2024-04-14T14:51:22.342 INFO:root:teuthology version: 0.0.1.dev267+g6c63784 2024-04-14T14:51:22.342 DEBUG:teuthology.run:Teuthology command: teuthology --description orch/cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v1only root} --verbose --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2024-04-08_20:08:14-orch-main-distro-default-smithi/7646059 --name teuthology-2024-04-08_20:08:14-orch-main-distro-default-smithi -- /home/teuthworker/archive/teuthology-2024-04-08_20:08:14-orch-main-distro-default-smithi/7646059/orig.config.yaml 2024-04-14T14:51:22.388 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-14T14:51:22.477 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-08_20:08:14-orch-main-distro-default-smithi/7646059 branch: main description: orch/cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v1only root} email: ceph-qa@ceph.com first_in_suite: false job_id: '7646059' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-04-08_20:08:14-orch-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug ms: 1 debug objecter: 20 debug rados: 20 global: ms bind msgr2: false ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mgrc: 20 debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon warn on pool no app: false paxos service trim min: 10 osd: debug ms: 1 debug osd: 20 osd class default list: '*' osd class load list: '*' osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd max backfills: 3 osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 crush_tunables: jewel flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - reached quota - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) mon_bind_msgr2: false sha1: a971fd5f5719038c70b817bafacd6d5400b1d888 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: cephadm_mode: root install: ceph: flavor: default sha1: a971fd5f5719038c70b817bafacd6d5400b1d888 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: main sha1: 05c9d46e7de5db2cb434c177a6bbfc56f472d09b owner: scheduled_teuthology@teuthology priority: 950 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 6509 sha1: a971fd5f5719038c70b817bafacd6d5400b1d888 sleep_before_teardown: 0 subset: 21/64 suite: orch suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph_05c9d46e7de5db2cb434c177a6bbfc56f472d09b/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 05c9d46e7de5db2cb434c177a6bbfc56f472d09b targets: smithi090.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF4r8vt0uUbDwufn8Oy7xXkk8cNtSGrn7iFpqL1+81e2O4nTg2MD0Wo8y2GwsCJ7wc/Tey3H3cn9h7qhcBDldRo= smithi092.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLpstLYsuLzhOW3itcBt92l/uZdJxnpz1kSmx0N5abFELPL7u07DE0DV9prrJGKdyyTPwim2WmYIu/WHGK4FB/U= tasks: - pexec: all: - sudo dnf install runc -y - sudo sed -i 's/^#runtime = "crun"/runtime = "runc"/g' /usr/share/containers/containers.conf - sudo sed -i 's/runtime = "crun"/#runtime = "crun"/g' /usr/share/containers/containers.conf - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - workunit: clients: client.0: - rados/test.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6c637841c215537a4502385240412f1966e0faab timestamp: 2024-04-08_20:08:14 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-14T14:51:22.477 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_05c9d46e7de5db2cb434c177a6bbfc56f472d09b/qa; will attempt to use it 2024-04-14T14:51:22.478 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_05c9d46e7de5db2cb434c177a6bbfc56f472d09b/qa/tasks 2024-04-14T14:51:22.478 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-14T14:51:22.480 INFO:teuthology.task.internal:Checking packages... 2024-04-14T14:51:22.505 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'a971fd5f5719038c70b817bafacd6d5400b1d888' 2024-04-14T14:51:22.505 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-14T14:51:22.505 INFO:teuthology.packaging:ref: None 2024-04-14T14:51:22.505 INFO:teuthology.packaging:tag: None 2024-04-14T14:51:22.505 INFO:teuthology.packaging:branch: main 2024-04-14T14:51:22.505 INFO:teuthology.packaging:sha1: a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:51:22.505 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-04-14T14:51:22.901 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3080.gc2d8c315 2024-04-14T14:51:22.902 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-14T14:51:22.911 INFO:teuthology.task.internal:no buildpackages task found 2024-04-14T14:51:22.912 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-14T14:51:22.918 INFO:teuthology.task.internal:Saving configuration 2024-04-14T14:51:22.930 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-14T14:51:22.936 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-14T14:51:22.961 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi090.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-08_20:08:14-orch-main-distro-default-smithi/7646059', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-14 14:45:31.756234', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF4r8vt0uUbDwufn8Oy7xXkk8cNtSGrn7iFpqL1+81e2O4nTg2MD0Wo8y2GwsCJ7wc/Tey3H3cn9h7qhcBDldRo='} 2024-04-14T14:51:22.982 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi092.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-08_20:08:14-orch-main-distro-default-smithi/7646059', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-14 14:45:31.758510', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLpstLYsuLzhOW3itcBt92l/uZdJxnpz1kSmx0N5abFELPL7u07DE0DV9prrJGKdyyTPwim2WmYIu/WHGK4FB/U='} 2024-04-14T14:51:22.982 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-14T14:51:22.988 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-04-14T14:51:22.989 INFO:teuthology.task.internal:roles: ubuntu@smithi092.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-04-14T14:51:22.989 INFO:teuthology.run_tasks:Running task console_log... 2024-04-14T14:51:23.062 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fec1fe8b0d0>, signals=[15]) 2024-04-14T14:51:23.062 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-14T14:51:23.068 INFO:teuthology.task.internal:Opening connections... 2024-04-14T14:51:23.069 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi090.front.sepia.ceph.com 2024-04-14T14:51:23.070 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-14T14:51:23.165 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi092.front.sepia.ceph.com 2024-04-14T14:51:23.166 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-14T14:51:23.256 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-14T14:51:23.264 DEBUG:teuthology.orchestra.run.smithi090:> uname -m 2024-04-14T14:51:23.281 INFO:teuthology.orchestra.run.smithi090.stdout:x86_64 2024-04-14T14:51:23.282 DEBUG:teuthology.orchestra.run.smithi090:> cat /etc/os-release 2024-04-14T14:51:23.338 INFO:teuthology.orchestra.run.smithi090.stdout:NAME="CentOS Stream" 2024-04-14T14:51:23.338 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION="9" 2024-04-14T14:51:23.338 INFO:teuthology.orchestra.run.smithi090.stdout:ID="centos" 2024-04-14T14:51:23.338 INFO:teuthology.orchestra.run.smithi090.stdout:ID_LIKE="rhel fedora" 2024-04-14T14:51:23.338 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION_ID="9" 2024-04-14T14:51:23.339 INFO:teuthology.orchestra.run.smithi090.stdout:PLATFORM_ID="platform:el9" 2024-04-14T14:51:23.339 INFO:teuthology.orchestra.run.smithi090.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-14T14:51:23.339 INFO:teuthology.orchestra.run.smithi090.stdout:ANSI_COLOR="0;31" 2024-04-14T14:51:23.339 INFO:teuthology.orchestra.run.smithi090.stdout:LOGO="fedora-logo-icon" 2024-04-14T14:51:23.339 INFO:teuthology.orchestra.run.smithi090.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-14T14:51:23.339 INFO:teuthology.orchestra.run.smithi090.stdout:HOME_URL="https://centos.org/" 2024-04-14T14:51:23.339 INFO:teuthology.orchestra.run.smithi090.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-14T14:51:23.339 INFO:teuthology.orchestra.run.smithi090.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-14T14:51:23.339 INFO:teuthology.orchestra.run.smithi090.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-14T14:51:23.340 INFO:teuthology.lock.ops:Updating smithi090.front.sepia.ceph.com on lock server 2024-04-14T14:51:23.370 DEBUG:teuthology.orchestra.run.smithi092:> uname -m 2024-04-14T14:51:23.388 INFO:teuthology.orchestra.run.smithi092.stdout:x86_64 2024-04-14T14:51:23.388 DEBUG:teuthology.orchestra.run.smithi092:> cat /etc/os-release 2024-04-14T14:51:23.445 INFO:teuthology.orchestra.run.smithi092.stdout:NAME="CentOS Stream" 2024-04-14T14:51:23.446 INFO:teuthology.orchestra.run.smithi092.stdout:VERSION="9" 2024-04-14T14:51:23.446 INFO:teuthology.orchestra.run.smithi092.stdout:ID="centos" 2024-04-14T14:51:23.446 INFO:teuthology.orchestra.run.smithi092.stdout:ID_LIKE="rhel fedora" 2024-04-14T14:51:23.446 INFO:teuthology.orchestra.run.smithi092.stdout:VERSION_ID="9" 2024-04-14T14:51:23.446 INFO:teuthology.orchestra.run.smithi092.stdout:PLATFORM_ID="platform:el9" 2024-04-14T14:51:23.446 INFO:teuthology.orchestra.run.smithi092.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-14T14:51:23.446 INFO:teuthology.orchestra.run.smithi092.stdout:ANSI_COLOR="0;31" 2024-04-14T14:51:23.446 INFO:teuthology.orchestra.run.smithi092.stdout:LOGO="fedora-logo-icon" 2024-04-14T14:51:23.446 INFO:teuthology.orchestra.run.smithi092.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-14T14:51:23.446 INFO:teuthology.orchestra.run.smithi092.stdout:HOME_URL="https://centos.org/" 2024-04-14T14:51:23.446 INFO:teuthology.orchestra.run.smithi092.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-14T14:51:23.446 INFO:teuthology.orchestra.run.smithi092.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-14T14:51:23.447 INFO:teuthology.orchestra.run.smithi092.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-14T14:51:23.447 INFO:teuthology.lock.ops:Updating smithi092.front.sepia.ceph.com on lock server 2024-04-14T14:51:23.470 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-14T14:51:23.480 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-14T14:51:23.485 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-14T14:51:23.486 DEBUG:teuthology.orchestra.run.smithi090:> test '!' -e /home/ubuntu/cephtest 2024-04-14T14:51:23.491 DEBUG:teuthology.orchestra.run.smithi092:> test '!' -e /home/ubuntu/cephtest 2024-04-14T14:51:23.507 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-14T14:51:23.513 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-14T14:51:23.513 DEBUG:teuthology.orchestra.run.smithi090:> test -z $(ls -A /var/lib/ceph) 2024-04-14T14:51:23.548 DEBUG:teuthology.orchestra.run.smithi092:> test -z $(ls -A /var/lib/ceph) 2024-04-14T14:51:23.632 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-14T14:51:23.688 INFO:teuthology.run_tasks:Running task kernel... 2024-04-14T14:51:23.702 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-14T14:51:23.702 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-14T14:51:23.702 DEBUG:teuthology.orchestra.run.smithi090:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-14T14:51:23.703 DEBUG:teuthology.orchestra.run.smithi092:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-14T14:51:23.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T14:51:23.719 DEBUG:teuthology.orchestra.run.smithi090:> uname -r 2024-04-14T14:51:23.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T14:51:23.720 DEBUG:teuthology.orchestra.run.smithi092:> uname -r 2024-04-14T14:51:23.776 INFO:teuthology.orchestra.run.smithi090.stdout:5.14.0-361.el9.x86_64 2024-04-14T14:51:23.776 INFO:teuthology.task.kernel:Running kernel on smithi090: 5.14.0-361.el9.x86_64 2024-04-14T14:51:23.777 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum install -y kernel 2024-04-14T14:51:23.778 INFO:teuthology.orchestra.run.smithi092.stdout:5.14.0-361.el9.x86_64 2024-04-14T14:51:23.778 INFO:teuthology.task.kernel:Running kernel on smithi092: 5.14.0-361.el9.x86_64 2024-04-14T14:51:23.778 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum install -y kernel 2024-04-14T14:51:26.294 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS-9 - AppStream 9.7 MB/s | 8.1 MB 00:00 2024-04-14T14:51:26.618 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS-9 - AppStream 9.0 MB/s | 8.1 MB 00:00 2024-04-14T14:51:28.210 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS-9 - BaseOS 4.4 MB/s | 2.5 MB 00:00 2024-04-14T14:51:28.585 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS-9 - BaseOS 3.8 MB/s | 2.5 MB 00:00 2024-04-14T14:51:29.684 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-04-14T14:51:30.644 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - BaseOS 4.1 MB/s | 8.0 MB 00:01 2024-04-14T14:51:34.819 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - AppStream 4.7 MB/s | 19 MB 00:04 2024-04-14T14:51:47.579 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - CRB 687 kB/s | 6.2 MB 00:09 2024-04-14T14:51:49.372 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - Extras packages 27 kB/s | 16 kB 00:00 2024-04-14T14:51:51.774 INFO:teuthology.orchestra.run.smithi090.stdout:Extra Packages for Enterprise Linux 9.3 MB/s | 21 MB 00:02 2024-04-14T14:51:56.841 INFO:teuthology.orchestra.run.smithi090.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-04-14T14:51:59.108 INFO:teuthology.orchestra.run.smithi090.stderr:Modular dependency problems: 2024-04-14T14:51:59.108 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-04-14T14:51:59.109 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 1: conflicting requests 2024-04-14T14:51:59.109 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.109 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 2: conflicting requests 2024-04-14T14:51:59.109 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.109 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 3: conflicting requests 2024-04-14T14:51:59.109 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.109 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 4: conflicting requests 2024-04-14T14:51:59.109 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.110 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 5: conflicting requests 2024-04-14T14:51:59.110 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.110 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 6: conflicting requests 2024-04-14T14:51:59.110 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.110 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 7: conflicting requests 2024-04-14T14:51:59.110 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.110 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 8: conflicting requests 2024-04-14T14:51:59.111 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.111 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 9: conflicting requests 2024-04-14T14:51:59.111 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.111 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 10: conflicting requests 2024-04-14T14:51:59.111 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.111 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 11: conflicting requests 2024-04-14T14:51:59.111 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.111 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 12: conflicting requests 2024-04-14T14:51:59.111 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.111 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 13: conflicting requests 2024-04-14T14:51:59.112 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.112 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 14: conflicting requests 2024-04-14T14:51:59.112 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.112 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 15: conflicting requests 2024-04-14T14:51:59.112 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.112 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 16: conflicting requests 2024-04-14T14:51:59.112 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.112 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 17: conflicting requests 2024-04-14T14:51:59.112 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.112 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 18: conflicting requests 2024-04-14T14:51:59.112 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.112 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 19: conflicting requests 2024-04-14T14:51:59.112 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.112 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 20: conflicting requests 2024-04-14T14:51:59.112 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.113 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 21: conflicting requests 2024-04-14T14:51:59.113 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.113 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 22: conflicting requests 2024-04-14T14:51:59.113 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.113 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 23: conflicting requests 2024-04-14T14:51:59.113 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.113 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 24: conflicting requests 2024-04-14T14:51:59.113 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.113 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 25: conflicting requests 2024-04-14T14:51:59.113 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.113 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 26: conflicting requests 2024-04-14T14:51:59.113 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.113 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 27: conflicting requests 2024-04-14T14:51:59.113 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.114 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 28: conflicting requests 2024-04-14T14:51:59.114 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.114 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 29: conflicting requests 2024-04-14T14:51:59.114 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.114 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 30: conflicting requests 2024-04-14T14:51:59.114 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.114 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 31: conflicting requests 2024-04-14T14:51:59.114 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.114 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 32: conflicting requests 2024-04-14T14:51:59.114 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.114 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 33: conflicting requests 2024-04-14T14:51:59.114 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.115 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 34: conflicting requests 2024-04-14T14:51:59.115 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.115 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 35: conflicting requests 2024-04-14T14:51:59.115 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.115 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 36: conflicting requests 2024-04-14T14:51:59.115 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.115 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 37: conflicting requests 2024-04-14T14:51:59.115 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.115 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.115 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.115 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 38: cannot install the best candidate for the job 2024-04-14T14:51:59.115 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.115 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.115 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.115 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.116 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 39: conflicting requests 2024-04-14T14:51:59.116 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.116 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.116 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.116 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.116 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.116 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.116 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 40: cannot install the best candidate for the job 2024-04-14T14:51:59.116 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.116 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.116 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.117 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.117 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 41: cannot install the best candidate for the job 2024-04-14T14:51:59.117 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.117 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.117 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.117 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.117 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 42: cannot install the best candidate for the job 2024-04-14T14:51:59.117 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.117 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.117 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.117 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.118 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 43: cannot install the best candidate for the job 2024-04-14T14:51:59.118 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.118 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.118 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.118 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.118 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 44: cannot install the best candidate for the job 2024-04-14T14:51:59.118 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.118 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.118 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.118 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.118 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 45: cannot install the best candidate for the job 2024-04-14T14:51:59.118 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.119 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.119 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.119 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-14T14:51:59.691 INFO:teuthology.orchestra.run.smithi090.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-14T14:51:59.691 INFO:teuthology.orchestra.run.smithi090.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-14T14:51:59.820 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-04-14T14:51:59.825 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-04-14T14:51:59.825 INFO:teuthology.orchestra.run.smithi090.stdout: Package Architecture Version Repository Size 2024-04-14T14:51:59.825 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-04-14T14:51:59.825 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-04-14T14:51:59.826 INFO:teuthology.orchestra.run.smithi090.stdout: kernel x86_64 5.14.0-435.el9 baseos 214 k 2024-04-14T14:51:59.826 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-04-14T14:51:59.826 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-core x86_64 5.14.0-435.el9 baseos 15 M 2024-04-14T14:51:59.826 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-modules x86_64 5.14.0-435.el9 baseos 34 M 2024-04-14T14:51:59.826 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-modules-core x86_64 5.14.0-435.el9 baseos 28 M 2024-04-14T14:51:59.826 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T14:51:59.826 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-04-14T14:51:59.826 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-04-14T14:51:59.826 INFO:teuthology.orchestra.run.smithi090.stdout:Install 4 Packages 2024-04-14T14:51:59.826 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T14:51:59.827 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 77 M 2024-04-14T14:51:59.827 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 123 M 2024-04-14T14:51:59.827 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-04-14T14:51:59.963 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - AppStream 685 kB/s | 19 MB 00:28 2024-04-14T14:52:00.227 INFO:teuthology.orchestra.run.smithi090.stdout:(1/4): kernel-5.14.0-435.el9.x86_64.rpm 718 kB/s | 214 kB 00:00 2024-04-14T14:52:01.636 INFO:teuthology.orchestra.run.smithi090.stdout:(2/4): kernel-core-5.14.0-435.el9.x86_64.rpm 8.9 MB/s | 15 MB 00:01 2024-04-14T14:52:02.677 INFO:teuthology.orchestra.run.smithi090.stdout:(3/4): kernel-modules-5.14.0-435.el9.x86_64.rpm 12 MB/s | 34 MB 00:02 2024-04-14T14:52:04.138 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.2 MB 00:00 2024-04-14T14:52:05.830 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - Extras packages 36 kB/s | 16 kB 00:00 2024-04-14T14:52:06.922 INFO:teuthology.orchestra.run.smithi090.stdout:(4/4): kernel-modules-core-5.14.0-435.el9.x86_6 4.1 MB/s | 28 MB 00:06 2024-04-14T14:52:06.923 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-04-14T14:52:06.923 INFO:teuthology.orchestra.run.smithi090.stdout:Total 11 MB/s | 77 MB 00:07 2024-04-14T14:52:07.590 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-04-14T14:52:08.005 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-04-14T14:52:08.005 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-04-14T14:52:09.033 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-04-14T14:52:09.033 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-04-14T14:52:09.056 INFO:teuthology.orchestra.run.smithi092.stdout:Extra Packages for Enterprise Linux 6.8 MB/s | 21 MB 00:03 2024-04-14T14:52:11.582 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-04-14T14:52:12.230 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : kernel-modules-core-5.14.0-435.el9.x86_64 1/4 2024-04-14T14:52:12.310 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : kernel-core-5.14.0-435.el9.x86_64 2/4 2024-04-14T14:52:14.064 INFO:teuthology.orchestra.run.smithi092.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-04-14T14:52:15.062 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: kernel-core-5.14.0-435.el9.x86_64 2/4 2024-04-14T14:52:15.195 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : kernel-modules-5.14.0-435.el9.x86_64 3/4 2024-04-14T14:52:16.310 INFO:teuthology.orchestra.run.smithi092.stderr:Modular dependency problems: 2024-04-14T14:52:16.310 INFO:teuthology.orchestra.run.smithi092.stderr: 2024-04-14T14:52:16.310 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 1: conflicting requests 2024-04-14T14:52:16.310 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.310 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 2: conflicting requests 2024-04-14T14:52:16.311 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.311 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 3: conflicting requests 2024-04-14T14:52:16.311 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.311 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 4: conflicting requests 2024-04-14T14:52:16.311 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.311 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 5: conflicting requests 2024-04-14T14:52:16.311 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.311 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 6: conflicting requests 2024-04-14T14:52:16.311 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.311 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 7: conflicting requests 2024-04-14T14:52:16.311 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.311 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 8: conflicting requests 2024-04-14T14:52:16.312 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.312 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 9: conflicting requests 2024-04-14T14:52:16.312 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.312 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 10: conflicting requests 2024-04-14T14:52:16.312 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.312 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 11: conflicting requests 2024-04-14T14:52:16.312 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.312 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 12: conflicting requests 2024-04-14T14:52:16.312 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.312 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 13: conflicting requests 2024-04-14T14:52:16.312 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.312 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 14: conflicting requests 2024-04-14T14:52:16.312 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.313 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 15: conflicting requests 2024-04-14T14:52:16.313 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.313 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 16: conflicting requests 2024-04-14T14:52:16.313 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.313 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 17: conflicting requests 2024-04-14T14:52:16.313 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.313 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 18: conflicting requests 2024-04-14T14:52:16.313 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.313 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 19: conflicting requests 2024-04-14T14:52:16.313 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.313 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 20: conflicting requests 2024-04-14T14:52:16.313 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.313 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 21: conflicting requests 2024-04-14T14:52:16.313 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.314 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 22: conflicting requests 2024-04-14T14:52:16.314 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.314 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 23: conflicting requests 2024-04-14T14:52:16.314 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.314 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 24: conflicting requests 2024-04-14T14:52:16.314 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.314 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 25: conflicting requests 2024-04-14T14:52:16.314 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.314 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 26: conflicting requests 2024-04-14T14:52:16.314 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.314 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 27: conflicting requests 2024-04-14T14:52:16.314 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.314 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 28: conflicting requests 2024-04-14T14:52:16.314 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.314 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 29: conflicting requests 2024-04-14T14:52:16.315 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.315 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 30: conflicting requests 2024-04-14T14:52:16.315 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.315 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 31: conflicting requests 2024-04-14T14:52:16.315 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.315 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 32: conflicting requests 2024-04-14T14:52:16.315 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.315 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 33: conflicting requests 2024-04-14T14:52:16.315 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.315 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 34: conflicting requests 2024-04-14T14:52:16.315 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.315 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 35: conflicting requests 2024-04-14T14:52:16.315 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.315 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 36: conflicting requests 2024-04-14T14:52:16.315 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.316 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 37: conflicting requests 2024-04-14T14:52:16.316 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.316 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.316 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.316 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 38: cannot install the best candidate for the job 2024-04-14T14:52:16.316 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.316 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.316 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.316 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.316 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 39: conflicting requests 2024-04-14T14:52:16.316 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.316 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.316 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.316 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.316 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.317 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.317 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 40: cannot install the best candidate for the job 2024-04-14T14:52:16.317 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.317 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.317 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.317 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.317 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 41: cannot install the best candidate for the job 2024-04-14T14:52:16.317 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.317 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.317 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.317 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.317 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 42: cannot install the best candidate for the job 2024-04-14T14:52:16.317 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.317 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.318 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.318 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.318 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 43: cannot install the best candidate for the job 2024-04-14T14:52:16.318 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.318 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.318 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.318 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.318 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 44: cannot install the best candidate for the job 2024-04-14T14:52:16.318 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.318 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.318 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.318 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.318 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 45: cannot install the best candidate for the job 2024-04-14T14:52:16.318 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.318 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.319 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.319 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-14T14:52:16.729 INFO:teuthology.orchestra.run.smithi092.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-14T14:52:16.729 INFO:teuthology.orchestra.run.smithi092.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-14T14:52:16.836 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2024-04-14T14:52:16.840 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-04-14T14:52:16.840 INFO:teuthology.orchestra.run.smithi092.stdout: Package Architecture Version Repository Size 2024-04-14T14:52:16.840 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-04-14T14:52:16.840 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2024-04-14T14:52:16.840 INFO:teuthology.orchestra.run.smithi092.stdout: kernel x86_64 5.14.0-435.el9 baseos 214 k 2024-04-14T14:52:16.841 INFO:teuthology.orchestra.run.smithi092.stdout:Installing dependencies: 2024-04-14T14:52:16.841 INFO:teuthology.orchestra.run.smithi092.stdout: kernel-core x86_64 5.14.0-435.el9 baseos 15 M 2024-04-14T14:52:16.841 INFO:teuthology.orchestra.run.smithi092.stdout: kernel-modules x86_64 5.14.0-435.el9 baseos 34 M 2024-04-14T14:52:16.841 INFO:teuthology.orchestra.run.smithi092.stdout: kernel-modules-core x86_64 5.14.0-435.el9 baseos 28 M 2024-04-14T14:52:16.841 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T14:52:16.841 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2024-04-14T14:52:16.841 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-04-14T14:52:16.841 INFO:teuthology.orchestra.run.smithi092.stdout:Install 4 Packages 2024-04-14T14:52:16.841 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T14:52:16.842 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 77 M 2024-04-14T14:52:16.842 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 123 M 2024-04-14T14:52:16.842 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2024-04-14T14:52:17.423 INFO:teuthology.orchestra.run.smithi092.stdout:(1/4): kernel-5.14.0-435.el9.x86_64.rpm 455 kB/s | 214 kB 00:00 2024-04-14T14:52:19.356 INFO:teuthology.orchestra.run.smithi092.stdout:(2/4): kernel-core-5.14.0-435.el9.x86_64.rpm 6.3 MB/s | 15 MB 00:02 2024-04-14T14:52:19.889 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: kernel-modules-5.14.0-435.el9.x86_64 3/4 2024-04-14T14:52:19.990 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : kernel-5.14.0-435.el9.x86_64 4/4 2024-04-14T14:52:20.248 INFO:teuthology.orchestra.run.smithi092.stdout:(3/4): kernel-modules-core-5.14.0-435.el9.x86_6 9.8 MB/s | 28 MB 00:02 2024-04-14T14:52:24.728 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: kernel-modules-core-5.14.0-435.el9.x86_64 4/4 2024-04-14T14:52:25.806 INFO:teuthology.orchestra.run.smithi092.stdout:(4/4): kernel-modules-5.14.0-435.el9.x86_64.rpm 3.8 MB/s | 34 MB 00:08 2024-04-14T14:52:25.808 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2024-04-14T14:52:25.809 INFO:teuthology.orchestra.run.smithi092.stdout:Total 8.5 MB/s | 77 MB 00:08 2024-04-14T14:52:26.474 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2024-04-14T14:52:26.845 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2024-04-14T14:52:26.846 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2024-04-14T14:52:27.852 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2024-04-14T14:52:27.853 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2024-04-14T14:52:30.312 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2024-04-14T14:52:31.014 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : kernel-modules-core-5.14.0-435.el9.x86_64 1/4 2024-04-14T14:52:31.105 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : kernel-core-5.14.0-435.el9.x86_64 2/4 2024-04-14T14:52:33.858 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: kernel-core-5.14.0-435.el9.x86_64 2/4 2024-04-14T14:52:33.991 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : kernel-modules-5.14.0-435.el9.x86_64 3/4 2024-04-14T14:52:38.706 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: kernel-modules-5.14.0-435.el9.x86_64 3/4 2024-04-14T14:52:38.826 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : kernel-5.14.0-435.el9.x86_64 4/4 2024-04-14T14:52:43.550 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: kernel-modules-core-5.14.0-435.el9.x86_64 4/4 2024-04-14T14:52:55.028 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: kernel-core-5.14.0-435.el9.x86_64 4/4 2024-04-14T14:52:55.028 INFO:teuthology.orchestra.run.smithi090.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-14T14:52:55.028 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T14:52:55.073 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: kernel-modules-5.14.0-435.el9.x86_64 4/4 2024-04-14T14:52:55.582 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: kernel-5.14.0-435.el9.x86_64 4/4 2024-04-14T14:52:55.582 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : kernel-5.14.0-435.el9.x86_64 1/4 2024-04-14T14:52:55.582 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : kernel-core-5.14.0-435.el9.x86_64 2/4 2024-04-14T14:52:55.582 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : kernel-modules-5.14.0-435.el9.x86_64 3/4 2024-04-14T14:52:56.039 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : kernel-modules-core-5.14.0-435.el9.x86_64 4/4 2024-04-14T14:52:56.040 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T14:52:56.040 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-04-14T14:52:56.040 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-5.14.0-435.el9.x86_64 2024-04-14T14:52:56.040 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-core-5.14.0-435.el9.x86_64 2024-04-14T14:52:56.040 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-modules-5.14.0-435.el9.x86_64 2024-04-14T14:52:56.040 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-modules-core-5.14.0-435.el9.x86_64 2024-04-14T14:52:56.040 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T14:52:56.040 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-04-14T14:52:56.437 DEBUG:teuthology.orchestra.run.smithi090:> rpm -q kernel | sort -rV | head -n 1 2024-04-14T14:52:56.550 INFO:teuthology.orchestra.run.smithi090.stdout:kernel-5.14.0-435.el9.x86_64 2024-04-14T14:52:56.551 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-435.el9.x86_64 2024-04-14T14:52:56.551 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-435.el9.x86_64 2024-04-14T14:52:56.551 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-14T14:52:56.551 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-14T14:52:56.551 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-14T14:52:56.551 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi090.front.sepia.ceph.com, path=None, version=distro) 2024-04-14T14:52:56.551 DEBUG:teuthology.orchestra.run.smithi090:> rpm -q kernel | sort -rV | head -n 1 2024-04-14T14:52:56.630 INFO:teuthology.orchestra.run.smithi090.stdout:kernel-5.14.0-435.el9.x86_64 2024-04-14T14:52:56.631 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-435.el9.x86_64 2024-04-14T14:52:56.632 DEBUG:teuthology.orchestra.run.smithi090:> sudo rpm -qi grub2-tools 2024-04-14T14:52:56.665 INFO:teuthology.orchestra.run.smithi090.stdout:Name : grub2-tools 2024-04-14T14:52:56.665 INFO:teuthology.orchestra.run.smithi090.stdout:Epoch : 1 2024-04-14T14:52:56.665 INFO:teuthology.orchestra.run.smithi090.stdout:Version : 2.06 2024-04-14T14:52:56.665 INFO:teuthology.orchestra.run.smithi090.stdout:Release : 68.el9 2024-04-14T14:52:56.665 INFO:teuthology.orchestra.run.smithi090.stdout:Architecture: x86_64 2024-04-14T14:52:56.665 INFO:teuthology.orchestra.run.smithi090.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-14T14:52:56.665 INFO:teuthology.orchestra.run.smithi090.stdout:Group : Unspecified 2024-04-14T14:52:56.666 INFO:teuthology.orchestra.run.smithi090.stdout:Size : 8264254 2024-04-14T14:52:56.666 INFO:teuthology.orchestra.run.smithi090.stdout:License : GPLv3+ 2024-04-14T14:52:56.666 INFO:teuthology.orchestra.run.smithi090.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-14T14:52:56.666 INFO:teuthology.orchestra.run.smithi090.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-14T14:52:56.666 INFO:teuthology.orchestra.run.smithi090.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-14T14:52:56.666 INFO:teuthology.orchestra.run.smithi090.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-14T14:52:56.666 INFO:teuthology.orchestra.run.smithi090.stdout:Packager : builder@centos.org 2024-04-14T14:52:56.666 INFO:teuthology.orchestra.run.smithi090.stdout:Vendor : CentOS 2024-04-14T14:52:56.666 INFO:teuthology.orchestra.run.smithi090.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-14T14:52:56.666 INFO:teuthology.orchestra.run.smithi090.stdout:Summary : Support tools for GRUB. 2024-04-14T14:52:56.666 INFO:teuthology.orchestra.run.smithi090.stdout:Description : 2024-04-14T14:52:56.666 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T14:52:56.666 INFO:teuthology.orchestra.run.smithi090.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-14T14:52:56.666 INFO:teuthology.orchestra.run.smithi090.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-14T14:52:56.667 INFO:teuthology.orchestra.run.smithi090.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-14T14:52:56.667 INFO:teuthology.orchestra.run.smithi090.stdout:hardware devices. 2024-04-14T14:52:56.667 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T14:52:56.667 INFO:teuthology.orchestra.run.smithi090.stdout:This subpackage provides tools for support of all platforms. 2024-04-14T14:52:56.668 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-14T14:52:56.668 INFO:teuthology.task.kernel:Updating grub on smithi090 to boot 5.14.0-435.el9.x86_64 2024-04-14T14:52:56.668 DEBUG:teuthology.orchestra.run.smithi090:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-14T14:52:57.298 INFO:teuthology.orchestra.run.smithi090.stderr:Generating grub configuration file ... 2024-04-14T14:52:59.182 INFO:teuthology.orchestra.run.smithi090.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-14T14:52:59.203 INFO:teuthology.orchestra.run.smithi090.stderr:done 2024-04-14T14:52:59.205 DEBUG:teuthology.orchestra.run.smithi090:> mktemp 2024-04-14T14:52:59.236 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/tmp.B9vRajIoJO 2024-04-14T14:52:59.236 DEBUG:teuthology.orchestra.run.smithi090:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.B9vRajIoJO 2024-04-14T14:52:59.304 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 0666 /tmp/tmp.B9vRajIoJO 2024-04-14T14:52:59.452 DEBUG:teuthology.orchestra.remote:smithi090:/tmp/tmp.B9vRajIoJO is 6KB 2024-04-14T14:52:59.501 DEBUG:teuthology.orchestra.run.smithi090:> rm -fr /tmp/tmp.B9vRajIoJO 2024-04-14T14:52:59.516 DEBUG:teuthology.orchestra.run.smithi090:> sudo /bin/ls /boot/loader/entries || true 2024-04-14T14:52:59.581 INFO:teuthology.orchestra.run.smithi090.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-14T14:52:59.581 INFO:teuthology.orchestra.run.smithi090.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-14T14:52:59.581 INFO:teuthology.orchestra.run.smithi090.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-14T14:52:59.581 INFO:teuthology.orchestra.run.smithi090.stdout:15949e3c88704300882482eddfd7baae-5.14.0-435.el9.x86_64.conf 2024-04-14T14:52:59.582 DEBUG:teuthology.orchestra.run.smithi090:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-435.el9.x86_64 2024-04-14T14:52:59.735 DEBUG:teuthology.orchestra.run.smithi090:> sudo shutdown -r now 2024-04-14T14:53:12.573 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: kernel-core-5.14.0-435.el9.x86_64 4/4 2024-04-14T14:53:12.573 INFO:teuthology.orchestra.run.smithi092.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-14T14:53:12.573 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T14:53:12.618 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: kernel-modules-5.14.0-435.el9.x86_64 4/4 2024-04-14T14:53:12.961 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: kernel-5.14.0-435.el9.x86_64 4/4 2024-04-14T14:53:12.961 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : kernel-5.14.0-435.el9.x86_64 1/4 2024-04-14T14:53:12.962 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : kernel-core-5.14.0-435.el9.x86_64 2/4 2024-04-14T14:53:12.962 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : kernel-modules-5.14.0-435.el9.x86_64 3/4 2024-04-14T14:53:13.411 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : kernel-modules-core-5.14.0-435.el9.x86_64 4/4 2024-04-14T14:53:13.411 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T14:53:13.411 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2024-04-14T14:53:13.411 INFO:teuthology.orchestra.run.smithi092.stdout: kernel-5.14.0-435.el9.x86_64 2024-04-14T14:53:13.411 INFO:teuthology.orchestra.run.smithi092.stdout: kernel-core-5.14.0-435.el9.x86_64 2024-04-14T14:53:13.411 INFO:teuthology.orchestra.run.smithi092.stdout: kernel-modules-5.14.0-435.el9.x86_64 2024-04-14T14:53:13.411 INFO:teuthology.orchestra.run.smithi092.stdout: kernel-modules-core-5.14.0-435.el9.x86_64 2024-04-14T14:53:13.411 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T14:53:13.411 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2024-04-14T14:53:13.747 DEBUG:teuthology.orchestra.run.smithi092:> rpm -q kernel | sort -rV | head -n 1 2024-04-14T14:53:13.852 INFO:teuthology.orchestra.run.smithi092.stdout:kernel-5.14.0-435.el9.x86_64 2024-04-14T14:53:13.853 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-435.el9.x86_64 2024-04-14T14:53:13.853 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-435.el9.x86_64 2024-04-14T14:53:13.853 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-14T14:53:13.853 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-14T14:53:13.853 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-04-14T14:53:13.853 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi092.front.sepia.ceph.com, path=None, version=distro) 2024-04-14T14:53:13.853 DEBUG:teuthology.orchestra.run.smithi092:> rpm -q kernel | sort -rV | head -n 1 2024-04-14T14:53:13.933 INFO:teuthology.orchestra.run.smithi092.stdout:kernel-5.14.0-435.el9.x86_64 2024-04-14T14:53:13.933 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-435.el9.x86_64 2024-04-14T14:53:13.933 DEBUG:teuthology.orchestra.run.smithi092:> sudo rpm -qi grub2-tools 2024-04-14T14:53:13.976 INFO:teuthology.orchestra.run.smithi092.stdout:Name : grub2-tools 2024-04-14T14:53:13.976 INFO:teuthology.orchestra.run.smithi092.stdout:Epoch : 1 2024-04-14T14:53:13.976 INFO:teuthology.orchestra.run.smithi092.stdout:Version : 2.06 2024-04-14T14:53:13.976 INFO:teuthology.orchestra.run.smithi092.stdout:Release : 68.el9 2024-04-14T14:53:13.976 INFO:teuthology.orchestra.run.smithi092.stdout:Architecture: x86_64 2024-04-14T14:53:13.976 INFO:teuthology.orchestra.run.smithi092.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-14T14:53:13.976 INFO:teuthology.orchestra.run.smithi092.stdout:Group : Unspecified 2024-04-14T14:53:13.977 INFO:teuthology.orchestra.run.smithi092.stdout:Size : 8264254 2024-04-14T14:53:13.977 INFO:teuthology.orchestra.run.smithi092.stdout:License : GPLv3+ 2024-04-14T14:53:13.977 INFO:teuthology.orchestra.run.smithi092.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-14T14:53:13.977 INFO:teuthology.orchestra.run.smithi092.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-14T14:53:13.977 INFO:teuthology.orchestra.run.smithi092.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-14T14:53:13.977 INFO:teuthology.orchestra.run.smithi092.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-14T14:53:13.977 INFO:teuthology.orchestra.run.smithi092.stdout:Packager : builder@centos.org 2024-04-14T14:53:13.977 INFO:teuthology.orchestra.run.smithi092.stdout:Vendor : CentOS 2024-04-14T14:53:13.977 INFO:teuthology.orchestra.run.smithi092.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-14T14:53:13.977 INFO:teuthology.orchestra.run.smithi092.stdout:Summary : Support tools for GRUB. 2024-04-14T14:53:13.977 INFO:teuthology.orchestra.run.smithi092.stdout:Description : 2024-04-14T14:53:13.977 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T14:53:13.977 INFO:teuthology.orchestra.run.smithi092.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-14T14:53:13.977 INFO:teuthology.orchestra.run.smithi092.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-14T14:53:13.977 INFO:teuthology.orchestra.run.smithi092.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-14T14:53:13.978 INFO:teuthology.orchestra.run.smithi092.stdout:hardware devices. 2024-04-14T14:53:13.978 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T14:53:13.978 INFO:teuthology.orchestra.run.smithi092.stdout:This subpackage provides tools for support of all platforms. 2024-04-14T14:53:13.979 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-14T14:53:13.979 INFO:teuthology.task.kernel:Updating grub on smithi092 to boot 5.14.0-435.el9.x86_64 2024-04-14T14:53:13.979 DEBUG:teuthology.orchestra.run.smithi092:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-14T14:53:14.573 INFO:teuthology.orchestra.run.smithi092.stderr:Generating grub configuration file ... 2024-04-14T14:53:15.940 INFO:teuthology.orchestra.run.smithi092.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-14T14:53:15.967 INFO:teuthology.orchestra.run.smithi092.stderr:done 2024-04-14T14:53:15.969 DEBUG:teuthology.orchestra.run.smithi092:> mktemp 2024-04-14T14:53:15.984 INFO:teuthology.orchestra.run.smithi092.stdout:/tmp/tmp.FJZzmgl8ie 2024-04-14T14:53:15.984 DEBUG:teuthology.orchestra.run.smithi092:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.FJZzmgl8ie 2024-04-14T14:53:16.062 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 0666 /tmp/tmp.FJZzmgl8ie 2024-04-14T14:53:16.205 DEBUG:teuthology.orchestra.remote:smithi092:/tmp/tmp.FJZzmgl8ie is 6KB 2024-04-14T14:53:16.253 DEBUG:teuthology.orchestra.run.smithi092:> rm -fr /tmp/tmp.FJZzmgl8ie 2024-04-14T14:53:16.267 DEBUG:teuthology.orchestra.run.smithi092:> sudo /bin/ls /boot/loader/entries || true 2024-04-14T14:53:16.331 INFO:teuthology.orchestra.run.smithi092.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-14T14:53:16.331 INFO:teuthology.orchestra.run.smithi092.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-14T14:53:16.331 INFO:teuthology.orchestra.run.smithi092.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-14T14:53:16.331 INFO:teuthology.orchestra.run.smithi092.stdout:15949e3c88704300882482eddfd7baae-5.14.0-435.el9.x86_64.conf 2024-04-14T14:53:16.333 DEBUG:teuthology.orchestra.run.smithi092:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-435.el9.x86_64 2024-04-14T14:53:16.486 DEBUG:teuthology.orchestra.run.smithi092:> sudo shutdown -r now 2024-04-14T14:53:29.751 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-14T14:53:29.751 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-04-14T14:53:29.752 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-14T14:53:46.503 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-04-14T14:53:46.504 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi092.front.sepia.ceph.com' 2024-04-14T14:53:46.505 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-14T14:54:04.319 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-04-14T14:54:13.326 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-04-14T14:54:13.327 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-14T14:54:16.382 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-04-14T14:54:28.392 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-04-14T14:54:28.393 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-14T14:54:46.524 DEBUG:teuthology.orchestra.remote:timed out 2024-04-14T14:54:46.818 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-04-14T14:54:55.534 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi092.front.sepia.ceph.com' 2024-04-14T14:54:55.535 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-14T14:54:55.872 DEBUG:teuthology.orchestra.run.smithi092:> true 2024-04-14T14:54:56.507 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi092.front.sepia.ceph.com' 2024-04-14T14:54:56.507 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-435.el9.x86_64"... 2024-04-14T14:54:56.507 DEBUG:teuthology.orchestra.run.smithi092:> uname -r 2024-04-14T14:54:56.521 INFO:teuthology.orchestra.run.smithi092.stdout:5.14.0-435.el9.x86_64 2024-04-14T14:54:56.521 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-435.el9.x86_64 vs 5.14.0-435.el9.x86_64 2024-04-14T14:54:56.521 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-14T14:54:56.521 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-14T14:54:57.522 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-14T14:54:57.522 DEBUG:teuthology.orchestra.run.smithi092:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-14T14:54:57.591 INFO:teuthology.orchestra.run.smithi092.stdout:ttyS1 2024-04-14T14:54:57.633 DEBUG:teuthology.parallel:result is None 2024-04-14T14:55:01.820 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-04-14T14:55:01.820 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-14T14:55:03.622 DEBUG:teuthology.orchestra.run.smithi090:> true 2024-04-14T14:55:04.307 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-04-14T14:55:04.307 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-435.el9.x86_64"... 2024-04-14T14:55:04.307 DEBUG:teuthology.orchestra.run.smithi090:> uname -r 2024-04-14T14:55:04.321 INFO:teuthology.orchestra.run.smithi090.stdout:5.14.0-435.el9.x86_64 2024-04-14T14:55:04.321 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-435.el9.x86_64 vs 5.14.0-435.el9.x86_64 2024-04-14T14:55:04.321 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-14T14:55:04.322 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-14T14:55:05.323 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-14T14:55:05.323 DEBUG:teuthology.orchestra.run.smithi090:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-14T14:55:05.427 INFO:teuthology.orchestra.run.smithi090.stdout:ttyS1 2024-04-14T14:55:05.475 DEBUG:teuthology.parallel:result is None 2024-04-14T14:55:05.475 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-14T14:55:05.483 INFO:teuthology.task.internal:Creating test directory... 2024-04-14T14:55:05.483 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-14T14:55:05.485 DEBUG:teuthology.orchestra.run.smithi092:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-14T14:55:05.503 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-14T14:55:05.509 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-14T14:55:05.517 INFO:teuthology.task.internal:Creating archive directory... 2024-04-14T14:55:05.517 DEBUG:teuthology.orchestra.run.smithi090:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-14T14:55:05.542 DEBUG:teuthology.orchestra.run.smithi092:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-14T14:55:05.590 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-14T14:55:05.597 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-14T14:55:05.598 DEBUG:teuthology.orchestra.run.smithi090:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-14T14:55:05.604 DEBUG:teuthology.orchestra.run.smithi092:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-14T14:55:05.639 INFO:teuthology.orchestra.run.smithi090.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-14T14:55:05.652 INFO:teuthology.orchestra.run.smithi090.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-14T14:55:05.668 INFO:teuthology.orchestra.run.smithi092.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-14T14:55:05.682 INFO:teuthology.orchestra.run.smithi092.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-14T14:55:05.684 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-14T14:55:05.691 INFO:teuthology.task.internal:Configuring sudo... 2024-04-14T14:55:05.691 DEBUG:teuthology.orchestra.run.smithi090:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-14T14:55:05.696 DEBUG:teuthology.orchestra.run.smithi092:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-14T14:55:05.753 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-14T14:55:05.795 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-14T14:55:05.795 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-14T14:55:05.800 DEBUG:teuthology.orchestra.run.smithi092:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-14T14:55:05.818 DEBUG:teuthology.orchestra.run.smithi090:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-14T14:55:05.905 DEBUG:teuthology.orchestra.run.smithi090:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-14T14:55:05.951 DEBUG:teuthology.orchestra.run.smithi090:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-14T14:55:06.046 DEBUG:teuthology.orchestra.run.smithi090:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-14T14:55:06.092 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-14T14:55:06.092 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-14T14:55:06.163 DEBUG:teuthology.orchestra.run.smithi092:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-14T14:55:06.207 DEBUG:teuthology.orchestra.run.smithi092:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-14T14:55:06.259 DEBUG:teuthology.orchestra.run.smithi092:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-14T14:55:06.345 DEBUG:teuthology.orchestra.run.smithi092:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-14T14:55:06.389 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-04-14T14:55:06.389 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-14T14:55:06.458 DEBUG:teuthology.orchestra.run.smithi090:> sudo service rsyslog restart 2024-04-14T14:55:06.460 DEBUG:teuthology.orchestra.run.smithi092:> sudo service rsyslog restart 2024-04-14T14:55:06.518 INFO:teuthology.orchestra.run.smithi090.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-14T14:55:06.562 INFO:teuthology.orchestra.run.smithi092.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-14T14:55:07.059 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-14T14:55:07.071 INFO:teuthology.task.internal:Starting timer... 2024-04-14T14:55:07.072 INFO:teuthology.run_tasks:Running task pcp... 2024-04-14T14:55:07.082 INFO:teuthology.run_tasks:Running task selinux... 2024-04-14T14:55:07.090 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-04-14T14:55:07.090 DEBUG:teuthology.orchestra.run.smithi090:> sudo service auditd rotate 2024-04-14T14:55:07.188 INFO:teuthology.orchestra.run.smithi090.stdout:Rotating logs: 2024-04-14T14:55:07.190 DEBUG:teuthology.orchestra.run.smithi092:> sudo service auditd rotate 2024-04-14T14:55:07.235 INFO:teuthology.orchestra.run.smithi092.stdout:Rotating logs: 2024-04-14T14:55:07.237 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-14T14:55:07.237 DEBUG:teuthology.orchestra.run.smithi090:> /usr/sbin/getenforce 2024-04-14T14:55:07.257 INFO:teuthology.orchestra.run.smithi090.stdout:Permissive 2024-04-14T14:55:07.257 DEBUG:teuthology.orchestra.run.smithi092:> /usr/sbin/getenforce 2024-04-14T14:55:07.311 INFO:teuthology.orchestra.run.smithi092.stdout:Permissive 2024-04-14T14:55:07.311 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi090.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi092.front.sepia.ceph.com': 'permissive'} 2024-04-14T14:55:07.311 DEBUG:teuthology.orchestra.run.smithi090:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-14T14:55:07.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T14:55:07.339 DEBUG:teuthology.orchestra.run.smithi092:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-14T14:55:07.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T14:55:07.379 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-14T14:55:07.380 DEBUG:teuthology.orchestra.run.smithi090:> sudo /usr/sbin/setenforce permissive 2024-04-14T14:55:07.415 DEBUG:teuthology.orchestra.run.smithi092:> sudo /usr/sbin/setenforce permissive 2024-04-14T14:55:07.454 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-14T14:55:07.464 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-14T14:55:07.469 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-14T14:55:07.585 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-14T14:55:07.600 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-14T14:55:07.601 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi090.front.sepia.ceph.com,smithi092.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-14T14:59:31.858 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi090.front.sepia.ceph.com'), Remote(name='ubuntu@smithi092.front.sepia.ceph.com')] 2024-04-14T14:59:31.860 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-04-14T14:59:31.861 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-14T14:59:31.955 DEBUG:teuthology.orchestra.run.smithi090:> true 2024-04-14T14:59:32.032 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-04-14T14:59:32.033 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi092.front.sepia.ceph.com' 2024-04-14T14:59:32.033 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-14T14:59:32.123 DEBUG:teuthology.orchestra.run.smithi092:> true 2024-04-14T14:59:32.205 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi092.front.sepia.ceph.com' 2024-04-14T14:59:32.206 INFO:teuthology.run_tasks:Running task clock... 2024-04-14T14:59:32.217 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-14T14:59:32.217 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-14T14:59:32.217 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-14T14:59:32.220 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-14T14:59:32.220 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-14T14:59:32.248 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-14T14:59:32.265 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-14T14:59:32.277 INFO:teuthology.orchestra.run.smithi092.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-14T14:59:32.293 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: ntpd: command not found 2024-04-14T14:59:32.294 INFO:teuthology.orchestra.run.smithi092.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-14T14:59:32.307 INFO:teuthology.orchestra.run.smithi090.stdout:506 Cannot talk to daemon 2024-04-14T14:59:32.323 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-14T14:59:32.324 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: ntpd: command not found 2024-04-14T14:59:32.337 INFO:teuthology.orchestra.run.smithi092.stdout:506 Cannot talk to daemon 2024-04-14T14:59:32.340 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-14T14:59:32.353 INFO:teuthology.orchestra.run.smithi092.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-14T14:59:32.370 INFO:teuthology.orchestra.run.smithi092.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-14T14:59:32.391 INFO:teuthology.orchestra.run.smithi090.stderr:bash: line 1: ntpq: command not found 2024-04-14T14:59:32.394 INFO:teuthology.orchestra.run.smithi090.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-14T14:59:32.394 INFO:teuthology.orchestra.run.smithi090.stdout:=============================================================================== 2024-04-14T14:59:32.394 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-14T14:59:32.394 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-14T14:59:32.394 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-14T14:59:32.394 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-14T14:59:32.422 INFO:teuthology.orchestra.run.smithi092.stderr:bash: line 1: ntpq: command not found 2024-04-14T14:59:32.425 INFO:teuthology.orchestra.run.smithi092.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-14T14:59:32.425 INFO:teuthology.orchestra.run.smithi092.stdout:=============================================================================== 2024-04-14T14:59:32.425 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-14T14:59:32.425 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-14T14:59:32.425 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-14T14:59:32.425 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-14T14:59:32.426 INFO:teuthology.run_tasks:Running task pexec... 2024-04-14T14:59:32.437 INFO:teuthology.task.pexec:Executing custom commands... 2024-04-14T14:59:32.437 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi090.front.sepia.ceph.com 2024-04-14T14:59:32.437 DEBUG:teuthology.orchestra.run.smithi090:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-04-14T14:59:32.438 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi092.front.sepia.ceph.com 2024-04-14T14:59:32.438 DEBUG:teuthology.orchestra.run.smithi092:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-04-14T14:59:33.053 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:00:52 ago on Sun 14 Apr 2024 02:58:40 PM UTC. 2024-04-14T14:59:33.085 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:00:57 ago on Sun 14 Apr 2024 02:58:35 PM UTC. 2024-04-14T14:59:33.342 INFO:teuthology.orchestra.run.smithi090.stderr:Modular dependency problems: 2024-04-14T14:59:33.342 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-04-14T14:59:33.342 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 1: conflicting requests 2024-04-14T14:59:33.342 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.342 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 2: conflicting requests 2024-04-14T14:59:33.342 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.343 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 3: conflicting requests 2024-04-14T14:59:33.343 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.343 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 4: conflicting requests 2024-04-14T14:59:33.343 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.343 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 5: conflicting requests 2024-04-14T14:59:33.343 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.343 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 6: conflicting requests 2024-04-14T14:59:33.343 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.343 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 7: conflicting requests 2024-04-14T14:59:33.343 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.343 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 8: conflicting requests 2024-04-14T14:59:33.343 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.343 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 9: conflicting requests 2024-04-14T14:59:33.343 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.343 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 10: conflicting requests 2024-04-14T14:59:33.344 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.344 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 11: conflicting requests 2024-04-14T14:59:33.344 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.344 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 12: conflicting requests 2024-04-14T14:59:33.344 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.344 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 13: conflicting requests 2024-04-14T14:59:33.344 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.344 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 14: conflicting requests 2024-04-14T14:59:33.344 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.344 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 15: conflicting requests 2024-04-14T14:59:33.344 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.344 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 16: conflicting requests 2024-04-14T14:59:33.344 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.344 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 17: conflicting requests 2024-04-14T14:59:33.344 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.345 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 18: conflicting requests 2024-04-14T14:59:33.345 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.345 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 19: conflicting requests 2024-04-14T14:59:33.345 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.345 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 20: conflicting requests 2024-04-14T14:59:33.345 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.345 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 21: conflicting requests 2024-04-14T14:59:33.345 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.345 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 22: conflicting requests 2024-04-14T14:59:33.345 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.345 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 23: conflicting requests 2024-04-14T14:59:33.345 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.345 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 24: conflicting requests 2024-04-14T14:59:33.345 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.345 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 25: conflicting requests 2024-04-14T14:59:33.346 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.346 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 26: conflicting requests 2024-04-14T14:59:33.346 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.346 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 27: conflicting requests 2024-04-14T14:59:33.346 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.346 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 28: conflicting requests 2024-04-14T14:59:33.346 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.346 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 29: conflicting requests 2024-04-14T14:59:33.346 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.346 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 30: conflicting requests 2024-04-14T14:59:33.346 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.346 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 31: conflicting requests 2024-04-14T14:59:33.346 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.346 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 32: conflicting requests 2024-04-14T14:59:33.346 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.347 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 33: conflicting requests 2024-04-14T14:59:33.347 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.347 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 34: conflicting requests 2024-04-14T14:59:33.347 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.347 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 35: conflicting requests 2024-04-14T14:59:33.347 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.347 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 36: conflicting requests 2024-04-14T14:59:33.347 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.347 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 37: conflicting requests 2024-04-14T14:59:33.347 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.347 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.347 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.347 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 38: cannot install the best candidate for the job 2024-04-14T14:59:33.347 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.348 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.348 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.348 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.348 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 39: conflicting requests 2024-04-14T14:59:33.348 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.348 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.348 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.348 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.348 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.348 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.348 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 40: cannot install the best candidate for the job 2024-04-14T14:59:33.348 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.348 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.348 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.348 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.349 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 41: cannot install the best candidate for the job 2024-04-14T14:59:33.349 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.349 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.349 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.349 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.349 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 42: cannot install the best candidate for the job 2024-04-14T14:59:33.349 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.349 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.349 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.349 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.349 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 43: cannot install the best candidate for the job 2024-04-14T14:59:33.349 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.349 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.349 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.349 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.349 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 44: cannot install the best candidate for the job 2024-04-14T14:59:33.350 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.350 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.350 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.350 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.350 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 45: cannot install the best candidate for the job 2024-04-14T14:59:33.350 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.350 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.350 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.350 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.371 INFO:teuthology.orchestra.run.smithi092.stderr:Modular dependency problems: 2024-04-14T14:59:33.371 INFO:teuthology.orchestra.run.smithi092.stderr: 2024-04-14T14:59:33.371 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 1: conflicting requests 2024-04-14T14:59:33.371 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.371 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 2: conflicting requests 2024-04-14T14:59:33.371 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.371 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 3: conflicting requests 2024-04-14T14:59:33.371 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.371 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 4: conflicting requests 2024-04-14T14:59:33.371 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.371 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 5: conflicting requests 2024-04-14T14:59:33.372 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.372 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 6: conflicting requests 2024-04-14T14:59:33.372 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.372 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 7: conflicting requests 2024-04-14T14:59:33.372 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.372 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 8: conflicting requests 2024-04-14T14:59:33.372 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.372 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 9: conflicting requests 2024-04-14T14:59:33.372 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.372 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 10: conflicting requests 2024-04-14T14:59:33.372 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.372 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 11: conflicting requests 2024-04-14T14:59:33.372 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.372 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 12: conflicting requests 2024-04-14T14:59:33.372 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.373 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 13: conflicting requests 2024-04-14T14:59:33.373 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.373 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 14: conflicting requests 2024-04-14T14:59:33.373 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.373 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 15: conflicting requests 2024-04-14T14:59:33.373 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.373 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 16: conflicting requests 2024-04-14T14:59:33.373 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.373 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 17: conflicting requests 2024-04-14T14:59:33.373 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.373 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 18: conflicting requests 2024-04-14T14:59:33.373 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.373 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 19: conflicting requests 2024-04-14T14:59:33.373 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.374 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 20: conflicting requests 2024-04-14T14:59:33.374 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.374 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 21: conflicting requests 2024-04-14T14:59:33.374 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.374 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 22: conflicting requests 2024-04-14T14:59:33.374 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.374 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 23: conflicting requests 2024-04-14T14:59:33.374 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.374 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 24: conflicting requests 2024-04-14T14:59:33.374 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.374 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 25: conflicting requests 2024-04-14T14:59:33.374 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.374 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 26: conflicting requests 2024-04-14T14:59:33.374 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.374 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 27: conflicting requests 2024-04-14T14:59:33.375 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.375 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 28: conflicting requests 2024-04-14T14:59:33.375 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.375 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 29: conflicting requests 2024-04-14T14:59:33.375 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.375 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 30: conflicting requests 2024-04-14T14:59:33.375 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.375 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 31: conflicting requests 2024-04-14T14:59:33.375 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.375 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 32: conflicting requests 2024-04-14T14:59:33.375 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.375 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 33: conflicting requests 2024-04-14T14:59:33.375 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.375 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 34: conflicting requests 2024-04-14T14:59:33.375 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.376 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 35: conflicting requests 2024-04-14T14:59:33.376 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.376 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 36: conflicting requests 2024-04-14T14:59:33.376 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.376 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 37: conflicting requests 2024-04-14T14:59:33.376 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.376 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.376 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.376 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 38: cannot install the best candidate for the job 2024-04-14T14:59:33.376 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.376 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.376 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.376 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.376 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 39: conflicting requests 2024-04-14T14:59:33.377 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.377 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.377 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.377 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.377 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.377 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.377 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 40: cannot install the best candidate for the job 2024-04-14T14:59:33.377 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.377 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.377 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.377 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.377 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 41: cannot install the best candidate for the job 2024-04-14T14:59:33.377 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.377 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.378 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.378 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.378 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 42: cannot install the best candidate for the job 2024-04-14T14:59:33.378 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.378 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.378 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.378 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.378 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 43: cannot install the best candidate for the job 2024-04-14T14:59:33.378 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.378 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.378 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.378 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.378 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 44: cannot install the best candidate for the job 2024-04-14T14:59:33.378 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.378 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.379 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.379 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.379 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 45: cannot install the best candidate for the job 2024-04-14T14:59:33.379 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.379 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.379 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.379 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-14T14:59:33.409 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-04-14T14:59:33.422 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-04-14T14:59:33.423 INFO:teuthology.orchestra.run.smithi090.stdout: Package Architecture Version Repository Size 2024-04-14T14:59:33.423 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-04-14T14:59:33.423 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-04-14T14:59:33.423 INFO:teuthology.orchestra.run.smithi090.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-04-14T14:59:33.423 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T14:59:33.423 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-04-14T14:59:33.423 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-04-14T14:59:33.423 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-04-14T14:59:33.423 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T14:59:33.423 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 3.1 M 2024-04-14T14:59:33.423 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 9.8 M 2024-04-14T14:59:33.423 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-04-14T14:59:33.438 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2024-04-14T14:59:33.439 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-04-14T14:59:33.439 INFO:teuthology.orchestra.run.smithi092.stdout: Package Architecture Version Repository Size 2024-04-14T14:59:33.439 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-04-14T14:59:33.440 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2024-04-14T14:59:33.440 INFO:teuthology.orchestra.run.smithi092.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-04-14T14:59:33.440 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T14:59:33.440 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2024-04-14T14:59:33.440 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-04-14T14:59:33.440 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2024-04-14T14:59:33.440 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T14:59:33.440 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 3.1 M 2024-04-14T14:59:33.440 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 9.8 M 2024-04-14T14:59:33.440 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2024-04-14T14:59:34.230 INFO:teuthology.orchestra.run.smithi090.stdout:runc-1.1.12-2.el9.x86_64.rpm 4.7 MB/s | 3.1 MB 00:00 2024-04-14T14:59:34.230 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-04-14T14:59:34.231 INFO:teuthology.orchestra.run.smithi090.stdout:Total 3.9 MB/s | 3.1 MB 00:00 2024-04-14T14:59:34.267 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-04-14T14:59:34.273 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-04-14T14:59:34.273 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-04-14T14:59:34.353 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-04-14T14:59:34.353 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-04-14T14:59:34.597 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-04-14T14:59:34.693 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-04-14T14:59:34.697 INFO:teuthology.orchestra.run.smithi092.stdout:runc-1.1.12-2.el9.x86_64.rpm 3.2 MB/s | 3.1 MB 00:00 2024-04-14T14:59:34.697 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2024-04-14T14:59:34.697 INFO:teuthology.orchestra.run.smithi092.stdout:Total 2.5 MB/s | 3.1 MB 00:01 2024-04-14T14:59:34.735 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2024-04-14T14:59:34.741 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2024-04-14T14:59:34.741 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2024-04-14T14:59:34.822 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2024-04-14T14:59:34.822 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2024-04-14T14:59:35.174 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2024-04-14T14:59:35.267 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-04-14T14:59:35.352 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-04-14T14:59:35.676 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-04-14T14:59:35.692 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-04-14T14:59:35.692 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T14:59:35.692 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-04-14T14:59:35.693 INFO:teuthology.orchestra.run.smithi090.stdout: runc-4:1.1.12-2.el9.x86_64 2024-04-14T14:59:35.693 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T14:59:35.693 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-04-14T14:59:35.999 DEBUG:teuthology.parallel:result is None 2024-04-14T14:59:36.008 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-04-14T14:59:36.008 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T14:59:36.009 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2024-04-14T14:59:36.009 INFO:teuthology.orchestra.run.smithi092.stdout: runc-4:1.1.12-2.el9.x86_64 2024-04-14T14:59:36.009 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T14:59:36.009 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2024-04-14T14:59:36.273 DEBUG:teuthology.parallel:result is None 2024-04-14T14:59:36.273 INFO:teuthology.run_tasks:Running task install... 2024-04-14T14:59:36.282 DEBUG:teuthology.task.install:project ceph 2024-04-14T14:59:36.282 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'a971fd5f5719038c70b817bafacd6d5400b1d888'}} 2024-04-14T14:59:36.283 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'a971fd5f5719038c70b817bafacd6d5400b1d888'} 2024-04-14T14:59:36.283 INFO:teuthology.task.install:Using flavor: default 2024-04-14T14:59:36.289 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-14T14:59:36.289 INFO:teuthology.task.install:extra packages: [] 2024-04-14T14:59:36.290 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'a971fd5f5719038c70b817bafacd6d5400b1d888', 'tag': None, 'wait_for_package': False} 2024-04-14T14:59:36.290 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:59:36.292 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'a971fd5f5719038c70b817bafacd6d5400b1d888', 'tag': None, 'wait_for_package': False} 2024-04-14T14:59:36.292 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:59:36.440 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/main/a971fd5f5719038c70b817bafacd6d5400b1d888/centos/9/flavors/default/ 2024-04-14T14:59:36.441 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2778.ga971fd5f 2024-04-14T14:59:36.460 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/main/a971fd5f5719038c70b817bafacd6d5400b1d888/centos/9/flavors/default/ 2024-04-14T14:59:36.460 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2778.ga971fd5f 2024-04-14T14:59:36.584 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/main/a971fd5f5719038c70b817bafacd6d5400b1d888/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/main/a971fd5f5719038c70b817bafacd6d5400b1d888/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/main/a971fd5f5719038c70b817bafacd6d5400b1d888/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-14T14:59:36.584 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-04-14T14:59:36.584 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-14T14:59:36.611 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-14T14:59:36.611 DEBUG:teuthology.orchestra.run.smithi092:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/a971fd5f5719038c70b817bafacd6d5400b1d888/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-14T14:59:36.645 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/main/a971fd5f5719038c70b817bafacd6d5400b1d888/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/main/a971fd5f5719038c70b817bafacd6d5400b1d888/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/main/a971fd5f5719038c70b817bafacd6d5400b1d888/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-14T14:59:36.645 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-14T14:59:36.645 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-14T14:59:36.673 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-14T14:59:36.673 DEBUG:teuthology.orchestra.run.smithi090:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/a971fd5f5719038c70b817bafacd6d5400b1d888/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-14T14:59:36.678 DEBUG:teuthology.orchestra.run.smithi092:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-14T14:59:36.740 DEBUG:teuthology.orchestra.run.smithi090:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-14T14:59:36.772 DEBUG:teuthology.orchestra.run.smithi092:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-14T14:59:36.797 INFO:teuthology.orchestra.run.smithi092.stdout:check_obsoletes = 1 2024-04-14T14:59:36.799 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum clean all 2024-04-14T14:59:36.821 DEBUG:teuthology.orchestra.run.smithi090:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-14T14:59:36.887 INFO:teuthology.orchestra.run.smithi090.stdout:check_obsoletes = 1 2024-04-14T14:59:36.889 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum clean all 2024-04-14T14:59:37.053 INFO:teuthology.orchestra.run.smithi092.stdout:74 files removed 2024-04-14T14:59:37.081 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-radosgw 2024-04-14T14:59:37.135 INFO:teuthology.orchestra.run.smithi090.stdout:74 files removed 2024-04-14T14:59:37.165 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-radosgw 2024-04-14T14:59:38.033 INFO:teuthology.orchestra.run.smithi090.stdout:ceph packages for x86_64 150 kB/s | 83 kB 00:00 2024-04-14T14:59:38.412 INFO:teuthology.orchestra.run.smithi092.stdout:ceph packages for x86_64 81 kB/s | 83 kB 00:01 2024-04-14T14:59:38.549 INFO:teuthology.orchestra.run.smithi090.stdout:ceph noarch packages 25 kB/s | 11 kB 00:00 2024-04-14T14:59:38.942 INFO:teuthology.orchestra.run.smithi090.stdout:ceph source packages 5.2 kB/s | 1.9 kB 00:00 2024-04-14T14:59:39.496 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS-9 - AppStream 16 MB/s | 8.1 MB 00:00 2024-04-14T14:59:39.955 INFO:teuthology.orchestra.run.smithi092.stdout:ceph noarch packages 7.7 kB/s | 11 kB 00:01 2024-04-14T14:59:40.355 INFO:teuthology.orchestra.run.smithi092.stdout:ceph source packages 5.2 kB/s | 1.9 kB 00:00 2024-04-14T14:59:41.324 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS-9 - AppStream 8.7 MB/s | 8.1 MB 00:00 2024-04-14T14:59:41.522 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS-9 - BaseOS 3.5 MB/s | 2.5 MB 00:00 2024-04-14T14:59:42.521 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-04-14T14:59:43.424 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS-9 - BaseOS 3.2 MB/s | 2.5 MB 00:00 2024-04-14T14:59:44.867 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - BaseOS 8.1 MB/s | 8.0 MB 00:00 2024-04-14T14:59:45.916 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - AppStream 8.2 MB/s | 19 MB 00:02 2024-04-14T14:59:46.633 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - AppStream 28 MB/s | 19 MB 00:00 2024-04-14T14:59:50.368 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - CRB 6.9 MB/s | 6.2 MB 00:00 2024-04-14T14:59:51.300 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - CRB 7.1 MB/s | 6.2 MB 00:00 2024-04-14T14:59:51.900 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - Extras packages 47 kB/s | 16 kB 00:00 2024-04-14T14:59:52.742 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - Extras packages 64 kB/s | 16 kB 00:00 2024-04-14T14:59:53.789 INFO:teuthology.orchestra.run.smithi090.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-04-14T14:59:56.226 INFO:teuthology.orchestra.run.smithi092.stdout:Extra Packages for Enterprise Linux 6.2 MB/s | 21 MB 00:03 2024-04-14T14:59:58.854 INFO:teuthology.orchestra.run.smithi090.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-04-14T15:00:01.310 INFO:teuthology.orchestra.run.smithi092.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-04-14T15:00:01.414 INFO:teuthology.orchestra.run.smithi090.stderr:Modular dependency problems: 2024-04-14T15:00:01.414 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-04-14T15:00:01.414 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 1: conflicting requests 2024-04-14T15:00:01.414 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.415 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 2: conflicting requests 2024-04-14T15:00:01.415 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.415 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 3: conflicting requests 2024-04-14T15:00:01.415 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.415 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 4: conflicting requests 2024-04-14T15:00:01.415 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.415 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 5: conflicting requests 2024-04-14T15:00:01.415 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.415 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 6: conflicting requests 2024-04-14T15:00:01.415 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.415 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 7: conflicting requests 2024-04-14T15:00:01.415 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.416 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 8: conflicting requests 2024-04-14T15:00:01.416 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.416 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 9: conflicting requests 2024-04-14T15:00:01.416 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.416 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 10: conflicting requests 2024-04-14T15:00:01.416 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.416 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 11: conflicting requests 2024-04-14T15:00:01.416 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.416 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 12: conflicting requests 2024-04-14T15:00:01.416 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.416 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 13: conflicting requests 2024-04-14T15:00:01.416 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.417 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 14: conflicting requests 2024-04-14T15:00:01.417 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.417 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 15: conflicting requests 2024-04-14T15:00:01.417 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.417 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 16: conflicting requests 2024-04-14T15:00:01.417 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.417 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 17: conflicting requests 2024-04-14T15:00:01.417 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.417 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 18: conflicting requests 2024-04-14T15:00:01.417 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.418 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 19: conflicting requests 2024-04-14T15:00:01.418 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.418 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 20: conflicting requests 2024-04-14T15:00:01.418 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.418 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 21: conflicting requests 2024-04-14T15:00:01.418 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.418 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 22: conflicting requests 2024-04-14T15:00:01.418 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.418 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 23: conflicting requests 2024-04-14T15:00:01.418 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.418 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 24: conflicting requests 2024-04-14T15:00:01.419 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.419 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 25: conflicting requests 2024-04-14T15:00:01.419 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.419 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 26: conflicting requests 2024-04-14T15:00:01.419 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.419 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 27: conflicting requests 2024-04-14T15:00:01.419 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.419 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 28: conflicting requests 2024-04-14T15:00:01.419 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.419 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 29: conflicting requests 2024-04-14T15:00:01.419 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.419 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 30: conflicting requests 2024-04-14T15:00:01.419 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.420 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 31: conflicting requests 2024-04-14T15:00:01.420 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.420 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 32: conflicting requests 2024-04-14T15:00:01.420 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.420 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 33: conflicting requests 2024-04-14T15:00:01.420 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.420 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 34: conflicting requests 2024-04-14T15:00:01.420 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.420 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 35: conflicting requests 2024-04-14T15:00:01.420 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.420 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 36: conflicting requests 2024-04-14T15:00:01.420 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.420 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 37: conflicting requests 2024-04-14T15:00:01.421 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.421 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.421 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.421 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 38: cannot install the best candidate for the job 2024-04-14T15:00:01.421 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.421 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.421 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.421 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.421 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 39: conflicting requests 2024-04-14T15:00:01.421 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.422 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.422 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.422 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.422 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.422 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.422 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 40: cannot install the best candidate for the job 2024-04-14T15:00:01.422 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.422 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.422 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.422 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.423 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 41: cannot install the best candidate for the job 2024-04-14T15:00:01.423 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.423 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.423 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.423 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.423 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 42: cannot install the best candidate for the job 2024-04-14T15:00:01.423 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.423 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.423 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.424 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.424 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 43: cannot install the best candidate for the job 2024-04-14T15:00:01.424 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.424 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.424 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.424 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.424 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 44: cannot install the best candidate for the job 2024-04-14T15:00:01.424 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.424 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.424 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.424 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.424 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 45: cannot install the best candidate for the job 2024-04-14T15:00:01.424 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.425 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.425 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.425 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-14T15:00:01.976 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-04-14T15:00:01.978 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-04-14T15:00:01.978 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-04-14T15:00:01.978 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-04-14T15:00:01.979 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-04-14T15:00:01.979 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-radosgw x86_64 2:19.0.0-2778.ga971fd5f.el9 ceph 10 M 2024-04-14T15:00:01.979 INFO:teuthology.orchestra.run.smithi090.stdout:Upgrading: 2024-04-14T15:00:01.979 INFO:teuthology.orchestra.run.smithi090.stdout: librados2 x86_64 2:19.0.0-2778.ga971fd5f.el9 ceph 3.4 M 2024-04-14T15:00:01.979 INFO:teuthology.orchestra.run.smithi090.stdout: librbd1 x86_64 2:19.0.0-2778.ga971fd5f.el9 ceph 3.1 M 2024-04-14T15:00:01.979 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-04-14T15:00:01.979 INFO:teuthology.orchestra.run.smithi090.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-14T15:00:01.979 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-base x86_64 2:19.0.0-2778.ga971fd5f.el9 ceph 5.4 M 2024-04-14T15:00:01.979 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-common x86_64 2:19.0.0-2778.ga971fd5f.el9 ceph 21 M 2024-04-14T15:00:01.979 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-selinux x86_64 2:19.0.0-2778.ga971fd5f.el9 ceph 25 k 2024-04-14T15:00:01.979 INFO:teuthology.orchestra.run.smithi090.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-14T15:00:01.979 INFO:teuthology.orchestra.run.smithi090.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-14T15:00:01.979 INFO:teuthology.orchestra.run.smithi090.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-14T15:00:01.980 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs2 x86_64 2:19.0.0-2778.ga971fd5f.el9 ceph 746 k 2024-04-14T15:00:01.980 INFO:teuthology.orchestra.run.smithi090.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-14T15:00:01.980 INFO:teuthology.orchestra.run.smithi090.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-14T15:00:01.980 INFO:teuthology.orchestra.run.smithi090.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-14T15:00:01.980 INFO:teuthology.orchestra.run.smithi090.stdout: libradosstriper1 x86_64 2:19.0.0-2778.ga971fd5f.el9 ceph 501 k 2024-04-14T15:00:01.980 INFO:teuthology.orchestra.run.smithi090.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-14T15:00:01.980 INFO:teuthology.orchestra.run.smithi090.stdout: librgw2 x86_64 2:19.0.0-2778.ga971fd5f.el9 ceph 5.2 M 2024-04-14T15:00:01.980 INFO:teuthology.orchestra.run.smithi090.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-14T15:00:01.980 INFO:teuthology.orchestra.run.smithi090.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-14T15:00:01.980 INFO:teuthology.orchestra.run.smithi090.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-14T15:00:01.980 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-argparse x86_64 2:19.0.0-2778.ga971fd5f.el9 ceph 45 k 2024-04-14T15:00:01.980 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-common x86_64 2:19.0.0-2778.ga971fd5f.el9 ceph 135 k 2024-04-14T15:00:01.980 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cephfs x86_64 2:19.0.0-2778.ga971fd5f.el9 ceph 161 k 2024-04-14T15:00:01.980 INFO:teuthology.orchestra.run.smithi090.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-14T15:00:01.980 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rados x86_64 2:19.0.0-2778.ga971fd5f.el9 ceph 321 k 2024-04-14T15:00:01.981 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rbd x86_64 2:19.0.0-2778.ga971fd5f.el9 ceph 298 k 2024-04-14T15:00:01.981 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rgw x86_64 2:19.0.0-2778.ga971fd5f.el9 ceph 99 k 2024-04-14T15:00:01.981 INFO:teuthology.orchestra.run.smithi090.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-14T15:00:01.981 INFO:teuthology.orchestra.run.smithi090.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-14T15:00:01.981 INFO:teuthology.orchestra.run.smithi090.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-14T15:00:01.981 INFO:teuthology.orchestra.run.smithi090.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-14T15:00:01.981 INFO:teuthology.orchestra.run.smithi090.stdout:Installing weak dependencies: 2024-04-14T15:00:01.981 INFO:teuthology.orchestra.run.smithi090.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-14T15:00:01.981 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:00:01.981 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-04-14T15:00:01.981 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-04-14T15:00:01.981 INFO:teuthology.orchestra.run.smithi090.stdout:Install 30 Packages 2024-04-14T15:00:01.981 INFO:teuthology.orchestra.run.smithi090.stdout:Upgrade 2 Packages 2024-04-14T15:00:01.982 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:00:01.982 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 59 M 2024-04-14T15:00:01.982 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-04-14T15:00:03.880 INFO:teuthology.orchestra.run.smithi092.stderr:Modular dependency problems: 2024-04-14T15:00:03.881 INFO:teuthology.orchestra.run.smithi092.stderr: 2024-04-14T15:00:03.881 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 1: conflicting requests 2024-04-14T15:00:03.881 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.881 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 2: conflicting requests 2024-04-14T15:00:03.881 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.881 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 3: conflicting requests 2024-04-14T15:00:03.881 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.881 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 4: conflicting requests 2024-04-14T15:00:03.881 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.881 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 5: conflicting requests 2024-04-14T15:00:03.881 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.881 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 6: conflicting requests 2024-04-14T15:00:03.881 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.882 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 7: conflicting requests 2024-04-14T15:00:03.882 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.882 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 8: conflicting requests 2024-04-14T15:00:03.882 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.882 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 9: conflicting requests 2024-04-14T15:00:03.882 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.882 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 10: conflicting requests 2024-04-14T15:00:03.882 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.882 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 11: conflicting requests 2024-04-14T15:00:03.882 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.882 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 12: conflicting requests 2024-04-14T15:00:03.882 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.882 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 13: conflicting requests 2024-04-14T15:00:03.883 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.883 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 14: conflicting requests 2024-04-14T15:00:03.883 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.883 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 15: conflicting requests 2024-04-14T15:00:03.883 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.883 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 16: conflicting requests 2024-04-14T15:00:03.883 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.883 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 17: conflicting requests 2024-04-14T15:00:03.883 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.883 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 18: conflicting requests 2024-04-14T15:00:03.883 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.883 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 19: conflicting requests 2024-04-14T15:00:03.883 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.883 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 20: conflicting requests 2024-04-14T15:00:03.884 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.884 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 21: conflicting requests 2024-04-14T15:00:03.884 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.884 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 22: conflicting requests 2024-04-14T15:00:03.884 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.884 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 23: conflicting requests 2024-04-14T15:00:03.884 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.884 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 24: conflicting requests 2024-04-14T15:00:03.884 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.884 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 25: conflicting requests 2024-04-14T15:00:03.884 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.884 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 26: conflicting requests 2024-04-14T15:00:03.884 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.884 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 27: conflicting requests 2024-04-14T15:00:03.885 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.885 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 28: conflicting requests 2024-04-14T15:00:03.885 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.885 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 29: conflicting requests 2024-04-14T15:00:03.885 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.885 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 30: conflicting requests 2024-04-14T15:00:03.885 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.885 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 31: conflicting requests 2024-04-14T15:00:03.885 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.885 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 32: conflicting requests 2024-04-14T15:00:03.885 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.885 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 33: conflicting requests 2024-04-14T15:00:03.885 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.885 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 34: conflicting requests 2024-04-14T15:00:03.885 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.886 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 35: conflicting requests 2024-04-14T15:00:03.886 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.886 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 36: conflicting requests 2024-04-14T15:00:03.886 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.886 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 37: conflicting requests 2024-04-14T15:00:03.886 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.886 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.886 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.886 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 38: cannot install the best candidate for the job 2024-04-14T15:00:03.886 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.886 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.886 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.886 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.886 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 39: conflicting requests 2024-04-14T15:00:03.887 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.887 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.887 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.887 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.887 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.887 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.887 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 40: cannot install the best candidate for the job 2024-04-14T15:00:03.887 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.887 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.887 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.887 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.887 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 41: cannot install the best candidate for the job 2024-04-14T15:00:03.887 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.887 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.887 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.888 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.888 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 42: cannot install the best candidate for the job 2024-04-14T15:00:03.888 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.888 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.888 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.888 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.888 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 43: cannot install the best candidate for the job 2024-04-14T15:00:03.888 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.888 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.888 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.888 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.888 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 44: cannot install the best candidate for the job 2024-04-14T15:00:03.888 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.888 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.889 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.889 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.889 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 45: cannot install the best candidate for the job 2024-04-14T15:00:03.889 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.889 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.889 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.889 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-14T15:00:03.997 INFO:teuthology.orchestra.run.smithi090.stdout:(1/32): ceph-common-19.0.0-2778.ga971fd5f.el9.x 13 MB/s | 21 MB 00:01 2024-04-14T15:00:04.089 INFO:teuthology.orchestra.run.smithi090.stdout:(2/32): ceph-base-19.0.0-2778.ga971fd5f.el9.x86 3.2 MB/s | 5.4 MB 00:01 2024-04-14T15:00:04.114 INFO:teuthology.orchestra.run.smithi090.stdout:(3/32): ceph-selinux-19.0.0-2778.ga971fd5f.el9. 215 kB/s | 25 kB 00:00 2024-04-14T15:00:04.173 INFO:teuthology.orchestra.run.smithi090.stdout:(4/32): libradosstriper1-19.0.0-2778.ga971fd5f. 8.4 MB/s | 501 kB 00:00 2024-04-14T15:00:04.274 INFO:teuthology.orchestra.run.smithi090.stdout:(5/32): libcephfs2-19.0.0-2778.ga971fd5f.el9.x8 4.0 MB/s | 746 kB 00:00 2024-04-14T15:00:04.374 INFO:teuthology.orchestra.run.smithi090.stdout:(6/32): python3-ceph-argparse-19.0.0-2778.ga971 452 kB/s | 45 kB 00:00 2024-04-14T15:00:04.394 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2024-04-14T15:00:04.396 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-04-14T15:00:04.396 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repository Size 2024-04-14T15:00:04.396 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-04-14T15:00:04.396 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2024-04-14T15:00:04.397 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-radosgw x86_64 2:19.0.0-2778.ga971fd5f.el9 ceph 10 M 2024-04-14T15:00:04.397 INFO:teuthology.orchestra.run.smithi092.stdout:Upgrading: 2024-04-14T15:00:04.397 INFO:teuthology.orchestra.run.smithi092.stdout: librados2 x86_64 2:19.0.0-2778.ga971fd5f.el9 ceph 3.4 M 2024-04-14T15:00:04.397 INFO:teuthology.orchestra.run.smithi092.stdout: librbd1 x86_64 2:19.0.0-2778.ga971fd5f.el9 ceph 3.1 M 2024-04-14T15:00:04.397 INFO:teuthology.orchestra.run.smithi092.stdout:Installing dependencies: 2024-04-14T15:00:04.397 INFO:teuthology.orchestra.run.smithi092.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-14T15:00:04.397 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-base x86_64 2:19.0.0-2778.ga971fd5f.el9 ceph 5.4 M 2024-04-14T15:00:04.397 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-common x86_64 2:19.0.0-2778.ga971fd5f.el9 ceph 21 M 2024-04-14T15:00:04.397 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-selinux x86_64 2:19.0.0-2778.ga971fd5f.el9 ceph 25 k 2024-04-14T15:00:04.397 INFO:teuthology.orchestra.run.smithi092.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-14T15:00:04.397 INFO:teuthology.orchestra.run.smithi092.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-14T15:00:04.397 INFO:teuthology.orchestra.run.smithi092.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-14T15:00:04.397 INFO:teuthology.orchestra.run.smithi092.stdout: libcephfs2 x86_64 2:19.0.0-2778.ga971fd5f.el9 ceph 746 k 2024-04-14T15:00:04.397 INFO:teuthology.orchestra.run.smithi092.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-14T15:00:04.398 INFO:teuthology.orchestra.run.smithi092.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-14T15:00:04.398 INFO:teuthology.orchestra.run.smithi092.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-14T15:00:04.398 INFO:teuthology.orchestra.run.smithi092.stdout: libradosstriper1 x86_64 2:19.0.0-2778.ga971fd5f.el9 ceph 501 k 2024-04-14T15:00:04.398 INFO:teuthology.orchestra.run.smithi092.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-14T15:00:04.398 INFO:teuthology.orchestra.run.smithi092.stdout: librgw2 x86_64 2:19.0.0-2778.ga971fd5f.el9 ceph 5.2 M 2024-04-14T15:00:04.398 INFO:teuthology.orchestra.run.smithi092.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-14T15:00:04.398 INFO:teuthology.orchestra.run.smithi092.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-14T15:00:04.398 INFO:teuthology.orchestra.run.smithi092.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-14T15:00:04.398 INFO:teuthology.orchestra.run.smithi092.stdout: python3-ceph-argparse x86_64 2:19.0.0-2778.ga971fd5f.el9 ceph 45 k 2024-04-14T15:00:04.398 INFO:teuthology.orchestra.run.smithi092.stdout: python3-ceph-common x86_64 2:19.0.0-2778.ga971fd5f.el9 ceph 135 k 2024-04-14T15:00:04.398 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cephfs x86_64 2:19.0.0-2778.ga971fd5f.el9 ceph 161 k 2024-04-14T15:00:04.398 INFO:teuthology.orchestra.run.smithi092.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-14T15:00:04.399 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rados x86_64 2:19.0.0-2778.ga971fd5f.el9 ceph 321 k 2024-04-14T15:00:04.399 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rbd x86_64 2:19.0.0-2778.ga971fd5f.el9 ceph 298 k 2024-04-14T15:00:04.399 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rgw x86_64 2:19.0.0-2778.ga971fd5f.el9 ceph 99 k 2024-04-14T15:00:04.399 INFO:teuthology.orchestra.run.smithi092.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-14T15:00:04.399 INFO:teuthology.orchestra.run.smithi092.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-14T15:00:04.399 INFO:teuthology.orchestra.run.smithi092.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-14T15:00:04.399 INFO:teuthology.orchestra.run.smithi092.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-14T15:00:04.399 INFO:teuthology.orchestra.run.smithi092.stdout:Installing weak dependencies: 2024-04-14T15:00:04.399 INFO:teuthology.orchestra.run.smithi092.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-14T15:00:04.399 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:00:04.399 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2024-04-14T15:00:04.399 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-04-14T15:00:04.399 INFO:teuthology.orchestra.run.smithi092.stdout:Install 30 Packages 2024-04-14T15:00:04.399 INFO:teuthology.orchestra.run.smithi092.stdout:Upgrade 2 Packages 2024-04-14T15:00:04.400 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:00:04.400 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 59 M 2024-04-14T15:00:04.400 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2024-04-14T15:00:04.533 INFO:teuthology.orchestra.run.smithi090.stdout:(7/32): librgw2-19.0.0-2778.ga971fd5f.el9.x86_6 15 MB/s | 5.2 MB 00:00 2024-04-14T15:00:04.558 INFO:teuthology.orchestra.run.smithi090.stdout:(8/32): python3-ceph-common-19.0.0-2778.ga971fd 734 kB/s | 135 kB 00:00 2024-04-14T15:00:04.583 INFO:teuthology.orchestra.run.smithi090.stdout:(9/32): python3-cephfs-19.0.0-2778.ga971fd5f.el 3.1 MB/s | 161 kB 00:00 2024-04-14T15:00:04.717 INFO:teuthology.orchestra.run.smithi090.stdout:(10/32): ceph-radosgw-19.0.0-2778.ga971fd5f.el9 4.4 MB/s | 10 MB 00:02 2024-04-14T15:00:04.751 INFO:teuthology.orchestra.run.smithi090.stdout:(11/32): python3-rbd-19.0.0-2778.ga971fd5f.el9. 1.7 MB/s | 298 kB 00:00 2024-04-14T15:00:04.776 INFO:teuthology.orchestra.run.smithi090.stdout:(12/32): python3-rados-19.0.0-2778.ga971fd5f.el 1.4 MB/s | 321 kB 00:00 2024-04-14T15:00:04.802 INFO:teuthology.orchestra.run.smithi090.stdout:(13/32): python3-rgw-19.0.0-2778.ga971fd5f.el9. 1.1 MB/s | 99 kB 00:00 2024-04-14T15:00:04.894 INFO:teuthology.orchestra.run.smithi090.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 493 kB/s | 45 kB 00:00 2024-04-14T15:00:04.920 INFO:teuthology.orchestra.run.smithi090.stdout:(15/32): boost-program-options-1.75.0-8.el9.x86 634 kB/s | 107 kB 00:00 2024-04-14T15:00:04.945 INFO:teuthology.orchestra.run.smithi090.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 950 kB/s | 160 kB 00:00 2024-04-14T15:00:04.987 INFO:teuthology.orchestra.run.smithi090.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.0 MB/s | 42 kB 00:00 2024-04-14T15:00:05.020 INFO:teuthology.orchestra.run.smithi090.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.8 MB/s | 292 kB 00:00 2024-04-14T15:00:05.054 INFO:teuthology.orchestra.run.smithi090.stdout:(19/32): librdkafka-1.6.1-102.el9.x86_64.rpm 4.1 MB/s | 662 kB 00:00 2024-04-14T15:00:05.080 INFO:teuthology.orchestra.run.smithi090.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.3 MB/s | 221 kB 00:00 2024-04-14T15:00:05.105 INFO:teuthology.orchestra.run.smithi090.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 428 kB/s | 36 kB 00:00 2024-04-14T15:00:05.131 INFO:teuthology.orchestra.run.smithi090.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 627 kB/s | 47 kB 00:00 2024-04-14T15:00:05.577 INFO:teuthology.orchestra.run.smithi092.stdout:(1/32): ceph-base-19.0.0-2778.ga971fd5f.el9.x86 7.0 MB/s | 5.4 MB 00:00 2024-04-14T15:00:05.580 INFO:teuthology.orchestra.run.smithi090.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 58 kB/s | 26 kB 00:00 2024-04-14T15:00:05.636 INFO:teuthology.orchestra.run.smithi092.stdout:(2/32): ceph-selinux-19.0.0-2778.ga971fd5f.el9. 432 kB/s | 25 kB 00:00 2024-04-14T15:00:05.769 INFO:teuthology.orchestra.run.smithi092.stdout:(3/32): libcephfs2-19.0.0-2778.ga971fd5f.el9.x8 5.5 MB/s | 746 kB 00:00 2024-04-14T15:00:05.780 INFO:teuthology.orchestra.run.smithi090.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 243 kB/s | 49 kB 00:00 2024-04-14T15:00:05.823 INFO:teuthology.orchestra.run.smithi090.stdout:(25/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 430 kB/s | 309 kB 00:00 2024-04-14T15:00:05.869 INFO:teuthology.orchestra.run.smithi092.stdout:(4/32): libradosstriper1-19.0.0-2778.ga971fd5f. 4.9 MB/s | 501 kB 00:00 2024-04-14T15:00:06.048 INFO:teuthology.orchestra.run.smithi090.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 252 kB/s | 67 kB 00:00 2024-04-14T15:00:06.190 INFO:teuthology.orchestra.run.smithi090.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.2 MB/s | 838 kB 00:00 2024-04-14T15:00:06.324 INFO:teuthology.orchestra.run.smithi090.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 696 kB/s | 192 kB 00:00 2024-04-14T15:00:06.466 INFO:teuthology.orchestra.run.smithi090.stdout:(29/32): libarrow-9.0.0-10.el9.x86_64.rpm 3.2 MB/s | 4.4 MB 00:01 2024-04-14T15:00:06.516 INFO:teuthology.orchestra.run.smithi090.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 4.9 MB/s | 1.6 MB 00:00 2024-04-14T15:00:06.528 INFO:teuthology.orchestra.run.smithi092.stdout:(5/32): ceph-radosgw-19.0.0-2778.ga971fd5f.el9. 5.8 MB/s | 10 MB 00:01 2024-04-14T15:00:06.603 INFO:teuthology.orchestra.run.smithi092.stdout:(6/32): librgw2-19.0.0-2778.ga971fd5f.el9.x86_6 7.1 MB/s | 5.2 MB 00:00 2024-04-14T15:00:06.620 INFO:teuthology.orchestra.run.smithi092.stdout:(7/32): python3-ceph-argparse-19.0.0-2778.ga971 492 kB/s | 45 kB 00:00 2024-04-14T15:00:06.670 INFO:teuthology.orchestra.run.smithi092.stdout:(8/32): python3-ceph-common-19.0.0-2778.ga971fd 2.0 MB/s | 135 kB 00:00 2024-04-14T15:00:06.721 INFO:teuthology.orchestra.run.smithi092.stdout:(9/32): python3-cephfs-19.0.0-2778.ga971fd5f.el 1.6 MB/s | 161 kB 00:00 2024-04-14T15:00:06.804 INFO:teuthology.orchestra.run.smithi092.stdout:(10/32): python3-rbd-19.0.0-2778.ga971fd5f.el9. 3.5 MB/s | 298 kB 00:00 2024-04-14T15:00:06.833 INFO:teuthology.orchestra.run.smithi090.stdout:(31/32): librbd1-19.0.0-2778.ga971fd5f.el9.x86_ 8.6 MB/s | 3.1 MB 00:00 2024-04-14T15:00:06.996 INFO:teuthology.orchestra.run.smithi092.stdout:(11/32): ceph-common-19.0.0-2778.ga971fd5f.el9. 9.5 MB/s | 21 MB 00:02 2024-04-14T15:00:07.021 INFO:teuthology.orchestra.run.smithi092.stdout:(12/32): python3-rados-19.0.0-2778.ga971fd5f.el 916 kB/s | 321 kB 00:00 2024-04-14T15:00:07.047 INFO:teuthology.orchestra.run.smithi092.stdout:(13/32): python3-rgw-19.0.0-2778.ga971fd5f.el9. 411 kB/s | 99 kB 00:00 2024-04-14T15:00:07.164 INFO:teuthology.orchestra.run.smithi092.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.1 MB/s | 160 kB 00:00 2024-04-14T15:00:07.183 INFO:teuthology.orchestra.run.smithi090.stdout:(32/32): librados2-19.0.0-2778.ga971fd5f.el9.x8 4.0 MB/s | 3.4 MB 00:00 2024-04-14T15:00:07.188 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-04-14T15:00:07.188 INFO:teuthology.orchestra.run.smithi090.stdout:Total 11 MB/s | 59 MB 00:05 2024-04-14T15:00:07.189 INFO:teuthology.orchestra.run.smithi092.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 319 kB/s | 45 kB 00:00 2024-04-14T15:00:07.206 INFO:teuthology.orchestra.run.smithi092.stdout:(16/32): boost-program-options-1.75.0-8.el9.x86 511 kB/s | 107 kB 00:00 2024-04-14T15:00:07.256 INFO:teuthology.orchestra.run.smithi092.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 844 kB/s | 42 kB 00:00 2024-04-14T15:00:07.281 INFO:teuthology.orchestra.run.smithi092.stdout:(18/32): librdkafka-1.6.1-102.el9.x86_64.rpm 5.5 MB/s | 662 kB 00:00 2024-04-14T15:00:07.289 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-04-14T15:00:07.311 INFO:teuthology.orchestra.run.smithi092.stdout:(19/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.3 MB/s | 292 kB 00:00 2024-04-14T15:00:07.324 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-04-14T15:00:07.324 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-04-14T15:00:07.332 INFO:teuthology.orchestra.run.smithi092.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 716 kB/s | 36 kB 00:00 2024-04-14T15:00:07.357 INFO:teuthology.orchestra.run.smithi092.stdout:(21/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.1 MB/s | 221 kB 00:00 2024-04-14T15:00:07.382 INFO:teuthology.orchestra.run.smithi092.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 666 kB/s | 47 kB 00:00 2024-04-14T15:00:07.474 INFO:teuthology.orchestra.run.smithi092.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 283 kB/s | 26 kB 00:00 2024-04-14T15:00:07.533 INFO:teuthology.orchestra.run.smithi092.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 830 kB/s | 49 kB 00:00 2024-04-14T15:00:07.558 INFO:teuthology.orchestra.run.smithi092.stdout:(25/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.3 MB/s | 309 kB 00:00 2024-04-14T15:00:07.592 INFO:teuthology.orchestra.run.smithi092.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-04-14T15:00:07.676 INFO:teuthology.orchestra.run.smithi092.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 2.3 MB/s | 192 kB 00:00 2024-04-14T15:00:07.785 INFO:teuthology.orchestra.run.smithi092.stdout:(28/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.6 MB/s | 838 kB 00:00 2024-04-14T15:00:07.813 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-04-14T15:00:07.813 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-04-14T15:00:07.935 INFO:teuthology.orchestra.run.smithi092.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 6.1 MB/s | 1.6 MB 00:00 2024-04-14T15:00:08.118 INFO:teuthology.orchestra.run.smithi092.stdout:(30/32): librados2-19.0.0-2778.ga971fd5f.el9.x8 10 MB/s | 3.4 MB 00:00 2024-04-14T15:00:08.335 INFO:teuthology.orchestra.run.smithi092.stdout:(31/32): librbd1-19.0.0-2778.ga971fd5f.el9.x86_ 7.9 MB/s | 3.1 MB 00:00 2024-04-14T15:00:08.402 INFO:teuthology.orchestra.run.smithi092.stdout:(32/32): libarrow-9.0.0-10.el9.x86_64.rpm 4.2 MB/s | 4.4 MB 00:01 2024-04-14T15:00:08.404 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2024-04-14T15:00:08.404 INFO:teuthology.orchestra.run.smithi092.stdout:Total 15 MB/s | 59 MB 00:04 2024-04-14T15:00:08.505 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2024-04-14T15:00:08.539 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2024-04-14T15:00:08.540 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2024-04-14T15:00:09.027 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2024-04-14T15:00:09.027 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2024-04-14T15:00:09.243 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-04-14T15:00:09.358 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-04-14T15:00:09.390 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-04-14T15:00:09.424 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-04-14T15:00:09.446 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-04-14T15:00:09.475 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-04-14T15:00:09.672 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-04-14T15:00:09.726 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-04-14T15:00:10.093 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-04-14T15:00:10.143 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : librados2-2:19.0.0-2778.ga971fd5f.el9.x86_64 7/34 2024-04-14T15:00:10.235 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librados2-2:19.0.0-2778.ga971fd5f.el9.x86_64 7/34 2024-04-14T15:00:10.324 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rados-2:19.0.0-2778.ga971fd5f.el9.x86_64 8/34 2024-04-14T15:00:10.353 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libcephfs2-2:19.0.0-2778.ga971fd5f.el9.x86_64 9/34 2024-04-14T15:00:10.417 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libcephfs2-2:19.0.0-2778.ga971fd5f.el9.x86_64 9/34 2024-04-14T15:00:10.446 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-04-14T15:00:10.473 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2024-04-14T15:00:10.490 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-ceph-argparse-2:19.0.0-2778.ga971fd5f.el9. 11/34 2024-04-14T15:00:10.560 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cephfs-2:19.0.0-2778.ga971fd5f.el9.x86_64 12/34 2024-04-14T15:00:10.571 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-04-14T15:00:10.590 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libradosstriper1-2:19.0.0-2778.ga971fd5f.el9.x86_6 13/34 2024-04-14T15:00:10.599 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-04-14T15:00:10.624 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-04-14T15:00:10.647 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-04-14T15:00:10.654 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2778.ga971fd5f.el9.x86_6 13/34 2024-04-14T15:00:10.663 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-04-14T15:00:10.689 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-04-14T15:00:10.799 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-04-14T15:00:10.823 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-04-14T15:00:10.835 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-04-14T15:00:10.872 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-04-14T15:00:10.878 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-04-14T15:00:10.905 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-04-14T15:00:10.934 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-04-14T15:00:10.967 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-04-14T15:00:11.222 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-04-14T15:00:11.284 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : librados2-2:19.0.0-2778.ga971fd5f.el9.x86_64 7/34 2024-04-14T15:00:11.284 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-04-14T15:00:11.315 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : librbd1-2:19.0.0-2778.ga971fd5f.el9.x86_64 22/34 2024-04-14T15:00:11.376 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librados2-2:19.0.0-2778.ga971fd5f.el9.x86_64 7/34 2024-04-14T15:00:11.410 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librbd1-2:19.0.0-2778.ga971fd5f.el9.x86_64 22/34 2024-04-14T15:00:11.452 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rbd-2:19.0.0-2778.ga971fd5f.el9.x86_64 23/34 2024-04-14T15:00:11.463 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-rados-2:19.0.0-2778.ga971fd5f.el9.x86_64 8/34 2024-04-14T15:00:11.485 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libcephfs2-2:19.0.0-2778.ga971fd5f.el9.x86_64 9/34 2024-04-14T15:00:11.507 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-04-14T15:00:11.540 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: libcephfs2-2:19.0.0-2778.ga971fd5f.el9.x86_64 9/34 2024-04-14T15:00:11.569 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-04-14T15:00:11.614 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-ceph-argparse-2:19.0.0-2778.ga971fd5f.el9. 11/34 2024-04-14T15:00:11.697 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cephfs-2:19.0.0-2778.ga971fd5f.el9.x86_64 12/34 2024-04-14T15:00:11.721 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libradosstriper1-2:19.0.0-2778.ga971fd5f.el9.x86_6 13/34 2024-04-14T15:00:11.802 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2778.ga971fd5f.el9.x86_6 13/34 2024-04-14T15:00:11.837 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-04-14T15:00:11.945 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-04-14T15:00:11.977 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-04-14T15:00:11.987 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-04-14T15:00:12.026 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librgw2-2:19.0.0-2778.ga971fd5f.el9.x86_64 26/34 2024-04-14T15:00:12.028 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-04-14T15:00:12.061 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-04-14T15:00:12.081 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librgw2-2:19.0.0-2778.ga971fd5f.el9.x86_64 26/34 2024-04-14T15:00:12.090 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-04-14T15:00:12.123 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-04-14T15:00:12.150 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rgw-2:19.0.0-2778.ga971fd5f.el9.x86_64 27/34 2024-04-14T15:00:12.179 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-ceph-common-2:19.0.0-2778.ga971fd5f.el9.x8 28/34 2024-04-14T15:00:12.436 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-04-14T15:00:12.461 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : librbd1-2:19.0.0-2778.ga971fd5f.el9.x86_64 22/34 2024-04-14T15:00:12.551 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librbd1-2:19.0.0-2778.ga971fd5f.el9.x86_64 22/34 2024-04-14T15:00:12.583 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-rbd-2:19.0.0-2778.ga971fd5f.el9.x86_64 23/34 2024-04-14T15:00:12.626 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-04-14T15:00:13.098 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-04-14T15:00:13.122 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : librgw2-2:19.0.0-2778.ga971fd5f.el9.x86_64 26/34 2024-04-14T15:00:13.176 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librgw2-2:19.0.0-2778.ga971fd5f.el9.x86_64 26/34 2024-04-14T15:00:13.245 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-rgw-2:19.0.0-2778.ga971fd5f.el9.x86_64 27/34 2024-04-14T15:00:13.274 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-ceph-common-2:19.0.0-2778.ga971fd5f.el9.x8 28/34 2024-04-14T15:00:14.515 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-common-2:19.0.0-2778.ga971fd5f.el9.x86_64 29/34 2024-04-14T15:00:14.553 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-common-2:19.0.0-2778.ga971fd5f.el9.x86_64 29/34 2024-04-14T15:00:15.113 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-common-2:19.0.0-2778.ga971fd5f.el9.x86_64 29/34 2024-04-14T15:00:15.156 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-base-2:19.0.0-2778.ga971fd5f.el9.x86_64 30/34 2024-04-14T15:00:15.248 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-base-2:19.0.0-2778.ga971fd5f.el9.x86_64 30/34 2024-04-14T15:00:15.248 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-14T15:00:15.248 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-14T15:00:15.249 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:00:15.276 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-selinux-2:19.0.0-2778.ga971fd5f.el9.x86_64 31/34 2024-04-14T15:00:15.568 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-common-2:19.0.0-2778.ga971fd5f.el9.x86_64 29/34 2024-04-14T15:00:15.604 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-common-2:19.0.0-2778.ga971fd5f.el9.x86_64 29/34 2024-04-14T15:00:16.166 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-common-2:19.0.0-2778.ga971fd5f.el9.x86_64 29/34 2024-04-14T15:00:16.207 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-base-2:19.0.0-2778.ga971fd5f.el9.x86_64 30/34 2024-04-14T15:00:16.300 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-base-2:19.0.0-2778.ga971fd5f.el9.x86_64 30/34 2024-04-14T15:00:16.300 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-14T15:00:16.300 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-14T15:00:16.300 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:00:16.334 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-selinux-2:19.0.0-2778.ga971fd5f.el9.x86_64 31/34 2024-04-14T15:00:25.689 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2778.ga971fd5f.el9.x86_64 31/34 2024-04-14T15:00:25.690 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /sys 2024-04-14T15:00:25.690 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /proc 2024-04-14T15:00:25.690 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /mnt 2024-04-14T15:00:25.690 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /var/tmp 2024-04-14T15:00:25.690 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /home 2024-04-14T15:00:25.690 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /root 2024-04-14T15:00:25.690 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /tmp 2024-04-14T15:00:25.690 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:00:25.791 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2778.ga971fd5f.el9.x86_64 31/34 2024-04-14T15:00:25.791 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /sys 2024-04-14T15:00:25.791 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /proc 2024-04-14T15:00:25.791 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /mnt 2024-04-14T15:00:25.792 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /var/tmp 2024-04-14T15:00:25.792 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /home 2024-04-14T15:00:25.792 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /root 2024-04-14T15:00:25.792 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /tmp 2024-04-14T15:00:25.792 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:00:26.695 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-radosgw-2:19.0.0-2778.ga971fd5f.el9.x86_64 32/34 2024-04-14T15:00:26.721 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2778.ga971fd5f.el9.x86_64 32/34 2024-04-14T15:00:26.721 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-14T15:00:26.721 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-14T15:00:26.721 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-14T15:00:26.721 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-14T15:00:26.721 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:00:26.723 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-14T15:00:26.723 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-radosgw-2:19.0.0-2778.ga971fd5f.el9.x86_64 32/34 2024-04-14T15:00:26.748 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2778.ga971fd5f.el9.x86_64 32/34 2024-04-14T15:00:26.748 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-14T15:00:26.749 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-14T15:00:26.749 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-14T15:00:26.749 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-14T15:00:26.749 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:00:26.750 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-14T15:00:26.808 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-14T15:00:26.809 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-14T15:00:26.834 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-14T15:00:26.834 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-14T15:00:27.942 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-14T15:00:27.942 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-base-2:19.0.0-2778.ga971fd5f.el9.x86_64 1/34 2024-04-14T15:00:27.942 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-common-2:19.0.0-2778.ga971fd5f.el9.x86_64 2/34 2024-04-14T15:00:27.942 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-radosgw-2:19.0.0-2778.ga971fd5f.el9.x86_64 3/34 2024-04-14T15:00:27.942 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-selinux-2:19.0.0-2778.ga971fd5f.el9.x86_64 4/34 2024-04-14T15:00:27.942 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libcephfs2-2:19.0.0-2778.ga971fd5f.el9.x86_64 5/34 2024-04-14T15:00:27.942 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libradosstriper1-2:19.0.0-2778.ga971fd5f.el9.x86_6 6/34 2024-04-14T15:00:27.942 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librgw2-2:19.0.0-2778.ga971fd5f.el9.x86_64 7/34 2024-04-14T15:00:27.942 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2778.ga971fd5f.el9. 8/34 2024-04-14T15:00:27.943 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-ceph-common-2:19.0.0-2778.ga971fd5f.el9.x8 9/34 2024-04-14T15:00:27.943 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cephfs-2:19.0.0-2778.ga971fd5f.el9.x86_64 10/34 2024-04-14T15:00:27.943 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-rados-2:19.0.0-2778.ga971fd5f.el9.x86_64 11/34 2024-04-14T15:00:27.943 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-rbd-2:19.0.0-2778.ga971fd5f.el9.x86_64 12/34 2024-04-14T15:00:27.943 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-rgw-2:19.0.0-2778.ga971fd5f.el9.x86_64 13/34 2024-04-14T15:00:27.943 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-04-14T15:00:27.943 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-04-14T15:00:27.943 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-04-14T15:00:27.943 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-04-14T15:00:27.943 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-04-14T15:00:27.943 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-04-14T15:00:27.943 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-04-14T15:00:27.943 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-04-14T15:00:27.943 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-04-14T15:00:27.944 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-04-14T15:00:27.944 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-04-14T15:00:27.944 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-04-14T15:00:27.944 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-04-14T15:00:27.944 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-04-14T15:00:27.944 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-04-14T15:00:27.944 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-04-14T15:00:27.944 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-04-14T15:00:27.944 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librados2-2:19.0.0-2778.ga971fd5f.el9.x86_64 31/34 2024-04-14T15:00:27.944 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-04-14T15:00:27.944 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librbd1-2:19.0.0-2778.ga971fd5f.el9.x86_64 33/34 2024-04-14T15:00:28.097 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-14T15:00:28.097 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-base-2:19.0.0-2778.ga971fd5f.el9.x86_64 1/34 2024-04-14T15:00:28.097 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-common-2:19.0.0-2778.ga971fd5f.el9.x86_64 2/34 2024-04-14T15:00:28.097 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-radosgw-2:19.0.0-2778.ga971fd5f.el9.x86_64 3/34 2024-04-14T15:00:28.097 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-selinux-2:19.0.0-2778.ga971fd5f.el9.x86_64 4/34 2024-04-14T15:00:28.097 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libcephfs2-2:19.0.0-2778.ga971fd5f.el9.x86_64 5/34 2024-04-14T15:00:28.097 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libradosstriper1-2:19.0.0-2778.ga971fd5f.el9.x86_6 6/34 2024-04-14T15:00:28.097 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librgw2-2:19.0.0-2778.ga971fd5f.el9.x86_64 7/34 2024-04-14T15:00:28.097 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2778.ga971fd5f.el9. 8/34 2024-04-14T15:00:28.097 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-ceph-common-2:19.0.0-2778.ga971fd5f.el9.x8 9/34 2024-04-14T15:00:28.098 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cephfs-2:19.0.0-2778.ga971fd5f.el9.x86_64 10/34 2024-04-14T15:00:28.098 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rados-2:19.0.0-2778.ga971fd5f.el9.x86_64 11/34 2024-04-14T15:00:28.098 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rbd-2:19.0.0-2778.ga971fd5f.el9.x86_64 12/34 2024-04-14T15:00:28.098 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rgw-2:19.0.0-2778.ga971fd5f.el9.x86_64 13/34 2024-04-14T15:00:28.098 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-04-14T15:00:28.098 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-04-14T15:00:28.098 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-04-14T15:00:28.098 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-04-14T15:00:28.098 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-04-14T15:00:28.098 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-04-14T15:00:28.098 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-04-14T15:00:28.098 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-04-14T15:00:28.098 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-04-14T15:00:28.098 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-04-14T15:00:28.099 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-04-14T15:00:28.099 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-04-14T15:00:28.099 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-04-14T15:00:28.099 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-04-14T15:00:28.099 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-04-14T15:00:28.099 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-04-14T15:00:28.099 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-04-14T15:00:28.100 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librados2-2:19.0.0-2778.ga971fd5f.el9.x86_64 31/34 2024-04-14T15:00:28.100 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-04-14T15:00:28.100 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librbd1-2:19.0.0-2778.ga971fd5f.el9.x86_64 33/34 2024-04-14T15:00:29.117 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-04-14T15:00:29.117 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:00:29.117 INFO:teuthology.orchestra.run.smithi092.stdout:Upgraded: 2024-04-14T15:00:29.117 INFO:teuthology.orchestra.run.smithi092.stdout: librados2-2:19.0.0-2778.ga971fd5f.el9.x86_64 2024-04-14T15:00:29.117 INFO:teuthology.orchestra.run.smithi092.stdout: librbd1-2:19.0.0-2778.ga971fd5f.el9.x86_64 2024-04-14T15:00:29.117 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2024-04-14T15:00:29.117 INFO:teuthology.orchestra.run.smithi092.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-14T15:00:29.117 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-base-2:19.0.0-2778.ga971fd5f.el9.x86_64 2024-04-14T15:00:29.117 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-common-2:19.0.0-2778.ga971fd5f.el9.x86_64 2024-04-14T15:00:29.117 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-radosgw-2:19.0.0-2778.ga971fd5f.el9.x86_64 2024-04-14T15:00:29.117 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-selinux-2:19.0.0-2778.ga971fd5f.el9.x86_64 2024-04-14T15:00:29.117 INFO:teuthology.orchestra.run.smithi092.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-14T15:00:29.118 INFO:teuthology.orchestra.run.smithi092.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-14T15:00:29.118 INFO:teuthology.orchestra.run.smithi092.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-14T15:00:29.118 INFO:teuthology.orchestra.run.smithi092.stdout: libcephfs2-2:19.0.0-2778.ga971fd5f.el9.x86_64 2024-04-14T15:00:29.118 INFO:teuthology.orchestra.run.smithi092.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-14T15:00:29.118 INFO:teuthology.orchestra.run.smithi092.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-14T15:00:29.118 INFO:teuthology.orchestra.run.smithi092.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-14T15:00:29.118 INFO:teuthology.orchestra.run.smithi092.stdout: libradosstriper1-2:19.0.0-2778.ga971fd5f.el9.x86_64 2024-04-14T15:00:29.118 INFO:teuthology.orchestra.run.smithi092.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-14T15:00:29.118 INFO:teuthology.orchestra.run.smithi092.stdout: librgw2-2:19.0.0-2778.ga971fd5f.el9.x86_64 2024-04-14T15:00:29.118 INFO:teuthology.orchestra.run.smithi092.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-14T15:00:29.118 INFO:teuthology.orchestra.run.smithi092.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-14T15:00:29.118 INFO:teuthology.orchestra.run.smithi092.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-14T15:00:29.118 INFO:teuthology.orchestra.run.smithi092.stdout: python3-ceph-argparse-2:19.0.0-2778.ga971fd5f.el9.x86_64 2024-04-14T15:00:29.118 INFO:teuthology.orchestra.run.smithi092.stdout: python3-ceph-common-2:19.0.0-2778.ga971fd5f.el9.x86_64 2024-04-14T15:00:29.119 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cephfs-2:19.0.0-2778.ga971fd5f.el9.x86_64 2024-04-14T15:00:29.119 INFO:teuthology.orchestra.run.smithi092.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-14T15:00:29.119 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rados-2:19.0.0-2778.ga971fd5f.el9.x86_64 2024-04-14T15:00:29.119 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rbd-2:19.0.0-2778.ga971fd5f.el9.x86_64 2024-04-14T15:00:29.119 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rgw-2:19.0.0-2778.ga971fd5f.el9.x86_64 2024-04-14T15:00:29.119 INFO:teuthology.orchestra.run.smithi092.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-14T15:00:29.119 INFO:teuthology.orchestra.run.smithi092.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-14T15:00:29.119 INFO:teuthology.orchestra.run.smithi092.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-14T15:00:29.119 INFO:teuthology.orchestra.run.smithi092.stdout: re2-1:20211101-3.el9.x86_64 2024-04-14T15:00:29.119 INFO:teuthology.orchestra.run.smithi092.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-14T15:00:29.119 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:00:29.119 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2024-04-14T15:00:29.249 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-04-14T15:00:29.249 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:00:29.249 INFO:teuthology.orchestra.run.smithi090.stdout:Upgraded: 2024-04-14T15:00:29.249 INFO:teuthology.orchestra.run.smithi090.stdout: librados2-2:19.0.0-2778.ga971fd5f.el9.x86_64 2024-04-14T15:00:29.249 INFO:teuthology.orchestra.run.smithi090.stdout: librbd1-2:19.0.0-2778.ga971fd5f.el9.x86_64 2024-04-14T15:00:29.249 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-04-14T15:00:29.249 INFO:teuthology.orchestra.run.smithi090.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-14T15:00:29.249 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-base-2:19.0.0-2778.ga971fd5f.el9.x86_64 2024-04-14T15:00:29.249 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-common-2:19.0.0-2778.ga971fd5f.el9.x86_64 2024-04-14T15:00:29.249 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-radosgw-2:19.0.0-2778.ga971fd5f.el9.x86_64 2024-04-14T15:00:29.250 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-selinux-2:19.0.0-2778.ga971fd5f.el9.x86_64 2024-04-14T15:00:29.250 INFO:teuthology.orchestra.run.smithi090.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-14T15:00:29.250 INFO:teuthology.orchestra.run.smithi090.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-14T15:00:29.250 INFO:teuthology.orchestra.run.smithi090.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-14T15:00:29.250 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs2-2:19.0.0-2778.ga971fd5f.el9.x86_64 2024-04-14T15:00:29.250 INFO:teuthology.orchestra.run.smithi090.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-14T15:00:29.250 INFO:teuthology.orchestra.run.smithi090.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-14T15:00:29.250 INFO:teuthology.orchestra.run.smithi090.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-14T15:00:29.250 INFO:teuthology.orchestra.run.smithi090.stdout: libradosstriper1-2:19.0.0-2778.ga971fd5f.el9.x86_64 2024-04-14T15:00:29.250 INFO:teuthology.orchestra.run.smithi090.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-14T15:00:29.250 INFO:teuthology.orchestra.run.smithi090.stdout: librgw2-2:19.0.0-2778.ga971fd5f.el9.x86_64 2024-04-14T15:00:29.251 INFO:teuthology.orchestra.run.smithi090.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-14T15:00:29.251 INFO:teuthology.orchestra.run.smithi090.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-14T15:00:29.251 INFO:teuthology.orchestra.run.smithi090.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-14T15:00:29.251 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-argparse-2:19.0.0-2778.ga971fd5f.el9.x86_64 2024-04-14T15:00:29.251 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-common-2:19.0.0-2778.ga971fd5f.el9.x86_64 2024-04-14T15:00:29.251 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cephfs-2:19.0.0-2778.ga971fd5f.el9.x86_64 2024-04-14T15:00:29.251 INFO:teuthology.orchestra.run.smithi090.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-14T15:00:29.251 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rados-2:19.0.0-2778.ga971fd5f.el9.x86_64 2024-04-14T15:00:29.251 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rbd-2:19.0.0-2778.ga971fd5f.el9.x86_64 2024-04-14T15:00:29.251 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rgw-2:19.0.0-2778.ga971fd5f.el9.x86_64 2024-04-14T15:00:29.251 INFO:teuthology.orchestra.run.smithi090.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-14T15:00:29.251 INFO:teuthology.orchestra.run.smithi090.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-14T15:00:29.251 INFO:teuthology.orchestra.run.smithi090.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-14T15:00:29.251 INFO:teuthology.orchestra.run.smithi090.stdout: re2-1:20211101-3.el9.x86_64 2024-04-14T15:00:29.252 INFO:teuthology.orchestra.run.smithi090.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-14T15:00:29.252 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:00:29.252 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-04-14T15:00:29.403 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-test 2024-04-14T15:00:29.682 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-test 2024-04-14T15:00:29.999 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:00:28 ago on Sun 14 Apr 2024 03:00:01 PM UTC. 2024-04-14T15:00:30.273 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:00:32 ago on Sun 14 Apr 2024 02:59:58 PM UTC. 2024-04-14T15:00:30.288 INFO:teuthology.orchestra.run.smithi092.stderr:Modular dependency problems: 2024-04-14T15:00:30.288 INFO:teuthology.orchestra.run.smithi092.stderr: 2024-04-14T15:00:30.288 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 1: conflicting requests 2024-04-14T15:00:30.288 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.288 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 2: conflicting requests 2024-04-14T15:00:30.288 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.288 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 3: conflicting requests 2024-04-14T15:00:30.288 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.288 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 4: conflicting requests 2024-04-14T15:00:30.288 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.289 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 5: conflicting requests 2024-04-14T15:00:30.289 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.289 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 6: conflicting requests 2024-04-14T15:00:30.289 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.289 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 7: conflicting requests 2024-04-14T15:00:30.289 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.289 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 8: conflicting requests 2024-04-14T15:00:30.289 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.289 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 9: conflicting requests 2024-04-14T15:00:30.289 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.289 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 10: conflicting requests 2024-04-14T15:00:30.289 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.289 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 11: conflicting requests 2024-04-14T15:00:30.289 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.290 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 12: conflicting requests 2024-04-14T15:00:30.290 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.290 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 13: conflicting requests 2024-04-14T15:00:30.290 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.290 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 14: conflicting requests 2024-04-14T15:00:30.290 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.290 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 15: conflicting requests 2024-04-14T15:00:30.290 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.290 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 16: conflicting requests 2024-04-14T15:00:30.290 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.290 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 17: conflicting requests 2024-04-14T15:00:30.290 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.290 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 18: conflicting requests 2024-04-14T15:00:30.290 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.291 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 19: conflicting requests 2024-04-14T15:00:30.291 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.291 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 20: conflicting requests 2024-04-14T15:00:30.291 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.291 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 21: conflicting requests 2024-04-14T15:00:30.291 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.291 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 22: conflicting requests 2024-04-14T15:00:30.291 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.291 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 23: conflicting requests 2024-04-14T15:00:30.291 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.291 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 24: conflicting requests 2024-04-14T15:00:30.291 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.291 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 25: conflicting requests 2024-04-14T15:00:30.292 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.292 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 26: conflicting requests 2024-04-14T15:00:30.292 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.292 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 27: conflicting requests 2024-04-14T15:00:30.292 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.292 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 28: conflicting requests 2024-04-14T15:00:30.292 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.292 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 29: conflicting requests 2024-04-14T15:00:30.292 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.292 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 30: conflicting requests 2024-04-14T15:00:30.292 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.292 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 31: conflicting requests 2024-04-14T15:00:30.292 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.292 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 32: conflicting requests 2024-04-14T15:00:30.293 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.293 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 33: conflicting requests 2024-04-14T15:00:30.293 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.293 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 34: conflicting requests 2024-04-14T15:00:30.293 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.293 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 35: conflicting requests 2024-04-14T15:00:30.293 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.293 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 36: conflicting requests 2024-04-14T15:00:30.293 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.293 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 37: conflicting requests 2024-04-14T15:00:30.293 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.293 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.293 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.294 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 38: cannot install the best candidate for the job 2024-04-14T15:00:30.294 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.294 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.294 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.294 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.294 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 39: conflicting requests 2024-04-14T15:00:30.294 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.294 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.294 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.294 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.294 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.294 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.294 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 40: cannot install the best candidate for the job 2024-04-14T15:00:30.294 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.295 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.295 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.295 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.295 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 41: cannot install the best candidate for the job 2024-04-14T15:00:30.295 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.295 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.295 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.295 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.295 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 42: cannot install the best candidate for the job 2024-04-14T15:00:30.295 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.295 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.295 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.295 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.295 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 43: cannot install the best candidate for the job 2024-04-14T15:00:30.295 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.296 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.296 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.296 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.296 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 44: cannot install the best candidate for the job 2024-04-14T15:00:30.296 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.296 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.296 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.296 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.296 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 45: cannot install the best candidate for the job 2024-04-14T15:00:30.296 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.296 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.296 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.296 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.360 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2024-04-14T15:00:30.361 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-04-14T15:00:30.361 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repository Size 2024-04-14T15:00:30.361 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-04-14T15:00:30.361 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2024-04-14T15:00:30.361 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-test x86_64 2:19.0.0-2778.ga971fd5f.el9 ceph 49 M 2024-04-14T15:00:30.361 INFO:teuthology.orchestra.run.smithi092.stdout:Installing dependencies: 2024-04-14T15:00:30.361 INFO:teuthology.orchestra.run.smithi092.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-14T15:00:30.361 INFO:teuthology.orchestra.run.smithi092.stdout: libcephsqlite x86_64 2:19.0.0-2778.ga971fd5f.el9 ceph 163 k 2024-04-14T15:00:30.362 INFO:teuthology.orchestra.run.smithi092.stdout: oniguruma x86_64 6.9.6-1.el9.5 baseos 218 k 2024-04-14T15:00:30.362 INFO:teuthology.orchestra.run.smithi092.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-14T15:00:30.362 INFO:teuthology.orchestra.run.smithi092.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-14T15:00:30.362 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:00:30.362 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2024-04-14T15:00:30.362 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-04-14T15:00:30.362 INFO:teuthology.orchestra.run.smithi092.stdout:Install 6 Packages 2024-04-14T15:00:30.362 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:00:30.363 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 49 M 2024-04-14T15:00:30.363 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 210 M 2024-04-14T15:00:30.363 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2024-04-14T15:00:30.562 INFO:teuthology.orchestra.run.smithi090.stderr:Modular dependency problems: 2024-04-14T15:00:30.562 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-04-14T15:00:30.562 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 1: conflicting requests 2024-04-14T15:00:30.562 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.562 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 2: conflicting requests 2024-04-14T15:00:30.562 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.562 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 3: conflicting requests 2024-04-14T15:00:30.562 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.563 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 4: conflicting requests 2024-04-14T15:00:30.563 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.563 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 5: conflicting requests 2024-04-14T15:00:30.563 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.563 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 6: conflicting requests 2024-04-14T15:00:30.563 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.563 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 7: conflicting requests 2024-04-14T15:00:30.563 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.563 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 8: conflicting requests 2024-04-14T15:00:30.563 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.563 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 9: conflicting requests 2024-04-14T15:00:30.563 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.563 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 10: conflicting requests 2024-04-14T15:00:30.563 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.564 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 11: conflicting requests 2024-04-14T15:00:30.564 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.564 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 12: conflicting requests 2024-04-14T15:00:30.564 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.564 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 13: conflicting requests 2024-04-14T15:00:30.564 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.564 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 14: conflicting requests 2024-04-14T15:00:30.564 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.564 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 15: conflicting requests 2024-04-14T15:00:30.564 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.564 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 16: conflicting requests 2024-04-14T15:00:30.564 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.565 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 17: conflicting requests 2024-04-14T15:00:30.565 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.565 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 18: conflicting requests 2024-04-14T15:00:30.565 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.565 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 19: conflicting requests 2024-04-14T15:00:30.565 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.565 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 20: conflicting requests 2024-04-14T15:00:30.565 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.565 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 21: conflicting requests 2024-04-14T15:00:30.565 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.565 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 22: conflicting requests 2024-04-14T15:00:30.565 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.565 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 23: conflicting requests 2024-04-14T15:00:30.565 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.566 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 24: conflicting requests 2024-04-14T15:00:30.566 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.566 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 25: conflicting requests 2024-04-14T15:00:30.566 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.566 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 26: conflicting requests 2024-04-14T15:00:30.566 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.566 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 27: conflicting requests 2024-04-14T15:00:30.566 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.566 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 28: conflicting requests 2024-04-14T15:00:30.566 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.566 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 29: conflicting requests 2024-04-14T15:00:30.566 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.567 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 30: conflicting requests 2024-04-14T15:00:30.567 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.567 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 31: conflicting requests 2024-04-14T15:00:30.567 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.567 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 32: conflicting requests 2024-04-14T15:00:30.567 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.567 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 33: conflicting requests 2024-04-14T15:00:30.567 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.567 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 34: conflicting requests 2024-04-14T15:00:30.567 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.567 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 35: conflicting requests 2024-04-14T15:00:30.567 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.567 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 36: conflicting requests 2024-04-14T15:00:30.568 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.568 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 37: conflicting requests 2024-04-14T15:00:30.568 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.568 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.568 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.568 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 38: cannot install the best candidate for the job 2024-04-14T15:00:30.568 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.568 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.568 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.568 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.568 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 39: conflicting requests 2024-04-14T15:00:30.568 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.568 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.569 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.569 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.569 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.569 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.569 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 40: cannot install the best candidate for the job 2024-04-14T15:00:30.569 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.569 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.569 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.569 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.569 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 41: cannot install the best candidate for the job 2024-04-14T15:00:30.569 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.569 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.569 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.570 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.570 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 42: cannot install the best candidate for the job 2024-04-14T15:00:30.570 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.570 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.570 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.570 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.570 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 43: cannot install the best candidate for the job 2024-04-14T15:00:30.570 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.570 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.570 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.570 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.570 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 44: cannot install the best candidate for the job 2024-04-14T15:00:30.570 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.571 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.571 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.571 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.571 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 45: cannot install the best candidate for the job 2024-04-14T15:00:30.571 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.571 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.571 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.571 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-14T15:00:30.648 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-04-14T15:00:30.649 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-04-14T15:00:30.649 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-04-14T15:00:30.649 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-04-14T15:00:30.649 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-04-14T15:00:30.649 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-test x86_64 2:19.0.0-2778.ga971fd5f.el9 ceph 49 M 2024-04-14T15:00:30.649 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-04-14T15:00:30.649 INFO:teuthology.orchestra.run.smithi090.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-14T15:00:30.650 INFO:teuthology.orchestra.run.smithi090.stdout: libcephsqlite x86_64 2:19.0.0-2778.ga971fd5f.el9 ceph 163 k 2024-04-14T15:00:30.650 INFO:teuthology.orchestra.run.smithi090.stdout: oniguruma x86_64 6.9.6-1.el9.5 baseos 218 k 2024-04-14T15:00:30.650 INFO:teuthology.orchestra.run.smithi090.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-14T15:00:30.650 INFO:teuthology.orchestra.run.smithi090.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-14T15:00:30.650 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:00:30.650 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-04-14T15:00:30.650 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-04-14T15:00:30.650 INFO:teuthology.orchestra.run.smithi090.stdout:Install 6 Packages 2024-04-14T15:00:30.650 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:00:30.651 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 49 M 2024-04-14T15:00:30.651 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 210 M 2024-04-14T15:00:30.651 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-04-14T15:00:31.055 INFO:teuthology.orchestra.run.smithi092.stdout:(1/6): libcephsqlite-19.0.0-2778.ga971fd5f.el9. 342 kB/s | 163 kB 00:00 2024-04-14T15:00:31.372 INFO:teuthology.orchestra.run.smithi092.stdout:(2/6): jq-1.6-16.el9.x86_64.rpm 236 kB/s | 187 kB 00:00 2024-04-14T15:00:31.422 INFO:teuthology.orchestra.run.smithi092.stdout:(3/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 594 kB/s | 218 kB 00:00 2024-04-14T15:00:31.660 INFO:teuthology.orchestra.run.smithi090.stdout:(1/6): jq-1.6-16.el9.x86_64.rpm 214 kB/s | 187 kB 00:00 2024-04-14T15:00:32.090 INFO:teuthology.orchestra.run.smithi092.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 95 kB/s | 64 kB 00:00 2024-04-14T15:00:32.423 INFO:teuthology.orchestra.run.smithi092.stdout:(5/6): socat-1.7.4.1-5.el9.x86_64.rpm 290 kB/s | 305 kB 00:01 2024-04-14T15:00:32.585 INFO:teuthology.orchestra.run.smithi090.stdout:(2/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 236 kB/s | 218 kB 00:00 2024-04-14T15:00:32.652 INFO:teuthology.orchestra.run.smithi090.stdout:(3/6): libcephsqlite-19.0.0-2778.ga971fd5f.el9. 87 kB/s | 163 kB 00:01 2024-04-14T15:00:33.520 INFO:teuthology.orchestra.run.smithi090.stdout:(4/6): ceph-test-19.0.0-2778.ga971fd5f.el9.x86_ 18 MB/s | 49 MB 00:02 2024-04-14T15:00:33.878 INFO:teuthology.orchestra.run.smithi090.stdout:(5/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 52 kB/s | 64 kB 00:01 2024-04-14T15:00:34.120 INFO:teuthology.orchestra.run.smithi090.stdout:(6/6): socat-1.7.4.1-5.el9.x86_64.rpm 199 kB/s | 305 kB 00:01 2024-04-14T15:00:34.122 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-04-14T15:00:34.122 INFO:teuthology.orchestra.run.smithi090.stdout:Total 14 MB/s | 49 MB 00:03 2024-04-14T15:00:34.173 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-04-14T15:00:34.193 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-04-14T15:00:34.193 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-04-14T15:00:34.588 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-04-14T15:00:34.589 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-04-14T15:00:35.223 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-04-14T15:00:35.306 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-04-14T15:00:35.352 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-04-14T15:00:35.396 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-04-14T15:00:35.438 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-04-14T15:00:35.466 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libcephsqlite-2:19.0.0-2778.ga971fd5f.el9.x86_64 5/6 2024-04-14T15:00:39.797 INFO:teuthology.orchestra.run.smithi092.stdout:(6/6): ceph-test-19.0.0-2778.ga971fd5f.el9.x86_ 5.3 MB/s | 49 MB 00:09 2024-04-14T15:00:39.799 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2024-04-14T15:00:39.799 INFO:teuthology.orchestra.run.smithi092.stdout:Total 5.2 MB/s | 49 MB 00:09 2024-04-14T15:00:39.849 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2024-04-14T15:00:39.868 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2024-04-14T15:00:39.869 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2024-04-14T15:00:40.065 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2778.ga971fd5f.el9.x86_64 5/6 2024-04-14T15:00:40.151 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-test-2:19.0.0-2778.ga971fd5f.el9.x86_64 6/6 2024-04-14T15:00:40.260 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2024-04-14T15:00:40.260 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2024-04-14T15:00:40.851 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2024-04-14T15:00:40.943 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-04-14T15:00:40.982 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-04-14T15:00:41.033 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-04-14T15:00:41.072 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-04-14T15:00:41.100 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libcephsqlite-2:19.0.0-2778.ga971fd5f.el9.x86_64 5/6 2024-04-14T15:00:43.309 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-test-2:19.0.0-2778.ga971fd5f.el9.x86_64 6/6 2024-04-14T15:00:43.309 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-test-2:19.0.0-2778.ga971fd5f.el9.x86_64 1/6 2024-04-14T15:00:43.309 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libcephsqlite-2:19.0.0-2778.ga971fd5f.el9.x86_64 2/6 2024-04-14T15:00:43.309 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : jq-1.6-16.el9.x86_64 3/6 2024-04-14T15:00:43.309 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 4/6 2024-04-14T15:00:43.309 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/6 2024-04-14T15:00:43.782 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/6 2024-04-14T15:00:43.782 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:00:43.782 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-04-14T15:00:43.782 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-test-2:19.0.0-2778.ga971fd5f.el9.x86_64 2024-04-14T15:00:43.782 INFO:teuthology.orchestra.run.smithi090.stdout: jq-1.6-16.el9.x86_64 2024-04-14T15:00:43.782 INFO:teuthology.orchestra.run.smithi090.stdout: libcephsqlite-2:19.0.0-2778.ga971fd5f.el9.x86_64 2024-04-14T15:00:43.782 INFO:teuthology.orchestra.run.smithi090.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-04-14T15:00:43.782 INFO:teuthology.orchestra.run.smithi090.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-14T15:00:43.782 INFO:teuthology.orchestra.run.smithi090.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-14T15:00:43.782 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:00:43.782 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-04-14T15:00:44.048 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph 2024-04-14T15:00:44.644 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:00:46 ago on Sun 14 Apr 2024 02:59:58 PM UTC. 2024-04-14T15:00:44.935 INFO:teuthology.orchestra.run.smithi090.stderr:Modular dependency problems: 2024-04-14T15:00:44.936 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-04-14T15:00:44.936 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 1: conflicting requests 2024-04-14T15:00:44.936 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.936 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 2: conflicting requests 2024-04-14T15:00:44.936 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.936 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 3: conflicting requests 2024-04-14T15:00:44.936 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.936 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 4: conflicting requests 2024-04-14T15:00:44.936 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.936 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 5: conflicting requests 2024-04-14T15:00:44.936 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.936 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 6: conflicting requests 2024-04-14T15:00:44.936 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.937 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 7: conflicting requests 2024-04-14T15:00:44.937 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.937 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 8: conflicting requests 2024-04-14T15:00:44.937 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.937 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 9: conflicting requests 2024-04-14T15:00:44.937 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.937 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 10: conflicting requests 2024-04-14T15:00:44.937 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.937 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 11: conflicting requests 2024-04-14T15:00:44.937 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.937 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 12: conflicting requests 2024-04-14T15:00:44.937 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.937 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 13: conflicting requests 2024-04-14T15:00:44.937 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.938 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 14: conflicting requests 2024-04-14T15:00:44.938 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.938 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 15: conflicting requests 2024-04-14T15:00:44.938 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.938 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 16: conflicting requests 2024-04-14T15:00:44.938 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.938 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 17: conflicting requests 2024-04-14T15:00:44.938 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.938 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 18: conflicting requests 2024-04-14T15:00:44.938 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.938 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 19: conflicting requests 2024-04-14T15:00:44.938 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.938 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 20: conflicting requests 2024-04-14T15:00:44.939 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.939 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 21: conflicting requests 2024-04-14T15:00:44.939 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.939 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 22: conflicting requests 2024-04-14T15:00:44.939 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.939 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 23: conflicting requests 2024-04-14T15:00:44.939 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.939 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 24: conflicting requests 2024-04-14T15:00:44.939 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.939 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 25: conflicting requests 2024-04-14T15:00:44.939 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.939 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 26: conflicting requests 2024-04-14T15:00:44.939 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.940 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 27: conflicting requests 2024-04-14T15:00:44.940 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.940 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 28: conflicting requests 2024-04-14T15:00:44.940 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.940 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 29: conflicting requests 2024-04-14T15:00:44.940 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.940 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 30: conflicting requests 2024-04-14T15:00:44.940 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.940 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 31: conflicting requests 2024-04-14T15:00:44.940 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.940 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 32: conflicting requests 2024-04-14T15:00:44.940 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.940 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 33: conflicting requests 2024-04-14T15:00:44.940 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.941 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 34: conflicting requests 2024-04-14T15:00:44.941 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.941 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 35: conflicting requests 2024-04-14T15:00:44.941 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.941 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 36: conflicting requests 2024-04-14T15:00:44.941 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.941 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 37: conflicting requests 2024-04-14T15:00:44.941 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.941 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.941 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.941 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 38: cannot install the best candidate for the job 2024-04-14T15:00:44.941 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.941 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.941 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.942 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.942 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 39: conflicting requests 2024-04-14T15:00:44.942 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.942 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.942 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.942 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.942 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.942 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.942 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 40: cannot install the best candidate for the job 2024-04-14T15:00:44.942 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.942 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.942 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.942 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.943 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 41: cannot install the best candidate for the job 2024-04-14T15:00:44.943 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.943 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.943 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.943 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.943 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 42: cannot install the best candidate for the job 2024-04-14T15:00:44.943 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.943 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.943 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.943 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.943 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 43: cannot install the best candidate for the job 2024-04-14T15:00:44.943 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.943 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.943 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.943 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.944 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 44: cannot install the best candidate for the job 2024-04-14T15:00:44.944 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.944 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.944 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.944 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.944 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 45: cannot install the best candidate for the job 2024-04-14T15:00:44.944 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.944 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.944 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-14T15:00:44.944 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-14T15:00:45.019 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-04-14T15:00:45.320 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-04-14T15:00:45.321 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-04-14T15:00:45.321 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-04-14T15:00:45.321 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-04-14T15:00:45.321 INFO:teuthology.orchestra.run.smithi090.stdout: ceph x86_64 2:19.0.0-2778.ga971fd5f.el9 ceph 6.6 k 2024-04-14T15:00:45.321 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-04-14T15:00:45.321 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mds x86_64 2:19.0.0-2778.ga971fd5f.el9 ceph 2.4 M 2024-04-14T15:00:45.322 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr x86_64 2:19.0.0-2778.ga971fd5f.el9 ceph 1.5 M 2024-04-14T15:00:45.322 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2778.ga971fd5f.el9 ceph-noarch 249 k 2024-04-14T15:00:45.322 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mon x86_64 2:19.0.0-2778.ga971fd5f.el9 ceph 4.7 M 2024-04-14T15:00:45.322 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-osd x86_64 2:19.0.0-2778.ga971fd5f.el9 ceph 17 M 2024-04-14T15:00:45.322 INFO:teuthology.orchestra.run.smithi090.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-14T15:00:45.322 INFO:teuthology.orchestra.run.smithi090.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-14T15:00:45.322 INFO:teuthology.orchestra.run.smithi090.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-14T15:00:45.322 INFO:teuthology.orchestra.run.smithi090.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-14T15:00:45.322 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-14T15:00:45.323 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-14T15:00:45.323 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-14T15:00:45.323 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-14T15:00:45.323 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-14T15:00:45.323 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-14T15:00:45.323 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-collections 2024-04-14T15:00:45.323 INFO:teuthology.orchestra.run.smithi090.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-04-14T15:00:45.323 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-14T15:00:45.323 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-14T15:00:45.324 INFO:teuthology.orchestra.run.smithi090.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-14T15:00:45.324 INFO:teuthology.orchestra.run.smithi090.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-14T15:00:45.324 INFO:teuthology.orchestra.run.smithi090.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-14T15:00:45.324 INFO:teuthology.orchestra.run.smithi090.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-14T15:00:45.324 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-14T15:00:45.324 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-14T15:00:45.325 INFO:teuthology.orchestra.run.smithi090.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-14T15:00:45.325 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-14T15:00:45.325 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-14T15:00:45.325 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-14T15:00:45.325 INFO:teuthology.orchestra.run.smithi090.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-14T15:00:45.325 INFO:teuthology.orchestra.run.smithi090.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-14T15:00:45.325 INFO:teuthology.orchestra.run.smithi090.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-14T15:00:45.326 INFO:teuthology.orchestra.run.smithi090.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-14T15:00:45.326 INFO:teuthology.orchestra.run.smithi090.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-14T15:00:45.326 INFO:teuthology.orchestra.run.smithi090.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-14T15:00:45.326 INFO:teuthology.orchestra.run.smithi090.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-14T15:00:45.326 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:00:45.326 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-04-14T15:00:45.326 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-04-14T15:00:45.326 INFO:teuthology.orchestra.run.smithi090.stdout:Install 36 Packages 2024-04-14T15:00:45.326 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:00:45.326 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 30 M 2024-04-14T15:00:45.326 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 108 M 2024-04-14T15:00:45.327 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-04-14T15:00:45.695 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2778.ga971fd5f.el9.x86_64 5/6 2024-04-14T15:00:45.774 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-test-2:19.0.0-2778.ga971fd5f.el9.x86_64 6/6 2024-04-14T15:00:45.792 INFO:teuthology.orchestra.run.smithi090.stdout:(1/36): ceph-19.0.0-2778.ga971fd5f.el9.x86_64.r 23 kB/s | 6.6 kB 00:00 2024-04-14T15:00:46.717 INFO:teuthology.orchestra.run.smithi090.stdout:(2/36): ceph-mds-19.0.0-2778.ga971fd5f.el9.x86_ 1.9 MB/s | 2.4 MB 00:01 2024-04-14T15:00:46.934 INFO:teuthology.orchestra.run.smithi090.stdout:(3/36): ceph-mgr-19.0.0-2778.ga971fd5f.el9.x86_ 1.0 MB/s | 1.5 MB 00:01 2024-04-14T15:00:47.084 INFO:teuthology.orchestra.run.smithi090.stdout:(4/36): ceph-mgr-modules-core-19.0.0-2778.ga971 1.6 MB/s | 249 kB 00:00 2024-04-14T15:00:47.218 INFO:teuthology.orchestra.run.smithi090.stdout:(5/36): ceph-mon-19.0.0-2778.ga971fd5f.el9.x86_ 3.3 MB/s | 4.7 MB 00:01 2024-04-14T15:00:47.760 INFO:teuthology.orchestra.run.smithi090.stdout:(6/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 375 kB/s | 253 kB 00:00 2024-04-14T15:00:47.868 INFO:teuthology.orchestra.run.smithi090.stdout:(7/36): python3-ply-3.11-14.el9.noarch.rpm 984 kB/s | 106 kB 00:00 2024-04-14T15:00:47.985 INFO:teuthology.orchestra.run.smithi090.stdout:(8/36): python3-pycparser-2.20-6.el9.noarch.rpm 1.1 MB/s | 135 kB 00:00 2024-04-14T15:00:48.102 INFO:teuthology.orchestra.run.smithi090.stdout:(9/36): python3-requests-2.25.1-8.el9.noarch.rp 1.0 MB/s | 125 kB 00:00 2024-04-14T15:00:48.153 INFO:teuthology.orchestra.run.smithi090.stdout:(10/36): python3-cryptography-36.0.1-4.el9.x86_ 1.3 MB/s | 1.2 MB 00:00 2024-04-14T15:00:48.161 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-test-2:19.0.0-2778.ga971fd5f.el9.x86_64 6/6 2024-04-14T15:00:48.161 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-test-2:19.0.0-2778.ga971fd5f.el9.x86_64 1/6 2024-04-14T15:00:48.161 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libcephsqlite-2:19.0.0-2778.ga971fd5f.el9.x86_64 2/6 2024-04-14T15:00:48.161 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : jq-1.6-16.el9.x86_64 3/6 2024-04-14T15:00:48.161 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 4/6 2024-04-14T15:00:48.161 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/6 2024-04-14T15:00:48.211 INFO:teuthology.orchestra.run.smithi090.stdout:(11/36): python3-urllib3-1.26.5-5.el9.noarch.rp 1.9 MB/s | 215 kB 00:00 2024-04-14T15:00:48.579 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/6 2024-04-14T15:00:48.579 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:00:48.579 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2024-04-14T15:00:48.579 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-test-2:19.0.0-2778.ga971fd5f.el9.x86_64 2024-04-14T15:00:48.579 INFO:teuthology.orchestra.run.smithi092.stdout: jq-1.6-16.el9.x86_64 2024-04-14T15:00:48.579 INFO:teuthology.orchestra.run.smithi092.stdout: libcephsqlite-2:19.0.0-2778.ga971fd5f.el9.x86_64 2024-04-14T15:00:48.580 INFO:teuthology.orchestra.run.smithi092.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-04-14T15:00:48.580 INFO:teuthology.orchestra.run.smithi092.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-14T15:00:48.580 INFO:teuthology.orchestra.run.smithi092.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-14T15:00:48.580 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:00:48.580 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2024-04-14T15:00:48.815 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph 2024-04-14T15:00:49.104 INFO:teuthology.orchestra.run.smithi090.stdout:(12/36): python3-mako-1.1.4-6.el9.noarch.rpm 193 kB/s | 172 kB 00:00 2024-04-14T15:00:49.130 INFO:teuthology.orchestra.run.smithi090.stdout:(13/36): lua-5.4.4-4.el9.x86_64.rpm 193 kB/s | 188 kB 00:00 2024-04-14T15:00:49.413 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:00:48 ago on Sun 14 Apr 2024 03:00:01 PM UTC. 2024-04-14T15:00:49.702 INFO:teuthology.orchestra.run.smithi092.stderr:Modular dependency problems: 2024-04-14T15:00:49.702 INFO:teuthology.orchestra.run.smithi092.stderr: 2024-04-14T15:00:49.702 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 1: conflicting requests 2024-04-14T15:00:49.703 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.703 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 2: conflicting requests 2024-04-14T15:00:49.703 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.703 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 3: conflicting requests 2024-04-14T15:00:49.703 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.703 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 4: conflicting requests 2024-04-14T15:00:49.703 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.703 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 5: conflicting requests 2024-04-14T15:00:49.703 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.703 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 6: conflicting requests 2024-04-14T15:00:49.703 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.703 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 7: conflicting requests 2024-04-14T15:00:49.703 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.703 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 8: conflicting requests 2024-04-14T15:00:49.704 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.704 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 9: conflicting requests 2024-04-14T15:00:49.704 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.704 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 10: conflicting requests 2024-04-14T15:00:49.704 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.704 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 11: conflicting requests 2024-04-14T15:00:49.704 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.704 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 12: conflicting requests 2024-04-14T15:00:49.704 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.704 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 13: conflicting requests 2024-04-14T15:00:49.704 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.704 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 14: conflicting requests 2024-04-14T15:00:49.704 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.704 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 15: conflicting requests 2024-04-14T15:00:49.705 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.705 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 16: conflicting requests 2024-04-14T15:00:49.705 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.705 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 17: conflicting requests 2024-04-14T15:00:49.705 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.705 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 18: conflicting requests 2024-04-14T15:00:49.705 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.705 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 19: conflicting requests 2024-04-14T15:00:49.705 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.705 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 20: conflicting requests 2024-04-14T15:00:49.705 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.705 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 21: conflicting requests 2024-04-14T15:00:49.705 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.706 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 22: conflicting requests 2024-04-14T15:00:49.706 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.706 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 23: conflicting requests 2024-04-14T15:00:49.706 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.706 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 24: conflicting requests 2024-04-14T15:00:49.706 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.706 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 25: conflicting requests 2024-04-14T15:00:49.706 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.706 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 26: conflicting requests 2024-04-14T15:00:49.706 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.706 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 27: conflicting requests 2024-04-14T15:00:49.706 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.706 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 28: conflicting requests 2024-04-14T15:00:49.707 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.707 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 29: conflicting requests 2024-04-14T15:00:49.707 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.707 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 30: conflicting requests 2024-04-14T15:00:49.707 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.707 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 31: conflicting requests 2024-04-14T15:00:49.707 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.707 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 32: conflicting requests 2024-04-14T15:00:49.707 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.707 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 33: conflicting requests 2024-04-14T15:00:49.707 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.707 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 34: conflicting requests 2024-04-14T15:00:49.707 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.708 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 35: conflicting requests 2024-04-14T15:00:49.708 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.708 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 36: conflicting requests 2024-04-14T15:00:49.708 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.708 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 37: conflicting requests 2024-04-14T15:00:49.708 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.708 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.708 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.708 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 38: cannot install the best candidate for the job 2024-04-14T15:00:49.708 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.708 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.708 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.708 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.708 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 39: conflicting requests 2024-04-14T15:00:49.709 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.709 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.709 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.709 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.709 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.709 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.709 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 40: cannot install the best candidate for the job 2024-04-14T15:00:49.709 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.709 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.709 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.709 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.709 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 41: cannot install the best candidate for the job 2024-04-14T15:00:49.709 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.709 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.710 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.710 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.710 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 42: cannot install the best candidate for the job 2024-04-14T15:00:49.710 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.710 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.710 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.710 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.710 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 43: cannot install the best candidate for the job 2024-04-14T15:00:49.710 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.710 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.710 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.710 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.710 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 44: cannot install the best candidate for the job 2024-04-14T15:00:49.710 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.711 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.711 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.711 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.711 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 45: cannot install the best candidate for the job 2024-04-14T15:00:49.711 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.711 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.711 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.711 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-14T15:00:49.747 INFO:teuthology.orchestra.run.smithi090.stdout:(14/36): ceph-osd-19.0.0-2778.ga971fd5f.el9.x86 5.6 MB/s | 17 MB 00:03 2024-04-14T15:00:49.772 INFO:teuthology.orchestra.run.smithi090.stdout:(15/36): python3-markupsafe-1.1.1-12.el9.x86_64 52 kB/s | 35 kB 00:00 2024-04-14T15:00:49.774 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2024-04-14T15:00:49.777 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-04-14T15:00:49.777 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repository Size 2024-04-14T15:00:49.777 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-04-14T15:00:49.777 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2024-04-14T15:00:49.777 INFO:teuthology.orchestra.run.smithi092.stdout: ceph x86_64 2:19.0.0-2778.ga971fd5f.el9 ceph 6.6 k 2024-04-14T15:00:49.778 INFO:teuthology.orchestra.run.smithi092.stdout:Installing dependencies: 2024-04-14T15:00:49.778 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mds x86_64 2:19.0.0-2778.ga971fd5f.el9 ceph 2.4 M 2024-04-14T15:00:49.778 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr x86_64 2:19.0.0-2778.ga971fd5f.el9 ceph 1.5 M 2024-04-14T15:00:49.778 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2778.ga971fd5f.el9 ceph-noarch 249 k 2024-04-14T15:00:49.778 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mon x86_64 2:19.0.0-2778.ga971fd5f.el9 ceph 4.7 M 2024-04-14T15:00:49.778 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-osd x86_64 2:19.0.0-2778.ga971fd5f.el9 ceph 17 M 2024-04-14T15:00:49.778 INFO:teuthology.orchestra.run.smithi092.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-14T15:00:49.778 INFO:teuthology.orchestra.run.smithi092.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-14T15:00:49.778 INFO:teuthology.orchestra.run.smithi092.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-14T15:00:49.778 INFO:teuthology.orchestra.run.smithi092.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-14T15:00:49.778 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-14T15:00:49.778 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-14T15:00:49.779 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-14T15:00:49.779 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-14T15:00:49.779 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-14T15:00:49.779 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-14T15:00:49.779 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-collections 2024-04-14T15:00:49.779 INFO:teuthology.orchestra.run.smithi092.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-04-14T15:00:49.779 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-14T15:00:49.779 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-14T15:00:49.779 INFO:teuthology.orchestra.run.smithi092.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-14T15:00:49.779 INFO:teuthology.orchestra.run.smithi092.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-14T15:00:49.779 INFO:teuthology.orchestra.run.smithi092.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-14T15:00:49.780 INFO:teuthology.orchestra.run.smithi092.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-14T15:00:49.780 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-14T15:00:49.780 INFO:teuthology.orchestra.run.smithi092.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-14T15:00:49.780 INFO:teuthology.orchestra.run.smithi092.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-14T15:00:49.780 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-14T15:00:49.780 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-14T15:00:49.780 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-14T15:00:49.780 INFO:teuthology.orchestra.run.smithi092.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-14T15:00:49.780 INFO:teuthology.orchestra.run.smithi092.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-14T15:00:49.780 INFO:teuthology.orchestra.run.smithi092.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-14T15:00:49.780 INFO:teuthology.orchestra.run.smithi092.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-14T15:00:49.780 INFO:teuthology.orchestra.run.smithi092.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-14T15:00:49.780 INFO:teuthology.orchestra.run.smithi092.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-14T15:00:49.781 INFO:teuthology.orchestra.run.smithi092.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-14T15:00:49.781 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:00:49.781 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2024-04-14T15:00:49.781 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-04-14T15:00:49.781 INFO:teuthology.orchestra.run.smithi092.stdout:Install 36 Packages 2024-04-14T15:00:49.781 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:00:49.781 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 30 M 2024-04-14T15:00:49.782 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 108 M 2024-04-14T15:00:49.782 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2024-04-14T15:00:49.798 INFO:teuthology.orchestra.run.smithi090.stdout:(16/36): python3-pytz-2021.1-5.el9.noarch.rpm 76 kB/s | 51 kB 00:00 2024-04-14T15:00:49.965 INFO:teuthology.orchestra.run.smithi090.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 116 kB/s | 22 kB 00:00 2024-04-14T15:00:50.157 INFO:teuthology.orchestra.run.smithi090.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 421 kB/s | 151 kB 00:00 2024-04-14T15:00:50.182 INFO:teuthology.orchestra.run.smithi090.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 200 kB/s | 43 kB 00:00 2024-04-14T15:00:50.332 INFO:teuthology.orchestra.run.smithi090.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 980 kB/s | 172 kB 00:00 2024-04-14T15:00:50.374 INFO:teuthology.orchestra.run.smithi090.stdout:(21/36): python3-toml-0.10.2-6.el9.noarch.rpm 66 kB/s | 42 kB 00:00 2024-04-14T15:00:50.408 INFO:teuthology.orchestra.run.smithi090.stdout:(22/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.6 MB/s | 358 kB 00:00 2024-04-14T15:00:50.434 INFO:teuthology.orchestra.run.smithi090.stdout:(23/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 106 kB/s | 11 kB 00:00 2024-04-14T15:00:50.492 INFO:teuthology.orchestra.run.smithi090.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 277 kB/s | 23 kB 00:00 2024-04-14T15:00:50.517 INFO:teuthology.orchestra.run.smithi090.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 233 kB/s | 19 kB 00:00 2024-04-14T15:00:50.568 INFO:teuthology.orchestra.run.smithi090.stdout:(26/36): python3-jaraco-classes-3.2.1-5.el9.noa 92 kB/s | 18 kB 00:00 2024-04-14T15:00:50.593 INFO:teuthology.orchestra.run.smithi090.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 196 kB/s | 20 kB 00:00 2024-04-14T15:00:50.618 INFO:teuthology.orchestra.run.smithi090.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 460 kB/s | 46 kB 00:00 2024-04-14T15:00:50.744 INFO:teuthology.orchestra.run.smithi090.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 448 kB/s | 79 kB 00:00 2024-04-14T15:00:50.769 INFO:teuthology.orchestra.run.smithi090.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 109 kB/s | 16 kB 00:00 2024-04-14T15:00:50.811 INFO:teuthology.orchestra.run.smithi090.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.2 MB/s | 272 kB 00:00 2024-04-14T15:00:50.853 INFO:teuthology.orchestra.run.smithi090.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 427 kB/s | 36 kB 00:00 2024-04-14T15:00:50.878 INFO:teuthology.orchestra.run.smithi090.stdout:(33/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 669 kB/s | 90 kB 00:00 2024-04-14T15:00:50.995 INFO:teuthology.orchestra.run.smithi090.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 171 kB/s | 20 kB 00:00 2024-04-14T15:00:51.029 INFO:teuthology.orchestra.run.smithi090.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.4 MB/s | 427 kB 00:00 2024-04-14T15:00:51.054 INFO:teuthology.orchestra.run.smithi090.stdout:(36/36): python3-webob-1.8.7-6.el9.noarch.rpm 946 kB/s | 230 kB 00:00 2024-04-14T15:00:51.057 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-04-14T15:00:51.057 INFO:teuthology.orchestra.run.smithi090.stdout:Total 5.0 MB/s | 30 MB 00:06 2024-04-14T15:00:51.199 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-04-14T15:00:51.235 INFO:teuthology.orchestra.run.smithi092.stdout:(1/36): ceph-19.0.0-2778.ga971fd5f.el9.x86_64.r 30 kB/s | 6.6 kB 00:00 2024-04-14T15:00:51.238 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-04-14T15:00:51.238 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-04-14T15:00:51.368 INFO:teuthology.orchestra.run.smithi092.stdout:(2/36): ceph-mgr-19.0.0-2778.ga971fd5f.el9.x86_ 4.2 MB/s | 1.5 MB 00:00 2024-04-14T15:00:51.686 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-04-14T15:00:51.686 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-04-14T15:00:52.193 INFO:teuthology.orchestra.run.smithi092.stdout:(3/36): ceph-osd-19.0.0-2778.ga971fd5f.el9.x86_ 20 MB/s | 17 MB 00:00 2024-04-14T15:00:52.243 INFO:teuthology.orchestra.run.smithi092.stdout:(4/36): ceph-mgr-modules-core-19.0.0-2778.ga971 4.9 MB/s | 249 kB 00:00 2024-04-14T15:00:52.335 INFO:teuthology.orchestra.run.smithi092.stdout:(5/36): ceph-mds-19.0.0-2778.ga971fd5f.el9.x86_ 1.8 MB/s | 2.4 MB 00:01 2024-04-14T15:00:52.711 INFO:teuthology.orchestra.run.smithi092.stdout:(6/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 542 kB/s | 253 kB 00:00 2024-04-14T15:00:52.852 INFO:teuthology.orchestra.run.smithi092.stdout:(7/36): ceph-mon-19.0.0-2778.ga971fd5f.el9.x86_ 2.9 MB/s | 4.7 MB 00:01 2024-04-14T15:00:52.878 INFO:teuthology.orchestra.run.smithi092.stdout:(8/36): python3-ply-3.11-14.el9.noarch.rpm 639 kB/s | 106 kB 00:00 2024-04-14T15:00:52.953 INFO:teuthology.orchestra.run.smithi092.stdout:(9/36): python3-cryptography-36.0.1-4.el9.x86_6 2.0 MB/s | 1.2 MB 00:00 2024-04-14T15:00:53.016 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-04-14T15:00:53.037 INFO:teuthology.orchestra.run.smithi092.stdout:(10/36): python3-requests-2.25.1-8.el9.noarch.r 787 kB/s | 125 kB 00:00 2024-04-14T15:00:53.062 INFO:teuthology.orchestra.run.smithi092.stdout:(11/36): python3-urllib3-1.26.5-5.el9.noarch.rp 1.9 MB/s | 215 kB 00:00 2024-04-14T15:00:53.083 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-04-14T15:00:53.133 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-04-14T15:00:53.169 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-04-14T15:00:53.195 INFO:teuthology.orchestra.run.smithi092.stdout:(12/36): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 188 kB 00:00 2024-04-14T15:00:53.221 INFO:teuthology.orchestra.run.smithi092.stdout:(13/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-04-14T15:00:53.237 INFO:teuthology.orchestra.run.smithi092.stdout:(14/36): python3-markupsafe-1.1.1-12.el9.x86_64 832 kB/s | 35 kB 00:00 2024-04-14T15:00:53.249 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-04-14T15:00:53.263 INFO:teuthology.orchestra.run.smithi092.stdout:(15/36): python3-pycparser-2.20-6.el9.noarch.rp 330 kB/s | 135 kB 00:00 2024-04-14T15:00:53.280 INFO:teuthology.orchestra.run.smithi092.stdout:(16/36): python3-pytz-2021.1-5.el9.noarch.rpm 865 kB/s | 51 kB 00:00 2024-04-14T15:00:53.293 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-04-14T15:00:53.296 INFO:teuthology.orchestra.run.smithi092.stdout:(17/36): python3-toml-0.10.2-6.el9.noarch.rpm 708 kB/s | 42 kB 00:00 2024-04-14T15:00:53.326 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-04-14T15:00:53.418 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-04-14T15:00:53.447 INFO:teuthology.orchestra.run.smithi092.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 290 kB/s | 43 kB 00:00 2024-04-14T15:00:53.472 INFO:teuthology.orchestra.run.smithi092.stdout:(19/36): lua-devel-5.4.4-4.el9.x86_64.rpm 107 kB/s | 22 kB 00:00 2024-04-14T15:00:53.498 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-04-14T15:00:53.522 INFO:teuthology.orchestra.run.smithi092.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 623 kB/s | 151 kB 00:00 2024-04-14T15:00:53.546 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-04-14T15:00:53.581 INFO:teuthology.orchestra.run.smithi092.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 184 kB/s | 11 kB 00:00 2024-04-14T15:00:53.581 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-04-14T15:00:53.606 INFO:teuthology.orchestra.run.smithi092.stdout:(22/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-04-14T15:00:53.612 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/36 2024-04-14T15:00:53.640 INFO:teuthology.orchestra.run.smithi092.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 301 kB/s | 18 kB 00:00 2024-04-14T15:00:53.647 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/36 2024-04-14T15:00:53.665 INFO:teuthology.orchestra.run.smithi092.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 394 kB/s | 23 kB 00:00 2024-04-14T15:00:53.680 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/36 2024-04-14T15:00:53.698 INFO:teuthology.orchestra.run.smithi092.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 333 kB/s | 19 kB 00:00 2024-04-14T15:00:53.721 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/36 2024-04-14T15:00:53.724 INFO:teuthology.orchestra.run.smithi092.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 336 kB/s | 20 kB 00:00 2024-04-14T15:00:53.749 INFO:teuthology.orchestra.run.smithi092.stdout:(27/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.3 MB/s | 358 kB 00:00 2024-04-14T15:00:53.764 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/36 2024-04-14T15:00:53.774 INFO:teuthology.orchestra.run.smithi092.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 612 kB/s | 46 kB 00:00 2024-04-14T15:00:53.799 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/36 2024-04-14T15:00:53.799 INFO:teuthology.orchestra.run.smithi092.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-04-14T15:00:53.825 INFO:teuthology.orchestra.run.smithi092.stdout:(30/36): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-04-14T15:00:53.837 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/36 2024-04-14T15:00:53.851 INFO:teuthology.orchestra.run.smithi092.stdout:(31/36): python3-portend-3.1.0-2.el9.noarch.rpm 217 kB/s | 16 kB 00:00 2024-04-14T15:00:53.876 INFO:teuthology.orchestra.run.smithi092.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-04-14T15:00:53.901 INFO:teuthology.orchestra.run.smithi092.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 473 kB/s | 36 kB 00:00 2024-04-14T15:00:53.918 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 18/36 2024-04-14T15:00:53.951 INFO:teuthology.orchestra.run.smithi092.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 401 kB/s | 20 kB 00:00 2024-04-14T15:00:53.976 INFO:teuthology.orchestra.run.smithi092.stdout:(35/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.8 MB/s | 230 kB 00:00 2024-04-14T15:00:54.001 INFO:teuthology.orchestra.run.smithi092.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.3 MB/s | 427 kB 00:00 2024-04-14T15:00:54.003 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2024-04-14T15:00:54.003 INFO:teuthology.orchestra.run.smithi092.stdout:Total 7.1 MB/s | 30 MB 00:04 2024-04-14T15:00:54.055 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 19/36 2024-04-14T15:00:54.140 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 20/36 2024-04-14T15:00:54.154 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2024-04-14T15:00:54.200 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2024-04-14T15:00:54.200 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2024-04-14T15:00:54.201 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 21/36 2024-04-14T15:00:54.248 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 22/36 2024-04-14T15:00:54.306 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-ply-3.11-14.el9.noarch 23/36 2024-04-14T15:00:54.370 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 24/36 2024-04-14T15:00:54.555 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 25/36 2024-04-14T15:00:54.611 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 26/36 2024-04-14T15:00:54.619 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2024-04-14T15:00:54.620 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2024-04-14T15:00:54.689 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 27/36 2024-04-14T15:00:54.779 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 28/36 2024-04-14T15:00:54.819 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 29/36 2024-04-14T15:00:54.936 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 30/36 2024-04-14T15:00:55.113 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2778.ga971fd5f.el9. 31/36 2024-04-14T15:00:55.156 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-2:19.0.0-2778.ga971fd5f.el9.x86_64 32/36 2024-04-14T15:00:55.182 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2778.ga971fd5f.el9.x86_64 32/36 2024-04-14T15:00:55.183 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-14T15:00:55.183 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-14T15:00:55.183 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-14T15:00:55.183 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-14T15:00:55.183 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:00:55.960 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2024-04-14T15:00:56.020 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-04-14T15:00:56.059 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-04-14T15:00:56.098 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-04-14T15:00:56.169 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-04-14T15:00:56.213 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-04-14T15:00:56.246 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-04-14T15:00:56.337 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-04-14T15:00:56.415 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-04-14T15:00:56.456 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-04-14T15:00:56.500 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-04-14T15:00:56.530 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/36 2024-04-14T15:00:56.565 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/36 2024-04-14T15:00:56.598 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/36 2024-04-14T15:00:56.632 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-osd-2:19.0.0-2778.ga971fd5f.el9.x86_64 33/36 2024-04-14T15:00:56.638 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/36 2024-04-14T15:00:56.659 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-osd-2:19.0.0-2778.ga971fd5f.el9.x86_64 33/36 2024-04-14T15:00:56.659 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-14T15:00:56.659 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-14T15:00:56.660 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-14T15:00:56.660 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-14T15:00:56.660 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:00:56.673 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/36 2024-04-14T15:00:56.708 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/36 2024-04-14T15:00:56.746 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/36 2024-04-14T15:00:56.819 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 18/36 2024-04-14T15:00:56.955 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 19/36 2024-04-14T15:00:57.037 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 20/36 2024-04-14T15:00:57.089 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 21/36 2024-04-14T15:00:57.108 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mon-2:19.0.0-2778.ga971fd5f.el9.x86_64 34/36 2024-04-14T15:00:57.128 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 22/36 2024-04-14T15:00:57.132 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mon-2:19.0.0-2778.ga971fd5f.el9.x86_64 34/36 2024-04-14T15:00:57.132 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-14T15:00:57.132 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-14T15:00:57.132 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-14T15:00:57.132 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-14T15:00:57.132 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:00:57.178 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-ply-3.11-14.el9.noarch 23/36 2024-04-14T15:00:57.232 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 24/36 2024-04-14T15:00:57.367 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mds-2:19.0.0-2778.ga971fd5f.el9.x86_64 35/36 2024-04-14T15:00:57.391 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mds-2:19.0.0-2778.ga971fd5f.el9.x86_64 35/36 2024-04-14T15:00:57.391 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-14T15:00:57.391 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-14T15:00:57.391 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-14T15:00:57.391 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-14T15:00:57.391 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:00:57.417 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 25/36 2024-04-14T15:00:57.464 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-2:19.0.0-2778.ga971fd5f.el9.x86_64 36/36 2024-04-14T15:00:57.465 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 26/36 2024-04-14T15:00:57.536 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 27/36 2024-04-14T15:00:57.622 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 28/36 2024-04-14T15:00:57.656 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 29/36 2024-04-14T15:00:57.773 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 30/36 2024-04-14T15:00:57.938 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2778.ga971fd5f.el9. 31/36 2024-04-14T15:00:57.967 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-2:19.0.0-2778.ga971fd5f.el9.x86_64 32/36 2024-04-14T15:00:57.992 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2778.ga971fd5f.el9.x86_64 32/36 2024-04-14T15:00:57.993 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-14T15:00:57.993 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-14T15:00:57.993 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-14T15:00:57.993 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-14T15:00:57.993 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:00:59.434 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-osd-2:19.0.0-2778.ga971fd5f.el9.x86_64 33/36 2024-04-14T15:00:59.460 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-osd-2:19.0.0-2778.ga971fd5f.el9.x86_64 33/36 2024-04-14T15:00:59.460 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-14T15:00:59.460 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-14T15:00:59.460 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-14T15:00:59.460 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-14T15:00:59.460 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:00:59.584 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-2:19.0.0-2778.ga971fd5f.el9.x86_64 36/36 2024-04-14T15:00:59.584 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-2:19.0.0-2778.ga971fd5f.el9.x86_64 1/36 2024-04-14T15:00:59.584 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mds-2:19.0.0-2778.ga971fd5f.el9.x86_64 2/36 2024-04-14T15:00:59.584 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-2:19.0.0-2778.ga971fd5f.el9.x86_64 3/36 2024-04-14T15:00:59.584 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mon-2:19.0.0-2778.ga971fd5f.el9.x86_64 4/36 2024-04-14T15:00:59.584 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-osd-2:19.0.0-2778.ga971fd5f.el9.x86_64 5/36 2024-04-14T15:00:59.585 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2778.ga971fd5f.el9. 6/36 2024-04-14T15:00:59.585 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/36 2024-04-14T15:00:59.585 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/36 2024-04-14T15:00:59.585 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/36 2024-04-14T15:00:59.585 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/36 2024-04-14T15:00:59.585 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/36 2024-04-14T15:00:59.585 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/36 2024-04-14T15:00:59.585 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/36 2024-04-14T15:00:59.585 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/36 2024-04-14T15:00:59.585 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/36 2024-04-14T15:00:59.585 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/36 2024-04-14T15:00:59.585 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/36 2024-04-14T15:00:59.585 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-04-14T15:00:59.586 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-04-14T15:00:59.586 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-04-14T15:00:59.586 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-14T15:00:59.586 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-04-14T15:00:59.586 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-04-14T15:00:59.586 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-04-14T15:00:59.586 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-04-14T15:00:59.586 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-04-14T15:00:59.586 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-04-14T15:00:59.587 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-04-14T15:00:59.587 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-04-14T15:00:59.587 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-14T15:00:59.587 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-04-14T15:00:59.588 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-04-14T15:00:59.588 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-04-14T15:00:59.588 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-04-14T15:00:59.588 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-04-14T15:00:59.901 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mon-2:19.0.0-2778.ga971fd5f.el9.x86_64 34/36 2024-04-14T15:00:59.926 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mon-2:19.0.0-2778.ga971fd5f.el9.x86_64 34/36 2024-04-14T15:00:59.926 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-14T15:00:59.926 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-14T15:00:59.926 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-14T15:00:59.926 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-14T15:00:59.926 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:01:00.152 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mds-2:19.0.0-2778.ga971fd5f.el9.x86_64 35/36 2024-04-14T15:01:00.175 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mds-2:19.0.0-2778.ga971fd5f.el9.x86_64 35/36 2024-04-14T15:01:00.175 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-14T15:01:00.176 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-14T15:01:00.176 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-14T15:01:00.176 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-14T15:01:00.176 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:01:00.241 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-2:19.0.0-2778.ga971fd5f.el9.x86_64 36/36 2024-04-14T15:01:00.626 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-04-14T15:01:00.626 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:01:00.626 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-04-14T15:01:00.626 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-2:19.0.0-2778.ga971fd5f.el9.x86_64 2024-04-14T15:01:00.626 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mds-2:19.0.0-2778.ga971fd5f.el9.x86_64 2024-04-14T15:01:00.626 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-2:19.0.0-2778.ga971fd5f.el9.x86_64 2024-04-14T15:01:00.626 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-modules-core-2:19.0.0-2778.ga971fd5f.el9.noarch 2024-04-14T15:01:00.626 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mon-2:19.0.0-2778.ga971fd5f.el9.x86_64 2024-04-14T15:01:00.627 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-osd-2:19.0.0-2778.ga971fd5f.el9.x86_64 2024-04-14T15:01:00.627 INFO:teuthology.orchestra.run.smithi090.stdout: lua-5.4.4-4.el9.x86_64 2024-04-14T15:01:00.627 INFO:teuthology.orchestra.run.smithi090.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-14T15:01:00.627 INFO:teuthology.orchestra.run.smithi090.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-14T15:01:00.627 INFO:teuthology.orchestra.run.smithi090.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-14T15:01:00.627 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-14T15:01:00.627 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-14T15:01:00.627 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-14T15:01:00.627 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-14T15:01:00.627 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-14T15:01:00.627 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-14T15:01:00.627 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-14T15:01:00.627 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-14T15:01:00.628 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-14T15:01:00.628 INFO:teuthology.orchestra.run.smithi090.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-14T15:01:00.628 INFO:teuthology.orchestra.run.smithi090.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-14T15:01:00.628 INFO:teuthology.orchestra.run.smithi090.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-14T15:01:00.628 INFO:teuthology.orchestra.run.smithi090.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-14T15:01:00.628 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-14T15:01:00.628 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ply-3.11-14.el9.noarch 2024-04-14T15:01:00.628 INFO:teuthology.orchestra.run.smithi090.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-14T15:01:00.628 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-14T15:01:00.628 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-14T15:01:00.628 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-14T15:01:00.628 INFO:teuthology.orchestra.run.smithi090.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-14T15:01:00.628 INFO:teuthology.orchestra.run.smithi090.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-14T15:01:00.628 INFO:teuthology.orchestra.run.smithi090.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-14T15:01:00.629 INFO:teuthology.orchestra.run.smithi090.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-14T15:01:00.629 INFO:teuthology.orchestra.run.smithi090.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-14T15:01:00.629 INFO:teuthology.orchestra.run.smithi090.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-14T15:01:00.629 INFO:teuthology.orchestra.run.smithi090.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-14T15:01:00.629 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:01:00.629 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-04-14T15:01:01.056 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-base 2024-04-14T15:01:01.653 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:03 ago on Sun 14 Apr 2024 02:59:58 PM UTC. 2024-04-14T15:01:01.948 INFO:teuthology.orchestra.run.smithi090.stderr:Modular dependency problems: 2024-04-14T15:01:01.948 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-04-14T15:01:01.948 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 1: conflicting requests 2024-04-14T15:01:01.948 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.948 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 2: conflicting requests 2024-04-14T15:01:01.948 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.948 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 3: conflicting requests 2024-04-14T15:01:01.948 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.949 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 4: conflicting requests 2024-04-14T15:01:01.949 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.949 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 5: conflicting requests 2024-04-14T15:01:01.949 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.949 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 6: conflicting requests 2024-04-14T15:01:01.949 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.949 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 7: conflicting requests 2024-04-14T15:01:01.949 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.949 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 8: conflicting requests 2024-04-14T15:01:01.949 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.949 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 9: conflicting requests 2024-04-14T15:01:01.949 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.949 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 10: conflicting requests 2024-04-14T15:01:01.949 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.950 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 11: conflicting requests 2024-04-14T15:01:01.950 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.950 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 12: conflicting requests 2024-04-14T15:01:01.950 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.950 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 13: conflicting requests 2024-04-14T15:01:01.950 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.950 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 14: conflicting requests 2024-04-14T15:01:01.950 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.950 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 15: conflicting requests 2024-04-14T15:01:01.950 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.950 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 16: conflicting requests 2024-04-14T15:01:01.950 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.950 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 17: conflicting requests 2024-04-14T15:01:01.951 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.951 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 18: conflicting requests 2024-04-14T15:01:01.951 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.951 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 19: conflicting requests 2024-04-14T15:01:01.951 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.951 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 20: conflicting requests 2024-04-14T15:01:01.951 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.951 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 21: conflicting requests 2024-04-14T15:01:01.951 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.951 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 22: conflicting requests 2024-04-14T15:01:01.951 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.951 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 23: conflicting requests 2024-04-14T15:01:01.951 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.952 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 24: conflicting requests 2024-04-14T15:01:01.952 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.952 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 25: conflicting requests 2024-04-14T15:01:01.952 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.952 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 26: conflicting requests 2024-04-14T15:01:01.952 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.952 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 27: conflicting requests 2024-04-14T15:01:01.952 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.952 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 28: conflicting requests 2024-04-14T15:01:01.952 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.952 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 29: conflicting requests 2024-04-14T15:01:01.952 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.952 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 30: conflicting requests 2024-04-14T15:01:01.952 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.953 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 31: conflicting requests 2024-04-14T15:01:01.953 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.953 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 32: conflicting requests 2024-04-14T15:01:01.953 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.953 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 33: conflicting requests 2024-04-14T15:01:01.953 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.953 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 34: conflicting requests 2024-04-14T15:01:01.953 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.953 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 35: conflicting requests 2024-04-14T15:01:01.953 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.953 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 36: conflicting requests 2024-04-14T15:01:01.953 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.953 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 37: conflicting requests 2024-04-14T15:01:01.953 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.954 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.954 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.954 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 38: cannot install the best candidate for the job 2024-04-14T15:01:01.954 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.954 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.954 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.954 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.954 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 39: conflicting requests 2024-04-14T15:01:01.954 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.954 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.954 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.954 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.954 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.955 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.955 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 40: cannot install the best candidate for the job 2024-04-14T15:01:01.955 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.955 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.955 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.955 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.955 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 41: cannot install the best candidate for the job 2024-04-14T15:01:01.955 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.955 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.955 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.955 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.955 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 42: cannot install the best candidate for the job 2024-04-14T15:01:01.955 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.955 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.956 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.956 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.956 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 43: cannot install the best candidate for the job 2024-04-14T15:01:01.956 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.956 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.956 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.956 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.956 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 44: cannot install the best candidate for the job 2024-04-14T15:01:01.956 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.956 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.956 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.956 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.956 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 45: cannot install the best candidate for the job 2024-04-14T15:01:01.956 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.957 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.957 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.957 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-14T15:01:01.957 INFO:teuthology.orchestra.run.smithi090.stdout:Package ceph-base-2:19.0.0-2778.ga971fd5f.el9.x86_64 is already installed. 2024-04-14T15:01:01.982 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-2:19.0.0-2778.ga971fd5f.el9.x86_64 36/36 2024-04-14T15:01:01.982 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-2:19.0.0-2778.ga971fd5f.el9.x86_64 1/36 2024-04-14T15:01:01.982 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mds-2:19.0.0-2778.ga971fd5f.el9.x86_64 2/36 2024-04-14T15:01:01.982 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-2:19.0.0-2778.ga971fd5f.el9.x86_64 3/36 2024-04-14T15:01:01.982 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mon-2:19.0.0-2778.ga971fd5f.el9.x86_64 4/36 2024-04-14T15:01:01.982 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-osd-2:19.0.0-2778.ga971fd5f.el9.x86_64 5/36 2024-04-14T15:01:01.982 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2778.ga971fd5f.el9. 6/36 2024-04-14T15:01:01.982 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/36 2024-04-14T15:01:01.982 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/36 2024-04-14T15:01:01.982 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/36 2024-04-14T15:01:01.982 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/36 2024-04-14T15:01:01.983 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/36 2024-04-14T15:01:01.983 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/36 2024-04-14T15:01:01.983 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/36 2024-04-14T15:01:01.983 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/36 2024-04-14T15:01:01.983 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/36 2024-04-14T15:01:01.983 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/36 2024-04-14T15:01:01.983 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/36 2024-04-14T15:01:01.983 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-04-14T15:01:01.983 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-04-14T15:01:01.983 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-04-14T15:01:01.983 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-14T15:01:01.983 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-04-14T15:01:01.983 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-04-14T15:01:01.983 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-04-14T15:01:01.983 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-04-14T15:01:01.984 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-04-14T15:01:01.984 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-04-14T15:01:01.984 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-04-14T15:01:01.984 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-04-14T15:01:01.984 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-14T15:01:01.984 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-04-14T15:01:01.984 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-04-14T15:01:01.984 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-04-14T15:01:01.984 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-04-14T15:01:01.984 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-04-14T15:01:02.016 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-04-14T15:01:02.017 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-04-14T15:01:02.017 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-04-14T15:01:02.121 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install cephadm 2024-04-14T15:01:02.705 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:04 ago on Sun 14 Apr 2024 02:59:58 PM UTC. 2024-04-14T15:01:02.872 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-04-14T15:01:02.872 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:01:02.873 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2024-04-14T15:01:02.873 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-2:19.0.0-2778.ga971fd5f.el9.x86_64 2024-04-14T15:01:02.873 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mds-2:19.0.0-2778.ga971fd5f.el9.x86_64 2024-04-14T15:01:02.873 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-2:19.0.0-2778.ga971fd5f.el9.x86_64 2024-04-14T15:01:02.873 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-modules-core-2:19.0.0-2778.ga971fd5f.el9.noarch 2024-04-14T15:01:02.873 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mon-2:19.0.0-2778.ga971fd5f.el9.x86_64 2024-04-14T15:01:02.873 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-osd-2:19.0.0-2778.ga971fd5f.el9.x86_64 2024-04-14T15:01:02.873 INFO:teuthology.orchestra.run.smithi092.stdout: lua-5.4.4-4.el9.x86_64 2024-04-14T15:01:02.873 INFO:teuthology.orchestra.run.smithi092.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-14T15:01:02.873 INFO:teuthology.orchestra.run.smithi092.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-14T15:01:02.873 INFO:teuthology.orchestra.run.smithi092.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-14T15:01:02.873 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-14T15:01:02.873 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-14T15:01:02.874 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-14T15:01:02.874 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-14T15:01:02.874 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-14T15:01:02.874 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-14T15:01:02.874 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-14T15:01:02.874 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-14T15:01:02.874 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-14T15:01:02.874 INFO:teuthology.orchestra.run.smithi092.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-14T15:01:02.874 INFO:teuthology.orchestra.run.smithi092.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-14T15:01:02.874 INFO:teuthology.orchestra.run.smithi092.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-14T15:01:02.874 INFO:teuthology.orchestra.run.smithi092.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-14T15:01:02.874 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-14T15:01:02.874 INFO:teuthology.orchestra.run.smithi092.stdout: python3-ply-3.11-14.el9.noarch 2024-04-14T15:01:02.875 INFO:teuthology.orchestra.run.smithi092.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-14T15:01:02.875 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-14T15:01:02.875 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-14T15:01:02.875 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-14T15:01:02.875 INFO:teuthology.orchestra.run.smithi092.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-14T15:01:02.875 INFO:teuthology.orchestra.run.smithi092.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-14T15:01:02.875 INFO:teuthology.orchestra.run.smithi092.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-14T15:01:02.875 INFO:teuthology.orchestra.run.smithi092.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-14T15:01:02.875 INFO:teuthology.orchestra.run.smithi092.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-14T15:01:02.875 INFO:teuthology.orchestra.run.smithi092.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-14T15:01:02.875 INFO:teuthology.orchestra.run.smithi092.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-14T15:01:02.875 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:01:02.875 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2024-04-14T15:01:02.991 INFO:teuthology.orchestra.run.smithi090.stderr:Modular dependency problems: 2024-04-14T15:01:02.991 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-04-14T15:01:02.991 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 1: conflicting requests 2024-04-14T15:01:02.991 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.991 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 2: conflicting requests 2024-04-14T15:01:02.991 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.991 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 3: conflicting requests 2024-04-14T15:01:02.991 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.992 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 4: conflicting requests 2024-04-14T15:01:02.992 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.992 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 5: conflicting requests 2024-04-14T15:01:02.992 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.992 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 6: conflicting requests 2024-04-14T15:01:02.992 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.992 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 7: conflicting requests 2024-04-14T15:01:02.992 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.992 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 8: conflicting requests 2024-04-14T15:01:02.992 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.992 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 9: conflicting requests 2024-04-14T15:01:02.992 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.992 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 10: conflicting requests 2024-04-14T15:01:02.992 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.992 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 11: conflicting requests 2024-04-14T15:01:02.993 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.993 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 12: conflicting requests 2024-04-14T15:01:02.993 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.993 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 13: conflicting requests 2024-04-14T15:01:02.993 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.993 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 14: conflicting requests 2024-04-14T15:01:02.993 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.993 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 15: conflicting requests 2024-04-14T15:01:02.993 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.993 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 16: conflicting requests 2024-04-14T15:01:02.993 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.993 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 17: conflicting requests 2024-04-14T15:01:02.993 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.994 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 18: conflicting requests 2024-04-14T15:01:02.994 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.994 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 19: conflicting requests 2024-04-14T15:01:02.994 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.994 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 20: conflicting requests 2024-04-14T15:01:02.994 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.994 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 21: conflicting requests 2024-04-14T15:01:02.994 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.994 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 22: conflicting requests 2024-04-14T15:01:02.994 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.994 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 23: conflicting requests 2024-04-14T15:01:02.994 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.994 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 24: conflicting requests 2024-04-14T15:01:02.994 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.995 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 25: conflicting requests 2024-04-14T15:01:02.995 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.995 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 26: conflicting requests 2024-04-14T15:01:02.995 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.995 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 27: conflicting requests 2024-04-14T15:01:02.995 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.995 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 28: conflicting requests 2024-04-14T15:01:02.995 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.995 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 29: conflicting requests 2024-04-14T15:01:02.995 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.995 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 30: conflicting requests 2024-04-14T15:01:02.995 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.995 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 31: conflicting requests 2024-04-14T15:01:02.995 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.996 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 32: conflicting requests 2024-04-14T15:01:02.996 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.996 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 33: conflicting requests 2024-04-14T15:01:02.996 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.996 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 34: conflicting requests 2024-04-14T15:01:02.996 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.996 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 35: conflicting requests 2024-04-14T15:01:02.996 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.996 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 36: conflicting requests 2024-04-14T15:01:02.996 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.996 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 37: conflicting requests 2024-04-14T15:01:02.996 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.996 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.996 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.997 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 38: cannot install the best candidate for the job 2024-04-14T15:01:02.997 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.997 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.997 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.997 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.997 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 39: conflicting requests 2024-04-14T15:01:02.997 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.997 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.997 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.997 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.997 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.997 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.997 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 40: cannot install the best candidate for the job 2024-04-14T15:01:02.997 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.997 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.998 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.998 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.998 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 41: cannot install the best candidate for the job 2024-04-14T15:01:02.998 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.998 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.998 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.998 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.998 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 42: cannot install the best candidate for the job 2024-04-14T15:01:02.998 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.998 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.998 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.998 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.998 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 43: cannot install the best candidate for the job 2024-04-14T15:01:02.999 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.999 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.999 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.999 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.999 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 44: cannot install the best candidate for the job 2024-04-14T15:01:02.999 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.999 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.999 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.999 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.999 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 45: cannot install the best candidate for the job 2024-04-14T15:01:02.999 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.999 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.999 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-14T15:01:02.999 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-14T15:01:03.057 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-04-14T15:01:03.058 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-04-14T15:01:03.058 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-04-14T15:01:03.058 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-04-14T15:01:03.059 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-04-14T15:01:03.059 INFO:teuthology.orchestra.run.smithi090.stdout: cephadm noarch 2:19.0.0-2778.ga971fd5f.el9 ceph-noarch 762 k 2024-04-14T15:01:03.059 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:01:03.059 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-04-14T15:01:03.059 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-04-14T15:01:03.059 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-04-14T15:01:03.059 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:01:03.059 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 762 k 2024-04-14T15:01:03.059 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 768 k 2024-04-14T15:01:03.059 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-04-14T15:01:03.218 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-base 2024-04-14T15:01:03.347 INFO:teuthology.orchestra.run.smithi090.stdout:cephadm-19.0.0-2778.ga971fd5f.el9.noarch.rpm 2.6 MB/s | 762 kB 00:00 2024-04-14T15:01:03.347 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-04-14T15:01:03.348 INFO:teuthology.orchestra.run.smithi090.stdout:Total 2.6 MB/s | 762 kB 00:00 2024-04-14T15:01:03.348 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-04-14T15:01:03.353 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-04-14T15:01:03.353 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-04-14T15:01:03.359 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-04-14T15:01:03.359 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-04-14T15:01:03.429 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-04-14T15:01:03.815 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:02 ago on Sun 14 Apr 2024 03:00:01 PM UTC. 2024-04-14T15:01:03.900 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: cephadm-2:19.0.0-2778.ga971fd5f.el9.noarch 1/1 2024-04-14T15:01:03.996 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : cephadm-2:19.0.0-2778.ga971fd5f.el9.noarch 1/1 2024-04-14T15:01:04.102 INFO:teuthology.orchestra.run.smithi092.stderr:Modular dependency problems: 2024-04-14T15:01:04.102 INFO:teuthology.orchestra.run.smithi092.stderr: 2024-04-14T15:01:04.102 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 1: conflicting requests 2024-04-14T15:01:04.102 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.102 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 2: conflicting requests 2024-04-14T15:01:04.102 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.102 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 3: conflicting requests 2024-04-14T15:01:04.102 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.102 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 4: conflicting requests 2024-04-14T15:01:04.102 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.102 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 5: conflicting requests 2024-04-14T15:01:04.103 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.103 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 6: conflicting requests 2024-04-14T15:01:04.103 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.103 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 7: conflicting requests 2024-04-14T15:01:04.103 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.103 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 8: conflicting requests 2024-04-14T15:01:04.103 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.103 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 9: conflicting requests 2024-04-14T15:01:04.103 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.103 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 10: conflicting requests 2024-04-14T15:01:04.103 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.103 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 11: conflicting requests 2024-04-14T15:01:04.103 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.103 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 12: conflicting requests 2024-04-14T15:01:04.104 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.104 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 13: conflicting requests 2024-04-14T15:01:04.104 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.104 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 14: conflicting requests 2024-04-14T15:01:04.104 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.104 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 15: conflicting requests 2024-04-14T15:01:04.104 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.104 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 16: conflicting requests 2024-04-14T15:01:04.104 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.104 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 17: conflicting requests 2024-04-14T15:01:04.104 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.104 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 18: conflicting requests 2024-04-14T15:01:04.104 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.104 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 19: conflicting requests 2024-04-14T15:01:04.105 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.105 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 20: conflicting requests 2024-04-14T15:01:04.105 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.105 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 21: conflicting requests 2024-04-14T15:01:04.105 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.105 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 22: conflicting requests 2024-04-14T15:01:04.105 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.105 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 23: conflicting requests 2024-04-14T15:01:04.105 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.105 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 24: conflicting requests 2024-04-14T15:01:04.105 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.105 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 25: conflicting requests 2024-04-14T15:01:04.105 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.105 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 26: conflicting requests 2024-04-14T15:01:04.106 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.106 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 27: conflicting requests 2024-04-14T15:01:04.106 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.106 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 28: conflicting requests 2024-04-14T15:01:04.106 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.106 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 29: conflicting requests 2024-04-14T15:01:04.106 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.106 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 30: conflicting requests 2024-04-14T15:01:04.106 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.106 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 31: conflicting requests 2024-04-14T15:01:04.106 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.106 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 32: conflicting requests 2024-04-14T15:01:04.107 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.107 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 33: conflicting requests 2024-04-14T15:01:04.107 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.107 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 34: conflicting requests 2024-04-14T15:01:04.107 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.107 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 35: conflicting requests 2024-04-14T15:01:04.107 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.107 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 36: conflicting requests 2024-04-14T15:01:04.107 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.107 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 37: conflicting requests 2024-04-14T15:01:04.107 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.107 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.107 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.107 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 38: cannot install the best candidate for the job 2024-04-14T15:01:04.107 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.107 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.108 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.108 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.108 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 39: conflicting requests 2024-04-14T15:01:04.108 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.108 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.108 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.108 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.108 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.108 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.108 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 40: cannot install the best candidate for the job 2024-04-14T15:01:04.108 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.108 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.108 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.109 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.109 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 41: cannot install the best candidate for the job 2024-04-14T15:01:04.109 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.109 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.109 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.109 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.109 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 42: cannot install the best candidate for the job 2024-04-14T15:01:04.109 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.109 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.109 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.109 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.109 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 43: cannot install the best candidate for the job 2024-04-14T15:01:04.109 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.109 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.109 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.110 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.110 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 44: cannot install the best candidate for the job 2024-04-14T15:01:04.110 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.110 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.110 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.110 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.110 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 45: cannot install the best candidate for the job 2024-04-14T15:01:04.110 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.110 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.110 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.110 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-14T15:01:04.111 INFO:teuthology.orchestra.run.smithi092.stdout:Package ceph-base-2:19.0.0-2778.ga971fd5f.el9.x86_64 is already installed. 2024-04-14T15:01:04.168 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2024-04-14T15:01:04.169 INFO:teuthology.orchestra.run.smithi092.stdout:Nothing to do. 2024-04-14T15:01:04.169 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2024-04-14T15:01:04.245 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install cephadm 2024-04-14T15:01:04.415 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: cephadm-2:19.0.0-2778.ga971fd5f.el9.noarch 1/1 2024-04-14T15:01:04.721 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : cephadm-2:19.0.0-2778.ga971fd5f.el9.noarch 1/1 2024-04-14T15:01:04.721 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:01:04.721 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-04-14T15:01:04.721 INFO:teuthology.orchestra.run.smithi090.stdout: cephadm-2:19.0.0-2778.ga971fd5f.el9.noarch 2024-04-14T15:01:04.721 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:01:04.721 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-04-14T15:01:04.830 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:03 ago on Sun 14 Apr 2024 03:00:01 PM UTC. 2024-04-14T15:01:04.928 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-immutable-object-cache 2024-04-14T15:01:05.118 INFO:teuthology.orchestra.run.smithi092.stderr:Modular dependency problems: 2024-04-14T15:01:05.119 INFO:teuthology.orchestra.run.smithi092.stderr: 2024-04-14T15:01:05.119 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 1: conflicting requests 2024-04-14T15:01:05.119 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.119 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 2: conflicting requests 2024-04-14T15:01:05.119 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.119 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 3: conflicting requests 2024-04-14T15:01:05.119 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.119 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 4: conflicting requests 2024-04-14T15:01:05.119 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.119 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 5: conflicting requests 2024-04-14T15:01:05.119 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.119 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 6: conflicting requests 2024-04-14T15:01:05.119 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.120 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 7: conflicting requests 2024-04-14T15:01:05.120 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.120 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 8: conflicting requests 2024-04-14T15:01:05.120 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.120 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 9: conflicting requests 2024-04-14T15:01:05.120 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.120 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 10: conflicting requests 2024-04-14T15:01:05.120 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.120 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 11: conflicting requests 2024-04-14T15:01:05.120 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.120 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 12: conflicting requests 2024-04-14T15:01:05.120 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.120 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 13: conflicting requests 2024-04-14T15:01:05.120 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.121 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 14: conflicting requests 2024-04-14T15:01:05.121 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.121 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 15: conflicting requests 2024-04-14T15:01:05.121 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.121 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 16: conflicting requests 2024-04-14T15:01:05.121 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.121 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 17: conflicting requests 2024-04-14T15:01:05.121 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.121 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 18: conflicting requests 2024-04-14T15:01:05.121 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.121 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 19: conflicting requests 2024-04-14T15:01:05.121 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.121 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 20: conflicting requests 2024-04-14T15:01:05.121 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.122 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 21: conflicting requests 2024-04-14T15:01:05.122 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.122 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 22: conflicting requests 2024-04-14T15:01:05.122 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.122 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 23: conflicting requests 2024-04-14T15:01:05.122 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.122 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 24: conflicting requests 2024-04-14T15:01:05.122 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.122 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 25: conflicting requests 2024-04-14T15:01:05.122 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.122 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 26: conflicting requests 2024-04-14T15:01:05.122 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.122 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 27: conflicting requests 2024-04-14T15:01:05.123 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.123 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 28: conflicting requests 2024-04-14T15:01:05.123 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.123 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 29: conflicting requests 2024-04-14T15:01:05.123 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.123 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 30: conflicting requests 2024-04-14T15:01:05.123 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.123 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 31: conflicting requests 2024-04-14T15:01:05.123 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.123 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 32: conflicting requests 2024-04-14T15:01:05.123 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.123 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 33: conflicting requests 2024-04-14T15:01:05.123 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.123 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 34: conflicting requests 2024-04-14T15:01:05.124 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.124 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 35: conflicting requests 2024-04-14T15:01:05.124 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.124 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 36: conflicting requests 2024-04-14T15:01:05.124 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.124 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 37: conflicting requests 2024-04-14T15:01:05.124 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.124 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.124 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.124 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 38: cannot install the best candidate for the job 2024-04-14T15:01:05.124 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.124 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.124 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.125 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.125 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 39: conflicting requests 2024-04-14T15:01:05.125 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.125 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.125 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.125 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.125 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.125 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.125 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 40: cannot install the best candidate for the job 2024-04-14T15:01:05.125 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.125 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.125 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.125 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.125 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 41: cannot install the best candidate for the job 2024-04-14T15:01:05.125 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.126 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.126 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.126 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.126 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 42: cannot install the best candidate for the job 2024-04-14T15:01:05.126 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.126 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.126 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.126 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.126 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 43: cannot install the best candidate for the job 2024-04-14T15:01:05.126 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.126 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.126 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.126 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.127 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 44: cannot install the best candidate for the job 2024-04-14T15:01:05.127 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.127 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.127 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.127 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.127 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 45: cannot install the best candidate for the job 2024-04-14T15:01:05.127 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.127 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.127 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.127 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.186 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2024-04-14T15:01:05.187 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-04-14T15:01:05.187 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repository Size 2024-04-14T15:01:05.187 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-04-14T15:01:05.188 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2024-04-14T15:01:05.188 INFO:teuthology.orchestra.run.smithi092.stdout: cephadm noarch 2:19.0.0-2778.ga971fd5f.el9 ceph-noarch 762 k 2024-04-14T15:01:05.188 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:01:05.188 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2024-04-14T15:01:05.188 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-04-14T15:01:05.188 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2024-04-14T15:01:05.188 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:01:05.188 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 762 k 2024-04-14T15:01:05.188 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 768 k 2024-04-14T15:01:05.188 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2024-04-14T15:01:05.458 INFO:teuthology.orchestra.run.smithi092.stdout:cephadm-19.0.0-2778.ga971fd5f.el9.noarch.rpm 2.8 MB/s | 762 kB 00:00 2024-04-14T15:01:05.458 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2024-04-14T15:01:05.458 INFO:teuthology.orchestra.run.smithi092.stdout:Total 2.7 MB/s | 762 kB 00:00 2024-04-14T15:01:05.459 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2024-04-14T15:01:05.463 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2024-04-14T15:01:05.464 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2024-04-14T15:01:05.470 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2024-04-14T15:01:05.470 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2024-04-14T15:01:05.534 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:07 ago on Sun 14 Apr 2024 02:59:58 PM UTC. 2024-04-14T15:01:05.548 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2024-04-14T15:01:05.829 INFO:teuthology.orchestra.run.smithi090.stderr:Modular dependency problems: 2024-04-14T15:01:05.829 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-04-14T15:01:05.829 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 1: conflicting requests 2024-04-14T15:01:05.829 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.829 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 2: conflicting requests 2024-04-14T15:01:05.829 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.830 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 3: conflicting requests 2024-04-14T15:01:05.830 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.830 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 4: conflicting requests 2024-04-14T15:01:05.830 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.830 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 5: conflicting requests 2024-04-14T15:01:05.830 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.830 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 6: conflicting requests 2024-04-14T15:01:05.830 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.830 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 7: conflicting requests 2024-04-14T15:01:05.830 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.830 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 8: conflicting requests 2024-04-14T15:01:05.830 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.831 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 9: conflicting requests 2024-04-14T15:01:05.831 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.831 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 10: conflicting requests 2024-04-14T15:01:05.831 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.831 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 11: conflicting requests 2024-04-14T15:01:05.831 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.831 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 12: conflicting requests 2024-04-14T15:01:05.831 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.831 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 13: conflicting requests 2024-04-14T15:01:05.831 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.831 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 14: conflicting requests 2024-04-14T15:01:05.831 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.831 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 15: conflicting requests 2024-04-14T15:01:05.831 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.832 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 16: conflicting requests 2024-04-14T15:01:05.832 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.832 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 17: conflicting requests 2024-04-14T15:01:05.832 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.832 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 18: conflicting requests 2024-04-14T15:01:05.832 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.832 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 19: conflicting requests 2024-04-14T15:01:05.832 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.832 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 20: conflicting requests 2024-04-14T15:01:05.832 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.832 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 21: conflicting requests 2024-04-14T15:01:05.832 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.832 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 22: conflicting requests 2024-04-14T15:01:05.833 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.833 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 23: conflicting requests 2024-04-14T15:01:05.833 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.833 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 24: conflicting requests 2024-04-14T15:01:05.833 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.833 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 25: conflicting requests 2024-04-14T15:01:05.833 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.833 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 26: conflicting requests 2024-04-14T15:01:05.833 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.833 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 27: conflicting requests 2024-04-14T15:01:05.833 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.833 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 28: conflicting requests 2024-04-14T15:01:05.833 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.834 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 29: conflicting requests 2024-04-14T15:01:05.834 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.834 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 30: conflicting requests 2024-04-14T15:01:05.834 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.834 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 31: conflicting requests 2024-04-14T15:01:05.834 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.834 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 32: conflicting requests 2024-04-14T15:01:05.834 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.834 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 33: conflicting requests 2024-04-14T15:01:05.834 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.834 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 34: conflicting requests 2024-04-14T15:01:05.834 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.834 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 35: conflicting requests 2024-04-14T15:01:05.834 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.835 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 36: conflicting requests 2024-04-14T15:01:05.835 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.835 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 37: conflicting requests 2024-04-14T15:01:05.835 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.835 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.835 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.835 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 38: cannot install the best candidate for the job 2024-04-14T15:01:05.835 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.835 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.835 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.836 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.836 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 39: conflicting requests 2024-04-14T15:01:05.836 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.836 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.836 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.836 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.836 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.836 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.836 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 40: cannot install the best candidate for the job 2024-04-14T15:01:05.836 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.836 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.836 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.836 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.837 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 41: cannot install the best candidate for the job 2024-04-14T15:01:05.837 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.837 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.837 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.837 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.837 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 42: cannot install the best candidate for the job 2024-04-14T15:01:05.837 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.837 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.837 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.837 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.837 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 43: cannot install the best candidate for the job 2024-04-14T15:01:05.837 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.837 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.837 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.838 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.838 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 44: cannot install the best candidate for the job 2024-04-14T15:01:05.838 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.838 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.838 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.838 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.838 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 45: cannot install the best candidate for the job 2024-04-14T15:01:05.838 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.838 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.838 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.838 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-14T15:01:05.899 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-04-14T15:01:05.899 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-04-14T15:01:05.899 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2024-04-14T15:01:05.899 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-04-14T15:01:05.899 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-04-14T15:01:05.899 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2778.ga971fd5f.el9 ceph 145 k 2024-04-14T15:01:05.899 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:01:05.899 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-04-14T15:01:05.900 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-04-14T15:01:05.900 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-04-14T15:01:05.900 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:01:05.900 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 145 k 2024-04-14T15:01:05.900 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 435 k 2024-04-14T15:01:05.900 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-04-14T15:01:05.991 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: cephadm-2:19.0.0-2778.ga971fd5f.el9.noarch 1/1 2024-04-14T15:01:06.070 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : cephadm-2:19.0.0-2778.ga971fd5f.el9.noarch 1/1 2024-04-14T15:01:06.319 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-immutable-object-cache-19.0.0-2778.ga971fd 345 kB/s | 145 kB 00:00 2024-04-14T15:01:06.319 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-04-14T15:01:06.320 INFO:teuthology.orchestra.run.smithi090.stdout:Total 344 kB/s | 145 kB 00:00 2024-04-14T15:01:06.320 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-04-14T15:01:06.327 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-04-14T15:01:06.327 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-04-14T15:01:06.384 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-04-14T15:01:06.385 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-04-14T15:01:06.448 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: cephadm-2:19.0.0-2778.ga971fd5f.el9.noarch 1/1 2024-04-14T15:01:06.523 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-04-14T15:01:06.587 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2778.ga971fd5f. 1/1 2024-04-14T15:01:06.611 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2778.ga971fd5f. 1/1 2024-04-14T15:01:06.611 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-14T15:01:06.612 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-14T15:01:06.612 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:01:06.765 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : cephadm-2:19.0.0-2778.ga971fd5f.el9.noarch 1/1 2024-04-14T15:01:06.765 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:01:06.765 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2024-04-14T15:01:06.765 INFO:teuthology.orchestra.run.smithi092.stdout: cephadm-2:19.0.0-2778.ga971fd5f.el9.noarch 2024-04-14T15:01:06.766 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:01:06.766 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2024-04-14T15:01:06.960 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-immutable-object-cache 2024-04-14T15:01:07.562 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:06 ago on Sun 14 Apr 2024 03:00:01 PM UTC. 2024-04-14T15:01:07.714 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2778.ga971fd5f. 1/1 2024-04-14T15:01:07.714 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:01:07.714 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-04-14T15:01:07.714 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-immutable-object-cache-2:19.0.0-2778.ga971fd5f.el9.x86_64 2024-04-14T15:01:07.715 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:01:07.715 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-04-14T15:01:07.857 INFO:teuthology.orchestra.run.smithi092.stderr:Modular dependency problems: 2024-04-14T15:01:07.857 INFO:teuthology.orchestra.run.smithi092.stderr: 2024-04-14T15:01:07.857 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 1: conflicting requests 2024-04-14T15:01:07.857 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.857 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 2: conflicting requests 2024-04-14T15:01:07.857 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.857 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 3: conflicting requests 2024-04-14T15:01:07.857 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.857 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 4: conflicting requests 2024-04-14T15:01:07.857 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.857 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 5: conflicting requests 2024-04-14T15:01:07.857 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.858 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 6: conflicting requests 2024-04-14T15:01:07.858 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.858 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 7: conflicting requests 2024-04-14T15:01:07.858 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.858 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 8: conflicting requests 2024-04-14T15:01:07.858 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.858 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 9: conflicting requests 2024-04-14T15:01:07.858 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.858 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 10: conflicting requests 2024-04-14T15:01:07.858 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.858 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 11: conflicting requests 2024-04-14T15:01:07.858 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.858 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 12: conflicting requests 2024-04-14T15:01:07.858 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.859 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 13: conflicting requests 2024-04-14T15:01:07.859 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.859 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 14: conflicting requests 2024-04-14T15:01:07.859 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.859 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 15: conflicting requests 2024-04-14T15:01:07.859 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.859 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 16: conflicting requests 2024-04-14T15:01:07.859 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.859 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 17: conflicting requests 2024-04-14T15:01:07.859 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.859 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 18: conflicting requests 2024-04-14T15:01:07.859 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.859 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 19: conflicting requests 2024-04-14T15:01:07.859 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.859 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 20: conflicting requests 2024-04-14T15:01:07.859 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.860 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 21: conflicting requests 2024-04-14T15:01:07.860 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.860 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 22: conflicting requests 2024-04-14T15:01:07.860 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.860 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 23: conflicting requests 2024-04-14T15:01:07.860 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.860 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 24: conflicting requests 2024-04-14T15:01:07.860 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.860 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 25: conflicting requests 2024-04-14T15:01:07.860 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.860 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 26: conflicting requests 2024-04-14T15:01:07.860 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.861 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 27: conflicting requests 2024-04-14T15:01:07.861 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.861 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 28: conflicting requests 2024-04-14T15:01:07.861 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.861 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 29: conflicting requests 2024-04-14T15:01:07.861 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.861 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 30: conflicting requests 2024-04-14T15:01:07.861 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.861 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 31: conflicting requests 2024-04-14T15:01:07.861 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.861 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 32: conflicting requests 2024-04-14T15:01:07.862 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.862 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 33: conflicting requests 2024-04-14T15:01:07.862 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.862 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 34: conflicting requests 2024-04-14T15:01:07.862 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.862 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 35: conflicting requests 2024-04-14T15:01:07.862 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.862 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 36: conflicting requests 2024-04-14T15:01:07.862 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.862 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 37: conflicting requests 2024-04-14T15:01:07.862 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.862 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.862 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.862 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 38: cannot install the best candidate for the job 2024-04-14T15:01:07.863 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.863 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.863 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.863 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.863 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 39: conflicting requests 2024-04-14T15:01:07.863 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.863 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.863 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.863 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.863 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.863 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.863 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 40: cannot install the best candidate for the job 2024-04-14T15:01:07.863 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.863 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.864 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.864 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.864 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 41: cannot install the best candidate for the job 2024-04-14T15:01:07.864 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.864 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.864 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.864 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.864 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 42: cannot install the best candidate for the job 2024-04-14T15:01:07.864 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.864 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.864 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.864 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.865 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 43: cannot install the best candidate for the job 2024-04-14T15:01:07.865 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.865 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.865 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.865 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.865 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 44: cannot install the best candidate for the job 2024-04-14T15:01:07.865 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.865 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.865 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.865 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.865 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 45: cannot install the best candidate for the job 2024-04-14T15:01:07.865 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.865 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.865 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.865 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-14T15:01:07.924 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2024-04-14T15:01:07.925 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-04-14T15:01:07.925 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repo Size 2024-04-14T15:01:07.925 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-04-14T15:01:07.925 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2024-04-14T15:01:07.925 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2778.ga971fd5f.el9 ceph 145 k 2024-04-14T15:01:07.925 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:01:07.925 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2024-04-14T15:01:07.925 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-04-14T15:01:07.926 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2024-04-14T15:01:07.926 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:01:07.926 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 145 k 2024-04-14T15:01:07.926 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 435 k 2024-04-14T15:01:07.926 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2024-04-14T15:01:07.982 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-mgr 2024-04-14T15:01:08.421 INFO:teuthology.orchestra.run.smithi092.stdout:ceph-immutable-object-cache-19.0.0-2778.ga971fd 292 kB/s | 145 kB 00:00 2024-04-14T15:01:08.422 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2024-04-14T15:01:08.422 INFO:teuthology.orchestra.run.smithi092.stdout:Total 291 kB/s | 145 kB 00:00 2024-04-14T15:01:08.422 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2024-04-14T15:01:08.430 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2024-04-14T15:01:08.430 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2024-04-14T15:01:08.488 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2024-04-14T15:01:08.489 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2024-04-14T15:01:08.580 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:10 ago on Sun 14 Apr 2024 02:59:58 PM UTC. 2024-04-14T15:01:08.629 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2024-04-14T15:01:08.706 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2778.ga971fd5f. 1/1 2024-04-14T15:01:08.732 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2778.ga971fd5f. 1/1 2024-04-14T15:01:08.732 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-14T15:01:08.732 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-14T15:01:08.732 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:01:08.867 INFO:teuthology.orchestra.run.smithi090.stderr:Modular dependency problems: 2024-04-14T15:01:08.867 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-04-14T15:01:08.867 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 1: conflicting requests 2024-04-14T15:01:08.867 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.867 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 2: conflicting requests 2024-04-14T15:01:08.867 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.867 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 3: conflicting requests 2024-04-14T15:01:08.867 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.867 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 4: conflicting requests 2024-04-14T15:01:08.867 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.868 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 5: conflicting requests 2024-04-14T15:01:08.868 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.868 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 6: conflicting requests 2024-04-14T15:01:08.868 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.868 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 7: conflicting requests 2024-04-14T15:01:08.868 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.868 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 8: conflicting requests 2024-04-14T15:01:08.868 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.868 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 9: conflicting requests 2024-04-14T15:01:08.868 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.868 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 10: conflicting requests 2024-04-14T15:01:08.868 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.868 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 11: conflicting requests 2024-04-14T15:01:08.868 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.868 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 12: conflicting requests 2024-04-14T15:01:08.869 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.869 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 13: conflicting requests 2024-04-14T15:01:08.869 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.869 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 14: conflicting requests 2024-04-14T15:01:08.869 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.869 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 15: conflicting requests 2024-04-14T15:01:08.869 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.869 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 16: conflicting requests 2024-04-14T15:01:08.869 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.869 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 17: conflicting requests 2024-04-14T15:01:08.869 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.870 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 18: conflicting requests 2024-04-14T15:01:08.870 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.870 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 19: conflicting requests 2024-04-14T15:01:08.870 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.870 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 20: conflicting requests 2024-04-14T15:01:08.870 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.870 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 21: conflicting requests 2024-04-14T15:01:08.870 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.870 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 22: conflicting requests 2024-04-14T15:01:08.870 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.870 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 23: conflicting requests 2024-04-14T15:01:08.870 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.870 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 24: conflicting requests 2024-04-14T15:01:08.870 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.871 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 25: conflicting requests 2024-04-14T15:01:08.871 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.871 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 26: conflicting requests 2024-04-14T15:01:08.871 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.871 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 27: conflicting requests 2024-04-14T15:01:08.871 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.871 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 28: conflicting requests 2024-04-14T15:01:08.871 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.871 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 29: conflicting requests 2024-04-14T15:01:08.871 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.871 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 30: conflicting requests 2024-04-14T15:01:08.871 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.871 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 31: conflicting requests 2024-04-14T15:01:08.871 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.871 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 32: conflicting requests 2024-04-14T15:01:08.872 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.872 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 33: conflicting requests 2024-04-14T15:01:08.872 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.872 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 34: conflicting requests 2024-04-14T15:01:08.872 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.872 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 35: conflicting requests 2024-04-14T15:01:08.872 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.872 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 36: conflicting requests 2024-04-14T15:01:08.872 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.872 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 37: conflicting requests 2024-04-14T15:01:08.872 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.872 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.872 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.872 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 38: cannot install the best candidate for the job 2024-04-14T15:01:08.872 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.872 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.873 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.873 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.873 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 39: conflicting requests 2024-04-14T15:01:08.873 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.873 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.873 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.873 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.873 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.873 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.873 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 40: cannot install the best candidate for the job 2024-04-14T15:01:08.873 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.873 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.873 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.873 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.873 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 41: cannot install the best candidate for the job 2024-04-14T15:01:08.873 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.874 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.874 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.874 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.874 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 42: cannot install the best candidate for the job 2024-04-14T15:01:08.874 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.874 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.874 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.874 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.874 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 43: cannot install the best candidate for the job 2024-04-14T15:01:08.874 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.874 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.874 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.874 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.874 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 44: cannot install the best candidate for the job 2024-04-14T15:01:08.875 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.875 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.875 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.875 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.875 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 45: cannot install the best candidate for the job 2024-04-14T15:01:08.875 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.875 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.875 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.875 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-14T15:01:08.875 INFO:teuthology.orchestra.run.smithi090.stdout:Package ceph-mgr-2:19.0.0-2778.ga971fd5f.el9.x86_64 is already installed. 2024-04-14T15:01:08.934 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-04-14T15:01:08.935 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-04-14T15:01:08.936 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-04-14T15:01:08.999 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-mgr-dashboard 2024-04-14T15:01:09.591 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:11 ago on Sun 14 Apr 2024 02:59:58 PM UTC. 2024-04-14T15:01:09.751 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2778.ga971fd5f. 1/1 2024-04-14T15:01:09.752 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:01:09.752 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2024-04-14T15:01:09.752 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-immutable-object-cache-2:19.0.0-2778.ga971fd5f.el9.x86_64 2024-04-14T15:01:09.752 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:01:09.752 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2024-04-14T15:01:09.881 INFO:teuthology.orchestra.run.smithi090.stderr:Modular dependency problems: 2024-04-14T15:01:09.882 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-04-14T15:01:09.882 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 1: conflicting requests 2024-04-14T15:01:09.882 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.882 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 2: conflicting requests 2024-04-14T15:01:09.882 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.882 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 3: conflicting requests 2024-04-14T15:01:09.882 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.882 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 4: conflicting requests 2024-04-14T15:01:09.882 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.882 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 5: conflicting requests 2024-04-14T15:01:09.882 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.883 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 6: conflicting requests 2024-04-14T15:01:09.883 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.883 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 7: conflicting requests 2024-04-14T15:01:09.883 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.883 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 8: conflicting requests 2024-04-14T15:01:09.883 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.883 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 9: conflicting requests 2024-04-14T15:01:09.883 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.883 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 10: conflicting requests 2024-04-14T15:01:09.883 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.883 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 11: conflicting requests 2024-04-14T15:01:09.883 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.883 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 12: conflicting requests 2024-04-14T15:01:09.883 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.883 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 13: conflicting requests 2024-04-14T15:01:09.883 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.884 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 14: conflicting requests 2024-04-14T15:01:09.884 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.884 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 15: conflicting requests 2024-04-14T15:01:09.884 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.884 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 16: conflicting requests 2024-04-14T15:01:09.884 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.884 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 17: conflicting requests 2024-04-14T15:01:09.884 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.884 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 18: conflicting requests 2024-04-14T15:01:09.884 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.884 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 19: conflicting requests 2024-04-14T15:01:09.884 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.884 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 20: conflicting requests 2024-04-14T15:01:09.884 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.884 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 21: conflicting requests 2024-04-14T15:01:09.885 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.885 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 22: conflicting requests 2024-04-14T15:01:09.885 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.885 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 23: conflicting requests 2024-04-14T15:01:09.885 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.885 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 24: conflicting requests 2024-04-14T15:01:09.885 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.885 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 25: conflicting requests 2024-04-14T15:01:09.885 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.885 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 26: conflicting requests 2024-04-14T15:01:09.885 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.885 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 27: conflicting requests 2024-04-14T15:01:09.885 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.885 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 28: conflicting requests 2024-04-14T15:01:09.885 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.886 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 29: conflicting requests 2024-04-14T15:01:09.886 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.886 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 30: conflicting requests 2024-04-14T15:01:09.886 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.886 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 31: conflicting requests 2024-04-14T15:01:09.886 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.886 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 32: conflicting requests 2024-04-14T15:01:09.886 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.886 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 33: conflicting requests 2024-04-14T15:01:09.886 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.886 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 34: conflicting requests 2024-04-14T15:01:09.886 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.886 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 35: conflicting requests 2024-04-14T15:01:09.887 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.887 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 36: conflicting requests 2024-04-14T15:01:09.887 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.887 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 37: conflicting requests 2024-04-14T15:01:09.887 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.887 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.887 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.887 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 38: cannot install the best candidate for the job 2024-04-14T15:01:09.887 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.887 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.887 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.887 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.888 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 39: conflicting requests 2024-04-14T15:01:09.888 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.888 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.888 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.888 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.888 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.888 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.888 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 40: cannot install the best candidate for the job 2024-04-14T15:01:09.888 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.888 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.888 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.888 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.888 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 41: cannot install the best candidate for the job 2024-04-14T15:01:09.888 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.889 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.889 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.889 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.889 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 42: cannot install the best candidate for the job 2024-04-14T15:01:09.889 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.889 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.889 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.889 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.890 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 43: cannot install the best candidate for the job 2024-04-14T15:01:09.890 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.890 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.890 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.890 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.890 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 44: cannot install the best candidate for the job 2024-04-14T15:01:09.890 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.890 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.890 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.890 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.890 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 45: cannot install the best candidate for the job 2024-04-14T15:01:09.891 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.891 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.891 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.891 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-14T15:01:09.950 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-04-14T15:01:09.952 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-04-14T15:01:09.952 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-04-14T15:01:09.952 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-04-14T15:01:09.952 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-04-14T15:01:09.952 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2778.ga971fd5f.el9 ceph-noarch 3.5 M 2024-04-14T15:01:09.952 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-04-14T15:01:09.952 INFO:teuthology.orchestra.run.smithi090.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-14T15:01:09.952 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2778.ga971fd5f.el9 ceph-noarch 34 k 2024-04-14T15:01:09.952 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2778.ga971fd5f.el9 ceph-noarch 16 k 2024-04-14T15:01:09.952 INFO:teuthology.orchestra.run.smithi090.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-14T15:01:09.952 INFO:teuthology.orchestra.run.smithi090.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-14T15:01:09.952 INFO:teuthology.orchestra.run.smithi090.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-14T15:01:09.953 INFO:teuthology.orchestra.run.smithi090.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-14T15:01:09.953 INFO:teuthology.orchestra.run.smithi090.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-14T15:01:09.953 INFO:teuthology.orchestra.run.smithi090.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-14T15:01:09.953 INFO:teuthology.orchestra.run.smithi090.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-14T15:01:09.953 INFO:teuthology.orchestra.run.smithi090.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-14T15:01:09.953 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:01:09.953 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-04-14T15:01:09.953 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-04-14T15:01:09.953 INFO:teuthology.orchestra.run.smithi090.stdout:Install 12 Packages 2024-04-14T15:01:09.953 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:01:09.954 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 8.6 M 2024-04-14T15:01:09.954 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 104 M 2024-04-14T15:01:09.954 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-04-14T15:01:10.025 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-mgr 2024-04-14T15:01:10.630 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:09 ago on Sun 14 Apr 2024 03:00:01 PM UTC. 2024-04-14T15:01:10.922 INFO:teuthology.orchestra.run.smithi092.stderr:Modular dependency problems: 2024-04-14T15:01:10.922 INFO:teuthology.orchestra.run.smithi092.stderr: 2024-04-14T15:01:10.922 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 1: conflicting requests 2024-04-14T15:01:10.922 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.922 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 2: conflicting requests 2024-04-14T15:01:10.923 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.923 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 3: conflicting requests 2024-04-14T15:01:10.923 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.923 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 4: conflicting requests 2024-04-14T15:01:10.923 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.923 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 5: conflicting requests 2024-04-14T15:01:10.923 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.923 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 6: conflicting requests 2024-04-14T15:01:10.923 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.923 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 7: conflicting requests 2024-04-14T15:01:10.923 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.923 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 8: conflicting requests 2024-04-14T15:01:10.923 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.923 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 9: conflicting requests 2024-04-14T15:01:10.924 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.924 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 10: conflicting requests 2024-04-14T15:01:10.924 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.924 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 11: conflicting requests 2024-04-14T15:01:10.924 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.924 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 12: conflicting requests 2024-04-14T15:01:10.924 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.924 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 13: conflicting requests 2024-04-14T15:01:10.924 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.924 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 14: conflicting requests 2024-04-14T15:01:10.924 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.924 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 15: conflicting requests 2024-04-14T15:01:10.924 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.924 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 16: conflicting requests 2024-04-14T15:01:10.924 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.925 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 17: conflicting requests 2024-04-14T15:01:10.925 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.925 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 18: conflicting requests 2024-04-14T15:01:10.925 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.925 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 19: conflicting requests 2024-04-14T15:01:10.925 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.925 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 20: conflicting requests 2024-04-14T15:01:10.925 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.925 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 21: conflicting requests 2024-04-14T15:01:10.925 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.925 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 22: conflicting requests 2024-04-14T15:01:10.925 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.925 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 23: conflicting requests 2024-04-14T15:01:10.925 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.926 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 24: conflicting requests 2024-04-14T15:01:10.926 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.926 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 25: conflicting requests 2024-04-14T15:01:10.926 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.926 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 26: conflicting requests 2024-04-14T15:01:10.926 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.926 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 27: conflicting requests 2024-04-14T15:01:10.926 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.926 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 28: conflicting requests 2024-04-14T15:01:10.926 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.926 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 29: conflicting requests 2024-04-14T15:01:10.926 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.926 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 30: conflicting requests 2024-04-14T15:01:10.927 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.927 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 31: conflicting requests 2024-04-14T15:01:10.927 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.927 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 32: conflicting requests 2024-04-14T15:01:10.927 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.927 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 33: conflicting requests 2024-04-14T15:01:10.927 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.927 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 34: conflicting requests 2024-04-14T15:01:10.927 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.927 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 35: conflicting requests 2024-04-14T15:01:10.927 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.927 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 36: conflicting requests 2024-04-14T15:01:10.927 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.928 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 37: conflicting requests 2024-04-14T15:01:10.928 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.928 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.928 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.928 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 38: cannot install the best candidate for the job 2024-04-14T15:01:10.928 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.928 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.928 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.928 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.928 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 39: conflicting requests 2024-04-14T15:01:10.928 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.928 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.928 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.928 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.928 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.928 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.929 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 40: cannot install the best candidate for the job 2024-04-14T15:01:10.929 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.929 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.929 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.929 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.929 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 41: cannot install the best candidate for the job 2024-04-14T15:01:10.929 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.929 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.929 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.929 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.929 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 42: cannot install the best candidate for the job 2024-04-14T15:01:10.929 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.929 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.929 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.930 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.930 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 43: cannot install the best candidate for the job 2024-04-14T15:01:10.930 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.930 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.930 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.930 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.930 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 44: cannot install the best candidate for the job 2024-04-14T15:01:10.930 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.930 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.930 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.930 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.930 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 45: cannot install the best candidate for the job 2024-04-14T15:01:10.930 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.931 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.931 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.931 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-14T15:01:10.931 INFO:teuthology.orchestra.run.smithi092.stdout:Package ceph-mgr-2:19.0.0-2778.ga971fd5f.el9.x86_64 is already installed. 2024-04-14T15:01:10.990 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2024-04-14T15:01:10.991 INFO:teuthology.orchestra.run.smithi092.stdout:Nothing to do. 2024-04-14T15:01:10.991 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2024-04-14T15:01:11.062 INFO:teuthology.orchestra.run.smithi090.stdout:(1/12): ceph-prometheus-alerts-19.0.0-2778.ga97 79 kB/s | 16 kB 00:00 2024-04-14T15:01:11.065 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-mgr-dashboard 2024-04-14T15:01:11.195 INFO:teuthology.orchestra.run.smithi090.stdout:(2/12): ceph-grafana-dashboards-19.0.0-2778.ga9 100 kB/s | 34 kB 00:00 2024-04-14T15:01:11.521 INFO:teuthology.orchestra.run.smithi090.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 828 kB/s | 269 kB 00:00 2024-04-14T15:01:11.562 INFO:teuthology.orchestra.run.smithi090.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 2.0 MB/s | 1.0 MB 00:00 2024-04-14T15:01:11.658 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:10 ago on Sun 14 Apr 2024 03:00:01 PM UTC. 2024-04-14T15:01:11.746 INFO:teuthology.orchestra.run.smithi090.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.7 MB/s | 863 kB 00:00 2024-04-14T15:01:11.813 INFO:teuthology.orchestra.run.smithi090.stdout:(6/12): ceph-mgr-dashboard-19.0.0-2778.ga971fd5 3.7 MB/s | 3.5 MB 00:00 2024-04-14T15:01:11.945 INFO:teuthology.orchestra.run.smithi092.stderr:Modular dependency problems: 2024-04-14T15:01:11.946 INFO:teuthology.orchestra.run.smithi092.stderr: 2024-04-14T15:01:11.946 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 1: conflicting requests 2024-04-14T15:01:11.946 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.946 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 2: conflicting requests 2024-04-14T15:01:11.946 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.946 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 3: conflicting requests 2024-04-14T15:01:11.946 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.946 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 4: conflicting requests 2024-04-14T15:01:11.946 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.946 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 5: conflicting requests 2024-04-14T15:01:11.946 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.947 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 6: conflicting requests 2024-04-14T15:01:11.947 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.947 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 7: conflicting requests 2024-04-14T15:01:11.947 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.947 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 8: conflicting requests 2024-04-14T15:01:11.947 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.947 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 9: conflicting requests 2024-04-14T15:01:11.947 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.947 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 10: conflicting requests 2024-04-14T15:01:11.947 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.947 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 11: conflicting requests 2024-04-14T15:01:11.947 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.947 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 12: conflicting requests 2024-04-14T15:01:11.947 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.948 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 13: conflicting requests 2024-04-14T15:01:11.948 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.948 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 14: conflicting requests 2024-04-14T15:01:11.948 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.948 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 15: conflicting requests 2024-04-14T15:01:11.948 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.948 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 16: conflicting requests 2024-04-14T15:01:11.948 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.948 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 17: conflicting requests 2024-04-14T15:01:11.948 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.948 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 18: conflicting requests 2024-04-14T15:01:11.948 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.948 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 19: conflicting requests 2024-04-14T15:01:11.949 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.949 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 20: conflicting requests 2024-04-14T15:01:11.949 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.949 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 21: conflicting requests 2024-04-14T15:01:11.949 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.949 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 22: conflicting requests 2024-04-14T15:01:11.949 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.949 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 23: conflicting requests 2024-04-14T15:01:11.949 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.949 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 24: conflicting requests 2024-04-14T15:01:11.950 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.950 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 25: conflicting requests 2024-04-14T15:01:11.950 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.950 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 26: conflicting requests 2024-04-14T15:01:11.950 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.950 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 27: conflicting requests 2024-04-14T15:01:11.950 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.950 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 28: conflicting requests 2024-04-14T15:01:11.950 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.950 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 29: conflicting requests 2024-04-14T15:01:11.950 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.950 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 30: conflicting requests 2024-04-14T15:01:11.950 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.951 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 31: conflicting requests 2024-04-14T15:01:11.951 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.951 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 32: conflicting requests 2024-04-14T15:01:11.951 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.951 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 33: conflicting requests 2024-04-14T15:01:11.951 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.951 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 34: conflicting requests 2024-04-14T15:01:11.951 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.951 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 35: conflicting requests 2024-04-14T15:01:11.951 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.952 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 36: conflicting requests 2024-04-14T15:01:11.952 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.952 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 37: conflicting requests 2024-04-14T15:01:11.952 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.952 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.952 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.952 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 38: cannot install the best candidate for the job 2024-04-14T15:01:11.952 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.952 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.952 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.952 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.952 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 39: conflicting requests 2024-04-14T15:01:11.953 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.953 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.953 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.953 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.953 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.953 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.953 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 40: cannot install the best candidate for the job 2024-04-14T15:01:11.953 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.953 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.953 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.953 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.954 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 41: cannot install the best candidate for the job 2024-04-14T15:01:11.954 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.954 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.954 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.954 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.954 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 42: cannot install the best candidate for the job 2024-04-14T15:01:11.954 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.954 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.954 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.954 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.954 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 43: cannot install the best candidate for the job 2024-04-14T15:01:11.955 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.955 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.955 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.955 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.955 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 44: cannot install the best candidate for the job 2024-04-14T15:01:11.955 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.955 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.955 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.955 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.955 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 45: cannot install the best candidate for the job 2024-04-14T15:01:11.956 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.956 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.956 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-14T15:01:11.956 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-14T15:01:12.016 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2024-04-14T15:01:12.017 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-04-14T15:01:12.018 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repository Size 2024-04-14T15:01:12.018 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-04-14T15:01:12.018 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2024-04-14T15:01:12.018 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2778.ga971fd5f.el9 ceph-noarch 3.5 M 2024-04-14T15:01:12.018 INFO:teuthology.orchestra.run.smithi092.stdout:Installing dependencies: 2024-04-14T15:01:12.018 INFO:teuthology.orchestra.run.smithi092.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-14T15:01:12.018 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2778.ga971fd5f.el9 ceph-noarch 34 k 2024-04-14T15:01:12.018 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2778.ga971fd5f.el9 ceph-noarch 16 k 2024-04-14T15:01:12.018 INFO:teuthology.orchestra.run.smithi092.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-14T15:01:12.018 INFO:teuthology.orchestra.run.smithi092.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-14T15:01:12.019 INFO:teuthology.orchestra.run.smithi092.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-14T15:01:12.019 INFO:teuthology.orchestra.run.smithi092.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-14T15:01:12.019 INFO:teuthology.orchestra.run.smithi092.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-14T15:01:12.019 INFO:teuthology.orchestra.run.smithi092.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-14T15:01:12.019 INFO:teuthology.orchestra.run.smithi092.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-14T15:01:12.019 INFO:teuthology.orchestra.run.smithi092.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-14T15:01:12.019 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:01:12.019 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2024-04-14T15:01:12.019 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-04-14T15:01:12.019 INFO:teuthology.orchestra.run.smithi092.stdout:Install 12 Packages 2024-04-14T15:01:12.019 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:01:12.020 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 8.6 M 2024-04-14T15:01:12.020 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 104 M 2024-04-14T15:01:12.020 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2024-04-14T15:01:12.072 INFO:teuthology.orchestra.run.smithi090.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.1 MB/s | 548 kB 00:00 2024-04-14T15:01:12.097 INFO:teuthology.orchestra.run.smithi090.stdout:(8/12): grpc-data-1.46.7-10.el9.noarch.rpm 55 kB/s | 19 kB 00:00 2024-04-14T15:01:12.464 INFO:teuthology.orchestra.run.smithi090.stdout:(9/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 3.1 MB/s | 2.0 MB 00:00 2024-04-14T15:01:12.489 INFO:teuthology.orchestra.run.smithi090.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 347 kB/s | 144 kB 00:00 2024-04-14T15:01:12.514 INFO:teuthology.orchestra.run.smithi090.stdout:(11/12): python3-repoze-lru-0.7-16.el9.noarch.r 74 kB/s | 31 kB 00:00 2024-04-14T15:01:12.540 INFO:teuthology.orchestra.run.smithi090.stdout:(12/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.4 MB/s | 188 kB 00:00 2024-04-14T15:01:12.544 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-04-14T15:01:12.544 INFO:teuthology.orchestra.run.smithi090.stdout:Total 3.3 MB/s | 8.6 MB 00:02 2024-04-14T15:01:12.592 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-04-14T15:01:12.609 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-04-14T15:01:12.609 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-04-14T15:01:12.792 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-04-14T15:01:12.793 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-04-14T15:01:13.016 INFO:teuthology.orchestra.run.smithi092.stdout:(1/12): ceph-prometheus-alerts-19.0.0-2778.ga97 85 kB/s | 16 kB 00:00 2024-04-14T15:01:13.066 INFO:teuthology.orchestra.run.smithi092.stdout:(2/12): ceph-grafana-dashboards-19.0.0-2778.ga9 140 kB/s | 34 kB 00:00 2024-04-14T15:01:13.300 INFO:teuthology.orchestra.run.smithi092.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.1 MB/s | 269 kB 00:00 2024-04-14T15:01:13.321 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-04-14T15:01:13.350 INFO:teuthology.orchestra.run.smithi092.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.0 MB/s | 1.0 MB 00:00 2024-04-14T15:01:13.399 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-04-14T15:01:13.425 INFO:teuthology.orchestra.run.smithi092.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 6.7 MB/s | 863 kB 00:00 2024-04-14T15:01:13.443 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-04-14T15:01:13.489 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-04-14T15:01:13.520 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-04-14T15:01:13.628 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-04-14T15:01:13.659 INFO:teuthology.orchestra.run.smithi092.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 83 kB/s | 19 kB 00:00 2024-04-14T15:01:13.754 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-04-14T15:01:13.851 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-04-14T15:01:13.868 INFO:teuthology.orchestra.run.smithi092.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.0 MB/s | 548 kB 00:00 2024-04-14T15:01:13.901 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-04-14T15:01:13.931 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-04-14T15:01:13.963 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2778.ga971fd5f.el9 10/12 2024-04-14T15:01:13.968 INFO:teuthology.orchestra.run.smithi092.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.4 MB/s | 144 kB 00:00 2024-04-14T15:01:14.068 INFO:teuthology.orchestra.run.smithi092.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 308 kB/s | 31 kB 00:00 2024-04-14T15:01:14.160 INFO:teuthology.orchestra.run.smithi092.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.0 MB/s | 188 kB 00:00 2024-04-14T15:01:14.210 INFO:teuthology.orchestra.run.smithi092.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 3.7 MB/s | 2.0 MB 00:00 2024-04-14T15:01:14.277 INFO:teuthology.orchestra.run.smithi092.stdout:(12/12): ceph-mgr-dashboard-19.0.0-2778.ga971fd 2.4 MB/s | 3.5 MB 00:01 2024-04-14T15:01:14.279 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2024-04-14T15:01:14.279 INFO:teuthology.orchestra.run.smithi092.stdout:Total 3.8 MB/s | 8.6 MB 00:02 2024-04-14T15:01:14.327 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2024-04-14T15:01:14.344 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2024-04-14T15:01:14.344 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2024-04-14T15:01:14.545 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2024-04-14T15:01:14.546 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2024-04-14T15:01:15.020 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2778.ga971fd5f.el 11/12 2024-04-14T15:01:15.067 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2024-04-14T15:01:15.098 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2778.ga971fd5f.el9.noa 12/12 2024-04-14T15:01:15.186 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-04-14T15:01:15.230 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-04-14T15:01:15.276 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-04-14T15:01:15.315 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-04-14T15:01:15.419 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-04-14T15:01:15.541 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-04-14T15:01:15.642 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-04-14T15:01:15.686 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-04-14T15:01:15.714 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-04-14T15:01:15.747 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2778.ga971fd5f.el9 10/12 2024-04-14T15:01:16.797 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2778.ga971fd5f.el 11/12 2024-04-14T15:01:16.839 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2778.ga971fd5f.el9.noa 12/12 2024-04-14T15:01:17.147 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2778.ga971fd5f.el9.noa 12/12 2024-04-14T15:01:17.147 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2778.ga971fd5f.el 1/12 2024-04-14T15:01:17.148 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2778.ga971fd5f.el9.noa 2/12 2024-04-14T15:01:17.148 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2778.ga971fd5f.el9 3/12 2024-04-14T15:01:17.148 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-04-14T15:01:17.148 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-04-14T15:01:17.148 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-04-14T15:01:17.148 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-04-14T15:01:17.148 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-04-14T15:01:17.148 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-04-14T15:01:17.148 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-04-14T15:01:17.148 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-04-14T15:01:17.506 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-04-14T15:01:17.506 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:01:17.507 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-04-14T15:01:17.507 INFO:teuthology.orchestra.run.smithi090.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-14T15:01:17.507 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-grafana-dashboards-2:19.0.0-2778.ga971fd5f.el9.noarch 2024-04-14T15:01:17.507 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-dashboard-2:19.0.0-2778.ga971fd5f.el9.noarch 2024-04-14T15:01:17.507 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-prometheus-alerts-2:19.0.0-2778.ga971fd5f.el9.noarch 2024-04-14T15:01:17.507 INFO:teuthology.orchestra.run.smithi090.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-14T15:01:17.507 INFO:teuthology.orchestra.run.smithi090.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-14T15:01:17.507 INFO:teuthology.orchestra.run.smithi090.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-14T15:01:17.507 INFO:teuthology.orchestra.run.smithi090.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-14T15:01:17.507 INFO:teuthology.orchestra.run.smithi090.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-14T15:01:17.507 INFO:teuthology.orchestra.run.smithi090.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-14T15:01:17.507 INFO:teuthology.orchestra.run.smithi090.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-14T15:01:17.507 INFO:teuthology.orchestra.run.smithi090.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-14T15:01:17.507 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:01:17.508 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-04-14T15:01:17.796 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-14T15:01:18.196 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2778.ga971fd5f.el9.noa 12/12 2024-04-14T15:01:18.197 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2778.ga971fd5f.el 1/12 2024-04-14T15:01:18.197 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2778.ga971fd5f.el9.noa 2/12 2024-04-14T15:01:18.197 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2778.ga971fd5f.el9 3/12 2024-04-14T15:01:18.197 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-04-14T15:01:18.197 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-04-14T15:01:18.197 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-04-14T15:01:18.197 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-04-14T15:01:18.197 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-04-14T15:01:18.197 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-04-14T15:01:18.197 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-04-14T15:01:18.197 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-04-14T15:01:18.381 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:20 ago on Sun 14 Apr 2024 02:59:58 PM UTC. 2024-04-14T15:01:18.576 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-04-14T15:01:18.576 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:01:18.576 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2024-04-14T15:01:18.577 INFO:teuthology.orchestra.run.smithi092.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-14T15:01:18.577 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-grafana-dashboards-2:19.0.0-2778.ga971fd5f.el9.noarch 2024-04-14T15:01:18.577 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-dashboard-2:19.0.0-2778.ga971fd5f.el9.noarch 2024-04-14T15:01:18.577 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-prometheus-alerts-2:19.0.0-2778.ga971fd5f.el9.noarch 2024-04-14T15:01:18.577 INFO:teuthology.orchestra.run.smithi092.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-14T15:01:18.577 INFO:teuthology.orchestra.run.smithi092.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-14T15:01:18.577 INFO:teuthology.orchestra.run.smithi092.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-14T15:01:18.577 INFO:teuthology.orchestra.run.smithi092.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-14T15:01:18.577 INFO:teuthology.orchestra.run.smithi092.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-14T15:01:18.577 INFO:teuthology.orchestra.run.smithi092.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-14T15:01:18.577 INFO:teuthology.orchestra.run.smithi092.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-14T15:01:18.577 INFO:teuthology.orchestra.run.smithi092.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-14T15:01:18.577 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:01:18.578 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2024-04-14T15:01:18.667 INFO:teuthology.orchestra.run.smithi090.stderr:Modular dependency problems: 2024-04-14T15:01:18.667 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-04-14T15:01:18.668 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 1: conflicting requests 2024-04-14T15:01:18.668 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.668 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 2: conflicting requests 2024-04-14T15:01:18.668 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.668 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 3: conflicting requests 2024-04-14T15:01:18.668 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.668 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 4: conflicting requests 2024-04-14T15:01:18.668 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.668 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 5: conflicting requests 2024-04-14T15:01:18.668 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.668 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 6: conflicting requests 2024-04-14T15:01:18.668 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.668 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 7: conflicting requests 2024-04-14T15:01:18.669 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.669 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 8: conflicting requests 2024-04-14T15:01:18.669 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.669 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 9: conflicting requests 2024-04-14T15:01:18.669 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.669 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 10: conflicting requests 2024-04-14T15:01:18.669 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.669 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 11: conflicting requests 2024-04-14T15:01:18.669 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.669 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 12: conflicting requests 2024-04-14T15:01:18.669 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.669 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 13: conflicting requests 2024-04-14T15:01:18.669 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.669 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 14: conflicting requests 2024-04-14T15:01:18.670 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.670 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 15: conflicting requests 2024-04-14T15:01:18.670 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.670 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 16: conflicting requests 2024-04-14T15:01:18.670 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.670 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 17: conflicting requests 2024-04-14T15:01:18.670 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.670 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 18: conflicting requests 2024-04-14T15:01:18.670 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.670 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 19: conflicting requests 2024-04-14T15:01:18.670 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.670 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 20: conflicting requests 2024-04-14T15:01:18.670 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.670 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 21: conflicting requests 2024-04-14T15:01:18.671 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.671 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 22: conflicting requests 2024-04-14T15:01:18.671 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.671 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 23: conflicting requests 2024-04-14T15:01:18.671 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.671 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 24: conflicting requests 2024-04-14T15:01:18.671 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.671 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 25: conflicting requests 2024-04-14T15:01:18.671 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.671 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 26: conflicting requests 2024-04-14T15:01:18.671 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.671 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 27: conflicting requests 2024-04-14T15:01:18.671 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.671 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 28: conflicting requests 2024-04-14T15:01:18.672 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.672 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 29: conflicting requests 2024-04-14T15:01:18.672 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.672 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 30: conflicting requests 2024-04-14T15:01:18.672 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.672 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 31: conflicting requests 2024-04-14T15:01:18.672 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.672 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 32: conflicting requests 2024-04-14T15:01:18.672 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.672 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 33: conflicting requests 2024-04-14T15:01:18.672 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.672 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 34: conflicting requests 2024-04-14T15:01:18.672 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.672 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 35: conflicting requests 2024-04-14T15:01:18.672 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.673 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 36: conflicting requests 2024-04-14T15:01:18.673 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.673 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 37: conflicting requests 2024-04-14T15:01:18.673 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.673 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.673 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.673 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 38: cannot install the best candidate for the job 2024-04-14T15:01:18.673 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.673 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.673 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.673 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.673 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 39: conflicting requests 2024-04-14T15:01:18.673 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.673 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.673 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.674 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.674 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.674 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.674 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 40: cannot install the best candidate for the job 2024-04-14T15:01:18.674 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.674 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.674 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.674 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.674 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 41: cannot install the best candidate for the job 2024-04-14T15:01:18.674 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.674 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.674 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.674 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.674 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 42: cannot install the best candidate for the job 2024-04-14T15:01:18.675 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.675 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.675 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.675 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.675 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 43: cannot install the best candidate for the job 2024-04-14T15:01:18.675 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.675 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.675 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.675 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.675 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 44: cannot install the best candidate for the job 2024-04-14T15:01:18.675 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.675 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.675 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.675 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.676 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 45: cannot install the best candidate for the job 2024-04-14T15:01:18.676 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.676 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.676 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.676 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-14T15:01:18.736 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-04-14T15:01:18.737 INFO:teuthology.orchestra.run.smithi090.stdout:======================================================================================= 2024-04-14T15:01:18.737 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-04-14T15:01:18.737 INFO:teuthology.orchestra.run.smithi090.stdout:======================================================================================= 2024-04-14T15:01:18.737 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-04-14T15:01:18.738 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2778.ga971fd5f.el9 ceph-noarch 7.4 M 2024-04-14T15:01:18.738 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-04-14T15:01:18.738 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-14T15:01:18.738 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-14T15:01:18.738 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-14T15:01:18.738 INFO:teuthology.orchestra.run.smithi090.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-14T15:01:18.738 INFO:teuthology.orchestra.run.smithi090.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-14T15:01:18.738 INFO:teuthology.orchestra.run.smithi090.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-14T15:01:18.738 INFO:teuthology.orchestra.run.smithi090.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-14T15:01:18.738 INFO:teuthology.orchestra.run.smithi090.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-14T15:01:18.738 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-14T15:01:18.738 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-14T15:01:18.738 INFO:teuthology.orchestra.run.smithi090.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-14T15:01:18.738 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:01:18.739 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-04-14T15:01:18.739 INFO:teuthology.orchestra.run.smithi090.stdout:======================================================================================= 2024-04-14T15:01:18.739 INFO:teuthology.orchestra.run.smithi090.stdout:Install 12 Packages 2024-04-14T15:01:18.739 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:01:18.739 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 38 M 2024-04-14T15:01:18.739 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 209 M 2024-04-14T15:01:18.739 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-04-14T15:01:18.842 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-14T15:01:19.122 INFO:teuthology.orchestra.run.smithi090.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.2 MB/s | 195 kB 00:00 2024-04-14T15:01:19.190 INFO:teuthology.orchestra.run.smithi090.stdout:(2/12): libgfortran-11.4.1-3.el9.x86_64.rpm 3.4 MB/s | 803 kB 00:00 2024-04-14T15:01:19.434 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:18 ago on Sun 14 Apr 2024 03:00:01 PM UTC. 2024-04-14T15:01:19.722 INFO:teuthology.orchestra.run.smithi092.stderr:Modular dependency problems: 2024-04-14T15:01:19.722 INFO:teuthology.orchestra.run.smithi092.stderr: 2024-04-14T15:01:19.722 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 1: conflicting requests 2024-04-14T15:01:19.722 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.722 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 2: conflicting requests 2024-04-14T15:01:19.722 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.722 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 3: conflicting requests 2024-04-14T15:01:19.723 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.723 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 4: conflicting requests 2024-04-14T15:01:19.723 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.723 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 5: conflicting requests 2024-04-14T15:01:19.723 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.723 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 6: conflicting requests 2024-04-14T15:01:19.723 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.723 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 7: conflicting requests 2024-04-14T15:01:19.723 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.723 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 8: conflicting requests 2024-04-14T15:01:19.723 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.723 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 9: conflicting requests 2024-04-14T15:01:19.723 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.723 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 10: conflicting requests 2024-04-14T15:01:19.723 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.724 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 11: conflicting requests 2024-04-14T15:01:19.724 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.724 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 12: conflicting requests 2024-04-14T15:01:19.724 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.724 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 13: conflicting requests 2024-04-14T15:01:19.724 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.724 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 14: conflicting requests 2024-04-14T15:01:19.724 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.724 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 15: conflicting requests 2024-04-14T15:01:19.724 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.724 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 16: conflicting requests 2024-04-14T15:01:19.724 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.724 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 17: conflicting requests 2024-04-14T15:01:19.724 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.724 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 18: conflicting requests 2024-04-14T15:01:19.725 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.725 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 19: conflicting requests 2024-04-14T15:01:19.725 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.725 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 20: conflicting requests 2024-04-14T15:01:19.725 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.725 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 21: conflicting requests 2024-04-14T15:01:19.725 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.725 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 22: conflicting requests 2024-04-14T15:01:19.725 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.725 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 23: conflicting requests 2024-04-14T15:01:19.725 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.725 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 24: conflicting requests 2024-04-14T15:01:19.725 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.725 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 25: conflicting requests 2024-04-14T15:01:19.725 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.726 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 26: conflicting requests 2024-04-14T15:01:19.726 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.726 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 27: conflicting requests 2024-04-14T15:01:19.726 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.726 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 28: conflicting requests 2024-04-14T15:01:19.726 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.726 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 29: conflicting requests 2024-04-14T15:01:19.726 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.726 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 30: conflicting requests 2024-04-14T15:01:19.726 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.726 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 31: conflicting requests 2024-04-14T15:01:19.726 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.726 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 32: conflicting requests 2024-04-14T15:01:19.726 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.726 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 33: conflicting requests 2024-04-14T15:01:19.727 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.727 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 34: conflicting requests 2024-04-14T15:01:19.727 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.727 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 35: conflicting requests 2024-04-14T15:01:19.727 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.727 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 36: conflicting requests 2024-04-14T15:01:19.727 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.727 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 37: conflicting requests 2024-04-14T15:01:19.727 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.727 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.727 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.727 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 38: cannot install the best candidate for the job 2024-04-14T15:01:19.727 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.727 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.727 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.727 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.728 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 39: conflicting requests 2024-04-14T15:01:19.728 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.728 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.728 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.728 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.728 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.728 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.728 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 40: cannot install the best candidate for the job 2024-04-14T15:01:19.728 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.728 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.728 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.728 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.728 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 41: cannot install the best candidate for the job 2024-04-14T15:01:19.728 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.728 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.729 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.729 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.729 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 42: cannot install the best candidate for the job 2024-04-14T15:01:19.729 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.729 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.729 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.729 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.729 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 43: cannot install the best candidate for the job 2024-04-14T15:01:19.729 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.729 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.729 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.729 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.729 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 44: cannot install the best candidate for the job 2024-04-14T15:01:19.729 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.729 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.729 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.730 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.730 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 45: cannot install the best candidate for the job 2024-04-14T15:01:19.730 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.730 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.730 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.730 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-14T15:01:19.732 INFO:teuthology.orchestra.run.smithi090.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 54 kB/s | 33 kB 00:00 2024-04-14T15:01:19.790 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2024-04-14T15:01:19.791 INFO:teuthology.orchestra.run.smithi092.stdout:======================================================================================= 2024-04-14T15:01:19.792 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repository Size 2024-04-14T15:01:19.792 INFO:teuthology.orchestra.run.smithi092.stdout:======================================================================================= 2024-04-14T15:01:19.792 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2024-04-14T15:01:19.792 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2778.ga971fd5f.el9 ceph-noarch 7.4 M 2024-04-14T15:01:19.792 INFO:teuthology.orchestra.run.smithi092.stdout:Installing dependencies: 2024-04-14T15:01:19.792 INFO:teuthology.orchestra.run.smithi092.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-14T15:01:19.792 INFO:teuthology.orchestra.run.smithi092.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-14T15:01:19.792 INFO:teuthology.orchestra.run.smithi092.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-14T15:01:19.792 INFO:teuthology.orchestra.run.smithi092.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-14T15:01:19.792 INFO:teuthology.orchestra.run.smithi092.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-14T15:01:19.792 INFO:teuthology.orchestra.run.smithi092.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-14T15:01:19.792 INFO:teuthology.orchestra.run.smithi092.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-14T15:01:19.792 INFO:teuthology.orchestra.run.smithi092.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-14T15:01:19.793 INFO:teuthology.orchestra.run.smithi092.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-14T15:01:19.793 INFO:teuthology.orchestra.run.smithi092.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-14T15:01:19.793 INFO:teuthology.orchestra.run.smithi092.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-14T15:01:19.793 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:01:19.793 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2024-04-14T15:01:19.793 INFO:teuthology.orchestra.run.smithi092.stdout:======================================================================================= 2024-04-14T15:01:19.793 INFO:teuthology.orchestra.run.smithi092.stdout:Install 12 Packages 2024-04-14T15:01:19.793 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:01:19.793 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 38 M 2024-04-14T15:01:19.794 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 209 M 2024-04-14T15:01:19.794 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2024-04-14T15:01:19.899 INFO:teuthology.orchestra.run.smithi090.stdout:(4/12): ceph-mgr-diskprediction-local-19.0.0-27 7.9 MB/s | 7.4 MB 00:00 2024-04-14T15:01:20.227 INFO:teuthology.orchestra.run.smithi092.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.4 MB/s | 195 kB 00:00 2024-04-14T15:01:20.268 INFO:teuthology.orchestra.run.smithi092.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 789 kB/s | 33 kB 00:00 2024-04-14T15:01:20.319 INFO:teuthology.orchestra.run.smithi092.stdout:(3/12): libgfortran-11.4.1-3.el9.x86_64.rpm 3.4 MB/s | 803 kB 00:00 2024-04-14T15:01:20.361 INFO:teuthology.orchestra.run.smithi092.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 436 kB/s | 18 kB 00:00 2024-04-14T15:01:20.374 INFO:teuthology.orchestra.run.smithi090.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 28 kB/s | 18 kB 00:00 2024-04-14T15:01:20.403 INFO:teuthology.orchestra.run.smithi092.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 850 kB/s | 35 kB 00:00 2024-04-14T15:01:20.524 INFO:teuthology.orchestra.run.smithi090.stdout:(6/12): openblas-0.3.21-2.el9.x86_64.rpm 56 kB/s | 35 kB 00:00 2024-04-14T15:01:20.587 INFO:teuthology.orchestra.run.smithi092.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 9.4 MB/s | 3.0 MB 00:00 2024-04-14T15:01:20.670 INFO:teuthology.orchestra.run.smithi092.stdout:(7/12): python3-devel-3.9.17-2.el9.x86_64.rpm 2.9 MB/s | 245 kB 00:00 2024-04-14T15:01:20.804 INFO:teuthology.orchestra.run.smithi092.stdout:(8/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 12 MB/s | 4.7 MB 00:00 2024-04-14T15:01:20.937 INFO:teuthology.orchestra.run.smithi092.stdout:(9/12): ceph-mgr-diskprediction-local-19.0.0-27 8.7 MB/s | 7.4 MB 00:00 2024-04-14T15:01:20.962 INFO:teuthology.orchestra.run.smithi092.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 1.5 MB/s | 250 kB 00:00 2024-04-14T15:01:21.129 INFO:teuthology.orchestra.run.smithi092.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 11 MB/s | 5.1 MB 00:00 2024-04-14T15:01:21.484 INFO:teuthology.orchestra.run.smithi090.stdout:(7/12): python3-devel-3.9.17-2.el9.x86_64.rpm 256 kB/s | 245 kB 00:00 2024-04-14T15:01:21.746 INFO:teuthology.orchestra.run.smithi092.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 20 MB/s | 16 MB 00:00 2024-04-14T15:01:21.747 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2024-04-14T15:01:21.747 INFO:teuthology.orchestra.run.smithi092.stdout:Total 19 MB/s | 38 MB 00:01 2024-04-14T15:01:22.100 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2024-04-14T15:01:22.127 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2024-04-14T15:01:22.127 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2024-04-14T15:01:22.562 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2024-04-14T15:01:22.562 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2024-04-14T15:01:23.339 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2024-04-14T15:01:23.417 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-04-14T15:01:23.453 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-04-14T15:01:23.559 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-04-14T15:01:23.596 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-04-14T15:01:23.700 INFO:teuthology.orchestra.run.smithi090.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 679 kB/s | 3.0 MB 00:04 2024-04-14T15:01:23.774 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-14T15:01:23.811 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-14T15:01:23.888 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-14T15:01:24.583 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-14T15:01:24.684 INFO:teuthology.orchestra.run.smithi090.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 255 kB/s | 250 kB 00:00 2024-04-14T15:01:24.710 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-14T15:01:26.040 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-14T15:01:27.079 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-14T15:01:27.118 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2778.ga971f 12/12 2024-04-14T15:01:27.484 INFO:teuthology.orchestra.run.smithi090.stdout:(10/12): openblas-openmp-0.3.21-2.el9.x86_64.rp 682 kB/s | 4.7 MB 00:07 2024-04-14T15:01:29.018 INFO:teuthology.orchestra.run.smithi090.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 693 kB/s | 5.1 MB 00:07 2024-04-14T15:01:29.579 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2778.ga971f 12/12 2024-04-14T15:01:29.579 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2778.ga971f 1/12 2024-04-14T15:01:29.579 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-14T15:01:29.579 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-04-14T15:01:29.580 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-04-14T15:01:29.580 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-04-14T15:01:29.580 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-04-14T15:01:29.580 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-04-14T15:01:29.580 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-04-14T15:01:29.580 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 9/12 2024-04-14T15:01:29.580 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-04-14T15:01:29.580 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-04-14T15:01:30.082 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-04-14T15:01:30.082 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:01:30.082 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2024-04-14T15:01:30.082 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2778.ga971fd5f.el9.noarch 2024-04-14T15:01:30.082 INFO:teuthology.orchestra.run.smithi092.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-14T15:01:30.082 INFO:teuthology.orchestra.run.smithi092.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-14T15:01:30.083 INFO:teuthology.orchestra.run.smithi092.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-14T15:01:30.083 INFO:teuthology.orchestra.run.smithi092.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-14T15:01:30.083 INFO:teuthology.orchestra.run.smithi092.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-14T15:01:30.083 INFO:teuthology.orchestra.run.smithi092.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-14T15:01:30.083 INFO:teuthology.orchestra.run.smithi092.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-14T15:01:30.083 INFO:teuthology.orchestra.run.smithi092.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-14T15:01:30.083 INFO:teuthology.orchestra.run.smithi092.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-14T15:01:30.083 INFO:teuthology.orchestra.run.smithi092.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-14T15:01:30.083 INFO:teuthology.orchestra.run.smithi092.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-14T15:01:30.083 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:01:30.083 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2024-04-14T15:01:30.481 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-mgr-rook 2024-04-14T15:01:31.088 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:30 ago on Sun 14 Apr 2024 03:00:01 PM UTC. 2024-04-14T15:01:31.376 INFO:teuthology.orchestra.run.smithi092.stderr:Modular dependency problems: 2024-04-14T15:01:31.376 INFO:teuthology.orchestra.run.smithi092.stderr: 2024-04-14T15:01:31.376 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 1: conflicting requests 2024-04-14T15:01:31.376 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.376 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 2: conflicting requests 2024-04-14T15:01:31.377 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.377 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 3: conflicting requests 2024-04-14T15:01:31.377 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.377 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 4: conflicting requests 2024-04-14T15:01:31.377 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.377 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 5: conflicting requests 2024-04-14T15:01:31.377 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.377 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 6: conflicting requests 2024-04-14T15:01:31.377 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.377 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 7: conflicting requests 2024-04-14T15:01:31.377 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.377 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 8: conflicting requests 2024-04-14T15:01:31.377 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.377 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 9: conflicting requests 2024-04-14T15:01:31.377 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.378 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 10: conflicting requests 2024-04-14T15:01:31.378 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.378 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 11: conflicting requests 2024-04-14T15:01:31.378 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.378 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 12: conflicting requests 2024-04-14T15:01:31.378 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.378 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 13: conflicting requests 2024-04-14T15:01:31.378 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.378 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 14: conflicting requests 2024-04-14T15:01:31.378 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.378 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 15: conflicting requests 2024-04-14T15:01:31.378 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.378 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 16: conflicting requests 2024-04-14T15:01:31.378 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.379 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 17: conflicting requests 2024-04-14T15:01:31.379 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.379 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 18: conflicting requests 2024-04-14T15:01:31.379 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.379 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 19: conflicting requests 2024-04-14T15:01:31.379 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.379 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 20: conflicting requests 2024-04-14T15:01:31.379 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.379 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 21: conflicting requests 2024-04-14T15:01:31.379 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.379 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 22: conflicting requests 2024-04-14T15:01:31.379 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.379 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 23: conflicting requests 2024-04-14T15:01:31.379 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.379 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 24: conflicting requests 2024-04-14T15:01:31.380 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.380 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 25: conflicting requests 2024-04-14T15:01:31.380 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.380 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 26: conflicting requests 2024-04-14T15:01:31.380 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.380 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 27: conflicting requests 2024-04-14T15:01:31.380 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.380 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 28: conflicting requests 2024-04-14T15:01:31.380 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.380 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 29: conflicting requests 2024-04-14T15:01:31.380 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.380 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 30: conflicting requests 2024-04-14T15:01:31.380 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.380 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 31: conflicting requests 2024-04-14T15:01:31.380 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.381 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 32: conflicting requests 2024-04-14T15:01:31.381 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.381 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 33: conflicting requests 2024-04-14T15:01:31.381 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.381 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 34: conflicting requests 2024-04-14T15:01:31.381 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.381 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 35: conflicting requests 2024-04-14T15:01:31.381 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.381 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 36: conflicting requests 2024-04-14T15:01:31.381 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.381 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 37: conflicting requests 2024-04-14T15:01:31.381 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.381 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.381 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.381 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 38: cannot install the best candidate for the job 2024-04-14T15:01:31.381 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.382 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.382 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.382 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.382 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 39: conflicting requests 2024-04-14T15:01:31.382 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.382 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.382 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.382 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.382 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.382 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.382 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 40: cannot install the best candidate for the job 2024-04-14T15:01:31.382 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.382 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.382 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.383 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.383 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 41: cannot install the best candidate for the job 2024-04-14T15:01:31.383 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.383 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.383 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.383 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.383 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 42: cannot install the best candidate for the job 2024-04-14T15:01:31.383 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.383 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.383 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.383 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.383 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 43: cannot install the best candidate for the job 2024-04-14T15:01:31.383 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.383 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.383 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.383 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.384 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 44: cannot install the best candidate for the job 2024-04-14T15:01:31.384 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.384 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.384 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.384 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.384 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 45: cannot install the best candidate for the job 2024-04-14T15:01:31.384 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.384 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.384 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.384 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-14T15:01:31.445 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2024-04-14T15:01:31.446 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-04-14T15:01:31.446 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repository Size 2024-04-14T15:01:31.446 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-04-14T15:01:31.446 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2024-04-14T15:01:31.446 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-rook noarch 2:19.0.0-2778.ga971fd5f.el9 ceph-noarch 49 k 2024-04-14T15:01:31.446 INFO:teuthology.orchestra.run.smithi092.stdout:Installing dependencies: 2024-04-14T15:01:31.447 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-14T15:01:31.447 INFO:teuthology.orchestra.run.smithi092.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-14T15:01:31.447 INFO:teuthology.orchestra.run.smithi092.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-14T15:01:31.447 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-14T15:01:31.447 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-14T15:01:31.447 INFO:teuthology.orchestra.run.smithi092.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-14T15:01:31.447 INFO:teuthology.orchestra.run.smithi092.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-14T15:01:31.447 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-14T15:01:31.447 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-14T15:01:31.447 INFO:teuthology.orchestra.run.smithi092.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-14T15:01:31.447 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-14T15:01:31.447 INFO:teuthology.orchestra.run.smithi092.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-14T15:01:31.448 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:01:31.448 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2024-04-14T15:01:31.448 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-04-14T15:01:31.448 INFO:teuthology.orchestra.run.smithi092.stdout:Install 13 Packages 2024-04-14T15:01:31.448 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:01:31.448 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 2.2 M 2024-04-14T15:01:31.448 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 26 M 2024-04-14T15:01:31.448 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2024-04-14T15:01:32.100 INFO:teuthology.orchestra.run.smithi092.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 91 kB/s | 19 kB 00:00 2024-04-14T15:01:32.125 INFO:teuthology.orchestra.run.smithi092.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 108 kB/s | 26 kB 00:00 2024-04-14T15:01:32.317 INFO:teuthology.orchestra.run.smithi092.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.0 MB/s | 222 kB 00:00 2024-04-14T15:01:32.342 INFO:teuthology.orchestra.run.smithi092.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 732 kB/s | 159 kB 00:00 2024-04-14T15:01:32.426 INFO:teuthology.orchestra.run.smithi092.stdout:(5/13): python3-requests-oauthlib-1.3.0-12.el9. 643 kB/s | 54 kB 00:00 2024-04-14T15:01:32.468 INFO:teuthology.orchestra.run.smithi092.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.8 MB/s | 279 kB 00:00 2024-04-14T15:01:32.760 INFO:teuthology.orchestra.run.smithi092.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 51 kB/s | 15 kB 00:00 2024-04-14T15:01:32.785 INFO:teuthology.orchestra.run.smithi092.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 90 kB/s | 32 kB 00:00 2024-04-14T15:01:33.069 INFO:teuthology.orchestra.run.smithi092.stdout:(9/13): python3-google-auth-2.28.2-1.el9.noarch 657 kB/s | 203 kB 00:00 2024-04-14T15:01:33.194 INFO:teuthology.orchestra.run.smithi092.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 473 kB/s | 59 kB 00:00 2024-04-14T15:01:33.253 INFO:teuthology.orchestra.run.smithi092.stdout:(11/13): python3-kubernetes-26.1.0-2.el9.noarch 2.2 MB/s | 1.0 MB 00:00 2024-04-14T15:01:33.352 INFO:teuthology.orchestra.run.smithi092.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 566 kB/s | 90 kB 00:00 2024-04-14T15:01:35.154 INFO:teuthology.orchestra.run.smithi092.stdout:(13/13): ceph-mgr-rook-19.0.0-2778.ga971fd5f.el 15 kB/s | 49 kB 00:03 2024-04-14T15:01:35.155 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2024-04-14T15:01:35.155 INFO:teuthology.orchestra.run.smithi092.stdout:Total 607 kB/s | 2.2 MB 00:03 2024-04-14T15:01:35.224 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2024-04-14T15:01:35.240 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2024-04-14T15:01:35.240 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2024-04-14T15:01:35.334 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2024-04-14T15:01:35.335 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2024-04-14T15:01:35.845 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2024-04-14T15:01:36.034 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-14T15:01:36.086 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-14T15:01:36.142 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-14T15:01:36.179 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-14T15:01:36.220 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-14T15:01:36.315 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-14T15:01:36.415 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-04-14T15:01:36.472 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-14T15:01:37.192 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-14T15:01:37.244 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-14T15:01:37.278 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-14T15:01:37.325 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-14T15:01:37.347 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-rook-2:19.0.0-2778.ga971fd5f.el9.noarch 13/13 2024-04-14T15:01:38.154 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2778.ga971fd5f.el9.noarch 13/13 2024-04-14T15:01:38.154 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2778.ga971fd5f.el9.noarch 1/13 2024-04-14T15:01:38.154 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-14T15:01:38.154 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-14T15:01:38.154 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-14T15:01:38.154 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-14T15:01:38.154 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-14T15:01:38.155 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-14T15:01:38.155 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-14T15:01:38.155 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-14T15:01:38.155 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-04-14T15:01:38.155 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-14T15:01:38.155 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-14T15:01:38.582 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-14T15:01:38.583 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:01:38.583 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2024-04-14T15:01:38.583 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-rook-2:19.0.0-2778.ga971fd5f.el9.noarch 2024-04-14T15:01:38.583 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-14T15:01:38.583 INFO:teuthology.orchestra.run.smithi092.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-14T15:01:38.583 INFO:teuthology.orchestra.run.smithi092.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-14T15:01:38.583 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-14T15:01:38.583 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-14T15:01:38.583 INFO:teuthology.orchestra.run.smithi092.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-14T15:01:38.583 INFO:teuthology.orchestra.run.smithi092.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-14T15:01:38.583 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-14T15:01:38.584 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-14T15:01:38.584 INFO:teuthology.orchestra.run.smithi092.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-14T15:01:38.584 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-14T15:01:38.584 INFO:teuthology.orchestra.run.smithi092.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-14T15:01:38.584 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:01:38.584 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2024-04-14T15:01:38.853 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-mgr-cephadm 2024-04-14T15:01:39.448 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:38 ago on Sun 14 Apr 2024 03:00:01 PM UTC. 2024-04-14T15:01:39.743 INFO:teuthology.orchestra.run.smithi092.stderr:Modular dependency problems: 2024-04-14T15:01:39.743 INFO:teuthology.orchestra.run.smithi092.stderr: 2024-04-14T15:01:39.743 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 1: conflicting requests 2024-04-14T15:01:39.743 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.743 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 2: conflicting requests 2024-04-14T15:01:39.744 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.744 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 3: conflicting requests 2024-04-14T15:01:39.744 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.744 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 4: conflicting requests 2024-04-14T15:01:39.744 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.744 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 5: conflicting requests 2024-04-14T15:01:39.744 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.744 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 6: conflicting requests 2024-04-14T15:01:39.744 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.744 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 7: conflicting requests 2024-04-14T15:01:39.744 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.744 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 8: conflicting requests 2024-04-14T15:01:39.744 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.744 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 9: conflicting requests 2024-04-14T15:01:39.745 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.745 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 10: conflicting requests 2024-04-14T15:01:39.745 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.745 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 11: conflicting requests 2024-04-14T15:01:39.745 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.745 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 12: conflicting requests 2024-04-14T15:01:39.745 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.745 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 13: conflicting requests 2024-04-14T15:01:39.745 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.745 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 14: conflicting requests 2024-04-14T15:01:39.745 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.745 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 15: conflicting requests 2024-04-14T15:01:39.745 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.746 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 16: conflicting requests 2024-04-14T15:01:39.746 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.746 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 17: conflicting requests 2024-04-14T15:01:39.746 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.746 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 18: conflicting requests 2024-04-14T15:01:39.746 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.746 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 19: conflicting requests 2024-04-14T15:01:39.746 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.746 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 20: conflicting requests 2024-04-14T15:01:39.746 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.746 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 21: conflicting requests 2024-04-14T15:01:39.746 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.746 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 22: conflicting requests 2024-04-14T15:01:39.746 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.747 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 23: conflicting requests 2024-04-14T15:01:39.747 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.747 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 24: conflicting requests 2024-04-14T15:01:39.747 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.747 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 25: conflicting requests 2024-04-14T15:01:39.747 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.747 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 26: conflicting requests 2024-04-14T15:01:39.747 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.747 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 27: conflicting requests 2024-04-14T15:01:39.747 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.747 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 28: conflicting requests 2024-04-14T15:01:39.747 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.747 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 29: conflicting requests 2024-04-14T15:01:39.747 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.748 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 30: conflicting requests 2024-04-14T15:01:39.748 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.748 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 31: conflicting requests 2024-04-14T15:01:39.748 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.748 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 32: conflicting requests 2024-04-14T15:01:39.748 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.748 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 33: conflicting requests 2024-04-14T15:01:39.748 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.748 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 34: conflicting requests 2024-04-14T15:01:39.748 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.748 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 35: conflicting requests 2024-04-14T15:01:39.748 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.748 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 36: conflicting requests 2024-04-14T15:01:39.748 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.748 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 37: conflicting requests 2024-04-14T15:01:39.749 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.749 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.749 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.749 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 38: cannot install the best candidate for the job 2024-04-14T15:01:39.749 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.749 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.749 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.749 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.749 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 39: conflicting requests 2024-04-14T15:01:39.749 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.749 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.749 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.749 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.749 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.749 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.750 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 40: cannot install the best candidate for the job 2024-04-14T15:01:39.750 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.750 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.750 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.750 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.750 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 41: cannot install the best candidate for the job 2024-04-14T15:01:39.750 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.750 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.750 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.750 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.750 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 42: cannot install the best candidate for the job 2024-04-14T15:01:39.750 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.750 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.750 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.751 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.751 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 43: cannot install the best candidate for the job 2024-04-14T15:01:39.751 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.751 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.751 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.751 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.751 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 44: cannot install the best candidate for the job 2024-04-14T15:01:39.751 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.751 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.751 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.751 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.751 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 45: cannot install the best candidate for the job 2024-04-14T15:01:39.751 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.752 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.752 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.752 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-14T15:01:39.811 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2024-04-14T15:01:39.812 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-04-14T15:01:39.812 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repository Size 2024-04-14T15:01:39.812 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-04-14T15:01:39.812 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2024-04-14T15:01:39.812 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2778.ga971fd5f.el9 ceph-noarch 141 k 2024-04-14T15:01:39.813 INFO:teuthology.orchestra.run.smithi092.stdout:Installing dependencies: 2024-04-14T15:01:39.813 INFO:teuthology.orchestra.run.smithi092.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-14T15:01:39.813 INFO:teuthology.orchestra.run.smithi092.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-14T15:01:39.813 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-14T15:01:39.813 INFO:teuthology.orchestra.run.smithi092.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-14T15:01:39.813 INFO:teuthology.orchestra.run.smithi092.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-14T15:01:39.813 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:01:39.813 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2024-04-14T15:01:39.813 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-04-14T15:01:39.813 INFO:teuthology.orchestra.run.smithi092.stdout:Install 6 Packages 2024-04-14T15:01:39.813 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:01:39.814 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 7.0 M 2024-04-14T15:01:39.814 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 33 M 2024-04-14T15:01:39.814 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2024-04-14T15:01:40.586 INFO:teuthology.orchestra.run.smithi092.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.7 MB/s | 248 kB 00:00 2024-04-14T15:01:40.736 INFO:teuthology.orchestra.run.smithi092.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 6.0 MB 00:00 2024-04-14T15:01:40.761 INFO:teuthology.orchestra.run.smithi092.stdout:(3/6): ceph-mgr-cephadm-19.0.0-2778.ga971fd5f.e 445 kB/s | 141 kB 00:00 2024-04-14T15:01:40.928 INFO:teuthology.orchestra.run.smithi092.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 301 kB/s | 58 kB 00:00 2024-04-14T15:01:40.953 INFO:teuthology.orchestra.run.smithi092.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 303 kB/s | 51 kB 00:00 2024-04-14T15:01:41.012 INFO:teuthology.orchestra.run.smithi092.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.3 MB/s | 548 kB 00:00 2024-04-14T15:01:41.015 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2024-04-14T15:01:41.015 INFO:teuthology.orchestra.run.smithi092.stdout:Total 5.8 MB/s | 7.0 MB 00:01 2024-04-14T15:01:41.086 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2024-04-14T15:01:41.097 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2024-04-14T15:01:41.097 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2024-04-14T15:01:41.199 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2024-04-14T15:01:41.200 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2024-04-14T15:01:41.518 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2024-04-14T15:01:41.675 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-14T15:01:41.730 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-14T15:01:42.241 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-14T15:01:42.332 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-14T15:01:42.394 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-14T15:01:42.417 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2778.ga971fd5f.el9.noarch 6/6 2024-04-14T15:01:43.246 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2778.ga971fd5f.el9.noarch 6/6 2024-04-14T15:01:43.247 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2778.ga971fd5f.el9.noarch 1/6 2024-04-14T15:01:43.247 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-14T15:01:43.247 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-14T15:01:43.247 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-14T15:01:43.247 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-14T15:01:43.577 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-14T15:01:43.578 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:01:43.578 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2024-04-14T15:01:43.578 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-cephadm-2:19.0.0-2778.ga971fd5f.el9.noarch 2024-04-14T15:01:43.578 INFO:teuthology.orchestra.run.smithi092.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-14T15:01:43.578 INFO:teuthology.orchestra.run.smithi092.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-14T15:01:43.578 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-14T15:01:43.578 INFO:teuthology.orchestra.run.smithi092.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-14T15:01:43.578 INFO:teuthology.orchestra.run.smithi092.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-14T15:01:43.578 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:01:43.578 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2024-04-14T15:01:43.861 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-fuse 2024-04-14T15:01:44.457 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:43 ago on Sun 14 Apr 2024 03:00:01 PM UTC. 2024-04-14T15:01:44.745 INFO:teuthology.orchestra.run.smithi092.stderr:Modular dependency problems: 2024-04-14T15:01:44.745 INFO:teuthology.orchestra.run.smithi092.stderr: 2024-04-14T15:01:44.745 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 1: conflicting requests 2024-04-14T15:01:44.745 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.745 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 2: conflicting requests 2024-04-14T15:01:44.745 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.746 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 3: conflicting requests 2024-04-14T15:01:44.746 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.746 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 4: conflicting requests 2024-04-14T15:01:44.746 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.746 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 5: conflicting requests 2024-04-14T15:01:44.746 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.746 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 6: conflicting requests 2024-04-14T15:01:44.746 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.746 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 7: conflicting requests 2024-04-14T15:01:44.746 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.746 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 8: conflicting requests 2024-04-14T15:01:44.747 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.747 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 9: conflicting requests 2024-04-14T15:01:44.747 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.747 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 10: conflicting requests 2024-04-14T15:01:44.747 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.747 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 11: conflicting requests 2024-04-14T15:01:44.747 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.747 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 12: conflicting requests 2024-04-14T15:01:44.747 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.747 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 13: conflicting requests 2024-04-14T15:01:44.747 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.747 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 14: conflicting requests 2024-04-14T15:01:44.747 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.747 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 15: conflicting requests 2024-04-14T15:01:44.747 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.748 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 16: conflicting requests 2024-04-14T15:01:44.748 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.748 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 17: conflicting requests 2024-04-14T15:01:44.748 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.748 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 18: conflicting requests 2024-04-14T15:01:44.748 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.748 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 19: conflicting requests 2024-04-14T15:01:44.748 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.748 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 20: conflicting requests 2024-04-14T15:01:44.748 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.748 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 21: conflicting requests 2024-04-14T15:01:44.748 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.748 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 22: conflicting requests 2024-04-14T15:01:44.749 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.749 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 23: conflicting requests 2024-04-14T15:01:44.749 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.749 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 24: conflicting requests 2024-04-14T15:01:44.749 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.749 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 25: conflicting requests 2024-04-14T15:01:44.749 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.749 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 26: conflicting requests 2024-04-14T15:01:44.749 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.749 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 27: conflicting requests 2024-04-14T15:01:44.749 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.750 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 28: conflicting requests 2024-04-14T15:01:44.750 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.750 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 29: conflicting requests 2024-04-14T15:01:44.750 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.750 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 30: conflicting requests 2024-04-14T15:01:44.750 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.750 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 31: conflicting requests 2024-04-14T15:01:44.750 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.750 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 32: conflicting requests 2024-04-14T15:01:44.750 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.750 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 33: conflicting requests 2024-04-14T15:01:44.750 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.750 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 34: conflicting requests 2024-04-14T15:01:44.751 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.751 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 35: conflicting requests 2024-04-14T15:01:44.751 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.751 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 36: conflicting requests 2024-04-14T15:01:44.751 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.751 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 37: conflicting requests 2024-04-14T15:01:44.751 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.751 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.751 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.751 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 38: cannot install the best candidate for the job 2024-04-14T15:01:44.751 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.751 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.751 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.752 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.752 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 39: conflicting requests 2024-04-14T15:01:44.752 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.752 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.752 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.752 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.752 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.752 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.752 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 40: cannot install the best candidate for the job 2024-04-14T15:01:44.752 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.752 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.752 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.752 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.753 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 41: cannot install the best candidate for the job 2024-04-14T15:01:44.753 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.753 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.753 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.753 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.753 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 42: cannot install the best candidate for the job 2024-04-14T15:01:44.753 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.753 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.753 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.753 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.753 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 43: cannot install the best candidate for the job 2024-04-14T15:01:44.754 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.754 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.754 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.754 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.754 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 44: cannot install the best candidate for the job 2024-04-14T15:01:44.754 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.754 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.754 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.754 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.754 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 45: cannot install the best candidate for the job 2024-04-14T15:01:44.754 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.754 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.755 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.755 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-14T15:01:44.815 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2024-04-14T15:01:44.816 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-04-14T15:01:44.816 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repository Size 2024-04-14T15:01:44.816 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-04-14T15:01:44.816 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2024-04-14T15:01:44.816 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-fuse x86_64 2:19.0.0-2778.ga971fd5f.el9 ceph 891 k 2024-04-14T15:01:44.816 INFO:teuthology.orchestra.run.smithi092.stdout:Installing dependencies: 2024-04-14T15:01:44.816 INFO:teuthology.orchestra.run.smithi092.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-14T15:01:44.816 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:01:44.816 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2024-04-14T15:01:44.816 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-04-14T15:01:44.816 INFO:teuthology.orchestra.run.smithi092.stdout:Install 2 Packages 2024-04-14T15:01:44.816 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:01:44.817 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 971 k 2024-04-14T15:01:44.817 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 2.8 M 2024-04-14T15:01:44.817 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2024-04-14T15:01:45.313 INFO:teuthology.orchestra.run.smithi092.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 187 kB/s | 80 kB 00:00 2024-04-14T15:01:45.547 INFO:teuthology.orchestra.run.smithi092.stdout:(2/2): ceph-fuse-19.0.0-2778.ga971fd5f.el9.x86_ 1.3 MB/s | 891 kB 00:00 2024-04-14T15:01:45.547 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2024-04-14T15:01:45.548 INFO:teuthology.orchestra.run.smithi092.stdout:Total 1.3 MB/s | 971 kB 00:00 2024-04-14T15:01:45.560 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2024-04-14T15:01:45.584 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2024-04-14T15:01:45.584 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2024-04-14T15:01:45.664 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2024-04-14T15:01:45.665 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2024-04-14T15:01:45.850 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2024-04-14T15:01:45.982 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-14T15:01:46.039 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-fuse-2:19.0.0-2778.ga971fd5f.el9.x86_64 2/2 2024-04-14T15:01:46.618 INFO:teuthology.orchestra.run.smithi090.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 759 kB/s | 16 MB 00:21 2024-04-14T15:01:46.619 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-04-14T15:01:46.620 INFO:teuthology.orchestra.run.smithi090.stdout:Total 1.4 MB/s | 38 MB 00:27 2024-04-14T15:01:46.733 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-fuse-2:19.0.0-2778.ga971fd5f.el9.x86_64 2/2 2024-04-14T15:01:46.733 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-fuse-2:19.0.0-2778.ga971fd5f.el9.x86_64 1/2 2024-04-14T15:01:46.970 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-04-14T15:01:46.998 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-04-14T15:01:46.998 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-04-14T15:01:47.062 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-14T15:01:47.062 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:01:47.062 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2024-04-14T15:01:47.062 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-fuse-2:19.0.0-2778.ga971fd5f.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-14T15:01:47.062 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:01:47.063 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2024-04-14T15:01:47.361 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-volume 2024-04-14T15:01:47.446 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-04-14T15:01:47.446 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-04-14T15:01:47.972 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:46 ago on Sun 14 Apr 2024 03:00:01 PM UTC. 2024-04-14T15:01:48.202 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-04-14T15:01:48.262 INFO:teuthology.orchestra.run.smithi092.stderr:Modular dependency problems: 2024-04-14T15:01:48.262 INFO:teuthology.orchestra.run.smithi092.stderr: 2024-04-14T15:01:48.262 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 1: conflicting requests 2024-04-14T15:01:48.262 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.262 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 2: conflicting requests 2024-04-14T15:01:48.262 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.262 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 3: conflicting requests 2024-04-14T15:01:48.262 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.263 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 4: conflicting requests 2024-04-14T15:01:48.263 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.263 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 5: conflicting requests 2024-04-14T15:01:48.263 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.263 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 6: conflicting requests 2024-04-14T15:01:48.263 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.263 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 7: conflicting requests 2024-04-14T15:01:48.263 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.263 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 8: conflicting requests 2024-04-14T15:01:48.263 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.263 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 9: conflicting requests 2024-04-14T15:01:48.263 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.263 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 10: conflicting requests 2024-04-14T15:01:48.263 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.264 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 11: conflicting requests 2024-04-14T15:01:48.264 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.264 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 12: conflicting requests 2024-04-14T15:01:48.264 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.264 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 13: conflicting requests 2024-04-14T15:01:48.264 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.264 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 14: conflicting requests 2024-04-14T15:01:48.264 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.264 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 15: conflicting requests 2024-04-14T15:01:48.264 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.264 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 16: conflicting requests 2024-04-14T15:01:48.264 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.264 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 17: conflicting requests 2024-04-14T15:01:48.264 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.265 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 18: conflicting requests 2024-04-14T15:01:48.265 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.265 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 19: conflicting requests 2024-04-14T15:01:48.265 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.265 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 20: conflicting requests 2024-04-14T15:01:48.265 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.265 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 21: conflicting requests 2024-04-14T15:01:48.265 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.265 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 22: conflicting requests 2024-04-14T15:01:48.265 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.265 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 23: conflicting requests 2024-04-14T15:01:48.265 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.265 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 24: conflicting requests 2024-04-14T15:01:48.265 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.266 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 25: conflicting requests 2024-04-14T15:01:48.266 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.266 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 26: conflicting requests 2024-04-14T15:01:48.266 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.266 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 27: conflicting requests 2024-04-14T15:01:48.266 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.266 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 28: conflicting requests 2024-04-14T15:01:48.266 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.266 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 29: conflicting requests 2024-04-14T15:01:48.266 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.266 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 30: conflicting requests 2024-04-14T15:01:48.266 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.266 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 31: conflicting requests 2024-04-14T15:01:48.267 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.267 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 32: conflicting requests 2024-04-14T15:01:48.267 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.267 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 33: conflicting requests 2024-04-14T15:01:48.267 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.267 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 34: conflicting requests 2024-04-14T15:01:48.267 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.267 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 35: conflicting requests 2024-04-14T15:01:48.267 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.267 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 36: conflicting requests 2024-04-14T15:01:48.267 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.267 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 37: conflicting requests 2024-04-14T15:01:48.267 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.268 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.268 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.268 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 38: cannot install the best candidate for the job 2024-04-14T15:01:48.268 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.268 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.268 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.268 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.268 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 39: conflicting requests 2024-04-14T15:01:48.268 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.268 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.268 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.268 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.268 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.268 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.269 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 40: cannot install the best candidate for the job 2024-04-14T15:01:48.269 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.269 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.269 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.269 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.269 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 41: cannot install the best candidate for the job 2024-04-14T15:01:48.269 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.269 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.269 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.269 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.269 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 42: cannot install the best candidate for the job 2024-04-14T15:01:48.269 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.269 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.270 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.270 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.270 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 43: cannot install the best candidate for the job 2024-04-14T15:01:48.270 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.270 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.270 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.270 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.270 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 44: cannot install the best candidate for the job 2024-04-14T15:01:48.270 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.270 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.270 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.270 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.270 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 45: cannot install the best candidate for the job 2024-04-14T15:01:48.271 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.271 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.271 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.271 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-14T15:01:48.271 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-04-14T15:01:48.317 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-04-14T15:01:48.330 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2024-04-14T15:01:48.331 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-04-14T15:01:48.331 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repository Size 2024-04-14T15:01:48.331 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-04-14T15:01:48.331 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2024-04-14T15:01:48.331 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-volume noarch 2:19.0.0-2778.ga971fd5f.el9 ceph-noarch 264 k 2024-04-14T15:01:48.331 INFO:teuthology.orchestra.run.smithi092.stdout:Installing dependencies: 2024-04-14T15:01:48.331 INFO:teuthology.orchestra.run.smithi092.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-14T15:01:48.331 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:01:48.332 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2024-04-14T15:01:48.332 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-04-14T15:01:48.332 INFO:teuthology.orchestra.run.smithi092.stdout:Install 2 Packages 2024-04-14T15:01:48.332 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:01:48.332 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 341 k 2024-04-14T15:01:48.332 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 1.5 M 2024-04-14T15:01:48.332 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2024-04-14T15:01:48.416 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-04-14T15:01:48.459 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-04-14T15:01:48.520 INFO:teuthology.orchestra.run.smithi092.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 700 kB/s | 77 kB 00:00 2024-04-14T15:01:48.637 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-14T15:01:48.667 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-14T15:01:48.678 INFO:teuthology.orchestra.run.smithi092.stdout:(2/2): ceph-volume-19.0.0-2778.ga971fd5f.el9.no 981 kB/s | 264 kB 00:00 2024-04-14T15:01:48.678 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2024-04-14T15:01:48.679 INFO:teuthology.orchestra.run.smithi092.stdout:Total 982 kB/s | 341 kB 00:00 2024-04-14T15:01:48.690 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2024-04-14T15:01:48.700 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2024-04-14T15:01:48.700 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2024-04-14T15:01:48.743 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2024-04-14T15:01:48.744 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2024-04-14T15:01:48.753 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-14T15:01:48.891 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2024-04-14T15:01:49.065 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-04-14T15:01:49.106 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-volume-2:19.0.0-2778.ga971fd5f.el9.noarch 2/2 2024-04-14T15:01:49.124 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-volume-2:19.0.0-2778.ga971fd5f.el9.noarch 2/2 2024-04-14T15:01:49.124 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-14T15:01:49.124 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-14T15:01:49.124 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:01:49.401 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-14T15:01:49.540 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-14T15:01:49.875 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-volume-2:19.0.0-2778.ga971fd5f.el9.noarch 1/2 2024-04-14T15:01:50.246 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-04-14T15:01:50.246 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:01:50.246 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2024-04-14T15:01:50.246 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-volume-2:19.0.0-2778.ga971fd5f.el9.noarch 2024-04-14T15:01:50.246 INFO:teuthology.orchestra.run.smithi092.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-14T15:01:50.246 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:01:50.246 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2024-04-14T15:01:50.570 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install librados-devel 2024-04-14T15:01:50.791 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-14T15:01:51.177 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:50 ago on Sun 14 Apr 2024 03:00:01 PM UTC. 2024-04-14T15:01:51.465 INFO:teuthology.orchestra.run.smithi092.stderr:Modular dependency problems: 2024-04-14T15:01:51.465 INFO:teuthology.orchestra.run.smithi092.stderr: 2024-04-14T15:01:51.465 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 1: conflicting requests 2024-04-14T15:01:51.465 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.465 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 2: conflicting requests 2024-04-14T15:01:51.466 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.466 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 3: conflicting requests 2024-04-14T15:01:51.466 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.466 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 4: conflicting requests 2024-04-14T15:01:51.466 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.466 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 5: conflicting requests 2024-04-14T15:01:51.466 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.466 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 6: conflicting requests 2024-04-14T15:01:51.466 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.466 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 7: conflicting requests 2024-04-14T15:01:51.466 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.466 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 8: conflicting requests 2024-04-14T15:01:51.466 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.467 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 9: conflicting requests 2024-04-14T15:01:51.467 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.467 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 10: conflicting requests 2024-04-14T15:01:51.467 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.467 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 11: conflicting requests 2024-04-14T15:01:51.467 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.467 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 12: conflicting requests 2024-04-14T15:01:51.467 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.467 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 13: conflicting requests 2024-04-14T15:01:51.467 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.467 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 14: conflicting requests 2024-04-14T15:01:51.467 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.467 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 15: conflicting requests 2024-04-14T15:01:51.468 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.468 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 16: conflicting requests 2024-04-14T15:01:51.468 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.468 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 17: conflicting requests 2024-04-14T15:01:51.468 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.468 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 18: conflicting requests 2024-04-14T15:01:51.468 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.468 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 19: conflicting requests 2024-04-14T15:01:51.468 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.468 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 20: conflicting requests 2024-04-14T15:01:51.468 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.468 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 21: conflicting requests 2024-04-14T15:01:51.468 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.469 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 22: conflicting requests 2024-04-14T15:01:51.469 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.469 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 23: conflicting requests 2024-04-14T15:01:51.469 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.469 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 24: conflicting requests 2024-04-14T15:01:51.469 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.469 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 25: conflicting requests 2024-04-14T15:01:51.469 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.469 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 26: conflicting requests 2024-04-14T15:01:51.469 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.469 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 27: conflicting requests 2024-04-14T15:01:51.469 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.469 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 28: conflicting requests 2024-04-14T15:01:51.470 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.470 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 29: conflicting requests 2024-04-14T15:01:51.470 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.470 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 30: conflicting requests 2024-04-14T15:01:51.470 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.470 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 31: conflicting requests 2024-04-14T15:01:51.470 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.470 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 32: conflicting requests 2024-04-14T15:01:51.470 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.470 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 33: conflicting requests 2024-04-14T15:01:51.470 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.470 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 34: conflicting requests 2024-04-14T15:01:51.470 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.471 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 35: conflicting requests 2024-04-14T15:01:51.471 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.471 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 36: conflicting requests 2024-04-14T15:01:51.471 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.471 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 37: conflicting requests 2024-04-14T15:01:51.471 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.471 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.471 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.471 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 38: cannot install the best candidate for the job 2024-04-14T15:01:51.471 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.471 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.471 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.471 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.471 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 39: conflicting requests 2024-04-14T15:01:51.471 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.472 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.472 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.472 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.472 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.472 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.472 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 40: cannot install the best candidate for the job 2024-04-14T15:01:51.472 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.472 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.472 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.472 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.472 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 41: cannot install the best candidate for the job 2024-04-14T15:01:51.472 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.472 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.472 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.472 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.473 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 42: cannot install the best candidate for the job 2024-04-14T15:01:51.473 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.473 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.473 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.473 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.473 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 43: cannot install the best candidate for the job 2024-04-14T15:01:51.473 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.473 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.473 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.473 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.473 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 44: cannot install the best candidate for the job 2024-04-14T15:01:51.473 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.473 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.473 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.473 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.473 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 45: cannot install the best candidate for the job 2024-04-14T15:01:51.474 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.474 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.474 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.474 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-14T15:01:51.533 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2024-04-14T15:01:51.534 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-04-14T15:01:51.534 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repo Size 2024-04-14T15:01:51.534 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-04-14T15:01:51.534 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2024-04-14T15:01:51.534 INFO:teuthology.orchestra.run.smithi092.stdout: librados-devel x86_64 2:19.0.0-2778.ga971fd5f.el9 ceph 127 k 2024-04-14T15:01:51.534 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:01:51.534 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2024-04-14T15:01:51.534 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-04-14T15:01:51.535 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2024-04-14T15:01:51.535 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:01:51.535 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 127 k 2024-04-14T15:01:51.535 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 456 k 2024-04-14T15:01:51.535 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2024-04-14T15:01:51.759 INFO:teuthology.orchestra.run.smithi092.stdout:librados-devel-19.0.0-2778.ga971fd5f.el9.x86_64 568 kB/s | 127 kB 00:00 2024-04-14T15:01:51.759 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2024-04-14T15:01:51.759 INFO:teuthology.orchestra.run.smithi092.stdout:Total 563 kB/s | 127 kB 00:00 2024-04-14T15:01:51.760 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2024-04-14T15:01:51.765 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2024-04-14T15:01:51.765 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2024-04-14T15:01:51.824 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2024-04-14T15:01:51.825 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2024-04-14T15:01:51.839 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-14T15:01:51.874 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2778.ga971f 12/12 2024-04-14T15:01:52.021 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2024-04-14T15:01:52.124 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : librados-devel-2:19.0.0-2778.ga971fd5f.el9.x86_64 1/1 2024-04-14T15:01:52.562 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librados-devel-2:19.0.0-2778.ga971fd5f.el9.x86_64 1/1 2024-04-14T15:01:52.883 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librados-devel-2:19.0.0-2778.ga971fd5f.el9.x86_64 1/1 2024-04-14T15:01:52.883 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:01:52.883 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2024-04-14T15:01:52.883 INFO:teuthology.orchestra.run.smithi092.stdout: librados-devel-2:19.0.0-2778.ga971fd5f.el9.x86_64 2024-04-14T15:01:52.883 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:01:52.883 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2024-04-14T15:01:53.111 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install libcephfs2 2024-04-14T15:01:53.720 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:52 ago on Sun 14 Apr 2024 03:00:01 PM UTC. 2024-04-14T15:01:54.010 INFO:teuthology.orchestra.run.smithi092.stderr:Modular dependency problems: 2024-04-14T15:01:54.010 INFO:teuthology.orchestra.run.smithi092.stderr: 2024-04-14T15:01:54.010 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 1: conflicting requests 2024-04-14T15:01:54.010 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.010 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 2: conflicting requests 2024-04-14T15:01:54.011 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.011 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 3: conflicting requests 2024-04-14T15:01:54.011 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.011 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 4: conflicting requests 2024-04-14T15:01:54.011 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.011 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 5: conflicting requests 2024-04-14T15:01:54.011 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.011 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 6: conflicting requests 2024-04-14T15:01:54.011 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.011 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 7: conflicting requests 2024-04-14T15:01:54.011 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.011 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 8: conflicting requests 2024-04-14T15:01:54.011 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.011 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 9: conflicting requests 2024-04-14T15:01:54.012 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.012 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 10: conflicting requests 2024-04-14T15:01:54.012 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.012 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 11: conflicting requests 2024-04-14T15:01:54.012 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.012 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 12: conflicting requests 2024-04-14T15:01:54.012 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.012 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 13: conflicting requests 2024-04-14T15:01:54.012 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.012 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 14: conflicting requests 2024-04-14T15:01:54.012 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.012 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 15: conflicting requests 2024-04-14T15:01:54.012 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.013 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 16: conflicting requests 2024-04-14T15:01:54.013 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.013 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 17: conflicting requests 2024-04-14T15:01:54.013 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.013 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 18: conflicting requests 2024-04-14T15:01:54.013 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.013 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 19: conflicting requests 2024-04-14T15:01:54.013 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.013 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 20: conflicting requests 2024-04-14T15:01:54.013 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.013 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 21: conflicting requests 2024-04-14T15:01:54.013 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.013 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 22: conflicting requests 2024-04-14T15:01:54.013 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.014 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 23: conflicting requests 2024-04-14T15:01:54.014 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.014 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 24: conflicting requests 2024-04-14T15:01:54.014 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.014 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 25: conflicting requests 2024-04-14T15:01:54.014 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.014 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 26: conflicting requests 2024-04-14T15:01:54.014 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.014 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 27: conflicting requests 2024-04-14T15:01:54.014 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.014 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 28: conflicting requests 2024-04-14T15:01:54.014 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.014 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 29: conflicting requests 2024-04-14T15:01:54.015 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.015 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 30: conflicting requests 2024-04-14T15:01:54.015 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.015 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 31: conflicting requests 2024-04-14T15:01:54.015 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.015 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 32: conflicting requests 2024-04-14T15:01:54.015 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.015 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 33: conflicting requests 2024-04-14T15:01:54.015 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.015 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 34: conflicting requests 2024-04-14T15:01:54.015 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.015 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 35: conflicting requests 2024-04-14T15:01:54.015 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.015 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 36: conflicting requests 2024-04-14T15:01:54.016 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.016 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 37: conflicting requests 2024-04-14T15:01:54.016 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.016 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.016 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.016 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 38: cannot install the best candidate for the job 2024-04-14T15:01:54.016 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.016 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.016 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.016 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.016 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 39: conflicting requests 2024-04-14T15:01:54.016 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.016 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.016 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.016 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.017 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.017 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.017 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 40: cannot install the best candidate for the job 2024-04-14T15:01:54.017 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.017 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.017 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.017 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.017 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 41: cannot install the best candidate for the job 2024-04-14T15:01:54.017 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.017 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.017 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.017 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.017 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 42: cannot install the best candidate for the job 2024-04-14T15:01:54.017 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.018 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.018 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.018 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.018 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 43: cannot install the best candidate for the job 2024-04-14T15:01:54.018 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.018 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.018 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.018 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.018 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 44: cannot install the best candidate for the job 2024-04-14T15:01:54.018 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.018 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.018 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.018 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.018 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 45: cannot install the best candidate for the job 2024-04-14T15:01:54.019 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.019 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.019 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.019 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-14T15:01:54.019 INFO:teuthology.orchestra.run.smithi092.stdout:Package libcephfs2-2:19.0.0-2778.ga971fd5f.el9.x86_64 is already installed. 2024-04-14T15:01:54.078 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2024-04-14T15:01:54.079 INFO:teuthology.orchestra.run.smithi092.stdout:Nothing to do. 2024-04-14T15:01:54.079 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2024-04-14T15:01:54.152 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install libcephfs-devel 2024-04-14T15:01:54.744 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2778.ga971f 12/12 2024-04-14T15:01:54.744 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2778.ga971f 1/12 2024-04-14T15:01:54.744 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-14T15:01:54.744 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-04-14T15:01:54.744 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-04-14T15:01:54.745 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-04-14T15:01:54.745 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-04-14T15:01:54.745 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-04-14T15:01:54.745 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-04-14T15:01:54.745 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 9/12 2024-04-14T15:01:54.745 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-04-14T15:01:54.745 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-04-14T15:01:54.747 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:53 ago on Sun 14 Apr 2024 03:00:01 PM UTC. 2024-04-14T15:01:55.039 INFO:teuthology.orchestra.run.smithi092.stderr:Modular dependency problems: 2024-04-14T15:01:55.039 INFO:teuthology.orchestra.run.smithi092.stderr: 2024-04-14T15:01:55.039 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 1: conflicting requests 2024-04-14T15:01:55.039 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.039 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 2: conflicting requests 2024-04-14T15:01:55.039 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.039 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 3: conflicting requests 2024-04-14T15:01:55.039 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.039 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 4: conflicting requests 2024-04-14T15:01:55.039 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.040 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 5: conflicting requests 2024-04-14T15:01:55.040 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.040 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 6: conflicting requests 2024-04-14T15:01:55.040 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.040 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 7: conflicting requests 2024-04-14T15:01:55.040 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.040 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 8: conflicting requests 2024-04-14T15:01:55.040 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.040 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 9: conflicting requests 2024-04-14T15:01:55.040 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.040 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 10: conflicting requests 2024-04-14T15:01:55.040 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.040 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 11: conflicting requests 2024-04-14T15:01:55.040 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.041 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 12: conflicting requests 2024-04-14T15:01:55.041 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.041 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 13: conflicting requests 2024-04-14T15:01:55.041 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.041 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 14: conflicting requests 2024-04-14T15:01:55.041 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.041 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 15: conflicting requests 2024-04-14T15:01:55.041 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.041 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 16: conflicting requests 2024-04-14T15:01:55.041 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.041 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 17: conflicting requests 2024-04-14T15:01:55.041 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.041 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 18: conflicting requests 2024-04-14T15:01:55.042 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.042 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 19: conflicting requests 2024-04-14T15:01:55.042 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.042 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 20: conflicting requests 2024-04-14T15:01:55.042 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.042 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 21: conflicting requests 2024-04-14T15:01:55.042 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.042 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 22: conflicting requests 2024-04-14T15:01:55.042 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.042 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 23: conflicting requests 2024-04-14T15:01:55.042 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.042 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 24: conflicting requests 2024-04-14T15:01:55.042 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.043 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 25: conflicting requests 2024-04-14T15:01:55.043 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.043 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 26: conflicting requests 2024-04-14T15:01:55.043 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.043 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 27: conflicting requests 2024-04-14T15:01:55.043 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.043 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 28: conflicting requests 2024-04-14T15:01:55.043 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.043 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 29: conflicting requests 2024-04-14T15:01:55.043 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.043 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 30: conflicting requests 2024-04-14T15:01:55.043 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.043 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 31: conflicting requests 2024-04-14T15:01:55.043 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.044 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 32: conflicting requests 2024-04-14T15:01:55.044 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.044 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 33: conflicting requests 2024-04-14T15:01:55.044 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.044 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 34: conflicting requests 2024-04-14T15:01:55.044 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.044 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 35: conflicting requests 2024-04-14T15:01:55.044 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.044 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 36: conflicting requests 2024-04-14T15:01:55.044 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.044 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 37: conflicting requests 2024-04-14T15:01:55.044 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.044 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.044 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.045 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 38: cannot install the best candidate for the job 2024-04-14T15:01:55.045 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.045 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.045 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.045 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.045 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 39: conflicting requests 2024-04-14T15:01:55.045 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.045 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.045 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.045 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.045 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.045 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.045 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 40: cannot install the best candidate for the job 2024-04-14T15:01:55.045 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.045 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.046 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.046 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.046 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 41: cannot install the best candidate for the job 2024-04-14T15:01:55.046 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.046 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.046 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.046 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.046 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 42: cannot install the best candidate for the job 2024-04-14T15:01:55.046 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.046 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.046 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.046 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.046 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 43: cannot install the best candidate for the job 2024-04-14T15:01:55.046 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.047 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.047 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.047 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.047 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 44: cannot install the best candidate for the job 2024-04-14T15:01:55.047 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.047 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.047 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.047 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.047 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 45: cannot install the best candidate for the job 2024-04-14T15:01:55.047 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.047 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.047 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.047 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-14T15:01:55.107 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2024-04-14T15:01:55.107 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-04-14T15:01:55.107 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repo Size 2024-04-14T15:01:55.107 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-04-14T15:01:55.107 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2024-04-14T15:01:55.107 INFO:teuthology.orchestra.run.smithi092.stdout: libcephfs-devel x86_64 2:19.0.0-2778.ga971fd5f.el9 ceph 32 k 2024-04-14T15:01:55.107 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:01:55.108 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2024-04-14T15:01:55.108 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-04-14T15:01:55.108 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2024-04-14T15:01:55.108 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:01:55.108 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 32 k 2024-04-14T15:01:55.108 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 142 k 2024-04-14T15:01:55.108 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2024-04-14T15:01:55.194 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-04-14T15:01:55.194 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:01:55.194 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-04-14T15:01:55.194 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2778.ga971fd5f.el9.noarch 2024-04-14T15:01:55.195 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-14T15:01:55.195 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-14T15:01:55.195 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-14T15:01:55.195 INFO:teuthology.orchestra.run.smithi090.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-14T15:01:55.195 INFO:teuthology.orchestra.run.smithi090.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-14T15:01:55.195 INFO:teuthology.orchestra.run.smithi090.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-14T15:01:55.195 INFO:teuthology.orchestra.run.smithi090.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-14T15:01:55.195 INFO:teuthology.orchestra.run.smithi090.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-14T15:01:55.195 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-14T15:01:55.195 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-14T15:01:55.195 INFO:teuthology.orchestra.run.smithi090.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-14T15:01:55.195 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:01:55.196 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-04-14T15:01:55.269 INFO:teuthology.orchestra.run.smithi092.stdout:libcephfs-devel-19.0.0-2778.ga971fd5f.el9.x86_6 197 kB/s | 32 kB 00:00 2024-04-14T15:01:55.270 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2024-04-14T15:01:55.270 INFO:teuthology.orchestra.run.smithi092.stdout:Total 195 kB/s | 32 kB 00:00 2024-04-14T15:01:55.271 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2024-04-14T15:01:55.273 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2024-04-14T15:01:55.274 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2024-04-14T15:01:55.304 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2024-04-14T15:01:55.305 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2024-04-14T15:01:55.392 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2024-04-14T15:01:55.493 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libcephfs-devel-2:19.0.0-2778.ga971fd5f.el9.x86_64 1/1 2024-04-14T15:01:55.709 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-mgr-rook 2024-04-14T15:01:55.875 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-2778.ga971fd5f.el9.x86_64 1/1 2024-04-14T15:01:56.157 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libcephfs-devel-2:19.0.0-2778.ga971fd5f.el9.x86_64 1/1 2024-04-14T15:01:56.158 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:01:56.158 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2024-04-14T15:01:56.158 INFO:teuthology.orchestra.run.smithi092.stdout: libcephfs-devel-2:19.0.0-2778.ga971fd5f.el9.x86_64 2024-04-14T15:01:56.158 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:01:56.158 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2024-04-14T15:01:56.313 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:58 ago on Sun 14 Apr 2024 02:59:58 PM UTC. 2024-04-14T15:01:56.350 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install librados2 2024-04-14T15:01:56.600 INFO:teuthology.orchestra.run.smithi090.stderr:Modular dependency problems: 2024-04-14T15:01:56.600 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-04-14T15:01:56.600 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 1: conflicting requests 2024-04-14T15:01:56.600 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.600 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 2: conflicting requests 2024-04-14T15:01:56.600 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.600 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 3: conflicting requests 2024-04-14T15:01:56.601 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.601 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 4: conflicting requests 2024-04-14T15:01:56.601 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.601 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 5: conflicting requests 2024-04-14T15:01:56.601 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.601 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 6: conflicting requests 2024-04-14T15:01:56.601 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.601 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 7: conflicting requests 2024-04-14T15:01:56.601 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.601 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 8: conflicting requests 2024-04-14T15:01:56.601 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.601 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 9: conflicting requests 2024-04-14T15:01:56.601 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.602 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 10: conflicting requests 2024-04-14T15:01:56.602 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.602 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 11: conflicting requests 2024-04-14T15:01:56.602 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.602 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 12: conflicting requests 2024-04-14T15:01:56.602 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.602 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 13: conflicting requests 2024-04-14T15:01:56.602 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.602 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 14: conflicting requests 2024-04-14T15:01:56.602 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.602 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 15: conflicting requests 2024-04-14T15:01:56.602 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.602 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 16: conflicting requests 2024-04-14T15:01:56.602 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.603 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 17: conflicting requests 2024-04-14T15:01:56.603 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.603 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 18: conflicting requests 2024-04-14T15:01:56.603 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.603 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 19: conflicting requests 2024-04-14T15:01:56.603 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.603 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 20: conflicting requests 2024-04-14T15:01:56.603 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.603 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 21: conflicting requests 2024-04-14T15:01:56.603 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.603 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 22: conflicting requests 2024-04-14T15:01:56.603 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.603 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 23: conflicting requests 2024-04-14T15:01:56.603 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.604 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 24: conflicting requests 2024-04-14T15:01:56.604 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.604 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 25: conflicting requests 2024-04-14T15:01:56.604 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.604 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 26: conflicting requests 2024-04-14T15:01:56.604 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.604 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 27: conflicting requests 2024-04-14T15:01:56.604 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.604 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 28: conflicting requests 2024-04-14T15:01:56.604 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.604 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 29: conflicting requests 2024-04-14T15:01:56.604 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.604 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 30: conflicting requests 2024-04-14T15:01:56.604 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.605 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 31: conflicting requests 2024-04-14T15:01:56.605 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.605 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 32: conflicting requests 2024-04-14T15:01:56.605 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.605 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 33: conflicting requests 2024-04-14T15:01:56.605 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.605 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 34: conflicting requests 2024-04-14T15:01:56.605 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.605 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 35: conflicting requests 2024-04-14T15:01:56.605 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.605 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 36: conflicting requests 2024-04-14T15:01:56.605 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.605 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 37: conflicting requests 2024-04-14T15:01:56.605 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.605 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.606 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.606 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 38: cannot install the best candidate for the job 2024-04-14T15:01:56.606 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.606 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.606 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.606 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.606 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 39: conflicting requests 2024-04-14T15:01:56.606 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.606 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.606 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.606 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.606 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.606 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.607 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 40: cannot install the best candidate for the job 2024-04-14T15:01:56.607 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.607 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.607 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.607 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.607 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 41: cannot install the best candidate for the job 2024-04-14T15:01:56.607 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.607 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.607 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.607 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.607 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 42: cannot install the best candidate for the job 2024-04-14T15:01:56.607 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.607 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.608 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.608 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.608 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 43: cannot install the best candidate for the job 2024-04-14T15:01:56.608 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.608 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.608 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.608 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.608 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 44: cannot install the best candidate for the job 2024-04-14T15:01:56.608 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.608 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.608 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.608 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.608 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 45: cannot install the best candidate for the job 2024-04-14T15:01:56.609 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.609 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.609 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.609 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-14T15:01:56.669 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-04-14T15:01:56.670 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-04-14T15:01:56.670 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-04-14T15:01:56.670 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-04-14T15:01:56.670 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-04-14T15:01:56.670 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-rook noarch 2:19.0.0-2778.ga971fd5f.el9 ceph-noarch 49 k 2024-04-14T15:01:56.670 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-04-14T15:01:56.670 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-14T15:01:56.671 INFO:teuthology.orchestra.run.smithi090.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-14T15:01:56.671 INFO:teuthology.orchestra.run.smithi090.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-14T15:01:56.671 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-14T15:01:56.671 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-14T15:01:56.671 INFO:teuthology.orchestra.run.smithi090.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-14T15:01:56.671 INFO:teuthology.orchestra.run.smithi090.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-14T15:01:56.671 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-14T15:01:56.671 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-14T15:01:56.671 INFO:teuthology.orchestra.run.smithi090.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-14T15:01:56.671 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-14T15:01:56.671 INFO:teuthology.orchestra.run.smithi090.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-14T15:01:56.671 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:01:56.671 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-04-14T15:01:56.672 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-04-14T15:01:56.672 INFO:teuthology.orchestra.run.smithi090.stdout:Install 13 Packages 2024-04-14T15:01:56.672 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:01:56.672 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 2.2 M 2024-04-14T15:01:56.672 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 26 M 2024-04-14T15:01:56.672 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-04-14T15:01:56.944 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:55 ago on Sun 14 Apr 2024 03:00:01 PM UTC. 2024-04-14T15:01:57.073 INFO:teuthology.orchestra.run.smithi090.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 174 kB/s | 19 kB 00:00 2024-04-14T15:01:57.099 INFO:teuthology.orchestra.run.smithi090.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 189 kB/s | 26 kB 00:00 2024-04-14T15:01:57.174 INFO:teuthology.orchestra.run.smithi090.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.2 MB/s | 222 kB 00:00 2024-04-14T15:01:57.225 INFO:teuthology.orchestra.run.smithi090.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 159 kB 00:00 2024-04-14T15:01:57.231 INFO:teuthology.orchestra.run.smithi092.stderr:Modular dependency problems: 2024-04-14T15:01:57.231 INFO:teuthology.orchestra.run.smithi092.stderr: 2024-04-14T15:01:57.231 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 1: conflicting requests 2024-04-14T15:01:57.231 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.232 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 2: conflicting requests 2024-04-14T15:01:57.232 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.232 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 3: conflicting requests 2024-04-14T15:01:57.232 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.232 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 4: conflicting requests 2024-04-14T15:01:57.232 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.232 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 5: conflicting requests 2024-04-14T15:01:57.232 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.232 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 6: conflicting requests 2024-04-14T15:01:57.232 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.232 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 7: conflicting requests 2024-04-14T15:01:57.233 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.233 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 8: conflicting requests 2024-04-14T15:01:57.233 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.233 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 9: conflicting requests 2024-04-14T15:01:57.233 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.233 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 10: conflicting requests 2024-04-14T15:01:57.233 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.233 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 11: conflicting requests 2024-04-14T15:01:57.233 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.233 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 12: conflicting requests 2024-04-14T15:01:57.233 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.233 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 13: conflicting requests 2024-04-14T15:01:57.233 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.234 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 14: conflicting requests 2024-04-14T15:01:57.234 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.234 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 15: conflicting requests 2024-04-14T15:01:57.234 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.234 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 16: conflicting requests 2024-04-14T15:01:57.234 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.234 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 17: conflicting requests 2024-04-14T15:01:57.234 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.234 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 18: conflicting requests 2024-04-14T15:01:57.234 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.234 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 19: conflicting requests 2024-04-14T15:01:57.234 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.234 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 20: conflicting requests 2024-04-14T15:01:57.234 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.235 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 21: conflicting requests 2024-04-14T15:01:57.235 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.235 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 22: conflicting requests 2024-04-14T15:01:57.235 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.235 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 23: conflicting requests 2024-04-14T15:01:57.235 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.235 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 24: conflicting requests 2024-04-14T15:01:57.235 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.235 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 25: conflicting requests 2024-04-14T15:01:57.235 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.235 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 26: conflicting requests 2024-04-14T15:01:57.235 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.235 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 27: conflicting requests 2024-04-14T15:01:57.235 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.236 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 28: conflicting requests 2024-04-14T15:01:57.236 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.236 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 29: conflicting requests 2024-04-14T15:01:57.236 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.236 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 30: conflicting requests 2024-04-14T15:01:57.236 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.236 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 31: conflicting requests 2024-04-14T15:01:57.236 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.236 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 32: conflicting requests 2024-04-14T15:01:57.236 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.236 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 33: conflicting requests 2024-04-14T15:01:57.236 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.236 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 34: conflicting requests 2024-04-14T15:01:57.236 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.237 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 35: conflicting requests 2024-04-14T15:01:57.237 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.237 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 36: conflicting requests 2024-04-14T15:01:57.237 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.237 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 37: conflicting requests 2024-04-14T15:01:57.237 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.237 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.237 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.238 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 38: cannot install the best candidate for the job 2024-04-14T15:01:57.238 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.238 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.238 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.238 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.238 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 39: conflicting requests 2024-04-14T15:01:57.239 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.239 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.239 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.239 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.239 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.239 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.239 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 40: cannot install the best candidate for the job 2024-04-14T15:01:57.239 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.239 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.239 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.239 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.239 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 41: cannot install the best candidate for the job 2024-04-14T15:01:57.239 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.240 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.240 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.240 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.240 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 42: cannot install the best candidate for the job 2024-04-14T15:01:57.240 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.240 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.240 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.240 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.240 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 43: cannot install the best candidate for the job 2024-04-14T15:01:57.240 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.240 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.240 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.240 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.240 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 44: cannot install the best candidate for the job 2024-04-14T15:01:57.241 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.241 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.241 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.241 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.241 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 45: cannot install the best candidate for the job 2024-04-14T15:01:57.241 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.241 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.241 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.241 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-14T15:01:57.241 INFO:teuthology.orchestra.run.smithi092.stdout:Package librados2-2:19.0.0-2778.ga971fd5f.el9.x86_64 is already installed. 2024-04-14T15:01:57.250 INFO:teuthology.orchestra.run.smithi090.stdout:(5/13): ceph-mgr-rook-19.0.0-2778.ga971fd5f.el9 172 kB/s | 49 kB 00:00 2024-04-14T15:01:57.275 INFO:teuthology.orchestra.run.smithi090.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-04-14T15:01:57.299 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2024-04-14T15:01:57.300 INFO:teuthology.orchestra.run.smithi092.stdout:Nothing to do. 2024-04-14T15:01:57.300 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2024-04-14T15:01:57.301 INFO:teuthology.orchestra.run.smithi090.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.2 MB/s | 279 kB 00:00 2024-04-14T15:01:57.359 INFO:teuthology.orchestra.run.smithi090.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 179 kB/s | 15 kB 00:00 2024-04-14T15:01:57.380 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install librbd1 2024-04-14T15:01:57.384 INFO:teuthology.orchestra.run.smithi090.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 240 kB/s | 32 kB 00:00 2024-04-14T15:01:57.451 INFO:teuthology.orchestra.run.smithi090.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 886 kB/s | 59 kB 00:00 2024-04-14T15:01:57.476 INFO:teuthology.orchestra.run.smithi090.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 1.1 MB/s | 203 kB 00:00 2024-04-14T15:01:57.510 INFO:teuthology.orchestra.run.smithi090.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.5 MB/s | 90 kB 00:00 2024-04-14T15:01:57.544 INFO:teuthology.orchestra.run.smithi090.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 5.5 MB/s | 1.0 MB 00:00 2024-04-14T15:01:57.544 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-04-14T15:01:57.544 INFO:teuthology.orchestra.run.smithi090.stdout:Total 2.5 MB/s | 2.2 MB 00:00 2024-04-14T15:01:57.607 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-04-14T15:01:57.622 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-04-14T15:01:57.622 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-04-14T15:01:57.716 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-04-14T15:01:57.717 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-04-14T15:01:57.975 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:56 ago on Sun 14 Apr 2024 03:00:01 PM UTC. 2024-04-14T15:01:58.236 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-04-14T15:01:58.266 INFO:teuthology.orchestra.run.smithi092.stderr:Modular dependency problems: 2024-04-14T15:01:58.266 INFO:teuthology.orchestra.run.smithi092.stderr: 2024-04-14T15:01:58.266 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 1: conflicting requests 2024-04-14T15:01:58.266 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.266 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 2: conflicting requests 2024-04-14T15:01:58.267 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.267 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 3: conflicting requests 2024-04-14T15:01:58.267 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.267 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 4: conflicting requests 2024-04-14T15:01:58.267 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.267 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 5: conflicting requests 2024-04-14T15:01:58.267 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.267 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 6: conflicting requests 2024-04-14T15:01:58.267 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.267 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 7: conflicting requests 2024-04-14T15:01:58.267 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.267 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 8: conflicting requests 2024-04-14T15:01:58.267 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.267 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 9: conflicting requests 2024-04-14T15:01:58.268 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.268 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 10: conflicting requests 2024-04-14T15:01:58.268 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.268 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 11: conflicting requests 2024-04-14T15:01:58.268 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.268 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 12: conflicting requests 2024-04-14T15:01:58.268 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.268 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 13: conflicting requests 2024-04-14T15:01:58.268 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.268 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 14: conflicting requests 2024-04-14T15:01:58.268 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.268 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 15: conflicting requests 2024-04-14T15:01:58.268 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.269 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 16: conflicting requests 2024-04-14T15:01:58.269 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.269 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 17: conflicting requests 2024-04-14T15:01:58.269 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.269 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 18: conflicting requests 2024-04-14T15:01:58.269 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.269 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 19: conflicting requests 2024-04-14T15:01:58.269 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.269 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 20: conflicting requests 2024-04-14T15:01:58.269 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.269 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 21: conflicting requests 2024-04-14T15:01:58.269 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.269 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 22: conflicting requests 2024-04-14T15:01:58.270 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.270 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 23: conflicting requests 2024-04-14T15:01:58.270 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.270 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 24: conflicting requests 2024-04-14T15:01:58.270 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.270 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 25: conflicting requests 2024-04-14T15:01:58.270 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.270 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 26: conflicting requests 2024-04-14T15:01:58.270 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.270 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 27: conflicting requests 2024-04-14T15:01:58.270 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.270 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 28: conflicting requests 2024-04-14T15:01:58.270 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.271 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 29: conflicting requests 2024-04-14T15:01:58.271 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.271 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 30: conflicting requests 2024-04-14T15:01:58.271 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.271 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 31: conflicting requests 2024-04-14T15:01:58.271 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.271 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 32: conflicting requests 2024-04-14T15:01:58.271 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.271 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 33: conflicting requests 2024-04-14T15:01:58.271 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.271 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 34: conflicting requests 2024-04-14T15:01:58.271 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.271 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 35: conflicting requests 2024-04-14T15:01:58.271 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.272 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 36: conflicting requests 2024-04-14T15:01:58.272 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.272 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 37: conflicting requests 2024-04-14T15:01:58.272 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.272 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.272 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.272 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 38: cannot install the best candidate for the job 2024-04-14T15:01:58.272 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.272 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.272 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.272 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.272 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 39: conflicting requests 2024-04-14T15:01:58.272 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.272 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.273 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.273 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.273 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.273 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.273 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 40: cannot install the best candidate for the job 2024-04-14T15:01:58.273 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.273 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.273 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.273 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.273 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 41: cannot install the best candidate for the job 2024-04-14T15:01:58.273 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.273 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.273 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.273 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.274 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 42: cannot install the best candidate for the job 2024-04-14T15:01:58.274 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.274 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.274 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.274 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.274 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 43: cannot install the best candidate for the job 2024-04-14T15:01:58.274 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.274 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.274 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.274 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.274 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 44: cannot install the best candidate for the job 2024-04-14T15:01:58.274 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.274 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.275 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.275 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.275 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 45: cannot install the best candidate for the job 2024-04-14T15:01:58.275 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.275 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.275 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.275 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-14T15:01:58.275 INFO:teuthology.orchestra.run.smithi092.stdout:Package librbd1-2:19.0.0-2778.ga971fd5f.el9.x86_64 is already installed. 2024-04-14T15:01:58.334 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2024-04-14T15:01:58.335 INFO:teuthology.orchestra.run.smithi092.stdout:Nothing to do. 2024-04-14T15:01:58.335 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2024-04-14T15:01:58.412 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install python3-rados 2024-04-14T15:01:58.417 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-14T15:01:58.481 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-14T15:01:58.544 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-14T15:01:58.581 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-14T15:01:58.623 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-14T15:01:58.717 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-14T15:01:58.818 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-04-14T15:01:58.875 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-14T15:01:59.007 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:58 ago on Sun 14 Apr 2024 03:00:01 PM UTC. 2024-04-14T15:01:59.298 INFO:teuthology.orchestra.run.smithi092.stderr:Modular dependency problems: 2024-04-14T15:01:59.299 INFO:teuthology.orchestra.run.smithi092.stderr: 2024-04-14T15:01:59.299 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 1: conflicting requests 2024-04-14T15:01:59.299 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.299 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 2: conflicting requests 2024-04-14T15:01:59.299 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.299 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 3: conflicting requests 2024-04-14T15:01:59.299 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.299 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 4: conflicting requests 2024-04-14T15:01:59.299 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.299 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 5: conflicting requests 2024-04-14T15:01:59.300 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.300 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 6: conflicting requests 2024-04-14T15:01:59.300 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.300 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 7: conflicting requests 2024-04-14T15:01:59.300 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.300 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 8: conflicting requests 2024-04-14T15:01:59.300 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.300 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 9: conflicting requests 2024-04-14T15:01:59.300 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.300 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 10: conflicting requests 2024-04-14T15:01:59.300 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.300 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 11: conflicting requests 2024-04-14T15:01:59.300 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.300 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 12: conflicting requests 2024-04-14T15:01:59.301 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.301 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 13: conflicting requests 2024-04-14T15:01:59.301 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.301 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 14: conflicting requests 2024-04-14T15:01:59.301 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.301 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 15: conflicting requests 2024-04-14T15:01:59.301 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.301 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 16: conflicting requests 2024-04-14T15:01:59.301 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.301 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 17: conflicting requests 2024-04-14T15:01:59.301 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.301 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 18: conflicting requests 2024-04-14T15:01:59.301 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.301 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 19: conflicting requests 2024-04-14T15:01:59.302 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.302 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 20: conflicting requests 2024-04-14T15:01:59.302 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.302 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 21: conflicting requests 2024-04-14T15:01:59.302 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.302 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 22: conflicting requests 2024-04-14T15:01:59.302 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.302 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 23: conflicting requests 2024-04-14T15:01:59.302 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.302 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 24: conflicting requests 2024-04-14T15:01:59.303 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.303 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 25: conflicting requests 2024-04-14T15:01:59.303 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.303 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 26: conflicting requests 2024-04-14T15:01:59.303 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.303 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 27: conflicting requests 2024-04-14T15:01:59.303 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.303 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 28: conflicting requests 2024-04-14T15:01:59.303 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.303 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 29: conflicting requests 2024-04-14T15:01:59.303 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.303 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 30: conflicting requests 2024-04-14T15:01:59.304 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.304 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 31: conflicting requests 2024-04-14T15:01:59.304 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.304 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 32: conflicting requests 2024-04-14T15:01:59.304 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.304 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 33: conflicting requests 2024-04-14T15:01:59.304 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.304 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 34: conflicting requests 2024-04-14T15:01:59.304 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.304 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 35: conflicting requests 2024-04-14T15:01:59.304 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.304 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 36: conflicting requests 2024-04-14T15:01:59.304 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.305 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 37: conflicting requests 2024-04-14T15:01:59.305 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.305 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.305 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.305 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 38: cannot install the best candidate for the job 2024-04-14T15:01:59.305 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.305 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.305 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.305 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.305 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 39: conflicting requests 2024-04-14T15:01:59.306 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.306 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.306 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.306 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.306 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.306 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.306 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 40: cannot install the best candidate for the job 2024-04-14T15:01:59.306 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.306 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.306 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.306 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.306 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 41: cannot install the best candidate for the job 2024-04-14T15:01:59.307 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.307 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.307 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.307 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.307 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 42: cannot install the best candidate for the job 2024-04-14T15:01:59.307 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.307 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.307 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.307 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.307 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 43: cannot install the best candidate for the job 2024-04-14T15:01:59.307 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.307 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.307 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.307 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.307 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 44: cannot install the best candidate for the job 2024-04-14T15:01:59.308 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.308 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.308 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.308 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.308 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 45: cannot install the best candidate for the job 2024-04-14T15:01:59.308 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.308 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.308 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.308 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-14T15:01:59.308 INFO:teuthology.orchestra.run.smithi092.stdout:Package python3-rados-2:19.0.0-2778.ga971fd5f.el9.x86_64 is already installed. 2024-04-14T15:01:59.365 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2024-04-14T15:01:59.367 INFO:teuthology.orchestra.run.smithi092.stdout:Nothing to do. 2024-04-14T15:01:59.367 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2024-04-14T15:01:59.446 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install python3-rgw 2024-04-14T15:01:59.604 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-14T15:01:59.666 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-14T15:01:59.691 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-14T15:01:59.731 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-14T15:01:59.761 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-rook-2:19.0.0-2778.ga971fd5f.el9.noarch 13/13 2024-04-14T15:02:00.048 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:59 ago on Sun 14 Apr 2024 03:00:01 PM UTC. 2024-04-14T15:02:00.349 INFO:teuthology.orchestra.run.smithi092.stderr:Modular dependency problems: 2024-04-14T15:02:00.350 INFO:teuthology.orchestra.run.smithi092.stderr: 2024-04-14T15:02:00.350 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 1: conflicting requests 2024-04-14T15:02:00.350 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.350 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 2: conflicting requests 2024-04-14T15:02:00.350 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.350 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 3: conflicting requests 2024-04-14T15:02:00.350 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.350 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 4: conflicting requests 2024-04-14T15:02:00.350 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.350 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 5: conflicting requests 2024-04-14T15:02:00.350 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.350 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 6: conflicting requests 2024-04-14T15:02:00.350 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.351 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 7: conflicting requests 2024-04-14T15:02:00.351 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.351 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 8: conflicting requests 2024-04-14T15:02:00.351 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.351 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 9: conflicting requests 2024-04-14T15:02:00.351 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.351 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 10: conflicting requests 2024-04-14T15:02:00.351 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.351 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 11: conflicting requests 2024-04-14T15:02:00.351 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.351 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 12: conflicting requests 2024-04-14T15:02:00.351 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.351 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 13: conflicting requests 2024-04-14T15:02:00.351 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.352 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 14: conflicting requests 2024-04-14T15:02:00.352 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.352 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 15: conflicting requests 2024-04-14T15:02:00.352 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.352 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 16: conflicting requests 2024-04-14T15:02:00.352 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.352 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 17: conflicting requests 2024-04-14T15:02:00.352 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.352 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 18: conflicting requests 2024-04-14T15:02:00.352 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.352 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 19: conflicting requests 2024-04-14T15:02:00.352 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.352 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 20: conflicting requests 2024-04-14T15:02:00.352 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.353 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 21: conflicting requests 2024-04-14T15:02:00.353 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.353 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 22: conflicting requests 2024-04-14T15:02:00.353 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.353 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 23: conflicting requests 2024-04-14T15:02:00.353 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.353 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 24: conflicting requests 2024-04-14T15:02:00.353 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.353 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 25: conflicting requests 2024-04-14T15:02:00.353 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.353 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 26: conflicting requests 2024-04-14T15:02:00.353 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.353 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 27: conflicting requests 2024-04-14T15:02:00.354 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.354 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 28: conflicting requests 2024-04-14T15:02:00.354 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.354 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 29: conflicting requests 2024-04-14T15:02:00.354 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.354 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 30: conflicting requests 2024-04-14T15:02:00.354 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.354 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 31: conflicting requests 2024-04-14T15:02:00.354 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.354 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 32: conflicting requests 2024-04-14T15:02:00.354 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.354 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 33: conflicting requests 2024-04-14T15:02:00.354 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.354 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 34: conflicting requests 2024-04-14T15:02:00.355 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.355 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 35: conflicting requests 2024-04-14T15:02:00.355 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.355 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 36: conflicting requests 2024-04-14T15:02:00.355 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.355 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 37: conflicting requests 2024-04-14T15:02:00.355 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.355 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.355 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.355 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 38: cannot install the best candidate for the job 2024-04-14T15:02:00.355 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.355 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.355 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.355 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.356 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 39: conflicting requests 2024-04-14T15:02:00.356 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.356 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.356 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.356 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.356 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.356 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.356 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 40: cannot install the best candidate for the job 2024-04-14T15:02:00.356 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.356 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.356 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.356 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.356 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 41: cannot install the best candidate for the job 2024-04-14T15:02:00.356 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.357 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.357 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.357 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.357 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 42: cannot install the best candidate for the job 2024-04-14T15:02:00.357 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.357 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.357 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.357 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.357 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 43: cannot install the best candidate for the job 2024-04-14T15:02:00.357 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.357 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.357 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.357 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.357 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 44: cannot install the best candidate for the job 2024-04-14T15:02:00.357 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.358 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.358 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.358 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.358 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 45: cannot install the best candidate for the job 2024-04-14T15:02:00.358 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.358 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.358 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.358 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-14T15:02:00.358 INFO:teuthology.orchestra.run.smithi092.stdout:Package python3-rgw-2:19.0.0-2778.ga971fd5f.el9.x86_64 is already installed. 2024-04-14T15:02:00.418 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2024-04-14T15:02:00.419 INFO:teuthology.orchestra.run.smithi092.stdout:Nothing to do. 2024-04-14T15:02:00.419 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2024-04-14T15:02:00.501 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install python3-cephfs 2024-04-14T15:02:00.705 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2778.ga971fd5f.el9.noarch 13/13 2024-04-14T15:02:00.705 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2778.ga971fd5f.el9.noarch 1/13 2024-04-14T15:02:00.705 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-14T15:02:00.705 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-14T15:02:00.705 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-14T15:02:00.705 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-14T15:02:00.705 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-14T15:02:00.705 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-14T15:02:00.705 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-14T15:02:00.706 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-14T15:02:00.706 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-04-14T15:02:00.706 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-14T15:02:00.706 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-14T15:02:01.070 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-14T15:02:01.070 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:02:01.070 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-04-14T15:02:01.070 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-rook-2:19.0.0-2778.ga971fd5f.el9.noarch 2024-04-14T15:02:01.070 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-14T15:02:01.070 INFO:teuthology.orchestra.run.smithi090.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-14T15:02:01.070 INFO:teuthology.orchestra.run.smithi090.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-14T15:02:01.070 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-14T15:02:01.071 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-14T15:02:01.071 INFO:teuthology.orchestra.run.smithi090.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-14T15:02:01.071 INFO:teuthology.orchestra.run.smithi090.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-14T15:02:01.071 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-14T15:02:01.071 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-14T15:02:01.071 INFO:teuthology.orchestra.run.smithi090.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-14T15:02:01.071 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-14T15:02:01.071 INFO:teuthology.orchestra.run.smithi090.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-14T15:02:01.071 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:02:01.071 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-04-14T15:02:01.099 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:02:00 ago on Sun 14 Apr 2024 03:00:01 PM UTC. 2024-04-14T15:02:01.356 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-mgr-cephadm 2024-04-14T15:02:01.389 INFO:teuthology.orchestra.run.smithi092.stderr:Modular dependency problems: 2024-04-14T15:02:01.389 INFO:teuthology.orchestra.run.smithi092.stderr: 2024-04-14T15:02:01.389 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 1: conflicting requests 2024-04-14T15:02:01.389 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.389 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 2: conflicting requests 2024-04-14T15:02:01.389 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.390 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 3: conflicting requests 2024-04-14T15:02:01.390 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.390 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 4: conflicting requests 2024-04-14T15:02:01.390 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.390 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 5: conflicting requests 2024-04-14T15:02:01.390 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.390 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 6: conflicting requests 2024-04-14T15:02:01.390 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.390 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 7: conflicting requests 2024-04-14T15:02:01.390 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.391 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 8: conflicting requests 2024-04-14T15:02:01.391 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.391 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 9: conflicting requests 2024-04-14T15:02:01.391 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.391 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 10: conflicting requests 2024-04-14T15:02:01.391 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.391 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 11: conflicting requests 2024-04-14T15:02:01.391 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.391 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 12: conflicting requests 2024-04-14T15:02:01.391 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.391 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 13: conflicting requests 2024-04-14T15:02:01.392 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.392 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 14: conflicting requests 2024-04-14T15:02:01.392 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.392 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 15: conflicting requests 2024-04-14T15:02:01.392 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.392 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 16: conflicting requests 2024-04-14T15:02:01.392 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.392 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 17: conflicting requests 2024-04-14T15:02:01.392 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.392 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 18: conflicting requests 2024-04-14T15:02:01.392 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.392 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 19: conflicting requests 2024-04-14T15:02:01.392 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.392 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 20: conflicting requests 2024-04-14T15:02:01.393 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.393 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 21: conflicting requests 2024-04-14T15:02:01.393 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.393 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 22: conflicting requests 2024-04-14T15:02:01.393 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.393 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 23: conflicting requests 2024-04-14T15:02:01.393 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.393 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 24: conflicting requests 2024-04-14T15:02:01.393 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.393 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 25: conflicting requests 2024-04-14T15:02:01.393 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.393 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 26: conflicting requests 2024-04-14T15:02:01.393 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.393 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 27: conflicting requests 2024-04-14T15:02:01.394 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.394 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 28: conflicting requests 2024-04-14T15:02:01.394 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.394 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 29: conflicting requests 2024-04-14T15:02:01.394 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.394 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 30: conflicting requests 2024-04-14T15:02:01.394 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.394 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 31: conflicting requests 2024-04-14T15:02:01.394 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.394 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 32: conflicting requests 2024-04-14T15:02:01.394 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.394 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 33: conflicting requests 2024-04-14T15:02:01.394 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.394 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 34: conflicting requests 2024-04-14T15:02:01.395 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.395 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 35: conflicting requests 2024-04-14T15:02:01.395 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.395 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 36: conflicting requests 2024-04-14T15:02:01.395 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.395 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 37: conflicting requests 2024-04-14T15:02:01.395 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.395 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.395 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.396 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 38: cannot install the best candidate for the job 2024-04-14T15:02:01.396 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.396 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.396 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.396 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.396 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 39: conflicting requests 2024-04-14T15:02:01.396 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.396 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.396 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.396 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.396 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.396 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.396 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 40: cannot install the best candidate for the job 2024-04-14T15:02:01.396 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.397 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.397 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.397 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.397 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 41: cannot install the best candidate for the job 2024-04-14T15:02:01.397 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.397 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.397 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.397 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.397 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 42: cannot install the best candidate for the job 2024-04-14T15:02:01.397 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.397 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.397 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.397 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.397 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 43: cannot install the best candidate for the job 2024-04-14T15:02:01.398 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.398 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.398 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.398 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.398 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 44: cannot install the best candidate for the job 2024-04-14T15:02:01.398 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.398 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.398 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.398 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.398 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 45: cannot install the best candidate for the job 2024-04-14T15:02:01.398 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.399 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.399 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.399 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-14T15:02:01.399 INFO:teuthology.orchestra.run.smithi092.stdout:Package python3-cephfs-2:19.0.0-2778.ga971fd5f.el9.x86_64 is already installed. 2024-04-14T15:02:01.456 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2024-04-14T15:02:01.457 INFO:teuthology.orchestra.run.smithi092.stdout:Nothing to do. 2024-04-14T15:02:01.457 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2024-04-14T15:02:01.542 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install python3-rbd 2024-04-14T15:02:01.954 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:02:03 ago on Sun 14 Apr 2024 02:59:58 PM UTC. 2024-04-14T15:02:02.141 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:02:01 ago on Sun 14 Apr 2024 03:00:01 PM UTC. 2024-04-14T15:02:02.244 INFO:teuthology.orchestra.run.smithi090.stderr:Modular dependency problems: 2024-04-14T15:02:02.244 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-04-14T15:02:02.244 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 1: conflicting requests 2024-04-14T15:02:02.245 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.245 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 2: conflicting requests 2024-04-14T15:02:02.245 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.245 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 3: conflicting requests 2024-04-14T15:02:02.245 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.245 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 4: conflicting requests 2024-04-14T15:02:02.245 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.245 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 5: conflicting requests 2024-04-14T15:02:02.245 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.245 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 6: conflicting requests 2024-04-14T15:02:02.245 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.245 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 7: conflicting requests 2024-04-14T15:02:02.245 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.246 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 8: conflicting requests 2024-04-14T15:02:02.246 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.246 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 9: conflicting requests 2024-04-14T15:02:02.246 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.246 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 10: conflicting requests 2024-04-14T15:02:02.246 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.246 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 11: conflicting requests 2024-04-14T15:02:02.246 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.246 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 12: conflicting requests 2024-04-14T15:02:02.246 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.246 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 13: conflicting requests 2024-04-14T15:02:02.246 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.246 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 14: conflicting requests 2024-04-14T15:02:02.247 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.247 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 15: conflicting requests 2024-04-14T15:02:02.247 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.247 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 16: conflicting requests 2024-04-14T15:02:02.247 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.247 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 17: conflicting requests 2024-04-14T15:02:02.247 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.247 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 18: conflicting requests 2024-04-14T15:02:02.247 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.247 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 19: conflicting requests 2024-04-14T15:02:02.247 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.248 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 20: conflicting requests 2024-04-14T15:02:02.248 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.248 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 21: conflicting requests 2024-04-14T15:02:02.248 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.248 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 22: conflicting requests 2024-04-14T15:02:02.248 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.249 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 23: conflicting requests 2024-04-14T15:02:02.249 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.249 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 24: conflicting requests 2024-04-14T15:02:02.249 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.249 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 25: conflicting requests 2024-04-14T15:02:02.249 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.249 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 26: conflicting requests 2024-04-14T15:02:02.249 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.249 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 27: conflicting requests 2024-04-14T15:02:02.250 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.250 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 28: conflicting requests 2024-04-14T15:02:02.250 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.250 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 29: conflicting requests 2024-04-14T15:02:02.250 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.250 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 30: conflicting requests 2024-04-14T15:02:02.250 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.250 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 31: conflicting requests 2024-04-14T15:02:02.250 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.250 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 32: conflicting requests 2024-04-14T15:02:02.250 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.251 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 33: conflicting requests 2024-04-14T15:02:02.251 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.251 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 34: conflicting requests 2024-04-14T15:02:02.251 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.251 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 35: conflicting requests 2024-04-14T15:02:02.251 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.251 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 36: conflicting requests 2024-04-14T15:02:02.251 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.251 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 37: conflicting requests 2024-04-14T15:02:02.251 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.251 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.251 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.251 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 38: cannot install the best candidate for the job 2024-04-14T15:02:02.251 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.251 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.252 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.252 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.252 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 39: conflicting requests 2024-04-14T15:02:02.252 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.252 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.252 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.252 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.252 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.252 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.252 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 40: cannot install the best candidate for the job 2024-04-14T15:02:02.252 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.252 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.252 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.252 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.252 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 41: cannot install the best candidate for the job 2024-04-14T15:02:02.253 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.253 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.253 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.253 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.253 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 42: cannot install the best candidate for the job 2024-04-14T15:02:02.253 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.253 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.253 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.253 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.253 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 43: cannot install the best candidate for the job 2024-04-14T15:02:02.253 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.253 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.254 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.254 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.254 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 44: cannot install the best candidate for the job 2024-04-14T15:02:02.254 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.254 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.255 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.255 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.255 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 45: cannot install the best candidate for the job 2024-04-14T15:02:02.255 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.255 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.255 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.256 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.314 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-04-14T15:02:02.314 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-04-14T15:02:02.315 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-04-14T15:02:02.315 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-04-14T15:02:02.315 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-04-14T15:02:02.315 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2778.ga971fd5f.el9 ceph-noarch 141 k 2024-04-14T15:02:02.315 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-04-14T15:02:02.315 INFO:teuthology.orchestra.run.smithi090.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-14T15:02:02.315 INFO:teuthology.orchestra.run.smithi090.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-14T15:02:02.315 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-14T15:02:02.315 INFO:teuthology.orchestra.run.smithi090.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-14T15:02:02.315 INFO:teuthology.orchestra.run.smithi090.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-14T15:02:02.315 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:02:02.315 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-04-14T15:02:02.315 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-04-14T15:02:02.315 INFO:teuthology.orchestra.run.smithi090.stdout:Install 6 Packages 2024-04-14T15:02:02.316 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:02:02.316 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 7.0 M 2024-04-14T15:02:02.316 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 33 M 2024-04-14T15:02:02.316 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-04-14T15:02:02.430 INFO:teuthology.orchestra.run.smithi092.stderr:Modular dependency problems: 2024-04-14T15:02:02.430 INFO:teuthology.orchestra.run.smithi092.stderr: 2024-04-14T15:02:02.431 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 1: conflicting requests 2024-04-14T15:02:02.431 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.431 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 2: conflicting requests 2024-04-14T15:02:02.431 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.431 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 3: conflicting requests 2024-04-14T15:02:02.431 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.431 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 4: conflicting requests 2024-04-14T15:02:02.431 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.431 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 5: conflicting requests 2024-04-14T15:02:02.431 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.431 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 6: conflicting requests 2024-04-14T15:02:02.431 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.432 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 7: conflicting requests 2024-04-14T15:02:02.432 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.432 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 8: conflicting requests 2024-04-14T15:02:02.432 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.432 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 9: conflicting requests 2024-04-14T15:02:02.432 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.432 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 10: conflicting requests 2024-04-14T15:02:02.433 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.433 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 11: conflicting requests 2024-04-14T15:02:02.433 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.433 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 12: conflicting requests 2024-04-14T15:02:02.433 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.433 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 13: conflicting requests 2024-04-14T15:02:02.433 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.433 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 14: conflicting requests 2024-04-14T15:02:02.434 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.434 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 15: conflicting requests 2024-04-14T15:02:02.434 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.434 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 16: conflicting requests 2024-04-14T15:02:02.434 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.434 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 17: conflicting requests 2024-04-14T15:02:02.434 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.434 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 18: conflicting requests 2024-04-14T15:02:02.434 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.434 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 19: conflicting requests 2024-04-14T15:02:02.434 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.435 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 20: conflicting requests 2024-04-14T15:02:02.435 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.435 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 21: conflicting requests 2024-04-14T15:02:02.435 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.435 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 22: conflicting requests 2024-04-14T15:02:02.435 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.435 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 23: conflicting requests 2024-04-14T15:02:02.435 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.435 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 24: conflicting requests 2024-04-14T15:02:02.436 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.436 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 25: conflicting requests 2024-04-14T15:02:02.436 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.436 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 26: conflicting requests 2024-04-14T15:02:02.436 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.436 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 27: conflicting requests 2024-04-14T15:02:02.436 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.436 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 28: conflicting requests 2024-04-14T15:02:02.436 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.436 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 29: conflicting requests 2024-04-14T15:02:02.436 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.436 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 30: conflicting requests 2024-04-14T15:02:02.436 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.437 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 31: conflicting requests 2024-04-14T15:02:02.437 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.437 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 32: conflicting requests 2024-04-14T15:02:02.437 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.437 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 33: conflicting requests 2024-04-14T15:02:02.437 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.437 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 34: conflicting requests 2024-04-14T15:02:02.437 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.437 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 35: conflicting requests 2024-04-14T15:02:02.437 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.437 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 36: conflicting requests 2024-04-14T15:02:02.437 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.437 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 37: conflicting requests 2024-04-14T15:02:02.438 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.438 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.438 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.438 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 38: cannot install the best candidate for the job 2024-04-14T15:02:02.438 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.438 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.438 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.438 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.438 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 39: conflicting requests 2024-04-14T15:02:02.438 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.438 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.438 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.438 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.438 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.439 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.439 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 40: cannot install the best candidate for the job 2024-04-14T15:02:02.439 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.439 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.439 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.439 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.439 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 41: cannot install the best candidate for the job 2024-04-14T15:02:02.439 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.439 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.439 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.439 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.439 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 42: cannot install the best candidate for the job 2024-04-14T15:02:02.440 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.440 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.440 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.440 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.440 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 43: cannot install the best candidate for the job 2024-04-14T15:02:02.440 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.440 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.440 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.440 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.440 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 44: cannot install the best candidate for the job 2024-04-14T15:02:02.440 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.440 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.440 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.440 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.441 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 45: cannot install the best candidate for the job 2024-04-14T15:02:02.441 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.441 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.441 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.441 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-14T15:02:02.441 INFO:teuthology.orchestra.run.smithi092.stdout:Package python3-rbd-2:19.0.0-2778.ga971fd5f.el9.x86_64 is already installed. 2024-04-14T15:02:02.498 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2024-04-14T15:02:02.499 INFO:teuthology.orchestra.run.smithi092.stdout:Nothing to do. 2024-04-14T15:02:02.499 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2024-04-14T15:02:02.566 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install rbd-fuse 2024-04-14T15:02:02.836 INFO:teuthology.orchestra.run.smithi090.stdout:(1/6): ceph-mgr-cephadm-19.0.0-2778.ga971fd5f.e 610 kB/s | 141 kB 00:00 2024-04-14T15:02:03.120 INFO:teuthology.orchestra.run.smithi090.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.9 MB/s | 548 kB 00:00 2024-04-14T15:02:03.165 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:02:02 ago on Sun 14 Apr 2024 03:00:01 PM UTC. 2024-04-14T15:02:03.187 INFO:teuthology.orchestra.run.smithi090.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 866 kB/s | 58 kB 00:00 2024-04-14T15:02:03.254 INFO:teuthology.orchestra.run.smithi090.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 764 kB/s | 51 kB 00:00 2024-04-14T15:02:03.362 INFO:teuthology.orchestra.run.smithi090.stdout:(5/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 327 kB/s | 248 kB 00:00 2024-04-14T15:02:03.457 INFO:teuthology.orchestra.run.smithi092.stderr:Modular dependency problems: 2024-04-14T15:02:03.457 INFO:teuthology.orchestra.run.smithi092.stderr: 2024-04-14T15:02:03.457 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 1: conflicting requests 2024-04-14T15:02:03.457 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.457 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 2: conflicting requests 2024-04-14T15:02:03.457 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.457 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 3: conflicting requests 2024-04-14T15:02:03.457 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.457 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 4: conflicting requests 2024-04-14T15:02:03.457 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.457 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 5: conflicting requests 2024-04-14T15:02:03.458 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.458 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 6: conflicting requests 2024-04-14T15:02:03.458 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.458 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 7: conflicting requests 2024-04-14T15:02:03.458 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.458 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 8: conflicting requests 2024-04-14T15:02:03.458 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.458 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 9: conflicting requests 2024-04-14T15:02:03.458 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.458 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 10: conflicting requests 2024-04-14T15:02:03.458 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.458 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 11: conflicting requests 2024-04-14T15:02:03.458 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.459 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 12: conflicting requests 2024-04-14T15:02:03.459 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.459 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 13: conflicting requests 2024-04-14T15:02:03.459 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.459 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 14: conflicting requests 2024-04-14T15:02:03.459 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.459 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 15: conflicting requests 2024-04-14T15:02:03.459 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.459 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 16: conflicting requests 2024-04-14T15:02:03.459 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.459 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 17: conflicting requests 2024-04-14T15:02:03.459 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.459 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 18: conflicting requests 2024-04-14T15:02:03.459 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.460 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 19: conflicting requests 2024-04-14T15:02:03.460 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.460 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 20: conflicting requests 2024-04-14T15:02:03.460 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.460 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 21: conflicting requests 2024-04-14T15:02:03.460 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.460 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 22: conflicting requests 2024-04-14T15:02:03.460 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.460 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 23: conflicting requests 2024-04-14T15:02:03.460 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.460 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 24: conflicting requests 2024-04-14T15:02:03.460 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.460 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 25: conflicting requests 2024-04-14T15:02:03.460 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.461 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 26: conflicting requests 2024-04-14T15:02:03.461 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.461 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 27: conflicting requests 2024-04-14T15:02:03.461 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.461 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 28: conflicting requests 2024-04-14T15:02:03.461 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.461 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 29: conflicting requests 2024-04-14T15:02:03.461 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.461 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 30: conflicting requests 2024-04-14T15:02:03.461 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.461 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 31: conflicting requests 2024-04-14T15:02:03.461 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.462 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 32: conflicting requests 2024-04-14T15:02:03.462 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.462 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 33: conflicting requests 2024-04-14T15:02:03.462 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.462 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 34: conflicting requests 2024-04-14T15:02:03.462 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.462 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 35: conflicting requests 2024-04-14T15:02:03.462 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.462 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 36: conflicting requests 2024-04-14T15:02:03.462 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.462 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 37: conflicting requests 2024-04-14T15:02:03.462 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.462 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.462 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.463 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 38: cannot install the best candidate for the job 2024-04-14T15:02:03.463 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.463 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.463 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.463 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.463 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 39: conflicting requests 2024-04-14T15:02:03.463 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.463 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.463 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.463 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.463 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.463 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.463 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 40: cannot install the best candidate for the job 2024-04-14T15:02:03.463 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.464 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.464 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.464 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.464 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 41: cannot install the best candidate for the job 2024-04-14T15:02:03.464 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.464 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.464 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.464 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.464 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 42: cannot install the best candidate for the job 2024-04-14T15:02:03.464 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.464 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.464 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.464 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.464 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 43: cannot install the best candidate for the job 2024-04-14T15:02:03.465 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.465 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.465 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.465 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.465 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 44: cannot install the best candidate for the job 2024-04-14T15:02:03.465 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.465 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.465 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.465 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.465 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 45: cannot install the best candidate for the job 2024-04-14T15:02:03.465 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.465 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.465 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.465 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-14T15:02:03.525 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2024-04-14T15:02:03.526 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-04-14T15:02:03.526 INFO:teuthology.orchestra.run.smithi092.stdout: Package Architecture Version Repository Size 2024-04-14T15:02:03.526 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-04-14T15:02:03.526 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2024-04-14T15:02:03.526 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-fuse x86_64 2:19.0.0-2778.ga971fd5f.el9 ceph 85 k 2024-04-14T15:02:03.526 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:02:03.526 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2024-04-14T15:02:03.526 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-04-14T15:02:03.526 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2024-04-14T15:02:03.527 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:02:03.527 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 85 k 2024-04-14T15:02:03.527 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 231 k 2024-04-14T15:02:03.527 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2024-04-14T15:02:03.747 INFO:teuthology.orchestra.run.smithi092.stdout:rbd-fuse-19.0.0-2778.ga971fd5f.el9.x86_64.rpm 387 kB/s | 85 kB 00:00 2024-04-14T15:02:03.747 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2024-04-14T15:02:03.748 INFO:teuthology.orchestra.run.smithi092.stdout:Total 385 kB/s | 85 kB 00:00 2024-04-14T15:02:03.748 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2024-04-14T15:02:03.754 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2024-04-14T15:02:03.754 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2024-04-14T15:02:03.813 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2024-04-14T15:02:03.814 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2024-04-14T15:02:03.950 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2024-04-14T15:02:04.037 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : rbd-fuse-2:19.0.0-2778.ga971fd5f.el9.x86_64 1/1 2024-04-14T15:02:04.421 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: rbd-fuse-2:19.0.0-2778.ga971fd5f.el9.x86_64 1/1 2024-04-14T15:02:04.730 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : rbd-fuse-2:19.0.0-2778.ga971fd5f.el9.x86_64 1/1 2024-04-14T15:02:04.730 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:02:04.730 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2024-04-14T15:02:04.730 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-fuse-2:19.0.0-2778.ga971fd5f.el9.x86_64 2024-04-14T15:02:04.730 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:02:04.730 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2024-04-14T15:02:04.946 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install rbd-mirror 2024-04-14T15:02:05.556 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:02:04 ago on Sun 14 Apr 2024 03:00:01 PM UTC. 2024-04-14T15:02:05.846 INFO:teuthology.orchestra.run.smithi092.stderr:Modular dependency problems: 2024-04-14T15:02:05.847 INFO:teuthology.orchestra.run.smithi092.stderr: 2024-04-14T15:02:05.847 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 1: conflicting requests 2024-04-14T15:02:05.847 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.847 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 2: conflicting requests 2024-04-14T15:02:05.847 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.847 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 3: conflicting requests 2024-04-14T15:02:05.847 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.847 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 4: conflicting requests 2024-04-14T15:02:05.847 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.847 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 5: conflicting requests 2024-04-14T15:02:05.847 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.847 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 6: conflicting requests 2024-04-14T15:02:05.848 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.848 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 7: conflicting requests 2024-04-14T15:02:05.848 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.848 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 8: conflicting requests 2024-04-14T15:02:05.848 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.848 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 9: conflicting requests 2024-04-14T15:02:05.848 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.848 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 10: conflicting requests 2024-04-14T15:02:05.848 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.848 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 11: conflicting requests 2024-04-14T15:02:05.848 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.848 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 12: conflicting requests 2024-04-14T15:02:05.848 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.848 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 13: conflicting requests 2024-04-14T15:02:05.849 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.849 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 14: conflicting requests 2024-04-14T15:02:05.849 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.849 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 15: conflicting requests 2024-04-14T15:02:05.849 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.849 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 16: conflicting requests 2024-04-14T15:02:05.849 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.849 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 17: conflicting requests 2024-04-14T15:02:05.849 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.849 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 18: conflicting requests 2024-04-14T15:02:05.849 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.849 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 19: conflicting requests 2024-04-14T15:02:05.849 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.850 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 20: conflicting requests 2024-04-14T15:02:05.850 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.850 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 21: conflicting requests 2024-04-14T15:02:05.850 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.850 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 22: conflicting requests 2024-04-14T15:02:05.850 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.850 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 23: conflicting requests 2024-04-14T15:02:05.850 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.850 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 24: conflicting requests 2024-04-14T15:02:05.850 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.850 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 25: conflicting requests 2024-04-14T15:02:05.850 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.850 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 26: conflicting requests 2024-04-14T15:02:05.851 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.851 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 27: conflicting requests 2024-04-14T15:02:05.851 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.851 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 28: conflicting requests 2024-04-14T15:02:05.851 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.851 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 29: conflicting requests 2024-04-14T15:02:05.851 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.851 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 30: conflicting requests 2024-04-14T15:02:05.851 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.851 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 31: conflicting requests 2024-04-14T15:02:05.851 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.851 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 32: conflicting requests 2024-04-14T15:02:05.851 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.851 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 33: conflicting requests 2024-04-14T15:02:05.852 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.852 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 34: conflicting requests 2024-04-14T15:02:05.852 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.852 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 35: conflicting requests 2024-04-14T15:02:05.852 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.852 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 36: conflicting requests 2024-04-14T15:02:05.852 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.852 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 37: conflicting requests 2024-04-14T15:02:05.852 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.852 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.852 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.852 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 38: cannot install the best candidate for the job 2024-04-14T15:02:05.852 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.852 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.853 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.853 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.853 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 39: conflicting requests 2024-04-14T15:02:05.853 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.853 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.853 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.853 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.853 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.853 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.853 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 40: cannot install the best candidate for the job 2024-04-14T15:02:05.853 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.853 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.853 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.854 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.854 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 41: cannot install the best candidate for the job 2024-04-14T15:02:05.854 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.854 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.854 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.854 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.854 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 42: cannot install the best candidate for the job 2024-04-14T15:02:05.854 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.854 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.854 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.854 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.854 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 43: cannot install the best candidate for the job 2024-04-14T15:02:05.854 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.854 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.855 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.855 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.855 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 44: cannot install the best candidate for the job 2024-04-14T15:02:05.855 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.855 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.855 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.855 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.855 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 45: cannot install the best candidate for the job 2024-04-14T15:02:05.855 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.855 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.855 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.855 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-14T15:02:05.913 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2024-04-14T15:02:05.914 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-04-14T15:02:05.914 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repo Size 2024-04-14T15:02:05.914 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-04-14T15:02:05.914 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2024-04-14T15:02:05.914 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-mirror x86_64 2:19.0.0-2778.ga971fd5f.el9 ceph 3.1 M 2024-04-14T15:02:05.914 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:02:05.914 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2024-04-14T15:02:05.915 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-04-14T15:02:05.915 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2024-04-14T15:02:05.915 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:02:05.915 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 3.1 M 2024-04-14T15:02:05.915 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 13 M 2024-04-14T15:02:05.915 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2024-04-14T15:02:06.307 INFO:teuthology.orchestra.run.smithi092.stdout:rbd-mirror-19.0.0-2778.ga971fd5f.el9.x86_64.rpm 7.9 MB/s | 3.1 MB 00:00 2024-04-14T15:02:06.308 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2024-04-14T15:02:06.308 INFO:teuthology.orchestra.run.smithi092.stdout:Total 7.8 MB/s | 3.1 MB 00:00 2024-04-14T15:02:06.308 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2024-04-14T15:02:06.317 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2024-04-14T15:02:06.317 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2024-04-14T15:02:06.393 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2024-04-14T15:02:06.393 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2024-04-14T15:02:06.814 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2024-04-14T15:02:06.875 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : rbd-mirror-2:19.0.0-2778.ga971fd5f.el9.x86_64 1/1 2024-04-14T15:02:06.899 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2778.ga971fd5f.el9.x86_64 1/1 2024-04-14T15:02:06.899 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-14T15:02:06.899 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-14T15:02:06.899 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-14T15:02:06.899 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-14T15:02:06.899 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:02:08.081 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : rbd-mirror-2:19.0.0-2778.ga971fd5f.el9.x86_64 1/1 2024-04-14T15:02:08.081 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:02:08.081 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2024-04-14T15:02:08.081 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-mirror-2:19.0.0-2778.ga971fd5f.el9.x86_64 2024-04-14T15:02:08.081 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:02:08.081 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2024-04-14T15:02:08.343 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install rbd-nbd 2024-04-14T15:02:08.952 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:02:07 ago on Sun 14 Apr 2024 03:00:01 PM UTC. 2024-04-14T15:02:09.243 INFO:teuthology.orchestra.run.smithi092.stderr:Modular dependency problems: 2024-04-14T15:02:09.244 INFO:teuthology.orchestra.run.smithi092.stderr: 2024-04-14T15:02:09.244 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 1: conflicting requests 2024-04-14T15:02:09.244 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.244 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 2: conflicting requests 2024-04-14T15:02:09.244 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.244 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 3: conflicting requests 2024-04-14T15:02:09.244 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.244 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 4: conflicting requests 2024-04-14T15:02:09.244 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.244 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 5: conflicting requests 2024-04-14T15:02:09.244 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.244 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 6: conflicting requests 2024-04-14T15:02:09.245 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.245 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 7: conflicting requests 2024-04-14T15:02:09.245 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.245 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 8: conflicting requests 2024-04-14T15:02:09.245 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.245 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 9: conflicting requests 2024-04-14T15:02:09.245 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.245 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 10: conflicting requests 2024-04-14T15:02:09.245 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.245 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 11: conflicting requests 2024-04-14T15:02:09.245 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.245 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 12: conflicting requests 2024-04-14T15:02:09.245 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.246 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 13: conflicting requests 2024-04-14T15:02:09.246 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.246 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 14: conflicting requests 2024-04-14T15:02:09.246 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.246 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 15: conflicting requests 2024-04-14T15:02:09.246 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.246 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 16: conflicting requests 2024-04-14T15:02:09.246 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.246 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 17: conflicting requests 2024-04-14T15:02:09.246 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.246 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 18: conflicting requests 2024-04-14T15:02:09.246 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.246 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 19: conflicting requests 2024-04-14T15:02:09.247 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.247 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 20: conflicting requests 2024-04-14T15:02:09.247 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.247 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 21: conflicting requests 2024-04-14T15:02:09.247 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.247 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 22: conflicting requests 2024-04-14T15:02:09.247 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.247 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 23: conflicting requests 2024-04-14T15:02:09.247 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.247 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 24: conflicting requests 2024-04-14T15:02:09.247 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.247 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 25: conflicting requests 2024-04-14T15:02:09.247 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.248 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 26: conflicting requests 2024-04-14T15:02:09.248 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.248 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 27: conflicting requests 2024-04-14T15:02:09.248 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.248 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 28: conflicting requests 2024-04-14T15:02:09.248 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.248 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 29: conflicting requests 2024-04-14T15:02:09.248 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.248 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 30: conflicting requests 2024-04-14T15:02:09.248 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.248 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 31: conflicting requests 2024-04-14T15:02:09.248 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.248 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 32: conflicting requests 2024-04-14T15:02:09.248 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.249 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 33: conflicting requests 2024-04-14T15:02:09.249 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.249 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 34: conflicting requests 2024-04-14T15:02:09.249 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.249 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 35: conflicting requests 2024-04-14T15:02:09.249 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.249 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 36: conflicting requests 2024-04-14T15:02:09.249 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.249 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 37: conflicting requests 2024-04-14T15:02:09.249 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.249 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.249 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.249 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 38: cannot install the best candidate for the job 2024-04-14T15:02:09.249 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.250 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.250 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.250 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.250 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 39: conflicting requests 2024-04-14T15:02:09.250 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.250 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.250 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.250 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.250 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.250 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.250 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 40: cannot install the best candidate for the job 2024-04-14T15:02:09.250 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.250 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.251 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.251 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.251 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 41: cannot install the best candidate for the job 2024-04-14T15:02:09.251 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.251 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.251 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.251 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.251 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 42: cannot install the best candidate for the job 2024-04-14T15:02:09.251 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.251 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.251 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.251 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.251 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 43: cannot install the best candidate for the job 2024-04-14T15:02:09.252 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.252 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.252 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.252 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.252 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 44: cannot install the best candidate for the job 2024-04-14T15:02:09.252 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.252 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.252 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.252 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.252 INFO:teuthology.orchestra.run.smithi092.stderr: Problem 45: cannot install the best candidate for the job 2024-04-14T15:02:09.252 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.252 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.252 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.252 INFO:teuthology.orchestra.run.smithi092.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-14T15:02:09.313 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2024-04-14T15:02:09.314 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-04-14T15:02:09.314 INFO:teuthology.orchestra.run.smithi092.stdout: Package Architecture Version Repository Size 2024-04-14T15:02:09.314 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-04-14T15:02:09.314 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2024-04-14T15:02:09.314 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-nbd x86_64 2:19.0.0-2778.ga971fd5f.el9 ceph 170 k 2024-04-14T15:02:09.314 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:02:09.314 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2024-04-14T15:02:09.315 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-04-14T15:02:09.315 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2024-04-14T15:02:09.315 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:02:09.315 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 170 k 2024-04-14T15:02:09.315 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 494 k 2024-04-14T15:02:09.315 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2024-04-14T15:02:09.685 INFO:teuthology.orchestra.run.smithi092.stdout:rbd-nbd-19.0.0-2778.ga971fd5f.el9.x86_64.rpm 461 kB/s | 170 kB 00:00 2024-04-14T15:02:09.686 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2024-04-14T15:02:09.686 INFO:teuthology.orchestra.run.smithi092.stdout:Total 459 kB/s | 170 kB 00:00 2024-04-14T15:02:09.686 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2024-04-14T15:02:09.694 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2024-04-14T15:02:09.694 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2024-04-14T15:02:09.755 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2024-04-14T15:02:09.755 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2024-04-14T15:02:09.908 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2024-04-14T15:02:10.017 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : rbd-nbd-2:19.0.0-2778.ga971fd5f.el9.x86_64 1/1 2024-04-14T15:02:10.446 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: rbd-nbd-2:19.0.0-2778.ga971fd5f.el9.x86_64 1/1 2024-04-14T15:02:10.761 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : rbd-nbd-2:19.0.0-2778.ga971fd5f.el9.x86_64 1/1 2024-04-14T15:02:10.761 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:02:10.761 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2024-04-14T15:02:10.761 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-nbd-2:19.0.0-2778.ga971fd5f.el9.x86_64 2024-04-14T15:02:10.761 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:02:10.761 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2024-04-14T15:02:11.005 DEBUG:teuthology.parallel:result is None 2024-04-14T15:02:15.021 INFO:teuthology.orchestra.run.smithi090.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 492 kB/s | 6.0 MB 00:12 2024-04-14T15:02:15.022 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-04-14T15:02:15.022 INFO:teuthology.orchestra.run.smithi090.stdout:Total 563 kB/s | 7.0 MB 00:12 2024-04-14T15:02:15.095 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-04-14T15:02:15.108 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-04-14T15:02:15.108 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-04-14T15:02:15.212 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-04-14T15:02:15.213 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-04-14T15:02:15.519 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-04-14T15:02:15.686 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-14T15:02:15.748 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-14T15:02:16.268 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-14T15:02:16.361 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-14T15:02:16.432 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-14T15:02:16.462 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2778.ga971fd5f.el9.noarch 6/6 2024-04-14T15:02:17.452 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2778.ga971fd5f.el9.noarch 6/6 2024-04-14T15:02:17.452 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2778.ga971fd5f.el9.noarch 1/6 2024-04-14T15:02:17.452 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-14T15:02:17.452 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-14T15:02:17.452 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-14T15:02:17.453 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-14T15:02:17.771 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-14T15:02:17.771 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:02:17.771 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-04-14T15:02:17.771 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-cephadm-2:19.0.0-2778.ga971fd5f.el9.noarch 2024-04-14T15:02:17.771 INFO:teuthology.orchestra.run.smithi090.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-14T15:02:17.771 INFO:teuthology.orchestra.run.smithi090.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-14T15:02:17.771 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-14T15:02:17.771 INFO:teuthology.orchestra.run.smithi090.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-14T15:02:17.772 INFO:teuthology.orchestra.run.smithi090.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-14T15:02:17.772 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:02:17.772 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-04-14T15:02:18.021 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-fuse 2024-04-14T15:02:18.609 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:02:20 ago on Sun 14 Apr 2024 02:59:58 PM UTC. 2024-04-14T15:02:18.899 INFO:teuthology.orchestra.run.smithi090.stderr:Modular dependency problems: 2024-04-14T15:02:18.899 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-04-14T15:02:18.899 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 1: conflicting requests 2024-04-14T15:02:18.899 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.899 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 2: conflicting requests 2024-04-14T15:02:18.900 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.900 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 3: conflicting requests 2024-04-14T15:02:18.900 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.900 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 4: conflicting requests 2024-04-14T15:02:18.900 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.900 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 5: conflicting requests 2024-04-14T15:02:18.900 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.900 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 6: conflicting requests 2024-04-14T15:02:18.900 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.900 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 7: conflicting requests 2024-04-14T15:02:18.900 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.900 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 8: conflicting requests 2024-04-14T15:02:18.900 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.901 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 9: conflicting requests 2024-04-14T15:02:18.901 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.901 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 10: conflicting requests 2024-04-14T15:02:18.901 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.901 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 11: conflicting requests 2024-04-14T15:02:18.901 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.901 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 12: conflicting requests 2024-04-14T15:02:18.901 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.901 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 13: conflicting requests 2024-04-14T15:02:18.901 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.902 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 14: conflicting requests 2024-04-14T15:02:18.902 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.902 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 15: conflicting requests 2024-04-14T15:02:18.902 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.902 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 16: conflicting requests 2024-04-14T15:02:18.902 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.902 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 17: conflicting requests 2024-04-14T15:02:18.902 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.902 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 18: conflicting requests 2024-04-14T15:02:18.902 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.902 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 19: conflicting requests 2024-04-14T15:02:18.902 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.902 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 20: conflicting requests 2024-04-14T15:02:18.903 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.903 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 21: conflicting requests 2024-04-14T15:02:18.903 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.903 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 22: conflicting requests 2024-04-14T15:02:18.903 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.903 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 23: conflicting requests 2024-04-14T15:02:18.903 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.903 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 24: conflicting requests 2024-04-14T15:02:18.903 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.903 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 25: conflicting requests 2024-04-14T15:02:18.903 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.903 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 26: conflicting requests 2024-04-14T15:02:18.904 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.904 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 27: conflicting requests 2024-04-14T15:02:18.904 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.904 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 28: conflicting requests 2024-04-14T15:02:18.904 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.904 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 29: conflicting requests 2024-04-14T15:02:18.904 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.904 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 30: conflicting requests 2024-04-14T15:02:18.904 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.904 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 31: conflicting requests 2024-04-14T15:02:18.904 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.904 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 32: conflicting requests 2024-04-14T15:02:18.904 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.905 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 33: conflicting requests 2024-04-14T15:02:18.905 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.905 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 34: conflicting requests 2024-04-14T15:02:18.905 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.905 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 35: conflicting requests 2024-04-14T15:02:18.905 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.905 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 36: conflicting requests 2024-04-14T15:02:18.905 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.905 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 37: conflicting requests 2024-04-14T15:02:18.905 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.905 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.905 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.905 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 38: cannot install the best candidate for the job 2024-04-14T15:02:18.905 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.906 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.906 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.906 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.906 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 39: conflicting requests 2024-04-14T15:02:18.906 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.906 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.906 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.906 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.906 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.906 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.906 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 40: cannot install the best candidate for the job 2024-04-14T15:02:18.906 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.906 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.907 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.907 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.907 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 41: cannot install the best candidate for the job 2024-04-14T15:02:18.907 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.907 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.907 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.907 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.907 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 42: cannot install the best candidate for the job 2024-04-14T15:02:18.907 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.907 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.907 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.907 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.907 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 43: cannot install the best candidate for the job 2024-04-14T15:02:18.908 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.908 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.908 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.908 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.908 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 44: cannot install the best candidate for the job 2024-04-14T15:02:18.908 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.908 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.908 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.908 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.908 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 45: cannot install the best candidate for the job 2024-04-14T15:02:18.908 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.908 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.908 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.909 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-14T15:02:18.969 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-04-14T15:02:18.970 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-04-14T15:02:18.970 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-04-14T15:02:18.970 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-04-14T15:02:18.970 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-04-14T15:02:18.971 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-fuse x86_64 2:19.0.0-2778.ga971fd5f.el9 ceph 891 k 2024-04-14T15:02:18.971 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-04-14T15:02:18.971 INFO:teuthology.orchestra.run.smithi090.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-14T15:02:18.971 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:02:18.971 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-04-14T15:02:18.971 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-04-14T15:02:18.971 INFO:teuthology.orchestra.run.smithi090.stdout:Install 2 Packages 2024-04-14T15:02:18.971 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:02:18.971 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 971 k 2024-04-14T15:02:18.972 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 2.8 M 2024-04-14T15:02:18.972 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-04-14T15:02:19.377 INFO:teuthology.orchestra.run.smithi090.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 604 kB/s | 80 kB 00:00 2024-04-14T15:02:19.577 INFO:teuthology.orchestra.run.smithi090.stdout:(2/2): ceph-fuse-19.0.0-2778.ga971fd5f.el9.x86_ 2.6 MB/s | 891 kB 00:00 2024-04-14T15:02:19.578 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-04-14T15:02:19.578 INFO:teuthology.orchestra.run.smithi090.stdout:Total 1.6 MB/s | 971 kB 00:00 2024-04-14T15:02:19.590 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-04-14T15:02:19.610 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-04-14T15:02:19.611 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-04-14T15:02:19.688 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-04-14T15:02:19.688 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-04-14T15:02:19.872 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-04-14T15:02:20.022 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-14T15:02:20.103 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-fuse-2:19.0.0-2778.ga971fd5f.el9.x86_64 2/2 2024-04-14T15:02:20.837 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-fuse-2:19.0.0-2778.ga971fd5f.el9.x86_64 2/2 2024-04-14T15:02:20.837 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-fuse-2:19.0.0-2778.ga971fd5f.el9.x86_64 1/2 2024-04-14T15:02:21.172 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-14T15:02:21.172 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:02:21.172 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-04-14T15:02:21.172 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-fuse-2:19.0.0-2778.ga971fd5f.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-14T15:02:21.172 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:02:21.172 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-04-14T15:02:21.525 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-volume 2024-04-14T15:02:22.131 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:02:24 ago on Sun 14 Apr 2024 02:59:58 PM UTC. 2024-04-14T15:02:22.418 INFO:teuthology.orchestra.run.smithi090.stderr:Modular dependency problems: 2024-04-14T15:02:22.418 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-04-14T15:02:22.418 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 1: conflicting requests 2024-04-14T15:02:22.418 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.418 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 2: conflicting requests 2024-04-14T15:02:22.418 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.418 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 3: conflicting requests 2024-04-14T15:02:22.419 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.419 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 4: conflicting requests 2024-04-14T15:02:22.419 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.419 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 5: conflicting requests 2024-04-14T15:02:22.419 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.419 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 6: conflicting requests 2024-04-14T15:02:22.419 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.419 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 7: conflicting requests 2024-04-14T15:02:22.419 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.419 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 8: conflicting requests 2024-04-14T15:02:22.419 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.419 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 9: conflicting requests 2024-04-14T15:02:22.419 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.420 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 10: conflicting requests 2024-04-14T15:02:22.420 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.420 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 11: conflicting requests 2024-04-14T15:02:22.420 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.420 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 12: conflicting requests 2024-04-14T15:02:22.420 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.420 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 13: conflicting requests 2024-04-14T15:02:22.420 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.420 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 14: conflicting requests 2024-04-14T15:02:22.420 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.420 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 15: conflicting requests 2024-04-14T15:02:22.420 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.421 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 16: conflicting requests 2024-04-14T15:02:22.421 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.421 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 17: conflicting requests 2024-04-14T15:02:22.421 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.421 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 18: conflicting requests 2024-04-14T15:02:22.421 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.421 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 19: conflicting requests 2024-04-14T15:02:22.421 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.421 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 20: conflicting requests 2024-04-14T15:02:22.421 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.421 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 21: conflicting requests 2024-04-14T15:02:22.421 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.421 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 22: conflicting requests 2024-04-14T15:02:22.421 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.421 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 23: conflicting requests 2024-04-14T15:02:22.422 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.422 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 24: conflicting requests 2024-04-14T15:02:22.422 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.422 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 25: conflicting requests 2024-04-14T15:02:22.422 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.422 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 26: conflicting requests 2024-04-14T15:02:22.422 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.422 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 27: conflicting requests 2024-04-14T15:02:22.422 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.422 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 28: conflicting requests 2024-04-14T15:02:22.422 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.422 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 29: conflicting requests 2024-04-14T15:02:22.422 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.423 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 30: conflicting requests 2024-04-14T15:02:22.423 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.423 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 31: conflicting requests 2024-04-14T15:02:22.423 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.423 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 32: conflicting requests 2024-04-14T15:02:22.423 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.423 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 33: conflicting requests 2024-04-14T15:02:22.423 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.423 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 34: conflicting requests 2024-04-14T15:02:22.423 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.423 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 35: conflicting requests 2024-04-14T15:02:22.423 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.423 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 36: conflicting requests 2024-04-14T15:02:22.423 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.424 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 37: conflicting requests 2024-04-14T15:02:22.424 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.424 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.424 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.424 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 38: cannot install the best candidate for the job 2024-04-14T15:02:22.424 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.424 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.424 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.424 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.424 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 39: conflicting requests 2024-04-14T15:02:22.424 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.424 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.424 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.425 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.425 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.425 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.425 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 40: cannot install the best candidate for the job 2024-04-14T15:02:22.425 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.425 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.425 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.425 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.425 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 41: cannot install the best candidate for the job 2024-04-14T15:02:22.425 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.425 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.425 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.425 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.425 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 42: cannot install the best candidate for the job 2024-04-14T15:02:22.426 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.426 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.426 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.426 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.426 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 43: cannot install the best candidate for the job 2024-04-14T15:02:22.426 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.426 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.426 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.426 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.426 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 44: cannot install the best candidate for the job 2024-04-14T15:02:22.426 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.426 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.427 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.427 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.427 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 45: cannot install the best candidate for the job 2024-04-14T15:02:22.427 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.427 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.427 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.427 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-14T15:02:22.486 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-04-14T15:02:22.487 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-04-14T15:02:22.487 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-04-14T15:02:22.487 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-04-14T15:02:22.487 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-04-14T15:02:22.488 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-volume noarch 2:19.0.0-2778.ga971fd5f.el9 ceph-noarch 264 k 2024-04-14T15:02:22.488 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-04-14T15:02:22.488 INFO:teuthology.orchestra.run.smithi090.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-14T15:02:22.488 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:02:22.488 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-04-14T15:02:22.488 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-04-14T15:02:22.488 INFO:teuthology.orchestra.run.smithi090.stdout:Install 2 Packages 2024-04-14T15:02:22.488 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:02:22.489 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 341 k 2024-04-14T15:02:22.489 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 1.5 M 2024-04-14T15:02:22.489 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-04-14T15:02:22.772 INFO:teuthology.orchestra.run.smithi090.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 383 kB/s | 77 kB 00:00 2024-04-14T15:02:22.847 INFO:teuthology.orchestra.run.smithi090.stdout:(2/2): ceph-volume-19.0.0-2778.ga971fd5f.el9.no 954 kB/s | 264 kB 00:00 2024-04-14T15:02:22.848 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-04-14T15:02:22.848 INFO:teuthology.orchestra.run.smithi090.stdout:Total 947 kB/s | 341 kB 00:00 2024-04-14T15:02:22.859 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-04-14T15:02:22.869 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-04-14T15:02:22.869 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-04-14T15:02:22.909 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-04-14T15:02:22.910 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-04-14T15:02:23.069 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-04-14T15:02:23.254 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-04-14T15:02:23.292 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-volume-2:19.0.0-2778.ga971fd5f.el9.noarch 2/2 2024-04-14T15:02:23.308 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-volume-2:19.0.0-2778.ga971fd5f.el9.noarch 2/2 2024-04-14T15:02:23.308 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-14T15:02:23.308 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-14T15:02:23.308 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:02:24.274 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-volume-2:19.0.0-2778.ga971fd5f.el9.noarch 1/2 2024-04-14T15:02:24.628 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-04-14T15:02:24.628 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:02:24.628 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-04-14T15:02:24.628 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-volume-2:19.0.0-2778.ga971fd5f.el9.noarch 2024-04-14T15:02:24.628 INFO:teuthology.orchestra.run.smithi090.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-14T15:02:24.628 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:02:24.628 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-04-14T15:02:25.015 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install librados-devel 2024-04-14T15:02:25.623 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:02:27 ago on Sun 14 Apr 2024 02:59:58 PM UTC. 2024-04-14T15:02:25.910 INFO:teuthology.orchestra.run.smithi090.stderr:Modular dependency problems: 2024-04-14T15:02:25.910 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-04-14T15:02:25.910 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 1: conflicting requests 2024-04-14T15:02:25.910 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.911 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 2: conflicting requests 2024-04-14T15:02:25.911 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.911 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 3: conflicting requests 2024-04-14T15:02:25.911 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.911 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 4: conflicting requests 2024-04-14T15:02:25.911 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.911 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 5: conflicting requests 2024-04-14T15:02:25.911 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.911 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 6: conflicting requests 2024-04-14T15:02:25.911 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.911 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 7: conflicting requests 2024-04-14T15:02:25.911 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.912 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 8: conflicting requests 2024-04-14T15:02:25.912 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.912 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 9: conflicting requests 2024-04-14T15:02:25.912 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.912 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 10: conflicting requests 2024-04-14T15:02:25.912 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.912 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 11: conflicting requests 2024-04-14T15:02:25.912 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.912 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 12: conflicting requests 2024-04-14T15:02:25.912 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.912 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 13: conflicting requests 2024-04-14T15:02:25.912 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.913 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 14: conflicting requests 2024-04-14T15:02:25.913 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.913 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 15: conflicting requests 2024-04-14T15:02:25.913 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.913 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 16: conflicting requests 2024-04-14T15:02:25.913 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.913 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 17: conflicting requests 2024-04-14T15:02:25.913 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.913 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 18: conflicting requests 2024-04-14T15:02:25.913 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.913 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 19: conflicting requests 2024-04-14T15:02:25.913 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.914 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 20: conflicting requests 2024-04-14T15:02:25.914 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.914 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 21: conflicting requests 2024-04-14T15:02:25.914 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.914 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 22: conflicting requests 2024-04-14T15:02:25.914 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.914 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 23: conflicting requests 2024-04-14T15:02:25.914 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.915 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 24: conflicting requests 2024-04-14T15:02:25.915 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.915 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 25: conflicting requests 2024-04-14T15:02:25.915 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.915 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 26: conflicting requests 2024-04-14T15:02:25.915 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.915 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 27: conflicting requests 2024-04-14T15:02:25.915 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.915 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 28: conflicting requests 2024-04-14T15:02:25.915 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.916 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 29: conflicting requests 2024-04-14T15:02:25.916 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.916 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 30: conflicting requests 2024-04-14T15:02:25.916 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.916 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 31: conflicting requests 2024-04-14T15:02:25.916 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.916 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 32: conflicting requests 2024-04-14T15:02:25.916 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.916 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 33: conflicting requests 2024-04-14T15:02:25.916 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.916 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 34: conflicting requests 2024-04-14T15:02:25.916 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.917 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 35: conflicting requests 2024-04-14T15:02:25.917 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.917 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 36: conflicting requests 2024-04-14T15:02:25.917 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.917 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 37: conflicting requests 2024-04-14T15:02:25.917 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.917 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.917 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.917 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 38: cannot install the best candidate for the job 2024-04-14T15:02:25.917 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.917 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.917 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.918 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.918 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 39: conflicting requests 2024-04-14T15:02:25.918 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.918 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.918 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.918 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.918 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.918 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.918 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 40: cannot install the best candidate for the job 2024-04-14T15:02:25.918 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.918 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.919 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.919 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.919 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 41: cannot install the best candidate for the job 2024-04-14T15:02:25.919 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.919 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.919 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.919 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.919 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 42: cannot install the best candidate for the job 2024-04-14T15:02:25.919 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.919 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.919 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.919 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.920 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 43: cannot install the best candidate for the job 2024-04-14T15:02:25.920 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.920 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.920 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.920 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.920 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 44: cannot install the best candidate for the job 2024-04-14T15:02:25.920 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.920 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.920 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.920 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.920 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 45: cannot install the best candidate for the job 2024-04-14T15:02:25.920 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.921 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.921 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.921 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-14T15:02:25.979 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-04-14T15:02:25.980 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-04-14T15:02:25.980 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2024-04-14T15:02:25.980 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-04-14T15:02:25.980 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-04-14T15:02:25.980 INFO:teuthology.orchestra.run.smithi090.stdout: librados-devel x86_64 2:19.0.0-2778.ga971fd5f.el9 ceph 127 k 2024-04-14T15:02:25.980 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:02:25.980 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-04-14T15:02:25.981 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-04-14T15:02:25.981 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-04-14T15:02:25.981 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:02:25.981 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 127 k 2024-04-14T15:02:25.981 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 456 k 2024-04-14T15:02:25.981 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-04-14T15:02:26.300 INFO:teuthology.orchestra.run.smithi090.stdout:librados-devel-19.0.0-2778.ga971fd5f.el9.x86_64 397 kB/s | 127 kB 00:00 2024-04-14T15:02:26.301 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-04-14T15:02:26.301 INFO:teuthology.orchestra.run.smithi090.stdout:Total 395 kB/s | 127 kB 00:00 2024-04-14T15:02:26.301 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-04-14T15:02:26.307 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-04-14T15:02:26.307 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-04-14T15:02:26.368 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-04-14T15:02:26.368 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-04-14T15:02:26.555 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-04-14T15:02:26.658 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librados-devel-2:19.0.0-2778.ga971fd5f.el9.x86_64 1/1 2024-04-14T15:02:27.175 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librados-devel-2:19.0.0-2778.ga971fd5f.el9.x86_64 1/1 2024-04-14T15:02:27.480 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librados-devel-2:19.0.0-2778.ga971fd5f.el9.x86_64 1/1 2024-04-14T15:02:27.480 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:02:27.480 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-04-14T15:02:27.480 INFO:teuthology.orchestra.run.smithi090.stdout: librados-devel-2:19.0.0-2778.ga971fd5f.el9.x86_64 2024-04-14T15:02:27.480 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:02:27.480 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-04-14T15:02:27.746 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install libcephfs2 2024-04-14T15:02:28.353 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:02:30 ago on Sun 14 Apr 2024 02:59:58 PM UTC. 2024-04-14T15:02:28.647 INFO:teuthology.orchestra.run.smithi090.stderr:Modular dependency problems: 2024-04-14T15:02:28.648 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-04-14T15:02:28.648 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 1: conflicting requests 2024-04-14T15:02:28.648 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.648 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 2: conflicting requests 2024-04-14T15:02:28.648 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.648 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 3: conflicting requests 2024-04-14T15:02:28.648 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.649 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 4: conflicting requests 2024-04-14T15:02:28.649 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.649 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 5: conflicting requests 2024-04-14T15:02:28.649 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.649 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 6: conflicting requests 2024-04-14T15:02:28.649 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.649 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 7: conflicting requests 2024-04-14T15:02:28.649 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.649 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 8: conflicting requests 2024-04-14T15:02:28.649 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.649 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 9: conflicting requests 2024-04-14T15:02:28.649 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.649 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 10: conflicting requests 2024-04-14T15:02:28.649 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.650 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 11: conflicting requests 2024-04-14T15:02:28.650 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.650 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 12: conflicting requests 2024-04-14T15:02:28.650 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.650 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 13: conflicting requests 2024-04-14T15:02:28.650 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.650 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 14: conflicting requests 2024-04-14T15:02:28.650 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.650 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 15: conflicting requests 2024-04-14T15:02:28.650 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.650 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 16: conflicting requests 2024-04-14T15:02:28.650 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.650 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 17: conflicting requests 2024-04-14T15:02:28.650 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.651 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 18: conflicting requests 2024-04-14T15:02:28.651 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.651 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 19: conflicting requests 2024-04-14T15:02:28.651 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.651 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 20: conflicting requests 2024-04-14T15:02:28.651 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.651 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 21: conflicting requests 2024-04-14T15:02:28.651 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.651 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 22: conflicting requests 2024-04-14T15:02:28.651 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.651 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 23: conflicting requests 2024-04-14T15:02:28.651 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.651 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 24: conflicting requests 2024-04-14T15:02:28.651 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.652 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 25: conflicting requests 2024-04-14T15:02:28.652 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.652 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 26: conflicting requests 2024-04-14T15:02:28.652 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.652 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 27: conflicting requests 2024-04-14T15:02:28.652 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.652 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 28: conflicting requests 2024-04-14T15:02:28.652 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.652 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 29: conflicting requests 2024-04-14T15:02:28.652 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.652 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 30: conflicting requests 2024-04-14T15:02:28.652 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.653 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 31: conflicting requests 2024-04-14T15:02:28.653 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.653 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 32: conflicting requests 2024-04-14T15:02:28.653 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.653 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 33: conflicting requests 2024-04-14T15:02:28.653 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.653 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 34: conflicting requests 2024-04-14T15:02:28.653 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.653 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 35: conflicting requests 2024-04-14T15:02:28.653 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.653 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 36: conflicting requests 2024-04-14T15:02:28.653 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.654 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 37: conflicting requests 2024-04-14T15:02:28.654 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.654 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.654 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.654 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 38: cannot install the best candidate for the job 2024-04-14T15:02:28.654 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.654 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.654 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.654 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.654 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 39: conflicting requests 2024-04-14T15:02:28.654 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.655 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.655 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.655 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.655 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.655 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.655 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 40: cannot install the best candidate for the job 2024-04-14T15:02:28.656 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.656 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.656 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.656 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.656 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 41: cannot install the best candidate for the job 2024-04-14T15:02:28.656 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.656 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.657 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.657 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.657 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 42: cannot install the best candidate for the job 2024-04-14T15:02:28.658 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.658 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.658 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.658 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.658 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 43: cannot install the best candidate for the job 2024-04-14T15:02:28.658 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.658 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.659 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.659 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.659 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 44: cannot install the best candidate for the job 2024-04-14T15:02:28.659 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.659 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.659 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.659 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.659 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 45: cannot install the best candidate for the job 2024-04-14T15:02:28.660 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.660 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.660 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.660 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-14T15:02:28.660 INFO:teuthology.orchestra.run.smithi090.stdout:Package libcephfs2-2:19.0.0-2778.ga971fd5f.el9.x86_64 is already installed. 2024-04-14T15:02:28.716 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-04-14T15:02:28.717 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-04-14T15:02:28.717 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-04-14T15:02:28.777 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install libcephfs-devel 2024-04-14T15:02:29.372 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:02:31 ago on Sun 14 Apr 2024 02:59:58 PM UTC. 2024-04-14T15:02:29.659 INFO:teuthology.orchestra.run.smithi090.stderr:Modular dependency problems: 2024-04-14T15:02:29.660 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-04-14T15:02:29.660 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 1: conflicting requests 2024-04-14T15:02:29.660 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.660 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 2: conflicting requests 2024-04-14T15:02:29.660 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.660 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 3: conflicting requests 2024-04-14T15:02:29.660 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.660 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 4: conflicting requests 2024-04-14T15:02:29.660 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.660 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 5: conflicting requests 2024-04-14T15:02:29.661 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.661 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 6: conflicting requests 2024-04-14T15:02:29.661 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.661 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 7: conflicting requests 2024-04-14T15:02:29.661 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.661 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 8: conflicting requests 2024-04-14T15:02:29.661 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.661 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 9: conflicting requests 2024-04-14T15:02:29.661 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.661 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 10: conflicting requests 2024-04-14T15:02:29.661 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.661 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 11: conflicting requests 2024-04-14T15:02:29.661 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.662 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 12: conflicting requests 2024-04-14T15:02:29.662 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.662 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 13: conflicting requests 2024-04-14T15:02:29.662 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.662 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 14: conflicting requests 2024-04-14T15:02:29.662 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.662 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 15: conflicting requests 2024-04-14T15:02:29.662 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.662 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 16: conflicting requests 2024-04-14T15:02:29.662 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.662 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 17: conflicting requests 2024-04-14T15:02:29.663 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.663 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 18: conflicting requests 2024-04-14T15:02:29.663 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.663 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 19: conflicting requests 2024-04-14T15:02:29.663 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.663 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 20: conflicting requests 2024-04-14T15:02:29.663 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.663 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 21: conflicting requests 2024-04-14T15:02:29.663 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.663 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 22: conflicting requests 2024-04-14T15:02:29.663 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.663 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 23: conflicting requests 2024-04-14T15:02:29.663 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.664 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 24: conflicting requests 2024-04-14T15:02:29.664 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.664 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 25: conflicting requests 2024-04-14T15:02:29.664 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.664 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 26: conflicting requests 2024-04-14T15:02:29.664 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.664 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 27: conflicting requests 2024-04-14T15:02:29.664 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.664 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 28: conflicting requests 2024-04-14T15:02:29.664 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.664 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 29: conflicting requests 2024-04-14T15:02:29.664 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.664 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 30: conflicting requests 2024-04-14T15:02:29.665 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.665 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 31: conflicting requests 2024-04-14T15:02:29.665 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.665 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 32: conflicting requests 2024-04-14T15:02:29.665 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.665 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 33: conflicting requests 2024-04-14T15:02:29.665 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.665 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 34: conflicting requests 2024-04-14T15:02:29.665 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.665 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 35: conflicting requests 2024-04-14T15:02:29.665 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.665 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 36: conflicting requests 2024-04-14T15:02:29.665 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.666 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 37: conflicting requests 2024-04-14T15:02:29.666 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.666 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.666 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.666 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 38: cannot install the best candidate for the job 2024-04-14T15:02:29.666 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.666 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.666 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.666 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.666 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 39: conflicting requests 2024-04-14T15:02:29.666 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.666 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.666 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.667 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.667 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.667 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.667 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 40: cannot install the best candidate for the job 2024-04-14T15:02:29.667 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.667 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.667 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.667 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.667 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 41: cannot install the best candidate for the job 2024-04-14T15:02:29.667 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.667 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.667 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.668 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.668 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 42: cannot install the best candidate for the job 2024-04-14T15:02:29.668 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.668 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.668 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.668 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.668 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 43: cannot install the best candidate for the job 2024-04-14T15:02:29.668 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.668 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.668 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.668 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.668 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 44: cannot install the best candidate for the job 2024-04-14T15:02:29.668 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.668 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.669 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.669 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.669 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 45: cannot install the best candidate for the job 2024-04-14T15:02:29.669 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.669 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.669 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.669 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-14T15:02:29.728 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-04-14T15:02:29.729 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-04-14T15:02:29.729 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2024-04-14T15:02:29.729 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-04-14T15:02:29.729 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-04-14T15:02:29.729 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs-devel x86_64 2:19.0.0-2778.ga971fd5f.el9 ceph 32 k 2024-04-14T15:02:29.729 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:02:29.729 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-04-14T15:02:29.729 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-04-14T15:02:29.729 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-04-14T15:02:29.729 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:02:29.729 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 32 k 2024-04-14T15:02:29.730 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 142 k 2024-04-14T15:02:29.730 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-04-14T15:02:29.968 INFO:teuthology.orchestra.run.smithi090.stdout:libcephfs-devel-19.0.0-2778.ga971fd5f.el9.x86_6 133 kB/s | 32 kB 00:00 2024-04-14T15:02:29.968 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-04-14T15:02:29.969 INFO:teuthology.orchestra.run.smithi090.stdout:Total 132 kB/s | 32 kB 00:00 2024-04-14T15:02:29.969 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-04-14T15:02:29.972 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-04-14T15:02:29.972 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-04-14T15:02:30.006 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-04-14T15:02:30.007 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-04-14T15:02:30.107 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-04-14T15:02:30.233 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libcephfs-devel-2:19.0.0-2778.ga971fd5f.el9.x86_64 1/1 2024-04-14T15:02:30.712 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-2778.ga971fd5f.el9.x86_64 1/1 2024-04-14T15:02:30.982 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libcephfs-devel-2:19.0.0-2778.ga971fd5f.el9.x86_64 1/1 2024-04-14T15:02:30.983 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:02:30.983 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-04-14T15:02:30.983 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs-devel-2:19.0.0-2778.ga971fd5f.el9.x86_64 2024-04-14T15:02:30.983 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:02:30.983 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-04-14T15:02:31.180 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install librados2 2024-04-14T15:02:31.779 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:02:33 ago on Sun 14 Apr 2024 02:59:58 PM UTC. 2024-04-14T15:02:32.066 INFO:teuthology.orchestra.run.smithi090.stderr:Modular dependency problems: 2024-04-14T15:02:32.066 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-04-14T15:02:32.066 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 1: conflicting requests 2024-04-14T15:02:32.066 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.066 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 2: conflicting requests 2024-04-14T15:02:32.066 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.067 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 3: conflicting requests 2024-04-14T15:02:32.067 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.067 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 4: conflicting requests 2024-04-14T15:02:32.067 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.067 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 5: conflicting requests 2024-04-14T15:02:32.067 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.067 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 6: conflicting requests 2024-04-14T15:02:32.067 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.067 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 7: conflicting requests 2024-04-14T15:02:32.067 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.067 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 8: conflicting requests 2024-04-14T15:02:32.067 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.067 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 9: conflicting requests 2024-04-14T15:02:32.068 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.068 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 10: conflicting requests 2024-04-14T15:02:32.068 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.068 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 11: conflicting requests 2024-04-14T15:02:32.068 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.068 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 12: conflicting requests 2024-04-14T15:02:32.068 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.068 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 13: conflicting requests 2024-04-14T15:02:32.068 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.068 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 14: conflicting requests 2024-04-14T15:02:32.068 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.068 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 15: conflicting requests 2024-04-14T15:02:32.068 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.069 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 16: conflicting requests 2024-04-14T15:02:32.069 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.069 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 17: conflicting requests 2024-04-14T15:02:32.069 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.069 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 18: conflicting requests 2024-04-14T15:02:32.069 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.069 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 19: conflicting requests 2024-04-14T15:02:32.069 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.069 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 20: conflicting requests 2024-04-14T15:02:32.069 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.069 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 21: conflicting requests 2024-04-14T15:02:32.069 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.069 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 22: conflicting requests 2024-04-14T15:02:32.070 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.070 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 23: conflicting requests 2024-04-14T15:02:32.070 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.070 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 24: conflicting requests 2024-04-14T15:02:32.070 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.070 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 25: conflicting requests 2024-04-14T15:02:32.070 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.070 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 26: conflicting requests 2024-04-14T15:02:32.070 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.070 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 27: conflicting requests 2024-04-14T15:02:32.070 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.070 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 28: conflicting requests 2024-04-14T15:02:32.070 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.070 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 29: conflicting requests 2024-04-14T15:02:32.071 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.071 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 30: conflicting requests 2024-04-14T15:02:32.071 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.071 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 31: conflicting requests 2024-04-14T15:02:32.071 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.071 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 32: conflicting requests 2024-04-14T15:02:32.071 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.071 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 33: conflicting requests 2024-04-14T15:02:32.071 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.071 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 34: conflicting requests 2024-04-14T15:02:32.071 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.071 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 35: conflicting requests 2024-04-14T15:02:32.071 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.072 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 36: conflicting requests 2024-04-14T15:02:32.072 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.072 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 37: conflicting requests 2024-04-14T15:02:32.072 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.072 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.072 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.072 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 38: cannot install the best candidate for the job 2024-04-14T15:02:32.072 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.072 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.072 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.072 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.072 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 39: conflicting requests 2024-04-14T15:02:32.072 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.073 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.073 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.073 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.073 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.073 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.073 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 40: cannot install the best candidate for the job 2024-04-14T15:02:32.073 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.073 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.073 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.073 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.073 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 41: cannot install the best candidate for the job 2024-04-14T15:02:32.073 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.073 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.073 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.074 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.074 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 42: cannot install the best candidate for the job 2024-04-14T15:02:32.074 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.074 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.074 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.074 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.074 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 43: cannot install the best candidate for the job 2024-04-14T15:02:32.074 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.074 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.074 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.074 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.074 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 44: cannot install the best candidate for the job 2024-04-14T15:02:32.075 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.075 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.075 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.075 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.075 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 45: cannot install the best candidate for the job 2024-04-14T15:02:32.075 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.075 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.075 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.075 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-14T15:02:32.075 INFO:teuthology.orchestra.run.smithi090.stdout:Package librados2-2:19.0.0-2778.ga971fd5f.el9.x86_64 is already installed. 2024-04-14T15:02:32.134 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-04-14T15:02:32.135 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-04-14T15:02:32.135 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-04-14T15:02:32.223 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install librbd1 2024-04-14T15:02:32.814 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:02:34 ago on Sun 14 Apr 2024 02:59:58 PM UTC. 2024-04-14T15:02:33.103 INFO:teuthology.orchestra.run.smithi090.stderr:Modular dependency problems: 2024-04-14T15:02:33.103 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-04-14T15:02:33.104 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 1: conflicting requests 2024-04-14T15:02:33.104 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.104 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 2: conflicting requests 2024-04-14T15:02:33.104 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.104 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 3: conflicting requests 2024-04-14T15:02:33.104 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.104 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 4: conflicting requests 2024-04-14T15:02:33.104 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.104 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 5: conflicting requests 2024-04-14T15:02:33.104 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.105 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 6: conflicting requests 2024-04-14T15:02:33.105 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.105 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 7: conflicting requests 2024-04-14T15:02:33.105 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.105 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 8: conflicting requests 2024-04-14T15:02:33.105 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.105 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 9: conflicting requests 2024-04-14T15:02:33.105 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.105 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 10: conflicting requests 2024-04-14T15:02:33.105 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.105 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 11: conflicting requests 2024-04-14T15:02:33.105 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.106 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 12: conflicting requests 2024-04-14T15:02:33.106 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.106 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 13: conflicting requests 2024-04-14T15:02:33.106 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.106 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 14: conflicting requests 2024-04-14T15:02:33.106 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.106 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 15: conflicting requests 2024-04-14T15:02:33.106 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.106 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 16: conflicting requests 2024-04-14T15:02:33.106 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.107 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 17: conflicting requests 2024-04-14T15:02:33.107 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.107 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 18: conflicting requests 2024-04-14T15:02:33.107 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.107 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 19: conflicting requests 2024-04-14T15:02:33.107 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.107 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 20: conflicting requests 2024-04-14T15:02:33.107 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.107 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 21: conflicting requests 2024-04-14T15:02:33.107 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.107 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 22: conflicting requests 2024-04-14T15:02:33.107 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.108 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 23: conflicting requests 2024-04-14T15:02:33.108 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.108 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 24: conflicting requests 2024-04-14T15:02:33.108 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.108 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 25: conflicting requests 2024-04-14T15:02:33.108 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.108 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 26: conflicting requests 2024-04-14T15:02:33.108 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.108 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 27: conflicting requests 2024-04-14T15:02:33.108 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.108 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 28: conflicting requests 2024-04-14T15:02:33.108 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.109 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 29: conflicting requests 2024-04-14T15:02:33.109 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.109 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 30: conflicting requests 2024-04-14T15:02:33.109 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.109 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 31: conflicting requests 2024-04-14T15:02:33.109 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.109 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 32: conflicting requests 2024-04-14T15:02:33.109 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.109 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 33: conflicting requests 2024-04-14T15:02:33.109 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.109 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 34: conflicting requests 2024-04-14T15:02:33.109 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.109 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 35: conflicting requests 2024-04-14T15:02:33.110 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.110 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 36: conflicting requests 2024-04-14T15:02:33.110 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.110 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 37: conflicting requests 2024-04-14T15:02:33.110 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.110 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.110 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.110 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 38: cannot install the best candidate for the job 2024-04-14T15:02:33.110 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.110 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.110 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.111 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.111 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 39: conflicting requests 2024-04-14T15:02:33.111 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.111 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.111 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.111 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.111 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.111 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.111 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 40: cannot install the best candidate for the job 2024-04-14T15:02:33.111 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.112 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.112 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.112 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.112 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 41: cannot install the best candidate for the job 2024-04-14T15:02:33.112 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.112 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.112 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.112 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.112 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 42: cannot install the best candidate for the job 2024-04-14T15:02:33.112 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.112 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.113 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.113 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.113 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 43: cannot install the best candidate for the job 2024-04-14T15:02:33.113 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.113 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.113 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.113 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.113 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 44: cannot install the best candidate for the job 2024-04-14T15:02:33.113 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.113 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.113 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.114 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.114 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 45: cannot install the best candidate for the job 2024-04-14T15:02:33.114 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.114 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.114 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.114 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-14T15:02:33.114 INFO:teuthology.orchestra.run.smithi090.stdout:Package librbd1-2:19.0.0-2778.ga971fd5f.el9.x86_64 is already installed. 2024-04-14T15:02:33.171 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-04-14T15:02:33.172 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-04-14T15:02:33.173 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-04-14T15:02:33.258 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install python3-rados 2024-04-14T15:02:33.850 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:02:35 ago on Sun 14 Apr 2024 02:59:58 PM UTC. 2024-04-14T15:02:34.144 INFO:teuthology.orchestra.run.smithi090.stderr:Modular dependency problems: 2024-04-14T15:02:34.144 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-04-14T15:02:34.144 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 1: conflicting requests 2024-04-14T15:02:34.144 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.144 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 2: conflicting requests 2024-04-14T15:02:34.144 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.144 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 3: conflicting requests 2024-04-14T15:02:34.144 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.144 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 4: conflicting requests 2024-04-14T15:02:34.144 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.144 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 5: conflicting requests 2024-04-14T15:02:34.145 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.145 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 6: conflicting requests 2024-04-14T15:02:34.145 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.145 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 7: conflicting requests 2024-04-14T15:02:34.145 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.145 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 8: conflicting requests 2024-04-14T15:02:34.145 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.145 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 9: conflicting requests 2024-04-14T15:02:34.145 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.145 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 10: conflicting requests 2024-04-14T15:02:34.145 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.145 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 11: conflicting requests 2024-04-14T15:02:34.145 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.145 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 12: conflicting requests 2024-04-14T15:02:34.146 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.146 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 13: conflicting requests 2024-04-14T15:02:34.146 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.146 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 14: conflicting requests 2024-04-14T15:02:34.146 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.146 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 15: conflicting requests 2024-04-14T15:02:34.146 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.146 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 16: conflicting requests 2024-04-14T15:02:34.146 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.146 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 17: conflicting requests 2024-04-14T15:02:34.146 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.146 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 18: conflicting requests 2024-04-14T15:02:34.147 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.147 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 19: conflicting requests 2024-04-14T15:02:34.147 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.147 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 20: conflicting requests 2024-04-14T15:02:34.147 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.147 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 21: conflicting requests 2024-04-14T15:02:34.147 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.147 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 22: conflicting requests 2024-04-14T15:02:34.147 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.147 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 23: conflicting requests 2024-04-14T15:02:34.148 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.148 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 24: conflicting requests 2024-04-14T15:02:34.148 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.148 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 25: conflicting requests 2024-04-14T15:02:34.148 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.148 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 26: conflicting requests 2024-04-14T15:02:34.148 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.148 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 27: conflicting requests 2024-04-14T15:02:34.148 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.148 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 28: conflicting requests 2024-04-14T15:02:34.148 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.148 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 29: conflicting requests 2024-04-14T15:02:34.148 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.148 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 30: conflicting requests 2024-04-14T15:02:34.149 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.149 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 31: conflicting requests 2024-04-14T15:02:34.149 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.149 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 32: conflicting requests 2024-04-14T15:02:34.149 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.149 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 33: conflicting requests 2024-04-14T15:02:34.149 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.149 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 34: conflicting requests 2024-04-14T15:02:34.149 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.149 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 35: conflicting requests 2024-04-14T15:02:34.149 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.149 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 36: conflicting requests 2024-04-14T15:02:34.149 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.149 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 37: conflicting requests 2024-04-14T15:02:34.150 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.150 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.150 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.150 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 38: cannot install the best candidate for the job 2024-04-14T15:02:34.150 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.150 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.150 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.150 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.150 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 39: conflicting requests 2024-04-14T15:02:34.150 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.150 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.150 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.150 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.151 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.151 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.151 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 40: cannot install the best candidate for the job 2024-04-14T15:02:34.151 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.151 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.151 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.151 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.151 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 41: cannot install the best candidate for the job 2024-04-14T15:02:34.151 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.151 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.151 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.151 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.151 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 42: cannot install the best candidate for the job 2024-04-14T15:02:34.151 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.152 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.152 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.152 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.152 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 43: cannot install the best candidate for the job 2024-04-14T15:02:34.152 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.152 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.152 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.152 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.152 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 44: cannot install the best candidate for the job 2024-04-14T15:02:34.152 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.153 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.153 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.153 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.153 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 45: cannot install the best candidate for the job 2024-04-14T15:02:34.153 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.153 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.153 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.153 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-14T15:02:34.154 INFO:teuthology.orchestra.run.smithi090.stdout:Package python3-rados-2:19.0.0-2778.ga971fd5f.el9.x86_64 is already installed. 2024-04-14T15:02:34.212 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-04-14T15:02:34.213 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-04-14T15:02:34.213 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-04-14T15:02:34.296 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install python3-rgw 2024-04-14T15:02:34.897 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:02:36 ago on Sun 14 Apr 2024 02:59:58 PM UTC. 2024-04-14T15:02:35.185 INFO:teuthology.orchestra.run.smithi090.stderr:Modular dependency problems: 2024-04-14T15:02:35.185 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-04-14T15:02:35.185 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 1: conflicting requests 2024-04-14T15:02:35.185 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.186 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 2: conflicting requests 2024-04-14T15:02:35.186 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.186 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 3: conflicting requests 2024-04-14T15:02:35.186 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.186 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 4: conflicting requests 2024-04-14T15:02:35.186 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.186 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 5: conflicting requests 2024-04-14T15:02:35.186 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.186 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 6: conflicting requests 2024-04-14T15:02:35.186 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.186 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 7: conflicting requests 2024-04-14T15:02:35.186 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.186 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 8: conflicting requests 2024-04-14T15:02:35.186 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.187 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 9: conflicting requests 2024-04-14T15:02:35.187 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.187 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 10: conflicting requests 2024-04-14T15:02:35.187 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.187 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 11: conflicting requests 2024-04-14T15:02:35.187 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.187 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 12: conflicting requests 2024-04-14T15:02:35.187 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.187 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 13: conflicting requests 2024-04-14T15:02:35.187 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.187 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 14: conflicting requests 2024-04-14T15:02:35.187 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.187 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 15: conflicting requests 2024-04-14T15:02:35.187 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.187 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 16: conflicting requests 2024-04-14T15:02:35.187 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.188 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 17: conflicting requests 2024-04-14T15:02:35.188 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.188 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 18: conflicting requests 2024-04-14T15:02:35.188 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.188 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 19: conflicting requests 2024-04-14T15:02:35.188 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.188 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 20: conflicting requests 2024-04-14T15:02:35.188 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.188 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 21: conflicting requests 2024-04-14T15:02:35.188 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.188 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 22: conflicting requests 2024-04-14T15:02:35.188 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.188 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 23: conflicting requests 2024-04-14T15:02:35.188 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.189 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 24: conflicting requests 2024-04-14T15:02:35.189 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.189 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 25: conflicting requests 2024-04-14T15:02:35.189 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.189 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 26: conflicting requests 2024-04-14T15:02:35.189 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.189 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 27: conflicting requests 2024-04-14T15:02:35.189 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.189 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 28: conflicting requests 2024-04-14T15:02:35.189 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.189 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 29: conflicting requests 2024-04-14T15:02:35.189 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.189 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 30: conflicting requests 2024-04-14T15:02:35.189 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.189 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 31: conflicting requests 2024-04-14T15:02:35.190 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.190 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 32: conflicting requests 2024-04-14T15:02:35.190 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.190 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 33: conflicting requests 2024-04-14T15:02:35.190 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.190 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 34: conflicting requests 2024-04-14T15:02:35.190 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.190 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 35: conflicting requests 2024-04-14T15:02:35.190 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.190 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 36: conflicting requests 2024-04-14T15:02:35.190 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.190 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 37: conflicting requests 2024-04-14T15:02:35.190 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.191 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.191 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.191 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 38: cannot install the best candidate for the job 2024-04-14T15:02:35.191 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.191 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.191 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.191 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.191 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 39: conflicting requests 2024-04-14T15:02:35.191 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.191 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.191 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.191 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.191 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.192 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.192 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 40: cannot install the best candidate for the job 2024-04-14T15:02:35.192 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.192 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.192 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.192 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.192 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 41: cannot install the best candidate for the job 2024-04-14T15:02:35.192 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.192 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.192 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.192 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.192 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 42: cannot install the best candidate for the job 2024-04-14T15:02:35.192 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.192 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.192 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.193 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.193 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 43: cannot install the best candidate for the job 2024-04-14T15:02:35.193 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.193 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.193 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.193 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.193 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 44: cannot install the best candidate for the job 2024-04-14T15:02:35.193 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.193 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.193 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.193 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.193 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 45: cannot install the best candidate for the job 2024-04-14T15:02:35.193 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.193 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.194 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.194 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-14T15:02:35.194 INFO:teuthology.orchestra.run.smithi090.stdout:Package python3-rgw-2:19.0.0-2778.ga971fd5f.el9.x86_64 is already installed. 2024-04-14T15:02:35.254 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-04-14T15:02:35.255 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-04-14T15:02:35.255 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-04-14T15:02:35.338 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install python3-cephfs 2024-04-14T15:02:35.932 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:02:37 ago on Sun 14 Apr 2024 02:59:58 PM UTC. 2024-04-14T15:02:36.220 INFO:teuthology.orchestra.run.smithi090.stderr:Modular dependency problems: 2024-04-14T15:02:36.220 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-04-14T15:02:36.220 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 1: conflicting requests 2024-04-14T15:02:36.220 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.220 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 2: conflicting requests 2024-04-14T15:02:36.220 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.220 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 3: conflicting requests 2024-04-14T15:02:36.220 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.220 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 4: conflicting requests 2024-04-14T15:02:36.220 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.221 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 5: conflicting requests 2024-04-14T15:02:36.221 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.221 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 6: conflicting requests 2024-04-14T15:02:36.221 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.221 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 7: conflicting requests 2024-04-14T15:02:36.221 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.221 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 8: conflicting requests 2024-04-14T15:02:36.221 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.221 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 9: conflicting requests 2024-04-14T15:02:36.221 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.221 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 10: conflicting requests 2024-04-14T15:02:36.221 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.221 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 11: conflicting requests 2024-04-14T15:02:36.221 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.222 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 12: conflicting requests 2024-04-14T15:02:36.222 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.222 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 13: conflicting requests 2024-04-14T15:02:36.222 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.222 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 14: conflicting requests 2024-04-14T15:02:36.222 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.222 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 15: conflicting requests 2024-04-14T15:02:36.222 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.222 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 16: conflicting requests 2024-04-14T15:02:36.222 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.222 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 17: conflicting requests 2024-04-14T15:02:36.222 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.223 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 18: conflicting requests 2024-04-14T15:02:36.223 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.223 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 19: conflicting requests 2024-04-14T15:02:36.223 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.223 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 20: conflicting requests 2024-04-14T15:02:36.223 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.223 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 21: conflicting requests 2024-04-14T15:02:36.223 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.223 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 22: conflicting requests 2024-04-14T15:02:36.223 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.223 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 23: conflicting requests 2024-04-14T15:02:36.223 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.223 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 24: conflicting requests 2024-04-14T15:02:36.223 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.224 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 25: conflicting requests 2024-04-14T15:02:36.224 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.224 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 26: conflicting requests 2024-04-14T15:02:36.224 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.224 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 27: conflicting requests 2024-04-14T15:02:36.224 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.224 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 28: conflicting requests 2024-04-14T15:02:36.224 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.224 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 29: conflicting requests 2024-04-14T15:02:36.224 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.224 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 30: conflicting requests 2024-04-14T15:02:36.224 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.224 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 31: conflicting requests 2024-04-14T15:02:36.224 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.224 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 32: conflicting requests 2024-04-14T15:02:36.225 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.225 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 33: conflicting requests 2024-04-14T15:02:36.225 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.225 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 34: conflicting requests 2024-04-14T15:02:36.225 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.225 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 35: conflicting requests 2024-04-14T15:02:36.225 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.225 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 36: conflicting requests 2024-04-14T15:02:36.225 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.225 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 37: conflicting requests 2024-04-14T15:02:36.225 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.225 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.225 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.225 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 38: cannot install the best candidate for the job 2024-04-14T15:02:36.226 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.226 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.226 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.226 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.226 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 39: conflicting requests 2024-04-14T15:02:36.226 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.226 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.226 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.226 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.226 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.226 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.226 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 40: cannot install the best candidate for the job 2024-04-14T15:02:36.226 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.226 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.227 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.227 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.227 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 41: cannot install the best candidate for the job 2024-04-14T15:02:36.227 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.227 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.227 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.227 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.227 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 42: cannot install the best candidate for the job 2024-04-14T15:02:36.227 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.227 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.227 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.227 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.227 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 43: cannot install the best candidate for the job 2024-04-14T15:02:36.227 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.227 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.228 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.228 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.228 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 44: cannot install the best candidate for the job 2024-04-14T15:02:36.228 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.228 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.228 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.228 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.228 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 45: cannot install the best candidate for the job 2024-04-14T15:02:36.228 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.228 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.228 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.228 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-14T15:02:36.229 INFO:teuthology.orchestra.run.smithi090.stdout:Package python3-cephfs-2:19.0.0-2778.ga971fd5f.el9.x86_64 is already installed. 2024-04-14T15:02:36.290 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-04-14T15:02:36.291 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-04-14T15:02:36.291 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-04-14T15:02:36.375 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install python3-rbd 2024-04-14T15:02:36.965 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:02:38 ago on Sun 14 Apr 2024 02:59:58 PM UTC. 2024-04-14T15:02:37.252 INFO:teuthology.orchestra.run.smithi090.stderr:Modular dependency problems: 2024-04-14T15:02:37.253 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-04-14T15:02:37.253 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 1: conflicting requests 2024-04-14T15:02:37.253 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.253 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 2: conflicting requests 2024-04-14T15:02:37.253 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.253 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 3: conflicting requests 2024-04-14T15:02:37.253 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.253 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 4: conflicting requests 2024-04-14T15:02:37.253 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.253 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 5: conflicting requests 2024-04-14T15:02:37.253 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.253 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 6: conflicting requests 2024-04-14T15:02:37.253 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.253 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 7: conflicting requests 2024-04-14T15:02:37.254 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.254 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 8: conflicting requests 2024-04-14T15:02:37.254 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.254 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 9: conflicting requests 2024-04-14T15:02:37.254 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.254 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 10: conflicting requests 2024-04-14T15:02:37.254 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.254 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 11: conflicting requests 2024-04-14T15:02:37.254 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.254 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 12: conflicting requests 2024-04-14T15:02:37.254 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.254 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 13: conflicting requests 2024-04-14T15:02:37.254 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.254 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 14: conflicting requests 2024-04-14T15:02:37.255 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.255 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 15: conflicting requests 2024-04-14T15:02:37.255 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.255 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 16: conflicting requests 2024-04-14T15:02:37.255 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.255 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 17: conflicting requests 2024-04-14T15:02:37.255 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.255 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 18: conflicting requests 2024-04-14T15:02:37.255 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.255 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 19: conflicting requests 2024-04-14T15:02:37.255 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.255 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 20: conflicting requests 2024-04-14T15:02:37.255 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.255 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 21: conflicting requests 2024-04-14T15:02:37.255 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.256 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 22: conflicting requests 2024-04-14T15:02:37.256 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.256 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 23: conflicting requests 2024-04-14T15:02:37.256 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.256 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 24: conflicting requests 2024-04-14T15:02:37.256 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.256 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 25: conflicting requests 2024-04-14T15:02:37.256 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.256 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 26: conflicting requests 2024-04-14T15:02:37.256 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.256 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 27: conflicting requests 2024-04-14T15:02:37.256 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.256 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 28: conflicting requests 2024-04-14T15:02:37.256 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.257 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 29: conflicting requests 2024-04-14T15:02:37.257 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.257 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 30: conflicting requests 2024-04-14T15:02:37.257 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.257 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 31: conflicting requests 2024-04-14T15:02:37.257 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.257 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 32: conflicting requests 2024-04-14T15:02:37.257 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.257 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 33: conflicting requests 2024-04-14T15:02:37.257 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.257 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 34: conflicting requests 2024-04-14T15:02:37.257 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.257 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 35: conflicting requests 2024-04-14T15:02:37.257 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.258 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 36: conflicting requests 2024-04-14T15:02:37.258 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.258 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 37: conflicting requests 2024-04-14T15:02:37.258 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.258 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.258 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.258 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 38: cannot install the best candidate for the job 2024-04-14T15:02:37.258 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.258 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.258 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.258 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.258 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 39: conflicting requests 2024-04-14T15:02:37.258 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.258 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.259 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.259 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.259 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.259 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.259 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 40: cannot install the best candidate for the job 2024-04-14T15:02:37.259 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.259 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.259 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.259 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.259 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 41: cannot install the best candidate for the job 2024-04-14T15:02:37.259 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.259 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.259 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.259 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.260 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 42: cannot install the best candidate for the job 2024-04-14T15:02:37.260 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.260 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.260 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.260 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.260 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 43: cannot install the best candidate for the job 2024-04-14T15:02:37.260 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.260 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.260 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.260 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.260 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 44: cannot install the best candidate for the job 2024-04-14T15:02:37.260 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.260 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.260 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.260 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.260 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 45: cannot install the best candidate for the job 2024-04-14T15:02:37.261 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.261 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.261 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.261 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-14T15:02:37.261 INFO:teuthology.orchestra.run.smithi090.stdout:Package python3-rbd-2:19.0.0-2778.ga971fd5f.el9.x86_64 is already installed. 2024-04-14T15:02:37.320 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-04-14T15:02:37.321 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-04-14T15:02:37.321 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-04-14T15:02:37.389 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install rbd-fuse 2024-04-14T15:02:37.983 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:02:39 ago on Sun 14 Apr 2024 02:59:58 PM UTC. 2024-04-14T15:02:38.273 INFO:teuthology.orchestra.run.smithi090.stderr:Modular dependency problems: 2024-04-14T15:02:38.273 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-04-14T15:02:38.273 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 1: conflicting requests 2024-04-14T15:02:38.273 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.273 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 2: conflicting requests 2024-04-14T15:02:38.273 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.273 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 3: conflicting requests 2024-04-14T15:02:38.273 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.273 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 4: conflicting requests 2024-04-14T15:02:38.273 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.274 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 5: conflicting requests 2024-04-14T15:02:38.274 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.274 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 6: conflicting requests 2024-04-14T15:02:38.274 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.274 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 7: conflicting requests 2024-04-14T15:02:38.274 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.274 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 8: conflicting requests 2024-04-14T15:02:38.274 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.274 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 9: conflicting requests 2024-04-14T15:02:38.274 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.274 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 10: conflicting requests 2024-04-14T15:02:38.274 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.274 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 11: conflicting requests 2024-04-14T15:02:38.274 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.275 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 12: conflicting requests 2024-04-14T15:02:38.275 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.275 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 13: conflicting requests 2024-04-14T15:02:38.275 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.275 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 14: conflicting requests 2024-04-14T15:02:38.275 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.275 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 15: conflicting requests 2024-04-14T15:02:38.275 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.275 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 16: conflicting requests 2024-04-14T15:02:38.275 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.275 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 17: conflicting requests 2024-04-14T15:02:38.275 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.275 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 18: conflicting requests 2024-04-14T15:02:38.275 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.275 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 19: conflicting requests 2024-04-14T15:02:38.276 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.276 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 20: conflicting requests 2024-04-14T15:02:38.276 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.276 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 21: conflicting requests 2024-04-14T15:02:38.276 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.276 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 22: conflicting requests 2024-04-14T15:02:38.276 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.276 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 23: conflicting requests 2024-04-14T15:02:38.276 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.276 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 24: conflicting requests 2024-04-14T15:02:38.276 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.276 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 25: conflicting requests 2024-04-14T15:02:38.276 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.276 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 26: conflicting requests 2024-04-14T15:02:38.276 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.276 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 27: conflicting requests 2024-04-14T15:02:38.277 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.277 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 28: conflicting requests 2024-04-14T15:02:38.277 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.277 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 29: conflicting requests 2024-04-14T15:02:38.277 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.277 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 30: conflicting requests 2024-04-14T15:02:38.277 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.277 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 31: conflicting requests 2024-04-14T15:02:38.277 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.277 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 32: conflicting requests 2024-04-14T15:02:38.277 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.277 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 33: conflicting requests 2024-04-14T15:02:38.277 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.278 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 34: conflicting requests 2024-04-14T15:02:38.278 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.278 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 35: conflicting requests 2024-04-14T15:02:38.278 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.278 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 36: conflicting requests 2024-04-14T15:02:38.278 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.278 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 37: conflicting requests 2024-04-14T15:02:38.278 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.278 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.278 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.278 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 38: cannot install the best candidate for the job 2024-04-14T15:02:38.278 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.278 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.279 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.279 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.279 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 39: conflicting requests 2024-04-14T15:02:38.279 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.279 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.279 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.279 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.279 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.279 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.279 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 40: cannot install the best candidate for the job 2024-04-14T15:02:38.279 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.279 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.279 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.279 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.280 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 41: cannot install the best candidate for the job 2024-04-14T15:02:38.280 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.280 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.280 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.280 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.280 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 42: cannot install the best candidate for the job 2024-04-14T15:02:38.280 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.280 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.280 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.280 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.280 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 43: cannot install the best candidate for the job 2024-04-14T15:02:38.280 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.280 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.280 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.281 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.281 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 44: cannot install the best candidate for the job 2024-04-14T15:02:38.281 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.281 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.281 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.281 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.281 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 45: cannot install the best candidate for the job 2024-04-14T15:02:38.281 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.281 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.281 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.281 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-14T15:02:38.341 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-04-14T15:02:38.342 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-04-14T15:02:38.342 INFO:teuthology.orchestra.run.smithi090.stdout: Package Architecture Version Repository Size 2024-04-14T15:02:38.342 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-04-14T15:02:38.342 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-04-14T15:02:38.342 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-fuse x86_64 2:19.0.0-2778.ga971fd5f.el9 ceph 85 k 2024-04-14T15:02:38.342 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:02:38.342 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-04-14T15:02:38.342 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-04-14T15:02:38.342 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-04-14T15:02:38.342 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:02:38.342 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 85 k 2024-04-14T15:02:38.342 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 231 k 2024-04-14T15:02:38.342 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-04-14T15:02:38.590 INFO:teuthology.orchestra.run.smithi090.stdout:rbd-fuse-19.0.0-2778.ga971fd5f.el9.x86_64.rpm 345 kB/s | 85 kB 00:00 2024-04-14T15:02:38.590 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-04-14T15:02:38.590 INFO:teuthology.orchestra.run.smithi090.stdout:Total 343 kB/s | 85 kB 00:00 2024-04-14T15:02:38.590 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-04-14T15:02:38.597 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-04-14T15:02:38.598 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-04-14T15:02:38.657 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-04-14T15:02:38.657 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-04-14T15:02:38.797 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-04-14T15:02:38.904 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : rbd-fuse-2:19.0.0-2778.ga971fd5f.el9.x86_64 1/1 2024-04-14T15:02:39.345 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: rbd-fuse-2:19.0.0-2778.ga971fd5f.el9.x86_64 1/1 2024-04-14T15:02:39.605 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : rbd-fuse-2:19.0.0-2778.ga971fd5f.el9.x86_64 1/1 2024-04-14T15:02:39.605 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:02:39.605 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-04-14T15:02:39.605 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-fuse-2:19.0.0-2778.ga971fd5f.el9.x86_64 2024-04-14T15:02:39.606 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:02:39.606 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-04-14T15:02:39.878 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install rbd-mirror 2024-04-14T15:02:40.475 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:02:42 ago on Sun 14 Apr 2024 02:59:58 PM UTC. 2024-04-14T15:02:40.766 INFO:teuthology.orchestra.run.smithi090.stderr:Modular dependency problems: 2024-04-14T15:02:40.767 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-04-14T15:02:40.767 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 1: conflicting requests 2024-04-14T15:02:40.767 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.767 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 2: conflicting requests 2024-04-14T15:02:40.767 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.767 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 3: conflicting requests 2024-04-14T15:02:40.767 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.767 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 4: conflicting requests 2024-04-14T15:02:40.767 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.767 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 5: conflicting requests 2024-04-14T15:02:40.767 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.767 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 6: conflicting requests 2024-04-14T15:02:40.767 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.767 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 7: conflicting requests 2024-04-14T15:02:40.768 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.768 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 8: conflicting requests 2024-04-14T15:02:40.768 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.768 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 9: conflicting requests 2024-04-14T15:02:40.768 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.768 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 10: conflicting requests 2024-04-14T15:02:40.768 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.768 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 11: conflicting requests 2024-04-14T15:02:40.768 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.768 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 12: conflicting requests 2024-04-14T15:02:40.768 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.768 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 13: conflicting requests 2024-04-14T15:02:40.768 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.768 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 14: conflicting requests 2024-04-14T15:02:40.768 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.769 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 15: conflicting requests 2024-04-14T15:02:40.769 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.769 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 16: conflicting requests 2024-04-14T15:02:40.769 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.769 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 17: conflicting requests 2024-04-14T15:02:40.769 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.769 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 18: conflicting requests 2024-04-14T15:02:40.769 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.769 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 19: conflicting requests 2024-04-14T15:02:40.769 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.769 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 20: conflicting requests 2024-04-14T15:02:40.769 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.769 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 21: conflicting requests 2024-04-14T15:02:40.769 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.769 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 22: conflicting requests 2024-04-14T15:02:40.770 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.770 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 23: conflicting requests 2024-04-14T15:02:40.770 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.770 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 24: conflicting requests 2024-04-14T15:02:40.770 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.770 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 25: conflicting requests 2024-04-14T15:02:40.770 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.770 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 26: conflicting requests 2024-04-14T15:02:40.770 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.770 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 27: conflicting requests 2024-04-14T15:02:40.770 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.770 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 28: conflicting requests 2024-04-14T15:02:40.770 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.770 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 29: conflicting requests 2024-04-14T15:02:40.771 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.771 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 30: conflicting requests 2024-04-14T15:02:40.771 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.771 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 31: conflicting requests 2024-04-14T15:02:40.771 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.771 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 32: conflicting requests 2024-04-14T15:02:40.771 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.771 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 33: conflicting requests 2024-04-14T15:02:40.771 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.771 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 34: conflicting requests 2024-04-14T15:02:40.771 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.771 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 35: conflicting requests 2024-04-14T15:02:40.771 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.771 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 36: conflicting requests 2024-04-14T15:02:40.771 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.771 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 37: conflicting requests 2024-04-14T15:02:40.772 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.772 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.772 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.772 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 38: cannot install the best candidate for the job 2024-04-14T15:02:40.772 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.772 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.772 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.772 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.772 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 39: conflicting requests 2024-04-14T15:02:40.772 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.772 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.772 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.772 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.772 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.773 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.773 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 40: cannot install the best candidate for the job 2024-04-14T15:02:40.773 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.773 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.773 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.773 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.773 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 41: cannot install the best candidate for the job 2024-04-14T15:02:40.773 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.773 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.773 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.773 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.773 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 42: cannot install the best candidate for the job 2024-04-14T15:02:40.773 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.773 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.773 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.773 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.774 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 43: cannot install the best candidate for the job 2024-04-14T15:02:40.774 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.774 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.774 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.774 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.774 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 44: cannot install the best candidate for the job 2024-04-14T15:02:40.774 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.774 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.774 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.774 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.774 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 45: cannot install the best candidate for the job 2024-04-14T15:02:40.774 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.774 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.774 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.775 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-14T15:02:40.834 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-04-14T15:02:40.834 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-04-14T15:02:40.834 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2024-04-14T15:02:40.834 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-04-14T15:02:40.834 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-04-14T15:02:40.834 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-mirror x86_64 2:19.0.0-2778.ga971fd5f.el9 ceph 3.1 M 2024-04-14T15:02:40.834 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:02:40.835 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-04-14T15:02:40.835 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-04-14T15:02:40.835 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-04-14T15:02:40.835 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:02:40.835 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 3.1 M 2024-04-14T15:02:40.835 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 13 M 2024-04-14T15:02:40.835 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-04-14T15:02:41.231 INFO:teuthology.orchestra.run.smithi090.stdout:rbd-mirror-19.0.0-2778.ga971fd5f.el9.x86_64.rpm 7.8 MB/s | 3.1 MB 00:00 2024-04-14T15:02:41.231 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-04-14T15:02:41.232 INFO:teuthology.orchestra.run.smithi090.stdout:Total 7.8 MB/s | 3.1 MB 00:00 2024-04-14T15:02:41.232 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-04-14T15:02:41.241 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-04-14T15:02:41.241 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-04-14T15:02:41.316 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-04-14T15:02:41.317 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-04-14T15:02:41.734 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-04-14T15:02:41.807 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : rbd-mirror-2:19.0.0-2778.ga971fd5f.el9.x86_64 1/1 2024-04-14T15:02:41.831 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2778.ga971fd5f.el9.x86_64 1/1 2024-04-14T15:02:41.831 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-14T15:02:41.831 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-14T15:02:41.831 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-14T15:02:41.831 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-14T15:02:41.831 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:02:43.068 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : rbd-mirror-2:19.0.0-2778.ga971fd5f.el9.x86_64 1/1 2024-04-14T15:02:43.069 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:02:43.069 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-04-14T15:02:43.069 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-mirror-2:19.0.0-2778.ga971fd5f.el9.x86_64 2024-04-14T15:02:43.069 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:02:43.069 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-04-14T15:02:43.360 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install rbd-nbd 2024-04-14T15:02:43.970 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:02:45 ago on Sun 14 Apr 2024 02:59:58 PM UTC. 2024-04-14T15:02:44.262 INFO:teuthology.orchestra.run.smithi090.stderr:Modular dependency problems: 2024-04-14T15:02:44.263 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-04-14T15:02:44.263 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 1: conflicting requests 2024-04-14T15:02:44.263 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.263 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 2: conflicting requests 2024-04-14T15:02:44.263 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.263 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 3: conflicting requests 2024-04-14T15:02:44.263 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.263 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 4: conflicting requests 2024-04-14T15:02:44.263 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.263 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 5: conflicting requests 2024-04-14T15:02:44.263 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.263 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 6: conflicting requests 2024-04-14T15:02:44.264 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.264 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 7: conflicting requests 2024-04-14T15:02:44.264 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.264 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 8: conflicting requests 2024-04-14T15:02:44.264 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.264 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 9: conflicting requests 2024-04-14T15:02:44.264 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.264 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 10: conflicting requests 2024-04-14T15:02:44.264 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.264 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 11: conflicting requests 2024-04-14T15:02:44.264 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.264 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 12: conflicting requests 2024-04-14T15:02:44.264 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.264 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 13: conflicting requests 2024-04-14T15:02:44.265 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.265 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 14: conflicting requests 2024-04-14T15:02:44.265 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.265 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 15: conflicting requests 2024-04-14T15:02:44.265 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.265 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 16: conflicting requests 2024-04-14T15:02:44.265 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.265 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 17: conflicting requests 2024-04-14T15:02:44.265 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.265 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 18: conflicting requests 2024-04-14T15:02:44.265 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.265 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 19: conflicting requests 2024-04-14T15:02:44.265 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.265 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 20: conflicting requests 2024-04-14T15:02:44.266 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.266 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 21: conflicting requests 2024-04-14T15:02:44.266 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.266 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 22: conflicting requests 2024-04-14T15:02:44.266 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.266 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 23: conflicting requests 2024-04-14T15:02:44.266 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.266 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 24: conflicting requests 2024-04-14T15:02:44.266 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.266 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 25: conflicting requests 2024-04-14T15:02:44.266 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.266 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 26: conflicting requests 2024-04-14T15:02:44.266 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.266 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 27: conflicting requests 2024-04-14T15:02:44.267 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.267 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 28: conflicting requests 2024-04-14T15:02:44.267 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.267 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 29: conflicting requests 2024-04-14T15:02:44.267 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.267 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 30: conflicting requests 2024-04-14T15:02:44.267 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.267 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 31: conflicting requests 2024-04-14T15:02:44.267 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.267 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 32: conflicting requests 2024-04-14T15:02:44.267 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.267 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 33: conflicting requests 2024-04-14T15:02:44.267 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.267 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 34: conflicting requests 2024-04-14T15:02:44.267 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.268 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 35: conflicting requests 2024-04-14T15:02:44.268 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.268 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 36: conflicting requests 2024-04-14T15:02:44.268 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.268 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 37: conflicting requests 2024-04-14T15:02:44.268 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.268 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.268 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.268 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 38: cannot install the best candidate for the job 2024-04-14T15:02:44.268 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.268 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.268 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.268 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.268 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 39: conflicting requests 2024-04-14T15:02:44.268 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.268 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.269 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.269 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.269 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.269 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.269 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 40: cannot install the best candidate for the job 2024-04-14T15:02:44.269 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.269 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.269 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.269 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.269 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 41: cannot install the best candidate for the job 2024-04-14T15:02:44.269 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.269 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.269 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.269 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.269 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 42: cannot install the best candidate for the job 2024-04-14T15:02:44.270 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.270 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.270 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.270 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.270 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 43: cannot install the best candidate for the job 2024-04-14T15:02:44.270 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.270 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.270 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.270 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.270 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 44: cannot install the best candidate for the job 2024-04-14T15:02:44.270 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.270 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.270 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.270 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.271 INFO:teuthology.orchestra.run.smithi090.stderr: Problem 45: cannot install the best candidate for the job 2024-04-14T15:02:44.271 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.271 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.271 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.271 INFO:teuthology.orchestra.run.smithi090.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-14T15:02:44.331 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-04-14T15:02:44.331 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-04-14T15:02:44.332 INFO:teuthology.orchestra.run.smithi090.stdout: Package Architecture Version Repository Size 2024-04-14T15:02:44.332 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-04-14T15:02:44.332 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-04-14T15:02:44.332 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-nbd x86_64 2:19.0.0-2778.ga971fd5f.el9 ceph 170 k 2024-04-14T15:02:44.332 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:02:44.332 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-04-14T15:02:44.332 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-04-14T15:02:44.332 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-04-14T15:02:44.332 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:02:44.333 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 170 k 2024-04-14T15:02:44.333 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 494 k 2024-04-14T15:02:44.333 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-04-14T15:02:44.732 INFO:teuthology.orchestra.run.smithi090.stdout:rbd-nbd-19.0.0-2778.ga971fd5f.el9.x86_64.rpm 427 kB/s | 170 kB 00:00 2024-04-14T15:02:44.732 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-04-14T15:02:44.733 INFO:teuthology.orchestra.run.smithi090.stdout:Total 426 kB/s | 170 kB 00:00 2024-04-14T15:02:44.733 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-04-14T15:02:44.740 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-04-14T15:02:44.740 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-04-14T15:02:44.800 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-04-14T15:02:44.801 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-04-14T15:02:44.949 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-04-14T15:02:45.056 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : rbd-nbd-2:19.0.0-2778.ga971fd5f.el9.x86_64 1/1 2024-04-14T15:02:45.630 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: rbd-nbd-2:19.0.0-2778.ga971fd5f.el9.x86_64 1/1 2024-04-14T15:02:45.959 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : rbd-nbd-2:19.0.0-2778.ga971fd5f.el9.x86_64 1/1 2024-04-14T15:02:45.959 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:02:45.959 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-04-14T15:02:45.959 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-nbd-2:19.0.0-2778.ga971fd5f.el9.x86_64 2024-04-14T15:02:45.960 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:02:45.960 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-04-14T15:02:46.248 DEBUG:teuthology.parallel:result is None 2024-04-14T15:02:46.249 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T15:02:46.396 DEBUG:teuthology.orchestra.run.smithi090:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-14T15:02:46.416 INFO:teuthology.orchestra.run.smithi090.stdout:19.0.0-2778.ga971fd5f.el9 2024-04-14T15:02:46.417 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2778.ga971fd5f.el9 2024-04-14T15:02:46.417 INFO:teuthology.task.install:The correct ceph version 19.0.0-2778.ga971fd5f is installed. 2024-04-14T15:02:46.419 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T15:02:46.820 DEBUG:teuthology.orchestra.run.smithi092:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-14T15:02:46.845 INFO:teuthology.orchestra.run.smithi092.stdout:19.0.0-2778.ga971fd5f.el9 2024-04-14T15:02:46.845 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2778.ga971fd5f.el9 2024-04-14T15:02:46.846 INFO:teuthology.task.install:The correct ceph version 19.0.0-2778.ga971fd5f is installed. 2024-04-14T15:02:46.847 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-14T15:02:46.847 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-14T15:02:46.847 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-14T15:02:46.878 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-04-14T15:02:46.878 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-14T15:02:46.916 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-14T15:02:46.917 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-14T15:02:46.917 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/usr/bin/daemon-helper 2024-04-14T15:02:46.945 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-14T15:02:47.013 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-04-14T15:02:47.013 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/usr/bin/daemon-helper 2024-04-14T15:02:47.042 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-14T15:02:47.109 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-14T15:02:47.109 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-14T15:02:47.109 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-14T15:02:47.136 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-14T15:02:47.203 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-04-14T15:02:47.203 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-14T15:02:47.230 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-14T15:02:47.297 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-14T15:02:47.297 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-14T15:02:47.297 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/usr/bin/stdin-killer 2024-04-14T15:02:47.324 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-14T15:02:47.391 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-04-14T15:02:47.391 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/usr/bin/stdin-killer 2024-04-14T15:02:47.420 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-14T15:02:47.488 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-14T15:02:47.587 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'client': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms bind msgr2': False, 'ms type': 'async'}, 'mon': {'debug mgrc': 20, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd class default list': '*', 'osd class load list': '*', 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'crush_tunables': 'jewel', 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'reached quota', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)'], 'mon_bind_msgr2': False, 'sha1': 'a971fd5f5719038c70b817bafacd6d5400b1d888', 'cephadm_mode': 'root'} 2024-04-14T15:02:47.587 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T15:02:47.588 INFO:tasks.cephadm:Cluster fsid is 0e288098-fa70-11ee-bc8e-c7b262605968 2024-04-14T15:02:47.588 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-14T15:02:47.588 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '[v1:172.21.15.90:6789]', 'mon.c': '[v1:172.21.15.90:6790]', 'mon.b': '[v1:172.21.15.92:6789]'} 2024-04-14T15:02:47.588 INFO:tasks.cephadm:First mon is mon.a on smithi090 2024-04-14T15:02:47.588 INFO:tasks.cephadm:First mgr is y 2024-04-14T15:02:47.588 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-14T15:02:47.588 DEBUG:teuthology.orchestra.run.smithi090:> sudo hostname $(hostname -s) 2024-04-14T15:02:47.615 DEBUG:teuthology.orchestra.run.smithi092:> sudo hostname $(hostname -s) 2024-04-14T15:02:47.642 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-04-14T15:02:47.643 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T15:02:47.784 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'a971fd5f5719038c70b817bafacd6d5400b1d888', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/54841/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2778-ga971fd5f', 'node_name': '172.21.4.69+confusa07', 'job_name': 'ceph-dev-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2778.ga971fd5f'}, 'url': 'https://4.chacra.ceph.com/r/ceph/main/a971fd5f5719038c70b817bafacd6d5400b1d888/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-07 15:35:40.226244', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'main', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/main/a971fd5f5719038c70b817bafacd6d5400b1d888/centos/9/flavors/default/', 'archs': ['arm64', 'source', 'x86_64'], 'distro': 'centos'}] 2024-04-14T15:02:48.008 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref main, sha1 a971fd5f5719038c70b817bafacd6d5400b1d888 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T15:02:48.009 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/main/a971fd5f5719038c70b817bafacd6d5400b1d888/centos/9/x86_64/flavors/default/cephadm 2024-04-14T15:02:48.009 DEBUG:teuthology.orchestra.run.smithi090:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/main/a971fd5f5719038c70b817bafacd6d5400b1d888/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-14T15:02:48.442 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r--. 1 ubuntu ubuntu 781494 Apr 14 15:02 /home/ubuntu/cephtest/cephadm 2024-04-14T15:02:48.442 DEBUG:teuthology.orchestra.run.smithi092:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/main/a971fd5f5719038c70b817bafacd6d5400b1d888/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-14T15:02:49.261 INFO:teuthology.orchestra.run.smithi092.stdout:-rw-r--r--. 1 ubuntu ubuntu 781494 Apr 14 15:02 /home/ubuntu/cephtest/cephadm 2024-04-14T15:02:49.262 DEBUG:teuthology.orchestra.run.smithi090:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-14T15:02:49.280 DEBUG:teuthology.orchestra.run.smithi092:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-14T15:02:49.307 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 on all hosts... 2024-04-14T15:02:49.307 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 pull 2024-04-14T15:02:49.323 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 pull 2024-04-14T15:02:50.122 INFO:teuthology.orchestra.run.smithi092.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888... 2024-04-14T15:02:50.350 INFO:teuthology.orchestra.run.smithi090.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888... 2024-04-14T15:03:52.032 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-14T15:03:52.033 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_version": "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)", 2024-04-14T15:03:52.033 INFO:teuthology.orchestra.run.smithi090.stdout: "image_id": "b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53", 2024-04-14T15:03:52.033 INFO:teuthology.orchestra.run.smithi090.stdout: "repo_digests": [ 2024-04-14T15:03:52.033 INFO:teuthology.orchestra.run.smithi090.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608" 2024-04-14T15:03:52.033 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:03:52.033 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-14T15:03:56.555 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-04-14T15:03:56.555 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph_version": "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)", 2024-04-14T15:03:56.555 INFO:teuthology.orchestra.run.smithi092.stdout: "image_id": "b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53", 2024-04-14T15:03:56.555 INFO:teuthology.orchestra.run.smithi092.stdout: "repo_digests": [ 2024-04-14T15:03:56.555 INFO:teuthology.orchestra.run.smithi092.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608" 2024-04-14T15:03:56.555 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:03:56.555 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-04-14T15:03:56.578 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /etc/ceph 2024-04-14T15:03:56.613 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /etc/ceph 2024-04-14T15:03:56.642 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 777 /etc/ceph 2024-04-14T15:03:56.684 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 777 /etc/ceph 2024-04-14T15:03:56.710 INFO:tasks.cephadm:Writing seed config... 2024-04-14T15:03:56.714 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-14T15:03:56.714 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-14T15:03:56.715 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-14T15:03:56.715 INFO:tasks.cephadm: override: [client] debug objecter = 20 2024-04-14T15:03:56.715 INFO:tasks.cephadm: override: [client] debug rados = 20 2024-04-14T15:03:56.715 INFO:tasks.cephadm: override: [global] ms bind msgr2 = False 2024-04-14T15:03:56.715 INFO:tasks.cephadm: override: [global] ms type = async 2024-04-14T15:03:56.715 INFO:tasks.cephadm: override: [mon] debug mgrc = 20 2024-04-14T15:03:56.715 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-14T15:03:56.715 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-14T15:03:56.715 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-14T15:03:56.715 INFO:tasks.cephadm: override: [mon] mon min osdmap epochs = 50 2024-04-14T15:03:56.715 INFO:tasks.cephadm: override: [mon] mon osdmap full prune interval = 2 2024-04-14T15:03:56.715 INFO:tasks.cephadm: override: [mon] mon osdmap full prune min = 15 2024-04-14T15:03:56.715 INFO:tasks.cephadm: override: [mon] mon osdmap full prune txsize = 2 2024-04-14T15:03:56.715 INFO:tasks.cephadm: override: [mon] mon warn on pool no app = False 2024-04-14T15:03:56.715 INFO:tasks.cephadm: override: [mon] paxos service trim min = 10 2024-04-14T15:03:56.716 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-14T15:03:56.716 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-14T15:03:56.716 INFO:tasks.cephadm: override: [osd] osd class default list = * 2024-04-14T15:03:56.716 INFO:tasks.cephadm: override: [osd] osd class load list = * 2024-04-14T15:03:56.716 INFO:tasks.cephadm: override: [osd] osd debug reject backfill probability = 0.3 2024-04-14T15:03:56.716 INFO:tasks.cephadm: override: [osd] osd delete sleep = 1 2024-04-14T15:03:56.716 INFO:tasks.cephadm: override: [osd] osd max backfills = 3 2024-04-14T15:03:56.716 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2024-04-14T15:03:56.716 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2024-04-14T15:03:56.716 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-14T15:03:56.716 INFO:tasks.cephadm: override: [osd] osd snap trim sleep = 2 2024-04-14T15:03:56.717 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-14T15:03:56.717 DEBUG:teuthology.orchestra.run.smithi090:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-14T15:03:56.741 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 0e288098-fa70-11ee-bc8e-c7b262605968 ms bind msgr2 = False ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd class default list = * osd class load list = * osd debug reject backfill probability = 0.3 osd delete sleep = 1 osd max backfills = 3 osd scrub min interval = 60 osd snap trim sleep = 2 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mgrc = 20 debug mon = 20 debug ms = 1 debug paxos = 20 mon min osdmap epochs = 50 mon osdmap full prune interval = 2 mon osdmap full prune min = 15 mon osdmap full prune txsize = 2 mon warn on pool no app = False paxos service trim min = 10 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] debug ms = 1 debug objecter = 20 debug rados = 20 2024-04-14T15:03:56.742 DEBUG:teuthology.orchestra.run.smithi090:mon.a> sudo journalctl -f -n 0 -u ceph-0e288098-fa70-11ee-bc8e-c7b262605968@mon.a.service 2024-04-14T15:03:56.785 DEBUG:teuthology.orchestra.run.smithi090:mgr.y> sudo journalctl -f -n 0 -u ceph-0e288098-fa70-11ee-bc8e-c7b262605968@mgr.y.service 2024-04-14T15:03:56.828 INFO:tasks.cephadm:Bootstrapping... 2024-04-14T15:03:56.828 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 -v bootstrap --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-addrv '[v1:172.21.15.90:6789]' --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-14T15:03:57.023 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-04-14T15:03:57.023 INFO:teuthology.orchestra.run.smithi090.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888', '-v', 'bootstrap', '--fsid', '0e288098-fa70-11ee-bc8e-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-addrv', '[v1:172.21.15.90:6789]', '--skip-admin-label'] 2024-04-14T15:03:57.024 INFO:teuthology.orchestra.run.smithi090.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-14T15:03:57.024 INFO:teuthology.orchestra.run.smithi090.stdout:Verifying podman|docker is present... 2024-04-14T15:03:57.050 INFO:teuthology.orchestra.run.smithi090.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-14T15:03:57.050 INFO:teuthology.orchestra.run.smithi090.stdout:Verifying lvm2 is present... 2024-04-14T15:03:57.050 INFO:teuthology.orchestra.run.smithi090.stdout:Verifying time synchronization is in place... 2024-04-14T15:03:57.057 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-14T15:03:57.057 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-14T15:03:57.063 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-14T15:03:57.063 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout inactive 2024-04-14T15:03:57.069 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout enabled 2024-04-14T15:03:57.075 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout active 2024-04-14T15:03:57.075 INFO:teuthology.orchestra.run.smithi090.stdout:Unit chronyd.service is enabled and running 2024-04-14T15:03:57.075 INFO:teuthology.orchestra.run.smithi090.stdout:Repeating the final host check... 2024-04-14T15:03:57.098 INFO:teuthology.orchestra.run.smithi090.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-14T15:03:57.099 INFO:teuthology.orchestra.run.smithi090.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-14T15:03:57.099 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl is present 2024-04-14T15:03:57.099 INFO:teuthology.orchestra.run.smithi090.stdout:lvcreate is present 2024-04-14T15:03:57.105 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-14T15:03:57.105 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-14T15:03:57.111 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-14T15:03:57.111 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout inactive 2024-04-14T15:03:57.117 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout enabled 2024-04-14T15:03:57.123 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout active 2024-04-14T15:03:57.123 INFO:teuthology.orchestra.run.smithi090.stdout:Unit chronyd.service is enabled and running 2024-04-14T15:03:57.123 INFO:teuthology.orchestra.run.smithi090.stdout:Host looks OK 2024-04-14T15:03:57.123 INFO:teuthology.orchestra.run.smithi090.stdout:Cluster fsid: 0e288098-fa70-11ee-bc8e-c7b262605968 2024-04-14T15:03:57.123 INFO:teuthology.orchestra.run.smithi090.stdout:Acquiring lock 139896804111024 on /run/cephadm/0e288098-fa70-11ee-bc8e-c7b262605968.lock 2024-04-14T15:03:57.123 INFO:teuthology.orchestra.run.smithi090.stdout:Lock 139896804111024 acquired on /run/cephadm/0e288098-fa70-11ee-bc8e-c7b262605968.lock 2024-04-14T15:03:57.124 INFO:teuthology.orchestra.run.smithi090.stdout:Verifying IP 172.21.15.90 port 6789 ... 2024-04-14T15:03:57.124 INFO:teuthology.orchestra.run.smithi090.stdout:Base mon IP(s) is [172.21.15.90:6789], mon addrv is [v1:172.21.15.90:6789] 2024-04-14T15:03:57.127 INFO:teuthology.orchestra.run.smithi090.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.90 metric 100 2024-04-14T15:03:57.127 INFO:teuthology.orchestra.run.smithi090.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.90 metric 100 2024-04-14T15:03:57.130 INFO:teuthology.orchestra.run.smithi090.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-14T15:03:57.130 INFO:teuthology.orchestra.run.smithi090.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-04-14T15:03:57.130 INFO:teuthology.orchestra.run.smithi090.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1747sec hoplimit 64 pref medium 2024-04-14T15:03:57.132 INFO:teuthology.orchestra.run.smithi090.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-14T15:03:57.132 INFO:teuthology.orchestra.run.smithi090.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-14T15:03:57.132 INFO:teuthology.orchestra.run.smithi090.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-14T15:03:57.132 INFO:teuthology.orchestra.run.smithi090.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-04-14T15:03:57.132 INFO:teuthology.orchestra.run.smithi090.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:d767/64 scope link 2024-04-14T15:03:57.133 INFO:teuthology.orchestra.run.smithi090.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-14T15:03:57.133 INFO:teuthology.orchestra.run.smithi090.stdout:Mon IP `172.21.15.90` is in CIDR network `172.21.0.0/20` 2024-04-14T15:03:57.133 INFO:teuthology.orchestra.run.smithi090.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20'] 2024-04-14T15:03:57.133 INFO:teuthology.orchestra.run.smithi090.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-14T15:03:57.134 INFO:teuthology.orchestra.run.smithi090.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888... 2024-04-14T15:03:57.760 INFO:teuthology.orchestra.run.smithi090.stdout:/bin/podman: stdout b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 2024-04-14T15:03:57.760 INFO:teuthology.orchestra.run.smithi090.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888... 2024-04-14T15:03:57.760 INFO:teuthology.orchestra.run.smithi090.stdout:/bin/podman: stderr Getting image source signatures 2024-04-14T15:03:57.760 INFO:teuthology.orchestra.run.smithi090.stdout:/bin/podman: stderr Copying blob sha256:4a9a0f674f5d00de2fb0d6279a6376d7e805966559c8ffba59b1bc2aa8fd87e7 2024-04-14T15:03:57.760 INFO:teuthology.orchestra.run.smithi090.stdout:/bin/podman: stderr Copying blob sha256:77070339c81bcb5c53bbbf3f31a47f7187e5393cb0e096452852a31192d97570 2024-04-14T15:03:57.760 INFO:teuthology.orchestra.run.smithi090.stdout:/bin/podman: stderr Copying config sha256:b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 2024-04-14T15:03:57.760 INFO:teuthology.orchestra.run.smithi090.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-14T15:03:58.667 INFO:teuthology.orchestra.run.smithi090.stdout:ceph: stdout ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev) 2024-04-14T15:03:58.667 INFO:teuthology.orchestra.run.smithi090.stdout:Ceph version: ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev) 2024-04-14T15:03:58.667 INFO:teuthology.orchestra.run.smithi090.stdout:Extracting ceph user uid/gid from container image... 2024-04-14T15:03:59.734 INFO:teuthology.orchestra.run.smithi090.stdout:stat: stdout 167 167 2024-04-14T15:03:59.734 INFO:teuthology.orchestra.run.smithi090.stdout:Creating initial keys... 2024-04-14T15:04:00.665 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph-authtool: stdout AQBg8Btm9Dg+BRAAspeJtvPfq1EEJnKVUYmhRw== 2024-04-14T15:04:01.589 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph-authtool: stdout AQBh8Btm4aqmABAAd8zbXUZ5gquKQsuesT2qGw== 2024-04-14T15:04:02.597 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph-authtool: stdout AQBh8Btmhbu7NxAAvF/WoFVyDqserZjmwYw+lQ== 2024-04-14T15:04:02.598 INFO:teuthology.orchestra.run.smithi090.stdout:Creating initial monmap... 2024-04-14T15:04:03.480 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-14T15:04:03.480 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-04-14T15:04:03.480 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 0e288098-fa70-11ee-bc8e-c7b262605968 2024-04-14T15:04:03.480 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-14T15:04:03.480 INFO:teuthology.orchestra.run.smithi090.stdout:monmaptool for a [v1:172.21.15.90:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-14T15:04:03.480 INFO:teuthology.orchestra.run.smithi090.stdout:setting min_mon_release = quincy 2024-04-14T15:04:03.480 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/monmaptool: set fsid to 0e288098-fa70-11ee-bc8e-c7b262605968 2024-04-14T15:04:03.480 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-14T15:04:03.480 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:04:03.481 INFO:teuthology.orchestra.run.smithi090.stdout:Creating mon... 2024-04-14T15:04:04.405 INFO:teuthology.orchestra.run.smithi090.stdout:create mon.a on 2024-04-14T15:04:04.600 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-14T15:04:04.737 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-14T15:04:04.910 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-0e288098-fa70-11ee-bc8e-c7b262605968.target → /etc/systemd/system/ceph-0e288098-fa70-11ee-bc8e-c7b262605968.target. 2024-04-14T15:04:04.910 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-0e288098-fa70-11ee-bc8e-c7b262605968.target → /etc/systemd/system/ceph-0e288098-fa70-11ee-bc8e-c7b262605968.target. 2024-04-14T15:04:05.131 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-0e288098-fa70-11ee-bc8e-c7b262605968@mon.a 2024-04-14T15:04:05.131 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stderr Failed to reset failed state of unit ceph-0e288098-fa70-11ee-bc8e-c7b262605968@mon.a.service: Unit ceph-0e288098-fa70-11ee-bc8e-c7b262605968@mon.a.service not loaded. 2024-04-14T15:04:05.294 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-0e288098-fa70-11ee-bc8e-c7b262605968.target.wants/ceph-0e288098-fa70-11ee-bc8e-c7b262605968@mon.a.service → /etc/systemd/system/ceph-0e288098-fa70-11ee-bc8e-c7b262605968@.service. 2024-04-14T15:04:05.865 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-14T15:04:05.865 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout disabled 2024-04-14T15:04:05.872 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-14T15:04:05.872 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout inactive 2024-04-14T15:04:05.872 INFO:teuthology.orchestra.run.smithi090.stdout:firewalld.service is not enabled 2024-04-14T15:04:05.872 INFO:teuthology.orchestra.run.smithi090.stdout:Not possible to enable service . firewalld.service is not available 2024-04-14T15:04:05.872 INFO:teuthology.orchestra.run.smithi090.stdout:Waiting for mon to start... 2024-04-14T15:04:05.873 INFO:teuthology.orchestra.run.smithi090.stdout:Waiting for mon... 2024-04-14T15:04:06.911 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout cluster: 2024-04-14T15:04:06.911 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout id: 0e288098-fa70-11ee-bc8e-c7b262605968 2024-04-14T15:04:06.911 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-14T15:04:06.911 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 2024-04-14T15:04:06.911 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout services: 2024-04-14T15:04:06.911 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.560117s) 2024-04-14T15:04:06.911 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-14T15:04:06.911 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-14T15:04:06.911 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 2024-04-14T15:04:06.912 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout data: 2024-04-14T15:04:06.912 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-14T15:04:06.912 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-14T15:04:06.912 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-14T15:04:06.912 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout pgs: 2024-04-14T15:04:06.912 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 2024-04-14T15:04:06.912 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:06.288+0000 7fc27ab21700 1 Processor -- start 2024-04-14T15:04:06.912 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:06.288+0000 7fc27ab21700 1 -- start start 2024-04-14T15:04:06.912 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:06.289+0000 7fc27ab21700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc27410a3b0 con 0x7fc274105f80 2024-04-14T15:04:06.912 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:06.289+0000 7fc2788bd700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fc274105f80 0x7fc2741063a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:34244/0 (socket says 172.21.15.90:34244) 2024-04-14T15:04:06.912 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:06.289+0000 7fc2788bd700 1 -- 172.21.15.90:0/402034382 learned_addr learned my addr 172.21.15.90:0/402034382 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:04:06.912 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:06.289+0000 7fc2737fe700 1 -- 172.21.15.90:0/402034382 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3028008291 0 0) 0x7fc27410a3b0 con 0x7fc274105f80 2024-04-14T15:04:06.912 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:06.289+0000 7fc2737fe700 1 -- 172.21.15.90:0/402034382 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc2640035b0 con 0x7fc274105f80 2024-04-14T15:04:06.913 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:06.289+0000 7fc2737fe700 1 -- 172.21.15.90:0/402034382 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2997252617 0 0) 0x7fc2640035b0 con 0x7fc274105f80 2024-04-14T15:04:06.913 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:06.289+0000 7fc2737fe700 1 -- 172.21.15.90:0/402034382 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc27410b590 con 0x7fc274105f80 2024-04-14T15:04:06.913 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:06.289+0000 7fc2737fe700 1 -- 172.21.15.90:0/402034382 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7fc268001580 con 0x7fc274105f80 2024-04-14T15:04:06.913 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:06.290+0000 7fc2737fe700 1 -- 172.21.15.90:0/402034382 <== mon.0 v1:172.21.15.90:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc2680017e0 con 0x7fc274105f80 2024-04-14T15:04:06.913 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:06.290+0000 7fc2737fe700 1 -- 172.21.15.90:0/402034382 <== mon.0 v1:172.21.15.90:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7fc268001b60 con 0x7fc274105f80 2024-04-14T15:04:06.913 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:06.290+0000 7fc27ab21700 1 -- 172.21.15.90:0/402034382 >> v1:172.21.15.90:6789/0 conn(0x7fc274105f80 legacy=0x7fc2741063a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:04:06.913 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:06.290+0000 7fc27ab21700 1 -- 172.21.15.90:0/402034382 shutdown_connections 2024-04-14T15:04:06.913 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:06.290+0000 7fc27ab21700 1 -- 172.21.15.90:0/402034382 >> 172.21.15.90:0/402034382 conn(0x7fc274101560 msgr2=0x7fc274103980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:04:06.913 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:06.290+0000 7fc27ab21700 1 -- 172.21.15.90:0/402034382 shutdown_connections 2024-04-14T15:04:06.913 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:06.290+0000 7fc27ab21700 1 -- 172.21.15.90:0/402034382 wait complete. 2024-04-14T15:04:06.913 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:06.291+0000 7fc27ab21700 1 librados: starting msgr at 2024-04-14T15:04:06.913 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:06.291+0000 7fc27ab21700 1 librados: starting objecter 2024-04-14T15:04:06.913 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:06.291+0000 7fc27ab21700 1 Processor -- start 2024-04-14T15:04:06.913 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:06.291+0000 7fc27ab21700 1 -- start start 2024-04-14T15:04:06.913 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:06.291+0000 7fc27ab21700 1 librados: setting wanted keys 2024-04-14T15:04:06.914 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:06.291+0000 7fc27ab21700 1 librados: calling monclient init 2024-04-14T15:04:06.914 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:06.291+0000 7fc27ab21700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc274199a00 con 0x7fc274105f80 2024-04-14T15:04:06.914 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:06.291+0000 7fc2788bd700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fc274105f80 0x7fc2741992f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:34254/0 (socket says 172.21.15.90:34254) 2024-04-14T15:04:06.914 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:06.291+0000 7fc2788bd700 1 -- 172.21.15.90:0/356732601 learned_addr learned my addr 172.21.15.90:0/356732601 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:04:06.914 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:06.291+0000 7fc271ffb700 10 client.?.objecter ms_handle_connect 0x7fc274105f80 2024-04-14T15:04:06.914 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:06.291+0000 7fc271ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:04:06.914 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:06.291+0000 7fc271ffb700 1 -- 172.21.15.90:0/356732601 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 641479476 0 0) 0x7fc274199a00 con 0x7fc274105f80 2024-04-14T15:04:06.914 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:06.291+0000 7fc271ffb700 1 -- 172.21.15.90:0/356732601 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc2640045b0 con 0x7fc274105f80 2024-04-14T15:04:06.914 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:06.292+0000 7fc271ffb700 1 -- 172.21.15.90:0/356732601 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1056723463 0 0) 0x7fc2640045b0 con 0x7fc274105f80 2024-04-14T15:04:06.914 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:06.292+0000 7fc271ffb700 1 -- 172.21.15.90:0/356732601 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc274199a00 con 0x7fc274105f80 2024-04-14T15:04:06.914 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:06.292+0000 7fc271ffb700 1 -- 172.21.15.90:0/356732601 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7fc268001480 con 0x7fc274105f80 2024-04-14T15:04:06.914 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:06.292+0000 7fc271ffb700 1 -- 172.21.15.90:0/356732601 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3233909164 0 0) 0x7fc274199a00 con 0x7fc274105f80 2024-04-14T15:04:06.914 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:06.292+0000 7fc271ffb700 1 -- 172.21.15.90:0/356732601 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc274199bd0 con 0x7fc274105f80 2024-04-14T15:04:06.914 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:06.292+0000 7fc27ab21700 1 -- 172.21.15.90:0/356732601 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc274199ec0 con 0x7fc274105f80 2024-04-14T15:04:06.915 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:06.292+0000 7fc271ffb700 1 -- 172.21.15.90:0/356732601 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc268004b90 con 0x7fc274105f80 2024-04-14T15:04:06.915 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:06.292+0000 7fc271ffb700 1 -- 172.21.15.90:0/356732601 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7fc268004ed0 con 0x7fc274105f80 2024-04-14T15:04:06.915 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:06.292+0000 7fc271ffb700 1 -- 172.21.15.90:0/356732601 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7fc268005440 con 0x7fc274105f80 2024-04-14T15:04:06.915 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:06.292+0000 7fc27ab21700 10 client.4098.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:04:06.915 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:06.292+0000 7fc27ab21700 1 -- 172.21.15.90:0/356732601 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc27419da10 con 0x7fc274105f80 2024-04-14T15:04:06.915 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:06.292+0000 7fc27ab21700 1 librados: init done 2024-04-14T15:04:06.915 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:06.292+0000 7fc271ffb700 1 -- 172.21.15.90:0/356732601 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 87150692 0 0) 0x7fc268005cf0 con 0x7fc274105f80 2024-04-14T15:04:06.915 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:06.292+0000 7fc271ffb700 10 client.4098.objecter ms_dispatch 0x7fc274001550 osd_map(1..1 src has 1..1) 2024-04-14T15:04:06.915 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:06.292+0000 7fc271ffb700 3 client.4098.objecter handle_osd_map got epochs [1,1] > 0 2024-04-14T15:04:06.915 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:06.292+0000 7fc271ffb700 3 client.4098.objecter handle_osd_map decoding full epoch 1 2024-04-14T15:04:06.915 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:06.292+0000 7fc271ffb700 20 client.4098.objecter dump_active .. 0 homeless 2024-04-14T15:04:06.915 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:06.293+0000 7fc27ab21700 1 -- 172.21.15.90:0/356732601 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc27410a6a0 con 0x7fc274105f80 2024-04-14T15:04:06.915 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:06.295+0000 7fc271ffb700 1 -- 172.21.15.90:0/356732601 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (unknown 1092875540 0 2268998900) 0x7fc268006780 con 0x7fc274105f80 2024-04-14T15:04:06.915 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:06.366+0000 7fc27ab21700 1 -- 172.21.15.90:0/356732601 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "status"} v 0) -- 0x7fc27402e040 con 0x7fc274105f80 2024-04-14T15:04:06.915 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:06.366+0000 7fc271ffb700 1 -- 172.21.15.90:0/356732601 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (unknown 1155462804 0 4064092613) 0x7fc268005f70 con 0x7fc274105f80 2024-04-14T15:04:06.915 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:06.367+0000 7fc27ab21700 10 librados: watch_flush enter 2024-04-14T15:04:06.916 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:06.367+0000 7fc27ab21700 10 librados: watch_flush exit 2024-04-14T15:04:06.916 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:06.367+0000 7fc27ab21700 20 client.4098.objecter shutdown clearing up homeless session... 2024-04-14T15:04:06.916 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:06.367+0000 7fc27ab21700 10 client.4098.objecter successfully canceled tick 2024-04-14T15:04:06.916 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:06.367+0000 7fc27ab21700 1 -- 172.21.15.90:0/356732601 >> v1:172.21.15.90:6789/0 conn(0x7fc274105f80 legacy=0x7fc2741992f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:04:06.916 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:06.367+0000 7fc27ab21700 1 -- 172.21.15.90:0/356732601 shutdown_connections 2024-04-14T15:04:06.916 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:06.367+0000 7fc27ab21700 1 -- 172.21.15.90:0/356732601 >> 172.21.15.90:0/356732601 conn(0x7fc274101560 msgr2=0x7fc2741904d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:04:06.916 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:06.367+0000 7fc27ab21700 1 -- 172.21.15.90:0/356732601 shutdown_connections 2024-04-14T15:04:06.916 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:06.367+0000 7fc27ab21700 1 -- 172.21.15.90:0/356732601 wait complete. 2024-04-14T15:04:06.916 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:06.367+0000 7fc27ab21700 1 librados: shutdown 2024-04-14T15:04:06.916 INFO:teuthology.orchestra.run.smithi090.stdout:mon is available 2024-04-14T15:04:06.916 INFO:teuthology.orchestra.run.smithi090.stdout:Assimilating anything we can from ceph.conf... 2024-04-14T15:04:07.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:07 smithi090 ceph-mon[29910]: from='client.? v1:172.21.15.90:0/1697621312' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-14T15:04:07.986 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 2024-04-14T15:04:07.986 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout [global] 2024-04-14T15:04:07.987 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout fsid = 0e288098-fa70-11ee-bc8e-c7b262605968 2024-04-14T15:04:07.987 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mon_host = [v1:172.21.15.90:6789] 2024-04-14T15:04:07.987 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-14T15:04:07.987 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-14T15:04:07.987 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-14T15:04:07.987 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-14T15:04:07.987 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 2024-04-14T15:04:07.987 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-14T15:04:07.987 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-14T15:04:07.987 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 2024-04-14T15:04:07.987 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout [osd] 2024-04-14T15:04:07.987 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-14T15:04:07.987 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-14T15:04:07.987 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:07.374+0000 7fa1c539e700 1 Processor -- start 2024-04-14T15:04:07.988 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:07.374+0000 7fa1c539e700 1 -- start start 2024-04-14T15:04:07.988 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:07.374+0000 7fa1c539e700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa1c0075bd0 con 0x7fa1c01057c0 2024-04-14T15:04:07.988 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:07.375+0000 7fa1bffff700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fa1c01057c0 0x7fa1c0105be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:34270/0 (socket says 172.21.15.90:34270) 2024-04-14T15:04:07.988 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:07.375+0000 7fa1bffff700 1 -- 172.21.15.90:0/3179319994 learned_addr learned my addr 172.21.15.90:0/3179319994 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:04:07.988 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:07.375+0000 7fa1beffd700 1 -- 172.21.15.90:0/3179319994 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 292457849 0 0) 0x7fa1c0075bd0 con 0x7fa1c01057c0 2024-04-14T15:04:07.988 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:07.375+0000 7fa1beffd700 1 -- 172.21.15.90:0/3179319994 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa1a80035b0 con 0x7fa1c01057c0 2024-04-14T15:04:07.988 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:07.375+0000 7fa1beffd700 1 -- 172.21.15.90:0/3179319994 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1619131225 0 0) 0x7fa1a80035b0 con 0x7fa1c01057c0 2024-04-14T15:04:07.988 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:07.375+0000 7fa1beffd700 1 -- 172.21.15.90:0/3179319994 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1c0107300 con 0x7fa1c01057c0 2024-04-14T15:04:07.988 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:07.375+0000 7fa1beffd700 1 -- 172.21.15.90:0/3179319994 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7fa1b4001580 con 0x7fa1c01057c0 2024-04-14T15:04:07.988 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:07.375+0000 7fa1beffd700 1 -- 172.21.15.90:0/3179319994 <== mon.0 v1:172.21.15.90:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa1b4001920 con 0x7fa1c01057c0 2024-04-14T15:04:07.988 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:07.375+0000 7fa1beffd700 1 -- 172.21.15.90:0/3179319994 <== mon.0 v1:172.21.15.90:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7fa1b4001ca0 con 0x7fa1c01057c0 2024-04-14T15:04:07.988 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:07.376+0000 7fa1c539e700 1 -- 172.21.15.90:0/3179319994 >> v1:172.21.15.90:6789/0 conn(0x7fa1c01057c0 legacy=0x7fa1c0105be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:04:07.988 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:07.376+0000 7fa1c539e700 1 -- 172.21.15.90:0/3179319994 shutdown_connections 2024-04-14T15:04:07.988 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:07.376+0000 7fa1c539e700 1 -- 172.21.15.90:0/3179319994 >> 172.21.15.90:0/3179319994 conn(0x7fa1c0101320 msgr2=0x7fa1c01037a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:04:07.988 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:07.376+0000 7fa1c539e700 1 -- 172.21.15.90:0/3179319994 shutdown_connections 2024-04-14T15:04:07.989 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:07.376+0000 7fa1c539e700 1 -- 172.21.15.90:0/3179319994 wait complete. 2024-04-14T15:04:07.989 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:07.376+0000 7fa1c539e700 1 librados: starting msgr at 2024-04-14T15:04:07.989 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:07.376+0000 7fa1c539e700 1 librados: starting objecter 2024-04-14T15:04:07.989 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:07.377+0000 7fa1c539e700 1 Processor -- start 2024-04-14T15:04:07.989 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:07.377+0000 7fa1c539e700 1 -- start start 2024-04-14T15:04:07.989 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:07.377+0000 7fa1c539e700 1 librados: setting wanted keys 2024-04-14T15:04:07.989 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:07.377+0000 7fa1c539e700 1 librados: calling monclient init 2024-04-14T15:04:07.989 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:07.377+0000 7fa1c539e700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa1c019cac0 con 0x7fa1c01057c0 2024-04-14T15:04:07.989 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:07.377+0000 7fa1bffff700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fa1c01057c0 0x7fa1c019c3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:34284/0 (socket says 172.21.15.90:34284) 2024-04-14T15:04:07.989 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:07.377+0000 7fa1bffff700 1 -- 172.21.15.90:0/1697621312 learned_addr learned my addr 172.21.15.90:0/1697621312 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:04:07.989 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:07.377+0000 7fa1bd7fa700 10 client.?.objecter ms_handle_connect 0x7fa1c01057c0 2024-04-14T15:04:07.989 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:07.377+0000 7fa1bd7fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:04:07.989 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:07.377+0000 7fa1bd7fa700 1 -- 172.21.15.90:0/1697621312 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2916649347 0 0) 0x7fa1c019cac0 con 0x7fa1c01057c0 2024-04-14T15:04:07.989 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:07.377+0000 7fa1bd7fa700 1 -- 172.21.15.90:0/1697621312 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa1a8004560 con 0x7fa1c01057c0 2024-04-14T15:04:07.990 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:07.378+0000 7fa1bd7fa700 1 -- 172.21.15.90:0/1697621312 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3907195944 0 0) 0x7fa1a8004560 con 0x7fa1c01057c0 2024-04-14T15:04:07.990 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:07.378+0000 7fa1bd7fa700 1 -- 172.21.15.90:0/1697621312 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa1c019cac0 con 0x7fa1c01057c0 2024-04-14T15:04:07.990 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:07.378+0000 7fa1bd7fa700 1 -- 172.21.15.90:0/1697621312 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7fa1b4001540 con 0x7fa1c01057c0 2024-04-14T15:04:07.990 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:07.378+0000 7fa1bd7fa700 1 -- 172.21.15.90:0/1697621312 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3931281436 0 0) 0x7fa1c019cac0 con 0x7fa1c01057c0 2024-04-14T15:04:07.990 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:07.378+0000 7fa1bd7fa700 1 -- 172.21.15.90:0/1697621312 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1c019cc90 con 0x7fa1c01057c0 2024-04-14T15:04:07.990 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:07.378+0000 7fa1c539e700 1 -- 172.21.15.90:0/1697621312 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa1c019cf80 con 0x7fa1c01057c0 2024-04-14T15:04:07.990 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:07.378+0000 7fa1c539e700 10 client.4100.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:04:07.990 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:07.378+0000 7fa1c539e700 1 -- 172.21.15.90:0/1697621312 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa1c01a0ad0 con 0x7fa1c01057c0 2024-04-14T15:04:07.990 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:07.378+0000 7fa1c539e700 1 librados: init done 2024-04-14T15:04:07.990 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:07.378+0000 7fa1bd7fa700 1 -- 172.21.15.90:0/1697621312 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa1b4001920 con 0x7fa1c01057c0 2024-04-14T15:04:07.990 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:07.379+0000 7fa1bd7fa700 1 -- 172.21.15.90:0/1697621312 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7fa1b4004de0 con 0x7fa1c01057c0 2024-04-14T15:04:07.990 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:07.379+0000 7fa1bd7fa700 1 -- 172.21.15.90:0/1697621312 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7fa1b4005310 con 0x7fa1c01057c0 2024-04-14T15:04:07.990 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:07.379+0000 7fa1bd7fa700 1 -- 172.21.15.90:0/1697621312 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 87150692 0 0) 0x7fa1b4006950 con 0x7fa1c01057c0 2024-04-14T15:04:07.991 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:07.379+0000 7fa1bd7fa700 10 client.4100.objecter ms_dispatch 0x7fa1c0001550 osd_map(1..1 src has 1..1) 2024-04-14T15:04:07.991 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:07.379+0000 7fa1bd7fa700 3 client.4100.objecter handle_osd_map got epochs [1,1] > 0 2024-04-14T15:04:07.991 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:07.379+0000 7fa1bd7fa700 3 client.4100.objecter handle_osd_map decoding full epoch 1 2024-04-14T15:04:07.991 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:07.379+0000 7fa1bd7fa700 20 client.4100.objecter dump_active .. 0 homeless 2024-04-14T15:04:07.991 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:07.379+0000 7fa1c539e700 1 -- 172.21.15.90:0/1697621312 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1ac005150 con 0x7fa1c01057c0 2024-04-14T15:04:07.991 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:07.381+0000 7fa1bd7fa700 1 -- 172.21.15.90:0/1697621312 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (unknown 1092875540 0 2268998900) 0x7fa1b4005790 con 0x7fa1c01057c0 2024-04-14T15:04:07.991 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:07.450+0000 7fa1c539e700 1 -- 172.21.15.90:0/1697621312 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7fa1ac002430 con 0x7fa1c01057c0 2024-04-14T15:04:07.991 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:07.453+0000 7fa1bd7fa700 1 -- 172.21.15.90:0/1697621312 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+317 (unknown 1213389831 0 3509117489) 0x7fa1b4005aa0 con 0x7fa1c01057c0 2024-04-14T15:04:07.991 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:07.453+0000 7fa1bd7fa700 1 -- 172.21.15.90:0/1697621312 <== mon.0 v1:172.21.15.90:6789/0 11 ==== config(26 keys) ==== 1050+0+0 (unknown 2744544961 0 0) 0x7fa1b4007000 con 0x7fa1c01057c0 2024-04-14T15:04:07.991 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:07.454+0000 7fa1c539e700 10 librados: watch_flush enter 2024-04-14T15:04:07.991 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:07.454+0000 7fa1c539e700 10 librados: watch_flush exit 2024-04-14T15:04:07.991 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:07.454+0000 7fa1c539e700 20 client.4100.objecter shutdown clearing up homeless session... 2024-04-14T15:04:07.991 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:07.454+0000 7fa1c539e700 10 client.4100.objecter successfully canceled tick 2024-04-14T15:04:07.991 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:07.454+0000 7fa1c539e700 1 -- 172.21.15.90:0/1697621312 >> v1:172.21.15.90:6789/0 conn(0x7fa1c01057c0 legacy=0x7fa1c019c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:04:07.991 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:07.454+0000 7fa1c539e700 1 -- 172.21.15.90:0/1697621312 shutdown_connections 2024-04-14T15:04:07.991 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:07.454+0000 7fa1c539e700 1 -- 172.21.15.90:0/1697621312 >> 172.21.15.90:0/1697621312 conn(0x7fa1c0101320 msgr2=0x7fa1c0192ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:04:07.992 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:07.455+0000 7fa1c539e700 1 -- 172.21.15.90:0/1697621312 shutdown_connections 2024-04-14T15:04:07.992 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:07.455+0000 7fa1c539e700 1 -- 172.21.15.90:0/1697621312 wait complete. 2024-04-14T15:04:07.992 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:07.455+0000 7fa1c539e700 1 librados: shutdown 2024-04-14T15:04:07.992 INFO:teuthology.orchestra.run.smithi090.stdout:Generating new minimal ceph.conf... 2024-04-14T15:04:09.062 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:08.389+0000 7f87a936e700 1 Processor -- start 2024-04-14T15:04:09.067 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:08.389+0000 7f87a936e700 1 -- start start 2024-04-14T15:04:09.068 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:08.389+0000 7f87a936e700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f87a407ba50 con 0x7f87a407ce50 2024-04-14T15:04:09.068 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:08.389+0000 7f87a2ffd700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f87a407ce50 0x7f87a407b2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:34292/0 (socket says 172.21.15.90:34292) 2024-04-14T15:04:09.068 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:08.389+0000 7f87a2ffd700 1 -- 172.21.15.90:0/4128111433 learned_addr learned my addr 172.21.15.90:0/4128111433 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:04:09.068 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:08.389+0000 7f87a1ffb700 1 -- 172.21.15.90:0/4128111433 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2954793033 0 0) 0x7f87a407ba50 con 0x7f87a407ce50 2024-04-14T15:04:09.068 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:08.389+0000 7f87a1ffb700 1 -- 172.21.15.90:0/4128111433 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f878c0035b0 con 0x7f87a407ce50 2024-04-14T15:04:09.068 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:08.390+0000 7f87a1ffb700 1 -- 172.21.15.90:0/4128111433 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3190667350 0 0) 0x7f878c0035b0 con 0x7f87a407ce50 2024-04-14T15:04:09.068 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:08.390+0000 7f87a1ffb700 1 -- 172.21.15.90:0/4128111433 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87a407bc20 con 0x7f87a407ce50 2024-04-14T15:04:09.068 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:08.390+0000 7f87a1ffb700 1 -- 172.21.15.90:0/4128111433 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f8798001580 con 0x7f87a407ce50 2024-04-14T15:04:09.068 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:08.390+0000 7f87a1ffb700 1 -- 172.21.15.90:0/4128111433 <== mon.0 v1:172.21.15.90:6789/0 4 ==== config(26 keys) ==== 1050+0+0 (unknown 2744544961 0 0) 0x7f8798001d30 con 0x7f87a407ce50 2024-04-14T15:04:09.068 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:08.390+0000 7f87a1ffb700 1 -- 172.21.15.90:0/4128111433 <== mon.0 v1:172.21.15.90:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f8798005960 con 0x7f87a407ce50 2024-04-14T15:04:09.068 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:08.390+0000 7f87a936e700 1 -- 172.21.15.90:0/4128111433 >> v1:172.21.15.90:6789/0 conn(0x7f87a407ce50 legacy=0x7f87a407b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:04:09.068 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:08.390+0000 7f87a936e700 1 -- 172.21.15.90:0/4128111433 shutdown_connections 2024-04-14T15:04:09.068 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:08.390+0000 7f87a936e700 1 -- 172.21.15.90:0/4128111433 >> 172.21.15.90:0/4128111433 conn(0x7f87a4102670 msgr2=0x7f87a4104a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:04:09.068 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:08.391+0000 7f87a936e700 1 -- 172.21.15.90:0/4128111433 shutdown_connections 2024-04-14T15:04:09.069 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:08.391+0000 7f87a936e700 1 -- 172.21.15.90:0/4128111433 wait complete. 2024-04-14T15:04:09.069 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:08.391+0000 7f87a936e700 1 librados: starting msgr at 2024-04-14T15:04:09.069 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:08.391+0000 7f87a936e700 1 librados: starting objecter 2024-04-14T15:04:09.069 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:08.391+0000 7f87a936e700 1 Processor -- start 2024-04-14T15:04:09.069 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:08.391+0000 7f87a936e700 1 -- start start 2024-04-14T15:04:09.069 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:08.391+0000 7f87a936e700 1 librados: setting wanted keys 2024-04-14T15:04:09.069 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:08.391+0000 7f87a936e700 1 librados: calling monclient init 2024-04-14T15:04:09.069 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:08.391+0000 7f87a936e700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f87a419d5e0 con 0x7f87a407ce50 2024-04-14T15:04:09.069 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:08.392+0000 7f87a2ffd700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f87a407ce50 0x7f87a419ced0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:34302/0 (socket says 172.21.15.90:34302) 2024-04-14T15:04:09.069 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:08.392+0000 7f87a2ffd700 1 -- 172.21.15.90:0/262313742 learned_addr learned my addr 172.21.15.90:0/262313742 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:04:09.069 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:08.392+0000 7f878bfff700 10 client.?.objecter ms_handle_connect 0x7f87a407ce50 2024-04-14T15:04:09.069 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:08.392+0000 7f878bfff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:04:09.069 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:08.392+0000 7f878bfff700 1 -- 172.21.15.90:0/262313742 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3371640736 0 0) 0x7f87a419d5e0 con 0x7f87a407ce50 2024-04-14T15:04:09.069 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:08.392+0000 7f878bfff700 1 -- 172.21.15.90:0/262313742 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f878c0056e0 con 0x7f87a407ce50 2024-04-14T15:04:09.069 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:08.392+0000 7f878bfff700 1 -- 172.21.15.90:0/262313742 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3246844406 0 0) 0x7f87a407bc20 con 0x7f87a407ce50 2024-04-14T15:04:09.070 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:08.392+0000 7f878bfff700 1 -- 172.21.15.90:0/262313742 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f87a419d5e0 con 0x7f87a407ce50 2024-04-14T15:04:09.070 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:08.392+0000 7f878bfff700 1 -- 172.21.15.90:0/262313742 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f8798001080 con 0x7f87a407ce50 2024-04-14T15:04:09.070 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:08.392+0000 7f878bfff700 1 -- 172.21.15.90:0/262313742 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3627296319 0 0) 0x7f87a419d5e0 con 0x7f87a407ce50 2024-04-14T15:04:09.070 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:08.392+0000 7f878bfff700 1 -- 172.21.15.90:0/262313742 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87a419d7b0 con 0x7f87a407ce50 2024-04-14T15:04:09.070 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:08.392+0000 7f87a936e700 1 -- 172.21.15.90:0/262313742 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f87a419daa0 con 0x7f87a407ce50 2024-04-14T15:04:09.070 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:08.392+0000 7f87a936e700 10 client.4102.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:04:09.070 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:08.392+0000 7f87a936e700 1 -- 172.21.15.90:0/262313742 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f87a41a15f0 con 0x7f87a407ce50 2024-04-14T15:04:09.070 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:08.392+0000 7f87a936e700 1 librados: init done 2024-04-14T15:04:09.070 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:08.392+0000 7f878bfff700 1 -- 172.21.15.90:0/262313742 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1050+0+0 (unknown 2744544961 0 0) 0x7f8798001d30 con 0x7f87a407ce50 2024-04-14T15:04:09.070 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:08.392+0000 7f878bfff700 1 -- 172.21.15.90:0/262313742 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f8798006040 con 0x7f87a407ce50 2024-04-14T15:04:09.070 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:08.392+0000 7f878bfff700 1 -- 172.21.15.90:0/262313742 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f87980066b0 con 0x7f87a407ce50 2024-04-14T15:04:09.070 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:08.393+0000 7f878bfff700 1 -- 172.21.15.90:0/262313742 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 87150692 0 0) 0x7f8798006ce0 con 0x7f87a407ce50 2024-04-14T15:04:09.070 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:08.393+0000 7f878bfff700 10 client.4102.objecter ms_dispatch 0x7f87a4001550 osd_map(1..1 src has 1..1) 2024-04-14T15:04:09.070 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:08.393+0000 7f878bfff700 3 client.4102.objecter handle_osd_map got epochs [1,1] > 0 2024-04-14T15:04:09.070 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:08.393+0000 7f878bfff700 3 client.4102.objecter handle_osd_map decoding full epoch 1 2024-04-14T15:04:09.071 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:08.393+0000 7f878bfff700 20 client.4102.objecter dump_active .. 0 homeless 2024-04-14T15:04:09.071 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:08.393+0000 7f87a936e700 1 -- 172.21.15.90:0/262313742 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8790005150 con 0x7f87a407ce50 2024-04-14T15:04:09.071 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:08.395+0000 7f878bfff700 1 -- 172.21.15.90:0/262313742 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (unknown 1092875540 0 2268998900) 0x7f878c0056e0 con 0x7f87a407ce50 2024-04-14T15:04:09.071 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:08.463+0000 7f87a936e700 1 -- 172.21.15.90:0/262313742 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f8790005440 con 0x7f87a407ce50 2024-04-14T15:04:09.071 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:08.463+0000 7f878bfff700 1 -- 172.21.15.90:0/262313742 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+147 (unknown 1452402520 0 2144695993) 0x7f8798001900 con 0x7f87a407ce50 2024-04-14T15:04:09.071 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:08.464+0000 7f87a936e700 10 librados: watch_flush enter 2024-04-14T15:04:09.071 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:08.464+0000 7f87a936e700 10 librados: watch_flush exit 2024-04-14T15:04:09.071 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:08.464+0000 7f87a936e700 20 client.4102.objecter shutdown clearing up homeless session... 2024-04-14T15:04:09.071 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:08.464+0000 7f87a936e700 10 client.4102.objecter successfully canceled tick 2024-04-14T15:04:09.071 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:08.464+0000 7f87a936e700 1 -- 172.21.15.90:0/262313742 >> v1:172.21.15.90:6789/0 conn(0x7f87a407ce50 legacy=0x7f87a419ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:04:09.071 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:08.464+0000 7f87a936e700 1 -- 172.21.15.90:0/262313742 shutdown_connections 2024-04-14T15:04:09.071 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:08.464+0000 7f87a936e700 1 -- 172.21.15.90:0/262313742 >> 172.21.15.90:0/262313742 conn(0x7f87a4102670 msgr2=0x7f87a410b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:04:09.071 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:08.464+0000 7f87a936e700 1 -- 172.21.15.90:0/262313742 shutdown_connections 2024-04-14T15:04:09.071 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:08.464+0000 7f87a936e700 1 -- 172.21.15.90:0/262313742 wait complete. 2024-04-14T15:04:09.071 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:08.464+0000 7f87a936e700 1 librados: shutdown 2024-04-14T15:04:09.072 INFO:teuthology.orchestra.run.smithi090.stdout:Restarting the monitor... 2024-04-14T15:04:09.087 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:08 smithi090 ceph-mon[29910]: from='client.? v1:172.21.15.90:0/262313742' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:04:09.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:09 smithi090 systemd[1]: Stopping Ceph mon.a for 0e288098-fa70-11ee-bc8e-c7b262605968... 2024-04-14T15:04:09.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:09 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mon-a[29887]: 2024-04-14T15:04:09.323+0000 7f441ee40700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T15:04:09.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:09 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mon-a[29887]: 2024-04-14T15:04:09.323+0000 7f441ee40700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-14T15:04:09.773 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:09 smithi090 podman[30243]: 2024-04-14 15:04:09.498253812 +0000 UTC m=+0.269270009 container died faa95a3cbccddabf8a5a1f95b04812794ef6b5156b9593a604f9bb1b2ef028df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mon-a, ceph=True, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=main-a971fd5, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-14T15:04:10.098 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:09 smithi090 podman[30243]: 2024-04-14 15:04:09.847514913 +0000 UTC m=+0.618531104 container cleanup faa95a3cbccddabf8a5a1f95b04812794ef6b5156b9593a604f9bb1b2ef028df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mon-a, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a971fd5, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-14T15:04:10.098 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:09 smithi090 bash[30243]: ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mon-a 2024-04-14T15:04:10.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 podman[30262]: 2024-04-14 15:04:10.098680103 +0000 UTC m=+0.599844493 container remove faa95a3cbccddabf8a5a1f95b04812794ef6b5156b9593a604f9bb1b2ef028df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mon-a, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=main-a971fd5) 2024-04-14T15:04:10.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 systemd[1]: ceph-0e288098-fa70-11ee-bc8e-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-14T15:04:10.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 systemd[1]: Stopped Ceph mon.a for 0e288098-fa70-11ee-bc8e-c7b262605968. 2024-04-14T15:04:10.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 systemd[1]: Starting Ceph mon.a for 0e288098-fa70-11ee-bc8e-c7b262605968... 2024-04-14T15:04:10.749 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 podman[30361]: 2024-04-14 15:04:10.466456197 +0000 UTC m=+0.103316069 container create fa32a6dcdb6ac5d02ad3385108a5352762c685236cdccde67fdc46db0001ec1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mon-a, org.label-schema.vendor=CentOS, RELEASE=main-a971fd5, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403) 2024-04-14T15:04:10.749 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 podman[30361]: 2024-04-14 15:04:10.38125693 +0000 UTC m=+0.018116787 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T15:04:10.749 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 podman[30361]: 2024-04-14 15:04:10.65240624 +0000 UTC m=+0.289266070 container init fa32a6dcdb6ac5d02ad3385108a5352762c685236cdccde67fdc46db0001ec1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mon-a, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=main-a971fd5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240403, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-14T15:04:10.749 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 podman[30361]: 2024-04-14 15:04:10.657538603 +0000 UTC m=+0.294398434 container start fa32a6dcdb6ac5d02ad3385108a5352762c685236cdccde67fdc46db0001ec1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mon-a, RELEASE=main-a971fd5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-14T15:04:10.749 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: set uid:gid to 167:167 (ceph:ceph) 2024-04-14T15:04:10.750 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev), process ceph-mon, pid 7 2024-04-14T15:04:10.750 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: pidfile_write: ignore empty --pid-file 2024-04-14T15:04:10.750 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: load: jerasure load: lrc 2024-04-14T15:04:10.750 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: RocksDB version: 7.9.2 2024-04-14T15:04:10.750 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Git sha 0 2024-04-14T15:04:10.750 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Compile date 2024-04-06 20:28:24 2024-04-14T15:04:10.750 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: DB SUMMARY 2024-04-14T15:04:10.750 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: DB Session ID: 2WLCR2B8JLLOHCKTH6O8 2024-04-14T15:04:10.750 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: CURRENT file: CURRENT 2024-04-14T15:04:10.750 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: IDENTITY file: IDENTITY 2024-04-14T15:04:10.750 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-14T15:04:10.750 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-14T15:04:10.751 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 94929 ; 2024-04-14T15:04:10.751 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.error_if_exists: 0 2024-04-14T15:04:10.751 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.create_if_missing: 0 2024-04-14T15:04:10.751 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.paranoid_checks: 1 2024-04-14T15:04:10.751 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-14T15:04:10.751 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-14T15:04:10.751 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-14T15:04:10.751 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.env: 0x562f43a65c60 2024-04-14T15:04:10.751 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.fs: PosixFileSystem 2024-04-14T15:04:10.751 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.info_log: 0x562f46271c40 2024-04-14T15:04:10.751 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.max_file_opening_threads: 16 2024-04-14T15:04:10.751 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.statistics: (nil) 2024-04-14T15:04:10.751 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.use_fsync: 0 2024-04-14T15:04:10.752 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.max_log_file_size: 0 2024-04-14T15:04:10.752 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-14T15:04:10.752 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-14T15:04:10.752 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.keep_log_file_num: 1000 2024-04-14T15:04:10.752 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.recycle_log_file_num: 0 2024-04-14T15:04:10.752 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.allow_fallocate: 1 2024-04-14T15:04:10.752 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.allow_mmap_reads: 0 2024-04-14T15:04:10.752 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.allow_mmap_writes: 0 2024-04-14T15:04:10.752 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.use_direct_reads: 0 2024-04-14T15:04:10.752 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-14T15:04:10.752 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.create_missing_column_families: 0 2024-04-14T15:04:10.752 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.db_log_dir: 2024-04-14T15:04:10.752 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.wal_dir: 2024-04-14T15:04:10.753 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-14T15:04:10.753 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-14T15:04:10.753 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-14T15:04:10.753 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-14T15:04:10.753 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-14T15:04:10.753 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-14T15:04:10.753 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.advise_random_on_open: 1 2024-04-14T15:04:10.753 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.db_write_buffer_size: 0 2024-04-14T15:04:10.753 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.write_buffer_manager: 0x562f462421e0 2024-04-14T15:04:10.753 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-14T15:04:10.753 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-14T15:04:10.753 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-14T15:04:10.753 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.rate_limiter: (nil) 2024-04-14T15:04:10.753 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-14T15:04:10.754 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.wal_recovery_mode: 2 2024-04-14T15:04:10.755 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.enable_thread_tracking: 0 2024-04-14T15:04:10.755 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.enable_pipelined_write: 0 2024-04-14T15:04:10.755 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.unordered_write: 0 2024-04-14T15:04:10.755 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-14T15:04:10.755 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-14T15:04:10.755 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-14T15:04:10.755 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-14T15:04:10.755 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.row_cache: None 2024-04-14T15:04:10.756 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.wal_filter: None 2024-04-14T15:04:10.756 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-14T15:04:10.756 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.allow_ingest_behind: 0 2024-04-14T15:04:10.756 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.two_write_queues: 0 2024-04-14T15:04:10.756 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.manual_wal_flush: 0 2024-04-14T15:04:10.756 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.wal_compression: 0 2024-04-14T15:04:10.756 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.atomic_flush: 0 2024-04-14T15:04:10.756 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-14T15:04:10.756 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-14T15:04:10.756 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-14T15:04:10.756 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.log_readahead_size: 0 2024-04-14T15:04:10.756 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-14T15:04:10.756 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.best_efforts_recovery: 0 2024-04-14T15:04:10.757 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-14T15:04:10.757 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-14T15:04:10.757 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.allow_data_in_errors: 0 2024-04-14T15:04:10.757 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.db_host_id: __hostname__ 2024-04-14T15:04:10.757 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-14T15:04:10.757 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.max_background_jobs: 2 2024-04-14T15:04:10.757 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.max_background_compactions: -1 2024-04-14T15:04:10.757 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.max_subcompactions: 1 2024-04-14T15:04:10.757 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-14T15:04:10.757 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-14T15:04:10.757 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-14T15:04:10.757 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.max_total_wal_size: 0 2024-04-14T15:04:10.757 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-14T15:04:10.757 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-14T15:04:10.758 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-14T15:04:10.758 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-14T15:04:10.758 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.max_open_files: -1 2024-04-14T15:04:10.758 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.bytes_per_sync: 0 2024-04-14T15:04:10.758 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-14T15:04:10.758 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-14T15:04:10.758 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.compaction_readahead_size: 0 2024-04-14T15:04:10.758 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.max_background_flushes: -1 2024-04-14T15:04:10.758 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Compression algorithms supported: 2024-04-14T15:04:10.758 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-14T15:04:10.758 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: kZSTD supported: 0 2024-04-14T15:04:10.758 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: kXpressCompression supported: 0 2024-04-14T15:04:10.758 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: kLZ4HCCompression supported: 1 2024-04-14T15:04:10.759 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: kZlibCompression supported: 1 2024-04-14T15:04:10.759 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: kSnappyCompression supported: 1 2024-04-14T15:04:10.759 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: kLZ4Compression supported: 1 2024-04-14T15:04:10.759 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: kBZip2Compression supported: 0 2024-04-14T15:04:10.759 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-14T15:04:10.759 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-14T15:04:10.759 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-04-14T15:04:10.759 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-14T15:04:10.759 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-14T15:04:10.759 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.merge_operator: 2024-04-14T15:04:10.759 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.compaction_filter: None 2024-04-14T15:04:10.759 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.compaction_filter_factory: None 2024-04-14T15:04:10.759 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.sst_partitioner_factory: None 2024-04-14T15:04:10.759 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-14T15:04:10.760 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-14T15:04:10.760 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x562f46271e00) 2024-04-14T15:04:10.760 INFO:journalctl@ceph.mon.a.smithi090.stdout: cache_index_and_filter_blocks: 1 2024-04-14T15:04:10.760 INFO:journalctl@ceph.mon.a.smithi090.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-14T15:04:10.760 INFO:journalctl@ceph.mon.a.smithi090.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-14T15:04:10.760 INFO:journalctl@ceph.mon.a.smithi090.stdout: pin_top_level_index_and_filter: 1 2024-04-14T15:04:10.760 INFO:journalctl@ceph.mon.a.smithi090.stdout: index_type: 0 2024-04-14T15:04:10.760 INFO:journalctl@ceph.mon.a.smithi090.stdout: data_block_index_type: 0 2024-04-14T15:04:10.760 INFO:journalctl@ceph.mon.a.smithi090.stdout: index_shortening: 1 2024-04-14T15:04:10.760 INFO:journalctl@ceph.mon.a.smithi090.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-14T15:04:10.760 INFO:journalctl@ceph.mon.a.smithi090.stdout: checksum: 4 2024-04-14T15:04:10.760 INFO:journalctl@ceph.mon.a.smithi090.stdout: no_block_cache: 0 2024-04-14T15:04:10.760 INFO:journalctl@ceph.mon.a.smithi090.stdout: block_cache: 0x562f46275090 2024-04-14T15:04:10.760 INFO:journalctl@ceph.mon.a.smithi090.stdout: block_cache_name: BinnedLRUCache 2024-04-14T15:04:10.761 INFO:journalctl@ceph.mon.a.smithi090.stdout: block_cache_options: 2024-04-14T15:04:10.761 INFO:journalctl@ceph.mon.a.smithi090.stdout: capacity : 536870912 2024-04-14T15:04:10.761 INFO:journalctl@ceph.mon.a.smithi090.stdout: num_shard_bits : 4 2024-04-14T15:04:10.761 INFO:journalctl@ceph.mon.a.smithi090.stdout: strict_capacity_limit : 0 2024-04-14T15:04:10.761 INFO:journalctl@ceph.mon.a.smithi090.stdout: high_pri_pool_ratio: 0.000 2024-04-14T15:04:10.761 INFO:journalctl@ceph.mon.a.smithi090.stdout: block_cache_compressed: (nil) 2024-04-14T15:04:10.761 INFO:journalctl@ceph.mon.a.smithi090.stdout: persistent_cache: (nil) 2024-04-14T15:04:10.761 INFO:journalctl@ceph.mon.a.smithi090.stdout: block_size: 4096 2024-04-14T15:04:10.761 INFO:journalctl@ceph.mon.a.smithi090.stdout: block_size_deviation: 10 2024-04-14T15:04:10.761 INFO:journalctl@ceph.mon.a.smithi090.stdout: block_restart_interval: 16 2024-04-14T15:04:10.761 INFO:journalctl@ceph.mon.a.smithi090.stdout: index_block_restart_interval: 1 2024-04-14T15:04:10.761 INFO:journalctl@ceph.mon.a.smithi090.stdout: metadata_block_size: 4096 2024-04-14T15:04:10.761 INFO:journalctl@ceph.mon.a.smithi090.stdout: partition_filters: 0 2024-04-14T15:04:10.761 INFO:journalctl@ceph.mon.a.smithi090.stdout: use_delta_encoding: 1 2024-04-14T15:04:10.762 INFO:journalctl@ceph.mon.a.smithi090.stdout: filter_policy: bloomfilter 2024-04-14T15:04:10.762 INFO:journalctl@ceph.mon.a.smithi090.stdout: whole_key_filtering: 1 2024-04-14T15:04:10.762 INFO:journalctl@ceph.mon.a.smithi090.stdout: verify_compression: 0 2024-04-14T15:04:10.762 INFO:journalctl@ceph.mon.a.smithi090.stdout: read_amp_bytes_per_bit: 0 2024-04-14T15:04:10.762 INFO:journalctl@ceph.mon.a.smithi090.stdout: format_version: 5 2024-04-14T15:04:10.762 INFO:journalctl@ceph.mon.a.smithi090.stdout: enable_index_compression: 1 2024-04-14T15:04:10.762 INFO:journalctl@ceph.mon.a.smithi090.stdout: block_align: 0 2024-04-14T15:04:10.762 INFO:journalctl@ceph.mon.a.smithi090.stdout: max_auto_readahead_size: 262144 2024-04-14T15:04:10.762 INFO:journalctl@ceph.mon.a.smithi090.stdout: prepopulate_block_cache: 0 2024-04-14T15:04:10.762 INFO:journalctl@ceph.mon.a.smithi090.stdout: initial_auto_readahead_size: 8192 2024-04-14T15:04:10.762 INFO:journalctl@ceph.mon.a.smithi090.stdout: num_file_reads_for_auto_readahead: 2 2024-04-14T15:04:10.762 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.write_buffer_size: 33554432 2024-04-14T15:04:10.762 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.max_write_buffer_number: 2 2024-04-14T15:04:10.763 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.compression: NoCompression 2024-04-14T15:04:10.763 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.bottommost_compression: Disabled 2024-04-14T15:04:10.763 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.prefix_extractor: nullptr 2024-04-14T15:04:10.763 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-14T15:04:10.763 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.num_levels: 7 2024-04-14T15:04:10.763 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-14T15:04:10.763 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-14T15:04:10.763 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-14T15:04:10.763 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-14T15:04:10.763 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-14T15:04:10.763 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-14T15:04:10.763 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-14T15:04:10.763 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-14T15:04:10.763 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-14T15:04:10.763 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-14T15:04:10.764 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-14T15:04:10.764 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-14T15:04:10.764 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-14T15:04:10.764 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.compression_opts.level: 32767 2024-04-14T15:04:10.764 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.compression_opts.strategy: 0 2024-04-14T15:04:10.764 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-14T15:04:10.764 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-14T15:04:10.764 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-14T15:04:10.764 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-14T15:04:10.764 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.compression_opts.enabled: false 2024-04-14T15:04:10.764 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-14T15:04:10.764 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-14T15:04:10.764 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-14T15:04:10.764 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-14T15:04:10.764 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.target_file_size_base: 67108864 2024-04-14T15:04:10.765 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-14T15:04:10.765 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-14T15:04:10.765 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-14T15:04:10.765 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-14T15:04:10.765 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-14T15:04:10.765 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-14T15:04:10.765 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-14T15:04:10.765 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-14T15:04:10.765 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-14T15:04:10.765 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-14T15:04:10.765 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-14T15:04:10.765 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-14T15:04:10.765 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-14T15:04:10.765 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-14T15:04:10.766 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.arena_block_size: 1048576 2024-04-14T15:04:10.766 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-14T15:04:10.766 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-14T15:04:10.766 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.disable_auto_compactions: 0 2024-04-14T15:04:10.766 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-14T15:04:10.766 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-14T15:04:10.766 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-14T15:04:10.766 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-14T15:04:10.766 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-14T15:04:10.766 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-14T15:04:10.766 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-14T15:04:10.766 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-14T15:04:10.766 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-14T15:04:10.767 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-14T15:04:10.767 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-14T15:04:10.767 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.inplace_update_support: 0 2024-04-14T15:04:10.767 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-14T15:04:10.767 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-14T15:04:10.767 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-14T15:04:10.767 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-14T15:04:10.767 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.bloom_locality: 0 2024-04-14T15:04:10.767 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.max_successive_merges: 0 2024-04-14T15:04:10.767 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-14T15:04:10.767 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.paranoid_file_checks: 0 2024-04-14T15:04:10.767 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.force_consistency_checks: 1 2024-04-14T15:04:10.767 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.report_bg_io_stats: 0 2024-04-14T15:04:10.767 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.ttl: 2592000 2024-04-14T15:04:10.767 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-14T15:04:10.768 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-14T15:04:10.768 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-14T15:04:10.768 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.enable_blob_files: false 2024-04-14T15:04:10.768 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.min_blob_size: 0 2024-04-14T15:04:10.768 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.blob_file_size: 268435456 2024-04-14T15:04:10.768 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-14T15:04:10.768 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-14T15:04:10.768 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-14T15:04:10.768 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-14T15:04:10.768 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-14T15:04:10.768 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.blob_file_starting_level: 0 2024-04-14T15:04:10.768 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-14T15:04:10.768 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-14T15:04:10.769 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-14T15:04:10.769 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 7431a7ce-35f4-476e-a4e3-0a5c6d49a165 2024-04-14T15:04:10.769 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713107050686628, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-14T15:04:10.769 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-14T15:04:10.769 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713107050687537, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 90584, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 294, "table_properties": {"data_size": 88639, "index_size": 265, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 12317, "raw_average_key_size": 48, "raw_value_size": 81970, "raw_average_value_size": 323, "num_data_blocks": 11, "num_entries": 253, "num_filter_entries": 253, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713107050, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "7431a7ce-35f4-476e-a4e3-0a5c6d49a165", "db_session_id": "2WLCR2B8JLLOHCKTH6O8", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-14T15:04:10.769 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713107050687618, "job": 1, "event": "recovery_finished"} 2024-04-14T15:04:10.769 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-14T15:04:10.769 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-14T15:04:10.769 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x562f46364000 2024-04-14T15:04:10.769 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: DB pointer 0x562f4634e000 2024-04-14T15:04:10.769 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-14T15:04:10.769 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-14T15:04:10.769 INFO:journalctl@ceph.mon.a.smithi090.stdout: ** DB Stats ** 2024-04-14T15:04:10.769 INFO:journalctl@ceph.mon.a.smithi090.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-14T15:04:10.770 INFO:journalctl@ceph.mon.a.smithi090.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-14T15:04:10.770 INFO:journalctl@ceph.mon.a.smithi090.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-14T15:04:10.770 INFO:journalctl@ceph.mon.a.smithi090.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-14T15:04:10.770 INFO:journalctl@ceph.mon.a.smithi090.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-14T15:04:10.770 INFO:journalctl@ceph.mon.a.smithi090.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-14T15:04:10.770 INFO:journalctl@ceph.mon.a.smithi090.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-14T15:04:10.770 INFO:journalctl@ceph.mon.a.smithi090.stdout: 2024-04-14T15:04:10.770 INFO:journalctl@ceph.mon.a.smithi090.stdout: ** Compaction Stats [default] ** 2024-04-14T15:04:10.770 INFO:journalctl@ceph.mon.a.smithi090.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-14T15:04:10.770 INFO:journalctl@ceph.mon.a.smithi090.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-14T15:04:10.770 INFO:journalctl@ceph.mon.a.smithi090.stdout: L0 2/0 90.29 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 147.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-14T15:04:10.770 INFO:journalctl@ceph.mon.a.smithi090.stdout: Sum 2/0 90.29 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 147.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-14T15:04:10.770 INFO:journalctl@ceph.mon.a.smithi090.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 147.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-14T15:04:10.771 INFO:journalctl@ceph.mon.a.smithi090.stdout: 2024-04-14T15:04:10.771 INFO:journalctl@ceph.mon.a.smithi090.stdout: ** Compaction Stats [default] ** 2024-04-14T15:04:10.771 INFO:journalctl@ceph.mon.a.smithi090.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-14T15:04:10.771 INFO:journalctl@ceph.mon.a.smithi090.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-14T15:04:10.771 INFO:journalctl@ceph.mon.a.smithi090.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 147.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-14T15:04:10.771 INFO:journalctl@ceph.mon.a.smithi090.stdout: 2024-04-14T15:04:10.771 INFO:journalctl@ceph.mon.a.smithi090.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-14T15:04:10.771 INFO:journalctl@ceph.mon.a.smithi090.stdout: 2024-04-14T15:04:10.771 INFO:journalctl@ceph.mon.a.smithi090.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-14T15:04:10.771 INFO:journalctl@ceph.mon.a.smithi090.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-14T15:04:10.771 INFO:journalctl@ceph.mon.a.smithi090.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-14T15:04:10.771 INFO:journalctl@ceph.mon.a.smithi090.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-14T15:04:10.771 INFO:journalctl@ceph.mon.a.smithi090.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-14T15:04:10.772 INFO:journalctl@ceph.mon.a.smithi090.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-14T15:04:10.772 INFO:journalctl@ceph.mon.a.smithi090.stdout: Cumulative compaction: 0.00 GB write, 30.42 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-14T15:04:10.772 INFO:journalctl@ceph.mon.a.smithi090.stdout: Interval compaction: 0.00 GB write, 30.42 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-14T15:04:10.772 INFO:journalctl@ceph.mon.a.smithi090.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-14T15:04:10.772 INFO:journalctl@ceph.mon.a.smithi090.stdout: Block cache BinnedLRUCache@0x562f46275090#7 capacity: 512.00 MB usage: 1.30 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-04-14T15:04:10.772 INFO:journalctl@ceph.mon.a.smithi090.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.83 KB,0.000157952%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-04-14T15:04:10.772 INFO:journalctl@ceph.mon.a.smithi090.stdout: 2024-04-14T15:04:10.772 INFO:journalctl@ceph.mon.a.smithi090.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-14T15:04:10.772 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: starting mon.a rank 0 at public addrs v1:172.21.15.90:6789/0 at bind addrs v1:172.21.15.90:6789/0 mon_data /var/lib/ceph/mon/ceph-a fsid 0e288098-fa70-11ee-bc8e-c7b262605968 2024-04-14T15:04:10.772 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: mon.a@-1(???) e1 preinit fsid 0e288098-fa70-11ee-bc8e-c7b262605968 2024-04-14T15:04:10.772 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: mon.a@-1(???).mds e1 new map 2024-04-14T15:04:10.772 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: mon.a@-1(???).mds e1 print_map 2024-04-14T15:04:10.772 INFO:journalctl@ceph.mon.a.smithi090.stdout: e1 2024-04-14T15:04:10.772 INFO:journalctl@ceph.mon.a.smithi090.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-14T15:04:10.772 INFO:journalctl@ceph.mon.a.smithi090.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-14T15:04:10.773 INFO:journalctl@ceph.mon.a.smithi090.stdout: legacy client fscid: -1 2024-04-14T15:04:10.773 INFO:journalctl@ceph.mon.a.smithi090.stdout: 2024-04-14T15:04:10.773 INFO:journalctl@ceph.mon.a.smithi090.stdout: No filesystems configured 2024-04-14T15:04:10.773 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-14T15:04:10.773 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-14T15:04:10.773 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-14T15:04:10.773 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-14T15:04:10.773 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-14T15:04:10.773 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-14T15:04:10.773 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: monmap epoch 1 2024-04-14T15:04:10.773 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: fsid 0e288098-fa70-11ee-bc8e-c7b262605968 2024-04-14T15:04:10.773 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: last_changed 2024-04-14T15:04:02.905926+0000 2024-04-14T15:04:10.773 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: created 2024-04-14T15:04:02.905926+0000 2024-04-14T15:04:10.773 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: min_mon_release 19 (squid) 2024-04-14T15:04:10.774 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: election_strategy: 1 2024-04-14T15:04:10.774 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: 0: v1:172.21.15.90:6789/0 mon.a 2024-04-14T15:04:10.774 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: fsmap 2024-04-14T15:04:10.774 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: osdmap e1: 0 total, 0 up, 0 in 2024-04-14T15:04:10.774 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 ceph-mon[30399]: mgrmap e1: no daemons active 2024-04-14T15:04:10.831 INFO:teuthology.orchestra.run.smithi090.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-14T15:04:11.023 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 bash[30361]: fa32a6dcdb6ac5d02ad3385108a5352762c685236cdccde67fdc46db0001ec1a 2024-04-14T15:04:11.023 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:10 smithi090 systemd[1]: Started Ceph mon.a for 0e288098-fa70-11ee-bc8e-c7b262605968. 2024-04-14T15:04:11.813 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:11.245+0000 7f2ab16be700 1 Processor -- start 2024-04-14T15:04:11.818 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:11.245+0000 7f2ab16be700 1 -- start start 2024-04-14T15:04:11.821 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:11.245+0000 7f2ab16be700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f2aac10b4c0 con 0x7f2aac107090 2024-04-14T15:04:11.821 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:11.245+0000 7f2aaaffd700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f2aac107090 0x7f2aac1074b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:55124/0 (socket says 172.21.15.90:55124) 2024-04-14T15:04:11.821 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:11.245+0000 7f2aaaffd700 1 -- 172.21.15.90:0/1550492966 learned_addr learned my addr 172.21.15.90:0/1550492966 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:04:11.821 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:11.246+0000 7f2aa9ffb700 1 -- 172.21.15.90:0/1550492966 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1266900469 0 0) 0x7f2aac10b4c0 con 0x7f2aac107090 2024-04-14T15:04:11.821 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:11.246+0000 7f2aa9ffb700 1 -- 172.21.15.90:0/1550492966 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2a940035b0 con 0x7f2aac107090 2024-04-14T15:04:11.821 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:11.246+0000 7f2aa9ffb700 1 -- 172.21.15.90:0/1550492966 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 174854319 0 0) 0x7f2a940035b0 con 0x7f2aac107090 2024-04-14T15:04:11.821 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:11.246+0000 7f2aa9ffb700 1 -- 172.21.15.90:0/1550492966 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2aac10c6a0 con 0x7f2aac107090 2024-04-14T15:04:11.822 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:11.246+0000 7f2aa9ffb700 1 -- 172.21.15.90:0/1550492966 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f2a9c001580 con 0x7f2aac107090 2024-04-14T15:04:11.822 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:11.246+0000 7f2aa9ffb700 1 -- 172.21.15.90:0/1550492966 <== mon.0 v1:172.21.15.90:6789/0 4 ==== config(26 keys) ==== 1050+0+0 (unknown 2744544961 0 0) 0x7f2a9c001d30 con 0x7f2aac107090 2024-04-14T15:04:11.822 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:11.246+0000 7f2aa9ffb700 1 -- 172.21.15.90:0/1550492966 <== mon.0 v1:172.21.15.90:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f2a9c005960 con 0x7f2aac107090 2024-04-14T15:04:11.822 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:11.246+0000 7f2ab16be700 1 -- 172.21.15.90:0/1550492966 >> v1:172.21.15.90:6789/0 conn(0x7f2aac107090 legacy=0x7f2aac1074b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:04:11.822 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:11.246+0000 7f2ab16be700 1 -- 172.21.15.90:0/1550492966 shutdown_connections 2024-04-14T15:04:11.822 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:11.246+0000 7f2ab16be700 1 -- 172.21.15.90:0/1550492966 >> 172.21.15.90:0/1550492966 conn(0x7f2aac102610 msgr2=0x7f2aac104a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:04:11.823 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:11.247+0000 7f2ab16be700 1 -- 172.21.15.90:0/1550492966 shutdown_connections 2024-04-14T15:04:11.823 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:11.247+0000 7f2ab16be700 1 -- 172.21.15.90:0/1550492966 wait complete. 2024-04-14T15:04:11.823 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:11.247+0000 7f2ab16be700 1 librados: starting msgr at 2024-04-14T15:04:11.823 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:11.247+0000 7f2ab16be700 1 librados: starting objecter 2024-04-14T15:04:11.823 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:11.247+0000 7f2ab16be700 1 Processor -- start 2024-04-14T15:04:11.823 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:11.247+0000 7f2ab16be700 1 -- start start 2024-04-14T15:04:11.823 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:11.247+0000 7f2ab16be700 1 librados: setting wanted keys 2024-04-14T15:04:11.823 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:11.247+0000 7f2ab16be700 1 librados: calling monclient init 2024-04-14T15:04:11.823 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:11.247+0000 7f2ab16be700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f2aac19b3b0 con 0x7f2aac107090 2024-04-14T15:04:11.823 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:11.247+0000 7f2aaaffd700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f2aac107090 0x7f2aac19aca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:55126/0 (socket says 172.21.15.90:55126) 2024-04-14T15:04:11.823 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:11.247+0000 7f2aaaffd700 1 -- 172.21.15.90:0/3343821424 learned_addr learned my addr 172.21.15.90:0/3343821424 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:04:11.823 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:11.248+0000 7f2a93fff700 10 client.?.objecter ms_handle_connect 0x7f2aac107090 2024-04-14T15:04:11.823 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:11.248+0000 7f2a93fff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:04:11.824 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:11.248+0000 7f2a93fff700 1 -- 172.21.15.90:0/3343821424 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 232146823 0 0) 0x7f2aac19b3b0 con 0x7f2aac107090 2024-04-14T15:04:11.824 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:11.248+0000 7f2a93fff700 1 -- 172.21.15.90:0/3343821424 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2a940056e0 con 0x7f2aac107090 2024-04-14T15:04:11.824 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:11.248+0000 7f2a93fff700 1 -- 172.21.15.90:0/3343821424 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 266971747 0 0) 0x7f2aac10c6a0 con 0x7f2aac107090 2024-04-14T15:04:11.824 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:11.248+0000 7f2a93fff700 1 -- 172.21.15.90:0/3343821424 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2aac19b3b0 con 0x7f2aac107090 2024-04-14T15:04:11.824 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:11.248+0000 7f2a93fff700 1 -- 172.21.15.90:0/3343821424 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f2a9c004870 con 0x7f2aac107090 2024-04-14T15:04:11.824 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:11.248+0000 7f2a93fff700 1 -- 172.21.15.90:0/3343821424 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 502022247 0 0) 0x7f2aac19b3b0 con 0x7f2aac107090 2024-04-14T15:04:11.824 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:11.248+0000 7f2a93fff700 1 -- 172.21.15.90:0/3343821424 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2aac19b580 con 0x7f2aac107090 2024-04-14T15:04:11.824 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:11.248+0000 7f2ab16be700 1 -- 172.21.15.90:0/3343821424 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2aac19b870 con 0x7f2aac107090 2024-04-14T15:04:11.824 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:11.248+0000 7f2ab16be700 10 client.14098.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:04:11.824 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:11.248+0000 7f2ab16be700 1 -- 172.21.15.90:0/3343821424 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f2aac19f3c0 con 0x7f2aac107090 2024-04-14T15:04:11.824 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:11.248+0000 7f2ab16be700 1 librados: init done 2024-04-14T15:04:11.824 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:11.248+0000 7f2a93fff700 1 -- 172.21.15.90:0/3343821424 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1050+0+0 (unknown 2744544961 0 0) 0x7f2a9c001d30 con 0x7f2aac107090 2024-04-14T15:04:11.824 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:11.248+0000 7f2a93fff700 1 -- 172.21.15.90:0/3343821424 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f2a9c006030 con 0x7f2aac107090 2024-04-14T15:04:11.824 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:11.248+0000 7f2a93fff700 1 -- 172.21.15.90:0/3343821424 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f2a9c006660 con 0x7f2aac107090 2024-04-14T15:04:11.824 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:11.248+0000 7f2a93fff700 1 -- 172.21.15.90:0/3343821424 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 87150692 0 0) 0x7f2a9c007d50 con 0x7f2aac107090 2024-04-14T15:04:11.825 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:11.248+0000 7f2a93fff700 10 client.14098.objecter ms_dispatch 0x7f2aac001550 osd_map(1..1 src has 1..1) 2024-04-14T15:04:11.825 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:11.248+0000 7f2a93fff700 3 client.14098.objecter handle_osd_map got epochs [1,1] > 0 2024-04-14T15:04:11.825 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:11.248+0000 7f2a93fff700 3 client.14098.objecter handle_osd_map decoding full epoch 1 2024-04-14T15:04:11.825 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:11.248+0000 7f2a93fff700 20 client.14098.objecter dump_active .. 0 homeless 2024-04-14T15:04:11.825 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:11.249+0000 7f2ab16be700 1 -- 172.21.15.90:0/3343821424 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a98005150 con 0x7f2aac107090 2024-04-14T15:04:11.825 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:11.251+0000 7f2a93fff700 1 -- 172.21.15.90:0/3343821424 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (unknown 1092875540 0 2268998900) 0x7f2a940056e0 con 0x7f2aac107090 2024-04-14T15:04:11.825 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:11.317+0000 7f2ab16be700 1 -- 172.21.15.90:0/3343821424 --> v1:172.21.15.90:6789/0 -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7f2a98005440 con 0x7f2aac107090 2024-04-14T15:04:11.825 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:11.319+0000 7f2a93fff700 1 -- 172.21.15.90:0/3343821424 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (unknown 534127980 0 0) 0x7f2a9c0056a0 con 0x7f2aac107090 2024-04-14T15:04:11.825 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:11.319+0000 7f2ab16be700 10 librados: watch_flush enter 2024-04-14T15:04:11.825 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:11.319+0000 7f2ab16be700 10 librados: watch_flush exit 2024-04-14T15:04:11.825 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:11.319+0000 7f2ab16be700 20 client.14098.objecter shutdown clearing up homeless session... 2024-04-14T15:04:11.825 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:11.319+0000 7f2ab16be700 10 client.14098.objecter successfully canceled tick 2024-04-14T15:04:11.825 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:11.319+0000 7f2ab16be700 1 -- 172.21.15.90:0/3343821424 >> v1:172.21.15.90:6789/0 conn(0x7f2aac107090 legacy=0x7f2aac19aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:04:11.825 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:11.320+0000 7f2ab16be700 1 -- 172.21.15.90:0/3343821424 shutdown_connections 2024-04-14T15:04:11.825 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:11.320+0000 7f2ab16be700 1 -- 172.21.15.90:0/3343821424 >> 172.21.15.90:0/3343821424 conn(0x7f2aac102610 msgr2=0x7f2aac104a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:04:11.826 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:11.320+0000 7f2ab16be700 1 -- 172.21.15.90:0/3343821424 shutdown_connections 2024-04-14T15:04:11.826 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:11.320+0000 7f2ab16be700 1 -- 172.21.15.90:0/3343821424 wait complete. 2024-04-14T15:04:11.826 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:11.320+0000 7f2ab16be700 1 librados: shutdown 2024-04-14T15:04:11.826 INFO:teuthology.orchestra.run.smithi090.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-14T15:04:11.843 INFO:teuthology.orchestra.run.smithi090.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-14T15:04:11.843 INFO:teuthology.orchestra.run.smithi090.stdout:Creating mgr... 2024-04-14T15:04:11.844 INFO:teuthology.orchestra.run.smithi090.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-14T15:04:11.844 INFO:teuthology.orchestra.run.smithi090.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-14T15:04:12.091 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-0e288098-fa70-11ee-bc8e-c7b262605968@mgr.y 2024-04-14T15:04:12.092 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stderr Failed to reset failed state of unit ceph-0e288098-fa70-11ee-bc8e-c7b262605968@mgr.y.service: Unit ceph-0e288098-fa70-11ee-bc8e-c7b262605968@mgr.y.service not loaded. 2024-04-14T15:04:12.237 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-0e288098-fa70-11ee-bc8e-c7b262605968.target.wants/ceph-0e288098-fa70-11ee-bc8e-c7b262605968@mgr.y.service → /etc/systemd/system/ceph-0e288098-fa70-11ee-bc8e-c7b262605968@.service. 2024-04-14T15:04:12.565 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:12 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3343821424' entity='client.admin' 2024-04-14T15:04:12.565 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:12 smithi090 podman[30641]: 2024-04-14 15:04:12.371976022 +0000 UTC m=+0.017631346 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T15:04:12.773 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-14T15:04:12.773 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout disabled 2024-04-14T15:04:12.781 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-14T15:04:12.781 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout inactive 2024-04-14T15:04:12.781 INFO:teuthology.orchestra.run.smithi090.stdout:firewalld.service is not enabled 2024-04-14T15:04:12.781 INFO:teuthology.orchestra.run.smithi090.stdout:Not possible to enable service . firewalld.service is not available 2024-04-14T15:04:12.789 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-14T15:04:12.789 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout disabled 2024-04-14T15:04:12.796 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-14T15:04:12.796 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout inactive 2024-04-14T15:04:12.796 INFO:teuthology.orchestra.run.smithi090.stdout:firewalld.service is not enabled 2024-04-14T15:04:12.797 INFO:teuthology.orchestra.run.smithi090.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-04-14T15:04:12.797 INFO:teuthology.orchestra.run.smithi090.stdout:Waiting for mgr to start... 2024-04-14T15:04:12.797 INFO:teuthology.orchestra.run.smithi090.stdout:Waiting for mgr... 2024-04-14T15:04:12.921 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:12 smithi090 podman[30641]: 2024-04-14 15:04:12.659259968 +0000 UTC m=+0.304915292 container init eb9ab6f9fe2001a8c8f0c236fc55a58b39133bc2d933d65b2f688b6110e3fff6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=main-a971fd5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS) 2024-04-14T15:04:12.921 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:12 smithi090 podman[30641]: 2024-04-14 15:04:12.664446541 +0000 UTC m=+0.310101866 container start eb9ab6f9fe2001a8c8f0c236fc55a58b39133bc2d933d65b2f688b6110e3fff6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=main-a971fd5, GIT_CLEAN=True, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-14T15:04:12.921 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:12 smithi090 bash[30641]: eb9ab6f9fe2001a8c8f0c236fc55a58b39133bc2d933d65b2f688b6110e3fff6 2024-04-14T15:04:12.921 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:12 smithi090 systemd[1]: Started Ceph mgr.y for 0e288098-fa70-11ee-bc8e-c7b262605968. 2024-04-14T15:04:13.691 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:13 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/988325006' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-14T15:04:13.692 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:13 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:04:13.224+0000 7f6eeeca1200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-14T15:04:13.692 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:13 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:04:13.323+0000 7f6eeeca1200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-14T15:04:14.322 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:13 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:04:13.983+0000 7f6eeeca1200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-14T15:04:14.323 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:14 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:04:14.075+0000 7f6eeeca1200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-14T15:04:14.323 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:14 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:04:14.225+0000 7f6eeeca1200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-14T15:04:14.323 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:14 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:04:14.321+0000 7f6eeeca1200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-14T15:04:14.666 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:14 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:04:14.403+0000 7f6eeeca1200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-14T15:04:14.666 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:14 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:04:14.665+0000 7f6eeeca1200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-14T15:04:14.941 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:14 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:04:14.774+0000 7f6eeeca1200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-14T15:04:14.942 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:14 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:04:14.859+0000 7f6eeeca1200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-14T15:04:15.441 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:14 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:04:14.953+0000 7f6eeeca1200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-14T15:04:15.442 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:15 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:04:15.131+0000 7f6eeeca1200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-14T15:04:16.441 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:16 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:04:16.107+0000 7f6eeeca1200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-14T15:04:16.441 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:16 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:04:16.238+0000 7f6eeeca1200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-14T15:04:16.851 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:16 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:04:16.569+0000 7f6eeeca1200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-14T15:04:16.851 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:16 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:04:16.657+0000 7f6eeeca1200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-14T15:04:16.851 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:16 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:04:16.850+0000 7f6eeeca1200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-14T15:04:17.191 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:17 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:04:17.032+0000 7f6eeeca1200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-14T15:04:17.672 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:17 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:04:17.314+0000 7f6eeeca1200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-14T15:04:17.672 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:17 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:04:17.572+0000 7f6eeeca1200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-14T15:04:17.941 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:17 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:04:17.807+0000 7f6eeeca1200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-14T15:04:18.941 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:18 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:04:18.609+0000 7f6eeeca1200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-14T15:04:19.441 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:19 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:04:19.174+0000 7f6eeeca1200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-14T15:04:19.441 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:19 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:04:19.269+0000 7f6eeeca1200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-14T15:04:19.942 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:19 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:04:19.484+0000 7f6eeeca1200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-14T15:04:19.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:19 smithi090 ceph-mon[30399]: Activating manager daemon y 2024-04-14T15:04:19.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:19 smithi090 ceph-mon[30399]: mgrmap e2: y(active, starting, since 0.00299364s) 2024-04-14T15:04:19.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:19 smithi090 ceph-mon[30399]: from='mgr.14100 v1:172.21.15.90:0/3581064477' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-14T15:04:19.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:19 smithi090 ceph-mon[30399]: from='mgr.14100 v1:172.21.15.90:0/3581064477' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-14T15:04:19.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:19 smithi090 ceph-mon[30399]: from='mgr.14100 v1:172.21.15.90:0/3581064477' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-14T15:04:19.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:19 smithi090 ceph-mon[30399]: from='mgr.14100 v1:172.21.15.90:0/3581064477' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-14T15:04:19.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:19 smithi090 ceph-mon[30399]: from='mgr.14100 v1:172.21.15.90:0/3581064477' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-14T15:04:19.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:19 smithi090 ceph-mon[30399]: from='mgr.14100 v1:172.21.15.90:0/3581064477' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-14T15:04:19.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:19 smithi090 ceph-mon[30399]: from='mgr.14100 v1:172.21.15.90:0/3581064477' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T15:04:19.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:19 smithi090 ceph-mon[30399]: from='mgr.14100 v1:172.21.15.90:0/3581064477' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-14T15:04:19.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:19 smithi090 ceph-mon[30399]: Manager daemon y is now available 2024-04-14T15:04:19.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:19 smithi090 ceph-mon[30399]: from='mgr.14100 v1:172.21.15.90:0/3581064477' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T15:04:19.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:19 smithi090 ceph-mon[30399]: from='mgr.14100 v1:172.21.15.90:0/3581064477' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T15:04:19.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:19 smithi090 ceph-mon[30399]: from='mgr.14100 v1:172.21.15.90:0/3581064477' entity='mgr.y' 2024-04-14T15:04:19.944 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:19 smithi090 ceph-mon[30399]: from='mgr.14100 v1:172.21.15.90:0/3581064477' entity='mgr.y' 2024-04-14T15:04:19.944 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:19 smithi090 ceph-mon[30399]: from='mgr.14100 v1:172.21.15.90:0/3581064477' entity='mgr.y' 2024-04-14T15:04:21.024 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 2024-04-14T15:04:21.027 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout { 2024-04-14T15:04:21.027 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "fsid": "0e288098-fa70-11ee-bc8e-c7b262605968", 2024-04-14T15:04:21.029 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "health": { 2024-04-14T15:04:21.029 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-14T15:04:21.029 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-14T15:04:21.029 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-14T15:04:21.029 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-14T15:04:21.030 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-14T15:04:21.030 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-14T15:04:21.030 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 0 2024-04-14T15:04:21.030 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout ], 2024-04-14T15:04:21.030 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-14T15:04:21.030 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "a" 2024-04-14T15:04:21.030 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout ], 2024-04-14T15:04:21.030 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-04-14T15:04:21.030 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-14T15:04:21.030 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-14T15:04:21.030 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-14T15:04:21.030 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-14T15:04:21.030 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-14T15:04:21.031 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-14T15:04:21.031 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-14T15:04:21.031 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-14T15:04:21.031 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-14T15:04:21.031 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-14T15:04:21.040 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-14T15:04:21.040 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-14T15:04:21.040 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-14T15:04:21.040 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-14T15:04:21.040 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-14T15:04:21.040 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-14T15:04:21.040 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-14T15:04:21.040 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-14T15:04:21.040 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-14T15:04:21.041 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-14T15:04:21.041 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-14T15:04:21.041 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-14T15:04:21.041 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-14T15:04:21.041 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-14T15:04:21.041 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-14T15:04:21.041 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-14T15:04:21.041 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-14T15:04:21.041 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-14T15:04:21.041 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-14T15:04:21.041 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-14T15:04:21.042 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-14T15:04:21.042 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-14T15:04:21.042 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-14T15:04:21.042 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-14T15:04:21.042 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-14T15:04:21.042 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "restful" 2024-04-14T15:04:21.042 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout ], 2024-04-14T15:04:21.042 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-14T15:04:21.042 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-14T15:04:21.042 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-14T15:04:21.042 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-14T15:04:21.043 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "modified": "2024-04-14T15:04:05.808764+0000", 2024-04-14T15:04:21.043 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-14T15:04:21.043 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-14T15:04:21.043 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-14T15:04:21.043 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout } 2024-04-14T15:04:21.043 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:13.249+0000 7ff5f1e2a700 1 Processor -- start 2024-04-14T15:04:21.043 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:13.249+0000 7ff5f1e2a700 1 -- start start 2024-04-14T15:04:21.043 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:13.249+0000 7ff5f1e2a700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff5ec075bf0 con 0x7ff5ec105960 2024-04-14T15:04:21.043 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:13.249+0000 7ff5eb7fe700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7ff5ec105960 0x7ff5ec105d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:55148/0 (socket says 172.21.15.90:55148) 2024-04-14T15:04:21.043 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:13.249+0000 7ff5eb7fe700 1 -- 172.21.15.90:0/1758173194 learned_addr learned my addr 172.21.15.90:0/1758173194 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:04:21.043 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:13.249+0000 7ff5ea7fc700 1 -- 172.21.15.90:0/1758173194 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 535641896 0 0) 0x7ff5ec075bf0 con 0x7ff5ec105960 2024-04-14T15:04:21.043 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:13.249+0000 7ff5ea7fc700 1 -- 172.21.15.90:0/1758173194 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff5d40035b0 con 0x7ff5ec105960 2024-04-14T15:04:21.044 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:13.250+0000 7ff5ea7fc700 1 -- 172.21.15.90:0/1758173194 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2422939957 0 0) 0x7ff5d40035b0 con 0x7ff5ec105960 2024-04-14T15:04:21.044 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:13.250+0000 7ff5ea7fc700 1 -- 172.21.15.90:0/1758173194 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5ec1074a0 con 0x7ff5ec105960 2024-04-14T15:04:21.044 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:13.250+0000 7ff5ea7fc700 1 -- 172.21.15.90:0/1758173194 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7ff5dc001580 con 0x7ff5ec105960 2024-04-14T15:04:21.044 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:13.250+0000 7ff5ea7fc700 1 -- 172.21.15.90:0/1758173194 <== mon.0 v1:172.21.15.90:6789/0 4 ==== config(26 keys) ==== 1050+0+0 (unknown 2744544961 0 0) 0x7ff5dc001d30 con 0x7ff5ec105960 2024-04-14T15:04:21.044 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:13.250+0000 7ff5ea7fc700 1 -- 172.21.15.90:0/1758173194 <== mon.0 v1:172.21.15.90:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7ff5dc005960 con 0x7ff5ec105960 2024-04-14T15:04:21.044 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:13.251+0000 7ff5f1e2a700 1 -- 172.21.15.90:0/1758173194 >> v1:172.21.15.90:6789/0 conn(0x7ff5ec105960 legacy=0x7ff5ec105d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:04:21.044 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:13.251+0000 7ff5f1e2a700 1 -- 172.21.15.90:0/1758173194 shutdown_connections 2024-04-14T15:04:21.044 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:13.251+0000 7ff5f1e2a700 1 -- 172.21.15.90:0/1758173194 >> 172.21.15.90:0/1758173194 conn(0x7ff5ec101500 msgr2=0x7ff5ec103960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:04:21.044 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:13.251+0000 7ff5f1e2a700 1 -- 172.21.15.90:0/1758173194 shutdown_connections 2024-04-14T15:04:21.044 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:13.251+0000 7ff5f1e2a700 1 -- 172.21.15.90:0/1758173194 wait complete. 2024-04-14T15:04:21.045 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:13.251+0000 7ff5f1e2a700 1 librados: starting msgr at 2024-04-14T15:04:21.045 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:13.251+0000 7ff5f1e2a700 1 librados: starting objecter 2024-04-14T15:04:21.045 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:13.251+0000 7ff5f1e2a700 1 Processor -- start 2024-04-14T15:04:21.045 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:13.251+0000 7ff5f1e2a700 1 -- start start 2024-04-14T15:04:21.045 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:13.251+0000 7ff5f1e2a700 1 librados: setting wanted keys 2024-04-14T15:04:21.045 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:13.251+0000 7ff5f1e2a700 1 librados: calling monclient init 2024-04-14T15:04:21.045 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:13.252+0000 7ff5f1e2a700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff5ec19d570 con 0x7ff5ec105960 2024-04-14T15:04:21.045 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:13.252+0000 7ff5eb7fe700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7ff5ec105960 0x7ff5ec19ce60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:55152/0 (socket says 172.21.15.90:55152) 2024-04-14T15:04:21.045 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:13.252+0000 7ff5eb7fe700 1 -- 172.21.15.90:0/988325006 learned_addr learned my addr 172.21.15.90:0/988325006 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:04:21.045 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:13.252+0000 7ff5e8ff9700 10 client.?.objecter ms_handle_connect 0x7ff5ec105960 2024-04-14T15:04:21.045 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:13.252+0000 7ff5e8ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:04:21.046 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:13.252+0000 7ff5e8ff9700 1 -- 172.21.15.90:0/988325006 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1046049733 0 0) 0x7ff5ec19d570 con 0x7ff5ec105960 2024-04-14T15:04:21.046 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:13.252+0000 7ff5e8ff9700 1 -- 172.21.15.90:0/988325006 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff5d40056e0 con 0x7ff5ec105960 2024-04-14T15:04:21.046 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:13.252+0000 7ff5e8ff9700 1 -- 172.21.15.90:0/988325006 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1437390379 0 0) 0x7ff5ec1074a0 con 0x7ff5ec105960 2024-04-14T15:04:21.046 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:13.252+0000 7ff5e8ff9700 1 -- 172.21.15.90:0/988325006 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff5ec19d570 con 0x7ff5ec105960 2024-04-14T15:04:21.046 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:13.252+0000 7ff5e8ff9700 1 -- 172.21.15.90:0/988325006 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7ff5dc001080 con 0x7ff5ec105960 2024-04-14T15:04:21.046 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:13.252+0000 7ff5e8ff9700 1 -- 172.21.15.90:0/988325006 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1194153004 0 0) 0x7ff5ec19d570 con 0x7ff5ec105960 2024-04-14T15:04:21.046 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:13.252+0000 7ff5e8ff9700 1 -- 172.21.15.90:0/988325006 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5ec19d740 con 0x7ff5ec105960 2024-04-14T15:04:21.046 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:13.252+0000 7ff5f1e2a700 1 -- 172.21.15.90:0/988325006 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff5ec19da30 con 0x7ff5ec105960 2024-04-14T15:04:21.046 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:13.252+0000 7ff5f1e2a700 10 client.14102.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:04:21.046 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:13.252+0000 7ff5f1e2a700 1 -- 172.21.15.90:0/988325006 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff5ec1a1580 con 0x7ff5ec105960 2024-04-14T15:04:21.046 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:13.252+0000 7ff5f1e2a700 1 librados: init done 2024-04-14T15:04:21.047 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:13.253+0000 7ff5e8ff9700 1 -- 172.21.15.90:0/988325006 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1050+0+0 (unknown 2744544961 0 0) 0x7ff5dc004ab0 con 0x7ff5ec105960 2024-04-14T15:04:21.047 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:13.253+0000 7ff5e8ff9700 1 -- 172.21.15.90:0/988325006 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7ff5dc006340 con 0x7ff5ec105960 2024-04-14T15:04:21.047 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:13.253+0000 7ff5e8ff9700 1 -- 172.21.15.90:0/988325006 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7ff5dc006930 con 0x7ff5ec105960 2024-04-14T15:04:21.047 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:13.253+0000 7ff5e8ff9700 1 -- 172.21.15.90:0/988325006 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 87150692 0 0) 0x7ff5dc007ff0 con 0x7ff5ec105960 2024-04-14T15:04:21.047 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:13.253+0000 7ff5e8ff9700 10 client.14102.objecter ms_dispatch 0x7ff5ec001550 osd_map(1..1 src has 1..1) 2024-04-14T15:04:21.047 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:13.253+0000 7ff5e8ff9700 3 client.14102.objecter handle_osd_map got epochs [1,1] > 0 2024-04-14T15:04:21.047 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:13.253+0000 7ff5e8ff9700 3 client.14102.objecter handle_osd_map decoding full epoch 1 2024-04-14T15:04:21.047 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:13.253+0000 7ff5e8ff9700 20 client.14102.objecter dump_active .. 0 homeless 2024-04-14T15:04:21.047 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:13.254+0000 7ff5f1e2a700 1 -- 172.21.15.90:0/988325006 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5ec10e660 con 0x7ff5ec105960 2024-04-14T15:04:21.047 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:13.257+0000 7ff5e8ff9700 1 -- 172.21.15.90:0/988325006 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (unknown 1092875540 0 2268998900) 0x7ff5d40056e0 con 0x7ff5ec105960 2024-04-14T15:04:21.047 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:13.331+0000 7ff5f1e2a700 1 -- 172.21.15.90:0/988325006 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7ff5ec1a1b00 con 0x7ff5ec105960 2024-04-14T15:04:21.048 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:13.332+0000 7ff5e8ff9700 1 -- 172.21.15.90:0/988325006 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (unknown 4201413639 0 3095052472) 0x7ff5dc005ce0 con 0x7ff5ec105960 2024-04-14T15:04:21.048 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:13.333+0000 7ff5f1e2a700 10 librados: watch_flush enter 2024-04-14T15:04:21.048 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:13.333+0000 7ff5f1e2a700 10 librados: watch_flush exit 2024-04-14T15:04:21.048 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:13.333+0000 7ff5f1e2a700 20 client.14102.objecter shutdown clearing up homeless session... 2024-04-14T15:04:21.048 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:13.333+0000 7ff5f1e2a700 10 client.14102.objecter successfully canceled tick 2024-04-14T15:04:21.048 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:13.333+0000 7ff5f1e2a700 1 -- 172.21.15.90:0/988325006 >> v1:172.21.15.90:6789/0 conn(0x7ff5ec105960 legacy=0x7ff5ec19ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:04:21.048 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:13.333+0000 7ff5f1e2a700 1 -- 172.21.15.90:0/988325006 shutdown_connections 2024-04-14T15:04:21.048 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:13.333+0000 7ff5f1e2a700 1 -- 172.21.15.90:0/988325006 >> 172.21.15.90:0/988325006 conn(0x7ff5ec101500 msgr2=0x7ff5ec103960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:04:21.048 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:13.333+0000 7ff5f1e2a700 1 -- 172.21.15.90:0/988325006 shutdown_connections 2024-04-14T15:04:21.048 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:13.333+0000 7ff5f1e2a700 1 -- 172.21.15.90:0/988325006 wait complete. 2024-04-14T15:04:21.048 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:13.333+0000 7ff5f1e2a700 1 librados: shutdown 2024-04-14T15:04:21.049 INFO:teuthology.orchestra.run.smithi090.stdout:mgr not available, waiting (1/15)... 2024-04-14T15:04:21.941 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:21 smithi090 ceph-mon[30399]: mgrmap e3: y(active, since 1.00703s) 2024-04-14T15:04:22.691 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:22 smithi090 ceph-mon[30399]: mgrmap e4: y(active, since 2s) 2024-04-14T15:04:27.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:27 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2905546529' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-14T15:04:30.374 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 2024-04-14T15:04:30.386 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout { 2024-04-14T15:04:30.387 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "fsid": "0e288098-fa70-11ee-bc8e-c7b262605968", 2024-04-14T15:04:30.387 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "health": { 2024-04-14T15:04:30.387 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-14T15:04:30.387 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-14T15:04:30.387 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-14T15:04:30.387 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-14T15:04:30.387 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-14T15:04:30.387 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-14T15:04:30.387 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 0 2024-04-14T15:04:30.387 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout ], 2024-04-14T15:04:30.387 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-14T15:04:30.387 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "a" 2024-04-14T15:04:30.387 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout ], 2024-04-14T15:04:30.387 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "quorum_age": 16, 2024-04-14T15:04:30.387 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-14T15:04:30.388 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-14T15:04:30.388 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-14T15:04:30.388 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-14T15:04:30.388 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-14T15:04:30.388 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-14T15:04:30.388 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-14T15:04:30.388 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-14T15:04:30.388 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-14T15:04:30.388 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-14T15:04:30.388 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-14T15:04:30.388 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-14T15:04:30.388 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-14T15:04:30.388 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-14T15:04:30.388 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-14T15:04:30.388 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-14T15:04:30.389 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-14T15:04:30.389 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-14T15:04:30.389 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-14T15:04:30.389 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-14T15:04:30.389 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-14T15:04:30.389 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-14T15:04:30.389 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-14T15:04:30.389 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-14T15:04:30.389 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-14T15:04:30.389 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-14T15:04:30.389 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-14T15:04:30.389 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-14T15:04:30.389 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-14T15:04:30.389 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-14T15:04:30.389 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-14T15:04:30.390 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-14T15:04:30.390 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-14T15:04:30.390 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-14T15:04:30.390 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-14T15:04:30.390 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "restful" 2024-04-14T15:04:30.390 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout ], 2024-04-14T15:04:30.390 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-14T15:04:30.390 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-14T15:04:30.390 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-14T15:04:30.390 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-14T15:04:30.390 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "modified": "2024-04-14T15:04:05.808764+0000", 2024-04-14T15:04:30.390 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-14T15:04:30.390 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-14T15:04:30.390 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-14T15:04:30.390 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout } 2024-04-14T15:04:30.391 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:26.914+0000 7f1e570fc700 1 Processor -- start 2024-04-14T15:04:30.391 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:26.915+0000 7f1e570fc700 1 -- start start 2024-04-14T15:04:30.391 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:26.915+0000 7f1e570fc700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1e5010b4c0 con 0x7f1e50107090 2024-04-14T15:04:30.391 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:26.915+0000 7f1e54e98700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f1e50107090 0x7f1e501074b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:32796/0 (socket says 172.21.15.90:32796) 2024-04-14T15:04:30.391 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:26.915+0000 7f1e54e98700 1 -- 172.21.15.90:0/1845919699 learned_addr learned my addr 172.21.15.90:0/1845919699 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:04:30.391 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:26.915+0000 7f1e4f7fe700 1 -- 172.21.15.90:0/1845919699 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1862758588 0 0) 0x7f1e5010b4c0 con 0x7f1e50107090 2024-04-14T15:04:30.391 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:26.915+0000 7f1e4f7fe700 1 -- 172.21.15.90:0/1845919699 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1e400035b0 con 0x7f1e50107090 2024-04-14T15:04:30.391 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:26.916+0000 7f1e4f7fe700 1 -- 172.21.15.90:0/1845919699 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2356861514 0 0) 0x7f1e400035b0 con 0x7f1e50107090 2024-04-14T15:04:30.391 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:26.916+0000 7f1e4f7fe700 1 -- 172.21.15.90:0/1845919699 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e5010c6a0 con 0x7f1e50107090 2024-04-14T15:04:30.391 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:26.916+0000 7f1e4f7fe700 1 -- 172.21.15.90:0/1845919699 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f1e44001580 con 0x7f1e50107090 2024-04-14T15:04:30.391 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:26.916+0000 7f1e4f7fe700 1 -- 172.21.15.90:0/1845919699 <== mon.0 v1:172.21.15.90:6789/0 4 ==== config(26 keys) ==== 1050+0+0 (unknown 2744544961 0 0) 0x7f1e44001d30 con 0x7f1e50107090 2024-04-14T15:04:30.391 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:26.916+0000 7f1e4f7fe700 1 -- 172.21.15.90:0/1845919699 <== mon.0 v1:172.21.15.90:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f1e44005960 con 0x7f1e50107090 2024-04-14T15:04:30.391 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:26.916+0000 7f1e570fc700 1 -- 172.21.15.90:0/1845919699 >> v1:172.21.15.90:6789/0 conn(0x7f1e50107090 legacy=0x7f1e501074b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:04:30.391 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:26.916+0000 7f1e570fc700 1 -- 172.21.15.90:0/1845919699 shutdown_connections 2024-04-14T15:04:30.391 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:26.916+0000 7f1e570fc700 1 -- 172.21.15.90:0/1845919699 >> 172.21.15.90:0/1845919699 conn(0x7f1e50102610 msgr2=0x7f1e50104a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:04:30.392 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:26.916+0000 7f1e570fc700 1 -- 172.21.15.90:0/1845919699 shutdown_connections 2024-04-14T15:04:30.392 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:26.916+0000 7f1e570fc700 1 -- 172.21.15.90:0/1845919699 wait complete. 2024-04-14T15:04:30.392 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:26.917+0000 7f1e570fc700 1 librados: starting msgr at 2024-04-14T15:04:30.392 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:26.917+0000 7f1e570fc700 1 librados: starting objecter 2024-04-14T15:04:30.392 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:26.917+0000 7f1e570fc700 1 Processor -- start 2024-04-14T15:04:30.392 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:26.917+0000 7f1e570fc700 1 -- start start 2024-04-14T15:04:30.392 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:26.917+0000 7f1e570fc700 1 librados: setting wanted keys 2024-04-14T15:04:30.392 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:26.917+0000 7f1e570fc700 1 librados: calling monclient init 2024-04-14T15:04:30.392 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:26.917+0000 7f1e570fc700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1e5019b3b0 con 0x7f1e50107090 2024-04-14T15:04:30.392 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:26.917+0000 7f1e54e98700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f1e50107090 0x7f1e5019aca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:32808/0 (socket says 172.21.15.90:32808) 2024-04-14T15:04:30.392 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:26.917+0000 7f1e54e98700 1 -- 172.21.15.90:0/2905546529 learned_addr learned my addr 172.21.15.90:0/2905546529 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:04:30.392 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:26.917+0000 7f1e4dffb700 10 client.?.objecter ms_handle_connect 0x7f1e50107090 2024-04-14T15:04:30.392 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:26.917+0000 7f1e4dffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:04:30.392 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:26.917+0000 7f1e4dffb700 1 -- 172.21.15.90:0/2905546529 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3802239848 0 0) 0x7f1e5019b3b0 con 0x7f1e50107090 2024-04-14T15:04:30.393 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:26.917+0000 7f1e4dffb700 1 -- 172.21.15.90:0/2905546529 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1e400056e0 con 0x7f1e50107090 2024-04-14T15:04:30.393 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:26.918+0000 7f1e4dffb700 1 -- 172.21.15.90:0/2905546529 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2994904703 0 0) 0x7f1e5010c6a0 con 0x7f1e50107090 2024-04-14T15:04:30.393 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:26.918+0000 7f1e4dffb700 1 -- 172.21.15.90:0/2905546529 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1e5019b3b0 con 0x7f1e50107090 2024-04-14T15:04:30.393 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:26.918+0000 7f1e4dffb700 1 -- 172.21.15.90:0/2905546529 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f1e44004870 con 0x7f1e50107090 2024-04-14T15:04:30.393 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:26.918+0000 7f1e4dffb700 1 -- 172.21.15.90:0/2905546529 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1993149518 0 0) 0x7f1e5019b3b0 con 0x7f1e50107090 2024-04-14T15:04:30.393 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:26.918+0000 7f1e4dffb700 1 -- 172.21.15.90:0/2905546529 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e5019b580 con 0x7f1e50107090 2024-04-14T15:04:30.393 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:26.918+0000 7f1e570fc700 1 -- 172.21.15.90:0/2905546529 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1e5019b870 con 0x7f1e50107090 2024-04-14T15:04:30.393 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:26.918+0000 7f1e4dffb700 1 -- 172.21.15.90:0/2905546529 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1050+0+0 (unknown 2744544961 0 0) 0x7f1e44004a40 con 0x7f1e50107090 2024-04-14T15:04:30.393 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:26.918+0000 7f1e4dffb700 1 -- 172.21.15.90:0/2905546529 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f1e44006350 con 0x7f1e50107090 2024-04-14T15:04:30.393 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:26.918+0000 7f1e570fc700 10 client.14110.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:04:30.393 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:26.918+0000 7f1e570fc700 1 -- 172.21.15.90:0/2905546529 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f1e5019f3c0 con 0x7f1e50107090 2024-04-14T15:04:30.393 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:26.918+0000 7f1e570fc700 1 librados: init done 2024-04-14T15:04:30.393 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:26.919+0000 7f1e4dffb700 1 -- 172.21.15.90:0/2905546529 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 4) ==== 46512+0+0 (unknown 367752499 0 0) 0x7f1e44011c20 con 0x7f1e50107090 2024-04-14T15:04:30.393 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:26.919+0000 7f1e570fc700 1 -- 172.21.15.90:0/2905546529 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e50064110 con 0x7f1e50107090 2024-04-14T15:04:30.394 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:26.919+0000 7f1e4dffb700 1 -- 172.21.15.90:0/2905546529 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 87150692 0 0) 0x7f1e44047f40 con 0x7f1e50107090 2024-04-14T15:04:30.394 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:26.919+0000 7f1e4dffb700 10 client.14110.objecter ms_dispatch 0x7f1e50001550 osd_map(1..1 src has 1..1) 2024-04-14T15:04:30.394 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:26.919+0000 7f1e4dffb700 3 client.14110.objecter handle_osd_map got epochs [1,1] > 0 2024-04-14T15:04:30.394 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:26.919+0000 7f1e4dffb700 3 client.14110.objecter handle_osd_map decoding full epoch 1 2024-04-14T15:04:30.394 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:26.919+0000 7f1e4dffb700 20 client.14110.objecter dump_active .. 0 homeless 2024-04-14T15:04:30.394 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:26.919+0000 7f1e4dffb700 10 client.14110.objecter ms_handle_connect 0x7f1e4003c120 2024-04-14T15:04:30.394 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:26.924+0000 7f1e4dffb700 1 -- 172.21.15.90:0/2905546529 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f1e400056e0 con 0x7f1e50107090 2024-04-14T15:04:30.394 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:27.202+0000 7f1e570fc700 1 -- 172.21.15.90:0/2905546529 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f1e50194140 con 0x7f1e50107090 2024-04-14T15:04:30.394 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:27.202+0000 7f1e4dffb700 1 -- 172.21.15.90:0/2905546529 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (unknown 4201413639 0 1948414751) 0x7f1e50194140 con 0x7f1e50107090 2024-04-14T15:04:30.394 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:27.204+0000 7f1e570fc700 10 librados: watch_flush enter 2024-04-14T15:04:30.394 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:27.204+0000 7f1e570fc700 10 librados: watch_flush exit 2024-04-14T15:04:30.394 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:27.204+0000 7f1e570fc700 20 client.14110.objecter shutdown clearing up homeless session... 2024-04-14T15:04:30.394 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:27.204+0000 7f1e570fc700 10 client.14110.objecter successfully canceled tick 2024-04-14T15:04:30.394 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:27.204+0000 7f1e570fc700 1 -- 172.21.15.90:0/2905546529 >> v1:172.21.15.90:6800/2306258962 conn(0x7f1e4003c120 legacy=0x7f1e4003e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:04:30.395 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:27.204+0000 7f1e570fc700 1 -- 172.21.15.90:0/2905546529 >> v1:172.21.15.90:6789/0 conn(0x7f1e50107090 legacy=0x7f1e5019aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:04:30.395 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:27.204+0000 7f1e570fc700 1 -- 172.21.15.90:0/2905546529 shutdown_connections 2024-04-14T15:04:30.395 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:27.204+0000 7f1e570fc700 1 -- 172.21.15.90:0/2905546529 >> 172.21.15.90:0/2905546529 conn(0x7f1e50102610 msgr2=0x7f1e50104a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:04:30.395 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:27.204+0000 7f1e570fc700 1 -- 172.21.15.90:0/2905546529 shutdown_connections 2024-04-14T15:04:30.395 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:27.204+0000 7f1e570fc700 1 -- 172.21.15.90:0/2905546529 wait complete. 2024-04-14T15:04:30.395 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:27.204+0000 7f1e570fc700 1 librados: shutdown 2024-04-14T15:04:30.395 INFO:teuthology.orchestra.run.smithi090.stdout:mgr is available 2024-04-14T15:04:31.387 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:31 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/628148210' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-14T15:04:31.668 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 2024-04-14T15:04:31.669 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout [global] 2024-04-14T15:04:31.669 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout fsid = 0e288098-fa70-11ee-bc8e-c7b262605968 2024-04-14T15:04:31.669 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-14T15:04:31.669 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-14T15:04:31.669 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-14T15:04:31.669 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-14T15:04:31.669 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 2024-04-14T15:04:31.669 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-14T15:04:31.670 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-14T15:04:31.670 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 2024-04-14T15:04:31.670 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout [osd] 2024-04-14T15:04:31.670 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-14T15:04:31.670 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-14T15:04:31.670 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:30.842+0000 7fa1f2e98700 1 Processor -- start 2024-04-14T15:04:31.670 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:30.842+0000 7fa1f2e98700 1 -- start start 2024-04-14T15:04:31.670 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:30.842+0000 7fa1f2e98700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa1ec10b4d0 con 0x7fa1ec1070a0 2024-04-14T15:04:31.671 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:30.842+0000 7fa1f0c34700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fa1ec1070a0 0x7fa1ec1074c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:39478/0 (socket says 172.21.15.90:39478) 2024-04-14T15:04:31.671 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:30.842+0000 7fa1f0c34700 1 -- 172.21.15.90:0/1356078389 learned_addr learned my addr 172.21.15.90:0/1356078389 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:04:31.671 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:30.843+0000 7fa1e37fe700 1 -- 172.21.15.90:0/1356078389 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1155427418 0 0) 0x7fa1ec10b4d0 con 0x7fa1ec1070a0 2024-04-14T15:04:31.671 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:30.843+0000 7fa1e37fe700 1 -- 172.21.15.90:0/1356078389 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa1d40035b0 con 0x7fa1ec1070a0 2024-04-14T15:04:31.671 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:30.843+0000 7fa1e37fe700 1 -- 172.21.15.90:0/1356078389 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 800161803 0 0) 0x7fa1d40035b0 con 0x7fa1ec1070a0 2024-04-14T15:04:31.671 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:30.843+0000 7fa1e37fe700 1 -- 172.21.15.90:0/1356078389 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1ec10c6b0 con 0x7fa1ec1070a0 2024-04-14T15:04:31.671 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:30.843+0000 7fa1e37fe700 1 -- 172.21.15.90:0/1356078389 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7fa1e8001900 con 0x7fa1ec1070a0 2024-04-14T15:04:31.671 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:30.843+0000 7fa1e37fe700 1 -- 172.21.15.90:0/1356078389 <== mon.0 v1:172.21.15.90:6789/0 4 ==== config(26 keys) ==== 1050+0+0 (unknown 2744544961 0 0) 0x7fa1e80020b0 con 0x7fa1ec1070a0 2024-04-14T15:04:31.671 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:30.843+0000 7fa1e37fe700 1 -- 172.21.15.90:0/1356078389 <== mon.0 v1:172.21.15.90:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7fa1e8005ce0 con 0x7fa1ec1070a0 2024-04-14T15:04:31.672 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:30.843+0000 7fa1f2e98700 1 -- 172.21.15.90:0/1356078389 >> v1:172.21.15.90:6789/0 conn(0x7fa1ec1070a0 legacy=0x7fa1ec1074c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:04:31.672 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:30.844+0000 7fa1f2e98700 1 -- 172.21.15.90:0/1356078389 shutdown_connections 2024-04-14T15:04:31.672 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:30.844+0000 7fa1f2e98700 1 -- 172.21.15.90:0/1356078389 >> 172.21.15.90:0/1356078389 conn(0x7fa1ec102640 msgr2=0x7fa1ec104a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:04:31.672 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:30.844+0000 7fa1f2e98700 1 -- 172.21.15.90:0/1356078389 shutdown_connections 2024-04-14T15:04:31.672 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:30.844+0000 7fa1f2e98700 1 -- 172.21.15.90:0/1356078389 wait complete. 2024-04-14T15:04:31.672 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:30.844+0000 7fa1f2e98700 1 librados: starting msgr at 2024-04-14T15:04:31.673 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:30.844+0000 7fa1f2e98700 1 librados: starting objecter 2024-04-14T15:04:31.673 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:30.844+0000 7fa1f2e98700 1 Processor -- start 2024-04-14T15:04:31.673 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:30.844+0000 7fa1f2e98700 1 -- start start 2024-04-14T15:04:31.673 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:30.844+0000 7fa1f2e98700 1 librados: setting wanted keys 2024-04-14T15:04:31.673 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:30.844+0000 7fa1f2e98700 1 librados: calling monclient init 2024-04-14T15:04:31.673 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:30.844+0000 7fa1f2e98700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa1ec19b3c0 con 0x7fa1ec1070a0 2024-04-14T15:04:31.673 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:30.844+0000 7fa1f0c34700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fa1ec1070a0 0x7fa1ec19acb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:39482/0 (socket says 172.21.15.90:39482) 2024-04-14T15:04:31.673 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:30.844+0000 7fa1f0c34700 1 -- 172.21.15.90:0/628148210 learned_addr learned my addr 172.21.15.90:0/628148210 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:04:31.673 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:30.844+0000 7fa1e1ffb700 10 client.?.objecter ms_handle_connect 0x7fa1ec1070a0 2024-04-14T15:04:31.673 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:30.844+0000 7fa1e1ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:04:31.674 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:30.845+0000 7fa1e1ffb700 1 -- 172.21.15.90:0/628148210 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 685510445 0 0) 0x7fa1ec19b3c0 con 0x7fa1ec1070a0 2024-04-14T15:04:31.674 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:30.845+0000 7fa1e1ffb700 1 -- 172.21.15.90:0/628148210 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa1d40056e0 con 0x7fa1ec1070a0 2024-04-14T15:04:31.674 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:30.845+0000 7fa1e1ffb700 1 -- 172.21.15.90:0/628148210 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 871988602 0 0) 0x7fa1ec10c6b0 con 0x7fa1ec1070a0 2024-04-14T15:04:31.674 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:30.845+0000 7fa1e1ffb700 1 -- 172.21.15.90:0/628148210 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa1ec19b3c0 con 0x7fa1ec1070a0 2024-04-14T15:04:31.674 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:30.845+0000 7fa1e1ffb700 1 -- 172.21.15.90:0/628148210 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7fa1e8004bf0 con 0x7fa1ec1070a0 2024-04-14T15:04:31.674 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:30.845+0000 7fa1e1ffb700 1 -- 172.21.15.90:0/628148210 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 9299679 0 0) 0x7fa1ec19b3c0 con 0x7fa1ec1070a0 2024-04-14T15:04:31.674 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:30.845+0000 7fa1e1ffb700 1 -- 172.21.15.90:0/628148210 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1ec19b590 con 0x7fa1ec1070a0 2024-04-14T15:04:31.674 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:30.845+0000 7fa1f2e98700 1 -- 172.21.15.90:0/628148210 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa1ec19b880 con 0x7fa1ec1070a0 2024-04-14T15:04:31.674 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:30.845+0000 7fa1e1ffb700 1 -- 172.21.15.90:0/628148210 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1050+0+0 (unknown 2744544961 0 0) 0x7fa1e8004dc0 con 0x7fa1ec1070a0 2024-04-14T15:04:31.675 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:30.845+0000 7fa1e1ffb700 1 -- 172.21.15.90:0/628148210 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7fa1e80065b0 con 0x7fa1ec1070a0 2024-04-14T15:04:31.675 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:30.845+0000 7fa1f2e98700 10 client.14112.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:04:31.675 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:30.845+0000 7fa1f2e98700 1 -- 172.21.15.90:0/628148210 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa1ec19f3d0 con 0x7fa1ec1070a0 2024-04-14T15:04:31.675 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:30.845+0000 7fa1f2e98700 1 librados: init done 2024-04-14T15:04:31.675 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:30.846+0000 7fa1e1ffb700 1 -- 172.21.15.90:0/628148210 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 4) ==== 46512+0+0 (unknown 367752499 0 0) 0x7fa1e8006770 con 0x7fa1ec1070a0 2024-04-14T15:04:31.675 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:30.846+0000 7fa1e1ffb700 1 -- 172.21.15.90:0/628148210 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 87150692 0 0) 0x7fa1e8048060 con 0x7fa1ec1070a0 2024-04-14T15:04:31.675 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:30.846+0000 7fa1e1ffb700 10 client.14112.objecter ms_dispatch 0x7fa1ec001550 osd_map(1..1 src has 1..1) 2024-04-14T15:04:31.675 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:30.846+0000 7fa1e1ffb700 3 client.14112.objecter handle_osd_map got epochs [1,1] > 0 2024-04-14T15:04:31.675 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:30.846+0000 7fa1e1ffb700 3 client.14112.objecter handle_osd_map decoding full epoch 1 2024-04-14T15:04:31.676 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:30.846+0000 7fa1e1ffb700 20 client.14112.objecter dump_active .. 0 homeless 2024-04-14T15:04:31.676 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:30.846+0000 7fa1f2e98700 1 -- 172.21.15.90:0/628148210 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1ec10c8e0 con 0x7fa1ec1070a0 2024-04-14T15:04:31.676 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:30.846+0000 7fa1e1ffb700 10 client.14112.objecter ms_handle_connect 0x7fa1d403c220 2024-04-14T15:04:31.676 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:30.851+0000 7fa1e1ffb700 1 -- 172.21.15.90:0/628148210 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fa1d40056e0 con 0x7fa1ec1070a0 2024-04-14T15:04:31.676 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:31.044+0000 7fa1f2e98700 1 -- 172.21.15.90:0/628148210 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7fa1ec064110 con 0x7fa1ec1070a0 2024-04-14T15:04:31.676 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:31.044+0000 7fa1e1ffb700 1 -- 172.21.15.90:0/628148210 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+282 (unknown 1187553405 0 1973701136) 0x7fa1e8017900 con 0x7fa1ec1070a0 2024-04-14T15:04:31.676 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:31.046+0000 7fa1f2e98700 10 librados: watch_flush enter 2024-04-14T15:04:31.676 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:31.046+0000 7fa1f2e98700 10 librados: watch_flush exit 2024-04-14T15:04:31.676 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:31.046+0000 7fa1f2e98700 20 client.14112.objecter shutdown clearing up homeless session... 2024-04-14T15:04:31.676 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:31.046+0000 7fa1f2e98700 10 client.14112.objecter successfully canceled tick 2024-04-14T15:04:31.676 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:31.046+0000 7fa1f2e98700 1 -- 172.21.15.90:0/628148210 >> v1:172.21.15.90:6800/2306258962 conn(0x7fa1d403c220 legacy=0x7fa1d403e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:04:31.677 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:31.046+0000 7fa1f2e98700 1 -- 172.21.15.90:0/628148210 >> v1:172.21.15.90:6789/0 conn(0x7fa1ec1070a0 legacy=0x7fa1ec19acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:04:31.677 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:31.047+0000 7fa1f2e98700 1 -- 172.21.15.90:0/628148210 shutdown_connections 2024-04-14T15:04:31.677 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:31.047+0000 7fa1f2e98700 1 -- 172.21.15.90:0/628148210 >> 172.21.15.90:0/628148210 conn(0x7fa1ec102640 msgr2=0x7fa1ec104a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:04:31.677 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:31.047+0000 7fa1f2e98700 1 -- 172.21.15.90:0/628148210 shutdown_connections 2024-04-14T15:04:31.678 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:31.047+0000 7fa1f2e98700 1 -- 172.21.15.90:0/628148210 wait complete. 2024-04-14T15:04:31.678 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:31.047+0000 7fa1f2e98700 1 librados: shutdown 2024-04-14T15:04:31.678 INFO:teuthology.orchestra.run.smithi090.stdout:Enabling cephadm module... 2024-04-14T15:04:32.688 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:32 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1841207414' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-14T15:04:32.688 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:32 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ignoring --setuser ceph since I am not root 2024-04-14T15:04:32.689 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:32 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ignoring --setgroup ceph since I am not root 2024-04-14T15:04:32.956 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:32.119+0000 7f47cd8cb700 1 Processor -- start 2024-04-14T15:04:32.956 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:32.119+0000 7f47cd8cb700 1 -- start start 2024-04-14T15:04:32.956 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:32.119+0000 7f47cd8cb700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f47c810a3b0 con 0x7f47c8105f80 2024-04-14T15:04:32.956 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:32.119+0000 7f47c6ffd700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f47c8105f80 0x7f47c81063a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:39498/0 (socket says 172.21.15.90:39498) 2024-04-14T15:04:32.956 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:32.119+0000 7f47c6ffd700 1 -- 172.21.15.90:0/3312937115 learned_addr learned my addr 172.21.15.90:0/3312937115 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:04:32.957 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:32.119+0000 7f47c5ffb700 1 -- 172.21.15.90:0/3312937115 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2292458000 0 0) 0x7f47c810a3b0 con 0x7f47c8105f80 2024-04-14T15:04:32.957 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:32.119+0000 7f47c5ffb700 1 -- 172.21.15.90:0/3312937115 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f47b00035b0 con 0x7f47c8105f80 2024-04-14T15:04:32.957 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:32.119+0000 7f47c5ffb700 1 -- 172.21.15.90:0/3312937115 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1187431775 0 0) 0x7f47b00035b0 con 0x7f47c8105f80 2024-04-14T15:04:32.957 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:32.119+0000 7f47c5ffb700 1 -- 172.21.15.90:0/3312937115 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47c810b590 con 0x7f47c8105f80 2024-04-14T15:04:32.957 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:32.119+0000 7f47c5ffb700 1 -- 172.21.15.90:0/3312937115 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f47b8001580 con 0x7f47c8105f80 2024-04-14T15:04:32.957 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:32.120+0000 7f47c5ffb700 1 -- 172.21.15.90:0/3312937115 <== mon.0 v1:172.21.15.90:6789/0 4 ==== config(26 keys) ==== 1050+0+0 (unknown 2744544961 0 0) 0x7f47b8001d30 con 0x7f47c8105f80 2024-04-14T15:04:32.957 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:32.120+0000 7f47c5ffb700 1 -- 172.21.15.90:0/3312937115 <== mon.0 v1:172.21.15.90:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f47b8005960 con 0x7f47c8105f80 2024-04-14T15:04:32.957 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:32.120+0000 7f47cd8cb700 1 -- 172.21.15.90:0/3312937115 >> v1:172.21.15.90:6789/0 conn(0x7f47c8105f80 legacy=0x7f47c81063a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:04:32.957 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:32.120+0000 7f47cd8cb700 1 -- 172.21.15.90:0/3312937115 shutdown_connections 2024-04-14T15:04:32.957 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:32.120+0000 7f47cd8cb700 1 -- 172.21.15.90:0/3312937115 >> 172.21.15.90:0/3312937115 conn(0x7f47c8101500 msgr2=0x7f47c8103960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:04:32.957 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:32.120+0000 7f47cd8cb700 1 -- 172.21.15.90:0/3312937115 shutdown_connections 2024-04-14T15:04:32.957 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:32.120+0000 7f47cd8cb700 1 -- 172.21.15.90:0/3312937115 wait complete. 2024-04-14T15:04:32.957 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:32.121+0000 7f47cd8cb700 1 librados: starting msgr at 2024-04-14T15:04:32.957 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:32.121+0000 7f47cd8cb700 1 librados: starting objecter 2024-04-14T15:04:32.957 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:32.121+0000 7f47cd8cb700 1 Processor -- start 2024-04-14T15:04:32.957 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:32.121+0000 7f47cd8cb700 1 -- start start 2024-04-14T15:04:32.958 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:32.121+0000 7f47cd8cb700 1 librados: setting wanted keys 2024-04-14T15:04:32.958 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:32.121+0000 7f47cd8cb700 1 librados: calling monclient init 2024-04-14T15:04:32.958 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:32.121+0000 7f47cd8cb700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f47c819b360 con 0x7f47c8105f80 2024-04-14T15:04:32.958 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:32.121+0000 7f47c6ffd700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f47c8105f80 0x7f47c819ac50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:39506/0 (socket says 172.21.15.90:39506) 2024-04-14T15:04:32.958 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:32.121+0000 7f47c6ffd700 1 -- 172.21.15.90:0/1841207414 learned_addr learned my addr 172.21.15.90:0/1841207414 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:04:32.958 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:32.121+0000 7f47cc8c9700 10 client.?.objecter ms_handle_connect 0x7f47c8105f80 2024-04-14T15:04:32.958 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:32.121+0000 7f47cc8c9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:04:32.958 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:32.121+0000 7f47cc8c9700 1 -- 172.21.15.90:0/1841207414 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1113675215 0 0) 0x7f47c819b360 con 0x7f47c8105f80 2024-04-14T15:04:32.958 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:32.121+0000 7f47cc8c9700 1 -- 172.21.15.90:0/1841207414 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f47b00056e0 con 0x7f47c8105f80 2024-04-14T15:04:32.958 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:32.122+0000 7f47cc8c9700 1 -- 172.21.15.90:0/1841207414 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2801860001 0 0) 0x7f47c810b590 con 0x7f47c8105f80 2024-04-14T15:04:32.958 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:32.122+0000 7f47cc8c9700 1 -- 172.21.15.90:0/1841207414 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f47c819b360 con 0x7f47c8105f80 2024-04-14T15:04:32.958 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:32.122+0000 7f47cc8c9700 1 -- 172.21.15.90:0/1841207414 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f47b8004870 con 0x7f47c8105f80 2024-04-14T15:04:32.959 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:32.122+0000 7f47cc8c9700 1 -- 172.21.15.90:0/1841207414 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3567163233 0 0) 0x7f47c819b360 con 0x7f47c8105f80 2024-04-14T15:04:32.959 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:32.122+0000 7f47cc8c9700 1 -- 172.21.15.90:0/1841207414 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47c819b530 con 0x7f47c8105f80 2024-04-14T15:04:32.959 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:32.122+0000 7f47cd8cb700 1 -- 172.21.15.90:0/1841207414 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f47c819b820 con 0x7f47c8105f80 2024-04-14T15:04:32.959 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:32.122+0000 7f47cd8cb700 10 client.14114.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:04:32.959 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:32.122+0000 7f47cd8cb700 1 -- 172.21.15.90:0/1841207414 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f47c819f370 con 0x7f47c8105f80 2024-04-14T15:04:32.959 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:32.122+0000 7f47cd8cb700 1 librados: init done 2024-04-14T15:04:32.959 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:32.122+0000 7f47cc8c9700 1 -- 172.21.15.90:0/1841207414 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1050+0+0 (unknown 2744544961 0 0) 0x7f47b8001be0 con 0x7f47c8105f80 2024-04-14T15:04:32.959 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:32.122+0000 7f47cc8c9700 1 -- 172.21.15.90:0/1841207414 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f47b8005fc0 con 0x7f47c8105f80 2024-04-14T15:04:32.959 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:32.123+0000 7f47cd8cb700 1 -- 172.21.15.90:0/1841207414 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47b4005150 con 0x7f47c8105f80 2024-04-14T15:04:32.959 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:32.123+0000 7f47cc8c9700 1 -- 172.21.15.90:0/1841207414 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 4) ==== 46512+0+0 (unknown 367752499 0 0) 0x7f47b80117a0 con 0x7f47c8105f80 2024-04-14T15:04:32.959 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:32.123+0000 7f47cc8c9700 1 -- 172.21.15.90:0/1841207414 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 87150692 0 0) 0x7f47b8047de0 con 0x7f47c8105f80 2024-04-14T15:04:32.959 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:32.123+0000 7f47cc8c9700 10 client.14114.objecter ms_dispatch 0x7f47c8001550 osd_map(1..1 src has 1..1) 2024-04-14T15:04:32.959 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:32.123+0000 7f47cc8c9700 3 client.14114.objecter handle_osd_map got epochs [1,1] > 0 2024-04-14T15:04:32.959 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:32.123+0000 7f47cc8c9700 3 client.14114.objecter handle_osd_map decoding full epoch 1 2024-04-14T15:04:32.959 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:32.123+0000 7f47cc8c9700 20 client.14114.objecter dump_active .. 0 homeless 2024-04-14T15:04:32.960 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:32.123+0000 7f47cc8c9700 10 client.14114.objecter ms_handle_connect 0x7f47b003c1d0 2024-04-14T15:04:32.960 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:32.128+0000 7f47cc8c9700 1 -- 172.21.15.90:0/1841207414 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f47b00056e0 con 0x7f47c8105f80 2024-04-14T15:04:32.960 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:32.380+0000 7f47cd8cb700 1 -- 172.21.15.90:0/1841207414 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f47b4005440 con 0x7f47c8105f80 2024-04-14T15:04:32.960 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:32.432+0000 7f47cc8c9700 1 -- 172.21.15.90:0/1841207414 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (unknown 2263024820 0 0) 0x7f47b8017630 con 0x7f47c8105f80 2024-04-14T15:04:32.960 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:32.433+0000 7f47cc8c9700 1 -- 172.21.15.90:0/1841207414 <== mon.0 v1:172.21.15.90:6789/0 11 ==== mgrmap(e 5) ==== 46523+0+0 (unknown 2962160406 0 0) 0x7f47b80467d0 con 0x7f47c8105f80 2024-04-14T15:04:32.960 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:32.435+0000 7f47cd8cb700 10 librados: watch_flush enter 2024-04-14T15:04:32.960 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:32.435+0000 7f47cd8cb700 10 librados: watch_flush exit 2024-04-14T15:04:32.960 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:32.435+0000 7f47cd8cb700 20 client.14114.objecter shutdown clearing up homeless session... 2024-04-14T15:04:32.960 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:32.435+0000 7f47cd8cb700 10 client.14114.objecter successfully canceled tick 2024-04-14T15:04:32.960 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:32.435+0000 7f47cd8cb700 1 -- 172.21.15.90:0/1841207414 >> v1:172.21.15.90:6800/2306258962 conn(0x7f47b003c1d0 legacy=0x7f47b003e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:04:32.960 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:32.435+0000 7f47cd8cb700 1 -- 172.21.15.90:0/1841207414 >> v1:172.21.15.90:6789/0 conn(0x7f47c8105f80 legacy=0x7f47c819ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:04:32.960 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:32.435+0000 7f47cd8cb700 1 -- 172.21.15.90:0/1841207414 shutdown_connections 2024-04-14T15:04:32.960 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:32.435+0000 7f47cd8cb700 1 -- 172.21.15.90:0/1841207414 >> 172.21.15.90:0/1841207414 conn(0x7f47c8101500 msgr2=0x7f47c8074e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:04:32.960 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:32.436+0000 7f47cd8cb700 1 -- 172.21.15.90:0/1841207414 shutdown_connections 2024-04-14T15:04:32.961 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:32.436+0000 7f47cd8cb700 1 -- 172.21.15.90:0/1841207414 wait complete. 2024-04-14T15:04:32.961 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:32.436+0000 7f47cd8cb700 1 librados: shutdown 2024-04-14T15:04:33.205 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:33 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:04:33.003+0000 7eff57813200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-14T15:04:33.205 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:33 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:04:33.099+0000 7eff57813200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-14T15:04:33.644 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:33 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1841207414' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-14T15:04:33.644 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:33 smithi090 ceph-mon[30399]: mgrmap e5: y(active, since 12s) 2024-04-14T15:04:33.941 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:33 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:04:33.778+0000 7eff57813200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-14T15:04:33.941 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:33 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:04:33.870+0000 7eff57813200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-14T15:04:34.301 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:34 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:04:34.010+0000 7eff57813200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-14T15:04:34.301 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:34 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:04:34.104+0000 7eff57813200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-14T15:04:34.301 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:34 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:04:34.185+0000 7eff57813200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-14T15:04:34.553 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:34 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3679146341' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-14T15:04:34.553 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:34 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:04:34.456+0000 7eff57813200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-14T15:04:34.562 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout { 2024-04-14T15:04:34.562 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-14T15:04:34.562 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-14T15:04:34.562 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-04-14T15:04:34.562 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-14T15:04:34.562 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout } 2024-04-14T15:04:34.562 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:33.436+0000 7f615b250700 1 Processor -- start 2024-04-14T15:04:34.562 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:33.436+0000 7f615b250700 1 -- start start 2024-04-14T15:04:34.562 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:33.436+0000 7f615b250700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f615410a400 con 0x7f6154107560 2024-04-14T15:04:34.562 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:33.436+0000 7f6158fec700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f6154107560 0x7f6154107980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:39530/0 (socket says 172.21.15.90:39530) 2024-04-14T15:04:34.563 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:33.436+0000 7f6158fec700 1 -- 172.21.15.90:0/2216716603 learned_addr learned my addr 172.21.15.90:0/2216716603 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:04:34.563 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:33.439+0000 7f61537fe700 1 -- 172.21.15.90:0/2216716603 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2505261577 0 0) 0x7f615410a400 con 0x7f6154107560 2024-04-14T15:04:34.563 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:33.439+0000 7f61537fe700 1 -- 172.21.15.90:0/2216716603 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f61440035b0 con 0x7f6154107560 2024-04-14T15:04:34.563 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:33.439+0000 7f61537fe700 1 -- 172.21.15.90:0/2216716603 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 406489019 0 0) 0x7f61440035b0 con 0x7f6154107560 2024-04-14T15:04:34.563 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:33.439+0000 7f61537fe700 1 -- 172.21.15.90:0/2216716603 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f615410b5e0 con 0x7f6154107560 2024-04-14T15:04:34.563 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:33.439+0000 7f61537fe700 1 -- 172.21.15.90:0/2216716603 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f6148001580 con 0x7f6154107560 2024-04-14T15:04:34.563 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:33.440+0000 7f61537fe700 1 -- 172.21.15.90:0/2216716603 <== mon.0 v1:172.21.15.90:6789/0 4 ==== config(26 keys) ==== 1050+0+0 (unknown 2744544961 0 0) 0x7f6148001d30 con 0x7f6154107560 2024-04-14T15:04:34.563 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:33.440+0000 7f61537fe700 1 -- 172.21.15.90:0/2216716603 <== mon.0 v1:172.21.15.90:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f6148005960 con 0x7f6154107560 2024-04-14T15:04:34.563 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:33.440+0000 7f615b250700 1 -- 172.21.15.90:0/2216716603 >> v1:172.21.15.90:6789/0 conn(0x7f6154107560 legacy=0x7f6154107980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:04:34.563 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:33.440+0000 7f615b250700 1 -- 172.21.15.90:0/2216716603 shutdown_connections 2024-04-14T15:04:34.563 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:33.440+0000 7f615b250700 1 -- 172.21.15.90:0/2216716603 >> 172.21.15.90:0/2216716603 conn(0x7f6154100cf0 msgr2=0x7f6154103110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:04:34.563 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:33.440+0000 7f615b250700 1 -- 172.21.15.90:0/2216716603 shutdown_connections 2024-04-14T15:04:34.563 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:33.440+0000 7f615b250700 1 -- 172.21.15.90:0/2216716603 wait complete. 2024-04-14T15:04:34.563 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:33.441+0000 7f615b250700 1 librados: starting msgr at 2024-04-14T15:04:34.564 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:33.441+0000 7f615b250700 1 librados: starting objecter 2024-04-14T15:04:34.564 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:33.441+0000 7f615b250700 1 Processor -- start 2024-04-14T15:04:34.564 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:33.441+0000 7f615b250700 1 -- start start 2024-04-14T15:04:34.564 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:33.441+0000 7f615b250700 1 librados: setting wanted keys 2024-04-14T15:04:34.564 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:33.441+0000 7f615b250700 1 librados: calling monclient init 2024-04-14T15:04:34.564 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:33.441+0000 7f615b250700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f615419b800 con 0x7f6154107560 2024-04-14T15:04:34.564 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:33.441+0000 7f6158fec700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f6154107560 0x7f615419fe10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:39540/0 (socket says 172.21.15.90:39540) 2024-04-14T15:04:34.564 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:33.441+0000 7f6158fec700 1 -- 172.21.15.90:0/3679146341 learned_addr learned my addr 172.21.15.90:0/3679146341 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:04:34.564 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:33.441+0000 7f6151ffb700 10 client.?.objecter ms_handle_connect 0x7f6154107560 2024-04-14T15:04:34.564 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:33.441+0000 7f6151ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:04:34.564 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:33.441+0000 7f6151ffb700 1 -- 172.21.15.90:0/3679146341 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1130220783 0 0) 0x7f615419b800 con 0x7f6154107560 2024-04-14T15:04:34.564 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:33.441+0000 7f6151ffb700 1 -- 172.21.15.90:0/3679146341 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f61440056e0 con 0x7f6154107560 2024-04-14T15:04:34.564 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:33.442+0000 7f6151ffb700 1 -- 172.21.15.90:0/3679146341 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2923930769 0 0) 0x7f615410b5e0 con 0x7f6154107560 2024-04-14T15:04:34.564 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:33.442+0000 7f6151ffb700 1 -- 172.21.15.90:0/3679146341 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f615419b800 con 0x7f6154107560 2024-04-14T15:04:34.565 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:33.442+0000 7f6151ffb700 1 -- 172.21.15.90:0/3679146341 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f6148001080 con 0x7f6154107560 2024-04-14T15:04:34.565 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:33.442+0000 7f6151ffb700 1 -- 172.21.15.90:0/3679146341 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 325853341 0 0) 0x7f615419b800 con 0x7f6154107560 2024-04-14T15:04:34.565 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:33.442+0000 7f6151ffb700 1 -- 172.21.15.90:0/3679146341 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6154105b30 con 0x7f6154107560 2024-04-14T15:04:34.565 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:33.442+0000 7f615b250700 1 -- 172.21.15.90:0/3679146341 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6154105e40 con 0x7f6154107560 2024-04-14T15:04:34.565 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:33.442+0000 7f6151ffb700 1 -- 172.21.15.90:0/3679146341 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1050+0+0 (unknown 2744544961 0 0) 0x7f6148004ab0 con 0x7f6154107560 2024-04-14T15:04:34.565 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:33.442+0000 7f6151ffb700 1 -- 172.21.15.90:0/3679146341 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f6148006270 con 0x7f6154107560 2024-04-14T15:04:34.565 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:33.442+0000 7f615b250700 10 client.14118.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:04:34.565 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:33.442+0000 7f615b250700 1 -- 172.21.15.90:0/3679146341 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f61541a1700 con 0x7f6154107560 2024-04-14T15:04:34.565 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:33.442+0000 7f615b250700 1 librados: init done 2024-04-14T15:04:34.565 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:33.443+0000 7f6151ffb700 1 -- 172.21.15.90:0/3679146341 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 5) ==== 46523+0+0 (unknown 2962160406 0 0) 0x7f6148011b50 con 0x7f6154107560 2024-04-14T15:04:34.565 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:33.443+0000 7f6151ffb700 1 -- 172.21.15.90:0/3679146341 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 87150692 0 0) 0x7f6148048fe0 con 0x7f6154107560 2024-04-14T15:04:34.565 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:33.443+0000 7f6153fff700 1 -- 172.21.15.90:0/3679146341 >> v1:172.21.15.90:6800/2306258962 conn(0x7f614403c3b0 legacy=0x7f614403e870 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.90:6800/2306258962 2024-04-14T15:04:34.565 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:33.443+0000 7f6151ffb700 10 client.14118.objecter ms_dispatch 0x7f6154001550 osd_map(1..1 src has 1..1) 2024-04-14T15:04:34.565 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:33.443+0000 7f6151ffb700 3 client.14118.objecter handle_osd_map got epochs [1,1] > 0 2024-04-14T15:04:34.565 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:33.443+0000 7f6151ffb700 3 client.14118.objecter handle_osd_map decoding full epoch 1 2024-04-14T15:04:34.566 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:33.443+0000 7f6151ffb700 20 client.14118.objecter dump_active .. 0 homeless 2024-04-14T15:04:34.566 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:33.443+0000 7f615b250700 1 -- 172.21.15.90:0/3679146341 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f615410b7b0 con 0x7f6154107560 2024-04-14T15:04:34.566 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:33.448+0000 7f6151ffb700 1 -- 172.21.15.90:0/3679146341 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f61440056e0 con 0x7f6154107560 2024-04-14T15:04:34.566 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:33.644+0000 7f6153fff700 1 -- 172.21.15.90:0/3679146341 >> v1:172.21.15.90:6800/2306258962 conn(0x7f614403c3b0 legacy=0x7f614403e870 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.90:6800/2306258962 2024-04-14T15:04:34.566 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:33.727+0000 7f615b250700 1 -- 172.21.15.90:0/3679146341 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f615402e040 con 0x7f6154107560 2024-04-14T15:04:34.566 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:33.727+0000 7f6151ffb700 1 -- 172.21.15.90:0/3679146341 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (unknown 3768197548 0 15966916) 0x7f61480179c0 con 0x7f6154107560 2024-04-14T15:04:34.566 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:33.729+0000 7f615b250700 10 librados: watch_flush enter 2024-04-14T15:04:34.566 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:33.729+0000 7f615b250700 10 librados: watch_flush exit 2024-04-14T15:04:34.566 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:33.729+0000 7f615b250700 20 client.14118.objecter shutdown clearing up homeless session... 2024-04-14T15:04:34.566 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:33.729+0000 7f615b250700 10 client.14118.objecter successfully canceled tick 2024-04-14T15:04:34.566 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:33.729+0000 7f615b250700 1 -- 172.21.15.90:0/3679146341 >> v1:172.21.15.90:6800/2306258962 conn(0x7f614403c3b0 legacy=0x7f614403e870 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-14T15:04:34.566 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:33.729+0000 7f615b250700 1 -- 172.21.15.90:0/3679146341 >> v1:172.21.15.90:6789/0 conn(0x7f6154107560 legacy=0x7f615419fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:04:34.566 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:33.729+0000 7f615b250700 1 -- 172.21.15.90:0/3679146341 shutdown_connections 2024-04-14T15:04:34.566 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:33.729+0000 7f615b250700 1 -- 172.21.15.90:0/3679146341 >> 172.21.15.90:0/3679146341 conn(0x7f6154100cf0 msgr2=0x7f61541908e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:04:34.566 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:33.729+0000 7f615b250700 1 -- 172.21.15.90:0/3679146341 shutdown_connections 2024-04-14T15:04:34.566 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:33.729+0000 7f615b250700 1 -- 172.21.15.90:0/3679146341 wait complete. 2024-04-14T15:04:34.567 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:33.730+0000 7f615b250700 1 librados: shutdown 2024-04-14T15:04:34.567 INFO:teuthology.orchestra.run.smithi090.stdout:Waiting for the mgr to restart... 2024-04-14T15:04:34.567 INFO:teuthology.orchestra.run.smithi090.stdout:Waiting for mgr epoch 5... 2024-04-14T15:04:34.816 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:34 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:04:34.567+0000 7eff57813200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-14T15:04:34.817 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:34 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:04:34.663+0000 7eff57813200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-14T15:04:34.817 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:34 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:04:34.757+0000 7eff57813200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-14T15:04:35.191 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:34 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:04:34.937+0000 7eff57813200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-14T15:04:36.191 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:35 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:04:35.893+0000 7eff57813200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-14T15:04:36.192 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:36 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:04:36.021+0000 7eff57813200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-14T15:04:36.633 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:36 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:04:36.350+0000 7eff57813200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-14T15:04:36.633 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:36 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:04:36.438+0000 7eff57813200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-14T15:04:36.633 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:36 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:04:36.632+0000 7eff57813200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-14T15:04:36.941 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:36 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:04:36.813+0000 7eff57813200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-14T15:04:37.350 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:37 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:04:37.093+0000 7eff57813200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-14T15:04:37.351 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:37 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:04:37.350+0000 7eff57813200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-14T15:04:37.691 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:37 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:04:37.584+0000 7eff57813200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-14T15:04:38.691 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:38 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:04:38.377+0000 7eff57813200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-14T15:04:39.191 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:38 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:04:38.940+0000 7eff57813200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-14T15:04:39.191 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:39 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:04:39.036+0000 7eff57813200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-14T15:04:39.692 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:39 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:04:39.249+0000 7eff57813200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-14T15:04:39.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:39 smithi090 ceph-mon[30399]: Active manager daemon y restarted 2024-04-14T15:04:39.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:39 smithi090 ceph-mon[30399]: Activating manager daemon y 2024-04-14T15:04:39.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:39 smithi090 ceph-mon[30399]: osdmap e2: 0 total, 0 up, 0 in 2024-04-14T15:04:39.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:39 smithi090 ceph-mon[30399]: mgrmap e6: y(active, starting, since 0.00342601s) 2024-04-14T15:04:39.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:39 smithi090 ceph-mon[30399]: from='mgr.14116 v1:172.21.15.90:0/2550531201' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T15:04:39.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:39 smithi090 ceph-mon[30399]: from='mgr.14116 v1:172.21.15.90:0/2550531201' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-14T15:04:39.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:39 smithi090 ceph-mon[30399]: from='mgr.14116 v1:172.21.15.90:0/2550531201' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-14T15:04:39.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:39 smithi090 ceph-mon[30399]: from='mgr.14116 v1:172.21.15.90:0/2550531201' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-14T15:04:39.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:39 smithi090 ceph-mon[30399]: from='mgr.14116 v1:172.21.15.90:0/2550531201' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-14T15:04:39.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:39 smithi090 ceph-mon[30399]: Manager daemon y is now available 2024-04-14T15:04:39.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:39 smithi090 ceph-mon[30399]: from='mgr.14116 v1:172.21.15.90:0/2550531201' entity='mgr.y' 2024-04-14T15:04:39.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:39 smithi090 ceph-mon[30399]: from='mgr.14116 v1:172.21.15.90:0/2550531201' entity='mgr.y' 2024-04-14T15:04:39.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:39 smithi090 ceph-mon[30399]: from='mgr.14116 v1:172.21.15.90:0/2550531201' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:04:39.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:39 smithi090 ceph-mon[30399]: from='mgr.14116 v1:172.21.15.90:0/2550531201' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:04:39.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:39 smithi090 ceph-mon[30399]: from='mgr.14116 v1:172.21.15.90:0/2550531201' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T15:04:39.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:39 smithi090 ceph-mon[30399]: from='mgr.14116 v1:172.21.15.90:0/2550531201' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T15:04:40.691 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:40 smithi090 ceph-mon[30399]: Found migration_current of "None". Setting to last migration. 2024-04-14T15:04:40.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:40 smithi090 ceph-mon[30399]: from='mgr.14116 v1:172.21.15.90:0/2550531201' entity='mgr.y' 2024-04-14T15:04:40.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:40 smithi090 ceph-mon[30399]: from='mgr.14116 v1:172.21.15.90:0/2550531201' entity='mgr.y' 2024-04-14T15:04:40.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:40 smithi090 ceph-mon[30399]: mgrmap e7: y(active, since 1.00534s) 2024-04-14T15:04:40.974 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout { 2024-04-14T15:04:40.974 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-14T15:04:40.974 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-14T15:04:40.974 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout } 2024-04-14T15:04:40.974 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:35.027+0000 7ff8e6457700 1 Processor -- start 2024-04-14T15:04:40.974 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:35.027+0000 7ff8e6457700 1 -- start start 2024-04-14T15:04:40.974 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:35.028+0000 7ff8e6457700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff8e007ae60 con 0x7ff8e0107b40 2024-04-14T15:04:40.974 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:35.028+0000 7ff8dffff700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7ff8e0107b40 0x7ff8e0109f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:39548/0 (socket says 172.21.15.90:39548) 2024-04-14T15:04:40.974 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:35.028+0000 7ff8dffff700 1 -- 172.21.15.90:0/1915429330 learned_addr learned my addr 172.21.15.90:0/1915429330 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:04:40.975 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:35.028+0000 7ff8deffd700 1 -- 172.21.15.90:0/1915429330 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 668255852 0 0) 0x7ff8e007ae60 con 0x7ff8e0107b40 2024-04-14T15:04:40.975 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:35.028+0000 7ff8deffd700 1 -- 172.21.15.90:0/1915429330 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff8c80035b0 con 0x7ff8e0107b40 2024-04-14T15:04:40.975 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:35.028+0000 7ff8deffd700 1 -- 172.21.15.90:0/1915429330 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3879713193 0 0) 0x7ff8c80035b0 con 0x7ff8e0107b40 2024-04-14T15:04:40.975 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:35.028+0000 7ff8deffd700 1 -- 172.21.15.90:0/1915429330 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8e010c660 con 0x7ff8e0107b40 2024-04-14T15:04:40.975 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:35.028+0000 7ff8deffd700 1 -- 172.21.15.90:0/1915429330 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7ff8d4001580 con 0x7ff8e0107b40 2024-04-14T15:04:40.975 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:35.028+0000 7ff8deffd700 1 -- 172.21.15.90:0/1915429330 <== mon.0 v1:172.21.15.90:6789/0 4 ==== config(26 keys) ==== 1050+0+0 (unknown 2744544961 0 0) 0x7ff8d4001d30 con 0x7ff8e0107b40 2024-04-14T15:04:40.975 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:35.028+0000 7ff8deffd700 1 -- 172.21.15.90:0/1915429330 <== mon.0 v1:172.21.15.90:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7ff8d4005960 con 0x7ff8e0107b40 2024-04-14T15:04:40.975 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:35.029+0000 7ff8e6457700 1 -- 172.21.15.90:0/1915429330 >> v1:172.21.15.90:6789/0 conn(0x7ff8e0107b40 legacy=0x7ff8e0109f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:04:40.975 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:35.029+0000 7ff8e6457700 1 -- 172.21.15.90:0/1915429330 shutdown_connections 2024-04-14T15:04:40.975 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:35.029+0000 7ff8e6457700 1 -- 172.21.15.90:0/1915429330 >> 172.21.15.90:0/1915429330 conn(0x7ff8e0101570 msgr2=0x7ff8e0103990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:04:40.975 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:35.029+0000 7ff8e6457700 1 -- 172.21.15.90:0/1915429330 shutdown_connections 2024-04-14T15:04:40.975 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:35.029+0000 7ff8e6457700 1 -- 172.21.15.90:0/1915429330 wait complete. 2024-04-14T15:04:40.975 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:35.030+0000 7ff8e6457700 1 librados: starting msgr at 2024-04-14T15:04:40.975 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:35.030+0000 7ff8e6457700 1 librados: starting objecter 2024-04-14T15:04:40.975 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:35.030+0000 7ff8e6457700 1 Processor -- start 2024-04-14T15:04:40.976 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:35.030+0000 7ff8e6457700 1 -- start start 2024-04-14T15:04:40.976 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:35.030+0000 7ff8e6457700 1 librados: setting wanted keys 2024-04-14T15:04:40.976 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:35.030+0000 7ff8e6457700 1 librados: calling monclient init 2024-04-14T15:04:40.976 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:35.030+0000 7ff8e6457700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff8e019b410 con 0x7ff8e0107b40 2024-04-14T15:04:40.976 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:35.030+0000 7ff8dffff700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7ff8e0107b40 0x7ff8e019ad00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:39558/0 (socket says 172.21.15.90:39558) 2024-04-14T15:04:40.976 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:35.030+0000 7ff8dffff700 1 -- 172.21.15.90:0/122826770 learned_addr learned my addr 172.21.15.90:0/122826770 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:04:40.976 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:35.030+0000 7ff8dd7fa700 10 client.?.objecter ms_handle_connect 0x7ff8e0107b40 2024-04-14T15:04:40.976 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:35.030+0000 7ff8dd7fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:04:40.976 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:35.030+0000 7ff8dd7fa700 1 -- 172.21.15.90:0/122826770 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3041449372 0 0) 0x7ff8e019b410 con 0x7ff8e0107b40 2024-04-14T15:04:40.976 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:35.030+0000 7ff8dd7fa700 1 -- 172.21.15.90:0/122826770 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff8c80056e0 con 0x7ff8e0107b40 2024-04-14T15:04:40.976 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:35.031+0000 7ff8dd7fa700 1 -- 172.21.15.90:0/122826770 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2424226463 0 0) 0x7ff8e010c660 con 0x7ff8e0107b40 2024-04-14T15:04:40.976 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:35.031+0000 7ff8dd7fa700 1 -- 172.21.15.90:0/122826770 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff8e019b410 con 0x7ff8e0107b40 2024-04-14T15:04:40.976 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:35.031+0000 7ff8dd7fa700 1 -- 172.21.15.90:0/122826770 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7ff8d4001080 con 0x7ff8e0107b40 2024-04-14T15:04:40.976 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:35.031+0000 7ff8dd7fa700 1 -- 172.21.15.90:0/122826770 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3481612949 0 0) 0x7ff8e019b410 con 0x7ff8e0107b40 2024-04-14T15:04:40.976 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:35.031+0000 7ff8dd7fa700 1 -- 172.21.15.90:0/122826770 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8e019b5e0 con 0x7ff8e0107b40 2024-04-14T15:04:40.977 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:35.031+0000 7ff8e6457700 1 -- 172.21.15.90:0/122826770 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff8e019b8d0 con 0x7ff8e0107b40 2024-04-14T15:04:40.977 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:35.031+0000 7ff8e6457700 10 client.14120.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:04:40.977 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:35.031+0000 7ff8e6457700 1 -- 172.21.15.90:0/122826770 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff8e019f410 con 0x7ff8e0107b40 2024-04-14T15:04:40.977 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:35.031+0000 7ff8e6457700 1 librados: init done 2024-04-14T15:04:40.977 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:35.031+0000 7ff8dd7fa700 1 -- 172.21.15.90:0/122826770 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1050+0+0 (unknown 2744544961 0 0) 0x7ff8d4001ba0 con 0x7ff8e0107b40 2024-04-14T15:04:40.977 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:35.031+0000 7ff8dd7fa700 1 -- 172.21.15.90:0/122826770 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7ff8d4005fc0 con 0x7ff8e0107b40 2024-04-14T15:04:40.977 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:35.032+0000 7ff8dd7fa700 1 -- 172.21.15.90:0/122826770 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 5) ==== 46523+0+0 (unknown 2962160406 0 0) 0x7ff8d4011730 con 0x7ff8e0107b40 2024-04-14T15:04:40.977 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:35.032+0000 7ff8dd7fa700 1 -- 172.21.15.90:0/122826770 --> v1:172.21.15.90:6800/2306258962 -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7ff8d4001080 con 0x7ff8c803c3c0 2024-04-14T15:04:40.977 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:35.032+0000 7ff8df7fe700 1 -- 172.21.15.90:0/122826770 >> v1:172.21.15.90:6800/2306258962 conn(0x7ff8c803c3c0 legacy=0x7ff8c803e880 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.90:6800/2306258962 2024-04-14T15:04:40.977 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:35.032+0000 7ff8dd7fa700 1 -- 172.21.15.90:0/122826770 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 87150692 0 0) 0x7ff8d4047dc0 con 0x7ff8e0107b40 2024-04-14T15:04:40.977 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:35.032+0000 7ff8dd7fa700 10 client.14120.objecter ms_dispatch 0x7ff8e0001550 osd_map(1..1 src has 1..1) 2024-04-14T15:04:40.977 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:35.032+0000 7ff8dd7fa700 3 client.14120.objecter handle_osd_map got epochs [1,1] > 0 2024-04-14T15:04:40.977 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:35.032+0000 7ff8dd7fa700 3 client.14120.objecter handle_osd_map decoding full epoch 1 2024-04-14T15:04:40.977 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:35.032+0000 7ff8dd7fa700 20 client.14120.objecter dump_active .. 0 homeless 2024-04-14T15:04:40.977 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:35.233+0000 7ff8df7fe700 1 -- 172.21.15.90:0/122826770 >> v1:172.21.15.90:6800/2306258962 conn(0x7ff8c803c3c0 legacy=0x7ff8c803e880 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.90:6800/2306258962 2024-04-14T15:04:40.978 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:35.634+0000 7ff8df7fe700 1 -- 172.21.15.90:0/122826770 >> v1:172.21.15.90:6800/2306258962 conn(0x7ff8c803c3c0 legacy=0x7ff8c803e880 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.90:6800/2306258962 2024-04-14T15:04:40.978 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:36.435+0000 7ff8df7fe700 1 -- 172.21.15.90:0/122826770 >> v1:172.21.15.90:6800/2306258962 conn(0x7ff8c803c3c0 legacy=0x7ff8c803e880 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.90:6800/2306258962 2024-04-14T15:04:40.978 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:38.036+0000 7ff8df7fe700 1 -- 172.21.15.90:0/122826770 >> v1:172.21.15.90:6800/2306258962 conn(0x7ff8c803c3c0 legacy=0x7ff8c803e880 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.90:6800/2306258962 2024-04-14T15:04:40.978 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:39.253+0000 7ff8dd7fa700 1 -- 172.21.15.90:0/122826770 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mgrmap(e 6) ==== 46325+0+0 (unknown 2960943484 0 0) 0x7ff8d4012ca0 con 0x7ff8e0107b40 2024-04-14T15:04:40.978 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:39.253+0000 7ff8dd7fa700 1 -- 172.21.15.90:0/122826770 >> v1:172.21.15.90:6800/2306258962 conn(0x7ff8c803c3c0 legacy=0x7ff8c803e880 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-14T15:04:40.978 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:40.032+0000 7ff8e5c56700 10 client.14120.objecter tick 2024-04-14T15:04:40.978 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:40.253+0000 7ff8dd7fa700 1 -- 172.21.15.90:0/122826770 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mgrmap(e 7) ==== 46417+0+0 (unknown 1908795166 0 0) 0x7ff8d40490a0 con 0x7ff8e0107b40 2024-04-14T15:04:40.978 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:40.254+0000 7ff8dd7fa700 1 -- 172.21.15.90:0/122826770 --> v1:172.21.15.90:6800/941091171 -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7ff8d4001080 con 0x7ff8c8040c30 2024-04-14T15:04:40.978 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:40.254+0000 7ff8dd7fa700 10 client.14120.objecter ms_handle_connect 0x7ff8c8040c30 2024-04-14T15:04:40.978 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:40.255+0000 7ff8dd7fa700 1 -- 172.21.15.90:0/122826770 <== mgr.14116 v1:172.21.15.90:6800/941091171 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (unknown 0 0 4109177037) 0x7ff8d4001080 con 0x7ff8c8040c30 2024-04-14T15:04:40.978 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:40.261+0000 7ff8e6457700 1 -- 172.21.15.90:0/122826770 --> v1:172.21.15.90:6800/941091171 -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7ff8cc002600 con 0x7ff8c8040c30 2024-04-14T15:04:40.978 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:40.262+0000 7ff8dd7fa700 1 -- 172.21.15.90:0/122826770 <== mgr.14116 v1:172.21.15.90:6800/941091171 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (unknown 0 0 96372106) 0x7ff8cc002600 con 0x7ff8c8040c30 2024-04-14T15:04:40.978 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:40.262+0000 7ff8e6457700 10 librados: watch_flush enter 2024-04-14T15:04:40.978 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:40.262+0000 7ff8e6457700 10 librados: watch_flush exit 2024-04-14T15:04:40.979 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:40.262+0000 7ff8e6457700 20 client.14120.objecter shutdown clearing up homeless session... 2024-04-14T15:04:40.979 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:40.262+0000 7ff8e6457700 10 client.14120.objecter successfully canceled tick 2024-04-14T15:04:40.979 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:40.262+0000 7ff8e6457700 1 -- 172.21.15.90:0/122826770 >> v1:172.21.15.90:6800/941091171 conn(0x7ff8c8040c30 legacy=0x7ff8c8043020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:04:40.979 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:40.262+0000 7ff8e6457700 1 -- 172.21.15.90:0/122826770 >> v1:172.21.15.90:6789/0 conn(0x7ff8e0107b40 legacy=0x7ff8e019ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:04:40.979 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:40.262+0000 7ff8e6457700 1 -- 172.21.15.90:0/122826770 shutdown_connections 2024-04-14T15:04:40.979 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:40.262+0000 7ff8e6457700 1 -- 172.21.15.90:0/122826770 >> 172.21.15.90:0/122826770 conn(0x7ff8e0101570 msgr2=0x7ff8e0191930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:04:40.979 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:40.262+0000 7ff8e6457700 1 -- 172.21.15.90:0/122826770 shutdown_connections 2024-04-14T15:04:40.979 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:40.262+0000 7ff8e6457700 1 -- 172.21.15.90:0/122826770 wait complete. 2024-04-14T15:04:40.979 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:40.262+0000 7ff8e6457700 1 librados: shutdown 2024-04-14T15:04:40.979 INFO:teuthology.orchestra.run.smithi090.stdout:mgr epoch 5 is available 2024-04-14T15:04:40.979 INFO:teuthology.orchestra.run.smithi090.stdout:Setting orchestrator backend to cephadm... 2024-04-14T15:04:41.608 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:41 smithi090 ceph-mon[30399]: from='client.14120 v1:172.21.15.90:0/122826770' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-14T15:04:41.608 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:41 smithi090 ceph-mon[30399]: from='client.14120 v1:172.21.15.90:0/122826770' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-14T15:04:41.608 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:41 smithi090 ceph-mon[30399]: [14/Apr/2024:15:04:40] ENGINE Bus STARTING 2024-04-14T15:04:41.608 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:41 smithi090 ceph-mon[30399]: from='mgr.14116 v1:172.21.15.90:0/2550531201' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:04:42.114 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:41.386+0000 7f3bd69e0700 1 Processor -- start 2024-04-14T15:04:42.114 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:41.386+0000 7f3bd69e0700 1 -- start start 2024-04-14T15:04:42.115 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:41.386+0000 7f3bd69e0700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3bd010a3f0 con 0x7f3bd01075e0 2024-04-14T15:04:42.115 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:41.386+0000 7f3bcffff700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f3bd01075e0 0x7f3bd0107a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:45684/0 (socket says 172.21.15.90:45684) 2024-04-14T15:04:42.115 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:41.386+0000 7f3bcffff700 1 -- 172.21.15.90:0/1741720446 learned_addr learned my addr 172.21.15.90:0/1741720446 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:04:42.115 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:41.386+0000 7f3bceffd700 1 -- 172.21.15.90:0/1741720446 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2580631252 0 0) 0x7f3bd010a3f0 con 0x7f3bd01075e0 2024-04-14T15:04:42.115 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:41.386+0000 7f3bceffd700 1 -- 172.21.15.90:0/1741720446 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3bb80035b0 con 0x7f3bd01075e0 2024-04-14T15:04:42.115 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:41.387+0000 7f3bceffd700 1 -- 172.21.15.90:0/1741720446 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2835575090 0 0) 0x7f3bb80035b0 con 0x7f3bd01075e0 2024-04-14T15:04:42.115 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:41.387+0000 7f3bceffd700 1 -- 172.21.15.90:0/1741720446 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bd010b5d0 con 0x7f3bd01075e0 2024-04-14T15:04:42.115 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:41.387+0000 7f3bceffd700 1 -- 172.21.15.90:0/1741720446 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f3bc0001580 con 0x7f3bd01075e0 2024-04-14T15:04:42.115 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:41.387+0000 7f3bceffd700 1 -- 172.21.15.90:0/1741720446 <== mon.0 v1:172.21.15.90:6789/0 4 ==== config(26 keys) ==== 1050+0+0 (unknown 2744544961 0 0) 0x7f3bc0001c60 con 0x7f3bd01075e0 2024-04-14T15:04:42.115 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:41.387+0000 7f3bceffd700 1 -- 172.21.15.90:0/1741720446 <== mon.0 v1:172.21.15.90:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f3bc0005960 con 0x7f3bd01075e0 2024-04-14T15:04:42.115 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:41.387+0000 7f3bd69e0700 1 -- 172.21.15.90:0/1741720446 >> v1:172.21.15.90:6789/0 conn(0x7f3bd01075e0 legacy=0x7f3bd0107a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:04:42.115 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:41.387+0000 7f3bd69e0700 1 -- 172.21.15.90:0/1741720446 shutdown_connections 2024-04-14T15:04:42.115 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:41.387+0000 7f3bd69e0700 1 -- 172.21.15.90:0/1741720446 >> 172.21.15.90:0/1741720446 conn(0x7f3bd0100c80 msgr2=0x7f3bd01030c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:04:42.115 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:41.387+0000 7f3bd69e0700 1 -- 172.21.15.90:0/1741720446 shutdown_connections 2024-04-14T15:04:42.116 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:41.387+0000 7f3bd69e0700 1 -- 172.21.15.90:0/1741720446 wait complete. 2024-04-14T15:04:42.116 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:41.388+0000 7f3bd69e0700 1 librados: starting msgr at 2024-04-14T15:04:42.116 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:41.388+0000 7f3bd69e0700 1 librados: starting objecter 2024-04-14T15:04:42.116 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:41.388+0000 7f3bd69e0700 1 Processor -- start 2024-04-14T15:04:42.116 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:41.388+0000 7f3bd69e0700 1 -- start start 2024-04-14T15:04:42.116 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:41.388+0000 7f3bd69e0700 1 librados: setting wanted keys 2024-04-14T15:04:42.116 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:41.388+0000 7f3bd69e0700 1 librados: calling monclient init 2024-04-14T15:04:42.116 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:41.388+0000 7f3bd69e0700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3bd019b890 con 0x7f3bd01075e0 2024-04-14T15:04:42.116 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:41.388+0000 7f3bcffff700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f3bd01075e0 0x7f3bd019fea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:45694/0 (socket says 172.21.15.90:45694) 2024-04-14T15:04:42.116 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:41.388+0000 7f3bcffff700 1 -- 172.21.15.90:0/2566312674 learned_addr learned my addr 172.21.15.90:0/2566312674 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:04:42.116 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:41.388+0000 7f3bcd7fa700 10 client.?.objecter ms_handle_connect 0x7f3bd01075e0 2024-04-14T15:04:42.116 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:41.388+0000 7f3bcd7fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:04:42.116 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:41.389+0000 7f3bcd7fa700 1 -- 172.21.15.90:0/2566312674 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 331588887 0 0) 0x7f3bd019b890 con 0x7f3bd01075e0 2024-04-14T15:04:42.116 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:41.389+0000 7f3bcd7fa700 1 -- 172.21.15.90:0/2566312674 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3bb80035b0 con 0x7f3bd01075e0 2024-04-14T15:04:42.117 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:41.389+0000 7f3bcd7fa700 1 -- 172.21.15.90:0/2566312674 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 334671825 0 0) 0x7f3bb80035b0 con 0x7f3bd01075e0 2024-04-14T15:04:42.117 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:41.389+0000 7f3bcd7fa700 1 -- 172.21.15.90:0/2566312674 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3bd019b890 con 0x7f3bd01075e0 2024-04-14T15:04:42.117 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:41.389+0000 7f3bcd7fa700 1 -- 172.21.15.90:0/2566312674 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f3bc0001080 con 0x7f3bd01075e0 2024-04-14T15:04:42.117 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:41.389+0000 7f3bcd7fa700 1 -- 172.21.15.90:0/2566312674 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4210465190 0 0) 0x7f3bd019b890 con 0x7f3bd01075e0 2024-04-14T15:04:42.117 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:41.389+0000 7f3bcd7fa700 1 -- 172.21.15.90:0/2566312674 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bd0105ad0 con 0x7f3bd01075e0 2024-04-14T15:04:42.117 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:41.389+0000 7f3bd69e0700 1 -- 172.21.15.90:0/2566312674 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3bd0105de0 con 0x7f3bd01075e0 2024-04-14T15:04:42.117 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:41.389+0000 7f3bd69e0700 10 client.14128.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:04:42.117 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:41.389+0000 7f3bd69e0700 1 -- 172.21.15.90:0/2566312674 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f3bd01a1790 con 0x7f3bd01075e0 2024-04-14T15:04:42.117 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:41.389+0000 7f3bd69e0700 1 librados: init done 2024-04-14T15:04:42.117 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:41.389+0000 7f3bcd7fa700 1 -- 172.21.15.90:0/2566312674 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1050+0+0 (unknown 2744544961 0 0) 0x7f3bc0001b30 con 0x7f3bd01075e0 2024-04-14T15:04:42.117 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:41.389+0000 7f3bcd7fa700 1 -- 172.21.15.90:0/2566312674 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f3bc0005fc0 con 0x7f3bd01075e0 2024-04-14T15:04:42.117 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:41.390+0000 7f3bcd7fa700 1 -- 172.21.15.90:0/2566312674 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 8) ==== 46523+0+0 (unknown 2307475143 0 0) 0x7f3bc00117f0 con 0x7f3bd01075e0 2024-04-14T15:04:42.117 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:41.390+0000 7f3bd69e0700 1 -- 172.21.15.90:0/2566312674 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bbc005150 con 0x7f3bd01075e0 2024-04-14T15:04:42.117 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:41.390+0000 7f3bcd7fa700 1 -- 172.21.15.90:0/2566312674 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (unknown 1250798154 0 0) 0x7f3bc0047e80 con 0x7f3bd01075e0 2024-04-14T15:04:42.117 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:41.390+0000 7f3bcd7fa700 10 client.14128.objecter ms_dispatch 0x7f3bd0001550 osd_map(2..2 src has 1..2) 2024-04-14T15:04:42.118 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:41.390+0000 7f3bcd7fa700 3 client.14128.objecter handle_osd_map got epochs [2,2] > 0 2024-04-14T15:04:42.118 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:41.390+0000 7f3bcd7fa700 3 client.14128.objecter handle_osd_map decoding full epoch 2 2024-04-14T15:04:42.118 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:41.390+0000 7f3bcd7fa700 20 client.14128.objecter dump_active .. 0 homeless 2024-04-14T15:04:42.118 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:41.390+0000 7f3bcd7fa700 10 client.14128.objecter ms_handle_connect 0x7f3bb803c370 2024-04-14T15:04:42.118 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:41.395+0000 7f3bcd7fa700 1 -- 172.21.15.90:0/2566312674 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f3bc00176e0 con 0x7f3bd01075e0 2024-04-14T15:04:42.118 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:41.608+0000 7f3bd69e0700 1 -- 172.21.15.90:0/2566312674 --> v1:172.21.15.90:6800/941091171 -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f3bbc000bc0 con 0x7f3bb803c370 2024-04-14T15:04:42.118 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:41.614+0000 7f3bcd7fa700 1 -- 172.21.15.90:0/2566312674 <== mgr.14116 v1:172.21.15.90:6800/941091171 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (unknown 0 0 0) 0x7f3bbc000bc0 con 0x7f3bb803c370 2024-04-14T15:04:42.118 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:41.617+0000 7f3bd69e0700 10 librados: watch_flush enter 2024-04-14T15:04:42.118 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:41.617+0000 7f3bd69e0700 10 librados: watch_flush exit 2024-04-14T15:04:42.118 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:41.617+0000 7f3bd69e0700 20 client.14128.objecter shutdown clearing up homeless session... 2024-04-14T15:04:42.118 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:41.617+0000 7f3bd69e0700 10 client.14128.objecter successfully canceled tick 2024-04-14T15:04:42.118 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:41.617+0000 7f3bd69e0700 1 -- 172.21.15.90:0/2566312674 >> v1:172.21.15.90:6800/941091171 conn(0x7f3bb803c370 legacy=0x7f3bb803e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:04:42.118 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:41.617+0000 7f3bd69e0700 1 -- 172.21.15.90:0/2566312674 >> v1:172.21.15.90:6789/0 conn(0x7f3bd01075e0 legacy=0x7f3bd019fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:04:42.118 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:41.617+0000 7f3bd69e0700 1 -- 172.21.15.90:0/2566312674 shutdown_connections 2024-04-14T15:04:42.119 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:41.617+0000 7f3bd69e0700 1 -- 172.21.15.90:0/2566312674 >> 172.21.15.90:0/2566312674 conn(0x7f3bd0100c80 msgr2=0x7f3bd0190950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:04:42.119 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:41.617+0000 7f3bd69e0700 1 -- 172.21.15.90:0/2566312674 shutdown_connections 2024-04-14T15:04:42.119 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:41.617+0000 7f3bd69e0700 1 -- 172.21.15.90:0/2566312674 wait complete. 2024-04-14T15:04:42.119 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:41.617+0000 7f3bd69e0700 1 librados: shutdown 2024-04-14T15:04:42.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:42 smithi090 ceph-mon[30399]: [14/Apr/2024:15:04:40] ENGINE Serving on http://172.21.15.90:8765 2024-04-14T15:04:42.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:42 smithi090 ceph-mon[30399]: [14/Apr/2024:15:04:40] ENGINE Serving on https://172.21.15.90:7150 2024-04-14T15:04:42.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:42 smithi090 ceph-mon[30399]: [14/Apr/2024:15:04:40] ENGINE Bus STARTED 2024-04-14T15:04:42.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:42 smithi090 ceph-mon[30399]: mgrmap e8: y(active, since 2s) 2024-04-14T15:04:42.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:42 smithi090 ceph-mon[30399]: from='mgr.14116 v1:172.21.15.90:0/2550531201' entity='mgr.y' 2024-04-14T15:04:42.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:42 smithi090 ceph-mon[30399]: from='mgr.14116 v1:172.21.15.90:0/2550531201' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:04:43.323 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:43 smithi090 ceph-mon[30399]: from='client.14128 v1:172.21.15.90:0/2566312674' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:04:43.332 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-14T15:04:43.332 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:42.558+0000 7f859d109700 1 Processor -- start 2024-04-14T15:04:43.332 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:42.558+0000 7f859d109700 1 -- start start 2024-04-14T15:04:43.332 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:42.559+0000 7f859d109700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f859810a3c0 con 0x7f8598105f90 2024-04-14T15:04:43.332 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:42.559+0000 7f8596d9d700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f8598105f90 0x7f85981063b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:45710/0 (socket says 172.21.15.90:45710) 2024-04-14T15:04:43.333 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:42.559+0000 7f8596d9d700 1 -- 172.21.15.90:0/2121845980 learned_addr learned my addr 172.21.15.90:0/2121845980 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:04:43.333 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:42.559+0000 7f8595d9b700 1 -- 172.21.15.90:0/2121845980 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1300894613 0 0) 0x7f859810a3c0 con 0x7f8598105f90 2024-04-14T15:04:43.333 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:42.559+0000 7f8595d9b700 1 -- 172.21.15.90:0/2121845980 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f85800035b0 con 0x7f8598105f90 2024-04-14T15:04:43.333 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:42.559+0000 7f8595d9b700 1 -- 172.21.15.90:0/2121845980 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2002900149 0 0) 0x7f85800035b0 con 0x7f8598105f90 2024-04-14T15:04:43.333 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:42.559+0000 7f8595d9b700 1 -- 172.21.15.90:0/2121845980 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f859810b5a0 con 0x7f8598105f90 2024-04-14T15:04:43.333 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:42.559+0000 7f8595d9b700 1 -- 172.21.15.90:0/2121845980 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f8588001580 con 0x7f8598105f90 2024-04-14T15:04:43.333 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:42.559+0000 7f8595d9b700 1 -- 172.21.15.90:0/2121845980 <== mon.0 v1:172.21.15.90:6789/0 4 ==== config(26 keys) ==== 1050+0+0 (unknown 2744544961 0 0) 0x7f8588001d30 con 0x7f8598105f90 2024-04-14T15:04:43.333 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:42.559+0000 7f8595d9b700 1 -- 172.21.15.90:0/2121845980 <== mon.0 v1:172.21.15.90:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f8588005960 con 0x7f8598105f90 2024-04-14T15:04:43.333 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:42.560+0000 7f859d109700 1 -- 172.21.15.90:0/2121845980 >> v1:172.21.15.90:6789/0 conn(0x7f8598105f90 legacy=0x7f85981063b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:04:43.333 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:42.560+0000 7f859d109700 1 -- 172.21.15.90:0/2121845980 shutdown_connections 2024-04-14T15:04:43.333 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:42.560+0000 7f859d109700 1 -- 172.21.15.90:0/2121845980 >> 172.21.15.90:0/2121845980 conn(0x7f8598101550 msgr2=0x7f8598103970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:04:43.333 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:42.560+0000 7f859d109700 1 -- 172.21.15.90:0/2121845980 shutdown_connections 2024-04-14T15:04:43.333 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:42.560+0000 7f859d109700 1 -- 172.21.15.90:0/2121845980 wait complete. 2024-04-14T15:04:43.334 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:42.560+0000 7f859d109700 1 librados: starting msgr at 2024-04-14T15:04:43.334 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:42.560+0000 7f859d109700 1 librados: starting objecter 2024-04-14T15:04:43.334 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:42.561+0000 7f859d109700 1 Processor -- start 2024-04-14T15:04:43.334 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:42.561+0000 7f859d109700 1 -- start start 2024-04-14T15:04:43.334 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:42.561+0000 7f859d109700 1 librados: setting wanted keys 2024-04-14T15:04:43.334 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:42.561+0000 7f859d109700 1 librados: calling monclient init 2024-04-14T15:04:43.334 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:42.561+0000 7f859d109700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f859819a2f0 con 0x7f8598105f90 2024-04-14T15:04:43.334 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:42.561+0000 7f8596d9d700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f8598105f90 0x7f8598199be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:45720/0 (socket says 172.21.15.90:45720) 2024-04-14T15:04:43.334 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:42.561+0000 7f8596d9d700 1 -- 172.21.15.90:0/1739639584 learned_addr learned my addr 172.21.15.90:0/1739639584 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:04:43.334 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:42.561+0000 7f857ffff700 10 client.?.objecter ms_handle_connect 0x7f8598105f90 2024-04-14T15:04:43.334 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:42.561+0000 7f857ffff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:04:43.334 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:42.561+0000 7f857ffff700 1 -- 172.21.15.90:0/1739639584 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3618157705 0 0) 0x7f859819a2f0 con 0x7f8598105f90 2024-04-14T15:04:43.334 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:42.561+0000 7f857ffff700 1 -- 172.21.15.90:0/1739639584 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f85800056e0 con 0x7f8598105f90 2024-04-14T15:04:43.335 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:42.561+0000 7f857ffff700 1 -- 172.21.15.90:0/1739639584 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 485539690 0 0) 0x7f859810b5a0 con 0x7f8598105f90 2024-04-14T15:04:43.335 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:42.561+0000 7f857ffff700 1 -- 172.21.15.90:0/1739639584 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f859819a2f0 con 0x7f8598105f90 2024-04-14T15:04:43.335 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:42.562+0000 7f857ffff700 1 -- 172.21.15.90:0/1739639584 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f8588004870 con 0x7f8598105f90 2024-04-14T15:04:43.335 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:42.562+0000 7f857ffff700 1 -- 172.21.15.90:0/1739639584 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3179826081 0 0) 0x7f859819a2f0 con 0x7f8598105f90 2024-04-14T15:04:43.335 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:42.562+0000 7f857ffff700 1 -- 172.21.15.90:0/1739639584 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f859819a4c0 con 0x7f8598105f90 2024-04-14T15:04:43.335 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:42.562+0000 7f859d109700 1 -- 172.21.15.90:0/1739639584 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f859819a7b0 con 0x7f8598105f90 2024-04-14T15:04:43.335 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:42.562+0000 7f859d109700 10 client.14130.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:04:43.335 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:42.562+0000 7f859d109700 1 -- 172.21.15.90:0/1739639584 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f859819e300 con 0x7f8598105f90 2024-04-14T15:04:43.335 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:42.562+0000 7f859d109700 1 librados: init done 2024-04-14T15:04:43.335 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:42.562+0000 7f857ffff700 1 -- 172.21.15.90:0/1739639584 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1050+0+0 (unknown 2744544961 0 0) 0x7f8588001be0 con 0x7f8598105f90 2024-04-14T15:04:43.335 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:42.562+0000 7f857ffff700 1 -- 172.21.15.90:0/1739639584 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f8588005fc0 con 0x7f8598105f90 2024-04-14T15:04:43.335 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:42.562+0000 7f857ffff700 1 -- 172.21.15.90:0/1739639584 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 8) ==== 46523+0+0 (unknown 2307475143 0 0) 0x7f8588011730 con 0x7f8598105f90 2024-04-14T15:04:43.335 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:42.563+0000 7f859d109700 1 -- 172.21.15.90:0/1739639584 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8584005150 con 0x7f8598105f90 2024-04-14T15:04:43.335 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:42.563+0000 7f857ffff700 1 -- 172.21.15.90:0/1739639584 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (unknown 1250798154 0 0) 0x7f8588047e00 con 0x7f8598105f90 2024-04-14T15:04:43.335 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:42.563+0000 7f857ffff700 10 client.14130.objecter ms_dispatch 0x7f8598001550 osd_map(2..2 src has 1..2) 2024-04-14T15:04:43.336 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:42.563+0000 7f857ffff700 3 client.14130.objecter handle_osd_map got epochs [2,2] > 0 2024-04-14T15:04:43.336 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:42.563+0000 7f857ffff700 3 client.14130.objecter handle_osd_map decoding full epoch 2 2024-04-14T15:04:43.336 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:42.563+0000 7f857ffff700 20 client.14130.objecter dump_active .. 0 homeless 2024-04-14T15:04:43.336 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:42.563+0000 7f857ffff700 10 client.14130.objecter ms_handle_connect 0x7f858003c270 2024-04-14T15:04:43.336 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:42.568+0000 7f857ffff700 1 -- 172.21.15.90:0/1739639584 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f85800056e0 con 0x7f8598105f90 2024-04-14T15:04:43.336 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:42.765+0000 7f859d109700 1 -- 172.21.15.90:0/1739639584 --> v1:172.21.15.90:6800/941091171 -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f8584000bc0 con 0x7f858003c270 2024-04-14T15:04:43.336 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:42.765+0000 7f857ffff700 1 -- 172.21.15.90:0/1739639584 <== mgr.14116 v1:172.21.15.90:6800/941091171 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (unknown 0 0 2070689548) 0x7f8584000bc0 con 0x7f858003c270 2024-04-14T15:04:43.336 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:42.768+0000 7f859d109700 10 librados: watch_flush enter 2024-04-14T15:04:43.336 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:42.768+0000 7f859d109700 10 librados: watch_flush exit 2024-04-14T15:04:43.336 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:42.768+0000 7f859d109700 20 client.14130.objecter shutdown clearing up homeless session... 2024-04-14T15:04:43.336 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:42.768+0000 7f859d109700 10 client.14130.objecter successfully canceled tick 2024-04-14T15:04:43.336 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:42.768+0000 7f859d109700 1 -- 172.21.15.90:0/1739639584 >> v1:172.21.15.90:6800/941091171 conn(0x7f858003c270 legacy=0x7f858003e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:04:43.336 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:42.768+0000 7f859d109700 1 -- 172.21.15.90:0/1739639584 >> v1:172.21.15.90:6789/0 conn(0x7f8598105f90 legacy=0x7f8598199be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:04:43.337 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:42.768+0000 7f859d109700 1 -- 172.21.15.90:0/1739639584 shutdown_connections 2024-04-14T15:04:43.337 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:42.768+0000 7f859d109700 1 -- 172.21.15.90:0/1739639584 >> 172.21.15.90:0/1739639584 conn(0x7f8598101550 msgr2=0x7f8598103940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:04:43.337 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:42.768+0000 7f859d109700 1 -- 172.21.15.90:0/1739639584 shutdown_connections 2024-04-14T15:04:43.337 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:42.768+0000 7f859d109700 1 -- 172.21.15.90:0/1739639584 wait complete. 2024-04-14T15:04:43.337 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:42.768+0000 7f859d109700 1 librados: shutdown 2024-04-14T15:04:43.337 INFO:teuthology.orchestra.run.smithi090.stdout:Generating ssh key... 2024-04-14T15:04:44.310 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:44 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: Generating public/private rsa key pair. 2024-04-14T15:04:44.310 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:44 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: Your identification has been saved in /tmp/tmp88lm4aej/key. 2024-04-14T15:04:44.310 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:44 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: Your public key has been saved in /tmp/tmp88lm4aej/key.pub. 2024-04-14T15:04:44.310 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:44 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: The key fingerprint is: 2024-04-14T15:04:44.310 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:44 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: SHA256:wf5WV8oTynrGWA3Zx+iu+iBe/Z7ACnN7joI5LwW24Cs ceph-0e288098-fa70-11ee-bc8e-c7b262605968 2024-04-14T15:04:44.310 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:44 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: The key's randomart image is: 2024-04-14T15:04:44.310 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:44 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: +---[RSA 3072]----+ 2024-04-14T15:04:44.310 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:44 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: | | 2024-04-14T15:04:44.310 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:44 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: | . o o | 2024-04-14T15:04:44.311 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:44 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: | o o + +| 2024-04-14T15:04:44.311 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:44 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: | . o. . . * = | 2024-04-14T15:04:44.311 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:44 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: | . o oS = B | 2024-04-14T15:04:44.311 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:44 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: | . . .. O o . | 2024-04-14T15:04:44.311 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:44 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: | . ++ O B . | 2024-04-14T15:04:44.311 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:44 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: | E . =..B B.+ . | 2024-04-14T15:04:44.311 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:44 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: | . +o.==+.+ | 2024-04-14T15:04:44.311 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:44 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: +----[SHA256]-----+ 2024-04-14T15:04:44.650 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:44 smithi090 ceph-mon[30399]: from='client.14130 v1:172.21.15.90:0/1739639584' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:04:44.650 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:44 smithi090 ceph-mon[30399]: from='mgr.14116 v1:172.21.15.90:0/2550531201' entity='mgr.y' 2024-04-14T15:04:44.650 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:44 smithi090 ceph-mon[30399]: from='mgr.14116 v1:172.21.15.90:0/2550531201' entity='mgr.y' 2024-04-14T15:04:44.658 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:43.809+0000 7fc9d76d2700 1 Processor -- start 2024-04-14T15:04:44.658 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:43.809+0000 7fc9d76d2700 1 -- start start 2024-04-14T15:04:44.659 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:43.809+0000 7fc9d76d2700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc9d007ae20 con 0x7fc9d0105980 2024-04-14T15:04:44.659 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:43.810+0000 7fc9d546e700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fc9d0105980 0x7fc9d0105da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:45722/0 (socket says 172.21.15.90:45722) 2024-04-14T15:04:44.659 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:43.810+0000 7fc9d546e700 1 -- 172.21.15.90:0/780343372 learned_addr learned my addr 172.21.15.90:0/780343372 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:04:44.659 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:43.810+0000 7fc9c7fff700 1 -- 172.21.15.90:0/780343372 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1169305346 0 0) 0x7fc9d007ae20 con 0x7fc9d0105980 2024-04-14T15:04:44.659 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:43.810+0000 7fc9c7fff700 1 -- 172.21.15.90:0/780343372 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9bc0035b0 con 0x7fc9d0105980 2024-04-14T15:04:44.659 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:43.810+0000 7fc9c7fff700 1 -- 172.21.15.90:0/780343372 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 354683272 0 0) 0x7fc9bc0035b0 con 0x7fc9d0105980 2024-04-14T15:04:44.659 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:43.810+0000 7fc9c7fff700 1 -- 172.21.15.90:0/780343372 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9d01074c0 con 0x7fc9d0105980 2024-04-14T15:04:44.659 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:43.810+0000 7fc9c7fff700 1 -- 172.21.15.90:0/780343372 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7fc9c0001580 con 0x7fc9d0105980 2024-04-14T15:04:44.659 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:43.810+0000 7fc9c7fff700 1 -- 172.21.15.90:0/780343372 <== mon.0 v1:172.21.15.90:6789/0 4 ==== config(26 keys) ==== 1050+0+0 (unknown 2744544961 0 0) 0x7fc9c0001d30 con 0x7fc9d0105980 2024-04-14T15:04:44.659 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:43.810+0000 7fc9c7fff700 1 -- 172.21.15.90:0/780343372 <== mon.0 v1:172.21.15.90:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7fc9c0005960 con 0x7fc9d0105980 2024-04-14T15:04:44.659 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:43.811+0000 7fc9d76d2700 1 -- 172.21.15.90:0/780343372 >> v1:172.21.15.90:6789/0 conn(0x7fc9d0105980 legacy=0x7fc9d0105da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:04:44.659 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:43.811+0000 7fc9d76d2700 1 -- 172.21.15.90:0/780343372 shutdown_connections 2024-04-14T15:04:44.660 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:43.811+0000 7fc9d76d2700 1 -- 172.21.15.90:0/780343372 >> 172.21.15.90:0/780343372 conn(0x7fc9d0101580 msgr2=0x7fc9d01039a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:04:44.660 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:43.811+0000 7fc9d76d2700 1 -- 172.21.15.90:0/780343372 shutdown_connections 2024-04-14T15:04:44.660 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:43.811+0000 7fc9d76d2700 1 -- 172.21.15.90:0/780343372 wait complete. 2024-04-14T15:04:44.660 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:43.811+0000 7fc9d76d2700 1 librados: starting msgr at 2024-04-14T15:04:44.660 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:43.811+0000 7fc9d76d2700 1 librados: starting objecter 2024-04-14T15:04:44.660 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:43.812+0000 7fc9d76d2700 1 Processor -- start 2024-04-14T15:04:44.660 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:43.812+0000 7fc9d76d2700 1 -- start start 2024-04-14T15:04:44.660 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:43.812+0000 7fc9d76d2700 1 librados: setting wanted keys 2024-04-14T15:04:44.660 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:43.812+0000 7fc9d76d2700 1 librados: calling monclient init 2024-04-14T15:04:44.660 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:43.812+0000 7fc9d76d2700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc9d019d600 con 0x7fc9d0105980 2024-04-14T15:04:44.660 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:43.812+0000 7fc9d546e700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fc9d0105980 0x7fc9d019cef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:45726/0 (socket says 172.21.15.90:45726) 2024-04-14T15:04:44.660 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:43.812+0000 7fc9d546e700 1 -- 172.21.15.90:0/429388251 learned_addr learned my addr 172.21.15.90:0/429388251 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:04:44.660 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:43.812+0000 7fc9c67fc700 10 client.?.objecter ms_handle_connect 0x7fc9d0105980 2024-04-14T15:04:44.660 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:43.812+0000 7fc9c67fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:04:44.661 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:43.812+0000 7fc9c67fc700 1 -- 172.21.15.90:0/429388251 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 286972912 0 0) 0x7fc9d019d600 con 0x7fc9d0105980 2024-04-14T15:04:44.661 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:43.812+0000 7fc9c67fc700 1 -- 172.21.15.90:0/429388251 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9bc0056e0 con 0x7fc9d0105980 2024-04-14T15:04:44.661 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:43.812+0000 7fc9c67fc700 1 -- 172.21.15.90:0/429388251 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2889277390 0 0) 0x7fc9d01074c0 con 0x7fc9d0105980 2024-04-14T15:04:44.661 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:43.812+0000 7fc9c67fc700 1 -- 172.21.15.90:0/429388251 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc9d019d600 con 0x7fc9d0105980 2024-04-14T15:04:44.661 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:43.812+0000 7fc9c67fc700 1 -- 172.21.15.90:0/429388251 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7fc9c0001080 con 0x7fc9d0105980 2024-04-14T15:04:44.661 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:43.813+0000 7fc9c67fc700 1 -- 172.21.15.90:0/429388251 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 97684846 0 0) 0x7fc9d019d600 con 0x7fc9d0105980 2024-04-14T15:04:44.661 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:43.813+0000 7fc9c67fc700 1 -- 172.21.15.90:0/429388251 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9d019d7d0 con 0x7fc9d0105980 2024-04-14T15:04:44.661 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:43.813+0000 7fc9d76d2700 1 -- 172.21.15.90:0/429388251 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc9d019dac0 con 0x7fc9d0105980 2024-04-14T15:04:44.661 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:43.813+0000 7fc9d76d2700 10 client.14132.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:04:44.661 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:43.813+0000 7fc9d76d2700 1 -- 172.21.15.90:0/429388251 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc9d01a1610 con 0x7fc9d0105980 2024-04-14T15:04:44.661 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:43.813+0000 7fc9d76d2700 1 librados: init done 2024-04-14T15:04:44.661 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:43.813+0000 7fc9c67fc700 1 -- 172.21.15.90:0/429388251 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1050+0+0 (unknown 2744544961 0 0) 0x7fc9c0001ba0 con 0x7fc9d0105980 2024-04-14T15:04:44.661 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:43.813+0000 7fc9c67fc700 1 -- 172.21.15.90:0/429388251 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7fc9c0005fc0 con 0x7fc9d0105980 2024-04-14T15:04:44.661 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:43.813+0000 7fc9c67fc700 1 -- 172.21.15.90:0/429388251 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 8) ==== 46523+0+0 (unknown 2307475143 0 0) 0x7fc9c0006160 con 0x7fc9d0105980 2024-04-14T15:04:44.661 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:43.813+0000 7fc9d76d2700 1 -- 172.21.15.90:0/429388251 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9b4005150 con 0x7fc9d0105980 2024-04-14T15:04:44.661 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:43.814+0000 7fc9c67fc700 1 -- 172.21.15.90:0/429388251 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (unknown 1250798154 0 0) 0x7fc9c0047dc0 con 0x7fc9d0105980 2024-04-14T15:04:44.662 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:43.814+0000 7fc9c67fc700 10 client.14132.objecter ms_dispatch 0x7fc9d0001550 osd_map(2..2 src has 1..2) 2024-04-14T15:04:44.662 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:43.814+0000 7fc9c67fc700 3 client.14132.objecter handle_osd_map got epochs [2,2] > 0 2024-04-14T15:04:44.662 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:43.814+0000 7fc9c67fc700 3 client.14132.objecter handle_osd_map decoding full epoch 2 2024-04-14T15:04:44.662 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:43.814+0000 7fc9c67fc700 20 client.14132.objecter dump_active .. 0 homeless 2024-04-14T15:04:44.662 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:43.814+0000 7fc9c67fc700 10 client.14132.objecter ms_handle_connect 0x7fc9bc03c410 2024-04-14T15:04:44.662 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:43.818+0000 7fc9c67fc700 1 -- 172.21.15.90:0/429388251 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fc9bc0056e0 con 0x7fc9d0105980 2024-04-14T15:04:44.662 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:44.015+0000 7fc9d76d2700 1 -- 172.21.15.90:0/429388251 --> v1:172.21.15.90:6800/941091171 -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7fc9b4000bc0 con 0x7fc9bc03c410 2024-04-14T15:04:44.662 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:44.082+0000 7fc9c67fc700 1 -- 172.21.15.90:0/429388251 <== mgr.14116 v1:172.21.15.90:6800/941091171 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (unknown 0 0 0) 0x7fc9b4000bc0 con 0x7fc9bc03c410 2024-04-14T15:04:44.662 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:44.084+0000 7fc9d76d2700 10 librados: watch_flush enter 2024-04-14T15:04:44.662 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:44.084+0000 7fc9d76d2700 10 librados: watch_flush exit 2024-04-14T15:04:44.662 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:44.084+0000 7fc9d76d2700 20 client.14132.objecter shutdown clearing up homeless session... 2024-04-14T15:04:44.662 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:44.084+0000 7fc9d76d2700 10 client.14132.objecter successfully canceled tick 2024-04-14T15:04:44.662 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:44.084+0000 7fc9d76d2700 1 -- 172.21.15.90:0/429388251 >> v1:172.21.15.90:6800/941091171 conn(0x7fc9bc03c410 legacy=0x7fc9bc03e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:04:44.662 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:44.084+0000 7fc9d76d2700 1 -- 172.21.15.90:0/429388251 >> v1:172.21.15.90:6789/0 conn(0x7fc9d0105980 legacy=0x7fc9d019cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:04:44.663 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:44.084+0000 7fc9d76d2700 1 -- 172.21.15.90:0/429388251 shutdown_connections 2024-04-14T15:04:44.663 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:44.084+0000 7fc9d76d2700 1 -- 172.21.15.90:0/429388251 >> 172.21.15.90:0/429388251 conn(0x7fc9d0101580 msgr2=0x7fc9d0193b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:04:44.663 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:44.084+0000 7fc9d76d2700 1 -- 172.21.15.90:0/429388251 shutdown_connections 2024-04-14T15:04:44.663 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:44.084+0000 7fc9d76d2700 1 -- 172.21.15.90:0/429388251 wait complete. 2024-04-14T15:04:44.663 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:44.084+0000 7fc9d76d2700 1 librados: shutdown 2024-04-14T15:04:45.571 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:45 smithi090 ceph-mon[30399]: from='client.14132 v1:172.21.15.90:0/429388251' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:04:45.571 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:45 smithi090 ceph-mon[30399]: Generating ssh key... 2024-04-14T15:04:45.843 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFHb0NOZETQs1TVRGlSy67VtsqfO3n76visoDbsAzvW2+lwxTcrtzpKMHBBm5kBv1+huZZPyu+tva5xAK3j+DDgzpkbigXgzzqEIIAOalTNTW7VuJEiOCx7iZChEu2Fqp8dl2/YfcxqvF78UQidkmoiBeTLt9guNjN4G2Fi1gz8ts28XwK2qCuIvlKCj7gOTcyf0fRoCOHIWT+6xSJAKByTaojGvnfHIdeGHwJvzVLUlC5mnVU8RoojOIJ+/C0PE38eqq5HoY9A3NkRobIh8D6pRNkus6CSY/IVyPcLtH3vfUmnsj6hE7UEz+ZUhDFCOPDS6eukJ7UJxB0676jfw08KXRWU4RnateGr2kwEqeZ+HfLAfdUfWuW/ZeHkvcUwgE9LYb687K/3B9DHm9VCgkue4XW5yteTghd2pJWjlFT+eZsuC08oX6GAoRG6DigIXp1mz20qks6i6WUJhOqK1foSWbvYnoy+ZhuTY3L0ifPgaY5aXpHs4nsIaASzFILSSM= ceph-0e288098-fa70-11ee-bc8e-c7b262605968 2024-04-14T15:04:45.843 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:45.068+0000 7ff2b49f5700 1 Processor -- start 2024-04-14T15:04:45.843 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:45.068+0000 7ff2b49f5700 1 -- start start 2024-04-14T15:04:45.843 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:45.068+0000 7ff2b49f5700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff2b007ba90 con 0x7ff2b007ce90 2024-04-14T15:04:45.843 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:45.068+0000 7ff2ae59c700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7ff2b007ce90 0x7ff2b007b2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:45730/0 (socket says 172.21.15.90:45730) 2024-04-14T15:04:45.843 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:45.068+0000 7ff2ae59c700 1 -- 172.21.15.90:0/1121341365 learned_addr learned my addr 172.21.15.90:0/1121341365 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:04:45.843 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:45.069+0000 7ff2ad59a700 1 -- 172.21.15.90:0/1121341365 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3941319249 0 0) 0x7ff2b007ba90 con 0x7ff2b007ce90 2024-04-14T15:04:45.843 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:45.069+0000 7ff2ad59a700 1 -- 172.21.15.90:0/1121341365 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff29c0035b0 con 0x7ff2b007ce90 2024-04-14T15:04:45.843 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:45.069+0000 7ff2ad59a700 1 -- 172.21.15.90:0/1121341365 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 126570205 0 0) 0x7ff29c0035b0 con 0x7ff2b007ce90 2024-04-14T15:04:45.843 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:45.069+0000 7ff2ad59a700 1 -- 172.21.15.90:0/1121341365 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2b007bc60 con 0x7ff2b007ce90 2024-04-14T15:04:45.843 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:45.069+0000 7ff2ad59a700 1 -- 172.21.15.90:0/1121341365 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7ff2a0001900 con 0x7ff2b007ce90 2024-04-14T15:04:45.844 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:45.069+0000 7ff2ad59a700 1 -- 172.21.15.90:0/1121341365 <== mon.0 v1:172.21.15.90:6789/0 4 ==== config(26 keys) ==== 1050+0+0 (unknown 2744544961 0 0) 0x7ff2a0001fe0 con 0x7ff2b007ce90 2024-04-14T15:04:45.844 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:45.069+0000 7ff2ad59a700 1 -- 172.21.15.90:0/1121341365 <== mon.0 v1:172.21.15.90:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7ff2a0005ce0 con 0x7ff2b007ce90 2024-04-14T15:04:45.844 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:45.070+0000 7ff2b49f5700 1 -- 172.21.15.90:0/1121341365 >> v1:172.21.15.90:6789/0 conn(0x7ff2b007ce90 legacy=0x7ff2b007b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:04:45.844 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:45.070+0000 7ff2b49f5700 1 -- 172.21.15.90:0/1121341365 shutdown_connections 2024-04-14T15:04:45.844 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:45.070+0000 7ff2b49f5700 1 -- 172.21.15.90:0/1121341365 >> 172.21.15.90:0/1121341365 conn(0x7ff2b0102610 msgr2=0x7ff2b0104a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:04:45.844 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:45.070+0000 7ff2b49f5700 1 -- 172.21.15.90:0/1121341365 shutdown_connections 2024-04-14T15:04:45.844 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:45.070+0000 7ff2b49f5700 1 -- 172.21.15.90:0/1121341365 wait complete. 2024-04-14T15:04:45.844 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:45.070+0000 7ff2b49f5700 1 librados: starting msgr at 2024-04-14T15:04:45.844 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:45.070+0000 7ff2b49f5700 1 librados: starting objecter 2024-04-14T15:04:45.844 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:45.070+0000 7ff2b49f5700 1 Processor -- start 2024-04-14T15:04:45.844 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:45.071+0000 7ff2b49f5700 1 -- start start 2024-04-14T15:04:45.844 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:45.071+0000 7ff2b49f5700 1 librados: setting wanted keys 2024-04-14T15:04:45.844 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:45.071+0000 7ff2b49f5700 1 librados: calling monclient init 2024-04-14T15:04:45.844 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:45.071+0000 7ff2b49f5700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff2b019d640 con 0x7ff2b007ce90 2024-04-14T15:04:45.845 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:45.071+0000 7ff2ae59c700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7ff2b007ce90 0x7ff2b019cf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:45732/0 (socket says 172.21.15.90:45732) 2024-04-14T15:04:45.845 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:45.071+0000 7ff2ae59c700 1 -- 172.21.15.90:0/1745837305 learned_addr learned my addr 172.21.15.90:0/1745837305 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:04:45.845 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:45.071+0000 7ff2a77fe700 10 client.?.objecter ms_handle_connect 0x7ff2b007ce90 2024-04-14T15:04:45.845 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:45.071+0000 7ff2a77fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:04:45.845 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:45.071+0000 7ff2a77fe700 1 -- 172.21.15.90:0/1745837305 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1067646384 0 0) 0x7ff2b019d640 con 0x7ff2b007ce90 2024-04-14T15:04:45.845 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:45.071+0000 7ff2a77fe700 1 -- 172.21.15.90:0/1745837305 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff29c0035b0 con 0x7ff2b007ce90 2024-04-14T15:04:45.845 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:45.071+0000 7ff2a77fe700 1 -- 172.21.15.90:0/1745837305 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 246988421 0 0) 0x7ff29c0035b0 con 0x7ff2b007ce90 2024-04-14T15:04:45.845 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:45.071+0000 7ff2a77fe700 1 -- 172.21.15.90:0/1745837305 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff2b019d640 con 0x7ff2b007ce90 2024-04-14T15:04:45.845 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:45.071+0000 7ff2a77fe700 1 -- 172.21.15.90:0/1745837305 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7ff2a0001400 con 0x7ff2b007ce90 2024-04-14T15:04:45.845 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:45.071+0000 7ff2a77fe700 1 -- 172.21.15.90:0/1745837305 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3172307654 0 0) 0x7ff2b019d640 con 0x7ff2b007ce90 2024-04-14T15:04:45.845 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:45.071+0000 7ff2a77fe700 1 -- 172.21.15.90:0/1745837305 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2b019d810 con 0x7ff2b007ce90 2024-04-14T15:04:45.845 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:45.071+0000 7ff2b49f5700 1 -- 172.21.15.90:0/1745837305 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff2b019db00 con 0x7ff2b007ce90 2024-04-14T15:04:45.845 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:45.072+0000 7ff2b49f5700 10 client.14134.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:04:45.846 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:45.072+0000 7ff2b49f5700 1 -- 172.21.15.90:0/1745837305 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff2b01a1650 con 0x7ff2b007ce90 2024-04-14T15:04:45.846 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:45.072+0000 7ff2b49f5700 1 librados: init done 2024-04-14T15:04:45.846 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:45.072+0000 7ff2a77fe700 1 -- 172.21.15.90:0/1745837305 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1050+0+0 (unknown 2744544961 0 0) 0x7ff2a0001f90 con 0x7ff2b007ce90 2024-04-14T15:04:45.846 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:45.072+0000 7ff2a77fe700 1 -- 172.21.15.90:0/1745837305 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7ff2a0006420 con 0x7ff2b007ce90 2024-04-14T15:04:45.846 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:45.072+0000 7ff2a77fe700 1 -- 172.21.15.90:0/1745837305 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 8) ==== 46523+0+0 (unknown 2307475143 0 0) 0x7ff2a0011b90 con 0x7ff2b007ce90 2024-04-14T15:04:45.846 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:45.073+0000 7ff2a77fe700 1 -- 172.21.15.90:0/1745837305 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (unknown 1250798154 0 0) 0x7ff2a0049170 con 0x7ff2b007ce90 2024-04-14T15:04:45.846 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:45.073+0000 7ff2a77fe700 10 client.14134.objecter ms_dispatch 0x7ff2b0001550 osd_map(2..2 src has 1..2) 2024-04-14T15:04:45.846 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:45.073+0000 7ff2a77fe700 3 client.14134.objecter handle_osd_map got epochs [2,2] > 0 2024-04-14T15:04:45.846 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:45.073+0000 7ff2a77fe700 3 client.14134.objecter handle_osd_map decoding full epoch 2 2024-04-14T15:04:45.846 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:45.073+0000 7ff2b49f5700 1 -- 172.21.15.90:0/1745837305 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff290005150 con 0x7ff2b007ce90 2024-04-14T15:04:45.846 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:45.073+0000 7ff2a77fe700 20 client.14134.objecter dump_active .. 0 homeless 2024-04-14T15:04:45.846 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:45.073+0000 7ff2a77fe700 10 client.14134.objecter ms_handle_connect 0x7ff29c03c320 2024-04-14T15:04:45.846 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:45.078+0000 7ff2a77fe700 1 -- 172.21.15.90:0/1745837305 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7ff2a0017aa0 con 0x7ff2b007ce90 2024-04-14T15:04:45.846 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:45.277+0000 7ff2b49f5700 1 -- 172.21.15.90:0/1745837305 --> v1:172.21.15.90:6800/941091171 -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7ff290000bc0 con 0x7ff29c03c320 2024-04-14T15:04:45.846 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:45.277+0000 7ff2a77fe700 1 -- 172.21.15.90:0/1745837305 <== mgr.14116 v1:172.21.15.90:6800/941091171 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (unknown 0 0 2898095094) 0x7ff290000bc0 con 0x7ff29c03c320 2024-04-14T15:04:45.847 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:45.279+0000 7ff2b49f5700 10 librados: watch_flush enter 2024-04-14T15:04:45.847 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:45.279+0000 7ff2b49f5700 10 librados: watch_flush exit 2024-04-14T15:04:45.847 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:45.279+0000 7ff2b49f5700 20 client.14134.objecter shutdown clearing up homeless session... 2024-04-14T15:04:45.847 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:45.279+0000 7ff2b49f5700 10 client.14134.objecter successfully canceled tick 2024-04-14T15:04:45.847 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:45.279+0000 7ff2b49f5700 1 -- 172.21.15.90:0/1745837305 >> v1:172.21.15.90:6800/941091171 conn(0x7ff29c03c320 legacy=0x7ff29c03e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:04:45.847 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:45.279+0000 7ff2b49f5700 1 -- 172.21.15.90:0/1745837305 >> v1:172.21.15.90:6789/0 conn(0x7ff2b007ce90 legacy=0x7ff2b019cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:04:45.847 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:45.279+0000 7ff2b49f5700 1 -- 172.21.15.90:0/1745837305 shutdown_connections 2024-04-14T15:04:45.847 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:45.279+0000 7ff2b49f5700 1 -- 172.21.15.90:0/1745837305 >> 172.21.15.90:0/1745837305 conn(0x7ff2b0102610 msgr2=0x7ff2b0193b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:04:45.847 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:45.279+0000 7ff2b49f5700 1 -- 172.21.15.90:0/1745837305 shutdown_connections 2024-04-14T15:04:45.847 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:45.279+0000 7ff2b49f5700 1 -- 172.21.15.90:0/1745837305 wait complete. 2024-04-14T15:04:45.847 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:45.280+0000 7ff2b49f5700 1 librados: shutdown 2024-04-14T15:04:45.847 INFO:teuthology.orchestra.run.smithi090.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-14T15:04:45.847 INFO:teuthology.orchestra.run.smithi090.stdout:Adding key to root@localhost authorized_keys... 2024-04-14T15:04:45.847 INFO:teuthology.orchestra.run.smithi090.stdout:Adding host smithi090... 2024-04-14T15:04:46.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:46 smithi090 ceph-mon[30399]: from='client.14134 v1:172.21.15.90:0/1745837305' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:04:47.421 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:47 smithi090 ceph-mon[30399]: from='client.14136 v1:172.21.15.90:0/1908992420' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi090", "addr": "172.21.15.90", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:04:48.691 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:48 smithi090 ceph-mon[30399]: Deploying cephadm binary to smithi090 2024-04-14T15:04:51.263 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout Added host 'smithi090' with addr '172.21.15.90' 2024-04-14T15:04:51.264 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:46.265+0000 7f83bb062700 1 Processor -- start 2024-04-14T15:04:51.264 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:46.265+0000 7f83bb062700 1 -- start start 2024-04-14T15:04:51.264 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:46.265+0000 7f83bb062700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f83b4108180 con 0x7f83b41075c0 2024-04-14T15:04:51.264 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:46.265+0000 7f83b8dfe700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f83b41075c0 0x7f83b41079e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:45746/0 (socket says 172.21.15.90:45746) 2024-04-14T15:04:51.264 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:46.265+0000 7f83b8dfe700 1 -- 172.21.15.90:0/3438273582 learned_addr learned my addr 172.21.15.90:0/3438273582 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:04:51.264 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:46.266+0000 7f83b37fe700 1 -- 172.21.15.90:0/3438273582 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3906672716 0 0) 0x7f83b4108180 con 0x7f83b41075c0 2024-04-14T15:04:51.264 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:46.266+0000 7f83b37fe700 1 -- 172.21.15.90:0/3438273582 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f839c0035b0 con 0x7f83b41075c0 2024-04-14T15:04:51.264 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:46.266+0000 7f83b37fe700 1 -- 172.21.15.90:0/3438273582 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3031697956 0 0) 0x7f839c0035b0 con 0x7f83b41075c0 2024-04-14T15:04:51.264 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:46.266+0000 7f83b37fe700 1 -- 172.21.15.90:0/3438273582 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83b4106160 con 0x7f83b41075c0 2024-04-14T15:04:51.264 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:46.266+0000 7f83b37fe700 1 -- 172.21.15.90:0/3438273582 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f83a4001580 con 0x7f83b41075c0 2024-04-14T15:04:51.265 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:46.266+0000 7f83b37fe700 1 -- 172.21.15.90:0/3438273582 <== mon.0 v1:172.21.15.90:6789/0 4 ==== config(26 keys) ==== 1050+0+0 (unknown 2744544961 0 0) 0x7f83a4001d30 con 0x7f83b41075c0 2024-04-14T15:04:51.265 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:46.266+0000 7f83b37fe700 1 -- 172.21.15.90:0/3438273582 <== mon.0 v1:172.21.15.90:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f83a4005960 con 0x7f83b41075c0 2024-04-14T15:04:51.265 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:46.267+0000 7f83bb062700 1 -- 172.21.15.90:0/3438273582 >> v1:172.21.15.90:6789/0 conn(0x7f83b41075c0 legacy=0x7f83b41079e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:04:51.265 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:46.267+0000 7f83bb062700 1 -- 172.21.15.90:0/3438273582 shutdown_connections 2024-04-14T15:04:51.265 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:46.267+0000 7f83bb062700 1 -- 172.21.15.90:0/3438273582 >> 172.21.15.90:0/3438273582 conn(0x7f83b4100cf0 msgr2=0x7f83b4103110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:04:51.265 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:46.267+0000 7f83bb062700 1 -- 172.21.15.90:0/3438273582 shutdown_connections 2024-04-14T15:04:51.265 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:46.267+0000 7f83bb062700 1 -- 172.21.15.90:0/3438273582 wait complete. 2024-04-14T15:04:51.265 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:46.267+0000 7f83bb062700 1 librados: starting msgr at 2024-04-14T15:04:51.265 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:46.267+0000 7f83bb062700 1 librados: starting objecter 2024-04-14T15:04:51.265 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:46.267+0000 7f83bb062700 1 Processor -- start 2024-04-14T15:04:51.265 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:46.267+0000 7f83bb062700 1 -- start start 2024-04-14T15:04:51.266 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:46.267+0000 7f83bb062700 1 librados: setting wanted keys 2024-04-14T15:04:51.266 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:46.267+0000 7f83bb062700 1 librados: calling monclient init 2024-04-14T15:04:51.266 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:46.268+0000 7f83bb062700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f83b419d640 con 0x7f83b41075c0 2024-04-14T15:04:51.266 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:46.268+0000 7f83b8dfe700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f83b41075c0 0x7f83b419cf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:45750/0 (socket says 172.21.15.90:45750) 2024-04-14T15:04:51.266 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:46.268+0000 7f83b8dfe700 1 -- 172.21.15.90:0/1908992420 learned_addr learned my addr 172.21.15.90:0/1908992420 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:04:51.266 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:46.268+0000 7f83b1ffb700 10 client.?.objecter ms_handle_connect 0x7f83b41075c0 2024-04-14T15:04:51.266 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:46.268+0000 7f83b1ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:04:51.266 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:46.268+0000 7f83b1ffb700 1 -- 172.21.15.90:0/1908992420 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3684609456 0 0) 0x7f83b419d640 con 0x7f83b41075c0 2024-04-14T15:04:51.266 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:46.268+0000 7f83b1ffb700 1 -- 172.21.15.90:0/1908992420 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f839c0056e0 con 0x7f83b41075c0 2024-04-14T15:04:51.266 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:46.268+0000 7f83b1ffb700 1 -- 172.21.15.90:0/1908992420 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3845784351 0 0) 0x7f83b4106160 con 0x7f83b41075c0 2024-04-14T15:04:51.266 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:46.268+0000 7f83b1ffb700 1 -- 172.21.15.90:0/1908992420 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f83b419d640 con 0x7f83b41075c0 2024-04-14T15:04:51.267 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:46.268+0000 7f83b1ffb700 1 -- 172.21.15.90:0/1908992420 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f83a4001080 con 0x7f83b41075c0 2024-04-14T15:04:51.267 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:46.268+0000 7f83b1ffb700 1 -- 172.21.15.90:0/1908992420 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 692580969 0 0) 0x7f83b419d640 con 0x7f83b41075c0 2024-04-14T15:04:51.267 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:46.268+0000 7f83b1ffb700 1 -- 172.21.15.90:0/1908992420 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83b419d810 con 0x7f83b41075c0 2024-04-14T15:04:51.267 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:46.268+0000 7f83bb062700 1 -- 172.21.15.90:0/1908992420 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f83b419db00 con 0x7f83b41075c0 2024-04-14T15:04:51.267 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:46.268+0000 7f83b1ffb700 1 -- 172.21.15.90:0/1908992420 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1050+0+0 (unknown 2744544961 0 0) 0x7f83a4001b80 con 0x7f83b41075c0 2024-04-14T15:04:51.267 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:46.268+0000 7f83b1ffb700 1 -- 172.21.15.90:0/1908992420 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f83a4006220 con 0x7f83b41075c0 2024-04-14T15:04:51.267 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:46.269+0000 7f83bb062700 10 client.14136.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:04:51.267 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:46.269+0000 7f83bb062700 1 -- 172.21.15.90:0/1908992420 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f83b41a1650 con 0x7f83b41075c0 2024-04-14T15:04:51.267 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:46.269+0000 7f83bb062700 1 librados: init done 2024-04-14T15:04:51.267 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:46.269+0000 7f83b1ffb700 1 -- 172.21.15.90:0/1908992420 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 8) ==== 46523+0+0 (unknown 2307475143 0 0) 0x7f83a40063e0 con 0x7f83b41075c0 2024-04-14T15:04:51.267 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:46.269+0000 7f83bb062700 1 -- 172.21.15.90:0/1908992420 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83b4064110 con 0x7f83b41075c0 2024-04-14T15:04:51.267 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:46.269+0000 7f83b1ffb700 1 -- 172.21.15.90:0/1908992420 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (unknown 1250798154 0 0) 0x7f83a4047dc0 con 0x7f83b41075c0 2024-04-14T15:04:51.268 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:46.270+0000 7f83b1ffb700 10 client.14136.objecter ms_dispatch 0x7f83b4001550 osd_map(2..2 src has 1..2) 2024-04-14T15:04:51.268 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:46.270+0000 7f83b1ffb700 3 client.14136.objecter handle_osd_map got epochs [2,2] > 0 2024-04-14T15:04:51.268 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:46.270+0000 7f83b1ffb700 3 client.14136.objecter handle_osd_map decoding full epoch 2 2024-04-14T15:04:51.268 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:46.270+0000 7f83b1ffb700 20 client.14136.objecter dump_active .. 0 homeless 2024-04-14T15:04:51.268 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:46.270+0000 7f83b1ffb700 10 client.14136.objecter ms_handle_connect 0x7f839c03c410 2024-04-14T15:04:51.268 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:46.274+0000 7f83b1ffb700 1 -- 172.21.15.90:0/1908992420 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f839c0056e0 con 0x7f83b41075c0 2024-04-14T15:04:51.268 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:46.476+0000 7f83bb062700 1 -- 172.21.15.90:0/1908992420 --> v1:172.21.15.90:6800/941091171 -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi090", "addr": "172.21.15.90", "target": ["mon-mgr", ""]}) -- 0x7f83b402e190 con 0x7f839c03c410 2024-04-14T15:04:51.268 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:50.692+0000 7f83b1ffb700 1 -- 172.21.15.90:0/1908992420 <== mgr.14116 v1:172.21.15.90:6800/941091171 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (unknown 0 0 3387189665) 0x7f83b402e190 con 0x7f839c03c410 2024-04-14T15:04:51.268 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:50.695+0000 7f83bb062700 10 librados: watch_flush enter 2024-04-14T15:04:51.268 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:50.695+0000 7f83bb062700 10 librados: watch_flush exit 2024-04-14T15:04:51.268 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:50.695+0000 7f83bb062700 20 client.14136.objecter shutdown clearing up homeless session... 2024-04-14T15:04:51.268 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:50.695+0000 7f83bb062700 10 client.14136.objecter successfully canceled tick 2024-04-14T15:04:51.269 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:50.695+0000 7f83bb062700 1 -- 172.21.15.90:0/1908992420 >> v1:172.21.15.90:6800/941091171 conn(0x7f839c03c410 legacy=0x7f839c03e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:04:51.269 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:50.695+0000 7f83bb062700 1 -- 172.21.15.90:0/1908992420 >> v1:172.21.15.90:6789/0 conn(0x7f83b41075c0 legacy=0x7f83b419cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:04:51.269 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:50.695+0000 7f83bb062700 1 -- 172.21.15.90:0/1908992420 shutdown_connections 2024-04-14T15:04:51.269 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:50.695+0000 7f83bb062700 1 -- 172.21.15.90:0/1908992420 >> 172.21.15.90:0/1908992420 conn(0x7f83b4100cf0 msgr2=0x7f83b4193b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:04:51.269 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:50.695+0000 7f83bb062700 1 -- 172.21.15.90:0/1908992420 shutdown_connections 2024-04-14T15:04:51.269 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:50.695+0000 7f83bb062700 1 -- 172.21.15.90:0/1908992420 wait complete. 2024-04-14T15:04:51.269 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:50.695+0000 7f83bb062700 1 librados: shutdown 2024-04-14T15:04:51.269 INFO:teuthology.orchestra.run.smithi090.stdout:Deploying unmanaged mon service... 2024-04-14T15:04:51.788 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:51 smithi090 ceph-mon[30399]: from='mgr.14116 v1:172.21.15.90:0/2550531201' entity='mgr.y' 2024-04-14T15:04:51.788 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:51 smithi090 ceph-mon[30399]: Added host smithi090 2024-04-14T15:04:51.788 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:51 smithi090 ceph-mon[30399]: from='mgr.14116 v1:172.21.15.90:0/2550531201' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:04:52.705 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-14T15:04:52.711 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:51.787+0000 7f318ee55700 1 Processor -- start 2024-04-14T15:04:52.711 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:51.787+0000 7f318ee55700 1 -- start start 2024-04-14T15:04:52.711 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:51.787+0000 7f318ee55700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f318807ae60 con 0x7f3188107b50 2024-04-14T15:04:52.711 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:51.787+0000 7f318cbf1700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f3188107b50 0x7f3188109f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:45788/0 (socket says 172.21.15.90:45788) 2024-04-14T15:04:52.711 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:51.787+0000 7f318cbf1700 1 -- 172.21.15.90:0/2775762158 learned_addr learned my addr 172.21.15.90:0/2775762158 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:04:52.711 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:51.788+0000 7f31877fe700 1 -- 172.21.15.90:0/2775762158 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 595178800 0 0) 0x7f318807ae60 con 0x7f3188107b50 2024-04-14T15:04:52.712 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:51.788+0000 7f31877fe700 1 -- 172.21.15.90:0/2775762158 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f31780035b0 con 0x7f3188107b50 2024-04-14T15:04:52.712 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:51.788+0000 7f31877fe700 1 -- 172.21.15.90:0/2775762158 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1126429955 0 0) 0x7f31780035b0 con 0x7f3188107b50 2024-04-14T15:04:52.712 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:51.788+0000 7f31877fe700 1 -- 172.21.15.90:0/2775762158 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f318810c670 con 0x7f3188107b50 2024-04-14T15:04:52.712 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:51.788+0000 7f31877fe700 1 -- 172.21.15.90:0/2775762158 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f317c001580 con 0x7f3188107b50 2024-04-14T15:04:52.712 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:51.788+0000 7f31877fe700 1 -- 172.21.15.90:0/2775762158 <== mon.0 v1:172.21.15.90:6789/0 4 ==== config(26 keys) ==== 1050+0+0 (unknown 2744544961 0 0) 0x7f317c001d30 con 0x7f3188107b50 2024-04-14T15:04:52.712 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:51.788+0000 7f31877fe700 1 -- 172.21.15.90:0/2775762158 <== mon.0 v1:172.21.15.90:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f317c005960 con 0x7f3188107b50 2024-04-14T15:04:52.712 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:51.788+0000 7f318ee55700 1 -- 172.21.15.90:0/2775762158 >> v1:172.21.15.90:6789/0 conn(0x7f3188107b50 legacy=0x7f3188109f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:04:52.712 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:51.789+0000 7f318ee55700 1 -- 172.21.15.90:0/2775762158 shutdown_connections 2024-04-14T15:04:52.712 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:51.789+0000 7f318ee55700 1 -- 172.21.15.90:0/2775762158 >> 172.21.15.90:0/2775762158 conn(0x7f31881015c0 msgr2=0x7f31881039e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:04:52.712 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:51.789+0000 7f318ee55700 1 -- 172.21.15.90:0/2775762158 shutdown_connections 2024-04-14T15:04:52.712 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:51.789+0000 7f318ee55700 1 -- 172.21.15.90:0/2775762158 wait complete. 2024-04-14T15:04:52.712 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:51.789+0000 7f318ee55700 1 librados: starting msgr at 2024-04-14T15:04:52.712 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:51.789+0000 7f318ee55700 1 librados: starting objecter 2024-04-14T15:04:52.712 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:51.789+0000 7f318ee55700 1 Processor -- start 2024-04-14T15:04:52.712 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:51.789+0000 7f318ee55700 1 -- start start 2024-04-14T15:04:52.713 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:51.789+0000 7f318ee55700 1 librados: setting wanted keys 2024-04-14T15:04:52.713 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:51.789+0000 7f318ee55700 1 librados: calling monclient init 2024-04-14T15:04:52.713 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:51.789+0000 7f318ee55700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f31881a0d60 con 0x7f3188107b50 2024-04-14T15:04:52.713 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:51.789+0000 7f318cbf1700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f3188107b50 0x7f31881073d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:45796/0 (socket says 172.21.15.90:45796) 2024-04-14T15:04:52.713 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:51.789+0000 7f318cbf1700 1 -- 172.21.15.90:0/2798662536 learned_addr learned my addr 172.21.15.90:0/2798662536 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:04:52.713 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:51.789+0000 7f3185ffb700 10 client.?.objecter ms_handle_connect 0x7f3188107b50 2024-04-14T15:04:52.713 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:51.789+0000 7f3185ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:04:52.713 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:51.790+0000 7f3185ffb700 1 -- 172.21.15.90:0/2798662536 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 952425830 0 0) 0x7f31881a0d60 con 0x7f3188107b50 2024-04-14T15:04:52.713 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:51.790+0000 7f3185ffb700 1 -- 172.21.15.90:0/2798662536 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f31780056e0 con 0x7f3188107b50 2024-04-14T15:04:52.713 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:51.790+0000 7f3185ffb700 1 -- 172.21.15.90:0/2798662536 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3699184363 0 0) 0x7f318810c670 con 0x7f3188107b50 2024-04-14T15:04:52.713 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:51.790+0000 7f3185ffb700 1 -- 172.21.15.90:0/2798662536 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f31881a0d60 con 0x7f3188107b50 2024-04-14T15:04:52.713 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:51.790+0000 7f3185ffb700 1 -- 172.21.15.90:0/2798662536 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f317c001080 con 0x7f3188107b50 2024-04-14T15:04:52.713 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:51.790+0000 7f3185ffb700 1 -- 172.21.15.90:0/2798662536 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4193308639 0 0) 0x7f31881a0d60 con 0x7f3188107b50 2024-04-14T15:04:52.713 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:51.790+0000 7f3185ffb700 1 -- 172.21.15.90:0/2798662536 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31881a1f40 con 0x7f3188107b50 2024-04-14T15:04:52.713 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:51.790+0000 7f318ee55700 1 -- 172.21.15.90:0/2798662536 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f31881a0f90 con 0x7f3188107b50 2024-04-14T15:04:52.713 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:51.790+0000 7f3185ffb700 1 -- 172.21.15.90:0/2798662536 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1050+0+0 (unknown 2744544961 0 0) 0x7f317c0048c0 con 0x7f3188107b50 2024-04-14T15:04:52.714 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:51.790+0000 7f3185ffb700 1 -- 172.21.15.90:0/2798662536 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f317c006350 con 0x7f3188107b50 2024-04-14T15:04:52.714 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:51.790+0000 7f318ee55700 10 client.14138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:04:52.714 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:51.790+0000 7f318ee55700 1 -- 172.21.15.90:0/2798662536 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f31881a14d0 con 0x7f3188107b50 2024-04-14T15:04:52.714 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:51.790+0000 7f318ee55700 1 librados: init done 2024-04-14T15:04:52.714 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:51.791+0000 7f3185ffb700 1 -- 172.21.15.90:0/2798662536 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 8) ==== 46523+0+0 (unknown 2307475143 0 0) 0x7f317c011c30 con 0x7f3188107b50 2024-04-14T15:04:52.714 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:51.791+0000 7f318ee55700 1 -- 172.21.15.90:0/2798662536 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3188064110 con 0x7f3188107b50 2024-04-14T15:04:52.714 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:51.791+0000 7f3185ffb700 1 -- 172.21.15.90:0/2798662536 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (unknown 1250798154 0 0) 0x7f317c047fd0 con 0x7f3188107b50 2024-04-14T15:04:52.714 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:51.791+0000 7f3185ffb700 10 client.14138.objecter ms_dispatch 0x7f3188001550 osd_map(2..2 src has 1..2) 2024-04-14T15:04:52.714 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:51.791+0000 7f3185ffb700 3 client.14138.objecter handle_osd_map got epochs [2,2] > 0 2024-04-14T15:04:52.714 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:51.791+0000 7f3185ffb700 3 client.14138.objecter handle_osd_map decoding full epoch 2 2024-04-14T15:04:52.714 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:51.791+0000 7f3185ffb700 20 client.14138.objecter dump_active .. 0 homeless 2024-04-14T15:04:52.714 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:51.791+0000 7f3185ffb700 10 client.14138.objecter ms_handle_connect 0x7f317803c370 2024-04-14T15:04:52.714 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:51.797+0000 7f3185ffb700 1 -- 172.21.15.90:0/2798662536 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f31780056e0 con 0x7f3188107b50 2024-04-14T15:04:52.715 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:52.017+0000 7f318ee55700 1 -- 172.21.15.90:0/2798662536 --> v1:172.21.15.90:6800/941091171 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f318802e110 con 0x7f317803c370 2024-04-14T15:04:52.715 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:52.025+0000 7f3185ffb700 1 -- 172.21.15.90:0/2798662536 <== mgr.14116 v1:172.21.15.90:6800/941091171 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (unknown 0 0 3265049985) 0x7f318802e110 con 0x7f317803c370 2024-04-14T15:04:52.715 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:52.026+0000 7f318ee55700 10 librados: watch_flush enter 2024-04-14T15:04:52.715 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:52.026+0000 7f318ee55700 10 librados: watch_flush exit 2024-04-14T15:04:52.715 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:52.026+0000 7f318ee55700 20 client.14138.objecter shutdown clearing up homeless session... 2024-04-14T15:04:52.715 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:52.026+0000 7f318ee55700 10 client.14138.objecter successfully canceled tick 2024-04-14T15:04:52.715 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:52.027+0000 7f318ee55700 1 -- 172.21.15.90:0/2798662536 >> v1:172.21.15.90:6800/941091171 conn(0x7f317803c370 legacy=0x7f317803e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:04:52.715 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:52.027+0000 7f318ee55700 1 -- 172.21.15.90:0/2798662536 >> v1:172.21.15.90:6789/0 conn(0x7f3188107b50 legacy=0x7f31881073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:04:52.715 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:52.027+0000 7f318ee55700 1 -- 172.21.15.90:0/2798662536 shutdown_connections 2024-04-14T15:04:52.715 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:52.027+0000 7f318ee55700 1 -- 172.21.15.90:0/2798662536 >> 172.21.15.90:0/2798662536 conn(0x7f31881015c0 msgr2=0x7f3188191940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:04:52.715 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:52.027+0000 7f318ee55700 1 -- 172.21.15.90:0/2798662536 shutdown_connections 2024-04-14T15:04:52.715 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:52.027+0000 7f318ee55700 1 -- 172.21.15.90:0/2798662536 wait complete. 2024-04-14T15:04:52.715 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:52.027+0000 7f318ee55700 1 librados: shutdown 2024-04-14T15:04:52.715 INFO:teuthology.orchestra.run.smithi090.stdout:Deploying unmanaged mgr service... 2024-04-14T15:04:53.435 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:53 smithi090 ceph-mon[30399]: from='client.14138 v1:172.21.15.90:0/2798662536' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:04:53.435 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:53 smithi090 ceph-mon[30399]: Saving service mon spec with placement count:5 2024-04-14T15:04:53.435 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:53 smithi090 ceph-mon[30399]: from='mgr.14116 v1:172.21.15.90:0/2550531201' entity='mgr.y' 2024-04-14T15:04:54.736 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-14T15:04:54.736 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:53.918+0000 7f77e132c700 1 Processor -- start 2024-04-14T15:04:54.736 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:53.918+0000 7f77e132c700 1 -- start start 2024-04-14T15:04:54.737 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:53.919+0000 7f77e132c700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f77dc10a3d0 con 0x7f77dc105fa0 2024-04-14T15:04:54.737 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:53.919+0000 7f77daffd700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f77dc105fa0 0x7f77dc1063c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:45798/0 (socket says 172.21.15.90:45798) 2024-04-14T15:04:54.737 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:53.919+0000 7f77daffd700 1 -- 172.21.15.90:0/936553396 learned_addr learned my addr 172.21.15.90:0/936553396 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:04:54.737 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:53.919+0000 7f77d9ffb700 1 -- 172.21.15.90:0/936553396 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 635609980 0 0) 0x7f77dc10a3d0 con 0x7f77dc105fa0 2024-04-14T15:04:54.737 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:53.919+0000 7f77d9ffb700 1 -- 172.21.15.90:0/936553396 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f77c40035b0 con 0x7f77dc105fa0 2024-04-14T15:04:54.737 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:53.919+0000 7f77d9ffb700 1 -- 172.21.15.90:0/936553396 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2023199185 0 0) 0x7f77c40035b0 con 0x7f77dc105fa0 2024-04-14T15:04:54.737 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:53.919+0000 7f77d9ffb700 1 -- 172.21.15.90:0/936553396 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77dc10b5b0 con 0x7f77dc105fa0 2024-04-14T15:04:54.737 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:53.919+0000 7f77d9ffb700 1 -- 172.21.15.90:0/936553396 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f77cc001580 con 0x7f77dc105fa0 2024-04-14T15:04:54.737 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:53.919+0000 7f77d9ffb700 1 -- 172.21.15.90:0/936553396 <== mon.0 v1:172.21.15.90:6789/0 4 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f77cc004930 con 0x7f77dc105fa0 2024-04-14T15:04:54.737 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:53.919+0000 7f77d9ffb700 1 -- 172.21.15.90:0/936553396 <== mon.0 v1:172.21.15.90:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f77cc005960 con 0x7f77dc105fa0 2024-04-14T15:04:54.737 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:53.920+0000 7f77e132c700 1 -- 172.21.15.90:0/936553396 >> v1:172.21.15.90:6789/0 conn(0x7f77dc105fa0 legacy=0x7f77dc1063c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:04:54.737 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:53.920+0000 7f77e132c700 1 -- 172.21.15.90:0/936553396 shutdown_connections 2024-04-14T15:04:54.737 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:53.920+0000 7f77e132c700 1 -- 172.21.15.90:0/936553396 >> 172.21.15.90:0/936553396 conn(0x7f77dc101580 msgr2=0x7f77dc1039a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:04:54.737 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:53.920+0000 7f77e132c700 1 -- 172.21.15.90:0/936553396 shutdown_connections 2024-04-14T15:04:54.737 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:53.920+0000 7f77e132c700 1 -- 172.21.15.90:0/936553396 wait complete. 2024-04-14T15:04:54.738 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:53.921+0000 7f77e132c700 1 librados: starting msgr at 2024-04-14T15:04:54.738 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:53.921+0000 7f77e132c700 1 librados: starting objecter 2024-04-14T15:04:54.738 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:53.921+0000 7f77e132c700 1 Processor -- start 2024-04-14T15:04:54.738 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:53.921+0000 7f77e132c700 1 -- start start 2024-04-14T15:04:54.738 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:53.921+0000 7f77e132c700 1 librados: setting wanted keys 2024-04-14T15:04:54.738 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:53.921+0000 7f77e132c700 1 librados: calling monclient init 2024-04-14T15:04:54.738 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:53.921+0000 7f77e132c700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f77dc1a0d40 con 0x7f77dc105fa0 2024-04-14T15:04:54.738 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:53.921+0000 7f77daffd700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f77dc105fa0 0x7f77dc1033f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:45812/0 (socket says 172.21.15.90:45812) 2024-04-14T15:04:54.738 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:53.921+0000 7f77daffd700 1 -- 172.21.15.90:0/8161762 learned_addr learned my addr 172.21.15.90:0/8161762 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:04:54.738 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:53.921+0000 7f77c3fff700 10 client.?.objecter ms_handle_connect 0x7f77dc105fa0 2024-04-14T15:04:54.738 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:53.921+0000 7f77c3fff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:04:54.738 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:53.921+0000 7f77c3fff700 1 -- 172.21.15.90:0/8161762 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2068446839 0 0) 0x7f77dc1a0d40 con 0x7f77dc105fa0 2024-04-14T15:04:54.738 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:53.921+0000 7f77c3fff700 1 -- 172.21.15.90:0/8161762 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f77c40056e0 con 0x7f77dc105fa0 2024-04-14T15:04:54.738 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:53.922+0000 7f77c3fff700 1 -- 172.21.15.90:0/8161762 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1557855025 0 0) 0x7f77dc10b5b0 con 0x7f77dc105fa0 2024-04-14T15:04:54.738 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:53.922+0000 7f77c3fff700 1 -- 172.21.15.90:0/8161762 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f77dc1a0d40 con 0x7f77dc105fa0 2024-04-14T15:04:54.739 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:53.922+0000 7f77c3fff700 1 -- 172.21.15.90:0/8161762 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f77cc001080 con 0x7f77dc105fa0 2024-04-14T15:04:54.739 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:53.922+0000 7f77c3fff700 1 -- 172.21.15.90:0/8161762 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3404311745 0 0) 0x7f77dc1a0d40 con 0x7f77dc105fa0 2024-04-14T15:04:54.739 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:53.922+0000 7f77c3fff700 1 -- 172.21.15.90:0/8161762 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77dc1a1f20 con 0x7f77dc105fa0 2024-04-14T15:04:54.739 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:53.922+0000 7f77e132c700 1 -- 172.21.15.90:0/8161762 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f77dc1a0f70 con 0x7f77dc105fa0 2024-04-14T15:04:54.739 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:53.922+0000 7f77e132c700 10 client.14140.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:04:54.739 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:53.922+0000 7f77e132c700 1 -- 172.21.15.90:0/8161762 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f77dc1a14b0 con 0x7f77dc105fa0 2024-04-14T15:04:54.739 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:53.922+0000 7f77e132c700 1 librados: init done 2024-04-14T15:04:54.739 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:53.922+0000 7f77c3fff700 1 -- 172.21.15.90:0/8161762 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f77cc004930 con 0x7f77dc105fa0 2024-04-14T15:04:54.739 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:53.922+0000 7f77c3fff700 1 -- 172.21.15.90:0/8161762 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f77cc006050 con 0x7f77dc105fa0 2024-04-14T15:04:54.739 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:53.923+0000 7f77c3fff700 1 -- 172.21.15.90:0/8161762 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 8) ==== 46523+0+0 (unknown 2307475143 0 0) 0x7f77cc011800 con 0x7f77dc105fa0 2024-04-14T15:04:54.739 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:53.923+0000 7f77e132c700 1 -- 172.21.15.90:0/8161762 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77c8005150 con 0x7f77dc105fa0 2024-04-14T15:04:54.739 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:53.923+0000 7f77c3fff700 1 -- 172.21.15.90:0/8161762 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (unknown 1250798154 0 0) 0x7f77cc004d10 con 0x7f77dc105fa0 2024-04-14T15:04:54.739 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:53.923+0000 7f77c3fff700 10 client.14140.objecter ms_dispatch 0x7f77dc001550 osd_map(2..2 src has 1..2) 2024-04-14T15:04:54.739 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:53.923+0000 7f77c3fff700 3 client.14140.objecter handle_osd_map got epochs [2,2] > 0 2024-04-14T15:04:54.739 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:53.923+0000 7f77c3fff700 3 client.14140.objecter handle_osd_map decoding full epoch 2 2024-04-14T15:04:54.739 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:53.923+0000 7f77c3fff700 20 client.14140.objecter dump_active .. 0 homeless 2024-04-14T15:04:54.740 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:53.923+0000 7f77c3fff700 10 client.14140.objecter ms_handle_connect 0x7f77c403c3c0 2024-04-14T15:04:54.740 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:53.928+0000 7f77c3fff700 1 -- 172.21.15.90:0/8161762 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f77c40056e0 con 0x7f77dc105fa0 2024-04-14T15:04:54.740 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:54.140+0000 7f77e132c700 1 -- 172.21.15.90:0/8161762 --> v1:172.21.15.90:6800/941091171 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f77c8000bc0 con 0x7f77c403c3c0 2024-04-14T15:04:54.740 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:54.148+0000 7f77c3fff700 1 -- 172.21.15.90:0/8161762 <== mgr.14116 v1:172.21.15.90:6800/941091171 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (unknown 0 0 325935098) 0x7f77c8000bc0 con 0x7f77c403c3c0 2024-04-14T15:04:54.740 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:54.150+0000 7f77e132c700 10 librados: watch_flush enter 2024-04-14T15:04:54.740 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:54.150+0000 7f77e132c700 10 librados: watch_flush exit 2024-04-14T15:04:54.740 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:54.150+0000 7f77e132c700 20 client.14140.objecter shutdown clearing up homeless session... 2024-04-14T15:04:54.740 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:54.150+0000 7f77e132c700 10 client.14140.objecter successfully canceled tick 2024-04-14T15:04:54.740 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:54.150+0000 7f77e132c700 1 -- 172.21.15.90:0/8161762 >> v1:172.21.15.90:6800/941091171 conn(0x7f77c403c3c0 legacy=0x7f77c403e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:04:54.740 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:54.150+0000 7f77e132c700 1 -- 172.21.15.90:0/8161762 >> v1:172.21.15.90:6789/0 conn(0x7f77dc105fa0 legacy=0x7f77dc1033f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:04:54.740 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:54.150+0000 7f77e132c700 1 -- 172.21.15.90:0/8161762 shutdown_connections 2024-04-14T15:04:54.740 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:54.150+0000 7f77e132c700 1 -- 172.21.15.90:0/8161762 >> 172.21.15.90:0/8161762 conn(0x7f77dc101580 msgr2=0x7f77dc079fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:04:54.740 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:54.150+0000 7f77e132c700 1 -- 172.21.15.90:0/8161762 shutdown_connections 2024-04-14T15:04:54.740 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:54.150+0000 7f77e132c700 1 -- 172.21.15.90:0/8161762 wait complete. 2024-04-14T15:04:54.740 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:54.150+0000 7f77e132c700 1 librados: shutdown 2024-04-14T15:04:54.983 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:54 smithi090 ceph-mon[30399]: from='mgr.14116 v1:172.21.15.90:0/2550531201' entity='mgr.y' 2024-04-14T15:04:54.983 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:54 smithi090 ceph-mon[30399]: from='client.14140 v1:172.21.15.90:0/8161762' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:04:54.983 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:54 smithi090 ceph-mon[30399]: Saving service mgr spec with placement count:2 2024-04-14T15:04:54.983 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:54 smithi090 ceph-mon[30399]: from='mgr.14116 v1:172.21.15.90:0/2550531201' entity='mgr.y' 2024-04-14T15:04:56.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:55 smithi090 ceph-mon[30399]: from='mgr.14116 v1:172.21.15.90:0/2550531201' entity='mgr.y' 2024-04-14T15:04:56.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:55 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/389924820' entity='client.admin' 2024-04-14T15:04:56.274 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:55.227+0000 7fd9dffc1700 1 Processor -- start 2024-04-14T15:04:56.281 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:55.227+0000 7fd9dffc1700 1 -- start start 2024-04-14T15:04:56.281 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:55.227+0000 7fd9dffc1700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd9d8109270 con 0x7fd9d81086b0 2024-04-14T15:04:56.282 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:55.228+0000 7fd9ddd5d700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fd9d81086b0 0x7fd9d8108ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:45822/0 (socket says 172.21.15.90:45822) 2024-04-14T15:04:56.282 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:55.228+0000 7fd9ddd5d700 1 -- 172.21.15.90:0/2932815263 learned_addr learned my addr 172.21.15.90:0/2932815263 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:04:56.282 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:55.228+0000 7fd9dcd5b700 1 -- 172.21.15.90:0/2932815263 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 428436300 0 0) 0x7fd9d8109270 con 0x7fd9d81086b0 2024-04-14T15:04:56.282 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:55.228+0000 7fd9dcd5b700 1 -- 172.21.15.90:0/2932815263 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9c40035b0 con 0x7fd9d81086b0 2024-04-14T15:04:56.282 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:55.228+0000 7fd9dcd5b700 1 -- 172.21.15.90:0/2932815263 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 797764565 0 0) 0x7fd9c40035b0 con 0x7fd9d81086b0 2024-04-14T15:04:56.282 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:55.228+0000 7fd9dcd5b700 1 -- 172.21.15.90:0/2932815263 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9d810c690 con 0x7fd9d81086b0 2024-04-14T15:04:56.282 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:55.228+0000 7fd9dcd5b700 1 -- 172.21.15.90:0/2932815263 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7fd9c8001580 con 0x7fd9d81086b0 2024-04-14T15:04:56.282 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:55.228+0000 7fd9dcd5b700 1 -- 172.21.15.90:0/2932815263 <== mon.0 v1:172.21.15.90:6789/0 4 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fd9c8004930 con 0x7fd9d81086b0 2024-04-14T15:04:56.282 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:55.228+0000 7fd9dcd5b700 1 -- 172.21.15.90:0/2932815263 <== mon.0 v1:172.21.15.90:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7fd9c8005a70 con 0x7fd9d81086b0 2024-04-14T15:04:56.282 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:55.229+0000 7fd9dffc1700 1 -- 172.21.15.90:0/2932815263 >> v1:172.21.15.90:6789/0 conn(0x7fd9d81086b0 legacy=0x7fd9d8108ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:04:56.282 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:55.229+0000 7fd9dffc1700 1 -- 172.21.15.90:0/2932815263 shutdown_connections 2024-04-14T15:04:56.282 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:55.229+0000 7fd9dffc1700 1 -- 172.21.15.90:0/2932815263 >> 172.21.15.90:0/2932815263 conn(0x7fd9d8100cf0 msgr2=0x7fd9d8103110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:04:56.282 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:55.229+0000 7fd9dffc1700 1 -- 172.21.15.90:0/2932815263 shutdown_connections 2024-04-14T15:04:56.282 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:55.229+0000 7fd9dffc1700 1 -- 172.21.15.90:0/2932815263 wait complete. 2024-04-14T15:04:56.283 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:55.230+0000 7fd9dffc1700 1 librados: starting msgr at 2024-04-14T15:04:56.283 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:55.230+0000 7fd9dffc1700 1 librados: starting objecter 2024-04-14T15:04:56.283 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:55.230+0000 7fd9dffc1700 1 Processor -- start 2024-04-14T15:04:56.283 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:55.230+0000 7fd9dffc1700 1 -- start start 2024-04-14T15:04:56.283 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:55.230+0000 7fd9dffc1700 1 librados: setting wanted keys 2024-04-14T15:04:56.283 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:55.230+0000 7fd9dffc1700 1 librados: calling monclient init 2024-04-14T15:04:56.283 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:55.230+0000 7fd9dffc1700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd9d81a0dd0 con 0x7fd9d81086b0 2024-04-14T15:04:56.283 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:55.230+0000 7fd9ddd5d700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fd9d81086b0 0x7fd9d8107cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:45832/0 (socket says 172.21.15.90:45832) 2024-04-14T15:04:56.283 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:55.230+0000 7fd9ddd5d700 1 -- 172.21.15.90:0/389924820 learned_addr learned my addr 172.21.15.90:0/389924820 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:04:56.283 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:55.230+0000 7fd9ceffd700 10 client.?.objecter ms_handle_connect 0x7fd9d81086b0 2024-04-14T15:04:56.283 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:55.230+0000 7fd9ceffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:04:56.283 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:55.230+0000 7fd9ceffd700 1 -- 172.21.15.90:0/389924820 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3330551427 0 0) 0x7fd9d81a0dd0 con 0x7fd9d81086b0 2024-04-14T15:04:56.283 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:55.230+0000 7fd9ceffd700 1 -- 172.21.15.90:0/389924820 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9c40056e0 con 0x7fd9d81086b0 2024-04-14T15:04:56.283 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:55.231+0000 7fd9ceffd700 1 -- 172.21.15.90:0/389924820 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3885099025 0 0) 0x7fd9d810c690 con 0x7fd9d81086b0 2024-04-14T15:04:56.283 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:55.231+0000 7fd9ceffd700 1 -- 172.21.15.90:0/389924820 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd9d81a0dd0 con 0x7fd9d81086b0 2024-04-14T15:04:56.284 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:55.231+0000 7fd9ceffd700 1 -- 172.21.15.90:0/389924820 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7fd9c8001080 con 0x7fd9d81086b0 2024-04-14T15:04:56.284 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:55.231+0000 7fd9ceffd700 1 -- 172.21.15.90:0/389924820 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3084431985 0 0) 0x7fd9d81a0dd0 con 0x7fd9d81086b0 2024-04-14T15:04:56.284 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:55.231+0000 7fd9ceffd700 1 -- 172.21.15.90:0/389924820 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9d81a1fb0 con 0x7fd9d81086b0 2024-04-14T15:04:56.284 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:55.231+0000 7fd9dffc1700 1 -- 172.21.15.90:0/389924820 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd9d81a1000 con 0x7fd9d81086b0 2024-04-14T15:04:56.284 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:55.231+0000 7fd9dffc1700 10 client.14142.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:04:56.284 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:55.231+0000 7fd9dffc1700 1 -- 172.21.15.90:0/389924820 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd9d81a1540 con 0x7fd9d81086b0 2024-04-14T15:04:56.284 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:55.231+0000 7fd9dffc1700 1 librados: init done 2024-04-14T15:04:56.284 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:55.231+0000 7fd9ceffd700 1 -- 172.21.15.90:0/389924820 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fd9c8004c80 con 0x7fd9d81086b0 2024-04-14T15:04:56.284 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:55.231+0000 7fd9ceffd700 1 -- 172.21.15.90:0/389924820 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7fd9c80062a0 con 0x7fd9d81086b0 2024-04-14T15:04:56.284 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:55.232+0000 7fd9ceffd700 1 -- 172.21.15.90:0/389924820 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 8) ==== 46523+0+0 (unknown 2307475143 0 0) 0x7fd9c8007570 con 0x7fd9d81086b0 2024-04-14T15:04:56.284 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:55.232+0000 7fd9ceffd700 1 -- 172.21.15.90:0/389924820 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (unknown 1250798154 0 0) 0x7fd9c8048da0 con 0x7fd9d81086b0 2024-04-14T15:04:56.284 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:55.232+0000 7fd9ceffd700 10 client.14142.objecter ms_dispatch 0x7fd9d8001550 osd_map(2..2 src has 1..2) 2024-04-14T15:04:56.284 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:55.232+0000 7fd9ceffd700 3 client.14142.objecter handle_osd_map got epochs [2,2] > 0 2024-04-14T15:04:56.285 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:55.232+0000 7fd9ceffd700 3 client.14142.objecter handle_osd_map decoding full epoch 2 2024-04-14T15:04:56.285 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:55.232+0000 7fd9ceffd700 20 client.14142.objecter dump_active .. 0 homeless 2024-04-14T15:04:56.285 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:55.232+0000 7fd9dffc1700 1 -- 172.21.15.90:0/389924820 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9bc005150 con 0x7fd9d81086b0 2024-04-14T15:04:56.285 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:55.232+0000 7fd9ceffd700 10 client.14142.objecter ms_handle_connect 0x7fd9c403c480 2024-04-14T15:04:56.285 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:55.237+0000 7fd9ceffd700 1 -- 172.21.15.90:0/389924820 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fd9c40056e0 con 0x7fd9d81086b0 2024-04-14T15:04:56.285 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:55.461+0000 7fd9dffc1700 1 -- 172.21.15.90:0/389924820 --> v1:172.21.15.90:6789/0 -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7fd9bc005440 con 0x7fd9d81086b0 2024-04-14T15:04:56.285 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:55.463+0000 7fd9ceffd700 1 -- 172.21.15.90:0/389924820 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (unknown 2684808024 0 0) 0x7fd9c8017670 con 0x7fd9d81086b0 2024-04-14T15:04:56.285 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:55.465+0000 7fd9dffc1700 10 librados: watch_flush enter 2024-04-14T15:04:56.285 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:55.465+0000 7fd9dffc1700 10 librados: watch_flush exit 2024-04-14T15:04:56.285 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:55.465+0000 7fd9dffc1700 20 client.14142.objecter shutdown clearing up homeless session... 2024-04-14T15:04:56.285 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:55.465+0000 7fd9dffc1700 10 client.14142.objecter successfully canceled tick 2024-04-14T15:04:56.285 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:55.466+0000 7fd9dffc1700 1 -- 172.21.15.90:0/389924820 >> v1:172.21.15.90:6800/941091171 conn(0x7fd9c403c480 legacy=0x7fd9c403e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:04:56.285 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:55.466+0000 7fd9dffc1700 1 -- 172.21.15.90:0/389924820 >> v1:172.21.15.90:6789/0 conn(0x7fd9d81086b0 legacy=0x7fd9d8107cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:04:56.285 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:55.466+0000 7fd9dffc1700 1 -- 172.21.15.90:0/389924820 shutdown_connections 2024-04-14T15:04:56.285 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:55.466+0000 7fd9dffc1700 1 -- 172.21.15.90:0/389924820 >> 172.21.15.90:0/389924820 conn(0x7fd9d8100cf0 msgr2=0x7fd9d81919a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:04:56.286 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:55.466+0000 7fd9dffc1700 1 -- 172.21.15.90:0/389924820 shutdown_connections 2024-04-14T15:04:56.286 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:55.466+0000 7fd9dffc1700 1 -- 172.21.15.90:0/389924820 wait complete. 2024-04-14T15:04:56.286 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:55.466+0000 7fd9dffc1700 1 librados: shutdown 2024-04-14T15:04:57.701 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:56.732+0000 7fc592a94700 1 Processor -- start 2024-04-14T15:04:57.701 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:56.732+0000 7fc592a94700 1 -- start start 2024-04-14T15:04:57.701 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:56.732+0000 7fc592a94700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc58c10a3d0 con 0x7fc58c107530 2024-04-14T15:04:57.701 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:56.732+0000 7fc590830700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fc58c107530 0x7fc58c107950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:45848/0 (socket says 172.21.15.90:45848) 2024-04-14T15:04:57.701 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:56.732+0000 7fc590830700 1 -- 172.21.15.90:0/107103986 learned_addr learned my addr 172.21.15.90:0/107103986 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:04:57.702 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:56.732+0000 7fc58b7fe700 1 -- 172.21.15.90:0/107103986 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3667737972 0 0) 0x7fc58c10a3d0 con 0x7fc58c107530 2024-04-14T15:04:57.702 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:56.733+0000 7fc58b7fe700 1 -- 172.21.15.90:0/107103986 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc57c0057c0 con 0x7fc58c107530 2024-04-14T15:04:57.702 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:56.733+0000 7fc58b7fe700 1 -- 172.21.15.90:0/107103986 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3946752727 0 0) 0x7fc57c0057c0 con 0x7fc58c107530 2024-04-14T15:04:57.702 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:56.733+0000 7fc58b7fe700 1 -- 172.21.15.90:0/107103986 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc58c10b5b0 con 0x7fc58c107530 2024-04-14T15:04:57.702 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:56.733+0000 7fc58b7fe700 1 -- 172.21.15.90:0/107103986 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7fc580001580 con 0x7fc58c107530 2024-04-14T15:04:57.702 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:56.733+0000 7fc58b7fe700 1 -- 172.21.15.90:0/107103986 <== mon.0 v1:172.21.15.90:6789/0 4 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fc580004930 con 0x7fc58c107530 2024-04-14T15:04:57.702 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:56.733+0000 7fc58b7fe700 1 -- 172.21.15.90:0/107103986 <== mon.0 v1:172.21.15.90:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7fc580005a70 con 0x7fc58c107530 2024-04-14T15:04:57.702 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:56.733+0000 7fc592a94700 1 -- 172.21.15.90:0/107103986 >> v1:172.21.15.90:6789/0 conn(0x7fc58c107530 legacy=0x7fc58c107950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:04:57.702 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:56.733+0000 7fc592a94700 1 -- 172.21.15.90:0/107103986 shutdown_connections 2024-04-14T15:04:57.702 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:56.733+0000 7fc592a94700 1 -- 172.21.15.90:0/107103986 >> 172.21.15.90:0/107103986 conn(0x7fc58c100c80 msgr2=0x7fc58c1030e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:04:57.702 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:56.733+0000 7fc592a94700 1 -- 172.21.15.90:0/107103986 shutdown_connections 2024-04-14T15:04:57.703 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:56.733+0000 7fc592a94700 1 -- 172.21.15.90:0/107103986 wait complete. 2024-04-14T15:04:57.703 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:56.734+0000 7fc592a94700 1 librados: starting msgr at 2024-04-14T15:04:57.703 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:56.734+0000 7fc592a94700 1 librados: starting objecter 2024-04-14T15:04:57.703 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:56.734+0000 7fc592a94700 1 Processor -- start 2024-04-14T15:04:57.703 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:56.734+0000 7fc592a94700 1 -- start start 2024-04-14T15:04:57.703 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:56.734+0000 7fc592a94700 1 librados: setting wanted keys 2024-04-14T15:04:57.703 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:56.734+0000 7fc592a94700 1 librados: calling monclient init 2024-04-14T15:04:57.703 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:56.734+0000 7fc592a94700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc58c1991c0 con 0x7fc58c107530 2024-04-14T15:04:57.703 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:56.734+0000 7fc590830700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fc58c107530 0x7fc58c198ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:45858/0 (socket says 172.21.15.90:45858) 2024-04-14T15:04:57.703 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:56.734+0000 7fc590830700 1 -- 172.21.15.90:0/2253686230 learned_addr learned my addr 172.21.15.90:0/2253686230 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:04:57.703 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:56.734+0000 7fc589ffb700 10 client.?.objecter ms_handle_connect 0x7fc58c107530 2024-04-14T15:04:57.703 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:56.734+0000 7fc589ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:04:57.704 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:56.734+0000 7fc589ffb700 1 -- 172.21.15.90:0/2253686230 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2052156221 0 0) 0x7fc58c1991c0 con 0x7fc58c107530 2024-04-14T15:04:57.704 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:56.734+0000 7fc589ffb700 1 -- 172.21.15.90:0/2253686230 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc57c0078f0 con 0x7fc58c107530 2024-04-14T15:04:57.704 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:56.735+0000 7fc589ffb700 1 -- 172.21.15.90:0/2253686230 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2520633101 0 0) 0x7fc58c10b5b0 con 0x7fc58c107530 2024-04-14T15:04:57.704 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:56.735+0000 7fc589ffb700 1 -- 172.21.15.90:0/2253686230 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc58c1991c0 con 0x7fc58c107530 2024-04-14T15:04:57.704 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:56.735+0000 7fc589ffb700 1 -- 172.21.15.90:0/2253686230 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7fc580001080 con 0x7fc58c107530 2024-04-14T15:04:57.704 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:56.735+0000 7fc589ffb700 1 -- 172.21.15.90:0/2253686230 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 460949396 0 0) 0x7fc58c1991c0 con 0x7fc58c107530 2024-04-14T15:04:57.704 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:56.735+0000 7fc589ffb700 1 -- 172.21.15.90:0/2253686230 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc58c199390 con 0x7fc58c107530 2024-04-14T15:04:57.704 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:56.735+0000 7fc592a94700 1 -- 172.21.15.90:0/2253686230 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc58c199680 con 0x7fc58c107530 2024-04-14T15:04:57.704 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:56.735+0000 7fc589ffb700 1 -- 172.21.15.90:0/2253686230 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fc580004c80 con 0x7fc58c107530 2024-04-14T15:04:57.704 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:56.735+0000 7fc589ffb700 1 -- 172.21.15.90:0/2253686230 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7fc580006320 con 0x7fc58c107530 2024-04-14T15:04:57.704 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:56.735+0000 7fc592a94700 10 client.14144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:04:57.704 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:56.735+0000 7fc592a94700 1 -- 172.21.15.90:0/2253686230 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc58c19d1d0 con 0x7fc58c107530 2024-04-14T15:04:57.704 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:56.735+0000 7fc592a94700 1 librados: init done 2024-04-14T15:04:57.705 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:56.736+0000 7fc589ffb700 1 -- 172.21.15.90:0/2253686230 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 8) ==== 46523+0+0 (unknown 2307475143 0 0) 0x7fc5800064e0 con 0x7fc58c107530 2024-04-14T15:04:57.705 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:56.736+0000 7fc592a94700 1 -- 172.21.15.90:0/2253686230 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc58c064110 con 0x7fc58c107530 2024-04-14T15:04:57.705 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:56.736+0000 7fc589ffb700 1 -- 172.21.15.90:0/2253686230 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (unknown 1250798154 0 0) 0x7fc580047fd0 con 0x7fc58c107530 2024-04-14T15:04:57.705 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:56.736+0000 7fc589ffb700 10 client.14144.objecter ms_dispatch 0x7fc58c001550 osd_map(2..2 src has 1..2) 2024-04-14T15:04:57.705 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:56.736+0000 7fc589ffb700 3 client.14144.objecter handle_osd_map got epochs [2,2] > 0 2024-04-14T15:04:57.705 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:56.736+0000 7fc589ffb700 3 client.14144.objecter handle_osd_map decoding full epoch 2 2024-04-14T15:04:57.705 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:56.736+0000 7fc589ffb700 20 client.14144.objecter dump_active .. 0 homeless 2024-04-14T15:04:57.705 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:56.736+0000 7fc589ffb700 10 client.14144.objecter ms_handle_connect 0x7fc57c03e670 2024-04-14T15:04:57.705 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:56.741+0000 7fc589ffb700 1 -- 172.21.15.90:0/2253686230 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fc57c0078f0 con 0x7fc58c107530 2024-04-14T15:04:57.705 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:56.938+0000 7fc592a94700 1 -- 172.21.15.90:0/2253686230 --> v1:172.21.15.90:6789/0 -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7fc58c10b220 con 0x7fc58c107530 2024-04-14T15:04:57.705 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:56.940+0000 7fc589ffb700 1 -- 172.21.15.90:0/2253686230 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (unknown 1496729260 0 0) 0x7fc5800176d0 con 0x7fc58c107530 2024-04-14T15:04:57.706 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:56.943+0000 7fc592a94700 10 librados: watch_flush enter 2024-04-14T15:04:57.706 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:56.943+0000 7fc592a94700 10 librados: watch_flush exit 2024-04-14T15:04:57.706 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:56.943+0000 7fc592a94700 20 client.14144.objecter shutdown clearing up homeless session... 2024-04-14T15:04:57.706 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:56.943+0000 7fc592a94700 10 client.14144.objecter successfully canceled tick 2024-04-14T15:04:57.706 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:56.943+0000 7fc592a94700 1 -- 172.21.15.90:0/2253686230 >> v1:172.21.15.90:6800/941091171 conn(0x7fc57c03e670 legacy=0x7fc57c040b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:04:57.706 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:56.943+0000 7fc592a94700 1 -- 172.21.15.90:0/2253686230 >> v1:172.21.15.90:6789/0 conn(0x7fc58c107530 legacy=0x7fc58c198ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:04:57.706 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:56.943+0000 7fc592a94700 1 -- 172.21.15.90:0/2253686230 shutdown_connections 2024-04-14T15:04:57.706 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:56.943+0000 7fc592a94700 1 -- 172.21.15.90:0/2253686230 >> 172.21.15.90:0/2253686230 conn(0x7fc58c100c80 msgr2=0x7fc58c1030e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:04:57.706 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:56.943+0000 7fc592a94700 1 -- 172.21.15.90:0/2253686230 shutdown_connections 2024-04-14T15:04:57.706 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:56.943+0000 7fc592a94700 1 -- 172.21.15.90:0/2253686230 wait complete. 2024-04-14T15:04:57.706 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:56.943+0000 7fc592a94700 1 librados: shutdown 2024-04-14T15:04:57.706 INFO:teuthology.orchestra.run.smithi090.stdout:Enabling the dashboard module... 2024-04-14T15:04:58.119 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:57 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2253686230' entity='client.admin' 2024-04-14T15:04:58.120 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:57 smithi090 ceph-mon[30399]: from='mgr.14116 v1:172.21.15.90:0/2550531201' entity='mgr.y' 2024-04-14T15:04:59.275 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:58 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2399910882' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-14T15:04:59.275 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:58 smithi090 ceph-mon[30399]: from='mgr.14116 v1:172.21.15.90:0/2550531201' entity='mgr.y' 2024-04-14T15:04:59.275 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:59 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ignoring --setuser ceph since I am not root 2024-04-14T15:04:59.275 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:59 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ignoring --setgroup ceph since I am not root 2024-04-14T15:04:59.552 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:58.127+0000 7f3bd8537700 1 Processor -- start 2024-04-14T15:04:59.552 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:58.127+0000 7f3bd8537700 1 -- start start 2024-04-14T15:04:59.552 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:58.127+0000 7f3bd8537700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3bd0109b60 con 0x7f3bd0105730 2024-04-14T15:04:59.552 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:58.127+0000 7f3bd62d3700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f3bd0105730 0x7f3bd0105b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:45864/0 (socket says 172.21.15.90:45864) 2024-04-14T15:04:59.552 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:58.127+0000 7f3bd62d3700 1 -- 172.21.15.90:0/1377341878 learned_addr learned my addr 172.21.15.90:0/1377341878 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:04:59.552 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:58.128+0000 7f3bd52d1700 1 -- 172.21.15.90:0/1377341878 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2344388214 0 0) 0x7f3bd0109b60 con 0x7f3bd0105730 2024-04-14T15:04:59.552 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:58.128+0000 7f3bd52d1700 1 -- 172.21.15.90:0/1377341878 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3bbc0034d0 con 0x7f3bd0105730 2024-04-14T15:04:59.552 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:58.128+0000 7f3bd52d1700 1 -- 172.21.15.90:0/1377341878 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1077007249 0 0) 0x7f3bbc0034d0 con 0x7f3bd0105730 2024-04-14T15:04:59.553 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:58.128+0000 7f3bd52d1700 1 -- 172.21.15.90:0/1377341878 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bd010ad40 con 0x7f3bd0105730 2024-04-14T15:04:59.553 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:58.128+0000 7f3bd52d1700 1 -- 172.21.15.90:0/1377341878 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f3bcc001580 con 0x7f3bd0105730 2024-04-14T15:04:59.553 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:58.128+0000 7f3bd52d1700 1 -- 172.21.15.90:0/1377341878 <== mon.0 v1:172.21.15.90:6789/0 4 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f3bcc004930 con 0x7f3bd0105730 2024-04-14T15:04:59.553 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:58.128+0000 7f3bd52d1700 1 -- 172.21.15.90:0/1377341878 <== mon.0 v1:172.21.15.90:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f3bcc005960 con 0x7f3bd0105730 2024-04-14T15:04:59.553 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:58.129+0000 7f3bd8537700 1 -- 172.21.15.90:0/1377341878 >> v1:172.21.15.90:6789/0 conn(0x7f3bd0105730 legacy=0x7f3bd0105b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:04:59.553 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:58.129+0000 7f3bd8537700 1 -- 172.21.15.90:0/1377341878 shutdown_connections 2024-04-14T15:04:59.553 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:58.129+0000 7f3bd8537700 1 -- 172.21.15.90:0/1377341878 >> 172.21.15.90:0/1377341878 conn(0x7f3bd0100cf0 msgr2=0x7f3bd0103110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:04:59.553 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:58.129+0000 7f3bd8537700 1 -- 172.21.15.90:0/1377341878 shutdown_connections 2024-04-14T15:04:59.553 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:58.129+0000 7f3bd8537700 1 -- 172.21.15.90:0/1377341878 wait complete. 2024-04-14T15:04:59.553 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:58.130+0000 7f3bd8537700 1 librados: starting msgr at 2024-04-14T15:04:59.553 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:58.130+0000 7f3bd8537700 1 librados: starting objecter 2024-04-14T15:04:59.553 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:58.130+0000 7f3bd8537700 1 Processor -- start 2024-04-14T15:04:59.554 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:58.130+0000 7f3bd8537700 1 -- start start 2024-04-14T15:04:59.554 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:58.130+0000 7f3bd8537700 1 librados: setting wanted keys 2024-04-14T15:04:59.554 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:58.130+0000 7f3bd8537700 1 librados: calling monclient init 2024-04-14T15:04:59.554 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:58.130+0000 7f3bd8537700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3bd019d5d0 con 0x7f3bd0105730 2024-04-14T15:04:59.554 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:58.130+0000 7f3bd62d3700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f3bd0105730 0x7f3bd019cec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:45878/0 (socket says 172.21.15.90:45878) 2024-04-14T15:04:59.554 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:58.130+0000 7f3bd62d3700 1 -- 172.21.15.90:0/2399910882 learned_addr learned my addr 172.21.15.90:0/2399910882 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:04:59.554 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:58.130+0000 7f3bc77fe700 10 client.?.objecter ms_handle_connect 0x7f3bd0105730 2024-04-14T15:04:59.554 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:58.130+0000 7f3bc77fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:04:59.554 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:58.130+0000 7f3bc77fe700 1 -- 172.21.15.90:0/2399910882 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1018954354 0 0) 0x7f3bd019d5d0 con 0x7f3bd0105730 2024-04-14T15:04:59.554 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:58.130+0000 7f3bc77fe700 1 -- 172.21.15.90:0/2399910882 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3bbc003480 con 0x7f3bd0105730 2024-04-14T15:04:59.554 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:58.130+0000 7f3bc77fe700 1 -- 172.21.15.90:0/2399910882 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 805044028 0 0) 0x7f3bbc003480 con 0x7f3bd0105730 2024-04-14T15:04:59.555 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:58.131+0000 7f3bc77fe700 1 -- 172.21.15.90:0/2399910882 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3bd019d5d0 con 0x7f3bd0105730 2024-04-14T15:04:59.555 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:58.131+0000 7f3bc77fe700 1 -- 172.21.15.90:0/2399910882 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f3bcc001080 con 0x7f3bd0105730 2024-04-14T15:04:59.555 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:58.131+0000 7f3bc77fe700 1 -- 172.21.15.90:0/2399910882 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4046238429 0 0) 0x7f3bd019d5d0 con 0x7f3bd0105730 2024-04-14T15:04:59.555 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:58.131+0000 7f3bc77fe700 1 -- 172.21.15.90:0/2399910882 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bd019d7a0 con 0x7f3bd0105730 2024-04-14T15:04:59.555 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:58.131+0000 7f3bd8537700 1 -- 172.21.15.90:0/2399910882 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3bd019da90 con 0x7f3bd0105730 2024-04-14T15:04:59.555 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:58.131+0000 7f3bd8537700 10 client.14146.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:04:59.555 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:58.131+0000 7f3bd8537700 1 -- 172.21.15.90:0/2399910882 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f3bd01a15e0 con 0x7f3bd0105730 2024-04-14T15:04:59.555 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:58.131+0000 7f3bd8537700 1 librados: init done 2024-04-14T15:04:59.555 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:58.131+0000 7f3bc77fe700 1 -- 172.21.15.90:0/2399910882 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f3bcc004930 con 0x7f3bd0105730 2024-04-14T15:04:59.555 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:58.131+0000 7f3bc77fe700 1 -- 172.21.15.90:0/2399910882 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f3bcc006090 con 0x7f3bd0105730 2024-04-14T15:04:59.555 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:58.132+0000 7f3bc77fe700 1 -- 172.21.15.90:0/2399910882 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 8) ==== 46523+0+0 (unknown 2307475143 0 0) 0x7f3bcc011820 con 0x7f3bd0105730 2024-04-14T15:04:59.555 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:58.132+0000 7f3bd8537700 1 -- 172.21.15.90:0/2399910882 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bb4005150 con 0x7f3bd0105730 2024-04-14T15:04:59.555 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:58.132+0000 7f3bc77fe700 1 -- 172.21.15.90:0/2399910882 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (unknown 1250798154 0 0) 0x7f3bcc048100 con 0x7f3bd0105730 2024-04-14T15:04:59.556 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:58.132+0000 7f3bc77fe700 10 client.14146.objecter ms_dispatch 0x7f3bd0001550 osd_map(2..2 src has 1..2) 2024-04-14T15:04:59.556 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:58.132+0000 7f3bc77fe700 3 client.14146.objecter handle_osd_map got epochs [2,2] > 0 2024-04-14T15:04:59.556 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:58.132+0000 7f3bc77fe700 3 client.14146.objecter handle_osd_map decoding full epoch 2 2024-04-14T15:04:59.556 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:58.132+0000 7f3bc77fe700 20 client.14146.objecter dump_active .. 0 homeless 2024-04-14T15:04:59.556 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:58.132+0000 7f3bc77fe700 10 client.14146.objecter ms_handle_connect 0x7f3bbc03c400 2024-04-14T15:04:59.556 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:58.137+0000 7f3bc77fe700 1 -- 172.21.15.90:0/2399910882 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f3bcc017750 con 0x7f3bd0105730 2024-04-14T15:04:59.556 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:58.402+0000 7f3bd8537700 1 -- 172.21.15.90:0/2399910882 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7f3bb4005440 con 0x7f3bd0105730 2024-04-14T15:04:59.556 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:58.951+0000 7f3bc77fe700 1 -- 172.21.15.90:0/2399910882 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (unknown 1498667528 0 0) 0x7f3bcc017050 con 0x7f3bd0105730 2024-04-14T15:04:59.556 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:58.953+0000 7f3bc77fe700 1 -- 172.21.15.90:0/2399910882 <== mon.0 v1:172.21.15.90:6789/0 11 ==== mgrmap(e 9) ==== 46536+0+0 (unknown 2298455986 0 0) 0x7f3bcc046aa0 con 0x7f3bd0105730 2024-04-14T15:04:59.556 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:58.954+0000 7f3bd8537700 10 librados: watch_flush enter 2024-04-14T15:04:59.556 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:58.954+0000 7f3bd8537700 10 librados: watch_flush exit 2024-04-14T15:04:59.556 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:58.954+0000 7f3bd8537700 20 client.14146.objecter shutdown clearing up homeless session... 2024-04-14T15:04:59.557 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:58.954+0000 7f3bd8537700 10 client.14146.objecter successfully canceled tick 2024-04-14T15:04:59.557 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:58.954+0000 7f3bd8537700 1 -- 172.21.15.90:0/2399910882 >> v1:172.21.15.90:6800/941091171 conn(0x7f3bbc03c400 legacy=0x7f3bbc03e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:04:59.557 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:58.955+0000 7f3bd8537700 1 -- 172.21.15.90:0/2399910882 >> v1:172.21.15.90:6789/0 conn(0x7f3bd0105730 legacy=0x7f3bd019cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:04:59.557 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:58.955+0000 7f3bd8537700 1 -- 172.21.15.90:0/2399910882 shutdown_connections 2024-04-14T15:04:59.557 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:58.955+0000 7f3bd8537700 1 -- 172.21.15.90:0/2399910882 >> 172.21.15.90:0/2399910882 conn(0x7f3bd0100cf0 msgr2=0x7f3bd018fc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:04:59.557 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:58.955+0000 7f3bd8537700 1 -- 172.21.15.90:0/2399910882 shutdown_connections 2024-04-14T15:04:59.557 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:58.955+0000 7f3bd8537700 1 -- 172.21.15.90:0/2399910882 wait complete. 2024-04-14T15:04:59.557 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:04:58.955+0000 7f3bd8537700 1 librados: shutdown 2024-04-14T15:04:59.789 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:59 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:04:59.531+0000 7f48f5deb200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-14T15:04:59.790 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:04:59 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:04:59.633+0000 7f48f5deb200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-14T15:05:00.118 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:59 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2399910882' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-14T15:05:00.118 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:04:59 smithi090 ceph-mon[30399]: mgrmap e9: y(active, since 19s) 2024-04-14T15:05:00.391 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:05:00 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:05:00.308+0000 7f48f5deb200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-14T15:05:00.644 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:05:00 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:05:00.407+0000 7f48f5deb200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-14T15:05:00.644 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:05:00 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:05:00.551+0000 7f48f5deb200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-14T15:05:00.941 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:05:00 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:05:00.646+0000 7f48f5deb200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-14T15:05:00.941 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:05:00 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:05:00.730+0000 7f48f5deb200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-14T15:05:01.198 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:00 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1917578779' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-14T15:05:01.198 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:05:01 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:05:01.002+0000 7f48f5deb200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-14T15:05:01.198 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:05:01 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:05:01.112+0000 7f48f5deb200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-14T15:05:01.475 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:05:01 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:05:01.198+0000 7f48f5deb200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-14T15:05:01.475 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:05:01 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:05:01.296+0000 7f48f5deb200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-14T15:05:01.829 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:05:01 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:05:01.475+0000 7f48f5deb200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-14T15:05:01.837 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout { 2024-04-14T15:05:01.837 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-14T15:05:01.837 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-14T15:05:01.837 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-04-14T15:05:01.837 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-14T15:05:01.837 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout } 2024-04-14T15:05:01.838 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:00.010+0000 7f450da79700 1 Processor -- start 2024-04-14T15:05:01.838 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:00.010+0000 7f450da79700 1 -- start start 2024-04-14T15:05:01.838 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:00.010+0000 7f450da79700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f450810a440 con 0x7f4508107630 2024-04-14T15:05:01.838 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:00.010+0000 7f45077fe700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f4508107630 0x7f4508107a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:45902/0 (socket says 172.21.15.90:45902) 2024-04-14T15:05:01.838 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:00.010+0000 7f45077fe700 1 -- 172.21.15.90:0/308913857 learned_addr learned my addr 172.21.15.90:0/308913857 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:05:01.838 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:00.011+0000 7f45067fc700 1 -- 172.21.15.90:0/308913857 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3525093724 0 0) 0x7f450810a440 con 0x7f4508107630 2024-04-14T15:05:01.838 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:00.011+0000 7f45067fc700 1 -- 172.21.15.90:0/308913857 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f44f00035b0 con 0x7f4508107630 2024-04-14T15:05:01.838 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:00.011+0000 7f45067fc700 1 -- 172.21.15.90:0/308913857 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3258469084 0 0) 0x7f44f00035b0 con 0x7f4508107630 2024-04-14T15:05:01.838 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:00.011+0000 7f45067fc700 1 -- 172.21.15.90:0/308913857 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f450810b620 con 0x7f4508107630 2024-04-14T15:05:01.838 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:00.011+0000 7f45067fc700 1 -- 172.21.15.90:0/308913857 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f44f8001560 con 0x7f4508107630 2024-04-14T15:05:01.838 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:00.011+0000 7f45067fc700 1 -- 172.21.15.90:0/308913857 <== mon.0 v1:172.21.15.90:6789/0 4 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f44f8001d30 con 0x7f4508107630 2024-04-14T15:05:01.838 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:00.012+0000 7f45067fc700 1 -- 172.21.15.90:0/308913857 <== mon.0 v1:172.21.15.90:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f44f8001560 con 0x7f4508107630 2024-04-14T15:05:01.838 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:00.012+0000 7f450da79700 1 -- 172.21.15.90:0/308913857 >> v1:172.21.15.90:6789/0 conn(0x7f4508107630 legacy=0x7f4508107a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:05:01.839 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:00.012+0000 7f450da79700 1 -- 172.21.15.90:0/308913857 shutdown_connections 2024-04-14T15:05:01.839 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:00.012+0000 7f450da79700 1 -- 172.21.15.90:0/308913857 >> 172.21.15.90:0/308913857 conn(0x7f4508100cf0 msgr2=0x7f4508103110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:05:01.839 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:00.012+0000 7f450da79700 1 -- 172.21.15.90:0/308913857 shutdown_connections 2024-04-14T15:05:01.839 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:00.012+0000 7f450da79700 1 -- 172.21.15.90:0/308913857 wait complete. 2024-04-14T15:05:01.839 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:00.013+0000 7f450da79700 1 librados: starting msgr at 2024-04-14T15:05:01.839 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:00.013+0000 7f450da79700 1 librados: starting objecter 2024-04-14T15:05:01.839 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:00.013+0000 7f450da79700 1 Processor -- start 2024-04-14T15:05:01.839 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:00.013+0000 7f450da79700 1 -- start start 2024-04-14T15:05:01.839 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:00.013+0000 7f450da79700 1 librados: setting wanted keys 2024-04-14T15:05:01.839 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:00.013+0000 7f450da79700 1 librados: calling monclient init 2024-04-14T15:05:01.839 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:00.013+0000 7f450da79700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f450819b660 con 0x7f4508107630 2024-04-14T15:05:01.839 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:00.013+0000 7f45077fe700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f4508107630 0x7f450819af50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:45904/0 (socket says 172.21.15.90:45904) 2024-04-14T15:05:01.839 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:00.013+0000 7f45077fe700 1 -- 172.21.15.90:0/1917578779 learned_addr learned my addr 172.21.15.90:0/1917578779 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:05:01.839 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:00.014+0000 7f4504ff9700 10 client.?.objecter ms_handle_connect 0x7f4508107630 2024-04-14T15:05:01.840 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:00.014+0000 7f4504ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:05:01.840 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:00.014+0000 7f4504ff9700 1 -- 172.21.15.90:0/1917578779 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3246982902 0 0) 0x7f450819b660 con 0x7f4508107630 2024-04-14T15:05:01.840 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:00.014+0000 7f4504ff9700 1 -- 172.21.15.90:0/1917578779 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f44f00031f0 con 0x7f4508107630 2024-04-14T15:05:01.840 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:00.014+0000 7f4504ff9700 1 -- 172.21.15.90:0/1917578779 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1863486258 0 0) 0x7f44f00031f0 con 0x7f4508107630 2024-04-14T15:05:01.840 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:00.014+0000 7f4504ff9700 1 -- 172.21.15.90:0/1917578779 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f450819b660 con 0x7f4508107630 2024-04-14T15:05:01.840 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:00.014+0000 7f4504ff9700 1 -- 172.21.15.90:0/1917578779 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f44f8005380 con 0x7f4508107630 2024-04-14T15:05:01.840 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:00.014+0000 7f4504ff9700 1 -- 172.21.15.90:0/1917578779 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2115809398 0 0) 0x7f450819b660 con 0x7f4508107630 2024-04-14T15:05:01.840 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:00.014+0000 7f4504ff9700 1 -- 172.21.15.90:0/1917578779 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f450819b830 con 0x7f4508107630 2024-04-14T15:05:01.840 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:00.014+0000 7f450da79700 1 -- 172.21.15.90:0/1917578779 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f450819bb20 con 0x7f4508107630 2024-04-14T15:05:01.840 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:00.014+0000 7f450da79700 10 client.14150.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:05:01.840 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:00.014+0000 7f450da79700 1 -- 172.21.15.90:0/1917578779 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f450819f670 con 0x7f4508107630 2024-04-14T15:05:01.840 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:00.014+0000 7f450da79700 1 librados: init done 2024-04-14T15:05:01.840 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:00.015+0000 7f4504ff9700 1 -- 172.21.15.90:0/1917578779 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f44f8004ee0 con 0x7f4508107630 2024-04-14T15:05:01.841 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:00.015+0000 7f4504ff9700 1 -- 172.21.15.90:0/1917578779 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f44f8006350 con 0x7f4508107630 2024-04-14T15:05:01.841 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:00.015+0000 7f4504ff9700 1 -- 172.21.15.90:0/1917578779 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 9) ==== 46536+0+0 (unknown 2298455986 0 0) 0x7f44f8011ba0 con 0x7f4508107630 2024-04-14T15:05:01.841 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:00.016+0000 7f4504ff9700 1 -- 172.21.15.90:0/1917578779 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (unknown 1250798154 0 0) 0x7f44f8049190 con 0x7f4508107630 2024-04-14T15:05:01.841 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:00.016+0000 7f4504ff9700 10 client.14150.objecter ms_dispatch 0x7f4508001550 osd_map(2..2 src has 1..2) 2024-04-14T15:05:01.841 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:00.016+0000 7f4504ff9700 3 client.14150.objecter handle_osd_map got epochs [2,2] > 0 2024-04-14T15:05:01.841 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:00.016+0000 7f4504ff9700 3 client.14150.objecter handle_osd_map decoding full epoch 2 2024-04-14T15:05:01.841 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:00.016+0000 7f4506ffd700 1 -- 172.21.15.90:0/1917578779 >> v1:172.21.15.90:6800/941091171 conn(0x7f44f003c4c0 legacy=0x7f44f003e980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.90:6800/941091171 2024-04-14T15:05:01.841 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:00.016+0000 7f4504ff9700 20 client.14150.objecter dump_active .. 0 homeless 2024-04-14T15:05:01.841 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:00.016+0000 7f450da79700 1 -- 172.21.15.90:0/1917578779 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f450810b870 con 0x7f4508107630 2024-04-14T15:05:01.841 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:00.025+0000 7f4504ff9700 1 -- 172.21.15.90:0/1917578779 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f44f8017a60 con 0x7f4508107630 2024-04-14T15:05:01.841 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:00.216+0000 7f4506ffd700 1 -- 172.21.15.90:0/1917578779 >> v1:172.21.15.90:6800/941091171 conn(0x7f44f003c4c0 legacy=0x7f44f003e980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.90:6800/941091171 2024-04-14T15:05:01.841 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:00.303+0000 7f450da79700 1 -- 172.21.15.90:0/1917578779 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f4508064110 con 0x7f4508107630 2024-04-14T15:05:01.842 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:00.303+0000 7f4504ff9700 1 -- 172.21.15.90:0/1917578779 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (unknown 2005831594 0 1748205903) 0x7f4508064110 con 0x7f4508107630 2024-04-14T15:05:01.842 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:00.305+0000 7f450da79700 10 librados: watch_flush enter 2024-04-14T15:05:01.842 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:00.305+0000 7f450da79700 10 librados: watch_flush exit 2024-04-14T15:05:01.842 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:00.305+0000 7f450da79700 20 client.14150.objecter shutdown clearing up homeless session... 2024-04-14T15:05:01.842 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:00.305+0000 7f450da79700 10 client.14150.objecter successfully canceled tick 2024-04-14T15:05:01.842 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:00.305+0000 7f450da79700 1 -- 172.21.15.90:0/1917578779 >> v1:172.21.15.90:6800/941091171 conn(0x7f44f003c4c0 legacy=0x7f44f003e980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-14T15:05:01.842 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:00.305+0000 7f450da79700 1 -- 172.21.15.90:0/1917578779 >> v1:172.21.15.90:6789/0 conn(0x7f4508107630 legacy=0x7f450819af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:05:01.842 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:00.305+0000 7f450da79700 1 -- 172.21.15.90:0/1917578779 shutdown_connections 2024-04-14T15:05:01.842 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:00.305+0000 7f450da79700 1 -- 172.21.15.90:0/1917578779 >> 172.21.15.90:0/1917578779 conn(0x7f4508100cf0 msgr2=0x7f45081050b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:05:01.842 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:00.305+0000 7f450da79700 1 -- 172.21.15.90:0/1917578779 shutdown_connections 2024-04-14T15:05:01.842 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:00.305+0000 7f450da79700 1 -- 172.21.15.90:0/1917578779 wait complete. 2024-04-14T15:05:01.842 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:00.305+0000 7f450da79700 1 librados: shutdown 2024-04-14T15:05:01.842 INFO:teuthology.orchestra.run.smithi090.stdout:Waiting for the mgr to restart... 2024-04-14T15:05:01.842 INFO:teuthology.orchestra.run.smithi090.stdout:Waiting for mgr epoch 9... 2024-04-14T15:05:02.735 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:05:02 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:05:02.463+0000 7f48f5deb200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-14T15:05:02.735 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:05:02 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:05:02.594+0000 7f48f5deb200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-14T15:05:03.021 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:05:02 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:05:02.931+0000 7f48f5deb200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-14T15:05:03.336 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:05:03 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:05:03.021+0000 7f48f5deb200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-14T15:05:03.336 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:05:03 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:05:03.217+0000 7f48f5deb200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-14T15:05:03.691 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:05:03 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:05:03.421+0000 7f48f5deb200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-14T15:05:03.967 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:05:03 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:05:03.709+0000 7f48f5deb200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-14T15:05:04.441 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:05:03 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:05:03.966+0000 7f48f5deb200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-14T15:05:04.441 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:05:04 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:05:04.201+0000 7f48f5deb200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-14T15:05:05.327 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:05:05 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:05:05.000+0000 7f48f5deb200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-14T15:05:05.666 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:05:05 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:05:05.567+0000 7f48f5deb200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-14T15:05:05.667 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:05:05 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:05:05.666+0000 7f48f5deb200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-14T15:05:05.933 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:05:05 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:05:05.880+0000 7f48f5deb200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-14T15:05:06.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:05 smithi090 ceph-mon[30399]: Active manager daemon y restarted 2024-04-14T15:05:06.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:05 smithi090 ceph-mon[30399]: Activating manager daemon y 2024-04-14T15:05:06.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:05 smithi090 ceph-mon[30399]: osdmap e3: 0 total, 0 up, 0 in 2024-04-14T15:05:06.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:05 smithi090 ceph-mon[30399]: mgrmap e10: y(active, starting, since 0.00324569s) 2024-04-14T15:05:06.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:05 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T15:05:06.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:05 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-14T15:05:06.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:05 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-14T15:05:06.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:05 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-14T15:05:06.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:05 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-14T15:05:06.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:05 smithi090 ceph-mon[30399]: Manager daemon y is now available 2024-04-14T15:05:06.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:05 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:05:06.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:05 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T15:05:06.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:05 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T15:05:07.785 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout { 2024-04-14T15:05:07.785 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-14T15:05:07.785 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-14T15:05:07.785 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout } 2024-04-14T15:05:07.785 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:02.317+0000 7f89e21f5700 1 Processor -- start 2024-04-14T15:05:07.785 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:02.317+0000 7f89e21f5700 1 -- start start 2024-04-14T15:05:07.786 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:02.317+0000 7f89e21f5700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f89dc10b640 con 0x7f89dc105390 2024-04-14T15:05:07.786 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:02.317+0000 7f89db7fe700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f89dc105390 0x7f89dc1057b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:40992/0 (socket says 172.21.15.90:40992) 2024-04-14T15:05:07.786 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:02.317+0000 7f89db7fe700 1 -- 172.21.15.90:0/2285632091 learned_addr learned my addr 172.21.15.90:0/2285632091 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:05:07.786 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:02.318+0000 7f89da7fc700 1 -- 172.21.15.90:0/2285632091 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 639433826 0 0) 0x7f89dc10b640 con 0x7f89dc105390 2024-04-14T15:05:07.786 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:02.318+0000 7f89da7fc700 1 -- 172.21.15.90:0/2285632091 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89c40035b0 con 0x7f89dc105390 2024-04-14T15:05:07.786 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:02.318+0000 7f89da7fc700 1 -- 172.21.15.90:0/2285632091 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4150385463 0 0) 0x7f89c40035b0 con 0x7f89dc105390 2024-04-14T15:05:07.786 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:02.318+0000 7f89da7fc700 1 -- 172.21.15.90:0/2285632091 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89dc10c820 con 0x7f89dc105390 2024-04-14T15:05:07.786 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:02.318+0000 7f89da7fc700 1 -- 172.21.15.90:0/2285632091 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f89d0001580 con 0x7f89dc105390 2024-04-14T15:05:07.786 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:02.318+0000 7f89da7fc700 1 -- 172.21.15.90:0/2285632091 <== mon.0 v1:172.21.15.90:6789/0 4 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f89d0001c80 con 0x7f89dc105390 2024-04-14T15:05:07.786 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:02.318+0000 7f89da7fc700 1 -- 172.21.15.90:0/2285632091 <== mon.0 v1:172.21.15.90:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f89d0005960 con 0x7f89dc105390 2024-04-14T15:05:07.786 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:02.319+0000 7f89e21f5700 1 -- 172.21.15.90:0/2285632091 >> v1:172.21.15.90:6789/0 conn(0x7f89dc105390 legacy=0x7f89dc1057b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:05:07.786 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:02.319+0000 7f89e21f5700 1 -- 172.21.15.90:0/2285632091 shutdown_connections 2024-04-14T15:05:07.787 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:02.319+0000 7f89e21f5700 1 -- 172.21.15.90:0/2285632091 >> 172.21.15.90:0/2285632091 conn(0x7f89dc100cf0 msgr2=0x7f89dc103110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:05:07.787 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:02.319+0000 7f89e21f5700 1 -- 172.21.15.90:0/2285632091 shutdown_connections 2024-04-14T15:05:07.787 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:02.319+0000 7f89e21f5700 1 -- 172.21.15.90:0/2285632091 wait complete. 2024-04-14T15:05:07.787 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:02.319+0000 7f89e21f5700 1 librados: starting msgr at 2024-04-14T15:05:07.787 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:02.319+0000 7f89e21f5700 1 librados: starting objecter 2024-04-14T15:05:07.787 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:02.319+0000 7f89e21f5700 1 Processor -- start 2024-04-14T15:05:07.787 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:02.320+0000 7f89e21f5700 1 -- start start 2024-04-14T15:05:07.787 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:02.320+0000 7f89e21f5700 1 librados: setting wanted keys 2024-04-14T15:05:07.787 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:02.320+0000 7f89e21f5700 1 librados: calling monclient init 2024-04-14T15:05:07.787 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:02.320+0000 7f89e21f5700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f89dc1a0d70 con 0x7f89dc105390 2024-04-14T15:05:07.787 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:02.320+0000 7f89db7fe700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f89dc105390 0x7f89dc107c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:40996/0 (socket says 172.21.15.90:40996) 2024-04-14T15:05:07.787 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:02.320+0000 7f89db7fe700 1 -- 172.21.15.90:0/60455662 learned_addr learned my addr 172.21.15.90:0/60455662 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:05:07.787 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:02.320+0000 7f89d8ff9700 10 client.?.objecter ms_handle_connect 0x7f89dc105390 2024-04-14T15:05:07.788 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:02.320+0000 7f89d8ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:05:07.788 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:02.320+0000 7f89d8ff9700 1 -- 172.21.15.90:0/60455662 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1270519927 0 0) 0x7f89dc1a0d70 con 0x7f89dc105390 2024-04-14T15:05:07.788 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:02.320+0000 7f89d8ff9700 1 -- 172.21.15.90:0/60455662 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89c40035b0 con 0x7f89dc105390 2024-04-14T15:05:07.788 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:02.320+0000 7f89d8ff9700 1 -- 172.21.15.90:0/60455662 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 89920431 0 0) 0x7f89c40035b0 con 0x7f89dc105390 2024-04-14T15:05:07.788 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:02.320+0000 7f89d8ff9700 1 -- 172.21.15.90:0/60455662 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f89dc1a0d70 con 0x7f89dc105390 2024-04-14T15:05:07.788 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:02.320+0000 7f89d8ff9700 1 -- 172.21.15.90:0/60455662 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f89d00017a0 con 0x7f89dc105390 2024-04-14T15:05:07.788 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:02.321+0000 7f89d8ff9700 1 -- 172.21.15.90:0/60455662 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2182096878 0 0) 0x7f89dc1a0d70 con 0x7f89dc105390 2024-04-14T15:05:07.788 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:02.321+0000 7f89d8ff9700 1 -- 172.21.15.90:0/60455662 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89dc1a1f50 con 0x7f89dc105390 2024-04-14T15:05:07.788 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:02.321+0000 7f89e21f5700 1 -- 172.21.15.90:0/60455662 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f89dc1a0fa0 con 0x7f89dc105390 2024-04-14T15:05:07.788 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:02.321+0000 7f89e21f5700 10 client.14152.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:05:07.788 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:02.321+0000 7f89e21f5700 1 -- 172.21.15.90:0/60455662 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f89dc1a14e0 con 0x7f89dc105390 2024-04-14T15:05:07.788 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:02.321+0000 7f89e21f5700 1 librados: init done 2024-04-14T15:05:07.789 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:02.321+0000 7f89d8ff9700 1 -- 172.21.15.90:0/60455662 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f89d00011c0 con 0x7f89dc105390 2024-04-14T15:05:07.789 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:02.321+0000 7f89d8ff9700 1 -- 172.21.15.90:0/60455662 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f89d0006270 con 0x7f89dc105390 2024-04-14T15:05:07.789 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:02.322+0000 7f89d8ff9700 1 -- 172.21.15.90:0/60455662 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 9) ==== 46536+0+0 (unknown 2298455986 0 0) 0x7f89d0011a00 con 0x7f89dc105390 2024-04-14T15:05:07.789 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:02.322+0000 7f89d8ff9700 1 -- 172.21.15.90:0/60455662 --> v1:172.21.15.90:6800/941091171 -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f89c403ff10 con 0x7f89c403c240 2024-04-14T15:05:07.789 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:02.322+0000 7f89d8ff9700 1 -- 172.21.15.90:0/60455662 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (unknown 1250798154 0 0) 0x7f89d00481c0 con 0x7f89dc105390 2024-04-14T15:05:07.789 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:02.322+0000 7f89d8ff9700 10 client.14152.objecter ms_dispatch 0x7f89dc001550 osd_map(2..2 src has 1..2) 2024-04-14T15:05:07.789 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:02.322+0000 7f89d8ff9700 3 client.14152.objecter handle_osd_map got epochs [2,2] > 0 2024-04-14T15:05:07.789 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:02.322+0000 7f89d8ff9700 3 client.14152.objecter handle_osd_map decoding full epoch 2 2024-04-14T15:05:07.789 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:02.322+0000 7f89daffd700 1 -- 172.21.15.90:0/60455662 >> v1:172.21.15.90:6800/941091171 conn(0x7f89c403c240 legacy=0x7f89c403e700 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.90:6800/941091171 2024-04-14T15:05:07.789 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:02.322+0000 7f89d8ff9700 20 client.14152.objecter dump_active .. 0 homeless 2024-04-14T15:05:07.789 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:02.523+0000 7f89daffd700 1 -- 172.21.15.90:0/60455662 >> v1:172.21.15.90:6800/941091171 conn(0x7f89c403c240 legacy=0x7f89c403e700 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.90:6800/941091171 2024-04-14T15:05:07.789 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:02.924+0000 7f89daffd700 1 -- 172.21.15.90:0/60455662 >> v1:172.21.15.90:6800/941091171 conn(0x7f89c403c240 legacy=0x7f89c403e700 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.90:6800/941091171 2024-04-14T15:05:07.790 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:03.725+0000 7f89daffd700 1 -- 172.21.15.90:0/60455662 >> v1:172.21.15.90:6800/941091171 conn(0x7f89c403c240 legacy=0x7f89c403e700 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.90:6800/941091171 2024-04-14T15:05:07.790 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:05.326+0000 7f89daffd700 1 -- 172.21.15.90:0/60455662 >> v1:172.21.15.90:6800/941091171 conn(0x7f89c403c240 legacy=0x7f89c403e700 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.90:6800/941091171 2024-04-14T15:05:07.790 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:05.883+0000 7f89d8ff9700 1 -- 172.21.15.90:0/60455662 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mgrmap(e 10) ==== 46338+0+0 (unknown 566446654 0 0) 0x7f89d0012fc0 con 0x7f89dc105390 2024-04-14T15:05:07.790 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:05.883+0000 7f89d8ff9700 1 -- 172.21.15.90:0/60455662 >> v1:172.21.15.90:6800/941091171 conn(0x7f89c403c240 legacy=0x7f89c403e700 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-14T15:05:07.790 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:06.885+0000 7f89d8ff9700 1 -- 172.21.15.90:0/60455662 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mgrmap(e 11) ==== 46430+0+0 (unknown 3765425799 0 0) 0x7f89d0049140 con 0x7f89dc105390 2024-04-14T15:05:07.790 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:06.885+0000 7f89d8ff9700 1 -- 172.21.15.90:0/60455662 --> v1:172.21.15.90:6800/491071118 -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f89c403ff10 con 0x7f89c4040da0 2024-04-14T15:05:07.790 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:06.886+0000 7f89d8ff9700 10 client.14152.objecter ms_handle_connect 0x7f89c4040da0 2024-04-14T15:05:07.790 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:06.886+0000 7f89d8ff9700 1 -- 172.21.15.90:0/60455662 <== mgr.14148 v1:172.21.15.90:6800/491071118 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (unknown 0 0 4109177037) 0x7f89c403ff10 con 0x7f89c4040da0 2024-04-14T15:05:07.790 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:06.893+0000 7f89e21f5700 1 -- 172.21.15.90:0/60455662 --> v1:172.21.15.90:6800/491071118 -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f89c8002630 con 0x7f89c4040da0 2024-04-14T15:05:07.790 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:06.893+0000 7f89d8ff9700 1 -- 172.21.15.90:0/60455662 <== mgr.14148 v1:172.21.15.90:6800/491071118 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (unknown 0 0 3086460295) 0x7f89c8002630 con 0x7f89c4040da0 2024-04-14T15:05:07.790 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:06.893+0000 7f89e21f5700 10 librados: watch_flush enter 2024-04-14T15:05:07.791 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:06.893+0000 7f89e21f5700 10 librados: watch_flush exit 2024-04-14T15:05:07.791 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:06.893+0000 7f89e21f5700 20 client.14152.objecter shutdown clearing up homeless session... 2024-04-14T15:05:07.791 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:06.893+0000 7f89e21f5700 10 client.14152.objecter successfully canceled tick 2024-04-14T15:05:07.791 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:06.893+0000 7f89e21f5700 1 -- 172.21.15.90:0/60455662 >> v1:172.21.15.90:6800/491071118 conn(0x7f89c4040da0 legacy=0x7f89c4043190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:05:07.791 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:06.893+0000 7f89e21f5700 1 -- 172.21.15.90:0/60455662 >> v1:172.21.15.90:6789/0 conn(0x7f89dc105390 legacy=0x7f89dc107c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:05:07.791 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:06.893+0000 7f89e21f5700 1 -- 172.21.15.90:0/60455662 shutdown_connections 2024-04-14T15:05:07.791 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:06.893+0000 7f89e21f5700 1 -- 172.21.15.90:0/60455662 >> 172.21.15.90:0/60455662 conn(0x7f89dc100cf0 msgr2=0x7f89dc191980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:05:07.791 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:06.894+0000 7f89e21f5700 1 -- 172.21.15.90:0/60455662 shutdown_connections 2024-04-14T15:05:07.791 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:06.894+0000 7f89e21f5700 1 -- 172.21.15.90:0/60455662 wait complete. 2024-04-14T15:05:07.791 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:06.894+0000 7f89e21f5700 1 librados: shutdown 2024-04-14T15:05:07.791 INFO:teuthology.orchestra.run.smithi090.stdout:mgr epoch 9 is available 2024-04-14T15:05:07.791 INFO:teuthology.orchestra.run.smithi090.stdout:Generating a dashboard self-signed certificate... 2024-04-14T15:05:08.153 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:07 smithi090 ceph-mon[30399]: [14/Apr/2024:15:05:06] ENGINE Bus STARTING 2024-04-14T15:05:08.153 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:07 smithi090 ceph-mon[30399]: [14/Apr/2024:15:05:06] ENGINE Serving on http://172.21.15.90:8765 2024-04-14T15:05:08.153 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:07 smithi090 ceph-mon[30399]: [14/Apr/2024:15:05:06] ENGINE Serving on https://172.21.15.90:7150 2024-04-14T15:05:08.153 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:07 smithi090 ceph-mon[30399]: [14/Apr/2024:15:05:06] ENGINE Bus STARTED 2024-04-14T15:05:08.153 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:07 smithi090 ceph-mon[30399]: from='client.14152 v1:172.21.15.90:0/60455662' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-14T15:05:08.153 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:07 smithi090 ceph-mon[30399]: mgrmap e11: y(active, since 1.00647s) 2024-04-14T15:05:08.153 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:07 smithi090 ceph-mon[30399]: from='client.14152 v1:172.21.15.90:0/60455662' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-14T15:05:08.154 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:07 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:09.133 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-14T15:05:09.134 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:08.255+0000 7f07cd0ed700 1 Processor -- start 2024-04-14T15:05:09.134 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:08.256+0000 7f07cd0ed700 1 -- start start 2024-04-14T15:05:09.134 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:08.256+0000 7f07cd0ed700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f07c8075c50 con 0x7f07c8107b10 2024-04-14T15:05:09.134 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:08.256+0000 7f07c6d9d700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f07c8107b10 0x7f07c8109f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:41076/0 (socket says 172.21.15.90:41076) 2024-04-14T15:05:09.134 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:08.256+0000 7f07c6d9d700 1 -- 172.21.15.90:0/1196800474 learned_addr learned my addr 172.21.15.90:0/1196800474 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:05:09.134 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:08.256+0000 7f07c5d9b700 1 -- 172.21.15.90:0/1196800474 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3892903818 0 0) 0x7f07c8075c50 con 0x7f07c8107b10 2024-04-14T15:05:09.134 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:08.256+0000 7f07c5d9b700 1 -- 172.21.15.90:0/1196800474 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f07b00035b0 con 0x7f07c8107b10 2024-04-14T15:05:09.134 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:08.256+0000 7f07c5d9b700 1 -- 172.21.15.90:0/1196800474 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 294459330 0 0) 0x7f07b00035b0 con 0x7f07c8107b10 2024-04-14T15:05:09.134 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:08.256+0000 7f07c5d9b700 1 -- 172.21.15.90:0/1196800474 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07c810c630 con 0x7f07c8107b10 2024-04-14T15:05:09.134 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:08.256+0000 7f07c5d9b700 1 -- 172.21.15.90:0/1196800474 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f07b8001580 con 0x7f07c8107b10 2024-04-14T15:05:09.134 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:08.257+0000 7f07c5d9b700 1 -- 172.21.15.90:0/1196800474 <== mon.0 v1:172.21.15.90:6789/0 4 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f07b8001c80 con 0x7f07c8107b10 2024-04-14T15:05:09.134 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:08.257+0000 7f07c5d9b700 1 -- 172.21.15.90:0/1196800474 <== mon.0 v1:172.21.15.90:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f07b8005960 con 0x7f07c8107b10 2024-04-14T15:05:09.134 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:08.257+0000 7f07cd0ed700 1 -- 172.21.15.90:0/1196800474 >> v1:172.21.15.90:6789/0 conn(0x7f07c8107b10 legacy=0x7f07c8109f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:05:09.135 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:08.257+0000 7f07cd0ed700 1 -- 172.21.15.90:0/1196800474 shutdown_connections 2024-04-14T15:05:09.135 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:08.257+0000 7f07cd0ed700 1 -- 172.21.15.90:0/1196800474 >> 172.21.15.90:0/1196800474 conn(0x7f07c8101500 msgr2=0x7f07c8103960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:05:09.135 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:08.257+0000 7f07cd0ed700 1 -- 172.21.15.90:0/1196800474 shutdown_connections 2024-04-14T15:05:09.135 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:08.257+0000 7f07cd0ed700 1 -- 172.21.15.90:0/1196800474 wait complete. 2024-04-14T15:05:09.135 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:08.258+0000 7f07cd0ed700 1 librados: starting msgr at 2024-04-14T15:05:09.135 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:08.258+0000 7f07cd0ed700 1 librados: starting objecter 2024-04-14T15:05:09.135 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:08.258+0000 7f07cd0ed700 1 Processor -- start 2024-04-14T15:05:09.135 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:08.258+0000 7f07cd0ed700 1 -- start start 2024-04-14T15:05:09.135 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:08.258+0000 7f07cd0ed700 1 librados: setting wanted keys 2024-04-14T15:05:09.135 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:08.258+0000 7f07cd0ed700 1 librados: calling monclient init 2024-04-14T15:05:09.135 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:08.258+0000 7f07cd0ed700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f07c819b420 con 0x7f07c8107b10 2024-04-14T15:05:09.135 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:08.259+0000 7f07c6d9d700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f07c8107b10 0x7f07c819ad10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:41084/0 (socket says 172.21.15.90:41084) 2024-04-14T15:05:09.135 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:08.259+0000 7f07c6d9d700 1 -- 172.21.15.90:0/1058972561 learned_addr learned my addr 172.21.15.90:0/1058972561 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:05:09.135 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:08.259+0000 7f07affff700 10 client.?.objecter ms_handle_connect 0x7f07c8107b10 2024-04-14T15:05:09.135 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:08.259+0000 7f07affff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:05:09.136 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:08.259+0000 7f07affff700 1 -- 172.21.15.90:0/1058972561 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4164625541 0 0) 0x7f07c819b420 con 0x7f07c8107b10 2024-04-14T15:05:09.136 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:08.259+0000 7f07affff700 1 -- 172.21.15.90:0/1058972561 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f07b00035b0 con 0x7f07c8107b10 2024-04-14T15:05:09.136 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:08.259+0000 7f07affff700 1 -- 172.21.15.90:0/1058972561 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 294206316 0 0) 0x7f07b00035b0 con 0x7f07c8107b10 2024-04-14T15:05:09.136 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:08.259+0000 7f07affff700 1 -- 172.21.15.90:0/1058972561 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f07c819b420 con 0x7f07c8107b10 2024-04-14T15:05:09.136 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:08.259+0000 7f07affff700 1 -- 172.21.15.90:0/1058972561 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f07b8001080 con 0x7f07c8107b10 2024-04-14T15:05:09.136 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:08.259+0000 7f07affff700 1 -- 172.21.15.90:0/1058972561 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 701595910 0 0) 0x7f07c819b420 con 0x7f07c8107b10 2024-04-14T15:05:09.136 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:08.259+0000 7f07affff700 1 -- 172.21.15.90:0/1058972561 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07c819b5f0 con 0x7f07c8107b10 2024-04-14T15:05:09.136 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:08.259+0000 7f07cd0ed700 1 -- 172.21.15.90:0/1058972561 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f07c819b8e0 con 0x7f07c8107b10 2024-04-14T15:05:09.136 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:08.259+0000 7f07cd0ed700 10 client.14160.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:05:09.136 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:08.259+0000 7f07cd0ed700 1 -- 172.21.15.90:0/1058972561 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f07c819f420 con 0x7f07c8107b10 2024-04-14T15:05:09.136 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:08.259+0000 7f07cd0ed700 1 librados: init done 2024-04-14T15:05:09.136 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:08.259+0000 7f07affff700 1 -- 172.21.15.90:0/1058972561 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f07b80017f0 con 0x7f07c8107b10 2024-04-14T15:05:09.136 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:08.259+0000 7f07affff700 1 -- 172.21.15.90:0/1058972561 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f07b80060d0 con 0x7f07c8107b10 2024-04-14T15:05:09.136 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:08.260+0000 7f07affff700 1 -- 172.21.15.90:0/1058972561 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 11) ==== 46430+0+0 (unknown 3765425799 0 0) 0x7f07b8011800 con 0x7f07c8107b10 2024-04-14T15:05:09.136 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:08.260+0000 7f07affff700 1 -- 172.21.15.90:0/1058972561 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(3..3 src has 1..3) ==== 1069+0+0 (unknown 1996007131 0 0) 0x7f07b8048ce0 con 0x7f07c8107b10 2024-04-14T15:05:09.137 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:08.260+0000 7f07affff700 10 client.14160.objecter ms_dispatch 0x7f07c8001550 osd_map(3..3 src has 1..3) 2024-04-14T15:05:09.137 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:08.260+0000 7f07affff700 3 client.14160.objecter handle_osd_map got epochs [3,3] > 0 2024-04-14T15:05:09.137 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:08.260+0000 7f07affff700 3 client.14160.objecter handle_osd_map decoding full epoch 3 2024-04-14T15:05:09.137 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:08.260+0000 7f07affff700 20 client.14160.objecter dump_active .. 0 homeless 2024-04-14T15:05:09.137 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:08.260+0000 7f07cd0ed700 1 -- 172.21.15.90:0/1058972561 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07b4005120 con 0x7f07c8107b10 2024-04-14T15:05:09.137 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:08.261+0000 7f07affff700 10 client.14160.objecter ms_handle_connect 0x7f07b003c330 2024-04-14T15:05:09.137 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:08.265+0000 7f07affff700 1 -- 172.21.15.90:0/1058972561 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f07b8017780 con 0x7f07c8107b10 2024-04-14T15:05:09.137 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:08.294+0000 7f07affff700 1 -- 172.21.15.90:0/1058972561 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mgrmap(e 12) ==== 46536+0+0 (unknown 4098975754 0 0) 0x7f07b8012dc0 con 0x7f07c8107b10 2024-04-14T15:05:09.137 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:08.493+0000 7f07cd0ed700 1 -- 172.21.15.90:0/1058972561 --> v1:172.21.15.90:6800/491071118 -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f07b4000b90 con 0x7f07b003c330 2024-04-14T15:05:09.137 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:08.590+0000 7f07affff700 1 -- 172.21.15.90:0/1058972561 <== mgr.14148 v1:172.21.15.90:6800/491071118 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (unknown 0 0 3317752739) 0x7f07b4000b90 con 0x7f07b003c330 2024-04-14T15:05:09.137 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:08.592+0000 7f07cd0ed700 10 librados: watch_flush enter 2024-04-14T15:05:09.137 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:08.592+0000 7f07cd0ed700 10 librados: watch_flush exit 2024-04-14T15:05:09.137 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:08.592+0000 7f07cd0ed700 20 client.14160.objecter shutdown clearing up homeless session... 2024-04-14T15:05:09.137 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:08.592+0000 7f07cd0ed700 10 client.14160.objecter successfully canceled tick 2024-04-14T15:05:09.137 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:08.592+0000 7f07cd0ed700 1 -- 172.21.15.90:0/1058972561 >> v1:172.21.15.90:6800/491071118 conn(0x7f07b003c330 legacy=0x7f07b003e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:05:09.138 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:08.592+0000 7f07cd0ed700 1 -- 172.21.15.90:0/1058972561 >> v1:172.21.15.90:6789/0 conn(0x7f07c8107b10 legacy=0x7f07c819ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:05:09.138 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:08.592+0000 7f07cd0ed700 1 -- 172.21.15.90:0/1058972561 shutdown_connections 2024-04-14T15:05:09.138 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:08.592+0000 7f07cd0ed700 1 -- 172.21.15.90:0/1058972561 >> 172.21.15.90:0/1058972561 conn(0x7f07c8101500 msgr2=0x7f07c8191940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:05:09.138 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:08.593+0000 7f07cd0ed700 1 -- 172.21.15.90:0/1058972561 shutdown_connections 2024-04-14T15:05:09.138 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:08.593+0000 7f07cd0ed700 1 -- 172.21.15.90:0/1058972561 wait complete. 2024-04-14T15:05:09.138 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:08.593+0000 7f07cd0ed700 1 librados: shutdown 2024-04-14T15:05:09.138 INFO:teuthology.orchestra.run.smithi090.stdout:Creating initial admin user... 2024-04-14T15:05:09.512 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:09 smithi090 ceph-mon[30399]: mgrmap e12: y(active, since 2s) 2024-04-14T15:05:09.512 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:09 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:09.512 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:09 smithi090 ceph-mon[30399]: from='client.14160 v1:172.21.15.90:0/1058972561' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:05:09.512 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:09 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:09.512 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:09 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:10.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:10 smithi090 ceph-mon[30399]: from='client.14162 v1:172.21.15.90:0/1321247698' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:05:10.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:10 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:11.344 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$KRfY6Kdma31tD5YMHdtT4e4/t/W4Klc8ItA/GwFEiAzg83Ddeibha", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1713107110, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-14T15:05:11.344 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:09.619+0000 7fc6c139f700 1 Processor -- start 2024-04-14T15:05:11.344 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:09.619+0000 7fc6c139f700 1 -- start start 2024-04-14T15:05:11.344 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:09.620+0000 7fc6c139f700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc6bc10a1f0 con 0x7fc6bc105dc0 2024-04-14T15:05:11.344 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:09.620+0000 7fc6bbfff700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fc6bc105dc0 0x7fc6bc1061e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:41092/0 (socket says 172.21.15.90:41092) 2024-04-14T15:05:11.344 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:09.620+0000 7fc6bbfff700 1 -- 172.21.15.90:0/317691043 learned_addr learned my addr 172.21.15.90:0/317691043 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:05:11.344 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:09.620+0000 7fc6baffd700 1 -- 172.21.15.90:0/317691043 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2452341225 0 0) 0x7fc6bc10a1f0 con 0x7fc6bc105dc0 2024-04-14T15:05:11.344 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:09.620+0000 7fc6baffd700 1 -- 172.21.15.90:0/317691043 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc6a40035b0 con 0x7fc6bc105dc0 2024-04-14T15:05:11.344 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:09.620+0000 7fc6baffd700 1 -- 172.21.15.90:0/317691043 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4093991401 0 0) 0x7fc6a40035b0 con 0x7fc6bc105dc0 2024-04-14T15:05:11.344 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:09.620+0000 7fc6baffd700 1 -- 172.21.15.90:0/317691043 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6bc10b3d0 con 0x7fc6bc105dc0 2024-04-14T15:05:11.344 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:09.620+0000 7fc6baffd700 1 -- 172.21.15.90:0/317691043 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7fc6ac001580 con 0x7fc6bc105dc0 2024-04-14T15:05:11.345 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:09.620+0000 7fc6baffd700 1 -- 172.21.15.90:0/317691043 <== mon.0 v1:172.21.15.90:6789/0 4 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fc6ac004930 con 0x7fc6bc105dc0 2024-04-14T15:05:11.345 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:09.621+0000 7fc6baffd700 1 -- 172.21.15.90:0/317691043 <== mon.0 v1:172.21.15.90:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7fc6ac005960 con 0x7fc6bc105dc0 2024-04-14T15:05:11.345 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:09.621+0000 7fc6c139f700 1 -- 172.21.15.90:0/317691043 >> v1:172.21.15.90:6789/0 conn(0x7fc6bc105dc0 legacy=0x7fc6bc1061e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:05:11.345 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:09.621+0000 7fc6c139f700 1 -- 172.21.15.90:0/317691043 shutdown_connections 2024-04-14T15:05:11.345 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:09.621+0000 7fc6c139f700 1 -- 172.21.15.90:0/317691043 >> 172.21.15.90:0/317691043 conn(0x7fc6bc101320 msgr2=0x7fc6bc1037a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:05:11.345 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:09.621+0000 7fc6c139f700 1 -- 172.21.15.90:0/317691043 shutdown_connections 2024-04-14T15:05:11.345 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:09.621+0000 7fc6c139f700 1 -- 172.21.15.90:0/317691043 wait complete. 2024-04-14T15:05:11.345 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:09.622+0000 7fc6c139f700 1 librados: starting msgr at 2024-04-14T15:05:11.345 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:09.622+0000 7fc6c139f700 1 librados: starting objecter 2024-04-14T15:05:11.345 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:09.622+0000 7fc6c139f700 1 Processor -- start 2024-04-14T15:05:11.345 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:09.622+0000 7fc6c139f700 1 -- start start 2024-04-14T15:05:11.345 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:09.622+0000 7fc6c139f700 1 librados: setting wanted keys 2024-04-14T15:05:11.345 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:09.622+0000 7fc6c139f700 1 librados: calling monclient init 2024-04-14T15:05:11.345 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:09.622+0000 7fc6c139f700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc6bc19b230 con 0x7fc6bc105dc0 2024-04-14T15:05:11.346 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:09.623+0000 7fc6bbfff700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fc6bc105dc0 0x7fc6bc19ab20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:41106/0 (socket says 172.21.15.90:41106) 2024-04-14T15:05:11.346 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:09.623+0000 7fc6bbfff700 1 -- 172.21.15.90:0/1321247698 learned_addr learned my addr 172.21.15.90:0/1321247698 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:05:11.346 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:09.623+0000 7fc6b97fa700 10 client.?.objecter ms_handle_connect 0x7fc6bc105dc0 2024-04-14T15:05:11.346 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:09.623+0000 7fc6b97fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:05:11.346 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:09.623+0000 7fc6b97fa700 1 -- 172.21.15.90:0/1321247698 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1833550898 0 0) 0x7fc6bc19b230 con 0x7fc6bc105dc0 2024-04-14T15:05:11.346 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:09.623+0000 7fc6b97fa700 1 -- 172.21.15.90:0/1321247698 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc6a40056e0 con 0x7fc6bc105dc0 2024-04-14T15:05:11.346 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:09.623+0000 7fc6b97fa700 1 -- 172.21.15.90:0/1321247698 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 225645565 0 0) 0x7fc6bc10b3d0 con 0x7fc6bc105dc0 2024-04-14T15:05:11.346 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:09.623+0000 7fc6b97fa700 1 -- 172.21.15.90:0/1321247698 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc6bc19b230 con 0x7fc6bc105dc0 2024-04-14T15:05:11.346 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:09.623+0000 7fc6b97fa700 1 -- 172.21.15.90:0/1321247698 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7fc6ac001080 con 0x7fc6bc105dc0 2024-04-14T15:05:11.346 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:09.623+0000 7fc6b97fa700 1 -- 172.21.15.90:0/1321247698 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1736693148 0 0) 0x7fc6bc19b230 con 0x7fc6bc105dc0 2024-04-14T15:05:11.346 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:09.623+0000 7fc6b97fa700 1 -- 172.21.15.90:0/1321247698 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6bc19b400 con 0x7fc6bc105dc0 2024-04-14T15:05:11.346 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:09.623+0000 7fc6c139f700 1 -- 172.21.15.90:0/1321247698 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc6bc19b6f0 con 0x7fc6bc105dc0 2024-04-14T15:05:11.347 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:09.623+0000 7fc6c139f700 10 client.14162.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:05:11.347 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:09.623+0000 7fc6c139f700 1 -- 172.21.15.90:0/1321247698 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc6bc19f240 con 0x7fc6bc105dc0 2024-04-14T15:05:11.347 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:09.623+0000 7fc6c139f700 1 librados: init done 2024-04-14T15:05:11.347 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:09.623+0000 7fc6b97fa700 1 -- 172.21.15.90:0/1321247698 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fc6ac004930 con 0x7fc6bc105dc0 2024-04-14T15:05:11.347 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:09.624+0000 7fc6b97fa700 1 -- 172.21.15.90:0/1321247698 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7fc6ac005f50 con 0x7fc6bc105dc0 2024-04-14T15:05:11.347 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:09.624+0000 7fc6b97fa700 1 -- 172.21.15.90:0/1321247698 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 12) ==== 46536+0+0 (unknown 4098975754 0 0) 0x7fc6ac0116e0 con 0x7fc6bc105dc0 2024-04-14T15:05:11.347 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:09.624+0000 7fc6c139f700 1 -- 172.21.15.90:0/1321247698 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6a8005150 con 0x7fc6bc105dc0 2024-04-14T15:05:11.347 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:09.625+0000 7fc6b97fa700 1 -- 172.21.15.90:0/1321247698 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(3..3 src has 1..3) ==== 1069+0+0 (unknown 1996007131 0 0) 0x7fc6ac048e30 con 0x7fc6bc105dc0 2024-04-14T15:05:11.347 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:09.625+0000 7fc6b97fa700 10 client.14162.objecter ms_dispatch 0x7fc6bc001550 osd_map(3..3 src has 1..3) 2024-04-14T15:05:11.347 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:09.625+0000 7fc6b97fa700 3 client.14162.objecter handle_osd_map got epochs [3,3] > 0 2024-04-14T15:05:11.347 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:09.625+0000 7fc6b97fa700 3 client.14162.objecter handle_osd_map decoding full epoch 3 2024-04-14T15:05:11.347 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:09.625+0000 7fc6b97fa700 20 client.14162.objecter dump_active .. 0 homeless 2024-04-14T15:05:11.347 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:09.625+0000 7fc6b97fa700 10 client.14162.objecter ms_handle_connect 0x7fc6a403c4c0 2024-04-14T15:05:11.347 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:09.630+0000 7fc6b97fa700 1 -- 172.21.15.90:0/1321247698 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fc6a40056e0 con 0x7fc6bc105dc0 2024-04-14T15:05:11.348 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:09.855+0000 7fc6c139f700 1 -- 172.21.15.90:0/1321247698 --> v1:172.21.15.90:6800/491071118 -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7fc6a8002430 con 0x7fc6a403c4c0 2024-04-14T15:05:11.348 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:10.080+0000 7fc6b97fa700 1 -- 172.21.15.90:0/1321247698 <== mgr.14148 v1:172.21.15.90:6800/491071118 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (unknown 0 0 1890303732) 0x7fc6a8002430 con 0x7fc6a403c4c0 2024-04-14T15:05:11.348 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:10.082+0000 7fc6c139f700 10 librados: watch_flush enter 2024-04-14T15:05:11.348 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:10.082+0000 7fc6c139f700 10 librados: watch_flush exit 2024-04-14T15:05:11.348 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:10.082+0000 7fc6c139f700 20 client.14162.objecter shutdown clearing up homeless session... 2024-04-14T15:05:11.348 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:10.082+0000 7fc6c139f700 10 client.14162.objecter successfully canceled tick 2024-04-14T15:05:11.348 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:10.082+0000 7fc6c139f700 1 -- 172.21.15.90:0/1321247698 >> v1:172.21.15.90:6800/491071118 conn(0x7fc6a403c4c0 legacy=0x7fc6a403e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:05:11.348 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:10.082+0000 7fc6c139f700 1 -- 172.21.15.90:0/1321247698 >> v1:172.21.15.90:6789/0 conn(0x7fc6bc105dc0 legacy=0x7fc6bc19ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:05:11.348 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:10.082+0000 7fc6c139f700 1 -- 172.21.15.90:0/1321247698 shutdown_connections 2024-04-14T15:05:11.348 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:10.082+0000 7fc6c139f700 1 -- 172.21.15.90:0/1321247698 >> 172.21.15.90:0/1321247698 conn(0x7fc6bc101320 msgr2=0x7fc6bc079fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:05:11.348 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:10.082+0000 7fc6c139f700 1 -- 172.21.15.90:0/1321247698 shutdown_connections 2024-04-14T15:05:11.348 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:10.082+0000 7fc6c139f700 1 -- 172.21.15.90:0/1321247698 wait complete. 2024-04-14T15:05:11.348 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:10.082+0000 7fc6c139f700 1 librados: shutdown 2024-04-14T15:05:11.348 INFO:teuthology.orchestra.run.smithi090.stdout:Fetching dashboard port number... 2024-04-14T15:05:12.570 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 8443 2024-04-14T15:05:12.570 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:11.789+0000 7fcc740f0700 1 Processor -- start 2024-04-14T15:05:12.570 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:11.790+0000 7fcc740f0700 1 -- start start 2024-04-14T15:05:12.570 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:11.790+0000 7fcc740f0700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fcc6c07ba90 con 0x7fcc6c07ce90 2024-04-14T15:05:12.570 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:11.790+0000 7fcc71e8c700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fcc6c07ce90 0x7fcc6c07b2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50010/0 (socket says 172.21.15.90:50010) 2024-04-14T15:05:12.570 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:11.790+0000 7fcc71e8c700 1 -- 172.21.15.90:0/4067438164 learned_addr learned my addr 172.21.15.90:0/4067438164 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:05:12.570 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:11.790+0000 7fcc70e8a700 1 -- 172.21.15.90:0/4067438164 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3735136658 0 0) 0x7fcc6c07ba90 con 0x7fcc6c07ce90 2024-04-14T15:05:12.571 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:11.790+0000 7fcc70e8a700 1 -- 172.21.15.90:0/4067438164 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc580035b0 con 0x7fcc6c07ce90 2024-04-14T15:05:12.571 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:11.790+0000 7fcc70e8a700 1 -- 172.21.15.90:0/4067438164 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2529573132 0 0) 0x7fcc580035b0 con 0x7fcc6c07ce90 2024-04-14T15:05:12.571 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:11.790+0000 7fcc70e8a700 1 -- 172.21.15.90:0/4067438164 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc6c07bc60 con 0x7fcc6c07ce90 2024-04-14T15:05:12.571 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:11.790+0000 7fcc70e8a700 1 -- 172.21.15.90:0/4067438164 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7fcc5c001580 con 0x7fcc6c07ce90 2024-04-14T15:05:12.571 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:11.791+0000 7fcc70e8a700 1 -- 172.21.15.90:0/4067438164 <== mon.0 v1:172.21.15.90:6789/0 4 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fcc5c004930 con 0x7fcc6c07ce90 2024-04-14T15:05:12.571 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:11.791+0000 7fcc70e8a700 1 -- 172.21.15.90:0/4067438164 <== mon.0 v1:172.21.15.90:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7fcc5c005a70 con 0x7fcc6c07ce90 2024-04-14T15:05:12.571 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:11.791+0000 7fcc740f0700 1 -- 172.21.15.90:0/4067438164 >> v1:172.21.15.90:6789/0 conn(0x7fcc6c07ce90 legacy=0x7fcc6c07b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:05:12.571 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:11.791+0000 7fcc740f0700 1 -- 172.21.15.90:0/4067438164 shutdown_connections 2024-04-14T15:05:12.571 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:11.791+0000 7fcc740f0700 1 -- 172.21.15.90:0/4067438164 >> 172.21.15.90:0/4067438164 conn(0x7fcc6c102640 msgr2=0x7fcc6c104a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:05:12.571 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:11.791+0000 7fcc740f0700 1 -- 172.21.15.90:0/4067438164 shutdown_connections 2024-04-14T15:05:12.571 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:11.791+0000 7fcc740f0700 1 -- 172.21.15.90:0/4067438164 wait complete. 2024-04-14T15:05:12.571 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:11.792+0000 7fcc740f0700 1 librados: starting msgr at 2024-04-14T15:05:12.571 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:11.792+0000 7fcc740f0700 1 librados: starting objecter 2024-04-14T15:05:12.571 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:11.792+0000 7fcc740f0700 1 Processor -- start 2024-04-14T15:05:12.571 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:11.792+0000 7fcc740f0700 1 -- start start 2024-04-14T15:05:12.572 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:11.792+0000 7fcc740f0700 1 librados: setting wanted keys 2024-04-14T15:05:12.572 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:11.792+0000 7fcc740f0700 1 librados: calling monclient init 2024-04-14T15:05:12.572 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:11.792+0000 7fcc740f0700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fcc6c19d650 con 0x7fcc6c07ce90 2024-04-14T15:05:12.572 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:11.792+0000 7fcc71e8c700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fcc6c07ce90 0x7fcc6c19cf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50020/0 (socket says 172.21.15.90:50020) 2024-04-14T15:05:12.572 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:11.792+0000 7fcc71e8c700 1 -- 172.21.15.90:0/752329104 learned_addr learned my addr 172.21.15.90:0/752329104 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:05:12.572 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:11.792+0000 7fcc62ffd700 10 client.?.objecter ms_handle_connect 0x7fcc6c07ce90 2024-04-14T15:05:12.572 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:11.792+0000 7fcc62ffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:05:12.572 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:11.792+0000 7fcc62ffd700 1 -- 172.21.15.90:0/752329104 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3716264616 0 0) 0x7fcc6c19d650 con 0x7fcc6c07ce90 2024-04-14T15:05:12.572 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:11.792+0000 7fcc62ffd700 1 -- 172.21.15.90:0/752329104 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc580056e0 con 0x7fcc6c07ce90 2024-04-14T15:05:12.572 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:11.792+0000 7fcc62ffd700 1 -- 172.21.15.90:0/752329104 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1124879316 0 0) 0x7fcc6c07bc60 con 0x7fcc6c07ce90 2024-04-14T15:05:12.572 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:11.792+0000 7fcc62ffd700 1 -- 172.21.15.90:0/752329104 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcc6c19d650 con 0x7fcc6c07ce90 2024-04-14T15:05:12.572 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:11.792+0000 7fcc62ffd700 1 -- 172.21.15.90:0/752329104 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7fcc5c001080 con 0x7fcc6c07ce90 2024-04-14T15:05:12.572 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:11.793+0000 7fcc62ffd700 1 -- 172.21.15.90:0/752329104 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2638975687 0 0) 0x7fcc6c19d650 con 0x7fcc6c07ce90 2024-04-14T15:05:12.572 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:11.793+0000 7fcc62ffd700 1 -- 172.21.15.90:0/752329104 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc6c19d820 con 0x7fcc6c07ce90 2024-04-14T15:05:12.572 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:11.793+0000 7fcc740f0700 1 -- 172.21.15.90:0/752329104 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fcc6c19db10 con 0x7fcc6c07ce90 2024-04-14T15:05:12.573 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:11.793+0000 7fcc740f0700 10 client.14164.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:05:12.573 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:11.793+0000 7fcc740f0700 1 -- 172.21.15.90:0/752329104 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fcc6c1a1660 con 0x7fcc6c07ce90 2024-04-14T15:05:12.573 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:11.793+0000 7fcc740f0700 1 librados: init done 2024-04-14T15:05:12.573 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:11.793+0000 7fcc62ffd700 1 -- 172.21.15.90:0/752329104 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fcc5c004930 con 0x7fcc6c07ce90 2024-04-14T15:05:12.573 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:11.793+0000 7fcc62ffd700 1 -- 172.21.15.90:0/752329104 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7fcc5c006090 con 0x7fcc6c07ce90 2024-04-14T15:05:12.573 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:11.794+0000 7fcc62ffd700 1 -- 172.21.15.90:0/752329104 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 12) ==== 46536+0+0 (unknown 4098975754 0 0) 0x7fcc5c011820 con 0x7fcc6c07ce90 2024-04-14T15:05:12.573 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:11.794+0000 7fcc740f0700 1 -- 172.21.15.90:0/752329104 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc50005150 con 0x7fcc6c07ce90 2024-04-14T15:05:12.573 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:11.794+0000 7fcc62ffd700 1 -- 172.21.15.90:0/752329104 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(3..3 src has 1..3) ==== 1069+0+0 (unknown 1996007131 0 0) 0x7fcc5c048020 con 0x7fcc6c07ce90 2024-04-14T15:05:12.573 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:11.794+0000 7fcc62ffd700 10 client.14164.objecter ms_dispatch 0x7fcc6c001550 osd_map(3..3 src has 1..3) 2024-04-14T15:05:12.573 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:11.794+0000 7fcc62ffd700 3 client.14164.objecter handle_osd_map got epochs [3,3] > 0 2024-04-14T15:05:12.573 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:11.794+0000 7fcc62ffd700 3 client.14164.objecter handle_osd_map decoding full epoch 3 2024-04-14T15:05:12.573 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:11.794+0000 7fcc62ffd700 20 client.14164.objecter dump_active .. 0 homeless 2024-04-14T15:05:12.573 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:11.794+0000 7fcc62ffd700 10 client.14164.objecter ms_handle_connect 0x7fcc5803c460 2024-04-14T15:05:12.573 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:11.799+0000 7fcc62ffd700 1 -- 172.21.15.90:0/752329104 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fcc580056e0 con 0x7fcc6c07ce90 2024-04-14T15:05:12.573 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:11.935+0000 7fcc62ffd700 1 -- 172.21.15.90:0/752329104 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mgrmap(e 13) ==== 46579+0+0 (unknown 2972057686 0 0) 0x7fcc5c046920 con 0x7fcc6c07ce90 2024-04-14T15:05:12.574 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:12.000+0000 7fcc740f0700 1 -- 172.21.15.90:0/752329104 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7fcc50005440 con 0x7fcc6c07ce90 2024-04-14T15:05:12.574 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:12.001+0000 7fcc62ffd700 1 -- 172.21.15.90:0/752329104 <== mon.0 v1:172.21.15.90:6789/0 11 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (unknown 3713421687 0 83753974) 0x7fcc5c0177a0 con 0x7fcc6c07ce90 2024-04-14T15:05:12.574 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:12.003+0000 7fcc740f0700 10 librados: watch_flush enter 2024-04-14T15:05:12.574 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:12.003+0000 7fcc740f0700 10 librados: watch_flush exit 2024-04-14T15:05:12.574 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:12.003+0000 7fcc740f0700 20 client.14164.objecter shutdown clearing up homeless session... 2024-04-14T15:05:12.574 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:12.003+0000 7fcc740f0700 10 client.14164.objecter successfully canceled tick 2024-04-14T15:05:12.574 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:12.003+0000 7fcc740f0700 1 -- 172.21.15.90:0/752329104 >> v1:172.21.15.90:6800/491071118 conn(0x7fcc5803c460 legacy=0x7fcc5803e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:05:12.574 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:12.003+0000 7fcc740f0700 1 -- 172.21.15.90:0/752329104 >> v1:172.21.15.90:6789/0 conn(0x7fcc6c07ce90 legacy=0x7fcc6c19cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:05:12.574 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:12.003+0000 7fcc740f0700 1 -- 172.21.15.90:0/752329104 shutdown_connections 2024-04-14T15:05:12.574 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:12.003+0000 7fcc740f0700 1 -- 172.21.15.90:0/752329104 >> 172.21.15.90:0/752329104 conn(0x7fcc6c102640 msgr2=0x7fcc6c193b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:05:12.574 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:12.003+0000 7fcc740f0700 1 -- 172.21.15.90:0/752329104 shutdown_connections 2024-04-14T15:05:12.574 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:12.003+0000 7fcc740f0700 1 -- 172.21.15.90:0/752329104 wait complete. 2024-04-14T15:05:12.574 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:12.003+0000 7fcc740f0700 1 librados: shutdown 2024-04-14T15:05:12.576 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-14T15:05:12.576 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout disabled 2024-04-14T15:05:12.583 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-14T15:05:12.583 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout inactive 2024-04-14T15:05:12.583 INFO:teuthology.orchestra.run.smithi090.stdout:firewalld.service is not enabled 2024-04-14T15:05:12.583 INFO:teuthology.orchestra.run.smithi090.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-14T15:05:12.585 INFO:teuthology.orchestra.run.smithi090.stdout:Ceph Dashboard is now available at: 2024-04-14T15:05:12.585 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:05:12.585 INFO:teuthology.orchestra.run.smithi090.stdout: URL: https://smithi090.front.sepia.ceph.com:8443/ 2024-04-14T15:05:12.585 INFO:teuthology.orchestra.run.smithi090.stdout: User: admin 2024-04-14T15:05:12.585 INFO:teuthology.orchestra.run.smithi090.stdout: Password: oe0rts8qes 2024-04-14T15:05:12.585 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:05:12.585 INFO:teuthology.orchestra.run.smithi090.stdout:Saving cluster configuration to /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/config directory 2024-04-14T15:05:12.586 INFO:teuthology.orchestra.run.smithi090.stdout:Enabling autotune for osd_memory_target 2024-04-14T15:05:13.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:12 smithi090 ceph-mon[30399]: mgrmap e13: y(active, since 6s) 2024-04-14T15:05:13.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:12 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/752329104' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-14T15:05:13.795 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:13.025+0000 7f5715960700 1 Processor -- start 2024-04-14T15:05:13.796 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:13.025+0000 7f5715960700 1 -- start start 2024-04-14T15:05:13.796 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:13.025+0000 7f5715960700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f571010b4c0 con 0x7f5710107090 2024-04-14T15:05:13.796 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:13.025+0000 7f570effd700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f5710107090 0x7f57101074b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50028/0 (socket says 172.21.15.90:50028) 2024-04-14T15:05:13.796 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:13.025+0000 7f570effd700 1 -- 172.21.15.90:0/1774827086 learned_addr learned my addr 172.21.15.90:0/1774827086 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:05:13.796 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:13.025+0000 7f570dffb700 1 -- 172.21.15.90:0/1774827086 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2792227851 0 0) 0x7f571010b4c0 con 0x7f5710107090 2024-04-14T15:05:13.796 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:13.025+0000 7f570dffb700 1 -- 172.21.15.90:0/1774827086 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f56fc0035b0 con 0x7f5710107090 2024-04-14T15:05:13.796 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:13.026+0000 7f570dffb700 1 -- 172.21.15.90:0/1774827086 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2470128528 0 0) 0x7f56fc0035b0 con 0x7f5710107090 2024-04-14T15:05:13.796 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:13.026+0000 7f570dffb700 1 -- 172.21.15.90:0/1774827086 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f571010c6a0 con 0x7f5710107090 2024-04-14T15:05:13.796 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:13.026+0000 7f570dffb700 1 -- 172.21.15.90:0/1774827086 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f5708001900 con 0x7f5710107090 2024-04-14T15:05:13.796 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:13.026+0000 7f570dffb700 1 -- 172.21.15.90:0/1774827086 <== mon.0 v1:172.21.15.90:6789/0 4 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f5708004cb0 con 0x7f5710107090 2024-04-14T15:05:13.796 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:13.026+0000 7f570dffb700 1 -- 172.21.15.90:0/1774827086 <== mon.0 v1:172.21.15.90:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f5708005ce0 con 0x7f5710107090 2024-04-14T15:05:13.796 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:13.026+0000 7f5715960700 1 -- 172.21.15.90:0/1774827086 >> v1:172.21.15.90:6789/0 conn(0x7f5710107090 legacy=0x7f57101074b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:05:13.797 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:13.026+0000 7f5715960700 1 -- 172.21.15.90:0/1774827086 shutdown_connections 2024-04-14T15:05:13.797 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:13.026+0000 7f5715960700 1 -- 172.21.15.90:0/1774827086 >> 172.21.15.90:0/1774827086 conn(0x7f5710102610 msgr2=0x7f5710104a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:05:13.797 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:13.026+0000 7f5715960700 1 -- 172.21.15.90:0/1774827086 shutdown_connections 2024-04-14T15:05:13.797 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:13.026+0000 7f5715960700 1 -- 172.21.15.90:0/1774827086 wait complete. 2024-04-14T15:05:13.797 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:13.027+0000 7f5715960700 1 librados: starting msgr at 2024-04-14T15:05:13.797 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:13.027+0000 7f5715960700 1 librados: starting objecter 2024-04-14T15:05:13.797 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:13.027+0000 7f5715960700 1 Processor -- start 2024-04-14T15:05:13.797 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:13.027+0000 7f5715960700 1 -- start start 2024-04-14T15:05:13.797 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:13.027+0000 7f5715960700 1 librados: setting wanted keys 2024-04-14T15:05:13.797 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:13.027+0000 7f5715960700 1 librados: calling monclient init 2024-04-14T15:05:13.797 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:13.027+0000 7f5715960700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f57101a0d30 con 0x7f5710107090 2024-04-14T15:05:13.797 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:13.028+0000 7f570effd700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f5710107090 0x7f571007cb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50034/0 (socket says 172.21.15.90:50034) 2024-04-14T15:05:13.797 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:13.028+0000 7f570effd700 1 -- 172.21.15.90:0/1387900003 learned_addr learned my addr 172.21.15.90:0/1387900003 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:05:13.797 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:13.028+0000 7f571495e700 10 client.?.objecter ms_handle_connect 0x7f5710107090 2024-04-14T15:05:13.797 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:13.028+0000 7f571495e700 10 client.?.objecter resend_mon_ops 2024-04-14T15:05:13.798 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:13.028+0000 7f571495e700 1 -- 172.21.15.90:0/1387900003 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 470271962 0 0) 0x7f57101a0d30 con 0x7f5710107090 2024-04-14T15:05:13.798 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:13.028+0000 7f571495e700 1 -- 172.21.15.90:0/1387900003 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f56fc0056e0 con 0x7f5710107090 2024-04-14T15:05:13.798 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:13.028+0000 7f571495e700 1 -- 172.21.15.90:0/1387900003 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3051338081 0 0) 0x7f571010c6a0 con 0x7f5710107090 2024-04-14T15:05:13.798 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:13.028+0000 7f571495e700 1 -- 172.21.15.90:0/1387900003 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f57101a0d30 con 0x7f5710107090 2024-04-14T15:05:13.798 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:13.028+0000 7f571495e700 1 -- 172.21.15.90:0/1387900003 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f5708001400 con 0x7f5710107090 2024-04-14T15:05:13.798 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:13.028+0000 7f571495e700 1 -- 172.21.15.90:0/1387900003 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 640897029 0 0) 0x7f57101a0d30 con 0x7f5710107090 2024-04-14T15:05:13.798 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:13.028+0000 7f571495e700 1 -- 172.21.15.90:0/1387900003 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57101a1f10 con 0x7f5710107090 2024-04-14T15:05:13.798 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:13.028+0000 7f5715960700 1 -- 172.21.15.90:0/1387900003 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f57101a0f60 con 0x7f5710107090 2024-04-14T15:05:13.798 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:13.028+0000 7f5715960700 10 client.14166.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:05:13.798 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:13.028+0000 7f5715960700 1 -- 172.21.15.90:0/1387900003 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f57101a14a0 con 0x7f5710107090 2024-04-14T15:05:13.798 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:13.028+0000 7f5715960700 1 librados: init done 2024-04-14T15:05:13.798 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:13.028+0000 7f571495e700 1 -- 172.21.15.90:0/1387900003 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f5708004cb0 con 0x7f5710107090 2024-04-14T15:05:13.799 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:13.028+0000 7f571495e700 1 -- 172.21.15.90:0/1387900003 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f57080062d0 con 0x7f5710107090 2024-04-14T15:05:13.799 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:13.029+0000 7f571495e700 1 -- 172.21.15.90:0/1387900003 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 13) ==== 46579+0+0 (unknown 2972057686 0 0) 0x7f5708011b50 con 0x7f5710107090 2024-04-14T15:05:13.799 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:13.029+0000 7f5715960700 1 -- 172.21.15.90:0/1387900003 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56f0005150 con 0x7f5710107090 2024-04-14T15:05:13.799 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:13.029+0000 7f571495e700 1 -- 172.21.15.90:0/1387900003 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(3..3 src has 1..3) ==== 1069+0+0 (unknown 1996007131 0 0) 0x7f5708048430 con 0x7f5710107090 2024-04-14T15:05:13.799 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:13.029+0000 7f571495e700 10 client.14166.objecter ms_dispatch 0x7f5710001550 osd_map(3..3 src has 1..3) 2024-04-14T15:05:13.799 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:13.029+0000 7f571495e700 3 client.14166.objecter handle_osd_map got epochs [3,3] > 0 2024-04-14T15:05:13.799 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:13.029+0000 7f571495e700 3 client.14166.objecter handle_osd_map decoding full epoch 3 2024-04-14T15:05:13.799 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:13.030+0000 7f571495e700 20 client.14166.objecter dump_active .. 0 homeless 2024-04-14T15:05:13.799 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:13.030+0000 7f571495e700 10 client.14166.objecter ms_handle_connect 0x7f56fc03c500 2024-04-14T15:05:13.799 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:13.034+0000 7f571495e700 1 -- 172.21.15.90:0/1387900003 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f56fc0056e0 con 0x7f5710107090 2024-04-14T15:05:13.799 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:13.232+0000 7f5715960700 1 -- 172.21.15.90:0/1387900003 --> v1:172.21.15.90:6789/0 -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7f56f0005440 con 0x7f5710107090 2024-04-14T15:05:13.799 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:13.233+0000 7f571495e700 1 -- 172.21.15.90:0/1387900003 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (unknown 689008156 0 0) 0x7f5708017b90 con 0x7f5710107090 2024-04-14T15:05:13.799 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:13.235+0000 7f5715960700 10 librados: watch_flush enter 2024-04-14T15:05:13.800 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:13.235+0000 7f5715960700 10 librados: watch_flush exit 2024-04-14T15:05:13.800 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:13.235+0000 7f5715960700 20 client.14166.objecter shutdown clearing up homeless session... 2024-04-14T15:05:13.800 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:13.235+0000 7f5715960700 10 client.14166.objecter successfully canceled tick 2024-04-14T15:05:13.800 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:13.235+0000 7f5715960700 1 -- 172.21.15.90:0/1387900003 >> v1:172.21.15.90:6800/491071118 conn(0x7f56fc03c500 legacy=0x7f56fc03e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:05:13.800 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:13.235+0000 7f5715960700 1 -- 172.21.15.90:0/1387900003 >> v1:172.21.15.90:6789/0 conn(0x7f5710107090 legacy=0x7f571007cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:05:13.800 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:13.235+0000 7f5715960700 1 -- 172.21.15.90:0/1387900003 shutdown_connections 2024-04-14T15:05:13.800 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:13.235+0000 7f5715960700 1 -- 172.21.15.90:0/1387900003 >> 172.21.15.90:0/1387900003 conn(0x7f5710102610 msgr2=0x7f5710104a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:05:13.800 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:13.235+0000 7f5715960700 1 -- 172.21.15.90:0/1387900003 shutdown_connections 2024-04-14T15:05:13.800 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:13.235+0000 7f5715960700 1 -- 172.21.15.90:0/1387900003 wait complete. 2024-04-14T15:05:13.800 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:13.235+0000 7f5715960700 1 librados: shutdown 2024-04-14T15:05:15.145 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:14.258+0000 7ff1048b9700 1 Processor -- start 2024-04-14T15:05:15.145 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:14.258+0000 7ff1048b9700 1 -- start start 2024-04-14T15:05:15.146 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:14.258+0000 7ff1048b9700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff10010a3b0 con 0x7ff100105f80 2024-04-14T15:05:15.146 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:14.258+0000 7ff0fe59c700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7ff100105f80 0x7ff1001063a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50046/0 (socket says 172.21.15.90:50046) 2024-04-14T15:05:15.146 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:14.258+0000 7ff0fe59c700 1 -- 172.21.15.90:0/1580975293 learned_addr learned my addr 172.21.15.90:0/1580975293 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:05:15.146 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:14.258+0000 7ff0fd59a700 1 -- 172.21.15.90:0/1580975293 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3200654836 0 0) 0x7ff10010a3b0 con 0x7ff100105f80 2024-04-14T15:05:15.146 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:14.258+0000 7ff0fd59a700 1 -- 172.21.15.90:0/1580975293 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0e80035b0 con 0x7ff100105f80 2024-04-14T15:05:15.146 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:14.259+0000 7ff0fd59a700 1 -- 172.21.15.90:0/1580975293 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1251894799 0 0) 0x7ff0e80035b0 con 0x7ff100105f80 2024-04-14T15:05:15.146 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:14.259+0000 7ff0fd59a700 1 -- 172.21.15.90:0/1580975293 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff10010b590 con 0x7ff100105f80 2024-04-14T15:05:15.146 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:14.259+0000 7ff0fd59a700 1 -- 172.21.15.90:0/1580975293 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7ff0f0001580 con 0x7ff100105f80 2024-04-14T15:05:15.146 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:14.259+0000 7ff0fd59a700 1 -- 172.21.15.90:0/1580975293 <== mon.0 v1:172.21.15.90:6789/0 4 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7ff0f0004930 con 0x7ff100105f80 2024-04-14T15:05:15.146 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:14.259+0000 7ff0fd59a700 1 -- 172.21.15.90:0/1580975293 <== mon.0 v1:172.21.15.90:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7ff0f0005960 con 0x7ff100105f80 2024-04-14T15:05:15.146 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:14.259+0000 7ff1048b9700 1 -- 172.21.15.90:0/1580975293 >> v1:172.21.15.90:6789/0 conn(0x7ff100105f80 legacy=0x7ff1001063a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:05:15.146 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:14.259+0000 7ff1048b9700 1 -- 172.21.15.90:0/1580975293 shutdown_connections 2024-04-14T15:05:15.146 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:14.259+0000 7ff1048b9700 1 -- 172.21.15.90:0/1580975293 >> 172.21.15.90:0/1580975293 conn(0x7ff100101560 msgr2=0x7ff100103980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:05:15.146 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:14.259+0000 7ff1048b9700 1 -- 172.21.15.90:0/1580975293 shutdown_connections 2024-04-14T15:05:15.147 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:14.259+0000 7ff1048b9700 1 -- 172.21.15.90:0/1580975293 wait complete. 2024-04-14T15:05:15.147 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:14.260+0000 7ff1048b9700 1 librados: starting msgr at 2024-04-14T15:05:15.147 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:14.260+0000 7ff1048b9700 1 librados: starting objecter 2024-04-14T15:05:15.147 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:14.260+0000 7ff1048b9700 1 Processor -- start 2024-04-14T15:05:15.147 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:14.260+0000 7ff1048b9700 1 -- start start 2024-04-14T15:05:15.147 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:14.260+0000 7ff1048b9700 1 librados: setting wanted keys 2024-04-14T15:05:15.147 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:14.260+0000 7ff1048b9700 1 librados: calling monclient init 2024-04-14T15:05:15.147 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:14.260+0000 7ff1048b9700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff10019b400 con 0x7ff100105f80 2024-04-14T15:05:15.147 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:14.260+0000 7ff0fe59c700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7ff100105f80 0x7ff10019acf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50062/0 (socket says 172.21.15.90:50062) 2024-04-14T15:05:15.147 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:14.260+0000 7ff0fe59c700 1 -- 172.21.15.90:0/3886437468 learned_addr learned my addr 172.21.15.90:0/3886437468 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:05:15.147 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:14.260+0000 7ff0ef7fe700 10 client.?.objecter ms_handle_connect 0x7ff100105f80 2024-04-14T15:05:15.147 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:14.260+0000 7ff0ef7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:05:15.147 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:14.260+0000 7ff0ef7fe700 1 -- 172.21.15.90:0/3886437468 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2332338622 0 0) 0x7ff10019b400 con 0x7ff100105f80 2024-04-14T15:05:15.147 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:14.260+0000 7ff0ef7fe700 1 -- 172.21.15.90:0/3886437468 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0e80056e0 con 0x7ff100105f80 2024-04-14T15:05:15.147 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:14.261+0000 7ff0ef7fe700 1 -- 172.21.15.90:0/3886437468 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3303619839 0 0) 0x7ff10010b590 con 0x7ff100105f80 2024-04-14T15:05:15.148 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:14.261+0000 7ff0ef7fe700 1 -- 172.21.15.90:0/3886437468 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff10019b400 con 0x7ff100105f80 2024-04-14T15:05:15.148 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:14.261+0000 7ff0ef7fe700 1 -- 172.21.15.90:0/3886437468 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7ff0f0001080 con 0x7ff100105f80 2024-04-14T15:05:15.148 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:14.261+0000 7ff0ef7fe700 1 -- 172.21.15.90:0/3886437468 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1325263946 0 0) 0x7ff10019b400 con 0x7ff100105f80 2024-04-14T15:05:15.148 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:14.261+0000 7ff0ef7fe700 1 -- 172.21.15.90:0/3886437468 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff10019b5d0 con 0x7ff100105f80 2024-04-14T15:05:15.148 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:14.261+0000 7ff1048b9700 1 -- 172.21.15.90:0/3886437468 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff10019b8c0 con 0x7ff100105f80 2024-04-14T15:05:15.148 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:14.261+0000 7ff1048b9700 10 client.14168.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:05:15.148 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:14.261+0000 7ff1048b9700 1 -- 172.21.15.90:0/3886437468 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff10019f400 con 0x7ff100105f80 2024-04-14T15:05:15.148 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:14.261+0000 7ff1048b9700 1 librados: init done 2024-04-14T15:05:15.148 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:14.261+0000 7ff0ef7fe700 1 -- 172.21.15.90:0/3886437468 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7ff0f0004930 con 0x7ff100105f80 2024-04-14T15:05:15.148 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:14.261+0000 7ff0ef7fe700 1 -- 172.21.15.90:0/3886437468 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7ff0f0005f50 con 0x7ff100105f80 2024-04-14T15:05:15.148 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:14.262+0000 7ff0ef7fe700 1 -- 172.21.15.90:0/3886437468 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 13) ==== 46579+0+0 (unknown 2972057686 0 0) 0x7ff0f0011710 con 0x7ff100105f80 2024-04-14T15:05:15.148 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:14.262+0000 7ff1048b9700 1 -- 172.21.15.90:0/3886437468 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0e0005150 con 0x7ff100105f80 2024-04-14T15:05:15.148 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:14.262+0000 7ff0ef7fe700 1 -- 172.21.15.90:0/3886437468 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(3..3 src has 1..3) ==== 1069+0+0 (unknown 1996007131 0 0) 0x7ff0f0048040 con 0x7ff100105f80 2024-04-14T15:05:15.148 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:14.262+0000 7ff0ef7fe700 10 client.14168.objecter ms_dispatch 0x7ff100001550 osd_map(3..3 src has 1..3) 2024-04-14T15:05:15.148 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:14.262+0000 7ff0ef7fe700 3 client.14168.objecter handle_osd_map got epochs [3,3] > 0 2024-04-14T15:05:15.149 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:14.262+0000 7ff0ef7fe700 3 client.14168.objecter handle_osd_map decoding full epoch 3 2024-04-14T15:05:15.149 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:14.262+0000 7ff0ef7fe700 20 client.14168.objecter dump_active .. 0 homeless 2024-04-14T15:05:15.149 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:14.262+0000 7ff0ef7fe700 10 client.14168.objecter ms_handle_connect 0x7ff0e803c560 2024-04-14T15:05:15.149 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:14.267+0000 7ff0ef7fe700 1 -- 172.21.15.90:0/3886437468 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7ff0e80056e0 con 0x7ff100105f80 2024-04-14T15:05:15.149 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:14.577+0000 7ff1048b9700 1 -- 172.21.15.90:0/3886437468 --> v1:172.21.15.90:6789/0 -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7ff0e0005440 con 0x7ff100105f80 2024-04-14T15:05:15.149 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:14.579+0000 7ff0ef7fe700 1 -- 172.21.15.90:0/3886437468 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) ==== 153+0+0 (unknown 1169358022 0 0) 0x7ff0f0017730 con 0x7ff100105f80 2024-04-14T15:05:15.149 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:14.581+0000 7ff1048b9700 10 librados: watch_flush enter 2024-04-14T15:05:15.149 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:14.581+0000 7ff1048b9700 10 librados: watch_flush exit 2024-04-14T15:05:15.149 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:14.581+0000 7ff1048b9700 20 client.14168.objecter shutdown clearing up homeless session... 2024-04-14T15:05:15.149 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:14.581+0000 7ff1048b9700 10 client.14168.objecter successfully canceled tick 2024-04-14T15:05:15.149 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:14.581+0000 7ff1048b9700 1 -- 172.21.15.90:0/3886437468 >> v1:172.21.15.90:6800/491071118 conn(0x7ff0e803c560 legacy=0x7ff0e803ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:05:15.149 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:14.581+0000 7ff1048b9700 1 -- 172.21.15.90:0/3886437468 >> v1:172.21.15.90:6789/0 conn(0x7ff100105f80 legacy=0x7ff10019acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:05:15.150 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:14.581+0000 7ff1048b9700 1 -- 172.21.15.90:0/3886437468 shutdown_connections 2024-04-14T15:05:15.150 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:14.581+0000 7ff1048b9700 1 -- 172.21.15.90:0/3886437468 >> 172.21.15.90:0/3886437468 conn(0x7ff100101560 msgr2=0x7ff100079fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:05:15.150 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:14.581+0000 7ff1048b9700 1 -- 172.21.15.90:0/3886437468 shutdown_connections 2024-04-14T15:05:15.150 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:14.581+0000 7ff1048b9700 1 -- 172.21.15.90:0/3886437468 wait complete. 2024-04-14T15:05:15.150 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-14T15:05:14.581+0000 7ff1048b9700 1 librados: shutdown 2024-04-14T15:05:15.150 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-14T15:05:15.150 INFO:teuthology.orchestra.run.smithi090.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-14T15:05:15.150 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:05:15.150 INFO:teuthology.orchestra.run.smithi090.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-14T15:05:15.150 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:05:15.150 INFO:teuthology.orchestra.run.smithi090.stdout:Or, if you are only running a single cluster on this host: 2024-04-14T15:05:15.150 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:05:15.150 INFO:teuthology.orchestra.run.smithi090.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-14T15:05:15.150 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:05:15.150 INFO:teuthology.orchestra.run.smithi090.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-14T15:05:15.150 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:05:15.151 INFO:teuthology.orchestra.run.smithi090.stdout: ceph telemetry on 2024-04-14T15:05:15.151 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:05:15.151 INFO:teuthology.orchestra.run.smithi090.stdout:For more information see: 2024-04-14T15:05:15.151 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:05:15.151 INFO:teuthology.orchestra.run.smithi090.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-14T15:05:15.151 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:05:15.151 INFO:teuthology.orchestra.run.smithi090.stdout:Bootstrap complete. 2024-04-14T15:05:15.184 INFO:tasks.cephadm:Fetching config... 2024-04-14T15:05:15.184 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-14T15:05:15.185 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-14T15:05:15.200 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-14T15:05:15.200 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-14T15:05:15.200 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-14T15:05:15.255 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-14T15:05:15.255 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-14T15:05:15.255 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-14T15:05:15.321 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-14T15:05:15.322 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-14T15:05:15.322 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-14T15:05:15.377 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-14T15:05:15.377 DEBUG:teuthology.orchestra.run.smithi090:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFHb0NOZETQs1TVRGlSy67VtsqfO3n76visoDbsAzvW2+lwxTcrtzpKMHBBm5kBv1+huZZPyu+tva5xAK3j+DDgzpkbigXgzzqEIIAOalTNTW7VuJEiOCx7iZChEu2Fqp8dl2/YfcxqvF78UQidkmoiBeTLt9guNjN4G2Fi1gz8ts28XwK2qCuIvlKCj7gOTcyf0fRoCOHIWT+6xSJAKByTaojGvnfHIdeGHwJvzVLUlC5mnVU8RoojOIJ+/C0PE38eqq5HoY9A3NkRobIh8D6pRNkus6CSY/IVyPcLtH3vfUmnsj6hE7UEz+ZUhDFCOPDS6eukJ7UJxB0676jfw08KXRWU4RnateGr2kwEqeZ+HfLAfdUfWuW/ZeHkvcUwgE9LYb687K/3B9DHm9VCgkue4XW5yteTghd2pJWjlFT+eZsuC08oX6GAoRG6DigIXp1mz20qks6i6WUJhOqK1foSWbvYnoy+ZhuTY3L0ifPgaY5aXpHs4nsIaASzFILSSM= ceph-0e288098-fa70-11ee-bc8e-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-14T15:05:15.463 INFO:teuthology.orchestra.run.smithi090.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFHb0NOZETQs1TVRGlSy67VtsqfO3n76visoDbsAzvW2+lwxTcrtzpKMHBBm5kBv1+huZZPyu+tva5xAK3j+DDgzpkbigXgzzqEIIAOalTNTW7VuJEiOCx7iZChEu2Fqp8dl2/YfcxqvF78UQidkmoiBeTLt9guNjN4G2Fi1gz8ts28XwK2qCuIvlKCj7gOTcyf0fRoCOHIWT+6xSJAKByTaojGvnfHIdeGHwJvzVLUlC5mnVU8RoojOIJ+/C0PE38eqq5HoY9A3NkRobIh8D6pRNkus6CSY/IVyPcLtH3vfUmnsj6hE7UEz+ZUhDFCOPDS6eukJ7UJxB0676jfw08KXRWU4RnateGr2kwEqeZ+HfLAfdUfWuW/ZeHkvcUwgE9LYb687K/3B9DHm9VCgkue4XW5yteTghd2pJWjlFT+eZsuC08oX6GAoRG6DigIXp1mz20qks6i6WUJhOqK1foSWbvYnoy+ZhuTY3L0ifPgaY5aXpHs4nsIaASzFILSSM= ceph-0e288098-fa70-11ee-bc8e-c7b262605968 2024-04-14T15:05:15.486 DEBUG:teuthology.orchestra.run.smithi092:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFHb0NOZETQs1TVRGlSy67VtsqfO3n76visoDbsAzvW2+lwxTcrtzpKMHBBm5kBv1+huZZPyu+tva5xAK3j+DDgzpkbigXgzzqEIIAOalTNTW7VuJEiOCx7iZChEu2Fqp8dl2/YfcxqvF78UQidkmoiBeTLt9guNjN4G2Fi1gz8ts28XwK2qCuIvlKCj7gOTcyf0fRoCOHIWT+6xSJAKByTaojGvnfHIdeGHwJvzVLUlC5mnVU8RoojOIJ+/C0PE38eqq5HoY9A3NkRobIh8D6pRNkus6CSY/IVyPcLtH3vfUmnsj6hE7UEz+ZUhDFCOPDS6eukJ7UJxB0676jfw08KXRWU4RnateGr2kwEqeZ+HfLAfdUfWuW/ZeHkvcUwgE9LYb687K/3B9DHm9VCgkue4XW5yteTghd2pJWjlFT+eZsuC08oX6GAoRG6DigIXp1mz20qks6i6WUJhOqK1foSWbvYnoy+ZhuTY3L0ifPgaY5aXpHs4nsIaASzFILSSM= ceph-0e288098-fa70-11ee-bc8e-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-14T15:05:15.532 INFO:teuthology.orchestra.run.smithi092.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFHb0NOZETQs1TVRGlSy67VtsqfO3n76visoDbsAzvW2+lwxTcrtzpKMHBBm5kBv1+huZZPyu+tva5xAK3j+DDgzpkbigXgzzqEIIAOalTNTW7VuJEiOCx7iZChEu2Fqp8dl2/YfcxqvF78UQidkmoiBeTLt9guNjN4G2Fi1gz8ts28XwK2qCuIvlKCj7gOTcyf0fRoCOHIWT+6xSJAKByTaojGvnfHIdeGHwJvzVLUlC5mnVU8RoojOIJ+/C0PE38eqq5HoY9A3NkRobIh8D6pRNkus6CSY/IVyPcLtH3vfUmnsj6hE7UEz+ZUhDFCOPDS6eukJ7UJxB0676jfw08KXRWU4RnateGr2kwEqeZ+HfLAfdUfWuW/ZeHkvcUwgE9LYb687K/3B9DHm9VCgkue4XW5yteTghd2pJWjlFT+eZsuC08oX6GAoRG6DigIXp1mz20qks6i6WUJhOqK1foSWbvYnoy+ZhuTY3L0ifPgaY5aXpHs4nsIaASzFILSSM= ceph-0e288098-fa70-11ee-bc8e-c7b262605968 2024-04-14T15:05:15.547 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-14T15:05:15.691 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:15 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3886437468' entity='client.admin' 2024-04-14T15:05:15.786 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:05:18.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:18.217+0000 7f3b05bcf700 1 -- 172.21.15.90:0/197382760 >> v1:172.21.15.90:6789/0 conn(0x7f3b00101330 legacy=0x7f3b00101750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:05:18.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:18.217+0000 7f3b05bcf700 1 -- 172.21.15.90:0/197382760 shutdown_connections 2024-04-14T15:05:18.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:18.217+0000 7f3b05bcf700 1 -- 172.21.15.90:0/197382760 >> 172.21.15.90:0/197382760 conn(0x7f3b000fc8d0 msgr2=0x7f3b000fed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:05:18.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:18.217+0000 7f3b05bcf700 1 -- 172.21.15.90:0/197382760 shutdown_connections 2024-04-14T15:05:18.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:18.217+0000 7f3b05bcf700 1 -- 172.21.15.90:0/197382760 wait complete. 2024-04-14T15:05:18.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:18.218+0000 7f3b05bcf700 1 librados: starting msgr at 2024-04-14T15:05:18.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:18.218+0000 7f3b05bcf700 1 librados: starting objecter 2024-04-14T15:05:18.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:18.218+0000 7f3b05bcf700 1 Processor -- start 2024-04-14T15:05:18.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:18.218+0000 7f3b05bcf700 1 -- start start 2024-04-14T15:05:18.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:18.218+0000 7f3b05bcf700 1 librados: setting wanted keys 2024-04-14T15:05:18.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:18.218+0000 7f3b05bcf700 1 librados: calling monclient init 2024-04-14T15:05:18.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:18.218+0000 7f3b05bcf700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3b00196520 con 0x7f3b00101330 2024-04-14T15:05:18.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:18.218+0000 7f3aff7fe700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f3b00101330 0x7f3b00195e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50078/0 (socket says 172.21.15.90:50078) 2024-04-14T15:05:18.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:18.218+0000 7f3aff7fe700 1 -- 172.21.15.90:0/306393794 learned_addr learned my addr 172.21.15.90:0/306393794 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:05:18.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:18.218+0000 7f3afcff9700 10 client.?.objecter ms_handle_connect 0x7f3b00101330 2024-04-14T15:05:18.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:18.218+0000 7f3afcff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:05:18.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:18.219+0000 7f3afcff9700 1 -- 172.21.15.90:0/306393794 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2764306389 0 0) 0x7f3b00196520 con 0x7f3b00101330 2024-04-14T15:05:18.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:18.219+0000 7f3afcff9700 1 -- 172.21.15.90:0/306393794 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3ae80056e0 con 0x7f3b00101330 2024-04-14T15:05:18.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:18.219+0000 7f3afcff9700 1 -- 172.21.15.90:0/306393794 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 236526626 0 0) 0x7f3b00106940 con 0x7f3b00101330 2024-04-14T15:05:18.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:18.219+0000 7f3afcff9700 1 -- 172.21.15.90:0/306393794 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3b00196520 con 0x7f3b00101330 2024-04-14T15:05:18.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:18.219+0000 7f3afcff9700 1 -- 172.21.15.90:0/306393794 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f3af0001480 con 0x7f3b00101330 2024-04-14T15:05:18.222 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:18 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:18.222 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:18 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:18.222 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:18 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi090", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:05:18.222 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:18 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:18.222 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:18 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:05:18.222 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:18 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:18.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:18.219+0000 7f3afcff9700 1 -- 172.21.15.90:0/306393794 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3264013318 0 0) 0x7f3b00196520 con 0x7f3b00101330 2024-04-14T15:05:18.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:18.219+0000 7f3afcff9700 1 -- 172.21.15.90:0/306393794 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b001966f0 con 0x7f3b00101330 2024-04-14T15:05:18.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:18.219+0000 7f3b05bcf700 1 -- 172.21.15.90:0/306393794 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3b001969e0 con 0x7f3b00101330 2024-04-14T15:05:18.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:18.219+0000 7f3b05bcf700 10 client.14170.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:05:18.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:18.219+0000 7f3b05bcf700 1 -- 172.21.15.90:0/306393794 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f3b0019a530 con 0x7f3b00101330 2024-04-14T15:05:18.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:18.219+0000 7f3b05bcf700 1 librados: init done 2024-04-14T15:05:18.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:18.219+0000 7f3afcff9700 1 -- 172.21.15.90:0/306393794 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f3af0003700 con 0x7f3b00101330 2024-04-14T15:05:18.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:18.219+0000 7f3afcff9700 1 -- 172.21.15.90:0/306393794 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f3af0004e00 con 0x7f3b00101330 2024-04-14T15:05:18.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:18.220+0000 7f3afcff9700 1 -- 172.21.15.90:0/306393794 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 13) ==== 46579+0+0 (unknown 2972057686 0 0) 0x7f3af0004fc0 con 0x7f3b00101330 2024-04-14T15:05:18.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:18.220+0000 7f3afcff9700 1 -- 172.21.15.90:0/306393794 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(3..3 src has 1..3) ==== 1069+0+0 (unknown 1996007131 0 0) 0x7f3af0047f70 con 0x7f3b00101330 2024-04-14T15:05:18.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:18.220+0000 7f3afcff9700 10 client.14170.objecter ms_dispatch 0x7f3b00001550 osd_map(3..3 src has 1..3) 2024-04-14T15:05:18.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:18.220+0000 7f3afcff9700 3 client.14170.objecter handle_osd_map got epochs [3,3] > 0 2024-04-14T15:05:18.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:18.220+0000 7f3afcff9700 3 client.14170.objecter handle_osd_map decoding full epoch 3 2024-04-14T15:05:18.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:18.220+0000 7f3afcff9700 20 client.14170.objecter dump_active .. 0 homeless 2024-04-14T15:05:18.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:18.220+0000 7f3b05bcf700 1 -- 172.21.15.90:0/306393794 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3aec005150 con 0x7f3b00101330 2024-04-14T15:05:18.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:18.221+0000 7f3afcff9700 10 client.14170.objecter ms_handle_connect 0x7f3ae803e5c0 2024-04-14T15:05:18.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:18.226+0000 7f3afcff9700 1 -- 172.21.15.90:0/306393794 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f3ae80056e0 con 0x7f3b00101330 2024-04-14T15:05:18.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:18.426+0000 7f3b05bcf700 1 -- 172.21.15.90:0/306393794 --> v1:172.21.15.90:6789/0 -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f3aec005440 con 0x7f3b00101330 2024-04-14T15:05:18.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:18.428+0000 7f3afcff9700 1 -- 172.21.15.90:0/306393794 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (unknown 3028693289 0 0) 0x7f3af0017680 con 0x7f3b00101330 2024-04-14T15:05:18.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:18.430+0000 7f3b05bcf700 10 librados: watch_flush enter 2024-04-14T15:05:18.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:18.430+0000 7f3b05bcf700 10 librados: watch_flush exit 2024-04-14T15:05:18.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:18.430+0000 7f3b05bcf700 20 client.14170.objecter shutdown clearing up homeless session... 2024-04-14T15:05:18.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:18.430+0000 7f3b05bcf700 10 client.14170.objecter successfully canceled tick 2024-04-14T15:05:18.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:18.430+0000 7f3b05bcf700 1 -- 172.21.15.90:0/306393794 >> v1:172.21.15.90:6800/491071118 conn(0x7f3ae803e5c0 legacy=0x7f3ae8040a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:05:18.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:18.430+0000 7f3b05bcf700 1 -- 172.21.15.90:0/306393794 >> v1:172.21.15.90:6789/0 conn(0x7f3b00101330 legacy=0x7f3b00195e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:05:18.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:18.431+0000 7f3b05bcf700 1 -- 172.21.15.90:0/306393794 shutdown_connections 2024-04-14T15:05:18.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:18.431+0000 7f3b05bcf700 1 -- 172.21.15.90:0/306393794 >> 172.21.15.90:0/306393794 conn(0x7f3b000fc8d0 msgr2=0x7f3b000fed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:05:18.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:18.431+0000 7f3b05bcf700 1 -- 172.21.15.90:0/306393794 shutdown_connections 2024-04-14T15:05:18.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:18.431+0000 7f3b05bcf700 1 -- 172.21.15.90:0/306393794 wait complete. 2024-04-14T15:05:18.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:18.431+0000 7f3b05bcf700 1 librados: shutdown 2024-04-14T15:05:19.081 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-14T15:05:19.081 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-14T15:05:19.610 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:19 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/306393794' entity='client.admin' 2024-04-14T15:05:19.611 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:19 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:05:19.611 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:19 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:19.647 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:05:21.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:21.168+0000 7f3d0571a700 1 -- 172.21.15.90:0/2430454051 >> v1:172.21.15.90:6789/0 conn(0x7f3d00101340 legacy=0x7f3d00101760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:05:21.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:21.168+0000 7f3d0571a700 1 -- 172.21.15.90:0/2430454051 shutdown_connections 2024-04-14T15:05:21.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:21.168+0000 7f3d0571a700 1 -- 172.21.15.90:0/2430454051 >> 172.21.15.90:0/2430454051 conn(0x7f3d000fc8e0 msgr2=0x7f3d000fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:05:21.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:21.168+0000 7f3d0571a700 1 -- 172.21.15.90:0/2430454051 shutdown_connections 2024-04-14T15:05:21.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:21.168+0000 7f3d0571a700 1 -- 172.21.15.90:0/2430454051 wait complete. 2024-04-14T15:05:21.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:21.169+0000 7f3d0571a700 1 librados: starting msgr at 2024-04-14T15:05:21.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:21.169+0000 7f3d0571a700 1 librados: starting objecter 2024-04-14T15:05:21.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:21.169+0000 7f3d0571a700 1 Processor -- start 2024-04-14T15:05:21.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:21.169+0000 7f3d0571a700 1 -- start start 2024-04-14T15:05:21.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:21.169+0000 7f3d0571a700 1 librados: setting wanted keys 2024-04-14T15:05:21.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:21.169+0000 7f3d0571a700 1 librados: calling monclient init 2024-04-14T15:05:21.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:21.169+0000 7f3d0571a700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3d001975e0 con 0x7f3d00101340 2024-04-14T15:05:21.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:21.170+0000 7f3cfeffd700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f3d00101340 0x7f3d00196ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50164/0 (socket says 172.21.15.90:50164) 2024-04-14T15:05:21.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:21.170+0000 7f3cfeffd700 1 -- 172.21.15.90:0/2140700756 learned_addr learned my addr 172.21.15.90:0/2140700756 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:05:21.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:21.170+0000 7f3ce7fff700 10 client.?.objecter ms_handle_connect 0x7f3d00101340 2024-04-14T15:05:21.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:21.170+0000 7f3ce7fff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:05:21.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:21.170+0000 7f3ce7fff700 1 -- 172.21.15.90:0/2140700756 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 198945375 0 0) 0x7f3d001975e0 con 0x7f3d00101340 2024-04-14T15:05:21.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:21.170+0000 7f3ce7fff700 1 -- 172.21.15.90:0/2140700756 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3ce80056e0 con 0x7f3d00101340 2024-04-14T15:05:21.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:21.170+0000 7f3ce7fff700 1 -- 172.21.15.90:0/2140700756 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2606738398 0 0) 0x7f3d00106950 con 0x7f3d00101340 2024-04-14T15:05:21.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:21.170+0000 7f3ce7fff700 1 -- 172.21.15.90:0/2140700756 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3d001975e0 con 0x7f3d00101340 2024-04-14T15:05:21.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:21.170+0000 7f3ce7fff700 1 -- 172.21.15.90:0/2140700756 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f3cf0001480 con 0x7f3d00101340 2024-04-14T15:05:21.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:21.170+0000 7f3ce7fff700 1 -- 172.21.15.90:0/2140700756 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 986020027 0 0) 0x7f3d001975e0 con 0x7f3d00101340 2024-04-14T15:05:21.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:21.170+0000 7f3ce7fff700 1 -- 172.21.15.90:0/2140700756 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d001977b0 con 0x7f3d00101340 2024-04-14T15:05:21.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:21.170+0000 7f3d0571a700 1 -- 172.21.15.90:0/2140700756 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3d00197aa0 con 0x7f3d00101340 2024-04-14T15:05:21.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:21.170+0000 7f3d0571a700 10 client.14172.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:05:21.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:21.170+0000 7f3d0571a700 1 -- 172.21.15.90:0/2140700756 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f3d0019b5f0 con 0x7f3d00101340 2024-04-14T15:05:21.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:21.170+0000 7f3d0571a700 1 librados: init done 2024-04-14T15:05:21.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:21.170+0000 7f3ce7fff700 1 -- 172.21.15.90:0/2140700756 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f3cf0001ba0 con 0x7f3d00101340 2024-04-14T15:05:21.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:21.170+0000 7f3ce7fff700 1 -- 172.21.15.90:0/2140700756 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f3cf0005110 con 0x7f3d00101340 2024-04-14T15:05:21.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:21.171+0000 7f3ce7fff700 1 -- 172.21.15.90:0/2140700756 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 13) ==== 46579+0+0 (unknown 2972057686 0 0) 0x7f3cf00052d0 con 0x7f3d00101340 2024-04-14T15:05:21.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:21.171+0000 7f3d0571a700 1 -- 172.21.15.90:0/2140700756 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cec005150 con 0x7f3d00101340 2024-04-14T15:05:21.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:21.171+0000 7f3ce7fff700 1 -- 172.21.15.90:0/2140700756 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(3..3 src has 1..3) ==== 1069+0+0 (unknown 1996007131 0 0) 0x7f3cf0048170 con 0x7f3d00101340 2024-04-14T15:05:21.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:21.171+0000 7f3ce7fff700 10 client.14172.objecter ms_dispatch 0x7f3d00001550 osd_map(3..3 src has 1..3) 2024-04-14T15:05:21.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:21.171+0000 7f3ce7fff700 3 client.14172.objecter handle_osd_map got epochs [3,3] > 0 2024-04-14T15:05:21.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:21.171+0000 7f3ce7fff700 3 client.14172.objecter handle_osd_map decoding full epoch 3 2024-04-14T15:05:21.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:21.172+0000 7f3ce7fff700 20 client.14172.objecter dump_active .. 0 homeless 2024-04-14T15:05:21.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:21.172+0000 7f3ce7fff700 10 client.14172.objecter ms_handle_connect 0x7f3ce803c2f0 2024-04-14T15:05:21.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:21.176+0000 7f3ce7fff700 1 -- 172.21.15.90:0/2140700756 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f3ce80056e0 con 0x7f3d00101340 2024-04-14T15:05:21.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:21.378+0000 7f3d0571a700 1 -- 172.21.15.90:0/2140700756 --> v1:172.21.15.90:6800/491071118 -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f3cec000bc0 con 0x7f3ce803c2f0 2024-04-14T15:05:21.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:21.381+0000 7f3ce7fff700 1 -- 172.21.15.90:0/2140700756 <== mgr.14148 v1:172.21.15.90:6800/491071118 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (unknown 0 0 0) 0x7f3cec000bc0 con 0x7f3ce803c2f0 2024-04-14T15:05:21.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:21.383+0000 7f3d0571a700 10 librados: watch_flush enter 2024-04-14T15:05:21.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:21.383+0000 7f3d0571a700 10 librados: watch_flush exit 2024-04-14T15:05:21.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:21.383+0000 7f3d0571a700 20 client.14172.objecter shutdown clearing up homeless session... 2024-04-14T15:05:21.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:21.383+0000 7f3d0571a700 10 client.14172.objecter successfully canceled tick 2024-04-14T15:05:21.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:21.383+0000 7f3d0571a700 1 -- 172.21.15.90:0/2140700756 >> v1:172.21.15.90:6800/491071118 conn(0x7f3ce803c2f0 legacy=0x7f3ce803e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:05:21.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:21.384+0000 7f3d0571a700 1 -- 172.21.15.90:0/2140700756 >> v1:172.21.15.90:6789/0 conn(0x7f3d00101340 legacy=0x7f3d00196ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:05:21.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:21.384+0000 7f3d0571a700 1 -- 172.21.15.90:0/2140700756 shutdown_connections 2024-04-14T15:05:21.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:21.384+0000 7f3d0571a700 1 -- 172.21.15.90:0/2140700756 >> 172.21.15.90:0/2140700756 conn(0x7f3d000fc8e0 msgr2=0x7f3d00104590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:05:21.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:21.384+0000 7f3d0571a700 1 -- 172.21.15.90:0/2140700756 shutdown_connections 2024-04-14T15:05:21.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:21.384+0000 7f3d0571a700 1 -- 172.21.15.90:0/2140700756 wait complete. 2024-04-14T15:05:21.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:21.384+0000 7f3d0571a700 1 librados: shutdown 2024-04-14T15:05:22.147 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi092 2024-04-14T15:05:22.147 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-04-14T15:05:22.147 DEBUG:teuthology.orchestra.run.smithi092:> dd of=/etc/ceph/ceph.conf 2024-04-14T15:05:22.165 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-04-14T15:05:22.165 DEBUG:teuthology.orchestra.run.smithi092:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-14T15:05:22.222 INFO:tasks.cephadm:Adding host smithi092 to orchestrator... 2024-04-14T15:05:22.222 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph orch host add smithi092 2024-04-14T15:05:22.487 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:05:22.591 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:22 smithi090 ceph-mon[30399]: from='client.14172 v1:172.21.15.90:0/2140700756' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:05:22.591 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:22 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:22.591 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:22 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:05:22.592 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:22 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:05:22.592 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:22 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:05:22.592 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:22 smithi090 ceph-mon[30399]: Updating smithi090:/etc/ceph/ceph.conf 2024-04-14T15:05:22.592 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:22 smithi090 ceph-mon[30399]: Updating smithi090:/var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T15:05:22.592 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:22 smithi090 ceph-mon[30399]: Updating smithi090:/etc/ceph/ceph.client.admin.keyring 2024-04-14T15:05:22.592 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:22 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:22.592 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:22 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:22.592 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:22 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:23.515 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:23 smithi090 ceph-mon[30399]: Updating smithi090:/var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T15:05:23.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:23.981+0000 7fb66b035700 1 -- 172.21.15.90:0/1000549000 >> v1:172.21.15.90:6789/0 conn(0x7fb6641084b0 legacy=0x7fb664108890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:05:23.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:23.981+0000 7fb66b035700 1 -- 172.21.15.90:0/1000549000 shutdown_connections 2024-04-14T15:05:23.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:23.981+0000 7fb66b035700 1 -- 172.21.15.90:0/1000549000 >> 172.21.15.90:0/1000549000 conn(0x7fb6640fdf30 msgr2=0x7fb664100350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:05:23.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:23.981+0000 7fb66b035700 1 -- 172.21.15.90:0/1000549000 shutdown_connections 2024-04-14T15:05:23.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:23.981+0000 7fb66b035700 1 -- 172.21.15.90:0/1000549000 wait complete. 2024-04-14T15:05:23.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:23.982+0000 7fb66b035700 1 librados: starting msgr at 2024-04-14T15:05:23.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:23.982+0000 7fb66b035700 1 librados: starting objecter 2024-04-14T15:05:23.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:23.982+0000 7fb66b035700 1 Processor -- start 2024-04-14T15:05:23.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:23.982+0000 7fb66b035700 1 -- start start 2024-04-14T15:05:23.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:23.982+0000 7fb66b035700 1 librados: setting wanted keys 2024-04-14T15:05:23.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:23.982+0000 7fb66b035700 1 librados: calling monclient init 2024-04-14T15:05:23.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:23.982+0000 7fb66b035700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb664197c30 con 0x7fb6641084b0 2024-04-14T15:05:23.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:23.982+0000 7fb668dd1700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fb6641084b0 0x7fb664197520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50186/0 (socket says 172.21.15.90:50186) 2024-04-14T15:05:23.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:23.982+0000 7fb668dd1700 1 -- 172.21.15.90:0/2588275619 learned_addr learned my addr 172.21.15.90:0/2588275619 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:05:23.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:23.982+0000 7fb661ffb700 10 client.?.objecter ms_handle_connect 0x7fb6641084b0 2024-04-14T15:05:23.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:23.982+0000 7fb661ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:05:23.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:23.982+0000 7fb661ffb700 1 -- 172.21.15.90:0/2588275619 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1584438191 0 0) 0x7fb664197c30 con 0x7fb6641084b0 2024-04-14T15:05:23.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:23.982+0000 7fb661ffb700 1 -- 172.21.15.90:0/2588275619 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6540056e0 con 0x7fb6641084b0 2024-04-14T15:05:23.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:23.983+0000 7fb661ffb700 1 -- 172.21.15.90:0/2588275619 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4004366838 0 0) 0x7fb6540056e0 con 0x7fb6641084b0 2024-04-14T15:05:23.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:23.983+0000 7fb661ffb700 1 -- 172.21.15.90:0/2588275619 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb664197c30 con 0x7fb6641084b0 2024-04-14T15:05:23.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:23.983+0000 7fb661ffb700 1 -- 172.21.15.90:0/2588275619 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7fb6580047f0 con 0x7fb6641084b0 2024-04-14T15:05:23.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:23.983+0000 7fb661ffb700 1 -- 172.21.15.90:0/2588275619 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3247613566 0 0) 0x7fb664197c30 con 0x7fb6641084b0 2024-04-14T15:05:23.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:23.983+0000 7fb661ffb700 1 -- 172.21.15.90:0/2588275619 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb664197e00 con 0x7fb6641084b0 2024-04-14T15:05:23.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:23.983+0000 7fb66b035700 1 -- 172.21.15.90:0/2588275619 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb6641980f0 con 0x7fb6641084b0 2024-04-14T15:05:23.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:23.983+0000 7fb661ffb700 1 -- 172.21.15.90:0/2588275619 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fb658003b20 con 0x7fb6641084b0 2024-04-14T15:05:23.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:23.983+0000 7fb661ffb700 1 -- 172.21.15.90:0/2588275619 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7fb658006120 con 0x7fb6641084b0 2024-04-14T15:05:23.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:23.983+0000 7fb66b035700 10 client.14174.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:05:23.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:23.983+0000 7fb66b035700 1 -- 172.21.15.90:0/2588275619 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb6641a02b0 con 0x7fb6641084b0 2024-04-14T15:05:23.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:23.983+0000 7fb66b035700 1 librados: init done 2024-04-14T15:05:23.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:23.984+0000 7fb661ffb700 1 -- 172.21.15.90:0/2588275619 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 13) ==== 46579+0+0 (unknown 2972057686 0 0) 0x7fb6580062c0 con 0x7fb6641084b0 2024-04-14T15:05:23.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:23.984+0000 7fb66b035700 1 -- 172.21.15.90:0/2588275619 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb664064110 con 0x7fb6641084b0 2024-04-14T15:05:23.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:23.984+0000 7fb661ffb700 1 -- 172.21.15.90:0/2588275619 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(3..3 src has 1..3) ==== 1069+0+0 (unknown 1996007131 0 0) 0x7fb658047fc0 con 0x7fb6641084b0 2024-04-14T15:05:23.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:23.984+0000 7fb661ffb700 10 client.14174.objecter ms_dispatch 0x7fb664001550 osd_map(3..3 src has 1..3) 2024-04-14T15:05:23.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:23.984+0000 7fb661ffb700 3 client.14174.objecter handle_osd_map got epochs [3,3] > 0 2024-04-14T15:05:23.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:23.984+0000 7fb661ffb700 3 client.14174.objecter handle_osd_map decoding full epoch 3 2024-04-14T15:05:23.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:23.984+0000 7fb661ffb700 20 client.14174.objecter dump_active .. 0 homeless 2024-04-14T15:05:23.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:23.984+0000 7fb661ffb700 10 client.14174.objecter ms_handle_connect 0x7fb65403c340 2024-04-14T15:05:23.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:23.989+0000 7fb661ffb700 1 -- 172.21.15.90:0/2588275619 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fb664109030 con 0x7fb6641084b0 2024-04-14T15:05:24.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:24.196+0000 7fb66b035700 1 -- 172.21.15.90:0/2588275619 --> v1:172.21.15.90:6800/491071118 -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi092", "target": ["mon-mgr", ""]}) -- 0x7fb66410d130 con 0x7fb65403c340 2024-04-14T15:05:25.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:25 smithi090 ceph-mon[30399]: from='client.14174 v1:172.21.15.90:0/2588275619' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi092", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:05:25.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:25 smithi090 ceph-mon[30399]: Deploying cephadm binary to smithi092 2024-04-14T15:05:27.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:26 smithi090 ceph-mon[30399]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:05:28.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:28.564+0000 7fb661ffb700 1 -- 172.21.15.90:0/2588275619 <== mgr.14148 v1:172.21.15.90:6800/491071118 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (unknown 0 0 3343670069) 0x7fb66410d130 con 0x7fb65403c340 2024-04-14T15:05:28.565 INFO:teuthology.orchestra.run.smithi090.stdout:Added host 'smithi092' with addr '172.21.15.92' 2024-04-14T15:05:28.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:28.566+0000 7fb66b035700 10 librados: watch_flush enter 2024-04-14T15:05:28.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:28.566+0000 7fb66b035700 10 librados: watch_flush exit 2024-04-14T15:05:28.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:28.566+0000 7fb66b035700 20 client.14174.objecter shutdown clearing up homeless session... 2024-04-14T15:05:28.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:28.566+0000 7fb66b035700 10 client.14174.objecter successfully canceled tick 2024-04-14T15:05:28.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:28.566+0000 7fb66b035700 1 -- 172.21.15.90:0/2588275619 >> v1:172.21.15.90:6800/491071118 conn(0x7fb65403c340 legacy=0x7fb65403e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:05:28.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:28.566+0000 7fb66b035700 1 -- 172.21.15.90:0/2588275619 >> v1:172.21.15.90:6789/0 conn(0x7fb6641084b0 legacy=0x7fb664197520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:05:28.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:28.566+0000 7fb66b035700 1 -- 172.21.15.90:0/2588275619 shutdown_connections 2024-04-14T15:05:28.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:28.566+0000 7fb66b035700 1 -- 172.21.15.90:0/2588275619 >> 172.21.15.90:0/2588275619 conn(0x7fb6640fdf30 msgr2=0x7fb66410a470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:05:28.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:28.566+0000 7fb66b035700 1 -- 172.21.15.90:0/2588275619 shutdown_connections 2024-04-14T15:05:28.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:28.566+0000 7fb66b035700 1 -- 172.21.15.90:0/2588275619 wait complete. 2024-04-14T15:05:28.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:28.566+0000 7fb66b035700 1 librados: shutdown 2024-04-14T15:05:29.133 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:28 smithi090 ceph-mon[30399]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:05:29.133 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:28 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:29.133 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:28 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:05:29.134 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph orch host ls --format=json 2024-04-14T15:05:29.369 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:05:30.140 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:29 smithi090 ceph-mon[30399]: Added host smithi092 2024-04-14T15:05:30.140 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:29 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:30.140 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:29 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:30.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:30.871+0000 7f78e1e62700 1 -- 172.21.15.90:0/3744169625 >> v1:172.21.15.90:6789/0 conn(0x7f78dc103640 legacy=0x7f78dc103a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:05:30.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:30.871+0000 7f78e1e62700 1 -- 172.21.15.90:0/3744169625 shutdown_connections 2024-04-14T15:05:30.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:30.871+0000 7f78e1e62700 1 -- 172.21.15.90:0/3744169625 >> 172.21.15.90:0/3744169625 conn(0x7f78dc0fcdf0 msgr2=0x7f78dc0ff210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:05:30.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:30.871+0000 7f78e1e62700 1 -- 172.21.15.90:0/3744169625 shutdown_connections 2024-04-14T15:05:30.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:30.871+0000 7f78e1e62700 1 -- 172.21.15.90:0/3744169625 wait complete. 2024-04-14T15:05:30.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:30.871+0000 7f78e1e62700 1 librados: starting msgr at 2024-04-14T15:05:30.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:30.871+0000 7f78e1e62700 1 librados: starting objecter 2024-04-14T15:05:30.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:30.871+0000 7f78e1e62700 1 Processor -- start 2024-04-14T15:05:30.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:30.872+0000 7f78e1e62700 1 -- start start 2024-04-14T15:05:30.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:30.872+0000 7f78e1e62700 1 librados: setting wanted keys 2024-04-14T15:05:30.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:30.872+0000 7f78e1e62700 1 librados: calling monclient init 2024-04-14T15:05:30.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:30.872+0000 7f78e1e62700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f78dc103190 con 0x7f78dc103640 2024-04-14T15:05:30.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:30.872+0000 7f78db7fe700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f78dc103640 0x7f78dc1012a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:59458/0 (socket says 172.21.15.90:59458) 2024-04-14T15:05:30.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:30.872+0000 7f78db7fe700 1 -- 172.21.15.90:0/1206337090 learned_addr learned my addr 172.21.15.90:0/1206337090 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:05:30.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:30.872+0000 7f78d8ff9700 10 client.?.objecter ms_handle_connect 0x7f78dc103640 2024-04-14T15:05:30.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:30.872+0000 7f78d8ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:05:30.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:30.872+0000 7f78d8ff9700 1 -- 172.21.15.90:0/1206337090 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 204986167 0 0) 0x7f78dc103190 con 0x7f78dc103640 2024-04-14T15:05:30.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:30.872+0000 7f78d8ff9700 1 -- 172.21.15.90:0/1206337090 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78c40056e0 con 0x7f78dc103640 2024-04-14T15:05:30.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:30.872+0000 7f78d8ff9700 1 -- 172.21.15.90:0/1206337090 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2624289020 0 0) 0x7f78c40056e0 con 0x7f78dc103640 2024-04-14T15:05:30.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:30.872+0000 7f78d8ff9700 1 -- 172.21.15.90:0/1206337090 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f78dc103190 con 0x7f78dc103640 2024-04-14T15:05:30.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:30.872+0000 7f78d8ff9700 1 -- 172.21.15.90:0/1206337090 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f78cc001480 con 0x7f78dc103640 2024-04-14T15:05:30.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:30.873+0000 7f78d8ff9700 1 -- 172.21.15.90:0/1206337090 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2866714575 0 0) 0x7f78dc103190 con 0x7f78dc103640 2024-04-14T15:05:30.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:30.873+0000 7f78d8ff9700 1 -- 172.21.15.90:0/1206337090 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78dc1019b0 con 0x7f78dc103640 2024-04-14T15:05:30.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:30.873+0000 7f78e1e62700 1 -- 172.21.15.90:0/1206337090 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f78dc101ca0 con 0x7f78dc103640 2024-04-14T15:05:30.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:30.873+0000 7f78e1e62700 10 client.14176.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:05:30.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:30.873+0000 7f78e1e62700 1 -- 172.21.15.90:0/1206337090 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f78dc1a24f0 con 0x7f78dc103640 2024-04-14T15:05:30.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:30.873+0000 7f78e1e62700 1 librados: init done 2024-04-14T15:05:30.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:30.873+0000 7f78d8ff9700 1 -- 172.21.15.90:0/1206337090 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f78cc0036c0 con 0x7f78dc103640 2024-04-14T15:05:30.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:30.873+0000 7f78d8ff9700 1 -- 172.21.15.90:0/1206337090 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f78cc004d00 con 0x7f78dc103640 2024-04-14T15:05:30.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:30.873+0000 7f78d8ff9700 1 -- 172.21.15.90:0/1206337090 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 13) ==== 46579+0+0 (unknown 2972057686 0 0) 0x7f78cc004ec0 con 0x7f78dc103640 2024-04-14T15:05:30.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:30.873+0000 7f78e1e62700 1 -- 172.21.15.90:0/1206337090 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78c8005120 con 0x7f78dc103640 2024-04-14T15:05:30.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:30.874+0000 7f78d8ff9700 1 -- 172.21.15.90:0/1206337090 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(3..3 src has 1..3) ==== 1069+0+0 (unknown 1996007131 0 0) 0x7f78cc047fe0 con 0x7f78dc103640 2024-04-14T15:05:30.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:30.874+0000 7f78d8ff9700 10 client.14176.objecter ms_dispatch 0x7f78dc001550 osd_map(3..3 src has 1..3) 2024-04-14T15:05:30.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:30.874+0000 7f78d8ff9700 3 client.14176.objecter handle_osd_map got epochs [3,3] > 0 2024-04-14T15:05:30.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:30.874+0000 7f78d8ff9700 3 client.14176.objecter handle_osd_map decoding full epoch 3 2024-04-14T15:05:30.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:30.874+0000 7f78d8ff9700 20 client.14176.objecter dump_active .. 0 homeless 2024-04-14T15:05:30.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:30.874+0000 7f78d8ff9700 10 client.14176.objecter ms_handle_connect 0x7f78c403c340 2024-04-14T15:05:30.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:30.879+0000 7f78d8ff9700 1 -- 172.21.15.90:0/1206337090 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f78cc017660 con 0x7f78dc103640 2024-04-14T15:05:31.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:31.081+0000 7f78e1e62700 1 -- 172.21.15.90:0/1206337090 --> v1:172.21.15.90:6800/491071118 -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f78c8000b90 con 0x7f78c403c340 2024-04-14T15:05:31.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:31.082+0000 7f78d8ff9700 1 -- 172.21.15.90:0/1206337090 <== mgr.14148 v1:172.21.15.90:6800/491071118 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+159 (unknown 0 0 2425897416) 0x7f78c8000b90 con 0x7f78c403c340 2024-04-14T15:05:31.083 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:05:31.083 INFO:teuthology.orchestra.run.smithi090.stdout:[{"addr": "172.21.15.90", "hostname": "smithi090", "labels": [], "status": ""}, {"addr": "172.21.15.92", "hostname": "smithi092", "labels": [], "status": ""}] 2024-04-14T15:05:31.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:31.084+0000 7f78e1e62700 10 librados: watch_flush enter 2024-04-14T15:05:31.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:31.084+0000 7f78e1e62700 10 librados: watch_flush exit 2024-04-14T15:05:31.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:31.084+0000 7f78e1e62700 20 client.14176.objecter shutdown clearing up homeless session... 2024-04-14T15:05:31.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:31.084+0000 7f78e1e62700 10 client.14176.objecter successfully canceled tick 2024-04-14T15:05:31.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:31.084+0000 7f78e1e62700 1 -- 172.21.15.90:0/1206337090 >> v1:172.21.15.90:6800/491071118 conn(0x7f78c403c340 legacy=0x7f78c403e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:05:31.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:31.084+0000 7f78e1e62700 1 -- 172.21.15.90:0/1206337090 >> v1:172.21.15.90:6789/0 conn(0x7f78dc103640 legacy=0x7f78dc1012a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:05:31.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:31.084+0000 7f78e1e62700 1 -- 172.21.15.90:0/1206337090 shutdown_connections 2024-04-14T15:05:31.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:31.084+0000 7f78e1e62700 1 -- 172.21.15.90:0/1206337090 >> 172.21.15.90:0/1206337090 conn(0x7f78dc0fcdf0 msgr2=0x7f78dc108cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:05:31.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:31.084+0000 7f78e1e62700 1 -- 172.21.15.90:0/1206337090 shutdown_connections 2024-04-14T15:05:31.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:31.084+0000 7f78e1e62700 1 -- 172.21.15.90:0/1206337090 wait complete. 2024-04-14T15:05:31.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:31.084+0000 7f78e1e62700 1 librados: shutdown 2024-04-14T15:05:31.095 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:30 smithi090 ceph-mon[30399]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:05:31.736 INFO:tasks.cephadm:Setting crush tunables to jewel 2024-04-14T15:05:31.737 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd crush tunables jewel 2024-04-14T15:05:31.971 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:05:32.401 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:32 smithi090 ceph-mon[30399]: from='client.14176 v1:172.21.15.90:0/1206337090' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:05:32.402 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:32 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:32.402 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:32 smithi090 ceph-mon[30399]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:05:33.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:33.666+0000 7f55fa2e6700 1 -- 172.21.15.90:0/2925519091 >> v1:172.21.15.90:6789/0 conn(0x7f55f4108460 legacy=0x7f55f4108840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:05:33.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:33.666+0000 7f55fa2e6700 1 -- 172.21.15.90:0/2925519091 shutdown_connections 2024-04-14T15:05:33.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:33.666+0000 7f55fa2e6700 1 -- 172.21.15.90:0/2925519091 >> 172.21.15.90:0/2925519091 conn(0x7f55f40fdee0 msgr2=0x7f55f4100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:05:33.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:33.666+0000 7f55fa2e6700 1 -- 172.21.15.90:0/2925519091 shutdown_connections 2024-04-14T15:05:33.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:33.666+0000 7f55fa2e6700 1 -- 172.21.15.90:0/2925519091 wait complete. 2024-04-14T15:05:33.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:33.666+0000 7f55fa2e6700 1 librados: starting msgr at 2024-04-14T15:05:33.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:33.666+0000 7f55fa2e6700 1 librados: starting objecter 2024-04-14T15:05:33.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:33.666+0000 7f55fa2e6700 1 Processor -- start 2024-04-14T15:05:33.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:33.666+0000 7f55fa2e6700 1 -- start start 2024-04-14T15:05:33.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:33.666+0000 7f55fa2e6700 1 librados: setting wanted keys 2024-04-14T15:05:33.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:33.666+0000 7f55fa2e6700 1 librados: calling monclient init 2024-04-14T15:05:33.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:33.667+0000 7f55fa2e6700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f55f4197b00 con 0x7f55f4108460 2024-04-14T15:05:33.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:33.667+0000 7f55f3fff700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f55f4108460 0x7f55f41973f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:59488/0 (socket says 172.21.15.90:59488) 2024-04-14T15:05:33.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:33.667+0000 7f55f3fff700 1 -- 172.21.15.90:0/2166157961 learned_addr learned my addr 172.21.15.90:0/2166157961 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:05:33.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:33.667+0000 7f55f17fa700 10 client.?.objecter ms_handle_connect 0x7f55f4108460 2024-04-14T15:05:33.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:33.667+0000 7f55f17fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:05:33.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:33.667+0000 7f55f17fa700 1 -- 172.21.15.90:0/2166157961 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1087801946 0 0) 0x7f55f4197b00 con 0x7f55f4108460 2024-04-14T15:05:33.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:33.667+0000 7f55f17fa700 1 -- 172.21.15.90:0/2166157961 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f55dc0056e0 con 0x7f55f4108460 2024-04-14T15:05:33.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:33.667+0000 7f55f17fa700 1 -- 172.21.15.90:0/2166157961 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3942101417 0 0) 0x7f55dc0056e0 con 0x7f55f4108460 2024-04-14T15:05:33.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:33.667+0000 7f55f17fa700 1 -- 172.21.15.90:0/2166157961 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f55f4197b00 con 0x7f55f4108460 2024-04-14T15:05:33.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:33.667+0000 7f55f17fa700 1 -- 172.21.15.90:0/2166157961 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f55e40047f0 con 0x7f55f4108460 2024-04-14T15:05:33.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:33.667+0000 7f55f17fa700 1 -- 172.21.15.90:0/2166157961 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2949030437 0 0) 0x7f55f4197b00 con 0x7f55f4108460 2024-04-14T15:05:33.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:33.667+0000 7f55f17fa700 1 -- 172.21.15.90:0/2166157961 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55f4197cd0 con 0x7f55f4108460 2024-04-14T15:05:33.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:33.667+0000 7f55fa2e6700 1 -- 172.21.15.90:0/2166157961 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f55f4197fc0 con 0x7f55f4108460 2024-04-14T15:05:33.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:33.667+0000 7f55fa2e6700 10 client.14178.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:05:33.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:33.667+0000 7f55fa2e6700 1 -- 172.21.15.90:0/2166157961 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f55f41a0180 con 0x7f55f4108460 2024-04-14T15:05:33.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:33.667+0000 7f55fa2e6700 1 librados: init done 2024-04-14T15:05:33.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:33.668+0000 7f55f17fa700 1 -- 172.21.15.90:0/2166157961 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f55e40013e0 con 0x7f55f4108460 2024-04-14T15:05:33.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:33.668+0000 7f55f17fa700 1 -- 172.21.15.90:0/2166157961 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f55e4005f40 con 0x7f55f4108460 2024-04-14T15:05:33.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:33.668+0000 7f55f17fa700 1 -- 172.21.15.90:0/2166157961 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 13) ==== 46579+0+0 (unknown 2972057686 0 0) 0x7f55e40116e0 con 0x7f55f4108460 2024-04-14T15:05:33.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:33.668+0000 7f55fa2e6700 1 -- 172.21.15.90:0/2166157961 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55e0005120 con 0x7f55f4108460 2024-04-14T15:05:33.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:33.669+0000 7f55f17fa700 1 -- 172.21.15.90:0/2166157961 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(3..3 src has 1..3) ==== 1069+0+0 (unknown 1996007131 0 0) 0x7f55e4047f30 con 0x7f55f4108460 2024-04-14T15:05:33.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:33.669+0000 7f55f17fa700 10 client.14178.objecter ms_dispatch 0x7f55f4001550 osd_map(3..3 src has 1..3) 2024-04-14T15:05:33.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:33.669+0000 7f55f17fa700 3 client.14178.objecter handle_osd_map got epochs [3,3] > 0 2024-04-14T15:05:33.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:33.669+0000 7f55f17fa700 3 client.14178.objecter handle_osd_map decoding full epoch 3 2024-04-14T15:05:33.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:33.669+0000 7f55f17fa700 20 client.14178.objecter dump_active .. 0 homeless 2024-04-14T15:05:33.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:33.669+0000 7f55f17fa700 10 client.14178.objecter ms_handle_connect 0x7f55dc03c2f0 2024-04-14T15:05:33.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:33.673+0000 7f55f17fa700 1 -- 172.21.15.90:0/2166157961 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f55f4108fe0 con 0x7f55f4108460 2024-04-14T15:05:33.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:33.877+0000 7f55fa2e6700 1 -- 172.21.15.90:0/2166157961 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "jewel"} v 0) -- 0x7f55e0005410 con 0x7f55f4108460 2024-04-14T15:05:33.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:33.929+0000 7f55f17fa700 1 -- 172.21.15.90:0/2166157961 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "jewel"}]=0 adjusted tunables profile to jewel v4) ==== 120+0+0 (unknown 2099115640 0 0) 0x7f55e4017630 con 0x7f55f4108460 2024-04-14T15:05:33.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:33.931+0000 7f55fa2e6700 10 librados: watch_flush enter 2024-04-14T15:05:33.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:33.931+0000 7f55fa2e6700 10 librados: watch_flush exit 2024-04-14T15:05:33.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:33.931+0000 7f55fa2e6700 20 client.14178.objecter shutdown clearing up homeless session... 2024-04-14T15:05:33.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:33.931+0000 7f55fa2e6700 10 client.14178.objecter successfully canceled tick 2024-04-14T15:05:33.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:33.931+0000 7f55fa2e6700 1 -- 172.21.15.90:0/2166157961 >> v1:172.21.15.90:6800/491071118 conn(0x7f55dc03c2f0 legacy=0x7f55dc03e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:05:33.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:33.931+0000 7f55fa2e6700 1 -- 172.21.15.90:0/2166157961 >> v1:172.21.15.90:6789/0 conn(0x7f55f4108460 legacy=0x7f55f41973f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:05:33.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:33.931+0000 7f55fa2e6700 1 -- 172.21.15.90:0/2166157961 shutdown_connections 2024-04-14T15:05:33.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:33.931+0000 7f55fa2e6700 1 -- 172.21.15.90:0/2166157961 >> 172.21.15.90:0/2166157961 conn(0x7f55f40fdee0 msgr2=0x7f55f410a410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:05:33.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:33.931+0000 7f55fa2e6700 1 -- 172.21.15.90:0/2166157961 shutdown_connections 2024-04-14T15:05:33.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:33.931+0000 7f55fa2e6700 1 -- 172.21.15.90:0/2166157961 wait complete. 2024-04-14T15:05:33.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:33.931+0000 7f55fa2e6700 1 librados: shutdown 2024-04-14T15:05:33.932 INFO:teuthology.orchestra.run.smithi090.stderr:adjusted tunables profile to jewel 2024-04-14T15:05:34.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:33 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2166157961' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "jewel"}]: dispatch 2024-04-14T15:05:34.510 INFO:tasks.cephadm:Adding mon.a on smithi090 2024-04-14T15:05:34.511 INFO:tasks.cephadm:Adding mon.c on smithi090 2024-04-14T15:05:34.511 INFO:tasks.cephadm:Adding mon.b on smithi092 2024-04-14T15:05:34.511 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph orch apply mon '3;smithi090:[v1:172.21.15.90:6789]=a;smithi090:[v1:172.21.15.90:6790]=c;smithi092:[v1:172.21.15.92:6789]=b' 2024-04-14T15:05:34.748 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-14T15:05:34.795 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-14T15:05:35.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:34 smithi090 ceph-mon[30399]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:05:35.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:34 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2166157961' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "jewel"}]': finished 2024-04-14T15:05:35.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:34 smithi090 ceph-mon[30399]: osdmap e4: 0 total, 0 up, 0 in 2024-04-14T15:05:35.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:34 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:35.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:34 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:35.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:34 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:35.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:34 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:35.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:34 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi092", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:05:35.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:34 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:05:35.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:34 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:05:36.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:35 smithi090 ceph-mon[30399]: Updating smithi092:/etc/ceph/ceph.conf 2024-04-14T15:05:36.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:35 smithi090 ceph-mon[30399]: Updating smithi092:/var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T15:05:36.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:35 smithi090 ceph-mon[30399]: Updating smithi092:/etc/ceph/ceph.client.admin.keyring 2024-04-14T15:05:36.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:35 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:36.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:35 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:36.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:35 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:36.265 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:36.263+0000 7f8a3d803700 1 -- 172.21.15.92:0/4004092046 >> v1:172.21.15.90:6789/0 conn(0x7f8a381024a0 legacy=0x7f8a38102880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:05:36.265 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:36.263+0000 7f8a3d803700 1 -- 172.21.15.92:0/4004092046 shutdown_connections 2024-04-14T15:05:36.265 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:36.263+0000 7f8a3d803700 1 -- 172.21.15.92:0/4004092046 >> 172.21.15.92:0/4004092046 conn(0x7f8a380fdd70 msgr2=0x7f8a38100190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:05:36.266 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:36.263+0000 7f8a3d803700 1 -- 172.21.15.92:0/4004092046 shutdown_connections 2024-04-14T15:05:36.266 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:36.263+0000 7f8a3d803700 1 -- 172.21.15.92:0/4004092046 wait complete. 2024-04-14T15:05:36.266 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:36.264+0000 7f8a3d803700 1 librados: starting msgr at 2024-04-14T15:05:36.266 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:36.264+0000 7f8a3d803700 1 librados: starting objecter 2024-04-14T15:05:36.266 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:36.264+0000 7f8a3d803700 1 Processor -- start 2024-04-14T15:05:36.266 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:36.264+0000 7f8a3d803700 1 -- start start 2024-04-14T15:05:36.266 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:36.264+0000 7f8a3d803700 1 librados: setting wanted keys 2024-04-14T15:05:36.266 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:36.264+0000 7f8a3d803700 1 librados: calling monclient init 2024-04-14T15:05:36.266 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:36.264+0000 7f8a3d803700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f8a381977f0 con 0x7f8a381024a0 2024-04-14T15:05:36.266 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:36.264+0000 7f8a36ffd700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f8a381024a0 0x7f8a381970e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.92:33876/0 (socket says 172.21.15.92:33876) 2024-04-14T15:05:36.266 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:36.264+0000 7f8a36ffd700 1 -- 172.21.15.92:0/3063846388 learned_addr learned my addr 172.21.15.92:0/3063846388 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-04-14T15:05:36.266 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:36.265+0000 7f8a3c801700 10 client.?.objecter ms_handle_connect 0x7f8a381024a0 2024-04-14T15:05:36.267 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:36.265+0000 7f8a3c801700 10 client.?.objecter resend_mon_ops 2024-04-14T15:05:36.267 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:36.265+0000 7f8a3c801700 1 -- 172.21.15.92:0/3063846388 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 99357128 0 0) 0x7f8a381977f0 con 0x7f8a381024a0 2024-04-14T15:05:36.267 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:36.265+0000 7f8a3c801700 1 -- 172.21.15.92:0/3063846388 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8a200056e0 con 0x7f8a381024a0 2024-04-14T15:05:36.268 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:36.265+0000 7f8a3c801700 1 -- 172.21.15.92:0/3063846388 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4057124792 0 0) 0x7f8a200056e0 con 0x7f8a381024a0 2024-04-14T15:05:36.268 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:36.265+0000 7f8a3c801700 1 -- 172.21.15.92:0/3063846388 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8a381977f0 con 0x7f8a381024a0 2024-04-14T15:05:36.268 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:36.265+0000 7f8a3c801700 1 -- 172.21.15.92:0/3063846388 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f8a2c004750 con 0x7f8a381024a0 2024-04-14T15:05:36.268 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:36.265+0000 7f8a3c801700 1 -- 172.21.15.92:0/3063846388 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2470967271 0 0) 0x7f8a381977f0 con 0x7f8a381024a0 2024-04-14T15:05:36.268 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:36.265+0000 7f8a3c801700 1 -- 172.21.15.92:0/3063846388 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a381979c0 con 0x7f8a381024a0 2024-04-14T15:05:36.269 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:36.265+0000 7f8a3d803700 1 -- 172.21.15.92:0/3063846388 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8a38197cb0 con 0x7f8a381024a0 2024-04-14T15:05:36.269 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:36.265+0000 7f8a3d803700 10 client.14180.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:05:36.269 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:36.265+0000 7f8a3d803700 1 -- 172.21.15.92:0/3063846388 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f8a3819fe70 con 0x7f8a381024a0 2024-04-14T15:05:36.269 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:36.265+0000 7f8a3d803700 1 librados: init done 2024-04-14T15:05:36.269 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:36.265+0000 7f8a3c801700 1 -- 172.21.15.92:0/3063846388 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f8a2c003760 con 0x7f8a381024a0 2024-04-14T15:05:36.269 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:36.265+0000 7f8a3c801700 1 -- 172.21.15.92:0/3063846388 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f8a2c004e20 con 0x7f8a381024a0 2024-04-14T15:05:36.269 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:36.266+0000 7f8a3d803700 1 -- 172.21.15.92:0/3063846388 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a24005120 con 0x7f8a381024a0 2024-04-14T15:05:36.269 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:36.266+0000 7f8a3c801700 1 -- 172.21.15.92:0/3063846388 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 13) ==== 46579+0+0 (unknown 2972057686 0 0) 0x7f8a2c005060 con 0x7f8a381024a0 2024-04-14T15:05:36.269 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:36.267+0000 7f8a3c801700 1 -- 172.21.15.92:0/3063846388 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(4..4 src has 1..4) ==== 1069+0+0 (unknown 1732540447 0 0) 0x7f8a2c0480c0 con 0x7f8a381024a0 2024-04-14T15:05:36.269 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:36.267+0000 7f8a3c801700 10 client.14180.objecter ms_dispatch 0x7f8a38001550 osd_map(4..4 src has 1..4) 2024-04-14T15:05:36.269 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:36.267+0000 7f8a3c801700 3 client.14180.objecter handle_osd_map got epochs [4,4] > 0 2024-04-14T15:05:36.269 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:36.267+0000 7f8a3c801700 3 client.14180.objecter handle_osd_map decoding full epoch 4 2024-04-14T15:05:36.270 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:36.267+0000 7f8a3c801700 20 client.14180.objecter dump_active .. 0 homeless 2024-04-14T15:05:36.270 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:36.267+0000 7f8a3c801700 10 client.14180.objecter ms_handle_connect 0x7f8a2003c2a0 2024-04-14T15:05:36.272 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:36.272+0000 7f8a3c801700 1 -- 172.21.15.92:0/3063846388 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f8a2c017740 con 0x7f8a381024a0 2024-04-14T15:05:36.480 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:36.479+0000 7f8a3d803700 1 -- 172.21.15.92:0/3063846388 --> v1:172.21.15.90:6800/491071118 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi090:[v1:172.21.15.90:6789]=a;smithi090:[v1:172.21.15.90:6790]=c;smithi092:[v1:172.21.15.92:6789]=b", "target": ["mon-mgr", ""]}) -- 0x7f8a24000c60 con 0x7f8a2003c2a0 2024-04-14T15:05:36.486 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:36.486+0000 7f8a3c801700 1 -- 172.21.15.92:0/3063846388 <== mgr.14148 v1:172.21.15.90:6800/491071118 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (unknown 0 0 3265049985) 0x7f8a24000c60 con 0x7f8a2003c2a0 2024-04-14T15:05:36.486 INFO:teuthology.orchestra.run.smithi092.stdout:Scheduled mon update... 2024-04-14T15:05:36.488 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:36.487+0000 7f8a3d803700 10 librados: watch_flush enter 2024-04-14T15:05:36.489 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:36.488+0000 7f8a3d803700 10 librados: watch_flush exit 2024-04-14T15:05:36.489 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:36.488+0000 7f8a3d803700 20 client.14180.objecter shutdown clearing up homeless session... 2024-04-14T15:05:36.489 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:36.488+0000 7f8a3d803700 10 client.14180.objecter successfully canceled tick 2024-04-14T15:05:36.489 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:36.488+0000 7f8a3d803700 1 -- 172.21.15.92:0/3063846388 >> v1:172.21.15.90:6800/491071118 conn(0x7f8a2003c2a0 legacy=0x7f8a2003e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:05:36.489 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:36.488+0000 7f8a3d803700 1 -- 172.21.15.92:0/3063846388 >> v1:172.21.15.90:6789/0 conn(0x7f8a381024a0 legacy=0x7f8a381970e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:05:36.489 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:36.488+0000 7f8a3d803700 1 -- 172.21.15.92:0/3063846388 shutdown_connections 2024-04-14T15:05:36.489 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:36.488+0000 7f8a3d803700 1 -- 172.21.15.92:0/3063846388 >> 172.21.15.92:0/3063846388 conn(0x7f8a380fdd70 msgr2=0x7f8a38109410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:05:36.489 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:36.488+0000 7f8a3d803700 1 -- 172.21.15.92:0/3063846388 shutdown_connections 2024-04-14T15:05:36.489 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:36.488+0000 7f8a3d803700 1 -- 172.21.15.92:0/3063846388 wait complete. 2024-04-14T15:05:36.489 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:36.488+0000 7f8a3d803700 1 librados: shutdown 2024-04-14T15:05:37.067 DEBUG:teuthology.orchestra.run.smithi090:mon.c> sudo journalctl -f -n 0 -u ceph-0e288098-fa70-11ee-bc8e-c7b262605968@mon.c.service 2024-04-14T15:05:37.070 DEBUG:teuthology.orchestra.run.smithi092:mon.b> sudo journalctl -f -n 0 -u ceph-0e288098-fa70-11ee-bc8e-c7b262605968@mon.b.service 2024-04-14T15:05:37.072 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-14T15:05:37.073 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph mon dump -f json 2024-04-14T15:05:37.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:36 smithi090 ceph-mon[30399]: Updating smithi092:/var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T15:05:37.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:36 smithi090 ceph-mon[30399]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:05:37.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:36 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:37.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:36 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:05:37.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:36 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:05:37.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:36 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:05:37.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:36 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:37.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:36 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T15:05:37.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:36 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:05:37.755 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T15:05:38.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:37 smithi090 ceph-mon[30399]: from='client.14180 v1:172.21.15.92:0/3063846388' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi090:[v1:172.21.15.90:6789]=a;smithi090:[v1:172.21.15.90:6790]=c;smithi092:[v1:172.21.15.92:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:05:38.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:37 smithi090 ceph-mon[30399]: Saving service mon spec with placement smithi090:[v1:172.21.15.90:6789]=a;smithi090:[v1:172.21.15.90:6790]=c;smithi092:[v1:172.21.15.92:6789]=b;count:3 2024-04-14T15:05:38.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:37 smithi090 ceph-mon[30399]: Deploying daemon mon.b on smithi092 2024-04-14T15:05:39.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:38 smithi090 ceph-mon[30399]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:05:40.638 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:40.635+0000 7f78c8853700 1 -- 172.21.15.92:0/1280950769 >> v1:172.21.15.90:6789/0 conn(0x7f78c41062c0 legacy=0x7f78c41066a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:05:40.638 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:40.635+0000 7f78c8853700 1 -- 172.21.15.92:0/1280950769 shutdown_connections 2024-04-14T15:05:40.639 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:40.635+0000 7f78c8853700 1 -- 172.21.15.92:0/1280950769 >> 172.21.15.92:0/1280950769 conn(0x7f78c40fc520 msgr2=0x7f78c40fe940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:05:40.639 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:40.635+0000 7f78c8853700 1 -- 172.21.15.92:0/1280950769 shutdown_connections 2024-04-14T15:05:40.639 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:40.635+0000 7f78c8853700 1 -- 172.21.15.92:0/1280950769 wait complete. 2024-04-14T15:05:40.639 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:40.636+0000 7f78c8853700 1 librados: starting msgr at 2024-04-14T15:05:40.639 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:40.636+0000 7f78c8853700 1 librados: starting objecter 2024-04-14T15:05:40.639 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:40.636+0000 7f78c8853700 1 Processor -- start 2024-04-14T15:05:40.639 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:40.636+0000 7f78c8853700 1 -- start start 2024-04-14T15:05:40.639 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:40.636+0000 7f78c8853700 1 librados: setting wanted keys 2024-04-14T15:05:40.639 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:40.636+0000 7f78c8853700 1 librados: calling monclient init 2024-04-14T15:05:40.639 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:40.636+0000 7f78c8853700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f78c40fec30 con 0x7f78c41062c0 2024-04-14T15:05:40.639 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:40.636+0000 7f78c259c700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f78c41062c0 0x7f78c40fcde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.92:50768/0 (socket says 172.21.15.92:50768) 2024-04-14T15:05:40.639 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:40.636+0000 7f78c259c700 1 -- 172.21.15.92:0/1654883679 learned_addr learned my addr 172.21.15.92:0/1654883679 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-04-14T15:05:40.639 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:40.636+0000 7f78b37fe700 10 client.?.objecter ms_handle_connect 0x7f78c41062c0 2024-04-14T15:05:40.639 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:40.636+0000 7f78b37fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:05:40.640 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:40.637+0000 7f78b37fe700 1 -- 172.21.15.92:0/1654883679 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2458399849 0 0) 0x7f78c40fec30 con 0x7f78c41062c0 2024-04-14T15:05:40.640 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:40.637+0000 7f78b37fe700 1 -- 172.21.15.92:0/1654883679 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78ac0056e0 con 0x7f78c41062c0 2024-04-14T15:05:40.640 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:40.637+0000 7f78b37fe700 1 -- 172.21.15.92:0/1654883679 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 903307583 0 0) 0x7f78ac0056e0 con 0x7f78c41062c0 2024-04-14T15:05:40.640 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:40.637+0000 7f78b37fe700 1 -- 172.21.15.92:0/1654883679 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f78c40fec30 con 0x7f78c41062c0 2024-04-14T15:05:40.640 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:40.637+0000 7f78b37fe700 1 -- 172.21.15.92:0/1654883679 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f78b4001480 con 0x7f78c41062c0 2024-04-14T15:05:40.640 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:40.637+0000 7f78b37fe700 1 -- 172.21.15.92:0/1654883679 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1705403197 0 0) 0x7f78c40fec30 con 0x7f78c41062c0 2024-04-14T15:05:40.640 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:40.637+0000 7f78b37fe700 1 -- 172.21.15.92:0/1654883679 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78c40fd4f0 con 0x7f78c41062c0 2024-04-14T15:05:40.640 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:40.637+0000 7f78c8853700 1 -- 172.21.15.92:0/1654883679 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f78c40fd7e0 con 0x7f78c41062c0 2024-04-14T15:05:40.640 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:40.637+0000 7f78c8853700 10 client.14182.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:05:40.640 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:40.637+0000 7f78c8853700 1 -- 172.21.15.92:0/1654883679 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f78c41a2560 con 0x7f78c41062c0 2024-04-14T15:05:40.640 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:40.637+0000 7f78c8853700 1 librados: init done 2024-04-14T15:05:40.640 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:40.637+0000 7f78b37fe700 1 -- 172.21.15.92:0/1654883679 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f78b40036c0 con 0x7f78c41062c0 2024-04-14T15:05:40.640 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:40.637+0000 7f78b37fe700 1 -- 172.21.15.92:0/1654883679 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2895163438 0 0) 0x7f78b4004d00 con 0x7f78c41062c0 2024-04-14T15:05:40.640 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:40.638+0000 7f78b37fe700 1 -- 172.21.15.92:0/1654883679 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 13) ==== 46579+0+0 (unknown 2972057686 0 0) 0x7f78b4004ec0 con 0x7f78c41062c0 2024-04-14T15:05:40.640 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:40.638+0000 7f78b37fe700 1 -- 172.21.15.92:0/1654883679 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(4..4 src has 1..4) ==== 1069+0+0 (unknown 1732540447 0 0) 0x7f78b4048df0 con 0x7f78c41062c0 2024-04-14T15:05:40.640 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:40.638+0000 7f78b37fe700 10 client.14182.objecter ms_dispatch 0x7f78c4001550 osd_map(4..4 src has 1..4) 2024-04-14T15:05:40.641 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:40.638+0000 7f78b37fe700 3 client.14182.objecter handle_osd_map got epochs [4,4] > 0 2024-04-14T15:05:40.641 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:40.638+0000 7f78b37fe700 3 client.14182.objecter handle_osd_map decoding full epoch 4 2024-04-14T15:05:40.641 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:40.638+0000 7f78b37fe700 20 client.14182.objecter dump_active .. 0 homeless 2024-04-14T15:05:40.641 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:40.638+0000 7f78c8853700 1 -- 172.21.15.92:0/1654883679 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78a4005120 con 0x7f78c41062c0 2024-04-14T15:05:40.641 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:40.639+0000 7f78b37fe700 10 client.14182.objecter ms_handle_connect 0x7f78ac03c340 2024-04-14T15:05:40.643 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:40.644+0000 7f78b37fe700 1 -- 172.21.15.92:0/1654883679 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f78b4017660 con 0x7f78c41062c0 2024-04-14T15:05:40.962 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:40.961+0000 7f78c8853700 1 -- 172.21.15.92:0/1654883679 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f78a4005410 con 0x7f78c41062c0 2024-04-14T15:05:40.962 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:40.962+0000 7f78b37fe700 1 -- 172.21.15.92:0/1654883679 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+694 (unknown 2237029548 0 873796233) 0x7f78b4016f60 con 0x7f78c41062c0 2024-04-14T15:05:40.962 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:05:40.962 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":1,"fsid":"0e288098-fa70-11ee-bc8e-c7b262605968","modified":"2024-04-14T15:04:02.905926Z","created":"2024-04-14T15:04:02.905926Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-14T15:05:40.964 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:40.964+0000 7f78c8853700 10 librados: watch_flush enter 2024-04-14T15:05:40.964 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:40.964+0000 7f78c8853700 10 librados: watch_flush exit 2024-04-14T15:05:40.964 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:40.964+0000 7f78c8853700 20 client.14182.objecter shutdown clearing up homeless session... 2024-04-14T15:05:40.964 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:40.964+0000 7f78c8853700 10 client.14182.objecter successfully canceled tick 2024-04-14T15:05:40.964 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:40.964+0000 7f78c8853700 1 -- 172.21.15.92:0/1654883679 >> v1:172.21.15.90:6800/491071118 conn(0x7f78ac03c340 legacy=0x7f78ac03e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:05:40.964 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:40.964+0000 7f78c8853700 1 -- 172.21.15.92:0/1654883679 >> v1:172.21.15.90:6789/0 conn(0x7f78c41062c0 legacy=0x7f78c40fcde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:05:40.964 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:40.964+0000 7f78c8853700 1 -- 172.21.15.92:0/1654883679 shutdown_connections 2024-04-14T15:05:40.964 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:40.964+0000 7f78c8853700 1 -- 172.21.15.92:0/1654883679 >> 172.21.15.92:0/1654883679 conn(0x7f78c40fc520 msgr2=0x7f78c4100a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:05:40.964 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:40.964+0000 7f78c8853700 1 -- 172.21.15.92:0/1654883679 shutdown_connections 2024-04-14T15:05:40.965 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:40.964+0000 7f78c8853700 1 -- 172.21.15.92:0/1654883679 wait complete. 2024-04-14T15:05:40.965 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:40.964+0000 7f78c8853700 1 librados: shutdown 2024-04-14T15:05:40.965 INFO:teuthology.orchestra.run.smithi092.stderr:dumped monmap epoch 1 2024-04-14T15:05:41.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:40 smithi090 ceph-mon[30399]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:05:41.793 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 podman[33927]: 2024-04-14 15:05:41.611801514 +0000 UTC m=+0.017641614 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608 2024-04-14T15:05:42.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 podman[33927]: 2024-04-14 15:05:41.877946919 +0000 UTC m=+0.283787005 container init 20e24def0d377ff86771074a51111996756791ca0d91c9254ac1aef978b70d70 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mon-b, ceph=True, RELEASE=main-a971fd5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-14T15:05:42.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 podman[33927]: 2024-04-14 15:05:41.883274557 +0000 UTC m=+0.289114653 container start 20e24def0d377ff86771074a51111996756791ca0d91c9254ac1aef978b70d70 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mon-b, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=main-a971fd5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-14T15:05:42.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: set uid:gid to 167:167 (ceph:ceph) 2024-04-14T15:05:42.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev), process ceph-mon, pid 7 2024-04-14T15:05:42.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: pidfile_write: ignore empty --pid-file 2024-04-14T15:05:42.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: load: jerasure load: lrc 2024-04-14T15:05:42.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: RocksDB version: 7.9.2 2024-04-14T15:05:42.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Git sha 0 2024-04-14T15:05:42.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Compile date 2024-04-06 20:28:24 2024-04-14T15:05:42.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: DB SUMMARY 2024-04-14T15:05:42.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: DB Session ID: XY83VFDDN66NMIHNHHYB 2024-04-14T15:05:42.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: CURRENT file: CURRENT 2024-04-14T15:05:42.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: IDENTITY file: IDENTITY 2024-04-14T15:05:42.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-14T15:05:42.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-04-14T15:05:42.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 476 ; 2024-04-14T15:05:42.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.error_if_exists: 0 2024-04-14T15:05:42.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.create_if_missing: 0 2024-04-14T15:05:42.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.paranoid_checks: 1 2024-04-14T15:05:42.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-14T15:05:42.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-14T15:05:42.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-14T15:05:42.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.env: 0x55d00f89fc60 2024-04-14T15:05:42.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.fs: PosixFileSystem 2024-04-14T15:05:42.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.info_log: 0x55d0114f7c60 2024-04-14T15:05:42.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.max_file_opening_threads: 16 2024-04-14T15:05:42.376 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.statistics: (nil) 2024-04-14T15:05:42.376 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.use_fsync: 0 2024-04-14T15:05:42.376 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.max_log_file_size: 0 2024-04-14T15:05:42.376 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-14T15:05:42.376 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-14T15:05:42.376 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.keep_log_file_num: 1000 2024-04-14T15:05:42.376 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.recycle_log_file_num: 0 2024-04-14T15:05:42.376 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.allow_fallocate: 1 2024-04-14T15:05:42.376 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.allow_mmap_reads: 0 2024-04-14T15:05:42.376 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.allow_mmap_writes: 0 2024-04-14T15:05:42.376 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.use_direct_reads: 0 2024-04-14T15:05:42.376 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-14T15:05:42.376 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.create_missing_column_families: 0 2024-04-14T15:05:42.377 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.db_log_dir: 2024-04-14T15:05:42.377 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.wal_dir: 2024-04-14T15:05:42.377 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-14T15:05:42.377 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-14T15:05:42.377 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-14T15:05:42.377 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-14T15:05:42.377 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-14T15:05:42.377 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-14T15:05:42.377 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.advise_random_on_open: 1 2024-04-14T15:05:42.377 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.db_write_buffer_size: 0 2024-04-14T15:05:42.377 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.write_buffer_manager: 0x55d0114c81e0 2024-04-14T15:05:42.377 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-14T15:05:42.377 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-14T15:05:42.377 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-14T15:05:42.377 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.rate_limiter: (nil) 2024-04-14T15:05:42.378 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-14T15:05:42.378 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.wal_recovery_mode: 2 2024-04-14T15:05:42.378 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.enable_thread_tracking: 0 2024-04-14T15:05:42.378 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.enable_pipelined_write: 0 2024-04-14T15:05:42.378 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.unordered_write: 0 2024-04-14T15:05:42.378 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-14T15:05:42.378 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-14T15:05:42.378 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-14T15:05:42.378 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-14T15:05:42.378 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.row_cache: None 2024-04-14T15:05:42.378 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.wal_filter: None 2024-04-14T15:05:42.378 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-14T15:05:42.378 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.allow_ingest_behind: 0 2024-04-14T15:05:42.378 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.two_write_queues: 0 2024-04-14T15:05:42.378 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.manual_wal_flush: 0 2024-04-14T15:05:42.379 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.wal_compression: 0 2024-04-14T15:05:42.379 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.atomic_flush: 0 2024-04-14T15:05:42.379 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-14T15:05:42.379 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-14T15:05:42.379 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-14T15:05:42.379 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.log_readahead_size: 0 2024-04-14T15:05:42.379 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-14T15:05:42.379 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.best_efforts_recovery: 0 2024-04-14T15:05:42.379 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-14T15:05:42.379 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-14T15:05:42.379 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.allow_data_in_errors: 0 2024-04-14T15:05:42.379 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.db_host_id: __hostname__ 2024-04-14T15:05:42.379 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-14T15:05:42.379 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.max_background_jobs: 2 2024-04-14T15:05:42.379 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.max_background_compactions: -1 2024-04-14T15:05:42.379 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.max_subcompactions: 1 2024-04-14T15:05:42.380 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-14T15:05:42.380 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-14T15:05:42.380 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-14T15:05:42.380 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.max_total_wal_size: 0 2024-04-14T15:05:42.380 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-14T15:05:42.380 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-14T15:05:42.380 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-14T15:05:42.380 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-14T15:05:42.380 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.max_open_files: -1 2024-04-14T15:05:42.380 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.bytes_per_sync: 0 2024-04-14T15:05:42.380 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-14T15:05:42.380 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-14T15:05:42.380 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.compaction_readahead_size: 0 2024-04-14T15:05:42.380 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.max_background_flushes: -1 2024-04-14T15:05:42.380 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Compression algorithms supported: 2024-04-14T15:05:42.380 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-14T15:05:42.381 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: kZSTD supported: 0 2024-04-14T15:05:42.381 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: kXpressCompression supported: 0 2024-04-14T15:05:42.381 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: kLZ4HCCompression supported: 1 2024-04-14T15:05:42.381 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: kZlibCompression supported: 1 2024-04-14T15:05:42.381 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: kSnappyCompression supported: 1 2024-04-14T15:05:42.381 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: kLZ4Compression supported: 1 2024-04-14T15:05:42.381 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: kBZip2Compression supported: 0 2024-04-14T15:05:42.381 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-14T15:05:42.381 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-14T15:05:42.381 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-04-14T15:05:42.381 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-14T15:05:42.381 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-14T15:05:42.381 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.merge_operator: 2024-04-14T15:05:42.381 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.compaction_filter: None 2024-04-14T15:05:42.382 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.compaction_filter_factory: None 2024-04-14T15:05:42.382 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.sst_partitioner_factory: None 2024-04-14T15:05:42.382 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-14T15:05:42.382 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-14T15:05:42.382 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55d0114f7e20) 2024-04-14T15:05:42.382 INFO:journalctl@ceph.mon.b.smithi092.stdout: cache_index_and_filter_blocks: 1 2024-04-14T15:05:42.382 INFO:journalctl@ceph.mon.b.smithi092.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-14T15:05:42.382 INFO:journalctl@ceph.mon.b.smithi092.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-14T15:05:42.382 INFO:journalctl@ceph.mon.b.smithi092.stdout: pin_top_level_index_and_filter: 1 2024-04-14T15:05:42.382 INFO:journalctl@ceph.mon.b.smithi092.stdout: index_type: 0 2024-04-14T15:05:42.382 INFO:journalctl@ceph.mon.b.smithi092.stdout: data_block_index_type: 0 2024-04-14T15:05:42.382 INFO:journalctl@ceph.mon.b.smithi092.stdout: index_shortening: 1 2024-04-14T15:05:42.382 INFO:journalctl@ceph.mon.b.smithi092.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-14T15:05:42.382 INFO:journalctl@ceph.mon.b.smithi092.stdout: checksum: 4 2024-04-14T15:05:42.382 INFO:journalctl@ceph.mon.b.smithi092.stdout: no_block_cache: 0 2024-04-14T15:05:42.383 INFO:journalctl@ceph.mon.b.smithi092.stdout: block_cache: 0x55d0114fb090 2024-04-14T15:05:42.383 INFO:journalctl@ceph.mon.b.smithi092.stdout: block_cache_name: BinnedLRUCache 2024-04-14T15:05:42.383 INFO:journalctl@ceph.mon.b.smithi092.stdout: block_cache_options: 2024-04-14T15:05:42.383 INFO:journalctl@ceph.mon.b.smithi092.stdout: capacity : 536870912 2024-04-14T15:05:42.383 INFO:journalctl@ceph.mon.b.smithi092.stdout: num_shard_bits : 4 2024-04-14T15:05:42.383 INFO:journalctl@ceph.mon.b.smithi092.stdout: strict_capacity_limit : 0 2024-04-14T15:05:42.383 INFO:journalctl@ceph.mon.b.smithi092.stdout: high_pri_pool_ratio: 0.000 2024-04-14T15:05:42.383 INFO:journalctl@ceph.mon.b.smithi092.stdout: block_cache_compressed: (nil) 2024-04-14T15:05:42.383 INFO:journalctl@ceph.mon.b.smithi092.stdout: persistent_cache: (nil) 2024-04-14T15:05:42.383 INFO:journalctl@ceph.mon.b.smithi092.stdout: block_size: 4096 2024-04-14T15:05:42.383 INFO:journalctl@ceph.mon.b.smithi092.stdout: block_size_deviation: 10 2024-04-14T15:05:42.383 INFO:journalctl@ceph.mon.b.smithi092.stdout: block_restart_interval: 16 2024-04-14T15:05:42.383 INFO:journalctl@ceph.mon.b.smithi092.stdout: index_block_restart_interval: 1 2024-04-14T15:05:42.383 INFO:journalctl@ceph.mon.b.smithi092.stdout: metadata_block_size: 4096 2024-04-14T15:05:42.383 INFO:journalctl@ceph.mon.b.smithi092.stdout: partition_filters: 0 2024-04-14T15:05:42.383 INFO:journalctl@ceph.mon.b.smithi092.stdout: use_delta_encoding: 1 2024-04-14T15:05:42.384 INFO:journalctl@ceph.mon.b.smithi092.stdout: filter_policy: bloomfilter 2024-04-14T15:05:42.384 INFO:journalctl@ceph.mon.b.smithi092.stdout: whole_key_filtering: 1 2024-04-14T15:05:42.384 INFO:journalctl@ceph.mon.b.smithi092.stdout: verify_compression: 0 2024-04-14T15:05:42.384 INFO:journalctl@ceph.mon.b.smithi092.stdout: read_amp_bytes_per_bit: 0 2024-04-14T15:05:42.384 INFO:journalctl@ceph.mon.b.smithi092.stdout: format_version: 5 2024-04-14T15:05:42.384 INFO:journalctl@ceph.mon.b.smithi092.stdout: enable_index_compression: 1 2024-04-14T15:05:42.384 INFO:journalctl@ceph.mon.b.smithi092.stdout: block_align: 0 2024-04-14T15:05:42.384 INFO:journalctl@ceph.mon.b.smithi092.stdout: max_auto_readahead_size: 262144 2024-04-14T15:05:42.384 INFO:journalctl@ceph.mon.b.smithi092.stdout: prepopulate_block_cache: 0 2024-04-14T15:05:42.384 INFO:journalctl@ceph.mon.b.smithi092.stdout: initial_auto_readahead_size: 8192 2024-04-14T15:05:42.384 INFO:journalctl@ceph.mon.b.smithi092.stdout: num_file_reads_for_auto_readahead: 2 2024-04-14T15:05:42.384 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.write_buffer_size: 33554432 2024-04-14T15:05:42.384 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.max_write_buffer_number: 2 2024-04-14T15:05:42.384 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.compression: NoCompression 2024-04-14T15:05:42.384 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.bottommost_compression: Disabled 2024-04-14T15:05:42.385 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.prefix_extractor: nullptr 2024-04-14T15:05:42.385 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-14T15:05:42.385 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.num_levels: 7 2024-04-14T15:05:42.385 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-14T15:05:42.385 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-14T15:05:42.385 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-14T15:05:42.385 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-14T15:05:42.385 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-14T15:05:42.385 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-14T15:05:42.385 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-14T15:05:42.385 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-14T15:05:42.385 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-14T15:05:42.385 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-14T15:05:42.385 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-14T15:05:42.385 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-14T15:05:42.385 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-14T15:05:42.385 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.compression_opts.level: 32767 2024-04-14T15:05:42.386 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.compression_opts.strategy: 0 2024-04-14T15:05:42.386 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-14T15:05:42.386 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-14T15:05:42.386 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-14T15:05:42.386 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-14T15:05:42.386 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.compression_opts.enabled: false 2024-04-14T15:05:42.386 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-14T15:05:42.386 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-14T15:05:42.386 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-14T15:05:42.386 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-14T15:05:42.386 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.target_file_size_base: 67108864 2024-04-14T15:05:42.386 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-14T15:05:42.386 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-14T15:05:42.386 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-14T15:05:42.386 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-14T15:05:42.386 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-14T15:05:42.387 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-14T15:05:42.387 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-14T15:05:42.387 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-14T15:05:42.387 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-14T15:05:42.387 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-14T15:05:42.387 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-14T15:05:42.387 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-14T15:05:42.387 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-14T15:05:42.387 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-14T15:05:42.387 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.arena_block_size: 1048576 2024-04-14T15:05:42.387 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-14T15:05:42.387 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-14T15:05:42.387 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.disable_auto_compactions: 0 2024-04-14T15:05:42.387 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-14T15:05:42.387 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-14T15:05:42.388 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-14T15:05:42.388 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-14T15:05:42.388 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-14T15:05:42.388 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-14T15:05:42.388 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-14T15:05:42.388 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-14T15:05:42.388 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-14T15:05:42.388 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-14T15:05:42.388 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-14T15:05:42.388 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.inplace_update_support: 0 2024-04-14T15:05:42.388 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-14T15:05:42.388 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-14T15:05:42.388 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-14T15:05:42.388 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-14T15:05:42.388 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.bloom_locality: 0 2024-04-14T15:05:42.388 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.max_successive_merges: 0 2024-04-14T15:05:42.388 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-14T15:05:42.389 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.paranoid_file_checks: 0 2024-04-14T15:05:42.389 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.force_consistency_checks: 1 2024-04-14T15:05:42.389 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.report_bg_io_stats: 0 2024-04-14T15:05:42.389 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.ttl: 2592000 2024-04-14T15:05:42.389 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-14T15:05:42.389 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-14T15:05:42.389 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-14T15:05:42.389 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.enable_blob_files: false 2024-04-14T15:05:42.389 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.min_blob_size: 0 2024-04-14T15:05:42.389 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.blob_file_size: 268435456 2024-04-14T15:05:42.389 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-14T15:05:42.389 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-14T15:05:42.389 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-14T15:05:42.389 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-14T15:05:42.389 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-14T15:05:42.389 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.blob_file_starting_level: 0 2024-04-14T15:05:42.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-14T15:05:42.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-14T15:05:42.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-14T15:05:42.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 1cacc95d-132a-4f97-8c72-a0937e4ed6d8 2024-04-14T15:05:42.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713107141913205, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-14T15:05:42.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-14T15:05:42.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713107141913619, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1613, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 488, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 366, "raw_average_value_size": 73, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713107141, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "1cacc95d-132a-4f97-8c72-a0937e4ed6d8", "db_session_id": "XY83VFDDN66NMIHNHHYB", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-14T15:05:42.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713107141913700, "job": 1, "event": "recovery_finished"} 2024-04-14T15:05:42.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-14T15:05:42.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-14T15:05:42.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55d0115ea000 2024-04-14T15:05:42.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: DB pointer 0x55d0115d4000 2024-04-14T15:05:42.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-14T15:05:42.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-14T15:05:42.391 INFO:journalctl@ceph.mon.b.smithi092.stdout: ** DB Stats ** 2024-04-14T15:05:42.391 INFO:journalctl@ceph.mon.b.smithi092.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-14T15:05:42.391 INFO:journalctl@ceph.mon.b.smithi092.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-14T15:05:42.391 INFO:journalctl@ceph.mon.b.smithi092.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-14T15:05:42.391 INFO:journalctl@ceph.mon.b.smithi092.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-14T15:05:42.391 INFO:journalctl@ceph.mon.b.smithi092.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-14T15:05:42.391 INFO:journalctl@ceph.mon.b.smithi092.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-14T15:05:42.391 INFO:journalctl@ceph.mon.b.smithi092.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-14T15:05:42.391 INFO:journalctl@ceph.mon.b.smithi092.stdout: 2024-04-14T15:05:42.391 INFO:journalctl@ceph.mon.b.smithi092.stdout: ** Compaction Stats [default] ** 2024-04-14T15:05:42.391 INFO:journalctl@ceph.mon.b.smithi092.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-14T15:05:42.391 INFO:journalctl@ceph.mon.b.smithi092.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-14T15:05:42.391 INFO:journalctl@ceph.mon.b.smithi092.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-14T15:05:42.391 INFO:journalctl@ceph.mon.b.smithi092.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-14T15:05:42.392 INFO:journalctl@ceph.mon.b.smithi092.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-14T15:05:42.392 INFO:journalctl@ceph.mon.b.smithi092.stdout: 2024-04-14T15:05:42.392 INFO:journalctl@ceph.mon.b.smithi092.stdout: ** Compaction Stats [default] ** 2024-04-14T15:05:42.392 INFO:journalctl@ceph.mon.b.smithi092.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-14T15:05:42.392 INFO:journalctl@ceph.mon.b.smithi092.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-14T15:05:42.392 INFO:journalctl@ceph.mon.b.smithi092.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-14T15:05:42.392 INFO:journalctl@ceph.mon.b.smithi092.stdout: 2024-04-14T15:05:42.392 INFO:journalctl@ceph.mon.b.smithi092.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-14T15:05:42.392 INFO:journalctl@ceph.mon.b.smithi092.stdout: 2024-04-14T15:05:42.392 INFO:journalctl@ceph.mon.b.smithi092.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-14T15:05:42.392 INFO:journalctl@ceph.mon.b.smithi092.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-14T15:05:42.392 INFO:journalctl@ceph.mon.b.smithi092.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-14T15:05:42.392 INFO:journalctl@ceph.mon.b.smithi092.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-14T15:05:42.392 INFO:journalctl@ceph.mon.b.smithi092.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-14T15:05:42.393 INFO:journalctl@ceph.mon.b.smithi092.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-14T15:05:42.393 INFO:journalctl@ceph.mon.b.smithi092.stdout: Cumulative compaction: 0.00 GB write, 0.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-14T15:05:42.393 INFO:journalctl@ceph.mon.b.smithi092.stdout: Interval compaction: 0.00 GB write, 0.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-14T15:05:42.393 INFO:journalctl@ceph.mon.b.smithi092.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-14T15:05:42.393 INFO:journalctl@ceph.mon.b.smithi092.stdout: Block cache BinnedLRUCache@0x55d0114fb090#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-14T15:05:42.393 INFO:journalctl@ceph.mon.b.smithi092.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-14T15:05:42.393 INFO:journalctl@ceph.mon.b.smithi092.stdout: 2024-04-14T15:05:42.393 INFO:journalctl@ceph.mon.b.smithi092.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-14T15:05:42.393 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-04-14T15:05:42.393 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: using public_addrv v1:172.21.15.92:6789/0 2024-04-14T15:05:42.393 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: starting mon.b rank -1 at public addrs v1:172.21.15.92:6789/0 at bind addrs v1:172.21.15.92:6789/0 mon_data /var/lib/ceph/mon/ceph-b fsid 0e288098-fa70-11ee-bc8e-c7b262605968 2024-04-14T15:05:42.393 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: mon.b@-1(???) e0 preinit fsid 0e288098-fa70-11ee-bc8e-c7b262605968 2024-04-14T15:05:42.393 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: mon.b@-1(synchronizing).mds e1 new map 2024-04-14T15:05:42.393 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: mon.b@-1(synchronizing).mds e1 print_map 2024-04-14T15:05:42.393 INFO:journalctl@ceph.mon.b.smithi092.stdout: e1 2024-04-14T15:05:42.393 INFO:journalctl@ceph.mon.b.smithi092.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-14T15:05:42.393 INFO:journalctl@ceph.mon.b.smithi092.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-14T15:05:42.394 INFO:journalctl@ceph.mon.b.smithi092.stdout: legacy client fscid: -1 2024-04-14T15:05:42.394 INFO:journalctl@ceph.mon.b.smithi092.stdout: 2024-04-14T15:05:42.394 INFO:journalctl@ceph.mon.b.smithi092.stdout: No filesystems configured 2024-04-14T15:05:42.394 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-14T15:05:42.394 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-14T15:05:42.394 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-14T15:05:42.394 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-14T15:05:42.394 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-14T15:05:42.394 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-14T15:05:42.394 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-14T15:05:42.394 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-14T15:05:42.394 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-14T15:05:42.394 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-14T15:05:42.394 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:05:42.394 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2166157961' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "jewel"}]': finished 2024-04-14T15:05:42.394 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: osdmap e4: 0 total, 0 up, 0 in 2024-04-14T15:05:42.395 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:42.395 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:42.395 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:42.395 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:42.395 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi092", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:05:42.395 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:05:42.395 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:05:42.395 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: Updating smithi092:/etc/ceph/ceph.conf 2024-04-14T15:05:42.395 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: Updating smithi092:/var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T15:05:42.395 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: Updating smithi092:/etc/ceph/ceph.client.admin.keyring 2024-04-14T15:05:42.395 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:42.395 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:42.395 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:42.395 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: Updating smithi092:/var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T15:05:42.395 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:05:42.396 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:42.396 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:05:42.396 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:05:42.396 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:05:42.396 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:42.396 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T15:05:42.396 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:05:42.396 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: from='client.14180 v1:172.21.15.92:0/3063846388' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi090:[v1:172.21.15.90:6789]=a;smithi090:[v1:172.21.15.90:6790]=c;smithi092:[v1:172.21.15.92:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:05:42.396 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: Saving service mon spec with placement smithi090:[v1:172.21.15.90:6789]=a;smithi090:[v1:172.21.15.90:6790]=c;smithi092:[v1:172.21.15.92:6789]=b;count:3 2024-04-14T15:05:42.396 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: Deploying daemon mon.b on smithi092 2024-04-14T15:05:42.396 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:05:42.396 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:05:42.396 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 ceph-mon[33966]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-14T15:05:42.396 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 bash[33927]: 20e24def0d377ff86771074a51111996756791ca0d91c9254ac1aef978b70d70 2024-04-14T15:05:42.396 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:41 smithi092 systemd[1]: Started Ceph mon.b for 0e288098-fa70-11ee-bc8e-c7b262605968. 2024-04-14T15:05:42.542 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-14T15:05:42.542 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph mon dump -f json 2024-04-14T15:05:42.781 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:05:46.953 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:46.950+0000 7f783ffb6700 1 -- 172.21.15.92:0/3612388043 >> v1:172.21.15.90:6789/0 conn(0x7f7838108510 legacy=0x7f78381088f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:05:46.953 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:46.950+0000 7f783ffb6700 1 -- 172.21.15.92:0/3612388043 shutdown_connections 2024-04-14T15:05:46.953 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:46.950+0000 7f783ffb6700 1 -- 172.21.15.92:0/3612388043 >> 172.21.15.92:0/3612388043 conn(0x7f7838077ba0 msgr2=0x7f7838077fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:05:46.953 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:46.950+0000 7f783ffb6700 1 -- 172.21.15.92:0/3612388043 shutdown_connections 2024-04-14T15:05:46.953 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:46.950+0000 7f783ffb6700 1 -- 172.21.15.92:0/3612388043 wait complete. 2024-04-14T15:05:46.954 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:46.951+0000 7f783ffb6700 1 librados: starting msgr at 2024-04-14T15:05:46.954 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:46.951+0000 7f783ffb6700 1 librados: starting objecter 2024-04-14T15:05:46.954 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:46.951+0000 7f783ffb6700 1 Processor -- start 2024-04-14T15:05:46.954 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:46.951+0000 7f783ffb6700 1 -- start start 2024-04-14T15:05:46.954 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:46.951+0000 7f783ffb6700 1 librados: setting wanted keys 2024-04-14T15:05:46.954 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:46.951+0000 7f783ffb6700 1 librados: calling monclient init 2024-04-14T15:05:46.954 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:46.952+0000 7f783ffb6700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7838102990 con 0x7f7838102500 2024-04-14T15:05:46.954 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:46.952+0000 7f783ffb6700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7838102b60 con 0x7f7838108510 2024-04-14T15:05:46.954 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:46.952+0000 7f783dd52700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f7838108510 0x7f7838101df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:56934/0 (socket says 172.21.15.92:56934) 2024-04-14T15:05:46.954 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:46.952+0000 7f783dd52700 1 -- 172.21.15.92:0/3320065632 learned_addr learned my addr 172.21.15.92:0/3320065632 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-04-14T15:05:46.954 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:46.952+0000 7f782effd700 10 client.?.objecter ms_handle_connect 0x7f7838108510 2024-04-14T15:05:46.955 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:46.952+0000 7f782effd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:05:46.955 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:46.952+0000 7f782effd700 10 client.?.objecter ms_handle_connect 0x7f7838102500 2024-04-14T15:05:46.955 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:46.952+0000 7f782effd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:05:46.955 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:46.952+0000 7f782effd700 1 -- 172.21.15.92:0/3320065632 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2033962932 0 0) 0x7f7838102990 con 0x7f7838102500 2024-04-14T15:05:46.955 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:46.952+0000 7f782effd700 1 -- 172.21.15.92:0/3320065632 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78240035b0 con 0x7f7838102500 2024-04-14T15:05:46.955 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:46.953+0000 7f782effd700 1 -- 172.21.15.92:0/3320065632 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3535069649 0 0) 0x7f78240035b0 con 0x7f7838102500 2024-04-14T15:05:46.955 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:46.953+0000 7f782effd700 1 -- 172.21.15.92:0/3320065632 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7838102990 con 0x7f7838102500 2024-04-14T15:05:46.955 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:46.953+0000 7f782effd700 1 -- 172.21.15.92:0/3320065632 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 239+0+0 (unknown 3211819122 0 0) 0x7f7834001710 con 0x7f7838102500 2024-04-14T15:05:46.956 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:46.953+0000 7f782effd700 1 -- 172.21.15.92:0/3320065632 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3773806670 0 0) 0x7f7838102990 con 0x7f7838102500 2024-04-14T15:05:46.956 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:46.953+0000 7f782effd700 1 -- 172.21.15.92:0/3320065632 >> v1:172.21.15.92:6789/0 conn(0x7f7838108510 legacy=0x7f7838101df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:05:46.956 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:46.953+0000 7f782effd700 1 -- 172.21.15.92:0/3320065632 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78381a7fa0 con 0x7f7838102500 2024-04-14T15:05:46.957 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:46.953+0000 7f783ffb6700 1 -- 172.21.15.92:0/3320065632 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f78381a5f80 con 0x7f7838102500 2024-04-14T15:05:46.957 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:46.953+0000 7f783ffb6700 10 client.14186.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:05:46.957 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:46.953+0000 7f783ffb6700 1 -- 172.21.15.92:0/3320065632 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f78381a6480 con 0x7f7838102500 2024-04-14T15:05:46.957 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:46.953+0000 7f783ffb6700 1 librados: init done 2024-04-14T15:05:46.957 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:46.953+0000 7f782effd700 1 -- 172.21.15.92:0/3320065632 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f7834003b20 con 0x7f7838102500 2024-04-14T15:05:46.957 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:46.953+0000 7f782effd700 1 -- 172.21.15.92:0/3320065632 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 239+0+0 (unknown 3211819122 0 0) 0x7f7834004a30 con 0x7f7838102500 2024-04-14T15:05:46.957 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:46.954+0000 7f782effd700 1 -- 172.21.15.92:0/3320065632 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 13) ==== 46579+0+0 (unknown 2972057686 0 0) 0x7f7834004bd0 con 0x7f7838102500 2024-04-14T15:05:46.957 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:46.954+0000 7f782effd700 1 -- 172.21.15.92:0/3320065632 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(4..4 src has 1..4) ==== 1069+0+0 (unknown 1732540447 0 0) 0x7f7834047a40 con 0x7f7838102500 2024-04-14T15:05:46.957 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:46.954+0000 7f782effd700 10 client.14186.objecter ms_dispatch 0x7f7838001550 osd_map(4..4 src has 1..4) 2024-04-14T15:05:46.957 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:46.954+0000 7f782effd700 3 client.14186.objecter handle_osd_map got epochs [4,4] > 0 2024-04-14T15:05:46.957 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:46.954+0000 7f782effd700 3 client.14186.objecter handle_osd_map decoding full epoch 4 2024-04-14T15:05:46.957 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:46.954+0000 7f782effd700 20 client.14186.objecter dump_active .. 0 homeless 2024-04-14T15:05:46.957 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:46.954+0000 7f783ffb6700 1 -- 172.21.15.92:0/3320065632 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f783806a9e0 con 0x7f7838102500 2024-04-14T15:05:46.957 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:46.955+0000 7f782effd700 10 client.14186.objecter ms_handle_connect 0x7f782403c3e0 2024-04-14T15:05:46.964 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:46.964+0000 7f782effd700 1 -- 172.21.15.92:0/3320065632 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f78340161b0 con 0x7f7838102500 2024-04-14T15:05:47.254 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:47.253+0000 7f783ffb6700 1 -- 172.21.15.92:0/3320065632 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f7838103320 con 0x7f7838102500 2024-04-14T15:05:47.254 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:47.253+0000 7f782effd700 1 -- 172.21.15.92:0/3320065632 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+909 (unknown 3557084514 0 2573337898) 0x7f7834015ab0 con 0x7f7838102500 2024-04-14T15:05:47.255 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:05:47.255 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":2,"fsid":"0e288098-fa70-11ee-bc8e-c7b262605968","modified":"2024-04-14T15:05:41.934198Z","created":"2024-04-14T15:04:02.905926Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6789","nonce":0}]},"addr":"172.21.15.92:6789/0","public_addr":"172.21.15.92:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-14T15:05:47.255 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:46 smithi092 ceph-mon[33966]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:05:47.255 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:47 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T15:05:47.255 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:47 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:05:47.255 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:47 smithi092 ceph-mon[33966]: mon.a calling monitor election 2024-04-14T15:05:47.255 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:47 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:05:47.255 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:47 smithi092 ceph-mon[33966]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:05:47.255 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:47 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:05:47.256 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:47 smithi092 ceph-mon[33966]: mon.b calling monitor election 2024-04-14T15:05:47.256 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:47 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:05:47.256 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:47 smithi092 ceph-mon[33966]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:05:47.256 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:47 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:05:47.256 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:47 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:05:47.256 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:47 smithi092 ceph-mon[33966]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-04-14T15:05:47.256 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:47 smithi092 ceph-mon[33966]: monmap epoch 2 2024-04-14T15:05:47.256 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:47 smithi092 ceph-mon[33966]: fsid 0e288098-fa70-11ee-bc8e-c7b262605968 2024-04-14T15:05:47.256 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:47 smithi092 ceph-mon[33966]: last_changed 2024-04-14T15:05:41.934198+0000 2024-04-14T15:05:47.256 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:47 smithi092 ceph-mon[33966]: created 2024-04-14T15:04:02.905926+0000 2024-04-14T15:05:47.256 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:47 smithi092 ceph-mon[33966]: min_mon_release 19 (squid) 2024-04-14T15:05:47.256 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:47 smithi092 ceph-mon[33966]: election_strategy: 1 2024-04-14T15:05:47.256 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:47 smithi092 ceph-mon[33966]: 0: v1:172.21.15.90:6789/0 mon.a 2024-04-14T15:05:47.257 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:47 smithi092 ceph-mon[33966]: 1: v1:172.21.15.92:6789/0 mon.b 2024-04-14T15:05:47.257 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:47 smithi092 ceph-mon[33966]: fsmap 2024-04-14T15:05:47.257 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:47 smithi092 ceph-mon[33966]: osdmap e4: 0 total, 0 up, 0 in 2024-04-14T15:05:47.257 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:47 smithi092 ceph-mon[33966]: mgrmap e13: y(active, since 41s) 2024-04-14T15:05:47.257 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:47 smithi092 ceph-mon[33966]: overall HEALTH_OK 2024-04-14T15:05:47.257 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:47 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:47.257 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:47 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:47.258 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:47.255+0000 7f783ffb6700 10 librados: watch_flush enter 2024-04-14T15:05:47.258 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:47.255+0000 7f783ffb6700 10 librados: watch_flush exit 2024-04-14T15:05:47.258 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:47.255+0000 7f783ffb6700 20 client.14186.objecter shutdown clearing up homeless session... 2024-04-14T15:05:47.258 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:47.255+0000 7f783ffb6700 10 client.14186.objecter successfully canceled tick 2024-04-14T15:05:47.258 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:47.256+0000 7f783ffb6700 1 -- 172.21.15.92:0/3320065632 >> v1:172.21.15.90:6800/491071118 conn(0x7f782403c3e0 legacy=0x7f782403e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:05:47.258 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:47.256+0000 7f783ffb6700 1 -- 172.21.15.92:0/3320065632 >> v1:172.21.15.90:6789/0 conn(0x7f7838102500 legacy=0x7f78381a4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:05:47.258 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:47.256+0000 7f783ffb6700 1 -- 172.21.15.92:0/3320065632 shutdown_connections 2024-04-14T15:05:47.259 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:47.256+0000 7f783ffb6700 1 -- 172.21.15.92:0/3320065632 >> 172.21.15.92:0/3320065632 conn(0x7f7838077ba0 msgr2=0x7f7838101400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:05:47.259 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:47.256+0000 7f783ffb6700 1 -- 172.21.15.92:0/3320065632 shutdown_connections 2024-04-14T15:05:47.259 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:47.256+0000 7f783ffb6700 1 -- 172.21.15.92:0/3320065632 wait complete. 2024-04-14T15:05:47.259 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:47.256+0000 7f783ffb6700 1 librados: shutdown 2024-04-14T15:05:47.259 INFO:teuthology.orchestra.run.smithi092.stderr:dumped monmap epoch 2 2024-04-14T15:05:47.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:46 smithi090 ceph-mon[30399]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:05:47.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:46 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T15:05:47.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:46 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:05:47.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:46 smithi090 ceph-mon[30399]: mon.a calling monitor election 2024-04-14T15:05:47.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:46 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:05:47.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:46 smithi090 ceph-mon[30399]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:05:47.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:46 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:05:47.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:46 smithi090 ceph-mon[30399]: mon.b calling monitor election 2024-04-14T15:05:47.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:46 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:05:47.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:46 smithi090 ceph-mon[30399]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:05:47.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:46 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:05:47.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:46 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:05:47.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:46 smithi090 ceph-mon[30399]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-04-14T15:05:47.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:46 smithi090 ceph-mon[30399]: monmap epoch 2 2024-04-14T15:05:47.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:46 smithi090 ceph-mon[30399]: fsid 0e288098-fa70-11ee-bc8e-c7b262605968 2024-04-14T15:05:47.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:46 smithi090 ceph-mon[30399]: last_changed 2024-04-14T15:05:41.934198+0000 2024-04-14T15:05:47.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:46 smithi090 ceph-mon[30399]: created 2024-04-14T15:04:02.905926+0000 2024-04-14T15:05:47.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:46 smithi090 ceph-mon[30399]: min_mon_release 19 (squid) 2024-04-14T15:05:47.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:46 smithi090 ceph-mon[30399]: election_strategy: 1 2024-04-14T15:05:47.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:46 smithi090 ceph-mon[30399]: 0: v1:172.21.15.90:6789/0 mon.a 2024-04-14T15:05:47.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:46 smithi090 ceph-mon[30399]: 1: v1:172.21.15.92:6789/0 mon.b 2024-04-14T15:05:47.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:46 smithi090 ceph-mon[30399]: fsmap 2024-04-14T15:05:47.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:47 smithi090 ceph-mon[30399]: osdmap e4: 0 total, 0 up, 0 in 2024-04-14T15:05:47.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:47 smithi090 ceph-mon[30399]: mgrmap e13: y(active, since 41s) 2024-04-14T15:05:47.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:47 smithi090 ceph-mon[30399]: overall HEALTH_OK 2024-04-14T15:05:47.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:47 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:47.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:47 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:48.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:48 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:48.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:48 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T15:05:48.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:48 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:05:48.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:48 smithi092 ceph-mon[33966]: Deploying daemon mon.c on smithi090 2024-04-14T15:05:48.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:48 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.92:0/3320065632' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-14T15:05:48.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:48 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:05:48.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:48 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:48.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:48 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T15:05:48.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:48 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:05:48.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:48 smithi090 ceph-mon[30399]: Deploying daemon mon.c on smithi090 2024-04-14T15:05:48.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:48 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.92:0/3320065632' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-14T15:05:48.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:48 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:05:48.874 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-14T15:05:48.875 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph mon dump -f json 2024-04-14T15:05:49.114 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:05:49.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:49 smithi090 ceph-mon[30399]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:05:49.192 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:05:48 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:05:48.933+0000 7f48abf8d700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-14T15:05:49.235 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:49 smithi092 ceph-mon[33966]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:05:50.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 podman[41633]: 2024-04-14 15:05:50.008019816 +0000 UTC m=+0.017438074 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608 2024-04-14T15:05:50.498 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 podman[41633]: 2024-04-14 15:05:50.238251167 +0000 UTC m=+0.247669396 container create bc18858f949463086f3893ab3d98f541b737bb3019d18609ec36256b6a69944b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mon-c, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=main-a971fd5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS) 2024-04-14T15:05:50.499 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 podman[41633]: 2024-04-14 15:05:50.424165314 +0000 UTC m=+0.433583540 container init bc18858f949463086f3893ab3d98f541b737bb3019d18609ec36256b6a69944b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-a971fd5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=) 2024-04-14T15:05:50.499 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 podman[41633]: 2024-04-14 15:05:50.429740912 +0000 UTC m=+0.439159139 container start bc18858f949463086f3893ab3d98f541b737bb3019d18609ec36256b6a69944b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mon-c, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240403, RELEASE=main-a971fd5) 2024-04-14T15:05:50.499 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: set uid:gid to 167:167 (ceph:ceph) 2024-04-14T15:05:50.499 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev), process ceph-mon, pid 7 2024-04-14T15:05:50.499 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: pidfile_write: ignore empty --pid-file 2024-04-14T15:05:50.499 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: load: jerasure load: lrc 2024-04-14T15:05:50.499 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: RocksDB version: 7.9.2 2024-04-14T15:05:50.499 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Git sha 0 2024-04-14T15:05:50.499 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Compile date 2024-04-06 20:28:24 2024-04-14T15:05:50.499 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: DB SUMMARY 2024-04-14T15:05:50.499 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: DB Session ID: 2FK8GATNGL0XWI57ONEN 2024-04-14T15:05:50.500 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: CURRENT file: CURRENT 2024-04-14T15:05:50.500 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: IDENTITY file: IDENTITY 2024-04-14T15:05:50.500 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-14T15:05:50.500 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-04-14T15:05:50.500 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 566 ; 2024-04-14T15:05:50.500 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.error_if_exists: 0 2024-04-14T15:05:50.500 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.create_if_missing: 0 2024-04-14T15:05:50.500 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.paranoid_checks: 1 2024-04-14T15:05:50.501 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-14T15:05:50.501 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-14T15:05:50.501 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-14T15:05:50.502 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.env: 0x55c59e031c60 2024-04-14T15:05:50.502 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.fs: PosixFileSystem 2024-04-14T15:05:50.502 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.info_log: 0x55c59fda3c20 2024-04-14T15:05:50.502 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.max_file_opening_threads: 16 2024-04-14T15:05:50.502 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.statistics: (nil) 2024-04-14T15:05:50.502 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.use_fsync: 0 2024-04-14T15:05:50.502 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.max_log_file_size: 0 2024-04-14T15:05:50.502 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-14T15:05:50.502 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-14T15:05:50.502 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.keep_log_file_num: 1000 2024-04-14T15:05:50.502 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.recycle_log_file_num: 0 2024-04-14T15:05:50.502 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.allow_fallocate: 1 2024-04-14T15:05:50.502 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.allow_mmap_reads: 0 2024-04-14T15:05:50.503 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.allow_mmap_writes: 0 2024-04-14T15:05:50.503 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.use_direct_reads: 0 2024-04-14T15:05:50.503 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-14T15:05:50.503 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.create_missing_column_families: 0 2024-04-14T15:05:50.503 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.db_log_dir: 2024-04-14T15:05:50.503 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.wal_dir: 2024-04-14T15:05:50.503 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-14T15:05:50.503 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-14T15:05:50.503 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-14T15:05:50.503 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-14T15:05:50.503 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-14T15:05:50.503 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-14T15:05:50.503 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.advise_random_on_open: 1 2024-04-14T15:05:50.503 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.db_write_buffer_size: 0 2024-04-14T15:05:50.504 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.write_buffer_manager: 0x55c59fd741e0 2024-04-14T15:05:50.504 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-14T15:05:50.504 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-14T15:05:50.504 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-14T15:05:50.504 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.rate_limiter: (nil) 2024-04-14T15:05:50.504 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-14T15:05:50.504 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.wal_recovery_mode: 2 2024-04-14T15:05:50.504 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.enable_thread_tracking: 0 2024-04-14T15:05:50.504 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.enable_pipelined_write: 0 2024-04-14T15:05:50.504 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.unordered_write: 0 2024-04-14T15:05:50.504 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-14T15:05:50.504 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-14T15:05:50.504 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-14T15:05:50.504 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-14T15:05:50.504 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.row_cache: None 2024-04-14T15:05:50.505 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.wal_filter: None 2024-04-14T15:05:50.505 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-14T15:05:50.505 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.allow_ingest_behind: 0 2024-04-14T15:05:50.505 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.two_write_queues: 0 2024-04-14T15:05:50.505 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.manual_wal_flush: 0 2024-04-14T15:05:50.505 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.wal_compression: 0 2024-04-14T15:05:50.505 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.atomic_flush: 0 2024-04-14T15:05:50.505 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-14T15:05:50.505 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-14T15:05:50.505 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-14T15:05:50.505 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.log_readahead_size: 0 2024-04-14T15:05:50.505 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-14T15:05:50.505 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.best_efforts_recovery: 0 2024-04-14T15:05:50.505 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-14T15:05:50.506 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-14T15:05:50.506 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.allow_data_in_errors: 0 2024-04-14T15:05:50.506 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.db_host_id: __hostname__ 2024-04-14T15:05:50.506 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-14T15:05:50.506 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.max_background_jobs: 2 2024-04-14T15:05:50.506 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.max_background_compactions: -1 2024-04-14T15:05:50.506 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.max_subcompactions: 1 2024-04-14T15:05:50.506 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-14T15:05:50.506 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-14T15:05:50.506 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-14T15:05:50.506 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.max_total_wal_size: 0 2024-04-14T15:05:50.506 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-14T15:05:50.506 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-14T15:05:50.506 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-14T15:05:50.507 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-14T15:05:50.507 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.max_open_files: -1 2024-04-14T15:05:50.507 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.bytes_per_sync: 0 2024-04-14T15:05:50.507 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-14T15:05:50.507 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-14T15:05:50.507 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.compaction_readahead_size: 0 2024-04-14T15:05:50.507 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.max_background_flushes: -1 2024-04-14T15:05:50.507 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Compression algorithms supported: 2024-04-14T15:05:50.507 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-14T15:05:50.507 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: kZSTD supported: 0 2024-04-14T15:05:50.507 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: kXpressCompression supported: 0 2024-04-14T15:05:50.507 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: kLZ4HCCompression supported: 1 2024-04-14T15:05:50.507 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: kZlibCompression supported: 1 2024-04-14T15:05:50.507 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: kSnappyCompression supported: 1 2024-04-14T15:05:50.508 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: kLZ4Compression supported: 1 2024-04-14T15:05:50.508 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: kBZip2Compression supported: 0 2024-04-14T15:05:50.508 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-14T15:05:50.508 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-14T15:05:50.508 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-04-14T15:05:50.508 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-14T15:05:50.508 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-14T15:05:50.508 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.merge_operator: 2024-04-14T15:05:50.508 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.compaction_filter: None 2024-04-14T15:05:50.508 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.compaction_filter_factory: None 2024-04-14T15:05:50.508 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.sst_partitioner_factory: None 2024-04-14T15:05:50.508 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-14T15:05:50.508 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-14T15:05:50.508 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55c59fda3de0) 2024-04-14T15:05:50.509 INFO:journalctl@ceph.mon.c.smithi090.stdout: cache_index_and_filter_blocks: 1 2024-04-14T15:05:50.509 INFO:journalctl@ceph.mon.c.smithi090.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-14T15:05:50.509 INFO:journalctl@ceph.mon.c.smithi090.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-14T15:05:50.509 INFO:journalctl@ceph.mon.c.smithi090.stdout: pin_top_level_index_and_filter: 1 2024-04-14T15:05:50.509 INFO:journalctl@ceph.mon.c.smithi090.stdout: index_type: 0 2024-04-14T15:05:50.509 INFO:journalctl@ceph.mon.c.smithi090.stdout: data_block_index_type: 0 2024-04-14T15:05:50.509 INFO:journalctl@ceph.mon.c.smithi090.stdout: index_shortening: 1 2024-04-14T15:05:50.509 INFO:journalctl@ceph.mon.c.smithi090.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-14T15:05:50.509 INFO:journalctl@ceph.mon.c.smithi090.stdout: checksum: 4 2024-04-14T15:05:50.509 INFO:journalctl@ceph.mon.c.smithi090.stdout: no_block_cache: 0 2024-04-14T15:05:50.509 INFO:journalctl@ceph.mon.c.smithi090.stdout: block_cache: 0x55c59fda7090 2024-04-14T15:05:50.509 INFO:journalctl@ceph.mon.c.smithi090.stdout: block_cache_name: BinnedLRUCache 2024-04-14T15:05:50.509 INFO:journalctl@ceph.mon.c.smithi090.stdout: block_cache_options: 2024-04-14T15:05:50.509 INFO:journalctl@ceph.mon.c.smithi090.stdout: capacity : 536870912 2024-04-14T15:05:50.509 INFO:journalctl@ceph.mon.c.smithi090.stdout: num_shard_bits : 4 2024-04-14T15:05:50.510 INFO:journalctl@ceph.mon.c.smithi090.stdout: strict_capacity_limit : 0 2024-04-14T15:05:50.510 INFO:journalctl@ceph.mon.c.smithi090.stdout: high_pri_pool_ratio: 0.000 2024-04-14T15:05:50.510 INFO:journalctl@ceph.mon.c.smithi090.stdout: block_cache_compressed: (nil) 2024-04-14T15:05:50.510 INFO:journalctl@ceph.mon.c.smithi090.stdout: persistent_cache: (nil) 2024-04-14T15:05:50.510 INFO:journalctl@ceph.mon.c.smithi090.stdout: block_size: 4096 2024-04-14T15:05:50.510 INFO:journalctl@ceph.mon.c.smithi090.stdout: block_size_deviation: 10 2024-04-14T15:05:50.510 INFO:journalctl@ceph.mon.c.smithi090.stdout: block_restart_interval: 16 2024-04-14T15:05:50.510 INFO:journalctl@ceph.mon.c.smithi090.stdout: index_block_restart_interval: 1 2024-04-14T15:05:50.510 INFO:journalctl@ceph.mon.c.smithi090.stdout: metadata_block_size: 4096 2024-04-14T15:05:50.510 INFO:journalctl@ceph.mon.c.smithi090.stdout: partition_filters: 0 2024-04-14T15:05:50.510 INFO:journalctl@ceph.mon.c.smithi090.stdout: use_delta_encoding: 1 2024-04-14T15:05:50.510 INFO:journalctl@ceph.mon.c.smithi090.stdout: filter_policy: bloomfilter 2024-04-14T15:05:50.510 INFO:journalctl@ceph.mon.c.smithi090.stdout: whole_key_filtering: 1 2024-04-14T15:05:50.510 INFO:journalctl@ceph.mon.c.smithi090.stdout: verify_compression: 0 2024-04-14T15:05:50.510 INFO:journalctl@ceph.mon.c.smithi090.stdout: read_amp_bytes_per_bit: 0 2024-04-14T15:05:50.511 INFO:journalctl@ceph.mon.c.smithi090.stdout: format_version: 5 2024-04-14T15:05:50.511 INFO:journalctl@ceph.mon.c.smithi090.stdout: enable_index_compression: 1 2024-04-14T15:05:50.511 INFO:journalctl@ceph.mon.c.smithi090.stdout: block_align: 0 2024-04-14T15:05:50.511 INFO:journalctl@ceph.mon.c.smithi090.stdout: max_auto_readahead_size: 262144 2024-04-14T15:05:50.511 INFO:journalctl@ceph.mon.c.smithi090.stdout: prepopulate_block_cache: 0 2024-04-14T15:05:50.511 INFO:journalctl@ceph.mon.c.smithi090.stdout: initial_auto_readahead_size: 8192 2024-04-14T15:05:50.511 INFO:journalctl@ceph.mon.c.smithi090.stdout: num_file_reads_for_auto_readahead: 2 2024-04-14T15:05:50.511 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.write_buffer_size: 33554432 2024-04-14T15:05:50.511 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.max_write_buffer_number: 2 2024-04-14T15:05:50.511 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.compression: NoCompression 2024-04-14T15:05:50.511 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.bottommost_compression: Disabled 2024-04-14T15:05:50.511 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.prefix_extractor: nullptr 2024-04-14T15:05:50.511 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-14T15:05:50.511 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.num_levels: 7 2024-04-14T15:05:50.511 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-14T15:05:50.512 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-14T15:05:50.512 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-14T15:05:50.512 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-14T15:05:50.512 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-14T15:05:50.512 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-14T15:05:50.512 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-14T15:05:50.512 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-14T15:05:50.512 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-14T15:05:50.512 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-14T15:05:50.512 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-14T15:05:50.512 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-14T15:05:50.512 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-14T15:05:50.512 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.compression_opts.level: 32767 2024-04-14T15:05:50.512 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.compression_opts.strategy: 0 2024-04-14T15:05:50.513 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-14T15:05:50.513 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-14T15:05:50.513 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-14T15:05:50.513 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-14T15:05:50.513 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.compression_opts.enabled: false 2024-04-14T15:05:50.513 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-14T15:05:50.513 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-14T15:05:50.513 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-14T15:05:50.513 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-14T15:05:50.513 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.target_file_size_base: 67108864 2024-04-14T15:05:50.513 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-14T15:05:50.513 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-14T15:05:50.513 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-14T15:05:50.513 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-14T15:05:50.514 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-14T15:05:50.514 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-14T15:05:50.514 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-14T15:05:50.514 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-14T15:05:50.514 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-14T15:05:50.514 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-14T15:05:50.514 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-14T15:05:50.514 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-14T15:05:50.514 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-14T15:05:50.514 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-14T15:05:50.514 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.arena_block_size: 1048576 2024-04-14T15:05:50.514 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-14T15:05:50.514 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-14T15:05:50.514 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.disable_auto_compactions: 0 2024-04-14T15:05:50.515 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-14T15:05:50.515 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-14T15:05:50.515 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-14T15:05:50.515 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-14T15:05:50.515 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-14T15:05:50.515 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-14T15:05:50.515 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-14T15:05:50.515 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-14T15:05:50.515 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-14T15:05:50.515 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-14T15:05:50.515 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-14T15:05:50.515 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.inplace_update_support: 0 2024-04-14T15:05:50.515 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-14T15:05:50.515 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-14T15:05:50.515 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-14T15:05:50.516 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-14T15:05:50.516 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.bloom_locality: 0 2024-04-14T15:05:50.516 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.max_successive_merges: 0 2024-04-14T15:05:50.516 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-14T15:05:50.516 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.paranoid_file_checks: 0 2024-04-14T15:05:50.516 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.force_consistency_checks: 1 2024-04-14T15:05:50.516 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.report_bg_io_stats: 0 2024-04-14T15:05:50.516 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.ttl: 2592000 2024-04-14T15:05:50.516 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-14T15:05:50.516 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-14T15:05:50.516 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-14T15:05:50.516 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.enable_blob_files: false 2024-04-14T15:05:50.516 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.min_blob_size: 0 2024-04-14T15:05:50.516 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.blob_file_size: 268435456 2024-04-14T15:05:50.516 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-14T15:05:50.517 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-14T15:05:50.517 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-14T15:05:50.517 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-14T15:05:50.517 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-14T15:05:50.517 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.blob_file_starting_level: 0 2024-04-14T15:05:50.517 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-14T15:05:50.517 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-14T15:05:50.517 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-14T15:05:50.517 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 1548fb4b-451f-4808-b552-1d8b8c59c494 2024-04-14T15:05:50.517 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713107150458808, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-14T15:05:50.517 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-14T15:05:50.517 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713107150459199, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1703, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 578, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 456, "raw_average_value_size": 91, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713107150, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "1548fb4b-451f-4808-b552-1d8b8c59c494", "db_session_id": "2FK8GATNGL0XWI57ONEN", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-14T15:05:50.517 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713107150459277, "job": 1, "event": "recovery_finished"} 2024-04-14T15:05:50.518 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-14T15:05:50.518 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-14T15:05:50.518 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55c59fe96000 2024-04-14T15:05:50.518 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: DB pointer 0x55c59fe80000 2024-04-14T15:05:50.518 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-14T15:05:50.518 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-14T15:05:50.518 INFO:journalctl@ceph.mon.c.smithi090.stdout: ** DB Stats ** 2024-04-14T15:05:50.518 INFO:journalctl@ceph.mon.c.smithi090.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-14T15:05:50.518 INFO:journalctl@ceph.mon.c.smithi090.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-14T15:05:50.518 INFO:journalctl@ceph.mon.c.smithi090.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-14T15:05:50.518 INFO:journalctl@ceph.mon.c.smithi090.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-14T15:05:50.518 INFO:journalctl@ceph.mon.c.smithi090.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-14T15:05:50.518 INFO:journalctl@ceph.mon.c.smithi090.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-14T15:05:50.519 INFO:journalctl@ceph.mon.c.smithi090.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-14T15:05:50.519 INFO:journalctl@ceph.mon.c.smithi090.stdout: 2024-04-14T15:05:50.519 INFO:journalctl@ceph.mon.c.smithi090.stdout: ** Compaction Stats [default] ** 2024-04-14T15:05:50.519 INFO:journalctl@ceph.mon.c.smithi090.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-14T15:05:50.519 INFO:journalctl@ceph.mon.c.smithi090.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-14T15:05:50.519 INFO:journalctl@ceph.mon.c.smithi090.stdout: L0 1/0 1.66 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-14T15:05:50.519 INFO:journalctl@ceph.mon.c.smithi090.stdout: Sum 1/0 1.66 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-14T15:05:50.519 INFO:journalctl@ceph.mon.c.smithi090.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-14T15:05:50.519 INFO:journalctl@ceph.mon.c.smithi090.stdout: 2024-04-14T15:05:50.519 INFO:journalctl@ceph.mon.c.smithi090.stdout: ** Compaction Stats [default] ** 2024-04-14T15:05:50.519 INFO:journalctl@ceph.mon.c.smithi090.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-14T15:05:50.519 INFO:journalctl@ceph.mon.c.smithi090.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-14T15:05:50.519 INFO:journalctl@ceph.mon.c.smithi090.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-14T15:05:50.519 INFO:journalctl@ceph.mon.c.smithi090.stdout: 2024-04-14T15:05:50.520 INFO:journalctl@ceph.mon.c.smithi090.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-14T15:05:50.520 INFO:journalctl@ceph.mon.c.smithi090.stdout: 2024-04-14T15:05:50.520 INFO:journalctl@ceph.mon.c.smithi090.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-14T15:05:50.520 INFO:journalctl@ceph.mon.c.smithi090.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-14T15:05:50.520 INFO:journalctl@ceph.mon.c.smithi090.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-14T15:05:50.520 INFO:journalctl@ceph.mon.c.smithi090.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-14T15:05:50.520 INFO:journalctl@ceph.mon.c.smithi090.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-14T15:05:50.520 INFO:journalctl@ceph.mon.c.smithi090.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-14T15:05:50.520 INFO:journalctl@ceph.mon.c.smithi090.stdout: Cumulative compaction: 0.00 GB write, 0.68 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-14T15:05:50.520 INFO:journalctl@ceph.mon.c.smithi090.stdout: Interval compaction: 0.00 GB write, 0.68 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-14T15:05:50.520 INFO:journalctl@ceph.mon.c.smithi090.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-14T15:05:50.520 INFO:journalctl@ceph.mon.c.smithi090.stdout: Block cache BinnedLRUCache@0x55c59fda7090#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2024-04-14T15:05:50.520 INFO:journalctl@ceph.mon.c.smithi090.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-14T15:05:50.520 INFO:journalctl@ceph.mon.c.smithi090.stdout: 2024-04-14T15:05:50.521 INFO:journalctl@ceph.mon.c.smithi090.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-14T15:05:50.521 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-04-14T15:05:50.521 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: using public_addrv v1:172.21.15.90:6790/0 2024-04-14T15:05:50.521 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: starting mon.c rank -1 at public addrs v1:172.21.15.90:6790/0 at bind addrs v1:172.21.15.90:6790/0 mon_data /var/lib/ceph/mon/ceph-c fsid 0e288098-fa70-11ee-bc8e-c7b262605968 2024-04-14T15:05:50.521 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: mon.c@-1(???) e0 preinit fsid 0e288098-fa70-11ee-bc8e-c7b262605968 2024-04-14T15:05:50.521 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: mon.c@-1(synchronizing).mds e1 new map 2024-04-14T15:05:50.521 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: mon.c@-1(synchronizing).mds e1 print_map 2024-04-14T15:05:50.521 INFO:journalctl@ceph.mon.c.smithi090.stdout: e1 2024-04-14T15:05:50.521 INFO:journalctl@ceph.mon.c.smithi090.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-14T15:05:50.521 INFO:journalctl@ceph.mon.c.smithi090.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-14T15:05:50.521 INFO:journalctl@ceph.mon.c.smithi090.stdout: legacy client fscid: -1 2024-04-14T15:05:50.521 INFO:journalctl@ceph.mon.c.smithi090.stdout: 2024-04-14T15:05:50.521 INFO:journalctl@ceph.mon.c.smithi090.stdout: No filesystems configured 2024-04-14T15:05:50.521 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-14T15:05:50.522 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-14T15:05:50.522 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-14T15:05:50.522 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-14T15:05:50.522 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-14T15:05:50.522 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-14T15:05:50.522 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-14T15:05:50.522 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-14T15:05:50.523 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-14T15:05:50.523 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-14T15:05:50.523 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:05:50.523 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2166157961' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "jewel"}]': finished 2024-04-14T15:05:50.523 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: osdmap e4: 0 total, 0 up, 0 in 2024-04-14T15:05:50.523 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:50.523 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:50.523 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:50.523 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:50.523 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi092", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:05:50.523 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:05:50.523 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:05:50.523 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: Updating smithi092:/etc/ceph/ceph.conf 2024-04-14T15:05:50.524 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: Updating smithi092:/var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T15:05:50.524 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: Updating smithi092:/etc/ceph/ceph.client.admin.keyring 2024-04-14T15:05:50.524 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:50.524 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:50.524 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:50.524 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: Updating smithi092:/var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T15:05:50.524 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:05:50.524 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:50.524 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:05:50.524 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:05:50.524 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:05:50.524 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:50.524 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T15:05:50.525 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:05:50.525 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: from='client.14180 v1:172.21.15.92:0/3063846388' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi090:[v1:172.21.15.90:6789]=a;smithi090:[v1:172.21.15.90:6790]=c;smithi092:[v1:172.21.15.92:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:05:50.525 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: Saving service mon spec with placement smithi090:[v1:172.21.15.90:6789]=a;smithi090:[v1:172.21.15.90:6790]=c;smithi092:[v1:172.21.15.92:6789]=b;count:3 2024-04-14T15:05:50.525 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: Deploying daemon mon.b on smithi092 2024-04-14T15:05:50.525 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:05:50.525 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:05:50.525 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:05:50.525 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T15:05:50.525 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:05:50.525 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: mon.a calling monitor election 2024-04-14T15:05:50.525 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:05:50.525 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:05:50.525 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:05:50.525 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: mon.b calling monitor election 2024-04-14T15:05:50.526 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:05:50.526 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:05:50.526 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:05:50.526 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:05:50.526 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-04-14T15:05:50.526 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: monmap epoch 2 2024-04-14T15:05:50.526 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: fsid 0e288098-fa70-11ee-bc8e-c7b262605968 2024-04-14T15:05:50.526 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: last_changed 2024-04-14T15:05:41.934198+0000 2024-04-14T15:05:50.526 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: created 2024-04-14T15:04:02.905926+0000 2024-04-14T15:05:50.526 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: min_mon_release 19 (squid) 2024-04-14T15:05:50.526 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: election_strategy: 1 2024-04-14T15:05:50.526 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: 0: v1:172.21.15.90:6789/0 mon.a 2024-04-14T15:05:50.526 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: 1: v1:172.21.15.92:6789/0 mon.b 2024-04-14T15:05:50.526 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: fsmap 2024-04-14T15:05:50.527 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: osdmap e4: 0 total, 0 up, 0 in 2024-04-14T15:05:50.527 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: mgrmap e13: y(active, since 41s) 2024-04-14T15:05:50.527 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: overall HEALTH_OK 2024-04-14T15:05:50.527 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:50.527 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:50.527 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:50.527 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T15:05:50.527 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:05:50.527 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: Deploying daemon mon.c on smithi090 2024-04-14T15:05:50.527 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.92:0/3320065632' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-14T15:05:50.527 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:05:50.527 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:05:50.527 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 ceph-mon[41670]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-14T15:05:50.527 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 bash[41633]: bc18858f949463086f3893ab3d98f541b737bb3019d18609ec36256b6a69944b 2024-04-14T15:05:50.528 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:50 smithi090 systemd[1]: Started Ceph mon.c for 0e288098-fa70-11ee-bc8e-c7b262605968. 2024-04-14T15:05:55.507 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:55.504+0000 7ff8bee5f700 1 -- 172.21.15.92:0/3912543375 >> v1:172.21.15.90:6789/0 conn(0x7ff89c0035f0 legacy=0x7ff89c005a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:05:55.507 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:55.505+0000 7ff8bee5f700 1 -- 172.21.15.92:0/3912543375 shutdown_connections 2024-04-14T15:05:55.507 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:55.505+0000 7ff8bee5f700 1 -- 172.21.15.92:0/3912543375 >> 172.21.15.92:0/3912543375 conn(0x7ff8b80fe030 msgr2=0x7ff8b8100450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:05:55.507 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:55.505+0000 7ff8bee5f700 1 -- 172.21.15.92:0/3912543375 shutdown_connections 2024-04-14T15:05:55.507 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:55.505+0000 7ff8bee5f700 1 -- 172.21.15.92:0/3912543375 wait complete. 2024-04-14T15:05:55.507 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:55.505+0000 7ff8bee5f700 1 librados: starting msgr at 2024-04-14T15:05:55.507 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:55.505+0000 7ff8bee5f700 1 librados: starting objecter 2024-04-14T15:05:55.507 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:55.505+0000 7ff8bee5f700 1 Processor -- start 2024-04-14T15:05:55.508 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:55.506+0000 7ff8bee5f700 1 -- start start 2024-04-14T15:05:55.508 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:55.506+0000 7ff8bee5f700 1 librados: setting wanted keys 2024-04-14T15:05:55.508 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:55.506+0000 7ff8bee5f700 1 librados: calling monclient init 2024-04-14T15:05:55.508 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:55.506+0000 7ff8bee5f700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff8b81a95a0 con 0x7ff8b8075950 2024-04-14T15:05:55.508 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:55.506+0000 7ff8bee5f700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff8b81aa7c0 con 0x7ff8b80754c0 2024-04-14T15:05:55.508 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:55.506+0000 7ff8bee5f700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff8b81aba00 con 0x7ff89c0035f0 2024-04-14T15:05:55.508 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:55.506+0000 7ff8b7fff700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7ff8b80754c0 0x7ff8b81a45b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:56946/0 (socket says 172.21.15.92:56946) 2024-04-14T15:05:55.508 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:55.506+0000 7ff8b7fff700 1 -- 172.21.15.92:0/3058178067 learned_addr learned my addr 172.21.15.92:0/3058178067 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-04-14T15:05:55.508 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:55.506+0000 7ff8b5ffb700 10 client.?.objecter ms_handle_connect 0x7ff8b80754c0 2024-04-14T15:05:55.508 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:55.506+0000 7ff8b5ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:05:55.508 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:55.506+0000 7ff8b5ffb700 10 client.?.objecter ms_handle_connect 0x7ff8b8075950 2024-04-14T15:05:55.508 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:55.506+0000 7ff8b5ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:05:55.508 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:55.506+0000 7ff8b5ffb700 10 client.?.objecter ms_handle_connect 0x7ff89c0035f0 2024-04-14T15:05:55.509 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:55.506+0000 7ff8b5ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:05:55.510 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:55.507+0000 7ff8b5ffb700 1 -- 172.21.15.92:0/3058178067 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1082493617 0 0) 0x7ff8b81a95a0 con 0x7ff8b8075950 2024-04-14T15:05:55.510 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:55.507+0000 7ff8b5ffb700 1 -- 172.21.15.92:0/3058178067 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff8a8003bf0 con 0x7ff8b8075950 2024-04-14T15:05:55.511 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:55.508+0000 7ff8b5ffb700 1 -- 172.21.15.92:0/3058178067 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4251292680 0 0) 0x7ff8a8003bf0 con 0x7ff8b8075950 2024-04-14T15:05:55.511 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:55.508+0000 7ff8b5ffb700 1 -- 172.21.15.92:0/3058178067 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff8b81a95a0 con 0x7ff8b8075950 2024-04-14T15:05:55.511 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:55.508+0000 7ff8b5ffb700 1 -- 172.21.15.92:0/3058178067 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7ff8b0001b90 con 0x7ff8b8075950 2024-04-14T15:05:55.511 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:55.508+0000 7ff8b5ffb700 1 -- 172.21.15.92:0/3058178067 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4062049250 0 0) 0x7ff8b81a95a0 con 0x7ff8b8075950 2024-04-14T15:05:55.511 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:55.508+0000 7ff8b5ffb700 1 -- 172.21.15.92:0/3058178067 >> v1:172.21.15.90:6790/0 conn(0x7ff89c0035f0 legacy=0x7ff8b8074db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:05:55.511 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:55.508+0000 7ff8b5ffb700 1 -- 172.21.15.92:0/3058178067 >> v1:172.21.15.92:6789/0 conn(0x7ff8b80754c0 legacy=0x7ff8b81a45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:05:55.511 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:55.508+0000 7ff8b5ffb700 1 -- 172.21.15.92:0/3058178067 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8b81acc40 con 0x7ff8b8075950 2024-04-14T15:05:55.511 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:55.508+0000 7ff8bee5f700 1 -- 172.21.15.92:0/3058178067 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff8b81a97d0 con 0x7ff8b8075950 2024-04-14T15:05:55.511 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:55.508+0000 7ff8b5ffb700 1 -- 172.21.15.92:0/3058178067 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7ff8b0003f60 con 0x7ff8b8075950 2024-04-14T15:05:55.511 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:55.508+0000 7ff8b5ffb700 1 -- 172.21.15.92:0/3058178067 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7ff8b0004e60 con 0x7ff8b8075950 2024-04-14T15:05:55.511 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:55.509+0000 7ff8bee5f700 10 client.14190.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:05:55.511 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:55.509+0000 7ff8bee5f700 1 -- 172.21.15.92:0/3058178067 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff8b81a9cd0 con 0x7ff8b8075950 2024-04-14T15:05:55.511 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:55.509+0000 7ff8bee5f700 1 librados: init done 2024-04-14T15:05:55.511 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:55.509+0000 7ff8b5ffb700 1 -- 172.21.15.92:0/3058178067 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 13) ==== 46579+0+0 (unknown 2972057686 0 0) 0x7ff8b0005020 con 0x7ff8b8075950 2024-04-14T15:05:55.512 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:55.509+0000 7ff8b5ffb700 1 -- 172.21.15.92:0/3058178067 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(4..4 src has 1..4) ==== 1069+0+0 (unknown 1732540447 0 0) 0x7ff8b0047ea0 con 0x7ff8b8075950 2024-04-14T15:05:55.512 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:55.510+0000 7ff8b5ffb700 10 client.14190.objecter ms_dispatch 0x7ff8b8001550 osd_map(4..4 src has 1..4) 2024-04-14T15:05:55.512 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:55.510+0000 7ff8b5ffb700 3 client.14190.objecter handle_osd_map got epochs [4,4] > 0 2024-04-14T15:05:55.512 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:55.510+0000 7ff8b5ffb700 3 client.14190.objecter handle_osd_map decoding full epoch 4 2024-04-14T15:05:55.512 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:55.510+0000 7ff8b5ffb700 20 client.14190.objecter dump_active .. 0 homeless 2024-04-14T15:05:55.512 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:55.510+0000 7ff8bee5f700 1 -- 172.21.15.92:0/3058178067 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8b8103c80 con 0x7ff8b8075950 2024-04-14T15:05:55.512 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:55.510+0000 7ff8b5ffb700 10 client.14190.objecter ms_handle_connect 0x7ff8a803c210 2024-04-14T15:05:55.516 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:55.516+0000 7ff8b5ffb700 1 -- 172.21.15.92:0/3058178067 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7ff8b0016610 con 0x7ff8b8075950 2024-04-14T15:05:55.810 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:55.810+0000 7ff8bee5f700 1 -- 172.21.15.92:0/3058178067 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7ff8b80763e0 con 0x7ff8b8075950 2024-04-14T15:05:55.811 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:55 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T15:05:55.811 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:55 smithi092 ceph-mon[33966]: mon.a calling monitor election 2024-04-14T15:05:55.811 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:55 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:05:55.811 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:55 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:05:55.811 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:55 smithi092 ceph-mon[33966]: mon.b calling monitor election 2024-04-14T15:05:55.811 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:55 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:05:55.811 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:55 smithi092 ceph-mon[33966]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:05:55.812 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:55 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:05:55.812 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:55 smithi092 ceph-mon[33966]: mon.c calling monitor election 2024-04-14T15:05:55.812 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:55 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:05:55.812 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:55 smithi092 ceph-mon[33966]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:05:55.812 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:55 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:05:55.812 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:55 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:05:55.812 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:55 smithi092 ceph-mon[33966]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-04-14T15:05:55.812 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:55 smithi092 ceph-mon[33966]: monmap epoch 3 2024-04-14T15:05:55.812 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:55 smithi092 ceph-mon[33966]: fsid 0e288098-fa70-11ee-bc8e-c7b262605968 2024-04-14T15:05:55.812 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:55 smithi092 ceph-mon[33966]: last_changed 2024-04-14T15:05:50.487567+0000 2024-04-14T15:05:55.812 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:55 smithi092 ceph-mon[33966]: created 2024-04-14T15:04:02.905926+0000 2024-04-14T15:05:55.813 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:55 smithi092 ceph-mon[33966]: min_mon_release 19 (squid) 2024-04-14T15:05:55.813 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:55 smithi092 ceph-mon[33966]: election_strategy: 1 2024-04-14T15:05:55.813 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:55 smithi092 ceph-mon[33966]: 0: v1:172.21.15.90:6789/0 mon.a 2024-04-14T15:05:55.813 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:55 smithi092 ceph-mon[33966]: 1: v1:172.21.15.92:6789/0 mon.b 2024-04-14T15:05:55.813 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:55 smithi092 ceph-mon[33966]: 2: v1:172.21.15.90:6790/0 mon.c 2024-04-14T15:05:55.813 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:55 smithi092 ceph-mon[33966]: fsmap 2024-04-14T15:05:55.813 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:55 smithi092 ceph-mon[33966]: osdmap e4: 0 total, 0 up, 0 in 2024-04-14T15:05:55.813 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:55 smithi092 ceph-mon[33966]: mgrmap e13: y(active, since 49s) 2024-04-14T15:05:55.813 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:55 smithi092 ceph-mon[33966]: overall HEALTH_OK 2024-04-14T15:05:55.813 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:55 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:55.813 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:55 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:55.813 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:55 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:55.814 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:55.810+0000 7ff8b5ffb700 1 -- 172.21.15.92:0/3058178067 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1124 (unknown 409899127 0 1008968738) 0x7ff8b0015f10 con 0x7ff8b8075950 2024-04-14T15:05:55.814 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-14T15:05:55.814 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":3,"fsid":"0e288098-fa70-11ee-bc8e-c7b262605968","modified":"2024-04-14T15:05:50.487567Z","created":"2024-04-14T15:04:02.905926Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6789","nonce":0}]},"addr":"172.21.15.92:6789/0","public_addr":"172.21.15.92:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6790","nonce":0}]},"addr":"172.21.15.90:6790/0","public_addr":"172.21.15.90:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-04-14T15:05:55.815 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:55.812+0000 7ff8bee5f700 10 librados: watch_flush enter 2024-04-14T15:05:55.815 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:55.812+0000 7ff8bee5f700 10 librados: watch_flush exit 2024-04-14T15:05:55.815 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:55.812+0000 7ff8bee5f700 20 client.14190.objecter shutdown clearing up homeless session... 2024-04-14T15:05:55.815 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:55.812+0000 7ff8bee5f700 10 client.14190.objecter successfully canceled tick 2024-04-14T15:05:55.815 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:55.812+0000 7ff8bee5f700 1 -- 172.21.15.92:0/3058178067 >> v1:172.21.15.90:6800/491071118 conn(0x7ff8a803c210 legacy=0x7ff8a803e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:05:55.815 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:55.812+0000 7ff8bee5f700 1 -- 172.21.15.92:0/3058178067 >> v1:172.21.15.90:6789/0 conn(0x7ff8b8075950 legacy=0x7ff8b81a7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:05:55.815 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:55.812+0000 7ff8bee5f700 1 -- 172.21.15.92:0/3058178067 shutdown_connections 2024-04-14T15:05:55.815 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:55.812+0000 7ff8bee5f700 1 -- 172.21.15.92:0/3058178067 >> 172.21.15.92:0/3058178067 conn(0x7ff8b80fe030 msgr2=0x7ff8b80fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:05:55.815 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:55.812+0000 7ff8bee5f700 1 -- 172.21.15.92:0/3058178067 shutdown_connections 2024-04-14T15:05:55.816 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:55.813+0000 7ff8bee5f700 1 -- 172.21.15.92:0/3058178067 wait complete. 2024-04-14T15:05:55.816 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:05:55.813+0000 7ff8bee5f700 1 librados: shutdown 2024-04-14T15:05:55.816 INFO:teuthology.orchestra.run.smithi092.stderr:dumped monmap epoch 3 2024-04-14T15:05:55.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:55 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T15:05:55.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:55 smithi090 ceph-mon[41670]: mon.a calling monitor election 2024-04-14T15:05:55.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:55 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:05:55.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:55 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:05:55.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:55 smithi090 ceph-mon[41670]: mon.b calling monitor election 2024-04-14T15:05:55.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:55 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:05:55.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:55 smithi090 ceph-mon[41670]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:05:55.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:55 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:05:55.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:55 smithi090 ceph-mon[41670]: mon.c calling monitor election 2024-04-14T15:05:55.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:55 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:05:55.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:55 smithi090 ceph-mon[41670]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:05:55.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:55 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:05:55.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:55 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:05:55.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:55 smithi090 ceph-mon[41670]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-04-14T15:05:55.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:55 smithi090 ceph-mon[41670]: monmap epoch 3 2024-04-14T15:05:55.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:55 smithi090 ceph-mon[41670]: fsid 0e288098-fa70-11ee-bc8e-c7b262605968 2024-04-14T15:05:55.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:55 smithi090 ceph-mon[41670]: last_changed 2024-04-14T15:05:50.487567+0000 2024-04-14T15:05:55.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:55 smithi090 ceph-mon[41670]: created 2024-04-14T15:04:02.905926+0000 2024-04-14T15:05:55.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:55 smithi090 ceph-mon[41670]: min_mon_release 19 (squid) 2024-04-14T15:05:55.944 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:55 smithi090 ceph-mon[41670]: election_strategy: 1 2024-04-14T15:05:55.944 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:55 smithi090 ceph-mon[41670]: 0: v1:172.21.15.90:6789/0 mon.a 2024-04-14T15:05:55.944 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:55 smithi090 ceph-mon[41670]: 1: v1:172.21.15.92:6789/0 mon.b 2024-04-14T15:05:55.944 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:55 smithi090 ceph-mon[41670]: 2: v1:172.21.15.90:6790/0 mon.c 2024-04-14T15:05:55.944 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:55 smithi090 ceph-mon[41670]: fsmap 2024-04-14T15:05:55.944 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:55 smithi090 ceph-mon[41670]: osdmap e4: 0 total, 0 up, 0 in 2024-04-14T15:05:55.944 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:55 smithi090 ceph-mon[41670]: mgrmap e13: y(active, since 49s) 2024-04-14T15:05:55.944 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:55 smithi090 ceph-mon[41670]: overall HEALTH_OK 2024-04-14T15:05:55.944 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:55 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:55.944 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:55 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:55.944 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:55 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:55.944 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:55 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T15:05:55.945 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:55 smithi090 ceph-mon[30399]: mon.a calling monitor election 2024-04-14T15:05:55.945 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:55 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:05:55.945 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:55 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:05:55.945 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:55 smithi090 ceph-mon[30399]: mon.b calling monitor election 2024-04-14T15:05:55.945 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:55 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:05:55.945 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:55 smithi090 ceph-mon[30399]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:05:55.945 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:55 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:05:55.945 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:55 smithi090 ceph-mon[30399]: mon.c calling monitor election 2024-04-14T15:05:55.945 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:55 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:05:55.945 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:55 smithi090 ceph-mon[30399]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:05:55.945 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:55 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:05:55.945 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:55 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:05:55.945 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:55 smithi090 ceph-mon[30399]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-04-14T15:05:55.945 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:55 smithi090 ceph-mon[30399]: monmap epoch 3 2024-04-14T15:05:55.946 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:55 smithi090 ceph-mon[30399]: fsid 0e288098-fa70-11ee-bc8e-c7b262605968 2024-04-14T15:05:55.946 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:55 smithi090 ceph-mon[30399]: last_changed 2024-04-14T15:05:50.487567+0000 2024-04-14T15:05:55.946 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:55 smithi090 ceph-mon[30399]: created 2024-04-14T15:04:02.905926+0000 2024-04-14T15:05:55.946 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:55 smithi090 ceph-mon[30399]: min_mon_release 19 (squid) 2024-04-14T15:05:55.946 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:55 smithi090 ceph-mon[30399]: election_strategy: 1 2024-04-14T15:05:55.946 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:55 smithi090 ceph-mon[30399]: 0: v1:172.21.15.90:6789/0 mon.a 2024-04-14T15:05:55.946 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:55 smithi090 ceph-mon[30399]: 1: v1:172.21.15.92:6789/0 mon.b 2024-04-14T15:05:55.946 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:55 smithi090 ceph-mon[30399]: 2: v1:172.21.15.90:6790/0 mon.c 2024-04-14T15:05:55.946 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:55 smithi090 ceph-mon[30399]: fsmap 2024-04-14T15:05:55.946 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:55 smithi090 ceph-mon[30399]: osdmap e4: 0 total, 0 up, 0 in 2024-04-14T15:05:55.946 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:55 smithi090 ceph-mon[30399]: mgrmap e13: y(active, since 49s) 2024-04-14T15:05:55.946 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:55 smithi090 ceph-mon[30399]: overall HEALTH_OK 2024-04-14T15:05:55.946 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:55 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:55.947 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:55 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:55.947 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:55 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:56.443 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-14T15:05:56.443 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph config generate-minimal-conf 2024-04-14T15:05:56.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:56 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:56.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:56 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:05:56.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:56 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.92:0/3058178067' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-14T15:05:56.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:56 smithi090 ceph-mon[30399]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:05:56.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:56 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:56.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:56 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:05:56.728 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:56 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:56.728 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:56 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:05:56.728 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:56 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.92:0/3058178067' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-14T15:05:56.728 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:56 smithi090 ceph-mon[41670]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:05:56.728 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:56 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:56.728 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:56 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:05:56.811 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:56 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:56.811 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:56 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:05:56.811 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:56 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.92:0/3058178067' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-14T15:05:56.811 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:56 smithi092 ceph-mon[33966]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:05:56.811 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:56 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:56.811 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:56 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:05:57.021 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:05:58.291 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:58 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:58.291 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:58 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:58.291 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:58 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:58.291 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:58 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:58.291 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:58 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:05:58.291 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:58 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:05:58.292 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:58 smithi090 ceph-mon[41670]: Updating smithi090:/etc/ceph/ceph.conf 2024-04-14T15:05:58.292 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:58 smithi090 ceph-mon[41670]: Updating smithi092:/etc/ceph/ceph.conf 2024-04-14T15:05:58.292 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:58 smithi090 ceph-mon[41670]: Updating smithi090:/var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T15:05:58.292 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:58 smithi090 ceph-mon[41670]: Updating smithi092:/var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T15:05:58.292 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:58 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:58.292 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:58 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:58.292 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:58 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:58.292 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:58 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:58.292 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:58 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:58.292 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:58 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:58.292 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:58 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:58.292 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:58 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:58.292 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:58 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:58.293 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:58 smithi090 ceph-mon[41670]: Reconfiguring mon.a (unknown last config time)... 2024-04-14T15:05:58.293 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:58 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T15:05:58.293 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:58 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T15:05:58.293 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:58 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:05:58.293 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:58 smithi090 ceph-mon[41670]: Reconfiguring daemon mon.a on smithi090 2024-04-14T15:05:58.293 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:58 smithi090 ceph-mon[41670]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:05:58.293 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:58 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:58.293 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:58 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:58.293 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:58 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:58.293 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:58 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:58.293 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:58 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:05:58.294 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:58 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:05:58.294 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:58 smithi090 ceph-mon[30399]: Updating smithi090:/etc/ceph/ceph.conf 2024-04-14T15:05:58.294 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:58 smithi090 ceph-mon[30399]: Updating smithi092:/etc/ceph/ceph.conf 2024-04-14T15:05:58.294 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:58 smithi090 ceph-mon[30399]: Updating smithi090:/var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T15:05:58.294 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:58 smithi090 ceph-mon[30399]: Updating smithi092:/var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T15:05:58.294 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:58 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:58.294 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:58 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:58.294 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:58 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:58.294 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:58 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:58.294 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:58 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:58.294 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:58 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:58.294 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:58 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:58.295 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:58 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:58.295 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:58 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:58.295 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:58 smithi090 ceph-mon[30399]: Reconfiguring mon.a (unknown last config time)... 2024-04-14T15:05:58.295 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:58 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T15:05:58.295 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:58 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T15:05:58.295 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:58 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:05:58.295 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:58 smithi090 ceph-mon[30399]: Reconfiguring daemon mon.a on smithi090 2024-04-14T15:05:58.295 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:58 smithi090 ceph-mon[30399]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:05:58.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:58 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:58.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:58 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:58.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:58 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:58.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:58 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:58.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:58 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:05:58.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:58 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:05:58.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:58 smithi092 ceph-mon[33966]: Updating smithi090:/etc/ceph/ceph.conf 2024-04-14T15:05:58.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:58 smithi092 ceph-mon[33966]: Updating smithi092:/etc/ceph/ceph.conf 2024-04-14T15:05:58.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:58 smithi092 ceph-mon[33966]: Updating smithi090:/var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T15:05:58.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:58 smithi092 ceph-mon[33966]: Updating smithi092:/var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T15:05:58.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:58 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:58.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:58 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:58.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:58 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:58.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:58 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:58.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:58 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:58.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:58 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:58.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:58 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:58.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:58 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:58.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:58 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:05:58.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:58 smithi092 ceph-mon[33966]: Reconfiguring mon.a (unknown last config time)... 2024-04-14T15:05:58.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:58 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T15:05:58.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:58 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T15:05:58.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:58 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:05:58.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:58 smithi092 ceph-mon[33966]: Reconfiguring daemon mon.a on smithi090 2024-04-14T15:05:58.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:58 smithi092 ceph-mon[33966]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:05:59.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:59.319+0000 7f23feb18700 1 -- 172.21.15.90:0/4245679698 >> v1:172.21.15.90:6789/0 conn(0x7f23f8108480 legacy=0x7f23f8108860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:05:59.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:59.319+0000 7f23feb18700 1 -- 172.21.15.90:0/4245679698 shutdown_connections 2024-04-14T15:05:59.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:59.319+0000 7f23feb18700 1 -- 172.21.15.90:0/4245679698 >> 172.21.15.90:0/4245679698 conn(0x7f23f80fc560 msgr2=0x7f23f80fe980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:05:59.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:59.319+0000 7f23feb18700 1 -- 172.21.15.90:0/4245679698 shutdown_connections 2024-04-14T15:05:59.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:59.319+0000 7f23feb18700 1 -- 172.21.15.90:0/4245679698 wait complete. 2024-04-14T15:05:59.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:59.320+0000 7f23feb18700 1 librados: starting msgr at 2024-04-14T15:05:59.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:59.320+0000 7f23feb18700 1 librados: starting objecter 2024-04-14T15:05:59.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:59.320+0000 7f23feb18700 1 Processor -- start 2024-04-14T15:05:59.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:59.320+0000 7f23feb18700 1 -- start start 2024-04-14T15:05:59.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:59.320+0000 7f23feb18700 1 librados: setting wanted keys 2024-04-14T15:05:59.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:59.320+0000 7f23feb18700 1 librados: calling monclient init 2024-04-14T15:05:59.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:59.320+0000 7f23feb18700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f23f81a7380 con 0x7f23f8108480 2024-04-14T15:05:59.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:59.320+0000 7f23feb18700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f23f81a85a0 con 0x7f23f8197b30 2024-04-14T15:05:59.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:59.320+0000 7f23feb18700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f23f81a97e0 con 0x7f23f81a36b0 2024-04-14T15:05:59.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:59.320+0000 7f23fc8b4700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f23f8108480 0x7f23f8197420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:54776/0 (socket says 172.21.15.90:54776) 2024-04-14T15:05:59.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:59.320+0000 7f23fc8b4700 1 -- 172.21.15.90:0/4031654467 learned_addr learned my addr 172.21.15.90:0/4031654467 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:05:59.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:59.320+0000 7f23f5ffb700 10 client.?.objecter ms_handle_connect 0x7f23f8108480 2024-04-14T15:05:59.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:59.320+0000 7f23f5ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:05:59.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:59.320+0000 7f23f5ffb700 10 client.?.objecter ms_handle_connect 0x7f23f81a36b0 2024-04-14T15:05:59.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:59.320+0000 7f23f5ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:05:59.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:59.320+0000 7f23f5ffb700 1 -- 172.21.15.90:0/4031654467 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3323067440 0 0) 0x7f23f81a7380 con 0x7f23f8108480 2024-04-14T15:05:59.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:59.320+0000 7f23f5ffb700 1 -- 172.21.15.90:0/4031654467 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f23e8003bf0 con 0x7f23f8108480 2024-04-14T15:05:59.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:59.320+0000 7f23f5ffb700 1 -- 172.21.15.90:0/4031654467 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1720011633 0 0) 0x7f23f81a97e0 con 0x7f23f81a36b0 2024-04-14T15:05:59.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:59.320+0000 7f23f5ffb700 1 -- 172.21.15.90:0/4031654467 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f23f81a7380 con 0x7f23f81a36b0 2024-04-14T15:05:59.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:59.321+0000 7f23f5ffb700 10 client.?.objecter ms_handle_connect 0x7f23f8197b30 2024-04-14T15:05:59.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:59.321+0000 7f23f5ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:05:59.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:59.321+0000 7f23f5ffb700 1 -- 172.21.15.90:0/4031654467 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2779154734 0 0) 0x7f23e8003bf0 con 0x7f23f8108480 2024-04-14T15:05:59.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:59.321+0000 7f23f5ffb700 1 -- 172.21.15.90:0/4031654467 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f23f81a97e0 con 0x7f23f8108480 2024-04-14T15:05:59.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:59.321+0000 7f23f5ffb700 1 -- 172.21.15.90:0/4031654467 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f23ec003ab0 con 0x7f23f8108480 2024-04-14T15:05:59.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:59.321+0000 7f23f5ffb700 1 -- 172.21.15.90:0/4031654467 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2213398119 0 0) 0x7f23f81a7380 con 0x7f23f81a36b0 2024-04-14T15:05:59.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:59.321+0000 7f23f5ffb700 1 -- 172.21.15.90:0/4031654467 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f23e8003bf0 con 0x7f23f81a36b0 2024-04-14T15:05:59.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:59.321+0000 7f23f5ffb700 1 -- 172.21.15.90:0/4031654467 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f23f0001b10 con 0x7f23f81a36b0 2024-04-14T15:05:59.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:59.321+0000 7f23f5ffb700 1 -- 172.21.15.90:0/4031654467 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 877832199 0 0) 0x7f23f81a85a0 con 0x7f23f8197b30 2024-04-14T15:05:59.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:59.321+0000 7f23f5ffb700 1 -- 172.21.15.90:0/4031654467 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f23f81a7380 con 0x7f23f8197b30 2024-04-14T15:05:59.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:59.321+0000 7f23f5ffb700 1 -- 172.21.15.90:0/4031654467 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3302733470 0 0) 0x7f23f81a97e0 con 0x7f23f8108480 2024-04-14T15:05:59.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:59.321+0000 7f23f5ffb700 1 -- 172.21.15.90:0/4031654467 >> v1:172.21.15.90:6790/0 conn(0x7f23f81a36b0 legacy=0x7f23f81a5aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:05:59.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:59.321+0000 7f23f5ffb700 1 -- 172.21.15.90:0/4031654467 >> v1:172.21.15.92:6789/0 conn(0x7f23f8197b30 legacy=0x7f23f81a1f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:05:59.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:59.321+0000 7f23f5ffb700 1 -- 172.21.15.90:0/4031654467 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23f81aaa20 con 0x7f23f8108480 2024-04-14T15:05:59.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:59.321+0000 7f23feb18700 1 -- 172.21.15.90:0/4031654467 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f23f81a9a10 con 0x7f23f8108480 2024-04-14T15:05:59.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:59.321+0000 7f23f5ffb700 1 -- 172.21.15.90:0/4031654467 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f23ec003c80 con 0x7f23f8108480 2024-04-14T15:05:59.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:59.321+0000 7f23f5ffb700 1 -- 172.21.15.90:0/4031654467 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f23ec005200 con 0x7f23f8108480 2024-04-14T15:05:59.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:59.321+0000 7f23feb18700 10 client.14196.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:05:59.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:59.321+0000 7f23feb18700 1 -- 172.21.15.90:0/4031654467 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f23f81a9ed0 con 0x7f23f8108480 2024-04-14T15:05:59.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:59.321+0000 7f23feb18700 1 librados: init done 2024-04-14T15:05:59.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:59.322+0000 7f23f5ffb700 1 -- 172.21.15.90:0/4031654467 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 13) ==== 46579+0+0 (unknown 2972057686 0 0) 0x7f23ec005490 con 0x7f23f8108480 2024-04-14T15:05:59.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:59.322+0000 7f23feb18700 1 -- 172.21.15.90:0/4031654467 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23f8064110 con 0x7f23f8108480 2024-04-14T15:05:59.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:59.322+0000 7f23f5ffb700 1 -- 172.21.15.90:0/4031654467 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(4..4 src has 1..4) ==== 1069+0+0 (unknown 1732540447 0 0) 0x7f23ec0481c0 con 0x7f23f8108480 2024-04-14T15:05:59.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:59.322+0000 7f23f5ffb700 10 client.14196.objecter ms_dispatch 0x7f23f8001550 osd_map(4..4 src has 1..4) 2024-04-14T15:05:59.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:59.322+0000 7f23f5ffb700 3 client.14196.objecter handle_osd_map got epochs [4,4] > 0 2024-04-14T15:05:59.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:59.322+0000 7f23f5ffb700 3 client.14196.objecter handle_osd_map decoding full epoch 4 2024-04-14T15:05:59.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:59.322+0000 7f23f5ffb700 20 client.14196.objecter dump_active .. 0 homeless 2024-04-14T15:05:59.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:59.323+0000 7f23f5ffb700 10 client.14196.objecter ms_handle_connect 0x7f23e803c350 2024-04-14T15:05:59.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:59.327+0000 7f23f5ffb700 1 -- 172.21.15.90:0/4031654467 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f23ec017840 con 0x7f23f8108480 2024-04-14T15:05:59.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:59.527+0000 7f23feb18700 1 -- 172.21.15.90:0/4031654467 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f23f810d0c0 con 0x7f23f8108480 2024-04-14T15:05:59.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:59.527+0000 7f23f5ffb700 1 -- 172.21.15.90:0/4031654467 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+190 (unknown 2189217731 0 296491422) 0x7f23ec017140 con 0x7f23f8108480 2024-04-14T15:05:59.528 INFO:teuthology.orchestra.run.smithi090.stdout:# minimal ceph.conf for 0e288098-fa70-11ee-bc8e-c7b262605968 2024-04-14T15:05:59.528 INFO:teuthology.orchestra.run.smithi090.stdout:[global] 2024-04-14T15:05:59.528 INFO:teuthology.orchestra.run.smithi090.stdout: fsid = 0e288098-fa70-11ee-bc8e-c7b262605968 2024-04-14T15:05:59.528 INFO:teuthology.orchestra.run.smithi090.stdout: mon_host = 172.21.15.90:6789/0 172.21.15.92:6789/0 v1:172.21.15.90:6790/0 2024-04-14T15:05:59.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:59.529+0000 7f23feb18700 10 librados: watch_flush enter 2024-04-14T15:05:59.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:59.529+0000 7f23feb18700 10 librados: watch_flush exit 2024-04-14T15:05:59.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:59.529+0000 7f23feb18700 20 client.14196.objecter shutdown clearing up homeless session... 2024-04-14T15:05:59.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:59.529+0000 7f23feb18700 10 client.14196.objecter successfully canceled tick 2024-04-14T15:05:59.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:59.529+0000 7f23feb18700 1 -- 172.21.15.90:0/4031654467 >> v1:172.21.15.90:6800/491071118 conn(0x7f23e803c350 legacy=0x7f23e803e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:05:59.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:59.529+0000 7f23feb18700 1 -- 172.21.15.90:0/4031654467 >> v1:172.21.15.90:6789/0 conn(0x7f23f8108480 legacy=0x7f23f8197420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:05:59.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:59.529+0000 7f23feb18700 1 -- 172.21.15.90:0/4031654467 shutdown_connections 2024-04-14T15:05:59.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:59.529+0000 7f23feb18700 1 -- 172.21.15.90:0/4031654467 >> 172.21.15.90:0/4031654467 conn(0x7f23f80fc560 msgr2=0x7f23f81099d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:05:59.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:59.529+0000 7f23feb18700 1 -- 172.21.15.90:0/4031654467 shutdown_connections 2024-04-14T15:05:59.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:59.529+0000 7f23feb18700 1 -- 172.21.15.90:0/4031654467 wait complete. 2024-04-14T15:05:59.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:05:59.529+0000 7f23feb18700 1 librados: shutdown 2024-04-14T15:06:00.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:59 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/4031654467' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:06:00.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:59 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:00.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:05:59 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:00.148 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:59 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/4031654467' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:06:00.148 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:59 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:00.148 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:05:59 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:00.149 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:59 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/4031654467' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:06:00.149 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:59 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:00.149 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:05:59 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:00.453 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-14T15:06:00.454 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-14T15:06:00.454 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/ceph/ceph.conf 2024-04-14T15:06:00.484 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-14T15:06:00.484 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-14T15:06:00.551 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-04-14T15:06:00.551 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/etc/ceph/ceph.conf 2024-04-14T15:06:00.587 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-04-14T15:06:00.587 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-14T15:06:00.657 INFO:tasks.cephadm:Adding mgr.y on smithi090 2024-04-14T15:06:00.657 INFO:tasks.cephadm:Adding mgr.x on smithi092 2024-04-14T15:06:00.657 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph orch apply mgr '2;smithi090=y;smithi092=x' 2024-04-14T15:06:00.932 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:06:01.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:00 smithi092 ceph-mon[33966]: Reconfiguring mon.c (monmap changed)... 2024-04-14T15:06:01.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:00 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T15:06:01.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:00 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T15:06:01.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:00 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:06:01.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:00 smithi092 ceph-mon[33966]: Reconfiguring daemon mon.c on smithi090 2024-04-14T15:06:01.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:00 smithi092 ceph-mon[33966]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:06:01.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:00 smithi090 ceph-mon[30399]: Reconfiguring mon.c (monmap changed)... 2024-04-14T15:06:01.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:00 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T15:06:01.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:00 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T15:06:01.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:00 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:06:01.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:00 smithi090 ceph-mon[30399]: Reconfiguring daemon mon.c on smithi090 2024-04-14T15:06:01.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:00 smithi090 ceph-mon[30399]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:06:01.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:00 smithi090 ceph-mon[41670]: Reconfiguring mon.c (monmap changed)... 2024-04-14T15:06:01.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:00 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T15:06:01.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:00 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T15:06:01.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:00 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:06:01.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:00 smithi090 ceph-mon[41670]: Reconfiguring daemon mon.c on smithi090 2024-04-14T15:06:01.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:00 smithi090 ceph-mon[41670]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:06:02.415 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.412+0000 7f90d309d700 1 -- 172.21.15.92:0/4031202302 >> v1:172.21.15.90:6789/0 conn(0x7f90cc101480 legacy=0x7f90cc1018a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:06:02.415 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.412+0000 7f90d309d700 1 -- 172.21.15.92:0/4031202302 shutdown_connections 2024-04-14T15:06:02.415 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.412+0000 7f90d309d700 1 -- 172.21.15.92:0/4031202302 >> 172.21.15.92:0/4031202302 conn(0x7f90cc0fca20 msgr2=0x7f90cc0fee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:06:02.415 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.412+0000 7f90d309d700 1 -- 172.21.15.92:0/4031202302 shutdown_connections 2024-04-14T15:06:02.416 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.412+0000 7f90d309d700 1 -- 172.21.15.92:0/4031202302 wait complete. 2024-04-14T15:06:02.416 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.412+0000 7f90d309d700 1 librados: starting msgr at 2024-04-14T15:06:02.416 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.412+0000 7f90d309d700 1 librados: starting objecter 2024-04-14T15:06:02.416 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.412+0000 7f90d309d700 1 Processor -- start 2024-04-14T15:06:02.416 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.412+0000 7f90d309d700 1 -- start start 2024-04-14T15:06:02.416 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.412+0000 7f90d309d700 1 librados: setting wanted keys 2024-04-14T15:06:02.416 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.412+0000 7f90d309d700 1 librados: calling monclient init 2024-04-14T15:06:02.416 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.412+0000 7f90d309d700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f90cc1a18b0 con 0x7f90cc101480 2024-04-14T15:06:02.416 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.412+0000 7f90d309d700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f90cc1a2ad0 con 0x7f90cc19dc40 2024-04-14T15:06:02.416 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.412+0000 7f90d309d700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f90cc1a3d10 con 0x7f90cc196750 2024-04-14T15:06:02.416 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.413+0000 7f90d0e39700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f90cc101480 0x7f90cc196040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.92:33834/0 (socket says 172.21.15.92:33834) 2024-04-14T15:06:02.416 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.413+0000 7f90d0e39700 1 -- 172.21.15.92:0/4180790710 learned_addr learned my addr 172.21.15.92:0/4180790710 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-04-14T15:06:02.416 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.413+0000 7f90c9ffb700 10 client.?.objecter ms_handle_connect 0x7f90cc19dc40 2024-04-14T15:06:02.417 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.413+0000 7f90c9ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:06:02.417 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.413+0000 7f90c9ffb700 10 client.?.objecter ms_handle_connect 0x7f90cc101480 2024-04-14T15:06:02.417 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.413+0000 7f90c9ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:06:02.417 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.413+0000 7f90c9ffb700 10 client.?.objecter ms_handle_connect 0x7f90cc196750 2024-04-14T15:06:02.417 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.413+0000 7f90c9ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:06:02.417 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.413+0000 7f90c9ffb700 1 -- 172.21.15.92:0/4180790710 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4246555699 0 0) 0x7f90cc1a2ad0 con 0x7f90cc19dc40 2024-04-14T15:06:02.417 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.413+0000 7f90c9ffb700 1 -- 172.21.15.92:0/4180790710 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f90b4003c40 con 0x7f90cc19dc40 2024-04-14T15:06:02.417 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.413+0000 7f90c9ffb700 1 -- 172.21.15.92:0/4180790710 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2862680393 0 0) 0x7f90cc1a18b0 con 0x7f90cc101480 2024-04-14T15:06:02.417 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.413+0000 7f90c9ffb700 1 -- 172.21.15.92:0/4180790710 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f90cc1a2ad0 con 0x7f90cc101480 2024-04-14T15:06:02.417 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.413+0000 7f90c9ffb700 1 -- 172.21.15.92:0/4180790710 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2243548699 0 0) 0x7f90cc1a3d10 con 0x7f90cc196750 2024-04-14T15:06:02.417 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.413+0000 7f90c9ffb700 1 -- 172.21.15.92:0/4180790710 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f90cc1a18b0 con 0x7f90cc196750 2024-04-14T15:06:02.417 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.413+0000 7f90c9ffb700 1 -- 172.21.15.92:0/4180790710 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4262123263 0 0) 0x7f90b4003c40 con 0x7f90cc19dc40 2024-04-14T15:06:02.417 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.413+0000 7f90c9ffb700 1 -- 172.21.15.92:0/4180790710 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f90cc1a3d10 con 0x7f90cc19dc40 2024-04-14T15:06:02.418 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.413+0000 7f90c9ffb700 1 -- 172.21.15.92:0/4180790710 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f90bc001810 con 0x7f90cc19dc40 2024-04-14T15:06:02.418 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.413+0000 7f90c9ffb700 1 -- 172.21.15.92:0/4180790710 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1786554198 0 0) 0x7f90cc1a2ad0 con 0x7f90cc101480 2024-04-14T15:06:02.418 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.413+0000 7f90c9ffb700 1 -- 172.21.15.92:0/4180790710 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f90b4003c40 con 0x7f90cc101480 2024-04-14T15:06:02.418 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.413+0000 7f90c9ffb700 1 -- 172.21.15.92:0/4180790710 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f90c4004850 con 0x7f90cc101480 2024-04-14T15:06:02.418 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.413+0000 7f90c9ffb700 1 -- 172.21.15.92:0/4180790710 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 718208135 0 0) 0x7f90cc1a18b0 con 0x7f90cc196750 2024-04-14T15:06:02.418 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.413+0000 7f90c9ffb700 1 -- 172.21.15.92:0/4180790710 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f90cc1a2ad0 con 0x7f90cc196750 2024-04-14T15:06:02.418 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.413+0000 7f90c9ffb700 1 -- 172.21.15.92:0/4180790710 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f90c0001750 con 0x7f90cc196750 2024-04-14T15:06:02.418 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.413+0000 7f90c9ffb700 1 -- 172.21.15.92:0/4180790710 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1519191122 0 0) 0x7f90cc1a3d10 con 0x7f90cc19dc40 2024-04-14T15:06:02.418 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.413+0000 7f90c9ffb700 1 -- 172.21.15.92:0/4180790710 >> v1:172.21.15.90:6790/0 conn(0x7f90cc196750 legacy=0x7f90cc19c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:06:02.418 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.413+0000 7f90c9ffb700 1 -- 172.21.15.92:0/4180790710 >> v1:172.21.15.90:6789/0 conn(0x7f90cc101480 legacy=0x7f90cc196040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:06:02.418 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.413+0000 7f90c9ffb700 1 -- 172.21.15.92:0/4180790710 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90cc1a4f50 con 0x7f90cc19dc40 2024-04-14T15:06:02.418 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.414+0000 7f90d309d700 1 -- 172.21.15.92:0/4180790710 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f90cc1a1ae0 con 0x7f90cc19dc40 2024-04-14T15:06:02.418 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.414+0000 7f90c9ffb700 1 -- 172.21.15.92:0/4180790710 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f90bc003be0 con 0x7f90cc19dc40 2024-04-14T15:06:02.418 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.414+0000 7f90c9ffb700 1 -- 172.21.15.92:0/4180790710 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f90bc004b60 con 0x7f90cc19dc40 2024-04-14T15:06:02.418 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.414+0000 7f90d309d700 10 client.24103.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:06:02.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.414+0000 7f90d309d700 1 -- 172.21.15.92:0/4180790710 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f90cc1a2020 con 0x7f90cc19dc40 2024-04-14T15:06:02.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.414+0000 7f90d309d700 1 librados: init done 2024-04-14T15:06:02.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.414+0000 7f90c9ffb700 1 -- 172.21.15.92:0/4180790710 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 13) ==== 46579+0+0 (unknown 2972057686 0 0) 0x7f90bc0103f0 con 0x7f90cc19dc40 2024-04-14T15:06:02.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.415+0000 7f90c9ffb700 1 -- 172.21.15.92:0/4180790710 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(4..4 src has 1..4) ==== 1069+0+0 (unknown 1732540447 0 0) 0x7f90bc047a90 con 0x7f90cc19dc40 2024-04-14T15:06:02.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.415+0000 7f90c9ffb700 10 client.24103.objecter ms_dispatch 0x7f90cc001550 osd_map(4..4 src has 1..4) 2024-04-14T15:06:02.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.415+0000 7f90c9ffb700 3 client.24103.objecter handle_osd_map got epochs [4,4] > 0 2024-04-14T15:06:02.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.415+0000 7f90c9ffb700 3 client.24103.objecter handle_osd_map decoding full epoch 4 2024-04-14T15:06:02.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.415+0000 7f90c9ffb700 20 client.24103.objecter dump_active .. 0 homeless 2024-04-14T15:06:02.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.415+0000 7f90d309d700 1 -- 172.21.15.92:0/4180790710 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90cc064110 con 0x7f90cc19dc40 2024-04-14T15:06:02.420 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.415+0000 7f90c9ffb700 10 client.24103.objecter ms_handle_connect 0x7f90b403e680 2024-04-14T15:06:02.420 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.420+0000 7f90c9ffb700 1 -- 172.21.15.92:0/4180790710 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f90bc016200 con 0x7f90cc19dc40 2024-04-14T15:06:02.635 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.634+0000 7f90d309d700 1 -- 172.21.15.92:0/4180790710 --> v1:172.21.15.90:6800/491071118 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi090=y;smithi092=x", "target": ["mon-mgr", ""]}) -- 0x7f90cc1a2380 con 0x7f90b403e680 2024-04-14T15:06:02.641 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.640+0000 7f90c9ffb700 1 -- 172.21.15.92:0/4180790710 <== mgr.14148 v1:172.21.15.90:6800/491071118 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (unknown 0 0 325935098) 0x7f90cc1a2380 con 0x7f90b403e680 2024-04-14T15:06:02.641 INFO:teuthology.orchestra.run.smithi092.stdout:Scheduled mgr update... 2024-04-14T15:06:02.643 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.642+0000 7f90d309d700 10 librados: watch_flush enter 2024-04-14T15:06:02.643 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.642+0000 7f90d309d700 10 librados: watch_flush exit 2024-04-14T15:06:02.643 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.642+0000 7f90d309d700 20 client.24103.objecter shutdown clearing up homeless session... 2024-04-14T15:06:02.643 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.642+0000 7f90d309d700 10 client.24103.objecter successfully canceled tick 2024-04-14T15:06:02.644 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.643+0000 7f90d309d700 1 -- 172.21.15.92:0/4180790710 >> v1:172.21.15.90:6800/491071118 conn(0x7f90b403e680 legacy=0x7f90b4040b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:06:02.644 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.643+0000 7f90d309d700 1 -- 172.21.15.92:0/4180790710 >> v1:172.21.15.92:6789/0 conn(0x7f90cc19dc40 legacy=0x7f90cc1a0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:06:02.644 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.643+0000 7f90d309d700 1 -- 172.21.15.92:0/4180790710 shutdown_connections 2024-04-14T15:06:02.644 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.643+0000 7f90d309d700 1 -- 172.21.15.92:0/4180790710 >> 172.21.15.92:0/4180790710 conn(0x7f90cc0fca20 msgr2=0x7f90cc0fee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:06:02.644 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.643+0000 7f90d309d700 1 -- 172.21.15.92:0/4180790710 shutdown_connections 2024-04-14T15:06:02.644 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.643+0000 7f90d309d700 1 -- 172.21.15.92:0/4180790710 wait complete. 2024-04-14T15:06:02.644 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:06:02.643+0000 7f90d309d700 1 librados: shutdown 2024-04-14T15:06:02.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:02 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:02.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:02 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:02.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:02 smithi092 ceph-mon[33966]: Reconfiguring mon.b (monmap changed)... 2024-04-14T15:06:02.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:02 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T15:06:02.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:02 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T15:06:02.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:02 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:06:02.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:02 smithi092 ceph-mon[33966]: Reconfiguring daemon mon.b on smithi092 2024-04-14T15:06:02.873 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:02 smithi092 ceph-mon[33966]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:06:02.873 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:02 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:02.941 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:02 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:02.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:02 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:02.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:02 smithi090 ceph-mon[30399]: Reconfiguring mon.b (monmap changed)... 2024-04-14T15:06:02.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:02 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T15:06:02.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:02 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T15:06:02.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:02 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:06:02.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:02 smithi090 ceph-mon[30399]: Reconfiguring daemon mon.b on smithi092 2024-04-14T15:06:02.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:02 smithi090 ceph-mon[30399]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:06:02.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:02 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:02.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:02 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:02.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:02 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:02.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:02 smithi090 ceph-mon[41670]: Reconfiguring mon.b (monmap changed)... 2024-04-14T15:06:02.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:02 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T15:06:02.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:02 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T15:06:02.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:02 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:06:02.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:02 smithi090 ceph-mon[41670]: Reconfiguring daemon mon.b on smithi092 2024-04-14T15:06:02.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:02 smithi090 ceph-mon[41670]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:06:02.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:02 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:03.247 DEBUG:teuthology.orchestra.run.smithi092:mgr.x> sudo journalctl -f -n 0 -u ceph-0e288098-fa70-11ee-bc8e-c7b262605968@mgr.x.service 2024-04-14T15:06:03.249 INFO:tasks.cephadm:Deploying OSDs... 2024-04-14T15:06:03.249 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-14T15:06:03.250 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/scratch_devs of=/dev/stdout 2024-04-14T15:06:03.268 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-14T15:06:03.269 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_1 2024-04-14T15:06:03.326 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-14T15:06:03.326 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-14T15:06:03.326 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-04-14T15:06:03.326 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-14T15:06:03.326 INFO:teuthology.orchestra.run.smithi090.stdout:Context: system_u:object_r:device_t:s0 2024-04-14T15:06:03.326 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2024-04-14 15:05:49.836235409 +0000 2024-04-14T15:06:03.326 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2024-04-14 14:59:07.233143664 +0000 2024-04-14T15:06:03.326 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2024-04-14 14:59:07.233143664 +0000 2024-04-14T15:06:03.327 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: 2024-04-14 14:59:07.233143664 +0000 2024-04-14T15:06:03.327 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-14T15:06:03.396 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2024-04-14T15:06:03.396 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2024-04-14T15:06:03.396 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.000203533 s, 2.5 MB/s 2024-04-14T15:06:03.397 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-14T15:06:03.455 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_2 2024-04-14T15:06:03.512 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-14T15:06:03.512 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-14T15:06:03.512 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-14T15:06:03.512 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-14T15:06:03.512 INFO:teuthology.orchestra.run.smithi090.stdout:Context: system_u:object_r:device_t:s0 2024-04-14T15:06:03.512 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2024-04-14 15:05:49.836235409 +0000 2024-04-14T15:06:03.512 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2024-04-14 14:59:07.572138266 +0000 2024-04-14T15:06:03.512 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2024-04-14 14:59:07.572138266 +0000 2024-04-14T15:06:03.512 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: 2024-04-14 14:59:07.572138266 +0000 2024-04-14T15:06:03.512 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-14T15:06:03.580 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2024-04-14T15:06:03.580 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2024-04-14T15:06:03.580 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.000199985 s, 2.6 MB/s 2024-04-14T15:06:03.581 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-14T15:06:03.638 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_3 2024-04-14T15:06:03.694 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-14T15:06:03.819 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-14T15:06:03.820 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-04-14T15:06:03.820 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-14T15:06:03.820 INFO:teuthology.orchestra.run.smithi090.stdout:Context: system_u:object_r:device_t:s0 2024-04-14T15:06:03.820 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2024-04-14 15:05:49.836235409 +0000 2024-04-14T15:06:03.820 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2024-04-14 14:59:07.961132073 +0000 2024-04-14T15:06:03.820 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2024-04-14 14:59:07.961132073 +0000 2024-04-14T15:06:03.820 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: 2024-04-14 14:59:07.961132073 +0000 2024-04-14T15:06:03.820 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-14T15:06:03.848 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2024-04-14T15:06:03.848 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2024-04-14T15:06:03.848 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.00022325 s, 2.3 MB/s 2024-04-14T15:06:03.850 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-14T15:06:03.907 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_4 2024-04-14T15:06:03.941 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:03 smithi090 ceph-mon[41670]: from='client.24103 v1:172.21.15.92:0/4180790710' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi090=y;smithi092=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:06:03.941 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:03 smithi090 ceph-mon[41670]: Saving service mgr spec with placement smithi090=y;smithi092=x;count:2 2024-04-14T15:06:03.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:03 smithi090 ceph-mon[30399]: from='client.24103 v1:172.21.15.92:0/4180790710' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi090=y;smithi092=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:06:03.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:03 smithi090 ceph-mon[30399]: Saving service mgr spec with placement smithi090=y;smithi092=x;count:2 2024-04-14T15:06:03.948 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:03 smithi092 ceph-mon[33966]: from='client.24103 v1:172.21.15.92:0/4180790710' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi090=y;smithi092=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:06:03.948 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:03 smithi092 ceph-mon[33966]: Saving service mgr spec with placement smithi090=y;smithi092=x;count:2 2024-04-14T15:06:03.957 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-14T15:06:03.957 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-14T15:06:03.958 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-04-14T15:06:03.958 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-14T15:06:03.958 INFO:teuthology.orchestra.run.smithi090.stdout:Context: system_u:object_r:device_t:s0 2024-04-14T15:06:03.958 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2024-04-14 15:05:49.837235393 +0000 2024-04-14T15:06:03.958 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2024-04-14 14:59:08.312126484 +0000 2024-04-14T15:06:03.958 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2024-04-14 14:59:08.312126484 +0000 2024-04-14T15:06:03.958 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: 2024-04-14 14:59:08.312126484 +0000 2024-04-14T15:06:03.958 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-14T15:06:04.025 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2024-04-14T15:06:04.025 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2024-04-14T15:06:04.025 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.000190059 s, 2.7 MB/s 2024-04-14T15:06:04.027 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-14T15:06:04.083 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-04-14T15:06:04.083 DEBUG:teuthology.orchestra.run.smithi092:> dd if=/scratch_devs of=/dev/stdout 2024-04-14T15:06:04.098 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-14T15:06:04.098 DEBUG:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_1 2024-04-14T15:06:04.153 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-14T15:06:04.153 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-14T15:06:04.153 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-04-14T15:06:04.153 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-14T15:06:04.153 INFO:teuthology.orchestra.run.smithi092.stdout:Context: system_u:object_r:device_t:s0 2024-04-14T15:06:04.154 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2024-04-14 15:05:41.007123073 +0000 2024-04-14T15:06:04.154 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2024-04-14 14:59:02.578576135 +0000 2024-04-14T15:06:04.154 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2024-04-14 14:59:02.578576135 +0000 2024-04-14T15:06:04.154 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: 2024-04-14 14:59:02.578576135 +0000 2024-04-14T15:06:04.154 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-14T15:06:04.219 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2024-04-14T15:06:04.219 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2024-04-14T15:06:04.219 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.000196117 s, 2.6 MB/s 2024-04-14T15:06:04.220 DEBUG:teuthology.orchestra.run.smithi092:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-14T15:06:04.276 DEBUG:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_2 2024-04-14T15:06:04.331 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-14T15:06:04.331 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-14T15:06:04.331 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-14T15:06:04.331 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-14T15:06:04.331 INFO:teuthology.orchestra.run.smithi092.stdout:Context: system_u:object_r:device_t:s0 2024-04-14T15:06:04.331 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2024-04-14 15:05:41.007123073 +0000 2024-04-14T15:06:04.332 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2024-04-14 14:59:02.935565567 +0000 2024-04-14T15:06:04.332 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2024-04-14 14:59:02.935565567 +0000 2024-04-14T15:06:04.332 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: 2024-04-14 14:59:02.935565567 +0000 2024-04-14T15:06:04.332 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-14T15:06:04.397 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2024-04-14T15:06:04.397 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2024-04-14T15:06:04.397 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.00020089 s, 2.5 MB/s 2024-04-14T15:06:04.398 DEBUG:teuthology.orchestra.run.smithi092:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-14T15:06:04.454 DEBUG:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_3 2024-04-14T15:06:04.509 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-14T15:06:04.509 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-14T15:06:04.509 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-04-14T15:06:04.509 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-14T15:06:04.509 INFO:teuthology.orchestra.run.smithi092.stdout:Context: system_u:object_r:device_t:s0 2024-04-14T15:06:04.509 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2024-04-14 15:05:41.007123073 +0000 2024-04-14T15:06:04.509 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2024-04-14 14:59:03.260555945 +0000 2024-04-14T15:06:04.510 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2024-04-14 14:59:03.260555945 +0000 2024-04-14T15:06:04.510 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: 2024-04-14 14:59:03.260555945 +0000 2024-04-14T15:06:04.510 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-14T15:06:04.576 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2024-04-14T15:06:04.576 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2024-04-14T15:06:04.576 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.000200482 s, 2.6 MB/s 2024-04-14T15:06:04.577 DEBUG:teuthology.orchestra.run.smithi092:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-14T15:06:04.633 DEBUG:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_4 2024-04-14T15:06:04.688 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-14T15:06:04.688 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-14T15:06:04.688 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-04-14T15:06:04.688 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-14T15:06:04.689 INFO:teuthology.orchestra.run.smithi092.stdout:Context: system_u:object_r:device_t:s0 2024-04-14T15:06:04.689 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2024-04-14 15:05:41.008123043 +0000 2024-04-14T15:06:04.689 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2024-04-14 14:59:03.612545524 +0000 2024-04-14T15:06:04.689 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2024-04-14 14:59:03.612545524 +0000 2024-04-14T15:06:04.689 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: 2024-04-14 14:59:03.612545524 +0000 2024-04-14T15:06:04.689 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-14T15:06:04.753 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2024-04-14T15:06:04.754 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2024-04-14T15:06:04.754 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.000231164 s, 2.2 MB/s 2024-04-14T15:06:04.755 DEBUG:teuthology.orchestra.run.smithi092:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-14T15:06:04.811 INFO:tasks.cephadm:Deploying osd.0 on smithi090 with /dev/vg_nvme/lv_4... 2024-04-14T15:06:04.811 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-14T15:06:05.055 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:06:05.176 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:04 smithi090 ceph-mon[30399]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:06:05.176 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:04 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:05.176 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:04 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:05.176 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:04 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:06:05.176 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:04 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:06:05.176 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:04 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:06:05.176 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:04 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:05.176 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:04 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T15:06:05.177 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:04 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-14T15:06:05.177 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:04 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-14T15:06:05.177 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:04 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:06:05.177 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:04 smithi090 ceph-mon[41670]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:06:05.177 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:04 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:05.177 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:04 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:05.177 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:04 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:06:05.177 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:04 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:06:05.177 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:04 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:06:05.177 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:04 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:05.177 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:04 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T15:06:05.177 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:04 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-14T15:06:05.177 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:04 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-14T15:06:05.178 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:04 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:06:05.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:04 smithi092 ceph-mon[33966]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:06:05.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:04 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:05.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:04 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:05.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:04 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:06:05.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:04 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:06:05.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:04 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:06:05.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:04 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:05.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:04 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T15:06:05.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:04 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-14T15:06:05.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:04 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-14T15:06:05.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:04 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:06:05.997 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:05 smithi092 ceph-mon[33966]: Deploying daemon mgr.x on smithi092 2024-04-14T15:06:06.284 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:06:06 smithi092 systemd[1]: Starting Ceph mgr.x for 0e288098-fa70-11ee-bc8e-c7b262605968... 2024-04-14T15:06:06.284 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:06:06 smithi092 podman[36029]: 2024-04-14 15:06:06.282623198 +0000 UTC m=+0.107549907 container create 503f8922bf5434a3ec443850ec202ae388254ad5c982bd674a19ec1e1176deb8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-x, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, RELEASE=main-a971fd5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0) 2024-04-14T15:06:06.285 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:05 smithi090 ceph-mon[30399]: Deploying daemon mgr.x on smithi092 2024-04-14T15:06:06.285 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:05 smithi090 ceph-mon[41670]: Deploying daemon mgr.x on smithi092 2024-04-14T15:06:06.592 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:06:06 smithi092 podman[36029]: 2024-04-14 15:06:06.19277295 +0000 UTC m=+0.017699615 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608 2024-04-14T15:06:06.593 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:06:06 smithi092 podman[36029]: 2024-04-14 15:06:06.476944999 +0000 UTC m=+0.301871663 container init 503f8922bf5434a3ec443850ec202ae388254ad5c982bd674a19ec1e1176deb8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-x, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a971fd5, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-14T15:06:06.593 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:06:06 smithi092 podman[36029]: 2024-04-14 15:06:06.482217325 +0000 UTC m=+0.307143989 container start 503f8922bf5434a3ec443850ec202ae388254ad5c982bd674a19ec1e1176deb8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-x, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=main-a971fd5, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True) 2024-04-14T15:06:06.872 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:06:06 smithi092 bash[36029]: 503f8922bf5434a3ec443850ec202ae388254ad5c982bd674a19ec1e1176deb8 2024-04-14T15:06:06.872 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:06:06 smithi092 systemd[1]: Started Ceph mgr.x for 0e288098-fa70-11ee-bc8e-c7b262605968. 2024-04-14T15:06:07.127 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-14T15:06:07.127 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-14T15:06:07.127 INFO:teuthology.orchestra.run.smithi090.stderr: stderr: 10+0 records in 2024-04-14T15:06:07.128 INFO:teuthology.orchestra.run.smithi090.stderr:10+0 records out 2024-04-14T15:06:07.128 INFO:teuthology.orchestra.run.smithi090.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0114061 s, 919 MB/s 2024-04-14T15:06:07.128 INFO:teuthology.orchestra.run.smithi090.stderr: stderr: 2024-04-14T15:06:07.128 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping successful for: 2024-04-14T15:06:07.372 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:06:07 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-x[36044]: 2024-04-14T15:06:07.038+0000 7f84906b4200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-14T15:06:07.372 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:06:07 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-x[36044]: 2024-04-14T15:06:07.134+0000 7f84906b4200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-14T15:06:07.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:06 smithi092 ceph-mon[33966]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:06:07.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:06 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:07.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:06 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:07.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:06 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:07.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:06 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:07.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:06 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:06:07.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:06 smithi090 ceph-mon[30399]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:06:07.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:06 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:07.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:06 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:07.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:06 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:07.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:06 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:07.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:06 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:06:07.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:06 smithi090 ceph-mon[41670]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:06:07.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:06 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:07.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:06 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:07.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:06 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:07.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:06 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:07.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:06 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:06:07.776 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph orch daemon add osd smithi090:vg_nvme/lv_4 2024-04-14T15:06:07.872 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:06:07 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-x[36044]: 2024-04-14T15:06:07.810+0000 7f84906b4200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-14T15:06:08.015 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:06:08.234 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:06:07 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-x[36044]: 2024-04-14T15:06:07.902+0000 7f84906b4200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-14T15:06:08.234 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:06:08 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-x[36044]: 2024-04-14T15:06:08.049+0000 7f84906b4200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-14T15:06:08.234 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:06:08 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-x[36044]: 2024-04-14T15:06:08.149+0000 7f84906b4200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-14T15:06:08.234 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:06:08 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-x[36044]: 2024-04-14T15:06:08.233+0000 7f84906b4200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-14T15:06:08.777 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:06:08 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-x[36044]: 2024-04-14T15:06:08.492+0000 7f84906b4200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-14T15:06:08.777 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:06:08 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-x[36044]: 2024-04-14T15:06:08.599+0000 7f84906b4200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-14T15:06:08.777 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:06:08 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-x[36044]: 2024-04-14T15:06:08.683+0000 7f84906b4200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-14T15:06:08.778 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:06:08 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-x[36044]: 2024-04-14T15:06:08.776+0000 7f84906b4200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-14T15:06:09.058 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:09 smithi090 ceph-mon[41670]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:06:09.058 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:09 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:09.058 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:09 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:09.058 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:09 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:06:09.058 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:09 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:06:09.058 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:09 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:09.058 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:09 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T15:06:09.058 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:09 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-14T15:06:09.059 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:09 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:06:09.059 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:09 smithi090 ceph-mon[30399]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:06:09.059 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:09 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:09.059 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:09 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:09.059 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:09 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:06:09.059 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:09 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:06:09.059 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:09 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:09.059 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:09 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T15:06:09.059 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:09 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-14T15:06:09.059 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:09 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:06:09.122 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:06:08 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-x[36044]: 2024-04-14T15:06:08.949+0000 7f84906b4200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-14T15:06:09.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:09 smithi092 ceph-mon[33966]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:06:09.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:09 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:09.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:09 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:09.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:09 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:06:09.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:09 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:06:09.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:09 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:09.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:09 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T15:06:09.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:09 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-14T15:06:09.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:09 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:06:10.356 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:10 smithi092 ceph-mon[33966]: Reconfiguring mgr.y (unknown last config time)... 2024-04-14T15:06:10.356 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:10 smithi092 ceph-mon[33966]: Reconfiguring daemon mgr.y on smithi090 2024-04-14T15:06:10.356 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:06:09 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-x[36044]: 2024-04-14T15:06:09.902+0000 7f84906b4200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-14T15:06:10.356 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:06:10 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-x[36044]: 2024-04-14T15:06:10.030+0000 7f84906b4200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-14T15:06:10.357 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:06:10 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-x[36044]: 2024-04-14T15:06:10.355+0000 7f84906b4200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-14T15:06:10.360 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:10 smithi090 ceph-mon[30399]: Reconfiguring mgr.y (unknown last config time)... 2024-04-14T15:06:10.360 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:10 smithi090 ceph-mon[30399]: Reconfiguring daemon mgr.y on smithi090 2024-04-14T15:06:10.360 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:10 smithi090 ceph-mon[41670]: Reconfiguring mgr.y (unknown last config time)... 2024-04-14T15:06:10.360 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:10 smithi090 ceph-mon[41670]: Reconfiguring daemon mgr.y on smithi090 2024-04-14T15:06:10.622 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:06:10 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-x[36044]: 2024-04-14T15:06:10.442+0000 7f84906b4200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-14T15:06:10.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:10.704+0000 7f8e50656700 1 -- 172.21.15.90:0/449044981 >> v1:172.21.15.90:6790/0 conn(0x7f8e48106d40 legacy=0x7f8e48109190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:06:10.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:10.704+0000 7f8e50656700 1 -- 172.21.15.90:0/449044981 shutdown_connections 2024-04-14T15:06:10.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:10.704+0000 7f8e50656700 1 -- 172.21.15.90:0/449044981 >> 172.21.15.90:0/449044981 conn(0x7f8e480fded0 msgr2=0x7f8e481002f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:06:10.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:10.704+0000 7f8e50656700 1 -- 172.21.15.90:0/449044981 shutdown_connections 2024-04-14T15:06:10.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:10.704+0000 7f8e50656700 1 -- 172.21.15.90:0/449044981 wait complete. 2024-04-14T15:06:10.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:10.705+0000 7f8e50656700 1 librados: starting msgr at 2024-04-14T15:06:10.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:10.705+0000 7f8e50656700 1 librados: starting objecter 2024-04-14T15:06:10.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:10.705+0000 7f8e50656700 1 Processor -- start 2024-04-14T15:06:10.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:10.705+0000 7f8e50656700 1 -- start start 2024-04-14T15:06:10.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:10.705+0000 7f8e50656700 1 librados: setting wanted keys 2024-04-14T15:06:10.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:10.705+0000 7f8e50656700 1 librados: calling monclient init 2024-04-14T15:06:10.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:10.705+0000 7f8e50656700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f8e4810cf90 con 0x7f8e48106d40 2024-04-14T15:06:10.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:10.705+0000 7f8e50656700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f8e481a1f10 con 0x7f8e48102910 2024-04-14T15:06:10.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:10.705+0000 7f8e50656700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f8e481a30f0 con 0x7f8e4810aa80 2024-04-14T15:06:10.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:10.705+0000 7f8e4dbf1700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f8e48106d40 0x7f8e4819c0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:49374/0 (socket says 172.21.15.90:49374) 2024-04-14T15:06:10.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:10.705+0000 7f8e4dbf1700 1 -- 172.21.15.90:0/4278891990 learned_addr learned my addr 172.21.15.90:0/4278891990 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:06:10.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:10.705+0000 7f8e3f7fe700 10 client.?.objecter ms_handle_connect 0x7f8e48106d40 2024-04-14T15:06:10.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:10.705+0000 7f8e3f7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:06:10.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:10.705+0000 7f8e3f7fe700 10 client.?.objecter ms_handle_connect 0x7f8e4810aa80 2024-04-14T15:06:10.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:10.705+0000 7f8e3f7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:06:10.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:10.705+0000 7f8e3f7fe700 10 client.?.objecter ms_handle_connect 0x7f8e48102910 2024-04-14T15:06:10.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:10.705+0000 7f8e3f7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:06:10.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:10.705+0000 7f8e3f7fe700 1 -- 172.21.15.90:0/4278891990 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3440618758 0 0) 0x7f8e4810cf90 con 0x7f8e48106d40 2024-04-14T15:06:10.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:10.705+0000 7f8e3f7fe700 1 -- 172.21.15.90:0/4278891990 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e34005240 con 0x7f8e48106d40 2024-04-14T15:06:10.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:10.706+0000 7f8e3f7fe700 1 -- 172.21.15.90:0/4278891990 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 508120982 0 0) 0x7f8e34005240 con 0x7f8e48106d40 2024-04-14T15:06:10.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:10.706+0000 7f8e3f7fe700 1 -- 172.21.15.90:0/4278891990 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8e4810cf90 con 0x7f8e48106d40 2024-04-14T15:06:10.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:10.706+0000 7f8e3f7fe700 1 -- 172.21.15.90:0/4278891990 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f8e380044c0 con 0x7f8e48106d40 2024-04-14T15:06:10.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:10.706+0000 7f8e3f7fe700 1 -- 172.21.15.90:0/4278891990 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2030359946 0 0) 0x7f8e4810cf90 con 0x7f8e48106d40 2024-04-14T15:06:10.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:10.706+0000 7f8e3f7fe700 1 -- 172.21.15.90:0/4278891990 >> v1:172.21.15.90:6790/0 conn(0x7f8e4810aa80 legacy=0x7f8e4819f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:06:10.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:10.706+0000 7f8e3f7fe700 1 -- 172.21.15.90:0/4278891990 >> v1:172.21.15.92:6789/0 conn(0x7f8e48102910 legacy=0x7f8e4810c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:06:10.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:10.706+0000 7f8e3f7fe700 1 -- 172.21.15.90:0/4278891990 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e481a3300 con 0x7f8e48106d40 2024-04-14T15:06:10.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:10.706+0000 7f8e50656700 1 -- 172.21.15.90:0/4278891990 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8e481a35f0 con 0x7f8e48106d40 2024-04-14T15:06:10.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:10.706+0000 7f8e50656700 10 client.14214.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:06:10.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:10.706+0000 7f8e50656700 1 -- 172.21.15.90:0/4278891990 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f8e481a2330 con 0x7f8e48106d40 2024-04-14T15:06:10.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:10.706+0000 7f8e50656700 1 librados: init done 2024-04-14T15:06:10.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:10.706+0000 7f8e3f7fe700 1 -- 172.21.15.90:0/4278891990 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f8e38001b10 con 0x7f8e48106d40 2024-04-14T15:06:10.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:10.706+0000 7f8e3f7fe700 1 -- 172.21.15.90:0/4278891990 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f8e38005000 con 0x7f8e48106d40 2024-04-14T15:06:10.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:10.707+0000 7f8e3f7fe700 1 -- 172.21.15.90:0/4278891990 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 13) ==== 46579+0+0 (unknown 2972057686 0 0) 0x7f8e380051c0 con 0x7f8e48106d40 2024-04-14T15:06:10.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:10.707+0000 7f8e50656700 1 -- 172.21.15.90:0/4278891990 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e2c005120 con 0x7f8e48106d40 2024-04-14T15:06:10.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:10.707+0000 7f8e3f7fe700 1 -- 172.21.15.90:0/4278891990 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(4..4 src has 1..4) ==== 1069+0+0 (unknown 1732540447 0 0) 0x7f8e38049040 con 0x7f8e48106d40 2024-04-14T15:06:10.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:10.707+0000 7f8e3f7fe700 10 client.14214.objecter ms_dispatch 0x7f8e48001550 osd_map(4..4 src has 1..4) 2024-04-14T15:06:10.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:10.707+0000 7f8e3f7fe700 3 client.14214.objecter handle_osd_map got epochs [4,4] > 0 2024-04-14T15:06:10.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:10.707+0000 7f8e3f7fe700 3 client.14214.objecter handle_osd_map decoding full epoch 4 2024-04-14T15:06:10.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:10.707+0000 7f8e3f7fe700 20 client.14214.objecter dump_active .. 0 homeless 2024-04-14T15:06:10.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:10.708+0000 7f8e3f7fe700 10 client.14214.objecter ms_handle_connect 0x7f8e3403c120 2024-04-14T15:06:10.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:10.718+0000 7f8e3f7fe700 1 -- 172.21.15.90:0/4278891990 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f8e380178d0 con 0x7f8e48106d40 2024-04-14T15:06:10.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:10.944+0000 7f8e50656700 1 -- 172.21.15.90:0/4278891990 --> v1:172.21.15.90:6800/491071118 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f8e2c000b90 con 0x7f8e3403c120 2024-04-14T15:06:11.009 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:06:10 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-x[36044]: 2024-04-14T15:06:10.632+0000 7f84906b4200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-14T15:06:11.009 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:06:10 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-x[36044]: 2024-04-14T15:06:10.816+0000 7f84906b4200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-14T15:06:11.353 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:06:11 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-x[36044]: 2024-04-14T15:06:11.098+0000 7f84906b4200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-14T15:06:11.353 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:06:11 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-x[36044]: 2024-04-14T15:06:11.352+0000 7f84906b4200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-14T15:06:11.353 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:11 smithi092 ceph-mon[33966]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:06:11.353 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:11 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:11.353 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:11 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:11.353 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:11 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:06:11.353 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:11 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:06:11.353 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:11 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:06:11.353 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:11 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:11.353 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:11 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:11.353 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:11 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T15:06:11.354 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:11 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T15:06:11.354 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:11 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:06:11.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:11 smithi090 ceph-mon[30399]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:06:11.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:11 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:11.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:11 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:11.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:11 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:06:11.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:11 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:06:11.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:11 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:06:11.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:11 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:11.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:11 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:11.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:11 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T15:06:11.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:11 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T15:06:11.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:11 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:06:11.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:11 smithi090 ceph-mon[41670]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:06:11.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:11 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:11.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:11 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:11.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:11 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:06:11.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:11 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:06:11.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:11 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:06:11.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:11 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:11.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:11 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:11.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:11 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T15:06:11.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:11 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T15:06:11.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:11 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:06:11.622 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:06:11 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-x[36044]: 2024-04-14T15:06:11.584+0000 7f84906b4200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-14T15:06:12.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:12 smithi090 ceph-mon[30399]: from='client.14214 v1:172.21.15.90:0/4278891990' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:06:12.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:12 smithi090 ceph-mon[41670]: from='client.14214 v1:172.21.15.90:0/4278891990' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:06:12.369 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:12 smithi092 ceph-mon[33966]: from='client.14214 v1:172.21.15.90:0/4278891990' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:06:12.622 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:06:12 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-x[36044]: 2024-04-14T15:06:12.368+0000 7f84906b4200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-14T15:06:13.233 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:13 smithi092 ceph-mon[33966]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:06:13.234 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:06:12 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-x[36044]: 2024-04-14T15:06:12.926+0000 7f84906b4200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-14T15:06:13.234 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:06:13 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-x[36044]: 2024-04-14T15:06:13.022+0000 7f84906b4200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-14T15:06:13.234 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:06:13 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-x[36044]: 2024-04-14T15:06:13.233+0000 7f84906b4200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-14T15:06:13.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:13 smithi090 ceph-mon[30399]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:06:13.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:13 smithi090 ceph-mon[41670]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:06:14.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:14.022+0000 7f8e3f7fe700 1 -- 172.21.15.90:0/4278891990 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mgrmap(e 14) ==== 92567+0+0 (unknown 4138089514 0 0) 0x7f8e38047940 con 0x7f8e48106d40 2024-04-14T15:06:14.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:14 smithi092 ceph-mon[33966]: Standby manager daemon x started 2024-04-14T15:06:14.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:14 smithi092 ceph-mon[33966]: from='mgr.? v1:172.21.15.92:0/3996866323' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-14T15:06:14.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:14 smithi092 ceph-mon[33966]: from='mgr.? v1:172.21.15.92:0/3996866323' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T15:06:14.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:14 smithi092 ceph-mon[33966]: from='mgr.? v1:172.21.15.92:0/3996866323' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-14T15:06:14.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:14 smithi092 ceph-mon[33966]: from='mgr.? v1:172.21.15.92:0/3996866323' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T15:06:14.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:14 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/4234200134' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "76772cfc-89d1-4013-b617-32284ed8dfd9"}]: dispatch 2024-04-14T15:06:14.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:14 smithi092 ceph-mon[33966]: from='client.24119 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "76772cfc-89d1-4013-b617-32284ed8dfd9"}]: dispatch 2024-04-14T15:06:14.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:14 smithi092 ceph-mon[33966]: from='client.24119 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "76772cfc-89d1-4013-b617-32284ed8dfd9"}]': finished 2024-04-14T15:06:14.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:14 smithi092 ceph-mon[33966]: osdmap e5: 1 total, 0 up, 1 in 2024-04-14T15:06:14.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:14 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T15:06:14.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:14 smithi090 ceph-mon[41670]: Standby manager daemon x started 2024-04-14T15:06:14.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:14 smithi090 ceph-mon[41670]: from='mgr.? v1:172.21.15.92:0/3996866323' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-14T15:06:14.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:14 smithi090 ceph-mon[41670]: from='mgr.? v1:172.21.15.92:0/3996866323' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T15:06:14.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:14 smithi090 ceph-mon[41670]: from='mgr.? v1:172.21.15.92:0/3996866323' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-14T15:06:14.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:14 smithi090 ceph-mon[41670]: from='mgr.? v1:172.21.15.92:0/3996866323' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T15:06:14.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:14 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/4234200134' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "76772cfc-89d1-4013-b617-32284ed8dfd9"}]: dispatch 2024-04-14T15:06:14.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:14 smithi090 ceph-mon[41670]: from='client.24119 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "76772cfc-89d1-4013-b617-32284ed8dfd9"}]: dispatch 2024-04-14T15:06:14.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:14 smithi090 ceph-mon[41670]: from='client.24119 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "76772cfc-89d1-4013-b617-32284ed8dfd9"}]': finished 2024-04-14T15:06:14.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:14 smithi090 ceph-mon[41670]: osdmap e5: 1 total, 0 up, 1 in 2024-04-14T15:06:14.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:14 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T15:06:14.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:14 smithi090 ceph-mon[30399]: Standby manager daemon x started 2024-04-14T15:06:14.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:14 smithi090 ceph-mon[30399]: from='mgr.? v1:172.21.15.92:0/3996866323' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-14T15:06:14.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:14 smithi090 ceph-mon[30399]: from='mgr.? v1:172.21.15.92:0/3996866323' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T15:06:14.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:14 smithi090 ceph-mon[30399]: from='mgr.? v1:172.21.15.92:0/3996866323' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-14T15:06:14.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:14 smithi090 ceph-mon[30399]: from='mgr.? v1:172.21.15.92:0/3996866323' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T15:06:14.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:14 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/4234200134' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "76772cfc-89d1-4013-b617-32284ed8dfd9"}]: dispatch 2024-04-14T15:06:14.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:14 smithi090 ceph-mon[30399]: from='client.24119 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "76772cfc-89d1-4013-b617-32284ed8dfd9"}]: dispatch 2024-04-14T15:06:14.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:14 smithi090 ceph-mon[30399]: from='client.24119 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "76772cfc-89d1-4013-b617-32284ed8dfd9"}]': finished 2024-04-14T15:06:14.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:14 smithi090 ceph-mon[30399]: osdmap e5: 1 total, 0 up, 1 in 2024-04-14T15:06:14.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:14 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T15:06:15.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:15 smithi092 ceph-mon[33966]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:06:15.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:15 smithi092 ceph-mon[33966]: mgrmap e14: y(active, since 68s), standbys: x 2024-04-14T15:06:15.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:15 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-14T15:06:15.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:15 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/246468551' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T15:06:15.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:15 smithi090 ceph-mon[30399]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:06:15.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:15 smithi090 ceph-mon[30399]: mgrmap e14: y(active, since 68s), standbys: x 2024-04-14T15:06:15.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:15 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-14T15:06:15.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:15 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/246468551' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T15:06:15.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:15 smithi090 ceph-mon[41670]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:06:15.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:15 smithi090 ceph-mon[41670]: mgrmap e14: y(active, since 68s), standbys: x 2024-04-14T15:06:15.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:15 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-14T15:06:15.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:15 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/246468551' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T15:06:15.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:15.707+0000 7f8e4fe55700 10 client.14214.objecter tick 2024-04-14T15:06:16.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:16 smithi092 ceph-mon[33966]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:06:16.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:16 smithi090 ceph-mon[30399]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:06:16.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:16 smithi090 ceph-mon[41670]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:06:19.002 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:18 smithi090 ceph-mon[30399]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:06:19.002 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:18 smithi090 ceph-mon[41670]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:06:19.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:18 smithi092 ceph-mon[33966]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:06:20.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:20.708+0000 7f8e4fe55700 10 client.14214.objecter tick 2024-04-14T15:06:21.279 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:20 smithi092 ceph-mon[33966]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:06:21.279 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:20 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-14T15:06:21.279 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:20 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:06:21.308 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:20 smithi090 ceph-mon[30399]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:06:21.308 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:20 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-14T15:06:21.308 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:20 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:06:21.308 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:20 smithi090 ceph-mon[41670]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:06:21.308 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:20 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-14T15:06:21.309 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:20 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:06:22.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:21 smithi090 ceph-mon[30399]: Deploying daemon osd.0 on smithi090 2024-04-14T15:06:22.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:21 smithi090 ceph-mon[41670]: Deploying daemon osd.0 on smithi090 2024-04-14T15:06:22.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:21 smithi092 ceph-mon[33966]: Deploying daemon osd.0 on smithi090 2024-04-14T15:06:23.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:22 smithi090 ceph-mon[30399]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:06:23.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:22 smithi090 ceph-mon[41670]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:06:23.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:22 smithi092 ceph-mon[33966]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:06:25.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:24 smithi090 ceph-mon[30399]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:06:25.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:24 smithi090 ceph-mon[41670]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:06:25.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:24 smithi092 ceph-mon[33966]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:06:25.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:25.709+0000 7f8e4fe55700 10 client.14214.objecter tick 2024-04-14T15:06:26.686 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:26 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:26.686 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:26 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:26.686 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:26 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:06:26.686 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:26 smithi090 ceph-mon[30399]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:06:26.687 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:26 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:26.687 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:26 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:26.687 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:26 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:06:26.687 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:26 smithi090 ceph-mon[41670]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:06:26.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:26 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:26.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:26 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:26.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:26 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:06:26.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:26 smithi092 ceph-mon[33966]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:06:29.153 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:28 smithi090 ceph-mon[41670]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:06:29.154 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:28 smithi090 ceph-mon[41670]: from='osd.0 v1:172.21.15.90:6801/3761806885' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-14T15:06:29.154 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:28 smithi090 ceph-mon[41670]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-14T15:06:29.154 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:28 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:29.154 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:28 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:29.154 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:28 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:06:29.154 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:28 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:06:29.154 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:28 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:29.154 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:28 smithi090 ceph-mon[30399]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:06:29.154 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:28 smithi090 ceph-mon[30399]: from='osd.0 v1:172.21.15.90:6801/3761806885' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-14T15:06:29.155 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:28 smithi090 ceph-mon[30399]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-14T15:06:29.155 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:28 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:29.155 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:28 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:29.155 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:28 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:06:29.155 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:28 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:06:29.155 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:28 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:29.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:28 smithi092 ceph-mon[33966]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:06:29.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:28 smithi092 ceph-mon[33966]: from='osd.0 v1:172.21.15.90:6801/3761806885' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-14T15:06:29.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:28 smithi092 ceph-mon[33966]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-14T15:06:29.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:28 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:29.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:28 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:29.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:28 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:06:29.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:28 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:06:29.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:28 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:29.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:29.495+0000 7f8e3f7fe700 1 -- 172.21.15.90:0/4278891990 <== mgr.14148 v1:172.21.15.90:6800/491071118 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (unknown 0 0 932195887) 0x7f8e2c000b90 con 0x7f8e3403c120 2024-04-14T15:06:29.497 INFO:teuthology.orchestra.run.smithi090.stdout:Created osd(s) 0 on host 'smithi090' 2024-04-14T15:06:29.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:29.497+0000 7f8e50656700 10 librados: watch_flush enter 2024-04-14T15:06:29.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:29.498+0000 7f8e50656700 10 librados: watch_flush exit 2024-04-14T15:06:29.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:29.498+0000 7f8e50656700 20 client.14214.objecter shutdown clearing up homeless session... 2024-04-14T15:06:29.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:29.498+0000 7f8e50656700 10 client.14214.objecter successfully canceled tick 2024-04-14T15:06:29.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:29.498+0000 7f8e50656700 1 -- 172.21.15.90:0/4278891990 >> v1:172.21.15.90:6800/491071118 conn(0x7f8e3403c120 legacy=0x7f8e3403e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:06:29.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:29.498+0000 7f8e50656700 1 -- 172.21.15.90:0/4278891990 >> v1:172.21.15.90:6789/0 conn(0x7f8e48106d40 legacy=0x7f8e4819c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:06:29.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:29.498+0000 7f8e50656700 1 -- 172.21.15.90:0/4278891990 shutdown_connections 2024-04-14T15:06:29.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:29.498+0000 7f8e50656700 1 -- 172.21.15.90:0/4278891990 >> 172.21.15.90:0/4278891990 conn(0x7f8e480fded0 msgr2=0x7f8e48109930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:06:29.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:29.498+0000 7f8e50656700 1 -- 172.21.15.90:0/4278891990 shutdown_connections 2024-04-14T15:06:29.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:29.498+0000 7f8e50656700 1 -- 172.21.15.90:0/4278891990 wait complete. 2024-04-14T15:06:29.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:29.498+0000 7f8e50656700 1 librados: shutdown 2024-04-14T15:06:30.077 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:29 smithi090 ceph-mon[41670]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-14T15:06:30.077 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:29 smithi090 ceph-mon[41670]: osdmap e6: 1 total, 0 up, 1 in 2024-04-14T15:06:30.077 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:29 smithi090 ceph-mon[41670]: from='osd.0 v1:172.21.15.90:6801/3761806885' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-04-14T15:06:30.077 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:29 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T15:06:30.077 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:29 smithi090 ceph-mon[41670]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-04-14T15:06:30.077 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:29 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:30.077 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:29 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:30.077 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:29 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:06:30.077 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:29 smithi090 ceph-mon[41670]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-04-14T15:06:30.078 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:29 smithi090 ceph-mon[41670]: osdmap e7: 1 total, 0 up, 1 in 2024-04-14T15:06:30.078 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:29 smithi090 ceph-mon[30399]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-14T15:06:30.078 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:29 smithi090 ceph-mon[30399]: osdmap e6: 1 total, 0 up, 1 in 2024-04-14T15:06:30.078 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:29 smithi090 ceph-mon[30399]: from='osd.0 v1:172.21.15.90:6801/3761806885' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-04-14T15:06:30.078 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:29 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T15:06:30.078 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:29 smithi090 ceph-mon[30399]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-04-14T15:06:30.078 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:29 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:30.078 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:29 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:30.078 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:29 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:06:30.078 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:29 smithi090 ceph-mon[30399]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-04-14T15:06:30.078 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:29 smithi090 ceph-mon[30399]: osdmap e7: 1 total, 0 up, 1 in 2024-04-14T15:06:30.105 DEBUG:teuthology.orchestra.run.smithi090:osd.0> sudo journalctl -f -n 0 -u ceph-0e288098-fa70-11ee-bc8e-c7b262605968@osd.0.service 2024-04-14T15:06:30.108 INFO:tasks.cephadm:Deploying osd.1 on smithi090 with /dev/vg_nvme/lv_3... 2024-04-14T15:06:30.108 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-14T15:06:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:29 smithi092 ceph-mon[33966]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-14T15:06:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:29 smithi092 ceph-mon[33966]: osdmap e6: 1 total, 0 up, 1 in 2024-04-14T15:06:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:29 smithi092 ceph-mon[33966]: from='osd.0 v1:172.21.15.90:6801/3761806885' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-04-14T15:06:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:29 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T15:06:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:29 smithi092 ceph-mon[33966]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-04-14T15:06:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:29 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:29 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:30.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:29 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:06:30.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:29 smithi092 ceph-mon[33966]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-04-14T15:06:30.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:29 smithi092 ceph-mon[33966]: osdmap e7: 1 total, 0 up, 1 in 2024-04-14T15:06:30.672 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 14 15:06:30 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-0[46039]: 2024-04-14T15:06:30.380+0000 7fb4b3eb2700 -1 osd.0 0 waiting for initial osdmap 2024-04-14T15:06:30.672 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 14 15:06:30 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-0[46039]: 2024-04-14T15:06:30.382+0000 7fb4abc9d700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-14T15:06:30.871 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:06:31.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:30 smithi092 ceph-mon[33966]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:06:31.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:30 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T15:06:31.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:30 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T15:06:31.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:30 smithi092 ceph-mon[33966]: from='osd.0 ' entity='osd.0' 2024-04-14T15:06:31.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:30 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T15:06:31.329 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:30 smithi090 ceph-mon[30399]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:06:31.330 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:30 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T15:06:31.330 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:30 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T15:06:31.330 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:30 smithi090 ceph-mon[30399]: from='osd.0 ' entity='osd.0' 2024-04-14T15:06:31.330 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:30 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T15:06:31.330 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:30 smithi090 ceph-mon[41670]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:06:31.330 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:30 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T15:06:31.330 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:30 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T15:06:31.330 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:30 smithi090 ceph-mon[41670]: from='osd.0 ' entity='osd.0' 2024-04-14T15:06:31.330 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:30 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T15:06:32.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:32 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:06:32.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:32 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:06:32.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:32 smithi090 ceph-mon[30399]: osd.0 v1:172.21.15.90:6801/3761806885 boot 2024-04-14T15:06:32.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:32 smithi090 ceph-mon[30399]: osdmap e8: 1 total, 1 up, 1 in 2024-04-14T15:06:32.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:32 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T15:06:32.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:32 smithi090 ceph-mon[30399]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:06:32.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:32 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:06:32.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:32 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:06:32.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:32 smithi090 ceph-mon[41670]: osd.0 v1:172.21.15.90:6801/3761806885 boot 2024-04-14T15:06:32.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:32 smithi090 ceph-mon[41670]: osdmap e8: 1 total, 1 up, 1 in 2024-04-14T15:06:32.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:32 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T15:06:32.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:32 smithi090 ceph-mon[41670]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:06:32.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:32 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:06:32.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:32 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:06:32.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:32 smithi092 ceph-mon[33966]: osd.0 v1:172.21.15.90:6801/3761806885 boot 2024-04-14T15:06:32.873 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:32 smithi092 ceph-mon[33966]: osdmap e8: 1 total, 1 up, 1 in 2024-04-14T15:06:32.873 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:32 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T15:06:32.873 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:32 smithi092 ceph-mon[33966]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:06:33.685 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-14T15:06:33.685 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-14T15:06:33.685 INFO:teuthology.orchestra.run.smithi090.stderr: stderr: 10+0 records in 2024-04-14T15:06:33.685 INFO:teuthology.orchestra.run.smithi090.stderr:10+0 records out 2024-04-14T15:06:33.685 INFO:teuthology.orchestra.run.smithi090.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0127292 s, 824 MB/s 2024-04-14T15:06:33.685 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping successful for: 2024-04-14T15:06:33.686 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:33 smithi090 ceph-mon[30399]: osdmap e9: 1 total, 1 up, 1 in 2024-04-14T15:06:33.686 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:33 smithi090 ceph-mon[41670]: osdmap e9: 1 total, 1 up, 1 in 2024-04-14T15:06:33.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:33 smithi092 ceph-mon[33966]: osdmap e9: 1 total, 1 up, 1 in 2024-04-14T15:06:34.630 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:34 smithi090 ceph-mon[30399]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:06:34.630 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:34 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:34.630 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:34 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:34.630 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:34 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:06:34.630 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:34 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:34.630 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:34 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:06:34.631 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:34 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:06:34.631 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:34 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:34.631 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:34 smithi090 ceph-mon[41670]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:06:34.631 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:34 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:34.631 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:34 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:34.631 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:34 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:06:34.631 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:34 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:34.631 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:34 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:06:34.631 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:34 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:06:34.631 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:34 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:34.656 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph orch daemon add osd smithi090:vg_nvme/lv_3 2024-04-14T15:06:34.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:34 smithi092 ceph-mon[33966]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:06:34.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:34 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:34.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:34 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:34.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:34 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:06:34.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:34 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:34.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:34 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:06:34.873 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:34 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:06:34.873 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:34 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:34.894 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:06:35.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:35 smithi090 ceph-mon[30399]: Detected new or changed devices on smithi090 2024-04-14T15:06:35.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:35 smithi090 ceph-mon[30399]: Adjusting osd_memory_target on smithi090 to 16064M 2024-04-14T15:06:35.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:35 smithi090 ceph-mon[41670]: Detected new or changed devices on smithi090 2024-04-14T15:06:35.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:35 smithi090 ceph-mon[41670]: Adjusting osd_memory_target on smithi090 to 16064M 2024-04-14T15:06:35.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:35 smithi092 ceph-mon[33966]: Detected new or changed devices on smithi090 2024-04-14T15:06:35.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:35 smithi092 ceph-mon[33966]: Adjusting osd_memory_target on smithi090 to 16064M 2024-04-14T15:06:36.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:36.472+0000 7f8e62e79700 1 -- 172.21.15.90:0/3744868062 >> v1:172.21.15.90:6790/0 conn(0x7f8e5c106f30 legacy=0x7f8e5c109320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:06:36.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:36.472+0000 7f8e62e79700 1 -- 172.21.15.90:0/3744868062 shutdown_connections 2024-04-14T15:06:36.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:36.472+0000 7f8e62e79700 1 -- 172.21.15.90:0/3744868062 >> 172.21.15.90:0/3744868062 conn(0x7f8e5c0fcd90 msgr2=0x7f8e5c0ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:06:36.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:36.472+0000 7f8e62e79700 1 -- 172.21.15.90:0/3744868062 shutdown_connections 2024-04-14T15:06:36.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:36.472+0000 7f8e62e79700 1 -- 172.21.15.90:0/3744868062 wait complete. 2024-04-14T15:06:36.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:36.473+0000 7f8e62e79700 1 librados: starting msgr at 2024-04-14T15:06:36.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:36.473+0000 7f8e62e79700 1 librados: starting objecter 2024-04-14T15:06:36.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:36.473+0000 7f8e62e79700 1 Processor -- start 2024-04-14T15:06:36.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:36.473+0000 7f8e62e79700 1 -- start start 2024-04-14T15:06:36.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:36.473+0000 7f8e62e79700 1 librados: setting wanted keys 2024-04-14T15:06:36.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:36.473+0000 7f8e62e79700 1 librados: calling monclient init 2024-04-14T15:06:36.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:36.473+0000 7f8e62e79700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f8e5c1a5010 con 0x7f8e5c10aa40 2024-04-14T15:06:36.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:36.473+0000 7f8e62e79700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f8e5c1a6250 con 0x7f8e5c103420 2024-04-14T15:06:36.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:36.473+0000 7f8e62e79700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f8e5c1a7490 con 0x7f8e5c106f30 2024-04-14T15:06:36.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:36.473+0000 7f8e5bfff700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f8e5c106f30 0x7f8e5c19fdf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:36932/0 (socket says 172.21.15.90:36932) 2024-04-14T15:06:36.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:36.473+0000 7f8e61416700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f8e5c10aa40 0x7f8e5c1a3710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:53238/0 (socket says 172.21.15.90:53238) 2024-04-14T15:06:36.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:36.473+0000 7f8e5bfff700 1 -- 172.21.15.90:0/2438324624 learned_addr learned my addr 172.21.15.90:0/2438324624 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:06:36.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:36.473+0000 7f8e59ffb700 10 client.?.objecter ms_handle_connect 0x7f8e5c10aa40 2024-04-14T15:06:36.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:36.473+0000 7f8e59ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:06:36.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:36.473+0000 7f8e59ffb700 10 client.?.objecter ms_handle_connect 0x7f8e5c106f30 2024-04-14T15:06:36.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:36.473+0000 7f8e59ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:06:36.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:36.473+0000 7f8e59ffb700 10 client.?.objecter ms_handle_connect 0x7f8e5c103420 2024-04-14T15:06:36.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:36.473+0000 7f8e59ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:06:36.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:36.474+0000 7f8e59ffb700 1 -- 172.21.15.90:0/2438324624 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1047215874 0 0) 0x7f8e5c1a7490 con 0x7f8e5c106f30 2024-04-14T15:06:36.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:36.474+0000 7f8e59ffb700 1 -- 172.21.15.90:0/2438324624 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e4c0035b0 con 0x7f8e5c106f30 2024-04-14T15:06:36.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:36.474+0000 7f8e59ffb700 1 -- 172.21.15.90:0/2438324624 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3916938571 0 0) 0x7f8e5c1a6250 con 0x7f8e5c103420 2024-04-14T15:06:36.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:36.474+0000 7f8e59ffb700 1 -- 172.21.15.90:0/2438324624 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e5c1a7490 con 0x7f8e5c103420 2024-04-14T15:06:36.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:36.474+0000 7f8e59ffb700 1 -- 172.21.15.90:0/2438324624 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1205704889 0 0) 0x7f8e5c1a5010 con 0x7f8e5c10aa40 2024-04-14T15:06:36.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:36.474+0000 7f8e59ffb700 1 -- 172.21.15.90:0/2438324624 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e5c1a6250 con 0x7f8e5c10aa40 2024-04-14T15:06:36.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:36.474+0000 7f8e59ffb700 1 -- 172.21.15.90:0/2438324624 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 489904703 0 0) 0x7f8e4c0035b0 con 0x7f8e5c106f30 2024-04-14T15:06:36.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:36.474+0000 7f8e59ffb700 1 -- 172.21.15.90:0/2438324624 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8e5c1a5010 con 0x7f8e5c106f30 2024-04-14T15:06:36.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:36.474+0000 7f8e59ffb700 1 -- 172.21.15.90:0/2438324624 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f8e48004470 con 0x7f8e5c106f30 2024-04-14T15:06:36.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:36.474+0000 7f8e59ffb700 1 -- 172.21.15.90:0/2438324624 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1989505091 0 0) 0x7f8e5c1a6250 con 0x7f8e5c10aa40 2024-04-14T15:06:36.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:36.474+0000 7f8e59ffb700 1 -- 172.21.15.90:0/2438324624 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8e4c0035b0 con 0x7f8e5c10aa40 2024-04-14T15:06:36.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:36.474+0000 7f8e59ffb700 1 -- 172.21.15.90:0/2438324624 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 921939168 0 0) 0x7f8e5c1a7490 con 0x7f8e5c103420 2024-04-14T15:06:36.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:36.474+0000 7f8e59ffb700 1 -- 172.21.15.90:0/2438324624 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8e5c1a6250 con 0x7f8e5c103420 2024-04-14T15:06:36.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:36.474+0000 7f8e59ffb700 1 -- 172.21.15.90:0/2438324624 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f8e540015b0 con 0x7f8e5c10aa40 2024-04-14T15:06:36.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:36.474+0000 7f8e59ffb700 1 -- 172.21.15.90:0/2438324624 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f8e500017b0 con 0x7f8e5c103420 2024-04-14T15:06:36.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:36.474+0000 7f8e59ffb700 1 -- 172.21.15.90:0/2438324624 <== mon.2 v1:172.21.15.90:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1901839384 0 0) 0x7f8e5c1a5010 con 0x7f8e5c106f30 2024-04-14T15:06:36.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:36.474+0000 7f8e59ffb700 1 -- 172.21.15.90:0/2438324624 >> v1:172.21.15.92:6789/0 conn(0x7f8e5c103420 legacy=0x7f8e5c102ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:06:36.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:36.474+0000 7f8e59ffb700 1 -- 172.21.15.90:0/2438324624 >> v1:172.21.15.90:6789/0 conn(0x7f8e5c10aa40 legacy=0x7f8e5c1a3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:06:36.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:36.474+0000 7f8e59ffb700 1 -- 172.21.15.90:0/2438324624 --> v1:172.21.15.90:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e5c1a86d0 con 0x7f8e5c106f30 2024-04-14T15:06:36.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:36.474+0000 7f8e62e79700 1 -- 172.21.15.90:0/2438324624 --> v1:172.21.15.90:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8e5c1a76c0 con 0x7f8e5c106f30 2024-04-14T15:06:36.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:36.474+0000 7f8e59ffb700 1 -- 172.21.15.90:0/2438324624 <== mon.2 v1:172.21.15.90:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f8e48001ab0 con 0x7f8e5c106f30 2024-04-14T15:06:36.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:36.474+0000 7f8e59ffb700 1 -- 172.21.15.90:0/2438324624 <== mon.2 v1:172.21.15.90:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f8e48005130 con 0x7f8e5c106f30 2024-04-14T15:06:36.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:36.474+0000 7f8e62e79700 10 client.24140.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:06:36.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:36.474+0000 7f8e62e79700 1 -- 172.21.15.90:0/2438324624 --> v1:172.21.15.90:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f8e5c1a7c00 con 0x7f8e5c106f30 2024-04-14T15:06:36.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:36.474+0000 7f8e62e79700 1 librados: init done 2024-04-14T15:06:36.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:36.475+0000 7f8e62e79700 1 -- 172.21.15.90:0/2438324624 --> v1:172.21.15.90:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e5c0693d0 con 0x7f8e5c106f30 2024-04-14T15:06:36.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:36.476+0000 7f8e59ffb700 1 -- 172.21.15.90:0/2438324624 <== mon.2 v1:172.21.15.90:6790/0 7 ==== mgrmap(e 14) ==== 92567+0+0 (unknown 4138089514 0 0) 0x7f8e48005440 con 0x7f8e5c106f30 2024-04-14T15:06:36.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:36.476+0000 7f8e59ffb700 1 -- 172.21.15.90:0/2438324624 <== mon.2 v1:172.21.15.90:6790/0 8 ==== osd_map(9..9 src has 1..9) ==== 1541+0+0 (unknown 2977167112 0 0) 0x7f8e48089190 con 0x7f8e5c106f30 2024-04-14T15:06:36.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:36.476+0000 7f8e59ffb700 10 client.24140.objecter ms_dispatch 0x7f8e5c001550 osd_map(9..9 src has 1..9) 2024-04-14T15:06:36.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:36.476+0000 7f8e59ffb700 3 client.24140.objecter handle_osd_map got epochs [9,9] > 0 2024-04-14T15:06:36.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:36.476+0000 7f8e59ffb700 3 client.24140.objecter handle_osd_map decoding full epoch 9 2024-04-14T15:06:36.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:36.476+0000 7f8e59ffb700 20 client.24140.objecter dump_active .. 0 homeless 2024-04-14T15:06:36.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:36.476+0000 7f8e59ffb700 10 client.24140.objecter ms_handle_connect 0x7f8e4c0714c0 2024-04-14T15:06:36.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:36.481+0000 7f8e59ffb700 1 -- 172.21.15.90:0/2438324624 <== mon.2 v1:172.21.15.90:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f8e4805fc30 con 0x7f8e5c106f30 2024-04-14T15:06:36.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:36.687+0000 7f8e62e79700 1 -- 172.21.15.90:0/2438324624 --> v1:172.21.15.90:6800/491071118 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f8e5c1a7ef0 con 0x7f8e4c0714c0 2024-04-14T15:06:36.688 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:36 smithi090 ceph-mon[41670]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:06:36.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:36 smithi090 ceph-mon[30399]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:06:36.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:36 smithi092 ceph-mon[33966]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:06:37.551 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:37 smithi090 ceph-mon[30399]: from='client.24140 v1:172.21.15.90:0/2438324624' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:06:37.551 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:37 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T15:06:37.551 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:37 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T15:06:37.551 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:37 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:06:37.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:37 smithi090 ceph-mon[41670]: from='client.24140 v1:172.21.15.90:0/2438324624' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:06:37.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:37 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T15:06:37.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:37 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T15:06:37.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:37 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:06:37.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:37 smithi092 ceph-mon[33966]: from='client.24140 v1:172.21.15.90:0/2438324624' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:06:37.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:37 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T15:06:37.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:37 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T15:06:37.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:37 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:06:38.613 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:38 smithi090 ceph-mon[30399]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:06:38.614 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:38 smithi090 ceph-mon[41670]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:06:38.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:38 smithi092 ceph-mon[33966]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:06:39.832 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:39 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3811944160' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "de4a7fa2-7d57-4ff3-a4fa-28c0cde8ef6c"}]: dispatch 2024-04-14T15:06:39.832 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:39 smithi090 ceph-mon[30399]: from='client.24146 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "de4a7fa2-7d57-4ff3-a4fa-28c0cde8ef6c"}]: dispatch 2024-04-14T15:06:39.832 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:39 smithi090 ceph-mon[30399]: from='client.24146 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "de4a7fa2-7d57-4ff3-a4fa-28c0cde8ef6c"}]': finished 2024-04-14T15:06:39.832 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:39 smithi090 ceph-mon[30399]: osdmap e10: 2 total, 1 up, 2 in 2024-04-14T15:06:39.832 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:39 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T15:06:39.832 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:39 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3811944160' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "de4a7fa2-7d57-4ff3-a4fa-28c0cde8ef6c"}]: dispatch 2024-04-14T15:06:39.832 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:39 smithi090 ceph-mon[41670]: from='client.24146 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "de4a7fa2-7d57-4ff3-a4fa-28c0cde8ef6c"}]: dispatch 2024-04-14T15:06:39.832 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:39 smithi090 ceph-mon[41670]: from='client.24146 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "de4a7fa2-7d57-4ff3-a4fa-28c0cde8ef6c"}]': finished 2024-04-14T15:06:39.832 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:39 smithi090 ceph-mon[41670]: osdmap e10: 2 total, 1 up, 2 in 2024-04-14T15:06:39.833 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:39 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T15:06:39.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:39 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3811944160' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "de4a7fa2-7d57-4ff3-a4fa-28c0cde8ef6c"}]: dispatch 2024-04-14T15:06:39.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:39 smithi092 ceph-mon[33966]: from='client.24146 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "de4a7fa2-7d57-4ff3-a4fa-28c0cde8ef6c"}]: dispatch 2024-04-14T15:06:39.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:39 smithi092 ceph-mon[33966]: from='client.24146 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "de4a7fa2-7d57-4ff3-a4fa-28c0cde8ef6c"}]': finished 2024-04-14T15:06:39.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:39 smithi092 ceph-mon[33966]: osdmap e10: 2 total, 1 up, 2 in 2024-04-14T15:06:39.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:39 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T15:06:40.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:40 smithi092 ceph-mon[33966]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:06:40.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:40 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/4263536195' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T15:06:40.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:40 smithi090 ceph-mon[30399]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:06:40.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:40 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/4263536195' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T15:06:40.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:40 smithi090 ceph-mon[41670]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:06:40.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:40 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/4263536195' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T15:06:41.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:41.475+0000 7f8e62678700 10 client.24140.objecter tick 2024-04-14T15:06:43.046 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:42 smithi090 ceph-mon[30399]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:06:43.046 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:42 smithi090 ceph-mon[41670]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:06:43.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:42 smithi092 ceph-mon[33966]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:06:45.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:44 smithi092 ceph-mon[33966]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:06:45.388 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:44 smithi090 ceph-mon[30399]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:06:45.388 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:44 smithi090 ceph-mon[41670]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:06:46.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:46.476+0000 7f8e62678700 10 client.24140.objecter tick 2024-04-14T15:06:46.735 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:46 smithi090 ceph-mon[41670]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:06:46.736 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:46 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-14T15:06:46.736 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:46 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:06:46.736 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:46 smithi090 ceph-mon[41670]: Deploying daemon osd.1 on smithi090 2024-04-14T15:06:46.736 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:46 smithi090 ceph-mon[30399]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:06:46.736 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:46 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-14T15:06:46.736 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:46 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:06:46.736 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:46 smithi090 ceph-mon[30399]: Deploying daemon osd.1 on smithi090 2024-04-14T15:06:47.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:46 smithi092 ceph-mon[33966]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:06:47.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:46 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-14T15:06:47.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:46 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:06:47.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:46 smithi092 ceph-mon[33966]: Deploying daemon osd.1 on smithi090 2024-04-14T15:06:49.349 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:48 smithi090 ceph-mon[30399]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:06:49.349 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:48 smithi090 ceph-mon[41670]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:06:49.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:48 smithi092 ceph-mon[33966]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:06:51.110 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:50 smithi090 ceph-mon[30399]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:06:51.110 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:50 smithi090 ceph-mon[41670]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:06:51.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:50 smithi092 ceph-mon[33966]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:06:51.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:51.477+0000 7f8e62678700 10 client.24140.objecter tick 2024-04-14T15:06:52.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:52 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:52.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:52 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:06:52.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:52 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:52.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:52 smithi092 ceph-mon[33966]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:06:52.691 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:52 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:52.691 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:52 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:06:52.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:52 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:52.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:52 smithi090 ceph-mon[30399]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:06:52.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:52 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:52.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:52 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:06:52.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:52 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:52.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:52 smithi090 ceph-mon[41670]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:06:54.994 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:54 smithi090 ceph-mon[30399]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:06:54.994 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:54 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:54.994 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:54 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:54.994 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:54 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:06:54.994 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:54 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:06:54.994 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:54 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:54.994 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:54 smithi090 ceph-mon[30399]: from='osd.1 v1:172.21.15.90:6805/4170361699' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-14T15:06:54.994 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:54 smithi090 ceph-mon[41670]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:06:54.995 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:54 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:54.995 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:54 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:54.995 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:54 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:06:54.995 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:54 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:06:54.995 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:54 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:54.995 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:54 smithi090 ceph-mon[41670]: from='osd.1 v1:172.21.15.90:6805/4170361699' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-14T15:06:55.042 INFO:teuthology.orchestra.run.smithi090.stdout:Created osd(s) 1 on host 'smithi090' 2024-04-14T15:06:55.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:55.041+0000 7f8e59ffb700 1 -- 172.21.15.90:0/2438324624 <== mgr.14148 v1:172.21.15.90:6800/491071118 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (unknown 0 0 1803927707) 0x7f8e5c1a7ef0 con 0x7f8e4c0714c0 2024-04-14T15:06:55.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:55.043+0000 7f8e62e79700 10 librados: watch_flush enter 2024-04-14T15:06:55.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:55.044+0000 7f8e62e79700 10 librados: watch_flush exit 2024-04-14T15:06:55.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:55.044+0000 7f8e62e79700 20 client.24140.objecter shutdown clearing up homeless session... 2024-04-14T15:06:55.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:55.044+0000 7f8e62e79700 10 client.24140.objecter successfully canceled tick 2024-04-14T15:06:55.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:55.044+0000 7f8e62e79700 1 -- 172.21.15.90:0/2438324624 >> v1:172.21.15.90:6800/491071118 conn(0x7f8e4c0714c0 legacy=0x7f8e4c073980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:06:55.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:55.044+0000 7f8e62e79700 1 -- 172.21.15.90:0/2438324624 >> v1:172.21.15.90:6790/0 conn(0x7f8e5c106f30 legacy=0x7f8e5c19fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:06:55.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:55.044+0000 7f8e62e79700 1 -- 172.21.15.90:0/2438324624 shutdown_connections 2024-04-14T15:06:55.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:55.044+0000 7f8e62e79700 1 -- 172.21.15.90:0/2438324624 >> 172.21.15.90:0/2438324624 conn(0x7f8e5c0fcd90 msgr2=0x7f8e5c0ff180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:06:55.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:55.044+0000 7f8e62e79700 1 -- 172.21.15.90:0/2438324624 shutdown_connections 2024-04-14T15:06:55.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:55.044+0000 7f8e62e79700 1 -- 172.21.15.90:0/2438324624 wait complete. 2024-04-14T15:06:55.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:06:55.044+0000 7f8e62e79700 1 librados: shutdown 2024-04-14T15:06:55.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:54 smithi092 ceph-mon[33966]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:06:55.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:54 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:55.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:54 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:55.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:54 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:06:55.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:54 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:06:55.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:54 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:55.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:54 smithi092 ceph-mon[33966]: from='osd.1 v1:172.21.15.90:6805/4170361699' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-14T15:06:55.675 DEBUG:teuthology.orchestra.run.smithi090:osd.1> sudo journalctl -f -n 0 -u ceph-0e288098-fa70-11ee-bc8e-c7b262605968@osd.1.service 2024-04-14T15:06:55.677 INFO:tasks.cephadm:Deploying osd.2 on smithi090 with /dev/vg_nvme/lv_2... 2024-04-14T15:06:55.677 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-14T15:06:56.236 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:56 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:56.236 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:56 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:56.236 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:56 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:06:56.236 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:56 smithi090 ceph-mon[41670]: from='osd.1 v1:172.21.15.90:6805/4170361699' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-14T15:06:56.236 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:56 smithi090 ceph-mon[41670]: osdmap e11: 2 total, 1 up, 2 in 2024-04-14T15:06:56.236 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:56 smithi090 ceph-mon[41670]: from='osd.1 v1:172.21.15.90:6805/4170361699' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-04-14T15:06:56.236 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:56 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T15:06:56.237 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:56 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:56.237 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:56 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:56.237 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:56 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:06:56.237 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:56 smithi090 ceph-mon[30399]: from='osd.1 v1:172.21.15.90:6805/4170361699' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-14T15:06:56.237 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:56 smithi090 ceph-mon[30399]: osdmap e11: 2 total, 1 up, 2 in 2024-04-14T15:06:56.237 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:56 smithi090 ceph-mon[30399]: from='osd.1 v1:172.21.15.90:6805/4170361699' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-04-14T15:06:56.237 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:56 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T15:06:56.237 INFO:journalctl@ceph.osd.1.smithi090.stdout:Apr 14 15:06:56 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-1[52165]: 2024-04-14T15:06:56.217+0000 7f18c3ad3700 -1 osd.1 0 waiting for initial osdmap 2024-04-14T15:06:56.238 INFO:journalctl@ceph.osd.1.smithi090.stdout:Apr 14 15:06:56 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-1[52165]: 2024-04-14T15:06:56.220+0000 7f18bf8c6700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-14T15:06:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:56 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:56 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:06:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:56 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:06:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:56 smithi092 ceph-mon[33966]: from='osd.1 v1:172.21.15.90:6805/4170361699' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-14T15:06:56.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:56 smithi092 ceph-mon[33966]: osdmap e11: 2 total, 1 up, 2 in 2024-04-14T15:06:56.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:56 smithi092 ceph-mon[33966]: from='osd.1 v1:172.21.15.90:6805/4170361699' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-04-14T15:06:56.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:56 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T15:06:56.461 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:06:57.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:57 smithi090 ceph-mon[30399]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:06:57.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:57 smithi090 ceph-mon[30399]: from='osd.1 v1:172.21.15.90:6805/4170361699' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-04-14T15:06:57.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:57 smithi090 ceph-mon[30399]: osdmap e12: 2 total, 1 up, 2 in 2024-04-14T15:06:57.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:57 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T15:06:57.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:57 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T15:06:57.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:57 smithi090 ceph-mon[41670]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:06:57.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:57 smithi090 ceph-mon[41670]: from='osd.1 v1:172.21.15.90:6805/4170361699' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-04-14T15:06:57.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:57 smithi090 ceph-mon[41670]: osdmap e12: 2 total, 1 up, 2 in 2024-04-14T15:06:57.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:57 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T15:06:57.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:57 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T15:06:57.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:57 smithi092 ceph-mon[33966]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:06:57.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:57 smithi092 ceph-mon[33966]: from='osd.1 v1:172.21.15.90:6805/4170361699' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-04-14T15:06:57.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:57 smithi092 ceph-mon[33966]: osdmap e12: 2 total, 1 up, 2 in 2024-04-14T15:06:57.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:57 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T15:06:57.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:57 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T15:06:58.402 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:58 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:06:58.403 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:58 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:06:58.403 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:58 smithi090 ceph-mon[30399]: osd.1 v1:172.21.15.90:6805/4170361699 boot 2024-04-14T15:06:58.403 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:58 smithi090 ceph-mon[30399]: osdmap e13: 2 total, 2 up, 2 in 2024-04-14T15:06:58.403 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:58 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T15:06:58.403 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:58 smithi090 ceph-mon[30399]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:06:58.403 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:58 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:06:58.403 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:58 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:06:58.403 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:58 smithi090 ceph-mon[41670]: osd.1 v1:172.21.15.90:6805/4170361699 boot 2024-04-14T15:06:58.403 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:58 smithi090 ceph-mon[41670]: osdmap e13: 2 total, 2 up, 2 in 2024-04-14T15:06:58.403 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:58 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T15:06:58.404 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:58 smithi090 ceph-mon[41670]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:06:58.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:58 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:06:58.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:58 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:06:58.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:58 smithi092 ceph-mon[33966]: osd.1 v1:172.21.15.90:6805/4170361699 boot 2024-04-14T15:06:58.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:58 smithi092 ceph-mon[33966]: osdmap e13: 2 total, 2 up, 2 in 2024-04-14T15:06:58.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:58 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T15:06:58.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:58 smithi092 ceph-mon[33966]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:06:59.007 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-14T15:06:59.007 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-14T15:06:59.007 INFO:teuthology.orchestra.run.smithi090.stderr: stderr: 10+0 records in 2024-04-14T15:06:59.007 INFO:teuthology.orchestra.run.smithi090.stderr:10+0 records out 2024-04-14T15:06:59.007 INFO:teuthology.orchestra.run.smithi090.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.015614 s, 672 MB/s 2024-04-14T15:06:59.007 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping successful for: 2024-04-14T15:06:59.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:06:59 smithi090 ceph-mon[30399]: osdmap e14: 2 total, 2 up, 2 in 2024-04-14T15:06:59.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:06:59 smithi090 ceph-mon[41670]: osdmap e14: 2 total, 2 up, 2 in 2024-04-14T15:06:59.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:06:59 smithi092 ceph-mon[33966]: osdmap e14: 2 total, 2 up, 2 in 2024-04-14T15:06:59.841 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph orch daemon add osd smithi090:vg_nvme/lv_2 2024-04-14T15:07:00.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:00 smithi092 ceph-mon[33966]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:07:00.401 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:00 smithi090 ceph-mon[41670]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:07:00.402 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:00 smithi090 ceph-mon[30399]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:07:00.533 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:07:01.602 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:01 smithi090 ceph-mon[41670]: Detected new or changed devices on smithi090 2024-04-14T15:07:01.602 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:01 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:01.602 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:01 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:01.602 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:01 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:07:01.602 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:01 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:07:01.603 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:01 smithi090 ceph-mon[41670]: Adjusting osd_memory_target on smithi090 to 8032M 2024-04-14T15:07:01.603 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:01 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:01.603 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:01 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:07:01.603 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:01 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:07:01.603 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:01 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:01.603 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:01 smithi090 ceph-mon[30399]: Detected new or changed devices on smithi090 2024-04-14T15:07:01.603 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:01 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:01.603 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:01 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:01.603 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:01 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:07:01.603 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:01 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:07:01.603 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:01 smithi090 ceph-mon[30399]: Adjusting osd_memory_target on smithi090 to 8032M 2024-04-14T15:07:01.603 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:01 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:01.604 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:01 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:07:01.604 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:01 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:07:01.604 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:01 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:01.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:01 smithi092 ceph-mon[33966]: Detected new or changed devices on smithi090 2024-04-14T15:07:01.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:01 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:01.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:01 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:01.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:01 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:07:01.873 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:01 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:07:01.873 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:01 smithi092 ceph-mon[33966]: Adjusting osd_memory_target on smithi090 to 8032M 2024-04-14T15:07:01.873 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:01 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:01.873 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:01 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:07:01.873 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:01 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:07:01.873 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:01 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:02.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:02.092+0000 7f0334533700 1 -- 172.21.15.90:0/1953105519 >> v1:172.21.15.90:6790/0 conn(0x7f032c106d10 legacy=0x7f032c109160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:07:02.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:02.093+0000 7f0334533700 1 -- 172.21.15.90:0/1953105519 shutdown_connections 2024-04-14T15:07:02.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:02.093+0000 7f0334533700 1 -- 172.21.15.90:0/1953105519 >> 172.21.15.90:0/1953105519 conn(0x7f032c0fde80 msgr2=0x7f032c1002c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:07:02.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:02.093+0000 7f0334533700 1 -- 172.21.15.90:0/1953105519 shutdown_connections 2024-04-14T15:07:02.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:02.093+0000 7f0334533700 1 -- 172.21.15.90:0/1953105519 wait complete. 2024-04-14T15:07:02.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:02.093+0000 7f0334533700 1 librados: starting msgr at 2024-04-14T15:07:02.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:02.093+0000 7f0334533700 1 librados: starting objecter 2024-04-14T15:07:02.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:02.093+0000 7f0334533700 1 Processor -- start 2024-04-14T15:07:02.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:02.093+0000 7f0334533700 1 -- start start 2024-04-14T15:07:02.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:02.093+0000 7f0334533700 1 librados: setting wanted keys 2024-04-14T15:07:02.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:02.093+0000 7f0334533700 1 librados: calling monclient init 2024-04-14T15:07:02.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:02.093+0000 7f0334533700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f032c198930 con 0x7f032c1028e0 2024-04-14T15:07:02.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:02.093+0000 7f0334533700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f032c1a4120 con 0x7f032c10aa50 2024-04-14T15:07:02.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:02.093+0000 7f0334533700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f032c1a5300 con 0x7f032c106d10 2024-04-14T15:07:02.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:02.093+0000 7f0331ace700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f032c106d10 0x7f032c19e2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:59132/0 (socket says 172.21.15.90:59132) 2024-04-14T15:07:02.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:02.093+0000 7f03322cf700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f032c1028e0 0x7f032c197de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:57892/0 (socket says 172.21.15.90:57892) 2024-04-14T15:07:02.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:02.093+0000 7f0331ace700 1 -- 172.21.15.90:0/3502143730 learned_addr learned my addr 172.21.15.90:0/3502143730 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:07:02.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:02.093+0000 7f03237fe700 10 client.?.objecter ms_handle_connect 0x7f032c106d10 2024-04-14T15:07:02.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:02.094+0000 7f03237fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:07:02.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:02.094+0000 7f03237fe700 10 client.?.objecter ms_handle_connect 0x7f032c1028e0 2024-04-14T15:07:02.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:02.094+0000 7f03237fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:07:02.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:02.094+0000 7f03237fe700 1 -- 172.21.15.90:0/3502143730 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 203373875 0 0) 0x7f032c1a5300 con 0x7f032c106d10 2024-04-14T15:07:02.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:02.094+0000 7f03237fe700 1 -- 172.21.15.90:0/3502143730 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f03180035b0 con 0x7f032c106d10 2024-04-14T15:07:02.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:02.094+0000 7f03237fe700 1 -- 172.21.15.90:0/3502143730 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4095313817 0 0) 0x7f032c198930 con 0x7f032c1028e0 2024-04-14T15:07:02.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:02.094+0000 7f03237fe700 1 -- 172.21.15.90:0/3502143730 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f032c1a5300 con 0x7f032c1028e0 2024-04-14T15:07:02.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:02.094+0000 7f03237fe700 10 client.?.objecter ms_handle_connect 0x7f032c10aa50 2024-04-14T15:07:02.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:02.094+0000 7f03237fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:07:02.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:02.094+0000 7f03237fe700 1 -- 172.21.15.90:0/3502143730 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2482256052 0 0) 0x7f03180035b0 con 0x7f032c106d10 2024-04-14T15:07:02.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:02.094+0000 7f03237fe700 1 -- 172.21.15.90:0/3502143730 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f032c198930 con 0x7f032c106d10 2024-04-14T15:07:02.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:02.094+0000 7f03237fe700 1 -- 172.21.15.90:0/3502143730 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3257137704 0 0) 0x7f032c1a4120 con 0x7f032c10aa50 2024-04-14T15:07:02.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:02.094+0000 7f03237fe700 1 -- 172.21.15.90:0/3502143730 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f03180035b0 con 0x7f032c10aa50 2024-04-14T15:07:02.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:02.094+0000 7f03237fe700 1 -- 172.21.15.90:0/3502143730 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3259457438 0 0) 0x7f032c1a5300 con 0x7f032c1028e0 2024-04-14T15:07:02.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:02.094+0000 7f03237fe700 1 -- 172.21.15.90:0/3502143730 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f032c1a4120 con 0x7f032c1028e0 2024-04-14T15:07:02.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:02.094+0000 7f03237fe700 1 -- 172.21.15.90:0/3502143730 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f031c001510 con 0x7f032c106d10 2024-04-14T15:07:02.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:02.094+0000 7f03237fe700 1 -- 172.21.15.90:0/3502143730 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f0328001810 con 0x7f032c1028e0 2024-04-14T15:07:02.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:02.094+0000 7f03237fe700 1 -- 172.21.15.90:0/3502143730 <== mon.2 v1:172.21.15.90:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2757198639 0 0) 0x7f032c198930 con 0x7f032c106d10 2024-04-14T15:07:02.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:02.094+0000 7f03237fe700 1 -- 172.21.15.90:0/3502143730 >> v1:172.21.15.92:6789/0 conn(0x7f032c10aa50 legacy=0x7f032c1a19f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:07:02.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:02.094+0000 7f03237fe700 1 -- 172.21.15.90:0/3502143730 >> v1:172.21.15.90:6789/0 conn(0x7f032c1028e0 legacy=0x7f032c197de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:07:02.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:02.094+0000 7f03237fe700 1 -- 172.21.15.90:0/3502143730 --> v1:172.21.15.90:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f032c1a6520 con 0x7f032c106d10 2024-04-14T15:07:02.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:02.094+0000 7f0334533700 1 -- 172.21.15.90:0/3502143730 --> v1:172.21.15.90:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f032c1a3170 con 0x7f032c106d10 2024-04-14T15:07:02.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:02.094+0000 7f0334533700 10 client.24167.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:07:02.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:02.094+0000 7f0334533700 1 -- 172.21.15.90:0/3502143730 --> v1:172.21.15.90:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f032c1a36b0 con 0x7f032c106d10 2024-04-14T15:07:02.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:02.094+0000 7f0334533700 1 librados: init done 2024-04-14T15:07:02.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:02.094+0000 7f03237fe700 1 -- 172.21.15.90:0/3502143730 <== mon.2 v1:172.21.15.90:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f031c001ac0 con 0x7f032c106d10 2024-04-14T15:07:02.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:02.094+0000 7f03237fe700 1 -- 172.21.15.90:0/3502143730 <== mon.2 v1:172.21.15.90:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f031c004f80 con 0x7f032c106d10 2024-04-14T15:07:02.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:02.095+0000 7f0334533700 1 -- 172.21.15.90:0/3502143730 --> v1:172.21.15.90:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0310005150 con 0x7f032c106d10 2024-04-14T15:07:02.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:02.096+0000 7f03237fe700 1 -- 172.21.15.90:0/3502143730 <== mon.2 v1:172.21.15.90:6790/0 7 ==== mgrmap(e 14) ==== 92567+0+0 (unknown 4138089514 0 0) 0x7f031c005120 con 0x7f032c106d10 2024-04-14T15:07:02.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:02.096+0000 7f03237fe700 1 -- 172.21.15.90:0/3502143730 <== mon.2 v1:172.21.15.90:6790/0 8 ==== osd_map(14..14 src has 1..14) ==== 1833+0+0 (unknown 3558096432 0 0) 0x7f031c0891a0 con 0x7f032c106d10 2024-04-14T15:07:02.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:02.096+0000 7f03237fe700 10 client.24167.objecter ms_dispatch 0x7f032c001550 osd_map(14..14 src has 1..14) 2024-04-14T15:07:02.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:02.096+0000 7f03237fe700 3 client.24167.objecter handle_osd_map got epochs [14,14] > 0 2024-04-14T15:07:02.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:02.096+0000 7f03237fe700 3 client.24167.objecter handle_osd_map decoding full epoch 14 2024-04-14T15:07:02.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:02.096+0000 7f03237fe700 20 client.24167.objecter dump_active .. 0 homeless 2024-04-14T15:07:02.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:02.096+0000 7f03237fe700 10 client.24167.objecter ms_handle_connect 0x7f03180711e0 2024-04-14T15:07:02.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:02.101+0000 7f03237fe700 1 -- 172.21.15.90:0/3502143730 <== mon.2 v1:172.21.15.90:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f031c05fac0 con 0x7f032c106d10 2024-04-14T15:07:02.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:02.310+0000 7f0334533700 1 -- 172.21.15.90:0/3502143730 --> v1:172.21.15.90:6800/491071118 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f0310000bc0 con 0x7f03180711e0 2024-04-14T15:07:02.691 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:02 smithi090 ceph-mon[41670]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:07:02.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:02 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T15:07:02.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:02 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T15:07:02.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:02 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:07:02.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:02 smithi090 ceph-mon[30399]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:07:02.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:02 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T15:07:02.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:02 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T15:07:02.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:02 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:07:02.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:02 smithi092 ceph-mon[33966]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:07:02.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:02 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T15:07:02.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:02 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T15:07:02.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:02 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:07:03.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:03 smithi092 ceph-mon[33966]: from='client.24167 v1:172.21.15.90:0/3502143730' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:07:03.941 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:03 smithi090 ceph-mon[30399]: from='client.24167 v1:172.21.15.90:0/3502143730' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:07:03.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:03 smithi090 ceph-mon[41670]: from='client.24167 v1:172.21.15.90:0/3502143730' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:07:04.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:04 smithi092 ceph-mon[33966]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:07:04.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:04 smithi090 ceph-mon[30399]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:07:04.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:04 smithi090 ceph-mon[41670]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:07:05.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:05 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1607538740' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9c39b5ff-6175-46ba-9d71-370581c6ff3d"}]: dispatch 2024-04-14T15:07:05.873 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:05 smithi092 ceph-mon[33966]: from='client.24173 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9c39b5ff-6175-46ba-9d71-370581c6ff3d"}]: dispatch 2024-04-14T15:07:05.873 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:05 smithi092 ceph-mon[33966]: from='client.24173 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9c39b5ff-6175-46ba-9d71-370581c6ff3d"}]': finished 2024-04-14T15:07:05.873 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:05 smithi092 ceph-mon[33966]: osdmap e15: 3 total, 2 up, 3 in 2024-04-14T15:07:05.873 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:05 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T15:07:05.941 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:05 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1607538740' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9c39b5ff-6175-46ba-9d71-370581c6ff3d"}]: dispatch 2024-04-14T15:07:05.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:05 smithi090 ceph-mon[30399]: from='client.24173 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9c39b5ff-6175-46ba-9d71-370581c6ff3d"}]: dispatch 2024-04-14T15:07:05.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:05 smithi090 ceph-mon[30399]: from='client.24173 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9c39b5ff-6175-46ba-9d71-370581c6ff3d"}]': finished 2024-04-14T15:07:05.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:05 smithi090 ceph-mon[30399]: osdmap e15: 3 total, 2 up, 3 in 2024-04-14T15:07:05.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:05 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T15:07:05.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:05 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1607538740' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9c39b5ff-6175-46ba-9d71-370581c6ff3d"}]: dispatch 2024-04-14T15:07:05.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:05 smithi090 ceph-mon[41670]: from='client.24173 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9c39b5ff-6175-46ba-9d71-370581c6ff3d"}]: dispatch 2024-04-14T15:07:05.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:05 smithi090 ceph-mon[41670]: from='client.24173 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9c39b5ff-6175-46ba-9d71-370581c6ff3d"}]': finished 2024-04-14T15:07:05.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:05 smithi090 ceph-mon[41670]: osdmap e15: 3 total, 2 up, 3 in 2024-04-14T15:07:05.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:05 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T15:07:07.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:07.095+0000 7f0333d32700 10 client.24167.objecter tick 2024-04-14T15:07:07.096 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:06 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/894324626' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T15:07:07.096 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:06 smithi090 ceph-mon[30399]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:07:07.096 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:06 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/894324626' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T15:07:07.096 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:06 smithi090 ceph-mon[41670]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:07:07.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:06 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/894324626' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T15:07:07.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:06 smithi092 ceph-mon[33966]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:07:09.232 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:08 smithi090 ceph-mon[30399]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:07:09.232 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:08 smithi090 ceph-mon[41670]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:07:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:08 smithi092 ceph-mon[33966]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:07:11.350 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:10 smithi090 ceph-mon[41670]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:07:11.351 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:10 smithi090 ceph-mon[30399]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:07:11.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:10 smithi092 ceph-mon[33966]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:07:12.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:12.095+0000 7f0333d32700 10 client.24167.objecter tick 2024-04-14T15:07:12.347 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:11 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-14T15:07:12.347 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:11 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:07:12.348 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:11 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-14T15:07:12.348 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:11 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:07:12.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:11 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-14T15:07:12.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:11 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:07:13.309 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:12 smithi090 ceph-mon[30399]: Deploying daemon osd.2 on smithi090 2024-04-14T15:07:13.310 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:12 smithi090 ceph-mon[30399]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:07:13.310 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:12 smithi090 ceph-mon[41670]: Deploying daemon osd.2 on smithi090 2024-04-14T15:07:13.310 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:12 smithi090 ceph-mon[41670]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:07:13.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:12 smithi092 ceph-mon[33966]: Deploying daemon osd.2 on smithi090 2024-04-14T15:07:13.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:12 smithi092 ceph-mon[33966]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:07:15.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:14 smithi092 ceph-mon[33966]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:07:15.381 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:14 smithi090 ceph-mon[30399]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:07:15.381 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:14 smithi090 ceph-mon[41670]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:07:17.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:17.096+0000 7f0333d32700 10 client.24167.objecter tick 2024-04-14T15:07:17.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:16 smithi092 ceph-mon[33966]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:07:17.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:16 smithi090 ceph-mon[30399]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:07:17.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:16 smithi090 ceph-mon[41670]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:07:18.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:18 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:18.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:18 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:18.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:18 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:07:18.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:18 smithi090 ceph-mon[30399]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:07:18.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:18 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:18.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:18 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:18.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:18 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:07:18.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:18 smithi090 ceph-mon[41670]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:07:18.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:18 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:18.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:18 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:18.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:18 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:07:18.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:18 smithi092 ceph-mon[33966]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:07:20.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:20.962+0000 7f03237fe700 1 -- 172.21.15.90:0/3502143730 <== mgr.14148 v1:172.21.15.90:6800/491071118 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (unknown 0 0 2411459399) 0x7f0310000bc0 con 0x7f03180711e0 2024-04-14T15:07:20.963 INFO:teuthology.orchestra.run.smithi090.stdout:Created osd(s) 2 on host 'smithi090' 2024-04-14T15:07:20.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:20.964+0000 7f0334533700 10 librados: watch_flush enter 2024-04-14T15:07:20.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:20.964+0000 7f0334533700 10 librados: watch_flush exit 2024-04-14T15:07:20.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:20.964+0000 7f0334533700 20 client.24167.objecter shutdown clearing up homeless session... 2024-04-14T15:07:20.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:20.964+0000 7f0334533700 10 client.24167.objecter successfully canceled tick 2024-04-14T15:07:20.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:20.964+0000 7f0334533700 1 -- 172.21.15.90:0/3502143730 >> v1:172.21.15.90:6800/491071118 conn(0x7f03180711e0 legacy=0x7f03180736a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:07:20.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:20.965+0000 7f0334533700 1 -- 172.21.15.90:0/3502143730 >> v1:172.21.15.90:6790/0 conn(0x7f032c106d10 legacy=0x7f032c19e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:07:20.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:20.965+0000 7f0334533700 1 -- 172.21.15.90:0/3502143730 shutdown_connections 2024-04-14T15:07:20.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:20.965+0000 7f0334533700 1 -- 172.21.15.90:0/3502143730 >> 172.21.15.90:0/3502143730 conn(0x7f032c0fde80 msgr2=0x7f032c109160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:07:20.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:20.965+0000 7f0334533700 1 -- 172.21.15.90:0/3502143730 shutdown_connections 2024-04-14T15:07:20.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:20.965+0000 7f0334533700 1 -- 172.21.15.90:0/3502143730 wait complete. 2024-04-14T15:07:20.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:20.965+0000 7f0334533700 1 librados: shutdown 2024-04-14T15:07:21.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:20 smithi090 ceph-mon[30399]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:07:21.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:20 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:21.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:20 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:21.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:20 smithi090 ceph-mon[30399]: from='osd.2 v1:172.21.15.90:6809/1686184251' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-14T15:07:21.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:20 smithi090 ceph-mon[30399]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-14T15:07:21.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:20 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:21.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:20 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:21.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:20 smithi090 ceph-mon[41670]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:07:21.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:20 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:21.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:20 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:21.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:20 smithi090 ceph-mon[41670]: from='osd.2 v1:172.21.15.90:6809/1686184251' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-14T15:07:21.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:20 smithi090 ceph-mon[41670]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-14T15:07:21.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:20 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:21.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:20 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:21.325 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:20 smithi092 ceph-mon[33966]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:07:21.326 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:20 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:21.326 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:20 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:21.326 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:20 smithi092 ceph-mon[33966]: from='osd.2 v1:172.21.15.90:6809/1686184251' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-14T15:07:21.326 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:20 smithi092 ceph-mon[33966]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-14T15:07:21.326 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:20 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:21.326 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:20 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:21.611 DEBUG:teuthology.orchestra.run.smithi090:osd.2> sudo journalctl -f -n 0 -u ceph-0e288098-fa70-11ee-bc8e-c7b262605968@osd.2.service 2024-04-14T15:07:21.613 INFO:tasks.cephadm:Deploying osd.3 on smithi090 with /dev/vg_nvme/lv_1... 2024-04-14T15:07:21.614 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-14T15:07:22.358 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:07:22.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:22 smithi090 ceph-mon[30399]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-14T15:07:22.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:22 smithi090 ceph-mon[30399]: osdmap e16: 3 total, 2 up, 3 in 2024-04-14T15:07:22.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:22 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T15:07:22.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:22 smithi090 ceph-mon[30399]: from='osd.2 v1:172.21.15.90:6809/1686184251' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-04-14T15:07:22.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:22 smithi090 ceph-mon[30399]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-04-14T15:07:22.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:22 smithi090 ceph-mon[30399]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:07:22.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:22 smithi090 ceph-mon[41670]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-14T15:07:22.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:22 smithi090 ceph-mon[41670]: osdmap e16: 3 total, 2 up, 3 in 2024-04-14T15:07:22.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:22 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T15:07:22.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:22 smithi090 ceph-mon[41670]: from='osd.2 v1:172.21.15.90:6809/1686184251' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-04-14T15:07:22.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:22 smithi090 ceph-mon[41670]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-04-14T15:07:22.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:22 smithi090 ceph-mon[41670]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:07:22.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:22 smithi092 ceph-mon[33966]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-14T15:07:22.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:22 smithi092 ceph-mon[33966]: osdmap e16: 3 total, 2 up, 3 in 2024-04-14T15:07:22.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:22 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T15:07:22.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:22 smithi092 ceph-mon[33966]: from='osd.2 v1:172.21.15.90:6809/1686184251' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-04-14T15:07:22.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:22 smithi092 ceph-mon[33966]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-04-14T15:07:22.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:22 smithi092 ceph-mon[33966]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:07:22.941 INFO:journalctl@ceph.osd.2.smithi090.stdout:Apr 14 15:07:22 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-2[58450]: 2024-04-14T15:07:22.596+0000 7f7928a72700 -1 osd.2 0 waiting for initial osdmap 2024-04-14T15:07:22.942 INFO:journalctl@ceph.osd.2.smithi090.stdout:Apr 14 15:07:22 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-2[58450]: 2024-04-14T15:07:22.600+0000 7f792085d700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-14T15:07:23.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:23 smithi090 ceph-mon[30399]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-04-14T15:07:23.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:23 smithi090 ceph-mon[30399]: osdmap e17: 3 total, 2 up, 3 in 2024-04-14T15:07:23.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:23 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T15:07:23.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:23 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T15:07:23.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:23 smithi090 ceph-mon[30399]: from='osd.2 ' entity='osd.2' 2024-04-14T15:07:23.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:23 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T15:07:23.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:23 smithi090 ceph-mon[41670]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-04-14T15:07:23.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:23 smithi090 ceph-mon[41670]: osdmap e17: 3 total, 2 up, 3 in 2024-04-14T15:07:23.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:23 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T15:07:23.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:23 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T15:07:23.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:23 smithi090 ceph-mon[41670]: from='osd.2 ' entity='osd.2' 2024-04-14T15:07:23.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:23 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T15:07:23.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:23 smithi092 ceph-mon[33966]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-04-14T15:07:23.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:23 smithi092 ceph-mon[33966]: osdmap e17: 3 total, 2 up, 3 in 2024-04-14T15:07:23.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:23 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T15:07:23.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:23 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T15:07:23.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:23 smithi092 ceph-mon[33966]: from='osd.2 ' entity='osd.2' 2024-04-14T15:07:23.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:23 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T15:07:24.405 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:24 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:07:24.406 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:24 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:07:24.406 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:24 smithi090 ceph-mon[41670]: osd.2 v1:172.21.15.90:6809/1686184251 boot 2024-04-14T15:07:24.406 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:24 smithi090 ceph-mon[41670]: osdmap e18: 3 total, 3 up, 3 in 2024-04-14T15:07:24.406 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:24 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T15:07:24.406 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:24 smithi090 ceph-mon[41670]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:07:24.406 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:24 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:07:24.406 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:24 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:07:24.406 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:24 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:07:24.406 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:24 smithi090 ceph-mon[30399]: osd.2 v1:172.21.15.90:6809/1686184251 boot 2024-04-14T15:07:24.406 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:24 smithi090 ceph-mon[30399]: osdmap e18: 3 total, 3 up, 3 in 2024-04-14T15:07:24.406 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:24 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T15:07:24.407 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:24 smithi090 ceph-mon[30399]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:07:24.407 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:24 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:07:24.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:24 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:07:24.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:24 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:07:24.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:24 smithi092 ceph-mon[33966]: osd.2 v1:172.21.15.90:6809/1686184251 boot 2024-04-14T15:07:24.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:24 smithi092 ceph-mon[33966]: osdmap e18: 3 total, 3 up, 3 in 2024-04-14T15:07:24.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:24 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T15:07:24.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:24 smithi092 ceph-mon[33966]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:07:24.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:24 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:07:24.773 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-14T15:07:24.773 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-14T15:07:24.773 INFO:teuthology.orchestra.run.smithi090.stderr: stderr: 10+0 records in 2024-04-14T15:07:24.774 INFO:teuthology.orchestra.run.smithi090.stderr:10+0 records out 2024-04-14T15:07:24.774 INFO:teuthology.orchestra.run.smithi090.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0126905 s, 826 MB/s 2024-04-14T15:07:24.774 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping successful for: 2024-04-14T15:07:25.596 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph orch daemon add osd smithi090:vg_nvme/lv_1 2024-04-14T15:07:25.723 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:25 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-14T15:07:25.723 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:25 smithi090 ceph-mon[41670]: osdmap e19: 3 total, 3 up, 3 in 2024-04-14T15:07:25.723 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:25 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:07:25.724 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:25 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-14T15:07:25.724 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:25 smithi090 ceph-mon[30399]: osdmap e19: 3 total, 3 up, 3 in 2024-04-14T15:07:25.724 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:25 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:07:25.873 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:25 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-14T15:07:25.874 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:25 smithi092 ceph-mon[33966]: osdmap e19: 3 total, 3 up, 3 in 2024-04-14T15:07:25.874 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:25 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:07:26.180 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:07:26.244 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:26 smithi090 sudo[62416]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-14T15:07:26.245 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:26 smithi090 sudo[62416]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-14T15:07:26.245 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:26 smithi090 sudo[62416]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-14T15:07:26.245 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:26 smithi090 sudo[62416]: pam_unix(sudo:session): session closed for user root 2024-04-14T15:07:26.245 INFO:journalctl@ceph.osd.1.smithi090.stdout:Apr 14 15:07:26 smithi090 sudo[62324]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-14T15:07:26.245 INFO:journalctl@ceph.osd.1.smithi090.stdout:Apr 14 15:07:26 smithi090 sudo[62324]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-14T15:07:26.245 INFO:journalctl@ceph.osd.1.smithi090.stdout:Apr 14 15:07:26 smithi090 sudo[62324]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-14T15:07:26.245 INFO:journalctl@ceph.osd.1.smithi090.stdout:Apr 14 15:07:26 smithi090 sudo[62324]: pam_unix(sudo:session): session closed for user root 2024-04-14T15:07:26.245 INFO:journalctl@ceph.osd.1.smithi090.stdout:Apr 14 15:07:26 smithi090 sudo[62327]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-14T15:07:26.245 INFO:journalctl@ceph.osd.1.smithi090.stdout:Apr 14 15:07:26 smithi090 sudo[62327]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-14T15:07:26.245 INFO:journalctl@ceph.osd.1.smithi090.stdout:Apr 14 15:07:26 smithi090 sudo[62327]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-14T15:07:26.246 INFO:journalctl@ceph.osd.1.smithi090.stdout:Apr 14 15:07:26 smithi090 sudo[62327]: pam_unix(sudo:session): session closed for user root 2024-04-14T15:07:26.246 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:26 smithi090 sudo[62336]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-14T15:07:26.246 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:26 smithi090 sudo[62336]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-14T15:07:26.246 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:26 smithi090 sudo[62336]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-14T15:07:26.246 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:26 smithi090 sudo[62336]: pam_unix(sudo:session): session closed for user root 2024-04-14T15:07:26.246 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:26 smithi090 sudo[62353]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-14T15:07:26.246 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:26 smithi090 sudo[62353]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-14T15:07:26.246 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:26 smithi090 sudo[62353]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-14T15:07:26.246 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:26 smithi090 sudo[62353]: pam_unix(sudo:session): session closed for user root 2024-04-14T15:07:26.247 INFO:journalctl@ceph.osd.2.smithi090.stdout:Apr 14 15:07:26 smithi090 sudo[62330]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-14T15:07:26.247 INFO:journalctl@ceph.osd.2.smithi090.stdout:Apr 14 15:07:26 smithi090 sudo[62330]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-14T15:07:26.247 INFO:journalctl@ceph.osd.2.smithi090.stdout:Apr 14 15:07:26 smithi090 sudo[62330]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-14T15:07:26.247 INFO:journalctl@ceph.osd.2.smithi090.stdout:Apr 14 15:07:26 smithi090 sudo[62330]: pam_unix(sudo:session): session closed for user root 2024-04-14T15:07:26.247 INFO:journalctl@ceph.osd.2.smithi090.stdout:Apr 14 15:07:26 smithi090 sudo[62333]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-14T15:07:26.247 INFO:journalctl@ceph.osd.2.smithi090.stdout:Apr 14 15:07:26 smithi090 sudo[62333]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-14T15:07:26.247 INFO:journalctl@ceph.osd.2.smithi090.stdout:Apr 14 15:07:26 smithi090 sudo[62333]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-14T15:07:26.247 INFO:journalctl@ceph.osd.2.smithi090.stdout:Apr 14 15:07:26 smithi090 sudo[62333]: pam_unix(sudo:session): session closed for user root 2024-04-14T15:07:26.247 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 14 15:07:25 smithi090 sudo[62318]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-14T15:07:26.247 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 14 15:07:25 smithi090 sudo[62318]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-14T15:07:26.247 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 14 15:07:25 smithi090 sudo[62318]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-14T15:07:26.248 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 14 15:07:26 smithi090 sudo[62318]: pam_unix(sudo:session): session closed for user root 2024-04-14T15:07:26.248 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 14 15:07:26 smithi090 sudo[62321]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-14T15:07:26.248 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 14 15:07:26 smithi090 sudo[62321]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-14T15:07:26.248 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 14 15:07:26 smithi090 sudo[62321]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-14T15:07:26.248 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 14 15:07:26 smithi090 sudo[62321]: pam_unix(sudo:session): session closed for user root 2024-04-14T15:07:26.604 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:26 smithi090 sudo[62419]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-14T15:07:26.604 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:26 smithi090 sudo[62419]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-14T15:07:26.604 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:26 smithi090 sudo[62419]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-14T15:07:26.604 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:26 smithi090 sudo[62419]: pam_unix(sudo:session): session closed for user root 2024-04-14T15:07:26.613 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:26 smithi092 sudo[36727]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-14T15:07:26.613 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:26 smithi092 sudo[36727]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-14T15:07:26.614 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:26 smithi092 sudo[36727]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-14T15:07:26.614 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:26 smithi092 sudo[36727]: pam_unix(sudo:session): session closed for user root 2024-04-14T15:07:26.614 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:26 smithi092 sudo[36730]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-14T15:07:26.614 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:26 smithi092 sudo[36730]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-14T15:07:26.614 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:26 smithi092 sudo[36730]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-14T15:07:26.614 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:26 smithi092 sudo[36730]: pam_unix(sudo:session): session closed for user root 2024-04-14T15:07:26.873 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:26 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-14T15:07:26.873 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:26 smithi092 ceph-mon[33966]: osdmap e20: 3 total, 3 up, 3 in 2024-04-14T15:07:26.873 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:26 smithi092 ceph-mon[33966]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:07:26.873 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:26 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:26.873 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:26 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:26.874 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:26 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:26.874 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:26 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:07:26.874 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:26 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:07:26.874 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:26 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:07:26.874 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:26 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:26.874 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:26 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:07:26.874 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:26 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:07:26.874 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:26 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T15:07:26.874 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:26 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:07:26.874 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:26 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:07:26.875 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:26 smithi092 ceph-mon[33966]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-14T15:07:26.875 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:26 smithi092 ceph-mon[33966]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-14T15:07:26.875 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:26 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:26.875 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:26 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T15:07:26.875 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:26 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:07:26.875 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:26 smithi092 ceph-mon[33966]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-14T15:07:26.875 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:26 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:07:26.875 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:26 smithi092 ceph-mon[33966]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-14T15:07:26.875 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:26 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:26.875 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:26 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T15:07:26.875 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:26 smithi092 ceph-mon[33966]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-14T15:07:26.875 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:26 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:07:26.875 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:26 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:07:26.875 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:26 smithi092 ceph-mon[33966]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-14T15:07:26.876 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:26 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:07:26.876 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:26 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:07:26.876 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:26 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:07:26.876 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:26 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:26.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:26 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-14T15:07:26.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:26 smithi090 ceph-mon[41670]: osdmap e20: 3 total, 3 up, 3 in 2024-04-14T15:07:26.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:26 smithi090 ceph-mon[41670]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:07:26.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:26 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:26.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:26 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:26.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:26 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:26.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:26 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:07:26.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:26 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:07:26.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:26 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:07:26.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:26 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:26.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:26 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:07:26.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:26 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:07:26.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:26 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T15:07:26.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:26 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:07:26.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:26 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:07:26.944 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:26 smithi090 ceph-mon[41670]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-14T15:07:26.944 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:26 smithi090 ceph-mon[41670]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-14T15:07:26.944 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:26 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:26.944 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:26 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T15:07:26.944 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:26 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:07:26.944 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:26 smithi090 ceph-mon[41670]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-14T15:07:26.944 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:26 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:07:26.944 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:26 smithi090 ceph-mon[41670]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-14T15:07:26.944 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:26 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:26.944 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:26 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T15:07:26.944 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:26 smithi090 ceph-mon[41670]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-14T15:07:26.944 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:26 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:07:26.945 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:26 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:07:26.945 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:26 smithi090 ceph-mon[41670]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-14T15:07:26.945 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:26 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:07:26.945 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:26 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:07:26.945 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:26 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:07:26.945 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:26 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:26.945 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:26 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-14T15:07:26.945 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:26 smithi090 ceph-mon[30399]: osdmap e20: 3 total, 3 up, 3 in 2024-04-14T15:07:26.945 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:26 smithi090 ceph-mon[30399]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:07:26.945 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:26 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:26.945 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:26 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:26.946 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:26 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:26.946 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:26 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:07:26.946 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:26 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:07:26.946 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:26 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:07:26.946 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:26 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:26.946 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:26 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:07:26.946 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:26 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:07:26.946 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:26 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T15:07:26.946 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:26 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:07:26.946 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:26 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:07:26.946 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:26 smithi090 ceph-mon[30399]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-14T15:07:26.946 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:26 smithi090 ceph-mon[30399]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-14T15:07:26.946 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:26 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:26.947 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:26 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T15:07:26.947 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:26 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:07:26.947 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:26 smithi090 ceph-mon[30399]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-14T15:07:26.947 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:26 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:07:26.947 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:26 smithi090 ceph-mon[30399]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-14T15:07:26.947 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:26 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:26.947 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:26 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T15:07:26.947 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:26 smithi090 ceph-mon[30399]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-14T15:07:26.947 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:26 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:07:26.947 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:26 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:07:26.947 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:26 smithi090 ceph-mon[30399]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-14T15:07:26.947 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:26 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:07:26.947 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:26 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:07:26.947 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:26 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:07:26.947 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:26 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:27.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:27.822+0000 7fb3efd9e700 1 -- 172.21.15.90:0/1100178446 >> v1:172.21.15.90:6790/0 conn(0x7fb3e8105c10 legacy=0x7fb3e8108060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:07:27.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:27.822+0000 7fb3efd9e700 1 -- 172.21.15.90:0/1100178446 shutdown_connections 2024-04-14T15:07:27.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:27.822+0000 7fb3efd9e700 1 -- 172.21.15.90:0/1100178446 >> 172.21.15.90:0/1100178446 conn(0x7fb3e80fcd80 msgr2=0x7fb3e80ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:07:27.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:27.822+0000 7fb3efd9e700 1 -- 172.21.15.90:0/1100178446 shutdown_connections 2024-04-14T15:07:27.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:27.822+0000 7fb3efd9e700 1 -- 172.21.15.90:0/1100178446 wait complete. 2024-04-14T15:07:27.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:27.823+0000 7fb3efd9e700 1 librados: starting msgr at 2024-04-14T15:07:27.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:27.823+0000 7fb3efd9e700 1 librados: starting objecter 2024-04-14T15:07:27.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:27.823+0000 7fb3efd9e700 1 Processor -- start 2024-04-14T15:07:27.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:27.823+0000 7fb3efd9e700 1 -- start start 2024-04-14T15:07:27.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:27.823+0000 7fb3efd9e700 1 librados: setting wanted keys 2024-04-14T15:07:27.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:27.823+0000 7fb3efd9e700 1 librados: calling monclient init 2024-04-14T15:07:27.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:27.823+0000 7fb3efd9e700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb3e8197850 con 0x7fb3e8105c10 2024-04-14T15:07:27.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:27.823+0000 7fb3efd9e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb3e81a3030 con 0x7fb3e8109950 2024-04-14T15:07:27.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:27.823+0000 7fb3efd9e700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb3e81a4210 con 0x7fb3e81017e0 2024-04-14T15:07:27.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:27.823+0000 7fb3edb3a700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7fb3e81017e0 0x7fb3e8196d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:53126/0 (socket says 172.21.15.90:53126) 2024-04-14T15:07:27.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:27.823+0000 7fb3edb3a700 1 -- 172.21.15.90:0/2875462947 learned_addr learned my addr 172.21.15.90:0/2875462947 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:07:27.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:27.823+0000 7fb3ed339700 1 --1- 172.21.15.90:0/2875462947 >> v1:172.21.15.90:6789/0 conn(0x7fb3e8105c10 0x7fb3e819d1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:56370/0 (socket says 172.21.15.90:56370) 2024-04-14T15:07:27.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:27.823+0000 7fb3deffd700 10 client.?.objecter ms_handle_connect 0x7fb3e81017e0 2024-04-14T15:07:27.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:27.823+0000 7fb3deffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:07:27.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:27.823+0000 7fb3deffd700 10 client.?.objecter ms_handle_connect 0x7fb3e8105c10 2024-04-14T15:07:27.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:27.823+0000 7fb3deffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:07:27.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:27.823+0000 7fb3deffd700 10 client.?.objecter ms_handle_connect 0x7fb3e8109950 2024-04-14T15:07:27.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:27.823+0000 7fb3deffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:07:27.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:27.823+0000 7fb3deffd700 1 -- 172.21.15.90:0/2875462947 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3174241532 0 0) 0x7fb3e81a4210 con 0x7fb3e81017e0 2024-04-14T15:07:27.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:27.823+0000 7fb3deffd700 1 -- 172.21.15.90:0/2875462947 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb3d40035b0 con 0x7fb3e81017e0 2024-04-14T15:07:27.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:27.823+0000 7fb3deffd700 1 -- 172.21.15.90:0/2875462947 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2345515462 0 0) 0x7fb3e8197850 con 0x7fb3e8105c10 2024-04-14T15:07:27.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:27.823+0000 7fb3deffd700 1 -- 172.21.15.90:0/2875462947 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb3e81a4210 con 0x7fb3e8105c10 2024-04-14T15:07:27.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:27.823+0000 7fb3deffd700 1 -- 172.21.15.90:0/2875462947 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 813015676 0 0) 0x7fb3e81a3030 con 0x7fb3e8109950 2024-04-14T15:07:27.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:27.823+0000 7fb3deffd700 1 -- 172.21.15.90:0/2875462947 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb3e8197850 con 0x7fb3e8109950 2024-04-14T15:07:27.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:27.824+0000 7fb3deffd700 1 -- 172.21.15.90:0/2875462947 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1316682338 0 0) 0x7fb3d40035b0 con 0x7fb3e81017e0 2024-04-14T15:07:27.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:27.824+0000 7fb3deffd700 1 -- 172.21.15.90:0/2875462947 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb3e81a3030 con 0x7fb3e81017e0 2024-04-14T15:07:27.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:27.824+0000 7fb3deffd700 1 -- 172.21.15.90:0/2875462947 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3712044832 0 0) 0x7fb3e81a4210 con 0x7fb3e8105c10 2024-04-14T15:07:27.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:27.824+0000 7fb3deffd700 1 -- 172.21.15.90:0/2875462947 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb3d40035b0 con 0x7fb3e8105c10 2024-04-14T15:07:27.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:27.824+0000 7fb3deffd700 1 -- 172.21.15.90:0/2875462947 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fb3d80018f0 con 0x7fb3e81017e0 2024-04-14T15:07:27.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:27.824+0000 7fb3deffd700 1 -- 172.21.15.90:0/2875462947 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fb3e4003710 con 0x7fb3e8105c10 2024-04-14T15:07:27.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:27.824+0000 7fb3deffd700 1 -- 172.21.15.90:0/2875462947 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4029959198 0 0) 0x7fb3e8197850 con 0x7fb3e8109950 2024-04-14T15:07:27.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:27.824+0000 7fb3deffd700 1 -- 172.21.15.90:0/2875462947 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb3e81a4210 con 0x7fb3e8109950 2024-04-14T15:07:27.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:27.824+0000 7fb3deffd700 1 -- 172.21.15.90:0/2875462947 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fb3e0001590 con 0x7fb3e8109950 2024-04-14T15:07:27.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:27.824+0000 7fb3deffd700 1 -- 172.21.15.90:0/2875462947 <== mon.2 v1:172.21.15.90:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3211837110 0 0) 0x7fb3e81a3030 con 0x7fb3e81017e0 2024-04-14T15:07:27.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:27.824+0000 7fb3deffd700 1 -- 172.21.15.90:0/2875462947 >> v1:172.21.15.92:6789/0 conn(0x7fb3e8109950 legacy=0x7fb3e81a0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:07:27.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:27.824+0000 7fb3deffd700 1 -- 172.21.15.90:0/2875462947 >> v1:172.21.15.90:6789/0 conn(0x7fb3e8105c10 legacy=0x7fb3e819d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:07:27.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:27.824+0000 7fb3deffd700 1 -- 172.21.15.90:0/2875462947 --> v1:172.21.15.90:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3e81a5430 con 0x7fb3e81017e0 2024-04-14T15:07:27.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:27.824+0000 7fb3efd9e700 1 -- 172.21.15.90:0/2875462947 --> v1:172.21.15.90:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb3e81a4440 con 0x7fb3e81017e0 2024-04-14T15:07:27.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:27.824+0000 7fb3efd9e700 10 client.24203.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:07:27.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:27.824+0000 7fb3efd9e700 1 -- 172.21.15.90:0/2875462947 --> v1:172.21.15.90:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fb3e81a48c0 con 0x7fb3e81017e0 2024-04-14T15:07:27.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:27.824+0000 7fb3efd9e700 1 librados: init done 2024-04-14T15:07:27.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:27.824+0000 7fb3deffd700 1 -- 172.21.15.90:0/2875462947 <== mon.2 v1:172.21.15.90:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fb3d8001cc0 con 0x7fb3e81017e0 2024-04-14T15:07:27.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:27.824+0000 7fb3deffd700 1 -- 172.21.15.90:0/2875462947 <== mon.2 v1:172.21.15.90:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fb3d8005d00 con 0x7fb3e81017e0 2024-04-14T15:07:27.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:27.825+0000 7fb3efd9e700 1 -- 172.21.15.90:0/2875462947 --> v1:172.21.15.90:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3cc005120 con 0x7fb3e81017e0 2024-04-14T15:07:27.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:27.825+0000 7fb3deffd700 1 -- 172.21.15.90:0/2875462947 <== mon.2 v1:172.21.15.90:6790/0 7 ==== mgrmap(e 14) ==== 92567+0+0 (unknown 4138089514 0 0) 0x7fb3d8003950 con 0x7fb3e81017e0 2024-04-14T15:07:27.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:27.826+0000 7fb3deffd700 1 -- 172.21.15.90:0/2875462947 <== mon.2 v1:172.21.15.90:6790/0 8 ==== osd_map(21..21 src has 1..21) ==== 2511+0+0 (unknown 1717232963 0 0) 0x7fb3d80890e0 con 0x7fb3e81017e0 2024-04-14T15:07:27.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:27.826+0000 7fb3deffd700 10 client.24203.objecter ms_dispatch 0x7fb3e8001550 osd_map(21..21 src has 1..21) 2024-04-14T15:07:27.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:27.826+0000 7fb3deffd700 3 client.24203.objecter handle_osd_map got epochs [21,21] > 0 2024-04-14T15:07:27.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:27.826+0000 7fb3deffd700 3 client.24203.objecter handle_osd_map decoding full epoch 21 2024-04-14T15:07:27.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:27.826+0000 7fb3deffd700 20 client.24203.objecter dump_active .. 0 homeless 2024-04-14T15:07:27.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:27.826+0000 7fb3deffd700 10 client.24203.objecter ms_handle_connect 0x7fb3d4071260 2024-04-14T15:07:27.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:27.831+0000 7fb3deffd700 1 -- 172.21.15.90:0/2875462947 <== mon.2 v1:172.21.15.90:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fb3d8005d00 con 0x7fb3e81017e0 2024-04-14T15:07:27.908 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:27 smithi090 ceph-mon[30399]: Detected new or changed devices on smithi090 2024-04-14T15:07:27.908 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:27 smithi090 ceph-mon[30399]: Adjusting osd_memory_target on smithi090 to 5354M 2024-04-14T15:07:27.908 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:27 smithi090 ceph-mon[30399]: osdmap e21: 3 total, 3 up, 3 in 2024-04-14T15:07:27.909 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:27 smithi090 ceph-mon[41670]: Detected new or changed devices on smithi090 2024-04-14T15:07:27.909 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:27 smithi090 ceph-mon[41670]: Adjusting osd_memory_target on smithi090 to 5354M 2024-04-14T15:07:27.909 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:27 smithi090 ceph-mon[41670]: osdmap e21: 3 total, 3 up, 3 in 2024-04-14T15:07:28.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:28.065+0000 7fb3efd9e700 1 -- 172.21.15.90:0/2875462947 --> v1:172.21.15.90:6800/491071118 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fb3cc000b90 con 0x7fb3d4071260 2024-04-14T15:07:28.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:27 smithi092 ceph-mon[33966]: Detected new or changed devices on smithi090 2024-04-14T15:07:28.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:27 smithi092 ceph-mon[33966]: Adjusting osd_memory_target on smithi090 to 5354M 2024-04-14T15:07:28.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:27 smithi092 ceph-mon[33966]: osdmap e21: 3 total, 3 up, 3 in 2024-04-14T15:07:28.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:28.746+0000 7fb3deffd700 1 -- 172.21.15.90:0/2875462947 <== mon.2 v1:172.21.15.90:6790/0 10 ==== mgrmap(e 15) ==== 92623+0+0 (unknown 2343906571 0 0) 0x7fb3d8052810 con 0x7fb3e81017e0 2024-04-14T15:07:28.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:28 smithi090 ceph-mon[41670]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:07:28.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:28 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T15:07:28.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:28 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T15:07:28.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:28 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:07:28.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:28 smithi090 ceph-mon[30399]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:07:28.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:28 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T15:07:28.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:28 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T15:07:28.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:28 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:07:29.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:28 smithi092 ceph-mon[33966]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:07:29.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:28 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T15:07:29.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:28 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T15:07:29.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:28 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:07:30.000 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:29 smithi090 ceph-mon[41670]: from='client.24203 v1:172.21.15.90:0/2875462947' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:07:30.000 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:29 smithi090 ceph-mon[41670]: mgrmap e15: y(active, since 2m), standbys: x 2024-04-14T15:07:30.000 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:29 smithi090 ceph-mon[30399]: from='client.24203 v1:172.21.15.90:0/2875462947' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:07:30.000 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:29 smithi090 ceph-mon[30399]: mgrmap e15: y(active, since 2m), standbys: x 2024-04-14T15:07:30.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:29 smithi092 ceph-mon[33966]: from='client.24203 v1:172.21.15.90:0/2875462947' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:07:30.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:29 smithi092 ceph-mon[33966]: mgrmap e15: y(active, since 2m), standbys: x 2024-04-14T15:07:31.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:30 smithi092 ceph-mon[33966]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:07:31.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:30 smithi090 ceph-mon[30399]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:07:31.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:30 smithi090 ceph-mon[41670]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:07:32.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:31 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2685773770' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "efd2ca45-5b80-4013-a571-499e2cb5b2e8"}]: dispatch 2024-04-14T15:07:32.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:31 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2685773770' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "efd2ca45-5b80-4013-a571-499e2cb5b2e8"}]': finished 2024-04-14T15:07:32.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:31 smithi092 ceph-mon[33966]: osdmap e22: 4 total, 3 up, 4 in 2024-04-14T15:07:32.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:31 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T15:07:32.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:31 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/571249304' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T15:07:32.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:31 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2685773770' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "efd2ca45-5b80-4013-a571-499e2cb5b2e8"}]: dispatch 2024-04-14T15:07:32.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:31 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2685773770' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "efd2ca45-5b80-4013-a571-499e2cb5b2e8"}]': finished 2024-04-14T15:07:32.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:31 smithi090 ceph-mon[41670]: osdmap e22: 4 total, 3 up, 4 in 2024-04-14T15:07:32.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:31 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T15:07:32.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:31 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/571249304' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T15:07:32.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:31 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2685773770' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "efd2ca45-5b80-4013-a571-499e2cb5b2e8"}]: dispatch 2024-04-14T15:07:32.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:31 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2685773770' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "efd2ca45-5b80-4013-a571-499e2cb5b2e8"}]': finished 2024-04-14T15:07:32.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:31 smithi090 ceph-mon[30399]: osdmap e22: 4 total, 3 up, 4 in 2024-04-14T15:07:32.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:31 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T15:07:32.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:31 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/571249304' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T15:07:32.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:32.825+0000 7fb3ef59d700 10 client.24203.objecter tick 2024-04-14T15:07:33.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:32 smithi092 ceph-mon[33966]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:07:33.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:32 smithi090 ceph-mon[41670]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:07:33.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:32 smithi090 ceph-mon[30399]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:07:35.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:34 smithi090 ceph-mon[41670]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:07:35.305 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:34 smithi090 ceph-mon[30399]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:07:35.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:34 smithi092 ceph-mon[33966]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:07:37.353 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:36 smithi090 ceph-mon[41670]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:07:37.353 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:36 smithi090 ceph-mon[30399]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:07:37.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:36 smithi092 ceph-mon[33966]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:07:37.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:37.826+0000 7fb3ef59d700 10 client.24203.objecter tick 2024-04-14T15:07:38.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:37 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-14T15:07:38.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:37 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:07:38.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:37 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-14T15:07:38.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:37 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:07:38.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:37 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-14T15:07:38.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:37 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:07:39.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:38 smithi090 ceph-mon[30399]: Deploying daemon osd.3 on smithi090 2024-04-14T15:07:39.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:38 smithi090 ceph-mon[30399]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:07:39.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:38 smithi090 ceph-mon[41670]: Deploying daemon osd.3 on smithi090 2024-04-14T15:07:39.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:38 smithi090 ceph-mon[41670]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:07:39.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:38 smithi092 ceph-mon[33966]: Deploying daemon osd.3 on smithi090 2024-04-14T15:07:39.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:38 smithi092 ceph-mon[33966]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:07:41.229 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:40 smithi090 ceph-mon[30399]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:07:41.229 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:40 smithi090 ceph-mon[41670]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:07:41.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:40 smithi092 ceph-mon[33966]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:07:42.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:42.827+0000 7fb3ef59d700 10 client.24203.objecter tick 2024-04-14T15:07:43.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:42 smithi090 ceph-mon[30399]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:07:43.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:42 smithi090 ceph-mon[41670]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:07:43.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:42 smithi092 ceph-mon[33966]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:07:44.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:44 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:44.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:44 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:44.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:44 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:07:44.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:44 smithi092 ceph-mon[33966]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:07:44.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:44 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:44.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:44 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:44.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:44 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:07:44.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:44 smithi090 ceph-mon[41670]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:07:44.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:44 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:44.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:44 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:44.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:44 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:07:44.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:44 smithi090 ceph-mon[30399]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:07:46.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:46.971+0000 7fb3deffd700 1 -- 172.21.15.90:0/2875462947 <== mgr.14148 v1:172.21.15.90:6800/491071118 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (unknown 0 0 3551403507) 0x7fb3cc000b90 con 0x7fb3d4071260 2024-04-14T15:07:46.972 INFO:teuthology.orchestra.run.smithi090.stdout:Created osd(s) 3 on host 'smithi090' 2024-04-14T15:07:46.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:46.973+0000 7fb3efd9e700 10 librados: watch_flush enter 2024-04-14T15:07:46.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:46.973+0000 7fb3efd9e700 10 librados: watch_flush exit 2024-04-14T15:07:46.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:46.973+0000 7fb3efd9e700 20 client.24203.objecter shutdown clearing up homeless session... 2024-04-14T15:07:46.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:46.973+0000 7fb3efd9e700 10 client.24203.objecter successfully canceled tick 2024-04-14T15:07:46.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:46.973+0000 7fb3efd9e700 1 -- 172.21.15.90:0/2875462947 >> v1:172.21.15.90:6800/491071118 conn(0x7fb3d4071260 legacy=0x7fb3d4073720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:07:46.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:46.973+0000 7fb3efd9e700 1 -- 172.21.15.90:0/2875462947 >> v1:172.21.15.90:6790/0 conn(0x7fb3e81017e0 legacy=0x7fb3e8196d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:07:46.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:46.973+0000 7fb3efd9e700 1 -- 172.21.15.90:0/2875462947 shutdown_connections 2024-04-14T15:07:46.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:46.973+0000 7fb3efd9e700 1 -- 172.21.15.90:0/2875462947 >> 172.21.15.90:0/2875462947 conn(0x7fb3e80fcd80 msgr2=0x7fb3e8108860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:07:46.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:46.973+0000 7fb3efd9e700 1 -- 172.21.15.90:0/2875462947 shutdown_connections 2024-04-14T15:07:46.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:46.973+0000 7fb3efd9e700 1 -- 172.21.15.90:0/2875462947 wait complete. 2024-04-14T15:07:46.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:07:46.973+0000 7fb3efd9e700 1 librados: shutdown 2024-04-14T15:07:47.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:46 smithi090 ceph-mon[41670]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:07:47.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:46 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:47.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:46 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:47.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:46 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:07:47.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:46 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:07:47.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:46 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:47.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:46 smithi090 ceph-mon[41670]: from='osd.3 v1:172.21.15.90:6813/2270166774' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-14T15:07:47.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:46 smithi090 ceph-mon[41670]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-14T15:07:47.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:46 smithi090 ceph-mon[30399]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:07:47.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:46 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:47.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:46 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:47.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:46 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:07:47.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:46 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:07:47.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:46 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:47.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:46 smithi090 ceph-mon[30399]: from='osd.3 v1:172.21.15.90:6813/2270166774' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-14T15:07:47.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:46 smithi090 ceph-mon[30399]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-14T15:07:47.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:46 smithi092 ceph-mon[33966]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:07:47.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:46 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:47.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:46 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:47.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:46 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:07:47.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:46 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:07:47.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:46 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:47.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:46 smithi092 ceph-mon[33966]: from='osd.3 v1:172.21.15.90:6813/2270166774' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-14T15:07:47.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:46 smithi092 ceph-mon[33966]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-14T15:07:47.610 DEBUG:teuthology.orchestra.run.smithi090:osd.3> sudo journalctl -f -n 0 -u ceph-0e288098-fa70-11ee-bc8e-c7b262605968@osd.3.service 2024-04-14T15:07:47.612 INFO:tasks.cephadm:Deploying osd.4 on smithi092 with /dev/vg_nvme/lv_4... 2024-04-14T15:07:47.613 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-14T15:07:47.857 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:07:48.151 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:47 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:48.151 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:47 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:07:48.151 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:47 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:48.151 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:47 smithi090 ceph-mon[30399]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-14T15:07:48.151 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:47 smithi090 ceph-mon[30399]: osdmap e23: 4 total, 3 up, 4 in 2024-04-14T15:07:48.151 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:47 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T15:07:48.151 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:47 smithi090 ceph-mon[30399]: from='osd.3 v1:172.21.15.90:6813/2270166774' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-04-14T15:07:48.151 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:47 smithi090 ceph-mon[30399]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-04-14T15:07:48.152 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:47 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:48.152 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:47 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:07:48.152 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:47 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:48.152 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:47 smithi090 ceph-mon[41670]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-14T15:07:48.152 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:47 smithi090 ceph-mon[41670]: osdmap e23: 4 total, 3 up, 4 in 2024-04-14T15:07:48.152 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:47 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T15:07:48.152 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:47 smithi090 ceph-mon[41670]: from='osd.3 v1:172.21.15.90:6813/2270166774' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-04-14T15:07:48.152 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:47 smithi090 ceph-mon[41670]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-04-14T15:07:48.152 INFO:journalctl@ceph.osd.3.smithi090.stdout:Apr 14 15:07:47 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-3[64642]: 2024-04-14T15:07:47.936+0000 7f57ddf34700 -1 osd.3 0 waiting for initial osdmap 2024-04-14T15:07:48.152 INFO:journalctl@ceph.osd.3.smithi090.stdout:Apr 14 15:07:47 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-3[64642]: 2024-04-14T15:07:47.939+0000 7f57d9d27700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-14T15:07:48.242 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:47 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:48.243 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:47 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:07:48.243 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:47 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:48.243 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:47 smithi092 ceph-mon[33966]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-14T15:07:48.243 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:47 smithi092 ceph-mon[33966]: osdmap e23: 4 total, 3 up, 4 in 2024-04-14T15:07:48.243 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:47 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T15:07:48.243 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:47 smithi092 ceph-mon[33966]: from='osd.3 v1:172.21.15.90:6813/2270166774' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-04-14T15:07:48.243 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:47 smithi092 ceph-mon[33966]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-04-14T15:07:49.048 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:48 smithi090 ceph-mon[30399]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:07:49.048 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:48 smithi090 ceph-mon[30399]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-04-14T15:07:49.048 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:48 smithi090 ceph-mon[30399]: osdmap e24: 4 total, 3 up, 4 in 2024-04-14T15:07:49.048 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:48 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T15:07:49.048 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:48 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T15:07:49.048 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:48 smithi090 ceph-mon[41670]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:07:49.048 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:48 smithi090 ceph-mon[41670]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-04-14T15:07:49.048 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:48 smithi090 ceph-mon[41670]: osdmap e24: 4 total, 3 up, 4 in 2024-04-14T15:07:49.049 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:48 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T15:07:49.049 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:48 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T15:07:49.199 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:48 smithi092 ceph-mon[33966]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:07:49.199 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:48 smithi092 ceph-mon[33966]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-04-14T15:07:49.199 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:48 smithi092 ceph-mon[33966]: osdmap e24: 4 total, 3 up, 4 in 2024-04-14T15:07:49.199 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:48 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T15:07:49.199 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:48 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T15:07:49.656 INFO:teuthology.orchestra.run.smithi092.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-14T15:07:49.656 INFO:teuthology.orchestra.run.smithi092.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-14T15:07:49.656 INFO:teuthology.orchestra.run.smithi092.stderr: stderr: 10+0 records in 2024-04-14T15:07:49.656 INFO:teuthology.orchestra.run.smithi092.stderr:10+0 records out 2024-04-14T15:07:49.656 INFO:teuthology.orchestra.run.smithi092.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114525 s, 916 MB/s 2024-04-14T15:07:49.656 INFO:teuthology.orchestra.run.smithi092.stderr:--> Zapping successful for: 2024-04-14T15:07:50.188 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:49 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:07:50.188 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:49 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:07:50.188 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:49 smithi090 ceph-mon[41670]: osd.3 v1:172.21.15.90:6813/2270166774 boot 2024-04-14T15:07:50.188 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:49 smithi090 ceph-mon[41670]: osdmap e25: 4 total, 4 up, 4 in 2024-04-14T15:07:50.188 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:49 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T15:07:50.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:49 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:07:50.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:49 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:07:50.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:49 smithi090 ceph-mon[30399]: osd.3 v1:172.21.15.90:6813/2270166774 boot 2024-04-14T15:07:50.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:49 smithi090 ceph-mon[30399]: osdmap e25: 4 total, 4 up, 4 in 2024-04-14T15:07:50.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:49 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T15:07:50.195 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:49 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:07:50.195 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:49 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:07:50.195 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:49 smithi092 ceph-mon[33966]: osd.3 v1:172.21.15.90:6813/2270166774 boot 2024-04-14T15:07:50.195 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:49 smithi092 ceph-mon[33966]: osdmap e25: 4 total, 4 up, 4 in 2024-04-14T15:07:50.195 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:49 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T15:07:50.221 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph orch daemon add osd smithi092:vg_nvme/lv_4 2024-04-14T15:07:50.456 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:07:51.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:50 smithi092 ceph-mon[33966]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-14T15:07:51.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:50 smithi092 ceph-mon[33966]: osdmap e26: 4 total, 4 up, 4 in 2024-04-14T15:07:51.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:50 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:07:51.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:50 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:07:51.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:50 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:51.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:50 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:51.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:50 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:07:51.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:50 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:07:51.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:50 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:07:51.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:50 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:07:51.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:50 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:51.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:50 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:07:51.124 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:50 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:07:51.124 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:50 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:51.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:50 smithi090 ceph-mon[30399]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-14T15:07:51.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:50 smithi090 ceph-mon[30399]: osdmap e26: 4 total, 4 up, 4 in 2024-04-14T15:07:51.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:50 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:07:51.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:50 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:07:51.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:50 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:51.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:50 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:51.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:50 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:07:51.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:50 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:07:51.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:50 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:07:51.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:50 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:07:51.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:50 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:51.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:50 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:07:51.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:50 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:07:51.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:50 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:51.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:50 smithi090 ceph-mon[41670]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-14T15:07:51.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:50 smithi090 ceph-mon[41670]: osdmap e26: 4 total, 4 up, 4 in 2024-04-14T15:07:51.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:50 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:07:51.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:50 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:07:51.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:50 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:51.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:50 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:51.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:50 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:07:51.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:50 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:07:51.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:50 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:07:51.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:50 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:07:51.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:50 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:51.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:50 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:07:51.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:50 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:07:51.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:50 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:07:51.999 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:07:51.994+0000 7f6cd29e6700 1 -- 172.21.15.92:0/1202102058 >> v1:172.21.15.92:6789/0 conn(0x7f6ccc1017f0 legacy=0x7f6ccc101c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:07:51.999 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:07:51.994+0000 7f6cd29e6700 1 -- 172.21.15.92:0/1202102058 shutdown_connections 2024-04-14T15:07:51.999 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:07:51.994+0000 7f6cd29e6700 1 -- 172.21.15.92:0/1202102058 >> 172.21.15.92:0/1202102058 conn(0x7f6ccc0fcdb0 msgr2=0x7f6ccc0ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:07:51.999 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:07:51.995+0000 7f6cd29e6700 1 -- 172.21.15.92:0/1202102058 shutdown_connections 2024-04-14T15:07:51.999 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:07:51.995+0000 7f6cd29e6700 1 -- 172.21.15.92:0/1202102058 wait complete. 2024-04-14T15:07:51.999 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:07:51.995+0000 7f6cd29e6700 1 librados: starting msgr at 2024-04-14T15:07:51.999 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:07:51.995+0000 7f6cd29e6700 1 librados: starting objecter 2024-04-14T15:07:51.999 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:07:51.995+0000 7f6cd29e6700 1 Processor -- start 2024-04-14T15:07:51.999 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:07:51.995+0000 7f6cd29e6700 1 -- start start 2024-04-14T15:07:52.000 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:07:51.995+0000 7f6cd29e6700 1 librados: setting wanted keys 2024-04-14T15:07:52.000 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:07:51.995+0000 7f6cd29e6700 1 librados: calling monclient init 2024-04-14T15:07:52.000 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:07:51.995+0000 7f6cd29e6700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6ccc197a20 con 0x7f6ccc105c20 2024-04-14T15:07:52.000 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:07:51.995+0000 7f6cd29e6700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6ccc1a3210 con 0x7f6ccc109960 2024-04-14T15:07:52.000 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:07:51.995+0000 7f6cd29e6700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6ccc1a43f0 con 0x7f6ccc1017f0 2024-04-14T15:07:52.000 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:07:51.996+0000 7f6cd0f83700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f6ccc109960 0x7f6ccc1a0ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:34812/0 (socket says 172.21.15.92:34812) 2024-04-14T15:07:52.000 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:07:51.996+0000 7f6cd0f83700 1 -- 172.21.15.92:0/719819081 learned_addr learned my addr 172.21.15.92:0/719819081 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-04-14T15:07:52.000 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:07:51.996+0000 7f6cc97fa700 10 client.?.objecter ms_handle_connect 0x7f6ccc109960 2024-04-14T15:07:52.000 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:07:51.996+0000 7f6cc97fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:07:52.000 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:07:51.996+0000 7f6cc97fa700 10 client.?.objecter ms_handle_connect 0x7f6ccc1017f0 2024-04-14T15:07:52.000 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:07:51.996+0000 7f6cc97fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:07:52.000 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:07:51.996+0000 7f6cc97fa700 10 client.?.objecter ms_handle_connect 0x7f6ccc105c20 2024-04-14T15:07:52.000 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:07:51.996+0000 7f6cc97fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:07:52.000 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:07:51.996+0000 7f6cc97fa700 1 -- 172.21.15.92:0/719819081 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 122604433 0 0) 0x7f6ccc1a3210 con 0x7f6ccc109960 2024-04-14T15:07:52.001 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:07:51.996+0000 7f6cc97fa700 1 -- 172.21.15.92:0/719819081 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6cb40035b0 con 0x7f6ccc109960 2024-04-14T15:07:52.001 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:07:51.996+0000 7f6cc97fa700 1 -- 172.21.15.92:0/719819081 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1095807965 0 0) 0x7f6ccc1a43f0 con 0x7f6ccc1017f0 2024-04-14T15:07:52.001 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:07:51.996+0000 7f6cc97fa700 1 -- 172.21.15.92:0/719819081 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ccc1a3210 con 0x7f6ccc1017f0 2024-04-14T15:07:52.001 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:07:51.996+0000 7f6cc97fa700 1 -- 172.21.15.92:0/719819081 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2432725364 0 0) 0x7f6ccc197a20 con 0x7f6ccc105c20 2024-04-14T15:07:52.001 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:07:51.996+0000 7f6cc97fa700 1 -- 172.21.15.92:0/719819081 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ccc1a43f0 con 0x7f6ccc105c20 2024-04-14T15:07:52.001 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:07:51.996+0000 7f6cc97fa700 1 -- 172.21.15.92:0/719819081 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3437779542 0 0) 0x7f6ccc1a3210 con 0x7f6ccc1017f0 2024-04-14T15:07:52.001 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:07:51.996+0000 7f6cc97fa700 1 -- 172.21.15.92:0/719819081 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6ccc197a20 con 0x7f6ccc1017f0 2024-04-14T15:07:52.001 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:07:51.996+0000 7f6cc97fa700 1 -- 172.21.15.92:0/719819081 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1255815227 0 0) 0x7f6ccc1a43f0 con 0x7f6ccc105c20 2024-04-14T15:07:52.001 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:07:51.996+0000 7f6cc97fa700 1 -- 172.21.15.92:0/719819081 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6ccc1a3210 con 0x7f6ccc105c20 2024-04-14T15:07:52.001 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:07:51.996+0000 7f6cc97fa700 1 -- 172.21.15.92:0/719819081 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3769255419 0 0) 0x7f6cb40035b0 con 0x7f6ccc109960 2024-04-14T15:07:52.001 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:07:51.996+0000 7f6cc97fa700 1 -- 172.21.15.92:0/719819081 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6ccc1a43f0 con 0x7f6ccc109960 2024-04-14T15:07:52.001 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:07:51.996+0000 7f6cc97fa700 1 -- 172.21.15.92:0/719819081 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f6cbc001430 con 0x7f6ccc1017f0 2024-04-14T15:07:52.001 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:07:51.996+0000 7f6cc97fa700 1 -- 172.21.15.92:0/719819081 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f6cc00016a0 con 0x7f6ccc105c20 2024-04-14T15:07:52.002 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:07:51.996+0000 7f6cc97fa700 1 -- 172.21.15.92:0/719819081 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f6cc4001bf0 con 0x7f6ccc109960 2024-04-14T15:07:52.002 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:07:51.996+0000 7f6cc97fa700 1 -- 172.21.15.92:0/719819081 <== mon.2 v1:172.21.15.90:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1678249744 0 0) 0x7f6ccc197a20 con 0x7f6ccc1017f0 2024-04-14T15:07:52.002 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:07:51.996+0000 7f6cc97fa700 1 -- 172.21.15.92:0/719819081 >> v1:172.21.15.92:6789/0 conn(0x7f6ccc109960 legacy=0x7f6ccc1a0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:07:52.002 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:07:51.996+0000 7f6cc97fa700 1 -- 172.21.15.92:0/719819081 >> v1:172.21.15.90:6789/0 conn(0x7f6ccc105c20 legacy=0x7f6ccc19d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:07:52.002 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:07:51.996+0000 7f6cc97fa700 1 -- 172.21.15.92:0/719819081 --> v1:172.21.15.90:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ccc1a5610 con 0x7f6ccc1017f0 2024-04-14T15:07:52.002 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:07:51.996+0000 7f6cd29e6700 1 -- 172.21.15.92:0/719819081 --> v1:172.21.15.90:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6ccc1a3440 con 0x7f6ccc1017f0 2024-04-14T15:07:52.002 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:07:51.996+0000 7f6cd29e6700 10 client.24230.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:07:52.002 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:07:51.997+0000 7f6cd29e6700 1 -- 172.21.15.92:0/719819081 --> v1:172.21.15.90:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f6ccc1a3980 con 0x7f6ccc1017f0 2024-04-14T15:07:52.002 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:07:51.997+0000 7f6cd29e6700 1 librados: init done 2024-04-14T15:07:52.002 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:07:51.997+0000 7f6cc97fa700 1 -- 172.21.15.92:0/719819081 <== mon.2 v1:172.21.15.90:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f6cbc001bb0 con 0x7f6ccc1017f0 2024-04-14T15:07:52.002 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:07:51.997+0000 7f6cc97fa700 1 -- 172.21.15.92:0/719819081 <== mon.2 v1:172.21.15.90:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f6cbc004ee0 con 0x7f6ccc1017f0 2024-04-14T15:07:52.002 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:07:51.997+0000 7f6cd29e6700 1 -- 172.21.15.92:0/719819081 --> v1:172.21.15.90:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cb8005120 con 0x7f6ccc1017f0 2024-04-14T15:07:52.002 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:07:51.998+0000 7f6cc97fa700 1 -- 172.21.15.92:0/719819081 <== mon.2 v1:172.21.15.90:6790/0 7 ==== mgrmap(e 15) ==== 92623+0+0 (unknown 2343906571 0 0) 0x7f6cbc003ab0 con 0x7f6ccc1017f0 2024-04-14T15:07:52.003 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:07:51.998+0000 7f6cc97fa700 1 -- 172.21.15.92:0/719819081 <== mon.2 v1:172.21.15.90:6790/0 8 ==== osd_map(27..27 src has 1..27) ==== 2803+0+0 (unknown 362424274 0 0) 0x7f6cbc0884f0 con 0x7f6ccc1017f0 2024-04-14T15:07:52.003 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:07:51.998+0000 7f6cc97fa700 10 client.24230.objecter ms_dispatch 0x7f6ccc001550 osd_map(27..27 src has 1..27) 2024-04-14T15:07:52.003 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:07:51.998+0000 7f6cc97fa700 3 client.24230.objecter handle_osd_map got epochs [27,27] > 0 2024-04-14T15:07:52.003 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:07:51.998+0000 7f6cc97fa700 3 client.24230.objecter handle_osd_map decoding full epoch 27 2024-04-14T15:07:52.003 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:07:51.998+0000 7f6cc97fa700 20 client.24230.objecter dump_active .. 0 homeless 2024-04-14T15:07:52.003 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:07:51.999+0000 7f6cc97fa700 10 client.24230.objecter ms_handle_connect 0x7f6cb40712e0 2024-04-14T15:07:52.004 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:07:52.003+0000 7f6cc97fa700 1 -- 172.21.15.92:0/719819081 <== mon.2 v1:172.21.15.90:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f6cbc0563f0 con 0x7f6ccc1017f0 2024-04-14T15:07:52.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:51 smithi092 ceph-mon[33966]: Detected new or changed devices on smithi090 2024-04-14T15:07:52.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:51 smithi092 ceph-mon[33966]: Adjusting osd_memory_target on smithi090 to 4016M 2024-04-14T15:07:52.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:51 smithi092 ceph-mon[33966]: osdmap e27: 4 total, 4 up, 4 in 2024-04-14T15:07:52.209 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:07:52.208+0000 7f6cd29e6700 1 -- 172.21.15.92:0/719819081 --> v1:172.21.15.90:6800/491071118 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f6cb8000b90 con 0x7f6cb40712e0 2024-04-14T15:07:52.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:51 smithi090 ceph-mon[30399]: Detected new or changed devices on smithi090 2024-04-14T15:07:52.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:51 smithi090 ceph-mon[30399]: Adjusting osd_memory_target on smithi090 to 4016M 2024-04-14T15:07:52.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:51 smithi090 ceph-mon[30399]: osdmap e27: 4 total, 4 up, 4 in 2024-04-14T15:07:52.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:51 smithi090 ceph-mon[41670]: Detected new or changed devices on smithi090 2024-04-14T15:07:52.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:51 smithi090 ceph-mon[41670]: Adjusting osd_memory_target on smithi090 to 4016M 2024-04-14T15:07:52.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:51 smithi090 ceph-mon[41670]: osdmap e27: 4 total, 4 up, 4 in 2024-04-14T15:07:53.097 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:52 smithi092 ceph-mon[33966]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-14T15:07:53.098 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:52 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T15:07:53.098 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:52 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T15:07:53.098 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:52 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:07:53.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:52 smithi090 ceph-mon[30399]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-14T15:07:53.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:52 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T15:07:53.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:52 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T15:07:53.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:52 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:07:53.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:52 smithi090 ceph-mon[41670]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-14T15:07:53.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:52 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T15:07:53.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:52 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T15:07:53.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:52 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:07:54.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:53 smithi092 ceph-mon[33966]: from='client.24230 v1:172.21.15.92:0/719819081' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:07:54.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:53 smithi090 ceph-mon[30399]: from='client.24230 v1:172.21.15.92:0/719819081' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:07:54.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:53 smithi090 ceph-mon[41670]: from='client.24230 v1:172.21.15.92:0/719819081' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:07:55.342 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:54 smithi090 ceph-mon[41670]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-14T15:07:55.343 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:54 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.92:0/608715014' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e3d9013d-5eb6-433c-96b3-3fc38ba7ac8d"}]: dispatch 2024-04-14T15:07:55.343 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:54 smithi090 ceph-mon[41670]: from='client.24232 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e3d9013d-5eb6-433c-96b3-3fc38ba7ac8d"}]: dispatch 2024-04-14T15:07:55.343 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:54 smithi090 ceph-mon[41670]: from='client.24232 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e3d9013d-5eb6-433c-96b3-3fc38ba7ac8d"}]': finished 2024-04-14T15:07:55.343 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:54 smithi090 ceph-mon[41670]: osdmap e28: 5 total, 4 up, 5 in 2024-04-14T15:07:55.343 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:54 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T15:07:55.343 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:54 smithi090 ceph-mon[30399]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-14T15:07:55.343 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:54 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.92:0/608715014' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e3d9013d-5eb6-433c-96b3-3fc38ba7ac8d"}]: dispatch 2024-04-14T15:07:55.343 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:54 smithi090 ceph-mon[30399]: from='client.24232 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e3d9013d-5eb6-433c-96b3-3fc38ba7ac8d"}]: dispatch 2024-04-14T15:07:55.343 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:54 smithi090 ceph-mon[30399]: from='client.24232 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e3d9013d-5eb6-433c-96b3-3fc38ba7ac8d"}]': finished 2024-04-14T15:07:55.343 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:54 smithi090 ceph-mon[30399]: osdmap e28: 5 total, 4 up, 5 in 2024-04-14T15:07:55.343 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:54 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T15:07:55.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:54 smithi092 ceph-mon[33966]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-14T15:07:55.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:54 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.92:0/608715014' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e3d9013d-5eb6-433c-96b3-3fc38ba7ac8d"}]: dispatch 2024-04-14T15:07:55.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:54 smithi092 ceph-mon[33966]: from='client.24232 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e3d9013d-5eb6-433c-96b3-3fc38ba7ac8d"}]: dispatch 2024-04-14T15:07:55.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:54 smithi092 ceph-mon[33966]: from='client.24232 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e3d9013d-5eb6-433c-96b3-3fc38ba7ac8d"}]': finished 2024-04-14T15:07:55.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:54 smithi092 ceph-mon[33966]: osdmap e28: 5 total, 4 up, 5 in 2024-04-14T15:07:55.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:54 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T15:07:56.358 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:55 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.92:0/1049758382' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T15:07:56.358 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:55 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.92:0/1049758382' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T15:07:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:55 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.92:0/1049758382' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T15:07:56.998 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:07:56.998+0000 7f6cd21e5700 10 client.24230.objecter tick 2024-04-14T15:07:57.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:56 smithi092 ceph-mon[33966]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-14T15:07:57.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:56 smithi090 ceph-mon[30399]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-14T15:07:57.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:56 smithi090 ceph-mon[41670]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-14T15:07:59.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:07:58 smithi092 ceph-mon[33966]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-14T15:07:59.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:07:58 smithi090 ceph-mon[41670]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-14T15:07:59.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:07:58 smithi090 ceph-mon[30399]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-14T15:08:01.080 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:00 smithi092 ceph-mon[33966]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-14T15:08:01.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:00 smithi090 ceph-mon[41670]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-14T15:08:01.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:00 smithi090 ceph-mon[30399]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-14T15:08:01.998 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:01.998+0000 7f6cd21e5700 10 client.24230.objecter tick 2024-04-14T15:08:02.293 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:01 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-14T15:08:02.294 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:01 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:08:02.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:01 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-14T15:08:02.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:01 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:08:02.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:01 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-14T15:08:02.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:01 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:08:03.266 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:02 smithi092 ceph-mon[33966]: Deploying daemon osd.4 on smithi092 2024-04-14T15:08:03.266 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:02 smithi092 ceph-mon[33966]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-14T15:08:03.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:02 smithi090 ceph-mon[30399]: Deploying daemon osd.4 on smithi092 2024-04-14T15:08:03.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:02 smithi090 ceph-mon[30399]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-14T15:08:03.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:02 smithi090 ceph-mon[41670]: Deploying daemon osd.4 on smithi092 2024-04-14T15:08:03.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:02 smithi090 ceph-mon[41670]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-14T15:08:05.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:04 smithi092 ceph-mon[33966]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-14T15:08:05.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:04 smithi090 ceph-mon[30399]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-14T15:08:05.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:04 smithi090 ceph-mon[41670]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-14T15:08:07.002 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:06 smithi092 ceph-mon[33966]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-14T15:08:07.002 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:06 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:07.002 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:06 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:08:07.002 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:06 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:07.002 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:06.999+0000 7f6cd21e5700 10 client.24230.objecter tick 2024-04-14T15:08:07.051 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:06 smithi090 ceph-mon[41670]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-14T15:08:07.051 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:06 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:07.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:06 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:08:07.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:06 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:07.052 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:06 smithi090 ceph-mon[30399]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-14T15:08:07.052 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:06 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:07.052 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:06 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:08:07.052 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:06 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:09.099 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:08 smithi092 ceph-mon[33966]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-14T15:08:09.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:08 smithi090 ceph-mon[41670]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-14T15:08:09.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:08 smithi090 ceph-mon[30399]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-14T15:08:10.069 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:10.068+0000 7f6cc97fa700 1 -- 172.21.15.92:0/719819081 <== mgr.14148 v1:172.21.15.90:6800/491071118 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (unknown 0 0 222605315) 0x7f6cb8000b90 con 0x7f6cb40712e0 2024-04-14T15:08:10.069 INFO:teuthology.orchestra.run.smithi092.stdout:Created osd(s) 4 on host 'smithi092' 2024-04-14T15:08:10.071 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:10.070+0000 7f6cd29e6700 10 librados: watch_flush enter 2024-04-14T15:08:10.071 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:10.070+0000 7f6cd29e6700 10 librados: watch_flush exit 2024-04-14T15:08:10.071 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:10.070+0000 7f6cd29e6700 20 client.24230.objecter shutdown clearing up homeless session... 2024-04-14T15:08:10.071 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:10.070+0000 7f6cd29e6700 10 client.24230.objecter successfully canceled tick 2024-04-14T15:08:10.071 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:10.070+0000 7f6cd29e6700 1 -- 172.21.15.92:0/719819081 >> v1:172.21.15.90:6800/491071118 conn(0x7f6cb40712e0 legacy=0x7f6cb40737a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:08:10.071 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:10.070+0000 7f6cd29e6700 1 -- 172.21.15.92:0/719819081 >> v1:172.21.15.90:6790/0 conn(0x7f6ccc1017f0 legacy=0x7f6ccc196ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:08:10.071 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:10.070+0000 7f6cd29e6700 1 -- 172.21.15.92:0/719819081 shutdown_connections 2024-04-14T15:08:10.071 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:10.070+0000 7f6cd29e6700 1 -- 172.21.15.92:0/719819081 >> 172.21.15.92:0/719819081 conn(0x7f6ccc0fcdb0 msgr2=0x7f6ccc0ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:08:10.072 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:10.070+0000 7f6cd29e6700 1 -- 172.21.15.92:0/719819081 shutdown_connections 2024-04-14T15:08:10.072 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:10.070+0000 7f6cd29e6700 1 -- 172.21.15.92:0/719819081 wait complete. 2024-04-14T15:08:10.072 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:10.070+0000 7f6cd29e6700 1 librados: shutdown 2024-04-14T15:08:10.365 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:10 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:10.365 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:10 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:10.365 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:10 smithi092 ceph-mon[33966]: from='osd.4 v1:172.21.15.92:6800/3887549733' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-14T15:08:10.365 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:10 smithi092 ceph-mon[33966]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-14T15:08:10.365 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:10 smithi092 ceph-mon[33966]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-14T15:08:10.365 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:10 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:10.365 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:10 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:10.667 DEBUG:teuthology.orchestra.run.smithi092:osd.4> sudo journalctl -f -n 0 -u ceph-0e288098-fa70-11ee-bc8e-c7b262605968@osd.4.service 2024-04-14T15:08:10.670 INFO:tasks.cephadm:Deploying osd.5 on smithi092 with /dev/vg_nvme/lv_3... 2024-04-14T15:08:10.670 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-14T15:08:10.691 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:10 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:10.691 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:10 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:10.691 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:10 smithi090 ceph-mon[41670]: from='osd.4 v1:172.21.15.92:6800/3887549733' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-14T15:08:10.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:10 smithi090 ceph-mon[41670]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-14T15:08:10.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:10 smithi090 ceph-mon[41670]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-14T15:08:10.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:10 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:10.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:10 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:10.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:10 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:10.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:10 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:10.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:10 smithi090 ceph-mon[30399]: from='osd.4 v1:172.21.15.92:6800/3887549733' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-14T15:08:10.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:10 smithi090 ceph-mon[30399]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-14T15:08:10.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:10 smithi090 ceph-mon[30399]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-14T15:08:10.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:10 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:10.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:10 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:10.960 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:08:11.368 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:11 smithi092 ceph-mon[33966]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-14T15:08:11.369 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:11 smithi092 ceph-mon[33966]: from='osd.4 v1:172.21.15.92:6800/3887549733' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-04-14T15:08:11.369 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:11 smithi092 ceph-mon[33966]: osdmap e29: 5 total, 4 up, 5 in 2024-04-14T15:08:11.369 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:11 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T15:08:11.369 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:11 smithi092 ceph-mon[33966]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-04-14T15:08:11.369 INFO:journalctl@ceph.osd.4.smithi092.stdout:Apr 14 15:08:11 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-4[39413]: 2024-04-14T15:08:11.251+0000 7f74597e7700 -1 osd.4 0 waiting for initial osdmap 2024-04-14T15:08:11.369 INFO:journalctl@ceph.osd.4.smithi092.stdout:Apr 14 15:08:11 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-4[39413]: 2024-04-14T15:08:11.255+0000 7f74555da700 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-14T15:08:11.691 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:11 smithi090 ceph-mon[30399]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-14T15:08:11.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:11 smithi090 ceph-mon[30399]: from='osd.4 v1:172.21.15.92:6800/3887549733' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-04-14T15:08:11.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:11 smithi090 ceph-mon[30399]: osdmap e29: 5 total, 4 up, 5 in 2024-04-14T15:08:11.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:11 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T15:08:11.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:11 smithi090 ceph-mon[30399]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-04-14T15:08:11.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:11 smithi090 ceph-mon[41670]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-14T15:08:11.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:11 smithi090 ceph-mon[41670]: from='osd.4 v1:172.21.15.92:6800/3887549733' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-04-14T15:08:11.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:11 smithi090 ceph-mon[41670]: osdmap e29: 5 total, 4 up, 5 in 2024-04-14T15:08:11.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:11 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T15:08:11.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:11 smithi090 ceph-mon[41670]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-04-14T15:08:12.313 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:12 smithi092 ceph-mon[33966]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2024-04-14T15:08:12.313 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:12 smithi092 ceph-mon[33966]: osdmap e30: 5 total, 4 up, 5 in 2024-04-14T15:08:12.314 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:12 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T15:08:12.314 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:12 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T15:08:12.314 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:12 smithi092 ceph-mon[33966]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-14T15:08:12.691 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:12 smithi090 ceph-mon[41670]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2024-04-14T15:08:12.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:12 smithi090 ceph-mon[41670]: osdmap e30: 5 total, 4 up, 5 in 2024-04-14T15:08:12.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:12 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T15:08:12.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:12 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T15:08:12.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:12 smithi090 ceph-mon[41670]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-14T15:08:12.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:12 smithi090 ceph-mon[30399]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2024-04-14T15:08:12.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:12 smithi090 ceph-mon[30399]: osdmap e30: 5 total, 4 up, 5 in 2024-04-14T15:08:12.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:12 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T15:08:12.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:12 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T15:08:12.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:12 smithi090 ceph-mon[30399]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-14T15:08:12.854 INFO:teuthology.orchestra.run.smithi092.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-14T15:08:12.854 INFO:teuthology.orchestra.run.smithi092.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-14T15:08:12.854 INFO:teuthology.orchestra.run.smithi092.stderr: stderr: 10+0 records in 2024-04-14T15:08:12.854 INFO:teuthology.orchestra.run.smithi092.stderr:10+0 records out 2024-04-14T15:08:12.854 INFO:teuthology.orchestra.run.smithi092.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.013537 s, 775 MB/s 2024-04-14T15:08:12.854 INFO:teuthology.orchestra.run.smithi092.stderr: stderr: 2024-04-14T15:08:12.854 INFO:teuthology.orchestra.run.smithi092.stderr:--> Zapping successful for: 2024-04-14T15:08:13.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:13 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:08:13.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:13 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:08:13.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:13 smithi092 ceph-mon[33966]: osd.4 v1:172.21.15.92:6800/3887549733 boot 2024-04-14T15:08:13.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:13 smithi092 ceph-mon[33966]: osdmap e31: 5 total, 5 up, 5 in 2024-04-14T15:08:13.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:13 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T15:08:13.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:13 smithi092 ceph-mon[33966]: osdmap e32: 5 total, 5 up, 5 in 2024-04-14T15:08:13.691 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:13 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:08:13.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:13 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:08:13.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:13 smithi090 ceph-mon[41670]: osd.4 v1:172.21.15.92:6800/3887549733 boot 2024-04-14T15:08:13.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:13 smithi090 ceph-mon[41670]: osdmap e31: 5 total, 5 up, 5 in 2024-04-14T15:08:13.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:13 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T15:08:13.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:13 smithi090 ceph-mon[41670]: osdmap e32: 5 total, 5 up, 5 in 2024-04-14T15:08:13.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:13 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:08:13.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:13 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:08:13.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:13 smithi090 ceph-mon[30399]: osd.4 v1:172.21.15.92:6800/3887549733 boot 2024-04-14T15:08:13.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:13 smithi090 ceph-mon[30399]: osdmap e31: 5 total, 5 up, 5 in 2024-04-14T15:08:13.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:13 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T15:08:13.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:13 smithi090 ceph-mon[30399]: osdmap e32: 5 total, 5 up, 5 in 2024-04-14T15:08:14.138 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph orch daemon add osd smithi092:vg_nvme/lv_3 2024-04-14T15:08:14.364 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:14 smithi092 ceph-mon[33966]: pgmap v116: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-14T15:08:14.364 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:14 smithi092 ceph-mon[33966]: osdmap e33: 5 total, 5 up, 5 in 2024-04-14T15:08:14.571 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:08:14.691 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:14 smithi090 ceph-mon[41670]: pgmap v116: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-14T15:08:14.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:14 smithi090 ceph-mon[41670]: osdmap e33: 5 total, 5 up, 5 in 2024-04-14T15:08:14.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:14 smithi090 ceph-mon[30399]: pgmap v116: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-14T15:08:14.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:14 smithi090 ceph-mon[30399]: osdmap e33: 5 total, 5 up, 5 in 2024-04-14T15:08:16.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:16 smithi092 ceph-mon[33966]: osdmap e34: 5 total, 5 up, 5 in 2024-04-14T15:08:16.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:16 smithi092 ceph-mon[33966]: pgmap v119: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-14T15:08:16.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:16 smithi092 ceph-mon[33966]: Detected new or changed devices on smithi092 2024-04-14T15:08:16.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:16 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:16.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:16 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:16.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:16 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:08:16.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:16 smithi092 ceph-mon[33966]: Adjusting osd_memory_target on smithi092 to 17088M 2024-04-14T15:08:16.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:16 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:16.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:16 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:08:16.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:16 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:08:16.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:16 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:16.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:16 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:08:16.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:16 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:08:16.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:16 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:08:16.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:16 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:16.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:16 smithi090 ceph-mon[30399]: osdmap e34: 5 total, 5 up, 5 in 2024-04-14T15:08:16.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:16 smithi090 ceph-mon[30399]: pgmap v119: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-14T15:08:16.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:16 smithi090 ceph-mon[30399]: Detected new or changed devices on smithi092 2024-04-14T15:08:16.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:16 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:16.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:16 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:16.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:16 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:08:16.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:16 smithi090 ceph-mon[30399]: Adjusting osd_memory_target on smithi092 to 17088M 2024-04-14T15:08:16.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:16 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:16.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:16 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:08:16.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:16 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:08:16.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:16 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:16.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:16 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:08:16.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:16 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:08:16.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:16 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:08:16.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:16 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:16.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:16 smithi090 ceph-mon[41670]: osdmap e34: 5 total, 5 up, 5 in 2024-04-14T15:08:16.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:16 smithi090 ceph-mon[41670]: pgmap v119: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-14T15:08:16.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:16 smithi090 ceph-mon[41670]: Detected new or changed devices on smithi092 2024-04-14T15:08:16.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:16 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:16.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:16 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:16.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:16 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:08:16.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:16 smithi090 ceph-mon[41670]: Adjusting osd_memory_target on smithi092 to 17088M 2024-04-14T15:08:16.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:16 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:16.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:16 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:08:16.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:16 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:08:16.694 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:16 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:16.694 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:16 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:08:16.694 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:16 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:08:16.694 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:16 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:08:16.694 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:16 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:16.885 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:16.881+0000 7f06104fb700 1 -- 172.21.15.92:0/1348770643 >> v1:172.21.15.92:6789/0 conn(0x7f0608106d20 legacy=0x7f0608109170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:08:16.885 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:16.881+0000 7f06104fb700 1 -- 172.21.15.92:0/1348770643 shutdown_connections 2024-04-14T15:08:16.885 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:16.881+0000 7f06104fb700 1 -- 172.21.15.92:0/1348770643 >> 172.21.15.92:0/1348770643 conn(0x7f06080fdeb0 msgr2=0x7f06081002d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:08:16.885 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:16.881+0000 7f06104fb700 1 -- 172.21.15.92:0/1348770643 shutdown_connections 2024-04-14T15:08:16.885 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:16.881+0000 7f06104fb700 1 -- 172.21.15.92:0/1348770643 wait complete. 2024-04-14T15:08:16.886 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:16.882+0000 7f06104fb700 1 librados: starting msgr at 2024-04-14T15:08:16.886 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:16.882+0000 7f06104fb700 1 librados: starting objecter 2024-04-14T15:08:16.886 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:16.882+0000 7f06104fb700 1 Processor -- start 2024-04-14T15:08:16.886 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:16.882+0000 7f06104fb700 1 -- start start 2024-04-14T15:08:16.886 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:16.882+0000 7f06104fb700 1 librados: setting wanted keys 2024-04-14T15:08:16.886 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:16.882+0000 7f06104fb700 1 librados: calling monclient init 2024-04-14T15:08:16.886 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:16.882+0000 7f06104fb700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f0608198b00 con 0x7f0608106d20 2024-04-14T15:08:16.886 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:16.882+0000 7f06104fb700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f06081a42f0 con 0x7f06081028f0 2024-04-14T15:08:16.886 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:16.882+0000 7f06104fb700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f06081a54d0 con 0x7f060810aa60 2024-04-14T15:08:16.886 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:16.882+0000 7f060e297700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f06081028f0 0x7f0608197fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:40616/0 (socket says 172.21.15.92:40616) 2024-04-14T15:08:16.886 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:16.882+0000 7f060e297700 1 -- 172.21.15.92:0/1632101521 learned_addr learned my addr 172.21.15.92:0/1632101521 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-04-14T15:08:16.886 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:16.882+0000 7f05ff7fe700 10 client.?.objecter ms_handle_connect 0x7f06081028f0 2024-04-14T15:08:16.886 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:16.882+0000 7f05ff7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:08:16.886 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:16.882+0000 7f05ff7fe700 10 client.?.objecter ms_handle_connect 0x7f0608106d20 2024-04-14T15:08:16.887 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:16.882+0000 7f05ff7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:08:16.887 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:16.882+0000 7f05ff7fe700 10 client.?.objecter ms_handle_connect 0x7f060810aa60 2024-04-14T15:08:16.887 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:16.882+0000 7f05ff7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:08:16.887 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:16.882+0000 7f05ff7fe700 1 -- 172.21.15.92:0/1632101521 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1172707452 0 0) 0x7f06081a42f0 con 0x7f06081028f0 2024-04-14T15:08:16.887 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:16.883+0000 7f05ff7fe700 1 -- 172.21.15.92:0/1632101521 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f05f40035b0 con 0x7f06081028f0 2024-04-14T15:08:16.887 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:16.883+0000 7f05ff7fe700 1 -- 172.21.15.92:0/1632101521 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2319275480 0 0) 0x7f06081a54d0 con 0x7f060810aa60 2024-04-14T15:08:16.887 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:16.883+0000 7f05ff7fe700 1 -- 172.21.15.92:0/1632101521 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f06081a42f0 con 0x7f060810aa60 2024-04-14T15:08:16.887 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:16.883+0000 7f05ff7fe700 1 -- 172.21.15.92:0/1632101521 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 265006792 0 0) 0x7f0608198b00 con 0x7f0608106d20 2024-04-14T15:08:16.887 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:16.883+0000 7f05ff7fe700 1 -- 172.21.15.92:0/1632101521 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f06081a54d0 con 0x7f0608106d20 2024-04-14T15:08:16.887 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:16.883+0000 7f05ff7fe700 1 -- 172.21.15.92:0/1632101521 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2746829236 0 0) 0x7f05f40035b0 con 0x7f06081028f0 2024-04-14T15:08:16.887 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:16.883+0000 7f05ff7fe700 1 -- 172.21.15.92:0/1632101521 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0608198b00 con 0x7f06081028f0 2024-04-14T15:08:16.887 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:16.883+0000 7f05ff7fe700 1 -- 172.21.15.92:0/1632101521 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f05f8001870 con 0x7f06081028f0 2024-04-14T15:08:16.887 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:16.883+0000 7f05ff7fe700 1 -- 172.21.15.92:0/1632101521 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 898572694 0 0) 0x7f06081a42f0 con 0x7f060810aa60 2024-04-14T15:08:16.888 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:16.883+0000 7f05ff7fe700 1 -- 172.21.15.92:0/1632101521 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f05f40035b0 con 0x7f060810aa60 2024-04-14T15:08:16.888 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:16.883+0000 7f05ff7fe700 1 -- 172.21.15.92:0/1632101521 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f0600001aa0 con 0x7f060810aa60 2024-04-14T15:08:16.888 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:16.883+0000 7f05ff7fe700 1 -- 172.21.15.92:0/1632101521 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4153743684 0 0) 0x7f06081a54d0 con 0x7f0608106d20 2024-04-14T15:08:16.888 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:16.883+0000 7f05ff7fe700 1 -- 172.21.15.92:0/1632101521 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f06081a42f0 con 0x7f0608106d20 2024-04-14T15:08:16.888 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:16.883+0000 7f05ff7fe700 1 -- 172.21.15.92:0/1632101521 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f06040015d0 con 0x7f0608106d20 2024-04-14T15:08:16.888 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:16.883+0000 7f05ff7fe700 1 -- 172.21.15.92:0/1632101521 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1898564206 0 0) 0x7f0608198b00 con 0x7f06081028f0 2024-04-14T15:08:16.888 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:16.883+0000 7f05ff7fe700 1 -- 172.21.15.92:0/1632101521 >> v1:172.21.15.90:6790/0 conn(0x7f060810aa60 legacy=0x7f06081a1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:08:16.888 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:16.883+0000 7f05ff7fe700 1 -- 172.21.15.92:0/1632101521 >> v1:172.21.15.90:6789/0 conn(0x7f0608106d20 legacy=0x7f060819e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:08:16.888 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:16.883+0000 7f05ff7fe700 1 -- 172.21.15.92:0/1632101521 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06081a66f0 con 0x7f06081028f0 2024-04-14T15:08:16.888 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:16.883+0000 7f06104fb700 1 -- 172.21.15.92:0/1632101521 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f06081a3340 con 0x7f06081028f0 2024-04-14T15:08:16.888 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:16.883+0000 7f06104fb700 10 client.24253.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:08:16.888 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:16.883+0000 7f06104fb700 1 -- 172.21.15.92:0/1632101521 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f06081a3880 con 0x7f06081028f0 2024-04-14T15:08:16.888 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:16.883+0000 7f06104fb700 1 librados: init done 2024-04-14T15:08:16.888 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:16.883+0000 7f05ff7fe700 1 -- 172.21.15.92:0/1632101521 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f05f8001c40 con 0x7f06081028f0 2024-04-14T15:08:16.889 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:16.883+0000 7f05ff7fe700 1 -- 172.21.15.92:0/1632101521 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f05f8005ca0 con 0x7f06081028f0 2024-04-14T15:08:16.889 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:16.884+0000 7f06104fb700 1 -- 172.21.15.92:0/1632101521 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05ec005120 con 0x7f06081028f0 2024-04-14T15:08:16.889 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:16.885+0000 7f05ff7fe700 1 -- 172.21.15.92:0/1632101521 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 15) ==== 92623+0+0 (unknown 2343906571 0 0) 0x7f05f801c680 con 0x7f06081028f0 2024-04-14T15:08:16.889 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:16.885+0000 7f05ff7fe700 1 -- 172.21.15.92:0/1632101521 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(35..35 src has 1..35) ==== 3205+0+0 (unknown 4153870733 0 0) 0x7f05f8089460 con 0x7f06081028f0 2024-04-14T15:08:16.889 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:16.885+0000 7f05ff7fe700 10 client.24253.objecter ms_dispatch 0x7f0608001550 osd_map(35..35 src has 1..35) 2024-04-14T15:08:16.889 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:16.885+0000 7f05ff7fe700 3 client.24253.objecter handle_osd_map got epochs [35,35] > 0 2024-04-14T15:08:16.889 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:16.885+0000 7f05ff7fe700 3 client.24253.objecter handle_osd_map decoding full epoch 35 2024-04-14T15:08:16.890 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:16.885+0000 7f05ff7fe700 20 client.24253.objecter dump_active .. 0 homeless 2024-04-14T15:08:16.890 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:16.886+0000 7f05ff7fe700 10 client.24253.objecter ms_handle_connect 0x7f05f4071280 2024-04-14T15:08:16.890 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:16.890+0000 7f05ff7fe700 1 -- 172.21.15.92:0/1632101521 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f05f8057230 con 0x7f06081028f0 2024-04-14T15:08:17.096 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:17.095+0000 7f06104fb700 1 -- 172.21.15.92:0/1632101521 --> v1:172.21.15.90:6800/491071118 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f05ec000b90 con 0x7f05f4071280 2024-04-14T15:08:17.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:17 smithi092 ceph-mon[33966]: osdmap e35: 5 total, 5 up, 5 in 2024-04-14T15:08:17.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:17 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T15:08:17.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:17 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T15:08:17.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:17 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:08:17.691 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:17 smithi090 ceph-mon[30399]: osdmap e35: 5 total, 5 up, 5 in 2024-04-14T15:08:17.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:17 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T15:08:17.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:17 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T15:08:17.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:17 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:08:17.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:17 smithi090 ceph-mon[41670]: osdmap e35: 5 total, 5 up, 5 in 2024-04-14T15:08:17.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:17 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T15:08:17.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:17 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T15:08:17.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:17 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:08:18.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:18 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:08:18.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:18 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:08:18.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:18 smithi092 ceph-mon[33966]: from='client.24253 v1:172.21.15.92:0/1632101521' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:08:18.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:18 smithi092 ceph-mon[33966]: osdmap e36: 5 total, 5 up, 5 in 2024-04-14T15:08:18.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:18 smithi092 ceph-mon[33966]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-14T15:08:18.691 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:18 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:08:18.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:18 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:08:18.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:18 smithi090 ceph-mon[30399]: from='client.24253 v1:172.21.15.92:0/1632101521' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:08:18.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:18 smithi090 ceph-mon[30399]: osdmap e36: 5 total, 5 up, 5 in 2024-04-14T15:08:18.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:18 smithi090 ceph-mon[30399]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-14T15:08:18.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:18 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:08:18.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:18 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:08:18.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:18 smithi090 ceph-mon[41670]: from='client.24253 v1:172.21.15.92:0/1632101521' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:08:18.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:18 smithi090 ceph-mon[41670]: osdmap e36: 5 total, 5 up, 5 in 2024-04-14T15:08:18.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:18 smithi090 ceph-mon[41670]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-14T15:08:20.107 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:19 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.92:0/1567714302' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1ec41805-6f82-4139-81bc-75305ba9e958"}]: dispatch 2024-04-14T15:08:20.107 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:19 smithi092 ceph-mon[33966]: from='client.24259 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1ec41805-6f82-4139-81bc-75305ba9e958"}]: dispatch 2024-04-14T15:08:20.107 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:19 smithi092 ceph-mon[33966]: from='client.24259 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1ec41805-6f82-4139-81bc-75305ba9e958"}]': finished 2024-04-14T15:08:20.107 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:19 smithi092 ceph-mon[33966]: osdmap e37: 6 total, 5 up, 6 in 2024-04-14T15:08:20.107 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:19 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T15:08:20.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:19 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.92:0/1567714302' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1ec41805-6f82-4139-81bc-75305ba9e958"}]: dispatch 2024-04-14T15:08:20.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:19 smithi090 ceph-mon[30399]: from='client.24259 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1ec41805-6f82-4139-81bc-75305ba9e958"}]: dispatch 2024-04-14T15:08:20.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:19 smithi090 ceph-mon[30399]: from='client.24259 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1ec41805-6f82-4139-81bc-75305ba9e958"}]': finished 2024-04-14T15:08:20.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:19 smithi090 ceph-mon[30399]: osdmap e37: 6 total, 5 up, 6 in 2024-04-14T15:08:20.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:19 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T15:08:20.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:19 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.92:0/1567714302' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1ec41805-6f82-4139-81bc-75305ba9e958"}]: dispatch 2024-04-14T15:08:20.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:19 smithi090 ceph-mon[41670]: from='client.24259 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1ec41805-6f82-4139-81bc-75305ba9e958"}]: dispatch 2024-04-14T15:08:20.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:19 smithi090 ceph-mon[41670]: from='client.24259 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1ec41805-6f82-4139-81bc-75305ba9e958"}]': finished 2024-04-14T15:08:20.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:19 smithi090 ceph-mon[41670]: osdmap e37: 6 total, 5 up, 6 in 2024-04-14T15:08:20.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:19 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T15:08:21.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:20 smithi092 ceph-mon[33966]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 124 KiB/s, 0 objects/s recovering 2024-04-14T15:08:21.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:20 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.92:0/2146309732' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T15:08:21.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:20 smithi090 ceph-mon[41670]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 124 KiB/s, 0 objects/s recovering 2024-04-14T15:08:21.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:20 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.92:0/2146309732' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T15:08:21.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:20 smithi090 ceph-mon[30399]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 124 KiB/s, 0 objects/s recovering 2024-04-14T15:08:21.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:20 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.92:0/2146309732' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T15:08:21.885 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:21.884+0000 7f060fcfa700 10 client.24253.objecter tick 2024-04-14T15:08:23.242 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:22 smithi092 ceph-mon[33966]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-14T15:08:23.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:22 smithi090 ceph-mon[30399]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-14T15:08:23.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:22 smithi090 ceph-mon[41670]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-14T15:08:25.101 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:24 smithi092 ceph-mon[33966]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-04-14T15:08:25.244 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:24 smithi090 ceph-mon[41670]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-04-14T15:08:25.244 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:24 smithi090 ceph-mon[30399]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-04-14T15:08:26.885 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:26.885+0000 7f060fcfa700 10 client.24253.objecter tick 2024-04-14T15:08:27.082 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:26 smithi092 ceph-mon[33966]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-14T15:08:27.132 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:26 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-14T15:08:27.132 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:26 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:08:27.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:26 smithi090 ceph-mon[30399]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-14T15:08:27.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:26 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-14T15:08:27.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:26 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:08:27.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:26 smithi090 ceph-mon[41670]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-14T15:08:27.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:26 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-14T15:08:27.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:26 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:08:27.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:27 smithi092 ceph-mon[33966]: Deploying daemon osd.5 on smithi092 2024-04-14T15:08:28.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:27 smithi090 ceph-mon[30399]: Deploying daemon osd.5 on smithi092 2024-04-14T15:08:28.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:27 smithi090 ceph-mon[41670]: Deploying daemon osd.5 on smithi092 2024-04-14T15:08:29.024 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:28 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:08:29.025 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:28 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:08:29.025 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:28 smithi092 ceph-mon[33966]: osdmap e38: 6 total, 5 up, 6 in 2024-04-14T15:08:29.025 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:28 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T15:08:29.025 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:28 smithi092 ceph-mon[33966]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-14T15:08:29.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:28 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:08:29.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:28 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:08:29.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:28 smithi090 ceph-mon[41670]: osdmap e38: 6 total, 5 up, 6 in 2024-04-14T15:08:29.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:28 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T15:08:29.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:28 smithi090 ceph-mon[41670]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-14T15:08:29.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:28 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:08:29.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:28 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:08:29.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:28 smithi090 ceph-mon[30399]: osdmap e38: 6 total, 5 up, 6 in 2024-04-14T15:08:29.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:28 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T15:08:29.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:28 smithi090 ceph-mon[30399]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-14T15:08:31.008 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:30 smithi092 ceph-mon[33966]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-14T15:08:31.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:30 smithi090 ceph-mon[30399]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-14T15:08:31.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:30 smithi090 ceph-mon[41670]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-14T15:08:31.887 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:31.886+0000 7f060fcfa700 10 client.24253.objecter tick 2024-04-14T15:08:32.829 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:32 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:32.829 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:32 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:32.829 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:32 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:08:32.829 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:32 smithi092 ceph-mon[33966]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-14T15:08:32.941 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:32 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:32.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:32 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:32.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:32 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:08:32.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:32 smithi090 ceph-mon[30399]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-14T15:08:32.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:32 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:32.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:32 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:32.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:32 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:08:32.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:32 smithi090 ceph-mon[41670]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-14T15:08:35.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:34 smithi092 ceph-mon[33966]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-14T15:08:35.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:34 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:35.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:34 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:35.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:34 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:08:35.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:34 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:08:35.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:34 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:35.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:34 smithi092 ceph-mon[33966]: from='osd.5 v1:172.21.15.92:6804/3244561440' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-14T15:08:35.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:34 smithi092 ceph-mon[33966]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-14T15:08:35.111 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:35.110+0000 7f05ff7fe700 1 -- 172.21.15.92:0/1632101521 <== mgr.14148 v1:172.21.15.90:6800/491071118 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (unknown 0 0 1364287159) 0x7f05ec000b90 con 0x7f05f4071280 2024-04-14T15:08:35.111 INFO:teuthology.orchestra.run.smithi092.stdout:Created osd(s) 5 on host 'smithi092' 2024-04-14T15:08:35.112 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:35.112+0000 7f06104fb700 10 librados: watch_flush enter 2024-04-14T15:08:35.112 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:35.112+0000 7f06104fb700 10 librados: watch_flush exit 2024-04-14T15:08:35.112 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:35.112+0000 7f06104fb700 20 client.24253.objecter shutdown clearing up homeless session... 2024-04-14T15:08:35.112 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:35.112+0000 7f06104fb700 10 client.24253.objecter successfully canceled tick 2024-04-14T15:08:35.112 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:35.112+0000 7f06104fb700 1 -- 172.21.15.92:0/1632101521 >> v1:172.21.15.90:6800/491071118 conn(0x7f05f4071280 legacy=0x7f05f4073740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:08:35.112 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:35.112+0000 7f06104fb700 1 -- 172.21.15.92:0/1632101521 >> v1:172.21.15.92:6789/0 conn(0x7f06081028f0 legacy=0x7f0608197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:08:35.112 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:35.112+0000 7f06104fb700 1 -- 172.21.15.92:0/1632101521 shutdown_connections 2024-04-14T15:08:35.112 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:35.112+0000 7f06104fb700 1 -- 172.21.15.92:0/1632101521 >> 172.21.15.92:0/1632101521 conn(0x7f06080fdeb0 msgr2=0x7f0608109170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:08:35.112 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:35.112+0000 7f06104fb700 1 -- 172.21.15.92:0/1632101521 shutdown_connections 2024-04-14T15:08:35.112 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:35.112+0000 7f06104fb700 1 -- 172.21.15.92:0/1632101521 wait complete. 2024-04-14T15:08:35.112 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:35.112+0000 7f06104fb700 1 librados: shutdown 2024-04-14T15:08:35.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:34 smithi090 ceph-mon[41670]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-14T15:08:35.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:34 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:35.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:34 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:35.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:34 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:08:35.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:34 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:08:35.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:34 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:35.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:34 smithi090 ceph-mon[41670]: from='osd.5 v1:172.21.15.92:6804/3244561440' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-14T15:08:35.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:34 smithi090 ceph-mon[41670]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-14T15:08:35.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:34 smithi090 ceph-mon[30399]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-14T15:08:35.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:34 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:35.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:34 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:35.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:34 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:08:35.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:34 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:08:35.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:34 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:35.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:34 smithi090 ceph-mon[30399]: from='osd.5 v1:172.21.15.92:6804/3244561440' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-14T15:08:35.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:34 smithi090 ceph-mon[30399]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-14T15:08:35.737 DEBUG:teuthology.orchestra.run.smithi092:osd.5> sudo journalctl -f -n 0 -u ceph-0e288098-fa70-11ee-bc8e-c7b262605968@osd.5.service 2024-04-14T15:08:35.739 INFO:tasks.cephadm:Deploying osd.6 on smithi092 with /dev/vg_nvme/lv_2... 2024-04-14T15:08:35.739 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-14T15:08:36.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:36 smithi092 ceph-mon[33966]: 1.0 scrub starts 2024-04-14T15:08:36.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:36 smithi092 ceph-mon[33966]: 1.0 scrub ok 2024-04-14T15:08:36.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:36 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:36.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:36 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:36.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:36 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:08:36.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:36 smithi092 ceph-mon[33966]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-14T15:08:36.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:36 smithi092 ceph-mon[33966]: osdmap e39: 6 total, 5 up, 6 in 2024-04-14T15:08:36.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:36 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T15:08:36.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:36 smithi092 ceph-mon[33966]: from='osd.5 v1:172.21.15.92:6804/3244561440' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-04-14T15:08:36.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:36 smithi092 ceph-mon[33966]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-04-14T15:08:36.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:36 smithi092 ceph-mon[33966]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-14T15:08:36.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:36 smithi090 ceph-mon[30399]: 1.0 scrub starts 2024-04-14T15:08:36.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:36 smithi090 ceph-mon[30399]: 1.0 scrub ok 2024-04-14T15:08:36.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:36 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:36.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:36 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:36.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:36 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:08:36.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:36 smithi090 ceph-mon[30399]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-14T15:08:36.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:36 smithi090 ceph-mon[30399]: osdmap e39: 6 total, 5 up, 6 in 2024-04-14T15:08:36.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:36 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T15:08:36.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:36 smithi090 ceph-mon[30399]: from='osd.5 v1:172.21.15.92:6804/3244561440' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-04-14T15:08:36.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:36 smithi090 ceph-mon[30399]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-04-14T15:08:36.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:36 smithi090 ceph-mon[30399]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-14T15:08:36.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:36 smithi090 ceph-mon[41670]: 1.0 scrub starts 2024-04-14T15:08:36.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:36 smithi090 ceph-mon[41670]: 1.0 scrub ok 2024-04-14T15:08:36.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:36 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:36.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:36 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:36.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:36 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:08:36.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:36 smithi090 ceph-mon[41670]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-14T15:08:36.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:36 smithi090 ceph-mon[41670]: osdmap e39: 6 total, 5 up, 6 in 2024-04-14T15:08:36.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:36 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T15:08:36.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:36 smithi090 ceph-mon[41670]: from='osd.5 v1:172.21.15.92:6804/3244561440' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-04-14T15:08:36.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:36 smithi090 ceph-mon[41670]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-04-14T15:08:36.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:36 smithi090 ceph-mon[41670]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-14T15:08:36.862 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:08:37.093 INFO:journalctl@ceph.osd.5.smithi092.stdout:Apr 14 15:08:36 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-5[45851]: 2024-04-14T15:08:36.755+0000 7f20f4a00700 -1 osd.5 0 waiting for initial osdmap 2024-04-14T15:08:37.094 INFO:journalctl@ceph.osd.5.smithi092.stdout:Apr 14 15:08:36 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-5[45851]: 2024-04-14T15:08:36.761+0000 7f20ec7eb700 -1 osd.5 40 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-14T15:08:37.345 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:37 smithi092 ceph-mon[33966]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2024-04-14T15:08:37.346 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:37 smithi092 ceph-mon[33966]: osdmap e40: 6 total, 5 up, 6 in 2024-04-14T15:08:37.346 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:37 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T15:08:37.346 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:37 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T15:08:37.346 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:37 smithi092 ceph-mon[33966]: from='osd.5 ' entity='osd.5' 2024-04-14T15:08:37.691 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:37 smithi090 ceph-mon[30399]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2024-04-14T15:08:37.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:37 smithi090 ceph-mon[30399]: osdmap e40: 6 total, 5 up, 6 in 2024-04-14T15:08:37.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:37 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T15:08:37.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:37 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T15:08:37.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:37 smithi090 ceph-mon[30399]: from='osd.5 ' entity='osd.5' 2024-04-14T15:08:37.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:37 smithi090 ceph-mon[41670]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2024-04-14T15:08:37.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:37 smithi090 ceph-mon[41670]: osdmap e40: 6 total, 5 up, 6 in 2024-04-14T15:08:37.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:37 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T15:08:37.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:37 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T15:08:37.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:37 smithi090 ceph-mon[41670]: from='osd.5 ' entity='osd.5' 2024-04-14T15:08:38.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:38 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:08:38.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:38 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:08:38.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:38 smithi092 ceph-mon[33966]: osd.5 v1:172.21.15.92:6804/3244561440 boot 2024-04-14T15:08:38.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:38 smithi092 ceph-mon[33966]: osdmap e41: 6 total, 6 up, 6 in 2024-04-14T15:08:38.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:38 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T15:08:38.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:38 smithi092 ceph-mon[33966]: pgmap v137: 1 pgs: 1 unknown; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-14T15:08:38.691 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:38 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:08:38.691 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:38 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:08:38.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:38 smithi090 ceph-mon[30399]: osd.5 v1:172.21.15.92:6804/3244561440 boot 2024-04-14T15:08:38.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:38 smithi090 ceph-mon[30399]: osdmap e41: 6 total, 6 up, 6 in 2024-04-14T15:08:38.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:38 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T15:08:38.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:38 smithi090 ceph-mon[30399]: pgmap v137: 1 pgs: 1 unknown; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-14T15:08:38.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:38 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:08:38.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:38 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:08:38.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:38 smithi090 ceph-mon[41670]: osd.5 v1:172.21.15.92:6804/3244561440 boot 2024-04-14T15:08:38.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:38 smithi090 ceph-mon[41670]: osdmap e41: 6 total, 6 up, 6 in 2024-04-14T15:08:38.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:38 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T15:08:38.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:38 smithi090 ceph-mon[41670]: pgmap v137: 1 pgs: 1 unknown; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-14T15:08:39.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:39 smithi092 ceph-mon[33966]: osdmap e42: 6 total, 6 up, 6 in 2024-04-14T15:08:39.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:39 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:39.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:39 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:39.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:39 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:08:39.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:39 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:08:39.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:39 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:39.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:39 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:08:39.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:39 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:08:39.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:39 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:39.691 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:39 smithi090 ceph-mon[30399]: osdmap e42: 6 total, 6 up, 6 in 2024-04-14T15:08:39.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:39 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:39.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:39 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:39.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:39 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:08:39.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:39 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:08:39.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:39 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:39.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:39 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:08:39.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:39 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:08:39.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:39 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:39.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:39 smithi090 ceph-mon[41670]: osdmap e42: 6 total, 6 up, 6 in 2024-04-14T15:08:39.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:39 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:39.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:39 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:39.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:39 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:08:39.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:39 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:08:39.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:39 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:39.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:39 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:08:39.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:39 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:08:39.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:39 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:39.804 INFO:teuthology.orchestra.run.smithi092.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-14T15:08:39.804 INFO:teuthology.orchestra.run.smithi092.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-14T15:08:39.804 INFO:teuthology.orchestra.run.smithi092.stderr: stderr: 10+0 records in 2024-04-14T15:08:39.804 INFO:teuthology.orchestra.run.smithi092.stderr:10+0 records out 2024-04-14T15:08:39.804 INFO:teuthology.orchestra.run.smithi092.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0127992 s, 819 MB/s 2024-04-14T15:08:39.805 INFO:teuthology.orchestra.run.smithi092.stderr:--> Zapping successful for: 2024-04-14T15:08:40.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:40 smithi092 ceph-mon[33966]: Detected new or changed devices on smithi092 2024-04-14T15:08:40.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:40 smithi092 ceph-mon[33966]: Adjusting osd_memory_target on smithi092 to 8544M 2024-04-14T15:08:40.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:40 smithi092 ceph-mon[33966]: osdmap e43: 6 total, 6 up, 6 in 2024-04-14T15:08:40.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:40 smithi092 ceph-mon[33966]: pgmap v140: 1 pgs: 1 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-14T15:08:40.442 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph orch daemon add osd smithi092:vg_nvme/lv_2 2024-04-14T15:08:40.680 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:08:40.691 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:40 smithi090 ceph-mon[30399]: Detected new or changed devices on smithi092 2024-04-14T15:08:40.691 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:40 smithi090 ceph-mon[30399]: Adjusting osd_memory_target on smithi092 to 8544M 2024-04-14T15:08:40.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:40 smithi090 ceph-mon[30399]: osdmap e43: 6 total, 6 up, 6 in 2024-04-14T15:08:40.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:40 smithi090 ceph-mon[30399]: pgmap v140: 1 pgs: 1 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-14T15:08:40.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:40 smithi090 ceph-mon[41670]: Detected new or changed devices on smithi092 2024-04-14T15:08:40.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:40 smithi090 ceph-mon[41670]: Adjusting osd_memory_target on smithi092 to 8544M 2024-04-14T15:08:40.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:40 smithi090 ceph-mon[41670]: osdmap e43: 6 total, 6 up, 6 in 2024-04-14T15:08:40.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:40 smithi090 ceph-mon[41670]: pgmap v140: 1 pgs: 1 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-14T15:08:41.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:41 smithi092 ceph-mon[33966]: osdmap e44: 6 total, 6 up, 6 in 2024-04-14T15:08:41.691 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:41 smithi090 ceph-mon[41670]: osdmap e44: 6 total, 6 up, 6 in 2024-04-14T15:08:41.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:41 smithi090 ceph-mon[30399]: osdmap e44: 6 total, 6 up, 6 in 2024-04-14T15:08:42.187 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:42.183+0000 7eff6515d700 1 -- 172.21.15.92:0/2752884796 >> v1:172.21.15.92:6789/0 conn(0x7eff60106d20 legacy=0x7eff60109170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:08:42.188 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:42.183+0000 7eff6515d700 1 -- 172.21.15.92:0/2752884796 shutdown_connections 2024-04-14T15:08:42.188 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:42.183+0000 7eff6515d700 1 -- 172.21.15.92:0/2752884796 >> 172.21.15.92:0/2752884796 conn(0x7eff600fdeb0 msgr2=0x7eff601002d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:08:42.188 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:42.183+0000 7eff6515d700 1 -- 172.21.15.92:0/2752884796 shutdown_connections 2024-04-14T15:08:42.188 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:42.183+0000 7eff6515d700 1 -- 172.21.15.92:0/2752884796 wait complete. 2024-04-14T15:08:42.188 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:42.184+0000 7eff6515d700 1 librados: starting msgr at 2024-04-14T15:08:42.188 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:42.184+0000 7eff6515d700 1 librados: starting objecter 2024-04-14T15:08:42.188 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:42.184+0000 7eff6515d700 1 Processor -- start 2024-04-14T15:08:42.188 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:42.184+0000 7eff6515d700 1 -- start start 2024-04-14T15:08:42.188 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:42.184+0000 7eff6515d700 1 librados: setting wanted keys 2024-04-14T15:08:42.188 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:42.184+0000 7eff6515d700 1 librados: calling monclient init 2024-04-14T15:08:42.188 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:42.184+0000 7eff6515d700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7eff60198b00 con 0x7eff6010aa60 2024-04-14T15:08:42.188 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:42.184+0000 7eff6515d700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7eff601a42f0 con 0x7eff601028f0 2024-04-14T15:08:42.188 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:42.184+0000 7eff6515d700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7eff601a54d0 con 0x7eff60106d20 2024-04-14T15:08:42.189 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:42.185+0000 7eff5ed9d700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7eff601028f0 0x7eff60197fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:57324/0 (socket says 172.21.15.92:57324) 2024-04-14T15:08:42.189 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:42.185+0000 7eff5ed9d700 1 -- 172.21.15.92:0/4253246752 learned_addr learned my addr 172.21.15.92:0/4253246752 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-04-14T15:08:42.189 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:42.185+0000 7eff47fff700 10 client.?.objecter ms_handle_connect 0x7eff601028f0 2024-04-14T15:08:42.189 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:42.185+0000 7eff47fff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:08:42.189 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:42.185+0000 7eff47fff700 10 client.?.objecter ms_handle_connect 0x7eff6010aa60 2024-04-14T15:08:42.189 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:42.185+0000 7eff47fff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:08:42.189 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:42.185+0000 7eff47fff700 10 client.?.objecter ms_handle_connect 0x7eff60106d20 2024-04-14T15:08:42.189 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:42.185+0000 7eff47fff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:08:42.189 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:42.185+0000 7eff47fff700 1 -- 172.21.15.92:0/4253246752 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 841567043 0 0) 0x7eff60198b00 con 0x7eff6010aa60 2024-04-14T15:08:42.189 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:42.185+0000 7eff47fff700 1 -- 172.21.15.92:0/4253246752 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff480035b0 con 0x7eff6010aa60 2024-04-14T15:08:42.189 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:42.185+0000 7eff47fff700 1 -- 172.21.15.92:0/4253246752 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2592161244 0 0) 0x7eff601a42f0 con 0x7eff601028f0 2024-04-14T15:08:42.189 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:42.185+0000 7eff47fff700 1 -- 172.21.15.92:0/4253246752 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff60198b00 con 0x7eff601028f0 2024-04-14T15:08:42.189 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:42.185+0000 7eff47fff700 1 -- 172.21.15.92:0/4253246752 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1749769105 0 0) 0x7eff601a54d0 con 0x7eff60106d20 2024-04-14T15:08:42.189 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:42.185+0000 7eff47fff700 1 -- 172.21.15.92:0/4253246752 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff601a42f0 con 0x7eff60106d20 2024-04-14T15:08:42.189 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:42.185+0000 7eff47fff700 1 -- 172.21.15.92:0/4253246752 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3157333783 0 0) 0x7eff60198b00 con 0x7eff601028f0 2024-04-14T15:08:42.190 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:42.185+0000 7eff47fff700 1 -- 172.21.15.92:0/4253246752 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7eff601a54d0 con 0x7eff601028f0 2024-04-14T15:08:42.190 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:42.185+0000 7eff47fff700 1 -- 172.21.15.92:0/4253246752 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7eff500018b0 con 0x7eff601028f0 2024-04-14T15:08:42.190 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:42.185+0000 7eff47fff700 1 -- 172.21.15.92:0/4253246752 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2711117215 0 0) 0x7eff480035b0 con 0x7eff6010aa60 2024-04-14T15:08:42.190 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:42.185+0000 7eff47fff700 1 -- 172.21.15.92:0/4253246752 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7eff60198b00 con 0x7eff6010aa60 2024-04-14T15:08:42.190 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:42.185+0000 7eff47fff700 1 -- 172.21.15.92:0/4253246752 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7eff58001a60 con 0x7eff6010aa60 2024-04-14T15:08:42.190 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:42.185+0000 7eff47fff700 1 -- 172.21.15.92:0/4253246752 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1507289442 0 0) 0x7eff601a42f0 con 0x7eff60106d20 2024-04-14T15:08:42.190 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:42.185+0000 7eff47fff700 1 -- 172.21.15.92:0/4253246752 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7eff480035b0 con 0x7eff60106d20 2024-04-14T15:08:42.190 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:42.185+0000 7eff47fff700 1 -- 172.21.15.92:0/4253246752 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7eff54001430 con 0x7eff60106d20 2024-04-14T15:08:42.190 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:42.185+0000 7eff47fff700 1 -- 172.21.15.92:0/4253246752 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2581377181 0 0) 0x7eff601a54d0 con 0x7eff601028f0 2024-04-14T15:08:42.190 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:42.185+0000 7eff47fff700 1 -- 172.21.15.92:0/4253246752 >> v1:172.21.15.90:6790/0 conn(0x7eff60106d20 legacy=0x7eff6019e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:08:42.190 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:42.185+0000 7eff47fff700 1 -- 172.21.15.92:0/4253246752 >> v1:172.21.15.90:6789/0 conn(0x7eff6010aa60 legacy=0x7eff601a1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:08:42.190 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:42.185+0000 7eff47fff700 1 -- 172.21.15.92:0/4253246752 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff601a66f0 con 0x7eff601028f0 2024-04-14T15:08:42.190 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:42.185+0000 7eff6515d700 1 -- 172.21.15.92:0/4253246752 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7eff601a3340 con 0x7eff601028f0 2024-04-14T15:08:42.191 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:42.185+0000 7eff6515d700 10 client.24280.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:08:42.191 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:42.185+0000 7eff6515d700 1 -- 172.21.15.92:0/4253246752 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7eff601a3880 con 0x7eff601028f0 2024-04-14T15:08:42.191 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:42.185+0000 7eff6515d700 1 librados: init done 2024-04-14T15:08:42.191 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:42.186+0000 7eff47fff700 1 -- 172.21.15.92:0/4253246752 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7eff50003b40 con 0x7eff601028f0 2024-04-14T15:08:42.191 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:42.186+0000 7eff47fff700 1 -- 172.21.15.92:0/4253246752 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7eff50005d10 con 0x7eff601028f0 2024-04-14T15:08:42.191 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:42.186+0000 7eff6515d700 1 -- 172.21.15.92:0/4253246752 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff4c005120 con 0x7eff601028f0 2024-04-14T15:08:42.191 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:42.187+0000 7eff47fff700 1 -- 172.21.15.92:0/4253246752 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 15) ==== 92623+0+0 (unknown 2343906571 0 0) 0x7eff5001c950 con 0x7eff601028f0 2024-04-14T15:08:42.191 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:42.187+0000 7eff47fff700 1 -- 172.21.15.92:0/4253246752 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(45..45 src has 1..45) ==== 3497+0+0 (unknown 3188684185 0 0) 0x7eff50089640 con 0x7eff601028f0 2024-04-14T15:08:42.192 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:42.187+0000 7eff47fff700 10 client.24280.objecter ms_dispatch 0x7eff60001550 osd_map(45..45 src has 1..45) 2024-04-14T15:08:42.192 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:42.187+0000 7eff47fff700 3 client.24280.objecter handle_osd_map got epochs [45,45] > 0 2024-04-14T15:08:42.192 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:42.187+0000 7eff47fff700 3 client.24280.objecter handle_osd_map decoding full epoch 45 2024-04-14T15:08:42.192 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:42.188+0000 7eff47fff700 20 client.24280.objecter dump_active .. 0 homeless 2024-04-14T15:08:42.192 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:42.188+0000 7eff47fff700 10 client.24280.objecter ms_handle_connect 0x7eff48071310 2024-04-14T15:08:42.192 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:42.192+0000 7eff47fff700 1 -- 172.21.15.92:0/4253246752 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7eff500572f0 con 0x7eff601028f0 2024-04-14T15:08:42.400 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:42.400+0000 7eff6515d700 1 -- 172.21.15.92:0/4253246752 --> v1:172.21.15.90:6800/491071118 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7eff4c000b90 con 0x7eff48071310 2024-04-14T15:08:42.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:42 smithi092 ceph-mon[33966]: osdmap e45: 6 total, 6 up, 6 in 2024-04-14T15:08:42.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:42 smithi092 ceph-mon[33966]: pgmap v143: 1 pgs: 1 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-14T15:08:42.691 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:42 smithi090 ceph-mon[30399]: osdmap e45: 6 total, 6 up, 6 in 2024-04-14T15:08:42.691 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:42 smithi090 ceph-mon[30399]: pgmap v143: 1 pgs: 1 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-14T15:08:42.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:42 smithi090 ceph-mon[41670]: osdmap e45: 6 total, 6 up, 6 in 2024-04-14T15:08:42.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:42 smithi090 ceph-mon[41670]: pgmap v143: 1 pgs: 1 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-14T15:08:43.625 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:43 smithi092 ceph-mon[33966]: from='client.24280 v1:172.21.15.92:0/4253246752' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:08:43.625 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:43 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T15:08:43.625 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:43 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T15:08:43.625 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:43 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:08:43.691 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:43 smithi090 ceph-mon[30399]: from='client.24280 v1:172.21.15.92:0/4253246752' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:08:43.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:43 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T15:08:43.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:43 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T15:08:43.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:43 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:08:43.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:43 smithi090 ceph-mon[41670]: from='client.24280 v1:172.21.15.92:0/4253246752' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:08:43.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:43 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T15:08:43.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:43 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T15:08:43.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:43 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:08:44.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:44 smithi092 ceph-mon[33966]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-04-14T15:08:44.691 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:44 smithi090 ceph-mon[30399]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-04-14T15:08:44.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:44 smithi090 ceph-mon[41670]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-04-14T15:08:45.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:45 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.92:0/1801099829' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ca977ab2-bf8d-4b3a-ae5e-0efa5bc7a3ad"}]: dispatch 2024-04-14T15:08:45.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:45 smithi092 ceph-mon[33966]: from='client.24286 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ca977ab2-bf8d-4b3a-ae5e-0efa5bc7a3ad"}]: dispatch 2024-04-14T15:08:45.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:45 smithi092 ceph-mon[33966]: from='client.24286 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ca977ab2-bf8d-4b3a-ae5e-0efa5bc7a3ad"}]': finished 2024-04-14T15:08:45.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:45 smithi092 ceph-mon[33966]: osdmap e46: 7 total, 6 up, 7 in 2024-04-14T15:08:45.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:45 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T15:08:45.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:45 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.92:0/1801099829' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ca977ab2-bf8d-4b3a-ae5e-0efa5bc7a3ad"}]: dispatch 2024-04-14T15:08:45.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:45 smithi090 ceph-mon[41670]: from='client.24286 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ca977ab2-bf8d-4b3a-ae5e-0efa5bc7a3ad"}]: dispatch 2024-04-14T15:08:45.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:45 smithi090 ceph-mon[41670]: from='client.24286 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ca977ab2-bf8d-4b3a-ae5e-0efa5bc7a3ad"}]': finished 2024-04-14T15:08:45.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:45 smithi090 ceph-mon[41670]: osdmap e46: 7 total, 6 up, 7 in 2024-04-14T15:08:45.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:45 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T15:08:45.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:45 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.92:0/1801099829' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ca977ab2-bf8d-4b3a-ae5e-0efa5bc7a3ad"}]: dispatch 2024-04-14T15:08:45.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:45 smithi090 ceph-mon[30399]: from='client.24286 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ca977ab2-bf8d-4b3a-ae5e-0efa5bc7a3ad"}]: dispatch 2024-04-14T15:08:45.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:45 smithi090 ceph-mon[30399]: from='client.24286 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ca977ab2-bf8d-4b3a-ae5e-0efa5bc7a3ad"}]': finished 2024-04-14T15:08:45.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:45 smithi090 ceph-mon[30399]: osdmap e46: 7 total, 6 up, 7 in 2024-04-14T15:08:45.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:45 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T15:08:46.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:46 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.92:0/3038716376' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T15:08:46.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:46 smithi092 ceph-mon[33966]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-14T15:08:46.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:46 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.92:0/3038716376' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T15:08:46.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:46 smithi090 ceph-mon[30399]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-14T15:08:46.694 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:46 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.92:0/3038716376' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T15:08:46.694 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:46 smithi090 ceph-mon[41670]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-14T15:08:47.186 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:47.186+0000 7eff5ffff700 10 client.24280.objecter tick 2024-04-14T15:08:49.277 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:48 smithi092 ceph-mon[33966]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-04-14T15:08:49.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:48 smithi090 ceph-mon[30399]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-04-14T15:08:49.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:48 smithi090 ceph-mon[41670]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-04-14T15:08:51.296 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:50 smithi092 ceph-mon[33966]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-04-14T15:08:51.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:50 smithi090 ceph-mon[30399]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-04-14T15:08:51.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:50 smithi090 ceph-mon[41670]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-04-14T15:08:52.188 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:52.187+0000 7eff5ffff700 10 client.24280.objecter tick 2024-04-14T15:08:52.286 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:51 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-14T15:08:52.286 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:51 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:08:52.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:51 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-14T15:08:52.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:51 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:08:52.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:51 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-14T15:08:52.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:51 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:08:53.253 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:52 smithi092 ceph-mon[33966]: Deploying daemon osd.6 on smithi092 2024-04-14T15:08:53.254 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:52 smithi092 ceph-mon[33966]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-14T15:08:53.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:52 smithi090 ceph-mon[30399]: Deploying daemon osd.6 on smithi092 2024-04-14T15:08:53.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:52 smithi090 ceph-mon[30399]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-14T15:08:53.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:52 smithi090 ceph-mon[41670]: Deploying daemon osd.6 on smithi092 2024-04-14T15:08:53.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:52 smithi090 ceph-mon[41670]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-14T15:08:55.244 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:54 smithi090 ceph-mon[41670]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-14T15:08:55.244 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:54 smithi090 ceph-mon[30399]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-14T15:08:55.321 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:54 smithi092 ceph-mon[33966]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-14T15:08:57.190 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:08:57.188+0000 7eff5ffff700 10 client.24280.objecter tick 2024-04-14T15:08:57.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:56 smithi092 ceph-mon[33966]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-14T15:08:57.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:56 smithi090 ceph-mon[41670]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-14T15:08:57.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:56 smithi090 ceph-mon[30399]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-14T15:08:58.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:58 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:58.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:58 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:58.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:58 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:08:58.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:08:58 smithi092 ceph-mon[33966]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-14T15:08:58.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:58 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:58.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:58 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:58.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:58 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:08:58.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:08:58 smithi090 ceph-mon[30399]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-14T15:08:58.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:58 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:58.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:58 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:08:58.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:58 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:08:58.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:08:58 smithi090 ceph-mon[41670]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-14T15:09:01.245 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:00 smithi092 ceph-mon[33966]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-14T15:09:01.246 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:00 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:01.246 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:00 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:01.246 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:00 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:09:01.246 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:00 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:09:01.246 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:00 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:01.246 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:00 smithi092 ceph-mon[33966]: from='osd.6 v1:172.21.15.92:6808/636125963' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-14T15:09:01.246 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:00 smithi092 ceph-mon[33966]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-14T15:09:01.315 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:01.314+0000 7eff47fff700 1 -- 172.21.15.92:0/4253246752 <== mgr.14148 v1:172.21.15.90:6800/491071118 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (unknown 0 0 3043978603) 0x7eff4c000b90 con 0x7eff48071310 2024-04-14T15:09:01.315 INFO:teuthology.orchestra.run.smithi092.stdout:Created osd(s) 6 on host 'smithi092' 2024-04-14T15:09:01.318 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:01.317+0000 7eff6515d700 10 librados: watch_flush enter 2024-04-14T15:09:01.318 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:01.317+0000 7eff6515d700 10 librados: watch_flush exit 2024-04-14T15:09:01.318 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:01.317+0000 7eff6515d700 20 client.24280.objecter shutdown clearing up homeless session... 2024-04-14T15:09:01.318 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:01.317+0000 7eff6515d700 10 client.24280.objecter successfully canceled tick 2024-04-14T15:09:01.318 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:01.317+0000 7eff6515d700 1 -- 172.21.15.92:0/4253246752 >> v1:172.21.15.90:6800/491071118 conn(0x7eff48071310 legacy=0x7eff480737d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:09:01.318 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:01.317+0000 7eff6515d700 1 -- 172.21.15.92:0/4253246752 >> v1:172.21.15.92:6789/0 conn(0x7eff601028f0 legacy=0x7eff60197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:09:01.318 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:01.317+0000 7eff6515d700 1 -- 172.21.15.92:0/4253246752 shutdown_connections 2024-04-14T15:09:01.318 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:01.317+0000 7eff6515d700 1 -- 172.21.15.92:0/4253246752 >> 172.21.15.92:0/4253246752 conn(0x7eff600fdeb0 msgr2=0x7eff60109170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:09:01.318 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:01.317+0000 7eff6515d700 1 -- 172.21.15.92:0/4253246752 shutdown_connections 2024-04-14T15:09:01.318 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:01.317+0000 7eff6515d700 1 -- 172.21.15.92:0/4253246752 wait complete. 2024-04-14T15:09:01.318 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:01.317+0000 7eff6515d700 1 librados: shutdown 2024-04-14T15:09:01.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:00 smithi090 ceph-mon[30399]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-14T15:09:01.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:00 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:01.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:00 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:01.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:00 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:09:01.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:00 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:09:01.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:00 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:01.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:00 smithi090 ceph-mon[30399]: from='osd.6 v1:172.21.15.92:6808/636125963' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-14T15:09:01.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:00 smithi090 ceph-mon[30399]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-14T15:09:01.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:00 smithi090 ceph-mon[41670]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-14T15:09:01.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:00 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:01.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:00 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:01.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:00 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:09:01.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:00 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:09:01.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:00 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:01.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:00 smithi090 ceph-mon[41670]: from='osd.6 v1:172.21.15.92:6808/636125963' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-14T15:09:01.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:00 smithi090 ceph-mon[41670]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-14T15:09:01.923 DEBUG:teuthology.orchestra.run.smithi092:osd.6> sudo journalctl -f -n 0 -u ceph-0e288098-fa70-11ee-bc8e-c7b262605968@osd.6.service 2024-04-14T15:09:01.925 INFO:tasks.cephadm:Deploying osd.7 on smithi092 with /dev/vg_nvme/lv_1... 2024-04-14T15:09:01.926 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-14T15:09:02.234 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:02 smithi092 ceph-mon[33966]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-14T15:09:02.234 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:02 smithi092 ceph-mon[33966]: osdmap e47: 7 total, 6 up, 7 in 2024-04-14T15:09:02.234 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:02 smithi092 ceph-mon[33966]: from='osd.6 v1:172.21.15.92:6808/636125963' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-04-14T15:09:02.234 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:02 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T15:09:02.234 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:02 smithi092 ceph-mon[33966]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-04-14T15:09:02.234 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:02 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:02.234 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:02 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:02.234 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:02 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:09:02.234 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:02 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:09:02.234 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:02 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:09:02.234 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:02 smithi092 ceph-mon[33966]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-14T15:09:02.485 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:09:02 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6[52258]: 2024-04-14T15:09:02.237+0000 7fb5de2f4700 -1 osd.6 0 waiting for initial osdmap 2024-04-14T15:09:02.485 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:09:02 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6[52258]: 2024-04-14T15:09:02.243+0000 7fb5da0e7700 -1 osd.6 48 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-14T15:09:02.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:02 smithi090 ceph-mon[30399]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-14T15:09:02.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:02 smithi090 ceph-mon[30399]: osdmap e47: 7 total, 6 up, 7 in 2024-04-14T15:09:02.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:02 smithi090 ceph-mon[30399]: from='osd.6 v1:172.21.15.92:6808/636125963' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-04-14T15:09:02.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:02 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T15:09:02.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:02 smithi090 ceph-mon[30399]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-04-14T15:09:02.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:02 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:02.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:02 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:02.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:02 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:09:02.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:02 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:09:02.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:02 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:09:02.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:02 smithi090 ceph-mon[30399]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-14T15:09:02.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:02 smithi090 ceph-mon[41670]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-14T15:09:02.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:02 smithi090 ceph-mon[41670]: osdmap e47: 7 total, 6 up, 7 in 2024-04-14T15:09:02.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:02 smithi090 ceph-mon[41670]: from='osd.6 v1:172.21.15.92:6808/636125963' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-04-14T15:09:02.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:02 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T15:09:02.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:02 smithi090 ceph-mon[41670]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-04-14T15:09:02.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:02 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:02.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:02 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:02.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:02 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:09:02.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:02 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:09:02.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:02 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:09:02.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:02 smithi090 ceph-mon[41670]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-14T15:09:02.694 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:09:03.503 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:03 smithi092 ceph-mon[33966]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2024-04-14T15:09:03.503 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:03 smithi092 ceph-mon[33966]: osdmap e48: 7 total, 6 up, 7 in 2024-04-14T15:09:03.503 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:03 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T15:09:03.503 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:03 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T15:09:03.691 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:03 smithi090 ceph-mon[30399]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2024-04-14T15:09:03.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:03 smithi090 ceph-mon[30399]: osdmap e48: 7 total, 6 up, 7 in 2024-04-14T15:09:03.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:03 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T15:09:03.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:03 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T15:09:03.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:03 smithi090 ceph-mon[41670]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2024-04-14T15:09:03.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:03 smithi090 ceph-mon[41670]: osdmap e48: 7 total, 6 up, 7 in 2024-04-14T15:09:03.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:03 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T15:09:03.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:03 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T15:09:04.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:04 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:09:04.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:04 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:09:04.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:04 smithi092 ceph-mon[33966]: osd.6 v1:172.21.15.92:6808/636125963 boot 2024-04-14T15:09:04.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:04 smithi092 ceph-mon[33966]: osdmap e49: 7 total, 7 up, 7 in 2024-04-14T15:09:04.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:04 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T15:09:04.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:04 smithi092 ceph-mon[33966]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-14T15:09:04.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:04 smithi092 ceph-mon[33966]: osdmap e50: 7 total, 7 up, 7 in 2024-04-14T15:09:04.691 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:04 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:09:04.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:04 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:09:04.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:04 smithi090 ceph-mon[30399]: osd.6 v1:172.21.15.92:6808/636125963 boot 2024-04-14T15:09:04.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:04 smithi090 ceph-mon[30399]: osdmap e49: 7 total, 7 up, 7 in 2024-04-14T15:09:04.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:04 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T15:09:04.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:04 smithi090 ceph-mon[30399]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-14T15:09:04.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:04 smithi090 ceph-mon[30399]: osdmap e50: 7 total, 7 up, 7 in 2024-04-14T15:09:04.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:04 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:09:04.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:04 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:09:04.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:04 smithi090 ceph-mon[41670]: osd.6 v1:172.21.15.92:6808/636125963 boot 2024-04-14T15:09:04.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:04 smithi090 ceph-mon[41670]: osdmap e49: 7 total, 7 up, 7 in 2024-04-14T15:09:04.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:04 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T15:09:04.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:04 smithi090 ceph-mon[41670]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-14T15:09:04.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:04 smithi090 ceph-mon[41670]: osdmap e50: 7 total, 7 up, 7 in 2024-04-14T15:09:05.571 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:05 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:09:05.572 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:05 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:09:05.572 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:05 smithi092 ceph-mon[33966]: osdmap e51: 7 total, 7 up, 7 in 2024-04-14T15:09:05.691 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:05 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:09:05.691 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:05 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:09:05.691 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:05 smithi090 ceph-mon[30399]: osdmap e51: 7 total, 7 up, 7 in 2024-04-14T15:09:05.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:05 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:09:05.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:05 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:09:05.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:05 smithi090 ceph-mon[41670]: osdmap e51: 7 total, 7 up, 7 in 2024-04-14T15:09:06.160 INFO:teuthology.orchestra.run.smithi092.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-14T15:09:06.161 INFO:teuthology.orchestra.run.smithi092.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-14T15:09:06.161 INFO:teuthology.orchestra.run.smithi092.stderr: stderr: 10+0 records in 2024-04-14T15:09:06.161 INFO:teuthology.orchestra.run.smithi092.stderr:10+0 records out 2024-04-14T15:09:06.161 INFO:teuthology.orchestra.run.smithi092.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0131816 s, 795 MB/s 2024-04-14T15:09:06.161 INFO:teuthology.orchestra.run.smithi092.stderr:--> Zapping successful for: 2024-04-14T15:09:06.767 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:06 smithi092 ceph-mon[33966]: Detected new or changed devices on smithi092 2024-04-14T15:09:06.767 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:06 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:06.767 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:06 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:06.767 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:06 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:09:06.767 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:06 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:09:06.767 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:06 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:09:06.767 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:06 smithi092 ceph-mon[33966]: Adjusting osd_memory_target on smithi092 to 5696M 2024-04-14T15:09:06.767 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:06 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:06.768 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:06 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:09:06.768 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:06 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:09:06.768 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:06 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:06.768 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:06 smithi092 ceph-mon[33966]: osdmap e52: 7 total, 7 up, 7 in 2024-04-14T15:09:06.768 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:06 smithi092 ceph-mon[33966]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-14T15:09:06.794 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph orch daemon add osd smithi092:vg_nvme/lv_1 2024-04-14T15:09:06.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:06 smithi090 ceph-mon[30399]: Detected new or changed devices on smithi092 2024-04-14T15:09:06.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:06 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:06.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:06 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:06.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:06 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:09:06.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:06 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:09:06.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:06 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:09:06.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:06 smithi090 ceph-mon[30399]: Adjusting osd_memory_target on smithi092 to 5696M 2024-04-14T15:09:06.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:06 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:06.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:06 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:09:06.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:06 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:09:06.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:06 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:06.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:06 smithi090 ceph-mon[30399]: osdmap e52: 7 total, 7 up, 7 in 2024-04-14T15:09:06.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:06 smithi090 ceph-mon[30399]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-14T15:09:06.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:06 smithi090 ceph-mon[41670]: Detected new or changed devices on smithi092 2024-04-14T15:09:06.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:06 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:06.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:06 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:06.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:06 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:09:06.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:06 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:09:06.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:06 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:09:06.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:06 smithi090 ceph-mon[41670]: Adjusting osd_memory_target on smithi092 to 5696M 2024-04-14T15:09:06.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:06 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:06.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:06 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:09:06.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:06 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:09:06.944 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:06 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:06.944 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:06 smithi090 ceph-mon[41670]: osdmap e52: 7 total, 7 up, 7 in 2024-04-14T15:09:06.944 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:06 smithi090 ceph-mon[41670]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-14T15:09:07.029 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:09:08.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:07 smithi092 ceph-mon[33966]: osdmap e53: 7 total, 7 up, 7 in 2024-04-14T15:09:08.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:07 smithi090 ceph-mon[30399]: osdmap e53: 7 total, 7 up, 7 in 2024-04-14T15:09:08.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:07 smithi090 ceph-mon[41670]: osdmap e53: 7 total, 7 up, 7 in 2024-04-14T15:09:08.760 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:08.756+0000 7efe6e6b3700 1 -- 172.21.15.92:0/1116037941 >> v1:172.21.15.92:6789/0 conn(0x7efe68109950 legacy=0x7efe6810be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:09:08.760 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:08.756+0000 7efe6e6b3700 1 -- 172.21.15.92:0/1116037941 shutdown_connections 2024-04-14T15:09:08.760 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:08.756+0000 7efe6e6b3700 1 -- 172.21.15.92:0/1116037941 >> 172.21.15.92:0/1116037941 conn(0x7efe680fcd80 msgr2=0x7efe680ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:09:08.760 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:08.756+0000 7efe6e6b3700 1 -- 172.21.15.92:0/1116037941 shutdown_connections 2024-04-14T15:09:08.760 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:08.756+0000 7efe6e6b3700 1 -- 172.21.15.92:0/1116037941 wait complete. 2024-04-14T15:09:08.760 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:08.757+0000 7efe6e6b3700 1 librados: starting msgr at 2024-04-14T15:09:08.760 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:08.757+0000 7efe6e6b3700 1 librados: starting objecter 2024-04-14T15:09:08.760 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:08.757+0000 7efe6e6b3700 1 Processor -- start 2024-04-14T15:09:08.760 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:08.757+0000 7efe6e6b3700 1 -- start start 2024-04-14T15:09:08.761 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:08.757+0000 7efe6e6b3700 1 librados: setting wanted keys 2024-04-14T15:09:08.761 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:08.757+0000 7efe6e6b3700 1 librados: calling monclient init 2024-04-14T15:09:08.761 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:08.757+0000 7efe6e6b3700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7efe68197a10 con 0x7efe681017e0 2024-04-14T15:09:08.761 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:08.757+0000 7efe6e6b3700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7efe681a3200 con 0x7efe68109950 2024-04-14T15:09:08.761 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:08.757+0000 7efe6e6b3700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7efe681a43e0 con 0x7efe68105c10 2024-04-14T15:09:08.761 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:08.757+0000 7efe6cc50700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7efe68109950 0x7efe681a0ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:48302/0 (socket says 172.21.15.92:48302) 2024-04-14T15:09:08.761 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:08.757+0000 7efe6cc50700 1 -- 172.21.15.92:0/2557091639 learned_addr learned my addr 172.21.15.92:0/2557091639 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-04-14T15:09:08.761 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:08.757+0000 7efe657fa700 10 client.?.objecter ms_handle_connect 0x7efe68109950 2024-04-14T15:09:08.761 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:08.757+0000 7efe657fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:09:08.761 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:08.757+0000 7efe657fa700 10 client.?.objecter ms_handle_connect 0x7efe681017e0 2024-04-14T15:09:08.761 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:08.757+0000 7efe657fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:09:08.761 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:08.757+0000 7efe657fa700 10 client.?.objecter ms_handle_connect 0x7efe68105c10 2024-04-14T15:09:08.761 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:08.757+0000 7efe657fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:09:08.762 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:08.757+0000 7efe657fa700 1 -- 172.21.15.92:0/2557091639 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4162551282 0 0) 0x7efe681a3200 con 0x7efe68109950 2024-04-14T15:09:08.762 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:08.757+0000 7efe657fa700 1 -- 172.21.15.92:0/2557091639 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efe500031f0 con 0x7efe68109950 2024-04-14T15:09:08.762 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:08.757+0000 7efe657fa700 1 -- 172.21.15.92:0/2557091639 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1143889375 0 0) 0x7efe68197a10 con 0x7efe681017e0 2024-04-14T15:09:08.762 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:08.757+0000 7efe657fa700 1 -- 172.21.15.92:0/2557091639 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efe681a3200 con 0x7efe681017e0 2024-04-14T15:09:08.762 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:08.758+0000 7efe657fa700 1 -- 172.21.15.92:0/2557091639 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1688573414 0 0) 0x7efe681a43e0 con 0x7efe68105c10 2024-04-14T15:09:08.762 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:08.758+0000 7efe657fa700 1 -- 172.21.15.92:0/2557091639 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efe68197a10 con 0x7efe68105c10 2024-04-14T15:09:08.762 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:08.758+0000 7efe657fa700 1 -- 172.21.15.92:0/2557091639 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4003512321 0 0) 0x7efe500031f0 con 0x7efe68109950 2024-04-14T15:09:08.762 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:08.758+0000 7efe657fa700 1 -- 172.21.15.92:0/2557091639 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efe681a43e0 con 0x7efe68109950 2024-04-14T15:09:08.762 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:08.758+0000 7efe657fa700 1 -- 172.21.15.92:0/2557091639 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7efe600017b0 con 0x7efe68109950 2024-04-14T15:09:08.762 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:08.758+0000 7efe657fa700 1 -- 172.21.15.92:0/2557091639 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3877546321 0 0) 0x7efe681a3200 con 0x7efe681017e0 2024-04-14T15:09:08.762 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:08.758+0000 7efe657fa700 1 -- 172.21.15.92:0/2557091639 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efe500031f0 con 0x7efe681017e0 2024-04-14T15:09:08.762 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:08.758+0000 7efe657fa700 1 -- 172.21.15.92:0/2557091639 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7efe58001830 con 0x7efe681017e0 2024-04-14T15:09:08.763 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:08.758+0000 7efe657fa700 1 -- 172.21.15.92:0/2557091639 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2929819596 0 0) 0x7efe68197a10 con 0x7efe68105c10 2024-04-14T15:09:08.763 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:08.758+0000 7efe657fa700 1 -- 172.21.15.92:0/2557091639 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efe681a3200 con 0x7efe68105c10 2024-04-14T15:09:08.763 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:08.758+0000 7efe657fa700 1 -- 172.21.15.92:0/2557091639 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7efe5c001870 con 0x7efe68105c10 2024-04-14T15:09:08.763 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:08.758+0000 7efe657fa700 1 -- 172.21.15.92:0/2557091639 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 74191848 0 0) 0x7efe681a43e0 con 0x7efe68109950 2024-04-14T15:09:08.763 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:08.758+0000 7efe657fa700 1 -- 172.21.15.92:0/2557091639 >> v1:172.21.15.90:6790/0 conn(0x7efe68105c10 legacy=0x7efe6819d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:09:08.763 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:08.758+0000 7efe657fa700 1 -- 172.21.15.92:0/2557091639 >> v1:172.21.15.90:6789/0 conn(0x7efe681017e0 legacy=0x7efe68196ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:09:08.763 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:08.758+0000 7efe657fa700 1 -- 172.21.15.92:0/2557091639 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe681a5600 con 0x7efe68109950 2024-04-14T15:09:08.763 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:08.758+0000 7efe6e6b3700 1 -- 172.21.15.92:0/2557091639 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7efe681a2250 con 0x7efe68109950 2024-04-14T15:09:08.763 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:08.758+0000 7efe6e6b3700 10 client.24307.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:09:08.763 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:08.758+0000 7efe6e6b3700 1 -- 172.21.15.92:0/2557091639 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7efe681a2790 con 0x7efe68109950 2024-04-14T15:09:08.763 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:08.758+0000 7efe6e6b3700 1 librados: init done 2024-04-14T15:09:08.763 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:08.758+0000 7efe657fa700 1 -- 172.21.15.92:0/2557091639 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7efe60001a10 con 0x7efe68109950 2024-04-14T15:09:08.763 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:08.758+0000 7efe657fa700 1 -- 172.21.15.92:0/2557091639 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7efe600052a0 con 0x7efe68109950 2024-04-14T15:09:08.763 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:08.759+0000 7efe6e6b3700 1 -- 172.21.15.92:0/2557091639 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe54005150 con 0x7efe68109950 2024-04-14T15:09:08.764 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:08.759+0000 7efe657fa700 1 -- 172.21.15.92:0/2557091639 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 15) ==== 92623+0+0 (unknown 2343906571 0 0) 0x7efe600054e0 con 0x7efe68109950 2024-04-14T15:09:08.764 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:08.760+0000 7efe657fa700 1 -- 172.21.15.92:0/2557091639 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(53..53 src has 1..53) ==== 3789+0+0 (unknown 2094627349 0 0) 0x7efe60089b40 con 0x7efe68109950 2024-04-14T15:09:08.764 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:08.760+0000 7efe657fa700 10 client.24307.objecter ms_dispatch 0x7efe68001550 osd_map(53..53 src has 1..53) 2024-04-14T15:09:08.764 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:08.760+0000 7efe657fa700 3 client.24307.objecter handle_osd_map got epochs [53,53] > 0 2024-04-14T15:09:08.764 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:08.760+0000 7efe657fa700 3 client.24307.objecter handle_osd_map decoding full epoch 53 2024-04-14T15:09:08.764 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:08.760+0000 7efe657fa700 20 client.24307.objecter dump_active .. 0 homeless 2024-04-14T15:09:08.764 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:08.760+0000 7efe657fa700 10 client.24307.objecter ms_handle_connect 0x7efe500712c0 2024-04-14T15:09:08.764 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:08.765+0000 7efe657fa700 1 -- 172.21.15.92:0/2557091639 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7efe600577b0 con 0x7efe68109950 2024-04-14T15:09:08.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:08 smithi092 ceph-mon[33966]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-14T15:09:08.976 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:08.976+0000 7efe6e6b3700 1 -- 172.21.15.92:0/2557091639 --> v1:172.21.15.90:6800/491071118 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7efe54000bc0 con 0x7efe500712c0 2024-04-14T15:09:09.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:08 smithi090 ceph-mon[30399]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-14T15:09:09.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:08 smithi090 ceph-mon[41670]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-14T15:09:09.865 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:09 smithi092 ceph-mon[33966]: from='client.24307 v1:172.21.15.92:0/2557091639' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:09:09.865 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:09 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T15:09:09.865 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:09 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T15:09:09.865 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:09 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:09:10.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:09 smithi090 ceph-mon[30399]: from='client.24307 v1:172.21.15.92:0/2557091639' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:09:10.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:09 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T15:09:10.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:09 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T15:09:10.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:09 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:09:10.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:09 smithi090 ceph-mon[41670]: from='client.24307 v1:172.21.15.92:0/2557091639' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:09:10.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:09 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T15:09:10.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:09 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T15:09:10.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:09 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:09:10.920 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:10 smithi092 ceph-mon[33966]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-04-14T15:09:11.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:10 smithi090 ceph-mon[30399]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-04-14T15:09:11.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:10 smithi090 ceph-mon[41670]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-04-14T15:09:12.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:11 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.92:0/1304297095' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e5354645-79ec-4d4c-b6e5-f9b0653915ee"}]: dispatch 2024-04-14T15:09:12.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:11 smithi092 ceph-mon[33966]: from='client.24313 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e5354645-79ec-4d4c-b6e5-f9b0653915ee"}]: dispatch 2024-04-14T15:09:12.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:11 smithi092 ceph-mon[33966]: from='client.24313 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e5354645-79ec-4d4c-b6e5-f9b0653915ee"}]': finished 2024-04-14T15:09:12.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:11 smithi092 ceph-mon[33966]: osdmap e54: 8 total, 7 up, 8 in 2024-04-14T15:09:12.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:11 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T15:09:12.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:11 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.92:0/1304297095' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e5354645-79ec-4d4c-b6e5-f9b0653915ee"}]: dispatch 2024-04-14T15:09:12.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:11 smithi090 ceph-mon[30399]: from='client.24313 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e5354645-79ec-4d4c-b6e5-f9b0653915ee"}]: dispatch 2024-04-14T15:09:12.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:11 smithi090 ceph-mon[30399]: from='client.24313 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e5354645-79ec-4d4c-b6e5-f9b0653915ee"}]': finished 2024-04-14T15:09:12.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:11 smithi090 ceph-mon[30399]: osdmap e54: 8 total, 7 up, 8 in 2024-04-14T15:09:12.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:11 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T15:09:12.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:11 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.92:0/1304297095' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e5354645-79ec-4d4c-b6e5-f9b0653915ee"}]: dispatch 2024-04-14T15:09:12.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:11 smithi090 ceph-mon[41670]: from='client.24313 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e5354645-79ec-4d4c-b6e5-f9b0653915ee"}]: dispatch 2024-04-14T15:09:12.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:11 smithi090 ceph-mon[41670]: from='client.24313 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e5354645-79ec-4d4c-b6e5-f9b0653915ee"}]': finished 2024-04-14T15:09:12.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:11 smithi090 ceph-mon[41670]: osdmap e54: 8 total, 7 up, 8 in 2024-04-14T15:09:12.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:11 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T15:09:13.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:12 smithi092 ceph-mon[33966]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-04-14T15:09:13.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:12 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.92:0/3811336293' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T15:09:13.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:12 smithi090 ceph-mon[30399]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-04-14T15:09:13.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:12 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.92:0/3811336293' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T15:09:13.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:12 smithi090 ceph-mon[41670]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-04-14T15:09:13.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:12 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.92:0/3811336293' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T15:09:13.758 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:13.758+0000 7efe6deb2700 10 client.24307.objecter tick 2024-04-14T15:09:15.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:14 smithi092 ceph-mon[33966]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-14T15:09:15.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:14 smithi090 ceph-mon[30399]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-14T15:09:15.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:14 smithi090 ceph-mon[41670]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-14T15:09:17.323 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:16 smithi092 ceph-mon[33966]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-04-14T15:09:17.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:16 smithi090 ceph-mon[30399]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-04-14T15:09:17.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:16 smithi090 ceph-mon[41670]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-04-14T15:09:18.759 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:18.759+0000 7efe6deb2700 10 client.24307.objecter tick 2024-04-14T15:09:19.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:18 smithi092 ceph-mon[33966]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-14T15:09:19.112 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:18 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-14T15:09:19.112 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:18 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:09:19.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:18 smithi090 ceph-mon[30399]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-14T15:09:19.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:18 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-14T15:09:19.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:18 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:09:19.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:18 smithi090 ceph-mon[41670]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-14T15:09:19.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:18 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-14T15:09:19.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:18 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:09:20.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:19 smithi092 ceph-mon[33966]: Deploying daemon osd.7 on smithi092 2024-04-14T15:09:20.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:19 smithi090 ceph-mon[30399]: Deploying daemon osd.7 on smithi092 2024-04-14T15:09:20.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:19 smithi090 ceph-mon[41670]: Deploying daemon osd.7 on smithi092 2024-04-14T15:09:21.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:20 smithi092 ceph-mon[33966]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-14T15:09:21.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:20 smithi090 ceph-mon[30399]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-14T15:09:21.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:20 smithi090 ceph-mon[41670]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-14T15:09:23.269 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:22 smithi092 ceph-mon[33966]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-14T15:09:23.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:22 smithi090 ceph-mon[30399]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-14T15:09:23.466 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:22 smithi090 ceph-mon[41670]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-14T15:09:23.762 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:23.760+0000 7efe6deb2700 10 client.24307.objecter tick 2024-04-14T15:09:25.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:25 smithi092 ceph-mon[33966]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-14T15:09:25.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:25 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:25.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:25 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:25.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:25 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:09:25.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:25 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:09:25.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:25 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:09:25.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:24 smithi090 ceph-mon[30399]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-14T15:09:25.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:24 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:25.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:24 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:25.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:24 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:09:25.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:24 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:09:25.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:24 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:09:25.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:24 smithi090 ceph-mon[41670]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-14T15:09:25.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:24 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:25.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:24 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:25.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:24 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:09:25.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:24 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:09:25.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:24 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:09:26.438 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:26 smithi092 ceph-mon[33966]: osdmap e55: 8 total, 7 up, 8 in 2024-04-14T15:09:26.438 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:26 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T15:09:26.438 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:26 smithi092 ceph-mon[33966]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-14T15:09:26.691 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:26 smithi090 ceph-mon[30399]: osdmap e55: 8 total, 7 up, 8 in 2024-04-14T15:09:26.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:26 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T15:09:26.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:26 smithi090 ceph-mon[30399]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-14T15:09:26.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:26 smithi090 ceph-mon[41670]: osdmap e55: 8 total, 7 up, 8 in 2024-04-14T15:09:26.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:26 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T15:09:26.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:26 smithi090 ceph-mon[41670]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-14T15:09:27.741 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:27.740+0000 7efe657fa700 1 -- 172.21.15.92:0/2557091639 <== mgr.14148 v1:172.21.15.90:6800/491071118 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (unknown 0 0 3917120479) 0x7efe54000bc0 con 0x7efe500712c0 2024-04-14T15:09:27.741 INFO:teuthology.orchestra.run.smithi092.stdout:Created osd(s) 7 on host 'smithi092' 2024-04-14T15:09:27.744 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:27.743+0000 7efe6e6b3700 10 librados: watch_flush enter 2024-04-14T15:09:27.744 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:27.743+0000 7efe6e6b3700 10 librados: watch_flush exit 2024-04-14T15:09:27.744 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:27.743+0000 7efe6e6b3700 20 client.24307.objecter shutdown clearing up homeless session... 2024-04-14T15:09:27.744 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:27.743+0000 7efe6e6b3700 10 client.24307.objecter successfully canceled tick 2024-04-14T15:09:27.744 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:27.743+0000 7efe6e6b3700 1 -- 172.21.15.92:0/2557091639 >> v1:172.21.15.90:6800/491071118 conn(0x7efe500712c0 legacy=0x7efe50073780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:09:27.744 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:27.743+0000 7efe6e6b3700 1 -- 172.21.15.92:0/2557091639 >> v1:172.21.15.92:6789/0 conn(0x7efe68109950 legacy=0x7efe681a0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:09:27.744 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:27.743+0000 7efe6e6b3700 1 -- 172.21.15.92:0/2557091639 shutdown_connections 2024-04-14T15:09:27.744 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:27.743+0000 7efe6e6b3700 1 -- 172.21.15.92:0/2557091639 >> 172.21.15.92:0/2557091639 conn(0x7efe680fcd80 msgr2=0x7efe68108790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:09:27.745 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:27.743+0000 7efe6e6b3700 1 -- 172.21.15.92:0/2557091639 shutdown_connections 2024-04-14T15:09:27.745 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:27.743+0000 7efe6e6b3700 1 -- 172.21.15.92:0/2557091639 wait complete. 2024-04-14T15:09:27.745 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:27.743+0000 7efe6e6b3700 1 librados: shutdown 2024-04-14T15:09:28.080 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:27 smithi092 ceph-mon[33966]: from='osd.7 v1:172.21.15.92:6812/1773706716' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-14T15:09:28.080 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:27 smithi092 ceph-mon[33966]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-14T15:09:28.081 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:27 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:09:28.081 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:27 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:09:28.081 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:27 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:28.081 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:27 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:28.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:27 smithi090 ceph-mon[30399]: from='osd.7 v1:172.21.15.92:6812/1773706716' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-14T15:09:28.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:27 smithi090 ceph-mon[30399]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-14T15:09:28.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:27 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:09:28.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:27 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:09:28.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:27 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:28.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:27 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:28.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:27 smithi090 ceph-mon[41670]: from='osd.7 v1:172.21.15.92:6812/1773706716' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-14T15:09:28.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:27 smithi090 ceph-mon[41670]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-14T15:09:28.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:27 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:09:28.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:27 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:09:28.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:27 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:28.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:27 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:28.408 DEBUG:teuthology.orchestra.run.smithi092:osd.7> sudo journalctl -f -n 0 -u ceph-0e288098-fa70-11ee-bc8e-c7b262605968@osd.7.service 2024-04-14T15:09:28.410 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-04-14T15:09:28.410 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd stat -f json 2024-04-14T15:09:28.663 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:09:29.103 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:28 smithi090 ceph-mon[41670]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-14T15:09:29.103 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:28 smithi090 ceph-mon[41670]: from='osd.7 v1:172.21.15.92:6812/1773706716' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-04-14T15:09:29.103 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:28 smithi090 ceph-mon[41670]: osdmap e56: 8 total, 7 up, 8 in 2024-04-14T15:09:29.103 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:28 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T15:09:29.103 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:28 smithi090 ceph-mon[41670]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-04-14T15:09:29.103 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:28 smithi090 ceph-mon[41670]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-14T15:09:29.103 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:28 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:29.103 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:28 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:29.103 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:28 smithi090 ceph-mon[30399]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-14T15:09:29.104 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:28 smithi090 ceph-mon[30399]: from='osd.7 v1:172.21.15.92:6812/1773706716' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-04-14T15:09:29.104 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:28 smithi090 ceph-mon[30399]: osdmap e56: 8 total, 7 up, 8 in 2024-04-14T15:09:29.104 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:28 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T15:09:29.104 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:28 smithi090 ceph-mon[30399]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-04-14T15:09:29.104 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:28 smithi090 ceph-mon[30399]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-14T15:09:29.104 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:28 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:29.104 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:28 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:29.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:28 smithi092 ceph-mon[33966]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-14T15:09:29.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:28 smithi092 ceph-mon[33966]: from='osd.7 v1:172.21.15.92:6812/1773706716' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-04-14T15:09:29.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:28 smithi092 ceph-mon[33966]: osdmap e56: 8 total, 7 up, 8 in 2024-04-14T15:09:29.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:28 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T15:09:29.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:28 smithi092 ceph-mon[33966]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-04-14T15:09:29.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:28 smithi092 ceph-mon[33966]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-14T15:09:29.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:28 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:29.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:28 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:29.123 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:09:28 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:09:28.772+0000 7fa00c5d8700 -1 osd.7 0 waiting for initial osdmap 2024-04-14T15:09:29.123 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:09:28 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:09:28.779+0000 7fa0083cb700 -1 osd.7 57 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-14T15:09:29.960 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:29 smithi092 ceph-mon[33966]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2024-04-14T15:09:29.960 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:29 smithi092 ceph-mon[33966]: osdmap e57: 8 total, 7 up, 8 in 2024-04-14T15:09:29.960 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:29 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T15:09:29.960 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:29 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T15:09:29.960 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:29 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T15:09:30.046 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:29 smithi090 ceph-mon[30399]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2024-04-14T15:09:30.046 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:29 smithi090 ceph-mon[30399]: osdmap e57: 8 total, 7 up, 8 in 2024-04-14T15:09:30.046 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:29 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T15:09:30.046 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:29 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T15:09:30.046 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:29 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T15:09:30.046 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:29 smithi090 ceph-mon[41670]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2024-04-14T15:09:30.046 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:29 smithi090 ceph-mon[41670]: osdmap e57: 8 total, 7 up, 8 in 2024-04-14T15:09:30.047 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:29 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T15:09:30.047 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:29 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T15:09:30.047 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:29 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T15:09:30.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.275+0000 7efc60265700 1 -- 172.21.15.90:0/3301549981 >> v1:172.21.15.90:6790/0 conn(0x7efc5810aa20 legacy=0x7efc5810cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:09:30.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.275+0000 7efc60265700 1 -- 172.21.15.90:0/3301549981 shutdown_connections 2024-04-14T15:09:30.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.275+0000 7efc60265700 1 -- 172.21.15.90:0/3301549981 >> 172.21.15.90:0/3301549981 conn(0x7efc580fcd80 msgr2=0x7efc580ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:09:30.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.275+0000 7efc60265700 1 -- 172.21.15.90:0/3301549981 shutdown_connections 2024-04-14T15:09:30.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.275+0000 7efc60265700 1 -- 172.21.15.90:0/3301549981 wait complete. 2024-04-14T15:09:30.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.275+0000 7efc60265700 1 librados: starting msgr at 2024-04-14T15:09:30.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.275+0000 7efc60265700 1 librados: starting objecter 2024-04-14T15:09:30.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.275+0000 7efc60265700 1 Processor -- start 2024-04-14T15:09:30.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.275+0000 7efc60265700 1 -- start start 2024-04-14T15:09:30.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.275+0000 7efc60265700 1 librados: setting wanted keys 2024-04-14T15:09:30.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.275+0000 7efc60265700 1 librados: calling monclient init 2024-04-14T15:09:30.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.276+0000 7efc60265700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7efc58198ac0 con 0x7efc5810aa20 2024-04-14T15:09:30.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.276+0000 7efc60265700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7efc581a42b0 con 0x7efc58103370 2024-04-14T15:09:30.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.276+0000 7efc60265700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7efc581a5490 con 0x7efc58106e80 2024-04-14T15:09:30.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.276+0000 7efc5d800700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7efc58106e80 0x7efc5819e450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:48724/0 (socket says 172.21.15.90:48724) 2024-04-14T15:09:30.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.276+0000 7efc5e802700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7efc5810aa20 0x7efc581a1b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:52614/0 (socket says 172.21.15.90:52614) 2024-04-14T15:09:30.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.276+0000 7efc5d800700 1 -- 172.21.15.90:0/3640264331 learned_addr learned my addr 172.21.15.90:0/3640264331 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:09:30.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.276+0000 7efc4effd700 10 client.?.objecter ms_handle_connect 0x7efc5810aa20 2024-04-14T15:09:30.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.276+0000 7efc4effd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:09:30.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.276+0000 7efc4effd700 10 client.?.objecter ms_handle_connect 0x7efc58106e80 2024-04-14T15:09:30.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.276+0000 7efc4effd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:09:30.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.276+0000 7efc4effd700 10 client.?.objecter ms_handle_connect 0x7efc58103370 2024-04-14T15:09:30.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.276+0000 7efc4effd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:09:30.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.276+0000 7efc4effd700 1 -- 172.21.15.90:0/3640264331 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2424986942 0 0) 0x7efc581a42b0 con 0x7efc58103370 2024-04-14T15:09:30.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.276+0000 7efc4effd700 1 -- 172.21.15.90:0/3640264331 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efc440031f0 con 0x7efc58103370 2024-04-14T15:09:30.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.276+0000 7efc4effd700 1 -- 172.21.15.90:0/3640264331 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3043865496 0 0) 0x7efc581a5490 con 0x7efc58106e80 2024-04-14T15:09:30.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.276+0000 7efc4effd700 1 -- 172.21.15.90:0/3640264331 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efc581a42b0 con 0x7efc58106e80 2024-04-14T15:09:30.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.276+0000 7efc4effd700 1 -- 172.21.15.90:0/3640264331 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3139234063 0 0) 0x7efc58198ac0 con 0x7efc5810aa20 2024-04-14T15:09:30.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.276+0000 7efc4effd700 1 -- 172.21.15.90:0/3640264331 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efc581a5490 con 0x7efc5810aa20 2024-04-14T15:09:30.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.276+0000 7efc4effd700 1 -- 172.21.15.90:0/3640264331 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 44872115 0 0) 0x7efc440031f0 con 0x7efc58103370 2024-04-14T15:09:30.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.276+0000 7efc4effd700 1 -- 172.21.15.90:0/3640264331 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efc58198ac0 con 0x7efc58103370 2024-04-14T15:09:30.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.276+0000 7efc4effd700 1 -- 172.21.15.90:0/3640264331 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7efc540018b0 con 0x7efc58103370 2024-04-14T15:09:30.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.276+0000 7efc4effd700 1 -- 172.21.15.90:0/3640264331 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3426775637 0 0) 0x7efc581a42b0 con 0x7efc58106e80 2024-04-14T15:09:30.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.276+0000 7efc4effd700 1 -- 172.21.15.90:0/3640264331 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efc440031f0 con 0x7efc58106e80 2024-04-14T15:09:30.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.276+0000 7efc4effd700 1 -- 172.21.15.90:0/3640264331 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7efc480018f0 con 0x7efc58106e80 2024-04-14T15:09:30.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.276+0000 7efc4effd700 1 -- 172.21.15.90:0/3640264331 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1445442013 0 0) 0x7efc581a5490 con 0x7efc5810aa20 2024-04-14T15:09:30.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.276+0000 7efc4effd700 1 -- 172.21.15.90:0/3640264331 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efc581a42b0 con 0x7efc5810aa20 2024-04-14T15:09:30.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.276+0000 7efc4effd700 1 -- 172.21.15.90:0/3640264331 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7efc500017b0 con 0x7efc5810aa20 2024-04-14T15:09:30.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.276+0000 7efc4effd700 1 -- 172.21.15.90:0/3640264331 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1111671065 0 0) 0x7efc58198ac0 con 0x7efc58103370 2024-04-14T15:09:30.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.276+0000 7efc4effd700 1 -- 172.21.15.90:0/3640264331 >> v1:172.21.15.90:6790/0 conn(0x7efc58106e80 legacy=0x7efc5819e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:09:30.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.276+0000 7efc4effd700 1 -- 172.21.15.90:0/3640264331 >> v1:172.21.15.90:6789/0 conn(0x7efc5810aa20 legacy=0x7efc581a1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:09:30.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.276+0000 7efc4effd700 1 -- 172.21.15.90:0/3640264331 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc581a66b0 con 0x7efc58103370 2024-04-14T15:09:30.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.277+0000 7efc60265700 1 -- 172.21.15.90:0/3640264331 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7efc581a56c0 con 0x7efc58103370 2024-04-14T15:09:30.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.277+0000 7efc60265700 10 client.24334.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:09:30.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.277+0000 7efc60265700 1 -- 172.21.15.90:0/3640264331 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7efc581a5c00 con 0x7efc58103370 2024-04-14T15:09:30.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.277+0000 7efc60265700 1 librados: init done 2024-04-14T15:09:30.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.277+0000 7efc4effd700 1 -- 172.21.15.90:0/3640264331 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7efc54001c60 con 0x7efc58103370 2024-04-14T15:09:30.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.277+0000 7efc4effd700 1 -- 172.21.15.90:0/3640264331 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7efc54004b00 con 0x7efc58103370 2024-04-14T15:09:30.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.277+0000 7efc60265700 1 -- 172.21.15.90:0/3640264331 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc3c005150 con 0x7efc58103370 2024-04-14T15:09:30.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.278+0000 7efc4effd700 1 -- 172.21.15.90:0/3640264331 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 15) ==== 92623+0+0 (unknown 2343906571 0 0) 0x7efc54003950 con 0x7efc58103370 2024-04-14T15:09:30.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.278+0000 7efc4effd700 1 -- 172.21.15.90:0/3640264331 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(58..58 src has 1..58) ==== 4114+0+0 (unknown 1431882447 0 0) 0x7efc540886b0 con 0x7efc58103370 2024-04-14T15:09:30.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.279+0000 7efc4effd700 10 client.24334.objecter ms_dispatch 0x7efc58001550 osd_map(58..58 src has 1..58) 2024-04-14T15:09:30.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.279+0000 7efc4effd700 3 client.24334.objecter handle_osd_map got epochs [58,58] > 0 2024-04-14T15:09:30.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.279+0000 7efc4effd700 3 client.24334.objecter handle_osd_map decoding full epoch 58 2024-04-14T15:09:30.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.279+0000 7efc4effd700 20 client.24334.objecter dump_active .. 0 homeless 2024-04-14T15:09:30.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.279+0000 7efc4effd700 10 client.24334.objecter ms_handle_connect 0x7efc44071270 2024-04-14T15:09:30.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.283+0000 7efc4effd700 1 -- 172.21.15.90:0/3640264331 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7efc54056090 con 0x7efc58103370 2024-04-14T15:09:30.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.481+0000 7efc60265700 1 -- 172.21.15.90:0/3640264331 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7efc3c005440 con 0x7efc58103370 2024-04-14T15:09:30.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.481+0000 7efc4effd700 1 -- 172.21.15.90:0/3640264331 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v58) ==== 74+0+130 (unknown 4023224526 0 3073086068) 0x7efc54055990 con 0x7efc58103370 2024-04-14T15:09:30.482 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:09:30.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.483+0000 7efc60265700 10 librados: watch_flush enter 2024-04-14T15:09:30.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.483+0000 7efc60265700 10 librados: watch_flush exit 2024-04-14T15:09:30.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.483+0000 7efc60265700 20 client.24334.objecter shutdown clearing up homeless session... 2024-04-14T15:09:30.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.483+0000 7efc60265700 10 client.24334.objecter successfully canceled tick 2024-04-14T15:09:30.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.484+0000 7efc60265700 1 -- 172.21.15.90:0/3640264331 >> v1:172.21.15.90:6800/491071118 conn(0x7efc44071270 legacy=0x7efc44073730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:09:30.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.484+0000 7efc60265700 1 -- 172.21.15.90:0/3640264331 >> v1:172.21.15.92:6789/0 conn(0x7efc58103370 legacy=0x7efc58197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:09:30.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.484+0000 7efc60265700 1 -- 172.21.15.90:0/3640264331 shutdown_connections 2024-04-14T15:09:30.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.484+0000 7efc60265700 1 -- 172.21.15.90:0/3640264331 >> 172.21.15.90:0/3640264331 conn(0x7efc580fcd80 msgr2=0x7efc580ff170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:09:30.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.484+0000 7efc60265700 1 -- 172.21.15.90:0/3640264331 shutdown_connections 2024-04-14T15:09:30.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.484+0000 7efc60265700 1 -- 172.21.15.90:0/3640264331 wait complete. 2024-04-14T15:09:30.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:30.484+0000 7efc60265700 1 librados: shutdown 2024-04-14T15:09:30.958 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:30 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:09:30.959 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:30 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:09:30.959 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:30 smithi092 ceph-mon[33966]: osd.7 v1:172.21.15.92:6812/1773706716 boot 2024-04-14T15:09:30.959 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:30 smithi092 ceph-mon[33966]: osdmap e58: 8 total, 8 up, 8 in 2024-04-14T15:09:30.959 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:30 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T15:09:30.959 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:30 smithi092 ceph-mon[33966]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-14T15:09:30.959 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:30 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3640264331' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-14T15:09:30.959 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:30 smithi092 ceph-mon[33966]: osdmap e59: 8 total, 8 up, 8 in 2024-04-14T15:09:31.083 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:30 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:09:31.083 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:30 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:09:31.083 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:30 smithi090 ceph-mon[41670]: osd.7 v1:172.21.15.92:6812/1773706716 boot 2024-04-14T15:09:31.083 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:30 smithi090 ceph-mon[41670]: osdmap e58: 8 total, 8 up, 8 in 2024-04-14T15:09:31.083 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:30 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T15:09:31.083 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:30 smithi090 ceph-mon[41670]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-14T15:09:31.083 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:30 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3640264331' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-14T15:09:31.083 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:30 smithi090 ceph-mon[41670]: osdmap e59: 8 total, 8 up, 8 in 2024-04-14T15:09:31.083 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:30 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:09:31.084 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:30 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:09:31.084 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:30 smithi090 ceph-mon[30399]: osd.7 v1:172.21.15.92:6812/1773706716 boot 2024-04-14T15:09:31.084 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:30 smithi090 ceph-mon[30399]: osdmap e58: 8 total, 8 up, 8 in 2024-04-14T15:09:31.084 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:30 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T15:09:31.084 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:30 smithi090 ceph-mon[30399]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-14T15:09:31.084 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:30 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3640264331' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-14T15:09:31.084 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:30 smithi090 ceph-mon[30399]: osdmap e59: 8 total, 8 up, 8 in 2024-04-14T15:09:31.109 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":58,"num_osds":8,"num_up_osds":8,"osd_up_since":1713107369,"num_in_osds":8,"osd_in_since":1713107351,"num_remapped_pgs":1} 2024-04-14T15:09:31.110 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd dump --format=json 2024-04-14T15:09:31.346 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:09:32.750 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:32 smithi090 ceph-mon[41670]: osdmap e60: 8 total, 8 up, 8 in 2024-04-14T15:09:32.750 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:32 smithi090 ceph-mon[41670]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:09:32.750 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:32 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:32.750 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:32 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:32.750 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:32 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:09:32.750 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:32 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:09:32.750 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:32 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:09:32.750 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:32 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:09:32.750 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:32 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:32.750 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:32 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:09:32.750 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:32 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:09:32.751 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:32 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:32.751 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:32 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:09:33.022 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:32 smithi090 ceph-mon[30399]: osdmap e60: 8 total, 8 up, 8 in 2024-04-14T15:09:33.022 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:32 smithi090 ceph-mon[30399]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:09:33.023 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:32 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:33.023 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:32 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:33.023 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:32 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:09:33.023 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:32 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:09:33.023 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:32 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:09:33.023 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:32 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:09:33.023 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:32 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:33.023 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:32 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:09:33.023 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:32 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:09:33.023 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:32 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:33.023 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:32 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:09:33.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.046+0000 7fce587d8700 1 -- 172.21.15.90:0/4127932709 >> v1:172.21.15.90:6789/0 conn(0x7fce5010a740 legacy=0x7fce5010cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:09:33.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.046+0000 7fce587d8700 1 -- 172.21.15.90:0/4127932709 shutdown_connections 2024-04-14T15:09:33.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.046+0000 7fce587d8700 1 -- 172.21.15.90:0/4127932709 >> 172.21.15.90:0/4127932709 conn(0x7fce500fd290 msgr2=0x7fce500ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:09:33.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.046+0000 7fce587d8700 1 -- 172.21.15.90:0/4127932709 shutdown_connections 2024-04-14T15:09:33.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.046+0000 7fce587d8700 1 -- 172.21.15.90:0/4127932709 wait complete. 2024-04-14T15:09:33.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.046+0000 7fce587d8700 1 librados: starting msgr at 2024-04-14T15:09:33.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.046+0000 7fce587d8700 1 librados: starting objecter 2024-04-14T15:09:33.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.046+0000 7fce587d8700 1 Processor -- start 2024-04-14T15:09:33.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.047+0000 7fce587d8700 1 -- start start 2024-04-14T15:09:33.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.047+0000 7fce587d8700 1 librados: setting wanted keys 2024-04-14T15:09:33.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.047+0000 7fce587d8700 1 librados: calling monclient init 2024-04-14T15:09:33.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.047+0000 7fce56574700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fce50107890 0x7fce5010db50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:33272/0 (socket says 172.21.15.90:33272) 2024-04-14T15:09:33.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.047+0000 7fce56574700 1 -- 172.21.15.90:0/3729797341 learned_addr learned my addr 172.21.15.90:0/3729797341 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:09:33.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.047+0000 7fce477fe700 10 client.?.objecter ms_handle_connect 0x7fce50107890 2024-04-14T15:09:33.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.047+0000 7fce477fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:09:33.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.047+0000 7fce477fe700 10 client.?.objecter ms_handle_connect 0x7fce5010a740 2024-04-14T15:09:33.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.047+0000 7fce477fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:09:33.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.047+0000 7fce587d8700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fce501a9d90 con 0x7fce50107890 2024-04-14T15:09:33.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.047+0000 7fce587d8700 1 -- 172.21.15.90:0/3729797341 --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fce501aafb0 con 0x7fce5010e310 2024-04-14T15:09:33.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.047+0000 7fce587d8700 1 -- 172.21.15.90:0/3729797341 --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fce501ac1f0 con 0x7fce5010a740 2024-04-14T15:09:33.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.047+0000 7fce477fe700 10 client.?.objecter ms_handle_connect 0x7fce5010e310 2024-04-14T15:09:33.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.047+0000 7fce477fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:09:33.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.047+0000 7fce477fe700 1 -- 172.21.15.90:0/3729797341 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 260639288 0 0) 0x7fce501a9d90 con 0x7fce50107890 2024-04-14T15:09:33.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.047+0000 7fce477fe700 1 -- 172.21.15.90:0/3729797341 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fce3c0031f0 con 0x7fce50107890 2024-04-14T15:09:33.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.047+0000 7fce477fe700 1 -- 172.21.15.90:0/3729797341 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3955580842 0 0) 0x7fce501aafb0 con 0x7fce5010e310 2024-04-14T15:09:33.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.047+0000 7fce477fe700 1 -- 172.21.15.90:0/3729797341 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fce501a9d90 con 0x7fce5010e310 2024-04-14T15:09:33.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.047+0000 7fce477fe700 1 -- 172.21.15.90:0/3729797341 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1115075730 0 0) 0x7fce501ac1f0 con 0x7fce5010a740 2024-04-14T15:09:33.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.047+0000 7fce477fe700 1 -- 172.21.15.90:0/3729797341 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fce501aafb0 con 0x7fce5010a740 2024-04-14T15:09:33.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.047+0000 7fce477fe700 1 -- 172.21.15.90:0/3729797341 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 370194831 0 0) 0x7fce3c0031f0 con 0x7fce50107890 2024-04-14T15:09:33.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.048+0000 7fce477fe700 1 -- 172.21.15.90:0/3729797341 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fce501ac1f0 con 0x7fce50107890 2024-04-14T15:09:33.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.048+0000 7fce477fe700 1 -- 172.21.15.90:0/3729797341 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fce40001930 con 0x7fce50107890 2024-04-14T15:09:33.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.048+0000 7fce477fe700 1 -- 172.21.15.90:0/3729797341 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 373363741 0 0) 0x7fce501a9d90 con 0x7fce5010e310 2024-04-14T15:09:33.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.048+0000 7fce477fe700 1 -- 172.21.15.90:0/3729797341 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fce3c0031f0 con 0x7fce5010e310 2024-04-14T15:09:33.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.048+0000 7fce477fe700 1 -- 172.21.15.90:0/3729797341 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fce480013f0 con 0x7fce5010e310 2024-04-14T15:09:33.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.048+0000 7fce477fe700 1 -- 172.21.15.90:0/3729797341 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1333293467 0 0) 0x7fce501aafb0 con 0x7fce5010a740 2024-04-14T15:09:33.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.048+0000 7fce477fe700 1 -- 172.21.15.90:0/3729797341 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fce501a9d90 con 0x7fce5010a740 2024-04-14T15:09:33.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.048+0000 7fce477fe700 1 -- 172.21.15.90:0/3729797341 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fce4c0037f0 con 0x7fce5010a740 2024-04-14T15:09:33.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.048+0000 7fce477fe700 1 -- 172.21.15.90:0/3729797341 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1622461240 0 0) 0x7fce3c0031f0 con 0x7fce5010e310 2024-04-14T15:09:33.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.048+0000 7fce477fe700 1 -- 172.21.15.90:0/3729797341 >> v1:172.21.15.90:6790/0 conn(0x7fce5010a740 legacy=0x7fce501a4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:09:33.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.048+0000 7fce477fe700 1 -- 172.21.15.90:0/3729797341 >> v1:172.21.15.90:6789/0 conn(0x7fce50107890 legacy=0x7fce5010db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:09:33.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.048+0000 7fce477fe700 1 -- 172.21.15.90:0/3729797341 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce501ab1c0 con 0x7fce5010e310 2024-04-14T15:09:33.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.048+0000 7fce587d8700 1 -- 172.21.15.90:0/3729797341 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fce501ab4b0 con 0x7fce5010e310 2024-04-14T15:09:33.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.048+0000 7fce587d8700 10 client.24340.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:09:33.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.048+0000 7fce587d8700 1 -- 172.21.15.90:0/3729797341 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fce501ab970 con 0x7fce5010e310 2024-04-14T15:09:33.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.048+0000 7fce587d8700 1 librados: init done 2024-04-14T15:09:33.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.048+0000 7fce477fe700 1 -- 172.21.15.90:0/3729797341 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fce480019a0 con 0x7fce5010e310 2024-04-14T15:09:33.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.048+0000 7fce477fe700 1 -- 172.21.15.90:0/3729797341 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fce48004d10 con 0x7fce5010e310 2024-04-14T15:09:33.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.049+0000 7fce587d8700 1 -- 172.21.15.90:0/3729797341 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce34005120 con 0x7fce5010e310 2024-04-14T15:09:33.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.049+0000 7fce477fe700 1 -- 172.21.15.90:0/3729797341 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 15) ==== 92623+0+0 (unknown 2343906571 0 0) 0x7fce48005020 con 0x7fce5010e310 2024-04-14T15:09:33.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.050+0000 7fce477fe700 1 -- 172.21.15.90:0/3729797341 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(61..61 src has 1..61) ==== 4081+0+0 (unknown 283209459 0 0) 0x7fce480889d0 con 0x7fce5010e310 2024-04-14T15:09:33.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.050+0000 7fce477fe700 10 client.24340.objecter ms_dispatch 0x7fce50001550 osd_map(61..61 src has 1..61) 2024-04-14T15:09:33.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.050+0000 7fce477fe700 3 client.24340.objecter handle_osd_map got epochs [61,61] > 0 2024-04-14T15:09:33.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.050+0000 7fce477fe700 3 client.24340.objecter handle_osd_map decoding full epoch 61 2024-04-14T15:09:33.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.050+0000 7fce477fe700 20 client.24340.objecter dump_active .. 0 homeless 2024-04-14T15:09:33.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.050+0000 7fce477fe700 10 client.24340.objecter ms_handle_connect 0x7fce3c071270 2024-04-14T15:09:33.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.054+0000 7fce477fe700 1 -- 172.21.15.90:0/3729797341 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fce480563d0 con 0x7fce5010e310 2024-04-14T15:09:33.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:32 smithi092 ceph-mon[33966]: osdmap e60: 8 total, 8 up, 8 in 2024-04-14T15:09:33.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:32 smithi092 ceph-mon[33966]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:09:33.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:32 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:33.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:32 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:33.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:32 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:09:33.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:32 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:09:33.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:32 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:09:33.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:32 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:09:33.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:32 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:33.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:32 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:09:33.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:32 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:09:33.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:32 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:33.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:32 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:09:33.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.261+0000 7fce587d8700 1 -- 172.21.15.90:0/3729797341 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fce34005410 con 0x7fce5010e310 2024-04-14T15:09:33.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.263+0000 7fce477fe700 1 -- 172.21.15.90:0/3729797341 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v61) ==== 74+0+11471 (unknown 1877459060 0 1734004835) 0x7fce48055cd0 con 0x7fce5010e310 2024-04-14T15:09:33.263 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:09:33.263 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":61,"fsid":"0e288098-fa70-11ee-bc8e-c7b262605968","created":"2024-04-14T15:04:05.808313+0000","modified":"2024-04-14T15:09:32.743231+0000","last_up_change":"2024-04-14T15:09:29.769352+0000","last_in_change":"2024-04-14T15:09:11.803164+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-14T15:07:23.959628+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"76772cfc-89d1-4013-b617-32284ed8dfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6801","nonce":3761806885}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6802","nonce":3761806885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6804","nonce":3761806885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6803","nonce":3761806885}]},"public_addr":"172.21.15.90:6801/3761806885","cluster_addr":"172.21.15.90:6802/3761806885","heartbeat_back_addr":"172.21.15.90:6804/3761806885","heartbeat_front_addr":"172.21.15.90:6803/3761806885","state":["exists","up"]},{"osd":1,"uuid":"de4a7fa2-7d57-4ff3-a4fa-28c0cde8ef6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6805","nonce":4170361699}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6806","nonce":4170361699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6808","nonce":4170361699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6807","nonce":4170361699}]},"public_addr":"172.21.15.90:6805/4170361699","cluster_addr":"172.21.15.90:6806/4170361699","heartbeat_back_addr":"172.21.15.90:6808/4170361699","heartbeat_front_addr":"172.21.15.90:6807/4170361699","state":["exists","up"]},{"osd":2,"uuid":"9c39b5ff-6175-46ba-9d71-370581c6ff3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6809","nonce":1686184251}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6810","nonce":1686184251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6812","nonce":1686184251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6811","nonce":1686184251}]},"public_addr":"172.21.15.90:6809/1686184251","cluster_addr":"172.21.15.90:6810/1686184251","heartbeat_back_addr":"172.21.15.90:6812/1686184251","heartbeat_front_addr":"172.21.15.90:6811/1686184251","state":["exists","up"]},{"osd":3,"uuid":"efd2ca45-5b80-4013-a571-499e2cb5b2e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6813","nonce":2270166774}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6814","nonce":2270166774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6816","nonce":2270166774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6815","nonce":2270166774}]},"public_addr":"172.21.15.90:6813/2270166774","cluster_addr":"172.21.15.90:6814/2270166774","heartbeat_back_addr":"172.21.15.90:6816/2270166774","heartbeat_front_addr":"172.21.15.90:6815/2270166774","state":["exists","up"]},{"osd":4,"uuid":"e3d9013d-5eb6-433c-96b3-3fc38ba7ac8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6800","nonce":3887549733}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6801","nonce":3887549733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6803","nonce":3887549733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6802","nonce":3887549733}]},"public_addr":"172.21.15.92:6800/3887549733","cluster_addr":"172.21.15.92:6801/3887549733","heartbeat_back_addr":"172.21.15.92:6803/3887549733","heartbeat_front_addr":"172.21.15.92:6802/3887549733","state":["exists","up"]},{"osd":5,"uuid":"1ec41805-6f82-4139-81bc-75305ba9e958","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6804","nonce":3244561440}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6805","nonce":3244561440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6807","nonce":3244561440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6806","nonce":3244561440}]},"public_addr":"172.21.15.92:6804/3244561440","cluster_addr":"172.21.15.92:6805/3244561440","heartbeat_back_addr":"172.21.15.92:6807/3244561440","heartbeat_front_addr":"172.21.15.92:6806/3244561440","state":["exists","up"]},{"osd":6,"uuid":"ca977ab2-bf8d-4b3a-ae5e-0efa5bc7a3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6808","nonce":636125963}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6809","nonce":636125963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6811","nonce":636125963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6810","nonce":636125963}]},"public_addr":"172.21.15.92:6808/636125963","cluster_addr":"172.21.15.92:6809/636125963","heartbeat_back_addr":"172.21.15.92:6811/636125963","heartbeat_front_addr":"172.21.15.92:6810/636125963","state":["exists","up"]},{"osd":7,"uuid":"e5354645-79ec-4d4c-b6e5-f9b0653915ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6812","nonce":1773706716}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6813","nonce":1773706716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6815","nonce":1773706716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6814","nonce":1773706716}]},"public_addr":"172.21.15.92:6812/1773706716","cluster_addr":"172.21.15.92:6813/1773706716","heartbeat_back_addr":"172.21.15.92:6815/1773706716","heartbeat_front_addr":"172.21.15.92:6814/1773706716","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T15:09:04.496851+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T15:09:24.387925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T15:08:27.374441+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T15:09:01.292185+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T15:09:27.428788+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T15:08:35.749690+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T15:09:01.317538+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T15:09:28.295984+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:0/1139868610":"2024-04-15T15:05:05.882361+0000","172.21.15.90:0/977818204":"2024-04-15T15:04:39.251711+0000","172.21.15.90:6800/941091171":"2024-04-15T15:05:05.882361+0000","172.21.15.90:0/3671453561":"2024-04-15T15:05:05.882361+0000","172.21.15.90:0/2611856200":"2024-04-15T15:05:05.882361+0000","172.21.15.90:6800/2306258962":"2024-04-15T15:04:39.251711+0000","172.21.15.90:0/4051692380":"2024-04-15T15:04:39.251711+0000","172.21.15.90:0/2681502561":"2024-04-15T15:04:39.251711+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-14T15:09:33.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.264+0000 7fce587d8700 10 librados: watch_flush enter 2024-04-14T15:09:33.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.264+0000 7fce587d8700 10 librados: watch_flush exit 2024-04-14T15:09:33.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.264+0000 7fce587d8700 20 client.24340.objecter shutdown clearing up homeless session... 2024-04-14T15:09:33.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.265+0000 7fce587d8700 10 client.24340.objecter successfully canceled tick 2024-04-14T15:09:33.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.265+0000 7fce587d8700 1 -- 172.21.15.90:0/3729797341 >> v1:172.21.15.90:6800/491071118 conn(0x7fce3c071270 legacy=0x7fce3c073730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:09:33.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.265+0000 7fce587d8700 1 -- 172.21.15.90:0/3729797341 >> v1:172.21.15.92:6789/0 conn(0x7fce5010e310 legacy=0x7fce501a8450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:09:33.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.265+0000 7fce587d8700 1 -- 172.21.15.90:0/3729797341 shutdown_connections 2024-04-14T15:09:33.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.265+0000 7fce587d8700 1 -- 172.21.15.90:0/3729797341 >> 172.21.15.90:0/3729797341 conn(0x7fce500fd290 msgr2=0x7fce50110fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:09:33.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.265+0000 7fce587d8700 1 -- 172.21.15.90:0/3729797341 shutdown_connections 2024-04-14T15:09:33.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.265+0000 7fce587d8700 1 -- 172.21.15.90:0/3729797341 wait complete. 2024-04-14T15:09:33.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:33.265+0000 7fce587d8700 1 librados: shutdown 2024-04-14T15:09:34.094 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:33 smithi090 ceph-mon[41670]: Detected new or changed devices on smithi092 2024-04-14T15:09:34.094 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:33 smithi090 ceph-mon[41670]: Adjusting osd_memory_target on smithi092 to 4272M 2024-04-14T15:09:34.094 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:33 smithi090 ceph-mon[41670]: osdmap e61: 8 total, 8 up, 8 in 2024-04-14T15:09:34.094 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:33 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3729797341' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-14T15:09:34.095 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:33 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:09:34.095 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:33 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:09:34.095 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:33 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:34.095 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:33 smithi090 ceph-mon[30399]: Detected new or changed devices on smithi092 2024-04-14T15:09:34.095 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:33 smithi090 ceph-mon[30399]: Adjusting osd_memory_target on smithi092 to 4272M 2024-04-14T15:09:34.095 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:33 smithi090 ceph-mon[30399]: osdmap e61: 8 total, 8 up, 8 in 2024-04-14T15:09:34.095 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:33 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3729797341' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-14T15:09:34.095 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:33 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:09:34.095 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:33 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:09:34.095 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:33 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:34.124 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-14T15:07:23.959628+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-14T15:09:34.124 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-14T15:09:34.125 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:33 smithi092 ceph-mon[33966]: Detected new or changed devices on smithi092 2024-04-14T15:09:34.125 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:33 smithi092 ceph-mon[33966]: Adjusting osd_memory_target on smithi092 to 4272M 2024-04-14T15:09:34.125 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:33 smithi092 ceph-mon[33966]: osdmap e61: 8 total, 8 up, 8 in 2024-04-14T15:09:34.125 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:33 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3729797341' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-14T15:09:34.125 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:33 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:09:34.125 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:33 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:09:34.125 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:33 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:34.361 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:09:35.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:34 smithi092 ceph-mon[33966]: osdmap e62: 8 total, 8 up, 8 in 2024-04-14T15:09:35.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:34 smithi092 ceph-mon[33966]: pgmap v186: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-14T15:09:35.168 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:34 smithi090 ceph-mon[41670]: osdmap e62: 8 total, 8 up, 8 in 2024-04-14T15:09:35.168 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:34 smithi090 ceph-mon[41670]: pgmap v186: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-14T15:09:35.168 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:34 smithi090 ceph-mon[30399]: osdmap e62: 8 total, 8 up, 8 in 2024-04-14T15:09:35.168 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:34 smithi090 ceph-mon[30399]: pgmap v186: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-14T15:09:35.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:35.963+0000 7f8481428700 1 -- 172.21.15.90:0/108159979 >> v1:172.21.15.90:6790/0 conn(0x7f847c10a740 legacy=0x7f847c10cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:09:35.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:35.963+0000 7f8481428700 1 -- 172.21.15.90:0/108159979 shutdown_connections 2024-04-14T15:09:35.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:35.963+0000 7f8481428700 1 -- 172.21.15.90:0/108159979 >> 172.21.15.90:0/108159979 conn(0x7f847c0fd290 msgr2=0x7f847c0ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:09:35.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:35.963+0000 7f8481428700 1 -- 172.21.15.90:0/108159979 shutdown_connections 2024-04-14T15:09:35.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:35.963+0000 7f8481428700 1 -- 172.21.15.90:0/108159979 wait complete. 2024-04-14T15:09:35.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:35.963+0000 7f8481428700 1 librados: starting msgr at 2024-04-14T15:09:35.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:35.963+0000 7f8481428700 1 librados: starting objecter 2024-04-14T15:09:35.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:35.963+0000 7f8481428700 1 Processor -- start 2024-04-14T15:09:35.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:35.963+0000 7f8481428700 1 -- start start 2024-04-14T15:09:35.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:35.963+0000 7f8481428700 1 librados: setting wanted keys 2024-04-14T15:09:35.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:35.963+0000 7f8481428700 1 librados: calling monclient init 2024-04-14T15:09:35.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:35.964+0000 7f8481428700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f847c1a6ae0 con 0x7f847c10e310 2024-04-14T15:09:35.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:35.964+0000 7f8481428700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f847c1a7d00 con 0x7f847c10a740 2024-04-14T15:09:35.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:35.964+0000 7f8481428700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f847c1a8f40 con 0x7f847c107890 2024-04-14T15:09:35.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:35.964+0000 7f847affd700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f847c107890 0x7f847c10dbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:47350/0 (socket says 172.21.15.90:47350) 2024-04-14T15:09:35.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:35.964+0000 7f847affd700 1 -- 172.21.15.90:0/434711940 learned_addr learned my addr 172.21.15.90:0/434711940 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:09:35.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:35.964+0000 7f847b7fe700 1 --1- 172.21.15.90:0/434711940 >> v1:172.21.15.90:6789/0 conn(0x7f847c10e310 0x7f847c1a51e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:33298/0 (socket says 172.21.15.90:33298) 2024-04-14T15:09:35.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:35.964+0000 7f8463fff700 10 client.?.objecter ms_handle_connect 0x7f847c107890 2024-04-14T15:09:35.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:35.964+0000 7f8463fff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:09:35.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:35.964+0000 7f8463fff700 10 client.?.objecter ms_handle_connect 0x7f847c10e310 2024-04-14T15:09:35.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:35.964+0000 7f8463fff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:09:35.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:35.964+0000 7f8463fff700 1 -- 172.21.15.90:0/434711940 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 407500031 0 0) 0x7f847c1a8f40 con 0x7f847c107890 2024-04-14T15:09:35.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:35.964+0000 7f8463fff700 1 -- 172.21.15.90:0/434711940 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f84640031f0 con 0x7f847c107890 2024-04-14T15:09:35.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:35.964+0000 7f8463fff700 1 -- 172.21.15.90:0/434711940 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2786079479 0 0) 0x7f847c1a6ae0 con 0x7f847c10e310 2024-04-14T15:09:35.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:35.964+0000 7f8463fff700 1 -- 172.21.15.90:0/434711940 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f847c1a8f40 con 0x7f847c10e310 2024-04-14T15:09:35.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:35.964+0000 7f8463fff700 10 client.?.objecter ms_handle_connect 0x7f847c10a740 2024-04-14T15:09:35.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:35.964+0000 7f8463fff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:09:35.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:35.964+0000 7f8463fff700 1 -- 172.21.15.90:0/434711940 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2283139178 0 0) 0x7f847c1a7d00 con 0x7f847c10a740 2024-04-14T15:09:35.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:35.964+0000 7f8463fff700 1 -- 172.21.15.90:0/434711940 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f847c1a6ae0 con 0x7f847c10a740 2024-04-14T15:09:35.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:35.964+0000 7f8463fff700 1 -- 172.21.15.90:0/434711940 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1058045041 0 0) 0x7f84640031f0 con 0x7f847c107890 2024-04-14T15:09:35.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:35.964+0000 7f8463fff700 1 -- 172.21.15.90:0/434711940 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f847c1a7d00 con 0x7f847c107890 2024-04-14T15:09:35.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:35.964+0000 7f8463fff700 1 -- 172.21.15.90:0/434711940 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f846c0018b0 con 0x7f847c107890 2024-04-14T15:09:35.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:35.964+0000 7f8463fff700 1 -- 172.21.15.90:0/434711940 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3912075479 0 0) 0x7f847c1a8f40 con 0x7f847c10e310 2024-04-14T15:09:35.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:35.964+0000 7f8463fff700 1 -- 172.21.15.90:0/434711940 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f84640031f0 con 0x7f847c10e310 2024-04-14T15:09:35.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:35.964+0000 7f8463fff700 1 -- 172.21.15.90:0/434711940 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f8474001c10 con 0x7f847c10e310 2024-04-14T15:09:35.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:35.964+0000 7f8463fff700 1 -- 172.21.15.90:0/434711940 <== mon.2 v1:172.21.15.90:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 213245552 0 0) 0x7f847c1a7d00 con 0x7f847c107890 2024-04-14T15:09:35.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:35.964+0000 7f8463fff700 1 -- 172.21.15.90:0/434711940 >> v1:172.21.15.92:6789/0 conn(0x7f847c10a740 legacy=0x7f847c1a1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:09:35.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:35.964+0000 7f8463fff700 1 -- 172.21.15.90:0/434711940 >> v1:172.21.15.90:6789/0 conn(0x7f847c10e310 legacy=0x7f847c1a51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:09:35.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:35.964+0000 7f8463fff700 1 -- 172.21.15.90:0/434711940 --> v1:172.21.15.90:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f847c1aa180 con 0x7f847c107890 2024-04-14T15:09:35.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:35.964+0000 7f8481428700 1 -- 172.21.15.90:0/434711940 --> v1:172.21.15.90:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f847c1a6d10 con 0x7f847c107890 2024-04-14T15:09:35.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:35.965+0000 7f8481428700 10 client.24326.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:09:35.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:35.965+0000 7f8481428700 1 -- 172.21.15.90:0/434711940 --> v1:172.21.15.90:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f847c1a71d0 con 0x7f847c107890 2024-04-14T15:09:35.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:35.965+0000 7f8481428700 1 librados: init done 2024-04-14T15:09:35.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:35.965+0000 7f8463fff700 1 -- 172.21.15.90:0/434711940 <== mon.2 v1:172.21.15.90:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f846c003b40 con 0x7f847c107890 2024-04-14T15:09:35.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:35.965+0000 7f8463fff700 1 -- 172.21.15.90:0/434711940 <== mon.2 v1:172.21.15.90:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f846c005ca0 con 0x7f847c107890 2024-04-14T15:09:35.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:35.966+0000 7f8481428700 1 -- 172.21.15.90:0/434711940 --> v1:172.21.15.90:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8468005120 con 0x7f847c107890 2024-04-14T15:09:35.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:35.966+0000 7f8463fff700 1 -- 172.21.15.90:0/434711940 <== mon.2 v1:172.21.15.90:6790/0 7 ==== mgrmap(e 15) ==== 92623+0+0 (unknown 2343906571 0 0) 0x7f846c01c900 con 0x7f847c107890 2024-04-14T15:09:35.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:35.966+0000 7f8463fff700 1 -- 172.21.15.90:0/434711940 <== mon.2 v1:172.21.15.90:6790/0 8 ==== osd_map(62..62 src has 12..62) ==== 4081+0+0 (unknown 3457423984 0 0) 0x7f846c089b30 con 0x7f847c107890 2024-04-14T15:09:35.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:35.966+0000 7f8463fff700 10 client.24326.objecter ms_dispatch 0x7f847c001550 osd_map(62..62 src has 12..62) 2024-04-14T15:09:35.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:35.966+0000 7f8463fff700 3 client.24326.objecter handle_osd_map got epochs [62,62] > 0 2024-04-14T15:09:35.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:35.966+0000 7f8463fff700 3 client.24326.objecter handle_osd_map decoding full epoch 62 2024-04-14T15:09:35.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:35.966+0000 7f8463fff700 20 client.24326.objecter dump_active .. 0 homeless 2024-04-14T15:09:35.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:35.967+0000 7f8463fff700 10 client.24326.objecter ms_handle_connect 0x7f8464071290 2024-04-14T15:09:35.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:35.971+0000 7f8463fff700 1 -- 172.21.15.90:0/434711940 <== mon.2 v1:172.21.15.90:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f846c057590 con 0x7f847c107890 2024-04-14T15:09:36.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:36.169+0000 7f8481428700 1 -- 172.21.15.90:0/434711940 --> v1:172.21.15.90:6790/0 -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8468005c20 con 0x7f847c107890 2024-04-14T15:09:36.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:36.169+0000 7f8463fff700 1 -- 172.21.15.90:0/434711940 <== mon.2 v1:172.21.15.90:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v62) ==== 93+0+10 (unknown 3447137061 0 2170607528) 0x7f846c056e90 con 0x7f847c107890 2024-04-14T15:09:36.170 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-04-14T15:09:36.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:36.171+0000 7f8481428700 10 librados: watch_flush enter 2024-04-14T15:09:36.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:36.171+0000 7f8481428700 10 librados: watch_flush exit 2024-04-14T15:09:36.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:36.171+0000 7f8481428700 20 client.24326.objecter shutdown clearing up homeless session... 2024-04-14T15:09:36.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:36.171+0000 7f8481428700 10 client.24326.objecter successfully canceled tick 2024-04-14T15:09:36.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:36.171+0000 7f8481428700 1 -- 172.21.15.90:0/434711940 >> v1:172.21.15.90:6800/491071118 conn(0x7f8464071290 legacy=0x7f8464073750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:09:36.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:36.171+0000 7f8481428700 1 -- 172.21.15.90:0/434711940 >> v1:172.21.15.90:6790/0 conn(0x7f847c107890 legacy=0x7f847c10dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:09:36.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:36.171+0000 7f8481428700 1 -- 172.21.15.90:0/434711940 shutdown_connections 2024-04-14T15:09:36.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:36.171+0000 7f8481428700 1 -- 172.21.15.90:0/434711940 >> 172.21.15.90:0/434711940 conn(0x7f847c0fd290 msgr2=0x7f847c0fed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:09:36.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:36.171+0000 7f8481428700 1 -- 172.21.15.90:0/434711940 shutdown_connections 2024-04-14T15:09:36.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:36.171+0000 7f8481428700 1 -- 172.21.15.90:0/434711940 wait complete. 2024-04-14T15:09:36.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:09:36.171+0000 7f8481428700 1 librados: shutdown 2024-04-14T15:09:36.758 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi090 2024-04-14T15:09:36.759 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph orch apply rgw foo.a --placement '1;smithi090=foo.a' 2024-04-14T15:09:37.018 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:09:37.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:36 smithi092 ceph-mon[33966]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-04-14T15:09:37.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:36 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/434711940' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-14T15:09:37.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:36 smithi090 ceph-mon[30399]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-04-14T15:09:37.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:36 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/434711940' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-14T15:09:37.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:36 smithi090 ceph-mon[41670]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-04-14T15:09:37.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:36 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/434711940' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-14T15:09:38.723 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.720+0000 7f1958062700 1 -- 172.21.15.92:0/487782957 >> v1:172.21.15.92:6789/0 conn(0x7f1950104cc0 legacy=0x7f195010ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:09:38.798 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.720+0000 7f1958062700 1 -- 172.21.15.92:0/487782957 shutdown_connections 2024-04-14T15:09:38.798 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.720+0000 7f1958062700 1 -- 172.21.15.92:0/487782957 >> 172.21.15.92:0/487782957 conn(0x7f19500fcd90 msgr2=0x7f19500ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:09:38.798 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.720+0000 7f1958062700 1 -- 172.21.15.92:0/487782957 shutdown_connections 2024-04-14T15:09:38.798 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.720+0000 7f1958062700 1 -- 172.21.15.92:0/487782957 wait complete. 2024-04-14T15:09:38.798 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.721+0000 7f1958062700 1 librados: starting msgr at 2024-04-14T15:09:38.798 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.721+0000 7f1958062700 1 librados: starting objecter 2024-04-14T15:09:38.798 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.721+0000 7f1958062700 1 Processor -- start 2024-04-14T15:09:38.798 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.721+0000 7f1958062700 1 -- start start 2024-04-14T15:09:38.798 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.721+0000 7f1958062700 1 librados: setting wanted keys 2024-04-14T15:09:38.798 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.721+0000 7f1958062700 1 librados: calling monclient init 2024-04-14T15:09:38.799 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.721+0000 7f1958062700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f195019f070 con 0x7f1950107180 2024-04-14T15:09:38.799 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.721+0000 7f1958062700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f19501aa860 con 0x7f1950104cc0 2024-04-14T15:09:38.799 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.721+0000 7f1958062700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f19501aba40 con 0x7f1950101170 2024-04-14T15:09:38.799 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.721+0000 7f19555fd700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f1950104cc0 0x7f19501a4a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:58062/0 (socket says 172.21.15.92:58062) 2024-04-14T15:09:38.799 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.721+0000 7f19555fd700 1 -- 172.21.15.92:0/1255812233 learned_addr learned my addr 172.21.15.92:0/1255812233 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-04-14T15:09:38.799 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.721+0000 7f1946ffd700 10 client.?.objecter ms_handle_connect 0x7f1950104cc0 2024-04-14T15:09:38.799 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.721+0000 7f1946ffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:09:38.799 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.721+0000 7f1946ffd700 10 client.?.objecter ms_handle_connect 0x7f1950101170 2024-04-14T15:09:38.799 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.721+0000 7f1946ffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:09:38.799 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.721+0000 7f1946ffd700 10 client.?.objecter ms_handle_connect 0x7f1950107180 2024-04-14T15:09:38.799 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.721+0000 7f1946ffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:09:38.799 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.721+0000 7f1946ffd700 1 -- 172.21.15.92:0/1255812233 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 951747374 0 0) 0x7f19501aa860 con 0x7f1950104cc0 2024-04-14T15:09:38.799 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.721+0000 7f1946ffd700 1 -- 172.21.15.92:0/1255812233 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f193c0035b0 con 0x7f1950104cc0 2024-04-14T15:09:38.799 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.721+0000 7f1946ffd700 1 -- 172.21.15.92:0/1255812233 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 61071829 0 0) 0x7f19501aba40 con 0x7f1950101170 2024-04-14T15:09:38.800 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.721+0000 7f1946ffd700 1 -- 172.21.15.92:0/1255812233 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f19501aa860 con 0x7f1950101170 2024-04-14T15:09:38.800 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.721+0000 7f1946ffd700 1 -- 172.21.15.92:0/1255812233 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2370990556 0 0) 0x7f195019f070 con 0x7f1950107180 2024-04-14T15:09:38.800 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.721+0000 7f1946ffd700 1 -- 172.21.15.92:0/1255812233 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f19501aba40 con 0x7f1950107180 2024-04-14T15:09:38.800 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.722+0000 7f1946ffd700 1 -- 172.21.15.92:0/1255812233 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3790000085 0 0) 0x7f193c0035b0 con 0x7f1950104cc0 2024-04-14T15:09:38.800 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.722+0000 7f1946ffd700 1 -- 172.21.15.92:0/1255812233 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f195019f070 con 0x7f1950104cc0 2024-04-14T15:09:38.800 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.722+0000 7f1946ffd700 1 -- 172.21.15.92:0/1255812233 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f1940001430 con 0x7f1950104cc0 2024-04-14T15:09:38.800 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.722+0000 7f1946ffd700 1 -- 172.21.15.92:0/1255812233 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3429829322 0 0) 0x7f19501aa860 con 0x7f1950101170 2024-04-14T15:09:38.800 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.722+0000 7f1946ffd700 1 -- 172.21.15.92:0/1255812233 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f193c0035b0 con 0x7f1950101170 2024-04-14T15:09:38.800 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.722+0000 7f1946ffd700 1 -- 172.21.15.92:0/1255812233 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f194c001830 con 0x7f1950101170 2024-04-14T15:09:38.800 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.722+0000 7f1946ffd700 1 -- 172.21.15.92:0/1255812233 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1349428013 0 0) 0x7f19501aba40 con 0x7f1950107180 2024-04-14T15:09:38.800 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.722+0000 7f1946ffd700 1 -- 172.21.15.92:0/1255812233 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f19501aa860 con 0x7f1950107180 2024-04-14T15:09:38.800 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.722+0000 7f1946ffd700 1 -- 172.21.15.92:0/1255812233 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f1948001ae0 con 0x7f1950107180 2024-04-14T15:09:38.800 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.722+0000 7f1946ffd700 1 -- 172.21.15.92:0/1255812233 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2838307180 0 0) 0x7f195019f070 con 0x7f1950104cc0 2024-04-14T15:09:38.801 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.722+0000 7f1946ffd700 1 -- 172.21.15.92:0/1255812233 >> v1:172.21.15.90:6790/0 conn(0x7f1950101170 legacy=0x7f195019e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:09:38.801 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.722+0000 7f1946ffd700 1 -- 172.21.15.92:0/1255812233 >> v1:172.21.15.90:6789/0 conn(0x7f1950107180 legacy=0x7f19501a8130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:09:38.801 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.722+0000 7f1946ffd700 1 -- 172.21.15.92:0/1255812233 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19501acc60 con 0x7f1950104cc0 2024-04-14T15:09:38.801 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.722+0000 7f1958062700 1 -- 172.21.15.92:0/1255812233 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f19501abc70 con 0x7f1950104cc0 2024-04-14T15:09:38.801 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.722+0000 7f1958062700 10 client.24349.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:09:38.801 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.722+0000 7f1958062700 1 -- 172.21.15.92:0/1255812233 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f19501ac1b0 con 0x7f1950104cc0 2024-04-14T15:09:38.801 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.722+0000 7f1958062700 1 librados: init done 2024-04-14T15:09:38.801 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.722+0000 7f1946ffd700 1 -- 172.21.15.92:0/1255812233 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f1940001720 con 0x7f1950104cc0 2024-04-14T15:09:38.801 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.722+0000 7f1946ffd700 1 -- 172.21.15.92:0/1255812233 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f1940004e80 con 0x7f1950104cc0 2024-04-14T15:09:38.801 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.723+0000 7f1958062700 1 -- 172.21.15.92:0/1255812233 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1934005150 con 0x7f1950104cc0 2024-04-14T15:09:38.801 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.723+0000 7f1946ffd700 1 -- 172.21.15.92:0/1255812233 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 15) ==== 92623+0+0 (unknown 2343906571 0 0) 0x7f19400050e0 con 0x7f1950104cc0 2024-04-14T15:09:38.802 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.724+0000 7f1946ffd700 1 -- 172.21.15.92:0/1255812233 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(62..62 src has 12..62) ==== 4081+0+0 (unknown 3457423984 0 0) 0x7f1940089a50 con 0x7f1950104cc0 2024-04-14T15:09:38.802 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.724+0000 7f1946ffd700 10 client.24349.objecter ms_dispatch 0x7f1950001550 osd_map(62..62 src has 12..62) 2024-04-14T15:09:38.802 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.724+0000 7f1946ffd700 3 client.24349.objecter handle_osd_map got epochs [62,62] > 0 2024-04-14T15:09:38.802 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.724+0000 7f1946ffd700 3 client.24349.objecter handle_osd_map decoding full epoch 62 2024-04-14T15:09:38.803 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.724+0000 7f1946ffd700 20 client.24349.objecter dump_active .. 0 homeless 2024-04-14T15:09:38.803 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.724+0000 7f1946ffd700 10 client.24349.objecter ms_handle_connect 0x7f193c071320 2024-04-14T15:09:38.803 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.729+0000 7f1946ffd700 1 -- 172.21.15.92:0/1255812233 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f1940004e80 con 0x7f1950104cc0 2024-04-14T15:09:38.938 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.938+0000 7f1958062700 1 -- 172.21.15.92:0/1255812233 --> v1:172.21.15.90:6800/491071118 -- mgr_command(tid 0: {"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi090=foo.a", "target": ["mon-mgr", ""]}) -- 0x7f1934000bc0 con 0x7f193c071320 2024-04-14T15:09:38.945 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.945+0000 7f1946ffd700 1 -- 172.21.15.92:0/1255812233 <== mgr.14148 v1:172.21.15.90:6800/491071118 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+30 (unknown 0 0 1123153589) 0x7f1934000bc0 con 0x7f193c071320 2024-04-14T15:09:38.945 INFO:teuthology.orchestra.run.smithi092.stdout:Scheduled rgw.foo.a update... 2024-04-14T15:09:38.947 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.948+0000 7f1958062700 10 librados: watch_flush enter 2024-04-14T15:09:38.947 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.948+0000 7f1958062700 10 librados: watch_flush exit 2024-04-14T15:09:38.947 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.948+0000 7f1958062700 20 client.24349.objecter shutdown clearing up homeless session... 2024-04-14T15:09:38.948 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.948+0000 7f1958062700 10 client.24349.objecter successfully canceled tick 2024-04-14T15:09:38.948 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.948+0000 7f1958062700 1 -- 172.21.15.92:0/1255812233 >> v1:172.21.15.90:6800/491071118 conn(0x7f193c071320 legacy=0x7f193c0737e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:09:38.948 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.948+0000 7f1958062700 1 -- 172.21.15.92:0/1255812233 >> v1:172.21.15.92:6789/0 conn(0x7f1950104cc0 legacy=0x7f19501a4a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:09:38.948 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.948+0000 7f1958062700 1 -- 172.21.15.92:0/1255812233 shutdown_connections 2024-04-14T15:09:38.948 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.948+0000 7f1958062700 1 -- 172.21.15.92:0/1255812233 >> 172.21.15.92:0/1255812233 conn(0x7f19500fcd90 msgr2=0x7f1950103560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:09:38.948 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.948+0000 7f1958062700 1 -- 172.21.15.92:0/1255812233 shutdown_connections 2024-04-14T15:09:38.948 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.948+0000 7f1958062700 1 -- 172.21.15.92:0/1255812233 wait complete. 2024-04-14T15:09:38.948 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:38.948+0000 7f1958062700 1 librados: shutdown 2024-04-14T15:09:39.273 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:38 smithi092 ceph-mon[33966]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-04-14T15:09:39.273 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:38 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:39.274 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:38 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:09:39.274 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:38 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:09:39.274 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:38 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:09:39.274 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:38 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:39.274 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:38 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-14T15:09:39.274 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:38 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-14T15:09:39.274 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:38 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:39.274 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:38 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:09:39.436 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:39 smithi090 ceph-mon[41670]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-04-14T15:09:39.436 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:39 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:39.436 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:39 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:09:39.436 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:39 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:09:39.436 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:39 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:09:39.436 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:39 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:39.436 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:39 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-14T15:09:39.436 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:39 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-14T15:09:39.436 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:39 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:39.436 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:39 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:09:39.436 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:38 smithi090 ceph-mon[30399]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-04-14T15:09:39.436 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:38 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:39.437 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:38 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:09:39.437 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:38 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:09:39.437 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:38 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:09:39.437 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:38 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:39.437 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:38 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-14T15:09:39.437 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:38 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-14T15:09:39.437 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:38 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:39.437 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:38 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:09:39.593 DEBUG:teuthology.orchestra.run.smithi090:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-0e288098-fa70-11ee-bc8e-c7b262605968@rgw.foo.a.service 2024-04-14T15:09:39.595 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi092 2024-04-14T15:09:39.595 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd pool create datapool 3 3 replicated 2024-04-14T15:09:39.833 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:09:40.268 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:40 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:09:40.268 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:40 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:09:40.269 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:40 smithi092 ceph-mon[33966]: from='client.24349 v1:172.21.15.92:0/1255812233' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi090=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:09:40.269 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:40 smithi092 ceph-mon[33966]: Saving service rgw.foo.a spec with placement smithi090=foo.a;count:1 2024-04-14T15:09:40.269 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:40 smithi092 ceph-mon[33966]: Deploying daemon rgw.foo.a on smithi090 2024-04-14T15:09:40.269 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:40 smithi092 ceph-mon[33966]: osdmap e63: 8 total, 8 up, 8 in 2024-04-14T15:09:40.306 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:40 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:09:40.306 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:40 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:09:40.306 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:40 smithi090 ceph-mon[41670]: from='client.24349 v1:172.21.15.92:0/1255812233' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi090=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:09:40.306 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:40 smithi090 ceph-mon[41670]: Saving service rgw.foo.a spec with placement smithi090=foo.a;count:1 2024-04-14T15:09:40.307 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:40 smithi090 ceph-mon[41670]: Deploying daemon rgw.foo.a on smithi090 2024-04-14T15:09:40.307 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:40 smithi090 ceph-mon[41670]: osdmap e63: 8 total, 8 up, 8 in 2024-04-14T15:09:40.307 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:40 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:09:40.307 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:40 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:09:40.307 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:40 smithi090 ceph-mon[30399]: from='client.24349 v1:172.21.15.92:0/1255812233' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi090=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:09:40.307 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:40 smithi090 ceph-mon[30399]: Saving service rgw.foo.a spec with placement smithi090=foo.a;count:1 2024-04-14T15:09:40.307 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:40 smithi090 ceph-mon[30399]: Deploying daemon rgw.foo.a on smithi090 2024-04-14T15:09:40.307 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:40 smithi090 ceph-mon[30399]: osdmap e63: 8 total, 8 up, 8 in 2024-04-14T15:09:41.291 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:41 smithi092 ceph-mon[33966]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:09:41.337 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:41 smithi090 ceph-mon[41670]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:09:41.337 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:41 smithi090 ceph-mon[30399]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:09:41.338 INFO:journalctl@ceph.rgw.foo.a.smithi090.stdout:Apr 14 15:09:41 smithi090 systemd[1]: Starting Ceph rgw.foo.a for 0e288098-fa70-11ee-bc8e-c7b262605968... 2024-04-14T15:09:41.534 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:41.531+0000 7f639acfd700 1 -- 172.21.15.92:0/3256370633 >> v1:172.21.15.92:6789/0 conn(0x7f6394105d00 legacy=0x7f63941080f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:09:41.534 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:41.531+0000 7f639acfd700 1 -- 172.21.15.92:0/3256370633 shutdown_connections 2024-04-14T15:09:41.534 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:41.531+0000 7f639acfd700 1 -- 172.21.15.92:0/3256370633 >> 172.21.15.92:0/3256370633 conn(0x7f63940fc550 msgr2=0x7f63940fe970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:09:41.534 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:41.531+0000 7f639acfd700 1 -- 172.21.15.92:0/3256370633 shutdown_connections 2024-04-14T15:09:41.535 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:41.532+0000 7f639acfd700 1 -- 172.21.15.92:0/3256370633 wait complete. 2024-04-14T15:09:41.535 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:41.532+0000 7f639acfd700 1 librados: starting msgr at 2024-04-14T15:09:41.535 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:41.532+0000 7f639acfd700 1 librados: starting objecter 2024-04-14T15:09:41.535 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:41.532+0000 7f639acfd700 1 Processor -- start 2024-04-14T15:09:41.535 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:41.532+0000 7f639acfd700 1 -- start start 2024-04-14T15:09:41.535 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:41.532+0000 7f639acfd700 1 librados: setting wanted keys 2024-04-14T15:09:41.535 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:41.532+0000 7f639acfd700 1 librados: calling monclient init 2024-04-14T15:09:41.535 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:41.532+0000 7f639acfd700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6394101670 con 0x7f6394102d80 2024-04-14T15:09:41.535 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:41.532+0000 7f639acfd700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f63941018a0 con 0x7f6394109930 2024-04-14T15:09:41.535 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:41.532+0000 7f639acfd700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f63941a76f0 con 0x7f6394105d00 2024-04-14T15:09:41.535 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:41.532+0000 7f639929a700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f6394109930 0x7f63941a3f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:58082/0 (socket says 172.21.15.92:58082) 2024-04-14T15:09:41.535 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:41.532+0000 7f639929a700 1 -- 172.21.15.92:0/2673024752 learned_addr learned my addr 172.21.15.92:0/2673024752 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-04-14T15:09:41.535 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:41.532+0000 7f6391ffb700 10 client.?.objecter ms_handle_connect 0x7f6394109930 2024-04-14T15:09:41.535 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:41.532+0000 7f6391ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:09:41.535 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:41.532+0000 7f6391ffb700 10 client.?.objecter ms_handle_connect 0x7f6394102d80 2024-04-14T15:09:41.536 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:41.532+0000 7f6391ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:09:41.536 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:41.532+0000 7f6391ffb700 10 client.?.objecter ms_handle_connect 0x7f6394105d00 2024-04-14T15:09:41.536 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:41.532+0000 7f6391ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:09:41.536 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:41.533+0000 7f6391ffb700 1 -- 172.21.15.92:0/2673024752 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3684331782 0 0) 0x7f6394101670 con 0x7f6394102d80 2024-04-14T15:09:41.536 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:41.533+0000 7f6391ffb700 1 -- 172.21.15.92:0/2673024752 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f63840035b0 con 0x7f6394102d80 2024-04-14T15:09:41.536 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:41.533+0000 7f6391ffb700 1 -- 172.21.15.92:0/2673024752 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 379510071 0 0) 0x7f63941018a0 con 0x7f6394109930 2024-04-14T15:09:41.536 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:41.533+0000 7f6391ffb700 1 -- 172.21.15.92:0/2673024752 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6394101670 con 0x7f6394109930 2024-04-14T15:09:41.536 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:41.533+0000 7f6391ffb700 1 -- 172.21.15.92:0/2673024752 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 535646566 0 0) 0x7f63941a76f0 con 0x7f6394105d00 2024-04-14T15:09:41.536 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:41.533+0000 7f6391ffb700 1 -- 172.21.15.92:0/2673024752 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f63941018a0 con 0x7f6394105d00 2024-04-14T15:09:41.536 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:41.533+0000 7f6391ffb700 1 -- 172.21.15.92:0/2673024752 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2279980204 0 0) 0x7f63840035b0 con 0x7f6394102d80 2024-04-14T15:09:41.536 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:41.533+0000 7f6391ffb700 1 -- 172.21.15.92:0/2673024752 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f63941a76f0 con 0x7f6394102d80 2024-04-14T15:09:41.536 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:41.533+0000 7f6391ffb700 1 -- 172.21.15.92:0/2673024752 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f6388001830 con 0x7f6394102d80 2024-04-14T15:09:41.536 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:41.533+0000 7f6391ffb700 1 -- 172.21.15.92:0/2673024752 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 344314538 0 0) 0x7f63941018a0 con 0x7f6394105d00 2024-04-14T15:09:41.536 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:41.533+0000 7f6391ffb700 1 -- 172.21.15.92:0/2673024752 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f63840035b0 con 0x7f6394105d00 2024-04-14T15:09:41.536 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:41.533+0000 7f6391ffb700 1 -- 172.21.15.92:0/2673024752 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f6380001370 con 0x7f6394105d00 2024-04-14T15:09:41.537 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:41.533+0000 7f6391ffb700 1 -- 172.21.15.92:0/2673024752 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 809200961 0 0) 0x7f6394101670 con 0x7f6394109930 2024-04-14T15:09:41.537 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:41.533+0000 7f6391ffb700 1 -- 172.21.15.92:0/2673024752 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f63941018a0 con 0x7f6394109930 2024-04-14T15:09:41.537 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:41.533+0000 7f6391ffb700 1 -- 172.21.15.92:0/2673024752 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f638c001bf0 con 0x7f6394109930 2024-04-14T15:09:41.537 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:41.533+0000 7f6391ffb700 1 -- 172.21.15.92:0/2673024752 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 235167884 0 0) 0x7f63941a76f0 con 0x7f6394102d80 2024-04-14T15:09:41.537 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:41.533+0000 7f6391ffb700 1 -- 172.21.15.92:0/2673024752 >> v1:172.21.15.90:6790/0 conn(0x7f6394105d00 legacy=0x7f63941a0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:09:41.537 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:41.533+0000 7f6391ffb700 1 -- 172.21.15.92:0/2673024752 >> v1:172.21.15.92:6789/0 conn(0x7f6394109930 legacy=0x7f63941a3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:09:41.537 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:41.533+0000 7f6391ffb700 1 -- 172.21.15.92:0/2673024752 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63941a8910 con 0x7f6394102d80 2024-04-14T15:09:41.537 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:41.533+0000 7f639acfd700 1 -- 172.21.15.92:0/2673024752 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f63941a7920 con 0x7f6394102d80 2024-04-14T15:09:41.537 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:41.533+0000 7f6391ffb700 1 -- 172.21.15.92:0/2673024752 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f6388003b40 con 0x7f6394102d80 2024-04-14T15:09:41.537 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:41.533+0000 7f6391ffb700 1 -- 172.21.15.92:0/2673024752 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f6388004b00 con 0x7f6394102d80 2024-04-14T15:09:41.537 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:41.533+0000 7f639acfd700 10 client.14451.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:09:41.537 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:41.533+0000 7f639acfd700 1 -- 172.21.15.92:0/2673024752 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f63941a7e60 con 0x7f6394102d80 2024-04-14T15:09:41.537 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:41.533+0000 7f639acfd700 1 librados: init done 2024-04-14T15:09:41.537 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:41.535+0000 7f639acfd700 1 -- 172.21.15.92:0/2673024752 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63940693d0 con 0x7f6394102d80 2024-04-14T15:09:41.537 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:41.535+0000 7f6391ffb700 1 -- 172.21.15.92:0/2673024752 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 15) ==== 92623+0+0 (unknown 2343906571 0 0) 0x7f63880040e0 con 0x7f6394102d80 2024-04-14T15:09:41.538 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:41.535+0000 7f6391ffb700 1 -- 172.21.15.92:0/2673024752 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(63..63 src has 12..63) ==== 4081+0+0 (unknown 2918512782 0 0) 0x7f6388088710 con 0x7f6394102d80 2024-04-14T15:09:41.538 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:41.535+0000 7f6391ffb700 10 client.14451.objecter ms_dispatch 0x7f6394001550 osd_map(63..63 src has 12..63) 2024-04-14T15:09:41.538 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:41.535+0000 7f6391ffb700 3 client.14451.objecter handle_osd_map got epochs [63,63] > 0 2024-04-14T15:09:41.538 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:41.535+0000 7f6391ffb700 3 client.14451.objecter handle_osd_map decoding full epoch 63 2024-04-14T15:09:41.538 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:41.535+0000 7f6391ffb700 20 client.14451.objecter dump_active .. 0 homeless 2024-04-14T15:09:41.538 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:41.535+0000 7f6391ffb700 10 client.14451.objecter ms_handle_connect 0x7f63840711e0 2024-04-14T15:09:41.539 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:41.540+0000 7f6391ffb700 1 -- 172.21.15.92:0/2673024752 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f6388056170 con 0x7f6394102d80 2024-04-14T15:09:41.640 INFO:journalctl@ceph.rgw.foo.a.smithi090.stdout:Apr 14 15:09:41 smithi090 podman[70038]: 2024-04-14 15:09:41.337711904 +0000 UTC m=+0.103975150 container create fc3f1f3469ebe7f93ba3030a48e39b911fa0f7188cea2aa30a085198b9a7c5f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-rgw-foo-a, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=main-a971fd5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-14T15:09:41.640 INFO:journalctl@ceph.rgw.foo.a.smithi090.stdout:Apr 14 15:09:41 smithi090 podman[70038]: 2024-04-14 15:09:41.251485943 +0000 UTC m=+0.017749205 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608 2024-04-14T15:09:41.640 INFO:journalctl@ceph.rgw.foo.a.smithi090.stdout:Apr 14 15:09:41 smithi090 podman[70038]: 2024-04-14 15:09:41.565385273 +0000 UTC m=+0.331648519 container init fc3f1f3469ebe7f93ba3030a48e39b911fa0f7188cea2aa30a085198b9a7c5f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-rgw-foo-a, org.label-schema.build-date=20240403, RELEASE=main-a971fd5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-14T15:09:41.640 INFO:journalctl@ceph.rgw.foo.a.smithi090.stdout:Apr 14 15:09:41 smithi090 podman[70038]: 2024-04-14 15:09:41.571059983 +0000 UTC m=+0.337323230 container start fc3f1f3469ebe7f93ba3030a48e39b911fa0f7188cea2aa30a085198b9a7c5f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-rgw-foo-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=main-a971fd5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-14T15:09:41.640 INFO:journalctl@ceph.rgw.foo.a.smithi090.stdout:Apr 14 15:09:41 smithi090 bash[70038]: fc3f1f3469ebe7f93ba3030a48e39b911fa0f7188cea2aa30a085198b9a7c5f6 2024-04-14T15:09:41.740 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:41.740+0000 7f639acfd700 1 -- 172.21.15.92:0/2673024752 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} v 0) -- 0x7f639410eca0 con 0x7f6394102d80 2024-04-14T15:09:41.942 INFO:journalctl@ceph.rgw.foo.a.smithi090.stdout:Apr 14 15:09:41 smithi090 systemd[1]: Started Ceph rgw.foo.a for 0e288098-fa70-11ee-bc8e-c7b262605968. 2024-04-14T15:09:42.005 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:42.006+0000 7f6391ffb700 1 -- 172.21.15.92:0/2673024752 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]=0 pool 'datapool' created v64) ==== 160+0+0 (unknown 2423491992 0 0) 0x7f6388055a70 con 0x7f6394102d80 2024-04-14T15:09:42.009 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:42.009+0000 7f639acfd700 10 librados: watch_flush enter 2024-04-14T15:09:42.009 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:42.009+0000 7f639acfd700 10 librados: watch_flush exit 2024-04-14T15:09:42.009 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:42.009+0000 7f639acfd700 20 client.14451.objecter shutdown clearing up homeless session... 2024-04-14T15:09:42.009 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:42.009+0000 7f639acfd700 10 client.14451.objecter successfully canceled tick 2024-04-14T15:09:42.009 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:42.009+0000 7f639acfd700 1 -- 172.21.15.92:0/2673024752 >> v1:172.21.15.90:6800/491071118 conn(0x7f63840711e0 legacy=0x7f63840736a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:09:42.009 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:42.009+0000 7f639acfd700 1 -- 172.21.15.92:0/2673024752 >> v1:172.21.15.90:6789/0 conn(0x7f6394102d80 legacy=0x7f639419d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:09:42.011 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:42.011+0000 7f639acfd700 1 -- 172.21.15.92:0/2673024752 shutdown_connections 2024-04-14T15:09:42.011 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:42.011+0000 7f639acfd700 1 -- 172.21.15.92:0/2673024752 >> 172.21.15.92:0/2673024752 conn(0x7f63940fc550 msgr2=0x7f63940fe940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:09:42.012 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:42.012+0000 7f639acfd700 1 -- 172.21.15.92:0/2673024752 shutdown_connections 2024-04-14T15:09:42.012 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:42.012+0000 7f639acfd700 1 -- 172.21.15.92:0/2673024752 wait complete. 2024-04-14T15:09:42.012 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:42.012+0000 7f639acfd700 1 librados: shutdown 2024-04-14T15:09:42.013 INFO:teuthology.orchestra.run.smithi092.stderr:pool 'datapool' created 2024-04-14T15:09:42.728 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:42 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:42.728 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:42 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:42.728 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:42 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:42.728 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:42 smithi092 ceph-mon[33966]: Saving service rgw.foo.a spec with placement smithi090=foo.a;count:1 2024-04-14T15:09:42.728 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:42 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:42.728 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:42 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:42.728 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:42 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.92:0/2673024752' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-04-14T15:09:42.728 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:42 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:09:42.729 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:42 smithi092 ceph-mon[33966]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:09:42.729 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:42 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.92:0/2673024752' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-04-14T15:09:42.729 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:42 smithi092 ceph-mon[33966]: osdmap e64: 8 total, 8 up, 8 in 2024-04-14T15:09:42.729 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:42 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2555498183' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-14T15:09:42.760 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- rbd pool init datapool 2024-04-14T15:09:42.941 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:42 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:42.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:42 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:42.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:42 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:42.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:42 smithi090 ceph-mon[30399]: Saving service rgw.foo.a spec with placement smithi090=foo.a;count:1 2024-04-14T15:09:42.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:42 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:42.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:42 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:42.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:42 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.92:0/2673024752' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-04-14T15:09:42.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:42 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:09:42.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:42 smithi090 ceph-mon[30399]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:09:42.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:42 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.92:0/2673024752' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-04-14T15:09:42.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:42 smithi090 ceph-mon[30399]: osdmap e64: 8 total, 8 up, 8 in 2024-04-14T15:09:42.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:42 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2555498183' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-14T15:09:42.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:42 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:42.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:42 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:42.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:42 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:42.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:42 smithi090 ceph-mon[41670]: Saving service rgw.foo.a spec with placement smithi090=foo.a;count:1 2024-04-14T15:09:42.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:42 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:42.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:42 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:42.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:42 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.92:0/2673024752' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-04-14T15:09:42.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:42 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:09:42.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:42 smithi090 ceph-mon[41670]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:09:42.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:42 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.92:0/2673024752' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-04-14T15:09:42.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:42 smithi090 ceph-mon[41670]: osdmap e64: 8 total, 8 up, 8 in 2024-04-14T15:09:42.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:42 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2555498183' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-14T15:09:42.998 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:09:44.159 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:44 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2555498183' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-04-14T15:09:44.159 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:44 smithi092 ceph-mon[33966]: osdmap e65: 8 total, 8 up, 8 in 2024-04-14T15:09:44.159 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:44 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:44.160 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:44 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:44.160 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:44 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:09:44.160 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:44 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:09:44.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:44 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2555498183' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-04-14T15:09:44.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:44 smithi090 ceph-mon[30399]: osdmap e65: 8 total, 8 up, 8 in 2024-04-14T15:09:44.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:44 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:44.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:44 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:44.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:44 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:09:44.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:44 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:09:44.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:44 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2555498183' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-04-14T15:09:44.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:44 smithi090 ceph-mon[41670]: osdmap e65: 8 total, 8 up, 8 in 2024-04-14T15:09:44.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:44 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:44.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:44 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:44.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:44 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:09:44.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:44 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:09:45.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:45 smithi092 ceph-mon[33966]: Checking dashboard <-> RGW credentials 2024-04-14T15:09:45.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:45 smithi092 ceph-mon[33966]: pgmap v194: 36 pgs: 8 creating+peering, 11 active+clean, 17 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:09:45.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:45 smithi092 ceph-mon[33966]: osdmap e66: 8 total, 8 up, 8 in 2024-04-14T15:09:45.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:45 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2924083629' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-04-14T15:09:45.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:45 smithi092 ceph-mon[33966]: from='client.24373 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-04-14T15:09:45.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:45 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2632170366' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-04-14T15:09:45.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:45 smithi092 ceph-mon[33966]: from='client.24383 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-04-14T15:09:45.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:45 smithi092 ceph-mon[33966]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:09:45.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:45 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.92:0/2365795386' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-04-14T15:09:45.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:45 smithi092 ceph-mon[33966]: from='client.24406 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-04-14T15:09:45.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:45 smithi090 ceph-mon[30399]: Checking dashboard <-> RGW credentials 2024-04-14T15:09:45.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:45 smithi090 ceph-mon[30399]: pgmap v194: 36 pgs: 8 creating+peering, 11 active+clean, 17 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:09:45.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:45 smithi090 ceph-mon[30399]: osdmap e66: 8 total, 8 up, 8 in 2024-04-14T15:09:45.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:45 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2924083629' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-04-14T15:09:45.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:45 smithi090 ceph-mon[30399]: from='client.24373 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-04-14T15:09:45.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:45 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2632170366' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-04-14T15:09:45.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:45 smithi090 ceph-mon[30399]: from='client.24383 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-04-14T15:09:45.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:45 smithi090 ceph-mon[30399]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:09:45.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:45 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.92:0/2365795386' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-04-14T15:09:45.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:45 smithi090 ceph-mon[30399]: from='client.24406 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-04-14T15:09:45.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:45 smithi090 ceph-mon[41670]: Checking dashboard <-> RGW credentials 2024-04-14T15:09:45.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:45 smithi090 ceph-mon[41670]: pgmap v194: 36 pgs: 8 creating+peering, 11 active+clean, 17 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:09:45.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:45 smithi090 ceph-mon[41670]: osdmap e66: 8 total, 8 up, 8 in 2024-04-14T15:09:45.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:45 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2924083629' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-04-14T15:09:45.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:45 smithi090 ceph-mon[41670]: from='client.24373 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-04-14T15:09:45.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:45 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2632170366' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-04-14T15:09:45.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:45 smithi090 ceph-mon[41670]: from='client.24383 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-04-14T15:09:45.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:45 smithi090 ceph-mon[41670]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:09:45.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:45 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.92:0/2365795386' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-04-14T15:09:45.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:45 smithi090 ceph-mon[41670]: from='client.24406 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-04-14T15:09:46.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:46 smithi092 ceph-mon[33966]: from='client.24373 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-04-14T15:09:46.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:46 smithi092 ceph-mon[33966]: from='client.24383 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-04-14T15:09:46.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:46 smithi092 ceph-mon[33966]: from='client.24406 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-04-14T15:09:46.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:46 smithi092 ceph-mon[33966]: osdmap e67: 8 total, 8 up, 8 in 2024-04-14T15:09:46.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:46 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:46.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:46 smithi090 ceph-mon[30399]: from='client.24373 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-04-14T15:09:46.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:46 smithi090 ceph-mon[30399]: from='client.24383 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-04-14T15:09:46.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:46 smithi090 ceph-mon[30399]: from='client.24406 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-04-14T15:09:46.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:46 smithi090 ceph-mon[30399]: osdmap e67: 8 total, 8 up, 8 in 2024-04-14T15:09:46.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:46 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:46.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:46 smithi090 ceph-mon[41670]: from='client.24373 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-04-14T15:09:46.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:46 smithi090 ceph-mon[41670]: from='client.24383 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-04-14T15:09:46.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:46 smithi090 ceph-mon[41670]: from='client.24406 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-04-14T15:09:46.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:46 smithi090 ceph-mon[41670]: osdmap e67: 8 total, 8 up, 8 in 2024-04-14T15:09:46.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:46 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:47.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:47 smithi092 ceph-mon[33966]: pgmap v197: 68 pgs: 9 creating+peering, 25 active+clean, 34 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 511 B/s wr, 1 op/s 2024-04-14T15:09:47.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:47 smithi092 ceph-mon[33966]: osdmap e68: 8 total, 8 up, 8 in 2024-04-14T15:09:47.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:47 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2924083629' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-04-14T15:09:47.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:47 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2632170366' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-04-14T15:09:47.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:47 smithi092 ceph-mon[33966]: from='client.24373 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-04-14T15:09:47.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:47 smithi092 ceph-mon[33966]: from='client.24383 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-04-14T15:09:47.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:47 smithi092 ceph-mon[33966]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-14T15:09:47.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:47 smithi092 ceph-mon[33966]: Cluster is now healthy 2024-04-14T15:09:47.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:47 smithi092 ceph-mon[33966]: from='client.24373 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-04-14T15:09:47.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:47 smithi092 ceph-mon[33966]: from='client.24383 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-04-14T15:09:47.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:47 smithi092 ceph-mon[33966]: osdmap e69: 8 total, 8 up, 8 in 2024-04-14T15:09:47.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:47 smithi090 ceph-mon[30399]: pgmap v197: 68 pgs: 9 creating+peering, 25 active+clean, 34 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 511 B/s wr, 1 op/s 2024-04-14T15:09:47.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:47 smithi090 ceph-mon[30399]: osdmap e68: 8 total, 8 up, 8 in 2024-04-14T15:09:47.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:47 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2924083629' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-04-14T15:09:47.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:47 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2632170366' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-04-14T15:09:47.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:47 smithi090 ceph-mon[30399]: from='client.24373 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-04-14T15:09:47.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:47 smithi090 ceph-mon[30399]: from='client.24383 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-04-14T15:09:47.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:47 smithi090 ceph-mon[30399]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-14T15:09:47.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:47 smithi090 ceph-mon[30399]: Cluster is now healthy 2024-04-14T15:09:47.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:47 smithi090 ceph-mon[30399]: from='client.24373 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-04-14T15:09:47.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:47 smithi090 ceph-mon[30399]: from='client.24383 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-04-14T15:09:47.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:47 smithi090 ceph-mon[30399]: osdmap e69: 8 total, 8 up, 8 in 2024-04-14T15:09:47.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:47 smithi090 ceph-mon[41670]: pgmap v197: 68 pgs: 9 creating+peering, 25 active+clean, 34 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 511 B/s wr, 1 op/s 2024-04-14T15:09:47.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:47 smithi090 ceph-mon[41670]: osdmap e68: 8 total, 8 up, 8 in 2024-04-14T15:09:47.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:47 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2924083629' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-04-14T15:09:47.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:47 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2632170366' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-04-14T15:09:47.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:47 smithi090 ceph-mon[41670]: from='client.24373 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-04-14T15:09:47.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:47 smithi090 ceph-mon[41670]: from='client.24383 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-04-14T15:09:47.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:47 smithi090 ceph-mon[41670]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-14T15:09:47.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:47 smithi090 ceph-mon[41670]: Cluster is now healthy 2024-04-14T15:09:47.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:47 smithi090 ceph-mon[41670]: from='client.24373 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-04-14T15:09:47.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:47 smithi090 ceph-mon[41670]: from='client.24383 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-04-14T15:09:47.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:47 smithi090 ceph-mon[41670]: osdmap e69: 8 total, 8 up, 8 in 2024-04-14T15:09:47.755 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.92 --placement '1;smithi092=iscsi.a' 2024-04-14T15:09:47.997 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:09:49.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:49 smithi092 ceph-mon[33966]: pgmap v200: 100 pgs: 4 creating+peering, 63 active+clean, 33 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-04-14T15:09:49.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:49 smithi092 ceph-mon[33966]: osdmap e70: 8 total, 8 up, 8 in 2024-04-14T15:09:49.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:49 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2632170366' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-04-14T15:09:49.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:49 smithi092 ceph-mon[33966]: from='client.24383 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-04-14T15:09:49.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:49 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2924083629' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-04-14T15:09:49.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:49 smithi092 ceph-mon[33966]: from='client.24373 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-04-14T15:09:49.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:49 smithi090 ceph-mon[41670]: pgmap v200: 100 pgs: 4 creating+peering, 63 active+clean, 33 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-04-14T15:09:49.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:49 smithi090 ceph-mon[41670]: osdmap e70: 8 total, 8 up, 8 in 2024-04-14T15:09:49.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:49 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2632170366' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-04-14T15:09:49.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:49 smithi090 ceph-mon[41670]: from='client.24383 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-04-14T15:09:49.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:49 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2924083629' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-04-14T15:09:49.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:49 smithi090 ceph-mon[41670]: from='client.24373 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-04-14T15:09:49.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:49 smithi090 ceph-mon[30399]: pgmap v200: 100 pgs: 4 creating+peering, 63 active+clean, 33 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-04-14T15:09:49.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:49 smithi090 ceph-mon[30399]: osdmap e70: 8 total, 8 up, 8 in 2024-04-14T15:09:49.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:49 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2632170366' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-04-14T15:09:49.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:49 smithi090 ceph-mon[30399]: from='client.24383 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-04-14T15:09:49.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:49 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2924083629' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-04-14T15:09:49.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:49 smithi090 ceph-mon[30399]: from='client.24373 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-04-14T15:09:49.695 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:49.692+0000 7fe0fa046700 1 -- 172.21.15.92:0/2896304464 >> v1:172.21.15.92:6789/0 conn(0x7fe0f41028e0 legacy=0x7fe0f4102d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:09:49.695 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:49.692+0000 7fe0fa046700 1 -- 172.21.15.92:0/2896304464 shutdown_connections 2024-04-14T15:09:49.695 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:49.692+0000 7fe0fa046700 1 -- 172.21.15.92:0/2896304464 >> 172.21.15.92:0/2896304464 conn(0x7fe0f40fde80 msgr2=0x7fe0f41002c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:09:49.695 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:49.692+0000 7fe0fa046700 1 -- 172.21.15.92:0/2896304464 shutdown_connections 2024-04-14T15:09:49.695 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:49.692+0000 7fe0fa046700 1 -- 172.21.15.92:0/2896304464 wait complete. 2024-04-14T15:09:49.696 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:49.692+0000 7fe0fa046700 1 librados: starting msgr at 2024-04-14T15:09:49.696 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:49.692+0000 7fe0fa046700 1 librados: starting objecter 2024-04-14T15:09:49.696 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:49.692+0000 7fe0fa046700 1 Processor -- start 2024-04-14T15:09:49.696 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:49.693+0000 7fe0fa046700 1 -- start start 2024-04-14T15:09:49.696 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:49.693+0000 7fe0fa046700 1 librados: setting wanted keys 2024-04-14T15:09:49.696 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:49.693+0000 7fe0fa046700 1 librados: calling monclient init 2024-04-14T15:09:49.696 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:49.693+0000 7fe0fa046700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe0f410e2d0 con 0x7fe0f4106d10 2024-04-14T15:09:49.696 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:49.693+0000 7fe0fa046700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe0f41a6510 con 0x7fe0f41028e0 2024-04-14T15:09:49.696 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:49.693+0000 7fe0fa046700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe0f41a76f0 con 0x7fe0f410aa50 2024-04-14T15:09:49.696 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:49.693+0000 7fe0f37fe700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fe0f41028e0 0x7fe0f410d780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:60412/0 (socket says 172.21.15.92:60412) 2024-04-14T15:09:49.696 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:49.693+0000 7fe0f37fe700 1 -- 172.21.15.92:0/3147218297 learned_addr learned my addr 172.21.15.92:0/3147218297 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-04-14T15:09:49.696 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:49.693+0000 7fe0f0ff9700 10 client.?.objecter ms_handle_connect 0x7fe0f41028e0 2024-04-14T15:09:49.697 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:49.693+0000 7fe0f0ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:09:49.697 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:49.693+0000 7fe0f0ff9700 10 client.?.objecter ms_handle_connect 0x7fe0f4106d10 2024-04-14T15:09:49.697 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:49.693+0000 7fe0f0ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:09:49.697 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:49.693+0000 7fe0f0ff9700 10 client.?.objecter ms_handle_connect 0x7fe0f410aa50 2024-04-14T15:09:49.697 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:49.693+0000 7fe0f0ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:09:49.697 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:49.693+0000 7fe0f0ff9700 1 -- 172.21.15.92:0/3147218297 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 223515952 0 0) 0x7fe0f41a6510 con 0x7fe0f41028e0 2024-04-14T15:09:49.697 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:49.693+0000 7fe0f0ff9700 1 -- 172.21.15.92:0/3147218297 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe0dc0035b0 con 0x7fe0f41028e0 2024-04-14T15:09:49.697 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:49.693+0000 7fe0f0ff9700 1 -- 172.21.15.92:0/3147218297 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2925117095 0 0) 0x7fe0f410e2d0 con 0x7fe0f4106d10 2024-04-14T15:09:49.697 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:49.693+0000 7fe0f0ff9700 1 -- 172.21.15.92:0/3147218297 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe0f41a6510 con 0x7fe0f4106d10 2024-04-14T15:09:49.697 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:49.693+0000 7fe0f0ff9700 1 -- 172.21.15.92:0/3147218297 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1508399775 0 0) 0x7fe0f41a76f0 con 0x7fe0f410aa50 2024-04-14T15:09:49.697 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:49.693+0000 7fe0f0ff9700 1 -- 172.21.15.92:0/3147218297 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe0f410e2d0 con 0x7fe0f410aa50 2024-04-14T15:09:49.697 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:49.693+0000 7fe0f0ff9700 1 -- 172.21.15.92:0/3147218297 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 228770394 0 0) 0x7fe0dc0035b0 con 0x7fe0f41028e0 2024-04-14T15:09:49.697 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:49.693+0000 7fe0f0ff9700 1 -- 172.21.15.92:0/3147218297 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe0f41a76f0 con 0x7fe0f41028e0 2024-04-14T15:09:49.697 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:49.693+0000 7fe0f0ff9700 1 -- 172.21.15.92:0/3147218297 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fe0e4001430 con 0x7fe0f41028e0 2024-04-14T15:09:49.698 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:49.694+0000 7fe0f0ff9700 1 -- 172.21.15.92:0/3147218297 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3149456857 0 0) 0x7fe0f41a6510 con 0x7fe0f4106d10 2024-04-14T15:09:49.698 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:49.694+0000 7fe0f0ff9700 1 -- 172.21.15.92:0/3147218297 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe0dc0035b0 con 0x7fe0f4106d10 2024-04-14T15:09:49.698 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:49.694+0000 7fe0f0ff9700 1 -- 172.21.15.92:0/3147218297 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fe0e80017e0 con 0x7fe0f4106d10 2024-04-14T15:09:49.698 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:49.694+0000 7fe0f0ff9700 1 -- 172.21.15.92:0/3147218297 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3000952074 0 0) 0x7fe0f41a76f0 con 0x7fe0f41028e0 2024-04-14T15:09:49.698 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:49.694+0000 7fe0f0ff9700 1 -- 172.21.15.92:0/3147218297 >> v1:172.21.15.90:6790/0 conn(0x7fe0f410aa50 legacy=0x7fe0f41a3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:09:49.698 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:49.694+0000 7fe0f0ff9700 1 -- 172.21.15.92:0/3147218297 >> v1:172.21.15.90:6789/0 conn(0x7fe0f4106d10 legacy=0x7fe0f41a06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:09:49.698 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:49.694+0000 7fe0f0ff9700 1 -- 172.21.15.92:0/3147218297 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0f41a8910 con 0x7fe0f41028e0 2024-04-14T15:09:49.698 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:49.694+0000 7fe0fa046700 1 -- 172.21.15.92:0/3147218297 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe0f41a5560 con 0x7fe0f41028e0 2024-04-14T15:09:49.698 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:49.694+0000 7fe0fa046700 10 client.24412.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:09:49.698 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:49.694+0000 7fe0fa046700 1 -- 172.21.15.92:0/3147218297 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe0f41a5aa0 con 0x7fe0f41028e0 2024-04-14T15:09:49.698 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:49.694+0000 7fe0fa046700 1 librados: init done 2024-04-14T15:09:49.699 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:49.694+0000 7fe0f0ff9700 1 -- 172.21.15.92:0/3147218297 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fe0e4001bb0 con 0x7fe0f41028e0 2024-04-14T15:09:49.699 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:49.694+0000 7fe0f0ff9700 1 -- 172.21.15.92:0/3147218297 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fe0e4005010 con 0x7fe0f41028e0 2024-04-14T15:09:49.699 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:49.695+0000 7fe0fa046700 1 -- 172.21.15.92:0/3147218297 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0e0005150 con 0x7fe0f41028e0 2024-04-14T15:09:49.700 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:49.695+0000 7fe0f0ff9700 1 -- 172.21.15.92:0/3147218297 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 15) ==== 92623+0+0 (unknown 2343906571 0 0) 0x7fe0e4003ab0 con 0x7fe0f41028e0 2024-04-14T15:09:49.700 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:49.696+0000 7fe0f0ff9700 1 -- 172.21.15.92:0/3147218297 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(71..71 src has 12..71) ==== 5926+0+0 (unknown 1859983057 0 0) 0x7fe0e408a180 con 0x7fe0f41028e0 2024-04-14T15:09:49.700 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:49.696+0000 7fe0f0ff9700 10 client.24412.objecter ms_dispatch 0x7fe0f4001550 osd_map(71..71 src has 12..71) 2024-04-14T15:09:49.700 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:49.696+0000 7fe0f0ff9700 3 client.24412.objecter handle_osd_map got epochs [71,71] > 0 2024-04-14T15:09:49.700 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:49.696+0000 7fe0f0ff9700 3 client.24412.objecter handle_osd_map decoding full epoch 71 2024-04-14T15:09:49.700 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:49.696+0000 7fe0f0ff9700 20 client.24412.objecter dump_active .. 0 homeless 2024-04-14T15:09:49.700 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:49.696+0000 7fe0f0ff9700 10 client.24412.objecter ms_handle_connect 0x7fe0dc070fe0 2024-04-14T15:09:49.700 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:49.700+0000 7fe0f0ff9700 1 -- 172.21.15.92:0/3147218297 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fe0e4057590 con 0x7fe0f41028e0 2024-04-14T15:09:49.910 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:49.910+0000 7fe0fa046700 1 -- 172.21.15.92:0/3147218297 --> v1:172.21.15.90:6800/491071118 -- mgr_command(tid 0: {"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.92", "placement": "1;smithi092=iscsi.a", "target": ["mon-mgr", ""]}) -- 0x7fe0e0000c90 con 0x7fe0dc070fe0 2024-04-14T15:09:49.918 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:49.918+0000 7fe0f0ff9700 1 -- 172.21.15.92:0/3147218297 <== mgr.14148 v1:172.21.15.90:6800/491071118 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+35 (unknown 0 0 803663096) 0x7fe0e0000c90 con 0x7fe0dc070fe0 2024-04-14T15:09:49.919 INFO:teuthology.orchestra.run.smithi092.stdout:Scheduled iscsi.datapool update... 2024-04-14T15:09:49.921 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:49.920+0000 7fe0fa046700 10 librados: watch_flush enter 2024-04-14T15:09:49.921 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:49.920+0000 7fe0fa046700 10 librados: watch_flush exit 2024-04-14T15:09:49.922 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:49.920+0000 7fe0fa046700 20 client.24412.objecter shutdown clearing up homeless session... 2024-04-14T15:09:49.922 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:49.920+0000 7fe0fa046700 10 client.24412.objecter successfully canceled tick 2024-04-14T15:09:49.922 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:49.920+0000 7fe0fa046700 1 -- 172.21.15.92:0/3147218297 >> v1:172.21.15.90:6800/491071118 conn(0x7fe0dc070fe0 legacy=0x7fe0dc0734a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:09:49.922 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:49.920+0000 7fe0fa046700 1 -- 172.21.15.92:0/3147218297 >> v1:172.21.15.92:6789/0 conn(0x7fe0f41028e0 legacy=0x7fe0f410d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:09:49.922 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:49.921+0000 7fe0fa046700 1 -- 172.21.15.92:0/3147218297 shutdown_connections 2024-04-14T15:09:49.922 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:49.921+0000 7fe0fa046700 1 -- 172.21.15.92:0/3147218297 >> 172.21.15.92:0/3147218297 conn(0x7fe0f40fde80 msgr2=0x7fe0f4109d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:09:49.922 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:49.921+0000 7fe0fa046700 1 -- 172.21.15.92:0/3147218297 shutdown_connections 2024-04-14T15:09:49.922 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:49.921+0000 7fe0fa046700 1 -- 172.21.15.92:0/3147218297 wait complete. 2024-04-14T15:09:49.922 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:49.921+0000 7fe0fa046700 1 librados: shutdown 2024-04-14T15:09:50.245 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:50 smithi092 ceph-mon[33966]: from='client.24383 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-04-14T15:09:50.245 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:50 smithi092 ceph-mon[33966]: from='client.24373 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-04-14T15:09:50.245 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:50 smithi092 ceph-mon[33966]: osdmap e71: 8 total, 8 up, 8 in 2024-04-14T15:09:50.245 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:50 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2924083629' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-14T15:09:50.245 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:50 smithi092 ceph-mon[33966]: from='client.24373 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-14T15:09:50.245 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:50 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2632170366' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-14T15:09:50.245 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:50 smithi092 ceph-mon[33966]: from='client.24383 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-14T15:09:50.245 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:50 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:50.245 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:50 smithi092 ceph-mon[33966]: from='client.24373 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-14T15:09:50.441 INFO:journalctl@ceph.rgw.foo.a.smithi090.stdout:Apr 14 15:09:50 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-rgw-foo-a[70053]: 2024-04-14T15:09:50.107+0000 7fbacf28bac0 -1 LDAP not started since no server URIs were provided in the configuration. 2024-04-14T15:09:50.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:50 smithi090 ceph-mon[41670]: from='client.24383 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-04-14T15:09:50.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:50 smithi090 ceph-mon[41670]: from='client.24373 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-04-14T15:09:50.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:50 smithi090 ceph-mon[41670]: osdmap e71: 8 total, 8 up, 8 in 2024-04-14T15:09:50.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:50 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2924083629' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-14T15:09:50.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:50 smithi090 ceph-mon[41670]: from='client.24373 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-14T15:09:50.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:50 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2632170366' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-14T15:09:50.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:50 smithi090 ceph-mon[41670]: from='client.24383 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-14T15:09:50.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:50 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:50.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:50 smithi090 ceph-mon[41670]: from='client.24373 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-14T15:09:50.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:50 smithi090 ceph-mon[30399]: from='client.24383 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-04-14T15:09:50.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:50 smithi090 ceph-mon[30399]: from='client.24373 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-04-14T15:09:50.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:50 smithi090 ceph-mon[30399]: osdmap e71: 8 total, 8 up, 8 in 2024-04-14T15:09:50.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:50 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2924083629' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-14T15:09:50.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:50 smithi090 ceph-mon[30399]: from='client.24373 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-14T15:09:50.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:50 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2632170366' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-14T15:09:50.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:50 smithi090 ceph-mon[30399]: from='client.24383 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-14T15:09:50.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:50 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:50.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:50 smithi090 ceph-mon[30399]: from='client.24373 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-14T15:09:50.598 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2024-04-14T15:09:50.598 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-14T15:09:50.598 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-04-14T15:09:50.628 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-04-14T15:09:50.628 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-04-14T15:09:50.657 DEBUG:teuthology.orchestra.run.smithi092:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-0e288098-fa70-11ee-bc8e-c7b262605968@iscsi.iscsi.a.service 2024-04-14T15:09:50.699 INFO:tasks.cephadm:Adding prometheus.a on smithi092 2024-04-14T15:09:50.700 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph orch apply prometheus '1;smithi092=a' 2024-04-14T15:09:51.249 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:51 smithi092 ceph-mon[33966]: from='client.24412 v1:172.21.15.92:0/3147218297' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.92", "placement": "1;smithi092=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:09:51.249 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:51 smithi092 ceph-mon[33966]: Saving service iscsi.datapool spec with placement smithi092=iscsi.a;count:1 2024-04-14T15:09:51.249 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:51 smithi092 ceph-mon[33966]: pgmap v203: 132 pgs: 14 creating+peering, 95 active+clean, 23 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-04-14T15:09:51.249 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:51 smithi092 ceph-mon[33966]: from='client.24383 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-14T15:09:51.249 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:51 smithi092 ceph-mon[33966]: osdmap e72: 8 total, 8 up, 8 in 2024-04-14T15:09:51.249 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:51 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:51.249 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:51 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:51.249 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:51 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:51.249 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:51 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-14T15:09:51.249 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:51 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-04-14T15:09:51.249 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:51 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:09:51.250 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:51 smithi092 ceph-mon[33966]: Deploying daemon iscsi.iscsi.a on smithi092 2024-04-14T15:09:51.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:51 smithi090 ceph-mon[30399]: from='client.24412 v1:172.21.15.92:0/3147218297' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.92", "placement": "1;smithi092=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:09:51.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:51 smithi090 ceph-mon[30399]: Saving service iscsi.datapool spec with placement smithi092=iscsi.a;count:1 2024-04-14T15:09:51.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:51 smithi090 ceph-mon[30399]: pgmap v203: 132 pgs: 14 creating+peering, 95 active+clean, 23 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-04-14T15:09:51.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:51 smithi090 ceph-mon[30399]: from='client.24383 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-14T15:09:51.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:51 smithi090 ceph-mon[30399]: osdmap e72: 8 total, 8 up, 8 in 2024-04-14T15:09:51.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:51 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:51.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:51 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:51.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:51 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:51.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:51 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-14T15:09:51.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:51 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-04-14T15:09:51.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:51 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:09:51.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:51 smithi090 ceph-mon[30399]: Deploying daemon iscsi.iscsi.a on smithi092 2024-04-14T15:09:51.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:51 smithi090 ceph-mon[41670]: from='client.24412 v1:172.21.15.92:0/3147218297' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.92", "placement": "1;smithi092=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:09:51.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:51 smithi090 ceph-mon[41670]: Saving service iscsi.datapool spec with placement smithi092=iscsi.a;count:1 2024-04-14T15:09:51.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:51 smithi090 ceph-mon[41670]: pgmap v203: 132 pgs: 14 creating+peering, 95 active+clean, 23 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-04-14T15:09:51.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:51 smithi090 ceph-mon[41670]: from='client.24383 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-14T15:09:51.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:51 smithi090 ceph-mon[41670]: osdmap e72: 8 total, 8 up, 8 in 2024-04-14T15:09:51.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:51 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:51.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:51 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:51.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:51 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:51.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:51 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-14T15:09:51.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:51 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-04-14T15:09:51.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:51 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:09:51.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:51 smithi090 ceph-mon[41670]: Deploying daemon iscsi.iscsi.a on smithi092 2024-04-14T15:09:51.573 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:09:52.263 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:52 smithi092 ceph-mon[33966]: pgmap v205: 132 pgs: 14 creating+peering, 105 active+clean, 13 unknown; 579 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2024-04-14T15:09:52.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:52 smithi090 ceph-mon[30399]: pgmap v205: 132 pgs: 14 creating+peering, 105 active+clean, 13 unknown; 579 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2024-04-14T15:09:52.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:52 smithi090 ceph-mon[41670]: pgmap v205: 132 pgs: 14 creating+peering, 105 active+clean, 13 unknown; 579 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2024-04-14T15:09:53.102 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:53 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:09:53.102 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:53 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:09:53.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:53 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:09:53.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:53 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:09:53.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:53 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:09:53.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:53 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:09:53.968 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:53.964+0000 7fb66d886700 1 -- 172.21.15.92:0/3769352590 >> v1:172.21.15.92:6789/0 conn(0x7fb668100ba0 legacy=0x7fb668100fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:09:53.969 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:53.964+0000 7fb66d886700 1 -- 172.21.15.92:0/3769352590 shutdown_connections 2024-04-14T15:09:53.969 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:53.964+0000 7fb66d886700 1 -- 172.21.15.92:0/3769352590 >> 172.21.15.92:0/3769352590 conn(0x7fb6680fc520 msgr2=0x7fb6680fe940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:09:53.969 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:53.964+0000 7fb66d886700 1 -- 172.21.15.92:0/3769352590 shutdown_connections 2024-04-14T15:09:53.969 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:53.965+0000 7fb66d886700 1 -- 172.21.15.92:0/3769352590 wait complete. 2024-04-14T15:09:53.969 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:53.965+0000 7fb66d886700 1 librados: starting msgr at 2024-04-14T15:09:53.969 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:53.965+0000 7fb66d886700 1 librados: starting objecter 2024-04-14T15:09:53.969 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:53.965+0000 7fb66d886700 1 Processor -- start 2024-04-14T15:09:53.969 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:53.965+0000 7fb66d886700 1 -- start start 2024-04-14T15:09:53.969 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:53.965+0000 7fb66d886700 1 librados: setting wanted keys 2024-04-14T15:09:53.969 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:53.965+0000 7fb66d886700 1 librados: calling monclient init 2024-04-14T15:09:53.969 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:53.966+0000 7fb66d886700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb668198910 con 0x7fb668106db0 2024-04-14T15:09:53.969 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:53.966+0000 7fb66d886700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb6681a4100 con 0x7fb66810aa10 2024-04-14T15:09:53.970 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:53.966+0000 7fb66d886700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb6681a52e0 con 0x7fb668100ba0 2024-04-14T15:09:53.970 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:53.966+0000 7fb6677fe700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fb66810aa10 0x7fb6681a19d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:60426/0 (socket says 172.21.15.92:60426) 2024-04-14T15:09:53.970 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:53.966+0000 7fb6677fe700 1 -- 172.21.15.92:0/309125602 learned_addr learned my addr 172.21.15.92:0/309125602 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-04-14T15:09:53.970 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:53.966+0000 7fb666ffd700 1 --1- 172.21.15.92:0/309125602 >> v1:172.21.15.90:6790/0 conn(0x7fb668100ba0 0x7fb668197dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.92:54852/0 (socket says 172.21.15.92:54852) 2024-04-14T15:09:53.970 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:53.966+0000 7fb66c884700 10 client.?.objecter ms_handle_connect 0x7fb66810aa10 2024-04-14T15:09:53.970 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:53.966+0000 7fb66c884700 10 client.?.objecter resend_mon_ops 2024-04-14T15:09:53.970 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:53.966+0000 7fb66c884700 10 client.?.objecter ms_handle_connect 0x7fb668106db0 2024-04-14T15:09:53.970 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:53.966+0000 7fb66c884700 10 client.?.objecter resend_mon_ops 2024-04-14T15:09:53.970 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:53.966+0000 7fb66c884700 10 client.?.objecter ms_handle_connect 0x7fb668100ba0 2024-04-14T15:09:53.970 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:53.966+0000 7fb66c884700 10 client.?.objecter resend_mon_ops 2024-04-14T15:09:53.970 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:53.966+0000 7fb66c884700 1 -- 172.21.15.92:0/309125602 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 556763079 0 0) 0x7fb668198910 con 0x7fb668106db0 2024-04-14T15:09:53.970 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:53.966+0000 7fb66c884700 1 -- 172.21.15.92:0/309125602 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6540035b0 con 0x7fb668106db0 2024-04-14T15:09:53.971 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:53.966+0000 7fb66c884700 1 -- 172.21.15.92:0/309125602 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2262771715 0 0) 0x7fb6681a4100 con 0x7fb66810aa10 2024-04-14T15:09:53.971 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:53.966+0000 7fb66c884700 1 -- 172.21.15.92:0/309125602 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb668198910 con 0x7fb66810aa10 2024-04-14T15:09:53.971 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:53.966+0000 7fb66c884700 1 -- 172.21.15.92:0/309125602 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1685080744 0 0) 0x7fb6681a52e0 con 0x7fb668100ba0 2024-04-14T15:09:53.971 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:53.966+0000 7fb66c884700 1 -- 172.21.15.92:0/309125602 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6681a4100 con 0x7fb668100ba0 2024-04-14T15:09:53.971 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:53.966+0000 7fb66c884700 1 -- 172.21.15.92:0/309125602 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 498366600 0 0) 0x7fb668198910 con 0x7fb66810aa10 2024-04-14T15:09:53.971 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:53.966+0000 7fb66c884700 1 -- 172.21.15.92:0/309125602 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb6681a52e0 con 0x7fb66810aa10 2024-04-14T15:09:53.971 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:53.966+0000 7fb66c884700 1 -- 172.21.15.92:0/309125602 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1150453137 0 0) 0x7fb6540035b0 con 0x7fb668106db0 2024-04-14T15:09:53.971 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:53.966+0000 7fb66c884700 1 -- 172.21.15.92:0/309125602 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb668198910 con 0x7fb668106db0 2024-04-14T15:09:53.971 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:53.967+0000 7fb66c884700 1 -- 172.21.15.92:0/309125602 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fb660001bd0 con 0x7fb66810aa10 2024-04-14T15:09:53.971 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:53.967+0000 7fb66c884700 1 -- 172.21.15.92:0/309125602 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fb6580017f0 con 0x7fb668106db0 2024-04-14T15:09:53.971 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:53.967+0000 7fb66c884700 1 -- 172.21.15.92:0/309125602 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2156960808 0 0) 0x7fb6681a4100 con 0x7fb668100ba0 2024-04-14T15:09:53.972 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:53.967+0000 7fb66c884700 1 -- 172.21.15.92:0/309125602 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb6540035b0 con 0x7fb668100ba0 2024-04-14T15:09:53.972 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:53.967+0000 7fb66c884700 1 -- 172.21.15.92:0/309125602 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fb6500015d0 con 0x7fb668100ba0 2024-04-14T15:09:53.972 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:53.967+0000 7fb66c884700 1 -- 172.21.15.92:0/309125602 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3986486498 0 0) 0x7fb6681a52e0 con 0x7fb66810aa10 2024-04-14T15:09:53.972 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:53.967+0000 7fb66c884700 1 -- 172.21.15.92:0/309125602 >> v1:172.21.15.90:6790/0 conn(0x7fb668100ba0 legacy=0x7fb668197dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:09:53.972 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:53.967+0000 7fb66c884700 1 -- 172.21.15.92:0/309125602 >> v1:172.21.15.90:6789/0 conn(0x7fb668106db0 legacy=0x7fb66819e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:09:53.972 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:53.967+0000 7fb66c884700 1 -- 172.21.15.92:0/309125602 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6681a6500 con 0x7fb66810aa10 2024-04-14T15:09:53.972 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:53.967+0000 7fb66d886700 1 -- 172.21.15.92:0/309125602 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb6681a3150 con 0x7fb66810aa10 2024-04-14T15:09:53.972 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:53.967+0000 7fb66d886700 10 client.24433.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:09:53.972 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:53.967+0000 7fb66d886700 1 -- 172.21.15.92:0/309125602 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb6681a3690 con 0x7fb66810aa10 2024-04-14T15:09:53.972 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:53.967+0000 7fb66d886700 1 librados: init done 2024-04-14T15:09:53.972 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:53.967+0000 7fb66c884700 1 -- 172.21.15.92:0/309125602 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fb660001fe0 con 0x7fb66810aa10 2024-04-14T15:09:53.972 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:53.967+0000 7fb66c884700 1 -- 172.21.15.92:0/309125602 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fb660006020 con 0x7fb66810aa10 2024-04-14T15:09:53.973 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:53.968+0000 7fb66d886700 1 -- 172.21.15.92:0/309125602 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6680693d0 con 0x7fb66810aa10 2024-04-14T15:09:53.973 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:53.969+0000 7fb66c884700 1 -- 172.21.15.92:0/309125602 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 15) ==== 92623+0+0 (unknown 2343906571 0 0) 0x7fb66001ca00 con 0x7fb66810aa10 2024-04-14T15:09:53.973 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:53.969+0000 7fb66c884700 1 -- 172.21.15.92:0/309125602 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(73..73 src has 12..73) ==== 5942+0+0 (unknown 2855617026 0 0) 0x7fb66008a360 con 0x7fb66810aa10 2024-04-14T15:09:53.974 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:53.969+0000 7fb66c884700 10 client.24433.objecter ms_dispatch 0x7fb668001550 osd_map(73..73 src has 12..73) 2024-04-14T15:09:53.974 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:53.969+0000 7fb66c884700 3 client.24433.objecter handle_osd_map got epochs [73,73] > 0 2024-04-14T15:09:53.974 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:53.969+0000 7fb66c884700 3 client.24433.objecter handle_osd_map decoding full epoch 73 2024-04-14T15:09:53.974 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:53.969+0000 7fb66c884700 20 client.24433.objecter dump_active .. 0 homeless 2024-04-14T15:09:53.974 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:53.970+0000 7fb66c884700 10 client.24433.objecter ms_handle_connect 0x7fb654071220 2024-04-14T15:09:53.974 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:53.974+0000 7fb66c884700 1 -- 172.21.15.92:0/309125602 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fb660057760 con 0x7fb66810aa10 2024-04-14T15:09:54.223 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:54.222+0000 7fb66d886700 1 -- 172.21.15.92:0/309125602 --> v1:172.21.15.90:6800/491071118 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi092=a", "target": ["mon-mgr", ""]}) -- 0x7fb6681a3a20 con 0x7fb654071220 2024-04-14T15:09:54.230 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:54.230+0000 7fb66c884700 1 -- 172.21.15.92:0/309125602 <== mgr.14148 v1:172.21.15.90:6800/491071118 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+31 (unknown 0 0 1342662408) 0x7fb6681a3a20 con 0x7fb654071220 2024-04-14T15:09:54.230 INFO:teuthology.orchestra.run.smithi092.stdout:Scheduled prometheus update... 2024-04-14T15:09:54.234 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:54.233+0000 7fb66d886700 10 librados: watch_flush enter 2024-04-14T15:09:54.234 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:54.233+0000 7fb66d886700 10 librados: watch_flush exit 2024-04-14T15:09:54.234 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:54.233+0000 7fb66d886700 20 client.24433.objecter shutdown clearing up homeless session... 2024-04-14T15:09:54.234 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:54.233+0000 7fb66d886700 10 client.24433.objecter successfully canceled tick 2024-04-14T15:09:54.234 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:54.233+0000 7fb66d886700 1 -- 172.21.15.92:0/309125602 >> v1:172.21.15.90:6800/491071118 conn(0x7fb654071220 legacy=0x7fb6540736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:09:54.234 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:54.233+0000 7fb66d886700 1 -- 172.21.15.92:0/309125602 >> v1:172.21.15.92:6789/0 conn(0x7fb66810aa10 legacy=0x7fb6681a19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:09:54.234 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:54.233+0000 7fb66d886700 1 -- 172.21.15.92:0/309125602 shutdown_connections 2024-04-14T15:09:54.234 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:54.233+0000 7fb66d886700 1 -- 172.21.15.92:0/309125602 >> 172.21.15.92:0/309125602 conn(0x7fb6680fc520 msgr2=0x7fb6680fe910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:09:54.234 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:54.233+0000 7fb66d886700 1 -- 172.21.15.92:0/309125602 shutdown_connections 2024-04-14T15:09:54.234 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:54.233+0000 7fb66d886700 1 -- 172.21.15.92:0/309125602 wait complete. 2024-04-14T15:09:54.234 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:54.233+0000 7fb66d886700 1 librados: shutdown 2024-04-14T15:09:54.359 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:54 smithi092 ceph-mon[33966]: osdmap e73: 8 total, 8 up, 8 in 2024-04-14T15:09:54.359 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:54 smithi092 ceph-mon[33966]: pgmap v207: 132 pgs: 7 creating+peering, 125 active+clean; 581 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 5.1 KiB/s wr, 146 op/s 2024-04-14T15:09:54.360 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:09:54 smithi092 podman[63492]: 2024-04-14 15:09:54.359606003 +0000 UTC m=+0.384795956 container init 5d5759575ca229bf030cf7ac396d66ce82e747187a59f1cf64ea0b55b0e0906a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a971fd5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-14T15:09:54.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:54 smithi090 ceph-mon[30399]: osdmap e73: 8 total, 8 up, 8 in 2024-04-14T15:09:54.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:54 smithi090 ceph-mon[30399]: pgmap v207: 132 pgs: 7 creating+peering, 125 active+clean; 581 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 5.1 KiB/s wr, 146 op/s 2024-04-14T15:09:54.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:54 smithi090 ceph-mon[41670]: osdmap e73: 8 total, 8 up, 8 in 2024-04-14T15:09:54.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:54 smithi090 ceph-mon[41670]: pgmap v207: 132 pgs: 7 creating+peering, 125 active+clean; 581 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 5.1 KiB/s wr, 146 op/s 2024-04-14T15:09:54.622 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:09:54 smithi092 podman[63492]: 2024-04-14 15:09:54.366078749 +0000 UTC m=+0.391268694 container start 5d5759575ca229bf030cf7ac396d66ce82e747187a59f1cf64ea0b55b0e0906a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=main-a971fd5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-14T15:09:54.622 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:09:54 smithi092 bash[63492]: 5d5759575ca229bf030cf7ac396d66ce82e747187a59f1cf64ea0b55b0e0906a 2024-04-14T15:09:54.622 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:09:54 smithi092 systemd[1]: Started Ceph iscsi.iscsi.a for 0e288098-fa70-11ee-bc8e-c7b262605968. 2024-04-14T15:09:54.987 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:09:54 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug Started the configuration object watcher 2024-04-14T15:09:54.987 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:09:54 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug Checking for config object changes every 1s 2024-04-14T15:09:54.987 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:09:54 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug Processing osd blocklist entries for this node 2024-04-14T15:09:54.987 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:09:54 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug Reading the configuration object to update local LIO configuration 2024-04-14T15:09:54.987 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:09:54 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug Configuration does not have an entry for this host(smithi092.front.sepia.ceph.com) - nothing to define to LIO 2024-04-14T15:09:55.141 DEBUG:teuthology.orchestra.run.smithi092:prometheus.a> sudo journalctl -f -n 0 -u ceph-0e288098-fa70-11ee-bc8e-c7b262605968@prometheus.a.service 2024-04-14T15:09:55.143 INFO:tasks.cephadm:Adding node-exporter.a on smithi090 2024-04-14T15:09:55.143 INFO:tasks.cephadm:Adding node-exporter.b on smithi092 2024-04-14T15:09:55.143 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph orch apply node-exporter '2;smithi090=a;smithi092=b' 2024-04-14T15:09:55.266 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:09:54 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug * Running on http://[::]:5000/ (Press CTRL+C to quit) 2024-04-14T15:09:55.267 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:55 smithi092 ceph-mon[33966]: from='client.24433 v1:172.21.15.92:0/309125602' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi092=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:09:55.267 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:55 smithi092 ceph-mon[33966]: Saving service prometheus spec with placement smithi092=a;count:1 2024-04-14T15:09:55.267 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:55 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:55.267 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:55 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:55.267 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:55 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:55.267 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:55 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:55.267 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:55 smithi092 ceph-mon[33966]: Checking pool "datapool" exists for service iscsi.datapool 2024-04-14T15:09:55.267 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:55 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:55.267 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:55 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:09:55.267 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:55 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.92:0/3847718528' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-14T15:09:55.607 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:55 smithi090 ceph-mon[30399]: from='client.24433 v1:172.21.15.92:0/309125602' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi092=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:09:55.608 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:55 smithi090 ceph-mon[30399]: Saving service prometheus spec with placement smithi092=a;count:1 2024-04-14T15:09:55.608 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:55 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:55.608 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:55 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:55.608 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:55 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:55.608 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:55 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:55.608 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:55 smithi090 ceph-mon[30399]: Checking pool "datapool" exists for service iscsi.datapool 2024-04-14T15:09:55.608 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:55 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:55.608 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:55 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:09:55.608 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:55 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.92:0/3847718528' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-14T15:09:55.608 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:55 smithi090 ceph-mon[41670]: from='client.24433 v1:172.21.15.92:0/309125602' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi092=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:09:55.608 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:55 smithi090 ceph-mon[41670]: Saving service prometheus spec with placement smithi092=a;count:1 2024-04-14T15:09:55.608 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:55 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:55.609 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:55 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:55.609 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:55 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:55.609 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:55 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:55.609 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:55 smithi090 ceph-mon[41670]: Checking pool "datapool" exists for service iscsi.datapool 2024-04-14T15:09:55.609 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:55 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:55.609 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:55 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:09:55.609 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:55 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.92:0/3847718528' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-14T15:09:55.723 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:09:56.941 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:56 smithi090 ceph-mon[30399]: osdmap e74: 8 total, 8 up, 8 in 2024-04-14T15:09:56.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:56 smithi090 ceph-mon[30399]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 5.7 KiB/s wr, 185 op/s 2024-04-14T15:09:56.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:56 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:56.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:56 smithi090 ceph-mon[41670]: osdmap e74: 8 total, 8 up, 8 in 2024-04-14T15:09:56.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:56 smithi090 ceph-mon[41670]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 5.7 KiB/s wr, 185 op/s 2024-04-14T15:09:56.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:56 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:57.090 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:56 smithi092 ceph-mon[33966]: osdmap e74: 8 total, 8 up, 8 in 2024-04-14T15:09:57.091 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:56 smithi092 ceph-mon[33966]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 5.7 KiB/s wr, 185 op/s 2024-04-14T15:09:57.091 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:56 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:58.113 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:57 smithi092 ceph-mon[33966]: mgrmap e16: y(active, since 4m), standbys: x 2024-04-14T15:09:58.221 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:58.217+0000 7fb647749700 1 -- 172.21.15.92:0/1035599944 >> v1:172.21.15.92:6789/0 conn(0x7fb640105c10 legacy=0x7fb640108060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:09:58.222 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:58.217+0000 7fb647749700 1 -- 172.21.15.92:0/1035599944 shutdown_connections 2024-04-14T15:09:58.222 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:58.217+0000 7fb647749700 1 -- 172.21.15.92:0/1035599944 >> 172.21.15.92:0/1035599944 conn(0x7fb6400fcd80 msgr2=0x7fb6400ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:09:58.222 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:58.217+0000 7fb647749700 1 -- 172.21.15.92:0/1035599944 shutdown_connections 2024-04-14T15:09:58.222 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:58.217+0000 7fb647749700 1 -- 172.21.15.92:0/1035599944 wait complete. 2024-04-14T15:09:58.222 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:58.217+0000 7fb647749700 1 librados: starting msgr at 2024-04-14T15:09:58.222 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:58.217+0000 7fb647749700 1 librados: starting objecter 2024-04-14T15:09:58.222 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:58.218+0000 7fb647749700 1 Processor -- start 2024-04-14T15:09:58.222 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:58.218+0000 7fb647749700 1 -- start start 2024-04-14T15:09:58.222 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:58.218+0000 7fb647749700 1 librados: setting wanted keys 2024-04-14T15:09:58.222 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:58.218+0000 7fb647749700 1 librados: calling monclient init 2024-04-14T15:09:58.222 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:58.218+0000 7fb647749700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb6401a5010 con 0x7fb6401017e0 2024-04-14T15:09:58.223 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:58.218+0000 7fb647749700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb6401a6250 con 0x7fb640105c10 2024-04-14T15:09:58.223 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:58.218+0000 7fb647749700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb6401a7490 con 0x7fb640109950 2024-04-14T15:09:58.223 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:58.218+0000 7fb644ce4700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fb640105c10 0x7fb64019fdf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:53550/0 (socket says 172.21.15.92:53550) 2024-04-14T15:09:58.223 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:58.218+0000 7fb644ce4700 1 -- 172.21.15.92:0/133232279 learned_addr learned my addr 172.21.15.92:0/133232279 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-04-14T15:09:58.223 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:58.218+0000 7fb6367fc700 10 client.?.objecter ms_handle_connect 0x7fb640105c10 2024-04-14T15:09:58.223 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:58.218+0000 7fb6367fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:09:58.223 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:58.218+0000 7fb6367fc700 10 client.?.objecter ms_handle_connect 0x7fb6401017e0 2024-04-14T15:09:58.223 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:58.218+0000 7fb6367fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:09:58.223 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:58.218+0000 7fb6367fc700 1 -- 172.21.15.92:0/133232279 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2664330361 0 0) 0x7fb6401a6250 con 0x7fb640105c10 2024-04-14T15:09:58.223 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:58.218+0000 7fb6367fc700 1 -- 172.21.15.92:0/133232279 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb62c0035b0 con 0x7fb640105c10 2024-04-14T15:09:58.223 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:58.218+0000 7fb6367fc700 10 client.?.objecter ms_handle_connect 0x7fb640109950 2024-04-14T15:09:58.223 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:58.218+0000 7fb6367fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:09:58.223 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:58.218+0000 7fb6367fc700 1 -- 172.21.15.92:0/133232279 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2532949623 0 0) 0x7fb6401a5010 con 0x7fb6401017e0 2024-04-14T15:09:58.223 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:58.218+0000 7fb6367fc700 1 -- 172.21.15.92:0/133232279 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6401a6250 con 0x7fb6401017e0 2024-04-14T15:09:58.224 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:58.218+0000 7fb6367fc700 1 -- 172.21.15.92:0/133232279 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1663062125 0 0) 0x7fb6401a7490 con 0x7fb640109950 2024-04-14T15:09:58.224 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:58.218+0000 7fb6367fc700 1 -- 172.21.15.92:0/133232279 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6401a5010 con 0x7fb640109950 2024-04-14T15:09:58.224 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:58.218+0000 7fb6367fc700 1 -- 172.21.15.92:0/133232279 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 811851232 0 0) 0x7fb62c0035b0 con 0x7fb640105c10 2024-04-14T15:09:58.224 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:58.218+0000 7fb6367fc700 1 -- 172.21.15.92:0/133232279 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb6401a7490 con 0x7fb640105c10 2024-04-14T15:09:58.224 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:58.218+0000 7fb6367fc700 1 -- 172.21.15.92:0/133232279 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fb630001430 con 0x7fb640105c10 2024-04-14T15:09:58.224 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:58.218+0000 7fb6367fc700 1 -- 172.21.15.92:0/133232279 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3976771561 0 0) 0x7fb6401a7490 con 0x7fb640105c10 2024-04-14T15:09:58.224 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:58.218+0000 7fb6367fc700 1 -- 172.21.15.92:0/133232279 >> v1:172.21.15.90:6790/0 conn(0x7fb640109950 legacy=0x7fb6401a3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:09:58.224 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:58.219+0000 7fb6367fc700 1 -- 172.21.15.92:0/133232279 >> v1:172.21.15.90:6789/0 conn(0x7fb6401017e0 legacy=0x7fb640107ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:09:58.224 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:58.219+0000 7fb6367fc700 1 -- 172.21.15.92:0/133232279 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6401a86d0 con 0x7fb640105c10 2024-04-14T15:09:58.224 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:58.219+0000 7fb647749700 1 -- 172.21.15.92:0/133232279 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb6401a5240 con 0x7fb640105c10 2024-04-14T15:09:58.224 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:58.219+0000 7fb647749700 10 client.24469.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:09:58.224 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:58.219+0000 7fb647749700 1 -- 172.21.15.92:0/133232279 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb6401a5780 con 0x7fb640105c10 2024-04-14T15:09:58.225 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:58.219+0000 7fb647749700 1 librados: init done 2024-04-14T15:09:58.225 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:58.219+0000 7fb6367fc700 1 -- 172.21.15.92:0/133232279 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fb630003720 con 0x7fb640105c10 2024-04-14T15:09:58.225 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:58.219+0000 7fb6367fc700 1 -- 172.21.15.92:0/133232279 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fb630004ea0 con 0x7fb640105c10 2024-04-14T15:09:58.225 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:58.219+0000 7fb647749700 1 -- 172.21.15.92:0/133232279 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb624005150 con 0x7fb640105c10 2024-04-14T15:09:58.225 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:58.220+0000 7fb6367fc700 1 -- 172.21.15.92:0/133232279 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 16) ==== 92674+0+0 (unknown 886309086 0 0) 0x7fb630003ab0 con 0x7fb640105c10 2024-04-14T15:09:58.225 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:58.221+0000 7fb6367fc700 1 -- 172.21.15.92:0/133232279 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(74..74 src has 12..74) ==== 5942+0+0 (unknown 4067678692 0 0) 0x7fb63008a570 con 0x7fb640105c10 2024-04-14T15:09:58.225 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:58.221+0000 7fb6367fc700 10 client.24469.objecter ms_dispatch 0x7fb640001550 osd_map(74..74 src has 12..74) 2024-04-14T15:09:58.225 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:58.221+0000 7fb6367fc700 3 client.24469.objecter handle_osd_map got epochs [74,74] > 0 2024-04-14T15:09:58.225 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:58.221+0000 7fb6367fc700 3 client.24469.objecter handle_osd_map decoding full epoch 74 2024-04-14T15:09:58.225 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:58.221+0000 7fb6367fc700 20 client.24469.objecter dump_active .. 0 homeless 2024-04-14T15:09:58.225 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:58.221+0000 7fb6367fc700 10 client.24469.objecter ms_handle_connect 0x7fb62c071380 2024-04-14T15:09:58.226 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:58.225+0000 7fb6367fc700 1 -- 172.21.15.92:0/133232279 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fb630057790 con 0x7fb640105c10 2024-04-14T15:09:58.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:57 smithi090 ceph-mon[30399]: mgrmap e16: y(active, since 4m), standbys: x 2024-04-14T15:09:58.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:57 smithi090 ceph-mon[41670]: mgrmap e16: y(active, since 4m), standbys: x 2024-04-14T15:09:58.466 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:58.466+0000 7fb647749700 1 -- 172.21.15.92:0/133232279 --> v1:172.21.15.90:6800/491071118 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi090=a;smithi092=b", "target": ["mon-mgr", ""]}) -- 0x7fb624000bc0 con 0x7fb62c071380 2024-04-14T15:09:58.475 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:58.474+0000 7fb6367fc700 1 -- 172.21.15.92:0/133232279 <== mgr.14148 v1:172.21.15.90:6800/491071118 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+34 (unknown 0 0 240551134) 0x7fb624000bc0 con 0x7fb62c071380 2024-04-14T15:09:58.475 INFO:teuthology.orchestra.run.smithi092.stdout:Scheduled node-exporter update... 2024-04-14T15:09:58.477 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:58.476+0000 7fb647749700 10 librados: watch_flush enter 2024-04-14T15:09:58.477 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:58.476+0000 7fb647749700 10 librados: watch_flush exit 2024-04-14T15:09:58.477 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:58.476+0000 7fb647749700 20 client.24469.objecter shutdown clearing up homeless session... 2024-04-14T15:09:58.477 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:58.476+0000 7fb647749700 10 client.24469.objecter successfully canceled tick 2024-04-14T15:09:58.477 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:58.476+0000 7fb647749700 1 -- 172.21.15.92:0/133232279 >> v1:172.21.15.90:6800/491071118 conn(0x7fb62c071380 legacy=0x7fb62c073840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:09:58.477 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:58.476+0000 7fb647749700 1 -- 172.21.15.92:0/133232279 >> v1:172.21.15.92:6789/0 conn(0x7fb640105c10 legacy=0x7fb64019fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:09:58.477 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:58.476+0000 7fb647749700 1 -- 172.21.15.92:0/133232279 shutdown_connections 2024-04-14T15:09:58.478 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:58.476+0000 7fb647749700 1 -- 172.21.15.92:0/133232279 >> 172.21.15.92:0/133232279 conn(0x7fb6400fcd80 msgr2=0x7fb64006aaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:09:58.478 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:58.476+0000 7fb647749700 1 -- 172.21.15.92:0/133232279 shutdown_connections 2024-04-14T15:09:58.478 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:58.476+0000 7fb647749700 1 -- 172.21.15.92:0/133232279 wait complete. 2024-04-14T15:09:58.478 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:09:58.477+0000 7fb647749700 1 librados: shutdown 2024-04-14T15:09:59.151 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:58 smithi092 ceph-mon[33966]: pgmap v210: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 4.4 KiB/s wr, 142 op/s 2024-04-14T15:09:59.151 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:58 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:59.177 DEBUG:teuthology.orchestra.run.smithi090:node-exporter.a> sudo journalctl -f -n 0 -u ceph-0e288098-fa70-11ee-bc8e-c7b262605968@node-exporter.a.service 2024-04-14T15:09:59.180 DEBUG:teuthology.orchestra.run.smithi092:node-exporter.b> sudo journalctl -f -n 0 -u ceph-0e288098-fa70-11ee-bc8e-c7b262605968@node-exporter.b.service 2024-04-14T15:09:59.182 INFO:tasks.cephadm:Adding alertmanager.a on smithi090 2024-04-14T15:09:59.183 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph orch apply alertmanager '1;smithi090=a' 2024-04-14T15:09:59.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:58 smithi090 ceph-mon[30399]: pgmap v210: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 4.4 KiB/s wr, 142 op/s 2024-04-14T15:09:59.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:58 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:59.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:58 smithi090 ceph-mon[41670]: pgmap v210: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 4.4 KiB/s wr, 142 op/s 2024-04-14T15:09:59.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:58 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:09:59.644 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:10:00.022 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:59 smithi092 ceph-mon[33966]: from='client.24469 v1:172.21.15.92:0/133232279' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi090=a;smithi092=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:10:00.022 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:59 smithi092 ceph-mon[33966]: Saving service node-exporter spec with placement smithi090=a;smithi092=b;count:2 2024-04-14T15:10:00.022 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:59 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:10:00.022 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:59 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:10:00.022 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:59 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:10:00.022 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:59 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:10:00.022 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:09:59 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:10:00.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:59 smithi090 ceph-mon[30399]: from='client.24469 v1:172.21.15.92:0/133232279' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi090=a;smithi092=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:10:00.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:59 smithi090 ceph-mon[30399]: Saving service node-exporter spec with placement smithi090=a;smithi092=b;count:2 2024-04-14T15:10:00.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:59 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:10:00.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:59 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:10:00.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:59 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:10:00.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:59 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:10:00.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:09:59 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:10:00.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:59 smithi090 ceph-mon[41670]: from='client.24469 v1:172.21.15.92:0/133232279' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi090=a;smithi092=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:10:00.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:59 smithi090 ceph-mon[41670]: Saving service node-exporter spec with placement smithi090=a;smithi092=b;count:2 2024-04-14T15:10:00.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:59 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:10:00.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:59 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:10:00.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:59 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:10:00.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:59 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:10:00.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:09:59 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:10:01.057 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:00 smithi092 ceph-mon[33966]: Deploying daemon prometheus.a on smithi092 2024-04-14T15:10:01.057 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:00 smithi092 ceph-mon[33966]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 4.2 KiB/s wr, 140 op/s 2024-04-14T15:10:01.057 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:00 smithi092 ceph-mon[33966]: overall HEALTH_OK 2024-04-14T15:10:01.057 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:00 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:10:01.295 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:01.274+0000 7f970f75d700 1 -- 172.21.15.92:0/1383373295 >> v1:172.21.15.92:6789/0 conn(0x7f9708109950 legacy=0x7f970810be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:01.296 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:01.274+0000 7f970f75d700 1 -- 172.21.15.92:0/1383373295 shutdown_connections 2024-04-14T15:10:01.296 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:01.274+0000 7f970f75d700 1 -- 172.21.15.92:0/1383373295 >> 172.21.15.92:0/1383373295 conn(0x7f97080fcd80 msgr2=0x7f97080ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:10:01.296 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:01.274+0000 7f970f75d700 1 -- 172.21.15.92:0/1383373295 shutdown_connections 2024-04-14T15:10:01.296 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:01.274+0000 7f970f75d700 1 -- 172.21.15.92:0/1383373295 wait complete. 2024-04-14T15:10:01.296 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:01.275+0000 7f970f75d700 1 librados: starting msgr at 2024-04-14T15:10:01.296 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:01.275+0000 7f970f75d700 1 librados: starting objecter 2024-04-14T15:10:01.296 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:01.275+0000 7f970f75d700 1 Processor -- start 2024-04-14T15:10:01.296 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:01.275+0000 7f970f75d700 1 -- start start 2024-04-14T15:10:01.296 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:01.275+0000 7f970f75d700 1 librados: setting wanted keys 2024-04-14T15:10:01.296 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:01.275+0000 7f970f75d700 1 librados: calling monclient init 2024-04-14T15:10:01.296 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:01.275+0000 7f970f75d700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f97081a5270 con 0x7f9708105c10 2024-04-14T15:10:01.297 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:01.275+0000 7f970f75d700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f97081a64b0 con 0x7f9708109950 2024-04-14T15:10:01.297 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:01.275+0000 7f970f75d700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f97081a76f0 con 0x7f97081017e0 2024-04-14T15:10:01.297 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:01.275+0000 7f970dcfa700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f9708109950 0x7f97081a3930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:53580/0 (socket says 172.21.15.92:53580) 2024-04-14T15:10:01.297 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:01.275+0000 7f970dcfa700 1 -- 172.21.15.92:0/1660582761 learned_addr learned my addr 172.21.15.92:0/1660582761 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-04-14T15:10:01.297 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:01.275+0000 7f96fe7fc700 10 client.?.objecter ms_handle_connect 0x7f9708109950 2024-04-14T15:10:01.297 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:01.275+0000 7f96fe7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:10:01.297 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:01.275+0000 7f96fe7fc700 10 client.?.objecter ms_handle_connect 0x7f9708105c10 2024-04-14T15:10:01.297 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:01.275+0000 7f96fe7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:10:01.297 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:01.275+0000 7f96fe7fc700 10 client.?.objecter ms_handle_connect 0x7f97081017e0 2024-04-14T15:10:01.297 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:01.275+0000 7f96fe7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:10:01.297 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:01.275+0000 7f96fe7fc700 1 -- 172.21.15.92:0/1660582761 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1379101675 0 0) 0x7f97081a64b0 con 0x7f9708109950 2024-04-14T15:10:01.297 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:01.276+0000 7f96fe7fc700 1 -- 172.21.15.92:0/1660582761 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f96f40035b0 con 0x7f9708109950 2024-04-14T15:10:01.298 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:01.276+0000 7f96fe7fc700 1 -- 172.21.15.92:0/1660582761 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1045903386 0 0) 0x7f97081a5270 con 0x7f9708105c10 2024-04-14T15:10:01.298 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:01.276+0000 7f96fe7fc700 1 -- 172.21.15.92:0/1660582761 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f97081a64b0 con 0x7f9708105c10 2024-04-14T15:10:01.298 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:01.276+0000 7f96fe7fc700 1 -- 172.21.15.92:0/1660582761 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1538377689 0 0) 0x7f97081a76f0 con 0x7f97081017e0 2024-04-14T15:10:01.298 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:01.276+0000 7f96fe7fc700 1 -- 172.21.15.92:0/1660582761 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f97081a5270 con 0x7f97081017e0 2024-04-14T15:10:01.298 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:01.276+0000 7f96fe7fc700 1 -- 172.21.15.92:0/1660582761 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3268314793 0 0) 0x7f96f40035b0 con 0x7f9708109950 2024-04-14T15:10:01.298 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:01.276+0000 7f96fe7fc700 1 -- 172.21.15.92:0/1660582761 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f97081a76f0 con 0x7f9708109950 2024-04-14T15:10:01.298 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:01.276+0000 7f96fe7fc700 1 -- 172.21.15.92:0/1660582761 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f97000017b0 con 0x7f9708109950 2024-04-14T15:10:01.298 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:01.276+0000 7f96fe7fc700 1 -- 172.21.15.92:0/1660582761 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1744438824 0 0) 0x7f97081a64b0 con 0x7f9708105c10 2024-04-14T15:10:01.298 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:01.276+0000 7f96fe7fc700 1 -- 172.21.15.92:0/1660582761 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f96f40035b0 con 0x7f9708105c10 2024-04-14T15:10:01.298 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:01.276+0000 7f96fe7fc700 1 -- 172.21.15.92:0/1660582761 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f96f80017e0 con 0x7f9708105c10 2024-04-14T15:10:01.298 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:01.276+0000 7f96fe7fc700 1 -- 172.21.15.92:0/1660582761 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4172713776 0 0) 0x7f97081a76f0 con 0x7f9708109950 2024-04-14T15:10:01.299 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:01.276+0000 7f96fe7fc700 1 -- 172.21.15.92:0/1660582761 >> v1:172.21.15.90:6790/0 conn(0x7f97081017e0 legacy=0x7f9708107a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:01.299 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:01.276+0000 7f96fe7fc700 1 -- 172.21.15.92:0/1660582761 >> v1:172.21.15.90:6789/0 conn(0x7f9708105c10 legacy=0x7f97081a0090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:01.299 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:01.276+0000 7f96fe7fc700 1 -- 172.21.15.92:0/1660582761 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97081a8930 con 0x7f9708109950 2024-04-14T15:10:01.299 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:01.276+0000 7f970f75d700 1 -- 172.21.15.92:0/1660582761 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f97081a54a0 con 0x7f9708109950 2024-04-14T15:10:01.299 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:01.276+0000 7f970f75d700 10 client.24475.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:10:01.299 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:01.276+0000 7f970f75d700 1 -- 172.21.15.92:0/1660582761 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f97081a59e0 con 0x7f9708109950 2024-04-14T15:10:01.299 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:01.276+0000 7f970f75d700 1 librados: init done 2024-04-14T15:10:01.299 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:01.276+0000 7f96fe7fc700 1 -- 172.21.15.92:0/1660582761 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f9700003a00 con 0x7f9708109950 2024-04-14T15:10:01.299 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:01.276+0000 7f96fe7fc700 1 -- 172.21.15.92:0/1660582761 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f9700005200 con 0x7f9708109950 2024-04-14T15:10:01.299 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:01.277+0000 7f970f75d700 1 -- 172.21.15.92:0/1660582761 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96ec005150 con 0x7f9708109950 2024-04-14T15:10:01.299 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:01.278+0000 7f96fe7fc700 1 -- 172.21.15.92:0/1660582761 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 16) ==== 92674+0+0 (unknown 886309086 0 0) 0x7f9700001ef0 con 0x7f9708109950 2024-04-14T15:10:01.299 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:01.278+0000 7f96fe7fc700 1 -- 172.21.15.92:0/1660582761 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(74..74 src has 12..74) ==== 5942+0+0 (unknown 4067678692 0 0) 0x7f970008a500 con 0x7f9708109950 2024-04-14T15:10:01.299 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:01.278+0000 7f96fe7fc700 10 client.24475.objecter ms_dispatch 0x7f9708001550 osd_map(74..74 src has 12..74) 2024-04-14T15:10:01.299 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:01.278+0000 7f96fe7fc700 3 client.24475.objecter handle_osd_map got epochs [74,74] > 0 2024-04-14T15:10:01.300 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:01.278+0000 7f96fe7fc700 3 client.24475.objecter handle_osd_map decoding full epoch 74 2024-04-14T15:10:01.300 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:01.278+0000 7f96fe7fc700 20 client.24475.objecter dump_active .. 0 homeless 2024-04-14T15:10:01.300 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:01.278+0000 7f96fe7fc700 10 client.24475.objecter ms_handle_connect 0x7f96f40710f0 2024-04-14T15:10:01.300 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:01.283+0000 7f96fe7fc700 1 -- 172.21.15.92:0/1660582761 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f9700057740 con 0x7f9708109950 2024-04-14T15:10:01.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:00 smithi090 ceph-mon[41670]: Deploying daemon prometheus.a on smithi092 2024-04-14T15:10:01.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:00 smithi090 ceph-mon[41670]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 4.2 KiB/s wr, 140 op/s 2024-04-14T15:10:01.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:00 smithi090 ceph-mon[41670]: overall HEALTH_OK 2024-04-14T15:10:01.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:00 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:10:01.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:00 smithi090 ceph-mon[30399]: Deploying daemon prometheus.a on smithi092 2024-04-14T15:10:01.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:00 smithi090 ceph-mon[30399]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 4.2 KiB/s wr, 140 op/s 2024-04-14T15:10:01.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:00 smithi090 ceph-mon[30399]: overall HEALTH_OK 2024-04-14T15:10:01.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:00 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:10:01.501 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:01.501+0000 7f970f75d700 1 -- 172.21.15.92:0/1660582761 --> v1:172.21.15.90:6800/491071118 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi090=a", "target": ["mon-mgr", ""]}) -- 0x7f96ec000bc0 con 0x7f96f40710f0 2024-04-14T15:10:01.509 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:01.508+0000 7f96fe7fc700 1 -- 172.21.15.92:0/1660582761 <== mgr.14148 v1:172.21.15.90:6800/491071118 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+33 (unknown 0 0 1850065467) 0x7f96ec000bc0 con 0x7f96f40710f0 2024-04-14T15:10:01.509 INFO:teuthology.orchestra.run.smithi092.stdout:Scheduled alertmanager update... 2024-04-14T15:10:01.511 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:01.510+0000 7f970f75d700 10 librados: watch_flush enter 2024-04-14T15:10:01.511 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:01.510+0000 7f970f75d700 10 librados: watch_flush exit 2024-04-14T15:10:01.511 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:01.510+0000 7f970f75d700 20 client.24475.objecter shutdown clearing up homeless session... 2024-04-14T15:10:01.512 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:01.510+0000 7f970f75d700 10 client.24475.objecter successfully canceled tick 2024-04-14T15:10:01.512 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:01.510+0000 7f970f75d700 1 -- 172.21.15.92:0/1660582761 >> v1:172.21.15.90:6800/491071118 conn(0x7f96f40710f0 legacy=0x7f96f40735b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:01.512 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:01.510+0000 7f970f75d700 1 -- 172.21.15.92:0/1660582761 >> v1:172.21.15.92:6789/0 conn(0x7f9708109950 legacy=0x7f97081a3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:01.512 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:01.510+0000 7f970f75d700 1 -- 172.21.15.92:0/1660582761 shutdown_connections 2024-04-14T15:10:01.512 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:01.510+0000 7f970f75d700 1 -- 172.21.15.92:0/1660582761 >> 172.21.15.92:0/1660582761 conn(0x7f97080fcd80 msgr2=0x7f9708109240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:10:01.512 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:01.511+0000 7f970f75d700 1 -- 172.21.15.92:0/1660582761 shutdown_connections 2024-04-14T15:10:01.512 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:01.511+0000 7f970f75d700 1 -- 172.21.15.92:0/1660582761 wait complete. 2024-04-14T15:10:01.512 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:01.511+0000 7f970f75d700 1 librados: shutdown 2024-04-14T15:10:02.532 DEBUG:teuthology.orchestra.run.smithi090:alertmanager.a> sudo journalctl -f -n 0 -u ceph-0e288098-fa70-11ee-bc8e-c7b262605968@alertmanager.a.service 2024-04-14T15:10:02.536 INFO:tasks.cephadm:Adding grafana.a on smithi092 2024-04-14T15:10:02.536 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph orch apply grafana '1;smithi092=a' 2024-04-14T15:10:02.796 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:10:02.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:02 smithi092 ceph-mon[33966]: from='client.24475 v1:172.21.15.92:0/1660582761' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi090=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:10:02.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:02 smithi092 ceph-mon[33966]: Saving service alertmanager spec with placement smithi090=a;count:1 2024-04-14T15:10:02.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:02 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:10:02.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:02 smithi092 ceph-mon[33966]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 1.4 KiB/s wr, 51 op/s 2024-04-14T15:10:02.941 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:02 smithi090 ceph-mon[30399]: from='client.24475 v1:172.21.15.92:0/1660582761' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi090=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:10:02.941 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:02 smithi090 ceph-mon[30399]: Saving service alertmanager spec with placement smithi090=a;count:1 2024-04-14T15:10:02.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:02 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:10:02.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:02 smithi090 ceph-mon[30399]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 1.4 KiB/s wr, 51 op/s 2024-04-14T15:10:02.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:02 smithi090 ceph-mon[41670]: from='client.24475 v1:172.21.15.92:0/1660582761' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi090=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:10:02.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:02 smithi090 ceph-mon[41670]: Saving service alertmanager spec with placement smithi090=a;count:1 2024-04-14T15:10:02.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:02 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:10:02.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:02 smithi090 ceph-mon[41670]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 1.4 KiB/s wr, 51 op/s 2024-04-14T15:10:04.872 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:10:04 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:10:05.177 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:04 smithi092 ceph-mon[33966]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 818 B/s wr, 27 op/s 2024-04-14T15:10:05.218 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:05.176+0000 7ff8df7f8700 1 -- 172.21.15.92:0/3211031027 >> v1:172.21.15.92:6789/0 conn(0x7ff8d8105d00 legacy=0x7ff8d81080f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:05.219 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:05.176+0000 7ff8df7f8700 1 -- 172.21.15.92:0/3211031027 shutdown_connections 2024-04-14T15:10:05.219 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:05.176+0000 7ff8df7f8700 1 -- 172.21.15.92:0/3211031027 >> 172.21.15.92:0/3211031027 conn(0x7ff8d80fc550 msgr2=0x7ff8d80fe970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:10:05.219 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:05.176+0000 7ff8df7f8700 1 -- 172.21.15.92:0/3211031027 shutdown_connections 2024-04-14T15:10:05.219 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:05.176+0000 7ff8df7f8700 1 -- 172.21.15.92:0/3211031027 wait complete. 2024-04-14T15:10:05.219 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:05.177+0000 7ff8df7f8700 1 librados: starting msgr at 2024-04-14T15:10:05.219 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:05.177+0000 7ff8df7f8700 1 librados: starting objecter 2024-04-14T15:10:05.219 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:05.177+0000 7ff8df7f8700 1 Processor -- start 2024-04-14T15:10:05.219 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:05.177+0000 7ff8df7f8700 1 -- start start 2024-04-14T15:10:05.219 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:05.177+0000 7ff8df7f8700 1 librados: setting wanted keys 2024-04-14T15:10:05.219 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:05.177+0000 7ff8df7f8700 1 librados: calling monclient init 2024-04-14T15:10:05.219 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:05.177+0000 7ff8df7f8700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff8d81a5220 con 0x7ff8d8109960 2024-04-14T15:10:05.219 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:05.177+0000 7ff8df7f8700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff8d81a6460 con 0x7ff8d8105d00 2024-04-14T15:10:05.219 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:05.177+0000 7ff8df7f8700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff8d81a76a0 con 0x7ff8d8102e10 2024-04-14T15:10:05.220 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:05.177+0000 7ff8dcd93700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7ff8d8105d00 0x7ff8d81a0000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:53600/0 (socket says 172.21.15.92:53600) 2024-04-14T15:10:05.220 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:05.177+0000 7ff8dcd93700 1 -- 172.21.15.92:0/2627173071 learned_addr learned my addr 172.21.15.92:0/2627173071 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-04-14T15:10:05.220 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:05.177+0000 7ff8ce7fc700 10 client.?.objecter ms_handle_connect 0x7ff8d8105d00 2024-04-14T15:10:05.220 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:05.177+0000 7ff8ce7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:10:05.220 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:05.178+0000 7ff8ce7fc700 10 client.?.objecter ms_handle_connect 0x7ff8d8109960 2024-04-14T15:10:05.220 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:05.178+0000 7ff8ce7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:10:05.220 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:05.178+0000 7ff8ce7fc700 10 client.?.objecter ms_handle_connect 0x7ff8d8102e10 2024-04-14T15:10:05.220 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:05.178+0000 7ff8ce7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:10:05.220 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:05.178+0000 7ff8ce7fc700 1 -- 172.21.15.92:0/2627173071 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2550693628 0 0) 0x7ff8d81a6460 con 0x7ff8d8105d00 2024-04-14T15:10:05.220 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:05.178+0000 7ff8ce7fc700 1 -- 172.21.15.92:0/2627173071 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff8c40035b0 con 0x7ff8d8105d00 2024-04-14T15:10:05.220 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:05.178+0000 7ff8ce7fc700 1 -- 172.21.15.92:0/2627173071 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1698110932 0 0) 0x7ff8d81a5220 con 0x7ff8d8109960 2024-04-14T15:10:05.220 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:05.178+0000 7ff8ce7fc700 1 -- 172.21.15.92:0/2627173071 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff8d81a6460 con 0x7ff8d8109960 2024-04-14T15:10:05.220 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:05.178+0000 7ff8ce7fc700 1 -- 172.21.15.92:0/2627173071 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2200427658 0 0) 0x7ff8d81a76a0 con 0x7ff8d8102e10 2024-04-14T15:10:05.220 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:05.178+0000 7ff8ce7fc700 1 -- 172.21.15.92:0/2627173071 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff8d81a5220 con 0x7ff8d8102e10 2024-04-14T15:10:05.221 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:05.178+0000 7ff8ce7fc700 1 -- 172.21.15.92:0/2627173071 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3118841638 0 0) 0x7ff8c40035b0 con 0x7ff8d8105d00 2024-04-14T15:10:05.221 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:05.178+0000 7ff8ce7fc700 1 -- 172.21.15.92:0/2627173071 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff8d81a76a0 con 0x7ff8d8105d00 2024-04-14T15:10:05.221 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:05.178+0000 7ff8ce7fc700 1 -- 172.21.15.92:0/2627173071 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7ff8c8001430 con 0x7ff8d8105d00 2024-04-14T15:10:05.221 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:05.178+0000 7ff8ce7fc700 1 -- 172.21.15.92:0/2627173071 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 33114122 0 0) 0x7ff8d81a6460 con 0x7ff8d8109960 2024-04-14T15:10:05.221 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:05.178+0000 7ff8ce7fc700 1 -- 172.21.15.92:0/2627173071 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff8c40035b0 con 0x7ff8d8109960 2024-04-14T15:10:05.221 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:05.178+0000 7ff8ce7fc700 1 -- 172.21.15.92:0/2627173071 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7ff8d0001aa0 con 0x7ff8d8109960 2024-04-14T15:10:05.221 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:05.178+0000 7ff8ce7fc700 1 -- 172.21.15.92:0/2627173071 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 187179858 0 0) 0x7ff8d81a76a0 con 0x7ff8d8105d00 2024-04-14T15:10:05.221 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:05.178+0000 7ff8ce7fc700 1 -- 172.21.15.92:0/2627173071 >> v1:172.21.15.90:6790/0 conn(0x7ff8d8102e10 legacy=0x7ff8d80fe3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:05.221 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:05.178+0000 7ff8ce7fc700 1 -- 172.21.15.92:0/2627173071 >> v1:172.21.15.90:6789/0 conn(0x7ff8d8109960 legacy=0x7ff8d81a3920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:05.221 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:05.178+0000 7ff8ce7fc700 1 -- 172.21.15.92:0/2627173071 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8d81a88e0 con 0x7ff8d8105d00 2024-04-14T15:10:05.221 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:05.178+0000 7ff8df7f8700 1 -- 172.21.15.92:0/2627173071 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff8d81a5450 con 0x7ff8d8105d00 2024-04-14T15:10:05.221 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:05.178+0000 7ff8df7f8700 10 client.24481.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:10:05.221 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:05.178+0000 7ff8df7f8700 1 -- 172.21.15.92:0/2627173071 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff8d81a5990 con 0x7ff8d8105d00 2024-04-14T15:10:05.221 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:05.178+0000 7ff8df7f8700 1 librados: init done 2024-04-14T15:10:05.222 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:05.178+0000 7ff8ce7fc700 1 -- 172.21.15.92:0/2627173071 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7ff8c80016e0 con 0x7ff8d8105d00 2024-04-14T15:10:05.222 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:05.178+0000 7ff8ce7fc700 1 -- 172.21.15.92:0/2627173071 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7ff8c8004ea0 con 0x7ff8d8105d00 2024-04-14T15:10:05.222 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:05.179+0000 7ff8df7f8700 1 -- 172.21.15.92:0/2627173071 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8bc005150 con 0x7ff8d8105d00 2024-04-14T15:10:05.222 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:05.180+0000 7ff8ce7fc700 1 -- 172.21.15.92:0/2627173071 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 16) ==== 92674+0+0 (unknown 886309086 0 0) 0x7ff8c8003ab0 con 0x7ff8d8105d00 2024-04-14T15:10:05.222 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:05.180+0000 7ff8ce7fc700 1 -- 172.21.15.92:0/2627173071 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(74..74 src has 12..74) ==== 5942+0+0 (unknown 4067678692 0 0) 0x7ff8c808a340 con 0x7ff8d8105d00 2024-04-14T15:10:05.222 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:05.180+0000 7ff8ce7fc700 10 client.24481.objecter ms_dispatch 0x7ff8d8001550 osd_map(74..74 src has 12..74) 2024-04-14T15:10:05.222 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:05.180+0000 7ff8ce7fc700 3 client.24481.objecter handle_osd_map got epochs [74,74] > 0 2024-04-14T15:10:05.222 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:05.180+0000 7ff8ce7fc700 3 client.24481.objecter handle_osd_map decoding full epoch 74 2024-04-14T15:10:05.222 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:05.180+0000 7ff8ce7fc700 20 client.24481.objecter dump_active .. 0 homeless 2024-04-14T15:10:05.222 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:05.180+0000 7ff8ce7fc700 10 client.24481.objecter ms_handle_connect 0x7ff8c4071080 2024-04-14T15:10:05.222 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:05.185+0000 7ff8ce7fc700 1 -- 172.21.15.92:0/2627173071 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7ff8c8057580 con 0x7ff8d8105d00 2024-04-14T15:10:05.400 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:05.399+0000 7ff8df7f8700 1 -- 172.21.15.92:0/2627173071 --> v1:172.21.15.90:6800/491071118 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi092=a", "target": ["mon-mgr", ""]}) -- 0x7ff8bc000bc0 con 0x7ff8c4071080 2024-04-14T15:10:05.407 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:05.406+0000 7ff8ce7fc700 1 -- 172.21.15.92:0/2627173071 <== mgr.14148 v1:172.21.15.90:6800/491071118 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (unknown 0 0 664801700) 0x7ff8bc000bc0 con 0x7ff8c4071080 2024-04-14T15:10:05.407 INFO:teuthology.orchestra.run.smithi092.stdout:Scheduled grafana update... 2024-04-14T15:10:05.409 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:05.408+0000 7ff8df7f8700 10 librados: watch_flush enter 2024-04-14T15:10:05.409 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:05.408+0000 7ff8df7f8700 10 librados: watch_flush exit 2024-04-14T15:10:05.409 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:05.408+0000 7ff8df7f8700 20 client.24481.objecter shutdown clearing up homeless session... 2024-04-14T15:10:05.409 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:05.408+0000 7ff8df7f8700 10 client.24481.objecter successfully canceled tick 2024-04-14T15:10:05.409 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:05.408+0000 7ff8df7f8700 1 -- 172.21.15.92:0/2627173071 >> v1:172.21.15.90:6800/491071118 conn(0x7ff8c4071080 legacy=0x7ff8c4073540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:05.409 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:05.408+0000 7ff8df7f8700 1 -- 172.21.15.92:0/2627173071 >> v1:172.21.15.92:6789/0 conn(0x7ff8d8105d00 legacy=0x7ff8d81a0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:05.409 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:05.408+0000 7ff8df7f8700 1 -- 172.21.15.92:0/2627173071 shutdown_connections 2024-04-14T15:10:05.409 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:05.408+0000 7ff8df7f8700 1 -- 172.21.15.92:0/2627173071 >> 172.21.15.92:0/2627173071 conn(0x7ff8d80fc550 msgr2=0x7ff8d81060e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:10:05.410 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:05.408+0000 7ff8df7f8700 1 -- 172.21.15.92:0/2627173071 shutdown_connections 2024-04-14T15:10:05.410 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:05.408+0000 7ff8df7f8700 1 -- 172.21.15.92:0/2627173071 wait complete. 2024-04-14T15:10:05.410 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:05.408+0000 7ff8df7f8700 1 librados: shutdown 2024-04-14T15:10:05.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:04 smithi090 ceph-mon[41670]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 818 B/s wr, 27 op/s 2024-04-14T15:10:05.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:04 smithi090 ceph-mon[30399]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 818 B/s wr, 27 op/s 2024-04-14T15:10:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:05 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:10:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:05 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:10:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:05 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]: dispatch 2024-04-14T15:10:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:05 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-04-14T15:10:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:05 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-04-14T15:10:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:05 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-04-14T15:10:06.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:05 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:10:06.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:05 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:10:06.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:05 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]: dispatch 2024-04-14T15:10:06.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:05 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-04-14T15:10:06.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:05 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-04-14T15:10:06.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:05 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-04-14T15:10:06.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:05 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:10:06.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:05 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:10:06.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:05 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]: dispatch 2024-04-14T15:10:06.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:05 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-04-14T15:10:06.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:05 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-04-14T15:10:06.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:05 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-04-14T15:10:07.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:06 smithi092 ceph-mon[33966]: from='client.24481 v1:172.21.15.92:0/2627173071' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi092=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:10:07.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:06 smithi092 ceph-mon[33966]: Saving service grafana spec with placement smithi092=a;count:1 2024-04-14T15:10:07.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:06 smithi092 ceph-mon[33966]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 803 B/s rd, 100 B/s wr, 1 op/s 2024-04-14T15:10:07.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:06 smithi092 ceph-mon[33966]: 1.0 scrub starts 2024-04-14T15:10:07.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:06 smithi092 ceph-mon[33966]: 1.0 scrub ok 2024-04-14T15:10:07.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:06 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]': finished 2024-04-14T15:10:07.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:06 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-04-14T15:10:07.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:06 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-04-14T15:10:07.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:06 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-04-14T15:10:07.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:06 smithi092 ceph-mon[33966]: osdmap e75: 8 total, 8 up, 8 in 2024-04-14T15:10:07.156 DEBUG:teuthology.orchestra.run.smithi092:grafana.a> sudo journalctl -f -n 0 -u ceph-0e288098-fa70-11ee-bc8e-c7b262605968@grafana.a.service 2024-04-14T15:10:07.158 INFO:tasks.cephadm:Setting up client nodes... 2024-04-14T15:10:07.158 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-14T15:10:07.409 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:10:07.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:06 smithi090 ceph-mon[30399]: from='client.24481 v1:172.21.15.92:0/2627173071' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi092=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:10:07.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:06 smithi090 ceph-mon[30399]: Saving service grafana spec with placement smithi092=a;count:1 2024-04-14T15:10:07.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:06 smithi090 ceph-mon[30399]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 803 B/s rd, 100 B/s wr, 1 op/s 2024-04-14T15:10:07.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:06 smithi090 ceph-mon[30399]: 1.0 scrub starts 2024-04-14T15:10:07.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:06 smithi090 ceph-mon[30399]: 1.0 scrub ok 2024-04-14T15:10:07.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:06 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]': finished 2024-04-14T15:10:07.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:06 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-04-14T15:10:07.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:06 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-04-14T15:10:07.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:06 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-04-14T15:10:07.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:06 smithi090 ceph-mon[30399]: osdmap e75: 8 total, 8 up, 8 in 2024-04-14T15:10:07.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:06 smithi090 ceph-mon[41670]: from='client.24481 v1:172.21.15.92:0/2627173071' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi092=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:10:07.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:06 smithi090 ceph-mon[41670]: Saving service grafana spec with placement smithi092=a;count:1 2024-04-14T15:10:07.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:06 smithi090 ceph-mon[41670]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 803 B/s rd, 100 B/s wr, 1 op/s 2024-04-14T15:10:07.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:06 smithi090 ceph-mon[41670]: 1.0 scrub starts 2024-04-14T15:10:07.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:06 smithi090 ceph-mon[41670]: 1.0 scrub ok 2024-04-14T15:10:07.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:06 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]': finished 2024-04-14T15:10:07.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:06 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-04-14T15:10:07.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:06 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-04-14T15:10:07.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:06 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-04-14T15:10:07.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:06 smithi090 ceph-mon[41670]: osdmap e75: 8 total, 8 up, 8 in 2024-04-14T15:10:08.814 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:08 smithi090 ceph-mon[41670]: osdmap e76: 8 total, 8 up, 8 in 2024-04-14T15:10:08.814 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:08 smithi090 ceph-mon[41670]: pgmap v217: 132 pgs: 2 peering, 130 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:10:08.814 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:08 smithi090 ceph-mon[30399]: osdmap e76: 8 total, 8 up, 8 in 2024-04-14T15:10:08.814 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:08 smithi090 ceph-mon[30399]: pgmap v217: 132 pgs: 2 peering, 130 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:10:09.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.053+0000 7fbb5b3e8700 1 -- 172.21.15.90:0/3107857045 >> v1:172.21.15.90:6789/0 conn(0x7fbb541017b0 legacy=0x7fbb54101bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:09.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.053+0000 7fbb5b3e8700 1 -- 172.21.15.90:0/3107857045 shutdown_connections 2024-04-14T15:10:09.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.053+0000 7fbb5b3e8700 1 -- 172.21.15.90:0/3107857045 >> 172.21.15.90:0/3107857045 conn(0x7fbb540fcd50 msgr2=0x7fbb540ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:10:09.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.053+0000 7fbb5b3e8700 1 -- 172.21.15.90:0/3107857045 shutdown_connections 2024-04-14T15:10:09.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.053+0000 7fbb5b3e8700 1 -- 172.21.15.90:0/3107857045 wait complete. 2024-04-14T15:10:09.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.053+0000 7fbb5b3e8700 1 librados: starting msgr at 2024-04-14T15:10:09.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.053+0000 7fbb5b3e8700 1 librados: starting objecter 2024-04-14T15:10:09.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.053+0000 7fbb5b3e8700 1 Processor -- start 2024-04-14T15:10:09.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.054+0000 7fbb5b3e8700 1 -- start start 2024-04-14T15:10:09.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.054+0000 7fbb5b3e8700 1 librados: setting wanted keys 2024-04-14T15:10:09.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.054+0000 7fbb5b3e8700 1 librados: calling monclient init 2024-04-14T15:10:09.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.054+0000 7fbb5b3e8700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbb5406b120 con 0x7fbb54105be0 2024-04-14T15:10:09.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.054+0000 7fbb5b3e8700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbb5406b350 con 0x7fbb541017b0 2024-04-14T15:10:09.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.054+0000 7fbb5b3e8700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbb541a76a0 con 0x7fbb54109920 2024-04-14T15:10:09.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.054+0000 7fbb58983700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fbb54105be0 0x7fbb541a07f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:56264/0 (socket says 172.21.15.90:56264) 2024-04-14T15:10:09.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.054+0000 7fbb58983700 1 -- 172.21.15.90:0/3198137979 learned_addr learned my addr 172.21.15.90:0/3198137979 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:10:09.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.054+0000 7fbb4a7fc700 10 client.?.objecter ms_handle_connect 0x7fbb54105be0 2024-04-14T15:10:09.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.054+0000 7fbb4a7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:10:09.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.054+0000 7fbb4a7fc700 10 client.?.objecter ms_handle_connect 0x7fbb54109920 2024-04-14T15:10:09.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.054+0000 7fbb4a7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:10:09.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.054+0000 7fbb4a7fc700 10 client.?.objecter ms_handle_connect 0x7fbb541017b0 2024-04-14T15:10:09.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.054+0000 7fbb4a7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:10:09.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.054+0000 7fbb4a7fc700 1 -- 172.21.15.90:0/3198137979 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3455622409 0 0) 0x7fbb5406b120 con 0x7fbb54105be0 2024-04-14T15:10:09.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.054+0000 7fbb4a7fc700 1 -- 172.21.15.90:0/3198137979 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb400035b0 con 0x7fbb54105be0 2024-04-14T15:10:09.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.054+0000 7fbb4a7fc700 1 -- 172.21.15.90:0/3198137979 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3634151876 0 0) 0x7fbb541a76a0 con 0x7fbb54109920 2024-04-14T15:10:09.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.054+0000 7fbb4a7fc700 1 -- 172.21.15.90:0/3198137979 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb5406b120 con 0x7fbb54109920 2024-04-14T15:10:09.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.054+0000 7fbb4a7fc700 1 -- 172.21.15.90:0/3198137979 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 538912515 0 0) 0x7fbb5406b350 con 0x7fbb541017b0 2024-04-14T15:10:09.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.054+0000 7fbb4a7fc700 1 -- 172.21.15.90:0/3198137979 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb541a76a0 con 0x7fbb541017b0 2024-04-14T15:10:09.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.054+0000 7fbb4a7fc700 1 -- 172.21.15.90:0/3198137979 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1608190339 0 0) 0x7fbb400035b0 con 0x7fbb54105be0 2024-04-14T15:10:09.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.054+0000 7fbb4a7fc700 1 -- 172.21.15.90:0/3198137979 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbb5406b350 con 0x7fbb54105be0 2024-04-14T15:10:09.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.054+0000 7fbb4a7fc700 1 -- 172.21.15.90:0/3198137979 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fbb440018b0 con 0x7fbb54105be0 2024-04-14T15:10:09.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.054+0000 7fbb4a7fc700 1 -- 172.21.15.90:0/3198137979 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2082525878 0 0) 0x7fbb5406b120 con 0x7fbb54109920 2024-04-14T15:10:09.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.054+0000 7fbb4a7fc700 1 -- 172.21.15.90:0/3198137979 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbb400035b0 con 0x7fbb54109920 2024-04-14T15:10:09.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.054+0000 7fbb4a7fc700 1 -- 172.21.15.90:0/3198137979 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 917639099 0 0) 0x7fbb541a76a0 con 0x7fbb541017b0 2024-04-14T15:10:09.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.054+0000 7fbb4a7fc700 1 -- 172.21.15.90:0/3198137979 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbb5406b120 con 0x7fbb541017b0 2024-04-14T15:10:09.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.054+0000 7fbb4a7fc700 1 -- 172.21.15.90:0/3198137979 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fbb4c001bb0 con 0x7fbb54109920 2024-04-14T15:10:09.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.054+0000 7fbb4a7fc700 1 -- 172.21.15.90:0/3198137979 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fbb50001430 con 0x7fbb541017b0 2024-04-14T15:10:09.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.055+0000 7fbb4a7fc700 1 -- 172.21.15.90:0/3198137979 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 717530182 0 0) 0x7fbb5406b350 con 0x7fbb54105be0 2024-04-14T15:10:09.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.055+0000 7fbb4a7fc700 1 -- 172.21.15.90:0/3198137979 >> v1:172.21.15.90:6790/0 conn(0x7fbb54109920 legacy=0x7fbb541a3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:09.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.055+0000 7fbb4a7fc700 1 -- 172.21.15.90:0/3198137979 >> v1:172.21.15.92:6789/0 conn(0x7fbb541017b0 legacy=0x7fbb5419d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:09.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.055+0000 7fbb4a7fc700 1 -- 172.21.15.90:0/3198137979 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb541a88c0 con 0x7fbb54105be0 2024-04-14T15:10:09.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.055+0000 7fbb5b3e8700 1 -- 172.21.15.90:0/3198137979 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbb541a78d0 con 0x7fbb54105be0 2024-04-14T15:10:09.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.055+0000 7fbb5b3e8700 10 client.14592.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:10:09.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.055+0000 7fbb5b3e8700 1 -- 172.21.15.90:0/3198137979 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fbb541a7e10 con 0x7fbb54105be0 2024-04-14T15:10:09.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.055+0000 7fbb5b3e8700 1 librados: init done 2024-04-14T15:10:09.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.055+0000 7fbb4a7fc700 1 -- 172.21.15.90:0/3198137979 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fbb44003b20 con 0x7fbb54105be0 2024-04-14T15:10:09.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.055+0000 7fbb4a7fc700 1 -- 172.21.15.90:0/3198137979 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fbb44005c90 con 0x7fbb54105be0 2024-04-14T15:10:09.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.056+0000 7fbb5b3e8700 1 -- 172.21.15.90:0/3198137979 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb38005120 con 0x7fbb54105be0 2024-04-14T15:10:09.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.056+0000 7fbb4a7fc700 1 -- 172.21.15.90:0/3198137979 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 16) ==== 92674+0+0 (unknown 886309086 0 0) 0x7fbb44003680 con 0x7fbb54105be0 2024-04-14T15:10:09.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.057+0000 7fbb4a7fc700 1 -- 172.21.15.90:0/3198137979 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(76..76 src has 12..76) ==== 6030+0+0 (unknown 3888776809 0 0) 0x7fbb4408a180 con 0x7fbb54105be0 2024-04-14T15:10:09.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.057+0000 7fbb4a7fc700 10 client.14592.objecter ms_dispatch 0x7fbb54001550 osd_map(76..76 src has 12..76) 2024-04-14T15:10:09.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.057+0000 7fbb4a7fc700 3 client.14592.objecter handle_osd_map got epochs [76,76] > 0 2024-04-14T15:10:09.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.057+0000 7fbb4a7fc700 3 client.14592.objecter handle_osd_map decoding full epoch 76 2024-04-14T15:10:09.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.057+0000 7fbb4a7fc700 20 client.14592.objecter dump_active .. 0 homeless 2024-04-14T15:10:09.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.057+0000 7fbb4a7fc700 10 client.14592.objecter ms_handle_connect 0x7fbb40079c30 2024-04-14T15:10:09.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.061+0000 7fbb4a7fc700 1 -- 172.21.15.90:0/3198137979 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fbb44057290 con 0x7fbb54105be0 2024-04-14T15:10:09.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:08 smithi092 ceph-mon[33966]: osdmap e76: 8 total, 8 up, 8 in 2024-04-14T15:10:09.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:08 smithi092 ceph-mon[33966]: pgmap v217: 132 pgs: 2 peering, 130 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:10:09.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.343+0000 7fbb5b3e8700 1 -- 172.21.15.90:0/3198137979 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7fbb38005410 con 0x7fbb54105be0 2024-04-14T15:10:09.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.345+0000 7fbb4a7fc700 1 -- 172.21.15.90:0/3198137979 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v16) ==== 170+0+59 (unknown 2149379880 0 542235392) 0x7fbb44056b90 con 0x7fbb54105be0 2024-04-14T15:10:09.345 INFO:teuthology.orchestra.run.smithi090.stdout:[client.0] 2024-04-14T15:10:09.345 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQDR8Rtm9EWMFBAAuNde3g9r+Ly6C12pyEImSQ== 2024-04-14T15:10:09.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.347+0000 7fbb5b3e8700 10 librados: watch_flush enter 2024-04-14T15:10:09.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.347+0000 7fbb5b3e8700 10 librados: watch_flush exit 2024-04-14T15:10:09.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.347+0000 7fbb5b3e8700 20 client.14592.objecter shutdown clearing up homeless session... 2024-04-14T15:10:09.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.347+0000 7fbb5b3e8700 10 client.14592.objecter successfully canceled tick 2024-04-14T15:10:09.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.347+0000 7fbb5b3e8700 1 -- 172.21.15.90:0/3198137979 >> v1:172.21.15.90:6800/491071118 conn(0x7fbb40079c30 legacy=0x7fbb4007c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:09.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.347+0000 7fbb5b3e8700 1 -- 172.21.15.90:0/3198137979 >> v1:172.21.15.90:6789/0 conn(0x7fbb54105be0 legacy=0x7fbb541a07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:09.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.347+0000 7fbb5b3e8700 1 -- 172.21.15.90:0/3198137979 shutdown_connections 2024-04-14T15:10:09.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.347+0000 7fbb5b3e8700 1 -- 172.21.15.90:0/3198137979 >> 172.21.15.90:0/3198137979 conn(0x7fbb540fcd50 msgr2=0x7fbb54108030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:10:09.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.347+0000 7fbb5b3e8700 1 -- 172.21.15.90:0/3198137979 shutdown_connections 2024-04-14T15:10:09.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.347+0000 7fbb5b3e8700 1 -- 172.21.15.90:0/3198137979 wait complete. 2024-04-14T15:10:09.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:09.347+0000 7fbb5b3e8700 1 librados: shutdown 2024-04-14T15:10:09.912 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:09 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3198137979' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-14T15:10:09.912 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:09 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3198137979' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-14T15:10:09.912 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:09 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3198137979' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-14T15:10:09.912 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:09 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3198137979' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-14T15:10:09.937 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-14T15:10:09.937 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-14T15:10:09.937 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-14T15:10:09.979 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-14T15:10:10.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:09 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3198137979' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-14T15:10:10.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:09 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3198137979' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-14T15:10:10.992 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:10 smithi092 ceph-mon[33966]: pgmap v218: 132 pgs: 2 peering, 130 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 13 B/s, 1 objects/s recovering 2024-04-14T15:10:11.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:10 smithi090 ceph-mon[30399]: pgmap v218: 132 pgs: 2 peering, 130 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 13 B/s, 1 objects/s recovering 2024-04-14T15:10:11.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:10 smithi090 ceph-mon[41670]: pgmap v218: 132 pgs: 2 peering, 130 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 13 B/s, 1 objects/s recovering 2024-04-14T15:10:11.987 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:10:13.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:12 smithi092 ceph-mon[33966]: pgmap v219: 132 pgs: 2 peering, 130 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 13 B/s, 1 objects/s recovering 2024-04-14T15:10:13.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:12 smithi090 ceph-mon[30399]: pgmap v219: 132 pgs: 2 peering, 130 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 13 B/s, 1 objects/s recovering 2024-04-14T15:10:13.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:12 smithi090 ceph-mon[41670]: pgmap v219: 132 pgs: 2 peering, 130 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 13 B/s, 1 objects/s recovering 2024-04-14T15:10:14.342 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:10:14 smithi092 podman[65369]: 2024-04-14 15:10:14.206758484 +0000 UTC m=+0.862261013 container create 9fcf2f777e893dca73a9aa39d89766781c5169ef7a3ecc734c55e7b883824974 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T15:10:14.530 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:14.530+0000 7fa60db7e700 1 -- 172.21.15.92:0/3296101938 >> v1:172.21.15.92:6789/0 conn(0x7fa6081017c0 legacy=0x7fa608101be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:14.530 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:14.530+0000 7fa60db7e700 1 -- 172.21.15.92:0/3296101938 shutdown_connections 2024-04-14T15:10:14.531 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:14.530+0000 7fa60db7e700 1 -- 172.21.15.92:0/3296101938 >> 172.21.15.92:0/3296101938 conn(0x7fa6080fcda0 msgr2=0x7fa6080ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:10:14.531 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:14.530+0000 7fa60db7e700 1 -- 172.21.15.92:0/3296101938 shutdown_connections 2024-04-14T15:10:14.531 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:14.530+0000 7fa60db7e700 1 -- 172.21.15.92:0/3296101938 wait complete. 2024-04-14T15:10:14.531 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:14.531+0000 7fa60db7e700 1 librados: starting msgr at 2024-04-14T15:10:14.531 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:14.531+0000 7fa60db7e700 1 librados: starting objecter 2024-04-14T15:10:14.531 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:14.531+0000 7fa60db7e700 1 Processor -- start 2024-04-14T15:10:14.531 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:14.531+0000 7fa60db7e700 1 -- start start 2024-04-14T15:10:14.531 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:14.531+0000 7fa60db7e700 1 librados: setting wanted keys 2024-04-14T15:10:14.531 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:14.531+0000 7fa60db7e700 1 librados: calling monclient init 2024-04-14T15:10:14.534 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:14.531+0000 7fa60db7e700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa608198b00 con 0x7fa608109930 2024-04-14T15:10:14.534 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:14.531+0000 7fa60db7e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa6081a42f0 con 0x7fa6081017c0 2024-04-14T15:10:14.534 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:14.531+0000 7fa60db7e700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa6081a54d0 con 0x7fa608105bf0 2024-04-14T15:10:14.534 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:14.531+0000 7fa6077fe700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fa6081017c0 0x7fa608197fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:45012/0 (socket says 172.21.15.92:45012) 2024-04-14T15:10:14.534 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:14.531+0000 7fa6077fe700 1 -- 172.21.15.92:0/3303914104 learned_addr learned my addr 172.21.15.92:0/3303914104 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-04-14T15:10:14.534 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:14.531+0000 7fa604ff9700 10 client.?.objecter ms_handle_connect 0x7fa6081017c0 2024-04-14T15:10:14.534 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:14.531+0000 7fa604ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:10:14.534 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:14.531+0000 7fa604ff9700 10 client.?.objecter ms_handle_connect 0x7fa608109930 2024-04-14T15:10:14.534 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:14.531+0000 7fa604ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:10:14.534 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:14.531+0000 7fa604ff9700 10 client.?.objecter ms_handle_connect 0x7fa608105bf0 2024-04-14T15:10:14.535 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:14.531+0000 7fa604ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:10:14.535 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:14.531+0000 7fa604ff9700 1 -- 172.21.15.92:0/3303914104 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3270833224 0 0) 0x7fa6081a42f0 con 0x7fa6081017c0 2024-04-14T15:10:14.535 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:14.531+0000 7fa604ff9700 1 -- 172.21.15.92:0/3303914104 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa5f00035b0 con 0x7fa6081017c0 2024-04-14T15:10:14.535 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:14.532+0000 7fa604ff9700 1 -- 172.21.15.92:0/3303914104 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3783403707 0 0) 0x7fa608198b00 con 0x7fa608109930 2024-04-14T15:10:14.535 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:14.532+0000 7fa604ff9700 1 -- 172.21.15.92:0/3303914104 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa6081a42f0 con 0x7fa608109930 2024-04-14T15:10:14.535 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:14.532+0000 7fa604ff9700 1 -- 172.21.15.92:0/3303914104 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3275591121 0 0) 0x7fa6081a54d0 con 0x7fa608105bf0 2024-04-14T15:10:14.535 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:14.532+0000 7fa604ff9700 1 -- 172.21.15.92:0/3303914104 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa608198b00 con 0x7fa608105bf0 2024-04-14T15:10:14.535 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:14.532+0000 7fa604ff9700 1 -- 172.21.15.92:0/3303914104 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1178785143 0 0) 0x7fa5f00035b0 con 0x7fa6081017c0 2024-04-14T15:10:14.535 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:14.532+0000 7fa604ff9700 1 -- 172.21.15.92:0/3303914104 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa6081a54d0 con 0x7fa6081017c0 2024-04-14T15:10:14.535 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:14.532+0000 7fa604ff9700 1 -- 172.21.15.92:0/3303914104 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fa5f8001430 con 0x7fa6081017c0 2024-04-14T15:10:14.535 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:14.532+0000 7fa604ff9700 1 -- 172.21.15.92:0/3303914104 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 143806062 0 0) 0x7fa6081a42f0 con 0x7fa608109930 2024-04-14T15:10:14.535 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:14.532+0000 7fa604ff9700 1 -- 172.21.15.92:0/3303914104 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa5f00035b0 con 0x7fa608109930 2024-04-14T15:10:14.535 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:14.532+0000 7fa604ff9700 1 -- 172.21.15.92:0/3303914104 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fa600001aa0 con 0x7fa608109930 2024-04-14T15:10:14.536 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:14.532+0000 7fa604ff9700 1 -- 172.21.15.92:0/3303914104 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2372192842 0 0) 0x7fa6081a54d0 con 0x7fa6081017c0 2024-04-14T15:10:14.536 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:14.532+0000 7fa604ff9700 1 -- 172.21.15.92:0/3303914104 >> v1:172.21.15.90:6790/0 conn(0x7fa608105bf0 legacy=0x7fa60819e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:14.536 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:14.532+0000 7fa604ff9700 1 -- 172.21.15.92:0/3303914104 >> v1:172.21.15.90:6789/0 conn(0x7fa608109930 legacy=0x7fa6081a1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:14.536 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:14.532+0000 7fa604ff9700 1 -- 172.21.15.92:0/3303914104 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6081a66f0 con 0x7fa6081017c0 2024-04-14T15:10:14.536 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:14.532+0000 7fa60db7e700 1 -- 172.21.15.92:0/3303914104 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa6081a3340 con 0x7fa6081017c0 2024-04-14T15:10:14.536 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:14.532+0000 7fa60db7e700 10 client.24493.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:10:14.536 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:14.532+0000 7fa60db7e700 1 -- 172.21.15.92:0/3303914104 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa6081a3880 con 0x7fa6081017c0 2024-04-14T15:10:14.536 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:14.532+0000 7fa60db7e700 1 librados: init done 2024-04-14T15:10:14.536 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:14.532+0000 7fa604ff9700 1 -- 172.21.15.92:0/3303914104 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fa5f8003700 con 0x7fa6081017c0 2024-04-14T15:10:14.536 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:14.532+0000 7fa604ff9700 1 -- 172.21.15.92:0/3303914104 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fa5f8004ea0 con 0x7fa6081017c0 2024-04-14T15:10:14.536 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:14.533+0000 7fa60db7e700 1 -- 172.21.15.92:0/3303914104 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5f4005150 con 0x7fa6081017c0 2024-04-14T15:10:14.536 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:14.533+0000 7fa604ff9700 1 -- 172.21.15.92:0/3303914104 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 16) ==== 92674+0+0 (unknown 886309086 0 0) 0x7fa5f8003ab0 con 0x7fa6081017c0 2024-04-14T15:10:14.536 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:14.534+0000 7fa604ff9700 1 -- 172.21.15.92:0/3303914104 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(76..76 src has 12..76) ==== 6030+0+0 (unknown 3888776809 0 0) 0x7fa5f808a390 con 0x7fa6081017c0 2024-04-14T15:10:14.537 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:14.534+0000 7fa604ff9700 10 client.24493.objecter ms_dispatch 0x7fa608001550 osd_map(76..76 src has 12..76) 2024-04-14T15:10:14.537 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:14.534+0000 7fa604ff9700 3 client.24493.objecter handle_osd_map got epochs [76,76] > 0 2024-04-14T15:10:14.537 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:14.534+0000 7fa604ff9700 3 client.24493.objecter handle_osd_map decoding full epoch 76 2024-04-14T15:10:14.537 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:14.534+0000 7fa604ff9700 20 client.24493.objecter dump_active .. 0 homeless 2024-04-14T15:10:14.537 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:14.534+0000 7fa604ff9700 10 client.24493.objecter ms_handle_connect 0x7fa5f0071070 2024-04-14T15:10:14.538 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:14.538+0000 7fa604ff9700 1 -- 172.21.15.92:0/3303914104 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fa5f8057580 con 0x7fa6081017c0 2024-04-14T15:10:14.667 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:10:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:10:14.668 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:10:14 smithi092 podman[65369]: 2024-04-14 15:10:14.668406901 +0000 UTC m=+1.323909430 container init 9fcf2f777e893dca73a9aa39d89766781c5169ef7a3ecc734c55e7b883824974 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T15:10:14.846 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:14.846+0000 7fa60db7e700 1 -- 172.21.15.92:0/3303914104 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7fa5f4005440 con 0x7fa6081017c0 2024-04-14T15:10:14.852 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:14.852+0000 7fa604ff9700 1 -- 172.21.15.92:0/3303914104 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v17) ==== 170+0+59 (unknown 1363465646 0 1927041944) 0x7fa5f8001bb0 con 0x7fa6081017c0 2024-04-14T15:10:14.852 INFO:teuthology.orchestra.run.smithi092.stdout:[client.1] 2024-04-14T15:10:14.852 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQDW8RtmAb+CMhAAT2AC42tDHz75mV/jsOIIQg== 2024-04-14T15:10:14.854 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:14.854+0000 7fa60db7e700 10 librados: watch_flush enter 2024-04-14T15:10:14.854 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:14.854+0000 7fa60db7e700 10 librados: watch_flush exit 2024-04-14T15:10:14.854 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:14.854+0000 7fa60db7e700 20 client.24493.objecter shutdown clearing up homeless session... 2024-04-14T15:10:14.854 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:14.854+0000 7fa60db7e700 10 client.24493.objecter successfully canceled tick 2024-04-14T15:10:14.855 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:14.855+0000 7fa60db7e700 1 -- 172.21.15.92:0/3303914104 >> v1:172.21.15.90:6800/491071118 conn(0x7fa5f0071070 legacy=0x7fa5f0073530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:14.855 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:14.855+0000 7fa60db7e700 1 -- 172.21.15.92:0/3303914104 >> v1:172.21.15.92:6789/0 conn(0x7fa6081017c0 legacy=0x7fa608197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:14.855 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:14.855+0000 7fa60db7e700 1 -- 172.21.15.92:0/3303914104 shutdown_connections 2024-04-14T15:10:14.855 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:14.855+0000 7fa60db7e700 1 -- 172.21.15.92:0/3303914104 >> 172.21.15.92:0/3303914104 conn(0x7fa6080fcda0 msgr2=0x7fa608109150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:10:14.855 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:14.855+0000 7fa60db7e700 1 -- 172.21.15.92:0/3303914104 shutdown_connections 2024-04-14T15:10:14.855 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:14.855+0000 7fa60db7e700 1 -- 172.21.15.92:0/3303914104 wait complete. 2024-04-14T15:10:14.855 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-14T15:10:14.855+0000 7fa60db7e700 1 librados: shutdown 2024-04-14T15:10:15.001 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:10:14 smithi092 podman[65369]: 2024-04-14 15:10:14.674473446 +0000 UTC m=+1.329975978 container start 9fcf2f777e893dca73a9aa39d89766781c5169ef7a3ecc734c55e7b883824974 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T15:10:15.001 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:10:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[65425]: ts=2024-04-14T15:10:14.709Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-04-14T15:10:15.001 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:10:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[65425]: ts=2024-04-14T15:10:14.709Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-04-14T15:10:15.001 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:10:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[65425]: ts=2024-04-14T15:10:14.709Z caller=main.go:570 level=info host_details="(Linux 5.14.0-435.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024 x86_64 smithi092 (none))" 2024-04-14T15:10:15.001 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:10:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[65425]: ts=2024-04-14T15:10:14.709Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-04-14T15:10:15.001 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:10:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[65425]: ts=2024-04-14T15:10:14.709Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-04-14T15:10:15.001 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:10:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[65425]: ts=2024-04-14T15:10:14.711Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-04-14T15:10:15.001 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:10:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[65425]: ts=2024-04-14T15:10:14.712Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-04-14T15:10:15.002 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:10:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[65425]: ts=2024-04-14T15:10:14.712Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-04-14T15:10:15.002 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:10:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[65425]: ts=2024-04-14T15:10:14.712Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-04-14T15:10:15.002 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:10:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[65425]: ts=2024-04-14T15:10:14.715Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-04-14T15:10:15.002 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:10:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[65425]: ts=2024-04-14T15:10:14.715Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.579µs 2024-04-14T15:10:15.002 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:10:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[65425]: ts=2024-04-14T15:10:14.715Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-04-14T15:10:15.002 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:10:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[65425]: ts=2024-04-14T15:10:14.715Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-04-14T15:10:15.002 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:10:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[65425]: ts=2024-04-14T15:10:14.715Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=17.945µs wal_replay_duration=213.881µs wbl_replay_duration=108ns total_replay_duration=249.415µs 2024-04-14T15:10:15.002 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:10:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[65425]: ts=2024-04-14T15:10:14.716Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-04-14T15:10:15.002 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:10:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[65425]: ts=2024-04-14T15:10:14.716Z caller=main.go:1029 level=info msg="TSDB started" 2024-04-14T15:10:15.002 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:10:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[65425]: ts=2024-04-14T15:10:14.716Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-04-14T15:10:15.002 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:10:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[65425]: ts=2024-04-14T15:10:14.736Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=20.212213ms db_storage=885ns remote_storage=1.051µs web_handler=331ns query_engine=620ns scrape=459.16µs scrape_sd=80.762µs notify=1.039µs notify_sd=2.418µs rules=19.420269ms tracing=6.327µs 2024-04-14T15:10:15.002 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:10:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[65425]: ts=2024-04-14T15:10:14.736Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-04-14T15:10:15.002 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:10:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[65425]: ts=2024-04-14T15:10:14.736Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-04-14T15:10:15.003 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:10:14 smithi092 bash[65369]: 9fcf2f777e893dca73a9aa39d89766781c5169ef7a3ecc734c55e7b883824974 2024-04-14T15:10:15.003 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:10:14 smithi092 systemd[1]: Started Ceph prometheus.a for 0e288098-fa70-11ee-bc8e-c7b262605968. 2024-04-14T15:10:15.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:15 smithi092 ceph-mon[33966]: pgmap v220: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 2 objects/s recovering 2024-04-14T15:10:15.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:15 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:10:15.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:15 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:10:15.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:15 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:10:15.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:15 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-14T15:10:15.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:15 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.92:0/3303914104' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-14T15:10:15.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:15 smithi092 ceph-mon[33966]: from='client.24493 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-14T15:10:15.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:15 smithi092 ceph-mon[33966]: from='client.24493 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-14T15:10:15.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:14 smithi090 ceph-mon[41670]: pgmap v220: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 2 objects/s recovering 2024-04-14T15:10:15.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:14 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:10:15.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:14 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:10:15.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:14 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:10:15.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:14 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-14T15:10:15.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:14 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.92:0/3303914104' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-14T15:10:15.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:14 smithi090 ceph-mon[41670]: from='client.24493 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-14T15:10:15.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:14 smithi090 ceph-mon[41670]: from='client.24493 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-14T15:10:15.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:14 smithi090 ceph-mon[30399]: pgmap v220: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 2 objects/s recovering 2024-04-14T15:10:15.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:14 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:10:15.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:14 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:10:15.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:14 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' 2024-04-14T15:10:15.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:14 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-14T15:10:15.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:14 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.92:0/3303914104' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-14T15:10:15.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:14 smithi090 ceph-mon[30399]: from='client.24493 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-14T15:10:15.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:14 smithi090 ceph-mon[30399]: from='client.24493 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-14T15:10:15.450 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-04-14T15:10:15.450 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-04-14T15:10:15.450 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-14T15:10:15.493 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-14T15:10:15.493 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-14T15:10:15.494 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph mgr dump --format=json 2024-04-14T15:10:15.744 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:10:16.142 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:15 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:10:16.142 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:15 smithi090 ceph-mon[41670]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-14T15:10:16.142 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:15 smithi090 ceph-mon[41670]: mgrmap e17: y(active, since 5m), standbys: x 2024-04-14T15:10:16.142 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:10:15 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ignoring --setuser ceph since I am not root 2024-04-14T15:10:16.143 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:10:15 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ignoring --setgroup ceph since I am not root 2024-04-14T15:10:16.143 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:15 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:10:16.143 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:15 smithi090 ceph-mon[30399]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-14T15:10:16.143 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:15 smithi090 ceph-mon[30399]: mgrmap e17: y(active, since 5m), standbys: x 2024-04-14T15:10:16.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:15 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:10:16.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:15 smithi092 ceph-mon[33966]: from='mgr.14148 v1:172.21.15.90:0/2214604766' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-14T15:10:16.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:15 smithi092 ceph-mon[33966]: mgrmap e17: y(active, since 5m), standbys: x 2024-04-14T15:10:16.372 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:10:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-x[36044]: ignoring --setuser ceph since I am not root 2024-04-14T15:10:16.372 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:10:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-x[36044]: ignoring --setgroup ceph since I am not root 2024-04-14T15:10:16.691 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:10:16 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:10:16.422+0000 7f5636b80200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-14T15:10:16.691 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:10:16 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:10:16.517+0000 7f5636b80200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-14T15:10:16.872 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:10:16 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-x[36044]: 2024-04-14T15:10:16.399+0000 7f629731b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-14T15:10:16.872 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:10:16 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-x[36044]: 2024-04-14T15:10:16.493+0000 7f629731b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-14T15:10:17.164 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:16 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:10:17.165 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:17 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:10:17.165 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:16 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:10:17.165 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:16 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:10:17.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:17 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:10:17.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:17 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:10:17.372 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:10:17 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-x[36044]: 2024-04-14T15:10:17.136+0000 7f629731b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-14T15:10:17.372 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:10:17 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-x[36044]: 2024-04-14T15:10:17.232+0000 7f629731b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-14T15:10:17.439 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:10:17 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:10:17.198+0000 7f5636b80200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-14T15:10:17.439 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:10:17 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:10:17.292+0000 7f5636b80200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-14T15:10:17.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:17.451+0000 7ff5fef05700 1 -- 172.21.15.90:0/2740203202 >> v1:172.21.15.90:6789/0 conn(0x7ff5f8107890 legacy=0x7ff5f8107c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:17.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:17.451+0000 7ff5fef05700 1 -- 172.21.15.90:0/2740203202 shutdown_connections 2024-04-14T15:10:17.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:17.451+0000 7ff5fef05700 1 -- 172.21.15.90:0/2740203202 >> 172.21.15.90:0/2740203202 conn(0x7ff5f80fd290 msgr2=0x7ff5f80ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:10:17.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:17.451+0000 7ff5fef05700 1 -- 172.21.15.90:0/2740203202 shutdown_connections 2024-04-14T15:10:17.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:17.451+0000 7ff5fef05700 1 -- 172.21.15.90:0/2740203202 wait complete. 2024-04-14T15:10:17.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:17.452+0000 7ff5fef05700 1 librados: starting msgr at 2024-04-14T15:10:17.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:17.452+0000 7ff5fef05700 1 librados: starting objecter 2024-04-14T15:10:17.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:17.452+0000 7ff5fef05700 1 Processor -- start 2024-04-14T15:10:17.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:17.452+0000 7ff5fef05700 1 -- start start 2024-04-14T15:10:17.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:17.452+0000 7ff5fef05700 1 librados: setting wanted keys 2024-04-14T15:10:17.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:17.452+0000 7ff5fef05700 1 librados: calling monclient init 2024-04-14T15:10:17.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:17.452+0000 7ff5fef05700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff5f810df80 con 0x7ff5f810e310 2024-04-14T15:10:17.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:17.452+0000 7ff5fef05700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff5f81a8f30 con 0x7ff5f810a740 2024-04-14T15:10:17.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:17.452+0000 7ff5fef05700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff5f81aa110 con 0x7ff5f8107890 2024-04-14T15:10:17.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:17.452+0000 7ff5fcca1700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7ff5f8107890 0x7ff5f806acf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:48902/0 (socket says 172.21.15.90:48902) 2024-04-14T15:10:17.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:17.452+0000 7ff5fcca1700 1 -- 172.21.15.90:0/1180789422 learned_addr learned my addr 172.21.15.90:0/1180789422 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:10:17.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:17.452+0000 7ff5f5ffb700 10 client.?.objecter ms_handle_connect 0x7ff5f8107890 2024-04-14T15:10:17.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:17.452+0000 7ff5f5ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:10:17.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:17.452+0000 7ff5f5ffb700 10 client.?.objecter ms_handle_connect 0x7ff5f810e310 2024-04-14T15:10:17.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:17.452+0000 7ff5f5ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:10:17.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:17.452+0000 7ff5f5ffb700 10 client.?.objecter ms_handle_connect 0x7ff5f810a740 2024-04-14T15:10:17.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:17.452+0000 7ff5f5ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:10:17.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:17.453+0000 7ff5f5ffb700 1 -- 172.21.15.90:0/1180789422 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1356761477 0 0) 0x7ff5f81a8f30 con 0x7ff5f810a740 2024-04-14T15:10:17.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:17.453+0000 7ff5f5ffb700 1 -- 172.21.15.90:0/1180789422 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff5e80031f0 con 0x7ff5f810a740 2024-04-14T15:10:17.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:17.453+0000 7ff5f5ffb700 1 -- 172.21.15.90:0/1180789422 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3135037437 0 0) 0x7ff5f81aa110 con 0x7ff5f8107890 2024-04-14T15:10:17.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:17.453+0000 7ff5f5ffb700 1 -- 172.21.15.90:0/1180789422 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff5f81a8f30 con 0x7ff5f8107890 2024-04-14T15:10:17.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:17.453+0000 7ff5f5ffb700 1 -- 172.21.15.90:0/1180789422 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2430996060 0 0) 0x7ff5f810df80 con 0x7ff5f810e310 2024-04-14T15:10:17.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:17.453+0000 7ff5f5ffb700 1 -- 172.21.15.90:0/1180789422 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff5f81aa110 con 0x7ff5f810e310 2024-04-14T15:10:17.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:17.453+0000 7ff5f5ffb700 1 -- 172.21.15.90:0/1180789422 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3379580121 0 0) 0x7ff5e80031f0 con 0x7ff5f810a740 2024-04-14T15:10:17.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:17.453+0000 7ff5f5ffb700 1 -- 172.21.15.90:0/1180789422 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff5f810df80 con 0x7ff5f810a740 2024-04-14T15:10:17.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:17.453+0000 7ff5f5ffb700 1 -- 172.21.15.90:0/1180789422 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7ff5e40012b0 con 0x7ff5f810a740 2024-04-14T15:10:17.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:17.453+0000 7ff5f5ffb700 1 -- 172.21.15.90:0/1180789422 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2558185110 0 0) 0x7ff5f81a8f30 con 0x7ff5f8107890 2024-04-14T15:10:17.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:17.453+0000 7ff5f5ffb700 1 -- 172.21.15.90:0/1180789422 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff5e80031f0 con 0x7ff5f8107890 2024-04-14T15:10:17.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:17.453+0000 7ff5f5ffb700 1 -- 172.21.15.90:0/1180789422 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7ff5ec004680 con 0x7ff5f8107890 2024-04-14T15:10:17.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:17.453+0000 7ff5f5ffb700 1 -- 172.21.15.90:0/1180789422 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3816152536 0 0) 0x7ff5f810df80 con 0x7ff5f810a740 2024-04-14T15:10:17.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:17.453+0000 7ff5f5ffb700 1 -- 172.21.15.90:0/1180789422 >> v1:172.21.15.90:6790/0 conn(0x7ff5f8107890 legacy=0x7ff5f806acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:17.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:17.453+0000 7ff5f5ffb700 1 -- 172.21.15.90:0/1180789422 >> v1:172.21.15.90:6789/0 conn(0x7ff5f810e310 legacy=0x7ff5f81a6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:17.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:17.453+0000 7ff5f5ffb700 1 -- 172.21.15.90:0/1180789422 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5f81ab2f0 con 0x7ff5f810a740 2024-04-14T15:10:17.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:17.453+0000 7ff5fef05700 1 -- 172.21.15.90:0/1180789422 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff5f81a9160 con 0x7ff5f810a740 2024-04-14T15:10:17.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:17.453+0000 7ff5f5ffb700 1 -- 172.21.15.90:0/1180789422 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7ff5e4003b20 con 0x7ff5f810a740 2024-04-14T15:10:17.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:17.453+0000 7ff5f5ffb700 1 -- 172.21.15.90:0/1180789422 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7ff5e4004920 con 0x7ff5f810a740 2024-04-14T15:10:17.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:17.453+0000 7ff5fef05700 10 client.24508.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:10:17.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:17.453+0000 7ff5fef05700 1 -- 172.21.15.90:0/1180789422 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff5f81a9620 con 0x7ff5f810a740 2024-04-14T15:10:17.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:17.454+0000 7ff5fef05700 1 librados: init done 2024-04-14T15:10:17.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:17.454+0000 7ff5fef05700 1 -- 172.21.15.90:0/1180789422 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5f80693d0 con 0x7ff5f810a740 2024-04-14T15:10:17.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:17.455+0000 7ff5f5ffb700 1 -- 172.21.15.90:0/1180789422 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 17) ==== 92688+0+0 (unknown 1179923196 0 0) 0x7ff5e4003680 con 0x7ff5f810a740 2024-04-14T15:10:17.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:17.455+0000 7ff5fcca1700 1 -- 172.21.15.90:0/1180789422 >> v1:172.21.15.90:6800/491071118 conn(0x7ff5e8070fb0 legacy=0x7ff5e8073470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.90:6800/491071118 2024-04-14T15:10:17.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:17.455+0000 7ff5f5ffb700 1 -- 172.21.15.90:0/1180789422 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(77..77 src has 12..77) ==== 6030+0+0 (unknown 3485272705 0 0) 0x7ff5e4088eb0 con 0x7ff5f810a740 2024-04-14T15:10:17.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:17.455+0000 7ff5f5ffb700 10 client.24508.objecter ms_dispatch 0x7ff5f8001550 osd_map(77..77 src has 12..77) 2024-04-14T15:10:17.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:17.455+0000 7ff5f5ffb700 3 client.24508.objecter handle_osd_map got epochs [77,77] > 0 2024-04-14T15:10:17.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:17.455+0000 7ff5f5ffb700 3 client.24508.objecter handle_osd_map decoding full epoch 77 2024-04-14T15:10:17.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:17.455+0000 7ff5f5ffb700 20 client.24508.objecter dump_active .. 0 homeless 2024-04-14T15:10:17.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:17.464+0000 7ff5f5ffb700 1 -- 172.21.15.90:0/1180789422 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7ff5e4055fc0 con 0x7ff5f810a740 2024-04-14T15:10:17.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:17.656+0000 7ff5fcca1700 1 -- 172.21.15.90:0/1180789422 >> v1:172.21.15.90:6800/491071118 conn(0x7ff5e8070fb0 legacy=0x7ff5e8073470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.90:6800/491071118 2024-04-14T15:10:17.691 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:10:17 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:10:17.438+0000 7f5636b80200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-14T15:10:17.691 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:10:17 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:10:17.538+0000 7f5636b80200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-14T15:10:17.691 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:10:17 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:10:17.621+0000 7f5636b80200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-14T15:10:17.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:17.730+0000 7ff5fef05700 1 -- 172.21.15.90:0/1180789422 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7ff5f810d320 con 0x7ff5f810a740 2024-04-14T15:10:17.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:17.735+0000 7ff5f5ffb700 1 -- 172.21.15.90:0/1180789422 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+177149 (unknown 831476310 0 312526934) 0x7ff5e40558c0 con 0x7ff5f810a740 2024-04-14T15:10:17.739 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:10:17.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:17.738+0000 7ff5fef05700 10 librados: watch_flush enter 2024-04-14T15:10:17.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:17.738+0000 7ff5fef05700 10 librados: watch_flush exit 2024-04-14T15:10:17.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:17.738+0000 7ff5fef05700 20 client.24508.objecter shutdown clearing up homeless session... 2024-04-14T15:10:17.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:17.738+0000 7ff5fef05700 10 client.24508.objecter successfully canceled tick 2024-04-14T15:10:17.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:17.738+0000 7ff5fef05700 1 -- 172.21.15.90:0/1180789422 >> v1:172.21.15.90:6800/491071118 conn(0x7ff5e8070fb0 legacy=0x7ff5e8073470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-14T15:10:17.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:17.738+0000 7ff5fef05700 1 -- 172.21.15.90:0/1180789422 >> v1:172.21.15.92:6789/0 conn(0x7ff5f810a740 legacy=0x7ff5f806b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:17.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:17.738+0000 7ff5fef05700 1 -- 172.21.15.90:0/1180789422 shutdown_connections 2024-04-14T15:10:17.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:17.738+0000 7ff5fef05700 1 -- 172.21.15.90:0/1180789422 >> 172.21.15.90:0/1180789422 conn(0x7ff5f80fd290 msgr2=0x7ff5f8111060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:10:17.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:17.738+0000 7ff5fef05700 1 -- 172.21.15.90:0/1180789422 shutdown_connections 2024-04-14T15:10:17.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:17.738+0000 7ff5fef05700 1 -- 172.21.15.90:0/1180789422 wait complete. 2024-04-14T15:10:17.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:17.738+0000 7ff5fef05700 1 librados: shutdown 2024-04-14T15:10:17.825 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:10:17 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-x[36044]: 2024-04-14T15:10:17.374+0000 7f629731b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-14T15:10:17.825 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:10:17 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-x[36044]: 2024-04-14T15:10:17.475+0000 7f629731b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-14T15:10:17.825 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:10:17 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-x[36044]: 2024-04-14T15:10:17.558+0000 7f629731b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-14T15:10:17.825 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:10:17 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-x[36044]: 2024-04-14T15:10:17.824+0000 7f629731b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-14T15:10:18.003 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:10:17 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:10:17.892+0000 7f5636b80200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-14T15:10:18.119 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:18 smithi092 ceph-mon[33966]: osdmap e77: 8 total, 8 up, 8 in 2024-04-14T15:10:18.119 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:18 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1180789422' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-14T15:10:18.119 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:10:17 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-x[36044]: 2024-04-14T15:10:17.934+0000 7f629731b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-14T15:10:18.119 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:10:18 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-x[36044]: 2024-04-14T15:10:18.022+0000 7f629731b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-14T15:10:18.119 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:10:18 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-x[36044]: 2024-04-14T15:10:18.118+0000 7f629731b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-14T15:10:18.369 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:18 smithi090 ceph-mon[30399]: osdmap e77: 8 total, 8 up, 8 in 2024-04-14T15:10:18.369 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:18 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1180789422' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-14T15:10:18.369 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:18 smithi090 ceph-mon[41670]: osdmap e77: 8 total, 8 up, 8 in 2024-04-14T15:10:18.369 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:18 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1180789422' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-14T15:10:18.370 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:10:18 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:10:18.007+0000 7f5636b80200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-14T15:10:18.370 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:10:18 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:10:18.097+0000 7f5636b80200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-14T15:10:18.370 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:10:18 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:10:18.191+0000 7f5636b80200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-14T15:10:18.372 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:10:18 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-x[36044]: 2024-04-14T15:10:18.295+0000 7f629731b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-14T15:10:18.511 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":17,"flags":0,"active_gid":14148,"active_name":"y","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6800","nonce":491071118}]},"active_addr":"172.21.15.90:6800/491071118","active_change":"2024-04-14T15:05:05.882467+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.90:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"cephadm","addrvec":[{"type":"v2","addr":"172.21.15.90:0","nonce":1823072952}]},{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.90:0","nonce":93763408}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.90:0","nonce":508768479}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.90:0","nonce":1337064035}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.90:0","nonce":769462306}]}]} 2024-04-14T15:10:18.515 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-14T15:10:18.515 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-14T15:10:18.515 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd dump --format=json 2024-04-14T15:10:18.691 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:10:18 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:10:18.368+0000 7f5636b80200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-14T15:10:18.791 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:10:19.441 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:10:19 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:10:19.368+0000 7f5636b80200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-14T15:10:19.622 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:10:19 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-x[36044]: 2024-04-14T15:10:19.255+0000 7f629731b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-14T15:10:19.622 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:10:19 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-x[36044]: 2024-04-14T15:10:19.386+0000 7f629731b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-14T15:10:19.842 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:10:19 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:10:19.500+0000 7f5636b80200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-14T15:10:19.842 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:10:19 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:10:19.842+0000 7f5636b80200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-14T15:10:20.009 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:10:19 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-x[36044]: 2024-04-14T15:10:19.719+0000 7f629731b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-14T15:10:20.009 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:10:19 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-x[36044]: 2024-04-14T15:10:19.810+0000 7f629731b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-14T15:10:20.009 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:10:20 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-x[36044]: 2024-04-14T15:10:20.008+0000 7f629731b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-14T15:10:20.135 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:10:19 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:10:19.934+0000 7f5636b80200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-14T15:10:20.372 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:10:20 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-x[36044]: 2024-04-14T15:10:20.196+0000 7f629731b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-14T15:10:20.441 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:10:20 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:10:20.141+0000 7f5636b80200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-14T15:10:20.441 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:10:20 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:10:20.335+0000 7f5636b80200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-14T15:10:20.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.474+0000 7f2012689700 1 -- 172.21.15.90:0/3623557843 >> v1:172.21.15.90:6789/0 conn(0x7f200c108980 legacy=0x7f200c108d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:20.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.474+0000 7f2012689700 1 -- 172.21.15.90:0/3623557843 shutdown_connections 2024-04-14T15:10:20.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.474+0000 7f2012689700 1 -- 172.21.15.90:0/3623557843 >> 172.21.15.90:0/3623557843 conn(0x7f200c076aa0 msgr2=0x7f200c076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:10:20.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.474+0000 7f2012689700 1 -- 172.21.15.90:0/3623557843 shutdown_connections 2024-04-14T15:10:20.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.474+0000 7f2012689700 1 -- 172.21.15.90:0/3623557843 wait complete. 2024-04-14T15:10:20.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.475+0000 7f2012689700 1 librados: starting msgr at 2024-04-14T15:10:20.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.475+0000 7f2012689700 1 librados: starting objecter 2024-04-14T15:10:20.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.475+0000 7f2012689700 1 Processor -- start 2024-04-14T15:10:20.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.475+0000 7f2012689700 1 -- start start 2024-04-14T15:10:20.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.475+0000 7f2012689700 1 librados: setting wanted keys 2024-04-14T15:10:20.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.475+0000 7f2012689700 1 librados: calling monclient init 2024-04-14T15:10:20.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.475+0000 7f2012689700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f200c1081d0 con 0x7f200c10f2f0 2024-04-14T15:10:20.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.475+0000 7f2012689700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f200c1a8f00 con 0x7f200c10b6f0 2024-04-14T15:10:20.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.475+0000 7f2012689700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f200c1aa0e0 con 0x7f200c108980 2024-04-14T15:10:20.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.475+0000 7f200bfff700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f200c108980 0x7f200c105020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:48920/0 (socket says 172.21.15.90:48920) 2024-04-14T15:10:20.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.475+0000 7f200bfff700 1 -- 172.21.15.90:0/2630886422 learned_addr learned my addr 172.21.15.90:0/2630886422 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:10:20.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.475+0000 7f20097fa700 10 client.?.objecter ms_handle_connect 0x7f200c10f2f0 2024-04-14T15:10:20.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.475+0000 7f20097fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:10:20.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.476+0000 7f20097fa700 10 client.?.objecter ms_handle_connect 0x7f200c108980 2024-04-14T15:10:20.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.476+0000 7f20097fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:10:20.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.476+0000 7f20097fa700 10 client.?.objecter ms_handle_connect 0x7f200c10b6f0 2024-04-14T15:10:20.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.476+0000 7f20097fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:10:20.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.476+0000 7f20097fa700 1 -- 172.21.15.90:0/2630886422 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 879914622 0 0) 0x7f200c1a8f00 con 0x7f200c10b6f0 2024-04-14T15:10:20.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.476+0000 7f20097fa700 1 -- 172.21.15.90:0/2630886422 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1ff40035b0 con 0x7f200c10b6f0 2024-04-14T15:10:20.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.476+0000 7f20097fa700 1 -- 172.21.15.90:0/2630886422 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3961364933 0 0) 0x7f200c1081d0 con 0x7f200c10f2f0 2024-04-14T15:10:20.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.476+0000 7f20097fa700 1 -- 172.21.15.90:0/2630886422 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f200c1a8f00 con 0x7f200c10f2f0 2024-04-14T15:10:20.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.476+0000 7f20097fa700 1 -- 172.21.15.90:0/2630886422 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 75626916 0 0) 0x7f200c1aa0e0 con 0x7f200c108980 2024-04-14T15:10:20.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.476+0000 7f20097fa700 1 -- 172.21.15.90:0/2630886422 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f200c1081d0 con 0x7f200c108980 2024-04-14T15:10:20.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.476+0000 7f20097fa700 1 -- 172.21.15.90:0/2630886422 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 865868964 0 0) 0x7f1ff40035b0 con 0x7f200c10b6f0 2024-04-14T15:10:20.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.476+0000 7f20097fa700 1 -- 172.21.15.90:0/2630886422 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f200c1aa0e0 con 0x7f200c10b6f0 2024-04-14T15:10:20.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.476+0000 7f20097fa700 1 -- 172.21.15.90:0/2630886422 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1876917902 0 0) 0x7f200c1a8f00 con 0x7f200c10f2f0 2024-04-14T15:10:20.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.476+0000 7f20097fa700 1 -- 172.21.15.90:0/2630886422 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1ff40035b0 con 0x7f200c10f2f0 2024-04-14T15:10:20.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.476+0000 7f20097fa700 1 -- 172.21.15.90:0/2630886422 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f2004001bb0 con 0x7f200c10f2f0 2024-04-14T15:10:20.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.476+0000 7f20097fa700 1 -- 172.21.15.90:0/2630886422 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1381343155 0 0) 0x7f200c1081d0 con 0x7f200c108980 2024-04-14T15:10:20.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.476+0000 7f20097fa700 1 -- 172.21.15.90:0/2630886422 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f200c1a8f00 con 0x7f200c108980 2024-04-14T15:10:20.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.476+0000 7f20097fa700 1 -- 172.21.15.90:0/2630886422 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f2000001430 con 0x7f200c108980 2024-04-14T15:10:20.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.476+0000 7f20097fa700 1 -- 172.21.15.90:0/2630886422 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f1ffc001910 con 0x7f200c10b6f0 2024-04-14T15:10:20.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.476+0000 7f20097fa700 1 -- 172.21.15.90:0/2630886422 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4232161825 0 0) 0x7f200c1aa0e0 con 0x7f200c10b6f0 2024-04-14T15:10:20.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.476+0000 7f20097fa700 1 -- 172.21.15.90:0/2630886422 >> v1:172.21.15.90:6790/0 conn(0x7f200c108980 legacy=0x7f200c105020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:20.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.476+0000 7f20097fa700 1 -- 172.21.15.90:0/2630886422 >> v1:172.21.15.90:6789/0 conn(0x7f200c10f2f0 legacy=0x7f200c1a67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:20.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.476+0000 7f20097fa700 1 -- 172.21.15.90:0/2630886422 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f200c1ab2c0 con 0x7f200c10b6f0 2024-04-14T15:10:20.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.476+0000 7f2012689700 1 -- 172.21.15.90:0/2630886422 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f200c1aa310 con 0x7f200c10b6f0 2024-04-14T15:10:20.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.476+0000 7f2012689700 10 client.24514.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:10:20.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.476+0000 7f2012689700 1 -- 172.21.15.90:0/2630886422 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f200c1aa710 con 0x7f200c10b6f0 2024-04-14T15:10:20.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.476+0000 7f2012689700 1 librados: init done 2024-04-14T15:10:20.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.476+0000 7f20097fa700 1 -- 172.21.15.90:0/2630886422 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f1ffc003b40 con 0x7f200c10b6f0 2024-04-14T15:10:20.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.476+0000 7f20097fa700 1 -- 172.21.15.90:0/2630886422 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f1ffc004a20 con 0x7f200c10b6f0 2024-04-14T15:10:20.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.477+0000 7f2012689700 1 -- 172.21.15.90:0/2630886422 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ff8005120 con 0x7f200c10b6f0 2024-04-14T15:10:20.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.478+0000 7f20097fa700 1 -- 172.21.15.90:0/2630886422 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 17) ==== 92688+0+0 (unknown 1179923196 0 0) 0x7f1ffc01b6c0 con 0x7f200c10b6f0 2024-04-14T15:10:20.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.478+0000 7f200bfff700 1 -- 172.21.15.90:0/2630886422 >> v1:172.21.15.90:6800/491071118 conn(0x7f1ff4071310 legacy=0x7f1ff40737d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.90:6800/491071118 2024-04-14T15:10:20.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.478+0000 7f20097fa700 1 -- 172.21.15.90:0/2630886422 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(77..77 src has 12..77) ==== 6030+0+0 (unknown 3485272705 0 0) 0x7f1ffc088fd0 con 0x7f200c10b6f0 2024-04-14T15:10:20.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.478+0000 7f20097fa700 10 client.24514.objecter ms_dispatch 0x7f200c001550 osd_map(77..77 src has 12..77) 2024-04-14T15:10:20.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.478+0000 7f20097fa700 3 client.24514.objecter handle_osd_map got epochs [77,77] > 0 2024-04-14T15:10:20.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.478+0000 7f20097fa700 3 client.24514.objecter handle_osd_map decoding full epoch 77 2024-04-14T15:10:20.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.478+0000 7f20097fa700 20 client.24514.objecter dump_active .. 0 homeless 2024-04-14T15:10:20.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.483+0000 7f20097fa700 1 -- 172.21.15.90:0/2630886422 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f1ffc0560e0 con 0x7f200c10b6f0 2024-04-14T15:10:20.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.679+0000 7f200bfff700 1 -- 172.21.15.90:0/2630886422 >> v1:172.21.15.90:6800/491071118 conn(0x7f1ff4071310 legacy=0x7f1ff40737d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.90:6800/491071118 2024-04-14T15:10:20.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.689+0000 7f2012689700 1 -- 172.21.15.90:0/2630886422 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1ff8005410 con 0x7f200c10b6f0 2024-04-14T15:10:20.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.690+0000 7f20097fa700 1 -- 172.21.15.90:0/2630886422 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) ==== 74+0+20693 (unknown 1875750491 0 3747541200) 0x7f1ffc0559e0 con 0x7f200c10b6f0 2024-04-14T15:10:20.691 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:10:20.691 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":77,"fsid":"0e288098-fa70-11ee-bc8e-c7b262605968","created":"2024-04-14T15:04:05.808313+0000","modified":"2024-04-14T15:10:16.999066+0000","last_up_change":"2024-04-14T15:09:29.769352+0000","last_in_change":"2024-04-14T15:09:11.803164+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-14T15:07:23.959628+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-04-14T15:09:41.608420+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"datapool","create_time":"2024-04-14T15:09:41.744391+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":69,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-04-14T15:09:43.088157+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-04-14T15:09:45.042722+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-04-14T15:09:47.111336+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"76772cfc-89d1-4013-b617-32284ed8dfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6801","nonce":3761806885}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6802","nonce":3761806885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6804","nonce":3761806885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6803","nonce":3761806885}]},"public_addr":"172.21.15.90:6801/3761806885","cluster_addr":"172.21.15.90:6802/3761806885","heartbeat_back_addr":"172.21.15.90:6804/3761806885","heartbeat_front_addr":"172.21.15.90:6803/3761806885","state":["exists","up"]},{"osd":1,"uuid":"de4a7fa2-7d57-4ff3-a4fa-28c0cde8ef6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6805","nonce":4170361699}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6806","nonce":4170361699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6808","nonce":4170361699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6807","nonce":4170361699}]},"public_addr":"172.21.15.90:6805/4170361699","cluster_addr":"172.21.15.90:6806/4170361699","heartbeat_back_addr":"172.21.15.90:6808/4170361699","heartbeat_front_addr":"172.21.15.90:6807/4170361699","state":["exists","up"]},{"osd":2,"uuid":"9c39b5ff-6175-46ba-9d71-370581c6ff3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6809","nonce":1686184251}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6810","nonce":1686184251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6812","nonce":1686184251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6811","nonce":1686184251}]},"public_addr":"172.21.15.90:6809/1686184251","cluster_addr":"172.21.15.90:6810/1686184251","heartbeat_back_addr":"172.21.15.90:6812/1686184251","heartbeat_front_addr":"172.21.15.90:6811/1686184251","state":["exists","up"]},{"osd":3,"uuid":"efd2ca45-5b80-4013-a571-499e2cb5b2e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6813","nonce":2270166774}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6814","nonce":2270166774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6816","nonce":2270166774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6815","nonce":2270166774}]},"public_addr":"172.21.15.90:6813/2270166774","cluster_addr":"172.21.15.90:6814/2270166774","heartbeat_back_addr":"172.21.15.90:6816/2270166774","heartbeat_front_addr":"172.21.15.90:6815/2270166774","state":["exists","up"]},{"osd":4,"uuid":"e3d9013d-5eb6-433c-96b3-3fc38ba7ac8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6800","nonce":3887549733}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6801","nonce":3887549733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6803","nonce":3887549733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6802","nonce":3887549733}]},"public_addr":"172.21.15.92:6800/3887549733","cluster_addr":"172.21.15.92:6801/3887549733","heartbeat_back_addr":"172.21.15.92:6803/3887549733","heartbeat_front_addr":"172.21.15.92:6802/3887549733","state":["exists","up"]},{"osd":5,"uuid":"1ec41805-6f82-4139-81bc-75305ba9e958","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6804","nonce":3244561440}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6805","nonce":3244561440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6807","nonce":3244561440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6806","nonce":3244561440}]},"public_addr":"172.21.15.92:6804/3244561440","cluster_addr":"172.21.15.92:6805/3244561440","heartbeat_back_addr":"172.21.15.92:6807/3244561440","heartbeat_front_addr":"172.21.15.92:6806/3244561440","state":["exists","up"]},{"osd":6,"uuid":"ca977ab2-bf8d-4b3a-ae5e-0efa5bc7a3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6808","nonce":636125963}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6809","nonce":636125963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6811","nonce":636125963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6810","nonce":636125963}]},"public_addr":"172.21.15.92:6808/636125963","cluster_addr":"172.21.15.92:6809/636125963","heartbeat_back_addr":"172.21.15.92:6811/636125963","heartbeat_front_addr":"172.21.15.92:6810/636125963","state":["exists","up"]},{"osd":7,"uuid":"e5354645-79ec-4d4c-b6e5-f9b0653915ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6812","nonce":1773706716}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6813","nonce":1773706716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6815","nonce":1773706716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6814","nonce":1773706716}]},"public_addr":"172.21.15.92:6812/1773706716","cluster_addr":"172.21.15.92:6813/1773706716","heartbeat_back_addr":"172.21.15.92:6815/1773706716","heartbeat_front_addr":"172.21.15.92:6814/1773706716","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T15:09:04.496851+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T15:09:24.387925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T15:09:38.429314+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T15:09:01.292185+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T15:09:27.428788+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T15:09:52.454083+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T15:10:16.090875+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T15:09:28.295984+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.1c","mappings":[{"from":7,"to":4}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:0/1139868610":"2024-04-15T15:05:05.882361+0000","172.21.15.90:0/977818204":"2024-04-15T15:04:39.251711+0000","172.21.15.90:6800/941091171":"2024-04-15T15:05:05.882361+0000","172.21.15.90:0/3671453561":"2024-04-15T15:05:05.882361+0000","172.21.15.90:0/2611856200":"2024-04-15T15:05:05.882361+0000","172.21.15.90:6800/2306258962":"2024-04-15T15:04:39.251711+0000","172.21.15.90:0/4051692380":"2024-04-15T15:04:39.251711+0000","172.21.15.90:0/2681502561":"2024-04-15T15:04:39.251711+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-14T15:10:20.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.692+0000 7f2012689700 10 librados: watch_flush enter 2024-04-14T15:10:20.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.693+0000 7f2012689700 10 librados: watch_flush exit 2024-04-14T15:10:20.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.693+0000 7f2012689700 20 client.24514.objecter shutdown clearing up homeless session... 2024-04-14T15:10:20.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.693+0000 7f2012689700 10 client.24514.objecter successfully canceled tick 2024-04-14T15:10:20.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.693+0000 7f2012689700 1 -- 172.21.15.90:0/2630886422 >> v1:172.21.15.90:6800/491071118 conn(0x7f1ff4071310 legacy=0x7f1ff40737d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-14T15:10:20.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.693+0000 7f2012689700 1 -- 172.21.15.90:0/2630886422 >> v1:172.21.15.92:6789/0 conn(0x7f200c10b6f0 legacy=0x7f200c105730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:20.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.693+0000 7f2012689700 1 -- 172.21.15.90:0/2630886422 shutdown_connections 2024-04-14T15:10:20.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.693+0000 7f2012689700 1 -- 172.21.15.90:0/2630886422 >> 172.21.15.90:0/2630886422 conn(0x7f200c076aa0 msgr2=0x7f200c1117b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:10:20.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.693+0000 7f2012689700 1 -- 172.21.15.90:0/2630886422 shutdown_connections 2024-04-14T15:10:20.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.693+0000 7f2012689700 1 -- 172.21.15.90:0/2630886422 wait complete. 2024-04-14T15:10:20.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:20.693+0000 7f2012689700 1 librados: shutdown 2024-04-14T15:10:20.742 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:10:20 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-x[36044]: 2024-04-14T15:10:20.485+0000 7f629731b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-14T15:10:20.743 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:10:20 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:10:20.638+0000 7f5636b80200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-14T15:10:21.043 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:20 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2630886422' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-14T15:10:21.044 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:20 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2630886422' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-14T15:10:21.044 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:10:20 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:10:20.906+0000 7f5636b80200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-14T15:10:21.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:20 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2630886422' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-14T15:10:21.122 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:10:20 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-x[36044]: 2024-04-14T15:10:20.746+0000 7f629731b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-14T15:10:21.122 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:10:20 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-x[36044]: 2024-04-14T15:10:20.986+0000 7f629731b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-14T15:10:21.419 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:10:21 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:10:21.150+0000 7f5636b80200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-14T15:10:21.454 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-14T15:10:21.455 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd dump --format=json 2024-04-14T15:10:21.702 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:10:22.013 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:21 smithi090 ceph-mon[41670]: osdmap e78: 8 total, 8 up, 8 in 2024-04-14T15:10:22.013 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:21 smithi090 ceph-mon[30399]: osdmap e78: 8 total, 8 up, 8 in 2024-04-14T15:10:22.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:21 smithi092 ceph-mon[33966]: osdmap e78: 8 total, 8 up, 8 in 2024-04-14T15:10:22.122 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:10:21 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-x[36044]: 2024-04-14T15:10:21.834+0000 7f629731b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-14T15:10:22.441 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:10:22 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:10:22.011+0000 7f5636b80200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-14T15:10:22.622 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:10:22 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-x[36044]: 2024-04-14T15:10:22.421+0000 7f629731b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-14T15:10:22.622 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:10:22 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-x[36044]: 2024-04-14T15:10:22.519+0000 7f629731b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-14T15:10:22.917 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:10:22 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:10:22.593+0000 7f5636b80200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-14T15:10:22.917 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:10:22 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:10:22.692+0000 7f5636b80200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-14T15:10:22.917 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:22 smithi090 ceph-mon[30399]: Standby manager daemon x restarted 2024-04-14T15:10:22.917 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:22 smithi090 ceph-mon[30399]: Standby manager daemon x started 2024-04-14T15:10:22.917 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:22 smithi090 ceph-mon[30399]: from='mgr.? v1:172.21.15.92:0/3471652138' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-14T15:10:22.917 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:22 smithi090 ceph-mon[30399]: from='mgr.? v1:172.21.15.92:0/3471652138' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T15:10:22.918 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:22 smithi090 ceph-mon[30399]: from='mgr.? v1:172.21.15.92:0/3471652138' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-14T15:10:22.918 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:22 smithi090 ceph-mon[30399]: from='mgr.? v1:172.21.15.92:0/3471652138' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T15:10:22.918 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:22 smithi090 ceph-mon[41670]: Standby manager daemon x restarted 2024-04-14T15:10:22.918 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:22 smithi090 ceph-mon[41670]: Standby manager daemon x started 2024-04-14T15:10:22.918 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:22 smithi090 ceph-mon[41670]: from='mgr.? v1:172.21.15.92:0/3471652138' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-14T15:10:22.918 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:22 smithi090 ceph-mon[41670]: from='mgr.? v1:172.21.15.92:0/3471652138' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T15:10:22.918 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:22 smithi090 ceph-mon[41670]: from='mgr.? v1:172.21.15.92:0/3471652138' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-14T15:10:22.918 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:22 smithi090 ceph-mon[41670]: from='mgr.? v1:172.21.15.92:0/3471652138' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T15:10:23.053 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:22 smithi092 ceph-mon[33966]: Standby manager daemon x restarted 2024-04-14T15:10:23.053 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:22 smithi092 ceph-mon[33966]: Standby manager daemon x started 2024-04-14T15:10:23.053 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:22 smithi092 ceph-mon[33966]: from='mgr.? v1:172.21.15.92:0/3471652138' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-14T15:10:23.053 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:22 smithi092 ceph-mon[33966]: from='mgr.? v1:172.21.15.92:0/3471652138' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T15:10:23.053 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:22 smithi092 ceph-mon[33966]: from='mgr.? v1:172.21.15.92:0/3471652138' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-14T15:10:23.053 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:22 smithi092 ceph-mon[33966]: from='mgr.? v1:172.21.15.92:0/3471652138' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T15:10:23.053 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:10:22 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-x[36044]: 2024-04-14T15:10:22.737+0000 7f629731b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-14T15:10:23.053 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:10:22 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-x[36044]: [14/Apr/2024:15:10:22] ENGINE Bus STARTING 2024-04-14T15:10:23.054 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:10:22 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-x[36044]: CherryPy Checker: 2024-04-14T15:10:23.054 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:10:22 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-x[36044]: The Application mounted at '' has an empty config. 2024-04-14T15:10:23.054 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:10:22 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-x[36044]: 2024-04-14T15:10:23.054 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:10:22 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-x[36044]: [14/Apr/2024:15:10:22] ENGINE Serving on http://:::9283 2024-04-14T15:10:23.054 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:10:22 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-x[36044]: [14/Apr/2024:15:10:22] ENGINE Bus STARTED 2024-04-14T15:10:23.191 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:10:22 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:10:22.916+0000 7f5636b80200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-14T15:10:23.192 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:10:23 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: [14/Apr/2024:15:10:23] ENGINE Bus STARTING 2024-04-14T15:10:23.192 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:10:23 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: CherryPy Checker: 2024-04-14T15:10:23.192 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:10:23 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: The Application mounted at '' has an empty config. 2024-04-14T15:10:23.192 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:10:23 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:10:23.192 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:10:23 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: [14/Apr/2024:15:10:23] ENGINE Serving on http://:::9283 2024-04-14T15:10:23.192 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:10:23 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: [14/Apr/2024:15:10:23] ENGINE Bus STARTED 2024-04-14T15:10:23.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:23.506+0000 7fcd92736700 1 -- 172.21.15.90:0/2793509386 >> v1:172.21.15.90:6789/0 conn(0x7fcd8c108870 legacy=0x7fcd8c108c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:23.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:23.506+0000 7fcd92736700 1 -- 172.21.15.90:0/2793509386 shutdown_connections 2024-04-14T15:10:23.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:23.506+0000 7fcd92736700 1 -- 172.21.15.90:0/2793509386 >> 172.21.15.90:0/2793509386 conn(0x7fcd8c0fe3a0 msgr2=0x7fcd8c1007c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:10:23.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:23.506+0000 7fcd92736700 1 -- 172.21.15.90:0/2793509386 shutdown_connections 2024-04-14T15:10:23.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:23.506+0000 7fcd92736700 1 -- 172.21.15.90:0/2793509386 wait complete. 2024-04-14T15:10:23.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:23.507+0000 7fcd92736700 1 librados: starting msgr at 2024-04-14T15:10:23.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:23.507+0000 7fcd92736700 1 librados: starting objecter 2024-04-14T15:10:23.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:23.507+0000 7fcd92736700 1 Processor -- start 2024-04-14T15:10:23.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:23.507+0000 7fcd92736700 1 -- start start 2024-04-14T15:10:23.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:23.507+0000 7fcd92736700 1 librados: setting wanted keys 2024-04-14T15:10:23.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:23.507+0000 7fcd92736700 1 librados: calling monclient init 2024-04-14T15:10:23.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:23.507+0000 7fcd92736700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fcd8c1a7ef0 con 0x7fcd8c10f2f0 2024-04-14T15:10:23.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:23.507+0000 7fcd92736700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fcd8c1a90d0 con 0x7fcd8c10b720 2024-04-14T15:10:23.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:23.507+0000 7fcd92736700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fcd8c1aa2b0 con 0x7fcd8c108870 2024-04-14T15:10:23.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:23.507+0000 7fcd90cd3700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fcd8c10f2f0 0x7fcd8c1a67d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:56422/0 (socket says 172.21.15.90:56422) 2024-04-14T15:10:23.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:23.507+0000 7fcd90cd3700 1 -- 172.21.15.90:0/707165538 learned_addr learned my addr 172.21.15.90:0/707165538 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:10:23.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:23.508+0000 7fcd897fa700 10 client.?.objecter ms_handle_connect 0x7fcd8c10f2f0 2024-04-14T15:10:23.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:23.508+0000 7fcd897fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:10:23.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:23.508+0000 7fcd897fa700 10 client.?.objecter ms_handle_connect 0x7fcd8c108870 2024-04-14T15:10:23.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:23.508+0000 7fcd897fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:10:23.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:23.508+0000 7fcd897fa700 10 client.?.objecter ms_handle_connect 0x7fcd8c10b720 2024-04-14T15:10:23.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:23.508+0000 7fcd897fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:10:23.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:23.508+0000 7fcd897fa700 1 -- 172.21.15.90:0/707165538 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 480578652 0 0) 0x7fcd8c1aa2b0 con 0x7fcd8c108870 2024-04-14T15:10:23.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:23.508+0000 7fcd897fa700 1 -- 172.21.15.90:0/707165538 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcd740035b0 con 0x7fcd8c108870 2024-04-14T15:10:23.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:23.508+0000 7fcd897fa700 1 -- 172.21.15.90:0/707165538 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1623817071 0 0) 0x7fcd8c1a7ef0 con 0x7fcd8c10f2f0 2024-04-14T15:10:23.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:23.508+0000 7fcd897fa700 1 -- 172.21.15.90:0/707165538 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcd8c1aa2b0 con 0x7fcd8c10f2f0 2024-04-14T15:10:23.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:23.508+0000 7fcd897fa700 1 -- 172.21.15.90:0/707165538 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2032904564 0 0) 0x7fcd8c1a90d0 con 0x7fcd8c10b720 2024-04-14T15:10:23.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:23.508+0000 7fcd897fa700 1 -- 172.21.15.90:0/707165538 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcd8c1a7ef0 con 0x7fcd8c10b720 2024-04-14T15:10:23.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:23.508+0000 7fcd897fa700 1 -- 172.21.15.90:0/707165538 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1468753867 0 0) 0x7fcd740035b0 con 0x7fcd8c108870 2024-04-14T15:10:23.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:23.508+0000 7fcd897fa700 1 -- 172.21.15.90:0/707165538 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcd8c1a90d0 con 0x7fcd8c108870 2024-04-14T15:10:23.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:23.508+0000 7fcd897fa700 1 -- 172.21.15.90:0/707165538 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3008985759 0 0) 0x7fcd8c1a7ef0 con 0x7fcd8c10b720 2024-04-14T15:10:23.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:23.508+0000 7fcd897fa700 1 -- 172.21.15.90:0/707165538 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcd74006ff0 con 0x7fcd8c10b720 2024-04-14T15:10:23.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:23.508+0000 7fcd897fa700 1 -- 172.21.15.90:0/707165538 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fcd800018b0 con 0x7fcd8c10b720 2024-04-14T15:10:23.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:23.508+0000 7fcd897fa700 1 -- 172.21.15.90:0/707165538 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3243299941 0 0) 0x7fcd8c1aa2b0 con 0x7fcd8c10f2f0 2024-04-14T15:10:23.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:23.508+0000 7fcd897fa700 1 -- 172.21.15.90:0/707165538 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcd8c1a7ef0 con 0x7fcd8c10f2f0 2024-04-14T15:10:23.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:23.508+0000 7fcd897fa700 1 -- 172.21.15.90:0/707165538 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fcd7c0015d0 con 0x7fcd8c108870 2024-04-14T15:10:23.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:23.508+0000 7fcd897fa700 1 -- 172.21.15.90:0/707165538 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fcd84001bf0 con 0x7fcd8c10f2f0 2024-04-14T15:10:23.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:23.508+0000 7fcd897fa700 1 -- 172.21.15.90:0/707165538 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 709494885 0 0) 0x7fcd74006ff0 con 0x7fcd8c10b720 2024-04-14T15:10:23.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:23.508+0000 7fcd897fa700 1 -- 172.21.15.90:0/707165538 >> v1:172.21.15.90:6790/0 conn(0x7fcd8c108870 legacy=0x7fcd8c10d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:23.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:23.508+0000 7fcd897fa700 1 -- 172.21.15.90:0/707165538 >> v1:172.21.15.90:6789/0 conn(0x7fcd8c10f2f0 legacy=0x7fcd8c1a67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:23.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:23.508+0000 7fcd897fa700 1 -- 172.21.15.90:0/707165538 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd8c1ab490 con 0x7fcd8c10b720 2024-04-14T15:10:23.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:23.509+0000 7fcd92736700 1 -- 172.21.15.90:0/707165538 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fcd8c1aa4e0 con 0x7fcd8c10b720 2024-04-14T15:10:23.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:23.509+0000 7fcd92736700 10 client.24538.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:10:23.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:23.509+0000 7fcd92736700 1 -- 172.21.15.90:0/707165538 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fcd8c1aa9a0 con 0x7fcd8c10b720 2024-04-14T15:10:23.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:23.509+0000 7fcd92736700 1 librados: init done 2024-04-14T15:10:23.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:23.509+0000 7fcd897fa700 1 -- 172.21.15.90:0/707165538 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fcd80003b40 con 0x7fcd8c10b720 2024-04-14T15:10:23.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:23.509+0000 7fcd897fa700 1 -- 172.21.15.90:0/707165538 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fcd80004910 con 0x7fcd8c10b720 2024-04-14T15:10:23.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:23.510+0000 7fcd92736700 1 -- 172.21.15.90:0/707165538 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd78005120 con 0x7fcd8c10b720 2024-04-14T15:10:23.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:23.510+0000 7fcd897fa700 1 -- 172.21.15.90:0/707165538 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 19) ==== 92340+0+0 (unknown 2114501483 0 0) 0x7fcd8001b1b0 con 0x7fcd8c10b720 2024-04-14T15:10:23.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:23.511+0000 7fcd897fa700 1 -- 172.21.15.90:0/707165538 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(79..79 src has 12..79) ==== 6288+0+0 (unknown 451107537 0 0) 0x7fcd80088870 con 0x7fcd8c10b720 2024-04-14T15:10:23.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:23.511+0000 7fcd897fa700 10 client.24538.objecter ms_dispatch 0x7fcd8c001550 osd_map(79..79 src has 12..79) 2024-04-14T15:10:23.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:23.511+0000 7fcd897fa700 3 client.24538.objecter handle_osd_map got epochs [79,79] > 0 2024-04-14T15:10:23.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:23.511+0000 7fcd897fa700 3 client.24538.objecter handle_osd_map decoding full epoch 79 2024-04-14T15:10:23.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:23.511+0000 7fcd897fa700 20 client.24538.objecter dump_active .. 0 homeless 2024-04-14T15:10:23.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:23.515+0000 7fcd897fa700 1 -- 172.21.15.90:0/707165538 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fcd80055de0 con 0x7fcd8c10b720 2024-04-14T15:10:23.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:23.722+0000 7fcd92736700 1 -- 172.21.15.90:0/707165538 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcd78005410 con 0x7fcd8c10b720 2024-04-14T15:10:23.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:23.723+0000 7fcd897fa700 1 -- 172.21.15.90:0/707165538 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) ==== 74+0+21063 (unknown 2549029860 0 3436291654) 0x7fcd800556e0 con 0x7fcd8c10b720 2024-04-14T15:10:23.724 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:10:23.724 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":79,"fsid":"0e288098-fa70-11ee-bc8e-c7b262605968","created":"2024-04-14T15:04:05.808313+0000","modified":"2024-04-14T15:10:22.918862+0000","last_up_change":"2024-04-14T15:09:29.769352+0000","last_in_change":"2024-04-14T15:09:11.803164+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-14T15:07:23.959628+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-04-14T15:09:41.608420+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"datapool","create_time":"2024-04-14T15:09:41.744391+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":69,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-04-14T15:09:43.088157+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-04-14T15:09:45.042722+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-04-14T15:09:47.111336+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"76772cfc-89d1-4013-b617-32284ed8dfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6801","nonce":3761806885}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6802","nonce":3761806885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6804","nonce":3761806885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6803","nonce":3761806885}]},"public_addr":"172.21.15.90:6801/3761806885","cluster_addr":"172.21.15.90:6802/3761806885","heartbeat_back_addr":"172.21.15.90:6804/3761806885","heartbeat_front_addr":"172.21.15.90:6803/3761806885","state":["exists","up"]},{"osd":1,"uuid":"de4a7fa2-7d57-4ff3-a4fa-28c0cde8ef6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6805","nonce":4170361699}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6806","nonce":4170361699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6808","nonce":4170361699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6807","nonce":4170361699}]},"public_addr":"172.21.15.90:6805/4170361699","cluster_addr":"172.21.15.90:6806/4170361699","heartbeat_back_addr":"172.21.15.90:6808/4170361699","heartbeat_front_addr":"172.21.15.90:6807/4170361699","state":["exists","up"]},{"osd":2,"uuid":"9c39b5ff-6175-46ba-9d71-370581c6ff3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6809","nonce":1686184251}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6810","nonce":1686184251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6812","nonce":1686184251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6811","nonce":1686184251}]},"public_addr":"172.21.15.90:6809/1686184251","cluster_addr":"172.21.15.90:6810/1686184251","heartbeat_back_addr":"172.21.15.90:6812/1686184251","heartbeat_front_addr":"172.21.15.90:6811/1686184251","state":["exists","up"]},{"osd":3,"uuid":"efd2ca45-5b80-4013-a571-499e2cb5b2e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6813","nonce":2270166774}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6814","nonce":2270166774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6816","nonce":2270166774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6815","nonce":2270166774}]},"public_addr":"172.21.15.90:6813/2270166774","cluster_addr":"172.21.15.90:6814/2270166774","heartbeat_back_addr":"172.21.15.90:6816/2270166774","heartbeat_front_addr":"172.21.15.90:6815/2270166774","state":["exists","up"]},{"osd":4,"uuid":"e3d9013d-5eb6-433c-96b3-3fc38ba7ac8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6800","nonce":3887549733}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6801","nonce":3887549733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6803","nonce":3887549733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6802","nonce":3887549733}]},"public_addr":"172.21.15.92:6800/3887549733","cluster_addr":"172.21.15.92:6801/3887549733","heartbeat_back_addr":"172.21.15.92:6803/3887549733","heartbeat_front_addr":"172.21.15.92:6802/3887549733","state":["exists","up"]},{"osd":5,"uuid":"1ec41805-6f82-4139-81bc-75305ba9e958","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6804","nonce":3244561440}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6805","nonce":3244561440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6807","nonce":3244561440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6806","nonce":3244561440}]},"public_addr":"172.21.15.92:6804/3244561440","cluster_addr":"172.21.15.92:6805/3244561440","heartbeat_back_addr":"172.21.15.92:6807/3244561440","heartbeat_front_addr":"172.21.15.92:6806/3244561440","state":["exists","up"]},{"osd":6,"uuid":"ca977ab2-bf8d-4b3a-ae5e-0efa5bc7a3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6808","nonce":636125963}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6809","nonce":636125963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6811","nonce":636125963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6810","nonce":636125963}]},"public_addr":"172.21.15.92:6808/636125963","cluster_addr":"172.21.15.92:6809/636125963","heartbeat_back_addr":"172.21.15.92:6811/636125963","heartbeat_front_addr":"172.21.15.92:6810/636125963","state":["exists","up"]},{"osd":7,"uuid":"e5354645-79ec-4d4c-b6e5-f9b0653915ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6812","nonce":1773706716}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6813","nonce":1773706716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6815","nonce":1773706716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6814","nonce":1773706716}]},"public_addr":"172.21.15.92:6812/1773706716","cluster_addr":"172.21.15.92:6813/1773706716","heartbeat_back_addr":"172.21.15.92:6815/1773706716","heartbeat_front_addr":"172.21.15.92:6814/1773706716","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T15:09:04.496851+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T15:09:24.387925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T15:09:38.429314+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T15:09:01.292185+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T15:09:27.428788+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T15:09:52.454083+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T15:10:16.090875+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T15:09:28.295984+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.1c","mappings":[{"from":7,"to":4}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:0/508768479":"2024-04-15T15:10:22.918674+0000","172.21.15.90:0/1823072952":"2024-04-15T15:10:22.918674+0000","172.21.15.90:6800/491071118":"2024-04-15T15:10:22.918674+0000","172.21.15.90:0/1337064035":"2024-04-15T15:10:22.918674+0000","172.21.15.90:0/769462306":"2024-04-15T15:10:22.918674+0000","172.21.15.90:0/1139868610":"2024-04-15T15:05:05.882361+0000","172.21.15.90:0/93763408":"2024-04-15T15:10:22.918674+0000","172.21.15.90:0/977818204":"2024-04-15T15:04:39.251711+0000","172.21.15.90:6800/941091171":"2024-04-15T15:05:05.882361+0000","172.21.15.90:0/3671453561":"2024-04-15T15:05:05.882361+0000","172.21.15.90:0/2611856200":"2024-04-15T15:05:05.882361+0000","172.21.15.90:6800/2306258962":"2024-04-15T15:04:39.251711+0000","172.21.15.90:0/4051692380":"2024-04-15T15:04:39.251711+0000","172.21.15.90:0/2681502561":"2024-04-15T15:04:39.251711+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-14T15:10:23.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:23.726+0000 7fcd92736700 10 librados: watch_flush enter 2024-04-14T15:10:23.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:23.726+0000 7fcd92736700 10 librados: watch_flush exit 2024-04-14T15:10:23.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:23.726+0000 7fcd92736700 20 client.24538.objecter shutdown clearing up homeless session... 2024-04-14T15:10:23.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:23.726+0000 7fcd92736700 10 client.24538.objecter successfully canceled tick 2024-04-14T15:10:23.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:23.726+0000 7fcd92736700 1 -- 172.21.15.90:0/707165538 >> v1:172.21.15.92:6789/0 conn(0x7fcd8c10b720 legacy=0x7fcd8c10e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:23.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:23.726+0000 7fcd92736700 1 -- 172.21.15.90:0/707165538 shutdown_connections 2024-04-14T15:10:23.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:23.726+0000 7fcd92736700 1 -- 172.21.15.90:0/707165538 >> 172.21.15.90:0/707165538 conn(0x7fcd8c0fe3a0 msgr2=0x7fcd8c102910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:10:23.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:23.726+0000 7fcd92736700 1 -- 172.21.15.90:0/707165538 shutdown_connections 2024-04-14T15:10:23.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:23.726+0000 7fcd92736700 1 -- 172.21.15.90:0/707165538 wait complete. 2024-04-14T15:10:23.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:23.726+0000 7fcd92736700 1 librados: shutdown 2024-04-14T15:10:24.107 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:10:23 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:10:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:10:24.108 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:23 smithi090 ceph-mon[30399]: mgrmap e18: y(active, since 5m), standbys: x 2024-04-14T15:10:24.108 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:23 smithi090 ceph-mon[30399]: Active manager daemon y restarted 2024-04-14T15:10:24.108 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:23 smithi090 ceph-mon[30399]: Activating manager daemon y 2024-04-14T15:10:24.108 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:23 smithi090 ceph-mon[30399]: osdmap e79: 8 total, 8 up, 8 in 2024-04-14T15:10:24.108 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:23 smithi090 ceph-mon[30399]: mgrmap e19: y(active, starting, since 0.0129801s), standbys: x 2024-04-14T15:10:24.108 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:23 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T15:10:24.108 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:23 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:10:24.108 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:23 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:10:24.108 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:23 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-14T15:10:24.108 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:23 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-14T15:10:24.108 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:23 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T15:10:24.108 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:23 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T15:10:24.108 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:23 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T15:10:24.108 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:23 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T15:10:24.108 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:23 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T15:10:24.109 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:23 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T15:10:24.109 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:23 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T15:10:24.109 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:23 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T15:10:24.109 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:23 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-14T15:10:24.109 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:23 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-14T15:10:24.109 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:23 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-14T15:10:24.109 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:23 smithi090 ceph-mon[30399]: Manager daemon y is now available 2024-04-14T15:10:24.109 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:23 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:10:24.109 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:23 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:10:24.109 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:23 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:10:24.109 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:23 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T15:10:24.109 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:23 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T15:10:24.109 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:23 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/707165538' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-14T15:10:24.109 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:23 smithi090 ceph-mon[41670]: mgrmap e18: y(active, since 5m), standbys: x 2024-04-14T15:10:24.110 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:23 smithi090 ceph-mon[41670]: Active manager daemon y restarted 2024-04-14T15:10:24.110 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:23 smithi090 ceph-mon[41670]: Activating manager daemon y 2024-04-14T15:10:24.110 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:23 smithi090 ceph-mon[41670]: osdmap e79: 8 total, 8 up, 8 in 2024-04-14T15:10:24.110 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:23 smithi090 ceph-mon[41670]: mgrmap e19: y(active, starting, since 0.0129801s), standbys: x 2024-04-14T15:10:24.110 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:23 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T15:10:24.110 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:23 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:10:24.110 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:23 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:10:24.110 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:23 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-14T15:10:24.110 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:23 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-14T15:10:24.110 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:23 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T15:10:24.110 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:23 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T15:10:24.110 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:23 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T15:10:24.110 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:23 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T15:10:24.110 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:23 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T15:10:24.111 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:23 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T15:10:24.111 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:23 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T15:10:24.111 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:23 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T15:10:24.111 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:23 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-14T15:10:24.111 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:23 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-14T15:10:24.111 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:23 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-14T15:10:24.111 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:23 smithi090 ceph-mon[41670]: Manager daemon y is now available 2024-04-14T15:10:24.111 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:23 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:10:24.111 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:23 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:10:24.111 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:23 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:10:24.111 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:23 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T15:10:24.111 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:23 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T15:10:24.111 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:23 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/707165538' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-14T15:10:24.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:23 smithi092 ceph-mon[33966]: mgrmap e18: y(active, since 5m), standbys: x 2024-04-14T15:10:24.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:23 smithi092 ceph-mon[33966]: Active manager daemon y restarted 2024-04-14T15:10:24.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:23 smithi092 ceph-mon[33966]: Activating manager daemon y 2024-04-14T15:10:24.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:23 smithi092 ceph-mon[33966]: osdmap e79: 8 total, 8 up, 8 in 2024-04-14T15:10:24.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:23 smithi092 ceph-mon[33966]: mgrmap e19: y(active, starting, since 0.0129801s), standbys: x 2024-04-14T15:10:24.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:23 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T15:10:24.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:23 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:10:24.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:23 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:10:24.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:23 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-14T15:10:24.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:23 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-14T15:10:24.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:23 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T15:10:24.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:23 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T15:10:24.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:23 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T15:10:24.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:23 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T15:10:24.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:23 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T15:10:24.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:23 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T15:10:24.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:23 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T15:10:24.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:23 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T15:10:24.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:23 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-14T15:10:24.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:23 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-14T15:10:24.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:23 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-14T15:10:24.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:23 smithi092 ceph-mon[33966]: Manager daemon y is now available 2024-04-14T15:10:24.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:23 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:10:24.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:23 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:10:24.124 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:23 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:10:24.124 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:23 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T15:10:24.124 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:23 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T15:10:24.124 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:23 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/707165538' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-14T15:10:24.418 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-14T15:10:24.418 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-14T15:10:24.419 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-14T15:10:24.419 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-14T15:10:24.419 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-14T15:10:24.420 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-14T15:10:24.420 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-14T15:10:24.421 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-14T15:10:25.070 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:24 smithi090 ceph-mon[41670]: [14/Apr/2024:15:10:23] ENGINE Bus STARTING 2024-04-14T15:10:25.070 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:24 smithi090 ceph-mon[41670]: mgrmap e20: y(active, since 1.01393s), standbys: x 2024-04-14T15:10:25.070 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:24 smithi090 ceph-mon[30399]: [14/Apr/2024:15:10:23] ENGINE Bus STARTING 2024-04-14T15:10:25.070 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:24 smithi090 ceph-mon[30399]: mgrmap e20: y(active, since 1.01393s), standbys: x 2024-04-14T15:10:25.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:24 smithi092 ceph-mon[33966]: [14/Apr/2024:15:10:23] ENGINE Bus STARTING 2024-04-14T15:10:25.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:24 smithi092 ceph-mon[33966]: mgrmap e20: y(active, since 1.01393s), standbys: x 2024-04-14T15:10:25.122 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:10:24 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:10:25.465 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:10:25.465 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:10:25.466 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:10:25.842 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:25 smithi090 ceph-mon[30399]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:10:25.842 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:25 smithi090 ceph-mon[30399]: [14/Apr/2024:15:10:23] ENGINE Serving on https://172.21.15.90:7150 2024-04-14T15:10:25.842 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:25 smithi090 ceph-mon[30399]: [14/Apr/2024:15:10:24] ENGINE Serving on http://172.21.15.90:8765 2024-04-14T15:10:25.842 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:25 smithi090 ceph-mon[30399]: [14/Apr/2024:15:10:24] ENGINE Bus STARTED 2024-04-14T15:10:25.843 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:25 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:10:25.843 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:25 smithi090 ceph-mon[30399]: osdmap e80: 8 total, 8 up, 8 in 2024-04-14T15:10:25.843 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:25 smithi090 ceph-mon[41670]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:10:25.843 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:25 smithi090 ceph-mon[41670]: [14/Apr/2024:15:10:23] ENGINE Serving on https://172.21.15.90:7150 2024-04-14T15:10:25.843 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:25 smithi090 ceph-mon[41670]: [14/Apr/2024:15:10:24] ENGINE Serving on http://172.21.15.90:8765 2024-04-14T15:10:25.843 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:25 smithi090 ceph-mon[41670]: [14/Apr/2024:15:10:24] ENGINE Bus STARTED 2024-04-14T15:10:25.843 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:25 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:10:25.843 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:25 smithi090 ceph-mon[41670]: osdmap e80: 8 total, 8 up, 8 in 2024-04-14T15:10:26.024 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:25 smithi092 ceph-mon[33966]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:10:26.024 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:25 smithi092 ceph-mon[33966]: [14/Apr/2024:15:10:23] ENGINE Serving on https://172.21.15.90:7150 2024-04-14T15:10:26.024 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:25 smithi092 ceph-mon[33966]: [14/Apr/2024:15:10:24] ENGINE Serving on http://172.21.15.90:8765 2024-04-14T15:10:26.025 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:25 smithi092 ceph-mon[33966]: [14/Apr/2024:15:10:24] ENGINE Bus STARTED 2024-04-14T15:10:26.025 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:25 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:10:26.025 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:25 smithi092 ceph-mon[33966]: osdmap e80: 8 total, 8 up, 8 in 2024-04-14T15:10:26.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:26 smithi090 ceph-mon[30399]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:10:26.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:26 smithi090 ceph-mon[30399]: mgrmap e21: y(active, since 2s), standbys: x 2024-04-14T15:10:26.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:26 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:10:26.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:26 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:10:26.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:26 smithi090 ceph-mon[41670]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:10:26.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:26 smithi090 ceph-mon[41670]: mgrmap e21: y(active, since 2s), standbys: x 2024-04-14T15:10:26.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:26 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:10:26.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:26 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:10:27.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:26 smithi092 ceph-mon[33966]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:10:27.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:26 smithi092 ceph-mon[33966]: mgrmap e21: y(active, since 2s), standbys: x 2024-04-14T15:10:27.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:26 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:10:27.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:26 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:10:28.688 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:10:28.688 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:10:28.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:28 smithi090 ceph-mon[30399]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:10:28.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:28 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:10:28.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:28 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:10:28.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:28 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:10:28.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:28 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:10:28.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:28 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:10:28.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:28 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:10:28.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:28 smithi090 ceph-mon[30399]: Adjusting osd_memory_target on smithi092 to 3760M 2024-04-14T15:10:28.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:28 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:10:28.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:28 smithi090 ceph-mon[30399]: mgrmap e22: y(active, since 4s), standbys: x 2024-04-14T15:10:28.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:28 smithi090 ceph-mon[41670]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:10:28.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:28 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:10:28.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:28 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:10:28.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:28 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:10:28.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:28 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:10:28.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:28 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:10:28.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:28 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:10:28.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:28 smithi090 ceph-mon[41670]: Adjusting osd_memory_target on smithi092 to 3760M 2024-04-14T15:10:28.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:28 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:10:28.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:28 smithi090 ceph-mon[41670]: mgrmap e22: y(active, since 4s), standbys: x 2024-04-14T15:10:28.694 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:10:28.694 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:10:28.694 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:10:28.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:28 smithi092 ceph-mon[33966]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:10:28.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:28 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:10:28.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:28 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:10:28.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:28 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:10:28.873 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:28 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:10:28.873 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:28 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:10:28.873 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:28 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:10:28.873 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:28 smithi092 ceph-mon[33966]: Adjusting osd_memory_target on smithi092 to 3760M 2024-04-14T15:10:28.873 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:28 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:10:28.873 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:28 smithi092 ceph-mon[33966]: mgrmap e22: y(active, since 4s), standbys: x 2024-04-14T15:10:29.652 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:29 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:10:29.652 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:29 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:10:29.652 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:29 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:10:29.652 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:29 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:10:29.653 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:29 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:10:29.653 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:29 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:10:29.653 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:29 smithi090 ceph-mon[41670]: osdmap e81: 8 total, 8 up, 8 in 2024-04-14T15:10:29.653 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:29 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:10:29.653 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:29 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:10:29.653 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:29 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:10:29.653 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:29 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:10:29.653 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:29 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:10:29.653 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:29 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:10:29.653 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:29 smithi090 ceph-mon[30399]: osdmap e81: 8 total, 8 up, 8 in 2024-04-14T15:10:29.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.671+0000 7f14c43a8700 1 -- 172.21.15.90:0/3472774790 >> v1:172.21.15.90:6789/0 conn(0x7f14bc10b7c0 legacy=0x7f14bc10dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:29.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.671+0000 7f14c43a8700 1 -- 172.21.15.90:0/3472774790 shutdown_connections 2024-04-14T15:10:29.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.671+0000 7f14c43a8700 1 -- 172.21.15.90:0/3472774790 >> 172.21.15.90:0/3472774790 conn(0x7f14bc076a70 msgr2=0x7f14bc076e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:10:29.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.671+0000 7f14c43a8700 1 -- 172.21.15.90:0/3472774790 shutdown_connections 2024-04-14T15:10:29.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.672+0000 7f14c43a8700 1 -- 172.21.15.90:0/3472774790 wait complete. 2024-04-14T15:10:29.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.672+0000 7f14c43a8700 1 librados: starting msgr at 2024-04-14T15:10:29.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.672+0000 7f14c43a8700 1 librados: starting objecter 2024-04-14T15:10:29.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.672+0000 7f14c43a8700 1 Processor -- start 2024-04-14T15:10:29.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.672+0000 7f14c43a8700 1 -- start start 2024-04-14T15:10:29.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.672+0000 7f14c43a8700 1 librados: setting wanted keys 2024-04-14T15:10:29.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.672+0000 7f14c43a8700 1 librados: calling monclient init 2024-04-14T15:10:29.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.672+0000 7f14c43a8700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f14bc103e10 con 0x7f14bc10f360 2024-04-14T15:10:29.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.672+0000 7f14c43a8700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f14bc104020 con 0x7f14bc10b7c0 2024-04-14T15:10:29.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.672+0000 7f14c43a8700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f14bc1ac350 con 0x7f14bc1012c0 2024-04-14T15:10:29.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.672+0000 7f14c2144700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f14bc1012c0 0x7f14bc102240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:38468/0 (socket says 172.21.15.90:38468) 2024-04-14T15:10:29.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.672+0000 7f14c2144700 1 -- 172.21.15.90:0/3430665722 learned_addr learned my addr 172.21.15.90:0/3430665722 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:10:29.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.672+0000 7f14b37fe700 10 client.?.objecter ms_handle_connect 0x7f14bc10f360 2024-04-14T15:10:29.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.672+0000 7f14b37fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:10:29.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.673+0000 7f14b37fe700 10 client.?.objecter ms_handle_connect 0x7f14bc1012c0 2024-04-14T15:10:29.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.673+0000 7f14b37fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:10:29.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.673+0000 7f14b37fe700 1 -- 172.21.15.90:0/3430665722 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2433530780 0 0) 0x7f14bc103e10 con 0x7f14bc10f360 2024-04-14T15:10:29.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.673+0000 7f14b37fe700 1 -- 172.21.15.90:0/3430665722 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f14a80035b0 con 0x7f14bc10f360 2024-04-14T15:10:29.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.673+0000 7f14b37fe700 10 client.?.objecter ms_handle_connect 0x7f14bc10b7c0 2024-04-14T15:10:29.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.673+0000 7f14b37fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:10:29.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.673+0000 7f14b37fe700 1 -- 172.21.15.90:0/3430665722 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1216539588 0 0) 0x7f14bc1ac350 con 0x7f14bc1012c0 2024-04-14T15:10:29.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.673+0000 7f14b37fe700 1 -- 172.21.15.90:0/3430665722 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f14bc103e10 con 0x7f14bc1012c0 2024-04-14T15:10:29.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.673+0000 7f14b37fe700 1 -- 172.21.15.90:0/3430665722 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 571576614 0 0) 0x7f14bc104020 con 0x7f14bc10b7c0 2024-04-14T15:10:29.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.673+0000 7f14b37fe700 1 -- 172.21.15.90:0/3430665722 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f14bc1ac350 con 0x7f14bc10b7c0 2024-04-14T15:10:29.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.673+0000 7f14b37fe700 1 -- 172.21.15.90:0/3430665722 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2437559339 0 0) 0x7f14a80035b0 con 0x7f14bc10f360 2024-04-14T15:10:29.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.673+0000 7f14b37fe700 1 -- 172.21.15.90:0/3430665722 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f14bc104020 con 0x7f14bc10f360 2024-04-14T15:10:29.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.673+0000 7f14b37fe700 1 -- 172.21.15.90:0/3430665722 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f14b4001c30 con 0x7f14bc10f360 2024-04-14T15:10:29.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.673+0000 7f14b37fe700 1 -- 172.21.15.90:0/3430665722 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1094432622 0 0) 0x7f14bc103e10 con 0x7f14bc1012c0 2024-04-14T15:10:29.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.673+0000 7f14b37fe700 1 -- 172.21.15.90:0/3430665722 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f14a80035b0 con 0x7f14bc1012c0 2024-04-14T15:10:29.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.673+0000 7f14b37fe700 1 -- 172.21.15.90:0/3430665722 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f14ac0018b0 con 0x7f14bc1012c0 2024-04-14T15:10:29.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.673+0000 7f14b37fe700 1 -- 172.21.15.90:0/3430665722 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2935606089 0 0) 0x7f14bc1ac350 con 0x7f14bc10b7c0 2024-04-14T15:10:29.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.673+0000 7f14b37fe700 1 -- 172.21.15.90:0/3430665722 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f14bc103e10 con 0x7f14bc10b7c0 2024-04-14T15:10:29.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.673+0000 7f14b37fe700 1 -- 172.21.15.90:0/3430665722 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3605215357 0 0) 0x7f14bc104020 con 0x7f14bc10f360 2024-04-14T15:10:29.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.673+0000 7f14b37fe700 1 -- 172.21.15.90:0/3430665722 >> v1:172.21.15.90:6790/0 conn(0x7f14bc1012c0 legacy=0x7f14bc102240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:29.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.673+0000 7f14b37fe700 1 -- 172.21.15.90:0/3430665722 >> v1:172.21.15.92:6789/0 conn(0x7f14bc10b7c0 legacy=0x7f14bc102950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:29.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.673+0000 7f14b37fe700 1 -- 172.21.15.90:0/3430665722 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14bc1ad530 con 0x7f14bc10f360 2024-04-14T15:10:29.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.673+0000 7f14c43a8700 1 -- 172.21.15.90:0/3430665722 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f14bc1ac580 con 0x7f14bc10f360 2024-04-14T15:10:29.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.673+0000 7f14c43a8700 10 client.14658.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:10:29.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.673+0000 7f14c43a8700 1 -- 172.21.15.90:0/3430665722 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f14bc1aca40 con 0x7f14bc10f360 2024-04-14T15:10:29.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.673+0000 7f14c43a8700 1 librados: init done 2024-04-14T15:10:29.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.673+0000 7f14b37fe700 1 -- 172.21.15.90:0/3430665722 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f14b4003f40 con 0x7f14bc10f360 2024-04-14T15:10:29.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.673+0000 7f14b37fe700 1 -- 172.21.15.90:0/3430665722 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f14b4004e80 con 0x7f14bc10f360 2024-04-14T15:10:29.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.674+0000 7f14c43a8700 10 client.14658.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:10:29.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.674+0000 7f14c43a8700 15 client.14658.objecter _session_command_op_assign -1 1 2024-04-14T15:10:29.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.674+0000 7f14c43a8700 20 client.14658.objecter _get_session osd=-1 returning homeless 2024-04-14T15:10:29.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.674+0000 7f14c43a8700 10 client.14658.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:10:29.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.674+0000 7f14c43a8700 1 -- 172.21.15.90:0/3430665722 --> v1:172.21.15.90:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f14bc06ab20 con 0x7f14bc10f360 2024-04-14T15:10:29.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.675+0000 7f14b37fe700 1 -- 172.21.15.90:0/3430665722 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f14b4003aa0 con 0x7f14bc10f360 2024-04-14T15:10:29.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.675+0000 7f14b37fe700 1 -- 172.21.15.90:0/3430665722 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(81..81 src has 12..81) ==== 6288+0+0 (unknown 899498440 0 0) 0x7f14b40893c0 con 0x7f14bc10f360 2024-04-14T15:10:29.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.675+0000 7f14b37fe700 10 client.14658.objecter ms_dispatch 0x7f14bc001550 osd_map(81..81 src has 12..81) 2024-04-14T15:10:29.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.675+0000 7f14b37fe700 3 client.14658.objecter handle_osd_map got epochs [81,81] > 0 2024-04-14T15:10:29.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.675+0000 7f14b37fe700 3 client.14658.objecter handle_osd_map decoding full epoch 81 2024-04-14T15:10:29.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.675+0000 7f14b37fe700 10 client.14658.objecter checking command 1 2024-04-14T15:10:29.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.675+0000 7f14b37fe700 20 client.14658.objecter _get_session s=0x7f14a807b520 osd=7 3 2024-04-14T15:10:29.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.675+0000 7f14b37fe700 20 client.14658.objecter put_session s=0x7f14a807b520 osd=7 3 2024-04-14T15:10:29.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.675+0000 7f14b37fe700 15 client.14658.objecter _session_command_op_remove -1 1 2024-04-14T15:10:29.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.675+0000 7f14b37fe700 20 client.14658.objecter _get_session s=0x7f14a807b520 osd=7 3 2024-04-14T15:10:29.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.675+0000 7f14b37fe700 20 client.14658.objecter get_session s=0x7f14a807b520 osd=7 3 2024-04-14T15:10:29.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.675+0000 7f14b37fe700 15 client.14658.objecter _session_command_op_assign 7 1 2024-04-14T15:10:29.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.675+0000 7f14b37fe700 20 client.14658.objecter put_session s=0x7f14a807b520 osd=7 4 2024-04-14T15:10:29.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.675+0000 7f14b37fe700 10 client.14658.objecter _send_command 1 2024-04-14T15:10:29.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.675+0000 7f14b37fe700 1 -- 172.21.15.90:0/3430665722 --> v1:172.21.15.92:6812/1773706716 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f14a807f8e0 con 0x7f14a807bbd0 2024-04-14T15:10:29.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.675+0000 7f14b37fe700 20 client.14658.objecter dump_active .. 0 homeless 2024-04-14T15:10:29.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.675+0000 7f14b37fe700 1 -- 172.21.15.90:0/3430665722 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (unknown 1708901252 0 0) 0x7f14b40897e0 con 0x7f14bc10f360 2024-04-14T15:10:29.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.675+0000 7f14b37fe700 10 client.14658.objecter ms_handle_connect 0x7f14a8071360 2024-04-14T15:10:29.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.676+0000 7f14b37fe700 10 client.14658.objecter ms_handle_connect 0x7f14a807bbd0 2024-04-14T15:10:29.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.677+0000 7f14b37fe700 1 -- 172.21.15.90:0/3430665722 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f14a807f8e0 con 0x7f14a807bbd0 2024-04-14T15:10:29.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.677+0000 7f14b37fe700 10 client.14658.objecter ms_dispatch 0x7f14bc001550 command_reply(tid 1: 0 ) 2024-04-14T15:10:29.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.677+0000 7f14b37fe700 10 client.14658.objecter _finish_command 1 = system:0 2024-04-14T15:10:29.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.677+0000 7f14b37fe700 20 client.14658.objecter put_session s=0x7f14a807b520 osd=7 4 2024-04-14T15:10:29.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.677+0000 7f14b37fe700 15 client.14658.objecter _session_command_op_remove 7 1 2024-04-14T15:10:29.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.693+0000 7f14c43a8700 10 client.14658.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-14T15:10:29.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.693+0000 7f14c43a8700 15 client.14658.objecter _session_command_op_assign -1 2 2024-04-14T15:10:29.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.693+0000 7f14c43a8700 20 client.14658.objecter _get_session s=0x7f14a807b520 osd=7 3 2024-04-14T15:10:29.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.693+0000 7f14c43a8700 20 client.14658.objecter put_session s=0x7f14a807b520 osd=7 3 2024-04-14T15:10:29.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.693+0000 7f14c43a8700 20 client.14658.objecter _get_session s=0x7f14a807b520 osd=7 3 2024-04-14T15:10:29.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.693+0000 7f14c43a8700 15 client.14658.objecter _session_command_op_remove -1 2 2024-04-14T15:10:29.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.693+0000 7f14c43a8700 20 client.14658.objecter get_session s=0x7f14a807b520 osd=7 3 2024-04-14T15:10:29.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.693+0000 7f14c43a8700 15 client.14658.objecter _session_command_op_assign 7 2 2024-04-14T15:10:29.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.693+0000 7f14c43a8700 20 client.14658.objecter put_session s=0x7f14a807b520 osd=7 4 2024-04-14T15:10:29.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.693+0000 7f14c43a8700 10 client.14658.objecter _send_command 2 2024-04-14T15:10:29.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.693+0000 7f14c43a8700 1 -- 172.21.15.90:0/3430665722 --> v1:172.21.15.92:6812/1773706716 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f14bc0693d0 con 0x7f14a807bbd0 2024-04-14T15:10:29.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.693+0000 7f14b37fe700 1 -- 172.21.15.90:0/3430665722 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (unknown 0 0 1117776133) 0x7f14bc0693d0 con 0x7f14a807bbd0 2024-04-14T15:10:29.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.693+0000 7f14b37fe700 10 client.14658.objecter ms_dispatch 0x7f14bc001550 command_reply(tid 2: 0 ) 2024-04-14T15:10:29.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.693+0000 7f14b37fe700 10 client.14658.objecter _finish_command 2 = system:0 2024-04-14T15:10:29.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.693+0000 7f14b37fe700 20 client.14658.objecter put_session s=0x7f14a807b520 osd=7 4 2024-04-14T15:10:29.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.693+0000 7f14b37fe700 15 client.14658.objecter _session_command_op_remove 7 2 2024-04-14T15:10:29.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.694+0000 7f14c43a8700 10 librados: watch_flush enter 2024-04-14T15:10:29.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.694+0000 7f14c43a8700 10 librados: watch_flush exit 2024-04-14T15:10:29.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.694+0000 7f14c43a8700 10 client.14658.objecter close_session for osd.7 2024-04-14T15:10:29.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.694+0000 7f14c43a8700 1 -- 172.21.15.90:0/3430665722 >> v1:172.21.15.92:6812/1773706716 conn(0x7f14a807bbd0 legacy=0x7f14a807e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:29.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.694+0000 7f14c43a8700 20 client.14658.objecter put_session s=0x7f14a807b520 osd=7 1 2024-04-14T15:10:29.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.694+0000 7f14c43a8700 20 client.14658.objecter shutdown clearing up homeless session... 2024-04-14T15:10:29.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.694+0000 7f14c43a8700 10 client.14658.objecter successfully canceled tick 2024-04-14T15:10:29.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.694+0000 7f14c43a8700 1 -- 172.21.15.90:0/3430665722 >> v1:172.21.15.90:6800/420935427 conn(0x7f14a8071360 legacy=0x7f14a8073820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:29.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.694+0000 7f14c43a8700 1 -- 172.21.15.90:0/3430665722 >> v1:172.21.15.90:6789/0 conn(0x7f14bc10f360 legacy=0x7f14bc1a8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:29.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.694+0000 7f14c2945700 1 -- 172.21.15.90:0/3430665722 reap_dead start 2024-04-14T15:10:29.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.694+0000 7f14c43a8700 1 -- 172.21.15.90:0/3430665722 shutdown_connections 2024-04-14T15:10:29.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.694+0000 7f14c43a8700 1 -- 172.21.15.90:0/3430665722 >> 172.21.15.90:0/3430665722 conn(0x7f14bc076a70 msgr2=0x7f14bc1117a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:10:29.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.694+0000 7f14c43a8700 1 -- 172.21.15.90:0/3430665722 shutdown_connections 2024-04-14T15:10:29.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.694+0000 7f14c43a8700 1 -- 172.21.15.90:0/3430665722 wait complete. 2024-04-14T15:10:29.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:29.694+0000 7f14c43a8700 1 librados: shutdown 2024-04-14T15:10:29.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:29 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:10:29.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:29 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:10:29.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:29 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:10:29.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:29 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:10:29.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:29 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:10:29.873 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:29 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:10:29.873 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:29 smithi092 ceph-mon[33966]: osdmap e81: 8 total, 8 up, 8 in 2024-04-14T15:10:30.376 INFO:teuthology.orchestra.run.smithi090.stdout:249108103183 2024-04-14T15:10:30.376 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-14T15:10:31.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:30 smithi092 ceph-mon[33966]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 24 op/s 2024-04-14T15:10:31.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:30 smithi092 ceph-mon[33966]: osdmap e82: 8 total, 8 up, 8 in 2024-04-14T15:10:31.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:30 smithi090 ceph-mon[41670]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 24 op/s 2024-04-14T15:10:31.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:30 smithi090 ceph-mon[41670]: osdmap e82: 8 total, 8 up, 8 in 2024-04-14T15:10:31.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:30 smithi090 ceph-mon[30399]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 24 op/s 2024-04-14T15:10:31.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:30 smithi090 ceph-mon[30399]: osdmap e82: 8 total, 8 up, 8 in 2024-04-14T15:10:32.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.764+0000 7f98efeac700 1 -- 172.21.15.90:0/584248684 >> v1:172.21.15.90:6789/0 conn(0x7f98e810f2f0 legacy=0x7f98e81117b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:32.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.765+0000 7f98efeac700 1 -- 172.21.15.90:0/584248684 shutdown_connections 2024-04-14T15:10:32.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.765+0000 7f98efeac700 1 -- 172.21.15.90:0/584248684 >> 172.21.15.90:0/584248684 conn(0x7f98e80fe3c0 msgr2=0x7f98e81007e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:10:32.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.765+0000 7f98efeac700 1 -- 172.21.15.90:0/584248684 shutdown_connections 2024-04-14T15:10:32.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.765+0000 7f98efeac700 1 -- 172.21.15.90:0/584248684 wait complete. 2024-04-14T15:10:32.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.765+0000 7f98efeac700 1 librados: starting msgr at 2024-04-14T15:10:32.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.765+0000 7f98efeac700 1 librados: starting objecter 2024-04-14T15:10:32.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.765+0000 7f98efeac700 1 Processor -- start 2024-04-14T15:10:32.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.765+0000 7f98efeac700 1 -- start start 2024-04-14T15:10:32.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.765+0000 7f98efeac700 1 librados: setting wanted keys 2024-04-14T15:10:32.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.765+0000 7f98efeac700 1 librados: calling monclient init 2024-04-14T15:10:32.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.765+0000 7f98efeac700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f98e8108070 con 0x7f98e810b720 2024-04-14T15:10:32.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.765+0000 7f98efeac700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f98e81a8f00 con 0x7f98e810f2f0 2024-04-14T15:10:32.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.765+0000 7f98efeac700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f98e81aa0e0 con 0x7f98e8108870 2024-04-14T15:10:32.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.766+0000 7f98ed447700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f98e810b720 0x7f98e81055d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:53440/0 (socket says 172.21.15.90:53440) 2024-04-14T15:10:32.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.766+0000 7f98ed447700 1 -- 172.21.15.90:0/1379011833 learned_addr learned my addr 172.21.15.90:0/1379011833 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:10:32.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.766+0000 7f98edc48700 1 --1- 172.21.15.90:0/1379011833 >> v1:172.21.15.90:6790/0 conn(0x7f98e8108870 0x7f98e8104ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:33972/0 (socket says 172.21.15.90:33972) 2024-04-14T15:10:32.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.766+0000 7f98deffd700 10 client.?.objecter ms_handle_connect 0x7f98e810b720 2024-04-14T15:10:32.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.766+0000 7f98deffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:10:32.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.766+0000 7f98deffd700 10 client.?.objecter ms_handle_connect 0x7f98e8108870 2024-04-14T15:10:32.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.766+0000 7f98deffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:10:32.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.766+0000 7f98deffd700 10 client.?.objecter ms_handle_connect 0x7f98e810f2f0 2024-04-14T15:10:32.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.766+0000 7f98deffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:10:32.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.766+0000 7f98deffd700 1 -- 172.21.15.90:0/1379011833 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1251123583 0 0) 0x7f98e8108070 con 0x7f98e810b720 2024-04-14T15:10:32.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.766+0000 7f98deffd700 1 -- 172.21.15.90:0/1379011833 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f98d40035b0 con 0x7f98e810b720 2024-04-14T15:10:32.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.766+0000 7f98deffd700 1 -- 172.21.15.90:0/1379011833 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2750285278 0 0) 0x7f98e81a8f00 con 0x7f98e810f2f0 2024-04-14T15:10:32.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.766+0000 7f98deffd700 1 -- 172.21.15.90:0/1379011833 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f98e8108070 con 0x7f98e810f2f0 2024-04-14T15:10:32.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.766+0000 7f98deffd700 1 -- 172.21.15.90:0/1379011833 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2795139783 0 0) 0x7f98e81aa0e0 con 0x7f98e8108870 2024-04-14T15:10:32.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.766+0000 7f98deffd700 1 -- 172.21.15.90:0/1379011833 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f98e81a8f00 con 0x7f98e8108870 2024-04-14T15:10:32.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.766+0000 7f98deffd700 1 -- 172.21.15.90:0/1379011833 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 755346346 0 0) 0x7f98d40035b0 con 0x7f98e810b720 2024-04-14T15:10:32.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.766+0000 7f98deffd700 1 -- 172.21.15.90:0/1379011833 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f98e81aa0e0 con 0x7f98e810b720 2024-04-14T15:10:32.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.766+0000 7f98deffd700 1 -- 172.21.15.90:0/1379011833 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f98d80018b0 con 0x7f98e810b720 2024-04-14T15:10:32.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.766+0000 7f98deffd700 1 -- 172.21.15.90:0/1379011833 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3142937153 0 0) 0x7f98e8108070 con 0x7f98e810f2f0 2024-04-14T15:10:32.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.766+0000 7f98deffd700 1 -- 172.21.15.90:0/1379011833 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f98d40035b0 con 0x7f98e810f2f0 2024-04-14T15:10:32.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.766+0000 7f98deffd700 1 -- 172.21.15.90:0/1379011833 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f98e00017b0 con 0x7f98e810f2f0 2024-04-14T15:10:32.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.766+0000 7f98deffd700 1 -- 172.21.15.90:0/1379011833 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 220113918 0 0) 0x7f98e81a8f00 con 0x7f98e8108870 2024-04-14T15:10:32.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.766+0000 7f98deffd700 1 -- 172.21.15.90:0/1379011833 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f98e8108070 con 0x7f98e8108870 2024-04-14T15:10:32.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.766+0000 7f98deffd700 1 -- 172.21.15.90:0/1379011833 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f98e40017f0 con 0x7f98e8108870 2024-04-14T15:10:32.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.766+0000 7f98deffd700 1 -- 172.21.15.90:0/1379011833 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2088356023 0 0) 0x7f98e81aa0e0 con 0x7f98e810b720 2024-04-14T15:10:32.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.766+0000 7f98deffd700 1 -- 172.21.15.90:0/1379011833 >> v1:172.21.15.90:6790/0 conn(0x7f98e8108870 legacy=0x7f98e8104ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:32.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.766+0000 7f98deffd700 1 -- 172.21.15.90:0/1379011833 >> v1:172.21.15.92:6789/0 conn(0x7f98e810f2f0 legacy=0x7f98e81a67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:32.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.766+0000 7f98deffd700 1 -- 172.21.15.90:0/1379011833 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98e81ab2c0 con 0x7f98e810b720 2024-04-14T15:10:32.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.766+0000 7f98efeac700 1 -- 172.21.15.90:0/1379011833 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f98e81aa310 con 0x7f98e810b720 2024-04-14T15:10:32.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.766+0000 7f98efeac700 10 client.14664.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:10:32.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.766+0000 7f98efeac700 1 -- 172.21.15.90:0/1379011833 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f98e81aa7d0 con 0x7f98e810b720 2024-04-14T15:10:32.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.767+0000 7f98efeac700 1 librados: init done 2024-04-14T15:10:32.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.767+0000 7f98deffd700 1 -- 172.21.15.90:0/1379011833 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f98d8003b20 con 0x7f98e810b720 2024-04-14T15:10:32.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.767+0000 7f98deffd700 1 -- 172.21.15.90:0/1379011833 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f98d8005c00 con 0x7f98e810b720 2024-04-14T15:10:32.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.767+0000 7f98efeac700 10 client.14664.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:10:32.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.767+0000 7f98efeac700 15 client.14664.objecter _session_command_op_assign -1 1 2024-04-14T15:10:32.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.767+0000 7f98efeac700 20 client.14664.objecter _get_session osd=-1 returning homeless 2024-04-14T15:10:32.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.767+0000 7f98efeac700 10 client.14664.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:10:32.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.767+0000 7f98efeac700 1 -- 172.21.15.90:0/1379011833 --> v1:172.21.15.90:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f98cc002920 con 0x7f98e810b720 2024-04-14T15:10:32.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.768+0000 7f98deffd700 1 -- 172.21.15.90:0/1379011833 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f98d801c8b0 con 0x7f98e810b720 2024-04-14T15:10:32.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.768+0000 7f98deffd700 1 -- 172.21.15.90:0/1379011833 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(82..82 src has 12..82) ==== 6288+0+0 (unknown 460659372 0 0) 0x7f98d808a170 con 0x7f98e810b720 2024-04-14T15:10:32.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.768+0000 7f98deffd700 10 client.14664.objecter ms_dispatch 0x7f98e8001550 osd_map(82..82 src has 12..82) 2024-04-14T15:10:32.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.768+0000 7f98deffd700 3 client.14664.objecter handle_osd_map got epochs [82,82] > 0 2024-04-14T15:10:32.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.768+0000 7f98deffd700 3 client.14664.objecter handle_osd_map decoding full epoch 82 2024-04-14T15:10:32.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.768+0000 7f98deffd700 10 client.14664.objecter checking command 1 2024-04-14T15:10:32.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.768+0000 7f98deffd700 20 client.14664.objecter _get_session s=0x7f98d407b570 osd=1 3 2024-04-14T15:10:32.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.768+0000 7f98deffd700 20 client.14664.objecter put_session s=0x7f98d407b570 osd=1 3 2024-04-14T15:10:32.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.768+0000 7f98deffd700 15 client.14664.objecter _session_command_op_remove -1 1 2024-04-14T15:10:32.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.768+0000 7f98deffd700 20 client.14664.objecter _get_session s=0x7f98d407b570 osd=1 3 2024-04-14T15:10:32.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.768+0000 7f98deffd700 20 client.14664.objecter get_session s=0x7f98d407b570 osd=1 3 2024-04-14T15:10:32.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.768+0000 7f98deffd700 15 client.14664.objecter _session_command_op_assign 1 1 2024-04-14T15:10:32.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.769+0000 7f98deffd700 20 client.14664.objecter put_session s=0x7f98d407b570 osd=1 4 2024-04-14T15:10:32.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.769+0000 7f98deffd700 10 client.14664.objecter _send_command 1 2024-04-14T15:10:32.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.769+0000 7f98deffd700 1 -- 172.21.15.90:0/1379011833 --> v1:172.21.15.90:6805/4170361699 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f98d407f930 con 0x7f98d407bc20 2024-04-14T15:10:32.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.769+0000 7f98deffd700 20 client.14664.objecter dump_active .. 0 homeless 2024-04-14T15:10:32.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.769+0000 7f98deffd700 1 -- 172.21.15.90:0/1379011833 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (unknown 1982047095 0 0) 0x7f98d808a5c0 con 0x7f98e810b720 2024-04-14T15:10:32.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.769+0000 7f98deffd700 10 client.14664.objecter ms_handle_connect 0x7f98d40713c0 2024-04-14T15:10:32.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.769+0000 7f98deffd700 10 client.14664.objecter ms_handle_connect 0x7f98d407bc20 2024-04-14T15:10:32.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.769+0000 7f98deffd700 1 -- 172.21.15.90:0/1379011833 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f98d407f930 con 0x7f98d407bc20 2024-04-14T15:10:32.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.769+0000 7f98deffd700 10 client.14664.objecter ms_dispatch 0x7f98e8001550 command_reply(tid 1: 0 ) 2024-04-14T15:10:32.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.769+0000 7f98deffd700 10 client.14664.objecter _finish_command 1 = system:0 2024-04-14T15:10:32.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.769+0000 7f98deffd700 20 client.14664.objecter put_session s=0x7f98d407b570 osd=1 4 2024-04-14T15:10:32.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.769+0000 7f98deffd700 15 client.14664.objecter _session_command_op_remove 1 1 2024-04-14T15:10:32.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.789+0000 7f98efeac700 10 client.14664.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-14T15:10:32.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.789+0000 7f98efeac700 15 client.14664.objecter _session_command_op_assign -1 2 2024-04-14T15:10:32.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.789+0000 7f98efeac700 20 client.14664.objecter _get_session s=0x7f98d407b570 osd=1 3 2024-04-14T15:10:32.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.789+0000 7f98efeac700 20 client.14664.objecter put_session s=0x7f98d407b570 osd=1 3 2024-04-14T15:10:32.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.789+0000 7f98efeac700 20 client.14664.objecter _get_session s=0x7f98d407b570 osd=1 3 2024-04-14T15:10:32.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.789+0000 7f98efeac700 15 client.14664.objecter _session_command_op_remove -1 2 2024-04-14T15:10:32.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.789+0000 7f98efeac700 20 client.14664.objecter get_session s=0x7f98d407b570 osd=1 3 2024-04-14T15:10:32.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.789+0000 7f98efeac700 15 client.14664.objecter _session_command_op_assign 1 2 2024-04-14T15:10:32.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.789+0000 7f98efeac700 20 client.14664.objecter put_session s=0x7f98d407b570 osd=1 4 2024-04-14T15:10:32.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.789+0000 7f98efeac700 10 client.14664.objecter _send_command 2 2024-04-14T15:10:32.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.789+0000 7f98efeac700 1 -- 172.21.15.90:0/1379011833 --> v1:172.21.15.90:6805/4170361699 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f98cc0011f0 con 0x7f98d407bc20 2024-04-14T15:10:32.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.789+0000 7f98deffd700 1 -- 172.21.15.90:0/1379011833 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 4200525082) 0x7f98cc0011f0 con 0x7f98d407bc20 2024-04-14T15:10:32.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.789+0000 7f98deffd700 10 client.14664.objecter ms_dispatch 0x7f98e8001550 command_reply(tid 2: 0 ) 2024-04-14T15:10:32.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.789+0000 7f98deffd700 10 client.14664.objecter _finish_command 2 = system:0 2024-04-14T15:10:32.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.789+0000 7f98deffd700 20 client.14664.objecter put_session s=0x7f98d407b570 osd=1 4 2024-04-14T15:10:32.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.789+0000 7f98deffd700 15 client.14664.objecter _session_command_op_remove 1 2 2024-04-14T15:10:32.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.789+0000 7f98efeac700 10 librados: watch_flush enter 2024-04-14T15:10:32.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.789+0000 7f98efeac700 10 librados: watch_flush exit 2024-04-14T15:10:32.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.790+0000 7f98efeac700 10 client.14664.objecter close_session for osd.1 2024-04-14T15:10:32.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.790+0000 7f98efeac700 1 -- 172.21.15.90:0/1379011833 >> v1:172.21.15.90:6805/4170361699 conn(0x7f98d407bc20 legacy=0x7f98d407e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:32.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.790+0000 7f98efeac700 20 client.14664.objecter put_session s=0x7f98d407b570 osd=1 1 2024-04-14T15:10:32.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.790+0000 7f98efeac700 20 client.14664.objecter shutdown clearing up homeless session... 2024-04-14T15:10:32.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.790+0000 7f98efeac700 10 client.14664.objecter successfully canceled tick 2024-04-14T15:10:32.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.790+0000 7f98efeac700 1 -- 172.21.15.90:0/1379011833 >> v1:172.21.15.90:6800/420935427 conn(0x7f98d40713c0 legacy=0x7f98d4073880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:32.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.790+0000 7f98efeac700 1 -- 172.21.15.90:0/1379011833 >> v1:172.21.15.90:6789/0 conn(0x7f98e810b720 legacy=0x7f98e81055d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:32.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.790+0000 7f98ee449700 1 -- 172.21.15.90:0/1379011833 reap_dead start 2024-04-14T15:10:32.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.790+0000 7f98efeac700 1 -- 172.21.15.90:0/1379011833 shutdown_connections 2024-04-14T15:10:32.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.790+0000 7f98efeac700 1 -- 172.21.15.90:0/1379011833 >> 172.21.15.90:0/1379011833 conn(0x7f98e80fe3c0 msgr2=0x7f98e8112810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:10:32.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.790+0000 7f98efeac700 1 -- 172.21.15.90:0/1379011833 shutdown_connections 2024-04-14T15:10:32.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.790+0000 7f98efeac700 1 -- 172.21.15.90:0/1379011833 wait complete. 2024-04-14T15:10:32.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:32.790+0000 7f98efeac700 1 librados: shutdown 2024-04-14T15:10:32.941 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:32 smithi090 ceph-mon[30399]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 20 op/s 2024-04-14T15:10:32.941 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:32 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:10:32.941 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:32 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:10:32.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:32 smithi090 ceph-mon[41670]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 20 op/s 2024-04-14T15:10:32.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:32 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:10:32.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:32 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:10:33.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:32 smithi092 ceph-mon[33966]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 20 op/s 2024-04-14T15:10:33.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:32 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:10:33.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:32 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:10:33.656 INFO:teuthology.orchestra.run.smithi090.stdout:55834574892 2024-04-14T15:10:33.656 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-14T15:10:33.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.870+0000 7f5dc6442700 1 -- 172.21.15.90:0/3752256327 >> v1:172.21.15.90:6790/0 conn(0x7f5dc0108880 legacy=0x7f5dc0108c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:33.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.870+0000 7f5dc6442700 1 -- 172.21.15.90:0/3752256327 shutdown_connections 2024-04-14T15:10:33.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.870+0000 7f5dc6442700 1 -- 172.21.15.90:0/3752256327 >> 172.21.15.90:0/3752256327 conn(0x7f5dc00fe3d0 msgr2=0x7f5dc01007f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:10:33.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.870+0000 7f5dc6442700 1 -- 172.21.15.90:0/3752256327 shutdown_connections 2024-04-14T15:10:33.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.870+0000 7f5dc6442700 1 -- 172.21.15.90:0/3752256327 wait complete. 2024-04-14T15:10:33.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.871+0000 7f5dc6442700 1 librados: starting msgr at 2024-04-14T15:10:33.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.871+0000 7f5dc6442700 1 librados: starting objecter 2024-04-14T15:10:33.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.871+0000 7f5dc6442700 1 Processor -- start 2024-04-14T15:10:33.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.871+0000 7f5dc6442700 1 -- start start 2024-04-14T15:10:33.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.871+0000 7f5dc6442700 1 librados: setting wanted keys 2024-04-14T15:10:33.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.871+0000 7f5dc6442700 1 librados: calling monclient init 2024-04-14T15:10:33.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.871+0000 7f5dc6442700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5dc01a5840 con 0x7f5dc0108880 2024-04-14T15:10:33.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.871+0000 7f5dc6442700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5dc01a6a60 con 0x7f5dc010b730 2024-04-14T15:10:33.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.871+0000 7f5dc6442700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5dc01a7ca0 con 0x7f5dc010f300 2024-04-14T15:10:33.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.872+0000 7f5dbffff700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f5dc0108880 0x7f5dc010eb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:53462/0 (socket says 172.21.15.90:53462) 2024-04-14T15:10:33.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.872+0000 7f5dbffff700 1 -- 172.21.15.90:0/1308952299 learned_addr learned my addr 172.21.15.90:0/1308952299 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:10:33.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.872+0000 7f5dbd7fa700 10 client.?.objecter ms_handle_connect 0x7f5dc0108880 2024-04-14T15:10:33.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.872+0000 7f5dbd7fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:10:33.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.872+0000 7f5dbd7fa700 10 client.?.objecter ms_handle_connect 0x7f5dc010b730 2024-04-14T15:10:33.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.872+0000 7f5dbd7fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:10:33.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.872+0000 7f5dbd7fa700 10 client.?.objecter ms_handle_connect 0x7f5dc010f300 2024-04-14T15:10:33.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.872+0000 7f5dbd7fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:10:33.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.872+0000 7f5dbd7fa700 1 -- 172.21.15.90:0/1308952299 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4130156890 0 0) 0x7f5dc01a6a60 con 0x7f5dc010b730 2024-04-14T15:10:33.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.872+0000 7f5dbd7fa700 1 -- 172.21.15.90:0/1308952299 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5da80039c0 con 0x7f5dc010b730 2024-04-14T15:10:33.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.872+0000 7f5dbd7fa700 1 -- 172.21.15.90:0/1308952299 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1696813651 0 0) 0x7f5dc01a5840 con 0x7f5dc0108880 2024-04-14T15:10:33.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.872+0000 7f5dbd7fa700 1 -- 172.21.15.90:0/1308952299 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5dc01a6a60 con 0x7f5dc0108880 2024-04-14T15:10:33.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.872+0000 7f5dbd7fa700 1 -- 172.21.15.90:0/1308952299 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3255830016 0 0) 0x7f5dc01a7ca0 con 0x7f5dc010f300 2024-04-14T15:10:33.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.872+0000 7f5dbd7fa700 1 -- 172.21.15.90:0/1308952299 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5dc01a5840 con 0x7f5dc010f300 2024-04-14T15:10:33.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.872+0000 7f5dbd7fa700 1 -- 172.21.15.90:0/1308952299 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1355137197 0 0) 0x7f5da80039c0 con 0x7f5dc010b730 2024-04-14T15:10:33.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.872+0000 7f5dbd7fa700 1 -- 172.21.15.90:0/1308952299 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5dc01a7ca0 con 0x7f5dc010b730 2024-04-14T15:10:33.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.872+0000 7f5dbd7fa700 1 -- 172.21.15.90:0/1308952299 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f5db4001a30 con 0x7f5dc010b730 2024-04-14T15:10:33.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.872+0000 7f5dbd7fa700 1 -- 172.21.15.90:0/1308952299 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3222671380 0 0) 0x7f5dc01a6a60 con 0x7f5dc0108880 2024-04-14T15:10:33.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.872+0000 7f5dbd7fa700 1 -- 172.21.15.90:0/1308952299 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5da80039c0 con 0x7f5dc0108880 2024-04-14T15:10:33.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.872+0000 7f5dbd7fa700 1 -- 172.21.15.90:0/1308952299 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f5db00017b0 con 0x7f5dc0108880 2024-04-14T15:10:33.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.872+0000 7f5dbd7fa700 1 -- 172.21.15.90:0/1308952299 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4189996501 0 0) 0x7f5dc01a5840 con 0x7f5dc010f300 2024-04-14T15:10:33.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.872+0000 7f5dbd7fa700 1 -- 172.21.15.90:0/1308952299 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5dc01a6a60 con 0x7f5dc010f300 2024-04-14T15:10:33.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.872+0000 7f5dbd7fa700 1 -- 172.21.15.90:0/1308952299 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f5db8001bf0 con 0x7f5dc010f300 2024-04-14T15:10:33.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.872+0000 7f5dbd7fa700 1 -- 172.21.15.90:0/1308952299 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4214318217 0 0) 0x7f5dc01a7ca0 con 0x7f5dc010b730 2024-04-14T15:10:33.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.872+0000 7f5dbd7fa700 1 -- 172.21.15.90:0/1308952299 >> v1:172.21.15.90:6790/0 conn(0x7f5dc010f300 legacy=0x7f5dc01a3f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:33.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.872+0000 7f5dbd7fa700 1 -- 172.21.15.90:0/1308952299 >> v1:172.21.15.90:6789/0 conn(0x7f5dc0108880 legacy=0x7f5dc010eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:33.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.872+0000 7f5dbd7fa700 1 -- 172.21.15.90:0/1308952299 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dc01a7ed0 con 0x7f5dc010b730 2024-04-14T15:10:33.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.872+0000 7f5dc6442700 1 -- 172.21.15.90:0/1308952299 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5dc01a6c30 con 0x7f5dc010b730 2024-04-14T15:10:33.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.873+0000 7f5dbd7fa700 1 -- 172.21.15.90:0/1308952299 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f5db4001ce0 con 0x7f5dc010b730 2024-04-14T15:10:33.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.873+0000 7f5dc6442700 10 client.24562.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:10:33.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.873+0000 7f5dbd7fa700 1 -- 172.21.15.90:0/1308952299 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f5db4004a40 con 0x7f5dc010b730 2024-04-14T15:10:33.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.873+0000 7f5dc6442700 1 -- 172.21.15.90:0/1308952299 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f5dc01a70f0 con 0x7f5dc010b730 2024-04-14T15:10:33.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.873+0000 7f5dc6442700 1 librados: init done 2024-04-14T15:10:33.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.873+0000 7f5dc6442700 10 client.24562.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:10:33.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.873+0000 7f5dc6442700 15 client.24562.objecter _session_command_op_assign -1 1 2024-04-14T15:10:33.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.873+0000 7f5dc6442700 20 client.24562.objecter _get_session osd=-1 returning homeless 2024-04-14T15:10:33.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.873+0000 7f5dc6442700 10 client.24562.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:10:33.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.873+0000 7f5dc6442700 1 -- 172.21.15.90:0/1308952299 --> v1:172.21.15.92:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f5dc01a81c0 con 0x7f5dc010b730 2024-04-14T15:10:33.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.874+0000 7f5dbd7fa700 1 -- 172.21.15.90:0/1308952299 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f5db401b760 con 0x7f5dc010b730 2024-04-14T15:10:33.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.874+0000 7f5dbd7fa700 1 -- 172.21.15.90:0/1308952299 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(83..83 src has 12..83) ==== 6288+0+0 (unknown 4253614559 0 0) 0x7f5db40890e0 con 0x7f5dc010b730 2024-04-14T15:10:33.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.874+0000 7f5dbd7fa700 10 client.24562.objecter ms_dispatch 0x7f5dc0001550 osd_map(83..83 src has 12..83) 2024-04-14T15:10:33.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.874+0000 7f5dbd7fa700 3 client.24562.objecter handle_osd_map got epochs [83,83] > 0 2024-04-14T15:10:33.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.874+0000 7f5dbd7fa700 3 client.24562.objecter handle_osd_map decoding full epoch 83 2024-04-14T15:10:33.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.875+0000 7f5dbd7fa700 10 client.24562.objecter checking command 1 2024-04-14T15:10:33.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.875+0000 7f5dbd7fa700 20 client.24562.objecter _get_session s=0x7f5da807b450 osd=3 3 2024-04-14T15:10:33.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.875+0000 7f5dbd7fa700 20 client.24562.objecter put_session s=0x7f5da807b450 osd=3 3 2024-04-14T15:10:33.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.875+0000 7f5dbd7fa700 15 client.24562.objecter _session_command_op_remove -1 1 2024-04-14T15:10:33.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.875+0000 7f5dbd7fa700 20 client.24562.objecter _get_session s=0x7f5da807b450 osd=3 3 2024-04-14T15:10:33.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.875+0000 7f5dbd7fa700 20 client.24562.objecter get_session s=0x7f5da807b450 osd=3 3 2024-04-14T15:10:33.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.875+0000 7f5dbd7fa700 15 client.24562.objecter _session_command_op_assign 3 1 2024-04-14T15:10:33.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.875+0000 7f5dbd7fa700 20 client.24562.objecter put_session s=0x7f5da807b450 osd=3 4 2024-04-14T15:10:33.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.875+0000 7f5dbd7fa700 10 client.24562.objecter _send_command 1 2024-04-14T15:10:33.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.875+0000 7f5dbd7fa700 1 -- 172.21.15.90:0/1308952299 --> v1:172.21.15.90:6813/2270166774 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5db401bf30 con 0x7f5da807bb00 2024-04-14T15:10:33.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.875+0000 7f5dbd7fa700 20 client.24562.objecter dump_active .. 0 homeless 2024-04-14T15:10:33.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.875+0000 7f5dbd7fa700 1 -- 172.21.15.90:0/1308952299 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_get_version_reply(handle=1 version=83) ==== 24+0+0 (unknown 2217717385 0 0) 0x7f5db4089470 con 0x7f5dc010b730 2024-04-14T15:10:33.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.875+0000 7f5dbd7fa700 10 client.24562.objecter ms_handle_connect 0x7f5da80713a0 2024-04-14T15:10:33.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.875+0000 7f5dbd7fa700 10 client.24562.objecter ms_handle_connect 0x7f5da807bb00 2024-04-14T15:10:33.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.876+0000 7f5dbd7fa700 1 -- 172.21.15.90:0/1308952299 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f5db401bf30 con 0x7f5da807bb00 2024-04-14T15:10:33.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.876+0000 7f5dbd7fa700 10 client.24562.objecter ms_dispatch 0x7f5dc0001550 command_reply(tid 1: 0 ) 2024-04-14T15:10:33.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.876+0000 7f5dbd7fa700 10 client.24562.objecter _finish_command 1 = system:0 2024-04-14T15:10:33.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.876+0000 7f5dbd7fa700 20 client.24562.objecter put_session s=0x7f5da807b450 osd=3 4 2024-04-14T15:10:33.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.876+0000 7f5dbd7fa700 15 client.24562.objecter _session_command_op_remove 3 1 2024-04-14T15:10:33.900 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:33 smithi090 ceph-mon[30399]: osdmap e83: 8 total, 8 up, 8 in 2024-04-14T15:10:33.900 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:33 smithi090 ceph-mon[41670]: osdmap e83: 8 total, 8 up, 8 in 2024-04-14T15:10:33.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.899+0000 7f5dc6442700 10 client.24562.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-14T15:10:33.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.899+0000 7f5dc6442700 15 client.24562.objecter _session_command_op_assign -1 2 2024-04-14T15:10:33.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.899+0000 7f5dc6442700 20 client.24562.objecter _get_session s=0x7f5da807b450 osd=3 3 2024-04-14T15:10:33.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.899+0000 7f5dc6442700 20 client.24562.objecter put_session s=0x7f5da807b450 osd=3 3 2024-04-14T15:10:33.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.899+0000 7f5dc6442700 20 client.24562.objecter _get_session s=0x7f5da807b450 osd=3 3 2024-04-14T15:10:33.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.899+0000 7f5dc6442700 15 client.24562.objecter _session_command_op_remove -1 2 2024-04-14T15:10:33.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.899+0000 7f5dc6442700 20 client.24562.objecter get_session s=0x7f5da807b450 osd=3 3 2024-04-14T15:10:33.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.899+0000 7f5dc6442700 15 client.24562.objecter _session_command_op_assign 3 2 2024-04-14T15:10:33.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.899+0000 7f5dc6442700 20 client.24562.objecter put_session s=0x7f5da807b450 osd=3 4 2024-04-14T15:10:33.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.899+0000 7f5dc6442700 10 client.24562.objecter _send_command 2 2024-04-14T15:10:33.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.899+0000 7f5dc6442700 1 -- 172.21.15.90:0/1308952299 --> v1:172.21.15.90:6813/2270166774 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5dc0103fc0 con 0x7f5da807bb00 2024-04-14T15:10:33.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.899+0000 7f5dbd7fa700 1 -- 172.21.15.90:0/1308952299 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (unknown 0 0 325371873) 0x7f5dc0103fc0 con 0x7f5da807bb00 2024-04-14T15:10:33.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.899+0000 7f5dbd7fa700 10 client.24562.objecter ms_dispatch 0x7f5dc0001550 command_reply(tid 2: 0 ) 2024-04-14T15:10:33.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.899+0000 7f5dbd7fa700 10 client.24562.objecter _finish_command 2 = system:0 2024-04-14T15:10:33.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.900+0000 7f5dbd7fa700 20 client.24562.objecter put_session s=0x7f5da807b450 osd=3 4 2024-04-14T15:10:33.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.900+0000 7f5dbd7fa700 15 client.24562.objecter _session_command_op_remove 3 2 2024-04-14T15:10:33.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.900+0000 7f5dc6442700 10 librados: watch_flush enter 2024-04-14T15:10:33.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.900+0000 7f5dc6442700 10 librados: watch_flush exit 2024-04-14T15:10:33.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.900+0000 7f5dc6442700 10 client.24562.objecter close_session for osd.3 2024-04-14T15:10:33.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.900+0000 7f5dc6442700 1 -- 172.21.15.90:0/1308952299 >> v1:172.21.15.90:6813/2270166774 conn(0x7f5da807bb00 legacy=0x7f5da807dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:33.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.900+0000 7f5dc6442700 20 client.24562.objecter put_session s=0x7f5da807b450 osd=3 1 2024-04-14T15:10:33.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.900+0000 7f5dc6442700 20 client.24562.objecter shutdown clearing up homeless session... 2024-04-14T15:10:33.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.900+0000 7f5dc6442700 10 client.24562.objecter successfully canceled tick 2024-04-14T15:10:33.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.901+0000 7f5dc6442700 1 -- 172.21.15.90:0/1308952299 >> v1:172.21.15.90:6800/420935427 conn(0x7f5da80713a0 legacy=0x7f5da8073860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:33.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.901+0000 7f5dc6442700 1 -- 172.21.15.90:0/1308952299 >> v1:172.21.15.92:6789/0 conn(0x7f5dc010b730 legacy=0x7f5dc01a0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:33.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.901+0000 7f5dc49df700 1 -- 172.21.15.90:0/1308952299 reap_dead start 2024-04-14T15:10:33.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.901+0000 7f5dc6442700 1 -- 172.21.15.90:0/1308952299 shutdown_connections 2024-04-14T15:10:33.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.901+0000 7f5dc6442700 1 -- 172.21.15.90:0/1308952299 >> 172.21.15.90:0/1308952299 conn(0x7f5dc00fe3d0 msgr2=0x7f5dc0111fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:10:33.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.901+0000 7f5dc6442700 1 -- 172.21.15.90:0/1308952299 shutdown_connections 2024-04-14T15:10:33.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.901+0000 7f5dc6442700 1 -- 172.21.15.90:0/1308952299 wait complete. 2024-04-14T15:10:33.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:33.901+0000 7f5dc6442700 1 librados: shutdown 2024-04-14T15:10:34.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:33 smithi092 ceph-mon[33966]: osdmap e83: 8 total, 8 up, 8 in 2024-04-14T15:10:34.191 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:10:33 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:10:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:10:34.941 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:34 smithi090 ceph-mon[41670]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 20 op/s 2024-04-14T15:10:34.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:34 smithi090 ceph-mon[30399]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 20 op/s 2024-04-14T15:10:35.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:34 smithi092 ceph-mon[33966]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 20 op/s 2024-04-14T15:10:35.122 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:10:34 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:10:36.088 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:35 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:10:36.088 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:35 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:10:36.088 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:35 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:10:36.089 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:35 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:10:36.089 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:35 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:10:36.089 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:35 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:10:36.117 INFO:teuthology.orchestra.run.smithi090.stdout:107374182434 2024-04-14T15:10:36.117 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-14T15:10:36.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:35 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:10:36.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:35 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:10:36.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:35 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:10:36.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.674+0000 7fc93aaf7700 1 -- 172.21.15.90:0/3057663114 >> v1:172.21.15.90:6789/0 conn(0x7fc93410b6f0 legacy=0x7fc93410dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:36.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.674+0000 7fc93aaf7700 1 -- 172.21.15.90:0/3057663114 shutdown_connections 2024-04-14T15:10:36.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.674+0000 7fc93aaf7700 1 -- 172.21.15.90:0/3057663114 >> 172.21.15.90:0/3057663114 conn(0x7fc934076aa0 msgr2=0x7fc934076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:10:36.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.674+0000 7fc93aaf7700 1 -- 172.21.15.90:0/3057663114 shutdown_connections 2024-04-14T15:10:36.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.674+0000 7fc93aaf7700 1 -- 172.21.15.90:0/3057663114 wait complete. 2024-04-14T15:10:36.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.674+0000 7fc93aaf7700 1 librados: starting msgr at 2024-04-14T15:10:36.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.674+0000 7fc93aaf7700 1 librados: starting objecter 2024-04-14T15:10:36.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.675+0000 7fc93aaf7700 1 Processor -- start 2024-04-14T15:10:36.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.675+0000 7fc93aaf7700 1 -- start start 2024-04-14T15:10:36.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.675+0000 7fc93aaf7700 1 librados: setting wanted keys 2024-04-14T15:10:36.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.675+0000 7fc93aaf7700 1 librados: calling monclient init 2024-04-14T15:10:36.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.675+0000 7fc93aaf7700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc9341a7920 con 0x7fc93410b6f0 2024-04-14T15:10:36.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.675+0000 7fc93aaf7700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc9341a8b40 con 0x7fc93410f2f0 2024-04-14T15:10:36.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.675+0000 7fc933fff700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fc93410b6f0 0x7fc9341a27a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:53482/0 (socket says 172.21.15.90:53482) 2024-04-14T15:10:36.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.675+0000 7fc933fff700 1 -- 172.21.15.90:0/2423652966 learned_addr learned my addr 172.21.15.90:0/2423652966 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:10:36.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.675+0000 7fc931ffb700 10 client.?.objecter ms_handle_connect 0x7fc93410b6f0 2024-04-14T15:10:36.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.675+0000 7fc931ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:10:36.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.675+0000 7fc931ffb700 10 client.?.objecter ms_handle_connect 0x7fc934108980 2024-04-14T15:10:36.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.675+0000 7fc931ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:10:36.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.675+0000 7fc93aaf7700 1 -- 172.21.15.90:0/2423652966 --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc9341a7af0 con 0x7fc934108980 2024-04-14T15:10:36.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.675+0000 7fc931ffb700 10 client.?.objecter ms_handle_connect 0x7fc93410f2f0 2024-04-14T15:10:36.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.675+0000 7fc931ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:10:36.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.675+0000 7fc931ffb700 1 -- 172.21.15.90:0/2423652966 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 792858255 0 0) 0x7fc9341a7920 con 0x7fc93410b6f0 2024-04-14T15:10:36.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.675+0000 7fc931ffb700 1 -- 172.21.15.90:0/2423652966 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9240039c0 con 0x7fc93410b6f0 2024-04-14T15:10:36.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.676+0000 7fc931ffb700 1 -- 172.21.15.90:0/2423652966 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1084855859 0 0) 0x7fc9341a8b40 con 0x7fc93410f2f0 2024-04-14T15:10:36.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.676+0000 7fc931ffb700 1 -- 172.21.15.90:0/2423652966 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9341a7920 con 0x7fc93410f2f0 2024-04-14T15:10:36.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.676+0000 7fc931ffb700 1 -- 172.21.15.90:0/2423652966 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3901820898 0 0) 0x7fc9341a7af0 con 0x7fc934108980 2024-04-14T15:10:36.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.676+0000 7fc931ffb700 1 -- 172.21.15.90:0/2423652966 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9341a8b40 con 0x7fc934108980 2024-04-14T15:10:36.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.676+0000 7fc931ffb700 1 -- 172.21.15.90:0/2423652966 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1462740919 0 0) 0x7fc9240039c0 con 0x7fc93410b6f0 2024-04-14T15:10:36.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.676+0000 7fc931ffb700 1 -- 172.21.15.90:0/2423652966 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc9341a7af0 con 0x7fc93410b6f0 2024-04-14T15:10:36.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.676+0000 7fc931ffb700 1 -- 172.21.15.90:0/2423652966 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fc920004530 con 0x7fc93410b6f0 2024-04-14T15:10:36.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.676+0000 7fc931ffb700 1 -- 172.21.15.90:0/2423652966 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2850428138 0 0) 0x7fc9341a7920 con 0x7fc93410f2f0 2024-04-14T15:10:36.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.676+0000 7fc931ffb700 1 -- 172.21.15.90:0/2423652966 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc9240039c0 con 0x7fc93410f2f0 2024-04-14T15:10:36.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.676+0000 7fc931ffb700 1 -- 172.21.15.90:0/2423652966 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fc92c001c30 con 0x7fc93410f2f0 2024-04-14T15:10:36.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.676+0000 7fc931ffb700 1 -- 172.21.15.90:0/2423652966 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 51045005 0 0) 0x7fc9341a8b40 con 0x7fc934108980 2024-04-14T15:10:36.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.676+0000 7fc931ffb700 1 -- 172.21.15.90:0/2423652966 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc9341a7920 con 0x7fc934108980 2024-04-14T15:10:36.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.676+0000 7fc931ffb700 1 -- 172.21.15.90:0/2423652966 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fc9280017c0 con 0x7fc934108980 2024-04-14T15:10:36.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.676+0000 7fc931ffb700 1 -- 172.21.15.90:0/2423652966 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2639145721 0 0) 0x7fc9341a7af0 con 0x7fc93410b6f0 2024-04-14T15:10:36.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.676+0000 7fc931ffb700 1 -- 172.21.15.90:0/2423652966 >> v1:172.21.15.90:6790/0 conn(0x7fc934108980 legacy=0x7fc934107cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:36.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.676+0000 7fc931ffb700 1 -- 172.21.15.90:0/2423652966 >> v1:172.21.15.92:6789/0 conn(0x7fc93410f2f0 legacy=0x7fc9341a5fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:36.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.676+0000 7fc931ffb700 1 -- 172.21.15.90:0/2423652966 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9341a7d00 con 0x7fc93410b6f0 2024-04-14T15:10:36.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.676+0000 7fc93aaf7700 1 -- 172.21.15.90:0/2423652966 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc9341a7ff0 con 0x7fc93410b6f0 2024-04-14T15:10:36.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.676+0000 7fc931ffb700 1 -- 172.21.15.90:0/2423652966 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fc920001b90 con 0x7fc93410b6f0 2024-04-14T15:10:36.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.676+0000 7fc931ffb700 1 -- 172.21.15.90:0/2423652966 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fc920005190 con 0x7fc93410b6f0 2024-04-14T15:10:36.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.676+0000 7fc93aaf7700 10 client.14676.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:10:36.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.677+0000 7fc93aaf7700 1 -- 172.21.15.90:0/2423652966 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc9341a84b0 con 0x7fc93410b6f0 2024-04-14T15:10:36.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.677+0000 7fc93aaf7700 1 librados: init done 2024-04-14T15:10:36.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.677+0000 7fc93aaf7700 10 client.14676.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:10:36.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.677+0000 7fc93aaf7700 15 client.14676.objecter _session_command_op_assign -1 1 2024-04-14T15:10:36.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.677+0000 7fc93aaf7700 20 client.14676.objecter _get_session osd=-1 returning homeless 2024-04-14T15:10:36.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.677+0000 7fc93aaf7700 10 client.14676.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:10:36.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.677+0000 7fc93aaf7700 1 -- 172.21.15.90:0/2423652966 --> v1:172.21.15.90:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fc93406a060 con 0x7fc93410b6f0 2024-04-14T15:10:36.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.678+0000 7fc931ffb700 1 -- 172.21.15.90:0/2423652966 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fc9200053d0 con 0x7fc93410b6f0 2024-04-14T15:10:36.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.679+0000 7fc931ffb700 1 -- 172.21.15.90:0/2423652966 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(83..83 src has 12..83) ==== 6288+0+0 (unknown 4253614559 0 0) 0x7fc92008a410 con 0x7fc93410b6f0 2024-04-14T15:10:36.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.679+0000 7fc931ffb700 10 client.14676.objecter ms_dispatch 0x7fc934001550 osd_map(83..83 src has 12..83) 2024-04-14T15:10:36.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.679+0000 7fc931ffb700 3 client.14676.objecter handle_osd_map got epochs [83,83] > 0 2024-04-14T15:10:36.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.679+0000 7fc931ffb700 3 client.14676.objecter handle_osd_map decoding full epoch 83 2024-04-14T15:10:36.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.679+0000 7fc931ffb700 10 client.14676.objecter checking command 1 2024-04-14T15:10:36.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.679+0000 7fc931ffb700 20 client.14676.objecter _get_session s=0x7fc92407b510 osd=2 3 2024-04-14T15:10:36.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.679+0000 7fc931ffb700 20 client.14676.objecter put_session s=0x7fc92407b510 osd=2 3 2024-04-14T15:10:36.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.679+0000 7fc931ffb700 15 client.14676.objecter _session_command_op_remove -1 1 2024-04-14T15:10:36.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.679+0000 7fc931ffb700 20 client.14676.objecter _get_session s=0x7fc92407b510 osd=2 3 2024-04-14T15:10:36.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.679+0000 7fc931ffb700 20 client.14676.objecter get_session s=0x7fc92407b510 osd=2 3 2024-04-14T15:10:36.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.679+0000 7fc931ffb700 15 client.14676.objecter _session_command_op_assign 2 1 2024-04-14T15:10:36.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.679+0000 7fc931ffb700 20 client.14676.objecter put_session s=0x7fc92407b510 osd=2 4 2024-04-14T15:10:36.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.679+0000 7fc931ffb700 10 client.14676.objecter _send_command 1 2024-04-14T15:10:36.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.679+0000 7fc931ffb700 1 -- 172.21.15.90:0/2423652966 --> v1:172.21.15.90:6809/1686184251 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc92407f8d0 con 0x7fc92407bbc0 2024-04-14T15:10:36.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.679+0000 7fc931ffb700 20 client.14676.objecter dump_active .. 0 homeless 2024-04-14T15:10:36.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.679+0000 7fc931ffb700 1 -- 172.21.15.90:0/2423652966 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_get_version_reply(handle=1 version=83) ==== 24+0+0 (unknown 2217717385 0 0) 0x7fc92008a850 con 0x7fc93410b6f0 2024-04-14T15:10:36.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.679+0000 7fc931ffb700 10 client.14676.objecter ms_handle_connect 0x7fc9240713d0 2024-04-14T15:10:36.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.679+0000 7fc931ffb700 10 client.14676.objecter ms_handle_connect 0x7fc92407bbc0 2024-04-14T15:10:36.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.680+0000 7fc931ffb700 1 -- 172.21.15.90:0/2423652966 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc92407f8d0 con 0x7fc92407bbc0 2024-04-14T15:10:36.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.680+0000 7fc931ffb700 10 client.14676.objecter ms_dispatch 0x7fc934001550 command_reply(tid 1: 0 ) 2024-04-14T15:10:36.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.680+0000 7fc931ffb700 10 client.14676.objecter _finish_command 1 = system:0 2024-04-14T15:10:36.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.680+0000 7fc931ffb700 20 client.14676.objecter put_session s=0x7fc92407b510 osd=2 4 2024-04-14T15:10:36.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.680+0000 7fc931ffb700 15 client.14676.objecter _session_command_op_remove 2 1 2024-04-14T15:10:36.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.698+0000 7fc93aaf7700 10 client.14676.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-14T15:10:36.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.698+0000 7fc93aaf7700 15 client.14676.objecter _session_command_op_assign -1 2 2024-04-14T15:10:36.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.698+0000 7fc93aaf7700 20 client.14676.objecter _get_session s=0x7fc92407b510 osd=2 3 2024-04-14T15:10:36.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.698+0000 7fc93aaf7700 20 client.14676.objecter put_session s=0x7fc92407b510 osd=2 3 2024-04-14T15:10:36.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.698+0000 7fc93aaf7700 20 client.14676.objecter _get_session s=0x7fc92407b510 osd=2 3 2024-04-14T15:10:36.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.698+0000 7fc93aaf7700 15 client.14676.objecter _session_command_op_remove -1 2 2024-04-14T15:10:36.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.698+0000 7fc93aaf7700 20 client.14676.objecter get_session s=0x7fc92407b510 osd=2 3 2024-04-14T15:10:36.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.698+0000 7fc93aaf7700 15 client.14676.objecter _session_command_op_assign 2 2 2024-04-14T15:10:36.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.698+0000 7fc93aaf7700 20 client.14676.objecter put_session s=0x7fc92407b510 osd=2 4 2024-04-14T15:10:36.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.698+0000 7fc93aaf7700 10 client.14676.objecter _send_command 2 2024-04-14T15:10:36.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.698+0000 7fc93aaf7700 1 -- 172.21.15.90:0/2423652966 --> v1:172.21.15.90:6809/1686184251 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc9340693d0 con 0x7fc92407bbc0 2024-04-14T15:10:36.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.699+0000 7fc931ffb700 1 -- 172.21.15.90:0/2423652966 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 3472381573) 0x7fc9340693d0 con 0x7fc92407bbc0 2024-04-14T15:10:36.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.699+0000 7fc931ffb700 10 client.14676.objecter ms_dispatch 0x7fc934001550 command_reply(tid 2: 0 ) 2024-04-14T15:10:36.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.699+0000 7fc931ffb700 10 client.14676.objecter _finish_command 2 = system:0 2024-04-14T15:10:36.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.699+0000 7fc931ffb700 20 client.14676.objecter put_session s=0x7fc92407b510 osd=2 4 2024-04-14T15:10:36.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.699+0000 7fc931ffb700 15 client.14676.objecter _session_command_op_remove 2 2 2024-04-14T15:10:36.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.699+0000 7fc93aaf7700 10 librados: watch_flush enter 2024-04-14T15:10:36.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.699+0000 7fc93aaf7700 10 librados: watch_flush exit 2024-04-14T15:10:36.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.699+0000 7fc93aaf7700 10 client.14676.objecter close_session for osd.2 2024-04-14T15:10:36.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.699+0000 7fc93aaf7700 1 -- 172.21.15.90:0/2423652966 >> v1:172.21.15.90:6809/1686184251 conn(0x7fc92407bbc0 legacy=0x7fc92407e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:36.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.699+0000 7fc93aaf7700 20 client.14676.objecter put_session s=0x7fc92407b510 osd=2 1 2024-04-14T15:10:36.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.699+0000 7fc93aaf7700 20 client.14676.objecter shutdown clearing up homeless session... 2024-04-14T15:10:36.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.699+0000 7fc93aaf7700 10 client.14676.objecter successfully canceled tick 2024-04-14T15:10:36.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.699+0000 7fc93aaf7700 1 -- 172.21.15.90:0/2423652966 >> v1:172.21.15.90:6800/420935427 conn(0x7fc9240713d0 legacy=0x7fc924073890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:36.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.699+0000 7fc93aaf7700 1 -- 172.21.15.90:0/2423652966 >> v1:172.21.15.90:6789/0 conn(0x7fc93410b6f0 legacy=0x7fc9341a27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:36.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.699+0000 7fc939094700 1 -- 172.21.15.90:0/2423652966 reap_dead start 2024-04-14T15:10:36.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.699+0000 7fc93aaf7700 1 -- 172.21.15.90:0/2423652966 shutdown_connections 2024-04-14T15:10:36.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.699+0000 7fc93aaf7700 1 -- 172.21.15.90:0/2423652966 >> 172.21.15.90:0/2423652966 conn(0x7fc934076aa0 msgr2=0x7fc934109eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:10:36.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.699+0000 7fc93aaf7700 1 -- 172.21.15.90:0/2423652966 shutdown_connections 2024-04-14T15:10:36.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.699+0000 7fc93aaf7700 1 -- 172.21.15.90:0/2423652966 wait complete. 2024-04-14T15:10:36.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:36.699+0000 7fc93aaf7700 1 librados: shutdown 2024-04-14T15:10:36.936 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:36 smithi090 ceph-mon[30399]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 20 op/s 2024-04-14T15:10:36.937 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:36 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:10:36.937 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:36 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:10:36.937 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:36 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:10:36.937 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:36 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:10:36.937 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:36 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:10:36.937 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:36 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:10:36.937 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:36 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:10:36.937 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:36 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:10:36.937 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:36 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:10:36.937 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:36 smithi090 ceph-mon[41670]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 20 op/s 2024-04-14T15:10:36.938 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:36 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:10:36.938 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:36 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:10:36.938 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:36 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:10:36.938 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:36 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:10:36.938 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:36 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:10:36.938 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:36 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:10:36.938 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:36 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:10:36.938 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:36 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:10:36.938 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:36 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:10:37.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:36 smithi092 ceph-mon[33966]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 20 op/s 2024-04-14T15:10:37.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:36 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:10:37.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:36 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:10:37.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:36 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:10:37.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:36 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:10:37.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:36 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:10:37.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:36 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:10:37.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:36 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:10:37.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:36 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:10:37.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:36 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:10:38.028 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:37 smithi090 ceph-mon[41670]: Adjusting osd_memory_target on smithi090 to 3760M 2024-04-14T15:10:38.028 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:37 smithi090 ceph-mon[41670]: Updating smithi090:/etc/ceph/ceph.conf 2024-04-14T15:10:38.028 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:37 smithi090 ceph-mon[41670]: Updating smithi092:/etc/ceph/ceph.conf 2024-04-14T15:10:38.028 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:37 smithi090 ceph-mon[41670]: Updating smithi090:/var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T15:10:38.028 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:37 smithi090 ceph-mon[41670]: Updating smithi092:/var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T15:10:38.028 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:37 smithi090 ceph-mon[41670]: Updating smithi090:/etc/ceph/ceph.client.admin.keyring 2024-04-14T15:10:38.028 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:37 smithi090 ceph-mon[41670]: Updating smithi092:/etc/ceph/ceph.client.admin.keyring 2024-04-14T15:10:38.029 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:37 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:10:38.029 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:37 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:10:38.029 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:37 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:10:38.029 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:37 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:10:38.029 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:37 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:10:38.029 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:37 smithi090 ceph-mon[30399]: Adjusting osd_memory_target on smithi090 to 3760M 2024-04-14T15:10:38.029 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:37 smithi090 ceph-mon[30399]: Updating smithi090:/etc/ceph/ceph.conf 2024-04-14T15:10:38.029 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:37 smithi090 ceph-mon[30399]: Updating smithi092:/etc/ceph/ceph.conf 2024-04-14T15:10:38.029 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:37 smithi090 ceph-mon[30399]: Updating smithi090:/var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T15:10:38.029 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:37 smithi090 ceph-mon[30399]: Updating smithi092:/var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T15:10:38.029 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:37 smithi090 ceph-mon[30399]: Updating smithi090:/etc/ceph/ceph.client.admin.keyring 2024-04-14T15:10:38.029 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:37 smithi090 ceph-mon[30399]: Updating smithi092:/etc/ceph/ceph.client.admin.keyring 2024-04-14T15:10:38.030 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:37 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:10:38.030 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:37 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:10:38.030 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:37 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:10:38.030 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:37 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:10:38.030 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:37 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:10:38.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:37 smithi092 ceph-mon[33966]: Adjusting osd_memory_target on smithi090 to 3760M 2024-04-14T15:10:38.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:37 smithi092 ceph-mon[33966]: Updating smithi090:/etc/ceph/ceph.conf 2024-04-14T15:10:38.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:37 smithi092 ceph-mon[33966]: Updating smithi092:/etc/ceph/ceph.conf 2024-04-14T15:10:38.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:37 smithi092 ceph-mon[33966]: Updating smithi090:/var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T15:10:38.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:37 smithi092 ceph-mon[33966]: Updating smithi092:/var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T15:10:38.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:37 smithi092 ceph-mon[33966]: Updating smithi090:/etc/ceph/ceph.client.admin.keyring 2024-04-14T15:10:38.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:37 smithi092 ceph-mon[33966]: Updating smithi092:/etc/ceph/ceph.client.admin.keyring 2024-04-14T15:10:38.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:37 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:10:38.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:37 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:10:38.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:37 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:10:38.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:37 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:10:38.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:37 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:10:38.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.716+0000 7f0c050ca700 1 -- 172.21.15.90:0/1199302171 >> v1:172.21.15.90:6789/0 conn(0x7f0c0010dcc0 legacy=0x7f0c001100b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:38.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.716+0000 7f0c050ca700 1 -- 172.21.15.90:0/1199302171 shutdown_connections 2024-04-14T15:10:38.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.716+0000 7f0c050ca700 1 -- 172.21.15.90:0/1199302171 >> 172.21.15.90:0/1199302171 conn(0x7f0c000fd290 msgr2=0x7f0c000ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:10:38.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.716+0000 7f0c050ca700 1 -- 172.21.15.90:0/1199302171 shutdown_connections 2024-04-14T15:10:38.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.716+0000 7f0c050ca700 1 -- 172.21.15.90:0/1199302171 wait complete. 2024-04-14T15:10:38.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.716+0000 7f0c050ca700 1 librados: starting msgr at 2024-04-14T15:10:38.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.716+0000 7f0c050ca700 1 librados: starting objecter 2024-04-14T15:10:38.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.717+0000 7f0c050ca700 1 Processor -- start 2024-04-14T15:10:38.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.717+0000 7f0c050ca700 1 -- start start 2024-04-14T15:10:38.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.717+0000 7f0c050ca700 1 librados: setting wanted keys 2024-04-14T15:10:38.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.717+0000 7f0c050ca700 1 librados: calling monclient init 2024-04-14T15:10:38.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.717+0000 7f0c050ca700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f0c0010d980 con 0x7f0c0010dcc0 2024-04-14T15:10:38.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.717+0000 7f0c050ca700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f0c001b3ab0 con 0x7f0c001016e0 2024-04-14T15:10:38.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.717+0000 7f0c050ca700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f0c001b4c90 con 0x7f0c001117d0 2024-04-14T15:10:38.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.718+0000 7f0bff59e700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f0c001117d0 0x7f0c0010af70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:34018/0 (socket says 172.21.15.90:34018) 2024-04-14T15:10:38.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.718+0000 7f0bff59e700 1 -- 172.21.15.90:0/1753306996 learned_addr learned my addr 172.21.15.90:0/1753306996 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:10:38.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.718+0000 7f0bfe59c700 1 --1- 172.21.15.90:0/1753306996 >> v1:172.21.15.90:6789/0 conn(0x7f0c0010dcc0 0x7f0c00074530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:53496/0 (socket says 172.21.15.90:53496) 2024-04-14T15:10:38.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.718+0000 7f0be7fff700 10 client.?.objecter ms_handle_connect 0x7f0c0010dcc0 2024-04-14T15:10:38.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.718+0000 7f0be7fff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:10:38.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.718+0000 7f0be7fff700 10 client.?.objecter ms_handle_connect 0x7f0c001117d0 2024-04-14T15:10:38.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.718+0000 7f0be7fff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:10:38.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.718+0000 7f0be7fff700 10 client.?.objecter ms_handle_connect 0x7f0c001016e0 2024-04-14T15:10:38.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.718+0000 7f0be7fff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:10:38.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.718+0000 7f0be7fff700 1 -- 172.21.15.90:0/1753306996 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3777673319 0 0) 0x7f0c0010d980 con 0x7f0c0010dcc0 2024-04-14T15:10:38.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.718+0000 7f0be7fff700 1 -- 172.21.15.90:0/1753306996 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0be80035b0 con 0x7f0c0010dcc0 2024-04-14T15:10:38.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.718+0000 7f0be7fff700 1 -- 172.21.15.90:0/1753306996 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3850909379 0 0) 0x7f0c001b4c90 con 0x7f0c001117d0 2024-04-14T15:10:38.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.718+0000 7f0be7fff700 1 -- 172.21.15.90:0/1753306996 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c0010d980 con 0x7f0c001117d0 2024-04-14T15:10:38.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.719+0000 7f0be7fff700 1 -- 172.21.15.90:0/1753306996 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2442365890 0 0) 0x7f0c0010d980 con 0x7f0c001117d0 2024-04-14T15:10:38.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.719+0000 7f0be7fff700 1 -- 172.21.15.90:0/1753306996 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0c001b4c90 con 0x7f0c001117d0 2024-04-14T15:10:38.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.719+0000 7f0be7fff700 1 -- 172.21.15.90:0/1753306996 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4208065501 0 0) 0x7f0be80035b0 con 0x7f0c0010dcc0 2024-04-14T15:10:38.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.719+0000 7f0be7fff700 1 -- 172.21.15.90:0/1753306996 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0c0010d980 con 0x7f0c0010dcc0 2024-04-14T15:10:38.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.719+0000 7f0be7fff700 1 -- 172.21.15.90:0/1753306996 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f0bf4001510 con 0x7f0c0010dcc0 2024-04-14T15:10:38.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.719+0000 7f0be7fff700 1 -- 172.21.15.90:0/1753306996 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f0bf8001bd0 con 0x7f0c001117d0 2024-04-14T15:10:38.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.719+0000 7f0be7fff700 1 -- 172.21.15.90:0/1753306996 <== mon.2 v1:172.21.15.90:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1896252 0 0) 0x7f0c001b4c90 con 0x7f0c001117d0 2024-04-14T15:10:38.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.719+0000 7f0be7fff700 1 -- 172.21.15.90:0/1753306996 >> v1:172.21.15.92:6789/0 conn(0x7f0c001016e0 legacy=0x7f0c0010a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:38.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.719+0000 7f0be7fff700 1 -- 172.21.15.90:0/1753306996 >> v1:172.21.15.90:6789/0 conn(0x7f0c0010dcc0 legacy=0x7f0c00074530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:38.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.719+0000 7f0be7fff700 1 -- 172.21.15.90:0/1753306996 --> v1:172.21.15.90:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c001b3c80 con 0x7f0c001117d0 2024-04-14T15:10:38.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.719+0000 7f0c050ca700 1 -- 172.21.15.90:0/1753306996 --> v1:172.21.15.90:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0c001b3f70 con 0x7f0c001117d0 2024-04-14T15:10:38.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.719+0000 7f0c050ca700 10 client.24548.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:10:38.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.719+0000 7f0c050ca700 1 -- 172.21.15.90:0/1753306996 --> v1:172.21.15.90:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f0c001b4430 con 0x7f0c001117d0 2024-04-14T15:10:38.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.719+0000 7f0c050ca700 1 librados: init done 2024-04-14T15:10:38.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.719+0000 7f0be7fff700 1 -- 172.21.15.90:0/1753306996 <== mon.2 v1:172.21.15.90:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f0bf8001d70 con 0x7f0c001117d0 2024-04-14T15:10:38.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.719+0000 7f0be7fff700 1 -- 172.21.15.90:0/1753306996 <== mon.2 v1:172.21.15.90:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f0bf8005fc0 con 0x7f0c001117d0 2024-04-14T15:10:38.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.720+0000 7f0c050ca700 10 client.24548.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:10:38.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.720+0000 7f0c050ca700 15 client.24548.objecter _session_command_op_assign -1 1 2024-04-14T15:10:38.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.720+0000 7f0c050ca700 20 client.24548.objecter _get_session osd=-1 returning homeless 2024-04-14T15:10:38.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.720+0000 7f0c050ca700 10 client.24548.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:10:38.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.720+0000 7f0c050ca700 1 -- 172.21.15.90:0/1753306996 --> v1:172.21.15.90:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f0bec002920 con 0x7f0c001117d0 2024-04-14T15:10:38.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.721+0000 7f0be7fff700 1 -- 172.21.15.90:0/1753306996 <== mon.2 v1:172.21.15.90:6790/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f0bf801cc30 con 0x7f0c001117d0 2024-04-14T15:10:38.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.721+0000 7f0be7fff700 1 -- 172.21.15.90:0/1753306996 <== mon.2 v1:172.21.15.90:6790/0 8 ==== osd_map(83..83 src has 12..83) ==== 6288+0+0 (unknown 4253614559 0 0) 0x7f0bf808a6d0 con 0x7f0c001117d0 2024-04-14T15:10:38.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.721+0000 7f0be7fff700 10 client.24548.objecter ms_dispatch 0x7f0c00001550 osd_map(83..83 src has 12..83) 2024-04-14T15:10:38.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.721+0000 7f0be7fff700 3 client.24548.objecter handle_osd_map got epochs [83,83] > 0 2024-04-14T15:10:38.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.721+0000 7f0be7fff700 3 client.24548.objecter handle_osd_map decoding full epoch 83 2024-04-14T15:10:38.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.721+0000 7f0be7fff700 10 client.24548.objecter checking command 1 2024-04-14T15:10:38.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.721+0000 7f0be7fff700 20 client.24548.objecter _get_session s=0x7f0be807b3c0 osd=5 3 2024-04-14T15:10:38.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.721+0000 7f0be7fff700 20 client.24548.objecter put_session s=0x7f0be807b3c0 osd=5 3 2024-04-14T15:10:38.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.721+0000 7f0be7fff700 15 client.24548.objecter _session_command_op_remove -1 1 2024-04-14T15:10:38.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.721+0000 7f0be7fff700 20 client.24548.objecter _get_session s=0x7f0be807b3c0 osd=5 3 2024-04-14T15:10:38.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.721+0000 7f0be7fff700 20 client.24548.objecter get_session s=0x7f0be807b3c0 osd=5 3 2024-04-14T15:10:38.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.721+0000 7f0be7fff700 15 client.24548.objecter _session_command_op_assign 5 1 2024-04-14T15:10:38.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.721+0000 7f0be7fff700 20 client.24548.objecter put_session s=0x7f0be807b3c0 osd=5 4 2024-04-14T15:10:38.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.721+0000 7f0be7fff700 10 client.24548.objecter _send_command 1 2024-04-14T15:10:38.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.721+0000 7f0be7fff700 1 -- 172.21.15.90:0/1753306996 --> v1:172.21.15.92:6804/3244561440 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0be807f780 con 0x7f0be807ba70 2024-04-14T15:10:38.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.721+0000 7f0be7fff700 20 client.24548.objecter dump_active .. 0 homeless 2024-04-14T15:10:38.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.721+0000 7f0be7fff700 1 -- 172.21.15.90:0/1753306996 <== mon.2 v1:172.21.15.90:6790/0 9 ==== mon_get_version_reply(handle=1 version=83) ==== 24+0+0 (unknown 2217717385 0 0) 0x7f0bf808ab40 con 0x7f0c001117d0 2024-04-14T15:10:38.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.722+0000 7f0be7fff700 10 client.24548.objecter ms_handle_connect 0x7f0be8071290 2024-04-14T15:10:38.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.722+0000 7f0be7fff700 10 client.24548.objecter ms_handle_connect 0x7f0be807ba70 2024-04-14T15:10:38.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.723+0000 7f0be7fff700 1 -- 172.21.15.90:0/1753306996 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f0be807f780 con 0x7f0be807ba70 2024-04-14T15:10:38.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.723+0000 7f0be7fff700 10 client.24548.objecter ms_dispatch 0x7f0c00001550 command_reply(tid 1: 0 ) 2024-04-14T15:10:38.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.723+0000 7f0be7fff700 10 client.24548.objecter _finish_command 1 = system:0 2024-04-14T15:10:38.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.723+0000 7f0be7fff700 20 client.24548.objecter put_session s=0x7f0be807b3c0 osd=5 4 2024-04-14T15:10:38.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.723+0000 7f0be7fff700 15 client.24548.objecter _session_command_op_remove 5 1 2024-04-14T15:10:38.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.739+0000 7f0c050ca700 10 client.24548.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-14T15:10:38.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.739+0000 7f0c050ca700 15 client.24548.objecter _session_command_op_assign -1 2 2024-04-14T15:10:38.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.739+0000 7f0c050ca700 20 client.24548.objecter _get_session s=0x7f0be807b3c0 osd=5 3 2024-04-14T15:10:38.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.739+0000 7f0c050ca700 20 client.24548.objecter put_session s=0x7f0be807b3c0 osd=5 3 2024-04-14T15:10:38.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.739+0000 7f0c050ca700 20 client.24548.objecter _get_session s=0x7f0be807b3c0 osd=5 3 2024-04-14T15:10:38.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.739+0000 7f0c050ca700 15 client.24548.objecter _session_command_op_remove -1 2 2024-04-14T15:10:38.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.739+0000 7f0c050ca700 20 client.24548.objecter get_session s=0x7f0be807b3c0 osd=5 3 2024-04-14T15:10:38.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.739+0000 7f0c050ca700 15 client.24548.objecter _session_command_op_assign 5 2 2024-04-14T15:10:38.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.739+0000 7f0c050ca700 20 client.24548.objecter put_session s=0x7f0be807b3c0 osd=5 4 2024-04-14T15:10:38.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.739+0000 7f0c050ca700 10 client.24548.objecter _send_command 2 2024-04-14T15:10:38.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.739+0000 7f0c050ca700 1 -- 172.21.15.90:0/1753306996 --> v1:172.21.15.92:6804/3244561440 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0bec0011f0 con 0x7f0be807ba70 2024-04-14T15:10:38.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.740+0000 7f0be7fff700 1 -- 172.21.15.90:0/1753306996 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (unknown 0 0 2286797969) 0x7f0bec0011f0 con 0x7f0be807ba70 2024-04-14T15:10:38.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.740+0000 7f0be7fff700 10 client.24548.objecter ms_dispatch 0x7f0c00001550 command_reply(tid 2: 0 ) 2024-04-14T15:10:38.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.740+0000 7f0be7fff700 10 client.24548.objecter _finish_command 2 = system:0 2024-04-14T15:10:38.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.740+0000 7f0be7fff700 20 client.24548.objecter put_session s=0x7f0be807b3c0 osd=5 4 2024-04-14T15:10:38.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.740+0000 7f0be7fff700 15 client.24548.objecter _session_command_op_remove 5 2 2024-04-14T15:10:38.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.740+0000 7f0c050ca700 10 librados: watch_flush enter 2024-04-14T15:10:38.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.740+0000 7f0c050ca700 10 librados: watch_flush exit 2024-04-14T15:10:38.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.740+0000 7f0c050ca700 10 client.24548.objecter close_session for osd.5 2024-04-14T15:10:38.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.740+0000 7f0c050ca700 1 -- 172.21.15.90:0/1753306996 >> v1:172.21.15.92:6804/3244561440 conn(0x7f0be807ba70 legacy=0x7f0be807df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:38.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.740+0000 7f0c050ca700 20 client.24548.objecter put_session s=0x7f0be807b3c0 osd=5 1 2024-04-14T15:10:38.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.740+0000 7f0c050ca700 20 client.24548.objecter shutdown clearing up homeless session... 2024-04-14T15:10:38.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.740+0000 7f0c050ca700 10 client.24548.objecter successfully canceled tick 2024-04-14T15:10:38.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.740+0000 7f0c050ca700 1 -- 172.21.15.90:0/1753306996 >> v1:172.21.15.90:6800/420935427 conn(0x7f0be8071290 legacy=0x7f0be8073750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:38.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.740+0000 7f0c050ca700 1 -- 172.21.15.90:0/1753306996 >> v1:172.21.15.90:6790/0 conn(0x7f0c001117d0 legacy=0x7f0c0010af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:38.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.740+0000 7f0bff59e700 1 -- 172.21.15.90:0/1753306996 reap_dead start 2024-04-14T15:10:38.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.740+0000 7f0c050ca700 1 -- 172.21.15.90:0/1753306996 shutdown_connections 2024-04-14T15:10:38.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.740+0000 7f0c050ca700 1 -- 172.21.15.90:0/1753306996 >> 172.21.15.90:0/1753306996 conn(0x7f0c000fd290 msgr2=0x7f0c000ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:10:38.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.740+0000 7f0c050ca700 1 -- 172.21.15.90:0/1753306996 shutdown_connections 2024-04-14T15:10:38.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.740+0000 7f0c050ca700 1 -- 172.21.15.90:0/1753306996 wait complete. 2024-04-14T15:10:38.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.740+0000 7f0c050ca700 1 librados: shutdown 2024-04-14T15:10:38.941 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:38 smithi090 ceph-mon[30399]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-14T15:10:38.941 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:38 smithi090 ceph-mon[30399]: Updating smithi090:/var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T15:10:38.941 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:38 smithi090 ceph-mon[30399]: Updating smithi092:/var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T15:10:38.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:38 smithi090 ceph-mon[30399]: Deploying daemon alertmanager.a on smithi090 2024-04-14T15:10:38.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:38 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:10:38.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:38 smithi090 ceph-mon[41670]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-14T15:10:38.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:38 smithi090 ceph-mon[41670]: Updating smithi090:/var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T15:10:38.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:38 smithi090 ceph-mon[41670]: Updating smithi092:/var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T15:10:38.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:38 smithi090 ceph-mon[41670]: Deploying daemon alertmanager.a on smithi090 2024-04-14T15:10:38.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:38 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:10:39.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:38 smithi092 ceph-mon[33966]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-14T15:10:39.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:38 smithi092 ceph-mon[33966]: Updating smithi090:/var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T15:10:39.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:38 smithi092 ceph-mon[33966]: Updating smithi092:/var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T15:10:39.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:38 smithi092 ceph-mon[33966]: Deploying daemon alertmanager.a on smithi090 2024-04-14T15:10:39.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:38 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:10:39.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.560+0000 7f7c4c349700 1 -- 172.21.15.90:0/3988993038 >> v1:172.21.15.90:6789/0 conn(0x7f7c44107780 legacy=0x7f7c44107b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:39.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.560+0000 7f7c4c349700 1 -- 172.21.15.90:0/3988993038 shutdown_connections 2024-04-14T15:10:39.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.560+0000 7f7c4c349700 1 -- 172.21.15.90:0/3988993038 >> 172.21.15.90:0/3988993038 conn(0x7f7c440fd290 msgr2=0x7f7c440ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:10:39.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.560+0000 7f7c4c349700 1 -- 172.21.15.90:0/3988993038 shutdown_connections 2024-04-14T15:10:39.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.560+0000 7f7c4c349700 1 -- 172.21.15.90:0/3988993038 wait complete. 2024-04-14T15:10:39.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.560+0000 7f7c4c349700 1 librados: starting msgr at 2024-04-14T15:10:39.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.560+0000 7f7c4c349700 1 librados: starting objecter 2024-04-14T15:10:39.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.560+0000 7f7c4c349700 1 Processor -- start 2024-04-14T15:10:39.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.560+0000 7f7c4c349700 1 -- start start 2024-04-14T15:10:39.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.560+0000 7f7c4c349700 1 librados: setting wanted keys 2024-04-14T15:10:39.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.560+0000 7f7c4c349700 1 librados: calling monclient init 2024-04-14T15:10:39.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.561+0000 7f7c4c349700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7c4410def0 con 0x7f7c44107780 2024-04-14T15:10:39.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.561+0000 7f7c4c349700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7c441a8e20 con 0x7f7c4410e200 2024-04-14T15:10:39.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.561+0000 7f7c4c349700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7c441aa000 con 0x7f7c4410a630 2024-04-14T15:10:39.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.561+0000 7f7c4a0e5700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f7c44107780 0x7f7c4406aed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:53516/0 (socket says 172.21.15.90:53516) 2024-04-14T15:10:39.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.561+0000 7f7c4a0e5700 1 -- 172.21.15.90:0/2958490715 learned_addr learned my addr 172.21.15.90:0/2958490715 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:10:39.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.561+0000 7f7c3b7fe700 10 client.?.objecter ms_handle_connect 0x7f7c4410a630 2024-04-14T15:10:39.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.561+0000 7f7c3b7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:10:39.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.561+0000 7f7c3b7fe700 10 client.?.objecter ms_handle_connect 0x7f7c44107780 2024-04-14T15:10:39.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.561+0000 7f7c3b7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:10:39.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.561+0000 7f7c3b7fe700 1 -- 172.21.15.90:0/2958490715 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4148371965 0 0) 0x7f7c441aa000 con 0x7f7c4410a630 2024-04-14T15:10:39.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.561+0000 7f7c3b7fe700 1 -- 172.21.15.90:0/2958490715 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c300031f0 con 0x7f7c4410a630 2024-04-14T15:10:39.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.561+0000 7f7c3b7fe700 10 client.?.objecter ms_handle_connect 0x7f7c4410e200 2024-04-14T15:10:39.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.561+0000 7f7c3b7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:10:39.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.561+0000 7f7c3b7fe700 1 -- 172.21.15.90:0/2958490715 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 243768916 0 0) 0x7f7c4410def0 con 0x7f7c44107780 2024-04-14T15:10:39.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.561+0000 7f7c3b7fe700 1 -- 172.21.15.90:0/2958490715 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c441aa000 con 0x7f7c44107780 2024-04-14T15:10:39.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.561+0000 7f7c3b7fe700 1 -- 172.21.15.90:0/2958490715 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 440212294 0 0) 0x7f7c441a8e20 con 0x7f7c4410e200 2024-04-14T15:10:39.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.561+0000 7f7c3b7fe700 1 -- 172.21.15.90:0/2958490715 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c4410def0 con 0x7f7c4410e200 2024-04-14T15:10:39.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.561+0000 7f7c3b7fe700 1 -- 172.21.15.90:0/2958490715 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1489535897 0 0) 0x7f7c300031f0 con 0x7f7c4410a630 2024-04-14T15:10:39.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.561+0000 7f7c3b7fe700 1 -- 172.21.15.90:0/2958490715 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7c441a8e20 con 0x7f7c4410a630 2024-04-14T15:10:39.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.561+0000 7f7c3b7fe700 1 -- 172.21.15.90:0/2958490715 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f7c34001830 con 0x7f7c4410a630 2024-04-14T15:10:39.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.561+0000 7f7c3b7fe700 1 -- 172.21.15.90:0/2958490715 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2642202124 0 0) 0x7f7c441aa000 con 0x7f7c44107780 2024-04-14T15:10:39.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.561+0000 7f7c3b7fe700 1 -- 172.21.15.90:0/2958490715 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7c300031f0 con 0x7f7c44107780 2024-04-14T15:10:39.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.561+0000 7f7c3b7fe700 1 -- 172.21.15.90:0/2958490715 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f7c3c003d10 con 0x7f7c44107780 2024-04-14T15:10:39.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.561+0000 7f7c3b7fe700 1 -- 172.21.15.90:0/2958490715 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 214480266 0 0) 0x7f7c4410def0 con 0x7f7c4410e200 2024-04-14T15:10:39.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.561+0000 7f7c3b7fe700 1 -- 172.21.15.90:0/2958490715 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7c441aa000 con 0x7f7c4410e200 2024-04-14T15:10:39.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.561+0000 7f7c3b7fe700 1 -- 172.21.15.90:0/2958490715 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f7c40001210 con 0x7f7c4410e200 2024-04-14T15:10:39.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.561+0000 7f7c3b7fe700 1 -- 172.21.15.90:0/2958490715 <== mon.2 v1:172.21.15.90:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 653794297 0 0) 0x7f7c441a8e20 con 0x7f7c4410a630 2024-04-14T15:10:39.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.561+0000 7f7c3b7fe700 1 -- 172.21.15.90:0/2958490715 >> v1:172.21.15.92:6789/0 conn(0x7f7c4410e200 legacy=0x7f7c441a66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:39.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.561+0000 7f7c3b7fe700 1 -- 172.21.15.90:0/2958490715 >> v1:172.21.15.90:6789/0 conn(0x7f7c44107780 legacy=0x7f7c4406aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:39.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.561+0000 7f7c3b7fe700 1 -- 172.21.15.90:0/2958490715 --> v1:172.21.15.90:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c441ab1e0 con 0x7f7c4410a630 2024-04-14T15:10:39.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.562+0000 7f7c4c349700 1 -- 172.21.15.90:0/2958490715 --> v1:172.21.15.90:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7c441a7e70 con 0x7f7c4410a630 2024-04-14T15:10:39.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.562+0000 7f7c4c349700 10 client.24560.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:10:39.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.562+0000 7f7c4c349700 1 -- 172.21.15.90:0/2958490715 --> v1:172.21.15.90:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f7c441a8330 con 0x7f7c4410a630 2024-04-14T15:10:39.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.562+0000 7f7c4c349700 1 librados: init done 2024-04-14T15:10:39.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.562+0000 7f7c3b7fe700 1 -- 172.21.15.90:0/2958490715 <== mon.2 v1:172.21.15.90:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f7c34003b20 con 0x7f7c4410a630 2024-04-14T15:10:39.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.562+0000 7f7c3b7fe700 1 -- 172.21.15.90:0/2958490715 <== mon.2 v1:172.21.15.90:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f7c34004990 con 0x7f7c4410a630 2024-04-14T15:10:39.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.562+0000 7f7c4c349700 10 client.24560.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:10:39.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.562+0000 7f7c4c349700 15 client.24560.objecter _session_command_op_assign -1 1 2024-04-14T15:10:39.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.562+0000 7f7c4c349700 20 client.24560.objecter _get_session osd=-1 returning homeless 2024-04-14T15:10:39.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.562+0000 7f7c4c349700 10 client.24560.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:10:39.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.562+0000 7f7c4c349700 1 -- 172.21.15.90:0/2958490715 --> v1:172.21.15.90:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f7c28002920 con 0x7f7c4410a630 2024-04-14T15:10:39.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.563+0000 7f7c3b7fe700 1 -- 172.21.15.90:0/2958490715 <== mon.2 v1:172.21.15.90:6790/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f7c3401b620 con 0x7f7c4410a630 2024-04-14T15:10:39.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.564+0000 7f7c3b7fe700 1 -- 172.21.15.90:0/2958490715 <== mon.2 v1:172.21.15.90:6790/0 8 ==== osd_map(83..83 src has 12..83) ==== 6288+0+0 (unknown 4253614559 0 0) 0x7f7c34088f90 con 0x7f7c4410a630 2024-04-14T15:10:39.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.564+0000 7f7c3b7fe700 10 client.24560.objecter ms_dispatch 0x7f7c44001550 osd_map(83..83 src has 12..83) 2024-04-14T15:10:39.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.564+0000 7f7c3b7fe700 3 client.24560.objecter handle_osd_map got epochs [83,83] > 0 2024-04-14T15:10:39.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.564+0000 7f7c3b7fe700 3 client.24560.objecter handle_osd_map decoding full epoch 83 2024-04-14T15:10:39.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.564+0000 7f7c3b7fe700 10 client.24560.objecter checking command 1 2024-04-14T15:10:39.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.564+0000 7f7c3b7fe700 20 client.24560.objecter _get_session s=0x7f7c3007b460 osd=4 3 2024-04-14T15:10:39.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.564+0000 7f7c3b7fe700 20 client.24560.objecter put_session s=0x7f7c3007b460 osd=4 3 2024-04-14T15:10:39.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.564+0000 7f7c3b7fe700 15 client.24560.objecter _session_command_op_remove -1 1 2024-04-14T15:10:39.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.564+0000 7f7c3b7fe700 20 client.24560.objecter _get_session s=0x7f7c3007b460 osd=4 3 2024-04-14T15:10:39.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.564+0000 7f7c3b7fe700 20 client.24560.objecter get_session s=0x7f7c3007b460 osd=4 3 2024-04-14T15:10:39.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.564+0000 7f7c3b7fe700 15 client.24560.objecter _session_command_op_assign 4 1 2024-04-14T15:10:39.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.564+0000 7f7c3b7fe700 20 client.24560.objecter put_session s=0x7f7c3007b460 osd=4 4 2024-04-14T15:10:39.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.564+0000 7f7c3b7fe700 10 client.24560.objecter _send_command 1 2024-04-14T15:10:39.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.564+0000 7f7c3b7fe700 1 -- 172.21.15.90:0/2958490715 --> v1:172.21.15.92:6800/3887549733 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7c3007f820 con 0x7f7c3007bb10 2024-04-14T15:10:39.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.564+0000 7f7c3b7fe700 20 client.24560.objecter dump_active .. 0 homeless 2024-04-14T15:10:39.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.564+0000 7f7c3b7fe700 1 -- 172.21.15.90:0/2958490715 <== mon.2 v1:172.21.15.90:6790/0 9 ==== mon_get_version_reply(handle=1 version=83) ==== 24+0+0 (unknown 2217717385 0 0) 0x7f7c34089400 con 0x7f7c4410a630 2024-04-14T15:10:39.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.564+0000 7f7c3b7fe700 10 client.24560.objecter ms_handle_connect 0x7f7c30071350 2024-04-14T15:10:39.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.564+0000 7f7c3b7fe700 10 client.24560.objecter ms_handle_connect 0x7f7c3007bb10 2024-04-14T15:10:39.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.565+0000 7f7c3b7fe700 1 -- 172.21.15.90:0/2958490715 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f7c3007f820 con 0x7f7c3007bb10 2024-04-14T15:10:39.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.565+0000 7f7c3b7fe700 10 client.24560.objecter ms_dispatch 0x7f7c44001550 command_reply(tid 1: 0 ) 2024-04-14T15:10:39.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.565+0000 7f7c3b7fe700 10 client.24560.objecter _finish_command 1 = system:0 2024-04-14T15:10:39.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.565+0000 7f7c3b7fe700 20 client.24560.objecter put_session s=0x7f7c3007b460 osd=4 4 2024-04-14T15:10:39.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.565+0000 7f7c3b7fe700 15 client.24560.objecter _session_command_op_remove 4 1 2024-04-14T15:10:39.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.581+0000 7f7c4c349700 10 client.24560.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-14T15:10:39.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.581+0000 7f7c4c349700 15 client.24560.objecter _session_command_op_assign -1 2 2024-04-14T15:10:39.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.581+0000 7f7c4c349700 20 client.24560.objecter _get_session s=0x7f7c3007b460 osd=4 3 2024-04-14T15:10:39.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.581+0000 7f7c4c349700 20 client.24560.objecter put_session s=0x7f7c3007b460 osd=4 3 2024-04-14T15:10:39.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.581+0000 7f7c4c349700 20 client.24560.objecter _get_session s=0x7f7c3007b460 osd=4 3 2024-04-14T15:10:39.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.581+0000 7f7c4c349700 15 client.24560.objecter _session_command_op_remove -1 2 2024-04-14T15:10:39.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.581+0000 7f7c4c349700 20 client.24560.objecter get_session s=0x7f7c3007b460 osd=4 3 2024-04-14T15:10:39.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.581+0000 7f7c4c349700 15 client.24560.objecter _session_command_op_assign 4 2 2024-04-14T15:10:39.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.581+0000 7f7c4c349700 20 client.24560.objecter put_session s=0x7f7c3007b460 osd=4 4 2024-04-14T15:10:39.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.581+0000 7f7c4c349700 10 client.24560.objecter _send_command 2 2024-04-14T15:10:39.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.581+0000 7f7c4c349700 1 -- 172.21.15.90:0/2958490715 --> v1:172.21.15.92:6800/3887549733 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7c280011f0 con 0x7f7c3007bb10 2024-04-14T15:10:39.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.581+0000 7f7c3b7fe700 1 -- 172.21.15.90:0/2958490715 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (unknown 0 0 3087939264) 0x7f7c280011f0 con 0x7f7c3007bb10 2024-04-14T15:10:39.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.581+0000 7f7c3b7fe700 10 client.24560.objecter ms_dispatch 0x7f7c44001550 command_reply(tid 2: 0 ) 2024-04-14T15:10:39.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.581+0000 7f7c3b7fe700 10 client.24560.objecter _finish_command 2 = system:0 2024-04-14T15:10:39.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.581+0000 7f7c3b7fe700 20 client.24560.objecter put_session s=0x7f7c3007b460 osd=4 4 2024-04-14T15:10:39.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.581+0000 7f7c3b7fe700 15 client.24560.objecter _session_command_op_remove 4 2 2024-04-14T15:10:39.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.581+0000 7f7c4c349700 10 librados: watch_flush enter 2024-04-14T15:10:39.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.581+0000 7f7c4c349700 10 librados: watch_flush exit 2024-04-14T15:10:39.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.581+0000 7f7c4c349700 10 client.24560.objecter close_session for osd.4 2024-04-14T15:10:39.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.581+0000 7f7c4c349700 1 -- 172.21.15.90:0/2958490715 >> v1:172.21.15.92:6800/3887549733 conn(0x7f7c3007bb10 legacy=0x7f7c3007dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:39.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.581+0000 7f7c4c349700 20 client.24560.objecter put_session s=0x7f7c3007b460 osd=4 1 2024-04-14T15:10:39.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.581+0000 7f7c4c349700 20 client.24560.objecter shutdown clearing up homeless session... 2024-04-14T15:10:39.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.581+0000 7f7c4c349700 10 client.24560.objecter successfully canceled tick 2024-04-14T15:10:39.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.582+0000 7f7c4c349700 1 -- 172.21.15.90:0/2958490715 >> v1:172.21.15.90:6800/420935427 conn(0x7f7c30071350 legacy=0x7f7c30073810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:39.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.582+0000 7f7c4c349700 1 -- 172.21.15.90:0/2958490715 >> v1:172.21.15.90:6790/0 conn(0x7f7c4410a630 legacy=0x7f7c4406b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:39.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.582+0000 7f7c4a8e6700 1 -- 172.21.15.90:0/2958490715 reap_dead start 2024-04-14T15:10:39.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.582+0000 7f7c4c349700 1 -- 172.21.15.90:0/2958490715 shutdown_connections 2024-04-14T15:10:39.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.582+0000 7f7c4c349700 1 -- 172.21.15.90:0/2958490715 >> 172.21.15.90:0/2958490715 conn(0x7f7c440fd290 msgr2=0x7f7c441117b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:10:39.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.582+0000 7f7c4c349700 1 -- 172.21.15.90:0/2958490715 shutdown_connections 2024-04-14T15:10:39.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.582+0000 7f7c4c349700 1 -- 172.21.15.90:0/2958490715 wait complete. 2024-04-14T15:10:39.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:39.582+0000 7f7c4c349700 1 librados: shutdown 2024-04-14T15:10:39.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.959+0000 7f14d2ae0700 1 -- 172.21.15.90:0/2772642566 >> v1:172.21.15.90:6790/0 conn(0x7f14cc10a620 legacy=0x7f14cc10ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:39.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.959+0000 7f14d2ae0700 1 -- 172.21.15.90:0/2772642566 shutdown_connections 2024-04-14T15:10:39.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.959+0000 7f14d2ae0700 1 -- 172.21.15.90:0/2772642566 >> 172.21.15.90:0/2772642566 conn(0x7f14cc0fd280 msgr2=0x7f14cc0ff6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:10:39.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.959+0000 7f14d2ae0700 1 -- 172.21.15.90:0/2772642566 shutdown_connections 2024-04-14T15:10:39.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.959+0000 7f14d2ae0700 1 -- 172.21.15.90:0/2772642566 wait complete. 2024-04-14T15:10:39.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.960+0000 7f14d2ae0700 1 librados: starting msgr at 2024-04-14T15:10:39.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.960+0000 7f14d2ae0700 1 librados: starting objecter 2024-04-14T15:10:39.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.960+0000 7f14d2ae0700 1 Processor -- start 2024-04-14T15:10:39.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.960+0000 7f14d2ae0700 1 -- start start 2024-04-14T15:10:39.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.960+0000 7f14d2ae0700 1 librados: setting wanted keys 2024-04-14T15:10:39.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.960+0000 7f14d2ae0700 1 librados: calling monclient init 2024-04-14T15:10:39.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.960+0000 7f14d2ae0700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f14cc1a9c50 con 0x7f14cc10a620 2024-04-14T15:10:39.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.960+0000 7f14d2ae0700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f14cc1aae70 con 0x7f14cc107770 2024-04-14T15:10:39.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.960+0000 7f14d2ae0700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f14cc1ac0b0 con 0x7f14cc10e1f0 2024-04-14T15:10:39.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.960+0000 7f14cbfff700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f14cc10a620 0x7f14cc1a4aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:53512/0 (socket says 172.21.15.90:53512) 2024-04-14T15:10:39.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.960+0000 7f14cbfff700 1 -- 172.21.15.90:0/3480869894 learned_addr learned my addr 172.21.15.90:0/3480869894 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:10:39.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.960+0000 7f14c9ffb700 10 client.?.objecter ms_handle_connect 0x7f14cc10e1f0 2024-04-14T15:10:39.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.960+0000 7f14c9ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:10:39.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.960+0000 7f14c9ffb700 10 client.?.objecter ms_handle_connect 0x7f14cc10a620 2024-04-14T15:10:39.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.960+0000 7f14c9ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:10:39.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.960+0000 7f14c9ffb700 1 -- 172.21.15.90:0/3480869894 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3657850877 0 0) 0x7f14cc1ac0b0 con 0x7f14cc10e1f0 2024-04-14T15:10:39.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.960+0000 7f14c9ffb700 1 -- 172.21.15.90:0/3480869894 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f14bc003480 con 0x7f14cc10e1f0 2024-04-14T15:10:39.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.960+0000 7f14c9ffb700 1 -- 172.21.15.90:0/3480869894 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2164893570 0 0) 0x7f14cc1a9c50 con 0x7f14cc10a620 2024-04-14T15:10:39.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.960+0000 7f14c9ffb700 1 -- 172.21.15.90:0/3480869894 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f14cc1ac0b0 con 0x7f14cc10a620 2024-04-14T15:10:39.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.960+0000 7f14c9ffb700 10 client.?.objecter ms_handle_connect 0x7f14cc107770 2024-04-14T15:10:39.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.960+0000 7f14c9ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:10:39.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.961+0000 7f14c9ffb700 1 -- 172.21.15.90:0/3480869894 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2409778756 0 0) 0x7f14cc1aae70 con 0x7f14cc107770 2024-04-14T15:10:39.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.961+0000 7f14c9ffb700 1 -- 172.21.15.90:0/3480869894 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f14cc1a9c50 con 0x7f14cc107770 2024-04-14T15:10:39.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.961+0000 7f14c9ffb700 1 -- 172.21.15.90:0/3480869894 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 554764025 0 0) 0x7f14cc1ac0b0 con 0x7f14cc10a620 2024-04-14T15:10:39.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.961+0000 7f14c9ffb700 1 -- 172.21.15.90:0/3480869894 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f14cc1aae70 con 0x7f14cc10a620 2024-04-14T15:10:39.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.961+0000 7f14c9ffb700 1 -- 172.21.15.90:0/3480869894 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3019574493 0 0) 0x7f14bc003480 con 0x7f14cc10e1f0 2024-04-14T15:10:39.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.961+0000 7f14c9ffb700 1 -- 172.21.15.90:0/3480869894 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f14cc1ac0b0 con 0x7f14cc10e1f0 2024-04-14T15:10:39.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.961+0000 7f14c9ffb700 1 -- 172.21.15.90:0/3480869894 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f14b8003820 con 0x7f14cc10a620 2024-04-14T15:10:39.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.961+0000 7f14c9ffb700 1 -- 172.21.15.90:0/3480869894 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f14c4001b70 con 0x7f14cc10e1f0 2024-04-14T15:10:39.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.961+0000 7f14c9ffb700 1 -- 172.21.15.90:0/3480869894 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1693635863 0 0) 0x7f14cc1a9c50 con 0x7f14cc107770 2024-04-14T15:10:39.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.961+0000 7f14c9ffb700 1 -- 172.21.15.90:0/3480869894 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f14bc003480 con 0x7f14cc107770 2024-04-14T15:10:39.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.961+0000 7f14c9ffb700 1 -- 172.21.15.90:0/3480869894 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2891364303 0 0) 0x7f14cc1aae70 con 0x7f14cc10a620 2024-04-14T15:10:39.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.961+0000 7f14c9ffb700 1 -- 172.21.15.90:0/3480869894 >> v1:172.21.15.90:6790/0 conn(0x7f14cc10e1f0 legacy=0x7f14cc1a8310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:39.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.961+0000 7f14c9ffb700 1 -- 172.21.15.90:0/3480869894 >> v1:172.21.15.92:6789/0 conn(0x7f14cc107770 legacy=0x7f14cc10da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:39.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.961+0000 7f14c9ffb700 1 -- 172.21.15.90:0/3480869894 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14cc1ad2f0 con 0x7f14cc10a620 2024-04-14T15:10:39.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.961+0000 7f14d2ae0700 1 -- 172.21.15.90:0/3480869894 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f14cc1a9e80 con 0x7f14cc10a620 2024-04-14T15:10:39.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.961+0000 7f14c9ffb700 1 -- 172.21.15.90:0/3480869894 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f14b8003a60 con 0x7f14cc10a620 2024-04-14T15:10:39.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.961+0000 7f14c9ffb700 1 -- 172.21.15.90:0/3480869894 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f14b80050c0 con 0x7f14cc10a620 2024-04-14T15:10:39.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.961+0000 7f14d2ae0700 10 client.14688.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:10:39.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.961+0000 7f14d2ae0700 1 -- 172.21.15.90:0/3480869894 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f14cc1aa340 con 0x7f14cc10a620 2024-04-14T15:10:39.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.961+0000 7f14d2ae0700 1 librados: init done 2024-04-14T15:10:39.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.962+0000 7f14d2ae0700 10 client.14688.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:10:39.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.962+0000 7f14d2ae0700 15 client.14688.objecter _session_command_op_assign -1 1 2024-04-14T15:10:39.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.962+0000 7f14d2ae0700 20 client.14688.objecter _get_session osd=-1 returning homeless 2024-04-14T15:10:39.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.962+0000 7f14d2ae0700 10 client.14688.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:10:39.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.962+0000 7f14d2ae0700 1 -- 172.21.15.90:0/3480869894 --> v1:172.21.15.90:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f14cc102eb0 con 0x7f14cc10a620 2024-04-14T15:10:39.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.962+0000 7f14c9ffb700 1 -- 172.21.15.90:0/3480869894 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f14b8005340 con 0x7f14cc10a620 2024-04-14T15:10:39.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.963+0000 7f14c9ffb700 1 -- 172.21.15.90:0/3480869894 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(83..83 src has 12..83) ==== 6288+0+0 (unknown 4253614559 0 0) 0x7f14b808a440 con 0x7f14cc10a620 2024-04-14T15:10:39.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.963+0000 7f14c9ffb700 10 client.14688.objecter ms_dispatch 0x7f14cc001550 osd_map(83..83 src has 12..83) 2024-04-14T15:10:39.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.963+0000 7f14c9ffb700 3 client.14688.objecter handle_osd_map got epochs [83,83] > 0 2024-04-14T15:10:39.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.963+0000 7f14c9ffb700 3 client.14688.objecter handle_osd_map decoding full epoch 83 2024-04-14T15:10:39.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.963+0000 7f14c9ffb700 10 client.14688.objecter checking command 1 2024-04-14T15:10:39.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.963+0000 7f14c9ffb700 20 client.14688.objecter _get_session s=0x7f14bc07b450 osd=6 3 2024-04-14T15:10:39.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.963+0000 7f14c9ffb700 20 client.14688.objecter put_session s=0x7f14bc07b450 osd=6 3 2024-04-14T15:10:39.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.963+0000 7f14c9ffb700 15 client.14688.objecter _session_command_op_remove -1 1 2024-04-14T15:10:39.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.963+0000 7f14c9ffb700 20 client.14688.objecter _get_session s=0x7f14bc07b450 osd=6 3 2024-04-14T15:10:39.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.963+0000 7f14c9ffb700 20 client.14688.objecter get_session s=0x7f14bc07b450 osd=6 3 2024-04-14T15:10:39.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.963+0000 7f14c9ffb700 15 client.14688.objecter _session_command_op_assign 6 1 2024-04-14T15:10:39.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.963+0000 7f14c9ffb700 20 client.14688.objecter put_session s=0x7f14bc07b450 osd=6 4 2024-04-14T15:10:39.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.963+0000 7f14c9ffb700 10 client.14688.objecter _send_command 1 2024-04-14T15:10:39.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.963+0000 7f14c9ffb700 1 -- 172.21.15.90:0/3480869894 --> v1:172.21.15.92:6808/636125963 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f14b8001ca0 con 0x7f14bc07bb00 2024-04-14T15:10:39.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.963+0000 7f14c9ffb700 20 client.14688.objecter dump_active .. 0 homeless 2024-04-14T15:10:39.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.963+0000 7f14c9ffb700 10 client.14688.objecter ms_handle_connect 0x7f14bc0713a0 2024-04-14T15:10:39.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.963+0000 7f14c9ffb700 1 -- 172.21.15.90:0/3480869894 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_get_version_reply(handle=1 version=83) ==== 24+0+0 (unknown 2217717385 0 0) 0x7f14b808a8b0 con 0x7f14cc10a620 2024-04-14T15:10:39.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.964+0000 7f14c9ffb700 10 client.14688.objecter ms_handle_connect 0x7f14bc07bb00 2024-04-14T15:10:39.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.965+0000 7f14c9ffb700 1 -- 172.21.15.90:0/3480869894 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f14b8001ca0 con 0x7f14bc07bb00 2024-04-14T15:10:39.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.965+0000 7f14c9ffb700 10 client.14688.objecter ms_dispatch 0x7f14cc001550 command_reply(tid 1: 0 ) 2024-04-14T15:10:39.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.965+0000 7f14c9ffb700 10 client.14688.objecter _finish_command 1 = system:0 2024-04-14T15:10:39.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.965+0000 7f14c9ffb700 20 client.14688.objecter put_session s=0x7f14bc07b450 osd=6 4 2024-04-14T15:10:39.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.965+0000 7f14c9ffb700 15 client.14688.objecter _session_command_op_remove 6 1 2024-04-14T15:10:39.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.981+0000 7f14d2ae0700 10 client.14688.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-14T15:10:39.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.981+0000 7f14d2ae0700 15 client.14688.objecter _session_command_op_assign -1 2 2024-04-14T15:10:39.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.981+0000 7f14d2ae0700 20 client.14688.objecter _get_session s=0x7f14bc07b450 osd=6 3 2024-04-14T15:10:39.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.981+0000 7f14d2ae0700 20 client.14688.objecter put_session s=0x7f14bc07b450 osd=6 3 2024-04-14T15:10:39.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.981+0000 7f14d2ae0700 20 client.14688.objecter _get_session s=0x7f14bc07b450 osd=6 3 2024-04-14T15:10:39.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.981+0000 7f14d2ae0700 15 client.14688.objecter _session_command_op_remove -1 2 2024-04-14T15:10:39.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.981+0000 7f14d2ae0700 20 client.14688.objecter get_session s=0x7f14bc07b450 osd=6 3 2024-04-14T15:10:39.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.981+0000 7f14d2ae0700 15 client.14688.objecter _session_command_op_assign 6 2 2024-04-14T15:10:39.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.981+0000 7f14d2ae0700 20 client.14688.objecter put_session s=0x7f14bc07b450 osd=6 4 2024-04-14T15:10:39.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.981+0000 7f14d2ae0700 10 client.14688.objecter _send_command 2 2024-04-14T15:10:39.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.981+0000 7f14d2ae0700 1 -- 172.21.15.90:0/3480869894 --> v1:172.21.15.92:6808/636125963 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f14cc0693d0 con 0x7f14bc07bb00 2024-04-14T15:10:39.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.982+0000 7f14c9ffb700 1 -- 172.21.15.90:0/3480869894 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (unknown 0 0 387455257) 0x7f14cc0693d0 con 0x7f14bc07bb00 2024-04-14T15:10:39.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.982+0000 7f14c9ffb700 10 client.14688.objecter ms_dispatch 0x7f14cc001550 command_reply(tid 2: 0 ) 2024-04-14T15:10:39.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.982+0000 7f14c9ffb700 10 client.14688.objecter _finish_command 2 = system:0 2024-04-14T15:10:39.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.982+0000 7f14c9ffb700 20 client.14688.objecter put_session s=0x7f14bc07b450 osd=6 4 2024-04-14T15:10:39.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.982+0000 7f14c9ffb700 15 client.14688.objecter _session_command_op_remove 6 2 2024-04-14T15:10:39.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.982+0000 7f14d2ae0700 10 librados: watch_flush enter 2024-04-14T15:10:39.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.982+0000 7f14d2ae0700 10 librados: watch_flush exit 2024-04-14T15:10:39.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.982+0000 7f14d2ae0700 10 client.14688.objecter close_session for osd.6 2024-04-14T15:10:39.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.982+0000 7f14d2ae0700 1 -- 172.21.15.90:0/3480869894 >> v1:172.21.15.92:6808/636125963 conn(0x7f14bc07bb00 legacy=0x7f14bc07dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:39.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.982+0000 7f14d2ae0700 20 client.14688.objecter put_session s=0x7f14bc07b450 osd=6 1 2024-04-14T15:10:39.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.982+0000 7f14d2ae0700 20 client.14688.objecter shutdown clearing up homeless session... 2024-04-14T15:10:39.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.982+0000 7f14d2ae0700 10 client.14688.objecter successfully canceled tick 2024-04-14T15:10:39.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.982+0000 7f14d2ae0700 1 -- 172.21.15.90:0/3480869894 >> v1:172.21.15.90:6800/420935427 conn(0x7f14bc0713a0 legacy=0x7f14bc073860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:39.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.982+0000 7f14d2ae0700 1 -- 172.21.15.90:0/3480869894 >> v1:172.21.15.90:6789/0 conn(0x7f14cc10a620 legacy=0x7f14cc1a4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:39.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.982+0000 7f14d107d700 1 -- 172.21.15.90:0/3480869894 reap_dead start 2024-04-14T15:10:39.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.983+0000 7f14d2ae0700 1 -- 172.21.15.90:0/3480869894 shutdown_connections 2024-04-14T15:10:39.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.983+0000 7f14d2ae0700 1 -- 172.21.15.90:0/3480869894 >> 172.21.15.90:0/3480869894 conn(0x7f14cc0fd280 msgr2=0x7f14cc0ffe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:10:39.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.983+0000 7f14d2ae0700 1 -- 172.21.15.90:0/3480869894 shutdown_connections 2024-04-14T15:10:39.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.983+0000 7f14d2ae0700 1 -- 172.21.15.90:0/3480869894 wait complete. 2024-04-14T15:10:39.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:38.983+0000 7f14d2ae0700 1 librados: shutdown 2024-04-14T15:10:40.807 INFO:teuthology.orchestra.run.smithi090.stdout:210453397524 2024-04-14T15:10:40.807 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-14T15:10:41.090 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:40 smithi090 ceph-mon[30399]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:10:41.091 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:40 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:10:41.091 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:40 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:10:41.091 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:40 smithi090 ceph-mon[41670]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:10:41.091 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:40 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:10:41.091 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:40 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:10:41.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:40 smithi092 ceph-mon[33966]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:10:41.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:40 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:10:41.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:40 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:10:41.399 INFO:teuthology.orchestra.run.smithi090.stdout:133143986206 2024-04-14T15:10:41.399 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-14T15:10:41.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.610+0000 7f05e312f700 1 -- 172.21.15.90:0/1879330812 >> v1:172.21.15.90:6789/0 conn(0x7f05dc10a740 legacy=0x7f05dc10cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:41.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.610+0000 7f05e312f700 1 -- 172.21.15.90:0/1879330812 shutdown_connections 2024-04-14T15:10:41.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.610+0000 7f05e312f700 1 -- 172.21.15.90:0/1879330812 >> 172.21.15.90:0/1879330812 conn(0x7f05dc0fd290 msgr2=0x7f05dc0ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:10:41.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.610+0000 7f05e312f700 1 -- 172.21.15.90:0/1879330812 shutdown_connections 2024-04-14T15:10:41.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.610+0000 7f05e312f700 1 -- 172.21.15.90:0/1879330812 wait complete. 2024-04-14T15:10:41.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.610+0000 7f05e312f700 1 librados: starting msgr at 2024-04-14T15:10:41.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.610+0000 7f05e312f700 1 librados: starting objecter 2024-04-14T15:10:41.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.610+0000 7f05e312f700 1 Processor -- start 2024-04-14T15:10:41.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.611+0000 7f05e312f700 1 -- start start 2024-04-14T15:10:41.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.611+0000 7f05e312f700 1 librados: setting wanted keys 2024-04-14T15:10:41.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.611+0000 7f05e312f700 1 librados: calling monclient init 2024-04-14T15:10:41.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.611+0000 7f05e312f700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f05dc10de70 con 0x7f05dc10a740 2024-04-14T15:10:41.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.611+0000 7f05e312f700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f05dc10e080 con 0x7f05dc10e310 2024-04-14T15:10:41.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.611+0000 7f05e312f700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f05dc1a8f30 con 0x7f05dc107890 2024-04-14T15:10:41.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.611+0000 7f05d3fff700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f05dc10a740 0x7f05dc10b560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:51352/0 (socket says 172.21.15.90:51352) 2024-04-14T15:10:41.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.611+0000 7f05d3fff700 1 -- 172.21.15.90:0/2579348565 learned_addr learned my addr 172.21.15.90:0/2579348565 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:10:41.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.611+0000 7f05e0ecb700 1 --1- 172.21.15.90:0/2579348565 >> v1:172.21.15.90:6790/0 conn(0x7f05dc107890 0x7f05dc10ae50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:55106/0 (socket says 172.21.15.90:55106) 2024-04-14T15:10:41.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.611+0000 7f05d1ffb700 10 client.?.objecter ms_handle_connect 0x7f05dc10a740 2024-04-14T15:10:41.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.611+0000 7f05d1ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:10:41.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.611+0000 7f05d1ffb700 10 client.?.objecter ms_handle_connect 0x7f05dc107890 2024-04-14T15:10:41.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.611+0000 7f05d1ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:10:41.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.611+0000 7f05d1ffb700 1 -- 172.21.15.90:0/2579348565 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2053766771 0 0) 0x7f05dc1a8f30 con 0x7f05dc107890 2024-04-14T15:10:41.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.611+0000 7f05d1ffb700 1 -- 172.21.15.90:0/2579348565 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f05c40035b0 con 0x7f05dc107890 2024-04-14T15:10:41.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.611+0000 7f05d1ffb700 10 client.?.objecter ms_handle_connect 0x7f05dc10e310 2024-04-14T15:10:41.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.611+0000 7f05d1ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:10:41.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.611+0000 7f05d1ffb700 1 -- 172.21.15.90:0/2579348565 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1319762919 0 0) 0x7f05dc10de70 con 0x7f05dc10a740 2024-04-14T15:10:41.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.611+0000 7f05d1ffb700 1 -- 172.21.15.90:0/2579348565 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f05dc1a8f30 con 0x7f05dc10a740 2024-04-14T15:10:41.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.611+0000 7f05d1ffb700 1 -- 172.21.15.90:0/2579348565 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3888219545 0 0) 0x7f05dc10e080 con 0x7f05dc10e310 2024-04-14T15:10:41.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.611+0000 7f05d1ffb700 1 -- 172.21.15.90:0/2579348565 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f05dc10de70 con 0x7f05dc10e310 2024-04-14T15:10:41.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.612+0000 7f05d1ffb700 1 -- 172.21.15.90:0/2579348565 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3708914256 0 0) 0x7f05c40035b0 con 0x7f05dc107890 2024-04-14T15:10:41.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.612+0000 7f05d1ffb700 1 -- 172.21.15.90:0/2579348565 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f05dc10e080 con 0x7f05dc107890 2024-04-14T15:10:41.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.612+0000 7f05d1ffb700 1 -- 172.21.15.90:0/2579348565 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3538301012 0 0) 0x7f05dc1a8f30 con 0x7f05dc10a740 2024-04-14T15:10:41.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.612+0000 7f05d1ffb700 1 -- 172.21.15.90:0/2579348565 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f05c40035b0 con 0x7f05dc10a740 2024-04-14T15:10:41.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.612+0000 7f05d1ffb700 1 -- 172.21.15.90:0/2579348565 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f05cc004590 con 0x7f05dc10a740 2024-04-14T15:10:41.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.612+0000 7f05d1ffb700 1 -- 172.21.15.90:0/2579348565 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 175004671 0 0) 0x7f05dc10de70 con 0x7f05dc10e310 2024-04-14T15:10:41.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.612+0000 7f05d1ffb700 1 -- 172.21.15.90:0/2579348565 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f05dc1a8f30 con 0x7f05dc10e310 2024-04-14T15:10:41.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.612+0000 7f05d1ffb700 1 -- 172.21.15.90:0/2579348565 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f05d8001c70 con 0x7f05dc107890 2024-04-14T15:10:41.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.612+0000 7f05d1ffb700 1 -- 172.21.15.90:0/2579348565 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1951146081 0 0) 0x7f05c40035b0 con 0x7f05dc10a740 2024-04-14T15:10:41.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.612+0000 7f05d1ffb700 1 -- 172.21.15.90:0/2579348565 >> v1:172.21.15.90:6790/0 conn(0x7f05dc107890 legacy=0x7f05dc10ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:41.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.612+0000 7f05d1ffb700 1 -- 172.21.15.90:0/2579348565 >> v1:172.21.15.92:6789/0 conn(0x7f05dc10e310 legacy=0x7f05dc1a57f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:41.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.612+0000 7f05d1ffb700 1 -- 172.21.15.90:0/2579348565 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05dc1aa110 con 0x7f05dc10a740 2024-04-14T15:10:41.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.612+0000 7f05e312f700 1 -- 172.21.15.90:0/2579348565 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f05dc1a9160 con 0x7f05dc10a740 2024-04-14T15:10:41.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.612+0000 7f05d1ffb700 1 -- 172.21.15.90:0/2579348565 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f05cc001b30 con 0x7f05dc10a740 2024-04-14T15:10:41.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.612+0000 7f05d1ffb700 1 -- 172.21.15.90:0/2579348565 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f05cc005050 con 0x7f05dc10a740 2024-04-14T15:10:41.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.612+0000 7f05e312f700 10 client.14700.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:10:41.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.612+0000 7f05e312f700 1 -- 172.21.15.90:0/2579348565 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f05dc1a95c0 con 0x7f05dc10a740 2024-04-14T15:10:41.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.612+0000 7f05e312f700 1 librados: init done 2024-04-14T15:10:41.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.614+0000 7f05e312f700 10 client.14700.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:10:41.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.614+0000 7f05e312f700 15 client.14700.objecter _session_command_op_assign -1 1 2024-04-14T15:10:41.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.614+0000 7f05e312f700 20 client.14700.objecter _get_session osd=-1 returning homeless 2024-04-14T15:10:41.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.614+0000 7f05e312f700 10 client.14700.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:10:41.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.614+0000 7f05e312f700 1 -- 172.21.15.90:0/2579348565 --> v1:172.21.15.90:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f05dc0693d0 con 0x7f05dc10a740 2024-04-14T15:10:41.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.614+0000 7f05d1ffb700 1 -- 172.21.15.90:0/2579348565 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f05cc005050 con 0x7f05dc10a740 2024-04-14T15:10:41.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.614+0000 7f05d1ffb700 1 -- 172.21.15.90:0/2579348565 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(84..84 src has 12..84) ==== 6288+0+0 (unknown 1185262692 0 0) 0x7f05cc08a330 con 0x7f05dc10a740 2024-04-14T15:10:41.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.614+0000 7f05d1ffb700 10 client.14700.objecter ms_dispatch 0x7f05dc001550 osd_map(84..84 src has 12..84) 2024-04-14T15:10:41.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.614+0000 7f05d1ffb700 3 client.14700.objecter handle_osd_map got epochs [84,84] > 0 2024-04-14T15:10:41.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.614+0000 7f05d1ffb700 3 client.14700.objecter handle_osd_map decoding full epoch 84 2024-04-14T15:10:41.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.615+0000 7f05d1ffb700 10 client.14700.objecter checking command 1 2024-04-14T15:10:41.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.615+0000 7f05d1ffb700 20 client.14700.objecter _get_session s=0x7f05c407b580 osd=0 3 2024-04-14T15:10:41.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.615+0000 7f05d1ffb700 20 client.14700.objecter put_session s=0x7f05c407b580 osd=0 3 2024-04-14T15:10:41.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.615+0000 7f05d1ffb700 15 client.14700.objecter _session_command_op_remove -1 1 2024-04-14T15:10:41.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.615+0000 7f05d1ffb700 20 client.14700.objecter _get_session s=0x7f05c407b580 osd=0 3 2024-04-14T15:10:41.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.615+0000 7f05d1ffb700 20 client.14700.objecter get_session s=0x7f05c407b580 osd=0 3 2024-04-14T15:10:41.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.615+0000 7f05d1ffb700 15 client.14700.objecter _session_command_op_assign 0 1 2024-04-14T15:10:41.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.615+0000 7f05d1ffb700 20 client.14700.objecter put_session s=0x7f05c407b580 osd=0 4 2024-04-14T15:10:41.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.615+0000 7f05d1ffb700 10 client.14700.objecter _send_command 1 2024-04-14T15:10:41.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.615+0000 7f05d1ffb700 1 -- 172.21.15.90:0/2579348565 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f05c407f940 con 0x7f05c407bc30 2024-04-14T15:10:41.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.615+0000 7f05d1ffb700 20 client.14700.objecter dump_active .. 0 homeless 2024-04-14T15:10:41.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.615+0000 7f05d1ffb700 1 -- 172.21.15.90:0/2579348565 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_get_version_reply(handle=1 version=84) ==== 24+0+0 (unknown 1372807825 0 0) 0x7f05cc08a7a0 con 0x7f05dc10a740 2024-04-14T15:10:41.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.615+0000 7f05d1ffb700 10 client.14700.objecter ms_handle_connect 0x7f05c40713d0 2024-04-14T15:10:41.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.615+0000 7f05d1ffb700 10 client.14700.objecter ms_handle_connect 0x7f05c407bc30 2024-04-14T15:10:41.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.616+0000 7f05d1ffb700 1 -- 172.21.15.90:0/2579348565 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f05c407f940 con 0x7f05c407bc30 2024-04-14T15:10:41.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.616+0000 7f05d1ffb700 10 client.14700.objecter ms_dispatch 0x7f05dc001550 command_reply(tid 1: 0 ) 2024-04-14T15:10:41.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.616+0000 7f05d1ffb700 10 client.14700.objecter _finish_command 1 = system:0 2024-04-14T15:10:41.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.616+0000 7f05d1ffb700 20 client.14700.objecter put_session s=0x7f05c407b580 osd=0 4 2024-04-14T15:10:41.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.616+0000 7f05d1ffb700 15 client.14700.objecter _session_command_op_remove 0 1 2024-04-14T15:10:41.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.641+0000 7f05e312f700 10 client.14700.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-14T15:10:41.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.641+0000 7f05e312f700 15 client.14700.objecter _session_command_op_assign -1 2 2024-04-14T15:10:41.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.641+0000 7f05e312f700 20 client.14700.objecter _get_session s=0x7f05c407b580 osd=0 3 2024-04-14T15:10:41.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.641+0000 7f05e312f700 20 client.14700.objecter put_session s=0x7f05c407b580 osd=0 3 2024-04-14T15:10:41.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.641+0000 7f05e312f700 20 client.14700.objecter _get_session s=0x7f05c407b580 osd=0 3 2024-04-14T15:10:41.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.641+0000 7f05e312f700 15 client.14700.objecter _session_command_op_remove -1 2 2024-04-14T15:10:41.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.641+0000 7f05e312f700 20 client.14700.objecter get_session s=0x7f05c407b580 osd=0 3 2024-04-14T15:10:41.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.641+0000 7f05e312f700 15 client.14700.objecter _session_command_op_assign 0 2 2024-04-14T15:10:41.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.641+0000 7f05e312f700 20 client.14700.objecter put_session s=0x7f05c407b580 osd=0 4 2024-04-14T15:10:41.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.641+0000 7f05e312f700 10 client.14700.objecter _send_command 2 2024-04-14T15:10:41.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.641+0000 7f05e312f700 1 -- 172.21.15.90:0/2579348565 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f05dc10c1f0 con 0x7f05c407bc30 2024-04-14T15:10:41.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.642+0000 7f05d1ffb700 1 -- 172.21.15.90:0/2579348565 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 3385987527) 0x7f05dc10c1f0 con 0x7f05c407bc30 2024-04-14T15:10:41.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.642+0000 7f05d1ffb700 10 client.14700.objecter ms_dispatch 0x7f05dc001550 command_reply(tid 2: 0 ) 2024-04-14T15:10:41.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.642+0000 7f05d1ffb700 10 client.14700.objecter _finish_command 2 = system:0 2024-04-14T15:10:41.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.642+0000 7f05d1ffb700 20 client.14700.objecter put_session s=0x7f05c407b580 osd=0 4 2024-04-14T15:10:41.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.642+0000 7f05d1ffb700 15 client.14700.objecter _session_command_op_remove 0 2 2024-04-14T15:10:41.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.643+0000 7f05e312f700 10 librados: watch_flush enter 2024-04-14T15:10:41.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.643+0000 7f05e312f700 10 librados: watch_flush exit 2024-04-14T15:10:41.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.643+0000 7f05e312f700 10 client.14700.objecter close_session for osd.0 2024-04-14T15:10:41.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.643+0000 7f05e312f700 1 -- 172.21.15.90:0/2579348565 >> v1:172.21.15.90:6801/3761806885 conn(0x7f05c407bc30 legacy=0x7f05c407e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:41.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.643+0000 7f05e312f700 20 client.14700.objecter put_session s=0x7f05c407b580 osd=0 1 2024-04-14T15:10:41.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.643+0000 7f05e312f700 20 client.14700.objecter shutdown clearing up homeless session... 2024-04-14T15:10:41.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.643+0000 7f05e312f700 10 client.14700.objecter successfully canceled tick 2024-04-14T15:10:41.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.643+0000 7f05e312f700 1 -- 172.21.15.90:0/2579348565 >> v1:172.21.15.90:6800/420935427 conn(0x7f05c40713d0 legacy=0x7f05c4073890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:41.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.643+0000 7f05e312f700 1 -- 172.21.15.90:0/2579348565 >> v1:172.21.15.90:6789/0 conn(0x7f05dc10a740 legacy=0x7f05dc10b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:41.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.643+0000 7f05e16cc700 1 -- 172.21.15.90:0/2579348565 reap_dead start 2024-04-14T15:10:41.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.644+0000 7f05e312f700 1 -- 172.21.15.90:0/2579348565 shutdown_connections 2024-04-14T15:10:41.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.644+0000 7f05e312f700 1 -- 172.21.15.90:0/2579348565 >> 172.21.15.90:0/2579348565 conn(0x7f05dc0fd290 msgr2=0x7f05dc1006f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:10:41.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.644+0000 7f05e312f700 1 -- 172.21.15.90:0/2579348565 shutdown_connections 2024-04-14T15:10:41.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.644+0000 7f05e312f700 1 -- 172.21.15.90:0/2579348565 wait complete. 2024-04-14T15:10:41.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:41.644+0000 7f05e312f700 1 librados: shutdown 2024-04-14T15:10:41.941 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:41 smithi090 ceph-mon[41670]: osdmap e84: 8 total, 8 up, 8 in 2024-04-14T15:10:41.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:41 smithi090 ceph-mon[30399]: osdmap e84: 8 total, 8 up, 8 in 2024-04-14T15:10:42.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:41 smithi092 ceph-mon[33966]: osdmap e84: 8 total, 8 up, 8 in 2024-04-14T15:10:42.134 INFO:teuthology.orchestra.run.smithi090.stdout:77309411369 2024-04-14T15:10:42.135 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-14T15:10:42.173 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:10:42.775 INFO:teuthology.orchestra.run.smithi090.stdout:176093659162 2024-04-14T15:10:42.775 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-14T15:10:43.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:42 smithi092 ceph-mon[33966]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 754 B/s rd, 0 op/s 2024-04-14T15:10:43.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:42 smithi090 ceph-mon[30399]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 754 B/s rd, 0 op/s 2024-04-14T15:10:43.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:42 smithi090 ceph-mon[41670]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 754 B/s rd, 0 op/s 2024-04-14T15:10:43.415 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:10:44.053 INFO:teuthology.orchestra.run.smithi090.stdout:34359738419 2024-04-14T15:10:44.053 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-14T15:10:44.077 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:10:43 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:10:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:10:44.078 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:43 smithi090 ceph-mon[41670]: 2.1 scrub starts 2024-04-14T15:10:44.078 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:43 smithi090 ceph-mon[41670]: 2.1 scrub ok 2024-04-14T15:10:44.078 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:43 smithi090 ceph-mon[41670]: 2.1d deep-scrub starts 2024-04-14T15:10:44.078 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:43 smithi090 ceph-mon[41670]: 2.1d deep-scrub ok 2024-04-14T15:10:44.078 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:43 smithi090 ceph-mon[30399]: 2.1 scrub starts 2024-04-14T15:10:44.078 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:43 smithi090 ceph-mon[30399]: 2.1 scrub ok 2024-04-14T15:10:44.078 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:43 smithi090 ceph-mon[30399]: 2.1d deep-scrub starts 2024-04-14T15:10:44.078 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:43 smithi090 ceph-mon[30399]: 2.1d deep-scrub ok 2024-04-14T15:10:44.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:43 smithi092 ceph-mon[33966]: 2.1 scrub starts 2024-04-14T15:10:44.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:43 smithi092 ceph-mon[33966]: 2.1 scrub ok 2024-04-14T15:10:44.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:43 smithi092 ceph-mon[33966]: 2.1d deep-scrub starts 2024-04-14T15:10:44.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:43 smithi092 ceph-mon[33966]: 2.1d deep-scrub ok 2024-04-14T15:10:44.404 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:10:45.089 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:44 smithi090 ceph-mon[41670]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 0 op/s 2024-04-14T15:10:45.089 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:44 smithi090 ceph-mon[41670]: 2.10 scrub starts 2024-04-14T15:10:45.090 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:44 smithi090 ceph-mon[41670]: 2.10 scrub ok 2024-04-14T15:10:45.090 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:44 smithi090 ceph-mon[41670]: 2.3 scrub starts 2024-04-14T15:10:45.090 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:44 smithi090 ceph-mon[41670]: 2.3 scrub ok 2024-04-14T15:10:45.090 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:44 smithi090 ceph-mon[30399]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 0 op/s 2024-04-14T15:10:45.090 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:44 smithi090 ceph-mon[30399]: 2.10 scrub starts 2024-04-14T15:10:45.090 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:44 smithi090 ceph-mon[30399]: 2.10 scrub ok 2024-04-14T15:10:45.090 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:44 smithi090 ceph-mon[30399]: 2.3 scrub starts 2024-04-14T15:10:45.090 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:44 smithi090 ceph-mon[30399]: 2.3 scrub ok 2024-04-14T15:10:45.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:44 smithi092 ceph-mon[33966]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 0 op/s 2024-04-14T15:10:45.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:44 smithi092 ceph-mon[33966]: 2.10 scrub starts 2024-04-14T15:10:45.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:44 smithi092 ceph-mon[33966]: 2.10 scrub ok 2024-04-14T15:10:45.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:44 smithi092 ceph-mon[33966]: 2.3 scrub starts 2024-04-14T15:10:45.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:44 smithi092 ceph-mon[33966]: 2.3 scrub ok 2024-04-14T15:10:45.123 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:10:44 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:10:45.941 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:45 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:10:45.941 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:45 smithi090 ceph-mon[41670]: osdmap e85: 8 total, 8 up, 8 in 2024-04-14T15:10:45.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:45 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:10:45.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:45 smithi090 ceph-mon[30399]: osdmap e85: 8 total, 8 up, 8 in 2024-04-14T15:10:46.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:45 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:10:46.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:45 smithi092 ceph-mon[33966]: osdmap e85: 8 total, 8 up, 8 in 2024-04-14T15:10:47.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:46 smithi092 ceph-mon[33966]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:10:47.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:46 smithi092 ceph-mon[33966]: 2.c scrub starts 2024-04-14T15:10:47.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:46 smithi092 ceph-mon[33966]: 2.c scrub ok 2024-04-14T15:10:47.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:46 smithi092 ceph-mon[33966]: 2.a scrub starts 2024-04-14T15:10:47.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:46 smithi092 ceph-mon[33966]: 2.a scrub ok 2024-04-14T15:10:47.158 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:46 smithi090 ceph-mon[41670]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:10:47.158 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:46 smithi090 ceph-mon[41670]: 2.c scrub starts 2024-04-14T15:10:47.158 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:46 smithi090 ceph-mon[41670]: 2.c scrub ok 2024-04-14T15:10:47.158 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:46 smithi090 ceph-mon[41670]: 2.a scrub starts 2024-04-14T15:10:47.158 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:46 smithi090 ceph-mon[41670]: 2.a scrub ok 2024-04-14T15:10:47.158 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:46 smithi090 ceph-mon[30399]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:10:47.158 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:46 smithi090 ceph-mon[30399]: 2.c scrub starts 2024-04-14T15:10:47.158 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:46 smithi090 ceph-mon[30399]: 2.c scrub ok 2024-04-14T15:10:47.158 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:46 smithi090 ceph-mon[30399]: 2.a scrub starts 2024-04-14T15:10:47.158 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:46 smithi090 ceph-mon[30399]: 2.a scrub ok 2024-04-14T15:10:48.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:47 smithi092 ceph-mon[33966]: 4.e deep-scrub starts 2024-04-14T15:10:48.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:47 smithi092 ceph-mon[33966]: 4.e deep-scrub ok 2024-04-14T15:10:48.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:47 smithi090 ceph-mon[30399]: 4.e deep-scrub starts 2024-04-14T15:10:48.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:47 smithi090 ceph-mon[30399]: 4.e deep-scrub ok 2024-04-14T15:10:48.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:47 smithi090 ceph-mon[41670]: 4.e deep-scrub starts 2024-04-14T15:10:48.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:47 smithi090 ceph-mon[41670]: 4.e deep-scrub ok 2024-04-14T15:10:48.727 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:10:49.016 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:48 smithi090 ceph-mon[30399]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-14T15:10:49.016 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:48 smithi090 ceph-mon[30399]: 4.3 scrub starts 2024-04-14T15:10:49.016 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:48 smithi090 ceph-mon[30399]: 4.3 scrub ok 2024-04-14T15:10:49.016 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:48 smithi090 ceph-mon[30399]: 5.1c scrub starts 2024-04-14T15:10:49.016 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:48 smithi090 ceph-mon[30399]: 5.1c scrub ok 2024-04-14T15:10:49.016 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:48 smithi090 ceph-mon[41670]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-14T15:10:49.016 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:48 smithi090 ceph-mon[41670]: 4.3 scrub starts 2024-04-14T15:10:49.016 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:48 smithi090 ceph-mon[41670]: 4.3 scrub ok 2024-04-14T15:10:49.016 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:48 smithi090 ceph-mon[41670]: 5.1c scrub starts 2024-04-14T15:10:49.017 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:48 smithi090 ceph-mon[41670]: 5.1c scrub ok 2024-04-14T15:10:49.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:48 smithi092 ceph-mon[33966]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-14T15:10:49.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:48 smithi092 ceph-mon[33966]: 4.3 scrub starts 2024-04-14T15:10:49.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:48 smithi092 ceph-mon[33966]: 4.3 scrub ok 2024-04-14T15:10:49.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:48 smithi092 ceph-mon[33966]: 5.1c scrub starts 2024-04-14T15:10:49.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:48 smithi092 ceph-mon[33966]: 5.1c scrub ok 2024-04-14T15:10:49.923 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:49 smithi090 ceph-mon[41670]: 2.8 scrub starts 2024-04-14T15:10:49.923 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:49 smithi090 ceph-mon[41670]: 2.8 scrub ok 2024-04-14T15:10:49.923 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:49 smithi090 ceph-mon[30399]: 2.8 scrub starts 2024-04-14T15:10:49.923 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:49 smithi090 ceph-mon[30399]: 2.8 scrub ok 2024-04-14T15:10:50.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:49 smithi092 ceph-mon[33966]: 2.8 scrub starts 2024-04-14T15:10:50.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:49 smithi092 ceph-mon[33966]: 2.8 scrub ok 2024-04-14T15:10:50.941 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:50 smithi090 ceph-mon[30399]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:10:50.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:50 smithi090 ceph-mon[30399]: 2.0 scrub starts 2024-04-14T15:10:50.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:50 smithi090 ceph-mon[30399]: 2.0 scrub ok 2024-04-14T15:10:50.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:50 smithi090 ceph-mon[30399]: 5.11 scrub starts 2024-04-14T15:10:50.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:50 smithi090 ceph-mon[30399]: 5.11 scrub ok 2024-04-14T15:10:50.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:50 smithi090 ceph-mon[41670]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:10:50.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:50 smithi090 ceph-mon[41670]: 2.0 scrub starts 2024-04-14T15:10:50.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:50 smithi090 ceph-mon[41670]: 2.0 scrub ok 2024-04-14T15:10:50.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:50 smithi090 ceph-mon[41670]: 5.11 scrub starts 2024-04-14T15:10:50.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:50 smithi090 ceph-mon[41670]: 5.11 scrub ok 2024-04-14T15:10:51.301 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:50 smithi092 ceph-mon[33966]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:10:51.301 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:50 smithi092 ceph-mon[33966]: 2.0 scrub starts 2024-04-14T15:10:51.301 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:50 smithi092 ceph-mon[33966]: 2.0 scrub ok 2024-04-14T15:10:51.301 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:50 smithi092 ceph-mon[33966]: 5.11 scrub starts 2024-04-14T15:10:51.301 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:50 smithi092 ceph-mon[33966]: 5.11 scrub ok 2024-04-14T15:10:51.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.506+0000 7fed3cd25700 1 -- 172.21.15.90:0/3317268347 >> v1:172.21.15.90:6789/0 conn(0x7fed3810b930 legacy=0x7fed3810dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:51.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.506+0000 7fed3cd25700 1 -- 172.21.15.90:0/3317268347 shutdown_connections 2024-04-14T15:10:51.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.507+0000 7fed3cd25700 1 -- 172.21.15.90:0/3317268347 >> 172.21.15.90:0/3317268347 conn(0x7fed380fd290 msgr2=0x7fed380ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:10:51.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.507+0000 7fed3cd25700 1 -- 172.21.15.90:0/3317268347 shutdown_connections 2024-04-14T15:10:51.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.507+0000 7fed3cd25700 1 -- 172.21.15.90:0/3317268347 wait complete. 2024-04-14T15:10:51.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.507+0000 7fed3cd25700 1 librados: starting msgr at 2024-04-14T15:10:51.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.507+0000 7fed3cd25700 1 librados: starting objecter 2024-04-14T15:10:51.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.507+0000 7fed3cd25700 1 Processor -- start 2024-04-14T15:10:51.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.508+0000 7fed3cd25700 1 -- start start 2024-04-14T15:10:51.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.508+0000 7fed3cd25700 1 librados: setting wanted keys 2024-04-14T15:10:51.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.508+0000 7fed3cd25700 1 librados: calling monclient init 2024-04-14T15:10:51.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.508+0000 7fed3cd25700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fed381a9cb0 con 0x7fed38103a40 2024-04-14T15:10:51.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.508+0000 7fed3cd25700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fed381aaed0 con 0x7fed3810f4d0 2024-04-14T15:10:51.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.508+0000 7fed3cd25700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fed381ac110 con 0x7fed3810b930 2024-04-14T15:10:51.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.508+0000 7fed3659c700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fed38103a40 0x7fed3810b0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:45244/0 (socket says 172.21.15.90:45244) 2024-04-14T15:10:51.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.508+0000 7fed3659c700 1 -- 172.21.15.90:0/257481144 learned_addr learned my addr 172.21.15.90:0/257481144 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:10:51.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.508+0000 7fed277fe700 10 client.?.objecter ms_handle_connect 0x7fed38103a40 2024-04-14T15:10:51.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.508+0000 7fed277fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:10:51.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.508+0000 7fed277fe700 10 client.?.objecter ms_handle_connect 0x7fed3810b930 2024-04-14T15:10:51.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.508+0000 7fed277fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:10:51.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.508+0000 7fed277fe700 10 client.?.objecter ms_handle_connect 0x7fed3810f4d0 2024-04-14T15:10:51.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.508+0000 7fed277fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:10:51.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.508+0000 7fed277fe700 1 -- 172.21.15.90:0/257481144 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1453913357 0 0) 0x7fed381aaed0 con 0x7fed3810f4d0 2024-04-14T15:10:51.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.508+0000 7fed277fe700 1 -- 172.21.15.90:0/257481144 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fed200031f0 con 0x7fed3810f4d0 2024-04-14T15:10:51.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.508+0000 7fed277fe700 1 -- 172.21.15.90:0/257481144 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2635480624 0 0) 0x7fed381a9cb0 con 0x7fed38103a40 2024-04-14T15:10:51.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.508+0000 7fed277fe700 1 -- 172.21.15.90:0/257481144 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fed381aaed0 con 0x7fed38103a40 2024-04-14T15:10:51.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.508+0000 7fed277fe700 1 -- 172.21.15.90:0/257481144 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 953652183 0 0) 0x7fed381ac110 con 0x7fed3810b930 2024-04-14T15:10:51.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.508+0000 7fed277fe700 1 -- 172.21.15.90:0/257481144 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fed381a9cb0 con 0x7fed3810b930 2024-04-14T15:10:51.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.508+0000 7fed277fe700 1 -- 172.21.15.90:0/257481144 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2528110692 0 0) 0x7fed200031f0 con 0x7fed3810f4d0 2024-04-14T15:10:51.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.509+0000 7fed277fe700 1 -- 172.21.15.90:0/257481144 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fed381ac110 con 0x7fed3810f4d0 2024-04-14T15:10:51.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.509+0000 7fed277fe700 1 -- 172.21.15.90:0/257481144 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fed30001b30 con 0x7fed3810f4d0 2024-04-14T15:10:51.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.509+0000 7fed277fe700 1 -- 172.21.15.90:0/257481144 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1591212271 0 0) 0x7fed381a9cb0 con 0x7fed3810b930 2024-04-14T15:10:51.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.509+0000 7fed277fe700 1 -- 172.21.15.90:0/257481144 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fed200031f0 con 0x7fed3810b930 2024-04-14T15:10:51.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.509+0000 7fed277fe700 1 -- 172.21.15.90:0/257481144 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fed28001880 con 0x7fed3810b930 2024-04-14T15:10:51.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.509+0000 7fed277fe700 1 -- 172.21.15.90:0/257481144 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1107852499 0 0) 0x7fed381aaed0 con 0x7fed38103a40 2024-04-14T15:10:51.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.509+0000 7fed277fe700 1 -- 172.21.15.90:0/257481144 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fed381a9cb0 con 0x7fed38103a40 2024-04-14T15:10:51.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.509+0000 7fed277fe700 1 -- 172.21.15.90:0/257481144 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fed2c001850 con 0x7fed38103a40 2024-04-14T15:10:51.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.509+0000 7fed277fe700 1 -- 172.21.15.90:0/257481144 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1388326045 0 0) 0x7fed381ac110 con 0x7fed3810f4d0 2024-04-14T15:10:51.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.509+0000 7fed277fe700 1 -- 172.21.15.90:0/257481144 >> v1:172.21.15.90:6790/0 conn(0x7fed3810b930 legacy=0x7fed381a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:51.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.509+0000 7fed277fe700 1 -- 172.21.15.90:0/257481144 >> v1:172.21.15.90:6789/0 conn(0x7fed38103a40 legacy=0x7fed3810b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:51.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.509+0000 7fed277fe700 1 -- 172.21.15.90:0/257481144 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed381ad350 con 0x7fed3810f4d0 2024-04-14T15:10:51.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.509+0000 7fed3cd25700 1 -- 172.21.15.90:0/257481144 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fed381ab100 con 0x7fed3810f4d0 2024-04-14T15:10:51.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.509+0000 7fed3cd25700 10 client.24598.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:10:51.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.509+0000 7fed3cd25700 1 -- 172.21.15.90:0/257481144 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fed381ab5c0 con 0x7fed3810f4d0 2024-04-14T15:10:51.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.509+0000 7fed3cd25700 1 librados: init done 2024-04-14T15:10:51.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.509+0000 7fed277fe700 1 -- 172.21.15.90:0/257481144 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fed30003ea0 con 0x7fed3810f4d0 2024-04-14T15:10:51.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.509+0000 7fed277fe700 1 -- 172.21.15.90:0/257481144 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fed30004db0 con 0x7fed3810f4d0 2024-04-14T15:10:51.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.510+0000 7fed3cd25700 1 -- 172.21.15.90:0/257481144 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed18005120 con 0x7fed3810f4d0 2024-04-14T15:10:51.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.510+0000 7fed277fe700 1 -- 172.21.15.90:0/257481144 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fed3001b960 con 0x7fed3810f4d0 2024-04-14T15:10:51.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.511+0000 7fed277fe700 1 -- 172.21.15.90:0/257481144 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(85..85 src has 12..85) ==== 6288+0+0 (unknown 2690234135 0 0) 0x7fed30089470 con 0x7fed3810f4d0 2024-04-14T15:10:51.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.511+0000 7fed277fe700 10 client.24598.objecter ms_dispatch 0x7fed38001550 osd_map(85..85 src has 12..85) 2024-04-14T15:10:51.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.511+0000 7fed277fe700 3 client.24598.objecter handle_osd_map got epochs [85,85] > 0 2024-04-14T15:10:51.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.511+0000 7fed277fe700 3 client.24598.objecter handle_osd_map decoding full epoch 85 2024-04-14T15:10:51.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.511+0000 7fed277fe700 20 client.24598.objecter dump_active .. 0 homeless 2024-04-14T15:10:51.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.511+0000 7fed277fe700 10 client.24598.objecter ms_handle_connect 0x7fed20071440 2024-04-14T15:10:51.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.515+0000 7fed277fe700 1 -- 172.21.15.90:0/257481144 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fed300565e0 con 0x7fed3810f4d0 2024-04-14T15:10:51.582 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:10:51 smithi090 systemd[1]: Starting Ceph alertmanager.a for 0e288098-fa70-11ee-bc8e-c7b262605968... 2024-04-14T15:10:51.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.720+0000 7fed3cd25700 1 -- 172.21.15.90:0/257481144 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fed18005410 con 0x7fed3810f4d0 2024-04-14T15:10:51.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.720+0000 7fed277fe700 1 -- 172.21.15.90:0/257481144 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (unknown 832126871 0 1580136542) 0x7fed3005a290 con 0x7fed3810f4d0 2024-04-14T15:10:51.721 INFO:teuthology.orchestra.run.smithi090.stdout:55834574896 2024-04-14T15:10:51.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.722+0000 7fed3cd25700 10 librados: watch_flush enter 2024-04-14T15:10:51.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.722+0000 7fed3cd25700 10 librados: watch_flush exit 2024-04-14T15:10:51.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.722+0000 7fed3cd25700 20 client.24598.objecter shutdown clearing up homeless session... 2024-04-14T15:10:51.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.722+0000 7fed3cd25700 10 client.24598.objecter successfully canceled tick 2024-04-14T15:10:51.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.722+0000 7fed3cd25700 1 -- 172.21.15.90:0/257481144 >> v1:172.21.15.90:6800/420935427 conn(0x7fed20071440 legacy=0x7fed20073900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:51.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.722+0000 7fed3cd25700 1 -- 172.21.15.90:0/257481144 >> v1:172.21.15.92:6789/0 conn(0x7fed3810f4d0 legacy=0x7fed381a8430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:51.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.723+0000 7fed3cd25700 1 -- 172.21.15.90:0/257481144 shutdown_connections 2024-04-14T15:10:51.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.723+0000 7fed3cd25700 1 -- 172.21.15.90:0/257481144 >> 172.21.15.90:0/257481144 conn(0x7fed380fd290 msgr2=0x7fed380ff680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:10:51.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.723+0000 7fed3cd25700 1 -- 172.21.15.90:0/257481144 shutdown_connections 2024-04-14T15:10:51.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.723+0000 7fed3cd25700 1 -- 172.21.15.90:0/257481144 wait complete. 2024-04-14T15:10:51.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:51.723+0000 7fed3cd25700 1 librados: shutdown 2024-04-14T15:10:52.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:51 smithi090 ceph-mon[30399]: 5.a scrub starts 2024-04-14T15:10:52.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:51 smithi090 ceph-mon[30399]: 5.a scrub ok 2024-04-14T15:10:52.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:51 smithi090 ceph-mon[30399]: 4.b scrub starts 2024-04-14T15:10:52.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:51 smithi090 ceph-mon[30399]: 4.b scrub ok 2024-04-14T15:10:52.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:51 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/257481144' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-14T15:10:52.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:51 smithi090 ceph-mon[41670]: 5.a scrub starts 2024-04-14T15:10:52.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:51 smithi090 ceph-mon[41670]: 5.a scrub ok 2024-04-14T15:10:52.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:51 smithi090 ceph-mon[41670]: 4.b scrub starts 2024-04-14T15:10:52.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:51 smithi090 ceph-mon[41670]: 4.b scrub ok 2024-04-14T15:10:52.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:51 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/257481144' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-14T15:10:52.244 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:51 smithi092 ceph-mon[33966]: 5.a scrub starts 2024-04-14T15:10:52.244 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:51 smithi092 ceph-mon[33966]: 5.a scrub ok 2024-04-14T15:10:52.244 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:51 smithi092 ceph-mon[33966]: 4.b scrub starts 2024-04-14T15:10:52.244 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:51 smithi092 ceph-mon[33966]: 4.b scrub ok 2024-04-14T15:10:52.244 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:51 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/257481144' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-14T15:10:53.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:52 smithi090 ceph-mon[41670]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:10:53.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:52 smithi090 ceph-mon[41670]: 4.6 scrub starts 2024-04-14T15:10:53.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:52 smithi090 ceph-mon[41670]: 4.6 scrub ok 2024-04-14T15:10:53.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:52 smithi090 ceph-mon[41670]: 5.9 deep-scrub starts 2024-04-14T15:10:53.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:52 smithi090 ceph-mon[41670]: 5.9 deep-scrub ok 2024-04-14T15:10:53.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:52 smithi090 ceph-mon[30399]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:10:53.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:52 smithi090 ceph-mon[30399]: 4.6 scrub starts 2024-04-14T15:10:53.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:52 smithi090 ceph-mon[30399]: 4.6 scrub ok 2024-04-14T15:10:53.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:52 smithi090 ceph-mon[30399]: 5.9 deep-scrub starts 2024-04-14T15:10:53.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:52 smithi090 ceph-mon[30399]: 5.9 deep-scrub ok 2024-04-14T15:10:53.333 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:10:53.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:52 smithi092 ceph-mon[33966]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:10:53.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:52 smithi092 ceph-mon[33966]: 4.6 scrub starts 2024-04-14T15:10:53.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:52 smithi092 ceph-mon[33966]: 4.6 scrub ok 2024-04-14T15:10:53.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:52 smithi092 ceph-mon[33966]: 5.9 deep-scrub starts 2024-04-14T15:10:53.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:52 smithi092 ceph-mon[33966]: 5.9 deep-scrub ok 2024-04-14T15:10:53.428 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:10:53.429 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:10:53.429 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:10:53.888 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:53 smithi090 ceph-mon[30399]: 2.18 scrub starts 2024-04-14T15:10:53.888 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:53 smithi090 ceph-mon[30399]: 2.18 scrub ok 2024-04-14T15:10:53.888 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:53 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:10:54.182 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:53 smithi090 ceph-mon[30399]: 4.9 scrub starts 2024-04-14T15:10:54.182 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:53 smithi090 ceph-mon[30399]: 4.9 scrub ok 2024-04-14T15:10:54.183 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:10:53 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:10:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:10:54.183 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:53 smithi090 ceph-mon[41670]: 2.18 scrub starts 2024-04-14T15:10:54.183 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:53 smithi090 ceph-mon[41670]: 2.18 scrub ok 2024-04-14T15:10:54.183 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:53 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:10:54.183 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:53 smithi090 ceph-mon[41670]: 4.9 scrub starts 2024-04-14T15:10:54.183 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:53 smithi090 ceph-mon[41670]: 4.9 scrub ok 2024-04-14T15:10:54.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.072+0000 7ff6df1bc700 1 -- 172.21.15.90:0/919046079 >> v1:172.21.15.90:6789/0 conn(0x7ff6d810e1e0 legacy=0x7ff6d81106a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:54.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.072+0000 7ff6df1bc700 1 -- 172.21.15.90:0/919046079 shutdown_connections 2024-04-14T15:10:54.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.072+0000 7ff6df1bc700 1 -- 172.21.15.90:0/919046079 >> 172.21.15.90:0/919046079 conn(0x7ff6d80fd290 msgr2=0x7ff6d80ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:10:54.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.072+0000 7ff6df1bc700 1 -- 172.21.15.90:0/919046079 shutdown_connections 2024-04-14T15:10:54.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.072+0000 7ff6df1bc700 1 -- 172.21.15.90:0/919046079 wait complete. 2024-04-14T15:10:54.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.073+0000 7ff6df1bc700 1 librados: starting msgr at 2024-04-14T15:10:54.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.073+0000 7ff6df1bc700 1 librados: starting objecter 2024-04-14T15:10:54.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.073+0000 7ff6df1bc700 1 Processor -- start 2024-04-14T15:10:54.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.073+0000 7ff6df1bc700 1 -- start start 2024-04-14T15:10:54.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.073+0000 7ff6df1bc700 1 librados: setting wanted keys 2024-04-14T15:10:54.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.073+0000 7ff6df1bc700 1 librados: calling monclient init 2024-04-14T15:10:54.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.073+0000 7ff6df1bc700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff6d81a7df0 con 0x7ff6d810a610 2024-04-14T15:10:54.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.073+0000 7ff6df1bc700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff6d81a8fd0 con 0x7ff6d810e1e0 2024-04-14T15:10:54.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.073+0000 7ff6df1bc700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff6d81aa1b0 con 0x7ff6d8107760 2024-04-14T15:10:54.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.073+0000 7ff6dcf58700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7ff6d8107760 0x7ff6d806a460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:55290/0 (socket says 172.21.15.90:55290) 2024-04-14T15:10:54.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.073+0000 7ff6d7fff700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7ff6d810a610 0x7ff6d806ab70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:45258/0 (socket says 172.21.15.90:45258) 2024-04-14T15:10:54.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.073+0000 7ff6dcf58700 1 -- 172.21.15.90:0/790404968 learned_addr learned my addr 172.21.15.90:0/790404968 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:10:54.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.073+0000 7ff6d5ffb700 10 client.?.objecter ms_handle_connect 0x7ff6d810a610 2024-04-14T15:10:54.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.073+0000 7ff6d5ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:10:54.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.073+0000 7ff6d5ffb700 10 client.?.objecter ms_handle_connect 0x7ff6d8107760 2024-04-14T15:10:54.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.073+0000 7ff6d5ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:10:54.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.073+0000 7ff6d5ffb700 10 client.?.objecter ms_handle_connect 0x7ff6d810e1e0 2024-04-14T15:10:54.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.073+0000 7ff6d5ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:10:54.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.073+0000 7ff6d5ffb700 1 -- 172.21.15.90:0/790404968 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 564304341 0 0) 0x7ff6d81a7df0 con 0x7ff6d810a610 2024-04-14T15:10:54.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.073+0000 7ff6d5ffb700 1 -- 172.21.15.90:0/790404968 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff6c80035b0 con 0x7ff6d810a610 2024-04-14T15:10:54.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.073+0000 7ff6d5ffb700 1 -- 172.21.15.90:0/790404968 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3040577262 0 0) 0x7ff6d81aa1b0 con 0x7ff6d8107760 2024-04-14T15:10:54.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.073+0000 7ff6d5ffb700 1 -- 172.21.15.90:0/790404968 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff6d81a7df0 con 0x7ff6d8107760 2024-04-14T15:10:54.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.074+0000 7ff6d5ffb700 1 -- 172.21.15.90:0/790404968 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3669818198 0 0) 0x7ff6d81a8fd0 con 0x7ff6d810e1e0 2024-04-14T15:10:54.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.074+0000 7ff6d5ffb700 1 -- 172.21.15.90:0/790404968 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff6d81aa1b0 con 0x7ff6d810e1e0 2024-04-14T15:10:54.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.074+0000 7ff6d5ffb700 1 -- 172.21.15.90:0/790404968 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1720459762 0 0) 0x7ff6c80035b0 con 0x7ff6d810a610 2024-04-14T15:10:54.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.074+0000 7ff6d5ffb700 1 -- 172.21.15.90:0/790404968 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6d81a8fd0 con 0x7ff6d810a610 2024-04-14T15:10:54.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.074+0000 7ff6d5ffb700 1 -- 172.21.15.90:0/790404968 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7ff6c40018b0 con 0x7ff6d810a610 2024-04-14T15:10:54.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.074+0000 7ff6d5ffb700 1 -- 172.21.15.90:0/790404968 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 592974038 0 0) 0x7ff6d81a7df0 con 0x7ff6d8107760 2024-04-14T15:10:54.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.074+0000 7ff6d5ffb700 1 -- 172.21.15.90:0/790404968 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6c80035b0 con 0x7ff6d8107760 2024-04-14T15:10:54.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.074+0000 7ff6d5ffb700 1 -- 172.21.15.90:0/790404968 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7ff6cc0018d0 con 0x7ff6d8107760 2024-04-14T15:10:54.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.074+0000 7ff6d5ffb700 1 -- 172.21.15.90:0/790404968 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1415645851 0 0) 0x7ff6d81aa1b0 con 0x7ff6d810e1e0 2024-04-14T15:10:54.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.074+0000 7ff6d5ffb700 1 -- 172.21.15.90:0/790404968 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6d81a7df0 con 0x7ff6d810e1e0 2024-04-14T15:10:54.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.074+0000 7ff6d5ffb700 1 -- 172.21.15.90:0/790404968 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2428513098 0 0) 0x7ff6d81a8fd0 con 0x7ff6d810a610 2024-04-14T15:10:54.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.074+0000 7ff6d5ffb700 1 -- 172.21.15.90:0/790404968 >> v1:172.21.15.90:6790/0 conn(0x7ff6d8107760 legacy=0x7ff6d806a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:54.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.074+0000 7ff6d5ffb700 1 -- 172.21.15.90:0/790404968 >> v1:172.21.15.92:6789/0 conn(0x7ff6d810e1e0 legacy=0x7ff6d81a66d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:54.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.074+0000 7ff6d5ffb700 1 -- 172.21.15.90:0/790404968 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6d81ab390 con 0x7ff6d810a610 2024-04-14T15:10:54.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.074+0000 7ff6df1bc700 1 -- 172.21.15.90:0/790404968 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff6d81aa3e0 con 0x7ff6d810a610 2024-04-14T15:10:54.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.074+0000 7ff6d5ffb700 1 -- 172.21.15.90:0/790404968 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7ff6c4003b40 con 0x7ff6d810a610 2024-04-14T15:10:54.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.074+0000 7ff6d5ffb700 1 -- 172.21.15.90:0/790404968 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7ff6c4005dc0 con 0x7ff6d810a610 2024-04-14T15:10:54.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.074+0000 7ff6df1bc700 10 client.14712.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:10:54.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.074+0000 7ff6df1bc700 1 -- 172.21.15.90:0/790404968 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff6d81aa8a0 con 0x7ff6d810a610 2024-04-14T15:10:54.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.074+0000 7ff6df1bc700 1 librados: init done 2024-04-14T15:10:54.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.075+0000 7ff6df1bc700 1 -- 172.21.15.90:0/790404968 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6d8102ea0 con 0x7ff6d810a610 2024-04-14T15:10:54.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.075+0000 7ff6d5ffb700 1 -- 172.21.15.90:0/790404968 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7ff6c401ca10 con 0x7ff6d810a610 2024-04-14T15:10:54.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.076+0000 7ff6d5ffb700 1 -- 172.21.15.90:0/790404968 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(85..85 src has 12..85) ==== 6288+0+0 (unknown 2690234135 0 0) 0x7ff6c408a300 con 0x7ff6d810a610 2024-04-14T15:10:54.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.076+0000 7ff6d5ffb700 10 client.14712.objecter ms_dispatch 0x7ff6d8001550 osd_map(85..85 src has 12..85) 2024-04-14T15:10:54.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.076+0000 7ff6d5ffb700 3 client.14712.objecter handle_osd_map got epochs [85,85] > 0 2024-04-14T15:10:54.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.076+0000 7ff6d5ffb700 3 client.14712.objecter handle_osd_map decoding full epoch 85 2024-04-14T15:10:54.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.076+0000 7ff6d5ffb700 20 client.14712.objecter dump_active .. 0 homeless 2024-04-14T15:10:54.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.076+0000 7ff6d5ffb700 10 client.14712.objecter ms_handle_connect 0x7ff6c8071440 2024-04-14T15:10:54.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.082+0000 7ff6d5ffb700 1 -- 172.21.15.90:0/790404968 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7ff6c40574d0 con 0x7ff6d810a610 2024-04-14T15:10:54.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.297+0000 7ff6df1bc700 1 -- 172.21.15.90:0/790404968 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7ff6d810d3a0 con 0x7ff6d810a610 2024-04-14T15:10:54.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.298+0000 7ff6d5ffb700 1 -- 172.21.15.90:0/790404968 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (unknown 383520633 0 2838807938) 0x7ff6c405b180 con 0x7ff6d810a610 2024-04-14T15:10:54.298 INFO:teuthology.orchestra.run.smithi090.stdout:107374182438 2024-04-14T15:10:54.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.300+0000 7ff6df1bc700 10 librados: watch_flush enter 2024-04-14T15:10:54.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.300+0000 7ff6df1bc700 10 librados: watch_flush exit 2024-04-14T15:10:54.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.300+0000 7ff6df1bc700 20 client.14712.objecter shutdown clearing up homeless session... 2024-04-14T15:10:54.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.300+0000 7ff6df1bc700 10 client.14712.objecter successfully canceled tick 2024-04-14T15:10:54.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.300+0000 7ff6df1bc700 1 -- 172.21.15.90:0/790404968 >> v1:172.21.15.90:6800/420935427 conn(0x7ff6c8071440 legacy=0x7ff6c8073900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:54.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.300+0000 7ff6df1bc700 1 -- 172.21.15.90:0/790404968 >> v1:172.21.15.90:6789/0 conn(0x7ff6d810a610 legacy=0x7ff6d806ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:54.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.300+0000 7ff6df1bc700 1 -- 172.21.15.90:0/790404968 shutdown_connections 2024-04-14T15:10:54.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.300+0000 7ff6df1bc700 1 -- 172.21.15.90:0/790404968 >> 172.21.15.90:0/790404968 conn(0x7ff6d80fd290 msgr2=0x7ff6d81117c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:10:54.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.300+0000 7ff6df1bc700 1 -- 172.21.15.90:0/790404968 shutdown_connections 2024-04-14T15:10:54.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.300+0000 7ff6df1bc700 1 -- 172.21.15.90:0/790404968 wait complete. 2024-04-14T15:10:54.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.300+0000 7ff6df1bc700 1 librados: shutdown 2024-04-14T15:10:54.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:53 smithi092 ceph-mon[33966]: 2.18 scrub starts 2024-04-14T15:10:54.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:53 smithi092 ceph-mon[33966]: 2.18 scrub ok 2024-04-14T15:10:54.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:53 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:10:54.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:53 smithi092 ceph-mon[33966]: 4.9 scrub starts 2024-04-14T15:10:54.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:53 smithi092 ceph-mon[33966]: 4.9 scrub ok 2024-04-14T15:10:55.122 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:10:54 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:10:55.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:54 smithi092 ceph-mon[33966]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:10:55.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:54 smithi092 ceph-mon[33966]: 4.0 scrub starts 2024-04-14T15:10:55.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:54 smithi092 ceph-mon[33966]: 4.0 scrub ok 2024-04-14T15:10:55.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:54 smithi092 ceph-mon[33966]: 6.9 scrub starts 2024-04-14T15:10:55.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:54 smithi092 ceph-mon[33966]: 6.9 scrub ok 2024-04-14T15:10:55.124 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:54 smithi092 ceph-mon[33966]: 5.16 deep-scrub starts 2024-04-14T15:10:55.124 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:54 smithi092 ceph-mon[33966]: 5.16 deep-scrub ok 2024-04-14T15:10:55.124 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:54 smithi092 ceph-mon[33966]: 2.5 scrub starts 2024-04-14T15:10:55.124 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:54 smithi092 ceph-mon[33966]: 2.5 scrub ok 2024-04-14T15:10:55.124 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:54 smithi092 ceph-mon[33966]: 2.17 scrub starts 2024-04-14T15:10:55.124 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:54 smithi092 ceph-mon[33966]: 2.17 scrub ok 2024-04-14T15:10:55.124 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:54 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/790404968' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-14T15:10:55.124 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:54 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/432226639' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-14T15:10:55.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:54 smithi090 ceph-mon[30399]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:10:55.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:54 smithi090 ceph-mon[30399]: 4.0 scrub starts 2024-04-14T15:10:55.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:54 smithi090 ceph-mon[30399]: 4.0 scrub ok 2024-04-14T15:10:55.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:54 smithi090 ceph-mon[30399]: 6.9 scrub starts 2024-04-14T15:10:55.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:54 smithi090 ceph-mon[30399]: 6.9 scrub ok 2024-04-14T15:10:55.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:54 smithi090 ceph-mon[30399]: 5.16 deep-scrub starts 2024-04-14T15:10:55.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:54 smithi090 ceph-mon[30399]: 5.16 deep-scrub ok 2024-04-14T15:10:55.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:54 smithi090 ceph-mon[30399]: 2.5 scrub starts 2024-04-14T15:10:55.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:54 smithi090 ceph-mon[30399]: 2.5 scrub ok 2024-04-14T15:10:55.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:54 smithi090 ceph-mon[30399]: 2.17 scrub starts 2024-04-14T15:10:55.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:54 smithi090 ceph-mon[30399]: 2.17 scrub ok 2024-04-14T15:10:55.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:54 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/790404968' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-14T15:10:55.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:54 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/432226639' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-14T15:10:55.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:54 smithi090 ceph-mon[41670]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:10:55.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:54 smithi090 ceph-mon[41670]: 4.0 scrub starts 2024-04-14T15:10:55.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:54 smithi090 ceph-mon[41670]: 4.0 scrub ok 2024-04-14T15:10:55.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:54 smithi090 ceph-mon[41670]: 6.9 scrub starts 2024-04-14T15:10:55.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:54 smithi090 ceph-mon[41670]: 6.9 scrub ok 2024-04-14T15:10:55.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:54 smithi090 ceph-mon[41670]: 5.16 deep-scrub starts 2024-04-14T15:10:55.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:54 smithi090 ceph-mon[41670]: 5.16 deep-scrub ok 2024-04-14T15:10:55.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:54 smithi090 ceph-mon[41670]: 2.5 scrub starts 2024-04-14T15:10:55.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:54 smithi090 ceph-mon[41670]: 2.5 scrub ok 2024-04-14T15:10:55.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:54 smithi090 ceph-mon[41670]: 2.17 scrub starts 2024-04-14T15:10:55.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:54 smithi090 ceph-mon[41670]: 2.17 scrub ok 2024-04-14T15:10:55.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:54 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/790404968' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-14T15:10:55.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:54 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/432226639' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-14T15:10:55.901 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574892 got 55834574896 for osd.1 2024-04-14T15:10:55.901 DEBUG:teuthology.parallel:result is None 2024-04-14T15:10:56.118 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:55 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:10:56.119 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:55 smithi090 ceph-mon[41670]: 6.18 scrub starts 2024-04-14T15:10:56.119 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:55 smithi090 ceph-mon[41670]: 6.18 scrub ok 2024-04-14T15:10:56.119 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:55 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:10:56.119 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:55 smithi090 ceph-mon[30399]: 6.18 scrub starts 2024-04-14T15:10:56.119 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:55 smithi090 ceph-mon[30399]: 6.18 scrub ok 2024-04-14T15:10:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:55 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:10:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:55 smithi092 ceph-mon[33966]: 6.18 scrub starts 2024-04-14T15:10:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:55 smithi092 ceph-mon[33966]: 6.18 scrub ok 2024-04-14T15:10:56.765 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182434 got 107374182438 for osd.3 2024-04-14T15:10:56.765 DEBUG:teuthology.parallel:result is None 2024-04-14T15:10:57.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:56 smithi090 ceph-mon[41670]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:10:57.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:56 smithi090 ceph-mon[41670]: 5.6 scrub starts 2024-04-14T15:10:57.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:56 smithi090 ceph-mon[41670]: 5.6 scrub ok 2024-04-14T15:10:57.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:56 smithi090 ceph-mon[30399]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:10:57.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:56 smithi090 ceph-mon[30399]: 5.6 scrub starts 2024-04-14T15:10:57.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:56 smithi090 ceph-mon[30399]: 5.6 scrub ok 2024-04-14T15:10:57.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.181+0000 7f3a8435a700 1 -- 172.21.15.90:0/2292467914 >> v1:172.21.15.90:6789/0 conn(0x7f3a7c10b720 legacy=0x7f3a7c10db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:57.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.182+0000 7f3a8435a700 1 -- 172.21.15.90:0/2292467914 shutdown_connections 2024-04-14T15:10:57.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.182+0000 7f3a8435a700 1 -- 172.21.15.90:0/2292467914 >> 172.21.15.90:0/2292467914 conn(0x7f3a7c0fe3a0 msgr2=0x7f3a7c1007c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:10:57.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.182+0000 7f3a8435a700 1 -- 172.21.15.90:0/2292467914 shutdown_connections 2024-04-14T15:10:57.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.182+0000 7f3a8435a700 1 -- 172.21.15.90:0/2292467914 wait complete. 2024-04-14T15:10:57.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.182+0000 7f3a8435a700 1 librados: starting msgr at 2024-04-14T15:10:57.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.182+0000 7f3a8435a700 1 librados: starting objecter 2024-04-14T15:10:57.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.183+0000 7f3a8435a700 1 Processor -- start 2024-04-14T15:10:57.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.183+0000 7f3a8435a700 1 -- start start 2024-04-14T15:10:57.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.183+0000 7f3a8435a700 1 librados: setting wanted keys 2024-04-14T15:10:57.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.183+0000 7f3a8435a700 1 librados: calling monclient init 2024-04-14T15:10:57.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.183+0000 7f3a8435a700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3a7c10efe0 con 0x7f3a7c10f2f0 2024-04-14T15:10:57.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.183+0000 7f3a8435a700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3a7c1a8f00 con 0x7f3a7c108870 2024-04-14T15:10:57.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.183+0000 7f3a8435a700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3a7c1aa0e0 con 0x7f3a7c10b720 2024-04-14T15:10:57.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.183+0000 7f3a818f5700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f3a7c10b720 0x7f3a7c10dfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:55310/0 (socket says 172.21.15.90:55310) 2024-04-14T15:10:57.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.183+0000 7f3a818f5700 1 -- 172.21.15.90:0/432226639 learned_addr learned my addr 172.21.15.90:0/432226639 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:10:57.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.183+0000 7f3a828f7700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f3a7c10f2f0 0x7f3a7c1a67d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:45270/0 (socket says 172.21.15.90:45270) 2024-04-14T15:10:57.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.183+0000 7f3a737fe700 10 client.?.objecter ms_handle_connect 0x7f3a7c10f2f0 2024-04-14T15:10:57.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.183+0000 7f3a737fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:10:57.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.183+0000 7f3a737fe700 10 client.?.objecter ms_handle_connect 0x7f3a7c10b720 2024-04-14T15:10:57.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.183+0000 7f3a737fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:10:57.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.183+0000 7f3a737fe700 1 -- 172.21.15.90:0/432226639 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1532320961 0 0) 0x7f3a7c10efe0 con 0x7f3a7c10f2f0 2024-04-14T15:10:57.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.183+0000 7f3a737fe700 1 -- 172.21.15.90:0/432226639 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3a680035b0 con 0x7f3a7c10f2f0 2024-04-14T15:10:57.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.183+0000 7f3a737fe700 1 -- 172.21.15.90:0/432226639 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4272217926 0 0) 0x7f3a7c1aa0e0 con 0x7f3a7c10b720 2024-04-14T15:10:57.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.183+0000 7f3a737fe700 1 -- 172.21.15.90:0/432226639 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3a7c10efe0 con 0x7f3a7c10b720 2024-04-14T15:10:57.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.184+0000 7f3a737fe700 1 -- 172.21.15.90:0/432226639 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 53721761 0 0) 0x7f3a680035b0 con 0x7f3a7c10f2f0 2024-04-14T15:10:57.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.184+0000 7f3a737fe700 1 -- 172.21.15.90:0/432226639 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3a7c1aa0e0 con 0x7f3a7c10f2f0 2024-04-14T15:10:57.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.184+0000 7f3a737fe700 10 client.?.objecter ms_handle_connect 0x7f3a7c108870 2024-04-14T15:10:57.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.184+0000 7f3a737fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:10:57.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.184+0000 7f3a737fe700 1 -- 172.21.15.90:0/432226639 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f3a74001c30 con 0x7f3a7c10f2f0 2024-04-14T15:10:57.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.184+0000 7f3a737fe700 1 -- 172.21.15.90:0/432226639 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2309877690 0 0) 0x7f3a7c10efe0 con 0x7f3a7c10b720 2024-04-14T15:10:57.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.184+0000 7f3a737fe700 1 -- 172.21.15.90:0/432226639 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3a680035b0 con 0x7f3a7c10b720 2024-04-14T15:10:57.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.184+0000 7f3a737fe700 1 -- 172.21.15.90:0/432226639 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1171824322 0 0) 0x7f3a7c1a8f00 con 0x7f3a7c108870 2024-04-14T15:10:57.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.184+0000 7f3a737fe700 1 -- 172.21.15.90:0/432226639 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3a7c10efe0 con 0x7f3a7c108870 2024-04-14T15:10:57.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.184+0000 7f3a737fe700 1 -- 172.21.15.90:0/432226639 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f3a6c0015d0 con 0x7f3a7c10b720 2024-04-14T15:10:57.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.184+0000 7f3a737fe700 1 -- 172.21.15.90:0/432226639 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3888329604 0 0) 0x7f3a7c1aa0e0 con 0x7f3a7c10f2f0 2024-04-14T15:10:57.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.184+0000 7f3a737fe700 1 -- 172.21.15.90:0/432226639 >> v1:172.21.15.90:6790/0 conn(0x7f3a7c10b720 legacy=0x7f3a7c10dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:57.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.184+0000 7f3a737fe700 1 -- 172.21.15.90:0/432226639 >> v1:172.21.15.92:6789/0 conn(0x7f3a7c108870 legacy=0x7f3a7c10d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:57.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.184+0000 7f3a737fe700 1 -- 172.21.15.90:0/432226639 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a7c1ab2c0 con 0x7f3a7c10f2f0 2024-04-14T15:10:57.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.184+0000 7f3a8435a700 1 -- 172.21.15.90:0/432226639 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3a7c1aa310 con 0x7f3a7c10f2f0 2024-04-14T15:10:57.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.184+0000 7f3a8435a700 10 client.14718.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:10:57.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.184+0000 7f3a8435a700 1 -- 172.21.15.90:0/432226639 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f3a7c1aa7d0 con 0x7f3a7c10f2f0 2024-04-14T15:10:57.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.184+0000 7f3a8435a700 1 librados: init done 2024-04-14T15:10:57.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.184+0000 7f3a737fe700 1 -- 172.21.15.90:0/432226639 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f3a74003ea0 con 0x7f3a7c10f2f0 2024-04-14T15:10:57.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.184+0000 7f3a737fe700 1 -- 172.21.15.90:0/432226639 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f3a74005ed0 con 0x7f3a7c10f2f0 2024-04-14T15:10:57.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.185+0000 7f3a8435a700 1 -- 172.21.15.90:0/432226639 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a60005120 con 0x7f3a7c10f2f0 2024-04-14T15:10:57.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.186+0000 7f3a737fe700 1 -- 172.21.15.90:0/432226639 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f3a74003a00 con 0x7f3a7c10f2f0 2024-04-14T15:10:57.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.186+0000 7f3a737fe700 1 -- 172.21.15.90:0/432226639 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(85..85 src has 12..85) ==== 6288+0+0 (unknown 2690234135 0 0) 0x7f3a7408a460 con 0x7f3a7c10f2f0 2024-04-14T15:10:57.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.186+0000 7f3a737fe700 10 client.14718.objecter ms_dispatch 0x7f3a7c001550 osd_map(85..85 src has 12..85) 2024-04-14T15:10:57.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.186+0000 7f3a737fe700 3 client.14718.objecter handle_osd_map got epochs [85,85] > 0 2024-04-14T15:10:57.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.186+0000 7f3a737fe700 3 client.14718.objecter handle_osd_map decoding full epoch 85 2024-04-14T15:10:57.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.186+0000 7f3a737fe700 20 client.14718.objecter dump_active .. 0 homeless 2024-04-14T15:10:57.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.186+0000 7f3a737fe700 10 client.14718.objecter ms_handle_connect 0x7f3a680713a0 2024-04-14T15:10:57.318 INFO:teuthology.orchestra.run.smithi090.stdout:249108103187 2024-04-14T15:10:57.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.191+0000 7f3a737fe700 1 -- 172.21.15.90:0/432226639 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f3a74057630 con 0x7f3a7c10f2f0 2024-04-14T15:10:57.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.400+0000 7f3a8435a700 1 -- 172.21.15.90:0/432226639 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f3a60005410 con 0x7f3a7c10f2f0 2024-04-14T15:10:57.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.400+0000 7f3a737fe700 1 -- 172.21.15.90:0/432226639 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (unknown 1482059429 0 282837521) 0x7f3a7405b2e0 con 0x7f3a7c10f2f0 2024-04-14T15:10:57.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.402+0000 7f3a8435a700 10 librados: watch_flush enter 2024-04-14T15:10:57.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.402+0000 7f3a8435a700 10 librados: watch_flush exit 2024-04-14T15:10:57.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.402+0000 7f3a8435a700 20 client.14718.objecter shutdown clearing up homeless session... 2024-04-14T15:10:57.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.402+0000 7f3a8435a700 10 client.14718.objecter successfully canceled tick 2024-04-14T15:10:57.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.402+0000 7f3a8435a700 1 -- 172.21.15.90:0/432226639 >> v1:172.21.15.90:6800/420935427 conn(0x7f3a680713a0 legacy=0x7f3a68073860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:57.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.402+0000 7f3a8435a700 1 -- 172.21.15.90:0/432226639 >> v1:172.21.15.90:6789/0 conn(0x7f3a7c10f2f0 legacy=0x7f3a7c1a67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:10:57.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.402+0000 7f3a8435a700 1 -- 172.21.15.90:0/432226639 shutdown_connections 2024-04-14T15:10:57.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.402+0000 7f3a8435a700 1 -- 172.21.15.90:0/432226639 >> 172.21.15.90:0/432226639 conn(0x7f3a7c0fe3a0 msgr2=0x7f3a7c112870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:10:57.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.403+0000 7f3a8435a700 1 -- 172.21.15.90:0/432226639 shutdown_connections 2024-04-14T15:10:57.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.403+0000 7f3a8435a700 1 -- 172.21.15.90:0/432226639 wait complete. 2024-04-14T15:10:57.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:10:54.403+0000 7f3a8435a700 1 librados: shutdown 2024-04-14T15:10:57.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:56 smithi092 ceph-mon[33966]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:10:57.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:56 smithi092 ceph-mon[33966]: 5.6 scrub starts 2024-04-14T15:10:57.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:56 smithi092 ceph-mon[33966]: 5.6 scrub ok 2024-04-14T15:10:58.026 INFO:tasks.cephadm.ceph_manager.ceph:need seq 249108103183 got 249108103187 for osd.7 2024-04-14T15:10:58.026 DEBUG:teuthology.parallel:result is None 2024-04-14T15:10:58.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:57 smithi090 ceph-mon[41670]: 6.10 scrub starts 2024-04-14T15:10:58.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:57 smithi090 ceph-mon[41670]: 6.10 scrub ok 2024-04-14T15:10:58.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:57 smithi090 ceph-mon[41670]: 4.15 scrub starts 2024-04-14T15:10:58.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:57 smithi090 ceph-mon[41670]: 4.15 scrub ok 2024-04-14T15:10:58.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:57 smithi090 ceph-mon[41670]: 6.14 scrub starts 2024-04-14T15:10:58.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:57 smithi090 ceph-mon[41670]: 6.14 scrub ok 2024-04-14T15:10:58.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:57 smithi090 ceph-mon[30399]: 6.10 scrub starts 2024-04-14T15:10:58.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:57 smithi090 ceph-mon[30399]: 6.10 scrub ok 2024-04-14T15:10:58.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:57 smithi090 ceph-mon[30399]: 4.15 scrub starts 2024-04-14T15:10:58.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:57 smithi090 ceph-mon[30399]: 4.15 scrub ok 2024-04-14T15:10:58.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:57 smithi090 ceph-mon[30399]: 6.14 scrub starts 2024-04-14T15:10:58.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:57 smithi090 ceph-mon[30399]: 6.14 scrub ok 2024-04-14T15:10:58.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:57 smithi092 ceph-mon[33966]: 6.10 scrub starts 2024-04-14T15:10:58.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:57 smithi092 ceph-mon[33966]: 6.10 scrub ok 2024-04-14T15:10:58.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:57 smithi092 ceph-mon[33966]: 4.15 scrub starts 2024-04-14T15:10:58.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:57 smithi092 ceph-mon[33966]: 4.15 scrub ok 2024-04-14T15:10:58.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:57 smithi092 ceph-mon[33966]: 6.14 scrub starts 2024-04-14T15:10:58.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:57 smithi092 ceph-mon[33966]: 6.14 scrub ok 2024-04-14T15:10:59.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:58 smithi090 ceph-mon[41670]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 915 B/s rd, 0 op/s 2024-04-14T15:10:59.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:58 smithi090 ceph-mon[41670]: 2.19 scrub starts 2024-04-14T15:10:59.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:58 smithi090 ceph-mon[41670]: 2.19 scrub ok 2024-04-14T15:10:59.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:58 smithi090 ceph-mon[41670]: 5.1f scrub starts 2024-04-14T15:10:59.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:58 smithi090 ceph-mon[41670]: 5.1f scrub ok 2024-04-14T15:10:59.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:58 smithi090 ceph-mon[30399]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 915 B/s rd, 0 op/s 2024-04-14T15:10:59.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:58 smithi090 ceph-mon[30399]: 2.19 scrub starts 2024-04-14T15:10:59.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:58 smithi090 ceph-mon[30399]: 2.19 scrub ok 2024-04-14T15:10:59.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:58 smithi090 ceph-mon[30399]: 5.1f scrub starts 2024-04-14T15:10:59.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:58 smithi090 ceph-mon[30399]: 5.1f scrub ok 2024-04-14T15:10:59.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:58 smithi092 ceph-mon[33966]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 915 B/s rd, 0 op/s 2024-04-14T15:10:59.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:58 smithi092 ceph-mon[33966]: 2.19 scrub starts 2024-04-14T15:10:59.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:58 smithi092 ceph-mon[33966]: 2.19 scrub ok 2024-04-14T15:10:59.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:58 smithi092 ceph-mon[33966]: 5.1f scrub starts 2024-04-14T15:10:59.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:58 smithi092 ceph-mon[33966]: 5.1f scrub ok 2024-04-14T15:11:00.181 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:59 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:11:00.181 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:59 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:11:00.181 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:59 smithi090 ceph-mon[30399]: osdmap e86: 8 total, 8 up, 8 in 2024-04-14T15:11:00.182 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:59 smithi090 ceph-mon[30399]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:11:00.182 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:59 smithi090 ceph-mon[30399]: 5.2 scrub starts 2024-04-14T15:11:00.182 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:10:59 smithi090 ceph-mon[30399]: 5.2 scrub ok 2024-04-14T15:11:00.182 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:59 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:11:00.182 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:59 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:11:00.182 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:59 smithi090 ceph-mon[41670]: osdmap e86: 8 total, 8 up, 8 in 2024-04-14T15:11:00.182 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:59 smithi090 ceph-mon[41670]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:11:00.182 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:59 smithi090 ceph-mon[41670]: 5.2 scrub starts 2024-04-14T15:11:00.182 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:10:59 smithi090 ceph-mon[41670]: 5.2 scrub ok 2024-04-14T15:11:00.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:59 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:11:00.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:59 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:11:00.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:59 smithi092 ceph-mon[33966]: osdmap e86: 8 total, 8 up, 8 in 2024-04-14T15:11:00.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:59 smithi092 ceph-mon[33966]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:11:00.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:59 smithi092 ceph-mon[33966]: 5.2 scrub starts 2024-04-14T15:11:00.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:10:59 smithi092 ceph-mon[33966]: 5.2 scrub ok 2024-04-14T15:11:01.104 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:00 smithi090 ceph-mon[41670]: 5.b scrub starts 2024-04-14T15:11:01.104 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:00 smithi090 ceph-mon[41670]: 5.b scrub ok 2024-04-14T15:11:01.104 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:00 smithi090 ceph-mon[41670]: 6.1c scrub starts 2024-04-14T15:11:01.105 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:00 smithi090 ceph-mon[41670]: 6.1c scrub ok 2024-04-14T15:11:01.105 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:00 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3384181037' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-14T15:11:01.105 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:00 smithi090 ceph-mon[41670]: osdmap e87: 8 total, 8 up, 8 in 2024-04-14T15:11:01.105 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:00 smithi090 ceph-mon[30399]: 5.b scrub starts 2024-04-14T15:11:01.105 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:00 smithi090 ceph-mon[30399]: 5.b scrub ok 2024-04-14T15:11:01.105 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:00 smithi090 ceph-mon[30399]: 6.1c scrub starts 2024-04-14T15:11:01.105 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:00 smithi090 ceph-mon[30399]: 6.1c scrub ok 2024-04-14T15:11:01.105 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:00 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3384181037' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-14T15:11:01.105 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:00 smithi090 ceph-mon[30399]: osdmap e87: 8 total, 8 up, 8 in 2024-04-14T15:11:01.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:00.288+0000 7f67a2c21700 1 -- 172.21.15.90:0/620476314 >> v1:172.21.15.90:6789/0 conn(0x7f679c10e200 legacy=0x7f679c1106c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:01.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:00.288+0000 7f67a2c21700 1 -- 172.21.15.90:0/620476314 shutdown_connections 2024-04-14T15:11:01.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:00.288+0000 7f67a2c21700 1 -- 172.21.15.90:0/620476314 >> 172.21.15.90:0/620476314 conn(0x7f679c0fd2b0 msgr2=0x7f679c0ff6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:11:01.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:00.288+0000 7f67a2c21700 1 -- 172.21.15.90:0/620476314 shutdown_connections 2024-04-14T15:11:01.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:00.288+0000 7f67a2c21700 1 -- 172.21.15.90:0/620476314 wait complete. 2024-04-14T15:11:01.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:00.289+0000 7f67a2c21700 1 librados: starting msgr at 2024-04-14T15:11:01.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:00.289+0000 7f67a2c21700 1 librados: starting objecter 2024-04-14T15:11:01.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:00.289+0000 7f67a2c21700 1 Processor -- start 2024-04-14T15:11:01.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:00.289+0000 7f67a2c21700 1 -- start start 2024-04-14T15:11:01.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:00.289+0000 7f67a2c21700 1 librados: setting wanted keys 2024-04-14T15:11:01.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:00.289+0000 7f67a2c21700 1 librados: calling monclient init 2024-04-14T15:11:01.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:00.289+0000 7f67a2c21700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f679c1a7e10 con 0x7f679c107780 2024-04-14T15:11:01.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:00.289+0000 7f67a2c21700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f679c1a8ff0 con 0x7f679c10e200 2024-04-14T15:11:01.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:00.289+0000 7f67a2c21700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f679c1aa1d0 con 0x7f679c10a630 2024-04-14T15:11:01.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:00.289+0000 7f67a09bd700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f679c107780 0x7f679c10be30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:45286/0 (socket says 172.21.15.90:45286) 2024-04-14T15:11:01.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:00.289+0000 7f67a09bd700 1 -- 172.21.15.90:0/3384181037 learned_addr learned my addr 172.21.15.90:0/3384181037 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:11:01.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:00.289+0000 7f6799ffb700 10 client.?.objecter ms_handle_connect 0x7f679c10a630 2024-04-14T15:11:01.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:00.289+0000 7f6799ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:11:01.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:00.289+0000 7f6799ffb700 10 client.?.objecter ms_handle_connect 0x7f679c107780 2024-04-14T15:11:01.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:00.289+0000 7f6799ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:11:01.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:00.290+0000 7f6799ffb700 1 -- 172.21.15.90:0/3384181037 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 516306369 0 0) 0x7f679c1aa1d0 con 0x7f679c10a630 2024-04-14T15:11:01.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:00.290+0000 7f6799ffb700 1 -- 172.21.15.90:0/3384181037 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f678c0035b0 con 0x7f679c10a630 2024-04-14T15:11:01.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:00.290+0000 7f6799ffb700 1 -- 172.21.15.90:0/3384181037 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3285829003 0 0) 0x7f679c1a7e10 con 0x7f679c107780 2024-04-14T15:11:01.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:00.290+0000 7f6799ffb700 1 -- 172.21.15.90:0/3384181037 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f679c1aa1d0 con 0x7f679c107780 2024-04-14T15:11:01.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:00.290+0000 7f6799ffb700 10 client.?.objecter ms_handle_connect 0x7f679c10e200 2024-04-14T15:11:01.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:00.290+0000 7f6799ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:11:01.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:00.290+0000 7f6799ffb700 1 -- 172.21.15.90:0/3384181037 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2118805408 0 0) 0x7f679c1aa1d0 con 0x7f679c107780 2024-04-14T15:11:01.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:00.290+0000 7f6799ffb700 1 -- 172.21.15.90:0/3384181037 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f679c1a7e10 con 0x7f679c107780 2024-04-14T15:11:01.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:00.290+0000 7f6799ffb700 1 -- 172.21.15.90:0/3384181037 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f67900018b0 con 0x7f679c107780 2024-04-14T15:11:01.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:00.290+0000 7f6799ffb700 1 -- 172.21.15.90:0/3384181037 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3525745933 0 0) 0x7f678c0035b0 con 0x7f679c10a630 2024-04-14T15:11:01.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:00.290+0000 7f6799ffb700 1 -- 172.21.15.90:0/3384181037 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f679c1aa1d0 con 0x7f679c10a630 2024-04-14T15:11:01.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:00.290+0000 7f6799ffb700 1 -- 172.21.15.90:0/3384181037 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f6788001850 con 0x7f679c10a630 2024-04-14T15:11:01.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:00.290+0000 7f6799ffb700 1 -- 172.21.15.90:0/3384181037 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 877355476 0 0) 0x7f679c1a8ff0 con 0x7f679c10e200 2024-04-14T15:11:01.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:00.290+0000 7f6799ffb700 1 -- 172.21.15.90:0/3384181037 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f678c0035b0 con 0x7f679c10e200 2024-04-14T15:11:01.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:00.290+0000 7f6799ffb700 1 -- 172.21.15.90:0/3384181037 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1874763003 0 0) 0x7f679c1a7e10 con 0x7f679c107780 2024-04-14T15:11:01.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:00.290+0000 7f6799ffb700 1 -- 172.21.15.90:0/3384181037 >> v1:172.21.15.90:6790/0 conn(0x7f679c10a630 legacy=0x7f679c10c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:01.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:00.290+0000 7f6799ffb700 1 -- 172.21.15.90:0/3384181037 >> v1:172.21.15.92:6789/0 conn(0x7f679c10e200 legacy=0x7f679c1a66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:01.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:00.290+0000 7f6799ffb700 1 -- 172.21.15.90:0/3384181037 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f679c1ab3b0 con 0x7f679c107780 2024-04-14T15:11:01.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:00.290+0000 7f67a2c21700 1 -- 172.21.15.90:0/3384181037 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f679c1a8040 con 0x7f679c107780 2024-04-14T15:11:01.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:00.290+0000 7f6799ffb700 1 -- 172.21.15.90:0/3384181037 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f6790003b40 con 0x7f679c107780 2024-04-14T15:11:01.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:00.290+0000 7f6799ffb700 1 -- 172.21.15.90:0/3384181037 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f6790005cf0 con 0x7f679c107780 2024-04-14T15:11:01.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:00.290+0000 7f67a2c21700 10 client.14724.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:11:01.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:00.290+0000 7f67a2c21700 1 -- 172.21.15.90:0/3384181037 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f679c1a8500 con 0x7f679c107780 2024-04-14T15:11:01.217 INFO:teuthology.orchestra.run.smithi090.stdout:210453397528 2024-04-14T15:11:01.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:00.290+0000 7f67a2c21700 1 librados: init done 2024-04-14T15:11:01.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:00.291+0000 7f67a2c21700 1 -- 172.21.15.90:0/3384181037 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f679c102ec0 con 0x7f679c107780 2024-04-14T15:11:01.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:00.292+0000 7f6799ffb700 1 -- 172.21.15.90:0/3384181037 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f67900036a0 con 0x7f679c107780 2024-04-14T15:11:01.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:00.292+0000 7f6799ffb700 1 -- 172.21.15.90:0/3384181037 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(86..86 src has 12..86) ==== 6288+0+0 (unknown 2394025075 0 0) 0x7f679008a510 con 0x7f679c107780 2024-04-14T15:11:01.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:00.292+0000 7f6799ffb700 10 client.14724.objecter ms_dispatch 0x7f679c001550 osd_map(86..86 src has 12..86) 2024-04-14T15:11:01.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:00.292+0000 7f6799ffb700 3 client.14724.objecter handle_osd_map got epochs [86,86] > 0 2024-04-14T15:11:01.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:00.292+0000 7f6799ffb700 3 client.14724.objecter handle_osd_map decoding full epoch 86 2024-04-14T15:11:01.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:00.292+0000 7f6799ffb700 20 client.14724.objecter dump_active .. 0 homeless 2024-04-14T15:11:01.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:00.292+0000 7f6799ffb700 10 client.14724.objecter ms_handle_connect 0x7f678c071360 2024-04-14T15:11:01.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:00.296+0000 7f6799ffb700 1 -- 172.21.15.90:0/3384181037 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f67900576e0 con 0x7f679c107780 2024-04-14T15:11:01.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:00.500+0000 7f67a2c21700 1 -- 172.21.15.90:0/3384181037 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f679c10d360 con 0x7f679c107780 2024-04-14T15:11:01.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:00.500+0000 7f6799ffb700 1 -- 172.21.15.90:0/3384181037 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (unknown 1274345170 0 3498544870) 0x7f679005b390 con 0x7f679c107780 2024-04-14T15:11:01.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:00.502+0000 7f67a2c21700 10 librados: watch_flush enter 2024-04-14T15:11:01.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:00.502+0000 7f67a2c21700 10 librados: watch_flush exit 2024-04-14T15:11:01.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:00.502+0000 7f67a2c21700 20 client.14724.objecter shutdown clearing up homeless session... 2024-04-14T15:11:01.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:00.503+0000 7f67a2c21700 10 client.14724.objecter successfully canceled tick 2024-04-14T15:11:01.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:00.503+0000 7f67a2c21700 1 -- 172.21.15.90:0/3384181037 >> v1:172.21.15.90:6800/420935427 conn(0x7f678c071360 legacy=0x7f678c073820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:01.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:00.503+0000 7f67a2c21700 1 -- 172.21.15.90:0/3384181037 >> v1:172.21.15.90:6789/0 conn(0x7f679c107780 legacy=0x7f679c10be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:01.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:00.503+0000 7f67a2c21700 1 -- 172.21.15.90:0/3384181037 shutdown_connections 2024-04-14T15:11:01.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:00.503+0000 7f67a2c21700 1 -- 172.21.15.90:0/3384181037 >> 172.21.15.90:0/3384181037 conn(0x7f679c0fd2b0 msgr2=0x7f679c100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:11:01.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:00.503+0000 7f67a2c21700 1 -- 172.21.15.90:0/3384181037 shutdown_connections 2024-04-14T15:11:01.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:00.503+0000 7f67a2c21700 1 -- 172.21.15.90:0/3384181037 wait complete. 2024-04-14T15:11:01.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:00.503+0000 7f67a2c21700 1 librados: shutdown 2024-04-14T15:11:01.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:00 smithi092 ceph-mon[33966]: 5.b scrub starts 2024-04-14T15:11:01.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:00 smithi092 ceph-mon[33966]: 5.b scrub ok 2024-04-14T15:11:01.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:00 smithi092 ceph-mon[33966]: 6.1c scrub starts 2024-04-14T15:11:01.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:00 smithi092 ceph-mon[33966]: 6.1c scrub ok 2024-04-14T15:11:01.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:00 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3384181037' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-14T15:11:01.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:00 smithi092 ceph-mon[33966]: osdmap e87: 8 total, 8 up, 8 in 2024-04-14T15:11:01.756 INFO:tasks.cephadm.ceph_manager.ceph:need seq 210453397524 got 210453397528 for osd.6 2024-04-14T15:11:01.756 DEBUG:teuthology.parallel:result is None 2024-04-14T15:11:02.105 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:01 smithi090 ceph-mon[41670]: 6.5 scrub starts 2024-04-14T15:11:02.105 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:01 smithi090 ceph-mon[41670]: 6.5 scrub ok 2024-04-14T15:11:02.105 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:01 smithi090 ceph-mon[41670]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:11:02.105 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:01 smithi090 ceph-mon[30399]: 6.5 scrub starts 2024-04-14T15:11:02.106 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:01 smithi090 ceph-mon[30399]: 6.5 scrub ok 2024-04-14T15:11:02.106 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:01 smithi090 ceph-mon[30399]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:11:02.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:01 smithi092 ceph-mon[33966]: 6.5 scrub starts 2024-04-14T15:11:02.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:01 smithi092 ceph-mon[33966]: 6.5 scrub ok 2024-04-14T15:11:02.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:01 smithi092 ceph-mon[33966]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:11:02.441 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:11:02 smithi090 podman[78655]: 2024-04-14 15:11:02.223554099 +0000 UTC m=+2.131293635 volume create bfd38cfe623e46a9dc77d1785ae43d3cd0a5f1a3e056066ba8ebfbb9d99cc625 2024-04-14T15:11:02.441 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:11:02 smithi090 podman[78655]: 2024-04-14 15:11:02.146221401 +0000 UTC m=+2.053960945 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-04-14T15:11:02.442 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:11:02 smithi090 podman[78655]: 2024-04-14 15:11:02.332460803 +0000 UTC m=+2.240200384 container create fc393084a9b6da0d9165587ad2f71d1ea3acf037ad5406bdf43ae8297b1a96fd (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-14T15:11:02.932 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:02 smithi090 ceph-mon[41670]: 6.c scrub starts 2024-04-14T15:11:02.933 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:02 smithi090 ceph-mon[41670]: 6.c scrub ok 2024-04-14T15:11:02.933 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:02 smithi090 ceph-mon[41670]: 2.b scrub starts 2024-04-14T15:11:02.933 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:02 smithi090 ceph-mon[41670]: 2.b scrub ok 2024-04-14T15:11:02.933 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:02 smithi090 ceph-mon[30399]: 6.c scrub starts 2024-04-14T15:11:02.933 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:02 smithi090 ceph-mon[30399]: 6.c scrub ok 2024-04-14T15:11:02.933 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:02 smithi090 ceph-mon[30399]: 2.b scrub starts 2024-04-14T15:11:02.933 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:02 smithi090 ceph-mon[30399]: 2.b scrub ok 2024-04-14T15:11:02.933 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:11:02 smithi090 podman[78655]: 2024-04-14 15:11:02.780103017 +0000 UTC m=+2.687842553 container init fc393084a9b6da0d9165587ad2f71d1ea3acf037ad5406bdf43ae8297b1a96fd (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-14T15:11:02.934 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:11:02 smithi090 podman[78655]: 2024-04-14 15:11:02.785908508 +0000 UTC m=+2.693648052 container start fc393084a9b6da0d9165587ad2f71d1ea3acf037ad5406bdf43ae8297b1a96fd (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-14T15:11:02.934 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:11:02 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-alertmanager-a[78793]: ts=2024-04-14T15:11:02.798Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-04-14T15:11:02.934 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:11:02 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-alertmanager-a[78793]: ts=2024-04-14T15:11:02.798Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-04-14T15:11:02.934 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:11:02 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-alertmanager-a[78793]: ts=2024-04-14T15:11:02.799Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.90 port=9094 2024-04-14T15:11:02.934 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:11:02 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-alertmanager-a[78793]: ts=2024-04-14T15:11:02.800Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-04-14T15:11:02.934 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:11:02 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-alertmanager-a[78793]: ts=2024-04-14T15:11:02.823Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-14T15:11:02.934 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:11:02 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-alertmanager-a[78793]: ts=2024-04-14T15:11:02.824Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-14T15:11:02.934 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:11:02 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-alertmanager-a[78793]: ts=2024-04-14T15:11:02.826Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-04-14T15:11:02.934 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:11:02 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-alertmanager-a[78793]: ts=2024-04-14T15:11:02.826Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-04-14T15:11:03.191 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:11:02 smithi090 bash[78655]: fc393084a9b6da0d9165587ad2f71d1ea3acf037ad5406bdf43ae8297b1a96fd 2024-04-14T15:11:03.191 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:11:03 smithi090 systemd[1]: Started Ceph alertmanager.a for 0e288098-fa70-11ee-bc8e-c7b262605968. 2024-04-14T15:11:03.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.230+0000 7fe651c05700 1 -- 172.21.15.90:0/1670448835 >> v1:172.21.15.90:6789/0 conn(0x7fe64c108980 legacy=0x7fe64c108d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:03.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.231+0000 7fe651c05700 1 -- 172.21.15.90:0/1670448835 shutdown_connections 2024-04-14T15:11:03.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.231+0000 7fe651c05700 1 -- 172.21.15.90:0/1670448835 >> 172.21.15.90:0/1670448835 conn(0x7fe64c076aa0 msgr2=0x7fe64c076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:11:03.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.231+0000 7fe651c05700 1 -- 172.21.15.90:0/1670448835 shutdown_connections 2024-04-14T15:11:03.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.231+0000 7fe651c05700 1 -- 172.21.15.90:0/1670448835 wait complete. 2024-04-14T15:11:03.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.231+0000 7fe651c05700 1 librados: starting msgr at 2024-04-14T15:11:03.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.231+0000 7fe651c05700 1 librados: starting objecter 2024-04-14T15:11:03.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.231+0000 7fe651c05700 1 Processor -- start 2024-04-14T15:11:03.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.231+0000 7fe651c05700 1 -- start start 2024-04-14T15:11:03.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.231+0000 7fe651c05700 1 librados: setting wanted keys 2024-04-14T15:11:03.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.231+0000 7fe651c05700 1 librados: calling monclient init 2024-04-14T15:11:03.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.232+0000 7fe651c05700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe64c1a9d40 con 0x7fe64c10b6f0 2024-04-14T15:11:03.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.232+0000 7fe651c05700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe64c1aaf60 con 0x7fe64c10f2f0 2024-04-14T15:11:03.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.232+0000 7fe651c05700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe64c1ac1a0 con 0x7fe64c108980 2024-04-14T15:11:03.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.232+0000 7fe64affd700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fe64c10b6f0 0x7fe64c1a4b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:40006/0 (socket says 172.21.15.90:40006) 2024-04-14T15:11:03.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.232+0000 7fe64affd700 1 -- 172.21.15.90:0/1294041795 learned_addr learned my addr 172.21.15.90:0/1294041795 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:11:03.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.232+0000 7fe648ff9700 10 client.?.objecter ms_handle_connect 0x7fe64c108980 2024-04-14T15:11:03.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.232+0000 7fe648ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:11:03.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.232+0000 7fe648ff9700 10 client.?.objecter ms_handle_connect 0x7fe64c10b6f0 2024-04-14T15:11:03.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.232+0000 7fe648ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:11:03.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.232+0000 7fe648ff9700 10 client.?.objecter ms_handle_connect 0x7fe64c10f2f0 2024-04-14T15:11:03.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.232+0000 7fe648ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:11:03.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.232+0000 7fe648ff9700 1 -- 172.21.15.90:0/1294041795 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2500464154 0 0) 0x7fe64c1a9d40 con 0x7fe64c10b6f0 2024-04-14T15:11:03.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.232+0000 7fe648ff9700 1 -- 172.21.15.90:0/1294041795 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe6340035b0 con 0x7fe64c10b6f0 2024-04-14T15:11:03.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.232+0000 7fe648ff9700 1 -- 172.21.15.90:0/1294041795 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1107245863 0 0) 0x7fe64c1aaf60 con 0x7fe64c10f2f0 2024-04-14T15:11:03.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.232+0000 7fe648ff9700 1 -- 172.21.15.90:0/1294041795 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe64c1a9d40 con 0x7fe64c10f2f0 2024-04-14T15:11:03.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.232+0000 7fe648ff9700 1 -- 172.21.15.90:0/1294041795 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 778543237 0 0) 0x7fe64c1ac1a0 con 0x7fe64c108980 2024-04-14T15:11:03.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.232+0000 7fe648ff9700 1 -- 172.21.15.90:0/1294041795 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe64c1aaf60 con 0x7fe64c108980 2024-04-14T15:11:03.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.232+0000 7fe648ff9700 1 -- 172.21.15.90:0/1294041795 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 785175301 0 0) 0x7fe64c1a9d40 con 0x7fe64c10f2f0 2024-04-14T15:11:03.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.232+0000 7fe648ff9700 1 -- 172.21.15.90:0/1294041795 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe64c1ac1a0 con 0x7fe64c10f2f0 2024-04-14T15:11:03.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.232+0000 7fe648ff9700 1 -- 172.21.15.90:0/1294041795 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1424526678 0 0) 0x7fe6340035b0 con 0x7fe64c10b6f0 2024-04-14T15:11:03.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.232+0000 7fe648ff9700 1 -- 172.21.15.90:0/1294041795 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe64c1a9d40 con 0x7fe64c10b6f0 2024-04-14T15:11:03.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.232+0000 7fe648ff9700 1 -- 172.21.15.90:0/1294041795 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fe644001bd0 con 0x7fe64c10f2f0 2024-04-14T15:11:03.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.232+0000 7fe648ff9700 1 -- 172.21.15.90:0/1294041795 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fe640001870 con 0x7fe64c10b6f0 2024-04-14T15:11:03.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.232+0000 7fe648ff9700 1 -- 172.21.15.90:0/1294041795 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3724572925 0 0) 0x7fe64c1aaf60 con 0x7fe64c108980 2024-04-14T15:11:03.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.232+0000 7fe648ff9700 1 -- 172.21.15.90:0/1294041795 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe6340035b0 con 0x7fe64c108980 2024-04-14T15:11:03.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.232+0000 7fe648ff9700 1 -- 172.21.15.90:0/1294041795 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fe63c001430 con 0x7fe64c108980 2024-04-14T15:11:03.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.232+0000 7fe648ff9700 1 -- 172.21.15.90:0/1294041795 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 350680478 0 0) 0x7fe64c1ac1a0 con 0x7fe64c10f2f0 2024-04-14T15:11:03.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.232+0000 7fe648ff9700 1 -- 172.21.15.90:0/1294041795 >> v1:172.21.15.90:6790/0 conn(0x7fe64c108980 legacy=0x7fe64c103c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:03.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.232+0000 7fe648ff9700 1 -- 172.21.15.90:0/1294041795 >> v1:172.21.15.90:6789/0 conn(0x7fe64c10b6f0 legacy=0x7fe64c1a4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:03.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.232+0000 7fe648ff9700 1 -- 172.21.15.90:0/1294041795 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe64c1ad3e0 con 0x7fe64c10f2f0 2024-04-14T15:11:03.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.233+0000 7fe651c05700 1 -- 172.21.15.90:0/1294041795 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe64c1a9f70 con 0x7fe64c10f2f0 2024-04-14T15:11:03.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.233+0000 7fe651c05700 10 client.24619.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:11:03.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.233+0000 7fe651c05700 1 -- 172.21.15.90:0/1294041795 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe64c1aa430 con 0x7fe64c10f2f0 2024-04-14T15:11:03.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.233+0000 7fe651c05700 1 librados: init done 2024-04-14T15:11:03.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.233+0000 7fe648ff9700 1 -- 172.21.15.90:0/1294041795 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fe644003ec0 con 0x7fe64c10f2f0 2024-04-14T15:11:03.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.233+0000 7fe648ff9700 1 -- 172.21.15.90:0/1294041795 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fe644004d00 con 0x7fe64c10f2f0 2024-04-14T15:11:03.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.234+0000 7fe651c05700 1 -- 172.21.15.90:0/1294041795 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe638005120 con 0x7fe64c10f2f0 2024-04-14T15:11:03.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.234+0000 7fe648ff9700 1 -- 172.21.15.90:0/1294041795 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fe644003a20 con 0x7fe64c10f2f0 2024-04-14T15:11:03.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.235+0000 7fe648ff9700 1 -- 172.21.15.90:0/1294041795 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(87..87 src has 12..87) ==== 6288+0+0 (unknown 1749939968 0 0) 0x7fe644089360 con 0x7fe64c10f2f0 2024-04-14T15:11:03.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.235+0000 7fe648ff9700 10 client.24619.objecter ms_dispatch 0x7fe64c001550 osd_map(87..87 src has 12..87) 2024-04-14T15:11:03.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.235+0000 7fe648ff9700 3 client.24619.objecter handle_osd_map got epochs [87,87] > 0 2024-04-14T15:11:03.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.235+0000 7fe648ff9700 3 client.24619.objecter handle_osd_map decoding full epoch 87 2024-04-14T15:11:03.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.235+0000 7fe648ff9700 20 client.24619.objecter dump_active .. 0 homeless 2024-04-14T15:11:03.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.235+0000 7fe648ff9700 10 client.24619.objecter ms_handle_connect 0x7fe6340714b0 2024-04-14T15:11:03.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.239+0000 7fe648ff9700 1 -- 172.21.15.90:0/1294041795 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fe6440564d0 con 0x7fe64c10f2f0 2024-04-14T15:11:03.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:02 smithi092 ceph-mon[33966]: 6.c scrub starts 2024-04-14T15:11:03.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:02 smithi092 ceph-mon[33966]: 6.c scrub ok 2024-04-14T15:11:03.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:02 smithi092 ceph-mon[33966]: 2.b scrub starts 2024-04-14T15:11:03.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:02 smithi092 ceph-mon[33966]: 2.b scrub ok 2024-04-14T15:11:03.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.456+0000 7fe651c05700 1 -- 172.21.15.90:0/1294041795 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fe638005410 con 0x7fe64c10f2f0 2024-04-14T15:11:03.457 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:11:03 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: [14/Apr/2024:15:11:03] ENGINE Bus STOPPING 2024-04-14T15:11:03.457 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:11:03 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: [14/Apr/2024:15:11:03] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-14T15:11:03.457 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:11:03 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: [14/Apr/2024:15:11:03] ENGINE Bus STOPPED 2024-04-14T15:11:03.457 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:11:03 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: [14/Apr/2024:15:11:03] ENGINE Bus STARTING 2024-04-14T15:11:03.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.456+0000 7fe648ff9700 1 -- 172.21.15.90:0/1294041795 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (unknown 2131975755 0 1451352899) 0x7fe64405a180 con 0x7fe64c10f2f0 2024-04-14T15:11:03.458 INFO:teuthology.orchestra.run.smithi090.stdout:176093659167 2024-04-14T15:11:03.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.459+0000 7fe651c05700 10 librados: watch_flush enter 2024-04-14T15:11:03.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.459+0000 7fe651c05700 10 librados: watch_flush exit 2024-04-14T15:11:03.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.459+0000 7fe651c05700 20 client.24619.objecter shutdown clearing up homeless session... 2024-04-14T15:11:03.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.459+0000 7fe651c05700 10 client.24619.objecter successfully canceled tick 2024-04-14T15:11:03.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.459+0000 7fe651c05700 1 -- 172.21.15.90:0/1294041795 >> v1:172.21.15.90:6800/420935427 conn(0x7fe6340714b0 legacy=0x7fe634073970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:03.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.459+0000 7fe651c05700 1 -- 172.21.15.90:0/1294041795 >> v1:172.21.15.92:6789/0 conn(0x7fe64c10f2f0 legacy=0x7fe64c1a8400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:03.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.459+0000 7fe651c05700 1 -- 172.21.15.90:0/1294041795 shutdown_connections 2024-04-14T15:11:03.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.459+0000 7fe651c05700 1 -- 172.21.15.90:0/1294041795 >> 172.21.15.90:0/1294041795 conn(0x7fe64c076aa0 msgr2=0x7fe64c1117b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:11:03.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.459+0000 7fe651c05700 1 -- 172.21.15.90:0/1294041795 shutdown_connections 2024-04-14T15:11:03.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.459+0000 7fe651c05700 1 -- 172.21.15.90:0/1294041795 wait complete. 2024-04-14T15:11:03.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:03.459+0000 7fe651c05700 1 librados: shutdown 2024-04-14T15:11:03.849 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:11:03 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: [14/Apr/2024:15:11:03] ENGINE Serving on http://:::9283 2024-04-14T15:11:03.850 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:11:03 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: [14/Apr/2024:15:11:03] ENGINE Bus STARTED 2024-04-14T15:11:04.142 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:11:03 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:11:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:11:04.143 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:04 smithi090 ceph-mon[41670]: 5.f scrub starts 2024-04-14T15:11:04.143 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:04 smithi090 ceph-mon[41670]: 5.f scrub ok 2024-04-14T15:11:04.143 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:04 smithi090 ceph-mon[41670]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-14T15:11:04.144 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:04 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:04.144 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:04 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:04.144 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:04 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:04.144 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:04 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:04.144 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:04 smithi090 ceph-mon[41670]: Regenerating cephadm self-signed grafana TLS certificates 2024-04-14T15:11:04.144 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:04 smithi090 ceph-mon[41670]: 3.0 scrub starts 2024-04-14T15:11:04.145 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:04 smithi090 ceph-mon[41670]: 3.0 scrub ok 2024-04-14T15:11:04.145 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:04 smithi090 ceph-mon[41670]: 2.14 scrub starts 2024-04-14T15:11:04.145 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:04 smithi090 ceph-mon[41670]: 2.14 scrub ok 2024-04-14T15:11:04.145 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:04 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:04.145 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:04 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:04.145 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:04 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-14T15:11:04.146 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:04 smithi090 ceph-mon[41670]: from='mon.0 v1:172.21.15.90:0/650083486' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-14T15:11:04.146 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:04 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:04.146 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:04 smithi090 ceph-mon[41670]: Deploying daemon grafana.a on smithi092 2024-04-14T15:11:04.146 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:04 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1294041795' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-14T15:11:04.147 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:04 smithi090 ceph-mon[30399]: 5.f scrub starts 2024-04-14T15:11:04.147 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:04 smithi090 ceph-mon[30399]: 5.f scrub ok 2024-04-14T15:11:04.147 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:04 smithi090 ceph-mon[30399]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-14T15:11:04.147 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:04 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:04.147 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:04 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:04.147 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:04 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:04.147 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:04 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:04.147 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:04 smithi090 ceph-mon[30399]: Regenerating cephadm self-signed grafana TLS certificates 2024-04-14T15:11:04.148 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:04 smithi090 ceph-mon[30399]: 3.0 scrub starts 2024-04-14T15:11:04.148 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:04 smithi090 ceph-mon[30399]: 3.0 scrub ok 2024-04-14T15:11:04.148 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:04 smithi090 ceph-mon[30399]: 2.14 scrub starts 2024-04-14T15:11:04.148 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:04 smithi090 ceph-mon[30399]: 2.14 scrub ok 2024-04-14T15:11:04.148 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:04 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:04.148 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:04 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:04.148 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:04 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-14T15:11:04.148 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:04 smithi090 ceph-mon[30399]: from='mon.0 v1:172.21.15.90:0/650083486' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-14T15:11:04.148 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:04 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:04.149 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:04 smithi090 ceph-mon[30399]: Deploying daemon grafana.a on smithi092 2024-04-14T15:11:04.149 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:04 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1294041795' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-14T15:11:04.168 INFO:tasks.cephadm.ceph_manager.ceph:need seq 176093659162 got 176093659167 for osd.5 2024-04-14T15:11:04.169 DEBUG:teuthology.parallel:result is None 2024-04-14T15:11:04.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:04 smithi092 ceph-mon[33966]: 5.f scrub starts 2024-04-14T15:11:04.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:04 smithi092 ceph-mon[33966]: 5.f scrub ok 2024-04-14T15:11:04.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:04 smithi092 ceph-mon[33966]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-14T15:11:04.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:04 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:04.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:04 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:04.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:04 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:04.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:04 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:04.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:04 smithi092 ceph-mon[33966]: Regenerating cephadm self-signed grafana TLS certificates 2024-04-14T15:11:04.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:04 smithi092 ceph-mon[33966]: 3.0 scrub starts 2024-04-14T15:11:04.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:04 smithi092 ceph-mon[33966]: 3.0 scrub ok 2024-04-14T15:11:04.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:04 smithi092 ceph-mon[33966]: 2.14 scrub starts 2024-04-14T15:11:04.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:04 smithi092 ceph-mon[33966]: 2.14 scrub ok 2024-04-14T15:11:04.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:04 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:04.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:04 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:04.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:04 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-14T15:11:04.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:04 smithi092 ceph-mon[33966]: from='mon.0 v1:172.21.15.90:0/650083486' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-14T15:11:04.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:04 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:04.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:04 smithi092 ceph-mon[33966]: Deploying daemon grafana.a on smithi092 2024-04-14T15:11:04.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:04 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1294041795' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-14T15:11:04.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:04.861+0000 7f927d032700 1 -- 172.21.15.90:0/2321766297 >> v1:172.21.15.90:6789/0 conn(0x7f927810f350 legacy=0x7f92781117f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:04.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:04.861+0000 7f927d032700 1 -- 172.21.15.90:0/2321766297 shutdown_connections 2024-04-14T15:11:04.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:04.861+0000 7f927d032700 1 -- 172.21.15.90:0/2321766297 >> 172.21.15.90:0/2321766297 conn(0x7f9278076a70 msgr2=0x7f9278076e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:11:04.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:04.861+0000 7f927d032700 1 -- 172.21.15.90:0/2321766297 shutdown_connections 2024-04-14T15:11:04.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:04.861+0000 7f927d032700 1 -- 172.21.15.90:0/2321766297 wait complete. 2024-04-14T15:11:04.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:04.862+0000 7f927d032700 1 librados: starting msgr at 2024-04-14T15:11:04.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:04.862+0000 7f927d032700 1 librados: starting objecter 2024-04-14T15:11:04.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:04.862+0000 7f927d032700 1 Processor -- start 2024-04-14T15:11:04.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:04.862+0000 7f927d032700 1 -- start start 2024-04-14T15:11:04.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:04.862+0000 7f927d032700 1 librados: setting wanted keys 2024-04-14T15:11:04.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:04.862+0000 7f927d032700 1 librados: calling monclient init 2024-04-14T15:11:04.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:04.862+0000 7f927d032700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f92781a9d50 con 0x7f927810f350 2024-04-14T15:11:04.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:04.862+0000 7f927d032700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f92781aaf30 con 0x7f92781013b0 2024-04-14T15:11:04.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:04.862+0000 7f927d032700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f92781ac170 con 0x7f927810b7b0 2024-04-14T15:11:04.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:04.862+0000 7f927659c700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f927810b7b0 0x7f92781a4bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:41444/0 (socket says 172.21.15.90:41444) 2024-04-14T15:11:04.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:04.862+0000 7f927659c700 1 -- 172.21.15.90:0/3244054673 learned_addr learned my addr 172.21.15.90:0/3244054673 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:11:04.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:04.863+0000 7f925ffff700 10 client.?.objecter ms_handle_connect 0x7f927810f350 2024-04-14T15:11:04.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:04.863+0000 7f925ffff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:11:04.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:04.863+0000 7f925ffff700 10 client.?.objecter ms_handle_connect 0x7f92781013b0 2024-04-14T15:11:04.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:04.863+0000 7f925ffff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:11:04.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:04.863+0000 7f925ffff700 10 client.?.objecter ms_handle_connect 0x7f927810b7b0 2024-04-14T15:11:04.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:04.863+0000 7f925ffff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:11:04.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:04.863+0000 7f925ffff700 1 -- 172.21.15.90:0/3244054673 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2891442792 0 0) 0x7f92781a9d50 con 0x7f927810f350 2024-04-14T15:11:04.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:04.863+0000 7f925ffff700 1 -- 172.21.15.90:0/3244054673 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f92600035b0 con 0x7f927810f350 2024-04-14T15:11:04.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:04.863+0000 7f925ffff700 1 -- 172.21.15.90:0/3244054673 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1278096793 0 0) 0x7f92781aaf30 con 0x7f92781013b0 2024-04-14T15:11:04.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:04.863+0000 7f925ffff700 1 -- 172.21.15.90:0/3244054673 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f92781a9d50 con 0x7f92781013b0 2024-04-14T15:11:04.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:04.863+0000 7f925ffff700 1 -- 172.21.15.90:0/3244054673 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1846807365 0 0) 0x7f92781ac170 con 0x7f927810b7b0 2024-04-14T15:11:04.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:04.863+0000 7f925ffff700 1 -- 172.21.15.90:0/3244054673 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f92781aaf30 con 0x7f927810b7b0 2024-04-14T15:11:04.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:04.863+0000 7f925ffff700 1 -- 172.21.15.90:0/3244054673 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1468865462 0 0) 0x7f92600035b0 con 0x7f927810f350 2024-04-14T15:11:04.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:04.863+0000 7f925ffff700 1 -- 172.21.15.90:0/3244054673 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f92781ac170 con 0x7f927810f350 2024-04-14T15:11:04.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:04.863+0000 7f925ffff700 1 -- 172.21.15.90:0/3244054673 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f92700017b0 con 0x7f927810f350 2024-04-14T15:11:04.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:04.863+0000 7f925ffff700 1 -- 172.21.15.90:0/3244054673 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1803818254 0 0) 0x7f92781a9d50 con 0x7f92781013b0 2024-04-14T15:11:04.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:04.863+0000 7f925ffff700 1 -- 172.21.15.90:0/3244054673 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f92600035b0 con 0x7f92781013b0 2024-04-14T15:11:04.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:04.863+0000 7f925ffff700 1 -- 172.21.15.90:0/3244054673 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3884038226 0 0) 0x7f92781aaf30 con 0x7f927810b7b0 2024-04-14T15:11:04.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:04.863+0000 7f925ffff700 1 -- 172.21.15.90:0/3244054673 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f92781a9d50 con 0x7f927810b7b0 2024-04-14T15:11:04.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:04.863+0000 7f925ffff700 1 -- 172.21.15.90:0/3244054673 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f9268001830 con 0x7f92781013b0 2024-04-14T15:11:04.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:04.863+0000 7f925ffff700 1 -- 172.21.15.90:0/3244054673 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f926c0017f0 con 0x7f927810b7b0 2024-04-14T15:11:04.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:04.863+0000 7f925ffff700 1 -- 172.21.15.90:0/3244054673 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3409807541 0 0) 0x7f92781ac170 con 0x7f927810f350 2024-04-14T15:11:04.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:04.863+0000 7f925ffff700 1 -- 172.21.15.90:0/3244054673 >> v1:172.21.15.90:6790/0 conn(0x7f927810b7b0 legacy=0x7f92781a4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:04.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:04.863+0000 7f925ffff700 1 -- 172.21.15.90:0/3244054673 >> v1:172.21.15.92:6789/0 conn(0x7f92781013b0 legacy=0x7f9278103d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:04.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:04.863+0000 7f925ffff700 1 -- 172.21.15.90:0/3244054673 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92781ad3b0 con 0x7f927810f350 2024-04-14T15:11:04.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:04.863+0000 7f927d032700 1 -- 172.21.15.90:0/3244054673 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f92781ab160 con 0x7f927810f350 2024-04-14T15:11:04.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:04.863+0000 7f927d032700 10 client.14736.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:11:04.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:04.863+0000 7f927d032700 1 -- 172.21.15.90:0/3244054673 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f92781ab6a0 con 0x7f927810f350 2024-04-14T15:11:04.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:04.863+0000 7f927d032700 1 librados: init done 2024-04-14T15:11:04.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:04.864+0000 7f925ffff700 1 -- 172.21.15.90:0/3244054673 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f9270001ef0 con 0x7f927810f350 2024-04-14T15:11:04.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:04.864+0000 7f925ffff700 1 -- 172.21.15.90:0/3244054673 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f9270005320 con 0x7f927810f350 2024-04-14T15:11:04.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:04.864+0000 7f927d032700 1 -- 172.21.15.90:0/3244054673 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9264005120 con 0x7f927810f350 2024-04-14T15:11:04.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:04.865+0000 7f925ffff700 1 -- 172.21.15.90:0/3244054673 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f927001bd30 con 0x7f927810f350 2024-04-14T15:11:04.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:04.865+0000 7f925ffff700 1 -- 172.21.15.90:0/3244054673 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(87..87 src has 12..87) ==== 6288+0+0 (unknown 1749939968 0 0) 0x7f92700893a0 con 0x7f927810f350 2024-04-14T15:11:04.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:04.865+0000 7f925ffff700 10 client.14736.objecter ms_dispatch 0x7f9278001550 osd_map(87..87 src has 12..87) 2024-04-14T15:11:04.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:04.865+0000 7f925ffff700 3 client.14736.objecter handle_osd_map got epochs [87,87] > 0 2024-04-14T15:11:04.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:04.865+0000 7f925ffff700 3 client.14736.objecter handle_osd_map decoding full epoch 87 2024-04-14T15:11:04.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:04.866+0000 7f925ffff700 20 client.14736.objecter dump_active .. 0 homeless 2024-04-14T15:11:04.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:04.866+0000 7f925ffff700 10 client.14736.objecter ms_handle_connect 0x7f92600715d0 2024-04-14T15:11:04.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:04.870+0000 7f925ffff700 1 -- 172.21.15.90:0/3244054673 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f9270056570 con 0x7f927810f350 2024-04-14T15:11:05.043 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:11:04 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-alertmanager-a[78793]: ts=2024-04-14T15:11:04.801Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.001005986s 2024-04-14T15:11:05.056 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:11:04 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:11:05.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.065+0000 7fae0ce82700 1 -- 172.21.15.90:0/3292701712 >> v1:172.21.15.90:6789/0 conn(0x7fae0810f3a0 legacy=0x7fae081117f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:05.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.065+0000 7fae0ce82700 1 -- 172.21.15.90:0/3292701712 shutdown_connections 2024-04-14T15:11:05.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.065+0000 7fae0ce82700 1 -- 172.21.15.90:0/3292701712 >> 172.21.15.90:0/3292701712 conn(0x7fae080fd290 msgr2=0x7fae080ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:11:05.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.065+0000 7fae0ce82700 1 -- 172.21.15.90:0/3292701712 shutdown_connections 2024-04-14T15:11:05.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.065+0000 7fae0ce82700 1 -- 172.21.15.90:0/3292701712 wait complete. 2024-04-14T15:11:05.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.066+0000 7fae0ce82700 1 librados: starting msgr at 2024-04-14T15:11:05.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.066+0000 7fae0ce82700 1 librados: starting objecter 2024-04-14T15:11:05.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.066+0000 7fae0ce82700 1 Processor -- start 2024-04-14T15:11:05.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.066+0000 7fae0ce82700 1 -- start start 2024-04-14T15:11:05.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.066+0000 7fae0ce82700 1 librados: setting wanted keys 2024-04-14T15:11:05.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.066+0000 7fae0ce82700 1 librados: calling monclient init 2024-04-14T15:11:05.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.066+0000 7fae0ce82700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fae081a78a0 con 0x7fae0810b890 2024-04-14T15:11:05.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.066+0000 7fae0ce82700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fae081a8ac0 con 0x7fae0810f3a0 2024-04-14T15:11:05.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.066+0000 7fae0ce82700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fae081a9d00 con 0x7fae08103a80 2024-04-14T15:11:05.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.066+0000 7fae05d9b700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fae0810b890 0x7fae081a2780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:40040/0 (socket says 172.21.15.90:40040) 2024-04-14T15:11:05.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.066+0000 7fae05d9b700 1 -- 172.21.15.90:0/3647845116 learned_addr learned my addr 172.21.15.90:0/3647845116 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:11:05.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.066+0000 7fae0659c700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7fae08103a80 0x7fae0810b000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:41456/0 (socket says 172.21.15.90:41456) 2024-04-14T15:11:05.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.066+0000 7fadf77fe700 10 client.?.objecter ms_handle_connect 0x7fae0810b890 2024-04-14T15:11:05.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.066+0000 7fadf77fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:11:05.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.066+0000 7fadf77fe700 10 client.?.objecter ms_handle_connect 0x7fae08103a80 2024-04-14T15:11:05.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.066+0000 7fadf77fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:11:05.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.066+0000 7fadf77fe700 10 client.?.objecter ms_handle_connect 0x7fae0810f3a0 2024-04-14T15:11:05.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.066+0000 7fadf77fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:11:05.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.066+0000 7fadf77fe700 1 -- 172.21.15.90:0/3647845116 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3379428836 0 0) 0x7fae081a9d00 con 0x7fae08103a80 2024-04-14T15:11:05.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.066+0000 7fadf77fe700 1 -- 172.21.15.90:0/3647845116 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fadf00031f0 con 0x7fae08103a80 2024-04-14T15:11:05.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.067+0000 7fadf77fe700 1 -- 172.21.15.90:0/3647845116 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1895121971 0 0) 0x7fae081a78a0 con 0x7fae0810b890 2024-04-14T15:11:05.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.067+0000 7fadf77fe700 1 -- 172.21.15.90:0/3647845116 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fae081a9d00 con 0x7fae0810b890 2024-04-14T15:11:05.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.067+0000 7fadf77fe700 1 -- 172.21.15.90:0/3647845116 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3805104046 0 0) 0x7fae081a8ac0 con 0x7fae0810f3a0 2024-04-14T15:11:05.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.067+0000 7fadf77fe700 1 -- 172.21.15.90:0/3647845116 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fae081a78a0 con 0x7fae0810f3a0 2024-04-14T15:11:05.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.067+0000 7fadf77fe700 1 -- 172.21.15.90:0/3647845116 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3666946140 0 0) 0x7fae081a78a0 con 0x7fae0810f3a0 2024-04-14T15:11:05.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.067+0000 7fadf77fe700 1 -- 172.21.15.90:0/3647845116 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fae081a8ac0 con 0x7fae0810f3a0 2024-04-14T15:11:05.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.067+0000 7fadf77fe700 1 -- 172.21.15.90:0/3647845116 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fae00001d10 con 0x7fae0810f3a0 2024-04-14T15:11:05.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.067+0000 7fadf77fe700 1 -- 172.21.15.90:0/3647845116 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1847386124 0 0) 0x7fae081a9d00 con 0x7fae0810b890 2024-04-14T15:11:05.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.067+0000 7fadf77fe700 1 -- 172.21.15.90:0/3647845116 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fae081a78a0 con 0x7fae0810b890 2024-04-14T15:11:05.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.067+0000 7fadf77fe700 1 -- 172.21.15.90:0/3647845116 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 102652592 0 0) 0x7fadf00031f0 con 0x7fae08103a80 2024-04-14T15:11:05.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.067+0000 7fadf77fe700 1 -- 172.21.15.90:0/3647845116 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fae081a9d00 con 0x7fae08103a80 2024-04-14T15:11:05.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.067+0000 7fadf77fe700 1 -- 172.21.15.90:0/3647845116 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fadf80019f0 con 0x7fae0810b890 2024-04-14T15:11:05.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.067+0000 7fadf77fe700 1 -- 172.21.15.90:0/3647845116 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fadfc000ed0 con 0x7fae08103a80 2024-04-14T15:11:05.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.067+0000 7fadf77fe700 1 -- 172.21.15.90:0/3647845116 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2972051060 0 0) 0x7fae081a8ac0 con 0x7fae0810f3a0 2024-04-14T15:11:05.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.067+0000 7fadf77fe700 1 -- 172.21.15.90:0/3647845116 >> v1:172.21.15.90:6790/0 conn(0x7fae08103a80 legacy=0x7fae0810b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:05.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.067+0000 7fadf77fe700 1 -- 172.21.15.90:0/3647845116 >> v1:172.21.15.90:6789/0 conn(0x7fae0810b890 legacy=0x7fae081a2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:05.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.067+0000 7fadf77fe700 1 -- 172.21.15.90:0/3647845116 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae081aaf40 con 0x7fae0810f3a0 2024-04-14T15:11:05.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.067+0000 7fae0ce82700 1 -- 172.21.15.90:0/3647845116 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fae081a9f30 con 0x7fae0810f3a0 2024-04-14T15:11:05.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.067+0000 7fae0ce82700 10 client.24631.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:11:05.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.067+0000 7fae0ce82700 1 -- 172.21.15.90:0/3647845116 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fae081aa3f0 con 0x7fae0810f3a0 2024-04-14T15:11:05.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.067+0000 7fae0ce82700 1 librados: init done 2024-04-14T15:11:05.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.067+0000 7fadf77fe700 1 -- 172.21.15.90:0/3647845116 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fae00002010 con 0x7fae0810f3a0 2024-04-14T15:11:05.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.067+0000 7fadf77fe700 1 -- 172.21.15.90:0/3647845116 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fae00004f40 con 0x7fae0810f3a0 2024-04-14T15:11:05.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.068+0000 7fae0ce82700 1 -- 172.21.15.90:0/3647845116 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fade8005120 con 0x7fae0810f3a0 2024-04-14T15:11:05.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.069+0000 7fadf77fe700 1 -- 172.21.15.90:0/3647845116 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fae00003a90 con 0x7fae0810f3a0 2024-04-14T15:11:05.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.069+0000 7fadf77fe700 1 -- 172.21.15.90:0/3647845116 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(87..87 src has 12..87) ==== 6288+0+0 (unknown 1749939968 0 0) 0x7fae00089320 con 0x7fae0810f3a0 2024-04-14T15:11:05.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.069+0000 7fadf77fe700 10 client.24631.objecter ms_dispatch 0x7fae08001550 osd_map(87..87 src has 12..87) 2024-04-14T15:11:05.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.069+0000 7fadf77fe700 3 client.24631.objecter handle_osd_map got epochs [87,87] > 0 2024-04-14T15:11:05.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.069+0000 7fadf77fe700 3 client.24631.objecter handle_osd_map decoding full epoch 87 2024-04-14T15:11:05.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.069+0000 7fadf77fe700 20 client.24631.objecter dump_active .. 0 homeless 2024-04-14T15:11:05.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.070+0000 7fadf77fe700 10 client.24631.objecter ms_handle_connect 0x7fadf0071340 2024-04-14T15:11:05.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.074+0000 7fadf77fe700 1 -- 172.21.15.90:0/3647845116 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fae00056490 con 0x7fae0810f3a0 2024-04-14T15:11:05.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.080+0000 7f927d032700 1 -- 172.21.15.90:0/3244054673 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f9264005410 con 0x7f927810f350 2024-04-14T15:11:05.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.081+0000 7f925ffff700 1 -- 172.21.15.90:0/3244054673 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (unknown 92182286 0 2868170512) 0x7f927005a220 con 0x7f927810f350 2024-04-14T15:11:05.081 INFO:teuthology.orchestra.run.smithi090.stdout:77309411375 2024-04-14T15:11:05.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.083+0000 7f927d032700 10 librados: watch_flush enter 2024-04-14T15:11:05.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.083+0000 7f927d032700 10 librados: watch_flush exit 2024-04-14T15:11:05.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.083+0000 7f927d032700 20 client.14736.objecter shutdown clearing up homeless session... 2024-04-14T15:11:05.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.083+0000 7f927d032700 10 client.14736.objecter successfully canceled tick 2024-04-14T15:11:05.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.083+0000 7f927d032700 1 -- 172.21.15.90:0/3244054673 >> v1:172.21.15.90:6800/420935427 conn(0x7f92600715d0 legacy=0x7f9260073a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:05.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.083+0000 7f927d032700 1 -- 172.21.15.90:0/3244054673 >> v1:172.21.15.90:6789/0 conn(0x7f927810f350 legacy=0x7f92781a83e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:05.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.083+0000 7f927d032700 1 -- 172.21.15.90:0/3244054673 shutdown_connections 2024-04-14T15:11:05.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.083+0000 7f927d032700 1 -- 172.21.15.90:0/3244054673 >> 172.21.15.90:0/3244054673 conn(0x7f9278076a70 msgr2=0x7f92781117b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:11:05.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.083+0000 7f927d032700 1 -- 172.21.15.90:0/3244054673 shutdown_connections 2024-04-14T15:11:05.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.083+0000 7f927d032700 1 -- 172.21.15.90:0/3244054673 wait complete. 2024-04-14T15:11:05.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.083+0000 7f927d032700 1 librados: shutdown 2024-04-14T15:11:05.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.277+0000 7fae0ce82700 1 -- 172.21.15.90:0/3647845116 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fade8005410 con 0x7fae0810f3a0 2024-04-14T15:11:05.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.277+0000 7fadf77fe700 1 -- 172.21.15.90:0/3647845116 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (unknown 1823589948 0 2012961777) 0x7fae00004f40 con 0x7fae0810f3a0 2024-04-14T15:11:05.278 INFO:teuthology.orchestra.run.smithi090.stdout:133143986211 2024-04-14T15:11:05.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.279+0000 7fae0ce82700 10 librados: watch_flush enter 2024-04-14T15:11:05.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.279+0000 7fae0ce82700 10 librados: watch_flush exit 2024-04-14T15:11:05.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.279+0000 7fae0ce82700 20 client.24631.objecter shutdown clearing up homeless session... 2024-04-14T15:11:05.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.279+0000 7fae0ce82700 10 client.24631.objecter successfully canceled tick 2024-04-14T15:11:05.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.279+0000 7fae0ce82700 1 -- 172.21.15.90:0/3647845116 >> v1:172.21.15.90:6800/420935427 conn(0x7fadf0071340 legacy=0x7fadf0073800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:05.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.279+0000 7fae0ce82700 1 -- 172.21.15.90:0/3647845116 >> v1:172.21.15.92:6789/0 conn(0x7fae0810f3a0 legacy=0x7fae081a5f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:05.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.279+0000 7fae0ce82700 1 -- 172.21.15.90:0/3647845116 shutdown_connections 2024-04-14T15:11:05.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.279+0000 7fae0ce82700 1 -- 172.21.15.90:0/3647845116 >> 172.21.15.90:0/3647845116 conn(0x7fae080fd290 msgr2=0x7fae080ff680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:11:05.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.280+0000 7fae0ce82700 1 -- 172.21.15.90:0/3647845116 shutdown_connections 2024-04-14T15:11:05.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.280+0000 7fae0ce82700 1 -- 172.21.15.90:0/3647845116 wait complete. 2024-04-14T15:11:05.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.280+0000 7fae0ce82700 1 librados: shutdown 2024-04-14T15:11:05.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:05 smithi092 ceph-mon[33966]: 6.19 scrub starts 2024-04-14T15:11:05.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:05 smithi092 ceph-mon[33966]: 6.19 scrub ok 2024-04-14T15:11:05.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:05 smithi092 ceph-mon[33966]: 2.d scrub starts 2024-04-14T15:11:05.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:05 smithi092 ceph-mon[33966]: 2.d scrub ok 2024-04-14T15:11:05.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:05 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:11:05.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:05 smithi090 ceph-mon[41670]: 6.19 scrub starts 2024-04-14T15:11:05.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:05 smithi090 ceph-mon[41670]: 6.19 scrub ok 2024-04-14T15:11:05.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:05 smithi090 ceph-mon[41670]: 2.d scrub starts 2024-04-14T15:11:05.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:05 smithi090 ceph-mon[41670]: 2.d scrub ok 2024-04-14T15:11:05.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:05 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:11:05.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:05 smithi090 ceph-mon[30399]: 6.19 scrub starts 2024-04-14T15:11:05.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:05 smithi090 ceph-mon[30399]: 6.19 scrub ok 2024-04-14T15:11:05.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:05 smithi090 ceph-mon[30399]: 2.d scrub starts 2024-04-14T15:11:05.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:05 smithi090 ceph-mon[30399]: 2.d scrub ok 2024-04-14T15:11:05.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:05 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:11:05.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.744+0000 7f4510386700 1 -- 172.21.15.90:0/1592246557 >> v1:172.21.15.90:6789/0 conn(0x7f4508107760 legacy=0x7f4508107b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:05.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.744+0000 7f4510386700 1 -- 172.21.15.90:0/1592246557 shutdown_connections 2024-04-14T15:11:05.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.744+0000 7f4510386700 1 -- 172.21.15.90:0/1592246557 >> 172.21.15.90:0/1592246557 conn(0x7f45080fd290 msgr2=0x7f45080ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:11:05.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.745+0000 7f4510386700 1 -- 172.21.15.90:0/1592246557 shutdown_connections 2024-04-14T15:11:05.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.745+0000 7f4510386700 1 -- 172.21.15.90:0/1592246557 wait complete. 2024-04-14T15:11:05.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.745+0000 7f4510386700 1 librados: starting msgr at 2024-04-14T15:11:05.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.745+0000 7f4510386700 1 librados: starting objecter 2024-04-14T15:11:05.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.745+0000 7f4510386700 1 Processor -- start 2024-04-14T15:11:05.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.745+0000 7f4510386700 1 -- start start 2024-04-14T15:11:05.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.745+0000 7f4510386700 1 librados: setting wanted keys 2024-04-14T15:11:05.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.745+0000 7f4510386700 1 librados: calling monclient init 2024-04-14T15:11:05.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.745+0000 7f4510386700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f450806b1d0 con 0x7f450810e1e0 2024-04-14T15:11:05.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.745+0000 7f4510386700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f450806b3a0 con 0x7f450810a610 2024-04-14T15:11:05.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.745+0000 7f4510386700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f45081a9e70 con 0x7f4508107760 2024-04-14T15:11:05.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.746+0000 7f450e923700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f450810e1e0 0x7f45081a66d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:40056/0 (socket says 172.21.15.90:40056) 2024-04-14T15:11:05.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.746+0000 7f450e122700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f4508107760 0x7f450810cc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:41484/0 (socket says 172.21.15.90:41484) 2024-04-14T15:11:05.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.746+0000 7f450e923700 1 -- 172.21.15.90:0/3063772151 learned_addr learned my addr 172.21.15.90:0/3063772151 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:11:05.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.746+0000 7f44ff7fe700 10 client.?.objecter ms_handle_connect 0x7f450810e1e0 2024-04-14T15:11:05.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.746+0000 7f44ff7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:11:05.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.746+0000 7f44ff7fe700 10 client.?.objecter ms_handle_connect 0x7f450810a610 2024-04-14T15:11:05.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.746+0000 7f44ff7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:11:05.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.746+0000 7f44ff7fe700 10 client.?.objecter ms_handle_connect 0x7f4508107760 2024-04-14T15:11:05.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.746+0000 7f44ff7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:11:05.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.746+0000 7f44ff7fe700 1 -- 172.21.15.90:0/3063772151 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 731590079 0 0) 0x7f450806b1d0 con 0x7f450810e1e0 2024-04-14T15:11:05.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.746+0000 7f44ff7fe700 1 -- 172.21.15.90:0/3063772151 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f44f40035b0 con 0x7f450810e1e0 2024-04-14T15:11:05.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.746+0000 7f44ff7fe700 1 -- 172.21.15.90:0/3063772151 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3730602716 0 0) 0x7f450806b3a0 con 0x7f450810a610 2024-04-14T15:11:05.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.746+0000 7f44ff7fe700 1 -- 172.21.15.90:0/3063772151 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f450806b1d0 con 0x7f450810a610 2024-04-14T15:11:05.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.746+0000 7f44ff7fe700 1 -- 172.21.15.90:0/3063772151 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2693634317 0 0) 0x7f45081a9e70 con 0x7f4508107760 2024-04-14T15:11:05.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.746+0000 7f44ff7fe700 1 -- 172.21.15.90:0/3063772151 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f450806b3a0 con 0x7f4508107760 2024-04-14T15:11:05.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.746+0000 7f44ff7fe700 1 -- 172.21.15.90:0/3063772151 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2967286976 0 0) 0x7f450806b1d0 con 0x7f450810a610 2024-04-14T15:11:05.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.746+0000 7f44ff7fe700 1 -- 172.21.15.90:0/3063772151 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f45081a9e70 con 0x7f450810a610 2024-04-14T15:11:05.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.746+0000 7f44ff7fe700 1 -- 172.21.15.90:0/3063772151 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 305906726 0 0) 0x7f44f40035b0 con 0x7f450810e1e0 2024-04-14T15:11:05.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.746+0000 7f44ff7fe700 1 -- 172.21.15.90:0/3063772151 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f450806b1d0 con 0x7f450810e1e0 2024-04-14T15:11:05.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.746+0000 7f44ff7fe700 1 -- 172.21.15.90:0/3063772151 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1670367298 0 0) 0x7f450806b3a0 con 0x7f4508107760 2024-04-14T15:11:05.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.746+0000 7f44ff7fe700 1 -- 172.21.15.90:0/3063772151 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f44f40035b0 con 0x7f4508107760 2024-04-14T15:11:05.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.746+0000 7f44ff7fe700 1 -- 172.21.15.90:0/3063772151 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f44f8001a70 con 0x7f450810a610 2024-04-14T15:11:05.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.746+0000 7f44ff7fe700 1 -- 172.21.15.90:0/3063772151 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f4500001c10 con 0x7f450810e1e0 2024-04-14T15:11:05.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.746+0000 7f44ff7fe700 1 -- 172.21.15.90:0/3063772151 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f4504004540 con 0x7f4508107760 2024-04-14T15:11:05.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.746+0000 7f44ff7fe700 1 -- 172.21.15.90:0/3063772151 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1687474136 0 0) 0x7f45081a9e70 con 0x7f450810a610 2024-04-14T15:11:05.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.746+0000 7f44ff7fe700 1 -- 172.21.15.90:0/3063772151 >> v1:172.21.15.90:6790/0 conn(0x7f4508107760 legacy=0x7f450810cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:05.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.746+0000 7f44ff7fe700 1 -- 172.21.15.90:0/3063772151 >> v1:172.21.15.90:6789/0 conn(0x7f450810e1e0 legacy=0x7f45081a66d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:05.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.747+0000 7f44ff7fe700 1 -- 172.21.15.90:0/3063772151 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45081ab0b0 con 0x7f450810a610 2024-04-14T15:11:05.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.747+0000 7f4510386700 1 -- 172.21.15.90:0/3063772151 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f45081aa0a0 con 0x7f450810a610 2024-04-14T15:11:05.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.747+0000 7f4510386700 10 client.24637.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:11:05.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.747+0000 7f4510386700 1 -- 172.21.15.90:0/3063772151 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f45081aa560 con 0x7f450810a610 2024-04-14T15:11:05.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.747+0000 7f4510386700 1 librados: init done 2024-04-14T15:11:05.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.747+0000 7f44ff7fe700 1 -- 172.21.15.90:0/3063772151 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f44f8003b20 con 0x7f450810a610 2024-04-14T15:11:05.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.747+0000 7f44ff7fe700 1 -- 172.21.15.90:0/3063772151 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f44f80049b0 con 0x7f450810a610 2024-04-14T15:11:05.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.748+0000 7f4510386700 1 -- 172.21.15.90:0/3063772151 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44ec005150 con 0x7f450810a610 2024-04-14T15:11:05.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.748+0000 7f44ff7fe700 1 -- 172.21.15.90:0/3063772151 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f44f8003680 con 0x7f450810a610 2024-04-14T15:11:05.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.749+0000 7f44ff7fe700 1 -- 172.21.15.90:0/3063772151 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(87..87 src has 12..87) ==== 6288+0+0 (unknown 1749939968 0 0) 0x7f44f8089110 con 0x7f450810a610 2024-04-14T15:11:05.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.749+0000 7f44ff7fe700 10 client.24637.objecter ms_dispatch 0x7f4508001550 osd_map(87..87 src has 12..87) 2024-04-14T15:11:05.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.749+0000 7f44ff7fe700 3 client.24637.objecter handle_osd_map got epochs [87,87] > 0 2024-04-14T15:11:05.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.749+0000 7f44ff7fe700 3 client.24637.objecter handle_osd_map decoding full epoch 87 2024-04-14T15:11:05.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.749+0000 7f44ff7fe700 20 client.24637.objecter dump_active .. 0 homeless 2024-04-14T15:11:05.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.749+0000 7f44ff7fe700 10 client.24637.objecter ms_handle_connect 0x7f44f4071510 2024-04-14T15:11:05.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.754+0000 7f44ff7fe700 1 -- 172.21.15.90:0/3063772151 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f44f8056280 con 0x7f450810a610 2024-04-14T15:11:05.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.959+0000 7f4510386700 1 -- 172.21.15.90:0/3063772151 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f44ec005440 con 0x7f450810a610 2024-04-14T15:11:05.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.959+0000 7f44ff7fe700 1 -- 172.21.15.90:0/3063772151 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (unknown 574334944 0 1451115110) 0x7f44f8059f30 con 0x7f450810a610 2024-04-14T15:11:05.960 INFO:teuthology.orchestra.run.smithi090.stdout:34359738424 2024-04-14T15:11:05.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.961+0000 7f4510386700 10 librados: watch_flush enter 2024-04-14T15:11:05.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.961+0000 7f4510386700 10 librados: watch_flush exit 2024-04-14T15:11:05.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.961+0000 7f4510386700 20 client.24637.objecter shutdown clearing up homeless session... 2024-04-14T15:11:05.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.961+0000 7f4510386700 10 client.24637.objecter successfully canceled tick 2024-04-14T15:11:05.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.961+0000 7f4510386700 1 -- 172.21.15.90:0/3063772151 >> v1:172.21.15.90:6800/420935427 conn(0x7f44f4071510 legacy=0x7f44f40739d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:05.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.961+0000 7f4510386700 1 -- 172.21.15.90:0/3063772151 >> v1:172.21.15.92:6789/0 conn(0x7f450810a610 legacy=0x7f450810d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:05.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.961+0000 7f4510386700 1 -- 172.21.15.90:0/3063772151 shutdown_connections 2024-04-14T15:11:05.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.961+0000 7f4510386700 1 -- 172.21.15.90:0/3063772151 >> 172.21.15.90:0/3063772151 conn(0x7f45080fd290 msgr2=0x7f45081003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:11:05.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.961+0000 7f4510386700 1 -- 172.21.15.90:0/3063772151 shutdown_connections 2024-04-14T15:11:05.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.961+0000 7f4510386700 1 -- 172.21.15.90:0/3063772151 wait complete. 2024-04-14T15:11:05.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:05.961+0000 7f4510386700 1 librados: shutdown 2024-04-14T15:11:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:06 smithi092 ceph-mon[33966]: 6.1e scrub starts 2024-04-14T15:11:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:06 smithi092 ceph-mon[33966]: 6.1e scrub ok 2024-04-14T15:11:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:06 smithi092 ceph-mon[33966]: 5.1a scrub starts 2024-04-14T15:11:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:06 smithi092 ceph-mon[33966]: 5.1a scrub ok 2024-04-14T15:11:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:06 smithi092 ceph-mon[33966]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:11:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:06 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3244054673' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-14T15:11:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:06 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3647845116' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-14T15:11:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:06 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3063772151' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-14T15:11:06.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:06 smithi090 ceph-mon[41670]: 6.1e scrub starts 2024-04-14T15:11:06.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:06 smithi090 ceph-mon[41670]: 6.1e scrub ok 2024-04-14T15:11:06.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:06 smithi090 ceph-mon[41670]: 5.1a scrub starts 2024-04-14T15:11:06.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:06 smithi090 ceph-mon[41670]: 5.1a scrub ok 2024-04-14T15:11:06.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:06 smithi090 ceph-mon[41670]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:11:06.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:06 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3244054673' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-14T15:11:06.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:06 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3647845116' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-14T15:11:06.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:06 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3063772151' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-14T15:11:06.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:06 smithi090 ceph-mon[30399]: 6.1e scrub starts 2024-04-14T15:11:06.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:06 smithi090 ceph-mon[30399]: 6.1e scrub ok 2024-04-14T15:11:06.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:06 smithi090 ceph-mon[30399]: 5.1a scrub starts 2024-04-14T15:11:06.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:06 smithi090 ceph-mon[30399]: 5.1a scrub ok 2024-04-14T15:11:06.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:06 smithi090 ceph-mon[30399]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:11:06.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:06 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3244054673' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-14T15:11:06.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:06 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3647845116' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-14T15:11:06.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:06 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3063772151' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-14T15:11:06.509 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986206 got 133143986211 for osd.4 2024-04-14T15:11:06.510 DEBUG:teuthology.parallel:result is None 2024-04-14T15:11:07.151 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:07 smithi090 ceph-mon[41670]: 5.10 scrub starts 2024-04-14T15:11:07.151 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:07 smithi090 ceph-mon[41670]: 5.10 scrub ok 2024-04-14T15:11:07.151 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:07 smithi090 ceph-mon[30399]: 5.10 scrub starts 2024-04-14T15:11:07.151 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:07 smithi090 ceph-mon[30399]: 5.10 scrub ok 2024-04-14T15:11:07.185 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738419 got 34359738424 for osd.0 2024-04-14T15:11:07.185 DEBUG:teuthology.parallel:result is None 2024-04-14T15:11:07.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:07 smithi092 ceph-mon[33966]: 5.10 scrub starts 2024-04-14T15:11:07.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:07 smithi092 ceph-mon[33966]: 5.10 scrub ok 2024-04-14T15:11:07.670 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411369 got 77309411375 for osd.2 2024-04-14T15:11:07.671 DEBUG:teuthology.parallel:result is None 2024-04-14T15:11:07.671 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-14T15:11:07.671 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph pg dump --format=json 2024-04-14T15:11:07.916 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:11:08.293 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:08 smithi090 ceph-mon[41670]: 4.11 scrub starts 2024-04-14T15:11:08.293 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:08 smithi090 ceph-mon[41670]: 4.11 scrub ok 2024-04-14T15:11:08.293 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:08 smithi090 ceph-mon[41670]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:11:08.293 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:08 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:08.293 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:08 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:11:08.294 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:08 smithi090 ceph-mon[30399]: 4.11 scrub starts 2024-04-14T15:11:08.294 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:08 smithi090 ceph-mon[30399]: 4.11 scrub ok 2024-04-14T15:11:08.294 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:08 smithi090 ceph-mon[30399]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:11:08.294 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:08 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:08.294 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:08 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:11:08.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:08 smithi092 ceph-mon[33966]: 4.11 scrub starts 2024-04-14T15:11:08.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:08 smithi092 ceph-mon[33966]: 4.11 scrub ok 2024-04-14T15:11:08.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:08 smithi092 ceph-mon[33966]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:11:08.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:08 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:08.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:08 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:11:09.338 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:09 smithi090 ceph-mon[41670]: 5.17 scrub starts 2024-04-14T15:11:09.338 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:09 smithi090 ceph-mon[41670]: 5.17 scrub ok 2024-04-14T15:11:09.338 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:09 smithi090 ceph-mon[41670]: 3.1 scrub starts 2024-04-14T15:11:09.338 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:09 smithi090 ceph-mon[41670]: 3.1 scrub ok 2024-04-14T15:11:09.338 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:09 smithi090 ceph-mon[41670]: 6.12 scrub starts 2024-04-14T15:11:09.338 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:09 smithi090 ceph-mon[41670]: 6.12 scrub ok 2024-04-14T15:11:09.338 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:09 smithi090 ceph-mon[41670]: 2.16 deep-scrub starts 2024-04-14T15:11:09.338 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:09 smithi090 ceph-mon[41670]: 2.16 deep-scrub ok 2024-04-14T15:11:09.338 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:09 smithi090 ceph-mon[30399]: 5.17 scrub starts 2024-04-14T15:11:09.339 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:09 smithi090 ceph-mon[30399]: 5.17 scrub ok 2024-04-14T15:11:09.339 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:09 smithi090 ceph-mon[30399]: 3.1 scrub starts 2024-04-14T15:11:09.339 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:09 smithi090 ceph-mon[30399]: 3.1 scrub ok 2024-04-14T15:11:09.339 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:09 smithi090 ceph-mon[30399]: 6.12 scrub starts 2024-04-14T15:11:09.339 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:09 smithi090 ceph-mon[30399]: 6.12 scrub ok 2024-04-14T15:11:09.339 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:09 smithi090 ceph-mon[30399]: 2.16 deep-scrub starts 2024-04-14T15:11:09.339 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:09 smithi090 ceph-mon[30399]: 2.16 deep-scrub ok 2024-04-14T15:11:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:09 smithi092 ceph-mon[33966]: 5.17 scrub starts 2024-04-14T15:11:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:09 smithi092 ceph-mon[33966]: 5.17 scrub ok 2024-04-14T15:11:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:09 smithi092 ceph-mon[33966]: 3.1 scrub starts 2024-04-14T15:11:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:09 smithi092 ceph-mon[33966]: 3.1 scrub ok 2024-04-14T15:11:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:09 smithi092 ceph-mon[33966]: 6.12 scrub starts 2024-04-14T15:11:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:09 smithi092 ceph-mon[33966]: 6.12 scrub ok 2024-04-14T15:11:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:09 smithi092 ceph-mon[33966]: 2.16 deep-scrub starts 2024-04-14T15:11:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:09 smithi092 ceph-mon[33966]: 2.16 deep-scrub ok 2024-04-14T15:11:09.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.596+0000 7fb5227c4700 1 -- 172.21.15.90:0/3805274477 >> v1:172.21.15.90:6789/0 conn(0x7fb51c10a630 legacy=0x7fb51c10ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:09.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.596+0000 7fb5227c4700 1 -- 172.21.15.90:0/3805274477 shutdown_connections 2024-04-14T15:11:09.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.596+0000 7fb5227c4700 1 -- 172.21.15.90:0/3805274477 >> 172.21.15.90:0/3805274477 conn(0x7fb51c0fd270 msgr2=0x7fb51c0ff690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:11:09.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.597+0000 7fb5227c4700 1 -- 172.21.15.90:0/3805274477 shutdown_connections 2024-04-14T15:11:09.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.597+0000 7fb5227c4700 1 -- 172.21.15.90:0/3805274477 wait complete. 2024-04-14T15:11:09.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.597+0000 7fb5227c4700 1 librados: starting msgr at 2024-04-14T15:11:09.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.597+0000 7fb5227c4700 1 librados: starting objecter 2024-04-14T15:11:09.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.597+0000 7fb5227c4700 1 Processor -- start 2024-04-14T15:11:09.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.597+0000 7fb5227c4700 1 -- start start 2024-04-14T15:11:09.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.597+0000 7fb5227c4700 1 librados: setting wanted keys 2024-04-14T15:11:09.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.597+0000 7fb5227c4700 1 librados: calling monclient init 2024-04-14T15:11:09.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.598+0000 7fb5227c4700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb51c10de60 con 0x7fb51c10a630 2024-04-14T15:11:09.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.598+0000 7fb5227c4700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb51c1ab210 con 0x7fb51c107780 2024-04-14T15:11:09.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.598+0000 7fb5227c4700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb51c1ac3f0 con 0x7fb51c10e200 2024-04-14T15:11:09.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.598+0000 7fb520d61700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7fb51c10e200 0x7fb51c1a8ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:41510/0 (socket says 172.21.15.90:41510) 2024-04-14T15:11:09.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.598+0000 7fb520d61700 1 -- 172.21.15.90:0/4176607233 learned_addr learned my addr 172.21.15.90:0/4176607233 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:11:09.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.598+0000 7fb51b7fe700 1 --1- 172.21.15.90:0/4176607233 >> v1:172.21.15.90:6789/0 conn(0x7fb51c10a630 0x7fb51c06b5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:40078/0 (socket says 172.21.15.90:40078) 2024-04-14T15:11:09.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.598+0000 7fb5197fa700 10 client.?.objecter ms_handle_connect 0x7fb51c10a630 2024-04-14T15:11:09.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.598+0000 7fb5197fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:11:09.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.598+0000 7fb5197fa700 10 client.?.objecter ms_handle_connect 0x7fb51c107780 2024-04-14T15:11:09.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.598+0000 7fb5197fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:11:09.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.598+0000 7fb5197fa700 10 client.?.objecter ms_handle_connect 0x7fb51c10e200 2024-04-14T15:11:09.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.598+0000 7fb5197fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:11:09.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.598+0000 7fb5197fa700 1 -- 172.21.15.90:0/4176607233 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2773193525 0 0) 0x7fb51c1ab210 con 0x7fb51c107780 2024-04-14T15:11:09.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.598+0000 7fb5197fa700 1 -- 172.21.15.90:0/4176607233 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb5040035b0 con 0x7fb51c107780 2024-04-14T15:11:09.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.598+0000 7fb5197fa700 1 -- 172.21.15.90:0/4176607233 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1571866797 0 0) 0x7fb51c10de60 con 0x7fb51c10a630 2024-04-14T15:11:09.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.598+0000 7fb5197fa700 1 -- 172.21.15.90:0/4176607233 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb51c1ab210 con 0x7fb51c10a630 2024-04-14T15:11:09.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.598+0000 7fb5197fa700 1 -- 172.21.15.90:0/4176607233 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 159260252 0 0) 0x7fb51c1ac3f0 con 0x7fb51c10e200 2024-04-14T15:11:09.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.598+0000 7fb5197fa700 1 -- 172.21.15.90:0/4176607233 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb51c10de60 con 0x7fb51c10e200 2024-04-14T15:11:09.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.598+0000 7fb5197fa700 1 -- 172.21.15.90:0/4176607233 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 663201817 0 0) 0x7fb5040035b0 con 0x7fb51c107780 2024-04-14T15:11:09.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.598+0000 7fb5197fa700 1 -- 172.21.15.90:0/4176607233 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb51c1ac3f0 con 0x7fb51c107780 2024-04-14T15:11:09.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.598+0000 7fb5197fa700 1 -- 172.21.15.90:0/4176607233 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fb50c0018b0 con 0x7fb51c107780 2024-04-14T15:11:09.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.598+0000 7fb5197fa700 1 -- 172.21.15.90:0/4176607233 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1798101927 0 0) 0x7fb51c10de60 con 0x7fb51c10e200 2024-04-14T15:11:09.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.598+0000 7fb5197fa700 1 -- 172.21.15.90:0/4176607233 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb5040035b0 con 0x7fb51c10e200 2024-04-14T15:11:09.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.598+0000 7fb5197fa700 1 -- 172.21.15.90:0/4176607233 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fb514001c30 con 0x7fb51c10e200 2024-04-14T15:11:09.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.598+0000 7fb5197fa700 1 -- 172.21.15.90:0/4176607233 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3858249065 0 0) 0x7fb51c1ab210 con 0x7fb51c10a630 2024-04-14T15:11:09.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.598+0000 7fb5197fa700 1 -- 172.21.15.90:0/4176607233 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb51c10de60 con 0x7fb51c10a630 2024-04-14T15:11:09.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.598+0000 7fb5197fa700 1 -- 172.21.15.90:0/4176607233 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fb5100015d0 con 0x7fb51c10a630 2024-04-14T15:11:09.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.598+0000 7fb5197fa700 1 -- 172.21.15.90:0/4176607233 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2700547299 0 0) 0x7fb51c1ac3f0 con 0x7fb51c107780 2024-04-14T15:11:09.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.598+0000 7fb5197fa700 1 -- 172.21.15.90:0/4176607233 >> v1:172.21.15.90:6790/0 conn(0x7fb51c10e200 legacy=0x7fb51c1a8ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:09.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.598+0000 7fb5197fa700 1 -- 172.21.15.90:0/4176607233 >> v1:172.21.15.90:6789/0 conn(0x7fb51c10a630 legacy=0x7fb51c06b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:09.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.598+0000 7fb5197fa700 1 -- 172.21.15.90:0/4176607233 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb51c1ad5d0 con 0x7fb51c107780 2024-04-14T15:11:09.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.599+0000 7fb5227c4700 1 -- 172.21.15.90:0/4176607233 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb51c1ab440 con 0x7fb51c107780 2024-04-14T15:11:09.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.599+0000 7fb5227c4700 10 client.24643.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:11:09.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.599+0000 7fb5227c4700 1 -- 172.21.15.90:0/4176607233 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb51c1ab900 con 0x7fb51c107780 2024-04-14T15:11:09.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.599+0000 7fb5227c4700 1 librados: init done 2024-04-14T15:11:09.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.599+0000 7fb5197fa700 1 -- 172.21.15.90:0/4176607233 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fb50c003b40 con 0x7fb51c107780 2024-04-14T15:11:09.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.599+0000 7fb5197fa700 1 -- 172.21.15.90:0/4176607233 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fb50c004a80 con 0x7fb51c107780 2024-04-14T15:11:09.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.600+0000 7fb5227c4700 1 -- 172.21.15.90:0/4176607233 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb508005120 con 0x7fb51c107780 2024-04-14T15:11:09.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.600+0000 7fb5197fa700 1 -- 172.21.15.90:0/4176607233 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fb50c0036a0 con 0x7fb51c107780 2024-04-14T15:11:09.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.601+0000 7fb5197fa700 1 -- 172.21.15.90:0/4176607233 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(87..87 src has 12..87) ==== 6288+0+0 (unknown 1749939968 0 0) 0x7fb50c089090 con 0x7fb51c107780 2024-04-14T15:11:09.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.601+0000 7fb5197fa700 10 client.24643.objecter ms_dispatch 0x7fb51c001550 osd_map(87..87 src has 12..87) 2024-04-14T15:11:09.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.601+0000 7fb5197fa700 3 client.24643.objecter handle_osd_map got epochs [87,87] > 0 2024-04-14T15:11:09.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.601+0000 7fb5197fa700 3 client.24643.objecter handle_osd_map decoding full epoch 87 2024-04-14T15:11:09.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.601+0000 7fb5197fa700 20 client.24643.objecter dump_active .. 0 homeless 2024-04-14T15:11:09.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.601+0000 7fb5197fa700 10 client.24643.objecter ms_handle_connect 0x7fb5040713b0 2024-04-14T15:11:09.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.606+0000 7fb5197fa700 1 -- 172.21.15.90:0/4176607233 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fb50c056200 con 0x7fb51c107780 2024-04-14T15:11:09.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.800+0000 7fb5227c4700 1 -- 172.21.15.90:0/4176607233 --> v1:172.21.15.90:6800/420935427 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb508000b90 con 0x7fb5040713b0 2024-04-14T15:11:09.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.810+0000 7fb5197fa700 1 -- 172.21.15.90:0/4176607233 <== mgr.14610 v1:172.21.15.90:6800/420935427 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+345451 (unknown 2965378022 0 3132230054) 0x7fb508000b90 con 0x7fb5040713b0 2024-04-14T15:11:09.815 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:11:09.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.813+0000 7fb5227c4700 10 librados: watch_flush enter 2024-04-14T15:11:09.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.813+0000 7fb5227c4700 10 librados: watch_flush exit 2024-04-14T15:11:09.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.813+0000 7fb5227c4700 20 client.24643.objecter shutdown clearing up homeless session... 2024-04-14T15:11:09.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.813+0000 7fb5227c4700 10 client.24643.objecter successfully canceled tick 2024-04-14T15:11:09.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.813+0000 7fb5227c4700 1 -- 172.21.15.90:0/4176607233 >> v1:172.21.15.90:6800/420935427 conn(0x7fb5040713b0 legacy=0x7fb504073870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:09.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.813+0000 7fb5227c4700 1 -- 172.21.15.90:0/4176607233 >> v1:172.21.15.92:6789/0 conn(0x7fb51c107780 legacy=0x7fb51c06ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:09.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.813+0000 7fb5227c4700 1 -- 172.21.15.90:0/4176607233 shutdown_connections 2024-04-14T15:11:09.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.813+0000 7fb5227c4700 1 -- 172.21.15.90:0/4176607233 >> 172.21.15.90:0/4176607233 conn(0x7fb51c0fd270 msgr2=0x7fb51c111840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:11:09.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.813+0000 7fb5227c4700 1 -- 172.21.15.90:0/4176607233 shutdown_connections 2024-04-14T15:11:09.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.813+0000 7fb5227c4700 1 -- 172.21.15.90:0/4176607233 wait complete. 2024-04-14T15:11:09.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:09.813+0000 7fb5227c4700 1 librados: shutdown 2024-04-14T15:11:09.818 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-14T15:11:10.291 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:10 smithi090 ceph-mon[41670]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:11:10.291 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:10 smithi090 ceph-mon[41670]: 4.a scrub starts 2024-04-14T15:11:10.291 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:10 smithi090 ceph-mon[41670]: 4.a scrub ok 2024-04-14T15:11:10.291 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:10 smithi090 ceph-mon[41670]: 4.d scrub starts 2024-04-14T15:11:10.291 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:10 smithi090 ceph-mon[41670]: 4.d scrub ok 2024-04-14T15:11:10.291 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:10 smithi090 ceph-mon[41670]: from='client.24643 v1:172.21.15.90:0/4176607233' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:11:10.292 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:10 smithi090 ceph-mon[30399]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:11:10.292 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:10 smithi090 ceph-mon[30399]: 4.a scrub starts 2024-04-14T15:11:10.292 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:10 smithi090 ceph-mon[30399]: 4.a scrub ok 2024-04-14T15:11:10.292 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:10 smithi090 ceph-mon[30399]: 4.d scrub starts 2024-04-14T15:11:10.292 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:10 smithi090 ceph-mon[30399]: 4.d scrub ok 2024-04-14T15:11:10.292 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:10 smithi090 ceph-mon[30399]: from='client.24643 v1:172.21.15.90:0/4176607233' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:11:10.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:10 smithi092 ceph-mon[33966]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:11:10.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:10 smithi092 ceph-mon[33966]: 4.a scrub starts 2024-04-14T15:11:10.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:10 smithi092 ceph-mon[33966]: 4.a scrub ok 2024-04-14T15:11:10.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:10 smithi092 ceph-mon[33966]: 4.d scrub starts 2024-04-14T15:11:10.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:10 smithi092 ceph-mon[33966]: 4.d scrub ok 2024-04-14T15:11:10.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:10 smithi092 ceph-mon[33966]: from='client.24643 v1:172.21.15.90:0/4176607233' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:11:10.720 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2024-04-14T15:11:08.947635+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1014,"num_read_kb":860,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":1771324,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":606,"ondisk_log_size":606,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":236576,"kb_used_data":7964,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":749724640,"statfs":{"total":767960285184,"available":767718031360,"internally_reserved":0,"allocated":8155136,"data_stored":4551736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":234016306},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002753"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.763793+0000","last_change":"2024-04-14T15:09:49.130210+0000","last_active":"2024-04-14T15:11:00.763793+0000","last_peered":"2024-04-14T15:11:00.763793+0000","last_clean":"2024-04-14T15:11:00.763793+0000","last_became_active":"2024-04-14T15:09:49.129636+0000","last_became_peered":"2024-04-14T15:09:49.129636+0000","last_unstale":"2024-04-14T15:11:00.763793+0000","last_undegraded":"2024-04-14T15:11:00.763793+0000","last_fullsized":"2024-04-14T15:11:00.763793+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:10.759713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":47,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.763604+0000","last_change":"2024-04-14T15:09:47.043210+0000","last_active":"2024-04-14T15:11:00.763604+0000","last_peered":"2024-04-14T15:11:00.763604+0000","last_clean":"2024-04-14T15:11:00.763604+0000","last_became_active":"2024-04-14T15:09:47.042538+0000","last_became_peered":"2024-04-14T15:09:47.042538+0000","last_unstale":"2024-04-14T15:11:00.763604+0000","last_undegraded":"2024-04-14T15:11:00.763604+0000","last_fullsized":"2024-04-14T15:11:00.763604+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:09:46.023663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:09.982965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.210463+0000","last_change":"2024-04-14T15:09:43.018880+0000","last_active":"2024-04-14T15:11:01.210463+0000","last_peered":"2024-04-14T15:11:01.210463+0000","last_clean":"2024-04-14T15:11:01.210463+0000","last_became_active":"2024-04-14T15:09:43.017351+0000","last_became_peered":"2024-04-14T15:09:43.017351+0000","last_unstale":"2024-04-14T15:11:01.210463+0000","last_undegraded":"2024-04-14T15:11:01.210463+0000","last_fullsized":"2024-04-14T15:11:01.210463+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:09:42.004317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:10.992803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"72'15","reported_seq":76,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764610+0000","last_change":"2024-04-14T15:09:45.034424+0000","last_active":"2024-04-14T15:11:00.764610+0000","last_peered":"2024-04-14T15:11:00.764610+0000","last_clean":"2024-04-14T15:11:00.764610+0000","last_became_active":"2024-04-14T15:09:45.034077+0000","last_became_peered":"2024-04-14T15:09:45.034077+0000","last_unstale":"2024-04-14T15:11:00.764610+0000","last_undegraded":"2024-04-14T15:11:00.764610+0000","last_fullsized":"2024-04-14T15:11:00.764610+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:08.933517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.763826+0000","last_change":"2024-04-14T15:09:49.059264+0000","last_active":"2024-04-14T15:11:00.763826+0000","last_peered":"2024-04-14T15:11:00.763826+0000","last_clean":"2024-04-14T15:11:00.763826+0000","last_became_active":"2024-04-14T15:09:49.058904+0000","last_became_peered":"2024-04-14T15:09:49.058904+0000","last_unstale":"2024-04-14T15:11:00.763826+0000","last_undegraded":"2024-04-14T15:11:00.763826+0000","last_fullsized":"2024-04-14T15:11:00.763826+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:13.856879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":47,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.763106+0000","last_change":"2024-04-14T15:09:47.041785+0000","last_active":"2024-04-14T15:11:00.763106+0000","last_peered":"2024-04-14T15:11:00.763106+0000","last_clean":"2024-04-14T15:11:00.763106+0000","last_became_active":"2024-04-14T15:09:47.041486+0000","last_became_peered":"2024-04-14T15:09:47.041486+0000","last_unstale":"2024-04-14T15:11:00.763106+0000","last_undegraded":"2024-04-14T15:11:00.763106+0000","last_fullsized":"2024-04-14T15:11:00.763106+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:09:46.023663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:03.085264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.765067+0000","last_change":"2024-04-14T15:09:43.023226+0000","last_active":"2024-04-14T15:11:00.765067+0000","last_peered":"2024-04-14T15:11:00.765067+0000","last_clean":"2024-04-14T15:11:00.765067+0000","last_became_active":"2024-04-14T15:09:43.022938+0000","last_became_peered":"2024-04-14T15:09:43.022938+0000","last_unstale":"2024-04-14T15:11:00.765067+0000","last_undegraded":"2024-04-14T15:11:00.765067+0000","last_fullsized":"2024-04-14T15:11:00.765067+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:09:42.004317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:10.564708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"72'9","reported_seq":67,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.763831+0000","last_change":"2024-04-14T15:09:45.032381+0000","last_active":"2024-04-14T15:11:00.763831+0000","last_peered":"2024-04-14T15:11:00.763831+0000","last_clean":"2024-04-14T15:11:00.763831+0000","last_became_active":"2024-04-14T15:09:45.032079+0000","last_became_peered":"2024-04-14T15:09:45.032079+0000","last_unstale":"2024-04-14T15:11:00.763831+0000","last_undegraded":"2024-04-14T15:11:00.763831+0000","last_fullsized":"2024-04-14T15:11:00.763831+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":51,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:03.745722+0000","last_change":"2024-04-14T15:11:03.745753+0000","last_active":"2024-04-14T15:11:03.745722+0000","last_peered":"2024-04-14T15:11:03.745722+0000","last_clean":"2024-04-14T15:11:03.745722+0000","last_became_active":"2024-04-14T15:09:49.069590+0000","last_became_peered":"2024-04-14T15:09:49.069590+0000","last_unstale":"2024-04-14T15:11:03.745722+0000","last_undegraded":"2024-04-14T15:11:03.745722+0000","last_fullsized":"2024-04-14T15:11:03.745722+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:03.745644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:03.745644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:29.376857+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:04.881903+0000","last_change":"2024-04-14T15:11:04.881923+0000","last_active":"2024-04-14T15:11:04.881903+0000","last_peered":"2024-04-14T15:11:04.881903+0000","last_clean":"2024-04-14T15:11:04.881903+0000","last_became_active":"2024-04-14T15:09:47.046517+0000","last_became_peered":"2024-04-14T15:09:47.046517+0000","last_unstale":"2024-04-14T15:11:04.881903+0000","last_undegraded":"2024-04-14T15:11:04.881903+0000","last_fullsized":"2024-04-14T15:11:04.881903+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:04.881846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:04.881846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:24.065968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":63,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.938224+0000","last_change":"2024-04-14T15:10:43.084925+0000","last_active":"2024-04-14T15:11:00.938224+0000","last_peered":"2024-04-14T15:11:00.938224+0000","last_clean":"2024-04-14T15:11:00.938224+0000","last_became_active":"2024-04-14T15:09:43.021986+0000","last_became_peered":"2024-04-14T15:09:43.021986+0000","last_unstale":"2024-04-14T15:11:00.938224+0000","last_undegraded":"2024-04-14T15:11:00.938224+0000","last_fullsized":"2024-04-14T15:11:00.938224+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:43.084820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:10:43.084820+0000","last_clean_scrub_stamp":"2024-04-14T15:10:43.084820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:55.633388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"72'5","reported_seq":61,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.763964+0000","last_change":"2024-04-14T15:09:45.028975+0000","last_active":"2024-04-14T15:11:00.763964+0000","last_peered":"2024-04-14T15:11:00.763964+0000","last_clean":"2024-04-14T15:11:00.763964+0000","last_became_active":"2024-04-14T15:09:45.028667+0000","last_became_peered":"2024-04-14T15:09:45.028667+0000","last_unstale":"2024-04-14T15:11:00.763964+0000","last_undegraded":"2024-04-14T15:11:00.763964+0000","last_fullsized":"2024-04-14T15:11:00.763964+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":51,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.209600+0000","last_change":"2024-04-14T15:10:54.765733+0000","last_active":"2024-04-14T15:11:01.209600+0000","last_peered":"2024-04-14T15:11:01.209600+0000","last_clean":"2024-04-14T15:11:01.209600+0000","last_became_active":"2024-04-14T15:09:49.061756+0000","last_became_peered":"2024-04-14T15:09:49.061756+0000","last_unstale":"2024-04-14T15:11:01.209600+0000","last_undegraded":"2024-04-14T15:11:01.209600+0000","last_fullsized":"2024-04-14T15:11:01.209600+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:54.765650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:10:54.765650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:59.282104+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":47,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.211280+0000","last_change":"2024-04-14T15:09:47.042682+0000","last_active":"2024-04-14T15:11:01.211280+0000","last_peered":"2024-04-14T15:11:01.211280+0000","last_clean":"2024-04-14T15:11:01.211280+0000","last_became_active":"2024-04-14T15:09:47.042384+0000","last_became_peered":"2024-04-14T15:09:47.042384+0000","last_unstale":"2024-04-14T15:11:01.211280+0000","last_undegraded":"2024-04-14T15:11:01.211280+0000","last_fullsized":"2024-04-14T15:11:01.211280+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:09:46.023663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:07.893460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764049+0000","last_change":"2024-04-14T15:10:07.764431+0000","last_active":"2024-04-14T15:11:00.764049+0000","last_peered":"2024-04-14T15:11:00.764049+0000","last_clean":"2024-04-14T15:11:00.764049+0000","last_became_active":"2024-04-14T15:10:07.764086+0000","last_became_peered":"2024-04-14T15:10:07.764086+0000","last_unstale":"2024-04-14T15:11:00.764049+0000","last_undegraded":"2024-04-14T15:11:00.764049+0000","last_fullsized":"2024-04-14T15:11:00.764049+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:09:42.004317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:08.369982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"72'9","reported_seq":67,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.763809+0000","last_change":"2024-04-14T15:09:45.030349+0000","last_active":"2024-04-14T15:11:00.763809+0000","last_peered":"2024-04-14T15:11:00.763809+0000","last_clean":"2024-04-14T15:11:00.763809+0000","last_became_active":"2024-04-14T15:09:45.030093+0000","last_became_peered":"2024-04-14T15:09:45.030093+0000","last_unstale":"2024-04-14T15:11:00.763809+0000","last_undegraded":"2024-04-14T15:11:00.763809+0000","last_fullsized":"2024-04-14T15:11:00.763809+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.763768+0000","last_change":"2024-04-14T15:09:49.128334+0000","last_active":"2024-04-14T15:11:00.763768+0000","last_peered":"2024-04-14T15:11:00.763768+0000","last_clean":"2024-04-14T15:11:00.763768+0000","last_became_active":"2024-04-14T15:09:49.128007+0000","last_became_peered":"2024-04-14T15:09:49.128007+0000","last_unstale":"2024-04-14T15:11:00.763768+0000","last_undegraded":"2024-04-14T15:11:00.763768+0000","last_fullsized":"2024-04-14T15:11:00.763768+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:04.103893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764349+0000","last_change":"2024-04-14T15:10:48.466668+0000","last_active":"2024-04-14T15:11:00.764349+0000","last_peered":"2024-04-14T15:11:00.764349+0000","last_clean":"2024-04-14T15:11:00.764349+0000","last_became_active":"2024-04-14T15:09:47.048045+0000","last_became_peered":"2024-04-14T15:09:47.048045+0000","last_unstale":"2024-04-14T15:11:00.764349+0000","last_undegraded":"2024-04-14T15:11:00.764349+0000","last_fullsized":"2024-04-14T15:11:00.764349+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:48.466556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:10:48.466556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:50.564963+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764000+0000","last_change":"2024-04-14T15:09:43.025155+0000","last_active":"2024-04-14T15:11:00.764000+0000","last_peered":"2024-04-14T15:11:00.764000+0000","last_clean":"2024-04-14T15:11:00.764000+0000","last_became_active":"2024-04-14T15:09:43.024853+0000","last_became_peered":"2024-04-14T15:09:43.024853+0000","last_unstale":"2024-04-14T15:11:00.764000+0000","last_undegraded":"2024-04-14T15:11:00.764000+0000","last_fullsized":"2024-04-14T15:11:00.764000+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:09:42.004317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:08.422553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"72'12","reported_seq":74,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.763590+0000","last_change":"2024-04-14T15:09:45.037659+0000","last_active":"2024-04-14T15:11:00.763590+0000","last_peered":"2024-04-14T15:11:00.763590+0000","last_clean":"2024-04-14T15:11:00.763590+0000","last_became_active":"2024-04-14T15:09:45.037173+0000","last_became_peered":"2024-04-14T15:09:45.037173+0000","last_unstale":"2024-04-14T15:11:00.763590+0000","last_undegraded":"2024-04-14T15:11:00.763590+0000","last_fullsized":"2024-04-14T15:11:00.763590+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:02.298243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":51,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:04.539857+0000","last_change":"2024-04-14T15:11:04.539877+0000","last_active":"2024-04-14T15:11:04.539857+0000","last_peered":"2024-04-14T15:11:04.539857+0000","last_clean":"2024-04-14T15:11:04.539857+0000","last_became_active":"2024-04-14T15:09:49.127979+0000","last_became_peered":"2024-04-14T15:09:49.127979+0000","last_unstale":"2024-04-14T15:11:04.539857+0000","last_undegraded":"2024-04-14T15:11:04.539857+0000","last_fullsized":"2024-04-14T15:11:04.539857+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:04.539789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:04.539789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:16.636271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":47,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764190+0000","last_change":"2024-04-14T15:09:47.043239+0000","last_active":"2024-04-14T15:11:00.764190+0000","last_peered":"2024-04-14T15:11:00.764190+0000","last_clean":"2024-04-14T15:11:00.764190+0000","last_became_active":"2024-04-14T15:09:47.042733+0000","last_became_peered":"2024-04-14T15:09:47.042733+0000","last_unstale":"2024-04-14T15:11:00.764190+0000","last_undegraded":"2024-04-14T15:11:00.764190+0000","last_fullsized":"2024-04-14T15:11:00.764190+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:09:46.023663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.940018+0000","last_change":"2024-04-14T15:09:43.022219+0000","last_active":"2024-04-14T15:11:00.940018+0000","last_peered":"2024-04-14T15:11:00.940018+0000","last_clean":"2024-04-14T15:11:00.940018+0000","last_became_active":"2024-04-14T15:09:43.021494+0000","last_became_peered":"2024-04-14T15:09:43.021494+0000","last_unstale":"2024-04-14T15:11:00.940018+0000","last_undegraded":"2024-04-14T15:11:00.940018+0000","last_fullsized":"2024-04-14T15:11:00.940018+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:09:42.004317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:02.243293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"72'15","reported_seq":76,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.211010+0000","last_change":"2024-04-14T15:09:45.033224+0000","last_active":"2024-04-14T15:11:01.211010+0000","last_peered":"2024-04-14T15:11:01.211010+0000","last_clean":"2024-04-14T15:11:01.211010+0000","last_became_active":"2024-04-14T15:09:45.032887+0000","last_became_peered":"2024-04-14T15:09:45.032887+0000","last_unstale":"2024-04-14T15:11:01.211010+0000","last_undegraded":"2024-04-14T15:11:01.211010+0000","last_fullsized":"2024-04-14T15:11:01.211010+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:04.849528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.763996+0000","last_change":"2024-04-14T15:09:49.057849+0000","last_active":"2024-04-14T15:11:00.763996+0000","last_peered":"2024-04-14T15:11:00.763996+0000","last_clean":"2024-04-14T15:11:00.763996+0000","last_became_active":"2024-04-14T15:09:49.057545+0000","last_became_peered":"2024-04-14T15:09:49.057545+0000","last_unstale":"2024-04-14T15:11:00.763996+0000","last_undegraded":"2024-04-14T15:11:00.763996+0000","last_fullsized":"2024-04-14T15:11:00.763996+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":47,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.211109+0000","last_change":"2024-04-14T15:09:47.046035+0000","last_active":"2024-04-14T15:11:01.211109+0000","last_peered":"2024-04-14T15:11:01.211109+0000","last_clean":"2024-04-14T15:11:01.211109+0000","last_became_active":"2024-04-14T15:09:47.045730+0000","last_became_peered":"2024-04-14T15:09:47.045730+0000","last_unstale":"2024-04-14T15:11:01.211109+0000","last_undegraded":"2024-04-14T15:11:01.211109+0000","last_fullsized":"2024-04-14T15:11:01.211109+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:09:46.023663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:11.966020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"65'2","reported_seq":70,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764422+0000","last_change":"2024-04-14T15:10:57.406574+0000","last_active":"2024-04-14T15:11:00.764422+0000","last_peered":"2024-04-14T15:11:00.764422+0000","last_clean":"2024-04-14T15:11:00.764422+0000","last_became_active":"2024-04-14T15:09:43.020401+0000","last_became_peered":"2024-04-14T15:09:43.020401+0000","last_unstale":"2024-04-14T15:11:00.764422+0000","last_undegraded":"2024-04-14T15:11:00.764422+0000","last_fullsized":"2024-04-14T15:11:00.764422+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-04-14T15:10:57.406478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:10:57.406478+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:25.577150+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"72'11","reported_seq":88,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.939685+0000","last_change":"2024-04-14T15:10:07.771479+0000","last_active":"2024-04-14T15:11:00.939685+0000","last_peered":"2024-04-14T15:11:00.939685+0000","last_clean":"2024-04-14T15:11:00.939685+0000","last_became_active":"2024-04-14T15:10:07.764183+0000","last_became_peered":"2024-04-14T15:10:07.764183+0000","last_unstale":"2024-04-14T15:11:00.939685+0000","last_undegraded":"2024-04-14T15:11:00.939685+0000","last_fullsized":"2024-04-14T15:11:00.939685+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:13.054416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"72'1","reported_seq":52,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.938612+0000","last_change":"2024-04-14T15:10:59.889342+0000","last_active":"2024-04-14T15:11:00.938612+0000","last_peered":"2024-04-14T15:11:00.938612+0000","last_clean":"2024-04-14T15:11:00.938612+0000","last_became_active":"2024-04-14T15:09:49.062051+0000","last_became_peered":"2024-04-14T15:09:49.062051+0000","last_unstale":"2024-04-14T15:11:00.938612+0000","last_undegraded":"2024-04-14T15:11:00.938612+0000","last_fullsized":"2024-04-14T15:11:00.938612+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-04-14T15:10:59.889236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:10:59.889236+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:03.689276+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"72'8","reported_seq":84,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:03.082668+0000","last_change":"2024-04-14T15:10:58.268612+0000","last_active":"2024-04-14T15:11:03.082668+0000","last_peered":"2024-04-14T15:11:03.082668+0000","last_clean":"2024-04-14T15:11:03.082668+0000","last_became_active":"2024-04-14T15:09:47.048216+0000","last_became_peered":"2024-04-14T15:09:47.048216+0000","last_unstale":"2024-04-14T15:11:03.082668+0000","last_undegraded":"2024-04-14T15:11:03.082668+0000","last_fullsized":"2024-04-14T15:11:03.082668+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:10:58.268488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:10:58.268488+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:04.149311+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":63,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.210561+0000","last_change":"2024-04-14T15:10:52.882061+0000","last_active":"2024-04-14T15:11:01.210561+0000","last_peered":"2024-04-14T15:11:01.210561+0000","last_clean":"2024-04-14T15:11:01.210561+0000","last_became_active":"2024-04-14T15:09:43.023257+0000","last_became_peered":"2024-04-14T15:09:43.023257+0000","last_unstale":"2024-04-14T15:11:01.210561+0000","last_undegraded":"2024-04-14T15:11:01.210561+0000","last_fullsized":"2024-04-14T15:11:01.210561+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:52.881916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:10:52.881916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:58.458431+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"72'10","reported_seq":66,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.211106+0000","last_change":"2024-04-14T15:09:45.181795+0000","last_active":"2024-04-14T15:11:01.211106+0000","last_peered":"2024-04-14T15:11:01.211106+0000","last_clean":"2024-04-14T15:11:01.211106+0000","last_became_active":"2024-04-14T15:09:45.181456+0000","last_became_peered":"2024-04-14T15:09:45.181456+0000","last_unstale":"2024-04-14T15:11:01.211106+0000","last_undegraded":"2024-04-14T15:11:01.211106+0000","last_fullsized":"2024-04-14T15:11:01.211106+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:05.909429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764295+0000","last_change":"2024-04-14T15:09:49.130071+0000","last_active":"2024-04-14T15:11:00.764295+0000","last_peered":"2024-04-14T15:11:00.764295+0000","last_clean":"2024-04-14T15:11:00.764295+0000","last_became_active":"2024-04-14T15:09:49.129392+0000","last_became_peered":"2024-04-14T15:09:49.129392+0000","last_unstale":"2024-04-14T15:11:00.764295+0000","last_undegraded":"2024-04-14T15:11:00.764295+0000","last_fullsized":"2024-04-14T15:11:00.764295+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:02.611406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:05.917034+0000","last_change":"2024-04-14T15:11:05.917054+0000","last_active":"2024-04-14T15:11:05.917034+0000","last_peered":"2024-04-14T15:11:05.917034+0000","last_clean":"2024-04-14T15:11:05.917034+0000","last_became_active":"2024-04-14T15:09:47.044665+0000","last_became_peered":"2024-04-14T15:09:47.044665+0000","last_unstale":"2024-04-14T15:11:05.917034+0000","last_undegraded":"2024-04-14T15:11:05.917034+0000","last_fullsized":"2024-04-14T15:11:05.917034+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:05.916966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:05.916966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:35.330112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":63,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.939825+0000","last_change":"2024-04-14T15:10:54.239518+0000","last_active":"2024-04-14T15:11:00.939825+0000","last_peered":"2024-04-14T15:11:00.939825+0000","last_clean":"2024-04-14T15:11:00.939825+0000","last_became_active":"2024-04-14T15:09:43.023220+0000","last_became_peered":"2024-04-14T15:09:43.023220+0000","last_unstale":"2024-04-14T15:11:00.939825+0000","last_undegraded":"2024-04-14T15:11:00.939825+0000","last_fullsized":"2024-04-14T15:11:00.939825+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:54.239409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:10:54.239409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:11.539313+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"72'11","reported_seq":78,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:06.492751+0000","last_change":"2024-04-14T15:11:06.492784+0000","last_active":"2024-04-14T15:11:06.492751+0000","last_peered":"2024-04-14T15:11:06.492751+0000","last_clean":"2024-04-14T15:11:06.492751+0000","last_became_active":"2024-04-14T15:09:45.180332+0000","last_became_peered":"2024-04-14T15:09:45.180332+0000","last_unstale":"2024-04-14T15:11:06.492751+0000","last_undegraded":"2024-04-14T15:11:06.492751+0000","last_fullsized":"2024-04-14T15:11:06.492751+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-04-14T15:11:06.492677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:06.492677+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:24.577564+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":51,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:07.901818+0000","last_change":"2024-04-14T15:11:07.901838+0000","last_active":"2024-04-14T15:11:07.901818+0000","last_peered":"2024-04-14T15:11:07.901818+0000","last_clean":"2024-04-14T15:11:07.901818+0000","last_became_active":"2024-04-14T15:09:49.061359+0000","last_became_peered":"2024-04-14T15:09:49.061359+0000","last_unstale":"2024-04-14T15:11:07.901818+0000","last_undegraded":"2024-04-14T15:11:07.901818+0000","last_fullsized":"2024-04-14T15:11:07.901818+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:07.901761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:07.901761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:35.316850+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.939704+0000","last_change":"2024-04-14T15:10:50.229008+0000","last_active":"2024-04-14T15:11:00.939704+0000","last_peered":"2024-04-14T15:11:00.939704+0000","last_clean":"2024-04-14T15:11:00.939704+0000","last_became_active":"2024-04-14T15:09:47.041074+0000","last_became_peered":"2024-04-14T15:09:47.041074+0000","last_unstale":"2024-04-14T15:11:00.939704+0000","last_undegraded":"2024-04-14T15:11:00.939704+0000","last_fullsized":"2024-04-14T15:11:00.939704+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:50.228903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:10:50.228903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:17.158842+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":63,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:08.712389+0000","last_change":"2024-04-14T15:11:08.712429+0000","last_active":"2024-04-14T15:11:08.712389+0000","last_peered":"2024-04-14T15:11:08.712389+0000","last_clean":"2024-04-14T15:11:08.712389+0000","last_became_active":"2024-04-14T15:09:43.026786+0000","last_became_peered":"2024-04-14T15:09:43.026786+0000","last_unstale":"2024-04-14T15:11:08.712389+0000","last_undegraded":"2024-04-14T15:11:08.712389+0000","last_fullsized":"2024-04-14T15:11:08.712389+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:08.712289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:08.712289+0000","last_clean_scrub_stamp":"2024-04-14T15:11:08.712289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:11.190004+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"72'4","reported_seq":57,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764146+0000","last_change":"2024-04-14T15:09:45.039145+0000","last_active":"2024-04-14T15:11:00.764146+0000","last_peered":"2024-04-14T15:11:00.764146+0000","last_clean":"2024-04-14T15:11:00.764146+0000","last_became_active":"2024-04-14T15:09:45.038815+0000","last_became_peered":"2024-04-14T15:09:45.038815+0000","last_unstale":"2024-04-14T15:11:00.764146+0000","last_undegraded":"2024-04-14T15:11:00.764146+0000","last_fullsized":"2024-04-14T15:11:00.764146+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:13.528834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764325+0000","last_change":"2024-04-14T15:09:49.066223+0000","last_active":"2024-04-14T15:11:00.764325+0000","last_peered":"2024-04-14T15:11:00.764325+0000","last_clean":"2024-04-14T15:11:00.764325+0000","last_became_active":"2024-04-14T15:09:49.065896+0000","last_became_peered":"2024-04-14T15:09:49.065896+0000","last_unstale":"2024-04-14T15:11:00.764325+0000","last_undegraded":"2024-04-14T15:11:00.764325+0000","last_fullsized":"2024-04-14T15:11:00.764325+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:09.025179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":47,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764746+0000","last_change":"2024-04-14T15:09:47.041972+0000","last_active":"2024-04-14T15:11:00.764746+0000","last_peered":"2024-04-14T15:11:00.764746+0000","last_clean":"2024-04-14T15:11:00.764746+0000","last_became_active":"2024-04-14T15:09:47.041683+0000","last_became_peered":"2024-04-14T15:09:47.041683+0000","last_unstale":"2024-04-14T15:11:00.764746+0000","last_undegraded":"2024-04-14T15:11:00.764746+0000","last_fullsized":"2024-04-14T15:11:00.764746+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:09:46.023663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:08.100556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764274+0000","last_change":"2024-04-14T15:09:43.023867+0000","last_active":"2024-04-14T15:11:00.764274+0000","last_peered":"2024-04-14T15:11:00.764274+0000","last_clean":"2024-04-14T15:11:00.764274+0000","last_became_active":"2024-04-14T15:09:43.023583+0000","last_became_peered":"2024-04-14T15:09:43.023583+0000","last_unstale":"2024-04-14T15:11:00.764274+0000","last_undegraded":"2024-04-14T15:11:00.764274+0000","last_fullsized":"2024-04-14T15:11:00.764274+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:09:42.004317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:05.031251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"72'11","reported_seq":70,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764103+0000","last_change":"2024-04-14T15:09:45.026357+0000","last_active":"2024-04-14T15:11:00.764103+0000","last_peered":"2024-04-14T15:11:00.764103+0000","last_clean":"2024-04-14T15:11:00.764103+0000","last_became_active":"2024-04-14T15:09:45.026065+0000","last_became_peered":"2024-04-14T15:09:45.026065+0000","last_unstale":"2024-04-14T15:11:00.764103+0000","last_undegraded":"2024-04-14T15:11:00.764103+0000","last_fullsized":"2024-04-14T15:11:00.764103+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:08.741092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":51,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.209828+0000","last_change":"2024-04-14T15:10:56.820210+0000","last_active":"2024-04-14T15:11:01.209828+0000","last_peered":"2024-04-14T15:11:01.209828+0000","last_clean":"2024-04-14T15:11:01.209828+0000","last_became_active":"2024-04-14T15:09:49.059711+0000","last_became_peered":"2024-04-14T15:09:49.059711+0000","last_unstale":"2024-04-14T15:11:01.209828+0000","last_undegraded":"2024-04-14T15:11:01.209828+0000","last_fullsized":"2024-04-14T15:11:01.209828+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:56.820123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:10:56.820123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:08.868464+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":47,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764103+0000","last_change":"2024-04-14T15:09:47.045205+0000","last_active":"2024-04-14T15:11:00.764103+0000","last_peered":"2024-04-14T15:11:00.764103+0000","last_clean":"2024-04-14T15:11:00.764103+0000","last_became_active":"2024-04-14T15:09:47.044891+0000","last_became_peered":"2024-04-14T15:09:47.044891+0000","last_unstale":"2024-04-14T15:11:00.764103+0000","last_undegraded":"2024-04-14T15:11:00.764103+0000","last_fullsized":"2024-04-14T15:11:00.764103+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:09:46.023663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":63,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:03.304615+0000","last_change":"2024-04-14T15:11:03.304640+0000","last_active":"2024-04-14T15:11:03.304615+0000","last_peered":"2024-04-14T15:11:03.304615+0000","last_clean":"2024-04-14T15:11:03.304615+0000","last_became_active":"2024-04-14T15:09:43.022480+0000","last_became_peered":"2024-04-14T15:09:43.022480+0000","last_unstale":"2024-04-14T15:11:03.304615+0000","last_undegraded":"2024-04-14T15:11:03.304615+0000","last_fullsized":"2024-04-14T15:11:03.304615+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:03.304541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:03.304541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:22.825961+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"72'9","reported_seq":67,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764699+0000","last_change":"2024-04-14T15:09:45.026649+0000","last_active":"2024-04-14T15:11:00.764699+0000","last_peered":"2024-04-14T15:11:00.764699+0000","last_clean":"2024-04-14T15:11:00.764699+0000","last_became_active":"2024-04-14T15:09:45.026363+0000","last_became_peered":"2024-04-14T15:09:45.026363+0000","last_unstale":"2024-04-14T15:11:00.764699+0000","last_undegraded":"2024-04-14T15:11:00.764699+0000","last_fullsized":"2024-04-14T15:11:00.764699+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:03.421380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764776+0000","last_change":"2024-04-14T15:09:49.065895+0000","last_active":"2024-04-14T15:11:00.764776+0000","last_peered":"2024-04-14T15:11:00.764776+0000","last_clean":"2024-04-14T15:11:00.764776+0000","last_became_active":"2024-04-14T15:09:49.065242+0000","last_became_peered":"2024-04-14T15:09:49.065242+0000","last_unstale":"2024-04-14T15:11:00.764776+0000","last_undegraded":"2024-04-14T15:11:00.764776+0000","last_fullsized":"2024-04-14T15:11:00.764776+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"72'8","reported_seq":76,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:03.083036+0000","last_change":"2024-04-14T15:09:47.047381+0000","last_active":"2024-04-14T15:11:03.083036+0000","last_peered":"2024-04-14T15:11:03.083036+0000","last_clean":"2024-04-14T15:11:03.083036+0000","last_became_active":"2024-04-14T15:09:47.047072+0000","last_became_peered":"2024-04-14T15:09:47.047072+0000","last_unstale":"2024-04-14T15:11:03.083036+0000","last_undegraded":"2024-04-14T15:11:03.083036+0000","last_fullsized":"2024-04-14T15:11:03.083036+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:09:46.023663+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.210904+0000","last_change":"2024-04-14T15:09:43.019612+0000","last_active":"2024-04-14T15:11:01.210904+0000","last_peered":"2024-04-14T15:11:01.210904+0000","last_clean":"2024-04-14T15:11:01.210904+0000","last_became_active":"2024-04-14T15:09:43.019301+0000","last_became_peered":"2024-04-14T15:09:43.019301+0000","last_unstale":"2024-04-14T15:11:01.210904+0000","last_undegraded":"2024-04-14T15:11:01.210904+0000","last_fullsized":"2024-04-14T15:11:01.210904+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:09:42.004317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:01.268097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"72'9","reported_seq":75,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.211409+0000","last_change":"2024-04-14T15:10:56.858475+0000","last_active":"2024-04-14T15:11:01.211409+0000","last_peered":"2024-04-14T15:11:01.211409+0000","last_clean":"2024-04-14T15:11:01.211409+0000","last_became_active":"2024-04-14T15:09:45.178857+0000","last_became_peered":"2024-04-14T15:09:45.178857+0000","last_unstale":"2024-04-14T15:11:01.211409+0000","last_undegraded":"2024-04-14T15:11:01.211409+0000","last_fullsized":"2024-04-14T15:11:01.211409+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-04-14T15:10:56.858323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:10:56.858323+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:13.702279+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.209643+0000","last_change":"2024-04-14T15:09:49.066448+0000","last_active":"2024-04-14T15:11:01.209643+0000","last_peered":"2024-04-14T15:11:01.209643+0000","last_clean":"2024-04-14T15:11:01.209643+0000","last_became_active":"2024-04-14T15:09:49.066068+0000","last_became_peered":"2024-04-14T15:09:49.066068+0000","last_unstale":"2024-04-14T15:11:01.209643+0000","last_undegraded":"2024-04-14T15:11:01.209643+0000","last_fullsized":"2024-04-14T15:11:01.209643+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:14.252283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"72'8","reported_seq":73,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:03.082718+0000","last_change":"2024-04-14T15:09:47.037002+0000","last_active":"2024-04-14T15:11:03.082718+0000","last_peered":"2024-04-14T15:11:03.082718+0000","last_clean":"2024-04-14T15:11:03.082718+0000","last_became_active":"2024-04-14T15:09:47.036688+0000","last_became_peered":"2024-04-14T15:09:47.036688+0000","last_unstale":"2024-04-14T15:11:03.082718+0000","last_undegraded":"2024-04-14T15:11:03.082718+0000","last_fullsized":"2024-04-14T15:11:03.082718+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:09:46.023663+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.210693+0000","last_change":"2024-04-14T15:09:43.028602+0000","last_active":"2024-04-14T15:11:01.210693+0000","last_peered":"2024-04-14T15:11:01.210693+0000","last_clean":"2024-04-14T15:11:01.210693+0000","last_became_active":"2024-04-14T15:09:43.027776+0000","last_became_peered":"2024-04-14T15:09:43.027776+0000","last_unstale":"2024-04-14T15:11:01.210693+0000","last_undegraded":"2024-04-14T15:11:01.210693+0000","last_fullsized":"2024-04-14T15:11:01.210693+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:09:42.004317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:01.870579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"72'10","reported_seq":66,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764159+0000","last_change":"2024-04-14T15:09:45.179495+0000","last_active":"2024-04-14T15:11:00.764159+0000","last_peered":"2024-04-14T15:11:00.764159+0000","last_clean":"2024-04-14T15:11:00.764159+0000","last_became_active":"2024-04-14T15:09:45.179165+0000","last_became_peered":"2024-04-14T15:09:45.179165+0000","last_unstale":"2024-04-14T15:11:00.764159+0000","last_undegraded":"2024-04-14T15:11:00.764159+0000","last_fullsized":"2024-04-14T15:11:00.764159+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.938088+0000","last_change":"2024-04-14T15:09:49.130725+0000","last_active":"2024-04-14T15:11:00.938088+0000","last_peered":"2024-04-14T15:11:00.938088+0000","last_clean":"2024-04-14T15:11:00.938088+0000","last_became_active":"2024-04-14T15:09:49.130438+0000","last_became_peered":"2024-04-14T15:09:49.130438+0000","last_unstale":"2024-04-14T15:11:00.938088+0000","last_undegraded":"2024-04-14T15:11:00.938088+0000","last_fullsized":"2024-04-14T15:11:00.938088+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:16.730190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.211603+0000","last_change":"2024-04-14T15:10:53.900979+0000","last_active":"2024-04-14T15:11:01.211603+0000","last_peered":"2024-04-14T15:11:01.211603+0000","last_clean":"2024-04-14T15:11:01.211603+0000","last_became_active":"2024-04-14T15:09:47.045349+0000","last_became_peered":"2024-04-14T15:09:47.045349+0000","last_unstale":"2024-04-14T15:11:01.211603+0000","last_undegraded":"2024-04-14T15:11:01.211603+0000","last_fullsized":"2024-04-14T15:11:01.211603+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:53.900878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:10:53.900878+0000","last_clean_scrub_stamp":"2024-04-14T15:10:53.900878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:12.123204+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.939789+0000","last_change":"2024-04-14T15:09:43.019353+0000","last_active":"2024-04-14T15:11:00.939789+0000","last_peered":"2024-04-14T15:11:00.939789+0000","last_clean":"2024-04-14T15:11:00.939789+0000","last_became_active":"2024-04-14T15:09:43.018987+0000","last_became_peered":"2024-04-14T15:09:43.018987+0000","last_unstale":"2024-04-14T15:11:00.939789+0000","last_undegraded":"2024-04-14T15:11:00.939789+0000","last_fullsized":"2024-04-14T15:11:00.939789+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:09:42.004317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"72'6","reported_seq":60,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.763707+0000","last_change":"2024-04-14T15:09:45.180445+0000","last_active":"2024-04-14T15:11:00.763707+0000","last_peered":"2024-04-14T15:11:00.763707+0000","last_clean":"2024-04-14T15:11:00.763707+0000","last_became_active":"2024-04-14T15:09:45.180037+0000","last_became_peered":"2024-04-14T15:09:45.180037+0000","last_unstale":"2024-04-14T15:11:00.763707+0000","last_undegraded":"2024-04-14T15:11:00.763707+0000","last_fullsized":"2024-04-14T15:11:00.763707+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:08.718920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":51,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.210053+0000","last_change":"2024-04-14T15:10:56.886642+0000","last_active":"2024-04-14T15:11:01.210053+0000","last_peered":"2024-04-14T15:11:01.210053+0000","last_clean":"2024-04-14T15:11:01.210053+0000","last_became_active":"2024-04-14T15:09:49.064296+0000","last_became_peered":"2024-04-14T15:09:49.064296+0000","last_unstale":"2024-04-14T15:11:01.210053+0000","last_undegraded":"2024-04-14T15:11:01.210053+0000","last_fullsized":"2024-04-14T15:11:01.210053+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:56.886513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:10:56.886513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:17.056264+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:07.485898+0000","last_change":"2024-04-14T15:11:07.485922+0000","last_active":"2024-04-14T15:11:07.485898+0000","last_peered":"2024-04-14T15:11:07.485898+0000","last_clean":"2024-04-14T15:11:07.485898+0000","last_became_active":"2024-04-14T15:09:47.044816+0000","last_became_peered":"2024-04-14T15:09:47.044816+0000","last_unstale":"2024-04-14T15:11:07.485898+0000","last_undegraded":"2024-04-14T15:11:07.485898+0000","last_fullsized":"2024-04-14T15:11:07.485898+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:07.485821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:07.485821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:21.021299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":63,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.210752+0000","last_change":"2024-04-14T15:10:43.844823+0000","last_active":"2024-04-14T15:11:01.210752+0000","last_peered":"2024-04-14T15:11:01.210752+0000","last_clean":"2024-04-14T15:11:01.210752+0000","last_became_active":"2024-04-14T15:09:43.016788+0000","last_became_peered":"2024-04-14T15:09:43.016788+0000","last_unstale":"2024-04-14T15:11:01.210752+0000","last_undegraded":"2024-04-14T15:11:01.210752+0000","last_fullsized":"2024-04-14T15:11:01.210752+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:43.844693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:10:43.844693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:51.339452+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"72'9","reported_seq":67,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.210984+0000","last_change":"2024-04-14T15:09:45.180769+0000","last_active":"2024-04-14T15:11:01.210984+0000","last_peered":"2024-04-14T15:11:01.210984+0000","last_clean":"2024-04-14T15:11:01.210984+0000","last_became_active":"2024-04-14T15:09:45.180375+0000","last_became_peered":"2024-04-14T15:09:45.180375+0000","last_unstale":"2024-04-14T15:11:01.210984+0000","last_undegraded":"2024-04-14T15:11:01.210984+0000","last_fullsized":"2024-04-14T15:11:01.210984+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:12.591629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764186+0000","last_change":"2024-04-14T15:09:49.069204+0000","last_active":"2024-04-14T15:11:00.764186+0000","last_peered":"2024-04-14T15:11:00.764186+0000","last_clean":"2024-04-14T15:11:00.764186+0000","last_became_active":"2024-04-14T15:09:49.068883+0000","last_became_peered":"2024-04-14T15:09:49.068883+0000","last_unstale":"2024-04-14T15:11:00.764186+0000","last_undegraded":"2024-04-14T15:11:00.764186+0000","last_fullsized":"2024-04-14T15:11:00.764186+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:12.232108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"65'3","reported_seq":73,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764604+0000","last_change":"2024-04-14T15:09:43.020827+0000","last_active":"2024-04-14T15:11:00.764604+0000","last_peered":"2024-04-14T15:11:00.764604+0000","last_clean":"2024-04-14T15:11:00.764604+0000","last_became_active":"2024-04-14T15:09:43.020490+0000","last_became_peered":"2024-04-14T15:09:43.020490+0000","last_unstale":"2024-04-14T15:11:00.764604+0000","last_undegraded":"2024-04-14T15:11:00.764604+0000","last_fullsized":"2024-04-14T15:11:00.764604+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:09:42.004317+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:07.143386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":528,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"72'12","reported_seq":82,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.763942+0000","last_change":"2024-04-14T15:10:53.448340+0000","last_active":"2024-04-14T15:11:00.763942+0000","last_peered":"2024-04-14T15:11:00.763942+0000","last_clean":"2024-04-14T15:11:00.763942+0000","last_became_active":"2024-04-14T15:09:45.032892+0000","last_became_peered":"2024-04-14T15:09:45.032892+0000","last_unstale":"2024-04-14T15:11:00.763942+0000","last_undegraded":"2024-04-14T15:11:00.763942+0000","last_fullsized":"2024-04-14T15:11:00.763942+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'12","last_scrub_stamp":"2024-04-14T15:10:53.448218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:10:53.448218+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:56.821958+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":47,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.210659+0000","last_change":"2024-04-14T15:09:47.049725+0000","last_active":"2024-04-14T15:11:01.210659+0000","last_peered":"2024-04-14T15:11:01.210659+0000","last_clean":"2024-04-14T15:11:01.210659+0000","last_became_active":"2024-04-14T15:09:47.049005+0000","last_became_peered":"2024-04-14T15:09:47.049005+0000","last_unstale":"2024-04-14T15:11:01.210659+0000","last_undegraded":"2024-04-14T15:11:01.210659+0000","last_fullsized":"2024-04-14T15:11:01.210659+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:09:46.023663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.209730+0000","last_change":"2024-04-14T15:09:49.067038+0000","last_active":"2024-04-14T15:11:01.209730+0000","last_peered":"2024-04-14T15:11:01.209730+0000","last_clean":"2024-04-14T15:11:01.209730+0000","last_became_active":"2024-04-14T15:09:49.066648+0000","last_became_peered":"2024-04-14T15:09:49.066648+0000","last_unstale":"2024-04-14T15:11:01.209730+0000","last_undegraded":"2024-04-14T15:11:01.209730+0000","last_fullsized":"2024-04-14T15:11:01.209730+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.763844+0000","last_change":"2024-04-14T15:09:43.022909+0000","last_active":"2024-04-14T15:11:00.763844+0000","last_peered":"2024-04-14T15:11:00.763844+0000","last_clean":"2024-04-14T15:11:00.763844+0000","last_became_active":"2024-04-14T15:09:43.022597+0000","last_became_peered":"2024-04-14T15:09:43.022597+0000","last_unstale":"2024-04-14T15:11:00.763844+0000","last_undegraded":"2024-04-14T15:11:00.763844+0000","last_fullsized":"2024-04-14T15:11:00.763844+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:09:42.004317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:11.088019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"72'10","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.210277+0000","last_change":"2024-04-14T15:10:07.770475+0000","last_active":"2024-04-14T15:11:01.210277+0000","last_peered":"2024-04-14T15:11:01.210277+0000","last_clean":"2024-04-14T15:11:01.210277+0000","last_became_active":"2024-04-14T15:10:07.764042+0000","last_became_peered":"2024-04-14T15:10:07.764042+0000","last_unstale":"2024-04-14T15:11:01.210277+0000","last_undegraded":"2024-04-14T15:11:01.210277+0000","last_fullsized":"2024-04-14T15:11:01.210277+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:04.539099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"72'8","reported_seq":73,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:03.082438+0000","last_change":"2024-04-14T15:09:47.048177+0000","last_active":"2024-04-14T15:11:03.082438+0000","last_peered":"2024-04-14T15:11:03.082438+0000","last_clean":"2024-04-14T15:11:03.082438+0000","last_became_active":"2024-04-14T15:09:47.047699+0000","last_became_peered":"2024-04-14T15:09:47.047699+0000","last_unstale":"2024-04-14T15:11:03.082438+0000","last_undegraded":"2024-04-14T15:11:03.082438+0000","last_fullsized":"2024-04-14T15:11:03.082438+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:09:46.023663+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.765065+0000","last_change":"2024-04-14T15:09:49.058715+0000","last_active":"2024-04-14T15:11:00.765065+0000","last_peered":"2024-04-14T15:11:00.765065+0000","last_clean":"2024-04-14T15:11:00.765065+0000","last_became_active":"2024-04-14T15:09:49.058387+0000","last_became_peered":"2024-04-14T15:09:49.058387+0000","last_unstale":"2024-04-14T15:11:00.765065+0000","last_undegraded":"2024-04-14T15:11:00.765065+0000","last_fullsized":"2024-04-14T15:11:00.765065+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:16.162642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.210796+0000","last_change":"2024-04-14T15:09:43.028748+0000","last_active":"2024-04-14T15:11:01.210796+0000","last_peered":"2024-04-14T15:11:01.210796+0000","last_clean":"2024-04-14T15:11:01.210796+0000","last_became_active":"2024-04-14T15:09:43.028362+0000","last_became_peered":"2024-04-14T15:09:43.028362+0000","last_unstale":"2024-04-14T15:11:01.210796+0000","last_undegraded":"2024-04-14T15:11:01.210796+0000","last_fullsized":"2024-04-14T15:11:01.210796+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:09:42.004317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"72'28","reported_seq":117,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:03.082440+0000","last_change":"2024-04-14T15:09:45.034501+0000","last_active":"2024-04-14T15:11:03.082440+0000","last_peered":"2024-04-14T15:11:03.082440+0000","last_clean":"2024-04-14T15:11:03.082440+0000","last_became_active":"2024-04-14T15:09:45.034168+0000","last_became_peered":"2024-04-14T15:09:45.034168+0000","last_unstale":"2024-04-14T15:11:03.082440+0000","last_undegraded":"2024-04-14T15:11:03.082440+0000","last_fullsized":"2024-04-14T15:11:03.082440+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":47,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.210782+0000","last_change":"2024-04-14T15:09:47.040508+0000","last_active":"2024-04-14T15:11:01.210782+0000","last_peered":"2024-04-14T15:11:01.210782+0000","last_clean":"2024-04-14T15:11:01.210782+0000","last_became_active":"2024-04-14T15:09:47.040185+0000","last_became_peered":"2024-04-14T15:09:47.040185+0000","last_unstale":"2024-04-14T15:11:01.210782+0000","last_undegraded":"2024-04-14T15:11:01.210782+0000","last_fullsized":"2024-04-14T15:11:01.210782+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:09:46.023663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:03.491052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":51,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.939013+0000","last_change":"2024-04-14T15:11:00.939052+0000","last_active":"2024-04-14T15:11:00.939013+0000","last_peered":"2024-04-14T15:11:00.939013+0000","last_clean":"2024-04-14T15:11:00.939013+0000","last_became_active":"2024-04-14T15:09:49.130307+0000","last_became_peered":"2024-04-14T15:09:49.130307+0000","last_unstale":"2024-04-14T15:11:00.939013+0000","last_undegraded":"2024-04-14T15:11:00.939013+0000","last_fullsized":"2024-04-14T15:11:00.939013+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:00.938917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:00.938917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:17.427163+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"73'5","reported_seq":153,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:07.762306+0000","last_change":"2024-04-14T15:11:03.192957+0000","last_active":"2024-04-14T15:11:07.762306+0000","last_peered":"2024-04-14T15:11:07.762306+0000","last_clean":"2024-04-14T15:11:07.762306+0000","last_became_active":"2024-04-14T15:09:43.016544+0000","last_became_peered":"2024-04-14T15:09:43.016544+0000","last_unstale":"2024-04-14T15:11:07.762306+0000","last_undegraded":"2024-04-14T15:11:07.762306+0000","last_fullsized":"2024-04-14T15:11:07.762306+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-04-14T15:11:03.192810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:03.192810+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:21.177179+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00056542099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":80,"num_read_kb":75,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":63,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.209765+0000","last_change":"2024-04-14T15:10:42.819764+0000","last_active":"2024-04-14T15:11:01.209765+0000","last_peered":"2024-04-14T15:11:01.209765+0000","last_clean":"2024-04-14T15:11:01.209765+0000","last_became_active":"2024-04-14T15:09:43.021188+0000","last_became_peered":"2024-04-14T15:09:43.021188+0000","last_unstale":"2024-04-14T15:11:01.209765+0000","last_undegraded":"2024-04-14T15:11:01.209765+0000","last_fullsized":"2024-04-14T15:11:01.209765+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:42.819681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:10:42.819681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:52.363715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"72'13","reported_seq":78,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764908+0000","last_change":"2024-04-14T15:09:45.034537+0000","last_active":"2024-04-14T15:11:00.764908+0000","last_peered":"2024-04-14T15:11:00.764908+0000","last_clean":"2024-04-14T15:11:00.764908+0000","last_became_active":"2024-04-14T15:09:45.033883+0000","last_became_peered":"2024-04-14T15:09:45.033883+0000","last_unstale":"2024-04-14T15:11:00.764908+0000","last_undegraded":"2024-04-14T15:11:00.764908+0000","last_fullsized":"2024-04-14T15:11:00.764908+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:04.692317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.210758+0000","last_change":"2024-04-14T15:10:55.841920+0000","last_active":"2024-04-14T15:11:01.210758+0000","last_peered":"2024-04-14T15:11:01.210758+0000","last_clean":"2024-04-14T15:11:01.210758+0000","last_became_active":"2024-04-14T15:09:47.049457+0000","last_became_peered":"2024-04-14T15:09:47.049457+0000","last_unstale":"2024-04-14T15:11:01.210758+0000","last_undegraded":"2024-04-14T15:11:01.210758+0000","last_fullsized":"2024-04-14T15:11:01.210758+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:55.841832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:10:55.841832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:09.497072+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.210445+0000","last_change":"2024-04-14T15:09:49.067392+0000","last_active":"2024-04-14T15:11:01.210445+0000","last_peered":"2024-04-14T15:11:01.210445+0000","last_clean":"2024-04-14T15:11:01.210445+0000","last_became_active":"2024-04-14T15:09:49.067031+0000","last_became_peered":"2024-04-14T15:09:49.067031+0000","last_unstale":"2024-04-14T15:11:01.210445+0000","last_undegraded":"2024-04-14T15:11:01.210445+0000","last_fullsized":"2024-04-14T15:11:01.210445+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:13.012527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"69'2","reported_seq":64,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.763287+0000","last_change":"2024-04-14T15:09:47.043261+0000","last_active":"2024-04-14T15:11:00.763287+0000","last_peered":"2024-04-14T15:11:00.763287+0000","last_clean":"2024-04-14T15:11:00.763287+0000","last_became_active":"2024-04-14T15:09:43.024882+0000","last_became_peered":"2024-04-14T15:09:43.024882+0000","last_unstale":"2024-04-14T15:11:00.763287+0000","last_undegraded":"2024-04-14T15:11:00.763287+0000","last_fullsized":"2024-04-14T15:11:00.763287+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:09:42.004317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:03.751335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0058046169999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":63,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.211222+0000","last_change":"2024-04-14T15:10:43.898230+0000","last_active":"2024-04-14T15:11:01.211222+0000","last_peered":"2024-04-14T15:11:01.211222+0000","last_clean":"2024-04-14T15:11:01.211222+0000","last_became_active":"2024-04-14T15:09:43.025421+0000","last_became_peered":"2024-04-14T15:09:43.025421+0000","last_unstale":"2024-04-14T15:11:01.211222+0000","last_undegraded":"2024-04-14T15:11:01.211222+0000","last_fullsized":"2024-04-14T15:11:01.211222+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:43.898117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:10:43.898117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:47.080377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":304,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.937548+0000","last_change":"2024-04-14T15:10:06.160908+0000","last_active":"2024-04-14T15:11:00.937548+0000","last_peered":"2024-04-14T15:11:00.937548+0000","last_clean":"2024-04-14T15:11:00.937548+0000","last_became_active":"2024-04-14T15:09:33.753828+0000","last_became_peered":"2024-04-14T15:09:33.753828+0000","last_unstale":"2024-04-14T15:11:00.937548+0000","last_undegraded":"2024-04-14T15:11:00.937548+0000","last_fullsized":"2024-04-14T15:11:00.937548+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'117","last_scrub_stamp":"2024-04-14T15:10:06.160794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:07:24.600370+0000","last_clean_scrub_stamp":"2024-04-14T15:10:06.160794+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:17.821751+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"72'16","reported_seq":91,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:03.082811+0000","last_change":"2024-04-14T15:09:45.180279+0000","last_active":"2024-04-14T15:11:03.082811+0000","last_peered":"2024-04-14T15:11:03.082811+0000","last_clean":"2024-04-14T15:11:03.082811+0000","last_became_active":"2024-04-14T15:09:45.179978+0000","last_became_peered":"2024-04-14T15:09:45.179978+0000","last_unstale":"2024-04-14T15:11:03.082811+0000","last_undegraded":"2024-04-14T15:11:03.082811+0000","last_fullsized":"2024-04-14T15:11:03.082811+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:11.891483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":47,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.938495+0000","last_change":"2024-04-14T15:09:47.047523+0000","last_active":"2024-04-14T15:11:00.938495+0000","last_peered":"2024-04-14T15:11:00.938495+0000","last_clean":"2024-04-14T15:11:00.938495+0000","last_became_active":"2024-04-14T15:09:47.047201+0000","last_became_peered":"2024-04-14T15:09:47.047201+0000","last_unstale":"2024-04-14T15:11:00.938495+0000","last_undegraded":"2024-04-14T15:11:00.938495+0000","last_fullsized":"2024-04-14T15:11:00.938495+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:09:46.023663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:13.363585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"72'1","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.763434+0000","last_change":"2024-04-14T15:09:49.065706+0000","last_active":"2024-04-14T15:11:00.763434+0000","last_peered":"2024-04-14T15:11:00.763434+0000","last_clean":"2024-04-14T15:11:00.763434+0000","last_became_active":"2024-04-14T15:09:49.065243+0000","last_became_peered":"2024-04-14T15:09:49.065243+0000","last_unstale":"2024-04-14T15:11:00.763434+0000","last_undegraded":"2024-04-14T15:11:00.763434+0000","last_fullsized":"2024-04-14T15:11:00.763434+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":70,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:07.696587+0000","last_change":"2024-04-14T15:11:07.696609+0000","last_active":"2024-04-14T15:11:07.696587+0000","last_peered":"2024-04-14T15:11:07.696587+0000","last_clean":"2024-04-14T15:11:07.696587+0000","last_became_active":"2024-04-14T15:09:43.018509+0000","last_became_peered":"2024-04-14T15:09:43.018509+0000","last_unstale":"2024-04-14T15:11:07.696587+0000","last_undegraded":"2024-04-14T15:11:07.696587+0000","last_fullsized":"2024-04-14T15:11:07.696587+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:07.696525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:07.696525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:17.969989+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0018592680000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":63,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.937410+0000","last_change":"2024-04-14T15:10:49.916989+0000","last_active":"2024-04-14T15:11:00.937410+0000","last_peered":"2024-04-14T15:11:00.937410+0000","last_clean":"2024-04-14T15:11:00.937410+0000","last_became_active":"2024-04-14T15:09:43.025095+0000","last_became_peered":"2024-04-14T15:09:43.025095+0000","last_unstale":"2024-04-14T15:11:00.937410+0000","last_undegraded":"2024-04-14T15:11:00.937410+0000","last_fullsized":"2024-04-14T15:11:00.937410+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:49.916884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:10:49.916884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:11.267411+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"72'12","reported_seq":77,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.210892+0000","last_change":"2024-04-14T15:10:51.840577+0000","last_active":"2024-04-14T15:11:01.210892+0000","last_peered":"2024-04-14T15:11:01.210892+0000","last_clean":"2024-04-14T15:11:01.210892+0000","last_became_active":"2024-04-14T15:09:45.037231+0000","last_became_peered":"2024-04-14T15:09:45.037231+0000","last_unstale":"2024-04-14T15:11:01.210892+0000","last_undegraded":"2024-04-14T15:11:01.210892+0000","last_fullsized":"2024-04-14T15:11:01.210892+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'12","last_scrub_stamp":"2024-04-14T15:10:51.840487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:10:51.840487+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:52.131140+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":47,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.211771+0000","last_change":"2024-04-14T15:09:47.037005+0000","last_active":"2024-04-14T15:11:01.211771+0000","last_peered":"2024-04-14T15:11:01.211771+0000","last_clean":"2024-04-14T15:11:01.211771+0000","last_became_active":"2024-04-14T15:09:47.036696+0000","last_became_peered":"2024-04-14T15:09:47.036696+0000","last_unstale":"2024-04-14T15:11:01.211771+0000","last_undegraded":"2024-04-14T15:11:01.211771+0000","last_fullsized":"2024-04-14T15:11:01.211771+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:09:46.023663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:06.058177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.763009+0000","last_change":"2024-04-14T15:09:49.128490+0000","last_active":"2024-04-14T15:11:00.763009+0000","last_peered":"2024-04-14T15:11:00.763009+0000","last_clean":"2024-04-14T15:11:00.763009+0000","last_became_active":"2024-04-14T15:09:49.128136+0000","last_became_peered":"2024-04-14T15:09:49.128136+0000","last_unstale":"2024-04-14T15:11:00.763009+0000","last_undegraded":"2024-04-14T15:11:00.763009+0000","last_fullsized":"2024-04-14T15:11:00.763009+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:09.879878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":63,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.938596+0000","last_change":"2024-04-14T15:10:53.992688+0000","last_active":"2024-04-14T15:11:00.938596+0000","last_peered":"2024-04-14T15:11:00.938596+0000","last_clean":"2024-04-14T15:11:00.938596+0000","last_became_active":"2024-04-14T15:09:43.021006+0000","last_became_peered":"2024-04-14T15:09:43.021006+0000","last_unstale":"2024-04-14T15:11:00.938596+0000","last_undegraded":"2024-04-14T15:11:00.938596+0000","last_fullsized":"2024-04-14T15:11:00.938596+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:53.992590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:10:53.992590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:16.131673+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"72'19","reported_seq":95,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.210795+0000","last_change":"2024-04-14T15:10:47.880742+0000","last_active":"2024-04-14T15:11:01.210795+0000","last_peered":"2024-04-14T15:11:01.210795+0000","last_clean":"2024-04-14T15:11:01.210795+0000","last_became_active":"2024-04-14T15:09:45.179182+0000","last_became_peered":"2024-04-14T15:09:45.179182+0000","last_unstale":"2024-04-14T15:11:01.210795+0000","last_undegraded":"2024-04-14T15:11:01.210795+0000","last_fullsized":"2024-04-14T15:11:01.210795+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'19","last_scrub_stamp":"2024-04-14T15:10:47.880623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:10:47.880623+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:07.612241+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.940078+0000","last_change":"2024-04-14T15:10:59.286474+0000","last_active":"2024-04-14T15:11:00.940078+0000","last_peered":"2024-04-14T15:11:00.940078+0000","last_clean":"2024-04-14T15:11:00.940078+0000","last_became_active":"2024-04-14T15:09:47.048427+0000","last_became_peered":"2024-04-14T15:09:47.048427+0000","last_unstale":"2024-04-14T15:11:00.940078+0000","last_undegraded":"2024-04-14T15:11:00.940078+0000","last_fullsized":"2024-04-14T15:11:00.940078+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:59.286374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:10:59.286374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:59.951701+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764213+0000","last_change":"2024-04-14T15:09:49.065988+0000","last_active":"2024-04-14T15:11:00.764213+0000","last_peered":"2024-04-14T15:11:00.764213+0000","last_clean":"2024-04-14T15:11:00.764213+0000","last_became_active":"2024-04-14T15:09:49.065684+0000","last_became_peered":"2024-04-14T15:09:49.065684+0000","last_unstale":"2024-04-14T15:11:00.764213+0000","last_undegraded":"2024-04-14T15:11:00.764213+0000","last_fullsized":"2024-04-14T15:11:00.764213+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:07.917647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"65'1","reported_seq":56,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.763647+0000","last_change":"2024-04-14T15:09:43.019515+0000","last_active":"2024-04-14T15:11:00.763647+0000","last_peered":"2024-04-14T15:11:00.763647+0000","last_clean":"2024-04-14T15:11:00.763647+0000","last_became_active":"2024-04-14T15:09:43.018478+0000","last_became_peered":"2024-04-14T15:09:43.018478+0000","last_unstale":"2024-04-14T15:11:00.763647+0000","last_undegraded":"2024-04-14T15:11:00.763647+0000","last_fullsized":"2024-04-14T15:11:00.763647+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:09:42.004317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:10.202051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"72'18","reported_seq":91,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764001+0000","last_change":"2024-04-14T15:10:53.457240+0000","last_active":"2024-04-14T15:11:00.764001+0000","last_peered":"2024-04-14T15:11:00.764001+0000","last_clean":"2024-04-14T15:11:00.764001+0000","last_became_active":"2024-04-14T15:09:45.181252+0000","last_became_peered":"2024-04-14T15:09:45.181252+0000","last_unstale":"2024-04-14T15:11:00.764001+0000","last_undegraded":"2024-04-14T15:11:00.764001+0000","last_fullsized":"2024-04-14T15:11:00.764001+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'18","last_scrub_stamp":"2024-04-14T15:10:53.457087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:10:53.457087+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:00.641833+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":47,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764213+0000","last_change":"2024-04-14T15:09:47.044907+0000","last_active":"2024-04-14T15:11:00.764213+0000","last_peered":"2024-04-14T15:11:00.764213+0000","last_clean":"2024-04-14T15:11:00.764213+0000","last_became_active":"2024-04-14T15:09:47.044610+0000","last_became_peered":"2024-04-14T15:09:47.044610+0000","last_unstale":"2024-04-14T15:11:00.764213+0000","last_undegraded":"2024-04-14T15:11:00.764213+0000","last_fullsized":"2024-04-14T15:11:00.764213+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:09:46.023663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:05.510613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.938142+0000","last_change":"2024-04-14T15:09:49.130011+0000","last_active":"2024-04-14T15:11:00.938142+0000","last_peered":"2024-04-14T15:11:00.938142+0000","last_clean":"2024-04-14T15:11:00.938142+0000","last_became_active":"2024-04-14T15:09:49.129610+0000","last_became_peered":"2024-04-14T15:09:49.129610+0000","last_unstale":"2024-04-14T15:11:00.938142+0000","last_undegraded":"2024-04-14T15:11:00.938142+0000","last_fullsized":"2024-04-14T15:11:00.938142+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:13.440963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.939826+0000","last_change":"2024-04-14T15:09:43.020785+0000","last_active":"2024-04-14T15:11:00.939826+0000","last_peered":"2024-04-14T15:11:00.939826+0000","last_clean":"2024-04-14T15:11:00.939826+0000","last_became_active":"2024-04-14T15:09:43.020503+0000","last_became_peered":"2024-04-14T15:09:43.020503+0000","last_unstale":"2024-04-14T15:11:00.939826+0000","last_undegraded":"2024-04-14T15:11:00.939826+0000","last_fullsized":"2024-04-14T15:11:00.939826+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:09:42.004317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:09.830404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"72'14","reported_seq":72,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.763783+0000","last_change":"2024-04-14T15:09:45.026006+0000","last_active":"2024-04-14T15:11:00.763783+0000","last_peered":"2024-04-14T15:11:00.763783+0000","last_clean":"2024-04-14T15:11:00.763783+0000","last_became_active":"2024-04-14T15:09:45.025673+0000","last_became_peered":"2024-04-14T15:09:45.025673+0000","last_unstale":"2024-04-14T15:11:00.763783+0000","last_undegraded":"2024-04-14T15:11:00.763783+0000","last_fullsized":"2024-04-14T15:11:00.763783+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"72'8","reported_seq":73,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:03.082724+0000","last_change":"2024-04-14T15:09:47.046875+0000","last_active":"2024-04-14T15:11:03.082724+0000","last_peered":"2024-04-14T15:11:03.082724+0000","last_clean":"2024-04-14T15:11:03.082724+0000","last_became_active":"2024-04-14T15:09:47.046560+0000","last_became_peered":"2024-04-14T15:09:47.046560+0000","last_unstale":"2024-04-14T15:11:03.082724+0000","last_undegraded":"2024-04-14T15:11:03.082724+0000","last_fullsized":"2024-04-14T15:11:03.082724+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:09:46.023663+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:10.044512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":51,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.501853+0000","last_change":"2024-04-14T15:11:01.501878+0000","last_active":"2024-04-14T15:11:01.501853+0000","last_peered":"2024-04-14T15:11:01.501853+0000","last_clean":"2024-04-14T15:11:01.501853+0000","last_became_active":"2024-04-14T15:09:49.128028+0000","last_became_peered":"2024-04-14T15:09:49.128028+0000","last_unstale":"2024-04-14T15:11:01.501853+0000","last_undegraded":"2024-04-14T15:11:01.501853+0000","last_fullsized":"2024-04-14T15:11:01.501853+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:01.501784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:01.501784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:23.865314+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:02.763474+0000","last_change":"2024-04-14T15:11:02.763497+0000","last_active":"2024-04-14T15:11:02.763474+0000","last_peered":"2024-04-14T15:11:02.763474+0000","last_clean":"2024-04-14T15:11:02.763474+0000","last_became_active":"2024-04-14T15:09:47.043021+0000","last_became_peered":"2024-04-14T15:09:47.043021+0000","last_unstale":"2024-04-14T15:11:02.763474+0000","last_undegraded":"2024-04-14T15:11:02.763474+0000","last_fullsized":"2024-04-14T15:11:02.763474+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:02.763399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:02.763399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:06.127325+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":63,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.937694+0000","last_change":"2024-04-14T15:10:48.939920+0000","last_active":"2024-04-14T15:11:00.937694+0000","last_peered":"2024-04-14T15:11:00.937694+0000","last_clean":"2024-04-14T15:11:00.937694+0000","last_became_active":"2024-04-14T15:09:43.018601+0000","last_became_peered":"2024-04-14T15:09:43.018601+0000","last_unstale":"2024-04-14T15:11:00.937694+0000","last_undegraded":"2024-04-14T15:11:00.937694+0000","last_fullsized":"2024-04-14T15:11:00.937694+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:48.939790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:10:48.939790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:14.888606+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"85'16","reported_seq":83,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764593+0000","last_change":"2024-04-14T15:10:47.456301+0000","last_active":"2024-04-14T15:11:00.764593+0000","last_peered":"2024-04-14T15:11:00.764593+0000","last_clean":"2024-04-14T15:11:00.764593+0000","last_became_active":"2024-04-14T15:09:45.028212+0000","last_became_peered":"2024-04-14T15:09:45.028212+0000","last_unstale":"2024-04-14T15:11:00.764593+0000","last_undegraded":"2024-04-14T15:11:00.764593+0000","last_fullsized":"2024-04-14T15:11:00.764593+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-14T15:10:47.456169+0000","last_deep_scrub":"85'16","last_deep_scrub_stamp":"2024-04-14T15:10:47.456169+0000","last_clean_scrub_stamp":"2024-04-14T15:10:47.456169+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:53.736949+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.210097+0000","last_change":"2024-04-14T15:09:49.059351+0000","last_active":"2024-04-14T15:11:01.210097+0000","last_peered":"2024-04-14T15:11:01.210097+0000","last_clean":"2024-04-14T15:11:01.210097+0000","last_became_active":"2024-04-14T15:09:49.058940+0000","last_became_peered":"2024-04-14T15:09:49.058940+0000","last_unstale":"2024-04-14T15:11:01.210097+0000","last_undegraded":"2024-04-14T15:11:01.210097+0000","last_fullsized":"2024-04-14T15:11:01.210097+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:03.127308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"72'8","reported_seq":73,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:03.082922+0000","last_change":"2024-04-14T15:09:47.051276+0000","last_active":"2024-04-14T15:11:03.082922+0000","last_peered":"2024-04-14T15:11:03.082922+0000","last_clean":"2024-04-14T15:11:03.082922+0000","last_became_active":"2024-04-14T15:09:47.050962+0000","last_became_peered":"2024-04-14T15:09:47.050962+0000","last_unstale":"2024-04-14T15:11:03.082922+0000","last_undegraded":"2024-04-14T15:11:03.082922+0000","last_fullsized":"2024-04-14T15:11:03.082922+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:09:46.023663+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.763245+0000","last_change":"2024-04-14T15:09:43.021175+0000","last_active":"2024-04-14T15:11:00.763245+0000","last_peered":"2024-04-14T15:11:00.763245+0000","last_clean":"2024-04-14T15:11:00.763245+0000","last_became_active":"2024-04-14T15:09:43.020878+0000","last_became_peered":"2024-04-14T15:09:43.020878+0000","last_unstale":"2024-04-14T15:11:00.763245+0000","last_undegraded":"2024-04-14T15:11:00.763245+0000","last_fullsized":"2024-04-14T15:11:00.763245+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:09:42.004317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"72'15","reported_seq":76,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764979+0000","last_change":"2024-04-14T15:09:45.033879+0000","last_active":"2024-04-14T15:11:00.764979+0000","last_peered":"2024-04-14T15:11:00.764979+0000","last_clean":"2024-04-14T15:11:00.764979+0000","last_became_active":"2024-04-14T15:09:45.033331+0000","last_became_peered":"2024-04-14T15:09:45.033331+0000","last_unstale":"2024-04-14T15:11:00.764979+0000","last_undegraded":"2024-04-14T15:11:00.764979+0000","last_fullsized":"2024-04-14T15:11:00.764979+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764272+0000","last_change":"2024-04-14T15:09:49.065592+0000","last_active":"2024-04-14T15:11:00.764272+0000","last_peered":"2024-04-14T15:11:00.764272+0000","last_clean":"2024-04-14T15:11:00.764272+0000","last_became_active":"2024-04-14T15:09:49.065274+0000","last_became_peered":"2024-04-14T15:09:49.065274+0000","last_unstale":"2024-04-14T15:11:00.764272+0000","last_undegraded":"2024-04-14T15:11:00.764272+0000","last_fullsized":"2024-04-14T15:11:00.764272+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:11.500131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"72'8","reported_seq":73,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:03.082488+0000","last_change":"2024-04-14T15:09:47.044537+0000","last_active":"2024-04-14T15:11:03.082488+0000","last_peered":"2024-04-14T15:11:03.082488+0000","last_clean":"2024-04-14T15:11:03.082488+0000","last_became_active":"2024-04-14T15:09:47.044085+0000","last_became_peered":"2024-04-14T15:09:47.044085+0000","last_unstale":"2024-04-14T15:11:03.082488+0000","last_undegraded":"2024-04-14T15:11:03.082488+0000","last_fullsized":"2024-04-14T15:11:03.082488+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:09:46.023663+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:08.095291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":63,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764648+0000","last_change":"2024-04-14T15:10:46.280858+0000","last_active":"2024-04-14T15:11:00.764648+0000","last_peered":"2024-04-14T15:11:00.764648+0000","last_clean":"2024-04-14T15:11:00.764648+0000","last_became_active":"2024-04-14T15:09:43.018871+0000","last_became_peered":"2024-04-14T15:09:43.018871+0000","last_unstale":"2024-04-14T15:11:00.764648+0000","last_undegraded":"2024-04-14T15:11:00.764648+0000","last_fullsized":"2024-04-14T15:11:00.764648+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:46.280773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:10:46.280773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:59.607706+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"72'10","reported_seq":66,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764245+0000","last_change":"2024-04-14T15:09:45.029530+0000","last_active":"2024-04-14T15:11:00.764245+0000","last_peered":"2024-04-14T15:11:00.764245+0000","last_clean":"2024-04-14T15:11:00.764245+0000","last_became_active":"2024-04-14T15:09:45.029170+0000","last_became_peered":"2024-04-14T15:09:45.029170+0000","last_unstale":"2024-04-14T15:11:00.764245+0000","last_undegraded":"2024-04-14T15:11:00.764245+0000","last_fullsized":"2024-04-14T15:11:00.764245+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:08.246963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.210058+0000","last_change":"2024-04-14T15:09:49.069916+0000","last_active":"2024-04-14T15:11:01.210058+0000","last_peered":"2024-04-14T15:11:01.210058+0000","last_clean":"2024-04-14T15:11:01.210058+0000","last_became_active":"2024-04-14T15:09:49.069594+0000","last_became_peered":"2024-04-14T15:09:49.069594+0000","last_unstale":"2024-04-14T15:11:01.210058+0000","last_undegraded":"2024-04-14T15:11:01.210058+0000","last_fullsized":"2024-04-14T15:11:01.210058+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":47,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.763544+0000","last_change":"2024-04-14T15:09:47.043481+0000","last_active":"2024-04-14T15:11:00.763544+0000","last_peered":"2024-04-14T15:11:00.763544+0000","last_clean":"2024-04-14T15:11:00.763544+0000","last_became_active":"2024-04-14T15:09:47.042810+0000","last_became_peered":"2024-04-14T15:09:47.042810+0000","last_unstale":"2024-04-14T15:11:00.763544+0000","last_undegraded":"2024-04-14T15:11:00.763544+0000","last_fullsized":"2024-04-14T15:11:00.763544+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:09:46.023663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:12.979380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":63,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.903230+0000","last_change":"2024-04-14T15:11:01.903252+0000","last_active":"2024-04-14T15:11:01.903230+0000","last_peered":"2024-04-14T15:11:01.903230+0000","last_clean":"2024-04-14T15:11:01.903230+0000","last_became_active":"2024-04-14T15:09:43.022351+0000","last_became_peered":"2024-04-14T15:09:43.022351+0000","last_unstale":"2024-04-14T15:11:01.903230+0000","last_undegraded":"2024-04-14T15:11:01.903230+0000","last_fullsized":"2024-04-14T15:11:01.903230+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:01.903150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:01.903150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:27.833698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"72'17","reported_seq":103,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764650+0000","last_change":"2024-04-14T15:10:07.775002+0000","last_active":"2024-04-14T15:11:00.764650+0000","last_peered":"2024-04-14T15:11:00.764650+0000","last_clean":"2024-04-14T15:11:00.764650+0000","last_became_active":"2024-04-14T15:10:07.766008+0000","last_became_peered":"2024-04-14T15:10:07.766008+0000","last_unstale":"2024-04-14T15:11:00.764650+0000","last_undegraded":"2024-04-14T15:11:00.764650+0000","last_fullsized":"2024-04-14T15:11:00.764650+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.937345+0000","last_change":"2024-04-14T15:09:49.065881+0000","last_active":"2024-04-14T15:11:00.937345+0000","last_peered":"2024-04-14T15:11:00.937345+0000","last_clean":"2024-04-14T15:11:00.937345+0000","last_became_active":"2024-04-14T15:09:49.065397+0000","last_became_peered":"2024-04-14T15:09:49.065397+0000","last_unstale":"2024-04-14T15:11:00.937345+0000","last_undegraded":"2024-04-14T15:11:00.937345+0000","last_fullsized":"2024-04-14T15:11:00.937345+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:13.399339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.210496+0000","last_change":"2024-04-14T15:10:59.793878+0000","last_active":"2024-04-14T15:11:01.210496+0000","last_peered":"2024-04-14T15:11:01.210496+0000","last_clean":"2024-04-14T15:11:01.210496+0000","last_became_active":"2024-04-14T15:09:47.045542+0000","last_became_peered":"2024-04-14T15:09:47.045542+0000","last_unstale":"2024-04-14T15:11:01.210496+0000","last_undegraded":"2024-04-14T15:11:01.210496+0000","last_fullsized":"2024-04-14T15:11:01.210496+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:59.793768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:10:59.793768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:24.997192+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":63,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.210883+0000","last_change":"2024-04-14T15:10:45.837586+0000","last_active":"2024-04-14T15:11:01.210883+0000","last_peered":"2024-04-14T15:11:01.210883+0000","last_clean":"2024-04-14T15:11:01.210883+0000","last_became_active":"2024-04-14T15:09:43.020268+0000","last_became_peered":"2024-04-14T15:09:43.020268+0000","last_unstale":"2024-04-14T15:11:01.210883+0000","last_undegraded":"2024-04-14T15:11:01.210883+0000","last_fullsized":"2024-04-14T15:11:01.210883+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:45.837506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:10:45.837506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:04.829576+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"72'19","reported_seq":82,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.939978+0000","last_change":"2024-04-14T15:09:45.179087+0000","last_active":"2024-04-14T15:11:00.939978+0000","last_peered":"2024-04-14T15:11:00.939978+0000","last_clean":"2024-04-14T15:11:00.939978+0000","last_became_active":"2024-04-14T15:09:45.178782+0000","last_became_peered":"2024-04-14T15:09:45.178782+0000","last_unstale":"2024-04-14T15:11:00.939978+0000","last_undegraded":"2024-04-14T15:11:00.939978+0000","last_fullsized":"2024-04-14T15:11:00.939978+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":51,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.209629+0000","last_change":"2024-04-14T15:10:53.789872+0000","last_active":"2024-04-14T15:11:01.209629+0000","last_peered":"2024-04-14T15:11:01.209629+0000","last_clean":"2024-04-14T15:11:01.209629+0000","last_became_active":"2024-04-14T15:09:49.061495+0000","last_became_peered":"2024-04-14T15:09:49.061495+0000","last_unstale":"2024-04-14T15:11:01.209629+0000","last_undegraded":"2024-04-14T15:11:01.209629+0000","last_fullsized":"2024-04-14T15:11:01.209629+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:53.789790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:10:53.789790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:54.748155+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.210353+0000","last_change":"2024-04-14T15:10:50.788471+0000","last_active":"2024-04-14T15:11:01.210353+0000","last_peered":"2024-04-14T15:11:01.210353+0000","last_clean":"2024-04-14T15:11:01.210353+0000","last_became_active":"2024-04-14T15:09:47.046153+0000","last_became_peered":"2024-04-14T15:09:47.046153+0000","last_unstale":"2024-04-14T15:11:01.210353+0000","last_undegraded":"2024-04-14T15:11:01.210353+0000","last_fullsized":"2024-04-14T15:11:01.210353+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:50.788350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:10:50.788350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:52.620524+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":63,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:04.223660+0000","last_change":"2024-04-14T15:11:04.223680+0000","last_active":"2024-04-14T15:11:04.223660+0000","last_peered":"2024-04-14T15:11:04.223660+0000","last_clean":"2024-04-14T15:11:04.223660+0000","last_became_active":"2024-04-14T15:09:43.020044+0000","last_became_peered":"2024-04-14T15:09:43.020044+0000","last_unstale":"2024-04-14T15:11:04.223660+0000","last_undegraded":"2024-04-14T15:11:04.223660+0000","last_fullsized":"2024-04-14T15:11:04.223660+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:04.223605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:04.223605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:11.422287+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"72'9","reported_seq":75,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.211199+0000","last_change":"2024-04-14T15:10:50.850226+0000","last_active":"2024-04-14T15:11:01.211199+0000","last_peered":"2024-04-14T15:11:01.211199+0000","last_clean":"2024-04-14T15:11:01.211199+0000","last_became_active":"2024-04-14T15:09:45.037179+0000","last_became_peered":"2024-04-14T15:09:45.037179+0000","last_unstale":"2024-04-14T15:11:01.211199+0000","last_undegraded":"2024-04-14T15:11:01.211199+0000","last_fullsized":"2024-04-14T15:11:01.211199+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-04-14T15:10:50.850116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:10:50.850116+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:06.196819+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.210115+0000","last_change":"2024-04-14T15:09:49.128389+0000","last_active":"2024-04-14T15:11:01.210115+0000","last_peered":"2024-04-14T15:11:01.210115+0000","last_clean":"2024-04-14T15:11:01.210115+0000","last_became_active":"2024-04-14T15:09:49.128070+0000","last_became_peered":"2024-04-14T15:09:49.128070+0000","last_unstale":"2024-04-14T15:11:01.210115+0000","last_undegraded":"2024-04-14T15:11:01.210115+0000","last_fullsized":"2024-04-14T15:11:01.210115+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:09.430362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.210984+0000","last_change":"2024-04-14T15:09:43.021909+0000","last_active":"2024-04-14T15:11:01.210984+0000","last_peered":"2024-04-14T15:11:01.210984+0000","last_clean":"2024-04-14T15:11:01.210984+0000","last_became_active":"2024-04-14T15:09:43.021460+0000","last_became_peered":"2024-04-14T15:09:43.021460+0000","last_unstale":"2024-04-14T15:11:01.210984+0000","last_undegraded":"2024-04-14T15:11:01.210984+0000","last_fullsized":"2024-04-14T15:11:01.210984+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:09:42.004317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:02.402497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"72'15","reported_seq":76,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.210565+0000","last_change":"2024-04-14T15:09:45.180505+0000","last_active":"2024-04-14T15:11:01.210565+0000","last_peered":"2024-04-14T15:11:01.210565+0000","last_clean":"2024-04-14T15:11:01.210565+0000","last_became_active":"2024-04-14T15:09:45.180160+0000","last_became_peered":"2024-04-14T15:09:45.180160+0000","last_unstale":"2024-04-14T15:11:01.210565+0000","last_undegraded":"2024-04-14T15:11:01.210565+0000","last_fullsized":"2024-04-14T15:11:01.210565+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"85'9","reported_seq":82,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:03.082688+0000","last_change":"2024-04-14T15:10:51.950673+0000","last_active":"2024-04-14T15:11:03.082688+0000","last_peered":"2024-04-14T15:11:03.082688+0000","last_clean":"2024-04-14T15:11:03.082688+0000","last_became_active":"2024-04-14T15:09:47.045441+0000","last_became_peered":"2024-04-14T15:09:47.045441+0000","last_unstale":"2024-04-14T15:11:03.082688+0000","last_undegraded":"2024-04-14T15:11:03.082688+0000","last_fullsized":"2024-04-14T15:11:03.082688+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9","last_scrub_stamp":"2024-04-14T15:10:51.950580+0000","last_deep_scrub":"85'9","last_deep_scrub_stamp":"2024-04-14T15:10:51.950580+0000","last_clean_scrub_stamp":"2024-04-14T15:10:51.950580+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:09.843217+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":398,"ondisk_log_size":398,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":80,"num_read_kb":75,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103191,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28368,"kb_used_data":1352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716784,"statfs":{"total":95995035648,"available":95965986816,"internally_reserved":0,"allocated":1384448,"data_stored":937840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397530,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28364,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716788,"statfs":{"total":95995035648,"available":95965990912,"internally_reserved":0,"allocated":1380352,"data_stored":938133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574900,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27820,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717332,"statfs":{"total":95995035648,"available":95966547968,"internally_reserved":0,"allocated":823296,"data_stored":347886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738425,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32536,"kb_used_data":1360,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712616,"statfs":{"total":95995035648,"available":95961718784,"internally_reserved":0,"allocated":1392640,"data_stored":938493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411376,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31932,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713220,"statfs":{"total":95995035648,"available":95962337280,"internally_reserved":0,"allocated":774144,"data_stored":346987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182442,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27816,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717336,"statfs":{"total":95995035648,"available":95966552064,"internally_reserved":0,"allocated":819200,"data_stored":348207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986212,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27796,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717356,"statfs":{"total":95995035648,"available":95966572544,"internally_reserved":0,"allocated":798720,"data_stored":347368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659169,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31944,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713208,"statfs":{"total":95995035648,"available":95962324992,"internally_reserved":0,"allocated":782336,"data_stored":346822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-14T15:11:10.727 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph pg dump --format=json 2024-04-14T15:11:10.969 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:11:11.364 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:11 smithi090 ceph-mon[41670]: 5.14 scrub starts 2024-04-14T15:11:11.364 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:11 smithi090 ceph-mon[41670]: 5.14 scrub ok 2024-04-14T15:11:11.364 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:11 smithi090 ceph-mon[41670]: 6.0 scrub starts 2024-04-14T15:11:11.364 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:11 smithi090 ceph-mon[41670]: 6.0 scrub ok 2024-04-14T15:11:11.364 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:11 smithi090 ceph-mon[30399]: 5.14 scrub starts 2024-04-14T15:11:11.364 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:11 smithi090 ceph-mon[30399]: 5.14 scrub ok 2024-04-14T15:11:11.364 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:11 smithi090 ceph-mon[30399]: 6.0 scrub starts 2024-04-14T15:11:11.365 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:11 smithi090 ceph-mon[30399]: 6.0 scrub ok 2024-04-14T15:11:11.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:11 smithi092 ceph-mon[33966]: 5.14 scrub starts 2024-04-14T15:11:11.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:11 smithi092 ceph-mon[33966]: 5.14 scrub ok 2024-04-14T15:11:11.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:11 smithi092 ceph-mon[33966]: 6.0 scrub starts 2024-04-14T15:11:11.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:11 smithi092 ceph-mon[33966]: 6.0 scrub ok 2024-04-14T15:11:12.344 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:12 smithi090 ceph-mon[41670]: 5.13 scrub starts 2024-04-14T15:11:12.344 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:12 smithi090 ceph-mon[41670]: 5.13 scrub ok 2024-04-14T15:11:12.344 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:12 smithi090 ceph-mon[41670]: 4.1b deep-scrub starts 2024-04-14T15:11:12.344 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:12 smithi090 ceph-mon[41670]: 4.1b deep-scrub ok 2024-04-14T15:11:12.344 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:12 smithi090 ceph-mon[41670]: 2.13 scrub starts 2024-04-14T15:11:12.344 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:12 smithi090 ceph-mon[41670]: 2.13 scrub ok 2024-04-14T15:11:12.345 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:12 smithi090 ceph-mon[41670]: 6.f scrub starts 2024-04-14T15:11:12.345 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:12 smithi090 ceph-mon[41670]: 6.f scrub ok 2024-04-14T15:11:12.345 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:12 smithi090 ceph-mon[41670]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1005 B/s rd, 0 op/s 2024-04-14T15:11:12.345 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:12 smithi090 ceph-mon[41670]: 2.9 scrub starts 2024-04-14T15:11:12.345 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:12 smithi090 ceph-mon[41670]: 2.9 scrub ok 2024-04-14T15:11:12.345 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:12 smithi090 ceph-mon[30399]: 5.13 scrub starts 2024-04-14T15:11:12.345 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:12 smithi090 ceph-mon[30399]: 5.13 scrub ok 2024-04-14T15:11:12.345 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:12 smithi090 ceph-mon[30399]: 4.1b deep-scrub starts 2024-04-14T15:11:12.345 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:12 smithi090 ceph-mon[30399]: 4.1b deep-scrub ok 2024-04-14T15:11:12.345 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:12 smithi090 ceph-mon[30399]: 2.13 scrub starts 2024-04-14T15:11:12.345 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:12 smithi090 ceph-mon[30399]: 2.13 scrub ok 2024-04-14T15:11:12.345 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:12 smithi090 ceph-mon[30399]: 6.f scrub starts 2024-04-14T15:11:12.346 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:12 smithi090 ceph-mon[30399]: 6.f scrub ok 2024-04-14T15:11:12.346 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:12 smithi090 ceph-mon[30399]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1005 B/s rd, 0 op/s 2024-04-14T15:11:12.346 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:12 smithi090 ceph-mon[30399]: 2.9 scrub starts 2024-04-14T15:11:12.346 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:12 smithi090 ceph-mon[30399]: 2.9 scrub ok 2024-04-14T15:11:12.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:12 smithi092 ceph-mon[33966]: 5.13 scrub starts 2024-04-14T15:11:12.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:12 smithi092 ceph-mon[33966]: 5.13 scrub ok 2024-04-14T15:11:12.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:12 smithi092 ceph-mon[33966]: 4.1b deep-scrub starts 2024-04-14T15:11:12.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:12 smithi092 ceph-mon[33966]: 4.1b deep-scrub ok 2024-04-14T15:11:12.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:12 smithi092 ceph-mon[33966]: 2.13 scrub starts 2024-04-14T15:11:12.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:12 smithi092 ceph-mon[33966]: 2.13 scrub ok 2024-04-14T15:11:12.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:12 smithi092 ceph-mon[33966]: 6.f scrub starts 2024-04-14T15:11:12.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:12 smithi092 ceph-mon[33966]: 6.f scrub ok 2024-04-14T15:11:12.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:12 smithi092 ceph-mon[33966]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1005 B/s rd, 0 op/s 2024-04-14T15:11:12.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:12 smithi092 ceph-mon[33966]: 2.9 scrub starts 2024-04-14T15:11:12.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:12 smithi092 ceph-mon[33966]: 2.9 scrub ok 2024-04-14T15:11:12.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.678+0000 7f10aa6fd700 1 -- 172.21.15.90:0/2210594049 >> v1:172.21.15.90:6789/0 conn(0x7f10a410f300 legacy=0x7f10a41117c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:12.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.678+0000 7f10aa6fd700 1 -- 172.21.15.90:0/2210594049 shutdown_connections 2024-04-14T15:11:12.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.678+0000 7f10aa6fd700 1 -- 172.21.15.90:0/2210594049 >> 172.21.15.90:0/2210594049 conn(0x7f10a40fe3d0 msgr2=0x7f10a41007f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:11:12.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.678+0000 7f10aa6fd700 1 -- 172.21.15.90:0/2210594049 shutdown_connections 2024-04-14T15:11:12.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.678+0000 7f10aa6fd700 1 -- 172.21.15.90:0/2210594049 wait complete. 2024-04-14T15:11:12.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.678+0000 7f10aa6fd700 1 librados: starting msgr at 2024-04-14T15:11:12.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.678+0000 7f10aa6fd700 1 librados: starting objecter 2024-04-14T15:11:12.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.678+0000 7f10aa6fd700 1 Processor -- start 2024-04-14T15:11:12.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.678+0000 7f10aa6fd700 1 -- start start 2024-04-14T15:11:12.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.678+0000 7f10aa6fd700 1 librados: setting wanted keys 2024-04-14T15:11:12.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.678+0000 7f10aa6fd700 1 librados: calling monclient init 2024-04-14T15:11:12.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.679+0000 7f10aa6fd700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f10a41a9c70 con 0x7f10a4108880 2024-04-14T15:11:12.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.679+0000 7f10aa6fd700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f10a41aae90 con 0x7f10a410b730 2024-04-14T15:11:12.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.679+0000 7f10aa6fd700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f10a41ac0d0 con 0x7f10a410f300 2024-04-14T15:11:12.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.679+0000 7f10a3fff700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f10a4108880 0x7f10a410eb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:39510/0 (socket says 172.21.15.90:39510) 2024-04-14T15:11:12.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.679+0000 7f10a3fff700 1 -- 172.21.15.90:0/746960986 learned_addr learned my addr 172.21.15.90:0/746960986 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:11:12.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.679+0000 7f10a17fa700 10 client.?.objecter ms_handle_connect 0x7f10a4108880 2024-04-14T15:11:12.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.679+0000 7f10a17fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:11:12.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.679+0000 7f10a17fa700 10 client.?.objecter ms_handle_connect 0x7f10a410f300 2024-04-14T15:11:12.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.679+0000 7f10a17fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:11:12.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.679+0000 7f10a17fa700 10 client.?.objecter ms_handle_connect 0x7f10a410b730 2024-04-14T15:11:12.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.679+0000 7f10a17fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:11:12.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.679+0000 7f10a17fa700 1 -- 172.21.15.90:0/746960986 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2946676375 0 0) 0x7f10a41a9c70 con 0x7f10a4108880 2024-04-14T15:11:12.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.679+0000 7f10a17fa700 1 -- 172.21.15.90:0/746960986 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f108c0039c0 con 0x7f10a4108880 2024-04-14T15:11:12.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.679+0000 7f10a17fa700 1 -- 172.21.15.90:0/746960986 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1602407354 0 0) 0x7f10a41ac0d0 con 0x7f10a410f300 2024-04-14T15:11:12.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.679+0000 7f10a17fa700 1 -- 172.21.15.90:0/746960986 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f10a41a9c70 con 0x7f10a410f300 2024-04-14T15:11:12.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.679+0000 7f10a17fa700 1 -- 172.21.15.90:0/746960986 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2018158176 0 0) 0x7f10a41aae90 con 0x7f10a410b730 2024-04-14T15:11:12.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.679+0000 7f10a17fa700 1 -- 172.21.15.90:0/746960986 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f10a41ac0d0 con 0x7f10a410b730 2024-04-14T15:11:12.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.680+0000 7f10a17fa700 1 -- 172.21.15.90:0/746960986 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3232585125 0 0) 0x7f10a41a9c70 con 0x7f10a410f300 2024-04-14T15:11:12.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.680+0000 7f10a17fa700 1 -- 172.21.15.90:0/746960986 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f10a41aae90 con 0x7f10a410f300 2024-04-14T15:11:12.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.680+0000 7f10a17fa700 1 -- 172.21.15.90:0/746960986 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f109c001930 con 0x7f10a410f300 2024-04-14T15:11:12.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.680+0000 7f10a17fa700 1 -- 172.21.15.90:0/746960986 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 21050615 0 0) 0x7f10a41ac0d0 con 0x7f10a410b730 2024-04-14T15:11:12.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.680+0000 7f10a17fa700 1 -- 172.21.15.90:0/746960986 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f10a41a9c70 con 0x7f10a410b730 2024-04-14T15:11:12.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.680+0000 7f10a17fa700 1 -- 172.21.15.90:0/746960986 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f10980017f0 con 0x7f10a410b730 2024-04-14T15:11:12.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.680+0000 7f10a17fa700 1 -- 172.21.15.90:0/746960986 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 177640039 0 0) 0x7f108c0039c0 con 0x7f10a4108880 2024-04-14T15:11:12.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.680+0000 7f10a17fa700 1 -- 172.21.15.90:0/746960986 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f10a41ac0d0 con 0x7f10a4108880 2024-04-14T15:11:12.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.680+0000 7f10a17fa700 1 -- 172.21.15.90:0/746960986 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f1094001a30 con 0x7f10a4108880 2024-04-14T15:11:12.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.680+0000 7f10a17fa700 1 -- 172.21.15.90:0/746960986 <== mon.2 v1:172.21.15.90:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 605027183 0 0) 0x7f10a41aae90 con 0x7f10a410f300 2024-04-14T15:11:12.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.680+0000 7f10a17fa700 1 -- 172.21.15.90:0/746960986 >> v1:172.21.15.92:6789/0 conn(0x7f10a410b730 legacy=0x7f10a41a4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:12.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.680+0000 7f10a17fa700 1 -- 172.21.15.90:0/746960986 >> v1:172.21.15.90:6789/0 conn(0x7f10a4108880 legacy=0x7f10a410eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:12.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.680+0000 7f10a17fa700 1 -- 172.21.15.90:0/746960986 --> v1:172.21.15.90:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10a41ad310 con 0x7f10a410f300 2024-04-14T15:11:12.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.680+0000 7f10aa6fd700 1 -- 172.21.15.90:0/746960986 --> v1:172.21.15.90:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f10a41ac300 con 0x7f10a410f300 2024-04-14T15:11:12.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.680+0000 7f10aa6fd700 10 client.24626.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:11:12.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.680+0000 7f10aa6fd700 1 -- 172.21.15.90:0/746960986 --> v1:172.21.15.90:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f10a41ac7c0 con 0x7f10a410f300 2024-04-14T15:11:12.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.680+0000 7f10aa6fd700 1 librados: init done 2024-04-14T15:11:12.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.680+0000 7f10a17fa700 1 -- 172.21.15.90:0/746960986 <== mon.2 v1:172.21.15.90:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f109c001480 con 0x7f10a410f300 2024-04-14T15:11:12.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.680+0000 7f10a17fa700 1 -- 172.21.15.90:0/746960986 <== mon.2 v1:172.21.15.90:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f109c005000 con 0x7f10a410f300 2024-04-14T15:11:12.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.681+0000 7f10aa6fd700 1 -- 172.21.15.90:0/746960986 --> v1:172.21.15.90:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1090005150 con 0x7f10a410f300 2024-04-14T15:11:12.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.681+0000 7f10a17fa700 1 -- 172.21.15.90:0/746960986 <== mon.2 v1:172.21.15.90:6790/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f109c003a00 con 0x7f10a410f300 2024-04-14T15:11:12.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.682+0000 7f10a17fa700 1 -- 172.21.15.90:0/746960986 <== mon.2 v1:172.21.15.90:6790/0 8 ==== osd_map(87..87 src has 12..87) ==== 6288+0+0 (unknown 1749939968 0 0) 0x7f109c0891f0 con 0x7f10a410f300 2024-04-14T15:11:12.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.682+0000 7f10a17fa700 10 client.24626.objecter ms_dispatch 0x7f10a4001550 osd_map(87..87 src has 12..87) 2024-04-14T15:11:12.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.682+0000 7f10a17fa700 3 client.24626.objecter handle_osd_map got epochs [87,87] > 0 2024-04-14T15:11:12.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.682+0000 7f10a17fa700 3 client.24626.objecter handle_osd_map decoding full epoch 87 2024-04-14T15:11:12.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.682+0000 7f10a17fa700 20 client.24626.objecter dump_active .. 0 homeless 2024-04-14T15:11:12.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.682+0000 7f10a17fa700 10 client.24626.objecter ms_handle_connect 0x7f108c071370 2024-04-14T15:11:12.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.687+0000 7f10a17fa700 1 -- 172.21.15.90:0/746960986 <== mon.2 v1:172.21.15.90:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f109c0564a0 con 0x7f10a410f300 2024-04-14T15:11:12.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.880+0000 7f10aa6fd700 1 -- 172.21.15.90:0/746960986 --> v1:172.21.15.90:6800/420935427 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1090000bc0 con 0x7f108c071370 2024-04-14T15:11:12.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.889+0000 7f10a17fa700 1 -- 172.21.15.90:0/746960986 <== mgr.14610 v1:172.21.15.90:6800/420935427 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+345451 (unknown 2965378022 0 773816603) 0x7f1090000bc0 con 0x7f108c071370 2024-04-14T15:11:12.894 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:11:12.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.892+0000 7f10aa6fd700 10 librados: watch_flush enter 2024-04-14T15:11:12.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.892+0000 7f10aa6fd700 10 librados: watch_flush exit 2024-04-14T15:11:12.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.892+0000 7f10aa6fd700 20 client.24626.objecter shutdown clearing up homeless session... 2024-04-14T15:11:12.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.892+0000 7f10aa6fd700 10 client.24626.objecter successfully canceled tick 2024-04-14T15:11:12.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.892+0000 7f10aa6fd700 1 -- 172.21.15.90:0/746960986 >> v1:172.21.15.90:6800/420935427 conn(0x7f108c071370 legacy=0x7f108c073830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:12.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.892+0000 7f10aa6fd700 1 -- 172.21.15.90:0/746960986 >> v1:172.21.15.90:6790/0 conn(0x7f10a410f300 legacy=0x7f10a41a8330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:12.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.893+0000 7f10aa6fd700 1 -- 172.21.15.90:0/746960986 shutdown_connections 2024-04-14T15:11:12.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.893+0000 7f10aa6fd700 1 -- 172.21.15.90:0/746960986 >> 172.21.15.90:0/746960986 conn(0x7f10a40fe3d0 msgr2=0x7f10a4101830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:11:12.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.893+0000 7f10aa6fd700 1 -- 172.21.15.90:0/746960986 shutdown_connections 2024-04-14T15:11:12.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.893+0000 7f10aa6fd700 1 -- 172.21.15.90:0/746960986 wait complete. 2024-04-14T15:11:12.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:12.893+0000 7f10aa6fd700 1 librados: shutdown 2024-04-14T15:11:12.897 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-14T15:11:12.942 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:11:12 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-alertmanager-a[78793]: ts=2024-04-14T15:11:12.803Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002854957s 2024-04-14T15:11:13.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:13 smithi092 ceph-mon[33966]: 4.14 scrub starts 2024-04-14T15:11:13.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:13 smithi092 ceph-mon[33966]: 4.14 scrub ok 2024-04-14T15:11:13.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:13 smithi092 ceph-mon[33966]: from='client.24626 v1:172.21.15.90:0/746960986' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:11:13.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:13 smithi090 ceph-mon[30399]: 4.14 scrub starts 2024-04-14T15:11:13.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:13 smithi090 ceph-mon[30399]: 4.14 scrub ok 2024-04-14T15:11:13.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:13 smithi090 ceph-mon[30399]: from='client.24626 v1:172.21.15.90:0/746960986' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:11:13.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:13 smithi090 ceph-mon[41670]: 4.14 scrub starts 2024-04-14T15:11:13.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:13 smithi090 ceph-mon[41670]: 4.14 scrub ok 2024-04-14T15:11:13.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:13 smithi090 ceph-mon[41670]: from='client.24626 v1:172.21.15.90:0/746960986' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:11:13.742 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2024-04-14T15:11:10.948145+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1014,"num_read_kb":860,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":1771324,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":606,"ondisk_log_size":606,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":236576,"kb_used_data":7964,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":749724640,"statfs":{"total":767960285184,"available":767718031360,"internally_reserved":0,"allocated":8155136,"data_stored":4551736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":234016306},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.186187"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.763793+0000","last_change":"2024-04-14T15:09:49.130210+0000","last_active":"2024-04-14T15:11:00.763793+0000","last_peered":"2024-04-14T15:11:00.763793+0000","last_clean":"2024-04-14T15:11:00.763793+0000","last_became_active":"2024-04-14T15:09:49.129636+0000","last_became_peered":"2024-04-14T15:09:49.129636+0000","last_unstale":"2024-04-14T15:11:00.763793+0000","last_undegraded":"2024-04-14T15:11:00.763793+0000","last_fullsized":"2024-04-14T15:11:00.763793+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:10.759713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":47,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.763604+0000","last_change":"2024-04-14T15:09:47.043210+0000","last_active":"2024-04-14T15:11:00.763604+0000","last_peered":"2024-04-14T15:11:00.763604+0000","last_clean":"2024-04-14T15:11:00.763604+0000","last_became_active":"2024-04-14T15:09:47.042538+0000","last_became_peered":"2024-04-14T15:09:47.042538+0000","last_unstale":"2024-04-14T15:11:00.763604+0000","last_undegraded":"2024-04-14T15:11:00.763604+0000","last_fullsized":"2024-04-14T15:11:00.763604+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:09:46.023663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:09.982965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.210463+0000","last_change":"2024-04-14T15:09:43.018880+0000","last_active":"2024-04-14T15:11:01.210463+0000","last_peered":"2024-04-14T15:11:01.210463+0000","last_clean":"2024-04-14T15:11:01.210463+0000","last_became_active":"2024-04-14T15:09:43.017351+0000","last_became_peered":"2024-04-14T15:09:43.017351+0000","last_unstale":"2024-04-14T15:11:01.210463+0000","last_undegraded":"2024-04-14T15:11:01.210463+0000","last_fullsized":"2024-04-14T15:11:01.210463+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:09:42.004317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:10.992803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"72'15","reported_seq":76,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764610+0000","last_change":"2024-04-14T15:09:45.034424+0000","last_active":"2024-04-14T15:11:00.764610+0000","last_peered":"2024-04-14T15:11:00.764610+0000","last_clean":"2024-04-14T15:11:00.764610+0000","last_became_active":"2024-04-14T15:09:45.034077+0000","last_became_peered":"2024-04-14T15:09:45.034077+0000","last_unstale":"2024-04-14T15:11:00.764610+0000","last_undegraded":"2024-04-14T15:11:00.764610+0000","last_fullsized":"2024-04-14T15:11:00.764610+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:08.933517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.763826+0000","last_change":"2024-04-14T15:09:49.059264+0000","last_active":"2024-04-14T15:11:00.763826+0000","last_peered":"2024-04-14T15:11:00.763826+0000","last_clean":"2024-04-14T15:11:00.763826+0000","last_became_active":"2024-04-14T15:09:49.058904+0000","last_became_peered":"2024-04-14T15:09:49.058904+0000","last_unstale":"2024-04-14T15:11:00.763826+0000","last_undegraded":"2024-04-14T15:11:00.763826+0000","last_fullsized":"2024-04-14T15:11:00.763826+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:13.856879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":47,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.763106+0000","last_change":"2024-04-14T15:09:47.041785+0000","last_active":"2024-04-14T15:11:00.763106+0000","last_peered":"2024-04-14T15:11:00.763106+0000","last_clean":"2024-04-14T15:11:00.763106+0000","last_became_active":"2024-04-14T15:09:47.041486+0000","last_became_peered":"2024-04-14T15:09:47.041486+0000","last_unstale":"2024-04-14T15:11:00.763106+0000","last_undegraded":"2024-04-14T15:11:00.763106+0000","last_fullsized":"2024-04-14T15:11:00.763106+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:09:46.023663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:03.085264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.765067+0000","last_change":"2024-04-14T15:09:43.023226+0000","last_active":"2024-04-14T15:11:00.765067+0000","last_peered":"2024-04-14T15:11:00.765067+0000","last_clean":"2024-04-14T15:11:00.765067+0000","last_became_active":"2024-04-14T15:09:43.022938+0000","last_became_peered":"2024-04-14T15:09:43.022938+0000","last_unstale":"2024-04-14T15:11:00.765067+0000","last_undegraded":"2024-04-14T15:11:00.765067+0000","last_fullsized":"2024-04-14T15:11:00.765067+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:09:42.004317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:10.564708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"72'9","reported_seq":67,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.763831+0000","last_change":"2024-04-14T15:09:45.032381+0000","last_active":"2024-04-14T15:11:00.763831+0000","last_peered":"2024-04-14T15:11:00.763831+0000","last_clean":"2024-04-14T15:11:00.763831+0000","last_became_active":"2024-04-14T15:09:45.032079+0000","last_became_peered":"2024-04-14T15:09:45.032079+0000","last_unstale":"2024-04-14T15:11:00.763831+0000","last_undegraded":"2024-04-14T15:11:00.763831+0000","last_fullsized":"2024-04-14T15:11:00.763831+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":51,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:03.745722+0000","last_change":"2024-04-14T15:11:03.745753+0000","last_active":"2024-04-14T15:11:03.745722+0000","last_peered":"2024-04-14T15:11:03.745722+0000","last_clean":"2024-04-14T15:11:03.745722+0000","last_became_active":"2024-04-14T15:09:49.069590+0000","last_became_peered":"2024-04-14T15:09:49.069590+0000","last_unstale":"2024-04-14T15:11:03.745722+0000","last_undegraded":"2024-04-14T15:11:03.745722+0000","last_fullsized":"2024-04-14T15:11:03.745722+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:03.745644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:03.745644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:29.376857+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:04.881903+0000","last_change":"2024-04-14T15:11:04.881923+0000","last_active":"2024-04-14T15:11:04.881903+0000","last_peered":"2024-04-14T15:11:04.881903+0000","last_clean":"2024-04-14T15:11:04.881903+0000","last_became_active":"2024-04-14T15:09:47.046517+0000","last_became_peered":"2024-04-14T15:09:47.046517+0000","last_unstale":"2024-04-14T15:11:04.881903+0000","last_undegraded":"2024-04-14T15:11:04.881903+0000","last_fullsized":"2024-04-14T15:11:04.881903+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:04.881846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:04.881846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:24.065968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":63,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.938224+0000","last_change":"2024-04-14T15:10:43.084925+0000","last_active":"2024-04-14T15:11:00.938224+0000","last_peered":"2024-04-14T15:11:00.938224+0000","last_clean":"2024-04-14T15:11:00.938224+0000","last_became_active":"2024-04-14T15:09:43.021986+0000","last_became_peered":"2024-04-14T15:09:43.021986+0000","last_unstale":"2024-04-14T15:11:00.938224+0000","last_undegraded":"2024-04-14T15:11:00.938224+0000","last_fullsized":"2024-04-14T15:11:00.938224+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:43.084820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:10:43.084820+0000","last_clean_scrub_stamp":"2024-04-14T15:10:43.084820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:55.633388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"72'5","reported_seq":61,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.763964+0000","last_change":"2024-04-14T15:09:45.028975+0000","last_active":"2024-04-14T15:11:00.763964+0000","last_peered":"2024-04-14T15:11:00.763964+0000","last_clean":"2024-04-14T15:11:00.763964+0000","last_became_active":"2024-04-14T15:09:45.028667+0000","last_became_peered":"2024-04-14T15:09:45.028667+0000","last_unstale":"2024-04-14T15:11:00.763964+0000","last_undegraded":"2024-04-14T15:11:00.763964+0000","last_fullsized":"2024-04-14T15:11:00.763964+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":51,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.209600+0000","last_change":"2024-04-14T15:10:54.765733+0000","last_active":"2024-04-14T15:11:01.209600+0000","last_peered":"2024-04-14T15:11:01.209600+0000","last_clean":"2024-04-14T15:11:01.209600+0000","last_became_active":"2024-04-14T15:09:49.061756+0000","last_became_peered":"2024-04-14T15:09:49.061756+0000","last_unstale":"2024-04-14T15:11:01.209600+0000","last_undegraded":"2024-04-14T15:11:01.209600+0000","last_fullsized":"2024-04-14T15:11:01.209600+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:54.765650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:10:54.765650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:59.282104+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":47,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.211280+0000","last_change":"2024-04-14T15:09:47.042682+0000","last_active":"2024-04-14T15:11:01.211280+0000","last_peered":"2024-04-14T15:11:01.211280+0000","last_clean":"2024-04-14T15:11:01.211280+0000","last_became_active":"2024-04-14T15:09:47.042384+0000","last_became_peered":"2024-04-14T15:09:47.042384+0000","last_unstale":"2024-04-14T15:11:01.211280+0000","last_undegraded":"2024-04-14T15:11:01.211280+0000","last_fullsized":"2024-04-14T15:11:01.211280+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:09:46.023663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:07.893460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764049+0000","last_change":"2024-04-14T15:10:07.764431+0000","last_active":"2024-04-14T15:11:00.764049+0000","last_peered":"2024-04-14T15:11:00.764049+0000","last_clean":"2024-04-14T15:11:00.764049+0000","last_became_active":"2024-04-14T15:10:07.764086+0000","last_became_peered":"2024-04-14T15:10:07.764086+0000","last_unstale":"2024-04-14T15:11:00.764049+0000","last_undegraded":"2024-04-14T15:11:00.764049+0000","last_fullsized":"2024-04-14T15:11:00.764049+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:09:42.004317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:08.369982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"72'9","reported_seq":67,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.763809+0000","last_change":"2024-04-14T15:09:45.030349+0000","last_active":"2024-04-14T15:11:00.763809+0000","last_peered":"2024-04-14T15:11:00.763809+0000","last_clean":"2024-04-14T15:11:00.763809+0000","last_became_active":"2024-04-14T15:09:45.030093+0000","last_became_peered":"2024-04-14T15:09:45.030093+0000","last_unstale":"2024-04-14T15:11:00.763809+0000","last_undegraded":"2024-04-14T15:11:00.763809+0000","last_fullsized":"2024-04-14T15:11:00.763809+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.763768+0000","last_change":"2024-04-14T15:09:49.128334+0000","last_active":"2024-04-14T15:11:00.763768+0000","last_peered":"2024-04-14T15:11:00.763768+0000","last_clean":"2024-04-14T15:11:00.763768+0000","last_became_active":"2024-04-14T15:09:49.128007+0000","last_became_peered":"2024-04-14T15:09:49.128007+0000","last_unstale":"2024-04-14T15:11:00.763768+0000","last_undegraded":"2024-04-14T15:11:00.763768+0000","last_fullsized":"2024-04-14T15:11:00.763768+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:04.103893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764349+0000","last_change":"2024-04-14T15:10:48.466668+0000","last_active":"2024-04-14T15:11:00.764349+0000","last_peered":"2024-04-14T15:11:00.764349+0000","last_clean":"2024-04-14T15:11:00.764349+0000","last_became_active":"2024-04-14T15:09:47.048045+0000","last_became_peered":"2024-04-14T15:09:47.048045+0000","last_unstale":"2024-04-14T15:11:00.764349+0000","last_undegraded":"2024-04-14T15:11:00.764349+0000","last_fullsized":"2024-04-14T15:11:00.764349+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:48.466556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:10:48.466556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:50.564963+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764000+0000","last_change":"2024-04-14T15:09:43.025155+0000","last_active":"2024-04-14T15:11:00.764000+0000","last_peered":"2024-04-14T15:11:00.764000+0000","last_clean":"2024-04-14T15:11:00.764000+0000","last_became_active":"2024-04-14T15:09:43.024853+0000","last_became_peered":"2024-04-14T15:09:43.024853+0000","last_unstale":"2024-04-14T15:11:00.764000+0000","last_undegraded":"2024-04-14T15:11:00.764000+0000","last_fullsized":"2024-04-14T15:11:00.764000+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:09:42.004317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:08.422553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"72'12","reported_seq":74,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.763590+0000","last_change":"2024-04-14T15:09:45.037659+0000","last_active":"2024-04-14T15:11:00.763590+0000","last_peered":"2024-04-14T15:11:00.763590+0000","last_clean":"2024-04-14T15:11:00.763590+0000","last_became_active":"2024-04-14T15:09:45.037173+0000","last_became_peered":"2024-04-14T15:09:45.037173+0000","last_unstale":"2024-04-14T15:11:00.763590+0000","last_undegraded":"2024-04-14T15:11:00.763590+0000","last_fullsized":"2024-04-14T15:11:00.763590+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:02.298243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":51,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:04.539857+0000","last_change":"2024-04-14T15:11:04.539877+0000","last_active":"2024-04-14T15:11:04.539857+0000","last_peered":"2024-04-14T15:11:04.539857+0000","last_clean":"2024-04-14T15:11:04.539857+0000","last_became_active":"2024-04-14T15:09:49.127979+0000","last_became_peered":"2024-04-14T15:09:49.127979+0000","last_unstale":"2024-04-14T15:11:04.539857+0000","last_undegraded":"2024-04-14T15:11:04.539857+0000","last_fullsized":"2024-04-14T15:11:04.539857+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:04.539789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:04.539789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:16.636271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":47,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764190+0000","last_change":"2024-04-14T15:09:47.043239+0000","last_active":"2024-04-14T15:11:00.764190+0000","last_peered":"2024-04-14T15:11:00.764190+0000","last_clean":"2024-04-14T15:11:00.764190+0000","last_became_active":"2024-04-14T15:09:47.042733+0000","last_became_peered":"2024-04-14T15:09:47.042733+0000","last_unstale":"2024-04-14T15:11:00.764190+0000","last_undegraded":"2024-04-14T15:11:00.764190+0000","last_fullsized":"2024-04-14T15:11:00.764190+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:09:46.023663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.940018+0000","last_change":"2024-04-14T15:09:43.022219+0000","last_active":"2024-04-14T15:11:00.940018+0000","last_peered":"2024-04-14T15:11:00.940018+0000","last_clean":"2024-04-14T15:11:00.940018+0000","last_became_active":"2024-04-14T15:09:43.021494+0000","last_became_peered":"2024-04-14T15:09:43.021494+0000","last_unstale":"2024-04-14T15:11:00.940018+0000","last_undegraded":"2024-04-14T15:11:00.940018+0000","last_fullsized":"2024-04-14T15:11:00.940018+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:09:42.004317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:02.243293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"72'15","reported_seq":76,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.211010+0000","last_change":"2024-04-14T15:09:45.033224+0000","last_active":"2024-04-14T15:11:01.211010+0000","last_peered":"2024-04-14T15:11:01.211010+0000","last_clean":"2024-04-14T15:11:01.211010+0000","last_became_active":"2024-04-14T15:09:45.032887+0000","last_became_peered":"2024-04-14T15:09:45.032887+0000","last_unstale":"2024-04-14T15:11:01.211010+0000","last_undegraded":"2024-04-14T15:11:01.211010+0000","last_fullsized":"2024-04-14T15:11:01.211010+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:04.849528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.763996+0000","last_change":"2024-04-14T15:09:49.057849+0000","last_active":"2024-04-14T15:11:00.763996+0000","last_peered":"2024-04-14T15:11:00.763996+0000","last_clean":"2024-04-14T15:11:00.763996+0000","last_became_active":"2024-04-14T15:09:49.057545+0000","last_became_peered":"2024-04-14T15:09:49.057545+0000","last_unstale":"2024-04-14T15:11:00.763996+0000","last_undegraded":"2024-04-14T15:11:00.763996+0000","last_fullsized":"2024-04-14T15:11:00.763996+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":47,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.211109+0000","last_change":"2024-04-14T15:09:47.046035+0000","last_active":"2024-04-14T15:11:01.211109+0000","last_peered":"2024-04-14T15:11:01.211109+0000","last_clean":"2024-04-14T15:11:01.211109+0000","last_became_active":"2024-04-14T15:09:47.045730+0000","last_became_peered":"2024-04-14T15:09:47.045730+0000","last_unstale":"2024-04-14T15:11:01.211109+0000","last_undegraded":"2024-04-14T15:11:01.211109+0000","last_fullsized":"2024-04-14T15:11:01.211109+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:09:46.023663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:11.966020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"65'2","reported_seq":70,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764422+0000","last_change":"2024-04-14T15:10:57.406574+0000","last_active":"2024-04-14T15:11:00.764422+0000","last_peered":"2024-04-14T15:11:00.764422+0000","last_clean":"2024-04-14T15:11:00.764422+0000","last_became_active":"2024-04-14T15:09:43.020401+0000","last_became_peered":"2024-04-14T15:09:43.020401+0000","last_unstale":"2024-04-14T15:11:00.764422+0000","last_undegraded":"2024-04-14T15:11:00.764422+0000","last_fullsized":"2024-04-14T15:11:00.764422+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-04-14T15:10:57.406478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:10:57.406478+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:25.577150+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"72'11","reported_seq":88,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.939685+0000","last_change":"2024-04-14T15:10:07.771479+0000","last_active":"2024-04-14T15:11:00.939685+0000","last_peered":"2024-04-14T15:11:00.939685+0000","last_clean":"2024-04-14T15:11:00.939685+0000","last_became_active":"2024-04-14T15:10:07.764183+0000","last_became_peered":"2024-04-14T15:10:07.764183+0000","last_unstale":"2024-04-14T15:11:00.939685+0000","last_undegraded":"2024-04-14T15:11:00.939685+0000","last_fullsized":"2024-04-14T15:11:00.939685+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:13.054416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"72'1","reported_seq":52,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.938612+0000","last_change":"2024-04-14T15:10:59.889342+0000","last_active":"2024-04-14T15:11:00.938612+0000","last_peered":"2024-04-14T15:11:00.938612+0000","last_clean":"2024-04-14T15:11:00.938612+0000","last_became_active":"2024-04-14T15:09:49.062051+0000","last_became_peered":"2024-04-14T15:09:49.062051+0000","last_unstale":"2024-04-14T15:11:00.938612+0000","last_undegraded":"2024-04-14T15:11:00.938612+0000","last_fullsized":"2024-04-14T15:11:00.938612+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-04-14T15:10:59.889236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:10:59.889236+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:03.689276+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"72'8","reported_seq":84,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:03.082668+0000","last_change":"2024-04-14T15:10:58.268612+0000","last_active":"2024-04-14T15:11:03.082668+0000","last_peered":"2024-04-14T15:11:03.082668+0000","last_clean":"2024-04-14T15:11:03.082668+0000","last_became_active":"2024-04-14T15:09:47.048216+0000","last_became_peered":"2024-04-14T15:09:47.048216+0000","last_unstale":"2024-04-14T15:11:03.082668+0000","last_undegraded":"2024-04-14T15:11:03.082668+0000","last_fullsized":"2024-04-14T15:11:03.082668+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:10:58.268488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:10:58.268488+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:04.149311+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":63,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.210561+0000","last_change":"2024-04-14T15:10:52.882061+0000","last_active":"2024-04-14T15:11:01.210561+0000","last_peered":"2024-04-14T15:11:01.210561+0000","last_clean":"2024-04-14T15:11:01.210561+0000","last_became_active":"2024-04-14T15:09:43.023257+0000","last_became_peered":"2024-04-14T15:09:43.023257+0000","last_unstale":"2024-04-14T15:11:01.210561+0000","last_undegraded":"2024-04-14T15:11:01.210561+0000","last_fullsized":"2024-04-14T15:11:01.210561+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:52.881916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:10:52.881916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:58.458431+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"72'10","reported_seq":66,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.211106+0000","last_change":"2024-04-14T15:09:45.181795+0000","last_active":"2024-04-14T15:11:01.211106+0000","last_peered":"2024-04-14T15:11:01.211106+0000","last_clean":"2024-04-14T15:11:01.211106+0000","last_became_active":"2024-04-14T15:09:45.181456+0000","last_became_peered":"2024-04-14T15:09:45.181456+0000","last_unstale":"2024-04-14T15:11:01.211106+0000","last_undegraded":"2024-04-14T15:11:01.211106+0000","last_fullsized":"2024-04-14T15:11:01.211106+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:05.909429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764295+0000","last_change":"2024-04-14T15:09:49.130071+0000","last_active":"2024-04-14T15:11:00.764295+0000","last_peered":"2024-04-14T15:11:00.764295+0000","last_clean":"2024-04-14T15:11:00.764295+0000","last_became_active":"2024-04-14T15:09:49.129392+0000","last_became_peered":"2024-04-14T15:09:49.129392+0000","last_unstale":"2024-04-14T15:11:00.764295+0000","last_undegraded":"2024-04-14T15:11:00.764295+0000","last_fullsized":"2024-04-14T15:11:00.764295+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:02.611406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:05.917034+0000","last_change":"2024-04-14T15:11:05.917054+0000","last_active":"2024-04-14T15:11:05.917034+0000","last_peered":"2024-04-14T15:11:05.917034+0000","last_clean":"2024-04-14T15:11:05.917034+0000","last_became_active":"2024-04-14T15:09:47.044665+0000","last_became_peered":"2024-04-14T15:09:47.044665+0000","last_unstale":"2024-04-14T15:11:05.917034+0000","last_undegraded":"2024-04-14T15:11:05.917034+0000","last_fullsized":"2024-04-14T15:11:05.917034+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:05.916966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:05.916966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:35.330112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":63,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.939825+0000","last_change":"2024-04-14T15:10:54.239518+0000","last_active":"2024-04-14T15:11:00.939825+0000","last_peered":"2024-04-14T15:11:00.939825+0000","last_clean":"2024-04-14T15:11:00.939825+0000","last_became_active":"2024-04-14T15:09:43.023220+0000","last_became_peered":"2024-04-14T15:09:43.023220+0000","last_unstale":"2024-04-14T15:11:00.939825+0000","last_undegraded":"2024-04-14T15:11:00.939825+0000","last_fullsized":"2024-04-14T15:11:00.939825+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:54.239409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:10:54.239409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:11.539313+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"72'11","reported_seq":78,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:06.492751+0000","last_change":"2024-04-14T15:11:06.492784+0000","last_active":"2024-04-14T15:11:06.492751+0000","last_peered":"2024-04-14T15:11:06.492751+0000","last_clean":"2024-04-14T15:11:06.492751+0000","last_became_active":"2024-04-14T15:09:45.180332+0000","last_became_peered":"2024-04-14T15:09:45.180332+0000","last_unstale":"2024-04-14T15:11:06.492751+0000","last_undegraded":"2024-04-14T15:11:06.492751+0000","last_fullsized":"2024-04-14T15:11:06.492751+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-04-14T15:11:06.492677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:06.492677+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:24.577564+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":51,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:07.901818+0000","last_change":"2024-04-14T15:11:07.901838+0000","last_active":"2024-04-14T15:11:07.901818+0000","last_peered":"2024-04-14T15:11:07.901818+0000","last_clean":"2024-04-14T15:11:07.901818+0000","last_became_active":"2024-04-14T15:09:49.061359+0000","last_became_peered":"2024-04-14T15:09:49.061359+0000","last_unstale":"2024-04-14T15:11:07.901818+0000","last_undegraded":"2024-04-14T15:11:07.901818+0000","last_fullsized":"2024-04-14T15:11:07.901818+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:07.901761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:07.901761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:35.316850+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.939704+0000","last_change":"2024-04-14T15:10:50.229008+0000","last_active":"2024-04-14T15:11:00.939704+0000","last_peered":"2024-04-14T15:11:00.939704+0000","last_clean":"2024-04-14T15:11:00.939704+0000","last_became_active":"2024-04-14T15:09:47.041074+0000","last_became_peered":"2024-04-14T15:09:47.041074+0000","last_unstale":"2024-04-14T15:11:00.939704+0000","last_undegraded":"2024-04-14T15:11:00.939704+0000","last_fullsized":"2024-04-14T15:11:00.939704+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:50.228903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:10:50.228903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:17.158842+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":63,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:08.712389+0000","last_change":"2024-04-14T15:11:08.712429+0000","last_active":"2024-04-14T15:11:08.712389+0000","last_peered":"2024-04-14T15:11:08.712389+0000","last_clean":"2024-04-14T15:11:08.712389+0000","last_became_active":"2024-04-14T15:09:43.026786+0000","last_became_peered":"2024-04-14T15:09:43.026786+0000","last_unstale":"2024-04-14T15:11:08.712389+0000","last_undegraded":"2024-04-14T15:11:08.712389+0000","last_fullsized":"2024-04-14T15:11:08.712389+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:08.712289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:08.712289+0000","last_clean_scrub_stamp":"2024-04-14T15:11:08.712289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:11.190004+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"72'4","reported_seq":57,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764146+0000","last_change":"2024-04-14T15:09:45.039145+0000","last_active":"2024-04-14T15:11:00.764146+0000","last_peered":"2024-04-14T15:11:00.764146+0000","last_clean":"2024-04-14T15:11:00.764146+0000","last_became_active":"2024-04-14T15:09:45.038815+0000","last_became_peered":"2024-04-14T15:09:45.038815+0000","last_unstale":"2024-04-14T15:11:00.764146+0000","last_undegraded":"2024-04-14T15:11:00.764146+0000","last_fullsized":"2024-04-14T15:11:00.764146+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:13.528834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764325+0000","last_change":"2024-04-14T15:09:49.066223+0000","last_active":"2024-04-14T15:11:00.764325+0000","last_peered":"2024-04-14T15:11:00.764325+0000","last_clean":"2024-04-14T15:11:00.764325+0000","last_became_active":"2024-04-14T15:09:49.065896+0000","last_became_peered":"2024-04-14T15:09:49.065896+0000","last_unstale":"2024-04-14T15:11:00.764325+0000","last_undegraded":"2024-04-14T15:11:00.764325+0000","last_fullsized":"2024-04-14T15:11:00.764325+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:09.025179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":47,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764746+0000","last_change":"2024-04-14T15:09:47.041972+0000","last_active":"2024-04-14T15:11:00.764746+0000","last_peered":"2024-04-14T15:11:00.764746+0000","last_clean":"2024-04-14T15:11:00.764746+0000","last_became_active":"2024-04-14T15:09:47.041683+0000","last_became_peered":"2024-04-14T15:09:47.041683+0000","last_unstale":"2024-04-14T15:11:00.764746+0000","last_undegraded":"2024-04-14T15:11:00.764746+0000","last_fullsized":"2024-04-14T15:11:00.764746+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:09:46.023663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:08.100556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764274+0000","last_change":"2024-04-14T15:09:43.023867+0000","last_active":"2024-04-14T15:11:00.764274+0000","last_peered":"2024-04-14T15:11:00.764274+0000","last_clean":"2024-04-14T15:11:00.764274+0000","last_became_active":"2024-04-14T15:09:43.023583+0000","last_became_peered":"2024-04-14T15:09:43.023583+0000","last_unstale":"2024-04-14T15:11:00.764274+0000","last_undegraded":"2024-04-14T15:11:00.764274+0000","last_fullsized":"2024-04-14T15:11:00.764274+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:09:42.004317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:05.031251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"72'11","reported_seq":70,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764103+0000","last_change":"2024-04-14T15:09:45.026357+0000","last_active":"2024-04-14T15:11:00.764103+0000","last_peered":"2024-04-14T15:11:00.764103+0000","last_clean":"2024-04-14T15:11:00.764103+0000","last_became_active":"2024-04-14T15:09:45.026065+0000","last_became_peered":"2024-04-14T15:09:45.026065+0000","last_unstale":"2024-04-14T15:11:00.764103+0000","last_undegraded":"2024-04-14T15:11:00.764103+0000","last_fullsized":"2024-04-14T15:11:00.764103+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:08.741092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":51,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.209828+0000","last_change":"2024-04-14T15:10:56.820210+0000","last_active":"2024-04-14T15:11:01.209828+0000","last_peered":"2024-04-14T15:11:01.209828+0000","last_clean":"2024-04-14T15:11:01.209828+0000","last_became_active":"2024-04-14T15:09:49.059711+0000","last_became_peered":"2024-04-14T15:09:49.059711+0000","last_unstale":"2024-04-14T15:11:01.209828+0000","last_undegraded":"2024-04-14T15:11:01.209828+0000","last_fullsized":"2024-04-14T15:11:01.209828+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:56.820123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:10:56.820123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:08.868464+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":47,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764103+0000","last_change":"2024-04-14T15:09:47.045205+0000","last_active":"2024-04-14T15:11:00.764103+0000","last_peered":"2024-04-14T15:11:00.764103+0000","last_clean":"2024-04-14T15:11:00.764103+0000","last_became_active":"2024-04-14T15:09:47.044891+0000","last_became_peered":"2024-04-14T15:09:47.044891+0000","last_unstale":"2024-04-14T15:11:00.764103+0000","last_undegraded":"2024-04-14T15:11:00.764103+0000","last_fullsized":"2024-04-14T15:11:00.764103+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:09:46.023663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":63,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:03.304615+0000","last_change":"2024-04-14T15:11:03.304640+0000","last_active":"2024-04-14T15:11:03.304615+0000","last_peered":"2024-04-14T15:11:03.304615+0000","last_clean":"2024-04-14T15:11:03.304615+0000","last_became_active":"2024-04-14T15:09:43.022480+0000","last_became_peered":"2024-04-14T15:09:43.022480+0000","last_unstale":"2024-04-14T15:11:03.304615+0000","last_undegraded":"2024-04-14T15:11:03.304615+0000","last_fullsized":"2024-04-14T15:11:03.304615+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:03.304541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:03.304541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:22.825961+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"72'9","reported_seq":67,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764699+0000","last_change":"2024-04-14T15:09:45.026649+0000","last_active":"2024-04-14T15:11:00.764699+0000","last_peered":"2024-04-14T15:11:00.764699+0000","last_clean":"2024-04-14T15:11:00.764699+0000","last_became_active":"2024-04-14T15:09:45.026363+0000","last_became_peered":"2024-04-14T15:09:45.026363+0000","last_unstale":"2024-04-14T15:11:00.764699+0000","last_undegraded":"2024-04-14T15:11:00.764699+0000","last_fullsized":"2024-04-14T15:11:00.764699+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:03.421380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764776+0000","last_change":"2024-04-14T15:09:49.065895+0000","last_active":"2024-04-14T15:11:00.764776+0000","last_peered":"2024-04-14T15:11:00.764776+0000","last_clean":"2024-04-14T15:11:00.764776+0000","last_became_active":"2024-04-14T15:09:49.065242+0000","last_became_peered":"2024-04-14T15:09:49.065242+0000","last_unstale":"2024-04-14T15:11:00.764776+0000","last_undegraded":"2024-04-14T15:11:00.764776+0000","last_fullsized":"2024-04-14T15:11:00.764776+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"72'8","reported_seq":76,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:03.083036+0000","last_change":"2024-04-14T15:09:47.047381+0000","last_active":"2024-04-14T15:11:03.083036+0000","last_peered":"2024-04-14T15:11:03.083036+0000","last_clean":"2024-04-14T15:11:03.083036+0000","last_became_active":"2024-04-14T15:09:47.047072+0000","last_became_peered":"2024-04-14T15:09:47.047072+0000","last_unstale":"2024-04-14T15:11:03.083036+0000","last_undegraded":"2024-04-14T15:11:03.083036+0000","last_fullsized":"2024-04-14T15:11:03.083036+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:09:46.023663+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.210904+0000","last_change":"2024-04-14T15:09:43.019612+0000","last_active":"2024-04-14T15:11:01.210904+0000","last_peered":"2024-04-14T15:11:01.210904+0000","last_clean":"2024-04-14T15:11:01.210904+0000","last_became_active":"2024-04-14T15:09:43.019301+0000","last_became_peered":"2024-04-14T15:09:43.019301+0000","last_unstale":"2024-04-14T15:11:01.210904+0000","last_undegraded":"2024-04-14T15:11:01.210904+0000","last_fullsized":"2024-04-14T15:11:01.210904+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:09:42.004317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:01.268097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"72'9","reported_seq":75,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.211409+0000","last_change":"2024-04-14T15:10:56.858475+0000","last_active":"2024-04-14T15:11:01.211409+0000","last_peered":"2024-04-14T15:11:01.211409+0000","last_clean":"2024-04-14T15:11:01.211409+0000","last_became_active":"2024-04-14T15:09:45.178857+0000","last_became_peered":"2024-04-14T15:09:45.178857+0000","last_unstale":"2024-04-14T15:11:01.211409+0000","last_undegraded":"2024-04-14T15:11:01.211409+0000","last_fullsized":"2024-04-14T15:11:01.211409+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-04-14T15:10:56.858323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:10:56.858323+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:13.702279+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.209643+0000","last_change":"2024-04-14T15:09:49.066448+0000","last_active":"2024-04-14T15:11:01.209643+0000","last_peered":"2024-04-14T15:11:01.209643+0000","last_clean":"2024-04-14T15:11:01.209643+0000","last_became_active":"2024-04-14T15:09:49.066068+0000","last_became_peered":"2024-04-14T15:09:49.066068+0000","last_unstale":"2024-04-14T15:11:01.209643+0000","last_undegraded":"2024-04-14T15:11:01.209643+0000","last_fullsized":"2024-04-14T15:11:01.209643+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:14.252283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"72'8","reported_seq":73,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:03.082718+0000","last_change":"2024-04-14T15:09:47.037002+0000","last_active":"2024-04-14T15:11:03.082718+0000","last_peered":"2024-04-14T15:11:03.082718+0000","last_clean":"2024-04-14T15:11:03.082718+0000","last_became_active":"2024-04-14T15:09:47.036688+0000","last_became_peered":"2024-04-14T15:09:47.036688+0000","last_unstale":"2024-04-14T15:11:03.082718+0000","last_undegraded":"2024-04-14T15:11:03.082718+0000","last_fullsized":"2024-04-14T15:11:03.082718+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:09:46.023663+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.210693+0000","last_change":"2024-04-14T15:09:43.028602+0000","last_active":"2024-04-14T15:11:01.210693+0000","last_peered":"2024-04-14T15:11:01.210693+0000","last_clean":"2024-04-14T15:11:01.210693+0000","last_became_active":"2024-04-14T15:09:43.027776+0000","last_became_peered":"2024-04-14T15:09:43.027776+0000","last_unstale":"2024-04-14T15:11:01.210693+0000","last_undegraded":"2024-04-14T15:11:01.210693+0000","last_fullsized":"2024-04-14T15:11:01.210693+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:09:42.004317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:01.870579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"72'10","reported_seq":66,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764159+0000","last_change":"2024-04-14T15:09:45.179495+0000","last_active":"2024-04-14T15:11:00.764159+0000","last_peered":"2024-04-14T15:11:00.764159+0000","last_clean":"2024-04-14T15:11:00.764159+0000","last_became_active":"2024-04-14T15:09:45.179165+0000","last_became_peered":"2024-04-14T15:09:45.179165+0000","last_unstale":"2024-04-14T15:11:00.764159+0000","last_undegraded":"2024-04-14T15:11:00.764159+0000","last_fullsized":"2024-04-14T15:11:00.764159+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.938088+0000","last_change":"2024-04-14T15:09:49.130725+0000","last_active":"2024-04-14T15:11:00.938088+0000","last_peered":"2024-04-14T15:11:00.938088+0000","last_clean":"2024-04-14T15:11:00.938088+0000","last_became_active":"2024-04-14T15:09:49.130438+0000","last_became_peered":"2024-04-14T15:09:49.130438+0000","last_unstale":"2024-04-14T15:11:00.938088+0000","last_undegraded":"2024-04-14T15:11:00.938088+0000","last_fullsized":"2024-04-14T15:11:00.938088+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:16.730190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.211603+0000","last_change":"2024-04-14T15:10:53.900979+0000","last_active":"2024-04-14T15:11:01.211603+0000","last_peered":"2024-04-14T15:11:01.211603+0000","last_clean":"2024-04-14T15:11:01.211603+0000","last_became_active":"2024-04-14T15:09:47.045349+0000","last_became_peered":"2024-04-14T15:09:47.045349+0000","last_unstale":"2024-04-14T15:11:01.211603+0000","last_undegraded":"2024-04-14T15:11:01.211603+0000","last_fullsized":"2024-04-14T15:11:01.211603+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:53.900878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:10:53.900878+0000","last_clean_scrub_stamp":"2024-04-14T15:10:53.900878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:12.123204+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.939789+0000","last_change":"2024-04-14T15:09:43.019353+0000","last_active":"2024-04-14T15:11:00.939789+0000","last_peered":"2024-04-14T15:11:00.939789+0000","last_clean":"2024-04-14T15:11:00.939789+0000","last_became_active":"2024-04-14T15:09:43.018987+0000","last_became_peered":"2024-04-14T15:09:43.018987+0000","last_unstale":"2024-04-14T15:11:00.939789+0000","last_undegraded":"2024-04-14T15:11:00.939789+0000","last_fullsized":"2024-04-14T15:11:00.939789+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:09:42.004317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"72'6","reported_seq":60,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.763707+0000","last_change":"2024-04-14T15:09:45.180445+0000","last_active":"2024-04-14T15:11:00.763707+0000","last_peered":"2024-04-14T15:11:00.763707+0000","last_clean":"2024-04-14T15:11:00.763707+0000","last_became_active":"2024-04-14T15:09:45.180037+0000","last_became_peered":"2024-04-14T15:09:45.180037+0000","last_unstale":"2024-04-14T15:11:00.763707+0000","last_undegraded":"2024-04-14T15:11:00.763707+0000","last_fullsized":"2024-04-14T15:11:00.763707+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:08.718920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":51,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.210053+0000","last_change":"2024-04-14T15:10:56.886642+0000","last_active":"2024-04-14T15:11:01.210053+0000","last_peered":"2024-04-14T15:11:01.210053+0000","last_clean":"2024-04-14T15:11:01.210053+0000","last_became_active":"2024-04-14T15:09:49.064296+0000","last_became_peered":"2024-04-14T15:09:49.064296+0000","last_unstale":"2024-04-14T15:11:01.210053+0000","last_undegraded":"2024-04-14T15:11:01.210053+0000","last_fullsized":"2024-04-14T15:11:01.210053+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:56.886513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:10:56.886513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:17.056264+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:07.485898+0000","last_change":"2024-04-14T15:11:07.485922+0000","last_active":"2024-04-14T15:11:07.485898+0000","last_peered":"2024-04-14T15:11:07.485898+0000","last_clean":"2024-04-14T15:11:07.485898+0000","last_became_active":"2024-04-14T15:09:47.044816+0000","last_became_peered":"2024-04-14T15:09:47.044816+0000","last_unstale":"2024-04-14T15:11:07.485898+0000","last_undegraded":"2024-04-14T15:11:07.485898+0000","last_fullsized":"2024-04-14T15:11:07.485898+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:07.485821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:07.485821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:21.021299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":63,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.210752+0000","last_change":"2024-04-14T15:10:43.844823+0000","last_active":"2024-04-14T15:11:01.210752+0000","last_peered":"2024-04-14T15:11:01.210752+0000","last_clean":"2024-04-14T15:11:01.210752+0000","last_became_active":"2024-04-14T15:09:43.016788+0000","last_became_peered":"2024-04-14T15:09:43.016788+0000","last_unstale":"2024-04-14T15:11:01.210752+0000","last_undegraded":"2024-04-14T15:11:01.210752+0000","last_fullsized":"2024-04-14T15:11:01.210752+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:43.844693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:10:43.844693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:51.339452+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"72'9","reported_seq":67,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.210984+0000","last_change":"2024-04-14T15:09:45.180769+0000","last_active":"2024-04-14T15:11:01.210984+0000","last_peered":"2024-04-14T15:11:01.210984+0000","last_clean":"2024-04-14T15:11:01.210984+0000","last_became_active":"2024-04-14T15:09:45.180375+0000","last_became_peered":"2024-04-14T15:09:45.180375+0000","last_unstale":"2024-04-14T15:11:01.210984+0000","last_undegraded":"2024-04-14T15:11:01.210984+0000","last_fullsized":"2024-04-14T15:11:01.210984+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:12.591629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764186+0000","last_change":"2024-04-14T15:09:49.069204+0000","last_active":"2024-04-14T15:11:00.764186+0000","last_peered":"2024-04-14T15:11:00.764186+0000","last_clean":"2024-04-14T15:11:00.764186+0000","last_became_active":"2024-04-14T15:09:49.068883+0000","last_became_peered":"2024-04-14T15:09:49.068883+0000","last_unstale":"2024-04-14T15:11:00.764186+0000","last_undegraded":"2024-04-14T15:11:00.764186+0000","last_fullsized":"2024-04-14T15:11:00.764186+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:12.232108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"65'3","reported_seq":73,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764604+0000","last_change":"2024-04-14T15:09:43.020827+0000","last_active":"2024-04-14T15:11:00.764604+0000","last_peered":"2024-04-14T15:11:00.764604+0000","last_clean":"2024-04-14T15:11:00.764604+0000","last_became_active":"2024-04-14T15:09:43.020490+0000","last_became_peered":"2024-04-14T15:09:43.020490+0000","last_unstale":"2024-04-14T15:11:00.764604+0000","last_undegraded":"2024-04-14T15:11:00.764604+0000","last_fullsized":"2024-04-14T15:11:00.764604+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:09:42.004317+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:07.143386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":528,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"72'12","reported_seq":82,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.763942+0000","last_change":"2024-04-14T15:10:53.448340+0000","last_active":"2024-04-14T15:11:00.763942+0000","last_peered":"2024-04-14T15:11:00.763942+0000","last_clean":"2024-04-14T15:11:00.763942+0000","last_became_active":"2024-04-14T15:09:45.032892+0000","last_became_peered":"2024-04-14T15:09:45.032892+0000","last_unstale":"2024-04-14T15:11:00.763942+0000","last_undegraded":"2024-04-14T15:11:00.763942+0000","last_fullsized":"2024-04-14T15:11:00.763942+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'12","last_scrub_stamp":"2024-04-14T15:10:53.448218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:10:53.448218+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:56.821958+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":47,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.210659+0000","last_change":"2024-04-14T15:09:47.049725+0000","last_active":"2024-04-14T15:11:01.210659+0000","last_peered":"2024-04-14T15:11:01.210659+0000","last_clean":"2024-04-14T15:11:01.210659+0000","last_became_active":"2024-04-14T15:09:47.049005+0000","last_became_peered":"2024-04-14T15:09:47.049005+0000","last_unstale":"2024-04-14T15:11:01.210659+0000","last_undegraded":"2024-04-14T15:11:01.210659+0000","last_fullsized":"2024-04-14T15:11:01.210659+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:09:46.023663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.209730+0000","last_change":"2024-04-14T15:09:49.067038+0000","last_active":"2024-04-14T15:11:01.209730+0000","last_peered":"2024-04-14T15:11:01.209730+0000","last_clean":"2024-04-14T15:11:01.209730+0000","last_became_active":"2024-04-14T15:09:49.066648+0000","last_became_peered":"2024-04-14T15:09:49.066648+0000","last_unstale":"2024-04-14T15:11:01.209730+0000","last_undegraded":"2024-04-14T15:11:01.209730+0000","last_fullsized":"2024-04-14T15:11:01.209730+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.763844+0000","last_change":"2024-04-14T15:09:43.022909+0000","last_active":"2024-04-14T15:11:00.763844+0000","last_peered":"2024-04-14T15:11:00.763844+0000","last_clean":"2024-04-14T15:11:00.763844+0000","last_became_active":"2024-04-14T15:09:43.022597+0000","last_became_peered":"2024-04-14T15:09:43.022597+0000","last_unstale":"2024-04-14T15:11:00.763844+0000","last_undegraded":"2024-04-14T15:11:00.763844+0000","last_fullsized":"2024-04-14T15:11:00.763844+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:09:42.004317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:11.088019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"72'10","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.210277+0000","last_change":"2024-04-14T15:10:07.770475+0000","last_active":"2024-04-14T15:11:01.210277+0000","last_peered":"2024-04-14T15:11:01.210277+0000","last_clean":"2024-04-14T15:11:01.210277+0000","last_became_active":"2024-04-14T15:10:07.764042+0000","last_became_peered":"2024-04-14T15:10:07.764042+0000","last_unstale":"2024-04-14T15:11:01.210277+0000","last_undegraded":"2024-04-14T15:11:01.210277+0000","last_fullsized":"2024-04-14T15:11:01.210277+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:04.539099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"72'8","reported_seq":73,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:03.082438+0000","last_change":"2024-04-14T15:09:47.048177+0000","last_active":"2024-04-14T15:11:03.082438+0000","last_peered":"2024-04-14T15:11:03.082438+0000","last_clean":"2024-04-14T15:11:03.082438+0000","last_became_active":"2024-04-14T15:09:47.047699+0000","last_became_peered":"2024-04-14T15:09:47.047699+0000","last_unstale":"2024-04-14T15:11:03.082438+0000","last_undegraded":"2024-04-14T15:11:03.082438+0000","last_fullsized":"2024-04-14T15:11:03.082438+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:09:46.023663+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.765065+0000","last_change":"2024-04-14T15:09:49.058715+0000","last_active":"2024-04-14T15:11:00.765065+0000","last_peered":"2024-04-14T15:11:00.765065+0000","last_clean":"2024-04-14T15:11:00.765065+0000","last_became_active":"2024-04-14T15:09:49.058387+0000","last_became_peered":"2024-04-14T15:09:49.058387+0000","last_unstale":"2024-04-14T15:11:00.765065+0000","last_undegraded":"2024-04-14T15:11:00.765065+0000","last_fullsized":"2024-04-14T15:11:00.765065+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:16.162642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.210796+0000","last_change":"2024-04-14T15:09:43.028748+0000","last_active":"2024-04-14T15:11:01.210796+0000","last_peered":"2024-04-14T15:11:01.210796+0000","last_clean":"2024-04-14T15:11:01.210796+0000","last_became_active":"2024-04-14T15:09:43.028362+0000","last_became_peered":"2024-04-14T15:09:43.028362+0000","last_unstale":"2024-04-14T15:11:01.210796+0000","last_undegraded":"2024-04-14T15:11:01.210796+0000","last_fullsized":"2024-04-14T15:11:01.210796+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:09:42.004317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"72'28","reported_seq":117,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:03.082440+0000","last_change":"2024-04-14T15:09:45.034501+0000","last_active":"2024-04-14T15:11:03.082440+0000","last_peered":"2024-04-14T15:11:03.082440+0000","last_clean":"2024-04-14T15:11:03.082440+0000","last_became_active":"2024-04-14T15:09:45.034168+0000","last_became_peered":"2024-04-14T15:09:45.034168+0000","last_unstale":"2024-04-14T15:11:03.082440+0000","last_undegraded":"2024-04-14T15:11:03.082440+0000","last_fullsized":"2024-04-14T15:11:03.082440+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":47,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.210782+0000","last_change":"2024-04-14T15:09:47.040508+0000","last_active":"2024-04-14T15:11:01.210782+0000","last_peered":"2024-04-14T15:11:01.210782+0000","last_clean":"2024-04-14T15:11:01.210782+0000","last_became_active":"2024-04-14T15:09:47.040185+0000","last_became_peered":"2024-04-14T15:09:47.040185+0000","last_unstale":"2024-04-14T15:11:01.210782+0000","last_undegraded":"2024-04-14T15:11:01.210782+0000","last_fullsized":"2024-04-14T15:11:01.210782+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:09:46.023663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:03.491052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":51,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.939013+0000","last_change":"2024-04-14T15:11:00.939052+0000","last_active":"2024-04-14T15:11:00.939013+0000","last_peered":"2024-04-14T15:11:00.939013+0000","last_clean":"2024-04-14T15:11:00.939013+0000","last_became_active":"2024-04-14T15:09:49.130307+0000","last_became_peered":"2024-04-14T15:09:49.130307+0000","last_unstale":"2024-04-14T15:11:00.939013+0000","last_undegraded":"2024-04-14T15:11:00.939013+0000","last_fullsized":"2024-04-14T15:11:00.939013+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:00.938917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:00.938917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:17.427163+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"73'5","reported_seq":153,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:07.762306+0000","last_change":"2024-04-14T15:11:03.192957+0000","last_active":"2024-04-14T15:11:07.762306+0000","last_peered":"2024-04-14T15:11:07.762306+0000","last_clean":"2024-04-14T15:11:07.762306+0000","last_became_active":"2024-04-14T15:09:43.016544+0000","last_became_peered":"2024-04-14T15:09:43.016544+0000","last_unstale":"2024-04-14T15:11:07.762306+0000","last_undegraded":"2024-04-14T15:11:07.762306+0000","last_fullsized":"2024-04-14T15:11:07.762306+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-04-14T15:11:03.192810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:03.192810+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:21.177179+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00056542099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":80,"num_read_kb":75,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":63,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.209765+0000","last_change":"2024-04-14T15:10:42.819764+0000","last_active":"2024-04-14T15:11:01.209765+0000","last_peered":"2024-04-14T15:11:01.209765+0000","last_clean":"2024-04-14T15:11:01.209765+0000","last_became_active":"2024-04-14T15:09:43.021188+0000","last_became_peered":"2024-04-14T15:09:43.021188+0000","last_unstale":"2024-04-14T15:11:01.209765+0000","last_undegraded":"2024-04-14T15:11:01.209765+0000","last_fullsized":"2024-04-14T15:11:01.209765+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:42.819681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:10:42.819681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:52.363715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"72'13","reported_seq":78,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764908+0000","last_change":"2024-04-14T15:09:45.034537+0000","last_active":"2024-04-14T15:11:00.764908+0000","last_peered":"2024-04-14T15:11:00.764908+0000","last_clean":"2024-04-14T15:11:00.764908+0000","last_became_active":"2024-04-14T15:09:45.033883+0000","last_became_peered":"2024-04-14T15:09:45.033883+0000","last_unstale":"2024-04-14T15:11:00.764908+0000","last_undegraded":"2024-04-14T15:11:00.764908+0000","last_fullsized":"2024-04-14T15:11:00.764908+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:04.692317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.210758+0000","last_change":"2024-04-14T15:10:55.841920+0000","last_active":"2024-04-14T15:11:01.210758+0000","last_peered":"2024-04-14T15:11:01.210758+0000","last_clean":"2024-04-14T15:11:01.210758+0000","last_became_active":"2024-04-14T15:09:47.049457+0000","last_became_peered":"2024-04-14T15:09:47.049457+0000","last_unstale":"2024-04-14T15:11:01.210758+0000","last_undegraded":"2024-04-14T15:11:01.210758+0000","last_fullsized":"2024-04-14T15:11:01.210758+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:55.841832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:10:55.841832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:09.497072+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.210445+0000","last_change":"2024-04-14T15:09:49.067392+0000","last_active":"2024-04-14T15:11:01.210445+0000","last_peered":"2024-04-14T15:11:01.210445+0000","last_clean":"2024-04-14T15:11:01.210445+0000","last_became_active":"2024-04-14T15:09:49.067031+0000","last_became_peered":"2024-04-14T15:09:49.067031+0000","last_unstale":"2024-04-14T15:11:01.210445+0000","last_undegraded":"2024-04-14T15:11:01.210445+0000","last_fullsized":"2024-04-14T15:11:01.210445+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:13.012527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"69'2","reported_seq":64,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.763287+0000","last_change":"2024-04-14T15:09:47.043261+0000","last_active":"2024-04-14T15:11:00.763287+0000","last_peered":"2024-04-14T15:11:00.763287+0000","last_clean":"2024-04-14T15:11:00.763287+0000","last_became_active":"2024-04-14T15:09:43.024882+0000","last_became_peered":"2024-04-14T15:09:43.024882+0000","last_unstale":"2024-04-14T15:11:00.763287+0000","last_undegraded":"2024-04-14T15:11:00.763287+0000","last_fullsized":"2024-04-14T15:11:00.763287+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:09:42.004317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:03.751335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0058046169999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":63,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.211222+0000","last_change":"2024-04-14T15:10:43.898230+0000","last_active":"2024-04-14T15:11:01.211222+0000","last_peered":"2024-04-14T15:11:01.211222+0000","last_clean":"2024-04-14T15:11:01.211222+0000","last_became_active":"2024-04-14T15:09:43.025421+0000","last_became_peered":"2024-04-14T15:09:43.025421+0000","last_unstale":"2024-04-14T15:11:01.211222+0000","last_undegraded":"2024-04-14T15:11:01.211222+0000","last_fullsized":"2024-04-14T15:11:01.211222+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:43.898117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:10:43.898117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:47.080377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":304,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.937548+0000","last_change":"2024-04-14T15:10:06.160908+0000","last_active":"2024-04-14T15:11:00.937548+0000","last_peered":"2024-04-14T15:11:00.937548+0000","last_clean":"2024-04-14T15:11:00.937548+0000","last_became_active":"2024-04-14T15:09:33.753828+0000","last_became_peered":"2024-04-14T15:09:33.753828+0000","last_unstale":"2024-04-14T15:11:00.937548+0000","last_undegraded":"2024-04-14T15:11:00.937548+0000","last_fullsized":"2024-04-14T15:11:00.937548+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'117","last_scrub_stamp":"2024-04-14T15:10:06.160794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:07:24.600370+0000","last_clean_scrub_stamp":"2024-04-14T15:10:06.160794+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:17.821751+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"72'16","reported_seq":91,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:03.082811+0000","last_change":"2024-04-14T15:09:45.180279+0000","last_active":"2024-04-14T15:11:03.082811+0000","last_peered":"2024-04-14T15:11:03.082811+0000","last_clean":"2024-04-14T15:11:03.082811+0000","last_became_active":"2024-04-14T15:09:45.179978+0000","last_became_peered":"2024-04-14T15:09:45.179978+0000","last_unstale":"2024-04-14T15:11:03.082811+0000","last_undegraded":"2024-04-14T15:11:03.082811+0000","last_fullsized":"2024-04-14T15:11:03.082811+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:11.891483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":47,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.938495+0000","last_change":"2024-04-14T15:09:47.047523+0000","last_active":"2024-04-14T15:11:00.938495+0000","last_peered":"2024-04-14T15:11:00.938495+0000","last_clean":"2024-04-14T15:11:00.938495+0000","last_became_active":"2024-04-14T15:09:47.047201+0000","last_became_peered":"2024-04-14T15:09:47.047201+0000","last_unstale":"2024-04-14T15:11:00.938495+0000","last_undegraded":"2024-04-14T15:11:00.938495+0000","last_fullsized":"2024-04-14T15:11:00.938495+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:09:46.023663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:13.363585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"72'1","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.763434+0000","last_change":"2024-04-14T15:09:49.065706+0000","last_active":"2024-04-14T15:11:00.763434+0000","last_peered":"2024-04-14T15:11:00.763434+0000","last_clean":"2024-04-14T15:11:00.763434+0000","last_became_active":"2024-04-14T15:09:49.065243+0000","last_became_peered":"2024-04-14T15:09:49.065243+0000","last_unstale":"2024-04-14T15:11:00.763434+0000","last_undegraded":"2024-04-14T15:11:00.763434+0000","last_fullsized":"2024-04-14T15:11:00.763434+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":70,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:07.696587+0000","last_change":"2024-04-14T15:11:07.696609+0000","last_active":"2024-04-14T15:11:07.696587+0000","last_peered":"2024-04-14T15:11:07.696587+0000","last_clean":"2024-04-14T15:11:07.696587+0000","last_became_active":"2024-04-14T15:09:43.018509+0000","last_became_peered":"2024-04-14T15:09:43.018509+0000","last_unstale":"2024-04-14T15:11:07.696587+0000","last_undegraded":"2024-04-14T15:11:07.696587+0000","last_fullsized":"2024-04-14T15:11:07.696587+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:07.696525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:07.696525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:17.969989+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0018592680000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":63,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.937410+0000","last_change":"2024-04-14T15:10:49.916989+0000","last_active":"2024-04-14T15:11:00.937410+0000","last_peered":"2024-04-14T15:11:00.937410+0000","last_clean":"2024-04-14T15:11:00.937410+0000","last_became_active":"2024-04-14T15:09:43.025095+0000","last_became_peered":"2024-04-14T15:09:43.025095+0000","last_unstale":"2024-04-14T15:11:00.937410+0000","last_undegraded":"2024-04-14T15:11:00.937410+0000","last_fullsized":"2024-04-14T15:11:00.937410+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:49.916884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:10:49.916884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:11.267411+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"72'12","reported_seq":77,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.210892+0000","last_change":"2024-04-14T15:10:51.840577+0000","last_active":"2024-04-14T15:11:01.210892+0000","last_peered":"2024-04-14T15:11:01.210892+0000","last_clean":"2024-04-14T15:11:01.210892+0000","last_became_active":"2024-04-14T15:09:45.037231+0000","last_became_peered":"2024-04-14T15:09:45.037231+0000","last_unstale":"2024-04-14T15:11:01.210892+0000","last_undegraded":"2024-04-14T15:11:01.210892+0000","last_fullsized":"2024-04-14T15:11:01.210892+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'12","last_scrub_stamp":"2024-04-14T15:10:51.840487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:10:51.840487+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:52.131140+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":47,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.211771+0000","last_change":"2024-04-14T15:09:47.037005+0000","last_active":"2024-04-14T15:11:01.211771+0000","last_peered":"2024-04-14T15:11:01.211771+0000","last_clean":"2024-04-14T15:11:01.211771+0000","last_became_active":"2024-04-14T15:09:47.036696+0000","last_became_peered":"2024-04-14T15:09:47.036696+0000","last_unstale":"2024-04-14T15:11:01.211771+0000","last_undegraded":"2024-04-14T15:11:01.211771+0000","last_fullsized":"2024-04-14T15:11:01.211771+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:09:46.023663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:06.058177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.763009+0000","last_change":"2024-04-14T15:09:49.128490+0000","last_active":"2024-04-14T15:11:00.763009+0000","last_peered":"2024-04-14T15:11:00.763009+0000","last_clean":"2024-04-14T15:11:00.763009+0000","last_became_active":"2024-04-14T15:09:49.128136+0000","last_became_peered":"2024-04-14T15:09:49.128136+0000","last_unstale":"2024-04-14T15:11:00.763009+0000","last_undegraded":"2024-04-14T15:11:00.763009+0000","last_fullsized":"2024-04-14T15:11:00.763009+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:09.879878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":63,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.938596+0000","last_change":"2024-04-14T15:10:53.992688+0000","last_active":"2024-04-14T15:11:00.938596+0000","last_peered":"2024-04-14T15:11:00.938596+0000","last_clean":"2024-04-14T15:11:00.938596+0000","last_became_active":"2024-04-14T15:09:43.021006+0000","last_became_peered":"2024-04-14T15:09:43.021006+0000","last_unstale":"2024-04-14T15:11:00.938596+0000","last_undegraded":"2024-04-14T15:11:00.938596+0000","last_fullsized":"2024-04-14T15:11:00.938596+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:53.992590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:10:53.992590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:16.131673+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"72'19","reported_seq":95,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.210795+0000","last_change":"2024-04-14T15:10:47.880742+0000","last_active":"2024-04-14T15:11:01.210795+0000","last_peered":"2024-04-14T15:11:01.210795+0000","last_clean":"2024-04-14T15:11:01.210795+0000","last_became_active":"2024-04-14T15:09:45.179182+0000","last_became_peered":"2024-04-14T15:09:45.179182+0000","last_unstale":"2024-04-14T15:11:01.210795+0000","last_undegraded":"2024-04-14T15:11:01.210795+0000","last_fullsized":"2024-04-14T15:11:01.210795+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'19","last_scrub_stamp":"2024-04-14T15:10:47.880623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:10:47.880623+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:07.612241+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.940078+0000","last_change":"2024-04-14T15:10:59.286474+0000","last_active":"2024-04-14T15:11:00.940078+0000","last_peered":"2024-04-14T15:11:00.940078+0000","last_clean":"2024-04-14T15:11:00.940078+0000","last_became_active":"2024-04-14T15:09:47.048427+0000","last_became_peered":"2024-04-14T15:09:47.048427+0000","last_unstale":"2024-04-14T15:11:00.940078+0000","last_undegraded":"2024-04-14T15:11:00.940078+0000","last_fullsized":"2024-04-14T15:11:00.940078+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:59.286374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:10:59.286374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:59.951701+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764213+0000","last_change":"2024-04-14T15:09:49.065988+0000","last_active":"2024-04-14T15:11:00.764213+0000","last_peered":"2024-04-14T15:11:00.764213+0000","last_clean":"2024-04-14T15:11:00.764213+0000","last_became_active":"2024-04-14T15:09:49.065684+0000","last_became_peered":"2024-04-14T15:09:49.065684+0000","last_unstale":"2024-04-14T15:11:00.764213+0000","last_undegraded":"2024-04-14T15:11:00.764213+0000","last_fullsized":"2024-04-14T15:11:00.764213+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:07.917647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"65'1","reported_seq":56,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.763647+0000","last_change":"2024-04-14T15:09:43.019515+0000","last_active":"2024-04-14T15:11:00.763647+0000","last_peered":"2024-04-14T15:11:00.763647+0000","last_clean":"2024-04-14T15:11:00.763647+0000","last_became_active":"2024-04-14T15:09:43.018478+0000","last_became_peered":"2024-04-14T15:09:43.018478+0000","last_unstale":"2024-04-14T15:11:00.763647+0000","last_undegraded":"2024-04-14T15:11:00.763647+0000","last_fullsized":"2024-04-14T15:11:00.763647+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:09:42.004317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:10.202051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"72'18","reported_seq":91,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764001+0000","last_change":"2024-04-14T15:10:53.457240+0000","last_active":"2024-04-14T15:11:00.764001+0000","last_peered":"2024-04-14T15:11:00.764001+0000","last_clean":"2024-04-14T15:11:00.764001+0000","last_became_active":"2024-04-14T15:09:45.181252+0000","last_became_peered":"2024-04-14T15:09:45.181252+0000","last_unstale":"2024-04-14T15:11:00.764001+0000","last_undegraded":"2024-04-14T15:11:00.764001+0000","last_fullsized":"2024-04-14T15:11:00.764001+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'18","last_scrub_stamp":"2024-04-14T15:10:53.457087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:10:53.457087+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:00.641833+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":47,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764213+0000","last_change":"2024-04-14T15:09:47.044907+0000","last_active":"2024-04-14T15:11:00.764213+0000","last_peered":"2024-04-14T15:11:00.764213+0000","last_clean":"2024-04-14T15:11:00.764213+0000","last_became_active":"2024-04-14T15:09:47.044610+0000","last_became_peered":"2024-04-14T15:09:47.044610+0000","last_unstale":"2024-04-14T15:11:00.764213+0000","last_undegraded":"2024-04-14T15:11:00.764213+0000","last_fullsized":"2024-04-14T15:11:00.764213+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:09:46.023663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:05.510613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.938142+0000","last_change":"2024-04-14T15:09:49.130011+0000","last_active":"2024-04-14T15:11:00.938142+0000","last_peered":"2024-04-14T15:11:00.938142+0000","last_clean":"2024-04-14T15:11:00.938142+0000","last_became_active":"2024-04-14T15:09:49.129610+0000","last_became_peered":"2024-04-14T15:09:49.129610+0000","last_unstale":"2024-04-14T15:11:00.938142+0000","last_undegraded":"2024-04-14T15:11:00.938142+0000","last_fullsized":"2024-04-14T15:11:00.938142+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:13.440963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.939826+0000","last_change":"2024-04-14T15:09:43.020785+0000","last_active":"2024-04-14T15:11:00.939826+0000","last_peered":"2024-04-14T15:11:00.939826+0000","last_clean":"2024-04-14T15:11:00.939826+0000","last_became_active":"2024-04-14T15:09:43.020503+0000","last_became_peered":"2024-04-14T15:09:43.020503+0000","last_unstale":"2024-04-14T15:11:00.939826+0000","last_undegraded":"2024-04-14T15:11:00.939826+0000","last_fullsized":"2024-04-14T15:11:00.939826+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:09:42.004317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:09.830404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"72'14","reported_seq":72,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.763783+0000","last_change":"2024-04-14T15:09:45.026006+0000","last_active":"2024-04-14T15:11:00.763783+0000","last_peered":"2024-04-14T15:11:00.763783+0000","last_clean":"2024-04-14T15:11:00.763783+0000","last_became_active":"2024-04-14T15:09:45.025673+0000","last_became_peered":"2024-04-14T15:09:45.025673+0000","last_unstale":"2024-04-14T15:11:00.763783+0000","last_undegraded":"2024-04-14T15:11:00.763783+0000","last_fullsized":"2024-04-14T15:11:00.763783+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"72'8","reported_seq":73,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:03.082724+0000","last_change":"2024-04-14T15:09:47.046875+0000","last_active":"2024-04-14T15:11:03.082724+0000","last_peered":"2024-04-14T15:11:03.082724+0000","last_clean":"2024-04-14T15:11:03.082724+0000","last_became_active":"2024-04-14T15:09:47.046560+0000","last_became_peered":"2024-04-14T15:09:47.046560+0000","last_unstale":"2024-04-14T15:11:03.082724+0000","last_undegraded":"2024-04-14T15:11:03.082724+0000","last_fullsized":"2024-04-14T15:11:03.082724+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:09:46.023663+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:10.044512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":51,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.501853+0000","last_change":"2024-04-14T15:11:01.501878+0000","last_active":"2024-04-14T15:11:01.501853+0000","last_peered":"2024-04-14T15:11:01.501853+0000","last_clean":"2024-04-14T15:11:01.501853+0000","last_became_active":"2024-04-14T15:09:49.128028+0000","last_became_peered":"2024-04-14T15:09:49.128028+0000","last_unstale":"2024-04-14T15:11:01.501853+0000","last_undegraded":"2024-04-14T15:11:01.501853+0000","last_fullsized":"2024-04-14T15:11:01.501853+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:01.501784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:01.501784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:23.865314+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:02.763474+0000","last_change":"2024-04-14T15:11:02.763497+0000","last_active":"2024-04-14T15:11:02.763474+0000","last_peered":"2024-04-14T15:11:02.763474+0000","last_clean":"2024-04-14T15:11:02.763474+0000","last_became_active":"2024-04-14T15:09:47.043021+0000","last_became_peered":"2024-04-14T15:09:47.043021+0000","last_unstale":"2024-04-14T15:11:02.763474+0000","last_undegraded":"2024-04-14T15:11:02.763474+0000","last_fullsized":"2024-04-14T15:11:02.763474+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:02.763399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:02.763399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:06.127325+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":63,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.937694+0000","last_change":"2024-04-14T15:10:48.939920+0000","last_active":"2024-04-14T15:11:00.937694+0000","last_peered":"2024-04-14T15:11:00.937694+0000","last_clean":"2024-04-14T15:11:00.937694+0000","last_became_active":"2024-04-14T15:09:43.018601+0000","last_became_peered":"2024-04-14T15:09:43.018601+0000","last_unstale":"2024-04-14T15:11:00.937694+0000","last_undegraded":"2024-04-14T15:11:00.937694+0000","last_fullsized":"2024-04-14T15:11:00.937694+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:48.939790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:10:48.939790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:14.888606+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"85'16","reported_seq":83,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764593+0000","last_change":"2024-04-14T15:10:47.456301+0000","last_active":"2024-04-14T15:11:00.764593+0000","last_peered":"2024-04-14T15:11:00.764593+0000","last_clean":"2024-04-14T15:11:00.764593+0000","last_became_active":"2024-04-14T15:09:45.028212+0000","last_became_peered":"2024-04-14T15:09:45.028212+0000","last_unstale":"2024-04-14T15:11:00.764593+0000","last_undegraded":"2024-04-14T15:11:00.764593+0000","last_fullsized":"2024-04-14T15:11:00.764593+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-14T15:10:47.456169+0000","last_deep_scrub":"85'16","last_deep_scrub_stamp":"2024-04-14T15:10:47.456169+0000","last_clean_scrub_stamp":"2024-04-14T15:10:47.456169+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:53.736949+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.210097+0000","last_change":"2024-04-14T15:09:49.059351+0000","last_active":"2024-04-14T15:11:01.210097+0000","last_peered":"2024-04-14T15:11:01.210097+0000","last_clean":"2024-04-14T15:11:01.210097+0000","last_became_active":"2024-04-14T15:09:49.058940+0000","last_became_peered":"2024-04-14T15:09:49.058940+0000","last_unstale":"2024-04-14T15:11:01.210097+0000","last_undegraded":"2024-04-14T15:11:01.210097+0000","last_fullsized":"2024-04-14T15:11:01.210097+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:03.127308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"72'8","reported_seq":73,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:03.082922+0000","last_change":"2024-04-14T15:09:47.051276+0000","last_active":"2024-04-14T15:11:03.082922+0000","last_peered":"2024-04-14T15:11:03.082922+0000","last_clean":"2024-04-14T15:11:03.082922+0000","last_became_active":"2024-04-14T15:09:47.050962+0000","last_became_peered":"2024-04-14T15:09:47.050962+0000","last_unstale":"2024-04-14T15:11:03.082922+0000","last_undegraded":"2024-04-14T15:11:03.082922+0000","last_fullsized":"2024-04-14T15:11:03.082922+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:09:46.023663+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.763245+0000","last_change":"2024-04-14T15:09:43.021175+0000","last_active":"2024-04-14T15:11:00.763245+0000","last_peered":"2024-04-14T15:11:00.763245+0000","last_clean":"2024-04-14T15:11:00.763245+0000","last_became_active":"2024-04-14T15:09:43.020878+0000","last_became_peered":"2024-04-14T15:09:43.020878+0000","last_unstale":"2024-04-14T15:11:00.763245+0000","last_undegraded":"2024-04-14T15:11:00.763245+0000","last_fullsized":"2024-04-14T15:11:00.763245+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:09:42.004317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"72'15","reported_seq":76,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764979+0000","last_change":"2024-04-14T15:09:45.033879+0000","last_active":"2024-04-14T15:11:00.764979+0000","last_peered":"2024-04-14T15:11:00.764979+0000","last_clean":"2024-04-14T15:11:00.764979+0000","last_became_active":"2024-04-14T15:09:45.033331+0000","last_became_peered":"2024-04-14T15:09:45.033331+0000","last_unstale":"2024-04-14T15:11:00.764979+0000","last_undegraded":"2024-04-14T15:11:00.764979+0000","last_fullsized":"2024-04-14T15:11:00.764979+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764272+0000","last_change":"2024-04-14T15:09:49.065592+0000","last_active":"2024-04-14T15:11:00.764272+0000","last_peered":"2024-04-14T15:11:00.764272+0000","last_clean":"2024-04-14T15:11:00.764272+0000","last_became_active":"2024-04-14T15:09:49.065274+0000","last_became_peered":"2024-04-14T15:09:49.065274+0000","last_unstale":"2024-04-14T15:11:00.764272+0000","last_undegraded":"2024-04-14T15:11:00.764272+0000","last_fullsized":"2024-04-14T15:11:00.764272+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:11.500131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"72'8","reported_seq":73,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:03.082488+0000","last_change":"2024-04-14T15:09:47.044537+0000","last_active":"2024-04-14T15:11:03.082488+0000","last_peered":"2024-04-14T15:11:03.082488+0000","last_clean":"2024-04-14T15:11:03.082488+0000","last_became_active":"2024-04-14T15:09:47.044085+0000","last_became_peered":"2024-04-14T15:09:47.044085+0000","last_unstale":"2024-04-14T15:11:03.082488+0000","last_undegraded":"2024-04-14T15:11:03.082488+0000","last_fullsized":"2024-04-14T15:11:03.082488+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:09:46.023663+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:08.095291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":63,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764648+0000","last_change":"2024-04-14T15:10:46.280858+0000","last_active":"2024-04-14T15:11:00.764648+0000","last_peered":"2024-04-14T15:11:00.764648+0000","last_clean":"2024-04-14T15:11:00.764648+0000","last_became_active":"2024-04-14T15:09:43.018871+0000","last_became_peered":"2024-04-14T15:09:43.018871+0000","last_unstale":"2024-04-14T15:11:00.764648+0000","last_undegraded":"2024-04-14T15:11:00.764648+0000","last_fullsized":"2024-04-14T15:11:00.764648+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:46.280773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:10:46.280773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:59.607706+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"72'10","reported_seq":66,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764245+0000","last_change":"2024-04-14T15:09:45.029530+0000","last_active":"2024-04-14T15:11:00.764245+0000","last_peered":"2024-04-14T15:11:00.764245+0000","last_clean":"2024-04-14T15:11:00.764245+0000","last_became_active":"2024-04-14T15:09:45.029170+0000","last_became_peered":"2024-04-14T15:09:45.029170+0000","last_unstale":"2024-04-14T15:11:00.764245+0000","last_undegraded":"2024-04-14T15:11:00.764245+0000","last_fullsized":"2024-04-14T15:11:00.764245+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:08.246963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.210058+0000","last_change":"2024-04-14T15:09:49.069916+0000","last_active":"2024-04-14T15:11:01.210058+0000","last_peered":"2024-04-14T15:11:01.210058+0000","last_clean":"2024-04-14T15:11:01.210058+0000","last_became_active":"2024-04-14T15:09:49.069594+0000","last_became_peered":"2024-04-14T15:09:49.069594+0000","last_unstale":"2024-04-14T15:11:01.210058+0000","last_undegraded":"2024-04-14T15:11:01.210058+0000","last_fullsized":"2024-04-14T15:11:01.210058+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":47,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.763544+0000","last_change":"2024-04-14T15:09:47.043481+0000","last_active":"2024-04-14T15:11:00.763544+0000","last_peered":"2024-04-14T15:11:00.763544+0000","last_clean":"2024-04-14T15:11:00.763544+0000","last_became_active":"2024-04-14T15:09:47.042810+0000","last_became_peered":"2024-04-14T15:09:47.042810+0000","last_unstale":"2024-04-14T15:11:00.763544+0000","last_undegraded":"2024-04-14T15:11:00.763544+0000","last_fullsized":"2024-04-14T15:11:00.763544+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:09:46.023663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:12.979380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":63,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.903230+0000","last_change":"2024-04-14T15:11:01.903252+0000","last_active":"2024-04-14T15:11:01.903230+0000","last_peered":"2024-04-14T15:11:01.903230+0000","last_clean":"2024-04-14T15:11:01.903230+0000","last_became_active":"2024-04-14T15:09:43.022351+0000","last_became_peered":"2024-04-14T15:09:43.022351+0000","last_unstale":"2024-04-14T15:11:01.903230+0000","last_undegraded":"2024-04-14T15:11:01.903230+0000","last_fullsized":"2024-04-14T15:11:01.903230+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:01.903150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:01.903150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:27.833698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"72'17","reported_seq":103,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764650+0000","last_change":"2024-04-14T15:10:07.775002+0000","last_active":"2024-04-14T15:11:00.764650+0000","last_peered":"2024-04-14T15:11:00.764650+0000","last_clean":"2024-04-14T15:11:00.764650+0000","last_became_active":"2024-04-14T15:10:07.766008+0000","last_became_peered":"2024-04-14T15:10:07.766008+0000","last_unstale":"2024-04-14T15:11:00.764650+0000","last_undegraded":"2024-04-14T15:11:00.764650+0000","last_fullsized":"2024-04-14T15:11:00.764650+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.937345+0000","last_change":"2024-04-14T15:09:49.065881+0000","last_active":"2024-04-14T15:11:00.937345+0000","last_peered":"2024-04-14T15:11:00.937345+0000","last_clean":"2024-04-14T15:11:00.937345+0000","last_became_active":"2024-04-14T15:09:49.065397+0000","last_became_peered":"2024-04-14T15:09:49.065397+0000","last_unstale":"2024-04-14T15:11:00.937345+0000","last_undegraded":"2024-04-14T15:11:00.937345+0000","last_fullsized":"2024-04-14T15:11:00.937345+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:13.399339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.210496+0000","last_change":"2024-04-14T15:10:59.793878+0000","last_active":"2024-04-14T15:11:01.210496+0000","last_peered":"2024-04-14T15:11:01.210496+0000","last_clean":"2024-04-14T15:11:01.210496+0000","last_became_active":"2024-04-14T15:09:47.045542+0000","last_became_peered":"2024-04-14T15:09:47.045542+0000","last_unstale":"2024-04-14T15:11:01.210496+0000","last_undegraded":"2024-04-14T15:11:01.210496+0000","last_fullsized":"2024-04-14T15:11:01.210496+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:59.793768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:10:59.793768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:24.997192+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":63,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.210883+0000","last_change":"2024-04-14T15:10:45.837586+0000","last_active":"2024-04-14T15:11:01.210883+0000","last_peered":"2024-04-14T15:11:01.210883+0000","last_clean":"2024-04-14T15:11:01.210883+0000","last_became_active":"2024-04-14T15:09:43.020268+0000","last_became_peered":"2024-04-14T15:09:43.020268+0000","last_unstale":"2024-04-14T15:11:01.210883+0000","last_undegraded":"2024-04-14T15:11:01.210883+0000","last_fullsized":"2024-04-14T15:11:01.210883+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:45.837506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:10:45.837506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:04.829576+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"72'19","reported_seq":82,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.939978+0000","last_change":"2024-04-14T15:09:45.179087+0000","last_active":"2024-04-14T15:11:00.939978+0000","last_peered":"2024-04-14T15:11:00.939978+0000","last_clean":"2024-04-14T15:11:00.939978+0000","last_became_active":"2024-04-14T15:09:45.178782+0000","last_became_peered":"2024-04-14T15:09:45.178782+0000","last_unstale":"2024-04-14T15:11:00.939978+0000","last_undegraded":"2024-04-14T15:11:00.939978+0000","last_fullsized":"2024-04-14T15:11:00.939978+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":51,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.209629+0000","last_change":"2024-04-14T15:10:53.789872+0000","last_active":"2024-04-14T15:11:01.209629+0000","last_peered":"2024-04-14T15:11:01.209629+0000","last_clean":"2024-04-14T15:11:01.209629+0000","last_became_active":"2024-04-14T15:09:49.061495+0000","last_became_peered":"2024-04-14T15:09:49.061495+0000","last_unstale":"2024-04-14T15:11:01.209629+0000","last_undegraded":"2024-04-14T15:11:01.209629+0000","last_fullsized":"2024-04-14T15:11:01.209629+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:53.789790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:10:53.789790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:54.748155+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.210353+0000","last_change":"2024-04-14T15:10:50.788471+0000","last_active":"2024-04-14T15:11:01.210353+0000","last_peered":"2024-04-14T15:11:01.210353+0000","last_clean":"2024-04-14T15:11:01.210353+0000","last_became_active":"2024-04-14T15:09:47.046153+0000","last_became_peered":"2024-04-14T15:09:47.046153+0000","last_unstale":"2024-04-14T15:11:01.210353+0000","last_undegraded":"2024-04-14T15:11:01.210353+0000","last_fullsized":"2024-04-14T15:11:01.210353+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:50.788350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:10:50.788350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:52.620524+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":63,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:04.223660+0000","last_change":"2024-04-14T15:11:04.223680+0000","last_active":"2024-04-14T15:11:04.223660+0000","last_peered":"2024-04-14T15:11:04.223660+0000","last_clean":"2024-04-14T15:11:04.223660+0000","last_became_active":"2024-04-14T15:09:43.020044+0000","last_became_peered":"2024-04-14T15:09:43.020044+0000","last_unstale":"2024-04-14T15:11:04.223660+0000","last_undegraded":"2024-04-14T15:11:04.223660+0000","last_fullsized":"2024-04-14T15:11:04.223660+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:04.223605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:04.223605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:11.422287+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"72'9","reported_seq":75,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.211199+0000","last_change":"2024-04-14T15:10:50.850226+0000","last_active":"2024-04-14T15:11:01.211199+0000","last_peered":"2024-04-14T15:11:01.211199+0000","last_clean":"2024-04-14T15:11:01.211199+0000","last_became_active":"2024-04-14T15:09:45.037179+0000","last_became_peered":"2024-04-14T15:09:45.037179+0000","last_unstale":"2024-04-14T15:11:01.211199+0000","last_undegraded":"2024-04-14T15:11:01.211199+0000","last_fullsized":"2024-04-14T15:11:01.211199+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-04-14T15:10:50.850116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:10:50.850116+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:06.196819+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.210115+0000","last_change":"2024-04-14T15:09:49.128389+0000","last_active":"2024-04-14T15:11:01.210115+0000","last_peered":"2024-04-14T15:11:01.210115+0000","last_clean":"2024-04-14T15:11:01.210115+0000","last_became_active":"2024-04-14T15:09:49.128070+0000","last_became_peered":"2024-04-14T15:09:49.128070+0000","last_unstale":"2024-04-14T15:11:01.210115+0000","last_undegraded":"2024-04-14T15:11:01.210115+0000","last_fullsized":"2024-04-14T15:11:01.210115+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:09.430362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.210984+0000","last_change":"2024-04-14T15:09:43.021909+0000","last_active":"2024-04-14T15:11:01.210984+0000","last_peered":"2024-04-14T15:11:01.210984+0000","last_clean":"2024-04-14T15:11:01.210984+0000","last_became_active":"2024-04-14T15:09:43.021460+0000","last_became_peered":"2024-04-14T15:09:43.021460+0000","last_unstale":"2024-04-14T15:11:01.210984+0000","last_undegraded":"2024-04-14T15:11:01.210984+0000","last_fullsized":"2024-04-14T15:11:01.210984+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:09:42.004317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:02.402497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"72'15","reported_seq":76,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.210565+0000","last_change":"2024-04-14T15:09:45.180505+0000","last_active":"2024-04-14T15:11:01.210565+0000","last_peered":"2024-04-14T15:11:01.210565+0000","last_clean":"2024-04-14T15:11:01.210565+0000","last_became_active":"2024-04-14T15:09:45.180160+0000","last_became_peered":"2024-04-14T15:09:45.180160+0000","last_unstale":"2024-04-14T15:11:01.210565+0000","last_undegraded":"2024-04-14T15:11:01.210565+0000","last_fullsized":"2024-04-14T15:11:01.210565+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"85'9","reported_seq":82,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:03.082688+0000","last_change":"2024-04-14T15:10:51.950673+0000","last_active":"2024-04-14T15:11:03.082688+0000","last_peered":"2024-04-14T15:11:03.082688+0000","last_clean":"2024-04-14T15:11:03.082688+0000","last_became_active":"2024-04-14T15:09:47.045441+0000","last_became_peered":"2024-04-14T15:09:47.045441+0000","last_unstale":"2024-04-14T15:11:03.082688+0000","last_undegraded":"2024-04-14T15:11:03.082688+0000","last_fullsized":"2024-04-14T15:11:03.082688+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9","last_scrub_stamp":"2024-04-14T15:10:51.950580+0000","last_deep_scrub":"85'9","last_deep_scrub_stamp":"2024-04-14T15:10:51.950580+0000","last_clean_scrub_stamp":"2024-04-14T15:10:51.950580+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:09.843217+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":398,"ondisk_log_size":398,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":80,"num_read_kb":75,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103191,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28368,"kb_used_data":1352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716784,"statfs":{"total":95995035648,"available":95965986816,"internally_reserved":0,"allocated":1384448,"data_stored":937840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397530,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28364,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716788,"statfs":{"total":95995035648,"available":95965990912,"internally_reserved":0,"allocated":1380352,"data_stored":938133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574900,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27820,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717332,"statfs":{"total":95995035648,"available":95966547968,"internally_reserved":0,"allocated":823296,"data_stored":347886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738425,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32536,"kb_used_data":1360,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712616,"statfs":{"total":95995035648,"available":95961718784,"internally_reserved":0,"allocated":1392640,"data_stored":938493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411376,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31932,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713220,"statfs":{"total":95995035648,"available":95962337280,"internally_reserved":0,"allocated":774144,"data_stored":346987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182442,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27816,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717336,"statfs":{"total":95995035648,"available":95966552064,"internally_reserved":0,"allocated":819200,"data_stored":348207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986212,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27796,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717356,"statfs":{"total":95995035648,"available":95966572544,"internally_reserved":0,"allocated":798720,"data_stored":347368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659169,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31944,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713208,"statfs":{"total":95995035648,"available":95962324992,"internally_reserved":0,"allocated":782336,"data_stored":346822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-14T15:11:13.747 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-14T15:11:13.747 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-14T15:11:13.747 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-14T15:11:13.748 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph health --format=json 2024-04-14T15:11:13.990 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:11:14.056 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:13 smithi092 systemd[1]: Starting Ceph grafana.a for 0e288098-fa70-11ee-bc8e-c7b262605968... 2024-04-14T15:11:14.084 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:11:13 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:11:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:11:14.329 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-mon[33966]: 5.8 scrub starts 2024-04-14T15:11:14.329 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-mon[33966]: 5.8 scrub ok 2024-04-14T15:11:14.329 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-mon[33966]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:11:14.329 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 podman[67581]: 2024-04-14 15:11:13.95941651 +0000 UTC m=+0.018428589 image pull 2bacad6d85d802cd176c146528adcc0e47a8a83e94bc4a8b7ba1a9b9ef2c65e7 quay.io/ceph/grafana:9.4.12 2024-04-14T15:11:14.329 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 podman[67581]: 2024-04-14 15:11:14.068826543 +0000 UTC m=+0.127838607 container create 176d192f7fa174bfd5cd7ffe9d1b180411f2e8347142e95e140739b2470aff40 (image=quay.io/ceph/grafana:9.4.12, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-04-14T15:11:14.329 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 podman[67581]: 2024-04-14 15:11:14.329625628 +0000 UTC m=+0.388637692 container init 176d192f7fa174bfd5cd7ffe9d1b180411f2e8347142e95e140739b2470aff40 (image=quay.io/ceph/grafana:9.4.12, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-04-14T15:11:14.350 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:14 smithi090 ceph-mon[30399]: 5.8 scrub starts 2024-04-14T15:11:14.350 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:14 smithi090 ceph-mon[30399]: 5.8 scrub ok 2024-04-14T15:11:14.350 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:14 smithi090 ceph-mon[30399]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:11:14.350 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:14 smithi090 ceph-mon[41670]: 5.8 scrub starts 2024-04-14T15:11:14.350 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:14 smithi090 ceph-mon[41670]: 5.8 scrub ok 2024-04-14T15:11:14.350 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:14 smithi090 ceph-mon[41670]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:11:14.580 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 podman[67581]: 2024-04-14 15:11:14.335543134 +0000 UTC m=+0.394555199 container start 176d192f7fa174bfd5cd7ffe9d1b180411f2e8347142e95e140739b2470aff40 (image=quay.io/ceph/grafana:9.4.12, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-04-14T15:11:14.580 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 bash[67581]: 176d192f7fa174bfd5cd7ffe9d1b180411f2e8347142e95e140739b2470aff40 2024-04-14T15:11:14.580 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 systemd[1]: Started Ceph grafana.a for 0e288098-fa70-11ee-bc8e-c7b262605968. 2024-04-14T15:11:14.580 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=settings t=2024-04-14T15:11:14.487708722Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-04-14T15:11:14.580 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=settings t=2024-04-14T15:11:14.487868218Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-04-14T15:11:14.580 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=settings t=2024-04-14T15:11:14.487879004Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-04-14T15:11:14.580 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=settings t=2024-04-14T15:11:14.487882082Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-04-14T15:11:14.580 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=settings t=2024-04-14T15:11:14.487884819Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana" 2024-04-14T15:11:14.580 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=settings t=2024-04-14T15:11:14.487887403Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana" 2024-04-14T15:11:14.580 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=settings t=2024-04-14T15:11:14.487890277Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins" 2024-04-14T15:11:14.580 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=settings t=2024-04-14T15:11:14.487892729Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning" 2024-04-14T15:11:14.581 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=settings t=2024-04-14T15:11:14.487895574Z level=info msg="Config overridden from command line" arg="default.log.mode=console" 2024-04-14T15:11:14.581 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=settings t=2024-04-14T15:11:14.487898262Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-04-14T15:11:14.581 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=settings t=2024-04-14T15:11:14.487904659Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-04-14T15:11:14.581 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=settings t=2024-04-14T15:11:14.487907086Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-04-14T15:11:14.581 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=settings t=2024-04-14T15:11:14.487909917Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-04-14T15:11:14.581 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=settings t=2024-04-14T15:11:14.487912385Z level=info msg="Path Home" path=/usr/share/grafana 2024-04-14T15:11:14.581 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=settings t=2024-04-14T15:11:14.487914841Z level=info msg="Path Data" path=/var/lib/grafana 2024-04-14T15:11:14.582 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=settings t=2024-04-14T15:11:14.487917134Z level=info msg="Path Logs" path=/var/log/grafana 2024-04-14T15:11:14.582 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=settings t=2024-04-14T15:11:14.487919337Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-04-14T15:11:14.582 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=settings t=2024-04-14T15:11:14.487922053Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-04-14T15:11:14.582 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=settings t=2024-04-14T15:11:14.487924539Z level=info msg="App mode production" 2024-04-14T15:11:14.582 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=sqlstore t=2024-04-14T15:11:14.487980611Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-04-14T15:11:14.582 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=sqlstore t=2024-04-14T15:11:14.487990703Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-04-14T15:11:14.582 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.488821121Z level=info msg="Starting DB migrations" 2024-04-14T15:11:14.582 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.488878951Z level=info msg="Executing migration" id="create migration_log table" 2024-04-14T15:11:14.582 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.48946497Z level=info msg="Executing migration" id="create user table" 2024-04-14T15:11:14.583 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.489834962Z level=info msg="Executing migration" id="add unique index user.login" 2024-04-14T15:11:14.583 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.49039786Z level=info msg="Executing migration" id="add unique index user.email" 2024-04-14T15:11:14.583 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.490785423Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-04-14T15:11:14.583 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.491154476Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-04-14T15:11:14.583 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.491565842Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-04-14T15:11:14.583 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.492279794Z level=info msg="Executing migration" id="create user table v2" 2024-04-14T15:11:14.583 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.492683911Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-04-14T15:11:14.583 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.493520251Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-04-14T15:11:14.583 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.494102235Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-04-14T15:11:14.583 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.494436508Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-04-14T15:11:14.583 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.494793769Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-04-14T15:11:14.583 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.495198972Z level=info msg="Executing migration" id="Update user table charset" 2024-04-14T15:11:14.583 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.495484905Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-04-14T15:11:14.583 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.495857976Z level=info msg="Executing migration" id="Add missing user data" 2024-04-14T15:11:14.583 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.49617063Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-04-14T15:11:14.584 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.496587967Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-04-14T15:11:14.584 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.496945216Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-04-14T15:11:14.584 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.497371142Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-04-14T15:11:14.584 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.4987223Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-04-14T15:11:14.584 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.499117998Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-04-14T15:11:14.584 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.499518152Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-04-14T15:11:14.584 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.499902231Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-04-14T15:11:14.584 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.500840126Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-04-14T15:11:14.584 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.501449051Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-04-14T15:11:14.584 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.502367744Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-04-14T15:11:14.584 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.503012708Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-04-14T15:11:14.584 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.503938974Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-04-14T15:11:14.584 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.504586867Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-04-14T15:11:14.584 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.505403475Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-04-14T15:11:14.584 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.50619969Z level=info msg="Executing migration" id="create temp_user v2" 2024-04-14T15:11:14.584 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.506992188Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-04-14T15:11:14.585 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.507787479Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-04-14T15:11:14.585 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.508172429Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-04-14T15:11:14.585 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.508983303Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-04-14T15:11:14.585 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.509820939Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-04-14T15:11:14.585 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.510541495Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-04-14T15:11:14.585 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.510880071Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-04-14T15:11:14.585 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.511411861Z level=info msg="Executing migration" id="create star table" 2024-04-14T15:11:14.585 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.511780594Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-04-14T15:11:14.585 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.512250216Z level=info msg="Executing migration" id="create org table v1" 2024-04-14T15:11:14.585 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.513020755Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-04-14T15:11:14.585 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.513917832Z level=info msg="Executing migration" id="create org_user table v1" 2024-04-14T15:11:14.585 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.514741127Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-04-14T15:11:14.585 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.515389308Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-04-14T15:11:14.585 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.51833909Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-04-14T15:11:14.585 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.518755914Z level=info msg="Executing migration" id="Update org table charset" 2024-04-14T15:11:14.586 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.521342951Z level=info msg="Executing migration" id="Update org_user table charset" 2024-04-14T15:11:14.586 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.521651266Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-04-14T15:11:14.586 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.52407811Z level=info msg="Executing migration" id="create dashboard table" 2024-04-14T15:11:14.586 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.524626474Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-04-14T15:11:14.586 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.526858846Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-04-14T15:11:14.586 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.529504234Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-04-14T15:11:14.586 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.53242759Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-04-14T15:11:14.586 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.535049088Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-04-14T15:11:14.586 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.537763386Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-04-14T15:11:14.586 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.540517273Z level=info msg="Executing migration" id="create dashboard v2" 2024-04-14T15:11:14.586 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.543149287Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-04-14T15:11:14.586 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.545970196Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-04-14T15:11:14.587 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.548833843Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-04-14T15:11:14.587 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.551797366Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-04-14T15:11:14.587 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.554533605Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-04-14T15:11:14.587 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.557407806Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-04-14T15:11:14.587 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.559968888Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-04-14T15:11:14.587 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.562514501Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-04-14T15:11:14.587 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.565216811Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-04-14T15:11:14.587 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.567580344Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-04-14T15:11:14.587 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.570045265Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-04-14T15:11:14.587 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.572674509Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-04-14T15:11:14.587 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.575229333Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-04-14T15:11:14.587 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.577766749Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-04-14T15:11:14.587 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.580346173Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-04-14T15:11:14.587 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.580890403Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-04-14T15:11:14.830 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:11:14.830 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.582707078Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-04-14T15:11:14.831 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.585344443Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-04-14T15:11:14.831 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.587798484Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-04-14T15:11:14.831 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.590379097Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-04-14T15:11:14.831 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.593216414Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-04-14T15:11:14.831 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.595630268Z level=info msg="Executing migration" id="Update dashboard title length" 2024-04-14T15:11:14.831 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.598070375Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-04-14T15:11:14.831 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.600776032Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-04-14T15:11:14.831 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.603501661Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-04-14T15:11:14.831 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.604761941Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-04-14T15:11:14.831 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.605565477Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-04-14T15:11:14.831 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.606331466Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-04-14T15:11:14.831 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.607229292Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-04-14T15:11:14.831 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.607801698Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-04-14T15:11:14.831 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.608574677Z level=info msg="Executing migration" id="Add check_sum column" 2024-04-14T15:11:14.832 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.609093383Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-04-14T15:11:14.832 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.609838269Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-04-14T15:11:14.832 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.610535802Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-04-14T15:11:14.832 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.611419786Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-04-14T15:11:14.832 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.612110239Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-04-14T15:11:14.832 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.612905303Z level=info msg="Executing migration" id="create data_source table" 2024-04-14T15:11:14.833 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.613624747Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-04-14T15:11:14.833 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.614376512Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-04-14T15:11:14.833 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.614774115Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-04-14T15:11:14.833 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.615464836Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-04-14T15:11:14.833 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.615849615Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-04-14T15:11:14.833 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.617046651Z level=info msg="Executing migration" id="create data_source table v2" 2024-04-14T15:11:14.833 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.617837984Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-04-14T15:11:14.833 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.618641257Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-04-14T15:11:14.833 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.619428199Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-04-14T15:11:14.833 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.620324844Z level=info msg="Executing migration" id="Add column with_credentials" 2024-04-14T15:11:14.833 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.62106411Z level=info msg="Executing migration" id="Add secure json data column" 2024-04-14T15:11:14.833 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.621900799Z level=info msg="Executing migration" id="Update data_source table charset" 2024-04-14T15:11:14.833 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.62220332Z level=info msg="Executing migration" id="Update initial version to 1" 2024-04-14T15:11:14.833 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.62249884Z level=info msg="Executing migration" id="Add read_only data column" 2024-04-14T15:11:14.833 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.623011525Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-04-14T15:11:14.834 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.62329308Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-04-14T15:11:14.834 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.623591858Z level=info msg="Executing migration" id="Add uid column" 2024-04-14T15:11:14.834 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.624106075Z level=info msg="Executing migration" id="Update uid value" 2024-04-14T15:11:14.834 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.624424328Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-04-14T15:11:14.834 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.624945926Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-04-14T15:11:14.834 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.625764674Z level=info msg="Executing migration" id="create api_key table" 2024-04-14T15:11:14.834 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.626560807Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-04-14T15:11:14.834 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.627313297Z level=info msg="Executing migration" id="add index api_key.key" 2024-04-14T15:11:14.834 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.628152727Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-04-14T15:11:14.834 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.629095333Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-04-14T15:11:14.834 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.629696824Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-04-14T15:11:14.834 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.630048646Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-04-14T15:11:14.834 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.630448945Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-04-14T15:11:14.834 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.631705664Z level=info msg="Executing migration" id="create api_key table v2" 2024-04-14T15:11:14.835 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.632092191Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-04-14T15:11:14.835 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.632495168Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-04-14T15:11:14.835 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.632882462Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-04-14T15:11:14.835 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.633276191Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-04-14T15:11:14.835 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.633583865Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-04-14T15:11:14.835 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.633905466Z level=info msg="Executing migration" id="Update api_key table charset" 2024-04-14T15:11:14.835 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.634146433Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-04-14T15:11:14.835 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.634754193Z level=info msg="Executing migration" id="Add service account foreign key" 2024-04-14T15:11:14.835 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.635283561Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-04-14T15:11:14.835 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.635581725Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-04-14T15:11:14.835 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.637481726Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-04-14T15:11:14.835 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.638251911Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-04-14T15:11:14.835 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.640290209Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-04-14T15:11:14.835 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.643049263Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-04-14T15:11:14.836 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.645649128Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-04-14T15:11:14.836 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.646251797Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-04-14T15:11:14.836 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.647141781Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-04-14T15:11:14.836 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.648128653Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-04-14T15:11:14.836 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.648855843Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-04-14T15:11:14.836 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.649102018Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-04-14T15:11:14.836 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.64973483Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-04-14T15:11:14.836 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.650306208Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-04-14T15:11:14.836 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.651123697Z level=info msg="Executing migration" id="create quota table v1" 2024-04-14T15:11:14.836 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.651938926Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-04-14T15:11:14.836 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.652376322Z level=info msg="Executing migration" id="Update quota table charset" 2024-04-14T15:11:14.836 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.65335099Z level=info msg="Executing migration" id="create plugin_setting table" 2024-04-14T15:11:14.837 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.654217325Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-04-14T15:11:14.837 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.654893653Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-04-14T15:11:14.837 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.655580339Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-04-14T15:11:14.838 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.656435986Z level=info msg="Executing migration" id="create session table" 2024-04-14T15:11:14.838 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.657301263Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-04-14T15:11:14.838 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.658108124Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-04-14T15:11:14.838 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.658414919Z level=info msg="Executing migration" id="create playlist table v2" 2024-04-14T15:11:14.838 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.659276111Z level=info msg="Executing migration" id="create playlist item table v2" 2024-04-14T15:11:14.838 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.660095657Z level=info msg="Executing migration" id="Update playlist table charset" 2024-04-14T15:11:14.838 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.661032623Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-04-14T15:11:14.838 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.661303207Z level=info msg="Executing migration" id="drop preferences table v2" 2024-04-14T15:11:14.838 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.66229222Z level=info msg="Executing migration" id="drop preferences table v3" 2024-04-14T15:11:14.838 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.663018513Z level=info msg="Executing migration" id="create preferences table v3" 2024-04-14T15:11:14.838 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.663989627Z level=info msg="Executing migration" id="Update preferences table charset" 2024-04-14T15:11:14.838 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.66437901Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-04-14T15:11:14.838 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.665257804Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-04-14T15:11:14.838 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.666265414Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-04-14T15:11:14.839 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.666963925Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-04-14T15:11:14.839 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.667597862Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-04-14T15:11:14.839 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.668579463Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-04-14T15:11:14.839 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.669316452Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-04-14T15:11:14.839 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.670439135Z level=info msg="Executing migration" id="create alert table v1" 2024-04-14T15:11:14.839 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.670962568Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-04-14T15:11:14.839 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.671855733Z level=info msg="Executing migration" id="add index alert state" 2024-04-14T15:11:14.839 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.672811367Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-04-14T15:11:14.839 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.67552939Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-04-14T15:11:14.839 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.675917954Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-04-14T15:11:14.839 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.678809551Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-04-14T15:11:14.839 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.681583565Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-04-14T15:11:14.840 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.684336926Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-04-14T15:11:14.840 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.684708932Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-04-14T15:11:14.840 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.687438091Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-04-14T15:11:14.840 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.690305119Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-04-14T15:11:14.840 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.693110642Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-04-14T15:11:14.840 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.693748607Z level=info msg="Executing migration" id="Add column is_default" 2024-04-14T15:11:14.840 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.696659768Z level=info msg="Executing migration" id="Add column frequency" 2024-04-14T15:11:14.840 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.699529199Z level=info msg="Executing migration" id="Add column send_reminder" 2024-04-14T15:11:14.840 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.702310002Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-04-14T15:11:14.840 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.703044287Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-04-14T15:11:14.840 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.705956634Z level=info msg="Executing migration" id="Update alert table charset" 2024-04-14T15:11:14.840 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.708791807Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-04-14T15:11:14.840 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.711480683Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-04-14T15:11:14.840 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.711908737Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-04-14T15:11:14.841 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.714615327Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-04-14T15:11:14.841 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.717366668Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-04-14T15:11:14.841 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.720152218Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-04-14T15:11:14.841 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.720588862Z level=info msg="Executing migration" id="Add for to alert table" 2024-04-14T15:11:14.841 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.723373244Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-04-14T15:11:14.841 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.72611724Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-04-14T15:11:14.841 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.728764745Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-04-14T15:11:14.841 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.729232428Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-04-14T15:11:14.841 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.732127585Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-04-14T15:11:14.841 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.734811468Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-04-14T15:11:14.841 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.737576742Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-04-14T15:11:14.841 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.738074973Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-04-14T15:11:14.841 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.740878038Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-04-14T15:11:14.841 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.743544773Z level=info msg="Executing migration" id="create annotation table v5" 2024-04-14T15:11:14.842 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.746341311Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-04-14T15:11:14.842 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.746849757Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-04-14T15:11:14.842 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.749609035Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-04-14T15:11:14.842 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.750563521Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-04-14T15:11:14.842 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.751298394Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-04-14T15:11:14.842 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.751760128Z level=info msg="Executing migration" id="Update annotation table charset" 2024-04-14T15:11:14.842 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.752545166Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-04-14T15:11:14.842 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.75331879Z level=info msg="Executing migration" id="Drop category_id index" 2024-04-14T15:11:14.842 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.753785024Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-04-14T15:11:14.842 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.754561776Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-04-14T15:11:14.842 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.754955465Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-04-14T15:11:14.842 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.755786385Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-04-14T15:11:14.842 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.7566264Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-04-14T15:11:14.842 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.758591147Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-04-14T15:11:14.843 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.758973525Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-04-14T15:11:14.843 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.759957056Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-04-14T15:11:14.843 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.760721177Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-04-14T15:11:14.843 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.761719107Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-04-14T15:11:14.843 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.762142659Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-04-14T15:11:14.843 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.76320062Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-04-14T15:11:14.843 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.764152556Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-04-14T15:11:14.843 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.764944829Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-04-14T15:11:14.843 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.765387577Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-04-14T15:11:14.843 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.766294363Z level=info msg="Executing migration" id="Add epoch_end column" 2024-04-14T15:11:14.843 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.767136728Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-04-14T15:11:14.843 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.768084525Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-04-14T15:11:14.843 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.76858513Z level=info msg="Executing migration" id="Move region to single row" 2024-04-14T15:11:14.843 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.769398Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-04-14T15:11:14.843 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.769881287Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-04-14T15:11:14.844 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.770373909Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-04-14T15:11:14.844 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.771396694Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-04-14T15:11:14.844 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.772079625Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-04-14T15:11:14.844 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.772930254Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-04-14T15:11:14.844 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.773942241Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-04-14T15:11:14.844 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.77472429Z level=info msg="Executing migration" id="create test_data table" 2024-04-14T15:11:14.844 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.775123702Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-04-14T15:11:14.844 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.775991149Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-04-14T15:11:14.844 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.776863643Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-04-14T15:11:14.844 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.779684979Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-04-14T15:11:14.844 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.780037404Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-04-14T15:11:14.844 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.782951276Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-04-14T15:11:14.844 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.785796208Z level=info msg="Executing migration" id="create team table" 2024-04-14T15:11:14.844 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.788652862Z level=info msg="Executing migration" id="add index team.org_id" 2024-04-14T15:11:14.844 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.789162726Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-04-14T15:11:14.845 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.791821056Z level=info msg="Executing migration" id="create team member table" 2024-04-14T15:11:14.845 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.794696023Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-04-14T15:11:14.845 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.797433311Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-04-14T15:11:14.845 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.798045842Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-04-14T15:11:14.845 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.801129696Z level=info msg="Executing migration" id="Add column email to team table" 2024-04-14T15:11:14.845 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.803853253Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-04-14T15:11:14.845 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.806885344Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-04-14T15:11:14.845 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.807814439Z level=info msg="Executing migration" id="create dashboard acl table" 2024-04-14T15:11:14.845 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.810614356Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-04-14T15:11:14.845 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.813457354Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-04-14T15:11:14.845 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.816291269Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-04-14T15:11:14.845 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.816839757Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-04-14T15:11:14.845 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.819596941Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-04-14T15:11:14.845 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.822018252Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-04-14T15:11:14.846 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.82460426Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-04-14T15:11:14.846 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.825038049Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-04-14T15:11:14.846 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.825438738Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-04-14T15:11:14.846 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.825734155Z level=info msg="Executing migration" id="create tag table" 2024-04-14T15:11:14.846 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.826119952Z level=info msg="Executing migration" id="add index tag.key_value" 2024-04-14T15:11:14.846 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.826557376Z level=info msg="Executing migration" id="create login attempt table" 2024-04-14T15:11:14.847 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.826931982Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-04-14T15:11:14.847 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.82737554Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-04-14T15:11:14.847 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.827802812Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-04-14T15:11:14.847 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.830142401Z level=info msg="Executing migration" id="create login_attempt v2" 2024-04-14T15:11:14.847 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.830548328Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-04-14T15:11:14.847 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.831008562Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-04-14T15:11:14.847 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.831338892Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-04-14T15:11:14.847 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.831704067Z level=info msg="Executing migration" id="create user auth table" 2024-04-14T15:11:14.847 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.832062388Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-04-14T15:11:14.847 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.832503761Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-04-14T15:11:14.847 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.832791874Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-04-14T15:11:14.847 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.833627979Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-04-14T15:11:14.847 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.834471808Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-04-14T15:11:14.847 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.835319037Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-04-14T15:11:14.848 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.836178874Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-04-14T15:11:14.848 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.836628185Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-04-14T15:11:14.848 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.837493666Z level=info msg="Executing migration" id="create server_lock table" 2024-04-14T15:11:14.848 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.837879515Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-04-14T15:11:14.848 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.838304519Z level=info msg="Executing migration" id="create user auth token table" 2024-04-14T15:11:14.848 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.838732942Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-04-14T15:11:15.081 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.841166547Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-04-14T15:11:15.081 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.841611214Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-04-14T15:11:15.081 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.84204508Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-04-14T15:11:15.081 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.842940514Z level=info msg="Executing migration" id="create cache_data table" 2024-04-14T15:11:15.081 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.843353536Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-04-14T15:11:15.082 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.843793205Z level=info msg="Executing migration" id="create short_url table v1" 2024-04-14T15:11:15.082 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.844226318Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-04-14T15:11:15.082 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.84469361Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-04-14T15:11:15.082 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.844961083Z level=info msg="Executing migration" id="delete alert_definition table" 2024-04-14T15:11:15.082 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.845241984Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-04-14T15:11:15.082 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.846939645Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-04-14T15:11:15.082 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.849496753Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-04-14T15:11:15.082 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.85201198Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-04-14T15:11:15.082 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.852338773Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-04-14T15:11:15.082 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.854493884Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-04-14T15:11:15.082 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.856995547Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-04-14T15:11:15.082 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.8597487Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-04-14T15:11:15.082 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.86214256Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-04-14T15:11:15.084 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.864632749Z level=info msg="Executing migration" id="drop alert_definition table" 2024-04-14T15:11:15.084 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.867120627Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-04-14T15:11:15.084 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.869689688Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-04-14T15:11:15.084 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.872314706Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-04-14T15:11:15.084 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.874668135Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-04-14T15:11:15.084 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.876998381Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-04-14T15:11:15.084 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.879762369Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-04-14T15:11:15.084 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.88233228Z level=info msg="Executing migration" id="create alert_instance table" 2024-04-14T15:11:15.084 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.883101514Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-04-14T15:11:15.084 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.883911267Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-04-14T15:11:15.084 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.884731064Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-04-14T15:11:15.084 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.885619802Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-04-14T15:11:15.085 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.88644933Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-04-14T15:11:15.085 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.887266446Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-04-14T15:11:15.085 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.89172038Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-04-14T15:11:15.085 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.896072882Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-04-14T15:11:15.085 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.896989185Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-04-14T15:11:15.085 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.897670243Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-04-14T15:11:15.085 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.898637306Z level=info msg="Executing migration" id="create alert_rule table" 2024-04-14T15:11:15.085 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.899052409Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-04-14T15:11:15.085 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.89994075Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-04-14T15:11:15.085 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.900632948Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-04-14T15:11:15.085 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.901474459Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-04-14T15:11:15.085 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.902353641Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-04-14T15:11:15.085 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.903401359Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-04-14T15:11:15.085 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.904303957Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-04-14T15:11:15.085 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.905220928Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-04-14T15:11:15.086 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.90598966Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-04-14T15:11:15.086 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.906909408Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-04-14T15:11:15.086 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.908005644Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-04-14T15:11:15.086 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.908909301Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-04-14T15:11:15.086 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.909819188Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-04-14T15:11:15.086 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.910875507Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-04-14T15:11:15.086 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.911858941Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-04-14T15:11:15.086 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.912790137Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-04-14T15:11:15.086 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.913411293Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-04-14T15:11:15.086 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.914344794Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-04-14T15:11:15.086 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.914939627Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-04-14T15:11:15.086 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.91520405Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-04-14T15:11:15.086 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.916163763Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-04-14T15:11:15.086 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.917107416Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-04-14T15:11:15.087 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.918049735Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-04-14T15:11:15.087 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.91899639Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-04-14T15:11:15.087 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.919951983Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-04-14T15:11:15.087 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.920879568Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-04-14T15:11:15.087 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.923750719Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-04-14T15:11:15.087 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.926434736Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-04-14T15:11:15.087 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.929402303Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-04-14T15:11:15.087 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.932235845Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-04-14T15:11:15.087 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.934636371Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-04-14T15:11:15.087 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.937532456Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-04-14T15:11:15.087 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.940232346Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-04-14T15:11:15.087 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.942916151Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-04-14T15:11:15.087 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.945450324Z level=info msg="Executing migration" id="create provenance_type table" 2024-04-14T15:11:15.087 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.948240601Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-04-14T15:11:15.088 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.951070507Z level=info msg="Executing migration" id="create alert_image table" 2024-04-14T15:11:15.088 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.95386956Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-04-14T15:11:15.088 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.956572387Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-04-14T15:11:15.088 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.959550097Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-04-14T15:11:15.088 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.962094418Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-04-14T15:11:15.088 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.962523477Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-04-14T15:11:15.088 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.962594669Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-04-14T15:11:15.088 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.965313434Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-04-14T15:11:15.088 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.968305333Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-04-14T15:11:15.088 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.971063433Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-04-14T15:11:15.088 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.97131612Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-04-14T15:11:15.088 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.971432826Z level=info msg="alerts found to migrate" alerts=0 2024-04-14T15:11:15.088 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.97350671Z level=info msg="Executing migration" id="create library_element table v1" 2024-04-14T15:11:15.088 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.976218825Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-04-14T15:11:15.089 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.978909737Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-04-14T15:11:15.089 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.981325945Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-04-14T15:11:15.089 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.983918473Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-04-14T15:11:15.089 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.986351686Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-04-14T15:11:15.089 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.988909479Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-04-14T15:11:15.089 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.991309701Z level=info msg="Executing migration" id="create data_keys table" 2024-04-14T15:11:15.089 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.99369161Z level=info msg="Executing migration" id="create secrets table" 2024-04-14T15:11:15.089 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:14.99624073Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-04-14T15:11:15.089 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.00150071Z level=info msg="Executing migration" id="add name column into data_keys" 2024-04-14T15:11:15.089 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.004096781Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-04-14T15:11:15.089 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.006524465Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-04-14T15:11:15.089 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.01213233Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-04-14T15:11:15.089 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.017535984Z level=info msg="Executing migration" id="create kv_store table v1" 2024-04-14T15:11:15.089 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.020171319Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-04-14T15:11:15.090 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.020642764Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-04-14T15:11:15.090 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.020933726Z level=info msg="Executing migration" id="create permission table" 2024-04-14T15:11:15.090 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.02130888Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-04-14T15:11:15.090 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.021758295Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-04-14T15:11:15.090 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.022179886Z level=info msg="Executing migration" id="create role table" 2024-04-14T15:11:15.090 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.022571701Z level=info msg="Executing migration" id="add column display_name" 2024-04-14T15:11:15.090 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.023649242Z level=info msg="Executing migration" id="add column group_name" 2024-04-14T15:11:15.090 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.024732165Z level=info msg="Executing migration" id="add index role.org_id" 2024-04-14T15:11:15.090 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.025152093Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-04-14T15:11:15.090 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.026505796Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-04-14T15:11:15.090 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.02932192Z level=info msg="Executing migration" id="create team role table" 2024-04-14T15:11:15.090 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.031993135Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-04-14T15:11:15.090 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.034898025Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-04-14T15:11:15.090 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.037697477Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-04-14T15:11:15.091 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.040514486Z level=info msg="Executing migration" id="create user role table" 2024-04-14T15:11:15.091 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.042938959Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-04-14T15:11:15.091 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.045497141Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-04-14T15:11:15.091 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.048155072Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-04-14T15:11:15.091 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.050855363Z level=info msg="Executing migration" id="create builtin role table" 2024-04-14T15:11:15.091 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.053463761Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-04-14T15:11:15.091 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.056247635Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-04-14T15:11:15.091 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.058883371Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-04-14T15:11:15.091 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.06159785Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-04-14T15:11:15.091 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.064341743Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-04-14T15:11:15.091 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.067077749Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-04-14T15:11:15.091 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.06990265Z level=info msg="Executing migration" id="add unique index role.uid" 2024-04-14T15:11:15.091 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.072525969Z level=info msg="Executing migration" id="create seed assignment table" 2024-04-14T15:11:15.091 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.075336928Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-04-14T15:11:15.091 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.075883163Z level=info msg="Executing migration" id="add column hidden to role table" 2024-04-14T15:11:15.092 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.078625353Z level=info msg="Executing migration" id="create query_history table v1" 2024-04-14T15:11:15.092 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.08140863Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-04-14T15:11:15.092 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.082109492Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-04-14T15:11:15.092 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.082412257Z level=info msg="Executing migration" id="teams permissions migration" 2024-04-14T15:11:15.092 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.083238564Z level=info msg="Executing migration" id="dashboard permissions" 2024-04-14T15:11:15.092 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.084070957Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-04-14T15:11:15.092 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.084911755Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-04-14T15:11:15.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-mon[33966]: 2.e scrub starts 2024-04-14T15:11:15.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-mon[33966]: 2.e scrub ok 2024-04-14T15:11:15.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-mon[33966]: 5.4 scrub starts 2024-04-14T15:11:15.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-mon[33966]: 5.4 scrub ok 2024-04-14T15:11:15.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:15.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:15.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:15.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:15.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-mon[33966]: Deploying daemon node-exporter.a on smithi090 2024-04-14T15:11:15.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:11:15.373 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.085282263Z level=info msg="Executing migration" id="alerting notification permissions" 2024-04-14T15:11:15.374 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.086063807Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-04-14T15:11:15.374 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.086851913Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-04-14T15:11:15.374 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.087735471Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-04-14T15:11:15.374 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.089426682Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-04-14T15:11:15.374 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.090339427Z level=info msg="Executing migration" id="create correlation table v1" 2024-04-14T15:11:15.374 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.091160746Z level=info msg="Executing migration" id="add index correlations.uid" 2024-04-14T15:11:15.374 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.091938453Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-04-14T15:11:15.374 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.092950859Z level=info msg="Executing migration" id="add correlation config column" 2024-04-14T15:11:15.374 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.094132974Z level=info msg="Executing migration" id="create entity_events table" 2024-04-14T15:11:15.374 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.094915978Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-04-14T15:11:15.374 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.095864799Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-04-14T15:11:15.374 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.095992254Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-04-14T15:11:15.375 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.096573684Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-04-14T15:11:15.375 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.096671882Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-04-14T15:11:15.375 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.096938748Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-04-14T15:11:15.375 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.097845298Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-04-14T15:11:15.375 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.098808218Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-04-14T15:11:15.375 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.099588803Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-04-14T15:11:15.375 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.100050541Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-04-14T15:11:15.375 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.100895469Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-04-14T15:11:15.375 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.101790892Z level=info msg="Executing migration" id="Drop public config table" 2024-04-14T15:11:15.375 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.102759206Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-04-14T15:11:15.375 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.103242026Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-04-14T15:11:15.375 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.104096292Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-04-14T15:11:15.375 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.105087165Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-04-14T15:11:15.375 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.105908173Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-04-14T15:11:15.376 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.109950884Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-04-14T15:11:15.376 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.111154417Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-04-14T15:11:15.376 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.11233871Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-04-14T15:11:15.376 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.113326529Z level=info msg="Executing migration" id="add share column" 2024-04-14T15:11:15.376 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.11470919Z level=info msg="Executing migration" id="create default alerting folders" 2024-04-14T15:11:15.376 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.117525879Z level=info msg="Executing migration" id="create file table" 2024-04-14T15:11:15.376 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.120397949Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-04-14T15:11:15.376 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.123216784Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-04-14T15:11:15.376 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.123724258Z level=info msg="Executing migration" id="create file_meta table" 2024-04-14T15:11:15.376 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.1267243Z level=info msg="Executing migration" id="file table idx: path key" 2024-04-14T15:11:15.376 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.12962344Z level=info msg="Executing migration" id="set path collation in file table" 2024-04-14T15:11:15.376 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.132284511Z level=info msg="Executing migration" id="managed permissions migration" 2024-04-14T15:11:15.376 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.132826657Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-04-14T15:11:15.376 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.135528036Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-04-14T15:11:15.377 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.135990394Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-04-14T15:11:15.377 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.138389408Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-04-14T15:11:15.377 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.141404339Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-04-14T15:11:15.377 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.144257206Z level=info msg="Executing migration" id="update group index for alert rules" 2024-04-14T15:11:15.377 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.147067185Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-04-14T15:11:15.377 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.149899555Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-04-14T15:11:15.377 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.152813306Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-04-14T15:11:15.377 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.154119704Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-04-14T15:11:15.377 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.15700678Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-04-14T15:11:15.377 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.159600636Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-04-14T15:11:15.377 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.172421048Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-04-14T15:11:15.377 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.172877606Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-04-14T15:11:15.377 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.175711856Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-04-14T15:11:15.377 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.180079936Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-04-14T15:11:15.378 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.18293066Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-04-14T15:11:15.378 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.183325936Z level=info msg="Executing migration" id="create folder table" 2024-04-14T15:11:15.378 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.186097703Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-04-14T15:11:15.378 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.188770413Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-04-14T15:11:15.378 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.191428704Z level=info msg="Executing migration" id="Update folder title length" 2024-04-14T15:11:15.378 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.191727899Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-04-14T15:11:15.378 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=migrator t=2024-04-14T15:11:15.194524498Z level=info msg="migrations completed" performed=485 skipped=0 duration=705.662703ms 2024-04-14T15:11:15.378 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=sqlstore t=2024-04-14T15:11:15.194956409Z level=info msg="Created default organization" 2024-04-14T15:11:15.378 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=plugin.finder t=2024-04-14T15:11:15.230732084Z level=warn msg="Skipping finding plugins as directory does not exist" path=/usr/share/grafana/plugins-bundled 2024-04-14T15:11:15.378 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=secrets t=2024-04-14T15:11:15.230903031Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-04-14T15:11:15.378 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=query_data t=2024-04-14T15:11:15.234127781Z level=info msg="Query Service initialization" 2024-04-14T15:11:15.378 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=live.push_http t=2024-04-14T15:11:15.236780586Z level=info msg="Live Push Gateway initialization" 2024-04-14T15:11:15.379 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:15 smithi090 ceph-mon[41670]: 2.e scrub starts 2024-04-14T15:11:15.379 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:15 smithi090 ceph-mon[41670]: 2.e scrub ok 2024-04-14T15:11:15.379 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:15 smithi090 ceph-mon[41670]: 5.4 scrub starts 2024-04-14T15:11:15.379 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:15 smithi090 ceph-mon[41670]: 5.4 scrub ok 2024-04-14T15:11:15.379 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:15 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:15.379 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:15 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:15.379 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:15 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:15.379 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:15 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:15.379 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:15 smithi090 ceph-mon[41670]: Deploying daemon node-exporter.a on smithi090 2024-04-14T15:11:15.379 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:15 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:11:15.379 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:15 smithi090 ceph-mon[30399]: 2.e scrub starts 2024-04-14T15:11:15.380 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:15 smithi090 ceph-mon[30399]: 2.e scrub ok 2024-04-14T15:11:15.380 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:15 smithi090 ceph-mon[30399]: 5.4 scrub starts 2024-04-14T15:11:15.380 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:15 smithi090 ceph-mon[30399]: 5.4 scrub ok 2024-04-14T15:11:15.380 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:15 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:15.380 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:15 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:15.380 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:15 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:15.380 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:15 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:15.380 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:15 smithi090 ceph-mon[30399]: Deploying daemon node-exporter.a on smithi090 2024-04-14T15:11:15.380 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:15 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:11:15.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.656+0000 7f58cc260700 1 -- 172.21.15.90:0/4118010931 >> v1:172.21.15.90:6789/0 conn(0x7f58c410f2f0 legacy=0x7f58c41117b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:15.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.656+0000 7f58cc260700 1 -- 172.21.15.90:0/4118010931 shutdown_connections 2024-04-14T15:11:15.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.656+0000 7f58cc260700 1 -- 172.21.15.90:0/4118010931 >> 172.21.15.90:0/4118010931 conn(0x7f58c40fe3a0 msgr2=0x7f58c41007c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:11:15.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.657+0000 7f58cc260700 1 -- 172.21.15.90:0/4118010931 shutdown_connections 2024-04-14T15:11:15.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.657+0000 7f58cc260700 1 -- 172.21.15.90:0/4118010931 wait complete. 2024-04-14T15:11:15.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.657+0000 7f58cc260700 1 librados: starting msgr at 2024-04-14T15:11:15.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.657+0000 7f58cc260700 1 librados: starting objecter 2024-04-14T15:11:15.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.657+0000 7f58cc260700 1 Processor -- start 2024-04-14T15:11:15.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.657+0000 7f58cc260700 1 -- start start 2024-04-14T15:11:15.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.657+0000 7f58cc260700 1 librados: setting wanted keys 2024-04-14T15:11:15.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.657+0000 7f58cc260700 1 librados: calling monclient init 2024-04-14T15:11:15.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.657+0000 7f58cc260700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f58c41a7a70 con 0x7f58c410b720 2024-04-14T15:11:15.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.657+0000 7f58cc260700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f58c41a8c90 con 0x7f58c4108870 2024-04-14T15:11:15.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.657+0000 7f58cc260700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f58c41a9ed0 con 0x7f58c410f2f0 2024-04-14T15:11:15.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.657+0000 7f58ca7fd700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f58c410f2f0 0x7f58c41a6130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:52610/0 (socket says 172.21.15.90:52610) 2024-04-14T15:11:15.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.657+0000 7f58ca7fd700 1 -- 172.21.15.90:0/116195567 learned_addr learned my addr 172.21.15.90:0/116195567 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:11:15.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.658+0000 7f58baffd700 10 client.?.objecter ms_handle_connect 0x7f58c410b720 2024-04-14T15:11:15.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.658+0000 7f58baffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:11:15.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.658+0000 7f58baffd700 10 client.?.objecter ms_handle_connect 0x7f58c4108870 2024-04-14T15:11:15.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.658+0000 7f58baffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:11:15.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.658+0000 7f58baffd700 10 client.?.objecter ms_handle_connect 0x7f58c410f2f0 2024-04-14T15:11:15.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.658+0000 7f58baffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:11:15.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.658+0000 7f58baffd700 1 -- 172.21.15.90:0/116195567 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 172959875 0 0) 0x7f58c41a7a70 con 0x7f58c410b720 2024-04-14T15:11:15.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.658+0000 7f58baffd700 1 -- 172.21.15.90:0/116195567 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f58b00031f0 con 0x7f58c410b720 2024-04-14T15:11:15.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.658+0000 7f58baffd700 1 -- 172.21.15.90:0/116195567 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3694840094 0 0) 0x7f58c41a8c90 con 0x7f58c4108870 2024-04-14T15:11:15.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.658+0000 7f58baffd700 1 -- 172.21.15.90:0/116195567 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f58c41a7a70 con 0x7f58c4108870 2024-04-14T15:11:15.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.658+0000 7f58baffd700 1 -- 172.21.15.90:0/116195567 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1584034703 0 0) 0x7f58c41a9ed0 con 0x7f58c410f2f0 2024-04-14T15:11:15.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.658+0000 7f58baffd700 1 -- 172.21.15.90:0/116195567 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f58c41a8c90 con 0x7f58c410f2f0 2024-04-14T15:11:15.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.658+0000 7f58baffd700 1 -- 172.21.15.90:0/116195567 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1441901149 0 0) 0x7f58b00031f0 con 0x7f58c410b720 2024-04-14T15:11:15.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.658+0000 7f58baffd700 1 -- 172.21.15.90:0/116195567 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f58c41a9ed0 con 0x7f58c410b720 2024-04-14T15:11:15.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.658+0000 7f58baffd700 1 -- 172.21.15.90:0/116195567 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f58b4001870 con 0x7f58c410b720 2024-04-14T15:11:15.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.658+0000 7f58baffd700 1 -- 172.21.15.90:0/116195567 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3631308361 0 0) 0x7f58c41a8c90 con 0x7f58c410f2f0 2024-04-14T15:11:15.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.658+0000 7f58baffd700 1 -- 172.21.15.90:0/116195567 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f58b00031f0 con 0x7f58c410f2f0 2024-04-14T15:11:15.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.658+0000 7f58baffd700 1 -- 172.21.15.90:0/116195567 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f58bc001bc0 con 0x7f58c410f2f0 2024-04-14T15:11:15.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.658+0000 7f58baffd700 1 -- 172.21.15.90:0/116195567 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3155393400 0 0) 0x7f58c41a7a70 con 0x7f58c4108870 2024-04-14T15:11:15.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.658+0000 7f58baffd700 1 -- 172.21.15.90:0/116195567 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f58c41a8c90 con 0x7f58c4108870 2024-04-14T15:11:15.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.658+0000 7f58baffd700 1 -- 172.21.15.90:0/116195567 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f58c0001870 con 0x7f58c4108870 2024-04-14T15:11:15.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.658+0000 7f58baffd700 1 -- 172.21.15.90:0/116195567 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3702775266 0 0) 0x7f58c41a9ed0 con 0x7f58c410b720 2024-04-14T15:11:15.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.658+0000 7f58baffd700 1 -- 172.21.15.90:0/116195567 >> v1:172.21.15.90:6790/0 conn(0x7f58c410f2f0 legacy=0x7f58c41a6130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:15.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.658+0000 7f58baffd700 1 -- 172.21.15.90:0/116195567 >> v1:172.21.15.92:6789/0 conn(0x7f58c4108870 legacy=0x7f58c410eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:15.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.658+0000 7f58baffd700 1 -- 172.21.15.90:0/116195567 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58c41ab110 con 0x7f58c410b720 2024-04-14T15:11:15.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.658+0000 7f58baffd700 1 -- 172.21.15.90:0/116195567 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f58b4003b20 con 0x7f58c410b720 2024-04-14T15:11:15.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.658+0000 7f58cc260700 1 -- 172.21.15.90:0/116195567 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f58c41a8ec0 con 0x7f58c410b720 2024-04-14T15:11:15.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.658+0000 7f58baffd700 1 -- 172.21.15.90:0/116195567 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f58b4004ae0 con 0x7f58c410b720 2024-04-14T15:11:15.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.659+0000 7f58cc260700 10 client.14766.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:11:15.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.659+0000 7f58cc260700 1 -- 172.21.15.90:0/116195567 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f58c41a9380 con 0x7f58c410b720 2024-04-14T15:11:15.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.659+0000 7f58cc260700 1 librados: init done 2024-04-14T15:11:15.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.660+0000 7f58cc260700 1 -- 172.21.15.90:0/116195567 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58c4103fb0 con 0x7f58c410b720 2024-04-14T15:11:15.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.660+0000 7f58baffd700 1 -- 172.21.15.90:0/116195567 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f58b4005e90 con 0x7f58c410b720 2024-04-14T15:11:15.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.661+0000 7f58baffd700 1 -- 172.21.15.90:0/116195567 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(87..87 src has 12..87) ==== 6288+0+0 (unknown 1749939968 0 0) 0x7f58b40891b0 con 0x7f58c410b720 2024-04-14T15:11:15.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.661+0000 7f58baffd700 10 client.14766.objecter ms_dispatch 0x7f58c4001550 osd_map(87..87 src has 12..87) 2024-04-14T15:11:15.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.661+0000 7f58baffd700 3 client.14766.objecter handle_osd_map got epochs [87,87] > 0 2024-04-14T15:11:15.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.661+0000 7f58baffd700 3 client.14766.objecter handle_osd_map decoding full epoch 87 2024-04-14T15:11:15.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.661+0000 7f58baffd700 20 client.14766.objecter dump_active .. 0 homeless 2024-04-14T15:11:15.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.661+0000 7f58baffd700 10 client.14766.objecter ms_handle_connect 0x7f58b0071460 2024-04-14T15:11:15.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.665+0000 7f58baffd700 1 -- 172.21.15.90:0/116195567 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f58b4056380 con 0x7f58c410b720 2024-04-14T15:11:15.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.936+0000 7f58cc260700 1 -- 172.21.15.90:0/116195567 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f58c410d1d0 con 0x7f58c410b720 2024-04-14T15:11:15.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.936+0000 7f58baffd700 1 -- 172.21.15.90:0/116195567 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (unknown 3730786198 0 4185958460) 0x7f58b405a030 con 0x7f58c410b720 2024-04-14T15:11:15.937 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:11:15.937 INFO:teuthology.orchestra.run.smithi090.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-14T15:11:15.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.938+0000 7f58cc260700 10 librados: watch_flush enter 2024-04-14T15:11:15.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.938+0000 7f58cc260700 10 librados: watch_flush exit 2024-04-14T15:11:15.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.938+0000 7f58cc260700 20 client.14766.objecter shutdown clearing up homeless session... 2024-04-14T15:11:15.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.938+0000 7f58cc260700 10 client.14766.objecter successfully canceled tick 2024-04-14T15:11:15.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.939+0000 7f58cc260700 1 -- 172.21.15.90:0/116195567 >> v1:172.21.15.90:6800/420935427 conn(0x7f58b0071460 legacy=0x7f58b0073920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:15.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.939+0000 7f58cc260700 1 -- 172.21.15.90:0/116195567 >> v1:172.21.15.90:6789/0 conn(0x7f58c410b720 legacy=0x7f58c41a27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:15.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.939+0000 7f58cc260700 1 -- 172.21.15.90:0/116195567 shutdown_connections 2024-04-14T15:11:15.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.939+0000 7f58cc260700 1 -- 172.21.15.90:0/116195567 >> 172.21.15.90:0/116195567 conn(0x7f58c40fe3a0 msgr2=0x7f58c40ffe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:11:15.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.939+0000 7f58cc260700 1 -- 172.21.15.90:0/116195567 shutdown_connections 2024-04-14T15:11:15.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.939+0000 7f58cc260700 1 -- 172.21.15.90:0/116195567 wait complete. 2024-04-14T15:11:15.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:15.939+0000 7f58cc260700 1 librados: shutdown 2024-04-14T15:11:16.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:16 smithi092 ceph-mon[33966]: 4.1d scrub starts 2024-04-14T15:11:16.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:16 smithi092 ceph-mon[33966]: 4.1d scrub ok 2024-04-14T15:11:16.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:16 smithi092 ceph-mon[33966]: 4.1 scrub starts 2024-04-14T15:11:16.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:16 smithi092 ceph-mon[33966]: 4.1 scrub ok 2024-04-14T15:11:16.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:16 smithi092 ceph-mon[33966]: 5.3 scrub starts 2024-04-14T15:11:16.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:16 smithi092 ceph-mon[33966]: 5.3 scrub ok 2024-04-14T15:11:16.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:16 smithi092 ceph-mon[33966]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:11:16.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:16 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/116195567' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-14T15:11:16.123 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=infra.usagestats.collector t=2024-04-14T15:11:15.839784903Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-04-14T15:11:16.123 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=provisioning.datasources t=2024-04-14T15:11:15.84071243Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-04-14T15:11:16.123 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=provisioning.datasources t=2024-04-14T15:11:15.851842774Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-04-14T15:11:16.123 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=provisioning.alerting t=2024-04-14T15:11:15.862526199Z level=info msg="starting to provision alerting" 2024-04-14T15:11:16.123 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=provisioning.alerting t=2024-04-14T15:11:15.862538541Z level=info msg="finished to provision alerting" 2024-04-14T15:11:16.123 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=ngalert.state.manager t=2024-04-14T15:11:15.862623127Z level=info msg="Warming state cache for startup" 2024-04-14T15:11:16.123 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=ngalert.state.manager t=2024-04-14T15:11:15.862817251Z level=info msg="State cache has been initialized" states=0 duration=192.175µs 2024-04-14T15:11:16.123 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=ticker t=2024-04-14T15:11:15.862895612Z level=info msg=starting first_tick=2024-04-14T15:11:20Z 2024-04-14T15:11:16.124 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=grafanaStorageLogger t=2024-04-14T15:11:15.862922155Z level=info msg="storage starting" 2024-04-14T15:11:16.124 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=ngalert.multiorg.alertmanager t=2024-04-14T15:11:15.863034307Z level=info msg="starting MultiOrg Alertmanager" 2024-04-14T15:11:16.124 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=http.server t=2024-04-14T15:11:15.86966219Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-04-14T15:11:16.124 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=sqlstore.transactions t=2024-04-14T15:11:15.879994469Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-04-14T15:11:16.124 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=sqlstore.transactions t=2024-04-14T15:11:15.890567067Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-04-14T15:11:16.124 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:11:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=sqlstore.transactions t=2024-04-14T15:11:15.900815555Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=2 code="database is locked" 2024-04-14T15:11:16.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:16 smithi090 ceph-mon[30399]: 4.1d scrub starts 2024-04-14T15:11:16.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:16 smithi090 ceph-mon[30399]: 4.1d scrub ok 2024-04-14T15:11:16.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:16 smithi090 ceph-mon[30399]: 4.1 scrub starts 2024-04-14T15:11:16.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:16 smithi090 ceph-mon[30399]: 4.1 scrub ok 2024-04-14T15:11:16.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:16 smithi090 ceph-mon[30399]: 5.3 scrub starts 2024-04-14T15:11:16.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:16 smithi090 ceph-mon[30399]: 5.3 scrub ok 2024-04-14T15:11:16.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:16 smithi090 ceph-mon[30399]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:11:16.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:16 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/116195567' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-14T15:11:16.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:16 smithi090 ceph-mon[41670]: 4.1d scrub starts 2024-04-14T15:11:16.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:16 smithi090 ceph-mon[41670]: 4.1d scrub ok 2024-04-14T15:11:16.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:16 smithi090 ceph-mon[41670]: 4.1 scrub starts 2024-04-14T15:11:16.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:16 smithi090 ceph-mon[41670]: 4.1 scrub ok 2024-04-14T15:11:16.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:16 smithi090 ceph-mon[41670]: 5.3 scrub starts 2024-04-14T15:11:16.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:16 smithi090 ceph-mon[41670]: 5.3 scrub ok 2024-04-14T15:11:16.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:16 smithi090 ceph-mon[41670]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:11:16.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:16 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/116195567' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-14T15:11:16.662 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-14T15:11:16.662 INFO:tasks.cephadm:Setup complete, yielding 2024-04-14T15:11:16.662 INFO:teuthology.run_tasks:Running task thrashosds... 2024-04-14T15:11:16.677 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-04-14T15:11:16.678 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms bind msgr2': False, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mgrc': 20, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd class default list': '*', 'osd class load list': '*', 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'crush_tunables': 'jewel', 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'reached quota', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)'], 'mon_bind_msgr2': False, 'sha1': 'a971fd5f5719038c70b817bafacd6d5400b1d888'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'cephadm': {'cephadm_mode': 'root'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'a971fd5f5719038c70b817bafacd6d5400b1d888'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'main', 'sha1': '05c9d46e7de5db2cb434c177a6bbfc56f472d09b'}} 2024-04-14T15:11:16.678 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-04-14T15:11:16.678 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-04-14T15:11:16.937 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:11:16.941 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:11:16 smithi090 bash[79996]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-04-14T15:11:17.348 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:17 smithi090 ceph-mon[30399]: 6.17 scrub starts 2024-04-14T15:11:17.349 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:17 smithi090 ceph-mon[30399]: 6.17 scrub ok 2024-04-14T15:11:17.349 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:17 smithi090 ceph-mon[30399]: 5.5 scrub starts 2024-04-14T15:11:17.349 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:17 smithi090 ceph-mon[30399]: 5.5 scrub ok 2024-04-14T15:11:17.349 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:17 smithi090 ceph-mon[30399]: 6.8 scrub starts 2024-04-14T15:11:17.349 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:17 smithi090 ceph-mon[30399]: 6.8 scrub ok 2024-04-14T15:11:17.349 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:17 smithi090 ceph-mon[30399]: 5.1d scrub starts 2024-04-14T15:11:17.349 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:17 smithi090 ceph-mon[30399]: 5.1d scrub ok 2024-04-14T15:11:17.349 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:17 smithi090 ceph-mon[41670]: 6.17 scrub starts 2024-04-14T15:11:17.350 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:17 smithi090 ceph-mon[41670]: 6.17 scrub ok 2024-04-14T15:11:17.350 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:17 smithi090 ceph-mon[41670]: 5.5 scrub starts 2024-04-14T15:11:17.350 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:17 smithi090 ceph-mon[41670]: 5.5 scrub ok 2024-04-14T15:11:17.350 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:17 smithi090 ceph-mon[41670]: 6.8 scrub starts 2024-04-14T15:11:17.350 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:17 smithi090 ceph-mon[41670]: 6.8 scrub ok 2024-04-14T15:11:17.350 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:17 smithi090 ceph-mon[41670]: 5.1d scrub starts 2024-04-14T15:11:17.350 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:17 smithi090 ceph-mon[41670]: 5.1d scrub ok 2024-04-14T15:11:17.350 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:11:17 smithi090 bash[79996]: Getting image source signatures 2024-04-14T15:11:17.350 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:11:17 smithi090 bash[79996]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-04-14T15:11:17.350 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:11:17 smithi090 bash[79996]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-04-14T15:11:17.351 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:11:17 smithi090 bash[79996]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-04-14T15:11:17.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:17 smithi092 ceph-mon[33966]: 6.17 scrub starts 2024-04-14T15:11:17.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:17 smithi092 ceph-mon[33966]: 6.17 scrub ok 2024-04-14T15:11:17.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:17 smithi092 ceph-mon[33966]: 5.5 scrub starts 2024-04-14T15:11:17.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:17 smithi092 ceph-mon[33966]: 5.5 scrub ok 2024-04-14T15:11:17.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:17 smithi092 ceph-mon[33966]: 6.8 scrub starts 2024-04-14T15:11:17.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:17 smithi092 ceph-mon[33966]: 6.8 scrub ok 2024-04-14T15:11:17.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:17 smithi092 ceph-mon[33966]: 5.1d scrub starts 2024-04-14T15:11:17.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:17 smithi092 ceph-mon[33966]: 5.1d scrub ok 2024-04-14T15:11:18.353 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:18 smithi090 ceph-mon[41670]: 6.3 scrub starts 2024-04-14T15:11:18.353 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:18 smithi090 ceph-mon[41670]: 6.3 scrub ok 2024-04-14T15:11:18.353 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:18 smithi090 ceph-mon[41670]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:11:18.353 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:18 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:18.353 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:18 smithi090 ceph-mon[30399]: 6.3 scrub starts 2024-04-14T15:11:18.353 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:18 smithi090 ceph-mon[30399]: 6.3 scrub ok 2024-04-14T15:11:18.353 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:18 smithi090 ceph-mon[30399]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:11:18.354 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:18 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:18.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:18 smithi092 ceph-mon[33966]: 6.3 scrub starts 2024-04-14T15:11:18.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:18 smithi092 ceph-mon[33966]: 6.3 scrub ok 2024-04-14T15:11:18.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:18 smithi092 ceph-mon[33966]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:11:18.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:18 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:19.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:19 smithi090 ceph-mon[41670]: 6.13 scrub starts 2024-04-14T15:11:19.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:19 smithi090 ceph-mon[41670]: 6.13 scrub ok 2024-04-14T15:11:19.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:19 smithi090 ceph-mon[41670]: 4.1a scrub starts 2024-04-14T15:11:19.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:19 smithi090 ceph-mon[41670]: 4.1a scrub ok 2024-04-14T15:11:19.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:19 smithi090 ceph-mon[41670]: 4.2 scrub starts 2024-04-14T15:11:19.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:19 smithi090 ceph-mon[41670]: 4.2 scrub ok 2024-04-14T15:11:19.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:19 smithi090 ceph-mon[41670]: 4.1e scrub starts 2024-04-14T15:11:19.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:19 smithi090 ceph-mon[41670]: 4.1e scrub ok 2024-04-14T15:11:19.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:19 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:11:19.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:19 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:11:19.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:19 smithi090 ceph-mon[41670]: 2.11 scrub starts 2024-04-14T15:11:19.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:19 smithi090 ceph-mon[41670]: 2.11 scrub ok 2024-04-14T15:11:19.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:19 smithi090 ceph-mon[41670]: 5.15 scrub starts 2024-04-14T15:11:19.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:19 smithi090 ceph-mon[41670]: 5.15 scrub ok 2024-04-14T15:11:19.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:19 smithi090 ceph-mon[30399]: 6.13 scrub starts 2024-04-14T15:11:19.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:19 smithi090 ceph-mon[30399]: 6.13 scrub ok 2024-04-14T15:11:19.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:19 smithi090 ceph-mon[30399]: 4.1a scrub starts 2024-04-14T15:11:19.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:19 smithi090 ceph-mon[30399]: 4.1a scrub ok 2024-04-14T15:11:19.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:19 smithi090 ceph-mon[30399]: 4.2 scrub starts 2024-04-14T15:11:19.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:19 smithi090 ceph-mon[30399]: 4.2 scrub ok 2024-04-14T15:11:19.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:19 smithi090 ceph-mon[30399]: 4.1e scrub starts 2024-04-14T15:11:19.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:19 smithi090 ceph-mon[30399]: 4.1e scrub ok 2024-04-14T15:11:19.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:19 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:11:19.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:19 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:11:19.196 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:19 smithi090 ceph-mon[30399]: 2.11 scrub starts 2024-04-14T15:11:19.196 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:19 smithi090 ceph-mon[30399]: 2.11 scrub ok 2024-04-14T15:11:19.196 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:19 smithi090 ceph-mon[30399]: 5.15 scrub starts 2024-04-14T15:11:19.196 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:19 smithi090 ceph-mon[30399]: 5.15 scrub ok 2024-04-14T15:11:19.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:19 smithi092 ceph-mon[33966]: 6.13 scrub starts 2024-04-14T15:11:19.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:19 smithi092 ceph-mon[33966]: 6.13 scrub ok 2024-04-14T15:11:19.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:19 smithi092 ceph-mon[33966]: 4.1a scrub starts 2024-04-14T15:11:19.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:19 smithi092 ceph-mon[33966]: 4.1a scrub ok 2024-04-14T15:11:19.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:19 smithi092 ceph-mon[33966]: 4.2 scrub starts 2024-04-14T15:11:19.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:19 smithi092 ceph-mon[33966]: 4.2 scrub ok 2024-04-14T15:11:19.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:19 smithi092 ceph-mon[33966]: 4.1e scrub starts 2024-04-14T15:11:19.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:19 smithi092 ceph-mon[33966]: 4.1e scrub ok 2024-04-14T15:11:19.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:19 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:11:19.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:19 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:11:19.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:19 smithi092 ceph-mon[33966]: 2.11 scrub starts 2024-04-14T15:11:19.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:19 smithi092 ceph-mon[33966]: 2.11 scrub ok 2024-04-14T15:11:19.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:19 smithi092 ceph-mon[33966]: 5.15 scrub starts 2024-04-14T15:11:19.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:19 smithi092 ceph-mon[33966]: 5.15 scrub ok 2024-04-14T15:11:19.519 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:11:19 smithi090 bash[79996]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-04-14T15:11:19.520 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:11:19 smithi090 bash[79996]: Writing manifest to image destination 2024-04-14T15:11:19.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:19.829+0000 7f0268905700 1 -- 172.21.15.90:0/934033636 >> v1:172.21.15.90:6789/0 conn(0x7f0264107760 legacy=0x7f0264107b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:19.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:19.829+0000 7f0268905700 1 -- 172.21.15.90:0/934033636 shutdown_connections 2024-04-14T15:11:19.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:19.829+0000 7f0268905700 1 -- 172.21.15.90:0/934033636 >> 172.21.15.90:0/934033636 conn(0x7f02640fd290 msgr2=0x7f02640ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:11:19.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:19.829+0000 7f0268905700 1 -- 172.21.15.90:0/934033636 shutdown_connections 2024-04-14T15:11:19.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:19.829+0000 7f0268905700 1 -- 172.21.15.90:0/934033636 wait complete. 2024-04-14T15:11:19.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:19.830+0000 7f0268905700 1 librados: starting msgr at 2024-04-14T15:11:19.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:19.830+0000 7f0268905700 1 librados: starting objecter 2024-04-14T15:11:19.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:19.830+0000 7f0268905700 1 Processor -- start 2024-04-14T15:11:19.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:19.830+0000 7f0268905700 1 -- start start 2024-04-14T15:11:19.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:19.830+0000 7f0268905700 1 librados: setting wanted keys 2024-04-14T15:11:19.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:19.830+0000 7f0268905700 1 librados: calling monclient init 2024-04-14T15:11:19.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:19.830+0000 7f0268905700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f026410dbf0 con 0x7f026410a610 2024-04-14T15:11:19.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:19.830+0000 7f0268905700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f026410ddc0 con 0x7f026410e1e0 2024-04-14T15:11:19.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:19.830+0000 7f0268905700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f026410df90 con 0x7f0264107760 2024-04-14T15:11:19.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:19.830+0000 7f0261d9b700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f026410a610 0x7f026410b100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:39556/0 (socket says 172.21.15.90:39556) 2024-04-14T15:11:19.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:19.830+0000 7f0261d9b700 1 -- 172.21.15.90:0/438005962 learned_addr learned my addr 172.21.15.90:0/438005962 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:11:19.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:19.831+0000 7f025b7fe700 10 client.?.objecter ms_handle_connect 0x7f0264107760 2024-04-14T15:11:19.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:19.831+0000 7f025b7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:11:19.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:19.831+0000 7f025b7fe700 10 client.?.objecter ms_handle_connect 0x7f026410a610 2024-04-14T15:11:19.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:19.831+0000 7f025b7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:11:19.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:19.831+0000 7f025b7fe700 1 -- 172.21.15.90:0/438005962 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3914555663 0 0) 0x7f026410dbf0 con 0x7f026410a610 2024-04-14T15:11:19.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:19.831+0000 7f025b7fe700 1 -- 172.21.15.90:0/438005962 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f02500031f0 con 0x7f026410a610 2024-04-14T15:11:19.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:19.831+0000 7f025b7fe700 10 client.?.objecter ms_handle_connect 0x7f026410e1e0 2024-04-14T15:11:19.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:19.831+0000 7f025b7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:11:19.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:19.831+0000 7f025b7fe700 1 -- 172.21.15.90:0/438005962 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 642216976 0 0) 0x7f026410df90 con 0x7f0264107760 2024-04-14T15:11:19.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:19.831+0000 7f025b7fe700 1 -- 172.21.15.90:0/438005962 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f026410dbf0 con 0x7f0264107760 2024-04-14T15:11:19.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:19.831+0000 7f025b7fe700 1 -- 172.21.15.90:0/438005962 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3098025698 0 0) 0x7f026410ddc0 con 0x7f026410e1e0 2024-04-14T15:11:19.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:19.831+0000 7f025b7fe700 1 -- 172.21.15.90:0/438005962 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f026410df90 con 0x7f026410e1e0 2024-04-14T15:11:19.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:19.831+0000 7f025b7fe700 1 -- 172.21.15.90:0/438005962 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2200812395 0 0) 0x7f02500031f0 con 0x7f026410a610 2024-04-14T15:11:19.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:19.831+0000 7f025b7fe700 1 -- 172.21.15.90:0/438005962 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f026410ddc0 con 0x7f026410a610 2024-04-14T15:11:19.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:19.831+0000 7f025b7fe700 1 -- 172.21.15.90:0/438005962 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f0254001870 con 0x7f026410a610 2024-04-14T15:11:19.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:19.831+0000 7f025b7fe700 1 -- 172.21.15.90:0/438005962 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3986475263 0 0) 0x7f026410df90 con 0x7f026410e1e0 2024-04-14T15:11:19.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:19.831+0000 7f025b7fe700 1 -- 172.21.15.90:0/438005962 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f02500031f0 con 0x7f026410e1e0 2024-04-14T15:11:19.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:19.831+0000 7f025b7fe700 1 -- 172.21.15.90:0/438005962 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f025c001c30 con 0x7f026410e1e0 2024-04-14T15:11:19.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:19.831+0000 7f025b7fe700 1 -- 172.21.15.90:0/438005962 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2040607438 0 0) 0x7f026410ddc0 con 0x7f026410a610 2024-04-14T15:11:19.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:19.831+0000 7f025b7fe700 1 -- 172.21.15.90:0/438005962 >> v1:172.21.15.90:6790/0 conn(0x7f0264107760 legacy=0x7f026410a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:19.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:19.831+0000 7f025b7fe700 1 -- 172.21.15.90:0/438005962 >> v1:172.21.15.92:6789/0 conn(0x7f026410e1e0 legacy=0x7f02641ad2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:19.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:19.831+0000 7f025b7fe700 1 -- 172.21.15.90:0/438005962 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02641b19f0 con 0x7f026410a610 2024-04-14T15:11:19.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:19.831+0000 7f0268905700 1 -- 172.21.15.90:0/438005962 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f02641aea20 con 0x7f026410a610 2024-04-14T15:11:19.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:19.831+0000 7f0268905700 10 client.14772.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:11:19.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:19.831+0000 7f0268905700 1 -- 172.21.15.90:0/438005962 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f02641aeee0 con 0x7f026410a610 2024-04-14T15:11:19.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:19.831+0000 7f0268905700 1 librados: init done 2024-04-14T15:11:19.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:19.831+0000 7f025b7fe700 1 -- 172.21.15.90:0/438005962 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f0254001c40 con 0x7f026410a610 2024-04-14T15:11:19.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:19.831+0000 7f025b7fe700 1 -- 172.21.15.90:0/438005962 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f0254005ca0 con 0x7f026410a610 2024-04-14T15:11:19.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:19.832+0000 7f0268905700 1 -- 172.21.15.90:0/438005962 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0244005120 con 0x7f026410a610 2024-04-14T15:11:19.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:19.833+0000 7f025b7fe700 1 -- 172.21.15.90:0/438005962 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f0254003950 con 0x7f026410a610 2024-04-14T15:11:19.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:19.833+0000 7f025b7fe700 1 -- 172.21.15.90:0/438005962 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(88..88 src has 12..88) ==== 6288+0+0 (unknown 4244947444 0 0) 0x7f025408a3a0 con 0x7f026410a610 2024-04-14T15:11:19.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:19.833+0000 7f025b7fe700 10 client.14772.objecter ms_dispatch 0x7f0264001550 osd_map(88..88 src has 12..88) 2024-04-14T15:11:19.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:19.833+0000 7f025b7fe700 3 client.14772.objecter handle_osd_map got epochs [88,88] > 0 2024-04-14T15:11:19.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:19.833+0000 7f025b7fe700 3 client.14772.objecter handle_osd_map decoding full epoch 88 2024-04-14T15:11:19.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:19.833+0000 7f025b7fe700 20 client.14772.objecter dump_active .. 0 homeless 2024-04-14T15:11:19.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:19.834+0000 7f025b7fe700 10 client.14772.objecter ms_handle_connect 0x7f02500712f0 2024-04-14T15:11:19.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:19.838+0000 7f025b7fe700 1 -- 172.21.15.90:0/438005962 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f0254057510 con 0x7f026410a610 2024-04-14T15:11:20.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:20.039+0000 7f0268905700 1 -- 172.21.15.90:0/438005962 --> v1:172.21.15.90:6789/0 -- mon_command([{prefix=config set, name=mgr_debug_aggressive_pg_num_changes}] v 0) -- 0x7f0244005410 con 0x7f026410a610 2024-04-14T15:11:20.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:20.042+0000 7f025b7fe700 1 -- 172.21.15.90:0/438005962 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{prefix=config set, name=mgr_debug_aggressive_pg_num_changes}]=0 v27)=0 v27) ==== 136+0+0 (unknown 398629962 0 0) 0x7f025405b1c0 con 0x7f026410a610 2024-04-14T15:11:20.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:20.045+0000 7f0268905700 10 librados: watch_flush enter 2024-04-14T15:11:20.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:20.045+0000 7f0268905700 10 librados: watch_flush exit 2024-04-14T15:11:20.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:20.045+0000 7f0268905700 20 client.14772.objecter shutdown clearing up homeless session... 2024-04-14T15:11:20.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:20.045+0000 7f0268905700 10 client.14772.objecter successfully canceled tick 2024-04-14T15:11:20.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:20.045+0000 7f0268905700 1 -- 172.21.15.90:0/438005962 >> v1:172.21.15.90:6800/420935427 conn(0x7f02500712f0 legacy=0x7f02500737b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:20.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:20.045+0000 7f0268905700 1 -- 172.21.15.90:0/438005962 >> v1:172.21.15.90:6789/0 conn(0x7f026410a610 legacy=0x7f026410b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:20.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:20.046+0000 7f0268905700 1 -- 172.21.15.90:0/438005962 shutdown_connections 2024-04-14T15:11:20.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:20.046+0000 7f0268905700 1 -- 172.21.15.90:0/438005962 >> 172.21.15.90:0/438005962 conn(0x7f02640fd290 msgr2=0x7f0264111700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:11:20.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:20.046+0000 7f0268905700 1 -- 172.21.15.90:0/438005962 shutdown_connections 2024-04-14T15:11:20.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:20.046+0000 7f0268905700 1 -- 172.21.15.90:0/438005962 wait complete. 2024-04-14T15:11:20.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:20.046+0000 7f0268905700 1 librados: shutdown 2024-04-14T15:11:20.099 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:11:19 smithi090 podman[79996]: 2024-04-14 15:11:19.970515376 +0000 UTC m=+3.206451684 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-04-14T15:11:20.100 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:11:20 smithi090 podman[79996]: 2024-04-14 15:11:20.053803693 +0000 UTC m=+3.289740000 container create 5f2702315825ca14b91f0cc47d486173ceed9ffcad5cdac18bfba74183f27629 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-14T15:11:20.100 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:11:20 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: [14/Apr/2024:15:11:20] ENGINE Bus STOPPING 2024-04-14T15:11:20.100 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:11:20 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: [14/Apr/2024:15:11:20] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-14T15:11:20.100 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:11:20 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: [14/Apr/2024:15:11:20] ENGINE Bus STOPPED 2024-04-14T15:11:20.100 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:11:20 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: [14/Apr/2024:15:11:20] ENGINE Bus STARTING 2024-04-14T15:11:20.101 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:20 smithi090 ceph-mon[30399]: 4.1c scrub starts 2024-04-14T15:11:20.101 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:20 smithi090 ceph-mon[30399]: 4.1c scrub ok 2024-04-14T15:11:20.101 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:20 smithi090 ceph-mon[30399]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:11:20.101 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:20 smithi090 ceph-mon[30399]: osdmap e88: 8 total, 8 up, 8 in 2024-04-14T15:11:20.101 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:20 smithi090 ceph-mon[30399]: 2.1a scrub starts 2024-04-14T15:11:20.101 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:20 smithi090 ceph-mon[30399]: 2.1a scrub ok 2024-04-14T15:11:20.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:20 smithi092 ceph-mon[33966]: 4.1c scrub starts 2024-04-14T15:11:20.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:20 smithi092 ceph-mon[33966]: 4.1c scrub ok 2024-04-14T15:11:20.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:20 smithi092 ceph-mon[33966]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:11:20.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:20 smithi092 ceph-mon[33966]: osdmap e88: 8 total, 8 up, 8 in 2024-04-14T15:11:20.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:20 smithi092 ceph-mon[33966]: 2.1a scrub starts 2024-04-14T15:11:20.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:20 smithi092 ceph-mon[33966]: 2.1a scrub ok 2024-04-14T15:11:20.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:20 smithi092 ceph-mon[33966]: 4.8 scrub starts 2024-04-14T15:11:20.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:20 smithi092 ceph-mon[33966]: 4.8 scrub ok 2024-04-14T15:11:20.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:20 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/438005962' entity='client.admin' 2024-04-14T15:11:20.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:20 smithi090 ceph-mon[30399]: 4.8 scrub starts 2024-04-14T15:11:20.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:20 smithi090 ceph-mon[30399]: 4.8 scrub ok 2024-04-14T15:11:20.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:20 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/438005962' entity='client.admin' 2024-04-14T15:11:20.442 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:11:20 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: [14/Apr/2024:15:11:20] ENGINE Serving on http://:::9283 2024-04-14T15:11:20.442 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:11:20 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: [14/Apr/2024:15:11:20] ENGINE Bus STARTED 2024-04-14T15:11:20.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:20 smithi090 ceph-mon[41670]: 4.1c scrub starts 2024-04-14T15:11:20.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:20 smithi090 ceph-mon[41670]: 4.1c scrub ok 2024-04-14T15:11:20.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:20 smithi090 ceph-mon[41670]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:11:20.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:20 smithi090 ceph-mon[41670]: osdmap e88: 8 total, 8 up, 8 in 2024-04-14T15:11:20.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:20 smithi090 ceph-mon[41670]: 2.1a scrub starts 2024-04-14T15:11:20.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:20 smithi090 ceph-mon[41670]: 2.1a scrub ok 2024-04-14T15:11:20.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:20 smithi090 ceph-mon[41670]: 4.8 scrub starts 2024-04-14T15:11:20.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:20 smithi090 ceph-mon[41670]: 4.8 scrub ok 2024-04-14T15:11:20.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:20 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/438005962' entity='client.admin' 2024-04-14T15:11:20.940 INFO:tasks.thrashosds:Beginning thrashosds... 2024-04-14T15:11:20.940 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-14T15:11:20.940 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph pg dump --format=json 2024-04-14T15:11:21.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:21 smithi090 ceph-mon[30399]: 2.7 scrub starts 2024-04-14T15:11:21.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:21 smithi090 ceph-mon[30399]: 2.7 scrub ok 2024-04-14T15:11:21.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:21 smithi090 ceph-mon[30399]: 2.2 scrub starts 2024-04-14T15:11:21.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:21 smithi090 ceph-mon[30399]: 2.2 scrub ok 2024-04-14T15:11:21.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:21 smithi090 ceph-mon[30399]: osdmap e89: 8 total, 8 up, 8 in 2024-04-14T15:11:21.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:21 smithi090 ceph-mon[41670]: 2.7 scrub starts 2024-04-14T15:11:21.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:21 smithi090 ceph-mon[41670]: 2.7 scrub ok 2024-04-14T15:11:21.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:21 smithi090 ceph-mon[41670]: 2.2 scrub starts 2024-04-14T15:11:21.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:21 smithi090 ceph-mon[41670]: 2.2 scrub ok 2024-04-14T15:11:21.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:21 smithi090 ceph-mon[41670]: osdmap e89: 8 total, 8 up, 8 in 2024-04-14T15:11:21.193 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:11:21 smithi090 podman[79996]: 2024-04-14 15:11:21.088211894 +0000 UTC m=+4.324148201 container init 5f2702315825ca14b91f0cc47d486173ceed9ffcad5cdac18bfba74183f27629 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-14T15:11:21.193 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:11:21 smithi090 podman[79996]: 2024-04-14 15:11:21.094125333 +0000 UTC m=+4.330061650 container start 5f2702315825ca14b91f0cc47d486173ceed9ffcad5cdac18bfba74183f27629 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-14T15:11:21.194 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:11:21 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-a[80246]: ts=2024-04-14T15:11:21.098Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-04-14T15:11:21.194 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:11:21 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-a[80246]: ts=2024-04-14T15:11:21.098Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-04-14T15:11:21.194 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:11:21 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-a[80246]: ts=2024-04-14T15:11:21.099Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-04-14T15:11:21.194 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:11:21 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-a[80246]: ts=2024-04-14T15:11:21.099Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-04-14T15:11:21.194 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:11:21 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-a[80246]: ts=2024-04-14T15:11:21.099Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-04-14T15:11:21.194 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:11:21 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-a[80246]: ts=2024-04-14T15:11:21.099Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-04-14T15:11:21.194 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:11:21 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-a[80246]: ts=2024-04-14T15:11:21.099Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-04-14T15:11:21.194 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:11:21 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-a[80246]: ts=2024-04-14T15:11:21.099Z caller=node_exporter.go:117 level=info collector=arp 2024-04-14T15:11:21.194 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:11:21 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-a[80246]: ts=2024-04-14T15:11:21.099Z caller=node_exporter.go:117 level=info collector=bcache 2024-04-14T15:11:21.195 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:11:21 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-a[80246]: ts=2024-04-14T15:11:21.099Z caller=node_exporter.go:117 level=info collector=bonding 2024-04-14T15:11:21.195 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:11:21 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-a[80246]: ts=2024-04-14T15:11:21.099Z caller=node_exporter.go:117 level=info collector=btrfs 2024-04-14T15:11:21.195 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:11:21 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-a[80246]: ts=2024-04-14T15:11:21.099Z caller=node_exporter.go:117 level=info collector=conntrack 2024-04-14T15:11:21.195 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:11:21 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-a[80246]: ts=2024-04-14T15:11:21.099Z caller=node_exporter.go:117 level=info collector=cpu 2024-04-14T15:11:21.195 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:11:21 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-a[80246]: ts=2024-04-14T15:11:21.099Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-04-14T15:11:21.195 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:11:21 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-a[80246]: ts=2024-04-14T15:11:21.099Z caller=node_exporter.go:117 level=info collector=diskstats 2024-04-14T15:11:21.195 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:11:21 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-a[80246]: ts=2024-04-14T15:11:21.099Z caller=node_exporter.go:117 level=info collector=dmi 2024-04-14T15:11:21.195 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:11:21 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-a[80246]: ts=2024-04-14T15:11:21.099Z caller=node_exporter.go:117 level=info collector=edac 2024-04-14T15:11:21.195 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:11:21 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-a[80246]: ts=2024-04-14T15:11:21.099Z caller=node_exporter.go:117 level=info collector=entropy 2024-04-14T15:11:21.196 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:11:21 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-a[80246]: ts=2024-04-14T15:11:21.099Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-04-14T15:11:21.196 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:11:21 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-a[80246]: ts=2024-04-14T15:11:21.099Z caller=node_exporter.go:117 level=info collector=filefd 2024-04-14T15:11:21.196 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:11:21 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-a[80246]: ts=2024-04-14T15:11:21.099Z caller=node_exporter.go:117 level=info collector=filesystem 2024-04-14T15:11:21.196 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:11:21 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-a[80246]: ts=2024-04-14T15:11:21.099Z caller=node_exporter.go:117 level=info collector=hwmon 2024-04-14T15:11:21.196 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:11:21 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-a[80246]: ts=2024-04-14T15:11:21.099Z caller=node_exporter.go:117 level=info collector=infiniband 2024-04-14T15:11:21.196 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:11:21 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-a[80246]: ts=2024-04-14T15:11:21.099Z caller=node_exporter.go:117 level=info collector=ipvs 2024-04-14T15:11:21.196 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:11:21 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-a[80246]: ts=2024-04-14T15:11:21.099Z caller=node_exporter.go:117 level=info collector=loadavg 2024-04-14T15:11:21.196 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:11:21 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-a[80246]: ts=2024-04-14T15:11:21.099Z caller=node_exporter.go:117 level=info collector=mdadm 2024-04-14T15:11:21.197 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:11:21 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-a[80246]: ts=2024-04-14T15:11:21.099Z caller=node_exporter.go:117 level=info collector=meminfo 2024-04-14T15:11:21.197 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:11:21 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-a[80246]: ts=2024-04-14T15:11:21.099Z caller=node_exporter.go:117 level=info collector=netclass 2024-04-14T15:11:21.197 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:11:21 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-a[80246]: ts=2024-04-14T15:11:21.099Z caller=node_exporter.go:117 level=info collector=netdev 2024-04-14T15:11:21.197 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:11:21 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-a[80246]: ts=2024-04-14T15:11:21.099Z caller=node_exporter.go:117 level=info collector=netstat 2024-04-14T15:11:21.197 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:11:21 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-a[80246]: ts=2024-04-14T15:11:21.099Z caller=node_exporter.go:117 level=info collector=nfs 2024-04-14T15:11:21.197 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:11:21 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-a[80246]: ts=2024-04-14T15:11:21.099Z caller=node_exporter.go:117 level=info collector=nfsd 2024-04-14T15:11:21.197 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:11:21 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-a[80246]: ts=2024-04-14T15:11:21.099Z caller=node_exporter.go:117 level=info collector=nvme 2024-04-14T15:11:21.197 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:11:21 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-a[80246]: ts=2024-04-14T15:11:21.099Z caller=node_exporter.go:117 level=info collector=os 2024-04-14T15:11:21.197 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:11:21 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-a[80246]: ts=2024-04-14T15:11:21.099Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-04-14T15:11:21.198 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:11:21 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-a[80246]: ts=2024-04-14T15:11:21.099Z caller=node_exporter.go:117 level=info collector=pressure 2024-04-14T15:11:21.198 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:11:21 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-a[80246]: ts=2024-04-14T15:11:21.099Z caller=node_exporter.go:117 level=info collector=rapl 2024-04-14T15:11:21.198 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:11:21 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-a[80246]: ts=2024-04-14T15:11:21.099Z caller=node_exporter.go:117 level=info collector=schedstat 2024-04-14T15:11:21.198 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:11:21 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-a[80246]: ts=2024-04-14T15:11:21.099Z caller=node_exporter.go:117 level=info collector=selinux 2024-04-14T15:11:21.198 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:11:21 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-a[80246]: ts=2024-04-14T15:11:21.099Z caller=node_exporter.go:117 level=info collector=sockstat 2024-04-14T15:11:21.198 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:11:21 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-a[80246]: ts=2024-04-14T15:11:21.099Z caller=node_exporter.go:117 level=info collector=softnet 2024-04-14T15:11:21.198 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:11:21 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-a[80246]: ts=2024-04-14T15:11:21.099Z caller=node_exporter.go:117 level=info collector=stat 2024-04-14T15:11:21.198 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:11:21 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-a[80246]: ts=2024-04-14T15:11:21.099Z caller=node_exporter.go:117 level=info collector=tapestats 2024-04-14T15:11:21.198 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:11:21 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-a[80246]: ts=2024-04-14T15:11:21.099Z caller=node_exporter.go:117 level=info collector=textfile 2024-04-14T15:11:21.199 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:11:21 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-a[80246]: ts=2024-04-14T15:11:21.099Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-04-14T15:11:21.199 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:11:21 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-a[80246]: ts=2024-04-14T15:11:21.099Z caller=node_exporter.go:117 level=info collector=time 2024-04-14T15:11:21.199 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:11:21 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-a[80246]: ts=2024-04-14T15:11:21.099Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-04-14T15:11:21.199 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:11:21 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-a[80246]: ts=2024-04-14T15:11:21.099Z caller=node_exporter.go:117 level=info collector=uname 2024-04-14T15:11:21.199 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:11:21 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-a[80246]: ts=2024-04-14T15:11:21.099Z caller=node_exporter.go:117 level=info collector=vmstat 2024-04-14T15:11:21.199 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:11:21 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-a[80246]: ts=2024-04-14T15:11:21.099Z caller=node_exporter.go:117 level=info collector=xfs 2024-04-14T15:11:21.199 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:11:21 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-a[80246]: ts=2024-04-14T15:11:21.099Z caller=node_exporter.go:117 level=info collector=zfs 2024-04-14T15:11:21.199 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:11:21 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-a[80246]: ts=2024-04-14T15:11:21.100Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-04-14T15:11:21.199 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:11:21 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-a[80246]: ts=2024-04-14T15:11:21.100Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-04-14T15:11:21.270 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:11:21.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:21 smithi092 ceph-mon[33966]: 2.7 scrub starts 2024-04-14T15:11:21.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:21 smithi092 ceph-mon[33966]: 2.7 scrub ok 2024-04-14T15:11:21.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:21 smithi092 ceph-mon[33966]: 2.2 scrub starts 2024-04-14T15:11:21.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:21 smithi092 ceph-mon[33966]: 2.2 scrub ok 2024-04-14T15:11:21.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:21 smithi092 ceph-mon[33966]: osdmap e89: 8 total, 8 up, 8 in 2024-04-14T15:11:21.515 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:11:21 smithi090 bash[79996]: 5f2702315825ca14b91f0cc47d486173ceed9ffcad5cdac18bfba74183f27629 2024-04-14T15:11:21.515 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:11:21 smithi090 systemd[1]: Started Ceph node-exporter.a for 0e288098-fa70-11ee-bc8e-c7b262605968. 2024-04-14T15:11:22.182 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:22 smithi092 ceph-mon[33966]: 3.2 scrub starts 2024-04-14T15:11:22.182 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:22 smithi092 ceph-mon[33966]: 3.2 scrub ok 2024-04-14T15:11:22.182 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:22 smithi092 ceph-mon[33966]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:11:22.182 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:22 smithi092 ceph-mon[33966]: 6.1d deep-scrub starts 2024-04-14T15:11:22.182 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:22 smithi092 ceph-mon[33966]: 6.1d deep-scrub ok 2024-04-14T15:11:22.183 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:22 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:22.183 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:22 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:22.183 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:22 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:11:22.183 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:22 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:11:22.183 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:22 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:22.183 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:22 smithi092 ceph-mon[33966]: Deploying daemon node-exporter.b on smithi092 2024-04-14T15:11:22.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:22 smithi090 ceph-mon[30399]: 3.2 scrub starts 2024-04-14T15:11:22.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:22 smithi090 ceph-mon[30399]: 3.2 scrub ok 2024-04-14T15:11:22.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:22 smithi090 ceph-mon[30399]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:11:22.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:22 smithi090 ceph-mon[30399]: 6.1d deep-scrub starts 2024-04-14T15:11:22.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:22 smithi090 ceph-mon[30399]: 6.1d deep-scrub ok 2024-04-14T15:11:22.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:22 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:22.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:22 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:22.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:22 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:11:22.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:22 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:11:22.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:22 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:22.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:22 smithi090 ceph-mon[30399]: Deploying daemon node-exporter.b on smithi092 2024-04-14T15:11:22.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:22 smithi090 ceph-mon[41670]: 3.2 scrub starts 2024-04-14T15:11:22.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:22 smithi090 ceph-mon[41670]: 3.2 scrub ok 2024-04-14T15:11:22.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:22 smithi090 ceph-mon[41670]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:11:22.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:22 smithi090 ceph-mon[41670]: 6.1d deep-scrub starts 2024-04-14T15:11:22.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:22 smithi090 ceph-mon[41670]: 6.1d deep-scrub ok 2024-04-14T15:11:22.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:22 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:22.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:22 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:22.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:22 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:11:22.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:22 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:11:22.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:22 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:22.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:22 smithi090 ceph-mon[41670]: Deploying daemon node-exporter.b on smithi092 2024-04-14T15:11:22.872 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:11:22 smithi092 bash[67889]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-04-14T15:11:22.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:22.989+0000 7f2f29a16700 1 -- 172.21.15.90:0/2754345900 >> v1:172.21.15.90:6789/0 conn(0x7f2f24108880 legacy=0x7f2f24108c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:22.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:22.989+0000 7f2f29a16700 1 -- 172.21.15.90:0/2754345900 shutdown_connections 2024-04-14T15:11:22.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:22.989+0000 7f2f29a16700 1 -- 172.21.15.90:0/2754345900 >> 172.21.15.90:0/2754345900 conn(0x7f2f240fe3d0 msgr2=0x7f2f241007f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:11:22.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:22.989+0000 7f2f29a16700 1 -- 172.21.15.90:0/2754345900 shutdown_connections 2024-04-14T15:11:22.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:22.989+0000 7f2f29a16700 1 -- 172.21.15.90:0/2754345900 wait complete. 2024-04-14T15:11:22.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:22.989+0000 7f2f29a16700 1 librados: starting msgr at 2024-04-14T15:11:22.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:22.989+0000 7f2f29a16700 1 librados: starting objecter 2024-04-14T15:11:22.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:22.989+0000 7f2f29a16700 1 Processor -- start 2024-04-14T15:11:22.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:22.990+0000 7f2f29a16700 1 -- start start 2024-04-14T15:11:22.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:22.990+0000 7f2f29a16700 1 librados: setting wanted keys 2024-04-14T15:11:22.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:22.990+0000 7f2f29a16700 1 librados: calling monclient init 2024-04-14T15:11:22.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:22.990+0000 7f2f29a16700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f2f2410f070 con 0x7f2f2410b730 2024-04-14T15:11:22.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:22.990+0000 7f2f29a16700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f2f241a8f10 con 0x7f2f24108880 2024-04-14T15:11:22.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:22.990+0000 7f2f29a16700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f2f241aa0f0 con 0x7f2f2410f300 2024-04-14T15:11:22.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:22.990+0000 7f2f1bfff700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f2f2410b730 0x7f2f2410c6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:59372/0 (socket says 172.21.15.90:59372) 2024-04-14T15:11:22.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:22.990+0000 7f2f237fe700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f2f2410f300 0x7f2f241a67e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:43862/0 (socket says 172.21.15.90:43862) 2024-04-14T15:11:22.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:22.990+0000 7f2f1bfff700 1 -- 172.21.15.90:0/1347748414 learned_addr learned my addr 172.21.15.90:0/1347748414 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:11:22.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:22.990+0000 7f2f28a14700 10 client.?.objecter ms_handle_connect 0x7f2f2410b730 2024-04-14T15:11:22.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:22.990+0000 7f2f28a14700 10 client.?.objecter resend_mon_ops 2024-04-14T15:11:22.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:22.990+0000 7f2f28a14700 10 client.?.objecter ms_handle_connect 0x7f2f24108880 2024-04-14T15:11:22.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:22.990+0000 7f2f28a14700 10 client.?.objecter resend_mon_ops 2024-04-14T15:11:22.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:22.990+0000 7f2f28a14700 10 client.?.objecter ms_handle_connect 0x7f2f2410f300 2024-04-14T15:11:22.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:22.990+0000 7f2f28a14700 10 client.?.objecter resend_mon_ops 2024-04-14T15:11:22.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:22.990+0000 7f2f28a14700 1 -- 172.21.15.90:0/1347748414 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1583725936 0 0) 0x7f2f241a8f10 con 0x7f2f24108880 2024-04-14T15:11:22.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:22.990+0000 7f2f28a14700 1 -- 172.21.15.90:0/1347748414 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f100035b0 con 0x7f2f24108880 2024-04-14T15:11:22.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:22.990+0000 7f2f28a14700 1 -- 172.21.15.90:0/1347748414 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2868850133 0 0) 0x7f2f2410f070 con 0x7f2f2410b730 2024-04-14T15:11:22.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:22.990+0000 7f2f28a14700 1 -- 172.21.15.90:0/1347748414 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f241a8f10 con 0x7f2f2410b730 2024-04-14T15:11:22.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:22.990+0000 7f2f28a14700 1 -- 172.21.15.90:0/1347748414 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1554080203 0 0) 0x7f2f241aa0f0 con 0x7f2f2410f300 2024-04-14T15:11:22.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:22.990+0000 7f2f28a14700 1 -- 172.21.15.90:0/1347748414 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f2410f070 con 0x7f2f2410f300 2024-04-14T15:11:22.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:22.990+0000 7f2f28a14700 1 -- 172.21.15.90:0/1347748414 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4262466167 0 0) 0x7f2f241a8f10 con 0x7f2f2410b730 2024-04-14T15:11:22.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:22.990+0000 7f2f28a14700 1 -- 172.21.15.90:0/1347748414 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2f241aa0f0 con 0x7f2f2410b730 2024-04-14T15:11:22.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:22.990+0000 7f2f28a14700 1 -- 172.21.15.90:0/1347748414 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f2f0c0018f0 con 0x7f2f2410b730 2024-04-14T15:11:22.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:22.990+0000 7f2f28a14700 1 -- 172.21.15.90:0/1347748414 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2356322317 0 0) 0x7f2f2410f070 con 0x7f2f2410f300 2024-04-14T15:11:22.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:22.990+0000 7f2f28a14700 1 -- 172.21.15.90:0/1347748414 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2f241a8f10 con 0x7f2f2410f300 2024-04-14T15:11:22.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:22.990+0000 7f2f28a14700 1 -- 172.21.15.90:0/1347748414 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f2f1c001bb0 con 0x7f2f2410f300 2024-04-14T15:11:22.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:22.990+0000 7f2f28a14700 1 -- 172.21.15.90:0/1347748414 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2760056087 0 0) 0x7f2f100035b0 con 0x7f2f24108880 2024-04-14T15:11:22.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:22.990+0000 7f2f28a14700 1 -- 172.21.15.90:0/1347748414 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2f2410f070 con 0x7f2f24108880 2024-04-14T15:11:22.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:22.990+0000 7f2f28a14700 1 -- 172.21.15.90:0/1347748414 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f2f140015d0 con 0x7f2f24108880 2024-04-14T15:11:22.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:22.991+0000 7f2f28a14700 1 -- 172.21.15.90:0/1347748414 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1837796250 0 0) 0x7f2f241aa0f0 con 0x7f2f2410b730 2024-04-14T15:11:22.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:22.991+0000 7f2f28a14700 1 -- 172.21.15.90:0/1347748414 >> v1:172.21.15.90:6790/0 conn(0x7f2f2410f300 legacy=0x7f2f241a67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:22.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:22.991+0000 7f2f28a14700 1 -- 172.21.15.90:0/1347748414 >> v1:172.21.15.92:6789/0 conn(0x7f2f24108880 legacy=0x7f2f2410bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:22.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:22.991+0000 7f2f28a14700 1 -- 172.21.15.90:0/1347748414 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f241ab310 con 0x7f2f2410b730 2024-04-14T15:11:22.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:22.991+0000 7f2f29a16700 1 -- 172.21.15.90:0/1347748414 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2f241a9140 con 0x7f2f2410b730 2024-04-14T15:11:22.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:22.991+0000 7f2f29a16700 10 client.14778.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:11:22.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:22.991+0000 7f2f29a16700 1 -- 172.21.15.90:0/1347748414 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f2f241a9600 con 0x7f2f2410b730 2024-04-14T15:11:22.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:22.991+0000 7f2f29a16700 1 librados: init done 2024-04-14T15:11:22.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:22.991+0000 7f2f28a14700 1 -- 172.21.15.90:0/1347748414 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f2f0c001d00 con 0x7f2f2410b730 2024-04-14T15:11:22.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:22.991+0000 7f2f28a14700 1 -- 172.21.15.90:0/1347748414 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f2f0c005dc0 con 0x7f2f2410b730 2024-04-14T15:11:22.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:22.991+0000 7f2f29a16700 1 -- 172.21.15.90:0/1347748414 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f04005120 con 0x7f2f2410b730 2024-04-14T15:11:22.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:22.992+0000 7f2f28a14700 1 -- 172.21.15.90:0/1347748414 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f2f0c003950 con 0x7f2f2410b730 2024-04-14T15:11:22.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:22.993+0000 7f2f28a14700 1 -- 172.21.15.90:0/1347748414 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(90..90 src has 12..90) ==== 6288+0+0 (unknown 890274275 0 0) 0x7f2f0c08a2a0 con 0x7f2f2410b730 2024-04-14T15:11:22.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:22.993+0000 7f2f28a14700 10 client.14778.objecter ms_dispatch 0x7f2f24001550 osd_map(90..90 src has 12..90) 2024-04-14T15:11:22.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:22.993+0000 7f2f28a14700 3 client.14778.objecter handle_osd_map got epochs [90,90] > 0 2024-04-14T15:11:22.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:22.993+0000 7f2f28a14700 3 client.14778.objecter handle_osd_map decoding full epoch 90 2024-04-14T15:11:22.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:22.993+0000 7f2f28a14700 20 client.14778.objecter dump_active .. 0 homeless 2024-04-14T15:11:22.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:22.993+0000 7f2f28a14700 10 client.14778.objecter ms_handle_connect 0x7f2f10071480 2024-04-14T15:11:22.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:22.997+0000 7f2f28a14700 1 -- 172.21.15.90:0/1347748414 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f2f0c057410 con 0x7f2f2410b730 2024-04-14T15:11:23.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:23.198+0000 7f2f29a16700 1 -- 172.21.15.90:0/1347748414 --> v1:172.21.15.90:6800/420935427 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2f04000b90 con 0x7f2f10071480 2024-04-14T15:11:23.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:23.208+0000 7f2f28a14700 1 -- 172.21.15.90:0/1347748414 <== mgr.14610 v1:172.21.15.90:6800/420935427 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+346058 (unknown 2965378022 0 3354114319) 0x7f2f04000b90 con 0x7f2f10071480 2024-04-14T15:11:23.212 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:11:23.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:23.210+0000 7f2f29a16700 10 librados: watch_flush enter 2024-04-14T15:11:23.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:23.210+0000 7f2f29a16700 10 librados: watch_flush exit 2024-04-14T15:11:23.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:23.210+0000 7f2f29a16700 20 client.14778.objecter shutdown clearing up homeless session... 2024-04-14T15:11:23.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:23.210+0000 7f2f29a16700 10 client.14778.objecter successfully canceled tick 2024-04-14T15:11:23.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:23.211+0000 7f2f29a16700 1 -- 172.21.15.90:0/1347748414 >> v1:172.21.15.90:6800/420935427 conn(0x7f2f10071480 legacy=0x7f2f10073940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:23.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:23.211+0000 7f2f29a16700 1 -- 172.21.15.90:0/1347748414 >> v1:172.21.15.90:6789/0 conn(0x7f2f2410b730 legacy=0x7f2f2410c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:23.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:23.211+0000 7f2f29a16700 1 -- 172.21.15.90:0/1347748414 shutdown_connections 2024-04-14T15:11:23.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:23.211+0000 7f2f29a16700 1 -- 172.21.15.90:0/1347748414 >> 172.21.15.90:0/1347748414 conn(0x7f2f240fe3d0 msgr2=0x7f2f241028e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:11:23.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:23.211+0000 7f2f29a16700 1 -- 172.21.15.90:0/1347748414 shutdown_connections 2024-04-14T15:11:23.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:23.211+0000 7f2f29a16700 1 -- 172.21.15.90:0/1347748414 wait complete. 2024-04-14T15:11:23.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:23.211+0000 7f2f29a16700 1 librados: shutdown 2024-04-14T15:11:23.216 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-14T15:11:23.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:23 smithi092 ceph-mon[33966]: 6.1f scrub starts 2024-04-14T15:11:23.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:23 smithi092 ceph-mon[33966]: 6.1f scrub ok 2024-04-14T15:11:23.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:23 smithi092 ceph-mon[33966]: 5.d scrub starts 2024-04-14T15:11:23.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:23 smithi092 ceph-mon[33966]: 5.d scrub ok 2024-04-14T15:11:23.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:23 smithi092 ceph-mon[33966]: 4.f scrub starts 2024-04-14T15:11:23.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:23 smithi092 ceph-mon[33966]: 4.f scrub ok 2024-04-14T15:11:23.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:23 smithi092 ceph-mon[33966]: osdmap e90: 8 total, 8 up, 8 in 2024-04-14T15:11:23.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:23 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:11:23.373 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:11:23 smithi092 bash[67889]: Getting image source signatures 2024-04-14T15:11:23.373 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:11:23 smithi092 bash[67889]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-04-14T15:11:23.373 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:11:23 smithi092 bash[67889]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-04-14T15:11:23.373 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:11:23 smithi092 bash[67889]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-04-14T15:11:23.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:23 smithi090 ceph-mon[30399]: 6.1f scrub starts 2024-04-14T15:11:23.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:23 smithi090 ceph-mon[30399]: 6.1f scrub ok 2024-04-14T15:11:23.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:23 smithi090 ceph-mon[30399]: 5.d scrub starts 2024-04-14T15:11:23.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:23 smithi090 ceph-mon[30399]: 5.d scrub ok 2024-04-14T15:11:23.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:23 smithi090 ceph-mon[30399]: 4.f scrub starts 2024-04-14T15:11:23.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:23 smithi090 ceph-mon[30399]: 4.f scrub ok 2024-04-14T15:11:23.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:23 smithi090 ceph-mon[30399]: osdmap e90: 8 total, 8 up, 8 in 2024-04-14T15:11:23.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:23 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:11:23.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:23 smithi090 ceph-mon[41670]: 6.1f scrub starts 2024-04-14T15:11:23.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:23 smithi090 ceph-mon[41670]: 6.1f scrub ok 2024-04-14T15:11:23.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:23 smithi090 ceph-mon[41670]: 5.d scrub starts 2024-04-14T15:11:23.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:23 smithi090 ceph-mon[41670]: 5.d scrub ok 2024-04-14T15:11:23.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:23 smithi090 ceph-mon[41670]: 4.f scrub starts 2024-04-14T15:11:23.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:23 smithi090 ceph-mon[41670]: 4.f scrub ok 2024-04-14T15:11:23.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:23 smithi090 ceph-mon[41670]: osdmap e90: 8 total, 8 up, 8 in 2024-04-14T15:11:23.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:23 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:11:23.910 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":44,"stamp":"2024-04-14T15:11:22.951169+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1024,"num_read_kb":870,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":1771324,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":608,"ondisk_log_size":608,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":236572,"kb_used_data":7964,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":749724644,"statfs":{"total":767960285184,"available":767718035456,"internally_reserved":0,"allocated":8155136,"data_stored":4551736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":234016306},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001428"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.763793+0000","last_change":"2024-04-14T15:09:49.130210+0000","last_active":"2024-04-14T15:11:00.763793+0000","last_peered":"2024-04-14T15:11:00.763793+0000","last_clean":"2024-04-14T15:11:00.763793+0000","last_became_active":"2024-04-14T15:09:49.129636+0000","last_became_peered":"2024-04-14T15:09:49.129636+0000","last_unstale":"2024-04-14T15:11:00.763793+0000","last_undegraded":"2024-04-14T15:11:00.763793+0000","last_fullsized":"2024-04-14T15:11:00.763793+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:10.759713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":47,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.763604+0000","last_change":"2024-04-14T15:09:47.043210+0000","last_active":"2024-04-14T15:11:00.763604+0000","last_peered":"2024-04-14T15:11:00.763604+0000","last_clean":"2024-04-14T15:11:00.763604+0000","last_became_active":"2024-04-14T15:09:47.042538+0000","last_became_peered":"2024-04-14T15:09:47.042538+0000","last_unstale":"2024-04-14T15:11:00.763604+0000","last_undegraded":"2024-04-14T15:11:00.763604+0000","last_fullsized":"2024-04-14T15:11:00.763604+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:09:46.023663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:09.982965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.210463+0000","last_change":"2024-04-14T15:09:43.018880+0000","last_active":"2024-04-14T15:11:01.210463+0000","last_peered":"2024-04-14T15:11:01.210463+0000","last_clean":"2024-04-14T15:11:01.210463+0000","last_became_active":"2024-04-14T15:09:43.017351+0000","last_became_peered":"2024-04-14T15:09:43.017351+0000","last_unstale":"2024-04-14T15:11:01.210463+0000","last_undegraded":"2024-04-14T15:11:01.210463+0000","last_fullsized":"2024-04-14T15:11:01.210463+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:09:42.004317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:10.992803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"72'15","reported_seq":76,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764610+0000","last_change":"2024-04-14T15:09:45.034424+0000","last_active":"2024-04-14T15:11:00.764610+0000","last_peered":"2024-04-14T15:11:00.764610+0000","last_clean":"2024-04-14T15:11:00.764610+0000","last_became_active":"2024-04-14T15:09:45.034077+0000","last_became_peered":"2024-04-14T15:09:45.034077+0000","last_unstale":"2024-04-14T15:11:00.764610+0000","last_undegraded":"2024-04-14T15:11:00.764610+0000","last_fullsized":"2024-04-14T15:11:00.764610+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:08.933517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.763826+0000","last_change":"2024-04-14T15:09:49.059264+0000","last_active":"2024-04-14T15:11:00.763826+0000","last_peered":"2024-04-14T15:11:00.763826+0000","last_clean":"2024-04-14T15:11:00.763826+0000","last_became_active":"2024-04-14T15:09:49.058904+0000","last_became_peered":"2024-04-14T15:09:49.058904+0000","last_unstale":"2024-04-14T15:11:00.763826+0000","last_undegraded":"2024-04-14T15:11:00.763826+0000","last_fullsized":"2024-04-14T15:11:00.763826+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:13.856879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":47,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.763106+0000","last_change":"2024-04-14T15:09:47.041785+0000","last_active":"2024-04-14T15:11:00.763106+0000","last_peered":"2024-04-14T15:11:00.763106+0000","last_clean":"2024-04-14T15:11:00.763106+0000","last_became_active":"2024-04-14T15:09:47.041486+0000","last_became_peered":"2024-04-14T15:09:47.041486+0000","last_unstale":"2024-04-14T15:11:00.763106+0000","last_undegraded":"2024-04-14T15:11:00.763106+0000","last_fullsized":"2024-04-14T15:11:00.763106+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:09:46.023663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:03.085264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.765067+0000","last_change":"2024-04-14T15:09:43.023226+0000","last_active":"2024-04-14T15:11:00.765067+0000","last_peered":"2024-04-14T15:11:00.765067+0000","last_clean":"2024-04-14T15:11:00.765067+0000","last_became_active":"2024-04-14T15:09:43.022938+0000","last_became_peered":"2024-04-14T15:09:43.022938+0000","last_unstale":"2024-04-14T15:11:00.765067+0000","last_undegraded":"2024-04-14T15:11:00.765067+0000","last_fullsized":"2024-04-14T15:11:00.765067+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:09:42.004317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:10.564708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"72'9","reported_seq":67,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.763831+0000","last_change":"2024-04-14T15:09:45.032381+0000","last_active":"2024-04-14T15:11:00.763831+0000","last_peered":"2024-04-14T15:11:00.763831+0000","last_clean":"2024-04-14T15:11:00.763831+0000","last_became_active":"2024-04-14T15:09:45.032079+0000","last_became_peered":"2024-04-14T15:09:45.032079+0000","last_unstale":"2024-04-14T15:11:00.763831+0000","last_undegraded":"2024-04-14T15:11:00.763831+0000","last_fullsized":"2024-04-14T15:11:00.763831+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":51,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:03.745722+0000","last_change":"2024-04-14T15:11:03.745753+0000","last_active":"2024-04-14T15:11:03.745722+0000","last_peered":"2024-04-14T15:11:03.745722+0000","last_clean":"2024-04-14T15:11:03.745722+0000","last_became_active":"2024-04-14T15:09:49.069590+0000","last_became_peered":"2024-04-14T15:09:49.069590+0000","last_unstale":"2024-04-14T15:11:03.745722+0000","last_undegraded":"2024-04-14T15:11:03.745722+0000","last_fullsized":"2024-04-14T15:11:03.745722+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:03.745644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:03.745644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:29.376857+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:04.881903+0000","last_change":"2024-04-14T15:11:04.881923+0000","last_active":"2024-04-14T15:11:04.881903+0000","last_peered":"2024-04-14T15:11:04.881903+0000","last_clean":"2024-04-14T15:11:04.881903+0000","last_became_active":"2024-04-14T15:09:47.046517+0000","last_became_peered":"2024-04-14T15:09:47.046517+0000","last_unstale":"2024-04-14T15:11:04.881903+0000","last_undegraded":"2024-04-14T15:11:04.881903+0000","last_fullsized":"2024-04-14T15:11:04.881903+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:04.881846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:04.881846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:24.065968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":63,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.938224+0000","last_change":"2024-04-14T15:10:43.084925+0000","last_active":"2024-04-14T15:11:00.938224+0000","last_peered":"2024-04-14T15:11:00.938224+0000","last_clean":"2024-04-14T15:11:00.938224+0000","last_became_active":"2024-04-14T15:09:43.021986+0000","last_became_peered":"2024-04-14T15:09:43.021986+0000","last_unstale":"2024-04-14T15:11:00.938224+0000","last_undegraded":"2024-04-14T15:11:00.938224+0000","last_fullsized":"2024-04-14T15:11:00.938224+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:43.084820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:10:43.084820+0000","last_clean_scrub_stamp":"2024-04-14T15:10:43.084820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:55.633388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"87'7","reported_seq":71,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:10.510879+0000","last_change":"2024-04-14T15:11:10.510929+0000","last_active":"2024-04-14T15:11:10.510879+0000","last_peered":"2024-04-14T15:11:10.510879+0000","last_clean":"2024-04-14T15:11:10.510879+0000","last_became_active":"2024-04-14T15:09:45.028667+0000","last_became_peered":"2024-04-14T15:09:45.028667+0000","last_unstale":"2024-04-14T15:11:10.510879+0000","last_undegraded":"2024-04-14T15:11:10.510879+0000","last_fullsized":"2024-04-14T15:11:10.510879+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'7","last_scrub_stamp":"2024-04-14T15:11:10.510785+0000","last_deep_scrub":"87'7","last_deep_scrub_stamp":"2024-04-14T15:11:10.510785+0000","last_clean_scrub_stamp":"2024-04-14T15:11:10.510785+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:17.842170+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":51,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.209600+0000","last_change":"2024-04-14T15:10:54.765733+0000","last_active":"2024-04-14T15:11:01.209600+0000","last_peered":"2024-04-14T15:11:01.209600+0000","last_clean":"2024-04-14T15:11:01.209600+0000","last_became_active":"2024-04-14T15:09:49.061756+0000","last_became_peered":"2024-04-14T15:09:49.061756+0000","last_unstale":"2024-04-14T15:11:01.209600+0000","last_undegraded":"2024-04-14T15:11:01.209600+0000","last_fullsized":"2024-04-14T15:11:01.209600+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:54.765650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:10:54.765650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:59.282104+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":47,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.211280+0000","last_change":"2024-04-14T15:09:47.042682+0000","last_active":"2024-04-14T15:11:01.211280+0000","last_peered":"2024-04-14T15:11:01.211280+0000","last_clean":"2024-04-14T15:11:01.211280+0000","last_became_active":"2024-04-14T15:09:47.042384+0000","last_became_peered":"2024-04-14T15:09:47.042384+0000","last_unstale":"2024-04-14T15:11:01.211280+0000","last_undegraded":"2024-04-14T15:11:01.211280+0000","last_fullsized":"2024-04-14T15:11:01.211280+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:09:46.023663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:07.893460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764049+0000","last_change":"2024-04-14T15:10:07.764431+0000","last_active":"2024-04-14T15:11:00.764049+0000","last_peered":"2024-04-14T15:11:00.764049+0000","last_clean":"2024-04-14T15:11:00.764049+0000","last_became_active":"2024-04-14T15:10:07.764086+0000","last_became_peered":"2024-04-14T15:10:07.764086+0000","last_unstale":"2024-04-14T15:11:00.764049+0000","last_undegraded":"2024-04-14T15:11:00.764049+0000","last_fullsized":"2024-04-14T15:11:00.764049+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:09:42.004317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:08.369982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"72'9","reported_seq":75,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:17.667754+0000","last_change":"2024-04-14T15:11:17.667793+0000","last_active":"2024-04-14T15:11:17.667754+0000","last_peered":"2024-04-14T15:11:17.667754+0000","last_clean":"2024-04-14T15:11:17.667754+0000","last_became_active":"2024-04-14T15:09:45.030093+0000","last_became_peered":"2024-04-14T15:09:45.030093+0000","last_unstale":"2024-04-14T15:11:17.667754+0000","last_undegraded":"2024-04-14T15:11:17.667754+0000","last_fullsized":"2024-04-14T15:11:17.667754+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-04-14T15:11:17.667682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:17.667682+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:31.898486+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.763768+0000","last_change":"2024-04-14T15:09:49.128334+0000","last_active":"2024-04-14T15:11:00.763768+0000","last_peered":"2024-04-14T15:11:00.763768+0000","last_clean":"2024-04-14T15:11:00.763768+0000","last_became_active":"2024-04-14T15:09:49.128007+0000","last_became_peered":"2024-04-14T15:09:49.128007+0000","last_unstale":"2024-04-14T15:11:00.763768+0000","last_undegraded":"2024-04-14T15:11:00.763768+0000","last_fullsized":"2024-04-14T15:11:00.763768+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:04.103893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764349+0000","last_change":"2024-04-14T15:10:48.466668+0000","last_active":"2024-04-14T15:11:00.764349+0000","last_peered":"2024-04-14T15:11:00.764349+0000","last_clean":"2024-04-14T15:11:00.764349+0000","last_became_active":"2024-04-14T15:09:47.048045+0000","last_became_peered":"2024-04-14T15:09:47.048045+0000","last_unstale":"2024-04-14T15:11:00.764349+0000","last_undegraded":"2024-04-14T15:11:00.764349+0000","last_fullsized":"2024-04-14T15:11:00.764349+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:48.466556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:10:48.466556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:50.564963+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764000+0000","last_change":"2024-04-14T15:09:43.025155+0000","last_active":"2024-04-14T15:11:00.764000+0000","last_peered":"2024-04-14T15:11:00.764000+0000","last_clean":"2024-04-14T15:11:00.764000+0000","last_became_active":"2024-04-14T15:09:43.024853+0000","last_became_peered":"2024-04-14T15:09:43.024853+0000","last_unstale":"2024-04-14T15:11:00.764000+0000","last_undegraded":"2024-04-14T15:11:00.764000+0000","last_fullsized":"2024-04-14T15:11:00.764000+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:09:42.004317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:08.422553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"72'12","reported_seq":82,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:14.461167+0000","last_change":"2024-04-14T15:11:14.461199+0000","last_active":"2024-04-14T15:11:14.461167+0000","last_peered":"2024-04-14T15:11:14.461167+0000","last_clean":"2024-04-14T15:11:14.461167+0000","last_became_active":"2024-04-14T15:09:45.037173+0000","last_became_peered":"2024-04-14T15:09:45.037173+0000","last_unstale":"2024-04-14T15:11:14.461167+0000","last_undegraded":"2024-04-14T15:11:14.461167+0000","last_fullsized":"2024-04-14T15:11:14.461167+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'12","last_scrub_stamp":"2024-04-14T15:11:14.461100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:14.461100+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:25.102035+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":51,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:04.539857+0000","last_change":"2024-04-14T15:11:04.539877+0000","last_active":"2024-04-14T15:11:04.539857+0000","last_peered":"2024-04-14T15:11:04.539857+0000","last_clean":"2024-04-14T15:11:04.539857+0000","last_became_active":"2024-04-14T15:09:49.127979+0000","last_became_peered":"2024-04-14T15:09:49.127979+0000","last_unstale":"2024-04-14T15:11:04.539857+0000","last_undegraded":"2024-04-14T15:11:04.539857+0000","last_fullsized":"2024-04-14T15:11:04.539857+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:04.539789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:04.539789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:16.636271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:16.297499+0000","last_change":"2024-04-14T15:11:16.297521+0000","last_active":"2024-04-14T15:11:16.297499+0000","last_peered":"2024-04-14T15:11:16.297499+0000","last_clean":"2024-04-14T15:11:16.297499+0000","last_became_active":"2024-04-14T15:09:47.042733+0000","last_became_peered":"2024-04-14T15:09:47.042733+0000","last_unstale":"2024-04-14T15:11:16.297499+0000","last_undegraded":"2024-04-14T15:11:16.297499+0000","last_fullsized":"2024-04-14T15:11:16.297499+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:16.297432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:16.297432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:20.307091+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.940018+0000","last_change":"2024-04-14T15:09:43.022219+0000","last_active":"2024-04-14T15:11:00.940018+0000","last_peered":"2024-04-14T15:11:00.940018+0000","last_clean":"2024-04-14T15:11:00.940018+0000","last_became_active":"2024-04-14T15:09:43.021494+0000","last_became_peered":"2024-04-14T15:09:43.021494+0000","last_unstale":"2024-04-14T15:11:00.940018+0000","last_undegraded":"2024-04-14T15:11:00.940018+0000","last_fullsized":"2024-04-14T15:11:00.940018+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:09:42.004317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:02.243293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"72'15","reported_seq":84,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:18.790176+0000","last_change":"2024-04-14T15:11:18.790209+0000","last_active":"2024-04-14T15:11:18.790176+0000","last_peered":"2024-04-14T15:11:18.790176+0000","last_clean":"2024-04-14T15:11:18.790176+0000","last_became_active":"2024-04-14T15:09:45.032887+0000","last_became_peered":"2024-04-14T15:09:45.032887+0000","last_unstale":"2024-04-14T15:11:18.790176+0000","last_undegraded":"2024-04-14T15:11:18.790176+0000","last_fullsized":"2024-04-14T15:11:18.790176+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-04-14T15:11:18.790117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:18.790117+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:48.354430+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.763996+0000","last_change":"2024-04-14T15:09:49.057849+0000","last_active":"2024-04-14T15:11:00.763996+0000","last_peered":"2024-04-14T15:11:00.763996+0000","last_clean":"2024-04-14T15:11:00.763996+0000","last_became_active":"2024-04-14T15:09:49.057545+0000","last_became_peered":"2024-04-14T15:09:49.057545+0000","last_unstale":"2024-04-14T15:11:00.763996+0000","last_undegraded":"2024-04-14T15:11:00.763996+0000","last_fullsized":"2024-04-14T15:11:00.763996+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":47,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.211109+0000","last_change":"2024-04-14T15:09:47.046035+0000","last_active":"2024-04-14T15:11:01.211109+0000","last_peered":"2024-04-14T15:11:01.211109+0000","last_clean":"2024-04-14T15:11:01.211109+0000","last_became_active":"2024-04-14T15:09:47.045730+0000","last_became_peered":"2024-04-14T15:09:47.045730+0000","last_unstale":"2024-04-14T15:11:01.211109+0000","last_undegraded":"2024-04-14T15:11:01.211109+0000","last_fullsized":"2024-04-14T15:11:01.211109+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:09:46.023663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:11.966020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"65'2","reported_seq":70,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764422+0000","last_change":"2024-04-14T15:10:57.406574+0000","last_active":"2024-04-14T15:11:00.764422+0000","last_peered":"2024-04-14T15:11:00.764422+0000","last_clean":"2024-04-14T15:11:00.764422+0000","last_became_active":"2024-04-14T15:09:43.020401+0000","last_became_peered":"2024-04-14T15:09:43.020401+0000","last_unstale":"2024-04-14T15:11:00.764422+0000","last_undegraded":"2024-04-14T15:11:00.764422+0000","last_fullsized":"2024-04-14T15:11:00.764422+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-04-14T15:10:57.406478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:10:57.406478+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:25.577150+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"72'11","reported_seq":88,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.939685+0000","last_change":"2024-04-14T15:10:07.771479+0000","last_active":"2024-04-14T15:11:00.939685+0000","last_peered":"2024-04-14T15:11:00.939685+0000","last_clean":"2024-04-14T15:11:00.939685+0000","last_became_active":"2024-04-14T15:10:07.764183+0000","last_became_peered":"2024-04-14T15:10:07.764183+0000","last_unstale":"2024-04-14T15:11:00.939685+0000","last_undegraded":"2024-04-14T15:11:00.939685+0000","last_fullsized":"2024-04-14T15:11:00.939685+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:13.054416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"72'1","reported_seq":52,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.938612+0000","last_change":"2024-04-14T15:10:59.889342+0000","last_active":"2024-04-14T15:11:00.938612+0000","last_peered":"2024-04-14T15:11:00.938612+0000","last_clean":"2024-04-14T15:11:00.938612+0000","last_became_active":"2024-04-14T15:09:49.062051+0000","last_became_peered":"2024-04-14T15:09:49.062051+0000","last_unstale":"2024-04-14T15:11:00.938612+0000","last_undegraded":"2024-04-14T15:11:00.938612+0000","last_fullsized":"2024-04-14T15:11:00.938612+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-04-14T15:10:59.889236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:10:59.889236+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:03.689276+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"72'8","reported_seq":86,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:13.084616+0000","last_change":"2024-04-14T15:10:58.268612+0000","last_active":"2024-04-14T15:11:13.084616+0000","last_peered":"2024-04-14T15:11:13.084616+0000","last_clean":"2024-04-14T15:11:13.084616+0000","last_became_active":"2024-04-14T15:09:47.048216+0000","last_became_peered":"2024-04-14T15:09:47.048216+0000","last_unstale":"2024-04-14T15:11:13.084616+0000","last_undegraded":"2024-04-14T15:11:13.084616+0000","last_fullsized":"2024-04-14T15:11:13.084616+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:10:58.268488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:10:58.268488+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:04.149311+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":63,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.210561+0000","last_change":"2024-04-14T15:10:52.882061+0000","last_active":"2024-04-14T15:11:01.210561+0000","last_peered":"2024-04-14T15:11:01.210561+0000","last_clean":"2024-04-14T15:11:01.210561+0000","last_became_active":"2024-04-14T15:09:43.023257+0000","last_became_peered":"2024-04-14T15:09:43.023257+0000","last_unstale":"2024-04-14T15:11:01.210561+0000","last_undegraded":"2024-04-14T15:11:01.210561+0000","last_fullsized":"2024-04-14T15:11:01.210561+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:52.881916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:10:52.881916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:58.458431+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"72'10","reported_seq":74,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:17.863713+0000","last_change":"2024-04-14T15:11:17.863747+0000","last_active":"2024-04-14T15:11:17.863713+0000","last_peered":"2024-04-14T15:11:17.863713+0000","last_clean":"2024-04-14T15:11:17.863713+0000","last_became_active":"2024-04-14T15:09:45.181456+0000","last_became_peered":"2024-04-14T15:09:45.181456+0000","last_unstale":"2024-04-14T15:11:17.863713+0000","last_undegraded":"2024-04-14T15:11:17.863713+0000","last_fullsized":"2024-04-14T15:11:17.863713+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-04-14T15:11:17.863645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:17.863645+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:27.587257+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":51,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:17.499776+0000","last_change":"2024-04-14T15:11:17.499797+0000","last_active":"2024-04-14T15:11:17.499776+0000","last_peered":"2024-04-14T15:11:17.499776+0000","last_clean":"2024-04-14T15:11:17.499776+0000","last_became_active":"2024-04-14T15:09:49.129392+0000","last_became_peered":"2024-04-14T15:09:49.129392+0000","last_unstale":"2024-04-14T15:11:17.499776+0000","last_undegraded":"2024-04-14T15:11:17.499776+0000","last_fullsized":"2024-04-14T15:11:17.499776+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:17.499708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:17.499708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:24.507730+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:05.917034+0000","last_change":"2024-04-14T15:11:05.917054+0000","last_active":"2024-04-14T15:11:05.917034+0000","last_peered":"2024-04-14T15:11:05.917034+0000","last_clean":"2024-04-14T15:11:05.917034+0000","last_became_active":"2024-04-14T15:09:47.044665+0000","last_became_peered":"2024-04-14T15:09:47.044665+0000","last_unstale":"2024-04-14T15:11:05.917034+0000","last_undegraded":"2024-04-14T15:11:05.917034+0000","last_fullsized":"2024-04-14T15:11:05.917034+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:05.916966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:05.916966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:35.330112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":63,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.939825+0000","last_change":"2024-04-14T15:10:54.239518+0000","last_active":"2024-04-14T15:11:00.939825+0000","last_peered":"2024-04-14T15:11:00.939825+0000","last_clean":"2024-04-14T15:11:00.939825+0000","last_became_active":"2024-04-14T15:09:43.023220+0000","last_became_peered":"2024-04-14T15:09:43.023220+0000","last_unstale":"2024-04-14T15:11:00.939825+0000","last_undegraded":"2024-04-14T15:11:00.939825+0000","last_fullsized":"2024-04-14T15:11:00.939825+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:54.239409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:10:54.239409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:11.539313+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"72'11","reported_seq":78,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:06.492751+0000","last_change":"2024-04-14T15:11:06.492784+0000","last_active":"2024-04-14T15:11:06.492751+0000","last_peered":"2024-04-14T15:11:06.492751+0000","last_clean":"2024-04-14T15:11:06.492751+0000","last_became_active":"2024-04-14T15:09:45.180332+0000","last_became_peered":"2024-04-14T15:09:45.180332+0000","last_unstale":"2024-04-14T15:11:06.492751+0000","last_undegraded":"2024-04-14T15:11:06.492751+0000","last_fullsized":"2024-04-14T15:11:06.492751+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-04-14T15:11:06.492677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:06.492677+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:24.577564+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":51,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:07.901818+0000","last_change":"2024-04-14T15:11:07.901838+0000","last_active":"2024-04-14T15:11:07.901818+0000","last_peered":"2024-04-14T15:11:07.901818+0000","last_clean":"2024-04-14T15:11:07.901818+0000","last_became_active":"2024-04-14T15:09:49.061359+0000","last_became_peered":"2024-04-14T15:09:49.061359+0000","last_unstale":"2024-04-14T15:11:07.901818+0000","last_undegraded":"2024-04-14T15:11:07.901818+0000","last_fullsized":"2024-04-14T15:11:07.901818+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:07.901761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:07.901761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:35.316850+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.939704+0000","last_change":"2024-04-14T15:10:50.229008+0000","last_active":"2024-04-14T15:11:00.939704+0000","last_peered":"2024-04-14T15:11:00.939704+0000","last_clean":"2024-04-14T15:11:00.939704+0000","last_became_active":"2024-04-14T15:09:47.041074+0000","last_became_peered":"2024-04-14T15:09:47.041074+0000","last_unstale":"2024-04-14T15:11:00.939704+0000","last_undegraded":"2024-04-14T15:11:00.939704+0000","last_fullsized":"2024-04-14T15:11:00.939704+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:50.228903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:10:50.228903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:17.158842+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":63,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:08.712389+0000","last_change":"2024-04-14T15:11:08.712429+0000","last_active":"2024-04-14T15:11:08.712389+0000","last_peered":"2024-04-14T15:11:08.712389+0000","last_clean":"2024-04-14T15:11:08.712389+0000","last_became_active":"2024-04-14T15:09:43.026786+0000","last_became_peered":"2024-04-14T15:09:43.026786+0000","last_unstale":"2024-04-14T15:11:08.712389+0000","last_undegraded":"2024-04-14T15:11:08.712389+0000","last_fullsized":"2024-04-14T15:11:08.712389+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:08.712289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:08.712289+0000","last_clean_scrub_stamp":"2024-04-14T15:11:08.712289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:11.190004+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"72'4","reported_seq":57,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764146+0000","last_change":"2024-04-14T15:09:45.039145+0000","last_active":"2024-04-14T15:11:00.764146+0000","last_peered":"2024-04-14T15:11:00.764146+0000","last_clean":"2024-04-14T15:11:00.764146+0000","last_became_active":"2024-04-14T15:09:45.038815+0000","last_became_peered":"2024-04-14T15:09:45.038815+0000","last_unstale":"2024-04-14T15:11:00.764146+0000","last_undegraded":"2024-04-14T15:11:00.764146+0000","last_fullsized":"2024-04-14T15:11:00.764146+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:13.528834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764325+0000","last_change":"2024-04-14T15:09:49.066223+0000","last_active":"2024-04-14T15:11:00.764325+0000","last_peered":"2024-04-14T15:11:00.764325+0000","last_clean":"2024-04-14T15:11:00.764325+0000","last_became_active":"2024-04-14T15:09:49.065896+0000","last_became_peered":"2024-04-14T15:09:49.065896+0000","last_unstale":"2024-04-14T15:11:00.764325+0000","last_undegraded":"2024-04-14T15:11:00.764325+0000","last_fullsized":"2024-04-14T15:11:00.764325+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:09.025179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":47,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764746+0000","last_change":"2024-04-14T15:09:47.041972+0000","last_active":"2024-04-14T15:11:00.764746+0000","last_peered":"2024-04-14T15:11:00.764746+0000","last_clean":"2024-04-14T15:11:00.764746+0000","last_became_active":"2024-04-14T15:09:47.041683+0000","last_became_peered":"2024-04-14T15:09:47.041683+0000","last_unstale":"2024-04-14T15:11:00.764746+0000","last_undegraded":"2024-04-14T15:11:00.764746+0000","last_fullsized":"2024-04-14T15:11:00.764746+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:09:46.023663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:08.100556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764274+0000","last_change":"2024-04-14T15:09:43.023867+0000","last_active":"2024-04-14T15:11:00.764274+0000","last_peered":"2024-04-14T15:11:00.764274+0000","last_clean":"2024-04-14T15:11:00.764274+0000","last_became_active":"2024-04-14T15:09:43.023583+0000","last_became_peered":"2024-04-14T15:09:43.023583+0000","last_unstale":"2024-04-14T15:11:00.764274+0000","last_undegraded":"2024-04-14T15:11:00.764274+0000","last_fullsized":"2024-04-14T15:11:00.764274+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:09:42.004317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:05.031251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"72'11","reported_seq":70,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764103+0000","last_change":"2024-04-14T15:09:45.026357+0000","last_active":"2024-04-14T15:11:00.764103+0000","last_peered":"2024-04-14T15:11:00.764103+0000","last_clean":"2024-04-14T15:11:00.764103+0000","last_became_active":"2024-04-14T15:09:45.026065+0000","last_became_peered":"2024-04-14T15:09:45.026065+0000","last_unstale":"2024-04-14T15:11:00.764103+0000","last_undegraded":"2024-04-14T15:11:00.764103+0000","last_fullsized":"2024-04-14T15:11:00.764103+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:08.741092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":51,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.209828+0000","last_change":"2024-04-14T15:10:56.820210+0000","last_active":"2024-04-14T15:11:01.209828+0000","last_peered":"2024-04-14T15:11:01.209828+0000","last_clean":"2024-04-14T15:11:01.209828+0000","last_became_active":"2024-04-14T15:09:49.059711+0000","last_became_peered":"2024-04-14T15:09:49.059711+0000","last_unstale":"2024-04-14T15:11:01.209828+0000","last_undegraded":"2024-04-14T15:11:01.209828+0000","last_fullsized":"2024-04-14T15:11:01.209828+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:56.820123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:10:56.820123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:08.868464+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:10.455387+0000","last_change":"2024-04-14T15:11:10.455426+0000","last_active":"2024-04-14T15:11:10.455387+0000","last_peered":"2024-04-14T15:11:10.455387+0000","last_clean":"2024-04-14T15:11:10.455387+0000","last_became_active":"2024-04-14T15:09:47.044891+0000","last_became_peered":"2024-04-14T15:09:47.044891+0000","last_unstale":"2024-04-14T15:11:10.455387+0000","last_undegraded":"2024-04-14T15:11:10.455387+0000","last_fullsized":"2024-04-14T15:11:10.455387+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:10.455300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:10.455300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:32.884911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":63,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:03.304615+0000","last_change":"2024-04-14T15:11:03.304640+0000","last_active":"2024-04-14T15:11:03.304615+0000","last_peered":"2024-04-14T15:11:03.304615+0000","last_clean":"2024-04-14T15:11:03.304615+0000","last_became_active":"2024-04-14T15:09:43.022480+0000","last_became_peered":"2024-04-14T15:09:43.022480+0000","last_unstale":"2024-04-14T15:11:03.304615+0000","last_undegraded":"2024-04-14T15:11:03.304615+0000","last_fullsized":"2024-04-14T15:11:03.304615+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:03.304541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:03.304541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:22.825961+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"72'9","reported_seq":67,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764699+0000","last_change":"2024-04-14T15:09:45.026649+0000","last_active":"2024-04-14T15:11:00.764699+0000","last_peered":"2024-04-14T15:11:00.764699+0000","last_clean":"2024-04-14T15:11:00.764699+0000","last_became_active":"2024-04-14T15:09:45.026363+0000","last_became_peered":"2024-04-14T15:09:45.026363+0000","last_unstale":"2024-04-14T15:11:00.764699+0000","last_undegraded":"2024-04-14T15:11:00.764699+0000","last_fullsized":"2024-04-14T15:11:00.764699+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:03.421380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":51,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:15.644773+0000","last_change":"2024-04-14T15:11:15.644792+0000","last_active":"2024-04-14T15:11:15.644773+0000","last_peered":"2024-04-14T15:11:15.644773+0000","last_clean":"2024-04-14T15:11:15.644773+0000","last_became_active":"2024-04-14T15:09:49.065242+0000","last_became_peered":"2024-04-14T15:09:49.065242+0000","last_unstale":"2024-04-14T15:11:15.644773+0000","last_undegraded":"2024-04-14T15:11:15.644773+0000","last_fullsized":"2024-04-14T15:11:15.644773+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:15.644714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:15.644714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:19.718378+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"72'8","reported_seq":86,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:13.084614+0000","last_change":"2024-04-14T15:11:09.485228+0000","last_active":"2024-04-14T15:11:13.084614+0000","last_peered":"2024-04-14T15:11:13.084614+0000","last_clean":"2024-04-14T15:11:13.084614+0000","last_became_active":"2024-04-14T15:09:47.047072+0000","last_became_peered":"2024-04-14T15:09:47.047072+0000","last_unstale":"2024-04-14T15:11:13.084614+0000","last_undegraded":"2024-04-14T15:11:13.084614+0000","last_fullsized":"2024-04-14T15:11:13.084614+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:11:09.485146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:09.485146+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:34.775008+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":63,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:10.762142+0000","last_change":"2024-04-14T15:11:10.762163+0000","last_active":"2024-04-14T15:11:10.762142+0000","last_peered":"2024-04-14T15:11:10.762142+0000","last_clean":"2024-04-14T15:11:10.762142+0000","last_became_active":"2024-04-14T15:09:43.019301+0000","last_became_peered":"2024-04-14T15:09:43.019301+0000","last_unstale":"2024-04-14T15:11:10.762142+0000","last_undegraded":"2024-04-14T15:11:10.762142+0000","last_fullsized":"2024-04-14T15:11:10.762142+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:10.762080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:10.762080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:13.869531+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"72'9","reported_seq":75,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.211409+0000","last_change":"2024-04-14T15:10:56.858475+0000","last_active":"2024-04-14T15:11:01.211409+0000","last_peered":"2024-04-14T15:11:01.211409+0000","last_clean":"2024-04-14T15:11:01.211409+0000","last_became_active":"2024-04-14T15:09:45.178857+0000","last_became_peered":"2024-04-14T15:09:45.178857+0000","last_unstale":"2024-04-14T15:11:01.211409+0000","last_undegraded":"2024-04-14T15:11:01.211409+0000","last_fullsized":"2024-04-14T15:11:01.211409+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-04-14T15:10:56.858323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:10:56.858323+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:13.702279+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.209643+0000","last_change":"2024-04-14T15:09:49.066448+0000","last_active":"2024-04-14T15:11:01.209643+0000","last_peered":"2024-04-14T15:11:01.209643+0000","last_clean":"2024-04-14T15:11:01.209643+0000","last_became_active":"2024-04-14T15:09:49.066068+0000","last_became_peered":"2024-04-14T15:09:49.066068+0000","last_unstale":"2024-04-14T15:11:01.209643+0000","last_undegraded":"2024-04-14T15:11:01.209643+0000","last_fullsized":"2024-04-14T15:11:01.209643+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:14.252283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"72'8","reported_seq":83,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:18.667170+0000","last_change":"2024-04-14T15:11:18.667210+0000","last_active":"2024-04-14T15:11:18.667170+0000","last_peered":"2024-04-14T15:11:18.667170+0000","last_clean":"2024-04-14T15:11:18.667170+0000","last_became_active":"2024-04-14T15:09:47.036688+0000","last_became_peered":"2024-04-14T15:09:47.036688+0000","last_unstale":"2024-04-14T15:11:18.667170+0000","last_undegraded":"2024-04-14T15:11:18.667170+0000","last_fullsized":"2024-04-14T15:11:18.667170+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:11:18.667049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:18.667049+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:36.115547+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.210693+0000","last_change":"2024-04-14T15:09:43.028602+0000","last_active":"2024-04-14T15:11:01.210693+0000","last_peered":"2024-04-14T15:11:01.210693+0000","last_clean":"2024-04-14T15:11:01.210693+0000","last_became_active":"2024-04-14T15:09:43.027776+0000","last_became_peered":"2024-04-14T15:09:43.027776+0000","last_unstale":"2024-04-14T15:11:01.210693+0000","last_undegraded":"2024-04-14T15:11:01.210693+0000","last_fullsized":"2024-04-14T15:11:01.210693+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:09:42.004317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:01.870579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"72'10","reported_seq":74,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:11.458657+0000","last_change":"2024-04-14T15:11:11.458687+0000","last_active":"2024-04-14T15:11:11.458657+0000","last_peered":"2024-04-14T15:11:11.458657+0000","last_clean":"2024-04-14T15:11:11.458657+0000","last_became_active":"2024-04-14T15:09:45.179165+0000","last_became_peered":"2024-04-14T15:09:45.179165+0000","last_unstale":"2024-04-14T15:11:11.458657+0000","last_undegraded":"2024-04-14T15:11:11.458657+0000","last_fullsized":"2024-04-14T15:11:11.458657+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-04-14T15:11:11.458594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:11.458594+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:19.766366+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.938088+0000","last_change":"2024-04-14T15:09:49.130725+0000","last_active":"2024-04-14T15:11:00.938088+0000","last_peered":"2024-04-14T15:11:00.938088+0000","last_clean":"2024-04-14T15:11:00.938088+0000","last_became_active":"2024-04-14T15:09:49.130438+0000","last_became_peered":"2024-04-14T15:09:49.130438+0000","last_unstale":"2024-04-14T15:11:00.938088+0000","last_undegraded":"2024-04-14T15:11:00.938088+0000","last_fullsized":"2024-04-14T15:11:00.938088+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:16.730190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.211603+0000","last_change":"2024-04-14T15:10:53.900979+0000","last_active":"2024-04-14T15:11:01.211603+0000","last_peered":"2024-04-14T15:11:01.211603+0000","last_clean":"2024-04-14T15:11:01.211603+0000","last_became_active":"2024-04-14T15:09:47.045349+0000","last_became_peered":"2024-04-14T15:09:47.045349+0000","last_unstale":"2024-04-14T15:11:01.211603+0000","last_undegraded":"2024-04-14T15:11:01.211603+0000","last_fullsized":"2024-04-14T15:11:01.211603+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:53.900878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:10:53.900878+0000","last_clean_scrub_stamp":"2024-04-14T15:10:53.900878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:12.123204+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":63,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:18.194409+0000","last_change":"2024-04-14T15:11:18.194434+0000","last_active":"2024-04-14T15:11:18.194409+0000","last_peered":"2024-04-14T15:11:18.194409+0000","last_clean":"2024-04-14T15:11:18.194409+0000","last_became_active":"2024-04-14T15:09:43.018987+0000","last_became_peered":"2024-04-14T15:09:43.018987+0000","last_unstale":"2024-04-14T15:11:18.194409+0000","last_undegraded":"2024-04-14T15:11:18.194409+0000","last_fullsized":"2024-04-14T15:11:18.194409+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:18.194339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:18.194339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:27.683203+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"72'6","reported_seq":60,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.763707+0000","last_change":"2024-04-14T15:09:45.180445+0000","last_active":"2024-04-14T15:11:00.763707+0000","last_peered":"2024-04-14T15:11:00.763707+0000","last_clean":"2024-04-14T15:11:00.763707+0000","last_became_active":"2024-04-14T15:09:45.180037+0000","last_became_peered":"2024-04-14T15:09:45.180037+0000","last_unstale":"2024-04-14T15:11:00.763707+0000","last_undegraded":"2024-04-14T15:11:00.763707+0000","last_fullsized":"2024-04-14T15:11:00.763707+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:08.718920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":51,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.210053+0000","last_change":"2024-04-14T15:10:56.886642+0000","last_active":"2024-04-14T15:11:01.210053+0000","last_peered":"2024-04-14T15:11:01.210053+0000","last_clean":"2024-04-14T15:11:01.210053+0000","last_became_active":"2024-04-14T15:09:49.064296+0000","last_became_peered":"2024-04-14T15:09:49.064296+0000","last_unstale":"2024-04-14T15:11:01.210053+0000","last_undegraded":"2024-04-14T15:11:01.210053+0000","last_fullsized":"2024-04-14T15:11:01.210053+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:56.886513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:10:56.886513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:17.056264+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:07.485898+0000","last_change":"2024-04-14T15:11:07.485922+0000","last_active":"2024-04-14T15:11:07.485898+0000","last_peered":"2024-04-14T15:11:07.485898+0000","last_clean":"2024-04-14T15:11:07.485898+0000","last_became_active":"2024-04-14T15:09:47.044816+0000","last_became_peered":"2024-04-14T15:09:47.044816+0000","last_unstale":"2024-04-14T15:11:07.485898+0000","last_undegraded":"2024-04-14T15:11:07.485898+0000","last_fullsized":"2024-04-14T15:11:07.485898+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:07.485821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:07.485821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:21.021299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":63,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.210752+0000","last_change":"2024-04-14T15:10:43.844823+0000","last_active":"2024-04-14T15:11:01.210752+0000","last_peered":"2024-04-14T15:11:01.210752+0000","last_clean":"2024-04-14T15:11:01.210752+0000","last_became_active":"2024-04-14T15:09:43.016788+0000","last_became_peered":"2024-04-14T15:09:43.016788+0000","last_unstale":"2024-04-14T15:11:01.210752+0000","last_undegraded":"2024-04-14T15:11:01.210752+0000","last_fullsized":"2024-04-14T15:11:01.210752+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:43.844693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:10:43.844693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:51.339452+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"72'9","reported_seq":67,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.210984+0000","last_change":"2024-04-14T15:09:45.180769+0000","last_active":"2024-04-14T15:11:01.210984+0000","last_peered":"2024-04-14T15:11:01.210984+0000","last_clean":"2024-04-14T15:11:01.210984+0000","last_became_active":"2024-04-14T15:09:45.180375+0000","last_became_peered":"2024-04-14T15:09:45.180375+0000","last_unstale":"2024-04-14T15:11:01.210984+0000","last_undegraded":"2024-04-14T15:11:01.210984+0000","last_fullsized":"2024-04-14T15:11:01.210984+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:12.591629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764186+0000","last_change":"2024-04-14T15:09:49.069204+0000","last_active":"2024-04-14T15:11:00.764186+0000","last_peered":"2024-04-14T15:11:00.764186+0000","last_clean":"2024-04-14T15:11:00.764186+0000","last_became_active":"2024-04-14T15:09:49.068883+0000","last_became_peered":"2024-04-14T15:09:49.068883+0000","last_unstale":"2024-04-14T15:11:00.764186+0000","last_undegraded":"2024-04-14T15:11:00.764186+0000","last_fullsized":"2024-04-14T15:11:00.764186+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:12.232108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"65'3","reported_seq":73,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764604+0000","last_change":"2024-04-14T15:09:43.020827+0000","last_active":"2024-04-14T15:11:00.764604+0000","last_peered":"2024-04-14T15:11:00.764604+0000","last_clean":"2024-04-14T15:11:00.764604+0000","last_became_active":"2024-04-14T15:09:43.020490+0000","last_became_peered":"2024-04-14T15:09:43.020490+0000","last_unstale":"2024-04-14T15:11:00.764604+0000","last_undegraded":"2024-04-14T15:11:00.764604+0000","last_fullsized":"2024-04-14T15:11:00.764604+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:09:42.004317+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:07.143386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":528,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"72'12","reported_seq":82,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.763942+0000","last_change":"2024-04-14T15:10:53.448340+0000","last_active":"2024-04-14T15:11:00.763942+0000","last_peered":"2024-04-14T15:11:00.763942+0000","last_clean":"2024-04-14T15:11:00.763942+0000","last_became_active":"2024-04-14T15:09:45.032892+0000","last_became_peered":"2024-04-14T15:09:45.032892+0000","last_unstale":"2024-04-14T15:11:00.763942+0000","last_undegraded":"2024-04-14T15:11:00.763942+0000","last_fullsized":"2024-04-14T15:11:00.763942+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'12","last_scrub_stamp":"2024-04-14T15:10:53.448218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:10:53.448218+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:56.821958+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:12.867579+0000","last_change":"2024-04-14T15:11:12.867617+0000","last_active":"2024-04-14T15:11:12.867579+0000","last_peered":"2024-04-14T15:11:12.867579+0000","last_clean":"2024-04-14T15:11:12.867579+0000","last_became_active":"2024-04-14T15:09:47.049005+0000","last_became_peered":"2024-04-14T15:09:47.049005+0000","last_unstale":"2024-04-14T15:11:12.867579+0000","last_undegraded":"2024-04-14T15:11:12.867579+0000","last_fullsized":"2024-04-14T15:11:12.867579+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:12.867489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:12.867489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:42.053319+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":51,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:09.756221+0000","last_change":"2024-04-14T15:11:09.756246+0000","last_active":"2024-04-14T15:11:09.756221+0000","last_peered":"2024-04-14T15:11:09.756221+0000","last_clean":"2024-04-14T15:11:09.756221+0000","last_became_active":"2024-04-14T15:09:49.066648+0000","last_became_peered":"2024-04-14T15:09:49.066648+0000","last_unstale":"2024-04-14T15:11:09.756221+0000","last_undegraded":"2024-04-14T15:11:09.756221+0000","last_fullsized":"2024-04-14T15:11:09.756221+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:09.756164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:09.756164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:32.890427+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.763844+0000","last_change":"2024-04-14T15:09:43.022909+0000","last_active":"2024-04-14T15:11:00.763844+0000","last_peered":"2024-04-14T15:11:00.763844+0000","last_clean":"2024-04-14T15:11:00.763844+0000","last_became_active":"2024-04-14T15:09:43.022597+0000","last_became_peered":"2024-04-14T15:09:43.022597+0000","last_unstale":"2024-04-14T15:11:00.763844+0000","last_undegraded":"2024-04-14T15:11:00.763844+0000","last_fullsized":"2024-04-14T15:11:00.763844+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:09:42.004317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:11.088019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"72'10","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:17.779479+0000","last_change":"2024-04-14T15:11:17.779528+0000","last_active":"2024-04-14T15:11:17.779479+0000","last_peered":"2024-04-14T15:11:17.779479+0000","last_clean":"2024-04-14T15:11:17.779479+0000","last_became_active":"2024-04-14T15:10:07.764042+0000","last_became_peered":"2024-04-14T15:10:07.764042+0000","last_unstale":"2024-04-14T15:11:17.779479+0000","last_undegraded":"2024-04-14T15:11:17.779479+0000","last_fullsized":"2024-04-14T15:11:17.779479+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-04-14T15:11:17.779383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:17.779383+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:33.747601+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"72'8","reported_seq":83,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:14.870904+0000","last_change":"2024-04-14T15:11:14.870904+0000","last_active":"2024-04-14T15:11:14.870904+0000","last_peered":"2024-04-14T15:11:14.870904+0000","last_clean":"2024-04-14T15:11:14.870904+0000","last_became_active":"2024-04-14T15:09:47.047699+0000","last_became_peered":"2024-04-14T15:09:47.047699+0000","last_unstale":"2024-04-14T15:11:14.870904+0000","last_undegraded":"2024-04-14T15:11:14.870904+0000","last_fullsized":"2024-04-14T15:11:14.870904+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:11:14.870772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:14.870772+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:19.702141+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.765065+0000","last_change":"2024-04-14T15:09:49.058715+0000","last_active":"2024-04-14T15:11:00.765065+0000","last_peered":"2024-04-14T15:11:00.765065+0000","last_clean":"2024-04-14T15:11:00.765065+0000","last_became_active":"2024-04-14T15:09:49.058387+0000","last_became_peered":"2024-04-14T15:09:49.058387+0000","last_unstale":"2024-04-14T15:11:00.765065+0000","last_undegraded":"2024-04-14T15:11:00.765065+0000","last_fullsized":"2024-04-14T15:11:00.765065+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:16.162642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.210796+0000","last_change":"2024-04-14T15:09:43.028748+0000","last_active":"2024-04-14T15:11:01.210796+0000","last_peered":"2024-04-14T15:11:01.210796+0000","last_clean":"2024-04-14T15:11:01.210796+0000","last_became_active":"2024-04-14T15:09:43.028362+0000","last_became_peered":"2024-04-14T15:09:43.028362+0000","last_unstale":"2024-04-14T15:11:01.210796+0000","last_undegraded":"2024-04-14T15:11:01.210796+0000","last_fullsized":"2024-04-14T15:11:01.210796+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:09:42.004317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"72'28","reported_seq":119,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:13.084520+0000","last_change":"2024-04-14T15:09:45.034501+0000","last_active":"2024-04-14T15:11:13.084520+0000","last_peered":"2024-04-14T15:11:13.084520+0000","last_clean":"2024-04-14T15:11:13.084520+0000","last_became_active":"2024-04-14T15:09:45.034168+0000","last_became_peered":"2024-04-14T15:09:45.034168+0000","last_unstale":"2024-04-14T15:11:13.084520+0000","last_undegraded":"2024-04-14T15:11:13.084520+0000","last_fullsized":"2024-04-14T15:11:13.084520+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:15.831871+0000","last_change":"2024-04-14T15:11:15.831911+0000","last_active":"2024-04-14T15:11:15.831871+0000","last_peered":"2024-04-14T15:11:15.831871+0000","last_clean":"2024-04-14T15:11:15.831871+0000","last_became_active":"2024-04-14T15:09:47.040185+0000","last_became_peered":"2024-04-14T15:09:47.040185+0000","last_unstale":"2024-04-14T15:11:15.831871+0000","last_undegraded":"2024-04-14T15:11:15.831871+0000","last_fullsized":"2024-04-14T15:11:15.831871+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:15.831779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:15.831779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:44.452408+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":51,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.939013+0000","last_change":"2024-04-14T15:11:00.939052+0000","last_active":"2024-04-14T15:11:00.939013+0000","last_peered":"2024-04-14T15:11:00.939013+0000","last_clean":"2024-04-14T15:11:00.939013+0000","last_became_active":"2024-04-14T15:09:49.130307+0000","last_became_peered":"2024-04-14T15:09:49.130307+0000","last_unstale":"2024-04-14T15:11:00.939013+0000","last_undegraded":"2024-04-14T15:11:00.939013+0000","last_fullsized":"2024-04-14T15:11:00.939013+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:00.938917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:00.938917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:17.427163+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"73'5","reported_seq":163,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:17.776950+0000","last_change":"2024-04-14T15:11:03.192957+0000","last_active":"2024-04-14T15:11:17.776950+0000","last_peered":"2024-04-14T15:11:17.776950+0000","last_clean":"2024-04-14T15:11:17.776950+0000","last_became_active":"2024-04-14T15:09:43.016544+0000","last_became_peered":"2024-04-14T15:09:43.016544+0000","last_unstale":"2024-04-14T15:11:17.776950+0000","last_undegraded":"2024-04-14T15:11:17.776950+0000","last_fullsized":"2024-04-14T15:11:17.776950+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-04-14T15:11:03.192810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:03.192810+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:21.177179+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00056542099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":90,"num_read_kb":85,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":63,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.209765+0000","last_change":"2024-04-14T15:10:42.819764+0000","last_active":"2024-04-14T15:11:01.209765+0000","last_peered":"2024-04-14T15:11:01.209765+0000","last_clean":"2024-04-14T15:11:01.209765+0000","last_became_active":"2024-04-14T15:09:43.021188+0000","last_became_peered":"2024-04-14T15:09:43.021188+0000","last_unstale":"2024-04-14T15:11:01.209765+0000","last_undegraded":"2024-04-14T15:11:01.209765+0000","last_fullsized":"2024-04-14T15:11:01.209765+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:42.819681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:10:42.819681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:52.363715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"72'13","reported_seq":78,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764908+0000","last_change":"2024-04-14T15:09:45.034537+0000","last_active":"2024-04-14T15:11:00.764908+0000","last_peered":"2024-04-14T15:11:00.764908+0000","last_clean":"2024-04-14T15:11:00.764908+0000","last_became_active":"2024-04-14T15:09:45.033883+0000","last_became_peered":"2024-04-14T15:09:45.033883+0000","last_unstale":"2024-04-14T15:11:00.764908+0000","last_undegraded":"2024-04-14T15:11:00.764908+0000","last_fullsized":"2024-04-14T15:11:00.764908+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:04.692317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.210758+0000","last_change":"2024-04-14T15:10:55.841920+0000","last_active":"2024-04-14T15:11:01.210758+0000","last_peered":"2024-04-14T15:11:01.210758+0000","last_clean":"2024-04-14T15:11:01.210758+0000","last_became_active":"2024-04-14T15:09:47.049457+0000","last_became_peered":"2024-04-14T15:09:47.049457+0000","last_unstale":"2024-04-14T15:11:01.210758+0000","last_undegraded":"2024-04-14T15:11:01.210758+0000","last_fullsized":"2024-04-14T15:11:01.210758+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:55.841832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:10:55.841832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:09.497072+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.210445+0000","last_change":"2024-04-14T15:09:49.067392+0000","last_active":"2024-04-14T15:11:01.210445+0000","last_peered":"2024-04-14T15:11:01.210445+0000","last_clean":"2024-04-14T15:11:01.210445+0000","last_became_active":"2024-04-14T15:09:49.067031+0000","last_became_peered":"2024-04-14T15:09:49.067031+0000","last_unstale":"2024-04-14T15:11:01.210445+0000","last_undegraded":"2024-04-14T15:11:01.210445+0000","last_fullsized":"2024-04-14T15:11:01.210445+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:13.012527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"69'2","reported_seq":64,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.763287+0000","last_change":"2024-04-14T15:09:47.043261+0000","last_active":"2024-04-14T15:11:00.763287+0000","last_peered":"2024-04-14T15:11:00.763287+0000","last_clean":"2024-04-14T15:11:00.763287+0000","last_became_active":"2024-04-14T15:09:43.024882+0000","last_became_peered":"2024-04-14T15:09:43.024882+0000","last_unstale":"2024-04-14T15:11:00.763287+0000","last_undegraded":"2024-04-14T15:11:00.763287+0000","last_fullsized":"2024-04-14T15:11:00.763287+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:09:42.004317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:03.751335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0058046169999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":63,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.211222+0000","last_change":"2024-04-14T15:10:43.898230+0000","last_active":"2024-04-14T15:11:01.211222+0000","last_peered":"2024-04-14T15:11:01.211222+0000","last_clean":"2024-04-14T15:11:01.211222+0000","last_became_active":"2024-04-14T15:09:43.025421+0000","last_became_peered":"2024-04-14T15:09:43.025421+0000","last_unstale":"2024-04-14T15:11:01.211222+0000","last_undegraded":"2024-04-14T15:11:01.211222+0000","last_fullsized":"2024-04-14T15:11:01.211222+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:43.898117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:10:43.898117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:47.080377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":304,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.937548+0000","last_change":"2024-04-14T15:10:06.160908+0000","last_active":"2024-04-14T15:11:00.937548+0000","last_peered":"2024-04-14T15:11:00.937548+0000","last_clean":"2024-04-14T15:11:00.937548+0000","last_became_active":"2024-04-14T15:09:33.753828+0000","last_became_peered":"2024-04-14T15:09:33.753828+0000","last_unstale":"2024-04-14T15:11:00.937548+0000","last_undegraded":"2024-04-14T15:11:00.937548+0000","last_fullsized":"2024-04-14T15:11:00.937548+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'117","last_scrub_stamp":"2024-04-14T15:10:06.160794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:07:24.600370+0000","last_clean_scrub_stamp":"2024-04-14T15:10:06.160794+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:17.821751+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"72'16","reported_seq":93,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:13.084671+0000","last_change":"2024-04-14T15:09:45.180279+0000","last_active":"2024-04-14T15:11:13.084671+0000","last_peered":"2024-04-14T15:11:13.084671+0000","last_clean":"2024-04-14T15:11:13.084671+0000","last_became_active":"2024-04-14T15:09:45.179978+0000","last_became_peered":"2024-04-14T15:09:45.179978+0000","last_unstale":"2024-04-14T15:11:13.084671+0000","last_undegraded":"2024-04-14T15:11:13.084671+0000","last_fullsized":"2024-04-14T15:11:13.084671+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:13.892928+0000","last_change":"2024-04-14T15:11:13.892949+0000","last_active":"2024-04-14T15:11:13.892928+0000","last_peered":"2024-04-14T15:11:13.892928+0000","last_clean":"2024-04-14T15:11:13.892928+0000","last_became_active":"2024-04-14T15:09:47.047201+0000","last_became_peered":"2024-04-14T15:09:47.047201+0000","last_unstale":"2024-04-14T15:11:13.892928+0000","last_undegraded":"2024-04-14T15:11:13.892928+0000","last_fullsized":"2024-04-14T15:11:13.892928+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:13.892863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:13.892863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:41.896681+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"72'1","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.763434+0000","last_change":"2024-04-14T15:09:49.065706+0000","last_active":"2024-04-14T15:11:00.763434+0000","last_peered":"2024-04-14T15:11:00.763434+0000","last_clean":"2024-04-14T15:11:00.763434+0000","last_became_active":"2024-04-14T15:09:49.065243+0000","last_became_peered":"2024-04-14T15:09:49.065243+0000","last_unstale":"2024-04-14T15:11:00.763434+0000","last_undegraded":"2024-04-14T15:11:00.763434+0000","last_fullsized":"2024-04-14T15:11:00.763434+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":70,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:07.696587+0000","last_change":"2024-04-14T15:11:07.696609+0000","last_active":"2024-04-14T15:11:07.696587+0000","last_peered":"2024-04-14T15:11:07.696587+0000","last_clean":"2024-04-14T15:11:07.696587+0000","last_became_active":"2024-04-14T15:09:43.018509+0000","last_became_peered":"2024-04-14T15:09:43.018509+0000","last_unstale":"2024-04-14T15:11:07.696587+0000","last_undegraded":"2024-04-14T15:11:07.696587+0000","last_fullsized":"2024-04-14T15:11:07.696587+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:07.696525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:07.696525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:17.969989+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0018592680000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":63,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.937410+0000","last_change":"2024-04-14T15:10:49.916989+0000","last_active":"2024-04-14T15:11:00.937410+0000","last_peered":"2024-04-14T15:11:00.937410+0000","last_clean":"2024-04-14T15:11:00.937410+0000","last_became_active":"2024-04-14T15:09:43.025095+0000","last_became_peered":"2024-04-14T15:09:43.025095+0000","last_unstale":"2024-04-14T15:11:00.937410+0000","last_undegraded":"2024-04-14T15:11:00.937410+0000","last_fullsized":"2024-04-14T15:11:00.937410+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:49.916884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:10:49.916884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:11.267411+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"72'12","reported_seq":77,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.210892+0000","last_change":"2024-04-14T15:10:51.840577+0000","last_active":"2024-04-14T15:11:01.210892+0000","last_peered":"2024-04-14T15:11:01.210892+0000","last_clean":"2024-04-14T15:11:01.210892+0000","last_became_active":"2024-04-14T15:09:45.037231+0000","last_became_peered":"2024-04-14T15:09:45.037231+0000","last_unstale":"2024-04-14T15:11:01.210892+0000","last_undegraded":"2024-04-14T15:11:01.210892+0000","last_fullsized":"2024-04-14T15:11:01.210892+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'12","last_scrub_stamp":"2024-04-14T15:10:51.840487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:10:51.840487+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:52.131140+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":47,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.211771+0000","last_change":"2024-04-14T15:09:47.037005+0000","last_active":"2024-04-14T15:11:01.211771+0000","last_peered":"2024-04-14T15:11:01.211771+0000","last_clean":"2024-04-14T15:11:01.211771+0000","last_became_active":"2024-04-14T15:09:47.036696+0000","last_became_peered":"2024-04-14T15:09:47.036696+0000","last_unstale":"2024-04-14T15:11:01.211771+0000","last_undegraded":"2024-04-14T15:11:01.211771+0000","last_fullsized":"2024-04-14T15:11:01.211771+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:09:46.023663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:06.058177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.763009+0000","last_change":"2024-04-14T15:09:49.128490+0000","last_active":"2024-04-14T15:11:00.763009+0000","last_peered":"2024-04-14T15:11:00.763009+0000","last_clean":"2024-04-14T15:11:00.763009+0000","last_became_active":"2024-04-14T15:09:49.128136+0000","last_became_peered":"2024-04-14T15:09:49.128136+0000","last_unstale":"2024-04-14T15:11:00.763009+0000","last_undegraded":"2024-04-14T15:11:00.763009+0000","last_fullsized":"2024-04-14T15:11:00.763009+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:09.879878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":63,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.938596+0000","last_change":"2024-04-14T15:10:53.992688+0000","last_active":"2024-04-14T15:11:00.938596+0000","last_peered":"2024-04-14T15:11:00.938596+0000","last_clean":"2024-04-14T15:11:00.938596+0000","last_became_active":"2024-04-14T15:09:43.021006+0000","last_became_peered":"2024-04-14T15:09:43.021006+0000","last_unstale":"2024-04-14T15:11:00.938596+0000","last_undegraded":"2024-04-14T15:11:00.938596+0000","last_fullsized":"2024-04-14T15:11:00.938596+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:53.992590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:10:53.992590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:16.131673+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"72'19","reported_seq":95,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.210795+0000","last_change":"2024-04-14T15:10:47.880742+0000","last_active":"2024-04-14T15:11:01.210795+0000","last_peered":"2024-04-14T15:11:01.210795+0000","last_clean":"2024-04-14T15:11:01.210795+0000","last_became_active":"2024-04-14T15:09:45.179182+0000","last_became_peered":"2024-04-14T15:09:45.179182+0000","last_unstale":"2024-04-14T15:11:01.210795+0000","last_undegraded":"2024-04-14T15:11:01.210795+0000","last_fullsized":"2024-04-14T15:11:01.210795+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'19","last_scrub_stamp":"2024-04-14T15:10:47.880623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:10:47.880623+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:07.612241+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.940078+0000","last_change":"2024-04-14T15:10:59.286474+0000","last_active":"2024-04-14T15:11:00.940078+0000","last_peered":"2024-04-14T15:11:00.940078+0000","last_clean":"2024-04-14T15:11:00.940078+0000","last_became_active":"2024-04-14T15:09:47.048427+0000","last_became_peered":"2024-04-14T15:09:47.048427+0000","last_unstale":"2024-04-14T15:11:00.940078+0000","last_undegraded":"2024-04-14T15:11:00.940078+0000","last_fullsized":"2024-04-14T15:11:00.940078+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:59.286374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:10:59.286374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:59.951701+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764213+0000","last_change":"2024-04-14T15:09:49.065988+0000","last_active":"2024-04-14T15:11:00.764213+0000","last_peered":"2024-04-14T15:11:00.764213+0000","last_clean":"2024-04-14T15:11:00.764213+0000","last_became_active":"2024-04-14T15:09:49.065684+0000","last_became_peered":"2024-04-14T15:09:49.065684+0000","last_unstale":"2024-04-14T15:11:00.764213+0000","last_undegraded":"2024-04-14T15:11:00.764213+0000","last_fullsized":"2024-04-14T15:11:00.764213+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:07.917647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"65'1","reported_seq":56,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.763647+0000","last_change":"2024-04-14T15:09:43.019515+0000","last_active":"2024-04-14T15:11:00.763647+0000","last_peered":"2024-04-14T15:11:00.763647+0000","last_clean":"2024-04-14T15:11:00.763647+0000","last_became_active":"2024-04-14T15:09:43.018478+0000","last_became_peered":"2024-04-14T15:09:43.018478+0000","last_unstale":"2024-04-14T15:11:00.763647+0000","last_undegraded":"2024-04-14T15:11:00.763647+0000","last_fullsized":"2024-04-14T15:11:00.763647+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:09:42.004317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:10.202051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"72'18","reported_seq":91,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764001+0000","last_change":"2024-04-14T15:10:53.457240+0000","last_active":"2024-04-14T15:11:00.764001+0000","last_peered":"2024-04-14T15:11:00.764001+0000","last_clean":"2024-04-14T15:11:00.764001+0000","last_became_active":"2024-04-14T15:09:45.181252+0000","last_became_peered":"2024-04-14T15:09:45.181252+0000","last_unstale":"2024-04-14T15:11:00.764001+0000","last_undegraded":"2024-04-14T15:11:00.764001+0000","last_fullsized":"2024-04-14T15:11:00.764001+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'18","last_scrub_stamp":"2024-04-14T15:10:53.457087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:10:53.457087+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:00.641833+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":47,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764213+0000","last_change":"2024-04-14T15:09:47.044907+0000","last_active":"2024-04-14T15:11:00.764213+0000","last_peered":"2024-04-14T15:11:00.764213+0000","last_clean":"2024-04-14T15:11:00.764213+0000","last_became_active":"2024-04-14T15:09:47.044610+0000","last_became_peered":"2024-04-14T15:09:47.044610+0000","last_unstale":"2024-04-14T15:11:00.764213+0000","last_undegraded":"2024-04-14T15:11:00.764213+0000","last_fullsized":"2024-04-14T15:11:00.764213+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:09:46.023663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:05.510613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":51,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:16.821238+0000","last_change":"2024-04-14T15:11:16.821265+0000","last_active":"2024-04-14T15:11:16.821238+0000","last_peered":"2024-04-14T15:11:16.821238+0000","last_clean":"2024-04-14T15:11:16.821238+0000","last_became_active":"2024-04-14T15:09:49.129610+0000","last_became_peered":"2024-04-14T15:09:49.129610+0000","last_unstale":"2024-04-14T15:11:16.821238+0000","last_undegraded":"2024-04-14T15:11:16.821238+0000","last_fullsized":"2024-04-14T15:11:16.821238+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:16.821140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:16.821140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:32.897119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.939826+0000","last_change":"2024-04-14T15:09:43.020785+0000","last_active":"2024-04-14T15:11:00.939826+0000","last_peered":"2024-04-14T15:11:00.939826+0000","last_clean":"2024-04-14T15:11:00.939826+0000","last_became_active":"2024-04-14T15:09:43.020503+0000","last_became_peered":"2024-04-14T15:09:43.020503+0000","last_unstale":"2024-04-14T15:11:00.939826+0000","last_undegraded":"2024-04-14T15:11:00.939826+0000","last_fullsized":"2024-04-14T15:11:00.939826+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:09:42.004317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:09.830404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"72'14","reported_seq":80,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:14.657145+0000","last_change":"2024-04-14T15:11:14.657178+0000","last_active":"2024-04-14T15:11:14.657145+0000","last_peered":"2024-04-14T15:11:14.657145+0000","last_clean":"2024-04-14T15:11:14.657145+0000","last_became_active":"2024-04-14T15:09:45.025673+0000","last_became_peered":"2024-04-14T15:09:45.025673+0000","last_unstale":"2024-04-14T15:11:14.657145+0000","last_undegraded":"2024-04-14T15:11:14.657145+0000","last_fullsized":"2024-04-14T15:11:14.657145+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'14","last_scrub_stamp":"2024-04-14T15:11:14.657075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:14.657075+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:37.152537+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"72'8","reported_seq":75,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:13.084518+0000","last_change":"2024-04-14T15:09:47.046875+0000","last_active":"2024-04-14T15:11:13.084518+0000","last_peered":"2024-04-14T15:11:13.084518+0000","last_clean":"2024-04-14T15:11:13.084518+0000","last_became_active":"2024-04-14T15:09:47.046560+0000","last_became_peered":"2024-04-14T15:09:47.046560+0000","last_unstale":"2024-04-14T15:11:13.084518+0000","last_undegraded":"2024-04-14T15:11:13.084518+0000","last_fullsized":"2024-04-14T15:11:13.084518+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:09:46.023663+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":51,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.501853+0000","last_change":"2024-04-14T15:11:01.501878+0000","last_active":"2024-04-14T15:11:01.501853+0000","last_peered":"2024-04-14T15:11:01.501853+0000","last_clean":"2024-04-14T15:11:01.501853+0000","last_became_active":"2024-04-14T15:09:49.128028+0000","last_became_peered":"2024-04-14T15:09:49.128028+0000","last_unstale":"2024-04-14T15:11:01.501853+0000","last_undegraded":"2024-04-14T15:11:01.501853+0000","last_fullsized":"2024-04-14T15:11:01.501853+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:01.501784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:01.501784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:23.865314+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:02.763474+0000","last_change":"2024-04-14T15:11:02.763497+0000","last_active":"2024-04-14T15:11:02.763474+0000","last_peered":"2024-04-14T15:11:02.763474+0000","last_clean":"2024-04-14T15:11:02.763474+0000","last_became_active":"2024-04-14T15:09:47.043021+0000","last_became_peered":"2024-04-14T15:09:47.043021+0000","last_unstale":"2024-04-14T15:11:02.763474+0000","last_undegraded":"2024-04-14T15:11:02.763474+0000","last_fullsized":"2024-04-14T15:11:02.763474+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:02.763399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:02.763399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:06.127325+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":63,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.937694+0000","last_change":"2024-04-14T15:10:48.939920+0000","last_active":"2024-04-14T15:11:00.937694+0000","last_peered":"2024-04-14T15:11:00.937694+0000","last_clean":"2024-04-14T15:11:00.937694+0000","last_became_active":"2024-04-14T15:09:43.018601+0000","last_became_peered":"2024-04-14T15:09:43.018601+0000","last_unstale":"2024-04-14T15:11:00.937694+0000","last_undegraded":"2024-04-14T15:11:00.937694+0000","last_fullsized":"2024-04-14T15:11:00.937694+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:48.939790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:10:48.939790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:14.888606+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"85'16","reported_seq":83,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764593+0000","last_change":"2024-04-14T15:10:47.456301+0000","last_active":"2024-04-14T15:11:00.764593+0000","last_peered":"2024-04-14T15:11:00.764593+0000","last_clean":"2024-04-14T15:11:00.764593+0000","last_became_active":"2024-04-14T15:09:45.028212+0000","last_became_peered":"2024-04-14T15:09:45.028212+0000","last_unstale":"2024-04-14T15:11:00.764593+0000","last_undegraded":"2024-04-14T15:11:00.764593+0000","last_fullsized":"2024-04-14T15:11:00.764593+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-14T15:10:47.456169+0000","last_deep_scrub":"85'16","last_deep_scrub_stamp":"2024-04-14T15:10:47.456169+0000","last_clean_scrub_stamp":"2024-04-14T15:10:47.456169+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:53.736949+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.210097+0000","last_change":"2024-04-14T15:09:49.059351+0000","last_active":"2024-04-14T15:11:01.210097+0000","last_peered":"2024-04-14T15:11:01.210097+0000","last_clean":"2024-04-14T15:11:01.210097+0000","last_became_active":"2024-04-14T15:09:49.058940+0000","last_became_peered":"2024-04-14T15:09:49.058940+0000","last_unstale":"2024-04-14T15:11:01.210097+0000","last_undegraded":"2024-04-14T15:11:01.210097+0000","last_fullsized":"2024-04-14T15:11:01.210097+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:03.127308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"72'8","reported_seq":75,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:13.084589+0000","last_change":"2024-04-14T15:09:47.051276+0000","last_active":"2024-04-14T15:11:13.084589+0000","last_peered":"2024-04-14T15:11:13.084589+0000","last_clean":"2024-04-14T15:11:13.084589+0000","last_became_active":"2024-04-14T15:09:47.050962+0000","last_became_peered":"2024-04-14T15:09:47.050962+0000","last_unstale":"2024-04-14T15:11:13.084589+0000","last_undegraded":"2024-04-14T15:11:13.084589+0000","last_fullsized":"2024-04-14T15:11:13.084589+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:09:46.023663+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":63,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:11.285661+0000","last_change":"2024-04-14T15:11:11.285681+0000","last_active":"2024-04-14T15:11:11.285661+0000","last_peered":"2024-04-14T15:11:11.285661+0000","last_clean":"2024-04-14T15:11:11.285661+0000","last_became_active":"2024-04-14T15:09:43.020878+0000","last_became_peered":"2024-04-14T15:09:43.020878+0000","last_unstale":"2024-04-14T15:11:11.285661+0000","last_undegraded":"2024-04-14T15:11:11.285661+0000","last_fullsized":"2024-04-14T15:11:11.285661+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:11.285588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:11.285588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:11.582228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"72'15","reported_seq":76,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764979+0000","last_change":"2024-04-14T15:09:45.033879+0000","last_active":"2024-04-14T15:11:00.764979+0000","last_peered":"2024-04-14T15:11:00.764979+0000","last_clean":"2024-04-14T15:11:00.764979+0000","last_became_active":"2024-04-14T15:09:45.033331+0000","last_became_peered":"2024-04-14T15:09:45.033331+0000","last_unstale":"2024-04-14T15:11:00.764979+0000","last_undegraded":"2024-04-14T15:11:00.764979+0000","last_fullsized":"2024-04-14T15:11:00.764979+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764272+0000","last_change":"2024-04-14T15:09:49.065592+0000","last_active":"2024-04-14T15:11:00.764272+0000","last_peered":"2024-04-14T15:11:00.764272+0000","last_clean":"2024-04-14T15:11:00.764272+0000","last_became_active":"2024-04-14T15:09:49.065274+0000","last_became_peered":"2024-04-14T15:09:49.065274+0000","last_unstale":"2024-04-14T15:11:00.764272+0000","last_undegraded":"2024-04-14T15:11:00.764272+0000","last_fullsized":"2024-04-14T15:11:00.764272+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:11.500131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"72'8","reported_seq":75,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:13.084481+0000","last_change":"2024-04-14T15:09:47.044537+0000","last_active":"2024-04-14T15:11:13.084481+0000","last_peered":"2024-04-14T15:11:13.084481+0000","last_clean":"2024-04-14T15:11:13.084481+0000","last_became_active":"2024-04-14T15:09:47.044085+0000","last_became_peered":"2024-04-14T15:09:47.044085+0000","last_unstale":"2024-04-14T15:11:13.084481+0000","last_undegraded":"2024-04-14T15:11:13.084481+0000","last_fullsized":"2024-04-14T15:11:13.084481+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:09:46.023663+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":63,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764648+0000","last_change":"2024-04-14T15:10:46.280858+0000","last_active":"2024-04-14T15:11:00.764648+0000","last_peered":"2024-04-14T15:11:00.764648+0000","last_clean":"2024-04-14T15:11:00.764648+0000","last_became_active":"2024-04-14T15:09:43.018871+0000","last_became_peered":"2024-04-14T15:09:43.018871+0000","last_unstale":"2024-04-14T15:11:00.764648+0000","last_undegraded":"2024-04-14T15:11:00.764648+0000","last_fullsized":"2024-04-14T15:11:00.764648+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:46.280773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:10:46.280773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:59.607706+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"72'10","reported_seq":66,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.764245+0000","last_change":"2024-04-14T15:09:45.029530+0000","last_active":"2024-04-14T15:11:00.764245+0000","last_peered":"2024-04-14T15:11:00.764245+0000","last_clean":"2024-04-14T15:11:00.764245+0000","last_became_active":"2024-04-14T15:09:45.029170+0000","last_became_peered":"2024-04-14T15:09:45.029170+0000","last_unstale":"2024-04-14T15:11:00.764245+0000","last_undegraded":"2024-04-14T15:11:00.764245+0000","last_fullsized":"2024-04-14T15:11:00.764245+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:08.246963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":51,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:10.852776+0000","last_change":"2024-04-14T15:11:10.852797+0000","last_active":"2024-04-14T15:11:10.852776+0000","last_peered":"2024-04-14T15:11:10.852776+0000","last_clean":"2024-04-14T15:11:10.852776+0000","last_became_active":"2024-04-14T15:09:49.069594+0000","last_became_peered":"2024-04-14T15:09:49.069594+0000","last_unstale":"2024-04-14T15:11:10.852776+0000","last_undegraded":"2024-04-14T15:11:10.852776+0000","last_fullsized":"2024-04-14T15:11:10.852776+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:10.852715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:10.852715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:18.595232+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":47,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:00.763544+0000","last_change":"2024-04-14T15:09:47.043481+0000","last_active":"2024-04-14T15:11:00.763544+0000","last_peered":"2024-04-14T15:11:00.763544+0000","last_clean":"2024-04-14T15:11:00.763544+0000","last_became_active":"2024-04-14T15:09:47.042810+0000","last_became_peered":"2024-04-14T15:09:47.042810+0000","last_unstale":"2024-04-14T15:11:00.763544+0000","last_undegraded":"2024-04-14T15:11:00.763544+0000","last_fullsized":"2024-04-14T15:11:00.763544+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:09:46.023663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:12.979380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":63,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.903230+0000","last_change":"2024-04-14T15:11:01.903252+0000","last_active":"2024-04-14T15:11:01.903230+0000","last_peered":"2024-04-14T15:11:01.903230+0000","last_clean":"2024-04-14T15:11:01.903230+0000","last_became_active":"2024-04-14T15:09:43.022351+0000","last_became_peered":"2024-04-14T15:09:43.022351+0000","last_unstale":"2024-04-14T15:11:01.903230+0000","last_undegraded":"2024-04-14T15:11:01.903230+0000","last_fullsized":"2024-04-14T15:11:01.903230+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:01.903150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:01.903150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:27.833698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"72'17","reported_seq":111,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:09.502490+0000","last_change":"2024-04-14T15:11:09.502519+0000","last_active":"2024-04-14T15:11:09.502490+0000","last_peered":"2024-04-14T15:11:09.502490+0000","last_clean":"2024-04-14T15:11:09.502490+0000","last_became_active":"2024-04-14T15:10:07.766008+0000","last_became_peered":"2024-04-14T15:10:07.766008+0000","last_unstale":"2024-04-14T15:11:09.502490+0000","last_undegraded":"2024-04-14T15:11:09.502490+0000","last_fullsized":"2024-04-14T15:11:09.502490+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'17","last_scrub_stamp":"2024-04-14T15:11:09.502431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:09.502431+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:29.834036+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":51,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:15.845003+0000","last_change":"2024-04-14T15:11:15.845024+0000","last_active":"2024-04-14T15:11:15.845003+0000","last_peered":"2024-04-14T15:11:15.845003+0000","last_clean":"2024-04-14T15:11:15.845003+0000","last_became_active":"2024-04-14T15:09:49.065397+0000","last_became_peered":"2024-04-14T15:09:49.065397+0000","last_unstale":"2024-04-14T15:11:15.845003+0000","last_undegraded":"2024-04-14T15:11:15.845003+0000","last_fullsized":"2024-04-14T15:11:15.845003+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:15.844942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:15.844942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:39.797576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.210496+0000","last_change":"2024-04-14T15:10:59.793878+0000","last_active":"2024-04-14T15:11:01.210496+0000","last_peered":"2024-04-14T15:11:01.210496+0000","last_clean":"2024-04-14T15:11:01.210496+0000","last_became_active":"2024-04-14T15:09:47.045542+0000","last_became_peered":"2024-04-14T15:09:47.045542+0000","last_unstale":"2024-04-14T15:11:01.210496+0000","last_undegraded":"2024-04-14T15:11:01.210496+0000","last_fullsized":"2024-04-14T15:11:01.210496+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:59.793768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:10:59.793768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:24.997192+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":63,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.210883+0000","last_change":"2024-04-14T15:10:45.837586+0000","last_active":"2024-04-14T15:11:01.210883+0000","last_peered":"2024-04-14T15:11:01.210883+0000","last_clean":"2024-04-14T15:11:01.210883+0000","last_became_active":"2024-04-14T15:09:43.020268+0000","last_became_peered":"2024-04-14T15:09:43.020268+0000","last_unstale":"2024-04-14T15:11:01.210883+0000","last_undegraded":"2024-04-14T15:11:01.210883+0000","last_fullsized":"2024-04-14T15:11:01.210883+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:45.837506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:10:45.837506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:04.829576+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"72'19","reported_seq":90,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:09.225427+0000","last_change":"2024-04-14T15:11:09.225462+0000","last_active":"2024-04-14T15:11:09.225427+0000","last_peered":"2024-04-14T15:11:09.225427+0000","last_clean":"2024-04-14T15:11:09.225427+0000","last_became_active":"2024-04-14T15:09:45.178782+0000","last_became_peered":"2024-04-14T15:09:45.178782+0000","last_unstale":"2024-04-14T15:11:09.225427+0000","last_undegraded":"2024-04-14T15:11:09.225427+0000","last_fullsized":"2024-04-14T15:11:09.225427+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'19","last_scrub_stamp":"2024-04-14T15:11:09.225359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:09.225359+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:29.861536+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":51,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.209629+0000","last_change":"2024-04-14T15:10:53.789872+0000","last_active":"2024-04-14T15:11:01.209629+0000","last_peered":"2024-04-14T15:11:01.209629+0000","last_clean":"2024-04-14T15:11:01.209629+0000","last_became_active":"2024-04-14T15:09:49.061495+0000","last_became_peered":"2024-04-14T15:09:49.061495+0000","last_unstale":"2024-04-14T15:11:01.209629+0000","last_undegraded":"2024-04-14T15:11:01.209629+0000","last_fullsized":"2024-04-14T15:11:01.209629+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:53.789790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:10:53.789790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:54.748155+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.210353+0000","last_change":"2024-04-14T15:10:50.788471+0000","last_active":"2024-04-14T15:11:01.210353+0000","last_peered":"2024-04-14T15:11:01.210353+0000","last_clean":"2024-04-14T15:11:01.210353+0000","last_became_active":"2024-04-14T15:09:47.046153+0000","last_became_peered":"2024-04-14T15:09:47.046153+0000","last_unstale":"2024-04-14T15:11:01.210353+0000","last_undegraded":"2024-04-14T15:11:01.210353+0000","last_fullsized":"2024-04-14T15:11:01.210353+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:50.788350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:10:50.788350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:52.620524+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":63,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:04.223660+0000","last_change":"2024-04-14T15:11:04.223680+0000","last_active":"2024-04-14T15:11:04.223660+0000","last_peered":"2024-04-14T15:11:04.223660+0000","last_clean":"2024-04-14T15:11:04.223660+0000","last_became_active":"2024-04-14T15:09:43.020044+0000","last_became_peered":"2024-04-14T15:09:43.020044+0000","last_unstale":"2024-04-14T15:11:04.223660+0000","last_undegraded":"2024-04-14T15:11:04.223660+0000","last_fullsized":"2024-04-14T15:11:04.223660+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:04.223605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:04.223605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:11.422287+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"72'9","reported_seq":75,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.211199+0000","last_change":"2024-04-14T15:10:50.850226+0000","last_active":"2024-04-14T15:11:01.211199+0000","last_peered":"2024-04-14T15:11:01.211199+0000","last_clean":"2024-04-14T15:11:01.211199+0000","last_became_active":"2024-04-14T15:09:45.037179+0000","last_became_peered":"2024-04-14T15:09:45.037179+0000","last_unstale":"2024-04-14T15:11:01.211199+0000","last_undegraded":"2024-04-14T15:11:01.211199+0000","last_fullsized":"2024-04-14T15:11:01.211199+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-04-14T15:10:50.850116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:10:50.850116+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:06.196819+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.210115+0000","last_change":"2024-04-14T15:09:49.128389+0000","last_active":"2024-04-14T15:11:01.210115+0000","last_peered":"2024-04-14T15:11:01.210115+0000","last_clean":"2024-04-14T15:11:01.210115+0000","last_became_active":"2024-04-14T15:09:49.128070+0000","last_became_peered":"2024-04-14T15:09:49.128070+0000","last_unstale":"2024-04-14T15:11:01.210115+0000","last_undegraded":"2024-04-14T15:11:01.210115+0000","last_fullsized":"2024-04-14T15:11:01.210115+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:09.430362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":63,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:13.838541+0000","last_change":"2024-04-14T15:11:13.838572+0000","last_active":"2024-04-14T15:11:13.838541+0000","last_peered":"2024-04-14T15:11:13.838541+0000","last_clean":"2024-04-14T15:11:13.838541+0000","last_became_active":"2024-04-14T15:09:43.021460+0000","last_became_peered":"2024-04-14T15:09:43.021460+0000","last_unstale":"2024-04-14T15:11:13.838541+0000","last_undegraded":"2024-04-14T15:11:13.838541+0000","last_fullsized":"2024-04-14T15:11:13.838541+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:13.838468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:13.838468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:31.461401+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"72'15","reported_seq":76,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:01.210565+0000","last_change":"2024-04-14T15:09:45.180505+0000","last_active":"2024-04-14T15:11:01.210565+0000","last_peered":"2024-04-14T15:11:01.210565+0000","last_clean":"2024-04-14T15:11:01.210565+0000","last_became_active":"2024-04-14T15:09:45.180160+0000","last_became_peered":"2024-04-14T15:09:45.180160+0000","last_unstale":"2024-04-14T15:11:01.210565+0000","last_undegraded":"2024-04-14T15:11:01.210565+0000","last_fullsized":"2024-04-14T15:11:01.210565+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"85'9","reported_seq":84,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-14T15:11:13.084634+0000","last_change":"2024-04-14T15:10:51.950673+0000","last_active":"2024-04-14T15:11:13.084634+0000","last_peered":"2024-04-14T15:11:13.084634+0000","last_clean":"2024-04-14T15:11:13.084634+0000","last_became_active":"2024-04-14T15:09:47.045441+0000","last_became_peered":"2024-04-14T15:09:47.045441+0000","last_unstale":"2024-04-14T15:11:13.084634+0000","last_undegraded":"2024-04-14T15:11:13.084634+0000","last_fullsized":"2024-04-14T15:11:13.084634+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9","last_scrub_stamp":"2024-04-14T15:10:51.950580+0000","last_deep_scrub":"85'9","last_deep_scrub_stamp":"2024-04-14T15:10:51.950580+0000","last_clean_scrub_stamp":"2024-04-14T15:10:51.950580+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:09.843217+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":400,"ondisk_log_size":400,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":90,"num_read_kb":85,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103193,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28368,"kb_used_data":1352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716784,"statfs":{"total":95995035648,"available":95965986816,"internally_reserved":0,"allocated":1384448,"data_stored":937840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397532,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28364,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716788,"statfs":{"total":95995035648,"available":95965990912,"internally_reserved":0,"allocated":1380352,"data_stored":938133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574902,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27820,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717332,"statfs":{"total":95995035648,"available":95966547968,"internally_reserved":0,"allocated":823296,"data_stored":347886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738427,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32536,"kb_used_data":1360,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712616,"statfs":{"total":95995035648,"available":95961718784,"internally_reserved":0,"allocated":1392640,"data_stored":938493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411378,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31932,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713220,"statfs":{"total":95995035648,"available":95962337280,"internally_reserved":0,"allocated":774144,"data_stored":346987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182444,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27816,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717336,"statfs":{"total":95995035648,"available":95966552064,"internally_reserved":0,"allocated":819200,"data_stored":348207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986214,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27796,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717356,"statfs":{"total":95995035648,"available":95966572544,"internally_reserved":0,"allocated":798720,"data_stored":347368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659171,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31940,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713212,"statfs":{"total":95995035648,"available":95962329088,"internally_reserved":0,"allocated":782336,"data_stored":346822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-14T15:11:23.915 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph pg dump --format=json 2024-04-14T15:11:24.161 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:11:24.191 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:11:23 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:11:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:11:24.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:24 smithi090 ceph-mon[30399]: 2.1b scrub starts 2024-04-14T15:11:24.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:24 smithi090 ceph-mon[30399]: 2.1b scrub ok 2024-04-14T15:11:24.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:24 smithi090 ceph-mon[30399]: 2.1f scrub starts 2024-04-14T15:11:24.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:24 smithi090 ceph-mon[30399]: 2.1f scrub ok 2024-04-14T15:11:24.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:24 smithi090 ceph-mon[30399]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:11:24.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:24 smithi090 ceph-mon[30399]: from='client.14778 v1:172.21.15.90:0/1347748414' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:11:24.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:24 smithi090 ceph-mon[41670]: 2.1b scrub starts 2024-04-14T15:11:24.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:24 smithi090 ceph-mon[41670]: 2.1b scrub ok 2024-04-14T15:11:24.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:24 smithi090 ceph-mon[41670]: 2.1f scrub starts 2024-04-14T15:11:24.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:24 smithi090 ceph-mon[41670]: 2.1f scrub ok 2024-04-14T15:11:24.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:24 smithi090 ceph-mon[41670]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:11:24.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:24 smithi090 ceph-mon[41670]: from='client.14778 v1:172.21.15.90:0/1347748414' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:11:24.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:24 smithi092 ceph-mon[33966]: 2.1b scrub starts 2024-04-14T15:11:24.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:24 smithi092 ceph-mon[33966]: 2.1b scrub ok 2024-04-14T15:11:24.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:24 smithi092 ceph-mon[33966]: 2.1f scrub starts 2024-04-14T15:11:24.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:24 smithi092 ceph-mon[33966]: 2.1f scrub ok 2024-04-14T15:11:24.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:24 smithi092 ceph-mon[33966]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:11:24.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:24 smithi092 ceph-mon[33966]: from='client.14778 v1:172.21.15.90:0/1347748414' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:11:25.110 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:11:24 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:11:25.350 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:25 smithi090 ceph-mon[30399]: 4.4 scrub starts 2024-04-14T15:11:25.350 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:25 smithi090 ceph-mon[30399]: 4.4 scrub ok 2024-04-14T15:11:25.350 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:25 smithi090 ceph-mon[30399]: 2.12 scrub starts 2024-04-14T15:11:25.350 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:25 smithi090 ceph-mon[30399]: 2.12 scrub ok 2024-04-14T15:11:25.350 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:25 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:11:25.350 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:25 smithi090 ceph-mon[41670]: 4.4 scrub starts 2024-04-14T15:11:25.350 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:25 smithi090 ceph-mon[41670]: 4.4 scrub ok 2024-04-14T15:11:25.350 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:25 smithi090 ceph-mon[41670]: 2.12 scrub starts 2024-04-14T15:11:25.350 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:25 smithi090 ceph-mon[41670]: 2.12 scrub ok 2024-04-14T15:11:25.351 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:25 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:11:25.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:25 smithi092 ceph-mon[33966]: 4.4 scrub starts 2024-04-14T15:11:25.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:25 smithi092 ceph-mon[33966]: 4.4 scrub ok 2024-04-14T15:11:25.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:25 smithi092 ceph-mon[33966]: 2.12 scrub starts 2024-04-14T15:11:25.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:25 smithi092 ceph-mon[33966]: 2.12 scrub ok 2024-04-14T15:11:25.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:25 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:11:25.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:25.865+0000 7f1a2fcc7700 1 -- 172.21.15.90:0/3197895597 >> v1:172.21.15.90:6789/0 conn(0x7f1a2810e200 legacy=0x7f1a281106c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:25.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:25.865+0000 7f1a2fcc7700 1 -- 172.21.15.90:0/3197895597 shutdown_connections 2024-04-14T15:11:25.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:25.865+0000 7f1a2fcc7700 1 -- 172.21.15.90:0/3197895597 >> 172.21.15.90:0/3197895597 conn(0x7f1a280fd290 msgr2=0x7f1a280ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:11:25.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:25.865+0000 7f1a2fcc7700 1 -- 172.21.15.90:0/3197895597 shutdown_connections 2024-04-14T15:11:25.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:25.865+0000 7f1a2fcc7700 1 -- 172.21.15.90:0/3197895597 wait complete. 2024-04-14T15:11:25.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:25.865+0000 7f1a2fcc7700 1 librados: starting msgr at 2024-04-14T15:11:25.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:25.865+0000 7f1a2fcc7700 1 librados: starting objecter 2024-04-14T15:11:25.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:25.866+0000 7f1a2fcc7700 1 Processor -- start 2024-04-14T15:11:25.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:25.866+0000 7f1a2fcc7700 1 -- start start 2024-04-14T15:11:25.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:25.866+0000 7f1a2fcc7700 1 librados: setting wanted keys 2024-04-14T15:11:25.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:25.866+0000 7f1a2fcc7700 1 librados: calling monclient init 2024-04-14T15:11:25.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:25.866+0000 7f1a2fcc7700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1a280ff4e0 con 0x7f1a2810a630 2024-04-14T15:11:25.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:25.866+0000 7f1a2fcc7700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1a281ab210 con 0x7f1a2810e200 2024-04-14T15:11:25.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:25.866+0000 7f1a2fcc7700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1a281ac3f0 con 0x7f1a28107780 2024-04-14T15:11:25.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:25.866+0000 7f1a2da63700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f1a28107780 0x7f1a280fd740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:43886/0 (socket says 172.21.15.90:43886) 2024-04-14T15:11:25.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:25.866+0000 7f1a2da63700 1 -- 172.21.15.90:0/3399860262 learned_addr learned my addr 172.21.15.90:0/3399860262 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:11:25.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:25.866+0000 7f1a2d262700 1 --1- 172.21.15.90:0/3399860262 >> v1:172.21.15.90:6789/0 conn(0x7f1a2810a630 0x7f1a280fde50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:59384/0 (socket says 172.21.15.90:59384) 2024-04-14T15:11:25.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:25.866+0000 7f1a1effd700 10 client.?.objecter ms_handle_connect 0x7f1a2810a630 2024-04-14T15:11:25.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:25.866+0000 7f1a1effd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:11:25.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:25.866+0000 7f1a1effd700 10 client.?.objecter ms_handle_connect 0x7f1a28107780 2024-04-14T15:11:25.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:25.866+0000 7f1a1effd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:11:25.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:25.866+0000 7f1a1effd700 10 client.?.objecter ms_handle_connect 0x7f1a2810e200 2024-04-14T15:11:25.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:25.866+0000 7f1a1effd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:11:25.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:25.866+0000 7f1a1effd700 1 -- 172.21.15.90:0/3399860262 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1613811870 0 0) 0x7f1a280ff4e0 con 0x7f1a2810a630 2024-04-14T15:11:25.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:25.866+0000 7f1a1effd700 1 -- 172.21.15.90:0/3399860262 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a140035b0 con 0x7f1a2810a630 2024-04-14T15:11:25.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:25.866+0000 7f1a1effd700 1 -- 172.21.15.90:0/3399860262 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1251008420 0 0) 0x7f1a281ab210 con 0x7f1a2810e200 2024-04-14T15:11:25.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:25.866+0000 7f1a1effd700 1 -- 172.21.15.90:0/3399860262 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a280ff4e0 con 0x7f1a2810e200 2024-04-14T15:11:25.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:25.866+0000 7f1a1effd700 1 -- 172.21.15.90:0/3399860262 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2946925794 0 0) 0x7f1a281ac3f0 con 0x7f1a28107780 2024-04-14T15:11:25.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:25.866+0000 7f1a1effd700 1 -- 172.21.15.90:0/3399860262 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a281ab210 con 0x7f1a28107780 2024-04-14T15:11:25.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:25.866+0000 7f1a1effd700 1 -- 172.21.15.90:0/3399860262 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1540100331 0 0) 0x7f1a280ff4e0 con 0x7f1a2810e200 2024-04-14T15:11:25.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:25.866+0000 7f1a1effd700 1 -- 172.21.15.90:0/3399860262 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1a281ac3f0 con 0x7f1a2810e200 2024-04-14T15:11:25.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:25.866+0000 7f1a1effd700 1 -- 172.21.15.90:0/3399860262 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f1a20001950 con 0x7f1a2810e200 2024-04-14T15:11:25.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:25.867+0000 7f1a1effd700 1 -- 172.21.15.90:0/3399860262 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3271675699 0 0) 0x7f1a140035b0 con 0x7f1a2810a630 2024-04-14T15:11:25.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:25.867+0000 7f1a1effd700 1 -- 172.21.15.90:0/3399860262 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1a280ff4e0 con 0x7f1a2810a630 2024-04-14T15:11:25.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:25.867+0000 7f1a1effd700 1 -- 172.21.15.90:0/3399860262 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f1a18001830 con 0x7f1a2810a630 2024-04-14T15:11:25.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:25.867+0000 7f1a1effd700 1 -- 172.21.15.90:0/3399860262 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3433475754 0 0) 0x7f1a281ab210 con 0x7f1a28107780 2024-04-14T15:11:25.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:25.867+0000 7f1a1effd700 1 -- 172.21.15.90:0/3399860262 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1a140035b0 con 0x7f1a28107780 2024-04-14T15:11:25.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:25.867+0000 7f1a1effd700 1 -- 172.21.15.90:0/3399860262 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f1a240018b0 con 0x7f1a28107780 2024-04-14T15:11:25.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:25.867+0000 7f1a1effd700 1 -- 172.21.15.90:0/3399860262 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1814662856 0 0) 0x7f1a281ac3f0 con 0x7f1a2810e200 2024-04-14T15:11:25.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:25.867+0000 7f1a1effd700 1 -- 172.21.15.90:0/3399860262 >> v1:172.21.15.90:6790/0 conn(0x7f1a28107780 legacy=0x7f1a280fd740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:25.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:25.867+0000 7f1a1effd700 1 -- 172.21.15.90:0/3399860262 >> v1:172.21.15.90:6789/0 conn(0x7f1a2810a630 legacy=0x7f1a280fde50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:25.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:25.867+0000 7f1a1effd700 1 -- 172.21.15.90:0/3399860262 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a281ad5d0 con 0x7f1a2810e200 2024-04-14T15:11:25.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:25.867+0000 7f1a2fcc7700 1 -- 172.21.15.90:0/3399860262 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1a281aa260 con 0x7f1a2810e200 2024-04-14T15:11:25.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:25.867+0000 7f1a2fcc7700 10 client.24673.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:11:25.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:25.867+0000 7f1a2fcc7700 1 -- 172.21.15.90:0/3399860262 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f1a281aa720 con 0x7f1a2810e200 2024-04-14T15:11:25.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:25.867+0000 7f1a2fcc7700 1 librados: init done 2024-04-14T15:11:25.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:25.867+0000 7f1a1effd700 1 -- 172.21.15.90:0/3399860262 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f1a20003a00 con 0x7f1a2810e200 2024-04-14T15:11:25.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:25.867+0000 7f1a1effd700 1 -- 172.21.15.90:0/3399860262 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f1a200050a0 con 0x7f1a2810e200 2024-04-14T15:11:25.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:25.868+0000 7f1a2fcc7700 1 -- 172.21.15.90:0/3399860262 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a0c005120 con 0x7f1a2810e200 2024-04-14T15:11:25.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:25.868+0000 7f1a1effd700 1 -- 172.21.15.90:0/3399860262 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f1a2001bab0 con 0x7f1a2810e200 2024-04-14T15:11:25.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:25.869+0000 7f1a1effd700 1 -- 172.21.15.90:0/3399860262 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(91..91 src has 12..91) ==== 6288+0+0 (unknown 3555498640 0 0) 0x7f1a20089470 con 0x7f1a2810e200 2024-04-14T15:11:25.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:25.869+0000 7f1a1effd700 10 client.24673.objecter ms_dispatch 0x7f1a28001550 osd_map(91..91 src has 12..91) 2024-04-14T15:11:25.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:25.869+0000 7f1a1effd700 3 client.24673.objecter handle_osd_map got epochs [91,91] > 0 2024-04-14T15:11:25.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:25.869+0000 7f1a1effd700 3 client.24673.objecter handle_osd_map decoding full epoch 91 2024-04-14T15:11:25.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:25.869+0000 7f1a1effd700 20 client.24673.objecter dump_active .. 0 homeless 2024-04-14T15:11:25.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:25.869+0000 7f1a1effd700 10 client.24673.objecter ms_handle_connect 0x7f1a140713c0 2024-04-14T15:11:25.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:25.873+0000 7f1a1effd700 1 -- 172.21.15.90:0/3399860262 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f1a200565e0 con 0x7f1a2810e200 2024-04-14T15:11:26.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:26.068+0000 7f1a2fcc7700 1 -- 172.21.15.90:0/3399860262 --> v1:172.21.15.90:6800/420935427 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1a0c000b90 con 0x7f1a140713c0 2024-04-14T15:11:26.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:26.078+0000 7f1a1effd700 1 -- 172.21.15.90:0/3399860262 <== mgr.14610 v1:172.21.15.90:6800/420935427 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+344680 (unknown 2965378022 0 1006476478) 0x7f1a0c000b90 con 0x7f1a140713c0 2024-04-14T15:11:26.082 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:11:26.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:26.081+0000 7f1a2fcc7700 10 librados: watch_flush enter 2024-04-14T15:11:26.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:26.081+0000 7f1a2fcc7700 10 librados: watch_flush exit 2024-04-14T15:11:26.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:26.081+0000 7f1a2fcc7700 20 client.24673.objecter shutdown clearing up homeless session... 2024-04-14T15:11:26.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:26.081+0000 7f1a2fcc7700 10 client.24673.objecter successfully canceled tick 2024-04-14T15:11:26.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:26.081+0000 7f1a2fcc7700 1 -- 172.21.15.90:0/3399860262 >> v1:172.21.15.90:6800/420935427 conn(0x7f1a140713c0 legacy=0x7f1a14073880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:26.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:26.081+0000 7f1a2fcc7700 1 -- 172.21.15.90:0/3399860262 >> v1:172.21.15.92:6789/0 conn(0x7f1a2810e200 legacy=0x7f1a281a8ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:26.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:26.081+0000 7f1a2fcc7700 1 -- 172.21.15.90:0/3399860262 shutdown_connections 2024-04-14T15:11:26.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:26.081+0000 7f1a2fcc7700 1 -- 172.21.15.90:0/3399860262 >> 172.21.15.90:0/3399860262 conn(0x7f1a280fd290 msgr2=0x7f1a281000f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:11:26.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:26.081+0000 7f1a2fcc7700 1 -- 172.21.15.90:0/3399860262 shutdown_connections 2024-04-14T15:11:26.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:26.081+0000 7f1a2fcc7700 1 -- 172.21.15.90:0/3399860262 wait complete. 2024-04-14T15:11:26.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:26.081+0000 7f1a2fcc7700 1 librados: shutdown 2024-04-14T15:11:26.086 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-14T15:11:26.122 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:11:25 smithi092 bash[67889]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-04-14T15:11:26.122 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:11:26 smithi092 bash[67889]: Writing manifest to image destination 2024-04-14T15:11:26.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:26 smithi092 ceph-mon[33966]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:11:26.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:26 smithi092 ceph-mon[33966]: 6.4 scrub starts 2024-04-14T15:11:26.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:26 smithi092 ceph-mon[33966]: 6.4 scrub ok 2024-04-14T15:11:26.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:26 smithi092 ceph-mon[33966]: 4.5 deep-scrub starts 2024-04-14T15:11:26.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:26 smithi092 ceph-mon[33966]: 4.5 deep-scrub ok 2024-04-14T15:11:26.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:26 smithi092 ceph-mon[33966]: 6.d scrub starts 2024-04-14T15:11:26.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:26 smithi092 ceph-mon[33966]: 6.d scrub ok 2024-04-14T15:11:26.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:26 smithi092 ceph-mon[33966]: osdmap e91: 8 total, 8 up, 8 in 2024-04-14T15:11:26.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:26 smithi090 ceph-mon[41670]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:11:26.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:26 smithi090 ceph-mon[41670]: 6.4 scrub starts 2024-04-14T15:11:26.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:26 smithi090 ceph-mon[41670]: 6.4 scrub ok 2024-04-14T15:11:26.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:26 smithi090 ceph-mon[41670]: 4.5 deep-scrub starts 2024-04-14T15:11:26.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:26 smithi090 ceph-mon[41670]: 4.5 deep-scrub ok 2024-04-14T15:11:26.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:26 smithi090 ceph-mon[41670]: 6.d scrub starts 2024-04-14T15:11:26.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:26 smithi090 ceph-mon[41670]: 6.d scrub ok 2024-04-14T15:11:26.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:26 smithi090 ceph-mon[41670]: osdmap e91: 8 total, 8 up, 8 in 2024-04-14T15:11:26.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:26 smithi090 ceph-mon[30399]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:11:26.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:26 smithi090 ceph-mon[30399]: 6.4 scrub starts 2024-04-14T15:11:26.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:26 smithi090 ceph-mon[30399]: 6.4 scrub ok 2024-04-14T15:11:26.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:26 smithi090 ceph-mon[30399]: 4.5 deep-scrub starts 2024-04-14T15:11:26.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:26 smithi090 ceph-mon[30399]: 4.5 deep-scrub ok 2024-04-14T15:11:26.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:26 smithi090 ceph-mon[30399]: 6.d scrub starts 2024-04-14T15:11:26.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:26 smithi090 ceph-mon[30399]: 6.d scrub ok 2024-04-14T15:11:26.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:26 smithi090 ceph-mon[30399]: osdmap e91: 8 total, 8 up, 8 in 2024-04-14T15:11:26.622 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:11:26 smithi092 podman[67889]: 2024-04-14 15:11:26.324962159 +0000 UTC m=+3.724730521 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-04-14T15:11:26.622 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:11:26 smithi092 podman[67889]: 2024-04-14 15:11:26.440477408 +0000 UTC m=+3.840245798 container create 3096ddbef6327091889e6a95eb5d901afc92a9b765fdd00a4d9350e17718db9d (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-14T15:11:26.992 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":46,"stamp":"2024-04-14T15:11:25.771659+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1029,"num_read_kb":875,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":1771324,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":608,"ondisk_log_size":608,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":236900,"kb_used_data":8252,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":749724316,"statfs":{"total":767960285184,"available":767717699584,"internally_reserved":0,"allocated":8450048,"data_stored":4708248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":234016306},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.671025"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.312324+0000","last_change":"2024-04-14T15:09:49.130210+0000","last_active":"2024-04-14T15:11:22.312324+0000","last_peered":"2024-04-14T15:11:22.312324+0000","last_clean":"2024-04-14T15:11:22.312324+0000","last_became_active":"2024-04-14T15:09:49.129636+0000","last_became_peered":"2024-04-14T15:09:49.129636+0000","last_unstale":"2024-04-14T15:11:22.312324+0000","last_undegraded":"2024-04-14T15:11:22.312324+0000","last_fullsized":"2024-04-14T15:11:22.312324+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.265140+0000","last_change":"2024-04-14T15:09:47.043210+0000","last_active":"2024-04-14T15:11:22.265140+0000","last_peered":"2024-04-14T15:11:22.265140+0000","last_clean":"2024-04-14T15:11:22.265140+0000","last_became_active":"2024-04-14T15:09:47.042538+0000","last_became_peered":"2024-04-14T15:09:47.042538+0000","last_unstale":"2024-04-14T15:11:22.265140+0000","last_undegraded":"2024-04-14T15:11:22.265140+0000","last_fullsized":"2024-04-14T15:11:22.265140+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:09:46.023663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":69,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.896984+0000","last_change":"2024-04-14T15:11:22.897018+0000","last_active":"2024-04-14T15:11:22.896984+0000","last_peered":"2024-04-14T15:11:22.896984+0000","last_clean":"2024-04-14T15:11:22.896984+0000","last_became_active":"2024-04-14T15:09:43.017351+0000","last_became_peered":"2024-04-14T15:09:43.017351+0000","last_unstale":"2024-04-14T15:11:22.896984+0000","last_undegraded":"2024-04-14T15:11:22.896984+0000","last_fullsized":"2024-04-14T15:11:22.896984+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:22.896926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:22.896926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:30.962446+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"72'15","reported_seq":82,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.312753+0000","last_change":"2024-04-14T15:09:45.034424+0000","last_active":"2024-04-14T15:11:22.312753+0000","last_peered":"2024-04-14T15:11:22.312753+0000","last_clean":"2024-04-14T15:11:22.312753+0000","last_became_active":"2024-04-14T15:09:45.034077+0000","last_became_peered":"2024-04-14T15:09:45.034077+0000","last_unstale":"2024-04-14T15:11:22.312753+0000","last_undegraded":"2024-04-14T15:11:22.312753+0000","last_fullsized":"2024-04-14T15:11:22.312753+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.265491+0000","last_change":"2024-04-14T15:09:49.059264+0000","last_active":"2024-04-14T15:11:22.265491+0000","last_peered":"2024-04-14T15:11:22.265491+0000","last_clean":"2024-04-14T15:11:22.265491+0000","last_became_active":"2024-04-14T15:09:49.058904+0000","last_became_peered":"2024-04-14T15:09:49.058904+0000","last_unstale":"2024-04-14T15:11:22.265491+0000","last_undegraded":"2024-04-14T15:11:22.265491+0000","last_fullsized":"2024-04-14T15:11:22.265491+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.266399+0000","last_change":"2024-04-14T15:09:47.041785+0000","last_active":"2024-04-14T15:11:22.266399+0000","last_peered":"2024-04-14T15:11:22.266399+0000","last_clean":"2024-04-14T15:11:22.266399+0000","last_became_active":"2024-04-14T15:09:47.041486+0000","last_became_peered":"2024-04-14T15:09:47.041486+0000","last_unstale":"2024-04-14T15:11:22.266399+0000","last_undegraded":"2024-04-14T15:11:22.266399+0000","last_fullsized":"2024-04-14T15:11:22.266399+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:09:46.023663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":61,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.313075+0000","last_change":"2024-04-14T15:09:43.023226+0000","last_active":"2024-04-14T15:11:22.313075+0000","last_peered":"2024-04-14T15:11:22.313075+0000","last_clean":"2024-04-14T15:11:22.313075+0000","last_became_active":"2024-04-14T15:09:43.022938+0000","last_became_peered":"2024-04-14T15:09:43.022938+0000","last_unstale":"2024-04-14T15:11:22.313075+0000","last_undegraded":"2024-04-14T15:11:22.313075+0000","last_fullsized":"2024-04-14T15:11:22.313075+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:09:42.004317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"72'9","reported_seq":73,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.265318+0000","last_change":"2024-04-14T15:09:45.032381+0000","last_active":"2024-04-14T15:11:22.265318+0000","last_peered":"2024-04-14T15:11:22.265318+0000","last_clean":"2024-04-14T15:11:22.265318+0000","last_became_active":"2024-04-14T15:09:45.032079+0000","last_became_peered":"2024-04-14T15:09:45.032079+0000","last_unstale":"2024-04-14T15:11:22.265318+0000","last_undegraded":"2024-04-14T15:11:22.265318+0000","last_fullsized":"2024-04-14T15:11:22.265318+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.313923+0000","last_change":"2024-04-14T15:11:03.745753+0000","last_active":"2024-04-14T15:11:22.313923+0000","last_peered":"2024-04-14T15:11:22.313923+0000","last_clean":"2024-04-14T15:11:22.313923+0000","last_became_active":"2024-04-14T15:09:49.069590+0000","last_became_peered":"2024-04-14T15:09:49.069590+0000","last_unstale":"2024-04-14T15:11:22.313923+0000","last_undegraded":"2024-04-14T15:11:22.313923+0000","last_fullsized":"2024-04-14T15:11:22.313923+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:03.745644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:03.745644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:29.376857+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":61,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.313134+0000","last_change":"2024-04-14T15:11:04.881923+0000","last_active":"2024-04-14T15:11:22.313134+0000","last_peered":"2024-04-14T15:11:22.313134+0000","last_clean":"2024-04-14T15:11:22.313134+0000","last_became_active":"2024-04-14T15:09:47.046517+0000","last_became_peered":"2024-04-14T15:09:47.046517+0000","last_unstale":"2024-04-14T15:11:22.313134+0000","last_undegraded":"2024-04-14T15:11:22.313134+0000","last_fullsized":"2024-04-14T15:11:22.313134+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:04.881846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:04.881846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:24.065968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":69,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.314051+0000","last_change":"2024-04-14T15:10:43.084925+0000","last_active":"2024-04-14T15:11:22.314051+0000","last_peered":"2024-04-14T15:11:22.314051+0000","last_clean":"2024-04-14T15:11:22.314051+0000","last_became_active":"2024-04-14T15:09:43.021986+0000","last_became_peered":"2024-04-14T15:09:43.021986+0000","last_unstale":"2024-04-14T15:11:22.314051+0000","last_undegraded":"2024-04-14T15:11:22.314051+0000","last_fullsized":"2024-04-14T15:11:22.314051+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:43.084820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:10:43.084820+0000","last_clean_scrub_stamp":"2024-04-14T15:10:43.084820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:55.633388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"87'7","reported_seq":77,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.265685+0000","last_change":"2024-04-14T15:11:10.510929+0000","last_active":"2024-04-14T15:11:22.265685+0000","last_peered":"2024-04-14T15:11:22.265685+0000","last_clean":"2024-04-14T15:11:22.265685+0000","last_became_active":"2024-04-14T15:09:45.028667+0000","last_became_peered":"2024-04-14T15:09:45.028667+0000","last_unstale":"2024-04-14T15:11:22.265685+0000","last_undegraded":"2024-04-14T15:11:22.265685+0000","last_fullsized":"2024-04-14T15:11:22.265685+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'7","last_scrub_stamp":"2024-04-14T15:11:10.510785+0000","last_deep_scrub":"87'7","last_deep_scrub_stamp":"2024-04-14T15:11:10.510785+0000","last_clean_scrub_stamp":"2024-04-14T15:11:10.510785+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:17.842170+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.312184+0000","last_change":"2024-04-14T15:10:54.765733+0000","last_active":"2024-04-14T15:11:22.312184+0000","last_peered":"2024-04-14T15:11:22.312184+0000","last_clean":"2024-04-14T15:11:22.312184+0000","last_became_active":"2024-04-14T15:09:49.061756+0000","last_became_peered":"2024-04-14T15:09:49.061756+0000","last_unstale":"2024-04-14T15:11:22.312184+0000","last_undegraded":"2024-04-14T15:11:22.312184+0000","last_fullsized":"2024-04-14T15:11:22.312184+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:54.765650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:10:54.765650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:59.282104+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.312908+0000","last_change":"2024-04-14T15:09:47.042682+0000","last_active":"2024-04-14T15:11:22.312908+0000","last_peered":"2024-04-14T15:11:22.312908+0000","last_clean":"2024-04-14T15:11:22.312908+0000","last_became_active":"2024-04-14T15:09:47.042384+0000","last_became_peered":"2024-04-14T15:09:47.042384+0000","last_unstale":"2024-04-14T15:11:22.312908+0000","last_undegraded":"2024-04-14T15:11:22.312908+0000","last_fullsized":"2024-04-14T15:11:22.312908+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:09:46.023663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.265516+0000","last_change":"2024-04-14T15:10:07.764431+0000","last_active":"2024-04-14T15:11:22.265516+0000","last_peered":"2024-04-14T15:11:22.265516+0000","last_clean":"2024-04-14T15:11:22.265516+0000","last_became_active":"2024-04-14T15:10:07.764086+0000","last_became_peered":"2024-04-14T15:10:07.764086+0000","last_unstale":"2024-04-14T15:11:22.265516+0000","last_undegraded":"2024-04-14T15:11:22.265516+0000","last_fullsized":"2024-04-14T15:11:22.265516+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:09:42.004317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"72'9","reported_seq":81,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.265545+0000","last_change":"2024-04-14T15:11:17.667793+0000","last_active":"2024-04-14T15:11:22.265545+0000","last_peered":"2024-04-14T15:11:22.265545+0000","last_clean":"2024-04-14T15:11:22.265545+0000","last_became_active":"2024-04-14T15:09:45.030093+0000","last_became_peered":"2024-04-14T15:09:45.030093+0000","last_unstale":"2024-04-14T15:11:22.265545+0000","last_undegraded":"2024-04-14T15:11:22.265545+0000","last_fullsized":"2024-04-14T15:11:22.265545+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-04-14T15:11:17.667682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:17.667682+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:31.898486+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.313283+0000","last_change":"2024-04-14T15:11:21.463365+0000","last_active":"2024-04-14T15:11:22.313283+0000","last_peered":"2024-04-14T15:11:22.313283+0000","last_clean":"2024-04-14T15:11:22.313283+0000","last_became_active":"2024-04-14T15:09:49.128007+0000","last_became_peered":"2024-04-14T15:09:49.128007+0000","last_unstale":"2024-04-14T15:11:22.313283+0000","last_undegraded":"2024-04-14T15:11:22.313283+0000","last_fullsized":"2024-04-14T15:11:22.313283+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:21.463285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:21.463285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:34.661425+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":61,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.265809+0000","last_change":"2024-04-14T15:10:48.466668+0000","last_active":"2024-04-14T15:11:22.265809+0000","last_peered":"2024-04-14T15:11:22.265809+0000","last_clean":"2024-04-14T15:11:22.265809+0000","last_became_active":"2024-04-14T15:09:47.048045+0000","last_became_peered":"2024-04-14T15:09:47.048045+0000","last_unstale":"2024-04-14T15:11:22.265809+0000","last_undegraded":"2024-04-14T15:11:22.265809+0000","last_fullsized":"2024-04-14T15:11:22.265809+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:48.466556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:10:48.466556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:50.564963+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":69,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.476164+0000","last_change":"2024-04-14T15:11:22.476202+0000","last_active":"2024-04-14T15:11:22.476164+0000","last_peered":"2024-04-14T15:11:22.476164+0000","last_clean":"2024-04-14T15:11:22.476164+0000","last_became_active":"2024-04-14T15:09:43.024853+0000","last_became_peered":"2024-04-14T15:09:43.024853+0000","last_unstale":"2024-04-14T15:11:22.476164+0000","last_undegraded":"2024-04-14T15:11:22.476164+0000","last_fullsized":"2024-04-14T15:11:22.476164+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:22.476087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:22.476087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:24.841050+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"72'12","reported_seq":88,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.312720+0000","last_change":"2024-04-14T15:11:14.461199+0000","last_active":"2024-04-14T15:11:22.312720+0000","last_peered":"2024-04-14T15:11:22.312720+0000","last_clean":"2024-04-14T15:11:22.312720+0000","last_became_active":"2024-04-14T15:09:45.037173+0000","last_became_peered":"2024-04-14T15:09:45.037173+0000","last_unstale":"2024-04-14T15:11:22.312720+0000","last_undegraded":"2024-04-14T15:11:22.312720+0000","last_fullsized":"2024-04-14T15:11:22.312720+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'12","last_scrub_stamp":"2024-04-14T15:11:14.461100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:14.461100+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:25.102035+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.265964+0000","last_change":"2024-04-14T15:11:04.539877+0000","last_active":"2024-04-14T15:11:22.265964+0000","last_peered":"2024-04-14T15:11:22.265964+0000","last_clean":"2024-04-14T15:11:22.265964+0000","last_became_active":"2024-04-14T15:09:49.127979+0000","last_became_peered":"2024-04-14T15:09:49.127979+0000","last_unstale":"2024-04-14T15:11:22.265964+0000","last_undegraded":"2024-04-14T15:11:22.265964+0000","last_fullsized":"2024-04-14T15:11:22.265964+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:04.539789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:04.539789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:16.636271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":61,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.267523+0000","last_change":"2024-04-14T15:11:16.297521+0000","last_active":"2024-04-14T15:11:22.267523+0000","last_peered":"2024-04-14T15:11:22.267523+0000","last_clean":"2024-04-14T15:11:22.267523+0000","last_became_active":"2024-04-14T15:09:47.042733+0000","last_became_peered":"2024-04-14T15:09:47.042733+0000","last_unstale":"2024-04-14T15:11:22.267523+0000","last_undegraded":"2024-04-14T15:11:22.267523+0000","last_fullsized":"2024-04-14T15:11:22.267523+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:16.297432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:16.297432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:20.307091+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":69,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.313647+0000","last_change":"2024-04-14T15:11:19.182322+0000","last_active":"2024-04-14T15:11:22.313647+0000","last_peered":"2024-04-14T15:11:22.313647+0000","last_clean":"2024-04-14T15:11:22.313647+0000","last_became_active":"2024-04-14T15:09:43.021494+0000","last_became_peered":"2024-04-14T15:09:43.021494+0000","last_unstale":"2024-04-14T15:11:22.313647+0000","last_undegraded":"2024-04-14T15:11:22.313647+0000","last_fullsized":"2024-04-14T15:11:22.313647+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:19.182163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:19.182163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:36.022685+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"72'15","reported_seq":90,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.268166+0000","last_change":"2024-04-14T15:11:18.790209+0000","last_active":"2024-04-14T15:11:22.268166+0000","last_peered":"2024-04-14T15:11:22.268166+0000","last_clean":"2024-04-14T15:11:22.268166+0000","last_became_active":"2024-04-14T15:09:45.032887+0000","last_became_peered":"2024-04-14T15:09:45.032887+0000","last_unstale":"2024-04-14T15:11:22.268166+0000","last_undegraded":"2024-04-14T15:11:22.268166+0000","last_fullsized":"2024-04-14T15:11:22.268166+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-04-14T15:11:18.790117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:18.790117+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:48.354430+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.267865+0000","last_change":"2024-04-14T15:11:21.247809+0000","last_active":"2024-04-14T15:11:22.267865+0000","last_peered":"2024-04-14T15:11:22.267865+0000","last_clean":"2024-04-14T15:11:22.267865+0000","last_became_active":"2024-04-14T15:09:49.057545+0000","last_became_peered":"2024-04-14T15:09:49.057545+0000","last_unstale":"2024-04-14T15:11:22.267865+0000","last_undegraded":"2024-04-14T15:11:22.267865+0000","last_fullsized":"2024-04-14T15:11:22.267865+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:21.247695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:21.247695+0000","last_clean_scrub_stamp":"2024-04-14T15:11:21.247695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:45.860565+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.312624+0000","last_change":"2024-04-14T15:09:47.046035+0000","last_active":"2024-04-14T15:11:22.312624+0000","last_peered":"2024-04-14T15:11:22.312624+0000","last_clean":"2024-04-14T15:11:22.312624+0000","last_became_active":"2024-04-14T15:09:47.045730+0000","last_became_peered":"2024-04-14T15:09:47.045730+0000","last_unstale":"2024-04-14T15:11:22.312624+0000","last_undegraded":"2024-04-14T15:11:22.312624+0000","last_fullsized":"2024-04-14T15:11:22.312624+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:09:46.023663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"65'2","reported_seq":76,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.312372+0000","last_change":"2024-04-14T15:10:57.406574+0000","last_active":"2024-04-14T15:11:22.312372+0000","last_peered":"2024-04-14T15:11:22.312372+0000","last_clean":"2024-04-14T15:11:22.312372+0000","last_became_active":"2024-04-14T15:09:43.020401+0000","last_became_peered":"2024-04-14T15:09:43.020401+0000","last_unstale":"2024-04-14T15:11:22.312372+0000","last_undegraded":"2024-04-14T15:11:22.312372+0000","last_fullsized":"2024-04-14T15:11:22.312372+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-04-14T15:10:57.406478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:10:57.406478+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:25.577150+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"72'11","reported_seq":94,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.313523+0000","last_change":"2024-04-14T15:10:07.771479+0000","last_active":"2024-04-14T15:11:22.313523+0000","last_peered":"2024-04-14T15:11:22.313523+0000","last_clean":"2024-04-14T15:11:22.313523+0000","last_became_active":"2024-04-14T15:10:07.764183+0000","last_became_peered":"2024-04-14T15:10:07.764183+0000","last_unstale":"2024-04-14T15:11:22.313523+0000","last_undegraded":"2024-04-14T15:11:22.313523+0000","last_fullsized":"2024-04-14T15:11:22.313523+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"72'1","reported_seq":58,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.313864+0000","last_change":"2024-04-14T15:10:59.889342+0000","last_active":"2024-04-14T15:11:22.313864+0000","last_peered":"2024-04-14T15:11:22.313864+0000","last_clean":"2024-04-14T15:11:22.313864+0000","last_became_active":"2024-04-14T15:09:49.062051+0000","last_became_peered":"2024-04-14T15:09:49.062051+0000","last_unstale":"2024-04-14T15:11:22.313864+0000","last_undegraded":"2024-04-14T15:11:22.313864+0000","last_fullsized":"2024-04-14T15:11:22.313864+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-04-14T15:10:59.889236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:10:59.889236+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:03.689276+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"72'8","reported_seq":93,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.313656+0000","last_change":"2024-04-14T15:10:58.268612+0000","last_active":"2024-04-14T15:11:22.313656+0000","last_peered":"2024-04-14T15:11:22.313656+0000","last_clean":"2024-04-14T15:11:22.313656+0000","last_became_active":"2024-04-14T15:09:47.048216+0000","last_became_peered":"2024-04-14T15:09:47.048216+0000","last_unstale":"2024-04-14T15:11:22.313656+0000","last_undegraded":"2024-04-14T15:11:22.313656+0000","last_fullsized":"2024-04-14T15:11:22.313656+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:10:58.268488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:10:58.268488+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:04.149311+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":69,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.312331+0000","last_change":"2024-04-14T15:10:52.882061+0000","last_active":"2024-04-14T15:11:22.312331+0000","last_peered":"2024-04-14T15:11:22.312331+0000","last_clean":"2024-04-14T15:11:22.312331+0000","last_became_active":"2024-04-14T15:09:43.023257+0000","last_became_peered":"2024-04-14T15:09:43.023257+0000","last_unstale":"2024-04-14T15:11:22.312331+0000","last_undegraded":"2024-04-14T15:11:22.312331+0000","last_fullsized":"2024-04-14T15:11:22.312331+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:52.881916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:10:52.881916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:58.458431+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"72'10","reported_seq":80,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.312602+0000","last_change":"2024-04-14T15:11:17.863747+0000","last_active":"2024-04-14T15:11:22.312602+0000","last_peered":"2024-04-14T15:11:22.312602+0000","last_clean":"2024-04-14T15:11:22.312602+0000","last_became_active":"2024-04-14T15:09:45.181456+0000","last_became_peered":"2024-04-14T15:09:45.181456+0000","last_unstale":"2024-04-14T15:11:22.312602+0000","last_undegraded":"2024-04-14T15:11:22.312602+0000","last_fullsized":"2024-04-14T15:11:22.312602+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-04-14T15:11:17.863645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:17.863645+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:27.587257+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.312772+0000","last_change":"2024-04-14T15:11:17.499797+0000","last_active":"2024-04-14T15:11:22.312772+0000","last_peered":"2024-04-14T15:11:22.312772+0000","last_clean":"2024-04-14T15:11:22.312772+0000","last_became_active":"2024-04-14T15:09:49.129392+0000","last_became_peered":"2024-04-14T15:09:49.129392+0000","last_unstale":"2024-04-14T15:11:22.312772+0000","last_undegraded":"2024-04-14T15:11:22.312772+0000","last_fullsized":"2024-04-14T15:11:22.312772+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:17.499708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:17.499708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:24.507730+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":61,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.312726+0000","last_change":"2024-04-14T15:11:05.917054+0000","last_active":"2024-04-14T15:11:22.312726+0000","last_peered":"2024-04-14T15:11:22.312726+0000","last_clean":"2024-04-14T15:11:22.312726+0000","last_became_active":"2024-04-14T15:09:47.044665+0000","last_became_peered":"2024-04-14T15:09:47.044665+0000","last_unstale":"2024-04-14T15:11:22.312726+0000","last_undegraded":"2024-04-14T15:11:22.312726+0000","last_fullsized":"2024-04-14T15:11:22.312726+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:05.916966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:05.916966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:35.330112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":69,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.313784+0000","last_change":"2024-04-14T15:10:54.239518+0000","last_active":"2024-04-14T15:11:22.313784+0000","last_peered":"2024-04-14T15:11:22.313784+0000","last_clean":"2024-04-14T15:11:22.313784+0000","last_became_active":"2024-04-14T15:09:43.023220+0000","last_became_peered":"2024-04-14T15:09:43.023220+0000","last_unstale":"2024-04-14T15:11:22.313784+0000","last_undegraded":"2024-04-14T15:11:22.313784+0000","last_fullsized":"2024-04-14T15:11:22.313784+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:54.239409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:10:54.239409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:11.539313+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"72'11","reported_seq":84,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.312673+0000","last_change":"2024-04-14T15:11:06.492784+0000","last_active":"2024-04-14T15:11:22.312673+0000","last_peered":"2024-04-14T15:11:22.312673+0000","last_clean":"2024-04-14T15:11:22.312673+0000","last_became_active":"2024-04-14T15:09:45.180332+0000","last_became_peered":"2024-04-14T15:09:45.180332+0000","last_unstale":"2024-04-14T15:11:22.312673+0000","last_undegraded":"2024-04-14T15:11:22.312673+0000","last_fullsized":"2024-04-14T15:11:22.312673+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-04-14T15:11:06.492677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:06.492677+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:24.577564+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.312914+0000","last_change":"2024-04-14T15:11:07.901838+0000","last_active":"2024-04-14T15:11:22.312914+0000","last_peered":"2024-04-14T15:11:22.312914+0000","last_clean":"2024-04-14T15:11:22.312914+0000","last_became_active":"2024-04-14T15:09:49.061359+0000","last_became_peered":"2024-04-14T15:09:49.061359+0000","last_unstale":"2024-04-14T15:11:22.312914+0000","last_undegraded":"2024-04-14T15:11:22.312914+0000","last_fullsized":"2024-04-14T15:11:22.312914+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:07.901761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:07.901761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:35.316850+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":61,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.313006+0000","last_change":"2024-04-14T15:10:50.229008+0000","last_active":"2024-04-14T15:11:22.313006+0000","last_peered":"2024-04-14T15:11:22.313006+0000","last_clean":"2024-04-14T15:11:22.313006+0000","last_became_active":"2024-04-14T15:09:47.041074+0000","last_became_peered":"2024-04-14T15:09:47.041074+0000","last_unstale":"2024-04-14T15:11:22.313006+0000","last_undegraded":"2024-04-14T15:11:22.313006+0000","last_fullsized":"2024-04-14T15:11:22.313006+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:50.228903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:10:50.228903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:17.158842+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":69,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.313644+0000","last_change":"2024-04-14T15:11:08.712429+0000","last_active":"2024-04-14T15:11:22.313644+0000","last_peered":"2024-04-14T15:11:22.313644+0000","last_clean":"2024-04-14T15:11:22.313644+0000","last_became_active":"2024-04-14T15:09:43.026786+0000","last_became_peered":"2024-04-14T15:09:43.026786+0000","last_unstale":"2024-04-14T15:11:22.313644+0000","last_undegraded":"2024-04-14T15:11:22.313644+0000","last_fullsized":"2024-04-14T15:11:22.313644+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:08.712289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:08.712289+0000","last_clean_scrub_stamp":"2024-04-14T15:11:08.712289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:11.190004+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"72'4","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.312044+0000","last_change":"2024-04-14T15:09:45.039145+0000","last_active":"2024-04-14T15:11:22.312044+0000","last_peered":"2024-04-14T15:11:22.312044+0000","last_clean":"2024-04-14T15:11:22.312044+0000","last_became_active":"2024-04-14T15:09:45.038815+0000","last_became_peered":"2024-04-14T15:09:45.038815+0000","last_unstale":"2024-04-14T15:11:22.312044+0000","last_undegraded":"2024-04-14T15:11:22.312044+0000","last_fullsized":"2024-04-14T15:11:22.312044+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.312211+0000","last_change":"2024-04-14T15:09:49.066223+0000","last_active":"2024-04-14T15:11:22.312211+0000","last_peered":"2024-04-14T15:11:22.312211+0000","last_clean":"2024-04-14T15:11:22.312211+0000","last_became_active":"2024-04-14T15:09:49.065896+0000","last_became_peered":"2024-04-14T15:09:49.065896+0000","last_unstale":"2024-04-14T15:11:22.312211+0000","last_undegraded":"2024-04-14T15:11:22.312211+0000","last_fullsized":"2024-04-14T15:11:22.312211+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.266906+0000","last_change":"2024-04-14T15:09:47.041972+0000","last_active":"2024-04-14T15:11:22.266906+0000","last_peered":"2024-04-14T15:11:22.266906+0000","last_clean":"2024-04-14T15:11:22.266906+0000","last_became_active":"2024-04-14T15:09:47.041683+0000","last_became_peered":"2024-04-14T15:09:47.041683+0000","last_unstale":"2024-04-14T15:11:22.266906+0000","last_undegraded":"2024-04-14T15:11:22.266906+0000","last_fullsized":"2024-04-14T15:11:22.266906+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:09:46.023663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":61,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.267615+0000","last_change":"2024-04-14T15:09:43.023867+0000","last_active":"2024-04-14T15:11:22.267615+0000","last_peered":"2024-04-14T15:11:22.267615+0000","last_clean":"2024-04-14T15:11:22.267615+0000","last_became_active":"2024-04-14T15:09:43.023583+0000","last_became_peered":"2024-04-14T15:09:43.023583+0000","last_unstale":"2024-04-14T15:11:22.267615+0000","last_undegraded":"2024-04-14T15:11:22.267615+0000","last_fullsized":"2024-04-14T15:11:22.267615+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:09:42.004317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"72'11","reported_seq":76,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.265856+0000","last_change":"2024-04-14T15:09:45.026357+0000","last_active":"2024-04-14T15:11:22.265856+0000","last_peered":"2024-04-14T15:11:22.265856+0000","last_clean":"2024-04-14T15:11:22.265856+0000","last_became_active":"2024-04-14T15:09:45.026065+0000","last_became_peered":"2024-04-14T15:09:45.026065+0000","last_unstale":"2024-04-14T15:11:22.265856+0000","last_undegraded":"2024-04-14T15:11:22.265856+0000","last_fullsized":"2024-04-14T15:11:22.265856+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.311816+0000","last_change":"2024-04-14T15:10:56.820210+0000","last_active":"2024-04-14T15:11:22.311816+0000","last_peered":"2024-04-14T15:11:22.311816+0000","last_clean":"2024-04-14T15:11:22.311816+0000","last_became_active":"2024-04-14T15:09:49.059711+0000","last_became_peered":"2024-04-14T15:09:49.059711+0000","last_unstale":"2024-04-14T15:11:22.311816+0000","last_undegraded":"2024-04-14T15:11:22.311816+0000","last_fullsized":"2024-04-14T15:11:22.311816+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:56.820123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:10:56.820123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:08.868464+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":61,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.312606+0000","last_change":"2024-04-14T15:11:10.455426+0000","last_active":"2024-04-14T15:11:22.312606+0000","last_peered":"2024-04-14T15:11:22.312606+0000","last_clean":"2024-04-14T15:11:22.312606+0000","last_became_active":"2024-04-14T15:09:47.044891+0000","last_became_peered":"2024-04-14T15:09:47.044891+0000","last_unstale":"2024-04-14T15:11:22.312606+0000","last_undegraded":"2024-04-14T15:11:22.312606+0000","last_fullsized":"2024-04-14T15:11:22.312606+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:10.455300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:10.455300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:32.884911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":69,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.312622+0000","last_change":"2024-04-14T15:11:03.304640+0000","last_active":"2024-04-14T15:11:22.312622+0000","last_peered":"2024-04-14T15:11:22.312622+0000","last_clean":"2024-04-14T15:11:22.312622+0000","last_became_active":"2024-04-14T15:09:43.022480+0000","last_became_peered":"2024-04-14T15:09:43.022480+0000","last_unstale":"2024-04-14T15:11:22.312622+0000","last_undegraded":"2024-04-14T15:11:22.312622+0000","last_fullsized":"2024-04-14T15:11:22.312622+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:03.304541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:03.304541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:22.825961+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"72'9","reported_seq":73,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.266834+0000","last_change":"2024-04-14T15:09:45.026649+0000","last_active":"2024-04-14T15:11:22.266834+0000","last_peered":"2024-04-14T15:11:22.266834+0000","last_clean":"2024-04-14T15:11:22.266834+0000","last_became_active":"2024-04-14T15:09:45.026363+0000","last_became_peered":"2024-04-14T15:09:45.026363+0000","last_unstale":"2024-04-14T15:11:22.266834+0000","last_undegraded":"2024-04-14T15:11:22.266834+0000","last_fullsized":"2024-04-14T15:11:22.266834+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.266073+0000","last_change":"2024-04-14T15:11:15.644792+0000","last_active":"2024-04-14T15:11:22.266073+0000","last_peered":"2024-04-14T15:11:22.266073+0000","last_clean":"2024-04-14T15:11:22.266073+0000","last_became_active":"2024-04-14T15:09:49.065242+0000","last_became_peered":"2024-04-14T15:09:49.065242+0000","last_unstale":"2024-04-14T15:11:22.266073+0000","last_undegraded":"2024-04-14T15:11:22.266073+0000","last_fullsized":"2024-04-14T15:11:22.266073+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:15.644714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:15.644714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:19.718378+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"72'8","reported_seq":93,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.313140+0000","last_change":"2024-04-14T15:11:09.485228+0000","last_active":"2024-04-14T15:11:22.313140+0000","last_peered":"2024-04-14T15:11:22.313140+0000","last_clean":"2024-04-14T15:11:22.313140+0000","last_became_active":"2024-04-14T15:09:47.047072+0000","last_became_peered":"2024-04-14T15:09:47.047072+0000","last_unstale":"2024-04-14T15:11:22.313140+0000","last_undegraded":"2024-04-14T15:11:22.313140+0000","last_fullsized":"2024-04-14T15:11:22.313140+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:11:09.485146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:09.485146+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:34.775008+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":69,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.312993+0000","last_change":"2024-04-14T15:11:10.762163+0000","last_active":"2024-04-14T15:11:22.312993+0000","last_peered":"2024-04-14T15:11:22.312993+0000","last_clean":"2024-04-14T15:11:22.312993+0000","last_became_active":"2024-04-14T15:09:43.019301+0000","last_became_peered":"2024-04-14T15:09:43.019301+0000","last_unstale":"2024-04-14T15:11:22.312993+0000","last_undegraded":"2024-04-14T15:11:22.312993+0000","last_fullsized":"2024-04-14T15:11:22.312993+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:10.762080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:10.762080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:13.869531+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"72'9","reported_seq":81,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.314391+0000","last_change":"2024-04-14T15:10:56.858475+0000","last_active":"2024-04-14T15:11:22.314391+0000","last_peered":"2024-04-14T15:11:22.314391+0000","last_clean":"2024-04-14T15:11:22.314391+0000","last_became_active":"2024-04-14T15:09:45.178857+0000","last_became_peered":"2024-04-14T15:09:45.178857+0000","last_unstale":"2024-04-14T15:11:22.314391+0000","last_undegraded":"2024-04-14T15:11:22.314391+0000","last_fullsized":"2024-04-14T15:11:22.314391+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-04-14T15:10:56.858323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:10:56.858323+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:13.702279+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.312063+0000","last_change":"2024-04-14T15:09:49.066448+0000","last_active":"2024-04-14T15:11:22.312063+0000","last_peered":"2024-04-14T15:11:22.312063+0000","last_clean":"2024-04-14T15:11:22.312063+0000","last_became_active":"2024-04-14T15:09:49.066068+0000","last_became_peered":"2024-04-14T15:09:49.066068+0000","last_unstale":"2024-04-14T15:11:22.312063+0000","last_undegraded":"2024-04-14T15:11:22.312063+0000","last_fullsized":"2024-04-14T15:11:22.312063+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"72'8","reported_seq":90,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.314581+0000","last_change":"2024-04-14T15:11:18.667210+0000","last_active":"2024-04-14T15:11:22.314581+0000","last_peered":"2024-04-14T15:11:22.314581+0000","last_clean":"2024-04-14T15:11:22.314581+0000","last_became_active":"2024-04-14T15:09:47.036688+0000","last_became_peered":"2024-04-14T15:09:47.036688+0000","last_unstale":"2024-04-14T15:11:22.314581+0000","last_undegraded":"2024-04-14T15:11:22.314581+0000","last_fullsized":"2024-04-14T15:11:22.314581+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:11:18.667049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:18.667049+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:36.115547+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":61,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.314333+0000","last_change":"2024-04-14T15:09:43.028602+0000","last_active":"2024-04-14T15:11:22.314333+0000","last_peered":"2024-04-14T15:11:22.314333+0000","last_clean":"2024-04-14T15:11:22.314333+0000","last_became_active":"2024-04-14T15:09:43.027776+0000","last_became_peered":"2024-04-14T15:09:43.027776+0000","last_unstale":"2024-04-14T15:11:22.314333+0000","last_undegraded":"2024-04-14T15:11:22.314333+0000","last_fullsized":"2024-04-14T15:11:22.314333+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:09:42.004317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"72'10","reported_seq":80,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.312889+0000","last_change":"2024-04-14T15:11:11.458687+0000","last_active":"2024-04-14T15:11:22.312889+0000","last_peered":"2024-04-14T15:11:22.312889+0000","last_clean":"2024-04-14T15:11:22.312889+0000","last_became_active":"2024-04-14T15:09:45.179165+0000","last_became_peered":"2024-04-14T15:09:45.179165+0000","last_unstale":"2024-04-14T15:11:22.312889+0000","last_undegraded":"2024-04-14T15:11:22.312889+0000","last_fullsized":"2024-04-14T15:11:22.312889+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-04-14T15:11:11.458594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:11.458594+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:19.766366+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.314019+0000","last_change":"2024-04-14T15:09:49.130725+0000","last_active":"2024-04-14T15:11:22.314019+0000","last_peered":"2024-04-14T15:11:22.314019+0000","last_clean":"2024-04-14T15:11:22.314019+0000","last_became_active":"2024-04-14T15:09:49.130438+0000","last_became_peered":"2024-04-14T15:09:49.130438+0000","last_unstale":"2024-04-14T15:11:22.314019+0000","last_undegraded":"2024-04-14T15:11:22.314019+0000","last_fullsized":"2024-04-14T15:11:22.314019+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":61,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.313564+0000","last_change":"2024-04-14T15:10:53.900979+0000","last_active":"2024-04-14T15:11:22.313564+0000","last_peered":"2024-04-14T15:11:22.313564+0000","last_clean":"2024-04-14T15:11:22.313564+0000","last_became_active":"2024-04-14T15:09:47.045349+0000","last_became_peered":"2024-04-14T15:09:47.045349+0000","last_unstale":"2024-04-14T15:11:22.313564+0000","last_undegraded":"2024-04-14T15:11:22.313564+0000","last_fullsized":"2024-04-14T15:11:22.313564+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:53.900878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:10:53.900878+0000","last_clean_scrub_stamp":"2024-04-14T15:10:53.900878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:12.123204+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":69,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.313089+0000","last_change":"2024-04-14T15:11:18.194434+0000","last_active":"2024-04-14T15:11:22.313089+0000","last_peered":"2024-04-14T15:11:22.313089+0000","last_clean":"2024-04-14T15:11:22.313089+0000","last_became_active":"2024-04-14T15:09:43.018987+0000","last_became_peered":"2024-04-14T15:09:43.018987+0000","last_unstale":"2024-04-14T15:11:22.313089+0000","last_undegraded":"2024-04-14T15:11:22.313089+0000","last_fullsized":"2024-04-14T15:11:22.313089+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:18.194339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:18.194339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:27.683203+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"72'6","reported_seq":66,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.313253+0000","last_change":"2024-04-14T15:09:45.180445+0000","last_active":"2024-04-14T15:11:22.313253+0000","last_peered":"2024-04-14T15:11:22.313253+0000","last_clean":"2024-04-14T15:11:22.313253+0000","last_became_active":"2024-04-14T15:09:45.180037+0000","last_became_peered":"2024-04-14T15:09:45.180037+0000","last_unstale":"2024-04-14T15:11:22.313253+0000","last_undegraded":"2024-04-14T15:11:22.313253+0000","last_fullsized":"2024-04-14T15:11:22.313253+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.268021+0000","last_change":"2024-04-14T15:10:56.886642+0000","last_active":"2024-04-14T15:11:22.268021+0000","last_peered":"2024-04-14T15:11:22.268021+0000","last_clean":"2024-04-14T15:11:22.268021+0000","last_became_active":"2024-04-14T15:09:49.064296+0000","last_became_peered":"2024-04-14T15:09:49.064296+0000","last_unstale":"2024-04-14T15:11:22.268021+0000","last_undegraded":"2024-04-14T15:11:22.268021+0000","last_fullsized":"2024-04-14T15:11:22.268021+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:56.886513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:10:56.886513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:17.056264+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":61,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.313349+0000","last_change":"2024-04-14T15:11:07.485922+0000","last_active":"2024-04-14T15:11:22.313349+0000","last_peered":"2024-04-14T15:11:22.313349+0000","last_clean":"2024-04-14T15:11:22.313349+0000","last_became_active":"2024-04-14T15:09:47.044816+0000","last_became_peered":"2024-04-14T15:09:47.044816+0000","last_unstale":"2024-04-14T15:11:22.313349+0000","last_undegraded":"2024-04-14T15:11:22.313349+0000","last_fullsized":"2024-04-14T15:11:22.313349+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:07.485821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:07.485821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:21.021299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":69,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.267205+0000","last_change":"2024-04-14T15:10:43.844823+0000","last_active":"2024-04-14T15:11:22.267205+0000","last_peered":"2024-04-14T15:11:22.267205+0000","last_clean":"2024-04-14T15:11:22.267205+0000","last_became_active":"2024-04-14T15:09:43.016788+0000","last_became_peered":"2024-04-14T15:09:43.016788+0000","last_unstale":"2024-04-14T15:11:22.267205+0000","last_undegraded":"2024-04-14T15:11:22.267205+0000","last_fullsized":"2024-04-14T15:11:22.267205+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:43.844693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:10:43.844693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:51.339452+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"72'9","reported_seq":73,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.312280+0000","last_change":"2024-04-14T15:09:45.180769+0000","last_active":"2024-04-14T15:11:22.312280+0000","last_peered":"2024-04-14T15:11:22.312280+0000","last_clean":"2024-04-14T15:11:22.312280+0000","last_became_active":"2024-04-14T15:09:45.180375+0000","last_became_peered":"2024-04-14T15:09:45.180375+0000","last_unstale":"2024-04-14T15:11:22.312280+0000","last_undegraded":"2024-04-14T15:11:22.312280+0000","last_fullsized":"2024-04-14T15:11:22.312280+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.312684+0000","last_change":"2024-04-14T15:09:49.069204+0000","last_active":"2024-04-14T15:11:22.312684+0000","last_peered":"2024-04-14T15:11:22.312684+0000","last_clean":"2024-04-14T15:11:22.312684+0000","last_became_active":"2024-04-14T15:09:49.068883+0000","last_became_peered":"2024-04-14T15:09:49.068883+0000","last_unstale":"2024-04-14T15:11:22.312684+0000","last_undegraded":"2024-04-14T15:11:22.312684+0000","last_fullsized":"2024-04-14T15:11:22.312684+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"65'3","reported_seq":79,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.265917+0000","last_change":"2024-04-14T15:09:43.020827+0000","last_active":"2024-04-14T15:11:22.265917+0000","last_peered":"2024-04-14T15:11:22.265917+0000","last_clean":"2024-04-14T15:11:22.265917+0000","last_became_active":"2024-04-14T15:09:43.020490+0000","last_became_peered":"2024-04-14T15:09:43.020490+0000","last_unstale":"2024-04-14T15:11:22.265917+0000","last_undegraded":"2024-04-14T15:11:22.265917+0000","last_fullsized":"2024-04-14T15:11:22.265917+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:09:42.004317+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":528,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"72'12","reported_seq":88,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.265387+0000","last_change":"2024-04-14T15:10:53.448340+0000","last_active":"2024-04-14T15:11:22.265387+0000","last_peered":"2024-04-14T15:11:22.265387+0000","last_clean":"2024-04-14T15:11:22.265387+0000","last_became_active":"2024-04-14T15:09:45.032892+0000","last_became_peered":"2024-04-14T15:09:45.032892+0000","last_unstale":"2024-04-14T15:11:22.265387+0000","last_undegraded":"2024-04-14T15:11:22.265387+0000","last_fullsized":"2024-04-14T15:11:22.265387+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'12","last_scrub_stamp":"2024-04-14T15:10:53.448218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:10:53.448218+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:56.821958+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":61,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.267201+0000","last_change":"2024-04-14T15:11:12.867617+0000","last_active":"2024-04-14T15:11:22.267201+0000","last_peered":"2024-04-14T15:11:22.267201+0000","last_clean":"2024-04-14T15:11:22.267201+0000","last_became_active":"2024-04-14T15:09:47.049005+0000","last_became_peered":"2024-04-14T15:09:47.049005+0000","last_unstale":"2024-04-14T15:11:22.267201+0000","last_undegraded":"2024-04-14T15:11:22.267201+0000","last_fullsized":"2024-04-14T15:11:22.267201+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:12.867489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:12.867489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:42.053319+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.311827+0000","last_change":"2024-04-14T15:11:09.756246+0000","last_active":"2024-04-14T15:11:22.311827+0000","last_peered":"2024-04-14T15:11:22.311827+0000","last_clean":"2024-04-14T15:11:22.311827+0000","last_became_active":"2024-04-14T15:09:49.066648+0000","last_became_peered":"2024-04-14T15:09:49.066648+0000","last_unstale":"2024-04-14T15:11:22.311827+0000","last_undegraded":"2024-04-14T15:11:22.311827+0000","last_fullsized":"2024-04-14T15:11:22.311827+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:09.756164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:09.756164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:32.890427+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":61,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.266976+0000","last_change":"2024-04-14T15:09:43.022909+0000","last_active":"2024-04-14T15:11:22.266976+0000","last_peered":"2024-04-14T15:11:22.266976+0000","last_clean":"2024-04-14T15:11:22.266976+0000","last_became_active":"2024-04-14T15:09:43.022597+0000","last_became_peered":"2024-04-14T15:09:43.022597+0000","last_unstale":"2024-04-14T15:11:22.266976+0000","last_undegraded":"2024-04-14T15:11:22.266976+0000","last_fullsized":"2024-04-14T15:11:22.266976+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:09:42.004317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"72'10","reported_seq":59,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.267611+0000","last_change":"2024-04-14T15:11:17.779528+0000","last_active":"2024-04-14T15:11:22.267611+0000","last_peered":"2024-04-14T15:11:22.267611+0000","last_clean":"2024-04-14T15:11:22.267611+0000","last_became_active":"2024-04-14T15:10:07.764042+0000","last_became_peered":"2024-04-14T15:10:07.764042+0000","last_unstale":"2024-04-14T15:11:22.267611+0000","last_undegraded":"2024-04-14T15:11:22.267611+0000","last_fullsized":"2024-04-14T15:11:22.267611+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-04-14T15:11:17.779383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:17.779383+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:33.747601+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"72'8","reported_seq":90,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.313152+0000","last_change":"2024-04-14T15:11:14.870904+0000","last_active":"2024-04-14T15:11:22.313152+0000","last_peered":"2024-04-14T15:11:22.313152+0000","last_clean":"2024-04-14T15:11:22.313152+0000","last_became_active":"2024-04-14T15:09:47.047699+0000","last_became_peered":"2024-04-14T15:09:47.047699+0000","last_unstale":"2024-04-14T15:11:22.313152+0000","last_undegraded":"2024-04-14T15:11:22.313152+0000","last_fullsized":"2024-04-14T15:11:22.313152+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:11:14.870772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:14.870772+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:19.702141+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.313112+0000","last_change":"2024-04-14T15:09:49.058715+0000","last_active":"2024-04-14T15:11:22.313112+0000","last_peered":"2024-04-14T15:11:22.313112+0000","last_clean":"2024-04-14T15:11:22.313112+0000","last_became_active":"2024-04-14T15:09:49.058387+0000","last_became_peered":"2024-04-14T15:09:49.058387+0000","last_unstale":"2024-04-14T15:11:22.313112+0000","last_undegraded":"2024-04-14T15:11:22.313112+0000","last_fullsized":"2024-04-14T15:11:22.313112+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":69,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.314347+0000","last_change":"2024-04-14T15:11:20.694315+0000","last_active":"2024-04-14T15:11:22.314347+0000","last_peered":"2024-04-14T15:11:22.314347+0000","last_clean":"2024-04-14T15:11:22.314347+0000","last_became_active":"2024-04-14T15:09:43.028362+0000","last_became_peered":"2024-04-14T15:09:43.028362+0000","last_unstale":"2024-04-14T15:11:22.314347+0000","last_undegraded":"2024-04-14T15:11:22.314347+0000","last_fullsized":"2024-04-14T15:11:22.314347+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:20.694225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:20.694225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:45.303474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"72'28","reported_seq":126,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.266923+0000","last_change":"2024-04-14T15:09:45.034501+0000","last_active":"2024-04-14T15:11:22.266923+0000","last_peered":"2024-04-14T15:11:22.266923+0000","last_clean":"2024-04-14T15:11:22.266923+0000","last_became_active":"2024-04-14T15:09:45.034168+0000","last_became_peered":"2024-04-14T15:09:45.034168+0000","last_unstale":"2024-04-14T15:11:22.266923+0000","last_undegraded":"2024-04-14T15:11:22.266923+0000","last_fullsized":"2024-04-14T15:11:22.266923+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":61,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.312739+0000","last_change":"2024-04-14T15:11:15.831911+0000","last_active":"2024-04-14T15:11:22.312739+0000","last_peered":"2024-04-14T15:11:22.312739+0000","last_clean":"2024-04-14T15:11:22.312739+0000","last_became_active":"2024-04-14T15:09:47.040185+0000","last_became_peered":"2024-04-14T15:09:47.040185+0000","last_unstale":"2024-04-14T15:11:22.312739+0000","last_undegraded":"2024-04-14T15:11:22.312739+0000","last_fullsized":"2024-04-14T15:11:22.312739+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:15.831779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:15.831779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:44.452408+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.314143+0000","last_change":"2024-04-14T15:11:00.939052+0000","last_active":"2024-04-14T15:11:22.314143+0000","last_peered":"2024-04-14T15:11:22.314143+0000","last_clean":"2024-04-14T15:11:22.314143+0000","last_became_active":"2024-04-14T15:09:49.130307+0000","last_became_peered":"2024-04-14T15:09:49.130307+0000","last_unstale":"2024-04-14T15:11:22.314143+0000","last_undegraded":"2024-04-14T15:11:22.314143+0000","last_fullsized":"2024-04-14T15:11:22.314143+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:00.938917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:00.938917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:17.427163+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"73'5","reported_seq":174,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.784290+0000","last_change":"2024-04-14T15:11:03.192957+0000","last_active":"2024-04-14T15:11:22.784290+0000","last_peered":"2024-04-14T15:11:22.784290+0000","last_clean":"2024-04-14T15:11:22.784290+0000","last_became_active":"2024-04-14T15:09:43.016544+0000","last_became_peered":"2024-04-14T15:09:43.016544+0000","last_unstale":"2024-04-14T15:11:22.784290+0000","last_undegraded":"2024-04-14T15:11:22.784290+0000","last_fullsized":"2024-04-14T15:11:22.784290+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-04-14T15:11:03.192810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:03.192810+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:21.177179+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00056542099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":95,"num_read_kb":90,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":69,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.267227+0000","last_change":"2024-04-14T15:10:42.819764+0000","last_active":"2024-04-14T15:11:22.267227+0000","last_peered":"2024-04-14T15:11:22.267227+0000","last_clean":"2024-04-14T15:11:22.267227+0000","last_became_active":"2024-04-14T15:09:43.021188+0000","last_became_peered":"2024-04-14T15:09:43.021188+0000","last_unstale":"2024-04-14T15:11:22.267227+0000","last_undegraded":"2024-04-14T15:11:22.267227+0000","last_fullsized":"2024-04-14T15:11:22.267227+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:42.819681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:10:42.819681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:52.363715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"72'13","reported_seq":84,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.267670+0000","last_change":"2024-04-14T15:09:45.034537+0000","last_active":"2024-04-14T15:11:22.267670+0000","last_peered":"2024-04-14T15:11:22.267670+0000","last_clean":"2024-04-14T15:11:22.267670+0000","last_became_active":"2024-04-14T15:09:45.033883+0000","last_became_peered":"2024-04-14T15:09:45.033883+0000","last_unstale":"2024-04-14T15:11:22.267670+0000","last_undegraded":"2024-04-14T15:11:22.267670+0000","last_fullsized":"2024-04-14T15:11:22.267670+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":61,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.267996+0000","last_change":"2024-04-14T15:10:55.841920+0000","last_active":"2024-04-14T15:11:22.267996+0000","last_peered":"2024-04-14T15:11:22.267996+0000","last_clean":"2024-04-14T15:11:22.267996+0000","last_became_active":"2024-04-14T15:09:47.049457+0000","last_became_peered":"2024-04-14T15:09:47.049457+0000","last_unstale":"2024-04-14T15:11:22.267996+0000","last_undegraded":"2024-04-14T15:11:22.267996+0000","last_fullsized":"2024-04-14T15:11:22.267996+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:55.841832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:10:55.841832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:09.497072+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.314204+0000","last_change":"2024-04-14T15:09:49.067392+0000","last_active":"2024-04-14T15:11:22.314204+0000","last_peered":"2024-04-14T15:11:22.314204+0000","last_clean":"2024-04-14T15:11:22.314204+0000","last_became_active":"2024-04-14T15:09:49.067031+0000","last_became_peered":"2024-04-14T15:09:49.067031+0000","last_unstale":"2024-04-14T15:11:22.314204+0000","last_undegraded":"2024-04-14T15:11:22.314204+0000","last_fullsized":"2024-04-14T15:11:22.314204+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"69'2","reported_seq":78,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.312381+0000","last_change":"2024-04-14T15:11:20.500383+0000","last_active":"2024-04-14T15:11:22.312381+0000","last_peered":"2024-04-14T15:11:22.312381+0000","last_clean":"2024-04-14T15:11:22.312381+0000","last_became_active":"2024-04-14T15:09:43.024882+0000","last_became_peered":"2024-04-14T15:09:43.024882+0000","last_unstale":"2024-04-14T15:11:22.312381+0000","last_undegraded":"2024-04-14T15:11:22.312381+0000","last_fullsized":"2024-04-14T15:11:22.312381+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-04-14T15:11:20.500271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:20.500271+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:20.673941+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0058046169999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":69,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.312824+0000","last_change":"2024-04-14T15:10:43.898230+0000","last_active":"2024-04-14T15:11:22.312824+0000","last_peered":"2024-04-14T15:11:22.312824+0000","last_clean":"2024-04-14T15:11:22.312824+0000","last_became_active":"2024-04-14T15:09:43.025421+0000","last_became_peered":"2024-04-14T15:09:43.025421+0000","last_unstale":"2024-04-14T15:11:22.312824+0000","last_undegraded":"2024-04-14T15:11:22.312824+0000","last_fullsized":"2024-04-14T15:11:22.312824+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:43.898117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:10:43.898117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:47.080377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":310,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.312230+0000","last_change":"2024-04-14T15:10:06.160908+0000","last_active":"2024-04-14T15:11:22.312230+0000","last_peered":"2024-04-14T15:11:22.312230+0000","last_clean":"2024-04-14T15:11:22.312230+0000","last_became_active":"2024-04-14T15:09:33.753828+0000","last_became_peered":"2024-04-14T15:09:33.753828+0000","last_unstale":"2024-04-14T15:11:22.312230+0000","last_undegraded":"2024-04-14T15:11:22.312230+0000","last_fullsized":"2024-04-14T15:11:22.312230+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'117","last_scrub_stamp":"2024-04-14T15:10:06.160794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:07:24.600370+0000","last_clean_scrub_stamp":"2024-04-14T15:10:06.160794+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"72'16","reported_seq":100,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.313360+0000","last_change":"2024-04-14T15:09:45.180279+0000","last_active":"2024-04-14T15:11:22.313360+0000","last_peered":"2024-04-14T15:11:22.313360+0000","last_clean":"2024-04-14T15:11:22.313360+0000","last_became_active":"2024-04-14T15:09:45.179978+0000","last_became_peered":"2024-04-14T15:09:45.179978+0000","last_unstale":"2024-04-14T15:11:22.313360+0000","last_undegraded":"2024-04-14T15:11:22.313360+0000","last_fullsized":"2024-04-14T15:11:22.313360+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":61,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.313719+0000","last_change":"2024-04-14T15:11:13.892949+0000","last_active":"2024-04-14T15:11:22.313719+0000","last_peered":"2024-04-14T15:11:22.313719+0000","last_clean":"2024-04-14T15:11:22.313719+0000","last_became_active":"2024-04-14T15:09:47.047201+0000","last_became_peered":"2024-04-14T15:09:47.047201+0000","last_unstale":"2024-04-14T15:11:22.313719+0000","last_undegraded":"2024-04-14T15:11:22.313719+0000","last_fullsized":"2024-04-14T15:11:22.313719+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:13.892863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:13.892863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:41.896681+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"72'1","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.266682+0000","last_change":"2024-04-14T15:09:49.065706+0000","last_active":"2024-04-14T15:11:22.266682+0000","last_peered":"2024-04-14T15:11:22.266682+0000","last_clean":"2024-04-14T15:11:22.266682+0000","last_became_active":"2024-04-14T15:09:49.065243+0000","last_became_peered":"2024-04-14T15:09:49.065243+0000","last_unstale":"2024-04-14T15:11:22.266682+0000","last_undegraded":"2024-04-14T15:11:22.266682+0000","last_fullsized":"2024-04-14T15:11:22.266682+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":76,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.312472+0000","last_change":"2024-04-14T15:11:07.696609+0000","last_active":"2024-04-14T15:11:22.312472+0000","last_peered":"2024-04-14T15:11:22.312472+0000","last_clean":"2024-04-14T15:11:22.312472+0000","last_became_active":"2024-04-14T15:09:43.018509+0000","last_became_peered":"2024-04-14T15:09:43.018509+0000","last_unstale":"2024-04-14T15:11:22.312472+0000","last_undegraded":"2024-04-14T15:11:22.312472+0000","last_fullsized":"2024-04-14T15:11:22.312472+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:07.696525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:07.696525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:17.969989+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0018592680000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":69,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.312122+0000","last_change":"2024-04-14T15:10:49.916989+0000","last_active":"2024-04-14T15:11:22.312122+0000","last_peered":"2024-04-14T15:11:22.312122+0000","last_clean":"2024-04-14T15:11:22.312122+0000","last_became_active":"2024-04-14T15:09:43.025095+0000","last_became_peered":"2024-04-14T15:09:43.025095+0000","last_unstale":"2024-04-14T15:11:22.312122+0000","last_undegraded":"2024-04-14T15:11:22.312122+0000","last_fullsized":"2024-04-14T15:11:22.312122+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:49.916884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:10:49.916884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:11.267411+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"72'12","reported_seq":83,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.312173+0000","last_change":"2024-04-14T15:10:51.840577+0000","last_active":"2024-04-14T15:11:22.312173+0000","last_peered":"2024-04-14T15:11:22.312173+0000","last_clean":"2024-04-14T15:11:22.312173+0000","last_became_active":"2024-04-14T15:09:45.037231+0000","last_became_peered":"2024-04-14T15:09:45.037231+0000","last_unstale":"2024-04-14T15:11:22.312173+0000","last_undegraded":"2024-04-14T15:11:22.312173+0000","last_fullsized":"2024-04-14T15:11:22.312173+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'12","last_scrub_stamp":"2024-04-14T15:10:51.840487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:10:51.840487+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:52.131140+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.314119+0000","last_change":"2024-04-14T15:09:47.037005+0000","last_active":"2024-04-14T15:11:22.314119+0000","last_peered":"2024-04-14T15:11:22.314119+0000","last_clean":"2024-04-14T15:11:22.314119+0000","last_became_active":"2024-04-14T15:09:47.036696+0000","last_became_peered":"2024-04-14T15:09:47.036696+0000","last_unstale":"2024-04-14T15:11:22.314119+0000","last_undegraded":"2024-04-14T15:11:22.314119+0000","last_fullsized":"2024-04-14T15:11:22.314119+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:09:46.023663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.266335+0000","last_change":"2024-04-14T15:09:49.128490+0000","last_active":"2024-04-14T15:11:22.266335+0000","last_peered":"2024-04-14T15:11:22.266335+0000","last_clean":"2024-04-14T15:11:22.266335+0000","last_became_active":"2024-04-14T15:09:49.128136+0000","last_became_peered":"2024-04-14T15:09:49.128136+0000","last_unstale":"2024-04-14T15:11:22.266335+0000","last_undegraded":"2024-04-14T15:11:22.266335+0000","last_fullsized":"2024-04-14T15:11:22.266335+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":69,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.314216+0000","last_change":"2024-04-14T15:10:53.992688+0000","last_active":"2024-04-14T15:11:22.314216+0000","last_peered":"2024-04-14T15:11:22.314216+0000","last_clean":"2024-04-14T15:11:22.314216+0000","last_became_active":"2024-04-14T15:09:43.021006+0000","last_became_peered":"2024-04-14T15:09:43.021006+0000","last_unstale":"2024-04-14T15:11:22.314216+0000","last_undegraded":"2024-04-14T15:11:22.314216+0000","last_fullsized":"2024-04-14T15:11:22.314216+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:53.992590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:10:53.992590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:16.131673+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"72'19","reported_seq":101,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.312982+0000","last_change":"2024-04-14T15:10:47.880742+0000","last_active":"2024-04-14T15:11:22.312982+0000","last_peered":"2024-04-14T15:11:22.312982+0000","last_clean":"2024-04-14T15:11:22.312982+0000","last_became_active":"2024-04-14T15:09:45.179182+0000","last_became_peered":"2024-04-14T15:09:45.179182+0000","last_unstale":"2024-04-14T15:11:22.312982+0000","last_undegraded":"2024-04-14T15:11:22.312982+0000","last_fullsized":"2024-04-14T15:11:22.312982+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'19","last_scrub_stamp":"2024-04-14T15:10:47.880623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:10:47.880623+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:07.612241+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":61,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.313726+0000","last_change":"2024-04-14T15:10:59.286474+0000","last_active":"2024-04-14T15:11:22.313726+0000","last_peered":"2024-04-14T15:11:22.313726+0000","last_clean":"2024-04-14T15:11:22.313726+0000","last_became_active":"2024-04-14T15:09:47.048427+0000","last_became_peered":"2024-04-14T15:09:47.048427+0000","last_unstale":"2024-04-14T15:11:22.313726+0000","last_undegraded":"2024-04-14T15:11:22.313726+0000","last_fullsized":"2024-04-14T15:11:22.313726+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:59.286374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:10:59.286374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:59.951701+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.265791+0000","last_change":"2024-04-14T15:09:49.065988+0000","last_active":"2024-04-14T15:11:22.265791+0000","last_peered":"2024-04-14T15:11:22.265791+0000","last_clean":"2024-04-14T15:11:22.265791+0000","last_became_active":"2024-04-14T15:09:49.065684+0000","last_became_peered":"2024-04-14T15:09:49.065684+0000","last_unstale":"2024-04-14T15:11:22.265791+0000","last_undegraded":"2024-04-14T15:11:22.265791+0000","last_fullsized":"2024-04-14T15:11:22.265791+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"65'1","reported_seq":62,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.267545+0000","last_change":"2024-04-14T15:09:43.019515+0000","last_active":"2024-04-14T15:11:22.267545+0000","last_peered":"2024-04-14T15:11:22.267545+0000","last_clean":"2024-04-14T15:11:22.267545+0000","last_became_active":"2024-04-14T15:09:43.018478+0000","last_became_peered":"2024-04-14T15:09:43.018478+0000","last_unstale":"2024-04-14T15:11:22.267545+0000","last_undegraded":"2024-04-14T15:11:22.267545+0000","last_fullsized":"2024-04-14T15:11:22.267545+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:09:42.004317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"72'18","reported_seq":97,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.311901+0000","last_change":"2024-04-14T15:10:53.457240+0000","last_active":"2024-04-14T15:11:22.311901+0000","last_peered":"2024-04-14T15:11:22.311901+0000","last_clean":"2024-04-14T15:11:22.311901+0000","last_became_active":"2024-04-14T15:09:45.181252+0000","last_became_peered":"2024-04-14T15:09:45.181252+0000","last_unstale":"2024-04-14T15:11:22.311901+0000","last_undegraded":"2024-04-14T15:11:22.311901+0000","last_fullsized":"2024-04-14T15:11:22.311901+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'18","last_scrub_stamp":"2024-04-14T15:10:53.457087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:10:53.457087+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:00.641833+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.265436+0000","last_change":"2024-04-14T15:09:47.044907+0000","last_active":"2024-04-14T15:11:22.265436+0000","last_peered":"2024-04-14T15:11:22.265436+0000","last_clean":"2024-04-14T15:11:22.265436+0000","last_became_active":"2024-04-14T15:09:47.044610+0000","last_became_peered":"2024-04-14T15:09:47.044610+0000","last_unstale":"2024-04-14T15:11:22.265436+0000","last_undegraded":"2024-04-14T15:11:22.265436+0000","last_fullsized":"2024-04-14T15:11:22.265436+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:09:46.023663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.312945+0000","last_change":"2024-04-14T15:11:16.821265+0000","last_active":"2024-04-14T15:11:22.312945+0000","last_peered":"2024-04-14T15:11:22.312945+0000","last_clean":"2024-04-14T15:11:22.312945+0000","last_became_active":"2024-04-14T15:09:49.129610+0000","last_became_peered":"2024-04-14T15:09:49.129610+0000","last_unstale":"2024-04-14T15:11:22.312945+0000","last_undegraded":"2024-04-14T15:11:22.312945+0000","last_fullsized":"2024-04-14T15:11:22.312945+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:16.821140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:16.821140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:32.897119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":69,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.313706+0000","last_change":"2024-04-14T15:11:20.219646+0000","last_active":"2024-04-14T15:11:22.313706+0000","last_peered":"2024-04-14T15:11:22.313706+0000","last_clean":"2024-04-14T15:11:22.313706+0000","last_became_active":"2024-04-14T15:09:43.020503+0000","last_became_peered":"2024-04-14T15:09:43.020503+0000","last_unstale":"2024-04-14T15:11:22.313706+0000","last_undegraded":"2024-04-14T15:11:22.313706+0000","last_fullsized":"2024-04-14T15:11:22.313706+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:20.219543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:20.219543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:39.250425+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"72'14","reported_seq":86,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.265287+0000","last_change":"2024-04-14T15:11:14.657178+0000","last_active":"2024-04-14T15:11:22.265287+0000","last_peered":"2024-04-14T15:11:22.265287+0000","last_clean":"2024-04-14T15:11:22.265287+0000","last_became_active":"2024-04-14T15:09:45.025673+0000","last_became_peered":"2024-04-14T15:09:45.025673+0000","last_unstale":"2024-04-14T15:11:22.265287+0000","last_undegraded":"2024-04-14T15:11:22.265287+0000","last_fullsized":"2024-04-14T15:11:22.265287+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'14","last_scrub_stamp":"2024-04-14T15:11:14.657075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:14.657075+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:37.152537+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"72'8","reported_seq":82,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.311891+0000","last_change":"2024-04-14T15:09:47.046875+0000","last_active":"2024-04-14T15:11:22.311891+0000","last_peered":"2024-04-14T15:11:22.311891+0000","last_clean":"2024-04-14T15:11:22.311891+0000","last_became_active":"2024-04-14T15:09:47.046560+0000","last_became_peered":"2024-04-14T15:09:47.046560+0000","last_unstale":"2024-04-14T15:11:22.311891+0000","last_undegraded":"2024-04-14T15:11:22.311891+0000","last_fullsized":"2024-04-14T15:11:22.311891+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:09:46.023663+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.312881+0000","last_change":"2024-04-14T15:11:01.501878+0000","last_active":"2024-04-14T15:11:22.312881+0000","last_peered":"2024-04-14T15:11:22.312881+0000","last_clean":"2024-04-14T15:11:22.312881+0000","last_became_active":"2024-04-14T15:09:49.128028+0000","last_became_peered":"2024-04-14T15:09:49.128028+0000","last_unstale":"2024-04-14T15:11:22.312881+0000","last_undegraded":"2024-04-14T15:11:22.312881+0000","last_fullsized":"2024-04-14T15:11:22.312881+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:01.501784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:01.501784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:23.865314+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":61,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.313979+0000","last_change":"2024-04-14T15:11:02.763497+0000","last_active":"2024-04-14T15:11:22.313979+0000","last_peered":"2024-04-14T15:11:22.313979+0000","last_clean":"2024-04-14T15:11:22.313979+0000","last_became_active":"2024-04-14T15:09:47.043021+0000","last_became_peered":"2024-04-14T15:09:47.043021+0000","last_unstale":"2024-04-14T15:11:22.313979+0000","last_undegraded":"2024-04-14T15:11:22.313979+0000","last_fullsized":"2024-04-14T15:11:22.313979+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:02.763399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:02.763399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:06.127325+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":69,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.312326+0000","last_change":"2024-04-14T15:10:48.939920+0000","last_active":"2024-04-14T15:11:22.312326+0000","last_peered":"2024-04-14T15:11:22.312326+0000","last_clean":"2024-04-14T15:11:22.312326+0000","last_became_active":"2024-04-14T15:09:43.018601+0000","last_became_peered":"2024-04-14T15:09:43.018601+0000","last_unstale":"2024-04-14T15:11:22.312326+0000","last_undegraded":"2024-04-14T15:11:22.312326+0000","last_fullsized":"2024-04-14T15:11:22.312326+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:48.939790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:10:48.939790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:14.888606+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"85'16","reported_seq":89,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.266028+0000","last_change":"2024-04-14T15:10:47.456301+0000","last_active":"2024-04-14T15:11:22.266028+0000","last_peered":"2024-04-14T15:11:22.266028+0000","last_clean":"2024-04-14T15:11:22.266028+0000","last_became_active":"2024-04-14T15:09:45.028212+0000","last_became_peered":"2024-04-14T15:09:45.028212+0000","last_unstale":"2024-04-14T15:11:22.266028+0000","last_undegraded":"2024-04-14T15:11:22.266028+0000","last_fullsized":"2024-04-14T15:11:22.266028+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-14T15:10:47.456169+0000","last_deep_scrub":"85'16","last_deep_scrub_stamp":"2024-04-14T15:10:47.456169+0000","last_clean_scrub_stamp":"2024-04-14T15:10:47.456169+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:53.736949+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.314638+0000","last_change":"2024-04-14T15:09:49.059351+0000","last_active":"2024-04-14T15:11:22.314638+0000","last_peered":"2024-04-14T15:11:22.314638+0000","last_clean":"2024-04-14T15:11:22.314638+0000","last_became_active":"2024-04-14T15:09:49.058940+0000","last_became_peered":"2024-04-14T15:09:49.058940+0000","last_unstale":"2024-04-14T15:11:22.314638+0000","last_undegraded":"2024-04-14T15:11:22.314638+0000","last_fullsized":"2024-04-14T15:11:22.314638+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"72'8","reported_seq":82,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.265809+0000","last_change":"2024-04-14T15:09:47.051276+0000","last_active":"2024-04-14T15:11:22.265809+0000","last_peered":"2024-04-14T15:11:22.265809+0000","last_clean":"2024-04-14T15:11:22.265809+0000","last_became_active":"2024-04-14T15:09:47.050962+0000","last_became_peered":"2024-04-14T15:09:47.050962+0000","last_unstale":"2024-04-14T15:11:22.265809+0000","last_undegraded":"2024-04-14T15:11:22.265809+0000","last_fullsized":"2024-04-14T15:11:22.265809+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:09:46.023663+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":69,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.266505+0000","last_change":"2024-04-14T15:11:11.285681+0000","last_active":"2024-04-14T15:11:22.266505+0000","last_peered":"2024-04-14T15:11:22.266505+0000","last_clean":"2024-04-14T15:11:22.266505+0000","last_became_active":"2024-04-14T15:09:43.020878+0000","last_became_peered":"2024-04-14T15:09:43.020878+0000","last_unstale":"2024-04-14T15:11:22.266505+0000","last_undegraded":"2024-04-14T15:11:22.266505+0000","last_fullsized":"2024-04-14T15:11:22.266505+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:11.285588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:11.285588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:11.582228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"72'15","reported_seq":90,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.267767+0000","last_change":"2024-04-14T15:11:22.219265+0000","last_active":"2024-04-14T15:11:22.267767+0000","last_peered":"2024-04-14T15:11:22.267767+0000","last_clean":"2024-04-14T15:11:22.267767+0000","last_became_active":"2024-04-14T15:09:45.033331+0000","last_became_peered":"2024-04-14T15:09:45.033331+0000","last_unstale":"2024-04-14T15:11:22.267767+0000","last_undegraded":"2024-04-14T15:11:22.267767+0000","last_fullsized":"2024-04-14T15:11:22.267767+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-04-14T15:11:22.219174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:22.219174+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:23.840704+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.265774+0000","last_change":"2024-04-14T15:09:49.065592+0000","last_active":"2024-04-14T15:11:22.265774+0000","last_peered":"2024-04-14T15:11:22.265774+0000","last_clean":"2024-04-14T15:11:22.265774+0000","last_became_active":"2024-04-14T15:09:49.065274+0000","last_became_peered":"2024-04-14T15:09:49.065274+0000","last_unstale":"2024-04-14T15:11:22.265774+0000","last_undegraded":"2024-04-14T15:11:22.265774+0000","last_fullsized":"2024-04-14T15:11:22.265774+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"72'8","reported_seq":90,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.267449+0000","last_change":"2024-04-14T15:11:21.827356+0000","last_active":"2024-04-14T15:11:22.267449+0000","last_peered":"2024-04-14T15:11:22.267449+0000","last_clean":"2024-04-14T15:11:22.267449+0000","last_became_active":"2024-04-14T15:09:47.044085+0000","last_became_peered":"2024-04-14T15:09:47.044085+0000","last_unstale":"2024-04-14T15:11:22.267449+0000","last_undegraded":"2024-04-14T15:11:22.267449+0000","last_fullsized":"2024-04-14T15:11:22.267449+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:11:21.827272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:21.827272+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:40.383654+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":69,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.266684+0000","last_change":"2024-04-14T15:10:46.280858+0000","last_active":"2024-04-14T15:11:22.266684+0000","last_peered":"2024-04-14T15:11:22.266684+0000","last_clean":"2024-04-14T15:11:22.266684+0000","last_became_active":"2024-04-14T15:09:43.018871+0000","last_became_peered":"2024-04-14T15:09:43.018871+0000","last_unstale":"2024-04-14T15:11:22.266684+0000","last_undegraded":"2024-04-14T15:11:22.266684+0000","last_fullsized":"2024-04-14T15:11:22.266684+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:46.280773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:10:46.280773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:59.607706+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"72'10","reported_seq":72,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.265665+0000","last_change":"2024-04-14T15:09:45.029530+0000","last_active":"2024-04-14T15:11:22.265665+0000","last_peered":"2024-04-14T15:11:22.265665+0000","last_clean":"2024-04-14T15:11:22.265665+0000","last_became_active":"2024-04-14T15:09:45.029170+0000","last_became_peered":"2024-04-14T15:09:45.029170+0000","last_unstale":"2024-04-14T15:11:22.265665+0000","last_undegraded":"2024-04-14T15:11:22.265665+0000","last_fullsized":"2024-04-14T15:11:22.265665+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.268272+0000","last_change":"2024-04-14T15:11:10.852797+0000","last_active":"2024-04-14T15:11:22.268272+0000","last_peered":"2024-04-14T15:11:22.268272+0000","last_clean":"2024-04-14T15:11:22.268272+0000","last_became_active":"2024-04-14T15:09:49.069594+0000","last_became_peered":"2024-04-14T15:09:49.069594+0000","last_unstale":"2024-04-14T15:11:22.268272+0000","last_undegraded":"2024-04-14T15:11:22.268272+0000","last_fullsized":"2024-04-14T15:11:22.268272+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:10.852715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:10.852715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:18.595232+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.266758+0000","last_change":"2024-04-14T15:09:47.043481+0000","last_active":"2024-04-14T15:11:22.266758+0000","last_peered":"2024-04-14T15:11:22.266758+0000","last_clean":"2024-04-14T15:11:22.266758+0000","last_became_active":"2024-04-14T15:09:47.042810+0000","last_became_peered":"2024-04-14T15:09:47.042810+0000","last_unstale":"2024-04-14T15:11:22.266758+0000","last_undegraded":"2024-04-14T15:11:22.266758+0000","last_fullsized":"2024-04-14T15:11:22.266758+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:09:46.023663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":69,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.312604+0000","last_change":"2024-04-14T15:11:01.903252+0000","last_active":"2024-04-14T15:11:22.312604+0000","last_peered":"2024-04-14T15:11:22.312604+0000","last_clean":"2024-04-14T15:11:22.312604+0000","last_became_active":"2024-04-14T15:09:43.022351+0000","last_became_peered":"2024-04-14T15:09:43.022351+0000","last_unstale":"2024-04-14T15:11:22.312604+0000","last_undegraded":"2024-04-14T15:11:22.312604+0000","last_fullsized":"2024-04-14T15:11:22.312604+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:01.903150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:01.903150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:27.833698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"72'17","reported_seq":117,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.266073+0000","last_change":"2024-04-14T15:11:09.502519+0000","last_active":"2024-04-14T15:11:22.266073+0000","last_peered":"2024-04-14T15:11:22.266073+0000","last_clean":"2024-04-14T15:11:22.266073+0000","last_became_active":"2024-04-14T15:10:07.766008+0000","last_became_peered":"2024-04-14T15:10:07.766008+0000","last_unstale":"2024-04-14T15:11:22.266073+0000","last_undegraded":"2024-04-14T15:11:22.266073+0000","last_fullsized":"2024-04-14T15:11:22.266073+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'17","last_scrub_stamp":"2024-04-14T15:11:09.502431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:09.502431+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:29.834036+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.312115+0000","last_change":"2024-04-14T15:11:15.845024+0000","last_active":"2024-04-14T15:11:22.312115+0000","last_peered":"2024-04-14T15:11:22.312115+0000","last_clean":"2024-04-14T15:11:22.312115+0000","last_became_active":"2024-04-14T15:09:49.065397+0000","last_became_peered":"2024-04-14T15:09:49.065397+0000","last_unstale":"2024-04-14T15:11:22.312115+0000","last_undegraded":"2024-04-14T15:11:22.312115+0000","last_fullsized":"2024-04-14T15:11:22.312115+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:15.844942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:15.844942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:39.797576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":61,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.267508+0000","last_change":"2024-04-14T15:10:59.793878+0000","last_active":"2024-04-14T15:11:22.267508+0000","last_peered":"2024-04-14T15:11:22.267508+0000","last_clean":"2024-04-14T15:11:22.267508+0000","last_became_active":"2024-04-14T15:09:47.045542+0000","last_became_peered":"2024-04-14T15:09:47.045542+0000","last_unstale":"2024-04-14T15:11:22.267508+0000","last_undegraded":"2024-04-14T15:11:22.267508+0000","last_fullsized":"2024-04-14T15:11:22.267508+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:59.793768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:10:59.793768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:24.997192+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":69,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.268115+0000","last_change":"2024-04-14T15:10:45.837586+0000","last_active":"2024-04-14T15:11:22.268115+0000","last_peered":"2024-04-14T15:11:22.268115+0000","last_clean":"2024-04-14T15:11:22.268115+0000","last_became_active":"2024-04-14T15:09:43.020268+0000","last_became_peered":"2024-04-14T15:09:43.020268+0000","last_unstale":"2024-04-14T15:11:22.268115+0000","last_undegraded":"2024-04-14T15:11:22.268115+0000","last_fullsized":"2024-04-14T15:11:22.268115+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:45.837506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:10:45.837506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:04.829576+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"72'19","reported_seq":96,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.313557+0000","last_change":"2024-04-14T15:11:09.225462+0000","last_active":"2024-04-14T15:11:22.313557+0000","last_peered":"2024-04-14T15:11:22.313557+0000","last_clean":"2024-04-14T15:11:22.313557+0000","last_became_active":"2024-04-14T15:09:45.178782+0000","last_became_peered":"2024-04-14T15:09:45.178782+0000","last_unstale":"2024-04-14T15:11:22.313557+0000","last_undegraded":"2024-04-14T15:11:22.313557+0000","last_fullsized":"2024-04-14T15:11:22.313557+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'19","last_scrub_stamp":"2024-04-14T15:11:09.225359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:09.225359+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:29.861536+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.312249+0000","last_change":"2024-04-14T15:10:53.789872+0000","last_active":"2024-04-14T15:11:22.312249+0000","last_peered":"2024-04-14T15:11:22.312249+0000","last_clean":"2024-04-14T15:11:22.312249+0000","last_became_active":"2024-04-14T15:09:49.061495+0000","last_became_peered":"2024-04-14T15:09:49.061495+0000","last_unstale":"2024-04-14T15:11:22.312249+0000","last_undegraded":"2024-04-14T15:11:22.312249+0000","last_fullsized":"2024-04-14T15:11:22.312249+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:53.789790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:10:53.789790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:54.748155+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":61,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.267823+0000","last_change":"2024-04-14T15:10:50.788471+0000","last_active":"2024-04-14T15:11:22.267823+0000","last_peered":"2024-04-14T15:11:22.267823+0000","last_clean":"2024-04-14T15:11:22.267823+0000","last_became_active":"2024-04-14T15:09:47.046153+0000","last_became_peered":"2024-04-14T15:09:47.046153+0000","last_unstale":"2024-04-14T15:11:22.267823+0000","last_undegraded":"2024-04-14T15:11:22.267823+0000","last_fullsized":"2024-04-14T15:11:22.267823+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:50.788350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:10:50.788350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:11:52.620524+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":69,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.267705+0000","last_change":"2024-04-14T15:11:04.223680+0000","last_active":"2024-04-14T15:11:22.267705+0000","last_peered":"2024-04-14T15:11:22.267705+0000","last_clean":"2024-04-14T15:11:22.267705+0000","last_became_active":"2024-04-14T15:09:43.020044+0000","last_became_peered":"2024-04-14T15:09:43.020044+0000","last_unstale":"2024-04-14T15:11:22.267705+0000","last_undegraded":"2024-04-14T15:11:22.267705+0000","last_fullsized":"2024-04-14T15:11:22.267705+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:04.223605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:04.223605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:11.422287+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"72'9","reported_seq":81,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.313097+0000","last_change":"2024-04-14T15:10:50.850226+0000","last_active":"2024-04-14T15:11:22.313097+0000","last_peered":"2024-04-14T15:11:22.313097+0000","last_clean":"2024-04-14T15:11:22.313097+0000","last_became_active":"2024-04-14T15:09:45.037179+0000","last_became_peered":"2024-04-14T15:09:45.037179+0000","last_unstale":"2024-04-14T15:11:22.313097+0000","last_undegraded":"2024-04-14T15:11:22.313097+0000","last_fullsized":"2024-04-14T15:11:22.313097+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-04-14T15:10:50.850116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:10:50.850116+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:06.196819+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.314229+0000","last_change":"2024-04-14T15:09:49.128389+0000","last_active":"2024-04-14T15:11:22.314229+0000","last_peered":"2024-04-14T15:11:22.314229+0000","last_clean":"2024-04-14T15:11:22.314229+0000","last_became_active":"2024-04-14T15:09:49.128070+0000","last_became_peered":"2024-04-14T15:09:49.128070+0000","last_unstale":"2024-04-14T15:11:22.314229+0000","last_undegraded":"2024-04-14T15:11:22.314229+0000","last_fullsized":"2024-04-14T15:11:22.314229+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":69,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.268112+0000","last_change":"2024-04-14T15:11:13.838572+0000","last_active":"2024-04-14T15:11:22.268112+0000","last_peered":"2024-04-14T15:11:22.268112+0000","last_clean":"2024-04-14T15:11:22.268112+0000","last_became_active":"2024-04-14T15:09:43.021460+0000","last_became_peered":"2024-04-14T15:09:43.021460+0000","last_unstale":"2024-04-14T15:11:22.268112+0000","last_undegraded":"2024-04-14T15:11:22.268112+0000","last_fullsized":"2024-04-14T15:11:22.268112+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:13.838468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:13.838468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:31.461401+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"72'15","reported_seq":90,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.312430+0000","last_change":"2024-04-14T15:11:19.680463+0000","last_active":"2024-04-14T15:11:22.312430+0000","last_peered":"2024-04-14T15:11:22.312430+0000","last_clean":"2024-04-14T15:11:22.312430+0000","last_became_active":"2024-04-14T15:09:45.180160+0000","last_became_peered":"2024-04-14T15:09:45.180160+0000","last_unstale":"2024-04-14T15:11:22.312430+0000","last_undegraded":"2024-04-14T15:11:22.312430+0000","last_fullsized":"2024-04-14T15:11:22.312430+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-04-14T15:11:19.680308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:19.680308+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:41.707865+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"85'9","reported_seq":91,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-14T15:11:22.312484+0000","last_change":"2024-04-14T15:10:51.950673+0000","last_active":"2024-04-14T15:11:22.312484+0000","last_peered":"2024-04-14T15:11:22.312484+0000","last_clean":"2024-04-14T15:11:22.312484+0000","last_became_active":"2024-04-14T15:09:47.045441+0000","last_became_peered":"2024-04-14T15:09:47.045441+0000","last_unstale":"2024-04-14T15:11:22.312484+0000","last_undegraded":"2024-04-14T15:11:22.312484+0000","last_fullsized":"2024-04-14T15:11:22.312484+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9","last_scrub_stamp":"2024-04-14T15:10:51.950580+0000","last_deep_scrub":"85'9","last_deep_scrub_stamp":"2024-04-14T15:10:51.950580+0000","last_clean_scrub_stamp":"2024-04-14T15:10:51.950580+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:09.843217+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":400,"ondisk_log_size":400,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":95,"num_read_kb":90,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103194,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28408,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716744,"statfs":{"total":95995035648,"available":95965945856,"internally_reserved":0,"allocated":1421312,"data_stored":957404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397533,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28404,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716748,"statfs":{"total":95995035648,"available":95965949952,"internally_reserved":0,"allocated":1417216,"data_stored":957697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574903,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27860,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717292,"statfs":{"total":95995035648,"available":95966507008,"internally_reserved":0,"allocated":860160,"data_stored":367450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738428,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32576,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712576,"statfs":{"total":95995035648,"available":95961677824,"internally_reserved":0,"allocated":1429504,"data_stored":958057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411379,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31976,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713176,"statfs":{"total":95995035648,"available":95962292224,"internally_reserved":0,"allocated":811008,"data_stored":366551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182445,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27860,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717292,"statfs":{"total":95995035648,"available":95966507008,"internally_reserved":0,"allocated":856064,"data_stored":367771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986215,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27836,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717316,"statfs":{"total":95995035648,"available":95966531584,"internally_reserved":0,"allocated":835584,"data_stored":366932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659172,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31980,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713172,"statfs":{"total":95995035648,"available":95962288128,"internally_reserved":0,"allocated":819200,"data_stored":366386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-14T15:11:26.997 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-14T15:11:26.998 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd dump 2024-04-14T15:11:27.114 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:11:26 smithi092 podman[67889]: 2024-04-14 15:11:26.642388567 +0000 UTC m=+4.042156916 container init 3096ddbef6327091889e6a95eb5d901afc92a9b765fdd00a4d9350e17718db9d (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-14T15:11:27.114 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:11:26 smithi092 podman[67889]: 2024-04-14 15:11:26.648204147 +0000 UTC m=+4.047972496 container start 3096ddbef6327091889e6a95eb5d901afc92a9b765fdd00a4d9350e17718db9d (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-14T15:11:27.114 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:11:26 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-b[67939]: ts=2024-04-14T15:11:26.651Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-04-14T15:11:27.114 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:11:26 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-b[67939]: ts=2024-04-14T15:11:26.652Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-04-14T15:11:27.114 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:11:26 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-b[67939]: ts=2024-04-14T15:11:26.652Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-04-14T15:11:27.114 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:11:26 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-b[67939]: ts=2024-04-14T15:11:26.652Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-04-14T15:11:27.114 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:11:26 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-b[67939]: ts=2024-04-14T15:11:26.652Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-04-14T15:11:27.114 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:11:26 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-b[67939]: ts=2024-04-14T15:11:26.653Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-04-14T15:11:27.114 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:11:26 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-b[67939]: ts=2024-04-14T15:11:26.653Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-04-14T15:11:27.114 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:11:26 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-b[67939]: ts=2024-04-14T15:11:26.653Z caller=node_exporter.go:117 level=info collector=arp 2024-04-14T15:11:27.115 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:11:26 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-b[67939]: ts=2024-04-14T15:11:26.653Z caller=node_exporter.go:117 level=info collector=bcache 2024-04-14T15:11:27.115 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:11:26 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-b[67939]: ts=2024-04-14T15:11:26.653Z caller=node_exporter.go:117 level=info collector=bonding 2024-04-14T15:11:27.115 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:11:26 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-b[67939]: ts=2024-04-14T15:11:26.653Z caller=node_exporter.go:117 level=info collector=btrfs 2024-04-14T15:11:27.115 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:11:26 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-b[67939]: ts=2024-04-14T15:11:26.653Z caller=node_exporter.go:117 level=info collector=conntrack 2024-04-14T15:11:27.115 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:11:26 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-b[67939]: ts=2024-04-14T15:11:26.653Z caller=node_exporter.go:117 level=info collector=cpu 2024-04-14T15:11:27.115 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:11:26 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-b[67939]: ts=2024-04-14T15:11:26.653Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-04-14T15:11:27.115 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:11:26 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-b[67939]: ts=2024-04-14T15:11:26.653Z caller=node_exporter.go:117 level=info collector=diskstats 2024-04-14T15:11:27.115 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:11:26 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-b[67939]: ts=2024-04-14T15:11:26.653Z caller=node_exporter.go:117 level=info collector=dmi 2024-04-14T15:11:27.115 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:11:26 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-b[67939]: ts=2024-04-14T15:11:26.653Z caller=node_exporter.go:117 level=info collector=edac 2024-04-14T15:11:27.115 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:11:26 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-b[67939]: ts=2024-04-14T15:11:26.653Z caller=node_exporter.go:117 level=info collector=entropy 2024-04-14T15:11:27.115 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:11:26 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-b[67939]: ts=2024-04-14T15:11:26.653Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-04-14T15:11:27.115 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:11:26 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-b[67939]: ts=2024-04-14T15:11:26.653Z caller=node_exporter.go:117 level=info collector=filefd 2024-04-14T15:11:27.116 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:11:26 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-b[67939]: ts=2024-04-14T15:11:26.653Z caller=node_exporter.go:117 level=info collector=filesystem 2024-04-14T15:11:27.116 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:11:26 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-b[67939]: ts=2024-04-14T15:11:26.653Z caller=node_exporter.go:117 level=info collector=hwmon 2024-04-14T15:11:27.116 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:11:26 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-b[67939]: ts=2024-04-14T15:11:26.653Z caller=node_exporter.go:117 level=info collector=infiniband 2024-04-14T15:11:27.116 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:11:26 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-b[67939]: ts=2024-04-14T15:11:26.653Z caller=node_exporter.go:117 level=info collector=ipvs 2024-04-14T15:11:27.116 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:11:26 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-b[67939]: ts=2024-04-14T15:11:26.653Z caller=node_exporter.go:117 level=info collector=loadavg 2024-04-14T15:11:27.116 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:11:26 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-b[67939]: ts=2024-04-14T15:11:26.653Z caller=node_exporter.go:117 level=info collector=mdadm 2024-04-14T15:11:27.116 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:11:26 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-b[67939]: ts=2024-04-14T15:11:26.653Z caller=node_exporter.go:117 level=info collector=meminfo 2024-04-14T15:11:27.116 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:11:26 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-b[67939]: ts=2024-04-14T15:11:26.653Z caller=node_exporter.go:117 level=info collector=netclass 2024-04-14T15:11:27.116 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:11:26 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-b[67939]: ts=2024-04-14T15:11:26.653Z caller=node_exporter.go:117 level=info collector=netdev 2024-04-14T15:11:27.116 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:11:26 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-b[67939]: ts=2024-04-14T15:11:26.653Z caller=node_exporter.go:117 level=info collector=netstat 2024-04-14T15:11:27.116 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:11:26 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-b[67939]: ts=2024-04-14T15:11:26.653Z caller=node_exporter.go:117 level=info collector=nfs 2024-04-14T15:11:27.116 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:11:26 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-b[67939]: ts=2024-04-14T15:11:26.653Z caller=node_exporter.go:117 level=info collector=nfsd 2024-04-14T15:11:27.116 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:11:26 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-b[67939]: ts=2024-04-14T15:11:26.653Z caller=node_exporter.go:117 level=info collector=nvme 2024-04-14T15:11:27.116 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:11:26 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-b[67939]: ts=2024-04-14T15:11:26.653Z caller=node_exporter.go:117 level=info collector=os 2024-04-14T15:11:27.117 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:11:26 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-b[67939]: ts=2024-04-14T15:11:26.653Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-04-14T15:11:27.117 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:11:26 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-b[67939]: ts=2024-04-14T15:11:26.653Z caller=node_exporter.go:117 level=info collector=pressure 2024-04-14T15:11:27.117 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:11:26 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-b[67939]: ts=2024-04-14T15:11:26.653Z caller=node_exporter.go:117 level=info collector=rapl 2024-04-14T15:11:27.117 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:11:26 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-b[67939]: ts=2024-04-14T15:11:26.653Z caller=node_exporter.go:117 level=info collector=schedstat 2024-04-14T15:11:27.117 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:11:26 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-b[67939]: ts=2024-04-14T15:11:26.653Z caller=node_exporter.go:117 level=info collector=selinux 2024-04-14T15:11:27.117 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:11:26 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-b[67939]: ts=2024-04-14T15:11:26.653Z caller=node_exporter.go:117 level=info collector=sockstat 2024-04-14T15:11:27.117 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:11:26 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-b[67939]: ts=2024-04-14T15:11:26.653Z caller=node_exporter.go:117 level=info collector=softnet 2024-04-14T15:11:27.117 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:11:26 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-b[67939]: ts=2024-04-14T15:11:26.653Z caller=node_exporter.go:117 level=info collector=stat 2024-04-14T15:11:27.117 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:11:26 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-b[67939]: ts=2024-04-14T15:11:26.653Z caller=node_exporter.go:117 level=info collector=tapestats 2024-04-14T15:11:27.118 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:11:26 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-b[67939]: ts=2024-04-14T15:11:26.653Z caller=node_exporter.go:117 level=info collector=textfile 2024-04-14T15:11:27.118 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:11:26 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-b[67939]: ts=2024-04-14T15:11:26.653Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-04-14T15:11:27.118 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:11:26 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-b[67939]: ts=2024-04-14T15:11:26.653Z caller=node_exporter.go:117 level=info collector=time 2024-04-14T15:11:27.118 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:11:26 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-b[67939]: ts=2024-04-14T15:11:26.653Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-04-14T15:11:27.118 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:11:26 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-b[67939]: ts=2024-04-14T15:11:26.653Z caller=node_exporter.go:117 level=info collector=uname 2024-04-14T15:11:27.118 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:11:26 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-b[67939]: ts=2024-04-14T15:11:26.653Z caller=node_exporter.go:117 level=info collector=vmstat 2024-04-14T15:11:27.118 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:11:26 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-b[67939]: ts=2024-04-14T15:11:26.653Z caller=node_exporter.go:117 level=info collector=xfs 2024-04-14T15:11:27.118 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:11:26 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-b[67939]: ts=2024-04-14T15:11:26.653Z caller=node_exporter.go:117 level=info collector=zfs 2024-04-14T15:11:27.118 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:11:26 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-b[67939]: ts=2024-04-14T15:11:26.654Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-04-14T15:11:27.118 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:11:26 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-b[67939]: ts=2024-04-14T15:11:26.654Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-04-14T15:11:27.118 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:11:26 smithi092 bash[67889]: 3096ddbef6327091889e6a95eb5d901afc92a9b765fdd00a4d9350e17718db9d 2024-04-14T15:11:27.118 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:11:26 smithi092 systemd[1]: Started Ceph node-exporter.b for 0e288098-fa70-11ee-bc8e-c7b262605968. 2024-04-14T15:11:27.268 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:11:27.352 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:27 smithi090 ceph-mon[41670]: 4.18 scrub starts 2024-04-14T15:11:27.352 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:27 smithi090 ceph-mon[41670]: 4.18 scrub ok 2024-04-14T15:11:27.352 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:27 smithi090 ceph-mon[41670]: from='client.24673 v1:172.21.15.90:0/3399860262' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:11:27.352 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:27 smithi090 ceph-mon[41670]: 5.7 scrub starts 2024-04-14T15:11:27.352 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:27 smithi090 ceph-mon[41670]: 5.7 scrub ok 2024-04-14T15:11:27.352 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:27 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:27.352 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:27 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:27.353 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:27 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:27.353 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:27 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:27.353 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:27 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:27.353 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:27 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:11:27.353 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:27 smithi090 ceph-mon[30399]: 4.18 scrub starts 2024-04-14T15:11:27.353 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:27 smithi090 ceph-mon[30399]: 4.18 scrub ok 2024-04-14T15:11:27.353 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:27 smithi090 ceph-mon[30399]: from='client.24673 v1:172.21.15.90:0/3399860262' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:11:27.353 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:27 smithi090 ceph-mon[30399]: 5.7 scrub starts 2024-04-14T15:11:27.353 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:27 smithi090 ceph-mon[30399]: 5.7 scrub ok 2024-04-14T15:11:27.353 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:27 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:27.353 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:27 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:27.353 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:27 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:27.354 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:27 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:27.354 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:27 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:27.354 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:27 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:11:27.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:27 smithi092 ceph-mon[33966]: 4.18 scrub starts 2024-04-14T15:11:27.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:27 smithi092 ceph-mon[33966]: 4.18 scrub ok 2024-04-14T15:11:27.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:27 smithi092 ceph-mon[33966]: from='client.24673 v1:172.21.15.90:0/3399860262' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:11:27.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:27 smithi092 ceph-mon[33966]: 5.7 scrub starts 2024-04-14T15:11:27.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:27 smithi092 ceph-mon[33966]: 5.7 scrub ok 2024-04-14T15:11:27.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:27 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:27.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:27 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:27.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:27 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:27.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:27 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:27.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:27 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:27.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:27 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:11:28.155 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:28 smithi092 ceph-mon[33966]: 4.17 scrub starts 2024-04-14T15:11:28.155 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:28 smithi092 ceph-mon[33966]: 4.17 scrub ok 2024-04-14T15:11:28.155 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:28 smithi092 ceph-mon[33966]: 5.1e scrub starts 2024-04-14T15:11:28.155 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:28 smithi092 ceph-mon[33966]: 5.1e scrub ok 2024-04-14T15:11:28.155 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:28 smithi092 ceph-mon[33966]: 6.15 scrub starts 2024-04-14T15:11:28.155 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:28 smithi092 ceph-mon[33966]: 6.15 scrub ok 2024-04-14T15:11:28.155 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:28 smithi092 ceph-mon[33966]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 827 B/s rd, 0 op/s 2024-04-14T15:11:28.155 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:28 smithi092 ceph-mon[33966]: 5.19 scrub starts 2024-04-14T15:11:28.155 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:28 smithi092 ceph-mon[33966]: 5.19 scrub ok 2024-04-14T15:11:28.156 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:28 smithi092 ceph-mon[33966]: 4.1f scrub starts 2024-04-14T15:11:28.156 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:28 smithi092 ceph-mon[33966]: 4.1f scrub ok 2024-04-14T15:11:28.156 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:28 smithi092 ceph-mon[33966]: 5.1b scrub starts 2024-04-14T15:11:28.156 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:28 smithi092 ceph-mon[33966]: 5.1b scrub ok 2024-04-14T15:11:28.156 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:28 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:28.390 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:28 smithi090 ceph-mon[41670]: 4.17 scrub starts 2024-04-14T15:11:28.390 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:28 smithi090 ceph-mon[41670]: 4.17 scrub ok 2024-04-14T15:11:28.390 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:28 smithi090 ceph-mon[41670]: 5.1e scrub starts 2024-04-14T15:11:28.390 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:28 smithi090 ceph-mon[41670]: 5.1e scrub ok 2024-04-14T15:11:28.390 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:28 smithi090 ceph-mon[41670]: 6.15 scrub starts 2024-04-14T15:11:28.390 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:28 smithi090 ceph-mon[41670]: 6.15 scrub ok 2024-04-14T15:11:28.390 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:28 smithi090 ceph-mon[41670]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 827 B/s rd, 0 op/s 2024-04-14T15:11:28.391 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:28 smithi090 ceph-mon[41670]: 5.19 scrub starts 2024-04-14T15:11:28.391 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:28 smithi090 ceph-mon[41670]: 5.19 scrub ok 2024-04-14T15:11:28.391 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:28 smithi090 ceph-mon[41670]: 4.1f scrub starts 2024-04-14T15:11:28.391 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:28 smithi090 ceph-mon[41670]: 4.1f scrub ok 2024-04-14T15:11:28.391 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:28 smithi090 ceph-mon[41670]: 5.1b scrub starts 2024-04-14T15:11:28.391 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:28 smithi090 ceph-mon[41670]: 5.1b scrub ok 2024-04-14T15:11:28.391 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:28 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:28.391 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:28 smithi090 ceph-mon[30399]: 4.17 scrub starts 2024-04-14T15:11:28.391 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:28 smithi090 ceph-mon[30399]: 4.17 scrub ok 2024-04-14T15:11:28.391 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:28 smithi090 ceph-mon[30399]: 5.1e scrub starts 2024-04-14T15:11:28.391 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:28 smithi090 ceph-mon[30399]: 5.1e scrub ok 2024-04-14T15:11:28.392 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:28 smithi090 ceph-mon[30399]: 6.15 scrub starts 2024-04-14T15:11:28.392 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:28 smithi090 ceph-mon[30399]: 6.15 scrub ok 2024-04-14T15:11:28.392 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:28 smithi090 ceph-mon[30399]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 827 B/s rd, 0 op/s 2024-04-14T15:11:28.392 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:28 smithi090 ceph-mon[30399]: 5.19 scrub starts 2024-04-14T15:11:28.392 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:28 smithi090 ceph-mon[30399]: 5.19 scrub ok 2024-04-14T15:11:28.392 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:28 smithi090 ceph-mon[30399]: 4.1f scrub starts 2024-04-14T15:11:28.392 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:28 smithi090 ceph-mon[30399]: 4.1f scrub ok 2024-04-14T15:11:28.392 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:28 smithi090 ceph-mon[30399]: 5.1b scrub starts 2024-04-14T15:11:28.392 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:28 smithi090 ceph-mon[30399]: 5.1b scrub ok 2024-04-14T15:11:28.392 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:28 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:29.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.222+0000 7f088352d700 1 -- 172.21.15.90:0/88451777 >> v1:172.21.15.90:6789/0 conn(0x7f087c10e310 legacy=0x7f087c1107d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:29.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.222+0000 7f088352d700 1 -- 172.21.15.90:0/88451777 shutdown_connections 2024-04-14T15:11:29.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.222+0000 7f088352d700 1 -- 172.21.15.90:0/88451777 >> 172.21.15.90:0/88451777 conn(0x7f087c0fd290 msgr2=0x7f087c0ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:11:29.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.222+0000 7f088352d700 1 -- 172.21.15.90:0/88451777 shutdown_connections 2024-04-14T15:11:29.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.223+0000 7f088352d700 1 -- 172.21.15.90:0/88451777 wait complete. 2024-04-14T15:11:29.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.223+0000 7f088352d700 1 librados: starting msgr at 2024-04-14T15:11:29.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.223+0000 7f088352d700 1 librados: starting objecter 2024-04-14T15:11:29.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.223+0000 7f088352d700 1 Processor -- start 2024-04-14T15:11:29.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.223+0000 7f088352d700 1 -- start start 2024-04-14T15:11:29.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.223+0000 7f088352d700 1 librados: setting wanted keys 2024-04-14T15:11:29.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.223+0000 7f088352d700 1 librados: calling monclient init 2024-04-14T15:11:29.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.224+0000 7f088352d700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f087c10e0b0 con 0x7f087c10a740 2024-04-14T15:11:29.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.224+0000 7f088352d700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f087c1a8f30 con 0x7f087c107890 2024-04-14T15:11:29.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.224+0000 7f088352d700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f087c1aa110 con 0x7f087c10e310 2024-04-14T15:11:29.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.224+0000 7f0880ac8700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f087c10a740 0x7f087c06b430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:59406/0 (socket says 172.21.15.90:59406) 2024-04-14T15:11:29.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.224+0000 7f0880ac8700 1 -- 172.21.15.90:0/1413818048 learned_addr learned my addr 172.21.15.90:0/1413818048 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:11:29.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.224+0000 7f08727fc700 10 client.?.objecter ms_handle_connect 0x7f087c10a740 2024-04-14T15:11:29.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.224+0000 7f08727fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:11:29.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.224+0000 7f08727fc700 10 client.?.objecter ms_handle_connect 0x7f087c107890 2024-04-14T15:11:29.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.224+0000 7f08727fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:11:29.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.224+0000 7f08727fc700 10 client.?.objecter ms_handle_connect 0x7f087c10e310 2024-04-14T15:11:29.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.224+0000 7f08727fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:11:29.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.224+0000 7f08727fc700 1 -- 172.21.15.90:0/1413818048 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 925384730 0 0) 0x7f087c1a8f30 con 0x7f087c107890 2024-04-14T15:11:29.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.224+0000 7f08727fc700 1 -- 172.21.15.90:0/1413818048 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f08680035b0 con 0x7f087c107890 2024-04-14T15:11:29.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.224+0000 7f08727fc700 1 -- 172.21.15.90:0/1413818048 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1702244147 0 0) 0x7f087c10e0b0 con 0x7f087c10a740 2024-04-14T15:11:29.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.224+0000 7f08727fc700 1 -- 172.21.15.90:0/1413818048 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f087c1a8f30 con 0x7f087c10a740 2024-04-14T15:11:29.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.224+0000 7f08727fc700 1 -- 172.21.15.90:0/1413818048 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1639031739 0 0) 0x7f087c1aa110 con 0x7f087c10e310 2024-04-14T15:11:29.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.224+0000 7f08727fc700 1 -- 172.21.15.90:0/1413818048 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f087c10e0b0 con 0x7f087c10e310 2024-04-14T15:11:29.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.224+0000 7f08727fc700 1 -- 172.21.15.90:0/1413818048 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3901869477 0 0) 0x7f08680035b0 con 0x7f087c107890 2024-04-14T15:11:29.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.224+0000 7f08727fc700 1 -- 172.21.15.90:0/1413818048 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f087c1aa110 con 0x7f087c107890 2024-04-14T15:11:29.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.224+0000 7f08727fc700 1 -- 172.21.15.90:0/1413818048 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f08780018b0 con 0x7f087c107890 2024-04-14T15:11:29.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.224+0000 7f08727fc700 1 -- 172.21.15.90:0/1413818048 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3220242374 0 0) 0x7f087c1a8f30 con 0x7f087c10a740 2024-04-14T15:11:29.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.224+0000 7f08727fc700 1 -- 172.21.15.90:0/1413818048 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f08680035b0 con 0x7f087c10a740 2024-04-14T15:11:29.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.224+0000 7f08727fc700 1 -- 172.21.15.90:0/1413818048 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f086c001870 con 0x7f087c10a740 2024-04-14T15:11:29.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.224+0000 7f08727fc700 1 -- 172.21.15.90:0/1413818048 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3162707652 0 0) 0x7f087c10e0b0 con 0x7f087c10e310 2024-04-14T15:11:29.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.224+0000 7f08727fc700 1 -- 172.21.15.90:0/1413818048 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f087c1a8f30 con 0x7f087c10e310 2024-04-14T15:11:29.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.224+0000 7f08727fc700 1 -- 172.21.15.90:0/1413818048 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f0874001bb0 con 0x7f087c10e310 2024-04-14T15:11:29.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.224+0000 7f08727fc700 1 -- 172.21.15.90:0/1413818048 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2118364292 0 0) 0x7f087c1aa110 con 0x7f087c107890 2024-04-14T15:11:29.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.224+0000 7f08727fc700 1 -- 172.21.15.90:0/1413818048 >> v1:172.21.15.90:6790/0 conn(0x7f087c10e310 legacy=0x7f087c1a6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:29.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.224+0000 7f08727fc700 1 -- 172.21.15.90:0/1413818048 >> v1:172.21.15.90:6789/0 conn(0x7f087c10a740 legacy=0x7f087c06b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:29.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.224+0000 7f08727fc700 1 -- 172.21.15.90:0/1413818048 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f087c1ab350 con 0x7f087c107890 2024-04-14T15:11:29.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.225+0000 7f088352d700 1 -- 172.21.15.90:0/1413818048 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f087c1a7f80 con 0x7f087c107890 2024-04-14T15:11:29.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.225+0000 7f088352d700 10 client.24679.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:11:29.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.225+0000 7f088352d700 1 -- 172.21.15.90:0/1413818048 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f087c1a8440 con 0x7f087c107890 2024-04-14T15:11:29.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.225+0000 7f088352d700 1 librados: init done 2024-04-14T15:11:29.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.225+0000 7f08727fc700 1 -- 172.21.15.90:0/1413818048 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f0878003b40 con 0x7f087c107890 2024-04-14T15:11:29.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.225+0000 7f08727fc700 1 -- 172.21.15.90:0/1413818048 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f0878004b00 con 0x7f087c107890 2024-04-14T15:11:29.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.226+0000 7f088352d700 1 -- 172.21.15.90:0/1413818048 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0860005150 con 0x7f087c107890 2024-04-14T15:11:29.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.226+0000 7f08727fc700 1 -- 172.21.15.90:0/1413818048 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f08780036a0 con 0x7f087c107890 2024-04-14T15:11:29.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.227+0000 7f08727fc700 1 -- 172.21.15.90:0/1413818048 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(91..91 src has 12..91) ==== 6288+0+0 (unknown 3555498640 0 0) 0x7f0878089010 con 0x7f087c107890 2024-04-14T15:11:29.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.227+0000 7f08727fc700 10 client.24679.objecter ms_dispatch 0x7f087c001550 osd_map(91..91 src has 12..91) 2024-04-14T15:11:29.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.227+0000 7f08727fc700 3 client.24679.objecter handle_osd_map got epochs [91,91] > 0 2024-04-14T15:11:29.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.227+0000 7f08727fc700 3 client.24679.objecter handle_osd_map decoding full epoch 91 2024-04-14T15:11:29.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.227+0000 7f08727fc700 20 client.24679.objecter dump_active .. 0 homeless 2024-04-14T15:11:29.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.227+0000 7f08727fc700 10 client.24679.objecter ms_handle_connect 0x7f0868071450 2024-04-14T15:11:29.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.232+0000 7f08727fc700 1 -- 172.21.15.90:0/1413818048 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f0878056180 con 0x7f087c107890 2024-04-14T15:11:29.277 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:29 smithi092 ceph-mon[33966]: 4.19 scrub starts 2024-04-14T15:11:29.278 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:29 smithi092 ceph-mon[33966]: 4.19 scrub ok 2024-04-14T15:11:29.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:29 smithi090 ceph-mon[30399]: 4.19 scrub starts 2024-04-14T15:11:29.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:29 smithi090 ceph-mon[30399]: 4.19 scrub ok 2024-04-14T15:11:29.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:29 smithi090 ceph-mon[41670]: 4.19 scrub starts 2024-04-14T15:11:29.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:29 smithi090 ceph-mon[41670]: 4.19 scrub ok 2024-04-14T15:11:29.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.440+0000 7f088352d700 1 -- 172.21.15.90:0/1413818048 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f0860005c50 con 0x7f087c107890 2024-04-14T15:11:29.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.441+0000 7f08727fc700 1 -- 172.21.15.90:0/1413818048 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v91) ==== 56+0+4441 (unknown 3557967957 0 540437653) 0x7f0878059e30 con 0x7f087c107890 2024-04-14T15:11:29.443 INFO:teuthology.orchestra.run.smithi090.stdout:epoch 91 2024-04-14T15:11:29.443 INFO:teuthology.orchestra.run.smithi090.stdout:fsid 0e288098-fa70-11ee-bc8e-c7b262605968 2024-04-14T15:11:29.443 INFO:teuthology.orchestra.run.smithi090.stdout:created 2024-04-14T15:04:05.808313+0000 2024-04-14T15:11:29.443 INFO:teuthology.orchestra.run.smithi090.stdout:modified 2024-04-14T15:11:25.765683+0000 2024-04-14T15:11:29.443 INFO:teuthology.orchestra.run.smithi090.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-14T15:11:29.443 INFO:teuthology.orchestra.run.smithi090.stdout:crush_version 18 2024-04-14T15:11:29.443 INFO:teuthology.orchestra.run.smithi090.stdout:full_ratio 0.95 2024-04-14T15:11:29.443 INFO:teuthology.orchestra.run.smithi090.stdout:backfillfull_ratio 0.9 2024-04-14T15:11:29.443 INFO:teuthology.orchestra.run.smithi090.stdout:nearfull_ratio 0.85 2024-04-14T15:11:29.443 INFO:teuthology.orchestra.run.smithi090.stdout:require_min_compat_client luminous 2024-04-14T15:11:29.444 INFO:teuthology.orchestra.run.smithi090.stdout:min_compat_client luminous 2024-04-14T15:11:29.444 INFO:teuthology.orchestra.run.smithi090.stdout:require_osd_release squid 2024-04-14T15:11:29.444 INFO:teuthology.orchestra.run.smithi090.stdout:stretch_mode_enabled false 2024-04-14T15:11:29.444 INFO:teuthology.orchestra.run.smithi090.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 7.89 2024-04-14T15:11:29.444 INFO:teuthology.orchestra.run.smithi090.stdout:pool 2 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 66 flags hashpspool stripe_width 0 application rgw read_balance_score 1.50 2024-04-14T15:11:29.444 INFO:teuthology.orchestra.run.smithi090.stdout:pool 3 'datapool' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 3 pgp_num 3 autoscale_mode off last_change 69 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.65 2024-04-14T15:11:29.444 INFO:teuthology.orchestra.run.smithi090.stdout:pool 4 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 68 flags hashpspool stripe_width 0 application rgw read_balance_score 2.25 2024-04-14T15:11:29.444 INFO:teuthology.orchestra.run.smithi090.stdout:pool 5 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 70 flags hashpspool stripe_width 0 application rgw read_balance_score 1.25 2024-04-14T15:11:29.444 INFO:teuthology.orchestra.run.smithi090.stdout:pool 6 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 72 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw read_balance_score 1.75 2024-04-14T15:11:29.444 INFO:teuthology.orchestra.run.smithi090.stdout:max_osd 8 2024-04-14T15:11:29.444 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0 up in weight 1 up_from 8 up_thru 70 down_at 0 last_clean_interval [0,0) v1:172.21.15.90:6801/3761806885 v1:172.21.15.90:6802/3761806885 exists,up 76772cfc-89d1-4013-b617-32284ed8dfd9 2024-04-14T15:11:29.444 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1 up in weight 1 up_from 13 up_thru 70 down_at 0 last_clean_interval [0,0) v1:172.21.15.90:6805/4170361699 v1:172.21.15.90:6806/4170361699 exists,up de4a7fa2-7d57-4ff3-a4fa-28c0cde8ef6c 2024-04-14T15:11:29.444 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2 up in weight 1 up_from 18 up_thru 75 down_at 0 last_clean_interval [0,0) v1:172.21.15.90:6809/1686184251 v1:172.21.15.90:6810/1686184251 exists,up 9c39b5ff-6175-46ba-9d71-370581c6ff3d 2024-04-14T15:11:29.444 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3 up in weight 1 up_from 25 up_thru 70 down_at 0 last_clean_interval [0,0) v1:172.21.15.90:6813/2270166774 v1:172.21.15.90:6814/2270166774 exists,up efd2ca45-5b80-4013-a571-499e2cb5b2e8 2024-04-14T15:11:29.445 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4 up in weight 1 up_from 31 up_thru 75 down_at 0 last_clean_interval [0,0) v1:172.21.15.92:6800/3887549733 v1:172.21.15.92:6801/3887549733 exists,up e3d9013d-5eb6-433c-96b3-3fc38ba7ac8d 2024-04-14T15:11:29.445 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5 up in weight 1 up_from 41 up_thru 70 down_at 0 last_clean_interval [0,0) v1:172.21.15.92:6804/3244561440 v1:172.21.15.92:6805/3244561440 exists,up 1ec41805-6f82-4139-81bc-75305ba9e958 2024-04-14T15:11:29.445 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6 up in weight 1 up_from 49 up_thru 75 down_at 0 last_clean_interval [0,0) v1:172.21.15.92:6808/636125963 v1:172.21.15.92:6809/636125963 exists,up ca977ab2-bf8d-4b3a-ae5e-0efa5bc7a3ad 2024-04-14T15:11:29.445 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7 up in weight 1 up_from 58 up_thru 70 down_at 0 last_clean_interval [0,0) v1:172.21.15.92:6812/1773706716 v1:172.21.15.92:6813/1773706716 exists,up e5354645-79ec-4d4c-b6e5-f9b0653915ee 2024-04-14T15:11:29.445 INFO:teuthology.orchestra.run.smithi090.stdout:pg_upmap_items 2.1c [7,4] 2024-04-14T15:11:29.445 INFO:teuthology.orchestra.run.smithi090.stdout:pg_upmap_items 4.2 [1,2] 2024-04-14T15:11:29.445 INFO:teuthology.orchestra.run.smithi090.stdout:pg_upmap_items 4.d [1,5] 2024-04-14T15:11:29.445 INFO:teuthology.orchestra.run.smithi090.stdout:pg_upmap_items 4.1f [1,2] 2024-04-14T15:11:29.445 INFO:teuthology.orchestra.run.smithi090.stdout:blocklist 172.21.15.90:0/508768479 expires 2024-04-15T15:10:22.918674+0000 2024-04-14T15:11:29.445 INFO:teuthology.orchestra.run.smithi090.stdout:blocklist 172.21.15.90:0/1823072952 expires 2024-04-15T15:10:22.918674+0000 2024-04-14T15:11:29.445 INFO:teuthology.orchestra.run.smithi090.stdout:blocklist 172.21.15.90:6800/491071118 expires 2024-04-15T15:10:22.918674+0000 2024-04-14T15:11:29.445 INFO:teuthology.orchestra.run.smithi090.stdout:blocklist 172.21.15.90:0/1337064035 expires 2024-04-15T15:10:22.918674+0000 2024-04-14T15:11:29.445 INFO:teuthology.orchestra.run.smithi090.stdout:blocklist 172.21.15.90:0/769462306 expires 2024-04-15T15:10:22.918674+0000 2024-04-14T15:11:29.445 INFO:teuthology.orchestra.run.smithi090.stdout:blocklist 172.21.15.90:0/1139868610 expires 2024-04-15T15:05:05.882361+0000 2024-04-14T15:11:29.445 INFO:teuthology.orchestra.run.smithi090.stdout:blocklist 172.21.15.90:0/93763408 expires 2024-04-15T15:10:22.918674+0000 2024-04-14T15:11:29.446 INFO:teuthology.orchestra.run.smithi090.stdout:blocklist 172.21.15.90:0/977818204 expires 2024-04-15T15:04:39.251711+0000 2024-04-14T15:11:29.446 INFO:teuthology.orchestra.run.smithi090.stdout:blocklist 172.21.15.90:6800/941091171 expires 2024-04-15T15:05:05.882361+0000 2024-04-14T15:11:29.446 INFO:teuthology.orchestra.run.smithi090.stdout:blocklist 172.21.15.90:0/3671453561 expires 2024-04-15T15:05:05.882361+0000 2024-04-14T15:11:29.446 INFO:teuthology.orchestra.run.smithi090.stdout:blocklist 172.21.15.90:0/2611856200 expires 2024-04-15T15:05:05.882361+0000 2024-04-14T15:11:29.446 INFO:teuthology.orchestra.run.smithi090.stdout:blocklist 172.21.15.90:6800/2306258962 expires 2024-04-15T15:04:39.251711+0000 2024-04-14T15:11:29.446 INFO:teuthology.orchestra.run.smithi090.stdout:blocklist 172.21.15.90:0/4051692380 expires 2024-04-15T15:04:39.251711+0000 2024-04-14T15:11:29.446 INFO:teuthology.orchestra.run.smithi090.stdout:blocklist 172.21.15.90:0/2681502561 expires 2024-04-15T15:04:39.251711+0000 2024-04-14T15:11:29.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.443+0000 7f088352d700 10 librados: watch_flush enter 2024-04-14T15:11:29.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.443+0000 7f088352d700 10 librados: watch_flush exit 2024-04-14T15:11:29.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.443+0000 7f088352d700 20 client.24679.objecter shutdown clearing up homeless session... 2024-04-14T15:11:29.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.443+0000 7f088352d700 10 client.24679.objecter successfully canceled tick 2024-04-14T15:11:29.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.443+0000 7f088352d700 1 -- 172.21.15.90:0/1413818048 >> v1:172.21.15.90:6800/420935427 conn(0x7f0868071450 legacy=0x7f0868073910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:29.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.443+0000 7f088352d700 1 -- 172.21.15.90:0/1413818048 >> v1:172.21.15.92:6789/0 conn(0x7f087c107890 legacy=0x7f087c06ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:29.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.444+0000 7f088352d700 1 -- 172.21.15.90:0/1413818048 shutdown_connections 2024-04-14T15:11:29.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.444+0000 7f088352d700 1 -- 172.21.15.90:0/1413818048 >> 172.21.15.90:0/1413818048 conn(0x7f087c0fd290 msgr2=0x7f087c101780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:11:29.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.444+0000 7f088352d700 1 -- 172.21.15.90:0/1413818048 shutdown_connections 2024-04-14T15:11:29.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.444+0000 7f088352d700 1 -- 172.21.15.90:0/1413818048 wait complete. 2024-04-14T15:11:29.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:29.444+0000 7f088352d700 1 librados: shutdown 2024-04-14T15:11:30.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:30 smithi090 ceph-mon[41670]: 1.0 deep-scrub starts 2024-04-14T15:11:30.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:30 smithi090 ceph-mon[41670]: 1.0 deep-scrub ok 2024-04-14T15:11:30.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:30 smithi090 ceph-mon[41670]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:11:30.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:30 smithi090 ceph-mon[41670]: 4.12 scrub starts 2024-04-14T15:11:30.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:30 smithi090 ceph-mon[41670]: 4.12 scrub ok 2024-04-14T15:11:30.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:30 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1413818048' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-14T15:11:30.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:30 smithi090 ceph-mon[30399]: 1.0 deep-scrub starts 2024-04-14T15:11:30.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:30 smithi090 ceph-mon[30399]: 1.0 deep-scrub ok 2024-04-14T15:11:30.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:30 smithi090 ceph-mon[30399]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:11:30.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:30 smithi090 ceph-mon[30399]: 4.12 scrub starts 2024-04-14T15:11:30.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:30 smithi090 ceph-mon[30399]: 4.12 scrub ok 2024-04-14T15:11:30.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:30 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1413818048' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-14T15:11:30.314 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 70 down_at 0 last_clean_interval [0,0) v1:172.21.15.90:6801/3761806885 v1:172.21.15.90:6802/3761806885 exists,up 76772cfc-89d1-4013-b617-32284ed8dfd9', 'osd.1 up in weight 1 up_from 13 up_thru 70 down_at 0 last_clean_interval [0,0) v1:172.21.15.90:6805/4170361699 v1:172.21.15.90:6806/4170361699 exists,up de4a7fa2-7d57-4ff3-a4fa-28c0cde8ef6c', 'osd.2 up in weight 1 up_from 18 up_thru 75 down_at 0 last_clean_interval [0,0) v1:172.21.15.90:6809/1686184251 v1:172.21.15.90:6810/1686184251 exists,up 9c39b5ff-6175-46ba-9d71-370581c6ff3d', 'osd.3 up in weight 1 up_from 25 up_thru 70 down_at 0 last_clean_interval [0,0) v1:172.21.15.90:6813/2270166774 v1:172.21.15.90:6814/2270166774 exists,up efd2ca45-5b80-4013-a571-499e2cb5b2e8', 'osd.4 up in weight 1 up_from 31 up_thru 75 down_at 0 last_clean_interval [0,0) v1:172.21.15.92:6800/3887549733 v1:172.21.15.92:6801/3887549733 exists,up e3d9013d-5eb6-433c-96b3-3fc38ba7ac8d', 'osd.5 up in weight 1 up_from 41 up_thru 70 down_at 0 last_clean_interval [0,0) v1:172.21.15.92:6804/3244561440 v1:172.21.15.92:6805/3244561440 exists,up 1ec41805-6f82-4139-81bc-75305ba9e958', 'osd.6 up in weight 1 up_from 49 up_thru 75 down_at 0 last_clean_interval [0,0) v1:172.21.15.92:6808/636125963 v1:172.21.15.92:6809/636125963 exists,up ca977ab2-bf8d-4b3a-ae5e-0efa5bc7a3ad', 'osd.7 up in weight 1 up_from 58 up_thru 70 down_at 0 last_clean_interval [0,0) v1:172.21.15.92:6812/1773706716 v1:172.21.15.92:6813/1773706716 exists,up e5354645-79ec-4d4c-b6e5-f9b0653915ee'] 2024-04-14T15:11:30.315 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon mon.a config show 2024-04-14T15:11:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:30 smithi092 ceph-mon[33966]: 1.0 deep-scrub starts 2024-04-14T15:11:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:30 smithi092 ceph-mon[33966]: 1.0 deep-scrub ok 2024-04-14T15:11:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:30 smithi092 ceph-mon[33966]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:11:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:30 smithi092 ceph-mon[33966]: 4.12 scrub starts 2024-04-14T15:11:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:30 smithi092 ceph-mon[33966]: 4.12 scrub ok 2024-04-14T15:11:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:30 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1413818048' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-14T15:11:30.900 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:11:31.334 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:31 smithi090 ceph-mon[41670]: 5.e scrub starts 2024-04-14T15:11:31.334 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:31 smithi090 ceph-mon[41670]: 5.e scrub ok 2024-04-14T15:11:31.335 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:31 smithi090 ceph-mon[30399]: 5.e scrub starts 2024-04-14T15:11:31.335 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:31 smithi090 ceph-mon[30399]: 5.e scrub ok 2024-04-14T15:11:31.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:31 smithi092 ceph-mon[33966]: 5.e scrub starts 2024-04-14T15:11:31.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:31 smithi092 ceph-mon[33966]: 5.e scrub ok 2024-04-14T15:11:32.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:32 smithi092 ceph-mon[33966]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:11:32.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:32 smithi092 ceph-mon[33966]: 4.7 scrub starts 2024-04-14T15:11:32.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:32 smithi092 ceph-mon[33966]: 4.7 scrub ok 2024-04-14T15:11:32.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:32 smithi092 ceph-mon[33966]: 6.a scrub starts 2024-04-14T15:11:32.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:32 smithi092 ceph-mon[33966]: 6.a scrub ok 2024-04-14T15:11:32.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:32 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:32.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:32 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:32.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:32 smithi090 ceph-mon[41670]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:11:32.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:32 smithi090 ceph-mon[41670]: 4.7 scrub starts 2024-04-14T15:11:32.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:32 smithi090 ceph-mon[41670]: 4.7 scrub ok 2024-04-14T15:11:32.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:32 smithi090 ceph-mon[41670]: 6.a scrub starts 2024-04-14T15:11:32.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:32 smithi090 ceph-mon[41670]: 6.a scrub ok 2024-04-14T15:11:32.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:32 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:32.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:32 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:32.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:32 smithi090 ceph-mon[30399]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:11:32.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:32 smithi090 ceph-mon[30399]: 4.7 scrub starts 2024-04-14T15:11:32.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:32 smithi090 ceph-mon[30399]: 4.7 scrub ok 2024-04-14T15:11:32.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:32 smithi090 ceph-mon[30399]: 6.a scrub starts 2024-04-14T15:11:32.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:32 smithi090 ceph-mon[30399]: 6.a scrub ok 2024-04-14T15:11:32.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:32 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:32.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:32 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:32.735 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-14T15:11:32.735 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "mon.a", 2024-04-14T15:11:32.735 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster": "ceph", 2024-04-14T15:11:32.736 INFO:teuthology.orchestra.run.smithi090.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-04-14T15:11:32.736 INFO:teuthology.orchestra.run.smithi090.stdout: "admin_socket_mode": "", 2024-04-14T15:11:32.736 INFO:teuthology.orchestra.run.smithi090.stdout: "auth_allow_insecure_global_id_reclaim": "false", 2024-04-14T15:11:32.736 INFO:teuthology.orchestra.run.smithi090.stdout: "auth_client_required": "cephx, none", 2024-04-14T15:11:32.736 INFO:teuthology.orchestra.run.smithi090.stdout: "auth_cluster_required": "cephx", 2024-04-14T15:11:32.736 INFO:teuthology.orchestra.run.smithi090.stdout: "auth_debug": "true", 2024-04-14T15:11:32.736 INFO:teuthology.orchestra.run.smithi090.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-04-14T15:11:32.736 INFO:teuthology.orchestra.run.smithi090.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-04-14T15:11:32.736 INFO:teuthology.orchestra.run.smithi090.stdout: "auth_service_required": "cephx", 2024-04-14T15:11:32.736 INFO:teuthology.orchestra.run.smithi090.stdout: "auth_service_ticket_ttl": "240.000000", 2024-04-14T15:11:32.736 INFO:teuthology.orchestra.run.smithi090.stdout: "auth_supported": "", 2024-04-14T15:11:32.736 INFO:teuthology.orchestra.run.smithi090.stdout: "bdev_aio": "true", 2024-04-14T15:11:32.736 INFO:teuthology.orchestra.run.smithi090.stdout: "bdev_aio_max_queue_depth": "1024", 2024-04-14T15:11:32.737 INFO:teuthology.orchestra.run.smithi090.stdout: "bdev_aio_poll_ms": "250", 2024-04-14T15:11:32.737 INFO:teuthology.orchestra.run.smithi090.stdout: "bdev_aio_reap_max": "16", 2024-04-14T15:11:32.737 INFO:teuthology.orchestra.run.smithi090.stdout: "bdev_async_discard_threads": "0", 2024-04-14T15:11:32.737 INFO:teuthology.orchestra.run.smithi090.stdout: "bdev_block_size": "4096", 2024-04-14T15:11:32.737 INFO:teuthology.orchestra.run.smithi090.stdout: "bdev_debug_aio": "false", 2024-04-14T15:11:32.737 INFO:teuthology.orchestra.run.smithi090.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-04-14T15:11:32.737 INFO:teuthology.orchestra.run.smithi090.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-04-14T15:11:32.737 INFO:teuthology.orchestra.run.smithi090.stdout: "bdev_debug_inflight_ios": "false", 2024-04-14T15:11:32.737 INFO:teuthology.orchestra.run.smithi090.stdout: "bdev_enable_discard": "false", 2024-04-14T15:11:32.737 INFO:teuthology.orchestra.run.smithi090.stdout: "bdev_flock_retry": "3", 2024-04-14T15:11:32.737 INFO:teuthology.orchestra.run.smithi090.stdout: "bdev_flock_retry_interval": "0.100000", 2024-04-14T15:11:32.737 INFO:teuthology.orchestra.run.smithi090.stdout: "bdev_inject_crash": "0", 2024-04-14T15:11:32.737 INFO:teuthology.orchestra.run.smithi090.stdout: "bdev_inject_crash_flush_delay": "2", 2024-04-14T15:11:32.738 INFO:teuthology.orchestra.run.smithi090.stdout: "bdev_ioring": "false", 2024-04-14T15:11:32.738 INFO:teuthology.orchestra.run.smithi090.stdout: "bdev_ioring_hipri": "false", 2024-04-14T15:11:32.738 INFO:teuthology.orchestra.run.smithi090.stdout: "bdev_ioring_sqthread_poll": "false", 2024-04-14T15:11:32.738 INFO:teuthology.orchestra.run.smithi090.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-04-14T15:11:32.738 INFO:teuthology.orchestra.run.smithi090.stdout: "bdev_read_buffer_alignment": "4096", 2024-04-14T15:11:32.738 INFO:teuthology.orchestra.run.smithi090.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-04-14T15:11:32.738 INFO:teuthology.orchestra.run.smithi090.stdout: "bdev_type": "", 2024-04-14T15:11:32.738 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_alloc_size": "1048576", 2024-04-14T15:11:32.738 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_allocator": "hybrid", 2024-04-14T15:11:32.738 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_buffered_io": "true", 2024-04-14T15:11:32.738 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_check_for_zeros": "false", 2024-04-14T15:11:32.738 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_check_volume_selector_often": "false", 2024-04-14T15:11:32.738 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-04-14T15:11:32.739 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_compact_log_sync": "false", 2024-04-14T15:11:32.739 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-04-14T15:11:32.739 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-04-14T15:11:32.739 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_log_compact_min_size": "16777216", 2024-04-14T15:11:32.739 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_log_replay_check_allocations": "true", 2024-04-14T15:11:32.739 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_max_log_runway": "4194304", 2024-04-14T15:11:32.739 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_max_prefetch": "1048576", 2024-04-14T15:11:32.739 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_min_flush_size": "524288", 2024-04-14T15:11:32.739 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_min_log_runway": "1048576", 2024-04-14T15:11:32.739 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_replay_recovery": "false", 2024-04-14T15:11:32.739 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-04-14T15:11:32.739 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_shared_alloc_size": "65536", 2024-04-14T15:11:32.739 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_sync_write": "false", 2024-04-14T15:11:32.740 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-04-14T15:11:32.740 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-04-14T15:11:32.740 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-04-14T15:11:32.740 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_allocation_from_file": "true", 2024-04-14T15:11:32.740 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_allocator": "hybrid", 2024-04-14T15:11:32.740 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-04-14T15:11:32.740 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-04-14T15:11:32.740 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-04-14T15:11:32.740 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-04-14T15:11:32.740 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-04-14T15:11:32.740 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-04-14T15:11:32.740 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_blobid_prealloc": "10240", 2024-04-14T15:11:32.740 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_block_create": "true", 2024-04-14T15:11:32.740 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_block_db_create": "false", 2024-04-14T15:11:32.741 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_block_db_path": "", 2024-04-14T15:11:32.741 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_block_db_size": "0", 2024-04-14T15:11:32.741 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_block_path": "", 2024-04-14T15:11:32.741 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_block_preallocate_file": "false", 2024-04-14T15:11:32.741 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_block_size": "107374182400", 2024-04-14T15:11:32.741 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_block_wal_create": "false", 2024-04-14T15:11:32.741 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_block_wal_path": "", 2024-04-14T15:11:32.741 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_block_wal_size": "100663296", 2024-04-14T15:11:32.741 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_bluefs": "true", 2024-04-14T15:11:32.741 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-04-14T15:11:32.741 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_bluefs_env_mirror": "false", 2024-04-14T15:11:32.741 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-04-14T15:11:32.741 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-04-14T15:11:32.741 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-04-14T15:11:32.742 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-04-14T15:11:32.742 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-04-14T15:11:32.742 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-04-14T15:11:32.742 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_autotune": "true", 2024-04-14T15:11:32.742 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-04-14T15:11:32.742 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-04-14T15:11:32.742 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-04-14T15:11:32.742 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-04-14T15:11:32.742 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_size": "0", 2024-04-14T15:11:32.742 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-04-14T15:11:32.742 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-04-14T15:11:32.742 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-04-14T15:11:32.742 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-04-14T15:11:32.743 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_type": "2q", 2024-04-14T15:11:32.743 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-04-14T15:11:32.743 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_clone_cow": "true", 2024-04-14T15:11:32.743 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_compression_algorithm": "snappy", 2024-04-14T15:11:32.743 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_compression_max_blob_size": "0", 2024-04-14T15:11:32.743 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-04-14T15:11:32.743 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-04-14T15:11:32.743 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_compression_min_blob_size": "0", 2024-04-14T15:11:32.743 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-04-14T15:11:32.743 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-04-14T15:11:32.743 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_compression_mode": "none", 2024-04-14T15:11:32.743 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-04-14T15:11:32.743 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_csum_type": "crc32c", 2024-04-14T15:11:32.743 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-04-14T15:11:32.744 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_debug_enforce_settings": "default", 2024-04-14T15:11:32.744 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_debug_freelist": "false", 2024-04-14T15:11:32.744 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_debug_fsck_abort": "false", 2024-04-14T15:11:32.744 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-04-14T15:11:32.744 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-04-14T15:11:32.744 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_debug_inject_read_err": "false", 2024-04-14T15:11:32.744 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_debug_legacy_omap": "false", 2024-04-14T15:11:32.744 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-04-14T15:11:32.744 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-04-14T15:11:32.744 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-04-14T15:11:32.744 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-04-14T15:11:32.744 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_debug_prefill": "0.000000", 2024-04-14T15:11:32.744 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-04-14T15:11:32.744 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-04-14T15:11:32.745 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-04-14T15:11:32.745 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_debug_small_allocations": "0", 2024-04-14T15:11:32.745 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-04-14T15:11:32.745 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_default_buffered_read": "true", 2024-04-14T15:11:32.745 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_default_buffered_write": "false", 2024-04-14T15:11:32.745 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_deferred_batch_ops": "0", 2024-04-14T15:11:32.745 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-04-14T15:11:32.745 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-04-14T15:11:32.745 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_elastic_shared_blobs": "true", 2024-04-14T15:11:32.745 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-04-14T15:11:32.745 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-04-14T15:11:32.745 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-04-14T15:11:32.745 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-04-14T15:11:32.745 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-04-14T15:11:32.746 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_fail_eio": "false", 2024-04-14T15:11:32.746 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-04-14T15:11:32.746 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-04-14T15:11:32.746 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-04-14T15:11:32.746 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-04-14T15:11:32.746 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_fsck_on_mkfs": "true", 2024-04-14T15:11:32.746 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-04-14T15:11:32.746 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_fsck_on_mount": "false", 2024-04-14T15:11:32.746 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-04-14T15:11:32.746 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_fsck_on_umount": "false", 2024-04-14T15:11:32.746 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-04-14T15:11:32.747 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-04-14T15:11:32.747 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-04-14T15:11:32.747 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-04-14T15:11:32.747 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-04-14T15:11:32.747 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-04-14T15:11:32.747 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-04-14T15:11:32.747 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-04-14T15:11:32.747 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_ignore_data_csum": "false", 2024-04-14T15:11:32.747 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-04-14T15:11:32.747 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_kvbackend": "rocksdb", 2024-04-14T15:11:32.747 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-04-14T15:11:32.747 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-04-14T15:11:32.748 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_log_op_age": "5.000000", 2024-04-14T15:11:32.748 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_max_alloc_size": "0", 2024-04-14T15:11:32.748 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_max_blob_size": "0", 2024-04-14T15:11:32.748 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-04-14T15:11:32.748 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-04-14T15:11:32.748 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_max_defer_interval": "3.000000", 2024-04-14T15:11:32.748 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_max_deferred_txc": "32", 2024-04-14T15:11:32.748 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_min_alloc_size": "0", 2024-04-14T15:11:32.748 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-04-14T15:11:32.748 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-04-14T15:11:32.748 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_nid_prealloc": "1024", 2024-04-14T15:11:32.748 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_prefer_deferred_size": "0", 2024-04-14T15:11:32.748 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-04-14T15:11:32.748 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-04-14T15:11:32.748 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_qfsck_on_mount": "true", 2024-04-14T15:11:32.749 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_retry_disk_reads": "3", 2024-04-14T15:11:32.749 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_rocksdb_cf": "true", 2024-04-14T15:11:32.749 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-04-14T15:11:32.750 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-04-14T15:11:32.750 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_rocksdb_options_annex": "", 2024-04-14T15:11:32.750 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_spdk_coremask": "0x1", 2024-04-14T15:11:32.750 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_spdk_io_sleep": "5", 2024-04-14T15:11:32.750 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_spdk_max_io_completion": "0", 2024-04-14T15:11:32.750 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_spdk_mem": "512", 2024-04-14T15:11:32.750 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_sync_submit_transaction": "false", 2024-04-14T15:11:32.750 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_throttle_bytes": "67108864", 2024-04-14T15:11:32.750 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_throttle_cost_per_io": "0", 2024-04-14T15:11:32.751 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-04-14T15:11:32.751 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-04-14T15:11:32.751 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-04-14T15:11:32.751 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-04-14T15:11:32.751 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_tracing": "false", 2024-04-14T15:11:32.751 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-04-14T15:11:32.751 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-04-14T15:11:32.751 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_volume_selection_reserved": "0", 2024-04-14T15:11:32.751 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-04-14T15:11:32.751 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-04-14T15:11:32.751 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-04-14T15:11:32.751 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-04-14T15:11:32.751 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-04-14T15:11:32.752 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-04-14T15:11:32.752 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_zero_block_detection": "false", 2024-04-14T15:11:32.752 INFO:teuthology.orchestra.run.smithi090.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-04-14T15:11:32.752 INFO:teuthology.orchestra.run.smithi090.stdout: "cephfs_mirror_action_update_interval": "2", 2024-04-14T15:11:32.752 INFO:teuthology.orchestra.run.smithi090.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-04-14T15:11:32.752 INFO:teuthology.orchestra.run.smithi090.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-04-14T15:11:32.752 INFO:teuthology.orchestra.run.smithi090.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-04-14T15:11:32.752 INFO:teuthology.orchestra.run.smithi090.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-04-14T15:11:32.752 INFO:teuthology.orchestra.run.smithi090.stdout: "cephfs_mirror_mount_timeout": "10", 2024-04-14T15:11:32.752 INFO:teuthology.orchestra.run.smithi090.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-04-14T15:11:32.752 INFO:teuthology.orchestra.run.smithi090.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-04-14T15:11:32.752 INFO:teuthology.orchestra.run.smithi090.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-04-14T15:11:32.752 INFO:teuthology.orchestra.run.smithi090.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-04-14T15:11:32.753 INFO:teuthology.orchestra.run.smithi090.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-04-14T15:11:32.753 INFO:teuthology.orchestra.run.smithi090.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-04-14T15:11:32.753 INFO:teuthology.orchestra.run.smithi090.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-04-14T15:11:32.753 INFO:teuthology.orchestra.run.smithi090.stdout: "cephx_cluster_require_signatures": "false", 2024-04-14T15:11:32.753 INFO:teuthology.orchestra.run.smithi090.stdout: "cephx_cluster_require_version": "2", 2024-04-14T15:11:32.753 INFO:teuthology.orchestra.run.smithi090.stdout: "cephx_require_signatures": "false", 2024-04-14T15:11:32.753 INFO:teuthology.orchestra.run.smithi090.stdout: "cephx_require_version": "2", 2024-04-14T15:11:32.753 INFO:teuthology.orchestra.run.smithi090.stdout: "cephx_service_require_signatures": "false", 2024-04-14T15:11:32.753 INFO:teuthology.orchestra.run.smithi090.stdout: "cephx_service_require_version": "2", 2024-04-14T15:11:32.753 INFO:teuthology.orchestra.run.smithi090.stdout: "cephx_sign_messages": "true", 2024-04-14T15:11:32.753 INFO:teuthology.orchestra.run.smithi090.stdout: "chdir": "", 2024-04-14T15:11:32.753 INFO:teuthology.orchestra.run.smithi090.stdout: "client_acl_type": "", 2024-04-14T15:11:32.753 INFO:teuthology.orchestra.run.smithi090.stdout: "client_asio_thread_count": "2", 2024-04-14T15:11:32.753 INFO:teuthology.orchestra.run.smithi090.stdout: "client_cache_mid": "0.750000", 2024-04-14T15:11:32.754 INFO:teuthology.orchestra.run.smithi090.stdout: "client_cache_size": "16384", 2024-04-14T15:11:32.754 INFO:teuthology.orchestra.run.smithi090.stdout: "client_caps_release_delay": "5", 2024-04-14T15:11:32.754 INFO:teuthology.orchestra.run.smithi090.stdout: "client_check_pool_perm": "true", 2024-04-14T15:11:32.754 INFO:teuthology.orchestra.run.smithi090.stdout: "client_collect_and_send_global_metrics": "false", 2024-04-14T15:11:32.754 INFO:teuthology.orchestra.run.smithi090.stdout: "client_debug_force_sync_read": "false", 2024-04-14T15:11:32.754 INFO:teuthology.orchestra.run.smithi090.stdout: "client_debug_getattr_caps": "false", 2024-04-14T15:11:32.754 INFO:teuthology.orchestra.run.smithi090.stdout: "client_debug_inject_tick_delay": "0", 2024-04-14T15:11:32.754 INFO:teuthology.orchestra.run.smithi090.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-04-14T15:11:32.754 INFO:teuthology.orchestra.run.smithi090.stdout: "client_die_on_failed_remount": "false", 2024-04-14T15:11:32.754 INFO:teuthology.orchestra.run.smithi090.stdout: "client_dirsize_rbytes": "true", 2024-04-14T15:11:32.754 INFO:teuthology.orchestra.run.smithi090.stdout: "client_force_lazyio": "false", 2024-04-14T15:11:32.754 INFO:teuthology.orchestra.run.smithi090.stdout: "client_fs": "", 2024-04-14T15:11:32.755 INFO:teuthology.orchestra.run.smithi090.stdout: "client_inject_fixed_oldest_tid": "false", 2024-04-14T15:11:32.755 INFO:teuthology.orchestra.run.smithi090.stdout: "client_inject_release_failure": "false", 2024-04-14T15:11:32.755 INFO:teuthology.orchestra.run.smithi090.stdout: "client_max_inline_size": "4096", 2024-04-14T15:11:32.755 INFO:teuthology.orchestra.run.smithi090.stdout: "client_max_retries_on_remount_failure": "5", 2024-04-14T15:11:32.755 INFO:teuthology.orchestra.run.smithi090.stdout: "client_mds_namespace": "", 2024-04-14T15:11:32.755 INFO:teuthology.orchestra.run.smithi090.stdout: "client_metadata": "", 2024-04-14T15:11:32.755 INFO:teuthology.orchestra.run.smithi090.stdout: "client_mount_gid": "-1", 2024-04-14T15:11:32.755 INFO:teuthology.orchestra.run.smithi090.stdout: "client_mount_timeout": "300", 2024-04-14T15:11:32.755 INFO:teuthology.orchestra.run.smithi090.stdout: "client_mount_uid": "-1", 2024-04-14T15:11:32.755 INFO:teuthology.orchestra.run.smithi090.stdout: "client_mountpoint": "/", 2024-04-14T15:11:32.755 INFO:teuthology.orchestra.run.smithi090.stdout: "client_notify_timeout": "10", 2024-04-14T15:11:32.755 INFO:teuthology.orchestra.run.smithi090.stdout: "client_oc": "true", 2024-04-14T15:11:32.755 INFO:teuthology.orchestra.run.smithi090.stdout: "client_oc_max_dirty": "104857600", 2024-04-14T15:11:32.755 INFO:teuthology.orchestra.run.smithi090.stdout: "client_oc_max_dirty_age": "5.000000", 2024-04-14T15:11:32.756 INFO:teuthology.orchestra.run.smithi090.stdout: "client_oc_max_objects": "1000", 2024-04-14T15:11:32.756 INFO:teuthology.orchestra.run.smithi090.stdout: "client_oc_size": "209715200", 2024-04-14T15:11:32.756 INFO:teuthology.orchestra.run.smithi090.stdout: "client_oc_target_dirty": "8388608", 2024-04-14T15:11:32.756 INFO:teuthology.orchestra.run.smithi090.stdout: "client_permissions": "true", 2024-04-14T15:11:32.756 INFO:teuthology.orchestra.run.smithi090.stdout: "client_quota": "true", 2024-04-14T15:11:32.756 INFO:teuthology.orchestra.run.smithi090.stdout: "client_quota_df": "true", 2024-04-14T15:11:32.756 INFO:teuthology.orchestra.run.smithi090.stdout: "client_readahead_max_bytes": "0", 2024-04-14T15:11:32.756 INFO:teuthology.orchestra.run.smithi090.stdout: "client_readahead_max_periods": "4", 2024-04-14T15:11:32.756 INFO:teuthology.orchestra.run.smithi090.stdout: "client_readahead_min": "131072", 2024-04-14T15:11:32.756 INFO:teuthology.orchestra.run.smithi090.stdout: "client_reconnect_stale": "false", 2024-04-14T15:11:32.756 INFO:teuthology.orchestra.run.smithi090.stdout: "client_shutdown_timeout": "30", 2024-04-14T15:11:32.757 INFO:teuthology.orchestra.run.smithi090.stdout: "client_snapdir": ".snap", 2024-04-14T15:11:32.757 INFO:teuthology.orchestra.run.smithi090.stdout: "client_tick_interval": "1", 2024-04-14T15:11:32.757 INFO:teuthology.orchestra.run.smithi090.stdout: "client_trace": "", 2024-04-14T15:11:32.757 INFO:teuthology.orchestra.run.smithi090.stdout: "client_try_dentry_invalidate": "false", 2024-04-14T15:11:32.757 INFO:teuthology.orchestra.run.smithi090.stdout: "client_use_faked_inos": "false", 2024-04-14T15:11:32.757 INFO:teuthology.orchestra.run.smithi090.stdout: "client_use_random_mds": "false", 2024-04-14T15:11:32.757 INFO:teuthology.orchestra.run.smithi090.stdout: "clog_to_graylog": "false", 2024-04-14T15:11:32.757 INFO:teuthology.orchestra.run.smithi090.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-04-14T15:11:32.757 INFO:teuthology.orchestra.run.smithi090.stdout: "clog_to_graylog_port": "12201", 2024-04-14T15:11:32.757 INFO:teuthology.orchestra.run.smithi090.stdout: "clog_to_monitors": "default=true", 2024-04-14T15:11:32.757 INFO:teuthology.orchestra.run.smithi090.stdout: "clog_to_syslog": "false", 2024-04-14T15:11:32.757 INFO:teuthology.orchestra.run.smithi090.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-04-14T15:11:32.757 INFO:teuthology.orchestra.run.smithi090.stdout: "clog_to_syslog_level": "info", 2024-04-14T15:11:32.758 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "-", 2024-04-14T15:11:32.758 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_network": "", 2024-04-14T15:11:32.758 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_network_interface": "", 2024-04-14T15:11:32.758 INFO:teuthology.orchestra.run.smithi090.stdout: "compressor_zlib_isal": "false", 2024-04-14T15:11:32.758 INFO:teuthology.orchestra.run.smithi090.stdout: "compressor_zlib_level": "5", 2024-04-14T15:11:32.758 INFO:teuthology.orchestra.run.smithi090.stdout: "compressor_zlib_winsize": "-15", 2024-04-14T15:11:32.758 INFO:teuthology.orchestra.run.smithi090.stdout: "compressor_zstd_level": "1", 2024-04-14T15:11:32.758 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608", 2024-04-14T15:11:32.759 INFO:teuthology.orchestra.run.smithi090.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-04-14T15:11:32.759 INFO:teuthology.orchestra.run.smithi090.stdout: "crimson_alien_op_num_threads": "6", 2024-04-14T15:11:32.759 INFO:teuthology.orchestra.run.smithi090.stdout: "crimson_alien_thread_cpu_cores": "", 2024-04-14T15:11:32.759 INFO:teuthology.orchestra.run.smithi090.stdout: "crimson_osd_obc_lru_size": "512", 2024-04-14T15:11:32.759 INFO:teuthology.orchestra.run.smithi090.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-04-14T15:11:32.759 INFO:teuthology.orchestra.run.smithi090.stdout: "crimson_seastar_cpu_cores": "", 2024-04-14T15:11:32.759 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_location": "", 2024-04-14T15:11:32.759 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_location_hook": "", 2024-04-14T15:11:32.759 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_location_hook_timeout": "10", 2024-04-14T15:11:32.759 INFO:teuthology.orchestra.run.smithi090.stdout: "daemonize": "false", 2024-04-14T15:11:32.759 INFO:teuthology.orchestra.run.smithi090.stdout: "daos_pool": "tank", 2024-04-14T15:11:32.759 INFO:teuthology.orchestra.run.smithi090.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-04-14T15:11:32.759 INFO:teuthology.orchestra.run.smithi090.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-04-14T15:11:32.759 INFO:teuthology.orchestra.run.smithi090.stdout: "dbstore_db_name_prefix": "dbstore", 2024-04-14T15:11:32.760 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_alienstore": "0/5", 2024-04-14T15:11:32.760 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_allow_any_pool_priority": "false", 2024-04-14T15:11:32.760 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_asok": "1/5", 2024-04-14T15:11:32.760 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_asok_assert_abort": "false", 2024-04-14T15:11:32.760 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_asserts_on_shutdown": "true", 2024-04-14T15:11:32.760 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_auth": "1/5", 2024-04-14T15:11:32.760 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_bdev": "1/3", 2024-04-14T15:11:32.760 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_bluefs": "1/5", 2024-04-14T15:11:32.760 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_bluestore": "1/5", 2024-04-14T15:11:32.760 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_buffer": "0/1", 2024-04-14T15:11:32.760 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_ceph_exporter": "1/5", 2024-04-14T15:11:32.760 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_cephfs_mirror": "0/5", 2024-04-14T15:11:32.760 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_cephsqlite": "0/5", 2024-04-14T15:11:32.760 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_client": "0/5", 2024-04-14T15:11:32.761 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_compressor": "1/5", 2024-04-14T15:11:32.761 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_context": "0/1", 2024-04-14T15:11:32.761 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_crimson_interrupt": "0/5", 2024-04-14T15:11:32.761 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_crush": "1/1", 2024-04-14T15:11:32.761 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_crypto": "1/5", 2024-04-14T15:11:32.761 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_cyanstore": "0/5", 2024-04-14T15:11:32.761 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_deliberately_leak_memory": "false", 2024-04-14T15:11:32.761 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_disable_randomized_ping": "false", 2024-04-14T15:11:32.761 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_dpdk": "1/5", 2024-04-14T15:11:32.762 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_eventtrace": "1/5", 2024-04-14T15:11:32.762 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_filer": "0/1", 2024-04-14T15:11:32.762 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_filestore": "1/3", 2024-04-14T15:11:32.762 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_finisher": "1/1", 2024-04-14T15:11:32.762 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_fuse": "1/5", 2024-04-14T15:11:32.762 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_heartbeat_testing_span": "0", 2024-04-14T15:11:32.762 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_heartbeatmap": "1/5", 2024-04-14T15:11:32.762 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_immutable_obj_cache": "0/5", 2024-04-14T15:11:32.762 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_javaclient": "1/5", 2024-04-14T15:11:32.762 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_journal": "1/3", 2024-04-14T15:11:32.762 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_journaler": "0/5", 2024-04-14T15:11:32.762 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_kstore": "1/5", 2024-04-14T15:11:32.762 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_lockdep": "0/1", 2024-04-14T15:11:32.763 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_mclock": "1/5", 2024-04-14T15:11:32.763 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_mds": "1/5", 2024-04-14T15:11:32.763 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_mds_balancer": "1/5", 2024-04-14T15:11:32.763 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_mds_locker": "1/5", 2024-04-14T15:11:32.763 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_mds_log": "1/5", 2024-04-14T15:11:32.763 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_mds_log_expire": "1/5", 2024-04-14T15:11:32.763 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_mds_migrator": "1/5", 2024-04-14T15:11:32.763 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_mds_quiesce": "3/5", 2024-04-14T15:11:32.763 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_memstore": "1/5", 2024-04-14T15:11:32.763 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_mgr": "2/5", 2024-04-14T15:11:32.763 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_mgrc": "20/20", 2024-04-14T15:11:32.763 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_mon": "20/20", 2024-04-14T15:11:32.763 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_monc": "0/10", 2024-04-14T15:11:32.764 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_ms": "1/1", 2024-04-14T15:11:32.764 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_none": "0/5", 2024-04-14T15:11:32.764 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_objclass": "0/5", 2024-04-14T15:11:32.764 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_objectcacher": "0/5", 2024-04-14T15:11:32.764 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_objecter": "0/1", 2024-04-14T15:11:32.764 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_optracker": "0/5", 2024-04-14T15:11:32.764 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_osd": "1/5", 2024-04-14T15:11:32.764 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_paxos": "20/20", 2024-04-14T15:11:32.764 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_perfcounter": "1/5", 2024-04-14T15:11:32.764 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_prioritycache": "1/5", 2024-04-14T15:11:32.764 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_rados": "0/5", 2024-04-14T15:11:32.764 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_rbd": "0/5", 2024-04-14T15:11:32.764 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_rbd_mirror": "0/5", 2024-04-14T15:11:32.764 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_rbd_pwl": "0/5", 2024-04-14T15:11:32.765 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_rbd_replay": "0/5", 2024-04-14T15:11:32.765 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_refs": "0/0", 2024-04-14T15:11:32.765 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_reserver": "1/1", 2024-04-14T15:11:32.765 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_rgw": "1/5", 2024-04-14T15:11:32.765 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_rgw_access": "1/5", 2024-04-14T15:11:32.765 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_rgw_datacache": "1/5", 2024-04-14T15:11:32.765 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_rgw_dbstore": "1/5", 2024-04-14T15:11:32.765 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_rgw_flight": "1/5", 2024-04-14T15:11:32.765 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_rgw_lifecycle": "1/5", 2024-04-14T15:11:32.765 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_rgw_notification": "1/5", 2024-04-14T15:11:32.765 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_rgw_sync": "1/5", 2024-04-14T15:11:32.765 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_rocksdb": "4/5", 2024-04-14T15:11:32.765 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_seastore": "0/5", 2024-04-14T15:11:32.765 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_seastore_backref": "0/5", 2024-04-14T15:11:32.765 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_seastore_cache": "0/5", 2024-04-14T15:11:32.766 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_seastore_cleaner": "0/5", 2024-04-14T15:11:32.766 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_seastore_device": "0/5", 2024-04-14T15:11:32.766 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_seastore_epm": "0/5", 2024-04-14T15:11:32.766 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-04-14T15:11:32.766 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_seastore_journal": "0/5", 2024-04-14T15:11:32.766 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_seastore_lba": "0/5", 2024-04-14T15:11:32.766 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_seastore_odata": "0/5", 2024-04-14T15:11:32.766 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_seastore_omap": "0/5", 2024-04-14T15:11:32.766 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_seastore_onode": "0/5", 2024-04-14T15:11:32.766 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_seastore_t": "0/5", 2024-04-14T15:11:32.766 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_seastore_tm": "0/5", 2024-04-14T15:11:32.766 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_striper": "0/1", 2024-04-14T15:11:32.766 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_test": "0/5", 2024-04-14T15:11:32.767 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_throttle": "1/1", 2024-04-14T15:11:32.767 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_timer": "0/1", 2024-04-14T15:11:32.767 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_tp": "0/5", 2024-04-14T15:11:32.767 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_trace": "1/5", 2024-04-14T15:11:32.767 INFO:teuthology.orchestra.run.smithi090.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-04-14T15:11:32.767 INFO:teuthology.orchestra.run.smithi090.stdout: "device_failure_prediction_mode": "none", 2024-04-14T15:11:32.767 INFO:teuthology.orchestra.run.smithi090.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-04-14T15:11:32.767 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-04-14T15:11:32.767 INFO:teuthology.orchestra.run.smithi090.stdout: "err_to_graylog": "false", 2024-04-14T15:11:32.767 INFO:teuthology.orchestra.run.smithi090.stdout: "err_to_journald": "false", 2024-04-14T15:11:32.767 INFO:teuthology.orchestra.run.smithi090.stdout: "err_to_stderr": "true", 2024-04-14T15:11:32.767 INFO:teuthology.orchestra.run.smithi090.stdout: "err_to_syslog": "false", 2024-04-14T15:11:32.767 INFO:teuthology.orchestra.run.smithi090.stdout: "event_tracing": "false", 2024-04-14T15:11:32.767 INFO:teuthology.orchestra.run.smithi090.stdout: "exporter_addr": "0.0.0.0", 2024-04-14T15:11:32.768 INFO:teuthology.orchestra.run.smithi090.stdout: "exporter_http_port": "9926", 2024-04-14T15:11:32.768 INFO:teuthology.orchestra.run.smithi090.stdout: "exporter_prio_limit": "5", 2024-04-14T15:11:32.768 INFO:teuthology.orchestra.run.smithi090.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-04-14T15:11:32.768 INFO:teuthology.orchestra.run.smithi090.stdout: "exporter_sort_metrics": "true", 2024-04-14T15:11:32.768 INFO:teuthology.orchestra.run.smithi090.stdout: "exporter_stats_period": "5", 2024-04-14T15:11:32.768 INFO:teuthology.orchestra.run.smithi090.stdout: "fake_statfs_for_testing": "0", 2024-04-14T15:11:32.768 INFO:teuthology.orchestra.run.smithi090.stdout: "fatal_signal_handlers": "true", 2024-04-14T15:11:32.768 INFO:teuthology.orchestra.run.smithi090.stdout: "filer_max_purge_ops": "10", 2024-04-14T15:11:32.768 INFO:teuthology.orchestra.run.smithi090.stdout: "filer_max_truncate_ops": "128", 2024-04-14T15:11:32.768 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_apply_finisher_threads": "1", 2024-04-14T15:11:32.768 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_blackhole": "false", 2024-04-14T15:11:32.768 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_btrfs_clone_range": "true", 2024-04-14T15:11:32.768 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_btrfs_snap": "true", 2024-04-14T15:11:32.768 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_caller_concurrency": "10", 2024-04-14T15:11:32.769 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_collect_device_partition_information": "true", 2024-04-14T15:11:32.769 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_commit_timeout": "600.000000", 2024-04-14T15:11:32.769 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_debug_inject_read_err": "false", 2024-04-14T15:11:32.769 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_debug_omap_check": "false", 2024-04-14T15:11:32.769 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_debug_verify_split": "false", 2024-04-14T15:11:32.769 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_dump_file": "", 2024-04-14T15:11:32.769 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-04-14T15:11:32.769 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-04-14T15:11:32.769 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_fadvise": "true", 2024-04-14T15:11:32.769 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_fail_eio": "true", 2024-04-14T15:11:32.769 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_fd_cache_shards": "16", 2024-04-14T15:11:32.769 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_fd_cache_size": "128", 2024-04-14T15:11:32.769 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_fiemap": "false", 2024-04-14T15:11:32.769 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_fiemap_threshold": "4096", 2024-04-14T15:11:32.769 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-04-14T15:11:32.770 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_index_retry_probability": "0.000000", 2024-04-14T15:11:32.770 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_inject_stall": "0", 2024-04-14T15:11:32.770 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_journal_parallel": "false", 2024-04-14T15:11:32.770 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_journal_trailing": "false", 2024-04-14T15:11:32.770 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_journal_writeahead": "false", 2024-04-14T15:11:32.770 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_kill_at": "0", 2024-04-14T15:11:32.770 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-04-14T15:11:32.770 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_max_inline_xattr_size": "0", 2024-04-14T15:11:32.770 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-04-14T15:11:32.770 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-04-14T15:11:32.770 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-04-14T15:11:32.770 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_max_inline_xattrs": "0", 2024-04-14T15:11:32.770 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-04-14T15:11:32.770 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_max_inline_xattrs_other": "2", 2024-04-14T15:11:32.771 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-04-14T15:11:32.771 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_max_sync_interval": "5.000000", 2024-04-14T15:11:32.771 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_max_xattr_value_size": "0", 2024-04-14T15:11:32.771 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-04-14T15:11:32.771 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-04-14T15:11:32.771 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-04-14T15:11:32.771 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_merge_threshold": "-10", 2024-04-14T15:11:32.771 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_min_sync_interval": "0.010000", 2024-04-14T15:11:32.771 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_odsync_write": "false", 2024-04-14T15:11:32.771 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_omap_backend": "rocksdb", 2024-04-14T15:11:32.771 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_omap_backend_path": "", 2024-04-14T15:11:32.771 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_omap_header_cache_size": "1024", 2024-04-14T15:11:32.771 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_ondisk_finisher_threads": "1", 2024-04-14T15:11:32.771 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-04-14T15:11:32.772 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_op_thread_timeout": "60", 2024-04-14T15:11:32.772 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_op_threads": "2", 2024-04-14T15:11:32.772 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_punch_hole": "false", 2024-04-14T15:11:32.772 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-04-14T15:11:32.772 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-04-14T15:11:32.772 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-04-14T15:11:32.772 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-04-14T15:11:32.772 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-04-14T15:11:32.772 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_queue_max_bytes": "104857600", 2024-04-14T15:11:32.772 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-04-14T15:11:32.772 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-04-14T15:11:32.772 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-04-14T15:11:32.772 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_queue_max_ops": "50", 2024-04-14T15:11:32.772 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-04-14T15:11:32.773 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_seek_data_hole": "false", 2024-04-14T15:11:32.773 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_sloppy_crc": "false", 2024-04-14T15:11:32.773 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-04-14T15:11:32.773 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_splice": "false", 2024-04-14T15:11:32.773 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_split_multiple": "2", 2024-04-14T15:11:32.773 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_split_rand_factor": "20", 2024-04-14T15:11:32.773 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_update_to": "1000", 2024-04-14T15:11:32.773 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-04-14T15:11:32.773 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-04-14T15:11:32.773 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-04-14T15:11:32.773 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-04-14T15:11:32.773 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-04-14T15:11:32.773 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-04-14T15:11:32.773 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_wbthrottle_enable": "true", 2024-04-14T15:11:32.774 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-04-14T15:11:32.774 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-04-14T15:11:32.774 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-04-14T15:11:32.774 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-04-14T15:11:32.774 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-04-14T15:11:32.774 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-04-14T15:11:32.774 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_xfs_extsize": "false", 2024-04-14T15:11:32.774 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_zfs_snap": "false", 2024-04-14T15:11:32.774 INFO:teuthology.orchestra.run.smithi090.stdout: "fio_dir": "/tmp/fio", 2024-04-14T15:11:32.774 INFO:teuthology.orchestra.run.smithi090.stdout: "fsid": "0e288098-fa70-11ee-bc8e-c7b262605968", 2024-04-14T15:11:32.774 INFO:teuthology.orchestra.run.smithi090.stdout: "fuse_allow_other": "true", 2024-04-14T15:11:32.774 INFO:teuthology.orchestra.run.smithi090.stdout: "fuse_atomic_o_trunc": "true", 2024-04-14T15:11:32.775 INFO:teuthology.orchestra.run.smithi090.stdout: "fuse_big_writes": "true", 2024-04-14T15:11:32.775 INFO:teuthology.orchestra.run.smithi090.stdout: "fuse_debug": "false", 2024-04-14T15:11:32.775 INFO:teuthology.orchestra.run.smithi090.stdout: "fuse_default_permissions": "false", 2024-04-14T15:11:32.775 INFO:teuthology.orchestra.run.smithi090.stdout: "fuse_disable_pagecache": "false", 2024-04-14T15:11:32.775 INFO:teuthology.orchestra.run.smithi090.stdout: "fuse_max_write": "0", 2024-04-14T15:11:32.775 INFO:teuthology.orchestra.run.smithi090.stdout: "fuse_multithreaded": "true", 2024-04-14T15:11:32.775 INFO:teuthology.orchestra.run.smithi090.stdout: "fuse_require_active_mds": "true", 2024-04-14T15:11:32.775 INFO:teuthology.orchestra.run.smithi090.stdout: "fuse_set_user_groups": "true", 2024-04-14T15:11:32.775 INFO:teuthology.orchestra.run.smithi090.stdout: "fuse_splice_move": "true", 2024-04-14T15:11:32.775 INFO:teuthology.orchestra.run.smithi090.stdout: "fuse_splice_read": "true", 2024-04-14T15:11:32.775 INFO:teuthology.orchestra.run.smithi090.stdout: "fuse_splice_write": "true", 2024-04-14T15:11:32.775 INFO:teuthology.orchestra.run.smithi090.stdout: "fuse_syncfs_on_mksnap": "true", 2024-04-14T15:11:32.775 INFO:teuthology.orchestra.run.smithi090.stdout: "fuse_use_invalidate_cb": "true", 2024-04-14T15:11:32.776 INFO:teuthology.orchestra.run.smithi090.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-04-14T15:11:32.776 INFO:teuthology.orchestra.run.smithi090.stdout: "gss_target_name": "ceph", 2024-04-14T15:11:32.776 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_file": "", 2024-04-14T15:11:32.776 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_inject_failure": "0", 2024-04-14T15:11:32.776 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_interval": "5", 2024-04-14T15:11:32.776 INFO:teuthology.orchestra.run.smithi090.stdout: "host": "smithi090", 2024-04-14T15:11:32.776 INFO:teuthology.orchestra.run.smithi090.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-04-14T15:11:32.776 INFO:teuthology.orchestra.run.smithi090.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-04-14T15:11:32.776 INFO:teuthology.orchestra.run.smithi090.stdout: "immutable_object_cache_max_size": "1073741824", 2024-04-14T15:11:32.776 INFO:teuthology.orchestra.run.smithi090.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-04-14T15:11:32.776 INFO:teuthology.orchestra.run.smithi090.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-04-14T15:11:32.776 INFO:teuthology.orchestra.run.smithi090.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-04-14T15:11:32.776 INFO:teuthology.orchestra.run.smithi090.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-04-14T15:11:32.777 INFO:teuthology.orchestra.run.smithi090.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-04-14T15:11:32.777 INFO:teuthology.orchestra.run.smithi090.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-04-14T15:11:32.777 INFO:teuthology.orchestra.run.smithi090.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-04-14T15:11:32.777 INFO:teuthology.orchestra.run.smithi090.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-04-14T15:11:32.777 INFO:teuthology.orchestra.run.smithi090.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-04-14T15:11:32.777 INFO:teuthology.orchestra.run.smithi090.stdout: "immutable_object_cache_watermark": "0.900000", 2024-04-14T15:11:32.777 INFO:teuthology.orchestra.run.smithi090.stdout: "inject_early_sigterm": "false", 2024-04-14T15:11:32.777 INFO:teuthology.orchestra.run.smithi090.stdout: "jaeger_agent_port": "6799", 2024-04-14T15:11:32.777 INFO:teuthology.orchestra.run.smithi090.stdout: "jaeger_tracing_enable": "false", 2024-04-14T15:11:32.777 INFO:teuthology.orchestra.run.smithi090.stdout: "journal_aio": "true", 2024-04-14T15:11:32.777 INFO:teuthology.orchestra.run.smithi090.stdout: "journal_align_min_size": "65536", 2024-04-14T15:11:32.777 INFO:teuthology.orchestra.run.smithi090.stdout: "journal_block_align": "true", 2024-04-14T15:11:32.777 INFO:teuthology.orchestra.run.smithi090.stdout: "journal_block_size": "4096", 2024-04-14T15:11:32.777 INFO:teuthology.orchestra.run.smithi090.stdout: "journal_dio": "true", 2024-04-14T15:11:32.778 INFO:teuthology.orchestra.run.smithi090.stdout: "journal_discard": "false", 2024-04-14T15:11:32.778 INFO:teuthology.orchestra.run.smithi090.stdout: "journal_force_aio": "false", 2024-04-14T15:11:32.778 INFO:teuthology.orchestra.run.smithi090.stdout: "journal_ignore_corruption": "false", 2024-04-14T15:11:32.778 INFO:teuthology.orchestra.run.smithi090.stdout: "journal_max_write_bytes": "10485760", 2024-04-14T15:11:32.778 INFO:teuthology.orchestra.run.smithi090.stdout: "journal_max_write_entries": "100", 2024-04-14T15:11:32.778 INFO:teuthology.orchestra.run.smithi090.stdout: "journal_replay_from": "0", 2024-04-14T15:11:32.778 INFO:teuthology.orchestra.run.smithi090.stdout: "journal_throttle_high_multiple": "0.000000", 2024-04-14T15:11:32.778 INFO:teuthology.orchestra.run.smithi090.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-04-14T15:11:32.778 INFO:teuthology.orchestra.run.smithi090.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-04-14T15:11:32.778 INFO:teuthology.orchestra.run.smithi090.stdout: "journal_throttle_max_multiple": "0.000000", 2024-04-14T15:11:32.778 INFO:teuthology.orchestra.run.smithi090.stdout: "journal_write_header_frequency": "0", 2024-04-14T15:11:32.778 INFO:teuthology.orchestra.run.smithi090.stdout: "journal_zero_on_create": "false", 2024-04-14T15:11:32.778 INFO:teuthology.orchestra.run.smithi090.stdout: "journaler_prefetch_periods": "10", 2024-04-14T15:11:32.779 INFO:teuthology.orchestra.run.smithi090.stdout: "journaler_prezero_periods": "5", 2024-04-14T15:11:32.779 INFO:teuthology.orchestra.run.smithi090.stdout: "journaler_write_head_interval": "15", 2024-04-14T15:11:32.779 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "", 2024-04-14T15:11:32.779 INFO:teuthology.orchestra.run.smithi090.stdout: "keyfile": "", 2024-04-14T15:11:32.779 INFO:teuthology.orchestra.run.smithi090.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-04-14T15:11:32.779 INFO:teuthology.orchestra.run.smithi090.stdout: "kstore_backend": "rocksdb", 2024-04-14T15:11:32.779 INFO:teuthology.orchestra.run.smithi090.stdout: "kstore_default_stripe_size": "65536", 2024-04-14T15:11:32.779 INFO:teuthology.orchestra.run.smithi090.stdout: "kstore_fsck_on_mount": "false", 2024-04-14T15:11:32.779 INFO:teuthology.orchestra.run.smithi090.stdout: "kstore_fsck_on_mount_deep": "true", 2024-04-14T15:11:32.779 INFO:teuthology.orchestra.run.smithi090.stdout: "kstore_max_bytes": "67108864", 2024-04-14T15:11:32.779 INFO:teuthology.orchestra.run.smithi090.stdout: "kstore_max_ops": "512", 2024-04-14T15:11:32.779 INFO:teuthology.orchestra.run.smithi090.stdout: "kstore_nid_prealloc": "1024", 2024-04-14T15:11:32.779 INFO:teuthology.orchestra.run.smithi090.stdout: "kstore_onode_map_size": "1024", 2024-04-14T15:11:32.779 INFO:teuthology.orchestra.run.smithi090.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-04-14T15:11:32.779 INFO:teuthology.orchestra.run.smithi090.stdout: "kstore_sync_submit_transaction": "false", 2024-04-14T15:11:32.780 INFO:teuthology.orchestra.run.smithi090.stdout: "kstore_sync_transaction": "false", 2024-04-14T15:11:32.780 INFO:teuthology.orchestra.run.smithi090.stdout: "librados_thread_count": "2", 2024-04-14T15:11:32.780 INFO:teuthology.orchestra.run.smithi090.stdout: "lockdep": "false", 2024-04-14T15:11:32.780 INFO:teuthology.orchestra.run.smithi090.stdout: "lockdep_force_backtrace": "false", 2024-04-14T15:11:32.780 INFO:teuthology.orchestra.run.smithi090.stdout: "log_coarse_timestamps": "true", 2024-04-14T15:11:32.780 INFO:teuthology.orchestra.run.smithi090.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-04-14T15:11:32.780 INFO:teuthology.orchestra.run.smithi090.stdout: "log_flush_on_exit": "false", 2024-04-14T15:11:32.780 INFO:teuthology.orchestra.run.smithi090.stdout: "log_graylog_host": "127.0.0.1", 2024-04-14T15:11:32.780 INFO:teuthology.orchestra.run.smithi090.stdout: "log_graylog_port": "12201", 2024-04-14T15:11:32.780 INFO:teuthology.orchestra.run.smithi090.stdout: "log_max_new": "1000", 2024-04-14T15:11:32.780 INFO:teuthology.orchestra.run.smithi090.stdout: "log_max_recent": "10000", 2024-04-14T15:11:32.780 INFO:teuthology.orchestra.run.smithi090.stdout: "log_stderr_prefix": "", 2024-04-14T15:11:32.780 INFO:teuthology.orchestra.run.smithi090.stdout: "log_stop_at_utilization": "0.970000", 2024-04-14T15:11:32.781 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": "true", 2024-04-14T15:11:32.781 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_graylog": "false", 2024-04-14T15:11:32.781 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_journald": "false", 2024-04-14T15:11:32.781 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_stderr": "false", 2024-04-14T15:11:32.781 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_syslog": "false", 2024-04-14T15:11:32.781 INFO:teuthology.orchestra.run.smithi090.stdout: "max_rotating_auth_attempts": "10", 2024-04-14T15:11:32.781 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-04-14T15:11:32.781 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_action_on_write_error": "1", 2024-04-14T15:11:32.781 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_alternate_name_max": "8192", 2024-04-14T15:11:32.781 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_asio_thread_count": "2", 2024-04-14T15:11:32.781 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_export_pin": "true", 2024-04-14T15:11:32.781 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_fragment_dirs": "true", 2024-04-14T15:11:32.781 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-04-14T15:11:32.781 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_fragment_interval": "5", 2024-04-14T15:11:32.782 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_fragment_size_max": "100000", 2024-04-14T15:11:32.782 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_idle_threshold": "0.000000", 2024-04-14T15:11:32.782 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_interval": "10", 2024-04-14T15:11:32.782 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_max": "-1", 2024-04-14T15:11:32.782 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_max_until": "-1", 2024-04-14T15:11:32.782 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_merge_size": "50", 2024-04-14T15:11:32.782 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_midchunk": "0.300000", 2024-04-14T15:11:32.782 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_min_rebalance": "0.100000", 2024-04-14T15:11:32.782 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_min_start": "0.200000", 2024-04-14T15:11:32.782 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_minchunk": "0.001000", 2024-04-14T15:11:32.782 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_mode": "0", 2024-04-14T15:11:32.782 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_need_max": "1.200000", 2024-04-14T15:11:32.782 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_need_min": "0.800000", 2024-04-14T15:11:32.782 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_overload_epochs": "2", 2024-04-14T15:11:32.783 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-04-14T15:11:32.783 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_sample_interval": "3.000000", 2024-04-14T15:11:32.783 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_split_bits": "3", 2024-04-14T15:11:32.783 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_split_rd": "25000.000000", 2024-04-14T15:11:32.783 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_split_size": "10000", 2024-04-14T15:11:32.783 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_split_wr": "10000.000000", 2024-04-14T15:11:32.783 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_target_decay": "10.000000", 2024-04-14T15:11:32.783 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-04-14T15:11:32.783 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_beacon_grace": "15.000000", 2024-04-14T15:11:32.783 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_beacon_interval": "4.000000", 2024-04-14T15:11:32.783 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_beacon_mon_down_grace": "60", 2024-04-14T15:11:32.783 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_cache_memory_limit": "4294967296", 2024-04-14T15:11:32.784 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_cache_mid": "0.700000", 2024-04-14T15:11:32.784 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-04-14T15:11:32.784 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_cache_quiesce_delay": "0", 2024-04-14T15:11:32.784 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_cache_quiesce_sleep": "200", 2024-04-14T15:11:32.784 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_cache_quiesce_splitauth": "true", 2024-04-14T15:11:32.784 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_cache_quiesce_threshold": "524288", 2024-04-14T15:11:32.784 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_cache_release_free_interval": "10", 2024-04-14T15:11:32.784 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_cache_reservation": "0.050000", 2024-04-14T15:11:32.784 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-04-14T15:11:32.784 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_cache_trim_interval": "1", 2024-04-14T15:11:32.784 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_cache_trim_threshold": "262144", 2024-04-14T15:11:32.784 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-04-14T15:11:32.784 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-04-14T15:11:32.784 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_client_delegate_inos_pct": "50", 2024-04-14T15:11:32.785 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_client_prealloc_inos": "1000", 2024-04-14T15:11:32.785 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-04-14T15:11:32.785 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_connect_bootstrapping": "false", 2024-04-14T15:11:32.785 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_damage_table_max_entries": "10000", 2024-04-14T15:11:32.785 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-04-14T15:11:32.785 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_debug_auth_pins": "false", 2024-04-14T15:11:32.785 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_debug_frag": "false", 2024-04-14T15:11:32.785 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_debug_scatterstat": "false", 2024-04-14T15:11:32.785 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_debug_subtrees": "false", 2024-04-14T15:11:32.785 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_decay_halflife": "5.000000", 2024-04-14T15:11:32.785 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_default_dir_hash": "2", 2024-04-14T15:11:32.785 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_defer_session_stale": "true", 2024-04-14T15:11:32.786 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_deny_all_reconnect": "false", 2024-04-14T15:11:32.786 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_dir_keys_per_op": "16384", 2024-04-14T15:11:32.786 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_dir_max_commit_size": "10", 2024-04-14T15:11:32.786 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_dir_max_entries": "0", 2024-04-14T15:11:32.786 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_dir_prefetch": "true", 2024-04-14T15:11:32.786 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_dirstat_min_interval": "1.000000", 2024-04-14T15:11:32.786 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_dump_cache_after_rejoin": "false", 2024-04-14T15:11:32.786 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_dump_cache_on_map": "false", 2024-04-14T15:11:32.786 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_dump_cache_threshold_file": "0", 2024-04-14T15:11:32.786 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-04-14T15:11:32.786 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_early_reply": "true", 2024-04-14T15:11:32.787 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_enable_op_tracker": "true", 2024-04-14T15:11:32.787 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_enforce_unique_name": "true", 2024-04-14T15:11:32.787 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_export_ephemeral_distributed": "true", 2024-04-14T15:11:32.787 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-04-14T15:11:32.787 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_export_ephemeral_random": "true", 2024-04-14T15:11:32.787 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-04-14T15:11:32.787 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-04-14T15:11:32.787 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_forward_all_requests_to_auth": "false", 2024-04-14T15:11:32.787 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-04-14T15:11:32.787 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-04-14T15:11:32.787 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-04-14T15:11:32.787 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-04-14T15:11:32.787 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_health_cache_threshold": "1.500000", 2024-04-14T15:11:32.788 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_health_summarize_threshold": "10", 2024-04-14T15:11:32.788 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_heartbeat_grace": "15.000000", 2024-04-14T15:11:32.788 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_heartbeat_reset_grace": "1000", 2024-04-14T15:11:32.788 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_inject_health_dummy": "false", 2024-04-14T15:11:32.788 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-04-14T15:11:32.788 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_inject_migrator_session_race": "false", 2024-04-14T15:11:32.788 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-04-14T15:11:32.788 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-04-14T15:11:32.788 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-04-14T15:11:32.788 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_join_fs": "", 2024-04-14T15:11:32.788 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_journal_format": "1", 2024-04-14T15:11:32.788 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_kill_create_at": "0", 2024-04-14T15:11:32.788 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_kill_export_at": "0", 2024-04-14T15:11:32.789 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_kill_import_at": "0", 2024-04-14T15:11:32.789 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_kill_journal_at": "0", 2024-04-14T15:11:32.789 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_kill_journal_expire_at": "0", 2024-04-14T15:11:32.789 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_kill_journal_replay_at": "0", 2024-04-14T15:11:32.789 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_kill_link_at": "0", 2024-04-14T15:11:32.789 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_kill_mdstable_at": "0", 2024-04-14T15:11:32.789 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_kill_openc_at": "0", 2024-04-14T15:11:32.789 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_kill_rename_at": "0", 2024-04-14T15:11:32.789 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_kill_shutdown_at": "0", 2024-04-14T15:11:32.789 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-04-14T15:11:32.789 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_log_event_large_threshold": "524288", 2024-04-14T15:11:32.789 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_log_events_per_segment": "1024", 2024-04-14T15:11:32.789 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_log_major_segment_event_ratio": "12", 2024-04-14T15:11:32.790 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_log_max_events": "-1", 2024-04-14T15:11:32.790 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_log_max_segments": "128", 2024-04-14T15:11:32.790 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_log_pause": "false", 2024-04-14T15:11:32.790 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_log_segment_size": "0", 2024-04-14T15:11:32.790 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_log_skip_corrupt_events": "false", 2024-04-14T15:11:32.790 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_log_skip_unbounded_events": "false", 2024-04-14T15:11:32.790 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-04-14T15:11:32.790 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_log_trim_threshold": "128", 2024-04-14T15:11:32.790 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-04-14T15:11:32.790 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_log_warn_factor": "2.000000", 2024-04-14T15:11:32.790 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_max_caps_per_client": "1048576", 2024-04-14T15:11:32.790 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_max_completed_flushes": "100000", 2024-04-14T15:11:32.791 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_max_completed_requests": "100000", 2024-04-14T15:11:32.791 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_max_export_size": "20971520", 2024-04-14T15:11:32.791 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_max_file_recover": "32", 2024-04-14T15:11:32.791 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_max_purge_files": "64", 2024-04-14T15:11:32.791 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_max_purge_ops": "8192", 2024-04-14T15:11:32.791 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-04-14T15:11:32.791 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-04-14T15:11:32.791 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_max_snaps_per_dir": "100", 2024-04-14T15:11:32.791 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_metrics_update_interval": "2", 2024-04-14T15:11:32.791 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_min_caps_per_client": "100", 2024-04-14T15:11:32.791 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_min_caps_working_set": "10000", 2024-04-14T15:11:32.791 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-04-14T15:11:32.792 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_numa_node": "-1", 2024-04-14T15:11:32.792 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-04-14T15:11:32.792 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_op_complaint_time": "30.000000", 2024-04-14T15:11:32.792 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_op_history_duration": "600", 2024-04-14T15:11:32.792 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_op_history_size": "20", 2024-04-14T15:11:32.792 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_op_history_slow_op_size": "20", 2024-04-14T15:11:32.792 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_op_history_slow_op_threshold": "10", 2024-04-14T15:11:32.792 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_op_log_threshold": "5", 2024-04-14T15:11:32.792 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_ping_grace": "15", 2024-04-14T15:11:32.792 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_ping_interval": "5", 2024-04-14T15:11:32.792 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-04-14T15:11:32.792 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-04-14T15:11:32.792 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_recall_max_caps": "30000", 2024-04-14T15:11:32.792 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-04-14T15:11:32.793 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_recall_max_decay_threshold": "131072", 2024-04-14T15:11:32.793 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-04-14T15:11:32.793 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_recall_warning_threshold": "262144", 2024-04-14T15:11:32.793 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_reconnect_timeout": "45.000000", 2024-04-14T15:11:32.793 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_replay_interval": "1.000000", 2024-04-14T15:11:32.793 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-04-14T15:11:32.793 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-04-14T15:11:32.793 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_root_ino_gid": "0", 2024-04-14T15:11:32.793 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_root_ino_uid": "0", 2024-04-14T15:11:32.793 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-04-14T15:11:32.793 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_session_blocklist_on_evict": "true", 2024-04-14T15:11:32.793 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_session_blocklist_on_timeout": "true", 2024-04-14T15:11:32.793 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-04-14T15:11:32.793 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-04-14T15:11:32.794 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-04-14T15:11:32.794 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-04-14T15:11:32.794 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-04-14T15:11:32.794 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_session_metadata_threshold": "16777216", 2024-04-14T15:11:32.794 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-04-14T15:11:32.794 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_shutdown_check": "0", 2024-04-14T15:11:32.794 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_skip_ino": "0", 2024-04-14T15:11:32.794 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_sleep_rank_change": "0.000000", 2024-04-14T15:11:32.794 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_snap_max_uid": "4294967294", 2024-04-14T15:11:32.794 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_snap_min_uid": "0", 2024-04-14T15:11:32.794 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_snap_rstat": "false", 2024-04-14T15:11:32.794 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_standby_replay_damaged": "false", 2024-04-14T15:11:32.794 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_symlink_recovery": "true", 2024-04-14T15:11:32.794 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_task_status_update_interval": "2.000000", 2024-04-14T15:11:32.795 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_thrash_exports": "0", 2024-04-14T15:11:32.795 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_thrash_fragments": "0", 2024-04-14T15:11:32.795 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_tick_interval": "5.000000", 2024-04-14T15:11:32.795 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_valgrind_exit": "false", 2024-04-14T15:11:32.795 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_verify_backtrace": "1", 2024-04-14T15:11:32.795 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_verify_scatter": "false", 2024-04-14T15:11:32.795 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_wipe_ino_prealloc": "false", 2024-04-14T15:11:32.795 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_wipe_sessions": "false", 2024-04-14T15:11:32.795 INFO:teuthology.orchestra.run.smithi090.stdout: "mempool_debug": "false", 2024-04-14T15:11:32.795 INFO:teuthology.orchestra.run.smithi090.stdout: "memstore_debug_omit_block_device_write": "false", 2024-04-14T15:11:32.795 INFO:teuthology.orchestra.run.smithi090.stdout: "memstore_device_bytes": "1073741824", 2024-04-14T15:11:32.795 INFO:teuthology.orchestra.run.smithi090.stdout: "memstore_page_set": "false", 2024-04-14T15:11:32.795 INFO:teuthology.orchestra.run.smithi090.stdout: "memstore_page_size": "65536", 2024-04-14T15:11:32.795 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_client_bytes": "134217728", 2024-04-14T15:11:32.795 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_client_messages": "512", 2024-04-14T15:11:32.795 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-04-14T15:11:32.796 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_connect_retry_interval": "1.000000", 2024-04-14T15:11:32.796 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-04-14T15:11:32.796 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-04-14T15:11:32.796 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_disabled_modules": "", 2024-04-14T15:11:32.796 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-04-14T15:11:32.796 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_max_pg_creating": "1024", 2024-04-14T15:11:32.796 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_max_pg_num_change": "128", 2024-04-14T15:11:32.796 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_mds_bytes": "134217728", 2024-04-14T15:11:32.796 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_mds_messages": "128", 2024-04-14T15:11:32.796 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-04-14T15:11:32.796 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_mon_bytes": "134217728", 2024-04-14T15:11:32.796 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_mon_messages": "128", 2024-04-14T15:11:32.796 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_osd_bytes": "536870912", 2024-04-14T15:11:32.797 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_osd_messages": "8192", 2024-04-14T15:11:32.797 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_pool": "true", 2024-04-14T15:11:32.797 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_service_beacon_grace": "60.000000", 2024-04-14T15:11:32.797 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_standby_modules": "true", 2024-04-14T15:11:32.797 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_stats_period": "5", 2024-04-14T15:11:32.797 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_stats_threshold": "5", 2024-04-14T15:11:32.797 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_tick_period": "2", 2024-04-14T15:11:32.797 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-04-14T15:11:32.797 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_accept_timeout_factor": "2.000000", 2024-04-14T15:11:32.797 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_allow_pool_delete": "true", 2024-04-14T15:11:32.797 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_allow_pool_size_one": "false", 2024-04-14T15:11:32.797 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_auth_validate_all_caps": "true", 2024-04-14T15:11:32.798 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-04-14T15:11:32.798 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-04-14T15:11:32.798 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_client_bytes": "104857600", 2024-04-14T15:11:32.798 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_client_directed_command_retry": "2", 2024-04-14T15:11:32.798 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_client_hunt_interval": "3.000000", 2024-04-14T15:11:32.798 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-04-14T15:11:32.798 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-04-14T15:11:32.798 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-04-14T15:11:32.798 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_client_hunt_parallel": "3", 2024-04-14T15:11:32.798 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_client_log_interval": "1.000000", 2024-04-14T15:11:32.798 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-04-14T15:11:32.798 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_client_ping_interval": "10.000000", 2024-04-14T15:11:32.798 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_client_ping_timeout": "30.000000", 2024-04-14T15:11:32.799 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_client_target_rank": "-1", 2024-04-14T15:11:32.799 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_clock_drift_allowed": "1.000000", 2024-04-14T15:11:32.799 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-04-14T15:11:32.799 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-04-14T15:11:32.799 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_cluster_log_file_level": "debug", 2024-04-14T15:11:32.799 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_cluster_log_to_file": "true", 2024-04-14T15:11:32.799 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_cluster_log_to_graylog": "false", 2024-04-14T15:11:32.799 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-04-14T15:11:32.799 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-04-14T15:11:32.799 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_cluster_log_to_journald": "true", 2024-04-14T15:11:32.799 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_cluster_log_to_stderr": "false", 2024-04-14T15:11:32.799 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-04-14T15:11:32.799 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-04-14T15:11:32.800 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-04-14T15:11:32.800 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_compact_on_bootstrap": "false", 2024-04-14T15:11:32.800 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_compact_on_start": "false", 2024-04-14T15:11:32.800 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_compact_on_trim": "true", 2024-04-14T15:11:32.800 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_con_tracker_persist_interval": "10", 2024-04-14T15:11:32.800 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_con_tracker_score_halflife": "43200", 2024-04-14T15:11:32.800 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_config_key_max_entry_size": "65536", 2024-04-14T15:11:32.800 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_cpu_threads": "4", 2024-04-14T15:11:32.800 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_crush_min_required_version": "hammer", 2024-04-14T15:11:32.800 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_daemon_bytes": "419430400", 2024-04-14T15:11:32.800 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-04-14T15:11:32.800 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_data_avail_crit": "5", 2024-04-14T15:11:32.800 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_data_avail_warn": "5", 2024-04-14T15:11:32.800 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_data_size_warn": "16106127360", 2024-04-14T15:11:32.801 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_debug_block_osdmap_trim": "false", 2024-04-14T15:11:32.801 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-04-14T15:11:32.801 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_debug_dump_json": "false", 2024-04-14T15:11:32.801 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-04-14T15:11:32.801 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_debug_dump_transactions": "false", 2024-04-14T15:11:32.801 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_debug_extra_checks": "false", 2024-04-14T15:11:32.801 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-04-14T15:11:32.801 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-04-14T15:11:32.801 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_debug_no_require_reef": "false", 2024-04-14T15:11:32.801 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_debug_no_require_squid": "false", 2024-04-14T15:11:32.801 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-04-14T15:11:32.801 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_delta_reset_interval": "10.000000", 2024-04-14T15:11:32.801 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_dns_srv_name": "ceph-mon", 2024-04-14T15:11:32.801 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_down_mkfs_grace": "60", 2024-04-14T15:11:32.801 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_down_uptime_grace": "60", 2024-04-14T15:11:32.801 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_election_default_strategy": "1", 2024-04-14T15:11:32.802 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_election_timeout": "5.000000", 2024-04-14T15:11:32.802 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-04-14T15:11:32.802 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_elector_ping_divisor": "2", 2024-04-14T15:11:32.802 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_elector_ping_timeout": "2.000000", 2024-04-14T15:11:32.802 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_enable_op_tracker": "true", 2024-04-14T15:11:32.802 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_fake_pool_delete": "false", 2024-04-14T15:11:32.802 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_force_quorum_join": "false", 2024-04-14T15:11:32.802 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_globalid_prealloc": "10000", 2024-04-14T15:11:32.802 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_health_detail_to_clog": "true", 2024-04-14T15:11:32.802 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_health_log_update_period": "5", 2024-04-14T15:11:32.802 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_health_max_detail": "50", 2024-04-14T15:11:32.802 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_health_to_clog": "true", 2024-04-14T15:11:32.802 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_health_to_clog_interval": "600", 2024-04-14T15:11:32.802 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-04-14T15:11:32.803 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_host": "172.21.15.90:6789/0", 2024-04-14T15:11:32.803 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_host_override": "", 2024-04-14T15:11:32.803 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_initial_members": "", 2024-04-14T15:11:32.803 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-04-14T15:11:32.803 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-04-14T15:11:32.803 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-04-14T15:11:32.803 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-04-14T15:11:32.803 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_keyvaluedb": "rocksdb", 2024-04-14T15:11:32.803 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_lease": "5.000000", 2024-04-14T15:11:32.803 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-04-14T15:11:32.803 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-04-14T15:11:32.803 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_log_full_interval": "50", 2024-04-14T15:11:32.804 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_log_max": "10000", 2024-04-14T15:11:32.804 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_log_max_summary": "50", 2024-04-14T15:11:32.804 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_max_log_entries_per_event": "4096", 2024-04-14T15:11:32.804 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_max_log_epochs": "500", 2024-04-14T15:11:32.804 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_max_mdsmap_epochs": "500", 2024-04-14T15:11:32.804 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_max_mgrmap_epochs": "500", 2024-04-14T15:11:32.804 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_max_osd": "10000", 2024-04-14T15:11:32.804 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_max_pg_per_osd": "10000", 2024-04-14T15:11:32.804 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_max_pool_pg_num": "65536", 2024-04-14T15:11:32.804 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-04-14T15:11:32.804 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-04-14T15:11:32.804 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_mds_force_trim_to": "0", 2024-04-14T15:11:32.804 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_mds_skip_sanity": "false", 2024-04-14T15:11:32.804 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_memory_autotune": "true", 2024-04-14T15:11:32.804 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_memory_target": "2147483648", 2024-04-14T15:11:32.805 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_mgr_beacon_grace": "30", 2024-04-14T15:11:32.805 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-04-14T15:11:32.805 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_mgr_digest_period": "5", 2024-04-14T15:11:32.805 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_mgr_inactive_grace": "60", 2024-04-14T15:11:32.805 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_mgr_mkfs_grace": "240", 2024-04-14T15:11:32.805 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-04-14T15:11:32.805 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_min_osdmap_epochs": "50", 2024-04-14T15:11:32.805 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_op_complaint_time": "30", 2024-04-14T15:11:32.805 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_op_history_duration": "600", 2024-04-14T15:11:32.805 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_op_history_size": "20", 2024-04-14T15:11:32.805 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_op_history_slow_op_size": "20", 2024-04-14T15:11:32.805 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_op_history_slow_op_threshold": "10", 2024-04-14T15:11:32.805 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_op_log_threshold": "5", 2024-04-14T15:11:32.805 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-04-14T15:11:32.805 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-04-14T15:11:32.805 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-04-14T15:11:32.805 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_auto_mark_in": "false", 2024-04-14T15:11:32.806 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_auto_mark_new_in": "true", 2024-04-14T15:11:32.806 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-04-14T15:11:32.806 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-04-14T15:11:32.806 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_cache_size": "500", 2024-04-14T15:11:32.806 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_cache_size_min": "134217728", 2024-04-14T15:11:32.806 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_crush_smoke_test": "true", 2024-04-14T15:11:32.806 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_destroyed_out_interval": "600", 2024-04-14T15:11:32.806 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_down_out_interval": "600", 2024-04-14T15:11:32.806 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-04-14T15:11:32.806 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-04-14T15:11:32.806 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_force_trim_to": "0", 2024-04-14T15:11:32.806 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_full_ratio": "0.950000", 2024-04-14T15:11:32.806 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-04-14T15:11:32.806 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_laggy_halflife": "3600", 2024-04-14T15:11:32.807 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_laggy_max_interval": "300", 2024-04-14T15:11:32.807 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_laggy_weight": "0.300000", 2024-04-14T15:11:32.807 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-04-14T15:11:32.807 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_max_initial_pgs": "1024", 2024-04-14T15:11:32.807 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_min_down_reporters": "2", 2024-04-14T15:11:32.807 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-04-14T15:11:32.807 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-04-14T15:11:32.807 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-04-14T15:11:32.807 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_prime_pg_temp": "true", 2024-04-14T15:11:32.807 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-04-14T15:11:32.807 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-04-14T15:11:32.807 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_report_timeout": "900", 2024-04-14T15:11:32.807 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-04-14T15:11:32.807 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-04-14T15:11:32.807 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_warn_num_repaired": "10", 2024-04-14T15:11:32.807 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_warn_op_age": "32.000000", 2024-04-14T15:11:32.808 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-04-14T15:11:32.808 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osdmap_full_prune_interval": "2", 2024-04-14T15:11:32.808 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osdmap_full_prune_min": "15", 2024-04-14T15:11:32.808 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-04-14T15:11:32.808 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-04-14T15:11:32.808 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_pg_stuck_threshold": "60", 2024-04-14T15:11:32.808 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-04-14T15:11:32.808 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_pg_warn_min_objects": "10000", 2024-04-14T15:11:32.808 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_pg_warn_min_per_osd": "0", 2024-04-14T15:11:32.808 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-04-14T15:11:32.808 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_pool_quota_crit_threshold": "0", 2024-04-14T15:11:32.808 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_pool_quota_warn_threshold": "0", 2024-04-14T15:11:32.808 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_probe_timeout": "2.000000", 2024-04-14T15:11:32.808 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_reweight_max_change": "0.050000", 2024-04-14T15:11:32.808 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_reweight_max_osds": "4", 2024-04-14T15:11:32.808 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-04-14T15:11:32.809 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-04-14T15:11:32.809 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-04-14T15:11:32.809 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-04-14T15:11:32.809 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-04-14T15:11:32.809 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_scrub_interval": "86400", 2024-04-14T15:11:32.809 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_scrub_max_keys": "100", 2024-04-14T15:11:32.809 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_scrub_timeout": "300", 2024-04-14T15:11:32.809 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_session_timeout": "300", 2024-04-14T15:11:32.809 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_smart_report_timeout": "5", 2024-04-14T15:11:32.809 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_stat_smooth_intervals": "6", 2024-04-14T15:11:32.809 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-04-14T15:11:32.809 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_stretch_pool_min_size": "2", 2024-04-14T15:11:32.811 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_stretch_pool_size": "4", 2024-04-14T15:11:32.811 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-04-14T15:11:32.811 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_subscribe_interval": "86400.000000", 2024-04-14T15:11:32.811 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_sync_debug": "false", 2024-04-14T15:11:32.811 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_sync_max_payload_keys": "2000", 2024-04-14T15:11:32.811 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_sync_max_payload_size": "1048576", 2024-04-14T15:11:32.811 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_sync_provider_kill_at": "0", 2024-04-14T15:11:32.811 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_sync_requester_kill_at": "0", 2024-04-14T15:11:32.811 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_sync_timeout": "60.000000", 2024-04-14T15:11:32.811 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_target_pg_per_osd": "100", 2024-04-14T15:11:32.812 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_tick_interval": "5", 2024-04-14T15:11:32.812 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_timecheck_interval": "300.000000", 2024-04-14T15:11:32.812 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-04-14T15:11:32.812 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_use_min_delay_socket": "false", 2024-04-14T15:11:32.812 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_warn_older_version_delay": "604800", 2024-04-14T15:11:32.812 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-04-14T15:11:32.812 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-04-14T15:11:32.812 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-04-14T15:11:32.812 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_warn_on_filestore_osds": "true", 2024-04-14T15:11:32.812 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-04-14T15:11:32.812 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-04-14T15:11:32.812 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-04-14T15:11:32.812 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_warn_on_misplaced": "false", 2024-04-14T15:11:32.812 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-04-14T15:11:32.813 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_warn_on_older_version": "true", 2024-04-14T15:11:32.813 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-04-14T15:11:32.813 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_warn_on_pool_no_app": "false", 2024-04-14T15:11:32.813 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-04-14T15:11:32.813 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-04-14T15:11:32.813 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-04-14T15:11:32.813 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-04-14T15:11:32.813 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_warn_on_too_few_osds": "false", 2024-04-14T15:11:32.813 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-04-14T15:11:32.813 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-04-14T15:11:32.813 INFO:teuthology.orchestra.run.smithi090.stdout: "monmap": "", 2024-04-14T15:11:32.813 INFO:teuthology.orchestra.run.smithi090.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-04-14T15:11:32.813 INFO:teuthology.orchestra.run.smithi090.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-04-14T15:11:32.814 INFO:teuthology.orchestra.run.smithi090.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-04-14T15:11:32.814 INFO:teuthology.orchestra.run.smithi090.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-04-14T15:11:32.814 INFO:teuthology.orchestra.run.smithi090.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-04-14T15:11:32.814 INFO:teuthology.orchestra.run.smithi090.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-04-14T15:11:32.814 INFO:teuthology.orchestra.run.smithi090.stdout: "motr_tracing_enabled": "false", 2024-04-14T15:11:32.814 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_async_op_threads": "3", 2024-04-14T15:11:32.814 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_async_rdma_buffer_size": "131072", 2024-04-14T15:11:32.814 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_async_rdma_cm": "false", 2024-04-14T15:11:32.814 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_async_rdma_device_name": "", 2024-04-14T15:11:32.814 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_async_rdma_dscp": "96", 2024-04-14T15:11:32.814 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-04-14T15:11:32.814 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_async_rdma_gid_idx": "0", 2024-04-14T15:11:32.814 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_async_rdma_local_gid": "", 2024-04-14T15:11:32.814 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_async_rdma_polling_us": "1000", 2024-04-14T15:11:32.815 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_async_rdma_port_num": "1", 2024-04-14T15:11:32.815 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-04-14T15:11:32.815 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-04-14T15:11:32.815 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_async_rdma_roce_ver": "1", 2024-04-14T15:11:32.815 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_async_rdma_send_buffers": "1024", 2024-04-14T15:11:32.815 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_async_rdma_sl": "3", 2024-04-14T15:11:32.815 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_async_rdma_support_srq": "true", 2024-04-14T15:11:32.815 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_async_rdma_type": "ib", 2024-04-14T15:11:32.815 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_async_reap_threshold": "5", 2024-04-14T15:11:32.815 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_bind_before_connect": "false", 2024-04-14T15:11:32.815 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_bind_ipv4": "true", 2024-04-14T15:11:32.815 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_bind_ipv6": "false", 2024-04-14T15:11:32.815 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_bind_msgr1": "true", 2024-04-14T15:11:32.815 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_bind_msgr2": "false", 2024-04-14T15:11:32.816 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_bind_port_max": "7568", 2024-04-14T15:11:32.816 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_bind_port_min": "6800", 2024-04-14T15:11:32.816 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_bind_prefer_ipv4": "false", 2024-04-14T15:11:32.816 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_bind_retry_count": "3", 2024-04-14T15:11:32.816 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_bind_retry_delay": "5", 2024-04-14T15:11:32.816 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_blackhole_client": "false", 2024-04-14T15:11:32.816 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_blackhole_mds": "false", 2024-04-14T15:11:32.816 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_blackhole_mgr": "false", 2024-04-14T15:11:32.816 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_blackhole_mon": "false", 2024-04-14T15:11:32.816 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_blackhole_osd": "false", 2024-04-14T15:11:32.816 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_client_mode": "crc secure", 2024-04-14T15:11:32.816 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_cluster_mode": "crc secure", 2024-04-14T15:11:32.816 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_cluster_type": "", 2024-04-14T15:11:32.817 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_compress_secure": "false", 2024-04-14T15:11:32.817 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_connection_idle_timeout": "900", 2024-04-14T15:11:32.817 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_connection_ready_timeout": "10", 2024-04-14T15:11:32.817 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_crc_data": "true", 2024-04-14T15:11:32.817 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_crc_header": "true", 2024-04-14T15:11:32.817 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_die_on_bad_msg": "false", 2024-04-14T15:11:32.817 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_die_on_bug": "true", 2024-04-14T15:11:32.817 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_die_on_old_message": "true", 2024-04-14T15:11:32.817 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_die_on_skipped_message": "false", 2024-04-14T15:11:32.817 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_die_on_unhandled_msg": "false", 2024-04-14T15:11:32.817 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-04-14T15:11:32.817 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_dpdk_coremask": "0xF", 2024-04-14T15:11:32.817 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-04-14T15:11:32.818 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_dpdk_devs_allowlist": "", 2024-04-14T15:11:32.818 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_dpdk_enable_tso": "true", 2024-04-14T15:11:32.818 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-04-14T15:11:32.818 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-04-14T15:11:32.818 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_dpdk_hugepages": "", 2024-04-14T15:11:32.818 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_dpdk_hw_flow_control": "true", 2024-04-14T15:11:32.818 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-04-14T15:11:32.818 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_dpdk_lro": "true", 2024-04-14T15:11:32.818 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_dpdk_memory_channel": "4", 2024-04-14T15:11:32.818 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-04-14T15:11:32.818 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_dpdk_pmd": "", 2024-04-14T15:11:32.818 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_dpdk_port_id": "0", 2024-04-14T15:11:32.819 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-04-14T15:11:32.819 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_dump_corrupt_message_level": "1", 2024-04-14T15:11:32.819 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_dump_on_send": "false", 2024-04-14T15:11:32.819 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_initial_backoff": "0.200000", 2024-04-14T15:11:32.819 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_inject_delay_max": "1.000000", 2024-04-14T15:11:32.819 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_inject_delay_probability": "0.000000", 2024-04-14T15:11:32.819 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_inject_delay_type": "", 2024-04-14T15:11:32.819 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_inject_internal_delays": "0.000000", 2024-04-14T15:11:32.819 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_inject_network_congestion": "0", 2024-04-14T15:11:32.819 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_inject_socket_failures": "0", 2024-04-14T15:11:32.819 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_learn_addr_from_peer": "true", 2024-04-14T15:11:32.819 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_max_accept_failures": "4", 2024-04-14T15:11:32.819 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_max_backoff": "15.000000", 2024-04-14T15:11:32.820 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_mon_client_mode": "secure crc", 2024-04-14T15:11:32.820 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_mon_cluster_mode": "secure crc", 2024-04-14T15:11:32.820 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_mon_service_mode": "secure crc", 2024-04-14T15:11:32.820 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_osd_compress_min_size": "1024", 2024-04-14T15:11:32.820 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_osd_compress_mode": "none", 2024-04-14T15:11:32.820 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_osd_compression_algorithm": "snappy", 2024-04-14T15:11:32.820 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-04-14T15:11:32.820 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_pq_min_cost": "65536", 2024-04-14T15:11:32.820 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_public_type": "", 2024-04-14T15:11:32.820 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_service_mode": "crc secure", 2024-04-14T15:11:32.820 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_tcp_listen_backlog": "512", 2024-04-14T15:11:32.820 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_tcp_nodelay": "true", 2024-04-14T15:11:32.821 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-04-14T15:11:32.821 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_tcp_rcvbuf": "0", 2024-04-14T15:11:32.821 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_type": "async+posix", 2024-04-14T15:11:32.821 INFO:teuthology.orchestra.run.smithi090.stdout: "no_config_file": "false", 2024-04-14T15:11:32.821 INFO:teuthology.orchestra.run.smithi090.stdout: "objecter_completion_locks_per_session": "32", 2024-04-14T15:11:32.821 INFO:teuthology.orchestra.run.smithi090.stdout: "objecter_debug_inject_relock_delay": "false", 2024-04-14T15:11:32.821 INFO:teuthology.orchestra.run.smithi090.stdout: "objecter_inflight_op_bytes": "104857600", 2024-04-14T15:11:32.821 INFO:teuthology.orchestra.run.smithi090.stdout: "objecter_inflight_ops": "1024", 2024-04-14T15:11:32.821 INFO:teuthology.orchestra.run.smithi090.stdout: "objecter_inject_no_watch_ping": "false", 2024-04-14T15:11:32.821 INFO:teuthology.orchestra.run.smithi090.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-04-14T15:11:32.821 INFO:teuthology.orchestra.run.smithi090.stdout: "objecter_tick_interval": "5.000000", 2024-04-14T15:11:32.821 INFO:teuthology.orchestra.run.smithi090.stdout: "objecter_timeout": "10.000000", 2024-04-14T15:11:32.821 INFO:teuthology.orchestra.run.smithi090.stdout: "objectstore_blackhole": "false", 2024-04-14T15:11:32.822 INFO:teuthology.orchestra.run.smithi090.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-04-14T15:11:32.822 INFO:teuthology.orchestra.run.smithi090.stdout: "openssl_engine_opts": "", 2024-04-14T15:11:32.822 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_agent_delay_time": "5.000000", 2024-04-14T15:11:32.822 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_agent_hist_halflife": "1000", 2024-04-14T15:11:32.822 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_agent_max_low_ops": "2", 2024-04-14T15:11:32.822 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_agent_max_ops": "4", 2024-04-14T15:11:32.822 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-04-14T15:11:32.822 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_agent_quantize_effort": "0.100000", 2024-04-14T15:11:32.822 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_agent_slop": "0.020000", 2024-04-14T15:11:32.822 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-04-14T15:11:32.822 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_allow_recovery_below_min_size": "true", 2024-04-14T15:11:32.822 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_asio_thread_count": "2", 2024-04-14T15:11:32.822 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_async_recovery_min_cost": "100", 2024-04-14T15:11:32.823 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_auto_mark_unfound_lost": "false", 2024-04-14T15:11:32.823 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_backfill_retry_interval": "30.000000", 2024-04-14T15:11:32.823 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_backfill_scan_max": "512", 2024-04-14T15:11:32.823 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_backfill_scan_min": "64", 2024-04-14T15:11:32.823 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_backoff_on_degraded": "false", 2024-04-14T15:11:32.823 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_backoff_on_peering": "false", 2024-04-14T15:11:32.823 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_backoff_on_unfound": "true", 2024-04-14T15:11:32.823 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_beacon_report_interval": "300", 2024-04-14T15:11:32.823 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_bench_duration": "30", 2024-04-14T15:11:32.823 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-04-14T15:11:32.823 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_bench_max_block_size": "67108864", 2024-04-14T15:11:32.823 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_bench_small_size_max_iops": "100", 2024-04-14T15:11:32.823 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_blkin_trace_all": "false", 2024-04-14T15:11:32.823 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_blocked_scrub_grace_period": "120", 2024-04-14T15:11:32.824 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-04-14T15:11:32.824 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-04-14T15:11:32.824 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-04-14T15:11:32.824 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_check_for_log_corruption": "false", 2024-04-14T15:11:32.824 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-04-14T15:11:32.824 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-04-14T15:11:32.824 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-04-14T15:11:32.824 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-04-14T15:11:32.824 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_class_update_on_start": "true", 2024-04-14T15:11:32.824 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_client_message_cap": "256", 2024-04-14T15:11:32.824 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_client_message_size_cap": "524288000", 2024-04-14T15:11:32.824 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_client_op_priority": "63", 2024-04-14T15:11:32.824 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_client_watch_timeout": "30", 2024-04-14T15:11:32.824 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_command_max_records": "256", 2024-04-14T15:11:32.825 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_compact_on_start": "false", 2024-04-14T15:11:32.825 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-04-14T15:11:32.825 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_crush_chooseleaf_type": "1", 2024-04-14T15:11:32.825 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_crush_initial_weight": "-1.000000", 2024-04-14T15:11:32.825 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_crush_update_on_start": "true", 2024-04-14T15:11:32.825 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_crush_update_weight_set": "true", 2024-04-14T15:11:32.825 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-04-14T15:11:32.825 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-04-14T15:11:32.825 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-04-14T15:11:32.825 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_drop_ping_duration": "0", 2024-04-14T15:11:32.825 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-04-14T15:11:32.825 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_feed_pullee": "-1", 2024-04-14T15:11:32.825 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-04-14T15:11:32.825 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-04-14T15:11:32.825 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-04-14T15:11:32.826 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_misdirected_ops": "false", 2024-04-14T15:11:32.826 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_no_acting_change": "false", 2024-04-14T15:11:32.826 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_no_purge_strays": "false", 2024-04-14T15:11:32.826 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_op_order": "false", 2024-04-14T15:11:32.826 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_pg_log_writeout": "false", 2024-04-14T15:11:32.826 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_pretend_recovery_active": "false", 2024-04-14T15:11:32.826 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-04-14T15:11:32.826 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-04-14T15:11:32.826 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_shutdown": "false", 2024-04-14T15:11:32.826 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-04-14T15:11:32.826 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-04-14T15:11:32.826 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_trim_objects": "false", 2024-04-14T15:11:32.826 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_verify_cached_snaps": "false", 2024-04-14T15:11:32.827 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_verify_missing_on_start": "false", 2024-04-14T15:11:32.827 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_verify_snaps": "false", 2024-04-14T15:11:32.827 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-04-14T15:11:32.827 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-04-14T15:11:32.827 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_deep_scrub_keys": "1024", 2024-04-14T15:11:32.827 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-04-14T15:11:32.827 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-04-14T15:11:32.827 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-04-14T15:11:32.827 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_deep_scrub_stride": "524288", 2024-04-14T15:11:32.827 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-04-14T15:11:32.827 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_default_data_pool_replay_window": "45", 2024-04-14T15:11:32.827 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_default_notify_timeout": "30", 2024-04-14T15:11:32.827 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_delete_sleep": "0.000000", 2024-04-14T15:11:32.828 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-04-14T15:11:32.828 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-04-14T15:11:32.828 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-04-14T15:11:32.828 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_discard_disconnected_ops": "true", 2024-04-14T15:11:32.828 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_enable_op_tracker": "true", 2024-04-14T15:11:32.828 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-04-14T15:11:32.828 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_extblkdev_plugins": "vdo", 2024-04-14T15:11:32.828 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-04-14T15:11:32.828 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-04-14T15:11:32.828 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_fast_info": "true", 2024-04-14T15:11:32.828 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_fast_shutdown": "true", 2024-04-14T15:11:32.828 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-04-14T15:11:32.828 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_fast_shutdown_timeout": "15", 2024-04-14T15:11:32.828 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-04-14T15:11:32.829 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-04-14T15:11:32.829 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-04-14T15:11:32.829 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_function_tracing": "false", 2024-04-14T15:11:32.829 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_heartbeat_grace": "20", 2024-04-14T15:11:32.829 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_heartbeat_interval": "6", 2024-04-14T15:11:32.829 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-04-14T15:11:32.829 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_heartbeat_min_peers": "10", 2024-04-14T15:11:32.829 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_heartbeat_min_size": "2000", 2024-04-14T15:11:32.829 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_heartbeat_stale": "600", 2024-04-14T15:11:32.829 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-04-14T15:11:32.829 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_hit_set_max_size": "100000", 2024-04-14T15:11:32.829 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_hit_set_min_size": "1000", 2024-04-14T15:11:32.829 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-04-14T15:11:32.829 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-04-14T15:11:32.830 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-04-14T15:11:32.830 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-04-14T15:11:32.830 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-04-14T15:11:32.830 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_journal_flush_on_shutdown": "true", 2024-04-14T15:11:32.830 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_journal_size": "5120", 2024-04-14T15:11:32.830 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_kill_backfill_at": "0", 2024-04-14T15:11:32.830 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_loop_before_reset_tphandle": "64", 2024-04-14T15:11:32.830 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_map_cache_size": "50", 2024-04-14T15:11:32.830 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_map_dedup": "true", 2024-04-14T15:11:32.830 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_map_message_max": "40", 2024-04-14T15:11:32.830 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_map_message_max_bytes": "10485760", 2024-04-14T15:11:32.830 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_map_share_max_epochs": "40", 2024-04-14T15:11:32.830 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_max_attr_name_len": "100", 2024-04-14T15:11:32.831 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_max_attr_size": "0", 2024-04-14T15:11:32.831 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_max_backfills": "1", 2024-04-14T15:11:32.831 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_max_markdown_count": "5", 2024-04-14T15:11:32.831 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_max_markdown_period": "600", 2024-04-14T15:11:32.831 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_max_object_name_len": "2048", 2024-04-14T15:11:32.831 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_max_object_namespace_len": "256", 2024-04-14T15:11:32.831 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_max_object_size": "134217728", 2024-04-14T15:11:32.831 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-04-14T15:11:32.831 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_max_omap_entries_per_request": "1024", 2024-04-14T15:11:32.831 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_max_pg_blocked_by": "16", 2024-04-14T15:11:32.831 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_max_pg_log_entries": "10000", 2024-04-14T15:11:32.831 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-04-14T15:11:32.831 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_max_pgls": "1024", 2024-04-14T15:11:32.831 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_max_push_cost": "8388608", 2024-04-14T15:11:32.831 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_max_push_objects": "10", 2024-04-14T15:11:32.832 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_max_scrubs": "3", 2024-04-14T15:11:32.832 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-04-14T15:11:32.832 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_max_trimming_pgs": "2", 2024-04-14T15:11:32.832 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_max_write_op_reply_len": "64", 2024-04-14T15:11:32.832 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_max_write_size": "90", 2024-04-14T15:11:32.832 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-04-14T15:11:32.832 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-04-14T15:11:32.832 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-04-14T15:11:32.832 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-04-14T15:11:32.832 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-04-14T15:11:32.832 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-04-14T15:11:32.832 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-04-14T15:11:32.832 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mclock_override_recovery_settings": "false", 2024-04-14T15:11:32.832 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mclock_profile": "balanced", 2024-04-14T15:11:32.833 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-04-14T15:11:32.833 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-04-14T15:11:32.833 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-04-14T15:11:32.833 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-04-14T15:11:32.833 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-04-14T15:11:32.833 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-04-14T15:11:32.833 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-04-14T15:11:32.833 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-04-14T15:11:32.833 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-04-14T15:11:32.833 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-04-14T15:11:32.833 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mclock_skip_benchmark": "false", 2024-04-14T15:11:32.833 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_memory_base": "805306368", 2024-04-14T15:11:32.833 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_memory_cache_min": "134217728", 2024-04-14T15:11:32.833 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-04-14T15:11:32.834 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-04-14T15:11:32.834 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_memory_target": "4294967296", 2024-04-14T15:11:32.834 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_memory_target_autotune": "false", 2024-04-14T15:11:32.834 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-04-14T15:11:32.834 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_min_pg_log_entries": "250", 2024-04-14T15:11:32.834 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_min_recovery_priority": "0", 2024-04-14T15:11:32.834 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mon_heartbeat_interval": "30", 2024-04-14T15:11:32.834 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-04-14T15:11:32.834 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mon_report_interval": "5", 2024-04-14T15:11:32.834 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mon_report_max_in_flight": "2", 2024-04-14T15:11:32.834 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-04-14T15:11:32.834 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_num_cache_shards": "32", 2024-04-14T15:11:32.834 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_num_op_tracker_shard": "32", 2024-04-14T15:11:32.834 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_numa_auto_affinity": "true", 2024-04-14T15:11:32.835 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_numa_node": "-1", 2024-04-14T15:11:32.835 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_numa_prefer_iface": "true", 2024-04-14T15:11:32.835 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-04-14T15:11:32.835 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_objecter_finishers": "1", 2024-04-14T15:11:32.835 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_objectstore": "bluestore", 2024-04-14T15:11:32.835 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_objectstore_fuse": "false", 2024-04-14T15:11:32.835 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_objectstore_tracing": "false", 2024-04-14T15:11:32.835 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_op_complaint_time": "30.000000", 2024-04-14T15:11:32.835 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_op_history_duration": "600", 2024-04-14T15:11:32.835 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_op_history_size": "20", 2024-04-14T15:11:32.835 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_op_history_slow_op_size": "20", 2024-04-14T15:11:32.835 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-04-14T15:11:32.835 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_op_log_threshold": "5", 2024-04-14T15:11:32.835 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_op_num_shards": "0", 2024-04-14T15:11:32.836 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_op_num_shards_hdd": "5", 2024-04-14T15:11:32.836 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_op_num_shards_ssd": "8", 2024-04-14T15:11:32.836 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_op_num_threads_per_shard": "0", 2024-04-14T15:11:32.836 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-04-14T15:11:32.836 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-04-14T15:11:32.836 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-04-14T15:11:32.836 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_op_pq_min_cost": "65536", 2024-04-14T15:11:32.836 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_op_queue": "mclock_scheduler", 2024-04-14T15:11:32.836 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_op_queue_cut_off": "high", 2024-04-14T15:11:32.836 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_op_thread_suicide_timeout": "150", 2024-04-14T15:11:32.836 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_op_thread_timeout": "15", 2024-04-14T15:11:32.836 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_open_classes_on_start": "true", 2024-04-14T15:11:32.836 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_os_flags": "0", 2024-04-14T15:11:32.837 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_peering_op_priority": "255", 2024-04-14T15:11:32.837 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pg_delete_cost": "1048576", 2024-04-14T15:11:32.837 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pg_delete_priority": "5", 2024-04-14T15:11:32.837 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-04-14T15:11:32.837 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-04-14T15:11:32.837 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pg_log_dups_tracked": "3000", 2024-04-14T15:11:32.837 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pg_log_trim_max": "10000", 2024-04-14T15:11:32.837 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pg_log_trim_min": "100", 2024-04-14T15:11:32.837 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-04-14T15:11:32.837 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pg_object_context_cache_count": "64", 2024-04-14T15:11:32.837 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-04-14T15:11:32.837 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-04-14T15:11:32.837 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-04-14T15:11:32.837 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-04-14T15:11:32.837 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-04-14T15:11:32.838 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-04-14T15:11:32.838 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-04-14T15:11:32.838 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-04-14T15:11:32.838 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_crimson": "false", 2024-04-14T15:11:32.838 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_crush_rule": "-1", 2024-04-14T15:11:32.838 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_ec_fast_read": "false", 2024-04-14T15:11:32.838 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-04-14T15:11:32.838 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_flag_bulk": "false", 2024-04-14T15:11:32.838 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-04-14T15:11:32.838 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_flag_nodelete": "false", 2024-04-14T15:11:32.838 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-04-14T15:11:32.838 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-04-14T15:11:32.838 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_flags": "0", 2024-04-14T15:11:32.839 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-04-14T15:11:32.839 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_min_size": "0", 2024-04-14T15:11:32.839 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-04-14T15:11:32.839 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_pg_num": "32", 2024-04-14T15:11:32.839 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_pgp_num": "0", 2024-04-14T15:11:32.839 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-04-14T15:11:32.839 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_read_ratio": "70", 2024-04-14T15:11:32.839 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_size": "3", 2024-04-14T15:11:32.839 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_type": "replicated", 2024-04-14T15:11:32.839 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-04-14T15:11:32.839 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_use_gmt_hitset": "true", 2024-04-14T15:11:32.839 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_push_per_object_cost": "1000", 2024-04-14T15:11:32.839 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_read_ec_check_for_errors": "false", 2024-04-14T15:11:32.839 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_recover_clone_overlap": "true", 2024-04-14T15:11:32.840 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_recover_clone_overlap_limit": "10", 2024-04-14T15:11:32.840 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_recovery_cost": "20971520", 2024-04-14T15:11:32.840 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_recovery_delay_start": "0.000000", 2024-04-14T15:11:32.840 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_recovery_max_active": "0", 2024-04-14T15:11:32.840 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_recovery_max_active_hdd": "3", 2024-04-14T15:11:32.840 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_recovery_max_active_ssd": "10", 2024-04-14T15:11:32.840 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_recovery_max_chunk": "8388608", 2024-04-14T15:11:32.840 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-04-14T15:11:32.840 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_recovery_max_single_start": "1", 2024-04-14T15:11:32.840 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_recovery_op_priority": "3", 2024-04-14T15:11:32.840 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_recovery_op_warn_multiple": "16", 2024-04-14T15:11:32.840 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_recovery_priority": "5", 2024-04-14T15:11:32.840 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_recovery_retry_interval": "30.000000", 2024-04-14T15:11:32.841 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_recovery_sleep": "0.000000", 2024-04-14T15:11:32.841 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-04-14T15:11:32.841 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-04-14T15:11:32.841 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-04-14T15:11:32.841 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_repair_during_recovery": "false", 2024-04-14T15:11:32.841 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_requested_scrub_priority": "120", 2024-04-14T15:11:32.841 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-04-14T15:11:32.841 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_rollback_to_cluster_snap": "", 2024-04-14T15:11:32.841 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_auto_repair": "false", 2024-04-14T15:11:32.841 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-04-14T15:11:32.841 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-04-14T15:11:32.841 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_begin_hour": "0", 2024-04-14T15:11:32.841 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_begin_week_day": "0", 2024-04-14T15:11:32.841 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_chunk_max": "25", 2024-04-14T15:11:32.842 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_chunk_min": "5", 2024-04-14T15:11:32.842 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_cost": "52428800", 2024-04-14T15:11:32.842 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-04-14T15:11:32.842 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_during_recovery": "false", 2024-04-14T15:11:32.842 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_end_hour": "0", 2024-04-14T15:11:32.842 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_end_week_day": "0", 2024-04-14T15:11:32.842 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_event_cost": "4096", 2024-04-14T15:11:32.842 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-04-14T15:11:32.842 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-04-14T15:11:32.842 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_invalid_stats": "true", 2024-04-14T15:11:32.842 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_load_threshold": "0.500000", 2024-04-14T15:11:32.842 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_max_interval": "604800.000000", 2024-04-14T15:11:32.842 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_max_preemptions": "5", 2024-04-14T15:11:32.842 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_min_interval": "86400.000000", 2024-04-14T15:11:32.843 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_priority": "5", 2024-04-14T15:11:32.843 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_reservation_timeout": "300000", 2024-04-14T15:11:32.843 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_sleep": "0.000000", 2024-04-14T15:11:32.843 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-04-14T15:11:32.843 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-04-14T15:11:32.843 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_shutdown_pgref_assert": "false", 2024-04-14T15:11:32.843 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-04-14T15:11:32.843 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_skip_data_digest": "false", 2024-04-14T15:11:32.843 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_smart_report_timeout": "5", 2024-04-14T15:11:32.843 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_snap_trim_cost": "1048576", 2024-04-14T15:11:32.843 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_snap_trim_priority": "5", 2024-04-14T15:11:32.843 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_snap_trim_sleep": "0.000000", 2024-04-14T15:11:32.843 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-04-14T15:11:32.843 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-04-14T15:11:32.844 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-04-14T15:11:32.844 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-04-14T15:11:32.844 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_stats_update_period_scrubbing": "15", 2024-04-14T15:11:32.844 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-04-14T15:11:32.844 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_target_transaction_size": "30", 2024-04-14T15:11:32.844 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-04-14T15:11:32.844 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-04-14T15:11:32.844 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-04-14T15:11:32.844 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-04-14T15:11:32.844 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-04-14T15:11:32.844 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-04-14T15:11:32.844 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-04-14T15:11:32.844 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_tier_default_cache_mode": "writeback", 2024-04-14T15:11:32.844 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-04-14T15:11:32.844 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-04-14T15:11:32.844 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_tracing": "false", 2024-04-14T15:11:32.845 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_use_stale_snap": "false", 2024-04-14T15:11:32.845 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-04-14T15:11:32.845 INFO:teuthology.orchestra.run.smithi090.stdout: "osdc_blkin_trace_all": "false", 2024-04-14T15:11:32.845 INFO:teuthology.orchestra.run.smithi090.stdout: "paxos_kill_at": "0", 2024-04-14T15:11:32.845 INFO:teuthology.orchestra.run.smithi090.stdout: "paxos_max_join_drift": "10", 2024-04-14T15:11:32.845 INFO:teuthology.orchestra.run.smithi090.stdout: "paxos_min": "500", 2024-04-14T15:11:32.845 INFO:teuthology.orchestra.run.smithi090.stdout: "paxos_min_wait": "0.050000", 2024-04-14T15:11:32.845 INFO:teuthology.orchestra.run.smithi090.stdout: "paxos_propose_interval": "1.000000", 2024-04-14T15:11:32.845 INFO:teuthology.orchestra.run.smithi090.stdout: "paxos_service_trim_max": "500", 2024-04-14T15:11:32.845 INFO:teuthology.orchestra.run.smithi090.stdout: "paxos_service_trim_max_multiplier": "20", 2024-04-14T15:11:32.845 INFO:teuthology.orchestra.run.smithi090.stdout: "paxos_service_trim_min": "10", 2024-04-14T15:11:32.845 INFO:teuthology.orchestra.run.smithi090.stdout: "paxos_stash_full_interval": "25", 2024-04-14T15:11:32.845 INFO:teuthology.orchestra.run.smithi090.stdout: "paxos_trim_max": "500", 2024-04-14T15:11:32.846 INFO:teuthology.orchestra.run.smithi090.stdout: "paxos_trim_min": "250", 2024-04-14T15:11:32.846 INFO:teuthology.orchestra.run.smithi090.stdout: "perf": "true", 2024-04-14T15:11:32.846 INFO:teuthology.orchestra.run.smithi090.stdout: "pid_file": "", 2024-04-14T15:11:32.846 INFO:teuthology.orchestra.run.smithi090.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-04-14T15:11:32.846 INFO:teuthology.orchestra.run.smithi090.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-04-14T15:11:32.846 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "-", 2024-04-14T15:11:32.849 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrv": "", 2024-04-14T15:11:32.849 INFO:teuthology.orchestra.run.smithi090.stdout: "public_bind_addr": "-", 2024-04-14T15:11:32.849 INFO:teuthology.orchestra.run.smithi090.stdout: "public_network": "", 2024-04-14T15:11:32.849 INFO:teuthology.orchestra.run.smithi090.stdout: "public_network_interface": "", 2024-04-14T15:11:32.850 INFO:teuthology.orchestra.run.smithi090.stdout: "qat_compressor_enabled": "false", 2024-04-14T15:11:32.850 INFO:teuthology.orchestra.run.smithi090.stdout: "qat_compressor_session_max_number": "256", 2024-04-14T15:11:32.850 INFO:teuthology.orchestra.run.smithi090.stdout: "rados_mon_op_timeout": "0", 2024-04-14T15:11:32.850 INFO:teuthology.orchestra.run.smithi090.stdout: "rados_osd_op_timeout": "0", 2024-04-14T15:11:32.850 INFO:teuthology.orchestra.run.smithi090.stdout: "rados_tracing": "false", 2024-04-14T15:11:32.850 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_atime_update_interval": "60", 2024-04-14T15:11:32.850 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-04-14T15:11:32.850 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_balance_parent_reads": "false", 2024-04-14T15:11:32.850 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_balance_snap_reads": "false", 2024-04-14T15:11:32.850 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_blkin_trace_all": "false", 2024-04-14T15:11:32.850 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_blocklist_expire_seconds": "0", 2024-04-14T15:11:32.850 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_blocklist_on_break_lock": "true", 2024-04-14T15:11:32.850 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_cache": "true", 2024-04-14T15:11:32.850 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_cache_block_writes_upfront": "false", 2024-04-14T15:11:32.851 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_cache_max_dirty": "25165824", 2024-04-14T15:11:32.851 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-04-14T15:11:32.851 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_cache_max_dirty_object": "0", 2024-04-14T15:11:32.851 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_cache_policy": "writearound", 2024-04-14T15:11:32.851 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_cache_size": "33554432", 2024-04-14T15:11:32.851 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_cache_target_dirty": "16777216", 2024-04-14T15:11:32.851 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-04-14T15:11:32.851 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_clone_copy_on_read": "false", 2024-04-14T15:11:32.851 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_compression_hint": "none", 2024-04-14T15:11:32.851 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_concurrent_management_ops": "10", 2024-04-14T15:11:32.851 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-04-14T15:11:32.851 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_default_clone_format": "auto", 2024-04-14T15:11:32.851 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_default_data_pool": "", 2024-04-14T15:11:32.852 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_default_features": "61", 2024-04-14T15:11:32.852 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_default_format": "2", 2024-04-14T15:11:32.852 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_default_map_options": "", 2024-04-14T15:11:32.852 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_default_order": "22", 2024-04-14T15:11:32.852 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_default_pool": "rbd", 2024-04-14T15:11:32.852 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-04-14T15:11:32.852 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_default_stripe_count": "0", 2024-04-14T15:11:32.852 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_default_stripe_unit": "0", 2024-04-14T15:11:32.852 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_disable_zero_copy_writes": "true", 2024-04-14T15:11:32.852 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_discard_granularity_bytes": "65536", 2024-04-14T15:11:32.852 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-04-14T15:11:32.852 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_enable_alloc_hint": "true", 2024-04-14T15:11:32.852 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-04-14T15:11:32.852 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_io_scheduler": "simple", 2024-04-14T15:11:32.852 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-04-14T15:11:32.853 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_journal_commit_age": "5.000000", 2024-04-14T15:11:32.853 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-04-14T15:11:32.853 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-04-14T15:11:32.853 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-04-14T15:11:32.853 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-04-14T15:11:32.853 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_journal_object_flush_interval": "0", 2024-04-14T15:11:32.853 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-04-14T15:11:32.853 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-04-14T15:11:32.853 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_journal_order": "24", 2024-04-14T15:11:32.853 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_journal_pool": "", 2024-04-14T15:11:32.853 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_journal_splay_width": "4", 2024-04-14T15:11:32.853 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_localize_parent_reads": "false", 2024-04-14T15:11:32.853 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_localize_snap_reads": "false", 2024-04-14T15:11:32.853 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-04-14T15:11:32.853 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-04-14T15:11:32.854 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-04-14T15:11:32.854 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-04-14T15:11:32.854 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-04-14T15:11:32.854 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-04-14T15:11:32.854 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_image_policy_type": "simple", 2024-04-14T15:11:32.854 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-04-14T15:11:32.854 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-04-14T15:11:32.854 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-04-14T15:11:32.854 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-04-14T15:11:32.854 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-04-14T15:11:32.854 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-04-14T15:11:32.854 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-04-14T15:11:32.855 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_memory_autotune": "true", 2024-04-14T15:11:32.855 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_memory_base": "805306368", 2024-04-14T15:11:32.855 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-04-14T15:11:32.855 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-04-14T15:11:32.855 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-04-14T15:11:32.855 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-04-14T15:11:32.855 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_memory_target": "4294967296", 2024-04-14T15:11:32.855 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-04-14T15:11:32.855 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-04-14T15:11:32.855 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-04-14T15:11:32.855 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirroring_delete_delay": "0", 2024-04-14T15:11:32.855 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-04-14T15:11:32.855 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirroring_replay_delay": "0", 2024-04-14T15:11:32.855 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-04-14T15:11:32.855 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-04-14T15:11:32.855 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_move_to_trash_on_remove": "false", 2024-04-14T15:11:32.856 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-04-14T15:11:32.856 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mtime_update_interval": "60", 2024-04-14T15:11:32.856 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_non_blocking_aio": "true", 2024-04-14T15:11:32.856 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_op_thread_timeout": "60", 2024-04-14T15:11:32.856 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_op_threads": "1", 2024-04-14T15:11:32.856 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_parent_cache_enabled": "false", 2024-04-14T15:11:32.856 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_persistent_cache_mode": "disabled", 2024-04-14T15:11:32.856 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_persistent_cache_path": "/tmp", 2024-04-14T15:11:32.856 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_persistent_cache_size": "1073741824", 2024-04-14T15:11:32.856 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_plugins": "", 2024-04-14T15:11:32.856 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_qos_bps_burst": "0", 2024-04-14T15:11:32.856 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-04-14T15:11:32.856 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_qos_bps_limit": "0", 2024-04-14T15:11:32.856 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_qos_exclude_ops": "0", 2024-04-14T15:11:32.856 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_qos_iops_burst": "0", 2024-04-14T15:11:32.857 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-04-14T15:11:32.857 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_qos_iops_limit": "0", 2024-04-14T15:11:32.857 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_qos_read_bps_burst": "0", 2024-04-14T15:11:32.857 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-04-14T15:11:32.857 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_qos_read_bps_limit": "0", 2024-04-14T15:11:32.857 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_qos_read_iops_burst": "0", 2024-04-14T15:11:32.857 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-04-14T15:11:32.857 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_qos_read_iops_limit": "0", 2024-04-14T15:11:32.857 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_qos_schedule_tick_min": "50", 2024-04-14T15:11:32.857 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_qos_write_bps_burst": "0", 2024-04-14T15:11:32.857 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-04-14T15:11:32.857 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_qos_write_bps_limit": "0", 2024-04-14T15:11:32.857 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_qos_write_iops_burst": "0", 2024-04-14T15:11:32.857 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-04-14T15:11:32.857 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_qos_write_iops_limit": "0", 2024-04-14T15:11:32.858 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_quiesce_notification_attempts": "10", 2024-04-14T15:11:32.858 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_read_from_replica_policy": "default", 2024-04-14T15:11:32.858 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-04-14T15:11:32.858 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_readahead_max_bytes": "524288", 2024-04-14T15:11:32.858 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_readahead_trigger_requests": "10", 2024-04-14T15:11:32.858 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_request_timed_out_seconds": "30", 2024-04-14T15:11:32.858 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_skip_partial_discard": "true", 2024-04-14T15:11:32.858 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-04-14T15:11:32.858 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_tracing": "false", 2024-04-14T15:11:32.858 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_validate_names": "true", 2024-04-14T15:11:32.858 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_validate_pool": "true", 2024-04-14T15:11:32.858 INFO:teuthology.orchestra.run.smithi090.stdout: "restapi_base_url": "", 2024-04-14T15:11:32.858 INFO:teuthology.orchestra.run.smithi090.stdout: "restapi_log_level": "", 2024-04-14T15:11:32.859 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_acl_grants_max_num": "100", 2024-04-14T15:11:32.859 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_admin_entry": "admin", 2024-04-14T15:11:32.859 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-04-14T15:11:32.859 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_backend_store": "rados", 2024-04-14T15:11:32.859 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_barbican_url": "", 2024-04-14T15:11:32.859 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_beast_enable_async": "true", 2024-04-14T15:11:32.859 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_bucket_counters_cache": "false", 2024-04-14T15:11:32.859 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-04-14T15:11:32.859 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-04-14T15:11:32.859 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-04-14T15:11:32.859 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_bucket_index_max_aio": "128", 2024-04-14T15:11:32.859 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-04-14T15:11:32.859 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-04-14T15:11:32.859 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_bucket_quota_ttl": "600", 2024-04-14T15:11:32.859 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-04-14T15:11:32.860 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_cache_enabled": "true", 2024-04-14T15:11:32.860 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_cache_expiry_interval": "900", 2024-04-14T15:11:32.860 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_cache_lru_size": "10000", 2024-04-14T15:11:32.860 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_config_store": "rados", 2024-04-14T15:11:32.860 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_content_length_compat": "false", 2024-04-14T15:11:32.860 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_copy_obj_progress": "true", 2024-04-14T15:11:32.860 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-04-14T15:11:32.860 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_cors_rules_max_num": "100", 2024-04-14T15:11:32.860 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_cross_domain_policy": "", 2024-04-14T15:11:32.860 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_default_encryption_key": "", 2024-04-14T15:11:32.860 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_kmip_addr": "", 2024-04-14T15:11:32.860 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_kmip_ca_path": "", 2024-04-14T15:11:32.860 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_kmip_client_cert": "", 2024-04-14T15:11:32.860 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_kmip_client_key": "", 2024-04-14T15:11:32.861 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-04-14T15:11:32.861 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_kmip_password": "", 2024-04-14T15:11:32.861 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-04-14T15:11:32.861 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_kmip_username": "", 2024-04-14T15:11:32.861 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_require_ssl": "true", 2024-04-14T15:11:32.861 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-04-14T15:11:32.861 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-04-14T15:11:32.861 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-04-14T15:11:32.861 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-04-14T15:11:32.861 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-04-14T15:11:32.861 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-04-14T15:11:32.861 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-04-14T15:11:32.861 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-04-14T15:11:32.861 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-04-14T15:11:32.861 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-04-14T15:11:32.862 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-04-14T15:11:32.862 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-04-14T15:11:32.862 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-04-14T15:11:32.862 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-04-14T15:11:32.862 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_suppress_logs": "true", 2024-04-14T15:11:32.862 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_vault_addr": "", 2024-04-14T15:11:32.862 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_vault_auth": "token", 2024-04-14T15:11:32.862 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_vault_namespace": "", 2024-04-14T15:11:32.862 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_vault_prefix": "", 2024-04-14T15:11:32.862 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-04-14T15:11:32.863 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-04-14T15:11:32.863 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-04-14T15:11:32.863 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-04-14T15:11:32.863 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_vault_token_file": "", 2024-04-14T15:11:32.863 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-04-14T15:11:32.863 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_curl_buffersize": "524288", 2024-04-14T15:11:32.863 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_curl_low_speed_limit": "1024", 2024-04-14T15:11:32.863 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_curl_low_speed_time": "300", 2024-04-14T15:11:32.863 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_curl_tcp_keepalive": "0", 2024-04-14T15:11:32.863 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-04-14T15:11:32.863 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-04-14T15:11:32.863 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-04-14T15:11:32.864 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-04-14T15:11:32.864 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-04-14T15:11:32.864 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_d3n_l1_fadvise": "4", 2024-04-14T15:11:32.864 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-04-14T15:11:32.864 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-04-14T15:11:32.864 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-04-14T15:11:32.864 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_d4n_host": "127.0.0.1", 2024-04-14T15:11:32.864 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_d4n_l1_datacache_persistent_path": "/tmp/rgw_d4n_datacache/", 2024-04-14T15:11:32.864 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_d4n_l1_datacache_size": "1073741824", 2024-04-14T15:11:32.864 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_d4n_l1_evict_cache_on_start": "true", 2024-04-14T15:11:32.864 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_d4n_l1_fadvise": "4", 2024-04-14T15:11:32.864 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_d4n_libaio_aio_num": "64", 2024-04-14T15:11:32.865 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_d4n_libaio_aio_threads": "20", 2024-04-14T15:11:32.865 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_d4n_port": "6379", 2024-04-14T15:11:32.865 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-04-14T15:11:32.865 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_data_log_changes_size": "1000", 2024-04-14T15:11:32.865 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_data_log_num_shards": "128", 2024-04-14T15:11:32.865 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-04-14T15:11:32.865 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_data_log_window": "30", 2024-04-14T15:11:32.865 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_data_notify_interval_msec": "0", 2024-04-14T15:11:32.865 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_data_sync_poll_interval": "20", 2024-04-14T15:11:32.865 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_data_sync_spawn_window": "20", 2024-04-14T15:11:32.865 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-04-14T15:11:32.866 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-04-14T15:11:32.866 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-04-14T15:11:32.866 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_default_data_log_backing": "fifo", 2024-04-14T15:11:32.866 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-04-14T15:11:32.866 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_default_region_info_oid": "default.region", 2024-04-14T15:11:32.866 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-04-14T15:11:32.866 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-04-14T15:11:32.866 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_defer_to_bucket_acls": "", 2024-04-14T15:11:32.866 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-04-14T15:11:32.866 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-04-14T15:11:32.866 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-04-14T15:11:32.867 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-04-14T15:11:32.867 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-04-14T15:11:32.867 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-04-14T15:11:32.867 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-04-14T15:11:32.867 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-04-14T15:11:32.867 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_dmclock_data_res": "500.000000", 2024-04-14T15:11:32.867 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-04-14T15:11:32.867 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-04-14T15:11:32.867 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-04-14T15:11:32.867 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-04-14T15:11:32.867 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_dns_name": "", 2024-04-14T15:11:32.867 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_dns_s3website_name": "", 2024-04-14T15:11:32.868 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_dynamic_resharding": "true", 2024-04-14T15:11:32.868 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-04-14T15:11:32.868 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_enable_gc_threads": "true", 2024-04-14T15:11:32.868 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_enable_lc_threads": "true", 2024-04-14T15:11:32.868 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_enable_ops_log": "false", 2024-04-14T15:11:32.868 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_enable_quota_threads": "true", 2024-04-14T15:11:32.868 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_enable_static_website": "false", 2024-04-14T15:11:32.868 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_enable_usage_log": "false", 2024-04-14T15:11:32.868 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_enforce_swift_acls": "true", 2024-04-14T15:11:32.868 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_exit_timeout_secs": "120", 2024-04-14T15:11:32.868 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_expose_bucket": "false", 2024-04-14T15:11:32.869 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_extended_http_attrs": "", 2024-04-14T15:11:32.869 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_filter": "none", 2024-04-14T15:11:32.869 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-04-14T15:11:32.869 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_frontends": "beast port=7480", 2024-04-14T15:11:32.869 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_gc_max_concurrent_io": "10", 2024-04-14T15:11:32.869 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_gc_max_deferred": "50", 2024-04-14T15:11:32.869 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-04-14T15:11:32.869 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_gc_max_objs": "32", 2024-04-14T15:11:32.869 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_gc_max_queue_size": "134213632", 2024-04-14T15:11:32.869 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_gc_max_trim_chunk": "16", 2024-04-14T15:11:32.869 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_gc_obj_min_wait": "7200", 2024-04-14T15:11:32.869 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_gc_processor_max_time": "3600", 2024-04-14T15:11:32.870 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_gc_processor_period": "3600", 2024-04-14T15:11:32.870 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-04-14T15:11:32.870 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_get_obj_window_size": "16777216", 2024-04-14T15:11:32.870 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_healthcheck_disabling_path": "", 2024-04-14T15:11:32.870 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_ignore_get_invalid_range": "false", 2024-04-14T15:11:32.870 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_init_timeout": "300", 2024-04-14T15:11:32.870 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-04-14T15:11:32.870 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-04-14T15:11:32.870 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_kafka_connection_idle": "300", 2024-04-14T15:11:32.870 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_kafka_message_timeout": "5000", 2024-04-14T15:11:32.870 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_kafka_sleep_timeout": "10", 2024-04-14T15:11:32.870 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-04-14T15:11:32.871 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-04-14T15:11:32.871 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-04-14T15:11:32.871 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_admin_domain": "", 2024-04-14T15:11:32.871 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_admin_password": "", 2024-04-14T15:11:32.871 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_admin_password_path": "", 2024-04-14T15:11:32.871 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_admin_project": "", 2024-04-14T15:11:32.871 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_admin_tenant": "", 2024-04-14T15:11:32.871 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_admin_token": "", 2024-04-14T15:11:32.871 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_admin_token_path": "", 2024-04-14T15:11:32.871 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_admin_user": "", 2024-04-14T15:11:32.871 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_api_version": "2", 2024-04-14T15:11:32.871 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_barbican_domain": "", 2024-04-14T15:11:32.871 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_barbican_password": "", 2024-04-14T15:11:32.872 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_barbican_project": "", 2024-04-14T15:11:32.872 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_barbican_tenant": "", 2024-04-14T15:11:32.872 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_barbican_user": "", 2024-04-14T15:11:32.872 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-04-14T15:11:32.872 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_implicit_tenants": "false", 2024-04-14T15:11:32.872 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-04-14T15:11:32.872 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_service_token_enabled": "false", 2024-04-14T15:11:32.872 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_token_cache_size": "10000", 2024-04-14T15:11:32.872 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_url": "", 2024-04-14T15:11:32.872 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_verify_ssl": "true", 2024-04-14T15:11:32.872 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_lc_debug_interval": "-1", 2024-04-14T15:11:32.872 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_lc_lock_max_time": "90", 2024-04-14T15:11:32.873 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_lc_max_objs": "32", 2024-04-14T15:11:32.873 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_lc_max_rules": "1000", 2024-04-14T15:11:32.873 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_lc_max_worker": "3", 2024-04-14T15:11:32.873 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_lc_max_wp_worker": "3", 2024-04-14T15:11:32.873 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_lc_thread_delay": "0", 2024-04-14T15:11:32.873 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-04-14T15:11:32.873 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_ldap_dnattr": "uid", 2024-04-14T15:11:32.873 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-04-14T15:11:32.873 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_ldap_searchfilter": "", 2024-04-14T15:11:32.873 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-04-14T15:11:32.873 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_ldap_uri": "", 2024-04-14T15:11:32.873 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_lfuda_sync_frequency": "60", 2024-04-14T15:11:32.874 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-04-14T15:11:32.874 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-04-14T15:11:32.874 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-04-14T15:11:32.874 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_local_cache_address": "127.0.0.1:6379", 2024-04-14T15:11:32.874 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_log_http_headers": "", 2024-04-14T15:11:32.874 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_log_nonexistent_bucket": "false", 2024-04-14T15:11:32.874 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-04-14T15:11:32.874 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_log_object_name_utc": "false", 2024-04-14T15:11:32.874 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-04-14T15:11:32.874 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-04-14T15:11:32.875 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_max_attr_name_len": "0", 2024-04-14T15:11:32.875 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_max_attr_size": "0", 2024-04-14T15:11:32.875 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_max_attrs_num_in_req": "0", 2024-04-14T15:11:32.875 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_max_chunk_size": "4194304", 2024-04-14T15:11:32.875 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_max_concurrent_requests": "1024", 2024-04-14T15:11:32.875 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-04-14T15:11:32.875 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_max_dynamic_shards": "1999", 2024-04-14T15:11:32.875 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_max_listing_results": "1000", 2024-04-14T15:11:32.875 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_max_notify_retries": "10", 2024-04-14T15:11:32.875 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_max_objs_per_shard": "100000", 2024-04-14T15:11:32.875 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_max_put_param_size": "1048576", 2024-04-14T15:11:32.875 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_max_put_size": "5368709120", 2024-04-14T15:11:32.876 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_max_slo_entries": "1000", 2024-04-14T15:11:32.876 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_md_log_max_shards": "64", 2024-04-14T15:11:32.876 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_md_notify_interval_msec": "200", 2024-04-14T15:11:32.876 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_meta_sync_poll_interval": "20", 2024-04-14T15:11:32.876 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_meta_sync_spawn_window": "20", 2024-04-14T15:11:32.876 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-04-14T15:11:32.876 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_mp_lock_max_time": "600", 2024-04-14T15:11:32.876 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-04-14T15:11:32.876 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_multipart_min_part_size": "5242880", 2024-04-14T15:11:32.876 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-04-14T15:11:32.876 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-04-14T15:11:32.876 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_nfs_fhcache_size": "2017", 2024-04-14T15:11:32.877 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-04-14T15:11:32.877 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-04-14T15:11:32.877 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_nfs_lru_lanes": "5", 2024-04-14T15:11:32.877 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_nfs_max_gc": "300", 2024-04-14T15:11:32.877 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-04-14T15:11:32.877 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_nfs_run_gc_threads": "false", 2024-04-14T15:11:32.877 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_nfs_run_lc_threads": "false", 2024-04-14T15:11:32.877 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_nfs_run_quota_threads": "false", 2024-04-14T15:11:32.877 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_nfs_run_sync_thread": "false", 2024-04-14T15:11:32.877 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-04-14T15:11:32.877 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-04-14T15:11:32.878 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_num_async_rados_threads": "32", 2024-04-14T15:11:32.878 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_num_control_oids": "8", 2024-04-14T15:11:32.878 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_numa_node": "-1", 2024-04-14T15:11:32.878 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_obj_stripe_size": "4194304", 2024-04-14T15:11:32.878 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-04-14T15:11:32.878 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_objexp_chunk_size": "100", 2024-04-14T15:11:32.878 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_objexp_gc_interval": "600", 2024-04-14T15:11:32.878 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_objexp_hints_num_shards": "127", 2024-04-14T15:11:32.878 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-04-14T15:11:32.878 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-04-14T15:11:32.878 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_op_thread_timeout": "600", 2024-04-14T15:11:32.878 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_op_tracing": "false", 2024-04-14T15:11:32.879 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_opa_token": "", 2024-04-14T15:11:32.879 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_opa_url": "", 2024-04-14T15:11:32.879 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_opa_verify_ssl": "true", 2024-04-14T15:11:32.879 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-04-14T15:11:32.879 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-04-14T15:11:32.879 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_ops_log_rados": "false", 2024-04-14T15:11:32.879 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_ops_log_socket_path": "", 2024-04-14T15:11:32.879 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-04-14T15:11:32.879 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-04-14T15:11:32.879 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-04-14T15:11:32.879 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_period_push_interval": "2.000000", 2024-04-14T15:11:32.880 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_period_push_interval_max": "30.000000", 2024-04-14T15:11:32.880 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_period_root_pool": ".rgw.root", 2024-04-14T15:11:32.880 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-04-14T15:11:32.880 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-04-14T15:11:32.880 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_posix_cache_lanes": "3", 2024-04-14T15:11:32.880 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-04-14T15:11:32.880 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_posix_cache_max_buckets": "100", 2024-04-14T15:11:32.880 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_posix_cache_partitions": "3", 2024-04-14T15:11:32.880 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-04-14T15:11:32.880 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_print_continue": "true", 2024-04-14T15:11:32.880 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_print_prohibited_content_length": "false", 2024-04-14T15:11:32.880 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-04-14T15:11:32.881 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-04-14T15:11:32.881 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-04-14T15:11:32.881 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-04-14T15:11:32.881 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_rados_tracing": "false", 2024-04-14T15:11:32.881 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_realm": "", 2024-04-14T15:11:32.881 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_realm_id": "", 2024-04-14T15:11:32.881 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-04-14T15:11:32.881 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_region": "", 2024-04-14T15:11:32.881 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_region_root_pool": ".rgw.root", 2024-04-14T15:11:32.881 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_relaxed_region_enforcement": "false", 2024-04-14T15:11:32.881 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-04-14T15:11:32.882 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-04-14T15:11:32.882 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_request_uri": "", 2024-04-14T15:11:32.882 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_reshard_batch_size": "64", 2024-04-14T15:11:32.882 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-04-14T15:11:32.882 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_reshard_max_aio": "128", 2024-04-14T15:11:32.882 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_reshard_num_logs": "16", 2024-04-14T15:11:32.882 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_reshard_thread_interval": "600", 2024-04-14T15:11:32.882 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_resolve_cname": "false", 2024-04-14T15:11:32.882 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_rest_getusage_op_compat": "false", 2024-04-14T15:11:32.882 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_run_sync_thread": "true", 2024-04-14T15:11:32.882 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-04-14T15:11:32.882 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-04-14T15:11:32.883 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_s3_auth_use_keystone": "false", 2024-04-14T15:11:32.883 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_s3_auth_use_ldap": "false", 2024-04-14T15:11:32.883 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_s3_auth_use_rados": "true", 2024-04-14T15:11:32.883 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_s3_auth_use_sts": "false", 2024-04-14T15:11:32.883 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-04-14T15:11:32.883 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_s3_success_create_obj_status": "0", 2024-04-14T15:11:32.883 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-04-14T15:11:32.883 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_scheduler_type": "throttler", 2024-04-14T15:11:32.883 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_script_uri": "", 2024-04-14T15:11:32.883 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_service_provider_name": "", 2024-04-14T15:11:32.883 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-04-14T15:11:32.883 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_sts_client_id": "", 2024-04-14T15:11:32.883 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_sts_client_secret": "", 2024-04-14T15:11:32.884 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_sts_entry": "sts", 2024-04-14T15:11:32.884 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_sts_key": "sts", 2024-04-14T15:11:32.884 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_sts_max_session_duration": "43200", 2024-04-14T15:11:32.884 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_sts_min_session_duration": "900", 2024-04-14T15:11:32.884 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_sts_token_introspection_url": "", 2024-04-14T15:11:32.884 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_swift_account_in_url": "false", 2024-04-14T15:11:32.884 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_swift_auth_entry": "auth", 2024-04-14T15:11:32.884 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_swift_auth_url": "", 2024-04-14T15:11:32.884 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_swift_custom_header": "", 2024-04-14T15:11:32.884 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_swift_enforce_content_length": "false", 2024-04-14T15:11:32.884 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_swift_need_stats": "true", 2024-04-14T15:11:32.884 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_swift_tenant_name": "", 2024-04-14T15:11:32.885 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_swift_token_expiration": "86400", 2024-04-14T15:11:32.885 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_swift_url": "", 2024-04-14T15:11:32.885 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_swift_url_prefix": "swift", 2024-04-14T15:11:32.885 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_swift_versioning_enabled": "false", 2024-04-14T15:11:32.885 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-04-14T15:11:32.885 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-04-14T15:11:32.885 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_sync_lease_period": "120", 2024-04-14T15:11:32.885 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-04-14T15:11:32.885 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_sync_log_trim_interval": "1200", 2024-04-14T15:11:32.885 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-04-14T15:11:32.885 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-04-14T15:11:32.885 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-04-14T15:11:32.886 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_sync_obj_etag_verify": "false", 2024-04-14T15:11:32.886 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_sync_trace_history_size": "4096", 2024-04-14T15:11:32.886 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-04-14T15:11:32.886 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-04-14T15:11:32.886 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_thread_pool_size": "512", 2024-04-14T15:11:32.886 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_topic_persistency_max_retries": "0", 2024-04-14T15:11:32.886 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-04-14T15:11:32.886 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-04-14T15:11:32.886 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_topic_require_publish_policy": "false", 2024-04-14T15:11:32.886 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_torrent_comment": "", 2024-04-14T15:11:32.886 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_torrent_createby": "", 2024-04-14T15:11:32.886 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_torrent_encoding": "", 2024-04-14T15:11:32.886 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_torrent_flag": "false", 2024-04-14T15:11:32.887 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_torrent_max_size": "5368709120", 2024-04-14T15:11:32.887 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_torrent_origin": "", 2024-04-14T15:11:32.887 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_torrent_sha_unit": "524288", 2024-04-14T15:11:32.887 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_torrent_tracker": "", 2024-04-14T15:11:32.887 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_trust_forwarded_https": "false", 2024-04-14T15:11:32.887 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-04-14T15:11:32.887 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_usage_log_tick_interval": "30", 2024-04-14T15:11:32.887 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_usage_max_shards": "32", 2024-04-14T15:11:32.887 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_usage_max_user_shards": "1", 2024-04-14T15:11:32.887 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_use_opa_authz": "false", 2024-04-14T15:11:32.887 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_user_counters_cache": "false", 2024-04-14T15:11:32.888 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_user_counters_cache_size": "10000", 2024-04-14T15:11:32.888 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-04-14T15:11:32.888 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_user_default_quota_max_size": "-1", 2024-04-14T15:11:32.888 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_user_max_buckets": "1000", 2024-04-14T15:11:32.888 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_user_policies_max_num": "100", 2024-04-14T15:11:32.888 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-04-14T15:11:32.888 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-04-14T15:11:32.888 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_user_quota_sync_interval": "86400", 2024-04-14T15:11:32.888 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-04-14T15:11:32.888 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_user_unique_email": "true", 2024-04-14T15:11:32.888 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_verify_ssl": "true", 2024-04-14T15:11:32.888 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_website_routing_rules_max_num": "50", 2024-04-14T15:11:32.889 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_zone": "", 2024-04-14T15:11:32.889 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_zone_id": "", 2024-04-14T15:11:32.889 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-04-14T15:11:32.889 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_zonegroup": "", 2024-04-14T15:11:32.889 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_zonegroup_id": "", 2024-04-14T15:11:32.889 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-04-14T15:11:32.889 INFO:teuthology.orchestra.run.smithi090.stdout: "rgwlc_auto_session_clear": "true", 2024-04-14T15:11:32.889 INFO:teuthology.orchestra.run.smithi090.stdout: "rgwlc_skip_bucket_step": "false", 2024-04-14T15:11:32.889 INFO:teuthology.orchestra.run.smithi090.stdout: "rocksdb_block_size": "4096", 2024-04-14T15:11:32.889 INFO:teuthology.orchestra.run.smithi090.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-04-14T15:11:32.889 INFO:teuthology.orchestra.run.smithi090.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-04-14T15:11:32.890 INFO:teuthology.orchestra.run.smithi090.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-04-14T15:11:32.890 INFO:teuthology.orchestra.run.smithi090.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-04-14T15:11:32.890 INFO:teuthology.orchestra.run.smithi090.stdout: "rocksdb_cache_shard_bits": "4", 2024-04-14T15:11:32.890 INFO:teuthology.orchestra.run.smithi090.stdout: "rocksdb_cache_size": "536870912", 2024-04-14T15:11:32.890 INFO:teuthology.orchestra.run.smithi090.stdout: "rocksdb_cache_type": "binned_lru", 2024-04-14T15:11:32.890 INFO:teuthology.orchestra.run.smithi090.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-04-14T15:11:32.890 INFO:teuthology.orchestra.run.smithi090.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-04-14T15:11:32.890 INFO:teuthology.orchestra.run.smithi090.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-04-14T15:11:32.890 INFO:teuthology.orchestra.run.smithi090.stdout: "rocksdb_collect_compaction_stats": "false", 2024-04-14T15:11:32.890 INFO:teuthology.orchestra.run.smithi090.stdout: "rocksdb_collect_extended_stats": "false", 2024-04-14T15:11:32.890 INFO:teuthology.orchestra.run.smithi090.stdout: "rocksdb_collect_memory_stats": "false", 2024-04-14T15:11:32.890 INFO:teuthology.orchestra.run.smithi090.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-04-14T15:11:32.891 INFO:teuthology.orchestra.run.smithi090.stdout: "rocksdb_index_type": "binary_search", 2024-04-14T15:11:32.891 INFO:teuthology.orchestra.run.smithi090.stdout: "rocksdb_log_to_ceph_log": "true", 2024-04-14T15:11:32.891 INFO:teuthology.orchestra.run.smithi090.stdout: "rocksdb_metadata_block_size": "4096", 2024-04-14T15:11:32.891 INFO:teuthology.orchestra.run.smithi090.stdout: "rocksdb_partition_filters": "false", 2024-04-14T15:11:32.891 INFO:teuthology.orchestra.run.smithi090.stdout: "rocksdb_perf": "false", 2024-04-14T15:11:32.891 INFO:teuthology.orchestra.run.smithi090.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-04-14T15:11:32.891 INFO:teuthology.orchestra.run.smithi090.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-04-14T15:11:32.891 INFO:teuthology.orchestra.run.smithi090.stdout: "rotating_keys_renewal_timeout": "10", 2024-04-14T15:11:32.891 INFO:teuthology.orchestra.run.smithi090.stdout: "run_dir": "/var/run/ceph", 2024-04-14T15:11:32.891 INFO:teuthology.orchestra.run.smithi090.stdout: "seastore_block_create": "true", 2024-04-14T15:11:32.891 INFO:teuthology.orchestra.run.smithi090.stdout: "seastore_cache_lru_size": "67108864", 2024-04-14T15:11:32.891 INFO:teuthology.orchestra.run.smithi090.stdout: "seastore_cbjournal_size": "5368709120", 2024-04-14T15:11:32.891 INFO:teuthology.orchestra.run.smithi090.stdout: "seastore_data_delta_based_overwrite": "0", 2024-04-14T15:11:32.892 INFO:teuthology.orchestra.run.smithi090.stdout: "seastore_default_max_object_size": "16777216", 2024-04-14T15:11:32.892 INFO:teuthology.orchestra.run.smithi090.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-04-14T15:11:32.892 INFO:teuthology.orchestra.run.smithi090.stdout: "seastore_device_size": "53687091200", 2024-04-14T15:11:32.892 INFO:teuthology.orchestra.run.smithi090.stdout: "seastore_journal_batch_capacity": "16", 2024-04-14T15:11:32.892 INFO:teuthology.orchestra.run.smithi090.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-04-14T15:11:32.892 INFO:teuthology.orchestra.run.smithi090.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-04-14T15:11:32.892 INFO:teuthology.orchestra.run.smithi090.stdout: "seastore_journal_iodepth_limit": "5", 2024-04-14T15:11:32.892 INFO:teuthology.orchestra.run.smithi090.stdout: "seastore_main_device_type": "SSD", 2024-04-14T15:11:32.892 INFO:teuthology.orchestra.run.smithi090.stdout: "seastore_max_concurrent_transactions": "8", 2024-04-14T15:11:32.892 INFO:teuthology.orchestra.run.smithi090.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-04-14T15:11:32.892 INFO:teuthology.orchestra.run.smithi090.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-04-14T15:11:32.893 INFO:teuthology.orchestra.run.smithi090.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-04-14T15:11:32.893 INFO:teuthology.orchestra.run.smithi090.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-04-14T15:11:32.893 INFO:teuthology.orchestra.run.smithi090.stdout: "seastore_segment_size": "67108864", 2024-04-14T15:11:32.893 INFO:teuthology.orchestra.run.smithi090.stdout: "set_keepcaps": "false", 2024-04-14T15:11:32.893 INFO:teuthology.orchestra.run.smithi090.stdout: "setgroup": "ceph", 2024-04-14T15:11:32.893 INFO:teuthology.orchestra.run.smithi090.stdout: "setuser": "ceph", 2024-04-14T15:11:32.893 INFO:teuthology.orchestra.run.smithi090.stdout: "setuser_match_path": "", 2024-04-14T15:11:32.893 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_misplaced_ratio": "0.050000", 2024-04-14T15:11:32.893 INFO:teuthology.orchestra.run.smithi090.stdout: "thp": "false", 2024-04-14T15:11:32.893 INFO:teuthology.orchestra.run.smithi090.stdout: "threadpool_default_timeout": "60", 2024-04-14T15:11:32.893 INFO:teuthology.orchestra.run.smithi090.stdout: "threadpool_empty_queue_max_wait": "2", 2024-04-14T15:11:32.893 INFO:teuthology.orchestra.run.smithi090.stdout: "throttler_perf_counter": "true" 2024-04-14T15:11:32.893 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-14T15:11:33.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:33 smithi090 ceph-mon[41670]: 6.7 scrub starts 2024-04-14T15:11:33.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:33 smithi090 ceph-mon[41670]: 6.7 scrub ok 2024-04-14T15:11:33.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:33 smithi090 ceph-mon[30399]: 6.7 scrub starts 2024-04-14T15:11:33.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:33 smithi090 ceph-mon[30399]: 6.7 scrub ok 2024-04-14T15:11:33.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:33 smithi092 ceph-mon[33966]: 6.7 scrub starts 2024-04-14T15:11:33.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:33 smithi092 ceph-mon[33966]: 6.7 scrub ok 2024-04-14T15:11:33.772 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-04-14T15:11:34.121 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:11:33 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:11:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:11:34.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:34 smithi090 ceph-mon[41670]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:11:34.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:34 smithi090 ceph-mon[41670]: 2.15 scrub starts 2024-04-14T15:11:34.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:34 smithi090 ceph-mon[41670]: 2.15 scrub ok 2024-04-14T15:11:34.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:34 smithi090 ceph-mon[30399]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:11:34.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:34 smithi090 ceph-mon[30399]: 2.15 scrub starts 2024-04-14T15:11:34.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:34 smithi090 ceph-mon[30399]: 2.15 scrub ok 2024-04-14T15:11:34.585 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:11:34.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:34 smithi092 ceph-mon[33966]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:11:34.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:34 smithi092 ceph-mon[33966]: 2.15 scrub starts 2024-04-14T15:11:34.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:34 smithi092 ceph-mon[33966]: 2.15 scrub ok 2024-04-14T15:11:35.122 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:11:34 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:11:35.262 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:35 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:11:35.263 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:35 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:11:35.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:35 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:11:36.404 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:36 smithi090 ceph-mon[41670]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:11:36.404 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:36 smithi090 ceph-mon[41670]: 5.12 scrub starts 2024-04-14T15:11:36.404 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:36 smithi090 ceph-mon[41670]: 5.12 scrub ok 2024-04-14T15:11:36.404 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:36 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:36.404 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:36 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:36.405 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:36 smithi090 ceph-mon[30399]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:11:36.405 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:36 smithi090 ceph-mon[30399]: 5.12 scrub starts 2024-04-14T15:11:36.405 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:36 smithi090 ceph-mon[30399]: 5.12 scrub ok 2024-04-14T15:11:36.405 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:36 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:36.405 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:36 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:36.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:36 smithi092 ceph-mon[33966]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:11:36.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:36 smithi092 ceph-mon[33966]: 5.12 scrub starts 2024-04-14T15:11:36.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:36 smithi092 ceph-mon[33966]: 5.12 scrub ok 2024-04-14T15:11:36.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:36 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:36.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:36 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:36.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.977+0000 7f5bb8857700 1 -- 172.21.15.90:0/2919233025 >> v1:172.21.15.90:6789/0 conn(0x7f5bb4103a30 legacy=0x7f5bb4103e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:36.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.977+0000 7f5bb8857700 1 -- 172.21.15.90:0/2919233025 shutdown_connections 2024-04-14T15:11:36.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.977+0000 7f5bb8857700 1 -- 172.21.15.90:0/2919233025 >> 172.21.15.90:0/2919233025 conn(0x7f5bb40fd280 msgr2=0x7f5bb40ff6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:11:36.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.977+0000 7f5bb8857700 1 -- 172.21.15.90:0/2919233025 shutdown_connections 2024-04-14T15:11:36.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.977+0000 7f5bb8857700 1 -- 172.21.15.90:0/2919233025 wait complete. 2024-04-14T15:11:36.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.978+0000 7f5bb8857700 1 librados: starting msgr at 2024-04-14T15:11:36.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.978+0000 7f5bb8857700 1 librados: starting objecter 2024-04-14T15:11:36.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.978+0000 7f5bb8857700 1 Processor -- start 2024-04-14T15:11:36.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.978+0000 7f5bb8857700 1 -- start start 2024-04-14T15:11:36.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.978+0000 7f5bb8857700 1 librados: setting wanted keys 2024-04-14T15:11:36.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.978+0000 7f5bb8857700 1 librados: calling monclient init 2024-04-14T15:11:36.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.979+0000 7f5bb8857700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5bb410b510 con 0x7f5bb410b840 2024-04-14T15:11:36.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.979+0000 7f5bb8857700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5bb41a8ed0 con 0x7f5bb4103a30 2024-04-14T15:11:36.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.979+0000 7f5bb8857700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5bb41aa0b0 con 0x7f5bb410f350 2024-04-14T15:11:36.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.979+0000 7f5bb1d9b700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f5bb410b840 0x7f5bb4108a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:34914/0 (socket says 172.21.15.90:34914) 2024-04-14T15:11:36.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.979+0000 7f5bb1d9b700 1 -- 172.21.15.90:0/848439010 learned_addr learned my addr 172.21.15.90:0/848439010 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:11:36.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.979+0000 7f5bb2d9d700 1 --1- 172.21.15.90:0/848439010 >> v1:172.21.15.90:6790/0 conn(0x7f5bb410f350 0x7f5bb41a67a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:50826/0 (socket says 172.21.15.90:50826) 2024-04-14T15:11:36.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.979+0000 7f5ba37fe700 10 client.?.objecter ms_handle_connect 0x7f5bb4103a30 2024-04-14T15:11:36.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.979+0000 7f5ba37fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:11:36.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.979+0000 7f5ba37fe700 10 client.?.objecter ms_handle_connect 0x7f5bb410f350 2024-04-14T15:11:36.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.979+0000 7f5ba37fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:11:36.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.979+0000 7f5ba37fe700 10 client.?.objecter ms_handle_connect 0x7f5bb410b840 2024-04-14T15:11:36.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.979+0000 7f5ba37fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:11:36.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.979+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2256699554 0 0) 0x7f5bb41a8ed0 con 0x7f5bb4103a30 2024-04-14T15:11:36.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.979+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5b9c0035b0 con 0x7f5bb4103a30 2024-04-14T15:11:36.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.979+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1124242465 0 0) 0x7f5bb410b510 con 0x7f5bb410b840 2024-04-14T15:11:36.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.979+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5bb41a8ed0 con 0x7f5bb410b840 2024-04-14T15:11:36.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.979+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1021197594 0 0) 0x7f5bb41aa0b0 con 0x7f5bb410f350 2024-04-14T15:11:36.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.979+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5bb410b510 con 0x7f5bb410f350 2024-04-14T15:11:36.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.979+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1789101589 0 0) 0x7f5b9c0035b0 con 0x7f5bb4103a30 2024-04-14T15:11:36.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.979+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5bb41aa0b0 con 0x7f5bb4103a30 2024-04-14T15:11:36.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.979+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f5ba40015d0 con 0x7f5bb4103a30 2024-04-14T15:11:36.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.979+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3063780359 0 0) 0x7f5bb410b510 con 0x7f5bb410f350 2024-04-14T15:11:36.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.979+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5b9c0035b0 con 0x7f5bb410f350 2024-04-14T15:11:36.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.979+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1038877452 0 0) 0x7f5bb41a8ed0 con 0x7f5bb410b840 2024-04-14T15:11:36.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.979+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5bb410b510 con 0x7f5bb410b840 2024-04-14T15:11:36.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.979+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f5bac001bb0 con 0x7f5bb410f350 2024-04-14T15:11:36.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.980+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f5ba80017f0 con 0x7f5bb410b840 2024-04-14T15:11:36.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.980+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1535156366 0 0) 0x7f5bb41aa0b0 con 0x7f5bb4103a30 2024-04-14T15:11:36.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.980+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 >> v1:172.21.15.90:6790/0 conn(0x7f5bb410f350 legacy=0x7f5bb41a67a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:36.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.980+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 >> v1:172.21.15.90:6789/0 conn(0x7f5bb410b840 legacy=0x7f5bb4108a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:36.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.980+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bb41ab290 con 0x7f5bb4103a30 2024-04-14T15:11:36.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.980+0000 7f5bb8857700 1 -- 172.21.15.90:0/848439010 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5bb41aa2e0 con 0x7f5bb4103a30 2024-04-14T15:11:36.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.980+0000 7f5bb8857700 10 client.24685.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:11:36.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.980+0000 7f5bb8857700 1 -- 172.21.15.90:0/848439010 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f5bb41aa7a0 con 0x7f5bb4103a30 2024-04-14T15:11:36.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.980+0000 7f5bb8857700 1 librados: init done 2024-04-14T15:11:36.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.980+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f5ba4001b70 con 0x7f5bb4103a30 2024-04-14T15:11:36.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.980+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f5ba4004e60 con 0x7f5bb4103a30 2024-04-14T15:11:36.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.981+0000 7f5bb8857700 1 -- 172.21.15.90:0/848439010 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f5b94005140 con 0x7f5bb4103a30 2024-04-14T15:11:36.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.982+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f5ba4003ab0 con 0x7f5bb4103a30 2024-04-14T15:11:36.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.982+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(91..91 src has 41..91) ==== 6288+0+0 (unknown 352955974 0 0) 0x7f5ba4089390 con 0x7f5bb4103a30 2024-04-14T15:11:36.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.982+0000 7f5ba37fe700 10 client.24685.objecter ms_dispatch 0x7f5bb4001550 osd_map(91..91 src has 41..91) 2024-04-14T15:11:36.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.982+0000 7f5ba37fe700 3 client.24685.objecter handle_osd_map got epochs [91,91] > 0 2024-04-14T15:11:36.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.982+0000 7f5ba37fe700 3 client.24685.objecter handle_osd_map decoding full epoch 91 2024-04-14T15:11:36.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.983+0000 7f5ba37fe700 20 client.24685.objecter dump_active .. 0 homeless 2024-04-14T15:11:36.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.983+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1124 (unknown 409899127 0 1008968738) 0x7f5ba4056500 con 0x7f5bb4103a30 2024-04-14T15:11:36.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.983+0000 7f5ba37fe700 10 client.24685.objecter ms_handle_connect 0x7f5b9c0714d0 2024-04-14T15:11:36.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.983+0000 7f5bb8857700 1 -- 172.21.15.90:0/848439010 --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 3) -- 0x7f5b94004dc0 con 0x7f5b940027a0 2024-04-14T15:11:36.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.983+0000 7f5ba37fe700 10 client.24685.objecter ms_handle_connect 0x7f5b940027a0 2024-04-14T15:11:36.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.983+0000 7f5ba37fe700 10 client.24685.objecter resend_mon_ops 2024-04-14T15:11:36.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.983+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3801133726 0 0) 0x7f5b94004dc0 con 0x7f5b940027a0 2024-04-14T15:11:36.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.983+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5bb41aa0b0 con 0x7f5b940027a0 2024-04-14T15:11:36.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.983+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2346119170 0 0) 0x7f5bb41aa0b0 con 0x7f5b940027a0 2024-04-14T15:11:36.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.983+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5b94004dc0 con 0x7f5b940027a0 2024-04-14T15:11:36.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.983+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f5bac003a20 con 0x7f5b940027a0 2024-04-14T15:11:36.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.984+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2937873729 0 0) 0x7f5b94004dc0 con 0x7f5b940027a0 2024-04-14T15:11:36.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.984+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 --> v1:172.21.15.90:6789/0 -- command(tid 2: {"prefix": "get_command_descriptions"}) -- 0x7f5b9400a740 con 0x7f5b940027a0 2024-04-14T15:11:36.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.984+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 <== mon.0 v1:172.21.15.90:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f5bac005bd0 con 0x7f5b940027a0 2024-04-14T15:11:36.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.984+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 >> v1:172.21.15.90:6789/0 conn(0x7f5b940027a0 legacy=0x7f5b940056f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:36.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.984+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 >> v1:172.21.15.90:6789/0 conn(0x7f5b940027a0 legacy=0x7f5b940056f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-14T15:11:36.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.991+0000 7f5bb8857700 1 -- 172.21.15.90:0/848439010 --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 3) -- 0x7f5b9400af20 con 0x7f5b9400aae0 2024-04-14T15:11:36.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.991+0000 7f5ba37fe700 10 client.24685.objecter ms_handle_connect 0x7f5b9400aae0 2024-04-14T15:11:36.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.991+0000 7f5ba37fe700 10 client.24685.objecter resend_mon_ops 2024-04-14T15:11:36.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.991+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2596972644 0 0) 0x7f5b9400af20 con 0x7f5b9400aae0 2024-04-14T15:11:36.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.991+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5b94004dc0 con 0x7f5b9400aae0 2024-04-14T15:11:36.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.992+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3474033043 0 0) 0x7f5b94004dc0 con 0x7f5b9400aae0 2024-04-14T15:11:36.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.992+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5b9400af20 con 0x7f5b9400aae0 2024-04-14T15:11:36.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.992+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f5bac003fd0 con 0x7f5b9400aae0 2024-04-14T15:11:36.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.992+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 785641655 0 0) 0x7f5b9400af20 con 0x7f5b9400aae0 2024-04-14T15:11:36.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.992+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 --> v1:172.21.15.90:6789/0 -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=0"]}) -- 0x7f5b9400c180 con 0x7f5b9400aae0 2024-04-14T15:11:36.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.993+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 <== mon.0 v1:172.21.15.90:6789/0 5 ==== command_reply(tid 3: 0 mon_osd_down_out_interval = '' (not observed, change may require restart) ) ==== 82+0+3 (unknown 285646418 0 2423995997) 0x7f5bac004290 con 0x7f5b9400aae0 2024-04-14T15:11:36.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.993+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 >> v1:172.21.15.90:6789/0 conn(0x7f5b9400aae0 legacy=0x7f5b94008e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:36.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.993+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 >> v1:172.21.15.90:6789/0 conn(0x7f5b9400aae0 legacy=0x7f5b94008e20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-14T15:11:36.994 INFO:teuthology.orchestra.run.smithi090.stdout:mon.a: {} 2024-04-14T15:11:36.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.994+0000 7f5bb8857700 1 -- 172.21.15.90:0/848439010 --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 3) -- 0x7f5b94002300 con 0x7f5b94001ed0 2024-04-14T15:11:36.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.994+0000 7f5ba37fe700 10 client.24685.objecter ms_handle_connect 0x7f5b94001ed0 2024-04-14T15:11:36.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.994+0000 7f5ba37fe700 10 client.24685.objecter resend_mon_ops 2024-04-14T15:11:36.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.994+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 609702935 0 0) 0x7f5b9c0035b0 con 0x7f5b94001ed0 2024-04-14T15:11:36.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.994+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5b9400af20 con 0x7f5b94001ed0 2024-04-14T15:11:36.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.994+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1353938394 0 0) 0x7f5b9400af20 con 0x7f5b94001ed0 2024-04-14T15:11:36.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.994+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5b9c0035b0 con 0x7f5b94001ed0 2024-04-14T15:11:36.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.994+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f5bac004850 con 0x7f5b94001ed0 2024-04-14T15:11:36.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.994+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2549610567 0 0) 0x7f5b9c0035b0 con 0x7f5b94001ed0 2024-04-14T15:11:36.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.995+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 --> v1:172.21.15.92:6789/0 -- command(tid 4: {"prefix": "get_command_descriptions"}) -- 0x7f5b94012ce0 con 0x7f5b94001ed0 2024-04-14T15:11:36.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.995+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 <== mon.1 v1:172.21.15.92:6789/0 5 ==== command_reply(tid 4: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f5bac008170 con 0x7f5b94001ed0 2024-04-14T15:11:36.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.995+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 >> v1:172.21.15.92:6789/0 conn(0x7f5b94001ed0 legacy=0x7f5b9400b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:36.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.995+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 >> v1:172.21.15.92:6789/0 conn(0x7f5b94001ed0 legacy=0x7f5b9400b8c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-14T15:11:36.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:36.995+0000 7f5bb2d9d700 1 -- 172.21.15.90:0/848439010 reap_dead start 2024-04-14T15:11:37.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:37.002+0000 7f5bb8857700 1 -- 172.21.15.90:0/848439010 --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 3) -- 0x7f5b94012ae0 con 0x7f5b94013070 2024-04-14T15:11:37.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:37.002+0000 7f5ba37fe700 10 client.24685.objecter ms_handle_connect 0x7f5b94013070 2024-04-14T15:11:37.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:37.002+0000 7f5ba37fe700 10 client.24685.objecter resend_mon_ops 2024-04-14T15:11:37.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:37.003+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3966439698 0 0) 0x7f5b94012ae0 con 0x7f5b94013070 2024-04-14T15:11:37.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:37.003+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5b9c0035b0 con 0x7f5b94013070 2024-04-14T15:11:37.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:37.003+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2890673030 0 0) 0x7f5b9c0035b0 con 0x7f5b94013070 2024-04-14T15:11:37.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:37.003+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5b94012ae0 con 0x7f5b94013070 2024-04-14T15:11:37.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:37.003+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f5bac0062c0 con 0x7f5b94013070 2024-04-14T15:11:37.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:37.003+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2759721943 0 0) 0x7f5b94012ae0 con 0x7f5b94013070 2024-04-14T15:11:37.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:37.003+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 --> v1:172.21.15.92:6789/0 -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=0"]}) -- 0x7f5b94014540 con 0x7f5b94013070 2024-04-14T15:11:37.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:37.005+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 <== mon.1 v1:172.21.15.92:6789/0 5 ==== command_reply(tid 5: 0 mon_osd_down_out_interval = '' (not observed, change may require restart) ) ==== 82+0+3 (unknown 285646418 0 2423995997) 0x7f5bac0066a0 con 0x7f5b94013070 2024-04-14T15:11:37.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:37.005+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 >> v1:172.21.15.92:6789/0 conn(0x7f5b94013070 legacy=0x7f5b940113c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:37.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:37.005+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 >> v1:172.21.15.92:6789/0 conn(0x7f5b94013070 legacy=0x7f5b940113c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-14T15:11:37.005 INFO:teuthology.orchestra.run.smithi090.stdout:mon.b: {} 2024-04-14T15:11:37.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:37.005+0000 7f5bb8857700 1 -- 172.21.15.90:0/848439010 --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 3) -- 0x7f5b94014240 con 0x7f5b940135e0 2024-04-14T15:11:37.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:37.005+0000 7f5ba37fe700 10 client.24685.objecter ms_handle_connect 0x7f5b940135e0 2024-04-14T15:11:37.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:37.005+0000 7f5ba37fe700 10 client.24685.objecter resend_mon_ops 2024-04-14T15:11:37.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:37.005+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 5377532 0 0) 0x7f5b94014240 con 0x7f5b940135e0 2024-04-14T15:11:37.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:37.005+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5b94012ae0 con 0x7f5b940135e0 2024-04-14T15:11:37.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:37.006+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 367351556 0 0) 0x7f5b94012ae0 con 0x7f5b940135e0 2024-04-14T15:11:37.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:37.006+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5b94014240 con 0x7f5b940135e0 2024-04-14T15:11:37.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:37.006+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f5bac006be0 con 0x7f5b940135e0 2024-04-14T15:11:37.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:37.006+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 <== mon.2 v1:172.21.15.90:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2058638322 0 0) 0x7f5b94014240 con 0x7f5b940135e0 2024-04-14T15:11:37.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:37.006+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 --> v1:172.21.15.90:6790/0 -- command(tid 6: {"prefix": "get_command_descriptions"}) -- 0x7f5b9401b150 con 0x7f5b940135e0 2024-04-14T15:11:37.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:37.006+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 <== mon.2 v1:172.21.15.90:6790/0 5 ==== command_reply(tid 6: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f5bac008320 con 0x7f5b940135e0 2024-04-14T15:11:37.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:37.006+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 >> v1:172.21.15.90:6790/0 conn(0x7f5b940135e0 legacy=0x7f5b940139c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:37.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:37.006+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 >> v1:172.21.15.90:6790/0 conn(0x7f5b940135e0 legacy=0x7f5b940139c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-14T15:11:37.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:37.020+0000 7f5bb8857700 1 -- 172.21.15.90:0/848439010 --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 3) -- 0x7f5b94018fb0 con 0x7f5b9401b390 2024-04-14T15:11:37.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:37.021+0000 7f5ba37fe700 10 client.24685.objecter ms_handle_connect 0x7f5b9401b390 2024-04-14T15:11:37.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:37.021+0000 7f5ba37fe700 10 client.24685.objecter resend_mon_ops 2024-04-14T15:11:37.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:37.021+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3166914867 0 0) 0x7f5b94018fb0 con 0x7f5b9401b390 2024-04-14T15:11:37.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:37.021+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5b94014240 con 0x7f5b9401b390 2024-04-14T15:11:37.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:37.021+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1964291533 0 0) 0x7f5b94014240 con 0x7f5b9401b390 2024-04-14T15:11:37.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:37.021+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5b94018fb0 con 0x7f5b9401b390 2024-04-14T15:11:37.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:37.021+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f5bac0071a0 con 0x7f5b9401b390 2024-04-14T15:11:37.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:37.021+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 <== mon.2 v1:172.21.15.90:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3323388634 0 0) 0x7f5b94018fb0 con 0x7f5b9401b390 2024-04-14T15:11:37.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:37.021+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 --> v1:172.21.15.90:6790/0 -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=0"]}) -- 0x7f5b9401a270 con 0x7f5b9401b390 2024-04-14T15:11:37.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:37.023+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 <== mon.2 v1:172.21.15.90:6790/0 5 ==== command_reply(tid 7: 0 mon_osd_down_out_interval = '' (not observed, change may require restart) ) ==== 82+0+3 (unknown 285646418 0 2423995997) 0x7f5bac0084d0 con 0x7f5b9401b390 2024-04-14T15:11:37.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:37.023+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 >> v1:172.21.15.90:6790/0 conn(0x7f5b9401b390 legacy=0x7f5b94017770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:37.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:37.023+0000 7f5ba37fe700 1 -- 172.21.15.90:0/848439010 >> v1:172.21.15.90:6790/0 conn(0x7f5b9401b390 legacy=0x7f5b94017770 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-14T15:11:37.023 INFO:teuthology.orchestra.run.smithi090.stdout:mon.c: {} 2024-04-14T15:11:37.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:37.023+0000 7f5bb8857700 10 librados: watch_flush enter 2024-04-14T15:11:37.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:37.023+0000 7f5bb8857700 10 librados: watch_flush exit 2024-04-14T15:11:37.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:37.023+0000 7f5bb8857700 20 client.24685.objecter shutdown clearing up homeless session... 2024-04-14T15:11:37.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:37.023+0000 7f5bb8857700 10 client.24685.objecter successfully canceled tick 2024-04-14T15:11:37.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:37.023+0000 7f5bb8857700 1 -- 172.21.15.90:0/848439010 >> v1:172.21.15.90:6800/420935427 conn(0x7f5b9c0714d0 legacy=0x7f5b9c073990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:37.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:37.023+0000 7f5bb8857700 1 -- 172.21.15.90:0/848439010 >> v1:172.21.15.92:6789/0 conn(0x7f5bb4103a30 legacy=0x7f5bb4108360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:37.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:37.023+0000 7f5bb2d9d700 1 -- 172.21.15.90:0/848439010 reap_dead start 2024-04-14T15:11:37.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:37.023+0000 7f5bb8857700 1 -- 172.21.15.90:0/848439010 shutdown_connections 2024-04-14T15:11:37.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:37.023+0000 7f5bb8857700 1 -- 172.21.15.90:0/848439010 >> 172.21.15.90:0/848439010 conn(0x7f5bb40fd280 msgr2=0x7f5bb4103e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:11:37.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:37.023+0000 7f5bb8857700 1 -- 172.21.15.90:0/848439010 shutdown_connections 2024-04-14T15:11:37.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:37.023+0000 7f5bb8857700 1 -- 172.21.15.90:0/848439010 wait complete. 2024-04-14T15:11:37.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:37.023+0000 7f5bb8857700 1 librados: shutdown 2024-04-14T15:11:37.025 INFO:teuthology.orchestra.run.smithi090.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-14T15:11:37.025 INFO:teuthology.orchestra.run.smithi090.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-14T15:11:37.025 INFO:teuthology.orchestra.run.smithi090.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-14T15:11:37.410 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:37 smithi090 ceph-mon[41670]: 6.1 scrub starts 2024-04-14T15:11:37.411 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:37 smithi090 ceph-mon[41670]: 6.1 scrub ok 2024-04-14T15:11:37.411 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:37 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:11:37.411 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:37 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:11:37.411 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:37 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:37.411 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:37 smithi090 ceph-mon[41670]: Reconfiguring alertmanager.a (dependencies changed)... 2024-04-14T15:11:37.411 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:37 smithi090 ceph-mon[41670]: Reconfiguring daemon alertmanager.a on smithi090 2024-04-14T15:11:37.411 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:37 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/848439010' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-14T15:11:37.411 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:37 smithi090 ceph-mon[30399]: 6.1 scrub starts 2024-04-14T15:11:37.411 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:37 smithi090 ceph-mon[30399]: 6.1 scrub ok 2024-04-14T15:11:37.411 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:37 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:11:37.411 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:37 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:11:37.412 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:37 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:37.412 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:37 smithi090 ceph-mon[30399]: Reconfiguring alertmanager.a (dependencies changed)... 2024-04-14T15:11:37.412 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:37 smithi090 ceph-mon[30399]: Reconfiguring daemon alertmanager.a on smithi090 2024-04-14T15:11:37.412 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:37 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/848439010' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-14T15:11:37.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:37 smithi092 ceph-mon[33966]: 6.1 scrub starts 2024-04-14T15:11:37.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:37 smithi092 ceph-mon[33966]: 6.1 scrub ok 2024-04-14T15:11:37.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:37 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:11:37.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:37 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:11:37.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:37 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:37.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:37 smithi092 ceph-mon[33966]: Reconfiguring alertmanager.a (dependencies changed)... 2024-04-14T15:11:37.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:37 smithi092 ceph-mon[33966]: Reconfiguring daemon alertmanager.a on smithi090 2024-04-14T15:11:37.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:37 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/848439010' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-14T15:11:38.026 INFO:teuthology.run_tasks:Running task workunit... 2024-04-14T15:11:38.039 INFO:tasks.workunit:Pulling workunits from ref 05c9d46e7de5db2cb434c177a6bbfc56f472d09b 2024-04-14T15:11:38.040 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-14T15:11:38.040 DEBUG:teuthology.orchestra.run.smithi090:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-14T15:11:38.040 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-04-14T15:11:38.040 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-14T15:11:38.040 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-14T15:11:38.041 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-04-14T15:11:38.041 INFO:tasks.thrashosds.thrasher:Setting osd 3 primary_affinity to 0.217857 2024-04-14T15:11:38.041 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd primary-affinity 3 0.21785660735935053 2024-04-14T15:11:38.041 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-04-14T15:11:38.041 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:11:38.042 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-04-14T15:11:38.042 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-14T15:11:38.042 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-04-14T15:11:38.042 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-14T15:11:38.043 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-04-14T15:11:38.043 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd set noscrub 2024-04-14T15:11:38.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:38.061 INFO:teuthology.orchestra.run.smithi090.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-04-14T15:11:38.061 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-04-14T15:11:38.072 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:38.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:38.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:11:38.092 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-04-14T15:11:38.093 DEBUG:teuthology.orchestra.run.smithi090:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-04-14T15:11:38.153 INFO:tasks.workunit:timeout=3h 2024-04-14T15:11:38.153 INFO:tasks.workunit:cleanup=True 2024-04-14T15:11:38.154 DEBUG:teuthology.orchestra.run.smithi090:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 05c9d46e7de5db2cb434c177a6bbfc56f472d09b 2024-04-14T15:11:38.173 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:11:38.202 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:38.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:38.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:11:38.208 INFO:tasks.workunit.client.0.smithi090.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-14T15:11:38.267 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:38 smithi090 ceph-mon[41670]: 5.1 scrub starts 2024-04-14T15:11:38.267 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:38 smithi090 ceph-mon[41670]: 5.1 scrub ok 2024-04-14T15:11:38.267 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:38 smithi090 ceph-mon[41670]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 915 B/s rd, 0 op/s 2024-04-14T15:11:38.267 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:38 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:11:38.267 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:38 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:11:38.267 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:38 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:11:38.267 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:38 smithi090 ceph-mon[30399]: 5.1 scrub starts 2024-04-14T15:11:38.267 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:38 smithi090 ceph-mon[30399]: 5.1 scrub ok 2024-04-14T15:11:38.268 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:38 smithi090 ceph-mon[30399]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 915 B/s rd, 0 op/s 2024-04-14T15:11:38.268 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:38 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:11:38.268 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:38 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:11:38.268 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:38 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:11:38.304 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:11:38.331 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:38.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:38.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:11:38.433 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:11:38.458 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:38.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:38.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:11:38.559 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:11:38.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:38.588 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:38.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:11:38.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:38 smithi092 ceph-mon[33966]: 5.1 scrub starts 2024-04-14T15:11:38.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:38 smithi092 ceph-mon[33966]: 5.1 scrub ok 2024-04-14T15:11:38.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:38 smithi092 ceph-mon[33966]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 915 B/s rd, 0 op/s 2024-04-14T15:11:38.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:38 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:11:38.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:38 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:11:38.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:38 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:11:38.689 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:11:38.715 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:38.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:38.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:11:38.818 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:11:38.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:38.846 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:38.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:11:38.947 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:11:38.974 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:38.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:38.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:11:39.075 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:11:39.084 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:11:39.103 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:39.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:39.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:11:39.204 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:11:39.231 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:39.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:39.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:11:39.332 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:11:39.357 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:39.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:39.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:11:39.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:39 smithi090 ceph-mon[41670]: 6.11 scrub starts 2024-04-14T15:11:39.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:39 smithi090 ceph-mon[41670]: 6.11 scrub ok 2024-04-14T15:11:39.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:39 smithi090 ceph-mon[41670]: 2.6 deep-scrub starts 2024-04-14T15:11:39.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:39 smithi090 ceph-mon[41670]: 2.6 deep-scrub ok 2024-04-14T15:11:39.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:39 smithi090 ceph-mon[41670]: osdmap e92: 8 total, 8 up, 8 in 2024-04-14T15:11:39.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:39 smithi090 ceph-mon[30399]: 6.11 scrub starts 2024-04-14T15:11:39.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:39 smithi090 ceph-mon[30399]: 6.11 scrub ok 2024-04-14T15:11:39.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:39 smithi090 ceph-mon[30399]: 2.6 deep-scrub starts 2024-04-14T15:11:39.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:39 smithi090 ceph-mon[30399]: 2.6 deep-scrub ok 2024-04-14T15:11:39.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:39 smithi090 ceph-mon[30399]: osdmap e92: 8 total, 8 up, 8 in 2024-04-14T15:11:39.459 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:11:39.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:39.484 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:39.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:11:39.585 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:11:39.612 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:39.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:39.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:11:39.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:39 smithi092 ceph-mon[33966]: 6.11 scrub starts 2024-04-14T15:11:39.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:39 smithi092 ceph-mon[33966]: 6.11 scrub ok 2024-04-14T15:11:39.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:39 smithi092 ceph-mon[33966]: 2.6 deep-scrub starts 2024-04-14T15:11:39.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:39 smithi092 ceph-mon[33966]: 2.6 deep-scrub ok 2024-04-14T15:11:39.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:39 smithi092 ceph-mon[33966]: osdmap e92: 8 total, 8 up, 8 in 2024-04-14T15:11:39.714 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:11:39.737 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:39.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:39.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:11:39.839 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:11:39.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:39.867 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:39.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:11:39.968 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:11:39.999 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:39.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:40.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:11:40.100 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:11:40.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:40.124 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:40.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:11:40.225 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:11:40.252 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:40.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:40.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:11:40.354 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:11:40.380 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:40.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:40.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:11:40.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:40 smithi090 ceph-mon[41670]: 2.f scrub starts 2024-04-14T15:11:40.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:40 smithi090 ceph-mon[41670]: 2.f scrub ok 2024-04-14T15:11:40.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:40 smithi090 ceph-mon[41670]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:11:40.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:40 smithi090 ceph-mon[30399]: 2.f scrub starts 2024-04-14T15:11:40.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:40 smithi090 ceph-mon[30399]: 2.f scrub ok 2024-04-14T15:11:40.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:40 smithi090 ceph-mon[30399]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:11:40.482 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:11:40.506 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:40.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:40.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:11:40.608 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:11:40.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:40 smithi092 ceph-mon[33966]: 2.f scrub starts 2024-04-14T15:11:40.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:40 smithi092 ceph-mon[33966]: 2.f scrub ok 2024-04-14T15:11:40.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:40 smithi092 ceph-mon[33966]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:11:40.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:40.634 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:40.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:11:40.735 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:11:40.762 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:40.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:40.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:11:40.864 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:11:40.891 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:40.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:40.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:11:40.993 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:11:41.017 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:41.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:41.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:11:41.118 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:11:41.146 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:41.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:41.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:11:41.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:41 smithi090 ceph-mon[41670]: 4.16 scrub starts 2024-04-14T15:11:41.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:41 smithi090 ceph-mon[41670]: 4.16 scrub ok 2024-04-14T15:11:41.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:41 smithi090 ceph-mon[30399]: 4.16 scrub starts 2024-04-14T15:11:41.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:41 smithi090 ceph-mon[30399]: 4.16 scrub ok 2024-04-14T15:11:41.248 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:11:41.272 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:41.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:41.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:11:41.374 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:11:41.401 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:41.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:41.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:11:41.503 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:11:41.527 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:41.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:41.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:11:41.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:41 smithi092 ceph-mon[33966]: 4.16 scrub starts 2024-04-14T15:11:41.686 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:41 smithi092 ceph-mon[33966]: 4.16 scrub ok 2024-04-14T15:11:41.686 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:11:41.700 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:11:41.713 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:41.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:41.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:11:41.815 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:11:41.843 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:41.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:41.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:11:41.945 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:11:41.960 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:11:41 smithi090 systemd[1]: Stopping Ceph alertmanager.a for 0e288098-fa70-11ee-bc8e-c7b262605968... 2024-04-14T15:11:41.970 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:41.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:41.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:11:42.071 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:11:42.095 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:42.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:42.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:11:42.197 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:11:42.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:42.226 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:42.226 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:11:42.327 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:11:42.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:42.354 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:42.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:11:42.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:42 smithi090 ceph-mon[41670]: 5.0 scrub starts 2024-04-14T15:11:42.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:42 smithi090 ceph-mon[41670]: 5.0 scrub ok 2024-04-14T15:11:42.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:42 smithi090 ceph-mon[41670]: 6.16 scrub starts 2024-04-14T15:11:42.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:42 smithi090 ceph-mon[41670]: 6.16 scrub ok 2024-04-14T15:11:42.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:42 smithi090 ceph-mon[41670]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:11:42.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:42 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:11:42.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:42 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:11:42.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:42 smithi090 ceph-mon[30399]: 5.0 scrub starts 2024-04-14T15:11:42.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:42 smithi090 ceph-mon[30399]: 5.0 scrub ok 2024-04-14T15:11:42.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:42 smithi090 ceph-mon[30399]: 6.16 scrub starts 2024-04-14T15:11:42.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:42 smithi090 ceph-mon[30399]: 6.16 scrub ok 2024-04-14T15:11:42.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:42 smithi090 ceph-mon[30399]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:11:42.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:42 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:11:42.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:42 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:11:42.455 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:11:42.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:42.483 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:42.484 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:11:42.584 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:11:42.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:42.609 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:42.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:11:42.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:42 smithi092 ceph-mon[33966]: 5.0 scrub starts 2024-04-14T15:11:42.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:42 smithi092 ceph-mon[33966]: 5.0 scrub ok 2024-04-14T15:11:42.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:42 smithi092 ceph-mon[33966]: 6.16 scrub starts 2024-04-14T15:11:42.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:42 smithi092 ceph-mon[33966]: 6.16 scrub ok 2024-04-14T15:11:42.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:42 smithi092 ceph-mon[33966]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:11:42.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:42 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:11:42.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:42 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:11:42.710 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:11:42.733 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:42.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:42.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:11:42.835 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:11:42.859 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:42.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:42.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:11:42.961 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:11:42.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:42.986 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:42.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:11:43.087 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:11:43.116 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:43.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:43.117 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:11:43.218 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:11:43.243 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:43.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:43.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:11:43.309 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:43 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:11:43.309 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:43 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:11:43.309 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:43 smithi090 ceph-mon[41670]: 6.2 scrub starts 2024-04-14T15:11:43.309 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:43 smithi090 ceph-mon[41670]: 6.2 scrub ok 2024-04-14T15:11:43.309 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:43 smithi090 ceph-mon[41670]: 2.4 scrub starts 2024-04-14T15:11:43.309 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:43 smithi090 ceph-mon[41670]: 2.4 scrub ok 2024-04-14T15:11:43.309 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:43 smithi090 ceph-mon[41670]: osdmap e93: 8 total, 8 up, 8 in 2024-04-14T15:11:43.309 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:43 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:11:43.309 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:43 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:11:43.309 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:43 smithi090 ceph-mon[30399]: 6.2 scrub starts 2024-04-14T15:11:43.310 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:43 smithi090 ceph-mon[30399]: 6.2 scrub ok 2024-04-14T15:11:43.310 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:43 smithi090 ceph-mon[30399]: 2.4 scrub starts 2024-04-14T15:11:43.310 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:43 smithi090 ceph-mon[30399]: 2.4 scrub ok 2024-04-14T15:11:43.310 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:43 smithi090 ceph-mon[30399]: osdmap e93: 8 total, 8 up, 8 in 2024-04-14T15:11:43.345 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:11:43.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:43.371 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:43.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:11:43.472 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:11:43.498 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:43.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:43.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:11:43.601 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:11:43.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:43 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:11:43.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:43 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:11:43.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:43 smithi092 ceph-mon[33966]: 6.2 scrub starts 2024-04-14T15:11:43.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:43 smithi092 ceph-mon[33966]: 6.2 scrub ok 2024-04-14T15:11:43.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:43 smithi092 ceph-mon[33966]: 2.4 scrub starts 2024-04-14T15:11:43.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:43 smithi092 ceph-mon[33966]: 2.4 scrub ok 2024-04-14T15:11:43.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:43 smithi092 ceph-mon[33966]: osdmap e93: 8 total, 8 up, 8 in 2024-04-14T15:11:43.627 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:43.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:43.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:11:43.729 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:11:43.752 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:43.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:43.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:11:43.854 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:11:43.878 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:43.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:43.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:11:43.980 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:11:44.008 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:44.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:44.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:11:44.110 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:11:44.111 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:11:43 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:11:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:11:44.135 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:44.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:44.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:11:44.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.178+0000 7f435f2aa700 1 -- 172.21.15.90:0/375146470 >> v1:172.21.15.90:6789/0 conn(0x7f435810b8b0 legacy=0x7f435810dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:44.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.178+0000 7f435f2aa700 1 -- 172.21.15.90:0/375146470 shutdown_connections 2024-04-14T15:11:44.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.178+0000 7f435f2aa700 1 -- 172.21.15.90:0/375146470 >> 172.21.15.90:0/375146470 conn(0x7f43580fd290 msgr2=0x7f43580ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:11:44.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.178+0000 7f435f2aa700 1 -- 172.21.15.90:0/375146470 shutdown_connections 2024-04-14T15:11:44.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.178+0000 7f435f2aa700 1 -- 172.21.15.90:0/375146470 wait complete. 2024-04-14T15:11:44.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.179+0000 7f435f2aa700 1 librados: starting msgr at 2024-04-14T15:11:44.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.179+0000 7f435f2aa700 1 librados: starting objecter 2024-04-14T15:11:44.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.179+0000 7f435f2aa700 1 Processor -- start 2024-04-14T15:11:44.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.179+0000 7f435f2aa700 1 -- start start 2024-04-14T15:11:44.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.179+0000 7f435f2aa700 1 librados: setting wanted keys 2024-04-14T15:11:44.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.179+0000 7f435f2aa700 1 librados: calling monclient init 2024-04-14T15:11:44.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.179+0000 7f435f2aa700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f435810b5b0 con 0x7f435810b8b0 2024-04-14T15:11:44.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.179+0000 7f435f2aa700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f43581a8ec0 con 0x7f4358103aa0 2024-04-14T15:11:44.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.179+0000 7f435f2aa700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f43581aa0a0 con 0x7f435810f3c0 2024-04-14T15:11:44.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.179+0000 7f435c845700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f435810b8b0 0x7f4358108b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:40132/0 (socket says 172.21.15.90:40132) 2024-04-14T15:11:44.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.179+0000 7f435c845700 1 -- 172.21.15.90:0/3157114834 learned_addr learned my addr 172.21.15.90:0/3157114834 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:11:44.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.179+0000 7f434e7fc700 10 client.?.objecter ms_handle_connect 0x7f435810b8b0 2024-04-14T15:11:44.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.179+0000 7f434e7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:11:44.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.179+0000 7f434e7fc700 10 client.?.objecter ms_handle_connect 0x7f435810f3c0 2024-04-14T15:11:44.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.179+0000 7f434e7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:11:44.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.179+0000 7f434e7fc700 10 client.?.objecter ms_handle_connect 0x7f4358103aa0 2024-04-14T15:11:44.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.179+0000 7f434e7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:11:44.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.179+0000 7f434e7fc700 1 -- 172.21.15.90:0/3157114834 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 150577520 0 0) 0x7f435810b5b0 con 0x7f435810b8b0 2024-04-14T15:11:44.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.179+0000 7f434e7fc700 1 -- 172.21.15.90:0/3157114834 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f43440035b0 con 0x7f435810b8b0 2024-04-14T15:11:44.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.180+0000 7f434e7fc700 1 -- 172.21.15.90:0/3157114834 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3198579240 0 0) 0x7f43581a8ec0 con 0x7f4358103aa0 2024-04-14T15:11:44.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.180+0000 7f434e7fc700 1 -- 172.21.15.90:0/3157114834 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f435810b5b0 con 0x7f4358103aa0 2024-04-14T15:11:44.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.180+0000 7f434e7fc700 1 -- 172.21.15.90:0/3157114834 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2512046331 0 0) 0x7f43581aa0a0 con 0x7f435810f3c0 2024-04-14T15:11:44.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.180+0000 7f434e7fc700 1 -- 172.21.15.90:0/3157114834 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f43581a8ec0 con 0x7f435810f3c0 2024-04-14T15:11:44.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.180+0000 7f434e7fc700 1 -- 172.21.15.90:0/3157114834 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 127774469 0 0) 0x7f43440035b0 con 0x7f435810b8b0 2024-04-14T15:11:44.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.180+0000 7f434e7fc700 1 -- 172.21.15.90:0/3157114834 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f43581aa0a0 con 0x7f435810b8b0 2024-04-14T15:11:44.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.180+0000 7f434e7fc700 1 -- 172.21.15.90:0/3157114834 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f4354001430 con 0x7f435810b8b0 2024-04-14T15:11:44.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.180+0000 7f434e7fc700 1 -- 172.21.15.90:0/3157114834 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3539775890 0 0) 0x7f435810b5b0 con 0x7f4358103aa0 2024-04-14T15:11:44.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.180+0000 7f434e7fc700 1 -- 172.21.15.90:0/3157114834 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f43440035b0 con 0x7f4358103aa0 2024-04-14T15:11:44.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.180+0000 7f434e7fc700 1 -- 172.21.15.90:0/3157114834 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4057076527 0 0) 0x7f43581a8ec0 con 0x7f435810f3c0 2024-04-14T15:11:44.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.180+0000 7f434e7fc700 1 -- 172.21.15.90:0/3157114834 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f435810b5b0 con 0x7f435810f3c0 2024-04-14T15:11:44.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.180+0000 7f434e7fc700 1 -- 172.21.15.90:0/3157114834 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f43480018d0 con 0x7f4358103aa0 2024-04-14T15:11:44.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.180+0000 7f434e7fc700 1 -- 172.21.15.90:0/3157114834 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f4350001af0 con 0x7f435810f3c0 2024-04-14T15:11:44.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.180+0000 7f434e7fc700 1 -- 172.21.15.90:0/3157114834 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 827474049 0 0) 0x7f43581aa0a0 con 0x7f435810b8b0 2024-04-14T15:11:44.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.180+0000 7f434e7fc700 1 -- 172.21.15.90:0/3157114834 >> v1:172.21.15.90:6790/0 conn(0x7f435810f3c0 legacy=0x7f43581a6790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:44.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.180+0000 7f434e7fc700 1 -- 172.21.15.90:0/3157114834 >> v1:172.21.15.92:6789/0 conn(0x7f4358103aa0 legacy=0x7f4358108400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:44.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.180+0000 7f434e7fc700 1 -- 172.21.15.90:0/3157114834 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43581ab280 con 0x7f435810b8b0 2024-04-14T15:11:44.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.180+0000 7f435f2aa700 1 -- 172.21.15.90:0/3157114834 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f43581a90f0 con 0x7f435810b8b0 2024-04-14T15:11:44.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.180+0000 7f435f2aa700 10 client.14808.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:11:44.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.180+0000 7f435f2aa700 1 -- 172.21.15.90:0/3157114834 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f43581a95b0 con 0x7f435810b8b0 2024-04-14T15:11:44.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.180+0000 7f435f2aa700 1 librados: init done 2024-04-14T15:11:44.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.180+0000 7f434e7fc700 1 -- 172.21.15.90:0/3157114834 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f4354001b30 con 0x7f435810b8b0 2024-04-14T15:11:44.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.180+0000 7f434e7fc700 1 -- 172.21.15.90:0/3157114834 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f4354004ea0 con 0x7f435810b8b0 2024-04-14T15:11:44.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.181+0000 7f435f2aa700 1 -- 172.21.15.90:0/3157114834 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f433c005100 con 0x7f435810b8b0 2024-04-14T15:11:44.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.182+0000 7f434e7fc700 1 -- 172.21.15.90:0/3157114834 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f4354003820 con 0x7f435810b8b0 2024-04-14T15:11:44.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.182+0000 7f434e7fc700 1 -- 172.21.15.90:0/3157114834 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(93..93 src has 41..93) ==== 6288+0+0 (unknown 1222187150 0 0) 0x7f435408a270 con 0x7f435810b8b0 2024-04-14T15:11:44.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.182+0000 7f434e7fc700 10 client.14808.objecter ms_dispatch 0x7f4358001550 osd_map(93..93 src has 41..93) 2024-04-14T15:11:44.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.182+0000 7f434e7fc700 3 client.14808.objecter handle_osd_map got epochs [93,93] > 0 2024-04-14T15:11:44.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.182+0000 7f434e7fc700 3 client.14808.objecter handle_osd_map decoding full epoch 93 2024-04-14T15:11:44.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.182+0000 7f434e7fc700 20 client.14808.objecter dump_active .. 0 homeless 2024-04-14T15:11:44.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.182+0000 7f434e7fc700 1 -- 172.21.15.90:0/3157114834 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v93) ==== 54+0+16 (unknown 296070313 0 2420000192) 0x7f435408a6d0 con 0x7f435810b8b0 2024-04-14T15:11:44.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.183+0000 7f434e7fc700 10 client.14808.objecter ms_handle_connect 0x7f43440713d0 2024-04-14T15:11:44.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.183+0000 7f435f2aa700 10 client.14808.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:11:44.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.183+0000 7f435f2aa700 15 client.14808.objecter _session_command_op_assign -1 1 2024-04-14T15:11:44.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.183+0000 7f435f2aa700 20 client.14808.objecter _get_session s=0x7f433c0053e0 osd=0 3 2024-04-14T15:11:44.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.183+0000 7f435f2aa700 20 client.14808.objecter put_session s=0x7f433c0053e0 osd=0 3 2024-04-14T15:11:44.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.183+0000 7f435f2aa700 20 client.14808.objecter _get_session s=0x7f433c0053e0 osd=0 3 2024-04-14T15:11:44.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.183+0000 7f435f2aa700 15 client.14808.objecter _session_command_op_remove -1 1 2024-04-14T15:11:44.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.183+0000 7f435f2aa700 20 client.14808.objecter get_session s=0x7f433c0053e0 osd=0 3 2024-04-14T15:11:44.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.183+0000 7f435f2aa700 15 client.14808.objecter _session_command_op_assign 0 1 2024-04-14T15:11:44.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.183+0000 7f435f2aa700 20 client.14808.objecter put_session s=0x7f433c0053e0 osd=0 4 2024-04-14T15:11:44.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.183+0000 7f435f2aa700 10 client.14808.objecter _send_command 1 2024-04-14T15:11:44.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.183+0000 7f435f2aa700 1 -- 172.21.15.90:0/3157114834 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f433c004b50 con 0x7f433c004730 2024-04-14T15:11:44.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.183+0000 7f434e7fc700 10 client.14808.objecter ms_handle_connect 0x7f433c004730 2024-04-14T15:11:44.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.184+0000 7f434e7fc700 1 -- 172.21.15.90:0/3157114834 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f433c004b50 con 0x7f433c004730 2024-04-14T15:11:44.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.184+0000 7f434e7fc700 10 client.14808.objecter ms_dispatch 0x7f4358001550 command_reply(tid 1: 0 ) 2024-04-14T15:11:44.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.184+0000 7f434e7fc700 10 client.14808.objecter _finish_command 1 = system:0 2024-04-14T15:11:44.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.184+0000 7f434e7fc700 20 client.14808.objecter put_session s=0x7f433c0053e0 osd=0 4 2024-04-14T15:11:44.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.184+0000 7f434e7fc700 15 client.14808.objecter _session_command_op_remove 0 1 2024-04-14T15:11:44.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.201+0000 7f435f2aa700 10 client.14808.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:11:44.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.201+0000 7f435f2aa700 15 client.14808.objecter _session_command_op_assign -1 2 2024-04-14T15:11:44.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.201+0000 7f435f2aa700 20 client.14808.objecter _get_session s=0x7f433c0053e0 osd=0 3 2024-04-14T15:11:44.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.201+0000 7f435f2aa700 20 client.14808.objecter put_session s=0x7f433c0053e0 osd=0 3 2024-04-14T15:11:44.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.201+0000 7f435f2aa700 20 client.14808.objecter _get_session s=0x7f433c0053e0 osd=0 3 2024-04-14T15:11:44.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.201+0000 7f435f2aa700 15 client.14808.objecter _session_command_op_remove -1 2 2024-04-14T15:11:44.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.201+0000 7f435f2aa700 20 client.14808.objecter get_session s=0x7f433c0053e0 osd=0 3 2024-04-14T15:11:44.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.201+0000 7f435f2aa700 15 client.14808.objecter _session_command_op_assign 0 2 2024-04-14T15:11:44.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.201+0000 7f435f2aa700 20 client.14808.objecter put_session s=0x7f433c0053e0 osd=0 4 2024-04-14T15:11:44.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.201+0000 7f435f2aa700 10 client.14808.objecter _send_command 2 2024-04-14T15:11:44.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.201+0000 7f435f2aa700 1 -- 172.21.15.90:0/3157114834 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f433c0079f0 con 0x7f433c004730 2024-04-14T15:11:44.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.203+0000 7f434e7fc700 1 -- 172.21.15.90:0/3157114834 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f433c0079f0 con 0x7f433c004730 2024-04-14T15:11:44.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.203+0000 7f434e7fc700 10 client.14808.objecter ms_dispatch 0x7f4358001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:11:44.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.203+0000 7f434e7fc700 10 client.14808.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:11:44.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.203+0000 7f434e7fc700 20 client.14808.objecter put_session s=0x7f433c0053e0 osd=0 4 2024-04-14T15:11:44.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.203+0000 7f434e7fc700 15 client.14808.objecter _session_command_op_remove 0 2 2024-04-14T15:11:44.203 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:11:44.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.203+0000 7f435f2aa700 10 client.14808.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:11:44.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.203+0000 7f435f2aa700 15 client.14808.objecter _session_command_op_assign -1 3 2024-04-14T15:11:44.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.203+0000 7f435f2aa700 20 client.14808.objecter _get_session s=0x7f433c004110 osd=1 3 2024-04-14T15:11:44.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.203+0000 7f435f2aa700 20 client.14808.objecter put_session s=0x7f433c004110 osd=1 3 2024-04-14T15:11:44.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.203+0000 7f435f2aa700 20 client.14808.objecter _get_session s=0x7f433c004110 osd=1 3 2024-04-14T15:11:44.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.203+0000 7f435f2aa700 15 client.14808.objecter _session_command_op_remove -1 3 2024-04-14T15:11:44.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.203+0000 7f435f2aa700 20 client.14808.objecter get_session s=0x7f433c004110 osd=1 3 2024-04-14T15:11:44.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.203+0000 7f435f2aa700 15 client.14808.objecter _session_command_op_assign 1 3 2024-04-14T15:11:44.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.203+0000 7f435f2aa700 20 client.14808.objecter put_session s=0x7f433c004110 osd=1 4 2024-04-14T15:11:44.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.203+0000 7f435f2aa700 10 client.14808.objecter _send_command 3 2024-04-14T15:11:44.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.203+0000 7f435f2aa700 1 -- 172.21.15.90:0/3157114834 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f433c002180 con 0x7f433c004280 2024-04-14T15:11:44.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.203+0000 7f434e7fc700 10 client.14808.objecter ms_handle_connect 0x7f433c004280 2024-04-14T15:11:44.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.204+0000 7f434e7fc700 1 -- 172.21.15.90:0/3157114834 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f433c002180 con 0x7f433c004280 2024-04-14T15:11:44.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.204+0000 7f434e7fc700 10 client.14808.objecter ms_dispatch 0x7f4358001550 command_reply(tid 3: 0 ) 2024-04-14T15:11:44.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.204+0000 7f434e7fc700 10 client.14808.objecter _finish_command 3 = system:0 2024-04-14T15:11:44.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.204+0000 7f434e7fc700 20 client.14808.objecter put_session s=0x7f433c004110 osd=1 4 2024-04-14T15:11:44.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.204+0000 7f434e7fc700 15 client.14808.objecter _session_command_op_remove 1 3 2024-04-14T15:11:44.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.220+0000 7f435f2aa700 10 client.14808.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:11:44.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.220+0000 7f435f2aa700 15 client.14808.objecter _session_command_op_assign -1 4 2024-04-14T15:11:44.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.220+0000 7f435f2aa700 20 client.14808.objecter _get_session s=0x7f433c004110 osd=1 3 2024-04-14T15:11:44.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.220+0000 7f435f2aa700 20 client.14808.objecter put_session s=0x7f433c004110 osd=1 3 2024-04-14T15:11:44.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.220+0000 7f435f2aa700 20 client.14808.objecter _get_session s=0x7f433c004110 osd=1 3 2024-04-14T15:11:44.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.220+0000 7f435f2aa700 15 client.14808.objecter _session_command_op_remove -1 4 2024-04-14T15:11:44.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.220+0000 7f435f2aa700 20 client.14808.objecter get_session s=0x7f433c004110 osd=1 3 2024-04-14T15:11:44.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.220+0000 7f435f2aa700 15 client.14808.objecter _session_command_op_assign 1 4 2024-04-14T15:11:44.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.220+0000 7f435f2aa700 20 client.14808.objecter put_session s=0x7f433c004110 osd=1 4 2024-04-14T15:11:44.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.220+0000 7f435f2aa700 10 client.14808.objecter _send_command 4 2024-04-14T15:11:44.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.220+0000 7f435f2aa700 1 -- 172.21.15.90:0/3157114834 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f433c001600 con 0x7f433c004280 2024-04-14T15:11:44.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.222+0000 7f434e7fc700 1 -- 172.21.15.90:0/3157114834 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f433c001600 con 0x7f433c004280 2024-04-14T15:11:44.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.222+0000 7f434e7fc700 10 client.14808.objecter ms_dispatch 0x7f4358001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:11:44.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.222+0000 7f434e7fc700 10 client.14808.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:11:44.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.222+0000 7f434e7fc700 20 client.14808.objecter put_session s=0x7f433c004110 osd=1 4 2024-04-14T15:11:44.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.222+0000 7f434e7fc700 15 client.14808.objecter _session_command_op_remove 1 4 2024-04-14T15:11:44.223 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:11:44.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.222+0000 7f435f2aa700 10 client.14808.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:11:44.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.222+0000 7f435f2aa700 15 client.14808.objecter _session_command_op_assign -1 5 2024-04-14T15:11:44.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.222+0000 7f435f2aa700 20 client.14808.objecter _get_session s=0x7f433c008370 osd=2 3 2024-04-14T15:11:44.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.222+0000 7f435f2aa700 20 client.14808.objecter put_session s=0x7f433c008370 osd=2 3 2024-04-14T15:11:44.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.222+0000 7f435f2aa700 20 client.14808.objecter _get_session s=0x7f433c008370 osd=2 3 2024-04-14T15:11:44.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.222+0000 7f435f2aa700 15 client.14808.objecter _session_command_op_remove -1 5 2024-04-14T15:11:44.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.222+0000 7f435f2aa700 20 client.14808.objecter get_session s=0x7f433c008370 osd=2 3 2024-04-14T15:11:44.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.222+0000 7f435f2aa700 15 client.14808.objecter _session_command_op_assign 2 5 2024-04-14T15:11:44.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.222+0000 7f435f2aa700 20 client.14808.objecter put_session s=0x7f433c008370 osd=2 4 2024-04-14T15:11:44.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.222+0000 7f435f2aa700 10 client.14808.objecter _send_command 5 2024-04-14T15:11:44.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.222+0000 7f435f2aa700 1 -- 172.21.15.90:0/3157114834 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f433c0110e0 con 0x7f433c00d5d0 2024-04-14T15:11:44.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.222+0000 7f434e7fc700 10 client.14808.objecter ms_handle_connect 0x7f433c00d5d0 2024-04-14T15:11:44.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.223+0000 7f434e7fc700 1 -- 172.21.15.90:0/3157114834 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f433c0110e0 con 0x7f433c00d5d0 2024-04-14T15:11:44.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.223+0000 7f434e7fc700 10 client.14808.objecter ms_dispatch 0x7f4358001550 command_reply(tid 5: 0 ) 2024-04-14T15:11:44.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.223+0000 7f434e7fc700 10 client.14808.objecter _finish_command 5 = system:0 2024-04-14T15:11:44.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.223+0000 7f434e7fc700 20 client.14808.objecter put_session s=0x7f433c008370 osd=2 4 2024-04-14T15:11:44.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.223+0000 7f434e7fc700 15 client.14808.objecter _session_command_op_remove 2 5 2024-04-14T15:11:44.237 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:11:44.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.239+0000 7f435f2aa700 10 client.14808.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:11:44.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.239+0000 7f435f2aa700 15 client.14808.objecter _session_command_op_assign -1 6 2024-04-14T15:11:44.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.239+0000 7f435f2aa700 20 client.14808.objecter _get_session s=0x7f433c008370 osd=2 3 2024-04-14T15:11:44.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.239+0000 7f435f2aa700 20 client.14808.objecter put_session s=0x7f433c008370 osd=2 3 2024-04-14T15:11:44.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.239+0000 7f435f2aa700 20 client.14808.objecter _get_session s=0x7f433c008370 osd=2 3 2024-04-14T15:11:44.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.239+0000 7f435f2aa700 15 client.14808.objecter _session_command_op_remove -1 6 2024-04-14T15:11:44.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.239+0000 7f435f2aa700 20 client.14808.objecter get_session s=0x7f433c008370 osd=2 3 2024-04-14T15:11:44.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.239+0000 7f435f2aa700 15 client.14808.objecter _session_command_op_assign 2 6 2024-04-14T15:11:44.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.239+0000 7f435f2aa700 20 client.14808.objecter put_session s=0x7f433c008370 osd=2 4 2024-04-14T15:11:44.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.239+0000 7f435f2aa700 10 client.14808.objecter _send_command 6 2024-04-14T15:11:44.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.239+0000 7f435f2aa700 1 -- 172.21.15.90:0/3157114834 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f433c007eb0 con 0x7f433c00d5d0 2024-04-14T15:11:44.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.241+0000 7f434e7fc700 1 -- 172.21.15.90:0/3157114834 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f433c007eb0 con 0x7f433c00d5d0 2024-04-14T15:11:44.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.241+0000 7f434e7fc700 10 client.14808.objecter ms_dispatch 0x7f4358001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:11:44.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.241+0000 7f434e7fc700 10 client.14808.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:11:44.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.241+0000 7f434e7fc700 20 client.14808.objecter put_session s=0x7f433c008370 osd=2 4 2024-04-14T15:11:44.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.241+0000 7f434e7fc700 15 client.14808.objecter _session_command_op_remove 2 6 2024-04-14T15:11:44.242 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:11:44.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.241+0000 7f435f2aa700 10 client.14808.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:11:44.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.241+0000 7f435f2aa700 15 client.14808.objecter _session_command_op_assign -1 7 2024-04-14T15:11:44.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.241+0000 7f435f2aa700 20 client.14808.objecter _get_session s=0x7f433c00cc20 osd=3 3 2024-04-14T15:11:44.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.241+0000 7f435f2aa700 20 client.14808.objecter put_session s=0x7f433c00cc20 osd=3 3 2024-04-14T15:11:44.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.241+0000 7f435f2aa700 20 client.14808.objecter _get_session s=0x7f433c00cc20 osd=3 3 2024-04-14T15:11:44.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.241+0000 7f435f2aa700 15 client.14808.objecter _session_command_op_remove -1 7 2024-04-14T15:11:44.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.241+0000 7f435f2aa700 20 client.14808.objecter get_session s=0x7f433c00cc20 osd=3 3 2024-04-14T15:11:44.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.241+0000 7f435f2aa700 15 client.14808.objecter _session_command_op_assign 3 7 2024-04-14T15:11:44.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.241+0000 7f435f2aa700 20 client.14808.objecter put_session s=0x7f433c00cc20 osd=3 4 2024-04-14T15:11:44.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.241+0000 7f435f2aa700 10 client.14808.objecter _send_command 7 2024-04-14T15:11:44.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.241+0000 7f435f2aa700 1 -- 172.21.15.90:0/3157114834 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f433c015f20 con 0x7f433c012410 2024-04-14T15:11:44.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.241+0000 7f434e7fc700 10 client.14808.objecter ms_handle_connect 0x7f433c012410 2024-04-14T15:11:44.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.242+0000 7f434e7fc700 1 -- 172.21.15.90:0/3157114834 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f433c015f20 con 0x7f433c012410 2024-04-14T15:11:44.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.242+0000 7f434e7fc700 10 client.14808.objecter ms_dispatch 0x7f4358001550 command_reply(tid 7: 0 ) 2024-04-14T15:11:44.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.242+0000 7f434e7fc700 10 client.14808.objecter _finish_command 7 = system:0 2024-04-14T15:11:44.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.242+0000 7f434e7fc700 20 client.14808.objecter put_session s=0x7f433c00cc20 osd=3 4 2024-04-14T15:11:44.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.242+0000 7f434e7fc700 15 client.14808.objecter _session_command_op_remove 3 7 2024-04-14T15:11:44.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.259+0000 7f435f2aa700 10 client.14808.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:11:44.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.259+0000 7f435f2aa700 15 client.14808.objecter _session_command_op_assign -1 8 2024-04-14T15:11:44.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.259+0000 7f435f2aa700 20 client.14808.objecter _get_session s=0x7f433c00cc20 osd=3 3 2024-04-14T15:11:44.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.259+0000 7f435f2aa700 20 client.14808.objecter put_session s=0x7f433c00cc20 osd=3 3 2024-04-14T15:11:44.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.259+0000 7f435f2aa700 20 client.14808.objecter _get_session s=0x7f433c00cc20 osd=3 3 2024-04-14T15:11:44.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.259+0000 7f435f2aa700 15 client.14808.objecter _session_command_op_remove -1 8 2024-04-14T15:11:44.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.259+0000 7f435f2aa700 20 client.14808.objecter get_session s=0x7f433c00cc20 osd=3 3 2024-04-14T15:11:44.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.259+0000 7f435f2aa700 15 client.14808.objecter _session_command_op_assign 3 8 2024-04-14T15:11:44.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.259+0000 7f435f2aa700 20 client.14808.objecter put_session s=0x7f433c00cc20 osd=3 4 2024-04-14T15:11:44.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.259+0000 7f435f2aa700 10 client.14808.objecter _send_command 8 2024-04-14T15:11:44.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.259+0000 7f435f2aa700 1 -- 172.21.15.90:0/3157114834 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f433c00c700 con 0x7f433c012410 2024-04-14T15:11:44.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.260+0000 7f434e7fc700 1 -- 172.21.15.90:0/3157114834 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f433c00c700 con 0x7f433c012410 2024-04-14T15:11:44.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.260+0000 7f434e7fc700 10 client.14808.objecter ms_dispatch 0x7f4358001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:11:44.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.260+0000 7f434e7fc700 10 client.14808.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:11:44.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.260+0000 7f434e7fc700 20 client.14808.objecter put_session s=0x7f433c00cc20 osd=3 4 2024-04-14T15:11:44.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.260+0000 7f434e7fc700 15 client.14808.objecter _session_command_op_remove 3 8 2024-04-14T15:11:44.261 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:11:44.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.260+0000 7f435f2aa700 10 client.14808.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:11:44.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.260+0000 7f435f2aa700 15 client.14808.objecter _session_command_op_assign -1 9 2024-04-14T15:11:44.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.260+0000 7f435f2aa700 20 client.14808.objecter _get_session s=0x7f433c00c950 osd=4 3 2024-04-14T15:11:44.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.260+0000 7f435f2aa700 20 client.14808.objecter put_session s=0x7f433c00c950 osd=4 3 2024-04-14T15:11:44.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.260+0000 7f435f2aa700 20 client.14808.objecter _get_session s=0x7f433c00c950 osd=4 3 2024-04-14T15:11:44.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.260+0000 7f435f2aa700 15 client.14808.objecter _session_command_op_remove -1 9 2024-04-14T15:11:44.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.260+0000 7f435f2aa700 20 client.14808.objecter get_session s=0x7f433c00c950 osd=4 3 2024-04-14T15:11:44.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.260+0000 7f435f2aa700 15 client.14808.objecter _session_command_op_assign 4 9 2024-04-14T15:11:44.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.260+0000 7f435f2aa700 20 client.14808.objecter put_session s=0x7f433c00c950 osd=4 4 2024-04-14T15:11:44.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.260+0000 7f435f2aa700 10 client.14808.objecter _send_command 9 2024-04-14T15:11:44.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.260+0000 7f435f2aa700 1 -- 172.21.15.90:0/3157114834 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f433c01a800 con 0x7f433c011f70 2024-04-14T15:11:44.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.261+0000 7f434e7fc700 10 client.14808.objecter ms_handle_connect 0x7f433c011f70 2024-04-14T15:11:44.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.262+0000 7f434e7fc700 1 -- 172.21.15.90:0/3157114834 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f433c01a800 con 0x7f433c011f70 2024-04-14T15:11:44.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.262+0000 7f434e7fc700 10 client.14808.objecter ms_dispatch 0x7f4358001550 command_reply(tid 9: 0 ) 2024-04-14T15:11:44.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.262+0000 7f434e7fc700 10 client.14808.objecter _finish_command 9 = system:0 2024-04-14T15:11:44.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.262+0000 7f434e7fc700 20 client.14808.objecter put_session s=0x7f433c00c950 osd=4 4 2024-04-14T15:11:44.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.262+0000 7f434e7fc700 15 client.14808.objecter _session_command_op_remove 4 9 2024-04-14T15:11:44.264 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:44.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:44.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:11:44.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.278+0000 7f435f2aa700 10 client.14808.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:11:44.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.278+0000 7f435f2aa700 15 client.14808.objecter _session_command_op_assign -1 10 2024-04-14T15:11:44.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.278+0000 7f435f2aa700 20 client.14808.objecter _get_session s=0x7f433c00c950 osd=4 3 2024-04-14T15:11:44.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.278+0000 7f435f2aa700 20 client.14808.objecter put_session s=0x7f433c00c950 osd=4 3 2024-04-14T15:11:44.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.278+0000 7f435f2aa700 20 client.14808.objecter _get_session s=0x7f433c00c950 osd=4 3 2024-04-14T15:11:44.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.278+0000 7f435f2aa700 15 client.14808.objecter _session_command_op_remove -1 10 2024-04-14T15:11:44.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.278+0000 7f435f2aa700 20 client.14808.objecter get_session s=0x7f433c00c950 osd=4 3 2024-04-14T15:11:44.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.278+0000 7f435f2aa700 15 client.14808.objecter _session_command_op_assign 4 10 2024-04-14T15:11:44.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.278+0000 7f435f2aa700 20 client.14808.objecter put_session s=0x7f433c00c950 osd=4 4 2024-04-14T15:11:44.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.278+0000 7f435f2aa700 10 client.14808.objecter _send_command 10 2024-04-14T15:11:44.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.278+0000 7f435f2aa700 1 -- 172.21.15.90:0/3157114834 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f433c01c1b0 con 0x7f433c011f70 2024-04-14T15:11:44.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.279+0000 7f434e7fc700 1 -- 172.21.15.90:0/3157114834 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f433c01c1b0 con 0x7f433c011f70 2024-04-14T15:11:44.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.279+0000 7f434e7fc700 10 client.14808.objecter ms_dispatch 0x7f4358001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:11:44.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.279+0000 7f434e7fc700 10 client.14808.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:11:44.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.279+0000 7f434e7fc700 20 client.14808.objecter put_session s=0x7f433c00c950 osd=4 4 2024-04-14T15:11:44.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.279+0000 7f434e7fc700 15 client.14808.objecter _session_command_op_remove 4 10 2024-04-14T15:11:44.280 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:11:44.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.280+0000 7f435f2aa700 10 client.14808.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:11:44.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.280+0000 7f435f2aa700 15 client.14808.objecter _session_command_op_assign -1 11 2024-04-14T15:11:44.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.280+0000 7f435f2aa700 20 client.14808.objecter _get_session s=0x7f433c01c1b0 osd=5 3 2024-04-14T15:11:44.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.280+0000 7f435f2aa700 20 client.14808.objecter put_session s=0x7f433c01c1b0 osd=5 3 2024-04-14T15:11:44.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.280+0000 7f435f2aa700 20 client.14808.objecter _get_session s=0x7f433c01c1b0 osd=5 3 2024-04-14T15:11:44.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.280+0000 7f435f2aa700 15 client.14808.objecter _session_command_op_remove -1 11 2024-04-14T15:11:44.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.280+0000 7f435f2aa700 20 client.14808.objecter get_session s=0x7f433c01c1b0 osd=5 3 2024-04-14T15:11:44.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.280+0000 7f435f2aa700 15 client.14808.objecter _session_command_op_assign 5 11 2024-04-14T15:11:44.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.280+0000 7f435f2aa700 20 client.14808.objecter put_session s=0x7f433c01c1b0 osd=5 4 2024-04-14T15:11:44.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.280+0000 7f435f2aa700 10 client.14808.objecter _send_command 11 2024-04-14T15:11:44.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.280+0000 7f435f2aa700 1 -- 172.21.15.90:0/3157114834 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f433c01cd30 con 0x7f433c01c8d0 2024-04-14T15:11:44.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.280+0000 7f434e7fc700 10 client.14808.objecter ms_handle_connect 0x7f433c01c8d0 2024-04-14T15:11:44.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.281+0000 7f434e7fc700 1 -- 172.21.15.90:0/3157114834 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f433c01cd30 con 0x7f433c01c8d0 2024-04-14T15:11:44.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.281+0000 7f434e7fc700 10 client.14808.objecter ms_dispatch 0x7f4358001550 command_reply(tid 11: 0 ) 2024-04-14T15:11:44.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.281+0000 7f434e7fc700 10 client.14808.objecter _finish_command 11 = system:0 2024-04-14T15:11:44.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.281+0000 7f434e7fc700 20 client.14808.objecter put_session s=0x7f433c01c1b0 osd=5 4 2024-04-14T15:11:44.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.281+0000 7f434e7fc700 15 client.14808.objecter _session_command_op_remove 5 11 2024-04-14T15:11:44.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.297+0000 7f435f2aa700 10 client.14808.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:11:44.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.297+0000 7f435f2aa700 15 client.14808.objecter _session_command_op_assign -1 12 2024-04-14T15:11:44.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.297+0000 7f435f2aa700 20 client.14808.objecter _get_session s=0x7f433c01c1b0 osd=5 3 2024-04-14T15:11:44.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.297+0000 7f435f2aa700 20 client.14808.objecter put_session s=0x7f433c01c1b0 osd=5 3 2024-04-14T15:11:44.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.297+0000 7f435f2aa700 20 client.14808.objecter _get_session s=0x7f433c01c1b0 osd=5 3 2024-04-14T15:11:44.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.297+0000 7f435f2aa700 15 client.14808.objecter _session_command_op_remove -1 12 2024-04-14T15:11:44.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.297+0000 7f435f2aa700 20 client.14808.objecter get_session s=0x7f433c01c1b0 osd=5 3 2024-04-14T15:11:44.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.297+0000 7f435f2aa700 15 client.14808.objecter _session_command_op_assign 5 12 2024-04-14T15:11:44.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.297+0000 7f435f2aa700 20 client.14808.objecter put_session s=0x7f433c01c1b0 osd=5 4 2024-04-14T15:11:44.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.297+0000 7f435f2aa700 10 client.14808.objecter _send_command 12 2024-04-14T15:11:44.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.297+0000 7f435f2aa700 1 -- 172.21.15.90:0/3157114834 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f433c01cfb0 con 0x7f433c01c8d0 2024-04-14T15:11:44.300 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:11:44.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.299+0000 7f434e7fc700 1 -- 172.21.15.90:0/3157114834 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f433c01cfb0 con 0x7f433c01c8d0 2024-04-14T15:11:44.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.299+0000 7f434e7fc700 10 client.14808.objecter ms_dispatch 0x7f4358001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:11:44.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.299+0000 7f434e7fc700 10 client.14808.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:11:44.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.299+0000 7f434e7fc700 20 client.14808.objecter put_session s=0x7f433c01c1b0 osd=5 4 2024-04-14T15:11:44.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.299+0000 7f434e7fc700 15 client.14808.objecter _session_command_op_remove 5 12 2024-04-14T15:11:44.301 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-14T15:11:44.301 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-14T15:11:44.301 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-14T15:11:44.301 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-14T15:11:44.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.300+0000 7f435f2aa700 10 client.14808.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:11:44.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.300+0000 7f435f2aa700 15 client.14808.objecter _session_command_op_assign -1 13 2024-04-14T15:11:44.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.300+0000 7f435f2aa700 20 client.14808.objecter _get_session s=0x7f433c01cd30 osd=6 3 2024-04-14T15:11:44.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.300+0000 7f435f2aa700 20 client.14808.objecter put_session s=0x7f433c01cd30 osd=6 3 2024-04-14T15:11:44.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.300+0000 7f435f2aa700 20 client.14808.objecter _get_session s=0x7f433c01cd30 osd=6 3 2024-04-14T15:11:44.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.300+0000 7f435f2aa700 15 client.14808.objecter _session_command_op_remove -1 13 2024-04-14T15:11:44.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.300+0000 7f435f2aa700 20 client.14808.objecter get_session s=0x7f433c01cd30 osd=6 3 2024-04-14T15:11:44.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.300+0000 7f435f2aa700 15 client.14808.objecter _session_command_op_assign 6 13 2024-04-14T15:11:44.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.300+0000 7f435f2aa700 20 client.14808.objecter put_session s=0x7f433c01cd30 osd=6 4 2024-04-14T15:11:44.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.300+0000 7f435f2aa700 10 client.14808.objecter _send_command 13 2024-04-14T15:11:44.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.300+0000 7f435f2aa700 1 -- 172.21.15.90:0/3157114834 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f433c01d1a0 con 0x7f433c01f8c0 2024-04-14T15:11:44.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.300+0000 7f434e7fc700 10 client.14808.objecter ms_handle_connect 0x7f433c01f8c0 2024-04-14T15:11:44.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.301+0000 7f434e7fc700 1 -- 172.21.15.90:0/3157114834 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f433c01d1a0 con 0x7f433c01f8c0 2024-04-14T15:11:44.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.301+0000 7f434e7fc700 10 client.14808.objecter ms_dispatch 0x7f4358001550 command_reply(tid 13: 0 ) 2024-04-14T15:11:44.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.301+0000 7f434e7fc700 10 client.14808.objecter _finish_command 13 = system:0 2024-04-14T15:11:44.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.301+0000 7f434e7fc700 20 client.14808.objecter put_session s=0x7f433c01cd30 osd=6 4 2024-04-14T15:11:44.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.301+0000 7f434e7fc700 15 client.14808.objecter _session_command_op_remove 6 13 2024-04-14T15:11:44.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.318+0000 7f435f2aa700 10 client.14808.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:11:44.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.318+0000 7f435f2aa700 15 client.14808.objecter _session_command_op_assign -1 14 2024-04-14T15:11:44.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.318+0000 7f435f2aa700 20 client.14808.objecter _get_session s=0x7f433c01cd30 osd=6 3 2024-04-14T15:11:44.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.318+0000 7f435f2aa700 20 client.14808.objecter put_session s=0x7f433c01cd30 osd=6 3 2024-04-14T15:11:44.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.318+0000 7f435f2aa700 20 client.14808.objecter _get_session s=0x7f433c01cd30 osd=6 3 2024-04-14T15:11:44.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.318+0000 7f435f2aa700 15 client.14808.objecter _session_command_op_remove -1 14 2024-04-14T15:11:44.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.318+0000 7f435f2aa700 20 client.14808.objecter get_session s=0x7f433c01cd30 osd=6 3 2024-04-14T15:11:44.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.318+0000 7f435f2aa700 15 client.14808.objecter _session_command_op_assign 6 14 2024-04-14T15:11:44.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.318+0000 7f435f2aa700 20 client.14808.objecter put_session s=0x7f433c01cd30 osd=6 4 2024-04-14T15:11:44.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.318+0000 7f435f2aa700 10 client.14808.objecter _send_command 14 2024-04-14T15:11:44.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.318+0000 7f435f2aa700 1 -- 172.21.15.90:0/3157114834 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f433c023400 con 0x7f433c01f8c0 2024-04-14T15:11:44.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.320+0000 7f434e7fc700 1 -- 172.21.15.90:0/3157114834 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f433c023400 con 0x7f433c01f8c0 2024-04-14T15:11:44.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.320+0000 7f434e7fc700 10 client.14808.objecter ms_dispatch 0x7f4358001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:11:44.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.320+0000 7f434e7fc700 10 client.14808.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:11:44.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.320+0000 7f434e7fc700 20 client.14808.objecter put_session s=0x7f433c01cd30 osd=6 4 2024-04-14T15:11:44.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.320+0000 7f434e7fc700 15 client.14808.objecter _session_command_op_remove 6 14 2024-04-14T15:11:44.320 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:11:44.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.320+0000 7f435f2aa700 10 client.14808.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:11:44.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.320+0000 7f435f2aa700 15 client.14808.objecter _session_command_op_assign -1 15 2024-04-14T15:11:44.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.320+0000 7f435f2aa700 20 client.14808.objecter _get_session s=0x7f433c023400 osd=7 3 2024-04-14T15:11:44.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.320+0000 7f435f2aa700 20 client.14808.objecter put_session s=0x7f433c023400 osd=7 3 2024-04-14T15:11:44.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.320+0000 7f435f2aa700 20 client.14808.objecter _get_session s=0x7f433c023400 osd=7 3 2024-04-14T15:11:44.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.320+0000 7f435f2aa700 15 client.14808.objecter _session_command_op_remove -1 15 2024-04-14T15:11:44.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.320+0000 7f435f2aa700 20 client.14808.objecter get_session s=0x7f433c023400 osd=7 3 2024-04-14T15:11:44.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.320+0000 7f435f2aa700 15 client.14808.objecter _session_command_op_assign 7 15 2024-04-14T15:11:44.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.320+0000 7f435f2aa700 20 client.14808.objecter put_session s=0x7f433c023400 osd=7 4 2024-04-14T15:11:44.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.320+0000 7f435f2aa700 10 client.14808.objecter _send_command 15 2024-04-14T15:11:44.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.320+0000 7f435f2aa700 1 -- 172.21.15.90:0/3157114834 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f433c024750 con 0x7f433c023b20 2024-04-14T15:11:44.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.320+0000 7f434e7fc700 10 client.14808.objecter ms_handle_connect 0x7f433c023b20 2024-04-14T15:11:44.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.321+0000 7f434e7fc700 1 -- 172.21.15.90:0/3157114834 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f433c024750 con 0x7f433c023b20 2024-04-14T15:11:44.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.321+0000 7f434e7fc700 10 client.14808.objecter ms_dispatch 0x7f4358001550 command_reply(tid 15: 0 ) 2024-04-14T15:11:44.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.321+0000 7f434e7fc700 10 client.14808.objecter _finish_command 15 = system:0 2024-04-14T15:11:44.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.321+0000 7f434e7fc700 20 client.14808.objecter put_session s=0x7f433c023400 osd=7 4 2024-04-14T15:11:44.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.321+0000 7f434e7fc700 15 client.14808.objecter _session_command_op_remove 7 15 2024-04-14T15:11:44.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.337+0000 7f435f2aa700 10 client.14808.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:11:44.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.337+0000 7f435f2aa700 15 client.14808.objecter _session_command_op_assign -1 16 2024-04-14T15:11:44.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.337+0000 7f435f2aa700 20 client.14808.objecter _get_session s=0x7f433c023400 osd=7 3 2024-04-14T15:11:44.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.337+0000 7f435f2aa700 20 client.14808.objecter put_session s=0x7f433c023400 osd=7 3 2024-04-14T15:11:44.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.337+0000 7f435f2aa700 20 client.14808.objecter _get_session s=0x7f433c023400 osd=7 3 2024-04-14T15:11:44.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.337+0000 7f435f2aa700 15 client.14808.objecter _session_command_op_remove -1 16 2024-04-14T15:11:44.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.337+0000 7f435f2aa700 20 client.14808.objecter get_session s=0x7f433c023400 osd=7 3 2024-04-14T15:11:44.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.337+0000 7f435f2aa700 15 client.14808.objecter _session_command_op_assign 7 16 2024-04-14T15:11:44.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.337+0000 7f435f2aa700 20 client.14808.objecter put_session s=0x7f433c023400 osd=7 4 2024-04-14T15:11:44.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.337+0000 7f435f2aa700 10 client.14808.objecter _send_command 16 2024-04-14T15:11:44.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.337+0000 7f435f2aa700 1 -- 172.21.15.90:0/3157114834 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f433c024750 con 0x7f433c023b20 2024-04-14T15:11:44.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.339+0000 7f434e7fc700 1 -- 172.21.15.90:0/3157114834 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f433c024750 con 0x7f433c023b20 2024-04-14T15:11:44.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.339+0000 7f434e7fc700 10 client.14808.objecter ms_dispatch 0x7f4358001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:11:44.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.339+0000 7f434e7fc700 10 client.14808.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:11:44.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.339+0000 7f434e7fc700 20 client.14808.objecter put_session s=0x7f433c023400 osd=7 4 2024-04-14T15:11:44.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.339+0000 7f434e7fc700 15 client.14808.objecter _session_command_op_remove 7 16 2024-04-14T15:11:44.340 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:11:44.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.339+0000 7f435f2aa700 10 librados: watch_flush enter 2024-04-14T15:11:44.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.339+0000 7f435f2aa700 10 librados: watch_flush exit 2024-04-14T15:11:44.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.339+0000 7f435f2aa700 10 client.14808.objecter close_session for osd.0 2024-04-14T15:11:44.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.339+0000 7f435f2aa700 1 -- 172.21.15.90:0/3157114834 >> v1:172.21.15.90:6801/3761806885 conn(0x7f433c004730 legacy=0x7f433c002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:44.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.339+0000 7f435f2aa700 20 client.14808.objecter put_session s=0x7f433c0053e0 osd=0 1 2024-04-14T15:11:44.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.339+0000 7f435f2aa700 10 client.14808.objecter close_session for osd.1 2024-04-14T15:11:44.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.339+0000 7f435f2aa700 1 -- 172.21.15.90:0/3157114834 >> v1:172.21.15.90:6805/4170361699 conn(0x7f433c004280 legacy=0x7f433c00ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:44.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.339+0000 7f435f2aa700 20 client.14808.objecter put_session s=0x7f433c004110 osd=1 1 2024-04-14T15:11:44.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.339+0000 7f435f2aa700 10 client.14808.objecter close_session for osd.2 2024-04-14T15:11:44.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.339+0000 7f435f2aa700 1 -- 172.21.15.90:0/3157114834 >> v1:172.21.15.90:6809/1686184251 conn(0x7f433c00d5d0 legacy=0x7f433c00f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:44.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.339+0000 7f435f2aa700 20 client.14808.objecter put_session s=0x7f433c008370 osd=2 1 2024-04-14T15:11:44.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.339+0000 7f435f2aa700 10 client.14808.objecter close_session for osd.3 2024-04-14T15:11:44.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.339+0000 7f435f2aa700 1 -- 172.21.15.90:0/3157114834 >> v1:172.21.15.90:6813/2270166774 conn(0x7f433c012410 legacy=0x7f433c014800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:44.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.339+0000 7f435f2aa700 20 client.14808.objecter put_session s=0x7f433c00cc20 osd=3 1 2024-04-14T15:11:44.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.339+0000 7f435f2aa700 10 client.14808.objecter close_session for osd.4 2024-04-14T15:11:44.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.339+0000 7f435f2aa700 1 -- 172.21.15.90:0/3157114834 >> v1:172.21.15.92:6800/3887549733 conn(0x7f433c011f70 legacy=0x7f433c0190b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:44.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.339+0000 7f435f2aa700 20 client.14808.objecter put_session s=0x7f433c00c950 osd=4 1 2024-04-14T15:11:44.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.339+0000 7f435f2aa700 10 client.14808.objecter close_session for osd.5 2024-04-14T15:11:44.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.339+0000 7f435f2aa700 1 -- 172.21.15.90:0/3157114834 >> v1:172.21.15.92:6804/3244561440 conn(0x7f433c01c8d0 legacy=0x7f433c01a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:44.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.339+0000 7f435f2aa700 20 client.14808.objecter put_session s=0x7f433c01c1b0 osd=5 1 2024-04-14T15:11:44.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.339+0000 7f435f2aa700 10 client.14808.objecter close_session for osd.6 2024-04-14T15:11:44.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.339+0000 7f435f2aa700 1 -- 172.21.15.90:0/3157114834 >> v1:172.21.15.92:6808/636125963 conn(0x7f433c01f8c0 legacy=0x7f433c021cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:44.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.339+0000 7f435f2aa700 20 client.14808.objecter put_session s=0x7f433c01cd30 osd=6 1 2024-04-14T15:11:44.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.339+0000 7f435f2aa700 10 client.14808.objecter close_session for osd.7 2024-04-14T15:11:44.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.339+0000 7f435f2aa700 1 -- 172.21.15.90:0/3157114834 >> v1:172.21.15.92:6812/1773706716 conn(0x7f433c023b20 legacy=0x7f433c023f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:44.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.339+0000 7f435f2aa700 20 client.14808.objecter put_session s=0x7f433c023400 osd=7 1 2024-04-14T15:11:44.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.339+0000 7f435f2aa700 20 client.14808.objecter shutdown clearing up homeless session... 2024-04-14T15:11:44.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.339+0000 7f435f2aa700 10 client.14808.objecter successfully canceled tick 2024-04-14T15:11:44.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.339+0000 7f435d847700 1 -- 172.21.15.90:0/3157114834 reap_dead start 2024-04-14T15:11:44.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.339+0000 7f435d847700 1 -- 172.21.15.90:0/3157114834 reap_dead start 2024-04-14T15:11:44.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.339+0000 7f435d847700 1 -- 172.21.15.90:0/3157114834 reap_dead start 2024-04-14T15:11:44.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.339+0000 7f435d847700 1 -- 172.21.15.90:0/3157114834 reap_dead start 2024-04-14T15:11:44.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.339+0000 7f435d847700 1 -- 172.21.15.90:0/3157114834 reap_dead start 2024-04-14T15:11:44.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.339+0000 7f435f2aa700 1 -- 172.21.15.90:0/3157114834 >> v1:172.21.15.90:6800/420935427 conn(0x7f43440713d0 legacy=0x7f4344073890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:44.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.339+0000 7f435f2aa700 1 -- 172.21.15.90:0/3157114834 >> v1:172.21.15.90:6789/0 conn(0x7f435810b8b0 legacy=0x7f4358108b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:44.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.339+0000 7f435f2aa700 1 -- 172.21.15.90:0/3157114834 shutdown_connections 2024-04-14T15:11:44.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.339+0000 7f435f2aa700 1 -- 172.21.15.90:0/3157114834 >> 172.21.15.90:0/3157114834 conn(0x7f43580fd290 msgr2=0x7f43580ff680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:11:44.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.339+0000 7f435f2aa700 1 -- 172.21.15.90:0/3157114834 shutdown_connections 2024-04-14T15:11:44.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.340+0000 7f435f2aa700 1 -- 172.21.15.90:0/3157114834 wait complete. 2024-04-14T15:11:44.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:44.340+0000 7f435f2aa700 1 librados: shutdown 2024-04-14T15:11:44.344 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:11:44.344 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:11:44.344 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:11:44.344 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:11:44.344 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:11:44.344 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:11:44.344 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:11:44.344 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:11:44.366 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:11:44.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:44.390 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:44.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:11:44.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:44 smithi090 ceph-mon[41670]: 4.c scrub starts 2024-04-14T15:11:44.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:44 smithi090 ceph-mon[41670]: 4.c scrub ok 2024-04-14T15:11:44.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:44 smithi090 ceph-mon[41670]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-14T15:11:44.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:44 smithi090 ceph-mon[30399]: 4.c scrub starts 2024-04-14T15:11:44.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:44 smithi090 ceph-mon[30399]: 4.c scrub ok 2024-04-14T15:11:44.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:44 smithi090 ceph-mon[30399]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-14T15:11:44.492 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:11:44.516 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:44.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:44.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:11:44.618 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:11:44.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:44 smithi092 ceph-mon[33966]: 4.c scrub starts 2024-04-14T15:11:44.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:44 smithi092 ceph-mon[33966]: 4.c scrub ok 2024-04-14T15:11:44.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:44 smithi092 ceph-mon[33966]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-14T15:11:44.641 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:44.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:44.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:11:44.743 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:11:44.766 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:44.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:44.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:11:44.868 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:11:44.892 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:44.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:44.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:11:44.994 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:11:45.022 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:45.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:45.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:11:45.122 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:11:44 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:11:45.124 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:11:45.148 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:45.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:45.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:11:45.152 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-14T15:11:45.250 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:11:45.273 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:45.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:45.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:11:45.375 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:11:45.398 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:45.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:45.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:11:45.409 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:11:45.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:45 smithi090 ceph-mon[41670]: 2.1e scrub starts 2024-04-14T15:11:45.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:45 smithi090 ceph-mon[41670]: 2.1e scrub ok 2024-04-14T15:11:45.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:45 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3157114834' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:11:45.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:45 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:11:45.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:45 smithi090 ceph-mon[30399]: 2.1e scrub starts 2024-04-14T15:11:45.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:45 smithi090 ceph-mon[30399]: 2.1e scrub ok 2024-04-14T15:11:45.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:45 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3157114834' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:11:45.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:45 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:11:45.500 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:11:45.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:45.525 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:45.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:11:45.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:45 smithi092 ceph-mon[33966]: 2.1e scrub starts 2024-04-14T15:11:45.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:45 smithi092 ceph-mon[33966]: 2.1e scrub ok 2024-04-14T15:11:45.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:45 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3157114834' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:11:45.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:45 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:11:45.626 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:11:45.653 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:45.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:45.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:11:45.755 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:11:45.781 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:45.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:45.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:11:45.794 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:11:45 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-alertmanager-a[78793]: ts=2024-04-14T15:11:45.575Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-04-14T15:11:45.883 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:11:45.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:45.908 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:45.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:11:46.009 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:11:46.036 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:46.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:46.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:11:46.109 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:11:45 smithi090 podman[83244]: 2024-04-14 15:11:45.795001775 +0000 UTC m=+0.607743967 container died fc393084a9b6da0d9165587ad2f71d1ea3acf037ad5406bdf43ae8297b1a96fd (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-14T15:11:46.137 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:11:46.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:46.167 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:46.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:11:46.268 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:11:46.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:46.293 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:46.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:11:46.394 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:11:46.422 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:46.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:46.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:11:46.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:46 smithi090 ceph-mon[30399]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:11:46.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:46 smithi090 ceph-mon[41670]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:11:46.442 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:11:46 smithi090 podman[83244]: 2024-04-14 15:11:46.300024665 +0000 UTC m=+1.112766858 container cleanup fc393084a9b6da0d9165587ad2f71d1ea3acf037ad5406bdf43ae8297b1a96fd (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-14T15:11:46.442 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:11:46 smithi090 bash[83244]: ceph-0e288098-fa70-11ee-bc8e-c7b262605968-alertmanager-a 2024-04-14T15:11:46.524 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:11:46.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:46.550 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:46.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:11:46.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:46 smithi092 ceph-mon[33966]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:11:46.650 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:11:46.674 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:46.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:46.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:11:46.776 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:11:46.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:46.800 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:46.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:11:46.901 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:11:46.929 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:46.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:46.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:11:46.931 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:11:46 smithi090 podman[83370]: 2024-04-14 15:11:46.459915007 +0000 UTC m=+0.837017086 container remove fc393084a9b6da0d9165587ad2f71d1ea3acf037ad5406bdf43ae8297b1a96fd (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-14T15:11:46.931 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:11:46 smithi090 podman[83370]: 2024-04-14 15:11:46.536854046 +0000 UTC m=+0.913956146 volume remove bfd38cfe623e46a9dc77d1785ae43d3cd0a5f1a3e056066ba8ebfbb9d99cc625 2024-04-14T15:11:47.030 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:11:47.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:47.059 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:47.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:11:47.160 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:11:47.187 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:47.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:47.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:11:47.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:47 smithi090 ceph-mon[41670]: 2.1c scrub starts 2024-04-14T15:11:47.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:47 smithi090 ceph-mon[41670]: 2.1c scrub ok 2024-04-14T15:11:47.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:47 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:11:47.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:47 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:11:47.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:47 smithi090 ceph-mon[41670]: osdmap e94: 8 total, 8 up, 8 in 2024-04-14T15:11:47.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:47 smithi090 ceph-mon[30399]: 2.1c scrub starts 2024-04-14T15:11:47.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:47 smithi090 ceph-mon[30399]: 2.1c scrub ok 2024-04-14T15:11:47.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:47 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:11:47.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:47 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:11:47.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:47 smithi090 ceph-mon[30399]: osdmap e94: 8 total, 8 up, 8 in 2024-04-14T15:11:47.289 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:11:47.316 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:47.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:47.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:11:47.418 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:11:47.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:47.442 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:47.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:11:47.543 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:11:47.570 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:47.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:47.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:11:47.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:47 smithi092 ceph-mon[33966]: 2.1c scrub starts 2024-04-14T15:11:47.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:47 smithi092 ceph-mon[33966]: 2.1c scrub ok 2024-04-14T15:11:47.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:47 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:11:47.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:47 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:11:47.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:47 smithi092 ceph-mon[33966]: osdmap e94: 8 total, 8 up, 8 in 2024-04-14T15:11:47.672 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:11:47.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:47.697 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:47.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:11:47.797 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:11:47.811 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:11:47.824 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:47.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:47.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:11:47.926 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:11:47.950 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:47.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:47.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:11:48.051 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:11:48.078 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:48.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:48.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:11:48.180 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:11:48.208 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:48.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:48.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:11:48.283 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:11:48.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:48.286+0000 7efe71dda700 1 -- 172.21.15.90:0/154094656 >> v1:172.21.15.90:6789/0 conn(0x7efe6c10f430 legacy=0x7efe6c1118d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:48.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:48.286+0000 7efe71dda700 1 -- 172.21.15.90:0/154094656 shutdown_connections 2024-04-14T15:11:48.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:48.286+0000 7efe71dda700 1 -- 172.21.15.90:0/154094656 >> 172.21.15.90:0/154094656 conn(0x7efe6c0fd290 msgr2=0x7efe6c0ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:11:48.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:48.286+0000 7efe71dda700 1 -- 172.21.15.90:0/154094656 shutdown_connections 2024-04-14T15:11:48.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:48.286+0000 7efe71dda700 1 -- 172.21.15.90:0/154094656 wait complete. 2024-04-14T15:11:48.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:48.286+0000 7efe71dda700 1 librados: starting msgr at 2024-04-14T15:11:48.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:48.286+0000 7efe71dda700 1 librados: starting objecter 2024-04-14T15:11:48.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:48.286+0000 7efe71dda700 1 Processor -- start 2024-04-14T15:11:48.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:48.287+0000 7efe71dda700 1 -- start start 2024-04-14T15:11:48.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:48.287+0000 7efe71dda700 1 librados: setting wanted keys 2024-04-14T15:11:48.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:48.287+0000 7efe71dda700 1 librados: calling monclient init 2024-04-14T15:11:48.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:48.287+0000 7efe71dda700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7efe6c1aa0a0 con 0x7efe6c10b890 2024-04-14T15:11:48.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:48.287+0000 7efe71dda700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7efe6c1ab280 con 0x7efe6c103a80 2024-04-14T15:11:48.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:48.287+0000 7efe71dda700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7efe6c1ac460 con 0x7efe6c10f430 2024-04-14T15:11:48.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:48.287+0000 7efe6affd700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7efe6c10b890 0x7efe6c1a5250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:40156/0 (socket says 172.21.15.90:40156) 2024-04-14T15:11:48.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:48.287+0000 7efe6affd700 1 -- 172.21.15.90:0/2995664348 learned_addr learned my addr 172.21.15.90:0/2995664348 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:11:48.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:48.287+0000 7efe68ff9700 10 client.?.objecter ms_handle_connect 0x7efe6c10b890 2024-04-14T15:11:48.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:48.287+0000 7efe68ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:11:48.289 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:48 smithi090 ceph-mon[41670]: 6.1b scrub starts 2024-04-14T15:11:48.289 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:48 smithi090 ceph-mon[41670]: 6.1b scrub ok 2024-04-14T15:11:48.289 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:48 smithi090 ceph-mon[41670]: pgmap v60: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-14T15:11:48.289 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:48 smithi090 ceph-mon[41670]: 2.3 deep-scrub starts 2024-04-14T15:11:48.289 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:48 smithi090 ceph-mon[41670]: 2.3 deep-scrub ok 2024-04-14T15:11:48.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:48.287+0000 7efe68ff9700 1 -- 172.21.15.90:0/2995664348 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 561269343 0 0) 0x7efe6c1aa0a0 con 0x7efe6c10b890 2024-04-14T15:11:48.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:48.287+0000 7efe68ff9700 1 -- 172.21.15.90:0/2995664348 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efe540035b0 con 0x7efe6c10b890 2024-04-14T15:11:48.290 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:11:48 smithi090 systemd[1]: ceph-0e288098-fa70-11ee-bc8e-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-04-14T15:11:48.290 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:11:48 smithi090 systemd[1]: Stopped Ceph alertmanager.a for 0e288098-fa70-11ee-bc8e-c7b262605968. 2024-04-14T15:11:48.290 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:11:48 smithi090 systemd[1]: Starting Ceph alertmanager.a for 0e288098-fa70-11ee-bc8e-c7b262605968... 2024-04-14T15:11:48.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:48.288+0000 7efe68ff9700 10 client.?.objecter ms_handle_connect 0x7efe6c103a80 2024-04-14T15:11:48.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:48.288+0000 7efe68ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:11:48.291 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:48 smithi090 ceph-mon[30399]: 6.1b scrub starts 2024-04-14T15:11:48.291 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:48 smithi090 ceph-mon[30399]: 6.1b scrub ok 2024-04-14T15:11:48.291 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:48 smithi090 ceph-mon[30399]: pgmap v60: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-14T15:11:48.291 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:48 smithi090 ceph-mon[30399]: 2.3 deep-scrub starts 2024-04-14T15:11:48.291 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:48 smithi090 ceph-mon[30399]: 2.3 deep-scrub ok 2024-04-14T15:11:48.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:48.288+0000 7efe68ff9700 10 client.?.objecter ms_handle_connect 0x7efe6c10f430 2024-04-14T15:11:48.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:48.288+0000 7efe68ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:11:48.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:48.288+0000 7efe68ff9700 1 -- 172.21.15.90:0/2995664348 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2060631989 0 0) 0x7efe540035b0 con 0x7efe6c10b890 2024-04-14T15:11:48.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:48.288+0000 7efe68ff9700 1 -- 172.21.15.90:0/2995664348 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efe6c1aa0a0 con 0x7efe6c10b890 2024-04-14T15:11:48.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:48.288+0000 7efe68ff9700 1 -- 172.21.15.90:0/2995664348 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7efe600018f0 con 0x7efe6c10b890 2024-04-14T15:11:48.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:48.288+0000 7efe68ff9700 1 -- 172.21.15.90:0/2995664348 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1274272283 0 0) 0x7efe6c1ab280 con 0x7efe6c103a80 2024-04-14T15:11:48.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:48.288+0000 7efe68ff9700 1 -- 172.21.15.90:0/2995664348 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efe540035b0 con 0x7efe6c103a80 2024-04-14T15:11:48.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:48.288+0000 7efe68ff9700 1 -- 172.21.15.90:0/2995664348 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3115727915 0 0) 0x7efe6c1aa0a0 con 0x7efe6c10b890 2024-04-14T15:11:48.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:48.288+0000 7efe68ff9700 1 -- 172.21.15.90:0/2995664348 >> v1:172.21.15.90:6790/0 conn(0x7efe6c10f430 legacy=0x7efe6c1a8980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:48.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:48.288+0000 7efe68ff9700 1 -- 172.21.15.90:0/2995664348 >> v1:172.21.15.92:6789/0 conn(0x7efe6c103a80 legacy=0x7efe6c1a0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:48.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:48.288+0000 7efe68ff9700 1 -- 172.21.15.90:0/2995664348 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe6c1ad640 con 0x7efe6c10b890 2024-04-14T15:11:48.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:48.288+0000 7efe71dda700 1 -- 172.21.15.90:0/2995664348 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7efe6c1ab4b0 con 0x7efe6c10b890 2024-04-14T15:11:48.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:48.288+0000 7efe71dda700 10 client.14814.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:11:48.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:48.288+0000 7efe71dda700 1 -- 172.21.15.90:0/2995664348 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7efe6c1ab970 con 0x7efe6c10b890 2024-04-14T15:11:48.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:48.288+0000 7efe71dda700 1 librados: init done 2024-04-14T15:11:48.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:48.288+0000 7efe68ff9700 1 -- 172.21.15.90:0/2995664348 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7efe60003bc0 con 0x7efe6c10b890 2024-04-14T15:11:48.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:48.288+0000 7efe68ff9700 1 -- 172.21.15.90:0/2995664348 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7efe60005db0 con 0x7efe6c10b890 2024-04-14T15:11:48.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:48.290+0000 7efe71dda700 1 -- 172.21.15.90:0/2995664348 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe6c06b590 con 0x7efe6c10b890 2024-04-14T15:11:48.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:48.290+0000 7efe68ff9700 1 -- 172.21.15.90:0/2995664348 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7efe60007060 con 0x7efe6c10b890 2024-04-14T15:11:48.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:48.291+0000 7efe68ff9700 1 -- 172.21.15.90:0/2995664348 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(94..94 src has 41..94) ==== 6288+0+0 (unknown 1714504682 0 0) 0x7efe6008b380 con 0x7efe6c10b890 2024-04-14T15:11:48.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:48.291+0000 7efe68ff9700 10 client.14814.objecter ms_dispatch 0x7efe6c001550 osd_map(94..94 src has 41..94) 2024-04-14T15:11:48.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:48.291+0000 7efe68ff9700 3 client.14814.objecter handle_osd_map got epochs [94,94] > 0 2024-04-14T15:11:48.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:48.291+0000 7efe68ff9700 3 client.14814.objecter handle_osd_map decoding full epoch 94 2024-04-14T15:11:48.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:48.291+0000 7efe68ff9700 20 client.14814.objecter dump_active .. 0 homeless 2024-04-14T15:11:48.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:48.291+0000 7efe68ff9700 10 client.14814.objecter ms_handle_connect 0x7efe54071300 2024-04-14T15:11:48.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:48.296+0000 7efe68ff9700 1 -- 172.21.15.90:0/2995664348 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7efe60058550 con 0x7efe6c10b890 2024-04-14T15:11:48.310 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:11:48.333 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:48.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:48.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:11:48.435 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:11:48.462 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:48.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:48.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:11:48.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:48.514+0000 7efe71dda700 1 -- 172.21.15.90:0/2995664348 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7efe6c19ba80 con 0x7efe6c10b890 2024-04-14T15:11:48.564 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:11:48.591 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:48.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:48.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:11:48.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:48 smithi092 ceph-mon[33966]: 6.1b scrub starts 2024-04-14T15:11:48.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:48 smithi092 ceph-mon[33966]: 6.1b scrub ok 2024-04-14T15:11:48.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:48 smithi092 ceph-mon[33966]: pgmap v60: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-14T15:11:48.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:48 smithi092 ceph-mon[33966]: 2.3 deep-scrub starts 2024-04-14T15:11:48.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:48 smithi092 ceph-mon[33966]: 2.3 deep-scrub ok 2024-04-14T15:11:48.693 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:11:48.721 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:48.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:48.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:11:48.823 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:11:48.849 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:48.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:48.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:11:48.950 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:11:48.957 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-14T15:11:48.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:48.976 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:48.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:11:49.078 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:11:49.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:49.103 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:49.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:11:49.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:49.184+0000 7efe68ff9700 1 -- 172.21.15.90:0/2995664348 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v95) ==== 87+0+0 (unknown 126117796 0 0) 0x7efe6005c200 con 0x7efe6c10b890 2024-04-14T15:11:49.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:49.186+0000 7efe71dda700 10 librados: watch_flush enter 2024-04-14T15:11:49.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:49.186+0000 7efe71dda700 10 librados: watch_flush exit 2024-04-14T15:11:49.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:49.186+0000 7efe71dda700 20 client.14814.objecter shutdown clearing up homeless session... 2024-04-14T15:11:49.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:49.186+0000 7efe71dda700 10 client.14814.objecter successfully canceled tick 2024-04-14T15:11:49.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:49.186+0000 7efe71dda700 1 -- 172.21.15.90:0/2995664348 >> v1:172.21.15.90:6800/420935427 conn(0x7efe54071300 legacy=0x7efe540737c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:49.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:49.186+0000 7efe71dda700 1 -- 172.21.15.90:0/2995664348 >> v1:172.21.15.90:6789/0 conn(0x7efe6c10b890 legacy=0x7efe6c1a5250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:49.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:49.186+0000 7efe71dda700 1 -- 172.21.15.90:0/2995664348 shutdown_connections 2024-04-14T15:11:49.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:49.186+0000 7efe71dda700 1 -- 172.21.15.90:0/2995664348 >> 172.21.15.90:0/2995664348 conn(0x7efe6c0fd290 msgr2=0x7efe6c1120a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:11:49.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:49.186+0000 7efe71dda700 1 -- 172.21.15.90:0/2995664348 shutdown_connections 2024-04-14T15:11:49.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:49.186+0000 7efe71dda700 1 -- 172.21.15.90:0/2995664348 wait complete. 2024-04-14T15:11:49.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:49.186+0000 7efe71dda700 1 librados: shutdown 2024-04-14T15:11:49.190 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is set 2024-04-14T15:11:49.204 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:11:49.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:49.231 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:49.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:11:49.332 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:11:49.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:49.358 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:49.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:11:49.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:49 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2995664348' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:11:49.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:49 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2995664348' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:11:49.459 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:11:49.483 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:49.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:49.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:11:49.585 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:11:49.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:49.609 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:49.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:11:49.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:49 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2995664348' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:11:49.710 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:11:49.738 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:49.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:49.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:11:49.840 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:11:49.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:49.868 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:49.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:11:49.969 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:11:49.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:49.998 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:49.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:11:50.098 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:11:50.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:50.124 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:50.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:11:50.225 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:11:50.253 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:50.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:50.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:11:50.354 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:11:50.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:50.380 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:50.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:11:50.435 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:50 smithi090 ceph-mon[30399]: 4.13 scrub starts 2024-04-14T15:11:50.435 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:50 smithi090 ceph-mon[30399]: 4.13 scrub ok 2024-04-14T15:11:50.435 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:50 smithi090 ceph-mon[30399]: pgmap v61: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:11:50.435 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:50 smithi090 ceph-mon[30399]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:11:50.435 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:50 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2995664348' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-14T15:11:50.435 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:50 smithi090 ceph-mon[30399]: osdmap e95: 8 total, 8 up, 8 in 2024-04-14T15:11:50.435 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:50 smithi090 ceph-mon[41670]: 4.13 scrub starts 2024-04-14T15:11:50.436 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:50 smithi090 ceph-mon[41670]: 4.13 scrub ok 2024-04-14T15:11:50.436 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:50 smithi090 ceph-mon[41670]: pgmap v61: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:11:50.436 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:50 smithi090 ceph-mon[41670]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:11:50.436 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:50 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2995664348' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-14T15:11:50.436 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:50 smithi090 ceph-mon[41670]: osdmap e95: 8 total, 8 up, 8 in 2024-04-14T15:11:50.482 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:11:50.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:50.507 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:50.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:11:50.608 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:11:50.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:50 smithi092 ceph-mon[33966]: 4.13 scrub starts 2024-04-14T15:11:50.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:50 smithi092 ceph-mon[33966]: 4.13 scrub ok 2024-04-14T15:11:50.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:50 smithi092 ceph-mon[33966]: pgmap v61: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:11:50.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:50 smithi092 ceph-mon[33966]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:11:50.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:50 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2995664348' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-14T15:11:50.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:50 smithi092 ceph-mon[33966]: osdmap e95: 8 total, 8 up, 8 in 2024-04-14T15:11:50.632 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:50.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:50.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:11:50.691 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:11:50 smithi090 podman[83857]: 2024-04-14 15:11:50.338109675 +0000 UTC m=+1.548651806 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-04-14T15:11:50.691 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:11:50 smithi090 podman[83857]: 2024-04-14 15:11:50.435903327 +0000 UTC m=+1.646445469 volume create 72c3f1ebb2f88129542df071edf80dc669c248e37d196c184d271ea1c320de81 2024-04-14T15:11:50.692 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:11:50 smithi090 podman[83857]: 2024-04-14 15:11:50.561779273 +0000 UTC m=+1.772321445 container create a1404ca24609bf7cccd9a82746fda5c420e34341a6dfd2db0b46113dda9ee0b0 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-14T15:11:50.734 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:11:50.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:50.762 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:50.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:11:50.863 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:11:50.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:50.889 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:50.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:11:50.990 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:11:51.014 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:51.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:51.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:11:51.116 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:11:51.139 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:51.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:51.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:11:51.192 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:11:50 smithi090 podman[83857]: 2024-04-14 15:11:50.856700993 +0000 UTC m=+2.067243121 container init a1404ca24609bf7cccd9a82746fda5c420e34341a6dfd2db0b46113dda9ee0b0 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-14T15:11:51.192 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:11:50 smithi090 podman[83857]: 2024-04-14 15:11:50.862108934 +0000 UTC m=+2.072651063 container start a1404ca24609bf7cccd9a82746fda5c420e34341a6dfd2db0b46113dda9ee0b0 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-14T15:11:51.192 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:11:50 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-alertmanager-a[84115]: ts=2024-04-14T15:11:50.874Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-04-14T15:11:51.192 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:11:50 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-alertmanager-a[84115]: ts=2024-04-14T15:11:50.874Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-04-14T15:11:51.192 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:11:50 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-alertmanager-a[84115]: ts=2024-04-14T15:11:50.875Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.90 port=9094 2024-04-14T15:11:51.192 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:11:50 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-alertmanager-a[84115]: ts=2024-04-14T15:11:50.876Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-04-14T15:11:51.192 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:11:50 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-alertmanager-a[84115]: ts=2024-04-14T15:11:50.900Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-14T15:11:51.192 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:11:50 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-alertmanager-a[84115]: ts=2024-04-14T15:11:50.900Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-14T15:11:51.192 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:11:50 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-alertmanager-a[84115]: ts=2024-04-14T15:11:50.903Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-04-14T15:11:51.192 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:11:50 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-alertmanager-a[84115]: ts=2024-04-14T15:11:50.903Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-04-14T15:11:51.193 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:11:50 smithi090 bash[83857]: a1404ca24609bf7cccd9a82746fda5c420e34341a6dfd2db0b46113dda9ee0b0 2024-04-14T15:11:51.193 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:11:50 smithi090 systemd[1]: Started Ceph alertmanager.a for 0e288098-fa70-11ee-bc8e-c7b262605968. 2024-04-14T15:11:51.241 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:11:51.264 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:51.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:51.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:11:51.366 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:11:51.392 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:51.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:51.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:11:51.494 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:11:51.518 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:51.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:51.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:11:51.620 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:11:51.643 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:51.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:51.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:11:51.745 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:11:51.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:51.770 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:51.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:11:51.871 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:11:51.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:51.895 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:51.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:11:51.936 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:11:51.996 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:11:52.019 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-14T15:11:52.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:52.021 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:52.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:11:52.122 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:11:52.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:52 smithi092 ceph-mon[33966]: pgmap v63: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:11:52.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:52 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:52.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:52 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:52.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:52 smithi092 ceph-mon[33966]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-14T15:11:52.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:52 smithi092 ceph-mon[33966]: Reconfiguring daemon prometheus.a on smithi092 2024-04-14T15:11:52.146 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:52.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:52.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:11:52.166 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:52 smithi090 ceph-mon[41670]: pgmap v63: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:11:52.166 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:52 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:52.166 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:52 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:52.166 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:52 smithi090 ceph-mon[41670]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-14T15:11:52.166 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:52 smithi090 ceph-mon[41670]: Reconfiguring daemon prometheus.a on smithi092 2024-04-14T15:11:52.167 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:52 smithi090 ceph-mon[30399]: pgmap v63: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:11:52.167 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:52 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:52.167 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:52 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:52.167 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:52 smithi090 ceph-mon[30399]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-14T15:11:52.167 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:52 smithi090 ceph-mon[30399]: Reconfiguring daemon prometheus.a on smithi092 2024-04-14T15:11:52.249 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:11:52.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:52.274 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:52.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:11:52.375 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:11:52.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:52.401 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:52.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:11:52.502 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:11:52.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:52.529 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:52.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:11:52.630 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:11:52.657 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:52.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:52.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:11:52.759 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:11:52.783 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:52.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:52.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:11:52.875 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-14T15:11:52.875 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:11:52.885 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:11:52.908 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:52.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:52.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:11:53.010 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:11:53.034 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:53.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:53.035 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:11:53.136 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:11:53.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:53.160 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:53.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:11:53.191 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:11:52 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-alertmanager-a[84115]: ts=2024-04-14T15:11:52.876Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000145644s 2024-04-14T15:11:53.261 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:11:53.285 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:53.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:53.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:11:53.387 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:11:53.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:53.411 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:53.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:11:53.512 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:11:53.535 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:53.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:53.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:11:53.637 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:11:53.662 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:53.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:53.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:11:53.764 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:11:53.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:53.788 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:53.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:11:53.889 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:11:53.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:53.913 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:53.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:11:54.014 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:11:54.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:54.039 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:54.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:11:54.141 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:11:54.164 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:54.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:54.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:11:54.191 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:11:53 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:11:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:11:54.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:54 smithi090 ceph-mon[41670]: pgmap v64: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-14T15:11:54.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:54 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:54.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:54 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:11:54.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:54 smithi090 ceph-mon[30399]: pgmap v64: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-14T15:11:54.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:54 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:54.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:54 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:11:54.266 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:11:54.291 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:54.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:54.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:11:54.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:54 smithi092 ceph-mon[33966]: pgmap v64: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-14T15:11:54.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:54 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:54.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:54 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:11:54.393 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:11:54.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:54.417 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:54.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:11:54.518 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:11:54.542 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:54.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:54.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:11:54.644 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:11:54.667 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:54.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:54.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:11:54.758 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-14T15:11:54.769 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:11:54.793 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:54.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:54.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:11:54.870 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:11:54 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:11:54.870 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:11:54 smithi092 systemd[1]: Stopping Ceph prometheus.a for 0e288098-fa70-11ee-bc8e-c7b262605968... 2024-04-14T15:11:54.895 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:11:54.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:54.919 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:54.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:11:55.020 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:11:55.043 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:55.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:55.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:11:55.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:55 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:11:55.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:55 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:11:55.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:55 smithi092 ceph-mon[33966]: osdmap e96: 8 total, 8 up, 8 in 2024-04-14T15:11:55.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:55 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:11:55.123 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:11:54 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[65425]: ts=2024-04-14T15:11:54.870Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-04-14T15:11:55.123 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:11:54 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[65425]: ts=2024-04-14T15:11:54.870Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-04-14T15:11:55.123 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:11:54 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[65425]: ts=2024-04-14T15:11:54.870Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-04-14T15:11:55.123 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:11:54 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[65425]: ts=2024-04-14T15:11:54.870Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-04-14T15:11:55.123 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:11:54 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[65425]: ts=2024-04-14T15:11:54.870Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-04-14T15:11:55.123 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:11:54 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[65425]: ts=2024-04-14T15:11:54.870Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-04-14T15:11:55.123 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:11:54 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[65425]: ts=2024-04-14T15:11:54.870Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-04-14T15:11:55.123 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:11:54 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[65425]: ts=2024-04-14T15:11:54.870Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-04-14T15:11:55.123 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:11:54 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[65425]: ts=2024-04-14T15:11:54.870Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-04-14T15:11:55.123 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:11:54 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[65425]: ts=2024-04-14T15:11:54.871Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-04-14T15:11:55.123 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:11:54 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[65425]: ts=2024-04-14T15:11:54.871Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-04-14T15:11:55.123 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:11:54 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[65425]: ts=2024-04-14T15:11:54.871Z caller=main.go:1147 level=info msg="See you next time!" 2024-04-14T15:11:55.123 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:11:54 smithi092 podman[70074]: 2024-04-14 15:11:54.880997145 +0000 UTC m=+0.105125183 container died 9fcf2f777e893dca73a9aa39d89766781c5169ef7a3ecc734c55e7b883824974 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T15:11:55.146 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:11:55.170 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:55.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:55.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:11:55.259 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:55 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:11:55.259 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:55 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:11:55.259 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:55 smithi090 ceph-mon[41670]: osdmap e96: 8 total, 8 up, 8 in 2024-04-14T15:11:55.259 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:55 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:11:55.259 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:55 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:11:55.260 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:55 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:11:55.260 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:55 smithi090 ceph-mon[30399]: osdmap e96: 8 total, 8 up, 8 in 2024-04-14T15:11:55.260 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:55 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:11:55.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.261+0000 7f1b513d7700 1 -- 172.21.15.90:0/2792779297 >> v1:172.21.15.92:6789/0 conn(0x7f1b4c10e310 legacy=0x7f1b4c1107d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:55.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.261+0000 7f1b513d7700 1 -- 172.21.15.90:0/2792779297 shutdown_connections 2024-04-14T15:11:55.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.261+0000 7f1b513d7700 1 -- 172.21.15.90:0/2792779297 >> 172.21.15.90:0/2792779297 conn(0x7f1b4c0fd290 msgr2=0x7f1b4c0ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:11:55.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.262+0000 7f1b513d7700 1 -- 172.21.15.90:0/2792779297 shutdown_connections 2024-04-14T15:11:55.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.262+0000 7f1b513d7700 1 -- 172.21.15.90:0/2792779297 wait complete. 2024-04-14T15:11:55.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.262+0000 7f1b513d7700 1 librados: starting msgr at 2024-04-14T15:11:55.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.262+0000 7f1b513d7700 1 librados: starting objecter 2024-04-14T15:11:55.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.262+0000 7f1b513d7700 1 Processor -- start 2024-04-14T15:11:55.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.262+0000 7f1b513d7700 1 -- start start 2024-04-14T15:11:55.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.262+0000 7f1b513d7700 1 librados: setting wanted keys 2024-04-14T15:11:55.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.262+0000 7f1b513d7700 1 librados: calling monclient init 2024-04-14T15:11:55.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.263+0000 7f1b513d7700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1b4c1a6a20 con 0x7f1b4c10e310 2024-04-14T15:11:55.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.263+0000 7f1b513d7700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1b4c1a7c60 con 0x7f1b4c10a740 2024-04-14T15:11:55.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.263+0000 7f1b513d7700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1b4c1a8ea0 con 0x7f1b4c107890 2024-04-14T15:11:55.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.263+0000 7f1b4affd700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f1b4c107890 0x7f1b4c10db20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:42456/0 (socket says 172.21.15.90:42456) 2024-04-14T15:11:55.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.263+0000 7f1b4affd700 1 -- 172.21.15.90:0/419239410 learned_addr learned my addr 172.21.15.90:0/419239410 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:11:55.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.263+0000 7f1b33fff700 10 client.?.objecter ms_handle_connect 0x7f1b4c10e310 2024-04-14T15:11:55.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.263+0000 7f1b33fff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:11:55.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.263+0000 7f1b33fff700 10 client.?.objecter ms_handle_connect 0x7f1b4c107890 2024-04-14T15:11:55.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.263+0000 7f1b33fff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:11:55.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.263+0000 7f1b33fff700 10 client.?.objecter ms_handle_connect 0x7f1b4c10a740 2024-04-14T15:11:55.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.263+0000 7f1b33fff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:11:55.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.263+0000 7f1b33fff700 1 -- 172.21.15.90:0/419239410 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3446904982 0 0) 0x7f1b4c1a8ea0 con 0x7f1b4c107890 2024-04-14T15:11:55.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.263+0000 7f1b33fff700 1 -- 172.21.15.90:0/419239410 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1b340031f0 con 0x7f1b4c107890 2024-04-14T15:11:55.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.263+0000 7f1b33fff700 1 -- 172.21.15.90:0/419239410 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3352982407 0 0) 0x7f1b4c1a7c60 con 0x7f1b4c10a740 2024-04-14T15:11:55.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.263+0000 7f1b33fff700 1 -- 172.21.15.90:0/419239410 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1b4c1a8ea0 con 0x7f1b4c10a740 2024-04-14T15:11:55.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.263+0000 7f1b33fff700 1 -- 172.21.15.90:0/419239410 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1082684434 0 0) 0x7f1b4c1a6a20 con 0x7f1b4c10e310 2024-04-14T15:11:55.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.263+0000 7f1b33fff700 1 -- 172.21.15.90:0/419239410 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1b4c1a7c60 con 0x7f1b4c10e310 2024-04-14T15:11:55.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.263+0000 7f1b33fff700 1 -- 172.21.15.90:0/419239410 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 690801925 0 0) 0x7f1b4c1a8ea0 con 0x7f1b4c10a740 2024-04-14T15:11:55.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.263+0000 7f1b33fff700 1 -- 172.21.15.90:0/419239410 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1b4c1a6a20 con 0x7f1b4c10a740 2024-04-14T15:11:55.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.263+0000 7f1b33fff700 1 -- 172.21.15.90:0/419239410 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f1b400017e0 con 0x7f1b4c10a740 2024-04-14T15:11:55.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.263+0000 7f1b33fff700 1 -- 172.21.15.90:0/419239410 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1821136655 0 0) 0x7f1b340031f0 con 0x7f1b4c107890 2024-04-14T15:11:55.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.263+0000 7f1b33fff700 1 -- 172.21.15.90:0/419239410 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1b4c1a8ea0 con 0x7f1b4c107890 2024-04-14T15:11:55.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.263+0000 7f1b33fff700 1 -- 172.21.15.90:0/419239410 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f1b3c0018d0 con 0x7f1b4c107890 2024-04-14T15:11:55.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.264+0000 7f1b33fff700 1 -- 172.21.15.90:0/419239410 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3094854916 0 0) 0x7f1b4c1a7c60 con 0x7f1b4c10e310 2024-04-14T15:11:55.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.264+0000 7f1b33fff700 1 -- 172.21.15.90:0/419239410 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1b340031f0 con 0x7f1b4c10e310 2024-04-14T15:11:55.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.264+0000 7f1b33fff700 1 -- 172.21.15.90:0/419239410 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f1b440016f0 con 0x7f1b4c10e310 2024-04-14T15:11:55.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.264+0000 7f1b33fff700 1 -- 172.21.15.90:0/419239410 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2460368846 0 0) 0x7f1b4c1a6a20 con 0x7f1b4c10a740 2024-04-14T15:11:55.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.264+0000 7f1b33fff700 1 -- 172.21.15.90:0/419239410 >> v1:172.21.15.90:6790/0 conn(0x7f1b4c107890 legacy=0x7f1b4c10db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:55.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.264+0000 7f1b33fff700 1 -- 172.21.15.90:0/419239410 >> v1:172.21.15.90:6789/0 conn(0x7f1b4c10e310 legacy=0x7f1b4c1a5120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:55.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.264+0000 7f1b33fff700 1 -- 172.21.15.90:0/419239410 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b4c1aa0e0 con 0x7f1b4c10a740 2024-04-14T15:11:55.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.264+0000 7f1b513d7700 1 -- 172.21.15.90:0/419239410 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1b4c1a90d0 con 0x7f1b4c10a740 2024-04-14T15:11:55.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.264+0000 7f1b513d7700 10 client.24715.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:11:55.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.264+0000 7f1b513d7700 1 -- 172.21.15.90:0/419239410 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f1b4c1a9590 con 0x7f1b4c10a740 2024-04-14T15:11:55.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.264+0000 7f1b513d7700 1 librados: init done 2024-04-14T15:11:55.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.264+0000 7f1b33fff700 1 -- 172.21.15.90:0/419239410 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f1b40003b40 con 0x7f1b4c10a740 2024-04-14T15:11:55.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.264+0000 7f1b33fff700 1 -- 172.21.15.90:0/419239410 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f1b40004aa0 con 0x7f1b4c10a740 2024-04-14T15:11:55.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.265+0000 7f1b513d7700 1 -- 172.21.15.90:0/419239410 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f1b38005100 con 0x7f1b4c10a740 2024-04-14T15:11:55.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.265+0000 7f1b33fff700 1 -- 172.21.15.90:0/419239410 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f1b4001b4b0 con 0x7f1b4c10a740 2024-04-14T15:11:55.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.266+0000 7f1b33fff700 1 -- 172.21.15.90:0/419239410 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(96..96 src has 41..96) ==== 6288+0+0 (unknown 4040766927 0 0) 0x7f1b40088ff0 con 0x7f1b4c10a740 2024-04-14T15:11:55.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.266+0000 7f1b33fff700 10 client.24715.objecter ms_dispatch 0x7f1b4c001550 osd_map(96..96 src has 41..96) 2024-04-14T15:11:55.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.266+0000 7f1b33fff700 3 client.24715.objecter handle_osd_map got epochs [96,96] > 0 2024-04-14T15:11:55.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.266+0000 7f1b33fff700 3 client.24715.objecter handle_osd_map decoding full epoch 96 2024-04-14T15:11:55.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.266+0000 7f1b33fff700 20 client.24715.objecter dump_active .. 0 homeless 2024-04-14T15:11:55.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.266+0000 7f1b33fff700 10 client.24715.objecter ms_handle_connect 0x7f1b340713b0 2024-04-14T15:11:55.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.266+0000 7f1b33fff700 1 -- 172.21.15.90:0/419239410 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v96) ==== 54+0+16 (unknown 2433924269 0 2420000192) 0x7f1b40089430 con 0x7f1b4c10a740 2024-04-14T15:11:55.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.266+0000 7f1b513d7700 10 client.24715.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:11:55.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.266+0000 7f1b513d7700 15 client.24715.objecter _session_command_op_assign -1 1 2024-04-14T15:11:55.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.266+0000 7f1b513d7700 20 client.24715.objecter _get_session s=0x7f1b380053e0 osd=0 3 2024-04-14T15:11:55.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.266+0000 7f1b513d7700 20 client.24715.objecter put_session s=0x7f1b380053e0 osd=0 3 2024-04-14T15:11:55.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.266+0000 7f1b513d7700 20 client.24715.objecter _get_session s=0x7f1b380053e0 osd=0 3 2024-04-14T15:11:55.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.266+0000 7f1b513d7700 15 client.24715.objecter _session_command_op_remove -1 1 2024-04-14T15:11:55.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.266+0000 7f1b513d7700 20 client.24715.objecter get_session s=0x7f1b380053e0 osd=0 3 2024-04-14T15:11:55.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.266+0000 7f1b513d7700 15 client.24715.objecter _session_command_op_assign 0 1 2024-04-14T15:11:55.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.266+0000 7f1b513d7700 20 client.24715.objecter put_session s=0x7f1b380053e0 osd=0 4 2024-04-14T15:11:55.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.266+0000 7f1b513d7700 10 client.24715.objecter _send_command 1 2024-04-14T15:11:55.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.266+0000 7f1b513d7700 1 -- 172.21.15.90:0/419239410 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1b38004b70 con 0x7f1b38004730 2024-04-14T15:11:55.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.267+0000 7f1b33fff700 10 client.24715.objecter ms_handle_connect 0x7f1b38004730 2024-04-14T15:11:55.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.267+0000 7f1b33fff700 1 -- 172.21.15.90:0/419239410 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1b38004b70 con 0x7f1b38004730 2024-04-14T15:11:55.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.267+0000 7f1b33fff700 10 client.24715.objecter ms_dispatch 0x7f1b4c001550 command_reply(tid 1: 0 ) 2024-04-14T15:11:55.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.267+0000 7f1b33fff700 10 client.24715.objecter _finish_command 1 = system:0 2024-04-14T15:11:55.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.267+0000 7f1b33fff700 20 client.24715.objecter put_session s=0x7f1b380053e0 osd=0 4 2024-04-14T15:11:55.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.267+0000 7f1b33fff700 15 client.24715.objecter _session_command_op_remove 0 1 2024-04-14T15:11:55.274 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:11:55.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.285+0000 7f1b513d7700 10 client.24715.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:11:55.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.285+0000 7f1b513d7700 15 client.24715.objecter _session_command_op_assign -1 2 2024-04-14T15:11:55.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.285+0000 7f1b513d7700 20 client.24715.objecter _get_session s=0x7f1b380053e0 osd=0 3 2024-04-14T15:11:55.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.285+0000 7f1b513d7700 20 client.24715.objecter put_session s=0x7f1b380053e0 osd=0 3 2024-04-14T15:11:55.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.285+0000 7f1b513d7700 20 client.24715.objecter _get_session s=0x7f1b380053e0 osd=0 3 2024-04-14T15:11:55.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.285+0000 7f1b513d7700 15 client.24715.objecter _session_command_op_remove -1 2 2024-04-14T15:11:55.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.285+0000 7f1b513d7700 20 client.24715.objecter get_session s=0x7f1b380053e0 osd=0 3 2024-04-14T15:11:55.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.285+0000 7f1b513d7700 15 client.24715.objecter _session_command_op_assign 0 2 2024-04-14T15:11:55.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.285+0000 7f1b513d7700 20 client.24715.objecter put_session s=0x7f1b380053e0 osd=0 4 2024-04-14T15:11:55.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.285+0000 7f1b513d7700 10 client.24715.objecter _send_command 2 2024-04-14T15:11:55.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.285+0000 7f1b513d7700 1 -- 172.21.15.90:0/419239410 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f1b380079f0 con 0x7f1b38004730 2024-04-14T15:11:55.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.287+0000 7f1b33fff700 1 -- 172.21.15.90:0/419239410 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1b380079f0 con 0x7f1b38004730 2024-04-14T15:11:55.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.287+0000 7f1b33fff700 10 client.24715.objecter ms_dispatch 0x7f1b4c001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:11:55.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.287+0000 7f1b33fff700 10 client.24715.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:11:55.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.287+0000 7f1b33fff700 20 client.24715.objecter put_session s=0x7f1b380053e0 osd=0 4 2024-04-14T15:11:55.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.287+0000 7f1b33fff700 15 client.24715.objecter _session_command_op_remove 0 2 2024-04-14T15:11:55.288 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:11:55.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.287+0000 7f1b513d7700 10 client.24715.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:11:55.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.287+0000 7f1b513d7700 15 client.24715.objecter _session_command_op_assign -1 3 2024-04-14T15:11:55.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.287+0000 7f1b513d7700 20 client.24715.objecter _get_session s=0x7f1b38004130 osd=1 3 2024-04-14T15:11:55.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.287+0000 7f1b513d7700 20 client.24715.objecter put_session s=0x7f1b38004130 osd=1 3 2024-04-14T15:11:55.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.287+0000 7f1b513d7700 20 client.24715.objecter _get_session s=0x7f1b38004130 osd=1 3 2024-04-14T15:11:55.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.287+0000 7f1b513d7700 15 client.24715.objecter _session_command_op_remove -1 3 2024-04-14T15:11:55.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.287+0000 7f1b513d7700 20 client.24715.objecter get_session s=0x7f1b38004130 osd=1 3 2024-04-14T15:11:55.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.287+0000 7f1b513d7700 15 client.24715.objecter _session_command_op_assign 1 3 2024-04-14T15:11:55.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.287+0000 7f1b513d7700 20 client.24715.objecter put_session s=0x7f1b38004130 osd=1 4 2024-04-14T15:11:55.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.287+0000 7f1b513d7700 10 client.24715.objecter _send_command 3 2024-04-14T15:11:55.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.287+0000 7f1b513d7700 1 -- 172.21.15.90:0/419239410 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f1b380022a0 con 0x7f1b38004310 2024-04-14T15:11:55.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.287+0000 7f1b33fff700 10 client.24715.objecter ms_handle_connect 0x7f1b38004310 2024-04-14T15:11:55.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.288+0000 7f1b33fff700 1 -- 172.21.15.90:0/419239410 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1b380022a0 con 0x7f1b38004310 2024-04-14T15:11:55.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.288+0000 7f1b33fff700 10 client.24715.objecter ms_dispatch 0x7f1b4c001550 command_reply(tid 3: 0 ) 2024-04-14T15:11:55.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.288+0000 7f1b33fff700 10 client.24715.objecter _finish_command 3 = system:0 2024-04-14T15:11:55.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.288+0000 7f1b33fff700 20 client.24715.objecter put_session s=0x7f1b38004130 osd=1 4 2024-04-14T15:11:55.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.288+0000 7f1b33fff700 15 client.24715.objecter _session_command_op_remove 1 3 2024-04-14T15:11:55.297 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:55.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:55.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:11:55.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.306+0000 7f1b513d7700 10 client.24715.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:11:55.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.306+0000 7f1b513d7700 15 client.24715.objecter _session_command_op_assign -1 4 2024-04-14T15:11:55.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.306+0000 7f1b513d7700 20 client.24715.objecter _get_session s=0x7f1b38004130 osd=1 3 2024-04-14T15:11:55.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.306+0000 7f1b513d7700 20 client.24715.objecter put_session s=0x7f1b38004130 osd=1 3 2024-04-14T15:11:55.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.306+0000 7f1b513d7700 20 client.24715.objecter _get_session s=0x7f1b38004130 osd=1 3 2024-04-14T15:11:55.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.306+0000 7f1b513d7700 15 client.24715.objecter _session_command_op_remove -1 4 2024-04-14T15:11:55.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.306+0000 7f1b513d7700 20 client.24715.objecter get_session s=0x7f1b38004130 osd=1 3 2024-04-14T15:11:55.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.306+0000 7f1b513d7700 15 client.24715.objecter _session_command_op_assign 1 4 2024-04-14T15:11:55.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.306+0000 7f1b513d7700 20 client.24715.objecter put_session s=0x7f1b38004130 osd=1 4 2024-04-14T15:11:55.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.306+0000 7f1b513d7700 10 client.24715.objecter _send_command 4 2024-04-14T15:11:55.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.306+0000 7f1b513d7700 1 -- 172.21.15.90:0/419239410 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f1b38001720 con 0x7f1b38004310 2024-04-14T15:11:55.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.308+0000 7f1b33fff700 1 -- 172.21.15.90:0/419239410 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1b38001720 con 0x7f1b38004310 2024-04-14T15:11:55.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.308+0000 7f1b33fff700 10 client.24715.objecter ms_dispatch 0x7f1b4c001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:11:55.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.308+0000 7f1b33fff700 10 client.24715.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:11:55.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.308+0000 7f1b33fff700 20 client.24715.objecter put_session s=0x7f1b38004130 osd=1 4 2024-04-14T15:11:55.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.308+0000 7f1b33fff700 15 client.24715.objecter _session_command_op_remove 1 4 2024-04-14T15:11:55.309 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:11:55.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.308+0000 7f1b513d7700 10 client.24715.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:11:55.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.308+0000 7f1b513d7700 15 client.24715.objecter _session_command_op_assign -1 5 2024-04-14T15:11:55.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.308+0000 7f1b513d7700 20 client.24715.objecter _get_session s=0x7f1b38001980 osd=2 3 2024-04-14T15:11:55.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.308+0000 7f1b513d7700 20 client.24715.objecter put_session s=0x7f1b38001980 osd=2 3 2024-04-14T15:11:55.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.308+0000 7f1b513d7700 20 client.24715.objecter _get_session s=0x7f1b38001980 osd=2 3 2024-04-14T15:11:55.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.308+0000 7f1b513d7700 15 client.24715.objecter _session_command_op_remove -1 5 2024-04-14T15:11:55.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.308+0000 7f1b513d7700 20 client.24715.objecter get_session s=0x7f1b38001980 osd=2 3 2024-04-14T15:11:55.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.308+0000 7f1b513d7700 15 client.24715.objecter _session_command_op_assign 2 5 2024-04-14T15:11:55.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.308+0000 7f1b513d7700 20 client.24715.objecter put_session s=0x7f1b38001980 osd=2 4 2024-04-14T15:11:55.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.308+0000 7f1b513d7700 10 client.24715.objecter _send_command 5 2024-04-14T15:11:55.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.308+0000 7f1b513d7700 1 -- 172.21.15.90:0/419239410 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f1b38013380 con 0x7f1b3800d5d0 2024-04-14T15:11:55.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.308+0000 7f1b33fff700 10 client.24715.objecter ms_handle_connect 0x7f1b3800d5d0 2024-04-14T15:11:55.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.309+0000 7f1b33fff700 1 -- 172.21.15.90:0/419239410 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1b38013380 con 0x7f1b3800d5d0 2024-04-14T15:11:55.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.309+0000 7f1b33fff700 10 client.24715.objecter ms_dispatch 0x7f1b4c001550 command_reply(tid 5: 0 ) 2024-04-14T15:11:55.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.309+0000 7f1b33fff700 10 client.24715.objecter _finish_command 5 = system:0 2024-04-14T15:11:55.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.309+0000 7f1b33fff700 20 client.24715.objecter put_session s=0x7f1b38001980 osd=2 4 2024-04-14T15:11:55.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.309+0000 7f1b33fff700 15 client.24715.objecter _session_command_op_remove 2 5 2024-04-14T15:11:55.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.326+0000 7f1b513d7700 10 client.24715.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:11:55.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.326+0000 7f1b513d7700 15 client.24715.objecter _session_command_op_assign -1 6 2024-04-14T15:11:55.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.326+0000 7f1b513d7700 20 client.24715.objecter _get_session s=0x7f1b38001980 osd=2 3 2024-04-14T15:11:55.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.326+0000 7f1b513d7700 20 client.24715.objecter put_session s=0x7f1b38001980 osd=2 3 2024-04-14T15:11:55.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.326+0000 7f1b513d7700 20 client.24715.objecter _get_session s=0x7f1b38001980 osd=2 3 2024-04-14T15:11:55.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.326+0000 7f1b513d7700 15 client.24715.objecter _session_command_op_remove -1 6 2024-04-14T15:11:55.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.326+0000 7f1b513d7700 20 client.24715.objecter get_session s=0x7f1b38001980 osd=2 3 2024-04-14T15:11:55.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.326+0000 7f1b513d7700 15 client.24715.objecter _session_command_op_assign 2 6 2024-04-14T15:11:55.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.326+0000 7f1b513d7700 20 client.24715.objecter put_session s=0x7f1b38001980 osd=2 4 2024-04-14T15:11:55.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.326+0000 7f1b513d7700 10 client.24715.objecter _send_command 6 2024-04-14T15:11:55.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.326+0000 7f1b513d7700 1 -- 172.21.15.90:0/419239410 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f1b38008100 con 0x7f1b3800d5d0 2024-04-14T15:11:55.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.327+0000 7f1b33fff700 1 -- 172.21.15.90:0/419239410 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1b38008100 con 0x7f1b3800d5d0 2024-04-14T15:11:55.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.327+0000 7f1b33fff700 10 client.24715.objecter ms_dispatch 0x7f1b4c001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:11:55.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.327+0000 7f1b33fff700 10 client.24715.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:11:55.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.327+0000 7f1b33fff700 20 client.24715.objecter put_session s=0x7f1b38001980 osd=2 4 2024-04-14T15:11:55.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.327+0000 7f1b33fff700 15 client.24715.objecter _session_command_op_remove 2 6 2024-04-14T15:11:55.328 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:11:55.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.327+0000 7f1b513d7700 10 client.24715.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:11:55.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.327+0000 7f1b513d7700 15 client.24715.objecter _session_command_op_assign -1 7 2024-04-14T15:11:55.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.327+0000 7f1b513d7700 20 client.24715.objecter _get_session s=0x7f1b38008390 osd=3 3 2024-04-14T15:11:55.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.327+0000 7f1b513d7700 20 client.24715.objecter put_session s=0x7f1b38008390 osd=3 3 2024-04-14T15:11:55.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.327+0000 7f1b513d7700 20 client.24715.objecter _get_session s=0x7f1b38008390 osd=3 3 2024-04-14T15:11:55.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.327+0000 7f1b513d7700 15 client.24715.objecter _session_command_op_remove -1 7 2024-04-14T15:11:55.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.327+0000 7f1b513d7700 20 client.24715.objecter get_session s=0x7f1b38008390 osd=3 3 2024-04-14T15:11:55.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.327+0000 7f1b513d7700 15 client.24715.objecter _session_command_op_assign 3 7 2024-04-14T15:11:55.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.327+0000 7f1b513d7700 20 client.24715.objecter put_session s=0x7f1b38008390 osd=3 4 2024-04-14T15:11:55.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.327+0000 7f1b513d7700 10 client.24715.objecter _send_command 7 2024-04-14T15:11:55.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.327+0000 7f1b513d7700 1 -- 172.21.15.90:0/419239410 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f1b380181c0 con 0x7f1b380146b0 2024-04-14T15:11:55.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.328+0000 7f1b33fff700 10 client.24715.objecter ms_handle_connect 0x7f1b380146b0 2024-04-14T15:11:55.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.328+0000 7f1b33fff700 1 -- 172.21.15.90:0/419239410 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1b380181c0 con 0x7f1b380146b0 2024-04-14T15:11:55.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.328+0000 7f1b33fff700 10 client.24715.objecter ms_dispatch 0x7f1b4c001550 command_reply(tid 7: 0 ) 2024-04-14T15:11:55.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.328+0000 7f1b33fff700 10 client.24715.objecter _finish_command 7 = system:0 2024-04-14T15:11:55.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.328+0000 7f1b33fff700 20 client.24715.objecter put_session s=0x7f1b38008390 osd=3 4 2024-04-14T15:11:55.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.328+0000 7f1b33fff700 15 client.24715.objecter _session_command_op_remove 3 7 2024-04-14T15:11:55.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.345+0000 7f1b513d7700 10 client.24715.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:11:55.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.345+0000 7f1b513d7700 15 client.24715.objecter _session_command_op_assign -1 8 2024-04-14T15:11:55.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.345+0000 7f1b513d7700 20 client.24715.objecter _get_session s=0x7f1b38008390 osd=3 3 2024-04-14T15:11:55.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.345+0000 7f1b513d7700 20 client.24715.objecter put_session s=0x7f1b38008390 osd=3 3 2024-04-14T15:11:55.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.345+0000 7f1b513d7700 20 client.24715.objecter _get_session s=0x7f1b38008390 osd=3 3 2024-04-14T15:11:55.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.345+0000 7f1b513d7700 15 client.24715.objecter _session_command_op_remove -1 8 2024-04-14T15:11:55.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.345+0000 7f1b513d7700 20 client.24715.objecter get_session s=0x7f1b38008390 osd=3 3 2024-04-14T15:11:55.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.345+0000 7f1b513d7700 15 client.24715.objecter _session_command_op_assign 3 8 2024-04-14T15:11:55.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.345+0000 7f1b513d7700 20 client.24715.objecter put_session s=0x7f1b38008390 osd=3 4 2024-04-14T15:11:55.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.345+0000 7f1b513d7700 10 client.24715.objecter _send_command 8 2024-04-14T15:11:55.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.345+0000 7f1b513d7700 1 -- 172.21.15.90:0/419239410 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f1b3800c9b0 con 0x7f1b380146b0 2024-04-14T15:11:55.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.348+0000 7f1b33fff700 1 -- 172.21.15.90:0/419239410 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1b3800c9b0 con 0x7f1b380146b0 2024-04-14T15:11:55.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.348+0000 7f1b33fff700 10 client.24715.objecter ms_dispatch 0x7f1b4c001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:11:55.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.348+0000 7f1b33fff700 10 client.24715.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:11:55.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.348+0000 7f1b33fff700 20 client.24715.objecter put_session s=0x7f1b38008390 osd=3 4 2024-04-14T15:11:55.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.348+0000 7f1b33fff700 15 client.24715.objecter _session_command_op_remove 3 8 2024-04-14T15:11:55.348 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:11:55.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.348+0000 7f1b513d7700 10 client.24715.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:11:55.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.348+0000 7f1b513d7700 15 client.24715.objecter _session_command_op_assign -1 9 2024-04-14T15:11:55.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.348+0000 7f1b513d7700 20 client.24715.objecter _get_session s=0x7f1b3800c9b0 osd=4 3 2024-04-14T15:11:55.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.348+0000 7f1b513d7700 20 client.24715.objecter put_session s=0x7f1b3800c9b0 osd=4 3 2024-04-14T15:11:55.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.348+0000 7f1b513d7700 20 client.24715.objecter _get_session s=0x7f1b3800c9b0 osd=4 3 2024-04-14T15:11:55.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.348+0000 7f1b513d7700 15 client.24715.objecter _session_command_op_remove -1 9 2024-04-14T15:11:55.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.348+0000 7f1b513d7700 20 client.24715.objecter get_session s=0x7f1b3800c9b0 osd=4 3 2024-04-14T15:11:55.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.348+0000 7f1b513d7700 15 client.24715.objecter _session_command_op_assign 4 9 2024-04-14T15:11:55.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.348+0000 7f1b513d7700 20 client.24715.objecter put_session s=0x7f1b3800c9b0 osd=4 4 2024-04-14T15:11:55.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.348+0000 7f1b513d7700 10 client.24715.objecter _send_command 9 2024-04-14T15:11:55.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.348+0000 7f1b513d7700 1 -- 172.21.15.90:0/419239410 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f1b3800cc90 con 0x7f1b38014210 2024-04-14T15:11:55.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.348+0000 7f1b33fff700 10 client.24715.objecter ms_handle_connect 0x7f1b38014210 2024-04-14T15:11:55.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.349+0000 7f1b33fff700 1 -- 172.21.15.90:0/419239410 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1b3800cc90 con 0x7f1b38014210 2024-04-14T15:11:55.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.349+0000 7f1b33fff700 10 client.24715.objecter ms_dispatch 0x7f1b4c001550 command_reply(tid 9: 0 ) 2024-04-14T15:11:55.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.349+0000 7f1b33fff700 10 client.24715.objecter _finish_command 9 = system:0 2024-04-14T15:11:55.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.349+0000 7f1b33fff700 20 client.24715.objecter put_session s=0x7f1b3800c9b0 osd=4 4 2024-04-14T15:11:55.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.349+0000 7f1b33fff700 15 client.24715.objecter _session_command_op_remove 4 9 2024-04-14T15:11:55.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.365+0000 7f1b513d7700 10 client.24715.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:11:55.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.365+0000 7f1b513d7700 15 client.24715.objecter _session_command_op_assign -1 10 2024-04-14T15:11:55.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.365+0000 7f1b513d7700 20 client.24715.objecter _get_session s=0x7f1b3800c9b0 osd=4 3 2024-04-14T15:11:55.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.365+0000 7f1b513d7700 20 client.24715.objecter put_session s=0x7f1b3800c9b0 osd=4 3 2024-04-14T15:11:55.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.365+0000 7f1b513d7700 20 client.24715.objecter _get_session s=0x7f1b3800c9b0 osd=4 3 2024-04-14T15:11:55.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.365+0000 7f1b513d7700 15 client.24715.objecter _session_command_op_remove -1 10 2024-04-14T15:11:55.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.365+0000 7f1b513d7700 20 client.24715.objecter get_session s=0x7f1b3800c9b0 osd=4 3 2024-04-14T15:11:55.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.365+0000 7f1b513d7700 15 client.24715.objecter _session_command_op_assign 4 10 2024-04-14T15:11:55.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.365+0000 7f1b513d7700 20 client.24715.objecter put_session s=0x7f1b3800c9b0 osd=4 4 2024-04-14T15:11:55.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.365+0000 7f1b513d7700 10 client.24715.objecter _send_command 10 2024-04-14T15:11:55.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.365+0000 7f1b513d7700 1 -- 172.21.15.90:0/419239410 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f1b3801e3f0 con 0x7f1b38014210 2024-04-14T15:11:55.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.367+0000 7f1b33fff700 1 -- 172.21.15.90:0/419239410 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1b3801e3f0 con 0x7f1b38014210 2024-04-14T15:11:55.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.367+0000 7f1b33fff700 10 client.24715.objecter ms_dispatch 0x7f1b4c001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:11:55.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.367+0000 7f1b33fff700 10 client.24715.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:11:55.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.367+0000 7f1b33fff700 20 client.24715.objecter put_session s=0x7f1b3800c9b0 osd=4 4 2024-04-14T15:11:55.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.367+0000 7f1b33fff700 15 client.24715.objecter _session_command_op_remove 4 10 2024-04-14T15:11:55.368 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:11:55.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.367+0000 7f1b513d7700 10 client.24715.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:11:55.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.367+0000 7f1b513d7700 15 client.24715.objecter _session_command_op_assign -1 11 2024-04-14T15:11:55.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.367+0000 7f1b513d7700 20 client.24715.objecter _get_session s=0x7f1b3801e3f0 osd=5 3 2024-04-14T15:11:55.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.367+0000 7f1b513d7700 20 client.24715.objecter put_session s=0x7f1b3801e3f0 osd=5 3 2024-04-14T15:11:55.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.367+0000 7f1b513d7700 20 client.24715.objecter _get_session s=0x7f1b3801e3f0 osd=5 3 2024-04-14T15:11:55.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.367+0000 7f1b513d7700 15 client.24715.objecter _session_command_op_remove -1 11 2024-04-14T15:11:55.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.367+0000 7f1b513d7700 20 client.24715.objecter get_session s=0x7f1b3801e3f0 osd=5 3 2024-04-14T15:11:55.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.367+0000 7f1b513d7700 15 client.24715.objecter _session_command_op_assign 5 11 2024-04-14T15:11:55.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.367+0000 7f1b513d7700 20 client.24715.objecter put_session s=0x7f1b3801e3f0 osd=5 4 2024-04-14T15:11:55.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.367+0000 7f1b513d7700 10 client.24715.objecter _send_command 11 2024-04-14T15:11:55.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.367+0000 7f1b513d7700 1 -- 172.21.15.90:0/419239410 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f1b3801efa0 con 0x7f1b3801eb10 2024-04-14T15:11:55.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.368+0000 7f1b33fff700 10 client.24715.objecter ms_handle_connect 0x7f1b3801eb10 2024-04-14T15:11:55.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.368+0000 7f1b33fff700 1 -- 172.21.15.90:0/419239410 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1b3801efa0 con 0x7f1b3801eb10 2024-04-14T15:11:55.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.368+0000 7f1b33fff700 10 client.24715.objecter ms_dispatch 0x7f1b4c001550 command_reply(tid 11: 0 ) 2024-04-14T15:11:55.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.368+0000 7f1b33fff700 10 client.24715.objecter _finish_command 11 = system:0 2024-04-14T15:11:55.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.368+0000 7f1b33fff700 20 client.24715.objecter put_session s=0x7f1b3801e3f0 osd=5 4 2024-04-14T15:11:55.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.368+0000 7f1b33fff700 15 client.24715.objecter _session_command_op_remove 5 11 2024-04-14T15:11:55.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.384+0000 7f1b513d7700 10 client.24715.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:11:55.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.384+0000 7f1b513d7700 15 client.24715.objecter _session_command_op_assign -1 12 2024-04-14T15:11:55.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.384+0000 7f1b513d7700 20 client.24715.objecter _get_session s=0x7f1b3801e3f0 osd=5 3 2024-04-14T15:11:55.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.384+0000 7f1b513d7700 20 client.24715.objecter put_session s=0x7f1b3801e3f0 osd=5 3 2024-04-14T15:11:55.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.384+0000 7f1b513d7700 20 client.24715.objecter _get_session s=0x7f1b3801e3f0 osd=5 3 2024-04-14T15:11:55.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.384+0000 7f1b513d7700 15 client.24715.objecter _session_command_op_remove -1 12 2024-04-14T15:11:55.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.384+0000 7f1b513d7700 20 client.24715.objecter get_session s=0x7f1b3801e3f0 osd=5 3 2024-04-14T15:11:55.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.384+0000 7f1b513d7700 15 client.24715.objecter _session_command_op_assign 5 12 2024-04-14T15:11:55.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.384+0000 7f1b513d7700 20 client.24715.objecter put_session s=0x7f1b3801e3f0 osd=5 4 2024-04-14T15:11:55.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.384+0000 7f1b513d7700 10 client.24715.objecter _send_command 12 2024-04-14T15:11:55.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.384+0000 7f1b513d7700 1 -- 172.21.15.90:0/419239410 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f1b3801f220 con 0x7f1b3801eb10 2024-04-14T15:11:55.386 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:11:55.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.386+0000 7f1b33fff700 1 -- 172.21.15.90:0/419239410 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1b3801f220 con 0x7f1b3801eb10 2024-04-14T15:11:55.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.386+0000 7f1b33fff700 10 client.24715.objecter ms_dispatch 0x7f1b4c001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:11:55.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.386+0000 7f1b33fff700 10 client.24715.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:11:55.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.386+0000 7f1b33fff700 20 client.24715.objecter put_session s=0x7f1b3801e3f0 osd=5 4 2024-04-14T15:11:55.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.386+0000 7f1b33fff700 15 client.24715.objecter _session_command_op_remove 5 12 2024-04-14T15:11:55.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.386+0000 7f1b513d7700 10 client.24715.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:11:55.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.386+0000 7f1b513d7700 15 client.24715.objecter _session_command_op_assign -1 13 2024-04-14T15:11:55.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.386+0000 7f1b513d7700 20 client.24715.objecter _get_session s=0x7f1b3801efa0 osd=6 3 2024-04-14T15:11:55.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.386+0000 7f1b513d7700 20 client.24715.objecter put_session s=0x7f1b3801efa0 osd=6 3 2024-04-14T15:11:55.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.386+0000 7f1b513d7700 20 client.24715.objecter _get_session s=0x7f1b3801efa0 osd=6 3 2024-04-14T15:11:55.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.386+0000 7f1b513d7700 15 client.24715.objecter _session_command_op_remove -1 13 2024-04-14T15:11:55.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.386+0000 7f1b513d7700 20 client.24715.objecter get_session s=0x7f1b3801efa0 osd=6 3 2024-04-14T15:11:55.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.386+0000 7f1b513d7700 15 client.24715.objecter _session_command_op_assign 6 13 2024-04-14T15:11:55.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.386+0000 7f1b513d7700 20 client.24715.objecter put_session s=0x7f1b3801efa0 osd=6 4 2024-04-14T15:11:55.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.386+0000 7f1b513d7700 10 client.24715.objecter _send_command 13 2024-04-14T15:11:55.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.386+0000 7f1b513d7700 1 -- 172.21.15.90:0/419239410 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f1b38025610 con 0x7f1b38021b00 2024-04-14T15:11:55.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.387+0000 7f1b33fff700 10 client.24715.objecter ms_handle_connect 0x7f1b38021b00 2024-04-14T15:11:55.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.388+0000 7f1b33fff700 1 -- 172.21.15.90:0/419239410 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1b38025610 con 0x7f1b38021b00 2024-04-14T15:11:55.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.388+0000 7f1b33fff700 10 client.24715.objecter ms_dispatch 0x7f1b4c001550 command_reply(tid 13: 0 ) 2024-04-14T15:11:55.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.388+0000 7f1b33fff700 10 client.24715.objecter _finish_command 13 = system:0 2024-04-14T15:11:55.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.388+0000 7f1b33fff700 20 client.24715.objecter put_session s=0x7f1b3801efa0 osd=6 4 2024-04-14T15:11:55.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.388+0000 7f1b33fff700 15 client.24715.objecter _session_command_op_remove 6 13 2024-04-14T15:11:55.399 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:11:55.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.404+0000 7f1b513d7700 10 client.24715.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:11:55.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.404+0000 7f1b513d7700 15 client.24715.objecter _session_command_op_assign -1 14 2024-04-14T15:11:55.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.404+0000 7f1b513d7700 20 client.24715.objecter _get_session s=0x7f1b3801efa0 osd=6 3 2024-04-14T15:11:55.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.404+0000 7f1b513d7700 20 client.24715.objecter put_session s=0x7f1b3801efa0 osd=6 3 2024-04-14T15:11:55.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.404+0000 7f1b513d7700 20 client.24715.objecter _get_session s=0x7f1b3801efa0 osd=6 3 2024-04-14T15:11:55.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.404+0000 7f1b513d7700 15 client.24715.objecter _session_command_op_remove -1 14 2024-04-14T15:11:55.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.404+0000 7f1b513d7700 20 client.24715.objecter get_session s=0x7f1b3801efa0 osd=6 3 2024-04-14T15:11:55.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.404+0000 7f1b513d7700 15 client.24715.objecter _session_command_op_assign 6 14 2024-04-14T15:11:55.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.404+0000 7f1b513d7700 20 client.24715.objecter put_session s=0x7f1b3801efa0 osd=6 4 2024-04-14T15:11:55.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.404+0000 7f1b513d7700 10 client.24715.objecter _send_command 14 2024-04-14T15:11:55.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.404+0000 7f1b513d7700 1 -- 172.21.15.90:0/419239410 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f1b38025610 con 0x7f1b38021b00 2024-04-14T15:11:55.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.406+0000 7f1b33fff700 1 -- 172.21.15.90:0/419239410 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1b38025610 con 0x7f1b38021b00 2024-04-14T15:11:55.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.406+0000 7f1b33fff700 10 client.24715.objecter ms_dispatch 0x7f1b4c001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:11:55.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.406+0000 7f1b33fff700 10 client.24715.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:11:55.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.406+0000 7f1b33fff700 20 client.24715.objecter put_session s=0x7f1b3801efa0 osd=6 4 2024-04-14T15:11:55.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.406+0000 7f1b33fff700 15 client.24715.objecter _session_command_op_remove 6 14 2024-04-14T15:11:55.406 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:11:55.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.406+0000 7f1b513d7700 10 client.24715.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:11:55.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.406+0000 7f1b513d7700 15 client.24715.objecter _session_command_op_assign -1 15 2024-04-14T15:11:55.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.406+0000 7f1b513d7700 20 client.24715.objecter _get_session s=0x7f1b38025610 osd=7 3 2024-04-14T15:11:55.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.406+0000 7f1b513d7700 20 client.24715.objecter put_session s=0x7f1b38025610 osd=7 3 2024-04-14T15:11:55.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.406+0000 7f1b513d7700 20 client.24715.objecter _get_session s=0x7f1b38025610 osd=7 3 2024-04-14T15:11:55.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.406+0000 7f1b513d7700 15 client.24715.objecter _session_command_op_remove -1 15 2024-04-14T15:11:55.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.406+0000 7f1b513d7700 20 client.24715.objecter get_session s=0x7f1b38025610 osd=7 3 2024-04-14T15:11:55.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.406+0000 7f1b513d7700 15 client.24715.objecter _session_command_op_assign 7 15 2024-04-14T15:11:55.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.406+0000 7f1b513d7700 20 client.24715.objecter put_session s=0x7f1b38025610 osd=7 4 2024-04-14T15:11:55.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.406+0000 7f1b513d7700 10 client.24715.objecter _send_command 15 2024-04-14T15:11:55.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.406+0000 7f1b513d7700 1 -- 172.21.15.90:0/419239410 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f1b38029990 con 0x7f1b38025c90 2024-04-14T15:11:55.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.406+0000 7f1b33fff700 10 client.24715.objecter ms_handle_connect 0x7f1b38025c90 2024-04-14T15:11:55.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.407+0000 7f1b33fff700 1 -- 172.21.15.90:0/419239410 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1b38029990 con 0x7f1b38025c90 2024-04-14T15:11:55.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.407+0000 7f1b33fff700 10 client.24715.objecter ms_dispatch 0x7f1b4c001550 command_reply(tid 15: 0 ) 2024-04-14T15:11:55.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.407+0000 7f1b33fff700 10 client.24715.objecter _finish_command 15 = system:0 2024-04-14T15:11:55.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.407+0000 7f1b33fff700 20 client.24715.objecter put_session s=0x7f1b38025610 osd=7 4 2024-04-14T15:11:55.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.407+0000 7f1b33fff700 15 client.24715.objecter _session_command_op_remove 7 15 2024-04-14T15:11:55.422 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:55.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:55.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:11:55.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.423+0000 7f1b513d7700 10 client.24715.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:11:55.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.423+0000 7f1b513d7700 15 client.24715.objecter _session_command_op_assign -1 16 2024-04-14T15:11:55.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.423+0000 7f1b513d7700 20 client.24715.objecter _get_session s=0x7f1b38025610 osd=7 3 2024-04-14T15:11:55.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.423+0000 7f1b513d7700 20 client.24715.objecter put_session s=0x7f1b38025610 osd=7 3 2024-04-14T15:11:55.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.423+0000 7f1b513d7700 20 client.24715.objecter _get_session s=0x7f1b38025610 osd=7 3 2024-04-14T15:11:55.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.423+0000 7f1b513d7700 15 client.24715.objecter _session_command_op_remove -1 16 2024-04-14T15:11:55.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.423+0000 7f1b513d7700 20 client.24715.objecter get_session s=0x7f1b38025610 osd=7 3 2024-04-14T15:11:55.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.423+0000 7f1b513d7700 15 client.24715.objecter _session_command_op_assign 7 16 2024-04-14T15:11:55.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.423+0000 7f1b513d7700 20 client.24715.objecter put_session s=0x7f1b38025610 osd=7 4 2024-04-14T15:11:55.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.423+0000 7f1b513d7700 10 client.24715.objecter _send_command 16 2024-04-14T15:11:55.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.423+0000 7f1b513d7700 1 -- 172.21.15.90:0/419239410 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f1b38029b70 con 0x7f1b38025c90 2024-04-14T15:11:55.425 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:11:55.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.425+0000 7f1b33fff700 1 -- 172.21.15.90:0/419239410 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1b38029b70 con 0x7f1b38025c90 2024-04-14T15:11:55.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.425+0000 7f1b33fff700 10 client.24715.objecter ms_dispatch 0x7f1b4c001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:11:55.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.425+0000 7f1b33fff700 10 client.24715.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:11:55.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.425+0000 7f1b33fff700 20 client.24715.objecter put_session s=0x7f1b38025610 osd=7 4 2024-04-14T15:11:55.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.425+0000 7f1b33fff700 15 client.24715.objecter _session_command_op_remove 7 16 2024-04-14T15:11:55.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.425+0000 7f1b513d7700 10 librados: watch_flush enter 2024-04-14T15:11:55.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.425+0000 7f1b513d7700 10 librados: watch_flush exit 2024-04-14T15:11:55.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.425+0000 7f1b513d7700 10 client.24715.objecter close_session for osd.0 2024-04-14T15:11:55.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.425+0000 7f1b513d7700 1 -- 172.21.15.90:0/419239410 >> v1:172.21.15.90:6801/3761806885 conn(0x7f1b38004730 legacy=0x7f1b38002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:55.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.425+0000 7f1b513d7700 20 client.24715.objecter put_session s=0x7f1b380053e0 osd=0 1 2024-04-14T15:11:55.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.425+0000 7f1b513d7700 10 client.24715.objecter close_session for osd.1 2024-04-14T15:11:55.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.425+0000 7f1b513d7700 1 -- 172.21.15.90:0/419239410 >> v1:172.21.15.90:6805/4170361699 conn(0x7f1b38004310 legacy=0x7f1b3800ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:55.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.425+0000 7f1b513d7700 20 client.24715.objecter put_session s=0x7f1b38004130 osd=1 1 2024-04-14T15:11:55.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.425+0000 7f1b513d7700 10 client.24715.objecter close_session for osd.2 2024-04-14T15:11:55.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.425+0000 7f1b513d7700 1 -- 172.21.15.90:0/419239410 >> v1:172.21.15.90:6809/1686184251 conn(0x7f1b3800d5d0 legacy=0x7f1b3800f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:55.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.425+0000 7f1b513d7700 20 client.24715.objecter put_session s=0x7f1b38001980 osd=2 1 2024-04-14T15:11:55.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.425+0000 7f1b513d7700 10 client.24715.objecter close_session for osd.3 2024-04-14T15:11:55.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.425+0000 7f1b513d7700 1 -- 172.21.15.90:0/419239410 >> v1:172.21.15.90:6813/2270166774 conn(0x7f1b380146b0 legacy=0x7f1b38016aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:55.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.425+0000 7f1b513d7700 20 client.24715.objecter put_session s=0x7f1b38008390 osd=3 1 2024-04-14T15:11:55.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.425+0000 7f1b513d7700 10 client.24715.objecter close_session for osd.4 2024-04-14T15:11:55.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.425+0000 7f1b513d7700 1 -- 172.21.15.90:0/419239410 >> v1:172.21.15.92:6800/3887549733 conn(0x7f1b38014210 legacy=0x7f1b3801b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:55.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.425+0000 7f1b513d7700 20 client.24715.objecter put_session s=0x7f1b3800c9b0 osd=4 1 2024-04-14T15:11:55.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.425+0000 7f1b513d7700 10 client.24715.objecter close_session for osd.5 2024-04-14T15:11:55.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.425+0000 7f1b513d7700 1 -- 172.21.15.90:0/419239410 >> v1:172.21.15.92:6804/3244561440 conn(0x7f1b3801eb10 legacy=0x7f1b3801cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:55.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.425+0000 7f1b513d7700 20 client.24715.objecter put_session s=0x7f1b3801e3f0 osd=5 1 2024-04-14T15:11:55.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.425+0000 7f1b513d7700 10 client.24715.objecter close_session for osd.6 2024-04-14T15:11:55.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.425+0000 7f1b513d7700 1 -- 172.21.15.90:0/419239410 >> v1:172.21.15.92:6808/636125963 conn(0x7f1b38021b00 legacy=0x7f1b38023ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:55.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.425+0000 7f1b4b7fe700 1 -- 172.21.15.90:0/419239410 reap_dead start 2024-04-14T15:11:55.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.425+0000 7f1b4b7fe700 1 -- 172.21.15.90:0/419239410 reap_dead start 2024-04-14T15:11:55.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.425+0000 7f1b4b7fe700 1 -- 172.21.15.90:0/419239410 reap_dead start 2024-04-14T15:11:55.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.426+0000 7f1b513d7700 20 client.24715.objecter put_session s=0x7f1b3801efa0 osd=6 1 2024-04-14T15:11:55.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.426+0000 7f1b513d7700 10 client.24715.objecter close_session for osd.7 2024-04-14T15:11:55.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.426+0000 7f1b513d7700 1 -- 172.21.15.90:0/419239410 >> v1:172.21.15.92:6812/1773706716 conn(0x7f1b38025c90 legacy=0x7f1b380280a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:55.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.426+0000 7f1b513d7700 20 client.24715.objecter put_session s=0x7f1b38025610 osd=7 1 2024-04-14T15:11:55.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.426+0000 7f1b513d7700 20 client.24715.objecter shutdown clearing up homeless session... 2024-04-14T15:11:55.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.426+0000 7f1b513d7700 10 client.24715.objecter successfully canceled tick 2024-04-14T15:11:55.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.426+0000 7f1b513d7700 1 -- 172.21.15.90:0/419239410 >> v1:172.21.15.90:6800/420935427 conn(0x7f1b340713b0 legacy=0x7f1b34073870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:55.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.426+0000 7f1b513d7700 1 -- 172.21.15.90:0/419239410 >> v1:172.21.15.92:6789/0 conn(0x7f1b4c10a740 legacy=0x7f1b4c1a17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:55.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.426+0000 7f1b513d7700 1 -- 172.21.15.90:0/419239410 shutdown_connections 2024-04-14T15:11:55.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.426+0000 7f1b513d7700 1 -- 172.21.15.90:0/419239410 >> 172.21.15.90:0/419239410 conn(0x7f1b4c0fd290 msgr2=0x7f1b4c101780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:11:55.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.426+0000 7f1b513d7700 1 -- 172.21.15.90:0/419239410 shutdown_connections 2024-04-14T15:11:55.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.426+0000 7f1b513d7700 1 -- 172.21.15.90:0/419239410 wait complete. 2024-04-14T15:11:55.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.426+0000 7f1b513d7700 1 librados: shutdown 2024-04-14T15:11:55.430 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:11:55.430 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:11:55.430 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:11:55.430 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:11:55.430 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:11:55.431 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:11:55.431 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:11:55.431 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:11:55.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.176+0000 7ff5ad65a700 1 -- 172.21.15.90:0/4179088662 >> v1:172.21.15.90:6789/0 conn(0x7ff5a810e1e0 legacy=0x7ff5a81106a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:55.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.176+0000 7ff5ad65a700 1 -- 172.21.15.90:0/4179088662 shutdown_connections 2024-04-14T15:11:55.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.176+0000 7ff5ad65a700 1 -- 172.21.15.90:0/4179088662 >> 172.21.15.90:0/4179088662 conn(0x7ff5a80fd290 msgr2=0x7ff5a80ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:11:55.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.176+0000 7ff5ad65a700 1 -- 172.21.15.90:0/4179088662 shutdown_connections 2024-04-14T15:11:55.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.176+0000 7ff5ad65a700 1 -- 172.21.15.90:0/4179088662 wait complete. 2024-04-14T15:11:55.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.176+0000 7ff5ad65a700 1 librados: starting msgr at 2024-04-14T15:11:55.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.176+0000 7ff5ad65a700 1 librados: starting objecter 2024-04-14T15:11:55.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.176+0000 7ff5ad65a700 1 Processor -- start 2024-04-14T15:11:55.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.177+0000 7ff5ad65a700 1 -- start start 2024-04-14T15:11:55.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.177+0000 7ff5ad65a700 1 librados: setting wanted keys 2024-04-14T15:11:55.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.177+0000 7ff5ad65a700 1 librados: calling monclient init 2024-04-14T15:11:55.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.177+0000 7ff5ad65a700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff5a810dd40 con 0x7ff5a8107760 2024-04-14T15:11:55.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.177+0000 7ff5ad65a700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff5a810df50 con 0x7ff5a810e1e0 2024-04-14T15:11:55.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.177+0000 7ff5ad65a700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff5a81ac200 con 0x7ff5a810a610 2024-04-14T15:11:55.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.177+0000 7ff5a6ffd700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7ff5a8107760 0x7ff5a806a4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:56930/0 (socket says 172.21.15.90:56930) 2024-04-14T15:11:55.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.177+0000 7ff5a6ffd700 1 -- 172.21.15.90:0/3479915152 learned_addr learned my addr 172.21.15.90:0/3479915152 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:11:55.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.177+0000 7ff58ffff700 10 client.?.objecter ms_handle_connect 0x7ff5a810a610 2024-04-14T15:11:55.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.177+0000 7ff58ffff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:11:55.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.177+0000 7ff58ffff700 10 client.?.objecter ms_handle_connect 0x7ff5a8107760 2024-04-14T15:11:55.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.177+0000 7ff58ffff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:11:55.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.177+0000 7ff58ffff700 10 client.?.objecter ms_handle_connect 0x7ff5a810e1e0 2024-04-14T15:11:55.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.177+0000 7ff58ffff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:11:55.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.177+0000 7ff58ffff700 1 -- 172.21.15.90:0/3479915152 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3472151369 0 0) 0x7ff5a81ac200 con 0x7ff5a810a610 2024-04-14T15:11:55.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.177+0000 7ff58ffff700 1 -- 172.21.15.90:0/3479915152 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff5900035b0 con 0x7ff5a810a610 2024-04-14T15:11:55.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.177+0000 7ff58ffff700 1 -- 172.21.15.90:0/3479915152 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2679345093 0 0) 0x7ff5a810dd40 con 0x7ff5a8107760 2024-04-14T15:11:55.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.177+0000 7ff58ffff700 1 -- 172.21.15.90:0/3479915152 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff5a81ac200 con 0x7ff5a8107760 2024-04-14T15:11:55.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.177+0000 7ff58ffff700 1 -- 172.21.15.90:0/3479915152 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3131165361 0 0) 0x7ff5a810df50 con 0x7ff5a810e1e0 2024-04-14T15:11:55.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.177+0000 7ff58ffff700 1 -- 172.21.15.90:0/3479915152 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff5a810dd40 con 0x7ff5a810e1e0 2024-04-14T15:11:55.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.177+0000 7ff58ffff700 1 -- 172.21.15.90:0/3479915152 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3709375779 0 0) 0x7ff5a81ac200 con 0x7ff5a8107760 2024-04-14T15:11:55.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.177+0000 7ff58ffff700 1 -- 172.21.15.90:0/3479915152 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff5a810df50 con 0x7ff5a8107760 2024-04-14T15:11:55.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.177+0000 7ff58ffff700 1 -- 172.21.15.90:0/3479915152 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7ff598001870 con 0x7ff5a8107760 2024-04-14T15:11:55.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.178+0000 7ff58ffff700 1 -- 172.21.15.90:0/3479915152 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1428547640 0 0) 0x7ff5900035b0 con 0x7ff5a810a610 2024-04-14T15:11:55.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.178+0000 7ff58ffff700 1 -- 172.21.15.90:0/3479915152 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff5a81ac200 con 0x7ff5a810a610 2024-04-14T15:11:55.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.178+0000 7ff58ffff700 1 -- 172.21.15.90:0/3479915152 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7ff59c0018b0 con 0x7ff5a810a610 2024-04-14T15:11:55.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.178+0000 7ff58ffff700 1 -- 172.21.15.90:0/3479915152 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1610962217 0 0) 0x7ff5a810df50 con 0x7ff5a8107760 2024-04-14T15:11:55.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.178+0000 7ff58ffff700 1 -- 172.21.15.90:0/3479915152 >> v1:172.21.15.90:6790/0 conn(0x7ff5a810a610 legacy=0x7ff5a806abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:55.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.178+0000 7ff58ffff700 1 -- 172.21.15.90:0/3479915152 >> v1:172.21.15.92:6789/0 conn(0x7ff5a810e1e0 legacy=0x7ff5a81a8ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:55.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.178+0000 7ff58ffff700 1 -- 172.21.15.90:0/3479915152 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5a81ad420 con 0x7ff5a8107760 2024-04-14T15:11:55.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.178+0000 7ff5ad65a700 1 -- 172.21.15.90:0/3479915152 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff5a81ac430 con 0x7ff5a8107760 2024-04-14T15:11:55.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.178+0000 7ff5ad65a700 10 client.14820.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:11:55.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.178+0000 7ff5ad65a700 1 -- 172.21.15.90:0/3479915152 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff5a81ac8f0 con 0x7ff5a8107760 2024-04-14T15:11:55.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.178+0000 7ff5ad65a700 1 librados: init done 2024-04-14T15:11:55.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.178+0000 7ff58ffff700 1 -- 172.21.15.90:0/3479915152 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7ff598003b40 con 0x7ff5a8107760 2024-04-14T15:11:55.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.178+0000 7ff58ffff700 1 -- 172.21.15.90:0/3479915152 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7ff598005cc0 con 0x7ff5a8107760 2024-04-14T15:11:55.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.179+0000 7ff5ad65a700 1 -- 172.21.15.90:0/3479915152 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff594005120 con 0x7ff5a8107760 2024-04-14T15:11:55.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.179+0000 7ff58ffff700 1 -- 172.21.15.90:0/3479915152 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7ff59801c890 con 0x7ff5a8107760 2024-04-14T15:11:55.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.180+0000 7ff58ffff700 1 -- 172.21.15.90:0/3479915152 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(96..96 src has 41..96) ==== 6288+0+0 (unknown 4040766927 0 0) 0x7ff59808a3c0 con 0x7ff5a8107760 2024-04-14T15:11:55.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.180+0000 7ff58ffff700 10 client.14820.objecter ms_dispatch 0x7ff5a8001550 osd_map(96..96 src has 41..96) 2024-04-14T15:11:55.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.180+0000 7ff58ffff700 3 client.14820.objecter handle_osd_map got epochs [96,96] > 0 2024-04-14T15:11:55.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.180+0000 7ff58ffff700 3 client.14820.objecter handle_osd_map decoding full epoch 96 2024-04-14T15:11:55.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.180+0000 7ff58ffff700 20 client.14820.objecter dump_active .. 0 homeless 2024-04-14T15:11:55.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.180+0000 7ff58ffff700 10 client.14820.objecter ms_handle_connect 0x7ff5900713d0 2024-04-14T15:11:55.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.184+0000 7ff58ffff700 1 -- 172.21.15.90:0/3479915152 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7ff598057590 con 0x7ff5a8107760 2024-04-14T15:11:55.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:55.409+0000 7ff5ad65a700 1 -- 172.21.15.90:0/3479915152 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd primary-affinity", "id": 3, "weight": 0.21785660735935053} v 0) -- 0x7ff594005410 con 0x7ff5a8107760 2024-04-14T15:11:55.524 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:11:55.535 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:11:55 smithi092 podman[70074]: 2024-04-14 15:11:55.259316059 +0000 UTC m=+0.483444090 container cleanup 9fcf2f777e893dca73a9aa39d89766781c5169ef7a3ecc734c55e7b883824974 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T15:11:55.535 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:11:55 smithi092 bash[70074]: ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a 2024-04-14T15:11:55.547 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:55.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:55.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:11:55.649 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:11:55.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:55.674 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:55.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:11:55.775 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:11:55.798 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:55.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:55.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:11:55.872 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:11:55 smithi092 podman[70109]: 2024-04-14 15:11:55.535615991 +0000 UTC m=+0.657445851 container remove 9fcf2f777e893dca73a9aa39d89766781c5169ef7a3ecc734c55e7b883824974 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T15:11:55.872 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:11:55 smithi092 systemd[1]: ceph-0e288098-fa70-11ee-bc8e-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-04-14T15:11:55.872 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:11:55 smithi092 systemd[1]: Stopped Ceph prometheus.a for 0e288098-fa70-11ee-bc8e-c7b262605968. 2024-04-14T15:11:55.872 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:11:55 smithi092 systemd[1]: ceph-0e288098-fa70-11ee-bc8e-c7b262605968@prometheus.a.service: Consumed 1.186s CPU time. 2024-04-14T15:11:55.872 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:11:55 smithi092 systemd[1]: Starting Ceph prometheus.a for 0e288098-fa70-11ee-bc8e-c7b262605968... 2024-04-14T15:11:55.900 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:11:55.924 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:55.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:55.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:11:56.026 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:11:56.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:56.050 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:56.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:11:56.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:56.104+0000 7ff58ffff700 1 -- 172.21.15.90:0/3479915152 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 3, "weight": 0.21785660735935053}]=0 set osd.3 primary-affinity to 0.217857 (8142772) v97) ==== 156+0+0 (unknown 3489263605 0 0) 0x7ff59805b240 con 0x7ff5a8107760 2024-04-14T15:11:56.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:56.108+0000 7ff5ad65a700 10 librados: watch_flush enter 2024-04-14T15:11:56.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:56.108+0000 7ff5ad65a700 10 librados: watch_flush exit 2024-04-14T15:11:56.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:56.108+0000 7ff5ad65a700 20 client.14820.objecter shutdown clearing up homeless session... 2024-04-14T15:11:56.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:56.108+0000 7ff5ad65a700 10 client.14820.objecter successfully canceled tick 2024-04-14T15:11:56.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:56.108+0000 7ff5ad65a700 1 -- 172.21.15.90:0/3479915152 >> v1:172.21.15.90:6800/420935427 conn(0x7ff5900713d0 legacy=0x7ff590073890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:56.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:56.108+0000 7ff5ad65a700 1 -- 172.21.15.90:0/3479915152 >> v1:172.21.15.90:6789/0 conn(0x7ff5a8107760 legacy=0x7ff5a806a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:56.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:56.112+0000 7ff5ad65a700 1 -- 172.21.15.90:0/3479915152 shutdown_connections 2024-04-14T15:11:56.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:56.112+0000 7ff5ad65a700 1 -- 172.21.15.90:0/3479915152 >> 172.21.15.90:0/3479915152 conn(0x7ff5a80fd290 msgr2=0x7ff5a8111700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:11:56.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:56.114+0000 7ff5ad65a700 1 -- 172.21.15.90:0/3479915152 shutdown_connections 2024-04-14T15:11:56.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:56.114+0000 7ff5ad65a700 1 -- 172.21.15.90:0/3479915152 wait complete. 2024-04-14T15:11:56.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:56.115+0000 7ff5ad65a700 1 librados: shutdown 2024-04-14T15:11:56.117 INFO:teuthology.orchestra.run.smithi090.stderr:set osd.3 primary-affinity to 0.217857 (8142772) 2024-04-14T15:11:56.147 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:56 smithi092 ceph-mon[33966]: pgmap v66: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:11:56.147 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:56 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/419239410' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:11:56.147 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:56 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3479915152' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.21785660735935053}]: dispatch 2024-04-14T15:11:56.147 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:11:55 smithi092 podman[70263]: 2024-04-14 15:11:55.783429371 +0000 UTC m=+0.021692920 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-04-14T15:11:56.147 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:11:55 smithi092 podman[70263]: 2024-04-14 15:11:55.911830813 +0000 UTC m=+0.150094349 container create 380bcfffec5aad23cb566afcb38e91626b634d53028d6979cda36041fdc7210a (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T15:11:56.151 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:11:56.175 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:56.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:56.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:11:56.277 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:11:56.302 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:56.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:56.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:11:56.306 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:56 smithi090 ceph-mon[41670]: pgmap v66: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:11:56.306 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:56 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/419239410' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:11:56.306 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:56 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3479915152' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.21785660735935053}]: dispatch 2024-04-14T15:11:56.306 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:56 smithi090 ceph-mon[30399]: pgmap v66: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:11:56.306 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:56 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/419239410' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:11:56.307 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:56 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3479915152' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.21785660735935053}]: dispatch 2024-04-14T15:11:56.405 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:11:56.428 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:56.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:56.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:11:56.529 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:11:56.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:56.555 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:56.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:11:56.601 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:11:56 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: [14/Apr/2024:15:11:56] ENGINE Bus STOPPING 2024-04-14T15:11:56.601 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:11:56 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: [14/Apr/2024:15:11:56] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-14T15:11:56.601 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:11:56 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: [14/Apr/2024:15:11:56] ENGINE Bus STOPPED 2024-04-14T15:11:56.601 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:11:56 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: [14/Apr/2024:15:11:56] ENGINE Bus STARTING 2024-04-14T15:11:56.601 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:11:56 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: [14/Apr/2024:15:11:56] ENGINE Serving on http://:::9283 2024-04-14T15:11:56.601 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:11:56 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: [14/Apr/2024:15:11:56] ENGINE Bus STARTED 2024-04-14T15:11:56.601 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:11:56 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: [14/Apr/2024:15:11:56] ENGINE Bus STOPPING 2024-04-14T15:11:56.601 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:11:56 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: [14/Apr/2024:15:11:56] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-14T15:11:56.601 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:11:56 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: [14/Apr/2024:15:11:56] ENGINE Bus STOPPED 2024-04-14T15:11:56.601 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:11:56 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: [14/Apr/2024:15:11:56] ENGINE Bus STARTING 2024-04-14T15:11:56.601 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:11:56 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: [14/Apr/2024:15:11:56] ENGINE Serving on http://:::9283 2024-04-14T15:11:56.601 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:11:56 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: [14/Apr/2024:15:11:56] ENGINE Bus STARTED 2024-04-14T15:11:56.602 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:11:56 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: [14/Apr/2024:15:11:56] ENGINE Bus STOPPING 2024-04-14T15:11:56.602 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:11:56 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: [14/Apr/2024:15:11:56] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-14T15:11:56.602 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:11:56 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: [14/Apr/2024:15:11:56] ENGINE Bus STOPPED 2024-04-14T15:11:56.602 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:11:56 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: [14/Apr/2024:15:11:56] ENGINE Bus STARTING 2024-04-14T15:11:56.622 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:11:56 smithi092 podman[70263]: 2024-04-14 15:11:56.14764609 +0000 UTC m=+0.385909627 container init 380bcfffec5aad23cb566afcb38e91626b634d53028d6979cda36041fdc7210a (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T15:11:56.622 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:11:56 smithi092 podman[70263]: 2024-04-14 15:11:56.153199384 +0000 UTC m=+0.391462931 container start 380bcfffec5aad23cb566afcb38e91626b634d53028d6979cda36041fdc7210a (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T15:11:56.623 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:11:56 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[70312]: ts=2024-04-14T15:11:56.178Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-04-14T15:11:56.623 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:11:56 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[70312]: ts=2024-04-14T15:11:56.178Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-04-14T15:11:56.623 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:11:56 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[70312]: ts=2024-04-14T15:11:56.178Z caller=main.go:570 level=info host_details="(Linux 5.14.0-435.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024 x86_64 smithi092 (none))" 2024-04-14T15:11:56.623 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:11:56 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[70312]: ts=2024-04-14T15:11:56.178Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-04-14T15:11:56.623 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:11:56 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[70312]: ts=2024-04-14T15:11:56.178Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-04-14T15:11:56.623 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:11:56 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[70312]: ts=2024-04-14T15:11:56.180Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-04-14T15:11:56.623 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:11:56 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[70312]: ts=2024-04-14T15:11:56.180Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-04-14T15:11:56.623 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:11:56 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[70312]: ts=2024-04-14T15:11:56.181Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-04-14T15:11:56.623 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:11:56 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[70312]: ts=2024-04-14T15:11:56.181Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-04-14T15:11:56.623 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:11:56 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[70312]: ts=2024-04-14T15:11:56.183Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-04-14T15:11:56.623 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:11:56 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[70312]: ts=2024-04-14T15:11:56.183Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.398µs 2024-04-14T15:11:56.623 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:11:56 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[70312]: ts=2024-04-14T15:11:56.183Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-04-14T15:11:56.624 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:11:56 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[70312]: ts=2024-04-14T15:11:56.184Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-04-14T15:11:56.624 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:11:56 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[70312]: ts=2024-04-14T15:11:56.184Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-04-14T15:11:56.624 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:11:56 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[70312]: ts=2024-04-14T15:11:56.184Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=22.34µs wal_replay_duration=566.075µs wbl_replay_duration=110ns total_replay_duration=608.542µs 2024-04-14T15:11:56.624 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:11:56 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[70312]: ts=2024-04-14T15:11:56.185Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-04-14T15:11:56.624 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:11:56 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[70312]: ts=2024-04-14T15:11:56.185Z caller=main.go:1029 level=info msg="TSDB started" 2024-04-14T15:11:56.624 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:11:56 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[70312]: ts=2024-04-14T15:11:56.185Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-04-14T15:11:56.624 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:11:56 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[70312]: ts=2024-04-14T15:11:56.205Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=20.63538ms db_storage=1.025µs remote_storage=1.04µs web_handler=409ns query_engine=489ns scrape=380.109µs scrape_sd=86.587µs notify=14.534µs notify_sd=27.339µs rules=19.769905ms tracing=5.788µs 2024-04-14T15:11:56.624 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:11:56 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[70312]: ts=2024-04-14T15:11:56.205Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-04-14T15:11:56.624 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:11:56 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[70312]: ts=2024-04-14T15:11:56.205Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-04-14T15:11:56.624 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:11:56 smithi092 bash[70263]: 380bcfffec5aad23cb566afcb38e91626b634d53028d6979cda36041fdc7210a 2024-04-14T15:11:56.624 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:11:56 smithi092 systemd[1]: Started Ceph prometheus.a for 0e288098-fa70-11ee-bc8e-c7b262605968. 2024-04-14T15:11:56.657 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:11:56.681 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:56.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:56.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:11:56.782 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:11:56.809 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:56.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:56.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:11:56.911 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:11:56.936 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:56.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:56.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:11:56.941 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:11:56 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: [14/Apr/2024:15:11:56] ENGINE Serving on http://:::9283 2024-04-14T15:11:56.941 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:11:56 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: [14/Apr/2024:15:11:56] ENGINE Bus STARTED 2024-04-14T15:11:57.038 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:11:57.061 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:57.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:57.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:11:57.138 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:11:57.138 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:11:57.163 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:11:57.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:57.190 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:57.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:11:57.291 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:11:57.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:57.316 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:57.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:11:57.370 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:57 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3479915152' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.21785660735935053}]': finished 2024-04-14T15:11:57.370 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:57 smithi090 ceph-mon[30399]: osdmap e97: 8 total, 8 up, 8 in 2024-04-14T15:11:57.370 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:57 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:57.370 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:57 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:57.370 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:57 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-14T15:11:57.370 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:57 smithi090 ceph-mon[30399]: from='mon.0 v1:172.21.15.90:0/650083486' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-14T15:11:57.370 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:57 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi090.front.sepia.ceph.com:9093"}]: dispatch 2024-04-14T15:11:57.370 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:57 smithi090 ceph-mon[30399]: from='mon.0 v1:172.21.15.90:0/650083486' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi090.front.sepia.ceph.com:9093"}]: dispatch 2024-04-14T15:11:57.370 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:57 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:57.371 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:57 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T15:11:57.371 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:57 smithi090 ceph-mon[30399]: from='mon.0 v1:172.21.15.90:0/650083486' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T15:11:57.371 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:57 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi092.front.sepia.ceph.com:9095"}]: dispatch 2024-04-14T15:11:57.371 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:57 smithi090 ceph-mon[30399]: from='mon.0 v1:172.21.15.90:0/650083486' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi092.front.sepia.ceph.com:9095"}]: dispatch 2024-04-14T15:11:57.371 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:57 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:57.371 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:57 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-14T15:11:57.371 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:57 smithi090 ceph-mon[30399]: from='mon.0 v1:172.21.15.90:0/650083486' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-14T15:11:57.371 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:57 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi092.front.sepia.ceph.com:3000"}]: dispatch 2024-04-14T15:11:57.371 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:57 smithi090 ceph-mon[30399]: from='mon.0 v1:172.21.15.90:0/650083486' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi092.front.sepia.ceph.com:3000"}]: dispatch 2024-04-14T15:11:57.371 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:57 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:57.371 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:57 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:11:57.371 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:57 smithi090 ceph-mon[30399]: osdmap e98: 8 total, 8 up, 8 in 2024-04-14T15:11:57.372 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:57 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3479915152' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.21785660735935053}]': finished 2024-04-14T15:11:57.372 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:57 smithi090 ceph-mon[41670]: osdmap e97: 8 total, 8 up, 8 in 2024-04-14T15:11:57.372 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:57 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:57.372 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:57 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:57.372 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:57 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-14T15:11:57.372 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:57 smithi090 ceph-mon[41670]: from='mon.0 v1:172.21.15.90:0/650083486' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-14T15:11:57.372 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:57 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi090.front.sepia.ceph.com:9093"}]: dispatch 2024-04-14T15:11:57.372 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:57 smithi090 ceph-mon[41670]: from='mon.0 v1:172.21.15.90:0/650083486' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi090.front.sepia.ceph.com:9093"}]: dispatch 2024-04-14T15:11:57.372 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:57 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:57.372 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:57 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T15:11:57.372 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:57 smithi090 ceph-mon[41670]: from='mon.0 v1:172.21.15.90:0/650083486' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T15:11:57.372 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:57 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi092.front.sepia.ceph.com:9095"}]: dispatch 2024-04-14T15:11:57.373 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:57 smithi090 ceph-mon[41670]: from='mon.0 v1:172.21.15.90:0/650083486' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi092.front.sepia.ceph.com:9095"}]: dispatch 2024-04-14T15:11:57.373 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:57 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:57.373 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:57 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-14T15:11:57.373 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:57 smithi090 ceph-mon[41670]: from='mon.0 v1:172.21.15.90:0/650083486' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-14T15:11:57.373 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:57 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi092.front.sepia.ceph.com:3000"}]: dispatch 2024-04-14T15:11:57.373 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:57 smithi090 ceph-mon[41670]: from='mon.0 v1:172.21.15.90:0/650083486' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi092.front.sepia.ceph.com:3000"}]: dispatch 2024-04-14T15:11:57.373 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:57 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:57.373 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:57 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:11:57.373 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:57 smithi090 ceph-mon[41670]: osdmap e98: 8 total, 8 up, 8 in 2024-04-14T15:11:57.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:57 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3479915152' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.21785660735935053}]': finished 2024-04-14T15:11:57.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:57 smithi092 ceph-mon[33966]: osdmap e97: 8 total, 8 up, 8 in 2024-04-14T15:11:57.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:57 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:57.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:57 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:57.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:57 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-14T15:11:57.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:57 smithi092 ceph-mon[33966]: from='mon.0 v1:172.21.15.90:0/650083486' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-14T15:11:57.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:57 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi090.front.sepia.ceph.com:9093"}]: dispatch 2024-04-14T15:11:57.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:57 smithi092 ceph-mon[33966]: from='mon.0 v1:172.21.15.90:0/650083486' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi090.front.sepia.ceph.com:9093"}]: dispatch 2024-04-14T15:11:57.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:57 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:57.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:57 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T15:11:57.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:57 smithi092 ceph-mon[33966]: from='mon.0 v1:172.21.15.90:0/650083486' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T15:11:57.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:57 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi092.front.sepia.ceph.com:9095"}]: dispatch 2024-04-14T15:11:57.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:57 smithi092 ceph-mon[33966]: from='mon.0 v1:172.21.15.90:0/650083486' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi092.front.sepia.ceph.com:9095"}]: dispatch 2024-04-14T15:11:57.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:57 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:57.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:57 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-14T15:11:57.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:57 smithi092 ceph-mon[33966]: from='mon.0 v1:172.21.15.90:0/650083486' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-14T15:11:57.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:57 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi092.front.sepia.ceph.com:3000"}]: dispatch 2024-04-14T15:11:57.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:57 smithi092 ceph-mon[33966]: from='mon.0 v1:172.21.15.90:0/650083486' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi092.front.sepia.ceph.com:3000"}]: dispatch 2024-04-14T15:11:57.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:57 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:11:57.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:57 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:11:57.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:57 smithi092 ceph-mon[33966]: osdmap e98: 8 total, 8 up, 8 in 2024-04-14T15:11:57.417 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:11:57.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:57.442 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:57.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:11:57.543 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:11:57.568 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:57.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:57.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:11:57.669 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:11:57.693 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:57.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:57.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:11:57.795 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:11:57.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:57.821 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:57.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:11:57.922 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:11:57.946 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:57.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:57.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:11:58.047 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:11:58.071 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:58.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:58.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:11:58.172 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:11:58.196 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:58.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:58.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:11:58.298 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:11:58.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:58.321 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:58.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:11:58.423 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:11:58.442 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-14T15:11:58.446 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:58.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:58.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:11:58.548 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:11:58.572 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:58.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:58.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:11:58.588 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:11:58 smithi090 ceph-mon[41670]: pgmap v68: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 658 B/s rd, 0 op/s 2024-04-14T15:11:58.588 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:11:58 smithi090 ceph-mon[30399]: pgmap v68: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 658 B/s rd, 0 op/s 2024-04-14T15:11:58.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:11:58 smithi092 ceph-mon[33966]: pgmap v68: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 658 B/s rd, 0 op/s 2024-04-14T15:11:58.674 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:11:58.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:58.700 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:58.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:11:58.801 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:11:58.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:58.826 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:58.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:11:58.927 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:11:58.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:58.952 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:58.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:11:59.053 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:11:59.077 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:59.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:59.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:11:59.179 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:11:59.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:59.204 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:59.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:11:59.305 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:11:59.330 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:59.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:59.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:11:59.432 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:11:59.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:59.456 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:59.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:11:59.557 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:11:59.580 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:59.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:59.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:11:59.682 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:11:59.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:59.710 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:59.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:11:59.811 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:11:59.836 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:11:59.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:59.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:11:59.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:59.911+0000 7f9bcadbe700 1 -- 172.21.15.90:0/1945977998 >> v1:172.21.15.90:6789/0 conn(0x7f9bc4107760 legacy=0x7f9bc4107b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:59.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:59.911+0000 7f9bcadbe700 1 -- 172.21.15.90:0/1945977998 shutdown_connections 2024-04-14T15:11:59.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:59.911+0000 7f9bcadbe700 1 -- 172.21.15.90:0/1945977998 >> 172.21.15.90:0/1945977998 conn(0x7f9bc40fd290 msgr2=0x7f9bc40ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:11:59.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:59.912+0000 7f9bcadbe700 1 -- 172.21.15.90:0/1945977998 shutdown_connections 2024-04-14T15:11:59.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:59.912+0000 7f9bcadbe700 1 -- 172.21.15.90:0/1945977998 wait complete. 2024-04-14T15:11:59.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:59.912+0000 7f9bcadbe700 1 librados: starting msgr at 2024-04-14T15:11:59.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:59.912+0000 7f9bcadbe700 1 librados: starting objecter 2024-04-14T15:11:59.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:59.912+0000 7f9bcadbe700 1 Processor -- start 2024-04-14T15:11:59.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:59.912+0000 7f9bcadbe700 1 -- start start 2024-04-14T15:11:59.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:59.912+0000 7f9bcadbe700 1 librados: setting wanted keys 2024-04-14T15:11:59.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:59.912+0000 7f9bcadbe700 1 librados: calling monclient init 2024-04-14T15:11:59.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:59.912+0000 7f9bcadbe700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f9bc410e000 con 0x7f9bc4107760 2024-04-14T15:11:59.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:59.912+0000 7f9bcadbe700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f9bc41a7e30 con 0x7f9bc410a610 2024-04-14T15:11:59.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:59.912+0000 7f9bcadbe700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f9bc41a9070 con 0x7f9bc410e1e0 2024-04-14T15:11:59.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:59.912+0000 7f9bc8b5a700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f9bc4107760 0x7f9bc410c860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:56958/0 (socket says 172.21.15.90:56958) 2024-04-14T15:11:59.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:59.912+0000 7f9bc8b5a700 1 -- 172.21.15.90:0/3547696861 learned_addr learned my addr 172.21.15.90:0/3547696861 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:11:59.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:59.913+0000 7f9bc1ffb700 10 client.?.objecter ms_handle_connect 0x7f9bc4107760 2024-04-14T15:11:59.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:59.913+0000 7f9bc1ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:11:59.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:59.913+0000 7f9bc1ffb700 10 client.?.objecter ms_handle_connect 0x7f9bc410e1e0 2024-04-14T15:11:59.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:59.913+0000 7f9bc1ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:11:59.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:59.913+0000 7f9bc1ffb700 10 client.?.objecter ms_handle_connect 0x7f9bc410a610 2024-04-14T15:11:59.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:59.913+0000 7f9bc1ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:11:59.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:59.913+0000 7f9bc1ffb700 1 -- 172.21.15.90:0/3547696861 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2155461971 0 0) 0x7f9bc41156a0 con 0x7f9bc4107760 2024-04-14T15:11:59.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:59.913+0000 7f9bc1ffb700 1 -- 172.21.15.90:0/3547696861 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9bb40035b0 con 0x7f9bc4107760 2024-04-14T15:11:59.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:59.913+0000 7f9bc1ffb700 1 -- 172.21.15.90:0/3547696861 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4062313425 0 0) 0x7f9bc41a9070 con 0x7f9bc410e1e0 2024-04-14T15:11:59.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:59.913+0000 7f9bc1ffb700 1 -- 172.21.15.90:0/3547696861 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9bc41156a0 con 0x7f9bc410e1e0 2024-04-14T15:11:59.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:59.913+0000 7f9bc1ffb700 1 -- 172.21.15.90:0/3547696861 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2769230155 0 0) 0x7f9bc41a7e30 con 0x7f9bc410a610 2024-04-14T15:11:59.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:59.913+0000 7f9bc1ffb700 1 -- 172.21.15.90:0/3547696861 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9bc41a9070 con 0x7f9bc410a610 2024-04-14T15:11:59.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:59.913+0000 7f9bc1ffb700 1 -- 172.21.15.90:0/3547696861 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4187054008 0 0) 0x7f9bb40035b0 con 0x7f9bc4107760 2024-04-14T15:11:59.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:59.913+0000 7f9bc1ffb700 1 -- 172.21.15.90:0/3547696861 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9bc41a7e30 con 0x7f9bc4107760 2024-04-14T15:11:59.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:59.913+0000 7f9bc1ffb700 1 -- 172.21.15.90:0/3547696861 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f9bb80015d0 con 0x7f9bc4107760 2024-04-14T15:11:59.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:59.913+0000 7f9bc1ffb700 1 -- 172.21.15.90:0/3547696861 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 387603376 0 0) 0x7f9bc41156a0 con 0x7f9bc410e1e0 2024-04-14T15:11:59.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:59.913+0000 7f9bc1ffb700 1 -- 172.21.15.90:0/3547696861 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9bb40035b0 con 0x7f9bc410e1e0 2024-04-14T15:11:59.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:59.913+0000 7f9bc1ffb700 1 -- 172.21.15.90:0/3547696861 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4006661434 0 0) 0x7f9bc41a9070 con 0x7f9bc410a610 2024-04-14T15:11:59.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:59.913+0000 7f9bc1ffb700 1 -- 172.21.15.90:0/3547696861 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9bc41156a0 con 0x7f9bc410a610 2024-04-14T15:11:59.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:59.913+0000 7f9bc1ffb700 1 -- 172.21.15.90:0/3547696861 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f9bb00017a0 con 0x7f9bc410a610 2024-04-14T15:11:59.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:59.913+0000 7f9bc1ffb700 1 -- 172.21.15.90:0/3547696861 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f9bbc001c30 con 0x7f9bc410e1e0 2024-04-14T15:11:59.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:59.913+0000 7f9bc1ffb700 1 -- 172.21.15.90:0/3547696861 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 63552802 0 0) 0x7f9bc41a7e30 con 0x7f9bc4107760 2024-04-14T15:11:59.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:59.913+0000 7f9bc1ffb700 1 -- 172.21.15.90:0/3547696861 >> v1:172.21.15.90:6790/0 conn(0x7f9bc410e1e0 legacy=0x7f9bc41a56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:59.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:59.913+0000 7f9bc1ffb700 1 -- 172.21.15.90:0/3547696861 >> v1:172.21.15.92:6789/0 conn(0x7f9bc410a610 legacy=0x7f9bc410cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:11:59.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:59.913+0000 7f9bc1ffb700 1 -- 172.21.15.90:0/3547696861 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bc41aa2b0 con 0x7f9bc4107760 2024-04-14T15:11:59.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:59.913+0000 7f9bcadbe700 1 -- 172.21.15.90:0/3547696861 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9bc41a92a0 con 0x7f9bc4107760 2024-04-14T15:11:59.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:59.913+0000 7f9bc1ffb700 1 -- 172.21.15.90:0/3547696861 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f9bb8001b70 con 0x7f9bc4107760 2024-04-14T15:11:59.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:59.913+0000 7f9bc1ffb700 1 -- 172.21.15.90:0/3547696861 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f9bb80050e0 con 0x7f9bc4107760 2024-04-14T15:11:59.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:59.914+0000 7f9bcadbe700 10 client.14832.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:11:59.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:59.914+0000 7f9bcadbe700 1 -- 172.21.15.90:0/3547696861 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f9bc41a9760 con 0x7f9bc4107760 2024-04-14T15:11:59.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:59.914+0000 7f9bcadbe700 1 librados: init done 2024-04-14T15:11:59.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:59.914+0000 7f9bcadbe700 1 -- 172.21.15.90:0/3547696861 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9bc4102ea0 con 0x7f9bc4107760 2024-04-14T15:11:59.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:59.915+0000 7f9bc1ffb700 1 -- 172.21.15.90:0/3547696861 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f9bb80050e0 con 0x7f9bc4107760 2024-04-14T15:11:59.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:59.915+0000 7f9bc1ffb700 1 -- 172.21.15.90:0/3547696861 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(98..98 src has 41..98) ==== 6320+0+0 (unknown 2718985133 0 0) 0x7f9bb808a390 con 0x7f9bc4107760 2024-04-14T15:11:59.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:59.915+0000 7f9bc1ffb700 10 client.14832.objecter ms_dispatch 0x7f9bc4001550 osd_map(98..98 src has 41..98) 2024-04-14T15:11:59.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:59.915+0000 7f9bc1ffb700 3 client.14832.objecter handle_osd_map got epochs [98,98] > 0 2024-04-14T15:11:59.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:59.915+0000 7f9bc1ffb700 3 client.14832.objecter handle_osd_map decoding full epoch 98 2024-04-14T15:11:59.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:59.915+0000 7f9bc1ffb700 20 client.14832.objecter dump_active .. 0 homeless 2024-04-14T15:11:59.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:59.916+0000 7f9bc1ffb700 10 client.14832.objecter ms_handle_connect 0x7f9bb4073680 2024-04-14T15:11:59.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:11:59.920+0000 7f9bc1ffb700 1 -- 172.21.15.90:0/3547696861 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f9bc410e000 con 0x7f9bc4107760 2024-04-14T15:11:59.937 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:11:59.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:11:59.962 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:11:59.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:12:00.063 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:12:00.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:00.088 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:00.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:12:00.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:00.140+0000 7f9bcadbe700 1 -- 172.21.15.90:0/3547696861 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7f9bc410c320 con 0x7f9bc4107760 2024-04-14T15:12:00.189 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:12:00.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:00.214 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:00.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:12:00.315 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:12:00.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:00.340 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:00.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:12:00.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:00.340+0000 7f9bc1ffb700 1 -- 172.21.15.90:0/3547696861 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v99) ==== 97+0+0 (unknown 4046948202 0 0) 0x7f9bb8057540 con 0x7f9bc4107760 2024-04-14T15:12:00.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:00.344+0000 7f9bcadbe700 10 librados: watch_flush enter 2024-04-14T15:12:00.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:00.344+0000 7f9bcadbe700 10 librados: watch_flush exit 2024-04-14T15:12:00.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:00.344+0000 7f9bcadbe700 20 client.14832.objecter shutdown clearing up homeless session... 2024-04-14T15:12:00.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:00.344+0000 7f9bcadbe700 10 client.14832.objecter successfully canceled tick 2024-04-14T15:12:00.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:00.344+0000 7f9bcadbe700 1 -- 172.21.15.90:0/3547696861 >> v1:172.21.15.90:6800/420935427 conn(0x7f9bb4073680 legacy=0x7f9bb4075b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:00.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:00.344+0000 7f9bcadbe700 1 -- 172.21.15.90:0/3547696861 >> v1:172.21.15.90:6789/0 conn(0x7f9bc4107760 legacy=0x7f9bc410c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:00.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:00.345+0000 7f9bcadbe700 1 -- 172.21.15.90:0/3547696861 shutdown_connections 2024-04-14T15:12:00.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:00.345+0000 7f9bcadbe700 1 -- 172.21.15.90:0/3547696861 >> 172.21.15.90:0/3547696861 conn(0x7f9bc40fd290 msgr2=0x7f9bc41117c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:12:00.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:00.345+0000 7f9bcadbe700 1 -- 172.21.15.90:0/3547696861 shutdown_connections 2024-04-14T15:12:00.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:00.345+0000 7f9bcadbe700 1 -- 172.21.15.90:0/3547696861 wait complete. 2024-04-14T15:12:00.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:00.345+0000 7f9bcadbe700 1 librados: shutdown 2024-04-14T15:12:00.347 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is set 2024-04-14T15:12:00.386 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:00 smithi090 ceph-mon[41670]: pgmap v70: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:12:00.387 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:00 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3547696861' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:12:00.387 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:00 smithi090 ceph-mon[30399]: pgmap v70: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:12:00.387 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:00 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3547696861' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:12:00.428 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-14T15:12:00.428 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-14T15:12:00.428 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-14T15:12:00.428 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-14T15:12:00.428 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.428 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.14154.0:137 1.0 e20/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 20'95, mlcod=20'92)", 2024-04-14T15:12:00.428 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:07:26.209007+0000", 2024-04-14T15:12:00.429 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 274.21684616300001, 2024-04-14T15:12:00.429 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0030129110000000001, 2024-04-14T15:12:00.429 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:12:00.429 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:12:00.429 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:12:00.429 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:12:00.429 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.429 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:12:00.429 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:07:26.209007+0000", 2024-04-14T15:12:00.429 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:12:00.429 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.429 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.430 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:12:00.430 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:07:26.209007+0000", 2024-04-14T15:12:00.430 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:12:00.430 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.430 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.430 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:12:00.430 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:07:26.209008+0000", 2024-04-14T15:12:00.431 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.035e-06 2024-04-14T15:12:00.431 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.431 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.431 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:12:00.431 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:07:26.209032+0000", 2024-04-14T15:12:00.431 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3694999999999999e-05 2024-04-14T15:12:00.431 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.431 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.432 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:12:00.432 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:07:26.209043+0000", 2024-04-14T15:12:00.432 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1353e-05 2024-04-14T15:12:00.432 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.432 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.432 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:12:00.432 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:07:26.209050+0000", 2024-04-14T15:12:00.432 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9410000000000001e-06 2024-04-14T15:12:00.433 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.433 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.433 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:12:00.433 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:07:26.209093+0000", 2024-04-14T15:12:00.433 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2759000000000002e-05 2024-04-14T15:12:00.433 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.433 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.433 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:12:00.433 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:07:26.209116+0000", 2024-04-14T15:12:00.433 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.353e-05 2024-04-14T15:12:00.433 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.433 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.433 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:12:00.433 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:07:26.211978+0000", 2024-04-14T15:12:00.434 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028616520000000001 2024-04-14T15:12:00.434 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.434 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.434 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:12:00.434 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:07:26.212020+0000", 2024-04-14T15:12:00.434 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1946000000000001e-05 2024-04-14T15:12:00.434 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:00.434 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:12:00.434 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:00.434 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.434 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.435 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.14154.0:138 1.0 e20/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 20'96, mlcod=20'92)", 2024-04-14T15:12:00.435 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:07:26.209648+0000", 2024-04-14T15:12:00.435 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 274.21620551900003, 2024-04-14T15:12:00.435 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002410564, 2024-04-14T15:12:00.435 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:12:00.435 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:12:00.435 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:12:00.435 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:12:00.435 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.435 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:12:00.435 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:07:26.209648+0000", 2024-04-14T15:12:00.435 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:12:00.436 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.436 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.436 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:12:00.436 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:07:26.209648+0000", 2024-04-14T15:12:00.436 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:12:00.436 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.436 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.436 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:12:00.436 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:07:26.209649+0000", 2024-04-14T15:12:00.436 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.051e-06 2024-04-14T15:12:00.436 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.436 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.437 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:12:00.437 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:07:26.209656+0000", 2024-04-14T15:12:00.437 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3590000000000002e-06 2024-04-14T15:12:00.437 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.437 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.437 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:12:00.437 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:07:26.209668+0000", 2024-04-14T15:12:00.437 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1779e-05 2024-04-14T15:12:00.437 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.437 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.437 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:12:00.437 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:07:26.209674+0000", 2024-04-14T15:12:00.437 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6189999999999996e-06 2024-04-14T15:12:00.438 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.438 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.438 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:12:00.438 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:07:26.211062+0000", 2024-04-14T15:12:00.438 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0013879090000000001 2024-04-14T15:12:00.438 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.438 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.438 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:12:00.438 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:07:26.211079+0000", 2024-04-14T15:12:00.439 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6891000000000001e-05 2024-04-14T15:12:00.439 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.439 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.439 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:12:00.439 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:07:26.212023+0000", 2024-04-14T15:12:00.441 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00094375899999999996 2024-04-14T15:12:00.441 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.441 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.441 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:12:00.441 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:07:26.212058+0000", 2024-04-14T15:12:00.441 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5197000000000002e-05 2024-04-14T15:12:00.442 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:00.442 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:12:00.442 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:00.442 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.442 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.442 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.14154.0:139 1.0 e20/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 20'97, mlcod=20'93)", 2024-04-14T15:12:00.442 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:07:26.210145+0000", 2024-04-14T15:12:00.443 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 274.21570795999997, 2024-04-14T15:12:00.443 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0019439220000000001, 2024-04-14T15:12:00.443 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:12:00.443 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:12:00.443 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:12:00.443 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:12:00.443 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.443 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:12:00.444 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:07:26.210145+0000", 2024-04-14T15:12:00.444 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:12:00.444 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.444 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.444 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:12:00.445 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:07:26.210145+0000", 2024-04-14T15:12:00.445 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:12:00.445 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.445 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.445 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:12:00.445 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:07:26.210146+0000", 2024-04-14T15:12:00.445 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1429999999999999e-06 2024-04-14T15:12:00.446 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.446 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.446 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:12:00.446 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:07:26.210153+0000", 2024-04-14T15:12:00.446 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3890000000000002e-06 2024-04-14T15:12:00.446 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.446 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.447 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:12:00.447 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:07:26.210165+0000", 2024-04-14T15:12:00.447 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1841000000000001e-05 2024-04-14T15:12:00.447 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.447 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.447 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:12:00.447 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:07:26.210172+0000", 2024-04-14T15:12:00.448 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.79e-06 2024-04-14T15:12:00.448 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.448 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.448 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:12:00.448 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:07:26.211269+0000", 2024-04-14T15:12:00.448 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001096671 2024-04-14T15:12:00.448 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.448 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.449 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:12:00.449 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:07:26.211287+0000", 2024-04-14T15:12:00.449 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7731e-05 2024-04-14T15:12:00.449 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.449 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.449 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:12:00.449 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:07:26.212060+0000", 2024-04-14T15:12:00.450 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00077284100000000002 2024-04-14T15:12:00.450 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.450 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.450 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:12:00.450 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:07:26.212089+0000", 2024-04-14T15:12:00.450 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9516e-05 2024-04-14T15:12:00.450 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:00.451 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:12:00.451 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:00.451 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.451 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.451 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.14154.0:140 1.0 e20/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 20'98, mlcod=20'93)", 2024-04-14T15:12:00.451 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:07:26.210540+0000", 2024-04-14T15:12:00.451 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 274.21531342499998, 2024-04-14T15:12:00.452 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001580651, 2024-04-14T15:12:00.452 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:12:00.452 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:12:00.452 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:12:00.452 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:12:00.452 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.452 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:12:00.453 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:07:26.210540+0000", 2024-04-14T15:12:00.453 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:12:00.453 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.453 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.453 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:12:00.453 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:07:26.210540+0000", 2024-04-14T15:12:00.453 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:12:00.453 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.454 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.454 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:12:00.454 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:07:26.210541+0000", 2024-04-14T15:12:00.454 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.06e-06 2024-04-14T15:12:00.454 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.454 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.454 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:12:00.454 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:07:26.210566+0000", 2024-04-14T15:12:00.455 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4864000000000001e-05 2024-04-14T15:12:00.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.455 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.455 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:12:00.455 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:07:26.210576+0000", 2024-04-14T15:12:00.455 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0885e-05 2024-04-14T15:12:00.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.456 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.456 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:12:00.456 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:07:26.210584+0000", 2024-04-14T15:12:00.456 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4309999999999997e-06 2024-04-14T15:12:00.456 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.456 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.456 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:12:00.457 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:07:26.211458+0000", 2024-04-14T15:12:00.457 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00087401999999999998 2024-04-14T15:12:00.457 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.457 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.457 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:12:00.457 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:07:26.211479+0000", 2024-04-14T15:12:00.457 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0965e-05 2024-04-14T15:12:00.457 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.457 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.458 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:12:00.458 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:07:26.212091+0000", 2024-04-14T15:12:00.458 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00061230500000000005 2024-04-14T15:12:00.458 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.458 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.458 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:12:00.458 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:07:26.212120+0000", 2024-04-14T15:12:00.458 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9121000000000001e-05 2024-04-14T15:12:00.458 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:00.458 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:12:00.458 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:00.458 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.458 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.458 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24373.0:44 4.b 4:d467b91b:::bucket.sync-target-hints.:head [call,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e67)", 2024-04-14T15:12:00.459 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:09:45.030729+0000", 2024-04-14T15:12:00.459 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 135.39512369400001, 2024-04-14T15:12:00.459 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0070804240000000001, 2024-04-14T15:12:00.459 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:12:00.459 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:12:00.459 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-04-14T15:12:00.459 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:12:00.459 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24373", 2024-04-14T15:12:00.459 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2924083629", 2024-04-14T15:12:00.459 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 44 2024-04-14T15:12:00.459 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.459 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:12:00.459 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.460 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:12:00.460 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:45.030729+0000", 2024-04-14T15:12:00.460 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:12:00.460 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.460 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.460 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:12:00.460 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:45.030729+0000", 2024-04-14T15:12:00.460 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:12:00.460 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.460 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.460 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:12:00.460 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:45.030732+0000", 2024-04-14T15:12:00.460 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9340000000000002e-06 2024-04-14T15:12:00.460 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.461 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.461 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:12:00.461 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:45.030743+0000", 2024-04-14T15:12:00.461 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0529e-05 2024-04-14T15:12:00.461 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.461 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.461 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:12:00.461 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:45.030757+0000", 2024-04-14T15:12:00.461 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3720000000000001e-05 2024-04-14T15:12:00.461 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.461 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.461 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:12:00.462 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:45.030779+0000", 2024-04-14T15:12:00.462 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2858e-05 2024-04-14T15:12:00.462 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.462 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.462 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:12:00.462 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:45.030867+0000", 2024-04-14T15:12:00.462 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7878000000000001e-05 2024-04-14T15:12:00.462 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.462 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.462 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:12:00.462 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:45.030877+0000", 2024-04-14T15:12:00.462 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6239999999999997e-06 2024-04-14T15:12:00.462 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.463 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.463 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:12:00.463 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:45.037699+0000", 2024-04-14T15:12:00.463 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.006821761 2024-04-14T15:12:00.463 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.463 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.463 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:12:00.463 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:45.037810+0000", 2024-04-14T15:12:00.463 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011112000000000001 2024-04-14T15:12:00.463 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:00.463 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:12:00.463 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:00.464 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.464 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.464 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24383.0:44 4.b 4:d467b91b:::bucket.sync-target-hints.:head [call,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e67)", 2024-04-14T15:12:00.464 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:09:45.035765+0000", 2024-04-14T15:12:00.464 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 135.390088574, 2024-04-14T15:12:00.464 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022164810000000002, 2024-04-14T15:12:00.464 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:12:00.464 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:12:00.464 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-04-14T15:12:00.464 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:12:00.464 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24383", 2024-04-14T15:12:00.464 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2632170366", 2024-04-14T15:12:00.464 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 44 2024-04-14T15:12:00.464 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.465 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:12:00.465 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.465 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:12:00.465 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:45.035765+0000", 2024-04-14T15:12:00.465 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:12:00.465 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.465 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.465 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:12:00.465 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:45.035765+0000", 2024-04-14T15:12:00.465 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:12:00.465 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.465 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.465 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:12:00.466 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:45.035767+0000", 2024-04-14T15:12:00.466 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.407e-06 2024-04-14T15:12:00.466 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.466 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.466 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:12:00.466 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:45.035777+0000", 2024-04-14T15:12:00.466 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6409999999999997e-06 2024-04-14T15:12:00.466 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.466 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.466 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:12:00.466 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:45.035790+0000", 2024-04-14T15:12:00.466 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.344e-05 2024-04-14T15:12:00.466 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.466 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.467 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:12:00.467 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:45.035802+0000", 2024-04-14T15:12:00.467 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1909000000000001e-05 2024-04-14T15:12:00.467 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.467 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.467 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:12:00.467 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:45.035879+0000", 2024-04-14T15:12:00.467 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6755999999999995e-05 2024-04-14T15:12:00.467 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.467 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.467 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:12:00.467 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:45.035888+0000", 2024-04-14T15:12:00.467 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6809999999999993e-06 2024-04-14T15:12:00.467 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.467 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.468 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:12:00.468 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:45.037893+0000", 2024-04-14T15:12:00.468 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020043420000000001 2024-04-14T15:12:00.468 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.468 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.468 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:12:00.468 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:45.037981+0000", 2024-04-14T15:12:00.468 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8305000000000002e-05 2024-04-14T15:12:00.468 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:00.468 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:12:00.468 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:00.468 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.468 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.468 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24383.0:56 5.3 5:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-04-14T15:12:00.469 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:09:47.058804+0000", 2024-04-14T15:12:00.469 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 133.36704869799999, 2024-04-14T15:12:00.469 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001453305, 2024-04-14T15:12:00.469 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:12:00.471 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:12:00.471 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:12:00.471 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:12:00.471 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:12:00.471 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24383", 2024-04-14T15:12:00.472 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2632170366", 2024-04-14T15:12:00.472 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 56 2024-04-14T15:12:00.472 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.472 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:12:00.472 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.472 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:12:00.472 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:47.058804+0000", 2024-04-14T15:12:00.472 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:12:00.472 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.472 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.472 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:12:00.472 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:47.058804+0000", 2024-04-14T15:12:00.472 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:12:00.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.473 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.473 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:12:00.473 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:47.058806+0000", 2024-04-14T15:12:00.473 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.968e-06 2024-04-14T15:12:00.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.473 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.473 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:12:00.473 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:47.058812+0000", 2024-04-14T15:12:00.473 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.857e-06 2024-04-14T15:12:00.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.473 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.474 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:12:00.474 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:47.058824+0000", 2024-04-14T15:12:00.474 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1605e-05 2024-04-14T15:12:00.474 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.474 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.474 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:12:00.474 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:47.058831+0000", 2024-04-14T15:12:00.474 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1470000000000004e-06 2024-04-14T15:12:00.474 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.474 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.475 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:12:00.475 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:47.058893+0000", 2024-04-14T15:12:00.475 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2175999999999996e-05 2024-04-14T15:12:00.475 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.475 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.475 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:12:00.475 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:47.059022+0000", 2024-04-14T15:12:00.475 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012836299999999999 2024-04-14T15:12:00.475 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.475 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.475 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 5,6", 2024-04-14T15:12:00.475 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:47.059149+0000", 2024-04-14T15:12:00.475 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012752400000000001 2024-04-14T15:12:00.476 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.476 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.476 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:12:00.476 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:47.059767+0000", 2024-04-14T15:12:00.476 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00061752200000000001 2024-04-14T15:12:00.476 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.476 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.476 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:12:00.476 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:47.060128+0000", 2024-04-14T15:12:00.476 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00036191400000000002 2024-04-14T15:12:00.476 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.476 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.476 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:12:00.476 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:47.060165+0000", 2024-04-14T15:12:00.476 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6628000000000002e-05 2024-04-14T15:12:00.477 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.477 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.477 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:12:00.477 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:47.060200+0000", 2024-04-14T15:12:00.477 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4983e-05 2024-04-14T15:12:00.477 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.477 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.477 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:12:00.477 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:47.060258+0000", 2024-04-14T15:12:00.477 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7618000000000003e-05 2024-04-14T15:12:00.477 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:00.477 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:12:00.477 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:00.477 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.478 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24383.0:57 5.3 5:c609908c:::notify.5:head [watch watch cookie 94329014964480] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-04-14T15:12:00.478 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:09:47.060406+0000", 2024-04-14T15:12:00.478 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 133.36544745200001, 2024-04-14T15:12:00.478 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001391558, 2024-04-14T15:12:00.478 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:12:00.478 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:12:00.478 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:12:00.478 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:12:00.478 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24383", 2024-04-14T15:12:00.478 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2632170366", 2024-04-14T15:12:00.478 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 57 2024-04-14T15:12:00.479 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.479 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:12:00.479 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.479 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:12:00.479 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:47.060406+0000", 2024-04-14T15:12:00.479 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:12:00.479 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.479 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.479 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:12:00.479 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:47.060406+0000", 2024-04-14T15:12:00.479 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:12:00.479 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.479 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.479 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:12:00.479 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:47.060407+0000", 2024-04-14T15:12:00.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.344e-06 2024-04-14T15:12:00.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.480 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:12:00.480 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:47.060412+0000", 2024-04-14T15:12:00.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8019999999999999e-06 2024-04-14T15:12:00.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.480 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:12:00.480 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:47.060420+0000", 2024-04-14T15:12:00.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4719999999999995e-06 2024-04-14T15:12:00.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.481 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:12:00.481 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:47.060428+0000", 2024-04-14T15:12:00.481 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3629999999999996e-06 2024-04-14T15:12:00.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.481 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:12:00.481 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:47.060527+0000", 2024-04-14T15:12:00.481 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9493000000000004e-05 2024-04-14T15:12:00.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.481 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:12:00.481 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:47.060616+0000", 2024-04-14T15:12:00.481 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9308999999999995e-05 2024-04-14T15:12:00.482 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.482 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.482 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 5,6", 2024-04-14T15:12:00.482 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:47.060767+0000", 2024-04-14T15:12:00.482 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015017900000000001 2024-04-14T15:12:00.482 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.482 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.482 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:12:00.482 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:47.061149+0000", 2024-04-14T15:12:00.482 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038233499999999998 2024-04-14T15:12:00.482 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.483 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.483 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:12:00.483 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:47.061469+0000", 2024-04-14T15:12:00.483 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031994199999999998 2024-04-14T15:12:00.483 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.483 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.483 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:12:00.483 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:47.061653+0000", 2024-04-14T15:12:00.483 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018366899999999999 2024-04-14T15:12:00.483 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.483 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.483 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:12:00.483 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:47.061711+0000", 2024-04-14T15:12:00.484 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8894000000000002e-05 2024-04-14T15:12:00.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.484 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.484 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:12:00.484 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:47.061797+0000", 2024-04-14T15:12:00.484 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5755999999999997e-05 2024-04-14T15:12:00.484 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:00.484 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:12:00.484 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:00.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.484 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.484 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24373.0:57 5.3 5:c609908c:::notify.5:head [watch watch cookie 94738481570944] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-04-14T15:12:00.484 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:09:47.060695+0000", 2024-04-14T15:12:00.485 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 133.36515841900001, 2024-04-14T15:12:00.485 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022602299999999998, 2024-04-14T15:12:00.485 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:12:00.485 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:12:00.485 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:12:00.485 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:12:00.485 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24373", 2024-04-14T15:12:00.485 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2924083629", 2024-04-14T15:12:00.485 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 57 2024-04-14T15:12:00.485 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.485 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:12:00.485 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.485 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:12:00.485 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:47.060695+0000", 2024-04-14T15:12:00.486 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:12:00.486 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.486 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.486 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:12:00.486 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:47.060695+0000", 2024-04-14T15:12:00.486 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:12:00.486 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.486 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.486 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:12:00.486 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:47.060696+0000", 2024-04-14T15:12:00.486 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5740000000000001e-06 2024-04-14T15:12:00.486 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.486 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.487 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:12:00.487 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:47.060703+0000", 2024-04-14T15:12:00.487 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7070000000000003e-06 2024-04-14T15:12:00.487 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.487 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.487 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:12:00.487 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:47.060712+0000", 2024-04-14T15:12:00.487 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3090000000000001e-06 2024-04-14T15:12:00.487 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.487 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.487 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:12:00.487 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:47.060720+0000", 2024-04-14T15:12:00.487 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1999999999999994e-06 2024-04-14T15:12:00.487 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.488 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:12:00.488 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:47.061015+0000", 2024-04-14T15:12:00.488 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029501700000000001 2024-04-14T15:12:00.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.488 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-14T15:12:00.488 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:47.061108+0000", 2024-04-14T15:12:00.488 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2114999999999995e-05 2024-04-14T15:12:00.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.488 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:12:00.488 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:47.061844+0000", 2024-04-14T15:12:00.489 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00073676999999999998 2024-04-14T15:12:00.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.489 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:12:00.489 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:47.061915+0000", 2024-04-14T15:12:00.489 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0871999999999995e-05 2024-04-14T15:12:00.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.489 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 5,6", 2024-04-14T15:12:00.489 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:47.062085+0000", 2024-04-14T15:12:00.489 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000169467 2024-04-14T15:12:00.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.489 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:12:00.490 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:47.062412+0000", 2024-04-14T15:12:00.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032695599999999999 2024-04-14T15:12:00.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.490 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.490 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:12:00.490 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:47.062801+0000", 2024-04-14T15:12:00.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000389763 2024-04-14T15:12:00.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.490 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.490 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:12:00.490 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:47.062846+0000", 2024-04-14T15:12:00.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4400999999999997e-05 2024-04-14T15:12:00.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.490 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.491 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:12:00.491 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:47.062898+0000", 2024-04-14T15:12:00.491 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1737999999999998e-05 2024-04-14T15:12:00.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.491 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.491 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:12:00.491 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:47.062955+0000", 2024-04-14T15:12:00.491 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7340999999999999e-05 2024-04-14T15:12:00.491 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:00.491 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:12:00.491 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:00.492 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.492 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.492 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24373.0:208 4.b 4:d242335b:gc::gc.2:head [create,call version.check_conds in=18b,call rgw_gc.rgw_gc_queue_init in=23b,call version.set in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-04-14T15:12:00.492 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:09:50.060298+0000", 2024-04-14T15:12:00.492 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 130.365555425, 2024-04-14T15:12:00.492 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0069434570000000001, 2024-04-14T15:12:00.492 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:12:00.492 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:12:00.492 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:12:00.492 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:12:00.492 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24373", 2024-04-14T15:12:00.492 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2924083629", 2024-04-14T15:12:00.492 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 208 2024-04-14T15:12:00.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.493 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:12:00.493 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.493 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:12:00.493 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.060298+0000", 2024-04-14T15:12:00.493 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:12:00.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.493 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.493 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:12:00.493 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.060298+0000", 2024-04-14T15:12:00.493 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:12:00.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.493 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.493 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:12:00.493 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.060301+0000", 2024-04-14T15:12:00.494 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.405e-06 2024-04-14T15:12:00.494 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.494 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.494 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:12:00.494 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.060313+0000", 2024-04-14T15:12:00.494 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1761e-05 2024-04-14T15:12:00.494 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.494 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.494 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:12:00.494 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.060334+0000", 2024-04-14T15:12:00.497 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1174000000000001e-05 2024-04-14T15:12:00.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.497 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.497 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:12:00.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.060346+0000", 2024-04-14T15:12:00.497 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2293000000000001e-05 2024-04-14T15:12:00.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.498 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.498 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:12:00.498 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.060441+0000", 2024-04-14T15:12:00.498 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4264000000000004e-05 2024-04-14T15:12:00.498 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.498 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.498 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2024-04-14T15:12:00.498 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.060448+0000", 2024-04-14T15:12:00.498 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1539999999999996e-06 2024-04-14T15:12:00.498 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.498 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.498 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:12:00.498 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.066055+0000", 2024-04-14T15:12:00.499 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0056070089999999996 2024-04-14T15:12:00.499 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.499 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.499 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:12:00.499 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.066149+0000", 2024-04-14T15:12:00.499 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4263000000000002e-05 2024-04-14T15:12:00.499 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.499 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.499 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,4", 2024-04-14T15:12:00.499 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.066338+0000", 2024-04-14T15:12:00.499 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018898299999999999 2024-04-14T15:12:00.499 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.499 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.499 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:12:00.500 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.066914+0000", 2024-04-14T15:12:00.500 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00057636600000000003 2024-04-14T15:12:00.500 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.500 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.500 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:12:00.500 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.067107+0000", 2024-04-14T15:12:00.500 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019238099999999999 2024-04-14T15:12:00.500 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.500 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.500 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:12:00.500 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.067187+0000", 2024-04-14T15:12:00.500 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9970999999999997e-05 2024-04-14T15:12:00.500 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.500 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.501 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:12:00.501 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.067219+0000", 2024-04-14T15:12:00.501 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2144000000000003e-05 2024-04-14T15:12:00.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.501 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.501 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:12:00.501 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.067241+0000", 2024-04-14T15:12:00.501 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2289e-05 2024-04-14T15:12:00.501 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:00.501 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:12:00.501 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:00.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.501 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.501 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24373.0:210 4.3 4:c60da066:gc::gc.4:head [create,call version.check_conds in=18b,call rgw_gc.rgw_gc_queue_init in=23b,call version.set in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-04-14T15:12:00.502 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:09:50.068920+0000", 2024-04-14T15:12:00.502 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 130.35693297700001, 2024-04-14T15:12:00.502 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00137711, 2024-04-14T15:12:00.502 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:12:00.502 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:12:00.502 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:12:00.502 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:12:00.502 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24373", 2024-04-14T15:12:00.502 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2924083629", 2024-04-14T15:12:00.502 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 210 2024-04-14T15:12:00.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.502 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:12:00.502 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.502 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:12:00.503 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.068920+0000", 2024-04-14T15:12:00.503 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:12:00.503 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.503 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.503 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:12:00.503 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.068920+0000", 2024-04-14T15:12:00.503 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:12:00.503 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.503 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.503 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:12:00.503 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.068922+0000", 2024-04-14T15:12:00.503 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6899999999999999e-06 2024-04-14T15:12:00.503 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.504 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.504 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:12:00.504 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.068929+0000", 2024-04-14T15:12:00.504 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7190000000000004e-06 2024-04-14T15:12:00.504 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.504 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.504 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:12:00.504 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.068940+0000", 2024-04-14T15:12:00.504 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1344e-05 2024-04-14T15:12:00.504 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.504 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.504 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:12:00.504 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.068949+0000", 2024-04-14T15:12:00.504 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6340000000000007e-06 2024-04-14T15:12:00.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.505 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:12:00.505 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.069017+0000", 2024-04-14T15:12:00.505 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8717e-05 2024-04-14T15:12:00.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.505 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:12:00.505 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.069092+0000", 2024-04-14T15:12:00.505 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5202000000000003e-05 2024-04-14T15:12:00.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.505 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 5,7", 2024-04-14T15:12:00.506 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.069254+0000", 2024-04-14T15:12:00.506 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000161452 2024-04-14T15:12:00.506 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.506 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.506 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:12:00.506 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.069651+0000", 2024-04-14T15:12:00.506 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039669999999999999 2024-04-14T15:12:00.506 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.506 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.506 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:12:00.506 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.070083+0000", 2024-04-14T15:12:00.506 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00043231200000000002 2024-04-14T15:12:00.506 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.506 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.507 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:12:00.507 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.070235+0000", 2024-04-14T15:12:00.507 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015176499999999999 2024-04-14T15:12:00.507 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.507 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.507 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:12:00.507 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.070270+0000", 2024-04-14T15:12:00.507 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.578e-05 2024-04-14T15:12:00.507 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.507 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.507 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:12:00.507 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.070297+0000", 2024-04-14T15:12:00.507 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6795e-05 2024-04-14T15:12:00.508 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:00.508 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:12:00.508 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:00.508 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.508 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.508 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24373.0:347 4.3 4:c3a682d4:reshard::reshard.0000000004:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-04-14T15:12:00.508 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:09:50.122271+0000", 2024-04-14T15:12:00.508 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 130.30358186699999, 2024-04-14T15:12:00.508 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016298020000000001, 2024-04-14T15:12:00.508 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:12:00.508 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:12:00.508 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:12:00.508 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:12:00.508 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24373", 2024-04-14T15:12:00.509 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2924083629", 2024-04-14T15:12:00.509 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 347 2024-04-14T15:12:00.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.509 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:12:00.509 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.509 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:12:00.509 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.122271+0000", 2024-04-14T15:12:00.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:12:00.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.509 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.509 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:12:00.509 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.122271+0000", 2024-04-14T15:12:00.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:12:00.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.510 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:12:00.510 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.122274+0000", 2024-04-14T15:12:00.510 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.593e-06 2024-04-14T15:12:00.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.510 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:12:00.510 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.122285+0000", 2024-04-14T15:12:00.510 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.111e-05 2024-04-14T15:12:00.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.510 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:12:00.510 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.122301+0000", 2024-04-14T15:12:00.510 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6291e-05 2024-04-14T15:12:00.511 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.511 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.511 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:12:00.511 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.122311+0000", 2024-04-14T15:12:00.511 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2979999999999997e-06 2024-04-14T15:12:00.511 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.511 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.511 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:12:00.511 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.122387+0000", 2024-04-14T15:12:00.511 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6217000000000006e-05 2024-04-14T15:12:00.511 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.511 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.511 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:12:00.511 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.122539+0000", 2024-04-14T15:12:00.512 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000151791 2024-04-14T15:12:00.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.512 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.512 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 5,7", 2024-04-14T15:12:00.512 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.122757+0000", 2024-04-14T15:12:00.512 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021822700000000001 2024-04-14T15:12:00.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.512 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.512 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:12:00.512 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.123229+0000", 2024-04-14T15:12:00.512 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00047240999999999998 2024-04-14T15:12:00.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.512 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.512 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:12:00.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.123781+0000", 2024-04-14T15:12:00.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00055220300000000003 2024-04-14T15:12:00.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.513 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.513 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:12:00.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.123832+0000", 2024-04-14T15:12:00.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0868999999999997e-05 2024-04-14T15:12:00.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.513 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.513 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:12:00.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.123868+0000", 2024-04-14T15:12:00.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5460000000000003e-05 2024-04-14T15:12:00.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.513 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.514 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:12:00.514 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.123901+0000", 2024-04-14T15:12:00.514 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3333000000000001e-05 2024-04-14T15:12:00.514 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:00.514 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:12:00.514 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:00.514 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.514 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.514 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24373.0:357 4.6 4:66924383:reshard::reshard.0000000005:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-04-14T15:12:00.514 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:09:50.125923+0000", 2024-04-14T15:12:00.514 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 130.29992997799999, 2024-04-14T15:12:00.514 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0018007920000000001, 2024-04-14T15:12:00.514 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:12:00.515 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:12:00.515 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:12:00.515 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:12:00.515 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24373", 2024-04-14T15:12:00.515 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2924083629", 2024-04-14T15:12:00.515 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 357 2024-04-14T15:12:00.515 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.515 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:12:00.515 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.515 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:12:00.515 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.125923+0000", 2024-04-14T15:12:00.515 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:12:00.515 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.515 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.515 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:12:00.516 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.125923+0000", 2024-04-14T15:12:00.516 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:12:00.516 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.516 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.516 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:12:00.516 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.125924+0000", 2024-04-14T15:12:00.516 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3599999999999999e-06 2024-04-14T15:12:00.516 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.516 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.516 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:12:00.516 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.125929+0000", 2024-04-14T15:12:00.516 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9960000000000001e-06 2024-04-14T15:12:00.516 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.516 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.517 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:12:00.517 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.125939+0000", 2024-04-14T15:12:00.517 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2720000000000003e-06 2024-04-14T15:12:00.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.517 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.517 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:12:00.517 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.125945+0000", 2024-04-14T15:12:00.517 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9209999999999997e-06 2024-04-14T15:12:00.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.517 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.517 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:12:00.517 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.126006+0000", 2024-04-14T15:12:00.517 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1080999999999993e-05 2024-04-14T15:12:00.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.518 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:12:00.518 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.126130+0000", 2024-04-14T15:12:00.518 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000124599 2024-04-14T15:12:00.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.518 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2024-04-14T15:12:00.518 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.126310+0000", 2024-04-14T15:12:00.518 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017950299999999999 2024-04-14T15:12:00.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.518 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:12:00.518 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.126975+0000", 2024-04-14T15:12:00.518 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00066551999999999996 2024-04-14T15:12:00.519 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.519 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.519 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:12:00.519 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.127354+0000", 2024-04-14T15:12:00.519 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037910599999999999 2024-04-14T15:12:00.519 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.519 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.519 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:12:00.519 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.127638+0000", 2024-04-14T15:12:00.519 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000283844 2024-04-14T15:12:00.519 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.519 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.519 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:12:00.519 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.127684+0000", 2024-04-14T15:12:00.520 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5337999999999999e-05 2024-04-14T15:12:00.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.520 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.520 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:12:00.520 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.127724+0000", 2024-04-14T15:12:00.520 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0252e-05 2024-04-14T15:12:00.520 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:00.520 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:12:00.520 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:00.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.520 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.520 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24373.0:364 4.6 4:66924383:reshard::reshard.0000000005:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-04-14T15:12:00.520 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:09:50.128326+0000", 2024-04-14T15:12:00.521 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 130.297527075, 2024-04-14T15:12:00.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001648505, 2024-04-14T15:12:00.521 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:12:00.521 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:12:00.521 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:12:00.521 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:12:00.521 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24373", 2024-04-14T15:12:00.521 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2924083629", 2024-04-14T15:12:00.521 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 364 2024-04-14T15:12:00.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.521 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:12:00.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.521 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:12:00.521 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.128326+0000", 2024-04-14T15:12:00.522 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:12:00.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.522 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:12:00.522 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.128326+0000", 2024-04-14T15:12:00.522 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:12:00.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.522 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:12:00.522 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.128327+0000", 2024-04-14T15:12:00.522 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.083e-06 2024-04-14T15:12:00.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.523 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:12:00.523 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.128340+0000", 2024-04-14T15:12:00.523 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2549e-05 2024-04-14T15:12:00.523 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.523 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.523 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:12:00.523 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.128350+0000", 2024-04-14T15:12:00.523 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0155e-05 2024-04-14T15:12:00.523 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.523 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.523 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:12:00.523 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.128356+0000", 2024-04-14T15:12:00.523 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7459999999999997e-06 2024-04-14T15:12:00.523 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.524 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:12:00.524 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.128400+0000", 2024-04-14T15:12:00.524 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4048999999999998e-05 2024-04-14T15:12:00.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.524 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:12:00.524 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.128474+0000", 2024-04-14T15:12:00.524 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4482999999999998e-05 2024-04-14T15:12:00.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.524 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2024-04-14T15:12:00.524 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.128644+0000", 2024-04-14T15:12:00.525 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017027000000000001 2024-04-14T15:12:00.525 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.525 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:12:00.525 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.129446+0000", 2024-04-14T15:12:00.525 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00080159100000000004 2024-04-14T15:12:00.525 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.525 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:12:00.525 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.129565+0000", 2024-04-14T15:12:00.525 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000119166 2024-04-14T15:12:00.525 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.525 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:12:00.526 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.129870+0000", 2024-04-14T15:12:00.526 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030449300000000003 2024-04-14T15:12:00.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.526 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.526 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:12:00.526 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.129934+0000", 2024-04-14T15:12:00.526 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4152999999999994e-05 2024-04-14T15:12:00.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.526 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.526 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:12:00.526 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.129975+0000", 2024-04-14T15:12:00.526 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0766999999999998e-05 2024-04-14T15:12:00.526 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:00.526 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:12:00.526 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:00.527 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.527 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.527 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24373.0:366 4.16 4:6dc3b799:reshard::reshard.0000000006:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-04-14T15:12:00.527 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:09:50.130219+0000", 2024-04-14T15:12:00.527 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 130.29563423600001, 2024-04-14T15:12:00.527 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001442835, 2024-04-14T15:12:00.527 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:12:00.527 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:12:00.527 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:12:00.527 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:12:00.527 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24373", 2024-04-14T15:12:00.527 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2924083629", 2024-04-14T15:12:00.527 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 366 2024-04-14T15:12:00.527 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.528 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:12:00.528 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.528 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:12:00.528 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.130219+0000", 2024-04-14T15:12:00.528 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:12:00.528 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.528 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.528 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:12:00.528 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.130219+0000", 2024-04-14T15:12:00.528 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:12:00.528 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.528 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.528 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:12:00.528 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.130220+0000", 2024-04-14T15:12:00.529 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.525e-06 2024-04-14T15:12:00.529 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.529 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:12:00.529 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.130227+0000", 2024-04-14T15:12:00.529 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.81e-06 2024-04-14T15:12:00.529 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.529 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:12:00.529 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.130239+0000", 2024-04-14T15:12:00.529 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1838000000000001e-05 2024-04-14T15:12:00.529 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.529 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:12:00.529 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.130245+0000", 2024-04-14T15:12:00.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3430000000000001e-06 2024-04-14T15:12:00.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.530 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:12:00.530 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.130301+0000", 2024-04-14T15:12:00.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5254999999999999e-05 2024-04-14T15:12:00.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.530 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:12:00.530 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.130435+0000", 2024-04-14T15:12:00.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013412 2024-04-14T15:12:00.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.530 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,7", 2024-04-14T15:12:00.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.130652+0000", 2024-04-14T15:12:00.531 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021719900000000001 2024-04-14T15:12:00.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.531 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:12:00.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.131159+0000", 2024-04-14T15:12:00.531 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00050754800000000003 2024-04-14T15:12:00.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.531 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:12:00.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.131484+0000", 2024-04-14T15:12:00.531 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032483000000000002 2024-04-14T15:12:00.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.532 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:12:00.532 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.131576+0000", 2024-04-14T15:12:00.532 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1452999999999993e-05 2024-04-14T15:12:00.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.532 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:12:00.532 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.131626+0000", 2024-04-14T15:12:00.532 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0705000000000003e-05 2024-04-14T15:12:00.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.532 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:12:00.532 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.131662+0000", 2024-04-14T15:12:00.532 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5209000000000001e-05 2024-04-14T15:12:00.532 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:00.533 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:12:00.533 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:00.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.533 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24373.0:408 4.6 4:64760185:::obj_delete_at_hint.0000000010:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-04-14T15:12:00.533 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:09:50.142979+0000", 2024-04-14T15:12:00.533 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 130.28287417000001, 2024-04-14T15:12:00.533 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001896586, 2024-04-14T15:12:00.533 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:12:00.533 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:12:00.533 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:12:00.533 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:12:00.533 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24373", 2024-04-14T15:12:00.533 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2924083629", 2024-04-14T15:12:00.534 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 408 2024-04-14T15:12:00.534 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.534 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:12:00.534 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.534 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:12:00.534 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.142979+0000", 2024-04-14T15:12:00.534 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:12:00.534 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.534 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.534 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:12:00.534 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.142979+0000", 2024-04-14T15:12:00.534 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:12:00.534 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.534 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.535 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:12:00.535 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.142981+0000", 2024-04-14T15:12:00.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.629e-06 2024-04-14T15:12:00.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.535 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.535 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:12:00.535 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.143001+0000", 2024-04-14T15:12:00.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0882000000000001e-05 2024-04-14T15:12:00.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.535 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.535 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:12:00.535 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.143016+0000", 2024-04-14T15:12:00.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4623e-05 2024-04-14T15:12:00.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.536 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.536 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:12:00.536 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.143024+0000", 2024-04-14T15:12:00.536 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9440000000000005e-06 2024-04-14T15:12:00.536 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.536 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.536 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:12:00.536 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.143088+0000", 2024-04-14T15:12:00.536 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4265999999999998e-05 2024-04-14T15:12:00.536 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.536 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.536 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:12:00.536 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.143216+0000", 2024-04-14T15:12:00.536 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012818399999999999 2024-04-14T15:12:00.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.537 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2024-04-14T15:12:00.537 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.143408+0000", 2024-04-14T15:12:00.537 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019185099999999999 2024-04-14T15:12:00.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.537 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:12:00.537 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.143874+0000", 2024-04-14T15:12:00.537 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00046543399999999998 2024-04-14T15:12:00.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.537 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:12:00.537 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.144658+0000", 2024-04-14T15:12:00.538 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00078440699999999999 2024-04-14T15:12:00.538 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.538 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.538 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:12:00.538 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.144794+0000", 2024-04-14T15:12:00.538 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000136012 2024-04-14T15:12:00.538 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.538 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.538 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:12:00.538 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.144839+0000", 2024-04-14T15:12:00.538 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4886999999999999e-05 2024-04-14T15:12:00.538 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.538 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.539 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:12:00.539 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.144876+0000", 2024-04-14T15:12:00.539 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6467000000000002e-05 2024-04-14T15:12:00.539 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:00.539 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:12:00.539 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:00.539 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.539 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.539 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24373.0:413 4.6 4:64760185:::obj_delete_at_hint.0000000010:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-04-14T15:12:00.539 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:09:50.145528+0000", 2024-04-14T15:12:00.539 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 130.28032501199999, 2024-04-14T15:12:00.539 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001455011, 2024-04-14T15:12:00.540 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:12:00.540 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:12:00.540 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:12:00.540 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:12:00.540 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24373", 2024-04-14T15:12:00.540 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2924083629", 2024-04-14T15:12:00.540 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 413 2024-04-14T15:12:00.540 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.540 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:12:00.540 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.540 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:12:00.540 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.145528+0000", 2024-04-14T15:12:00.540 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:12:00.541 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.541 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.541 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:12:00.541 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.145528+0000", 2024-04-14T15:12:00.541 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:12:00.541 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.541 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.541 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:12:00.541 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.145529+0000", 2024-04-14T15:12:00.541 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2440000000000001e-06 2024-04-14T15:12:00.541 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.541 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.541 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:12:00.541 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.145535+0000", 2024-04-14T15:12:00.541 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8150000000000002e-06 2024-04-14T15:12:00.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.542 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:12:00.542 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.145545+0000", 2024-04-14T15:12:00.542 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6930000000000002e-06 2024-04-14T15:12:00.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.542 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:12:00.542 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.145553+0000", 2024-04-14T15:12:00.542 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6570000000000003e-06 2024-04-14T15:12:00.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.542 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:12:00.542 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.145816+0000", 2024-04-14T15:12:00.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000262119 2024-04-14T15:12:00.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.543 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:12:00.543 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.145893+0000", 2024-04-14T15:12:00.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7331999999999998e-05 2024-04-14T15:12:00.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.543 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2024-04-14T15:12:00.543 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.146067+0000", 2024-04-14T15:12:00.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017389599999999999 2024-04-14T15:12:00.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.543 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:12:00.543 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.146603+0000", 2024-04-14T15:12:00.544 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00053653000000000001 2024-04-14T15:12:00.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.544 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.544 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:12:00.544 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.146863+0000", 2024-04-14T15:12:00.544 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025920499999999998 2024-04-14T15:12:00.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.544 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.544 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:12:00.544 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.146909+0000", 2024-04-14T15:12:00.544 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5942000000000002e-05 2024-04-14T15:12:00.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.544 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.544 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:12:00.544 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.146945+0000", 2024-04-14T15:12:00.545 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6044999999999998e-05 2024-04-14T15:12:00.545 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.545 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.545 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:12:00.545 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.146983+0000", 2024-04-14T15:12:00.545 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8532999999999999e-05 2024-04-14T15:12:00.545 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:00.545 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:12:00.545 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:00.545 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.545 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.545 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24373.0:469 4.3 4:c60da066:gc::gc.4:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-04-14T15:12:00.545 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:09:50.164217+0000", 2024-04-14T15:12:00.545 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 130.261635977, 2024-04-14T15:12:00.546 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0014609239999999999, 2024-04-14T15:12:00.546 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:12:00.546 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:12:00.546 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:12:00.546 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:12:00.546 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24373", 2024-04-14T15:12:00.546 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2924083629", 2024-04-14T15:12:00.546 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 469 2024-04-14T15:12:00.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.546 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:12:00.546 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.546 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:12:00.546 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.164217+0000", 2024-04-14T15:12:00.547 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:12:00.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.547 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:12:00.547 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.164217+0000", 2024-04-14T15:12:00.547 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:12:00.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.547 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:12:00.547 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.164219+0000", 2024-04-14T15:12:00.547 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2280000000000001e-06 2024-04-14T15:12:00.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.547 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:12:00.548 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.164227+0000", 2024-04-14T15:12:00.548 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1110000000000008e-06 2024-04-14T15:12:00.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.548 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:12:00.548 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.164242+0000", 2024-04-14T15:12:00.548 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4533e-05 2024-04-14T15:12:00.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.548 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:12:00.548 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.164249+0000", 2024-04-14T15:12:00.548 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1980000000000004e-06 2024-04-14T15:12:00.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:12:00.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.164314+0000", 2024-04-14T15:12:00.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5180999999999998e-05 2024-04-14T15:12:00.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:12:00.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.164407+0000", 2024-04-14T15:12:00.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2604999999999999e-05 2024-04-14T15:12:00.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 5,7", 2024-04-14T15:12:00.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.164593+0000", 2024-04-14T15:12:00.550 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000185687 2024-04-14T15:12:00.550 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.550 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:12:00.550 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.164883+0000", 2024-04-14T15:12:00.550 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029016799999999999 2024-04-14T15:12:00.550 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.550 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:12:00.550 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.165540+0000", 2024-04-14T15:12:00.550 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00065722000000000003 2024-04-14T15:12:00.550 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.551 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:12:00.551 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.165588+0000", 2024-04-14T15:12:00.551 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7757999999999999e-05 2024-04-14T15:12:00.551 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.551 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.551 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:12:00.551 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.165638+0000", 2024-04-14T15:12:00.551 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9789000000000001e-05 2024-04-14T15:12:00.551 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.551 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.551 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:12:00.551 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.165678+0000", 2024-04-14T15:12:00.551 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0445999999999999e-05 2024-04-14T15:12:00.551 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:00.551 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:12:00.552 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:00.552 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.552 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.552 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24373.0:686 4.b 4:d5ecafc5:::obj_delete_at_hint.0000000046:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-04-14T15:12:00.552 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:09:50.243966+0000", 2024-04-14T15:12:00.552 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 130.18188709699999, 2024-04-14T15:12:00.552 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0014550920000000001, 2024-04-14T15:12:00.552 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:12:00.552 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:12:00.552 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:12:00.552 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:12:00.552 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24373", 2024-04-14T15:12:00.553 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2924083629", 2024-04-14T15:12:00.553 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 686 2024-04-14T15:12:00.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.553 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:12:00.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.553 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:12:00.553 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.243966+0000", 2024-04-14T15:12:00.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:12:00.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.554 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:12:00.554 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.243966+0000", 2024-04-14T15:12:00.554 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:12:00.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.554 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:12:00.554 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.243967+0000", 2024-04-14T15:12:00.554 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2899999999999999e-06 2024-04-14T15:12:00.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.555 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:12:00.555 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.243973+0000", 2024-04-14T15:12:00.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6890000000000001e-06 2024-04-14T15:12:00.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.555 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:12:00.556 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.243984+0000", 2024-04-14T15:12:00.556 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1209000000000001e-05 2024-04-14T15:12:00.556 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.556 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.556 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:12:00.556 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.243999+0000", 2024-04-14T15:12:00.556 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4814000000000001e-05 2024-04-14T15:12:00.556 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.556 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:12:00.557 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.244054+0000", 2024-04-14T15:12:00.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5488999999999997e-05 2024-04-14T15:12:00.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:12:00.557 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.244180+0000", 2024-04-14T15:12:00.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012581800000000001 2024-04-14T15:12:00.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,4", 2024-04-14T15:12:00.558 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.244357+0000", 2024-04-14T15:12:00.558 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017636600000000001 2024-04-14T15:12:00.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:12:00.558 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.244657+0000", 2024-04-14T15:12:00.558 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030061900000000002 2024-04-14T15:12:00.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.559 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:12:00.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.244799+0000", 2024-04-14T15:12:00.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000141717 2024-04-14T15:12:00.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.559 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:12:00.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.245368+0000", 2024-04-14T15:12:00.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00056878099999999997 2024-04-14T15:12:00.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.559 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:12:00.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.245398+0000", 2024-04-14T15:12:00.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9801000000000001e-05 2024-04-14T15:12:00.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.560 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:12:00.560 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:09:50.245421+0000", 2024-04-14T15:12:00.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3499e-05 2024-04-14T15:12:00.560 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:00.560 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:12:00.560 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:00.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.560 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.560 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.14634.0:2 3.1 3:90ab8ff9:::rbd_task:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-04-14T15:12:00.560 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:10:23.037903+0000", 2024-04-14T15:12:00.560 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 97.387950248999999, 2024-04-14T15:12:00.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001702727, 2024-04-14T15:12:00.560 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:12:00.560 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:12:00.561 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-04-14T15:12:00.561 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:12:00.561 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.14634", 2024-04-14T15:12:00.561 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1646083328", 2024-04-14T15:12:00.561 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2 2024-04-14T15:12:00.561 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.561 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:12:00.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.561 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:12:00.561 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:10:23.037903+0000", 2024-04-14T15:12:00.561 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:12:00.561 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.561 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:12:00.561 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:10:23.037903+0000", 2024-04-14T15:12:00.562 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:12:00.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.562 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.562 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:12:00.562 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:10:23.037905+0000", 2024-04-14T15:12:00.562 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.559e-06 2024-04-14T15:12:00.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.562 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.562 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:12:00.562 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:10:23.037914+0000", 2024-04-14T15:12:00.562 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2740000000000007e-06 2024-04-14T15:12:00.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.562 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.563 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:12:00.563 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:10:23.037928+0000", 2024-04-14T15:12:00.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4022999999999999e-05 2024-04-14T15:12:00.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.563 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:12:00.563 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:10:23.037941+0000", 2024-04-14T15:12:00.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2934e-05 2024-04-14T15:12:00.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.563 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:12:00.563 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:10:23.038025+0000", 2024-04-14T15:12:00.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4060000000000005e-05 2024-04-14T15:12:00.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.564 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2024-04-14T15:12:00.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:10:23.038031+0000", 2024-04-14T15:12:00.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4960000000000001e-06 2024-04-14T15:12:00.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.564 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:12:00.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:10:23.039519+0000", 2024-04-14T15:12:00.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00148807 2024-04-14T15:12:00.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:00.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:00.564 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:12:00.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:10:23.039606+0000", 2024-04-14T15:12:00.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6310999999999994e-05 2024-04-14T15:12:00.564 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:00.564 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:12:00.565 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:00.565 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:00.565 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:12:00.565 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-14T15:12:00.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:00.566 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:00.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:12:00.600 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:00 smithi092 ceph-mon[33966]: pgmap v70: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:12:00.600 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:00 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3547696861' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:12:00.667 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:12:00.691 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:00.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:00.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:12:00.793 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:12:00.817 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:00.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:00.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:12:00.919 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:12:00.941 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:12:00 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-alertmanager-a[84115]: ts=2024-04-14T15:12:00.879Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003066064s 2024-04-14T15:12:00.943 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:00.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:00.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:12:01.045 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:12:01.068 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:01.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:01.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:12:01.170 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:12:01.195 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:01.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:01.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:12:01.297 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:12:01.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:01.322 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:01.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:12:01.423 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:12:01.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:01.449 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:01.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:12:01.550 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:12:01.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:01.574 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:01.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:12:01.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:01 smithi092 ceph-mon[33966]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:12:01.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:01 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3547696861' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-14T15:12:01.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:01 smithi092 ceph-mon[33966]: osdmap e99: 8 total, 8 up, 8 in 2024-04-14T15:12:01.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:01 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:12:01.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:01 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:12:01.628 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:01 smithi090 ceph-mon[41670]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:12:01.629 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:01 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3547696861' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-14T15:12:01.629 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:01 smithi090 ceph-mon[41670]: osdmap e99: 8 total, 8 up, 8 in 2024-04-14T15:12:01.629 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:01 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:12:01.629 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:01 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:12:01.630 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:01 smithi090 ceph-mon[30399]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:12:01.630 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:01 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3547696861' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-14T15:12:01.630 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:01 smithi090 ceph-mon[30399]: osdmap e99: 8 total, 8 up, 8 in 2024-04-14T15:12:01.630 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:01 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:12:01.630 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:01 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:12:01.675 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:12:01.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:01.699 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:01.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:12:01.800 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:12:01.823 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:01.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:01.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:12:01.925 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:12:01.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:01.954 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:01.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:12:02.056 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:12:02.083 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:02.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:02.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:12:02.126 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-14T15:12:02.126 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-14T15:12:02.126 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-04-14T15:12:02.126 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-04-14T15:12:02.127 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph pg dump --format=json 2024-04-14T15:12:02.185 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:12:02.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:02.210 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:02.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:12:02.311 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:12:02.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:02.336 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:02.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:12:02.437 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:12:02.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:02.461 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:02.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:12:02.562 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:12:02.590 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:02.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:02.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:12:02.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:02 smithi092 ceph-mon[33966]: pgmap v72: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:12:02.692 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:12:02.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:02 smithi090 ceph-mon[41670]: pgmap v72: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:12:02.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:02 smithi090 ceph-mon[30399]: pgmap v72: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:12:02.720 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:02.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:02.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:12:02.754 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-14T15:12:02.756 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:12:02.777 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:12:02.821 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:12:02.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:02.847 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:02.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:12:02.948 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:12:02.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:02.973 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:02.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:12:03.074 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:12:03.100 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:03.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:03.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:12:03.202 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:12:03.226 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:03.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:03.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:12:03.329 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:12:03.353 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:03.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:03.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:12:03.454 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:12:03.479 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:03.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:03.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:12:03.581 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:12:03.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:03.605 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:03.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:12:03.655 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd unset noscrub 2024-04-14T15:12:03.706 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:12:03.732 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:03.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:03.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:12:03.834 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:12:03.857 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:03.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:03.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:12:03.959 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:12:03.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:03.984 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:03.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:12:04.085 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:12:04.111 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:04.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:04.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:12:04.128 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:12:04.213 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:12:04.225 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:12:03 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:12:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:12:04.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:04.242 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:04.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:12:04.343 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:12:04.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:04.367 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:04.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:12:04.468 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:12:04.495 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:04.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:04.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:12:04.597 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:12:04.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:04 smithi092 ceph-mon[33966]: pgmap v73: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:12:04.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:04.623 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:04.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:12:04.685 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:04 smithi090 ceph-mon[41670]: pgmap v73: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:12:04.685 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:04 smithi090 ceph-mon[30399]: pgmap v73: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:12:04.724 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:12:04.752 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:04.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:04.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:12:04.853 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:12:04.877 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:04.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:04.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:12:04.979 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:12:05.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:05.007 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:05.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:12:05.108 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:12:05.122 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:12:04 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:12:05.133 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:05.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:05.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:12:05.236 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:12:05.259 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:05.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:05.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:12:05.361 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:12:05.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:05.389 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:05.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:12:05.490 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:12:05.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:05.515 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:05.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:12:05.616 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:12:05.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:05 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:12:05.643 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:05.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:05.644 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:12:05.691 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:05 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:12:05.691 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:05 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:12:05.745 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:12:05.769 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:05.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:05.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:12:05.871 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:12:05.898 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:05.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:05.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:12:06.000 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:12:06.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:06.030 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:06.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:12:06.131 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:12:06.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:06.157 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:06.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:12:06.258 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:12:06.282 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:06.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:06.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:12:06.384 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:12:06.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:06.408 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:06.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:12:06.509 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:12:06.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:06.537 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:06.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:12:06.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:06 smithi092 ceph-mon[33966]: pgmap v74: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:12:06.638 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:12:06.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:06.663 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:06.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:12:06.691 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:06 smithi090 ceph-mon[41670]: pgmap v74: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:12:06.691 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:06 smithi090 ceph-mon[30399]: pgmap v74: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:12:06.764 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:12:06.787 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:06.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:06.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:12:06.889 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:12:06.917 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:06.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:06.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:12:07.019 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:12:07.042 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:07.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:07.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:12:07.144 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:12:07.170 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:07.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:07.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:12:07.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.165+0000 7f878d546700 1 -- 172.21.15.90:0/22180212 >> v1:172.21.15.90:6789/0 conn(0x7f878810b750 legacy=0x7f878810db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:07.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.165+0000 7f878d546700 1 -- 172.21.15.90:0/22180212 shutdown_connections 2024-04-14T15:12:07.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.165+0000 7f878d546700 1 -- 172.21.15.90:0/22180212 >> 172.21.15.90:0/22180212 conn(0x7f87880fe390 msgr2=0x7f87881007b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:12:07.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.165+0000 7f878d546700 1 -- 172.21.15.90:0/22180212 shutdown_connections 2024-04-14T15:12:07.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.165+0000 7f878d546700 1 -- 172.21.15.90:0/22180212 wait complete. 2024-04-14T15:12:07.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.166+0000 7f878d546700 1 librados: starting msgr at 2024-04-14T15:12:07.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.166+0000 7f878d546700 1 librados: starting objecter 2024-04-14T15:12:07.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.166+0000 7f878d546700 1 Processor -- start 2024-04-14T15:12:07.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.166+0000 7f878d546700 1 -- start start 2024-04-14T15:12:07.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.166+0000 7f878d546700 1 librados: setting wanted keys 2024-04-14T15:12:07.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.166+0000 7f878d546700 1 librados: calling monclient init 2024-04-14T15:12:07.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.167+0000 7f878d546700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f87881a7820 con 0x7f878810b750 2024-04-14T15:12:07.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.167+0000 7f878d546700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f87881a8a40 con 0x7f87881088a0 2024-04-14T15:12:07.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.167+0000 7f878d546700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f87881a9c80 con 0x7f878810f320 2024-04-14T15:12:07.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.167+0000 7f87867fc700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f878810b750 0x7f87881a27d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:42108/0 (socket says 172.21.15.90:42108) 2024-04-14T15:12:07.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.167+0000 7f87867fc700 1 -- 172.21.15.90:0/3031699205 learned_addr learned my addr 172.21.15.90:0/3031699205 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:12:07.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.167+0000 7f87877fe700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f878810f320 0x7f87881a5f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:39676/0 (socket says 172.21.15.90:39676) 2024-04-14T15:12:07.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.167+0000 7f876ffff700 10 client.?.objecter ms_handle_connect 0x7f878810b750 2024-04-14T15:12:07.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.167+0000 7f876ffff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:12:07.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.167+0000 7f876ffff700 10 client.?.objecter ms_handle_connect 0x7f878810f320 2024-04-14T15:12:07.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.167+0000 7f876ffff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:12:07.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.167+0000 7f876ffff700 1 -- 172.21.15.90:0/3031699205 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 867863283 0 0) 0x7f87881a7820 con 0x7f878810b750 2024-04-14T15:12:07.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.167+0000 7f876ffff700 1 -- 172.21.15.90:0/3031699205 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f87700035b0 con 0x7f878810b750 2024-04-14T15:12:07.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.167+0000 7f876ffff700 10 client.?.objecter ms_handle_connect 0x7f87881088a0 2024-04-14T15:12:07.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.167+0000 7f876ffff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:12:07.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.167+0000 7f876ffff700 1 -- 172.21.15.90:0/3031699205 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 356596142 0 0) 0x7f87881a9c80 con 0x7f878810f320 2024-04-14T15:12:07.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.167+0000 7f876ffff700 1 -- 172.21.15.90:0/3031699205 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f87881a7820 con 0x7f878810f320 2024-04-14T15:12:07.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.167+0000 7f876ffff700 1 -- 172.21.15.90:0/3031699205 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1711871316 0 0) 0x7f87881a8a40 con 0x7f87881088a0 2024-04-14T15:12:07.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.167+0000 7f876ffff700 1 -- 172.21.15.90:0/3031699205 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f87881a9c80 con 0x7f87881088a0 2024-04-14T15:12:07.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.167+0000 7f876ffff700 1 -- 172.21.15.90:0/3031699205 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1075999053 0 0) 0x7f87881a7820 con 0x7f878810f320 2024-04-14T15:12:07.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.167+0000 7f876ffff700 1 -- 172.21.15.90:0/3031699205 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f87881a8a40 con 0x7f878810f320 2024-04-14T15:12:07.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.167+0000 7f876ffff700 1 -- 172.21.15.90:0/3031699205 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f8780001db0 con 0x7f878810f320 2024-04-14T15:12:07.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.167+0000 7f876ffff700 1 -- 172.21.15.90:0/3031699205 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1340818722 0 0) 0x7f87700035b0 con 0x7f878810b750 2024-04-14T15:12:07.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.167+0000 7f876ffff700 1 -- 172.21.15.90:0/3031699205 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f87881a7820 con 0x7f878810b750 2024-04-14T15:12:07.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.167+0000 7f876ffff700 1 -- 172.21.15.90:0/3031699205 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f877c004500 con 0x7f878810b750 2024-04-14T15:12:07.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.167+0000 7f876ffff700 1 -- 172.21.15.90:0/3031699205 <== mon.2 v1:172.21.15.90:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3779214778 0 0) 0x7f87881a8a40 con 0x7f878810f320 2024-04-14T15:12:07.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.167+0000 7f876ffff700 1 -- 172.21.15.90:0/3031699205 >> v1:172.21.15.92:6789/0 conn(0x7f87881088a0 legacy=0x7f878810e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:07.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.167+0000 7f876ffff700 1 -- 172.21.15.90:0/3031699205 >> v1:172.21.15.90:6789/0 conn(0x7f878810b750 legacy=0x7f87881a27d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:07.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.167+0000 7f876ffff700 1 -- 172.21.15.90:0/3031699205 --> v1:172.21.15.90:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87881aaec0 con 0x7f878810f320 2024-04-14T15:12:07.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.168+0000 7f878d546700 1 -- 172.21.15.90:0/3031699205 --> v1:172.21.15.90:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f87881a9eb0 con 0x7f878810f320 2024-04-14T15:12:07.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.168+0000 7f878d546700 10 client.24704.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:12:07.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.168+0000 7f878d546700 1 -- 172.21.15.90:0/3031699205 --> v1:172.21.15.90:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f87881aa370 con 0x7f878810f320 2024-04-14T15:12:07.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.168+0000 7f878d546700 1 librados: init done 2024-04-14T15:12:07.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.168+0000 7f876ffff700 1 -- 172.21.15.90:0/3031699205 <== mon.2 v1:172.21.15.90:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f87800020a0 con 0x7f878810f320 2024-04-14T15:12:07.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.168+0000 7f876ffff700 1 -- 172.21.15.90:0/3031699205 <== mon.2 v1:172.21.15.90:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f8780005fe0 con 0x7f878810f320 2024-04-14T15:12:07.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.169+0000 7f878d546700 1 -- 172.21.15.90:0/3031699205 --> v1:172.21.15.90:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f8774005100 con 0x7f878810f320 2024-04-14T15:12:07.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.169+0000 7f876ffff700 1 -- 172.21.15.90:0/3031699205 <== mon.2 v1:172.21.15.90:6790/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f8780007270 con 0x7f878810f320 2024-04-14T15:12:07.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.170+0000 7f876ffff700 1 -- 172.21.15.90:0/3031699205 <== mon.2 v1:172.21.15.90:6790/0 8 ==== osd_map(99..99 src has 41..99) ==== 6320+0+0 (unknown 3476349625 0 0) 0x7f878008a3c0 con 0x7f878810f320 2024-04-14T15:12:07.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.170+0000 7f876ffff700 10 client.24704.objecter ms_dispatch 0x7f8788001550 osd_map(99..99 src has 41..99) 2024-04-14T15:12:07.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.170+0000 7f876ffff700 3 client.24704.objecter handle_osd_map got epochs [99,99] > 0 2024-04-14T15:12:07.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.170+0000 7f876ffff700 3 client.24704.objecter handle_osd_map decoding full epoch 99 2024-04-14T15:12:07.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.170+0000 7f876ffff700 20 client.24704.objecter dump_active .. 0 homeless 2024-04-14T15:12:07.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.170+0000 7f876ffff700 1 -- 172.21.15.90:0/3031699205 <== mon.2 v1:172.21.15.90:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v99) ==== 54+0+16 (unknown 2384750422 0 2420000192) 0x7f878008c8b0 con 0x7f878810f320 2024-04-14T15:12:07.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.170+0000 7f876ffff700 10 client.24704.objecter ms_handle_connect 0x7f87700713d0 2024-04-14T15:12:07.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.170+0000 7f878d546700 10 client.24704.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:12:07.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.170+0000 7f878d546700 15 client.24704.objecter _session_command_op_assign -1 1 2024-04-14T15:12:07.216 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:12:07.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.170+0000 7f878d546700 20 client.24704.objecter _get_session s=0x7f87740053e0 osd=0 3 2024-04-14T15:12:07.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.170+0000 7f878d546700 20 client.24704.objecter put_session s=0x7f87740053e0 osd=0 3 2024-04-14T15:12:07.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.170+0000 7f878d546700 20 client.24704.objecter _get_session s=0x7f87740053e0 osd=0 3 2024-04-14T15:12:07.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.170+0000 7f878d546700 15 client.24704.objecter _session_command_op_remove -1 1 2024-04-14T15:12:07.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.170+0000 7f878d546700 20 client.24704.objecter get_session s=0x7f87740053e0 osd=0 3 2024-04-14T15:12:07.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.170+0000 7f878d546700 15 client.24704.objecter _session_command_op_assign 0 1 2024-04-14T15:12:07.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.170+0000 7f878d546700 20 client.24704.objecter put_session s=0x7f87740053e0 osd=0 4 2024-04-14T15:12:07.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.170+0000 7f878d546700 10 client.24704.objecter _send_command 1 2024-04-14T15:12:07.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.170+0000 7f878d546700 1 -- 172.21.15.90:0/3031699205 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8774004b70 con 0x7f8774004730 2024-04-14T15:12:07.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.170+0000 7f876ffff700 10 client.24704.objecter ms_handle_connect 0x7f8774004730 2024-04-14T15:12:07.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.171+0000 7f876ffff700 1 -- 172.21.15.90:0/3031699205 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f8774004b70 con 0x7f8774004730 2024-04-14T15:12:07.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.171+0000 7f876ffff700 10 client.24704.objecter ms_dispatch 0x7f8788001550 command_reply(tid 1: 0 ) 2024-04-14T15:12:07.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.171+0000 7f876ffff700 10 client.24704.objecter _finish_command 1 = system:0 2024-04-14T15:12:07.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.171+0000 7f876ffff700 20 client.24704.objecter put_session s=0x7f87740053e0 osd=0 4 2024-04-14T15:12:07.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.171+0000 7f876ffff700 15 client.24704.objecter _session_command_op_remove 0 1 2024-04-14T15:12:07.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.188+0000 7f878d546700 10 client.24704.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:12:07.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.188+0000 7f878d546700 15 client.24704.objecter _session_command_op_assign -1 2 2024-04-14T15:12:07.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.188+0000 7f878d546700 20 client.24704.objecter _get_session s=0x7f87740053e0 osd=0 3 2024-04-14T15:12:07.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.188+0000 7f878d546700 20 client.24704.objecter put_session s=0x7f87740053e0 osd=0 3 2024-04-14T15:12:07.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.188+0000 7f878d546700 20 client.24704.objecter _get_session s=0x7f87740053e0 osd=0 3 2024-04-14T15:12:07.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.188+0000 7f878d546700 15 client.24704.objecter _session_command_op_remove -1 2 2024-04-14T15:12:07.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.188+0000 7f878d546700 20 client.24704.objecter get_session s=0x7f87740053e0 osd=0 3 2024-04-14T15:12:07.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.188+0000 7f878d546700 15 client.24704.objecter _session_command_op_assign 0 2 2024-04-14T15:12:07.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.188+0000 7f878d546700 20 client.24704.objecter put_session s=0x7f87740053e0 osd=0 4 2024-04-14T15:12:07.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.188+0000 7f878d546700 10 client.24704.objecter _send_command 2 2024-04-14T15:12:07.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.188+0000 7f878d546700 1 -- 172.21.15.90:0/3031699205 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f877400be00 con 0x7f8774004730 2024-04-14T15:12:07.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.189+0000 7f876ffff700 1 -- 172.21.15.90:0/3031699205 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f877400be00 con 0x7f8774004730 2024-04-14T15:12:07.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.189+0000 7f876ffff700 10 client.24704.objecter ms_dispatch 0x7f8788001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:12:07.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.189+0000 7f876ffff700 10 client.24704.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:12:07.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.189+0000 7f876ffff700 20 client.24704.objecter put_session s=0x7f87740053e0 osd=0 4 2024-04-14T15:12:07.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.189+0000 7f876ffff700 15 client.24704.objecter _session_command_op_remove 0 2 2024-04-14T15:12:07.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.190+0000 7f878d546700 10 client.24704.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:12:07.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.190+0000 7f878d546700 15 client.24704.objecter _session_command_op_assign -1 3 2024-04-14T15:12:07.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.190+0000 7f878d546700 20 client.24704.objecter _get_session s=0x7f87740040f0 osd=1 3 2024-04-14T15:12:07.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.190+0000 7f878d546700 20 client.24704.objecter put_session s=0x7f87740040f0 osd=1 3 2024-04-14T15:12:07.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.190+0000 7f878d546700 20 client.24704.objecter _get_session s=0x7f87740040f0 osd=1 3 2024-04-14T15:12:07.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.190+0000 7f878d546700 15 client.24704.objecter _session_command_op_remove -1 3 2024-04-14T15:12:07.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.190+0000 7f878d546700 20 client.24704.objecter get_session s=0x7f87740040f0 osd=1 3 2024-04-14T15:12:07.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.190+0000 7f878d546700 15 client.24704.objecter _session_command_op_assign 1 3 2024-04-14T15:12:07.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.190+0000 7f878d546700 20 client.24704.objecter put_session s=0x7f87740040f0 osd=1 4 2024-04-14T15:12:07.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.190+0000 7f878d546700 10 client.24704.objecter _send_command 3 2024-04-14T15:12:07.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.190+0000 7f878d546700 1 -- 172.21.15.90:0/3031699205 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f8774004430 con 0x7f8774002100 2024-04-14T15:12:07.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.190+0000 7f876ffff700 10 client.24704.objecter ms_handle_connect 0x7f8774002100 2024-04-14T15:12:07.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.191+0000 7f876ffff700 1 -- 172.21.15.90:0/3031699205 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f8774004430 con 0x7f8774002100 2024-04-14T15:12:07.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.191+0000 7f876ffff700 10 client.24704.objecter ms_dispatch 0x7f8788001550 command_reply(tid 3: 0 ) 2024-04-14T15:12:07.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.191+0000 7f876ffff700 10 client.24704.objecter _finish_command 3 = system:0 2024-04-14T15:12:07.221 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:12:07.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.191+0000 7f876ffff700 20 client.24704.objecter put_session s=0x7f87740040f0 osd=1 4 2024-04-14T15:12:07.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.191+0000 7f876ffff700 15 client.24704.objecter _session_command_op_remove 1 3 2024-04-14T15:12:07.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.207+0000 7f878d546700 10 client.24704.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:12:07.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.207+0000 7f878d546700 15 client.24704.objecter _session_command_op_assign -1 4 2024-04-14T15:12:07.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.207+0000 7f878d546700 20 client.24704.objecter _get_session s=0x7f87740040f0 osd=1 3 2024-04-14T15:12:07.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.207+0000 7f878d546700 20 client.24704.objecter put_session s=0x7f87740040f0 osd=1 3 2024-04-14T15:12:07.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.207+0000 7f878d546700 20 client.24704.objecter _get_session s=0x7f87740040f0 osd=1 3 2024-04-14T15:12:07.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.207+0000 7f878d546700 15 client.24704.objecter _session_command_op_remove -1 4 2024-04-14T15:12:07.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.207+0000 7f878d546700 20 client.24704.objecter get_session s=0x7f87740040f0 osd=1 3 2024-04-14T15:12:07.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.207+0000 7f878d546700 15 client.24704.objecter _session_command_op_assign 1 4 2024-04-14T15:12:07.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.207+0000 7f878d546700 20 client.24704.objecter put_session s=0x7f87740040f0 osd=1 4 2024-04-14T15:12:07.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.207+0000 7f878d546700 10 client.24704.objecter _send_command 4 2024-04-14T15:12:07.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.207+0000 7f878d546700 1 -- 172.21.15.90:0/3031699205 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f87740016c0 con 0x7f8774002100 2024-04-14T15:12:07.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.209+0000 7f876ffff700 1 -- 172.21.15.90:0/3031699205 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f87740016c0 con 0x7f8774002100 2024-04-14T15:12:07.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.209+0000 7f876ffff700 10 client.24704.objecter ms_dispatch 0x7f8788001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:12:07.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.209+0000 7f876ffff700 10 client.24704.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:12:07.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.209+0000 7f876ffff700 20 client.24704.objecter put_session s=0x7f87740040f0 osd=1 4 2024-04-14T15:12:07.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.209+0000 7f876ffff700 15 client.24704.objecter _session_command_op_remove 1 4 2024-04-14T15:12:07.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.209+0000 7f878d546700 10 client.24704.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:12:07.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.209+0000 7f878d546700 15 client.24704.objecter _session_command_op_assign -1 5 2024-04-14T15:12:07.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.209+0000 7f878d546700 20 client.24704.objecter _get_session s=0x7f877400c780 osd=2 3 2024-04-14T15:12:07.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.209+0000 7f878d546700 20 client.24704.objecter put_session s=0x7f877400c780 osd=2 3 2024-04-14T15:12:07.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.209+0000 7f878d546700 20 client.24704.objecter _get_session s=0x7f877400c780 osd=2 3 2024-04-14T15:12:07.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.209+0000 7f878d546700 15 client.24704.objecter _session_command_op_remove -1 5 2024-04-14T15:12:07.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.209+0000 7f878d546700 20 client.24704.objecter get_session s=0x7f877400c780 osd=2 3 2024-04-14T15:12:07.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.209+0000 7f878d546700 15 client.24704.objecter _session_command_op_assign 2 5 2024-04-14T15:12:07.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.209+0000 7f878d546700 20 client.24704.objecter put_session s=0x7f877400c780 osd=2 4 2024-04-14T15:12:07.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.209+0000 7f878d546700 10 client.24704.objecter _send_command 5 2024-04-14T15:12:07.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.209+0000 7f878d546700 1 -- 172.21.15.90:0/3031699205 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f87740154f0 con 0x7f87740119e0 2024-04-14T15:12:07.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.209+0000 7f876ffff700 10 client.24704.objecter ms_handle_connect 0x7f87740119e0 2024-04-14T15:12:07.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.210+0000 7f876ffff700 1 -- 172.21.15.90:0/3031699205 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f87740154f0 con 0x7f87740119e0 2024-04-14T15:12:07.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.210+0000 7f876ffff700 10 client.24704.objecter ms_dispatch 0x7f8788001550 command_reply(tid 5: 0 ) 2024-04-14T15:12:07.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.210+0000 7f876ffff700 10 client.24704.objecter _finish_command 5 = system:0 2024-04-14T15:12:07.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.210+0000 7f876ffff700 20 client.24704.objecter put_session s=0x7f877400c780 osd=2 4 2024-04-14T15:12:07.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.210+0000 7f876ffff700 15 client.24704.objecter _session_command_op_remove 2 5 2024-04-14T15:12:07.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.228+0000 7f878d546700 10 client.24704.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:12:07.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.228+0000 7f878d546700 15 client.24704.objecter _session_command_op_assign -1 6 2024-04-14T15:12:07.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.228+0000 7f878d546700 20 client.24704.objecter _get_session s=0x7f877400c780 osd=2 3 2024-04-14T15:12:07.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.228+0000 7f878d546700 20 client.24704.objecter put_session s=0x7f877400c780 osd=2 3 2024-04-14T15:12:07.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.228+0000 7f878d546700 20 client.24704.objecter _get_session s=0x7f877400c780 osd=2 3 2024-04-14T15:12:07.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.228+0000 7f878d546700 15 client.24704.objecter _session_command_op_remove -1 6 2024-04-14T15:12:07.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.228+0000 7f878d546700 20 client.24704.objecter get_session s=0x7f877400c780 osd=2 3 2024-04-14T15:12:07.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.228+0000 7f878d546700 15 client.24704.objecter _session_command_op_assign 2 6 2024-04-14T15:12:07.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.228+0000 7f878d546700 20 client.24704.objecter put_session s=0x7f877400c780 osd=2 4 2024-04-14T15:12:07.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.228+0000 7f878d546700 10 client.24704.objecter _send_command 6 2024-04-14T15:12:07.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.228+0000 7f878d546700 1 -- 172.21.15.90:0/3031699205 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f877400c300 con 0x7f87740119e0 2024-04-14T15:12:07.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.229+0000 7f876ffff700 1 -- 172.21.15.90:0/3031699205 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f877400c300 con 0x7f87740119e0 2024-04-14T15:12:07.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.229+0000 7f876ffff700 10 client.24704.objecter ms_dispatch 0x7f8788001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:12:07.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.229+0000 7f876ffff700 10 client.24704.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:12:07.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.229+0000 7f876ffff700 20 client.24704.objecter put_session s=0x7f877400c780 osd=2 4 2024-04-14T15:12:07.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.229+0000 7f876ffff700 15 client.24704.objecter _session_command_op_remove 2 6 2024-04-14T15:12:07.230 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:12:07.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.229+0000 7f878d546700 10 client.24704.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:12:07.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.229+0000 7f878d546700 15 client.24704.objecter _session_command_op_assign -1 7 2024-04-14T15:12:07.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.229+0000 7f878d546700 20 client.24704.objecter _get_session s=0x7f877400c510 osd=3 3 2024-04-14T15:12:07.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.229+0000 7f878d546700 20 client.24704.objecter put_session s=0x7f877400c510 osd=3 3 2024-04-14T15:12:07.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.229+0000 7f878d546700 20 client.24704.objecter _get_session s=0x7f877400c510 osd=3 3 2024-04-14T15:12:07.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.229+0000 7f878d546700 15 client.24704.objecter _session_command_op_remove -1 7 2024-04-14T15:12:07.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.229+0000 7f878d546700 20 client.24704.objecter get_session s=0x7f877400c510 osd=3 3 2024-04-14T15:12:07.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.229+0000 7f878d546700 15 client.24704.objecter _session_command_op_assign 3 7 2024-04-14T15:12:07.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.229+0000 7f878d546700 20 client.24704.objecter put_session s=0x7f877400c510 osd=3 4 2024-04-14T15:12:07.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.230+0000 7f878d546700 10 client.24704.objecter _send_command 7 2024-04-14T15:12:07.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.230+0000 7f878d546700 1 -- 172.21.15.90:0/3031699205 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f87740117c0 con 0x7f8774016820 2024-04-14T15:12:07.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.230+0000 7f876ffff700 10 client.24704.objecter ms_handle_connect 0x7f8774016820 2024-04-14T15:12:07.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.230+0000 7f876ffff700 1 -- 172.21.15.90:0/3031699205 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f87740117c0 con 0x7f8774016820 2024-04-14T15:12:07.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.230+0000 7f876ffff700 10 client.24704.objecter ms_dispatch 0x7f8788001550 command_reply(tid 7: 0 ) 2024-04-14T15:12:07.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.230+0000 7f876ffff700 10 client.24704.objecter _finish_command 7 = system:0 2024-04-14T15:12:07.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.230+0000 7f876ffff700 20 client.24704.objecter put_session s=0x7f877400c510 osd=3 4 2024-04-14T15:12:07.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.230+0000 7f876ffff700 15 client.24704.objecter _session_command_op_remove 3 7 2024-04-14T15:12:07.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.248+0000 7f878d546700 10 client.24704.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:12:07.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.248+0000 7f878d546700 15 client.24704.objecter _session_command_op_assign -1 8 2024-04-14T15:12:07.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.248+0000 7f878d546700 20 client.24704.objecter _get_session s=0x7f877400c510 osd=3 3 2024-04-14T15:12:07.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.248+0000 7f878d546700 20 client.24704.objecter put_session s=0x7f877400c510 osd=3 3 2024-04-14T15:12:07.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.248+0000 7f878d546700 20 client.24704.objecter _get_session s=0x7f877400c510 osd=3 3 2024-04-14T15:12:07.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.248+0000 7f878d546700 15 client.24704.objecter _session_command_op_remove -1 8 2024-04-14T15:12:07.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.248+0000 7f878d546700 20 client.24704.objecter get_session s=0x7f877400c510 osd=3 3 2024-04-14T15:12:07.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.248+0000 7f878d546700 15 client.24704.objecter _session_command_op_assign 3 8 2024-04-14T15:12:07.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.248+0000 7f878d546700 20 client.24704.objecter put_session s=0x7f877400c510 osd=3 4 2024-04-14T15:12:07.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.248+0000 7f878d546700 10 client.24704.objecter _send_command 8 2024-04-14T15:12:07.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.248+0000 7f878d546700 1 -- 172.21.15.90:0/3031699205 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f8774017bf0 con 0x7f8774016820 2024-04-14T15:12:07.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.249+0000 7f876ffff700 1 -- 172.21.15.90:0/3031699205 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f8780007020 con 0x7f8774016820 2024-04-14T15:12:07.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.249+0000 7f876ffff700 10 client.24704.objecter ms_dispatch 0x7f8788001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:12:07.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.249+0000 7f876ffff700 10 client.24704.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:12:07.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.249+0000 7f876ffff700 20 client.24704.objecter put_session s=0x7f877400c510 osd=3 4 2024-04-14T15:12:07.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.249+0000 7f876ffff700 15 client.24704.objecter _session_command_op_remove 3 8 2024-04-14T15:12:07.250 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:12:07.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.249+0000 7f878d546700 10 client.24704.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:12:07.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.249+0000 7f878d546700 15 client.24704.objecter _session_command_op_assign -1 9 2024-04-14T15:12:07.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.249+0000 7f878d546700 20 client.24704.objecter _get_session s=0x7f8774017e00 osd=4 3 2024-04-14T15:12:07.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.249+0000 7f878d546700 20 client.24704.objecter put_session s=0x7f8774017e00 osd=4 3 2024-04-14T15:12:07.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.249+0000 7f878d546700 20 client.24704.objecter _get_session s=0x7f8774017e00 osd=4 3 2024-04-14T15:12:07.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.249+0000 7f878d546700 15 client.24704.objecter _session_command_op_remove -1 9 2024-04-14T15:12:07.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.249+0000 7f878d546700 20 client.24704.objecter get_session s=0x7f8774017e00 osd=4 3 2024-04-14T15:12:07.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.249+0000 7f878d546700 15 client.24704.objecter _session_command_op_assign 4 9 2024-04-14T15:12:07.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.249+0000 7f878d546700 20 client.24704.objecter put_session s=0x7f8774017e00 osd=4 4 2024-04-14T15:12:07.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.249+0000 7f878d546700 10 client.24704.objecter _send_command 9 2024-04-14T15:12:07.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.249+0000 7f878d546700 1 -- 172.21.15.90:0/3031699205 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f877401f050 con 0x7f877401b540 2024-04-14T15:12:07.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.250+0000 7f876ffff700 10 client.24704.objecter ms_handle_connect 0x7f877401b540 2024-04-14T15:12:07.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.251+0000 7f876ffff700 1 -- 172.21.15.90:0/3031699205 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f877401f050 con 0x7f877401b540 2024-04-14T15:12:07.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.251+0000 7f876ffff700 10 client.24704.objecter ms_dispatch 0x7f8788001550 command_reply(tid 9: 0 ) 2024-04-14T15:12:07.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.251+0000 7f876ffff700 10 client.24704.objecter _finish_command 9 = system:0 2024-04-14T15:12:07.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.251+0000 7f876ffff700 20 client.24704.objecter put_session s=0x7f8774017e00 osd=4 4 2024-04-14T15:12:07.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.251+0000 7f876ffff700 15 client.24704.objecter _session_command_op_remove 4 9 2024-04-14T15:12:07.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.267+0000 7f878d546700 10 client.24704.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:12:07.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.267+0000 7f878d546700 15 client.24704.objecter _session_command_op_assign -1 10 2024-04-14T15:12:07.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.267+0000 7f878d546700 20 client.24704.objecter _get_session s=0x7f8774017e00 osd=4 3 2024-04-14T15:12:07.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.267+0000 7f878d546700 20 client.24704.objecter put_session s=0x7f8774017e00 osd=4 3 2024-04-14T15:12:07.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.267+0000 7f878d546700 20 client.24704.objecter _get_session s=0x7f8774017e00 osd=4 3 2024-04-14T15:12:07.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.267+0000 7f878d546700 15 client.24704.objecter _session_command_op_remove -1 10 2024-04-14T15:12:07.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.267+0000 7f878d546700 20 client.24704.objecter get_session s=0x7f8774017e00 osd=4 3 2024-04-14T15:12:07.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.267+0000 7f878d546700 15 client.24704.objecter _session_command_op_assign 4 10 2024-04-14T15:12:07.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.267+0000 7f878d546700 20 client.24704.objecter put_session s=0x7f8774017e00 osd=4 4 2024-04-14T15:12:07.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.267+0000 7f878d546700 10 client.24704.objecter _send_command 10 2024-04-14T15:12:07.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.267+0000 7f878d546700 1 -- 172.21.15.90:0/3031699205 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f877401f050 con 0x7f877401b540 2024-04-14T15:12:07.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.269+0000 7f876ffff700 1 -- 172.21.15.90:0/3031699205 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f877401f050 con 0x7f877401b540 2024-04-14T15:12:07.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.269+0000 7f876ffff700 10 client.24704.objecter ms_dispatch 0x7f8788001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:12:07.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.269+0000 7f876ffff700 10 client.24704.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:12:07.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.269+0000 7f876ffff700 20 client.24704.objecter put_session s=0x7f8774017e00 osd=4 4 2024-04-14T15:12:07.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.269+0000 7f876ffff700 15 client.24704.objecter _session_command_op_remove 4 10 2024-04-14T15:12:07.270 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:12:07.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.270+0000 7f878d546700 10 client.24704.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:12:07.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.270+0000 7f878d546700 15 client.24704.objecter _session_command_op_assign -1 11 2024-04-14T15:12:07.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.270+0000 7f878d546700 20 client.24704.objecter _get_session s=0x7f877401f050 osd=5 3 2024-04-14T15:12:07.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.270+0000 7f878d546700 20 client.24704.objecter put_session s=0x7f877401f050 osd=5 3 2024-04-14T15:12:07.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.270+0000 7f878d546700 20 client.24704.objecter _get_session s=0x7f877401f050 osd=5 3 2024-04-14T15:12:07.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.270+0000 7f878d546700 15 client.24704.objecter _session_command_op_remove -1 11 2024-04-14T15:12:07.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.270+0000 7f878d546700 20 client.24704.objecter get_session s=0x7f877401f050 osd=5 3 2024-04-14T15:12:07.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.270+0000 7f878d546700 15 client.24704.objecter _session_command_op_assign 5 11 2024-04-14T15:12:07.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.270+0000 7f878d546700 20 client.24704.objecter put_session s=0x7f877401f050 osd=5 4 2024-04-14T15:12:07.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.270+0000 7f878d546700 10 client.24704.objecter _send_command 11 2024-04-14T15:12:07.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.270+0000 7f878d546700 1 -- 172.21.15.90:0/3031699205 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f8774023430 con 0x7f877401f770 2024-04-14T15:12:07.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.270+0000 7f876ffff700 10 client.24704.objecter ms_handle_connect 0x7f877401f770 2024-04-14T15:12:07.272 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:12:07.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.271+0000 7f876ffff700 1 -- 172.21.15.90:0/3031699205 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f8774023430 con 0x7f877401f770 2024-04-14T15:12:07.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.271+0000 7f876ffff700 10 client.24704.objecter ms_dispatch 0x7f8788001550 command_reply(tid 11: 0 ) 2024-04-14T15:12:07.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.271+0000 7f876ffff700 10 client.24704.objecter _finish_command 11 = system:0 2024-04-14T15:12:07.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.271+0000 7f876ffff700 20 client.24704.objecter put_session s=0x7f877401f050 osd=5 4 2024-04-14T15:12:07.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.271+0000 7f876ffff700 15 client.24704.objecter _session_command_op_remove 5 11 2024-04-14T15:12:07.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.288+0000 7f878d546700 10 client.24704.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:12:07.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.288+0000 7f878d546700 15 client.24704.objecter _session_command_op_assign -1 12 2024-04-14T15:12:07.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.288+0000 7f878d546700 20 client.24704.objecter _get_session s=0x7f877401f050 osd=5 3 2024-04-14T15:12:07.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.288+0000 7f878d546700 20 client.24704.objecter put_session s=0x7f877401f050 osd=5 3 2024-04-14T15:12:07.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.288+0000 7f878d546700 20 client.24704.objecter _get_session s=0x7f877401f050 osd=5 3 2024-04-14T15:12:07.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.288+0000 7f878d546700 15 client.24704.objecter _session_command_op_remove -1 12 2024-04-14T15:12:07.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.288+0000 7f878d546700 20 client.24704.objecter get_session s=0x7f877401f050 osd=5 3 2024-04-14T15:12:07.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.288+0000 7f878d546700 15 client.24704.objecter _session_command_op_assign 5 12 2024-04-14T15:12:07.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.288+0000 7f878d546700 20 client.24704.objecter put_session s=0x7f877401f050 osd=5 4 2024-04-14T15:12:07.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.288+0000 7f878d546700 10 client.24704.objecter _send_command 12 2024-04-14T15:12:07.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.288+0000 7f878d546700 1 -- 172.21.15.90:0/3031699205 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f8774024de0 con 0x7f877401f770 2024-04-14T15:12:07.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.290+0000 7f876ffff700 1 -- 172.21.15.90:0/3031699205 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f8774024de0 con 0x7f877401f770 2024-04-14T15:12:07.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.290+0000 7f876ffff700 10 client.24704.objecter ms_dispatch 0x7f8788001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:12:07.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.290+0000 7f876ffff700 10 client.24704.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:12:07.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.290+0000 7f876ffff700 20 client.24704.objecter put_session s=0x7f877401f050 osd=5 4 2024-04-14T15:12:07.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.290+0000 7f876ffff700 15 client.24704.objecter _session_command_op_remove 5 12 2024-04-14T15:12:07.291 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:12:07.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.290+0000 7f878d546700 10 client.24704.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:12:07.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.290+0000 7f878d546700 15 client.24704.objecter _session_command_op_assign -1 13 2024-04-14T15:12:07.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.290+0000 7f878d546700 20 client.24704.objecter _get_session s=0x7f8774024de0 osd=6 3 2024-04-14T15:12:07.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.290+0000 7f878d546700 20 client.24704.objecter put_session s=0x7f8774024de0 osd=6 3 2024-04-14T15:12:07.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.290+0000 7f878d546700 20 client.24704.objecter _get_session s=0x7f8774024de0 osd=6 3 2024-04-14T15:12:07.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.290+0000 7f878d546700 15 client.24704.objecter _session_command_op_remove -1 13 2024-04-14T15:12:07.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.290+0000 7f878d546700 20 client.24704.objecter get_session s=0x7f8774024de0 osd=6 3 2024-04-14T15:12:07.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.290+0000 7f878d546700 15 client.24704.objecter _session_command_op_assign 6 13 2024-04-14T15:12:07.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.290+0000 7f878d546700 20 client.24704.objecter put_session s=0x7f8774024de0 osd=6 4 2024-04-14T15:12:07.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.290+0000 7f878d546700 10 client.24704.objecter _send_command 13 2024-04-14T15:12:07.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.290+0000 7f878d546700 1 -- 172.21.15.90:0/3031699205 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f8774025980 con 0x7f8774025500 2024-04-14T15:12:07.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.290+0000 7f876ffff700 10 client.24704.objecter ms_handle_connect 0x7f8774025500 2024-04-14T15:12:07.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.292+0000 7f876ffff700 1 -- 172.21.15.90:0/3031699205 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f8774025980 con 0x7f8774025500 2024-04-14T15:12:07.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.292+0000 7f876ffff700 10 client.24704.objecter ms_dispatch 0x7f8788001550 command_reply(tid 13: 0 ) 2024-04-14T15:12:07.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.292+0000 7f876ffff700 10 client.24704.objecter _finish_command 13 = system:0 2024-04-14T15:12:07.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.292+0000 7f876ffff700 20 client.24704.objecter put_session s=0x7f8774024de0 osd=6 4 2024-04-14T15:12:07.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.292+0000 7f876ffff700 15 client.24704.objecter _session_command_op_remove 6 13 2024-04-14T15:12:07.300 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:07.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:07.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:12:07.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.310+0000 7f878d546700 10 client.24704.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:12:07.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.310+0000 7f878d546700 15 client.24704.objecter _session_command_op_assign -1 14 2024-04-14T15:12:07.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.310+0000 7f878d546700 20 client.24704.objecter _get_session s=0x7f8774024de0 osd=6 3 2024-04-14T15:12:07.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.310+0000 7f878d546700 20 client.24704.objecter put_session s=0x7f8774024de0 osd=6 3 2024-04-14T15:12:07.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.310+0000 7f878d546700 20 client.24704.objecter _get_session s=0x7f8774024de0 osd=6 3 2024-04-14T15:12:07.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.310+0000 7f878d546700 15 client.24704.objecter _session_command_op_remove -1 14 2024-04-14T15:12:07.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.310+0000 7f878d546700 20 client.24704.objecter get_session s=0x7f8774024de0 osd=6 3 2024-04-14T15:12:07.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.310+0000 7f878d546700 15 client.24704.objecter _session_command_op_assign 6 14 2024-04-14T15:12:07.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.310+0000 7f878d546700 20 client.24704.objecter put_session s=0x7f8774024de0 osd=6 4 2024-04-14T15:12:07.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.310+0000 7f878d546700 10 client.24704.objecter _send_command 14 2024-04-14T15:12:07.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.310+0000 7f878d546700 1 -- 172.21.15.90:0/3031699205 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f8774025980 con 0x7f8774025500 2024-04-14T15:12:07.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.312+0000 7f876ffff700 1 -- 172.21.15.90:0/3031699205 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f8774025980 con 0x7f8774025500 2024-04-14T15:12:07.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.312+0000 7f876ffff700 10 client.24704.objecter ms_dispatch 0x7f8788001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:12:07.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.312+0000 7f876ffff700 10 client.24704.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:12:07.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.312+0000 7f876ffff700 20 client.24704.objecter put_session s=0x7f8774024de0 osd=6 4 2024-04-14T15:12:07.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.312+0000 7f876ffff700 15 client.24704.objecter _session_command_op_remove 6 14 2024-04-14T15:12:07.313 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:12:07.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.312+0000 7f878d546700 10 client.24704.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:12:07.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.312+0000 7f878d546700 15 client.24704.objecter _session_command_op_assign -1 15 2024-04-14T15:12:07.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.312+0000 7f878d546700 20 client.24704.objecter _get_session s=0x7f8774025980 osd=7 3 2024-04-14T15:12:07.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.312+0000 7f878d546700 20 client.24704.objecter put_session s=0x7f8774025980 osd=7 3 2024-04-14T15:12:07.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.312+0000 7f878d546700 20 client.24704.objecter _get_session s=0x7f8774025980 osd=7 3 2024-04-14T15:12:07.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.312+0000 7f878d546700 15 client.24704.objecter _session_command_op_remove -1 15 2024-04-14T15:12:07.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.312+0000 7f878d546700 20 client.24704.objecter get_session s=0x7f8774025980 osd=7 3 2024-04-14T15:12:07.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.312+0000 7f878d546700 15 client.24704.objecter _session_command_op_assign 7 15 2024-04-14T15:12:07.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.312+0000 7f878d546700 20 client.24704.objecter put_session s=0x7f8774025980 osd=7 4 2024-04-14T15:12:07.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.312+0000 7f878d546700 10 client.24704.objecter _send_command 15 2024-04-14T15:12:07.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.312+0000 7f878d546700 1 -- 172.21.15.90:0/3031699205 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f877402bdd0 con 0x7f8774025af0 2024-04-14T15:12:07.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.313+0000 7f876ffff700 10 client.24704.objecter ms_handle_connect 0x7f8774025af0 2024-04-14T15:12:07.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.314+0000 7f876ffff700 1 -- 172.21.15.90:0/3031699205 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f877402bdd0 con 0x7f8774025af0 2024-04-14T15:12:07.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.314+0000 7f876ffff700 10 client.24704.objecter ms_dispatch 0x7f8788001550 command_reply(tid 15: 0 ) 2024-04-14T15:12:07.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.314+0000 7f876ffff700 10 client.24704.objecter _finish_command 15 = system:0 2024-04-14T15:12:07.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.314+0000 7f876ffff700 20 client.24704.objecter put_session s=0x7f8774025980 osd=7 4 2024-04-14T15:12:07.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.314+0000 7f876ffff700 15 client.24704.objecter _session_command_op_remove 7 15 2024-04-14T15:12:07.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.330+0000 7f878d546700 10 client.24704.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:12:07.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.330+0000 7f878d546700 15 client.24704.objecter _session_command_op_assign -1 16 2024-04-14T15:12:07.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.330+0000 7f878d546700 20 client.24704.objecter _get_session s=0x7f8774025980 osd=7 3 2024-04-14T15:12:07.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.330+0000 7f878d546700 20 client.24704.objecter put_session s=0x7f8774025980 osd=7 3 2024-04-14T15:12:07.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.330+0000 7f878d546700 20 client.24704.objecter _get_session s=0x7f8774025980 osd=7 3 2024-04-14T15:12:07.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.330+0000 7f878d546700 15 client.24704.objecter _session_command_op_remove -1 16 2024-04-14T15:12:07.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.330+0000 7f878d546700 20 client.24704.objecter get_session s=0x7f8774025980 osd=7 3 2024-04-14T15:12:07.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.330+0000 7f878d546700 15 client.24704.objecter _session_command_op_assign 7 16 2024-04-14T15:12:07.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.330+0000 7f878d546700 20 client.24704.objecter put_session s=0x7f8774025980 osd=7 4 2024-04-14T15:12:07.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.330+0000 7f878d546700 10 client.24704.objecter _send_command 16 2024-04-14T15:12:07.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.330+0000 7f878d546700 1 -- 172.21.15.90:0/3031699205 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f877402bd90 con 0x7f8774025af0 2024-04-14T15:12:07.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.333+0000 7f876ffff700 1 -- 172.21.15.90:0/3031699205 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f877402bd90 con 0x7f8774025af0 2024-04-14T15:12:07.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.333+0000 7f876ffff700 10 client.24704.objecter ms_dispatch 0x7f8788001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:12:07.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.333+0000 7f876ffff700 10 client.24704.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:12:07.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.333+0000 7f876ffff700 20 client.24704.objecter put_session s=0x7f8774025980 osd=7 4 2024-04-14T15:12:07.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.333+0000 7f876ffff700 15 client.24704.objecter _session_command_op_remove 7 16 2024-04-14T15:12:07.334 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:12:07.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.333+0000 7f878d546700 10 librados: watch_flush enter 2024-04-14T15:12:07.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.333+0000 7f878d546700 10 librados: watch_flush exit 2024-04-14T15:12:07.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.333+0000 7f878d546700 10 client.24704.objecter close_session for osd.0 2024-04-14T15:12:07.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.333+0000 7f878d546700 1 -- 172.21.15.90:0/3031699205 >> v1:172.21.15.90:6801/3761806885 conn(0x7f8774004730 legacy=0x7f8774002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:07.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.333+0000 7f878d546700 20 client.24704.objecter put_session s=0x7f87740053e0 osd=0 1 2024-04-14T15:12:07.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.333+0000 7f878d546700 10 client.24704.objecter close_session for osd.1 2024-04-14T15:12:07.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.333+0000 7f878d546700 1 -- 172.21.15.90:0/3031699205 >> v1:172.21.15.90:6805/4170361699 conn(0x7f8774002100 legacy=0x7f877400f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:07.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.333+0000 7f878d546700 20 client.24704.objecter put_session s=0x7f87740040f0 osd=1 1 2024-04-14T15:12:07.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.333+0000 7f878d546700 10 client.24704.objecter close_session for osd.2 2024-04-14T15:12:07.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.333+0000 7f878d546700 1 -- 172.21.15.90:0/3031699205 >> v1:172.21.15.90:6809/1686184251 conn(0x7f87740119e0 legacy=0x7f8774013dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:07.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.333+0000 7f878d546700 20 client.24704.objecter put_session s=0x7f877400c780 osd=2 1 2024-04-14T15:12:07.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.333+0000 7f878d546700 10 client.24704.objecter close_session for osd.3 2024-04-14T15:12:07.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.333+0000 7f878d546700 1 -- 172.21.15.90:0/3031699205 >> v1:172.21.15.90:6813/2270166774 conn(0x7f8774016820 legacy=0x7f8774016ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:07.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.333+0000 7f878d546700 20 client.24704.objecter put_session s=0x7f877400c510 osd=3 1 2024-04-14T15:12:07.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.333+0000 7f878d546700 10 client.24704.objecter close_session for osd.4 2024-04-14T15:12:07.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.333+0000 7f878d546700 1 -- 172.21.15.90:0/3031699205 >> v1:172.21.15.92:6800/3887549733 conn(0x7f877401b540 legacy=0x7f877401d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:07.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.333+0000 7f878d546700 20 client.24704.objecter put_session s=0x7f8774017e00 osd=4 1 2024-04-14T15:12:07.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.333+0000 7f878d546700 10 client.24704.objecter close_session for osd.5 2024-04-14T15:12:07.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.333+0000 7f878d546700 1 -- 172.21.15.90:0/3031699205 >> v1:172.21.15.92:6804/3244561440 conn(0x7f877401f770 legacy=0x7f8774021b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:07.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.333+0000 7f878d546700 20 client.24704.objecter put_session s=0x7f877401f050 osd=5 1 2024-04-14T15:12:07.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.333+0000 7f878d546700 10 client.24704.objecter close_session for osd.6 2024-04-14T15:12:07.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.333+0000 7f878d546700 1 -- 172.21.15.90:0/3031699205 >> v1:172.21.15.92:6808/636125963 conn(0x7f8774025500 legacy=0x7f8774023610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:07.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.333+0000 7f878d546700 20 client.24704.objecter put_session s=0x7f8774024de0 osd=6 1 2024-04-14T15:12:07.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.333+0000 7f878d546700 10 client.24704.objecter close_session for osd.7 2024-04-14T15:12:07.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.333+0000 7f878d546700 1 -- 172.21.15.90:0/3031699205 >> v1:172.21.15.92:6812/1773706716 conn(0x7f8774025af0 legacy=0x7f877402a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:07.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.333+0000 7f878d546700 20 client.24704.objecter put_session s=0x7f8774025980 osd=7 1 2024-04-14T15:12:07.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.333+0000 7f878d546700 20 client.24704.objecter shutdown clearing up homeless session... 2024-04-14T15:12:07.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.333+0000 7f878d546700 10 client.24704.objecter successfully canceled tick 2024-04-14T15:12:07.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.333+0000 7f87877fe700 1 -- 172.21.15.90:0/3031699205 reap_dead start 2024-04-14T15:12:07.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.333+0000 7f87877fe700 1 -- 172.21.15.90:0/3031699205 reap_dead start 2024-04-14T15:12:07.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.333+0000 7f87877fe700 1 -- 172.21.15.90:0/3031699205 reap_dead start 2024-04-14T15:12:07.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.333+0000 7f87877fe700 1 -- 172.21.15.90:0/3031699205 reap_dead start 2024-04-14T15:12:07.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.333+0000 7f87877fe700 1 -- 172.21.15.90:0/3031699205 reap_dead start 2024-04-14T15:12:07.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.333+0000 7f878d546700 1 -- 172.21.15.90:0/3031699205 >> v1:172.21.15.90:6800/420935427 conn(0x7f87700713d0 legacy=0x7f8770073890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:07.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.333+0000 7f878d546700 1 -- 172.21.15.90:0/3031699205 >> v1:172.21.15.90:6790/0 conn(0x7f878810f320 legacy=0x7f87881a5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:07.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.333+0000 7f878d546700 1 -- 172.21.15.90:0/3031699205 shutdown_connections 2024-04-14T15:12:07.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.333+0000 7f878d546700 1 -- 172.21.15.90:0/3031699205 >> 172.21.15.90:0/3031699205 conn(0x7f87880fe390 msgr2=0x7f8788111fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:12:07.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.334+0000 7f878d546700 1 -- 172.21.15.90:0/3031699205 shutdown_connections 2024-04-14T15:12:07.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.334+0000 7f878d546700 1 -- 172.21.15.90:0/3031699205 wait complete. 2024-04-14T15:12:07.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:07.334+0000 7f878d546700 1 librados: shutdown 2024-04-14T15:12:07.339 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:12:07.339 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:12:07.339 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:12:07.339 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:12:07.339 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:12:07.339 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:12:07.339 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:12:07.340 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:12:07.402 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:12:07.430 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:07.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:07.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:12:07.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:07 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3031699205' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:12:07.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:07 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3031699205' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:12:07.532 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:12:07.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:07.557 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:07.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:12:07.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:07 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3031699205' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:12:07.657 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:12:07.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:07.681 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:07.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:12:07.782 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:12:07.810 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:07.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:07.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:12:07.911 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:12:07.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:07.936 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:07.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:12:08.037 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:12:08.060 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:08.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:08.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:12:08.162 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:12:08.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:08.187 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:08.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:12:08.288 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:12:08.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:08.312 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:08.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:12:08.413 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:12:08.441 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:08.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:08.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:12:08.543 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:12:08.566 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:08.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:08.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:12:08.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:08 smithi092 ceph-mon[33966]: pgmap v75: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:12:08.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:08 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:12:08.668 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:12:08.691 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:08 smithi090 ceph-mon[41670]: pgmap v75: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:12:08.691 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:08 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:12:08.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:08 smithi090 ceph-mon[30399]: pgmap v75: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:12:08.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:08 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:12:08.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:08.694 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:08.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:12:08.795 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:12:08.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:08.819 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:08.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:12:08.920 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:12:08.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:08.948 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:08.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:12:09.049 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:12:09.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:09.077 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:09.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:12:09.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.122+0000 7fb5242af700 1 -- 172.21.15.90:0/3434392844 >> v1:172.21.15.90:6789/0 conn(0x7fb51c10b760 legacy=0x7fb51c10db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:09.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.122+0000 7fb5242af700 1 -- 172.21.15.90:0/3434392844 shutdown_connections 2024-04-14T15:12:09.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.122+0000 7fb5242af700 1 -- 172.21.15.90:0/3434392844 >> 172.21.15.90:0/3434392844 conn(0x7fb51c0fe3c0 msgr2=0x7fb51c1007e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:12:09.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.122+0000 7fb5242af700 1 -- 172.21.15.90:0/3434392844 shutdown_connections 2024-04-14T15:12:09.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.122+0000 7fb5242af700 1 -- 172.21.15.90:0/3434392844 wait complete. 2024-04-14T15:12:09.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.123+0000 7fb5242af700 1 librados: starting msgr at 2024-04-14T15:12:09.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.123+0000 7fb5242af700 1 librados: starting objecter 2024-04-14T15:12:09.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.123+0000 7fb5242af700 1 Processor -- start 2024-04-14T15:12:09.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.123+0000 7fb5242af700 1 -- start start 2024-04-14T15:12:09.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.123+0000 7fb5242af700 1 librados: setting wanted keys 2024-04-14T15:12:09.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.123+0000 7fb5242af700 1 librados: calling monclient init 2024-04-14T15:12:09.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.123+0000 7fb5242af700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb51c10efa0 con 0x7fb51c10b760 2024-04-14T15:12:09.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.123+0000 7fb5242af700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb51c1a8f40 con 0x7fb51c1088b0 2024-04-14T15:12:09.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.123+0000 7fb5242af700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb51c1aa120 con 0x7fb51c10f330 2024-04-14T15:12:09.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.123+0000 7fb52284c700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7fb51c10f330 0x7fb51c1a6810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:39698/0 (socket says 172.21.15.90:39698) 2024-04-14T15:12:09.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.123+0000 7fb52184a700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fb51c10b760 0x7fb51c10c500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:42124/0 (socket says 172.21.15.90:42124) 2024-04-14T15:12:09.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.123+0000 7fb52284c700 1 -- 172.21.15.90:0/2508519724 learned_addr learned my addr 172.21.15.90:0/2508519724 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:12:09.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.124+0000 7fb5137fe700 10 client.?.objecter ms_handle_connect 0x7fb51c10b760 2024-04-14T15:12:09.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.124+0000 7fb5137fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:12:09.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.124+0000 7fb5137fe700 10 client.?.objecter ms_handle_connect 0x7fb51c10f330 2024-04-14T15:12:09.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.124+0000 7fb5137fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:12:09.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.124+0000 7fb5137fe700 1 -- 172.21.15.90:0/2508519724 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 903264020 0 0) 0x7fb51c10efa0 con 0x7fb51c10b760 2024-04-14T15:12:09.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.124+0000 7fb5137fe700 1 -- 172.21.15.90:0/2508519724 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb5080035b0 con 0x7fb51c10b760 2024-04-14T15:12:09.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.124+0000 7fb5137fe700 10 client.?.objecter ms_handle_connect 0x7fb51c1088b0 2024-04-14T15:12:09.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.124+0000 7fb5137fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:12:09.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.124+0000 7fb5137fe700 1 -- 172.21.15.90:0/2508519724 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2067319489 0 0) 0x7fb51c1aa120 con 0x7fb51c10f330 2024-04-14T15:12:09.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.124+0000 7fb5137fe700 1 -- 172.21.15.90:0/2508519724 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb51c10efa0 con 0x7fb51c10f330 2024-04-14T15:12:09.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.124+0000 7fb5137fe700 1 -- 172.21.15.90:0/2508519724 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1665365561 0 0) 0x7fb51c1a8f40 con 0x7fb51c1088b0 2024-04-14T15:12:09.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.124+0000 7fb5137fe700 1 -- 172.21.15.90:0/2508519724 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb51c1aa120 con 0x7fb51c1088b0 2024-04-14T15:12:09.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.124+0000 7fb5137fe700 1 -- 172.21.15.90:0/2508519724 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3607195819 0 0) 0x7fb51c10efa0 con 0x7fb51c10f330 2024-04-14T15:12:09.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.124+0000 7fb5137fe700 1 -- 172.21.15.90:0/2508519724 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb51c1a8f40 con 0x7fb51c10f330 2024-04-14T15:12:09.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.124+0000 7fb5137fe700 1 -- 172.21.15.90:0/2508519724 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 334311786 0 0) 0x7fb5080035b0 con 0x7fb51c10b760 2024-04-14T15:12:09.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.124+0000 7fb5137fe700 1 -- 172.21.15.90:0/2508519724 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb51c10efa0 con 0x7fb51c10b760 2024-04-14T15:12:09.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.124+0000 7fb5137fe700 1 -- 172.21.15.90:0/2508519724 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 165300001 0 0) 0x7fb51c1aa120 con 0x7fb51c1088b0 2024-04-14T15:12:09.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.124+0000 7fb5137fe700 1 -- 172.21.15.90:0/2508519724 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb5080035b0 con 0x7fb51c1088b0 2024-04-14T15:12:09.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.124+0000 7fb5137fe700 1 -- 172.21.15.90:0/2508519724 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fb50c0015d0 con 0x7fb51c10b760 2024-04-14T15:12:09.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.124+0000 7fb5137fe700 1 -- 172.21.15.90:0/2508519724 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fb514001bf0 con 0x7fb51c10f330 2024-04-14T15:12:09.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.124+0000 7fb5137fe700 1 -- 172.21.15.90:0/2508519724 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fb518001830 con 0x7fb51c1088b0 2024-04-14T15:12:09.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.124+0000 7fb5137fe700 1 -- 172.21.15.90:0/2508519724 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1921959977 0 0) 0x7fb51c10efa0 con 0x7fb51c10b760 2024-04-14T15:12:09.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.124+0000 7fb5137fe700 1 -- 172.21.15.90:0/2508519724 >> v1:172.21.15.90:6790/0 conn(0x7fb51c10f330 legacy=0x7fb51c1a6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:09.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.124+0000 7fb5137fe700 1 -- 172.21.15.90:0/2508519724 >> v1:172.21.15.92:6789/0 conn(0x7fb51c1088b0 legacy=0x7fb51c10bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:09.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.124+0000 7fb5137fe700 1 -- 172.21.15.90:0/2508519724 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb51c1ab300 con 0x7fb51c10b760 2024-04-14T15:12:09.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.125+0000 7fb5242af700 1 -- 172.21.15.90:0/2508519724 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb51c1aa350 con 0x7fb51c10b760 2024-04-14T15:12:09.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.125+0000 7fb5242af700 10 client.14844.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:12:09.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.125+0000 7fb5242af700 1 -- 172.21.15.90:0/2508519724 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb51c1aa810 con 0x7fb51c10b760 2024-04-14T15:12:09.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.125+0000 7fb5242af700 1 librados: init done 2024-04-14T15:12:09.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.125+0000 7fb5137fe700 1 -- 172.21.15.90:0/2508519724 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fb50c001b70 con 0x7fb51c10b760 2024-04-14T15:12:09.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.125+0000 7fb5137fe700 1 -- 172.21.15.90:0/2508519724 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fb50c004d80 con 0x7fb51c10b760 2024-04-14T15:12:09.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.126+0000 7fb5242af700 1 -- 172.21.15.90:0/2508519724 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb500005150 con 0x7fb51c10b760 2024-04-14T15:12:09.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.126+0000 7fb5137fe700 1 -- 172.21.15.90:0/2508519724 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fb50c003a10 con 0x7fb51c10b760 2024-04-14T15:12:09.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.127+0000 7fb5137fe700 1 -- 172.21.15.90:0/2508519724 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(99..99 src has 41..99) ==== 6320+0+0 (unknown 3476349625 0 0) 0x7fb50c08a270 con 0x7fb51c10b760 2024-04-14T15:12:09.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.127+0000 7fb5137fe700 10 client.14844.objecter ms_dispatch 0x7fb51c001550 osd_map(99..99 src has 41..99) 2024-04-14T15:12:09.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.127+0000 7fb5137fe700 3 client.14844.objecter handle_osd_map got epochs [99,99] > 0 2024-04-14T15:12:09.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.127+0000 7fb5137fe700 3 client.14844.objecter handle_osd_map decoding full epoch 99 2024-04-14T15:12:09.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.127+0000 7fb5137fe700 20 client.14844.objecter dump_active .. 0 homeless 2024-04-14T15:12:09.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.127+0000 7fb5137fe700 10 client.14844.objecter ms_handle_connect 0x7fb508075810 2024-04-14T15:12:09.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.132+0000 7fb5137fe700 1 -- 172.21.15.90:0/2508519724 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fb50c0573c0 con 0x7fb51c10b760 2024-04-14T15:12:09.178 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:12:09.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:09.205 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:09.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:12:09.306 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:12:09.335 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:09.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:09.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:12:09.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.354+0000 7fb5242af700 1 -- 172.21.15.90:0/2508519724 --> v1:172.21.15.90:6800/420935427 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb500000bc0 con 0x7fb508075810 2024-04-14T15:12:09.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.366+0000 7fb5137fe700 1 -- 172.21.15.90:0/2508519724 <== mgr.14610 v1:172.21.15.90:6800/420935427 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+346312 (unknown 2965378022 0 1851289687) 0x7fb500000bc0 con 0x7fb508075810 2024-04-14T15:12:09.371 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:12:09.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.369+0000 7fb5242af700 10 librados: watch_flush enter 2024-04-14T15:12:09.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.369+0000 7fb5242af700 10 librados: watch_flush exit 2024-04-14T15:12:09.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.369+0000 7fb5242af700 20 client.14844.objecter shutdown clearing up homeless session... 2024-04-14T15:12:09.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.369+0000 7fb5242af700 10 client.14844.objecter successfully canceled tick 2024-04-14T15:12:09.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.369+0000 7fb5242af700 1 -- 172.21.15.90:0/2508519724 >> v1:172.21.15.90:6800/420935427 conn(0x7fb508075810 legacy=0x7fb508077cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:09.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.369+0000 7fb5242af700 1 -- 172.21.15.90:0/2508519724 >> v1:172.21.15.90:6789/0 conn(0x7fb51c10b760 legacy=0x7fb51c10c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:09.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.369+0000 7fb5242af700 1 -- 172.21.15.90:0/2508519724 shutdown_connections 2024-04-14T15:12:09.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.369+0000 7fb5242af700 1 -- 172.21.15.90:0/2508519724 >> 172.21.15.90:0/2508519724 conn(0x7fb51c0fe3c0 msgr2=0x7fb51c1028b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:12:09.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.369+0000 7fb5242af700 1 -- 172.21.15.90:0/2508519724 shutdown_connections 2024-04-14T15:12:09.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.369+0000 7fb5242af700 1 -- 172.21.15.90:0/2508519724 wait complete. 2024-04-14T15:12:09.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:09.369+0000 7fb5242af700 1 librados: shutdown 2024-04-14T15:12:09.374 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-14T15:12:09.437 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:12:09.461 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:09.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:09.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:12:09.563 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:12:09.587 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:09.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:09.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:12:09.688 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:12:09.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:09.719 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:09.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:12:09.820 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:12:09.843 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:09.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:09.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:12:09.945 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:12:09.968 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:09.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:09.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:12:10.070 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:12:10.093 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:10.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:10.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:12:10.194 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:12:10.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:10.223 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:10.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:12:10.324 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:12:10.351 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:10.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:10.352 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:12:10.453 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:12:10.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:10.480 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:10.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:12:10.581 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:12:10.606 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:10.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:10.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:12:10.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:10 smithi092 ceph-mon[33966]: pgmap v76: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:12:10.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:10 smithi092 ceph-mon[33966]: from='client.14844 v1:172.21.15.90:0/2508519724' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:12:10.626 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:10 smithi090 ceph-mon[41670]: pgmap v76: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:12:10.627 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:10 smithi090 ceph-mon[41670]: from='client.14844 v1:172.21.15.90:0/2508519724' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:12:10.627 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:10 smithi090 ceph-mon[30399]: pgmap v76: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:12:10.627 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:10 smithi090 ceph-mon[30399]: from='client.14844 v1:172.21.15.90:0/2508519724' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:12:10.709 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:12:10.735 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:10.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:10.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:12:10.837 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:12:10.860 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:10.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:10.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:12:10.962 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:12:10.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:10.986 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:10.987 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:12:11.087 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:12:11.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:11.112 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:11.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:12:11.213 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:12:11.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:11.242 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:11.242 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:12:11.343 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:12:11.367 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:11.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:11.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:12:11.469 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:12:11.492 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:11.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:11.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:12:11.509 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-14T15:12:11.509 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:12:11.595 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:12:11.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:11 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:12:11.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:11 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:12:11.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:11 smithi092 ceph-mon[33966]: osdmap e100: 8 total, 8 up, 8 in 2024-04-14T15:12:11.622 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:11.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:11.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:12:11.691 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:11 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:12:11.691 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:11 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:12:11.691 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:11 smithi090 ceph-mon[30399]: osdmap e100: 8 total, 8 up, 8 in 2024-04-14T15:12:11.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:11 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:12:11.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:11 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:12:11.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:11 smithi090 ceph-mon[41670]: osdmap e100: 8 total, 8 up, 8 in 2024-04-14T15:12:11.726 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:12:11.753 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:11.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:11.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:12:11.855 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:12:11.881 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:11.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:11.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:12:11.983 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:12:12.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:12.007 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:12.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:12:12.108 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:12:12.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:12.136 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:12.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:12:12.237 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:12:12.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:12.264 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:12.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:12:12.366 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:12:12.441 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:12.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:12.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:12:12.539 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":76,"stamp":"2024-04-14T15:12:08.961495+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1062,"num_read_kb":908,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":1771324,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":621,"ondisk_log_size":621,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":240060,"kb_used_data":7260,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":749721156,"statfs":{"total":767960285184,"available":767714463744,"internally_reserved":0,"allocated":7434240,"data_stored":4427592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":238276146},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002478"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":75,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:01.260332+0000","last_change":"2024-04-14T15:11:46.589109+0000","last_active":"2024-04-14T15:12:01.260332+0000","last_peered":"2024-04-14T15:12:01.260332+0000","last_clean":"2024-04-14T15:12:01.260332+0000","last_became_active":"2024-04-14T15:09:49.129636+0000","last_became_peered":"2024-04-14T15:09:49.129636+0000","last_unstale":"2024-04-14T15:12:01.260332+0000","last_undegraded":"2024-04-14T15:12:01.260332+0000","last_fullsized":"2024-04-14T15:12:01.260332+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:46.589000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:46.589000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:52.616155+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":71,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:01.017163+0000","last_change":"2024-04-14T15:09:47.043210+0000","last_active":"2024-04-14T15:12:01.017163+0000","last_peered":"2024-04-14T15:12:01.017163+0000","last_clean":"2024-04-14T15:12:01.017163+0000","last_became_active":"2024-04-14T15:09:47.042538+0000","last_became_peered":"2024-04-14T15:09:47.042538+0000","last_unstale":"2024-04-14T15:12:01.017163+0000","last_undegraded":"2024-04-14T15:12:01.017163+0000","last_fullsized":"2024-04-14T15:12:01.017163+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:09:46.023663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:02.604380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":88,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.344398+0000","last_change":"2024-04-14T15:11:22.897018+0000","last_active":"2024-04-14T15:12:00.344398+0000","last_peered":"2024-04-14T15:12:00.344398+0000","last_clean":"2024-04-14T15:12:00.344398+0000","last_became_active":"2024-04-14T15:09:43.017351+0000","last_became_peered":"2024-04-14T15:09:43.017351+0000","last_unstale":"2024-04-14T15:12:00.344398+0000","last_undegraded":"2024-04-14T15:12:00.344398+0000","last_fullsized":"2024-04-14T15:12:00.344398+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:22.896926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:22.896926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:30.962446+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"72'15","reported_seq":58,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.510034+0000","last_change":"2024-04-14T15:11:57.121276+0000","last_active":"2024-04-14T15:12:00.510034+0000","last_peered":"2024-04-14T15:12:00.510034+0000","last_clean":"2024-04-14T15:12:00.510034+0000","last_became_active":"2024-04-14T15:11:57.120945+0000","last_became_peered":"2024-04-14T15:11:57.120945+0000","last_unstale":"2024-04-14T15:12:00.510034+0000","last_undegraded":"2024-04-14T15:12:00.510034+0000","last_fullsized":"2024-04-14T15:12:00.510034+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-04-14T15:11:27.622904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:27.622904+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:53.046182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":67,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:01.017308+0000","last_change":"2024-04-14T15:09:49.059264+0000","last_active":"2024-04-14T15:12:01.017308+0000","last_peered":"2024-04-14T15:12:01.017308+0000","last_clean":"2024-04-14T15:12:01.017308+0000","last_became_active":"2024-04-14T15:09:49.058904+0000","last_became_peered":"2024-04-14T15:09:49.058904+0000","last_unstale":"2024-04-14T15:12:01.017308+0000","last_undegraded":"2024-04-14T15:12:01.017308+0000","last_fullsized":"2024-04-14T15:12:01.017308+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:02.604431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":79,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.343968+0000","last_change":"2024-04-14T15:11:27.058062+0000","last_active":"2024-04-14T15:12:00.343968+0000","last_peered":"2024-04-14T15:12:00.343968+0000","last_clean":"2024-04-14T15:12:00.343968+0000","last_became_active":"2024-04-14T15:09:47.041486+0000","last_became_peered":"2024-04-14T15:09:47.041486+0000","last_unstale":"2024-04-14T15:12:00.343968+0000","last_undegraded":"2024-04-14T15:12:00.343968+0000","last_fullsized":"2024-04-14T15:12:00.343968+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:27.057962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:27.057962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:31.266985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.345073+0000","last_change":"2024-04-14T15:11:57.122422+0000","last_active":"2024-04-14T15:12:00.345073+0000","last_peered":"2024-04-14T15:12:00.345073+0000","last_clean":"2024-04-14T15:12:00.345073+0000","last_became_active":"2024-04-14T15:11:57.122120+0000","last_became_peered":"2024-04-14T15:11:57.122120+0000","last_unstale":"2024-04-14T15:12:00.345073+0000","last_undegraded":"2024-04-14T15:12:00.345073+0000","last_fullsized":"2024-04-14T15:12:00.345073+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:43.589625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:43.589625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:58.565475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"72'9","reported_seq":99,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:01.017315+0000","last_change":"2024-04-14T15:11:25.742331+0000","last_active":"2024-04-14T15:12:01.017315+0000","last_peered":"2024-04-14T15:12:01.017315+0000","last_clean":"2024-04-14T15:12:01.017315+0000","last_became_active":"2024-04-14T15:09:45.032079+0000","last_became_peered":"2024-04-14T15:09:45.032079+0000","last_unstale":"2024-04-14T15:12:01.017315+0000","last_undegraded":"2024-04-14T15:12:01.017315+0000","last_fullsized":"2024-04-14T15:12:01.017315+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-04-14T15:11:25.742244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:25.742244+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:49.471602+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":75,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:01.259672+0000","last_change":"2024-04-14T15:11:03.745753+0000","last_active":"2024-04-14T15:12:01.259672+0000","last_peered":"2024-04-14T15:12:01.259672+0000","last_clean":"2024-04-14T15:12:01.259672+0000","last_became_active":"2024-04-14T15:09:49.069590+0000","last_became_peered":"2024-04-14T15:09:49.069590+0000","last_unstale":"2024-04-14T15:12:01.259672+0000","last_undegraded":"2024-04-14T15:12:01.259672+0000","last_fullsized":"2024-04-14T15:12:01.259672+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:03.745644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:03.745644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:29.376857+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":79,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.345270+0000","last_change":"2024-04-14T15:11:04.881923+0000","last_active":"2024-04-14T15:12:00.345270+0000","last_peered":"2024-04-14T15:12:00.345270+0000","last_clean":"2024-04-14T15:12:00.345270+0000","last_became_active":"2024-04-14T15:09:47.046517+0000","last_became_peered":"2024-04-14T15:09:47.046517+0000","last_unstale":"2024-04-14T15:12:00.345270+0000","last_undegraded":"2024-04-14T15:12:00.345270+0000","last_fullsized":"2024-04-14T15:12:00.345270+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:04.881846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:04.881846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:24.065968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":87,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.345463+0000","last_change":"2024-04-14T15:10:43.084925+0000","last_active":"2024-04-14T15:12:00.345463+0000","last_peered":"2024-04-14T15:12:00.345463+0000","last_clean":"2024-04-14T15:12:00.345463+0000","last_became_active":"2024-04-14T15:09:43.021986+0000","last_became_peered":"2024-04-14T15:09:43.021986+0000","last_unstale":"2024-04-14T15:12:00.345463+0000","last_undegraded":"2024-04-14T15:12:00.345463+0000","last_fullsized":"2024-04-14T15:12:00.345463+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:43.084820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:10:43.084820+0000","last_clean_scrub_stamp":"2024-04-14T15:10:43.084820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:00.869023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"87'7","reported_seq":95,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:01.017394+0000","last_change":"2024-04-14T15:11:10.510929+0000","last_active":"2024-04-14T15:12:01.017394+0000","last_peered":"2024-04-14T15:12:01.017394+0000","last_clean":"2024-04-14T15:12:01.017394+0000","last_became_active":"2024-04-14T15:09:45.028667+0000","last_became_peered":"2024-04-14T15:09:45.028667+0000","last_unstale":"2024-04-14T15:12:01.017394+0000","last_undegraded":"2024-04-14T15:12:01.017394+0000","last_fullsized":"2024-04-14T15:12:01.017394+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'7","last_scrub_stamp":"2024-04-14T15:11:10.510785+0000","last_deep_scrub":"87'7","last_deep_scrub_stamp":"2024-04-14T15:11:10.510785+0000","last_clean_scrub_stamp":"2024-04-14T15:11:10.510785+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:17.842170+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":75,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.343805+0000","last_change":"2024-04-14T15:10:54.765733+0000","last_active":"2024-04-14T15:12:00.343805+0000","last_peered":"2024-04-14T15:12:00.343805+0000","last_clean":"2024-04-14T15:12:00.343805+0000","last_became_active":"2024-04-14T15:09:49.061756+0000","last_became_peered":"2024-04-14T15:09:49.061756+0000","last_unstale":"2024-04-14T15:12:00.343805+0000","last_undegraded":"2024-04-14T15:12:00.343805+0000","last_fullsized":"2024-04-14T15:12:00.343805+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:54.765650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:10:54.765650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":79,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:01.259904+0000","last_change":"2024-04-14T15:11:27.645381+0000","last_active":"2024-04-14T15:12:01.259904+0000","last_peered":"2024-04-14T15:12:01.259904+0000","last_clean":"2024-04-14T15:12:01.259904+0000","last_became_active":"2024-04-14T15:09:47.042384+0000","last_became_peered":"2024-04-14T15:09:47.042384+0000","last_unstale":"2024-04-14T15:12:01.259904+0000","last_undegraded":"2024-04-14T15:12:01.259904+0000","last_fullsized":"2024-04-14T15:12:01.259904+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:27.645307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:27.645307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:31.416607+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":65,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:01.017436+0000","last_change":"2024-04-14T15:11:45.791911+0000","last_active":"2024-04-14T15:12:01.017436+0000","last_peered":"2024-04-14T15:12:01.017436+0000","last_clean":"2024-04-14T15:12:01.017436+0000","last_became_active":"2024-04-14T15:10:07.764086+0000","last_became_peered":"2024-04-14T15:10:07.764086+0000","last_unstale":"2024-04-14T15:12:01.017436+0000","last_undegraded":"2024-04-14T15:12:01.017436+0000","last_fullsized":"2024-04-14T15:12:01.017436+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:45.791810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:45.791810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:03.789308+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"72'9","reported_seq":99,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:01.017355+0000","last_change":"2024-04-14T15:11:17.667793+0000","last_active":"2024-04-14T15:12:01.017355+0000","last_peered":"2024-04-14T15:12:01.017355+0000","last_clean":"2024-04-14T15:12:01.017355+0000","last_became_active":"2024-04-14T15:09:45.030093+0000","last_became_peered":"2024-04-14T15:09:45.030093+0000","last_unstale":"2024-04-14T15:12:01.017355+0000","last_undegraded":"2024-04-14T15:12:01.017355+0000","last_fullsized":"2024-04-14T15:12:01.017355+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-04-14T15:11:17.667682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:17.667682+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:31.898486+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":15,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:01.067128+0000","last_change":"2024-04-14T15:11:57.118519+0000","last_active":"2024-04-14T15:12:01.067128+0000","last_peered":"2024-04-14T15:12:01.067128+0000","last_clean":"2024-04-14T15:12:01.067128+0000","last_became_active":"2024-04-14T15:11:57.118128+0000","last_became_peered":"2024-04-14T15:11:57.118128+0000","last_unstale":"2024-04-14T15:12:01.067128+0000","last_undegraded":"2024-04-14T15:12:01.067128+0000","last_fullsized":"2024-04-14T15:12:01.067128+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:21.463285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:21.463285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:48.369458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":79,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:01.017672+0000","last_change":"2024-04-14T15:10:48.466668+0000","last_active":"2024-04-14T15:12:01.017672+0000","last_peered":"2024-04-14T15:12:01.017672+0000","last_clean":"2024-04-14T15:12:01.017672+0000","last_became_active":"2024-04-14T15:09:47.048045+0000","last_became_peered":"2024-04-14T15:09:47.048045+0000","last_unstale":"2024-04-14T15:12:01.017672+0000","last_undegraded":"2024-04-14T15:12:01.017672+0000","last_fullsized":"2024-04-14T15:12:01.017672+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:48.466556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:10:48.466556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:02.604448+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.345482+0000","last_change":"2024-04-14T15:11:57.118279+0000","last_active":"2024-04-14T15:12:00.345482+0000","last_peered":"2024-04-14T15:12:00.345482+0000","last_clean":"2024-04-14T15:12:00.345482+0000","last_became_active":"2024-04-14T15:11:57.117958+0000","last_became_peered":"2024-04-14T15:11:57.117958+0000","last_unstale":"2024-04-14T15:12:00.345482+0000","last_undegraded":"2024-04-14T15:12:00.345482+0000","last_fullsized":"2024-04-14T15:12:00.345482+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:22.476087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:22.476087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:41.933418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"72'12","reported_seq":56,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.345458+0000","last_change":"2024-04-14T15:11:57.119014+0000","last_active":"2024-04-14T15:12:00.345458+0000","last_peered":"2024-04-14T15:12:00.345458+0000","last_clean":"2024-04-14T15:12:00.345458+0000","last_became_active":"2024-04-14T15:11:57.118631+0000","last_became_peered":"2024-04-14T15:11:57.118631+0000","last_unstale":"2024-04-14T15:12:00.345458+0000","last_undegraded":"2024-04-14T15:12:00.345458+0000","last_fullsized":"2024-04-14T15:12:00.345458+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'12","last_scrub_stamp":"2024-04-14T15:11:14.461100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:14.461100+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:29.078742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":76,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:01.017888+0000","last_change":"2024-04-14T15:11:04.539877+0000","last_active":"2024-04-14T15:12:01.017888+0000","last_peered":"2024-04-14T15:12:01.017888+0000","last_clean":"2024-04-14T15:12:01.017888+0000","last_became_active":"2024-04-14T15:09:49.127979+0000","last_became_peered":"2024-04-14T15:09:49.127979+0000","last_unstale":"2024-04-14T15:12:01.017888+0000","last_undegraded":"2024-04-14T15:12:01.017888+0000","last_fullsized":"2024-04-14T15:12:01.017888+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:04.539789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:04.539789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:16.636271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":79,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.345180+0000","last_change":"2024-04-14T15:11:16.297521+0000","last_active":"2024-04-14T15:12:00.345180+0000","last_peered":"2024-04-14T15:12:00.345180+0000","last_clean":"2024-04-14T15:12:00.345180+0000","last_became_active":"2024-04-14T15:09:47.042733+0000","last_became_peered":"2024-04-14T15:09:47.042733+0000","last_unstale":"2024-04-14T15:12:00.345180+0000","last_undegraded":"2024-04-14T15:12:00.345180+0000","last_fullsized":"2024-04-14T15:12:00.345180+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:16.297432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:16.297432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:20.307091+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":87,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:01.067299+0000","last_change":"2024-04-14T15:11:19.182322+0000","last_active":"2024-04-14T15:12:01.067299+0000","last_peered":"2024-04-14T15:12:01.067299+0000","last_clean":"2024-04-14T15:12:01.067299+0000","last_became_active":"2024-04-14T15:09:43.021494+0000","last_became_peered":"2024-04-14T15:09:43.021494+0000","last_unstale":"2024-04-14T15:12:01.067299+0000","last_undegraded":"2024-04-14T15:12:01.067299+0000","last_fullsized":"2024-04-14T15:12:01.067299+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:19.182163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:19.182163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:36.022685+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"72'15","reported_seq":108,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.510343+0000","last_change":"2024-04-14T15:11:18.790209+0000","last_active":"2024-04-14T15:12:00.510343+0000","last_peered":"2024-04-14T15:12:00.510343+0000","last_clean":"2024-04-14T15:12:00.510343+0000","last_became_active":"2024-04-14T15:09:45.032887+0000","last_became_peered":"2024-04-14T15:09:45.032887+0000","last_unstale":"2024-04-14T15:12:00.510343+0000","last_undegraded":"2024-04-14T15:12:00.510343+0000","last_fullsized":"2024-04-14T15:12:00.510343+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-04-14T15:11:18.790117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:18.790117+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:48.354430+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":75,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.344970+0000","last_change":"2024-04-14T15:11:21.247809+0000","last_active":"2024-04-14T15:12:00.344970+0000","last_peered":"2024-04-14T15:12:00.344970+0000","last_clean":"2024-04-14T15:12:00.344970+0000","last_became_active":"2024-04-14T15:09:49.057545+0000","last_became_peered":"2024-04-14T15:09:49.057545+0000","last_unstale":"2024-04-14T15:12:00.344970+0000","last_undegraded":"2024-04-14T15:12:00.344970+0000","last_fullsized":"2024-04-14T15:12:00.344970+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:21.247695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:21.247695+0000","last_clean_scrub_stamp":"2024-04-14T15:11:21.247695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:45.860565+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":80,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.345194+0000","last_change":"2024-04-14T15:11:26.852424+0000","last_active":"2024-04-14T15:12:00.345194+0000","last_peered":"2024-04-14T15:12:00.345194+0000","last_clean":"2024-04-14T15:12:00.345194+0000","last_became_active":"2024-04-14T15:09:47.045730+0000","last_became_peered":"2024-04-14T15:09:47.045730+0000","last_unstale":"2024-04-14T15:12:00.345194+0000","last_undegraded":"2024-04-14T15:12:00.345194+0000","last_fullsized":"2024-04-14T15:12:00.345194+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:26.852293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:26.852293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:35.208466+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"65'2","reported_seq":23,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:01.066799+0000","last_change":"2024-04-14T15:11:57.118531+0000","last_active":"2024-04-14T15:12:01.066799+0000","last_peered":"2024-04-14T15:12:01.066799+0000","last_clean":"2024-04-14T15:12:01.066799+0000","last_became_active":"2024-04-14T15:11:57.118202+0000","last_became_peered":"2024-04-14T15:11:57.118202+0000","last_unstale":"2024-04-14T15:12:01.066799+0000","last_undegraded":"2024-04-14T15:12:01.066799+0000","last_fullsized":"2024-04-14T15:12:01.066799+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-04-14T15:10:57.406478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:10:57.406478+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:11.340721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"72'11","reported_seq":121,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:01.067321+0000","last_change":"2024-04-14T15:11:27.102639+0000","last_active":"2024-04-14T15:12:01.067321+0000","last_peered":"2024-04-14T15:12:01.067321+0000","last_clean":"2024-04-14T15:12:01.067321+0000","last_became_active":"2024-04-14T15:10:07.764183+0000","last_became_peered":"2024-04-14T15:10:07.764183+0000","last_unstale":"2024-04-14T15:12:01.067321+0000","last_undegraded":"2024-04-14T15:12:01.067321+0000","last_fullsized":"2024-04-14T15:12:01.067321+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-04-14T15:11:27.102549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:27.102549+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:47.191755+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"72'1","reported_seq":76,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.345640+0000","last_change":"2024-04-14T15:10:59.889342+0000","last_active":"2024-04-14T15:12:00.345640+0000","last_peered":"2024-04-14T15:12:00.345640+0000","last_clean":"2024-04-14T15:12:00.345640+0000","last_became_active":"2024-04-14T15:09:49.062051+0000","last_became_peered":"2024-04-14T15:09:49.062051+0000","last_unstale":"2024-04-14T15:12:00.345640+0000","last_undegraded":"2024-04-14T15:12:00.345640+0000","last_fullsized":"2024-04-14T15:12:00.345640+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-04-14T15:10:59.889236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:10:59.889236+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:03.689276+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"72'8","reported_seq":121,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:03.093821+0000","last_change":"2024-04-14T15:10:58.268612+0000","last_active":"2024-04-14T15:12:03.093821+0000","last_peered":"2024-04-14T15:12:03.093821+0000","last_clean":"2024-04-14T15:12:03.093821+0000","last_became_active":"2024-04-14T15:09:47.048216+0000","last_became_peered":"2024-04-14T15:09:47.048216+0000","last_unstale":"2024-04-14T15:12:03.093821+0000","last_undegraded":"2024-04-14T15:12:03.093821+0000","last_fullsized":"2024-04-14T15:12:03.093821+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:10:58.268488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:10:58.268488+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:12.986919+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":88,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:01.259769+0000","last_change":"2024-04-14T15:10:52.882061+0000","last_active":"2024-04-14T15:12:01.259769+0000","last_peered":"2024-04-14T15:12:01.259769+0000","last_clean":"2024-04-14T15:12:01.259769+0000","last_became_active":"2024-04-14T15:09:43.023257+0000","last_became_peered":"2024-04-14T15:09:43.023257+0000","last_unstale":"2024-04-14T15:12:01.259769+0000","last_undegraded":"2024-04-14T15:12:01.259769+0000","last_fullsized":"2024-04-14T15:12:01.259769+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:52.881916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:10:52.881916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:12.736485+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"72'10","reported_seq":99,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.345159+0000","last_change":"2024-04-14T15:11:17.863747+0000","last_active":"2024-04-14T15:12:00.345159+0000","last_peered":"2024-04-14T15:12:00.345159+0000","last_clean":"2024-04-14T15:12:00.345159+0000","last_became_active":"2024-04-14T15:09:45.181456+0000","last_became_peered":"2024-04-14T15:09:45.181456+0000","last_unstale":"2024-04-14T15:12:00.345159+0000","last_undegraded":"2024-04-14T15:12:00.345159+0000","last_fullsized":"2024-04-14T15:12:00.345159+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-04-14T15:11:17.863645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:17.863645+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:27.587257+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":15,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.344491+0000","last_change":"2024-04-14T15:11:57.122684+0000","last_active":"2024-04-14T15:12:00.344491+0000","last_peered":"2024-04-14T15:12:00.344491+0000","last_clean":"2024-04-14T15:12:00.344491+0000","last_became_active":"2024-04-14T15:11:57.122266+0000","last_became_peered":"2024-04-14T15:11:57.122266+0000","last_unstale":"2024-04-14T15:12:00.344491+0000","last_undegraded":"2024-04-14T15:12:00.344491+0000","last_fullsized":"2024-04-14T15:12:00.344491+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:17.499708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:17.499708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:22.796876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":80,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.345407+0000","last_change":"2024-04-14T15:11:05.917054+0000","last_active":"2024-04-14T15:12:00.345407+0000","last_peered":"2024-04-14T15:12:00.345407+0000","last_clean":"2024-04-14T15:12:00.345407+0000","last_became_active":"2024-04-14T15:09:47.044665+0000","last_became_peered":"2024-04-14T15:09:47.044665+0000","last_unstale":"2024-04-14T15:12:00.345407+0000","last_undegraded":"2024-04-14T15:12:00.345407+0000","last_fullsized":"2024-04-14T15:12:00.345407+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:05.916966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:05.916966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:35.330112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":88,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:01.067494+0000","last_change":"2024-04-14T15:10:54.239518+0000","last_active":"2024-04-14T15:12:01.067494+0000","last_peered":"2024-04-14T15:12:01.067494+0000","last_clean":"2024-04-14T15:12:01.067494+0000","last_became_active":"2024-04-14T15:09:43.023220+0000","last_became_peered":"2024-04-14T15:09:43.023220+0000","last_unstale":"2024-04-14T15:12:01.067494+0000","last_undegraded":"2024-04-14T15:12:01.067494+0000","last_fullsized":"2024-04-14T15:12:01.067494+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:54.239409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:10:54.239409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:11.539313+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"72'11","reported_seq":52,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.345145+0000","last_change":"2024-04-14T15:11:57.118379+0000","last_active":"2024-04-14T15:12:00.345145+0000","last_peered":"2024-04-14T15:12:00.345145+0000","last_clean":"2024-04-14T15:12:00.345145+0000","last_became_active":"2024-04-14T15:11:57.118033+0000","last_became_peered":"2024-04-14T15:11:57.118033+0000","last_unstale":"2024-04-14T15:12:00.345145+0000","last_undegraded":"2024-04-14T15:12:00.345145+0000","last_fullsized":"2024-04-14T15:12:00.345145+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-04-14T15:11:06.492677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:06.492677+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:07.866350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":76,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.345169+0000","last_change":"2024-04-14T15:11:07.901838+0000","last_active":"2024-04-14T15:12:00.345169+0000","last_peered":"2024-04-14T15:12:00.345169+0000","last_clean":"2024-04-14T15:12:00.345169+0000","last_became_active":"2024-04-14T15:09:49.061359+0000","last_became_peered":"2024-04-14T15:09:49.061359+0000","last_unstale":"2024-04-14T15:12:00.345169+0000","last_undegraded":"2024-04-14T15:12:00.345169+0000","last_fullsized":"2024-04-14T15:12:00.345169+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:07.901761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:07.901761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:35.316850+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":80,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:01.067181+0000","last_change":"2024-04-14T15:10:50.229008+0000","last_active":"2024-04-14T15:12:01.067181+0000","last_peered":"2024-04-14T15:12:01.067181+0000","last_clean":"2024-04-14T15:12:01.067181+0000","last_became_active":"2024-04-14T15:09:47.041074+0000","last_became_peered":"2024-04-14T15:09:47.041074+0000","last_unstale":"2024-04-14T15:12:01.067181+0000","last_undegraded":"2024-04-14T15:12:01.067181+0000","last_fullsized":"2024-04-14T15:12:01.067181+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:50.228903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:10:50.228903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:17.158842+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":88,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:01.260311+0000","last_change":"2024-04-14T15:11:08.712429+0000","last_active":"2024-04-14T15:12:01.260311+0000","last_peered":"2024-04-14T15:12:01.260311+0000","last_clean":"2024-04-14T15:12:01.260311+0000","last_became_active":"2024-04-14T15:09:43.026786+0000","last_became_peered":"2024-04-14T15:09:43.026786+0000","last_unstale":"2024-04-14T15:12:01.260311+0000","last_undegraded":"2024-04-14T15:12:01.260311+0000","last_fullsized":"2024-04-14T15:12:01.260311+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:08.712289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:08.712289+0000","last_clean_scrub_stamp":"2024-04-14T15:11:08.712289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:11.190004+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"72'4","reported_seq":82,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:01.259709+0000","last_change":"2024-04-14T15:09:45.039145+0000","last_active":"2024-04-14T15:12:01.259709+0000","last_peered":"2024-04-14T15:12:01.259709+0000","last_clean":"2024-04-14T15:12:01.259709+0000","last_became_active":"2024-04-14T15:09:45.038815+0000","last_became_peered":"2024-04-14T15:09:45.038815+0000","last_unstale":"2024-04-14T15:12:01.259709+0000","last_undegraded":"2024-04-14T15:12:01.259709+0000","last_fullsized":"2024-04-14T15:12:01.259709+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:09:44.008350+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":15,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.344639+0000","last_change":"2024-04-14T15:11:57.118649+0000","last_active":"2024-04-14T15:12:00.344639+0000","last_peered":"2024-04-14T15:12:00.344639+0000","last_clean":"2024-04-14T15:12:00.344639+0000","last_became_active":"2024-04-14T15:11:57.118363+0000","last_became_peered":"2024-04-14T15:11:57.118363+0000","last_unstale":"2024-04-14T15:12:00.344639+0000","last_undegraded":"2024-04-14T15:12:00.344639+0000","last_fullsized":"2024-04-14T15:12:00.344639+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:37.585236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:37.585236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:39.774953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":80,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.343752+0000","last_change":"2024-04-14T15:11:35.007456+0000","last_active":"2024-04-14T15:12:00.343752+0000","last_peered":"2024-04-14T15:12:00.343752+0000","last_clean":"2024-04-14T15:12:00.343752+0000","last_became_active":"2024-04-14T15:09:47.041683+0000","last_became_peered":"2024-04-14T15:09:47.041683+0000","last_unstale":"2024-04-14T15:12:00.343752+0000","last_undegraded":"2024-04-14T15:12:00.343752+0000","last_fullsized":"2024-04-14T15:12:00.343752+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:35.007228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:35.007228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:46.618105+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":88,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.345307+0000","last_change":"2024-04-14T15:11:33.055306+0000","last_active":"2024-04-14T15:12:00.345307+0000","last_peered":"2024-04-14T15:12:00.345307+0000","last_clean":"2024-04-14T15:12:00.345307+0000","last_became_active":"2024-04-14T15:09:43.023583+0000","last_became_peered":"2024-04-14T15:09:43.023583+0000","last_unstale":"2024-04-14T15:12:00.345307+0000","last_undegraded":"2024-04-14T15:12:00.345307+0000","last_fullsized":"2024-04-14T15:12:00.345307+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:33.055225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:33.055225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:34.922264+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"72'11","reported_seq":103,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:01.017602+0000","last_change":"2024-04-14T15:11:48.717808+0000","last_active":"2024-04-14T15:12:01.017602+0000","last_peered":"2024-04-14T15:12:01.017602+0000","last_clean":"2024-04-14T15:12:01.017602+0000","last_became_active":"2024-04-14T15:09:45.026065+0000","last_became_peered":"2024-04-14T15:09:45.026065+0000","last_unstale":"2024-04-14T15:12:01.017602+0000","last_undegraded":"2024-04-14T15:12:01.017602+0000","last_fullsized":"2024-04-14T15:12:01.017602+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-04-14T15:11:48.717697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:48.717697+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:02.580348+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":76,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.343409+0000","last_change":"2024-04-14T15:10:56.820210+0000","last_active":"2024-04-14T15:12:00.343409+0000","last_peered":"2024-04-14T15:12:00.343409+0000","last_clean":"2024-04-14T15:12:00.343409+0000","last_became_active":"2024-04-14T15:09:49.059711+0000","last_became_peered":"2024-04-14T15:09:49.059711+0000","last_unstale":"2024-04-14T15:12:00.343409+0000","last_undegraded":"2024-04-14T15:12:00.343409+0000","last_fullsized":"2024-04-14T15:12:00.343409+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:56.820123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:10:56.820123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.344642+0000","last_change":"2024-04-14T15:11:57.122543+0000","last_active":"2024-04-14T15:12:00.344642+0000","last_peered":"2024-04-14T15:12:00.344642+0000","last_clean":"2024-04-14T15:12:00.344642+0000","last_became_active":"2024-04-14T15:11:57.121871+0000","last_became_peered":"2024-04-14T15:11:57.121871+0000","last_unstale":"2024-04-14T15:12:00.344642+0000","last_undegraded":"2024-04-14T15:12:00.344642+0000","last_fullsized":"2024-04-14T15:12:00.344642+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:10.455300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:10.455300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:14.673176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":88,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:01.067254+0000","last_change":"2024-04-14T15:11:03.304640+0000","last_active":"2024-04-14T15:12:01.067254+0000","last_peered":"2024-04-14T15:12:01.067254+0000","last_clean":"2024-04-14T15:12:01.067254+0000","last_became_active":"2024-04-14T15:09:43.022480+0000","last_became_peered":"2024-04-14T15:09:43.022480+0000","last_unstale":"2024-04-14T15:12:01.067254+0000","last_undegraded":"2024-04-14T15:12:01.067254+0000","last_fullsized":"2024-04-14T15:12:01.067254+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:03.304541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:03.304541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:22.825961+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"72'9","reported_seq":100,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.343659+0000","last_change":"2024-04-14T15:11:29.021013+0000","last_active":"2024-04-14T15:12:00.343659+0000","last_peered":"2024-04-14T15:12:00.343659+0000","last_clean":"2024-04-14T15:12:00.343659+0000","last_became_active":"2024-04-14T15:09:45.026363+0000","last_became_peered":"2024-04-14T15:09:45.026363+0000","last_unstale":"2024-04-14T15:12:00.343659+0000","last_undegraded":"2024-04-14T15:12:00.343659+0000","last_fullsized":"2024-04-14T15:12:00.343659+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-04-14T15:11:29.020859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:29.020859+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:30.854130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":75,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:01.017957+0000","last_change":"2024-04-14T15:11:15.644792+0000","last_active":"2024-04-14T15:12:01.017957+0000","last_peered":"2024-04-14T15:12:01.017957+0000","last_clean":"2024-04-14T15:12:01.017957+0000","last_became_active":"2024-04-14T15:09:49.065242+0000","last_became_peered":"2024-04-14T15:09:49.065242+0000","last_unstale":"2024-04-14T15:12:01.017957+0000","last_undegraded":"2024-04-14T15:12:01.017957+0000","last_fullsized":"2024-04-14T15:12:01.017957+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:15.644714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:15.644714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:19.718378+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"98'9","reported_seq":45,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:03.093844+0000","last_change":"2024-04-14T15:11:57.124620+0000","last_active":"2024-04-14T15:12:03.093844+0000","last_peered":"2024-04-14T15:12:03.093844+0000","last_clean":"2024-04-14T15:12:03.093844+0000","last_became_active":"2024-04-14T15:11:57.124325+0000","last_became_peered":"2024-04-14T15:11:57.124325+0000","last_unstale":"2024-04-14T15:12:03.093844+0000","last_undegraded":"2024-04-14T15:12:03.093844+0000","last_fullsized":"2024-04-14T15:12:03.093844+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:11:09.485146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:09.485146+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:10.271965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":88,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.344877+0000","last_change":"2024-04-14T15:11:10.762163+0000","last_active":"2024-04-14T15:12:00.344877+0000","last_peered":"2024-04-14T15:12:00.344877+0000","last_clean":"2024-04-14T15:12:00.344877+0000","last_became_active":"2024-04-14T15:09:43.019301+0000","last_became_peered":"2024-04-14T15:09:43.019301+0000","last_unstale":"2024-04-14T15:12:00.344877+0000","last_undegraded":"2024-04-14T15:12:00.344877+0000","last_fullsized":"2024-04-14T15:12:00.344877+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:10.762080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:10.762080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:13.869531+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"72'9","reported_seq":99,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:01.259520+0000","last_change":"2024-04-14T15:10:56.858475+0000","last_active":"2024-04-14T15:12:01.259520+0000","last_peered":"2024-04-14T15:12:01.259520+0000","last_clean":"2024-04-14T15:12:01.259520+0000","last_became_active":"2024-04-14T15:09:45.178857+0000","last_became_peered":"2024-04-14T15:09:45.178857+0000","last_unstale":"2024-04-14T15:12:01.259520+0000","last_undegraded":"2024-04-14T15:12:01.259520+0000","last_fullsized":"2024-04-14T15:12:01.259520+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-04-14T15:10:56.858323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:10:56.858323+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:13.702279+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":75,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.344861+0000","last_change":"2024-04-14T15:11:40.922318+0000","last_active":"2024-04-14T15:12:00.344861+0000","last_peered":"2024-04-14T15:12:00.344861+0000","last_clean":"2024-04-14T15:12:00.344861+0000","last_became_active":"2024-04-14T15:09:49.066068+0000","last_became_peered":"2024-04-14T15:09:49.066068+0000","last_unstale":"2024-04-14T15:12:00.344861+0000","last_undegraded":"2024-04-14T15:12:00.344861+0000","last_fullsized":"2024-04-14T15:12:00.344861+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:40.922231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:40.922231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:59.161159+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"72'8","reported_seq":117,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:03.093825+0000","last_change":"2024-04-14T15:11:18.667210+0000","last_active":"2024-04-14T15:12:03.093825+0000","last_peered":"2024-04-14T15:12:03.093825+0000","last_clean":"2024-04-14T15:12:03.093825+0000","last_became_active":"2024-04-14T15:09:47.036688+0000","last_became_peered":"2024-04-14T15:09:47.036688+0000","last_unstale":"2024-04-14T15:12:03.093825+0000","last_undegraded":"2024-04-14T15:12:03.093825+0000","last_fullsized":"2024-04-14T15:12:03.093825+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:11:18.667049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:18.667049+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:36.115547+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":88,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:01.259850+0000","last_change":"2024-04-14T15:11:24.634784+0000","last_active":"2024-04-14T15:12:01.259850+0000","last_peered":"2024-04-14T15:12:01.259850+0000","last_clean":"2024-04-14T15:12:01.259850+0000","last_became_active":"2024-04-14T15:09:43.027776+0000","last_became_peered":"2024-04-14T15:09:43.027776+0000","last_unstale":"2024-04-14T15:12:01.259850+0000","last_undegraded":"2024-04-14T15:12:01.259850+0000","last_fullsized":"2024-04-14T15:12:01.259850+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:24.634676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:24.634676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:48.431623+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"72'10","reported_seq":48,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.344826+0000","last_change":"2024-04-14T15:11:57.126842+0000","last_active":"2024-04-14T15:12:00.344826+0000","last_peered":"2024-04-14T15:12:00.344826+0000","last_clean":"2024-04-14T15:12:00.344826+0000","last_became_active":"2024-04-14T15:11:57.126453+0000","last_became_peered":"2024-04-14T15:11:57.126453+0000","last_unstale":"2024-04-14T15:12:00.344826+0000","last_undegraded":"2024-04-14T15:12:00.344826+0000","last_fullsized":"2024-04-14T15:12:00.344826+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-04-14T15:11:11.458594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:11.458594+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:21.001376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":75,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.345451+0000","last_change":"2024-04-14T15:11:26.912839+0000","last_active":"2024-04-14T15:12:00.345451+0000","last_peered":"2024-04-14T15:12:00.345451+0000","last_clean":"2024-04-14T15:12:00.345451+0000","last_became_active":"2024-04-14T15:09:49.130438+0000","last_became_peered":"2024-04-14T15:09:49.130438+0000","last_unstale":"2024-04-14T15:12:00.345451+0000","last_undegraded":"2024-04-14T15:12:00.345451+0000","last_fullsized":"2024-04-14T15:12:00.345451+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:26.912753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:26.912753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:54.042078+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":79,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:01.260254+0000","last_change":"2024-04-14T15:10:53.900979+0000","last_active":"2024-04-14T15:12:01.260254+0000","last_peered":"2024-04-14T15:12:01.260254+0000","last_clean":"2024-04-14T15:12:01.260254+0000","last_became_active":"2024-04-14T15:09:47.045349+0000","last_became_peered":"2024-04-14T15:09:47.045349+0000","last_unstale":"2024-04-14T15:12:01.260254+0000","last_undegraded":"2024-04-14T15:12:01.260254+0000","last_fullsized":"2024-04-14T15:12:01.260254+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:53.900878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:10:53.900878+0000","last_clean_scrub_stamp":"2024-04-14T15:10:53.900878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:12.123204+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":88,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:01.066944+0000","last_change":"2024-04-14T15:11:18.194434+0000","last_active":"2024-04-14T15:12:01.066944+0000","last_peered":"2024-04-14T15:12:01.066944+0000","last_clean":"2024-04-14T15:12:01.066944+0000","last_became_active":"2024-04-14T15:09:43.018987+0000","last_became_peered":"2024-04-14T15:09:43.018987+0000","last_unstale":"2024-04-14T15:12:01.066944+0000","last_undegraded":"2024-04-14T15:12:01.066944+0000","last_fullsized":"2024-04-14T15:12:01.066944+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:18.194339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:18.194339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:27.683203+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"72'6","reported_seq":92,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:01.259811+0000","last_change":"2024-04-14T15:11:26.575575+0000","last_active":"2024-04-14T15:12:01.259811+0000","last_peered":"2024-04-14T15:12:01.259811+0000","last_clean":"2024-04-14T15:12:01.259811+0000","last_became_active":"2024-04-14T15:09:45.180037+0000","last_became_peered":"2024-04-14T15:09:45.180037+0000","last_unstale":"2024-04-14T15:12:01.259811+0000","last_undegraded":"2024-04-14T15:12:01.259811+0000","last_fullsized":"2024-04-14T15:12:01.259811+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'6","last_scrub_stamp":"2024-04-14T15:11:26.575440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:26.575440+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:27.111076+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":75,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.510152+0000","last_change":"2024-04-14T15:10:56.886642+0000","last_active":"2024-04-14T15:12:00.510152+0000","last_peered":"2024-04-14T15:12:00.510152+0000","last_clean":"2024-04-14T15:12:00.510152+0000","last_became_active":"2024-04-14T15:09:49.064296+0000","last_became_peered":"2024-04-14T15:09:49.064296+0000","last_unstale":"2024-04-14T15:12:00.510152+0000","last_undegraded":"2024-04-14T15:12:00.510152+0000","last_fullsized":"2024-04-14T15:12:00.510152+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:56.886513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:10:56.886513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:17.056264+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.343795+0000","last_change":"2024-04-14T15:11:57.119781+0000","last_active":"2024-04-14T15:12:00.343795+0000","last_peered":"2024-04-14T15:12:00.343795+0000","last_clean":"2024-04-14T15:12:00.343795+0000","last_became_active":"2024-04-14T15:11:57.119468+0000","last_became_peered":"2024-04-14T15:11:57.119468+0000","last_unstale":"2024-04-14T15:12:00.343795+0000","last_undegraded":"2024-04-14T15:12:00.343795+0000","last_fullsized":"2024-04-14T15:12:00.343795+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:07.485821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:07.485821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:25.390628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":88,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.509980+0000","last_change":"2024-04-14T15:10:43.844823+0000","last_active":"2024-04-14T15:12:00.509980+0000","last_peered":"2024-04-14T15:12:00.509980+0000","last_clean":"2024-04-14T15:12:00.509980+0000","last_became_active":"2024-04-14T15:09:43.016788+0000","last_became_peered":"2024-04-14T15:09:43.016788+0000","last_unstale":"2024-04-14T15:12:00.509980+0000","last_undegraded":"2024-04-14T15:12:00.509980+0000","last_fullsized":"2024-04-14T15:12:00.509980+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:43.844693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:10:43.844693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:08.479796+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"72'9","reported_seq":99,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.345034+0000","last_change":"2024-04-14T15:11:39.966261+0000","last_active":"2024-04-14T15:12:00.345034+0000","last_peered":"2024-04-14T15:12:00.345034+0000","last_clean":"2024-04-14T15:12:00.345034+0000","last_became_active":"2024-04-14T15:09:45.180375+0000","last_became_peered":"2024-04-14T15:09:45.180375+0000","last_unstale":"2024-04-14T15:12:00.345034+0000","last_undegraded":"2024-04-14T15:12:00.345034+0000","last_fullsized":"2024-04-14T15:12:00.345034+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-04-14T15:11:39.966171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:39.966171+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:49.895272+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":15,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.345329+0000","last_change":"2024-04-14T15:11:57.119085+0000","last_active":"2024-04-14T15:12:00.345329+0000","last_peered":"2024-04-14T15:12:00.345329+0000","last_clean":"2024-04-14T15:12:00.345329+0000","last_became_active":"2024-04-14T15:11:57.118796+0000","last_became_peered":"2024-04-14T15:11:57.118796+0000","last_unstale":"2024-04-14T15:12:00.345329+0000","last_undegraded":"2024-04-14T15:12:00.345329+0000","last_fullsized":"2024-04-14T15:12:00.345329+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:02.841041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"65'3","reported_seq":106,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:01.017860+0000","last_change":"2024-04-14T15:11:38.827544+0000","last_active":"2024-04-14T15:12:01.017860+0000","last_peered":"2024-04-14T15:12:01.017860+0000","last_clean":"2024-04-14T15:12:01.017860+0000","last_became_active":"2024-04-14T15:09:43.020490+0000","last_became_peered":"2024-04-14T15:09:43.020490+0000","last_unstale":"2024-04-14T15:12:01.017860+0000","last_undegraded":"2024-04-14T15:12:01.017860+0000","last_fullsized":"2024-04-14T15:12:01.017860+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3","last_scrub_stamp":"2024-04-14T15:11:38.827423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:38.827423+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:51.153401+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":528,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"72'12","reported_seq":107,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:01.017522+0000","last_change":"2024-04-14T15:10:53.448340+0000","last_active":"2024-04-14T15:12:01.017522+0000","last_peered":"2024-04-14T15:12:01.017522+0000","last_clean":"2024-04-14T15:12:01.017522+0000","last_became_active":"2024-04-14T15:09:45.032892+0000","last_became_peered":"2024-04-14T15:09:45.032892+0000","last_unstale":"2024-04-14T15:12:01.017522+0000","last_undegraded":"2024-04-14T15:12:01.017522+0000","last_fullsized":"2024-04-14T15:12:01.017522+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'12","last_scrub_stamp":"2024-04-14T15:10:53.448218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:10:53.448218+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:02.604482+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":80,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.509937+0000","last_change":"2024-04-14T15:11:12.867617+0000","last_active":"2024-04-14T15:12:00.509937+0000","last_peered":"2024-04-14T15:12:00.509937+0000","last_clean":"2024-04-14T15:12:00.509937+0000","last_became_active":"2024-04-14T15:09:47.049005+0000","last_became_peered":"2024-04-14T15:09:47.049005+0000","last_unstale":"2024-04-14T15:12:00.509937+0000","last_undegraded":"2024-04-14T15:12:00.509937+0000","last_fullsized":"2024-04-14T15:12:00.509937+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:12.867489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:12.867489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:42.053319+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":76,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.343466+0000","last_change":"2024-04-14T15:11:09.756246+0000","last_active":"2024-04-14T15:12:00.343466+0000","last_peered":"2024-04-14T15:12:00.343466+0000","last_clean":"2024-04-14T15:12:00.343466+0000","last_became_active":"2024-04-14T15:09:49.066648+0000","last_became_peered":"2024-04-14T15:09:49.066648+0000","last_unstale":"2024-04-14T15:12:00.343466+0000","last_undegraded":"2024-04-14T15:12:00.343466+0000","last_fullsized":"2024-04-14T15:12:00.343466+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:09.756164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:09.756164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:32.890427+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":88,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.345559+0000","last_change":"2024-04-14T15:11:42.065911+0000","last_active":"2024-04-14T15:12:00.345559+0000","last_peered":"2024-04-14T15:12:00.345559+0000","last_clean":"2024-04-14T15:12:00.345559+0000","last_became_active":"2024-04-14T15:09:43.022597+0000","last_became_peered":"2024-04-14T15:09:43.022597+0000","last_unstale":"2024-04-14T15:12:00.345559+0000","last_undegraded":"2024-04-14T15:12:00.345559+0000","last_fullsized":"2024-04-14T15:12:00.345559+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:42.065812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:42.065812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:03.153303+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"72'10","reported_seq":77,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.509977+0000","last_change":"2024-04-14T15:11:17.779528+0000","last_active":"2024-04-14T15:12:00.509977+0000","last_peered":"2024-04-14T15:12:00.509977+0000","last_clean":"2024-04-14T15:12:00.509977+0000","last_became_active":"2024-04-14T15:10:07.764042+0000","last_became_peered":"2024-04-14T15:10:07.764042+0000","last_unstale":"2024-04-14T15:12:00.509977+0000","last_undegraded":"2024-04-14T15:12:00.509977+0000","last_fullsized":"2024-04-14T15:12:00.509977+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-04-14T15:11:17.779383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:17.779383+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:33.747601+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"72'8","reported_seq":118,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:03.093550+0000","last_change":"2024-04-14T15:11:14.870904+0000","last_active":"2024-04-14T15:12:03.093550+0000","last_peered":"2024-04-14T15:12:03.093550+0000","last_clean":"2024-04-14T15:12:03.093550+0000","last_became_active":"2024-04-14T15:09:47.047699+0000","last_became_peered":"2024-04-14T15:09:47.047699+0000","last_unstale":"2024-04-14T15:12:03.093550+0000","last_undegraded":"2024-04-14T15:12:03.093550+0000","last_fullsized":"2024-04-14T15:12:03.093550+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:11:14.870772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:14.870772+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:19.702141+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":15,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:01.017710+0000","last_change":"2024-04-14T15:11:57.118237+0000","last_active":"2024-04-14T15:12:01.017710+0000","last_peered":"2024-04-14T15:12:01.017710+0000","last_clean":"2024-04-14T15:12:01.017710+0000","last_became_active":"2024-04-14T15:11:57.117900+0000","last_became_peered":"2024-04-14T15:11:57.117900+0000","last_unstale":"2024-04-14T15:12:01.017710+0000","last_undegraded":"2024-04-14T15:12:01.017710+0000","last_fullsized":"2024-04-14T15:12:01.017710+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:02.604415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":88,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:01.259872+0000","last_change":"2024-04-14T15:11:20.694315+0000","last_active":"2024-04-14T15:12:01.259872+0000","last_peered":"2024-04-14T15:12:01.259872+0000","last_clean":"2024-04-14T15:12:01.259872+0000","last_became_active":"2024-04-14T15:09:43.028362+0000","last_became_peered":"2024-04-14T15:09:43.028362+0000","last_unstale":"2024-04-14T15:12:01.259872+0000","last_undegraded":"2024-04-14T15:12:01.259872+0000","last_fullsized":"2024-04-14T15:12:01.259872+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:20.694225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:20.694225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:45.303474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"72'28","reported_seq":162,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:03.093852+0000","last_change":"2024-04-14T15:11:24.126821+0000","last_active":"2024-04-14T15:12:03.093852+0000","last_peered":"2024-04-14T15:12:03.093852+0000","last_clean":"2024-04-14T15:12:03.093852+0000","last_became_active":"2024-04-14T15:09:45.034168+0000","last_became_peered":"2024-04-14T15:09:45.034168+0000","last_unstale":"2024-04-14T15:12:03.093852+0000","last_undegraded":"2024-04-14T15:12:03.093852+0000","last_fullsized":"2024-04-14T15:12:03.093852+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'28","last_scrub_stamp":"2024-04-14T15:11:24.126715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:24.126715+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:40.303628+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":80,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.344901+0000","last_change":"2024-04-14T15:11:15.831911+0000","last_active":"2024-04-14T15:12:00.344901+0000","last_peered":"2024-04-14T15:12:00.344901+0000","last_clean":"2024-04-14T15:12:00.344901+0000","last_became_active":"2024-04-14T15:09:47.040185+0000","last_became_peered":"2024-04-14T15:09:47.040185+0000","last_unstale":"2024-04-14T15:12:00.344901+0000","last_undegraded":"2024-04-14T15:12:00.344901+0000","last_fullsized":"2024-04-14T15:12:00.344901+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:15.831779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:15.831779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:44.452408+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":76,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.345608+0000","last_change":"2024-04-14T15:11:00.939052+0000","last_active":"2024-04-14T15:12:00.345608+0000","last_peered":"2024-04-14T15:12:00.345608+0000","last_clean":"2024-04-14T15:12:00.345608+0000","last_became_active":"2024-04-14T15:09:49.130307+0000","last_became_peered":"2024-04-14T15:09:49.130307+0000","last_unstale":"2024-04-14T15:12:00.345608+0000","last_undegraded":"2024-04-14T15:12:00.345608+0000","last_fullsized":"2024-04-14T15:12:00.345608+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:00.938917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:00.938917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:17.427163+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"73'5","reported_seq":238,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:07.856860+0000","last_change":"2024-04-14T15:11:03.192957+0000","last_active":"2024-04-14T15:12:07.856860+0000","last_peered":"2024-04-14T15:12:07.856860+0000","last_clean":"2024-04-14T15:12:07.856860+0000","last_became_active":"2024-04-14T15:09:43.016544+0000","last_became_peered":"2024-04-14T15:09:43.016544+0000","last_unstale":"2024-04-14T15:12:07.856860+0000","last_undegraded":"2024-04-14T15:12:07.856860+0000","last_fullsized":"2024-04-14T15:12:07.856860+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-04-14T15:11:03.192810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:03.192810+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:21.177179+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00056542099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":140,"num_read_kb":135,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":88,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.509931+0000","last_change":"2024-04-14T15:10:42.819764+0000","last_active":"2024-04-14T15:12:00.509931+0000","last_peered":"2024-04-14T15:12:00.509931+0000","last_clean":"2024-04-14T15:12:00.509931+0000","last_became_active":"2024-04-14T15:09:43.021188+0000","last_became_peered":"2024-04-14T15:09:43.021188+0000","last_unstale":"2024-04-14T15:12:00.509931+0000","last_undegraded":"2024-04-14T15:12:00.509931+0000","last_fullsized":"2024-04-14T15:12:00.509931+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:42.819681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:10:42.819681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:02.400357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"72'13","reported_seq":111,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.343708+0000","last_change":"2024-04-14T15:11:31.082411+0000","last_active":"2024-04-14T15:12:00.343708+0000","last_peered":"2024-04-14T15:12:00.343708+0000","last_clean":"2024-04-14T15:12:00.343708+0000","last_became_active":"2024-04-14T15:09:45.033883+0000","last_became_peered":"2024-04-14T15:09:45.033883+0000","last_unstale":"2024-04-14T15:12:00.343708+0000","last_undegraded":"2024-04-14T15:12:00.343708+0000","last_fullsized":"2024-04-14T15:12:00.343708+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'13","last_scrub_stamp":"2024-04-14T15:11:31.082324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:31.082324+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:39.988990+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":80,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.510276+0000","last_change":"2024-04-14T15:10:55.841920+0000","last_active":"2024-04-14T15:12:00.510276+0000","last_peered":"2024-04-14T15:12:00.510276+0000","last_clean":"2024-04-14T15:12:00.510276+0000","last_became_active":"2024-04-14T15:09:47.049457+0000","last_became_peered":"2024-04-14T15:09:47.049457+0000","last_unstale":"2024-04-14T15:12:00.510276+0000","last_undegraded":"2024-04-14T15:12:00.510276+0000","last_fullsized":"2024-04-14T15:12:00.510276+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:55.841832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:10:55.841832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:09.497072+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":76,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:01.260490+0000","last_change":"2024-04-14T15:11:32.672340+0000","last_active":"2024-04-14T15:12:01.260490+0000","last_peered":"2024-04-14T15:12:01.260490+0000","last_clean":"2024-04-14T15:12:01.260490+0000","last_became_active":"2024-04-14T15:09:49.067031+0000","last_became_peered":"2024-04-14T15:09:49.067031+0000","last_unstale":"2024-04-14T15:12:01.260490+0000","last_undegraded":"2024-04-14T15:12:01.260490+0000","last_fullsized":"2024-04-14T15:12:01.260490+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:32.672255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:32.672255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:53.163772+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"69'2","reported_seq":39,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:01.259972+0000","last_change":"2024-04-14T15:11:57.122075+0000","last_active":"2024-04-14T15:12:01.259972+0000","last_peered":"2024-04-14T15:12:01.259972+0000","last_clean":"2024-04-14T15:12:01.259972+0000","last_became_active":"2024-04-14T15:11:57.121809+0000","last_became_peered":"2024-04-14T15:11:57.121809+0000","last_unstale":"2024-04-14T15:12:01.259972+0000","last_undegraded":"2024-04-14T15:12:01.259972+0000","last_fullsized":"2024-04-14T15:12:01.259972+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-04-14T15:11:20.500271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:20.500271+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:48.187402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":96,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:01.259864+0000","last_change":"2024-04-14T15:11:47.627424+0000","last_active":"2024-04-14T15:12:01.259864+0000","last_peered":"2024-04-14T15:12:01.259864+0000","last_clean":"2024-04-14T15:12:01.259864+0000","last_became_active":"2024-04-14T15:09:43.025421+0000","last_became_peered":"2024-04-14T15:09:43.025421+0000","last_unstale":"2024-04-14T15:12:01.259864+0000","last_undegraded":"2024-04-14T15:12:01.259864+0000","last_fullsized":"2024-04-14T15:12:01.259864+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:47.627305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:47.627305+0000","last_clean_scrub_stamp":"2024-04-14T15:11:47.627305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:06.227750+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"91'130","reported_seq":339,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.345146+0000","last_change":"2024-04-14T15:11:28.875662+0000","last_active":"2024-04-14T15:12:00.345146+0000","last_peered":"2024-04-14T15:12:00.345146+0000","last_clean":"2024-04-14T15:12:00.345146+0000","last_became_active":"2024-04-14T15:09:33.753828+0000","last_became_peered":"2024-04-14T15:09:33.753828+0000","last_unstale":"2024-04-14T15:12:00.345146+0000","last_undegraded":"2024-04-14T15:12:00.345146+0000","last_fullsized":"2024-04-14T15:12:00.345146+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:11:28.875570+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:51.231957+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"90'24","reported_seq":144,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:03.093841+0000","last_change":"2024-04-14T15:11:25.120788+0000","last_active":"2024-04-14T15:12:03.093841+0000","last_peered":"2024-04-14T15:12:03.093841+0000","last_clean":"2024-04-14T15:12:03.093841+0000","last_became_active":"2024-04-14T15:09:45.179978+0000","last_became_peered":"2024-04-14T15:09:45.179978+0000","last_unstale":"2024-04-14T15:12:03.093841+0000","last_undegraded":"2024-04-14T15:12:03.093841+0000","last_fullsized":"2024-04-14T15:12:03.093841+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'24","last_scrub_stamp":"2024-04-14T15:11:25.120692+0000","last_deep_scrub":"90'24","last_deep_scrub_stamp":"2024-04-14T15:11:25.120692+0000","last_clean_scrub_stamp":"2024-04-14T15:11:25.120692+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:53.531902+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":80,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.345540+0000","last_change":"2024-04-14T15:11:13.892949+0000","last_active":"2024-04-14T15:12:00.345540+0000","last_peered":"2024-04-14T15:12:00.345540+0000","last_clean":"2024-04-14T15:12:00.345540+0000","last_became_active":"2024-04-14T15:09:47.047201+0000","last_became_peered":"2024-04-14T15:09:47.047201+0000","last_unstale":"2024-04-14T15:12:00.345540+0000","last_undegraded":"2024-04-14T15:12:00.345540+0000","last_fullsized":"2024-04-14T15:12:00.345540+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:13.892863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:13.892863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:41.896681+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"72'1","reported_seq":77,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.344635+0000","last_change":"2024-04-14T15:11:25.095270+0000","last_active":"2024-04-14T15:12:00.344635+0000","last_peered":"2024-04-14T15:12:00.344635+0000","last_clean":"2024-04-14T15:12:00.344635+0000","last_became_active":"2024-04-14T15:09:49.065243+0000","last_became_peered":"2024-04-14T15:09:49.065243+0000","last_unstale":"2024-04-14T15:12:00.344635+0000","last_undegraded":"2024-04-14T15:12:00.344635+0000","last_fullsized":"2024-04-14T15:12:00.344635+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-04-14T15:11:25.095182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:25.095182+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:45.977272+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":95,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.344731+0000","last_change":"2024-04-14T15:11:07.696609+0000","last_active":"2024-04-14T15:12:00.344731+0000","last_peered":"2024-04-14T15:12:00.344731+0000","last_clean":"2024-04-14T15:12:00.344731+0000","last_became_active":"2024-04-14T15:09:43.018509+0000","last_became_peered":"2024-04-14T15:09:43.018509+0000","last_unstale":"2024-04-14T15:12:00.344731+0000","last_undegraded":"2024-04-14T15:12:00.344731+0000","last_fullsized":"2024-04-14T15:12:00.344731+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:07.696525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:07.696525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:17.969989+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0018592680000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":88,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.345049+0000","last_change":"2024-04-14T15:10:49.916989+0000","last_active":"2024-04-14T15:12:00.345049+0000","last_peered":"2024-04-14T15:12:00.345049+0000","last_clean":"2024-04-14T15:12:00.345049+0000","last_became_active":"2024-04-14T15:09:43.025095+0000","last_became_peered":"2024-04-14T15:09:43.025095+0000","last_unstale":"2024-04-14T15:12:00.345049+0000","last_undegraded":"2024-04-14T15:12:00.345049+0000","last_fullsized":"2024-04-14T15:12:00.345049+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:49.916884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:10:49.916884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:11.267411+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"72'12","reported_seq":102,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.344962+0000","last_change":"2024-04-14T15:10:51.840577+0000","last_active":"2024-04-14T15:12:00.344962+0000","last_peered":"2024-04-14T15:12:00.344962+0000","last_clean":"2024-04-14T15:12:00.344962+0000","last_became_active":"2024-04-14T15:09:45.037231+0000","last_became_peered":"2024-04-14T15:09:45.037231+0000","last_unstale":"2024-04-14T15:12:00.344962+0000","last_undegraded":"2024-04-14T15:12:00.344962+0000","last_fullsized":"2024-04-14T15:12:00.344962+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'12","last_scrub_stamp":"2024-04-14T15:10:51.840487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:10:51.840487+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:04.052874+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":80,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:01.260487+0000","last_change":"2024-04-14T15:11:26.662123+0000","last_active":"2024-04-14T15:12:01.260487+0000","last_peered":"2024-04-14T15:12:01.260487+0000","last_clean":"2024-04-14T15:12:01.260487+0000","last_became_active":"2024-04-14T15:09:47.036696+0000","last_became_peered":"2024-04-14T15:09:47.036696+0000","last_unstale":"2024-04-14T15:12:01.260487+0000","last_undegraded":"2024-04-14T15:12:01.260487+0000","last_fullsized":"2024-04-14T15:12:01.260487+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:26.662032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:26.662032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:28.937693+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":76,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.343874+0000","last_change":"2024-04-14T15:11:35.993757+0000","last_active":"2024-04-14T15:12:00.343874+0000","last_peered":"2024-04-14T15:12:00.343874+0000","last_clean":"2024-04-14T15:12:00.343874+0000","last_became_active":"2024-04-14T15:09:49.128136+0000","last_became_peered":"2024-04-14T15:09:49.128136+0000","last_unstale":"2024-04-14T15:12:00.343874+0000","last_undegraded":"2024-04-14T15:12:00.343874+0000","last_fullsized":"2024-04-14T15:12:00.343874+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:35.993681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:35.993681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:46.320717+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":88,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.345644+0000","last_change":"2024-04-14T15:10:53.992688+0000","last_active":"2024-04-14T15:12:00.345644+0000","last_peered":"2024-04-14T15:12:00.345644+0000","last_clean":"2024-04-14T15:12:00.345644+0000","last_became_active":"2024-04-14T15:09:43.021006+0000","last_became_peered":"2024-04-14T15:09:43.021006+0000","last_unstale":"2024-04-14T15:12:00.345644+0000","last_undegraded":"2024-04-14T15:12:00.345644+0000","last_fullsized":"2024-04-14T15:12:00.345644+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:53.992590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:10:53.992590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:16.131673+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"72'19","reported_seq":120,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.344846+0000","last_change":"2024-04-14T15:10:47.880742+0000","last_active":"2024-04-14T15:12:00.344846+0000","last_peered":"2024-04-14T15:12:00.344846+0000","last_clean":"2024-04-14T15:12:00.344846+0000","last_became_active":"2024-04-14T15:09:45.179182+0000","last_became_peered":"2024-04-14T15:09:45.179182+0000","last_unstale":"2024-04-14T15:12:00.344846+0000","last_undegraded":"2024-04-14T15:12:00.344846+0000","last_fullsized":"2024-04-14T15:12:00.344846+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'19","last_scrub_stamp":"2024-04-14T15:10:47.880623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:10:47.880623+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:07.612241+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":80,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:01.067316+0000","last_change":"2024-04-14T15:10:59.286474+0000","last_active":"2024-04-14T15:12:01.067316+0000","last_peered":"2024-04-14T15:12:01.067316+0000","last_clean":"2024-04-14T15:12:01.067316+0000","last_became_active":"2024-04-14T15:09:47.048427+0000","last_became_peered":"2024-04-14T15:09:47.048427+0000","last_unstale":"2024-04-14T15:12:01.067316+0000","last_undegraded":"2024-04-14T15:12:01.067316+0000","last_fullsized":"2024-04-14T15:12:01.067316+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:59.286374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:10:59.286374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:06.066070+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":76,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:01.017592+0000","last_change":"2024-04-14T15:11:41.838377+0000","last_active":"2024-04-14T15:12:01.017592+0000","last_peered":"2024-04-14T15:12:01.017592+0000","last_clean":"2024-04-14T15:12:01.017592+0000","last_became_active":"2024-04-14T15:09:49.065684+0000","last_became_peered":"2024-04-14T15:09:49.065684+0000","last_unstale":"2024-04-14T15:12:01.017592+0000","last_undegraded":"2024-04-14T15:12:01.017592+0000","last_fullsized":"2024-04-14T15:12:01.017592+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:41.838273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:41.838273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:50.252756+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"91'2","reported_seq":90,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.343159+0000","last_change":"2024-04-14T15:11:37.974863+0000","last_active":"2024-04-14T15:12:00.343159+0000","last_peered":"2024-04-14T15:12:00.343159+0000","last_clean":"2024-04-14T15:12:00.343159+0000","last_became_active":"2024-04-14T15:09:43.018478+0000","last_became_peered":"2024-04-14T15:09:43.018478+0000","last_unstale":"2024-04-14T15:12:00.343159+0000","last_undegraded":"2024-04-14T15:12:00.343159+0000","last_fullsized":"2024-04-14T15:12:00.343159+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'2","last_scrub_stamp":"2024-04-14T15:11:37.974770+0000","last_deep_scrub":"91'2","last_deep_scrub_stamp":"2024-04-14T15:11:37.974770+0000","last_clean_scrub_stamp":"2024-04-14T15:11:37.974770+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:05.657376+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"72'18","reported_seq":65,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.345378+0000","last_change":"2024-04-14T15:11:57.119345+0000","last_active":"2024-04-14T15:12:00.345378+0000","last_peered":"2024-04-14T15:12:00.345378+0000","last_clean":"2024-04-14T15:12:00.345378+0000","last_became_active":"2024-04-14T15:11:57.118744+0000","last_became_peered":"2024-04-14T15:11:57.118744+0000","last_unstale":"2024-04-14T15:12:00.345378+0000","last_undegraded":"2024-04-14T15:12:00.345378+0000","last_fullsized":"2024-04-14T15:12:00.345378+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'18","last_scrub_stamp":"2024-04-14T15:10:53.457087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:10:53.457087+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:21.316436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":33,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":80,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:01.017453+0000","last_change":"2024-04-14T15:11:36.850301+0000","last_active":"2024-04-14T15:12:01.017453+0000","last_peered":"2024-04-14T15:12:01.017453+0000","last_clean":"2024-04-14T15:12:01.017453+0000","last_became_active":"2024-04-14T15:09:47.044610+0000","last_became_peered":"2024-04-14T15:09:47.044610+0000","last_unstale":"2024-04-14T15:12:01.017453+0000","last_undegraded":"2024-04-14T15:12:01.017453+0000","last_fullsized":"2024-04-14T15:12:01.017453+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:36.850204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:36.850204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:46.499366+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":76,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.345651+0000","last_change":"2024-04-14T15:11:16.821265+0000","last_active":"2024-04-14T15:12:00.345651+0000","last_peered":"2024-04-14T15:12:00.345651+0000","last_clean":"2024-04-14T15:12:00.345651+0000","last_became_active":"2024-04-14T15:09:49.129610+0000","last_became_peered":"2024-04-14T15:09:49.129610+0000","last_unstale":"2024-04-14T15:12:00.345651+0000","last_undegraded":"2024-04-14T15:12:00.345651+0000","last_fullsized":"2024-04-14T15:12:00.345651+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:16.821140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:16.821140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:32.897119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":88,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:01.067429+0000","last_change":"2024-04-14T15:11:20.219646+0000","last_active":"2024-04-14T15:12:01.067429+0000","last_peered":"2024-04-14T15:12:01.067429+0000","last_clean":"2024-04-14T15:12:01.067429+0000","last_became_active":"2024-04-14T15:09:43.020503+0000","last_became_peered":"2024-04-14T15:09:43.020503+0000","last_unstale":"2024-04-14T15:12:01.067429+0000","last_undegraded":"2024-04-14T15:12:01.067429+0000","last_fullsized":"2024-04-14T15:12:01.067429+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:20.219543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:20.219543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:39.250425+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"72'14","reported_seq":105,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:01.017298+0000","last_change":"2024-04-14T15:11:14.657178+0000","last_active":"2024-04-14T15:12:01.017298+0000","last_peered":"2024-04-14T15:12:01.017298+0000","last_clean":"2024-04-14T15:12:01.017298+0000","last_became_active":"2024-04-14T15:09:45.025673+0000","last_became_peered":"2024-04-14T15:09:45.025673+0000","last_unstale":"2024-04-14T15:12:01.017298+0000","last_undegraded":"2024-04-14T15:12:01.017298+0000","last_fullsized":"2024-04-14T15:12:01.017298+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'14","last_scrub_stamp":"2024-04-14T15:11:14.657075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:14.657075+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:37.152537+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"98'9","reported_seq":42,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:03.093574+0000","last_change":"2024-04-14T15:11:57.118444+0000","last_active":"2024-04-14T15:12:03.093574+0000","last_peered":"2024-04-14T15:12:03.093574+0000","last_clean":"2024-04-14T15:12:03.093574+0000","last_became_active":"2024-04-14T15:11:57.118068+0000","last_became_peered":"2024-04-14T15:11:57.118068+0000","last_unstale":"2024-04-14T15:12:03.093574+0000","last_undegraded":"2024-04-14T15:12:03.093574+0000","last_fullsized":"2024-04-14T15:12:03.093574+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:11:40.562108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:40.562108+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:41.606283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":15,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:01.067109+0000","last_change":"2024-04-14T15:11:57.121216+0000","last_active":"2024-04-14T15:12:01.067109+0000","last_peered":"2024-04-14T15:12:01.067109+0000","last_clean":"2024-04-14T15:12:01.067109+0000","last_became_active":"2024-04-14T15:11:57.120861+0000","last_became_peered":"2024-04-14T15:11:57.120861+0000","last_unstale":"2024-04-14T15:12:01.067109+0000","last_undegraded":"2024-04-14T15:12:01.067109+0000","last_fullsized":"2024-04-14T15:12:01.067109+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:01.501784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:01.501784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:18.808269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":80,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:01.260449+0000","last_change":"2024-04-14T15:11:02.763497+0000","last_active":"2024-04-14T15:12:01.260449+0000","last_peered":"2024-04-14T15:12:01.260449+0000","last_clean":"2024-04-14T15:12:01.260449+0000","last_became_active":"2024-04-14T15:09:47.043021+0000","last_became_peered":"2024-04-14T15:09:47.043021+0000","last_unstale":"2024-04-14T15:12:01.260449+0000","last_undegraded":"2024-04-14T15:12:01.260449+0000","last_fullsized":"2024-04-14T15:12:01.260449+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:02.763399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:02.763399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:06.127325+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":88,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.345149+0000","last_change":"2024-04-14T15:10:48.939920+0000","last_active":"2024-04-14T15:12:00.345149+0000","last_peered":"2024-04-14T15:12:00.345149+0000","last_clean":"2024-04-14T15:12:00.345149+0000","last_became_active":"2024-04-14T15:09:43.018601+0000","last_became_peered":"2024-04-14T15:09:43.018601+0000","last_unstale":"2024-04-14T15:12:00.345149+0000","last_undegraded":"2024-04-14T15:12:00.345149+0000","last_fullsized":"2024-04-14T15:12:00.345149+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:48.939790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:10:48.939790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:14.888606+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"85'16","reported_seq":108,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:01.017931+0000","last_change":"2024-04-14T15:10:47.456301+0000","last_active":"2024-04-14T15:12:01.017931+0000","last_peered":"2024-04-14T15:12:01.017931+0000","last_clean":"2024-04-14T15:12:01.017931+0000","last_became_active":"2024-04-14T15:09:45.028212+0000","last_became_peered":"2024-04-14T15:09:45.028212+0000","last_unstale":"2024-04-14T15:12:01.017931+0000","last_undegraded":"2024-04-14T15:12:01.017931+0000","last_fullsized":"2024-04-14T15:12:01.017931+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-14T15:10:47.456169+0000","last_deep_scrub":"85'16","last_deep_scrub_stamp":"2024-04-14T15:10:47.456169+0000","last_clean_scrub_stamp":"2024-04-14T15:10:47.456169+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:06.615924+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":76,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:01.260318+0000","last_change":"2024-04-14T15:11:25.638017+0000","last_active":"2024-04-14T15:12:01.260318+0000","last_peered":"2024-04-14T15:12:01.260318+0000","last_clean":"2024-04-14T15:12:01.260318+0000","last_became_active":"2024-04-14T15:09:49.058940+0000","last_became_peered":"2024-04-14T15:09:49.058940+0000","last_unstale":"2024-04-14T15:12:01.260318+0000","last_undegraded":"2024-04-14T15:12:01.260318+0000","last_fullsized":"2024-04-14T15:12:01.260318+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:25.637888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:25.637888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:55.011351+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"72'8","reported_seq":118,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:03.944893+0000","last_change":"2024-04-14T15:11:29.730498+0000","last_active":"2024-04-14T15:12:03.944893+0000","last_peered":"2024-04-14T15:12:03.944893+0000","last_clean":"2024-04-14T15:12:03.944893+0000","last_became_active":"2024-04-14T15:09:47.050962+0000","last_became_peered":"2024-04-14T15:09:47.050962+0000","last_unstale":"2024-04-14T15:12:03.944893+0000","last_undegraded":"2024-04-14T15:12:03.944893+0000","last_fullsized":"2024-04-14T15:12:03.944893+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:11:29.730407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:29.730407+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:53.855034+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":88,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.344132+0000","last_change":"2024-04-14T15:11:11.285681+0000","last_active":"2024-04-14T15:12:00.344132+0000","last_peered":"2024-04-14T15:12:00.344132+0000","last_clean":"2024-04-14T15:12:00.344132+0000","last_became_active":"2024-04-14T15:09:43.020878+0000","last_became_peered":"2024-04-14T15:09:43.020878+0000","last_unstale":"2024-04-14T15:12:00.344132+0000","last_undegraded":"2024-04-14T15:12:00.344132+0000","last_fullsized":"2024-04-14T15:12:00.344132+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:11.285588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:11.285588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:11.582228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"72'15","reported_seq":109,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.343911+0000","last_change":"2024-04-14T15:11:22.219265+0000","last_active":"2024-04-14T15:12:00.343911+0000","last_peered":"2024-04-14T15:12:00.343911+0000","last_clean":"2024-04-14T15:12:00.343911+0000","last_became_active":"2024-04-14T15:09:45.033331+0000","last_became_peered":"2024-04-14T15:09:45.033331+0000","last_unstale":"2024-04-14T15:12:00.343911+0000","last_undegraded":"2024-04-14T15:12:00.343911+0000","last_fullsized":"2024-04-14T15:12:00.343911+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-04-14T15:11:22.219174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:22.219174+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:23.840704+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":68,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:01.017778+0000","last_change":"2024-04-14T15:09:49.065592+0000","last_active":"2024-04-14T15:12:01.017778+0000","last_peered":"2024-04-14T15:12:01.017778+0000","last_clean":"2024-04-14T15:12:01.017778+0000","last_became_active":"2024-04-14T15:09:49.065274+0000","last_became_peered":"2024-04-14T15:09:49.065274+0000","last_unstale":"2024-04-14T15:12:01.017778+0000","last_undegraded":"2024-04-14T15:12:01.017778+0000","last_fullsized":"2024-04-14T15:12:01.017778+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"72'8","reported_seq":118,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:03.942457+0000","last_change":"2024-04-14T15:11:21.827356+0000","last_active":"2024-04-14T15:12:03.942457+0000","last_peered":"2024-04-14T15:12:03.942457+0000","last_clean":"2024-04-14T15:12:03.942457+0000","last_became_active":"2024-04-14T15:09:47.044085+0000","last_became_peered":"2024-04-14T15:09:47.044085+0000","last_unstale":"2024-04-14T15:12:03.942457+0000","last_undegraded":"2024-04-14T15:12:03.942457+0000","last_fullsized":"2024-04-14T15:12:03.942457+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:11:21.827272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:21.827272+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:40.383654+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":88,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.343515+0000","last_change":"2024-04-14T15:10:46.280858+0000","last_active":"2024-04-14T15:12:00.343515+0000","last_peered":"2024-04-14T15:12:00.343515+0000","last_clean":"2024-04-14T15:12:00.343515+0000","last_became_active":"2024-04-14T15:09:43.018871+0000","last_became_peered":"2024-04-14T15:09:43.018871+0000","last_unstale":"2024-04-14T15:12:00.343515+0000","last_undegraded":"2024-04-14T15:12:00.343515+0000","last_fullsized":"2024-04-14T15:12:00.343515+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:46.280773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:10:46.280773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"72'10","reported_seq":99,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:01.017584+0000","last_change":"2024-04-14T15:11:42.802009+0000","last_active":"2024-04-14T15:12:01.017584+0000","last_peered":"2024-04-14T15:12:01.017584+0000","last_clean":"2024-04-14T15:12:01.017584+0000","last_became_active":"2024-04-14T15:09:45.029170+0000","last_became_peered":"2024-04-14T15:09:45.029170+0000","last_unstale":"2024-04-14T15:12:01.017584+0000","last_undegraded":"2024-04-14T15:12:01.017584+0000","last_fullsized":"2024-04-14T15:12:01.017584+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-04-14T15:11:42.801888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:42.801888+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:09.615426+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":76,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.510423+0000","last_change":"2024-04-14T15:11:10.852797+0000","last_active":"2024-04-14T15:12:00.510423+0000","last_peered":"2024-04-14T15:12:00.510423+0000","last_clean":"2024-04-14T15:12:00.510423+0000","last_became_active":"2024-04-14T15:09:49.069594+0000","last_became_peered":"2024-04-14T15:09:49.069594+0000","last_unstale":"2024-04-14T15:12:00.510423+0000","last_undegraded":"2024-04-14T15:12:00.510423+0000","last_fullsized":"2024-04-14T15:12:00.510423+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:10.852715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:10.852715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:18.595232+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":72,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.344778+0000","last_change":"2024-04-14T15:09:47.043481+0000","last_active":"2024-04-14T15:12:00.344778+0000","last_peered":"2024-04-14T15:12:00.344778+0000","last_clean":"2024-04-14T15:12:00.344778+0000","last_became_active":"2024-04-14T15:09:47.042810+0000","last_became_peered":"2024-04-14T15:09:47.042810+0000","last_unstale":"2024-04-14T15:12:00.344778+0000","last_undegraded":"2024-04-14T15:12:00.344778+0000","last_fullsized":"2024-04-14T15:12:00.344778+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:09:46.023663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":88,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.345242+0000","last_change":"2024-04-14T15:11:01.903252+0000","last_active":"2024-04-14T15:12:00.345242+0000","last_peered":"2024-04-14T15:12:00.345242+0000","last_clean":"2024-04-14T15:12:00.345242+0000","last_became_active":"2024-04-14T15:09:43.022351+0000","last_became_peered":"2024-04-14T15:09:43.022351+0000","last_unstale":"2024-04-14T15:12:00.345242+0000","last_undegraded":"2024-04-14T15:12:00.345242+0000","last_fullsized":"2024-04-14T15:12:00.345242+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:01.903150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:01.903150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:27.833698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"72'17","reported_seq":136,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:01.017841+0000","last_change":"2024-04-14T15:11:09.502519+0000","last_active":"2024-04-14T15:12:01.017841+0000","last_peered":"2024-04-14T15:12:01.017841+0000","last_clean":"2024-04-14T15:12:01.017841+0000","last_became_active":"2024-04-14T15:10:07.766008+0000","last_became_peered":"2024-04-14T15:10:07.766008+0000","last_unstale":"2024-04-14T15:12:01.017841+0000","last_undegraded":"2024-04-14T15:12:01.017841+0000","last_fullsized":"2024-04-14T15:12:01.017841+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'17","last_scrub_stamp":"2024-04-14T15:11:09.502431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:09.502431+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:29.834036+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":76,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.345009+0000","last_change":"2024-04-14T15:11:15.845024+0000","last_active":"2024-04-14T15:12:00.345009+0000","last_peered":"2024-04-14T15:12:00.345009+0000","last_clean":"2024-04-14T15:12:00.345009+0000","last_became_active":"2024-04-14T15:09:49.065397+0000","last_became_peered":"2024-04-14T15:09:49.065397+0000","last_unstale":"2024-04-14T15:12:00.345009+0000","last_undegraded":"2024-04-14T15:12:00.345009+0000","last_fullsized":"2024-04-14T15:12:00.345009+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:15.844942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:15.844942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:39.797576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":80,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.510257+0000","last_change":"2024-04-14T15:10:59.793878+0000","last_active":"2024-04-14T15:12:00.510257+0000","last_peered":"2024-04-14T15:12:00.510257+0000","last_clean":"2024-04-14T15:12:00.510257+0000","last_became_active":"2024-04-14T15:09:47.045542+0000","last_became_peered":"2024-04-14T15:09:47.045542+0000","last_unstale":"2024-04-14T15:12:00.510257+0000","last_undegraded":"2024-04-14T15:12:00.510257+0000","last_fullsized":"2024-04-14T15:12:00.510257+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:59.793768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:10:59.793768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:24.997192+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":88,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.510298+0000","last_change":"2024-04-14T15:10:45.837586+0000","last_active":"2024-04-14T15:12:00.510298+0000","last_peered":"2024-04-14T15:12:00.510298+0000","last_clean":"2024-04-14T15:12:00.510298+0000","last_became_active":"2024-04-14T15:09:43.020268+0000","last_became_peered":"2024-04-14T15:09:43.020268+0000","last_unstale":"2024-04-14T15:12:00.510298+0000","last_undegraded":"2024-04-14T15:12:00.510298+0000","last_fullsized":"2024-04-14T15:12:00.510298+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:45.837506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:10:45.837506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:04.829576+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"72'19","reported_seq":115,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:01.067201+0000","last_change":"2024-04-14T15:11:09.225462+0000","last_active":"2024-04-14T15:12:01.067201+0000","last_peered":"2024-04-14T15:12:01.067201+0000","last_clean":"2024-04-14T15:12:01.067201+0000","last_became_active":"2024-04-14T15:09:45.178782+0000","last_became_peered":"2024-04-14T15:09:45.178782+0000","last_unstale":"2024-04-14T15:12:01.067201+0000","last_undegraded":"2024-04-14T15:12:01.067201+0000","last_fullsized":"2024-04-14T15:12:01.067201+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'19","last_scrub_stamp":"2024-04-14T15:11:09.225359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:09.225359+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:29.861536+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":76,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.344420+0000","last_change":"2024-04-14T15:10:53.789872+0000","last_active":"2024-04-14T15:12:00.344420+0000","last_peered":"2024-04-14T15:12:00.344420+0000","last_clean":"2024-04-14T15:12:00.344420+0000","last_became_active":"2024-04-14T15:09:49.061495+0000","last_became_peered":"2024-04-14T15:09:49.061495+0000","last_unstale":"2024-04-14T15:12:00.344420+0000","last_undegraded":"2024-04-14T15:12:00.344420+0000","last_fullsized":"2024-04-14T15:12:00.344420+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:53.789790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:10:53.789790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:04.052893+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":80,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.510172+0000","last_change":"2024-04-14T15:10:50.788471+0000","last_active":"2024-04-14T15:12:00.510172+0000","last_peered":"2024-04-14T15:12:00.510172+0000","last_clean":"2024-04-14T15:12:00.510172+0000","last_became_active":"2024-04-14T15:09:47.046153+0000","last_became_peered":"2024-04-14T15:09:47.046153+0000","last_unstale":"2024-04-14T15:12:00.510172+0000","last_undegraded":"2024-04-14T15:12:00.510172+0000","last_fullsized":"2024-04-14T15:12:00.510172+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:50.788350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:10:50.788350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:08.479830+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":88,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.345357+0000","last_change":"2024-04-14T15:11:04.223680+0000","last_active":"2024-04-14T15:12:00.345357+0000","last_peered":"2024-04-14T15:12:00.345357+0000","last_clean":"2024-04-14T15:12:00.345357+0000","last_became_active":"2024-04-14T15:09:43.020044+0000","last_became_peered":"2024-04-14T15:09:43.020044+0000","last_unstale":"2024-04-14T15:12:00.345357+0000","last_undegraded":"2024-04-14T15:12:00.345357+0000","last_fullsized":"2024-04-14T15:12:00.345357+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:04.223605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:04.223605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:11.422287+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"72'9","reported_seq":100,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.344927+0000","last_change":"2024-04-14T15:10:50.850226+0000","last_active":"2024-04-14T15:12:00.344927+0000","last_peered":"2024-04-14T15:12:00.344927+0000","last_clean":"2024-04-14T15:12:00.344927+0000","last_became_active":"2024-04-14T15:09:45.037179+0000","last_became_peered":"2024-04-14T15:09:45.037179+0000","last_unstale":"2024-04-14T15:12:00.344927+0000","last_undegraded":"2024-04-14T15:12:00.344927+0000","last_fullsized":"2024-04-14T15:12:00.344927+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-04-14T15:10:50.850116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:10:50.850116+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:06.196819+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":76,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:01.259771+0000","last_change":"2024-04-14T15:11:31.682399+0000","last_active":"2024-04-14T15:12:01.259771+0000","last_peered":"2024-04-14T15:12:01.259771+0000","last_clean":"2024-04-14T15:12:01.259771+0000","last_became_active":"2024-04-14T15:09:49.128070+0000","last_became_peered":"2024-04-14T15:09:49.128070+0000","last_unstale":"2024-04-14T15:12:01.259771+0000","last_undegraded":"2024-04-14T15:12:01.259771+0000","last_fullsized":"2024-04-14T15:12:01.259771+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:31.682321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:31.682321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:52.297577+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":88,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:00.510376+0000","last_change":"2024-04-14T15:11:13.838572+0000","last_active":"2024-04-14T15:12:00.510376+0000","last_peered":"2024-04-14T15:12:00.510376+0000","last_clean":"2024-04-14T15:12:00.510376+0000","last_became_active":"2024-04-14T15:09:43.021460+0000","last_became_peered":"2024-04-14T15:09:43.021460+0000","last_unstale":"2024-04-14T15:12:00.510376+0000","last_undegraded":"2024-04-14T15:12:00.510376+0000","last_fullsized":"2024-04-14T15:12:00.510376+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:13.838468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:13.838468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:31.461401+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"72'15","reported_seq":109,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:01.259505+0000","last_change":"2024-04-14T15:11:19.680463+0000","last_active":"2024-04-14T15:12:01.259505+0000","last_peered":"2024-04-14T15:12:01.259505+0000","last_clean":"2024-04-14T15:12:01.259505+0000","last_became_active":"2024-04-14T15:09:45.180160+0000","last_became_peered":"2024-04-14T15:09:45.180160+0000","last_unstale":"2024-04-14T15:12:01.259505+0000","last_undegraded":"2024-04-14T15:12:01.259505+0000","last_fullsized":"2024-04-14T15:12:01.259505+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-04-14T15:11:19.680308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:19.680308+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:41.707865+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"85'9","reported_seq":119,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-04-14T15:12:03.945421+0000","last_change":"2024-04-14T15:10:51.950673+0000","last_active":"2024-04-14T15:12:03.945421+0000","last_peered":"2024-04-14T15:12:03.945421+0000","last_clean":"2024-04-14T15:12:03.945421+0000","last_became_active":"2024-04-14T15:09:47.045441+0000","last_became_peered":"2024-04-14T15:09:47.045441+0000","last_unstale":"2024-04-14T15:12:03.945421+0000","last_undegraded":"2024-04-14T15:12:03.945421+0000","last_fullsized":"2024-04-14T15:12:03.945421+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9","last_scrub_stamp":"2024-04-14T15:10:51.950580+0000","last_deep_scrub":"85'9","last_deep_scrub_stamp":"2024-04-14T15:10:51.950580+0000","last_clean_scrub_stamp":"2024-04-14T15:10:51.950580+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:09.843217+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67,"ondisk_log_size":67,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":693,"num_read_kb":450,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":408,"ondisk_log_size":408,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":140,"num_read_kb":135,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103203,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28284,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716868,"statfs":{"total":95995035648,"available":95966072832,"internally_reserved":0,"allocated":1294336,"data_stored":922322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397542,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28280,"kb_used_data":1260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716872,"statfs":{"total":95995035648,"available":95966076928,"internally_reserved":0,"allocated":1290240,"data_stored":922615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574912,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31896,"kb_used_data":716,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713256,"statfs":{"total":95995035648,"available":95962374144,"internally_reserved":0,"allocated":733184,"data_stored":332368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738437,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32452,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712700,"statfs":{"total":95995035648,"available":95961804800,"internally_reserved":0,"allocated":1302528,"data_stored":922975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411388,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31848,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713304,"statfs":{"total":95995035648,"available":95962423296,"internally_reserved":0,"allocated":684032,"data_stored":331469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182454,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27732,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717420,"statfs":{"total":95995035648,"available":95966638080,"internally_reserved":0,"allocated":729088,"data_stored":332689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986224,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717440,"statfs":{"total":95995035648,"available":95966658560,"internally_reserved":0,"allocated":708608,"data_stored":331850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659181,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31856,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713296,"statfs":{"total":95995035648,"available":95962415104,"internally_reserved":0,"allocated":692224,"data_stored":331304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-14T15:12:12.547 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj -9, currently 1 2024-04-14T15:12:12.547 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:12:12.574 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:12.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:12.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:12:12.629 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-14T15:12:12.676 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:12:12.703 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:12.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:12.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:12:12.777 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:12 smithi090 ceph-mon[41670]: pgmap v78: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:12:12.777 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:12 smithi090 ceph-mon[30399]: pgmap v78: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:12:12.805 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:12:12.831 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:12.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:12.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:12:12.933 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:12:12.957 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:12.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:12.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:12:13.043 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:12:13.060 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:12:13.086 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-14T15:12:13.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:13.095 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:13.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:12:13.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:12 smithi092 ceph-mon[33966]: pgmap v78: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:12:13.196 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:12:13.221 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:13.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:13.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:12:13.323 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:12:13.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:13.352 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:13.352 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:12:13.453 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:12:13.478 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:13.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:13.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:12:13.579 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:12:13.606 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:13.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:13.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:12:13.710 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:12:13.738 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:13.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:13.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:12:13.840 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:12:13.863 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:13.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:13.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:12:13.965 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:12:13.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:13.989 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:13.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:12:14.090 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:12:14.114 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:14.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:14.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:12:14.191 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:12:13 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:12:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:12:14.216 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:12:14.243 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:14.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:14.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:12:14.345 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:12:14.372 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:14.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:14.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:12:14.474 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:12:14.498 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:14.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:14.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:12:14.600 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:12:14.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:14.624 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:14.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:12:14.725 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:12:14.749 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:14.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:14.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:12:14.851 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:12:14.877 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:14.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:14.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:12:14.979 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:12:15.002 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:15.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:15.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:12:15.104 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:12:15.122 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:12:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:12:15.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:14 smithi092 ceph-mon[33966]: pgmap v79: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:12:15.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:14 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:12:15.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:14 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:12:15.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:14 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:12:15.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:14 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:12:15.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:14 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:12:15.128 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:15.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:15.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:12:15.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:14 smithi090 ceph-mon[30399]: pgmap v79: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:12:15.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:14 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:12:15.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:14 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:12:15.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:14 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:12:15.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:14 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:12:15.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:14 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:12:15.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:14 smithi090 ceph-mon[41670]: pgmap v79: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:12:15.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:14 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:12:15.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:14 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:12:15.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:14 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:12:15.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:14 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:12:15.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:14 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:12:15.230 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:12:15.258 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:15.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:15.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:12:15.340 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:12:15.359 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:12:15.376 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:12:15.388 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:15.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:15.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:12:15.490 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:12:15.517 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:15.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:15.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:12:15.619 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:12:15.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:15.649 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:15.649 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:12:15.750 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:12:15.776 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:15.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:15.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:12:15.878 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:12:15.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:15.906 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:15.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:12:15.941 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:15 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:12:15.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:15 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:12:16.007 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:12:16.034 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:16.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:16.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:12:16.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:15 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:12:16.135 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:12:16.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:16.159 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:16.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:12:16.260 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:12:16.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:16.288 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:16.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:12:16.389 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:12:16.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:16.414 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:16.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:12:16.515 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:12:16.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:16.543 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:16.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:12:16.645 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:12:16.670 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:16.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:16.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:12:16.772 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:12:16.799 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:16.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:16.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:12:16.901 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:12:16.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:16.925 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:16.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:12:17.026 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:12:17.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:17.052 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:17.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:12:17.087 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:16 smithi090 ceph-mon[41670]: pgmap v80: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:12:17.087 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:16 smithi090 ceph-mon[30399]: pgmap v80: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:12:17.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:16 smithi092 ceph-mon[33966]: pgmap v80: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:12:17.153 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:12:17.181 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:17.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:17.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:12:17.283 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:12:17.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:17.311 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:17.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:12:17.412 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:12:17.438 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:17.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:17.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:12:17.542 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:12:17.548 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-14T15:12:17.548 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-14T15:12:17.549 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-04-14T15:12:17.549 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph pg dump --format=json 2024-04-14T15:12:17.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:17.568 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:17.568 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:12:17.669 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:12:17.697 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:17.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:17.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:12:17.799 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:12:17.823 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:17.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:17.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:12:17.925 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:12:17.953 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:17.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:17.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:12:18.055 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:12:18.082 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:18.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:18.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:12:18.183 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:12:18.208 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:18.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:18.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:12:18.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.237+0000 7f472d014700 1 -- 172.21.15.90:0/2178797228 >> v1:172.21.15.90:6789/0 conn(0x7f472810a740 legacy=0x7f472810cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:18.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.237+0000 7f472d014700 1 -- 172.21.15.90:0/2178797228 shutdown_connections 2024-04-14T15:12:18.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.237+0000 7f472d014700 1 -- 172.21.15.90:0/2178797228 >> 172.21.15.90:0/2178797228 conn(0x7f47280fd290 msgr2=0x7f47280ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:12:18.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.237+0000 7f472d014700 1 -- 172.21.15.90:0/2178797228 shutdown_connections 2024-04-14T15:12:18.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.237+0000 7f472d014700 1 -- 172.21.15.90:0/2178797228 wait complete. 2024-04-14T15:12:18.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.238+0000 7f472d014700 1 librados: starting msgr at 2024-04-14T15:12:18.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.238+0000 7f472d014700 1 librados: starting objecter 2024-04-14T15:12:18.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.238+0000 7f472d014700 1 Processor -- start 2024-04-14T15:12:18.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.238+0000 7f472d014700 1 -- start start 2024-04-14T15:12:18.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.238+0000 7f472d014700 1 librados: setting wanted keys 2024-04-14T15:12:18.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.238+0000 7f472d014700 1 librados: calling monclient init 2024-04-14T15:12:18.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.239+0000 7f472d014700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f47281a9cc0 con 0x7f4728107890 2024-04-14T15:12:18.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.239+0000 7f472d014700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f47281aaee0 con 0x7f472810a740 2024-04-14T15:12:18.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.239+0000 7f472d014700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f47281ac120 con 0x7f472810e310 2024-04-14T15:12:18.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.239+0000 7f4726d9d700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f4728107890 0x7f472810da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:45792/0 (socket says 172.21.15.90:45792) 2024-04-14T15:12:18.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.239+0000 7f4726d9d700 1 -- 172.21.15.90:0/1990617137 learned_addr learned my addr 172.21.15.90:0/1990617137 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:12:18.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.239+0000 7f470ffff700 10 client.?.objecter ms_handle_connect 0x7f472810e310 2024-04-14T15:12:18.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.239+0000 7f470ffff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:12:18.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.239+0000 7f470ffff700 10 client.?.objecter ms_handle_connect 0x7f4728107890 2024-04-14T15:12:18.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.239+0000 7f470ffff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:12:18.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.239+0000 7f470ffff700 10 client.?.objecter ms_handle_connect 0x7f472810a740 2024-04-14T15:12:18.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.239+0000 7f470ffff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:12:18.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.239+0000 7f470ffff700 1 -- 172.21.15.90:0/1990617137 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3011442659 0 0) 0x7f47281ac120 con 0x7f472810e310 2024-04-14T15:12:18.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.239+0000 7f470ffff700 1 -- 172.21.15.90:0/1990617137 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f47100035b0 con 0x7f472810e310 2024-04-14T15:12:18.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.239+0000 7f470ffff700 1 -- 172.21.15.90:0/1990617137 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 509817115 0 0) 0x7f47281a9cc0 con 0x7f4728107890 2024-04-14T15:12:18.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.239+0000 7f470ffff700 1 -- 172.21.15.90:0/1990617137 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f47281ac120 con 0x7f4728107890 2024-04-14T15:12:18.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.239+0000 7f470ffff700 1 -- 172.21.15.90:0/1990617137 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1157550530 0 0) 0x7f47281aaee0 con 0x7f472810a740 2024-04-14T15:12:18.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.239+0000 7f470ffff700 1 -- 172.21.15.90:0/1990617137 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f47281a9cc0 con 0x7f472810a740 2024-04-14T15:12:18.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.239+0000 7f470ffff700 1 -- 172.21.15.90:0/1990617137 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2070038114 0 0) 0x7f47100035b0 con 0x7f472810e310 2024-04-14T15:12:18.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.240+0000 7f470ffff700 1 -- 172.21.15.90:0/1990617137 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f47281aaee0 con 0x7f472810e310 2024-04-14T15:12:18.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.240+0000 7f470ffff700 1 -- 172.21.15.90:0/1990617137 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f4720001bb0 con 0x7f472810e310 2024-04-14T15:12:18.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.240+0000 7f470ffff700 1 -- 172.21.15.90:0/1990617137 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2217533570 0 0) 0x7f47281a9cc0 con 0x7f472810a740 2024-04-14T15:12:18.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.240+0000 7f470ffff700 1 -- 172.21.15.90:0/1990617137 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f47100035b0 con 0x7f472810a740 2024-04-14T15:12:18.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.240+0000 7f470ffff700 1 -- 172.21.15.90:0/1990617137 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f471c004550 con 0x7f472810a740 2024-04-14T15:12:18.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.240+0000 7f470ffff700 1 -- 172.21.15.90:0/1990617137 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2780028821 0 0) 0x7f47281ac120 con 0x7f4728107890 2024-04-14T15:12:18.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.240+0000 7f470ffff700 1 -- 172.21.15.90:0/1990617137 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f47281a9cc0 con 0x7f4728107890 2024-04-14T15:12:18.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.240+0000 7f470ffff700 1 -- 172.21.15.90:0/1990617137 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f47180018b0 con 0x7f4728107890 2024-04-14T15:12:18.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.240+0000 7f470ffff700 1 -- 172.21.15.90:0/1990617137 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3947781145 0 0) 0x7f47100035b0 con 0x7f472810a740 2024-04-14T15:12:18.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.240+0000 7f470ffff700 1 -- 172.21.15.90:0/1990617137 >> v1:172.21.15.90:6790/0 conn(0x7f472810e310 legacy=0x7f47281a8380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:18.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.240+0000 7f470ffff700 1 -- 172.21.15.90:0/1990617137 >> v1:172.21.15.90:6789/0 conn(0x7f4728107890 legacy=0x7f472810da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:18.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.240+0000 7f470ffff700 1 -- 172.21.15.90:0/1990617137 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47281ad360 con 0x7f472810a740 2024-04-14T15:12:18.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.240+0000 7f472d014700 1 -- 172.21.15.90:0/1990617137 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f47281a9ef0 con 0x7f472810a740 2024-04-14T15:12:18.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.240+0000 7f472d014700 10 client.24739.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:12:18.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.240+0000 7f470ffff700 1 -- 172.21.15.90:0/1990617137 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f471c001080 con 0x7f472810a740 2024-04-14T15:12:18.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.240+0000 7f472d014700 1 -- 172.21.15.90:0/1990617137 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f47281aa3b0 con 0x7f472810a740 2024-04-14T15:12:18.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.240+0000 7f472d014700 1 librados: init done 2024-04-14T15:12:18.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.240+0000 7f470ffff700 1 -- 172.21.15.90:0/1990617137 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f471c004e10 con 0x7f472810a740 2024-04-14T15:12:18.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.241+0000 7f472d014700 1 -- 172.21.15.90:0/1990617137 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47280693d0 con 0x7f472810a740 2024-04-14T15:12:18.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.242+0000 7f470ffff700 1 -- 172.21.15.90:0/1990617137 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f471c003710 con 0x7f472810a740 2024-04-14T15:12:18.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.242+0000 7f470ffff700 1 -- 172.21.15.90:0/1990617137 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(100..100 src has 41..100) ==== 6320+0+0 (unknown 3374787364 0 0) 0x7f471c0890a0 con 0x7f472810a740 2024-04-14T15:12:18.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.242+0000 7f470ffff700 10 client.24739.objecter ms_dispatch 0x7f4728001550 osd_map(100..100 src has 41..100) 2024-04-14T15:12:18.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.242+0000 7f470ffff700 3 client.24739.objecter handle_osd_map got epochs [100,100] > 0 2024-04-14T15:12:18.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.242+0000 7f470ffff700 3 client.24739.objecter handle_osd_map decoding full epoch 100 2024-04-14T15:12:18.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.243+0000 7f470ffff700 20 client.24739.objecter dump_active .. 0 homeless 2024-04-14T15:12:18.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.243+0000 7f470ffff700 10 client.24739.objecter ms_handle_connect 0x7f4710071400 2024-04-14T15:12:18.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.247+0000 7f470ffff700 1 -- 172.21.15.90:0/1990617137 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f471c0561f0 con 0x7f472810a740 2024-04-14T15:12:18.310 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:12:18.338 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:18.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:18.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:12:18.440 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:12:18.466 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:18.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.466+0000 7f472d014700 1 -- 172.21.15.90:0/1990617137 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7f472810b030 con 0x7f472810a740 2024-04-14T15:12:18.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:18.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:12:18.569 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:12:18.595 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:18.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:18.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:12:18.697 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:12:18.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:18.723 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:18.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:12:18.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.792+0000 7f470ffff700 1 -- 172.21.15.90:0/1990617137 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v101) ==== 91+0+0 (unknown 1264733201 0 0) 0x7f471c059ea0 con 0x7f472810a740 2024-04-14T15:12:18.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.795+0000 7f472d014700 10 librados: watch_flush enter 2024-04-14T15:12:18.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.795+0000 7f472d014700 10 librados: watch_flush exit 2024-04-14T15:12:18.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.795+0000 7f472d014700 20 client.24739.objecter shutdown clearing up homeless session... 2024-04-14T15:12:18.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.795+0000 7f472d014700 10 client.24739.objecter successfully canceled tick 2024-04-14T15:12:18.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.795+0000 7f472d014700 1 -- 172.21.15.90:0/1990617137 >> v1:172.21.15.90:6800/420935427 conn(0x7f4710071400 legacy=0x7f47100738c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:18.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.795+0000 7f472d014700 1 -- 172.21.15.90:0/1990617137 >> v1:172.21.15.92:6789/0 conn(0x7f472810a740 legacy=0x7f47281a4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:18.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.795+0000 7f472d014700 1 -- 172.21.15.90:0/1990617137 shutdown_connections 2024-04-14T15:12:18.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.795+0000 7f472d014700 1 -- 172.21.15.90:0/1990617137 >> 172.21.15.90:0/1990617137 conn(0x7f47280fd290 msgr2=0x7f4728111810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:12:18.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.795+0000 7f472d014700 1 -- 172.21.15.90:0/1990617137 shutdown_connections 2024-04-14T15:12:18.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.795+0000 7f472d014700 1 -- 172.21.15.90:0/1990617137 wait complete. 2024-04-14T15:12:18.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:18.795+0000 7f472d014700 1 librados: shutdown 2024-04-14T15:12:18.797 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is unset 2024-04-14T15:12:18.823 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:12:18.850 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:18.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:18.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:12:18.952 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:12:18.980 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:18.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:18.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:12:19.082 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:12:19.106 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:19.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:19.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:12:19.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:18 smithi092 ceph-mon[33966]: pgmap v81: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:12:19.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:18 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1990617137' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:12:19.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:18 smithi092 ceph-mon[33966]: from='client.24739 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:12:19.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:18 smithi090 ceph-mon[30399]: pgmap v81: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:12:19.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:18 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1990617137' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:12:19.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:18 smithi090 ceph-mon[30399]: from='client.24739 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:12:19.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:18 smithi090 ceph-mon[41670]: pgmap v81: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:12:19.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:18 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1990617137' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:12:19.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:18 smithi090 ceph-mon[41670]: from='client.24739 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:12:19.208 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:12:19.231 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:19.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:19.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:12:19.334 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:12:19.361 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:19.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:19.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:12:19.463 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:12:19.487 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:19.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:19.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:12:19.589 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:12:19.613 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:19.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:19.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:12:19.715 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:12:19.743 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:19.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:19.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:12:19.844 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:12:19.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:19.868 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:19.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:12:19.970 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:12:19.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:19.998 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:19.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:12:20.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:19 smithi090 ceph-mon[41670]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:12:20.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:19 smithi090 ceph-mon[41670]: from='client.24739 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-14T15:12:20.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:19 smithi090 ceph-mon[41670]: osdmap e101: 8 total, 8 up, 8 in 2024-04-14T15:12:20.057 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:19 smithi090 ceph-mon[41670]: 5.c scrub starts 2024-04-14T15:12:20.057 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:19 smithi090 ceph-mon[41670]: 5.c scrub ok 2024-04-14T15:12:20.057 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:19 smithi090 ceph-mon[30399]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:12:20.057 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:19 smithi090 ceph-mon[30399]: from='client.24739 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-14T15:12:20.057 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:19 smithi090 ceph-mon[30399]: osdmap e101: 8 total, 8 up, 8 in 2024-04-14T15:12:20.057 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:19 smithi090 ceph-mon[30399]: 5.c scrub starts 2024-04-14T15:12:20.057 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:19 smithi090 ceph-mon[30399]: 5.c scrub ok 2024-04-14T15:12:20.100 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:12:20.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:19 smithi092 ceph-mon[33966]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:12:20.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:19 smithi092 ceph-mon[33966]: from='client.24739 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-14T15:12:20.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:19 smithi092 ceph-mon[33966]: osdmap e101: 8 total, 8 up, 8 in 2024-04-14T15:12:20.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:19 smithi092 ceph-mon[33966]: 5.c scrub starts 2024-04-14T15:12:20.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:19 smithi092 ceph-mon[33966]: 5.c scrub ok 2024-04-14T15:12:20.127 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:20.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:20.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:12:20.229 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:12:20.252 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:20.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:20.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:12:20.354 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:12:20.382 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:20.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:20.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:12:20.484 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:12:20.509 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:20.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:20.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:12:20.611 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:12:20.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:20.639 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:20.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:12:20.740 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:12:20.766 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:20.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:20.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:12:20.868 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:12:20.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:20.895 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:20.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:12:20.996 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:12:21.017 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:20 smithi090 ceph-mon[41670]: pgmap v83: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:12:21.017 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:20 smithi090 ceph-mon[41670]: 5.3 scrub starts 2024-04-14T15:12:21.017 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:20 smithi090 ceph-mon[41670]: 5.3 scrub ok 2024-04-14T15:12:21.018 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:20 smithi090 ceph-mon[30399]: pgmap v83: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:12:21.018 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:20 smithi090 ceph-mon[30399]: 5.3 scrub starts 2024-04-14T15:12:21.018 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:20 smithi090 ceph-mon[30399]: 5.3 scrub ok 2024-04-14T15:12:21.022 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:21.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:21.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:12:21.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.974+0000 7fbc32784700 1 -- 172.21.15.90:0/1135593781 >> v1:172.21.15.90:6789/0 conn(0x7fbc2c108890 legacy=0x7fbc2c108c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:21.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.974+0000 7fbc32784700 1 -- 172.21.15.90:0/1135593781 shutdown_connections 2024-04-14T15:12:21.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.974+0000 7fbc32784700 1 -- 172.21.15.90:0/1135593781 >> 172.21.15.90:0/1135593781 conn(0x7fbc2c0fe380 msgr2=0x7fbc2c1007a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:12:21.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.974+0000 7fbc32784700 1 -- 172.21.15.90:0/1135593781 shutdown_connections 2024-04-14T15:12:21.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.974+0000 7fbc32784700 1 -- 172.21.15.90:0/1135593781 wait complete. 2024-04-14T15:12:21.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.974+0000 7fbc32784700 1 librados: starting msgr at 2024-04-14T15:12:21.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.974+0000 7fbc32784700 1 librados: starting objecter 2024-04-14T15:12:21.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.975+0000 7fbc32784700 1 Processor -- start 2024-04-14T15:12:21.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.975+0000 7fbc32784700 1 -- start start 2024-04-14T15:12:21.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.975+0000 7fbc32784700 1 librados: setting wanted keys 2024-04-14T15:12:21.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.975+0000 7fbc32784700 1 librados: calling monclient init 2024-04-14T15:12:21.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.975+0000 7fbc32784700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbc2c10f000 con 0x7fbc2c10b740 2024-04-14T15:12:21.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.975+0000 7fbc32784700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbc2c1a8f20 con 0x7fbc2c10f310 2024-04-14T15:12:21.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.975+0000 7fbc32784700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbc2c1aa100 con 0x7fbc2c108890 2024-04-14T15:12:21.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.975+0000 7fbc2bfff700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7fbc2c108890 0x7fbc2c10d890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:46928/0 (socket says 172.21.15.90:46928) 2024-04-14T15:12:21.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.975+0000 7fbc2bfff700 1 -- 172.21.15.90:0/2291512326 learned_addr learned my addr 172.21.15.90:0/2291512326 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:12:21.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.975+0000 7fbc297fa700 10 client.?.objecter ms_handle_connect 0x7fbc2c10b740 2024-04-14T15:12:21.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.975+0000 7fbc297fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:12:21.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.975+0000 7fbc297fa700 10 client.?.objecter ms_handle_connect 0x7fbc2c108890 2024-04-14T15:12:21.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.975+0000 7fbc297fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:12:21.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.975+0000 7fbc297fa700 1 -- 172.21.15.90:0/2291512326 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1820431254 0 0) 0x7fbc2c10f000 con 0x7fbc2c10b740 2024-04-14T15:12:21.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.975+0000 7fbc297fa700 1 -- 172.21.15.90:0/2291512326 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc140035b0 con 0x7fbc2c10b740 2024-04-14T15:12:21.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.975+0000 7fbc297fa700 1 -- 172.21.15.90:0/2291512326 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1507271952 0 0) 0x7fbc2c1aa100 con 0x7fbc2c108890 2024-04-14T15:12:21.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.975+0000 7fbc297fa700 1 -- 172.21.15.90:0/2291512326 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc2c10f000 con 0x7fbc2c108890 2024-04-14T15:12:21.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.975+0000 7fbc297fa700 10 client.?.objecter ms_handle_connect 0x7fbc2c10f310 2024-04-14T15:12:21.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.975+0000 7fbc297fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:12:21.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.976+0000 7fbc297fa700 1 -- 172.21.15.90:0/2291512326 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 771425640 0 0) 0x7fbc2c1a8f20 con 0x7fbc2c10f310 2024-04-14T15:12:21.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.976+0000 7fbc297fa700 1 -- 172.21.15.90:0/2291512326 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc2c1aa100 con 0x7fbc2c10f310 2024-04-14T15:12:21.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.976+0000 7fbc297fa700 1 -- 172.21.15.90:0/2291512326 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3060678815 0 0) 0x7fbc2c10f000 con 0x7fbc2c108890 2024-04-14T15:12:21.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.976+0000 7fbc297fa700 1 -- 172.21.15.90:0/2291512326 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbc2c1a8f20 con 0x7fbc2c108890 2024-04-14T15:12:21.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.976+0000 7fbc297fa700 1 -- 172.21.15.90:0/2291512326 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4236981417 0 0) 0x7fbc140035b0 con 0x7fbc2c10b740 2024-04-14T15:12:21.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.976+0000 7fbc297fa700 1 -- 172.21.15.90:0/2291512326 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbc2c10f000 con 0x7fbc2c10b740 2024-04-14T15:12:21.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.976+0000 7fbc297fa700 1 -- 172.21.15.90:0/2291512326 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fbc1c001430 con 0x7fbc2c108890 2024-04-14T15:12:21.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.976+0000 7fbc297fa700 1 -- 172.21.15.90:0/2291512326 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fbc200018d0 con 0x7fbc2c10b740 2024-04-14T15:12:21.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.976+0000 7fbc297fa700 1 -- 172.21.15.90:0/2291512326 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 33767074 0 0) 0x7fbc2c1aa100 con 0x7fbc2c10f310 2024-04-14T15:12:21.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.976+0000 7fbc297fa700 1 -- 172.21.15.90:0/2291512326 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbc140035b0 con 0x7fbc2c10f310 2024-04-14T15:12:21.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.976+0000 7fbc297fa700 1 -- 172.21.15.90:0/2291512326 <== mon.2 v1:172.21.15.90:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 888263910 0 0) 0x7fbc2c1a8f20 con 0x7fbc2c108890 2024-04-14T15:12:21.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.976+0000 7fbc297fa700 1 -- 172.21.15.90:0/2291512326 >> v1:172.21.15.92:6789/0 conn(0x7fbc2c10f310 legacy=0x7fbc2c1a67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:21.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.976+0000 7fbc297fa700 1 -- 172.21.15.90:0/2291512326 >> v1:172.21.15.90:6789/0 conn(0x7fbc2c10b740 legacy=0x7fbc2c10dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:21.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.976+0000 7fbc297fa700 1 -- 172.21.15.90:0/2291512326 --> v1:172.21.15.90:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc2c1ab2e0 con 0x7fbc2c108890 2024-04-14T15:12:21.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.976+0000 7fbc32784700 1 -- 172.21.15.90:0/2291512326 --> v1:172.21.15.90:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbc2c1aa330 con 0x7fbc2c108890 2024-04-14T15:12:21.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.976+0000 7fbc32784700 10 client.24722.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:12:21.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.976+0000 7fbc32784700 1 -- 172.21.15.90:0/2291512326 --> v1:172.21.15.90:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fbc2c1aa7f0 con 0x7fbc2c108890 2024-04-14T15:12:21.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.976+0000 7fbc32784700 1 librados: init done 2024-04-14T15:12:21.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.976+0000 7fbc297fa700 1 -- 172.21.15.90:0/2291512326 <== mon.2 v1:172.21.15.90:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fbc1c001b30 con 0x7fbc2c108890 2024-04-14T15:12:21.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.976+0000 7fbc297fa700 1 -- 172.21.15.90:0/2291512326 <== mon.2 v1:172.21.15.90:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fbc1c004e20 con 0x7fbc2c108890 2024-04-14T15:12:21.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.977+0000 7fbc32784700 1 -- 172.21.15.90:0/2291512326 --> v1:172.21.15.90:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fbc18005100 con 0x7fbc2c108890 2024-04-14T15:12:21.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.978+0000 7fbc297fa700 1 -- 172.21.15.90:0/2291512326 <== mon.2 v1:172.21.15.90:6790/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fbc1c003900 con 0x7fbc2c108890 2024-04-14T15:12:21.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.978+0000 7fbc297fa700 1 -- 172.21.15.90:0/2291512326 <== mon.2 v1:172.21.15.90:6790/0 8 ==== osd_map(101..101 src has 51..101) ==== 6320+0+0 (unknown 1370162482 0 0) 0x7fbc1c08a2f0 con 0x7fbc2c108890 2024-04-14T15:12:21.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.978+0000 7fbc297fa700 10 client.24722.objecter ms_dispatch 0x7fbc2c001550 osd_map(101..101 src has 51..101) 2024-04-14T15:12:21.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.978+0000 7fbc297fa700 3 client.24722.objecter handle_osd_map got epochs [101,101] > 0 2024-04-14T15:12:21.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.978+0000 7fbc297fa700 3 client.24722.objecter handle_osd_map decoding full epoch 101 2024-04-14T15:12:21.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.979+0000 7fbc297fa700 20 client.24722.objecter dump_active .. 0 homeless 2024-04-14T15:12:21.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.979+0000 7fbc297fa700 1 -- 172.21.15.90:0/2291512326 <== mon.2 v1:172.21.15.90:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v101) ==== 54+0+16 (unknown 2957627552 0 2420000192) 0x7fbc1c08a750 con 0x7fbc2c108890 2024-04-14T15:12:21.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.979+0000 7fbc297fa700 10 client.24722.objecter ms_handle_connect 0x7fbc14071330 2024-04-14T15:12:21.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.979+0000 7fbc32784700 10 client.24722.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:12:21.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.979+0000 7fbc32784700 15 client.24722.objecter _session_command_op_assign -1 1 2024-04-14T15:12:21.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.979+0000 7fbc32784700 20 client.24722.objecter _get_session s=0x7fbc180053e0 osd=0 3 2024-04-14T15:12:21.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.979+0000 7fbc32784700 20 client.24722.objecter put_session s=0x7fbc180053e0 osd=0 3 2024-04-14T15:12:21.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.979+0000 7fbc32784700 20 client.24722.objecter _get_session s=0x7fbc180053e0 osd=0 3 2024-04-14T15:12:21.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.979+0000 7fbc32784700 15 client.24722.objecter _session_command_op_remove -1 1 2024-04-14T15:12:21.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.979+0000 7fbc32784700 20 client.24722.objecter get_session s=0x7fbc180053e0 osd=0 3 2024-04-14T15:12:21.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.979+0000 7fbc32784700 15 client.24722.objecter _session_command_op_assign 0 1 2024-04-14T15:12:21.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.979+0000 7fbc32784700 20 client.24722.objecter put_session s=0x7fbc180053e0 osd=0 4 2024-04-14T15:12:21.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.979+0000 7fbc32784700 10 client.24722.objecter _send_command 1 2024-04-14T15:12:21.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.979+0000 7fbc32784700 1 -- 172.21.15.90:0/2291512326 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbc18004b50 con 0x7fbc18004730 2024-04-14T15:12:21.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.979+0000 7fbc297fa700 10 client.24722.objecter ms_handle_connect 0x7fbc18004730 2024-04-14T15:12:21.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.980+0000 7fbc297fa700 1 -- 172.21.15.90:0/2291512326 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fbc18004b50 con 0x7fbc18004730 2024-04-14T15:12:21.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.980+0000 7fbc297fa700 10 client.24722.objecter ms_dispatch 0x7fbc2c001550 command_reply(tid 1: 0 ) 2024-04-14T15:12:21.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.980+0000 7fbc297fa700 10 client.24722.objecter _finish_command 1 = system:0 2024-04-14T15:12:21.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.980+0000 7fbc297fa700 20 client.24722.objecter put_session s=0x7fbc180053e0 osd=0 4 2024-04-14T15:12:21.097 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:12:21.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.980+0000 7fbc297fa700 15 client.24722.objecter _session_command_op_remove 0 1 2024-04-14T15:12:21.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.997+0000 7fbc32784700 10 client.24722.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:12:21.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.997+0000 7fbc32784700 15 client.24722.objecter _session_command_op_assign -1 2 2024-04-14T15:12:21.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.997+0000 7fbc32784700 20 client.24722.objecter _get_session s=0x7fbc180053e0 osd=0 3 2024-04-14T15:12:21.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.997+0000 7fbc32784700 20 client.24722.objecter put_session s=0x7fbc180053e0 osd=0 3 2024-04-14T15:12:21.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.997+0000 7fbc32784700 20 client.24722.objecter _get_session s=0x7fbc180053e0 osd=0 3 2024-04-14T15:12:21.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.997+0000 7fbc32784700 15 client.24722.objecter _session_command_op_remove -1 2 2024-04-14T15:12:21.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.997+0000 7fbc32784700 20 client.24722.objecter get_session s=0x7fbc180053e0 osd=0 3 2024-04-14T15:12:21.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.997+0000 7fbc32784700 15 client.24722.objecter _session_command_op_assign 0 2 2024-04-14T15:12:21.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.997+0000 7fbc32784700 20 client.24722.objecter put_session s=0x7fbc180053e0 osd=0 4 2024-04-14T15:12:21.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.997+0000 7fbc32784700 10 client.24722.objecter _send_command 2 2024-04-14T15:12:21.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.997+0000 7fbc32784700 1 -- 172.21.15.90:0/2291512326 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fbc180079f0 con 0x7fbc18004730 2024-04-14T15:12:21.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.998+0000 7fbc297fa700 1 -- 172.21.15.90:0/2291512326 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fbc180079f0 con 0x7fbc18004730 2024-04-14T15:12:21.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.998+0000 7fbc297fa700 10 client.24722.objecter ms_dispatch 0x7fbc2c001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:12:21.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.998+0000 7fbc297fa700 10 client.24722.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:12:21.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.998+0000 7fbc297fa700 20 client.24722.objecter put_session s=0x7fbc180053e0 osd=0 4 2024-04-14T15:12:21.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.998+0000 7fbc297fa700 15 client.24722.objecter _session_command_op_remove 0 2 2024-04-14T15:12:21.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.999+0000 7fbc32784700 10 client.24722.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:12:21.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.999+0000 7fbc32784700 15 client.24722.objecter _session_command_op_assign -1 3 2024-04-14T15:12:21.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.999+0000 7fbc32784700 20 client.24722.objecter _get_session s=0x7fbc18004110 osd=1 3 2024-04-14T15:12:21.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.999+0000 7fbc32784700 20 client.24722.objecter put_session s=0x7fbc18004110 osd=1 3 2024-04-14T15:12:21.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.999+0000 7fbc32784700 20 client.24722.objecter _get_session s=0x7fbc18004110 osd=1 3 2024-04-14T15:12:21.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.999+0000 7fbc32784700 15 client.24722.objecter _session_command_op_remove -1 3 2024-04-14T15:12:21.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.999+0000 7fbc32784700 20 client.24722.objecter get_session s=0x7fbc18004110 osd=1 3 2024-04-14T15:12:21.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.999+0000 7fbc32784700 15 client.24722.objecter _session_command_op_assign 1 3 2024-04-14T15:12:21.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.999+0000 7fbc32784700 20 client.24722.objecter put_session s=0x7fbc18004110 osd=1 4 2024-04-14T15:12:21.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.999+0000 7fbc32784700 10 client.24722.objecter _send_command 3 2024-04-14T15:12:21.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.999+0000 7fbc32784700 1 -- 172.21.15.90:0/2291512326 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fbc180020b0 con 0x7fbc18004280 2024-04-14T15:12:21.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:20.999+0000 7fbc297fa700 10 client.24722.objecter ms_handle_connect 0x7fbc18004280 2024-04-14T15:12:21.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.000+0000 7fbc297fa700 1 -- 172.21.15.90:0/2291512326 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fbc180020b0 con 0x7fbc18004280 2024-04-14T15:12:21.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.000+0000 7fbc297fa700 10 client.24722.objecter ms_dispatch 0x7fbc2c001550 command_reply(tid 3: 0 ) 2024-04-14T15:12:21.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.000+0000 7fbc297fa700 10 client.24722.objecter _finish_command 3 = system:0 2024-04-14T15:12:21.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.000+0000 7fbc297fa700 20 client.24722.objecter put_session s=0x7fbc18004110 osd=1 4 2024-04-14T15:12:21.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.000+0000 7fbc297fa700 15 client.24722.objecter _session_command_op_remove 1 3 2024-04-14T15:12:21.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.017+0000 7fbc32784700 10 client.24722.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:12:21.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.017+0000 7fbc32784700 15 client.24722.objecter _session_command_op_assign -1 4 2024-04-14T15:12:21.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.017+0000 7fbc32784700 20 client.24722.objecter _get_session s=0x7fbc18004110 osd=1 3 2024-04-14T15:12:21.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.017+0000 7fbc32784700 20 client.24722.objecter put_session s=0x7fbc18004110 osd=1 3 2024-04-14T15:12:21.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.017+0000 7fbc32784700 20 client.24722.objecter _get_session s=0x7fbc18004110 osd=1 3 2024-04-14T15:12:21.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.017+0000 7fbc32784700 15 client.24722.objecter _session_command_op_remove -1 4 2024-04-14T15:12:21.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.017+0000 7fbc32784700 20 client.24722.objecter get_session s=0x7fbc18004110 osd=1 3 2024-04-14T15:12:21.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.017+0000 7fbc32784700 15 client.24722.objecter _session_command_op_assign 1 4 2024-04-14T15:12:21.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.017+0000 7fbc32784700 20 client.24722.objecter put_session s=0x7fbc18004110 osd=1 4 2024-04-14T15:12:21.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.017+0000 7fbc32784700 10 client.24722.objecter _send_command 4 2024-04-14T15:12:21.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.017+0000 7fbc32784700 1 -- 172.21.15.90:0/2291512326 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fbc18001600 con 0x7fbc18004280 2024-04-14T15:12:21.101 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:12:21.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.018+0000 7fbc297fa700 1 -- 172.21.15.90:0/2291512326 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fbc18001600 con 0x7fbc18004280 2024-04-14T15:12:21.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.018+0000 7fbc297fa700 10 client.24722.objecter ms_dispatch 0x7fbc2c001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:12:21.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.018+0000 7fbc297fa700 10 client.24722.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:12:21.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.018+0000 7fbc297fa700 20 client.24722.objecter put_session s=0x7fbc18004110 osd=1 4 2024-04-14T15:12:21.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.018+0000 7fbc297fa700 15 client.24722.objecter _session_command_op_remove 1 4 2024-04-14T15:12:21.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.019+0000 7fbc32784700 10 client.24722.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:12:21.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.019+0000 7fbc32784700 15 client.24722.objecter _session_command_op_assign -1 5 2024-04-14T15:12:21.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.019+0000 7fbc32784700 20 client.24722.objecter _get_session s=0x7fbc18001a00 osd=2 3 2024-04-14T15:12:21.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.019+0000 7fbc32784700 20 client.24722.objecter put_session s=0x7fbc18001a00 osd=2 3 2024-04-14T15:12:21.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.019+0000 7fbc32784700 20 client.24722.objecter _get_session s=0x7fbc18001a00 osd=2 3 2024-04-14T15:12:21.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.019+0000 7fbc32784700 15 client.24722.objecter _session_command_op_remove -1 5 2024-04-14T15:12:21.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.019+0000 7fbc32784700 20 client.24722.objecter get_session s=0x7fbc18001a00 osd=2 3 2024-04-14T15:12:21.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.019+0000 7fbc32784700 15 client.24722.objecter _session_command_op_assign 2 5 2024-04-14T15:12:21.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.019+0000 7fbc32784700 20 client.24722.objecter put_session s=0x7fbc18001a00 osd=2 4 2024-04-14T15:12:21.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.019+0000 7fbc32784700 10 client.24722.objecter _send_command 5 2024-04-14T15:12:21.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.019+0000 7fbc32784700 1 -- 172.21.15.90:0/2291512326 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fbc18015350 con 0x7fbc18008880 2024-04-14T15:12:21.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.019+0000 7fbc297fa700 10 client.24722.objecter ms_handle_connect 0x7fbc18008880 2024-04-14T15:12:21.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.020+0000 7fbc297fa700 1 -- 172.21.15.90:0/2291512326 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fbc18015350 con 0x7fbc18008880 2024-04-14T15:12:21.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.020+0000 7fbc297fa700 10 client.24722.objecter ms_dispatch 0x7fbc2c001550 command_reply(tid 5: 0 ) 2024-04-14T15:12:21.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.020+0000 7fbc297fa700 10 client.24722.objecter _finish_command 5 = system:0 2024-04-14T15:12:21.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.020+0000 7fbc297fa700 20 client.24722.objecter put_session s=0x7fbc18001a00 osd=2 4 2024-04-14T15:12:21.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.020+0000 7fbc297fa700 15 client.24722.objecter _session_command_op_remove 2 5 2024-04-14T15:12:21.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.037+0000 7fbc32784700 10 client.24722.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:12:21.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.037+0000 7fbc32784700 15 client.24722.objecter _session_command_op_assign -1 6 2024-04-14T15:12:21.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.037+0000 7fbc32784700 20 client.24722.objecter _get_session s=0x7fbc18001a00 osd=2 3 2024-04-14T15:12:21.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.037+0000 7fbc32784700 20 client.24722.objecter put_session s=0x7fbc18001a00 osd=2 3 2024-04-14T15:12:21.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.037+0000 7fbc32784700 20 client.24722.objecter _get_session s=0x7fbc18001a00 osd=2 3 2024-04-14T15:12:21.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.037+0000 7fbc32784700 15 client.24722.objecter _session_command_op_remove -1 6 2024-04-14T15:12:21.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.037+0000 7fbc32784700 20 client.24722.objecter get_session s=0x7fbc18001a00 osd=2 3 2024-04-14T15:12:21.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.037+0000 7fbc32784700 15 client.24722.objecter _session_command_op_assign 2 6 2024-04-14T15:12:21.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.037+0000 7fbc32784700 20 client.24722.objecter put_session s=0x7fbc18001a00 osd=2 4 2024-04-14T15:12:21.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.037+0000 7fbc32784700 10 client.24722.objecter _send_command 6 2024-04-14T15:12:21.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.037+0000 7fbc32784700 1 -- 172.21.15.90:0/2291512326 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fbc18007f10 con 0x7fbc18008880 2024-04-14T15:12:21.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.039+0000 7fbc297fa700 1 -- 172.21.15.90:0/2291512326 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fbc18007f10 con 0x7fbc18008880 2024-04-14T15:12:21.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.039+0000 7fbc297fa700 10 client.24722.objecter ms_dispatch 0x7fbc2c001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:12:21.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.039+0000 7fbc297fa700 10 client.24722.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:12:21.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.039+0000 7fbc297fa700 20 client.24722.objecter put_session s=0x7fbc18001a00 osd=2 4 2024-04-14T15:12:21.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.039+0000 7fbc297fa700 15 client.24722.objecter _session_command_op_remove 2 6 2024-04-14T15:12:21.104 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:12:21.104 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:12:21.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.039+0000 7fbc32784700 10 client.24722.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:12:21.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.039+0000 7fbc32784700 15 client.24722.objecter _session_command_op_assign -1 7 2024-04-14T15:12:21.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.039+0000 7fbc32784700 20 client.24722.objecter _get_session s=0x7fbc180081a0 osd=3 3 2024-04-14T15:12:21.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.039+0000 7fbc32784700 20 client.24722.objecter put_session s=0x7fbc180081a0 osd=3 3 2024-04-14T15:12:21.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.039+0000 7fbc32784700 20 client.24722.objecter _get_session s=0x7fbc180081a0 osd=3 3 2024-04-14T15:12:21.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.039+0000 7fbc32784700 15 client.24722.objecter _session_command_op_remove -1 7 2024-04-14T15:12:21.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.039+0000 7fbc32784700 20 client.24722.objecter get_session s=0x7fbc180081a0 osd=3 3 2024-04-14T15:12:21.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.039+0000 7fbc32784700 15 client.24722.objecter _session_command_op_assign 3 7 2024-04-14T15:12:21.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.039+0000 7fbc32784700 20 client.24722.objecter put_session s=0x7fbc180081a0 osd=3 4 2024-04-14T15:12:21.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.039+0000 7fbc32784700 10 client.24722.objecter _send_command 7 2024-04-14T15:12:21.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.039+0000 7fbc32784700 1 -- 172.21.15.90:0/2291512326 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fbc180160a0 con 0x7fbc18015530 2024-04-14T15:12:21.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.039+0000 7fbc297fa700 10 client.24722.objecter ms_handle_connect 0x7fbc18015530 2024-04-14T15:12:21.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.040+0000 7fbc297fa700 1 -- 172.21.15.90:0/2291512326 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fbc180160a0 con 0x7fbc18015530 2024-04-14T15:12:21.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.040+0000 7fbc297fa700 10 client.24722.objecter ms_dispatch 0x7fbc2c001550 command_reply(tid 7: 0 ) 2024-04-14T15:12:21.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.040+0000 7fbc297fa700 10 client.24722.objecter _finish_command 7 = system:0 2024-04-14T15:12:21.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.040+0000 7fbc297fa700 20 client.24722.objecter put_session s=0x7fbc180081a0 osd=3 4 2024-04-14T15:12:21.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.040+0000 7fbc297fa700 15 client.24722.objecter _session_command_op_remove 3 7 2024-04-14T15:12:21.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.059+0000 7fbc32784700 10 client.24722.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:12:21.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.059+0000 7fbc32784700 15 client.24722.objecter _session_command_op_assign -1 8 2024-04-14T15:12:21.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.059+0000 7fbc32784700 20 client.24722.objecter _get_session s=0x7fbc180081a0 osd=3 3 2024-04-14T15:12:21.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.059+0000 7fbc32784700 20 client.24722.objecter put_session s=0x7fbc180081a0 osd=3 3 2024-04-14T15:12:21.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.059+0000 7fbc32784700 20 client.24722.objecter _get_session s=0x7fbc180081a0 osd=3 3 2024-04-14T15:12:21.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.059+0000 7fbc32784700 15 client.24722.objecter _session_command_op_remove -1 8 2024-04-14T15:12:21.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.059+0000 7fbc32784700 20 client.24722.objecter get_session s=0x7fbc180081a0 osd=3 3 2024-04-14T15:12:21.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.059+0000 7fbc32784700 15 client.24722.objecter _session_command_op_assign 3 8 2024-04-14T15:12:21.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.059+0000 7fbc32784700 20 client.24722.objecter put_session s=0x7fbc180081a0 osd=3 4 2024-04-14T15:12:21.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.059+0000 7fbc32784700 10 client.24722.objecter _send_command 8 2024-04-14T15:12:21.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.059+0000 7fbc32784700 1 -- 172.21.15.90:0/2291512326 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fbc18016a50 con 0x7fbc18015530 2024-04-14T15:12:21.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.060+0000 7fbc297fa700 1 -- 172.21.15.90:0/2291512326 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fbc18016a50 con 0x7fbc18015530 2024-04-14T15:12:21.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.060+0000 7fbc297fa700 10 client.24722.objecter ms_dispatch 0x7fbc2c001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:12:21.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.060+0000 7fbc297fa700 10 client.24722.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:12:21.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.061+0000 7fbc297fa700 20 client.24722.objecter put_session s=0x7fbc180081a0 osd=3 4 2024-04-14T15:12:21.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.061+0000 7fbc297fa700 15 client.24722.objecter _session_command_op_remove 3 8 2024-04-14T15:12:21.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.061+0000 7fbc32784700 10 client.24722.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:12:21.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.061+0000 7fbc32784700 15 client.24722.objecter _session_command_op_assign -1 9 2024-04-14T15:12:21.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.061+0000 7fbc32784700 20 client.24722.objecter _get_session s=0x7fbc18017bd0 osd=4 3 2024-04-14T15:12:21.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.061+0000 7fbc32784700 20 client.24722.objecter put_session s=0x7fbc18017bd0 osd=4 3 2024-04-14T15:12:21.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.061+0000 7fbc32784700 20 client.24722.objecter _get_session s=0x7fbc18017bd0 osd=4 3 2024-04-14T15:12:21.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.061+0000 7fbc32784700 15 client.24722.objecter _session_command_op_remove -1 9 2024-04-14T15:12:21.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.061+0000 7fbc32784700 20 client.24722.objecter get_session s=0x7fbc18017bd0 osd=4 3 2024-04-14T15:12:21.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.061+0000 7fbc32784700 15 client.24722.objecter _session_command_op_assign 4 9 2024-04-14T15:12:21.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.061+0000 7fbc32784700 20 client.24722.objecter put_session s=0x7fbc18017bd0 osd=4 4 2024-04-14T15:12:21.108 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:12:21.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.061+0000 7fbc32784700 10 client.24722.objecter _send_command 9 2024-04-14T15:12:21.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.061+0000 7fbc32784700 1 -- 172.21.15.90:0/2291512326 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fbc1801eb50 con 0x7fbc1801af10 2024-04-14T15:12:21.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.061+0000 7fbc297fa700 10 client.24722.objecter ms_handle_connect 0x7fbc1801af10 2024-04-14T15:12:21.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.062+0000 7fbc297fa700 1 -- 172.21.15.90:0/2291512326 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fbc1801eb50 con 0x7fbc1801af10 2024-04-14T15:12:21.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.062+0000 7fbc297fa700 10 client.24722.objecter ms_dispatch 0x7fbc2c001550 command_reply(tid 9: 0 ) 2024-04-14T15:12:21.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.062+0000 7fbc297fa700 10 client.24722.objecter _finish_command 9 = system:0 2024-04-14T15:12:21.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.062+0000 7fbc297fa700 20 client.24722.objecter put_session s=0x7fbc18017bd0 osd=4 4 2024-04-14T15:12:21.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.062+0000 7fbc297fa700 15 client.24722.objecter _session_command_op_remove 4 9 2024-04-14T15:12:21.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.081+0000 7fbc32784700 10 client.24722.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:12:21.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.081+0000 7fbc32784700 15 client.24722.objecter _session_command_op_assign -1 10 2024-04-14T15:12:21.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.081+0000 7fbc32784700 20 client.24722.objecter _get_session s=0x7fbc18017bd0 osd=4 3 2024-04-14T15:12:21.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.081+0000 7fbc32784700 20 client.24722.objecter put_session s=0x7fbc18017bd0 osd=4 3 2024-04-14T15:12:21.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.081+0000 7fbc32784700 20 client.24722.objecter _get_session s=0x7fbc18017bd0 osd=4 3 2024-04-14T15:12:21.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.081+0000 7fbc32784700 15 client.24722.objecter _session_command_op_remove -1 10 2024-04-14T15:12:21.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.081+0000 7fbc32784700 20 client.24722.objecter get_session s=0x7fbc18017bd0 osd=4 3 2024-04-14T15:12:21.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.081+0000 7fbc32784700 15 client.24722.objecter _session_command_op_assign 4 10 2024-04-14T15:12:21.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.081+0000 7fbc32784700 20 client.24722.objecter put_session s=0x7fbc18017bd0 osd=4 4 2024-04-14T15:12:21.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.081+0000 7fbc32784700 10 client.24722.objecter _send_command 10 2024-04-14T15:12:21.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.081+0000 7fbc32784700 1 -- 172.21.15.90:0/2291512326 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fbc1801eb10 con 0x7fbc1801af10 2024-04-14T15:12:21.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.083+0000 7fbc297fa700 1 -- 172.21.15.90:0/2291512326 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fbc1801eb10 con 0x7fbc1801af10 2024-04-14T15:12:21.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.083+0000 7fbc297fa700 10 client.24722.objecter ms_dispatch 0x7fbc2c001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:12:21.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.083+0000 7fbc297fa700 10 client.24722.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:12:21.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.083+0000 7fbc297fa700 20 client.24722.objecter put_session s=0x7fbc18017bd0 osd=4 4 2024-04-14T15:12:21.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.083+0000 7fbc297fa700 15 client.24722.objecter _session_command_op_remove 4 10 2024-04-14T15:12:21.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.083+0000 7fbc32784700 10 client.24722.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:12:21.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.083+0000 7fbc32784700 15 client.24722.objecter _session_command_op_assign -1 11 2024-04-14T15:12:21.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.083+0000 7fbc32784700 20 client.24722.objecter _get_session s=0x7fbc18010c50 osd=5 3 2024-04-14T15:12:21.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.083+0000 7fbc32784700 20 client.24722.objecter put_session s=0x7fbc18010c50 osd=5 3 2024-04-14T15:12:21.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.083+0000 7fbc32784700 20 client.24722.objecter _get_session s=0x7fbc18010c50 osd=5 3 2024-04-14T15:12:21.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.083+0000 7fbc32784700 15 client.24722.objecter _session_command_op_remove -1 11 2024-04-14T15:12:21.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.083+0000 7fbc32784700 20 client.24722.objecter get_session s=0x7fbc18010c50 osd=5 3 2024-04-14T15:12:21.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.083+0000 7fbc32784700 15 client.24722.objecter _session_command_op_assign 5 11 2024-04-14T15:12:21.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.083+0000 7fbc32784700 20 client.24722.objecter put_session s=0x7fbc18010c50 osd=5 4 2024-04-14T15:12:21.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.083+0000 7fbc32784700 10 client.24722.objecter _send_command 11 2024-04-14T15:12:21.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.083+0000 7fbc32784700 1 -- 172.21.15.90:0/2291512326 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fbc180202c0 con 0x7fbc1801f740 2024-04-14T15:12:21.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.083+0000 7fbc297fa700 10 client.24722.objecter ms_handle_connect 0x7fbc1801f740 2024-04-14T15:12:21.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.084+0000 7fbc297fa700 1 -- 172.21.15.90:0/2291512326 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fbc180202c0 con 0x7fbc1801f740 2024-04-14T15:12:21.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.084+0000 7fbc297fa700 10 client.24722.objecter ms_dispatch 0x7fbc2c001550 command_reply(tid 11: 0 ) 2024-04-14T15:12:21.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.084+0000 7fbc297fa700 10 client.24722.objecter _finish_command 11 = system:0 2024-04-14T15:12:21.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.084+0000 7fbc297fa700 20 client.24722.objecter put_session s=0x7fbc18010c50 osd=5 4 2024-04-14T15:12:21.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.084+0000 7fbc297fa700 15 client.24722.objecter _session_command_op_remove 5 11 2024-04-14T15:12:21.111 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:12:21.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.104+0000 7fbc32784700 10 client.24722.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:12:21.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.104+0000 7fbc32784700 15 client.24722.objecter _session_command_op_assign -1 12 2024-04-14T15:12:21.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.104+0000 7fbc32784700 20 client.24722.objecter _get_session s=0x7fbc18010c50 osd=5 3 2024-04-14T15:12:21.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.104+0000 7fbc32784700 20 client.24722.objecter put_session s=0x7fbc18010c50 osd=5 3 2024-04-14T15:12:21.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.104+0000 7fbc32784700 20 client.24722.objecter _get_session s=0x7fbc18010c50 osd=5 3 2024-04-14T15:12:21.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.104+0000 7fbc32784700 15 client.24722.objecter _session_command_op_remove -1 12 2024-04-14T15:12:21.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.104+0000 7fbc32784700 20 client.24722.objecter get_session s=0x7fbc18010c50 osd=5 3 2024-04-14T15:12:21.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.104+0000 7fbc32784700 15 client.24722.objecter _session_command_op_assign 5 12 2024-04-14T15:12:21.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.104+0000 7fbc32784700 20 client.24722.objecter put_session s=0x7fbc18010c50 osd=5 4 2024-04-14T15:12:21.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.104+0000 7fbc32784700 10 client.24722.objecter _send_command 12 2024-04-14T15:12:21.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.104+0000 7fbc32784700 1 -- 172.21.15.90:0/2291512326 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fbc1801f4f0 con 0x7fbc1801f740 2024-04-14T15:12:21.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.105+0000 7fbc297fa700 1 -- 172.21.15.90:0/2291512326 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fbc1801f4f0 con 0x7fbc1801f740 2024-04-14T15:12:21.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.105+0000 7fbc297fa700 10 client.24722.objecter ms_dispatch 0x7fbc2c001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:12:21.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.106+0000 7fbc297fa700 10 client.24722.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:12:21.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.106+0000 7fbc297fa700 20 client.24722.objecter put_session s=0x7fbc18010c50 osd=5 4 2024-04-14T15:12:21.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.106+0000 7fbc297fa700 15 client.24722.objecter _session_command_op_remove 5 12 2024-04-14T15:12:21.113 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:12:21.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.106+0000 7fbc32784700 10 client.24722.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:12:21.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.106+0000 7fbc32784700 15 client.24722.objecter _session_command_op_assign -1 13 2024-04-14T15:12:21.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.106+0000 7fbc32784700 20 client.24722.objecter _get_session s=0x7fbc18020300 osd=6 3 2024-04-14T15:12:21.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.106+0000 7fbc32784700 20 client.24722.objecter put_session s=0x7fbc18020300 osd=6 3 2024-04-14T15:12:21.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.106+0000 7fbc32784700 20 client.24722.objecter _get_session s=0x7fbc18020300 osd=6 3 2024-04-14T15:12:21.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.106+0000 7fbc32784700 15 client.24722.objecter _session_command_op_remove -1 13 2024-04-14T15:12:21.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.106+0000 7fbc32784700 20 client.24722.objecter get_session s=0x7fbc18020300 osd=6 3 2024-04-14T15:12:21.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.106+0000 7fbc32784700 15 client.24722.objecter _session_command_op_assign 6 13 2024-04-14T15:12:21.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.106+0000 7fbc32784700 20 client.24722.objecter put_session s=0x7fbc18020300 osd=6 4 2024-04-14T15:12:21.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.106+0000 7fbc32784700 10 client.24722.objecter _send_command 13 2024-04-14T15:12:21.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.106+0000 7fbc32784700 1 -- 172.21.15.90:0/2291512326 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fbc18027540 con 0x7fbc18023a30 2024-04-14T15:12:21.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.106+0000 7fbc297fa700 10 client.24722.objecter ms_handle_connect 0x7fbc18023a30 2024-04-14T15:12:21.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.107+0000 7fbc297fa700 1 -- 172.21.15.90:0/2291512326 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fbc18027540 con 0x7fbc18023a30 2024-04-14T15:12:21.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.107+0000 7fbc297fa700 10 client.24722.objecter ms_dispatch 0x7fbc2c001550 command_reply(tid 13: 0 ) 2024-04-14T15:12:21.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.107+0000 7fbc297fa700 10 client.24722.objecter _finish_command 13 = system:0 2024-04-14T15:12:21.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.107+0000 7fbc297fa700 20 client.24722.objecter put_session s=0x7fbc18020300 osd=6 4 2024-04-14T15:12:21.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.107+0000 7fbc297fa700 15 client.24722.objecter _session_command_op_remove 6 13 2024-04-14T15:12:21.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:20 smithi092 ceph-mon[33966]: pgmap v83: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:12:21.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:20 smithi092 ceph-mon[33966]: 5.3 scrub starts 2024-04-14T15:12:21.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:20 smithi092 ceph-mon[33966]: 5.3 scrub ok 2024-04-14T15:12:21.124 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:12:21.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.129+0000 7fbc32784700 10 client.24722.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:12:21.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.129+0000 7fbc32784700 15 client.24722.objecter _session_command_op_assign -1 14 2024-04-14T15:12:21.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.129+0000 7fbc32784700 20 client.24722.objecter _get_session s=0x7fbc18020300 osd=6 3 2024-04-14T15:12:21.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.129+0000 7fbc32784700 20 client.24722.objecter put_session s=0x7fbc18020300 osd=6 3 2024-04-14T15:12:21.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.129+0000 7fbc32784700 20 client.24722.objecter _get_session s=0x7fbc18020300 osd=6 3 2024-04-14T15:12:21.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.129+0000 7fbc32784700 15 client.24722.objecter _session_command_op_remove -1 14 2024-04-14T15:12:21.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.129+0000 7fbc32784700 20 client.24722.objecter get_session s=0x7fbc18020300 osd=6 3 2024-04-14T15:12:21.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.129+0000 7fbc32784700 15 client.24722.objecter _session_command_op_assign 6 14 2024-04-14T15:12:21.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.129+0000 7fbc32784700 20 client.24722.objecter put_session s=0x7fbc18020300 osd=6 4 2024-04-14T15:12:21.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.129+0000 7fbc32784700 10 client.24722.objecter _send_command 14 2024-04-14T15:12:21.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.129+0000 7fbc32784700 1 -- 172.21.15.90:0/2291512326 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fbc1801ef10 con 0x7fbc18023a30 2024-04-14T15:12:21.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.131+0000 7fbc297fa700 1 -- 172.21.15.90:0/2291512326 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fbc1801ef10 con 0x7fbc18023a30 2024-04-14T15:12:21.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.131+0000 7fbc297fa700 10 client.24722.objecter ms_dispatch 0x7fbc2c001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:12:21.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.131+0000 7fbc297fa700 10 client.24722.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:12:21.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.131+0000 7fbc297fa700 20 client.24722.objecter put_session s=0x7fbc18020300 osd=6 4 2024-04-14T15:12:21.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.131+0000 7fbc297fa700 15 client.24722.objecter _session_command_op_remove 6 14 2024-04-14T15:12:21.132 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:12:21.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.131+0000 7fbc32784700 10 client.24722.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:12:21.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.131+0000 7fbc32784700 15 client.24722.objecter _session_command_op_assign -1 15 2024-04-14T15:12:21.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.131+0000 7fbc32784700 20 client.24722.objecter _get_session s=0x7fbc18017a40 osd=7 3 2024-04-14T15:12:21.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.131+0000 7fbc32784700 20 client.24722.objecter put_session s=0x7fbc18017a40 osd=7 3 2024-04-14T15:12:21.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.131+0000 7fbc32784700 20 client.24722.objecter _get_session s=0x7fbc18017a40 osd=7 3 2024-04-14T15:12:21.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.131+0000 7fbc32784700 15 client.24722.objecter _session_command_op_remove -1 15 2024-04-14T15:12:21.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.131+0000 7fbc32784700 20 client.24722.objecter get_session s=0x7fbc18017a40 osd=7 3 2024-04-14T15:12:21.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.131+0000 7fbc32784700 15 client.24722.objecter _session_command_op_assign 7 15 2024-04-14T15:12:21.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.131+0000 7fbc32784700 20 client.24722.objecter put_session s=0x7fbc18017a40 osd=7 4 2024-04-14T15:12:21.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.131+0000 7fbc32784700 10 client.24722.objecter _send_command 15 2024-04-14T15:12:21.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.131+0000 7fbc32784700 1 -- 172.21.15.90:0/2291512326 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fbc18016f60 con 0x7fbc1801ecc0 2024-04-14T15:12:21.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.132+0000 7fbc297fa700 10 client.24722.objecter ms_handle_connect 0x7fbc1801ecc0 2024-04-14T15:12:21.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.133+0000 7fbc297fa700 1 -- 172.21.15.90:0/2291512326 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fbc18016f60 con 0x7fbc1801ecc0 2024-04-14T15:12:21.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.133+0000 7fbc297fa700 10 client.24722.objecter ms_dispatch 0x7fbc2c001550 command_reply(tid 15: 0 ) 2024-04-14T15:12:21.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.133+0000 7fbc297fa700 10 client.24722.objecter _finish_command 15 = system:0 2024-04-14T15:12:21.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.133+0000 7fbc297fa700 20 client.24722.objecter put_session s=0x7fbc18017a40 osd=7 4 2024-04-14T15:12:21.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.133+0000 7fbc297fa700 15 client.24722.objecter _session_command_op_remove 7 15 2024-04-14T15:12:21.149 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:21.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:21.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:12:21.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.150+0000 7fbc32784700 10 client.24722.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:12:21.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.150+0000 7fbc32784700 15 client.24722.objecter _session_command_op_assign -1 16 2024-04-14T15:12:21.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.150+0000 7fbc32784700 20 client.24722.objecter _get_session s=0x7fbc18017a40 osd=7 3 2024-04-14T15:12:21.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.150+0000 7fbc32784700 20 client.24722.objecter put_session s=0x7fbc18017a40 osd=7 3 2024-04-14T15:12:21.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.150+0000 7fbc32784700 20 client.24722.objecter _get_session s=0x7fbc18017a40 osd=7 3 2024-04-14T15:12:21.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.150+0000 7fbc32784700 15 client.24722.objecter _session_command_op_remove -1 16 2024-04-14T15:12:21.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.150+0000 7fbc32784700 20 client.24722.objecter get_session s=0x7fbc18017a40 osd=7 3 2024-04-14T15:12:21.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.150+0000 7fbc32784700 15 client.24722.objecter _session_command_op_assign 7 16 2024-04-14T15:12:21.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.150+0000 7fbc32784700 20 client.24722.objecter put_session s=0x7fbc18017a40 osd=7 4 2024-04-14T15:12:21.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.150+0000 7fbc32784700 10 client.24722.objecter _send_command 16 2024-04-14T15:12:21.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.150+0000 7fbc32784700 1 -- 172.21.15.90:0/2291512326 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fbc18016f60 con 0x7fbc1801ecc0 2024-04-14T15:12:21.152 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:12:21.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.151+0000 7fbc297fa700 1 -- 172.21.15.90:0/2291512326 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fbc18016f60 con 0x7fbc1801ecc0 2024-04-14T15:12:21.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.151+0000 7fbc297fa700 10 client.24722.objecter ms_dispatch 0x7fbc2c001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:12:21.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.151+0000 7fbc297fa700 10 client.24722.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:12:21.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.151+0000 7fbc297fa700 20 client.24722.objecter put_session s=0x7fbc18017a40 osd=7 4 2024-04-14T15:12:21.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.151+0000 7fbc297fa700 15 client.24722.objecter _session_command_op_remove 7 16 2024-04-14T15:12:21.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.152+0000 7fbc32784700 10 librados: watch_flush enter 2024-04-14T15:12:21.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.152+0000 7fbc32784700 10 librados: watch_flush exit 2024-04-14T15:12:21.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.152+0000 7fbc32784700 10 client.24722.objecter close_session for osd.0 2024-04-14T15:12:21.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.152+0000 7fbc32784700 1 -- 172.21.15.90:0/2291512326 >> v1:172.21.15.90:6801/3761806885 conn(0x7fbc18004730 legacy=0x7fbc18002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:21.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.152+0000 7fbc32784700 20 client.24722.objecter put_session s=0x7fbc180053e0 osd=0 1 2024-04-14T15:12:21.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.152+0000 7fbc32784700 10 client.24722.objecter close_session for osd.1 2024-04-14T15:12:21.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.152+0000 7fbc32784700 1 -- 172.21.15.90:0/2291512326 >> v1:172.21.15.90:6805/4170361699 conn(0x7fbc18004280 legacy=0x7fbc18008d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:21.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.152+0000 7fbc32784700 20 client.24722.objecter put_session s=0x7fbc18004110 osd=1 1 2024-04-14T15:12:21.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.152+0000 7fbc32784700 10 client.24722.objecter close_session for osd.2 2024-04-14T15:12:21.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.152+0000 7fbc32784700 1 -- 172.21.15.90:0/2291512326 >> v1:172.21.15.90:6809/1686184251 conn(0x7fbc18008880 legacy=0x7fbc18013ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:21.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.152+0000 7fbc32784700 20 client.24722.objecter put_session s=0x7fbc18001a00 osd=2 1 2024-04-14T15:12:21.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.152+0000 7fbc32784700 10 client.24722.objecter close_session for osd.3 2024-04-14T15:12:21.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.152+0000 7fbc32784700 1 -- 172.21.15.90:0/2291512326 >> v1:172.21.15.90:6813/2270166774 conn(0x7fbc18015530 legacy=0x7fbc18015990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:21.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.152+0000 7fbc32784700 20 client.24722.objecter put_session s=0x7fbc180081a0 osd=3 1 2024-04-14T15:12:21.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.152+0000 7fbc32784700 10 client.24722.objecter close_session for osd.4 2024-04-14T15:12:21.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.152+0000 7fbc32784700 1 -- 172.21.15.90:0/2291512326 >> v1:172.21.15.92:6800/3887549733 conn(0x7fbc1801af10 legacy=0x7fbc1801d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:21.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.152+0000 7fbc32784700 20 client.24722.objecter put_session s=0x7fbc18017bd0 osd=4 1 2024-04-14T15:12:21.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.152+0000 7fbc32784700 10 client.24722.objecter close_session for osd.5 2024-04-14T15:12:21.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.152+0000 7fbc32784700 1 -- 172.21.15.90:0/2291512326 >> v1:172.21.15.92:6804/3244561440 conn(0x7fbc1801f740 legacy=0x7fbc1801fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:21.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.152+0000 7fbc32784700 20 client.24722.objecter put_session s=0x7fbc18010c50 osd=5 1 2024-04-14T15:12:21.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.152+0000 7fbc30d21700 1 -- 172.21.15.90:0/2291512326 reap_dead start 2024-04-14T15:12:21.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.152+0000 7fbc32784700 10 client.24722.objecter close_session for osd.6 2024-04-14T15:12:21.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.152+0000 7fbc32784700 1 -- 172.21.15.90:0/2291512326 >> v1:172.21.15.92:6808/636125963 conn(0x7fbc18023a30 legacy=0x7fbc18025e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:21.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.152+0000 7fbc30d21700 1 -- 172.21.15.90:0/2291512326 reap_dead start 2024-04-14T15:12:21.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.152+0000 7fbc30d21700 1 -- 172.21.15.90:0/2291512326 reap_dead start 2024-04-14T15:12:21.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.152+0000 7fbc32784700 20 client.24722.objecter put_session s=0x7fbc18020300 osd=6 1 2024-04-14T15:12:21.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.152+0000 7fbc30d21700 1 -- 172.21.15.90:0/2291512326 reap_dead start 2024-04-14T15:12:21.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.152+0000 7fbc32784700 10 client.24722.objecter close_session for osd.7 2024-04-14T15:12:21.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.152+0000 7fbc32784700 1 -- 172.21.15.90:0/2291512326 >> v1:172.21.15.92:6812/1773706716 conn(0x7fbc1801ecc0 legacy=0x7fbc180286c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:21.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.152+0000 7fbc32784700 20 client.24722.objecter put_session s=0x7fbc18017a40 osd=7 1 2024-04-14T15:12:21.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.152+0000 7fbc32784700 20 client.24722.objecter shutdown clearing up homeless session... 2024-04-14T15:12:21.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.152+0000 7fbc32784700 10 client.24722.objecter successfully canceled tick 2024-04-14T15:12:21.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.152+0000 7fbc32784700 1 -- 172.21.15.90:0/2291512326 >> v1:172.21.15.90:6800/420935427 conn(0x7fbc14071330 legacy=0x7fbc140737f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:21.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.152+0000 7fbc32784700 1 -- 172.21.15.90:0/2291512326 >> v1:172.21.15.90:6790/0 conn(0x7fbc2c108890 legacy=0x7fbc2c10d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:21.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.152+0000 7fbc32784700 1 -- 172.21.15.90:0/2291512326 shutdown_connections 2024-04-14T15:12:21.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.152+0000 7fbc32784700 1 -- 172.21.15.90:0/2291512326 >> 172.21.15.90:0/2291512326 conn(0x7fbc2c0fe380 msgr2=0x7fbc2c112840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:12:21.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.152+0000 7fbc32784700 1 -- 172.21.15.90:0/2291512326 shutdown_connections 2024-04-14T15:12:21.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.152+0000 7fbc32784700 1 -- 172.21.15.90:0/2291512326 wait complete. 2024-04-14T15:12:21.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:21.153+0000 7fbc32784700 1 librados: shutdown 2024-04-14T15:12:21.156 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:12:21.156 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:12:21.156 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:12:21.156 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:12:21.156 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:12:21.157 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:12:21.157 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:12:21.157 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:12:21.250 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:12:21.275 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:21.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:21.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:12:21.377 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:12:21.378 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-14T15:12:21.378 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-14T15:12:21.378 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-14T15:12:21.378 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-14T15:12:21.378 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-14T15:12:21.404 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:21.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:21.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:12:21.507 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:12:21.531 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:21.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:21.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:12:21.632 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:12:21.655 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:21.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:21.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:12:21.757 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:12:21.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:21.781 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:21.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:12:21.882 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:12:21.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:21.907 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:21.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:12:21.962 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-14T15:12:21.966 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:21 smithi090 ceph-mon[41670]: 2.c scrub starts 2024-04-14T15:12:21.966 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:21 smithi090 ceph-mon[41670]: 2.c scrub ok 2024-04-14T15:12:21.966 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:21 smithi090 ceph-mon[41670]: 5.18 scrub starts 2024-04-14T15:12:21.966 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:21 smithi090 ceph-mon[41670]: 5.18 scrub ok 2024-04-14T15:12:21.966 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:21 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2291512326' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:12:21.967 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:21 smithi090 ceph-mon[30399]: 2.c scrub starts 2024-04-14T15:12:21.967 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:21 smithi090 ceph-mon[30399]: 2.c scrub ok 2024-04-14T15:12:21.967 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:21 smithi090 ceph-mon[30399]: 5.18 scrub starts 2024-04-14T15:12:21.967 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:21 smithi090 ceph-mon[30399]: 5.18 scrub ok 2024-04-14T15:12:21.967 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:21 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2291512326' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:12:22.008 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:12:22.038 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:22.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:22.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:12:22.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:21 smithi092 ceph-mon[33966]: 2.c scrub starts 2024-04-14T15:12:22.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:21 smithi092 ceph-mon[33966]: 2.c scrub ok 2024-04-14T15:12:22.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:21 smithi092 ceph-mon[33966]: 5.18 scrub starts 2024-04-14T15:12:22.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:21 smithi092 ceph-mon[33966]: 5.18 scrub ok 2024-04-14T15:12:22.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:21 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2291512326' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:12:22.140 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:12:22.165 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:22.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:22.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:12:22.267 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:12:22.291 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:22.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:22.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:12:22.393 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:12:22.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:22.419 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:22.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:12:22.520 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:12:22.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:22.548 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:22.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:12:22.649 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:12:22.672 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:22.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:22.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:12:22.774 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:12:22.798 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:22.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:22.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:12:22.813 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:22 smithi090 ceph-mon[30399]: pgmap v84: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:12:22.813 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:22 smithi090 ceph-mon[30399]: 4.10 scrub starts 2024-04-14T15:12:22.813 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:22 smithi090 ceph-mon[30399]: 4.10 scrub ok 2024-04-14T15:12:22.813 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:22 smithi090 ceph-mon[30399]: 4.6 scrub starts 2024-04-14T15:12:22.813 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:22 smithi090 ceph-mon[30399]: 4.6 scrub ok 2024-04-14T15:12:22.900 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:12:22.925 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:22.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:22.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:12:23.026 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:12:23.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:23.052 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:23.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:12:23.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:22 smithi092 ceph-mon[33966]: pgmap v84: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:12:23.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:22 smithi092 ceph-mon[33966]: 4.10 scrub starts 2024-04-14T15:12:23.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:22 smithi092 ceph-mon[33966]: 4.10 scrub ok 2024-04-14T15:12:23.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:22 smithi092 ceph-mon[33966]: 4.6 scrub starts 2024-04-14T15:12:23.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:22 smithi092 ceph-mon[33966]: 4.6 scrub ok 2024-04-14T15:12:23.153 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:12:23.176 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:23.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:23.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:12:23.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:22 smithi090 ceph-mon[41670]: pgmap v84: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:12:23.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:22 smithi090 ceph-mon[41670]: 4.10 scrub starts 2024-04-14T15:12:23.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:22 smithi090 ceph-mon[41670]: 4.10 scrub ok 2024-04-14T15:12:23.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:22 smithi090 ceph-mon[41670]: 4.6 scrub starts 2024-04-14T15:12:23.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:22 smithi090 ceph-mon[41670]: 4.6 scrub ok 2024-04-14T15:12:23.278 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:12:23.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:23.302 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:23.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:12:23.403 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:12:23.431 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:23.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:23.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:12:23.457 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-14T15:12:23.533 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:12:23.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:23.558 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:23.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:12:23.659 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:12:23.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:23.687 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:23.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:12:23.788 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:12:23.813 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:23.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:23.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:12:23.915 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:12:23.939 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:23.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:23.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:12:24.041 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:12:24.066 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:24.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:24.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:12:24.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:23 smithi092 ceph-mon[33966]: 2.18 scrub starts 2024-04-14T15:12:24.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:23 smithi092 ceph-mon[33966]: 2.18 scrub ok 2024-04-14T15:12:24.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:23 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:12:24.135 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:12:24.168 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:12:24.191 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:12:23 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:12:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:12:24.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:23 smithi090 ceph-mon[41670]: 2.18 scrub starts 2024-04-14T15:12:24.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:23 smithi090 ceph-mon[41670]: 2.18 scrub ok 2024-04-14T15:12:24.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:23 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:12:24.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:23 smithi090 ceph-mon[30399]: 2.18 scrub starts 2024-04-14T15:12:24.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:23 smithi090 ceph-mon[30399]: 2.18 scrub ok 2024-04-14T15:12:24.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:23 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:12:24.192 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:24.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:24.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:12:24.294 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:12:24.318 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:24.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:24.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:12:24.420 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:12:24.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:24.449 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:24.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:12:24.550 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:12:24.564 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-14T15:12:24.575 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:24.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:24.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:12:24.677 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:12:24.701 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:24.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:24.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:12:24.803 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:12:24.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:24.829 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:24.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:12:24.860 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:24 smithi090 ceph-mon[41670]: pgmap v85: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:12:24.860 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:24 smithi090 ceph-mon[41670]: 5.2 scrub starts 2024-04-14T15:12:24.860 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:24 smithi090 ceph-mon[41670]: 5.2 scrub ok 2024-04-14T15:12:24.860 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:24 smithi090 ceph-mon[41670]: 2.a scrub starts 2024-04-14T15:12:24.861 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:24 smithi090 ceph-mon[41670]: 2.a scrub ok 2024-04-14T15:12:24.861 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:24 smithi090 ceph-mon[41670]: 6.e scrub starts 2024-04-14T15:12:24.861 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:24 smithi090 ceph-mon[30399]: pgmap v85: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:12:24.861 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:24 smithi090 ceph-mon[30399]: 5.2 scrub starts 2024-04-14T15:12:24.861 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:24 smithi090 ceph-mon[30399]: 5.2 scrub ok 2024-04-14T15:12:24.861 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:24 smithi090 ceph-mon[30399]: 2.a scrub starts 2024-04-14T15:12:24.861 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:24 smithi090 ceph-mon[30399]: 2.a scrub ok 2024-04-14T15:12:24.861 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:24 smithi090 ceph-mon[30399]: 6.e scrub starts 2024-04-14T15:12:24.930 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:12:24.956 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:24.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:24.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:12:25.058 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:12:25.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:25.083 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:25.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:12:25.122 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:12:24 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:12:25.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:24 smithi092 ceph-mon[33966]: pgmap v85: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:12:25.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:24 smithi092 ceph-mon[33966]: 5.2 scrub starts 2024-04-14T15:12:25.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:24 smithi092 ceph-mon[33966]: 5.2 scrub ok 2024-04-14T15:12:25.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:24 smithi092 ceph-mon[33966]: 2.a scrub starts 2024-04-14T15:12:25.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:24 smithi092 ceph-mon[33966]: 2.a scrub ok 2024-04-14T15:12:25.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:24 smithi092 ceph-mon[33966]: 6.e scrub starts 2024-04-14T15:12:25.185 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:12:25.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:25.210 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:25.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:12:25.311 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:12:25.326 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:12:25.336 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:25.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:25.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:12:25.438 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:12:25.462 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:25.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:25.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:12:25.564 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:12:25.588 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:25.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:25.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:12:25.690 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:12:25.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:25.714 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:25.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:12:25.814 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:12:25.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:25.842 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:25.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:12:25.943 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:12:25.969 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:25.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:25.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:12:26.071 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:12:26.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:26.097 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:26.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:12:26.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:25 smithi092 ceph-mon[33966]: 6.e scrub ok 2024-04-14T15:12:26.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:25 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:12:26.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:25 smithi092 ceph-mon[33966]: 6.9 scrub starts 2024-04-14T15:12:26.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:25 smithi092 ceph-mon[33966]: 6.9 scrub ok 2024-04-14T15:12:26.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:25 smithi092 ceph-mon[33966]: 5.1f scrub starts 2024-04-14T15:12:26.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:25 smithi092 ceph-mon[33966]: 5.1f scrub ok 2024-04-14T15:12:26.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:25 smithi090 ceph-mon[30399]: 6.e scrub ok 2024-04-14T15:12:26.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:25 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:12:26.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:25 smithi090 ceph-mon[30399]: 6.9 scrub starts 2024-04-14T15:12:26.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:25 smithi090 ceph-mon[30399]: 6.9 scrub ok 2024-04-14T15:12:26.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:25 smithi090 ceph-mon[30399]: 5.1f scrub starts 2024-04-14T15:12:26.190 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:25 smithi090 ceph-mon[30399]: 5.1f scrub ok 2024-04-14T15:12:26.190 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:25 smithi090 ceph-mon[41670]: 6.e scrub ok 2024-04-14T15:12:26.190 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:25 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:12:26.190 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:25 smithi090 ceph-mon[41670]: 6.9 scrub starts 2024-04-14T15:12:26.190 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:25 smithi090 ceph-mon[41670]: 6.9 scrub ok 2024-04-14T15:12:26.190 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:25 smithi090 ceph-mon[41670]: 5.1f scrub starts 2024-04-14T15:12:26.190 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:25 smithi090 ceph-mon[41670]: 5.1f scrub ok 2024-04-14T15:12:26.198 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:12:26.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:26.228 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:26.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:12:26.329 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:12:26.357 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:26.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:26.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:12:26.459 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:12:26.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:26.485 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:26.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:12:26.586 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:12:26.614 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:26.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:26.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:12:26.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.705+0000 7f410dbb2700 1 -- 172.21.15.90:0/476491655 >> v1:172.21.15.90:6789/0 conn(0x7f410810e1e0 legacy=0x7f41081106a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:26.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.705+0000 7f410dbb2700 1 -- 172.21.15.90:0/476491655 shutdown_connections 2024-04-14T15:12:26.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.705+0000 7f410dbb2700 1 -- 172.21.15.90:0/476491655 >> 172.21.15.90:0/476491655 conn(0x7f41080fd290 msgr2=0x7f41080ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:12:26.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.705+0000 7f410dbb2700 1 -- 172.21.15.90:0/476491655 shutdown_connections 2024-04-14T15:12:26.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.705+0000 7f410dbb2700 1 -- 172.21.15.90:0/476491655 wait complete. 2024-04-14T15:12:26.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.705+0000 7f410dbb2700 1 librados: starting msgr at 2024-04-14T15:12:26.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.705+0000 7f410dbb2700 1 librados: starting objecter 2024-04-14T15:12:26.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.706+0000 7f410dbb2700 1 Processor -- start 2024-04-14T15:12:26.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.706+0000 7f410dbb2700 1 -- start start 2024-04-14T15:12:26.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.706+0000 7f410dbb2700 1 librados: setting wanted keys 2024-04-14T15:12:26.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.706+0000 7f410dbb2700 1 librados: calling monclient init 2024-04-14T15:12:26.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.706+0000 7f410dbb2700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f41081a7df0 con 0x7f4108107760 2024-04-14T15:12:26.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.706+0000 7f410dbb2700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f41081a8fd0 con 0x7f410810a610 2024-04-14T15:12:26.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.706+0000 7f410dbb2700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f41081aa1b0 con 0x7f410810e1e0 2024-04-14T15:12:26.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.706+0000 7f41077fe700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f4108107760 0x7f410806a460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:56012/0 (socket says 172.21.15.90:56012) 2024-04-14T15:12:26.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.706+0000 7f41077fe700 1 -- 172.21.15.90:0/3648728731 learned_addr learned my addr 172.21.15.90:0/3648728731 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:12:26.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.706+0000 7f4104ff9700 10 client.?.objecter ms_handle_connect 0x7f4108107760 2024-04-14T15:12:26.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.706+0000 7f4104ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:12:26.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.706+0000 7f4104ff9700 10 client.?.objecter ms_handle_connect 0x7f410810e1e0 2024-04-14T15:12:26.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.706+0000 7f4104ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:12:26.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.707+0000 7f4104ff9700 10 client.?.objecter ms_handle_connect 0x7f410810a610 2024-04-14T15:12:26.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.707+0000 7f4104ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:12:26.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.707+0000 7f4104ff9700 1 -- 172.21.15.90:0/3648728731 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4156118824 0 0) 0x7f41081a7df0 con 0x7f4108107760 2024-04-14T15:12:26.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.707+0000 7f4104ff9700 1 -- 172.21.15.90:0/3648728731 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f40f00031f0 con 0x7f4108107760 2024-04-14T15:12:26.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.707+0000 7f4104ff9700 1 -- 172.21.15.90:0/3648728731 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2563948337 0 0) 0x7f41081aa1b0 con 0x7f410810e1e0 2024-04-14T15:12:26.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.707+0000 7f4104ff9700 1 -- 172.21.15.90:0/3648728731 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f41081a7df0 con 0x7f410810e1e0 2024-04-14T15:12:26.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.707+0000 7f4104ff9700 1 -- 172.21.15.90:0/3648728731 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 175360302 0 0) 0x7f41081a8fd0 con 0x7f410810a610 2024-04-14T15:12:26.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.707+0000 7f4104ff9700 1 -- 172.21.15.90:0/3648728731 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f41081aa1b0 con 0x7f410810a610 2024-04-14T15:12:26.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.707+0000 7f4104ff9700 1 -- 172.21.15.90:0/3648728731 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3926308019 0 0) 0x7f40f00031f0 con 0x7f4108107760 2024-04-14T15:12:26.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.707+0000 7f4104ff9700 1 -- 172.21.15.90:0/3648728731 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f41081a8fd0 con 0x7f4108107760 2024-04-14T15:12:26.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.707+0000 7f4104ff9700 1 -- 172.21.15.90:0/3648728731 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f40f80018b0 con 0x7f4108107760 2024-04-14T15:12:26.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.707+0000 7f4104ff9700 1 -- 172.21.15.90:0/3648728731 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2203907149 0 0) 0x7f41081a7df0 con 0x7f410810e1e0 2024-04-14T15:12:26.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.707+0000 7f4104ff9700 1 -- 172.21.15.90:0/3648728731 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f40f00031f0 con 0x7f410810e1e0 2024-04-14T15:12:26.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.707+0000 7f4104ff9700 1 -- 172.21.15.90:0/3648728731 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f4100001950 con 0x7f410810e1e0 2024-04-14T15:12:26.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.707+0000 7f4104ff9700 1 -- 172.21.15.90:0/3648728731 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1337838082 0 0) 0x7f41081a8fd0 con 0x7f4108107760 2024-04-14T15:12:26.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.707+0000 7f4104ff9700 1 -- 172.21.15.90:0/3648728731 >> v1:172.21.15.90:6790/0 conn(0x7f410810e1e0 legacy=0x7f41081a66d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:26.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.707+0000 7f4104ff9700 1 -- 172.21.15.90:0/3648728731 >> v1:172.21.15.92:6789/0 conn(0x7f410810a610 legacy=0x7f410806ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:26.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.707+0000 7f4104ff9700 1 -- 172.21.15.90:0/3648728731 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41081ab390 con 0x7f4108107760 2024-04-14T15:12:26.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.708+0000 7f410dbb2700 1 -- 172.21.15.90:0/3648728731 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f41081aa3e0 con 0x7f4108107760 2024-04-14T15:12:26.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.708+0000 7f410dbb2700 10 client.14862.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:12:26.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.708+0000 7f410dbb2700 1 -- 172.21.15.90:0/3648728731 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f41081aa8a0 con 0x7f4108107760 2024-04-14T15:12:26.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.708+0000 7f410dbb2700 1 librados: init done 2024-04-14T15:12:26.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.708+0000 7f4104ff9700 1 -- 172.21.15.90:0/3648728731 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f40f8001ba0 con 0x7f4108107760 2024-04-14T15:12:26.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.708+0000 7f4104ff9700 1 -- 172.21.15.90:0/3648728731 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f40f8005d70 con 0x7f4108107760 2024-04-14T15:12:26.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.709+0000 7f410dbb2700 1 -- 172.21.15.90:0/3648728731 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40f4005150 con 0x7f4108107760 2024-04-14T15:12:26.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.710+0000 7f4104ff9700 1 -- 172.21.15.90:0/3648728731 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f40f8003950 con 0x7f4108107760 2024-04-14T15:12:26.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.710+0000 7f4104ff9700 1 -- 172.21.15.90:0/3648728731 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(101..101 src has 51..101) ==== 6320+0+0 (unknown 1370162482 0 0) 0x7f40f808a210 con 0x7f4108107760 2024-04-14T15:12:26.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.710+0000 7f4104ff9700 10 client.14862.objecter ms_dispatch 0x7f4108001550 osd_map(101..101 src has 51..101) 2024-04-14T15:12:26.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.710+0000 7f4104ff9700 3 client.14862.objecter handle_osd_map got epochs [101,101] > 0 2024-04-14T15:12:26.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.710+0000 7f4104ff9700 3 client.14862.objecter handle_osd_map decoding full epoch 101 2024-04-14T15:12:26.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.710+0000 7f4104ff9700 20 client.14862.objecter dump_active .. 0 homeless 2024-04-14T15:12:26.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.711+0000 7f4104ff9700 10 client.14862.objecter ms_handle_connect 0x7f40f0075480 2024-04-14T15:12:26.715 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:12:26.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.719+0000 7f4104ff9700 1 -- 172.21.15.90:0/3648728731 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f40f8057360 con 0x7f4108107760 2024-04-14T15:12:26.743 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:26.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:26.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:12:26.844 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:12:26.868 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:26.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:26.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:12:26.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.946+0000 7f49ba103700 1 -- 172.21.15.90:0/311492703 >> v1:172.21.15.90:6789/0 conn(0x7f49b410b930 legacy=0x7f49b410dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:26.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.946+0000 7f49ba103700 1 -- 172.21.15.90:0/311492703 shutdown_connections 2024-04-14T15:12:26.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.946+0000 7f49ba103700 1 -- 172.21.15.90:0/311492703 >> 172.21.15.90:0/311492703 conn(0x7f49b40fd290 msgr2=0x7f49b40ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:12:26.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.946+0000 7f49ba103700 1 -- 172.21.15.90:0/311492703 shutdown_connections 2024-04-14T15:12:26.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.946+0000 7f49ba103700 1 -- 172.21.15.90:0/311492703 wait complete. 2024-04-14T15:12:26.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.947+0000 7f49ba103700 1 librados: starting msgr at 2024-04-14T15:12:26.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.947+0000 7f49ba103700 1 librados: starting objecter 2024-04-14T15:12:26.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.947+0000 7f49ba103700 1 Processor -- start 2024-04-14T15:12:26.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.947+0000 7f49ba103700 1 -- start start 2024-04-14T15:12:26.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.947+0000 7f49ba103700 1 librados: setting wanted keys 2024-04-14T15:12:26.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.947+0000 7f49ba103700 1 librados: calling monclient init 2024-04-14T15:12:26.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.947+0000 7f49ba103700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f49b41ae150 con 0x7f49b4103a40 2024-04-14T15:12:26.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.947+0000 7f49ba103700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f49b41af370 con 0x7f49b410b930 2024-04-14T15:12:26.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.947+0000 7f49ba103700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f49b41b0590 con 0x7f49b410f4d0 2024-04-14T15:12:26.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.948+0000 7f49b2ffd700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f49b410b930 0x7f49b41a9100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.90:44822/0 (socket says 172.21.15.90:44822) 2024-04-14T15:12:26.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.948+0000 7f49b2ffd700 1 -- 172.21.15.90:0/3030676872 learned_addr learned my addr 172.21.15.90:0/3030676872 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:12:26.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.948+0000 7f49b0ff9700 10 client.?.objecter ms_handle_connect 0x7f49b410f4d0 2024-04-14T15:12:26.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.948+0000 7f49b0ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:12:26.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.948+0000 7f49b0ff9700 10 client.?.objecter ms_handle_connect 0x7f49b4103a40 2024-04-14T15:12:26.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.948+0000 7f49b0ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:12:26.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.948+0000 7f49b0ff9700 10 client.?.objecter ms_handle_connect 0x7f49b410b930 2024-04-14T15:12:26.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.948+0000 7f49b0ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:12:26.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.948+0000 7f49b0ff9700 1 -- 172.21.15.90:0/3030676872 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3281831956 0 0) 0x7f49b41b0590 con 0x7f49b410f4d0 2024-04-14T15:12:26.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.948+0000 7f49b0ff9700 1 -- 172.21.15.90:0/3030676872 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f499c0035b0 con 0x7f49b410f4d0 2024-04-14T15:12:26.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.948+0000 7f49b0ff9700 1 -- 172.21.15.90:0/3030676872 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1878991061 0 0) 0x7f49b41ae150 con 0x7f49b4103a40 2024-04-14T15:12:26.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.948+0000 7f49b0ff9700 1 -- 172.21.15.90:0/3030676872 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f49b41b0590 con 0x7f49b4103a40 2024-04-14T15:12:26.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.948+0000 7f49b0ff9700 1 -- 172.21.15.90:0/3030676872 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3080890604 0 0) 0x7f49b41af370 con 0x7f49b410b930 2024-04-14T15:12:26.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.948+0000 7f49b0ff9700 1 -- 172.21.15.90:0/3030676872 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f49b41ae150 con 0x7f49b410b930 2024-04-14T15:12:26.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.949+0000 7f49b0ff9700 1 -- 172.21.15.90:0/3030676872 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3373281174 0 0) 0x7f499c0035b0 con 0x7f49b410f4d0 2024-04-14T15:12:26.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.949+0000 7f49b0ff9700 1 -- 172.21.15.90:0/3030676872 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f49b41af370 con 0x7f49b410f4d0 2024-04-14T15:12:26.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.949+0000 7f49b0ff9700 1 -- 172.21.15.90:0/3030676872 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f49ac001ba0 con 0x7f49b410f4d0 2024-04-14T15:12:26.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.949+0000 7f49b0ff9700 1 -- 172.21.15.90:0/3030676872 <== mon.2 v1:172.21.15.90:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3193050225 0 0) 0x7f49b41af370 con 0x7f49b410f4d0 2024-04-14T15:12:26.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.949+0000 7f49b0ff9700 1 -- 172.21.15.90:0/3030676872 >> v1:172.21.15.92:6789/0 conn(0x7f49b410b930 legacy=0x7f49b41a9100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:26.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.949+0000 7f49b0ff9700 1 -- 172.21.15.90:0/3030676872 >> v1:172.21.15.90:6789/0 conn(0x7f49b4103a40 legacy=0x7f49b410afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:26.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.949+0000 7f49b0ff9700 1 -- 172.21.15.90:0/3030676872 --> v1:172.21.15.90:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49b41b17d0 con 0x7f49b410f4d0 2024-04-14T15:12:26.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.949+0000 7f49ba103700 1 -- 172.21.15.90:0/3030676872 --> v1:172.21.15.90:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f49b41af5a0 con 0x7f49b410f4d0 2024-04-14T15:12:26.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.949+0000 7f49ba103700 10 client.24734.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:12:26.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.949+0000 7f49ba103700 1 -- 172.21.15.90:0/3030676872 --> v1:172.21.15.90:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f49b41afa60 con 0x7f49b410f4d0 2024-04-14T15:12:26.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.950+0000 7f49ba103700 1 librados: init done 2024-04-14T15:12:26.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.950+0000 7f49b0ff9700 1 -- 172.21.15.90:0/3030676872 <== mon.2 v1:172.21.15.90:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f49ac003ec0 con 0x7f49b410f4d0 2024-04-14T15:12:26.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.950+0000 7f49b0ff9700 1 -- 172.21.15.90:0/3030676872 <== mon.2 v1:172.21.15.90:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f49ac004e10 con 0x7f49b410f4d0 2024-04-14T15:12:26.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.952+0000 7f49ba103700 1 -- 172.21.15.90:0/3030676872 --> v1:172.21.15.90:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49a0005120 con 0x7f49b410f4d0 2024-04-14T15:12:26.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.952+0000 7f49b0ff9700 1 -- 172.21.15.90:0/3030676872 <== mon.2 v1:172.21.15.90:6790/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f49ac01b9e0 con 0x7f49b410f4d0 2024-04-14T15:12:26.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.953+0000 7f49b0ff9700 1 -- 172.21.15.90:0/3030676872 <== mon.2 v1:172.21.15.90:6790/0 8 ==== osd_map(101..101 src has 51..101) ==== 6320+0+0 (unknown 1370162482 0 0) 0x7f49ac089520 con 0x7f49b410f4d0 2024-04-14T15:12:26.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.953+0000 7f49b0ff9700 10 client.24734.objecter ms_dispatch 0x7f49b4001550 osd_map(101..101 src has 51..101) 2024-04-14T15:12:26.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.953+0000 7f49b0ff9700 3 client.24734.objecter handle_osd_map got epochs [101,101] > 0 2024-04-14T15:12:26.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.953+0000 7f49b0ff9700 3 client.24734.objecter handle_osd_map decoding full epoch 101 2024-04-14T15:12:26.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.953+0000 7f49b0ff9700 20 client.24734.objecter dump_active .. 0 homeless 2024-04-14T15:12:26.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.953+0000 7f49b0ff9700 10 client.24734.objecter ms_handle_connect 0x7f499c0714b0 2024-04-14T15:12:26.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:26.957+0000 7f49b0ff9700 1 -- 172.21.15.90:0/3030676872 <== mon.2 v1:172.21.15.90:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f49ac057590 con 0x7f49b410f4d0 2024-04-14T15:12:26.958 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:26 smithi090 ceph-mon[30399]: pgmap v86: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:12:26.958 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:26 smithi090 ceph-mon[30399]: 6.18 scrub starts 2024-04-14T15:12:26.958 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:26 smithi090 ceph-mon[30399]: 6.18 scrub ok 2024-04-14T15:12:26.958 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:26 smithi090 ceph-mon[30399]: 6.b scrub starts 2024-04-14T15:12:26.958 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:26 smithi090 ceph-mon[30399]: 6.b scrub ok 2024-04-14T15:12:26.959 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:26 smithi090 ceph-mon[30399]: 2.19 scrub starts 2024-04-14T15:12:26.959 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:26 smithi090 ceph-mon[30399]: 2.19 scrub ok 2024-04-14T15:12:26.959 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:26 smithi090 ceph-mon[30399]: 2.d scrub starts 2024-04-14T15:12:26.959 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:26 smithi090 ceph-mon[30399]: 2.d scrub ok 2024-04-14T15:12:26.959 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:26 smithi090 ceph-mon[41670]: pgmap v86: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:12:26.959 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:26 smithi090 ceph-mon[41670]: 6.18 scrub starts 2024-04-14T15:12:26.959 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:26 smithi090 ceph-mon[41670]: 6.18 scrub ok 2024-04-14T15:12:26.959 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:26 smithi090 ceph-mon[41670]: 6.b scrub starts 2024-04-14T15:12:26.959 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:26 smithi090 ceph-mon[41670]: 6.b scrub ok 2024-04-14T15:12:26.959 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:26 smithi090 ceph-mon[41670]: 2.19 scrub starts 2024-04-14T15:12:26.959 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:26 smithi090 ceph-mon[41670]: 2.19 scrub ok 2024-04-14T15:12:26.959 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:26 smithi090 ceph-mon[41670]: 2.d scrub starts 2024-04-14T15:12:26.960 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:26 smithi090 ceph-mon[41670]: 2.d scrub ok 2024-04-14T15:12:26.970 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:12:26.998 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:26.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:26.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:12:27.099 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:12:27.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:27.114+0000 7f410dbb2700 1 -- 172.21.15.90:0/3648728731 --> v1:172.21.15.90:6800/420935427 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f40f4000bc0 con 0x7f40f0075480 2024-04-14T15:12:27.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:26 smithi092 ceph-mon[33966]: pgmap v86: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:12:27.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:26 smithi092 ceph-mon[33966]: 6.18 scrub starts 2024-04-14T15:12:27.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:26 smithi092 ceph-mon[33966]: 6.18 scrub ok 2024-04-14T15:12:27.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:26 smithi092 ceph-mon[33966]: 6.b scrub starts 2024-04-14T15:12:27.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:26 smithi092 ceph-mon[33966]: 6.b scrub ok 2024-04-14T15:12:27.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:26 smithi092 ceph-mon[33966]: 2.19 scrub starts 2024-04-14T15:12:27.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:26 smithi092 ceph-mon[33966]: 2.19 scrub ok 2024-04-14T15:12:27.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:26 smithi092 ceph-mon[33966]: 2.d scrub starts 2024-04-14T15:12:27.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:26 smithi092 ceph-mon[33966]: 2.d scrub ok 2024-04-14T15:12:27.127 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:27.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:27.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:12:27.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:27.131+0000 7f4104ff9700 1 -- 172.21.15.90:0/3648728731 <== mgr.14610 v1:172.21.15.90:6800/420935427 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+346203 (unknown 2965378022 0 329524823) 0x7f40f4000bc0 con 0x7f40f0075480 2024-04-14T15:12:27.136 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:12:27.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:27.136+0000 7f410dbb2700 10 librados: watch_flush enter 2024-04-14T15:12:27.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:27.136+0000 7f410dbb2700 10 librados: watch_flush exit 2024-04-14T15:12:27.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:27.136+0000 7f410dbb2700 20 client.14862.objecter shutdown clearing up homeless session... 2024-04-14T15:12:27.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:27.136+0000 7f410dbb2700 10 client.14862.objecter successfully canceled tick 2024-04-14T15:12:27.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:27.136+0000 7f410dbb2700 1 -- 172.21.15.90:0/3648728731 >> v1:172.21.15.90:6800/420935427 conn(0x7f40f0075480 legacy=0x7f40f0077940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:27.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:27.136+0000 7f410dbb2700 1 -- 172.21.15.90:0/3648728731 >> v1:172.21.15.90:6789/0 conn(0x7f4108107760 legacy=0x7f410806a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:27.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:27.137+0000 7f410dbb2700 1 -- 172.21.15.90:0/3648728731 shutdown_connections 2024-04-14T15:12:27.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:27.137+0000 7f410dbb2700 1 -- 172.21.15.90:0/3648728731 >> 172.21.15.90:0/3648728731 conn(0x7f41080fd290 msgr2=0x7f4108111780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:12:27.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:27.137+0000 7f410dbb2700 1 -- 172.21.15.90:0/3648728731 shutdown_connections 2024-04-14T15:12:27.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:27.137+0000 7f410dbb2700 1 -- 172.21.15.90:0/3648728731 wait complete. 2024-04-14T15:12:27.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:27.137+0000 7f410dbb2700 1 librados: shutdown 2024-04-14T15:12:27.141 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-14T15:12:27.228 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:12:27.255 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:27.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:27.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:12:27.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:27.319+0000 7f49ba103700 1 -- 172.21.15.90:0/3030676872 --> v1:172.21.15.90:6790/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7f49a0005410 con 0x7f49b410f4d0 2024-04-14T15:12:27.357 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:12:27.398 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:27.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:27.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:12:27.500 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:12:27.527 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:27.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:27.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:12:27.629 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:12:27.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:27.663 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:27.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:12:27.764 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:12:27.794 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:27.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:27.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:12:27.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:27.846+0000 7f49b0ff9700 1 -- 172.21.15.90:0/3030676872 <== mon.2 v1:172.21.15.90:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v102) ==== 101+0+0 (unknown 4057388096 0 0) 0x7f49ac05b240 con 0x7f49b410f4d0 2024-04-14T15:12:27.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:27.850+0000 7f49ba103700 10 librados: watch_flush enter 2024-04-14T15:12:27.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:27.850+0000 7f49ba103700 10 librados: watch_flush exit 2024-04-14T15:12:27.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:27.850+0000 7f49ba103700 20 client.24734.objecter shutdown clearing up homeless session... 2024-04-14T15:12:27.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:27.850+0000 7f49ba103700 10 client.24734.objecter successfully canceled tick 2024-04-14T15:12:27.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:27.850+0000 7f49ba103700 1 -- 172.21.15.90:0/3030676872 >> v1:172.21.15.90:6800/420935427 conn(0x7f499c0714b0 legacy=0x7f499c073970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:27.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:27.850+0000 7f49ba103700 1 -- 172.21.15.90:0/3030676872 >> v1:172.21.15.90:6790/0 conn(0x7f49b410f4d0 legacy=0x7f49b41ac830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:27.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:27.850+0000 7f49ba103700 1 -- 172.21.15.90:0/3030676872 shutdown_connections 2024-04-14T15:12:27.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:27.850+0000 7f49ba103700 1 -- 172.21.15.90:0/3030676872 >> 172.21.15.90:0/3030676872 conn(0x7f49b40fd290 msgr2=0x7f49b4110210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:12:27.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:27.850+0000 7f49ba103700 1 -- 172.21.15.90:0/3030676872 shutdown_connections 2024-04-14T15:12:27.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:27.851+0000 7f49ba103700 1 -- 172.21.15.90:0/3030676872 wait complete. 2024-04-14T15:12:27.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:27.851+0000 7f49ba103700 1 librados: shutdown 2024-04-14T15:12:27.854 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is unset 2024-04-14T15:12:27.863 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":87,"stamp":"2024-04-14T15:12:26.965034+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1077,"num_read_kb":923,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":1771324,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":621,"ondisk_log_size":621,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":240232,"kb_used_data":7452,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":749720984,"statfs":{"total":767960285184,"available":767714287616,"internally_reserved":0,"allocated":7630848,"data_stored":4532312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":238276146},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001945"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":80,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:18.792045+0000","last_change":"2024-04-14T15:11:46.589109+0000","last_active":"2024-04-14T15:12:18.792045+0000","last_peered":"2024-04-14T15:12:18.792045+0000","last_clean":"2024-04-14T15:12:18.792045+0000","last_became_active":"2024-04-14T15:09:49.129636+0000","last_became_peered":"2024-04-14T15:09:49.129636+0000","last_unstale":"2024-04-14T15:12:18.792045+0000","last_undegraded":"2024-04-14T15:12:18.792045+0000","last_fullsized":"2024-04-14T15:12:18.792045+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:46.589000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:46.589000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:52.616155+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":84,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:20.610242+0000","last_change":"2024-04-14T15:12:20.610268+0000","last_active":"2024-04-14T15:12:20.610242+0000","last_peered":"2024-04-14T15:12:20.610242+0000","last_clean":"2024-04-14T15:12:20.610242+0000","last_became_active":"2024-04-14T15:09:47.042538+0000","last_became_peered":"2024-04-14T15:09:47.042538+0000","last_unstale":"2024-04-14T15:12:20.610242+0000","last_undegraded":"2024-04-14T15:12:20.610242+0000","last_fullsized":"2024-04-14T15:12:20.610242+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:20.610159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:12:20.610159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:42.966823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":92,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.059287+0000","last_change":"2024-04-14T15:11:22.897018+0000","last_active":"2024-04-14T15:12:19.059287+0000","last_peered":"2024-04-14T15:12:19.059287+0000","last_clean":"2024-04-14T15:12:19.059287+0000","last_became_active":"2024-04-14T15:09:43.017351+0000","last_became_peered":"2024-04-14T15:09:43.017351+0000","last_unstale":"2024-04-14T15:12:19.059287+0000","last_undegraded":"2024-04-14T15:12:19.059287+0000","last_fullsized":"2024-04-14T15:12:19.059287+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:22.896926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:22.896926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:30.962446+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"72'15","reported_seq":62,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.526440+0000","last_change":"2024-04-14T15:11:57.121276+0000","last_active":"2024-04-14T15:12:19.526440+0000","last_peered":"2024-04-14T15:12:19.526440+0000","last_clean":"2024-04-14T15:12:19.526440+0000","last_became_active":"2024-04-14T15:11:57.120945+0000","last_became_peered":"2024-04-14T15:11:57.120945+0000","last_unstale":"2024-04-14T15:12:19.526440+0000","last_undegraded":"2024-04-14T15:12:19.526440+0000","last_fullsized":"2024-04-14T15:12:19.526440+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-04-14T15:11:27.622904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:27.622904+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:53.046182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":72,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:18.790339+0000","last_change":"2024-04-14T15:09:49.059264+0000","last_active":"2024-04-14T15:12:18.790339+0000","last_peered":"2024-04-14T15:12:18.790339+0000","last_clean":"2024-04-14T15:12:18.790339+0000","last_became_active":"2024-04-14T15:09:49.058904+0000","last_became_peered":"2024-04-14T15:09:49.058904+0000","last_unstale":"2024-04-14T15:12:18.790339+0000","last_undegraded":"2024-04-14T15:12:18.790339+0000","last_fullsized":"2024-04-14T15:12:18.790339+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:20.571091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":84,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:18.791621+0000","last_change":"2024-04-14T15:11:27.058062+0000","last_active":"2024-04-14T15:12:18.791621+0000","last_peered":"2024-04-14T15:12:18.791621+0000","last_clean":"2024-04-14T15:12:18.791621+0000","last_became_active":"2024-04-14T15:09:47.041486+0000","last_became_peered":"2024-04-14T15:09:47.041486+0000","last_unstale":"2024-04-14T15:12:18.791621+0000","last_undegraded":"2024-04-14T15:12:18.791621+0000","last_fullsized":"2024-04-14T15:12:18.791621+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:27.057962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:27.057962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:31.266985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":18,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.059294+0000","last_change":"2024-04-14T15:11:57.122422+0000","last_active":"2024-04-14T15:12:19.059294+0000","last_peered":"2024-04-14T15:12:19.059294+0000","last_clean":"2024-04-14T15:12:19.059294+0000","last_became_active":"2024-04-14T15:11:57.122120+0000","last_became_peered":"2024-04-14T15:11:57.122120+0000","last_unstale":"2024-04-14T15:12:19.059294+0000","last_undegraded":"2024-04-14T15:12:19.059294+0000","last_fullsized":"2024-04-14T15:12:19.059294+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:43.589625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:43.589625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:58.565475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"72'9","reported_seq":104,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:18.790161+0000","last_change":"2024-04-14T15:11:25.742331+0000","last_active":"2024-04-14T15:12:18.790161+0000","last_peered":"2024-04-14T15:12:18.790161+0000","last_clean":"2024-04-14T15:12:18.790161+0000","last_became_active":"2024-04-14T15:09:45.032079+0000","last_became_peered":"2024-04-14T15:09:45.032079+0000","last_unstale":"2024-04-14T15:12:18.790161+0000","last_undegraded":"2024-04-14T15:12:18.790161+0000","last_fullsized":"2024-04-14T15:12:18.790161+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-04-14T15:11:25.742244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:25.742244+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:49.471602+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":80,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.615429+0000","last_change":"2024-04-14T15:11:03.745753+0000","last_active":"2024-04-14T15:12:19.615429+0000","last_peered":"2024-04-14T15:12:19.615429+0000","last_clean":"2024-04-14T15:12:19.615429+0000","last_became_active":"2024-04-14T15:09:49.069590+0000","last_became_peered":"2024-04-14T15:09:49.069590+0000","last_unstale":"2024-04-14T15:12:19.615429+0000","last_undegraded":"2024-04-14T15:12:19.615429+0000","last_fullsized":"2024-04-14T15:12:19.615429+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:03.745644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:03.745644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:29.376857+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":84,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.615076+0000","last_change":"2024-04-14T15:11:04.881923+0000","last_active":"2024-04-14T15:12:19.615076+0000","last_peered":"2024-04-14T15:12:19.615076+0000","last_clean":"2024-04-14T15:12:19.615076+0000","last_became_active":"2024-04-14T15:09:47.046517+0000","last_became_peered":"2024-04-14T15:09:47.046517+0000","last_unstale":"2024-04-14T15:12:19.615076+0000","last_undegraded":"2024-04-14T15:12:19.615076+0000","last_fullsized":"2024-04-14T15:12:19.615076+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:04.881846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:04.881846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:24.065968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":92,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.616284+0000","last_change":"2024-04-14T15:10:43.084925+0000","last_active":"2024-04-14T15:12:19.616284+0000","last_peered":"2024-04-14T15:12:19.616284+0000","last_clean":"2024-04-14T15:12:19.616284+0000","last_became_active":"2024-04-14T15:09:43.021986+0000","last_became_peered":"2024-04-14T15:09:43.021986+0000","last_unstale":"2024-04-14T15:12:19.616284+0000","last_undegraded":"2024-04-14T15:12:19.616284+0000","last_fullsized":"2024-04-14T15:12:19.616284+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:43.084820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:10:43.084820+0000","last_clean_scrub_stamp":"2024-04-14T15:10:43.084820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:20.010329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"87'7","reported_seq":100,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:18.790421+0000","last_change":"2024-04-14T15:11:10.510929+0000","last_active":"2024-04-14T15:12:18.790421+0000","last_peered":"2024-04-14T15:12:18.790421+0000","last_clean":"2024-04-14T15:12:18.790421+0000","last_became_active":"2024-04-14T15:09:45.028667+0000","last_became_peered":"2024-04-14T15:09:45.028667+0000","last_unstale":"2024-04-14T15:12:18.790421+0000","last_undegraded":"2024-04-14T15:12:18.790421+0000","last_fullsized":"2024-04-14T15:12:18.790421+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'7","last_scrub_stamp":"2024-04-14T15:11:10.510785+0000","last_deep_scrub":"87'7","last_deep_scrub_stamp":"2024-04-14T15:11:10.510785+0000","last_clean_scrub_stamp":"2024-04-14T15:11:10.510785+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":80,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.058809+0000","last_change":"2024-04-14T15:10:54.765733+0000","last_active":"2024-04-14T15:12:19.058809+0000","last_peered":"2024-04-14T15:12:19.058809+0000","last_clean":"2024-04-14T15:12:19.058809+0000","last_became_active":"2024-04-14T15:09:49.061756+0000","last_became_peered":"2024-04-14T15:09:49.061756+0000","last_unstale":"2024-04-14T15:12:19.058809+0000","last_undegraded":"2024-04-14T15:12:19.058809+0000","last_fullsized":"2024-04-14T15:12:19.058809+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:54.765650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:10:54.765650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:22.053079+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":84,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.617019+0000","last_change":"2024-04-14T15:11:27.645381+0000","last_active":"2024-04-14T15:12:19.617019+0000","last_peered":"2024-04-14T15:12:19.617019+0000","last_clean":"2024-04-14T15:12:19.617019+0000","last_became_active":"2024-04-14T15:09:47.042384+0000","last_became_peered":"2024-04-14T15:09:47.042384+0000","last_unstale":"2024-04-14T15:12:19.617019+0000","last_undegraded":"2024-04-14T15:12:19.617019+0000","last_fullsized":"2024-04-14T15:12:19.617019+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:27.645307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:27.645307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:31.416607+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":69,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:18.790428+0000","last_change":"2024-04-14T15:11:45.791911+0000","last_active":"2024-04-14T15:12:18.790428+0000","last_peered":"2024-04-14T15:12:18.790428+0000","last_clean":"2024-04-14T15:12:18.790428+0000","last_became_active":"2024-04-14T15:10:07.764086+0000","last_became_peered":"2024-04-14T15:10:07.764086+0000","last_unstale":"2024-04-14T15:12:18.790428+0000","last_undegraded":"2024-04-14T15:12:18.790428+0000","last_fullsized":"2024-04-14T15:12:18.790428+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:45.791810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:45.791810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:03.789308+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"72'9","reported_seq":104,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:18.790264+0000","last_change":"2024-04-14T15:11:17.667793+0000","last_active":"2024-04-14T15:12:18.790264+0000","last_peered":"2024-04-14T15:12:18.790264+0000","last_clean":"2024-04-14T15:12:18.790264+0000","last_became_active":"2024-04-14T15:09:45.030093+0000","last_became_peered":"2024-04-14T15:09:45.030093+0000","last_unstale":"2024-04-14T15:12:18.790264+0000","last_undegraded":"2024-04-14T15:12:18.790264+0000","last_fullsized":"2024-04-14T15:12:18.790264+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-04-14T15:11:17.667682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:17.667682+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:31.898486+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.128054+0000","last_change":"2024-04-14T15:11:57.118519+0000","last_active":"2024-04-14T15:12:19.128054+0000","last_peered":"2024-04-14T15:12:19.128054+0000","last_clean":"2024-04-14T15:12:19.128054+0000","last_became_active":"2024-04-14T15:11:57.118128+0000","last_became_peered":"2024-04-14T15:11:57.118128+0000","last_unstale":"2024-04-14T15:12:19.128054+0000","last_undegraded":"2024-04-14T15:12:19.128054+0000","last_fullsized":"2024-04-14T15:12:19.128054+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:21.463285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:21.463285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:48.369458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":84,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:18.790772+0000","last_change":"2024-04-14T15:10:48.466668+0000","last_active":"2024-04-14T15:12:18.790772+0000","last_peered":"2024-04-14T15:12:18.790772+0000","last_clean":"2024-04-14T15:12:18.790772+0000","last_became_active":"2024-04-14T15:09:47.048045+0000","last_became_peered":"2024-04-14T15:09:47.048045+0000","last_unstale":"2024-04-14T15:12:18.790772+0000","last_undegraded":"2024-04-14T15:12:18.790772+0000","last_fullsized":"2024-04-14T15:12:18.790772+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:48.466556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:10:48.466556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:20.571111+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":18,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.616131+0000","last_change":"2024-04-14T15:11:57.118279+0000","last_active":"2024-04-14T15:12:19.616131+0000","last_peered":"2024-04-14T15:12:19.616131+0000","last_clean":"2024-04-14T15:12:19.616131+0000","last_became_active":"2024-04-14T15:11:57.117958+0000","last_became_peered":"2024-04-14T15:11:57.117958+0000","last_unstale":"2024-04-14T15:12:19.616131+0000","last_undegraded":"2024-04-14T15:12:19.616131+0000","last_fullsized":"2024-04-14T15:12:19.616131+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:22.476087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:22.476087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:41.933418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"72'12","reported_seq":60,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:18.793462+0000","last_change":"2024-04-14T15:11:57.119014+0000","last_active":"2024-04-14T15:12:18.793462+0000","last_peered":"2024-04-14T15:12:18.793462+0000","last_clean":"2024-04-14T15:12:18.793462+0000","last_became_active":"2024-04-14T15:11:57.118631+0000","last_became_peered":"2024-04-14T15:11:57.118631+0000","last_unstale":"2024-04-14T15:12:18.793462+0000","last_undegraded":"2024-04-14T15:12:18.793462+0000","last_fullsized":"2024-04-14T15:12:18.793462+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'12","last_scrub_stamp":"2024-04-14T15:11:14.461100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:14.461100+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:29.078742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":80,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:18.791098+0000","last_change":"2024-04-14T15:11:04.539877+0000","last_active":"2024-04-14T15:12:18.791098+0000","last_peered":"2024-04-14T15:12:18.791098+0000","last_clean":"2024-04-14T15:12:18.791098+0000","last_became_active":"2024-04-14T15:09:49.127979+0000","last_became_peered":"2024-04-14T15:09:49.127979+0000","last_unstale":"2024-04-14T15:12:18.791098+0000","last_undegraded":"2024-04-14T15:12:18.791098+0000","last_fullsized":"2024-04-14T15:12:18.791098+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:04.539789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:04.539789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":84,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:18.793397+0000","last_change":"2024-04-14T15:11:16.297521+0000","last_active":"2024-04-14T15:12:18.793397+0000","last_peered":"2024-04-14T15:12:18.793397+0000","last_clean":"2024-04-14T15:12:18.793397+0000","last_became_active":"2024-04-14T15:09:47.042733+0000","last_became_peered":"2024-04-14T15:09:47.042733+0000","last_unstale":"2024-04-14T15:12:18.793397+0000","last_undegraded":"2024-04-14T15:12:18.793397+0000","last_fullsized":"2024-04-14T15:12:18.793397+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:16.297432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:16.297432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:20.307091+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":92,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.127704+0000","last_change":"2024-04-14T15:11:19.182322+0000","last_active":"2024-04-14T15:12:19.127704+0000","last_peered":"2024-04-14T15:12:19.127704+0000","last_clean":"2024-04-14T15:12:19.127704+0000","last_became_active":"2024-04-14T15:09:43.021494+0000","last_became_peered":"2024-04-14T15:09:43.021494+0000","last_unstale":"2024-04-14T15:12:19.127704+0000","last_undegraded":"2024-04-14T15:12:19.127704+0000","last_fullsized":"2024-04-14T15:12:19.127704+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:19.182163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:19.182163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:36.022685+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"72'15","reported_seq":113,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.526751+0000","last_change":"2024-04-14T15:11:18.790209+0000","last_active":"2024-04-14T15:12:19.526751+0000","last_peered":"2024-04-14T15:12:19.526751+0000","last_clean":"2024-04-14T15:12:19.526751+0000","last_became_active":"2024-04-14T15:09:45.032887+0000","last_became_peered":"2024-04-14T15:09:45.032887+0000","last_unstale":"2024-04-14T15:12:19.526751+0000","last_undegraded":"2024-04-14T15:12:19.526751+0000","last_fullsized":"2024-04-14T15:12:19.526751+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-04-14T15:11:18.790117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:18.790117+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:48.354430+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":80,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:18.793264+0000","last_change":"2024-04-14T15:11:21.247809+0000","last_active":"2024-04-14T15:12:18.793264+0000","last_peered":"2024-04-14T15:12:18.793264+0000","last_clean":"2024-04-14T15:12:18.793264+0000","last_became_active":"2024-04-14T15:09:49.057545+0000","last_became_peered":"2024-04-14T15:09:49.057545+0000","last_unstale":"2024-04-14T15:12:18.793264+0000","last_undegraded":"2024-04-14T15:12:18.793264+0000","last_fullsized":"2024-04-14T15:12:18.793264+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:21.247695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:21.247695+0000","last_clean_scrub_stamp":"2024-04-14T15:11:21.247695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:45.860565+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":84,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.059358+0000","last_change":"2024-04-14T15:11:26.852424+0000","last_active":"2024-04-14T15:12:19.059358+0000","last_peered":"2024-04-14T15:12:19.059358+0000","last_clean":"2024-04-14T15:12:19.059358+0000","last_became_active":"2024-04-14T15:09:47.045730+0000","last_became_peered":"2024-04-14T15:09:47.045730+0000","last_unstale":"2024-04-14T15:12:19.059358+0000","last_undegraded":"2024-04-14T15:12:19.059358+0000","last_fullsized":"2024-04-14T15:12:19.059358+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:26.852293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:26.852293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:35.208466+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"65'2","reported_seq":27,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.127412+0000","last_change":"2024-04-14T15:11:57.118531+0000","last_active":"2024-04-14T15:12:19.127412+0000","last_peered":"2024-04-14T15:12:19.127412+0000","last_clean":"2024-04-14T15:12:19.127412+0000","last_became_active":"2024-04-14T15:11:57.118202+0000","last_became_peered":"2024-04-14T15:11:57.118202+0000","last_unstale":"2024-04-14T15:12:19.127412+0000","last_undegraded":"2024-04-14T15:12:19.127412+0000","last_fullsized":"2024-04-14T15:12:19.127412+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-04-14T15:10:57.406478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:10:57.406478+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:23.099745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"72'11","reported_seq":125,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.128106+0000","last_change":"2024-04-14T15:11:27.102639+0000","last_active":"2024-04-14T15:12:19.128106+0000","last_peered":"2024-04-14T15:12:19.128106+0000","last_clean":"2024-04-14T15:12:19.128106+0000","last_became_active":"2024-04-14T15:10:07.764183+0000","last_became_peered":"2024-04-14T15:10:07.764183+0000","last_unstale":"2024-04-14T15:12:19.128106+0000","last_undegraded":"2024-04-14T15:12:19.128106+0000","last_fullsized":"2024-04-14T15:12:19.128106+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-04-14T15:11:27.102549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:27.102549+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:47.191755+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"72'1","reported_seq":81,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.616487+0000","last_change":"2024-04-14T15:10:59.889342+0000","last_active":"2024-04-14T15:12:19.616487+0000","last_peered":"2024-04-14T15:12:19.616487+0000","last_clean":"2024-04-14T15:12:19.616487+0000","last_became_active":"2024-04-14T15:09:49.062051+0000","last_became_peered":"2024-04-14T15:09:49.062051+0000","last_unstale":"2024-04-14T15:12:19.616487+0000","last_undegraded":"2024-04-14T15:12:19.616487+0000","last_fullsized":"2024-04-14T15:12:19.616487+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-04-14T15:10:59.889236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:10:59.889236+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:20.010349+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"72'8","reported_seq":128,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.127342+0000","last_change":"2024-04-14T15:10:58.268612+0000","last_active":"2024-04-14T15:12:19.127342+0000","last_peered":"2024-04-14T15:12:19.127342+0000","last_clean":"2024-04-14T15:12:19.127342+0000","last_became_active":"2024-04-14T15:09:47.048216+0000","last_became_peered":"2024-04-14T15:09:47.048216+0000","last_unstale":"2024-04-14T15:12:19.127342+0000","last_undegraded":"2024-04-14T15:12:19.127342+0000","last_fullsized":"2024-04-14T15:12:19.127342+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:10:58.268488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:10:58.268488+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:23.099727+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":100,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:22.784691+0000","last_change":"2024-04-14T15:12:22.784729+0000","last_active":"2024-04-14T15:12:22.784691+0000","last_peered":"2024-04-14T15:12:22.784691+0000","last_clean":"2024-04-14T15:12:22.784691+0000","last_became_active":"2024-04-14T15:09:43.023257+0000","last_became_peered":"2024-04-14T15:09:43.023257+0000","last_unstale":"2024-04-14T15:12:22.784691+0000","last_undegraded":"2024-04-14T15:12:22.784691+0000","last_fullsized":"2024-04-14T15:12:22.784691+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:22.784606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:12:22.784606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:27.049003+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"72'10","reported_seq":103,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.059354+0000","last_change":"2024-04-14T15:11:17.863747+0000","last_active":"2024-04-14T15:12:19.059354+0000","last_peered":"2024-04-14T15:12:19.059354+0000","last_clean":"2024-04-14T15:12:19.059354+0000","last_became_active":"2024-04-14T15:09:45.181456+0000","last_became_peered":"2024-04-14T15:09:45.181456+0000","last_unstale":"2024-04-14T15:12:19.059354+0000","last_undegraded":"2024-04-14T15:12:19.059354+0000","last_fullsized":"2024-04-14T15:12:19.059354+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-04-14T15:11:17.863645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:17.863645+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:27.587257+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.058761+0000","last_change":"2024-04-14T15:11:57.122684+0000","last_active":"2024-04-14T15:12:19.058761+0000","last_peered":"2024-04-14T15:12:19.058761+0000","last_clean":"2024-04-14T15:12:19.058761+0000","last_became_active":"2024-04-14T15:11:57.122266+0000","last_became_peered":"2024-04-14T15:11:57.122266+0000","last_unstale":"2024-04-14T15:12:19.058761+0000","last_undegraded":"2024-04-14T15:12:19.058761+0000","last_fullsized":"2024-04-14T15:12:19.058761+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:17.499708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:17.499708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:22.796876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":84,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.615064+0000","last_change":"2024-04-14T15:11:05.917054+0000","last_active":"2024-04-14T15:12:19.615064+0000","last_peered":"2024-04-14T15:12:19.615064+0000","last_clean":"2024-04-14T15:12:19.615064+0000","last_became_active":"2024-04-14T15:09:47.044665+0000","last_became_peered":"2024-04-14T15:09:47.044665+0000","last_unstale":"2024-04-14T15:12:19.615064+0000","last_undegraded":"2024-04-14T15:12:19.615064+0000","last_fullsized":"2024-04-14T15:12:19.615064+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:05.916966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:05.916966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:35.330112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":92,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.128177+0000","last_change":"2024-04-14T15:10:54.239518+0000","last_active":"2024-04-14T15:12:19.128177+0000","last_peered":"2024-04-14T15:12:19.128177+0000","last_clean":"2024-04-14T15:12:19.128177+0000","last_became_active":"2024-04-14T15:09:43.023220+0000","last_became_peered":"2024-04-14T15:09:43.023220+0000","last_unstale":"2024-04-14T15:12:19.128177+0000","last_undegraded":"2024-04-14T15:12:19.128177+0000","last_fullsized":"2024-04-14T15:12:19.128177+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:54.239409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:10:54.239409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:23.099763+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"72'11","reported_seq":56,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.615541+0000","last_change":"2024-04-14T15:11:57.118379+0000","last_active":"2024-04-14T15:12:19.615541+0000","last_peered":"2024-04-14T15:12:19.615541+0000","last_clean":"2024-04-14T15:12:19.615541+0000","last_became_active":"2024-04-14T15:11:57.118033+0000","last_became_peered":"2024-04-14T15:11:57.118033+0000","last_unstale":"2024-04-14T15:12:19.615541+0000","last_undegraded":"2024-04-14T15:12:19.615541+0000","last_fullsized":"2024-04-14T15:12:19.615541+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-04-14T15:11:06.492677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:06.492677+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:20.010369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":80,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.615762+0000","last_change":"2024-04-14T15:11:07.901838+0000","last_active":"2024-04-14T15:12:19.615762+0000","last_peered":"2024-04-14T15:12:19.615762+0000","last_clean":"2024-04-14T15:12:19.615762+0000","last_became_active":"2024-04-14T15:09:49.061359+0000","last_became_peered":"2024-04-14T15:09:49.061359+0000","last_unstale":"2024-04-14T15:12:19.615762+0000","last_undegraded":"2024-04-14T15:12:19.615762+0000","last_fullsized":"2024-04-14T15:12:19.615762+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:07.901761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:07.901761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:35.316850+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":84,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.127262+0000","last_change":"2024-04-14T15:10:50.229008+0000","last_active":"2024-04-14T15:12:19.127262+0000","last_peered":"2024-04-14T15:12:19.127262+0000","last_clean":"2024-04-14T15:12:19.127262+0000","last_became_active":"2024-04-14T15:09:47.041074+0000","last_became_peered":"2024-04-14T15:09:47.041074+0000","last_unstale":"2024-04-14T15:12:19.127262+0000","last_undegraded":"2024-04-14T15:12:19.127262+0000","last_fullsized":"2024-04-14T15:12:19.127262+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:50.228903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:10:50.228903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:23.099779+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":92,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.616704+0000","last_change":"2024-04-14T15:11:08.712429+0000","last_active":"2024-04-14T15:12:19.616704+0000","last_peered":"2024-04-14T15:12:19.616704+0000","last_clean":"2024-04-14T15:12:19.616704+0000","last_became_active":"2024-04-14T15:09:43.026786+0000","last_became_peered":"2024-04-14T15:09:43.026786+0000","last_unstale":"2024-04-14T15:12:19.616704+0000","last_undegraded":"2024-04-14T15:12:19.616704+0000","last_fullsized":"2024-04-14T15:12:19.616704+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:08.712289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:08.712289+0000","last_clean_scrub_stamp":"2024-04-14T15:11:08.712289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:22.765251+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"72'4","reported_seq":94,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:21.453369+0000","last_change":"2024-04-14T15:12:21.453394+0000","last_active":"2024-04-14T15:12:21.453369+0000","last_peered":"2024-04-14T15:12:21.453369+0000","last_clean":"2024-04-14T15:12:21.453369+0000","last_became_active":"2024-04-14T15:09:45.038815+0000","last_became_peered":"2024-04-14T15:09:45.038815+0000","last_unstale":"2024-04-14T15:12:21.453369+0000","last_undegraded":"2024-04-14T15:12:21.453369+0000","last_fullsized":"2024-04-14T15:12:21.453369+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4","last_scrub_stamp":"2024-04-14T15:12:21.453301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:12:21.453301+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:30.663485+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.058792+0000","last_change":"2024-04-14T15:11:57.118649+0000","last_active":"2024-04-14T15:12:19.058792+0000","last_peered":"2024-04-14T15:12:19.058792+0000","last_clean":"2024-04-14T15:12:19.058792+0000","last_became_active":"2024-04-14T15:11:57.118363+0000","last_became_peered":"2024-04-14T15:11:57.118363+0000","last_unstale":"2024-04-14T15:12:19.058792+0000","last_undegraded":"2024-04-14T15:12:19.058792+0000","last_fullsized":"2024-04-14T15:12:19.058792+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:37.585236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:37.585236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:39.774953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":84,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:18.792283+0000","last_change":"2024-04-14T15:11:35.007456+0000","last_active":"2024-04-14T15:12:18.792283+0000","last_peered":"2024-04-14T15:12:18.792283+0000","last_clean":"2024-04-14T15:12:18.792283+0000","last_became_active":"2024-04-14T15:09:47.041683+0000","last_became_peered":"2024-04-14T15:09:47.041683+0000","last_unstale":"2024-04-14T15:12:18.792283+0000","last_undegraded":"2024-04-14T15:12:18.792283+0000","last_fullsized":"2024-04-14T15:12:18.792283+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:35.007228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:35.007228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:46.618105+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":92,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:18.793371+0000","last_change":"2024-04-14T15:11:33.055306+0000","last_active":"2024-04-14T15:12:18.793371+0000","last_peered":"2024-04-14T15:12:18.793371+0000","last_clean":"2024-04-14T15:12:18.793371+0000","last_became_active":"2024-04-14T15:09:43.023583+0000","last_became_peered":"2024-04-14T15:09:43.023583+0000","last_unstale":"2024-04-14T15:12:18.793371+0000","last_undegraded":"2024-04-14T15:12:18.793371+0000","last_fullsized":"2024-04-14T15:12:18.793371+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:33.055225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:33.055225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:34.922264+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"72'11","reported_seq":107,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:18.790502+0000","last_change":"2024-04-14T15:11:48.717808+0000","last_active":"2024-04-14T15:12:18.790502+0000","last_peered":"2024-04-14T15:12:18.790502+0000","last_clean":"2024-04-14T15:12:18.790502+0000","last_became_active":"2024-04-14T15:09:45.026065+0000","last_became_peered":"2024-04-14T15:09:45.026065+0000","last_unstale":"2024-04-14T15:12:18.790502+0000","last_undegraded":"2024-04-14T15:12:18.790502+0000","last_fullsized":"2024-04-14T15:12:18.790502+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-04-14T15:11:48.717697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:48.717697+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:02.580348+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":80,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.058427+0000","last_change":"2024-04-14T15:10:56.820210+0000","last_active":"2024-04-14T15:12:19.058427+0000","last_peered":"2024-04-14T15:12:19.058427+0000","last_clean":"2024-04-14T15:12:19.058427+0000","last_became_active":"2024-04-14T15:09:49.059711+0000","last_became_peered":"2024-04-14T15:09:49.059711+0000","last_unstale":"2024-04-14T15:12:19.058427+0000","last_undegraded":"2024-04-14T15:12:19.058427+0000","last_fullsized":"2024-04-14T15:12:19.058427+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:56.820123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:10:56.820123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:22.053174+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":18,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.058940+0000","last_change":"2024-04-14T15:11:57.122543+0000","last_active":"2024-04-14T15:12:19.058940+0000","last_peered":"2024-04-14T15:12:19.058940+0000","last_clean":"2024-04-14T15:12:19.058940+0000","last_became_active":"2024-04-14T15:11:57.121871+0000","last_became_peered":"2024-04-14T15:11:57.121871+0000","last_unstale":"2024-04-14T15:12:19.058940+0000","last_undegraded":"2024-04-14T15:12:19.058940+0000","last_fullsized":"2024-04-14T15:12:19.058940+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:10.455300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:10.455300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:22.053238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":92,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.127808+0000","last_change":"2024-04-14T15:11:03.304640+0000","last_active":"2024-04-14T15:12:19.127808+0000","last_peered":"2024-04-14T15:12:19.127808+0000","last_clean":"2024-04-14T15:12:19.127808+0000","last_became_active":"2024-04-14T15:09:43.022480+0000","last_became_peered":"2024-04-14T15:09:43.022480+0000","last_unstale":"2024-04-14T15:12:19.127808+0000","last_undegraded":"2024-04-14T15:12:19.127808+0000","last_fullsized":"2024-04-14T15:12:19.127808+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:03.304541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:03.304541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:22.825961+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"72'9","reported_seq":104,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:18.792196+0000","last_change":"2024-04-14T15:11:29.021013+0000","last_active":"2024-04-14T15:12:18.792196+0000","last_peered":"2024-04-14T15:12:18.792196+0000","last_clean":"2024-04-14T15:12:18.792196+0000","last_became_active":"2024-04-14T15:09:45.026363+0000","last_became_peered":"2024-04-14T15:09:45.026363+0000","last_unstale":"2024-04-14T15:12:18.792196+0000","last_undegraded":"2024-04-14T15:12:18.792196+0000","last_fullsized":"2024-04-14T15:12:18.792196+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-04-14T15:11:29.020859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:29.020859+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:30.854130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":80,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:18.791161+0000","last_change":"2024-04-14T15:11:15.644792+0000","last_active":"2024-04-14T15:12:18.791161+0000","last_peered":"2024-04-14T15:12:18.791161+0000","last_clean":"2024-04-14T15:12:18.791161+0000","last_became_active":"2024-04-14T15:09:49.065242+0000","last_became_peered":"2024-04-14T15:09:49.065242+0000","last_unstale":"2024-04-14T15:12:18.791161+0000","last_undegraded":"2024-04-14T15:12:18.791161+0000","last_fullsized":"2024-04-14T15:12:18.791161+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:15.644714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:15.644714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:19.718378+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"98'9","reported_seq":52,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.617024+0000","last_change":"2024-04-14T15:11:57.124620+0000","last_active":"2024-04-14T15:12:19.617024+0000","last_peered":"2024-04-14T15:12:19.617024+0000","last_clean":"2024-04-14T15:12:19.617024+0000","last_became_active":"2024-04-14T15:11:57.124325+0000","last_became_peered":"2024-04-14T15:11:57.124325+0000","last_unstale":"2024-04-14T15:12:19.617024+0000","last_undegraded":"2024-04-14T15:12:19.617024+0000","last_fullsized":"2024-04-14T15:12:19.617024+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:11:09.485146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:09.485146+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":92,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.059079+0000","last_change":"2024-04-14T15:11:10.762163+0000","last_active":"2024-04-14T15:12:19.059079+0000","last_peered":"2024-04-14T15:12:19.059079+0000","last_clean":"2024-04-14T15:12:19.059079+0000","last_became_active":"2024-04-14T15:09:43.019301+0000","last_became_peered":"2024-04-14T15:09:43.019301+0000","last_unstale":"2024-04-14T15:12:19.059079+0000","last_undegraded":"2024-04-14T15:12:19.059079+0000","last_fullsized":"2024-04-14T15:12:19.059079+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:10.762080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:10.762080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:22.053210+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"72'9","reported_seq":104,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.616461+0000","last_change":"2024-04-14T15:10:56.858475+0000","last_active":"2024-04-14T15:12:19.616461+0000","last_peered":"2024-04-14T15:12:19.616461+0000","last_clean":"2024-04-14T15:12:19.616461+0000","last_became_active":"2024-04-14T15:09:45.178857+0000","last_became_peered":"2024-04-14T15:09:45.178857+0000","last_unstale":"2024-04-14T15:12:19.616461+0000","last_undegraded":"2024-04-14T15:12:19.616461+0000","last_fullsized":"2024-04-14T15:12:19.616461+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-04-14T15:10:56.858323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:10:56.858323+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:21.806401+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":80,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.059075+0000","last_change":"2024-04-14T15:11:40.922318+0000","last_active":"2024-04-14T15:12:19.059075+0000","last_peered":"2024-04-14T15:12:19.059075+0000","last_clean":"2024-04-14T15:12:19.059075+0000","last_became_active":"2024-04-14T15:09:49.066068+0000","last_became_peered":"2024-04-14T15:09:49.066068+0000","last_unstale":"2024-04-14T15:12:19.059075+0000","last_undegraded":"2024-04-14T15:12:19.059075+0000","last_fullsized":"2024-04-14T15:12:19.059075+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:40.922231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:40.922231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:59.161159+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"72'8","reported_seq":125,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.615286+0000","last_change":"2024-04-14T15:11:18.667210+0000","last_active":"2024-04-14T15:12:19.615286+0000","last_peered":"2024-04-14T15:12:19.615286+0000","last_clean":"2024-04-14T15:12:19.615286+0000","last_became_active":"2024-04-14T15:09:47.036688+0000","last_became_peered":"2024-04-14T15:09:47.036688+0000","last_unstale":"2024-04-14T15:12:19.615286+0000","last_undegraded":"2024-04-14T15:12:19.615286+0000","last_fullsized":"2024-04-14T15:12:19.615286+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:11:18.667049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:18.667049+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:36.115547+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":92,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.616313+0000","last_change":"2024-04-14T15:11:24.634784+0000","last_active":"2024-04-14T15:12:19.616313+0000","last_peered":"2024-04-14T15:12:19.616313+0000","last_clean":"2024-04-14T15:12:19.616313+0000","last_became_active":"2024-04-14T15:09:43.027776+0000","last_became_peered":"2024-04-14T15:09:43.027776+0000","last_unstale":"2024-04-14T15:12:19.616313+0000","last_undegraded":"2024-04-14T15:12:19.616313+0000","last_fullsized":"2024-04-14T15:12:19.616313+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:24.634676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:24.634676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:48.431623+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"72'10","reported_seq":52,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:18.792838+0000","last_change":"2024-04-14T15:11:57.126842+0000","last_active":"2024-04-14T15:12:18.792838+0000","last_peered":"2024-04-14T15:12:18.792838+0000","last_clean":"2024-04-14T15:12:18.792838+0000","last_became_active":"2024-04-14T15:11:57.126453+0000","last_became_peered":"2024-04-14T15:11:57.126453+0000","last_unstale":"2024-04-14T15:12:18.792838+0000","last_undegraded":"2024-04-14T15:12:18.792838+0000","last_fullsized":"2024-04-14T15:12:18.792838+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-04-14T15:11:11.458594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:11.458594+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:21.001376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":80,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.615959+0000","last_change":"2024-04-14T15:11:26.912839+0000","last_active":"2024-04-14T15:12:19.615959+0000","last_peered":"2024-04-14T15:12:19.615959+0000","last_clean":"2024-04-14T15:12:19.615959+0000","last_became_active":"2024-04-14T15:09:49.130438+0000","last_became_peered":"2024-04-14T15:09:49.130438+0000","last_unstale":"2024-04-14T15:12:19.615959+0000","last_undegraded":"2024-04-14T15:12:19.615959+0000","last_fullsized":"2024-04-14T15:12:19.615959+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:26.912753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:26.912753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:54.042078+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":84,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.615658+0000","last_change":"2024-04-14T15:10:53.900979+0000","last_active":"2024-04-14T15:12:19.615658+0000","last_peered":"2024-04-14T15:12:19.615658+0000","last_clean":"2024-04-14T15:12:19.615658+0000","last_became_active":"2024-04-14T15:09:47.045349+0000","last_became_peered":"2024-04-14T15:09:47.045349+0000","last_unstale":"2024-04-14T15:12:19.615658+0000","last_undegraded":"2024-04-14T15:12:19.615658+0000","last_fullsized":"2024-04-14T15:12:19.615658+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:53.900878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:10:53.900878+0000","last_clean_scrub_stamp":"2024-04-14T15:10:53.900878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:22.765272+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":92,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.127493+0000","last_change":"2024-04-14T15:11:18.194434+0000","last_active":"2024-04-14T15:12:19.127493+0000","last_peered":"2024-04-14T15:12:19.127493+0000","last_clean":"2024-04-14T15:12:19.127493+0000","last_became_active":"2024-04-14T15:09:43.018987+0000","last_became_peered":"2024-04-14T15:09:43.018987+0000","last_unstale":"2024-04-14T15:12:19.127493+0000","last_undegraded":"2024-04-14T15:12:19.127493+0000","last_fullsized":"2024-04-14T15:12:19.127493+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:18.194339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:18.194339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:27.683203+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"72'6","reported_seq":97,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:18.793280+0000","last_change":"2024-04-14T15:11:26.575575+0000","last_active":"2024-04-14T15:12:18.793280+0000","last_peered":"2024-04-14T15:12:18.793280+0000","last_clean":"2024-04-14T15:12:18.793280+0000","last_became_active":"2024-04-14T15:09:45.180037+0000","last_became_peered":"2024-04-14T15:09:45.180037+0000","last_unstale":"2024-04-14T15:12:18.793280+0000","last_undegraded":"2024-04-14T15:12:18.793280+0000","last_fullsized":"2024-04-14T15:12:18.793280+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'6","last_scrub_stamp":"2024-04-14T15:11:26.575440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:26.575440+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:27.111076+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":80,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.526579+0000","last_change":"2024-04-14T15:10:56.886642+0000","last_active":"2024-04-14T15:12:19.526579+0000","last_peered":"2024-04-14T15:12:19.526579+0000","last_clean":"2024-04-14T15:12:19.526579+0000","last_became_active":"2024-04-14T15:09:49.064296+0000","last_became_peered":"2024-04-14T15:09:49.064296+0000","last_unstale":"2024-04-14T15:12:19.526579+0000","last_undegraded":"2024-04-14T15:12:19.526579+0000","last_fullsized":"2024-04-14T15:12:19.526579+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:56.886513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:10:56.886513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":18,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:18.793867+0000","last_change":"2024-04-14T15:11:57.119781+0000","last_active":"2024-04-14T15:12:18.793867+0000","last_peered":"2024-04-14T15:12:18.793867+0000","last_clean":"2024-04-14T15:12:18.793867+0000","last_became_active":"2024-04-14T15:11:57.119468+0000","last_became_peered":"2024-04-14T15:11:57.119468+0000","last_unstale":"2024-04-14T15:12:18.793867+0000","last_undegraded":"2024-04-14T15:12:18.793867+0000","last_fullsized":"2024-04-14T15:12:18.793867+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:07.485821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:07.485821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:25.390628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":92,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.526411+0000","last_change":"2024-04-14T15:10:43.844823+0000","last_active":"2024-04-14T15:12:19.526411+0000","last_peered":"2024-04-14T15:12:19.526411+0000","last_clean":"2024-04-14T15:12:19.526411+0000","last_became_active":"2024-04-14T15:09:43.016788+0000","last_became_peered":"2024-04-14T15:09:43.016788+0000","last_unstale":"2024-04-14T15:12:19.526411+0000","last_undegraded":"2024-04-14T15:12:19.526411+0000","last_fullsized":"2024-04-14T15:12:19.526411+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:43.844693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:10:43.844693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:20.522926+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"72'9","reported_seq":104,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.059289+0000","last_change":"2024-04-14T15:11:39.966261+0000","last_active":"2024-04-14T15:12:19.059289+0000","last_peered":"2024-04-14T15:12:19.059289+0000","last_clean":"2024-04-14T15:12:19.059289+0000","last_became_active":"2024-04-14T15:09:45.180375+0000","last_became_peered":"2024-04-14T15:09:45.180375+0000","last_unstale":"2024-04-14T15:12:19.059289+0000","last_undegraded":"2024-04-14T15:12:19.059289+0000","last_fullsized":"2024-04-14T15:12:19.059289+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-04-14T15:11:39.966171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:39.966171+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:49.895272+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.615898+0000","last_change":"2024-04-14T15:11:57.119085+0000","last_active":"2024-04-14T15:12:19.615898+0000","last_peered":"2024-04-14T15:12:19.615898+0000","last_clean":"2024-04-14T15:12:19.615898+0000","last_became_active":"2024-04-14T15:11:57.118796+0000","last_became_peered":"2024-04-14T15:11:57.118796+0000","last_unstale":"2024-04-14T15:12:19.615898+0000","last_undegraded":"2024-04-14T15:12:19.615898+0000","last_fullsized":"2024-04-14T15:12:19.615898+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:20.010309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"65'3","reported_seq":110,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:18.791092+0000","last_change":"2024-04-14T15:11:38.827544+0000","last_active":"2024-04-14T15:12:18.791092+0000","last_peered":"2024-04-14T15:12:18.791092+0000","last_clean":"2024-04-14T15:12:18.791092+0000","last_became_active":"2024-04-14T15:09:43.020490+0000","last_became_peered":"2024-04-14T15:09:43.020490+0000","last_unstale":"2024-04-14T15:12:18.791092+0000","last_undegraded":"2024-04-14T15:12:18.791092+0000","last_fullsized":"2024-04-14T15:12:18.791092+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3","last_scrub_stamp":"2024-04-14T15:11:38.827423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:38.827423+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:51.153401+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":528,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"72'12","reported_seq":111,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:18.790319+0000","last_change":"2024-04-14T15:10:53.448340+0000","last_active":"2024-04-14T15:12:18.790319+0000","last_peered":"2024-04-14T15:12:18.790319+0000","last_clean":"2024-04-14T15:12:18.790319+0000","last_became_active":"2024-04-14T15:09:45.032892+0000","last_became_peered":"2024-04-14T15:09:45.032892+0000","last_unstale":"2024-04-14T15:12:18.790319+0000","last_undegraded":"2024-04-14T15:12:18.790319+0000","last_fullsized":"2024-04-14T15:12:18.790319+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'12","last_scrub_stamp":"2024-04-14T15:10:53.448218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:10:53.448218+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:20.571151+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":84,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.526209+0000","last_change":"2024-04-14T15:11:12.867617+0000","last_active":"2024-04-14T15:12:19.526209+0000","last_peered":"2024-04-14T15:12:19.526209+0000","last_clean":"2024-04-14T15:12:19.526209+0000","last_became_active":"2024-04-14T15:09:47.049005+0000","last_became_peered":"2024-04-14T15:09:47.049005+0000","last_unstale":"2024-04-14T15:12:19.526209+0000","last_undegraded":"2024-04-14T15:12:19.526209+0000","last_fullsized":"2024-04-14T15:12:19.526209+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:12.867489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:12.867489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:42.053319+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":80,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.058550+0000","last_change":"2024-04-14T15:11:09.756246+0000","last_active":"2024-04-14T15:12:19.058550+0000","last_peered":"2024-04-14T15:12:19.058550+0000","last_clean":"2024-04-14T15:12:19.058550+0000","last_became_active":"2024-04-14T15:09:49.066648+0000","last_became_peered":"2024-04-14T15:09:49.066648+0000","last_unstale":"2024-04-14T15:12:19.058550+0000","last_undegraded":"2024-04-14T15:12:19.058550+0000","last_fullsized":"2024-04-14T15:12:19.058550+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:09.756164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:09.756164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:32.890427+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":92,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:18.793080+0000","last_change":"2024-04-14T15:11:42.065911+0000","last_active":"2024-04-14T15:12:18.793080+0000","last_peered":"2024-04-14T15:12:18.793080+0000","last_clean":"2024-04-14T15:12:18.793080+0000","last_became_active":"2024-04-14T15:09:43.022597+0000","last_became_peered":"2024-04-14T15:09:43.022597+0000","last_unstale":"2024-04-14T15:12:18.793080+0000","last_undegraded":"2024-04-14T15:12:18.793080+0000","last_fullsized":"2024-04-14T15:12:18.793080+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:42.065812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:42.065812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:03.153303+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"72'10","reported_seq":82,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.526494+0000","last_change":"2024-04-14T15:11:17.779528+0000","last_active":"2024-04-14T15:12:19.526494+0000","last_peered":"2024-04-14T15:12:19.526494+0000","last_clean":"2024-04-14T15:12:19.526494+0000","last_became_active":"2024-04-14T15:10:07.764042+0000","last_became_peered":"2024-04-14T15:10:07.764042+0000","last_unstale":"2024-04-14T15:12:19.526494+0000","last_undegraded":"2024-04-14T15:12:19.526494+0000","last_fullsized":"2024-04-14T15:12:19.526494+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-04-14T15:11:17.779383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:17.779383+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:33.747601+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"72'8","reported_seq":133,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:20.089824+0000","last_change":"2024-04-14T15:12:20.089824+0000","last_active":"2024-04-14T15:12:20.089824+0000","last_peered":"2024-04-14T15:12:20.089824+0000","last_clean":"2024-04-14T15:12:20.089824+0000","last_became_active":"2024-04-14T15:09:47.047699+0000","last_became_peered":"2024-04-14T15:09:47.047699+0000","last_unstale":"2024-04-14T15:12:20.089824+0000","last_undegraded":"2024-04-14T15:12:20.089824+0000","last_fullsized":"2024-04-14T15:12:20.089824+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:12:20.089708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:12:20.089708+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:27.759583+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:18.790784+0000","last_change":"2024-04-14T15:11:57.118237+0000","last_active":"2024-04-14T15:12:18.790784+0000","last_peered":"2024-04-14T15:12:18.790784+0000","last_clean":"2024-04-14T15:12:18.790784+0000","last_became_active":"2024-04-14T15:11:57.117900+0000","last_became_peered":"2024-04-14T15:11:57.117900+0000","last_unstale":"2024-04-14T15:12:18.790784+0000","last_undegraded":"2024-04-14T15:12:18.790784+0000","last_fullsized":"2024-04-14T15:12:18.790784+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:20.571074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":92,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.616418+0000","last_change":"2024-04-14T15:11:20.694315+0000","last_active":"2024-04-14T15:12:19.616418+0000","last_peered":"2024-04-14T15:12:19.616418+0000","last_clean":"2024-04-14T15:12:19.616418+0000","last_became_active":"2024-04-14T15:09:43.028362+0000","last_became_peered":"2024-04-14T15:09:43.028362+0000","last_unstale":"2024-04-14T15:12:19.616418+0000","last_undegraded":"2024-04-14T15:12:19.616418+0000","last_fullsized":"2024-04-14T15:12:19.616418+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:20.694225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:20.694225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:45.303474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"72'28","reported_seq":169,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:18.793032+0000","last_change":"2024-04-14T15:11:24.126821+0000","last_active":"2024-04-14T15:12:18.793032+0000","last_peered":"2024-04-14T15:12:18.793032+0000","last_clean":"2024-04-14T15:12:18.793032+0000","last_became_active":"2024-04-14T15:09:45.034168+0000","last_became_peered":"2024-04-14T15:09:45.034168+0000","last_unstale":"2024-04-14T15:12:18.793032+0000","last_undegraded":"2024-04-14T15:12:18.793032+0000","last_fullsized":"2024-04-14T15:12:18.793032+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'28","last_scrub_stamp":"2024-04-14T15:11:24.126715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:24.126715+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:40.303628+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":84,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.058368+0000","last_change":"2024-04-14T15:11:15.831911+0000","last_active":"2024-04-14T15:12:19.058368+0000","last_peered":"2024-04-14T15:12:19.058368+0000","last_clean":"2024-04-14T15:12:19.058368+0000","last_became_active":"2024-04-14T15:09:47.040185+0000","last_became_peered":"2024-04-14T15:09:47.040185+0000","last_unstale":"2024-04-14T15:12:19.058368+0000","last_undegraded":"2024-04-14T15:12:19.058368+0000","last_fullsized":"2024-04-14T15:12:19.058368+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:15.831779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:15.831779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:44.452408+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":80,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.616940+0000","last_change":"2024-04-14T15:11:00.939052+0000","last_active":"2024-04-14T15:12:19.616940+0000","last_peered":"2024-04-14T15:12:19.616940+0000","last_clean":"2024-04-14T15:12:19.616940+0000","last_became_active":"2024-04-14T15:09:49.130307+0000","last_became_peered":"2024-04-14T15:09:49.130307+0000","last_unstale":"2024-04-14T15:12:19.616940+0000","last_undegraded":"2024-04-14T15:12:19.616940+0000","last_fullsized":"2024-04-14T15:12:19.616940+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:00.938917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:00.938917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:23.003251+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"73'5","reported_seq":257,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:22.880214+0000","last_change":"2024-04-14T15:11:03.192957+0000","last_active":"2024-04-14T15:12:22.880214+0000","last_peered":"2024-04-14T15:12:22.880214+0000","last_clean":"2024-04-14T15:12:22.880214+0000","last_became_active":"2024-04-14T15:09:43.016544+0000","last_became_peered":"2024-04-14T15:09:43.016544+0000","last_unstale":"2024-04-14T15:12:22.880214+0000","last_undegraded":"2024-04-14T15:12:22.880214+0000","last_fullsized":"2024-04-14T15:12:22.880214+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-04-14T15:11:03.192810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:03.192810+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00056542099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":155,"num_read_kb":150,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":92,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.526348+0000","last_change":"2024-04-14T15:10:42.819764+0000","last_active":"2024-04-14T15:12:19.526348+0000","last_peered":"2024-04-14T15:12:19.526348+0000","last_clean":"2024-04-14T15:12:19.526348+0000","last_became_active":"2024-04-14T15:09:43.021188+0000","last_became_peered":"2024-04-14T15:09:43.021188+0000","last_unstale":"2024-04-14T15:12:19.526348+0000","last_undegraded":"2024-04-14T15:12:19.526348+0000","last_fullsized":"2024-04-14T15:12:19.526348+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:42.819681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:10:42.819681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:20.522945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"72'13","reported_seq":115,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:18.793743+0000","last_change":"2024-04-14T15:11:31.082411+0000","last_active":"2024-04-14T15:12:18.793743+0000","last_peered":"2024-04-14T15:12:18.793743+0000","last_clean":"2024-04-14T15:12:18.793743+0000","last_became_active":"2024-04-14T15:09:45.033883+0000","last_became_peered":"2024-04-14T15:09:45.033883+0000","last_unstale":"2024-04-14T15:12:18.793743+0000","last_undegraded":"2024-04-14T15:12:18.793743+0000","last_fullsized":"2024-04-14T15:12:18.793743+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'13","last_scrub_stamp":"2024-04-14T15:11:31.082324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:31.082324+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:39.988990+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":84,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.526680+0000","last_change":"2024-04-14T15:10:55.841920+0000","last_active":"2024-04-14T15:12:19.526680+0000","last_peered":"2024-04-14T15:12:19.526680+0000","last_clean":"2024-04-14T15:12:19.526680+0000","last_became_active":"2024-04-14T15:09:47.049457+0000","last_became_peered":"2024-04-14T15:09:47.049457+0000","last_unstale":"2024-04-14T15:12:19.526680+0000","last_undegraded":"2024-04-14T15:12:19.526680+0000","last_fullsized":"2024-04-14T15:12:19.526680+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:55.841832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:10:55.841832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:20.522980+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":80,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.616908+0000","last_change":"2024-04-14T15:11:32.672340+0000","last_active":"2024-04-14T15:12:19.616908+0000","last_peered":"2024-04-14T15:12:19.616908+0000","last_clean":"2024-04-14T15:12:19.616908+0000","last_became_active":"2024-04-14T15:09:49.067031+0000","last_became_peered":"2024-04-14T15:09:49.067031+0000","last_unstale":"2024-04-14T15:12:19.616908+0000","last_undegraded":"2024-04-14T15:12:19.616908+0000","last_fullsized":"2024-04-14T15:12:19.616908+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:32.672255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:32.672255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:53.163772+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"69'2","reported_seq":43,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.616980+0000","last_change":"2024-04-14T15:11:57.122075+0000","last_active":"2024-04-14T15:12:19.616980+0000","last_peered":"2024-04-14T15:12:19.616980+0000","last_clean":"2024-04-14T15:12:19.616980+0000","last_became_active":"2024-04-14T15:11:57.121809+0000","last_became_peered":"2024-04-14T15:11:57.121809+0000","last_unstale":"2024-04-14T15:12:19.616980+0000","last_undegraded":"2024-04-14T15:12:19.616980+0000","last_fullsized":"2024-04-14T15:12:19.616980+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-04-14T15:11:20.500271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:20.500271+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:48.187402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":100,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.616685+0000","last_change":"2024-04-14T15:11:47.627424+0000","last_active":"2024-04-14T15:12:19.616685+0000","last_peered":"2024-04-14T15:12:19.616685+0000","last_clean":"2024-04-14T15:12:19.616685+0000","last_became_active":"2024-04-14T15:09:43.025421+0000","last_became_peered":"2024-04-14T15:09:43.025421+0000","last_unstale":"2024-04-14T15:12:19.616685+0000","last_undegraded":"2024-04-14T15:12:19.616685+0000","last_fullsized":"2024-04-14T15:12:19.616685+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:47.627305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:47.627305+0000","last_clean_scrub_stamp":"2024-04-14T15:11:47.627305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:06.227750+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"91'130","reported_seq":343,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.615547+0000","last_change":"2024-04-14T15:11:28.875662+0000","last_active":"2024-04-14T15:12:19.615547+0000","last_peered":"2024-04-14T15:12:19.615547+0000","last_clean":"2024-04-14T15:12:19.615547+0000","last_became_active":"2024-04-14T15:09:33.753828+0000","last_became_peered":"2024-04-14T15:09:33.753828+0000","last_unstale":"2024-04-14T15:12:19.615547+0000","last_undegraded":"2024-04-14T15:12:19.615547+0000","last_fullsized":"2024-04-14T15:12:19.615547+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:11:28.875570+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:51.231957+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"90'24","reported_seq":151,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.128040+0000","last_change":"2024-04-14T15:11:25.120788+0000","last_active":"2024-04-14T15:12:19.128040+0000","last_peered":"2024-04-14T15:12:19.128040+0000","last_clean":"2024-04-14T15:12:19.128040+0000","last_became_active":"2024-04-14T15:09:45.179978+0000","last_became_peered":"2024-04-14T15:09:45.179978+0000","last_unstale":"2024-04-14T15:12:19.128040+0000","last_undegraded":"2024-04-14T15:12:19.128040+0000","last_fullsized":"2024-04-14T15:12:19.128040+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'24","last_scrub_stamp":"2024-04-14T15:11:25.120692+0000","last_deep_scrub":"90'24","last_deep_scrub_stamp":"2024-04-14T15:11:25.120692+0000","last_clean_scrub_stamp":"2024-04-14T15:11:25.120692+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:53.531902+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":84,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.616059+0000","last_change":"2024-04-14T15:11:13.892949+0000","last_active":"2024-04-14T15:12:19.616059+0000","last_peered":"2024-04-14T15:12:19.616059+0000","last_clean":"2024-04-14T15:12:19.616059+0000","last_became_active":"2024-04-14T15:09:47.047201+0000","last_became_peered":"2024-04-14T15:09:47.047201+0000","last_unstale":"2024-04-14T15:12:19.616059+0000","last_undegraded":"2024-04-14T15:12:19.616059+0000","last_fullsized":"2024-04-14T15:12:19.616059+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:13.892863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:13.892863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:41.896681+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"72'1","reported_seq":81,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:18.792663+0000","last_change":"2024-04-14T15:11:25.095270+0000","last_active":"2024-04-14T15:12:18.792663+0000","last_peered":"2024-04-14T15:12:18.792663+0000","last_clean":"2024-04-14T15:12:18.792663+0000","last_became_active":"2024-04-14T15:09:49.065243+0000","last_became_peered":"2024-04-14T15:09:49.065243+0000","last_unstale":"2024-04-14T15:12:18.792663+0000","last_undegraded":"2024-04-14T15:12:18.792663+0000","last_fullsized":"2024-04-14T15:12:18.792663+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-04-14T15:11:25.095182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:25.095182+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:45.977272+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":99,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.058863+0000","last_change":"2024-04-14T15:11:07.696609+0000","last_active":"2024-04-14T15:12:19.058863+0000","last_peered":"2024-04-14T15:12:19.058863+0000","last_clean":"2024-04-14T15:12:19.058863+0000","last_became_active":"2024-04-14T15:09:43.018509+0000","last_became_peered":"2024-04-14T15:09:43.018509+0000","last_unstale":"2024-04-14T15:12:19.058863+0000","last_undegraded":"2024-04-14T15:12:19.058863+0000","last_fullsized":"2024-04-14T15:12:19.058863+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:07.696525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:07.696525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:24.057779+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0018592680000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":92,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.615430+0000","last_change":"2024-04-14T15:10:49.916989+0000","last_active":"2024-04-14T15:12:19.615430+0000","last_peered":"2024-04-14T15:12:19.615430+0000","last_clean":"2024-04-14T15:12:19.615430+0000","last_became_active":"2024-04-14T15:09:43.025095+0000","last_became_peered":"2024-04-14T15:09:43.025095+0000","last_unstale":"2024-04-14T15:12:19.615430+0000","last_undegraded":"2024-04-14T15:12:19.615430+0000","last_fullsized":"2024-04-14T15:12:19.615430+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:49.916884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:10:49.916884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"72'12","reported_seq":114,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:22.083797+0000","last_change":"2024-04-14T15:12:22.083844+0000","last_active":"2024-04-14T15:12:22.083797+0000","last_peered":"2024-04-14T15:12:22.083797+0000","last_clean":"2024-04-14T15:12:22.083797+0000","last_became_active":"2024-04-14T15:09:45.037231+0000","last_became_peered":"2024-04-14T15:09:45.037231+0000","last_unstale":"2024-04-14T15:12:22.083797+0000","last_undegraded":"2024-04-14T15:12:22.083797+0000","last_fullsized":"2024-04-14T15:12:22.083797+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'12","last_scrub_stamp":"2024-04-14T15:12:22.083718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:12:22.083718+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:41.128151+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":84,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.615423+0000","last_change":"2024-04-14T15:11:26.662123+0000","last_active":"2024-04-14T15:12:19.615423+0000","last_peered":"2024-04-14T15:12:19.615423+0000","last_clean":"2024-04-14T15:12:19.615423+0000","last_became_active":"2024-04-14T15:09:47.036696+0000","last_became_peered":"2024-04-14T15:09:47.036696+0000","last_unstale":"2024-04-14T15:12:19.615423+0000","last_undegraded":"2024-04-14T15:12:19.615423+0000","last_fullsized":"2024-04-14T15:12:19.615423+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:26.662032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:26.662032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:28.937693+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":80,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:18.791528+0000","last_change":"2024-04-14T15:11:35.993757+0000","last_active":"2024-04-14T15:12:18.791528+0000","last_peered":"2024-04-14T15:12:18.791528+0000","last_clean":"2024-04-14T15:12:18.791528+0000","last_became_active":"2024-04-14T15:09:49.128136+0000","last_became_peered":"2024-04-14T15:09:49.128136+0000","last_unstale":"2024-04-14T15:12:18.791528+0000","last_undegraded":"2024-04-14T15:12:18.791528+0000","last_fullsized":"2024-04-14T15:12:18.791528+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:35.993681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:35.993681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:46.320717+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":92,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.616600+0000","last_change":"2024-04-14T15:10:53.992688+0000","last_active":"2024-04-14T15:12:19.616600+0000","last_peered":"2024-04-14T15:12:19.616600+0000","last_clean":"2024-04-14T15:12:19.616600+0000","last_became_active":"2024-04-14T15:09:43.021006+0000","last_became_peered":"2024-04-14T15:09:43.021006+0000","last_unstale":"2024-04-14T15:12:19.616600+0000","last_undegraded":"2024-04-14T15:12:19.616600+0000","last_fullsized":"2024-04-14T15:12:19.616600+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:53.992590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:10:53.992590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:23.003229+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"72'19","reported_seq":124,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.059089+0000","last_change":"2024-04-14T15:10:47.880742+0000","last_active":"2024-04-14T15:12:19.059089+0000","last_peered":"2024-04-14T15:12:19.059089+0000","last_clean":"2024-04-14T15:12:19.059089+0000","last_became_active":"2024-04-14T15:09:45.179182+0000","last_became_peered":"2024-04-14T15:09:45.179182+0000","last_unstale":"2024-04-14T15:12:19.059089+0000","last_undegraded":"2024-04-14T15:12:19.059089+0000","last_fullsized":"2024-04-14T15:12:19.059089+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'19","last_scrub_stamp":"2024-04-14T15:10:47.880623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:10:47.880623+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:22.053146+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":84,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.127801+0000","last_change":"2024-04-14T15:10:59.286474+0000","last_active":"2024-04-14T15:12:19.127801+0000","last_peered":"2024-04-14T15:12:19.127801+0000","last_clean":"2024-04-14T15:12:19.127801+0000","last_became_active":"2024-04-14T15:09:47.048427+0000","last_became_peered":"2024-04-14T15:09:47.048427+0000","last_unstale":"2024-04-14T15:12:19.127801+0000","last_undegraded":"2024-04-14T15:12:19.127801+0000","last_fullsized":"2024-04-14T15:12:19.127801+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:59.286374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:10:59.286374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:23.099706+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":80,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:18.790568+0000","last_change":"2024-04-14T15:11:41.838377+0000","last_active":"2024-04-14T15:12:18.790568+0000","last_peered":"2024-04-14T15:12:18.790568+0000","last_clean":"2024-04-14T15:12:18.790568+0000","last_became_active":"2024-04-14T15:09:49.065684+0000","last_became_peered":"2024-04-14T15:09:49.065684+0000","last_unstale":"2024-04-14T15:12:18.790568+0000","last_undegraded":"2024-04-14T15:12:18.790568+0000","last_fullsized":"2024-04-14T15:12:18.790568+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:41.838273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:41.838273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:50.252756+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"91'2","reported_seq":94,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:18.793818+0000","last_change":"2024-04-14T15:11:37.974863+0000","last_active":"2024-04-14T15:12:18.793818+0000","last_peered":"2024-04-14T15:12:18.793818+0000","last_clean":"2024-04-14T15:12:18.793818+0000","last_became_active":"2024-04-14T15:09:43.018478+0000","last_became_peered":"2024-04-14T15:09:43.018478+0000","last_unstale":"2024-04-14T15:12:18.793818+0000","last_undegraded":"2024-04-14T15:12:18.793818+0000","last_fullsized":"2024-04-14T15:12:18.793818+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'2","last_scrub_stamp":"2024-04-14T15:11:37.974770+0000","last_deep_scrub":"91'2","last_deep_scrub_stamp":"2024-04-14T15:11:37.974770+0000","last_clean_scrub_stamp":"2024-04-14T15:11:37.974770+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:05.657376+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"72'18","reported_seq":69,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.615829+0000","last_change":"2024-04-14T15:11:57.119345+0000","last_active":"2024-04-14T15:12:19.615829+0000","last_peered":"2024-04-14T15:12:19.615829+0000","last_clean":"2024-04-14T15:12:19.615829+0000","last_became_active":"2024-04-14T15:11:57.118744+0000","last_became_peered":"2024-04-14T15:11:57.118744+0000","last_unstale":"2024-04-14T15:12:19.615829+0000","last_undegraded":"2024-04-14T15:12:19.615829+0000","last_fullsized":"2024-04-14T15:12:19.615829+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'18","last_scrub_stamp":"2024-04-14T15:10:53.457087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:10:53.457087+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:21.316436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":33,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":84,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:18.790327+0000","last_change":"2024-04-14T15:11:36.850301+0000","last_active":"2024-04-14T15:12:18.790327+0000","last_peered":"2024-04-14T15:12:18.790327+0000","last_clean":"2024-04-14T15:12:18.790327+0000","last_became_active":"2024-04-14T15:09:47.044610+0000","last_became_peered":"2024-04-14T15:09:47.044610+0000","last_unstale":"2024-04-14T15:12:18.790327+0000","last_undegraded":"2024-04-14T15:12:18.790327+0000","last_fullsized":"2024-04-14T15:12:18.790327+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:36.850204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:36.850204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:46.499366+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":80,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.616303+0000","last_change":"2024-04-14T15:11:16.821265+0000","last_active":"2024-04-14T15:12:19.616303+0000","last_peered":"2024-04-14T15:12:19.616303+0000","last_clean":"2024-04-14T15:12:19.616303+0000","last_became_active":"2024-04-14T15:09:49.129610+0000","last_became_peered":"2024-04-14T15:09:49.129610+0000","last_unstale":"2024-04-14T15:12:19.616303+0000","last_undegraded":"2024-04-14T15:12:19.616303+0000","last_fullsized":"2024-04-14T15:12:19.616303+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:16.821140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:16.821140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:32.897119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":92,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.128143+0000","last_change":"2024-04-14T15:11:20.219646+0000","last_active":"2024-04-14T15:12:19.128143+0000","last_peered":"2024-04-14T15:12:19.128143+0000","last_clean":"2024-04-14T15:12:19.128143+0000","last_became_active":"2024-04-14T15:09:43.020503+0000","last_became_peered":"2024-04-14T15:09:43.020503+0000","last_unstale":"2024-04-14T15:12:19.128143+0000","last_undegraded":"2024-04-14T15:12:19.128143+0000","last_fullsized":"2024-04-14T15:12:19.128143+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:20.219543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:20.219543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:39.250425+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"72'14","reported_seq":109,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:18.790155+0000","last_change":"2024-04-14T15:11:14.657178+0000","last_active":"2024-04-14T15:12:18.790155+0000","last_peered":"2024-04-14T15:12:18.790155+0000","last_clean":"2024-04-14T15:12:18.790155+0000","last_became_active":"2024-04-14T15:09:45.025673+0000","last_became_peered":"2024-04-14T15:09:45.025673+0000","last_unstale":"2024-04-14T15:12:18.790155+0000","last_undegraded":"2024-04-14T15:12:18.790155+0000","last_fullsized":"2024-04-14T15:12:18.790155+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'14","last_scrub_stamp":"2024-04-14T15:11:14.657075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:14.657075+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:37.152537+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"98'9","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:18.791221+0000","last_change":"2024-04-14T15:11:57.118444+0000","last_active":"2024-04-14T15:12:18.791221+0000","last_peered":"2024-04-14T15:12:18.791221+0000","last_clean":"2024-04-14T15:12:18.791221+0000","last_became_active":"2024-04-14T15:11:57.118068+0000","last_became_peered":"2024-04-14T15:11:57.118068+0000","last_unstale":"2024-04-14T15:12:18.791221+0000","last_undegraded":"2024-04-14T15:12:18.791221+0000","last_fullsized":"2024-04-14T15:12:18.791221+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:11:40.562108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:40.562108+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:41.606283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.127743+0000","last_change":"2024-04-14T15:11:57.121216+0000","last_active":"2024-04-14T15:12:19.127743+0000","last_peered":"2024-04-14T15:12:19.127743+0000","last_clean":"2024-04-14T15:12:19.127743+0000","last_became_active":"2024-04-14T15:11:57.120861+0000","last_became_peered":"2024-04-14T15:11:57.120861+0000","last_unstale":"2024-04-14T15:12:19.127743+0000","last_undegraded":"2024-04-14T15:12:19.127743+0000","last_fullsized":"2024-04-14T15:12:19.127743+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:01.501784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:01.501784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":84,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.615562+0000","last_change":"2024-04-14T15:11:02.763497+0000","last_active":"2024-04-14T15:12:19.615562+0000","last_peered":"2024-04-14T15:12:19.615562+0000","last_clean":"2024-04-14T15:12:19.615562+0000","last_became_active":"2024-04-14T15:09:47.043021+0000","last_became_peered":"2024-04-14T15:09:47.043021+0000","last_unstale":"2024-04-14T15:12:19.615562+0000","last_undegraded":"2024-04-14T15:12:19.615562+0000","last_fullsized":"2024-04-14T15:12:19.615562+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:02.763399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:02.763399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:21.806381+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":92,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.615546+0000","last_change":"2024-04-14T15:10:48.939920+0000","last_active":"2024-04-14T15:12:19.615546+0000","last_peered":"2024-04-14T15:12:19.615546+0000","last_clean":"2024-04-14T15:12:19.615546+0000","last_became_active":"2024-04-14T15:09:43.018601+0000","last_became_peered":"2024-04-14T15:09:43.018601+0000","last_unstale":"2024-04-14T15:12:19.615546+0000","last_undegraded":"2024-04-14T15:12:19.615546+0000","last_fullsized":"2024-04-14T15:12:19.615546+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:48.939790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:10:48.939790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:20.010406+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"85'16","reported_seq":112,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:18.791129+0000","last_change":"2024-04-14T15:10:47.456301+0000","last_active":"2024-04-14T15:12:18.791129+0000","last_peered":"2024-04-14T15:12:18.791129+0000","last_clean":"2024-04-14T15:12:18.791129+0000","last_became_active":"2024-04-14T15:09:45.028212+0000","last_became_peered":"2024-04-14T15:09:45.028212+0000","last_unstale":"2024-04-14T15:12:18.791129+0000","last_undegraded":"2024-04-14T15:12:18.791129+0000","last_fullsized":"2024-04-14T15:12:18.791129+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-14T15:10:47.456169+0000","last_deep_scrub":"85'16","last_deep_scrub_stamp":"2024-04-14T15:10:47.456169+0000","last_clean_scrub_stamp":"2024-04-14T15:10:47.456169+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:20.571131+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":80,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.617082+0000","last_change":"2024-04-14T15:11:25.638017+0000","last_active":"2024-04-14T15:12:19.617082+0000","last_peered":"2024-04-14T15:12:19.617082+0000","last_clean":"2024-04-14T15:12:19.617082+0000","last_became_active":"2024-04-14T15:09:49.058940+0000","last_became_peered":"2024-04-14T15:09:49.058940+0000","last_unstale":"2024-04-14T15:12:19.617082+0000","last_undegraded":"2024-04-14T15:12:19.617082+0000","last_fullsized":"2024-04-14T15:12:19.617082+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:25.637888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:25.637888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:55.011351+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"72'8","reported_seq":125,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:18.791013+0000","last_change":"2024-04-14T15:11:29.730498+0000","last_active":"2024-04-14T15:12:18.791013+0000","last_peered":"2024-04-14T15:12:18.791013+0000","last_clean":"2024-04-14T15:12:18.791013+0000","last_became_active":"2024-04-14T15:09:47.050962+0000","last_became_peered":"2024-04-14T15:09:47.050962+0000","last_unstale":"2024-04-14T15:12:18.791013+0000","last_undegraded":"2024-04-14T15:12:18.791013+0000","last_fullsized":"2024-04-14T15:12:18.791013+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:11:29.730407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:29.730407+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:53.855034+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":92,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:18.791772+0000","last_change":"2024-04-14T15:11:11.285681+0000","last_active":"2024-04-14T15:12:18.791772+0000","last_peered":"2024-04-14T15:12:18.791772+0000","last_clean":"2024-04-14T15:12:18.791772+0000","last_became_active":"2024-04-14T15:09:43.020878+0000","last_became_peered":"2024-04-14T15:09:43.020878+0000","last_unstale":"2024-04-14T15:12:18.791772+0000","last_undegraded":"2024-04-14T15:12:18.791772+0000","last_fullsized":"2024-04-14T15:12:18.791772+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:11.285588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:11.285588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"72'15","reported_seq":113,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:18.793914+0000","last_change":"2024-04-14T15:11:22.219265+0000","last_active":"2024-04-14T15:12:18.793914+0000","last_peered":"2024-04-14T15:12:18.793914+0000","last_clean":"2024-04-14T15:12:18.793914+0000","last_became_active":"2024-04-14T15:09:45.033331+0000","last_became_peered":"2024-04-14T15:09:45.033331+0000","last_unstale":"2024-04-14T15:12:18.793914+0000","last_undegraded":"2024-04-14T15:12:18.793914+0000","last_fullsized":"2024-04-14T15:12:18.793914+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-04-14T15:11:22.219174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:22.219174+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:23.840704+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":72,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:18.790877+0000","last_change":"2024-04-14T15:09:49.065592+0000","last_active":"2024-04-14T15:12:18.790877+0000","last_peered":"2024-04-14T15:12:18.790877+0000","last_clean":"2024-04-14T15:12:18.790877+0000","last_became_active":"2024-04-14T15:09:49.065274+0000","last_became_peered":"2024-04-14T15:09:49.065274+0000","last_unstale":"2024-04-14T15:12:18.790877+0000","last_undegraded":"2024-04-14T15:12:18.790877+0000","last_fullsized":"2024-04-14T15:12:18.790877+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:09:48.039252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:20.571055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"72'8","reported_seq":125,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.526702+0000","last_change":"2024-04-14T15:11:21.827356+0000","last_active":"2024-04-14T15:12:19.526702+0000","last_peered":"2024-04-14T15:12:19.526702+0000","last_clean":"2024-04-14T15:12:19.526702+0000","last_became_active":"2024-04-14T15:09:47.044085+0000","last_became_peered":"2024-04-14T15:09:47.044085+0000","last_unstale":"2024-04-14T15:12:19.526702+0000","last_undegraded":"2024-04-14T15:12:19.526702+0000","last_fullsized":"2024-04-14T15:12:19.526702+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:11:21.827272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:21.827272+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:40.383654+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":92,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:18.792090+0000","last_change":"2024-04-14T15:10:46.280858+0000","last_active":"2024-04-14T15:12:18.792090+0000","last_peered":"2024-04-14T15:12:18.792090+0000","last_clean":"2024-04-14T15:12:18.792090+0000","last_became_active":"2024-04-14T15:09:43.018871+0000","last_became_peered":"2024-04-14T15:09:43.018871+0000","last_unstale":"2024-04-14T15:12:18.792090+0000","last_undegraded":"2024-04-14T15:12:18.792090+0000","last_fullsized":"2024-04-14T15:12:18.792090+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:46.280773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:10:46.280773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"72'10","reported_seq":103,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:18.790637+0000","last_change":"2024-04-14T15:11:42.802009+0000","last_active":"2024-04-14T15:12:18.790637+0000","last_peered":"2024-04-14T15:12:18.790637+0000","last_clean":"2024-04-14T15:12:18.790637+0000","last_became_active":"2024-04-14T15:09:45.029170+0000","last_became_peered":"2024-04-14T15:09:45.029170+0000","last_unstale":"2024-04-14T15:12:18.790637+0000","last_undegraded":"2024-04-14T15:12:18.790637+0000","last_fullsized":"2024-04-14T15:12:18.790637+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-04-14T15:11:42.801888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:42.801888+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:09.615426+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":80,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.526909+0000","last_change":"2024-04-14T15:11:10.852797+0000","last_active":"2024-04-14T15:12:19.526909+0000","last_peered":"2024-04-14T15:12:19.526909+0000","last_clean":"2024-04-14T15:12:19.526909+0000","last_became_active":"2024-04-14T15:09:49.069594+0000","last_became_peered":"2024-04-14T15:09:49.069594+0000","last_unstale":"2024-04-14T15:12:19.526909+0000","last_undegraded":"2024-04-14T15:12:19.526909+0000","last_fullsized":"2024-04-14T15:12:19.526909+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:10.852715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:10.852715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":84,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.226794+0000","last_change":"2024-04-14T15:12:19.226834+0000","last_active":"2024-04-14T15:12:19.226794+0000","last_peered":"2024-04-14T15:12:19.226794+0000","last_clean":"2024-04-14T15:12:19.226794+0000","last_became_active":"2024-04-14T15:09:47.042810+0000","last_became_peered":"2024-04-14T15:09:47.042810+0000","last_unstale":"2024-04-14T15:12:19.226794+0000","last_undegraded":"2024-04-14T15:12:19.226794+0000","last_fullsized":"2024-04-14T15:12:19.226794+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:19.226701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:12:19.226701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:27.237894+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":92,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.615794+0000","last_change":"2024-04-14T15:11:01.903252+0000","last_active":"2024-04-14T15:12:19.615794+0000","last_peered":"2024-04-14T15:12:19.615794+0000","last_clean":"2024-04-14T15:12:19.615794+0000","last_became_active":"2024-04-14T15:09:43.022351+0000","last_became_peered":"2024-04-14T15:09:43.022351+0000","last_unstale":"2024-04-14T15:12:19.615794+0000","last_undegraded":"2024-04-14T15:12:19.615794+0000","last_fullsized":"2024-04-14T15:12:19.615794+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:01.903150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:01.903150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:27.833698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"72'17","reported_seq":140,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:18.790960+0000","last_change":"2024-04-14T15:11:09.502519+0000","last_active":"2024-04-14T15:12:18.790960+0000","last_peered":"2024-04-14T15:12:18.790960+0000","last_clean":"2024-04-14T15:12:18.790960+0000","last_became_active":"2024-04-14T15:10:07.766008+0000","last_became_peered":"2024-04-14T15:10:07.766008+0000","last_unstale":"2024-04-14T15:12:18.790960+0000","last_undegraded":"2024-04-14T15:12:18.790960+0000","last_fullsized":"2024-04-14T15:12:18.790960+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'17","last_scrub_stamp":"2024-04-14T15:11:09.502431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:09.502431+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:29.834036+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":80,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.615294+0000","last_change":"2024-04-14T15:11:15.845024+0000","last_active":"2024-04-14T15:12:19.615294+0000","last_peered":"2024-04-14T15:12:19.615294+0000","last_clean":"2024-04-14T15:12:19.615294+0000","last_became_active":"2024-04-14T15:09:49.065397+0000","last_became_peered":"2024-04-14T15:09:49.065397+0000","last_unstale":"2024-04-14T15:12:19.615294+0000","last_undegraded":"2024-04-14T15:12:19.615294+0000","last_fullsized":"2024-04-14T15:12:19.615294+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:15.844942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:15.844942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:39.797576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":84,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.526513+0000","last_change":"2024-04-14T15:10:59.793878+0000","last_active":"2024-04-14T15:12:19.526513+0000","last_peered":"2024-04-14T15:12:19.526513+0000","last_clean":"2024-04-14T15:12:19.526513+0000","last_became_active":"2024-04-14T15:09:47.045542+0000","last_became_peered":"2024-04-14T15:09:47.045542+0000","last_unstale":"2024-04-14T15:12:19.526513+0000","last_undegraded":"2024-04-14T15:12:19.526513+0000","last_fullsized":"2024-04-14T15:12:19.526513+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:59.793768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:10:59.793768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:24.997192+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":100,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:20.500151+0000","last_change":"2024-04-14T15:12:20.500190+0000","last_active":"2024-04-14T15:12:20.500151+0000","last_peered":"2024-04-14T15:12:20.500151+0000","last_clean":"2024-04-14T15:12:20.500151+0000","last_became_active":"2024-04-14T15:09:43.020268+0000","last_became_peered":"2024-04-14T15:09:43.020268+0000","last_unstale":"2024-04-14T15:12:20.500151+0000","last_undegraded":"2024-04-14T15:12:20.500151+0000","last_fullsized":"2024-04-14T15:12:20.500151+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:20.500037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:12:20.500037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:33.689835+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"72'19","reported_seq":119,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.127685+0000","last_change":"2024-04-14T15:11:09.225462+0000","last_active":"2024-04-14T15:12:19.127685+0000","last_peered":"2024-04-14T15:12:19.127685+0000","last_clean":"2024-04-14T15:12:19.127685+0000","last_became_active":"2024-04-14T15:09:45.178782+0000","last_became_peered":"2024-04-14T15:09:45.178782+0000","last_unstale":"2024-04-14T15:12:19.127685+0000","last_undegraded":"2024-04-14T15:12:19.127685+0000","last_fullsized":"2024-04-14T15:12:19.127685+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'19","last_scrub_stamp":"2024-04-14T15:11:09.225359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:09.225359+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:29.861536+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":80,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.058606+0000","last_change":"2024-04-14T15:10:53.789872+0000","last_active":"2024-04-14T15:12:19.058606+0000","last_peered":"2024-04-14T15:12:19.058606+0000","last_clean":"2024-04-14T15:12:19.058606+0000","last_became_active":"2024-04-14T15:09:49.061495+0000","last_became_peered":"2024-04-14T15:09:49.061495+0000","last_unstale":"2024-04-14T15:12:19.058606+0000","last_undegraded":"2024-04-14T15:12:19.058606+0000","last_fullsized":"2024-04-14T15:12:19.058606+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:53.789790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:10:53.789790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:22.053050+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":84,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.526209+0000","last_change":"2024-04-14T15:10:50.788471+0000","last_active":"2024-04-14T15:12:19.526209+0000","last_peered":"2024-04-14T15:12:19.526209+0000","last_clean":"2024-04-14T15:12:19.526209+0000","last_became_active":"2024-04-14T15:09:47.046153+0000","last_became_peered":"2024-04-14T15:09:47.046153+0000","last_unstale":"2024-04-14T15:12:19.526209+0000","last_undegraded":"2024-04-14T15:12:19.526209+0000","last_fullsized":"2024-04-14T15:12:19.526209+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:10:50.788350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:10:50.788350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:20.522964+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":92,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:18.793451+0000","last_change":"2024-04-14T15:11:04.223680+0000","last_active":"2024-04-14T15:12:18.793451+0000","last_peered":"2024-04-14T15:12:18.793451+0000","last_clean":"2024-04-14T15:12:18.793451+0000","last_became_active":"2024-04-14T15:09:43.020044+0000","last_became_peered":"2024-04-14T15:09:43.020044+0000","last_unstale":"2024-04-14T15:12:18.793451+0000","last_undegraded":"2024-04-14T15:12:18.793451+0000","last_fullsized":"2024-04-14T15:12:18.793451+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:04.223605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:04.223605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"72'9","reported_seq":104,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.059180+0000","last_change":"2024-04-14T15:10:50.850226+0000","last_active":"2024-04-14T15:12:19.059180+0000","last_peered":"2024-04-14T15:12:19.059180+0000","last_clean":"2024-04-14T15:12:19.059180+0000","last_became_active":"2024-04-14T15:09:45.037179+0000","last_became_peered":"2024-04-14T15:09:45.037179+0000","last_unstale":"2024-04-14T15:12:19.059180+0000","last_undegraded":"2024-04-14T15:12:19.059180+0000","last_fullsized":"2024-04-14T15:12:19.059180+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-04-14T15:10:50.850116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:10:50.850116+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:22.053111+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":80,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.616208+0000","last_change":"2024-04-14T15:11:31.682399+0000","last_active":"2024-04-14T15:12:19.616208+0000","last_peered":"2024-04-14T15:12:19.616208+0000","last_clean":"2024-04-14T15:12:19.616208+0000","last_became_active":"2024-04-14T15:09:49.128070+0000","last_became_peered":"2024-04-14T15:09:49.128070+0000","last_unstale":"2024-04-14T15:12:19.616208+0000","last_undegraded":"2024-04-14T15:12:19.616208+0000","last_fullsized":"2024-04-14T15:12:19.616208+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:31.682321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:31.682321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:52.297577+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":92,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.526810+0000","last_change":"2024-04-14T15:11:13.838572+0000","last_active":"2024-04-14T15:12:19.526810+0000","last_peered":"2024-04-14T15:12:19.526810+0000","last_clean":"2024-04-14T15:12:19.526810+0000","last_became_active":"2024-04-14T15:09:43.021460+0000","last_became_peered":"2024-04-14T15:09:43.021460+0000","last_unstale":"2024-04-14T15:12:19.526810+0000","last_undegraded":"2024-04-14T15:12:19.526810+0000","last_fullsized":"2024-04-14T15:12:19.526810+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:13.838468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:13.838468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:31.461401+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"72'15","reported_seq":113,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.615881+0000","last_change":"2024-04-14T15:11:19.680463+0000","last_active":"2024-04-14T15:12:19.615881+0000","last_peered":"2024-04-14T15:12:19.615881+0000","last_clean":"2024-04-14T15:12:19.615881+0000","last_became_active":"2024-04-14T15:09:45.180160+0000","last_became_peered":"2024-04-14T15:09:45.180160+0000","last_unstale":"2024-04-14T15:12:19.615881+0000","last_undegraded":"2024-04-14T15:12:19.615881+0000","last_fullsized":"2024-04-14T15:12:19.615881+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-04-14T15:11:19.680308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:19.680308+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:12:41.707865+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"85'9","reported_seq":126,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-14T15:12:19.615287+0000","last_change":"2024-04-14T15:10:51.950673+0000","last_active":"2024-04-14T15:12:19.615287+0000","last_peered":"2024-04-14T15:12:19.615287+0000","last_clean":"2024-04-14T15:12:19.615287+0000","last_became_active":"2024-04-14T15:09:47.045441+0000","last_became_peered":"2024-04-14T15:09:47.045441+0000","last_unstale":"2024-04-14T15:12:19.615287+0000","last_undegraded":"2024-04-14T15:12:19.615287+0000","last_fullsized":"2024-04-14T15:12:19.615287+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9","last_scrub_stamp":"2024-04-14T15:10:51.950580+0000","last_deep_scrub":"85'9","last_deep_scrub_stamp":"2024-04-14T15:10:51.950580+0000","last_clean_scrub_stamp":"2024-04-14T15:10:51.950580+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67,"ondisk_log_size":67,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":693,"num_read_kb":450,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":408,"ondisk_log_size":408,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":155,"num_read_kb":150,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103206,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28308,"kb_used_data":1288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716844,"statfs":{"total":95995035648,"available":95966048256,"internally_reserved":0,"allocated":1318912,"data_stored":935412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397545,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28304,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716848,"statfs":{"total":95995035648,"available":95966052352,"internally_reserved":0,"allocated":1314816,"data_stored":935705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574915,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31916,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713236,"statfs":{"total":95995035648,"available":95962353664,"internally_reserved":0,"allocated":757760,"data_stored":345458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738440,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32472,"kb_used_data":1296,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712680,"statfs":{"total":95995035648,"available":95961784320,"internally_reserved":0,"allocated":1327104,"data_stored":936065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411391,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31872,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713280,"statfs":{"total":95995035648,"available":95962398720,"internally_reserved":0,"allocated":708608,"data_stored":344559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182457,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27752,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717400,"statfs":{"total":95995035648,"available":95966617600,"internally_reserved":0,"allocated":753664,"data_stored":345779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986227,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27732,"kb_used_data":716,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717420,"statfs":{"total":95995035648,"available":95966638080,"internally_reserved":0,"allocated":733184,"data_stored":344940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659184,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31876,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713276,"statfs":{"total":95995035648,"available":95962394624,"internally_reserved":0,"allocated":716800,"data_stored":344394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-14T15:12:27.869 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd pool set .mgr pgp_num 1 2024-04-14T15:12:27.897 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:12:27.945 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:27.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:27.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:12:28.047 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:12:28.074 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:28.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:28.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:12:28.095 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:27 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3030676872' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:12:28.095 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:27 smithi090 ceph-mon[30399]: from='client.24734 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:12:28.095 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:27 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3030676872' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:12:28.095 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:27 smithi090 ceph-mon[41670]: from='client.24734 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:12:28.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:27 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3030676872' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:12:28.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:27 smithi092 ceph-mon[33966]: from='client.24734 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:12:28.176 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:12:28.202 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:28.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:28.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:12:28.304 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:12:28.338 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:28.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:28.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:12:28.440 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:12:28.467 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:28.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:28.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:12:28.533 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:12:28.569 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:12:28.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:28.598 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:28.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:12:28.699 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:12:28.732 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:28.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:28.733 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:12:28.834 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:12:28.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:28.863 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:28.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:12:28.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:28 smithi090 ceph-mon[41670]: pgmap v87: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:12:28.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:28 smithi090 ceph-mon[41670]: from='client.14862 v1:172.21.15.90:0/3648728731' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:12:28.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:28 smithi090 ceph-mon[41670]: 5.f scrub starts 2024-04-14T15:12:28.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:28 smithi090 ceph-mon[41670]: 5.f scrub ok 2024-04-14T15:12:28.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:28 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:12:28.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:28 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:12:28.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:28 smithi090 ceph-mon[41670]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-14T15:12:28.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:28 smithi090 ceph-mon[41670]: Cluster is now healthy 2024-04-14T15:12:28.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:28 smithi090 ceph-mon[41670]: from='client.24734 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-14T15:12:28.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:28 smithi090 ceph-mon[41670]: osdmap e102: 8 total, 8 up, 8 in 2024-04-14T15:12:28.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:28 smithi090 ceph-mon[41670]: 2.1d scrub starts 2024-04-14T15:12:28.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:28 smithi090 ceph-mon[41670]: 2.1d scrub ok 2024-04-14T15:12:28.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:28 smithi090 ceph-mon[41670]: 2.17 scrub starts 2024-04-14T15:12:28.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:28 smithi090 ceph-mon[41670]: 2.17 scrub ok 2024-04-14T15:12:28.944 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:28 smithi090 ceph-mon[30399]: pgmap v87: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:12:28.944 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:28 smithi090 ceph-mon[30399]: from='client.14862 v1:172.21.15.90:0/3648728731' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:12:28.944 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:28 smithi090 ceph-mon[30399]: 5.f scrub starts 2024-04-14T15:12:28.944 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:28 smithi090 ceph-mon[30399]: 5.f scrub ok 2024-04-14T15:12:28.944 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:28 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:12:28.944 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:28 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:12:28.944 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:28 smithi090 ceph-mon[30399]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-14T15:12:28.944 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:28 smithi090 ceph-mon[30399]: Cluster is now healthy 2024-04-14T15:12:28.944 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:28 smithi090 ceph-mon[30399]: from='client.24734 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-14T15:12:28.944 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:28 smithi090 ceph-mon[30399]: osdmap e102: 8 total, 8 up, 8 in 2024-04-14T15:12:28.944 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:28 smithi090 ceph-mon[30399]: 2.1d scrub starts 2024-04-14T15:12:28.944 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:28 smithi090 ceph-mon[30399]: 2.1d scrub ok 2024-04-14T15:12:28.944 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:28 smithi090 ceph-mon[30399]: 2.17 scrub starts 2024-04-14T15:12:28.944 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:28 smithi090 ceph-mon[30399]: 2.17 scrub ok 2024-04-14T15:12:28.964 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:12:28.992 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:28.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:28.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:12:29.094 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:12:29.117 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:29.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:29.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:12:29.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:28 smithi092 ceph-mon[33966]: pgmap v87: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:12:29.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:28 smithi092 ceph-mon[33966]: from='client.14862 v1:172.21.15.90:0/3648728731' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:12:29.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:28 smithi092 ceph-mon[33966]: 5.f scrub starts 2024-04-14T15:12:29.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:28 smithi092 ceph-mon[33966]: 5.f scrub ok 2024-04-14T15:12:29.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:28 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:12:29.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:28 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:12:29.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:28 smithi092 ceph-mon[33966]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-14T15:12:29.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:28 smithi092 ceph-mon[33966]: Cluster is now healthy 2024-04-14T15:12:29.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:28 smithi092 ceph-mon[33966]: from='client.24734 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-14T15:12:29.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:28 smithi092 ceph-mon[33966]: osdmap e102: 8 total, 8 up, 8 in 2024-04-14T15:12:29.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:28 smithi092 ceph-mon[33966]: 2.1d scrub starts 2024-04-14T15:12:29.124 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:28 smithi092 ceph-mon[33966]: 2.1d scrub ok 2024-04-14T15:12:29.124 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:28 smithi092 ceph-mon[33966]: 2.17 scrub starts 2024-04-14T15:12:29.124 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:28 smithi092 ceph-mon[33966]: 2.17 scrub ok 2024-04-14T15:12:29.220 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:12:29.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:29.255 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:29.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:12:29.357 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:12:29.384 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:29.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:29.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:12:29.486 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:12:29.513 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:29.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:29.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:12:29.615 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:12:29.642 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:29.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:29.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:12:29.744 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:12:29.777 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:29.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:29.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:12:29.879 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:12:29.907 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:29.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:29.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:12:30.009 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:12:30.036 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:30.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:30.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:12:30.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:29 smithi092 ceph-mon[33966]: 2.16 scrub starts 2024-04-14T15:12:30.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:29 smithi092 ceph-mon[33966]: 2.16 scrub ok 2024-04-14T15:12:30.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:29 smithi092 ceph-mon[33966]: 4.b scrub starts 2024-04-14T15:12:30.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:29 smithi092 ceph-mon[33966]: 4.b scrub ok 2024-04-14T15:12:30.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:29 smithi092 ceph-mon[33966]: 6.1c scrub starts 2024-04-14T15:12:30.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:29 smithi092 ceph-mon[33966]: 6.1c scrub ok 2024-04-14T15:12:30.139 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:12:30.171 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:30.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:30.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:12:30.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:29 smithi090 ceph-mon[41670]: 2.16 scrub starts 2024-04-14T15:12:30.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:29 smithi090 ceph-mon[41670]: 2.16 scrub ok 2024-04-14T15:12:30.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:29 smithi090 ceph-mon[41670]: 4.b scrub starts 2024-04-14T15:12:30.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:29 smithi090 ceph-mon[41670]: 4.b scrub ok 2024-04-14T15:12:30.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:29 smithi090 ceph-mon[41670]: 6.1c scrub starts 2024-04-14T15:12:30.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:29 smithi090 ceph-mon[41670]: 6.1c scrub ok 2024-04-14T15:12:30.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:29 smithi090 ceph-mon[30399]: 2.16 scrub starts 2024-04-14T15:12:30.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:29 smithi090 ceph-mon[30399]: 2.16 scrub ok 2024-04-14T15:12:30.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:29 smithi090 ceph-mon[30399]: 4.b scrub starts 2024-04-14T15:12:30.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:29 smithi090 ceph-mon[30399]: 4.b scrub ok 2024-04-14T15:12:30.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:29 smithi090 ceph-mon[30399]: 6.1c scrub starts 2024-04-14T15:12:30.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:29 smithi090 ceph-mon[30399]: 6.1c scrub ok 2024-04-14T15:12:30.273 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:12:30.303 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:30.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:30.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:12:30.405 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:12:30.437 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:30.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:30.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:12:30.539 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:12:30.567 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:30.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:30.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:12:30.669 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:12:30.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:30.693 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:30.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:12:30.794 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:12:30.817 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:30.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:30.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:12:30.919 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:12:30.948 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:30.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:30.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:12:31.034 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd set noscrub 2024-04-14T15:12:31.050 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:12:31.112 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:31.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:31.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:12:31.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:30 smithi092 ceph-mon[33966]: pgmap v89: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:12:31.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:30 smithi092 ceph-mon[33966]: 5.16 scrub starts 2024-04-14T15:12:31.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:30 smithi092 ceph-mon[33966]: 5.16 scrub ok 2024-04-14T15:12:31.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:30 smithi090 ceph-mon[41670]: pgmap v89: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:12:31.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:30 smithi090 ceph-mon[41670]: 5.16 scrub starts 2024-04-14T15:12:31.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:30 smithi090 ceph-mon[41670]: 5.16 scrub ok 2024-04-14T15:12:31.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:30 smithi090 ceph-mon[30399]: pgmap v89: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:12:31.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:30 smithi090 ceph-mon[30399]: 5.16 scrub starts 2024-04-14T15:12:31.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:30 smithi090 ceph-mon[30399]: 5.16 scrub ok 2024-04-14T15:12:31.214 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:12:31.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:31.243 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:31.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:12:31.344 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:12:31.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:31.368 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:31.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:12:31.469 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:12:31.496 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:31.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:31.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:12:31.598 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:12:31.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:31.627 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:31.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:12:31.729 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:12:31.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:31.756 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:31.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:12:31.857 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:12:31.881 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:31.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:31.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:12:31.983 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:12:32.006 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:32.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:32.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:12:32.108 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:12:32.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:31 smithi092 ceph-mon[33966]: 2.10 scrub starts 2024-04-14T15:12:32.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:31 smithi092 ceph-mon[33966]: 2.10 scrub ok 2024-04-14T15:12:32.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:31 smithi092 ceph-mon[33966]: 4.3 deep-scrub starts 2024-04-14T15:12:32.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:31 smithi092 ceph-mon[33966]: 4.3 deep-scrub ok 2024-04-14T15:12:32.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:31 smithi092 ceph-mon[33966]: 5.11 scrub starts 2024-04-14T15:12:32.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:31 smithi092 ceph-mon[33966]: 5.11 scrub ok 2024-04-14T15:12:32.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:31 smithi092 ceph-mon[33966]: 2.9 scrub starts 2024-04-14T15:12:32.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:31 smithi092 ceph-mon[33966]: 2.9 scrub ok 2024-04-14T15:12:32.136 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:32.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:32.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:12:32.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:31 smithi090 ceph-mon[41670]: 2.10 scrub starts 2024-04-14T15:12:32.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:31 smithi090 ceph-mon[41670]: 2.10 scrub ok 2024-04-14T15:12:32.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:31 smithi090 ceph-mon[41670]: 4.3 deep-scrub starts 2024-04-14T15:12:32.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:31 smithi090 ceph-mon[41670]: 4.3 deep-scrub ok 2024-04-14T15:12:32.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:31 smithi090 ceph-mon[41670]: 5.11 scrub starts 2024-04-14T15:12:32.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:31 smithi090 ceph-mon[41670]: 5.11 scrub ok 2024-04-14T15:12:32.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:31 smithi090 ceph-mon[41670]: 2.9 scrub starts 2024-04-14T15:12:32.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:31 smithi090 ceph-mon[41670]: 2.9 scrub ok 2024-04-14T15:12:32.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:31 smithi090 ceph-mon[30399]: 2.10 scrub starts 2024-04-14T15:12:32.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:31 smithi090 ceph-mon[30399]: 2.10 scrub ok 2024-04-14T15:12:32.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:31 smithi090 ceph-mon[30399]: 4.3 deep-scrub starts 2024-04-14T15:12:32.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:31 smithi090 ceph-mon[30399]: 4.3 deep-scrub ok 2024-04-14T15:12:32.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:31 smithi090 ceph-mon[30399]: 5.11 scrub starts 2024-04-14T15:12:32.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:31 smithi090 ceph-mon[30399]: 5.11 scrub ok 2024-04-14T15:12:32.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:31 smithi090 ceph-mon[30399]: 2.9 scrub starts 2024-04-14T15:12:32.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:31 smithi090 ceph-mon[30399]: 2.9 scrub ok 2024-04-14T15:12:32.237 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:12:32.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:32.265 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:32.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:12:32.367 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:12:32.400 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:32.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:32.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:12:32.502 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:12:32.529 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:32.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:32.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:12:32.631 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:12:32.663 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:32.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:32.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:12:32.765 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:12:32.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:32.792 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:32.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:12:32.893 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:12:32.919 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:32.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:32.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:12:33.021 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:12:33.046 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:33.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:33.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:12:33.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:32 smithi092 ceph-mon[33966]: pgmap v90: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:12:33.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:32 smithi092 ceph-mon[33966]: 6.10 scrub starts 2024-04-14T15:12:33.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:32 smithi092 ceph-mon[33966]: 6.10 scrub ok 2024-04-14T15:12:33.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:32 smithi092 ceph-mon[33966]: 5.1d scrub starts 2024-04-14T15:12:33.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:32 smithi092 ceph-mon[33966]: 5.1d scrub ok 2024-04-14T15:12:33.148 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:12:33.173 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:33.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:33.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:12:33.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:32 smithi090 ceph-mon[30399]: pgmap v90: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:12:33.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:32 smithi090 ceph-mon[30399]: 6.10 scrub starts 2024-04-14T15:12:33.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:32 smithi090 ceph-mon[30399]: 6.10 scrub ok 2024-04-14T15:12:33.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:32 smithi090 ceph-mon[30399]: 5.1d scrub starts 2024-04-14T15:12:33.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:32 smithi090 ceph-mon[30399]: 5.1d scrub ok 2024-04-14T15:12:33.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:32 smithi090 ceph-mon[41670]: pgmap v90: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:12:33.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:32 smithi090 ceph-mon[41670]: 6.10 scrub starts 2024-04-14T15:12:33.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:32 smithi090 ceph-mon[41670]: 6.10 scrub ok 2024-04-14T15:12:33.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:32 smithi090 ceph-mon[41670]: 5.1d scrub starts 2024-04-14T15:12:33.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:32 smithi090 ceph-mon[41670]: 5.1d scrub ok 2024-04-14T15:12:33.275 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:12:33.302 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:33.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:33.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:12:33.403 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:12:33.429 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:33.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:33.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:12:33.531 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:12:33.555 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:33.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:33.556 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:12:33.657 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:12:33.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:33.681 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:33.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:12:33.782 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:12:33.806 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:33.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:33.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:12:33.907 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:12:33.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:33.934 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:33.934 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:12:34.035 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:12:34.068 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:34.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:34.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:12:34.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:33 smithi092 ceph-mon[33966]: 4.17 scrub starts 2024-04-14T15:12:34.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:33 smithi092 ceph-mon[33966]: 4.17 scrub ok 2024-04-14T15:12:34.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:33 smithi092 ceph-mon[33966]: 4.15 scrub starts 2024-04-14T15:12:34.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:33 smithi092 ceph-mon[33966]: 4.15 scrub ok 2024-04-14T15:12:34.170 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:12:34.191 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:12:33 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:12:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:12:34.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:33 smithi090 ceph-mon[41670]: 4.17 scrub starts 2024-04-14T15:12:34.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:33 smithi090 ceph-mon[41670]: 4.17 scrub ok 2024-04-14T15:12:34.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:33 smithi090 ceph-mon[41670]: 4.15 scrub starts 2024-04-14T15:12:34.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:33 smithi090 ceph-mon[41670]: 4.15 scrub ok 2024-04-14T15:12:34.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:33 smithi090 ceph-mon[30399]: 4.17 scrub starts 2024-04-14T15:12:34.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:33 smithi090 ceph-mon[30399]: 4.17 scrub ok 2024-04-14T15:12:34.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:33 smithi090 ceph-mon[30399]: 4.15 scrub starts 2024-04-14T15:12:34.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:33 smithi090 ceph-mon[30399]: 4.15 scrub ok 2024-04-14T15:12:34.194 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:34.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:34.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:12:34.296 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:12:34.321 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:34.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:34.322 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:12:34.423 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:12:34.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:34.449 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:34.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:12:34.550 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:12:34.584 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:34.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:34.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:12:34.686 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:12:34.712 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:34.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:34.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:12:34.814 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:12:34.841 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:34.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:34.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:12:34.943 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:12:34.975 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:34.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:34.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:12:35.077 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:12:35.105 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:35.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:35.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:12:35.122 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:12:34 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:12:35.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:34 smithi092 ceph-mon[33966]: pgmap v91: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:12:35.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:34 smithi092 ceph-mon[33966]: 2.13 scrub starts 2024-04-14T15:12:35.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:34 smithi092 ceph-mon[33966]: 2.13 scrub ok 2024-04-14T15:12:35.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:34 smithi092 ceph-mon[33966]: 6.c scrub starts 2024-04-14T15:12:35.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:34 smithi092 ceph-mon[33966]: 6.c scrub ok 2024-04-14T15:12:35.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:34 smithi092 ceph-mon[33966]: 4.14 scrub starts 2024-04-14T15:12:35.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:34 smithi092 ceph-mon[33966]: 4.14 scrub ok 2024-04-14T15:12:35.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:34 smithi090 ceph-mon[41670]: pgmap v91: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:12:35.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:34 smithi090 ceph-mon[41670]: 2.13 scrub starts 2024-04-14T15:12:35.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:34 smithi090 ceph-mon[41670]: 2.13 scrub ok 2024-04-14T15:12:35.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:34 smithi090 ceph-mon[41670]: 6.c scrub starts 2024-04-14T15:12:35.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:34 smithi090 ceph-mon[41670]: 6.c scrub ok 2024-04-14T15:12:35.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:34 smithi090 ceph-mon[41670]: 4.14 scrub starts 2024-04-14T15:12:35.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:34 smithi090 ceph-mon[41670]: 4.14 scrub ok 2024-04-14T15:12:35.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:34 smithi090 ceph-mon[30399]: pgmap v91: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:12:35.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:34 smithi090 ceph-mon[30399]: 2.13 scrub starts 2024-04-14T15:12:35.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:34 smithi090 ceph-mon[30399]: 2.13 scrub ok 2024-04-14T15:12:35.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:34 smithi090 ceph-mon[30399]: 6.c scrub starts 2024-04-14T15:12:35.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:34 smithi090 ceph-mon[30399]: 6.c scrub ok 2024-04-14T15:12:35.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:34 smithi090 ceph-mon[30399]: 4.14 scrub starts 2024-04-14T15:12:35.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:34 smithi090 ceph-mon[30399]: 4.14 scrub ok 2024-04-14T15:12:35.207 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:12:35.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:35.235 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:35.235 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:12:35.337 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:12:35.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:35.361 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:35.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:12:35.462 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:12:35.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:35.490 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:35.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:12:35.591 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:12:35.623 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:35.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:35.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:12:35.725 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:12:35.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:35.750 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:35.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:12:35.851 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:12:35.879 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:35.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:35.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:12:35.941 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:35 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:12:35.941 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:35 smithi090 ceph-mon[41670]: 6.6 scrub starts 2024-04-14T15:12:35.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:35 smithi090 ceph-mon[41670]: 6.6 scrub ok 2024-04-14T15:12:35.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:35 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:12:35.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:35 smithi090 ceph-mon[30399]: 6.6 scrub starts 2024-04-14T15:12:35.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:35 smithi090 ceph-mon[30399]: 6.6 scrub ok 2024-04-14T15:12:35.980 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:12:36.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:36.009 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:36.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:12:36.110 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:12:36.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:35 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:12:36.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:35 smithi092 ceph-mon[33966]: 6.6 scrub starts 2024-04-14T15:12:36.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:35 smithi092 ceph-mon[33966]: 6.6 scrub ok 2024-04-14T15:12:36.135 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:36.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:36.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:12:36.237 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:12:36.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:36.266 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:36.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:12:36.367 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:12:36.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:36.404 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:36.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:12:36.505 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:12:36.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:36.534 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:36.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:12:36.635 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:12:36.662 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:36.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:36.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:12:36.765 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:12:36.791 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:36.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:36.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:12:36.893 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:12:36.921 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:36.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:36.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:12:36.934 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:12:37.022 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:12:37.050 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:37.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:37.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:12:37.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:36 smithi092 ceph-mon[33966]: pgmap v92: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:12:37.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:36 smithi092 ceph-mon[33966]: 4.11 deep-scrub starts 2024-04-14T15:12:37.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:36 smithi092 ceph-mon[33966]: 4.11 deep-scrub ok 2024-04-14T15:12:37.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:36 smithi092 ceph-mon[33966]: 5.13 scrub starts 2024-04-14T15:12:37.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:36 smithi092 ceph-mon[33966]: 5.13 scrub ok 2024-04-14T15:12:37.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:36 smithi092 ceph-mon[33966]: 2.14 scrub starts 2024-04-14T15:12:37.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:36 smithi092 ceph-mon[33966]: 2.14 scrub ok 2024-04-14T15:12:37.152 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:12:37.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:37.176 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:37.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:12:37.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:36 smithi090 ceph-mon[41670]: pgmap v92: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:12:37.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:36 smithi090 ceph-mon[41670]: 4.11 deep-scrub starts 2024-04-14T15:12:37.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:36 smithi090 ceph-mon[41670]: 4.11 deep-scrub ok 2024-04-14T15:12:37.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:36 smithi090 ceph-mon[41670]: 5.13 scrub starts 2024-04-14T15:12:37.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:36 smithi090 ceph-mon[41670]: 5.13 scrub ok 2024-04-14T15:12:37.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:36 smithi090 ceph-mon[41670]: 2.14 scrub starts 2024-04-14T15:12:37.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:36 smithi090 ceph-mon[41670]: 2.14 scrub ok 2024-04-14T15:12:37.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:36 smithi090 ceph-mon[30399]: pgmap v92: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:12:37.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:36 smithi090 ceph-mon[30399]: 4.11 deep-scrub starts 2024-04-14T15:12:37.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:36 smithi090 ceph-mon[30399]: 4.11 deep-scrub ok 2024-04-14T15:12:37.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:36 smithi090 ceph-mon[30399]: 5.13 scrub starts 2024-04-14T15:12:37.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:36 smithi090 ceph-mon[30399]: 5.13 scrub ok 2024-04-14T15:12:37.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:36 smithi090 ceph-mon[30399]: 2.14 scrub starts 2024-04-14T15:12:37.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:36 smithi090 ceph-mon[30399]: 2.14 scrub ok 2024-04-14T15:12:37.277 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:12:37.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:37.302 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:37.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:12:37.404 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:12:37.432 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:37.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:37.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:12:37.534 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:12:37.562 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:37.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:37.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:12:37.664 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:12:37.691 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:37.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:37.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:12:37.793 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:12:37.821 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:37.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:37.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:12:37.876 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:37 smithi090 ceph-mon[30399]: 5.7 scrub starts 2024-04-14T15:12:37.876 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:37 smithi090 ceph-mon[30399]: 5.7 scrub ok 2024-04-14T15:12:37.877 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:37 smithi090 ceph-mon[30399]: 5.9 scrub starts 2024-04-14T15:12:37.877 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:37 smithi090 ceph-mon[30399]: 5.9 scrub ok 2024-04-14T15:12:37.877 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:37 smithi090 ceph-mon[30399]: 2.11 scrub starts 2024-04-14T15:12:37.877 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:37 smithi090 ceph-mon[30399]: 2.11 scrub ok 2024-04-14T15:12:37.877 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:37 smithi090 ceph-mon[30399]: 6.1a scrub starts 2024-04-14T15:12:37.877 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:37 smithi090 ceph-mon[30399]: 6.1a scrub ok 2024-04-14T15:12:37.923 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:12:37.951 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:37.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:37.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:12:38.053 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:12:38.087 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:38.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:38.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:12:38.106 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:12:38.142 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:37 smithi090 ceph-mon[41670]: 5.7 scrub starts 2024-04-14T15:12:38.142 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:37 smithi090 ceph-mon[41670]: 5.7 scrub ok 2024-04-14T15:12:38.142 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:37 smithi090 ceph-mon[41670]: 5.9 scrub starts 2024-04-14T15:12:38.142 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:37 smithi090 ceph-mon[41670]: 5.9 scrub ok 2024-04-14T15:12:38.142 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:37 smithi090 ceph-mon[41670]: 2.11 scrub starts 2024-04-14T15:12:38.142 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:37 smithi090 ceph-mon[41670]: 2.11 scrub ok 2024-04-14T15:12:38.142 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:37 smithi090 ceph-mon[41670]: 6.1a scrub starts 2024-04-14T15:12:38.142 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:37 smithi090 ceph-mon[41670]: 6.1a scrub ok 2024-04-14T15:12:38.189 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:12:38.217 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:38.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:38.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:12:38.319 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:12:38.347 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:38.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:38.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:12:38.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:37 smithi092 ceph-mon[33966]: 5.7 scrub starts 2024-04-14T15:12:38.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:37 smithi092 ceph-mon[33966]: 5.7 scrub ok 2024-04-14T15:12:38.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:37 smithi092 ceph-mon[33966]: 5.9 scrub starts 2024-04-14T15:12:38.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:37 smithi092 ceph-mon[33966]: 5.9 scrub ok 2024-04-14T15:12:38.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:37 smithi092 ceph-mon[33966]: 2.11 scrub starts 2024-04-14T15:12:38.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:37 smithi092 ceph-mon[33966]: 2.11 scrub ok 2024-04-14T15:12:38.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:37 smithi092 ceph-mon[33966]: 6.1a scrub starts 2024-04-14T15:12:38.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:37 smithi092 ceph-mon[33966]: 6.1a scrub ok 2024-04-14T15:12:38.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.389+0000 7fd6fd34a700 1 -- 172.21.15.90:0/1365374736 >> v1:172.21.15.90:6789/0 conn(0x7fd6f8111510 legacy=0x7fd6f81139b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:38.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.389+0000 7fd6fd34a700 1 -- 172.21.15.90:0/1365374736 shutdown_connections 2024-04-14T15:12:38.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.389+0000 7fd6fd34a700 1 -- 172.21.15.90:0/1365374736 >> 172.21.15.90:0/1365374736 conn(0x7fd6f80fd290 msgr2=0x7fd6f80ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:12:38.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.389+0000 7fd6fd34a700 1 -- 172.21.15.90:0/1365374736 shutdown_connections 2024-04-14T15:12:38.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.389+0000 7fd6fd34a700 1 -- 172.21.15.90:0/1365374736 wait complete. 2024-04-14T15:12:38.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.390+0000 7fd6fd34a700 1 librados: starting msgr at 2024-04-14T15:12:38.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.390+0000 7fd6fd34a700 1 librados: starting objecter 2024-04-14T15:12:38.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.390+0000 7fd6fd34a700 1 Processor -- start 2024-04-14T15:12:38.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.390+0000 7fd6fd34a700 1 -- start start 2024-04-14T15:12:38.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.390+0000 7fd6fd34a700 1 librados: setting wanted keys 2024-04-14T15:12:38.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.390+0000 7fd6fd34a700 1 librados: calling monclient init 2024-04-14T15:12:38.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.390+0000 7fd6fd34a700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd6f81a9a90 con 0x7fd6f8111510 2024-04-14T15:12:38.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.390+0000 7fd6fd34a700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd6f81aacb0 con 0x7fd6f8101610 2024-04-14T15:12:38.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.390+0000 7fd6fd34a700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd6f81abef0 con 0x7fd6f81021f0 2024-04-14T15:12:38.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.390+0000 7fd6f77fe700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fd6f8111510 0x7fd6f81a8150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:52066/0 (socket says 172.21.15.90:52066) 2024-04-14T15:12:38.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.390+0000 7fd6f77fe700 1 -- 172.21.15.90:0/2196878314 learned_addr learned my addr 172.21.15.90:0/2196878314 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:12:38.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.390+0000 7fd6f67fc700 1 --1- 172.21.15.90:0/2196878314 >> v1:172.21.15.90:6790/0 conn(0x7fd6f81021f0 0x7fd6f81a4970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:33102/0 (socket says 172.21.15.90:33102) 2024-04-14T15:12:38.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.391+0000 7fd6dffff700 10 client.?.objecter ms_handle_connect 0x7fd6f8111510 2024-04-14T15:12:38.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.391+0000 7fd6dffff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:12:38.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.391+0000 7fd6dffff700 10 client.?.objecter ms_handle_connect 0x7fd6f81021f0 2024-04-14T15:12:38.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.391+0000 7fd6dffff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:12:38.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.391+0000 7fd6dffff700 1 -- 172.21.15.90:0/2196878314 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2634500567 0 0) 0x7fd6f81a9a90 con 0x7fd6f8111510 2024-04-14T15:12:38.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.391+0000 7fd6dffff700 1 -- 172.21.15.90:0/2196878314 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd6e00035b0 con 0x7fd6f8111510 2024-04-14T15:12:38.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.391+0000 7fd6dffff700 1 -- 172.21.15.90:0/2196878314 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3063095862 0 0) 0x7fd6f81abef0 con 0x7fd6f81021f0 2024-04-14T15:12:38.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.391+0000 7fd6dffff700 1 -- 172.21.15.90:0/2196878314 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd6f81a9a90 con 0x7fd6f81021f0 2024-04-14T15:12:38.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.391+0000 7fd6dffff700 10 client.?.objecter ms_handle_connect 0x7fd6f8101610 2024-04-14T15:12:38.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.391+0000 7fd6dffff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:12:38.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.391+0000 7fd6dffff700 1 -- 172.21.15.90:0/2196878314 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3413517659 0 0) 0x7fd6e00035b0 con 0x7fd6f8111510 2024-04-14T15:12:38.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.392+0000 7fd6dffff700 1 -- 172.21.15.90:0/2196878314 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd6f81abef0 con 0x7fd6f8111510 2024-04-14T15:12:38.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.392+0000 7fd6dffff700 1 -- 172.21.15.90:0/2196878314 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3939262001 0 0) 0x7fd6f81aacb0 con 0x7fd6f8101610 2024-04-14T15:12:38.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.392+0000 7fd6dffff700 1 -- 172.21.15.90:0/2196878314 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd6e00035b0 con 0x7fd6f8101610 2024-04-14T15:12:38.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.392+0000 7fd6dffff700 1 -- 172.21.15.90:0/2196878314 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fd6f0004830 con 0x7fd6f8111510 2024-04-14T15:12:38.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.392+0000 7fd6dffff700 1 -- 172.21.15.90:0/2196878314 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2335592973 0 0) 0x7fd6f81abef0 con 0x7fd6f8111510 2024-04-14T15:12:38.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.392+0000 7fd6dffff700 1 -- 172.21.15.90:0/2196878314 >> v1:172.21.15.90:6790/0 conn(0x7fd6f81021f0 legacy=0x7fd6f81a4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:38.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.392+0000 7fd6dffff700 1 -- 172.21.15.90:0/2196878314 >> v1:172.21.15.92:6789/0 conn(0x7fd6f8101610 legacy=0x7fd6f810fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:38.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.392+0000 7fd6dffff700 1 -- 172.21.15.90:0/2196878314 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6f81ad130 con 0x7fd6f8111510 2024-04-14T15:12:38.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.392+0000 7fd6fd34a700 1 -- 172.21.15.90:0/2196878314 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd6f81a9cc0 con 0x7fd6f8111510 2024-04-14T15:12:38.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.392+0000 7fd6fd34a700 10 client.14874.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:12:38.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.393+0000 7fd6fd34a700 1 -- 172.21.15.90:0/2196878314 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd6f81aa180 con 0x7fd6f8111510 2024-04-14T15:12:38.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.393+0000 7fd6fd34a700 1 librados: init done 2024-04-14T15:12:38.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.393+0000 7fd6dffff700 1 -- 172.21.15.90:0/2196878314 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fd6f0004a90 con 0x7fd6f8111510 2024-04-14T15:12:38.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.393+0000 7fd6dffff700 1 -- 172.21.15.90:0/2196878314 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fd6f0005590 con 0x7fd6f8111510 2024-04-14T15:12:38.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.394+0000 7fd6fd34a700 1 -- 172.21.15.90:0/2196878314 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fd6f806ace0 con 0x7fd6f8111510 2024-04-14T15:12:38.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.394+0000 7fd6dffff700 1 -- 172.21.15.90:0/2196878314 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fd6f0003ef0 con 0x7fd6f8111510 2024-04-14T15:12:38.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.395+0000 7fd6dffff700 1 -- 172.21.15.90:0/2196878314 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(102..102 src has 51..102) ==== 6320+0+0 (unknown 3871776270 0 0) 0x7fd6f008a800 con 0x7fd6f8111510 2024-04-14T15:12:38.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.395+0000 7fd6dffff700 10 client.14874.objecter ms_dispatch 0x7fd6f8001550 osd_map(102..102 src has 51..102) 2024-04-14T15:12:38.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.395+0000 7fd6dffff700 3 client.14874.objecter handle_osd_map got epochs [102,102] > 0 2024-04-14T15:12:38.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.395+0000 7fd6dffff700 3 client.14874.objecter handle_osd_map decoding full epoch 102 2024-04-14T15:12:38.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.395+0000 7fd6dffff700 20 client.14874.objecter dump_active .. 0 homeless 2024-04-14T15:12:38.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.395+0000 7fd6dffff700 1 -- 172.21.15.90:0/2196878314 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v102) ==== 54+0+16 (unknown 2944338779 0 2420000192) 0x7fd6f008abf0 con 0x7fd6f8111510 2024-04-14T15:12:38.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.395+0000 7fd6dffff700 10 client.14874.objecter ms_handle_connect 0x7fd6e0070d40 2024-04-14T15:12:38.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.396+0000 7fd6ddffb700 10 client.14874.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:12:38.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.396+0000 7fd6ddffb700 15 client.14874.objecter _session_command_op_assign -1 1 2024-04-14T15:12:38.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.396+0000 7fd6ddffb700 20 client.14874.objecter _get_session s=0x7fd6f8117910 osd=0 3 2024-04-14T15:12:38.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.396+0000 7fd6ddffb700 20 client.14874.objecter put_session s=0x7fd6f8117910 osd=0 3 2024-04-14T15:12:38.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.396+0000 7fd6ddffb700 20 client.14874.objecter _get_session s=0x7fd6f8117910 osd=0 3 2024-04-14T15:12:38.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.396+0000 7fd6ddffb700 15 client.14874.objecter _session_command_op_remove -1 1 2024-04-14T15:12:38.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.396+0000 7fd6ddffb700 20 client.14874.objecter get_session s=0x7fd6f8117910 osd=0 3 2024-04-14T15:12:38.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.396+0000 7fd6ddffb700 15 client.14874.objecter _session_command_op_assign 0 1 2024-04-14T15:12:38.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.396+0000 7fd6ddffb700 20 client.14874.objecter put_session s=0x7fd6f8117910 osd=0 4 2024-04-14T15:12:38.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.396+0000 7fd6ddffb700 10 client.14874.objecter _send_command 1 2024-04-14T15:12:38.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.396+0000 7fd6ddffb700 1 -- 172.21.15.90:0/2196878314 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd6f810e0f0 con 0x7fd6f81ac0c0 2024-04-14T15:12:38.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.396+0000 7fd6dffff700 10 client.14874.objecter ms_handle_connect 0x7fd6f81ac0c0 2024-04-14T15:12:38.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.397+0000 7fd6dffff700 1 -- 172.21.15.90:0/2196878314 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd6f810e0f0 con 0x7fd6f81ac0c0 2024-04-14T15:12:38.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.397+0000 7fd6dffff700 10 client.14874.objecter ms_dispatch 0x7fd6f8001550 command_reply(tid 1: 0 ) 2024-04-14T15:12:38.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.397+0000 7fd6dffff700 10 client.14874.objecter _finish_command 1 = system:0 2024-04-14T15:12:38.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.397+0000 7fd6dffff700 20 client.14874.objecter put_session s=0x7fd6f8117910 osd=0 4 2024-04-14T15:12:38.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.397+0000 7fd6dffff700 15 client.14874.objecter _session_command_op_remove 0 1 2024-04-14T15:12:38.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.430+0000 7fd6ddffb700 10 client.14874.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:12:38.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.430+0000 7fd6ddffb700 15 client.14874.objecter _session_command_op_assign -1 2 2024-04-14T15:12:38.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.430+0000 7fd6ddffb700 20 client.14874.objecter _get_session s=0x7fd6f8117910 osd=0 3 2024-04-14T15:12:38.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.430+0000 7fd6ddffb700 20 client.14874.objecter put_session s=0x7fd6f8117910 osd=0 3 2024-04-14T15:12:38.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.430+0000 7fd6ddffb700 20 client.14874.objecter _get_session s=0x7fd6f8117910 osd=0 3 2024-04-14T15:12:38.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.430+0000 7fd6ddffb700 15 client.14874.objecter _session_command_op_remove -1 2 2024-04-14T15:12:38.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.430+0000 7fd6ddffb700 20 client.14874.objecter get_session s=0x7fd6f8117910 osd=0 3 2024-04-14T15:12:38.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.430+0000 7fd6ddffb700 15 client.14874.objecter _session_command_op_assign 0 2 2024-04-14T15:12:38.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.430+0000 7fd6ddffb700 20 client.14874.objecter put_session s=0x7fd6f8117910 osd=0 4 2024-04-14T15:12:38.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.430+0000 7fd6ddffb700 10 client.14874.objecter _send_command 2 2024-04-14T15:12:38.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.430+0000 7fd6ddffb700 1 -- 172.21.15.90:0/2196878314 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd6f80693d0 con 0x7fd6f81ac0c0 2024-04-14T15:12:38.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.433+0000 7fd6dffff700 1 -- 172.21.15.90:0/2196878314 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd6f80693d0 con 0x7fd6f81ac0c0 2024-04-14T15:12:38.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.433+0000 7fd6dffff700 10 client.14874.objecter ms_dispatch 0x7fd6f8001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:12:38.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.433+0000 7fd6dffff700 10 client.14874.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:12:38.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.433+0000 7fd6dffff700 20 client.14874.objecter put_session s=0x7fd6f8117910 osd=0 4 2024-04-14T15:12:38.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.433+0000 7fd6dffff700 15 client.14874.objecter _session_command_op_remove 0 2 2024-04-14T15:12:38.434 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:12:38.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.433+0000 7fd6ddffb700 10 client.14874.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:12:38.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.433+0000 7fd6ddffb700 15 client.14874.objecter _session_command_op_assign -1 3 2024-04-14T15:12:38.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.433+0000 7fd6ddffb700 20 client.14874.objecter _get_session s=0x7fd6f81ace00 osd=1 3 2024-04-14T15:12:38.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.433+0000 7fd6ddffb700 20 client.14874.objecter put_session s=0x7fd6f81ace00 osd=1 3 2024-04-14T15:12:38.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.433+0000 7fd6ddffb700 20 client.14874.objecter _get_session s=0x7fd6f81ace00 osd=1 3 2024-04-14T15:12:38.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.433+0000 7fd6ddffb700 15 client.14874.objecter _session_command_op_remove -1 3 2024-04-14T15:12:38.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.433+0000 7fd6ddffb700 20 client.14874.objecter get_session s=0x7fd6f81ace00 osd=1 3 2024-04-14T15:12:38.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.433+0000 7fd6ddffb700 15 client.14874.objecter _session_command_op_assign 1 3 2024-04-14T15:12:38.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.433+0000 7fd6ddffb700 20 client.14874.objecter put_session s=0x7fd6f81ace00 osd=1 4 2024-04-14T15:12:38.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.433+0000 7fd6ddffb700 10 client.14874.objecter _send_command 3 2024-04-14T15:12:38.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.433+0000 7fd6ddffb700 1 -- 172.21.15.90:0/2196878314 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fd6f81b2b00 con 0x7fd6f8112e80 2024-04-14T15:12:38.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.434+0000 7fd6dffff700 10 client.14874.objecter ms_handle_connect 0x7fd6f8112e80 2024-04-14T15:12:38.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.435+0000 7fd6dffff700 1 -- 172.21.15.90:0/2196878314 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd6f81b2b00 con 0x7fd6f8112e80 2024-04-14T15:12:38.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.435+0000 7fd6dffff700 10 client.14874.objecter ms_dispatch 0x7fd6f8001550 command_reply(tid 3: 0 ) 2024-04-14T15:12:38.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.435+0000 7fd6dffff700 10 client.14874.objecter _finish_command 3 = system:0 2024-04-14T15:12:38.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.435+0000 7fd6dffff700 20 client.14874.objecter put_session s=0x7fd6f81ace00 osd=1 4 2024-04-14T15:12:38.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.435+0000 7fd6dffff700 15 client.14874.objecter _session_command_op_remove 1 3 2024-04-14T15:12:38.449 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:12:38.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.467+0000 7fd6ddffb700 10 client.14874.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:12:38.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.467+0000 7fd6ddffb700 15 client.14874.objecter _session_command_op_assign -1 4 2024-04-14T15:12:38.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.467+0000 7fd6ddffb700 20 client.14874.objecter _get_session s=0x7fd6f81ace00 osd=1 3 2024-04-14T15:12:38.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.467+0000 7fd6ddffb700 20 client.14874.objecter put_session s=0x7fd6f81ace00 osd=1 3 2024-04-14T15:12:38.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.467+0000 7fd6ddffb700 20 client.14874.objecter _get_session s=0x7fd6f81ace00 osd=1 3 2024-04-14T15:12:38.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.467+0000 7fd6ddffb700 15 client.14874.objecter _session_command_op_remove -1 4 2024-04-14T15:12:38.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.467+0000 7fd6ddffb700 20 client.14874.objecter get_session s=0x7fd6f81ace00 osd=1 3 2024-04-14T15:12:38.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.467+0000 7fd6ddffb700 15 client.14874.objecter _session_command_op_assign 1 4 2024-04-14T15:12:38.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.467+0000 7fd6ddffb700 20 client.14874.objecter put_session s=0x7fd6f81ace00 osd=1 4 2024-04-14T15:12:38.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.467+0000 7fd6ddffb700 10 client.14874.objecter _send_command 4 2024-04-14T15:12:38.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.467+0000 7fd6ddffb700 1 -- 172.21.15.90:0/2196878314 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd6f81ab8a0 con 0x7fd6f8112e80 2024-04-14T15:12:38.469 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:12:38.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.469+0000 7fd6dffff700 1 -- 172.21.15.90:0/2196878314 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd6f81ab8a0 con 0x7fd6f8112e80 2024-04-14T15:12:38.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.469+0000 7fd6dffff700 10 client.14874.objecter ms_dispatch 0x7fd6f8001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:12:38.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.469+0000 7fd6dffff700 10 client.14874.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:12:38.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.469+0000 7fd6dffff700 20 client.14874.objecter put_session s=0x7fd6f81ace00 osd=1 4 2024-04-14T15:12:38.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.469+0000 7fd6dffff700 15 client.14874.objecter _session_command_op_remove 1 4 2024-04-14T15:12:38.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.469+0000 7fd6ddffb700 10 client.14874.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:12:38.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.469+0000 7fd6ddffb700 15 client.14874.objecter _session_command_op_assign -1 5 2024-04-14T15:12:38.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.469+0000 7fd6ddffb700 20 client.14874.objecter _get_session s=0x7fd6f81abca0 osd=2 3 2024-04-14T15:12:38.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.469+0000 7fd6ddffb700 20 client.14874.objecter put_session s=0x7fd6f81abca0 osd=2 3 2024-04-14T15:12:38.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.469+0000 7fd6ddffb700 20 client.14874.objecter _get_session s=0x7fd6f81abca0 osd=2 3 2024-04-14T15:12:38.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.469+0000 7fd6ddffb700 15 client.14874.objecter _session_command_op_remove -1 5 2024-04-14T15:12:38.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.469+0000 7fd6ddffb700 20 client.14874.objecter get_session s=0x7fd6f81abca0 osd=2 3 2024-04-14T15:12:38.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.469+0000 7fd6ddffb700 15 client.14874.objecter _session_command_op_assign 2 5 2024-04-14T15:12:38.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.469+0000 7fd6ddffb700 20 client.14874.objecter put_session s=0x7fd6f81abca0 osd=2 4 2024-04-14T15:12:38.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.469+0000 7fd6ddffb700 10 client.14874.objecter _send_command 5 2024-04-14T15:12:38.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.469+0000 7fd6ddffb700 1 -- 172.21.15.90:0/2196878314 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fd6f81b75f0 con 0x7fd6f8063bd0 2024-04-14T15:12:38.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.470+0000 7fd6dffff700 10 client.14874.objecter ms_handle_connect 0x7fd6f8063bd0 2024-04-14T15:12:38.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.471+0000 7fd6dffff700 1 -- 172.21.15.90:0/2196878314 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd6f81b75f0 con 0x7fd6f8063bd0 2024-04-14T15:12:38.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.471+0000 7fd6dffff700 10 client.14874.objecter ms_dispatch 0x7fd6f8001550 command_reply(tid 5: 0 ) 2024-04-14T15:12:38.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.471+0000 7fd6dffff700 10 client.14874.objecter _finish_command 5 = system:0 2024-04-14T15:12:38.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.471+0000 7fd6dffff700 20 client.14874.objecter put_session s=0x7fd6f81abca0 osd=2 4 2024-04-14T15:12:38.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.471+0000 7fd6dffff700 15 client.14874.objecter _session_command_op_remove 2 5 2024-04-14T15:12:38.476 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:38.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:38.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:12:38.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.503+0000 7fd6ddffb700 10 client.14874.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:12:38.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.503+0000 7fd6ddffb700 15 client.14874.objecter _session_command_op_assign -1 6 2024-04-14T15:12:38.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.503+0000 7fd6ddffb700 20 client.14874.objecter _get_session s=0x7fd6f81abca0 osd=2 3 2024-04-14T15:12:38.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.503+0000 7fd6ddffb700 20 client.14874.objecter put_session s=0x7fd6f81abca0 osd=2 3 2024-04-14T15:12:38.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.503+0000 7fd6ddffb700 20 client.14874.objecter _get_session s=0x7fd6f81abca0 osd=2 3 2024-04-14T15:12:38.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.503+0000 7fd6ddffb700 15 client.14874.objecter _session_command_op_remove -1 6 2024-04-14T15:12:38.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.503+0000 7fd6ddffb700 20 client.14874.objecter get_session s=0x7fd6f81abca0 osd=2 3 2024-04-14T15:12:38.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.503+0000 7fd6ddffb700 15 client.14874.objecter _session_command_op_assign 2 6 2024-04-14T15:12:38.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.503+0000 7fd6ddffb700 20 client.14874.objecter put_session s=0x7fd6f81abca0 osd=2 4 2024-04-14T15:12:38.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.503+0000 7fd6ddffb700 10 client.14874.objecter _send_command 6 2024-04-14T15:12:38.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.503+0000 7fd6ddffb700 1 -- 172.21.15.90:0/2196878314 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd6f8063260 con 0x7fd6f8063bd0 2024-04-14T15:12:38.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.505+0000 7fd6dffff700 1 -- 172.21.15.90:0/2196878314 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd6f8063260 con 0x7fd6f8063bd0 2024-04-14T15:12:38.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.505+0000 7fd6dffff700 10 client.14874.objecter ms_dispatch 0x7fd6f8001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:12:38.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.505+0000 7fd6dffff700 10 client.14874.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:12:38.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.505+0000 7fd6dffff700 20 client.14874.objecter put_session s=0x7fd6f81abca0 osd=2 4 2024-04-14T15:12:38.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.505+0000 7fd6dffff700 15 client.14874.objecter _session_command_op_remove 2 6 2024-04-14T15:12:38.506 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:12:38.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.505+0000 7fd6ddffb700 10 client.14874.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:12:38.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.505+0000 7fd6ddffb700 15 client.14874.objecter _session_command_op_assign -1 7 2024-04-14T15:12:38.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.505+0000 7fd6ddffb700 20 client.14874.objecter _get_session s=0x7fd6f80634c0 osd=3 3 2024-04-14T15:12:38.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.505+0000 7fd6ddffb700 20 client.14874.objecter put_session s=0x7fd6f80634c0 osd=3 3 2024-04-14T15:12:38.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.505+0000 7fd6ddffb700 20 client.14874.objecter _get_session s=0x7fd6f80634c0 osd=3 3 2024-04-14T15:12:38.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.505+0000 7fd6ddffb700 15 client.14874.objecter _session_command_op_remove -1 7 2024-04-14T15:12:38.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.505+0000 7fd6ddffb700 20 client.14874.objecter get_session s=0x7fd6f80634c0 osd=3 3 2024-04-14T15:12:38.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.505+0000 7fd6ddffb700 15 client.14874.objecter _session_command_op_assign 3 7 2024-04-14T15:12:38.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.505+0000 7fd6ddffb700 20 client.14874.objecter put_session s=0x7fd6f80634c0 osd=3 4 2024-04-14T15:12:38.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.505+0000 7fd6ddffb700 10 client.14874.objecter _send_command 7 2024-04-14T15:12:38.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.505+0000 7fd6ddffb700 1 -- 172.21.15.90:0/2196878314 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fd6f81b94a0 con 0x7fd6f81b8920 2024-04-14T15:12:38.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.506+0000 7fd6dffff700 10 client.14874.objecter ms_handle_connect 0x7fd6f81b8920 2024-04-14T15:12:38.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.507+0000 7fd6dffff700 1 -- 172.21.15.90:0/2196878314 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd6f81b94a0 con 0x7fd6f81b8920 2024-04-14T15:12:38.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.507+0000 7fd6dffff700 10 client.14874.objecter ms_dispatch 0x7fd6f8001550 command_reply(tid 7: 0 ) 2024-04-14T15:12:38.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.507+0000 7fd6dffff700 10 client.14874.objecter _finish_command 7 = system:0 2024-04-14T15:12:38.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.507+0000 7fd6dffff700 20 client.14874.objecter put_session s=0x7fd6f80634c0 osd=3 4 2024-04-14T15:12:38.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.507+0000 7fd6dffff700 15 client.14874.objecter _session_command_op_remove 3 7 2024-04-14T15:12:38.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.524+0000 7fd6ddffb700 10 client.14874.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:12:38.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.524+0000 7fd6ddffb700 15 client.14874.objecter _session_command_op_assign -1 8 2024-04-14T15:12:38.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.524+0000 7fd6ddffb700 20 client.14874.objecter _get_session s=0x7fd6f80634c0 osd=3 3 2024-04-14T15:12:38.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.524+0000 7fd6ddffb700 20 client.14874.objecter put_session s=0x7fd6f80634c0 osd=3 3 2024-04-14T15:12:38.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.524+0000 7fd6ddffb700 20 client.14874.objecter _get_session s=0x7fd6f80634c0 osd=3 3 2024-04-14T15:12:38.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.524+0000 7fd6ddffb700 15 client.14874.objecter _session_command_op_remove -1 8 2024-04-14T15:12:38.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.524+0000 7fd6ddffb700 20 client.14874.objecter get_session s=0x7fd6f80634c0 osd=3 3 2024-04-14T15:12:38.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.524+0000 7fd6ddffb700 15 client.14874.objecter _session_command_op_assign 3 8 2024-04-14T15:12:38.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.524+0000 7fd6ddffb700 20 client.14874.objecter put_session s=0x7fd6f80634c0 osd=3 4 2024-04-14T15:12:38.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.524+0000 7fd6ddffb700 10 client.14874.objecter _send_command 8 2024-04-14T15:12:38.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.524+0000 7fd6ddffb700 1 -- 172.21.15.90:0/2196878314 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd6f81b9e20 con 0x7fd6f81b8920 2024-04-14T15:12:38.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.526+0000 7fd6dffff700 1 -- 172.21.15.90:0/2196878314 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd6f81b9e20 con 0x7fd6f81b8920 2024-04-14T15:12:38.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.526+0000 7fd6dffff700 10 client.14874.objecter ms_dispatch 0x7fd6f8001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:12:38.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.526+0000 7fd6dffff700 10 client.14874.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:12:38.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.526+0000 7fd6dffff700 20 client.14874.objecter put_session s=0x7fd6f80634c0 osd=3 4 2024-04-14T15:12:38.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.526+0000 7fd6dffff700 15 client.14874.objecter _session_command_op_remove 3 8 2024-04-14T15:12:38.530 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:12:38.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.526+0000 7fd6ddffb700 10 client.14874.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:12:38.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.526+0000 7fd6ddffb700 15 client.14874.objecter _session_command_op_assign -1 9 2024-04-14T15:12:38.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.526+0000 7fd6ddffb700 20 client.14874.objecter _get_session s=0x7fd6f81b9e20 osd=4 3 2024-04-14T15:12:38.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.526+0000 7fd6ddffb700 20 client.14874.objecter put_session s=0x7fd6f81b9e20 osd=4 3 2024-04-14T15:12:38.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.526+0000 7fd6ddffb700 20 client.14874.objecter _get_session s=0x7fd6f81b9e20 osd=4 3 2024-04-14T15:12:38.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.526+0000 7fd6ddffb700 15 client.14874.objecter _session_command_op_remove -1 9 2024-04-14T15:12:38.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.526+0000 7fd6ddffb700 20 client.14874.objecter get_session s=0x7fd6f81b9e20 osd=4 3 2024-04-14T15:12:38.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.526+0000 7fd6ddffb700 15 client.14874.objecter _session_command_op_assign 4 9 2024-04-14T15:12:38.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.526+0000 7fd6ddffb700 20 client.14874.objecter put_session s=0x7fd6f81b9e20 osd=4 4 2024-04-14T15:12:38.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.526+0000 7fd6ddffb700 10 client.14874.objecter _send_command 9 2024-04-14T15:12:38.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.526+0000 7fd6ddffb700 1 -- 172.21.15.90:0/2196878314 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fd6f81c0d80 con 0x7fd6f81bd130 2024-04-14T15:12:38.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.527+0000 7fd6dffff700 10 client.14874.objecter ms_handle_connect 0x7fd6f81bd130 2024-04-14T15:12:38.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.527+0000 7fd6dffff700 1 -- 172.21.15.90:0/2196878314 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd6f81c0d80 con 0x7fd6f81bd130 2024-04-14T15:12:38.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.527+0000 7fd6dffff700 10 client.14874.objecter ms_dispatch 0x7fd6f8001550 command_reply(tid 9: 0 ) 2024-04-14T15:12:38.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.527+0000 7fd6dffff700 10 client.14874.objecter _finish_command 9 = system:0 2024-04-14T15:12:38.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.527+0000 7fd6dffff700 20 client.14874.objecter put_session s=0x7fd6f81b9e20 osd=4 4 2024-04-14T15:12:38.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.527+0000 7fd6dffff700 15 client.14874.objecter _session_command_op_remove 4 9 2024-04-14T15:12:38.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.544+0000 7fd6ddffb700 10 client.14874.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:12:38.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.544+0000 7fd6ddffb700 15 client.14874.objecter _session_command_op_assign -1 10 2024-04-14T15:12:38.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.544+0000 7fd6ddffb700 20 client.14874.objecter _get_session s=0x7fd6f81b9e20 osd=4 3 2024-04-14T15:12:38.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.544+0000 7fd6ddffb700 20 client.14874.objecter put_session s=0x7fd6f81b9e20 osd=4 3 2024-04-14T15:12:38.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.544+0000 7fd6ddffb700 20 client.14874.objecter _get_session s=0x7fd6f81b9e20 osd=4 3 2024-04-14T15:12:38.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.544+0000 7fd6ddffb700 15 client.14874.objecter _session_command_op_remove -1 10 2024-04-14T15:12:38.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.544+0000 7fd6ddffb700 20 client.14874.objecter get_session s=0x7fd6f81b9e20 osd=4 3 2024-04-14T15:12:38.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.544+0000 7fd6ddffb700 15 client.14874.objecter _session_command_op_assign 4 10 2024-04-14T15:12:38.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.544+0000 7fd6ddffb700 20 client.14874.objecter put_session s=0x7fd6f81b9e20 osd=4 4 2024-04-14T15:12:38.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.544+0000 7fd6ddffb700 10 client.14874.objecter _send_command 10 2024-04-14T15:12:38.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.544+0000 7fd6ddffb700 1 -- 172.21.15.90:0/2196878314 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd6f81c2730 con 0x7fd6f81bd130 2024-04-14T15:12:38.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.546+0000 7fd6dffff700 1 -- 172.21.15.90:0/2196878314 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd6f81c2730 con 0x7fd6f81bd130 2024-04-14T15:12:38.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.546+0000 7fd6dffff700 10 client.14874.objecter ms_dispatch 0x7fd6f8001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:12:38.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.546+0000 7fd6dffff700 10 client.14874.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:12:38.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.546+0000 7fd6dffff700 20 client.14874.objecter put_session s=0x7fd6f81b9e20 osd=4 4 2024-04-14T15:12:38.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.546+0000 7fd6dffff700 15 client.14874.objecter _session_command_op_remove 4 10 2024-04-14T15:12:38.547 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:12:38.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.546+0000 7fd6ddffb700 10 client.14874.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:12:38.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.546+0000 7fd6ddffb700 15 client.14874.objecter _session_command_op_assign -1 11 2024-04-14T15:12:38.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.546+0000 7fd6ddffb700 20 client.14874.objecter _get_session s=0x7fd6f81c2730 osd=5 3 2024-04-14T15:12:38.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.546+0000 7fd6ddffb700 20 client.14874.objecter put_session s=0x7fd6f81c2730 osd=5 3 2024-04-14T15:12:38.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.546+0000 7fd6ddffb700 20 client.14874.objecter _get_session s=0x7fd6f81c2730 osd=5 3 2024-04-14T15:12:38.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.546+0000 7fd6ddffb700 15 client.14874.objecter _session_command_op_remove -1 11 2024-04-14T15:12:38.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.546+0000 7fd6ddffb700 20 client.14874.objecter get_session s=0x7fd6f81c2730 osd=5 3 2024-04-14T15:12:38.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.546+0000 7fd6ddffb700 15 client.14874.objecter _session_command_op_assign 5 11 2024-04-14T15:12:38.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.546+0000 7fd6ddffb700 20 client.14874.objecter put_session s=0x7fd6f81c2730 osd=5 4 2024-04-14T15:12:38.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.546+0000 7fd6ddffb700 10 client.14874.objecter _send_command 11 2024-04-14T15:12:38.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.546+0000 7fd6ddffb700 1 -- 172.21.15.90:0/2196878314 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fd6f81c3300 con 0x7fd6f81c2e50 2024-04-14T15:12:38.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.547+0000 7fd6dffff700 10 client.14874.objecter ms_handle_connect 0x7fd6f81c2e50 2024-04-14T15:12:38.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.548+0000 7fd6dffff700 1 -- 172.21.15.90:0/2196878314 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd6f81c3300 con 0x7fd6f81c2e50 2024-04-14T15:12:38.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.548+0000 7fd6dffff700 10 client.14874.objecter ms_dispatch 0x7fd6f8001550 command_reply(tid 11: 0 ) 2024-04-14T15:12:38.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.548+0000 7fd6dffff700 10 client.14874.objecter _finish_command 11 = system:0 2024-04-14T15:12:38.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.548+0000 7fd6dffff700 20 client.14874.objecter put_session s=0x7fd6f81c2730 osd=5 4 2024-04-14T15:12:38.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.548+0000 7fd6dffff700 15 client.14874.objecter _session_command_op_remove 5 11 2024-04-14T15:12:38.577 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:12:38.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.580+0000 7fd6ddffb700 10 client.14874.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:12:38.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.580+0000 7fd6ddffb700 15 client.14874.objecter _session_command_op_assign -1 12 2024-04-14T15:12:38.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.580+0000 7fd6ddffb700 20 client.14874.objecter _get_session s=0x7fd6f81c2730 osd=5 3 2024-04-14T15:12:38.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.580+0000 7fd6ddffb700 20 client.14874.objecter put_session s=0x7fd6f81c2730 osd=5 3 2024-04-14T15:12:38.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.580+0000 7fd6ddffb700 20 client.14874.objecter _get_session s=0x7fd6f81c2730 osd=5 3 2024-04-14T15:12:38.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.580+0000 7fd6ddffb700 15 client.14874.objecter _session_command_op_remove -1 12 2024-04-14T15:12:38.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.580+0000 7fd6ddffb700 20 client.14874.objecter get_session s=0x7fd6f81c2730 osd=5 3 2024-04-14T15:12:38.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.580+0000 7fd6ddffb700 15 client.14874.objecter _session_command_op_assign 5 12 2024-04-14T15:12:38.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.580+0000 7fd6ddffb700 20 client.14874.objecter put_session s=0x7fd6f81c2730 osd=5 4 2024-04-14T15:12:38.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.580+0000 7fd6ddffb700 10 client.14874.objecter _send_command 12 2024-04-14T15:12:38.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.580+0000 7fd6ddffb700 1 -- 172.21.15.90:0/2196878314 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd6f81c3300 con 0x7fd6f81c2e50 2024-04-14T15:12:38.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.582+0000 7fd6dffff700 1 -- 172.21.15.90:0/2196878314 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd6f81c3300 con 0x7fd6f81c2e50 2024-04-14T15:12:38.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.582+0000 7fd6dffff700 10 client.14874.objecter ms_dispatch 0x7fd6f8001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:12:38.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.582+0000 7fd6dffff700 10 client.14874.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:12:38.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.582+0000 7fd6dffff700 20 client.14874.objecter put_session s=0x7fd6f81c2730 osd=5 4 2024-04-14T15:12:38.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.582+0000 7fd6dffff700 15 client.14874.objecter _session_command_op_remove 5 12 2024-04-14T15:12:38.583 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:12:38.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.582+0000 7fd6ddffb700 10 client.14874.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:12:38.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.582+0000 7fd6ddffb700 15 client.14874.objecter _session_command_op_assign -1 13 2024-04-14T15:12:38.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.582+0000 7fd6ddffb700 20 client.14874.objecter _get_session s=0x7fd6f81c3300 osd=6 3 2024-04-14T15:12:38.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.582+0000 7fd6ddffb700 20 client.14874.objecter put_session s=0x7fd6f81c3300 osd=6 3 2024-04-14T15:12:38.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.582+0000 7fd6ddffb700 20 client.14874.objecter _get_session s=0x7fd6f81c3300 osd=6 3 2024-04-14T15:12:38.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.582+0000 7fd6ddffb700 15 client.14874.objecter _session_command_op_remove -1 13 2024-04-14T15:12:38.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.582+0000 7fd6ddffb700 20 client.14874.objecter get_session s=0x7fd6f81c3300 osd=6 3 2024-04-14T15:12:38.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.582+0000 7fd6ddffb700 15 client.14874.objecter _session_command_op_assign 6 13 2024-04-14T15:12:38.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.582+0000 7fd6ddffb700 20 client.14874.objecter put_session s=0x7fd6f81c3300 osd=6 4 2024-04-14T15:12:38.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.582+0000 7fd6ddffb700 10 client.14874.objecter _send_command 13 2024-04-14T15:12:38.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.582+0000 7fd6ddffb700 1 -- 172.21.15.90:0/2196878314 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fd6f81c9950 con 0x7fd6f81c5e40 2024-04-14T15:12:38.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.583+0000 7fd6dffff700 10 client.14874.objecter ms_handle_connect 0x7fd6f81c5e40 2024-04-14T15:12:38.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.584+0000 7fd6dffff700 1 -- 172.21.15.90:0/2196878314 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd6f81c9950 con 0x7fd6f81c5e40 2024-04-14T15:12:38.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.584+0000 7fd6dffff700 10 client.14874.objecter ms_dispatch 0x7fd6f8001550 command_reply(tid 13: 0 ) 2024-04-14T15:12:38.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.584+0000 7fd6dffff700 10 client.14874.objecter _finish_command 13 = system:0 2024-04-14T15:12:38.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.584+0000 7fd6dffff700 20 client.14874.objecter put_session s=0x7fd6f81c3300 osd=6 4 2024-04-14T15:12:38.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.584+0000 7fd6dffff700 15 client.14874.objecter _session_command_op_remove 6 13 2024-04-14T15:12:38.607 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:38.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:38.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:12:38.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.616+0000 7fd6ddffb700 10 client.14874.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:12:38.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.616+0000 7fd6ddffb700 15 client.14874.objecter _session_command_op_assign -1 14 2024-04-14T15:12:38.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.616+0000 7fd6ddffb700 20 client.14874.objecter _get_session s=0x7fd6f81c3300 osd=6 3 2024-04-14T15:12:38.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.616+0000 7fd6ddffb700 20 client.14874.objecter put_session s=0x7fd6f81c3300 osd=6 3 2024-04-14T15:12:38.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.616+0000 7fd6ddffb700 20 client.14874.objecter _get_session s=0x7fd6f81c3300 osd=6 3 2024-04-14T15:12:38.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.616+0000 7fd6ddffb700 15 client.14874.objecter _session_command_op_remove -1 14 2024-04-14T15:12:38.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.616+0000 7fd6ddffb700 20 client.14874.objecter get_session s=0x7fd6f81c3300 osd=6 3 2024-04-14T15:12:38.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.616+0000 7fd6ddffb700 15 client.14874.objecter _session_command_op_assign 6 14 2024-04-14T15:12:38.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.616+0000 7fd6ddffb700 20 client.14874.objecter put_session s=0x7fd6f81c3300 osd=6 4 2024-04-14T15:12:38.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.616+0000 7fd6ddffb700 10 client.14874.objecter _send_command 14 2024-04-14T15:12:38.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.616+0000 7fd6ddffb700 1 -- 172.21.15.90:0/2196878314 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd6f81c9950 con 0x7fd6f81c5e40 2024-04-14T15:12:38.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.618+0000 7fd6dffff700 1 -- 172.21.15.90:0/2196878314 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd6f81c9950 con 0x7fd6f81c5e40 2024-04-14T15:12:38.618 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:12:38.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.618+0000 7fd6dffff700 10 client.14874.objecter ms_dispatch 0x7fd6f8001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:12:38.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.618+0000 7fd6dffff700 10 client.14874.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:12:38.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.618+0000 7fd6dffff700 20 client.14874.objecter put_session s=0x7fd6f81c3300 osd=6 4 2024-04-14T15:12:38.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.618+0000 7fd6dffff700 15 client.14874.objecter _session_command_op_remove 6 14 2024-04-14T15:12:38.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.618+0000 7fd6ddffb700 10 client.14874.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:12:38.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.618+0000 7fd6ddffb700 15 client.14874.objecter _session_command_op_assign -1 15 2024-04-14T15:12:38.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.618+0000 7fd6ddffb700 20 client.14874.objecter _get_session s=0x7fd6f81c9950 osd=7 3 2024-04-14T15:12:38.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.618+0000 7fd6ddffb700 20 client.14874.objecter put_session s=0x7fd6f81c9950 osd=7 3 2024-04-14T15:12:38.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.618+0000 7fd6ddffb700 20 client.14874.objecter _get_session s=0x7fd6f81c9950 osd=7 3 2024-04-14T15:12:38.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.618+0000 7fd6ddffb700 15 client.14874.objecter _session_command_op_remove -1 15 2024-04-14T15:12:38.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.618+0000 7fd6ddffb700 20 client.14874.objecter get_session s=0x7fd6f81c9950 osd=7 3 2024-04-14T15:12:38.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.618+0000 7fd6ddffb700 15 client.14874.objecter _session_command_op_assign 7 15 2024-04-14T15:12:38.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.618+0000 7fd6ddffb700 20 client.14874.objecter put_session s=0x7fd6f81c9950 osd=7 4 2024-04-14T15:12:38.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.618+0000 7fd6ddffb700 10 client.14874.objecter _send_command 15 2024-04-14T15:12:38.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.618+0000 7fd6ddffb700 1 -- 172.21.15.90:0/2196878314 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fd6f81cdc90 con 0x7fd6f81c9fd0 2024-04-14T15:12:38.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.619+0000 7fd6dffff700 10 client.14874.objecter ms_handle_connect 0x7fd6f81c9fd0 2024-04-14T15:12:38.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.619+0000 7fd6dffff700 1 -- 172.21.15.90:0/2196878314 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd6f81cdc90 con 0x7fd6f81c9fd0 2024-04-14T15:12:38.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.619+0000 7fd6dffff700 10 client.14874.objecter ms_dispatch 0x7fd6f8001550 command_reply(tid 15: 0 ) 2024-04-14T15:12:38.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.619+0000 7fd6dffff700 10 client.14874.objecter _finish_command 15 = system:0 2024-04-14T15:12:38.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.619+0000 7fd6dffff700 20 client.14874.objecter put_session s=0x7fd6f81c9950 osd=7 4 2024-04-14T15:12:38.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.619+0000 7fd6dffff700 15 client.14874.objecter _session_command_op_remove 7 15 2024-04-14T15:12:38.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.652+0000 7fd6ddffb700 10 client.14874.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:12:38.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.652+0000 7fd6ddffb700 15 client.14874.objecter _session_command_op_assign -1 16 2024-04-14T15:12:38.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.652+0000 7fd6ddffb700 20 client.14874.objecter _get_session s=0x7fd6f81c9950 osd=7 3 2024-04-14T15:12:38.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.652+0000 7fd6ddffb700 20 client.14874.objecter put_session s=0x7fd6f81c9950 osd=7 3 2024-04-14T15:12:38.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.652+0000 7fd6ddffb700 20 client.14874.objecter _get_session s=0x7fd6f81c9950 osd=7 3 2024-04-14T15:12:38.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.652+0000 7fd6ddffb700 15 client.14874.objecter _session_command_op_remove -1 16 2024-04-14T15:12:38.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.652+0000 7fd6ddffb700 20 client.14874.objecter get_session s=0x7fd6f81c9950 osd=7 3 2024-04-14T15:12:38.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.652+0000 7fd6ddffb700 15 client.14874.objecter _session_command_op_assign 7 16 2024-04-14T15:12:38.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.652+0000 7fd6ddffb700 20 client.14874.objecter put_session s=0x7fd6f81c9950 osd=7 4 2024-04-14T15:12:38.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.652+0000 7fd6ddffb700 10 client.14874.objecter _send_command 16 2024-04-14T15:12:38.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.652+0000 7fd6ddffb700 1 -- 172.21.15.90:0/2196878314 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd6f81cdea0 con 0x7fd6f81c9fd0 2024-04-14T15:12:38.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.654+0000 7fd6dffff700 1 -- 172.21.15.90:0/2196878314 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd6f81cdea0 con 0x7fd6f81c9fd0 2024-04-14T15:12:38.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.654+0000 7fd6dffff700 10 client.14874.objecter ms_dispatch 0x7fd6f8001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:12:38.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.654+0000 7fd6dffff700 10 client.14874.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:12:38.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.654+0000 7fd6dffff700 20 client.14874.objecter put_session s=0x7fd6f81c9950 osd=7 4 2024-04-14T15:12:38.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.654+0000 7fd6dffff700 15 client.14874.objecter _session_command_op_remove 7 16 2024-04-14T15:12:38.655 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:12:38.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.654+0000 7fd6ddffb700 10 librados: watch_flush enter 2024-04-14T15:12:38.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.654+0000 7fd6ddffb700 10 librados: watch_flush exit 2024-04-14T15:12:38.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.654+0000 7fd6ddffb700 10 client.14874.objecter close_session for osd.0 2024-04-14T15:12:38.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.654+0000 7fd6ddffb700 1 -- 172.21.15.90:0/2196878314 >> v1:172.21.15.90:6801/3761806885 conn(0x7fd6f81ac0c0 legacy=0x7fd6f81ac4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:38.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.655+0000 7fd6ddffb700 20 client.14874.objecter put_session s=0x7fd6f8117910 osd=0 1 2024-04-14T15:12:38.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.655+0000 7fd6ddffb700 10 client.14874.objecter close_session for osd.1 2024-04-14T15:12:38.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.655+0000 7fd6ddffb700 1 -- 172.21.15.90:0/2196878314 >> v1:172.21.15.90:6805/4170361699 conn(0x7fd6f8112e80 legacy=0x7fd6f81b13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:38.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.655+0000 7fd6ddffb700 20 client.14874.objecter put_session s=0x7fd6f81ace00 osd=1 1 2024-04-14T15:12:38.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.655+0000 7fd6ddffb700 10 client.14874.objecter close_session for osd.2 2024-04-14T15:12:38.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.655+0000 7fd6ddffb700 1 -- 172.21.15.90:0/2196878314 >> v1:172.21.15.90:6809/1686184251 conn(0x7fd6f8063bd0 legacy=0x7fd6f81b5e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:38.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.655+0000 7fd6ddffb700 20 client.14874.objecter put_session s=0x7fd6f81abca0 osd=2 1 2024-04-14T15:12:38.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.655+0000 7fd6ddffb700 10 client.14874.objecter close_session for osd.3 2024-04-14T15:12:38.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.655+0000 7fd6ddffb700 1 -- 172.21.15.90:0/2196878314 >> v1:172.21.15.90:6813/2270166774 conn(0x7fd6f81b8920 legacy=0x7fd6f81b8d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:38.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.655+0000 7fd6ddffb700 20 client.14874.objecter put_session s=0x7fd6f80634c0 osd=3 1 2024-04-14T15:12:38.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.655+0000 7fd6ddffb700 10 client.14874.objecter close_session for osd.4 2024-04-14T15:12:38.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.655+0000 7fd6ddffb700 1 -- 172.21.15.90:0/2196878314 >> v1:172.21.15.92:6800/3887549733 conn(0x7fd6f81bd130 legacy=0x7fd6f81bf520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:38.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.655+0000 7fd6ddffb700 20 client.14874.objecter put_session s=0x7fd6f81b9e20 osd=4 1 2024-04-14T15:12:38.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.655+0000 7fd6ddffb700 10 client.14874.objecter close_session for osd.5 2024-04-14T15:12:38.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.655+0000 7fd6ddffb700 1 -- 172.21.15.90:0/2196878314 >> v1:172.21.15.92:6804/3244561440 conn(0x7fd6f81c2e50 legacy=0x7fd6f81c0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:38.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.655+0000 7fd6ddffb700 20 client.14874.objecter put_session s=0x7fd6f81c2730 osd=5 1 2024-04-14T15:12:38.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.655+0000 7fd6ddffb700 10 client.14874.objecter close_session for osd.6 2024-04-14T15:12:38.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.655+0000 7fd6ddffb700 1 -- 172.21.15.90:0/2196878314 >> v1:172.21.15.92:6808/636125963 conn(0x7fd6f81c5e40 legacy=0x7fd6f81c8230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:38.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.655+0000 7fd6ddffb700 20 client.14874.objecter put_session s=0x7fd6f81c3300 osd=6 1 2024-04-14T15:12:38.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.655+0000 7fd6ddffb700 10 client.14874.objecter close_session for osd.7 2024-04-14T15:12:38.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.655+0000 7fd6ddffb700 1 -- 172.21.15.90:0/2196878314 >> v1:172.21.15.92:6812/1773706716 conn(0x7fd6f81c9fd0 legacy=0x7fd6f81cc3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:38.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.655+0000 7fd6ddffb700 20 client.14874.objecter put_session s=0x7fd6f81c9950 osd=7 1 2024-04-14T15:12:38.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.655+0000 7fd6ddffb700 20 client.14874.objecter shutdown clearing up homeless session... 2024-04-14T15:12:38.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.655+0000 7fd6ddffb700 10 client.14874.objecter successfully canceled tick 2024-04-14T15:12:38.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.655+0000 7fd6f77fe700 1 -- 172.21.15.90:0/2196878314 reap_dead start 2024-04-14T15:12:38.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.655+0000 7fd6f77fe700 1 -- 172.21.15.90:0/2196878314 reap_dead start 2024-04-14T15:12:38.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.655+0000 7fd6f77fe700 1 -- 172.21.15.90:0/2196878314 reap_dead start 2024-04-14T15:12:38.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.655+0000 7fd6f77fe700 1 -- 172.21.15.90:0/2196878314 reap_dead start 2024-04-14T15:12:38.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.655+0000 7fd6f77fe700 1 -- 172.21.15.90:0/2196878314 reap_dead start 2024-04-14T15:12:38.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.655+0000 7fd6ddffb700 1 -- 172.21.15.90:0/2196878314 >> v1:172.21.15.90:6800/420935427 conn(0x7fd6e0070d40 legacy=0x7fd6e0073200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:38.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.655+0000 7fd6ddffb700 1 -- 172.21.15.90:0/2196878314 >> v1:172.21.15.90:6789/0 conn(0x7fd6f8111510 legacy=0x7fd6f81a8150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:38.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.655+0000 7fd6ddffb700 1 -- 172.21.15.90:0/2196878314 shutdown_connections 2024-04-14T15:12:38.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.655+0000 7fd6ddffb700 1 -- 172.21.15.90:0/2196878314 >> 172.21.15.90:0/2196878314 conn(0x7fd6f80fd290 msgr2=0x7fd6f80ff680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:12:38.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.655+0000 7fd6ddffb700 1 -- 172.21.15.90:0/2196878314 shutdown_connections 2024-04-14T15:12:38.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.655+0000 7fd6ddffb700 1 -- 172.21.15.90:0/2196878314 wait complete. 2024-04-14T15:12:38.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:38.655+0000 7fd6ddffb700 1 librados: shutdown 2024-04-14T15:12:38.659 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:12:38.659 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:12:38.659 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:12:38.659 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:12:38.659 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:12:38.659 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:12:38.659 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:12:38.660 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:12:38.709 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:12:38.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:38.734 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:38.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:12:38.835 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:12:38.863 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:38.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:38.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:12:38.913 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:38 smithi090 ceph-mon[41670]: pgmap v93: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:12:38.913 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:38 smithi090 ceph-mon[41670]: 2.1 scrub starts 2024-04-14T15:12:38.913 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:38 smithi090 ceph-mon[41670]: 2.1 scrub ok 2024-04-14T15:12:38.913 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:38 smithi090 ceph-mon[41670]: 6.19 scrub starts 2024-04-14T15:12:38.913 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:38 smithi090 ceph-mon[41670]: 6.19 scrub ok 2024-04-14T15:12:38.913 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:38 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:12:38.913 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:38 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:12:38.913 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:38 smithi090 ceph-mon[41670]: 4.a scrub starts 2024-04-14T15:12:38.913 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:38 smithi090 ceph-mon[41670]: 4.a scrub ok 2024-04-14T15:12:38.913 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:38 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2196878314' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:12:38.914 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:38 smithi090 ceph-mon[30399]: pgmap v93: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:12:38.914 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:38 smithi090 ceph-mon[30399]: 2.1 scrub starts 2024-04-14T15:12:38.914 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:38 smithi090 ceph-mon[30399]: 2.1 scrub ok 2024-04-14T15:12:38.914 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:38 smithi090 ceph-mon[30399]: 6.19 scrub starts 2024-04-14T15:12:38.914 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:38 smithi090 ceph-mon[30399]: 6.19 scrub ok 2024-04-14T15:12:38.914 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:38 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:12:38.914 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:38 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:12:38.914 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:38 smithi090 ceph-mon[30399]: 4.a scrub starts 2024-04-14T15:12:38.914 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:38 smithi090 ceph-mon[30399]: 4.a scrub ok 2024-04-14T15:12:38.914 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:38 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2196878314' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:12:38.965 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:12:38.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:38.992 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:38.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:12:39.093 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:12:39.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:39.119 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:39.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:12:39.220 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:12:39.243 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:39.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:39.244 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:12:39.345 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:12:39.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:39.371 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:39.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:12:39.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:38 smithi092 ceph-mon[33966]: pgmap v93: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:12:39.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:38 smithi092 ceph-mon[33966]: 2.1 scrub starts 2024-04-14T15:12:39.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:38 smithi092 ceph-mon[33966]: 2.1 scrub ok 2024-04-14T15:12:39.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:38 smithi092 ceph-mon[33966]: 6.19 scrub starts 2024-04-14T15:12:39.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:38 smithi092 ceph-mon[33966]: 6.19 scrub ok 2024-04-14T15:12:39.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:38 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:12:39.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:38 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:12:39.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:38 smithi092 ceph-mon[33966]: 4.a scrub starts 2024-04-14T15:12:39.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:38 smithi092 ceph-mon[33966]: 4.a scrub ok 2024-04-14T15:12:39.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:38 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2196878314' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:12:39.394 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-14T15:12:39.394 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:12:39.472 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:12:39.496 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:39.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:39.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:12:39.598 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:12:39.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:39.623 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:39.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:12:39.724 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:12:39.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:39.750 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:39.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:12:39.851 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:12:39.876 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:39.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:39.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:12:39.978 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:12:40.010 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:40.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:40.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:12:40.111 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:12:40.145 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:40.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:40.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:12:40.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:39 smithi090 ceph-mon[41670]: 5.a scrub starts 2024-04-14T15:12:40.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:39 smithi090 ceph-mon[41670]: 5.a scrub ok 2024-04-14T15:12:40.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:39 smithi090 ceph-mon[41670]: 5.1b scrub starts 2024-04-14T15:12:40.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:39 smithi090 ceph-mon[41670]: 5.1b scrub ok 2024-04-14T15:12:40.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:39 smithi090 ceph-mon[41670]: 5.14 scrub starts 2024-04-14T15:12:40.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:39 smithi090 ceph-mon[41670]: 5.14 scrub ok 2024-04-14T15:12:40.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:39 smithi090 ceph-mon[41670]: 3.1 scrub starts 2024-04-14T15:12:40.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:39 smithi090 ceph-mon[41670]: 3.1 scrub ok 2024-04-14T15:12:40.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:39 smithi090 ceph-mon[41670]: 5.1c scrub starts 2024-04-14T15:12:40.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:39 smithi090 ceph-mon[41670]: 5.1c scrub ok 2024-04-14T15:12:40.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:39 smithi090 ceph-mon[30399]: 5.a scrub starts 2024-04-14T15:12:40.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:39 smithi090 ceph-mon[30399]: 5.a scrub ok 2024-04-14T15:12:40.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:39 smithi090 ceph-mon[30399]: 5.1b scrub starts 2024-04-14T15:12:40.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:39 smithi090 ceph-mon[30399]: 5.1b scrub ok 2024-04-14T15:12:40.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:39 smithi090 ceph-mon[30399]: 5.14 scrub starts 2024-04-14T15:12:40.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:39 smithi090 ceph-mon[30399]: 5.14 scrub ok 2024-04-14T15:12:40.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:39 smithi090 ceph-mon[30399]: 3.1 scrub starts 2024-04-14T15:12:40.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:39 smithi090 ceph-mon[30399]: 3.1 scrub ok 2024-04-14T15:12:40.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:39 smithi090 ceph-mon[30399]: 5.1c scrub starts 2024-04-14T15:12:40.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:39 smithi090 ceph-mon[30399]: 5.1c scrub ok 2024-04-14T15:12:40.247 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:12:40.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:40.274 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:40.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:12:40.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:39 smithi092 ceph-mon[33966]: 5.a scrub starts 2024-04-14T15:12:40.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:39 smithi092 ceph-mon[33966]: 5.a scrub ok 2024-04-14T15:12:40.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:39 smithi092 ceph-mon[33966]: 5.1b scrub starts 2024-04-14T15:12:40.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:39 smithi092 ceph-mon[33966]: 5.1b scrub ok 2024-04-14T15:12:40.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:39 smithi092 ceph-mon[33966]: 5.14 scrub starts 2024-04-14T15:12:40.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:39 smithi092 ceph-mon[33966]: 5.14 scrub ok 2024-04-14T15:12:40.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:39 smithi092 ceph-mon[33966]: 3.1 scrub starts 2024-04-14T15:12:40.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:39 smithi092 ceph-mon[33966]: 3.1 scrub ok 2024-04-14T15:12:40.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:39 smithi092 ceph-mon[33966]: 5.1c scrub starts 2024-04-14T15:12:40.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:39 smithi092 ceph-mon[33966]: 5.1c scrub ok 2024-04-14T15:12:40.376 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:12:40.403 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:40.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:40.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:12:40.505 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:12:40.532 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:40.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:40.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:12:40.634 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:12:40.661 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:40.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:40.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:12:40.763 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:12:40.804 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:40.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:40.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:12:40.906 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:12:40.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:40.931 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:40.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:12:41.001 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:40 smithi090 ceph-mon[30399]: pgmap v94: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-14T15:12:41.001 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:40 smithi090 ceph-mon[30399]: 5.6 scrub starts 2024-04-14T15:12:41.001 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:40 smithi090 ceph-mon[30399]: 5.6 scrub ok 2024-04-14T15:12:41.001 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:40 smithi090 ceph-mon[30399]: 4.e scrub starts 2024-04-14T15:12:41.001 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:40 smithi090 ceph-mon[30399]: 4.e scrub ok 2024-04-14T15:12:41.001 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:40 smithi090 ceph-mon[41670]: pgmap v94: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-14T15:12:41.001 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:40 smithi090 ceph-mon[41670]: 5.6 scrub starts 2024-04-14T15:12:41.001 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:40 smithi090 ceph-mon[41670]: 5.6 scrub ok 2024-04-14T15:12:41.001 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:40 smithi090 ceph-mon[41670]: 4.e scrub starts 2024-04-14T15:12:41.001 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:40 smithi090 ceph-mon[41670]: 4.e scrub ok 2024-04-14T15:12:41.032 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:12:41.060 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:41.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:41.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:12:41.162 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:12:41.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:41.188 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:41.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:12:41.219 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-14T15:12:41.290 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:12:41.317 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:41.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:41.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:12:41.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:40 smithi092 ceph-mon[33966]: pgmap v94: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-14T15:12:41.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:40 smithi092 ceph-mon[33966]: 5.6 scrub starts 2024-04-14T15:12:41.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:40 smithi092 ceph-mon[33966]: 5.6 scrub ok 2024-04-14T15:12:41.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:40 smithi092 ceph-mon[33966]: 4.e scrub starts 2024-04-14T15:12:41.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:40 smithi092 ceph-mon[33966]: 4.e scrub ok 2024-04-14T15:12:41.420 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:12:41.452 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:41.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:41.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:12:41.554 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:12:41.582 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:41.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:41.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:12:41.684 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:12:41.712 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:41.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:41.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:12:41.813 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:12:41.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:41.848 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:41.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:12:41.949 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:12:41.978 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:41.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:41.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:12:42.059 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:41 smithi090 ceph-mon[41670]: 5.15 scrub starts 2024-04-14T15:12:42.059 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:41 smithi090 ceph-mon[41670]: 5.15 scrub ok 2024-04-14T15:12:42.059 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:41 smithi090 ceph-mon[41670]: 2.0 scrub starts 2024-04-14T15:12:42.059 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:41 smithi090 ceph-mon[41670]: 2.0 scrub ok 2024-04-14T15:12:42.059 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:41 smithi090 ceph-mon[41670]: 3.0 scrub starts 2024-04-14T15:12:42.059 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:41 smithi090 ceph-mon[41670]: 3.0 scrub ok 2024-04-14T15:12:42.060 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:41 smithi090 ceph-mon[30399]: 5.15 scrub starts 2024-04-14T15:12:42.060 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:41 smithi090 ceph-mon[30399]: 5.15 scrub ok 2024-04-14T15:12:42.060 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:41 smithi090 ceph-mon[30399]: 2.0 scrub starts 2024-04-14T15:12:42.060 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:41 smithi090 ceph-mon[30399]: 2.0 scrub ok 2024-04-14T15:12:42.060 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:41 smithi090 ceph-mon[30399]: 3.0 scrub starts 2024-04-14T15:12:42.060 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:41 smithi090 ceph-mon[30399]: 3.0 scrub ok 2024-04-14T15:12:42.079 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:12:42.114 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:42.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:42.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:12:42.216 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:12:42.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:42.240 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:42.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:12:42.341 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:12:42.369 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:42.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:42.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:12:42.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:41 smithi092 ceph-mon[33966]: 5.15 scrub starts 2024-04-14T15:12:42.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:41 smithi092 ceph-mon[33966]: 5.15 scrub ok 2024-04-14T15:12:42.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:41 smithi092 ceph-mon[33966]: 2.0 scrub starts 2024-04-14T15:12:42.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:41 smithi092 ceph-mon[33966]: 2.0 scrub ok 2024-04-14T15:12:42.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:41 smithi092 ceph-mon[33966]: 3.0 scrub starts 2024-04-14T15:12:42.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:41 smithi092 ceph-mon[33966]: 3.0 scrub ok 2024-04-14T15:12:42.471 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:12:42.504 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:42.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:42.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:12:42.606 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:12:42.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.616+0000 7f8eb9345700 1 -- 172.21.15.90:0/2016633810 >> v1:172.21.15.90:6789/0 conn(0x7f8eb410e360 legacy=0x7f8eb4110820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:42.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.616+0000 7f8eb9345700 1 -- 172.21.15.90:0/2016633810 shutdown_connections 2024-04-14T15:12:42.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.616+0000 7f8eb9345700 1 -- 172.21.15.90:0/2016633810 >> 172.21.15.90:0/2016633810 conn(0x7f8eb40fd2c0 msgr2=0x7f8eb40ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:12:42.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.616+0000 7f8eb9345700 1 -- 172.21.15.90:0/2016633810 shutdown_connections 2024-04-14T15:12:42.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.616+0000 7f8eb9345700 1 -- 172.21.15.90:0/2016633810 wait complete. 2024-04-14T15:12:42.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.616+0000 7f8eb9345700 1 librados: starting msgr at 2024-04-14T15:12:42.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.616+0000 7f8eb9345700 1 librados: starting objecter 2024-04-14T15:12:42.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.617+0000 7f8eb9345700 1 Processor -- start 2024-04-14T15:12:42.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.617+0000 7f8eb9345700 1 -- start start 2024-04-14T15:12:42.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.617+0000 7f8eb9345700 1 librados: setting wanted keys 2024-04-14T15:12:42.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.617+0000 7f8eb9345700 1 librados: calling monclient init 2024-04-14T15:12:42.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.617+0000 7f8eb2ffd700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f8eb41078e0 0x7f8eb410ab70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:49398/0 (socket says 172.21.15.90:49398) 2024-04-14T15:12:42.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.617+0000 7f8eb2ffd700 1 -- 172.21.15.90:0/1999173477 learned_addr learned my addr 172.21.15.90:0/1999173477 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:12:42.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.617+0000 7f8e9bfff700 10 client.?.objecter ms_handle_connect 0x7f8eb41078e0 2024-04-14T15:12:42.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.617+0000 7f8e9bfff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:12:42.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.617+0000 7f8eb9345700 1 -- 172.21.15.90:0/1999173477 --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f8eb410e030 con 0x7f8eb41078e0 2024-04-14T15:12:42.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.617+0000 7f8eb9345700 1 -- 172.21.15.90:0/1999173477 --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f8eb41a5f50 con 0x7f8eb410e360 2024-04-14T15:12:42.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.617+0000 7f8eb9345700 1 -- 172.21.15.90:0/1999173477 --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f8eb41a7130 con 0x7f8eb410a790 2024-04-14T15:12:42.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.618+0000 7f8e9bfff700 1 -- 172.21.15.90:0/1999173477 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 751565460 0 0) 0x7f8eb410e030 con 0x7f8eb41078e0 2024-04-14T15:12:42.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.618+0000 7f8e9bfff700 1 -- 172.21.15.90:0/1999173477 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e9c0035b0 con 0x7f8eb41078e0 2024-04-14T15:12:42.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.618+0000 7f8e9bfff700 10 client.?.objecter ms_handle_connect 0x7f8eb410a790 2024-04-14T15:12:42.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.618+0000 7f8e9bfff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:12:42.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.618+0000 7f8e9bfff700 1 -- 172.21.15.90:0/1999173477 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2600390828 0 0) 0x7f8e9c0035b0 con 0x7f8eb41078e0 2024-04-14T15:12:42.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.618+0000 7f8e9bfff700 1 -- 172.21.15.90:0/1999173477 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8eb410e030 con 0x7f8eb41078e0 2024-04-14T15:12:42.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.618+0000 7f8e9bfff700 1 -- 172.21.15.90:0/1999173477 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f8ea4001930 con 0x7f8eb41078e0 2024-04-14T15:12:42.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.618+0000 7f8e9bfff700 10 client.?.objecter ms_handle_connect 0x7f8eb410e360 2024-04-14T15:12:42.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.618+0000 7f8e9bfff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:12:42.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.618+0000 7f8e9bfff700 1 -- 172.21.15.90:0/1999173477 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 407201488 0 0) 0x7f8eb41a7130 con 0x7f8eb410a790 2024-04-14T15:12:42.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.618+0000 7f8e9bfff700 1 -- 172.21.15.90:0/1999173477 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e9c0035b0 con 0x7f8eb410a790 2024-04-14T15:12:42.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.618+0000 7f8e9bfff700 1 -- 172.21.15.90:0/1999173477 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1907179050 0 0) 0x7f8eb410e030 con 0x7f8eb41078e0 2024-04-14T15:12:42.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.618+0000 7f8e9bfff700 1 -- 172.21.15.90:0/1999173477 >> v1:172.21.15.90:6790/0 conn(0x7f8eb410a790 legacy=0x7f8eb410b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:42.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.618+0000 7f8e9bfff700 1 -- 172.21.15.90:0/1999173477 >> v1:172.21.15.92:6789/0 conn(0x7f8eb410e360 legacy=0x7f8eb41a4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:42.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.618+0000 7f8e9bfff700 1 -- 172.21.15.90:0/1999173477 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8eb41b9400 con 0x7f8eb41078e0 2024-04-14T15:12:42.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.618+0000 7f8eb9345700 1 -- 172.21.15.90:0/1999173477 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8eb41a6180 con 0x7f8eb41078e0 2024-04-14T15:12:42.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.618+0000 7f8eb9345700 10 client.14880.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:12:42.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.618+0000 7f8eb9345700 1 -- 172.21.15.90:0/1999173477 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f8eb41a6640 con 0x7f8eb41078e0 2024-04-14T15:12:42.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.619+0000 7f8eb9345700 1 librados: init done 2024-04-14T15:12:42.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.619+0000 7f8e9bfff700 1 -- 172.21.15.90:0/1999173477 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f8ea4001cd0 con 0x7f8eb41078e0 2024-04-14T15:12:42.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.619+0000 7f8e9bfff700 1 -- 172.21.15.90:0/1999173477 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f8ea4005d90 con 0x7f8eb41078e0 2024-04-14T15:12:42.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.620+0000 7f8eb9345700 1 -- 172.21.15.90:0/1999173477 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ea0005120 con 0x7f8eb41078e0 2024-04-14T15:12:42.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.621+0000 7f8e9bfff700 1 -- 172.21.15.90:0/1999173477 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f8ea40039d0 con 0x7f8eb41078e0 2024-04-14T15:12:42.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.622+0000 7f8e9bfff700 1 -- 172.21.15.90:0/1999173477 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(103..103 src has 51..103) ==== 6320+0+0 (unknown 2346850074 0 0) 0x7f8ea408a580 con 0x7f8eb41078e0 2024-04-14T15:12:42.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.622+0000 7f8e9bfff700 10 client.14880.objecter ms_dispatch 0x7f8eb4001550 osd_map(103..103 src has 51..103) 2024-04-14T15:12:42.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.622+0000 7f8e9bfff700 3 client.14880.objecter handle_osd_map got epochs [103,103] > 0 2024-04-14T15:12:42.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.622+0000 7f8e9bfff700 3 client.14880.objecter handle_osd_map decoding full epoch 103 2024-04-14T15:12:42.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.622+0000 7f8e9bfff700 20 client.14880.objecter dump_active .. 0 homeless 2024-04-14T15:12:42.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.622+0000 7f8e9bfff700 10 client.14880.objecter ms_handle_connect 0x7f8e9c071420 2024-04-14T15:12:42.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.630+0000 7f8e9bfff700 1 -- 172.21.15.90:0/1999173477 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f8ea40576d0 con 0x7f8eb41078e0 2024-04-14T15:12:42.634 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:42.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:42.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:12:42.736 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:12:42.752 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-14T15:12:42.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:42.761 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:42.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:12:42.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.851+0000 7facd44df700 1 -- 172.21.15.90:0/1808139193 >> v1:172.21.15.90:6789/0 conn(0x7faccc111040 legacy=0x7faccc10ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:42.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.851+0000 7facd44df700 1 -- 172.21.15.90:0/1808139193 shutdown_connections 2024-04-14T15:12:42.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.851+0000 7facd44df700 1 -- 172.21.15.90:0/1808139193 >> 172.21.15.90:0/1808139193 conn(0x7faccc06d990 msgr2=0x7faccc06dda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:12:42.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.851+0000 7facd44df700 1 -- 172.21.15.90:0/1808139193 shutdown_connections 2024-04-14T15:12:42.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.851+0000 7facd44df700 1 -- 172.21.15.90:0/1808139193 wait complete. 2024-04-14T15:12:42.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.852+0000 7facd44df700 1 librados: starting msgr at 2024-04-14T15:12:42.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.852+0000 7facd44df700 1 librados: starting objecter 2024-04-14T15:12:42.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.852+0000 7facd44df700 1 Processor -- start 2024-04-14T15:12:42.856 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:12:42.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.852+0000 7facd44df700 1 -- start start 2024-04-14T15:12:42.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.852+0000 7facd44df700 1 librados: setting wanted keys 2024-04-14T15:12:42.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.852+0000 7facd44df700 1 librados: calling monclient init 2024-04-14T15:12:42.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.852+0000 7facd44df700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7faccc10f7a0 con 0x7faccc111040 2024-04-14T15:12:42.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.852+0000 7facd44df700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7faccc1b7d50 con 0x7faccc1114e0 2024-04-14T15:12:42.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.852+0000 7facd44df700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7faccc1b8f30 con 0x7faccc11e2b0 2024-04-14T15:12:42.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.852+0000 7facd227b700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7faccc111040 0x7faccc10c640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:49426/0 (socket says 172.21.15.90:49426) 2024-04-14T15:12:42.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.852+0000 7facd227b700 1 -- 172.21.15.90:0/3733252215 learned_addr learned my addr 172.21.15.90:0/3733252215 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:12:42.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.853+0000 7facc37fe700 10 client.?.objecter ms_handle_connect 0x7faccc1114e0 2024-04-14T15:12:42.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.853+0000 7facc37fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:12:42.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.853+0000 7facc37fe700 1 -- 172.21.15.90:0/3733252215 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3506061846 0 0) 0x7faccc1b7d50 con 0x7faccc1114e0 2024-04-14T15:12:42.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.853+0000 7facc37fe700 1 -- 172.21.15.90:0/3733252215 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7facb80035b0 con 0x7faccc1114e0 2024-04-14T15:12:42.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.853+0000 7facc37fe700 10 client.?.objecter ms_handle_connect 0x7faccc111040 2024-04-14T15:12:42.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.853+0000 7facc37fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:12:42.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.853+0000 7facc37fe700 1 -- 172.21.15.90:0/3733252215 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3724542510 0 0) 0x7facb80035b0 con 0x7faccc1114e0 2024-04-14T15:12:42.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.853+0000 7facc37fe700 1 -- 172.21.15.90:0/3733252215 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faccc1b7d50 con 0x7faccc1114e0 2024-04-14T15:12:42.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.853+0000 7facc37fe700 1 -- 172.21.15.90:0/3733252215 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7facc8001760 con 0x7faccc1114e0 2024-04-14T15:12:42.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.854+0000 7facc37fe700 1 -- 172.21.15.90:0/3733252215 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 323752100 0 0) 0x7faccc1b7d50 con 0x7faccc1114e0 2024-04-14T15:12:42.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.854+0000 7facc37fe700 1 -- 172.21.15.90:0/3733252215 >> v1:172.21.15.90:6790/0 conn(0x7faccc11e2b0 legacy=0x7faccc1b5620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:42.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.854+0000 7facc37fe700 1 -- 172.21.15.90:0/3733252215 >> v1:172.21.15.90:6789/0 conn(0x7faccc111040 legacy=0x7faccc10c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:42.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.854+0000 7facc37fe700 1 -- 172.21.15.90:0/3733252215 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faccc1ba110 con 0x7faccc1114e0 2024-04-14T15:12:42.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.854+0000 7facc37fe700 10 client.?.objecter ms_handle_connect 0x7faccc11e2b0 2024-04-14T15:12:42.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.854+0000 7facc37fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:12:42.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.854+0000 7facd44df700 1 -- 172.21.15.90:0/3733252215 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7faccc1b6da0 con 0x7faccc1114e0 2024-04-14T15:12:42.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.854+0000 7facd44df700 10 client.24772.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:12:42.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.854+0000 7facd44df700 1 -- 172.21.15.90:0/3733252215 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7faccc1b7260 con 0x7faccc1114e0 2024-04-14T15:12:42.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.854+0000 7facd44df700 1 librados: init done 2024-04-14T15:12:42.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.854+0000 7facc37fe700 1 -- 172.21.15.90:0/3733252215 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7facc8003b40 con 0x7faccc1114e0 2024-04-14T15:12:42.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.854+0000 7facc37fe700 1 -- 172.21.15.90:0/3733252215 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7facc8004a60 con 0x7faccc1114e0 2024-04-14T15:12:42.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.858+0000 7facd44df700 1 -- 172.21.15.90:0/3733252215 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7facb0005120 con 0x7faccc1114e0 2024-04-14T15:12:42.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.858+0000 7facc37fe700 1 -- 172.21.15.90:0/3733252215 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7facc8003680 con 0x7faccc1114e0 2024-04-14T15:12:42.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.858+0000 7facc37fe700 1 -- 172.21.15.90:0/3733252215 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(103..103 src has 51..103) ==== 6320+0+0 (unknown 2346850074 0 0) 0x7facc80890b0 con 0x7faccc1114e0 2024-04-14T15:12:42.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.858+0000 7facc37fe700 10 client.24772.objecter ms_dispatch 0x7faccc001550 osd_map(103..103 src has 51..103) 2024-04-14T15:12:42.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.858+0000 7facc37fe700 3 client.24772.objecter handle_osd_map got epochs [103,103] > 0 2024-04-14T15:12:42.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.858+0000 7facc37fe700 3 client.24772.objecter handle_osd_map decoding full epoch 103 2024-04-14T15:12:42.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.859+0000 7facc37fe700 20 client.24772.objecter dump_active .. 0 homeless 2024-04-14T15:12:42.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.859+0000 7facc37fe700 10 client.24772.objecter ms_handle_connect 0x7facb80711f0 2024-04-14T15:12:42.862 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:12:42.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.866+0000 7facc37fe700 1 -- 172.21.15.90:0/3733252215 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7facc8056200 con 0x7faccc1114e0 2024-04-14T15:12:42.901 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:42 smithi090 ceph-mon[30399]: pgmap v95: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:12:42.901 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:42 smithi090 ceph-mon[30399]: 6.14 scrub starts 2024-04-14T15:12:42.901 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:42 smithi090 ceph-mon[30399]: 6.14 scrub ok 2024-04-14T15:12:42.901 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:42 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:12:42.902 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:42 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:12:42.902 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:42 smithi090 ceph-mon[30399]: osdmap e103: 8 total, 8 up, 8 in 2024-04-14T15:12:42.902 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:42 smithi090 ceph-mon[30399]: 6.13 scrub starts 2024-04-14T15:12:42.902 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:42 smithi090 ceph-mon[30399]: 6.13 scrub ok 2024-04-14T15:12:42.902 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:42 smithi090 ceph-mon[30399]: 4.f scrub starts 2024-04-14T15:12:42.902 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:42 smithi090 ceph-mon[30399]: 4.f scrub ok 2024-04-14T15:12:42.902 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:42 smithi090 ceph-mon[30399]: 4.9 scrub starts 2024-04-14T15:12:42.902 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:42 smithi090 ceph-mon[30399]: 4.9 scrub ok 2024-04-14T15:12:42.902 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:42 smithi090 ceph-mon[41670]: pgmap v95: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:12:42.902 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:42 smithi090 ceph-mon[41670]: 6.14 scrub starts 2024-04-14T15:12:42.903 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:42 smithi090 ceph-mon[41670]: 6.14 scrub ok 2024-04-14T15:12:42.903 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:42 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:12:42.903 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:42 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:12:42.903 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:42 smithi090 ceph-mon[41670]: osdmap e103: 8 total, 8 up, 8 in 2024-04-14T15:12:42.903 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:42 smithi090 ceph-mon[41670]: 6.13 scrub starts 2024-04-14T15:12:42.903 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:42 smithi090 ceph-mon[41670]: 6.13 scrub ok 2024-04-14T15:12:42.903 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:42 smithi090 ceph-mon[41670]: 4.f scrub starts 2024-04-14T15:12:42.903 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:42 smithi090 ceph-mon[41670]: 4.f scrub ok 2024-04-14T15:12:42.903 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:42 smithi090 ceph-mon[41670]: 4.9 scrub starts 2024-04-14T15:12:42.903 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:42 smithi090 ceph-mon[41670]: 4.9 scrub ok 2024-04-14T15:12:42.907 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:42.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:42.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:12:42.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:42.945+0000 7f8eb9345700 1 -- 172.21.15.90:0/1999173477 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} v 0) -- 0x7f8ea0005410 con 0x7f8eb41078e0 2024-04-14T15:12:43.009 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:12:43.037 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:43.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:43.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:12:43.139 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:12:43.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:43.178 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:43.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:12:43.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:43.199+0000 7facd44df700 1 -- 172.21.15.90:0/3733252215 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7facb0005410 con 0x7faccc1114e0 2024-04-14T15:12:43.279 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:12:43.307 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:43.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:43.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:12:43.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:42 smithi092 ceph-mon[33966]: pgmap v95: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:12:43.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:42 smithi092 ceph-mon[33966]: 6.14 scrub starts 2024-04-14T15:12:43.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:42 smithi092 ceph-mon[33966]: 6.14 scrub ok 2024-04-14T15:12:43.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:42 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:12:43.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:42 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:12:43.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:42 smithi092 ceph-mon[33966]: osdmap e103: 8 total, 8 up, 8 in 2024-04-14T15:12:43.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:42 smithi092 ceph-mon[33966]: 6.13 scrub starts 2024-04-14T15:12:43.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:42 smithi092 ceph-mon[33966]: 6.13 scrub ok 2024-04-14T15:12:43.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:42 smithi092 ceph-mon[33966]: 4.f scrub starts 2024-04-14T15:12:43.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:42 smithi092 ceph-mon[33966]: 4.f scrub ok 2024-04-14T15:12:43.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:42 smithi092 ceph-mon[33966]: 4.9 scrub starts 2024-04-14T15:12:43.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:42 smithi092 ceph-mon[33966]: 4.9 scrub ok 2024-04-14T15:12:43.409 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:12:43.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:43.434 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:43.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:12:43.535 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:12:43.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:43.564 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:43.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:12:43.665 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:12:43.681 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:12:43.693 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:43.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:43.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:12:43.795 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:12:43.823 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:43.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:43.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:12:43.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:43.898+0000 7f8e9bfff700 1 -- 172.21.15.90:0/1999173477 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]=0 set pool 1 pgp_num to 1 v104) ==== 129+0+0 (unknown 4204382599 0 0) 0x7f8ea405b380 con 0x7f8eb41078e0 2024-04-14T15:12:43.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:43.901+0000 7f8eb9345700 10 librados: watch_flush enter 2024-04-14T15:12:43.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:43.901+0000 7f8eb9345700 10 librados: watch_flush exit 2024-04-14T15:12:43.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:43.901+0000 7f8eb9345700 20 client.14880.objecter shutdown clearing up homeless session... 2024-04-14T15:12:43.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:43.901+0000 7f8eb9345700 10 client.14880.objecter successfully canceled tick 2024-04-14T15:12:43.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:43.902+0000 7f8eb9345700 1 -- 172.21.15.90:0/1999173477 >> v1:172.21.15.90:6800/420935427 conn(0x7f8e9c071420 legacy=0x7f8e9c0738e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:43.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:43.902+0000 7f8eb9345700 1 -- 172.21.15.90:0/1999173477 >> v1:172.21.15.90:6789/0 conn(0x7f8eb41078e0 legacy=0x7f8eb410ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:43.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:43.902+0000 7f8eb9345700 1 -- 172.21.15.90:0/1999173477 shutdown_connections 2024-04-14T15:12:43.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:43.902+0000 7f8eb9345700 1 -- 172.21.15.90:0/1999173477 >> 172.21.15.90:0/1999173477 conn(0x7f8eb40fd2c0 msgr2=0x7f8eb406b590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:12:43.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:43.903+0000 7f8eb9345700 1 -- 172.21.15.90:0/1999173477 shutdown_connections 2024-04-14T15:12:43.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:43.906+0000 7f8eb9345700 1 -- 172.21.15.90:0/1999173477 wait complete. 2024-04-14T15:12:43.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:43.906+0000 7f8eb9345700 1 librados: shutdown 2024-04-14T15:12:43.907 INFO:teuthology.orchestra.run.smithi090.stderr:set pool 1 pgp_num to 1 2024-04-14T15:12:43.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:43.910+0000 7facc37fe700 1 -- 172.21.15.90:0/3733252215 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v104) ==== 87+0+0 (unknown 1783246304 0 0) 0x7facc8059eb0 con 0x7faccc1114e0 2024-04-14T15:12:43.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:43.913+0000 7facd44df700 10 librados: watch_flush enter 2024-04-14T15:12:43.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:43.913+0000 7facd44df700 10 librados: watch_flush exit 2024-04-14T15:12:43.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:43.913+0000 7facd44df700 20 client.24772.objecter shutdown clearing up homeless session... 2024-04-14T15:12:43.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:43.913+0000 7facd44df700 10 client.24772.objecter successfully canceled tick 2024-04-14T15:12:43.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:43.913+0000 7facd44df700 1 -- 172.21.15.90:0/3733252215 >> v1:172.21.15.90:6800/420935427 conn(0x7facb80711f0 legacy=0x7facb80736b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:43.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:43.913+0000 7facd44df700 1 -- 172.21.15.90:0/3733252215 >> v1:172.21.15.92:6789/0 conn(0x7faccc1114e0 legacy=0x7faccc10cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:43.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:43.913+0000 7facd44df700 1 -- 172.21.15.90:0/3733252215 shutdown_connections 2024-04-14T15:12:43.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:43.913+0000 7facd44df700 1 -- 172.21.15.90:0/3733252215 >> 172.21.15.90:0/3733252215 conn(0x7faccc06d990 msgr2=0x7faccc073230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:12:43.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:43.913+0000 7facd44df700 1 -- 172.21.15.90:0/3733252215 shutdown_connections 2024-04-14T15:12:43.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:43.913+0000 7facd44df700 1 -- 172.21.15.90:0/3733252215 wait complete. 2024-04-14T15:12:43.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:43.913+0000 7facd44df700 1 librados: shutdown 2024-04-14T15:12:43.915 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is set 2024-04-14T15:12:43.925 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:12:43.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:43.964 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:43.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:12:44.065 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:12:44.096 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:44.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:44.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:12:44.098 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:12:43 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:12:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:12:44.099 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:43 smithi090 ceph-mon[41670]: 6.f scrub starts 2024-04-14T15:12:44.099 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:43 smithi090 ceph-mon[41670]: 6.f scrub ok 2024-04-14T15:12:44.099 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:43 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1999173477' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-04-14T15:12:44.099 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:43 smithi090 ceph-mon[41670]: 2.8 scrub starts 2024-04-14T15:12:44.099 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:43 smithi090 ceph-mon[41670]: 2.8 scrub ok 2024-04-14T15:12:44.099 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:43 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3733252215' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:12:44.099 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:43 smithi090 ceph-mon[41670]: from='client.24772 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:12:44.099 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:43 smithi090 ceph-mon[30399]: 6.f scrub starts 2024-04-14T15:12:44.100 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:43 smithi090 ceph-mon[30399]: 6.f scrub ok 2024-04-14T15:12:44.100 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:43 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1999173477' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-04-14T15:12:44.100 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:43 smithi090 ceph-mon[30399]: 2.8 scrub starts 2024-04-14T15:12:44.100 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:43 smithi090 ceph-mon[30399]: 2.8 scrub ok 2024-04-14T15:12:44.100 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:43 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3733252215' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:12:44.100 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:43 smithi090 ceph-mon[30399]: from='client.24772 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:12:44.198 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:12:44.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:44.227 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:44.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:12:44.328 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:12:44.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:44.363 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:44.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:12:44.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:43 smithi092 ceph-mon[33966]: 6.f scrub starts 2024-04-14T15:12:44.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:43 smithi092 ceph-mon[33966]: 6.f scrub ok 2024-04-14T15:12:44.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:43 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1999173477' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-04-14T15:12:44.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:43 smithi092 ceph-mon[33966]: 2.8 scrub starts 2024-04-14T15:12:44.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:43 smithi092 ceph-mon[33966]: 2.8 scrub ok 2024-04-14T15:12:44.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:43 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3733252215' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:12:44.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:43 smithi092 ceph-mon[33966]: from='client.24772 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:12:44.464 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:12:44.491 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:44.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:44.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:12:44.593 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:12:44.621 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:44.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:44.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:12:44.723 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:12:44.750 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:44.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:44.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:12:44.852 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:12:44.885 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:44.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:44.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:12:44.987 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:12:45.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:45.013 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:45.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:12:45.114 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:12:45.122 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:12:44 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:12:45.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:44 smithi092 ceph-mon[33966]: pgmap v97: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:12:45.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:44 smithi092 ceph-mon[33966]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:12:45.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:44 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1999173477' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-04-14T15:12:45.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:44 smithi092 ceph-mon[33966]: from='client.24772 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-14T15:12:45.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:44 smithi092 ceph-mon[33966]: osdmap e104: 8 total, 8 up, 8 in 2024-04-14T15:12:45.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:44 smithi092 ceph-mon[33966]: 2.1a scrub starts 2024-04-14T15:12:45.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:44 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:12:45.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:44 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:12:45.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:45.141 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:45.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:12:45.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:44 smithi090 ceph-mon[30399]: pgmap v97: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:12:45.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:44 smithi090 ceph-mon[30399]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:12:45.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:44 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1999173477' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-04-14T15:12:45.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:44 smithi090 ceph-mon[30399]: from='client.24772 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-14T15:12:45.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:44 smithi090 ceph-mon[30399]: osdmap e104: 8 total, 8 up, 8 in 2024-04-14T15:12:45.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:44 smithi090 ceph-mon[30399]: 2.1a scrub starts 2024-04-14T15:12:45.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:44 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:12:45.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:44 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:12:45.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:44 smithi090 ceph-mon[41670]: pgmap v97: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:12:45.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:44 smithi090 ceph-mon[41670]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:12:45.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:44 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1999173477' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-04-14T15:12:45.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:44 smithi090 ceph-mon[41670]: from='client.24772 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-14T15:12:45.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:44 smithi090 ceph-mon[41670]: osdmap e104: 8 total, 8 up, 8 in 2024-04-14T15:12:45.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:44 smithi090 ceph-mon[41670]: 2.1a scrub starts 2024-04-14T15:12:45.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:44 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:12:45.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:44 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:12:45.242 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:12:45.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:45.266 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:45.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:12:45.367 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:12:45.395 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:45.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:45.396 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:12:45.497 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:12:45.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:45.523 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:45.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:12:45.624 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:12:45.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:45.651 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:45.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:12:45.752 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:12:45.780 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:45.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:45.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:12:45.882 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:12:45.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:45.911 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:45.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:12:46.012 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:12:46.044 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:46.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:46.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:12:46.124 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:45 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:12:46.124 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:45 smithi090 ceph-mon[41670]: osdmap e105: 8 total, 8 up, 8 in 2024-04-14T15:12:46.124 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:45 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:12:46.124 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:45 smithi090 ceph-mon[30399]: osdmap e105: 8 total, 8 up, 8 in 2024-04-14T15:12:46.147 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:12:46.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:46.171 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:46.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:12:46.273 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:12:46.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:46.301 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:46.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:12:46.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:45 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:12:46.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:45 smithi092 ceph-mon[33966]: osdmap e105: 8 total, 8 up, 8 in 2024-04-14T15:12:46.403 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:12:46.435 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:46.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:46.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:12:46.536 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:12:46.564 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:46.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:46.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:12:46.667 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:12:46.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:46.694 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:46.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:12:46.795 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:12:46.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:46.822 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:46.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:12:46.923 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:12:46.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:46 smithi090 ceph-mon[30399]: pgmap v100: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:12:46.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:46 smithi090 ceph-mon[41670]: pgmap v100: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:12:46.950 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:46.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:46.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:12:47.052 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:12:47.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:47.082 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:47.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:12:47.183 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:12:47.215 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:47.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:47.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:12:47.317 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:12:47.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:47.343 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:47.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:12:47.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:46 smithi092 ceph-mon[33966]: pgmap v100: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:12:47.443 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:12:47.448 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-14T15:12:47.448 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:12:47.471 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:47.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:47.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:12:47.573 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:12:47.605 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:47.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:47.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:12:47.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.660+0000 7f8474e8d700 1 -- 172.21.15.90:0/439218971 >> v1:172.21.15.90:6789/0 conn(0x7f847010d940 legacy=0x7f847010fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:47.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.660+0000 7f8474e8d700 1 -- 172.21.15.90:0/439218971 shutdown_connections 2024-04-14T15:12:47.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.660+0000 7f8474e8d700 1 -- 172.21.15.90:0/439218971 >> 172.21.15.90:0/439218971 conn(0x7f8470076aa0 msgr2=0x7f8470076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:12:47.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.660+0000 7f8474e8d700 1 -- 172.21.15.90:0/439218971 shutdown_connections 2024-04-14T15:12:47.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.660+0000 7f8474e8d700 1 -- 172.21.15.90:0/439218971 wait complete. 2024-04-14T15:12:47.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.661+0000 7f8474e8d700 1 librados: starting msgr at 2024-04-14T15:12:47.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.661+0000 7f8474e8d700 1 librados: starting objecter 2024-04-14T15:12:47.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.661+0000 7f8474e8d700 1 Processor -- start 2024-04-14T15:12:47.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.662+0000 7f8474e8d700 1 -- start start 2024-04-14T15:12:47.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.662+0000 7f8474e8d700 1 librados: setting wanted keys 2024-04-14T15:12:47.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.662+0000 7f8474e8d700 1 librados: calling monclient init 2024-04-14T15:12:47.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.662+0000 7f8474e8d700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f84701a5770 con 0x7f8470109da0 2024-04-14T15:12:47.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.662+0000 7f846dd9b700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f8470109da0 0x7f84700fe880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:49442/0 (socket says 172.21.15.90:49442) 2024-04-14T15:12:47.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.662+0000 7f846dd9b700 1 -- 172.21.15.90:0/2735371829 learned_addr learned my addr 172.21.15.90:0/2735371829 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:12:47.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.662+0000 7f8474e8d700 1 -- 172.21.15.90:0/2735371829 --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f84701b79b0 con 0x7f8470106f80 2024-04-14T15:12:47.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.662+0000 7f845f7fe700 10 client.?.objecter ms_handle_connect 0x7f847010d940 2024-04-14T15:12:47.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.662+0000 7f845f7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:12:47.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.662+0000 7f845f7fe700 10 client.?.objecter ms_handle_connect 0x7f8470109da0 2024-04-14T15:12:47.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.662+0000 7f845f7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:12:47.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.662+0000 7f845f7fe700 10 client.?.objecter ms_handle_connect 0x7f8470106f80 2024-04-14T15:12:47.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.662+0000 7f845f7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:12:47.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.662+0000 7f8474e8d700 1 -- 172.21.15.90:0/2735371829 --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f84701a5980 con 0x7f847010d940 2024-04-14T15:12:47.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.662+0000 7f845f7fe700 1 -- 172.21.15.90:0/2735371829 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2782267968 0 0) 0x7f84701b79b0 con 0x7f8470106f80 2024-04-14T15:12:47.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.662+0000 7f845f7fe700 1 -- 172.21.15.90:0/2735371829 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f84580046c0 con 0x7f8470106f80 2024-04-14T15:12:47.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.662+0000 7f845f7fe700 1 -- 172.21.15.90:0/2735371829 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 660071226 0 0) 0x7f84701a5770 con 0x7f8470109da0 2024-04-14T15:12:47.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.662+0000 7f845f7fe700 1 -- 172.21.15.90:0/2735371829 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8458003d00 con 0x7f8470109da0 2024-04-14T15:12:47.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.663+0000 7f845f7fe700 1 -- 172.21.15.90:0/2735371829 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2592145234 0 0) 0x7f84701a5980 con 0x7f847010d940 2024-04-14T15:12:47.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.663+0000 7f845f7fe700 1 -- 172.21.15.90:0/2735371829 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f84701a5770 con 0x7f847010d940 2024-04-14T15:12:47.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.663+0000 7f845f7fe700 1 -- 172.21.15.90:0/2735371829 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3140963114 0 0) 0x7f8458003d00 con 0x7f8470109da0 2024-04-14T15:12:47.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.663+0000 7f845f7fe700 1 -- 172.21.15.90:0/2735371829 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f84701a5980 con 0x7f8470109da0 2024-04-14T15:12:47.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.663+0000 7f845f7fe700 1 -- 172.21.15.90:0/2735371829 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f8464001a70 con 0x7f8470109da0 2024-04-14T15:12:47.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.663+0000 7f845f7fe700 1 -- 172.21.15.90:0/2735371829 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2213911771 0 0) 0x7f84580046c0 con 0x7f8470106f80 2024-04-14T15:12:47.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.663+0000 7f845f7fe700 1 -- 172.21.15.90:0/2735371829 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8458003d00 con 0x7f8470106f80 2024-04-14T15:12:47.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.663+0000 7f845f7fe700 1 -- 172.21.15.90:0/2735371829 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f84600018b0 con 0x7f8470106f80 2024-04-14T15:12:47.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.663+0000 7f845f7fe700 1 -- 172.21.15.90:0/2735371829 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2441268763 0 0) 0x7f84701a5980 con 0x7f8470109da0 2024-04-14T15:12:47.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.663+0000 7f845f7fe700 1 -- 172.21.15.90:0/2735371829 >> v1:172.21.15.90:6790/0 conn(0x7f847010d940 legacy=0x7f84701a3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:47.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.663+0000 7f845f7fe700 1 -- 172.21.15.90:0/2735371829 >> v1:172.21.15.92:6789/0 conn(0x7f8470106f80 legacy=0x7f84700fe170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:47.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.663+0000 7f845f7fe700 1 -- 172.21.15.90:0/2735371829 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84701a5b90 con 0x7f8470109da0 2024-04-14T15:12:47.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.663+0000 7f8474e8d700 1 -- 172.21.15.90:0/2735371829 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f84701a5e80 con 0x7f8470109da0 2024-04-14T15:12:47.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.663+0000 7f8474e8d700 10 client.14892.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:12:47.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.663+0000 7f8474e8d700 1 -- 172.21.15.90:0/2735371829 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f84701a6340 con 0x7f8470109da0 2024-04-14T15:12:47.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.663+0000 7f8474e8d700 1 librados: init done 2024-04-14T15:12:47.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.664+0000 7f845f7fe700 1 -- 172.21.15.90:0/2735371829 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f8464001cd0 con 0x7f8470109da0 2024-04-14T15:12:47.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.664+0000 7f845f7fe700 1 -- 172.21.15.90:0/2735371829 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f8464005e10 con 0x7f8470109da0 2024-04-14T15:12:47.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.665+0000 7f8474e8d700 1 -- 172.21.15.90:0/2735371829 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f8450005100 con 0x7f8470109da0 2024-04-14T15:12:47.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.666+0000 7f845f7fe700 1 -- 172.21.15.90:0/2735371829 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f8464007060 con 0x7f8470109da0 2024-04-14T15:12:47.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.667+0000 7f845f7fe700 1 -- 172.21.15.90:0/2735371829 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(105..105 src has 51..105) ==== 6320+0+0 (unknown 2277868576 0 0) 0x7f846408a470 con 0x7f8470109da0 2024-04-14T15:12:47.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.667+0000 7f845f7fe700 10 client.14892.objecter ms_dispatch 0x7f8470001550 osd_map(105..105 src has 51..105) 2024-04-14T15:12:47.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.667+0000 7f845f7fe700 3 client.14892.objecter handle_osd_map got epochs [105,105] > 0 2024-04-14T15:12:47.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.667+0000 7f845f7fe700 3 client.14892.objecter handle_osd_map decoding full epoch 105 2024-04-14T15:12:47.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.667+0000 7f845f7fe700 20 client.14892.objecter dump_active .. 0 homeless 2024-04-14T15:12:47.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.667+0000 7f845f7fe700 1 -- 172.21.15.90:0/2735371829 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v105) ==== 54+0+16 (unknown 3432169292 0 2420000192) 0x7f846408c8a0 con 0x7f8470109da0 2024-04-14T15:12:47.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.667+0000 7f845f7fe700 10 client.14892.objecter ms_handle_connect 0x7f8458071540 2024-04-14T15:12:47.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.667+0000 7f845d7fa700 10 client.14892.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:12:47.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.667+0000 7f845d7fa700 15 client.14892.objecter _session_command_op_assign -1 1 2024-04-14T15:12:47.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.667+0000 7f845d7fa700 20 client.14892.objecter _get_session s=0x7f84540028e0 osd=0 3 2024-04-14T15:12:47.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.667+0000 7f845d7fa700 20 client.14892.objecter put_session s=0x7f84540028e0 osd=0 3 2024-04-14T15:12:47.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.667+0000 7f845d7fa700 20 client.14892.objecter _get_session s=0x7f84540028e0 osd=0 3 2024-04-14T15:12:47.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.667+0000 7f845d7fa700 15 client.14892.objecter _session_command_op_remove -1 1 2024-04-14T15:12:47.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.667+0000 7f845d7fa700 20 client.14892.objecter get_session s=0x7f84540028e0 osd=0 3 2024-04-14T15:12:47.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.667+0000 7f845d7fa700 15 client.14892.objecter _session_command_op_assign 0 1 2024-04-14T15:12:47.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.667+0000 7f845d7fa700 20 client.14892.objecter put_session s=0x7f84540028e0 osd=0 4 2024-04-14T15:12:47.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.668+0000 7f845d7fa700 10 client.14892.objecter _send_command 1 2024-04-14T15:12:47.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.668+0000 7f845d7fa700 1 -- 172.21.15.90:0/2735371829 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8454006d40 con 0x7f8454002fd0 2024-04-14T15:12:47.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.668+0000 7f845f7fe700 10 client.14892.objecter ms_handle_connect 0x7f8454002fd0 2024-04-14T15:12:47.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.668+0000 7f845f7fe700 1 -- 172.21.15.90:0/2735371829 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f8454006d40 con 0x7f8454002fd0 2024-04-14T15:12:47.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.668+0000 7f845f7fe700 10 client.14892.objecter ms_dispatch 0x7f8470001550 command_reply(tid 1: 0 ) 2024-04-14T15:12:47.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.668+0000 7f845f7fe700 10 client.14892.objecter _finish_command 1 = system:0 2024-04-14T15:12:47.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.668+0000 7f845f7fe700 20 client.14892.objecter put_session s=0x7f84540028e0 osd=0 4 2024-04-14T15:12:47.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.668+0000 7f845f7fe700 15 client.14892.objecter _session_command_op_remove 0 1 2024-04-14T15:12:47.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.694+0000 7f845d7fa700 10 client.14892.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:12:47.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.694+0000 7f845d7fa700 15 client.14892.objecter _session_command_op_assign -1 2 2024-04-14T15:12:47.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.694+0000 7f845d7fa700 20 client.14892.objecter _get_session s=0x7f84540028e0 osd=0 3 2024-04-14T15:12:47.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.694+0000 7f845d7fa700 20 client.14892.objecter put_session s=0x7f84540028e0 osd=0 3 2024-04-14T15:12:47.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.694+0000 7f845d7fa700 20 client.14892.objecter _get_session s=0x7f84540028e0 osd=0 3 2024-04-14T15:12:47.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.694+0000 7f845d7fa700 15 client.14892.objecter _session_command_op_remove -1 2 2024-04-14T15:12:47.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.694+0000 7f845d7fa700 20 client.14892.objecter get_session s=0x7f84540028e0 osd=0 3 2024-04-14T15:12:47.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.694+0000 7f845d7fa700 15 client.14892.objecter _session_command_op_assign 0 2 2024-04-14T15:12:47.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.694+0000 7f845d7fa700 20 client.14892.objecter put_session s=0x7f84540028e0 osd=0 4 2024-04-14T15:12:47.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.694+0000 7f845d7fa700 10 client.14892.objecter _send_command 2 2024-04-14T15:12:47.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.694+0000 7f845d7fa700 1 -- 172.21.15.90:0/2735371829 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f8454001240 con 0x7f8454002fd0 2024-04-14T15:12:47.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.697+0000 7f845f7fe700 1 -- 172.21.15.90:0/2735371829 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f8454001240 con 0x7f8454002fd0 2024-04-14T15:12:47.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.697+0000 7f845f7fe700 10 client.14892.objecter ms_dispatch 0x7f8470001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:12:47.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.697+0000 7f845f7fe700 10 client.14892.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:12:47.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.697+0000 7f845f7fe700 20 client.14892.objecter put_session s=0x7f84540028e0 osd=0 4 2024-04-14T15:12:47.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.697+0000 7f845f7fe700 15 client.14892.objecter _session_command_op_remove 0 2 2024-04-14T15:12:47.698 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:12:47.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.697+0000 7f845d7fa700 10 client.14892.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:12:47.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.697+0000 7f845d7fa700 15 client.14892.objecter _session_command_op_assign -1 3 2024-04-14T15:12:47.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.697+0000 7f845d7fa700 20 client.14892.objecter _get_session s=0x7f8454001f50 osd=1 3 2024-04-14T15:12:47.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.697+0000 7f845d7fa700 20 client.14892.objecter put_session s=0x7f8454001f50 osd=1 3 2024-04-14T15:12:47.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.697+0000 7f845d7fa700 20 client.14892.objecter _get_session s=0x7f8454001f50 osd=1 3 2024-04-14T15:12:47.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.697+0000 7f845d7fa700 15 client.14892.objecter _session_command_op_remove -1 3 2024-04-14T15:12:47.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.697+0000 7f845d7fa700 20 client.14892.objecter get_session s=0x7f8454001f50 osd=1 3 2024-04-14T15:12:47.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.697+0000 7f845d7fa700 15 client.14892.objecter _session_command_op_assign 1 3 2024-04-14T15:12:47.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.697+0000 7f845d7fa700 20 client.14892.objecter put_session s=0x7f8454001f50 osd=1 4 2024-04-14T15:12:47.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.697+0000 7f845d7fa700 10 client.14892.objecter _send_command 3 2024-04-14T15:12:47.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.697+0000 7f845d7fa700 1 -- 172.21.15.90:0/2735371829 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f8454008cc0 con 0x7f8454008100 2024-04-14T15:12:47.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.698+0000 7f845f7fe700 10 client.14892.objecter ms_handle_connect 0x7f8454008100 2024-04-14T15:12:47.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.699+0000 7f845f7fe700 1 -- 172.21.15.90:0/2735371829 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f8454008cc0 con 0x7f8454008100 2024-04-14T15:12:47.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.699+0000 7f845f7fe700 10 client.14892.objecter ms_dispatch 0x7f8470001550 command_reply(tid 3: 0 ) 2024-04-14T15:12:47.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.699+0000 7f845f7fe700 10 client.14892.objecter _finish_command 3 = system:0 2024-04-14T15:12:47.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.699+0000 7f845f7fe700 20 client.14892.objecter put_session s=0x7f8454001f50 osd=1 4 2024-04-14T15:12:47.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.699+0000 7f845f7fe700 15 client.14892.objecter _session_command_op_remove 1 3 2024-04-14T15:12:47.707 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:12:47.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.731+0000 7f845d7fa700 10 client.14892.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:12:47.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.731+0000 7f845d7fa700 15 client.14892.objecter _session_command_op_assign -1 4 2024-04-14T15:12:47.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.731+0000 7f845d7fa700 20 client.14892.objecter _get_session s=0x7f8454001f50 osd=1 3 2024-04-14T15:12:47.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.731+0000 7f845d7fa700 20 client.14892.objecter put_session s=0x7f8454001f50 osd=1 3 2024-04-14T15:12:47.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.731+0000 7f845d7fa700 20 client.14892.objecter _get_session s=0x7f8454001f50 osd=1 3 2024-04-14T15:12:47.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.731+0000 7f845d7fa700 15 client.14892.objecter _session_command_op_remove -1 4 2024-04-14T15:12:47.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.731+0000 7f845d7fa700 20 client.14892.objecter get_session s=0x7f8454001f50 osd=1 3 2024-04-14T15:12:47.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.731+0000 7f845d7fa700 15 client.14892.objecter _session_command_op_assign 1 4 2024-04-14T15:12:47.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.731+0000 7f845d7fa700 20 client.14892.objecter put_session s=0x7f8454001f50 osd=1 4 2024-04-14T15:12:47.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.731+0000 7f845d7fa700 10 client.14892.objecter _send_command 4 2024-04-14T15:12:47.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.731+0000 7f845d7fa700 1 -- 172.21.15.90:0/2735371829 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f84540096b0 con 0x7f8454008100 2024-04-14T15:12:47.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:47.733 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:47.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:12:47.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.733+0000 7f845f7fe700 1 -- 172.21.15.90:0/2735371829 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f84540096b0 con 0x7f8454008100 2024-04-14T15:12:47.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.733+0000 7f845f7fe700 10 client.14892.objecter ms_dispatch 0x7f8470001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:12:47.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.733+0000 7f845f7fe700 10 client.14892.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:12:47.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.733+0000 7f845f7fe700 20 client.14892.objecter put_session s=0x7f8454001f50 osd=1 4 2024-04-14T15:12:47.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.733+0000 7f845f7fe700 15 client.14892.objecter _session_command_op_remove 1 4 2024-04-14T15:12:47.735 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:12:47.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.734+0000 7f845d7fa700 10 client.14892.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:12:47.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.734+0000 7f845d7fa700 15 client.14892.objecter _session_command_op_assign -1 5 2024-04-14T15:12:47.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.734+0000 7f845d7fa700 20 client.14892.objecter _get_session s=0x7f8454001c90 osd=2 3 2024-04-14T15:12:47.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.734+0000 7f845d7fa700 20 client.14892.objecter put_session s=0x7f8454001c90 osd=2 3 2024-04-14T15:12:47.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.734+0000 7f845d7fa700 20 client.14892.objecter _get_session s=0x7f8454001c90 osd=2 3 2024-04-14T15:12:47.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.734+0000 7f845d7fa700 15 client.14892.objecter _session_command_op_remove -1 5 2024-04-14T15:12:47.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.734+0000 7f845d7fa700 20 client.14892.objecter get_session s=0x7f8454001c90 osd=2 3 2024-04-14T15:12:47.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.734+0000 7f845d7fa700 15 client.14892.objecter _session_command_op_assign 2 5 2024-04-14T15:12:47.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.734+0000 7f845d7fa700 20 client.14892.objecter put_session s=0x7f8454001c90 osd=2 4 2024-04-14T15:12:47.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.734+0000 7f845d7fa700 10 client.14892.objecter _send_command 5 2024-04-14T15:12:47.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.734+0000 7f845d7fa700 1 -- 172.21.15.90:0/2735371829 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f8454010ab0 con 0x7f845400ce70 2024-04-14T15:12:47.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.735+0000 7f845f7fe700 10 client.14892.objecter ms_handle_connect 0x7f845400ce70 2024-04-14T15:12:47.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.735+0000 7f845f7fe700 1 -- 172.21.15.90:0/2735371829 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f8454010ab0 con 0x7f845400ce70 2024-04-14T15:12:47.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.735+0000 7f845f7fe700 10 client.14892.objecter ms_dispatch 0x7f8470001550 command_reply(tid 5: 0 ) 2024-04-14T15:12:47.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.735+0000 7f845f7fe700 10 client.14892.objecter _finish_command 5 = system:0 2024-04-14T15:12:47.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.735+0000 7f845f7fe700 20 client.14892.objecter put_session s=0x7f8454001c90 osd=2 4 2024-04-14T15:12:47.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.735+0000 7f845f7fe700 15 client.14892.objecter _session_command_op_remove 2 5 2024-04-14T15:12:47.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.752+0000 7f845d7fa700 10 client.14892.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:12:47.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.752+0000 7f845d7fa700 15 client.14892.objecter _session_command_op_assign -1 6 2024-04-14T15:12:47.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.752+0000 7f845d7fa700 20 client.14892.objecter _get_session s=0x7f8454001c90 osd=2 3 2024-04-14T15:12:47.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.752+0000 7f845d7fa700 20 client.14892.objecter put_session s=0x7f8454001c90 osd=2 3 2024-04-14T15:12:47.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.752+0000 7f845d7fa700 20 client.14892.objecter _get_session s=0x7f8454001c90 osd=2 3 2024-04-14T15:12:47.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.752+0000 7f845d7fa700 15 client.14892.objecter _session_command_op_remove -1 6 2024-04-14T15:12:47.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.752+0000 7f845d7fa700 20 client.14892.objecter get_session s=0x7f8454001c90 osd=2 3 2024-04-14T15:12:47.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.752+0000 7f845d7fa700 15 client.14892.objecter _session_command_op_assign 2 6 2024-04-14T15:12:47.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.752+0000 7f845d7fa700 20 client.14892.objecter put_session s=0x7f8454001c90 osd=2 4 2024-04-14T15:12:47.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.752+0000 7f845d7fa700 10 client.14892.objecter _send_command 6 2024-04-14T15:12:47.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.752+0000 7f845d7fa700 1 -- 172.21.15.90:0/2735371829 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f8454009220 con 0x7f845400ce70 2024-04-14T15:12:47.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.754+0000 7f845f7fe700 1 -- 172.21.15.90:0/2735371829 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f8454009220 con 0x7f845400ce70 2024-04-14T15:12:47.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.754+0000 7f845f7fe700 10 client.14892.objecter ms_dispatch 0x7f8470001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:12:47.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.754+0000 7f845f7fe700 10 client.14892.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:12:47.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.754+0000 7f845f7fe700 20 client.14892.objecter put_session s=0x7f8454001c90 osd=2 4 2024-04-14T15:12:47.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.754+0000 7f845f7fe700 15 client.14892.objecter _session_command_op_remove 2 6 2024-04-14T15:12:47.755 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:12:47.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.755+0000 7f845d7fa700 10 client.14892.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:12:47.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.755+0000 7f845d7fa700 15 client.14892.objecter _session_command_op_assign -1 7 2024-04-14T15:12:47.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.755+0000 7f845d7fa700 20 client.14892.objecter _get_session s=0x7f84540094c0 osd=3 3 2024-04-14T15:12:47.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.755+0000 7f845d7fa700 20 client.14892.objecter put_session s=0x7f84540094c0 osd=3 3 2024-04-14T15:12:47.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.755+0000 7f845d7fa700 20 client.14892.objecter _get_session s=0x7f84540094c0 osd=3 3 2024-04-14T15:12:47.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.755+0000 7f845d7fa700 15 client.14892.objecter _session_command_op_remove -1 7 2024-04-14T15:12:47.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.755+0000 7f845d7fa700 20 client.14892.objecter get_session s=0x7f84540094c0 osd=3 3 2024-04-14T15:12:47.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.755+0000 7f845d7fa700 15 client.14892.objecter _session_command_op_assign 3 7 2024-04-14T15:12:47.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.755+0000 7f845d7fa700 20 client.14892.objecter put_session s=0x7f84540094c0 osd=3 4 2024-04-14T15:12:47.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.755+0000 7f845d7fa700 10 client.14892.objecter _send_command 7 2024-04-14T15:12:47.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.755+0000 7f845d7fa700 1 -- 172.21.15.90:0/2735371829 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f84540117d0 con 0x7f8454010c60 2024-04-14T15:12:47.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.755+0000 7f845f7fe700 10 client.14892.objecter ms_handle_connect 0x7f8454010c60 2024-04-14T15:12:47.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.756+0000 7f845f7fe700 1 -- 172.21.15.90:0/2735371829 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f84540117d0 con 0x7f8454010c60 2024-04-14T15:12:47.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.756+0000 7f845f7fe700 10 client.14892.objecter ms_dispatch 0x7f8470001550 command_reply(tid 7: 0 ) 2024-04-14T15:12:47.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.756+0000 7f845f7fe700 10 client.14892.objecter _finish_command 7 = system:0 2024-04-14T15:12:47.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.756+0000 7f845f7fe700 20 client.14892.objecter put_session s=0x7f84540094c0 osd=3 4 2024-04-14T15:12:47.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.756+0000 7f845f7fe700 15 client.14892.objecter _session_command_op_remove 3 7 2024-04-14T15:12:47.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.773+0000 7f845d7fa700 10 client.14892.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:12:47.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.773+0000 7f845d7fa700 15 client.14892.objecter _session_command_op_assign -1 8 2024-04-14T15:12:47.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.773+0000 7f845d7fa700 20 client.14892.objecter _get_session s=0x7f84540094c0 osd=3 3 2024-04-14T15:12:47.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.773+0000 7f845d7fa700 20 client.14892.objecter put_session s=0x7f84540094c0 osd=3 3 2024-04-14T15:12:47.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.773+0000 7f845d7fa700 20 client.14892.objecter _get_session s=0x7f84540094c0 osd=3 3 2024-04-14T15:12:47.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.773+0000 7f845d7fa700 15 client.14892.objecter _session_command_op_remove -1 8 2024-04-14T15:12:47.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.773+0000 7f845d7fa700 20 client.14892.objecter get_session s=0x7f84540094c0 osd=3 3 2024-04-14T15:12:47.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.773+0000 7f845d7fa700 15 client.14892.objecter _session_command_op_assign 3 8 2024-04-14T15:12:47.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.773+0000 7f845d7fa700 20 client.14892.objecter put_session s=0x7f84540094c0 osd=3 4 2024-04-14T15:12:47.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.773+0000 7f845d7fa700 10 client.14892.objecter _send_command 8 2024-04-14T15:12:47.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.773+0000 7f845d7fa700 1 -- 172.21.15.90:0/2735371829 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f8454012150 con 0x7f8454010c60 2024-04-14T15:12:47.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.775+0000 7f845f7fe700 1 -- 172.21.15.90:0/2735371829 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f8454012150 con 0x7f8454010c60 2024-04-14T15:12:47.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.776+0000 7f845f7fe700 10 client.14892.objecter ms_dispatch 0x7f8470001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:12:47.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.776+0000 7f845f7fe700 10 client.14892.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:12:47.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.776+0000 7f845f7fe700 20 client.14892.objecter put_session s=0x7f84540094c0 osd=3 4 2024-04-14T15:12:47.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.776+0000 7f845f7fe700 15 client.14892.objecter _session_command_op_remove 3 8 2024-04-14T15:12:47.776 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:12:47.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.776+0000 7f845d7fa700 10 client.14892.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:12:47.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.776+0000 7f845d7fa700 15 client.14892.objecter _session_command_op_assign -1 9 2024-04-14T15:12:47.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.776+0000 7f845d7fa700 20 client.14892.objecter _get_session s=0x7f8454011bd0 osd=4 3 2024-04-14T15:12:47.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.776+0000 7f845d7fa700 20 client.14892.objecter put_session s=0x7f8454011bd0 osd=4 3 2024-04-14T15:12:47.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.776+0000 7f845d7fa700 20 client.14892.objecter _get_session s=0x7f8454011bd0 osd=4 3 2024-04-14T15:12:47.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.776+0000 7f845d7fa700 15 client.14892.objecter _session_command_op_remove -1 9 2024-04-14T15:12:47.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.776+0000 7f845d7fa700 20 client.14892.objecter get_session s=0x7f8454011bd0 osd=4 3 2024-04-14T15:12:47.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.776+0000 7f845d7fa700 15 client.14892.objecter _session_command_op_assign 4 9 2024-04-14T15:12:47.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.776+0000 7f845d7fa700 20 client.14892.objecter put_session s=0x7f8454011bd0 osd=4 4 2024-04-14T15:12:47.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.776+0000 7f845d7fa700 10 client.14892.objecter _send_command 9 2024-04-14T15:12:47.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.776+0000 7f845d7fa700 1 -- 172.21.15.90:0/2735371829 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f845401a3f0 con 0x7f8454011d40 2024-04-14T15:12:47.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.777+0000 7f845f7fe700 10 client.14892.objecter ms_handle_connect 0x7f8454011d40 2024-04-14T15:12:47.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.778+0000 7f845f7fe700 1 -- 172.21.15.90:0/2735371829 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f845401a3f0 con 0x7f8454011d40 2024-04-14T15:12:47.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.778+0000 7f845f7fe700 10 client.14892.objecter ms_dispatch 0x7f8470001550 command_reply(tid 9: 0 ) 2024-04-14T15:12:47.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.778+0000 7f845f7fe700 10 client.14892.objecter _finish_command 9 = system:0 2024-04-14T15:12:47.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.778+0000 7f845f7fe700 20 client.14892.objecter put_session s=0x7f8454011bd0 osd=4 4 2024-04-14T15:12:47.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.778+0000 7f845f7fe700 15 client.14892.objecter _session_command_op_remove 4 9 2024-04-14T15:12:47.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.794+0000 7f845d7fa700 10 client.14892.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:12:47.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.794+0000 7f845d7fa700 15 client.14892.objecter _session_command_op_assign -1 10 2024-04-14T15:12:47.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.794+0000 7f845d7fa700 20 client.14892.objecter _get_session s=0x7f8454011bd0 osd=4 3 2024-04-14T15:12:47.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.794+0000 7f845d7fa700 20 client.14892.objecter put_session s=0x7f8454011bd0 osd=4 3 2024-04-14T15:12:47.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.794+0000 7f845d7fa700 20 client.14892.objecter _get_session s=0x7f8454011bd0 osd=4 3 2024-04-14T15:12:47.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.794+0000 7f845d7fa700 15 client.14892.objecter _session_command_op_remove -1 10 2024-04-14T15:12:47.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.794+0000 7f845d7fa700 20 client.14892.objecter get_session s=0x7f8454011bd0 osd=4 3 2024-04-14T15:12:47.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.794+0000 7f845d7fa700 15 client.14892.objecter _session_command_op_assign 4 10 2024-04-14T15:12:47.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.794+0000 7f845d7fa700 20 client.14892.objecter put_session s=0x7f8454011bd0 osd=4 4 2024-04-14T15:12:47.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.794+0000 7f845d7fa700 10 client.14892.objecter _send_command 10 2024-04-14T15:12:47.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.794+0000 7f845d7fa700 1 -- 172.21.15.90:0/2735371829 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f8454012300 con 0x7f8454011d40 2024-04-14T15:12:47.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.796+0000 7f845f7fe700 1 -- 172.21.15.90:0/2735371829 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f8454012300 con 0x7f8454011d40 2024-04-14T15:12:47.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.796+0000 7f845f7fe700 10 client.14892.objecter ms_dispatch 0x7f8470001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:12:47.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.796+0000 7f845f7fe700 10 client.14892.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:12:47.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.796+0000 7f845f7fe700 20 client.14892.objecter put_session s=0x7f8454011bd0 osd=4 4 2024-04-14T15:12:47.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.796+0000 7f845f7fe700 15 client.14892.objecter _session_command_op_remove 4 10 2024-04-14T15:12:47.799 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:12:47.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.797+0000 7f845d7fa700 10 client.14892.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:12:47.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.797+0000 7f845d7fa700 15 client.14892.objecter _session_command_op_assign -1 11 2024-04-14T15:12:47.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.797+0000 7f845d7fa700 20 client.14892.objecter _get_session s=0x7f8454012300 osd=5 3 2024-04-14T15:12:47.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.797+0000 7f845d7fa700 20 client.14892.objecter put_session s=0x7f8454012300 osd=5 3 2024-04-14T15:12:47.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.797+0000 7f845d7fa700 20 client.14892.objecter _get_session s=0x7f8454012300 osd=5 3 2024-04-14T15:12:47.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.797+0000 7f845d7fa700 15 client.14892.objecter _session_command_op_remove -1 11 2024-04-14T15:12:47.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.797+0000 7f845d7fa700 20 client.14892.objecter get_session s=0x7f8454012300 osd=5 3 2024-04-14T15:12:47.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.797+0000 7f845d7fa700 15 client.14892.objecter _session_command_op_assign 5 11 2024-04-14T15:12:47.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.797+0000 7f845d7fa700 20 client.14892.objecter put_session s=0x7f8454012300 osd=5 4 2024-04-14T15:12:47.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.797+0000 7f845d7fa700 10 client.14892.objecter _send_command 11 2024-04-14T15:12:47.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.797+0000 7f845d7fa700 1 -- 172.21.15.90:0/2735371829 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f8454012e60 con 0x7f8454012a20 2024-04-14T15:12:47.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.797+0000 7f845f7fe700 10 client.14892.objecter ms_handle_connect 0x7f8454012a20 2024-04-14T15:12:47.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.798+0000 7f845f7fe700 1 -- 172.21.15.90:0/2735371829 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f8454012e60 con 0x7f8454012a20 2024-04-14T15:12:47.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.798+0000 7f845f7fe700 10 client.14892.objecter ms_dispatch 0x7f8470001550 command_reply(tid 11: 0 ) 2024-04-14T15:12:47.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.798+0000 7f845f7fe700 10 client.14892.objecter _finish_command 11 = system:0 2024-04-14T15:12:47.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.798+0000 7f845f7fe700 20 client.14892.objecter put_session s=0x7f8454012300 osd=5 4 2024-04-14T15:12:47.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.798+0000 7f845f7fe700 15 client.14892.objecter _session_command_op_remove 5 11 2024-04-14T15:12:47.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.831+0000 7f845d7fa700 10 client.14892.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:12:47.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.831+0000 7f845d7fa700 15 client.14892.objecter _session_command_op_assign -1 12 2024-04-14T15:12:47.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.831+0000 7f845d7fa700 20 client.14892.objecter _get_session s=0x7f8454012300 osd=5 3 2024-04-14T15:12:47.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.831+0000 7f845d7fa700 20 client.14892.objecter put_session s=0x7f8454012300 osd=5 3 2024-04-14T15:12:47.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.831+0000 7f845d7fa700 20 client.14892.objecter _get_session s=0x7f8454012300 osd=5 3 2024-04-14T15:12:47.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.831+0000 7f845d7fa700 15 client.14892.objecter _session_command_op_remove -1 12 2024-04-14T15:12:47.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.831+0000 7f845d7fa700 20 client.14892.objecter get_session s=0x7f8454012300 osd=5 3 2024-04-14T15:12:47.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.831+0000 7f845d7fa700 15 client.14892.objecter _session_command_op_assign 5 12 2024-04-14T15:12:47.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.831+0000 7f845d7fa700 20 client.14892.objecter put_session s=0x7f8454012300 osd=5 4 2024-04-14T15:12:47.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.831+0000 7f845d7fa700 10 client.14892.objecter _send_command 12 2024-04-14T15:12:47.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.831+0000 7f845d7fa700 1 -- 172.21.15.90:0/2735371829 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f8454012e60 con 0x7f8454012a20 2024-04-14T15:12:47.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.833+0000 7f845f7fe700 1 -- 172.21.15.90:0/2735371829 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f8454012e60 con 0x7f8454012a20 2024-04-14T15:12:47.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.833+0000 7f845f7fe700 10 client.14892.objecter ms_dispatch 0x7f8470001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:12:47.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.833+0000 7f845f7fe700 10 client.14892.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:12:47.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.833+0000 7f845f7fe700 20 client.14892.objecter put_session s=0x7f8454012300 osd=5 4 2024-04-14T15:12:47.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.833+0000 7f845f7fe700 15 client.14892.objecter _session_command_op_remove 5 12 2024-04-14T15:12:47.834 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:12:47.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.833+0000 7f845d7fa700 10 client.14892.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:12:47.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.833+0000 7f845d7fa700 15 client.14892.objecter _session_command_op_assign -1 13 2024-04-14T15:12:47.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.833+0000 7f845d7fa700 20 client.14892.objecter _get_session s=0x7f8454012e60 osd=6 3 2024-04-14T15:12:47.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.833+0000 7f845d7fa700 20 client.14892.objecter put_session s=0x7f8454012e60 osd=6 3 2024-04-14T15:12:47.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.833+0000 7f845d7fa700 20 client.14892.objecter _get_session s=0x7f8454012e60 osd=6 3 2024-04-14T15:12:47.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.833+0000 7f845d7fa700 15 client.14892.objecter _session_command_op_remove -1 13 2024-04-14T15:12:47.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.833+0000 7f845d7fa700 20 client.14892.objecter get_session s=0x7f8454012e60 osd=6 3 2024-04-14T15:12:47.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.833+0000 7f845d7fa700 15 client.14892.objecter _session_command_op_assign 6 13 2024-04-14T15:12:47.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.833+0000 7f845d7fa700 20 client.14892.objecter put_session s=0x7f8454012e60 osd=6 4 2024-04-14T15:12:47.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.833+0000 7f845d7fa700 10 client.14892.objecter _send_command 13 2024-04-14T15:12:47.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.833+0000 7f845d7fa700 1 -- 172.21.15.90:0/2735371829 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f8454022d70 con 0x7f84540130e0 2024-04-14T15:12:47.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.834+0000 7f845f7fe700 10 client.14892.objecter ms_handle_connect 0x7f84540130e0 2024-04-14T15:12:47.836 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:12:47.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.835+0000 7f845f7fe700 1 -- 172.21.15.90:0/2735371829 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f8454022d70 con 0x7f84540130e0 2024-04-14T15:12:47.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.835+0000 7f845f7fe700 10 client.14892.objecter ms_dispatch 0x7f8470001550 command_reply(tid 13: 0 ) 2024-04-14T15:12:47.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.835+0000 7f845f7fe700 10 client.14892.objecter _finish_command 13 = system:0 2024-04-14T15:12:47.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.835+0000 7f845f7fe700 20 client.14892.objecter put_session s=0x7f8454012e60 osd=6 4 2024-04-14T15:12:47.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.835+0000 7f845f7fe700 15 client.14892.objecter _session_command_op_remove 6 13 2024-04-14T15:12:47.864 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:47.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:47.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:12:47.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.868+0000 7f845d7fa700 10 client.14892.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:12:47.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.868+0000 7f845d7fa700 15 client.14892.objecter _session_command_op_assign -1 14 2024-04-14T15:12:47.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.868+0000 7f845d7fa700 20 client.14892.objecter _get_session s=0x7f8454012e60 osd=6 3 2024-04-14T15:12:47.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.868+0000 7f845d7fa700 20 client.14892.objecter put_session s=0x7f8454012e60 osd=6 3 2024-04-14T15:12:47.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.868+0000 7f845d7fa700 20 client.14892.objecter _get_session s=0x7f8454012e60 osd=6 3 2024-04-14T15:12:47.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.868+0000 7f845d7fa700 15 client.14892.objecter _session_command_op_remove -1 14 2024-04-14T15:12:47.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.868+0000 7f845d7fa700 20 client.14892.objecter get_session s=0x7f8454012e60 osd=6 3 2024-04-14T15:12:47.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.868+0000 7f845d7fa700 15 client.14892.objecter _session_command_op_assign 6 14 2024-04-14T15:12:47.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.868+0000 7f845d7fa700 20 client.14892.objecter put_session s=0x7f8454012e60 osd=6 4 2024-04-14T15:12:47.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.868+0000 7f845d7fa700 10 client.14892.objecter _send_command 14 2024-04-14T15:12:47.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.868+0000 7f845d7fa700 1 -- 172.21.15.90:0/2735371829 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f8454007420 con 0x7f84540130e0 2024-04-14T15:12:47.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.870+0000 7f845f7fe700 1 -- 172.21.15.90:0/2735371829 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f8454007420 con 0x7f84540130e0 2024-04-14T15:12:47.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.870+0000 7f845f7fe700 10 client.14892.objecter ms_dispatch 0x7f8470001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:12:47.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.870+0000 7f845f7fe700 10 client.14892.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:12:47.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.870+0000 7f845f7fe700 20 client.14892.objecter put_session s=0x7f8454012e60 osd=6 4 2024-04-14T15:12:47.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.870+0000 7f845f7fe700 15 client.14892.objecter _session_command_op_remove 6 14 2024-04-14T15:12:47.870 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:12:47.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.870+0000 7f845d7fa700 10 client.14892.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:12:47.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.870+0000 7f845d7fa700 15 client.14892.objecter _session_command_op_assign -1 15 2024-04-14T15:12:47.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.870+0000 7f845d7fa700 20 client.14892.objecter _get_session s=0x7f8454011a40 osd=7 3 2024-04-14T15:12:47.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.870+0000 7f845d7fa700 20 client.14892.objecter put_session s=0x7f8454011a40 osd=7 3 2024-04-14T15:12:47.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.870+0000 7f845d7fa700 20 client.14892.objecter _get_session s=0x7f8454011a40 osd=7 3 2024-04-14T15:12:47.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.870+0000 7f845d7fa700 15 client.14892.objecter _session_command_op_remove -1 15 2024-04-14T15:12:47.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.870+0000 7f845d7fa700 20 client.14892.objecter get_session s=0x7f8454011a40 osd=7 3 2024-04-14T15:12:47.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.870+0000 7f845d7fa700 15 client.14892.objecter _session_command_op_assign 7 15 2024-04-14T15:12:47.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.870+0000 7f845d7fa700 20 client.14892.objecter put_session s=0x7f8454011a40 osd=7 4 2024-04-14T15:12:47.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.870+0000 7f845d7fa700 10 client.14892.objecter _send_command 15 2024-04-14T15:12:47.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.870+0000 7f845d7fa700 1 -- 172.21.15.90:0/2735371829 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f8454024530 con 0x7f84540071d0 2024-04-14T15:12:47.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.871+0000 7f845f7fe700 10 client.14892.objecter ms_handle_connect 0x7f84540071d0 2024-04-14T15:12:47.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.871+0000 7f845f7fe700 1 -- 172.21.15.90:0/2735371829 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f8454008cc0 con 0x7f84540071d0 2024-04-14T15:12:47.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.871+0000 7f845f7fe700 10 client.14892.objecter ms_dispatch 0x7f8470001550 command_reply(tid 15: 0 ) 2024-04-14T15:12:47.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.871+0000 7f845f7fe700 10 client.14892.objecter _finish_command 15 = system:0 2024-04-14T15:12:47.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.871+0000 7f845f7fe700 20 client.14892.objecter put_session s=0x7f8454011a40 osd=7 4 2024-04-14T15:12:47.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.871+0000 7f845f7fe700 15 client.14892.objecter _session_command_op_remove 7 15 2024-04-14T15:12:47.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.904+0000 7f845d7fa700 10 client.14892.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:12:47.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.904+0000 7f845d7fa700 15 client.14892.objecter _session_command_op_assign -1 16 2024-04-14T15:12:47.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.904+0000 7f845d7fa700 20 client.14892.objecter _get_session s=0x7f8454011a40 osd=7 3 2024-04-14T15:12:47.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.904+0000 7f845d7fa700 20 client.14892.objecter put_session s=0x7f8454011a40 osd=7 3 2024-04-14T15:12:47.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.904+0000 7f845d7fa700 20 client.14892.objecter _get_session s=0x7f8454011a40 osd=7 3 2024-04-14T15:12:47.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.904+0000 7f845d7fa700 15 client.14892.objecter _session_command_op_remove -1 16 2024-04-14T15:12:47.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.904+0000 7f845d7fa700 20 client.14892.objecter get_session s=0x7f8454011a40 osd=7 3 2024-04-14T15:12:47.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.904+0000 7f845d7fa700 15 client.14892.objecter _session_command_op_assign 7 16 2024-04-14T15:12:47.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.904+0000 7f845d7fa700 20 client.14892.objecter put_session s=0x7f8454011a40 osd=7 4 2024-04-14T15:12:47.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.904+0000 7f845d7fa700 10 client.14892.objecter _send_command 16 2024-04-14T15:12:47.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.904+0000 7f845d7fa700 1 -- 172.21.15.90:0/2735371829 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f8454008cc0 con 0x7f84540071d0 2024-04-14T15:12:47.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.906+0000 7f845f7fe700 1 -- 172.21.15.90:0/2735371829 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f8454008cc0 con 0x7f84540071d0 2024-04-14T15:12:47.906 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:12:47.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.906+0000 7f845f7fe700 10 client.14892.objecter ms_dispatch 0x7f8470001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:12:47.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.906+0000 7f845f7fe700 10 client.14892.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:12:47.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.906+0000 7f845f7fe700 20 client.14892.objecter put_session s=0x7f8454011a40 osd=7 4 2024-04-14T15:12:47.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.906+0000 7f845f7fe700 15 client.14892.objecter _session_command_op_remove 7 16 2024-04-14T15:12:47.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.907+0000 7f845d7fa700 10 librados: watch_flush enter 2024-04-14T15:12:47.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.907+0000 7f845d7fa700 10 librados: watch_flush exit 2024-04-14T15:12:47.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.907+0000 7f845d7fa700 10 client.14892.objecter close_session for osd.0 2024-04-14T15:12:47.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.907+0000 7f845d7fa700 1 -- 172.21.15.90:0/2735371829 >> v1:172.21.15.90:6801/3761806885 conn(0x7f8454002fd0 legacy=0x7f8454005490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:47.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.908+0000 7f845d7fa700 20 client.14892.objecter put_session s=0x7f84540028e0 osd=0 1 2024-04-14T15:12:47.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.908+0000 7f845d7fa700 10 client.14892.objecter close_session for osd.1 2024-04-14T15:12:47.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.908+0000 7f845d7fa700 1 -- 172.21.15.90:0/2735371829 >> v1:172.21.15.90:6805/4170361699 conn(0x7f8454008100 legacy=0x7f84540085b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:47.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.908+0000 7f845d7fa700 20 client.14892.objecter put_session s=0x7f8454001f50 osd=1 1 2024-04-14T15:12:47.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.908+0000 7f845d7fa700 10 client.14892.objecter close_session for osd.2 2024-04-14T15:12:47.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.908+0000 7f845d7fa700 1 -- 172.21.15.90:0/2735371829 >> v1:172.21.15.90:6809/1686184251 conn(0x7f845400ce70 legacy=0x7f845400f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:47.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.908+0000 7f846ed9d700 1 -- 172.21.15.90:0/2735371829 reap_dead start 2024-04-14T15:12:47.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.908+0000 7f845d7fa700 20 client.14892.objecter put_session s=0x7f8454001c90 osd=2 1 2024-04-14T15:12:47.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.909+0000 7f845d7fa700 10 client.14892.objecter close_session for osd.3 2024-04-14T15:12:47.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.909+0000 7f845d7fa700 1 -- 172.21.15.90:0/2735371829 >> v1:172.21.15.90:6813/2270166774 conn(0x7f8454010c60 legacy=0x7f84540110c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:47.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.909+0000 7f845d7fa700 20 client.14892.objecter put_session s=0x7f84540094c0 osd=3 1 2024-04-14T15:12:47.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.909+0000 7f845d7fa700 10 client.14892.objecter close_session for osd.4 2024-04-14T15:12:47.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.909+0000 7f845d7fa700 1 -- 172.21.15.90:0/2735371829 >> v1:172.21.15.92:6800/3887549733 conn(0x7f8454011d40 legacy=0x7f8454018b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:47.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.909+0000 7f845d7fa700 20 client.14892.objecter put_session s=0x7f8454011bd0 osd=4 1 2024-04-14T15:12:47.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.909+0000 7f845d7fa700 10 client.14892.objecter close_session for osd.5 2024-04-14T15:12:47.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.910+0000 7f845d7fa700 1 -- 172.21.15.90:0/2735371829 >> v1:172.21.15.92:6804/3244561440 conn(0x7f8454012a20 legacy=0x7f845401a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:47.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.910+0000 7f845d7fa700 20 client.14892.objecter put_session s=0x7f8454012300 osd=5 1 2024-04-14T15:12:47.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.910+0000 7f845d7fa700 10 client.14892.objecter close_session for osd.6 2024-04-14T15:12:47.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.910+0000 7f845d7fa700 1 -- 172.21.15.90:0/2735371829 >> v1:172.21.15.92:6808/636125963 conn(0x7f84540130e0 legacy=0x7f84540214c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:47.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.911+0000 7f845d7fa700 20 client.14892.objecter put_session s=0x7f8454012e60 osd=6 1 2024-04-14T15:12:47.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.911+0000 7f845d7fa700 10 client.14892.objecter close_session for osd.7 2024-04-14T15:12:47.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.911+0000 7f845d7fa700 1 -- 172.21.15.90:0/2735371829 >> v1:172.21.15.92:6812/1773706716 conn(0x7f84540071d0 legacy=0x7f8454022d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:47.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.911+0000 7f846ed9d700 1 -- 172.21.15.90:0/2735371829 reap_dead start 2024-04-14T15:12:47.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.911+0000 7f845d7fa700 20 client.14892.objecter put_session s=0x7f8454011a40 osd=7 1 2024-04-14T15:12:47.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.912+0000 7f845d7fa700 20 client.14892.objecter shutdown clearing up homeless session... 2024-04-14T15:12:47.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.912+0000 7f845d7fa700 10 client.14892.objecter successfully canceled tick 2024-04-14T15:12:47.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.912+0000 7f845d7fa700 1 -- 172.21.15.90:0/2735371829 >> v1:172.21.15.90:6800/420935427 conn(0x7f8458071540 legacy=0x7f8458073a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:47.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.912+0000 7f845d7fa700 1 -- 172.21.15.90:0/2735371829 >> v1:172.21.15.90:6789/0 conn(0x7f8470109da0 legacy=0x7f84700fe880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:47.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.913+0000 7f845d7fa700 1 -- 172.21.15.90:0/2735371829 shutdown_connections 2024-04-14T15:12:47.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.913+0000 7f845d7fa700 1 -- 172.21.15.90:0/2735371829 >> 172.21.15.90:0/2735371829 conn(0x7f8470076aa0 msgr2=0x7f8470111f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:12:47.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.914+0000 7f845d7fa700 1 -- 172.21.15.90:0/2735371829 shutdown_connections 2024-04-14T15:12:47.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.914+0000 7f845d7fa700 1 -- 172.21.15.90:0/2735371829 wait complete. 2024-04-14T15:12:47.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:47.914+0000 7f845d7fa700 1 librados: shutdown 2024-04-14T15:12:47.917 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:12:47.917 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:12:47.917 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:12:47.917 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:12:47.917 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:12:47.917 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:12:47.917 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:12:47.917 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:12:47.967 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:12:47.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:47.991 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:47.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:12:48.092 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:12:48.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:48.120 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:48.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:12:48.152 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-14T15:12:48.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:47 smithi090 ceph-mon[41670]: pgmap v101: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:12:48.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:47 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2735371829' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:12:48.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:47 smithi090 ceph-mon[30399]: pgmap v101: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:12:48.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:47 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2735371829' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:12:48.221 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:12:48.254 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:48.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:48.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:12:48.356 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:12:48.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:47 smithi092 ceph-mon[33966]: pgmap v101: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:12:48.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:47 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2735371829' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:12:48.379 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:48.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:48.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:12:48.481 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:12:48.508 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:48.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:48.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:12:48.610 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:12:48.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:48.640 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:48.640 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:12:48.689 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-14T15:12:48.741 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:12:48.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:48.765 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:48.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:12:48.867 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:12:48.892 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:48.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:48.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:12:48.995 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:12:49.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:49.022 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:49.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:12:49.123 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:12:49.153 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:49.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:49.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:12:49.221 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:12:49.227 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:12:49.255 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:12:49.284 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:49.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:49.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:12:49.386 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:12:49.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:49.410 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:49.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:12:49.511 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:12:49.538 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:49.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:49.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:12:49.640 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:12:49.668 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:49.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:49.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:12:49.769 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:12:49.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:49.795 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:49.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:12:49.896 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:12:49.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:49.921 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:49.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:12:50.022 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:12:50.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:50.047 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:50.047 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:12:50.148 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:12:50.176 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:50.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:50.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:12:50.278 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:12:50.307 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:50.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:50.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:12:50.336 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-14T15:12:50.336 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-14T15:12:50.336 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-04-14T15:12:50.337 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd dump -f json-pretty 2024-04-14T15:12:50.360 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:50 smithi090 ceph-mon[41670]: pgmap v102: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-14T15:12:50.360 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:50 smithi090 ceph-mon[30399]: pgmap v102: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-14T15:12:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:50 smithi092 ceph-mon[33966]: pgmap v102: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-14T15:12:50.409 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:12:50.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:50.438 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:50.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:12:50.539 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:12:50.565 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:50.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:50.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:12:50.667 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:12:50.691 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:50.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:50.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:12:50.793 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:12:50.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:50.821 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:50.821 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:12:50.922 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:12:50.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:50.953 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:50.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:12:51.053 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:12:51.076 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:51.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:51.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:12:51.091 INFO:tasks.workunit.client.0.smithi090.stderr:Updating files: 94% (11832/12473) Updating files: 95% (11850/12473) Updating files: 96% (11975/12473) Updating files: 97% (12099/12473) Updating files: 98% (12224/12473) Updating files: 99% (12349/12473) Updating files: 100% (12473/12473) Updating files: 100% (12473/12473), done. 2024-04-14T15:12:51.173 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-14T15:12:51.178 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:12:51.239 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:51.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:51.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:12:51.260 INFO:tasks.workunit.client.0.smithi090.stderr:Note: switching to '05c9d46e7de5db2cb434c177a6bbfc56f472d09b'. 2024-04-14T15:12:51.260 INFO:tasks.workunit.client.0.smithi090.stderr: 2024-04-14T15:12:51.260 INFO:tasks.workunit.client.0.smithi090.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-14T15:12:51.260 INFO:tasks.workunit.client.0.smithi090.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-14T15:12:51.260 INFO:tasks.workunit.client.0.smithi090.stderr:state without impacting any branches by switching back to a branch. 2024-04-14T15:12:51.260 INFO:tasks.workunit.client.0.smithi090.stderr: 2024-04-14T15:12:51.260 INFO:tasks.workunit.client.0.smithi090.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-14T15:12:51.260 INFO:tasks.workunit.client.0.smithi090.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-14T15:12:51.260 INFO:tasks.workunit.client.0.smithi090.stderr: 2024-04-14T15:12:51.260 INFO:tasks.workunit.client.0.smithi090.stderr: git switch -c 2024-04-14T15:12:51.261 INFO:tasks.workunit.client.0.smithi090.stderr: 2024-04-14T15:12:51.261 INFO:tasks.workunit.client.0.smithi090.stderr:Or undo this operation with: 2024-04-14T15:12:51.261 INFO:tasks.workunit.client.0.smithi090.stderr: 2024-04-14T15:12:51.261 INFO:tasks.workunit.client.0.smithi090.stderr: git switch - 2024-04-14T15:12:51.261 INFO:tasks.workunit.client.0.smithi090.stderr: 2024-04-14T15:12:51.261 INFO:tasks.workunit.client.0.smithi090.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-14T15:12:51.261 INFO:tasks.workunit.client.0.smithi090.stderr: 2024-04-14T15:12:51.261 INFO:tasks.workunit.client.0.smithi090.stderr:HEAD is now at 05c9d46e7de Merge pull request #56637 from adk3798/node-proxy-service-expected-daemon-count 2024-04-14T15:12:51.262 DEBUG:teuthology.orchestra.run.smithi090:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-14T15:12:51.342 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:12:51.351 INFO:tasks.workunit.client.0.smithi090.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-14T15:12:51.353 INFO:tasks.workunit.client.0.smithi090.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-14T15:12:51.353 INFO:tasks.workunit.client.0.smithi090.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-14T15:12:51.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:51.370 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:51.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:12:51.471 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:12:51.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:51.496 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:51.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:12:51.597 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:12:51.620 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:51.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:51.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:12:51.722 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:12:51.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:51.752 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:51.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:12:51.853 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:12:51.881 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:51.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:51.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:12:51.983 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:12:52.010 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:52.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:52.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:12:52.112 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:12:52.135 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:52.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:52.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:12:52.238 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:12:52.262 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:52.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:52.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:12:52.347 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:52 smithi090 ceph-mon[41670]: pgmap v103: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:12:52.347 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:52 smithi090 ceph-mon[30399]: pgmap v103: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:12:52.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:52.346+0000 7f1cafa31700 1 -- 172.21.15.90:0/2657269430 >> v1:172.21.15.90:6789/0 conn(0x7f1ca810f300 legacy=0x7f1ca81117c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:52.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:52.346+0000 7f1cafa31700 1 -- 172.21.15.90:0/2657269430 shutdown_connections 2024-04-14T15:12:52.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:52.346+0000 7f1cafa31700 1 -- 172.21.15.90:0/2657269430 >> 172.21.15.90:0/2657269430 conn(0x7f1ca80fe3d0 msgr2=0x7f1ca81007f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:12:52.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:52.346+0000 7f1cafa31700 1 -- 172.21.15.90:0/2657269430 shutdown_connections 2024-04-14T15:12:52.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:52.346+0000 7f1cafa31700 1 -- 172.21.15.90:0/2657269430 wait complete. 2024-04-14T15:12:52.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:52.347+0000 7f1cafa31700 1 librados: starting msgr at 2024-04-14T15:12:52.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:52.347+0000 7f1cafa31700 1 librados: starting objecter 2024-04-14T15:12:52.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:52.347+0000 7f1cafa31700 1 Processor -- start 2024-04-14T15:12:52.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:52.347+0000 7f1cafa31700 1 -- start start 2024-04-14T15:12:52.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:52.347+0000 7f1cafa31700 1 librados: setting wanted keys 2024-04-14T15:12:52.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:52.347+0000 7f1cafa31700 1 librados: calling monclient init 2024-04-14T15:12:52.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:52.347+0000 7f1cafa31700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1ca810efb0 con 0x7f1ca8108880 2024-04-14T15:12:52.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:52.347+0000 7f1cafa31700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1ca81a8f10 con 0x7f1ca810f300 2024-04-14T15:12:52.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:52.347+0000 7f1cafa31700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1ca81aa0f0 con 0x7f1ca810b730 2024-04-14T15:12:52.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:52.347+0000 7f1cad7cd700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f1ca8108880 0x7f1ca810be00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:54642/0 (socket says 172.21.15.90:54642) 2024-04-14T15:12:52.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:52.347+0000 7f1cad7cd700 1 -- 172.21.15.90:0/3274751583 learned_addr learned my addr 172.21.15.90:0/3274751583 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:12:52.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:52.348+0000 7f1c9e7fc700 10 client.?.objecter ms_handle_connect 0x7f1ca8108880 2024-04-14T15:12:52.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:52.348+0000 7f1c9e7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:12:52.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:52.348+0000 7f1c9e7fc700 10 client.?.objecter ms_handle_connect 0x7f1ca810b730 2024-04-14T15:12:52.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:52.348+0000 7f1c9e7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:12:52.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:52.348+0000 7f1c9e7fc700 10 client.?.objecter ms_handle_connect 0x7f1ca810f300 2024-04-14T15:12:52.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:52.348+0000 7f1c9e7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:12:52.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:52.348+0000 7f1c9e7fc700 1 -- 172.21.15.90:0/3274751583 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4282585381 0 0) 0x7f1ca810efb0 con 0x7f1ca8108880 2024-04-14T15:12:52.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:52.348+0000 7f1c9e7fc700 1 -- 172.21.15.90:0/3274751583 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1c940031f0 con 0x7f1ca8108880 2024-04-14T15:12:52.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:52.348+0000 7f1c9e7fc700 1 -- 172.21.15.90:0/3274751583 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3456159274 0 0) 0x7f1ca81aa0f0 con 0x7f1ca810b730 2024-04-14T15:12:52.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:52.348+0000 7f1c9e7fc700 1 -- 172.21.15.90:0/3274751583 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1ca810efb0 con 0x7f1ca810b730 2024-04-14T15:12:52.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:52.348+0000 7f1c9e7fc700 1 -- 172.21.15.90:0/3274751583 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 533059237 0 0) 0x7f1ca81a8f10 con 0x7f1ca810f300 2024-04-14T15:12:52.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:52.348+0000 7f1c9e7fc700 1 -- 172.21.15.90:0/3274751583 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1ca81aa0f0 con 0x7f1ca810f300 2024-04-14T15:12:52.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:52.348+0000 7f1c9e7fc700 1 -- 172.21.15.90:0/3274751583 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1767157716 0 0) 0x7f1c940031f0 con 0x7f1ca8108880 2024-04-14T15:12:52.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:52.348+0000 7f1c9e7fc700 1 -- 172.21.15.90:0/3274751583 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1ca81a8f10 con 0x7f1ca8108880 2024-04-14T15:12:52.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:52.348+0000 7f1c9e7fc700 1 -- 172.21.15.90:0/3274751583 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f1c980018d0 con 0x7f1ca8108880 2024-04-14T15:12:52.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:52.348+0000 7f1c9e7fc700 1 -- 172.21.15.90:0/3274751583 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2795352727 0 0) 0x7f1ca810efb0 con 0x7f1ca810b730 2024-04-14T15:12:52.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:52.348+0000 7f1c9e7fc700 1 -- 172.21.15.90:0/3274751583 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1c940031f0 con 0x7f1ca810b730 2024-04-14T15:12:52.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:52.348+0000 7f1c9e7fc700 1 -- 172.21.15.90:0/3274751583 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f1ca4001830 con 0x7f1ca810b730 2024-04-14T15:12:52.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:52.348+0000 7f1c9e7fc700 1 -- 172.21.15.90:0/3274751583 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1555089571 0 0) 0x7f1ca81aa0f0 con 0x7f1ca810f300 2024-04-14T15:12:52.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:52.348+0000 7f1c9e7fc700 1 -- 172.21.15.90:0/3274751583 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1ca810efb0 con 0x7f1ca810f300 2024-04-14T15:12:52.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:52.348+0000 7f1c9e7fc700 1 -- 172.21.15.90:0/3274751583 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f1ca00017b0 con 0x7f1ca810f300 2024-04-14T15:12:52.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:52.348+0000 7f1c9e7fc700 1 -- 172.21.15.90:0/3274751583 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 44835277 0 0) 0x7f1ca81a8f10 con 0x7f1ca8108880 2024-04-14T15:12:52.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:52.348+0000 7f1c9e7fc700 1 -- 172.21.15.90:0/3274751583 >> v1:172.21.15.90:6790/0 conn(0x7f1ca810b730 legacy=0x7f1ca810c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:52.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:52.348+0000 7f1c9e7fc700 1 -- 172.21.15.90:0/3274751583 >> v1:172.21.15.92:6789/0 conn(0x7f1ca810f300 legacy=0x7f1ca81a67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:52.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:52.348+0000 7f1c9e7fc700 1 -- 172.21.15.90:0/3274751583 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ca81ab2d0 con 0x7f1ca8108880 2024-04-14T15:12:52.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:52.348+0000 7f1cafa31700 1 -- 172.21.15.90:0/3274751583 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1ca81a7f60 con 0x7f1ca8108880 2024-04-14T15:12:52.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:52.348+0000 7f1cafa31700 10 client.14898.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:12:52.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:52.348+0000 7f1cafa31700 1 -- 172.21.15.90:0/3274751583 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f1ca81a8420 con 0x7f1ca8108880 2024-04-14T15:12:52.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:52.348+0000 7f1cafa31700 1 librados: init done 2024-04-14T15:12:52.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:52.349+0000 7f1c9e7fc700 1 -- 172.21.15.90:0/3274751583 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f1c98003b40 con 0x7f1ca8108880 2024-04-14T15:12:52.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:52.349+0000 7f1c9e7fc700 1 -- 172.21.15.90:0/3274751583 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f1c98005ce0 con 0x7f1ca8108880 2024-04-14T15:12:52.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:52.349+0000 7f1cafa31700 1 -- 172.21.15.90:0/3274751583 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c8c005150 con 0x7f1ca8108880 2024-04-14T15:12:52.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:52.350+0000 7f1c9e7fc700 1 -- 172.21.15.90:0/3274751583 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f1c980036a0 con 0x7f1ca8108880 2024-04-14T15:12:52.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:52.350+0000 7f1c9e7fc700 1 -- 172.21.15.90:0/3274751583 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(105..105 src has 51..105) ==== 6320+0+0 (unknown 2277868576 0 0) 0x7f1c9808a220 con 0x7f1ca8108880 2024-04-14T15:12:52.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:52.350+0000 7f1c9e7fc700 10 client.14898.objecter ms_dispatch 0x7f1ca8001550 osd_map(105..105 src has 51..105) 2024-04-14T15:12:52.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:52.350+0000 7f1c9e7fc700 3 client.14898.objecter handle_osd_map got epochs [105,105] > 0 2024-04-14T15:12:52.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:52.350+0000 7f1c9e7fc700 3 client.14898.objecter handle_osd_map decoding full epoch 105 2024-04-14T15:12:52.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:52.350+0000 7f1c9e7fc700 20 client.14898.objecter dump_active .. 0 homeless 2024-04-14T15:12:52.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:52.351+0000 7f1c9e7fc700 10 client.14898.objecter ms_handle_connect 0x7f1c94071490 2024-04-14T15:12:52.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:52.355+0000 7f1c9e7fc700 1 -- 172.21.15.90:0/3274751583 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f1c98057370 con 0x7f1ca8108880 2024-04-14T15:12:52.363 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:12:52.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:52 smithi092 ceph-mon[33966]: pgmap v103: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:12:52.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:52.393 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:52.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:12:52.403 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:12:52.493 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:12:52.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:52.523 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:52.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:12:52.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:52.577+0000 7f1cafa31700 1 -- 172.21.15.90:0/3274751583 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7f1c8c005440 con 0x7f1ca8108880 2024-04-14T15:12:52.624 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:12:52.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:52.648 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:52.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:12:52.749 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:12:52.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:52.773 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:52.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:12:52.874 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:12:52.898 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:52.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:52.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:12:53.000 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:12:53.019 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:12:53.028 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:53.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:53.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:12:53.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:53.032+0000 7f1c9e7fc700 1 -- 172.21.15.90:0/3274751583 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v106) ==== 97+0+0 (unknown 4175496361 0 0) 0x7f1c9805b020 con 0x7f1ca8108880 2024-04-14T15:12:53.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:53.035+0000 7f1cafa31700 10 librados: watch_flush enter 2024-04-14T15:12:53.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:53.035+0000 7f1cafa31700 10 librados: watch_flush exit 2024-04-14T15:12:53.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:53.035+0000 7f1cafa31700 20 client.14898.objecter shutdown clearing up homeless session... 2024-04-14T15:12:53.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:53.035+0000 7f1cafa31700 10 client.14898.objecter successfully canceled tick 2024-04-14T15:12:53.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:53.036+0000 7f1cafa31700 1 -- 172.21.15.90:0/3274751583 >> v1:172.21.15.90:6800/420935427 conn(0x7f1c94071490 legacy=0x7f1c94073950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:53.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:53.036+0000 7f1cafa31700 1 -- 172.21.15.90:0/3274751583 >> v1:172.21.15.90:6789/0 conn(0x7f1ca8108880 legacy=0x7f1ca810be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:53.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:53.036+0000 7f1cafa31700 1 -- 172.21.15.90:0/3274751583 shutdown_connections 2024-04-14T15:12:53.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:53.036+0000 7f1cafa31700 1 -- 172.21.15.90:0/3274751583 >> 172.21.15.90:0/3274751583 conn(0x7f1ca80fe3d0 msgr2=0x7f1ca8112880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:12:53.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:53.037+0000 7f1cafa31700 1 -- 172.21.15.90:0/3274751583 shutdown_connections 2024-04-14T15:12:53.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:53.038+0000 7f1cafa31700 1 -- 172.21.15.90:0/3274751583 wait complete. 2024-04-14T15:12:53.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:53.038+0000 7f1cafa31700 1 librados: shutdown 2024-04-14T15:12:53.039 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is set 2024-04-14T15:12:53.130 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:12:53.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:53.155 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:53.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:12:53.256 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:12:53.284 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:53.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:53.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:12:53.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:53 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3274751583' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:12:53.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:53 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3274751583' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:12:53.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:53 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3274751583' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:12:53.453 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:12:53.476 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:53.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:53.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:12:53.578 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:12:53.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:53.603 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:53.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:12:53.704 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:12:53.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:53.734 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:53.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:12:53.759 INFO:tasks.workunit.client.0.smithi090.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-14T15:12:53.835 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:12:53.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:53.860 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:53.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:12:53.939 INFO:tasks.workunit.client.0.smithi090.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-14T15:12:53.962 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:12:53.980 INFO:tasks.workunit.client.0.smithi090.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-14T15:12:53.982 INFO:tasks.workunit.client.0.smithi090.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-14T15:12:53.982 INFO:tasks.workunit.client.0.smithi090.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-14T15:12:53.990 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:53.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:53.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:12:54.022 INFO:tasks.workunit.client.0.smithi090.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-14T15:12:54.026 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-14T15:12:54.026 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-14T15:12:54.042 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2024-04-14T15:12:54.044 INFO:tasks.workunit:Running workunit rados/test.sh... 2024-04-14T15:12:54.045 DEBUG:teuthology.orchestra.run.smithi090:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=05c9d46e7de5db2cb434c177a6bbfc56f472d09b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2024-04-14T15:12:54.092 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:12:54.098 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:12:53 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:12:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:12:54.098 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:54 smithi090 ceph-mon[41670]: pgmap v104: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T15:12:54.098 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:54 smithi090 ceph-mon[41670]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:12:54.099 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:54 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3274751583' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-14T15:12:54.099 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:54 smithi090 ceph-mon[41670]: osdmap e106: 8 total, 8 up, 8 in 2024-04-14T15:12:54.099 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:54 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:12:54.099 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:54 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:12:54.099 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:54 smithi090 ceph-mon[30399]: pgmap v104: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T15:12:54.099 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:54 smithi090 ceph-mon[30399]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:12:54.099 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:54 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3274751583' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-14T15:12:54.099 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:54 smithi090 ceph-mon[30399]: osdmap e106: 8 total, 8 up, 8 in 2024-04-14T15:12:54.099 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:54 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:12:54.099 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:54 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:12:54.108 INFO:tasks.workunit.client.0.smithi090.stderr:+ parallel=1 2024-04-14T15:12:54.108 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' '' = --serial ']' 2024-04-14T15:12:54.108 INFO:tasks.workunit.client.0.smithi090.stderr:+ crimson=0 2024-04-14T15:12:54.108 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' '' = --crimson ']' 2024-04-14T15:12:54.108 INFO:tasks.workunit.client.0.smithi090.stderr:+ color= 2024-04-14T15:12:54.108 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -t 1 ']' 2024-04-14T15:12:54.108 INFO:tasks.workunit.client.0.smithi090.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2024-04-14T15:12:54.108 INFO:tasks.workunit.client.0.smithi090.stderr:+ GTEST_OUTPUT_DIR=/home/ubuntu/cephtest/archive/unit_test_xml_report 2024-04-14T15:12:54.108 INFO:tasks.workunit.client.0.smithi090.stderr:+ mkdir -p /home/ubuntu/cephtest/archive/unit_test_xml_report 2024-04-14T15:12:54.109 INFO:tasks.workunit.client.0.smithi090.stderr:+ declare -A pids 2024-04-14T15:12:54.109 INFO:tasks.workunit.client.0.smithi090.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-14T15:12:54.110 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' 1 -eq 1 ']' 2024-04-14T15:12:54.110 INFO:tasks.workunit.client.0.smithi090.stderr:++ printf %25s api_aio 2024-04-14T15:12:54.110 INFO:tasks.workunit.client.0.smithi090.stderr:+ r=' api_aio' 2024-04-14T15:12:54.110 INFO:tasks.workunit.client.0.smithi090.stderr:++ echo api_aio 2024-04-14T15:12:54.110 INFO:tasks.workunit.client.0.smithi090.stderr:++ awk '{print $1}' 2024-04-14T15:12:54.111 INFO:tasks.workunit.client.0.smithi090.stderr:+ ff=api_aio 2024-04-14T15:12:54.111 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=93276 2024-04-14T15:12:54.111 INFO:tasks.workunit.client.0.smithi090.stderr:+ echo 'test api_aio on pid 93276' 2024-04-14T15:12:54.111 INFO:tasks.workunit.client.0.smithi090.stdout:test api_aio on pid 93276 2024-04-14T15:12:54.115 INFO:tasks.workunit.client.0.smithi090.stderr:+ pids[$f]=93276 2024-04-14T15:12:54.115 INFO:tasks.workunit.client.0.smithi090.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-14T15:12:54.115 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' 1 -eq 1 ']' 2024-04-14T15:12:54.115 INFO:tasks.workunit.client.0.smithi090.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2024-04-14T15:12:54.115 INFO:tasks.workunit.client.0.smithi090.stderr:++ printf %25s api_aio_pp 2024-04-14T15:12:54.115 INFO:tasks.workunit.client.0.smithi090.stderr:+ r=' api_aio_pp' 2024-04-14T15:12:54.115 INFO:tasks.workunit.client.0.smithi090.stderr:++ echo api_aio_pp 2024-04-14T15:12:54.115 INFO:tasks.workunit.client.0.smithi090.stderr:++ awk '{print $1}' 2024-04-14T15:12:54.115 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -f /etc/bashrc ']' 2024-04-14T15:12:54.115 INFO:tasks.workunit.client.0.smithi090.stderr:+ . /etc/bashrc 2024-04-14T15:12:54.115 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -z '' ']' 2024-04-14T15:12:54.115 INFO:tasks.workunit.client.0.smithi090.stderr:++ BASHRCSOURCED=Y 2024-04-14T15:12:54.116 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.116 INFO:tasks.workunit.client.0.smithi090.stderr:++ shopt -q login_shell 2024-04-14T15:12:54.116 INFO:tasks.workunit.client.0.smithi090.stderr:+++ umask 2024-04-14T15:12:54.116 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' 0022 -eq 0 ']' 2024-04-14T15:12:54.116 INFO:tasks.workunit.client.0.smithi090.stderr:++ SHELL=/bin/bash 2024-04-14T15:12:54.116 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.116 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-14T15:12:54.116 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.116 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-14T15:12:54.116 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-14T15:12:54.116 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.116 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-14T15:12:54.116 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.116 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-14T15:12:54.116 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.117 INFO:tasks.workunit.client.0.smithi090.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.117 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.117 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-14T15:12:54.117 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.117 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-14T15:12:54.117 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.117 INFO:tasks.workunit.client.0.smithi090.stderr:+ ff=api_aio_pp 2024-04-14T15:12:54.117 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=93283 2024-04-14T15:12:54.118 INFO:tasks.workunit.client.0.smithi090.stdout:test api_aio_pp on pid 93283 2024-04-14T15:12:54.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:54.122 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:54.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:12:54.124 INFO:tasks.workunit.client.0.smithi090.stderr:+ echo 'test api_aio_pp on pid 93283' 2024-04-14T15:12:54.124 INFO:tasks.workunit.client.0.smithi090.stderr:+ pids[$f]=93283 2024-04-14T15:12:54.124 INFO:tasks.workunit.client.0.smithi090.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-14T15:12:54.124 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' 1 -eq 1 ']' 2024-04-14T15:12:54.124 INFO:tasks.workunit.client.0.smithi090.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2024-04-14T15:12:54.124 INFO:tasks.workunit.client.0.smithi090.stderr:++ printf %25s api_io 2024-04-14T15:12:54.124 INFO:tasks.workunit.client.0.smithi090.stderr:+ r=' api_io' 2024-04-14T15:12:54.124 INFO:tasks.workunit.client.0.smithi090.stderr:++ echo api_io 2024-04-14T15:12:54.124 INFO:tasks.workunit.client.0.smithi090.stderr:++ awk '{print $1}' 2024-04-14T15:12:54.124 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -f /etc/bashrc ']' 2024-04-14T15:12:54.125 INFO:tasks.workunit.client.0.smithi090.stderr:+ . /etc/bashrc 2024-04-14T15:12:54.125 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -z '' ']' 2024-04-14T15:12:54.125 INFO:tasks.workunit.client.0.smithi090.stderr:++ BASHRCSOURCED=Y 2024-04-14T15:12:54.125 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.125 INFO:tasks.workunit.client.0.smithi090.stderr:++ shopt -q login_shell 2024-04-14T15:12:54.125 INFO:tasks.workunit.client.0.smithi090.stderr:+++ umask 2024-04-14T15:12:54.125 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'grep=grep --color=auto' 2024-04-14T15:12:54.125 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' 0022 -eq 0 ']' 2024-04-14T15:12:54.125 INFO:tasks.workunit.client.0.smithi090.stderr:++ SHELL=/bin/bash 2024-04-14T15:12:54.125 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-14T15:12:54.125 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-14T15:12:54.125 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.125 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.125 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-14T15:12:54.126 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-14T15:12:54.126 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.126 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-14T15:12:54.126 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.126 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorls.sh 2024-04-14T15:12:54.126 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' '!' -t 0 ']' 2024-04-14T15:12:54.126 INFO:tasks.workunit.client.0.smithi090.stderr:+++ return 2024-04-14T15:12:54.126 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-14T15:12:54.126 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.126 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.126 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-14T15:12:54.126 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-14T15:12:54.126 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.126 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.127 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-14T15:12:54.127 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-14T15:12:54.127 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.127 INFO:tasks.workunit.client.0.smithi090.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.127 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.127 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-14T15:12:54.127 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.127 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-14T15:12:54.127 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export S_COLORS=auto 2024-04-14T15:12:54.127 INFO:tasks.workunit.client.0.smithi090.stderr:+++ S_COLORS=auto 2024-04-14T15:12:54.127 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.127 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.127 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-14T15:12:54.127 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.127 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-14T15:12:54.128 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.128 INFO:tasks.workunit.client.0.smithi090.stderr:+ ff=api_io 2024-04-14T15:12:54.128 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=93292 2024-04-14T15:12:54.128 INFO:tasks.workunit.client.0.smithi090.stdout:test api_io on pid 93292 2024-04-14T15:12:54.132 INFO:tasks.workunit.client.0.smithi090.stderr:+ echo 'test api_io on pid 93292' 2024-04-14T15:12:54.132 INFO:tasks.workunit.client.0.smithi090.stderr:+ pids[$f]=93292 2024-04-14T15:12:54.132 INFO:tasks.workunit.client.0.smithi090.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-14T15:12:54.132 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' 1 -eq 1 ']' 2024-04-14T15:12:54.132 INFO:tasks.workunit.client.0.smithi090.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2024-04-14T15:12:54.132 INFO:tasks.workunit.client.0.smithi090.stderr:++ printf %25s api_io_pp 2024-04-14T15:12:54.132 INFO:tasks.workunit.client.0.smithi090.stderr:+ r=' api_io_pp' 2024-04-14T15:12:54.132 INFO:tasks.workunit.client.0.smithi090.stderr:++ echo api_io_pp 2024-04-14T15:12:54.132 INFO:tasks.workunit.client.0.smithi090.stderr:++ awk '{print $1}' 2024-04-14T15:12:54.133 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -f /etc/bashrc ']' 2024-04-14T15:12:54.133 INFO:tasks.workunit.client.0.smithi090.stderr:+ . /etc/bashrc 2024-04-14T15:12:54.133 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -z '' ']' 2024-04-14T15:12:54.133 INFO:tasks.workunit.client.0.smithi090.stderr:++ BASHRCSOURCED=Y 2024-04-14T15:12:54.133 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.133 INFO:tasks.workunit.client.0.smithi090.stderr:++ shopt -q login_shell 2024-04-14T15:12:54.133 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'grep=grep --color=auto' 2024-04-14T15:12:54.133 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-14T15:12:54.133 INFO:tasks.workunit.client.0.smithi090.stderr:+++ umask 2024-04-14T15:12:54.133 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-14T15:12:54.134 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.134 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-14T15:12:54.134 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.134 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorls.sh 2024-04-14T15:12:54.134 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' '!' -t 0 ']' 2024-04-14T15:12:54.134 INFO:tasks.workunit.client.0.smithi090.stderr:+++ return 2024-04-14T15:12:54.134 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.134 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-14T15:12:54.134 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.134 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-14T15:12:54.134 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' 0022 -eq 0 ']' 2024-04-14T15:12:54.135 INFO:tasks.workunit.client.0.smithi090.stderr:++ SHELL=/bin/bash 2024-04-14T15:12:54.135 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export S_COLORS=auto 2024-04-14T15:12:54.135 INFO:tasks.workunit.client.0.smithi090.stderr:+++ S_COLORS=auto 2024-04-14T15:12:54.135 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-14T15:12:54.135 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.135 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-14T15:12:54.135 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.135 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-14T15:12:54.135 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-14T15:12:54.135 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.135 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-14T15:12:54.135 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.135 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.135 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-14T15:12:54.135 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-14T15:12:54.136 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.136 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-14T15:12:54.136 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.136 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-14T15:12:54.136 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-14T15:12:54.136 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.136 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-14T15:12:54.136 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.136 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-14T15:12:54.137 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-14T15:12:54.137 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.137 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.137 INFO:tasks.workunit.client.0.smithi090.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.137 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.137 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-14T15:12:54.137 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.138 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-14T15:12:54.138 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.138 INFO:tasks.workunit.client.0.smithi090.stderr:+ ff=api_io_pp 2024-04-14T15:12:54.138 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=93304 2024-04-14T15:12:54.139 INFO:tasks.workunit.client.0.smithi090.stdout:test api_io_pp on pid 93304 2024-04-14T15:12:54.150 INFO:tasks.workunit.client.0.smithi090.stderr:+ echo 'test api_io_pp on pid 93304' 2024-04-14T15:12:54.150 INFO:tasks.workunit.client.0.smithi090.stderr:+ pids[$f]=93304 2024-04-14T15:12:54.150 INFO:tasks.workunit.client.0.smithi090.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-14T15:12:54.150 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' 1 -eq 1 ']' 2024-04-14T15:12:54.151 INFO:tasks.workunit.client.0.smithi090.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2024-04-14T15:12:54.151 INFO:tasks.workunit.client.0.smithi090.stderr:++ printf %25s api_asio 2024-04-14T15:12:54.151 INFO:tasks.workunit.client.0.smithi090.stderr:+ r=' api_asio' 2024-04-14T15:12:54.151 INFO:tasks.workunit.client.0.smithi090.stderr:++ echo api_asio 2024-04-14T15:12:54.151 INFO:tasks.workunit.client.0.smithi090.stderr:++ awk '{print $1}' 2024-04-14T15:12:54.151 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-14T15:12:54.152 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-14T15:12:54.152 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-14T15:12:54.152 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.152 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-14T15:12:54.152 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.152 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-14T15:12:54.153 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -f /etc/bashrc ']' 2024-04-14T15:12:54.153 INFO:tasks.workunit.client.0.smithi090.stderr:+ . /etc/bashrc 2024-04-14T15:12:54.153 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-14T15:12:54.153 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.153 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-14T15:12:54.153 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-14T15:12:54.154 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.154 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/gawk.sh 2024-04-14T15:12:54.154 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-14T15:12:54.154 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-14T15:12:54.154 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.154 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-14T15:12:54.155 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.155 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-14T15:12:54.155 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -z '' ']' 2024-04-14T15:12:54.155 INFO:tasks.workunit.client.0.smithi090.stderr:++ BASHRCSOURCED=Y 2024-04-14T15:12:54.155 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.155 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.156 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-14T15:12:54.156 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-14T15:12:54.156 INFO:tasks.workunit.client.0.smithi090.stderr:++ shopt -q login_shell 2024-04-14T15:12:54.156 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.156 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/lang.sh 2024-04-14T15:12:54.156 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.157 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'grep=grep --color=auto' 2024-04-14T15:12:54.157 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-14T15:12:54.157 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup 2024-04-14T15:12:54.157 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-14T15:12:54.157 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.157 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-14T15:12:54.157 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.158 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorls.sh 2024-04-14T15:12:54.158 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' '!' -t 0 ']' 2024-04-14T15:12:54.158 INFO:tasks.workunit.client.0.smithi090.stderr:+++ return 2024-04-14T15:12:54.158 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.158 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-14T15:12:54.158 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.159 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-14T15:12:54.159 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export S_COLORS=auto 2024-04-14T15:12:54.159 INFO:tasks.workunit.client.0.smithi090.stderr:+++ S_COLORS=auto 2024-04-14T15:12:54.159 INFO:tasks.workunit.client.0.smithi090.stderr:+++ umask 2024-04-14T15:12:54.159 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.159 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-14T15:12:54.160 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.160 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-14T15:12:54.160 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/locale 2024-04-14T15:12:54.160 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.160 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' 0022 -eq 0 ']' 2024-04-14T15:12:54.160 INFO:tasks.workunit.client.0.smithi090.stderr:++ SHELL=/bin/bash 2024-04-14T15:12:54.160 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.160 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-14T15:12:54.161 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.161 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-14T15:12:54.161 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-14T15:12:54.161 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.161 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-14T15:12:54.161 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.161 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-14T15:12:54.161 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.161 INFO:tasks.workunit.client.0.smithi090.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.162 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.162 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-14T15:12:54.162 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.162 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-14T15:12:54.162 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.162 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.162 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-14T15:12:54.162 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.162 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-14T15:12:54.163 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-14T15:12:54.163 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-14T15:12:54.163 INFO:tasks.workunit.client.0.smithi090.stderr:+ ff=api_asio 2024-04-14T15:12:54.163 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=93319 2024-04-14T15:12:54.163 INFO:tasks.workunit.client.0.smithi090.stdout:test api_asio on pid 93319 2024-04-14T15:12:54.175 INFO:tasks.workunit.client.0.smithi090.stderr:+ echo 'test api_asio on pid 93319' 2024-04-14T15:12:54.176 INFO:tasks.workunit.client.0.smithi090.stderr:+ pids[$f]=93319 2024-04-14T15:12:54.176 INFO:tasks.workunit.client.0.smithi090.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-14T15:12:54.176 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' 1 -eq 1 ']' 2024-04-14T15:12:54.176 INFO:tasks.workunit.client.0.smithi090.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2024-04-14T15:12:54.176 INFO:tasks.workunit.client.0.smithi090.stderr:++ printf %25s api_list 2024-04-14T15:12:54.176 INFO:tasks.workunit.client.0.smithi090.stderr:+ r=' api_list' 2024-04-14T15:12:54.177 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-14T15:12:54.177 INFO:tasks.workunit.client.0.smithi090.stderr:++ echo api_list 2024-04-14T15:12:54.177 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-14T15:12:54.177 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-14T15:12:54.177 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.177 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-14T15:12:54.177 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.177 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-14T15:12:54.177 INFO:tasks.workunit.client.0.smithi090.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-14T15:12:54.178 INFO:tasks.workunit.client.0.smithi090.stderr:++ awk '{print $1}' 2024-04-14T15:12:54.178 INFO:tasks.workunit.client.0.smithi090.stderr:++++ export LANG=en_US.UTF-8 2024-04-14T15:12:54.178 INFO:tasks.workunit.client.0.smithi090.stderr:++++ LANG=en_US.UTF-8 2024-04-14T15:12:54.178 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-14T15:12:54.178 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.178 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-14T15:12:54.178 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.178 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/gawk.sh 2024-04-14T15:12:54.178 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.178 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-14T15:12:54.178 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.179 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG=en_US.UTF-8 2024-04-14T15:12:54.179 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup config 2024-04-14T15:12:54.179 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.179 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.179 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-14T15:12:54.179 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.179 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/lang.sh 2024-04-14T15:12:54.179 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup 2024-04-14T15:12:54.179 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.180 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' dumb = linux ']' 2024-04-14T15:12:54.180 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.180 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-14T15:12:54.180 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-14T15:12:54.180 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.180 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/less.sh 2024-04-14T15:12:54.181 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -f /etc/bashrc ']' 2024-04-14T15:12:54.181 INFO:tasks.workunit.client.0.smithi090.stderr:+ . /etc/bashrc 2024-04-14T15:12:54.181 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-14T15:12:54.181 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-14T15:12:54.181 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-14T15:12:54.181 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.182 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.182 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-14T15:12:54.182 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-14T15:12:54.182 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.182 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/which2.sh 2024-04-14T15:12:54.182 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.183 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-14T15:12:54.183 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-14T15:12:54.183 INFO:tasks.workunit.client.0.smithi090.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-14T15:12:54.183 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.183 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/locale 2024-04-14T15:12:54.183 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -z '' ']' 2024-04-14T15:12:54.184 INFO:tasks.workunit.client.0.smithi090.stderr:++ BASHRCSOURCED=Y 2024-04-14T15:12:54.184 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.184 INFO:tasks.workunit.client.0.smithi090.stderr:++ shopt -q login_shell 2024-04-14T15:12:54.184 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'grep=grep --color=auto' 2024-04-14T15:12:54.184 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-14T15:12:54.184 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-14T15:12:54.185 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.185 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-14T15:12:54.185 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.185 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorls.sh 2024-04-14T15:12:54.185 INFO:tasks.workunit.client.0.smithi090.stderr:+++ umask 2024-04-14T15:12:54.185 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' '!' -t 0 ']' 2024-04-14T15:12:54.185 INFO:tasks.workunit.client.0.smithi090.stderr:+++ return 2024-04-14T15:12:54.186 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.186 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-14T15:12:54.186 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.186 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-14T15:12:54.186 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' 0022 -eq 0 ']' 2024-04-14T15:12:54.186 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export S_COLORS=auto 2024-04-14T15:12:54.186 INFO:tasks.workunit.client.0.smithi090.stderr:++ SHELL=/bin/bash 2024-04-14T15:12:54.187 INFO:tasks.workunit.client.0.smithi090.stderr:+++ S_COLORS=auto 2024-04-14T15:12:54.187 INFO:tasks.workunit.client.0.smithi090.stderr:+++++ readlink /proc/93276/exe 2024-04-14T15:12:54.187 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.187 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-14T15:12:54.187 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.187 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-14T15:12:54.187 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.188 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-14T15:12:54.188 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.188 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.188 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-14T15:12:54.188 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-14T15:12:54.188 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.189 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-14T15:12:54.189 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.189 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-14T15:12:54.189 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.189 INFO:tasks.workunit.client.0.smithi090.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.189 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.189 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-14T15:12:54.190 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.190 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-14T15:12:54.190 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.190 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.190 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-14T15:12:54.190 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.191 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-14T15:12:54.191 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-14T15:12:54.191 INFO:tasks.workunit.client.0.smithi090.stderr:++++ basename /usr/bin/bash 2024-04-14T15:12:54.191 INFO:tasks.workunit.client.0.smithi090.stderr:+ ff=api_list 2024-04-14T15:12:54.191 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-14T15:12:54.191 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=93336 2024-04-14T15:12:54.192 INFO:tasks.workunit.client.0.smithi090.stdout:test api_list on pid 93336 2024-04-14T15:12:54.201 INFO:tasks.workunit.client.0.smithi090.stderr:+ echo 'test api_list on pid 93336' 2024-04-14T15:12:54.201 INFO:tasks.workunit.client.0.smithi090.stderr:+ pids[$f]=93336 2024-04-14T15:12:54.202 INFO:tasks.workunit.client.0.smithi090.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-14T15:12:54.202 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' 1 -eq 1 ']' 2024-04-14T15:12:54.202 INFO:tasks.workunit.client.0.smithi090.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2024-04-14T15:12:54.202 INFO:tasks.workunit.client.0.smithi090.stderr:++ printf %25s api_lock 2024-04-14T15:12:54.202 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_declare='declare -f' 2024-04-14T15:12:54.202 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_opt=-f 2024-04-14T15:12:54.203 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export which_declare 2024-04-14T15:12:54.203 INFO:tasks.workunit.client.0.smithi090.stderr:+ r=' api_lock' 2024-04-14T15:12:54.203 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export -f which 2024-04-14T15:12:54.203 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset i 2024-04-14T15:12:54.203 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset -f pathmunge 2024-04-14T15:12:54.203 INFO:tasks.workunit.client.0.smithi090.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-14T15:12:54.204 INFO:tasks.workunit.client.0.smithi090.stderr:+ export PATH 2024-04-14T15:12:54.204 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-14T15:12:54.204 INFO:tasks.workunit.client.0.smithi090.stderr:+ unset rc 2024-04-14T15:12:54.204 INFO:tasks.workunit.client.0.smithi090.stderr:++ echo api_lock 2024-04-14T15:12:54.204 INFO:tasks.workunit.client.0.smithi090.stderr:+ ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2024-04-14T15:12:54.204 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-14T15:12:54.205 INFO:tasks.workunit.client.0.smithi090.stderr:++ awk '{print $1}' 2024-04-14T15:12:54.205 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-14T15:12:54.205 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-14T15:12:54.205 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.205 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-14T15:12:54.205 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.205 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-14T15:12:54.206 INFO:tasks.workunit.client.0.smithi090.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-14T15:12:54.206 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -f /etc/bashrc ']' 2024-04-14T15:12:54.206 INFO:tasks.workunit.client.0.smithi090.stderr:+ . /etc/bashrc 2024-04-14T15:12:54.206 INFO:tasks.workunit.client.0.smithi090.stderr:++++ export LANG=en_US.UTF-8 2024-04-14T15:12:54.206 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-14T15:12:54.206 INFO:tasks.workunit.client.0.smithi090.stderr:++++ LANG=en_US.UTF-8 2024-04-14T15:12:54.207 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.207 INFO:tasks.workunit.client.0.smithi090.stderr:+ tee ceph_test_rados_api_aio.log 2024-04-14T15:12:54.207 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-14T15:12:54.207 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.207 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.207 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/gawk.sh 2024-04-14T15:12:54.208 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-14T15:12:54.208 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.208 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG=en_US.UTF-8 2024-04-14T15:12:54.208 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup config 2024-04-14T15:12:54.208 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.208 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -z '' ']' 2024-04-14T15:12:54.209 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.209 INFO:tasks.workunit.client.0.smithi090.stderr:++ BASHRCSOURCED=Y 2024-04-14T15:12:54.209 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-14T15:12:54.209 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.209 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-14T15:12:54.209 INFO:tasks.workunit.client.0.smithi090.stderr:++ shopt -q login_shell 2024-04-14T15:12:54.209 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.210 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/lang.sh 2024-04-14T15:12:54.210 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-14T15:12:54.210 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.210 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup 2024-04-14T15:12:54.210 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-14T15:12:54.210 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' dumb = linux ']' 2024-04-14T15:12:54.210 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.211 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.211 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-14T15:12:54.211 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-14T15:12:54.211 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.211 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.211 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/less.sh 2024-04-14T15:12:54.211 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-14T15:12:54.212 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-14T15:12:54.212 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-14T15:12:54.212 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.212 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.212 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-14T15:12:54.212 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.212 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/which2.sh 2024-04-14T15:12:54.212 INFO:tasks.workunit.client.0.smithi090.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-14T15:12:54.212 INFO:tasks.workunit.client.0.smithi090.stderr:+ sed 's/^/ api_aio: /' 2024-04-14T15:12:54.212 INFO:tasks.workunit.client.0.smithi090.stderr:+++ umask 2024-04-14T15:12:54.212 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'grep=grep --color=auto' 2024-04-14T15:12:54.213 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-14T15:12:54.213 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-14T15:12:54.213 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.213 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-14T15:12:54.213 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.213 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorls.sh 2024-04-14T15:12:54.213 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/locale 2024-04-14T15:12:54.213 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' '!' -t 0 ']' 2024-04-14T15:12:54.213 INFO:tasks.workunit.client.0.smithi090.stderr:+++ return 2024-04-14T15:12:54.213 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.213 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-14T15:12:54.213 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.213 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-14T15:12:54.213 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export S_COLORS=auto 2024-04-14T15:12:54.214 INFO:tasks.workunit.client.0.smithi090.stderr:+++ S_COLORS=auto 2024-04-14T15:12:54.214 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.214 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-14T15:12:54.214 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.214 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-14T15:12:54.214 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.214 INFO:tasks.workunit.client.0.smithi090.stderr:+++++ readlink /proc/93283/exe 2024-04-14T15:12:54.214 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' 0022 -eq 0 ']' 2024-04-14T15:12:54.214 INFO:tasks.workunit.client.0.smithi090.stderr:++ SHELL=/bin/bash 2024-04-14T15:12:54.214 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.214 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-14T15:12:54.214 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.214 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-14T15:12:54.214 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-14T15:12:54.214 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.215 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-14T15:12:54.215 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.215 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-14T15:12:54.215 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.215 INFO:tasks.workunit.client.0.smithi090.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.215 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.215 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-14T15:12:54.215 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.215 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-14T15:12:54.215 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.215 INFO:tasks.workunit.client.0.smithi090.stdout:test api_lock on pid 93355 2024-04-14T15:12:54.220 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.220 INFO:tasks.workunit.client.0.smithi090.stderr:+ ff=api_lock 2024-04-14T15:12:54.220 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-14T15:12:54.220 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.220 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-14T15:12:54.220 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-14T15:12:54.220 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=93355 2024-04-14T15:12:54.220 INFO:tasks.workunit.client.0.smithi090.stderr:+ echo 'test api_lock on pid 93355' 2024-04-14T15:12:54.220 INFO:tasks.workunit.client.0.smithi090.stderr:+ pids[$f]=93355 2024-04-14T15:12:54.220 INFO:tasks.workunit.client.0.smithi090.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-14T15:12:54.220 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' 1 -eq 1 ']' 2024-04-14T15:12:54.220 INFO:tasks.workunit.client.0.smithi090.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2024-04-14T15:12:54.220 INFO:tasks.workunit.client.0.smithi090.stderr:++++ basename /usr/bin/bash 2024-04-14T15:12:54.221 INFO:tasks.workunit.client.0.smithi090.stderr:++ printf %25s api_lock_pp 2024-04-14T15:12:54.221 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-14T15:12:54.221 INFO:tasks.workunit.client.0.smithi090.stderr:+ r=' api_lock_pp' 2024-04-14T15:12:54.221 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_declare='declare -f' 2024-04-14T15:12:54.221 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-14T15:12:54.221 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_opt=-f 2024-04-14T15:12:54.221 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-14T15:12:54.221 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export which_declare 2024-04-14T15:12:54.221 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-14T15:12:54.221 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export -f which 2024-04-14T15:12:54.221 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.221 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset i 2024-04-14T15:12:54.221 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-14T15:12:54.221 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset -f pathmunge 2024-04-14T15:12:54.221 INFO:tasks.workunit.client.0.smithi090.stderr:++ echo api_lock_pp 2024-04-14T15:12:54.222 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.222 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-14T15:12:54.222 INFO:tasks.workunit.client.0.smithi090.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-14T15:12:54.222 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-14T15:12:54.222 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.222 INFO:tasks.workunit.client.0.smithi090.stderr:++ awk '{print $1}' 2024-04-14T15:12:54.222 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-14T15:12:54.222 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.222 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/gawk.sh 2024-04-14T15:12:54.222 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -f /etc/bashrc ']' 2024-04-14T15:12:54.222 INFO:tasks.workunit.client.0.smithi090.stderr:+ . /etc/bashrc 2024-04-14T15:12:54.222 INFO:tasks.workunit.client.0.smithi090.stderr:+ export PATH 2024-04-14T15:12:54.222 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-14T15:12:54.222 INFO:tasks.workunit.client.0.smithi090.stderr:+ unset rc 2024-04-14T15:12:54.222 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.223 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-14T15:12:54.223 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.223 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/lang.sh 2024-04-14T15:12:54.223 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -z '' ']' 2024-04-14T15:12:54.223 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup 2024-04-14T15:12:54.223 INFO:tasks.workunit.client.0.smithi090.stderr:++ BASHRCSOURCED=Y 2024-04-14T15:12:54.223 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.223 INFO:tasks.workunit.client.0.smithi090.stderr:++ shopt -q login_shell 2024-04-14T15:12:54.223 INFO:tasks.workunit.client.0.smithi090.stderr:+ ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2024-04-14T15:12:54.223 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-14T15:12:54.223 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-14T15:12:54.223 INFO:tasks.workunit.client.0.smithi090.stderr:+++ umask 2024-04-14T15:12:54.223 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-14T15:12:54.223 INFO:tasks.workunit.client.0.smithi090.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-14T15:12:54.223 INFO:tasks.workunit.client.0.smithi090.stderr:+ tee ceph_test_rados_api_aio_pp.log 2024-04-14T15:12:54.224 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.224 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-14T15:12:54.224 INFO:tasks.workunit.client.0.smithi090.stderr:++++ export LANG=en_US.UTF-8 2024-04-14T15:12:54.224 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.224 INFO:tasks.workunit.client.0.smithi090.stderr:++++ LANG=en_US.UTF-8 2024-04-14T15:12:54.224 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-14T15:12:54.224 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.224 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-14T15:12:54.224 INFO:tasks.workunit.client.0.smithi090.stderr:+ sed 's/^/ api_aio_pp: /' 2024-04-14T15:12:54.224 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-14T15:12:54.224 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.224 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.224 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG=en_US.UTF-8 2024-04-14T15:12:54.224 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup config 2024-04-14T15:12:54.224 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' 0022 -eq 0 ']' 2024-04-14T15:12:54.225 INFO:tasks.workunit.client.0.smithi090.stderr:++ SHELL=/bin/bash 2024-04-14T15:12:54.225 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.225 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.225 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.225 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-14T15:12:54.225 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' dumb = linux ']' 2024-04-14T15:12:54.225 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.225 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-14T15:12:54.225 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.225 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-14T15:12:54.225 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.225 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/less.sh 2024-04-14T15:12:54.225 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-14T15:12:54.225 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-14T15:12:54.225 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.226 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.226 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-14T15:12:54.226 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-14T15:12:54.226 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.226 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.226 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/which2.sh 2024-04-14T15:12:54.226 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-14T15:12:54.226 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.226 INFO:tasks.workunit.client.0.smithi090.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.226 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.226 INFO:tasks.workunit.client.0.smithi090.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-14T15:12:54.226 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-14T15:12:54.226 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.226 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-14T15:12:54.226 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.227 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/locale 2024-04-14T15:12:54.227 INFO:tasks.workunit.client.0.smithi090.stderr:+++++ readlink /proc/93292/exe 2024-04-14T15:12:54.227 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'grep=grep --color=auto' 2024-04-14T15:12:54.227 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-14T15:12:54.227 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-14T15:12:54.227 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.227 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-14T15:12:54.227 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.227 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorls.sh 2024-04-14T15:12:54.227 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' '!' -t 0 ']' 2024-04-14T15:12:54.227 INFO:tasks.workunit.client.0.smithi090.stderr:+++ return 2024-04-14T15:12:54.227 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.227 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-14T15:12:54.227 INFO:tasks.workunit.client.0.smithi090.stderr:++++ basename /usr/bin/bash 2024-04-14T15:12:54.227 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.228 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-14T15:12:54.228 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export S_COLORS=auto 2024-04-14T15:12:54.228 INFO:tasks.workunit.client.0.smithi090.stderr:+++ S_COLORS=auto 2024-04-14T15:12:54.228 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.228 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-14T15:12:54.228 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.228 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-14T15:12:54.228 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.228 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.228 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-14T15:12:54.228 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.228 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-14T15:12:54.228 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-14T15:12:54.228 INFO:tasks.workunit.client.0.smithi090.stderr:+ ff=api_lock_pp 2024-04-14T15:12:54.229 INFO:tasks.workunit.client.0.smithi090.stdout:test api_lock_pp on pid 93377 2024-04-14T15:12:54.229 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:12:54.233 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=93377 2024-04-14T15:12:54.233 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-14T15:12:54.234 INFO:tasks.workunit.client.0.smithi090.stderr:+ echo 'test api_lock_pp on pid 93377' 2024-04-14T15:12:54.234 INFO:tasks.workunit.client.0.smithi090.stderr:+ pids[$f]=93377 2024-04-14T15:12:54.234 INFO:tasks.workunit.client.0.smithi090.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-14T15:12:54.234 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' 1 -eq 1 ']' 2024-04-14T15:12:54.234 INFO:tasks.workunit.client.0.smithi090.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2024-04-14T15:12:54.234 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_declare='declare -f' 2024-04-14T15:12:54.234 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_opt=-f 2024-04-14T15:12:54.234 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export which_declare 2024-04-14T15:12:54.234 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export -f which 2024-04-14T15:12:54.234 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset i 2024-04-14T15:12:54.234 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset -f pathmunge 2024-04-14T15:12:54.234 INFO:tasks.workunit.client.0.smithi090.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-14T15:12:54.234 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-14T15:12:54.235 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-14T15:12:54.235 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-14T15:12:54.235 INFO:tasks.workunit.client.0.smithi090.stderr:+ export PATH 2024-04-14T15:12:54.235 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.235 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-14T15:12:54.235 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-14T15:12:54.235 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.235 INFO:tasks.workunit.client.0.smithi090.stderr:+ unset rc 2024-04-14T15:12:54.235 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-14T15:12:54.235 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-14T15:12:54.235 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.235 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-14T15:12:54.235 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.235 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/gawk.sh 2024-04-14T15:12:54.236 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.236 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-14T15:12:54.236 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.236 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/lang.sh 2024-04-14T15:12:54.236 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup 2024-04-14T15:12:54.236 INFO:tasks.workunit.client.0.smithi090.stderr:+ tee ceph_test_rados_api_io.log 2024-04-14T15:12:54.236 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/locale 2024-04-14T15:12:54.236 INFO:tasks.workunit.client.0.smithi090.stderr:+ sed 's/^/ api_io: /' 2024-04-14T15:12:54.236 INFO:tasks.workunit.client.0.smithi090.stderr:+ ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2024-04-14T15:12:54.236 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -f /etc/bashrc ']' 2024-04-14T15:12:54.236 INFO:tasks.workunit.client.0.smithi090.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-14T15:12:54.236 INFO:tasks.workunit.client.0.smithi090.stderr:++++ export LANG=en_US.UTF-8 2024-04-14T15:12:54.236 INFO:tasks.workunit.client.0.smithi090.stderr:++++ LANG=en_US.UTF-8 2024-04-14T15:12:54.236 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.236 INFO:tasks.workunit.client.0.smithi090.stderr:+ . /etc/bashrc 2024-04-14T15:12:54.237 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-14T15:12:54.237 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.237 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG=en_US.UTF-8 2024-04-14T15:12:54.237 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup config 2024-04-14T15:12:54.237 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.237 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -z '' ']' 2024-04-14T15:12:54.237 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.237 INFO:tasks.workunit.client.0.smithi090.stderr:++ BASHRCSOURCED=Y 2024-04-14T15:12:54.237 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' dumb = linux ']' 2024-04-14T15:12:54.237 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.237 INFO:tasks.workunit.client.0.smithi090.stderr:++ shopt -q login_shell 2024-04-14T15:12:54.237 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.237 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-14T15:12:54.237 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.237 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/less.sh 2024-04-14T15:12:54.238 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-14T15:12:54.238 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.238 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-14T15:12:54.238 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.238 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/which2.sh 2024-04-14T15:12:54.238 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'grep=grep --color=auto' 2024-04-14T15:12:54.238 INFO:tasks.workunit.client.0.smithi090.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-14T15:12:54.238 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-14T15:12:54.238 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-14T15:12:54.238 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.238 INFO:tasks.workunit.client.0.smithi090.stderr:+++ umask 2024-04-14T15:12:54.238 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-14T15:12:54.238 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.238 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorls.sh 2024-04-14T15:12:54.238 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' '!' -t 0 ']' 2024-04-14T15:12:54.239 INFO:tasks.workunit.client.0.smithi090.stderr:+++ return 2024-04-14T15:12:54.239 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.239 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-14T15:12:54.239 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.239 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-14T15:12:54.239 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export S_COLORS=auto 2024-04-14T15:12:54.239 INFO:tasks.workunit.client.0.smithi090.stderr:+++ S_COLORS=auto 2024-04-14T15:12:54.239 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.239 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-14T15:12:54.239 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.239 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-14T15:12:54.239 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' 0022 -eq 0 ']' 2024-04-14T15:12:54.239 INFO:tasks.workunit.client.0.smithi090.stderr:++ SHELL=/bin/bash 2024-04-14T15:12:54.239 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.239 INFO:tasks.workunit.client.0.smithi090.stderr:++ printf %25s api_misc 2024-04-14T15:12:54.240 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.240 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-14T15:12:54.240 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.240 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-14T15:12:54.240 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-14T15:12:54.240 INFO:tasks.workunit.client.0.smithi090.stderr:+ r=' api_misc' 2024-04-14T15:12:54.240 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.240 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-14T15:12:54.240 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.240 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-14T15:12:54.240 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.240 INFO:tasks.workunit.client.0.smithi090.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.240 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.240 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-14T15:12:54.240 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.240 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-14T15:12:54.241 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-14T15:12:54.241 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.241 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-14T15:12:54.241 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-14T15:12:54.241 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.241 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-14T15:12:54.241 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.241 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-14T15:12:54.241 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-14T15:12:54.241 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.241 INFO:tasks.workunit.client.0.smithi090.stderr:+++++ readlink /proc/93304/exe 2024-04-14T15:12:54.241 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.241 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-14T15:12:54.241 INFO:tasks.workunit.client.0.smithi090.stderr:++ awk '{print $1}' 2024-04-14T15:12:54.241 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.242 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-14T15:12:54.242 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-14T15:12:54.242 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-14T15:12:54.242 INFO:tasks.workunit.client.0.smithi090.stderr:++++ basename /usr/bin/bash 2024-04-14T15:12:54.242 INFO:tasks.workunit.client.0.smithi090.stderr:++ echo api_misc 2024-04-14T15:12:54.242 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_declare='declare -f' 2024-04-14T15:12:54.242 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_opt=-f 2024-04-14T15:12:54.242 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export which_declare 2024-04-14T15:12:54.242 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export -f which 2024-04-14T15:12:54.242 INFO:tasks.workunit.client.0.smithi090.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-14T15:12:54.242 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-14T15:12:54.242 INFO:tasks.workunit.client.0.smithi090.stderr:++++ export LANG=en_US.UTF-8 2024-04-14T15:12:54.242 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset i 2024-04-14T15:12:54.242 INFO:tasks.workunit.client.0.smithi090.stderr:++++ LANG=en_US.UTF-8 2024-04-14T15:12:54.243 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset -f pathmunge 2024-04-14T15:12:54.243 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-14T15:12:54.243 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-14T15:12:54.243 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.243 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-14T15:12:54.243 INFO:tasks.workunit.client.0.smithi090.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-14T15:12:54.243 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.243 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-14T15:12:54.243 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.243 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.243 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-14T15:12:54.243 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG=en_US.UTF-8 2024-04-14T15:12:54.243 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup config 2024-04-14T15:12:54.243 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-14T15:12:54.243 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.243 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.244 INFO:tasks.workunit.client.0.smithi090.stderr:+ export PATH 2024-04-14T15:12:54.244 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-14T15:12:54.244 INFO:tasks.workunit.client.0.smithi090.stderr:+ unset rc 2024-04-14T15:12:54.244 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.244 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' dumb = linux ']' 2024-04-14T15:12:54.244 INFO:tasks.workunit.client.0.smithi090.stdout:test api_misc on pid 93399 2024-04-14T15:12:54.249 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.249 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-14T15:12:54.249 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.249 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/less.sh 2024-04-14T15:12:54.249 INFO:tasks.workunit.client.0.smithi090.stderr:+ ff=api_misc 2024-04-14T15:12:54.249 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-14T15:12:54.249 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.249 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-14T15:12:54.249 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.249 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/which2.sh 2024-04-14T15:12:54.249 INFO:tasks.workunit.client.0.smithi090.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-14T15:12:54.249 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=93399 2024-04-14T15:12:54.249 INFO:tasks.workunit.client.0.smithi090.stderr:+ echo 'test api_misc on pid 93399' 2024-04-14T15:12:54.250 INFO:tasks.workunit.client.0.smithi090.stderr:+ pids[$f]=93399 2024-04-14T15:12:54.250 INFO:tasks.workunit.client.0.smithi090.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-14T15:12:54.250 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' 1 -eq 1 ']' 2024-04-14T15:12:54.250 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'grep=grep --color=auto' 2024-04-14T15:12:54.250 INFO:tasks.workunit.client.0.smithi090.stderr:+ ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2024-04-14T15:12:54.250 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-14T15:12:54.250 INFO:tasks.workunit.client.0.smithi090.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2024-04-14T15:12:54.250 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-14T15:12:54.250 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.250 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-14T15:12:54.250 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.250 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorls.sh 2024-04-14T15:12:54.250 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' '!' -t 0 ']' 2024-04-14T15:12:54.250 INFO:tasks.workunit.client.0.smithi090.stderr:+++ return 2024-04-14T15:12:54.251 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.251 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-14T15:12:54.251 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.251 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-14T15:12:54.251 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export S_COLORS=auto 2024-04-14T15:12:54.251 INFO:tasks.workunit.client.0.smithi090.stderr:+++ S_COLORS=auto 2024-04-14T15:12:54.251 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.251 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-14T15:12:54.251 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.251 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-14T15:12:54.251 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.251 INFO:tasks.workunit.client.0.smithi090.stderr:+ tee ceph_test_rados_api_io_pp.log 2024-04-14T15:12:54.251 INFO:tasks.workunit.client.0.smithi090.stderr:+++++ readlink /proc/93319/exe 2024-04-14T15:12:54.251 INFO:tasks.workunit.client.0.smithi090.stderr:+ sed 's/^/ api_io_pp: /' 2024-04-14T15:12:54.251 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -f /etc/bashrc ']' 2024-04-14T15:12:54.252 INFO:tasks.workunit.client.0.smithi090.stderr:+ . /etc/bashrc 2024-04-14T15:12:54.252 INFO:tasks.workunit.client.0.smithi090.stderr:++++ basename /usr/bin/bash 2024-04-14T15:12:54.252 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -z '' ']' 2024-04-14T15:12:54.252 INFO:tasks.workunit.client.0.smithi090.stderr:++ BASHRCSOURCED=Y 2024-04-14T15:12:54.252 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.252 INFO:tasks.workunit.client.0.smithi090.stderr:++ shopt -q login_shell 2024-04-14T15:12:54.252 INFO:tasks.workunit.client.0.smithi090.stderr:++ printf %25s api_misc_pp 2024-04-14T15:12:54.252 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_declare='declare -f' 2024-04-14T15:12:54.252 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_opt=-f 2024-04-14T15:12:54.252 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export which_declare 2024-04-14T15:12:54.252 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export -f which 2024-04-14T15:12:54.252 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset i 2024-04-14T15:12:54.252 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset -f pathmunge 2024-04-14T15:12:54.252 INFO:tasks.workunit.client.0.smithi090.stderr:+ r=' api_misc_pp' 2024-04-14T15:12:54.252 INFO:tasks.workunit.client.0.smithi090.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-14T15:12:54.253 INFO:tasks.workunit.client.0.smithi090.stderr:+ export PATH 2024-04-14T15:12:54.253 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-14T15:12:54.253 INFO:tasks.workunit.client.0.smithi090.stderr:+ unset rc 2024-04-14T15:12:54.253 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-14T15:12:54.253 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-14T15:12:54.253 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-14T15:12:54.253 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.253 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-14T15:12:54.253 INFO:tasks.workunit.client.0.smithi090.stderr:+++ umask 2024-04-14T15:12:54.253 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.253 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-14T15:12:54.253 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-14T15:12:54.253 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.253 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-14T15:12:54.253 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.253 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/gawk.sh 2024-04-14T15:12:54.254 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' 0022 -eq 0 ']' 2024-04-14T15:12:54.254 INFO:tasks.workunit.client.0.smithi090.stderr:++ SHELL=/bin/bash 2024-04-14T15:12:54.254 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.254 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-14T15:12:54.254 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.254 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/lang.sh 2024-04-14T15:12:54.254 INFO:tasks.workunit.client.0.smithi090.stderr:+ ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2024-04-14T15:12:54.254 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup 2024-04-14T15:12:54.254 INFO:tasks.workunit.client.0.smithi090.stderr:+ tee ceph_test_rados_api_asio.log 2024-04-14T15:12:54.254 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.254 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-14T15:12:54.254 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.254 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-14T15:12:54.254 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-14T15:12:54.255 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.255 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-14T15:12:54.255 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.255 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-14T15:12:54.255 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.255 INFO:tasks.workunit.client.0.smithi090.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.255 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.255 INFO:tasks.workunit.client.0.smithi090.stderr:++ awk '{print $1}' 2024-04-14T15:12:54.255 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-14T15:12:54.255 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.255 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-14T15:12:54.255 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.255 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/locale 2024-04-14T15:12:54.255 INFO:tasks.workunit.client.0.smithi090.stderr:+ sed 's/^/ api_asio: /' 2024-04-14T15:12:54.255 INFO:tasks.workunit.client.0.smithi090.stderr:++ echo api_misc_pp 2024-04-14T15:12:54.256 INFO:tasks.workunit.client.0.smithi090.stdout:test api_misc_pp on pid 93419 2024-04-14T15:12:54.260 INFO:tasks.workunit.client.0.smithi090.stderr:+ ff=api_misc_pp 2024-04-14T15:12:54.260 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=93419 2024-04-14T15:12:54.261 INFO:tasks.workunit.client.0.smithi090.stderr:+ echo 'test api_misc_pp on pid 93419' 2024-04-14T15:12:54.261 INFO:tasks.workunit.client.0.smithi090.stderr:+ pids[$f]=93419 2024-04-14T15:12:54.261 INFO:tasks.workunit.client.0.smithi090.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-14T15:12:54.261 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' 1 -eq 1 ']' 2024-04-14T15:12:54.261 INFO:tasks.workunit.client.0.smithi090.stderr:++ printf %25s api_tier_pp 2024-04-14T15:12:54.261 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.261 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-14T15:12:54.261 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-14T15:12:54.261 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-14T15:12:54.261 INFO:tasks.workunit.client.0.smithi090.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2024-04-14T15:12:54.261 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.261 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-14T15:12:54.261 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-14T15:12:54.261 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-14T15:12:54.262 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.262 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-14T15:12:54.262 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.262 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-14T15:12:54.262 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-14T15:12:54.262 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.262 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-14T15:12:54.262 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.262 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/gawk.sh 2024-04-14T15:12:54.262 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.262 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-14T15:12:54.262 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.262 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/lang.sh 2024-04-14T15:12:54.262 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup 2024-04-14T15:12:54.263 INFO:tasks.workunit.client.0.smithi090.stderr:+ r=' api_tier_pp' 2024-04-14T15:12:54.263 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'grep=grep --color=auto' 2024-04-14T15:12:54.263 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-14T15:12:54.263 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-14T15:12:54.263 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-14T15:12:54.263 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.263 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-14T15:12:54.263 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/locale 2024-04-14T15:12:54.263 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.263 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorls.sh 2024-04-14T15:12:54.263 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' '!' -t 0 ']' 2024-04-14T15:12:54.263 INFO:tasks.workunit.client.0.smithi090.stderr:+++ return 2024-04-14T15:12:54.263 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.263 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-14T15:12:54.264 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.264 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-14T15:12:54.264 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export S_COLORS=auto 2024-04-14T15:12:54.264 INFO:tasks.workunit.client.0.smithi090.stderr:+++ S_COLORS=auto 2024-04-14T15:12:54.264 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.264 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-14T15:12:54.264 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.264 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-14T15:12:54.264 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.264 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -f /etc/bashrc ']' 2024-04-14T15:12:54.264 INFO:tasks.workunit.client.0.smithi090.stderr:+ . /etc/bashrc 2024-04-14T15:12:54.264 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -z '' ']' 2024-04-14T15:12:54.264 INFO:tasks.workunit.client.0.smithi090.stderr:++ BASHRCSOURCED=Y 2024-04-14T15:12:54.264 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.264 INFO:tasks.workunit.client.0.smithi090.stderr:++ shopt -q login_shell 2024-04-14T15:12:54.265 INFO:tasks.workunit.client.0.smithi090.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-14T15:12:54.265 INFO:tasks.workunit.client.0.smithi090.stderr:++++ export LANG=en_US.UTF-8 2024-04-14T15:12:54.265 INFO:tasks.workunit.client.0.smithi090.stderr:++++ LANG=en_US.UTF-8 2024-04-14T15:12:54.265 INFO:tasks.workunit.client.0.smithi090.stderr:+++ umask 2024-04-14T15:12:54.265 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.265 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-14T15:12:54.265 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.265 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG=en_US.UTF-8 2024-04-14T15:12:54.265 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup config 2024-04-14T15:12:54.265 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.265 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.265 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' 0022 -eq 0 ']' 2024-04-14T15:12:54.265 INFO:tasks.workunit.client.0.smithi090.stderr:++ SHELL=/bin/bash 2024-04-14T15:12:54.265 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-14T15:12:54.266 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.266 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' dumb = linux ']' 2024-04-14T15:12:54.266 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.266 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-14T15:12:54.266 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.266 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-14T15:12:54.266 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-14T15:12:54.266 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.266 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.266 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/less.sh 2024-04-14T15:12:54.266 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-14T15:12:54.266 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.266 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-14T15:12:54.266 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-14T15:12:54.267 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.267 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-14T15:12:54.267 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.267 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/which2.sh 2024-04-14T15:12:54.267 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-14T15:12:54.267 INFO:tasks.workunit.client.0.smithi090.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-14T15:12:54.267 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.267 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-14T15:12:54.267 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.267 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-14T15:12:54.267 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.267 INFO:tasks.workunit.client.0.smithi090.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.267 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.267 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-14T15:12:54.268 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.268 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-14T15:12:54.268 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.268 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-14T15:12:54.268 INFO:tasks.workunit.client.0.smithi090.stderr:+++++ readlink /proc/93355/exe 2024-04-14T15:12:54.268 INFO:tasks.workunit.client.0.smithi090.stderr:++++ basename /usr/bin/bash 2024-04-14T15:12:54.268 INFO:tasks.workunit.client.0.smithi090.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-14T15:12:54.268 INFO:tasks.workunit.client.0.smithi090.stderr:++++ export LANG=en_US.UTF-8 2024-04-14T15:12:54.268 INFO:tasks.workunit.client.0.smithi090.stderr:++++ LANG=en_US.UTF-8 2024-04-14T15:12:54.268 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.268 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-14T15:12:54.268 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.268 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG=en_US.UTF-8 2024-04-14T15:12:54.268 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup config 2024-04-14T15:12:54.268 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.269 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.269 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' dumb = linux ']' 2024-04-14T15:12:54.269 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.269 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-14T15:12:54.269 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.269 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/less.sh 2024-04-14T15:12:54.269 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-14T15:12:54.269 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.269 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-14T15:12:54.269 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.269 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/which2.sh 2024-04-14T15:12:54.269 INFO:tasks.workunit.client.0.smithi090.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-14T15:12:54.269 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_declare='declare -f' 2024-04-14T15:12:54.269 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_opt=-f 2024-04-14T15:12:54.269 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export which_declare 2024-04-14T15:12:54.270 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export -f which 2024-04-14T15:12:54.270 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset i 2024-04-14T15:12:54.270 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset -f pathmunge 2024-04-14T15:12:54.270 INFO:tasks.workunit.client.0.smithi090.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-14T15:12:54.270 INFO:tasks.workunit.client.0.smithi090.stderr:+ export PATH 2024-04-14T15:12:54.270 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-14T15:12:54.270 INFO:tasks.workunit.client.0.smithi090.stderr:+ unset rc 2024-04-14T15:12:54.270 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-14T15:12:54.270 INFO:tasks.workunit.client.0.smithi090.stderr:+++++ readlink /proc/93336/exe 2024-04-14T15:12:54.270 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-14T15:12:54.270 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-14T15:12:54.270 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.270 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-14T15:12:54.270 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.270 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-14T15:12:54.271 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-14T15:12:54.271 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.271 INFO:tasks.workunit.client.0.smithi090.stderr:+ ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2024-04-14T15:12:54.271 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-14T15:12:54.271 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-14T15:12:54.271 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-14T15:12:54.271 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.271 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-14T15:12:54.271 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.271 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-14T15:12:54.271 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-14T15:12:54.271 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.271 INFO:tasks.workunit.client.0.smithi090.stderr:++++ basename /usr/bin/bash 2024-04-14T15:12:54.271 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_declare='declare -f' 2024-04-14T15:12:54.271 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_opt=-f 2024-04-14T15:12:54.272 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export which_declare 2024-04-14T15:12:54.272 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export -f which 2024-04-14T15:12:54.272 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset i 2024-04-14T15:12:54.272 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset -f pathmunge 2024-04-14T15:12:54.272 INFO:tasks.workunit.client.0.smithi090.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-14T15:12:54.272 INFO:tasks.workunit.client.0.smithi090.stderr:+ export PATH 2024-04-14T15:12:54.272 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-14T15:12:54.272 INFO:tasks.workunit.client.0.smithi090.stderr:+ unset rc 2024-04-14T15:12:54.272 INFO:tasks.workunit.client.0.smithi090.stderr:+ sed 's/^/ api_list: /' 2024-04-14T15:12:54.272 INFO:tasks.workunit.client.0.smithi090.stderr:+ ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2024-04-14T15:12:54.272 INFO:tasks.workunit.client.0.smithi090.stderr:++ echo api_tier_pp 2024-04-14T15:12:54.272 INFO:tasks.workunit.client.0.smithi090.stderr:++ awk '{print $1}' 2024-04-14T15:12:54.272 INFO:tasks.workunit.client.0.smithi090.stderr:+ tee ceph_test_rados_api_lock.log 2024-04-14T15:12:54.272 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-14T15:12:54.273 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-14T15:12:54.273 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-14T15:12:54.273 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.273 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-14T15:12:54.273 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.273 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-14T15:12:54.273 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-14T15:12:54.273 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.273 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-14T15:12:54.273 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.273 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/gawk.sh 2024-04-14T15:12:54.273 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.273 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-14T15:12:54.273 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.273 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/lang.sh 2024-04-14T15:12:54.274 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup 2024-04-14T15:12:54.274 INFO:tasks.workunit.client.0.smithi090.stderr:+ tee ceph_test_rados_api_list.log 2024-04-14T15:12:54.274 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/locale 2024-04-14T15:12:54.274 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.274 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-14T15:12:54.274 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.274 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-14T15:12:54.274 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-14T15:12:54.274 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'grep=grep --color=auto' 2024-04-14T15:12:54.274 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-14T15:12:54.274 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-14T15:12:54.274 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.274 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-14T15:12:54.274 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.274 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorls.sh 2024-04-14T15:12:54.275 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' '!' -t 0 ']' 2024-04-14T15:12:54.275 INFO:tasks.workunit.client.0.smithi090.stderr:+++ return 2024-04-14T15:12:54.275 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.275 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-14T15:12:54.275 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.275 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-14T15:12:54.275 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export S_COLORS=auto 2024-04-14T15:12:54.275 INFO:tasks.workunit.client.0.smithi090.stderr:+++ S_COLORS=auto 2024-04-14T15:12:54.275 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.275 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-14T15:12:54.275 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.275 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-14T15:12:54.275 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.275 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-14T15:12:54.276 INFO:tasks.workunit.client.0.smithi090.stderr:+ ff=api_tier_pp 2024-04-14T15:12:54.276 INFO:tasks.workunit.client.0.smithi090.stdout:test api_tier_pp on pid 93452 2024-04-14T15:12:54.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:54.279 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:54.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:12:54.281 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=93452 2024-04-14T15:12:54.281 INFO:tasks.workunit.client.0.smithi090.stderr:+ echo 'test api_tier_pp on pid 93452' 2024-04-14T15:12:54.281 INFO:tasks.workunit.client.0.smithi090.stderr:+ pids[$f]=93452 2024-04-14T15:12:54.281 INFO:tasks.workunit.client.0.smithi090.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-14T15:12:54.281 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' 1 -eq 1 ']' 2024-04-14T15:12:54.281 INFO:tasks.workunit.client.0.smithi090.stderr:+ sed 's/^/ api_lock: /' 2024-04-14T15:12:54.281 INFO:tasks.workunit.client.0.smithi090.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-14T15:12:54.281 INFO:tasks.workunit.client.0.smithi090.stderr:++++ export LANG=en_US.UTF-8 2024-04-14T15:12:54.281 INFO:tasks.workunit.client.0.smithi090.stderr:++++ LANG=en_US.UTF-8 2024-04-14T15:12:54.281 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.281 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-14T15:12:54.282 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.282 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG=en_US.UTF-8 2024-04-14T15:12:54.282 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup config 2024-04-14T15:12:54.282 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.282 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.282 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' dumb = linux ']' 2024-04-14T15:12:54.282 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.282 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-14T15:12:54.282 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.282 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/less.sh 2024-04-14T15:12:54.282 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-14T15:12:54.282 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.282 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-14T15:12:54.282 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.283 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/which2.sh 2024-04-14T15:12:54.283 INFO:tasks.workunit.client.0.smithi090.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-14T15:12:54.283 INFO:tasks.workunit.client.0.smithi090.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2024-04-14T15:12:54.283 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -f /etc/bashrc ']' 2024-04-14T15:12:54.283 INFO:tasks.workunit.client.0.smithi090.stderr:+ . /etc/bashrc 2024-04-14T15:12:54.283 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -z '' ']' 2024-04-14T15:12:54.283 INFO:tasks.workunit.client.0.smithi090.stderr:++ BASHRCSOURCED=Y 2024-04-14T15:12:54.283 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.283 INFO:tasks.workunit.client.0.smithi090.stderr:++ shopt -q login_shell 2024-04-14T15:12:54.283 INFO:tasks.workunit.client.0.smithi090.stderr:+++ umask 2024-04-14T15:12:54.283 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' 0022 -eq 0 ']' 2024-04-14T15:12:54.283 INFO:tasks.workunit.client.0.smithi090.stderr:++ SHELL=/bin/bash 2024-04-14T15:12:54.283 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.283 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-14T15:12:54.283 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.284 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-14T15:12:54.284 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-14T15:12:54.284 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.284 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-14T15:12:54.284 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.284 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-14T15:12:54.284 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.284 INFO:tasks.workunit.client.0.smithi090.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.284 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.284 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-14T15:12:54.284 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.284 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-14T15:12:54.284 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.284 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-14T15:12:54.284 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-14T15:12:54.285 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-14T15:12:54.285 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.285 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-14T15:12:54.285 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.285 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-14T15:12:54.285 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-14T15:12:54.285 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.285 INFO:tasks.workunit.client.0.smithi090.stderr:+++++ readlink /proc/93377/exe 2024-04-14T15:12:54.285 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-14T15:12:54.285 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-14T15:12:54.285 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-14T15:12:54.285 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.285 INFO:tasks.workunit.client.0.smithi090.stderr:++++ basename /usr/bin/bash 2024-04-14T15:12:54.285 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-14T15:12:54.285 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.286 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-14T15:12:54.286 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-14T15:12:54.286 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.286 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-14T15:12:54.286 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.286 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/gawk.sh 2024-04-14T15:12:54.286 INFO:tasks.workunit.client.0.smithi090.stderr:++ printf %25s api_pool 2024-04-14T15:12:54.286 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.286 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-14T15:12:54.286 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.286 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/lang.sh 2024-04-14T15:12:54.286 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup 2024-04-14T15:12:54.286 INFO:tasks.workunit.client.0.smithi090.stderr:+ r=' api_pool' 2024-04-14T15:12:54.286 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/locale 2024-04-14T15:12:54.286 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_declare='declare -f' 2024-04-14T15:12:54.287 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_opt=-f 2024-04-14T15:12:54.287 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export which_declare 2024-04-14T15:12:54.287 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export -f which 2024-04-14T15:12:54.287 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset i 2024-04-14T15:12:54.287 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset -f pathmunge 2024-04-14T15:12:54.287 INFO:tasks.workunit.client.0.smithi090.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-14T15:12:54.287 INFO:tasks.workunit.client.0.smithi090.stderr:+ export PATH 2024-04-14T15:12:54.287 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-14T15:12:54.287 INFO:tasks.workunit.client.0.smithi090.stderr:+ unset rc 2024-04-14T15:12:54.287 INFO:tasks.workunit.client.0.smithi090.stderr:+ sed 's/^/ api_lock_pp: /' 2024-04-14T15:12:54.287 INFO:tasks.workunit.client.0.smithi090.stderr:+ tee ceph_test_rados_api_lock_pp.log 2024-04-14T15:12:54.287 INFO:tasks.workunit.client.0.smithi090.stderr:++ echo api_pool 2024-04-14T15:12:54.287 INFO:tasks.workunit.client.0.smithi090.stderr:+ ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2024-04-14T15:12:54.287 INFO:tasks.workunit.client.0.smithi090.stderr:++ awk '{print $1}' 2024-04-14T15:12:54.287 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.288 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-14T15:12:54.288 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.288 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-14T15:12:54.288 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-14T15:12:54.288 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-14T15:12:54.288 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-14T15:12:54.288 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-14T15:12:54.288 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.288 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-14T15:12:54.288 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.288 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-14T15:12:54.288 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-14T15:12:54.288 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.288 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-14T15:12:54.288 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.288 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/gawk.sh 2024-04-14T15:12:54.289 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.289 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-14T15:12:54.289 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.289 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/lang.sh 2024-04-14T15:12:54.289 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup 2024-04-14T15:12:54.289 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/locale 2024-04-14T15:12:54.289 INFO:tasks.workunit.client.0.smithi090.stdout:test api_pool on pid 93499 2024-04-14T15:12:54.293 INFO:tasks.workunit.client.0.smithi090.stderr:+ ff=api_pool 2024-04-14T15:12:54.293 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=93499 2024-04-14T15:12:54.293 INFO:tasks.workunit.client.0.smithi090.stderr:+ echo 'test api_pool on pid 93499' 2024-04-14T15:12:54.293 INFO:tasks.workunit.client.0.smithi090.stderr:+ pids[$f]=93499 2024-04-14T15:12:54.293 INFO:tasks.workunit.client.0.smithi090.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-14T15:12:54.293 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' 1 -eq 1 ']' 2024-04-14T15:12:54.293 INFO:tasks.workunit.client.0.smithi090.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2024-04-14T15:12:54.293 INFO:tasks.workunit.client.0.smithi090.stderr:++ printf %25s api_snapshots 2024-04-14T15:12:54.293 INFO:tasks.workunit.client.0.smithi090.stderr:+ r=' api_snapshots' 2024-04-14T15:12:54.293 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -f /etc/bashrc ']' 2024-04-14T15:12:54.293 INFO:tasks.workunit.client.0.smithi090.stderr:+ . /etc/bashrc 2024-04-14T15:12:54.294 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -z '' ']' 2024-04-14T15:12:54.294 INFO:tasks.workunit.client.0.smithi090.stderr:++ BASHRCSOURCED=Y 2024-04-14T15:12:54.294 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.294 INFO:tasks.workunit.client.0.smithi090.stderr:++ shopt -q login_shell 2024-04-14T15:12:54.294 INFO:tasks.workunit.client.0.smithi090.stderr:++ echo api_snapshots 2024-04-14T15:12:54.294 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-14T15:12:54.294 INFO:tasks.workunit.client.0.smithi090.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-14T15:12:54.294 INFO:tasks.workunit.client.0.smithi090.stderr:++++ export LANG=en_US.UTF-8 2024-04-14T15:12:54.294 INFO:tasks.workunit.client.0.smithi090.stderr:++++ LANG=en_US.UTF-8 2024-04-14T15:12:54.294 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.294 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-14T15:12:54.294 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.294 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG=en_US.UTF-8 2024-04-14T15:12:54.294 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup config 2024-04-14T15:12:54.295 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.295 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.295 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' dumb = linux ']' 2024-04-14T15:12:54.295 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.295 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-14T15:12:54.295 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.295 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/less.sh 2024-04-14T15:12:54.295 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-14T15:12:54.295 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.295 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-14T15:12:54.295 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.295 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/which2.sh 2024-04-14T15:12:54.295 INFO:tasks.workunit.client.0.smithi090.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-14T15:12:54.295 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'grep=grep --color=auto' 2024-04-14T15:12:54.295 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-14T15:12:54.296 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-14T15:12:54.296 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.296 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-14T15:12:54.296 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.296 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorls.sh 2024-04-14T15:12:54.296 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' '!' -t 0 ']' 2024-04-14T15:12:54.296 INFO:tasks.workunit.client.0.smithi090.stderr:+++ return 2024-04-14T15:12:54.296 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.296 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-14T15:12:54.296 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.296 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-14T15:12:54.296 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export S_COLORS=auto 2024-04-14T15:12:54.296 INFO:tasks.workunit.client.0.smithi090.stderr:+++ S_COLORS=auto 2024-04-14T15:12:54.296 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.296 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-14T15:12:54.297 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.297 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-14T15:12:54.297 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.297 INFO:tasks.workunit.client.0.smithi090.stderr:++ awk '{print $1}' 2024-04-14T15:12:54.297 INFO:tasks.workunit.client.0.smithi090.stderr:+++ umask 2024-04-14T15:12:54.297 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.297 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-14T15:12:54.297 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.297 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-14T15:12:54.297 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-14T15:12:54.297 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' 0022 -eq 0 ']' 2024-04-14T15:12:54.297 INFO:tasks.workunit.client.0.smithi090.stderr:++ SHELL=/bin/bash 2024-04-14T15:12:54.297 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-14T15:12:54.297 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.297 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-14T15:12:54.298 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.298 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-14T15:12:54.298 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-14T15:12:54.298 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.298 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-14T15:12:54.298 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.298 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-14T15:12:54.298 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.298 INFO:tasks.workunit.client.0.smithi090.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.298 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.298 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-14T15:12:54.298 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.298 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-14T15:12:54.298 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.298 INFO:tasks.workunit.client.0.smithi090.stderr:+ ff=api_snapshots 2024-04-14T15:12:54.299 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=93555 2024-04-14T15:12:54.299 INFO:tasks.workunit.client.0.smithi090.stdout:test api_snapshots on pid 93555 2024-04-14T15:12:54.300 INFO:tasks.workunit.client.0.smithi090.stderr:+ echo 'test api_snapshots on pid 93555' 2024-04-14T15:12:54.300 INFO:tasks.workunit.client.0.smithi090.stderr:+ pids[$f]=93555 2024-04-14T15:12:54.300 INFO:tasks.workunit.client.0.smithi090.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-14T15:12:54.301 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' 1 -eq 1 ']' 2024-04-14T15:12:54.301 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-14T15:12:54.301 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-14T15:12:54.301 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-14T15:12:54.301 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.301 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-14T15:12:54.301 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.301 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-14T15:12:54.301 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-14T15:12:54.301 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.301 INFO:tasks.workunit.client.0.smithi090.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2024-04-14T15:12:54.301 INFO:tasks.workunit.client.0.smithi090.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-14T15:12:54.301 INFO:tasks.workunit.client.0.smithi090.stderr:++++ export LANG=en_US.UTF-8 2024-04-14T15:12:54.301 INFO:tasks.workunit.client.0.smithi090.stderr:++++ LANG=en_US.UTF-8 2024-04-14T15:12:54.301 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.302 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-14T15:12:54.302 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.302 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG=en_US.UTF-8 2024-04-14T15:12:54.302 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup config 2024-04-14T15:12:54.302 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.302 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.302 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' dumb = linux ']' 2024-04-14T15:12:54.302 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.302 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-14T15:12:54.302 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.302 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/less.sh 2024-04-14T15:12:54.302 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-14T15:12:54.302 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.302 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -f /etc/bashrc ']' 2024-04-14T15:12:54.303 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-14T15:12:54.303 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.303 INFO:tasks.workunit.client.0.smithi090.stderr:+ . /etc/bashrc 2024-04-14T15:12:54.303 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/which2.sh 2024-04-14T15:12:54.303 INFO:tasks.workunit.client.0.smithi090.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-14T15:12:54.303 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -z '' ']' 2024-04-14T15:12:54.303 INFO:tasks.workunit.client.0.smithi090.stderr:++ BASHRCSOURCED=Y 2024-04-14T15:12:54.303 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.303 INFO:tasks.workunit.client.0.smithi090.stderr:++ shopt -q login_shell 2024-04-14T15:12:54.303 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'grep=grep --color=auto' 2024-04-14T15:12:54.303 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-14T15:12:54.303 INFO:tasks.workunit.client.0.smithi090.stderr:+++++ readlink /proc/93399/exe 2024-04-14T15:12:54.303 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-14T15:12:54.303 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.303 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-14T15:12:54.304 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.304 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorls.sh 2024-04-14T15:12:54.304 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' '!' -t 0 ']' 2024-04-14T15:12:54.304 INFO:tasks.workunit.client.0.smithi090.stderr:+++ umask 2024-04-14T15:12:54.304 INFO:tasks.workunit.client.0.smithi090.stderr:+++ return 2024-04-14T15:12:54.304 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.304 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-14T15:12:54.304 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.304 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-14T15:12:54.304 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export S_COLORS=auto 2024-04-14T15:12:54.304 INFO:tasks.workunit.client.0.smithi090.stderr:+++ S_COLORS=auto 2024-04-14T15:12:54.304 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.304 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-14T15:12:54.304 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.304 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-14T15:12:54.305 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.305 INFO:tasks.workunit.client.0.smithi090.stderr:+++++ readlink /proc/93419/exe 2024-04-14T15:12:54.305 INFO:tasks.workunit.client.0.smithi090.stderr:++ printf %25s api_snapshots_pp 2024-04-14T15:12:54.305 INFO:tasks.workunit.client.0.smithi090.stderr:+ r=' api_snapshots_pp' 2024-04-14T15:12:54.305 INFO:tasks.workunit.client.0.smithi090.stderr:++++ basename /usr/bin/bash 2024-04-14T15:12:54.305 INFO:tasks.workunit.client.0.smithi090.stderr:++ echo api_snapshots_pp 2024-04-14T15:12:54.305 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' 0022 -eq 0 ']' 2024-04-14T15:12:54.305 INFO:tasks.workunit.client.0.smithi090.stderr:++ SHELL=/bin/bash 2024-04-14T15:12:54.305 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.305 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-14T15:12:54.305 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.305 INFO:tasks.workunit.client.0.smithi090.stderr:++++ basename /usr/bin/bash 2024-04-14T15:12:54.305 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-14T15:12:54.305 INFO:tasks.workunit.client.0.smithi090.stderr:++ awk '{print $1}' 2024-04-14T15:12:54.306 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-14T15:12:54.306 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.306 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-14T15:12:54.306 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.306 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-14T15:12:54.306 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.306 INFO:tasks.workunit.client.0.smithi090.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.306 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.306 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-14T15:12:54.306 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.306 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-14T15:12:54.306 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.306 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_declare='declare -f' 2024-04-14T15:12:54.306 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_opt=-f 2024-04-14T15:12:54.306 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export which_declare 2024-04-14T15:12:54.307 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export -f which 2024-04-14T15:12:54.307 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset i 2024-04-14T15:12:54.307 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset -f pathmunge 2024-04-14T15:12:54.307 INFO:tasks.workunit.client.0.smithi090.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-14T15:12:54.307 INFO:tasks.workunit.client.0.smithi090.stderr:+ export PATH 2024-04-14T15:12:54.307 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-14T15:12:54.307 INFO:tasks.workunit.client.0.smithi090.stderr:+ unset rc 2024-04-14T15:12:54.307 INFO:tasks.workunit.client.0.smithi090.stdout:test api_snapshots_pp on pid 93590 2024-04-14T15:12:54.309 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-14T15:12:54.309 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-14T15:12:54.309 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-14T15:12:54.309 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.309 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-14T15:12:54.309 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.309 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-14T15:12:54.309 INFO:tasks.workunit.client.0.smithi090.stderr:+ sed 's/^/ api_misc: /' 2024-04-14T15:12:54.310 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-14T15:12:54.310 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.310 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-14T15:12:54.310 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.310 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/gawk.sh 2024-04-14T15:12:54.310 INFO:tasks.workunit.client.0.smithi090.stderr:+ ff=api_snapshots_pp 2024-04-14T15:12:54.310 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.310 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-14T15:12:54.310 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.310 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/lang.sh 2024-04-14T15:12:54.310 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup 2024-04-14T15:12:54.310 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=93590 2024-04-14T15:12:54.310 INFO:tasks.workunit.client.0.smithi090.stderr:+ echo 'test api_snapshots_pp on pid 93590' 2024-04-14T15:12:54.310 INFO:tasks.workunit.client.0.smithi090.stderr:+ pids[$f]=93590 2024-04-14T15:12:54.310 INFO:tasks.workunit.client.0.smithi090.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-14T15:12:54.311 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' 1 -eq 1 ']' 2024-04-14T15:12:54.311 INFO:tasks.workunit.client.0.smithi090.stderr:++ printf %25s api_stat 2024-04-14T15:12:54.311 INFO:tasks.workunit.client.0.smithi090.stderr:+ r=' api_stat' 2024-04-14T15:12:54.311 INFO:tasks.workunit.client.0.smithi090.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2024-04-14T15:12:54.311 INFO:tasks.workunit.client.0.smithi090.stderr:+ ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2024-04-14T15:12:54.311 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_declare='declare -f' 2024-04-14T15:12:54.311 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_opt=-f 2024-04-14T15:12:54.311 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export which_declare 2024-04-14T15:12:54.311 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export -f which 2024-04-14T15:12:54.311 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset i 2024-04-14T15:12:54.311 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset -f pathmunge 2024-04-14T15:12:54.311 INFO:tasks.workunit.client.0.smithi090.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-14T15:12:54.311 INFO:tasks.workunit.client.0.smithi090.stderr:+ export PATH 2024-04-14T15:12:54.311 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-14T15:12:54.312 INFO:tasks.workunit.client.0.smithi090.stderr:+ unset rc 2024-04-14T15:12:54.312 INFO:tasks.workunit.client.0.smithi090.stderr:++ echo api_stat 2024-04-14T15:12:54.312 INFO:tasks.workunit.client.0.smithi090.stderr:+ ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2024-04-14T15:12:54.312 INFO:tasks.workunit.client.0.smithi090.stderr:+ sed 's/^/ api_misc_pp: /' 2024-04-14T15:12:54.312 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -f /etc/bashrc ']' 2024-04-14T15:12:54.312 INFO:tasks.workunit.client.0.smithi090.stderr:+ . /etc/bashrc 2024-04-14T15:12:54.312 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -z '' ']' 2024-04-14T15:12:54.312 INFO:tasks.workunit.client.0.smithi090.stderr:++ BASHRCSOURCED=Y 2024-04-14T15:12:54.312 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.312 INFO:tasks.workunit.client.0.smithi090.stderr:++ shopt -q login_shell 2024-04-14T15:12:54.312 INFO:tasks.workunit.client.0.smithi090.stderr:++ awk '{print $1}' 2024-04-14T15:12:54.312 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-14T15:12:54.312 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-14T15:12:54.312 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-14T15:12:54.312 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.313 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-14T15:12:54.313 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.313 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-14T15:12:54.313 INFO:tasks.workunit.client.0.smithi090.stderr:+++ umask 2024-04-14T15:12:54.313 INFO:tasks.workunit.client.0.smithi090.stdout:test api_stat on pid 93615 2024-04-14T15:12:54.319 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' 0022 -eq 0 ']' 2024-04-14T15:12:54.319 INFO:tasks.workunit.client.0.smithi090.stderr:++ SHELL=/bin/bash 2024-04-14T15:12:54.319 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.319 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-14T15:12:54.319 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.319 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-14T15:12:54.319 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-14T15:12:54.319 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-14T15:12:54.319 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.319 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.320 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-14T15:12:54.320 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.320 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-14T15:12:54.320 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.320 INFO:tasks.workunit.client.0.smithi090.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.320 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.320 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-14T15:12:54.320 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.320 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-14T15:12:54.320 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.320 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'grep=grep --color=auto' 2024-04-14T15:12:54.320 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-14T15:12:54.320 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-14T15:12:54.320 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.321 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-14T15:12:54.321 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.321 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorls.sh 2024-04-14T15:12:54.321 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' '!' -t 0 ']' 2024-04-14T15:12:54.321 INFO:tasks.workunit.client.0.smithi090.stderr:+++ return 2024-04-14T15:12:54.321 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.321 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-14T15:12:54.321 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.321 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-14T15:12:54.321 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export S_COLORS=auto 2024-04-14T15:12:54.321 INFO:tasks.workunit.client.0.smithi090.stderr:+++ S_COLORS=auto 2024-04-14T15:12:54.321 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.321 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-14T15:12:54.321 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.322 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-14T15:12:54.322 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.322 INFO:tasks.workunit.client.0.smithi090.stderr:+ tee ceph_test_rados_api_misc_pp.log 2024-04-14T15:12:54.322 INFO:tasks.workunit.client.0.smithi090.stderr:+ ff=api_stat 2024-04-14T15:12:54.322 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=93615 2024-04-14T15:12:54.322 INFO:tasks.workunit.client.0.smithi090.stderr:+ echo 'test api_stat on pid 93615' 2024-04-14T15:12:54.322 INFO:tasks.workunit.client.0.smithi090.stderr:+ pids[$f]=93615 2024-04-14T15:12:54.322 INFO:tasks.workunit.client.0.smithi090.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-14T15:12:54.322 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' 1 -eq 1 ']' 2024-04-14T15:12:54.322 INFO:tasks.workunit.client.0.smithi090.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2024-04-14T15:12:54.322 INFO:tasks.workunit.client.0.smithi090.stderr:+ tee ceph_test_rados_api_misc.log 2024-04-14T15:12:54.322 INFO:tasks.workunit.client.0.smithi090.stderr:++ printf %25s api_stat_pp 2024-04-14T15:12:54.322 INFO:tasks.workunit.client.0.smithi090.stderr:+ r=' api_stat_pp' 2024-04-14T15:12:54.322 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/locale 2024-04-14T15:12:54.322 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.323 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-14T15:12:54.323 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.323 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-14T15:12:54.323 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-14T15:12:54.323 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -f /etc/bashrc ']' 2024-04-14T15:12:54.323 INFO:tasks.workunit.client.0.smithi090.stderr:+ . /etc/bashrc 2024-04-14T15:12:54.323 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-14T15:12:54.323 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -z '' ']' 2024-04-14T15:12:54.323 INFO:tasks.workunit.client.0.smithi090.stderr:++ BASHRCSOURCED=Y 2024-04-14T15:12:54.323 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.323 INFO:tasks.workunit.client.0.smithi090.stderr:++ shopt -q login_shell 2024-04-14T15:12:54.323 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-14T15:12:54.323 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-14T15:12:54.323 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-14T15:12:54.324 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.324 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-14T15:12:54.324 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.324 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-14T15:12:54.324 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-14T15:12:54.324 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.324 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-14T15:12:54.324 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.324 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/gawk.sh 2024-04-14T15:12:54.324 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.324 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-14T15:12:54.324 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.324 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/lang.sh 2024-04-14T15:12:54.324 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup 2024-04-14T15:12:54.324 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/locale 2024-04-14T15:12:54.325 INFO:tasks.workunit.client.0.smithi090.stderr:+++ umask 2024-04-14T15:12:54.325 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' 0022 -eq 0 ']' 2024-04-14T15:12:54.325 INFO:tasks.workunit.client.0.smithi090.stderr:++ SHELL=/bin/bash 2024-04-14T15:12:54.325 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.325 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-14T15:12:54.325 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.325 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-14T15:12:54.325 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-14T15:12:54.325 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-14T15:12:54.325 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-14T15:12:54.325 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.325 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-14T15:12:54.325 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-14T15:12:54.325 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.325 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.326 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-14T15:12:54.326 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-14T15:12:54.326 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.326 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-14T15:12:54.326 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-14T15:12:54.326 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.326 INFO:tasks.workunit.client.0.smithi090.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.326 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.326 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.326 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-14T15:12:54.326 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.326 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-14T15:12:54.326 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.326 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'grep=grep --color=auto' 2024-04-14T15:12:54.327 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-14T15:12:54.327 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-14T15:12:54.327 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.327 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-14T15:12:54.327 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.327 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorls.sh 2024-04-14T15:12:54.327 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' '!' -t 0 ']' 2024-04-14T15:12:54.327 INFO:tasks.workunit.client.0.smithi090.stderr:+++ return 2024-04-14T15:12:54.327 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.327 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-14T15:12:54.327 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.327 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-14T15:12:54.327 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export S_COLORS=auto 2024-04-14T15:12:54.327 INFO:tasks.workunit.client.0.smithi090.stderr:+++ S_COLORS=auto 2024-04-14T15:12:54.327 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.327 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-14T15:12:54.328 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.328 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-14T15:12:54.328 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.328 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.328 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-14T15:12:54.328 INFO:tasks.workunit.client.0.smithi090.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-14T15:12:54.328 INFO:tasks.workunit.client.0.smithi090.stderr:++++ export LANG=en_US.UTF-8 2024-04-14T15:12:54.328 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.328 INFO:tasks.workunit.client.0.smithi090.stderr:++++ LANG=en_US.UTF-8 2024-04-14T15:12:54.328 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-14T15:12:54.328 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-14T15:12:54.328 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.328 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-14T15:12:54.328 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.328 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG=en_US.UTF-8 2024-04-14T15:12:54.329 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup config 2024-04-14T15:12:54.329 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.329 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.329 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' dumb = linux ']' 2024-04-14T15:12:54.329 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.329 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-14T15:12:54.329 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.329 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/less.sh 2024-04-14T15:12:54.329 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-14T15:12:54.329 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.329 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-14T15:12:54.329 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.329 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/which2.sh 2024-04-14T15:12:54.329 INFO:tasks.workunit.client.0.smithi090.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-14T15:12:54.329 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-14T15:12:54.329 INFO:tasks.workunit.client.0.smithi090.stderr:+++++ readlink /proc/93452/exe 2024-04-14T15:12:54.330 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'grep=grep --color=auto' 2024-04-14T15:12:54.330 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-14T15:12:54.330 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-14T15:12:54.330 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.330 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-14T15:12:54.330 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.330 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorls.sh 2024-04-14T15:12:54.330 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' '!' -t 0 ']' 2024-04-14T15:12:54.330 INFO:tasks.workunit.client.0.smithi090.stderr:+++ return 2024-04-14T15:12:54.330 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.330 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-14T15:12:54.330 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.330 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-14T15:12:54.330 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export S_COLORS=auto 2024-04-14T15:12:54.330 INFO:tasks.workunit.client.0.smithi090.stderr:+++ S_COLORS=auto 2024-04-14T15:12:54.331 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.331 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-14T15:12:54.331 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.331 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-14T15:12:54.331 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.331 INFO:tasks.workunit.client.0.smithi090.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-14T15:12:54.331 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-14T15:12:54.331 INFO:tasks.workunit.client.0.smithi090.stderr:++++ export LANG=en_US.UTF-8 2024-04-14T15:12:54.331 INFO:tasks.workunit.client.0.smithi090.stderr:++++ LANG=en_US.UTF-8 2024-04-14T15:12:54.331 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-14T15:12:54.331 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-14T15:12:54.331 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.331 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-14T15:12:54.331 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.331 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-14T15:12:54.332 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.332 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.332 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-14T15:12:54.332 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG=en_US.UTF-8 2024-04-14T15:12:54.332 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup config 2024-04-14T15:12:54.332 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.332 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-14T15:12:54.332 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.332 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-14T15:12:54.332 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.332 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/gawk.sh 2024-04-14T15:12:54.332 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.332 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' dumb = linux ']' 2024-04-14T15:12:54.332 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.332 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-14T15:12:54.332 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.333 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/less.sh 2024-04-14T15:12:54.333 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.333 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-14T15:12:54.333 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.333 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/lang.sh 2024-04-14T15:12:54.333 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-14T15:12:54.333 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.333 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup 2024-04-14T15:12:54.333 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-14T15:12:54.333 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.333 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/which2.sh 2024-04-14T15:12:54.333 INFO:tasks.workunit.client.0.smithi090.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-14T15:12:54.333 INFO:tasks.workunit.client.0.smithi090.stderr:++ awk '{print $1}' 2024-04-14T15:12:54.333 INFO:tasks.workunit.client.0.smithi090.stderr:+++++ readlink /proc/93499/exe 2024-04-14T15:12:54.333 INFO:tasks.workunit.client.0.smithi090.stderr:++++ basename /usr/bin/bash 2024-04-14T15:12:54.334 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-14T15:12:54.334 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-14T15:12:54.334 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-14T15:12:54.334 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.334 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-14T15:12:54.334 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.334 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-14T15:12:54.334 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-14T15:12:54.334 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.334 INFO:tasks.workunit.client.0.smithi090.stderr:++ echo api_stat_pp 2024-04-14T15:12:54.334 INFO:tasks.workunit.client.0.smithi090.stderr:++++ basename /usr/bin/bash 2024-04-14T15:12:54.334 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_declare='declare -f' 2024-04-14T15:12:54.334 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_opt=-f 2024-04-14T15:12:54.334 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export which_declare 2024-04-14T15:12:54.334 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export -f which 2024-04-14T15:12:54.335 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset i 2024-04-14T15:12:54.335 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset -f pathmunge 2024-04-14T15:12:54.335 INFO:tasks.workunit.client.0.smithi090.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-14T15:12:54.335 INFO:tasks.workunit.client.0.smithi090.stderr:+ export PATH 2024-04-14T15:12:54.335 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-14T15:12:54.335 INFO:tasks.workunit.client.0.smithi090.stderr:+ unset rc 2024-04-14T15:12:54.335 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_declare='declare -f' 2024-04-14T15:12:54.335 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_opt=-f 2024-04-14T15:12:54.335 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export which_declare 2024-04-14T15:12:54.335 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export -f which 2024-04-14T15:12:54.335 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset i 2024-04-14T15:12:54.335 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset -f pathmunge 2024-04-14T15:12:54.335 INFO:tasks.workunit.client.0.smithi090.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-14T15:12:54.335 INFO:tasks.workunit.client.0.smithi090.stderr:+ tee ceph_test_rados_api_tier_pp.log 2024-04-14T15:12:54.335 INFO:tasks.workunit.client.0.smithi090.stderr:+ export PATH 2024-04-14T15:12:54.335 INFO:tasks.workunit.client.0.smithi090.stderr:+ sed 's/^/ api_tier_pp: /' 2024-04-14T15:12:54.336 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-14T15:12:54.336 INFO:tasks.workunit.client.0.smithi090.stderr:+ unset rc 2024-04-14T15:12:54.336 INFO:tasks.workunit.client.0.smithi090.stderr:+ sed 's/^/ api_pool: /' 2024-04-14T15:12:54.336 INFO:tasks.workunit.client.0.smithi090.stderr:+ ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2024-04-14T15:12:54.336 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/locale 2024-04-14T15:12:54.336 INFO:tasks.workunit.client.0.smithi090.stderr:+ ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2024-04-14T15:12:54.336 INFO:tasks.workunit.client.0.smithi090.stderr:+ tee ceph_test_rados_api_pool.log 2024-04-14T15:12:54.336 INFO:tasks.workunit.client.0.smithi090.stderr:+ ff=api_stat_pp 2024-04-14T15:12:54.336 INFO:tasks.workunit.client.0.smithi090.stdout:test api_stat_pp on pid 93665 2024-04-14T15:12:54.339 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.340 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-14T15:12:54.340 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=93665 2024-04-14T15:12:54.340 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.340 INFO:tasks.workunit.client.0.smithi090.stderr:+ echo 'test api_stat_pp on pid 93665' 2024-04-14T15:12:54.340 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-14T15:12:54.340 INFO:tasks.workunit.client.0.smithi090.stderr:+ pids[$f]=93665 2024-04-14T15:12:54.340 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-14T15:12:54.340 INFO:tasks.workunit.client.0.smithi090.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-14T15:12:54.340 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' 1 -eq 1 ']' 2024-04-14T15:12:54.340 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-14T15:12:54.340 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-14T15:12:54.340 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-14T15:12:54.340 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-14T15:12:54.341 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.341 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-14T15:12:54.341 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.341 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-14T15:12:54.341 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-14T15:12:54.341 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.341 INFO:tasks.workunit.client.0.smithi090.stderr:++ printf %25s api_watch_notify 2024-04-14T15:12:54.341 INFO:tasks.workunit.client.0.smithi090.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-14T15:12:54.341 INFO:tasks.workunit.client.0.smithi090.stderr:++++ export LANG=en_US.UTF-8 2024-04-14T15:12:54.341 INFO:tasks.workunit.client.0.smithi090.stderr:++++ LANG=en_US.UTF-8 2024-04-14T15:12:54.341 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.341 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-14T15:12:54.341 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.341 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG=en_US.UTF-8 2024-04-14T15:12:54.341 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup config 2024-04-14T15:12:54.342 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.342 INFO:tasks.workunit.client.0.smithi090.stderr:+ r=' api_watch_notify' 2024-04-14T15:12:54.342 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.342 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' dumb = linux ']' 2024-04-14T15:12:54.342 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.342 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-14T15:12:54.342 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.342 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/less.sh 2024-04-14T15:12:54.342 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-14T15:12:54.342 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.342 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-14T15:12:54.342 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.342 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/which2.sh 2024-04-14T15:12:54.342 INFO:tasks.workunit.client.0.smithi090.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-14T15:12:54.342 INFO:tasks.workunit.client.0.smithi090.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2024-04-14T15:12:54.343 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-14T15:12:54.343 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-14T15:12:54.343 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-14T15:12:54.343 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.343 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-14T15:12:54.343 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.343 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-14T15:12:54.343 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-14T15:12:54.343 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.343 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-14T15:12:54.343 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.343 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/gawk.sh 2024-04-14T15:12:54.343 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.343 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-14T15:12:54.343 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.344 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/lang.sh 2024-04-14T15:12:54.344 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup 2024-04-14T15:12:54.344 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -f /etc/bashrc ']' 2024-04-14T15:12:54.344 INFO:tasks.workunit.client.0.smithi090.stderr:+ . /etc/bashrc 2024-04-14T15:12:54.344 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -z '' ']' 2024-04-14T15:12:54.344 INFO:tasks.workunit.client.0.smithi090.stderr:++ BASHRCSOURCED=Y 2024-04-14T15:12:54.344 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.344 INFO:tasks.workunit.client.0.smithi090.stderr:++ shopt -q login_shell 2024-04-14T15:12:54.344 INFO:tasks.workunit.client.0.smithi090.stderr:++ awk '{print $1}' 2024-04-14T15:12:54.344 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/locale 2024-04-14T15:12:54.344 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-14T15:12:54.344 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-14T15:12:54.344 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-14T15:12:54.344 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.344 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-14T15:12:54.344 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.345 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-14T15:12:54.345 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-14T15:12:54.345 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.345 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-14T15:12:54.345 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.345 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/gawk.sh 2024-04-14T15:12:54.345 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.345 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-14T15:12:54.345 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.345 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/lang.sh 2024-04-14T15:12:54.345 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup 2024-04-14T15:12:54.345 INFO:tasks.workunit.client.0.smithi090.stderr:+++++ readlink /proc/93555/exe 2024-04-14T15:12:54.345 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.345 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-14T15:12:54.345 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.346 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-14T15:12:54.346 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-14T15:12:54.346 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/locale 2024-04-14T15:12:54.346 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-14T15:12:54.346 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.346 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-14T15:12:54.346 INFO:tasks.workunit.client.0.smithi090.stderr:+++ umask 2024-04-14T15:12:54.346 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.346 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-14T15:12:54.346 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-14T15:12:54.346 INFO:tasks.workunit.client.0.smithi090.stderr:++ echo api_watch_notify 2024-04-14T15:12:54.346 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' 0022 -eq 0 ']' 2024-04-14T15:12:54.346 INFO:tasks.workunit.client.0.smithi090.stderr:++ SHELL=/bin/bash 2024-04-14T15:12:54.346 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.346 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-14T15:12:54.347 INFO:tasks.workunit.client.0.smithi090.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-14T15:12:54.347 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.347 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-14T15:12:54.347 INFO:tasks.workunit.client.0.smithi090.stderr:++++ export LANG=en_US.UTF-8 2024-04-14T15:12:54.347 INFO:tasks.workunit.client.0.smithi090.stderr:++++ LANG=en_US.UTF-8 2024-04-14T15:12:54.347 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.347 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-14T15:12:54.347 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.347 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG=en_US.UTF-8 2024-04-14T15:12:54.347 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup config 2024-04-14T15:12:54.347 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.347 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.347 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-14T15:12:54.347 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' dumb = linux ']' 2024-04-14T15:12:54.347 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.347 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.348 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-14T15:12:54.348 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-14T15:12:54.348 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.348 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-14T15:12:54.348 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.348 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/less.sh 2024-04-14T15:12:54.348 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.348 INFO:tasks.workunit.client.0.smithi090.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.348 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.348 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-14T15:12:54.348 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-14T15:12:54.348 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.348 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.348 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-14T15:12:54.348 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-14T15:12:54.349 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.349 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/which2.sh 2024-04-14T15:12:54.349 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.349 INFO:tasks.workunit.client.0.smithi090.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-14T15:12:54.349 INFO:tasks.workunit.client.0.smithi090.stderr:+ ff=api_watch_notify 2024-04-14T15:12:54.349 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=93703 2024-04-14T15:12:54.349 INFO:tasks.workunit.client.0.smithi090.stdout:test api_watch_notify on pid 93703 2024-04-14T15:12:54.354 INFO:tasks.workunit.client.0.smithi090.stderr:+ echo 'test api_watch_notify on pid 93703' 2024-04-14T15:12:54.354 INFO:tasks.workunit.client.0.smithi090.stderr:+ pids[$f]=93703 2024-04-14T15:12:54.354 INFO:tasks.workunit.client.0.smithi090.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-14T15:12:54.354 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' 1 -eq 1 ']' 2024-04-14T15:12:54.354 INFO:tasks.workunit.client.0.smithi090.stderr:++++ basename /usr/bin/bash 2024-04-14T15:12:54.354 INFO:tasks.workunit.client.0.smithi090.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2024-04-14T15:12:54.354 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-14T15:12:54.354 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_declare='declare -f' 2024-04-14T15:12:54.354 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_opt=-f 2024-04-14T15:12:54.354 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export which_declare 2024-04-14T15:12:54.354 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export -f which 2024-04-14T15:12:54.354 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset i 2024-04-14T15:12:54.355 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset -f pathmunge 2024-04-14T15:12:54.355 INFO:tasks.workunit.client.0.smithi090.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-14T15:12:54.355 INFO:tasks.workunit.client.0.smithi090.stderr:+ export PATH 2024-04-14T15:12:54.355 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-14T15:12:54.355 INFO:tasks.workunit.client.0.smithi090.stderr:+ unset rc 2024-04-14T15:12:54.355 INFO:tasks.workunit.client.0.smithi090.stderr:+++++ readlink /proc/93590/exe 2024-04-14T15:12:54.355 INFO:tasks.workunit.client.0.smithi090.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-14T15:12:54.355 INFO:tasks.workunit.client.0.smithi090.stderr:++++ export LANG=en_US.UTF-8 2024-04-14T15:12:54.355 INFO:tasks.workunit.client.0.smithi090.stderr:++++ LANG=en_US.UTF-8 2024-04-14T15:12:54.355 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.355 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-14T15:12:54.355 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.355 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG=en_US.UTF-8 2024-04-14T15:12:54.355 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup config 2024-04-14T15:12:54.355 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.356 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.356 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' dumb = linux ']' 2024-04-14T15:12:54.356 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.356 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-14T15:12:54.356 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.356 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/less.sh 2024-04-14T15:12:54.356 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-14T15:12:54.356 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.356 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-14T15:12:54.356 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.356 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/which2.sh 2024-04-14T15:12:54.356 INFO:tasks.workunit.client.0.smithi090.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-14T15:12:54.356 INFO:tasks.workunit.client.0.smithi090.stderr:++++ basename /usr/bin/bash 2024-04-14T15:12:54.356 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -f /etc/bashrc ']' 2024-04-14T15:12:54.356 INFO:tasks.workunit.client.0.smithi090.stderr:+ . /etc/bashrc 2024-04-14T15:12:54.357 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -z '' ']' 2024-04-14T15:12:54.357 INFO:tasks.workunit.client.0.smithi090.stderr:++ BASHRCSOURCED=Y 2024-04-14T15:12:54.357 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.357 INFO:tasks.workunit.client.0.smithi090.stderr:++ shopt -q login_shell 2024-04-14T15:12:54.357 INFO:tasks.workunit.client.0.smithi090.stderr:+++ umask 2024-04-14T15:12:54.357 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' 0022 -eq 0 ']' 2024-04-14T15:12:54.357 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_declare='declare -f' 2024-04-14T15:12:54.357 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_opt=-f 2024-04-14T15:12:54.357 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export which_declare 2024-04-14T15:12:54.357 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export -f which 2024-04-14T15:12:54.357 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset i 2024-04-14T15:12:54.357 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset -f pathmunge 2024-04-14T15:12:54.357 INFO:tasks.workunit.client.0.smithi090.stderr:++ SHELL=/bin/bash 2024-04-14T15:12:54.357 INFO:tasks.workunit.client.0.smithi090.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-14T15:12:54.357 INFO:tasks.workunit.client.0.smithi090.stderr:+ export PATH 2024-04-14T15:12:54.358 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.358 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-14T15:12:54.358 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-14T15:12:54.358 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.358 INFO:tasks.workunit.client.0.smithi090.stderr:+ unset rc 2024-04-14T15:12:54.358 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-14T15:12:54.358 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-14T15:12:54.358 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.358 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-14T15:12:54.358 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.358 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-14T15:12:54.358 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.358 INFO:tasks.workunit.client.0.smithi090.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.358 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.358 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-14T15:12:54.359 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.359 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-14T15:12:54.359 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.359 INFO:tasks.workunit.client.0.smithi090.stderr:+ ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2024-04-14T15:12:54.359 INFO:tasks.workunit.client.0.smithi090.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2024-04-14T15:12:54.359 INFO:tasks.workunit.client.0.smithi090.stderr:++ printf %25s api_watch_notify_pp 2024-04-14T15:12:54.359 INFO:tasks.workunit.client.0.smithi090.stderr:+ sed 's/^/ api_snapshots_pp: /' 2024-04-14T15:12:54.359 INFO:tasks.workunit.client.0.smithi090.stderr:+ r=' api_watch_notify_pp' 2024-04-14T15:12:54.359 INFO:tasks.workunit.client.0.smithi090.stderr:+ tee ceph_test_rados_api_snapshots.log 2024-04-14T15:12:54.359 INFO:tasks.workunit.client.0.smithi090.stderr:+ sed 's/^/ api_snapshots: /' 2024-04-14T15:12:54.359 INFO:tasks.workunit.client.0.smithi090.stderr:+ ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2024-04-14T15:12:54.359 INFO:tasks.workunit.client.0.smithi090.stderr:+++++ readlink /proc/93615/exe 2024-04-14T15:12:54.359 INFO:tasks.workunit.client.0.smithi090.stderr:++++ basename /usr/bin/bash 2024-04-14T15:12:54.359 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'grep=grep --color=auto' 2024-04-14T15:12:54.359 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-14T15:12:54.360 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-14T15:12:54.360 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.360 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-14T15:12:54.360 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.360 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorls.sh 2024-04-14T15:12:54.360 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' '!' -t 0 ']' 2024-04-14T15:12:54.360 INFO:tasks.workunit.client.0.smithi090.stderr:+++ return 2024-04-14T15:12:54.360 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.360 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-14T15:12:54.360 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.360 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-14T15:12:54.360 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export S_COLORS=auto 2024-04-14T15:12:54.360 INFO:tasks.workunit.client.0.smithi090.stderr:+++ S_COLORS=auto 2024-04-14T15:12:54.360 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.361 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-14T15:12:54.361 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.361 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-14T15:12:54.361 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.361 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_declare='declare -f' 2024-04-14T15:12:54.361 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_opt=-f 2024-04-14T15:12:54.361 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export which_declare 2024-04-14T15:12:54.361 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export -f which 2024-04-14T15:12:54.361 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset i 2024-04-14T15:12:54.361 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset -f pathmunge 2024-04-14T15:12:54.361 INFO:tasks.workunit.client.0.smithi090.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-14T15:12:54.361 INFO:tasks.workunit.client.0.smithi090.stderr:+ export PATH 2024-04-14T15:12:54.361 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-14T15:12:54.361 INFO:tasks.workunit.client.0.smithi090.stderr:+ unset rc 2024-04-14T15:12:54.361 INFO:tasks.workunit.client.0.smithi090.stderr:++ awk '{print $1}' 2024-04-14T15:12:54.362 INFO:tasks.workunit.client.0.smithi090.stderr:+ ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2024-04-14T15:12:54.362 INFO:tasks.workunit.client.0.smithi090.stderr:++ echo api_watch_notify_pp 2024-04-14T15:12:54.362 INFO:tasks.workunit.client.0.smithi090.stdout:test api_watch_notify_pp on pid 93737 2024-04-14T15:12:54.364 INFO:tasks.workunit.client.0.smithi090.stderr:+ ff=api_watch_notify_pp 2024-04-14T15:12:54.364 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=93737 2024-04-14T15:12:54.365 INFO:tasks.workunit.client.0.smithi090.stderr:+ echo 'test api_watch_notify_pp on pid 93737' 2024-04-14T15:12:54.365 INFO:tasks.workunit.client.0.smithi090.stderr:+ pids[$f]=93737 2024-04-14T15:12:54.365 INFO:tasks.workunit.client.0.smithi090.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-14T15:12:54.365 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' 1 -eq 1 ']' 2024-04-14T15:12:54.365 INFO:tasks.workunit.client.0.smithi090.stderr:++ printf %25s api_cmd 2024-04-14T15:12:54.365 INFO:tasks.workunit.client.0.smithi090.stderr:+ r=' api_cmd' 2024-04-14T15:12:54.365 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'grep=grep --color=auto' 2024-04-14T15:12:54.365 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-14T15:12:54.365 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-14T15:12:54.365 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.365 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-14T15:12:54.365 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.365 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorls.sh 2024-04-14T15:12:54.365 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' '!' -t 0 ']' 2024-04-14T15:12:54.366 INFO:tasks.workunit.client.0.smithi090.stderr:+++ return 2024-04-14T15:12:54.366 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.366 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-14T15:12:54.366 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.366 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-14T15:12:54.366 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export S_COLORS=auto 2024-04-14T15:12:54.366 INFO:tasks.workunit.client.0.smithi090.stderr:+++ S_COLORS=auto 2024-04-14T15:12:54.366 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.366 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-14T15:12:54.366 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.366 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-14T15:12:54.366 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.366 INFO:tasks.workunit.client.0.smithi090.stderr:++ awk '{print $1}' 2024-04-14T15:12:54.366 INFO:tasks.workunit.client.0.smithi090.stderr:++ echo api_cmd 2024-04-14T15:12:54.367 INFO:tasks.workunit.client.0.smithi090.stderr:+ tee ceph_test_rados_api_stat.log 2024-04-14T15:12:54.367 INFO:tasks.workunit.client.0.smithi090.stderr:+ sed 's/^/ api_stat: /' 2024-04-14T15:12:54.367 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-14T15:12:54.367 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-14T15:12:54.367 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-14T15:12:54.367 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.367 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-14T15:12:54.367 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.367 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-14T15:12:54.367 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-14T15:12:54.367 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.367 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-14T15:12:54.367 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-14T15:12:54.367 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-14T15:12:54.367 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.368 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-14T15:12:54.368 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.368 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-14T15:12:54.368 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-14T15:12:54.368 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.368 INFO:tasks.workunit.client.0.smithi090.stderr:+ ff=api_cmd 2024-04-14T15:12:54.368 INFO:tasks.workunit.client.0.smithi090.stdout:test api_cmd on pid 93753 2024-04-14T15:12:54.369 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=93753 2024-04-14T15:12:54.369 INFO:tasks.workunit.client.0.smithi090.stderr:+ echo 'test api_cmd on pid 93753' 2024-04-14T15:12:54.369 INFO:tasks.workunit.client.0.smithi090.stderr:+ pids[$f]=93753 2024-04-14T15:12:54.369 INFO:tasks.workunit.client.0.smithi090.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-14T15:12:54.369 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' 1 -eq 1 ']' 2024-04-14T15:12:54.370 INFO:tasks.workunit.client.0.smithi090.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2024-04-14T15:12:54.370 INFO:tasks.workunit.client.0.smithi090.stderr:++ printf %25s api_cmd_pp 2024-04-14T15:12:54.370 INFO:tasks.workunit.client.0.smithi090.stderr:+ r=' api_cmd_pp' 2024-04-14T15:12:54.370 INFO:tasks.workunit.client.0.smithi090.stderr:++ awk '{print $1}' 2024-04-14T15:12:54.370 INFO:tasks.workunit.client.0.smithi090.stderr:++ echo api_cmd_pp 2024-04-14T15:12:54.370 INFO:tasks.workunit.client.0.smithi090.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2024-04-14T15:12:54.370 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -f /etc/bashrc ']' 2024-04-14T15:12:54.370 INFO:tasks.workunit.client.0.smithi090.stderr:+ . /etc/bashrc 2024-04-14T15:12:54.370 INFO:tasks.workunit.client.0.smithi090.stdout:test api_cmd_pp on pid 93772 2024-04-14T15:12:54.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:54 smithi092 ceph-mon[33966]: pgmap v104: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T15:12:54.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:54 smithi092 ceph-mon[33966]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:12:54.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:54 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3274751583' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-14T15:12:54.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:54 smithi092 ceph-mon[33966]: osdmap e106: 8 total, 8 up, 8 in 2024-04-14T15:12:54.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:54 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:12:54.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:54 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:12:54.373 INFO:tasks.workunit.client.0.smithi090.stderr:+ ff=api_cmd_pp 2024-04-14T15:12:54.373 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=93772 2024-04-14T15:12:54.373 INFO:tasks.workunit.client.0.smithi090.stderr:+ echo 'test api_cmd_pp on pid 93772' 2024-04-14T15:12:54.373 INFO:tasks.workunit.client.0.smithi090.stderr:+ pids[$f]=93772 2024-04-14T15:12:54.373 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -z '' ']' 2024-04-14T15:12:54.373 INFO:tasks.workunit.client.0.smithi090.stderr:++ BASHRCSOURCED=Y 2024-04-14T15:12:54.374 INFO:tasks.workunit.client.0.smithi090.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-14T15:12:54.374 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' 1 -eq 1 ']' 2024-04-14T15:12:54.374 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.374 INFO:tasks.workunit.client.0.smithi090.stderr:++ shopt -q login_shell 2024-04-14T15:12:54.374 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -f /etc/bashrc ']' 2024-04-14T15:12:54.374 INFO:tasks.workunit.client.0.smithi090.stderr:+ . /etc/bashrc 2024-04-14T15:12:54.374 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -z '' ']' 2024-04-14T15:12:54.374 INFO:tasks.workunit.client.0.smithi090.stderr:++ BASHRCSOURCED=Y 2024-04-14T15:12:54.374 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.374 INFO:tasks.workunit.client.0.smithi090.stderr:++ shopt -q login_shell 2024-04-14T15:12:54.374 INFO:tasks.workunit.client.0.smithi090.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2024-04-14T15:12:54.374 INFO:tasks.workunit.client.0.smithi090.stderr:++ printf %25s api_service 2024-04-14T15:12:54.374 INFO:tasks.workunit.client.0.smithi090.stderr:+++ umask 2024-04-14T15:12:54.374 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' 0022 -eq 0 ']' 2024-04-14T15:12:54.374 INFO:tasks.workunit.client.0.smithi090.stderr:++ SHELL=/bin/bash 2024-04-14T15:12:54.375 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.375 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-14T15:12:54.375 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.375 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-14T15:12:54.375 INFO:tasks.workunit.client.0.smithi090.stderr:+ r=' api_service' 2024-04-14T15:12:54.375 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-14T15:12:54.375 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.375 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-14T15:12:54.375 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.375 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-14T15:12:54.375 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.375 INFO:tasks.workunit.client.0.smithi090.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.375 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.375 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-14T15:12:54.375 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.376 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-14T15:12:54.376 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.376 INFO:tasks.workunit.client.0.smithi090.stderr:++ awk '{print $1}' 2024-04-14T15:12:54.376 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-14T15:12:54.376 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-14T15:12:54.376 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-14T15:12:54.376 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.376 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-14T15:12:54.376 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.376 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-14T15:12:54.376 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-14T15:12:54.376 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.376 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-14T15:12:54.376 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.376 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/gawk.sh 2024-04-14T15:12:54.376 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.377 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-14T15:12:54.377 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.377 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/lang.sh 2024-04-14T15:12:54.377 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup 2024-04-14T15:12:54.377 INFO:tasks.workunit.client.0.smithi090.stderr:++ echo api_service 2024-04-14T15:12:54.377 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -f /etc/bashrc ']' 2024-04-14T15:12:54.377 INFO:tasks.workunit.client.0.smithi090.stderr:+ . /etc/bashrc 2024-04-14T15:12:54.377 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -z '' ']' 2024-04-14T15:12:54.377 INFO:tasks.workunit.client.0.smithi090.stderr:++ BASHRCSOURCED=Y 2024-04-14T15:12:54.377 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.377 INFO:tasks.workunit.client.0.smithi090.stderr:++ shopt -q login_shell 2024-04-14T15:12:54.377 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/locale 2024-04-14T15:12:54.378 INFO:tasks.workunit.client.0.smithi090.stdout:test api_service on pid 93788 2024-04-14T15:12:54.380 INFO:tasks.workunit.client.0.smithi090.stderr:+ ff=api_service 2024-04-14T15:12:54.380 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=93788 2024-04-14T15:12:54.380 INFO:tasks.workunit.client.0.smithi090.stderr:+ echo 'test api_service on pid 93788' 2024-04-14T15:12:54.380 INFO:tasks.workunit.client.0.smithi090.stderr:+ pids[$f]=93788 2024-04-14T15:12:54.380 INFO:tasks.workunit.client.0.smithi090.stderr:+++ umask 2024-04-14T15:12:54.380 INFO:tasks.workunit.client.0.smithi090.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-14T15:12:54.380 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' 1 -eq 1 ']' 2024-04-14T15:12:54.380 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'grep=grep --color=auto' 2024-04-14T15:12:54.380 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-14T15:12:54.380 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-14T15:12:54.380 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' 0022 -eq 0 ']' 2024-04-14T15:12:54.381 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.381 INFO:tasks.workunit.client.0.smithi090.stderr:++ SHELL=/bin/bash 2024-04-14T15:12:54.381 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-14T15:12:54.381 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.381 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorls.sh 2024-04-14T15:12:54.381 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.381 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' '!' -t 0 ']' 2024-04-14T15:12:54.381 INFO:tasks.workunit.client.0.smithi090.stderr:+++ return 2024-04-14T15:12:54.381 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.381 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-14T15:12:54.381 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.381 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-14T15:12:54.381 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export S_COLORS=auto 2024-04-14T15:12:54.381 INFO:tasks.workunit.client.0.smithi090.stderr:+++ S_COLORS=auto 2024-04-14T15:12:54.381 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-14T15:12:54.382 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.382 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-14T15:12:54.382 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.382 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-14T15:12:54.382 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.382 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-14T15:12:54.382 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-14T15:12:54.382 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.382 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-14T15:12:54.382 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-14T15:12:54.382 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-14T15:12:54.382 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.382 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-14T15:12:54.382 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.383 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-14T15:12:54.383 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.383 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-14T15:12:54.383 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.383 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-14T15:12:54.383 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.383 INFO:tasks.workunit.client.0.smithi090.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.383 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.383 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-14T15:12:54.383 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-14T15:12:54.383 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.383 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-14T15:12:54.383 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.383 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-14T15:12:54.383 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.384 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.384 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/gawk.sh 2024-04-14T15:12:54.384 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.384 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-14T15:12:54.384 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.384 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-14T15:12:54.384 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.384 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/lang.sh 2024-04-14T15:12:54.384 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.384 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-14T15:12:54.384 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup 2024-04-14T15:12:54.384 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-14T15:12:54.384 INFO:tasks.workunit.client.0.smithi090.stderr:++ printf %25s api_service_pp 2024-04-14T15:12:54.384 INFO:tasks.workunit.client.0.smithi090.stderr:+ r=' api_service_pp' 2024-04-14T15:12:54.384 INFO:tasks.workunit.client.0.smithi090.stderr:++ echo api_service_pp 2024-04-14T15:12:54.385 INFO:tasks.workunit.client.0.smithi090.stderr:++ awk '{print $1}' 2024-04-14T15:12:54.385 INFO:tasks.workunit.client.0.smithi090.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2024-04-14T15:12:54.385 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-14T15:12:54.385 INFO:tasks.workunit.client.0.smithi090.stdout:test api_service_pp on pid 93813 2024-04-14T15:12:54.386 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:12:54.389 INFO:tasks.workunit.client.0.smithi090.stderr:+ ff=api_service_pp 2024-04-14T15:12:54.389 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/locale 2024-04-14T15:12:54.389 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=93813 2024-04-14T15:12:54.389 INFO:tasks.workunit.client.0.smithi090.stderr:+ echo 'test api_service_pp on pid 93813' 2024-04-14T15:12:54.389 INFO:tasks.workunit.client.0.smithi090.stderr:+ pids[$f]=93813 2024-04-14T15:12:54.390 INFO:tasks.workunit.client.0.smithi090.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-14T15:12:54.390 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' 1 -eq 1 ']' 2024-04-14T15:12:54.390 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'grep=grep --color=auto' 2024-04-14T15:12:54.390 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-14T15:12:54.390 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-14T15:12:54.390 INFO:tasks.workunit.client.0.smithi090.stderr:+++ umask 2024-04-14T15:12:54.390 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.390 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-14T15:12:54.390 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.390 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorls.sh 2024-04-14T15:12:54.390 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' '!' -t 0 ']' 2024-04-14T15:12:54.390 INFO:tasks.workunit.client.0.smithi090.stderr:+++ return 2024-04-14T15:12:54.390 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.390 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-14T15:12:54.391 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.391 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-14T15:12:54.391 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export S_COLORS=auto 2024-04-14T15:12:54.391 INFO:tasks.workunit.client.0.smithi090.stderr:+++ S_COLORS=auto 2024-04-14T15:12:54.391 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.391 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-14T15:12:54.391 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.391 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-14T15:12:54.391 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' 0022 -eq 0 ']' 2024-04-14T15:12:54.391 INFO:tasks.workunit.client.0.smithi090.stderr:++ SHELL=/bin/bash 2024-04-14T15:12:54.391 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.391 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.391 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-14T15:12:54.391 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.391 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-14T15:12:54.392 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-14T15:12:54.392 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.392 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-14T15:12:54.392 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.392 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-14T15:12:54.392 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.392 INFO:tasks.workunit.client.0.smithi090.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.392 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.392 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-14T15:12:54.392 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.392 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-14T15:12:54.392 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.392 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.392 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-14T15:12:54.392 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.393 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-14T15:12:54.393 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-14T15:12:54.393 INFO:tasks.workunit.client.0.smithi090.stderr:++ printf %25s api_c_write_operations 2024-04-14T15:12:54.393 INFO:tasks.workunit.client.0.smithi090.stderr:+ r=' api_c_write_operations' 2024-04-14T15:12:54.393 INFO:tasks.workunit.client.0.smithi090.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2024-04-14T15:12:54.393 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -f /etc/bashrc ']' 2024-04-14T15:12:54.393 INFO:tasks.workunit.client.0.smithi090.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-14T15:12:54.393 INFO:tasks.workunit.client.0.smithi090.stderr:++++ export LANG=en_US.UTF-8 2024-04-14T15:12:54.393 INFO:tasks.workunit.client.0.smithi090.stderr:++++ LANG=en_US.UTF-8 2024-04-14T15:12:54.393 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.393 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-14T15:12:54.393 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.393 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG=en_US.UTF-8 2024-04-14T15:12:54.393 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-14T15:12:54.393 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup config 2024-04-14T15:12:54.394 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-14T15:12:54.394 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.394 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-14T15:12:54.394 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.394 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-14T15:12:54.394 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.394 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.394 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' dumb = linux ']' 2024-04-14T15:12:54.394 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.394 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-14T15:12:54.394 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.394 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/less.sh 2024-04-14T15:12:54.394 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-14T15:12:54.394 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.394 INFO:tasks.workunit.client.0.smithi090.stderr:++ awk '{print $1}' 2024-04-14T15:12:54.395 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-14T15:12:54.395 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.395 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/which2.sh 2024-04-14T15:12:54.395 INFO:tasks.workunit.client.0.smithi090.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-14T15:12:54.395 INFO:tasks.workunit.client.0.smithi090.stderr:+ . /etc/bashrc 2024-04-14T15:12:54.395 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-14T15:12:54.395 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-14T15:12:54.395 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.395 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -z '' ']' 2024-04-14T15:12:54.395 INFO:tasks.workunit.client.0.smithi090.stderr:++ BASHRCSOURCED=Y 2024-04-14T15:12:54.395 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -f /etc/bashrc ']' 2024-04-14T15:12:54.395 INFO:tasks.workunit.client.0.smithi090.stderr:+ . /etc/bashrc 2024-04-14T15:12:54.395 INFO:tasks.workunit.client.0.smithi090.stderr:++ echo api_c_write_operations 2024-04-14T15:12:54.395 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -z '' ']' 2024-04-14T15:12:54.395 INFO:tasks.workunit.client.0.smithi090.stderr:++ BASHRCSOURCED=Y 2024-04-14T15:12:54.396 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.396 INFO:tasks.workunit.client.0.smithi090.stderr:++ shopt -q login_shell 2024-04-14T15:12:54.396 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'grep=grep --color=auto' 2024-04-14T15:12:54.396 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-14T15:12:54.396 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-14T15:12:54.396 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.396 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-14T15:12:54.396 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.396 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorls.sh 2024-04-14T15:12:54.396 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' '!' -t 0 ']' 2024-04-14T15:12:54.396 INFO:tasks.workunit.client.0.smithi090.stderr:+++ return 2024-04-14T15:12:54.396 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.396 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-14T15:12:54.396 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.396 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-14T15:12:54.397 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.397 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export S_COLORS=auto 2024-04-14T15:12:54.397 INFO:tasks.workunit.client.0.smithi090.stderr:+ ff=api_c_write_operations 2024-04-14T15:12:54.397 INFO:tasks.workunit.client.0.smithi090.stderr:+++ S_COLORS=auto 2024-04-14T15:12:54.397 INFO:tasks.workunit.client.0.smithi090.stderr:++ shopt -q login_shell 2024-04-14T15:12:54.397 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.397 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-14T15:12:54.397 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.397 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-14T15:12:54.397 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.397 INFO:tasks.workunit.client.0.smithi090.stdout:test api_c_write_operations on pid 93841 2024-04-14T15:12:54.406 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=93841 2024-04-14T15:12:54.406 INFO:tasks.workunit.client.0.smithi090.stderr:+ echo 'test api_c_write_operations on pid 93841' 2024-04-14T15:12:54.406 INFO:tasks.workunit.client.0.smithi090.stderr:+ pids[$f]=93841 2024-04-14T15:12:54.406 INFO:tasks.workunit.client.0.smithi090.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-14T15:12:54.406 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' 1 -eq 1 ']' 2024-04-14T15:12:54.406 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-14T15:12:54.407 INFO:tasks.workunit.client.0.smithi090.stderr:+++++ readlink /proc/93665/exe 2024-04-14T15:12:54.407 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-14T15:12:54.407 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-14T15:12:54.407 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-14T15:12:54.407 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.407 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-14T15:12:54.407 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.407 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-14T15:12:54.407 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-14T15:12:54.407 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.407 INFO:tasks.workunit.client.0.smithi090.stderr:+++ umask 2024-04-14T15:12:54.407 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' 0022 -eq 0 ']' 2024-04-14T15:12:54.407 INFO:tasks.workunit.client.0.smithi090.stderr:++ SHELL=/bin/bash 2024-04-14T15:12:54.408 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.408 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-14T15:12:54.408 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.408 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-14T15:12:54.408 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-14T15:12:54.408 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.408 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-14T15:12:54.408 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.408 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-14T15:12:54.408 INFO:tasks.workunit.client.0.smithi090.stderr:++++ basename /usr/bin/bash 2024-04-14T15:12:54.408 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.408 INFO:tasks.workunit.client.0.smithi090.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.408 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.409 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-14T15:12:54.409 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.409 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-14T15:12:54.409 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.409 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_declare='declare -f' 2024-04-14T15:12:54.409 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_opt=-f 2024-04-14T15:12:54.409 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export which_declare 2024-04-14T15:12:54.409 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export -f which 2024-04-14T15:12:54.409 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset i 2024-04-14T15:12:54.409 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset -f pathmunge 2024-04-14T15:12:54.409 INFO:tasks.workunit.client.0.smithi090.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-14T15:12:54.409 INFO:tasks.workunit.client.0.smithi090.stderr:+ export PATH 2024-04-14T15:12:54.409 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-14T15:12:54.409 INFO:tasks.workunit.client.0.smithi090.stderr:+ unset rc 2024-04-14T15:12:54.410 INFO:tasks.workunit.client.0.smithi090.stderr:+++ umask 2024-04-14T15:12:54.410 INFO:tasks.workunit.client.0.smithi090.stderr:++ printf %25s api_c_read_operations 2024-04-14T15:12:54.410 INFO:tasks.workunit.client.0.smithi090.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-14T15:12:54.410 INFO:tasks.workunit.client.0.smithi090.stderr:++++ export LANG=en_US.UTF-8 2024-04-14T15:12:54.410 INFO:tasks.workunit.client.0.smithi090.stderr:++++ LANG=en_US.UTF-8 2024-04-14T15:12:54.410 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.410 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-14T15:12:54.410 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.410 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG=en_US.UTF-8 2024-04-14T15:12:54.410 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup config 2024-04-14T15:12:54.410 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.410 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.410 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' dumb = linux ']' 2024-04-14T15:12:54.411 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.411 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-14T15:12:54.411 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.411 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/less.sh 2024-04-14T15:12:54.411 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-14T15:12:54.411 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.411 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-14T15:12:54.411 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.411 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/which2.sh 2024-04-14T15:12:54.411 INFO:tasks.workunit.client.0.smithi090.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-14T15:12:54.411 INFO:tasks.workunit.client.0.smithi090.stderr:+ r=' api_c_read_operations' 2024-04-14T15:12:54.411 INFO:tasks.workunit.client.0.smithi090.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2024-04-14T15:12:54.411 INFO:tasks.workunit.client.0.smithi090.stderr:+ ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2024-04-14T15:12:54.412 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' 0022 -eq 0 ']' 2024-04-14T15:12:54.412 INFO:tasks.workunit.client.0.smithi090.stderr:++ SHELL=/bin/bash 2024-04-14T15:12:54.412 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.412 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-14T15:12:54.412 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.412 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-14T15:12:54.412 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-14T15:12:54.412 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.412 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-14T15:12:54.412 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.412 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-14T15:12:54.412 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.412 INFO:tasks.workunit.client.0.smithi090.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.412 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -f /etc/bashrc ']' 2024-04-14T15:12:54.412 INFO:tasks.workunit.client.0.smithi090.stderr:+ . /etc/bashrc 2024-04-14T15:12:54.413 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-14T15:12:54.413 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.413 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-14T15:12:54.413 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.413 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-14T15:12:54.413 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-14T15:12:54.413 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.413 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-14T15:12:54.413 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.413 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-14T15:12:54.413 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -z '' ']' 2024-04-14T15:12:54.413 INFO:tasks.workunit.client.0.smithi090.stderr:++ BASHRCSOURCED=Y 2024-04-14T15:12:54.413 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.413 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.414 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-14T15:12:54.414 INFO:tasks.workunit.client.0.smithi090.stderr:++ shopt -q login_shell 2024-04-14T15:12:54.414 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-14T15:12:54.414 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.414 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-14T15:12:54.414 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.414 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/gawk.sh 2024-04-14T15:12:54.414 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.414 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-14T15:12:54.414 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.414 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/lang.sh 2024-04-14T15:12:54.415 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup 2024-04-14T15:12:54.415 INFO:tasks.workunit.client.0.smithi090.stderr:+++ umask 2024-04-14T15:12:54.415 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' 0022 -eq 0 ']' 2024-04-14T15:12:54.415 INFO:tasks.workunit.client.0.smithi090.stderr:++ SHELL=/bin/bash 2024-04-14T15:12:54.415 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.415 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-14T15:12:54.415 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.415 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-14T15:12:54.415 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-14T15:12:54.415 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-14T15:12:54.415 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-14T15:12:54.415 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-14T15:12:54.415 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.415 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.416 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-14T15:12:54.416 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-14T15:12:54.416 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.416 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.416 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-14T15:12:54.416 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-14T15:12:54.416 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.416 INFO:tasks.workunit.client.0.smithi090.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.416 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.416 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-14T15:12:54.416 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-14T15:12:54.416 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.416 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.416 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-14T15:12:54.416 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-14T15:12:54.417 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.417 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.417 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/gawk.sh 2024-04-14T15:12:54.417 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.417 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-14T15:12:54.417 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.417 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/lang.sh 2024-04-14T15:12:54.417 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup 2024-04-14T15:12:54.417 INFO:tasks.workunit.client.0.smithi090.stderr:++ awk '{print $1}' 2024-04-14T15:12:54.417 INFO:tasks.workunit.client.0.smithi090.stderr:+ sed 's/^/ api_stat_pp: /' 2024-04-14T15:12:54.417 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-14T15:12:54.417 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-14T15:12:54.417 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-14T15:12:54.417 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.418 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-14T15:12:54.418 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.418 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-14T15:12:54.418 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-14T15:12:54.418 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.418 INFO:tasks.workunit.client.0.smithi090.stderr:+ tee ceph_test_rados_api_stat_pp.log 2024-04-14T15:12:54.418 INFO:tasks.workunit.client.0.smithi090.stderr:++ echo api_c_read_operations 2024-04-14T15:12:54.418 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'grep=grep --color=auto' 2024-04-14T15:12:54.418 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-14T15:12:54.418 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-14T15:12:54.418 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.418 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-14T15:12:54.418 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.418 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorls.sh 2024-04-14T15:12:54.419 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' '!' -t 0 ']' 2024-04-14T15:12:54.419 INFO:tasks.workunit.client.0.smithi090.stderr:+++ return 2024-04-14T15:12:54.419 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.419 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-14T15:12:54.419 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.419 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-14T15:12:54.419 INFO:tasks.workunit.client.0.smithi090.stderr:+++++ readlink /proc/93703/exe 2024-04-14T15:12:54.419 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export S_COLORS=auto 2024-04-14T15:12:54.419 INFO:tasks.workunit.client.0.smithi090.stderr:+++ S_COLORS=auto 2024-04-14T15:12:54.419 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.419 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-14T15:12:54.419 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.419 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-14T15:12:54.419 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.420 INFO:tasks.workunit.client.0.smithi090.stdout:test api_c_read_operations on pid 93910 2024-04-14T15:12:54.424 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/locale 2024-04-14T15:12:54.425 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/locale 2024-04-14T15:12:54.425 INFO:tasks.workunit.client.0.smithi090.stderr:+ ff=api_c_read_operations 2024-04-14T15:12:54.425 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=93910 2024-04-14T15:12:54.425 INFO:tasks.workunit.client.0.smithi090.stderr:+ echo 'test api_c_read_operations on pid 93910' 2024-04-14T15:12:54.425 INFO:tasks.workunit.client.0.smithi090.stderr:+ pids[$f]=93910 2024-04-14T15:12:54.425 INFO:tasks.workunit.client.0.smithi090.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-14T15:12:54.425 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' 1 -eq 1 ']' 2024-04-14T15:12:54.425 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'grep=grep --color=auto' 2024-04-14T15:12:54.425 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-14T15:12:54.425 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-14T15:12:54.425 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.425 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-14T15:12:54.425 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.426 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorls.sh 2024-04-14T15:12:54.426 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' '!' -t 0 ']' 2024-04-14T15:12:54.426 INFO:tasks.workunit.client.0.smithi090.stderr:+++ return 2024-04-14T15:12:54.426 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.426 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-14T15:12:54.426 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.426 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-14T15:12:54.426 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export S_COLORS=auto 2024-04-14T15:12:54.426 INFO:tasks.workunit.client.0.smithi090.stderr:+++ S_COLORS=auto 2024-04-14T15:12:54.426 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.426 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-14T15:12:54.426 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.427 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-14T15:12:54.427 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.427 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'grep=grep --color=auto' 2024-04-14T15:12:54.427 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-14T15:12:54.427 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-14T15:12:54.427 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.427 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-14T15:12:54.427 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.427 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorls.sh 2024-04-14T15:12:54.427 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' '!' -t 0 ']' 2024-04-14T15:12:54.427 INFO:tasks.workunit.client.0.smithi090.stderr:+++ return 2024-04-14T15:12:54.427 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.427 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-14T15:12:54.427 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.428 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-14T15:12:54.428 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export S_COLORS=auto 2024-04-14T15:12:54.428 INFO:tasks.workunit.client.0.smithi090.stderr:+++ S_COLORS=auto 2024-04-14T15:12:54.428 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.428 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-14T15:12:54.428 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.428 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-14T15:12:54.428 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.428 INFO:tasks.workunit.client.0.smithi090.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2024-04-14T15:12:54.428 INFO:tasks.workunit.client.0.smithi090.stderr:++ printf %25s api_cls_remote_reads 2024-04-14T15:12:54.428 INFO:tasks.workunit.client.0.smithi090.stderr:+ r=' api_cls_remote_reads' 2024-04-14T15:12:54.428 INFO:tasks.workunit.client.0.smithi090.stderr:++ awk '{print $1}' 2024-04-14T15:12:54.428 INFO:tasks.workunit.client.0.smithi090.stderr:++ echo api_cls_remote_reads 2024-04-14T15:12:54.429 INFO:tasks.workunit.client.0.smithi090.stderr:++++ basename /usr/bin/bash 2024-04-14T15:12:54.429 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-14T15:12:54.429 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-14T15:12:54.429 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-14T15:12:54.429 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.429 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-14T15:12:54.429 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.429 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-14T15:12:54.429 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-14T15:12:54.429 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.429 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.429 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-14T15:12:54.429 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-14T15:12:54.429 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.430 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/gawk.sh 2024-04-14T15:12:54.430 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.430 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-14T15:12:54.430 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-14T15:12:54.430 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.430 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-14T15:12:54.430 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.430 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/lang.sh 2024-04-14T15:12:54.430 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup 2024-04-14T15:12:54.430 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.430 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-14T15:12:54.430 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_declare='declare -f' 2024-04-14T15:12:54.430 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_opt=-f 2024-04-14T15:12:54.431 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.431 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-14T15:12:54.431 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-14T15:12:54.431 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export which_declare 2024-04-14T15:12:54.431 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export -f which 2024-04-14T15:12:54.431 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset i 2024-04-14T15:12:54.431 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset -f pathmunge 2024-04-14T15:12:54.431 INFO:tasks.workunit.client.0.smithi090.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-14T15:12:54.431 INFO:tasks.workunit.client.0.smithi090.stderr:+ export PATH 2024-04-14T15:12:54.431 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-14T15:12:54.431 INFO:tasks.workunit.client.0.smithi090.stderr:+ unset rc 2024-04-14T15:12:54.431 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -f /etc/bashrc ']' 2024-04-14T15:12:54.431 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/locale 2024-04-14T15:12:54.431 INFO:tasks.workunit.client.0.smithi090.stderr:+ . /etc/bashrc 2024-04-14T15:12:54.432 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -z '' ']' 2024-04-14T15:12:54.432 INFO:tasks.workunit.client.0.smithi090.stderr:++ BASHRCSOURCED=Y 2024-04-14T15:12:54.432 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.432 INFO:tasks.workunit.client.0.smithi090.stderr:++ shopt -q login_shell 2024-04-14T15:12:54.432 INFO:tasks.workunit.client.0.smithi090.stderr:+ ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2024-04-14T15:12:54.432 INFO:tasks.workunit.client.0.smithi090.stderr:+ tee ceph_test_rados_api_watch_notify.log 2024-04-14T15:12:54.432 INFO:tasks.workunit.client.0.smithi090.stdout:test api_cls_remote_reads on pid 93939 2024-04-14T15:12:54.435 INFO:tasks.workunit.client.0.smithi090.stderr:+ ff=api_cls_remote_reads 2024-04-14T15:12:54.435 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=93939 2024-04-14T15:12:54.435 INFO:tasks.workunit.client.0.smithi090.stderr:+ echo 'test api_cls_remote_reads on pid 93939' 2024-04-14T15:12:54.435 INFO:tasks.workunit.client.0.smithi090.stderr:+ pids[$f]=93939 2024-04-14T15:12:54.435 INFO:tasks.workunit.client.0.smithi090.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-14T15:12:54.435 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' 1 -eq 1 ']' 2024-04-14T15:12:54.435 INFO:tasks.workunit.client.0.smithi090.stderr:+ sed 's/^/ api_watch_notify: /' 2024-04-14T15:12:54.435 INFO:tasks.workunit.client.0.smithi090.stderr:++ printf %25s list_parallel 2024-04-14T15:12:54.436 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.436 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-14T15:12:54.436 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.436 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-14T15:12:54.436 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-14T15:12:54.436 INFO:tasks.workunit.client.0.smithi090.stderr:+ r=' list_parallel' 2024-04-14T15:12:54.436 INFO:tasks.workunit.client.0.smithi090.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cls_remote_reads.xml 2>&1 | tee ceph_test_rados_api_cls_remote_reads.log | sed "s/^/ api_cls_remote_reads: /"' 2024-04-14T15:12:54.436 INFO:tasks.workunit.client.0.smithi090.stderr:++ awk '{print $1}' 2024-04-14T15:12:54.436 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-14T15:12:54.436 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -f /etc/bashrc ']' 2024-04-14T15:12:54.436 INFO:tasks.workunit.client.0.smithi090.stderr:+ . /etc/bashrc 2024-04-14T15:12:54.436 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -z '' ']' 2024-04-14T15:12:54.436 INFO:tasks.workunit.client.0.smithi090.stderr:++ BASHRCSOURCED=Y 2024-04-14T15:12:54.437 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.437 INFO:tasks.workunit.client.0.smithi090.stderr:++ shopt -q login_shell 2024-04-14T15:12:54.437 INFO:tasks.workunit.client.0.smithi090.stderr:++ echo list_parallel 2024-04-14T15:12:54.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:54.437 INFO:tasks.workunit.client.0.smithi090.stdout:test list_parallel on pid 93960 2024-04-14T15:12:54.438 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:54.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:12:54.443 INFO:tasks.workunit.client.0.smithi090.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-14T15:12:54.443 INFO:tasks.workunit.client.0.smithi090.stderr:++++ export LANG=en_US.UTF-8 2024-04-14T15:12:54.443 INFO:tasks.workunit.client.0.smithi090.stderr:++++ LANG=en_US.UTF-8 2024-04-14T15:12:54.443 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.443 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-14T15:12:54.443 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.443 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG=en_US.UTF-8 2024-04-14T15:12:54.443 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup config 2024-04-14T15:12:54.443 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.443 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.443 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' dumb = linux ']' 2024-04-14T15:12:54.443 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-14T15:12:54.444 INFO:tasks.workunit.client.0.smithi090.stderr:+ ff=list_parallel 2024-04-14T15:12:54.444 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.444 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-14T15:12:54.444 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.444 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/less.sh 2024-04-14T15:12:54.444 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-14T15:12:54.444 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.444 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-14T15:12:54.444 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.444 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/which2.sh 2024-04-14T15:12:54.444 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=93960 2024-04-14T15:12:54.444 INFO:tasks.workunit.client.0.smithi090.stderr:+ echo 'test list_parallel on pid 93960' 2024-04-14T15:12:54.444 INFO:tasks.workunit.client.0.smithi090.stderr:+ pids[$f]=93960 2024-04-14T15:12:54.444 INFO:tasks.workunit.client.0.smithi090.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-14T15:12:54.445 INFO:tasks.workunit.client.0.smithi090.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-14T15:12:54.445 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' 1 -eq 1 ']' 2024-04-14T15:12:54.445 INFO:tasks.workunit.client.0.smithi090.stderr:+++ umask 2024-04-14T15:12:54.445 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' 0022 -eq 0 ']' 2024-04-14T15:12:54.445 INFO:tasks.workunit.client.0.smithi090.stderr:++ SHELL=/bin/bash 2024-04-14T15:12:54.445 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.445 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-14T15:12:54.445 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.445 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-14T15:12:54.445 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-14T15:12:54.445 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.445 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-14T15:12:54.445 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.445 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-14T15:12:54.445 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.446 INFO:tasks.workunit.client.0.smithi090.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.446 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.446 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-14T15:12:54.446 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.446 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-14T15:12:54.446 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.446 INFO:tasks.workunit.client.0.smithi090.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-14T15:12:54.446 INFO:tasks.workunit.client.0.smithi090.stderr:++++ export LANG=en_US.UTF-8 2024-04-14T15:12:54.446 INFO:tasks.workunit.client.0.smithi090.stderr:++++ LANG=en_US.UTF-8 2024-04-14T15:12:54.446 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.446 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-14T15:12:54.446 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.446 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG=en_US.UTF-8 2024-04-14T15:12:54.446 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup config 2024-04-14T15:12:54.447 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.447 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.447 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' dumb = linux ']' 2024-04-14T15:12:54.447 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.447 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-14T15:12:54.447 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-14T15:12:54.447 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-14T15:12:54.447 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-14T15:12:54.447 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.447 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-14T15:12:54.447 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.447 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-14T15:12:54.447 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-14T15:12:54.447 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.447 INFO:tasks.workunit.client.0.smithi090.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2024-04-14T15:12:54.448 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -f /etc/bashrc ']' 2024-04-14T15:12:54.448 INFO:tasks.workunit.client.0.smithi090.stderr:+ . /etc/bashrc 2024-04-14T15:12:54.448 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -z '' ']' 2024-04-14T15:12:54.448 INFO:tasks.workunit.client.0.smithi090.stderr:++ BASHRCSOURCED=Y 2024-04-14T15:12:54.448 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.448 INFO:tasks.workunit.client.0.smithi090.stderr:++ shopt -q login_shell 2024-04-14T15:12:54.448 INFO:tasks.workunit.client.0.smithi090.stderr:+++ umask 2024-04-14T15:12:54.448 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' 0022 -eq 0 ']' 2024-04-14T15:12:54.448 INFO:tasks.workunit.client.0.smithi090.stderr:++ SHELL=/bin/bash 2024-04-14T15:12:54.448 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.448 INFO:tasks.workunit.client.0.smithi090.stderr:+++ umask 2024-04-14T15:12:54.448 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.448 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-14T15:12:54.448 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/less.sh 2024-04-14T15:12:54.449 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.449 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-14T15:12:54.449 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-14T15:12:54.449 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.449 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' 0022 -eq 0 ']' 2024-04-14T15:12:54.449 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-14T15:12:54.449 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.449 INFO:tasks.workunit.client.0.smithi090.stderr:++ SHELL=/bin/bash 2024-04-14T15:12:54.449 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-14T15:12:54.449 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.449 INFO:tasks.workunit.client.0.smithi090.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.449 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.449 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-14T15:12:54.449 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.449 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-14T15:12:54.450 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.450 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-14T15:12:54.450 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.450 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.450 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-14T15:12:54.450 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-14T15:12:54.450 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.450 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-14T15:12:54.450 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.450 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-14T15:12:54.450 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.450 INFO:tasks.workunit.client.0.smithi090.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.450 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.451 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-14T15:12:54.451 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.451 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-14T15:12:54.451 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.451 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-14T15:12:54.451 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-14T15:12:54.451 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.451 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-14T15:12:54.451 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.451 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/which2.sh 2024-04-14T15:12:54.451 INFO:tasks.workunit.client.0.smithi090.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-14T15:12:54.451 INFO:tasks.workunit.client.0.smithi090.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-14T15:12:54.451 INFO:tasks.workunit.client.0.smithi090.stderr:++++ export LANG=en_US.UTF-8 2024-04-14T15:12:54.452 INFO:tasks.workunit.client.0.smithi090.stderr:++++ LANG=en_US.UTF-8 2024-04-14T15:12:54.452 INFO:tasks.workunit.client.0.smithi090.stderr:++ printf %25s open_pools_parallel 2024-04-14T15:12:54.452 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.452 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-14T15:12:54.452 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.452 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG=en_US.UTF-8 2024-04-14T15:12:54.452 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup config 2024-04-14T15:12:54.452 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.452 INFO:tasks.workunit.client.0.smithi090.stderr:+ r=' open_pools_parallel' 2024-04-14T15:12:54.452 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.452 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' dumb = linux ']' 2024-04-14T15:12:54.452 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.452 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-14T15:12:54.452 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.453 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/less.sh 2024-04-14T15:12:54.453 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-14T15:12:54.453 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-14T15:12:54.453 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-14T15:12:54.453 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.453 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-14T15:12:54.453 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-14T15:12:54.453 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.453 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/which2.sh 2024-04-14T15:12:54.453 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.453 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-14T15:12:54.453 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.453 INFO:tasks.workunit.client.0.smithi090.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-14T15:12:54.453 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-14T15:12:54.453 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-14T15:12:54.454 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.454 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'grep=grep --color=auto' 2024-04-14T15:12:54.454 INFO:tasks.workunit.client.0.smithi090.stderr:+++++ readlink /proc/93772/exe 2024-04-14T15:12:54.454 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-14T15:12:54.454 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-14T15:12:54.454 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.454 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-14T15:12:54.454 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.454 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorls.sh 2024-04-14T15:12:54.454 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' '!' -t 0 ']' 2024-04-14T15:12:54.454 INFO:tasks.workunit.client.0.smithi090.stderr:+++ return 2024-04-14T15:12:54.454 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.454 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-14T15:12:54.454 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.455 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-14T15:12:54.455 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export S_COLORS=auto 2024-04-14T15:12:54.455 INFO:tasks.workunit.client.0.smithi090.stderr:+++ S_COLORS=auto 2024-04-14T15:12:54.455 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.455 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-14T15:12:54.455 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.455 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-14T15:12:54.455 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.455 INFO:tasks.workunit.client.0.smithi090.stderr:++++ basename /usr/bin/bash 2024-04-14T15:12:54.455 INFO:tasks.workunit.client.0.smithi090.stderr:+++++ readlink /proc/93737/exe 2024-04-14T15:12:54.455 INFO:tasks.workunit.client.0.smithi090.stderr:++ awk '{print $1}' 2024-04-14T15:12:54.455 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_declare='declare -f' 2024-04-14T15:12:54.455 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_opt=-f 2024-04-14T15:12:54.455 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export which_declare 2024-04-14T15:12:54.455 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export -f which 2024-04-14T15:12:54.456 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset i 2024-04-14T15:12:54.456 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset -f pathmunge 2024-04-14T15:12:54.456 INFO:tasks.workunit.client.0.smithi090.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-14T15:12:54.456 INFO:tasks.workunit.client.0.smithi090.stderr:+ export PATH 2024-04-14T15:12:54.456 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-14T15:12:54.456 INFO:tasks.workunit.client.0.smithi090.stderr:+ unset rc 2024-04-14T15:12:54.456 INFO:tasks.workunit.client.0.smithi090.stderr:+ ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2024-04-14T15:12:54.456 INFO:tasks.workunit.client.0.smithi090.stderr:++++ basename /usr/bin/bash 2024-04-14T15:12:54.456 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-14T15:12:54.456 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-14T15:12:54.456 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-14T15:12:54.456 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.456 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-14T15:12:54.456 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.457 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-14T15:12:54.457 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-14T15:12:54.457 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.457 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-14T15:12:54.457 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.457 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/gawk.sh 2024-04-14T15:12:54.457 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'grep=grep --color=auto' 2024-04-14T15:12:54.457 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-14T15:12:54.457 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-14T15:12:54.457 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.457 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-14T15:12:54.457 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.457 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorls.sh 2024-04-14T15:12:54.457 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' '!' -t 0 ']' 2024-04-14T15:12:54.457 INFO:tasks.workunit.client.0.smithi090.stderr:+++ return 2024-04-14T15:12:54.458 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.458 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-14T15:12:54.458 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_declare='declare -f' 2024-04-14T15:12:54.458 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.458 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-14T15:12:54.458 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_opt=-f 2024-04-14T15:12:54.458 INFO:tasks.workunit.client.0.smithi090.stderr:++ echo open_pools_parallel 2024-04-14T15:12:54.458 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export which_declare 2024-04-14T15:12:54.458 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export S_COLORS=auto 2024-04-14T15:12:54.458 INFO:tasks.workunit.client.0.smithi090.stderr:+++ S_COLORS=auto 2024-04-14T15:12:54.458 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export -f which 2024-04-14T15:12:54.458 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.458 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-14T15:12:54.458 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.459 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset i 2024-04-14T15:12:54.459 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-14T15:12:54.459 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset -f pathmunge 2024-04-14T15:12:54.459 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.459 INFO:tasks.workunit.client.0.smithi090.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-14T15:12:54.459 INFO:tasks.workunit.client.0.smithi090.stderr:+ export PATH 2024-04-14T15:12:54.459 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-14T15:12:54.459 INFO:tasks.workunit.client.0.smithi090.stderr:+ unset rc 2024-04-14T15:12:54.459 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-14T15:12:54.459 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-14T15:12:54.459 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-14T15:12:54.459 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.459 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-14T15:12:54.459 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.460 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-14T15:12:54.460 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-14T15:12:54.460 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.460 INFO:tasks.workunit.client.0.smithi090.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2024-04-14T15:12:54.460 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.460 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-14T15:12:54.460 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.460 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/lang.sh 2024-04-14T15:12:54.460 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup 2024-04-14T15:12:54.460 INFO:tasks.workunit.client.0.smithi090.stderr:+++++ readlink /proc/93753/exe 2024-04-14T15:12:54.460 INFO:tasks.workunit.client.0.smithi090.stderr:+ ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2024-04-14T15:12:54.460 INFO:tasks.workunit.client.0.smithi090.stderr:+ ff=open_pools_parallel 2024-04-14T15:12:54.460 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-14T15:12:54.460 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-14T15:12:54.460 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-14T15:12:54.461 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.461 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-14T15:12:54.461 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.461 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-14T15:12:54.461 INFO:tasks.workunit.client.0.smithi090.stderr:++++ basename /usr/bin/bash 2024-04-14T15:12:54.461 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-14T15:12:54.461 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.461 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'grep=grep --color=auto' 2024-04-14T15:12:54.461 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-14T15:12:54.461 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-14T15:12:54.461 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.461 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-14T15:12:54.461 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_declare='declare -f' 2024-04-14T15:12:54.461 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.462 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_opt=-f 2024-04-14T15:12:54.462 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorls.sh 2024-04-14T15:12:54.462 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export which_declare 2024-04-14T15:12:54.462 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' '!' -t 0 ']' 2024-04-14T15:12:54.462 INFO:tasks.workunit.client.0.smithi090.stderr:+++ return 2024-04-14T15:12:54.462 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export -f which 2024-04-14T15:12:54.462 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.462 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-14T15:12:54.462 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.462 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset i 2024-04-14T15:12:54.462 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-14T15:12:54.462 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset -f pathmunge 2024-04-14T15:12:54.462 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export S_COLORS=auto 2024-04-14T15:12:54.462 INFO:tasks.workunit.client.0.smithi090.stderr:+++ S_COLORS=auto 2024-04-14T15:12:54.463 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.463 INFO:tasks.workunit.client.0.smithi090.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-14T15:12:54.463 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-14T15:12:54.463 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.463 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-14T15:12:54.463 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.463 INFO:tasks.workunit.client.0.smithi090.stderr:+ export PATH 2024-04-14T15:12:54.463 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-14T15:12:54.463 INFO:tasks.workunit.client.0.smithi090.stderr:+ unset rc 2024-04-14T15:12:54.463 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=94006 2024-04-14T15:12:54.464 INFO:tasks.workunit.client.0.smithi090.stdout:test open_pools_parallel on pid 94006 2024-04-14T15:12:54.468 INFO:tasks.workunit.client.0.smithi090.stderr:+ echo 'test open_pools_parallel on pid 94006' 2024-04-14T15:12:54.468 INFO:tasks.workunit.client.0.smithi090.stderr:+ pids[$f]=94006 2024-04-14T15:12:54.469 INFO:tasks.workunit.client.0.smithi090.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-14T15:12:54.469 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' 1 -eq 1 ']' 2024-04-14T15:12:54.469 INFO:tasks.workunit.client.0.smithi090.stderr:+ sed 's/^/ api_cmd_pp: /' 2024-04-14T15:12:54.469 INFO:tasks.workunit.client.0.smithi090.stderr:++ printf %25s delete_pools_parallel 2024-04-14T15:12:54.469 INFO:tasks.workunit.client.0.smithi090.stderr:+ r=' delete_pools_parallel' 2024-04-14T15:12:54.469 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-14T15:12:54.469 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-14T15:12:54.469 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-14T15:12:54.469 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.469 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-14T15:12:54.469 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.469 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-14T15:12:54.469 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-14T15:12:54.470 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.470 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-14T15:12:54.470 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.470 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/gawk.sh 2024-04-14T15:12:54.470 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.470 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-14T15:12:54.470 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.470 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/lang.sh 2024-04-14T15:12:54.470 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup 2024-04-14T15:12:54.470 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/locale 2024-04-14T15:12:54.470 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-14T15:12:54.470 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-14T15:12:54.470 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-14T15:12:54.470 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.471 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-14T15:12:54.471 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.471 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-14T15:12:54.471 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-14T15:12:54.471 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.471 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-14T15:12:54.471 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.471 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/gawk.sh 2024-04-14T15:12:54.471 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.471 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-14T15:12:54.471 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.471 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/lang.sh 2024-04-14T15:12:54.471 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup 2024-04-14T15:12:54.471 INFO:tasks.workunit.client.0.smithi090.stderr:+ ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2024-04-14T15:12:54.471 INFO:tasks.workunit.client.0.smithi090.stderr:+ tee ceph_test_rados_api_cmd.log 2024-04-14T15:12:54.472 INFO:tasks.workunit.client.0.smithi090.stderr:++ echo delete_pools_parallel 2024-04-14T15:12:54.472 INFO:tasks.workunit.client.0.smithi090.stderr:++ awk '{print $1}' 2024-04-14T15:12:54.472 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.472 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-14T15:12:54.472 INFO:tasks.workunit.client.0.smithi090.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2024-04-14T15:12:54.472 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.472 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-14T15:12:54.472 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-14T15:12:54.472 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-14T15:12:54.472 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-14T15:12:54.472 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-14T15:12:54.472 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.472 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-14T15:12:54.472 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.473 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/locale 2024-04-14T15:12:54.473 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-14T15:12:54.473 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-14T15:12:54.473 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.473 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.473 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-14T15:12:54.473 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.473 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-14T15:12:54.473 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-14T15:12:54.473 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-14T15:12:54.473 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-14T15:12:54.473 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-14T15:12:54.473 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.474 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-14T15:12:54.474 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.474 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-14T15:12:54.474 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-14T15:12:54.474 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-14T15:12:54.474 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.474 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/locale 2024-04-14T15:12:54.474 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -f /etc/bashrc ']' 2024-04-14T15:12:54.474 INFO:tasks.workunit.client.0.smithi090.stderr:+ . /etc/bashrc 2024-04-14T15:12:54.474 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -z '' ']' 2024-04-14T15:12:54.474 INFO:tasks.workunit.client.0.smithi090.stderr:++ BASHRCSOURCED=Y 2024-04-14T15:12:54.474 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.474 INFO:tasks.workunit.client.0.smithi090.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-14T15:12:54.474 INFO:tasks.workunit.client.0.smithi090.stderr:++++ export LANG=en_US.UTF-8 2024-04-14T15:12:54.475 INFO:tasks.workunit.client.0.smithi090.stderr:++++ LANG=en_US.UTF-8 2024-04-14T15:12:54.475 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.475 INFO:tasks.workunit.client.0.smithi090.stderr:++ shopt -q login_shell 2024-04-14T15:12:54.475 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-14T15:12:54.475 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.475 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG=en_US.UTF-8 2024-04-14T15:12:54.475 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup config 2024-04-14T15:12:54.475 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.475 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.475 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-14T15:12:54.475 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.475 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' dumb = linux ']' 2024-04-14T15:12:54.475 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.475 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-14T15:12:54.476 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.476 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-14T15:12:54.476 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-14T15:12:54.476 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.476 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/less.sh 2024-04-14T15:12:54.476 INFO:tasks.workunit.client.0.smithi090.stderr:+++ umask 2024-04-14T15:12:54.476 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-14T15:12:54.476 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.476 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-14T15:12:54.476 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.476 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/which2.sh 2024-04-14T15:12:54.476 INFO:tasks.workunit.client.0.smithi090.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-14T15:12:54.476 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' 0022 -eq 0 ']' 2024-04-14T15:12:54.476 INFO:tasks.workunit.client.0.smithi090.stderr:++ SHELL=/bin/bash 2024-04-14T15:12:54.477 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-14T15:12:54.477 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.477 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-14T15:12:54.477 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.477 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-14T15:12:54.477 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-14T15:12:54.477 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.477 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-14T15:12:54.477 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.477 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-14T15:12:54.477 INFO:tasks.workunit.client.0.smithi090.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2024-04-14T15:12:54.477 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.477 INFO:tasks.workunit.client.0.smithi090.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.478 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.478 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-14T15:12:54.478 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.478 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-14T15:12:54.478 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.478 INFO:tasks.workunit.client.0.smithi090.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2024-04-14T15:12:54.478 INFO:tasks.workunit.client.0.smithi090.stderr:+ ff=delete_pools_parallel 2024-04-14T15:12:54.478 INFO:tasks.workunit.client.0.smithi090.stdout:test delete_pools_parallel on pid 94043 2024-04-14T15:12:54.482 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=94043 2024-04-14T15:12:54.482 INFO:tasks.workunit.client.0.smithi090.stderr:+ echo 'test delete_pools_parallel on pid 94043' 2024-04-14T15:12:54.482 INFO:tasks.workunit.client.0.smithi090.stderr:+ pids[$f]=94043 2024-04-14T15:12:54.482 INFO:tasks.workunit.client.0.smithi090.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-14T15:12:54.482 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' 1 -eq 1 ']' 2024-04-14T15:12:54.482 INFO:tasks.workunit.client.0.smithi090.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-14T15:12:54.482 INFO:tasks.workunit.client.0.smithi090.stderr:++++ export LANG=en_US.UTF-8 2024-04-14T15:12:54.482 INFO:tasks.workunit.client.0.smithi090.stderr:++++ LANG=en_US.UTF-8 2024-04-14T15:12:54.483 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.483 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-14T15:12:54.483 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.483 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG=en_US.UTF-8 2024-04-14T15:12:54.483 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup config 2024-04-14T15:12:54.483 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.483 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-14T15:12:54.483 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.483 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-14T15:12:54.483 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' dumb = linux ']' 2024-04-14T15:12:54.483 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-14T15:12:54.483 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.483 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-14T15:12:54.484 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-14T15:12:54.484 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.484 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/less.sh 2024-04-14T15:12:54.484 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.484 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-14T15:12:54.484 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.484 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-14T15:12:54.484 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-14T15:12:54.484 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.484 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-14T15:12:54.484 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.484 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/which2.sh 2024-04-14T15:12:54.484 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-14T15:12:54.485 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.485 INFO:tasks.workunit.client.0.smithi090.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-14T15:12:54.485 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-14T15:12:54.485 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.485 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/gawk.sh 2024-04-14T15:12:54.485 INFO:tasks.workunit.client.0.smithi090.stderr:+ sed 's/^/ api_cmd: /' 2024-04-14T15:12:54.485 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.485 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-14T15:12:54.485 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.485 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/lang.sh 2024-04-14T15:12:54.485 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup 2024-04-14T15:12:54.485 INFO:tasks.workunit.client.0.smithi090.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2024-04-14T15:12:54.485 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-14T15:12:54.486 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-14T15:12:54.486 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-14T15:12:54.486 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/locale 2024-04-14T15:12:54.486 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.486 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-14T15:12:54.486 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.486 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-14T15:12:54.486 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-14T15:12:54.486 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.486 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-14T15:12:54.486 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.486 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/gawk.sh 2024-04-14T15:12:54.486 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.487 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-14T15:12:54.487 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.487 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/lang.sh 2024-04-14T15:12:54.487 INFO:tasks.workunit.client.0.smithi090.stderr:+++++ readlink /proc/93841/exe 2024-04-14T15:12:54.487 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup 2024-04-14T15:12:54.487 INFO:tasks.workunit.client.0.smithi090.stderr:++ printf %25s cls 2024-04-14T15:12:54.487 INFO:tasks.workunit.client.0.smithi090.stderr:+ r=' cls' 2024-04-14T15:12:54.487 INFO:tasks.workunit.client.0.smithi090.stderr:++++ basename /usr/bin/bash 2024-04-14T15:12:54.487 INFO:tasks.workunit.client.0.smithi090.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-14T15:12:54.487 INFO:tasks.workunit.client.0.smithi090.stderr:++++ export LANG=en_US.UTF-8 2024-04-14T15:12:54.487 INFO:tasks.workunit.client.0.smithi090.stderr:++++ LANG=en_US.UTF-8 2024-04-14T15:12:54.487 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.487 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-14T15:12:54.487 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.488 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG=en_US.UTF-8 2024-04-14T15:12:54.488 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup config 2024-04-14T15:12:54.488 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.488 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.488 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -f /etc/bashrc ']' 2024-04-14T15:12:54.488 INFO:tasks.workunit.client.0.smithi090.stderr:+ . /etc/bashrc 2024-04-14T15:12:54.488 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' dumb = linux ']' 2024-04-14T15:12:54.488 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.488 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-14T15:12:54.488 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.488 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/less.sh 2024-04-14T15:12:54.489 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -z '' ']' 2024-04-14T15:12:54.489 INFO:tasks.workunit.client.0.smithi090.stderr:++ BASHRCSOURCED=Y 2024-04-14T15:12:54.489 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-14T15:12:54.489 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.489 INFO:tasks.workunit.client.0.smithi090.stderr:++ shopt -q login_shell 2024-04-14T15:12:54.489 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.489 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-14T15:12:54.489 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.489 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/which2.sh 2024-04-14T15:12:54.489 INFO:tasks.workunit.client.0.smithi090.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-14T15:12:54.489 INFO:tasks.workunit.client.0.smithi090.stderr:+++ umask 2024-04-14T15:12:54.490 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/locale 2024-04-14T15:12:54.490 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_declare='declare -f' 2024-04-14T15:12:54.490 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_opt=-f 2024-04-14T15:12:54.490 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' 0022 -eq 0 ']' 2024-04-14T15:12:54.490 INFO:tasks.workunit.client.0.smithi090.stderr:++ SHELL=/bin/bash 2024-04-14T15:12:54.490 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export which_declare 2024-04-14T15:12:54.490 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export -f which 2024-04-14T15:12:54.490 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset i 2024-04-14T15:12:54.490 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset -f pathmunge 2024-04-14T15:12:54.490 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.490 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-14T15:12:54.490 INFO:tasks.workunit.client.0.smithi090.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-14T15:12:54.491 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.491 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-14T15:12:54.491 INFO:tasks.workunit.client.0.smithi090.stderr:+ export PATH 2024-04-14T15:12:54.491 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'grep=grep --color=auto' 2024-04-14T15:12:54.491 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-14T15:12:54.491 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-14T15:12:54.491 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.491 INFO:tasks.workunit.client.0.smithi090.stderr:+ unset rc 2024-04-14T15:12:54.491 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-14T15:12:54.491 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-14T15:12:54.492 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.492 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-14T15:12:54.492 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-14T15:12:54.492 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.492 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.492 INFO:tasks.workunit.client.0.smithi090.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.492 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-14T15:12:54.492 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.492 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-14T15:12:54.492 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.492 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorls.sh 2024-04-14T15:12:54.492 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.492 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-14T15:12:54.493 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.493 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' '!' -t 0 ']' 2024-04-14T15:12:54.493 INFO:tasks.workunit.client.0.smithi090.stderr:+++ return 2024-04-14T15:12:54.493 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.493 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-14T15:12:54.493 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.493 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-14T15:12:54.493 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export S_COLORS=auto 2024-04-14T15:12:54.493 INFO:tasks.workunit.client.0.smithi090.stderr:+++ S_COLORS=auto 2024-04-14T15:12:54.493 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.493 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-14T15:12:54.493 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.494 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-14T15:12:54.494 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.494 INFO:tasks.workunit.client.0.smithi090.stderr:+ ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2024-04-14T15:12:54.494 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.494 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-14T15:12:54.494 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.494 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-14T15:12:54.494 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-14T15:12:54.494 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-14T15:12:54.494 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.494 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-14T15:12:54.494 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.495 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-14T15:12:54.495 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-14T15:12:54.495 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-14T15:12:54.495 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-14T15:12:54.495 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-14T15:12:54.495 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.495 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-14T15:12:54.495 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.495 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-14T15:12:54.495 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-14T15:12:54.495 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.495 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-14T15:12:54.495 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.496 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/gawk.sh 2024-04-14T15:12:54.496 INFO:tasks.workunit.client.0.smithi090.stderr:++ echo cls 2024-04-14T15:12:54.496 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.496 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-14T15:12:54.496 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.496 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/lang.sh 2024-04-14T15:12:54.496 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup 2024-04-14T15:12:54.496 INFO:tasks.workunit.client.0.smithi090.stderr:+ sed 's/^/ api_c_write_operations: /' 2024-04-14T15:12:54.496 INFO:tasks.workunit.client.0.smithi090.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-14T15:12:54.496 INFO:tasks.workunit.client.0.smithi090.stderr:++++ export LANG=en_US.UTF-8 2024-04-14T15:12:54.496 INFO:tasks.workunit.client.0.smithi090.stderr:++++ LANG=en_US.UTF-8 2024-04-14T15:12:54.496 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.497 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-14T15:12:54.497 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.497 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG=en_US.UTF-8 2024-04-14T15:12:54.497 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup config 2024-04-14T15:12:54.497 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.497 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.497 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' dumb = linux ']' 2024-04-14T15:12:54.497 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.497 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-14T15:12:54.497 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.497 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/less.sh 2024-04-14T15:12:54.497 INFO:tasks.workunit.client.0.smithi090.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2024-04-14T15:12:54.497 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-14T15:12:54.497 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.498 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-14T15:12:54.498 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.498 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/which2.sh 2024-04-14T15:12:54.498 INFO:tasks.workunit.client.0.smithi090.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-14T15:12:54.498 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-14T15:12:54.498 INFO:tasks.workunit.client.0.smithi090.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-14T15:12:54.498 INFO:tasks.workunit.client.0.smithi090.stderr:++++ export LANG=en_US.UTF-8 2024-04-14T15:12:54.498 INFO:tasks.workunit.client.0.smithi090.stderr:++++ LANG=en_US.UTF-8 2024-04-14T15:12:54.498 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.498 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-14T15:12:54.498 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.499 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG=en_US.UTF-8 2024-04-14T15:12:54.499 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup config 2024-04-14T15:12:54.499 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.499 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.499 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' dumb = linux ']' 2024-04-14T15:12:54.499 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.499 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-14T15:12:54.499 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.499 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/less.sh 2024-04-14T15:12:54.499 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-14T15:12:54.499 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.499 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-14T15:12:54.499 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.499 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/which2.sh 2024-04-14T15:12:54.499 INFO:tasks.workunit.client.0.smithi090.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-14T15:12:54.500 INFO:tasks.workunit.client.0.smithi090.stderr:+++++ readlink /proc/93788/exe 2024-04-14T15:12:54.500 INFO:tasks.workunit.client.0.smithi090.stderr:+++++ readlink /proc/93813/exe 2024-04-14T15:12:54.500 INFO:tasks.workunit.client.0.smithi090.stderr:++ awk '{print $1}' 2024-04-14T15:12:54.500 INFO:tasks.workunit.client.0.smithi090.stderr:++++ basename /usr/bin/bash 2024-04-14T15:12:54.500 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_declare='declare -f' 2024-04-14T15:12:54.500 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_opt=-f 2024-04-14T15:12:54.500 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export which_declare 2024-04-14T15:12:54.500 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export -f which 2024-04-14T15:12:54.500 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset i 2024-04-14T15:12:54.500 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset -f pathmunge 2024-04-14T15:12:54.500 INFO:tasks.workunit.client.0.smithi090.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-14T15:12:54.500 INFO:tasks.workunit.client.0.smithi090.stderr:+ export PATH 2024-04-14T15:12:54.500 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-14T15:12:54.500 INFO:tasks.workunit.client.0.smithi090.stderr:+ unset rc 2024-04-14T15:12:54.500 INFO:tasks.workunit.client.0.smithi090.stderr:++++ basename /usr/bin/bash 2024-04-14T15:12:54.501 INFO:tasks.workunit.client.0.smithi090.stderr:+++++ readlink /proc/93960/exe 2024-04-14T15:12:54.501 INFO:tasks.workunit.client.0.smithi090.stdout:test cls on pid 94102 2024-04-14T15:12:54.508 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'grep=grep --color=auto' 2024-04-14T15:12:54.508 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-14T15:12:54.508 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-14T15:12:54.508 INFO:tasks.workunit.client.0.smithi090.stderr:+ ff=cls 2024-04-14T15:12:54.509 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.509 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-14T15:12:54.509 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.509 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorls.sh 2024-04-14T15:12:54.509 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/locale 2024-04-14T15:12:54.509 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' '!' -t 0 ']' 2024-04-14T15:12:54.509 INFO:tasks.workunit.client.0.smithi090.stderr:+++ return 2024-04-14T15:12:54.509 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.509 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-14T15:12:54.509 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.509 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-14T15:12:54.509 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=94102 2024-04-14T15:12:54.510 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export S_COLORS=auto 2024-04-14T15:12:54.510 INFO:tasks.workunit.client.0.smithi090.stderr:+++ S_COLORS=auto 2024-04-14T15:12:54.510 INFO:tasks.workunit.client.0.smithi090.stderr:+ echo 'test cls on pid 94102' 2024-04-14T15:12:54.510 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.510 INFO:tasks.workunit.client.0.smithi090.stderr:+ pids[$f]=94102 2024-04-14T15:12:54.510 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-14T15:12:54.510 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.510 INFO:tasks.workunit.client.0.smithi090.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-14T15:12:54.510 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-14T15:12:54.510 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' 1 -eq 1 ']' 2024-04-14T15:12:54.510 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.510 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_declare='declare -f' 2024-04-14T15:12:54.510 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_opt=-f 2024-04-14T15:12:54.510 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export which_declare 2024-04-14T15:12:54.511 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export -f which 2024-04-14T15:12:54.511 INFO:tasks.workunit.client.0.smithi090.stderr:+++++ readlink /proc/93939/exe 2024-04-14T15:12:54.511 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset i 2024-04-14T15:12:54.511 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset -f pathmunge 2024-04-14T15:12:54.511 INFO:tasks.workunit.client.0.smithi090.stderr:++++ basename /usr/bin/bash 2024-04-14T15:12:54.511 INFO:tasks.workunit.client.0.smithi090.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-14T15:12:54.511 INFO:tasks.workunit.client.0.smithi090.stderr:+ export PATH 2024-04-14T15:12:54.511 INFO:tasks.workunit.client.0.smithi090.stderr:+ sed 's/^/ api_service: /' 2024-04-14T15:12:54.511 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-14T15:12:54.511 INFO:tasks.workunit.client.0.smithi090.stderr:+ unset rc 2024-04-14T15:12:54.511 INFO:tasks.workunit.client.0.smithi090.stderr:+ ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2024-04-14T15:12:54.511 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_declare='declare -f' 2024-04-14T15:12:54.511 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_opt=-f 2024-04-14T15:12:54.511 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export which_declare 2024-04-14T15:12:54.511 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export -f which 2024-04-14T15:12:54.512 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset i 2024-04-14T15:12:54.512 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset -f pathmunge 2024-04-14T15:12:54.512 INFO:tasks.workunit.client.0.smithi090.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-14T15:12:54.512 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.512 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-14T15:12:54.512 INFO:tasks.workunit.client.0.smithi090.stderr:+ export PATH 2024-04-14T15:12:54.512 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-14T15:12:54.512 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.512 INFO:tasks.workunit.client.0.smithi090.stderr:+ unset rc 2024-04-14T15:12:54.512 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-14T15:12:54.512 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-14T15:12:54.512 INFO:tasks.workunit.client.0.smithi090.stderr:+ tee ceph_test_rados_api_service.log 2024-04-14T15:12:54.512 INFO:tasks.workunit.client.0.smithi090.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cls 2>&1 | tee ceph_test_neorados_cls.log | sed "s/^/ cls: /"' 2024-04-14T15:12:54.512 INFO:tasks.workunit.client.0.smithi090.stderr:++++ basename /usr/bin/bash 2024-04-14T15:12:54.513 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_declare='declare -f' 2024-04-14T15:12:54.513 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_opt=-f 2024-04-14T15:12:54.513 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export which_declare 2024-04-14T15:12:54.513 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export -f which 2024-04-14T15:12:54.513 INFO:tasks.workunit.client.0.smithi090.stderr:+ sed 's/^/ list_parallel: /' 2024-04-14T15:12:54.513 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset i 2024-04-14T15:12:54.513 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset -f pathmunge 2024-04-14T15:12:54.513 INFO:tasks.workunit.client.0.smithi090.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-14T15:12:54.513 INFO:tasks.workunit.client.0.smithi090.stderr:+ export PATH 2024-04-14T15:12:54.513 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-14T15:12:54.513 INFO:tasks.workunit.client.0.smithi090.stderr:+ unset rc 2024-04-14T15:12:54.513 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-14T15:12:54.513 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-14T15:12:54.513 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-14T15:12:54.513 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.514 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-14T15:12:54.514 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.514 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-14T15:12:54.514 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-14T15:12:54.514 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.514 INFO:tasks.workunit.client.0.smithi090.stderr:+ tee ceph_test_rados_api_service_pp.log 2024-04-14T15:12:54.514 INFO:tasks.workunit.client.0.smithi090.stderr:+ ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cls_remote_reads.xml 2024-04-14T15:12:54.514 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-14T15:12:54.514 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -f /etc/bashrc ']' 2024-04-14T15:12:54.514 INFO:tasks.workunit.client.0.smithi090.stderr:+ . /etc/bashrc 2024-04-14T15:12:54.514 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -z '' ']' 2024-04-14T15:12:54.514 INFO:tasks.workunit.client.0.smithi090.stderr:+ sed 's/^/ api_cls_remote_reads: /' 2024-04-14T15:12:54.514 INFO:tasks.workunit.client.0.smithi090.stderr:+ tee ceph_test_rados_list_parallel.log 2024-04-14T15:12:54.514 INFO:tasks.workunit.client.0.smithi090.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-14T15:12:54.515 INFO:tasks.workunit.client.0.smithi090.stderr:++++ export LANG=en_US.UTF-8 2024-04-14T15:12:54.515 INFO:tasks.workunit.client.0.smithi090.stderr:++++ LANG=en_US.UTF-8 2024-04-14T15:12:54.515 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.515 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-14T15:12:54.515 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.515 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG=en_US.UTF-8 2024-04-14T15:12:54.515 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup config 2024-04-14T15:12:54.515 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.515 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.515 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' dumb = linux ']' 2024-04-14T15:12:54.515 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.515 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-14T15:12:54.515 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.515 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/less.sh 2024-04-14T15:12:54.515 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-14T15:12:54.516 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.516 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-14T15:12:54.516 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.516 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/which2.sh 2024-04-14T15:12:54.516 INFO:tasks.workunit.client.0.smithi090.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-14T15:12:54.516 INFO:tasks.workunit.client.0.smithi090.stderr:++ printf %25s cmd 2024-04-14T15:12:54.516 INFO:tasks.workunit.client.0.smithi090.stderr:+ r=' cmd' 2024-04-14T15:12:54.516 INFO:tasks.workunit.client.0.smithi090.stderr:+ ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2024-04-14T15:12:54.516 INFO:tasks.workunit.client.0.smithi090.stderr:+++++ readlink /proc/93910/exe 2024-04-14T15:12:54.516 INFO:tasks.workunit.client.0.smithi090.stderr:++ BASHRCSOURCED=Y 2024-04-14T15:12:54.516 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.516 INFO:tasks.workunit.client.0.smithi090.stderr:++ shopt -q login_shell 2024-04-14T15:12:54.516 INFO:tasks.workunit.client.0.smithi090.stderr:+ tee ceph_test_rados_api_cls_remote_reads.log 2024-04-14T15:12:54.516 INFO:tasks.workunit.client.0.smithi090.stderr:+ ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2024-04-14T15:12:54.517 INFO:tasks.workunit.client.0.smithi090.stderr:++++ basename /usr/bin/bash 2024-04-14T15:12:54.517 INFO:tasks.workunit.client.0.smithi090.stderr:+ sed 's/^/ api_service_pp: /' 2024-04-14T15:12:54.517 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_declare='declare -f' 2024-04-14T15:12:54.517 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_opt=-f 2024-04-14T15:12:54.517 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export which_declare 2024-04-14T15:12:54.517 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export -f which 2024-04-14T15:12:54.517 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset i 2024-04-14T15:12:54.517 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset -f pathmunge 2024-04-14T15:12:54.517 INFO:tasks.workunit.client.0.smithi090.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-14T15:12:54.517 INFO:tasks.workunit.client.0.smithi090.stderr:+ export PATH 2024-04-14T15:12:54.517 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-14T15:12:54.517 INFO:tasks.workunit.client.0.smithi090.stderr:+ unset rc 2024-04-14T15:12:54.517 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-14T15:12:54.517 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-14T15:12:54.517 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-14T15:12:54.518 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.518 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-14T15:12:54.518 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.518 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-14T15:12:54.518 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-14T15:12:54.518 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.518 INFO:tasks.workunit.client.0.smithi090.stderr:+++ umask 2024-04-14T15:12:54.518 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' 0022 -eq 0 ']' 2024-04-14T15:12:54.518 INFO:tasks.workunit.client.0.smithi090.stderr:++ SHELL=/bin/bash 2024-04-14T15:12:54.518 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.518 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-14T15:12:54.518 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.518 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-14T15:12:54.518 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-14T15:12:54.518 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.519 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-14T15:12:54.519 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.519 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-14T15:12:54.519 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.519 INFO:tasks.workunit.client.0.smithi090.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.519 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.519 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-14T15:12:54.519 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.519 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-14T15:12:54.519 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.519 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-14T15:12:54.519 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-14T15:12:54.519 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-14T15:12:54.519 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.519 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-14T15:12:54.520 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.520 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-14T15:12:54.520 INFO:tasks.workunit.client.0.smithi090.stderr:+ ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2024-04-14T15:12:54.520 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-14T15:12:54.520 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.520 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-14T15:12:54.520 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.520 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/gawk.sh 2024-04-14T15:12:54.520 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.520 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-14T15:12:54.520 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.520 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/lang.sh 2024-04-14T15:12:54.521 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup 2024-04-14T15:12:54.521 INFO:tasks.workunit.client.0.smithi090.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2024-04-14T15:12:54.521 INFO:tasks.workunit.client.0.smithi090.stderr:+ sed 's/^/ api_c_read_operations: /' 2024-04-14T15:12:54.521 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'grep=grep --color=auto' 2024-04-14T15:12:54.521 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-14T15:12:54.521 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-14T15:12:54.521 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.521 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-14T15:12:54.521 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.521 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorls.sh 2024-04-14T15:12:54.521 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' '!' -t 0 ']' 2024-04-14T15:12:54.521 INFO:tasks.workunit.client.0.smithi090.stderr:+++ return 2024-04-14T15:12:54.521 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.521 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-14T15:12:54.521 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.521 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-14T15:12:54.522 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export S_COLORS=auto 2024-04-14T15:12:54.522 INFO:tasks.workunit.client.0.smithi090.stderr:+++ S_COLORS=auto 2024-04-14T15:12:54.522 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.522 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-14T15:12:54.522 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.522 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-14T15:12:54.522 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.522 INFO:tasks.workunit.client.0.smithi090.stderr:++ echo cmd 2024-04-14T15:12:54.522 INFO:tasks.workunit.client.0.smithi090.stderr:++ awk '{print $1}' 2024-04-14T15:12:54.522 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-14T15:12:54.522 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-14T15:12:54.522 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-14T15:12:54.522 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.522 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-14T15:12:54.523 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.523 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-14T15:12:54.523 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-14T15:12:54.523 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.523 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-14T15:12:54.523 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.523 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/gawk.sh 2024-04-14T15:12:54.523 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.523 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-14T15:12:54.523 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.523 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/lang.sh 2024-04-14T15:12:54.523 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup 2024-04-14T15:12:54.523 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/locale 2024-04-14T15:12:54.524 INFO:tasks.workunit.client.0.smithi090.stdout:test cmd on pid 94168 2024-04-14T15:12:54.531 INFO:tasks.workunit.client.0.smithi090.stderr:+ ff=cmd 2024-04-14T15:12:54.531 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=94168 2024-04-14T15:12:54.531 INFO:tasks.workunit.client.0.smithi090.stderr:+ echo 'test cmd on pid 94168' 2024-04-14T15:12:54.531 INFO:tasks.workunit.client.0.smithi090.stderr:+ pids[$f]=94168 2024-04-14T15:12:54.531 INFO:tasks.workunit.client.0.smithi090.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-14T15:12:54.531 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' 1 -eq 1 ']' 2024-04-14T15:12:54.531 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/locale 2024-04-14T15:12:54.531 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-14T15:12:54.531 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-14T15:12:54.531 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-14T15:12:54.531 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.531 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-14T15:12:54.532 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.532 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-14T15:12:54.532 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-14T15:12:54.532 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.532 INFO:tasks.workunit.client.0.smithi090.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cmd 2>&1 | tee ceph_test_neorados_cmd.log | sed "s/^/ cmd: /"' 2024-04-14T15:12:54.532 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -f /etc/bashrc ']' 2024-04-14T15:12:54.532 INFO:tasks.workunit.client.0.smithi090.stderr:+ . /etc/bashrc 2024-04-14T15:12:54.532 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -z '' ']' 2024-04-14T15:12:54.532 INFO:tasks.workunit.client.0.smithi090.stderr:++ printf %25s handler_error 2024-04-14T15:12:54.532 INFO:tasks.workunit.client.0.smithi090.stderr:++ BASHRCSOURCED=Y 2024-04-14T15:12:54.532 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.532 INFO:tasks.workunit.client.0.smithi090.stderr:++ shopt -q login_shell 2024-04-14T15:12:54.532 INFO:tasks.workunit.client.0.smithi090.stderr:+ r=' handler_error' 2024-04-14T15:12:54.532 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.532 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-14T15:12:54.533 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.533 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-14T15:12:54.533 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-14T15:12:54.533 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-14T15:12:54.533 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.533 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-14T15:12:54.533 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.533 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-14T15:12:54.533 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-14T15:12:54.533 INFO:tasks.workunit.client.0.smithi090.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-14T15:12:54.533 INFO:tasks.workunit.client.0.smithi090.stderr:++++ export LANG=en_US.UTF-8 2024-04-14T15:12:54.533 INFO:tasks.workunit.client.0.smithi090.stderr:++++ LANG=en_US.UTF-8 2024-04-14T15:12:54.533 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.533 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-14T15:12:54.534 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.534 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG=en_US.UTF-8 2024-04-14T15:12:54.534 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup config 2024-04-14T15:12:54.534 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.534 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.534 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' dumb = linux ']' 2024-04-14T15:12:54.534 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.534 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-14T15:12:54.534 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.534 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/less.sh 2024-04-14T15:12:54.534 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-14T15:12:54.534 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.534 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-14T15:12:54.534 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.535 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/which2.sh 2024-04-14T15:12:54.535 INFO:tasks.workunit.client.0.smithi090.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-14T15:12:54.535 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-14T15:12:54.535 INFO:tasks.workunit.client.0.smithi090.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-14T15:12:54.535 INFO:tasks.workunit.client.0.smithi090.stderr:++++ export LANG=en_US.UTF-8 2024-04-14T15:12:54.535 INFO:tasks.workunit.client.0.smithi090.stderr:++++ LANG=en_US.UTF-8 2024-04-14T15:12:54.535 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.535 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-14T15:12:54.535 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.535 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG=en_US.UTF-8 2024-04-14T15:12:54.535 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup config 2024-04-14T15:12:54.535 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.535 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.535 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' dumb = linux ']' 2024-04-14T15:12:54.536 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.536 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-14T15:12:54.536 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.536 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/less.sh 2024-04-14T15:12:54.536 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-14T15:12:54.536 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.536 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-14T15:12:54.536 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.536 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/which2.sh 2024-04-14T15:12:54.536 INFO:tasks.workunit.client.0.smithi090.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-14T15:12:54.536 INFO:tasks.workunit.client.0.smithi090.stderr:++ awk '{print $1}' 2024-04-14T15:12:54.536 INFO:tasks.workunit.client.0.smithi090.stderr:+++++ readlink /proc/94006/exe 2024-04-14T15:12:54.536 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-14T15:12:54.536 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-14T15:12:54.537 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-14T15:12:54.537 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.537 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-14T15:12:54.537 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.537 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-14T15:12:54.537 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-14T15:12:54.537 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.537 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-14T15:12:54.537 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.537 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/gawk.sh 2024-04-14T15:12:54.537 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.537 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-14T15:12:54.537 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.537 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/lang.sh 2024-04-14T15:12:54.537 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup 2024-04-14T15:12:54.538 INFO:tasks.workunit.client.0.smithi090.stderr:+++ umask 2024-04-14T15:12:54.538 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' 0022 -eq 0 ']' 2024-04-14T15:12:54.538 INFO:tasks.workunit.client.0.smithi090.stderr:++ SHELL=/bin/bash 2024-04-14T15:12:54.538 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.538 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-14T15:12:54.538 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.538 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-14T15:12:54.538 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-14T15:12:54.538 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.538 INFO:tasks.workunit.client.0.smithi090.stderr:++++ basename /usr/bin/bash 2024-04-14T15:12:54.538 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-14T15:12:54.538 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.538 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-14T15:12:54.538 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.539 INFO:tasks.workunit.client.0.smithi090.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.539 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.539 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-14T15:12:54.539 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.539 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-14T15:12:54.539 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.539 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_declare='declare -f' 2024-04-14T15:12:54.539 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_opt=-f 2024-04-14T15:12:54.539 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export which_declare 2024-04-14T15:12:54.539 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export -f which 2024-04-14T15:12:54.539 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset i 2024-04-14T15:12:54.539 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/locale 2024-04-14T15:12:54.539 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset -f pathmunge 2024-04-14T15:12:54.539 INFO:tasks.workunit.client.0.smithi090.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-14T15:12:54.539 INFO:tasks.workunit.client.0.smithi090.stderr:+ export PATH 2024-04-14T15:12:54.540 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-14T15:12:54.540 INFO:tasks.workunit.client.0.smithi090.stderr:+ unset rc 2024-04-14T15:12:54.540 INFO:tasks.workunit.client.0.smithi090.stderr:+++++ readlink /proc/94043/exe 2024-04-14T15:12:54.540 INFO:tasks.workunit.client.0.smithi090.stderr:++++ basename /usr/bin/bash 2024-04-14T15:12:54.540 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_declare='declare -f' 2024-04-14T15:12:54.540 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.540 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_opt=-f 2024-04-14T15:12:54.540 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-14T15:12:54.540 INFO:tasks.workunit.client.0.smithi090.stderr:+ ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2024-04-14T15:12:54.540 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export which_declare 2024-04-14T15:12:54.540 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export -f which 2024-04-14T15:12:54.540 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.540 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-14T15:12:54.540 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-14T15:12:54.540 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset i 2024-04-14T15:12:54.541 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset -f pathmunge 2024-04-14T15:12:54.541 INFO:tasks.workunit.client.0.smithi090.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-14T15:12:54.541 INFO:tasks.workunit.client.0.smithi090.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2024-04-14T15:12:54.541 INFO:tasks.workunit.client.0.smithi090.stderr:+ export PATH 2024-04-14T15:12:54.541 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-14T15:12:54.541 INFO:tasks.workunit.client.0.smithi090.stderr:+ unset rc 2024-04-14T15:12:54.541 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-14T15:12:54.541 INFO:tasks.workunit.client.0.smithi090.stderr:+ sed 's/^/ open_pools_parallel: /' 2024-04-14T15:12:54.541 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'grep=grep --color=auto' 2024-04-14T15:12:54.541 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-14T15:12:54.541 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-14T15:12:54.541 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.541 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-14T15:12:54.541 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.541 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorls.sh 2024-04-14T15:12:54.542 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' '!' -t 0 ']' 2024-04-14T15:12:54.542 INFO:tasks.workunit.client.0.smithi090.stderr:+++ return 2024-04-14T15:12:54.542 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.542 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-14T15:12:54.542 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.542 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-14T15:12:54.542 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export S_COLORS=auto 2024-04-14T15:12:54.542 INFO:tasks.workunit.client.0.smithi090.stderr:+++ S_COLORS=auto 2024-04-14T15:12:54.542 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.542 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-14T15:12:54.542 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.542 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-14T15:12:54.542 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.542 INFO:tasks.workunit.client.0.smithi090.stderr:++ echo handler_error 2024-04-14T15:12:54.543 INFO:tasks.workunit.client.0.smithi090.stderr:+ ff=handler_error 2024-04-14T15:12:54.543 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=94252 2024-04-14T15:12:54.543 INFO:tasks.workunit.client.0.smithi090.stderr:+ echo 'test handler_error on pid 94252' 2024-04-14T15:12:54.543 INFO:tasks.workunit.client.0.smithi090.stdout:test handler_error on pid 94252 2024-04-14T15:12:54.545 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:12:54.547 INFO:tasks.workunit.client.0.smithi090.stderr:+ pids[$f]=94252 2024-04-14T15:12:54.547 INFO:tasks.workunit.client.0.smithi090.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-14T15:12:54.547 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' 1 -eq 1 ']' 2024-04-14T15:12:54.547 INFO:tasks.workunit.client.0.smithi090.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-14T15:12:54.547 INFO:tasks.workunit.client.0.smithi090.stderr:++++ export LANG=en_US.UTF-8 2024-04-14T15:12:54.547 INFO:tasks.workunit.client.0.smithi090.stderr:++++ LANG=en_US.UTF-8 2024-04-14T15:12:54.547 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.547 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-14T15:12:54.547 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.547 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG=en_US.UTF-8 2024-04-14T15:12:54.547 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup config 2024-04-14T15:12:54.548 INFO:tasks.workunit.client.0.smithi090.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_handler_error 2>&1 | tee ceph_test_neorados_handler_error.log | sed "s/^/ handler_error: /"' 2024-04-14T15:12:54.548 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.548 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.548 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' dumb = linux ']' 2024-04-14T15:12:54.548 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.548 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-14T15:12:54.548 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.548 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/less.sh 2024-04-14T15:12:54.548 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-14T15:12:54.548 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.548 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-14T15:12:54.548 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.548 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/which2.sh 2024-04-14T15:12:54.548 INFO:tasks.workunit.client.0.smithi090.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-14T15:12:54.549 INFO:tasks.workunit.client.0.smithi090.stderr:+++++ readlink /proc/94102/exe 2024-04-14T15:12:54.549 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -f /etc/bashrc ']' 2024-04-14T15:12:54.549 INFO:tasks.workunit.client.0.smithi090.stderr:+ . /etc/bashrc 2024-04-14T15:12:54.549 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -z '' ']' 2024-04-14T15:12:54.549 INFO:tasks.workunit.client.0.smithi090.stderr:++ BASHRCSOURCED=Y 2024-04-14T15:12:54.549 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.549 INFO:tasks.workunit.client.0.smithi090.stderr:++ shopt -q login_shell 2024-04-14T15:12:54.549 INFO:tasks.workunit.client.0.smithi090.stderr:++++ basename /usr/bin/bash 2024-04-14T15:12:54.549 INFO:tasks.workunit.client.0.smithi090.stderr:++ printf %25s io 2024-04-14T15:12:54.549 INFO:tasks.workunit.client.0.smithi090.stderr:+ ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2024-04-14T15:12:54.549 INFO:tasks.workunit.client.0.smithi090.stderr:+ r=' io' 2024-04-14T15:12:54.549 INFO:tasks.workunit.client.0.smithi090.stderr:+ sed 's/^/ delete_pools_parallel: /' 2024-04-14T15:12:54.549 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_declare='declare -f' 2024-04-14T15:12:54.549 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_opt=-f 2024-04-14T15:12:54.549 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export which_declare 2024-04-14T15:12:54.550 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export -f which 2024-04-14T15:12:54.550 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset i 2024-04-14T15:12:54.550 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset -f pathmunge 2024-04-14T15:12:54.550 INFO:tasks.workunit.client.0.smithi090.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-14T15:12:54.550 INFO:tasks.workunit.client.0.smithi090.stderr:+ export PATH 2024-04-14T15:12:54.550 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-14T15:12:54.550 INFO:tasks.workunit.client.0.smithi090.stderr:+ unset rc 2024-04-14T15:12:54.550 INFO:tasks.workunit.client.0.smithi090.stderr:+ ceph_test_neorados_cls 2024-04-14T15:12:54.550 INFO:tasks.workunit.client.0.smithi090.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2024-04-14T15:12:54.550 INFO:tasks.workunit.client.0.smithi090.stderr:+ tee ceph_test_neorados_cls.log 2024-04-14T15:12:54.550 INFO:tasks.workunit.client.0.smithi090.stderr:++ echo io 2024-04-14T15:12:54.550 INFO:tasks.workunit.client.0.smithi090.stderr:+ sed 's/^/ cls: /' 2024-04-14T15:12:54.550 INFO:tasks.workunit.client.0.smithi090.stderr:++ awk '{print $1}' 2024-04-14T15:12:54.551 INFO:tasks.workunit.client.0.smithi090.stderr:+++ umask 2024-04-14T15:12:54.551 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' 0022 -eq 0 ']' 2024-04-14T15:12:54.551 INFO:tasks.workunit.client.0.smithi090.stderr:++ SHELL=/bin/bash 2024-04-14T15:12:54.551 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.551 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-14T15:12:54.551 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.551 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-14T15:12:54.551 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-14T15:12:54.551 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.551 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-14T15:12:54.551 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.551 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-14T15:12:54.551 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.551 INFO:tasks.workunit.client.0.smithi090.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.551 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.552 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-14T15:12:54.552 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.552 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-14T15:12:54.552 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.552 INFO:tasks.workunit.client.0.smithi090.stdout:test io on pid 94305 2024-04-14T15:12:54.556 INFO:tasks.workunit.client.0.smithi090.stderr:+ ff=io 2024-04-14T15:12:54.556 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=94305 2024-04-14T15:12:54.556 INFO:tasks.workunit.client.0.smithi090.stderr:+ echo 'test io on pid 94305' 2024-04-14T15:12:54.556 INFO:tasks.workunit.client.0.smithi090.stderr:+ pids[$f]=94305 2024-04-14T15:12:54.556 INFO:tasks.workunit.client.0.smithi090.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-14T15:12:54.556 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' 1 -eq 1 ']' 2024-04-14T15:12:54.556 INFO:tasks.workunit.client.0.smithi090.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_io 2>&1 | tee ceph_test_neorados_io.log | sed "s/^/ io: /"' 2024-04-14T15:12:54.556 INFO:tasks.workunit.client.0.smithi090.stderr:++ printf %25s ec_io 2024-04-14T15:12:54.556 INFO:tasks.workunit.client.0.smithi090.stderr:+ r=' ec_io' 2024-04-14T15:12:54.556 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-14T15:12:54.556 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-14T15:12:54.556 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-14T15:12:54.556 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.557 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-14T15:12:54.557 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.557 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-14T15:12:54.557 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-14T15:12:54.557 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.557 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -f /etc/bashrc ']' 2024-04-14T15:12:54.557 INFO:tasks.workunit.client.0.smithi090.stderr:+ . /etc/bashrc 2024-04-14T15:12:54.557 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -z '' ']' 2024-04-14T15:12:54.557 INFO:tasks.workunit.client.0.smithi090.stderr:++ BASHRCSOURCED=Y 2024-04-14T15:12:54.557 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.557 INFO:tasks.workunit.client.0.smithi090.stderr:++ shopt -q login_shell 2024-04-14T15:12:54.557 INFO:tasks.workunit.client.0.smithi090.stderr:+++ umask 2024-04-14T15:12:54.557 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' 0022 -eq 0 ']' 2024-04-14T15:12:54.557 INFO:tasks.workunit.client.0.smithi090.stderr:++ SHELL=/bin/bash 2024-04-14T15:12:54.557 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.558 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-14T15:12:54.558 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.558 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-14T15:12:54.558 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-14T15:12:54.558 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.558 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-14T15:12:54.558 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.558 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-14T15:12:54.558 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.558 INFO:tasks.workunit.client.0.smithi090.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.558 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.558 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-14T15:12:54.558 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.558 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-14T15:12:54.559 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.559 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'grep=grep --color=auto' 2024-04-14T15:12:54.559 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-14T15:12:54.559 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-14T15:12:54.559 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.559 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-14T15:12:54.559 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.559 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorls.sh 2024-04-14T15:12:54.559 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' '!' -t 0 ']' 2024-04-14T15:12:54.559 INFO:tasks.workunit.client.0.smithi090.stderr:+++ return 2024-04-14T15:12:54.559 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.559 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-14T15:12:54.559 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.559 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-14T15:12:54.560 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export S_COLORS=auto 2024-04-14T15:12:54.560 INFO:tasks.workunit.client.0.smithi090.stderr:+++ S_COLORS=auto 2024-04-14T15:12:54.560 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.560 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-14T15:12:54.560 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.560 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-14T15:12:54.560 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.560 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-14T15:12:54.560 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-14T15:12:54.560 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-14T15:12:54.560 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.560 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-14T15:12:54.560 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.560 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-14T15:12:54.560 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-14T15:12:54.561 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.561 INFO:tasks.workunit.client.0.smithi090.stderr:++ awk '{print $1}' 2024-04-14T15:12:54.561 INFO:tasks.workunit.client.0.smithi090.stderr:++ echo ec_io 2024-04-14T15:12:54.561 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'grep=grep --color=auto' 2024-04-14T15:12:54.561 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-14T15:12:54.561 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-14T15:12:54.561 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.561 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-14T15:12:54.561 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.561 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorls.sh 2024-04-14T15:12:54.561 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' '!' -t 0 ']' 2024-04-14T15:12:54.561 INFO:tasks.workunit.client.0.smithi090.stderr:+++ return 2024-04-14T15:12:54.561 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.561 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-14T15:12:54.562 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.562 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-14T15:12:54.562 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export S_COLORS=auto 2024-04-14T15:12:54.562 INFO:tasks.workunit.client.0.smithi090.stderr:+++ S_COLORS=auto 2024-04-14T15:12:54.562 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.562 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-14T15:12:54.562 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.562 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-14T15:12:54.562 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.562 INFO:tasks.workunit.client.0.smithi090.stderr:+ ff=ec_io 2024-04-14T15:12:54.563 INFO:tasks.workunit.client.0.smithi090.stdout:test ec_io on pid 94378 2024-04-14T15:12:54.566 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=94378 2024-04-14T15:12:54.566 INFO:tasks.workunit.client.0.smithi090.stderr:+ echo 'test ec_io on pid 94378' 2024-04-14T15:12:54.566 INFO:tasks.workunit.client.0.smithi090.stderr:+ pids[$f]=94378 2024-04-14T15:12:54.566 INFO:tasks.workunit.client.0.smithi090.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-14T15:12:54.566 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' 1 -eq 1 ']' 2024-04-14T15:12:54.566 INFO:tasks.workunit.client.0.smithi090.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_io 2>&1 | tee ceph_test_neorados_ec_io.log | sed "s/^/ ec_io: /"' 2024-04-14T15:12:54.566 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -f /etc/bashrc ']' 2024-04-14T15:12:54.567 INFO:tasks.workunit.client.0.smithi090.stderr:+ . /etc/bashrc 2024-04-14T15:12:54.567 INFO:tasks.workunit.client.0.smithi090.stderr:++ printf %25s list 2024-04-14T15:12:54.567 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-14T15:12:54.567 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -z '' ']' 2024-04-14T15:12:54.567 INFO:tasks.workunit.client.0.smithi090.stderr:++ BASHRCSOURCED=Y 2024-04-14T15:12:54.567 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.567 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-14T15:12:54.567 INFO:tasks.workunit.client.0.smithi090.stderr:++ shopt -q login_shell 2024-04-14T15:12:54.567 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-14T15:12:54.567 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.567 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-14T15:12:54.567 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.567 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-14T15:12:54.567 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-14T15:12:54.567 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.568 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-14T15:12:54.568 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.568 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/gawk.sh 2024-04-14T15:12:54.568 INFO:tasks.workunit.client.0.smithi090.stderr:+++ umask 2024-04-14T15:12:54.568 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.568 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-14T15:12:54.568 INFO:tasks.workunit.client.0.smithi090.stderr:+ r=' list' 2024-04-14T15:12:54.568 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.568 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/lang.sh 2024-04-14T15:12:54.568 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' 0022 -eq 0 ']' 2024-04-14T15:12:54.568 INFO:tasks.workunit.client.0.smithi090.stderr:++ SHELL=/bin/bash 2024-04-14T15:12:54.568 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup 2024-04-14T15:12:54.568 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.568 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-14T15:12:54.568 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.569 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-14T15:12:54.569 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-14T15:12:54.569 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.569 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-14T15:12:54.569 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.569 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-14T15:12:54.569 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.569 INFO:tasks.workunit.client.0.smithi090.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.569 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.569 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-14T15:12:54.569 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.569 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-14T15:12:54.569 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.569 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/locale 2024-04-14T15:12:54.569 INFO:tasks.workunit.client.0.smithi090.stderr:++ awk '{print $1}' 2024-04-14T15:12:54.570 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.570 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-14T15:12:54.570 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.570 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-14T15:12:54.570 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-14T15:12:54.570 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-14T15:12:54.570 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-14T15:12:54.570 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-14T15:12:54.570 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.570 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-14T15:12:54.570 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-14T15:12:54.570 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.570 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-14T15:12:54.570 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-14T15:12:54.571 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.571 INFO:tasks.workunit.client.0.smithi090.stderr:++ echo list 2024-04-14T15:12:54.571 INFO:tasks.workunit.client.0.smithi090.stderr:+ ff=list 2024-04-14T15:12:54.571 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-14T15:12:54.571 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-14T15:12:54.571 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-14T15:12:54.571 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.571 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-14T15:12:54.571 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.571 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-14T15:12:54.571 INFO:tasks.workunit.client.0.smithi090.stdout:test list on pid 94418 2024-04-14T15:12:54.574 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-14T15:12:54.574 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.574 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-14T15:12:54.574 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.574 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/gawk.sh 2024-04-14T15:12:54.574 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=94418 2024-04-14T15:12:54.574 INFO:tasks.workunit.client.0.smithi090.stderr:+ echo 'test list on pid 94418' 2024-04-14T15:12:54.575 INFO:tasks.workunit.client.0.smithi090.stderr:+ pids[$f]=94418 2024-04-14T15:12:54.575 INFO:tasks.workunit.client.0.smithi090.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-14T15:12:54.575 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' 1 -eq 1 ']' 2024-04-14T15:12:54.575 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.575 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-14T15:12:54.575 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.575 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/lang.sh 2024-04-14T15:12:54.575 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup 2024-04-14T15:12:54.575 INFO:tasks.workunit.client.0.smithi090.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_list 2>&1 | tee ceph_test_neorados_list.log | sed "s/^/ list: /"' 2024-04-14T15:12:54.575 INFO:tasks.workunit.client.0.smithi090.stderr:++ printf %25s ec_list 2024-04-14T15:12:54.575 INFO:tasks.workunit.client.0.smithi090.stderr:+ r=' ec_list' 2024-04-14T15:12:54.575 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -f /etc/bashrc ']' 2024-04-14T15:12:54.575 INFO:tasks.workunit.client.0.smithi090.stderr:+ . /etc/bashrc 2024-04-14T15:12:54.575 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -z '' ']' 2024-04-14T15:12:54.576 INFO:tasks.workunit.client.0.smithi090.stderr:++ BASHRCSOURCED=Y 2024-04-14T15:12:54.576 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.576 INFO:tasks.workunit.client.0.smithi090.stderr:++ shopt -q login_shell 2024-04-14T15:12:54.576 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/locale 2024-04-14T15:12:54.576 INFO:tasks.workunit.client.0.smithi090.stderr:+++ umask 2024-04-14T15:12:54.576 INFO:tasks.workunit.client.0.smithi090.stderr:++ awk '{print $1}' 2024-04-14T15:12:54.576 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'grep=grep --color=auto' 2024-04-14T15:12:54.576 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-14T15:12:54.576 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-14T15:12:54.576 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.576 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-14T15:12:54.576 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.576 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorls.sh 2024-04-14T15:12:54.576 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' '!' -t 0 ']' 2024-04-14T15:12:54.577 INFO:tasks.workunit.client.0.smithi090.stderr:+++ return 2024-04-14T15:12:54.577 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.577 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-14T15:12:54.577 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.577 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-14T15:12:54.577 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export S_COLORS=auto 2024-04-14T15:12:54.577 INFO:tasks.workunit.client.0.smithi090.stderr:+++ S_COLORS=auto 2024-04-14T15:12:54.577 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.577 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-14T15:12:54.577 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.577 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-14T15:12:54.577 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.577 INFO:tasks.workunit.client.0.smithi090.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-14T15:12:54.577 INFO:tasks.workunit.client.0.smithi090.stderr:++++ export LANG=en_US.UTF-8 2024-04-14T15:12:54.578 INFO:tasks.workunit.client.0.smithi090.stderr:++++ LANG=en_US.UTF-8 2024-04-14T15:12:54.578 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.578 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-14T15:12:54.578 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.578 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG=en_US.UTF-8 2024-04-14T15:12:54.578 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup config 2024-04-14T15:12:54.578 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.578 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.578 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' dumb = linux ']' 2024-04-14T15:12:54.578 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.578 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-14T15:12:54.578 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.578 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/less.sh 2024-04-14T15:12:54.578 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-14T15:12:54.579 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.579 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-14T15:12:54.579 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.579 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/which2.sh 2024-04-14T15:12:54.579 INFO:tasks.workunit.client.0.smithi090.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-14T15:12:54.579 INFO:tasks.workunit.client.0.smithi090.stderr:+++++ readlink /proc/94305/exe 2024-04-14T15:12:54.579 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-14T15:12:54.579 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-14T15:12:54.579 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-14T15:12:54.579 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.579 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-14T15:12:54.579 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.579 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-14T15:12:54.579 INFO:tasks.workunit.client.0.smithi090.stderr:++ echo ec_list 2024-04-14T15:12:54.580 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-14T15:12:54.580 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.580 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-14T15:12:54.580 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.580 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/gawk.sh 2024-04-14T15:12:54.580 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-14T15:12:54.580 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-14T15:12:54.580 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-14T15:12:54.580 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.580 INFO:tasks.workunit.client.0.smithi090.stderr:++++ basename /usr/bin/bash 2024-04-14T15:12:54.580 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-14T15:12:54.580 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.580 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.580 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-14T15:12:54.580 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-14T15:12:54.581 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.581 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/lang.sh 2024-04-14T15:12:54.581 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-14T15:12:54.581 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.581 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup 2024-04-14T15:12:54.581 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/locale 2024-04-14T15:12:54.581 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' 0022 -eq 0 ']' 2024-04-14T15:12:54.581 INFO:tasks.workunit.client.0.smithi090.stderr:++ SHELL=/bin/bash 2024-04-14T15:12:54.581 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.581 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-14T15:12:54.581 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.581 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-14T15:12:54.581 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_declare='declare -f' 2024-04-14T15:12:54.581 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_opt=-f 2024-04-14T15:12:54.582 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export which_declare 2024-04-14T15:12:54.582 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-14T15:12:54.582 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export -f which 2024-04-14T15:12:54.582 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.582 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-14T15:12:54.582 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset i 2024-04-14T15:12:54.582 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.582 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset -f pathmunge 2024-04-14T15:12:54.582 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-14T15:12:54.582 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.582 INFO:tasks.workunit.client.0.smithi090.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.582 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.582 INFO:tasks.workunit.client.0.smithi090.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-14T15:12:54.582 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-14T15:12:54.583 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.583 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-14T15:12:54.583 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.583 INFO:tasks.workunit.client.0.smithi090.stderr:+ export PATH 2024-04-14T15:12:54.583 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-14T15:12:54.583 INFO:tasks.workunit.client.0.smithi090.stderr:+ unset rc 2024-04-14T15:12:54.583 INFO:tasks.workunit.client.0.smithi090.stderr:+ tee ceph_test_neorados_io.log 2024-04-14T15:12:54.583 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.583 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-14T15:12:54.583 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.583 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-14T15:12:54.583 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-14T15:12:54.583 INFO:tasks.workunit.client.0.smithi090.stderr:+ ff=ec_list 2024-04-14T15:12:54.583 INFO:tasks.workunit.client.0.smithi090.stderr:+ sed 's/^/ io: /' 2024-04-14T15:12:54.583 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=94456 2024-04-14T15:12:54.584 INFO:tasks.workunit.client.0.smithi090.stdout:test ec_list on pid 94456 2024-04-14T15:12:54.586 INFO:tasks.workunit.client.0.smithi090.stderr:+ echo 'test ec_list on pid 94456' 2024-04-14T15:12:54.586 INFO:tasks.workunit.client.0.smithi090.stderr:+ pids[$f]=94456 2024-04-14T15:12:54.586 INFO:tasks.workunit.client.0.smithi090.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-14T15:12:54.586 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' 1 -eq 1 ']' 2024-04-14T15:12:54.586 INFO:tasks.workunit.client.0.smithi090.stderr:++ printf %25s misc 2024-04-14T15:12:54.586 INFO:tasks.workunit.client.0.smithi090.stderr:+ r=' misc' 2024-04-14T15:12:54.586 INFO:tasks.workunit.client.0.smithi090.stderr:++ echo misc 2024-04-14T15:12:54.586 INFO:tasks.workunit.client.0.smithi090.stderr:++ awk '{print $1}' 2024-04-14T15:12:54.586 INFO:tasks.workunit.client.0.smithi090.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_list 2>&1 | tee ceph_test_neorados_ec_list.log | sed "s/^/ ec_list: /"' 2024-04-14T15:12:54.586 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'grep=grep --color=auto' 2024-04-14T15:12:54.586 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-14T15:12:54.586 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-14T15:12:54.587 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.587 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-14T15:12:54.587 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.587 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorls.sh 2024-04-14T15:12:54.587 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' '!' -t 0 ']' 2024-04-14T15:12:54.587 INFO:tasks.workunit.client.0.smithi090.stderr:+++ return 2024-04-14T15:12:54.587 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.587 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-14T15:12:54.587 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.587 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-14T15:12:54.587 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export S_COLORS=auto 2024-04-14T15:12:54.587 INFO:tasks.workunit.client.0.smithi090.stderr:+++ S_COLORS=auto 2024-04-14T15:12:54.587 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.587 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-14T15:12:54.588 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.588 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-14T15:12:54.588 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.588 INFO:tasks.workunit.client.0.smithi090.stderr:+ ceph_test_neorados_io 2024-04-14T15:12:54.588 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -f /etc/bashrc ']' 2024-04-14T15:12:54.588 INFO:tasks.workunit.client.0.smithi090.stderr:+ . /etc/bashrc 2024-04-14T15:12:54.588 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -z '' ']' 2024-04-14T15:12:54.588 INFO:tasks.workunit.client.0.smithi090.stderr:++ BASHRCSOURCED=Y 2024-04-14T15:12:54.588 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.588 INFO:tasks.workunit.client.0.smithi090.stderr:++ shopt -q login_shell 2024-04-14T15:12:54.589 INFO:tasks.workunit.client.0.smithi090.stdout:test misc on pid 94468 2024-04-14T15:12:54.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:54.590 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:54.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:12:54.596 INFO:tasks.workunit.client.0.smithi090.stderr:+ ff=misc 2024-04-14T15:12:54.596 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=94468 2024-04-14T15:12:54.596 INFO:tasks.workunit.client.0.smithi090.stderr:+ echo 'test misc on pid 94468' 2024-04-14T15:12:54.596 INFO:tasks.workunit.client.0.smithi090.stderr:+ pids[$f]=94468 2024-04-14T15:12:54.596 INFO:tasks.workunit.client.0.smithi090.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-14T15:12:54.596 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' 1 -eq 1 ']' 2024-04-14T15:12:54.596 INFO:tasks.workunit.client.0.smithi090.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_misc 2>&1 | tee ceph_test_neorados_misc.log | sed "s/^/ misc: /"' 2024-04-14T15:12:54.597 INFO:tasks.workunit.client.0.smithi090.stderr:+++ umask 2024-04-14T15:12:54.597 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' 0022 -eq 0 ']' 2024-04-14T15:12:54.597 INFO:tasks.workunit.client.0.smithi090.stderr:++ SHELL=/bin/bash 2024-04-14T15:12:54.597 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.597 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-14T15:12:54.597 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.597 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-14T15:12:54.597 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-14T15:12:54.597 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.597 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-14T15:12:54.597 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.597 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-14T15:12:54.597 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.597 INFO:tasks.workunit.client.0.smithi090.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.598 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.598 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-14T15:12:54.598 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.598 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-14T15:12:54.598 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.598 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-14T15:12:54.598 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-14T15:12:54.598 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-14T15:12:54.598 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.598 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-14T15:12:54.598 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.598 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-14T15:12:54.598 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-14T15:12:54.598 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.599 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.599 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-14T15:12:54.599 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-14T15:12:54.599 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.599 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/gawk.sh 2024-04-14T15:12:54.599 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.599 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-14T15:12:54.599 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-14T15:12:54.599 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.599 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-14T15:12:54.599 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.599 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/lang.sh 2024-04-14T15:12:54.599 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup 2024-04-14T15:12:54.599 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/locale 2024-04-14T15:12:54.599 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-14T15:12:54.600 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-14T15:12:54.600 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-14T15:12:54.600 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-14T15:12:54.600 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.600 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-14T15:12:54.600 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.600 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-14T15:12:54.600 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-14T15:12:54.600 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.600 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.600 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-14T15:12:54.600 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.600 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-14T15:12:54.600 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-14T15:12:54.601 INFO:tasks.workunit.client.0.smithi090.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-14T15:12:54.601 INFO:tasks.workunit.client.0.smithi090.stderr:++++ export LANG=en_US.UTF-8 2024-04-14T15:12:54.601 INFO:tasks.workunit.client.0.smithi090.stderr:++++ LANG=en_US.UTF-8 2024-04-14T15:12:54.601 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.601 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-14T15:12:54.601 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.601 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG=en_US.UTF-8 2024-04-14T15:12:54.601 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup config 2024-04-14T15:12:54.601 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.601 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.601 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' dumb = linux ']' 2024-04-14T15:12:54.601 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.601 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-14T15:12:54.601 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.602 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/less.sh 2024-04-14T15:12:54.602 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-14T15:12:54.602 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.602 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-14T15:12:54.602 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.602 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/which2.sh 2024-04-14T15:12:54.602 INFO:tasks.workunit.client.0.smithi090.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-14T15:12:54.602 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-14T15:12:54.602 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-14T15:12:54.602 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -f /etc/bashrc ']' 2024-04-14T15:12:54.602 INFO:tasks.workunit.client.0.smithi090.stderr:+ . /etc/bashrc 2024-04-14T15:12:54.602 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -z '' ']' 2024-04-14T15:12:54.602 INFO:tasks.workunit.client.0.smithi090.stderr:++ BASHRCSOURCED=Y 2024-04-14T15:12:54.602 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.602 INFO:tasks.workunit.client.0.smithi090.stderr:++ shopt -q login_shell 2024-04-14T15:12:54.603 INFO:tasks.workunit.client.0.smithi090.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-14T15:12:54.603 INFO:tasks.workunit.client.0.smithi090.stderr:++++ export LANG=en_US.UTF-8 2024-04-14T15:12:54.603 INFO:tasks.workunit.client.0.smithi090.stderr:++++ LANG=en_US.UTF-8 2024-04-14T15:12:54.603 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.603 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-14T15:12:54.603 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.603 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG=en_US.UTF-8 2024-04-14T15:12:54.603 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup config 2024-04-14T15:12:54.603 INFO:tasks.workunit.client.0.smithi090.stderr:+++ umask 2024-04-14T15:12:54.603 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.603 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.603 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' dumb = linux ']' 2024-04-14T15:12:54.603 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.603 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-14T15:12:54.603 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.604 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/less.sh 2024-04-14T15:12:54.604 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' 0022 -eq 0 ']' 2024-04-14T15:12:54.604 INFO:tasks.workunit.client.0.smithi090.stderr:++ SHELL=/bin/bash 2024-04-14T15:12:54.604 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-14T15:12:54.604 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.604 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-14T15:12:54.604 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.604 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/which2.sh 2024-04-14T15:12:54.604 INFO:tasks.workunit.client.0.smithi090.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-14T15:12:54.604 INFO:tasks.workunit.client.0.smithi090.stderr:++++ export LANG=en_US.UTF-8 2024-04-14T15:12:54.604 INFO:tasks.workunit.client.0.smithi090.stderr:++++ LANG=en_US.UTF-8 2024-04-14T15:12:54.604 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.604 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-14T15:12:54.604 INFO:tasks.workunit.client.0.smithi090.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-14T15:12:54.604 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.605 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG=en_US.UTF-8 2024-04-14T15:12:54.605 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-14T15:12:54.605 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup config 2024-04-14T15:12:54.605 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-14T15:12:54.605 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.605 INFO:tasks.workunit.client.0.smithi090.stderr:++ printf %25s pool 2024-04-14T15:12:54.605 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-14T15:12:54.605 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.605 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-14T15:12:54.605 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.605 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.605 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.605 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' dumb = linux ']' 2024-04-14T15:12:54.605 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-14T15:12:54.605 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-14T15:12:54.606 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.606 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.606 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-14T15:12:54.606 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-14T15:12:54.606 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.606 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/less.sh 2024-04-14T15:12:54.606 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-14T15:12:54.606 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.606 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-14T15:12:54.606 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.606 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-14T15:12:54.606 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/gawk.sh 2024-04-14T15:12:54.606 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.606 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-14T15:12:54.607 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.607 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/which2.sh 2024-04-14T15:12:54.607 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-14T15:12:54.607 INFO:tasks.workunit.client.0.smithi090.stderr:+ r=' pool' 2024-04-14T15:12:54.607 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.607 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-14T15:12:54.607 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.607 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-14T15:12:54.607 INFO:tasks.workunit.client.0.smithi090.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-14T15:12:54.609 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.609 INFO:tasks.workunit.client.0.smithi090.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.609 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.609 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.609 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-14T15:12:54.609 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-14T15:12:54.609 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.610 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.610 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/lang.sh 2024-04-14T15:12:54.610 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-14T15:12:54.610 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.610 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup 2024-04-14T15:12:54.610 INFO:tasks.workunit.client.0.smithi090.stderr:+++++ readlink /proc/94378/exe 2024-04-14T15:12:54.610 INFO:tasks.workunit.client.0.smithi090.stderr:++ awk '{print $1}' 2024-04-14T15:12:54.610 INFO:tasks.workunit.client.0.smithi090.stderr:++++ basename /usr/bin/bash 2024-04-14T15:12:54.610 INFO:tasks.workunit.client.0.smithi090.stderr:+++++ readlink /proc/94252/exe 2024-04-14T15:12:54.610 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/locale 2024-04-14T15:12:54.610 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_declare='declare -f' 2024-04-14T15:12:54.610 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_opt=-f 2024-04-14T15:12:54.610 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export which_declare 2024-04-14T15:12:54.610 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export -f which 2024-04-14T15:12:54.611 INFO:tasks.workunit.client.0.smithi090.stderr:++ echo pool 2024-04-14T15:12:54.611 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset i 2024-04-14T15:12:54.611 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset -f pathmunge 2024-04-14T15:12:54.611 INFO:tasks.workunit.client.0.smithi090.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-14T15:12:54.611 INFO:tasks.workunit.client.0.smithi090.stderr:+ export PATH 2024-04-14T15:12:54.611 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-14T15:12:54.611 INFO:tasks.workunit.client.0.smithi090.stderr:+ unset rc 2024-04-14T15:12:54.611 INFO:tasks.workunit.client.0.smithi090.stderr:++++ basename /usr/bin/bash 2024-04-14T15:12:54.611 INFO:tasks.workunit.client.0.smithi090.stderr:+ sed 's/^/ ec_io: /' 2024-04-14T15:12:54.611 INFO:tasks.workunit.client.0.smithi090.stderr:+ tee ceph_test_neorados_ec_io.log 2024-04-14T15:12:54.611 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_declare='declare -f' 2024-04-14T15:12:54.611 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_opt=-f 2024-04-14T15:12:54.612 INFO:tasks.workunit.client.0.smithi090.stdout:test pool on pid 94527 2024-04-14T15:12:54.619 INFO:tasks.workunit.client.0.smithi090.stderr:+ ff=pool 2024-04-14T15:12:54.619 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=94527 2024-04-14T15:12:54.619 INFO:tasks.workunit.client.0.smithi090.stderr:+ echo 'test pool on pid 94527' 2024-04-14T15:12:54.619 INFO:tasks.workunit.client.0.smithi090.stderr:+ pids[$f]=94527 2024-04-14T15:12:54.619 INFO:tasks.workunit.client.0.smithi090.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-14T15:12:54.619 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' 1 -eq 1 ']' 2024-04-14T15:12:54.619 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export which_declare 2024-04-14T15:12:54.619 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export -f which 2024-04-14T15:12:54.619 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset i 2024-04-14T15:12:54.620 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset -f pathmunge 2024-04-14T15:12:54.620 INFO:tasks.workunit.client.0.smithi090.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-14T15:12:54.620 INFO:tasks.workunit.client.0.smithi090.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_pool 2>&1 | tee ceph_test_neorados_pool.log | sed "s/^/ pool: /"' 2024-04-14T15:12:54.620 INFO:tasks.workunit.client.0.smithi090.stderr:+ export PATH 2024-04-14T15:12:54.620 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-14T15:12:54.620 INFO:tasks.workunit.client.0.smithi090.stderr:+ unset rc 2024-04-14T15:12:54.620 INFO:tasks.workunit.client.0.smithi090.stderr:+ sed 's/^/ handler_error: /' 2024-04-14T15:12:54.620 INFO:tasks.workunit.client.0.smithi090.stderr:+ tee ceph_test_neorados_handler_error.log 2024-04-14T15:12:54.620 INFO:tasks.workunit.client.0.smithi090.stderr:+ ceph_test_neorados_handler_error 2024-04-14T15:12:54.620 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -f /etc/bashrc ']' 2024-04-14T15:12:54.620 INFO:tasks.workunit.client.0.smithi090.stderr:+ . /etc/bashrc 2024-04-14T15:12:54.620 INFO:tasks.workunit.client.0.smithi090.stderr:+ ceph_test_neorados_ec_io 2024-04-14T15:12:54.620 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -z '' ']' 2024-04-14T15:12:54.620 INFO:tasks.workunit.client.0.smithi090.stderr:+++++ readlink /proc/94168/exe 2024-04-14T15:12:54.621 INFO:tasks.workunit.client.0.smithi090.stderr:++ BASHRCSOURCED=Y 2024-04-14T15:12:54.621 INFO:tasks.workunit.client.0.smithi090.stderr:++++ basename /usr/bin/bash 2024-04-14T15:12:54.621 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.621 INFO:tasks.workunit.client.0.smithi090.stderr:++ printf %25s read_operations 2024-04-14T15:12:54.621 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-14T15:12:54.621 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.621 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-14T15:12:54.621 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-14T15:12:54.621 INFO:tasks.workunit.client.0.smithi090.stderr:+ r=' read_operations' 2024-04-14T15:12:54.621 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_declare='declare -f' 2024-04-14T15:12:54.621 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_opt=-f 2024-04-14T15:12:54.621 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export which_declare 2024-04-14T15:12:54.621 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export -f which 2024-04-14T15:12:54.621 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset i 2024-04-14T15:12:54.622 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset -f pathmunge 2024-04-14T15:12:54.622 INFO:tasks.workunit.client.0.smithi090.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-14T15:12:54.622 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.622 INFO:tasks.workunit.client.0.smithi090.stderr:++ shopt -q login_shell 2024-04-14T15:12:54.622 INFO:tasks.workunit.client.0.smithi090.stderr:+ export PATH 2024-04-14T15:12:54.622 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-14T15:12:54.622 INFO:tasks.workunit.client.0.smithi090.stderr:+ unset rc 2024-04-14T15:12:54.622 INFO:tasks.workunit.client.0.smithi090.stderr:+++ umask 2024-04-14T15:12:54.622 INFO:tasks.workunit.client.0.smithi090.stderr:+ ceph_test_neorados_cmd 2024-04-14T15:12:54.622 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' 0022 -eq 0 ']' 2024-04-14T15:12:54.622 INFO:tasks.workunit.client.0.smithi090.stderr:++ SHELL=/bin/bash 2024-04-14T15:12:54.622 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'grep=grep --color=auto' 2024-04-14T15:12:54.622 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-14T15:12:54.623 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'grep=grep --color=auto' 2024-04-14T15:12:54.623 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-14T15:12:54.623 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.623 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-14T15:12:54.623 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-14T15:12:54.623 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.623 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.623 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-14T15:12:54.623 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-14T15:12:54.623 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-14T15:12:54.623 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.623 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorls.sh 2024-04-14T15:12:54.623 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.623 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-14T15:12:54.624 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.624 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorls.sh 2024-04-14T15:12:54.624 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' '!' -t 0 ']' 2024-04-14T15:12:54.624 INFO:tasks.workunit.client.0.smithi090.stderr:+++ return 2024-04-14T15:12:54.624 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.624 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-14T15:12:54.624 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' '!' -t 0 ']' 2024-04-14T15:12:54.624 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.624 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-14T15:12:54.624 INFO:tasks.workunit.client.0.smithi090.stderr:+++ return 2024-04-14T15:12:54.624 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-14T15:12:54.624 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.624 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.624 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-14T15:12:54.625 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-14T15:12:54.625 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.625 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.625 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-14T15:12:54.625 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export S_COLORS=auto 2024-04-14T15:12:54.625 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-14T15:12:54.625 INFO:tasks.workunit.client.0.smithi090.stderr:+++ S_COLORS=auto 2024-04-14T15:12:54.625 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.625 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-14T15:12:54.625 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export S_COLORS=auto 2024-04-14T15:12:54.625 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.625 INFO:tasks.workunit.client.0.smithi090.stderr:+++ S_COLORS=auto 2024-04-14T15:12:54.625 INFO:tasks.workunit.client.0.smithi090.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.626 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.626 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-14T15:12:54.626 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.626 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.626 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-14T15:12:54.626 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-14T15:12:54.626 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.626 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-14T15:12:54.626 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.626 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-14T15:12:54.626 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.626 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.626 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.626 INFO:tasks.workunit.client.0.smithi090.stderr:+ tee ceph_test_neorados_cmd.log 2024-04-14T15:12:54.627 INFO:tasks.workunit.client.0.smithi090.stderr:+ sed 's/^/ cmd: /' 2024-04-14T15:12:54.627 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-14T15:12:54.627 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-14T15:12:54.627 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-14T15:12:54.627 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.627 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-14T15:12:54.627 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.627 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-14T15:12:54.627 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-14T15:12:54.627 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.627 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-14T15:12:54.627 INFO:tasks.workunit.client.0.smithi090.stderr:++ echo read_operations 2024-04-14T15:12:54.627 INFO:tasks.workunit.client.0.smithi090.stderr:++ awk '{print $1}' 2024-04-14T15:12:54.627 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-14T15:12:54.628 INFO:tasks.workunit.client.0.smithi090.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-14T15:12:54.628 INFO:tasks.workunit.client.0.smithi090.stderr:++++ export LANG=en_US.UTF-8 2024-04-14T15:12:54.628 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-14T15:12:54.628 INFO:tasks.workunit.client.0.smithi090.stderr:++++ LANG=en_US.UTF-8 2024-04-14T15:12:54.628 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-14T15:12:54.628 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.628 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.628 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-14T15:12:54.628 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-14T15:12:54.628 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.628 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-14T15:12:54.628 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.628 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG=en_US.UTF-8 2024-04-14T15:12:54.628 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup config 2024-04-14T15:12:54.629 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.629 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-14T15:12:54.629 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.629 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-14T15:12:54.629 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.629 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/gawk.sh 2024-04-14T15:12:54.629 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.629 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' dumb = linux ']' 2024-04-14T15:12:54.629 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.629 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-14T15:12:54.629 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.629 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/less.sh 2024-04-14T15:12:54.629 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.629 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-14T15:12:54.630 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.630 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/lang.sh 2024-04-14T15:12:54.630 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-14T15:12:54.630 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.630 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-14T15:12:54.630 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup 2024-04-14T15:12:54.630 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.630 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/which2.sh 2024-04-14T15:12:54.630 INFO:tasks.workunit.client.0.smithi090.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-14T15:12:54.630 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/locale 2024-04-14T15:12:54.630 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'grep=grep --color=auto' 2024-04-14T15:12:54.630 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-14T15:12:54.630 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-14T15:12:54.631 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.631 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-14T15:12:54.631 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.631 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorls.sh 2024-04-14T15:12:54.631 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' '!' -t 0 ']' 2024-04-14T15:12:54.631 INFO:tasks.workunit.client.0.smithi090.stderr:+++ return 2024-04-14T15:12:54.631 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.631 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-14T15:12:54.631 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.631 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-14T15:12:54.631 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export S_COLORS=auto 2024-04-14T15:12:54.631 INFO:tasks.workunit.client.0.smithi090.stderr:+++ S_COLORS=auto 2024-04-14T15:12:54.631 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.632 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-14T15:12:54.632 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.632 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-14T15:12:54.632 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.632 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-14T15:12:54.632 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-14T15:12:54.632 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-14T15:12:54.632 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.632 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-14T15:12:54.632 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.632 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-14T15:12:54.632 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-14T15:12:54.632 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.632 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.633 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-14T15:12:54.633 INFO:tasks.workunit.client.0.smithi090.stderr:+++++ readlink /proc/94418/exe 2024-04-14T15:12:54.633 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.633 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-14T15:12:54.633 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-14T15:12:54.633 INFO:tasks.workunit.client.0.smithi090.stderr:++++ basename /usr/bin/bash 2024-04-14T15:12:54.633 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_declare='declare -f' 2024-04-14T15:12:54.633 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_opt=-f 2024-04-14T15:12:54.633 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export which_declare 2024-04-14T15:12:54.633 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export -f which 2024-04-14T15:12:54.633 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset i 2024-04-14T15:12:54.633 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset -f pathmunge 2024-04-14T15:12:54.633 INFO:tasks.workunit.client.0.smithi090.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-14T15:12:54.634 INFO:tasks.workunit.client.0.smithi090.stderr:+ export PATH 2024-04-14T15:12:54.634 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-14T15:12:54.634 INFO:tasks.workunit.client.0.smithi090.stderr:+ unset rc 2024-04-14T15:12:54.634 INFO:tasks.workunit.client.0.smithi090.stderr:+ sed 's/^/ list: /' 2024-04-14T15:12:54.634 INFO:tasks.workunit.client.0.smithi090.stdout:test read_operations on pid 94615 2024-04-14T15:12:54.641 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-14T15:12:54.641 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-14T15:12:54.641 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-14T15:12:54.641 INFO:tasks.workunit.client.0.smithi090.stderr:+ ceph_test_neorados_list 2024-04-14T15:12:54.641 INFO:tasks.workunit.client.0.smithi090.stderr:+ ff=read_operations 2024-04-14T15:12:54.641 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=94615 2024-04-14T15:12:54.641 INFO:tasks.workunit.client.0.smithi090.stderr:+ echo 'test read_operations on pid 94615' 2024-04-14T15:12:54.641 INFO:tasks.workunit.client.0.smithi090.stderr:+ pids[$f]=94615 2024-04-14T15:12:54.642 INFO:tasks.workunit.client.0.smithi090.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-14T15:12:54.642 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' 1 -eq 1 ']' 2024-04-14T15:12:54.642 INFO:tasks.workunit.client.0.smithi090.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_read_operations 2>&1 | tee ceph_test_neorados_read_operations.log | sed "s/^/ read_operations: /"' 2024-04-14T15:12:54.642 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-14T15:12:54.642 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.642 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-14T15:12:54.642 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.642 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-14T15:12:54.642 INFO:tasks.workunit.client.0.smithi090.stderr:+ tee ceph_test_neorados_list.log 2024-04-14T15:12:54.642 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-14T15:12:54.642 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.642 INFO:tasks.workunit.client.0.smithi090.stderr:++ printf %25s snapshots 2024-04-14T15:12:54.642 INFO:tasks.workunit.client.0.smithi090.stderr:+ r=' snapshots' 2024-04-14T15:12:54.642 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-14T15:12:54.643 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-14T15:12:54.643 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-14T15:12:54.643 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.643 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-14T15:12:54.643 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.643 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-14T15:12:54.643 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-14T15:12:54.643 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.643 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-14T15:12:54.643 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.643 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/gawk.sh 2024-04-14T15:12:54.643 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.643 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-14T15:12:54.643 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.644 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/lang.sh 2024-04-14T15:12:54.644 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup 2024-04-14T15:12:54.644 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -f /etc/bashrc ']' 2024-04-14T15:12:54.644 INFO:tasks.workunit.client.0.smithi090.stderr:+ . /etc/bashrc 2024-04-14T15:12:54.644 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -z '' ']' 2024-04-14T15:12:54.644 INFO:tasks.workunit.client.0.smithi090.stderr:++ BASHRCSOURCED=Y 2024-04-14T15:12:54.644 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.644 INFO:tasks.workunit.client.0.smithi090.stderr:++ shopt -q login_shell 2024-04-14T15:12:54.644 INFO:tasks.workunit.client.0.smithi090.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-14T15:12:54.644 INFO:tasks.workunit.client.0.smithi090.stderr:++++ export LANG=en_US.UTF-8 2024-04-14T15:12:54.644 INFO:tasks.workunit.client.0.smithi090.stderr:++++ LANG=en_US.UTF-8 2024-04-14T15:12:54.644 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.644 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-14T15:12:54.644 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.645 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG=en_US.UTF-8 2024-04-14T15:12:54.645 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup config 2024-04-14T15:12:54.645 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.645 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.645 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' dumb = linux ']' 2024-04-14T15:12:54.645 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.645 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-14T15:12:54.645 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.645 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/less.sh 2024-04-14T15:12:54.645 INFO:tasks.workunit.client.0.smithi090.stderr:+++ umask 2024-04-14T15:12:54.645 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-14T15:12:54.645 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.645 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-14T15:12:54.646 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.646 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/which2.sh 2024-04-14T15:12:54.646 INFO:tasks.workunit.client.0.smithi090.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-14T15:12:54.646 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' 0022 -eq 0 ']' 2024-04-14T15:12:54.646 INFO:tasks.workunit.client.0.smithi090.stderr:++ SHELL=/bin/bash 2024-04-14T15:12:54.646 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.646 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-14T15:12:54.646 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.646 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-14T15:12:54.646 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-14T15:12:54.646 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.646 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/locale 2024-04-14T15:12:54.646 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-14T15:12:54.647 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.647 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-14T15:12:54.647 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.647 INFO:tasks.workunit.client.0.smithi090.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.647 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.647 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-14T15:12:54.647 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.647 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-14T15:12:54.647 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.647 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-14T15:12:54.647 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-14T15:12:54.647 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-14T15:12:54.647 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.647 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-14T15:12:54.648 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.648 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-14T15:12:54.648 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-14T15:12:54.648 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.648 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-14T15:12:54.648 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.648 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/gawk.sh 2024-04-14T15:12:54.648 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.648 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-14T15:12:54.648 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.648 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/lang.sh 2024-04-14T15:12:54.648 INFO:tasks.workunit.client.0.smithi090.stderr:+++++ readlink /proc/94468/exe 2024-04-14T15:12:54.648 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup 2024-04-14T15:12:54.648 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.648 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-14T15:12:54.649 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.649 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-14T15:12:54.649 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-14T15:12:54.649 INFO:tasks.workunit.client.0.smithi090.stderr:++++ basename /usr/bin/bash 2024-04-14T15:12:54.649 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_declare='declare -f' 2024-04-14T15:12:54.649 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_opt=-f 2024-04-14T15:12:54.649 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export which_declare 2024-04-14T15:12:54.649 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export -f which 2024-04-14T15:12:54.649 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset i 2024-04-14T15:12:54.649 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset -f pathmunge 2024-04-14T15:12:54.649 INFO:tasks.workunit.client.0.smithi090.stderr:++ awk '{print $1}' 2024-04-14T15:12:54.649 INFO:tasks.workunit.client.0.smithi090.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-14T15:12:54.649 INFO:tasks.workunit.client.0.smithi090.stderr:+ export PATH 2024-04-14T15:12:54.649 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-14T15:12:54.650 INFO:tasks.workunit.client.0.smithi090.stderr:+ unset rc 2024-04-14T15:12:54.650 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'grep=grep --color=auto' 2024-04-14T15:12:54.650 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-14T15:12:54.650 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-14T15:12:54.650 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.650 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-14T15:12:54.650 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.650 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorls.sh 2024-04-14T15:12:54.650 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' '!' -t 0 ']' 2024-04-14T15:12:54.650 INFO:tasks.workunit.client.0.smithi090.stderr:+++ return 2024-04-14T15:12:54.650 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.650 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-14T15:12:54.650 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.650 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-14T15:12:54.650 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export S_COLORS=auto 2024-04-14T15:12:54.651 INFO:tasks.workunit.client.0.smithi090.stderr:+++ S_COLORS=auto 2024-04-14T15:12:54.651 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.651 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-14T15:12:54.651 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.651 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-14T15:12:54.651 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.651 INFO:tasks.workunit.client.0.smithi090.stderr:+ tee ceph_test_neorados_misc.log 2024-04-14T15:12:54.651 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/locale 2024-04-14T15:12:54.651 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-14T15:12:54.651 INFO:tasks.workunit.client.0.smithi090.stderr:+ sed 's/^/ misc: /' 2024-04-14T15:12:54.651 INFO:tasks.workunit.client.0.smithi090.stderr:+ ceph_test_neorados_misc 2024-04-14T15:12:54.651 INFO:tasks.workunit.client.0.smithi090.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-14T15:12:54.651 INFO:tasks.workunit.client.0.smithi090.stderr:++++ export LANG=en_US.UTF-8 2024-04-14T15:12:54.651 INFO:tasks.workunit.client.0.smithi090.stderr:++++ LANG=en_US.UTF-8 2024-04-14T15:12:54.651 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.652 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-14T15:12:54.652 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.652 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG=en_US.UTF-8 2024-04-14T15:12:54.652 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup config 2024-04-14T15:12:54.652 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.652 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.652 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' dumb = linux ']' 2024-04-14T15:12:54.652 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.652 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-14T15:12:54.652 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.652 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/less.sh 2024-04-14T15:12:54.652 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-14T15:12:54.652 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.652 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-14T15:12:54.653 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.653 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/which2.sh 2024-04-14T15:12:54.653 INFO:tasks.workunit.client.0.smithi090.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-14T15:12:54.653 INFO:tasks.workunit.client.0.smithi090.stderr:+++++ readlink /proc/94456/exe 2024-04-14T15:12:54.653 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.653 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-14T15:12:54.653 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.653 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-14T15:12:54.653 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-14T15:12:54.653 INFO:tasks.workunit.client.0.smithi090.stderr:++++ basename /usr/bin/bash 2024-04-14T15:12:54.653 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-14T15:12:54.653 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-14T15:12:54.653 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-14T15:12:54.653 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.653 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-14T15:12:54.654 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.654 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-14T15:12:54.654 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-14T15:12:54.654 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.654 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_declare='declare -f' 2024-04-14T15:12:54.654 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_opt=-f 2024-04-14T15:12:54.654 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export which_declare 2024-04-14T15:12:54.654 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export -f which 2024-04-14T15:12:54.654 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset i 2024-04-14T15:12:54.654 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset -f pathmunge 2024-04-14T15:12:54.654 INFO:tasks.workunit.client.0.smithi090.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-14T15:12:54.654 INFO:tasks.workunit.client.0.smithi090.stderr:+ export PATH 2024-04-14T15:12:54.654 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-14T15:12:54.654 INFO:tasks.workunit.client.0.smithi090.stderr:+ unset rc 2024-04-14T15:12:54.655 INFO:tasks.workunit.client.0.smithi090.stderr:+ ceph_test_neorados_ec_list 2024-04-14T15:12:54.655 INFO:tasks.workunit.client.0.smithi090.stderr:+ sed 's/^/ ec_list: /' 2024-04-14T15:12:54.655 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-14T15:12:54.655 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-14T15:12:54.655 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-14T15:12:54.655 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.655 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-14T15:12:54.655 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.655 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-14T15:12:54.655 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-14T15:12:54.655 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.655 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-14T15:12:54.655 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.655 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/gawk.sh 2024-04-14T15:12:54.655 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.655 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-14T15:12:54.656 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.656 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/lang.sh 2024-04-14T15:12:54.656 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup 2024-04-14T15:12:54.656 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/locale 2024-04-14T15:12:54.656 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-14T15:12:54.656 INFO:tasks.workunit.client.0.smithi090.stderr:++ echo snapshots 2024-04-14T15:12:54.656 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.656 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-14T15:12:54.656 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.656 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-14T15:12:54.656 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-14T15:12:54.656 INFO:tasks.workunit.client.0.smithi090.stderr:+ ff=snapshots 2024-04-14T15:12:54.657 INFO:tasks.workunit.client.0.smithi090.stdout:test snapshots on pid 94680 2024-04-14T15:12:54.659 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=94680 2024-04-14T15:12:54.659 INFO:tasks.workunit.client.0.smithi090.stderr:+ echo 'test snapshots on pid 94680' 2024-04-14T15:12:54.659 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-14T15:12:54.659 INFO:tasks.workunit.client.0.smithi090.stderr:+ pids[$f]=94680 2024-04-14T15:12:54.659 INFO:tasks.workunit.client.0.smithi090.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-14T15:12:54.659 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' 1 -eq 1 ']' 2024-04-14T15:12:54.659 INFO:tasks.workunit.client.0.smithi090.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_snapshots 2>&1 | tee ceph_test_neorados_snapshots.log | sed "s/^/ snapshots: /"' 2024-04-14T15:12:54.659 INFO:tasks.workunit.client.0.smithi090.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-14T15:12:54.659 INFO:tasks.workunit.client.0.smithi090.stderr:++++ export LANG=en_US.UTF-8 2024-04-14T15:12:54.659 INFO:tasks.workunit.client.0.smithi090.stderr:++++ LANG=en_US.UTF-8 2024-04-14T15:12:54.659 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.659 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-14T15:12:54.660 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.660 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG=en_US.UTF-8 2024-04-14T15:12:54.660 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup config 2024-04-14T15:12:54.660 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.660 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.660 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' dumb = linux ']' 2024-04-14T15:12:54.660 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.660 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-14T15:12:54.660 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.660 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/less.sh 2024-04-14T15:12:54.660 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-14T15:12:54.660 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.660 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-14T15:12:54.660 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.660 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/which2.sh 2024-04-14T15:12:54.661 INFO:tasks.workunit.client.0.smithi090.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-14T15:12:54.661 INFO:tasks.workunit.client.0.smithi090.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-14T15:12:54.661 INFO:tasks.workunit.client.0.smithi090.stderr:++++ export LANG=en_US.UTF-8 2024-04-14T15:12:54.661 INFO:tasks.workunit.client.0.smithi090.stderr:++++ LANG=en_US.UTF-8 2024-04-14T15:12:54.661 INFO:tasks.workunit.client.0.smithi090.stderr:++ printf %25s watch_notify 2024-04-14T15:12:54.661 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.661 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-14T15:12:54.661 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.661 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG=en_US.UTF-8 2024-04-14T15:12:54.661 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup config 2024-04-14T15:12:54.661 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -f /etc/bashrc ']' 2024-04-14T15:12:54.661 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.661 INFO:tasks.workunit.client.0.smithi090.stderr:+ . /etc/bashrc 2024-04-14T15:12:54.661 INFO:tasks.workunit.client.0.smithi090.stderr:+ r=' watch_notify' 2024-04-14T15:12:54.661 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.662 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' dumb = linux ']' 2024-04-14T15:12:54.662 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.662 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-14T15:12:54.662 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.662 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/less.sh 2024-04-14T15:12:54.662 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -z '' ']' 2024-04-14T15:12:54.662 INFO:tasks.workunit.client.0.smithi090.stderr:++ BASHRCSOURCED=Y 2024-04-14T15:12:54.662 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.662 INFO:tasks.workunit.client.0.smithi090.stderr:++ shopt -q login_shell 2024-04-14T15:12:54.662 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-14T15:12:54.662 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.662 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-14T15:12:54.662 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.662 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/which2.sh 2024-04-14T15:12:54.663 INFO:tasks.workunit.client.0.smithi090.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-14T15:12:54.663 INFO:tasks.workunit.client.0.smithi090.stderr:+++ umask 2024-04-14T15:12:54.663 INFO:tasks.workunit.client.0.smithi090.stderr:++ echo watch_notify 2024-04-14T15:12:54.663 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' 0022 -eq 0 ']' 2024-04-14T15:12:54.663 INFO:tasks.workunit.client.0.smithi090.stderr:++ SHELL=/bin/bash 2024-04-14T15:12:54.663 INFO:tasks.workunit.client.0.smithi090.stderr:++ awk '{print $1}' 2024-04-14T15:12:54.663 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.663 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-14T15:12:54.663 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.663 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-14T15:12:54.663 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-14T15:12:54.663 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.663 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-14T15:12:54.663 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.663 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-14T15:12:54.664 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.664 INFO:tasks.workunit.client.0.smithi090.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.664 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.664 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-14T15:12:54.664 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.664 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-14T15:12:54.664 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.664 INFO:tasks.workunit.client.0.smithi090.stderr:+ tee ceph_test_neorados_ec_list.log 2024-04-14T15:12:54.664 INFO:tasks.workunit.client.0.smithi090.stderr:+++++ readlink /proc/94615/exe 2024-04-14T15:12:54.664 INFO:tasks.workunit.client.0.smithi090.stderr:++++ basename /usr/bin/bash 2024-04-14T15:12:54.664 INFO:tasks.workunit.client.0.smithi090.stderr:+++++ readlink /proc/94527/exe 2024-04-14T15:12:54.664 INFO:tasks.workunit.client.0.smithi090.stdout:test watch_notify on pid 94701 2024-04-14T15:12:54.667 INFO:tasks.workunit.client.0.smithi090.stderr:+ ff=watch_notify 2024-04-14T15:12:54.668 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=94701 2024-04-14T15:12:54.668 INFO:tasks.workunit.client.0.smithi090.stderr:+ echo 'test watch_notify on pid 94701' 2024-04-14T15:12:54.668 INFO:tasks.workunit.client.0.smithi090.stderr:+ pids[$f]=94701 2024-04-14T15:12:54.668 INFO:tasks.workunit.client.0.smithi090.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-14T15:12:54.668 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' 1 -eq 1 ']' 2024-04-14T15:12:54.668 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_declare='declare -f' 2024-04-14T15:12:54.668 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_opt=-f 2024-04-14T15:12:54.668 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export which_declare 2024-04-14T15:12:54.668 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export -f which 2024-04-14T15:12:54.668 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset i 2024-04-14T15:12:54.668 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset -f pathmunge 2024-04-14T15:12:54.668 INFO:tasks.workunit.client.0.smithi090.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-14T15:12:54.668 INFO:tasks.workunit.client.0.smithi090.stderr:++++ basename /usr/bin/bash 2024-04-14T15:12:54.669 INFO:tasks.workunit.client.0.smithi090.stderr:+ export PATH 2024-04-14T15:12:54.669 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-14T15:12:54.669 INFO:tasks.workunit.client.0.smithi090.stderr:+ unset rc 2024-04-14T15:12:54.669 INFO:tasks.workunit.client.0.smithi090.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_watch_notify 2>&1 | tee ceph_test_neorados_watch_notify.log | sed "s/^/ watch_notify: /"' 2024-04-14T15:12:54.669 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_declare='declare -f' 2024-04-14T15:12:54.669 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_opt=-f 2024-04-14T15:12:54.669 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export which_declare 2024-04-14T15:12:54.669 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export -f which 2024-04-14T15:12:54.669 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset i 2024-04-14T15:12:54.669 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset -f pathmunge 2024-04-14T15:12:54.669 INFO:tasks.workunit.client.0.smithi090.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-14T15:12:54.669 INFO:tasks.workunit.client.0.smithi090.stderr:+ export PATH 2024-04-14T15:12:54.669 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-14T15:12:54.669 INFO:tasks.workunit.client.0.smithi090.stderr:+ unset rc 2024-04-14T15:12:54.669 INFO:tasks.workunit.client.0.smithi090.stderr:+ tee ceph_test_neorados_pool.log 2024-04-14T15:12:54.670 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -f /etc/bashrc ']' 2024-04-14T15:12:54.670 INFO:tasks.workunit.client.0.smithi090.stderr:+ . /etc/bashrc 2024-04-14T15:12:54.670 INFO:tasks.workunit.client.0.smithi090.stderr:+ ceph_test_neorados_read_operations 2024-04-14T15:12:54.670 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -z '' ']' 2024-04-14T15:12:54.670 INFO:tasks.workunit.client.0.smithi090.stderr:++ BASHRCSOURCED=Y 2024-04-14T15:12:54.670 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.670 INFO:tasks.workunit.client.0.smithi090.stderr:++ shopt -q login_shell 2024-04-14T15:12:54.670 INFO:tasks.workunit.client.0.smithi090.stderr:+++ umask 2024-04-14T15:12:54.670 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' 0022 -eq 0 ']' 2024-04-14T15:12:54.670 INFO:tasks.workunit.client.0.smithi090.stderr:++ SHELL=/bin/bash 2024-04-14T15:12:54.670 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.670 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-14T15:12:54.670 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.670 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-14T15:12:54.671 INFO:tasks.workunit.client.0.smithi090.stderr:++ printf %25s write_operations 2024-04-14T15:12:54.671 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-14T15:12:54.671 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.671 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-14T15:12:54.671 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.671 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-14T15:12:54.671 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.671 INFO:tasks.workunit.client.0.smithi090.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.671 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.671 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-14T15:12:54.671 INFO:tasks.workunit.client.0.smithi090.stderr:+ tee ceph_test_neorados_read_operations.log 2024-04-14T15:12:54.671 INFO:tasks.workunit.client.0.smithi090.stderr:+ r=' write_operations' 2024-04-14T15:12:54.671 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.671 INFO:tasks.workunit.client.0.smithi090.stderr:+ sed 's/^/ pool: /' 2024-04-14T15:12:54.671 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-14T15:12:54.672 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.672 INFO:tasks.workunit.client.0.smithi090.stderr:+ sed 's/^/ read_operations: /' 2024-04-14T15:12:54.672 INFO:tasks.workunit.client.0.smithi090.stderr:+ ceph_test_neorados_pool 2024-04-14T15:12:54.672 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'grep=grep --color=auto' 2024-04-14T15:12:54.672 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-14T15:12:54.672 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-14T15:12:54.672 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.672 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-14T15:12:54.672 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.672 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorls.sh 2024-04-14T15:12:54.672 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' '!' -t 0 ']' 2024-04-14T15:12:54.672 INFO:tasks.workunit.client.0.smithi090.stderr:+++ return 2024-04-14T15:12:54.672 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.672 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-14T15:12:54.672 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.673 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-14T15:12:54.673 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export S_COLORS=auto 2024-04-14T15:12:54.673 INFO:tasks.workunit.client.0.smithi090.stderr:+++ S_COLORS=auto 2024-04-14T15:12:54.673 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.673 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-14T15:12:54.673 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.673 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-14T15:12:54.673 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.673 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-14T15:12:54.673 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-14T15:12:54.673 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-14T15:12:54.673 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.673 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-14T15:12:54.673 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.673 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-14T15:12:54.673 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-14T15:12:54.674 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.674 INFO:tasks.workunit.client.0.smithi090.stderr:++ echo write_operations 2024-04-14T15:12:54.674 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-14T15:12:54.674 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-14T15:12:54.674 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-14T15:12:54.674 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.674 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-14T15:12:54.674 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.674 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-14T15:12:54.674 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-14T15:12:54.674 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.674 INFO:tasks.workunit.client.0.smithi090.stderr:++ awk '{print $1}' 2024-04-14T15:12:54.674 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-14T15:12:54.675 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.675 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/gawk.sh 2024-04-14T15:12:54.675 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.675 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-14T15:12:54.675 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.675 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/lang.sh 2024-04-14T15:12:54.675 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup 2024-04-14T15:12:54.675 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/locale 2024-04-14T15:12:54.675 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.675 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-14T15:12:54.675 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.675 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-14T15:12:54.675 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-14T15:12:54.675 INFO:tasks.workunit.client.0.smithi090.stderr:+ ff=write_operations 2024-04-14T15:12:54.675 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=94756 2024-04-14T15:12:54.676 INFO:tasks.workunit.client.0.smithi090.stderr:+ echo 'test write_operations on pid 94756' 2024-04-14T15:12:54.676 INFO:tasks.workunit.client.0.smithi090.stdout:test write_operations on pid 94756 2024-04-14T15:12:54.682 INFO:tasks.workunit.client.0.smithi090.stderr:+ pids[$f]=94756 2024-04-14T15:12:54.682 INFO:tasks.workunit.client.0.smithi090.stderr:+ ret=0 2024-04-14T15:12:54.682 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' 1 -eq 1 ']' 2024-04-14T15:12:54.682 INFO:tasks.workunit.client.0.smithi090.stderr:+ for t in "${!pids[@]}" 2024-04-14T15:12:54.683 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-14T15:12:54.683 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=93452 2024-04-14T15:12:54.683 INFO:tasks.workunit.client.0.smithi090.stderr:+ wait 93452 2024-04-14T15:12:54.683 INFO:tasks.workunit.client.0.smithi090.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_write_operations 2>&1 | tee ceph_test_neorados_write_operations.log | sed "s/^/ write_operations: /"' 2024-04-14T15:12:54.683 INFO:tasks.workunit.client.0.smithi090.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-14T15:12:54.683 INFO:tasks.workunit.client.0.smithi090.stderr:++++ export LANG=en_US.UTF-8 2024-04-14T15:12:54.683 INFO:tasks.workunit.client.0.smithi090.stderr:++++ LANG=en_US.UTF-8 2024-04-14T15:12:54.683 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.683 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-14T15:12:54.683 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.683 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG=en_US.UTF-8 2024-04-14T15:12:54.683 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup config 2024-04-14T15:12:54.683 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.683 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.684 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' dumb = linux ']' 2024-04-14T15:12:54.684 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.684 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-14T15:12:54.684 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.684 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/less.sh 2024-04-14T15:12:54.684 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-14T15:12:54.684 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.684 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-14T15:12:54.684 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.684 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/which2.sh 2024-04-14T15:12:54.684 INFO:tasks.workunit.client.0.smithi090.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-14T15:12:54.684 INFO:tasks.workunit.client.0.smithi090.stderr:+++++ readlink /proc/94680/exe 2024-04-14T15:12:54.684 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -f /etc/bashrc ']' 2024-04-14T15:12:54.684 INFO:tasks.workunit.client.0.smithi090.stderr:+ . /etc/bashrc 2024-04-14T15:12:54.684 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -z '' ']' 2024-04-14T15:12:54.685 INFO:tasks.workunit.client.0.smithi090.stderr:++ BASHRCSOURCED=Y 2024-04-14T15:12:54.685 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'grep=grep --color=auto' 2024-04-14T15:12:54.685 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-14T15:12:54.685 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.685 INFO:tasks.workunit.client.0.smithi090.stderr:++ shopt -q login_shell 2024-04-14T15:12:54.685 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-14T15:12:54.685 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.685 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-14T15:12:54.685 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.685 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorls.sh 2024-04-14T15:12:54.685 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' '!' -t 0 ']' 2024-04-14T15:12:54.685 INFO:tasks.workunit.client.0.smithi090.stderr:+++ return 2024-04-14T15:12:54.685 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.685 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-14T15:12:54.685 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.686 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-14T15:12:54.686 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export S_COLORS=auto 2024-04-14T15:12:54.686 INFO:tasks.workunit.client.0.smithi090.stderr:+++ S_COLORS=auto 2024-04-14T15:12:54.686 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.686 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-14T15:12:54.686 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.686 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-14T15:12:54.686 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.686 INFO:tasks.workunit.client.0.smithi090.stderr:+++ umask 2024-04-14T15:12:54.686 INFO:tasks.workunit.client.0.smithi090.stderr:++++ basename /usr/bin/bash 2024-04-14T15:12:54.686 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' 0022 -eq 0 ']' 2024-04-14T15:12:54.686 INFO:tasks.workunit.client.0.smithi090.stderr:++ SHELL=/bin/bash 2024-04-14T15:12:54.686 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.687 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-14T15:12:54.687 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.687 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-14T15:12:54.687 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-14T15:12:54.687 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.687 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-14T15:12:54.687 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.687 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-14T15:12:54.687 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.687 INFO:tasks.workunit.client.0.smithi090.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-14T15:12:54.687 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.687 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-14T15:12:54.687 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.688 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-14T15:12:54.688 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.688 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_declare='declare -f' 2024-04-14T15:12:54.688 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_opt=-f 2024-04-14T15:12:54.688 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export which_declare 2024-04-14T15:12:54.688 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export -f which 2024-04-14T15:12:54.688 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset i 2024-04-14T15:12:54.688 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset -f pathmunge 2024-04-14T15:12:54.688 INFO:tasks.workunit.client.0.smithi090.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-14T15:12:54.688 INFO:tasks.workunit.client.0.smithi090.stderr:+ export PATH 2024-04-14T15:12:54.688 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-14T15:12:54.688 INFO:tasks.workunit.client.0.smithi090.stderr:+ unset rc 2024-04-14T15:12:54.688 INFO:tasks.workunit.client.0.smithi090.stderr:+ tee ceph_test_neorados_snapshots.log 2024-04-14T15:12:54.688 INFO:tasks.workunit.client.0.smithi090.stderr:+ sed 's/^/ snapshots: /' 2024-04-14T15:12:54.689 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-14T15:12:54.689 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-14T15:12:54.689 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-14T15:12:54.689 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.689 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-14T15:12:54.689 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.689 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-14T15:12:54.689 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-14T15:12:54.689 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.689 INFO:tasks.workunit.client.0.smithi090.stderr:+ ceph_test_neorados_snapshots 2024-04-14T15:12:54.689 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'grep=grep --color=auto' 2024-04-14T15:12:54.689 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-14T15:12:54.689 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-14T15:12:54.689 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.690 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-14T15:12:54.690 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.690 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorls.sh 2024-04-14T15:12:54.690 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' '!' -t 0 ']' 2024-04-14T15:12:54.690 INFO:tasks.workunit.client.0.smithi090.stderr:+++ return 2024-04-14T15:12:54.690 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.690 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-14T15:12:54.690 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.690 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-14T15:12:54.690 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export S_COLORS=auto 2024-04-14T15:12:54.690 INFO:tasks.workunit.client.0.smithi090.stderr:+++ S_COLORS=auto 2024-04-14T15:12:54.690 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.690 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-14T15:12:54.690 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.690 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-14T15:12:54.691 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.691 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-14T15:12:54.691 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-14T15:12:54.691 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-14T15:12:54.691 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.691 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-14T15:12:54.691 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.691 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-14T15:12:54.691 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-14T15:12:54.691 INFO:tasks.workunit.client.0.smithi090.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-14T15:12:54.691 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-14T15:12:54.691 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-14T15:12:54.691 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-14T15:12:54.691 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.692 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-14T15:12:54.692 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.692 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-14T15:12:54.692 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-14T15:12:54.692 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.692 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-14T15:12:54.692 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.692 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/gawk.sh 2024-04-14T15:12:54.692 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.692 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-14T15:12:54.692 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.692 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/lang.sh 2024-04-14T15:12:54.692 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup 2024-04-14T15:12:54.692 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/locale 2024-04-14T15:12:54.692 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.693 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-14T15:12:54.693 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.693 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-14T15:12:54.693 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-14T15:12:54.693 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-14T15:12:54.693 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-14T15:12:54.693 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-14T15:12:54.693 INFO:tasks.workunit.client.0.smithi090.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-14T15:12:54.693 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.693 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-14T15:12:54.693 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.693 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-14T15:12:54.693 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-14T15:12:54.693 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.693 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-14T15:12:54.694 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.694 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/gawk.sh 2024-04-14T15:12:54.694 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.694 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-14T15:12:54.694 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.694 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/lang.sh 2024-04-14T15:12:54.694 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup 2024-04-14T15:12:54.694 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/locale 2024-04-14T15:12:54.694 INFO:tasks.workunit.client.0.smithi090.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-14T15:12:54.694 INFO:tasks.workunit.client.0.smithi090.stderr:++++ export LANG=en_US.UTF-8 2024-04-14T15:12:54.694 INFO:tasks.workunit.client.0.smithi090.stderr:++++ LANG=en_US.UTF-8 2024-04-14T15:12:54.694 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.694 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-14T15:12:54.694 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.695 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG=en_US.UTF-8 2024-04-14T15:12:54.695 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup config 2024-04-14T15:12:54.695 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.695 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.695 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.695 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-14T15:12:54.695 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' dumb = linux ']' 2024-04-14T15:12:54.695 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.695 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-14T15:12:54.695 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.695 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/less.sh 2024-04-14T15:12:54.695 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.695 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-14T15:12:54.695 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-14T15:12:54.695 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-14T15:12:54.696 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.696 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-14T15:12:54.696 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.696 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/which2.sh 2024-04-14T15:12:54.696 INFO:tasks.workunit.client.0.smithi090.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-14T15:12:54.696 INFO:tasks.workunit.client.0.smithi090.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-14T15:12:54.696 INFO:tasks.workunit.client.0.smithi090.stderr:+++++ readlink /proc/94701/exe 2024-04-14T15:12:54.696 INFO:tasks.workunit.client.0.smithi090.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-14T15:12:54.696 INFO:tasks.workunit.client.0.smithi090.stderr:++++ export LANG=en_US.UTF-8 2024-04-14T15:12:54.696 INFO:tasks.workunit.client.0.smithi090.stderr:++++ LANG=en_US.UTF-8 2024-04-14T15:12:54.696 INFO:tasks.workunit.client.0.smithi090.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-14T15:12:54.696 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-14T15:12:54.696 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.696 INFO:tasks.workunit.client.0.smithi090.stderr:+++ LANG=en_US.UTF-8 2024-04-14T15:12:54.697 INFO:tasks.workunit.client.0.smithi090.stderr:+++ unset LANG_backup config 2024-04-14T15:12:54.697 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n '' ']' 2024-04-14T15:12:54.697 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-14T15:12:54.697 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' dumb = linux ']' 2024-04-14T15:12:54.697 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.697 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-14T15:12:54.697 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.697 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/less.sh 2024-04-14T15:12:54.697 INFO:tasks.workunit.client.0.smithi090.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-14T15:12:54.697 INFO:tasks.workunit.client.0.smithi090.stderr:++ for i in /etc/profile.d/*.sh 2024-04-14T15:12:54.697 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-14T15:12:54.697 INFO:tasks.workunit.client.0.smithi090.stderr:++ '[' '' ']' 2024-04-14T15:12:54.697 INFO:tasks.workunit.client.0.smithi090.stderr:++ . /etc/profile.d/which2.sh 2024-04-14T15:12:54.697 INFO:tasks.workunit.client.0.smithi090.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-14T15:12:54.697 INFO:tasks.workunit.client.0.smithi090.stderr:+++++ readlink /proc/94756/exe 2024-04-14T15:12:54.698 INFO:tasks.workunit.client.0.smithi090.stderr:++++ basename /usr/bin/bash 2024-04-14T15:12:54.698 INFO:tasks.workunit.client.0.smithi090.stderr:++++ basename /usr/bin/bash 2024-04-14T15:12:54.698 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_declare='declare -f' 2024-04-14T15:12:54.698 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_opt=-f 2024-04-14T15:12:54.698 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export which_declare 2024-04-14T15:12:54.698 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export -f which 2024-04-14T15:12:54.698 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset i 2024-04-14T15:12:54.698 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset -f pathmunge 2024-04-14T15:12:54.698 INFO:tasks.workunit.client.0.smithi090.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-14T15:12:54.698 INFO:tasks.workunit.client.0.smithi090.stderr:+ export PATH 2024-04-14T15:12:54.698 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-14T15:12:54.698 INFO:tasks.workunit.client.0.smithi090.stderr:+ unset rc 2024-04-14T15:12:54.698 INFO:tasks.workunit.client.0.smithi090.stderr:+ sed 's/^/ write_operations: /' 2024-04-14T15:12:54.699 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_declare='declare -f' 2024-04-14T15:12:54.699 INFO:tasks.workunit.client.0.smithi090.stderr:+++ which_opt=-f 2024-04-14T15:12:54.699 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export which_declare 2024-04-14T15:12:54.699 INFO:tasks.workunit.client.0.smithi090.stderr:+++ export -f which 2024-04-14T15:12:54.699 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset i 2024-04-14T15:12:54.699 INFO:tasks.workunit.client.0.smithi090.stderr:++ unset -f pathmunge 2024-04-14T15:12:54.699 INFO:tasks.workunit.client.0.smithi090.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-14T15:12:54.699 INFO:tasks.workunit.client.0.smithi090.stderr:+ export PATH 2024-04-14T15:12:54.699 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-14T15:12:54.699 INFO:tasks.workunit.client.0.smithi090.stderr:+ unset rc 2024-04-14T15:12:54.699 INFO:tasks.workunit.client.0.smithi090.stderr:+ ceph_test_neorados_write_operations 2024-04-14T15:12:54.699 INFO:tasks.workunit.client.0.smithi090.stderr:+ ceph_test_neorados_watch_notify 2024-04-14T15:12:54.699 INFO:tasks.workunit.client.0.smithi090.stderr:+ sed 's/^/ watch_notify: /' 2024-04-14T15:12:54.699 INFO:tasks.workunit.client.0.smithi090.stderr:+ tee ceph_test_neorados_watch_notify.log 2024-04-14T15:12:54.699 INFO:tasks.workunit.client.0.smithi090.stderr:+ tee ceph_test_neorados_write_operations.log 2024-04-14T15:12:54.700 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:12:54.724 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:54.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:54.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:12:54.826 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:12:54.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:54.849 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:54.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:12:54.895 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-14T15:12:54.895 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-14T15:12:54.895 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-14T15:12:54.895 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-14T15:12:54.895 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-14T15:12:54.951 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:12:54.975 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:54.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:54.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:12:55.049 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:12:54 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:12:55.077 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:12:55.119 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:55.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:55.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:12:55.221 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:12:55.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:55.246 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:55.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:12:55.323 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:55 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/309942895' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi090-93998-1"}]: dispatch 2024-04-14T15:12:55.323 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:55 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/309942895' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi090-93998-1"}]: dispatch 2024-04-14T15:12:55.323 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:55 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/309942895' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi090-93998-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:12:55.323 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:55 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2377984195' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-04-14T15:12:55.323 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:55 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1118408389' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi090-94522-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:12:55.323 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:55 smithi090 ceph-mon[41670]: from='client.25021 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi090-94522-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:12:55.324 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:55 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1465058495' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi090-94662-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:12:55.324 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:55 smithi090 ceph-mon[41670]: from='client.25001 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi090-94662-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:12:55.324 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:55 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:12:55.324 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:55 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/309942895' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi090-93998-1"}]: dispatch 2024-04-14T15:12:55.324 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:55 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/309942895' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi090-93998-1"}]: dispatch 2024-04-14T15:12:55.324 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:55 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/309942895' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi090-93998-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:12:55.324 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:55 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2377984195' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-04-14T15:12:55.324 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:55 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1118408389' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi090-94522-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:12:55.324 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:55 smithi090 ceph-mon[30399]: from='client.25021 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi090-94522-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:12:55.324 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:55 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1465058495' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi090-94662-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:12:55.324 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:55 smithi090 ceph-mon[30399]: from='client.25001 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi090-94662-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:12:55.325 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:55 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:12:55.347 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:12:55.371 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:55.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:55.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:12:55.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:55 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/309942895' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi090-93998-1"}]: dispatch 2024-04-14T15:12:55.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:55 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/309942895' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi090-93998-1"}]: dispatch 2024-04-14T15:12:55.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:55 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/309942895' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi090-93998-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:12:55.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:55 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2377984195' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-04-14T15:12:55.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:55 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1118408389' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi090-94522-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:12:55.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:55 smithi092 ceph-mon[33966]: from='client.25021 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi090-94522-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:12:55.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:55 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1465058495' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi090-94662-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:12:55.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:55 smithi092 ceph-mon[33966]: from='client.25001 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi090-94662-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:12:55.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:55 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:12:55.473 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:12:55.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:55.497 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:55.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:12:55.598 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:12:55.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:55.623 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:55.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:12:55.724 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:12:55.748 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:55.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:55.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:12:55.850 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:12:55.875 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:55.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:55.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:12:55.977 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:12:56.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:56.002 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:56.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:12:56.102 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:12:56.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:56.146 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:56.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:12:56.247 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:12:56.265 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[41670]: pgmap v106: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:12:56.265 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/309942895' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi090-93998-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:12:56.266 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[41670]: from='client.25021 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi090-94522-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:12:56.266 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[41670]: from='client.25001 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi090-94662-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:12:56.266 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1118408389' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi090-94522-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi090-94522-1"}]: dispatch 2024-04-14T15:12:56.266 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[41670]: osdmap e107: 8 total, 8 up, 8 in 2024-04-14T15:12:56.266 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/309942895' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi090-93998-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi090-93998-1"}]: dispatch 2024-04-14T15:12:56.266 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1465058495' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi090-94662-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi090-94662-1"}]: dispatch 2024-04-14T15:12:56.266 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[41670]: from='client.25021 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi090-94522-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi090-94522-1"}]: dispatch 2024-04-14T15:12:56.266 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/117387996' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94114-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.266 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1106328377' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi090-93379-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.266 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi090-93444-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.266 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3757631150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93995-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.267 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2145372594' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi090-93342-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.267 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/4103332675' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi090-93363-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.267 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/473680876' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi090-93398-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.267 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3828074346' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi090-93436-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.267 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi090-93601-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.267 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1684644655' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi090-93724-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.268 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1680088077' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi090-93709-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.268 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3182938272' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi090-93930-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.268 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/757959112' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94011-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.268 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/766142007' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi090-93864-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.268 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/4254723003' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi090-94131-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.268 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/861388440' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.268 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[41670]: from='client.24955 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94114-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.268 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[41670]: from='client.25001 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi090-94662-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi090-94662-1"}]: dispatch 2024-04-14T15:12:56.268 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2381966651' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.268 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[41670]: from='client.24802 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi090-93379-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.269 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[41670]: from='client.24860 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi090-93864-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.269 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2763816171' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi090-93731-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.269 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[41670]: from='client.24829 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi090-93444-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.269 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/495867689' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi090-93652-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.269 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[41670]: from='client.24991 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93995-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.269 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1412026529' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi090-93476-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.269 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[41670]: from='client.24923 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi090-94131-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.269 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[41670]: from='client.24796 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi090-93342-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.269 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[41670]: from='client.24896 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.269 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[41670]: from='client.24808 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi090-93363-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.269 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[41670]: from='client.24811 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi090-93398-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.269 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[41670]: from='client.24823 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi090-93436-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.269 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[41670]: from='client.24853 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi090-93601-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.270 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[41670]: from='client.24868 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi090-93724-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.270 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[41670]: from='client.24871 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi090-93709-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.270 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[41670]: from='client.24889 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi090-93930-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.270 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[41670]: from='client.25000 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94011-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.270 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[41670]: from='client.24835 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi090-93476-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.270 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[30399]: pgmap v106: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:12:56.270 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/309942895' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi090-93998-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:12:56.270 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[30399]: from='client.25021 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi090-94522-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:12:56.271 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[30399]: from='client.25001 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi090-94662-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:12:56.271 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1118408389' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi090-94522-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi090-94522-1"}]: dispatch 2024-04-14T15:12:56.271 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[30399]: osdmap e107: 8 total, 8 up, 8 in 2024-04-14T15:12:56.271 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/309942895' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi090-93998-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi090-93998-1"}]: dispatch 2024-04-14T15:12:56.271 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1465058495' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi090-94662-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi090-94662-1"}]: dispatch 2024-04-14T15:12:56.271 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[30399]: from='client.25021 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi090-94522-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi090-94522-1"}]: dispatch 2024-04-14T15:12:56.271 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/117387996' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94114-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.271 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1106328377' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi090-93379-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.271 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi090-93444-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.271 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3757631150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93995-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.271 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2145372594' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi090-93342-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.271 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/4103332675' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi090-93363-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.271 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/473680876' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi090-93398-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.272 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3828074346' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi090-93436-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.272 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi090-93601-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.272 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1684644655' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi090-93724-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.272 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1680088077' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi090-93709-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.272 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3182938272' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi090-93930-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.272 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/757959112' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94011-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.272 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/766142007' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi090-93864-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.272 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/4254723003' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi090-94131-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.272 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/861388440' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.272 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[30399]: from='client.24955 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94114-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.272 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[30399]: from='client.25001 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi090-94662-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi090-94662-1"}]: dispatch 2024-04-14T15:12:56.272 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2381966651' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.272 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[30399]: from='client.24802 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi090-93379-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.272 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[30399]: from='client.24860 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi090-93864-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.273 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2763816171' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi090-93731-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.273 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[30399]: from='client.24829 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi090-93444-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.273 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/495867689' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi090-93652-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.273 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[30399]: from='client.24991 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93995-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.273 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1412026529' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi090-93476-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.273 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[30399]: from='client.24923 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi090-94131-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.273 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[30399]: from='client.24796 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi090-93342-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.273 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[30399]: from='client.24896 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.273 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[30399]: from='client.24808 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi090-93363-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.273 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[30399]: from='client.24811 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi090-93398-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.273 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[30399]: from='client.24823 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi090-93436-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.273 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[30399]: from='client.24853 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi090-93601-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.273 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[30399]: from='client.24868 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi090-93724-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.273 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[30399]: from='client.24871 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi090-93709-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.274 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[30399]: from='client.24889 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi090-93930-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.274 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[30399]: from='client.25000 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94011-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.274 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:56 smithi090 ceph-mon[30399]: from='client.24835 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi090-93476-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.301 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:56.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:56.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:12:56.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:56 smithi092 ceph-mon[33966]: pgmap v106: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:12:56.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:56 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/309942895' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi090-93998-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:12:56.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:56 smithi092 ceph-mon[33966]: from='client.25021 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi090-94522-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:12:56.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:56 smithi092 ceph-mon[33966]: from='client.25001 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi090-94662-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:12:56.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:56 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1118408389' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi090-94522-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi090-94522-1"}]: dispatch 2024-04-14T15:12:56.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:56 smithi092 ceph-mon[33966]: osdmap e107: 8 total, 8 up, 8 in 2024-04-14T15:12:56.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:56 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/309942895' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi090-93998-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi090-93998-1"}]: dispatch 2024-04-14T15:12:56.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:56 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1465058495' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi090-94662-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi090-94662-1"}]: dispatch 2024-04-14T15:12:56.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:56 smithi092 ceph-mon[33966]: from='client.25021 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi090-94522-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi090-94522-1"}]: dispatch 2024-04-14T15:12:56.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:56 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/117387996' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94114-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:56 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1106328377' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi090-93379-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:56 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi090-93444-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:56 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3757631150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93995-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:56 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2145372594' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi090-93342-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:56 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/4103332675' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi090-93363-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:56 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/473680876' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi090-93398-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:56 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3828074346' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi090-93436-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:56 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi090-93601-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:56 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1684644655' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi090-93724-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:56 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1680088077' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi090-93709-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:56 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3182938272' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi090-93930-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:56 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/757959112' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94011-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:56 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/766142007' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi090-93864-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:56 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/4254723003' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi090-94131-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:56 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/861388440' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:56 smithi092 ceph-mon[33966]: from='client.24955 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94114-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:56 smithi092 ceph-mon[33966]: from='client.25001 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi090-94662-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi090-94662-1"}]: dispatch 2024-04-14T15:12:56.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:56 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2381966651' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:56 smithi092 ceph-mon[33966]: from='client.24802 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi090-93379-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:56 smithi092 ceph-mon[33966]: from='client.24860 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi090-93864-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.376 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:56 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2763816171' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi090-93731-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.376 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:56 smithi092 ceph-mon[33966]: from='client.24829 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi090-93444-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.376 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:56 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/495867689' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi090-93652-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.376 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:56 smithi092 ceph-mon[33966]: from='client.24991 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93995-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.376 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:56 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1412026529' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi090-93476-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.376 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:56 smithi092 ceph-mon[33966]: from='client.24923 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi090-94131-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.376 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:56 smithi092 ceph-mon[33966]: from='client.24796 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi090-93342-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.376 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:56 smithi092 ceph-mon[33966]: from='client.24896 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.376 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:56 smithi092 ceph-mon[33966]: from='client.24808 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi090-93363-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.376 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:56 smithi092 ceph-mon[33966]: from='client.24811 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi090-93398-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.376 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:56 smithi092 ceph-mon[33966]: from='client.24823 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi090-93436-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.376 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:56 smithi092 ceph-mon[33966]: from='client.24853 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi090-93601-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.376 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:56 smithi092 ceph-mon[33966]: from='client.24868 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi090-93724-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.376 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:56 smithi092 ceph-mon[33966]: from='client.24871 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi090-93709-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.377 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:56 smithi092 ceph-mon[33966]: from='client.24889 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi090-93930-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.377 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:56 smithi092 ceph-mon[33966]: from='client.25000 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94011-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.377 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:56 smithi092 ceph-mon[33966]: from='client.24835 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi090-93476-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:56.403 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:12:56.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:56.439 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:56.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:12:56.540 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:12:56.567 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:56.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:56.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:12:56.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:56.589+0000 7f281e41f700 1 -- 172.21.15.90:0/1085148371 >> v1:172.21.15.92:6789/0 conn(0x7f28100add20 legacy=0x7f28100b4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:56.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:56.589+0000 7f281e41f700 1 -- 172.21.15.90:0/1085148371 shutdown_connections 2024-04-14T15:12:56.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:56.589+0000 7f281e41f700 1 -- 172.21.15.90:0/1085148371 >> 172.21.15.90:0/1085148371 conn(0x7f281001a930 msgr2=0x7f281001ad40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:12:56.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:56.590+0000 7f281e41f700 1 -- 172.21.15.90:0/1085148371 shutdown_connections 2024-04-14T15:12:56.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:56.593+0000 7f281e41f700 1 -- 172.21.15.90:0/1085148371 wait complete. 2024-04-14T15:12:56.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:56.594+0000 7f281e41f700 1 librados: starting msgr at 2024-04-14T15:12:56.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:56.594+0000 7f281e41f700 1 librados: starting objecter 2024-04-14T15:12:56.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:56.594+0000 7f281e41f700 1 Processor -- start 2024-04-14T15:12:56.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:56.594+0000 7f281e41f700 1 -- start start 2024-04-14T15:12:56.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:56.594+0000 7f281e41f700 1 librados: setting wanted keys 2024-04-14T15:12:56.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:56.594+0000 7f281e41f700 1 librados: calling monclient init 2024-04-14T15:12:56.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:56.595+0000 7f281e41f700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f281014e760 con 0x7f2810146da0 2024-04-14T15:12:56.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:56.595+0000 7f281d41d700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f28100add20 0x7f28100b5e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.90:33978/0 (socket says 172.21.15.90:33978) 2024-04-14T15:12:56.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:56.595+0000 7f281d41d700 1 -- 172.21.15.90:0/2645816859 learned_addr learned my addr 172.21.15.90:0/2645816859 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:12:56.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:56.595+0000 7f281e41f700 1 -- 172.21.15.90:0/2645816859 --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f281014f9a0 con 0x7f28100add20 2024-04-14T15:12:56.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:56.595+0000 7f281e41f700 1 -- 172.21.15.90:0/2645816859 --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f2810150be0 con 0x7f281014a9a0 2024-04-14T15:12:56.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:56.597+0000 7f280e7fc700 10 client.?.objecter ms_handle_connect 0x7f28100add20 2024-04-14T15:12:56.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:56.597+0000 7f280e7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:12:56.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:56.597+0000 7f280e7fc700 10 client.?.objecter ms_handle_connect 0x7f2810146da0 2024-04-14T15:12:56.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:56.597+0000 7f280e7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:12:56.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:56.597+0000 7f280e7fc700 10 client.?.objecter ms_handle_connect 0x7f281014a9a0 2024-04-14T15:12:56.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:56.597+0000 7f280e7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:12:56.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:56.597+0000 7f280e7fc700 1 -- 172.21.15.90:0/2645816859 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1763445414 0 0) 0x7f281014f9a0 con 0x7f28100add20 2024-04-14T15:12:56.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:56.597+0000 7f280e7fc700 1 -- 172.21.15.90:0/2645816859 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f28040035b0 con 0x7f28100add20 2024-04-14T15:12:56.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:56.597+0000 7f280e7fc700 1 -- 172.21.15.90:0/2645816859 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2007795644 0 0) 0x7f281014e760 con 0x7f2810146da0 2024-04-14T15:12:56.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:56.597+0000 7f280e7fc700 1 -- 172.21.15.90:0/2645816859 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f281014f9a0 con 0x7f2810146da0 2024-04-14T15:12:56.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:56.597+0000 7f280e7fc700 1 -- 172.21.15.90:0/2645816859 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1727195462 0 0) 0x7f2810150be0 con 0x7f281014a9a0 2024-04-14T15:12:56.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:56.597+0000 7f280e7fc700 1 -- 172.21.15.90:0/2645816859 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f281014e760 con 0x7f281014a9a0 2024-04-14T15:12:56.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:56.598+0000 7f280e7fc700 1 -- 172.21.15.90:0/2645816859 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4008343859 0 0) 0x7f281014f9a0 con 0x7f2810146da0 2024-04-14T15:12:56.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:56.598+0000 7f280e7fc700 1 -- 172.21.15.90:0/2645816859 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2810150be0 con 0x7f2810146da0 2024-04-14T15:12:56.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:56.598+0000 7f280e7fc700 1 -- 172.21.15.90:0/2645816859 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f2818068290 con 0x7f2810146da0 2024-04-14T15:12:56.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:56.598+0000 7f280e7fc700 1 -- 172.21.15.90:0/2645816859 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3153899708 0 0) 0x7f28040035b0 con 0x7f28100add20 2024-04-14T15:12:56.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:56.598+0000 7f280e7fc700 1 -- 172.21.15.90:0/2645816859 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f281014f9a0 con 0x7f28100add20 2024-04-14T15:12:56.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:56.598+0000 7f280e7fc700 1 -- 172.21.15.90:0/2645816859 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f2808001830 con 0x7f28100add20 2024-04-14T15:12:56.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:56.599+0000 7f280e7fc700 1 -- 172.21.15.90:0/2645816859 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3837173266 0 0) 0x7f281014f9a0 con 0x7f28100add20 2024-04-14T15:12:56.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:56.599+0000 7f280e7fc700 1 -- 172.21.15.90:0/2645816859 >> v1:172.21.15.90:6790/0 conn(0x7f281014a9a0 legacy=0x7f281014ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:56.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:56.599+0000 7f280e7fc700 1 -- 172.21.15.90:0/2645816859 >> v1:172.21.15.90:6789/0 conn(0x7f2810146da0 legacy=0x7f2810149190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:56.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:56.599+0000 7f280e7fc700 1 -- 172.21.15.90:0/2645816859 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2810151e20 con 0x7f28100add20 2024-04-14T15:12:56.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:56.599+0000 7f281e41f700 1 -- 172.21.15.90:0/2645816859 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f281014e990 con 0x7f28100add20 2024-04-14T15:12:56.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:56.599+0000 7f281e41f700 10 client.25108.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:12:56.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:56.599+0000 7f281e41f700 1 -- 172.21.15.90:0/2645816859 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f281014ee50 con 0x7f28100add20 2024-04-14T15:12:56.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:56.599+0000 7f281e41f700 1 librados: init done 2024-04-14T15:12:56.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:56.601+0000 7f280e7fc700 1 -- 172.21.15.90:0/2645816859 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f2808003b20 con 0x7f28100add20 2024-04-14T15:12:56.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:56.601+0000 7f280e7fc700 1 -- 172.21.15.90:0/2645816859 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f2808005c10 con 0x7f28100add20 2024-04-14T15:12:56.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:56.602+0000 7f280e7fc700 1 -- 172.21.15.90:0/2645816859 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f280801c880 con 0x7f28100add20 2024-04-14T15:12:56.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:56.602+0000 7f281e41f700 1 -- 172.21.15.90:0/2645816859 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28100a6510 con 0x7f28100add20 2024-04-14T15:12:56.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:56.603+0000 7f280e7fc700 1 -- 172.21.15.90:0/2645816859 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(108..108 src has 51..108) ==== 19763+0+0 (unknown 2802138315 0 0) 0x7f2808055cc0 con 0x7f28100add20 2024-04-14T15:12:56.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:56.603+0000 7f280e7fc700 10 client.25108.objecter ms_dispatch 0x7f2818001550 osd_map(108..108 src has 51..108) 2024-04-14T15:12:56.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:56.603+0000 7f280e7fc700 3 client.25108.objecter handle_osd_map got epochs [108,108] > 0 2024-04-14T15:12:56.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:56.603+0000 7f280e7fc700 3 client.25108.objecter handle_osd_map decoding full epoch 108 2024-04-14T15:12:56.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:56.603+0000 7f280e7fc700 20 client.25108.objecter dump_active .. 0 homeless 2024-04-14T15:12:56.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:56.605+0000 7f280e7fc700 10 client.25108.objecter ms_handle_connect 0x7f28040716d0 2024-04-14T15:12:56.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:56.609+0000 7f280e7fc700 1 -- 172.21.15.90:0/2645816859 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f280805bd70 con 0x7f28100add20 2024-04-14T15:12:56.630 INFO:tasks.workunit.client.0.smithi090.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2024-04-14T15:12:56.630 INFO:tasks.workunit.client.0.smithi090.stdout: api_asio: [----------] Global test environment set-up. 2024-04-14T15:12:56.630 INFO:tasks.workunit.client.0.smithi090.stdout: api_asio: [----------] 12 tests from AsioRados 2024-04-14T15:12:56.630 INFO:tasks.workunit.client.0.smithi090.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2024-04-14T15:12:56.630 INFO:tasks.workunit.client.0.smithi090.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (2 ms) 2024-04-14T15:12:56.630 INFO:tasks.workunit.client.0.smithi090.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2024-04-14T15:12:56.630 INFO:tasks.workunit.client.0.smithi090.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (0 ms) 2024-04-14T15:12:56.631 INFO:tasks.workunit.client.0.smithi090.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2024-04-14T15:12:56.631 INFO:tasks.workunit.client.0.smithi090.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (1 ms) 2024-04-14T15:12:56.631 INFO:tasks.workunit.client.0.smithi090.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2024-04-14T15:12:56.631 INFO:tasks.workunit.client.0.smithi090.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (63 ms) 2024-04-14T15:12:56.631 INFO:tasks.workunit.client.0.smithi090.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2024-04-14T15:12:56.631 INFO:tasks.workunit.client.0.smithi090.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (13 ms) 2024-04-14T15:12:56.631 INFO:tasks.workunit.client.0.smithi090.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2024-04-14T15:12:56.631 INFO:tasks.workunit.client.0.smithi090.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (12 ms) 2024-04-14T15:12:56.631 INFO:tasks.workunit.client.0.smithi090.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2024-04-14T15:12:56.631 INFO:tasks.workunit.client.0.smithi090.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (2 ms) 2024-04-14T15:12:56.631 INFO:tasks.workunit.client.0.smithi090.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2024-04-14T15:12:56.631 INFO:tasks.workunit.client.0.smithi090.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (0 ms) 2024-04-14T15:12:56.631 INFO:tasks.workunit.client.0.smithi090.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2024-04-14T15:12:56.631 INFO:tasks.workunit.client.0.smithi090.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (1 ms) 2024-04-14T15:12:56.631 INFO:tasks.workunit.client.0.smithi090.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2024-04-14T15:12:56.632 INFO:tasks.workunit.client.0.smithi090.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (5 ms) 2024-04-14T15:12:56.632 INFO:tasks.workunit.client.0.smithi090.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2024-04-14T15:12:56.632 INFO:tasks.workunit.client.0.smithi090.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (3 ms) 2024-04-14T15:12:56.632 INFO:tasks.workunit.client.0.smithi090.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2024-04-14T15:12:56.632 INFO:tasks.workunit.client.0.smithi090.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (5 ms) 2024-04-14T15:12:56.632 INFO:tasks.workunit.client.0.smithi090.stdout: api_asio: [----------] 12 tests from AsioRados (107 ms total) 2024-04-14T15:12:56.632 INFO:tasks.workunit.client.0.smithi090.stdout: api_asio: 2024-04-14T15:12:56.632 INFO:tasks.workunit.client.0.smithi090.stdout: api_asio: [----------] Global test environment tear-down 2024-04-14T15:12:56.632 INFO:tasks.workunit.client.0.smithi090.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (2435 ms total) 2024-04-14T15:12:56.632 INFO:tasks.workunit.client.0.smithi090.stdout: api_asio: [ PASSED ] 12 tests. 2024-04-14T15:12:56.669 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:12:56.701 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:56.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:56.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:12:56.803 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:12:56.837 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:56.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:56.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:12:56.939 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:12:56.979 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:56.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:56.980 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:12:56.990 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: [==========] Running 11 tests from 3 test suites. 2024-04-14T15:12:56.991 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: [----------] Global test environment set-up. 2024-04-14T15:12:56.991 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: [----------] 7 tests from LibRadosList 2024-04-14T15:12:56.991 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2024-04-14T15:12:56.991 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: [ OK ] LibRadosList.ListObjects (594 ms) 2024-04-14T15:12:56.991 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2024-04-14T15:12:56.991 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (41 ms) 2024-04-14T15:12:56.991 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2024-04-14T15:12:56.991 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: myset foo1,foo2,foo3 2024-04-14T15:12:56.991 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: foo1 2024-04-14T15:12:56.991 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: foo2 2024-04-14T15:12:56.991 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: foo3 2024-04-14T15:12:56.991 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: myset foo1,foo4,foo5 2024-04-14T15:12:56.991 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: foo4 2024-04-14T15:12:56.991 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: foo5 2024-04-14T15:12:56.991 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: foo1 2024-04-14T15:12:56.992 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: myset foo6,foo7 2024-04-14T15:12:56.992 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: foo7 2024-04-14T15:12:56.992 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: foo6 2024-04-14T15:12:56.992 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-04-14T15:12:56.992 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: ns1:foo4 2024-04-14T15:12:56.992 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: ns1:foo5 2024-04-14T15:12:56.992 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: ns2:foo7 2024-04-14T15:12:56.992 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: ns2:foo6 2024-04-14T15:12:56.992 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: ns1:foo1 2024-04-14T15:12:56.992 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: :foo1 2024-04-14T15:12:56.992 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: :foo2 2024-04-14T15:12:56.992 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: :foo3 2024-04-14T15:12:56.992 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (105 ms) 2024-04-14T15:12:56.992 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2024-04-14T15:12:56.993 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: 1 0 2024-04-14T15:12:56.993 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: 10 0 2024-04-14T15:12:56.993 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: 13 0 2024-04-14T15:12:56.993 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: 7 0 2024-04-14T15:12:56.993 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: 14 0 2024-04-14T15:12:56.993 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: 0 0 2024-04-14T15:12:56.993 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: 15 0 2024-04-14T15:12:56.993 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: 11 0 2024-04-14T15:12:56.993 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: 5 0 2024-04-14T15:12:56.993 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: 8 0 2024-04-14T15:12:56.993 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: 6 0 2024-04-14T15:12:56.993 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: 3 0 2024-04-14T15:12:56.993 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: 4 0 2024-04-14T15:12:56.993 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: 12 0 2024-04-14T15:12:56.993 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: 9 0 2024-04-14T15:12:56.993 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: 2 0 2024-04-14T15:12:56.994 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-04-14T15:12:56.994 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (85 ms) 2024-04-14T15:12:56.994 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2024-04-14T15:12:56.994 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: x cursor=MIN 2024-04-14T15:12:56.994 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: > oid=1 cursor=13:02547ec2:::1:head 2024-04-14T15:12:56.994 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: > oid=10 cursor=13:52ea6a34:::10:head 2024-04-14T15:12:56.994 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: > oid=13 cursor=13:566253c9:::13:head 2024-04-14T15:12:56.994 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: > oid=7 cursor=13:5c6b0b28:::7:head 2024-04-14T15:12:56.994 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: > oid=14 cursor=13:62a1935d:::14:head 2024-04-14T15:12:56.994 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: > oid=0 cursor=13:6cac518f:::0:head 2024-04-14T15:12:56.994 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: > oid=15 cursor=13:863748b0:::15:head 2024-04-14T15:12:56.994 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: > oid=11 cursor=13:89d3ae78:::11:head 2024-04-14T15:12:56.994 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: > oid=5 cursor=13:b29083e3:::5:head 2024-04-14T15:12:56.994 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: > oid=8 cursor=13:bd63b0f1:::8:head 2024-04-14T15:12:56.995 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: > oid=6 cursor=13:c4fdafeb:::6:head 2024-04-14T15:12:56.995 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: > oid=3 cursor=13:cfc208b3:::3:head 2024-04-14T15:12:56.995 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: > oid=4 cursor=13:d83876eb:::4:head 2024-04-14T15:12:56.995 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: > oid=12 cursor=13:de5d7c5f:::12:head 2024-04-14T15:12:56.995 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: > oid=9 cursor=13:e960b815:::9:head 2024-04-14T15:12:56.995 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: > oid=2 cursor=13:f905c69b:::2:head 2024-04-14T15:12:56.995 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: FIRST> seek to MIN oid=1 2024-04-14T15:12:56.995 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : oid=1 cursor=13:02547ec2:::1:head 2024-04-14T15:12:56.995 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : seek to 13:02547ec2:::1:head 2024-04-14T15:12:56.995 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: > 13:02547ec2:::1:head -> 1 2024-04-14T15:12:56.995 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : oid=10 cursor=13:52ea6a34:::10:head 2024-04-14T15:12:56.995 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : seek to 13:52ea6a34:::10:head 2024-04-14T15:12:56.995 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: > 13:52ea6a34:::10:head -> 10 2024-04-14T15:12:56.995 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : oid=13 cursor=13:566253c9:::13:head 2024-04-14T15:12:56.995 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : seek to 13:566253c9:::13:head 2024-04-14T15:12:56.996 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: > 13:566253c9:::13:head -> 13 2024-04-14T15:12:56.996 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : oid=7 cursor=13:5c6b0b28:::7:head 2024-04-14T15:12:56.996 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : seek to 13:5c6b0b28:::7:head 2024-04-14T15:12:56.996 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: > 13:5c6b0b28:::7:head -> 7 2024-04-14T15:12:57.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:57.028+0000 7f281e41f700 1 -- 172.21.15.90:0/2645816859 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) -- 0x7f28100b5590 con 0x7f28100add20 2024-04-14T15:12:57.040 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:12:57.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:57.039+0000 7f280e7fc700 1 -- 172.21.15.90:0/2645816859 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v108) ==== 81+0+139472 (unknown 2905176985 0 1581721321) 0x7f2808060630 con 0x7f28100add20 2024-04-14T15:12:57.041 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-14T15:12:57.041 INFO:teuthology.orchestra.run.smithi090.stdout: "epoch": 108, 2024-04-14T15:12:57.041 INFO:teuthology.orchestra.run.smithi090.stdout: "fsid": "0e288098-fa70-11ee-bc8e-c7b262605968", 2024-04-14T15:12:57.041 INFO:teuthology.orchestra.run.smithi090.stdout: "created": "2024-04-14T15:04:05.808313+0000", 2024-04-14T15:12:57.041 INFO:teuthology.orchestra.run.smithi090.stdout: "modified": "2024-04-14T15:12:56.046703+0000", 2024-04-14T15:12:57.041 INFO:teuthology.orchestra.run.smithi090.stdout: "last_up_change": "2024-04-14T15:09:29.769352+0000", 2024-04-14T15:12:57.041 INFO:teuthology.orchestra.run.smithi090.stdout: "last_in_change": "2024-04-14T15:09:11.803164+0000", 2024-04-14T15:12:57.041 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-14T15:12:57.041 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_num": 5806080, 2024-04-14T15:12:57.041 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_set": [ 2024-04-14T15:12:57.041 INFO:teuthology.orchestra.run.smithi090.stdout: "nodeep-scrub", 2024-04-14T15:12:57.041 INFO:teuthology.orchestra.run.smithi090.stdout: "noscrub", 2024-04-14T15:12:57.042 INFO:teuthology.orchestra.run.smithi090.stdout: "pglog_hardlimit", 2024-04-14T15:12:57.042 INFO:teuthology.orchestra.run.smithi090.stdout: "purged_snapdirs", 2024-04-14T15:12:57.042 INFO:teuthology.orchestra.run.smithi090.stdout: "recovery_deletes", 2024-04-14T15:12:57.042 INFO:teuthology.orchestra.run.smithi090.stdout: "sortbitwise" 2024-04-14T15:12:57.042 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-14T15:12:57.042 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_version": 19, 2024-04-14T15:12:57.042 INFO:teuthology.orchestra.run.smithi090.stdout: "full_ratio": 0.94999998807907104, 2024-04-14T15:12:57.042 INFO:teuthology.orchestra.run.smithi090.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-14T15:12:57.042 INFO:teuthology.orchestra.run.smithi090.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-14T15:12:57.042 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_snapshot": "", 2024-04-14T15:12:57.042 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_max": 41, 2024-04-14T15:12:57.042 INFO:teuthology.orchestra.run.smithi090.stdout: "max_osd": 8, 2024-04-14T15:12:57.042 INFO:teuthology.orchestra.run.smithi090.stdout: "require_min_compat_client": "luminous", 2024-04-14T15:12:57.043 INFO:teuthology.orchestra.run.smithi090.stdout: "min_compat_client": "luminous", 2024-04-14T15:12:57.043 INFO:teuthology.orchestra.run.smithi090.stdout: "require_osd_release": "squid", 2024-04-14T15:12:57.043 INFO:teuthology.orchestra.run.smithi090.stdout: "allow_crimson": false, 2024-04-14T15:12:57.043 INFO:teuthology.orchestra.run.smithi090.stdout: "pools": [ 2024-04-14T15:12:57.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.043 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 1, 2024-04-14T15:12:57.043 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": ".mgr", 2024-04-14T15:12:57.043 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:07:23.959628+0000", 2024-04-14T15:12:57.043 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-14T15:12:57.043 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool", 2024-04-14T15:12:57.043 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:12:57.043 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:12:57.043 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:12:57.043 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:12:57.044 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:12:57.044 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:12:57.044 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:12:57.044 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:12:57.044 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:12:57.044 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:12:57.044 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 1, 2024-04-14T15:12:57.044 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 1, 2024-04-14T15:12:57.044 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 1, 2024-04-14T15:12:57.044 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 1, 2024-04-14T15:12:57.044 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 1, 2024-04-14T15:12:57.045 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:12:57.045 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:12:57.045 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:12:57.045 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:12:57.045 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:12:57.045 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:12:57.045 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:12:57.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.045 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "104", 2024-04-14T15:12:57.045 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:12:57.045 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:12:57.045 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:12:57.045 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:12:57.045 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:12:57.046 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:12:57.046 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:12:57.046 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:12:57.046 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:12:57.046 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:12:57.046 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:12:57.046 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:12:57.046 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:12:57.046 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:12:57.046 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:12:57.046 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:12:57.046 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:12:57.046 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:12:57.047 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:12:57.047 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:12:57.047 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:12:57.047 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:12:57.047 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:12:57.047 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:12:57.047 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:12:57.047 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:12:57.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.047 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:12:57.047 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:12:57.047 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:12:57.047 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:12:57.047 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:12:57.048 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:12:57.048 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:12:57.048 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:12:57.048 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:12:57.048 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:12:57.048 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:12:57.048 INFO:teuthology.orchestra.run.smithi090.stdout: "options": { 2024-04-14T15:12:57.048 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_max": 32, 2024-04-14T15:12:57.048 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_min": 1 2024-04-14T15:12:57.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.048 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:12:57.048 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": {} 2024-04-14T15:12:57.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.048 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:12:57.049 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:12:57.049 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 7.8899998664855957, 2024-04-14T15:12:57.049 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 7.8899998664855957, 2024-04-14T15:12:57.049 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 0.37999999523162842, 2024-04-14T15:12:57.049 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 3, 2024-04-14T15:12:57.049 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 3, 2024-04-14T15:12:57.049 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.90223121643066406, 2024-04-14T15:12:57.049 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 0.89999997615814209, 2024-04-14T15:12:57.049 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-04-14T15:12:57.049 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.049 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 2, 2024-04-14T15:12:57.050 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": ".rgw.root", 2024-04-14T15:12:57.050 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:09:41.608420+0000", 2024-04-14T15:12:57.050 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-14T15:12:57.050 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool", 2024-04-14T15:12:57.050 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:12:57.050 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:12:57.050 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:12:57.050 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:12:57.050 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:12:57.050 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:12:57.050 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:12:57.050 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:12:57.050 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:12:57.050 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:12:57.051 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:12:57.051 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:12:57.051 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:12:57.051 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:12:57.051 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:12:57.051 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:12:57.051 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:12:57.051 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:12:57.051 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:12:57.051 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:12:57.051 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:12:57.051 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:12:57.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.051 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "66", 2024-04-14T15:12:57.052 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:12:57.052 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:12:57.052 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:12:57.052 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:12:57.052 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:12:57.052 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:12:57.052 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:12:57.052 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:12:57.052 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:12:57.052 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:12:57.052 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:12:57.052 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:12:57.052 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:12:57.052 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:12:57.052 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:12:57.053 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:12:57.053 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:12:57.053 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:12:57.053 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:12:57.053 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:12:57.053 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:12:57.053 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:12:57.053 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:12:57.053 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:12:57.053 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:12:57.053 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:12:57.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.053 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:12:57.053 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:12:57.054 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:12:57.054 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:12:57.054 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:12:57.054 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:12:57.054 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:12:57.054 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:12:57.054 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:12:57.054 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:12:57.054 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:12:57.054 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:12:57.054 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:12:57.054 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw": {} 2024-04-14T15:12:57.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.054 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:12:57.054 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:12:57.055 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 1.5, 2024-04-14T15:12:57.055 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 1.5, 2024-04-14T15:12:57.055 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:12:57.055 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 1.5, 2024-04-14T15:12:57.055 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 1.5, 2024-04-14T15:12:57.055 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.90223121643066406, 2024-04-14T15:12:57.055 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 0.89999997615814209, 2024-04-14T15:12:57.055 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-04-14T15:12:57.055 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.055 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 3, 2024-04-14T15:12:57.055 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "datapool", 2024-04-14T15:12:57.055 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:09:41.744391+0000", 2024-04-14T15:12:57.056 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 8193, 2024-04-14T15:12:57.056 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-14T15:12:57.056 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:12:57.056 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:12:57.056 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:12:57.056 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:12:57.056 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:12:57.056 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:12:57.056 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:12:57.056 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:12:57.056 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:12:57.056 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:12:57.056 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 3, 2024-04-14T15:12:57.056 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 3, 2024-04-14T15:12:57.057 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 3, 2024-04-14T15:12:57.057 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 3, 2024-04-14T15:12:57.057 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 3, 2024-04-14T15:12:57.057 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:12:57.057 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:12:57.057 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:12:57.057 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:12:57.057 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:12:57.057 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:12:57.058 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:12:57.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.058 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "69", 2024-04-14T15:12:57.058 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:12:57.058 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:12:57.058 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:12:57.058 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:12:57.058 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:12:57.058 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 2, 2024-04-14T15:12:57.058 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 69, 2024-04-14T15:12:57.058 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:12:57.058 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:12:57.059 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:12:57.059 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:12:57.059 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:12:57.059 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:12:57.059 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:12:57.059 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:12:57.059 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:12:57.059 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:12:57.059 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:12:57.059 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:12:57.059 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:12:57.059 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:12:57.059 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:12:57.059 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:12:57.059 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:12:57.060 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:12:57.060 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:12:57.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.060 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:12:57.060 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:12:57.060 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:12:57.060 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:12:57.060 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:12:57.060 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:12:57.060 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:12:57.060 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:12:57.060 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:12:57.060 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:12:57.060 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:12:57.061 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:12:57.061 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:12:57.061 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd": {} 2024-04-14T15:12:57.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.061 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:12:57.061 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:12:57.061 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 2.6500000953674316, 2024-04-14T15:12:57.061 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 2.6500000953674316, 2024-04-14T15:12:57.061 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 0.87999999523162842, 2024-04-14T15:12:57.061 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 2.3299999237060547, 2024-04-14T15:12:57.061 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 2.3299999237060547, 2024-04-14T15:12:57.061 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.90223121643066406, 2024-04-14T15:12:57.061 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 0.89999997615814209, 2024-04-14T15:12:57.061 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-04-14T15:12:57.062 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.062 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 4, 2024-04-14T15:12:57.062 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "default.rgw.log", 2024-04-14T15:12:57.062 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:09:43.088157+0000", 2024-04-14T15:12:57.062 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-14T15:12:57.062 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool", 2024-04-14T15:12:57.062 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:12:57.062 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:12:57.062 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:12:57.062 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:12:57.062 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:12:57.063 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:12:57.063 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:12:57.063 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:12:57.063 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:12:57.063 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:12:57.063 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:12:57.063 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:12:57.063 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:12:57.063 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:12:57.063 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:12:57.063 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:12:57.063 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:12:57.063 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:12:57.063 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:12:57.064 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:12:57.064 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:12:57.064 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:12:57.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.065 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "68", 2024-04-14T15:12:57.065 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:12:57.065 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:12:57.065 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:12:57.065 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:12:57.066 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:12:57.066 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:12:57.066 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:12:57.066 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:12:57.066 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:12:57.066 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:12:57.066 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:12:57.066 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:12:57.066 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:12:57.066 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:12:57.066 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:12:57.066 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:12:57.066 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:12:57.066 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:12:57.066 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:12:57.067 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:12:57.067 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:12:57.067 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:12:57.067 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:12:57.067 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:12:57.067 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:12:57.067 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:12:57.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.067 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:12:57.067 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:12:57.067 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:12:57.067 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:12:57.067 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:12:57.067 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:12:57.068 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:12:57.068 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:12:57.068 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:12:57.068 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:12:57.068 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:12:57.068 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:12:57.068 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:12:57.068 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw": {} 2024-04-14T15:12:57.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.068 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:12:57.068 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:12:57.068 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 2.25, 2024-04-14T15:12:57.068 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 2.25, 2024-04-14T15:12:57.068 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:12:57.068 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 2.25, 2024-04-14T15:12:57.069 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 2.25, 2024-04-14T15:12:57.069 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.90223121643066406, 2024-04-14T15:12:57.069 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 0.89999997615814209, 2024-04-14T15:12:57.069 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-04-14T15:12:57.069 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.069 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 5, 2024-04-14T15:12:57.069 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "default.rgw.control", 2024-04-14T15:12:57.069 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:09:45.042722+0000", 2024-04-14T15:12:57.069 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-14T15:12:57.069 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool", 2024-04-14T15:12:57.069 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:12:57.069 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:12:57.069 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:12:57.070 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:12:57.070 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:12:57.070 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:12:57.070 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:12:57.070 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:12:57.070 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:12:57.070 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:12:57.070 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:12:57.070 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:12:57.070 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:12:57.070 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:12:57.070 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:12:57.070 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:12:57.070 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:12:57.070 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:12:57.071 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:12:57.071 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:12:57.071 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:12:57.071 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:12:57.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.071 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "70", 2024-04-14T15:12:57.071 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:12:57.071 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:12:57.071 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:12:57.071 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:12:57.071 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:12:57.071 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:12:57.071 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:12:57.071 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:12:57.071 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:12:57.071 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:12:57.072 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:12:57.072 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:12:57.072 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:12:57.072 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:12:57.072 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:12:57.072 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:12:57.072 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:12:57.072 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:12:57.072 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:12:57.072 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:12:57.072 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:12:57.072 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:12:57.072 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:12:57.073 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:12:57.073 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:12:57.073 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:12:57.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.073 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:12:57.073 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:12:57.073 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:12:57.073 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:12:57.073 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:12:57.073 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:12:57.073 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:12:57.073 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:12:57.073 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:12:57.073 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:12:57.073 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:12:57.074 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:12:57.074 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:12:57.074 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw": {} 2024-04-14T15:12:57.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.074 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:12:57.074 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:12:57.074 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 1.5, 2024-04-14T15:12:57.074 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 1.5, 2024-04-14T15:12:57.074 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:12:57.074 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 1.5, 2024-04-14T15:12:57.074 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 1.5, 2024-04-14T15:12:57.074 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.90223121643066406, 2024-04-14T15:12:57.074 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 0.89999997615814209, 2024-04-14T15:12:57.075 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-04-14T15:12:57.075 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.075 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 6, 2024-04-14T15:12:57.075 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "default.rgw.meta", 2024-04-14T15:12:57.075 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:09:47.111336+0000", 2024-04-14T15:12:57.075 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-14T15:12:57.075 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool", 2024-04-14T15:12:57.075 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:12:57.075 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:12:57.075 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:12:57.075 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:12:57.075 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:12:57.075 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:12:57.076 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:12:57.076 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:12:57.076 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:12:57.076 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:12:57.076 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:12:57.076 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:12:57.076 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:12:57.076 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:12:57.076 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:12:57.076 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:12:57.076 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:12:57.076 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:12:57.076 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:12:57.076 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:12:57.076 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:12:57.077 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:12:57.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.077 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "72", 2024-04-14T15:12:57.077 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:12:57.077 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:12:57.077 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:12:57.077 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:12:57.077 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:12:57.077 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:12:57.077 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:12:57.077 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:12:57.077 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:12:57.077 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:12:57.077 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:12:57.077 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:12:57.078 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:12:57.078 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:12:57.078 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:12:57.078 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:12:57.078 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:12:57.078 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:12:57.078 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:12:57.078 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:12:57.078 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:12:57.078 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:12:57.078 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:12:57.078 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:12:57.078 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:12:57.078 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:12:57.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.079 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:12:57.079 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:12:57.079 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:12:57.079 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:12:57.079 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:12:57.079 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:12:57.079 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:12:57.079 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:12:57.079 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:12:57.079 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:12:57.079 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:12:57.079 INFO:teuthology.orchestra.run.smithi090.stdout: "options": { 2024-04-14T15:12:57.079 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_bias": 4 2024-04-14T15:12:57.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.080 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:12:57.080 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw": {} 2024-04-14T15:12:57.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.080 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:12:57.080 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:12:57.080 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 1.75, 2024-04-14T15:12:57.080 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 1.75, 2024-04-14T15:12:57.080 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:12:57.080 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 1.75, 2024-04-14T15:12:57.080 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 1.75, 2024-04-14T15:12:57.080 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.90223121643066406, 2024-04-14T15:12:57.080 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 0.89999997615814209, 2024-04-14T15:12:57.080 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-04-14T15:12:57.080 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.081 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 7, 2024-04-14T15:12:57.081 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "TooBig_smithi090-93342-1", 2024-04-14T15:12:57.081 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:12:54.194258+0000", 2024-04-14T15:12:57.081 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 32769, 2024-04-14T15:12:57.081 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,creating", 2024-04-14T15:12:57.081 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:12:57.081 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:12:57.081 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:12:57.081 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:12:57.081 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:12:57.081 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:12:57.081 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:12:57.081 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:12:57.082 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:12:57.082 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:12:57.082 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:12:57.082 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:12:57.082 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:12:57.082 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:12:57.082 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:12:57.082 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:12:57.082 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:12:57.082 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:12:57.082 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:12:57.082 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:12:57.082 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:12:57.082 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:12:57.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.083 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "108", 2024-04-14T15:12:57.083 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:12:57.083 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:12:57.083 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:12:57.083 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:12:57.083 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:12:57.083 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:12:57.083 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:12:57.083 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:12:57.083 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:12:57.083 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:12:57.083 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:12:57.083 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:12:57.083 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:12:57.084 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:12:57.084 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:12:57.084 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:12:57.084 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:12:57.084 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:12:57.084 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:12:57.084 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:12:57.084 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:12:57.084 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:12:57.084 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:12:57.084 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:12:57.084 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:12:57.084 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:12:57.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.085 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:12:57.085 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:12:57.085 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:12:57.085 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:12:57.085 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:12:57.085 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:12:57.085 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:12:57.085 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:12:57.085 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:12:57.085 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:12:57.085 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:12:57.085 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:12:57.085 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:12:57.085 INFO:teuthology.orchestra.run.smithi090.stdout: "rados": {} 2024-04-14T15:12:57.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.086 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:12:57.086 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:12:57.086 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 1.75, 2024-04-14T15:12:57.086 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 1.75, 2024-04-14T15:12:57.086 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:12:57.086 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 1.75, 2024-04-14T15:12:57.086 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 1.75, 2024-04-14T15:12:57.086 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.90223121643066406, 2024-04-14T15:12:57.086 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 0.89999997615814209, 2024-04-14T15:12:57.086 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-04-14T15:12:57.086 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.086 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 8, 2024-04-14T15:12:57.087 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "LibRadosIo_smithi090-93379-1", 2024-04-14T15:12:57.087 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:12:54.203865+0000", 2024-04-14T15:12:57.087 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 32769, 2024-04-14T15:12:57.087 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,creating", 2024-04-14T15:12:57.087 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:12:57.087 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:12:57.087 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:12:57.087 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:12:57.087 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:12:57.087 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:12:57.087 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:12:57.087 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:12:57.087 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:12:57.087 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:12:57.087 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:12:57.088 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:12:57.088 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:12:57.088 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:12:57.088 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:12:57.088 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:12:57.088 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:12:57.088 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:12:57.088 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:12:57.088 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:12:57.088 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:12:57.088 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:12:57.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.088 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "108", 2024-04-14T15:12:57.088 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:12:57.088 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:12:57.089 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:12:57.089 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:12:57.089 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:12:57.089 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:12:57.089 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:12:57.089 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:12:57.089 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:12:57.089 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:12:57.089 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:12:57.089 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:12:57.089 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:12:57.089 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:12:57.089 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:12:57.089 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:12:57.089 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:12:57.090 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:12:57.090 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:12:57.090 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:12:57.090 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:12:57.091 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:12:57.091 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:12:57.091 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:12:57.091 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:12:57.091 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:12:57.091 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:12:57.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.092 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:12:57.092 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:12:57.092 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:12:57.092 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:12:57.092 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:12:57.092 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:12:57.092 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:12:57.092 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:12:57.092 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:12:57.092 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:12:57.092 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:12:57.092 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:12:57.092 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:12:57.093 INFO:teuthology.orchestra.run.smithi090.stdout: "rados": {} 2024-04-14T15:12:57.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.093 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:12:57.093 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:12:57.093 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 1.75, 2024-04-14T15:12:57.093 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 1.75, 2024-04-14T15:12:57.093 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:12:57.093 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 1.75, 2024-04-14T15:12:57.093 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 1.75, 2024-04-14T15:12:57.093 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.90223121643066406, 2024-04-14T15:12:57.093 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 0.89999997615814209, 2024-04-14T15:12:57.093 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-04-14T15:12:57.093 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.094 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 9, 2024-04-14T15:12:57.094 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "TooBigPP_smithi090-93363-1", 2024-04-14T15:12:57.094 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:12:54.211449+0000", 2024-04-14T15:12:57.094 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 32769, 2024-04-14T15:12:57.094 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,creating", 2024-04-14T15:12:57.094 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:12:57.094 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:12:57.094 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:12:57.094 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:12:57.094 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:12:57.094 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:12:57.094 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:12:57.094 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:12:57.094 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:12:57.094 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:12:57.095 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:12:57.095 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:12:57.095 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:12:57.095 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:12:57.095 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:12:57.095 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:12:57.095 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:12:57.095 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:12:57.095 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:12:57.095 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:12:57.095 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:12:57.095 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:12:57.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.095 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "108", 2024-04-14T15:12:57.095 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:12:57.096 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:12:57.096 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:12:57.096 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:12:57.096 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:12:57.096 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:12:57.096 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:12:57.096 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:12:57.096 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:12:57.096 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:12:57.096 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:12:57.096 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:12:57.096 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:12:57.096 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:12:57.096 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:12:57.096 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:12:57.097 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:12:57.097 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:12:57.097 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:12:57.097 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:12:57.097 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:12:57.097 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:12:57.097 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:12:57.097 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:12:57.097 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:12:57.097 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:12:57.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.097 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:12:57.098 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:12:57.098 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:12:57.098 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:12:57.098 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:12:57.098 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:12:57.098 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:12:57.098 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:12:57.098 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:12:57.098 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:12:57.098 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:12:57.098 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:12:57.098 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:12:57.098 INFO:teuthology.orchestra.run.smithi090.stdout: "rados": {} 2024-04-14T15:12:57.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.098 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:12:57.099 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:12:57.099 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 2, 2024-04-14T15:12:57.099 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 2, 2024-04-14T15:12:57.099 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:12:57.099 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 2, 2024-04-14T15:12:57.099 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 2, 2024-04-14T15:12:57.099 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.90223121643066406, 2024-04-14T15:12:57.099 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 0.89999997615814209, 2024-04-14T15:12:57.099 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-04-14T15:12:57.099 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.099 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 10, 2024-04-14T15:12:57.099 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "LibRadosIoPP_smithi090-93398-1", 2024-04-14T15:12:57.099 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:12:54.219277+0000", 2024-04-14T15:12:57.100 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 32769, 2024-04-14T15:12:57.100 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,creating", 2024-04-14T15:12:57.100 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:12:57.100 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:12:57.100 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:12:57.100 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:12:57.100 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:12:57.100 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:12:57.100 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:12:57.100 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:12:57.100 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:12:57.100 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:12:57.100 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:12:57.100 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:12:57.100 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:12:57.101 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:12:57.101 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:12:57.101 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:12:57.101 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:12:57.101 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:12:57.101 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:12:57.101 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:12:57.101 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:12:57.101 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:12:57.101 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.101 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "108", 2024-04-14T15:12:57.101 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:12:57.101 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:12:57.101 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:12:57.101 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:12:57.102 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:12:57.102 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:12:57.102 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:12:57.102 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:12:57.102 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:12:57.102 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:12:57.102 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:12:57.102 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:12:57.102 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:12:57.102 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:12:57.102 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:12:57.102 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:12:57.102 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:12:57.102 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:12:57.102 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:12:57.103 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:12:57.103 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:12:57.103 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:12:57.103 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:12:57.103 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:12:57.103 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:12:57.103 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:12:57.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.103 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:12:57.103 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:12:57.103 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:12:57.103 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:12:57.103 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:12:57.103 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:12:57.103 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:12:57.104 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:12:57.104 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:12:57.104 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:12:57.104 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:12:57.104 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:12:57.104 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:12:57.104 INFO:teuthology.orchestra.run.smithi090.stdout: "rados": {} 2024-04-14T15:12:57.104 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.104 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:12:57.104 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:12:57.104 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 2.5, 2024-04-14T15:12:57.104 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 2.5, 2024-04-14T15:12:57.104 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:12:57.104 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 2.5, 2024-04-14T15:12:57.104 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 2.5, 2024-04-14T15:12:57.105 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.90223121643066406, 2024-04-14T15:12:57.105 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 0.89999997615814209, 2024-04-14T15:12:57.105 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-04-14T15:12:57.105 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.105 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 11, 2024-04-14T15:12:57.105 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "ceph_test_rados_api_asio", 2024-04-14T15:12:57.105 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:12:54.225671+0000", 2024-04-14T15:12:57.105 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 32769, 2024-04-14T15:12:57.105 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,creating", 2024-04-14T15:12:57.105 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:12:57.105 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:12:57.105 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:12:57.105 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:12:57.106 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:12:57.106 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:12:57.106 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:12:57.106 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:12:57.106 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:12:57.106 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:12:57.106 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:12:57.106 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:12:57.106 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:12:57.106 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:12:57.106 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:12:57.106 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:12:57.106 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:12:57.106 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:12:57.107 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:12:57.107 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:12:57.107 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:12:57.107 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:12:57.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.107 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "107", 2024-04-14T15:12:57.107 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:12:57.107 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:12:57.107 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:12:57.107 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:12:57.107 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:12:57.108 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:12:57.108 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:12:57.108 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:12:57.108 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:12:57.108 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:12:57.108 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:12:57.108 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:12:57.108 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:12:57.108 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:12:57.108 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:12:57.108 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:12:57.108 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:12:57.108 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:12:57.109 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:12:57.109 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:12:57.109 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:12:57.109 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:12:57.109 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:12:57.109 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:12:57.109 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:12:57.109 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:12:57.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.109 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:12:57.109 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:12:57.109 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:12:57.109 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:12:57.109 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:12:57.110 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:12:57.110 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:12:57.110 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:12:57.110 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:12:57.110 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:12:57.110 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:12:57.110 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:12:57.110 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": {}, 2024-04-14T15:12:57.110 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:12:57.110 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:12:57.110 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 1.75, 2024-04-14T15:12:57.110 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 1.75, 2024-04-14T15:12:57.110 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:12:57.110 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 1.75, 2024-04-14T15:12:57.110 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 1.75, 2024-04-14T15:12:57.111 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.90223121643066406, 2024-04-14T15:12:57.111 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 0.89999997615814209, 2024-04-14T15:12:57.111 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-04-14T15:12:57.111 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.111 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.111 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.111 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 12, 2024-04-14T15:12:57.111 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "LibRadosLock_smithi090-93436-1", 2024-04-14T15:12:57.111 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:12:54.234106+0000", 2024-04-14T15:12:57.111 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 32769, 2024-04-14T15:12:57.111 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,creating", 2024-04-14T15:12:57.111 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:12:57.111 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:12:57.111 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:12:57.111 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:12:57.112 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:12:57.112 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:12:57.112 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:12:57.112 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:12:57.112 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:12:57.112 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:12:57.112 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:12:57.112 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:12:57.112 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:12:57.112 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:12:57.112 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:12:57.112 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:12:57.112 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:12:57.112 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:12:57.112 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:12:57.113 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:12:57.113 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:12:57.113 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:12:57.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.113 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "108", 2024-04-14T15:12:57.113 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:12:57.113 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:12:57.113 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:12:57.113 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:12:57.113 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:12:57.113 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:12:57.113 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:12:57.113 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:12:57.113 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:12:57.114 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:12:57.114 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:12:57.114 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:12:57.114 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:12:57.114 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:12:57.114 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:12:57.114 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:12:57.114 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:12:57.114 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:12:57.114 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:12:57.114 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:12:57.114 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:12:57.114 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:12:57.114 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:12:57.114 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:12:57.115 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:12:57.115 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:12:57.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.115 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:12:57.115 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:12:57.115 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:12:57.115 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:12:57.115 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:12:57.115 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:12:57.115 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:12:57.115 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:12:57.115 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:12:57.115 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:12:57.115 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:12:57.115 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:12:57.116 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:12:57.116 INFO:teuthology.orchestra.run.smithi090.stdout: "rados": {} 2024-04-14T15:12:57.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.116 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:12:57.116 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:12:57.116 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 2, 2024-04-14T15:12:57.116 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 2, 2024-04-14T15:12:57.116 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:12:57.116 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 2, 2024-04-14T15:12:57.116 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 2, 2024-04-14T15:12:57.116 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.90223121643066406, 2024-04-14T15:12:57.116 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 0.89999997615814209, 2024-04-14T15:12:57.116 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-04-14T15:12:57.116 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.117 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.117 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 13, 2024-04-14T15:12:57.117 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "LibRadosList_smithi090-93444-1", 2024-04-14T15:12:57.117 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:12:54.244611+0000", 2024-04-14T15:12:57.117 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 32769, 2024-04-14T15:12:57.117 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,creating", 2024-04-14T15:12:57.117 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:12:57.117 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:12:57.117 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:12:57.117 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:12:57.117 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:12:57.117 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:12:57.117 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:12:57.117 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:12:57.117 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:12:57.117 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:12:57.118 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:12:57.118 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:12:57.118 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:12:57.118 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:12:57.118 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:12:57.118 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:12:57.118 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:12:57.118 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:12:57.118 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:12:57.118 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:12:57.118 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:12:57.118 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:12:57.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.118 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "108", 2024-04-14T15:12:57.119 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:12:57.119 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:12:57.119 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:12:57.119 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:12:57.119 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:12:57.119 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:12:57.119 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:12:57.119 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:12:57.119 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:12:57.120 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:12:57.120 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:12:57.120 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:12:57.120 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:12:57.120 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:12:57.120 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:12:57.120 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:12:57.120 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:12:57.120 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:12:57.120 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:12:57.120 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:12:57.120 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:12:57.120 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:12:57.120 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:12:57.121 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:12:57.121 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:12:57.121 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:12:57.121 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.121 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:12:57.121 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:12:57.121 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:12:57.121 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:12:57.121 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:12:57.121 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:12:57.121 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:12:57.121 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:12:57.121 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:12:57.121 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:12:57.121 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:12:57.121 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:12:57.122 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:12:57.122 INFO:teuthology.orchestra.run.smithi090.stdout: "rados": {} 2024-04-14T15:12:57.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.122 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:12:57.122 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:12:57.122 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 1.75, 2024-04-14T15:12:57.122 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 1.75, 2024-04-14T15:12:57.122 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:12:57.122 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 1.75, 2024-04-14T15:12:57.122 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 1.75, 2024-04-14T15:12:57.122 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.90223121643066406, 2024-04-14T15:12:57.122 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 0.89999997615814209, 2024-04-14T15:12:57.122 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-04-14T15:12:57.122 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.123 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 14, 2024-04-14T15:12:57.123 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "LibRadosLockPP_smithi090-93476-1", 2024-04-14T15:12:57.123 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:12:54.263258+0000", 2024-04-14T15:12:57.123 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 32769, 2024-04-14T15:12:57.123 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,creating", 2024-04-14T15:12:57.123 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:12:57.123 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:12:57.123 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:12:57.123 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:12:57.123 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:12:57.123 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:12:57.123 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:12:57.123 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:12:57.124 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:12:57.124 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:12:57.124 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:12:57.124 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:12:57.124 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:12:57.124 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:12:57.124 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:12:57.124 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:12:57.124 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:12:57.124 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:12:57.124 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:12:57.124 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:12:57.124 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:12:57.124 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:12:57.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.125 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "108", 2024-04-14T15:12:57.125 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:12:57.125 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:12:57.125 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:12:57.125 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:12:57.125 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:12:57.125 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:12:57.125 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:12:57.125 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:12:57.125 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:12:57.125 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:12:57.125 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:12:57.125 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:12:57.125 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:12:57.125 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:12:57.126 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:12:57.126 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:12:57.126 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:12:57.126 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:12:57.126 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:12:57.126 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:12:57.126 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:12:57.126 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:12:57.126 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:12:57.126 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:12:57.126 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:12:57.126 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:12:57.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.126 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:12:57.127 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:12:57.127 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:12:57.127 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:12:57.127 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:12:57.127 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:12:57.127 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:12:57.127 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:12:57.127 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:12:57.127 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:12:57.127 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:12:57.127 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:12:57.127 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:12:57.127 INFO:teuthology.orchestra.run.smithi090.stdout: "rados": {} 2024-04-14T15:12:57.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.127 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:12:57.128 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:12:57.128 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 2.75, 2024-04-14T15:12:57.128 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 2.75, 2024-04-14T15:12:57.128 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:12:57.128 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 2.75, 2024-04-14T15:12:57.128 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 2.75, 2024-04-14T15:12:57.128 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.90223121643066406, 2024-04-14T15:12:57.128 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 0.89999997615814209, 2024-04-14T15:12:57.128 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-04-14T15:12:57.128 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.128 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 15, 2024-04-14T15:12:57.128 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "test-rados-api-smithi090-93656-1", 2024-04-14T15:12:57.129 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:12:54.290883+0000", 2024-04-14T15:12:57.129 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 32769, 2024-04-14T15:12:57.129 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,creating", 2024-04-14T15:12:57.129 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:12:57.129 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:12:57.129 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:12:57.129 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:12:57.129 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:12:57.129 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:12:57.129 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:12:57.129 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:12:57.129 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:12:57.129 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:12:57.129 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:12:57.129 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:12:57.130 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:12:57.130 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:12:57.130 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:12:57.130 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:12:57.130 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:12:57.130 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:12:57.130 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:12:57.130 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:12:57.130 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:12:57.130 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:12:57.130 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.130 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "108", 2024-04-14T15:12:57.130 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:12:57.130 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:12:57.131 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:12:57.131 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:12:57.131 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:12:57.131 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:12:57.131 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:12:57.131 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:12:57.131 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:12:57.131 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:12:57.131 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:12:57.131 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:12:57.131 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:12:57.131 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:12:57.131 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:12:57.131 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:12:57.131 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:12:57.132 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:12:57.132 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:12:57.132 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:12:57.132 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:12:57.132 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:12:57.132 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:12:57.132 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:12:57.132 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:12:57.132 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:12:57.132 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.132 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:12:57.132 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:12:57.132 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:12:57.132 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:12:57.132 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:12:57.133 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:12:57.133 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:12:57.133 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:12:57.133 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:12:57.133 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:12:57.133 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:12:57.133 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:12:57.133 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:12:57.133 INFO:teuthology.orchestra.run.smithi090.stdout: "rados": {} 2024-04-14T15:12:57.133 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.133 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:12:57.133 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:12:57.133 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 2, 2024-04-14T15:12:57.133 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 2, 2024-04-14T15:12:57.134 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:12:57.134 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 2, 2024-04-14T15:12:57.134 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 2, 2024-04-14T15:12:57.134 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.90223121643066406, 2024-04-14T15:12:57.134 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 0.89999997615814209, 2024-04-14T15:12:57.134 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-04-14T15:12:57.134 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.134 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.134 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.134 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 16, 2024-04-14T15:12:57.134 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "LibRadosMiscPP_smithi090-93601-1", 2024-04-14T15:12:57.134 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:12:54.300251+0000", 2024-04-14T15:12:57.134 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 32769, 2024-04-14T15:12:57.134 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,creating", 2024-04-14T15:12:57.135 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:12:57.135 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:12:57.135 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:12:57.135 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:12:57.135 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:12:57.135 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:12:57.135 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:12:57.135 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:12:57.135 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:12:57.135 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:12:57.135 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:12:57.135 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:12:57.135 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:12:57.135 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:12:57.136 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:12:57.136 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:12:57.136 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:12:57.136 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:12:57.136 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:12:57.136 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:12:57.136 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:12:57.136 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:12:57.136 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.136 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "108", 2024-04-14T15:12:57.136 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:12:57.136 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:12:57.136 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:12:57.136 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:12:57.136 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:12:57.137 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:12:57.137 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:12:57.137 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:12:57.137 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:12:57.137 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:12:57.137 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:12:57.137 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:12:57.137 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:12:57.137 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:12:57.137 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:12:57.137 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:12:57.137 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:12:57.138 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : oid=14 cu handler_error: Running main() from gmock_main.cc 2024-04-14T15:12:57.138 INFO:tasks.workunit.client.0.smithi090.stdout: handler_error: [==========] Running 1 test from 1 test suite. 2024-04-14T15:12:57.138 INFO:tasks.workunit.client.0.smithi090.stdout: handler_error: [----------] Global test environment set-up. 2024-04-14T15:12:57.139 INFO:tasks.workunit.client.0.smithi090.stdout: handler_error: [----------] 1 test from neocls_handler_error 2024-04-14T15:12:57.139 INFO:tasks.workunit.client.0.smithi090.stdout: handler_error: [ RUN ] neocls_handler_error.test_handler_error 2024-04-14T15:12:57.139 INFO:tasks.workunit.client.0.smithi090.stdout: handler_error: [ OK ] neocls_handler_error.test_handler_error (2553 ms) 2024-04-14T15:12:57.139 INFO:tasks.workunit.client.0.smithi090.stdout: handler_error: [----------] 1 test from neocls_handler_error (2553 ms total) 2024-04-14T15:12:57.139 INFO:tasks.workunit.client.0.smithi090.stdout: handler_error: 2024-04-14T15:12:57.139 INFO:tasks.workunit.client.0.smithi090.stdout: handler_error: [----------] Global test environment tear-down 2024-04-14T15:12:57.139 INFO:tasks.workunit.client.0.smithi090.stdout: handler_error: [==========] 1 test from 1 test suite ran. (2553 ms total) 2024-04-14T15:12:57.139 INFO:tasks.workunit.client.0.smithi090.stdout: handler_error: [ PASSED ] 1 test. 2024-04-14T15:12:57.139 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:12:57.139 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:12:57.139 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:12:57.140 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:12:57.140 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:12:57.140 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:12:57.140 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:12:57.140 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:12:57.140 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:12:57.140 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.140 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:12:57.140 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:12:57.140 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:12:57.140 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:12:57.140 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:12:57.140 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:12:57.140 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:12:57.141 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:12:57.141 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:12:57.141 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:12:57.141 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:12:57.141 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:12:57.141 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:12:57.141 INFO:teuthology.orchestra.run.smithi090.stdout: "rados": {} 2024-04-14T15:12:57.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.141 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:12:57.141 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:12:57.141 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 2, 2024-04-14T15:12:57.141 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 2, 2024-04-14T15:12:57.141 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:12:57.141 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 2, 2024-04-14T15:12:57.141 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 2, 2024-04-14T15:12:57.142 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.90223121643066406, 2024-04-14T15:12:57.142 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 0.89999997615814209, 2024-04-14T15:12:57.142 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-04-14T15:12:57.142 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.142 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.142 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.142 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 17, 2024-04-14T15:12:57.142 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "LibRadosSnapshotsPP_smithi090-93724-1", 2024-04-14T15:12:57.142 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:12:54.309050+0000", 2024-04-14T15:12:57.142 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 32769, 2024-04-14T15:12:57.142 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,creating", 2024-04-14T15:12:57.142 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:12:57.142 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:12:57.142 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:12:57.143 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:12:57.143 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:12:57.143 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:12:57.143 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:12:57.143 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:12:57.143 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:12:57.143 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:12:57.143 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:12:57.143 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:12:57.143 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:12:57.143 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:12:57.143 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:12:57.143 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:12:57.143 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:12:57.143 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:12:57.144 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:12:57.144 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:12:57.144 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:12:57.144 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:12:57.144 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.144 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "108", 2024-04-14T15:12:57.144 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:12:57.144 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:12:57.144 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:12:57.144 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:12:57.144 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:12:57.144 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:12:57.144 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:12:57.144 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:12:57.145 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:12:57.145 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:12:57.145 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:12:57.145 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:12:57.145 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:12:57.145 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:12:57.145 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:12:57.145 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:12:57.145 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:12:57.145 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:12:57.145 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:12:57.145 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:12:57.145 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:12:57.145 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:12:57.145 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:12:57.145 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:12:57.146 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:12:57.146 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:12:57.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.146 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:12:57.146 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:12:57.146 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:12:57.146 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:12:57.146 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:12:57.146 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:12:57.146 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:12:57.146 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:12:57.146 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:12:57.146 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:12:57.146 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:12:57.147 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:12:57.147 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:12:57.147 INFO:teuthology.orchestra.run.smithi090.stdout: "rados": {} 2024-04-14T15:12:57.147 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.147 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:12:57.147 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:12:57.147 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 2, 2024-04-14T15:12:57.147 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 2, 2024-04-14T15:12:57.147 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:12:57.147 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 2, 2024-04-14T15:12:57.147 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 2, 2024-04-14T15:12:57.147 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.90223121643066406, 2024-04-14T15:12:57.147 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 0.89999997615814209, 2024-04-14T15:12:57.147 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-04-14T15:12:57.148 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.148 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.148 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.148 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 18, 2024-04-14T15:12:57.148 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "LibRadosSnapshots_smithi090-93709-1", 2024-04-14T15:12:57.148 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:12:54.322168+0000", 2024-04-14T15:12:57.148 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 32769, 2024-04-14T15:12:57.148 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,creating", 2024-04-14T15:12:57.148 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:12:57.148 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:12:57.148 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:12:57.148 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:12:57.148 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:12:57.148 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:12:57.148 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:12:57.149 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:12:57.149 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:12:57.149 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:12:57.149 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:12:57.149 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:12:57.149 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:12:57.149 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:12:57.149 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:12:57.149 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:12:57.149 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:12:57.149 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:12:57.149 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:12:57.149 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:12:57.149 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:12:57.149 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:12:57.149 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.150 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "108", 2024-04-14T15:12:57.150 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:12:57.150 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:12:57.150 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:12:57.150 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:12:57.150 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:12:57.150 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:12:57.150 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:12:57.150 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:12:57.150 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:12:57.150 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:12:57.150 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:12:57.150 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:12:57.150 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:12:57.151 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:12:57.151 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:12:57.151 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:12:57.151 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:12:57.151 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:12:57.151 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:12:57.151 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:12:57.151 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:12:57.151 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:12:57.151 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:12:57.151 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:12:57.151 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:12:57.151 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:12:57.151 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.151 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:12:57.152 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:12:57.152 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:12:57.152 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:12:57.152 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:12:57.152 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:12:57.152 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:12:57.152 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:12:57.152 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:12:57.152 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:12:57.152 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:12:57.152 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:12:57.152 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:12:57.152 INFO:teuthology.orchestra.run.smithi090.stdout: "rados": {} 2024-04-14T15:12:57.152 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.152 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:12:57.153 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:12:57.153 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 1.75, 2024-04-14T15:12:57.153 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 1.75, 2024-04-14T15:12:57.153 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:12:57.153 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 1.75, 2024-04-14T15:12:57.153 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 1.75, 2024-04-14T15:12:57.153 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.90223121643066406, 2024-04-14T15:12:57.153 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 0.89999997615814209, 2024-04-14T15:12:57.153 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-04-14T15:12:57.153 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.153 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.153 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.153 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 19, 2024-04-14T15:12:57.153 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "LibRadosStat_smithi090-93731-1", 2024-04-14T15:12:57.153 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:12:54.329704+0000", 2024-04-14T15:12:57.154 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 32769, 2024-04-14T15:12:57.154 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,creating", 2024-04-14T15:12:57.154 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:12:57.154 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:12:57.154 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:12:57.154 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:12:57.154 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:12:57.154 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:12:57.154 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:12:57.154 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:12:57.154 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:12:57.154 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:12:57.155 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_read_operations: Running main() from gmock_main.cc 2024-04-14T15:12:57.155 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_read_operations: [==========] Running 17 tests from 1 test suite. 2024-04-14T15:12:57.155 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_read_operations: [----------] Global test environment set-up. 2024-04-14T15:12:57.155 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest 2024-04-14T15:12:57.155 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2024-04-14T15:12:57.155 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2024-04-14T15:12:57.155 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2024-04-14T15:12:57.155 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (504 ms) 2024-04-14T15:12:57.155 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2024-04-14T15:12:57.155 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (9 ms) 2024-04-14T15:12:57.155 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2024-04-14T15:12:57.155 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (43 ms) 2024-04-14T15:12:57.155 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2024-04-14T15:12:57.156 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (31 ms) 2024-04-14T15:12:57.156 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2024-04-14T15:12:57.156 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (9 ms) 2024-04-14T15:12:57.156 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2024-04-14T15:12:57.156 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (10 ms) 2024-04-14T15:12:57.156 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2024-04-14T15:12:57.156 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (8 ms) 2024-04-14T15:12:57.156 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2024-04-14T15:12:57.156 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (7 ms) 2024-04-14T15:12:57.156 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2024-04-14T15:12:57.156 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (4 ms) 2024-04-14T15:12:57.156 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2024-04-14T15:12:57.156 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (6 ms) 2024-04-14T15:12:57.156 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2024-04-14T15:12:57.157 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (9 ms) 2024-04-14T15:12:57.157 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat2 2024-04-14T15:12:57.157 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat2 (7 ms) 2024-04-14T15:12:57.157 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2024-04-14T15:12:57.157 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (13 ms) 2024-04-14T15:12:57.157 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2024-04-14T15:12:57.157 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (15 ms) 2024-04-14T15:12:57.157 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2024-04-14T15:12:57.157 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (9 ms) 2024-04-14T15:12:57.157 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2024-04-14T15:12:57.157 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (4 ms) 2024-04-14T15:12:57.157 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest (688 ms total) 2024-04-14T15:12:57.157 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_read_operations: 2024-04-14T15:12:57.157 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_read_operations: [----------] Global test environment tear-down 2024-04-14T15:12:57.157 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_read_operations: [==========] 17 tests from 1 test suite ran. (2711 ms total) 2024-04-14T15:12:57.157 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_read_operations: [ PASSED ] 17 tests. 2024-04-14T15:12:57.158 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:12:57.158 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:12:57.158 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:12:57.158 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:12:57.158 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:12:57.158 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:12:57.158 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:12:57.158 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:12:57.158 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:12:57.158 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:12:57.159 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:12:57.159 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:12:57.159 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.159 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "108", 2024-04-14T15:12:57.159 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:12:57.159 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:12:57.159 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:12:57.159 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:12:57.159 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:12:57.159 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:12:57.159 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:12:57.159 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:12:57.159 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:12:57.159 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:12:57.159 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:12:57.160 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:12:57.160 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:12:57.160 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:12:57.160 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:12:57.160 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:12:57.160 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:12:57.160 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:12:57.160 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:12:57.160 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:12:57.160 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:12:57.160 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:12:57.160 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:12:57.160 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:12:57.160 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:12:57.161 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:12:57.161 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.161 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:12:57.161 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:12:57.161 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:12:57.161 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:12:57.161 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:12:57.161 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:12:57.161 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:12:57.161 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:12:57.161 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:12:57.161 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:12:57.161 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:12:57.161 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:12:57.161 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:12:57.162 INFO:teuthology.orchestra.run.smithi090.stdout: "rados": {} 2024-04-14T15:12:57.162 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.162 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:12:57.162 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:12:57.162 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 1.75, 2024-04-14T15:12:57.162 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 1.75, 2024-04-14T15:12:57.162 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:12:57.162 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 1.75, 2024-04-14T15:12:57.162 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 1.75, 2024-04-14T15:12:57.162 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.90223121643066406, 2024-04-14T15:12:57.162 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 0.89999997615814209, 2024-04-14T15:12:57.162 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-04-14T15:12:57.162 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.162 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.163 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.163 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 20, 2024-04-14T15:12:57.163 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "LibRadosTierPP_smithi090-93652-1", 2024-04-14T15:12:57.163 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:12:54.336842+0000", 2024-04-14T15:12:57.163 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 32769, 2024-04-14T15:12:57.163 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,creating", 2024-04-14T15:12:57.163 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:12:57.163 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:12:57.163 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:12:57.163 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:12:57.163 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:12:57.163 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:12:57.163 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:12:57.163 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:12:57.164 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:12:57.164 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:12:57.164 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:12:57.164 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:12:57.164 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:12:57.164 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:12:57.164 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:12:57.164 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:12:57.164 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:12:57.164 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:12:57.164 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:12:57.164 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:12:57.165 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:12:57.165 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:12:57.165 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.165 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "108", 2024-04-14T15:12:57.165 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:12:57.165 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:12:57.165 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:12:57.165 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:12:57.165 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:12:57.165 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:12:57.165 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:12:57.165 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:12:57.165 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:12:57.165 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:12:57.165 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:12:57.166 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:12:57.166 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:12:57.166 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:12:57.166 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:12:57.166 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:12:57.166 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:12:57.166 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:12:57.166 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:12:57.166 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:12:57.166 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:12:57.166 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:12:57.166 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:12:57.166 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:12:57.166 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:12:57.167 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:12:57.167 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.167 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:12:57.167 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:12:57.167 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:12:57.167 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:12:57.167 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:12:57.167 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:12:57.167 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:12:57.167 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:12:57.167 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:12:57.167 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:12:57.167 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:12:57.167 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:12:57.167 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:12:57.168 INFO:teuthology.orchestra.run.smithi090.stdout: "rados": {} 2024-04-14T15:12:57.168 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.168 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:12:57.168 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:12:57.168 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 2, 2024-04-14T15:12:57.168 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 2, 2024-04-14T15:12:57.168 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:12:57.168 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 2, 2024-04-14T15:12:57.168 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 2, 2024-04-14T15:12:57.168 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.90223121643066406, 2024-04-14T15:12:57.168 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 0.89999997615814209, 2024-04-14T15:12:57.168 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-04-14T15:12:57.168 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.168 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.168 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.169 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 21, 2024-04-14T15:12:57.169 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "LibRadosStatPP_smithi090-93864-1", 2024-04-14T15:12:57.169 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:12:54.363752+0000", 2024-04-14T15:12:57.169 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 32769, 2024-04-14T15:12:57.169 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,creating", 2024-04-14T15:12:57.169 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:12:57.169 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:12:57.169 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:12:57.169 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:12:57.169 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:12:57.169 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:12:57.169 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:12:57.169 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:12:57.169 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:12:57.169 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:12:57.170 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:12:57.170 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:12:57.170 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:12:57.170 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:12:57.170 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:12:57.170 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:12:57.170 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:12:57.170 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:12:57.170 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:12:57.170 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:12:57.170 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:12:57.170 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:12:57.170 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.170 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "108", 2024-04-14T15:12:57.170 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:12:57.171 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:12:57.171 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:12:57.171 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:12:57.171 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:12:57.171 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:12:57.171 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:12:57.171 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:12:57.171 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:12:57.171 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:12:57.171 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:12:57.171 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:12:57.171 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:12:57.171 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:12:57.171 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:12:57.171 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:12:57.172 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:12:57.172 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:12:57.172 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:12:57.172 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:12:57.172 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:12:57.172 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:12:57.172 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:12:57.172 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:12:57.172 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:12:57.172 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:12:57.172 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.172 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:12:57.172 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:12:57.172 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:12:57.172 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:12:57.173 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:12:57.173 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:12:57.173 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:12:57.173 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:12:57.173 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:12:57.173 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:12:57.173 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:12:57.173 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:12:57.173 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:12:57.173 INFO:teuthology.orchestra.run.smithi090.stdout: "rados": {} 2024-04-14T15:12:57.173 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.174 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd_pp: Running main() from gmock_main.cc 2024-04-14T15:12:57.174 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2024-04-14T15:12:57.174 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd_pp: [----------] Global test environment set-up. 2024-04-14T15:12:57.174 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2024-04-14T15:12:57.174 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2024-04-14T15:12:57.174 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (52 ms) 2024-04-14T15:12:57.174 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2024-04-14T15:12:57.174 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (72 ms) 2024-04-14T15:12:57.174 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2024-04-14T15:12:57.174 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (2668 ms) 2024-04-14T15:12:57.174 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (2793 ms total) 2024-04-14T15:12:57.174 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd_pp: 2024-04-14T15:12:57.174 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd_pp: [----------] Global test environment tear-down 2024-04-14T15:12:57.175 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (2793 ms total) 2024-04-14T15:12:57.175 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2024-04-14T15:12:57.175 INFO:tasks.workunit.client.0.smithi090.stdout: api_cls_remote_reads: [==========] Running 1 test from 1 test suite. 2024-04-14T15:12:57.176 INFO:tasks.workunit.client.0.smithi090.stdout: api_cls_remote_reads: [----------] Global test environment set-up. 2024-04-14T15:12:57.176 INFO:tasks.workunit.client.0.smithi090.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads 2024-04-14T15:12:57.176 INFO:tasks.workunit.client.0.smithi090.stdout: api_cls_remote_reads: [ RUN ] ClsTestRemoteReads.TestGather 2024-04-14T15:12:57.176 INFO:tasks.workunit.client.0.smithi090.stdout: api_cls_remote_reads: [ OK ] ClsTestRemoteReads.TestGather (2671 ms) 2024-04-14T15:12:57.176 INFO:tasks.workunit.client.0.smithi090.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads (2671 ms total) 2024-04-14T15:12:57.176 INFO:tasks.workunit.client.0.smithi090.stdout: api_cls_remote_reads: 2024-04-14T15:12:57.176 INFO:tasks.workunit.client.0.smithi090.stdout: api_cls_remote_reads: [----------] Global test environment tear-down 2024-04-14T15:12:57.176 INFO:tasks.workunit.client.0.smithi090.stdout: api_cls_remote_reads: [==========] 1 test from 1 test suite ran. (2671 ms total) 2024-04-14T15:12:57.176 INFO:tasks.workunit.client.0.smithi090.stdout: api_cls_remote_reads: [ PASSED ] 1 test. 2024-04-14T15:12:57.176 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:12:57.176 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:12:57.177 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 1.75, 2024-04-14T15:12:57.177 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 1.75, 2024-04-14T15:12:57.177 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:12:57.177 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 1.75, 2024-04-14T15:12:57.177 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 1.75, 2024-04-14T15:12:57.177 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.90223121643066406, 2024-04-14T15:12:57.177 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 0.89999997615814209, 2024-04-14T15:12:57.177 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-04-14T15:12:57.177 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.177 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.177 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.177 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 22, 2024-04-14T15:12:57.177 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "LibRadosWatchNotify_smithi090-93930-1", 2024-04-14T15:12:57.177 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:12:54.373375+0000", 2024-04-14T15:12:57.178 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 32769, 2024-04-14T15:12:57.178 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,creating", 2024-04-14T15:12:57.178 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:12:57.178 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:12:57.178 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:12:57.178 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:12:57.178 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:12:57.178 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:12:57.178 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:12:57.178 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:12:57.178 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:12:57.178 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:12:57.178 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:12:57.178 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:12:57.179 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:12:57.179 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:12:57.179 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:12:57.179 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:12:57.179 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:12:57.179 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:12:57.179 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:12:57.179 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:12:57.179 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:12:57.179 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:12:57.179 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.179 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "108", 2024-04-14T15:12:57.179 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:12:57.179 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:12:57.180 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:12:57.180 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:12:57.180 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:12:57.180 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:12:57.180 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:12:57.180 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:12:57.180 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:12:57.180 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:12:57.180 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:12:57.180 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:12:57.180 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:12:57.180 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:12:57.180 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:12:57.180 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:12:57.180 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:12:57.181 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:12:57.181 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:12:57.181 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:12:57.181 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:12:57.181 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:12:57.181 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:12:57.181 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:12:57.181 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:12:57.181 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:12:57.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.181 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:12:57.181 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:12:57.181 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:12:57.181 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:12:57.182 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:12:57.182 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:12:57.182 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:12:57.182 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:12:57.182 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:12:57.182 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:12:57.182 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:12:57.182 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:12:57.182 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:12:57.182 INFO:teuthology.orchestra.run.smithi090.stdout: "rados": {} 2024-04-14T15:12:57.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.182 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:12:57.182 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:12:57.182 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 2, 2024-04-14T15:12:57.182 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 2, 2024-04-14T15:12:57.183 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:12:57.183 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 2, 2024-04-14T15:12:57.183 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 2, 2024-04-14T15:12:57.183 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.90223121643066406, 2024-04-14T15:12:57.183 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 0.89999997615814209, 2024-04-14T15:12:57.183 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-04-14T15:12:57.183 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.183 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 23, 2024-04-14T15:12:57.183 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "test-rados-api-smithi090-94066-1", 2024-04-14T15:12:57.183 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:12:54.442611+0000", 2024-04-14T15:12:57.183 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 32769, 2024-04-14T15:12:57.183 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,creating", 2024-04-14T15:12:57.184 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:12:57.184 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:12:57.184 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:12:57.184 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:12:57.184 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:12:57.184 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:12:57.184 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:12:57.184 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:12:57.184 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:12:57.184 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:12:57.184 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:12:57.184 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:12:57.184 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:12:57.184 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:12:57.184 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:12:57.185 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:12:57.185 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:12:57.185 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:12:57.185 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:12:57.185 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:12:57.185 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:12:57.185 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:12:57.185 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.185 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "108", 2024-04-14T15:12:57.185 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:12:57.185 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:12:57.185 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:12:57.185 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:12:57.185 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:12:57.185 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:12:57.186 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:12:57.186 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:12:57.186 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:12:57.186 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:12:57.186 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:12:57.186 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:12:57.186 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:12:57.186 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:12:57.186 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:12:57.186 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:12:57.186 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:12:57.186 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:12:57.186 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:12:57.186 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:12:57.186 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:12:57.187 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:12:57.187 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:12:57.187 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:12:57.187 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:12:57.187 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:12:57.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.187 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:12:57.187 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:12:57.187 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:12:57.187 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:12:57.187 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:12:57.187 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:12:57.187 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:12:57.187 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:12:57.188 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:12:57.188 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:12:57.188 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:12:57.188 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:12:57.188 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:12:57.188 INFO:teuthology.orchestra.run.smithi090.stdout: "rados": {} 2024-04-14T15:12:57.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.188 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:12:57.188 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:12:57.188 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 2.25, 2024-04-14T15:12:57.188 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 2.25, 2024-04-14T15:12:57.188 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:12:57.188 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 2.25, 2024-04-14T15:12:57.188 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 2.25, 2024-04-14T15:12:57.189 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.90223121643066406, 2024-04-14T15:12:57.189 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 0.89999997615814209, 2024-04-14T15:12:57.189 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-04-14T15:12:57.189 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.189 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 24, 2024-04-14T15:12:57.189 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "ceph_test_rados_list_parallel.smithi090-94111", 2024-04-14T15:12:57.189 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:12:54.454594+0000", 2024-04-14T15:12:57.189 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 32769, 2024-04-14T15:12:57.189 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,creating", 2024-04-14T15:12:57.189 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:12:57.189 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:12:57.189 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:12:57.190 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:12:57.190 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:12:57.190 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:12:57.190 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:12:57.190 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:12:57.190 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:12:57.190 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:12:57.190 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:12:57.190 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:12:57.190 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:12:57.190 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:12:57.190 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:12:57.190 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:12:57.190 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:12:57.191 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:12:57.191 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:12:57.191 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:12:57.191 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:12:57.191 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:12:57.191 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.191 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "107", 2024-04-14T15:12:57.191 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:12:57.191 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:12:57.191 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:12:57.191 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:12:57.191 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:12:57.191 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:12:57.191 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:12:57.191 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:12:57.192 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:12:57.192 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:12:57.192 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:12:57.192 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:12:57.192 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:12:57.192 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:12:57.192 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:12:57.192 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:12:57.192 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:12:57.192 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:12:57.192 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:12:57.192 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:12:57.192 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:12:57.192 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:12:57.192 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:12:57.193 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:12:57.193 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:12:57.193 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:12:57.193 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.193 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:12:57.193 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:12:57.193 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:12:57.193 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:12:57.193 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:12:57.193 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:12:57.193 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:12:57.193 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:12:57.193 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:12:57.193 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:12:57.193 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:12:57.194 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:12:57.194 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": {}, 2024-04-14T15:12:57.194 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:12:57.194 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:12:57.194 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 1.5, 2024-04-14T15:12:57.194 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 1.5, 2024-04-14T15:12:57.194 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:12:57.194 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 1.5, 2024-04-14T15:12:57.194 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 1.5, 2024-04-14T15:12:57.194 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.90223121643066406, 2024-04-14T15:12:57.194 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 0.89999997615814209, 2024-04-14T15:12:57.195 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-04-14T15:12:57.195 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.195 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 25, 2024-04-14T15:12:57.195 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "CReadOpsTest_smithi090-94131-1", 2024-04-14T15:12:57.195 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:12:54.477230+0000", 2024-04-14T15:12:57.195 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 32769, 2024-04-14T15:12:57.195 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,creating", 2024-04-14T15:12:57.195 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:12:57.195 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:12:57.195 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:12:57.195 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:12:57.195 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:12:57.195 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:12:57.196 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:12:57.196 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:12:57.196 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:12:57.196 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:12:57.196 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:12:57.196 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:12:57.196 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:12:57.196 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:12:57.196 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:12:57.196 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:12:57.196 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:12:57.196 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:12:57.196 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:12:57.196 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:12:57.196 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:12:57.197 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:12:57.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.197 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "108", 2024-04-14T15:12:57.197 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:12:57.197 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:12:57.197 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:12:57.197 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:12:57.197 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:12:57.197 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:12:57.197 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:12:57.197 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:12:57.197 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:12:57.197 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:12:57.197 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:12:57.197 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:12:57.198 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:12:57.198 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:12:57.198 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:12:57.198 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:12:57.198 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:12:57.198 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:12:57.198 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:12:57.198 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:12:57.198 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:12:57.198 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:12:57.198 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:12:57.198 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:12:57.198 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:12:57.198 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:12:57.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.199 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:12:57.199 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:12:57.199 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:12:57.199 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:12:57.199 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:12:57.199 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:12:57.199 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:12:57.199 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:12:57.199 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:12:57.199 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:12:57.199 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:12:57.199 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:12:57.199 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:12:57.200 INFO:teuthology.orchestra.run.smithi090.stdout: "rados": {} 2024-04-14T15:12:57.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.200 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:12:57.200 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:12:57.200 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 2.25, 2024-04-14T15:12:57.200 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 2.25, 2024-04-14T15:12:57.200 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:12:57.200 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 2.25, 2024-04-14T15:12:57.200 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 2.25, 2024-04-14T15:12:57.200 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.90223121643066406, 2024-04-14T15:12:57.200 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 0.89999997615814209, 2024-04-14T15:12:57.200 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-04-14T15:12:57.200 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.201 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 26, 2024-04-14T15:12:57.201 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "test-rados-api-smithi090-94114-1", 2024-04-14T15:12:57.201 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:12:54.488445+0000", 2024-04-14T15:12:57.201 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 32769, 2024-04-14T15:12:57.201 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,creating", 2024-04-14T15:12:57.201 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:12:57.201 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:12:57.201 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:12:57.201 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:12:57.201 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:12:57.201 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:12:57.201 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:12:57.201 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:12:57.201 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:12:57.201 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:12:57.202 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:12:57.202 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:12:57.202 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:12:57.202 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:12:57.202 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:12:57.202 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:12:57.202 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:12:57.202 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:12:57.202 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:12:57.202 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:12:57.202 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:12:57.202 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:12:57.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.202 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "108", 2024-04-14T15:12:57.202 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:12:57.203 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:12:57.203 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:12:57.203 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:12:57.203 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:12:57.203 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:12:57.203 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:12:57.203 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:12:57.203 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:12:57.203 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:12:57.203 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:12:57.203 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:12:57.203 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:12:57.203 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:12:57.203 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:12:57.203 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:12:57.204 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:12:57.204 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:12:57.204 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:12:57.204 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:12:57.204 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:12:57.204 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:12:57.204 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:12:57.204 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:12:57.204 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:12:57.204 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:12:57.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.204 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:12:57.204 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:12:57.204 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:12:57.204 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:12:57.205 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:12:57.205 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:12:57.205 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:12:57.205 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:12:57.205 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:12:57.205 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:12:57.205 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:12:57.205 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:12:57.205 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:12:57.205 INFO:teuthology.orchestra.run.smithi090.stdout: "rados": {} 2024-04-14T15:12:57.205 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.205 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:12:57.205 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:12:57.205 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 1.75, 2024-04-14T15:12:57.205 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 1.75, 2024-04-14T15:12:57.206 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:12:57.206 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 1.75, 2024-04-14T15:12:57.206 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 1.75, 2024-04-14T15:12:57.206 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.90223121643066406, 2024-04-14T15:12:57.206 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 0.89999997615814209, 2024-04-14T15:12:57.206 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-04-14T15:12:57.206 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.206 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.206 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.206 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 27, 2024-04-14T15:12:57.206 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "DNEsmithi090-94274-1", 2024-04-14T15:12:57.206 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:12:54.514999+0000", 2024-04-14T15:12:57.206 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 32769, 2024-04-14T15:12:57.206 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,creating", 2024-04-14T15:12:57.206 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:12:57.207 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:12:57.207 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:12:57.207 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:12:57.207 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:12:57.207 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:12:57.207 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:12:57.208 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:12:57.208 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:12:57.208 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:12:57.208 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:12:57.208 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:12:57.208 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:12:57.208 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:12:57.208 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:12:57.208 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:12:57.208 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:12:57.208 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:12:57.208 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:12:57.208 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:12:57.208 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:12:57.208 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:12:57.209 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.209 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "107", 2024-04-14T15:12:57.209 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:12:57.209 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:12:57.209 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:12:57.209 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:12:57.209 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:12:57.209 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:12:57.209 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:12:57.209 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:12:57.209 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:12:57.209 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:12:57.209 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:12:57.209 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:12:57.210 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:12:57.210 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:12:57.210 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:12:57.210 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:12:57.210 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:12:57.210 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:12:57.210 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:12:57.210 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:12:57.210 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:12:57.210 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:12:57.210 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:12:57.210 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:12:57.210 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:12:57.210 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:12:57.211 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.211 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:12:57.211 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:12:57.211 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:12:57.211 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:12:57.211 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:12:57.211 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:12:57.211 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:12:57.211 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:12:57.211 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:12:57.211 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:12:57.211 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:12:57.211 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:12:57.211 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": {}, 2024-04-14T15:12:57.212 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:12:57.212 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:12:57.212 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 1.75, 2024-04-14T15:12:57.212 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 1.75, 2024-04-14T15:12:57.212 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:12:57.212 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 1.75, 2024-04-14T15:12:57.212 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 1.75, 2024-04-14T15:12:57.212 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.90223121643066406, 2024-04-14T15:12:57.212 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 0.89999997615814209, 2024-04-14T15:12:57.212 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-04-14T15:12:57.212 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.212 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.212 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.212 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 28, 2024-04-14T15:12:57.212 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "ceph_test_rados_delete_pools_parallel.smithi090-94244", 2024-04-14T15:12:57.213 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:12:54.526462+0000", 2024-04-14T15:12:57.213 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 32769, 2024-04-14T15:12:57.213 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,creating", 2024-04-14T15:12:57.213 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:12:57.213 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:12:57.213 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:12:57.213 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:12:57.213 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:12:57.213 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:12:57.213 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:12:57.213 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:12:57.213 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:12:57.213 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:12:57.213 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:12:57.213 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:12:57.214 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:12:57.214 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:12:57.214 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:12:57.214 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:12:57.214 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:12:57.214 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:12:57.214 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:12:57.214 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:12:57.214 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:12:57.214 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:12:57.214 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.214 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "107", 2024-04-14T15:12:57.214 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:12:57.214 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:12:57.214 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:12:57.215 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:12:57.215 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:12:57.215 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:12:57.215 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:12:57.215 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:12:57.215 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:12:57.215 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:12:57.215 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:12:57.215 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:12:57.215 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:12:57.215 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:12:57.215 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:12:57.215 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:12:57.215 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:12:57.216 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:12:57.216 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:12:57.216 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:12:57.216 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:12:57.216 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:12:57.216 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:12:57.216 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:12:57.216 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:12:57.216 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:12:57.216 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.216 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:12:57.216 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:12:57.216 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:12:57.216 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:12:57.216 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:12:57.217 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:12:57.217 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:12:57.217 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:12:57.217 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:12:57.217 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:12:57.217 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:12:57.217 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:12:57.217 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": {}, 2024-04-14T15:12:57.217 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:12:57.217 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:12:57.217 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 1.5, 2024-04-14T15:12:57.217 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 1.5, 2024-04-14T15:12:57.217 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:12:57.217 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 1.5, 2024-04-14T15:12:57.217 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 1.5, 2024-04-14T15:12:57.217 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.90223121643066406, 2024-04-14T15:12:57.218 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 0.89999997615814209, 2024-04-14T15:12:57.218 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-04-14T15:12:57.218 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.218 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.218 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.218 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 29, 2024-04-14T15:12:57.218 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "test-rados-api-smithi090-93995-1", 2024-04-14T15:12:57.218 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:12:54.535126+0000", 2024-04-14T15:12:57.218 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 32769, 2024-04-14T15:12:57.218 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,creating", 2024-04-14T15:12:57.218 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:12:57.218 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:12:57.218 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:12:57.219 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:12:57.219 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:12:57.219 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:12:57.219 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:12:57.219 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:12:57.219 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:12:57.219 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:12:57.219 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:12:57.219 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:12:57.219 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:12:57.219 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:12:57.219 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:12:57.219 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:12:57.220 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:12:57.220 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:12:57.220 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:12:57.220 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:12:57.220 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:12:57.220 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:12:57.220 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.220 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "108", 2024-04-14T15:12:57.220 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:12:57.220 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:12:57.221 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:12:57.221 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:12:57.221 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:12:57.221 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:12:57.221 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:12:57.221 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:12:57.221 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:12:57.221 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:12:57.221 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:12:57.221 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:12:57.221 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:12:57.221 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:12:57.221 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:12:57.221 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:12:57.221 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:12:57.222 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:12:57.222 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:12:57.222 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:12:57.222 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:12:57.222 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:12:57.222 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:12:57.222 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:12:57.222 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:12:57.222 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:12:57.222 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.222 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:12:57.222 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:12:57.222 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:12:57.222 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:12:57.223 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:12:57.223 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:12:57.223 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:12:57.223 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:12:57.223 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:12:57.223 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:12:57.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:57.144+0000 7f281e41f700 10 librados: watch_flush enter 2024-04-14T15:12:57.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:57.144+0000 7f281e41f700 10 librados: watch_flush exit 2024-04-14T15:12:57.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:57.144+0000 7f281e41f700 20 client.25108.objecter shutdown clearing up homeless session... 2024-04-14T15:12:57.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:57.144+0000 7f281e41f700 10 client.25108.objecter successfully canceled tick 2024-04-14T15:12:57.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:57.145+0000 7f281e41f700 1 -- 172.21.15.90:0/2645816859 >> v1:172.21.15.90:6800/420935427 conn(0x7f28040716d0 legacy=0x7f2804073b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:57.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:57.145+0000 7f281e41f700 1 -- 172.21.15.90:0/2645816859 >> v1:172.21.15.92:6789/0 conn(0x7f28100add20 legacy=0x7f28100b5e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:12:57.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:57.145+0000 7f281e41f700 1 -- 172.21.15.90:0/2645816859 shutdown_connections 2024-04-14T15:12:57.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:57.145+0000 7f281e41f700 1 -- 172.21.15.90:0/2645816859 >> 172.21.15.90:0/2645816859 conn(0x7f281001a930 msgr2=0x7f28100acc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:12:57.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:57.147+0000 7f281e41f700 1 -- 172.21.15.90:0/2645816859 shutdown_connections 2024-04-14T15:12:57.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:57.147+0000 7f281e41f700 1 -- 172.21.15.90:0/2645816859 wait complete. 2024-04-14T15:12:57.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:12:57.147+0000 7f281e41f700 1 librados: shutdown 2024-04-14T15:12:57.224 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:12:57.225 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:12:57.225 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:12:57.225 INFO:teuthology.orchestra.run.smithi090.stdout: "rados": {} 2024-04-14T15:12:57.225 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.225 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:12:57.225 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:12:57.225 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 2.75, 2024-04-14T15:12:57.225 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 2.75, 2024-04-14T15:12:57.225 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:12:57.225 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 2.75, 2024-04-14T15:12:57.225 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 2.75, 2024-04-14T15:12:57.225 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.90223121643066406, 2024-04-14T15:12:57.225 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 0.89999997615814209, 2024-04-14T15:12:57.225 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-04-14T15:12:57.226 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.226 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.226 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.226 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 30, 2024-04-14T15:12:57.226 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "ceph_test_rados_open_pools_parallel.smithi090-94230", 2024-04-14T15:12:57.226 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:12:54.542481+0000", 2024-04-14T15:12:57.226 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 32769, 2024-04-14T15:12:57.226 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,creating", 2024-04-14T15:12:57.226 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:12:57.226 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:12:57.226 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:12:57.226 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:12:57.226 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:12:57.226 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:12:57.227 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:12:57.227 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:12:57.227 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:12:57.227 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:12:57.227 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:12:57.227 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:12:57.227 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:12:57.227 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:12:57.227 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:12:57.227 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:12:57.227 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:12:57.227 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:12:57.227 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:12:57.227 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:12:57.227 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:12:57.228 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:12:57.228 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.228 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "107", 2024-04-14T15:12:57.228 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:12:57.228 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:12:57.228 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:12:57.228 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:12:57.228 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:12:57.228 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:12:57.228 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:12:57.228 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:12:57.228 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:12:57.228 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:12:57.228 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:12:57.228 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:12:57.229 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:12:57.229 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:12:57.229 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:12:57.229 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:12:57.229 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:12:57.229 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:12:57.229 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:12:57.229 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:12:57.229 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:12:57.229 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:12:57.229 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:12:57.229 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:12:57.229 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:12:57.229 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:12:57.230 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.230 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:12:57.230 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:12:57.230 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:12:57.230 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:12:57.230 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:12:57.230 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:12:57.230 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:12:57.230 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:12:57.230 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:12:57.230 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:12:57.230 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:12:57.230 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:12:57.230 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": {}, 2024-04-14T15:12:57.230 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:12:57.231 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:12:57.231 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 2.25, 2024-04-14T15:12:57.231 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 2.25, 2024-04-14T15:12:57.231 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:12:57.231 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 2.25, 2024-04-14T15:12:57.231 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 2.25, 2024-04-14T15:12:57.231 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.90223121643066406, 2024-04-14T15:12:57.231 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 0.89999997615814209, 2024-04-14T15:12:57.231 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-04-14T15:12:57.231 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.231 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.231 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 31, 2024-04-14T15:12:57.231 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "test-rados-api-smithi090-94011-1", 2024-04-14T15:12:57.231 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:12:54.549723+0000", 2024-04-14T15:12:57.232 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 32769, 2024-04-14T15:12:57.232 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,creating", 2024-04-14T15:12:57.232 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:12:57.232 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:12:57.232 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:12:57.232 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:12:57.232 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:12:57.232 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:12:57.232 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:12:57.232 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:12:57.232 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:12:57.232 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:12:57.232 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:12:57.232 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:12:57.232 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:12:57.233 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:12:57.233 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:12:57.233 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:12:57.233 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:12:57.233 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:12:57.233 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:12:57.233 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:12:57.233 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:12:57.233 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:12:57.233 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.233 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "108", 2024-04-14T15:12:57.233 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:12:57.233 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:12:57.233 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:12:57.233 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:12:57.234 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:12:57.234 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:12:57.234 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:12:57.234 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:12:57.234 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:12:57.234 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:12:57.234 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:12:57.234 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:12:57.234 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:12:57.234 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:12:57.234 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:12:57.234 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:12:57.234 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:12:57.234 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:12:57.234 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:12:57.235 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:12:57.235 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:12:57.235 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:12:57.235 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:12:57.235 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:12:57.235 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:12:57.235 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:12:57.235 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.235 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:12:57.235 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:12:57.235 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:12:57.235 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:12:57.235 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:12:57.235 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:12:57.235 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:12:57.236 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:12:57.236 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:12:57.236 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:12:57.236 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:12:57.236 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:12:57.236 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:12:57.236 INFO:teuthology.orchestra.run.smithi090.stdout: "rados": {} 2024-04-14T15:12:57.236 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.236 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:12:57.236 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:12:57.236 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 1.75, 2024-04-14T15:12:57.236 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 1.75, 2024-04-14T15:12:57.236 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:12:57.236 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 1.75, 2024-04-14T15:12:57.236 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 1.75, 2024-04-14T15:12:57.237 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.90223121643066406, 2024-04-14T15:12:57.237 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 0.89999997615814209, 2024-04-14T15:12:57.237 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-04-14T15:12:57.237 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.237 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.237 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.237 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 32, 2024-04-14T15:12:57.237 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "Limitssmithi090-94451-1", 2024-04-14T15:12:57.237 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:12:54.558188+0000", 2024-04-14T15:12:57.237 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 32769, 2024-04-14T15:12:57.237 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,creating", 2024-04-14T15:12:57.237 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:12:57.237 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:12:57.237 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:12:57.237 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:12:57.238 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:12:57.238 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:12:57.238 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:12:57.238 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:12:57.238 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:12:57.238 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:12:57.238 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:12:57.238 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:12:57.238 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:12:57.238 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:12:57.238 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:12:57.238 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:12:57.238 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:12:57.238 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:12:57.238 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:12:57.239 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:12:57.239 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:12:57.239 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:12:57.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.239 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "107", 2024-04-14T15:12:57.239 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:12:57.239 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:12:57.239 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:12:57.239 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:12:57.239 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:12:57.239 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:12:57.239 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:12:57.239 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:12:57.239 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:12:57.240 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:12:57.240 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:12:57.240 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:12:57.240 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:12:57.240 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:12:57.240 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:12:57.240 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:12:57.240 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:12:57.240 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:12:57.240 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:12:57.240 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:12:57.240 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:12:57.240 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:12:57.240 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:12:57.240 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:12:57.241 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:12:57.241 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:12:57.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.241 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:12:57.241 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:12:57.241 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:12:57.241 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:12:57.241 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:12:57.241 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:12:57.241 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:12:57.241 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:12:57.241 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:12:57.241 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:12:57.241 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:12:57.241 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:12:57.242 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": {}, 2024-04-14T15:12:57.242 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:12:57.242 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:12:57.242 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 1.75, 2024-04-14T15:12:57.242 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 1.75, 2024-04-14T15:12:57.242 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:12:57.242 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 1.75, 2024-04-14T15:12:57.242 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 1.75, 2024-04-14T15:12:57.242 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.90223121643066406, 2024-04-14T15:12:57.242 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 0.89999997615814209, 2024-04-14T15:12:57.242 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-04-14T15:12:57.242 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.242 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.242 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.243 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 34, 2024-04-14T15:12:57.243 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "test_handler_errorsmithi090-94532-1", 2024-04-14T15:12:57.243 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:12:54.581004+0000", 2024-04-14T15:12:57.243 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 32769, 2024-04-14T15:12:57.243 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,creating", 2024-04-14T15:12:57.243 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:12:57.243 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:12:57.243 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:12:57.243 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:12:57.243 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:12:57.243 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:12:57.243 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:12:57.243 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:12:57.243 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:12:57.243 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:12:57.244 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:12:57.244 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:12:57.244 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:12:57.244 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:12:57.244 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:12:57.244 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:12:57.244 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:12:57.244 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:12:57.244 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:12:57.244 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:12:57.244 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:12:57.244 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:12:57.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.244 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "107", 2024-04-14T15:12:57.245 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:12:57.245 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:12:57.245 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:12:57.245 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:12:57.245 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:12:57.245 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:12:57.245 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:12:57.245 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:12:57.245 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:12:57.245 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:12:57.245 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:12:57.245 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:12:57.245 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:12:57.245 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:12:57.245 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:12:57.246 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:12:57.246 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:12:57.246 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:12:57.246 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:12:57.246 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:12:57.246 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:12:57.246 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:12:57.246 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:12:57.246 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:12:57.246 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:12:57.246 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:12:57.246 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.246 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:12:57.246 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:12:57.247 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:12:57.247 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:12:57.247 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:12:57.247 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:12:57.247 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:12:57.247 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:12:57.247 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:12:57.247 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:12:57.247 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:12:57.247 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:12:57.247 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": {}, 2024-04-14T15:12:57.247 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:12:57.247 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:12:57.247 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 1.75, 2024-04-14T15:12:57.247 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 1.75, 2024-04-14T15:12:57.248 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:12:57.248 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 1.75, 2024-04-14T15:12:57.248 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 1.75, 2024-04-14T15:12:57.248 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.90223121643066406, 2024-04-14T15:12:57.248 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 0.89999997615814209, 2024-04-14T15:12:57.248 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-04-14T15:12:57.248 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.248 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.248 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.248 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 35, 2024-04-14T15:12:57.248 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "ListObjectssmithi090-94600-1", 2024-04-14T15:12:57.248 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:12:54.588799+0000", 2024-04-14T15:12:57.248 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 32769, 2024-04-14T15:12:57.249 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,creating", 2024-04-14T15:12:57.249 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:12:57.249 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:12:57.249 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:12:57.249 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:12:57.249 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:12:57.249 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:12:57.249 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:12:57.249 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:12:57.249 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:12:57.249 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:12:57.249 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:12:57.249 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:12:57.249 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:12:57.249 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:12:57.250 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:12:57.250 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:12:57.250 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:12:57.250 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:12:57.250 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:12:57.250 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:12:57.250 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:12:57.250 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:12:57.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.250 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "107", 2024-04-14T15:12:57.250 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:12:57.250 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:12:57.250 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:12:57.250 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:12:57.251 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:12:57.251 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:12:57.251 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:12:57.251 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:12:57.251 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:12:57.251 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:12:57.251 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:12:57.251 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:12:57.251 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:12:57.251 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:12:57.251 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:12:57.251 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:12:57.251 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:12:57.251 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:12:57.251 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:12:57.251 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:12:57.252 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:12:57.252 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:12:57.252 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:12:57.252 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:12:57.252 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:12:57.252 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:12:57.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.252 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:12:57.252 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:12:57.252 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:12:57.252 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:12:57.252 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:12:57.252 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:12:57.252 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:12:57.252 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:12:57.253 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:12:57.253 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:12:57.253 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:12:57.253 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:12:57.253 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": {}, 2024-04-14T15:12:57.253 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:12:57.253 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:12:57.253 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 2.5, 2024-04-14T15:12:57.253 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 2.5, 2024-04-14T15:12:57.253 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:12:57.253 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 2.5, 2024-04-14T15:12:57.253 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 2.5, 2024-04-14T15:12:57.253 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.90223121643066406, 2024-04-14T15:12:57.253 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 0.89999997615814209, 2024-04-14T15:12:57.253 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-04-14T15:12:57.254 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.254 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.254 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 38, 2024-04-14T15:12:57.254 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "SetOpFlagssmithi090-94703-1", 2024-04-14T15:12:57.254 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:12:54.615131+0000", 2024-04-14T15:12:57.254 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 32769, 2024-04-14T15:12:57.254 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,creating", 2024-04-14T15:12:57.254 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:12:57.254 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:12:57.254 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:12:57.254 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:12:57.254 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:12:57.254 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:12:57.255 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:12:57.255 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:12:57.255 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:12:57.255 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:12:57.255 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:12:57.255 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:12:57.255 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:12:57.255 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:12:57.255 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:12:57.255 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:12:57.255 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:12:57.255 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:12:57.255 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:12:57.255 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:12:57.255 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:12:57.256 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:12:57.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.256 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "107", 2024-04-14T15:12:57.256 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:12:57.256 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:12:57.256 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:12:57.256 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:12:57.256 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:12:57.256 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:12:57.256 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:12:57.256 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:12:57.256 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:12:57.256 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:12:57.256 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:12:57.257 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:12:57.257 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:12:57.257 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:12:57.257 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:12:57.257 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:12:57.257 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:12:57.257 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:12:57.257 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:12:57.257 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:12:57.257 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:12:57.257 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:12:57.257 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:12:57.257 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:12:57.257 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:12:57.257 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:12:57.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.258 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:12:57.258 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:12:57.258 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:12:57.258 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:12:57.258 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:12:57.258 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:12:57.258 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:12:57.258 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:12:57.258 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:12:57.258 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:12:57.258 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:12:57.258 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:12:57.258 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": {}, 2024-04-14T15:12:57.258 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:12:57.259 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:12:57.259 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 2.25, 2024-04-14T15:12:57.259 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 2.25, 2024-04-14T15:12:57.259 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:12:57.259 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 2.25, 2024-04-14T15:12:57.259 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 2.25, 2024-04-14T15:12:57.259 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.90223121643066406, 2024-04-14T15:12:57.259 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 0.89999997615814209, 2024-04-14T15:12:57.259 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-04-14T15:12:57.259 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.259 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 39, 2024-04-14T15:12:57.259 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "SnapListsmithi090-94770-1", 2024-04-14T15:12:57.260 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:12:54.631880+0000", 2024-04-14T15:12:57.260 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 32769, 2024-04-14T15:12:57.260 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,creating", 2024-04-14T15:12:57.260 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:12:57.260 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:12:57.260 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:12:57.260 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:12:57.260 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:12:57.260 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:12:57.260 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:12:57.260 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:12:57.260 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:12:57.260 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:12:57.261 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:12:57.261 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:12:57.261 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:12:57.261 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:12:57.261 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:12:57.261 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:12:57.261 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:12:57.262 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:12:57.262 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:12:57.262 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:12:57.262 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:12:57.262 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:12:57.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.262 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "107", 2024-04-14T15:12:57.262 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:12:57.263 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:12:57.263 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:12:57.263 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:12:57.263 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:12:57.263 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:12:57.263 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:12:57.263 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:12:57.263 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:12:57.263 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:12:57.263 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:12:57.264 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:12:57.264 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:12:57.264 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:12:57.264 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:12:57.264 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:12:57.264 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:12:57.264 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:12:57.264 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:12:57.264 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:12:57.264 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:12:57.264 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:12:57.264 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:12:57.264 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:12:57.264 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:12:57.264 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:12:57.265 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.265 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:12:57.265 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:12:57.265 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:12:57.265 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:12:57.265 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:12:57.265 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:12:57.265 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:12:57.265 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:12:57.265 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:12:57.265 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:12:57.265 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:12:57.265 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:12:57.265 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": {}, 2024-04-14T15:12:57.265 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:12:57.266 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:12:57.266 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 1.5, 2024-04-14T15:12:57.266 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 1.5, 2024-04-14T15:12:57.266 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:12:57.266 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 1.5, 2024-04-14T15:12:57.266 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 1.5, 2024-04-14T15:12:57.266 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.90223121643066406, 2024-04-14T15:12:57.266 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 0.89999997615814209, 2024-04-14T15:12:57.266 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-04-14T15:12:57.266 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.266 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.266 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.266 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 40, 2024-04-14T15:12:57.266 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "WatchNotifysmithi090-94850-1", 2024-04-14T15:12:57.267 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:12:54.635366+0000", 2024-04-14T15:12:57.267 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 32769, 2024-04-14T15:12:57.267 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,creating", 2024-04-14T15:12:57.267 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:12:57.269 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:12:57.269 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:12:57.269 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:12:57.269 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:12:57.269 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:12:57.270 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:12:57.270 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:12:57.270 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:12:57.270 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:12:57.270 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:12:57.270 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:12:57.270 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:12:57.270 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:12:57.270 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:12:57.270 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:12:57.270 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:12:57.270 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:12:57.270 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:12:57.270 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:12:57.271 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:12:57.271 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:12:57.271 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.271 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "107", 2024-04-14T15:12:57.271 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:12:57.271 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:12:57.271 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:12:57.271 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:12:57.271 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:12:57.271 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:12:57.271 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:12:57.271 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:12:57.271 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:12:57.271 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:12:57.272 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:12:57.272 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:12:57.272 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:12:57.272 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:12:57.272 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:12:57.272 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:12:57.272 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:12:57.272 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:12:57.272 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:12:57.272 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:12:57.272 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:12:57.272 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:12:57.272 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:12:57.272 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:12:57.273 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:12:57.273 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:12:57.273 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.273 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:12:57.273 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:12:57.273 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:12:57.273 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:12:57.273 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:12:57.273 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:12:57.273 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:12:57.273 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:12:57.273 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:12:57.273 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:12:57.274 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:12:57.274 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:12:57.274 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": {}, 2024-04-14T15:12:57.274 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:12:57.274 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:12:57.274 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 2, 2024-04-14T15:12:57.275 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 2, 2024-04-14T15:12:57.275 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:12:57.275 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 2, 2024-04-14T15:12:57.275 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 2, 2024-04-14T15:12:57.275 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.90223121643066406, 2024-04-14T15:12:57.275 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 0.89999997615814209, 2024-04-14T15:12:57.275 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-04-14T15:12:57.275 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.275 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.275 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 41, 2024-04-14T15:12:57.275 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "AssertExistssmithi090-94846-1", 2024-04-14T15:12:57.275 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:12:54.638413+0000", 2024-04-14T15:12:57.275 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 32769, 2024-04-14T15:12:57.276 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,creating", 2024-04-14T15:12:57.276 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:12:57.276 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:12:57.276 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:12:57.276 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:12:57.276 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:12:57.276 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:12:57.276 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:12:57.276 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:12:57.276 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:12:57.276 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:12:57.276 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:12:57.276 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:12:57.276 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:12:57.276 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:12:57.277 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:12:57.277 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:12:57.277 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:12:57.277 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:12:57.277 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:12:57.277 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:12:57.277 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:12:57.277 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:12:57.277 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.277 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "107", 2024-04-14T15:12:57.277 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:12:57.277 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:12:57.277 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:12:57.277 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:12:57.277 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:12:57.278 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:12:57.278 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:12:57.278 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:12:57.278 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:12:57.278 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:12:57.278 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:12:57.278 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:12:57.278 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:12:57.278 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:12:57.278 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:12:57.278 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:12:57.278 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:12:57.278 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:12:57.278 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:12:57.279 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:12:57.279 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:12:57.279 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:12:57.279 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:12:57.279 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:12:57.279 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:12:57.279 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:12:57.279 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.279 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:12:57.279 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:12:57.279 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:12:57.279 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:12:57.279 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:12:57.279 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:12:57.279 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:12:57.280 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:12:57.280 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:12:57.280 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:12:57.280 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:12:57.280 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:12:57.280 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": {}, 2024-04-14T15:12:57.280 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:12:57.280 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:12:57.280 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 2, 2024-04-14T15:12:57.280 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 2, 2024-04-14T15:12:57.280 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:12:57.280 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 2, 2024-04-14T15:12:57.280 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 2, 2024-04-14T15:12:57.280 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.90223121643066406, 2024-04-14T15:12:57.280 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 0.89999997615814209, 2024-04-14T15:12:57.281 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-04-14T15:12:57.281 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.281 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.281 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-14T15:12:57.281 INFO:teuthology.orchestra.run.smithi090.stdout: "osds": [ 2024-04-14T15:12:57.281 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.281 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 0, 2024-04-14T15:12:57.281 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "76772cfc-89d1-4013-b617-32284ed8dfd9", 2024-04-14T15:12:57.281 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-14T15:12:57.281 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-14T15:12:57.281 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-14T15:12:57.281 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-14T15:12:57.281 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-14T15:12:57.281 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-14T15:12:57.282 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 8, 2024-04-14T15:12:57.282 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 107, 2024-04-14T15:12:57.282 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-14T15:12:57.282 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-14T15:12:57.282 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-14T15:12:57.282 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:12:57.282 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.282 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:12:57.282 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6801", 2024-04-14T15:12:57.282 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3761806885 2024-04-14T15:12:57.282 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.282 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:12:57.282 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.282 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-14T15:12:57.282 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:12:57.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.283 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:12:57.283 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6802", 2024-04-14T15:12:57.283 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3761806885 2024-04-14T15:12:57.283 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.283 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:12:57.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.283 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-14T15:12:57.283 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:12:57.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.283 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:12:57.283 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6804", 2024-04-14T15:12:57.283 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3761806885 2024-04-14T15:12:57.283 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.284 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:12:57.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.284 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-14T15:12:57.284 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:12:57.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.284 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:12:57.284 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6803", 2024-04-14T15:12:57.284 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3761806885 2024-04-14T15:12:57.284 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.284 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:12:57.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.284 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6801/3761806885", 2024-04-14T15:12:57.284 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6802/3761806885", 2024-04-14T15:12:57.284 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6804/3761806885", 2024-04-14T15:12:57.284 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6803/3761806885", 2024-04-14T15:12:57.285 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-14T15:12:57.285 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-14T15:12:57.285 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-14T15:12:57.285 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:12:57.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.285 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 1, 2024-04-14T15:12:57.285 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "de4a7fa2-7d57-4ff3-a4fa-28c0cde8ef6c", 2024-04-14T15:12:57.285 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-14T15:12:57.285 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-14T15:12:57.285 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-14T15:12:57.285 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-14T15:12:57.285 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-14T15:12:57.285 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-14T15:12:57.285 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2024-04-14T15:12:57.286 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 107, 2024-04-14T15:12:57.286 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-14T15:12:57.286 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-14T15:12:57.286 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-14T15:12:57.286 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:12:57.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.286 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:12:57.286 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6805", 2024-04-14T15:12:57.286 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 4170361699 2024-04-14T15:12:57.286 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.286 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:12:57.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.286 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-14T15:12:57.286 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:12:57.287 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.287 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:12:57.287 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6806", 2024-04-14T15:12:57.287 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 4170361699 2024-04-14T15:12:57.287 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.287 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:12:57.287 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.287 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-14T15:12:57.287 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:12:57.287 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.287 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:12:57.287 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6808", 2024-04-14T15:12:57.287 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 4170361699 2024-04-14T15:12:57.287 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.287 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:12:57.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.288 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-14T15:12:57.288 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:12:57.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.288 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:12:57.288 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6807", 2024-04-14T15:12:57.288 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 4170361699 2024-04-14T15:12:57.288 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.288 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:12:57.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.288 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6805/4170361699", 2024-04-14T15:12:57.288 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6806/4170361699", 2024-04-14T15:12:57.288 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6808/4170361699", 2024-04-14T15:12:57.288 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6807/4170361699", 2024-04-14T15:12:57.288 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-14T15:12:57.289 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-14T15:12:57.289 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-14T15:12:57.289 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:12:57.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.289 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 2, 2024-04-14T15:12:57.289 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "9c39b5ff-6175-46ba-9d71-370581c6ff3d", 2024-04-14T15:12:57.289 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-14T15:12:57.289 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-14T15:12:57.289 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-14T15:12:57.289 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-14T15:12:57.289 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-14T15:12:57.289 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-14T15:12:57.289 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 18, 2024-04-14T15:12:57.289 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 107, 2024-04-14T15:12:57.290 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-14T15:12:57.290 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-14T15:12:57.290 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-14T15:12:57.290 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:12:57.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.290 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:12:57.290 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6809", 2024-04-14T15:12:57.290 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 1686184251 2024-04-14T15:12:57.290 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.290 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:12:57.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.290 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-14T15:12:57.290 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:12:57.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.290 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:12:57.291 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6810", 2024-04-14T15:12:57.291 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 1686184251 2024-04-14T15:12:57.291 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.291 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:12:57.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:57.292 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:57.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:12:57.293 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.293 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-14T15:12:57.293 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:12:57.293 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.293 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:12:57.293 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6812", 2024-04-14T15:12:57.293 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 1686184251 2024-04-14T15:12:57.293 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.293 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:12:57.293 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.293 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-14T15:12:57.293 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:12:57.293 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.294 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:12:57.294 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6811", 2024-04-14T15:12:57.294 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 1686184251 2024-04-14T15:12:57.294 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.294 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:12:57.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.294 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6809/1686184251", 2024-04-14T15:12:57.294 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6810/1686184251", 2024-04-14T15:12:57.294 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6812/1686184251", 2024-04-14T15:12:57.294 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6811/1686184251", 2024-04-14T15:12:57.294 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-14T15:12:57.294 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-14T15:12:57.294 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-14T15:12:57.294 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:12:57.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.295 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.295 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 3, 2024-04-14T15:12:57.295 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "efd2ca45-5b80-4013-a571-499e2cb5b2e8", 2024-04-14T15:12:57.295 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-14T15:12:57.295 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-14T15:12:57.295 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-14T15:12:57.295 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 0.2178497314453125, 2024-04-14T15:12:57.295 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-14T15:12:57.295 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-14T15:12:57.295 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 25, 2024-04-14T15:12:57.295 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 107, 2024-04-14T15:12:57.295 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-14T15:12:57.295 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-14T15:12:57.296 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-14T15:12:57.296 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:12:57.296 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.296 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:12:57.296 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6813", 2024-04-14T15:12:57.296 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 2270166774 2024-04-14T15:12:57.296 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.296 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:12:57.296 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.296 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-14T15:12:57.296 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:12:57.296 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.296 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:12:57.296 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6814", 2024-04-14T15:12:57.297 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 2270166774 2024-04-14T15:12:57.297 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.297 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:12:57.297 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.297 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-14T15:12:57.297 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:12:57.297 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.297 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:12:57.297 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6816", 2024-04-14T15:12:57.297 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 2270166774 2024-04-14T15:12:57.297 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.297 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:12:57.297 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.298 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-14T15:12:57.298 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:12:57.298 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.298 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:12:57.298 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6815", 2024-04-14T15:12:57.298 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 2270166774 2024-04-14T15:12:57.298 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.298 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:12:57.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.298 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6813/2270166774", 2024-04-14T15:12:57.298 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6814/2270166774", 2024-04-14T15:12:57.298 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6816/2270166774", 2024-04-14T15:12:57.298 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6815/2270166774", 2024-04-14T15:12:57.298 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-14T15:12:57.299 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-14T15:12:57.299 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-14T15:12:57.299 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:12:57.299 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.299 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.299 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 4, 2024-04-14T15:12:57.299 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "e3d9013d-5eb6-433c-96b3-3fc38ba7ac8d", 2024-04-14T15:12:57.299 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-14T15:12:57.299 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-14T15:12:57.299 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-14T15:12:57.299 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-14T15:12:57.299 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-14T15:12:57.299 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-14T15:12:57.299 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 31, 2024-04-14T15:12:57.300 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 107, 2024-04-14T15:12:57.300 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-14T15:12:57.300 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-14T15:12:57.300 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-14T15:12:57.300 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:12:57.300 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.300 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:12:57.300 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6800", 2024-04-14T15:12:57.300 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3887549733 2024-04-14T15:12:57.300 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.300 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:12:57.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.300 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-14T15:12:57.300 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:12:57.300 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.301 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:12:57.301 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6801", 2024-04-14T15:12:57.301 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3887549733 2024-04-14T15:12:57.301 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.301 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:12:57.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.301 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-14T15:12:57.301 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:12:57.301 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.301 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:12:57.301 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6803", 2024-04-14T15:12:57.301 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3887549733 2024-04-14T15:12:57.301 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.301 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:12:57.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.302 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-14T15:12:57.302 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:12:57.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.302 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:12:57.302 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6802", 2024-04-14T15:12:57.302 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3887549733 2024-04-14T15:12:57.302 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.302 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:12:57.302 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.302 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.92:6800/3887549733", 2024-04-14T15:12:57.302 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.92:6801/3887549733", 2024-04-14T15:12:57.302 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.92:6803/3887549733", 2024-04-14T15:12:57.302 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.92:6802/3887549733", 2024-04-14T15:12:57.302 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-14T15:12:57.303 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-14T15:12:57.303 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-14T15:12:57.303 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:12:57.303 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.303 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.303 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 5, 2024-04-14T15:12:57.303 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "1ec41805-6f82-4139-81bc-75305ba9e958", 2024-04-14T15:12:57.303 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-14T15:12:57.303 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-14T15:12:57.303 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-14T15:12:57.303 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-14T15:12:57.303 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-14T15:12:57.303 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-14T15:12:57.303 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 41, 2024-04-14T15:12:57.303 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 107, 2024-04-14T15:12:57.304 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-14T15:12:57.304 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-14T15:12:57.304 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-14T15:12:57.304 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:12:57.304 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.304 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:12:57.304 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6804", 2024-04-14T15:12:57.304 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3244561440 2024-04-14T15:12:57.304 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.304 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:12:57.304 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.304 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-14T15:12:57.304 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:12:57.304 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.305 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:12:57.305 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6805", 2024-04-14T15:12:57.305 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3244561440 2024-04-14T15:12:57.305 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.305 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:12:57.305 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.305 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-14T15:12:57.305 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:12:57.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.305 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:12:57.305 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6807", 2024-04-14T15:12:57.305 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3244561440 2024-04-14T15:12:57.305 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.305 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:12:57.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.306 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-14T15:12:57.306 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:12:57.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.306 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:12:57.306 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6806", 2024-04-14T15:12:57.306 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3244561440 2024-04-14T15:12:57.306 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.306 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:12:57.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.306 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.92:6804/3244561440", 2024-04-14T15:12:57.306 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.92:6805/3244561440", 2024-04-14T15:12:57.306 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.92:6807/3244561440", 2024-04-14T15:12:57.306 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.92:6806/3244561440", 2024-04-14T15:12:57.306 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-14T15:12:57.307 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-14T15:12:57.307 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-14T15:12:57.307 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:12:57.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.307 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 6, 2024-04-14T15:12:57.307 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "ca977ab2-bf8d-4b3a-ae5e-0efa5bc7a3ad", 2024-04-14T15:12:57.307 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-14T15:12:57.307 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-14T15:12:57.307 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-14T15:12:57.307 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-14T15:12:57.307 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-14T15:12:57.307 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-14T15:12:57.307 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 49, 2024-04-14T15:12:57.307 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 107, 2024-04-14T15:12:57.308 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-14T15:12:57.308 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-14T15:12:57.308 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-14T15:12:57.308 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:12:57.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.308 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:12:57.308 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6808", 2024-04-14T15:12:57.308 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 636125963 2024-04-14T15:12:57.308 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.308 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:12:57.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.308 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-14T15:12:57.308 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:12:57.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.308 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:12:57.309 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6809", 2024-04-14T15:12:57.309 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 636125963 2024-04-14T15:12:57.309 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.309 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:12:57.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.309 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-14T15:12:57.309 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:12:57.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.309 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:12:57.309 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6811", 2024-04-14T15:12:57.309 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 636125963 2024-04-14T15:12:57.309 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.309 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:12:57.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.309 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-14T15:12:57.310 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:12:57.310 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.310 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:12:57.310 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6810", 2024-04-14T15:12:57.310 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 636125963 2024-04-14T15:12:57.310 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.310 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:12:57.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.310 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.92:6808/636125963", 2024-04-14T15:12:57.310 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.92:6809/636125963", 2024-04-14T15:12:57.310 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.92:6811/636125963", 2024-04-14T15:12:57.310 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.92:6810/636125963", 2024-04-14T15:12:57.310 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-14T15:12:57.310 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-14T15:12:57.310 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-14T15:12:57.311 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:12:57.311 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.311 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 7, 2024-04-14T15:12:57.311 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "e5354645-79ec-4d4c-b6e5-f9b0653915ee", 2024-04-14T15:12:57.311 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-14T15:12:57.311 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-14T15:12:57.311 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-14T15:12:57.311 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-14T15:12:57.311 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-14T15:12:57.311 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-14T15:12:57.311 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 58, 2024-04-14T15:12:57.311 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 107, 2024-04-14T15:12:57.311 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-14T15:12:57.312 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-14T15:12:57.312 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-14T15:12:57.312 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:12:57.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.312 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:12:57.312 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6812", 2024-04-14T15:12:57.312 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 1773706716 2024-04-14T15:12:57.312 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.312 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:12:57.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.312 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-14T15:12:57.312 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:12:57.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.312 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:12:57.312 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6813", 2024-04-14T15:12:57.313 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 1773706716 2024-04-14T15:12:57.313 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.313 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:12:57.313 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.313 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-14T15:12:57.313 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:12:57.313 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.313 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:12:57.313 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6815", 2024-04-14T15:12:57.313 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 1773706716 2024-04-14T15:12:57.313 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.313 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:12:57.313 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.313 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-14T15:12:57.313 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:12:57.313 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.314 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:12:57.314 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6814", 2024-04-14T15:12:57.314 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 1773706716 2024-04-14T15:12:57.314 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.314 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:12:57.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.314 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.92:6812/1773706716", 2024-04-14T15:12:57.314 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.92:6813/1773706716", 2024-04-14T15:12:57.314 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.92:6815/1773706716", 2024-04-14T15:12:57.314 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.92:6814/1773706716", 2024-04-14T15:12:57.314 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-14T15:12:57.314 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-14T15:12:57.314 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-14T15:12:57.314 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:12:57.315 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.315 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-14T15:12:57.315 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_xinfo": [ 2024-04-14T15:12:57.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.315 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 0, 2024-04-14T15:12:57.315 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-14T15:12:57.315 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-14T15:12:57.315 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-14T15:12:57.315 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540701547738038271, 2024-04-14T15:12:57.315 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-14T15:12:57.315 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-14T15:11:46.086720+0000", 2024-04-14T15:12:57.315 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-14T15:12:57.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.316 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 1, 2024-04-14T15:12:57.316 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-14T15:12:57.316 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-14T15:12:57.316 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-14T15:12:57.316 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540701547738038271, 2024-04-14T15:12:57.316 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-14T15:12:57.316 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-14T15:12:44.215601+0000", 2024-04-14T15:12:57.316 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-14T15:12:57.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.316 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 2, 2024-04-14T15:12:57.316 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-14T15:12:57.316 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-14T15:12:57.316 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-14T15:12:57.316 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540701547738038271, 2024-04-14T15:12:57.317 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-14T15:12:57.317 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-14T15:12:10.327706+0000", 2024-04-14T15:12:57.317 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-14T15:12:57.317 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.317 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 3, 2024-04-14T15:12:57.317 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-14T15:12:57.317 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-14T15:12:57.317 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-14T15:12:57.317 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540701547738038271, 2024-04-14T15:12:57.317 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-14T15:12:57.317 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-14T15:11:53.872813+0000", 2024-04-14T15:12:57.317 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-14T15:12:57.317 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.318 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 4, 2024-04-14T15:12:57.318 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-14T15:12:57.318 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-14T15:12:57.318 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-14T15:12:57.318 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540701547738038271, 2024-04-14T15:12:57.318 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-14T15:12:57.318 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-14T15:11:41.834590+0000", 2024-04-14T15:12:57.318 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-14T15:12:57.318 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.318 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.318 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 5, 2024-04-14T15:12:57.318 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-14T15:12:57.318 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-14T15:12:57.318 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-14T15:12:57.318 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540701547738038271, 2024-04-14T15:12:57.319 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-14T15:12:57.319 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-14T15:12:27.778595+0000", 2024-04-14T15:12:57.319 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-14T15:12:57.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.319 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.319 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 6, 2024-04-14T15:12:57.319 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-14T15:12:57.319 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-14T15:12:57.319 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-14T15:12:57.319 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540701547738038271, 2024-04-14T15:12:57.319 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-14T15:12:57.319 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-14T15:12:41.863374+0000", 2024-04-14T15:12:57.319 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-14T15:12:57.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.320 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 7, 2024-04-14T15:12:57.320 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-14T15:12:57.320 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-14T15:12:57.320 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-14T15:12:57.320 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540701547738038271, 2024-04-14T15:12:57.320 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-14T15:12:57.320 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-14T15:11:41.238050+0000", 2024-04-14T15:12:57.320 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-14T15:12:57.320 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.320 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-14T15:12:57.321 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_upmap": [], 2024-04-14T15:12:57.321 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_upmap_items": [ 2024-04-14T15:12:57.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.321 INFO:teuthology.orchestra.run.smithi090.stdout: "pgid": "2.1c", 2024-04-14T15:12:57.321 INFO:teuthology.orchestra.run.smithi090.stdout: "mappings": [ 2024-04-14T15:12:57.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.321 INFO:teuthology.orchestra.run.smithi090.stdout: "from": 7, 2024-04-14T15:12:57.321 INFO:teuthology.orchestra.run.smithi090.stdout: "to": 4 2024-04-14T15:12:57.321 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.321 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:12:57.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.321 INFO:teuthology.orchestra.run.smithi090.stdout: "pgid": "4.2", 2024-04-14T15:12:57.321 INFO:teuthology.orchestra.run.smithi090.stdout: "mappings": [ 2024-04-14T15:12:57.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.322 INFO:teuthology.orchestra.run.smithi090.stdout: "from": 1, 2024-04-14T15:12:57.322 INFO:teuthology.orchestra.run.smithi090.stdout: "to": 2 2024-04-14T15:12:57.322 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.322 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:12:57.322 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.322 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.322 INFO:teuthology.orchestra.run.smithi090.stdout: "pgid": "4.d", 2024-04-14T15:12:57.322 INFO:teuthology.orchestra.run.smithi090.stdout: "mappings": [ 2024-04-14T15:12:57.322 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.322 INFO:teuthology.orchestra.run.smithi090.stdout: "from": 1, 2024-04-14T15:12:57.322 INFO:teuthology.orchestra.run.smithi090.stdout: "to": 5 2024-04-14T15:12:57.322 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.322 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:12:57.322 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.322 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.323 INFO:teuthology.orchestra.run.smithi090.stdout: "pgid": "4.1f", 2024-04-14T15:12:57.323 INFO:teuthology.orchestra.run.smithi090.stdout: "mappings": [ 2024-04-14T15:12:57.323 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:12:57.323 INFO:teuthology.orchestra.run.smithi090.stdout: "from": 1, 2024-04-14T15:12:57.323 INFO:teuthology.orchestra.run.smithi090.stdout: "to": 2 2024-04-14T15:12:57.323 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.323 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:12:57.323 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.323 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-14T15:12:57.323 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_upmap_primaries": [], 2024-04-14T15:12:57.323 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_temp": [], 2024-04-14T15:12:57.323 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_temp": [], 2024-04-14T15:12:57.323 INFO:teuthology.orchestra.run.smithi090.stdout: "blocklist": { 2024-04-14T15:12:57.323 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:0/508768479": "2024-04-15T15:10:22.918674+0000", 2024-04-14T15:12:57.323 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:0/1823072952": "2024-04-15T15:10:22.918674+0000", 2024-04-14T15:12:57.324 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:6800/491071118": "2024-04-15T15:10:22.918674+0000", 2024-04-14T15:12:57.324 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:0/1337064035": "2024-04-15T15:10:22.918674+0000", 2024-04-14T15:12:57.324 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:0/769462306": "2024-04-15T15:10:22.918674+0000", 2024-04-14T15:12:57.324 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:0/1139868610": "2024-04-15T15:05:05.882361+0000", 2024-04-14T15:12:57.324 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:0/93763408": "2024-04-15T15:10:22.918674+0000", 2024-04-14T15:12:57.324 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:0/977818204": "2024-04-15T15:04:39.251711+0000", 2024-04-14T15:12:57.324 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:6800/941091171": "2024-04-15T15:05:05.882361+0000", 2024-04-14T15:12:57.324 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:0/3671453561": "2024-04-15T15:05:05.882361+0000", 2024-04-14T15:12:57.324 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:0/2611856200": "2024-04-15T15:05:05.882361+0000", 2024-04-14T15:12:57.324 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:6800/2306258962": "2024-04-15T15:04:39.251711+0000", 2024-04-14T15:12:57.324 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:0/4051692380": "2024-04-15T15:04:39.251711+0000", 2024-04-14T15:12:57.324 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:0/2681502561": "2024-04-15T15:04:39.251711+0000" 2024-04-14T15:12:57.324 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.324 INFO:teuthology.orchestra.run.smithi090.stdout: "range_blocklist": {}, 2024-04-14T15:12:57.325 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profiles": { 2024-04-14T15:12:57.325 INFO:teuthology.orchestra.run.smithi090.stdout: "default": { 2024-04-14T15:12:57.325 INFO:teuthology.orchestra.run.smithi090.stdout: "crush-failure-domain": "osd", 2024-04-14T15:12:57.325 INFO:teuthology.orchestra.run.smithi090.stdout: "k": "2", 2024-04-14T15:12:57.325 INFO:teuthology.orchestra.run.smithi090.stdout: "m": "1", 2024-04-14T15:12:57.325 INFO:teuthology.orchestra.run.smithi090.stdout: "plugin": "jerasure", 2024-04-14T15:12:57.325 INFO:teuthology.orchestra.run.smithi090.stdout: "technique": "reed_sol_van" 2024-04-14T15:12:57.325 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.325 INFO:teuthology.orchestra.run.smithi090.stdout: "testprofile-LibRadosWatchNotifyECPP_smithi090-93998-1": { 2024-04-14T15:12:57.325 INFO:teuthology.orchestra.run.smithi090.stdout: "crush-device-class": "", 2024-04-14T15:12:57.325 INFO:teuthology.orchestra.run.smithi090.stdout: "crush-failure-domain": "osd", 2024-04-14T15:12:57.325 INFO:teuthology.orchestra.run.smithi090.stdout: "crush-num-failure-domains": "0", 2024-04-14T15:12:57.325 INFO:teuthology.orchestra.run.smithi090.stdout: "crush-osds-per-failure-domain": "0", 2024-04-14T15:12:57.325 INFO:teuthology.orchestra.run.smithi090.stdout: "crush-root": "default", 2024-04-14T15:12:57.325 INFO:teuthology.orchestra.run.smithi090.stdout: "jerasure-per-chunk-alignment": "false", 2024-04-14T15:12:57.325 INFO:teuthology.orchestra.run.smithi090.stdout: "k": "2", 2024-04-14T15:12:57.326 INFO:teuthology.orchestra.run.smithi090.stdout: "m": "1", 2024-04-14T15:12:57.326 INFO:teuthology.orchestra.run.smithi090.stdout: "plugin": "jerasure", 2024-04-14T15:12:57.326 INFO:teuthology.orchestra.run.smithi090.stdout: "technique": "reed_sol_van", 2024-04-14T15:12:57.326 INFO:teuthology.orchestra.run.smithi090.stdout: "w": "8" 2024-04-14T15:12:57.326 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.326 INFO:teuthology.orchestra.run.smithi090.stdout: "testprofile-ListObjectssmithi090-94662-1": { 2024-04-14T15:12:57.326 INFO:teuthology.orchestra.run.smithi090.stdout: "crush-device-class": "", 2024-04-14T15:12:57.326 INFO:teuthology.orchestra.run.smithi090.stdout: "crush-failure-domain": "osd", 2024-04-14T15:12:57.326 INFO:teuthology.orchestra.run.smithi090.stdout: "crush-num-failure-domains": "0", 2024-04-14T15:12:57.326 INFO:teuthology.orchestra.run.smithi090.stdout: "crush-osds-per-failure-domain": "0", 2024-04-14T15:12:57.326 INFO:teuthology.orchestra.run.smithi090.stdout: "crush-root": "default", 2024-04-14T15:12:57.326 INFO:teuthology.orchestra.run.smithi090.stdout: "jerasure-per-chunk-alignment": "false", 2024-04-14T15:12:57.326 INFO:teuthology.orchestra.run.smithi090.stdout: "k": "2", 2024-04-14T15:12:57.326 INFO:teuthology.orchestra.run.smithi090.stdout: "m": "1", 2024-04-14T15:12:57.326 INFO:teuthology.orchestra.run.smithi090.stdout: "plugin": "jerasure", 2024-04-14T15:12:57.327 INFO:teuthology.orchestra.run.smithi090.stdout: "technique": "reed_sol_van", 2024-04-14T15:12:57.327 INFO:teuthology.orchestra.run.smithi090.stdout: "w": "8" 2024-04-14T15:12:57.327 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.327 INFO:teuthology.orchestra.run.smithi090.stdout: "testprofile-SimpleWritesmithi090-94522-1": { 2024-04-14T15:12:57.327 INFO:teuthology.orchestra.run.smithi090.stdout: "crush-device-class": "", 2024-04-14T15:12:57.327 INFO:teuthology.orchestra.run.smithi090.stdout: "crush-failure-domain": "osd", 2024-04-14T15:12:57.327 INFO:teuthology.orchestra.run.smithi090.stdout: "crush-num-failure-domains": "0", 2024-04-14T15:12:57.327 INFO:teuthology.orchestra.run.smithi090.stdout: "crush-osds-per-failure-domain": "0", 2024-04-14T15:12:57.327 INFO:teuthology.orchestra.run.smithi090.stdout: "crush-root": "default", 2024-04-14T15:12:57.327 INFO:teuthology.orchestra.run.smithi090.stdout: "jerasure-per-chunk-alignment": "false", 2024-04-14T15:12:57.327 INFO:teuthology.orchestra.run.smithi090.stdout: "k": "2", 2024-04-14T15:12:57.327 INFO:teuthology.orchestra.run.smithi090.stdout: "m": "1", 2024-04-14T15:12:57.327 INFO:teuthology.orchestra.run.smithi090.stdout: "plugin": "jerasure", 2024-04-14T15:12:57.327 INFO:teuthology.orchestra.run.smithi090.stdout: "technique": "reed_sol_van", 2024-04-14T15:12:57.327 INFO:teuthology.orchestra.run.smithi090.stdout: "w": "8" 2024-04-14T15:12:57.328 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:12:57.328 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps_queue": [], 2024-04-14T15:12:57.328 INFO:teuthology.orchestra.run.smithi090.stdout: "new_removed_snaps": [], 2024-04-14T15:12:57.328 INFO:teuthology.orchestra.run.smithi090.stdout: "new_purged_snaps": [], 2024-04-14T15:12:57.328 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_node_flags": {}, 2024-04-14T15:12:57.328 INFO:teuthology.orchestra.run.smithi090.stdout: "device_class_flags": {}, 2024-04-14T15:12:57.328 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_mode": { 2024-04-14T15:12:57.328 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_mode_enabled": false, 2024-04-14T15:12:57.328 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_bucket_count": 0, 2024-04-14T15:12:57.328 INFO:teuthology.orchestra.run.smithi090.stdout: "degraded_stretch_mode": 0, 2024-04-14T15:12:57.328 INFO:teuthology.orchestra.run.smithi090.stdout: "recovering_stretch_mode": 0, 2024-04-14T15:12:57.328 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_mode_bucket": 0 2024-04-14T15:12:57.328 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:12:57.329 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-14T15:12:57.329 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:57 smithi090 ceph-mon[41670]: from='client.24955 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94114-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.329 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:57 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2381966651' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.329 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:57 smithi090 ceph-mon[41670]: from='client.24802 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi090-93379-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.329 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:57 smithi090 ceph-mon[41670]: from='client.24860 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi090-93864-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.329 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:57 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2763816171' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi090-93731-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.329 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:57 smithi090 ceph-mon[41670]: from='client.24829 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi090-93444-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.329 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:57 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/495867689' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi090-93652-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.329 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:57 smithi090 ceph-mon[41670]: from='client.24991 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93995-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.330 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:57 smithi090 ceph-mon[41670]: from='client.24923 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi090-94131-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.330 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:57 smithi090 ceph-mon[41670]: from='client.24796 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi090-93342-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.330 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:57 smithi090 ceph-mon[41670]: from='client.24896 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.330 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:57 smithi090 ceph-mon[41670]: from='client.24808 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi090-93363-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.330 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:57 smithi090 ceph-mon[41670]: from='client.24811 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi090-93398-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.330 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:57 smithi090 ceph-mon[41670]: from='client.24823 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi090-93436-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.330 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:57 smithi090 ceph-mon[41670]: from='client.24853 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi090-93601-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.330 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:57 smithi090 ceph-mon[41670]: from='client.24868 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi090-93724-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.330 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:57 smithi090 ceph-mon[41670]: from='client.24871 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi090-93709-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.330 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:57 smithi090 ceph-mon[41670]: from='client.24889 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi090-93930-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.330 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:57 smithi090 ceph-mon[41670]: from='client.25000 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94011-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.330 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:57 smithi090 ceph-mon[41670]: from='client.24835 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi090-93476-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.330 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:57 smithi090 ceph-mon[41670]: osdmap e108: 8 total, 8 up, 8 in 2024-04-14T15:12:57.330 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:57 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2645816859' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-14T15:12:57.331 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:57 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/309942895' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi090-93998-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi090-93998-1"}]': finished 2024-04-14T15:12:57.331 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:57 smithi090 ceph-mon[41670]: from='client.25021 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi090-94522-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi090-94522-1"}]': finished 2024-04-14T15:12:57.331 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:57 smithi090 ceph-mon[41670]: from='client.25001 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi090-94662-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi090-94662-1"}]': finished 2024-04-14T15:12:57.331 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:57 smithi090 ceph-mon[41670]: osdmap e109: 8 total, 8 up, 8 in 2024-04-14T15:12:57.331 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:57 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1327720256' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:57.331 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:57 smithi090 ceph-mon[41670]: from='client.25114 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:57.331 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:57 smithi090 ceph-mon[30399]: from='client.24955 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94114-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.331 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:57 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2381966651' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.331 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:57 smithi090 ceph-mon[30399]: from='client.24802 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi090-93379-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.350 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd unset noscrub 2024-04-14T15:12:57.359 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: Running main() from gmock_main.cc 2024-04-14T15:12:57.359 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2024-04-14T15:12:57.359 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: [----------] Global test environment set-up. 2024-04-14T15:12:57.359 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2024-04-14T15:12:57.359 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2024-04-14T15:12:57.359 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (75 ms) 2024-04-14T15:12:57.359 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2024-04-14T15:12:57.359 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (63 ms) 2024-04-14T15:12:57.360 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2024-04-14T15:12:57.360 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (2666 ms) 2024-04-14T15:12:57.360 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2024-04-14T15:12:57.360 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:56.054815+0000 mon.a [INF] from='client.24955 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94114-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.360 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:56.054896+0000 mon.a [INF] from='client.? v1:172.21.15.90:0/2381966651' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.360 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:56.054953+0000 mon.a [INF] from='client.24802 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi090-93379-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.360 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:56.055017+0000 mon.a [INF] from='client.24860 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi090-93864-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.360 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:56.055065+0000 mon.a [INF] from='client.? v1:172.21.15.90:0/2763816171' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi090-93731-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.360 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:56.055126+0000 mon.a [INF] from='client.24829 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi090-93444-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.360 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:56.055195+0000 mon.a [INF] from='client.? v1:172.21.15.90:0/495867689' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi090-93652-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.360 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:56.055232+0000 mon.a [INF] from='client.24991 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93995-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.360 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:56.055295+0000 mon.a [INF] from='client.24923 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi090-94131-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.360 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:56.055362+0000 mon.a [INF] from='client.24796 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi090-93342-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.360 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:56.055445+0000 mon.a [INF] from='client.24896 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.361 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:56.055526+0000 mon.a [INF] from='client.24808 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi090-93363-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.393 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:12:57.417 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:57.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:57.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:12:57.465 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:56.055607+0000 mon.a [INF] from='client.24811 ' entity='client.admin' cmd='[{"prefix":rsor=13:62a1935d:::14:head 2024-04-14T15:12:57.465 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : seek to 13:62a1935d:::14:head 2024-04-14T15:12:57.465 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: > 13:62a1935d:::14:head -> 14 2024-04-14T15:12:57.465 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : oid=0 cursor=13:6cac518f:::0:head 2024-04-14T15:12:57.465 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : seek to 13:6cac518f:::0:head 2024-04-14T15:12:57.465 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: > 13:6cac518f:::0:head -> 0 2024-04-14T15:12:57.465 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : oid=15 cursor=13:863748b0:::15:head 2024-04-14T15:12:57.465 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : seek to 13:863748b0:::15:head 2024-04-14T15:12:57.465 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: > 13:863748b0:::15:head -> 15 2024-04-14T15:12:57.465 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : oid=11 cursor=13:89d3ae78:::11:head 2024-04-14T15:12:57.465 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : seek to 13:89d3ae78:::11:head 2024-04-14T15:12:57.466 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: > 13:89d3ae78:::11:head -> 11 2024-04-14T15:12:57.466 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : oid=5 cursor=13:b29083e3:::5:head 2024-04-14T15:12:57.466 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : seek to 13:b29083e3:::5:head 2024-04-14T15:12:57.466 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: > 13:b29083e3:::5:head -> 5 2024-04-14T15:12:57.466 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : oid=8 cursor=13:bd63b0f1:::8:head 2024-04-14T15:12:57.466 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : seek to 13:bd63b0f1:::8:head 2024-04-14T15:12:57.466 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: > 13:bd63b0f1:::8:head -> 8 2024-04-14T15:12:57.466 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : oid=6 cursor=13:c4fdafeb:::6:head 2024-04-14T15:12:57.466 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : seek to 13:c4fdafeb:::6:head 2024-04-14T15:12:57.466 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: > 13:c4fdafeb:::6:head -> 6 2024-04-14T15:12:57.466 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : oid=3 cursor=13:cfc208b3:::3:head 2024-04-14T15:12:57.466 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : seek to 13:cfc208b3:::3:head 2024-04-14T15:12:57.466 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: > 13:cfc208b3:::3:head -> 3 2024-04-14T15:12:57.466 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : oid=4 cursor=13:d83876eb:::4:head 2024-04-14T15:12:57.467 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : seek to 13:d83876eb:::4:head 2024-04-14T15:12:57.467 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: > 13:d83876eb:::4:head -> 4 2024-04-14T15:12:57.467 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : oid=12 cursor=13:de5d7c5f:::12:head 2024-04-14T15:12:57.467 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : seek to 13:de5d7c5f:::12:head 2024-04-14T15:12:57.467 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: > 13:de5d7c5f:::12:head -> 12 2024-04-14T15:12:57.467 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : oid=9 cursor=13:e960b815:::9:head 2024-04-14T15:12:57.467 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : seek to 13:e960b815:::9:head 2024-04-14T15:12:57.467 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: > 13:e960b815:::9:head -> 9 2024-04-14T15:12:57.467 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : oid=2 cursor=13:f905c69b:::2:head 2024-04-14T15:12:57.467 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : seek to 13:f905c69b:::2:head 2024-04-14T15:12:57.467 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: > 13:f905c69b:::2:head -> 2 2024-04-14T15:12:57.467 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : seek to 13:d83876eb:::4:head 2024-04-14T15:12:57.467 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : cursor()=13:d83876eb:::4:head expected=13:d83876eb:::4:head 2024-04-14T15:12:57.467 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: > 13:d83876eb:::4:head -> 4 2024-04-14T15:12:57.467 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : entry=4 expected=4 2024-04-14T15:12:57.468 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : seek to 13:cfc208b3:::3:head 2024-04-14T15:12:57.468 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : cursor()=13:cfc208b3:::3:head expected=13:cfc208b3:::3:head 2024-04-14T15:12:57.468 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: > 13:cfc208b3:::3:head -> 3 2024-04-14T15:12:57.468 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : entry=3 expected=3 2024-04-14T15:12:57.468 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : seek to 13:5c6b0b28:::7:head 2024-04-14T15:12:57.468 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : cursor()=13:5c6b0b28:::7:head expected=13:5c6b0b28:::7:head 2024-04-14T15:12:57.468 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: > 13:5c6b0b28:::7:head -> 7 2024-04-14T15:12:57.468 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : entry=7 expected=7 2024-04-14T15:12:57.468 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : seek to 13:b29083e3:::5:head 2024-04-14T15:12:57.468 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : cursor()=13:b29083e3:::5:head expected=13:b29083e3:::5:head 2024-04-14T15:12:57.468 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: > 13:b29083e3:::5:head -> 5 2024-04-14T15:12:57.468 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : entry=5 expected=5 2024-04-14T15:12:57.468 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : seek to 13:89d3ae78:::11:head 2024-04-14T15:12:57.468 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : cursor()=13:89d3ae78:::11:head expected=13:89d3ae78:::11:head 2024-04-14T15:12:57.468 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: > 13:89d3ae78:::11:head -> 11 2024-04-14T15:12:57.469 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : entry=11 expected=11 2024-04-14T15:12:57.469 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : seek to 13:863748b0:::15:head 2024-04-14T15:12:57.469 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : cursor()=13:863748b0:::15:head expected=13:863748b0:::15:head 2024-04-14T15:12:57.469 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: > 13:863748b0:::15:head -> 15 2024-04-14T15:12:57.469 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : entry=15 expected=15 2024-04-14T15:12:57.469 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : seek to 13:6cac518f:::0:head 2024-04-14T15:12:57.469 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : cursor()=13:6cac518f:::0:head expected=13:6cac518f:::0:head 2024-04-14T15:12:57.469 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: > 13:6cac518f:::0:head -> 0 2024-04-14T15:12:57.469 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : entry=0 expected=0 2024-04-14T15:12:57.469 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : seek to 13:62a1935d:::14:head 2024-04-14T15:12:57.469 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : cursor()=13:62a1935d:::14:head expected=13:62a1935d:::14:head 2024-04-14T15:12:57.469 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: > 13:62a1935d:::14:head -> 14 2024-04-14T15:12:57.519 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:12:57.569 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:57.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:57.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:12:57.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:57 smithi092 ceph-mon[33966]: from='client.24955 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94114-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:57 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2381966651' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:57 smithi092 ceph-mon[33966]: from='client.24802 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi090-93379-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:57 smithi092 ceph-mon[33966]: from='client.24860 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi090-93864-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:57 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2763816171' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi090-93731-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:57 smithi092 ceph-mon[33966]: from='client.24829 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi090-93444-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:57 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/495867689' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi090-93652-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:57 smithi092 ceph-mon[33966]: from='client.24991 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93995-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:57 smithi092 ceph-mon[33966]: from='client.24923 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi090-94131-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:57 smithi092 ceph-mon[33966]: from='client.24796 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi090-93342-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:57 smithi092 ceph-mon[33966]: from='client.24896 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:57 smithi092 ceph-mon[33966]: from='client.24808 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi090-93363-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:57 smithi092 ceph-mon[33966]: from='client.24811 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi090-93398-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:57 smithi092 ceph-mon[33966]: from='client.24823 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi090-93436-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:57 smithi092 ceph-mon[33966]: from='client.24853 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi090-93601-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:57 smithi092 ceph-mon[33966]: from='client.24868 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi090-93724-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:57 smithi092 ceph-mon[33966]: from='client.24871 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi090-93709-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:57 smithi092 ceph-mon[33966]: from='client.24889 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi090-93930-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:57 smithi092 ceph-mon[33966]: from='client.25000 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94011-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:57 smithi092 ceph-mon[33966]: from='client.24835 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi090-93476-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:57 smithi092 ceph-mon[33966]: osdmap e108: 8 total, 8 up, 8 in 2024-04-14T15:12:57.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:57 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2645816859' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-14T15:12:57.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:57 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/309942895' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi090-93998-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi090-93998-1"}]': finished 2024-04-14T15:12:57.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:57 smithi092 ceph-mon[33966]: from='client.25021 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi090-94522-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi090-94522-1"}]': finished 2024-04-14T15:12:57.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:57 smithi092 ceph-mon[33966]: from='client.25001 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi090-94662-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi090-94662-1"}]': finished 2024-04-14T15:12:57.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:57 smithi092 ceph-mon[33966]: osdmap e109: 8 total, 8 up, 8 in 2024-04-14T15:12:57.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:57 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1327720256' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:57.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:57 smithi092 ceph-mon[33966]: from='client.25114 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:57.625 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : en list_parallel: process_1_[94164]: starting. 2024-04-14T15:12:57.625 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_1_[94164]: creating pool ceph_test_rados_list_parallel.smithi090-94111 2024-04-14T15:12:57.625 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_1_[94164]: created object 0... 2024-04-14T15:12:57.625 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_1_[94164]: created object 25... 2024-04-14T15:12:57.625 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_1_[94164]: created object 49... 2024-04-14T15:12:57.625 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_1_[94164]: finishing. 2024-04-14T15:12:57.625 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_1_[94164]: shutting down. 2024-04-14T15:12:57.625 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_2_[94165]: starting. 2024-04-14T15:12:57.625 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_2_[94165]: listing objects. 2024-04-14T15:12:57.626 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_2_[94165]: listed object 0... 2024-04-14T15:12:57.626 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_2_[94165]: listed object 25... 2024-04-14T15:12:57.626 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_2_[94165]: saw 50 objects 2024-04-14T15:12:57.626 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_2_[94165]: shutting down. 2024-04-14T15:12:57.626 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: ******************************* 2024-04-14T15:12:57.626 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_3_[95336]: starting. 2024-04-14T15:12:57.626 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_3_[95336]: creating pool ceph_test_rados_list_parallel.smithi090-94111 2024-04-14T15:12:57.626 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_3_[95336]: created object 0... 2024-04-14T15:12:57.626 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_3_[95336]: created object 25... 2024-04-14T15:12:57.626 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_3_[95336]: created object 49... 2024-04-14T15:12:57.626 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_3_[95336]: finishing. 2024-04-14T15:12:57.626 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_3_[95336]: shutting down. 2024-04-14T15:12:57.626 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: ******************************* 2024-04-14T15:12:57.626 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_4_[95337]: starting. 2024-04-14T15:12:57.627 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_4_[95337]: listing objects. 2024-04-14T15:12:57.627 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_4_[95337]: listed object 0... 2024-04-14T15:12:57.627 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_4_[95337]: listed object 25... 2024-04-14T15:12:57.627 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_4_[95337]: saw 41 objects 2024-04-14T15:12:57.627 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_4_[95337]: shutting down. 2024-04-14T15:12:57.627 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: ******************************* 2024-04-14T15:12:57.627 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_5_[95339]: starting. 2024-04-14T15:12:57.627 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_5_[95339]: removed 25 objects... 2024-04-14T15:12:57.627 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_5_[95339]: removed half of the objects 2024-04-14T15:12:57.627 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_5_[95339]: removed 50 objects... 2024-04-14T15:12:57.627 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_5_[95339]: removed 50 objects 2024-04-14T15:12:57.627 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_5_[95339]: shutting down. 2024-04-14T15:12:57.627 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: ******************************* 2024-04-14T15:12:57.627 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: ******************************* 2024-04-14T15:12:57.627 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_6_[95404]: starting. 2024-04-14T15:12:57.627 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_6_[95404]: creating pool ceph_test_rados_list_parallel.smithi090-94111 2024-04-14T15:12:57.628 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_6_[95404]: created object 0... 2024-04-14T15:12:57.628 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_6_[95404]: created object 25... 2024-04-14T15:12:57.628 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_6_[95404]: created object 49... 2024-04-14T15:12:57.628 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_6_[95404]: finishing. 2024-04-14T15:12:57.628 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_6_[95404]: shutting down. 2024-04-14T15:12:57.628 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: ******************************* 2024-04-14T15:12:57.628 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: ******************************* 2024-04-14T15:12:57.628 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_7_[95405]: starting. 2024-04-14T15:12:57.628 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_7_[95405]: listing objects. 2024-04-14T15:12:57.628 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_7_[95405]: listed object 0... 2024-04-14T15:12:57.628 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_7_[95405]: listed object 25... 2024-04-14T15:12:57.628 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_7_[95405]: listed object 50... 2024-04-14T15:12:57.628 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_7_[95405]: saw 56 objects 2024-04-14T15:12:57.628 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_7_[95405]: shutting down. 2024-04-14T15:12:57.628 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: ******************************* 2024-04-14T15:12:57.629 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: ******************************* 2024-04-14T15:12:57.629 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_8_[95406]: starting. 2024-04-14T15:12:57.629 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_8_[95406]: added 25 objects... 2024-04-14T15:12:57.629 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_8_[95406]: added half of the objects 2024-04-14T15:12:57.629 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_8_[95406]: added 50 objects... 2024-04-14T15:12:57.629 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_8_[95406]: added 50 objects 2024-04-14T15:12:57.629 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_8_[95406]: shutting down. 2024-04-14T15:12:57.629 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: ******************************* 2024-04-14T15:12:57.629 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: ******************************* 2024-04-14T15:12:57.670 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:12:57.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:57 smithi090 ceph-mon[30399]: from='client.24860 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi090-93864-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:57 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2763816171' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi090-93731-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:57 smithi090 ceph-mon[30399]: from='client.24829 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi090-93444-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:57 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/495867689' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi090-93652-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:57 smithi090 ceph-mon[30399]: from='client.24991 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93995-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:57 smithi090 ceph-mon[30399]: from='client.24923 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi090-94131-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:57 smithi090 ceph-mon[30399]: from='client.24796 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi090-93342-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:57 smithi090 ceph-mon[30399]: from='client.24896 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:57 smithi090 ceph-mon[30399]: from='client.24808 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi090-93363-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:57 smithi090 ceph-mon[30399]: from='client.24811 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi090-93398-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:57 smithi090 ceph-mon[30399]: from='client.24823 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi090-93436-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:57 smithi090 ceph-mon[30399]: from='client.24853 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi090-93601-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:57 smithi090 ceph-mon[30399]: from='client.24868 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi090-93724-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:57 smithi090 ceph-mon[30399]: from='client.24871 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi090-93709-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:57 smithi090 ceph-mon[30399]: from='client.24889 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi090-93930-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:57 smithi090 ceph-mon[30399]: from='client.25000 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94011-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:57 smithi090 ceph-mon[30399]: from='client.24835 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi090-93476-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:57.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:57 smithi090 ceph-mon[30399]: osdmap e108: 8 total, 8 up, 8 in 2024-04-14T15:12:57.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:57 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2645816859' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-14T15:12:57.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:57 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/309942895' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi090-93998-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi090-93998-1"}]': finished 2024-04-14T15:12:57.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:57 smithi090 ceph-mon[30399]: from='client.25021 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi090-94522-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi090-94522-1"}]': finished 2024-04-14T15:12:57.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:57 smithi090 ceph-mon[30399]: from='client.25001 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi090-94662-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi090-94662-1"}]': finished 2024-04-14T15:12:57.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:57 smithi090 ceph-mon[30399]: osdmap e109: 8 total, 8 up, 8 in 2024-04-14T15:12:57.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:57 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1327720256' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:57.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:57 smithi090 ceph-mon[30399]: from='client.25114 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:57.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:57.706 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:57.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:12:57.807 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:12:57.835 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:57.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:57.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:12:57.838 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: ******************************* 2024-04-14T15:12:57.838 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_9_[95845]: starting. 2024-04-14T15:12:57.838 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_9_[95845]: creating pool ceph_test_rados_list_parallel.smithi090-94111 2024-04-14T15:12:57.838 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_9_[95845]: created object 0... 2024-04-14T15:12:57.838 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_9_[95845]: created object 25... 2024-04-14T15:12:57.838 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_9_[95845]: created object 49... 2024-04-14T15:12:57.839 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_9_[95845]: finishing. 2024-04-14T15:12:57.839 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_9_[95845]: shutting down. 2024-04-14T15:12:57.839 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: ******************************* 2024-04-14T15:12:57.839 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: ******************************* 2024-04-14T15:12:57.839 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: ******************************* 2024-04-14T15:12:57.839 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_10_[95846]: starting. 2024-04-14T15:12:57.839 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_10_[95846]: listing objects. 2024-04-14T15:12:57.839 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_10_[95846]: listed object 0... 2024-04-14T15:12:57.839 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_10_[95846]: listed object 25... 2024-04-14T15:12:57.839 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_10_[95846]: listed object 50... 2024-04-14T15:12:57.839 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_10_[95846]: listed object 75... 2024-04-14T15:12:57.840 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_10_[95846]: saw 97 objects 2024-04-14T15:12:57.840 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_10_[95846]: shutting down. 2024-04-14T15:12:57.840 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: ******************************* 2024-04-14T15:12:57.840 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: ******************************* 2024-04-14T15:12:57.840 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: ******************************* 2024-04-14T15:12:57.840 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_13_[95849]: starting. 2024-04-14T15:12:57.840 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_13_[95849]: removed 25 objects... 2024-04-14T15:12:57.840 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_13_[95849]: removed half of the objects 2024-04-14T15:12:57.840 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_13_[95849]: removed 50 objects... 2024-04-14T15:12:57.840 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_13_[95849]: removed 50 objects 2024-04-14T15:12:57.840 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_13_[95849]: shutting down. 2024-04-14T15:12:57.840 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: ******************************* 2024-04-14T15:12:57.840 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: ******************************* 2024-04-14T15:12:57.841 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: ******************************* 2024-04-14T15:12:57.841 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_11_[95847]: starting. 2024-04-14T15:12:57.841 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_11_[95847]: added 25 objects... 2024-04-14T15:12:57.841 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_11_[95847]: added half of the objects 2024-04-14T15:12:57.841 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_11_[95847]: added 50 objects... 2024-04-14T15:12:57.841 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_11_[95847]: added 50 objects 2024-04-14T15:12:57.841 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_11_[95847]: shutting down. 2024-04-14T15:12:57.841 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: ******************************* 2024-04-14T15:12:57.841 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: ******************************* 2024-04-14T15:12:57.841 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: ******************************* 2024-04-14T15:12:57.841 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_12_[95848]: starting. 2024-04-14T15:12:57.842 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_12_[95848]: added 25 objects... 2024-04-14T15:12:57.842 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_12_[95848]: added half of the objects 2024-04-14T15:12:57.842 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_12_[95848]: added 50 objects... 2024-04-14T15:12:57.842 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_12_[95848]: added 50 objects 2024-04-14T15:12:57.842 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_12_[95848]: shutting down. 2024-04-14T15:12:57.842 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: ******************************* 2024-04-14T15:12:57.842 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: ******************************* 2024-04-14T15:12:57.842 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: ******************************* 2024-04-14T15:12:57.842 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: ******************************* 2024-04-14T15:12:57.842 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_14_[95971]: starting. 2024-04-14T15:12:57.842 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_14_[95971]: creating pool ceph_test_rados_list_parallel.smithi090-94111 2024-04-14T15:12:57.842 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_14_[95971]: created object 0... 2024-04-14T15:12:57.842 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_14_[95971]: created object 25... 2024-04-14T15:12:57.842 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_14_[95971]: created object 49... 2024-04-14T15:12:57.843 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_14_[95971]: finishing. 2024-04-14T15:12:57.843 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_14_[95971]: shutting down. 2024-04-14T15:12:57.843 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: ******************************* 2024-04-14T15:12:57.843 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: ******************************* 2024-04-14T15:12:57.843 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: ******************************* 2024-04-14T15:12:57.843 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: ******************************* 2024-04-14T15:12:57.843 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_15_[95972]: starting. 2024-04-14T15:12:57.843 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_15_[95972]: listing objects. 2024-04-14T15:12:57.843 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_15_[95972]: listed object 0... 2024-04-14T15:12:57.937 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:12:57.962 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:57.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:57.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:12:58.064 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:12:58.077 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_15_[95972]: listed object 25... 2024-04-14T15:12:58.077 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_15_[95972]: listed object 50... 2024-04-14T15:12:58.077 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_15_[95972]: listed object 75... 2024-04-14T15:12:58.077 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_15_[95972]: listed object 100... 2024-04-14T15:12:58.077 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_15_[95972]: listed object 125... 2024-04-14T15:12:58.077 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_15_[95972]: saw 150 objects 2024-04-14T15:12:58.078 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_15_[95972]: shutting down. 2024-04-14T15:12:58.078 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: ******************************* 2024-04-14T15:12:58.078 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: ******************************* 2024-04-14T15:12:58.078 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: ******************************* 2024-04-14T15:12:58.078 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: ******************************* 2024-04-14T15:12:58.078 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_16_[95973]: starting. 2024-04-14T15:12:58.078 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_16_[95973]: added 25 objects... 2024-04-14T15:12:58.078 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_16_[95973]: added half of the objects 2024-04-14T15:12:58.078 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_16_[95973]: added 50 objects... 2024-04-14T15:12:58.078 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_16_[95973]: added 50 objects 2024-04-14T15:12:58.078 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: process_16_[95973]: shutting down. 2024-04-14T15:12:58.078 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: ******************************* 2024-04-14T15:12:58.078 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: ******************************* 2024-04-14T15:12:58.078 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: ******************************* 2024-04-14T15:12:58.079 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: ******************************* 2024-04-14T15:12:58.079 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: ******************************* 2024-04-14T15:12:58.079 INFO:tasks.workunit.client.0.smithi090.stdout: list_parallel: ******* SUCCESS ********** 2024-04-14T15:12:58.116 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:58.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:58.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:12:58.200 INFO:tasks.workunit.client.0.smithi090.stdout: "osd pool application enable","pool": "LibRadosIoPP_smithi090-93398-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:58.201 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:56.055659+0000 mon.a [INF] from='client.24823 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi090-93436-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:58.201 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:56.055714+0000 mon.a [INF] from='client.24853 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi090-93601-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:58.201 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:56.055772+0000 mon.a [INF] from='client.24868 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi090-93724-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:58.201 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:56.055838+0000 mon.a [INF] from='client.24871 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi090-93709-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:58.201 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:56.055877+0000 mon.a [INF] from='client.24889 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi090-93930-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:58.201 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:56.055917+0000 mon.a [INF] from='client.25000 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94011-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:58.201 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:56.055955+0000 mon.a [INF] from='client.24835 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi090-93476-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:58.201 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:57.060400+0000 mon.a [INF] from='client.? v1:172.21.15.90:0/309942895' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi090-93998-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi090-93998-1"}]': finished 2024-04-14T15:12:58.201 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:57.060523+0000 mon.a [INF] from='client.25021 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi090-94522-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi090-94522-1"}]': finished 2024-04-14T15:12:58.201 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:57.060610+0000 mon.a [INF] from='client.25001 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi090-94662-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi090-94662-1"}]': finished 2024-04-14T15:12:58.201 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:57.074993+0000 mon.b [INF] from='client.? v1:172.21.15.90:0/1327720256' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:58.201 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:57.106102+0000 mon.a [INF] from='client.25114 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:58.201 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:57.179328+0000 mon.a [WRN] Health check failed: 12 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:12:58.202 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:57.345444+0000 mon.b [INF] from='client.? v1:172.21.15.90:0/4291843131' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi090-93398-23"}]: dispatch 2024-04-14T15:12:58.202 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:57.346178+0000 mon.b [INF] from='client.? v1:172.21.15.90:0/3178004355' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi090-93379-16"}]: dispatch 2024-04-14T15:12:58.202 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:57.347770+0000 mon.b [INF] from='client.? v1:172.21.15.90:0/3274182089' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi090-93731-7"}]: dispatch 2024-04-14T15:12:58.202 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:57.349449+0000 mon.b [INF] from='client.? v1:172.21.15.90:0/4159084683' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-04-14T15:12:58.202 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:57.350855+0000 client.admin [INF] onexx 2024-04-14T15:12:58.202 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:57.368141+0000 mon.b [INF] from='client.? v1:172.21.15.90:0/1324117929' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi090-93864-7"}]: dispatch 2024-04-14T15:12:58.202 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:57.416292+0000 mon.a [INF] from='client.25243 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi090-93398-23"}]: dispatch 2024-04-14T15:12:58.202 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:57.416705+0000 mon.a [INF] from='client.25240 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi090-93379-16"}]: dispatch 2024-04-14T15:12:58.202 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:57.417036+0000 mon.a [INF] from='client.25249 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi090-93731-7"}]: dispatch 2024-04-14T15:12:58.202 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:57.417530+0000 mon.b [INF] from='client.? v1:172.21.15.90:0/3178004355' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi090-93379-16"}]: dispatch 2024-04-14T15:12:58.202 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:57.418640+0000 mon.b [INF] from='client.? v1:172.21.15.90:0/4291843131' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi090-93398-23"}]: dispatch 2024-04-14T15:12:58.203 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:57.418785+0000 mon.b [INF] from='client.? v1:172.21.15.90:0/3274182089' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi090-93731-7"}]: dispatch 2024-04-14T15:12:58.203 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:57.438471+0000 mon.a [INF] from='client.25252 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-04-14T15:12:58.203 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:57.469174+0000 mon.a [INF] from='client.25267 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi090-93864-7"}]: dispatch 2024-04-14T15:12:58.203 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:57.469462+0000 mon.a [INF] from='client.25240 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi090-93379-16"}]: dispatch 2024-04-14T15:12:58.203 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:57.469703+0000 mon.a [INF] from='client.25243 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi090-93398-23"}]: dispatch 2024-04-14T15:12:58.203 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:57.469902+0000 mon.a [INF] from='client.25249 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi090-93731-7"}]: dispatch 2024-04-14T15:12:58.203 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:57.470969+0000 mon.b [INF] from='client.? v1:172.21.15.90:0/3178004355' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi090-93379-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:12:58.203 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:57.471132+0000 mon.b [INF] from='client.? v1:172.21.15.90:0/4291843131' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi090-93398-23", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:12:58.203 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:57.471325+0000 mon.b [INF] from='client.? v1:172.21.15.90:0/1324117929' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi090-93864-7"}]: dispatch 2024-04-14T15:12:58.203 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:57.471481+0000 mon.b [INF] from='client.? v1:172.21.15.90:0/3274182089' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi090-93731-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:12:58.203 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:57.471763+0000 mon.a [INF] from='client.25240 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi090-93379-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:12:58.203 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:57.472513+0000 mon.a [INF] from='client.25243 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi090-93398-23", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:12:58.203 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:57.472847+0000 mon.a [INF] from='client.25267 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi090-93864-7"}]: dispatch 2024-04-14T15:12:58.204 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:57.473227+0000 mon.a [INF] from='client.25249 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi090-93731-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:12:58.204 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:57.475531+0000 mon.b [INF] from='client.? v1:172.21.15.90:0/1324117929' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi090-93864-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:12:58.204 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:57.476140+0000 mon.a [INF] from='client.25267 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi090-93864-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:12:58.204 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:58.064728+0000 mon.a [INF] from='client.25114 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:58.204 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:58.064809+0000 mon.a [INF] from='client.25240 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi090-93379-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:12:58.204 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:58.064870+0000 mon.a [INF] from='client.25243 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi090-93398-23", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:12:58.204 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:58.064932+0000 mon.a [INF] from='client.25249 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi090-93731-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:12:58.204 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:58.065003+0000 mon.a [INF] from='client.25267 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi090-93864-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:12:58.204 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:58.072918+0000 mon.b [INF] from='client.? v1:172.21.15.90:0/3178004355' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi090-93379-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi090-93379-16"}]: dispatch 2024-04-14T15:12:58.204 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:58.073311+0000 mon.b [INF] from='client.? v1:172.21.15.90:0/4291843131' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi090-93398-23", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi090-93398-23"}]: dispatch 2024-04-14T15:12:58.204 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:58.073622+0000 mon.b [INF] from='client.? v1:172.21.15.90:0/1324117929' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi090-93864-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi090-93864-7"}]: dispatch 2024-04-14T15:12:58.204 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:58.074043+0000 mon.b [INF] from='client.? v1:172.21.15.90:0/3274182089' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi090-93731-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi090-93731-7"}]: dispatch 2024-04-14T15:12:58.204 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:58.074293+0000 mon.a [INF] from='client.25240 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi090-93379-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi090-93379-16"}]: dispatch 2024-04-14T15:12:58.205 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:58.074691+0000 mon.a [INF] from='client.25243 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi090-93398-23", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi090-93398-23"}]: dispatch 2024-04-14T15:12:58.205 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:58.075273+0000 mon.b [INF] from='client.? v1:172.21.15.90:0/1118408389' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi090-94522-1"}]: dispatch 2024-04-14T15:12:58.205 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:58.075746+0000 mon.a [INF] from='client.25267 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi090-93864-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi090-93864-7"}]: dispatch 2024-04-14T15:12:58.205 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:58.076020+0000 mon.b [INF] from='client.? v1:172.21.15.90:0/246759086' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi090-93342-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:58.205 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:58.076227+0000 mon.a [INF] from='client.25249 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi090-93731-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi090-93731-7"}]: dispatch 2024-04-14T15:12:58.205 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:58.076366+0000 mon.b [INF] from='client.? v1:172.21.15.90:0/1067094721' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:58.205 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:58.076601+0000 mon.b [INF] from='client.? v1:172.21.15.90:0/532832389' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi090-93363-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:58.205 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:58.076852+0000 mon.b [INF] from='client.? v1:172.21.15.90:0/265571846' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:58.205 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:58.080497+0000 mon.a [INF] from='client.25021 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi090-94522-1"}]: dispatch 2024-04-14T15:12:58.217 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:12:58.242 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:58.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:58.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:12:58.344 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:12:58.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:58.369 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:58.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:12:58.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[41670]: pgmap v109: 1156 pgs: 1024 unknown, 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:12:58.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[41670]: Health check failed: 12 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:12:58.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/4291843131' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi090-93398-23"}]: dispatch 2024-04-14T15:12:58.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3178004355' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi090-93379-16"}]: dispatch 2024-04-14T15:12:58.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3274182089' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi090-93731-7"}]: dispatch 2024-04-14T15:12:58.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/4159084683' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-04-14T15:12:58.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[41670]: onexx 2024-04-14T15:12:58.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1324117929' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi090-93864-7"}]: dispatch 2024-04-14T15:12:58.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[41670]: from='client.25243 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi090-93398-23"}]: dispatch 2024-04-14T15:12:58.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[41670]: from='client.25240 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi090-93379-16"}]: dispatch 2024-04-14T15:12:58.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[41670]: from='client.25249 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi090-93731-7"}]: dispatch 2024-04-14T15:12:58.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3178004355' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi090-93379-16"}]: dispatch 2024-04-14T15:12:58.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/4291843131' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi090-93398-23"}]: dispatch 2024-04-14T15:12:58.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3274182089' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi090-93731-7"}]: dispatch 2024-04-14T15:12:58.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[41670]: from='client.25252 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-04-14T15:12:58.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[41670]: from='client.25267 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi090-93864-7"}]: dispatch 2024-04-14T15:12:58.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[41670]: from='client.25240 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi090-93379-16"}]: dispatch 2024-04-14T15:12:58.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[41670]: from='client.25243 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi090-93398-23"}]: dispatch 2024-04-14T15:12:58.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[41670]: from='client.25249 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi090-93731-7"}]: dispatch 2024-04-14T15:12:58.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3178004355' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi090-93379-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:12:58.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/4291843131' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi090-93398-23", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:12:58.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[30399]: pgmap v109: 1156 pgs: 1024 unknown, 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:12:58.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[30399]: Health check failed: 12 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:12:58.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/4291843131' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi090-93398-23"}]: dispatch 2024-04-14T15:12:58.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3178004355' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi090-93379-16"}]: dispatch 2024-04-14T15:12:58.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3274182089' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi090-93731-7"}]: dispatch 2024-04-14T15:12:58.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/4159084683' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-04-14T15:12:58.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[30399]: onexx 2024-04-14T15:12:58.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1324117929' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi090-93864-7"}]: dispatch 2024-04-14T15:12:58.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[30399]: from='client.25243 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi090-93398-23"}]: dispatch 2024-04-14T15:12:58.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[30399]: from='client.25240 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi090-93379-16"}]: dispatch 2024-04-14T15:12:58.445 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[30399]: from='client.25249 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi090-93731-7"}]: dispatch 2024-04-14T15:12:58.445 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3178004355' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi090-93379-16"}]: dispatch 2024-04-14T15:12:58.445 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/4291843131' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi090-93398-23"}]: dispatch 2024-04-14T15:12:58.445 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3274182089' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi090-93731-7"}]: dispatch 2024-04-14T15:12:58.445 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[30399]: from='client.25252 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-04-14T15:12:58.445 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[30399]: from='client.25267 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi090-93864-7"}]: dispatch 2024-04-14T15:12:58.445 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[30399]: from='client.25240 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi090-93379-16"}]: dispatch 2024-04-14T15:12:58.445 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[30399]: from='client.25243 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi090-93398-23"}]: dispatch 2024-04-14T15:12:58.445 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[30399]: from='client.25249 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi090-93731-7"}]: dispatch 2024-04-14T15:12:58.445 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3178004355' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi090-93379-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:12:58.445 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/4291843131' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi090-93398-23", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:12:58.445 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1324117929' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi090-93864-7"}]: dispatch 2024-04-14T15:12:58.445 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3274182089' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi090-93731-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:12:58.445 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[30399]: from='client.25240 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi090-93379-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:12:58.446 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[30399]: from='client.25243 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi090-93398-23", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:12:58.446 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[30399]: from='client.25267 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi090-93864-7"}]: dispatch 2024-04-14T15:12:58.446 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[30399]: from='client.25249 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi090-93731-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:12:58.446 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1324117929' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi090-93864-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:12:58.446 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[30399]: from='client.25267 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi090-93864-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:12:58.447 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:12:58.447 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[30399]: from='client.25114 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:58.447 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[30399]: from='client.25240 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi090-93379-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:12:58.447 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[30399]: from='client.25243 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi090-93398-23", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:12:58.447 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[30399]: from='client.25249 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi090-93731-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:12:58.447 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[30399]: from='client.25267 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi090-93864-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:12:58.447 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[30399]: osdmap e110: 8 total, 8 up, 8 in 2024-04-14T15:12:58.447 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3178004355' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi090-93379-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi090-93379-16"}]: dispatch 2024-04-14T15:12:58.447 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/4291843131' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi090-93398-23", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi090-93398-23"}]: dispatch 2024-04-14T15:12:58.447 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1324117929' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi090-93864-7"}]: dispatch 2024-04-14T15:12:58.448 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3274182089' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi090-93731-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:12:58.448 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[41670]: from='client.25240 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi090-93379-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:12:58.448 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[41670]: from='client.25243 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi090-93398-23", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:12:58.448 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[41670]: from='client.25267 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi090-93864-7"}]: dispatch 2024-04-14T15:12:58.448 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[41670]: from='client.25249 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi090-93731-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:12:58.448 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1324117929' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi090-93864-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:12:58.448 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[41670]: from='client.25267 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi090-93864-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:12:58.448 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:12:58.448 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[41670]: from='client.25114 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:58.448 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[41670]: from='client.25240 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi090-93379-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:12:58.448 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[41670]: from='client.25243 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi090-93398-23", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:12:58.448 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[41670]: from='client.25249 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi090-93731-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:12:58.448 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[41670]: from='client.25267 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi090-93864-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:12:58.448 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[41670]: osdmap e110: 8 total, 8 up, 8 in 2024-04-14T15:12:58.449 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3178004355' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi090-93379-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi090-93379-16"}]: dispatch 2024-04-14T15:12:58.449 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/4291843131' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi090-93398-23", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi090-93398-23"}]: dispatch 2024-04-14T15:12:58.449 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1324117929' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi090-93864-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi090-93864-7"}]: dispatch 2024-04-14T15:12:58.449 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3274182089' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi090-93731-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi090-93731-7"}]: dispatch 2024-04-14T15:12:58.449 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[30399]: from='client.25240 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi090-93379-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi090-93379-16"}]: dispatch 2024-04-14T15:12:58.449 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[30399]: from='client.25243 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi090-93398-23", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi090-93398-23"}]: dispatch 2024-04-14T15:12:58.449 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1118408389' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi090-94522-1"}]: dispatch 2024-04-14T15:12:58.449 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[30399]: from='client.25267 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi090-93864-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi090-93864-7"}]: dispatch 2024-04-14T15:12:58.449 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/246759086' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi090-93342-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:58.449 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[30399]: from='client.25249 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi090-93731-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi090-93731-7"}]: dispatch 2024-04-14T15:12:58.449 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1067094721' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:58.450 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/532832389' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi090-93363-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:58.450 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/265571846' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:58.450 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[30399]: from='client.25021 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi090-94522-1"}]: dispatch 2024-04-14T15:12:58.450 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[30399]: from='client.25192 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi090-93342-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:58.450 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[30399]: from='client.25258 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:58.450 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[30399]: from='client.25207 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi090-93363-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:58.450 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[30399]: from='client.25246 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:58.450 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1324117929' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi090-93864-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi090-93864-7"}]: dispatch 2024-04-14T15:12:58.450 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3274182089' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi090-93731-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi090-93731-7"}]: dispatch 2024-04-14T15:12:58.450 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[41670]: from='client.25240 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi090-93379-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi090-93379-16"}]: dispatch 2024-04-14T15:12:58.450 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[41670]: from='client.25243 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi090-93398-23", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi090-93398-23"}]: dispatch 2024-04-14T15:12:58.451 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1118408389' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi090-94522-1"}]: dispatch 2024-04-14T15:12:58.451 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[41670]: from='client.25267 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi090-93864-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi090-93864-7"}]: dispatch 2024-04-14T15:12:58.451 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/246759086' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi090-93342-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:58.451 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[41670]: from='client.25249 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi090-93731-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi090-93731-7"}]: dispatch 2024-04-14T15:12:58.451 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1067094721' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:58.451 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/532832389' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi090-93363-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:58.451 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/265571846' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:58.451 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[41670]: from='client.25021 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi090-94522-1"}]: dispatch 2024-04-14T15:12:58.451 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[41670]: from='client.25192 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi090-93342-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:58.451 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[41670]: from='client.25258 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:58.451 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[41670]: from='client.25207 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi090-93363-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:58.451 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:58 smithi090 ceph-mon[41670]: from='client.25246 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:58.470 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:12:58.495 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:58.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:58.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:12:58.597 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:12:58.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:58.622 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:58.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:12:58.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:58 smithi092 ceph-mon[33966]: pgmap v109: 1156 pgs: 1024 unknown, 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:12:58.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:58 smithi092 ceph-mon[33966]: Health check failed: 12 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:12:58.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:58 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/4291843131' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi090-93398-23"}]: dispatch 2024-04-14T15:12:58.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:58 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3178004355' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi090-93379-16"}]: dispatch 2024-04-14T15:12:58.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:58 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3274182089' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi090-93731-7"}]: dispatch 2024-04-14T15:12:58.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:58 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/4159084683' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-04-14T15:12:58.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:58 smithi092 ceph-mon[33966]: onexx 2024-04-14T15:12:58.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:58 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1324117929' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi090-93864-7"}]: dispatch 2024-04-14T15:12:58.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:58 smithi092 ceph-mon[33966]: from='client.25243 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi090-93398-23"}]: dispatch 2024-04-14T15:12:58.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:58 smithi092 ceph-mon[33966]: from='client.25240 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi090-93379-16"}]: dispatch 2024-04-14T15:12:58.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:58 smithi092 ceph-mon[33966]: from='client.25249 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi090-93731-7"}]: dispatch 2024-04-14T15:12:58.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:58 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3178004355' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi090-93379-16"}]: dispatch 2024-04-14T15:12:58.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:58 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/4291843131' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi090-93398-23"}]: dispatch 2024-04-14T15:12:58.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:58 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3274182089' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi090-93731-7"}]: dispatch 2024-04-14T15:12:58.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:58 smithi092 ceph-mon[33966]: from='client.25252 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-04-14T15:12:58.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:58 smithi092 ceph-mon[33966]: from='client.25267 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi090-93864-7"}]: dispatch 2024-04-14T15:12:58.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:58 smithi092 ceph-mon[33966]: from='client.25240 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi090-93379-16"}]: dispatch 2024-04-14T15:12:58.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:58 smithi092 ceph-mon[33966]: from='client.25243 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi090-93398-23"}]: dispatch 2024-04-14T15:12:58.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:58 smithi092 ceph-mon[33966]: from='client.25249 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi090-93731-7"}]: dispatch 2024-04-14T15:12:58.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:58 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3178004355' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi090-93379-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:12:58.625 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:58 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/4291843131' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi090-93398-23", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:12:58.625 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:58 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1324117929' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi090-93864-7"}]: dispatch 2024-04-14T15:12:58.625 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:58 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3274182089' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi090-93731-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:12:58.625 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:58 smithi092 ceph-mon[33966]: from='client.25240 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi090-93379-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:12:58.625 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:58 smithi092 ceph-mon[33966]: from='client.25243 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi090-93398-23", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:12:58.625 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:58 smithi092 ceph-mon[33966]: from='client.25267 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi090-93864-7"}]: dispatch 2024-04-14T15:12:58.625 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:58 smithi092 ceph-mon[33966]: from='client.25249 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi090-93731-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:12:58.625 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:58 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1324117929' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi090-93864-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:12:58.625 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:58 smithi092 ceph-mon[33966]: from='client.25267 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi090-93864-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:12:58.625 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:58 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:12:58.625 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:58 smithi092 ceph-mon[33966]: from='client.25114 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:58.625 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:58 smithi092 ceph-mon[33966]: from='client.25240 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi090-93379-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:12:58.625 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:58 smithi092 ceph-mon[33966]: from='client.25243 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi090-93398-23", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:12:58.625 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:58 smithi092 ceph-mon[33966]: from='client.25249 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi090-93731-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:12:58.626 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:58 smithi092 ceph-mon[33966]: from='client.25267 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi090-93864-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:12:58.626 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:58 smithi092 ceph-mon[33966]: osdmap e110: 8 total, 8 up, 8 in 2024-04-14T15:12:58.626 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:58 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3178004355' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi090-93379-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi090-93379-16"}]: dispatch 2024-04-14T15:12:58.626 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:58 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/4291843131' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi090-93398-23", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi090-93398-23"}]: dispatch 2024-04-14T15:12:58.626 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:58 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1324117929' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi090-93864-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi090-93864-7"}]: dispatch 2024-04-14T15:12:58.626 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:58 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3274182089' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi090-93731-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi090-93731-7"}]: dispatch 2024-04-14T15:12:58.626 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:58 smithi092 ceph-mon[33966]: from='client.25240 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi090-93379-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi090-93379-16"}]: dispatch 2024-04-14T15:12:58.626 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:58 smithi092 ceph-mon[33966]: from='client.25243 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi090-93398-23", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi090-93398-23"}]: dispatch 2024-04-14T15:12:58.626 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:58 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1118408389' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi090-94522-1"}]: dispatch 2024-04-14T15:12:58.626 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:58 smithi092 ceph-mon[33966]: from='client.25267 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi090-93864-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi090-93864-7"}]: dispatch 2024-04-14T15:12:58.626 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:58 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/246759086' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi090-93342-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:58.626 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:58 smithi092 ceph-mon[33966]: from='client.25249 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi090-93731-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi090-93731-7"}]: dispatch 2024-04-14T15:12:58.627 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:58 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1067094721' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:58.627 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:58 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/532832389' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi090-93363-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:58.627 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:58 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/265571846' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:58.627 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:58 smithi092 ceph-mon[33966]: from='client.25021 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi090-94522-1"}]: dispatch 2024-04-14T15:12:58.627 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:58 smithi092 ceph-mon[33966]: from='client.25192 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi090-93342-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:58.627 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:58 smithi092 ceph-mon[33966]: from='client.25258 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:58.627 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:58 smithi092 ceph-mon[33966]: from='client.25207 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi090-93363-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:58.627 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:58 smithi092 ceph-mon[33966]: from='client.25246 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:58.724 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:12:58.748 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:58.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:58.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:12:58.850 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:12:58.877 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:58.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:58.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:12:58.979 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:12:59.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:59.003 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:59.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:12:59.104 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:12:59.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:59.161 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:59.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:12:59.262 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:12:59.290 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:59.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:59.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:12:59.392 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:12:59.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:59.417 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:59.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:12:59.518 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:12:59.561 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:59.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:59.562 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:12:59.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:59 smithi092 ceph-mon[33966]: from='client.25252 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-04-14T15:12:59.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:59 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/4159084683' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-04-14T15:12:59.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:59 smithi092 ceph-mon[33966]: twoxx 2024-04-14T15:12:59.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:59 smithi092 ceph-mon[33966]: from='client.25252 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-04-14T15:12:59.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:59 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:12:59.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:59 smithi092 ceph-mon[33966]: from='client.25021 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi090-94522-1"}]': finished 2024-04-14T15:12:59.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:59 smithi092 ceph-mon[33966]: from='client.25192 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi090-93342-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:59.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:59 smithi092 ceph-mon[33966]: from='client.25258 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:59.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:59 smithi092 ceph-mon[33966]: from='client.25207 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi090-93363-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:59.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:59 smithi092 ceph-mon[33966]: from='client.25246 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:59.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:59 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1118408389' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi090-94522-1"}]: dispatch 2024-04-14T15:12:59.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:59 smithi092 ceph-mon[33966]: osdmap e111: 8 total, 8 up, 8 in 2024-04-14T15:12:59.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:59 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1327720256' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:59.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:59 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1465058495' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi090-94662-1"}]: dispatch 2024-04-14T15:12:59.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:59 smithi092 ceph-mon[33966]: from='client.25021 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi090-94522-1"}]: dispatch 2024-04-14T15:12:59.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:59 smithi092 ceph-mon[33966]: from='client.25001 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi090-94662-1"}]: dispatch 2024-04-14T15:12:59.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:59 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/309942895' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi090-93998-1"}]: dispatch 2024-04-14T15:12:59.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:59 smithi092 ceph-mon[33966]: from='client.25114 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:59.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:59 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/274362209' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi090-93436-10"}]: dispatch 2024-04-14T15:12:59.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:12:59 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/274362209' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi090-93436-10"}]: dispatch 2024-04-14T15:12:59.663 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:12:59.686 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:59.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:59.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:12:59.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:59 smithi090 ceph-mon[30399]: from='client.25252 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-04-14T15:12:59.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:59 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/4159084683' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-04-14T15:12:59.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:59 smithi090 ceph-mon[30399]: twoxx 2024-04-14T15:12:59.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:59 smithi090 ceph-mon[30399]: from='client.25252 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-04-14T15:12:59.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:59 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:12:59.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:59 smithi090 ceph-mon[30399]: from='client.25021 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi090-94522-1"}]': finished 2024-04-14T15:12:59.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:59 smithi090 ceph-mon[30399]: from='client.25192 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi090-93342-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:59.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:59 smithi090 ceph-mon[30399]: from='client.25258 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:59.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:59 smithi090 ceph-mon[30399]: from='client.25207 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi090-93363-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:59.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:59 smithi090 ceph-mon[30399]: from='client.25246 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:59.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:59 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1118408389' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi090-94522-1"}]: dispatch 2024-04-14T15:12:59.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:59 smithi090 ceph-mon[30399]: osdmap e111: 8 total, 8 up, 8 in 2024-04-14T15:12:59.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:59 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1327720256' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:59.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:59 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1465058495' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi090-94662-1"}]: dispatch 2024-04-14T15:12:59.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:59 smithi090 ceph-mon[30399]: from='client.25021 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi090-94522-1"}]: dispatch 2024-04-14T15:12:59.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:59 smithi090 ceph-mon[30399]: from='client.25001 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi090-94662-1"}]: dispatch 2024-04-14T15:12:59.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:59 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/309942895' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi090-93998-1"}]: dispatch 2024-04-14T15:12:59.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:59 smithi090 ceph-mon[30399]: from='client.25114 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:59.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:59 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/274362209' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi090-93436-10"}]: dispatch 2024-04-14T15:12:59.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:12:59 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/274362209' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi090-93436-10"}]: dispatch 2024-04-14T15:12:59.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:59 smithi090 ceph-mon[41670]: from='client.25252 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-04-14T15:12:59.694 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:59 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/4159084683' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-04-14T15:12:59.694 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:59 smithi090 ceph-mon[41670]: twoxx 2024-04-14T15:12:59.694 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:59 smithi090 ceph-mon[41670]: from='client.25252 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-04-14T15:12:59.694 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:59 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:12:59.694 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:59 smithi090 ceph-mon[41670]: from='client.25021 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi090-94522-1"}]': finished 2024-04-14T15:12:59.694 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:59 smithi090 ceph-mon[41670]: from='client.25192 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi090-93342-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:59.694 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:59 smithi090 ceph-mon[41670]: from='client.25258 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:59.694 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:59 smithi090 ceph-mon[41670]: from='client.25207 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi090-93363-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:59.694 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:59 smithi090 ceph-mon[41670]: from='client.25246 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:12:59.694 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:59 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1118408389' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi090-94522-1"}]: dispatch 2024-04-14T15:12:59.694 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:59 smithi090 ceph-mon[41670]: osdmap e111: 8 total, 8 up, 8 in 2024-04-14T15:12:59.694 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:59 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1327720256' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:59.694 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:59 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1465058495' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi090-94662-1"}]: dispatch 2024-04-14T15:12:59.695 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:59 smithi090 ceph-mon[41670]: from='client.25021 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi090-94522-1"}]: dispatch 2024-04-14T15:12:59.695 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:59 smithi090 ceph-mon[41670]: from='client.25001 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi090-94662-1"}]: dispatch 2024-04-14T15:12:59.695 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:59 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/309942895' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi090-93998-1"}]: dispatch 2024-04-14T15:12:59.695 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:59 smithi090 ceph-mon[41670]: from='client.25114 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:12:59.695 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:59 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/274362209' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi090-93436-10"}]: dispatch 2024-04-14T15:12:59.695 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:12:59 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/274362209' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi090-93436-10"}]: dispatch 2024-04-14T15:12:59.789 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:12:59.812 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:12:59.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:59.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:12:59.908 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-14T15:12:59.914 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:12:59.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:12:59.975 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:12:59.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:13:00.077 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:13:00.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:00.110 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:00.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:13:00.125 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:58.092635+0000 mon.a [INF] from='client.25192 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi090-93342-2","app": "rados","yes_i_really_mea open_pools_parallel: process_1_[94322]: starting. 2024-04-14T15:13:00.125 INFO:tasks.workunit.client.0.smithi090.stdout: open_pools_parallel: process_1_[94322]: creating pool ceph_test_rados_open_pools_parallel.smithi090-94230 2024-04-14T15:13:00.125 INFO:tasks.workunit.client.0.smithi090.stdout: open_pools_parallel: process_1_[94322]: created object 0... 2024-04-14T15:13:00.125 INFO:tasks.workunit.client.0.smithi090.stdout: open_pools_parallel: process_1_[94322]: created object 25... 2024-04-14T15:13:00.125 INFO:tasks.workunit.client.0.smithi090.stdout: open_pools_parallel: process_1_[94322]: created object 49... 2024-04-14T15:13:00.125 INFO:tasks.workunit.client.0.smithi090.stdout: open_pools_parallel: process_1_[94322]: finishing. 2024-04-14T15:13:00.125 INFO:tasks.workunit.client.0.smithi090.stdout: open_pools_parallel: process_1_[94322]: shutting down. 2024-04-14T15:13:00.125 INFO:tasks.workunit.client.0.smithi090.stdout: open_pools_parallel: process_2_[94325]: starting. 2024-04-14T15:13:00.125 INFO:tasks.workunit.client.0.smithi090.stdout: open_pools_parallel: process_2_[94325]: rados_pool_create. 2024-04-14T15:13:00.126 INFO:tasks.workunit.client.0.smithi090.stdout: open_pools_parallel: process_2_[94325]: rados_ioctx_create. 2024-04-14T15:13:00.126 INFO:tasks.workunit.client.0.smithi090.stdout: open_pools_parallel: process_2_[94325]: shutting down. 2024-04-14T15:13:00.126 INFO:tasks.workunit.client.0.smithi090.stdout: open_pools_parallel: ******************************* 2024-04-14T15:13:00.126 INFO:tasks.workunit.client.0.smithi090.stdout: open_pools_parallel: process_3_[95470]: starting. 2024-04-14T15:13:00.126 INFO:tasks.workunit.client.0.smithi090.stdout: open_pools_parallel: process_3_[95470]: creating pool ceph_test_rados_open_pools_parallel.smithi090-94230 2024-04-14T15:13:00.126 INFO:tasks.workunit.client.0.smithi090.stdout: open_pools_parallel: process_3_[95470]: created object 0... 2024-04-14T15:13:00.126 INFO:tasks.workunit.client.0.smithi090.stdout: open_pools_parallel: process_3_[95470]: created object 25... 2024-04-14T15:13:00.126 INFO:tasks.workunit.client.0.smithi090.stdout: open_pools_parallel: process_3_[95470]: created object 49... 2024-04-14T15:13:00.126 INFO:tasks.workunit.client.0.smithi090.stdout: open_pools_parallel: process_3_[95470]: finishing. 2024-04-14T15:13:00.126 INFO:tasks.workunit.client.0.smithi090.stdout: open_pools_parallel: process_3_[95470]: shutting down. 2024-04-14T15:13:00.126 INFO:tasks.workunit.client.0.smithi090.stdout: open_pools_parallel: ******************************* 2024-04-14T15:13:00.126 INFO:tasks.workunit.client.0.smithi090.stdout: open_pools_parallel: process_4_[95471]: starting. 2024-04-14T15:13:00.126 INFO:tasks.workunit.client.0.smithi090.stdout: open_pools_parallel: process_4_[95471]: rados_pool_create. 2024-04-14T15:13:00.127 INFO:tasks.workunit.client.0.smithi090.stdout: open_pools_parallel: process_4_[95471]: rados_ioctx_create. 2024-04-14T15:13:00.127 INFO:tasks.workunit.client.0.smithi090.stdout: open_pools_parallel: process_4_[95471]: shutting down. 2024-04-14T15:13:00.127 INFO:tasks.workunit.client.0.smithi090.stdout: open_pools_parallel: ******************************* 2024-04-14T15:13:00.127 INFO:tasks.workunit.client.0.smithi090.stdout: open_pools_parallel: ******************************* 2024-04-14T15:13:00.127 INFO:tasks.workunit.client.0.smithi090.stdout: open_pools_parallel: ******* SUCCESS ********** 2024-04-14T15:13:00.133 INFO:tasks.workunit.client.0.smithi090.stdout: cls: Running main() from gmock_main.cc 2024-04-14T15:13:00.133 INFO:tasks.workunit.client.0.smithi090.stdout: cls: [==========] Running 2 tests from 1 test suite. 2024-04-14T15:13:00.134 INFO:tasks.workunit.client.0.smithi090.stdout: cls: [----------] Global test environment set-up. 2024-04-14T15:13:00.134 INFO:tasks.workunit.client.0.smithi090.stdout: cls: [----------] 2 tests from NeoRadosCls 2024-04-14T15:13:00.134 INFO:tasks.workunit.client.0.smithi090.stdout: cls: [ RUN ] NeoRadosCls.DNE 2024-04-14T15:13:00.134 INFO:tasks.workunit.client.0.smithi090.stdout: cls: [ OK ] NeoRadosCls.DNE (2630 ms) 2024-04-14T15:13:00.134 INFO:tasks.workunit.client.0.smithi090.stdout: cls: [ RUN ] NeoRadosCls.RemoteReads 2024-04-14T15:13:00.134 INFO:tasks.workunit.client.0.smithi090.stdout: cls: [ OK ] NeoRadosCls.RemoteReads (3043 ms) 2024-04-14T15:13:00.134 INFO:tasks.workunit.client.0.smithi090.stdout: cls: [----------] 2 tests from NeoRadosCls (5673 ms total) 2024-04-14T15:13:00.134 INFO:tasks.workunit.client.0.smithi090.stdout: cls: 2024-04-14T15:13:00.134 INFO:tasks.workunit.client.0.smithi090.stdout: cls: [----------] Global test environment tear-down 2024-04-14T15:13:00.134 INFO:tasks.workunit.client.0.smithi090.stdout: cls: [==========] 2 tests from 1 test suite ran. (5673 ms total) 2024-04-14T15:13:00.134 INFO:tasks.workunit.client.0.smithi090.stdout: cls: [ PASSED ] 2 tests. 2024-04-14T15:13:00.135 INFO:tasks.workunit.client.0.smithi090.stdout: delete_pools_parallel: process_1_[94381]: starting. 2024-04-14T15:13:00.135 INFO:tasks.workunit.client.0.smithi090.stdout: delete_pools_parallel: process_1_[94381]: creating pool ceph_test_rados_delete_pools_parallel.smithi090-94244 2024-04-14T15:13:00.135 INFO:tasks.workunit.client.0.smithi090.stdout: delete_pools_parallel: process_1_[94381]: created object 0... 2024-04-14T15:13:00.135 INFO:tasks.workunit.client.0.smithi090.stdout: delete_pools_parallel: process_1_[94381]: created object 25... 2024-04-14T15:13:00.135 INFO:tasks.workunit.client.0.smithi090.stdout: delete_pools_parallel: process_1_[94381]: created object 49... 2024-04-14T15:13:00.135 INFO:tasks.workunit.client.0.smithi090.stdout: delete_pools_parallel: process_1_[94381]: finishing. 2024-04-14T15:13:00.135 INFO:tasks.workunit.client.0.smithi090.stdout: delete_pools_parallel: process_1_[94381]: shutting down. 2024-04-14T15:13:00.135 INFO:tasks.workunit.client.0.smithi090.stdout: delete_pools_parallel: process_2_[94385]: starting. 2024-04-14T15:13:00.135 INFO:tasks.workunit.client.0.smithi090.stdout: delete_pools_parallel: process_2_[94385]: deleting pool ceph_test_rados_delete_pools_parallel.smithi090-94244 2024-04-14T15:13:00.136 INFO:tasks.workunit.client.0.smithi090.stdout: delete_pools_parallel: process_2_[94385]: shutting down. 2024-04-14T15:13:00.136 INFO:tasks.workunit.client.0.smithi090.stdout: delete_pools_parallel: ******************************* 2024-04-14T15:13:00.136 INFO:tasks.workunit.client.0.smithi090.stdout: delete_pools_parallel: process_3_[95501]: starting. 2024-04-14T15:13:00.136 INFO:tasks.workunit.client.0.smithi090.stdout: delete_pools_parallel: process_3_[95501]: creating pool ceph_test_rados_delete_pools_parallel.smithi090-94244 2024-04-14T15:13:00.136 INFO:tasks.workunit.client.0.smithi090.stdout: delete_pools_parallel: process_3_[95501]: created object 0... 2024-04-14T15:13:00.136 INFO:tasks.workunit.client.0.smithi090.stdout: delete_pools_parallel: process_3_[95501]: created object 25... 2024-04-14T15:13:00.136 INFO:tasks.workunit.client.0.smithi090.stdout: delete_pools_parallel: process_3_[95501]: created object 49... 2024-04-14T15:13:00.136 INFO:tasks.workunit.client.0.smithi090.stdout: delete_pools_parallel: process_3_[95501]: finishing. 2024-04-14T15:13:00.136 INFO:tasks.workunit.client.0.smithi090.stdout: delete_pools_parallel: process_3_[95501]: shutting down. 2024-04-14T15:13:00.136 INFO:tasks.workunit.client.0.smithi090.stdout: delete_pools_parallel: ******************************* 2024-04-14T15:13:00.136 INFO:tasks.workunit.client.0.smithi090.stdout: delete_pools_parallel: process_5_[95504]: starting. 2024-04-14T15:13:00.136 INFO:tasks.workunit.client.0.smithi090.stdout: delete_pools_parallel: process_5_[95504]: listing objects. 2024-04-14T15:13:00.136 INFO:tasks.workunit.client.0.smithi090.stdout: delete_pools_parallel: process_5_[95504]: listed object 0... 2024-04-14T15:13:00.136 INFO:tasks.workunit.client.0.smithi090.stdout: delete_pools_parallel: process_5_[95504]: listed object 25... 2024-04-14T15:13:00.137 INFO:tasks.workunit.client.0.smithi090.stdout: delete_pools_parallel: process_5_[95504]: saw 50 objects 2024-04-14T15:13:00.137 INFO:tasks.workunit.client.0.smithi090.stdout: delete_pools_parallel: process_5_[95504]: shutting down. 2024-04-14T15:13:00.137 INFO:tasks.workunit.client.0.smithi090.stdout: delete_pools_parallel: ******************************* 2024-04-14T15:13:00.137 INFO:tasks.workunit.client.0.smithi090.stdout: delete_pools_parallel: process_4_[95502]: starting. 2024-04-14T15:13:00.137 INFO:tasks.workunit.client.0.smithi090.stdout: delete_pools_parallel: process_4_[95502]: deleting pool ceph_test_rados_delete_pools_parallel.smithi090-94244 2024-04-14T15:13:00.137 INFO:tasks.workunit.client.0.smithi090.stdout: delete_pools_parallel: process_4_[95502]: shutting down. 2024-04-14T15:13:00.137 INFO:tasks.workunit.client.0.smithi090.stdout: delete_pools_parallel: ******************************* 2024-04-14T15:13:00.137 INFO:tasks.workunit.client.0.smithi090.stdout: delete_pools_parallel: ******************************* 2024-04-14T15:13:00.137 INFO:tasks.workunit.client.0.smithi090.stdout: delete_pools_parallel: ******* SUCCESS ********** 2024-04-14T15:13:00.211 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:13:00.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:00.245 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:00.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:13:00.346 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:13:00.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:00.371 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:00.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:13:00.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[30399]: pgmap v112: 948 pgs: 416 creating+peering, 532 active+clean; 25 MiB data, 282 MiB used, 715 GiB / 715 GiB avail; 9.7 KiB/s rd, 6.0 MiB/s wr, 562 op/s 2024-04-14T15:13:00.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/274362209' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi090-93436-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:13:00.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1713883211' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi090-93476-10"}]: dispatch 2024-04-14T15:13:00.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[30399]: from='client.25217 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi090-93476-10"}]: dispatch 2024-04-14T15:13:00.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[30399]: from='client.25252 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-04-14T15:13:00.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1713883211' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi090-93476-10"}]: dispatch 2024-04-14T15:13:00.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[30399]: from='client.25217 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi090-93476-10"}]: dispatch 2024-04-14T15:13:00.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1713883211' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi090-93476-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:13:00.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[30399]: from='client.25217 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi090-93476-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:13:00.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:00.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2051960200' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-04-14T15:13:00.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.0"}]: dispatch 2024-04-14T15:13:00.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[30399]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.0"}]: dispatch 2024-04-14T15:13:00.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.1"}]: dispatch 2024-04-14T15:13:00.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[30399]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.1"}]: dispatch 2024-04-14T15:13:00.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.2"}]: dispatch 2024-04-14T15:13:00.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[30399]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.2"}]: dispatch 2024-04-14T15:13:00.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.3"}]: dispatch 2024-04-14T15:13:00.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[30399]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.3"}]: dispatch 2024-04-14T15:13:00.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.4"}]: dispatch 2024-04-14T15:13:00.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[30399]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.4"}]: dispatch 2024-04-14T15:13:00.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.5"}]: dispatch 2024-04-14T15:13:00.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[41670]: pgmap v112: 948 pgs: 416 creating+peering, 532 active+clean; 25 MiB data, 282 MiB used, 715 GiB / 715 GiB avail; 9.7 KiB/s rd, 6.0 MiB/s wr, 562 op/s 2024-04-14T15:13:00.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/274362209' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi090-93436-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:13:00.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1713883211' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi090-93476-10"}]: dispatch 2024-04-14T15:13:00.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[41670]: from='client.25217 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi090-93476-10"}]: dispatch 2024-04-14T15:13:00.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[41670]: from='client.25252 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-04-14T15:13:00.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1713883211' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi090-93476-10"}]: dispatch 2024-04-14T15:13:00.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[41670]: from='client.25217 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi090-93476-10"}]: dispatch 2024-04-14T15:13:00.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1713883211' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi090-93476-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:13:00.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[41670]: from='client.25217 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi090-93476-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:13:00.445 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:00.445 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2051960200' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-04-14T15:13:00.445 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.0"}]: dispatch 2024-04-14T15:13:00.445 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[41670]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.0"}]: dispatch 2024-04-14T15:13:00.445 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.1"}]: dispatch 2024-04-14T15:13:00.445 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[41670]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.1"}]: dispatch 2024-04-14T15:13:00.445 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.2"}]: dispatch 2024-04-14T15:13:00.445 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[41670]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.2"}]: dispatch 2024-04-14T15:13:00.445 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.3"}]: dispatch 2024-04-14T15:13:00.445 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[41670]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.3"}]: dispatch 2024-04-14T15:13:00.445 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.4"}]: dispatch 2024-04-14T15:13:00.445 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[41670]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.4"}]: dispatch 2024-04-14T15:13:00.445 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.5"}]: dispatch 2024-04-14T15:13:00.446 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[30399]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.5"}]: dispatch 2024-04-14T15:13:00.446 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.6"}]: dispatch 2024-04-14T15:13:00.446 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[30399]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.6"}]: dispatch 2024-04-14T15:13:00.446 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.7"}]: dispatch 2024-04-14T15:13:00.446 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[30399]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.7"}]: dispatch 2024-04-14T15:13:00.446 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.8"}]: dispatch 2024-04-14T15:13:00.446 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[30399]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.8"}]: dispatch 2024-04-14T15:13:00.446 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.9"}]: dispatch 2024-04-14T15:13:00.447 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[30399]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.9"}]: dispatch 2024-04-14T15:13:00.447 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[30399]: from='client.25240 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi090-93379-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi090-93379-16"}]': finished 2024-04-14T15:13:00.447 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[30399]: from='client.25243 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi090-93398-23", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi090-93398-23"}]': finished 2024-04-14T15:13:00.447 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[30399]: from='client.25267 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi090-93864-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi090-93864-7"}]': finished 2024-04-14T15:13:00.447 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[30399]: from='client.25249 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi090-93731-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi090-93731-7"}]': finished 2024-04-14T15:13:00.447 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[30399]: from='client.25021 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi090-94522-1"}]': finished 2024-04-14T15:13:00.447 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[30399]: from='client.25001 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi090-94662-1"}]': finished 2024-04-14T15:13:00.447 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/309942895' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi090-93998-1"}]': finished 2024-04-14T15:13:00.447 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[30399]: from='client.25114 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:00.447 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/274362209' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi090-93436-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:13:00.447 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[30399]: from='client.25217 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi090-93476-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:13:00.448 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[41670]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.5"}]: dispatch 2024-04-14T15:13:00.448 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.6"}]: dispatch 2024-04-14T15:13:00.448 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[41670]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.6"}]: dispatch 2024-04-14T15:13:00.448 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.7"}]: dispatch 2024-04-14T15:13:00.448 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[41670]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.7"}]: dispatch 2024-04-14T15:13:00.448 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.8"}]: dispatch 2024-04-14T15:13:00.448 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[41670]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.8"}]: dispatch 2024-04-14T15:13:00.448 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.9"}]: dispatch 2024-04-14T15:13:00.448 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[41670]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.9"}]: dispatch 2024-04-14T15:13:00.448 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[41670]: from='client.25240 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi090-93379-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi090-93379-16"}]': finished 2024-04-14T15:13:00.449 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[41670]: from='client.25243 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi090-93398-23", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi090-93398-23"}]': finished 2024-04-14T15:13:00.449 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[41670]: from='client.25267 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi090-93864-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi090-93864-7"}]': finished 2024-04-14T15:13:00.449 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[41670]: from='client.25249 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi090-93731-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi090-93731-7"}]': finished 2024-04-14T15:13:00.449 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[41670]: from='client.25021 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi090-94522-1"}]': finished 2024-04-14T15:13:00.449 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[41670]: from='client.25001 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi090-94662-1"}]': finished 2024-04-14T15:13:00.450 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/309942895' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi090-93998-1"}]': finished 2024-04-14T15:13:00.450 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[41670]: from='client.25114 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:00.450 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/274362209' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi090-93436-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:13:00.450 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[41670]: from='client.25217 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi090-93476-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:13:00.450 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[30399]: osdmap e112: 8 total, 8 up, 8 in 2024-04-14T15:13:00.450 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/309942895' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi090-93998-1"}]: dispatch 2024-04-14T15:13:00.450 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1713883211' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi090-93476-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi090-93476-10"}]: dispatch 2024-04-14T15:13:00.451 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1465058495' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi090-94662-1"}]: dispatch 2024-04-14T15:13:00.451 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/274362209' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi090-93436-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi090-93436-10"}]: dispatch 2024-04-14T15:13:00.451 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[30399]: from='client.25217 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi090-93476-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi090-93476-10"}]: dispatch 2024-04-14T15:13:00.451 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[30399]: from='client.25001 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi090-94662-1"}]: dispatch 2024-04-14T15:13:00.451 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/532832389' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi090-93363-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:00.451 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1327720256' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-4", "tierpool": "test-rados-api-smithi090-93652-4-cache"}]: dispatch 2024-04-14T15:13:00.451 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[30399]: from='client.25207 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi090-93363-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:00.451 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[30399]: from='client.25114 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-4", "tierpool": "test-rados-api-smithi090-93652-4-cache"}]: dispatch 2024-04-14T15:13:00.451 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/340734729' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi090-94522-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:13:00.451 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[30399]: from='client.25489 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi090-94522-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:13:00.452 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[41670]: osdmap e112: 8 total, 8 up, 8 in 2024-04-14T15:13:00.452 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/309942895' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi090-93998-1"}]: dispatch 2024-04-14T15:13:00.452 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1713883211' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi090-93476-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi090-93476-10"}]: dispatch 2024-04-14T15:13:00.452 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1465058495' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi090-94662-1"}]: dispatch 2024-04-14T15:13:00.452 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/274362209' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi090-93436-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi090-93436-10"}]: dispatch 2024-04-14T15:13:00.452 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[41670]: from='client.25217 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi090-93476-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi090-93476-10"}]: dispatch 2024-04-14T15:13:00.452 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[41670]: from='client.25001 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi090-94662-1"}]: dispatch 2024-04-14T15:13:00.452 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/532832389' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi090-93363-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:00.452 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1327720256' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-4", "tierpool": "test-rados-api-smithi090-93652-4-cache"}]: dispatch 2024-04-14T15:13:00.452 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[41670]: from='client.25207 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi090-93363-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:00.452 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[41670]: from='client.25114 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-4", "tierpool": "test-rados-api-smithi090-93652-4-cache"}]: dispatch 2024-04-14T15:13:00.453 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/340734729' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi090-94522-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:13:00.453 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:00 smithi090 ceph-mon[41670]: from='client.25489 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi090-94522-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:13:00.472 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:13:00.496 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:00.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:00.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:13:00.598 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:13:00.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:00.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:00 smithi092 ceph-mon[33966]: pgmap v112: 948 pgs: 416 creating+peering, 532 active+clean; 25 MiB data, 282 MiB used, 715 GiB / 715 GiB avail; 9.7 KiB/s rd, 6.0 MiB/s wr, 562 op/s 2024-04-14T15:13:00.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:00 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/274362209' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi090-93436-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:13:00.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:00 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1713883211' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi090-93476-10"}]: dispatch 2024-04-14T15:13:00.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:00 smithi092 ceph-mon[33966]: from='client.25217 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi090-93476-10"}]: dispatch 2024-04-14T15:13:00.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:00 smithi092 ceph-mon[33966]: from='client.25252 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-04-14T15:13:00.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:00 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1713883211' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi090-93476-10"}]: dispatch 2024-04-14T15:13:00.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:00 smithi092 ceph-mon[33966]: from='client.25217 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi090-93476-10"}]: dispatch 2024-04-14T15:13:00.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:00 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1713883211' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi090-93476-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:13:00.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:00 smithi092 ceph-mon[33966]: from='client.25217 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi090-93476-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:13:00.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:00 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:00.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:00 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2051960200' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-04-14T15:13:00.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:00 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.0"}]: dispatch 2024-04-14T15:13:00.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:00 smithi092 ceph-mon[33966]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.0"}]: dispatch 2024-04-14T15:13:00.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:00 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.1"}]: dispatch 2024-04-14T15:13:00.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:00 smithi092 ceph-mon[33966]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.1"}]: dispatch 2024-04-14T15:13:00.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:00 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.2"}]: dispatch 2024-04-14T15:13:00.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:00 smithi092 ceph-mon[33966]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.2"}]: dispatch 2024-04-14T15:13:00.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:00 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.3"}]: dispatch 2024-04-14T15:13:00.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:00 smithi092 ceph-mon[33966]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.3"}]: dispatch 2024-04-14T15:13:00.625 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:00 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.4"}]: dispatch 2024-04-14T15:13:00.625 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:00 smithi092 ceph-mon[33966]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.4"}]: dispatch 2024-04-14T15:13:00.625 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:00 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.5"}]: dispatch 2024-04-14T15:13:00.625 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:00 smithi092 ceph-mon[33966]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.5"}]: dispatch 2024-04-14T15:13:00.625 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:00 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.6"}]: dispatch 2024-04-14T15:13:00.625 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:00 smithi092 ceph-mon[33966]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.6"}]: dispatch 2024-04-14T15:13:00.625 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:00 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.7"}]: dispatch 2024-04-14T15:13:00.625 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:00 smithi092 ceph-mon[33966]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.7"}]: dispatch 2024-04-14T15:13:00.625 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:00 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.8"}]: dispatch 2024-04-14T15:13:00.625 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:00 smithi092 ceph-mon[33966]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.8"}]: dispatch 2024-04-14T15:13:00.625 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:00 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.9"}]: dispatch 2024-04-14T15:13:00.625 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:00 smithi092 ceph-mon[33966]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.9"}]: dispatch 2024-04-14T15:13:00.625 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:00 smithi092 ceph-mon[33966]: from='client.25240 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi090-93379-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi090-93379-16"}]': finished 2024-04-14T15:13:00.625 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:00 smithi092 ceph-mon[33966]: from='client.25243 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi090-93398-23", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi090-93398-23"}]': finished 2024-04-14T15:13:00.625 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:00 smithi092 ceph-mon[33966]: from='client.25267 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi090-93864-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi090-93864-7"}]': finished 2024-04-14T15:13:00.626 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:00 smithi092 ceph-mon[33966]: from='client.25249 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi090-93731-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi090-93731-7"}]': finished 2024-04-14T15:13:00.626 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:00 smithi092 ceph-mon[33966]: from='client.25021 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi090-94522-1"}]': finished 2024-04-14T15:13:00.626 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:00 smithi092 ceph-mon[33966]: from='client.25001 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi090-94662-1"}]': finished 2024-04-14T15:13:00.626 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:00 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/309942895' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi090-93998-1"}]': finished 2024-04-14T15:13:00.626 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:00 smithi092 ceph-mon[33966]: from='client.25114 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:00.626 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:00 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/274362209' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi090-93436-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:13:00.626 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:00 smithi092 ceph-mon[33966]: from='client.25217 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi090-93476-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:13:00.626 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:00 smithi092 ceph-mon[33966]: osdmap e112: 8 total, 8 up, 8 in 2024-04-14T15:13:00.626 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:00 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/309942895' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi090-93998-1"}]: dispatch 2024-04-14T15:13:00.626 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:00 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1713883211' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi090-93476-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi090-93476-10"}]: dispatch 2024-04-14T15:13:00.626 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:00 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1465058495' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi090-94662-1"}]: dispatch 2024-04-14T15:13:00.626 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:00 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/274362209' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi090-93436-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi090-93436-10"}]: dispatch 2024-04-14T15:13:00.626 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:00 smithi092 ceph-mon[33966]: from='client.25217 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi090-93476-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi090-93476-10"}]: dispatch 2024-04-14T15:13:00.627 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:00 smithi092 ceph-mon[33966]: from='client.25001 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi090-94662-1"}]: dispatch 2024-04-14T15:13:00.627 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:00 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/532832389' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi090-93363-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:00.627 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:00 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1327720256' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-4", "tierpool": "test-rados-api-smithi090-93652-4-cache"}]: dispatch 2024-04-14T15:13:00.627 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:00 smithi092 ceph-mon[33966]: from='client.25207 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi090-93363-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:00.627 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:00 smithi092 ceph-mon[33966]: from='client.25114 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-4", "tierpool": "test-rados-api-smithi090-93652-4-cache"}]: dispatch 2024-04-14T15:13:00.627 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:00 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/340734729' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi090-94522-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:13:00.627 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:00 smithi092 ceph-mon[33966]: from='client.25489 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi090-94522-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:13:00.627 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:00.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:13:00.729 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:13:00.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:00.755 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:00.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:13:00.824 INFO:tasks.workunit.client.0.smithi090.stdout: cmd: Running main() from gmock_main.cc 2024-04-14T15:13:00.825 INFO:tasks.workunit.client.0.smithi090.stdout: cmd: [==========] Running 3 tests from 1 test suite. 2024-04-14T15:13:00.825 INFO:tasks.workunit.client.0.smithi090.stdout: cmd: [----------] Global test environment set-up. 2024-04-14T15:13:00.825 INFO:tasks.workunit.client.0.smithi090.stdout: cmd: [----------] 3 tests from NeoRadosCmd 2024-04-14T15:13:00.825 INFO:tasks.workunit.client.0.smithi090.stdout: cmd: [ RUN ] NeoRadosCmd.MonDescribe 2024-04-14T15:13:00.825 INFO:tasks.workunit.client.0.smithi090.stdout: cmd: [ OK ] NeoRadosCmd.MonDescribe (1574 ms) 2024-04-14T15:13:00.825 INFO:tasks.workunit.client.0.smithi090.stdout: cmd: [ RUN ] NeoRadosCmd.OSDCmd 2024-04-14T15:13:00.825 INFO:tasks.workunit.client.0.smithi090.stdout: cmd: [ OK ] NeoRadosCmd.OSDCmd (1977 ms) 2024-04-14T15:13:00.825 INFO:tasks.workunit.client.0.smithi090.stdout: cmd: [ RUN ] NeoRadosCmd.PGCmd 2024-04-14T15:13:00.825 INFO:tasks.workunit.client.0.smithi090.stdout: cmd: [ OK ] NeoRadosCmd.PGCmd (2739 ms) 2024-04-14T15:13:00.825 INFO:tasks.workunit.client.0.smithi090.stdout: cmd: [----------] 3 tests from NeoRadosCmd (6290 ms total) 2024-04-14T15:13:00.825 INFO:tasks.workunit.client.0.smithi090.stdout: cmd: 2024-04-14T15:13:00.825 INFO:tasks.workunit.client.0.smithi090.stdout: cmd: [----------] Global test environment tear-down 2024-04-14T15:13:00.825 INFO:tasks.workunit.client.0.smithi090.stdout: cmd: [==========] 3 tests from 1 test suite ran. (6290 ms total) 2024-04-14T15:13:00.825 INFO:tasks.workunit.client.0.smithi090.stdout: cmd: [ PASSED ] 3 tests. 2024-04-14T15:13:00.855 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:13:00.863 INFO:tasks.thrashosds.thrasher:j is {'epoch': 108, 'fsid': '0e288098-fa70-11ee-bc8e-c7b262605968', 'created': '2024-04-14T15:04:05.808313+0000', 'modified': '2024-04-14T15:12:56.046703+0000', 'last_up_change': '2024-04-14T15:09:29.769352+0000', 'last_in_change': '2024-04-14T15:09:11.803164+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 19, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 41, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-14T15:07:23.959628+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '104', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9022312164306641, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-04-14T15:09:41.608420+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9022312164306641, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 3, 'pool_name': 'datapool', 'create_time': '2024-04-14T15:09:41.744391+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 69, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 0.9022312164306641, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-04-14T15:09:43.088157+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9022312164306641, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-04-14T15:09:45.042722+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9022312164306641, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-04-14T15:09:47.111336+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.9022312164306641, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 7, 'pool_name': 'TooBig_smithi090-93342-1', 'create_time': '2024-04-14T15:12:54.194258+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.9022312164306641, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 8, 'pool_name': 'LibRadosIo_smithi090-93379-1', 'create_time': '2024-04-14T15:12:54.203865+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.9022312164306641, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 9, 'pool_name': 'TooBigPP_smithi090-93363-1', 'create_time': '2024-04-14T15:12:54.211449+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9022312164306641, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 10, 'pool_name': 'LibRadosIoPP_smithi090-93398-1', 'create_time': '2024-04-14T15:12:54.219277+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 0.9022312164306641, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 11, 'pool_name': 'ceph_test_rados_api_asio', 'create_time': '2024-04-14T15:12:54.225671+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.9022312164306641, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 12, 'pool_name': 'LibRadosLock_smithi090-93436-1', 'create_time': '2024-04-14T15:12:54.234106+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9022312164306641, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 13, 'pool_name': 'LibRadosList_smithi090-93444-1', 'create_time': '2024-04-14T15:12:54.244611+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.9022312164306641, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 14, 'pool_name': 'LibRadosLockPP_smithi090-93476-1', 'create_time': '2024-04-14T15:12:54.263258+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.75, 'score_stable': 2.75, 'optimal_score': 1, 'raw_score_acting': 2.75, 'raw_score_stable': 2.75, 'primary_affinity_weighted': 0.9022312164306641, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 15, 'pool_name': 'test-rados-api-smithi090-93656-1', 'create_time': '2024-04-14T15:12:54.290883+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9022312164306641, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 16, 'pool_name': 'LibRadosMiscPP_smithi090-93601-1', 'create_time': '2024-04-14T15:12:54.300251+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9022312164306641, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 17, 'pool_name': 'LibRadosSnapshotsPP_smithi090-93724-1', 'create_time': '2024-04-14T15:12:54.309050+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9022312164306641, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 18, 'pool_name': 'LibRadosSnapshots_smithi090-93709-1', 'create_time': '2024-04-14T15:12:54.322168+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.9022312164306641, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 19, 'pool_name': 'LibRadosStat_smithi090-93731-1', 'create_time': '2024-04-14T15:12:54.329704+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.9022312164306641, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 20, 'pool_name': 'LibRadosTierPP_smithi090-93652-1', 'create_time': '2024-04-14T15:12:54.336842+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9022312164306641, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 21, 'pool_name': 'LibRadosStatPP_smithi090-93864-1', 'create_time': '2024-04-14T15:12:54.363752+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.9022312164306641, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 22, 'pool_name': 'LibRadosWatchNotify_smithi090-93930-1', 'create_time': '2024-04-14T15:12:54.373375+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9022312164306641, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 23, 'pool_name': 'test-rados-api-smithi090-94066-1', 'create_time': '2024-04-14T15:12:54.442611+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9022312164306641, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 24, 'pool_name': 'ceph_test_rados_list_parallel.smithi090-94111', 'create_time': '2024-04-14T15:12:54.454594+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9022312164306641, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 25, 'pool_name': 'CReadOpsTest_smithi090-94131-1', 'create_time': '2024-04-14T15:12:54.477230+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9022312164306641, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 26, 'pool_name': 'test-rados-api-smithi090-94114-1', 'create_time': '2024-04-14T15:12:54.488445+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.9022312164306641, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 27, 'pool_name': 'DNEsmithi090-94274-1', 'create_time': '2024-04-14T15:12:54.514999+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.9022312164306641, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 28, 'pool_name': 'ceph_test_rados_delete_pools_parallel.smithi090-94244', 'create_time': '2024-04-14T15:12:54.526462+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9022312164306641, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 29, 'pool_name': 'test-rados-api-smithi090-93995-1', 'create_time': '2024-04-14T15:12:54.535126+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.75, 'score_stable': 2.75, 'optimal_score': 1, 'raw_score_acting': 2.75, 'raw_score_stable': 2.75, 'primary_affinity_weighted': 0.9022312164306641, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 30, 'pool_name': 'ceph_test_rados_open_pools_parallel.smithi090-94230', 'create_time': '2024-04-14T15:12:54.542481+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9022312164306641, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 31, 'pool_name': 'test-rados-api-smithi090-94011-1', 'create_time': '2024-04-14T15:12:54.549723+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.9022312164306641, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 32, 'pool_name': 'Limitssmithi090-94451-1', 'create_time': '2024-04-14T15:12:54.558188+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.9022312164306641, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 34, 'pool_name': 'test_handler_errorsmithi090-94532-1', 'create_time': '2024-04-14T15:12:54.581004+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.9022312164306641, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 35, 'pool_name': 'ListObjectssmithi090-94600-1', 'create_time': '2024-04-14T15:12:54.588799+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 0.9022312164306641, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 38, 'pool_name': 'SetOpFlagssmithi090-94703-1', 'create_time': '2024-04-14T15:12:54.615131+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9022312164306641, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 39, 'pool_name': 'SnapListsmithi090-94770-1', 'create_time': '2024-04-14T15:12:54.631880+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9022312164306641, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 40, 'pool_name': 'WatchNotifysmithi090-94850-1', 'create_time': '2024-04-14T15:12:54.635366+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9022312164306641, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 41, 'pool_name': 'AssertExistssmithi090-94846-1', 'create_time': '2024-04-14T15:12:54.638413+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9022312164306641, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}], 'osds': [{'osd': 0, 'uuid': '76772cfc-89d1-4013-b617-32284ed8dfd9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 107, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6801', 'nonce': 3761806885}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6802', 'nonce': 3761806885}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6804', 'nonce': 3761806885}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6803', 'nonce': 3761806885}]}, 'public_addr': '172.21.15.90:6801/3761806885', 'cluster_addr': '172.21.15.90:6802/3761806885', 'heartbeat_back_addr': '172.21.15.90:6804/3761806885', 'heartbeat_front_addr': '172.21.15.90:6803/3761806885', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'de4a7fa2-7d57-4ff3-a4fa-28c0cde8ef6c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 107, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6805', 'nonce': 4170361699}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6806', 'nonce': 4170361699}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6808', 'nonce': 4170361699}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6807', 'nonce': 4170361699}]}, 'public_addr': '172.21.15.90:6805/4170361699', 'cluster_addr': '172.21.15.90:6806/4170361699', 'heartbeat_back_addr': '172.21.15.90:6808/4170361699', 'heartbeat_front_addr': '172.21.15.90:6807/4170361699', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '9c39b5ff-6175-46ba-9d71-370581c6ff3d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 107, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6809', 'nonce': 1686184251}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6810', 'nonce': 1686184251}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6812', 'nonce': 1686184251}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6811', 'nonce': 1686184251}]}, 'public_addr': '172.21.15.90:6809/1686184251', 'cluster_addr': '172.21.15.90:6810/1686184251', 'heartbeat_back_addr': '172.21.15.90:6812/1686184251', 'heartbeat_front_addr': '172.21.15.90:6811/1686184251', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'efd2ca45-5b80-4013-a571-499e2cb5b2e8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.2178497314453125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 107, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6813', 'nonce': 2270166774}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6814', 'nonce': 2270166774}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6816', 'nonce': 2270166774}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6815', 'nonce': 2270166774}]}, 'public_addr': '172.21.15.90:6813/2270166774', 'cluster_addr': '172.21.15.90:6814/2270166774', 'heartbeat_back_addr': '172.21.15.90:6816/2270166774', 'heartbeat_front_addr': '172.21.15.90:6815/2270166774', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'e3d9013d-5eb6-433c-96b3-3fc38ba7ac8d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 107, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6800', 'nonce': 3887549733}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6801', 'nonce': 3887549733}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6803', 'nonce': 3887549733}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6802', 'nonce': 3887549733}]}, 'public_addr': '172.21.15.92:6800/3887549733', 'cluster_addr': '172.21.15.92:6801/3887549733', 'heartbeat_back_addr': '172.21.15.92:6803/3887549733', 'heartbeat_front_addr': '172.21.15.92:6802/3887549733', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '1ec41805-6f82-4139-81bc-75305ba9e958', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 41, 'up_thru': 107, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6804', 'nonce': 3244561440}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6805', 'nonce': 3244561440}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6807', 'nonce': 3244561440}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6806', 'nonce': 3244561440}]}, 'public_addr': '172.21.15.92:6804/3244561440', 'cluster_addr': '172.21.15.92:6805/3244561440', 'heartbeat_back_addr': '172.21.15.92:6807/3244561440', 'heartbeat_front_addr': '172.21.15.92:6806/3244561440', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'ca977ab2-bf8d-4b3a-ae5e-0efa5bc7a3ad', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 49, 'up_thru': 107, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6808', 'nonce': 636125963}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6809', 'nonce': 636125963}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6811', 'nonce': 636125963}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6810', 'nonce': 636125963}]}, 'public_addr': '172.21.15.92:6808/636125963', 'cluster_addr': '172.21.15.92:6809/636125963', 'heartbeat_back_addr': '172.21.15.92:6811/636125963', 'heartbeat_front_addr': '172.21.15.92:6810/636125963', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'e5354645-79ec-4d4c-b6e5-f9b0653915ee', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 58, 'up_thru': 107, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6812', 'nonce': 1773706716}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6813', 'nonce': 1773706716}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6815', 'nonce': 1773706716}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6814', 'nonce': 1773706716}]}, 'public_addr': '172.21.15.92:6812/1773706716', 'cluster_addr': '172.21.15.92:6813/1773706716', 'heartbeat_back_addr': '172.21.15.92:6815/1773706716', 'heartbeat_front_addr': '172.21.15.92:6814/1773706716', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-14T15:11:46.086720+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-14T15:12:44.215601+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-14T15:12:10.327706+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-14T15:11:53.872813+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-14T15:11:41.834590+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-14T15:12:27.778595+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-14T15:12:41.863374+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-14T15:11:41.238050+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '2.1c', 'mappings': [{'from': 7, 'to': 4}]}, {'pgid': '4.2', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.d', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.1f', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.90:0/508768479': '2024-04-15T15:10:22.918674+0000', '172.21.15.90:0/1823072952': '2024-04-15T15:10:22.918674+0000', '172.21.15.90:6800/491071118': '2024-04-15T15:10:22.918674+0000', '172.21.15.90:0/1337064035': '2024-04-15T15:10:22.918674+0000', '172.21.15.90:0/769462306': '2024-04-15T15:10:22.918674+0000', '172.21.15.90:0/1139868610': '2024-04-15T15:05:05.882361+0000', '172.21.15.90:0/93763408': '2024-04-15T15:10:22.918674+0000', '172.21.15.90:0/977818204': '2024-04-15T15:04:39.251711+0000', '172.21.15.90:6800/941091171': '2024-04-15T15:05:05.882361+0000', '172.21.15.90:0/3671453561': '2024-04-15T15:05:05.882361+0000', '172.21.15.90:0/2611856200': '2024-04-15T15:05:05.882361+0000', '172.21.15.90:6800/2306258962': '2024-04-15T15:04:39.251711+0000', '172.21.15.90:0/4051692380': '2024-04-15T15:04:39.251711+0000', '172.21.15.90:0/2681502561': '2024-04-15T15:04:39.251711+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}, 'testprofile-LibRadosWatchNotifyECPP_smithi090-93998-1': {'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-num-failure-domains': '0', 'crush-osds-per-failure-domain': '0', 'crush-root': 'default', 'jerasure-per-chunk-alignment': 'false', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van', 'w': '8'}, 'testprofile-ListObjectssmithi090-94662-1': {'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-num-failure-domains': '0', 'crush-osds-per-failure-domain': '0', 'crush-root': 'default', 'jerasure-per-chunk-alignment': 'false', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van', 'w': '8'}, 'testprofile-SimpleWritesmithi090-94522-1': {'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-num-failure-domains': '0', 'crush-osds-per-failure-domain': '0', 'crush-root': 'default', 'jerasure-per-chunk-alignment': 'false', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van', 'w': '8'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-14T15:13:00.864 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 4.d 2024-04-14T15:13:00.864 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd rm-pg-upmap-items 4.d 2024-04-14T15:13:00.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:00.897 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:00.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:13:00.985 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: Running main() from gmock_main.cc 2024-04-14T15:13:00.986 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [==========] Running 38 tests from 2 test suites. 2024-04-14T15:13:00.986 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [----------] Global test environment set-up. 2024-04-14T15:13:00.986 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [----------] 21 tests from LibRadosIoPP 2024-04-14T15:13:00.986 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: seed 93398 2024-04-14T15:13:00.986 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2024-04-14T15:13:00.986 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2024-04-14T15:13:00.986 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2024-04-14T15:13:00.986 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (481 ms) 2024-04-14T15:13:00.986 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2024-04-14T15:13:00.986 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (7 ms) 2024-04-14T15:13:00.986 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2024-04-14T15:13:00.986 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (42 ms) 2024-04-14T15:13:00.987 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadExtentArrayOpPP 2024-04-14T15:13:00.987 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadExtentArrayOpPP (381 ms) 2024-04-14T15:13:00.987 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2024-04-14T15:13:00.987 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (4 ms) 2024-04-14T15:13:00.987 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2024-04-14T15:13:00.987 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (2 ms) 2024-04-14T15:13:00.987 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2024-04-14T15:13:00.987 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (2 ms) 2024-04-14T15:13:00.987 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2024-04-14T15:13:00.987 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (4 ms) 2024-04-14T15:13:00.987 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2024-04-14T15:13:00.987 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (4 ms) 2024-04-14T15:13:00.987 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2024-04-14T15:13:00.987 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (4 ms) 2024-04-14T15:13:00.987 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2024-04-14T15:13:00.988 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (4 ms) 2024-04-14T15:13:00.988 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2024-04-14T15:13:00.988 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (5 ms) 2024-04-14T15:13:00.988 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2024-04-14T15:13:00.988 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (3 ms) 2024-04-14T15:13:00.988 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2024-04-14T15:13:00.988 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (3 ms) 2024-04-14T15:13:00.988 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2024-04-14T15:13:00.988 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (3 ms) 2024-04-14T15:13:00.988 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2024-04-14T15:13:00.988 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (10 ms) 2024-04-14T15:13:00.988 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-04-14T15:13:00.988 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (5 ms) 2024-04-14T15:13:00.989 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2024-04-14T15:13:00.989 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (8 ms) 2024-04-14T15:13:00.989 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2024-04-14T15:13:00.989 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (3 ms) 2024-04-14T15:13:00.989 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2024-04-14T15:13:00.989 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (6 ms) 2024-04-14T15:13:00.989 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [----------] 21 tests from LibRadosIoPP (981 ms total) 2024-04-14T15:13:00.989 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: 2024-04-14T15:13:00.989 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP 2024-04-14T15:13:00.989 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2024-04-14T15:13:00.989 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (847 ms) 2024-04-14T15:13:00.989 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2024-04-14T15:13:00.989 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (24 ms) 2024-04-14T15:13:00.989 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2024-04-14T15:13:00.990 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (7 ms) 2024-04-14T15:13:00.990 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2024-04-14T15:13:00.990 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (2 ms) 2024-04-14T15:13:00.998 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:13:01.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:01.028 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:01.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:13:01.129 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:13:01.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:01.154 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:01.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:13:01.160 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:13:01.225 INFO:tasks.workunit.client.0.smithi090.stdout: n_it": true}]: dispatch 2024-04-14T15:13:01.225 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:58.093242+0000 mon.a [INF] from='client.25258 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:01.225 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:58.093848+0000 mon.a [INF] from='client.25207 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi090-93363-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:01.225 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:58.094246+0000 mon.a [INF] from='client.25246 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:01.226 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:59.200778+0000 mon.a [INF] from='client.? v1:172.21.15.90:0/274362209' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi090-93436-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:13:01.226 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:59.203364+0000 mon.c [INF] from='client.? v1:172.21.15.90:0/1713883211' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi090-93476-10"}]: dispatch 2024-04-14T15:13:01.226 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:59.211415+0000 mon.a [INF] from='client.25217 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi090-93476-10"}]: dispatch 2024-04-14T15:13:01.226 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:59.214357+0000 mon.a [INF] from='client.25252 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-04-14T15:13:01.226 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:59.214640+0000 mon.c [INF] from='client.? v1:172.21.15.90:0/1713883211' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi090-93476-10"}]: dispatch 2024-04-14T15:13:01.226 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:59.236234+0000 mon.a [INF] from='client.25217 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi090-93476-10"}]: dispatch 2024-04-14T15:13:01.226 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:59.237713+0000 mon.c [INF] from='client.? v1:172.21.15.90:0/1713883211' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi090-93476-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:13:01.226 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:59.239875+0000 mon.a [INF] from='client.25217 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi090-93476-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:13:01.226 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:59.653567+0000 mon.b [INF] from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.0"}]: dispatch 2024-04-14T15:13:01.226 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:59.654213+0000 mon.b [INF] from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.1"}]: dispatch 2024-04-14T15:13:01.226 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:59.654616+0000 mon.b [INF] from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.2"}]: dispatch 2024-04-14T15:13:01.226 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:59.655045+0000 mon.b [INF] from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.3"}]: dispatch 2024-04-14T15:13:01.226 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:59.655456+0000 mon.b [INF] from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.4"}]: dispatch 2024-04-14T15:13:01.227 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:59.655833+0000 mon.b [INF] from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.5"}]: dispatch 2024-04-14T15:13:01.227 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:59.656203+0000 mon.b [INF] from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.6"}]: dispatch 2024-04-14T15:13:01.227 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:59.656596+0000 mon.b [INF] from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.7"}]: dispatch 2024-04-14T15:13:01.227 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:59.656940+0000 mon.b [INF] from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.8"}]: dispatch 2024-04-14T15:13:01.227 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:12:59.657308+0000 mon.b [INF] from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.9"}]: dispatch 2024-04-14T15:13:01.227 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:13:00.088145+0000 mon.a [INF] from='client.25240 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi090-93379-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi090-93379-16"}]': finished 2024-04-14T15:13:01.227 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:13:00.088208+0000 mon.a [INF] from='client.25243 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi090-93398-23", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi090-93398-23"}]': finished 2024-04-14T15:13:01.227 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:13:00.088252+0000 mon.a [INF] from='client.25267 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi090-93864-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi090-93864-7"}]': finished 2024-04-14T15:13:01.227 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:13:00.088322+0000 mon.a [INF] from='client.25249 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi090-93731-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi090-93731-7"}]': finished 2024-04-14T15:13:01.227 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:13:00.088364+0000 mon.a [INF] from='client.25021 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi090-94522-1"}]': finished 2024-04-14T15:13:01.227 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:13:00.088405+0000 mon.a [INF] from='client.25001 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi090-94662-1"}]': finished 2024-04-14T15:13:01.227 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:13:00.088454+0000 mon.a [INF] from='client.? v1:172.21.15.90:0/309942895' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi090-93998-1"}]': finished 2024-04-14T15:13:01.227 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:13:00.088499+0000 mon.a [INF] from='client.25114 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:01.227 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:13:00.088578+0000 mon.a [INF] from='client.? v1:172.21.15.90:0/274362209' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi090-93436-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:13:01.228 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:13:00.088620+0000 mon.a [INF] from='client.25217 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi090-93476-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:13:01.228 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:13:00.093971+0000 mon.a [INF] from='client.? v1:172.21.15.90:0/309942895' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi090-93998-1"}]: dispatch 2024-04-14T15:13:01.254 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:13:00.094273+0000 mon.c [INF] from='client.? v1:172.21.15.90:0/1713883211' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi090-93476-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi090-93476-10"}]: dispatch 2024-04-14T15:13:01.254 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:13:00.096263+0000 mon.c [INF] from='client.? v1:172.21.15.90:0/1465058495' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi090-94662-1"}]: dispatch 2024-04-14T15:13:01.254 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:13:00.097680+0000 mon.a [INF] from='client.? v1:172.21.15.90:0/274362209' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi090-93436-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi090-93436-10"}]: dispatch 2024-04-14T15:13:01.254 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:13:00.098172+0000 mon.a [INF] from='client.25217 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi090-93476-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi090-93476-10"}]: dispatch 2024-04-14T15:13:01.254 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:13:00.100102+0000 mon.a [INF] from='client.25001 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi090-94662-1"}]: dispatch 2024-04-14T15:13:01.254 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:13:00.105553+0000 mon.b [INF] from='client.? v1:172.21.15.90:0/532832389' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi090-93363-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:01.254 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:13:00.108929+0000 mon.b [INF] from='client.? v1:172.21.15.90:0/1327720256' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-4", "tierpool": "test-rados-api-smithi090-93652-4-cache"}]: dispatch 2024-04-14T15:13:01.254 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:13:00.109705+0000 mon.a [INF] from='client.25207 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi090-93363-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:01.254 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:13:00.110378+0000 mon.a [INF] from='client.25114 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-4", "tierpool": "test-rados-api-smithi090-93652-4-cache"}]: dispatch 2024-04-14T15:13:01.254 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:13:00.168170+0000 mon.b [INF] from='client.? v1:172.21.15.90:0/340734729' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi090-94522-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:13:01.255 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:13:00.194509+0000 mon.a [INF] from='client.25489 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi090-94522-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:13:01.255 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:13:00.786490+0000 mon.a [INF] from='client.? v1:172.21.15.90:0/309942895' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi090-93998-1"}]': finished 2024-04-14T15:13:01.255 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:13:00.786574+0000 mon.a [INF] from='client.25001 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi090-94662-1"}]': finished 2024-04-14T15:13:01.255 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:13:00.786650+0000 mon.a [INF] from='client.25207 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi090-93363-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:01.255 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:13:00.786720+0000 mon.a [INF] from='client.25114 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-4", "tierpool": "test-rados-api-smithi090-93652-4-cache"}]': finished 2024-04-14T15:13:01.255 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:13:00.786795+0000 mon.a [INF] from='client.25489 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi090-94522-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:13:01.255 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:13:00.795066+0000 mon.b [INF] from='client.? v1:172.21.15.90:0/532832389' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi090-93363-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-04-14T15:13:01.255 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:13:00.796319+0000 mon.b [INF] from='client.? v1:172.21.15.90:0/1327720256' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-4", "overlaypool": "test-rados-api-smithi090-93652-4-cache"}]: dispatch 2024-04-14T15:13:01.255 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:13:00.801861+0000 mon.b [INF] from='client.? v1:172.21.15.90:0/340734729' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi090-94522-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi090-94522-2"}]: dispatch 2024-04-14T15:13:01.255 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:13:00.818085+0000 mon.a [INF] from='client.25207 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi090-93363-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-04-14T15:13:01.255 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:13:00.818629+0000 mon.a [INF] from='client.25114 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-4", "overlaypool": "test-rados-api-smithi090-93652-4-cache"}]: dispatch 2024-04-14T15:13:01.255 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:13:00.820645+0000 mon.c [INF] from='client.? v1:172.21.15.90:0/2676581337' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:01.255 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:13:00.825052+0000 mon.b [INF] from='client.? v1:172.21.15.90:0/3158227083' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi090-93342-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:01.255 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:13:00.825481+0000 mon.b [INF] from='client.? v1:172.21.15.90:0/3022029644' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:01.256 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:13:00.857855+0000 mon.a [INF] from='client.25489 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi090-94522-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi090-94522-2"}]: dispatch 2024-04-14T15:13:01.256 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:13:00.858802+0000 mon.a [INF] from='client.25361 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:01.256 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:13:00.859890+0000 mon.a [INF] from='client.25516 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi090-93342-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:01.256 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:13:00.861139+0000 mon.a [INF] from='client.25504 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:01.256 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:13:00.886314+0000 mon.a [INF] from='client.? v1:172.21.15.90:0/1715119515' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi090-94662-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:13:01.256 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:13:01.225979+0000 mon.b [INF] from='client.? v1:172.21.15.90:0/4159084683' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-04-14T15:13:01.256 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:13:01.281 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:01.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:01.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:13:01.383 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:13:01.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.380+0000 7f575a236700 1 -- 172.21.15.90:0/501867522 >> v1:172.21.15.90:6790/0 conn(0x7f5754108880 legacy=0x7f5754108c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:01.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.380+0000 7f575a236700 1 -- 172.21.15.90:0/501867522 shutdown_connections 2024-04-14T15:13:01.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.380+0000 7f575a236700 1 -- 172.21.15.90:0/501867522 >> 172.21.15.90:0/501867522 conn(0x7f57540fe3d0 msgr2=0x7f57541007f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:13:01.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.380+0000 7f575a236700 1 -- 172.21.15.90:0/501867522 shutdown_connections 2024-04-14T15:13:01.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.380+0000 7f575a236700 1 -- 172.21.15.90:0/501867522 wait complete. 2024-04-14T15:13:01.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.381+0000 7f575a236700 1 librados: starting msgr at 2024-04-14T15:13:01.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.381+0000 7f575a236700 1 librados: starting objecter 2024-04-14T15:13:01.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.381+0000 7f575a236700 1 Processor -- start 2024-04-14T15:13:01.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.381+0000 7f575a236700 1 -- start start 2024-04-14T15:13:01.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.381+0000 7f575a236700 1 librados: setting wanted keys 2024-04-14T15:13:01.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.381+0000 7f575a236700 1 librados: calling monclient init 2024-04-14T15:13:01.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.381+0000 7f575a236700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f57541a9c70 con 0x7f5754108880 2024-04-14T15:13:01.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.381+0000 7f575a236700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f57541aae90 con 0x7f575410f300 2024-04-14T15:13:01.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.381+0000 7f575a236700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f57541ac0d0 con 0x7f575410b730 2024-04-14T15:13:01.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.381+0000 7f57537fe700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f5754108880 0x7f575410eb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:48320/0 (socket says 172.21.15.90:48320) 2024-04-14T15:13:01.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.381+0000 7f57537fe700 1 -- 172.21.15.90:0/1965288322 learned_addr learned my addr 172.21.15.90:0/1965288322 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:13:01.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.381+0000 7f57517fa700 10 client.?.objecter ms_handle_connect 0x7f5754108880 2024-04-14T15:13:01.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.381+0000 7f57517fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:13:01.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.381+0000 7f57517fa700 10 client.?.objecter ms_handle_connect 0x7f575410b730 2024-04-14T15:13:01.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.381+0000 7f57517fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:13:01.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.382+0000 7f57517fa700 10 client.?.objecter ms_handle_connect 0x7f575410f300 2024-04-14T15:13:01.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.382+0000 7f57517fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:13:01.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.382+0000 7f57517fa700 1 -- 172.21.15.90:0/1965288322 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3874292091 0 0) 0x7f57541a9c70 con 0x7f5754108880 2024-04-14T15:13:01.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.382+0000 7f57517fa700 1 -- 172.21.15.90:0/1965288322 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5740003aa0 con 0x7f5754108880 2024-04-14T15:13:01.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.382+0000 7f57517fa700 1 -- 172.21.15.90:0/1965288322 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1248165395 0 0) 0x7f57541ac0d0 con 0x7f575410b730 2024-04-14T15:13:01.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.382+0000 7f57517fa700 1 -- 172.21.15.90:0/1965288322 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f57541a9c70 con 0x7f575410b730 2024-04-14T15:13:01.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.382+0000 7f57517fa700 1 -- 172.21.15.90:0/1965288322 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4228861025 0 0) 0x7f57541aae90 con 0x7f575410f300 2024-04-14T15:13:01.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.382+0000 7f57517fa700 1 -- 172.21.15.90:0/1965288322 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f57541ac0d0 con 0x7f575410f300 2024-04-14T15:13:01.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.382+0000 7f57517fa700 1 -- 172.21.15.90:0/1965288322 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1903979039 0 0) 0x7f57541a9c70 con 0x7f575410b730 2024-04-14T15:13:01.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.382+0000 7f57517fa700 1 -- 172.21.15.90:0/1965288322 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f57541aae90 con 0x7f575410b730 2024-04-14T15:13:01.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.382+0000 7f57517fa700 1 -- 172.21.15.90:0/1965288322 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 436919534 0 0) 0x7f57541ac0d0 con 0x7f575410f300 2024-04-14T15:13:01.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.382+0000 7f57517fa700 1 -- 172.21.15.90:0/1965288322 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f57541a9c70 con 0x7f575410f300 2024-04-14T15:13:01.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.382+0000 7f57517fa700 1 -- 172.21.15.90:0/1965288322 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 367260136 0 0) 0x7f5740003aa0 con 0x7f5754108880 2024-04-14T15:13:01.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.382+0000 7f57517fa700 1 -- 172.21.15.90:0/1965288322 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f57541ac0d0 con 0x7f5754108880 2024-04-14T15:13:01.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.382+0000 7f57517fa700 1 -- 172.21.15.90:0/1965288322 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f573c001830 con 0x7f575410b730 2024-04-14T15:13:01.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.382+0000 7f57517fa700 1 -- 172.21.15.90:0/1965288322 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f574c0013f0 con 0x7f575410f300 2024-04-14T15:13:01.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.382+0000 7f57517fa700 1 -- 172.21.15.90:0/1965288322 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f57440043a0 con 0x7f5754108880 2024-04-14T15:13:01.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.382+0000 7f57517fa700 1 -- 172.21.15.90:0/1965288322 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 612210183 0 0) 0x7f57541a9c70 con 0x7f575410f300 2024-04-14T15:13:01.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.382+0000 7f57517fa700 1 -- 172.21.15.90:0/1965288322 >> v1:172.21.15.90:6790/0 conn(0x7f575410b730 legacy=0x7f57541a4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:01.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.382+0000 7f57517fa700 1 -- 172.21.15.90:0/1965288322 >> v1:172.21.15.90:6789/0 conn(0x7f5754108880 legacy=0x7f575410eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:01.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.382+0000 7f57517fa700 1 -- 172.21.15.90:0/1965288322 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57541ad310 con 0x7f575410f300 2024-04-14T15:13:01.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.382+0000 7f575a236700 1 -- 172.21.15.90:0/1965288322 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f57541ac300 con 0x7f575410f300 2024-04-14T15:13:01.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.382+0000 7f575a236700 10 client.25546.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:13:01.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.382+0000 7f575a236700 1 -- 172.21.15.90:0/1965288322 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f57541ac7c0 con 0x7f575410f300 2024-04-14T15:13:01.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.382+0000 7f575a236700 1 librados: init done 2024-04-14T15:13:01.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.382+0000 7f57517fa700 1 -- 172.21.15.90:0/1965288322 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f574c0015d0 con 0x7f575410f300 2024-04-14T15:13:01.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.382+0000 7f57517fa700 1 -- 172.21.15.90:0/1965288322 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f574c004c80 con 0x7f575410f300 2024-04-14T15:13:01.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.383+0000 7f575a236700 1 -- 172.21.15.90:0/1965288322 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f5734005130 con 0x7f575410f300 2024-04-14T15:13:01.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.384+0000 7f57517fa700 1 -- 172.21.15.90:0/1965288322 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f574c003a00 con 0x7f575410f300 2024-04-14T15:13:01.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.384+0000 7f57517fa700 1 -- 172.21.15.90:0/1965288322 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(113..113 src has 51..113) ==== 19069+0+0 (unknown 2100304605 0 0) 0x7f574c08c3a0 con 0x7f575410f300 2024-04-14T15:13:01.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.384+0000 7f57517fa700 10 client.25546.objecter ms_dispatch 0x7f5754001550 osd_map(113..113 src has 51..113) 2024-04-14T15:13:01.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.384+0000 7f57517fa700 3 client.25546.objecter handle_osd_map got epochs [113,113] > 0 2024-04-14T15:13:01.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.384+0000 7f57517fa700 3 client.25546.objecter handle_osd_map decoding full epoch 113 2024-04-14T15:13:01.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.385+0000 7f57517fa700 20 client.25546.objecter dump_active .. 0 homeless 2024-04-14T15:13:01.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.385+0000 7f57517fa700 10 client.25546.objecter ms_handle_connect 0x7f57400713b0 2024-04-14T15:13:01.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.385+0000 7f57517fa700 1 -- 172.21.15.90:0/1965288322 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v113) ==== 54+0+16 (unknown 891576468 0 2420000192) 0x7f574c08e7d0 con 0x7f575410f300 2024-04-14T15:13:01.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.385+0000 7f575a236700 10 client.25546.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:13:01.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.385+0000 7f575a236700 15 client.25546.objecter _session_command_op_assign -1 1 2024-04-14T15:13:01.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.385+0000 7f575a236700 20 client.25546.objecter _get_session s=0x7f5734005410 osd=0 3 2024-04-14T15:13:01.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.385+0000 7f575a236700 20 client.25546.objecter put_session s=0x7f5734005410 osd=0 3 2024-04-14T15:13:01.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.385+0000 7f575a236700 20 client.25546.objecter _get_session s=0x7f5734005410 osd=0 3 2024-04-14T15:13:01.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.385+0000 7f575a236700 15 client.25546.objecter _session_command_op_remove -1 1 2024-04-14T15:13:01.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.385+0000 7f575a236700 20 client.25546.objecter get_session s=0x7f5734005410 osd=0 3 2024-04-14T15:13:01.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.385+0000 7f575a236700 15 client.25546.objecter _session_command_op_assign 0 1 2024-04-14T15:13:01.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.385+0000 7f575a236700 20 client.25546.objecter put_session s=0x7f5734005410 osd=0 4 2024-04-14T15:13:01.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.385+0000 7f575a236700 10 client.25546.objecter _send_command 1 2024-04-14T15:13:01.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.385+0000 7f575a236700 1 -- 172.21.15.90:0/1965288322 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5734004b80 con 0x7f5734004760 2024-04-14T15:13:01.390 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:01 smithi090 ceph-mon[41670]: 16.2 deep-scrub starts 2024-04-14T15:13:01.391 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:01 smithi090 ceph-mon[41670]: 16.2 deep-scrub ok 2024-04-14T15:13:01.391 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:01 smithi090 ceph-mon[41670]: 16.3 deep-scrub starts 2024-04-14T15:13:01.391 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:01 smithi090 ceph-mon[41670]: 16.3 deep-scrub ok 2024-04-14T15:13:01.391 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:01 smithi090 ceph-mon[41670]: 16.9 deep-scrub starts 2024-04-14T15:13:01.391 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:01 smithi090 ceph-mon[41670]: 16.9 deep-scrub ok 2024-04-14T15:13:01.391 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:01 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:01.391 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:01 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/309942895' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi090-93998-1"}]': finished 2024-04-14T15:13:01.391 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:01 smithi090 ceph-mon[41670]: from='client.25001 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi090-94662-1"}]': finished 2024-04-14T15:13:01.391 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:01 smithi090 ceph-mon[41670]: from='client.25207 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi090-93363-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:01.391 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:01 smithi090 ceph-mon[41670]: from='client.25114 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-4", "tierpool": "test-rados-api-smithi090-93652-4-cache"}]': finished 2024-04-14T15:13:01.391 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:01 smithi090 ceph-mon[41670]: from='client.25489 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi090-94522-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:13:01.391 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:01 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/532832389' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi090-93363-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-04-14T15:13:01.392 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:01 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1327720256' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-4", "overlaypool": "test-rados-api-smithi090-93652-4-cache"}]: dispatch 2024-04-14T15:13:01.392 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:01 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/340734729' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi090-94522-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi090-94522-2"}]: dispatch 2024-04-14T15:13:01.392 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:01 smithi090 ceph-mon[41670]: osdmap e113: 8 total, 8 up, 8 in 2024-04-14T15:13:01.392 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:01 smithi090 ceph-mon[41670]: from='client.25207 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi090-93363-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-04-14T15:13:01.392 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:01 smithi090 ceph-mon[41670]: from='client.25114 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-4", "overlaypool": "test-rados-api-smithi090-93652-4-cache"}]: dispatch 2024-04-14T15:13:01.392 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:01 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2676581337' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:01.392 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:01 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3158227083' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi090-93342-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:01.392 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:01 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3022029644' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:01.392 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:01 smithi090 ceph-mon[41670]: from='client.25489 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi090-94522-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi090-94522-2"}]: dispatch 2024-04-14T15:13:01.392 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:01 smithi090 ceph-mon[41670]: from='client.25361 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:01.392 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:01 smithi090 ceph-mon[41670]: from='client.25516 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi090-93342-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:01.392 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:01 smithi090 ceph-mon[41670]: from='client.25504 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:01.392 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:01 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1715119515' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi090-94662-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:13:01.393 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:01 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/4159084683' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-04-14T15:13:01.393 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:01 smithi090 ceph-mon[41670]: threexx 2024-04-14T15:13:01.393 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:01 smithi090 ceph-mon[41670]: from='client.25252 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-04-14T15:13:01.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.386+0000 7f57517fa700 10 client.25546.objecter ms_handle_connect 0x7f5734004760 2024-04-14T15:13:01.393 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:01 smithi090 ceph-mon[30399]: 16.2 deep-scrub starts 2024-04-14T15:13:01.394 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:01 smithi090 ceph-mon[30399]: 16.2 deep-scrub ok 2024-04-14T15:13:01.394 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:01 smithi090 ceph-mon[30399]: 16.3 deep-scrub starts 2024-04-14T15:13:01.394 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:01 smithi090 ceph-mon[30399]: 16.3 deep-scrub ok 2024-04-14T15:13:01.394 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:01 smithi090 ceph-mon[30399]: 16.9 deep-scrub starts 2024-04-14T15:13:01.394 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:01 smithi090 ceph-mon[30399]: 16.9 deep-scrub ok 2024-04-14T15:13:01.394 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:01 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:01.394 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:01 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/309942895' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi090-93998-1"}]': finished 2024-04-14T15:13:01.394 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:01 smithi090 ceph-mon[30399]: from='client.25001 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi090-94662-1"}]': finished 2024-04-14T15:13:01.394 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:01 smithi090 ceph-mon[30399]: from='client.25207 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi090-93363-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:01.394 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:01 smithi090 ceph-mon[30399]: from='client.25114 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-4", "tierpool": "test-rados-api-smithi090-93652-4-cache"}]': finished 2024-04-14T15:13:01.394 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:01 smithi090 ceph-mon[30399]: from='client.25489 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi090-94522-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:13:01.394 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:01 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/532832389' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi090-93363-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-04-14T15:13:01.394 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:01 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1327720256' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-4", "overlaypool": "test-rados-api-smithi090-93652-4-cache"}]: dispatch 2024-04-14T15:13:01.395 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:01 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/340734729' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi090-94522-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi090-94522-2"}]: dispatch 2024-04-14T15:13:01.395 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:01 smithi090 ceph-mon[30399]: osdmap e113: 8 total, 8 up, 8 in 2024-04-14T15:13:01.395 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:01 smithi090 ceph-mon[30399]: from='client.25207 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi090-93363-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-04-14T15:13:01.395 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:01 smithi090 ceph-mon[30399]: from='client.25114 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-4", "overlaypool": "test-rados-api-smithi090-93652-4-cache"}]: dispatch 2024-04-14T15:13:01.395 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:01 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2676581337' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:01.395 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:01 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3158227083' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi090-93342-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:01.395 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:01 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3022029644' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:01.395 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:01 smithi090 ceph-mon[30399]: from='client.25489 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi090-94522-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi090-94522-2"}]: dispatch 2024-04-14T15:13:01.395 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:01 smithi090 ceph-mon[30399]: from='client.25361 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:01.395 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:01 smithi090 ceph-mon[30399]: from='client.25516 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi090-93342-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:01.395 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:01 smithi090 ceph-mon[30399]: from='client.25504 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:01.395 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:01 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1715119515' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi090-94662-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:13:01.395 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:01 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/4159084683' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-04-14T15:13:01.396 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:01 smithi090 ceph-mon[30399]: threexx 2024-04-14T15:13:01.396 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:01 smithi090 ceph-mon[30399]: from='client.25252 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-04-14T15:13:01.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.387+0000 7f57517fa700 1 -- 172.21.15.90:0/1965288322 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f5734004b80 con 0x7f5734004760 2024-04-14T15:13:01.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.387+0000 7f57517fa700 10 client.25546.objecter ms_dispatch 0x7f5754001550 command_reply(tid 1: 0 ) 2024-04-14T15:13:01.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.387+0000 7f57517fa700 10 client.25546.objecter _finish_command 1 = system:0 2024-04-14T15:13:01.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.387+0000 7f57517fa700 20 client.25546.objecter put_session s=0x7f5734005410 osd=0 4 2024-04-14T15:13:01.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.387+0000 7f57517fa700 15 client.25546.objecter _session_command_op_remove 0 1 2024-04-14T15:13:01.397 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:13:01.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:01.411 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:01.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:13:01.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.411+0000 7f575a236700 10 client.25546.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:13:01.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.411+0000 7f575a236700 15 client.25546.objecter _session_command_op_assign -1 2 2024-04-14T15:13:01.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.411+0000 7f575a236700 20 client.25546.objecter _get_session s=0x7f5734005410 osd=0 3 2024-04-14T15:13:01.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.411+0000 7f575a236700 20 client.25546.objecter put_session s=0x7f5734005410 osd=0 3 2024-04-14T15:13:01.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.411+0000 7f575a236700 20 client.25546.objecter _get_session s=0x7f5734005410 osd=0 3 2024-04-14T15:13:01.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.411+0000 7f575a236700 15 client.25546.objecter _session_command_op_remove -1 2 2024-04-14T15:13:01.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.411+0000 7f575a236700 20 client.25546.objecter get_session s=0x7f5734005410 osd=0 3 2024-04-14T15:13:01.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.411+0000 7f575a236700 15 client.25546.objecter _session_command_op_assign 0 2 2024-04-14T15:13:01.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.411+0000 7f575a236700 20 client.25546.objecter put_session s=0x7f5734005410 osd=0 4 2024-04-14T15:13:01.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.411+0000 7f575a236700 10 client.25546.objecter _send_command 2 2024-04-14T15:13:01.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.411+0000 7f575a236700 1 -- 172.21.15.90:0/1965288322 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f5734007b30 con 0x7f5734004760 2024-04-14T15:13:01.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.413+0000 7f57517fa700 1 -- 172.21.15.90:0/1965288322 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f5734007b30 con 0x7f5734004760 2024-04-14T15:13:01.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.413+0000 7f57517fa700 10 client.25546.objecter ms_dispatch 0x7f5754001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:13:01.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.413+0000 7f57517fa700 10 client.25546.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:13:01.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.413+0000 7f57517fa700 20 client.25546.objecter put_session s=0x7f5734005410 osd=0 4 2024-04-14T15:13:01.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.413+0000 7f57517fa700 15 client.25546.objecter _session_command_op_remove 0 2 2024-04-14T15:13:01.414 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:13:01.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.414+0000 7f575a236700 10 client.25546.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:13:01.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.414+0000 7f575a236700 15 client.25546.objecter _session_command_op_assign -1 3 2024-04-14T15:13:01.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.414+0000 7f575a236700 20 client.25546.objecter _get_session s=0x7f57340041b0 osd=1 3 2024-04-14T15:13:01.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.414+0000 7f575a236700 20 client.25546.objecter put_session s=0x7f57340041b0 osd=1 3 2024-04-14T15:13:01.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.414+0000 7f575a236700 20 client.25546.objecter _get_session s=0x7f57340041b0 osd=1 3 2024-04-14T15:13:01.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.414+0000 7f575a236700 15 client.25546.objecter _session_command_op_remove -1 3 2024-04-14T15:13:01.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.414+0000 7f575a236700 20 client.25546.objecter get_session s=0x7f57340041b0 osd=1 3 2024-04-14T15:13:01.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.414+0000 7f575a236700 15 client.25546.objecter _session_command_op_assign 1 3 2024-04-14T15:13:01.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.414+0000 7f575a236700 20 client.25546.objecter put_session s=0x7f57340041b0 osd=1 4 2024-04-14T15:13:01.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.414+0000 7f575a236700 10 client.25546.objecter _send_command 3 2024-04-14T15:13:01.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.414+0000 7f575a236700 1 -- 172.21.15.90:0/1965288322 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f5734002230 con 0x7f5734004320 2024-04-14T15:13:01.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.414+0000 7f57517fa700 10 client.25546.objecter ms_handle_connect 0x7f5734004320 2024-04-14T15:13:01.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.415+0000 7f57517fa700 1 -- 172.21.15.90:0/1965288322 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f5734002230 con 0x7f5734004320 2024-04-14T15:13:01.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.415+0000 7f57517fa700 10 client.25546.objecter ms_dispatch 0x7f5754001550 command_reply(tid 3: 0 ) 2024-04-14T15:13:01.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.415+0000 7f57517fa700 10 client.25546.objecter _finish_command 3 = system:0 2024-04-14T15:13:01.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.415+0000 7f57517fa700 20 client.25546.objecter put_session s=0x7f57340041b0 osd=1 4 2024-04-14T15:13:01.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.415+0000 7f57517fa700 15 client.25546.objecter _session_command_op_remove 1 3 2024-04-14T15:13:01.431 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:13:01.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.437+0000 7f575a236700 10 client.25546.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:13:01.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.437+0000 7f575a236700 15 client.25546.objecter _session_command_op_assign -1 4 2024-04-14T15:13:01.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.437+0000 7f575a236700 20 client.25546.objecter _get_session s=0x7f57340041b0 osd=1 3 2024-04-14T15:13:01.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.437+0000 7f575a236700 20 client.25546.objecter put_session s=0x7f57340041b0 osd=1 3 2024-04-14T15:13:01.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.437+0000 7f575a236700 20 client.25546.objecter _get_session s=0x7f57340041b0 osd=1 3 2024-04-14T15:13:01.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.437+0000 7f575a236700 15 client.25546.objecter _session_command_op_remove -1 4 2024-04-14T15:13:01.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.437+0000 7f575a236700 20 client.25546.objecter get_session s=0x7f57340041b0 osd=1 3 2024-04-14T15:13:01.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.437+0000 7f575a236700 15 client.25546.objecter _session_command_op_assign 1 4 2024-04-14T15:13:01.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.437+0000 7f575a236700 20 client.25546.objecter put_session s=0x7f57340041b0 osd=1 4 2024-04-14T15:13:01.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.437+0000 7f575a236700 10 client.25546.objecter _send_command 4 2024-04-14T15:13:01.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.437+0000 7f575a236700 1 -- 172.21.15.90:0/1965288322 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f5734001690 con 0x7f5734004320 2024-04-14T15:13:01.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.439+0000 7f57517fa700 1 -- 172.21.15.90:0/1965288322 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f5734001690 con 0x7f5734004320 2024-04-14T15:13:01.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.439+0000 7f57517fa700 10 client.25546.objecter ms_dispatch 0x7f5754001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:13:01.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.439+0000 7f57517fa700 10 client.25546.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:13:01.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.439+0000 7f57517fa700 20 client.25546.objecter put_session s=0x7f57340041b0 osd=1 4 2024-04-14T15:13:01.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.439+0000 7f57517fa700 15 client.25546.objecter _session_command_op_remove 1 4 2024-04-14T15:13:01.440 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:13:01.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.439+0000 7f575a236700 10 client.25546.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:13:01.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.439+0000 7f575a236700 15 client.25546.objecter _session_command_op_assign -1 5 2024-04-14T15:13:01.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.439+0000 7f575a236700 20 client.25546.objecter _get_session s=0x7f57340084b0 osd=2 3 2024-04-14T15:13:01.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.439+0000 7f575a236700 20 client.25546.objecter put_session s=0x7f57340084b0 osd=2 3 2024-04-14T15:13:01.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.439+0000 7f575a236700 20 client.25546.objecter _get_session s=0x7f57340084b0 osd=2 3 2024-04-14T15:13:01.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.439+0000 7f575a236700 15 client.25546.objecter _session_command_op_remove -1 5 2024-04-14T15:13:01.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.439+0000 7f575a236700 20 client.25546.objecter get_session s=0x7f57340084b0 osd=2 3 2024-04-14T15:13:01.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.439+0000 7f575a236700 15 client.25546.objecter _session_command_op_assign 2 5 2024-04-14T15:13:01.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.439+0000 7f575a236700 20 client.25546.objecter put_session s=0x7f57340084b0 osd=2 4 2024-04-14T15:13:01.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.439+0000 7f575a236700 10 client.25546.objecter _send_command 5 2024-04-14T15:13:01.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.439+0000 7f575a236700 1 -- 172.21.15.90:0/1965288322 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f5734011220 con 0x7f573400d710 2024-04-14T15:13:01.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.440+0000 7f57517fa700 10 client.25546.objecter ms_handle_connect 0x7f573400d710 2024-04-14T15:13:01.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.440+0000 7f57517fa700 1 -- 172.21.15.90:0/1965288322 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f5734011220 con 0x7f573400d710 2024-04-14T15:13:01.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.440+0000 7f57517fa700 10 client.25546.objecter ms_dispatch 0x7f5754001550 command_reply(tid 5: 0 ) 2024-04-14T15:13:01.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.440+0000 7f57517fa700 10 client.25546.objecter _finish_command 5 = system:0 2024-04-14T15:13:01.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.440+0000 7f57517fa700 20 client.25546.objecter put_session s=0x7f57340084b0 osd=2 4 2024-04-14T15:13:01.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.440+0000 7f57517fa700 15 client.25546.objecter _session_command_op_remove 2 5 2024-04-14T15:13:01.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.458+0000 7f575a236700 10 client.25546.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:13:01.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.458+0000 7f575a236700 15 client.25546.objecter _session_command_op_assign -1 6 2024-04-14T15:13:01.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.458+0000 7f575a236700 20 client.25546.objecter _get_session s=0x7f57340084b0 osd=2 3 2024-04-14T15:13:01.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.458+0000 7f575a236700 20 client.25546.objecter put_session s=0x7f57340084b0 osd=2 3 2024-04-14T15:13:01.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.458+0000 7f575a236700 20 client.25546.objecter _get_session s=0x7f57340084b0 osd=2 3 2024-04-14T15:13:01.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.458+0000 7f575a236700 15 client.25546.objecter _session_command_op_remove -1 6 2024-04-14T15:13:01.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.458+0000 7f575a236700 20 client.25546.objecter get_session s=0x7f57340084b0 osd=2 3 2024-04-14T15:13:01.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.458+0000 7f575a236700 15 client.25546.objecter _session_command_op_assign 2 6 2024-04-14T15:13:01.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.458+0000 7f575a236700 20 client.25546.objecter put_session s=0x7f57340084b0 osd=2 4 2024-04-14T15:13:01.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.458+0000 7f575a236700 10 client.25546.objecter _send_command 6 2024-04-14T15:13:01.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.458+0000 7f575a236700 1 -- 172.21.15.90:0/1965288322 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f5734007ff0 con 0x7f573400d710 2024-04-14T15:13:01.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.459+0000 7f57517fa700 1 -- 172.21.15.90:0/1965288322 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f5734007ff0 con 0x7f573400d710 2024-04-14T15:13:01.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.459+0000 7f57517fa700 10 client.25546.objecter ms_dispatch 0x7f5754001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:13:01.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.459+0000 7f57517fa700 10 client.25546.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:13:01.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.459+0000 7f57517fa700 20 client.25546.objecter put_session s=0x7f57340084b0 osd=2 4 2024-04-14T15:13:01.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.459+0000 7f57517fa700 15 client.25546.objecter _session_command_op_remove 2 6 2024-04-14T15:13:01.460 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:13:01.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.460+0000 7f575a236700 10 client.25546.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:13:01.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.460+0000 7f575a236700 15 client.25546.objecter _session_command_op_assign -1 7 2024-04-14T15:13:01.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.460+0000 7f575a236700 20 client.25546.objecter _get_session s=0x7f573400cd60 osd=3 3 2024-04-14T15:13:01.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.460+0000 7f575a236700 20 client.25546.objecter put_session s=0x7f573400cd60 osd=3 3 2024-04-14T15:13:01.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.460+0000 7f575a236700 20 client.25546.objecter _get_session s=0x7f573400cd60 osd=3 3 2024-04-14T15:13:01.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.460+0000 7f575a236700 15 client.25546.objecter _session_command_op_remove -1 7 2024-04-14T15:13:01.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.460+0000 7f575a236700 20 client.25546.objecter get_session s=0x7f573400cd60 osd=3 3 2024-04-14T15:13:01.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.460+0000 7f575a236700 15 client.25546.objecter _session_command_op_assign 3 7 2024-04-14T15:13:01.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.460+0000 7f575a236700 20 client.25546.objecter put_session s=0x7f573400cd60 osd=3 4 2024-04-14T15:13:01.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.460+0000 7f575a236700 10 client.25546.objecter _send_command 7 2024-04-14T15:13:01.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.460+0000 7f575a236700 1 -- 172.21.15.90:0/1965288322 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f5734016060 con 0x7f5734012550 2024-04-14T15:13:01.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.460+0000 7f57517fa700 10 client.25546.objecter ms_handle_connect 0x7f5734012550 2024-04-14T15:13:01.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.461+0000 7f57517fa700 1 -- 172.21.15.90:0/1965288322 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f5734016060 con 0x7f5734012550 2024-04-14T15:13:01.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.461+0000 7f57517fa700 10 client.25546.objecter ms_dispatch 0x7f5754001550 command_reply(tid 7: 0 ) 2024-04-14T15:13:01.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.461+0000 7f57517fa700 10 client.25546.objecter _finish_command 7 = system:0 2024-04-14T15:13:01.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.461+0000 7f57517fa700 20 client.25546.objecter put_session s=0x7f573400cd60 osd=3 4 2024-04-14T15:13:01.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.461+0000 7f57517fa700 15 client.25546.objecter _session_command_op_remove 3 7 2024-04-14T15:13:01.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.477+0000 7f575a236700 10 client.25546.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:13:01.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.477+0000 7f575a236700 15 client.25546.objecter _session_command_op_assign -1 8 2024-04-14T15:13:01.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.477+0000 7f575a236700 20 client.25546.objecter _get_session s=0x7f573400cd60 osd=3 3 2024-04-14T15:13:01.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.477+0000 7f575a236700 20 client.25546.objecter put_session s=0x7f573400cd60 osd=3 3 2024-04-14T15:13:01.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.477+0000 7f575a236700 20 client.25546.objecter _get_session s=0x7f573400cd60 osd=3 3 2024-04-14T15:13:01.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.477+0000 7f575a236700 15 client.25546.objecter _session_command_op_remove -1 8 2024-04-14T15:13:01.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.477+0000 7f575a236700 20 client.25546.objecter get_session s=0x7f573400cd60 osd=3 3 2024-04-14T15:13:01.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.477+0000 7f575a236700 15 client.25546.objecter _session_command_op_assign 3 8 2024-04-14T15:13:01.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.477+0000 7f575a236700 20 client.25546.objecter put_session s=0x7f573400cd60 osd=3 4 2024-04-14T15:13:01.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.477+0000 7f575a236700 10 client.25546.objecter _send_command 8 2024-04-14T15:13:01.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.477+0000 7f575a236700 1 -- 172.21.15.90:0/1965288322 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f573400c8a0 con 0x7f5734012550 2024-04-14T15:13:01.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.479+0000 7f57517fa700 1 -- 172.21.15.90:0/1965288322 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f573400c8a0 con 0x7f5734012550 2024-04-14T15:13:01.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.479+0000 7f57517fa700 10 client.25546.objecter ms_dispatch 0x7f5754001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:13:01.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.479+0000 7f57517fa700 10 client.25546.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:13:01.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.479+0000 7f57517fa700 20 client.25546.objecter put_session s=0x7f573400cd60 osd=3 4 2024-04-14T15:13:01.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.479+0000 7f57517fa700 15 client.25546.objecter _session_command_op_remove 3 8 2024-04-14T15:13:01.480 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:13:01.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.479+0000 7f575a236700 10 client.25546.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:13:01.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.479+0000 7f575a236700 15 client.25546.objecter _session_command_op_assign -1 9 2024-04-14T15:13:01.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.479+0000 7f575a236700 20 client.25546.objecter _get_session s=0x7f573400c8a0 osd=4 3 2024-04-14T15:13:01.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.479+0000 7f575a236700 20 client.25546.objecter put_session s=0x7f573400c8a0 osd=4 3 2024-04-14T15:13:01.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.479+0000 7f575a236700 20 client.25546.objecter _get_session s=0x7f573400c8a0 osd=4 3 2024-04-14T15:13:01.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.479+0000 7f575a236700 15 client.25546.objecter _session_command_op_remove -1 9 2024-04-14T15:13:01.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.479+0000 7f575a236700 20 client.25546.objecter get_session s=0x7f573400c8a0 osd=4 3 2024-04-14T15:13:01.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.479+0000 7f575a236700 15 client.25546.objecter _session_command_op_assign 4 9 2024-04-14T15:13:01.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.479+0000 7f575a236700 20 client.25546.objecter put_session s=0x7f573400c8a0 osd=4 4 2024-04-14T15:13:01.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.479+0000 7f575a236700 10 client.25546.objecter _send_command 9 2024-04-14T15:13:01.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.479+0000 7f575a236700 1 -- 172.21.15.90:0/1965288322 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f573401aac0 con 0x7f57340120b0 2024-04-14T15:13:01.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.479+0000 7f57517fa700 10 client.25546.objecter ms_handle_connect 0x7f57340120b0 2024-04-14T15:13:01.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.480+0000 7f57517fa700 1 -- 172.21.15.90:0/1965288322 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f573401aac0 con 0x7f57340120b0 2024-04-14T15:13:01.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.480+0000 7f57517fa700 10 client.25546.objecter ms_dispatch 0x7f5754001550 command_reply(tid 9: 0 ) 2024-04-14T15:13:01.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.480+0000 7f57517fa700 10 client.25546.objecter _finish_command 9 = system:0 2024-04-14T15:13:01.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.480+0000 7f57517fa700 20 client.25546.objecter put_session s=0x7f573400c8a0 osd=4 4 2024-04-14T15:13:01.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.480+0000 7f57517fa700 15 client.25546.objecter _session_command_op_remove 4 9 2024-04-14T15:13:01.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.496+0000 7f575a236700 10 client.25546.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:13:01.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.497+0000 7f575a236700 15 client.25546.objecter _session_command_op_assign -1 10 2024-04-14T15:13:01.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.497+0000 7f575a236700 20 client.25546.objecter _get_session s=0x7f573400c8a0 osd=4 3 2024-04-14T15:13:01.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.497+0000 7f575a236700 20 client.25546.objecter put_session s=0x7f573400c8a0 osd=4 3 2024-04-14T15:13:01.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.497+0000 7f575a236700 20 client.25546.objecter _get_session s=0x7f573400c8a0 osd=4 3 2024-04-14T15:13:01.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.497+0000 7f575a236700 15 client.25546.objecter _session_command_op_remove -1 10 2024-04-14T15:13:01.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.497+0000 7f575a236700 20 client.25546.objecter get_session s=0x7f573400c8a0 osd=4 3 2024-04-14T15:13:01.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.497+0000 7f575a236700 15 client.25546.objecter _session_command_op_assign 4 10 2024-04-14T15:13:01.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.497+0000 7f575a236700 20 client.25546.objecter put_session s=0x7f573400c8a0 osd=4 4 2024-04-14T15:13:01.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.497+0000 7f575a236700 10 client.25546.objecter _send_command 10 2024-04-14T15:13:01.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.497+0000 7f575a236700 1 -- 172.21.15.90:0/1965288322 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f573401aac0 con 0x7f57340120b0 2024-04-14T15:13:01.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.498+0000 7f57517fa700 1 -- 172.21.15.90:0/1965288322 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f573401aac0 con 0x7f57340120b0 2024-04-14T15:13:01.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.498+0000 7f57517fa700 10 client.25546.objecter ms_dispatch 0x7f5754001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:13:01.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.498+0000 7f57517fa700 10 client.25546.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:13:01.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.498+0000 7f57517fa700 20 client.25546.objecter put_session s=0x7f573400c8a0 osd=4 4 2024-04-14T15:13:01.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.498+0000 7f57517fa700 15 client.25546.objecter _session_command_op_remove 4 10 2024-04-14T15:13:01.499 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:13:01.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.499+0000 7f575a236700 10 client.25546.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:13:01.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.499+0000 7f575a236700 15 client.25546.objecter _session_command_op_assign -1 11 2024-04-14T15:13:01.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.499+0000 7f575a236700 20 client.25546.objecter _get_session s=0x7f573401aac0 osd=5 3 2024-04-14T15:13:01.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.499+0000 7f575a236700 20 client.25546.objecter put_session s=0x7f573401aac0 osd=5 3 2024-04-14T15:13:01.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.499+0000 7f575a236700 20 client.25546.objecter _get_session s=0x7f573401aac0 osd=5 3 2024-04-14T15:13:01.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.499+0000 7f575a236700 15 client.25546.objecter _session_command_op_remove -1 11 2024-04-14T15:13:01.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.499+0000 7f575a236700 20 client.25546.objecter get_session s=0x7f573401aac0 osd=5 3 2024-04-14T15:13:01.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.499+0000 7f575a236700 15 client.25546.objecter _session_command_op_assign 5 11 2024-04-14T15:13:01.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.499+0000 7f575a236700 20 client.25546.objecter put_session s=0x7f573401aac0 osd=5 4 2024-04-14T15:13:01.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.499+0000 7f575a236700 10 client.25546.objecter _send_command 11 2024-04-14T15:13:01.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.499+0000 7f575a236700 1 -- 172.21.15.90:0/1965288322 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f573401ee00 con 0x7f573401b140 2024-04-14T15:13:01.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.499+0000 7f57517fa700 10 client.25546.objecter ms_handle_connect 0x7f573401b140 2024-04-14T15:13:01.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.500+0000 7f57517fa700 1 -- 172.21.15.90:0/1965288322 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f573401ee00 con 0x7f573401b140 2024-04-14T15:13:01.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.500+0000 7f57517fa700 10 client.25546.objecter ms_dispatch 0x7f5754001550 command_reply(tid 11: 0 ) 2024-04-14T15:13:01.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.500+0000 7f57517fa700 10 client.25546.objecter _finish_command 11 = system:0 2024-04-14T15:13:01.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.500+0000 7f57517fa700 20 client.25546.objecter put_session s=0x7f573401aac0 osd=5 4 2024-04-14T15:13:01.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.500+0000 7f57517fa700 15 client.25546.objecter _session_command_op_remove 5 11 2024-04-14T15:13:01.511 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:13:01.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.516+0000 7f575a236700 10 client.25546.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:13:01.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.516+0000 7f575a236700 15 client.25546.objecter _session_command_op_assign -1 12 2024-04-14T15:13:01.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.516+0000 7f575a236700 20 client.25546.objecter _get_session s=0x7f573401aac0 osd=5 3 2024-04-14T15:13:01.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.516+0000 7f575a236700 20 client.25546.objecter put_session s=0x7f573401aac0 osd=5 3 2024-04-14T15:13:01.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.516+0000 7f575a236700 20 client.25546.objecter _get_session s=0x7f573401aac0 osd=5 3 2024-04-14T15:13:01.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.516+0000 7f575a236700 15 client.25546.objecter _session_command_op_remove -1 12 2024-04-14T15:13:01.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.516+0000 7f575a236700 20 client.25546.objecter get_session s=0x7f573401aac0 osd=5 3 2024-04-14T15:13:01.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.516+0000 7f575a236700 15 client.25546.objecter _session_command_op_assign 5 12 2024-04-14T15:13:01.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.516+0000 7f575a236700 20 client.25546.objecter put_session s=0x7f573401aac0 osd=5 4 2024-04-14T15:13:01.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.516+0000 7f575a236700 10 client.25546.objecter _send_command 12 2024-04-14T15:13:01.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.516+0000 7f575a236700 1 -- 172.21.15.90:0/1965288322 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f57340207b0 con 0x7f573401b140 2024-04-14T15:13:01.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.518+0000 7f57517fa700 1 -- 172.21.15.90:0/1965288322 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f57340207b0 con 0x7f573401b140 2024-04-14T15:13:01.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.518+0000 7f57517fa700 10 client.25546.objecter ms_dispatch 0x7f5754001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:13:01.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.518+0000 7f57517fa700 10 client.25546.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:13:01.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.518+0000 7f57517fa700 20 client.25546.objecter put_session s=0x7f573401aac0 osd=5 4 2024-04-14T15:13:01.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.518+0000 7f57517fa700 15 client.25546.objecter _session_command_op_remove 5 12 2024-04-14T15:13:01.522 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:13:01.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.518+0000 7f575a236700 10 client.25546.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:13:01.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.518+0000 7f575a236700 15 client.25546.objecter _session_command_op_assign -1 13 2024-04-14T15:13:01.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.518+0000 7f575a236700 20 client.25546.objecter _get_session s=0x7f57340207b0 osd=6 3 2024-04-14T15:13:01.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.518+0000 7f575a236700 20 client.25546.objecter put_session s=0x7f57340207b0 osd=6 3 2024-04-14T15:13:01.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.518+0000 7f575a236700 20 client.25546.objecter _get_session s=0x7f57340207b0 osd=6 3 2024-04-14T15:13:01.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.518+0000 7f575a236700 15 client.25546.objecter _session_command_op_remove -1 13 2024-04-14T15:13:01.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.518+0000 7f575a236700 20 client.25546.objecter get_session s=0x7f57340207b0 osd=6 3 2024-04-14T15:13:01.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.518+0000 7f575a236700 15 client.25546.objecter _session_command_op_assign 6 13 2024-04-14T15:13:01.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.518+0000 7f575a236700 20 client.25546.objecter put_session s=0x7f57340207b0 osd=6 4 2024-04-14T15:13:01.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.518+0000 7f575a236700 10 client.25546.objecter _send_command 13 2024-04-14T15:13:01.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.518+0000 7f575a236700 1 -- 172.21.15.90:0/1965288322 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f5734021310 con 0x7f5734020ed0 2024-04-14T15:13:01.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.518+0000 7f57517fa700 10 client.25546.objecter ms_handle_connect 0x7f5734020ed0 2024-04-14T15:13:01.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.519+0000 7f57517fa700 1 -- 172.21.15.90:0/1965288322 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f5734021310 con 0x7f5734020ed0 2024-04-14T15:13:01.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.519+0000 7f57517fa700 10 client.25546.objecter ms_dispatch 0x7f5754001550 command_reply(tid 13: 0 ) 2024-04-14T15:13:01.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.519+0000 7f57517fa700 10 client.25546.objecter _finish_command 13 = system:0 2024-04-14T15:13:01.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.519+0000 7f57517fa700 20 client.25546.objecter put_session s=0x7f57340207b0 osd=6 4 2024-04-14T15:13:01.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.519+0000 7f57517fa700 15 client.25546.objecter _session_command_op_remove 6 13 2024-04-14T15:13:01.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:01.537 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:01.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:13:01.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.535+0000 7f575a236700 10 client.25546.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:13:01.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.535+0000 7f575a236700 15 client.25546.objecter _session_command_op_assign -1 14 2024-04-14T15:13:01.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.535+0000 7f575a236700 20 client.25546.objecter _get_session s=0x7f57340207b0 osd=6 3 2024-04-14T15:13:01.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.535+0000 7f575a236700 20 client.25546.objecter put_session s=0x7f57340207b0 osd=6 3 2024-04-14T15:13:01.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.535+0000 7f575a236700 20 client.25546.objecter _get_session s=0x7f57340207b0 osd=6 3 2024-04-14T15:13:01.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.535+0000 7f575a236700 15 client.25546.objecter _session_command_op_remove -1 14 2024-04-14T15:13:01.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.535+0000 7f575a236700 20 client.25546.objecter get_session s=0x7f57340207b0 osd=6 3 2024-04-14T15:13:01.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.535+0000 7f575a236700 15 client.25546.objecter _session_command_op_assign 6 14 2024-04-14T15:13:01.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.535+0000 7f575a236700 20 client.25546.objecter put_session s=0x7f57340207b0 osd=6 4 2024-04-14T15:13:01.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.535+0000 7f575a236700 10 client.25546.objecter _send_command 14 2024-04-14T15:13:01.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.535+0000 7f575a236700 1 -- 172.21.15.90:0/1965288322 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f5734021310 con 0x7f5734020ed0 2024-04-14T15:13:01.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.537+0000 7f57517fa700 1 -- 172.21.15.90:0/1965288322 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f5734021310 con 0x7f5734020ed0 2024-04-14T15:13:01.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.537+0000 7f57517fa700 10 client.25546.objecter ms_dispatch 0x7f5754001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:13:01.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.537+0000 7f57517fa700 10 client.25546.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:13:01.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.537+0000 7f57517fa700 20 client.25546.objecter put_session s=0x7f57340207b0 osd=6 4 2024-04-14T15:13:01.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.537+0000 7f57517fa700 15 client.25546.objecter _session_command_op_remove 6 14 2024-04-14T15:13:01.541 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:13:01.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.537+0000 7f575a236700 10 client.25546.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:13:01.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.537+0000 7f575a236700 15 client.25546.objecter _session_command_op_assign -1 15 2024-04-14T15:13:01.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.537+0000 7f575a236700 20 client.25546.objecter _get_session s=0x7f5734021310 osd=7 3 2024-04-14T15:13:01.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.537+0000 7f575a236700 20 client.25546.objecter put_session s=0x7f5734021310 osd=7 3 2024-04-14T15:13:01.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.537+0000 7f575a236700 20 client.25546.objecter _get_session s=0x7f5734021310 osd=7 3 2024-04-14T15:13:01.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.537+0000 7f575a236700 15 client.25546.objecter _session_command_op_remove -1 15 2024-04-14T15:13:01.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.537+0000 7f575a236700 20 client.25546.objecter get_session s=0x7f5734021310 osd=7 3 2024-04-14T15:13:01.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.537+0000 7f575a236700 15 client.25546.objecter _session_command_op_assign 7 15 2024-04-14T15:13:01.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.537+0000 7f575a236700 20 client.25546.objecter put_session s=0x7f5734021310 osd=7 4 2024-04-14T15:13:01.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.537+0000 7f575a236700 10 client.25546.objecter _send_command 15 2024-04-14T15:13:01.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.537+0000 7f575a236700 1 -- 172.21.15.90:0/1965288322 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f57340216f0 con 0x7f5734023ec0 2024-04-14T15:13:01.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.537+0000 7f57517fa700 10 client.25546.objecter ms_handle_connect 0x7f5734023ec0 2024-04-14T15:13:01.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.538+0000 7f57517fa700 1 -- 172.21.15.90:0/1965288322 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f57340216f0 con 0x7f5734023ec0 2024-04-14T15:13:01.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.538+0000 7f57517fa700 10 client.25546.objecter ms_dispatch 0x7f5754001550 command_reply(tid 15: 0 ) 2024-04-14T15:13:01.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.538+0000 7f57517fa700 10 client.25546.objecter _finish_command 15 = system:0 2024-04-14T15:13:01.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.538+0000 7f57517fa700 20 client.25546.objecter put_session s=0x7f5734021310 osd=7 4 2024-04-14T15:13:01.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.538+0000 7f57517fa700 15 client.25546.objecter _session_command_op_remove 7 15 2024-04-14T15:13:01.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.556+0000 7f575a236700 10 client.25546.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:13:01.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.556+0000 7f575a236700 15 client.25546.objecter _session_command_op_assign -1 16 2024-04-14T15:13:01.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.556+0000 7f575a236700 20 client.25546.objecter _get_session s=0x7f5734021310 osd=7 3 2024-04-14T15:13:01.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.556+0000 7f575a236700 20 client.25546.objecter put_session s=0x7f5734021310 osd=7 3 2024-04-14T15:13:01.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.556+0000 7f575a236700 20 client.25546.objecter _get_session s=0x7f5734021310 osd=7 3 2024-04-14T15:13:01.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.556+0000 7f575a236700 15 client.25546.objecter _session_command_op_remove -1 16 2024-04-14T15:13:01.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.556+0000 7f575a236700 20 client.25546.objecter get_session s=0x7f5734021310 osd=7 3 2024-04-14T15:13:01.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.556+0000 7f575a236700 15 client.25546.objecter _session_command_op_assign 7 16 2024-04-14T15:13:01.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.556+0000 7f575a236700 20 client.25546.objecter put_session s=0x7f5734021310 osd=7 4 2024-04-14T15:13:01.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.556+0000 7f575a236700 10 client.25546.objecter _send_command 16 2024-04-14T15:13:01.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.556+0000 7f575a236700 1 -- 172.21.15.90:0/1965288322 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f57340279d0 con 0x7f5734023ec0 2024-04-14T15:13:01.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.558+0000 7f57517fa700 1 -- 172.21.15.90:0/1965288322 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f57340279d0 con 0x7f5734023ec0 2024-04-14T15:13:01.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.558+0000 7f57517fa700 10 client.25546.objecter ms_dispatch 0x7f5754001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:13:01.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.558+0000 7f57517fa700 10 client.25546.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:13:01.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.558+0000 7f57517fa700 20 client.25546.objecter put_session s=0x7f5734021310 osd=7 4 2024-04-14T15:13:01.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.558+0000 7f57517fa700 15 client.25546.objecter _session_command_op_remove 7 16 2024-04-14T15:13:01.560 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:13:01.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.559+0000 7f575a236700 10 librados: watch_flush enter 2024-04-14T15:13:01.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.559+0000 7f575a236700 10 librados: watch_flush exit 2024-04-14T15:13:01.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.559+0000 7f575a236700 10 client.25546.objecter close_session for osd.0 2024-04-14T15:13:01.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.559+0000 7f575a236700 1 -- 172.21.15.90:0/1965288322 >> v1:172.21.15.90:6801/3761806885 conn(0x7f5734004760 legacy=0x7f57340027c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:01.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.559+0000 7f575a236700 20 client.25546.objecter put_session s=0x7f5734005410 osd=0 1 2024-04-14T15:13:01.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.559+0000 7f575a236700 10 client.25546.objecter close_session for osd.1 2024-04-14T15:13:01.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.559+0000 7f575a236700 1 -- 172.21.15.90:0/1965288322 >> v1:172.21.15.90:6805/4170361699 conn(0x7f5734004320 legacy=0x7f573400ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:01.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.559+0000 7f575a236700 20 client.25546.objecter put_session s=0x7f57340041b0 osd=1 1 2024-04-14T15:13:01.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.559+0000 7f575a236700 10 client.25546.objecter close_session for osd.2 2024-04-14T15:13:01.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.559+0000 7f575a236700 1 -- 172.21.15.90:0/1965288322 >> v1:172.21.15.90:6809/1686184251 conn(0x7f573400d710 legacy=0x7f573400fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:01.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.559+0000 7f5753fff700 1 -- 172.21.15.90:0/1965288322 reap_dead start 2024-04-14T15:13:01.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.559+0000 7f575a236700 20 client.25546.objecter put_session s=0x7f57340084b0 osd=2 1 2024-04-14T15:13:01.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.559+0000 7f575a236700 10 client.25546.objecter close_session for osd.3 2024-04-14T15:13:01.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.559+0000 7f575a236700 1 -- 172.21.15.90:0/1965288322 >> v1:172.21.15.90:6813/2270166774 conn(0x7f5734012550 legacy=0x7f5734014940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:01.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.559+0000 7f575a236700 20 client.25546.objecter put_session s=0x7f573400cd60 osd=3 1 2024-04-14T15:13:01.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.559+0000 7f575a236700 10 client.25546.objecter close_session for osd.4 2024-04-14T15:13:01.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.559+0000 7f575a236700 1 -- 172.21.15.90:0/1965288322 >> v1:172.21.15.92:6800/3887549733 conn(0x7f57340120b0 legacy=0x7f57340193a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:01.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.559+0000 7f575a236700 20 client.25546.objecter put_session s=0x7f573400c8a0 osd=4 1 2024-04-14T15:13:01.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.559+0000 7f575a236700 10 client.25546.objecter close_session for osd.5 2024-04-14T15:13:01.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.559+0000 7f575a236700 1 -- 172.21.15.90:0/1965288322 >> v1:172.21.15.92:6804/3244561440 conn(0x7f573401b140 legacy=0x7f573401d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:01.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.559+0000 7f575a236700 20 client.25546.objecter put_session s=0x7f573401aac0 osd=5 1 2024-04-14T15:13:01.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.559+0000 7f575a236700 10 client.25546.objecter close_session for osd.6 2024-04-14T15:13:01.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.559+0000 7f575a236700 1 -- 172.21.15.90:0/1965288322 >> v1:172.21.15.92:6808/636125963 conn(0x7f5734020ed0 legacy=0x7f573401efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:01.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.559+0000 7f5753fff700 1 -- 172.21.15.90:0/1965288322 reap_dead start 2024-04-14T15:13:01.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.559+0000 7f575a236700 20 client.25546.objecter put_session s=0x7f57340207b0 osd=6 1 2024-04-14T15:13:01.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.559+0000 7f575a236700 10 client.25546.objecter close_session for osd.7 2024-04-14T15:13:01.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.559+0000 7f575a236700 1 -- 172.21.15.90:0/1965288322 >> v1:172.21.15.92:6812/1773706716 conn(0x7f5734023ec0 legacy=0x7f57340262b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:01.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.559+0000 7f575a236700 20 client.25546.objecter put_session s=0x7f5734021310 osd=7 1 2024-04-14T15:13:01.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.559+0000 7f575a236700 20 client.25546.objecter shutdown clearing up homeless session... 2024-04-14T15:13:01.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.559+0000 7f575a236700 10 client.25546.objecter successfully canceled tick 2024-04-14T15:13:01.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.559+0000 7f5753fff700 1 -- 172.21.15.90:0/1965288322 reap_dead start 2024-04-14T15:13:01.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.559+0000 7f575a236700 1 -- 172.21.15.90:0/1965288322 >> v1:172.21.15.90:6800/420935427 conn(0x7f57400713b0 legacy=0x7f5740073870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:01.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.559+0000 7f575a236700 1 -- 172.21.15.90:0/1965288322 >> v1:172.21.15.92:6789/0 conn(0x7f575410f300 legacy=0x7f57541a8370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:01.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.559+0000 7f575a236700 1 -- 172.21.15.90:0/1965288322 shutdown_connections 2024-04-14T15:13:01.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.559+0000 7f575a236700 1 -- 172.21.15.90:0/1965288322 >> 172.21.15.90:0/1965288322 conn(0x7f57540fe3d0 msgr2=0x7f5754112070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:13:01.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.559+0000 7f575a236700 1 -- 172.21.15.90:0/1965288322 shutdown_connections 2024-04-14T15:13:01.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.559+0000 7f575a236700 1 -- 172.21.15.90:0/1965288322 wait complete. 2024-04-14T15:13:01.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:01.559+0000 7f575a236700 1 librados: shutdown 2024-04-14T15:13:01.566 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:13:01.566 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:13:01.566 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:13:01.566 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:13:01.566 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:13:01.566 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:13:01.567 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:13:01.567 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:13:01.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:01 smithi092 ceph-mon[33966]: 16.2 deep-scrub starts 2024-04-14T15:13:01.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:01 smithi092 ceph-mon[33966]: 16.2 deep-scrub ok 2024-04-14T15:13:01.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:01 smithi092 ceph-mon[33966]: 16.3 deep-scrub starts 2024-04-14T15:13:01.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:01 smithi092 ceph-mon[33966]: 16.3 deep-scrub ok 2024-04-14T15:13:01.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:01 smithi092 ceph-mon[33966]: 16.9 deep-scrub starts 2024-04-14T15:13:01.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:01 smithi092 ceph-mon[33966]: 16.9 deep-scrub ok 2024-04-14T15:13:01.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:01 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:01.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:01 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/309942895' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi090-93998-1"}]': finished 2024-04-14T15:13:01.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:01 smithi092 ceph-mon[33966]: from='client.25001 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi090-94662-1"}]': finished 2024-04-14T15:13:01.625 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:01 smithi092 ceph-mon[33966]: from='client.25207 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi090-93363-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:01.625 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:01 smithi092 ceph-mon[33966]: from='client.25114 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-4", "tierpool": "test-rados-api-smithi090-93652-4-cache"}]': finished 2024-04-14T15:13:01.625 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:01 smithi092 ceph-mon[33966]: from='client.25489 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi090-94522-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:13:01.625 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:01 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/532832389' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi090-93363-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-04-14T15:13:01.625 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:01 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1327720256' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-4", "overlaypool": "test-rados-api-smithi090-93652-4-cache"}]: dispatch 2024-04-14T15:13:01.625 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:01 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/340734729' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi090-94522-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi090-94522-2"}]: dispatch 2024-04-14T15:13:01.625 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:01 smithi092 ceph-mon[33966]: osdmap e113: 8 total, 8 up, 8 in 2024-04-14T15:13:01.626 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:01 smithi092 ceph-mon[33966]: from='client.25207 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi090-93363-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-04-14T15:13:01.626 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:01 smithi092 ceph-mon[33966]: from='client.25114 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-4", "overlaypool": "test-rados-api-smithi090-93652-4-cache"}]: dispatch 2024-04-14T15:13:01.626 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:01 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2676581337' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:01.626 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:01 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3158227083' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi090-93342-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:01.626 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:01 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3022029644' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:01.626 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:01 smithi092 ceph-mon[33966]: from='client.25489 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi090-94522-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi090-94522-2"}]: dispatch 2024-04-14T15:13:01.627 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:01 smithi092 ceph-mon[33966]: from='client.25361 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:01.627 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:01 smithi092 ceph-mon[33966]: from='client.25516 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi090-93342-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:01.627 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:01 smithi092 ceph-mon[33966]: from='client.25504 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:01.627 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:01 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1715119515' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi090-94662-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:13:01.627 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:01 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/4159084683' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-04-14T15:13:01.628 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:01 smithi092 ceph-mon[33966]: threexx 2024-04-14T15:13:01.628 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:01 smithi092 ceph-mon[33966]: from='client.25252 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-04-14T15:13:01.639 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:13:01.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:01.666 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:01.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:13:01.767 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:13:01.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:01.793 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:01.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:13:01.895 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:13:01.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:01.929 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:01.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:13:02.031 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:13:02.057 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:02.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:02.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:13:02.159 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:13:02.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:02.185 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:02.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:13:02.286 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:13:02.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:02.312 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:02.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:13:02.414 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:13:02.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:02.439 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:02.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:13:02.540 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:13:02.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:02.564 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:02.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:13:02.572 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[30399]: 16.1 deep-scrub starts 2024-04-14T15:13:02.572 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[30399]: 16.1 deep-scrub ok 2024-04-14T15:13:02.572 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[30399]: pgmap v116: 836 pgs: 384 unknown, 64 creating+peering, 388 active+clean; 25 MiB data, 282 MiB used, 715 GiB / 715 GiB avail; 4.1 KiB/s rd, 6.2 MiB/s wr, 555 op/s 2024-04-14T15:13:02.572 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[30399]: from='client.25252 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-04-14T15:13:02.572 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/4159084683' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-04-14T15:13:02.572 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[30399]: fourxx 2024-04-14T15:13:02.572 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[30399]: from='client.25252 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-04-14T15:13:02.572 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[30399]: 16.4 deep-scrub starts 2024-04-14T15:13:02.572 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[30399]: 16.4 deep-scrub ok 2024-04-14T15:13:02.572 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1965288322' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:13:02.573 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:02.573 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2859819370' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-04-14T15:13:02.573 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/274362209' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi090-93436-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi090-93436-10"}]': finished 2024-04-14T15:13:02.573 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[30399]: from='client.25217 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi090-93476-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi090-93476-10"}]': finished 2024-04-14T15:13:02.573 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[30399]: from='client.25207 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi090-93363-3", "field": "max_bytes", "val": "4096"}]': finished 2024-04-14T15:13:02.573 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[30399]: from='client.25114 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-4", "overlaypool": "test-rados-api-smithi090-93652-4-cache"}]': finished 2024-04-14T15:13:02.573 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[30399]: from='client.25361 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:02.573 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[30399]: from='client.25516 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi090-93342-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:02.573 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[30399]: from='client.25504 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:02.573 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1715119515' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi090-94662-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:13:02.573 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1327720256' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-4-cache", "mode": "writeback"}]: dispatch 2024-04-14T15:13:02.573 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[30399]: osdmap e114: 8 total, 8 up, 8 in 2024-04-14T15:13:02.573 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3684793142' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi090-93582-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:02.574 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2105945719' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi090-93998-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:02.574 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3178004355' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi090-93379-16"}]: dispatch 2024-04-14T15:13:02.574 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3274182089' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi090-93731-7"}]: dispatch 2024-04-14T15:13:02.574 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1324117929' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi090-93864-7"}]: dispatch 2024-04-14T15:13:02.574 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/4291843131' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi090-93398-23"}]: dispatch 2024-04-14T15:13:02.574 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1715119515' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi090-94662-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi090-94662-2"}]: dispatch 2024-04-14T15:13:02.574 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[30399]: from='client.25114 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-4-cache", "mode": "writeback"}]: dispatch 2024-04-14T15:13:02.574 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[30399]: from='client.25379 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi090-93582-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:02.574 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[30399]: from='client.25240 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi090-93379-16"}]: dispatch 2024-04-14T15:13:02.574 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[30399]: from='client.25376 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi090-93998-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:02.574 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[30399]: from='client.25249 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi090-93731-7"}]: dispatch 2024-04-14T15:13:02.574 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[30399]: from='client.25267 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi090-93864-7"}]: dispatch 2024-04-14T15:13:02.574 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[30399]: from='client.25243 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi090-93398-23"}]: dispatch 2024-04-14T15:13:02.575 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[41670]: 16.1 deep-scrub starts 2024-04-14T15:13:02.575 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[41670]: 16.1 deep-scrub ok 2024-04-14T15:13:02.575 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[41670]: pgmap v116: 836 pgs: 384 unknown, 64 creating+peering, 388 active+clean; 25 MiB data, 282 MiB used, 715 GiB / 715 GiB avail; 4.1 KiB/s rd, 6.2 MiB/s wr, 555 op/s 2024-04-14T15:13:02.575 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[41670]: from='client.25252 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-04-14T15:13:02.575 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/4159084683' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-04-14T15:13:02.575 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[41670]: fourxx 2024-04-14T15:13:02.575 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[41670]: from='client.25252 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-04-14T15:13:02.575 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[41670]: 16.4 deep-scrub starts 2024-04-14T15:13:02.575 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[41670]: 16.4 deep-scrub ok 2024-04-14T15:13:02.575 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1965288322' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:13:02.576 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:02.576 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2859819370' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-04-14T15:13:02.576 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/274362209' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi090-93436-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi090-93436-10"}]': finished 2024-04-14T15:13:02.576 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[41670]: from='client.25217 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi090-93476-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi090-93476-10"}]': finished 2024-04-14T15:13:02.576 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[41670]: from='client.25207 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi090-93363-3", "field": "max_bytes", "val": "4096"}]': finished 2024-04-14T15:13:02.576 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[41670]: from='client.25114 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-4", "overlaypool": "test-rados-api-smithi090-93652-4-cache"}]': finished 2024-04-14T15:13:02.576 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[41670]: from='client.25361 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:02.576 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[41670]: from='client.25516 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi090-93342-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:02.576 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[41670]: from='client.25504 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:02.576 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1715119515' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi090-94662-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:13:02.576 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1327720256' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-4-cache", "mode": "writeback"}]: dispatch 2024-04-14T15:13:02.576 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[41670]: osdmap e114: 8 total, 8 up, 8 in 2024-04-14T15:13:02.576 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3684793142' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi090-93582-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:02.576 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2105945719' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi090-93998-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:02.577 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3178004355' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi090-93379-16"}]: dispatch 2024-04-14T15:13:02.577 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3274182089' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi090-93731-7"}]: dispatch 2024-04-14T15:13:02.577 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1324117929' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi090-93864-7"}]: dispatch 2024-04-14T15:13:02.577 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/4291843131' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi090-93398-23"}]: dispatch 2024-04-14T15:13:02.577 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1715119515' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi090-94662-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi090-94662-2"}]: dispatch 2024-04-14T15:13:02.577 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[41670]: from='client.25114 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-4-cache", "mode": "writeback"}]: dispatch 2024-04-14T15:13:02.577 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[41670]: from='client.25379 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi090-93582-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:02.577 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[41670]: from='client.25240 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi090-93379-16"}]: dispatch 2024-04-14T15:13:02.577 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[41670]: from='client.25376 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi090-93998-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:02.577 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[41670]: from='client.25249 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi090-93731-7"}]: dispatch 2024-04-14T15:13:02.577 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[41670]: from='client.25267 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi090-93864-7"}]: dispatch 2024-04-14T15:13:02.577 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:02 smithi090 ceph-mon[41670]: from='client.25243 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi090-93398-23"}]: dispatch 2024-04-14T15:13:02.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:02 smithi092 ceph-mon[33966]: 16.1 deep-scrub starts 2024-04-14T15:13:02.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:02 smithi092 ceph-mon[33966]: 16.1 deep-scrub ok 2024-04-14T15:13:02.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:02 smithi092 ceph-mon[33966]: pgmap v116: 836 pgs: 384 unknown, 64 creating+peering, 388 active+clean; 25 MiB data, 282 MiB used, 715 GiB / 715 GiB avail; 4.1 KiB/s rd, 6.2 MiB/s wr, 555 op/s 2024-04-14T15:13:02.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:02 smithi092 ceph-mon[33966]: from='client.25252 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-04-14T15:13:02.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:02 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/4159084683' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-04-14T15:13:02.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:02 smithi092 ceph-mon[33966]: fourxx 2024-04-14T15:13:02.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:02 smithi092 ceph-mon[33966]: from='client.25252 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-04-14T15:13:02.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:02 smithi092 ceph-mon[33966]: 16.4 deep-scrub starts 2024-04-14T15:13:02.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:02 smithi092 ceph-mon[33966]: 16.4 deep-scrub ok 2024-04-14T15:13:02.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:02 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1965288322' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:13:02.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:02 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:02.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:02 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2859819370' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-04-14T15:13:02.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:02 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/274362209' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi090-93436-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi090-93436-10"}]': finished 2024-04-14T15:13:02.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:02 smithi092 ceph-mon[33966]: from='client.25217 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi090-93476-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi090-93476-10"}]': finished 2024-04-14T15:13:02.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:02 smithi092 ceph-mon[33966]: from='client.25207 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi090-93363-3", "field": "max_bytes", "val": "4096"}]': finished 2024-04-14T15:13:02.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:02 smithi092 ceph-mon[33966]: from='client.25114 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-4", "overlaypool": "test-rados-api-smithi090-93652-4-cache"}]': finished 2024-04-14T15:13:02.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:02 smithi092 ceph-mon[33966]: from='client.25361 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:02.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:02 smithi092 ceph-mon[33966]: from='client.25516 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi090-93342-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:02.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:02 smithi092 ceph-mon[33966]: from='client.25504 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:02.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:02 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1715119515' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi090-94662-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:13:02.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:02 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1327720256' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-4-cache", "mode": "writeback"}]: dispatch 2024-04-14T15:13:02.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:02 smithi092 ceph-mon[33966]: osdmap e114: 8 total, 8 up, 8 in 2024-04-14T15:13:02.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:02 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3684793142' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi090-93582-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:02.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:02 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2105945719' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi090-93998-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:02.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:02 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3178004355' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi090-93379-16"}]: dispatch 2024-04-14T15:13:02.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:02 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3274182089' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi090-93731-7"}]: dispatch 2024-04-14T15:13:02.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:02 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1324117929' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi090-93864-7"}]: dispatch 2024-04-14T15:13:02.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:02 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/4291843131' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi090-93398-23"}]: dispatch 2024-04-14T15:13:02.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:02 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1715119515' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi090-94662-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi090-94662-2"}]: dispatch 2024-04-14T15:13:02.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:02 smithi092 ceph-mon[33966]: from='client.25114 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-4-cache", "mode": "writeback"}]: dispatch 2024-04-14T15:13:02.625 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:02 smithi092 ceph-mon[33966]: from='client.25379 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi090-93582-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:02.625 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:02 smithi092 ceph-mon[33966]: from='client.25240 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi090-93379-16"}]: dispatch 2024-04-14T15:13:02.625 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:02 smithi092 ceph-mon[33966]: from='client.25376 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi090-93998-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:02.625 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:02 smithi092 ceph-mon[33966]: from='client.25249 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi090-93731-7"}]: dispatch 2024-04-14T15:13:02.625 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:02 smithi092 ceph-mon[33966]: from='client.25267 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi090-93864-7"}]: dispatch 2024-04-14T15:13:02.625 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:02 smithi092 ceph-mon[33966]: from='client.25243 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi090-93398-23"}]: dispatch 2024-04-14T15:13:02.665 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:13:02.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:02.690 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:02.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:13:02.792 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:13:02.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:02.829 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:02.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:13:02.831 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024- list: Running main() from gmock_main.cc 2024-04-14T15:13:02.831 INFO:tasks.workunit.client.0.smithi090.stdout: list: [==========] Running 3 tests from 1 test suite. 2024-04-14T15:13:02.831 INFO:tasks.workunit.client.0.smithi090.stdout: list: [----------] Global test environment set-up. 2024-04-14T15:13:02.831 INFO:tasks.workunit.client.0.smithi090.stdout: list: [----------] 3 tests from NeoradosList 2024-04-14T15:13:02.831 INFO:tasks.workunit.client.0.smithi090.stdout: list: [ RUN ] NeoradosList.ListObjects 2024-04-14T15:13:02.831 INFO:tasks.workunit.client.0.smithi090.stdout: list: [ OK ] NeoradosList.ListObjects (2529 ms) 2024-04-14T15:13:02.831 INFO:tasks.workunit.client.0.smithi090.stdout: list: [ RUN ] NeoradosList.ListObjectsNS 2024-04-14T15:13:02.831 INFO:tasks.workunit.client.0.smithi090.stdout: list: [ OK ] NeoradosList.ListObjectsNS (3031 ms) 2024-04-14T15:13:02.831 INFO:tasks.workunit.client.0.smithi090.stdout: list: [ RUN ] NeoradosList.ListObjectsMany 2024-04-14T15:13:02.831 INFO:tasks.workunit.client.0.smithi090.stdout: list: [ OK ] NeoradosList.ListObjectsMany (2720 ms) 2024-04-14T15:13:02.831 INFO:tasks.workunit.client.0.smithi090.stdout: list: [----------] 3 tests from NeoradosList (8280 ms total) 2024-04-14T15:13:02.831 INFO:tasks.workunit.client.0.smithi090.stdout: list: 2024-04-14T15:13:02.832 INFO:tasks.workunit.client.0.smithi090.stdout: list: [----------] Global test environment tear-down 2024-04-14T15:13:02.832 INFO:tasks.workunit.client.0.smithi090.stdout: list: [==========] 3 tests from 1 test suite ran. (8280 ms total) 2024-04-14T15:13:02.832 INFO:tasks.workunit.client.0.smithi090.stdout: list: [ PASSED ] 3 tests. 2024-04-14T15:13:02.929 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:13:02.979 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:02.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:02.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:13:03.081 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:13:03.105 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:03.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:03.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:13:03.207 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:13:03.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:03.232 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:03.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:13:03.333 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:13:03.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:03.357 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:03.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:13:03.458 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:13:03.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:03.482 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:03.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:13:03.570 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:03 smithi090 ceph-mon[41670]: from='client.25252 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-04-14T15:13:03.570 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:03 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:03.570 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:03 smithi090 ceph-mon[41670]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:13:03.570 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:03 smithi090 ceph-mon[41670]: from='client.25489 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReadOpsmithi090-94522-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi090-94522-2"}]': finished 2024-04-14T15:13:03.570 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:03 smithi090 ceph-mon[41670]: from='client.25114 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-4-cache", "mode": "writeback"}]': finished 2024-04-14T15:13:03.570 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:03 smithi090 ceph-mon[41670]: from='client.25379 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi090-93582-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:03.570 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:03 smithi090 ceph-mon[41670]: from='client.25240 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi090-93379-16"}]': finished 2024-04-14T15:13:03.570 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:03 smithi090 ceph-mon[41670]: from='client.25376 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi090-93998-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:03.571 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:03 smithi090 ceph-mon[41670]: from='client.25249 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi090-93731-7"}]': finished 2024-04-14T15:13:03.571 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:03 smithi090 ceph-mon[41670]: from='client.25267 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi090-93864-7"}]': finished 2024-04-14T15:13:03.571 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:03 smithi090 ceph-mon[41670]: from='client.25243 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi090-93398-23"}]': finished 2024-04-14T15:13:03.571 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:03 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1324117929' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi090-93864-7"}]: dispatch 2024-04-14T15:13:03.571 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:03 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3274182089' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi090-93731-7"}]: dispatch 2024-04-14T15:13:03.571 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:03 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/4291843131' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi090-93398-23"}]: dispatch 2024-04-14T15:13:03.571 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:03 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3178004355' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi090-93379-16"}]: dispatch 2024-04-14T15:13:03.571 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:03 smithi090 ceph-mon[41670]: osdmap e115: 8 total, 8 up, 8 in 2024-04-14T15:13:03.571 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:03 smithi090 ceph-mon[41670]: from='client.25267 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi090-93864-7"}]: dispatch 2024-04-14T15:13:03.571 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:03 smithi090 ceph-mon[41670]: from='client.25249 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi090-93731-7"}]: dispatch 2024-04-14T15:13:03.571 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:03 smithi090 ceph-mon[41670]: from='client.25243 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi090-93398-23"}]: dispatch 2024-04-14T15:13:03.571 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:03 smithi090 ceph-mon[41670]: from='client.25240 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi090-93379-16"}]: dispatch 2024-04-14T15:13:03.571 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:03 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2105945719' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi090-93998-6", "pg_num": 4}]: dispatch 2024-04-14T15:13:03.571 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:03 smithi090 ceph-mon[41670]: from='client.25376 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi090-93998-6", "pg_num": 4}]: dispatch 2024-04-14T15:13:03.571 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:03 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3684793142' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-04-14T15:13:03.572 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:03 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1327720256' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-4"}]: dispatch 2024-04-14T15:13:03.572 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:03 smithi090 ceph-mon[41670]: from='client.25114 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-4"}]: dispatch 2024-04-14T15:13:03.572 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:03 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3684793142' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi090-93582-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:03.572 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:03 smithi090 ceph-mon[41670]: from='client.25379 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi090-93582-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:03.572 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:03 smithi090 ceph-mon[30399]: from='client.25252 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-04-14T15:13:03.572 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:03 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:03.572 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:03 smithi090 ceph-mon[30399]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:13:03.572 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:03 smithi090 ceph-mon[30399]: from='client.25489 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReadOpsmithi090-94522-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi090-94522-2"}]': finished 2024-04-14T15:13:03.572 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:03 smithi090 ceph-mon[30399]: from='client.25114 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-4-cache", "mode": "writeback"}]': finished 2024-04-14T15:13:03.572 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:03 smithi090 ceph-mon[30399]: from='client.25379 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi090-93582-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:03.573 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:03 smithi090 ceph-mon[30399]: from='client.25240 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi090-93379-16"}]': finished 2024-04-14T15:13:03.573 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:03 smithi090 ceph-mon[30399]: from='client.25376 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi090-93998-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:03.573 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:03 smithi090 ceph-mon[30399]: from='client.25249 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi090-93731-7"}]': finished 2024-04-14T15:13:03.573 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:03 smithi090 ceph-mon[30399]: from='client.25267 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi090-93864-7"}]': finished 2024-04-14T15:13:03.573 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:03 smithi090 ceph-mon[30399]: from='client.25243 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi090-93398-23"}]': finished 2024-04-14T15:13:03.573 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:03 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1324117929' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi090-93864-7"}]: dispatch 2024-04-14T15:13:03.573 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:03 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3274182089' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi090-93731-7"}]: dispatch 2024-04-14T15:13:03.573 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:03 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/4291843131' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi090-93398-23"}]: dispatch 2024-04-14T15:13:03.573 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:03 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3178004355' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi090-93379-16"}]: dispatch 2024-04-14T15:13:03.573 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:03 smithi090 ceph-mon[30399]: osdmap e115: 8 total, 8 up, 8 in 2024-04-14T15:13:03.573 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:03 smithi090 ceph-mon[30399]: from='client.25267 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi090-93864-7"}]: dispatch 2024-04-14T15:13:03.573 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:03 smithi090 ceph-mon[30399]: from='client.25249 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi090-93731-7"}]: dispatch 2024-04-14T15:13:03.573 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:03 smithi090 ceph-mon[30399]: from='client.25243 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi090-93398-23"}]: dispatch 2024-04-14T15:13:03.573 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:03 smithi090 ceph-mon[30399]: from='client.25240 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi090-93379-16"}]: dispatch 2024-04-14T15:13:03.573 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:03 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2105945719' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi090-93998-6", "pg_num": 4}]: dispatch 2024-04-14T15:13:03.574 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:03 smithi090 ceph-mon[30399]: from='client.25376 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi090-93998-6", "pg_num": 4}]: dispatch 2024-04-14T15:13:03.574 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:03 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3684793142' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-04-14T15:13:03.574 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:03 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1327720256' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-4"}]: dispatch 2024-04-14T15:13:03.574 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:03 smithi090 ceph-mon[30399]: from='client.25114 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-4"}]: dispatch 2024-04-14T15:13:03.574 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:03 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3684793142' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi090-93582-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:03.574 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:03 smithi090 ceph-mon[30399]: from='client.25379 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi090-93582-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:03.583 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:13:03.609 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:03.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:03.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:13:03.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:03 smithi092 ceph-mon[33966]: from='client.25252 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-04-14T15:13:03.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:03 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:03.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:03 smithi092 ceph-mon[33966]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:13:03.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:03 smithi092 ceph-mon[33966]: from='client.25489 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReadOpsmithi090-94522-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi090-94522-2"}]': finished 2024-04-14T15:13:03.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:03 smithi092 ceph-mon[33966]: from='client.25114 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-4-cache", "mode": "writeback"}]': finished 2024-04-14T15:13:03.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:03 smithi092 ceph-mon[33966]: from='client.25379 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi090-93582-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:03.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:03 smithi092 ceph-mon[33966]: from='client.25240 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi090-93379-16"}]': finished 2024-04-14T15:13:03.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:03 smithi092 ceph-mon[33966]: from='client.25376 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi090-93998-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:03.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:03 smithi092 ceph-mon[33966]: from='client.25249 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi090-93731-7"}]': finished 2024-04-14T15:13:03.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:03 smithi092 ceph-mon[33966]: from='client.25267 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi090-93864-7"}]': finished 2024-04-14T15:13:03.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:03 smithi092 ceph-mon[33966]: from='client.25243 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi090-93398-23"}]': finished 2024-04-14T15:13:03.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:03 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1324117929' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi090-93864-7"}]: dispatch 2024-04-14T15:13:03.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:03 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3274182089' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi090-93731-7"}]: dispatch 2024-04-14T15:13:03.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:03 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/4291843131' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi090-93398-23"}]: dispatch 2024-04-14T15:13:03.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:03 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3178004355' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi090-93379-16"}]: dispatch 2024-04-14T15:13:03.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:03 smithi092 ceph-mon[33966]: osdmap e115: 8 total, 8 up, 8 in 2024-04-14T15:13:03.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:03 smithi092 ceph-mon[33966]: from='client.25267 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi090-93864-7"}]: dispatch 2024-04-14T15:13:03.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:03 smithi092 ceph-mon[33966]: from='client.25249 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi090-93731-7"}]: dispatch 2024-04-14T15:13:03.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:03 smithi092 ceph-mon[33966]: from='client.25243 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi090-93398-23"}]: dispatch 2024-04-14T15:13:03.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:03 smithi092 ceph-mon[33966]: from='client.25240 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi090-93379-16"}]: dispatch 2024-04-14T15:13:03.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:03 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2105945719' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi090-93998-6", "pg_num": 4}]: dispatch 2024-04-14T15:13:03.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:03 smithi092 ceph-mon[33966]: from='client.25376 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi090-93998-6", "pg_num": 4}]: dispatch 2024-04-14T15:13:03.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:03 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3684793142' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-04-14T15:13:03.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:03 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1327720256' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-4"}]: dispatch 2024-04-14T15:13:03.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:03 smithi092 ceph-mon[33966]: from='client.25114 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-4"}]: dispatch 2024-04-14T15:13:03.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:03 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3684793142' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi090-93582-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:03.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:03 smithi092 ceph-mon[33966]: from='client.25379 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi090-93582-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:03.644 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: [==========] Running 4 tests from 1 test suite. 2024-04-14T15:13:03.644 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: [----------] Global test environment set-up. 2024-04-14T15:13:03.644 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: [----------] 4 tests from LibRadosService 2024-04-14T15:13:03.644 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2024-04-14T15:13:03.644 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5061 ms) 2024-04-14T15:13:03.644 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2024-04-14T15:13:03.645 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: [ OK ] LibRadosService.RegisterLate (20 ms) 2024-04-14T15:13:03.645 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2024-04-14T15:13:03.645 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: cluster: 2024-04-14T15:13:03.645 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: id: 0e288098-fa70-11ee-bc8e-c7b262605968 2024-04-14T15:13:03.645 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: health: HEALTH_WARN 2024-04-14T15:13:03.645 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: noscrub,nodeep-scrub flag(s) set 2024-04-14T15:13:03.645 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: 18 pool(s) do not have an application enabled 2024-04-14T15:13:03.645 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: 2024-04-14T15:13:03.645 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: services: 2024-04-14T15:13:03.645 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 7m) 2024-04-14T15:13:03.645 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: mgr: y(active, since 2m), standbys: x 2024-04-14T15:13:03.645 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: osd: 8 osds: 8 up (since 3m), 8 in (since 3m) 2024-04-14T15:13:03.645 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: flags noscrub,nodeep-scrub 2024-04-14T15:13:03.645 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: laundry: 2 daemons active (1 hosts) 2024-04-14T15:13:03.645 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-04-14T15:13:03.646 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: 2024-04-14T15:13:03.646 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: data: 2024-04-14T15:13:03.646 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: pools: 33 pools, 948 pgs 2024-04-14T15:13:03.646 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: objects: 269 objects, 25 MiB 2024-04-14T15:13:03.646 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: usage: 282 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:13:03.646 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: pgs: 43.882% pgs not active 2024-04-14T15:13:03.646 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: 532 active+clean 2024-04-14T15:13:03.646 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: 416 creating+peering 2024-04-14T15:13:03.646 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: 2024-04-14T15:13:03.646 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: io: 2024-04-14T15:13:03.646 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: client: 9.7 KiB/s rd, 6.0 MiB/s wr, 265 op/s rd, 296 op/s wr 2024-04-14T15:13:03.646 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: 2024-04-14T15:13:03.646 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: 2024-04-14T15:13:03.646 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: cluster: 2024-04-14T15:13:03.647 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: id: 0e288098-fa70-11ee-bc8e-c7b262605968 2024-04-14T15:13:03.647 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: health: HEALTH_WARN 2024-04-14T15:13:03.647 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: noscrub,nodeep-scrub flag(s) set 2024-04-14T15:13:03.647 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: 18 pool(s) do not have an application enabled 2024-04-14T15:13:03.647 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: 2024-04-14T15:13:03.647 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: services: 2024-04-14T15:13:03.647 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 7m) 2024-04-14T15:13:03.647 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: mgr: y(active, since 2m), standbys: x 2024-04-14T15:13:03.647 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: osd: 8 osds: 8 up (since 3m), 8 in (since 3m) 2024-04-14T15:13:03.647 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: flags noscrub,nodeep-scrub 2024-04-14T15:13:03.647 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: laundry: 2 daemons active (1 hosts) 2024-04-14T15:13:03.647 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-04-14T15:13:03.647 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: 2024-04-14T15:13:03.647 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: data: 2024-04-14T15:13:03.647 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: pools: 33 pools, 948 pgs 2024-04-14T15:13:03.648 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: objects: 269 objects, 25 MiB 2024-04-14T15:13:03.648 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: usage: 282 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:13:03.648 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: pgs: 43.882% pgs not active 2024-04-14T15:13:03.648 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: 532 active+clean 2024-04-14T15:13:03.648 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: 416 creating+peering 2024-04-14T15:13:03.648 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: 2024-04-14T15:13:03.648 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: io: 2024-04-14T15:13:03.648 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: client: 9.7 KiB/s rd, 6.0 MiB/s wr, 265 op/s rd, 296 op/s wr 2024-04-14T15:13:03.648 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: 2024-04-14T15:13:03.648 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: 2024-04-14T15:13:03.648 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: cluster: 2024-04-14T15:13:03.648 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: id: 0e288098-fa70-11ee-bc8e-c7b262605968 2024-04-14T15:13:03.648 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: health: HEALTH_WARN 2024-04-14T15:13:03.648 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: 1 cache pools are missing hit_sets 2024-04-14T15:13:03.648 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: noscrub,nodeep-scrub flag(s) set 2024-04-14T15:13:03.649 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: 15 pool(s) do not have an application enabled 2024-04-14T15:13:03.649 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: 2024-04-14T15:13:03.649 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: services: 2024-04-14T15:13:03.649 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 7m) 2024-04-14T15:13:03.710 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:13:03.733 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:03.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:03.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:13:03.819 INFO:tasks.workunit.client.0.smithi090.stdout: api_ api_stat: Running main() from gmock_main.cc 2024-04-14T15:13:03.819 INFO:tasks.workunit.client.0.smithi090.stdout: api_stat: [==========] Running 9 tests from 2 test suites. 2024-04-14T15:13:03.819 INFO:tasks.workunit.client.0.smithi090.stdout: api_stat: [----------] Global test environment set-up. 2024-04-14T15:13:03.819 INFO:tasks.workunit.client.0.smithi090.stdout: api_stat: [----------] 5 tests from LibRadosStat 2024-04-14T15:13:03.819 INFO:tasks.workunit.client.0.smithi090.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2024-04-14T15:13:03.819 INFO:tasks.workunit.client.0.smithi090.stdout: api_stat: [ OK ] LibRadosStat.Stat (457 ms) 2024-04-14T15:13:03.819 INFO:tasks.workunit.client.0.smithi090.stdout: api_stat: [ RUN ] LibRadosStat.Stat2 2024-04-14T15:13:03.819 INFO:tasks.workunit.client.0.smithi090.stdout: api_stat: [ OK ] LibRadosStat.Stat2 (14 ms) 2024-04-14T15:13:03.819 INFO:tasks.workunit.client.0.smithi090.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2024-04-14T15:13:03.819 INFO:tasks.workunit.client.0.smithi090.stdout: api_stat: [ OK ] LibRadosStat.StatNS (44 ms) 2024-04-14T15:13:03.820 INFO:tasks.workunit.client.0.smithi090.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2024-04-14T15:13:03.820 INFO:tasks.workunit.client.0.smithi090.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (1 ms) 2024-04-14T15:13:03.820 INFO:tasks.workunit.client.0.smithi090.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2024-04-14T15:13:03.820 INFO:tasks.workunit.client.0.smithi090.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (4 ms) 2024-04-14T15:13:03.820 INFO:tasks.workunit.client.0.smithi090.stdout: api_stat: [----------] 5 tests from LibRadosStat (520 ms total) 2024-04-14T15:13:03.820 INFO:tasks.workunit.client.0.smithi090.stdout: api_stat: 2024-04-14T15:13:03.820 INFO:tasks.workunit.client.0.smithi090.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2024-04-14T15:13:03.820 INFO:tasks.workunit.client.0.smithi090.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2024-04-14T15:13:03.820 INFO:tasks.workunit.client.0.smithi090.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (855 ms) 2024-04-14T15:13:03.820 INFO:tasks.workunit.client.0.smithi090.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2024-04-14T15:13:03.820 INFO:tasks.workunit.client.0.smithi090.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (14 ms) 2024-04-14T15:13:03.820 INFO:tasks.workunit.client.0.smithi090.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2024-04-14T15:13:03.820 INFO:tasks.workunit.client.0.smithi090.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (0 ms) 2024-04-14T15:13:03.821 INFO:tasks.workunit.client.0.smithi090.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2024-04-14T15:13:03.821 INFO:tasks.workunit.client.0.smithi090.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (3 ms) 2024-04-14T15:13:03.821 INFO:tasks.workunit.client.0.smithi090.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (872 ms total) 2024-04-14T15:13:03.821 INFO:tasks.workunit.client.0.smithi090.stdout: api_stat: 2024-04-14T15:13:03.821 INFO:tasks.workunit.client.0.smithi090.stdout: api_stat: [----------] Global test environment tear-down 2024-04-14T15:13:03.821 INFO:tasks.workunit.client.0.smithi090.stdout: api_stat: [==========] 9 tests from 2 test suites ran. (9551 ms total) 2024-04-14T15:13:03.821 INFO:tasks.workunit.client.0.smithi090.stdout: api_stat: [ PASSED ] 9 tests. 2024-04-14T15:13:03.823 INFO:tasks.workunit.client.0.smithi090.stdout:api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2024-04-14T15:13:03.823 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP2 (3 ms) 2024-04-14T15:13:03.823 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.OverlappingWriteRoundTripPP 2024-04-14T15:13:03.823 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (5 ms) 2024-04-14T15:13:03.823 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2024-04-14T15:13:03.823 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (7 ms) 2024-04-14T15:13:03.824 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2024-04-14T15:13:03.824 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (10 ms) 2024-04-14T15:13:03.824 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2024-04-14T15:13:03.824 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (6 ms) 2024-04-14T15:13:03.824 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2024-04-14T15:13:03.824 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (6 ms) 2024-04-14T15:13:03.824 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2024-04-14T15:13:03.824 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (3 ms) 2024-04-14T15:13:03.824 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2024-04-14T15:13:03.824 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (3 ms) 2024-04-14T15:13:03.824 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2024-04-14T15:13:03.824 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (7 ms) 2024-04-14T15:13:03.824 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2024-04-14T15:13:03.825 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (4 ms) 2024-04-14T15:13:03.825 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2024-04-14T15:13:03.825 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (5 ms) 2024-04-14T15:13:03.825 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2024-04-14T15:13:03.825 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (3 ms) 2024-04-14T15:13:03.825 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2024-04-14T15:13:03.825 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (5 ms) 2024-04-14T15:13:03.825 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP (947 ms total) 2024-04-14T15:13:03.825 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: 2024-04-14T15:13:03.825 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [----------] Global test environment tear-down 2024-04-14T15:13:03.825 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [==========] 38 tests from 2 test suites ran. (9663 ms total) 2024-04-14T15:13:03.825 INFO:tasks.workunit.client.0.smithi090.stdout: api_io_pp: [ PASSED ] 38 tests. 2024-04-14T15:13:03.826 INFO:tasks.workunit.client.0.smithi090.stdout: api_stat_pp: Running main() from gmock_main.cc 2024-04-14T15:13:03.826 INFO:tasks.workunit.client.0.smithi090.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2024-04-14T15:13:03.827 INFO:tasks.workunit.client.0.smithi090.stdout: api_stat_pp: [----------] Global test environment set-up. 2024-04-14T15:13:03.827 INFO:tasks.workunit.client.0.smithi090.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2024-04-14T15:13:03.827 INFO:tasks.workunit.client.0.smithi090.stdout: api_stat_pp: seed 93864 2024-04-14T15:13:03.827 INFO:tasks.workunit.client.0.smithi090.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2024-04-14T15:13:03.827 INFO:tasks.workunit.client.0.smithi090.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (429 ms) 2024-04-14T15:13:03.827 INFO:tasks.workunit.client.0.smithi090.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2024-04-14T15:13:03.827 INFO:tasks.workunit.client.0.smithi090.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (46 ms) 2024-04-14T15:13:03.827 INFO:tasks.workunit.client.0.smithi090.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2024-04-14T15:13:03.827 INFO:tasks.workunit.client.0.smithi090.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (0 ms) 2024-04-14T15:13:03.827 INFO:tasks.workunit.client.0.smithi090.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2024-04-14T15:13:03.827 INFO:tasks.workunit.client.0.smithi090.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (57 ms) 2024-04-14T15:13:03.827 INFO:tasks.workunit.client.0.smithi090.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2024-04-14T15:13:03.827 INFO:tasks.workunit.client.0.smithi090.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (21 ms) 2024-04-14T15:13:03.828 INFO:tasks.workunit.client.0.smithi090.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (553 ms total) 2024-04-14T15:13:03.828 INFO:tasks.workunit.client.0.smithi090.stdout: api_stat_pp: 2024-04-14T15:13:03.828 INFO:tasks.workunit.client.0.smithi090.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2024-04-14T15:13:03.828 INFO:tasks.workunit.client.0.smithi090.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2024-04-14T15:13:03.828 INFO:tasks.workunit.client.0.smithi090.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (865 ms) 2024-04-14T15:13:03.828 INFO:tasks.workunit.client.0.smithi090.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2024-04-14T15:13:03.828 INFO:tasks.workunit.client.0.smithi090.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (0 ms) 2024-04-14T15:13:03.828 INFO:tasks.workunit.client.0.smithi090.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2024-04-14T15:13:03.828 INFO:tasks.workunit.client.0.smithi090.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (2 ms) 2024-04-14T15:13:03.828 INFO:tasks.workunit.client.0.smithi090.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2024-04-14T15:13:03.828 INFO:tasks.workunit.client.0.smithi090.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (9 ms) 2024-04-14T15:13:03.828 INFO:tasks.workunit.client.0.smithi090.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (876 ms total) 2024-04-14T15:13:03.828 INFO:tasks.workunit.client.0.smithi090.stdout: api_stat_pp: 2024-04-14T15:13:03.828 INFO:tasks.workunit.client.0.smithi090.stdout: api_stat_pp: [----------] Global test environment tear-down 2024-04-14T15:13:03.829 INFO:tasks.workunit.client.0.smithi090.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (9518 ms total) 2024-04-14T15:13:03.829 INFO:tasks.workunit.client.0.smithi090.stdout: api_stat_pp: [ PASSED ] 9 tests. 2024-04-14T15:13:03.833 INFO:tasks.workunit.client.0.smithi090.stdout: api_io: Running main() from gmock_main.cc 2024-04-14T15:13:03.833 INFO:tasks.workunit.client.0.smithi090.stdout: api_io: [==========] Running 24 tests from 2 test suites. 2024-04-14T15:13:03.833 INFO:tasks.workunit.client.0.smithi090.stdout: api_io: [----------] Global test environment set-up. 2024-04-14T15:13:03.833 INFO:tasks.workunit.client.0.smithi090.stdout: api_io: [----------] 14 tests from LibRadosIo 2024-04-14T15:13:03.833 INFO:tasks.workunit.client.0.smithi090.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2024-04-14T15:13:03.833 INFO:tasks.workunit.client.0.smithi090.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (451 ms) 2024-04-14T15:13:03.833 INFO:tasks.workunit.client.0.smithi090.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2024-04-14T15:13:03.834 INFO:tasks.workunit.client.0.smithi090.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2024-04-14T15:13:03.834 INFO:tasks.workunit.client.0.smithi090.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2024-04-14T15:13:03.834 INFO:tasks.workunit.client.0.smithi090.stdout: api_io: no timeout :/ 2024-04-14T15:13:03.834 INFO:tasks.workunit.client.0.smithi090.stdout: api_io: no timeout :/ 2024-04-14T15:13:03.834 INFO:tasks.workunit.client.0.smithi090.stdout: api_io: no timeout :/ 2024-04-14T15:13:03.834 INFO:tasks.workunit.client.0.smithi090.stdout: api_io: no timeout :/ 2024-04-14T15:13:03.834 INFO:tasks.workunit.client.0.smithi090.stdout: api_io: no timeout :/ 2024-04-14T15:13:03.834 INFO:tasks.workunit.client.0.smithi090.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (78 ms) 2024-04-14T15:13:03.834 INFO:tasks.workunit.client.0.smithi090.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2024-04-14T15:13:03.834 INFO:tasks.workunit.client.0.smithi090.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (23 ms) 2024-04-14T15:13:03.834 INFO:tasks.workunit.client.0.smithi090.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2024-04-14T15:13:03.834 INFO:tasks.workunit.client.0.smithi090.stdout: api_io: [ OK ] LibRadosIo.Checksum (5 ms) 2024-04-14T15:13:03.835 INFO:tasks.workunit.client.0.smithi090.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2024-04-14T15:13:03.835 INFO:tasks.workunit.client.0.smithi090.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (4 ms) 2024-04-14T15:13:03.835 INFO:tasks.workunit.client.0.smithi090.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2024-04-14T15:13:03.835 INFO:tasks.workunit.client.0.smithi090.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (7 ms) 2024-04-14T15:13:03.835 INFO:tasks.workunit.client.0.smithi090.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2024-04-14T15:13:03.835 INFO:tasks.workunit.client.0.smithi090.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (10 ms) 2024-04-14T15:13:03.835 INFO:tasks.workunit.client.0.smithi090.stdout: api_io: [ RUN ] LibRadosIo.ZeroLenZero 2024-04-14T15:13:03.835 INFO:tasks.workunit.client.0.smithi090.stdout: api_io: [ OK ] LibRadosIo.ZeroLenZero (2 ms) 2024-04-14T15:13:03.835 INFO:tasks.workunit.client.0.smithi090.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2024-04-14T15:13:03.835 INFO:tasks.workunit.client.0.smithi090.stdout: api_io: [ OK ] LibRadosIo.TruncTest (5 ms) 2024-04-14T15:13:03.835 INFO:tasks.workunit.client.0.smithi090.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2024-04-14T15:13:03.835 INFO:tasks.workunit.client.0.smithi090.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (6 ms) 2024-04-14T15:13:03.836 INFO:tasks.workunit.client.0.smithi090.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2024-04-14T15:13:03.836 INFO:tasks.workunit.client.0.smithi090.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (7 ms) 2024-04-14T15:13:03.836 INFO:tasks.workunit.client.0.smithi090.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2024-04-14T15:13:03.836 INFO:tasks.workunit.client.0.smithi090.stdout: api_io: [ OK ] LibRadosIo.RmXattr (17 ms) 2024-04-14T15:13:03.836 INFO:tasks.workunit.client.0.smithi090.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2024-04-14T15:13:03.836 INFO:tasks.workunit.client.0.smithi090.stdout: api_io: [ OK ] LibRadosIo.XattrIter (8 ms) 2024-04-14T15:13:03.836 INFO:tasks.workunit.client.0.smithi090.stdout: api_io: [----------] 14 tests from LibRadosIo (623 ms total) 2024-04-14T15:13:03.836 INFO:tasks.workunit.client.0.smithi090.stdout: api_io: 2024-04-14T15:13:03.836 INFO:tasks.workunit.client.0.smithi090.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2024-04-14T15:13:03.836 INFO:tasks.workunit.client.0.smithi090.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2024-04-14T15:13:03.836 INFO:tasks.workunit.client.0.smithi090.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (848 ms) 2024-04-14T15:13:03.836 INFO:tasks.workunit.client.0.smithi090.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2024-04-14T15:13:03.836 INFO:tasks.workunit.client.0.smithi090.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (13 ms) 2024-04-14T15:13:03.836 INFO:tasks.workunit.client.0.smithi090.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2024-04-14T15:13:03.837 INFO:tasks.workunit.client.0.smithi090.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (7 ms) 2024-04-14T15:13:03.837 INFO:tasks.workunit.client.0.smithi090.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2024-04-14T15:13:03.837 INFO:tasks.workunit.client.0.smithi090.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (6 ms) 2024-04-14T15:13:03.837 INFO:tasks.workunit.client.0.smithi090.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2024-04-14T15:13:03.837 INFO:tasks.workunit.client.0.smithi090.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (7 ms) 2024-04-14T15:13:03.837 INFO:tasks.workunit.client.0.smithi090.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2024-04-14T15:13:03.837 INFO:tasks.workunit.client.0.smithi090.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (5 ms) 2024-04-14T15:13:03.837 INFO:tasks.workunit.client.0.smithi090.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2024-04-14T15:13:03.837 INFO:tasks.workunit.client.0.smithi090.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (3 ms) 2024-04-14T15:13:03.837 INFO:tasks.workunit.client.0.smithi090.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2024-04-14T15:13:03.837 INFO:tasks.workunit.client.0.smithi090.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (3 ms) 2024-04-14T15:13:03.837 INFO:tasks.workunit.client.0.smithi090.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2024-04-14T15:13:03.837 INFO:tasks.workunit.client.0.smithi090.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (10 ms) 2024-04-14T15:13:03.837 INFO:tasks.workunit.client.0.smithi090.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2024-04-14T15:13:03.837 INFO:tasks.workunit.client.0.smithi090.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (6 ms) 2024-04-14T15:13:03.838 INFO:tasks.workunit.client.0.smithi090.stdout: api_io: [----------] 10 tests from LibRadosIoEC (908 ms total) 2024-04-14T15:13:03.838 INFO:tasks.workunit.client.0.smithi090.stdout: api_io: 2024-04-14T15:13:03.838 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:13:03.839 INFO:tasks.workunit.client.0.smithi090.stdout: api_io: [----------] Global test environment tear-down 2024-04-14T15:13:03.839 INFO:tasks.workunit.client.0.smithi090.stdout: api_io: [==========] 24 tests from 2 test suites ran. (9681 ms total) 2024-04-14T15:13:03.839 INFO:tasks.workunit.client.0.smithi090.stdout: api_io: [ PASSED ] 24 tests. 2024-04-14T15:13:03.839 INFO:tasks.workunit.client.0.smithi090.stdout: watch_notify: Running main() from gmock_main.cc 2024-04-14T15:13:03.839 INFO:tasks.workunit.client.0.smithi090.stdout: watch_notify: [==========] Running 2 tests from 1 test suite. 2024-04-14T15:13:03.839 INFO:tasks.workunit.client.0.smithi090.stdout: watch_notify: [----------] Global test environment set-up. 2024-04-14T15:13:03.839 INFO:tasks.workunit.client.0.smithi090.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify 2024-04-14T15:13:03.839 INFO:tasks.workunit.client.0.smithi090.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotify 2024-04-14T15:13:03.839 INFO:tasks.workunit.client.0.smithi090.stdout: watch_notify: handle_notify cookie 94006567794064 notify_id 463856467968 notifier_gid 25028 2024-04-14T15:13:03.839 INFO:tasks.workunit.client.0.smithi090.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotify (2495 ms) 2024-04-14T15:13:03.839 INFO:tasks.workunit.client.0.smithi090.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotifyTimeout 2024-04-14T15:13:03.840 INFO:tasks.workunit.client.0.smithi090.stdout: watch_notify: Trying... 2024-04-14T15:13:03.840 INFO:tasks.workunit.client.0.smithi090.stdout: watch_notify: handle_notify cookie 94006580515472 notify_id 476741369856 notifier_gid 25255 2024-04-14T15:13:03.840 INFO:tasks.workunit.client.0.smithi090.stdout: watch_notify: Waiting for 3.000000000s 2024-04-14T15:13:03.840 INFO:tasks.workunit.client.0.smithi090.stdout: watch_notify: Timed out. 2024-04-14T15:13:03.840 INFO:tasks.workunit.client.0.smithi090.stdout: watch_notify: Flushing... 2024-04-14T15:13:03.840 INFO:tasks.workunit.client.0.smithi090.stdout: watch_notify: Flushed... 2024-04-14T15:13:03.840 INFO:tasks.workunit.client.0.smithi090.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotifyTimeout (6726 ms) 2024-04-14T15:13:03.840 INFO:tasks.workunit.client.0.smithi090.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify (9221 ms total) 2024-04-14T15:13:03.840 INFO:tasks.workunit.client.0.smithi090.stdout: watch_notify: 2024-04-14T15:13:03.840 INFO:tasks.workunit.client.0.smithi090.stdout: watch_notify: [----------] Global test environment tear-down 2024-04-14T15:13:03.840 INFO:tasks.workunit.client.0.smithi090.stdout: watch_notify: [==========] 2 tests from 1 test suite ran. (9221 ms total) 2024-04-14T15:13:03.840 INFO:tasks.workunit.client.0.smithi090.stdout: watch_notify: [ PASSED ] 2 tests. 2024-04-14T15:13:03.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:03.907 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:03.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:13:04.009 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:13:04.033 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:04.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:04.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:13:04.135 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:13:04.159 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:04.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:04.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:13:04.256 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:13:03 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:13:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:13:04.261 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:13:04.262 INFO:tasks.workunit.client.0.smithi090.stdout:04-14T15:13:01.226091+0000 client.admin [INF] threexx 2024-04-14T15:13:04.262 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: got: 2024-04-14T15:13:01.226339+0000 mon.a [INF] from='client.25252 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-04-14T15:13:04.262 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (7105 ms) 2024-04-14T15:13:04.262 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (9909 ms total) 2024-04-14T15:13:04.262 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: 2024-04-14T15:13:04.262 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: [----------] Global test environment tear-down 2024-04-14T15:13:04.262 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (9910 ms total) 2024-04-14T15:13:04.262 INFO:tasks.workunit.client.0.smithi090.stdout: api_cmd: [ PASSED ] 4 tests. 2024-04-14T15:13:04.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:04.285 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:04.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:13:04.386 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:13:04.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:04.412 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:04.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:13:04.513 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:13:04.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:04.536 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:04.537 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:13:04.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:04 smithi092 ceph-mon[33966]: pgmap v119: 700 pgs: 248 unknown, 64 creating+peering, 388 active+clean; 25 MiB data, 282 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:13:04.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:04 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:04.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:04 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3592743837' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-04-14T15:13:04.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:04 smithi092 ceph-mon[33966]: Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:13:04.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:04 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1715119515' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi090-94662-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi090-94662-2"}]': finished 2024-04-14T15:13:04.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:04 smithi092 ceph-mon[33966]: from='client.25267 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi090-93864-7"}]': finished 2024-04-14T15:13:04.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:04 smithi092 ceph-mon[33966]: from='client.25249 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi090-93731-7"}]': finished 2024-04-14T15:13:04.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:04 smithi092 ceph-mon[33966]: from='client.25243 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi090-93398-23"}]': finished 2024-04-14T15:13:04.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:04 smithi092 ceph-mon[33966]: from='client.25240 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi090-93379-16"}]': finished 2024-04-14T15:13:04.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:04 smithi092 ceph-mon[33966]: from='client.25376 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi090-93998-6", "pg_num": 4}]': finished 2024-04-14T15:13:04.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:04 smithi092 ceph-mon[33966]: from='client.25114 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-4"}]': finished 2024-04-14T15:13:04.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:04 smithi092 ceph-mon[33966]: from='client.25379 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi090-93582-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:04.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:04 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1327720256' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-4", "tierpool": "test-rados-api-smithi090-93652-4-cache"}]: dispatch 2024-04-14T15:13:04.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:04 smithi092 ceph-mon[33966]: osdmap e116: 8 total, 8 up, 8 in 2024-04-14T15:13:04.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:04 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2105945719' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi090-93998-4", "tierpool": "test-rados-api-smithi090-93998-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:13:04.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:04 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3684793142' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi090-93582-1","app": "app2"}]: dispatch 2024-04-14T15:13:04.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:04 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/773685218' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi090-93342-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:04.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:04 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3684793142' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi090-93582-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:04.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:04 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1015294680' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:04.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:04 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1279585499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:04.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:04 smithi092 ceph-mon[33966]: from='client.25114 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-4", "tierpool": "test-rados-api-smithi090-93652-4-cache"}]: dispatch 2024-04-14T15:13:04.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:04 smithi092 ceph-mon[33966]: from='client.25376 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi090-93998-4", "tierpool": "test-rados-api-smithi090-93998-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:13:04.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:04 smithi092 ceph-mon[33966]: from='client.25379 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi090-93582-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:04.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:04 smithi092 ceph-mon[33966]: from='client.25579 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi090-93342-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:04.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:04 smithi092 ceph-mon[33966]: from='client.25588 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:04.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:04 smithi092 ceph-mon[33966]: from='client.25570 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:04.637 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:13:04.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:04.662 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:04.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:13:04.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:04 smithi090 ceph-mon[30399]: pgmap v119: 700 pgs: 248 unknown, 64 creating+peering, 388 active+clean; 25 MiB data, 282 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:13:04.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:04 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:04.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:04 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3592743837' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-04-14T15:13:04.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:04 smithi090 ceph-mon[30399]: Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:13:04.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:04 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1715119515' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi090-94662-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi090-94662-2"}]': finished 2024-04-14T15:13:04.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:04 smithi090 ceph-mon[30399]: from='client.25267 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi090-93864-7"}]': finished 2024-04-14T15:13:04.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:04 smithi090 ceph-mon[30399]: from='client.25249 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi090-93731-7"}]': finished 2024-04-14T15:13:04.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:04 smithi090 ceph-mon[30399]: from='client.25243 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi090-93398-23"}]': finished 2024-04-14T15:13:04.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:04 smithi090 ceph-mon[30399]: from='client.25240 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi090-93379-16"}]': finished 2024-04-14T15:13:04.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:04 smithi090 ceph-mon[30399]: from='client.25376 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi090-93998-6", "pg_num": 4}]': finished 2024-04-14T15:13:04.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:04 smithi090 ceph-mon[30399]: from='client.25114 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-4"}]': finished 2024-04-14T15:13:04.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:04 smithi090 ceph-mon[30399]: from='client.25379 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi090-93582-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:04.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:04 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1327720256' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-4", "tierpool": "test-rados-api-smithi090-93652-4-cache"}]: dispatch 2024-04-14T15:13:04.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:04 smithi090 ceph-mon[30399]: osdmap e116: 8 total, 8 up, 8 in 2024-04-14T15:13:04.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:04 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2105945719' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi090-93998-4", "tierpool": "test-rados-api-smithi090-93998-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:13:04.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:04 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3684793142' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi090-93582-1","app": "app2"}]: dispatch 2024-04-14T15:13:04.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:04 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/773685218' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi090-93342-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:04.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:04 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3684793142' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi090-93582-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:04.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:04 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1015294680' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:04.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:04 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1279585499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:04.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:04 smithi090 ceph-mon[30399]: from='client.25114 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-4", "tierpool": "test-rados-api-smithi090-93652-4-cache"}]: dispatch 2024-04-14T15:13:04.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:04 smithi090 ceph-mon[30399]: from='client.25376 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi090-93998-4", "tierpool": "test-rados-api-smithi090-93998-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:13:04.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:04 smithi090 ceph-mon[30399]: from='client.25379 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi090-93582-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:04.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:04 smithi090 ceph-mon[30399]: from='client.25579 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi090-93342-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:04.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:04 smithi090 ceph-mon[30399]: from='client.25588 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:04.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:04 smithi090 ceph-mon[30399]: from='client.25570 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:04.694 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:04 smithi090 ceph-mon[41670]: pgmap v119: 700 pgs: 248 unknown, 64 creating+peering, 388 active+clean; 25 MiB data, 282 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:13:04.694 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:04 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:04.694 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:04 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3592743837' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-04-14T15:13:04.694 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:04 smithi090 ceph-mon[41670]: Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:13:04.694 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:04 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1715119515' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi090-94662-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi090-94662-2"}]': finished 2024-04-14T15:13:04.694 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:04 smithi090 ceph-mon[41670]: from='client.25267 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi090-93864-7"}]': finished 2024-04-14T15:13:04.694 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:04 smithi090 ceph-mon[41670]: from='client.25249 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi090-93731-7"}]': finished 2024-04-14T15:13:04.695 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:04 smithi090 ceph-mon[41670]: from='client.25243 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi090-93398-23"}]': finished 2024-04-14T15:13:04.695 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:04 smithi090 ceph-mon[41670]: from='client.25240 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi090-93379-16"}]': finished 2024-04-14T15:13:04.695 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:04 smithi090 ceph-mon[41670]: from='client.25376 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi090-93998-6", "pg_num": 4}]': finished 2024-04-14T15:13:04.696 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:04 smithi090 ceph-mon[41670]: from='client.25114 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-4"}]': finished 2024-04-14T15:13:04.696 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:04 smithi090 ceph-mon[41670]: from='client.25379 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi090-93582-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:04.696 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:04 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1327720256' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-4", "tierpool": "test-rados-api-smithi090-93652-4-cache"}]: dispatch 2024-04-14T15:13:04.696 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:04 smithi090 ceph-mon[41670]: osdmap e116: 8 total, 8 up, 8 in 2024-04-14T15:13:04.696 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:04 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2105945719' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi090-93998-4", "tierpool": "test-rados-api-smithi090-93998-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:13:04.696 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:04 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3684793142' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi090-93582-1","app": "app2"}]: dispatch 2024-04-14T15:13:04.696 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:04 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/773685218' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi090-93342-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:04.696 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:04 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3684793142' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi090-93582-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:04.696 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:04 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1015294680' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:04.696 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:04 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1279585499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:04.696 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:04 smithi090 ceph-mon[41670]: from='client.25114 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-4", "tierpool": "test-rados-api-smithi090-93652-4-cache"}]: dispatch 2024-04-14T15:13:04.696 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:04 smithi090 ceph-mon[41670]: from='client.25376 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi090-93998-4", "tierpool": "test-rados-api-smithi090-93998-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:13:04.696 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:04 smithi090 ceph-mon[41670]: from='client.25379 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi090-93582-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:04.696 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:04 smithi090 ceph-mon[41670]: from='client.25579 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi090-93342-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:04.697 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:04 smithi090 ceph-mon[41670]: from='client.25588 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:04.697 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:04 smithi090 ceph-mon[41670]: from='client.25570 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:04.763 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:13:04.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:04.787 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:04.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:13:04.888 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:13:04.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:04.923 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:04.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:13:05.024 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:13:05.051 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:05.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:05.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:13:05.122 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:13:04 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:13:05.153 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:13:05.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:05.178 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:05.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:13:05.279 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:13:05.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:05.303 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:05.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:13:05.404 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:13:05.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:05.429 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:05.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:13:05.530 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:13:05.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:05.554 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:05.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:13:05.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:05 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:05.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:05 smithi092 ceph-mon[33966]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:13:05.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:05 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:13:05.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:05 smithi092 ceph-mon[33966]: from='client.25114 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-4", "tierpool": "test-rados-api-smithi090-93652-4-cache"}]': finished 2024-04-14T15:13:05.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:05 smithi092 ceph-mon[33966]: from='client.25376 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi090-93998-4", "tierpool": "test-rados-api-smithi090-93998-6", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:13:05.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:05 smithi092 ceph-mon[33966]: from='client.25379 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi090-93582-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:05.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:05 smithi092 ceph-mon[33966]: from='client.25579 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi090-93342-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:05.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:05 smithi092 ceph-mon[33966]: from='client.25588 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:05.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:05 smithi092 ceph-mon[33966]: from='client.25570 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:05.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:05 smithi092 ceph-mon[33966]: osdmap e117: 8 total, 8 up, 8 in 2024-04-14T15:13:05.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:05 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2105945719' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi090-93998-4", "overlaypool": "test-rados-api-smithi090-93998-6"}]: dispatch 2024-04-14T15:13:05.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:05 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/340734729' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi090-94522-2"}]: dispatch 2024-04-14T15:13:05.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:05 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3684793142' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi090-93582-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-04-14T15:13:05.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:05 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3684793142' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi090-93582-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-04-14T15:13:05.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:05 smithi092 ceph-mon[33966]: from='client.25376 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi090-93998-4", "overlaypool": "test-rados-api-smithi090-93998-6"}]: dispatch 2024-04-14T15:13:05.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:05 smithi092 ceph-mon[33966]: from='client.25379 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi090-93582-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-04-14T15:13:05.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:05 smithi092 ceph-mon[33966]: from='client.25489 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi090-94522-2"}]: dispatch 2024-04-14T15:13:05.656 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:13:05.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:05.680 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:05.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:13:05.691 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:05 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:05.691 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:05 smithi090 ceph-mon[41670]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:13:05.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:05 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:13:05.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:05 smithi090 ceph-mon[41670]: from='client.25114 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-4", "tierpool": "test-rados-api-smithi090-93652-4-cache"}]': finished 2024-04-14T15:13:05.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:05 smithi090 ceph-mon[41670]: from='client.25376 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi090-93998-4", "tierpool": "test-rados-api-smithi090-93998-6", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:13:05.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:05 smithi090 ceph-mon[41670]: from='client.25379 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi090-93582-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:05.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:05 smithi090 ceph-mon[41670]: from='client.25579 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi090-93342-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:05.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:05 smithi090 ceph-mon[41670]: from='client.25588 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:05.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:05 smithi090 ceph-mon[41670]: from='client.25570 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:05.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:05 smithi090 ceph-mon[41670]: osdmap e117: 8 total, 8 up, 8 in 2024-04-14T15:13:05.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:05 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2105945719' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi090-93998-4", "overlaypool": "test-rados-api-smithi090-93998-6"}]: dispatch 2024-04-14T15:13:05.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:05 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/340734729' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi090-94522-2"}]: dispatch 2024-04-14T15:13:05.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:05 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3684793142' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi090-93582-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-04-14T15:13:05.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:05 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3684793142' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi090-93582-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-04-14T15:13:05.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:05 smithi090 ceph-mon[41670]: from='client.25376 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi090-93998-4", "overlaypool": "test-rados-api-smithi090-93998-6"}]: dispatch 2024-04-14T15:13:05.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:05 smithi090 ceph-mon[41670]: from='client.25379 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi090-93582-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-04-14T15:13:05.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:05 smithi090 ceph-mon[41670]: from='client.25489 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi090-94522-2"}]: dispatch 2024-04-14T15:13:05.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:05 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:05.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:05 smithi090 ceph-mon[30399]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:13:05.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:05 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:13:05.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:05 smithi090 ceph-mon[30399]: from='client.25114 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-4", "tierpool": "test-rados-api-smithi090-93652-4-cache"}]': finished 2024-04-14T15:13:05.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:05 smithi090 ceph-mon[30399]: from='client.25376 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi090-93998-4", "tierpool": "test-rados-api-smithi090-93998-6", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:13:05.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:05 smithi090 ceph-mon[30399]: from='client.25379 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi090-93582-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:05.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:05 smithi090 ceph-mon[30399]: from='client.25579 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi090-93342-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:05.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:05 smithi090 ceph-mon[30399]: from='client.25588 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:05.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:05 smithi090 ceph-mon[30399]: from='client.25570 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:05.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:05 smithi090 ceph-mon[30399]: osdmap e117: 8 total, 8 up, 8 in 2024-04-14T15:13:05.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:05 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2105945719' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi090-93998-4", "overlaypool": "test-rados-api-smithi090-93998-6"}]: dispatch 2024-04-14T15:13:05.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:05 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/340734729' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi090-94522-2"}]: dispatch 2024-04-14T15:13:05.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:05 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3684793142' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi090-93582-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-04-14T15:13:05.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:05 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3684793142' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi090-93582-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-04-14T15:13:05.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:05 smithi090 ceph-mon[30399]: from='client.25376 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi090-93998-4", "overlaypool": "test-rados-api-smithi090-93998-6"}]: dispatch 2024-04-14T15:13:05.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:05 smithi090 ceph-mon[30399]: from='client.25379 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi090-93582-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-04-14T15:13:05.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:05 smithi090 ceph-mon[30399]: from='client.25489 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi090-94522-2"}]: dispatch 2024-04-14T15:13:05.781 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:13:05.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:05.831 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:05.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:13:05.932 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:13:05.959 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:05.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:05.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:13:06.060 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:13:06.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:06.086 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:06.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:13:06.190 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:13:06.214 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:06.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:06.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:13:06.316 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:13:06.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:06.341 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:06.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:13:06.446 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:13:06.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:06.471 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:06.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:13:06.572 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:13:06.575 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:06 smithi090 ceph-mon[30399]: pgmap v122: 832 pgs: 204 creating+peering, 96 unknown, 532 active+clean; 145 MiB data, 767 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 31 MiB/s wr, 13 op/s 2024-04-14T15:13:06.575 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:06 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:13:06.575 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:06 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:13:06.575 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:06 smithi090 ceph-mon[30399]: 16.6 deep-scrub starts 2024-04-14T15:13:06.575 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:06 smithi090 ceph-mon[30399]: 16.6 deep-scrub ok 2024-04-14T15:13:06.575 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:06 smithi090 ceph-mon[30399]: 16.8 deep-scrub starts 2024-04-14T15:13:06.575 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:06 smithi090 ceph-mon[30399]: 16.8 deep-scrub ok 2024-04-14T15:13:06.575 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:06 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:06.575 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:06 smithi090 ceph-mon[30399]: from='client.25376 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi090-93998-4", "overlaypool": "test-rados-api-smithi090-93998-6"}]': finished 2024-04-14T15:13:06.575 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:06 smithi090 ceph-mon[30399]: from='client.25379 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi090-93582-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-04-14T15:13:06.575 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:06 smithi090 ceph-mon[30399]: from='client.25489 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi090-94522-2"}]': finished 2024-04-14T15:13:06.576 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:06 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/340734729' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi090-94522-2"}]: dispatch 2024-04-14T15:13:06.576 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:06 smithi090 ceph-mon[30399]: osdmap e118: 8 total, 8 up, 8 in 2024-04-14T15:13:06.576 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:06 smithi090 ceph-mon[30399]: from='client.25489 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi090-94522-2"}]: dispatch 2024-04-14T15:13:06.576 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:06 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1715119515' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi090-94662-2"}]: dispatch 2024-04-14T15:13:06.576 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:06 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3684793142' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi090-93582-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-04-14T15:13:06.576 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:06 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/274362209' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi090-93436-10"}]: dispatch 2024-04-14T15:13:06.576 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:06 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2105945719' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93998-6", "mode": "writeback"}]: dispatch 2024-04-14T15:13:06.576 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:06 smithi090 ceph-mon[30399]: from='client.25379 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi090-93582-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-04-14T15:13:06.576 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:06 smithi090 ceph-mon[30399]: from='client.25376 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93998-6", "mode": "writeback"}]: dispatch 2024-04-14T15:13:06.576 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:06 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/434516291' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi090-93709-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:06.576 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:06 smithi090 ceph-mon[30399]: from='client.25457 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi090-93709-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:06.576 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:06 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1713883211' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi090-93476-10"}]: dispatch 2024-04-14T15:13:06.576 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:06 smithi090 ceph-mon[30399]: from='client.25217 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi090-93476-10"}]: dispatch 2024-04-14T15:13:06.576 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:06 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3852633403' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi090-93930-12"}]: dispatch 2024-04-14T15:13:06.576 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:06 smithi090 ceph-mon[30399]: from='client.25636 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi090-93930-12"}]: dispatch 2024-04-14T15:13:06.577 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:06 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3852633403' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi090-93930-12"}]: dispatch 2024-04-14T15:13:06.577 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:06 smithi090 ceph-mon[30399]: from='client.25636 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi090-93930-12"}]: dispatch 2024-04-14T15:13:06.577 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:06 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3852633403' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi090-93930-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:13:06.577 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:06 smithi090 ceph-mon[30399]: from='client.25636 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi090-93930-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:13:06.577 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:06 smithi090 ceph-mon[41670]: pgmap v122: 832 pgs: 204 creating+peering, 96 unknown, 532 active+clean; 145 MiB data, 767 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 31 MiB/s wr, 13 op/s 2024-04-14T15:13:06.577 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:06 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:13:06.577 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:06 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:13:06.577 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:06 smithi090 ceph-mon[41670]: 16.6 deep-scrub starts 2024-04-14T15:13:06.577 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:06 smithi090 ceph-mon[41670]: 16.6 deep-scrub ok 2024-04-14T15:13:06.578 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:06 smithi090 ceph-mon[41670]: 16.8 deep-scrub starts 2024-04-14T15:13:06.578 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:06 smithi090 ceph-mon[41670]: 16.8 deep-scrub ok 2024-04-14T15:13:06.578 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:06 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:06.578 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:06 smithi090 ceph-mon[41670]: from='client.25376 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi090-93998-4", "overlaypool": "test-rados-api-smithi090-93998-6"}]': finished 2024-04-14T15:13:06.578 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:06 smithi090 ceph-mon[41670]: from='client.25379 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi090-93582-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-04-14T15:13:06.578 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:06 smithi090 ceph-mon[41670]: from='client.25489 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi090-94522-2"}]': finished 2024-04-14T15:13:06.578 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:06 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/340734729' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi090-94522-2"}]: dispatch 2024-04-14T15:13:06.578 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:06 smithi090 ceph-mon[41670]: osdmap e118: 8 total, 8 up, 8 in 2024-04-14T15:13:06.578 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:06 smithi090 ceph-mon[41670]: from='client.25489 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi090-94522-2"}]: dispatch 2024-04-14T15:13:06.579 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:06 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1715119515' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi090-94662-2"}]: dispatch 2024-04-14T15:13:06.579 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:06 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3684793142' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi090-93582-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-04-14T15:13:06.579 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:06 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/274362209' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi090-93436-10"}]: dispatch 2024-04-14T15:13:06.579 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:06 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2105945719' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93998-6", "mode": "writeback"}]: dispatch 2024-04-14T15:13:06.579 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:06 smithi090 ceph-mon[41670]: from='client.25379 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi090-93582-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-04-14T15:13:06.579 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:06 smithi090 ceph-mon[41670]: from='client.25376 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93998-6", "mode": "writeback"}]: dispatch 2024-04-14T15:13:06.580 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:06 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/434516291' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi090-93709-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:06.580 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:06 smithi090 ceph-mon[41670]: from='client.25457 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi090-93709-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:06.580 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:06 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1713883211' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi090-93476-10"}]: dispatch 2024-04-14T15:13:06.580 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:06 smithi090 ceph-mon[41670]: from='client.25217 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi090-93476-10"}]: dispatch 2024-04-14T15:13:06.580 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:06 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3852633403' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi090-93930-12"}]: dispatch 2024-04-14T15:13:06.580 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:06 smithi090 ceph-mon[41670]: from='client.25636 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi090-93930-12"}]: dispatch 2024-04-14T15:13:06.580 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:06 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3852633403' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi090-93930-12"}]: dispatch 2024-04-14T15:13:06.581 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:06 smithi090 ceph-mon[41670]: from='client.25636 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi090-93930-12"}]: dispatch 2024-04-14T15:13:06.581 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:06 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3852633403' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi090-93930-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:13:06.581 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:06 smithi090 ceph-mon[41670]: from='client.25636 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi090-93930-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:13:06.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:06.596 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:06.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:13:06.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:06 smithi092 ceph-mon[33966]: pgmap v122: 832 pgs: 204 creating+peering, 96 unknown, 532 active+clean; 145 MiB data, 767 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 31 MiB/s wr, 13 op/s 2024-04-14T15:13:06.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:06 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:13:06.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:06 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:13:06.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:06 smithi092 ceph-mon[33966]: 16.6 deep-scrub starts 2024-04-14T15:13:06.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:06 smithi092 ceph-mon[33966]: 16.6 deep-scrub ok 2024-04-14T15:13:06.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:06 smithi092 ceph-mon[33966]: 16.8 deep-scrub starts 2024-04-14T15:13:06.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:06 smithi092 ceph-mon[33966]: 16.8 deep-scrub ok 2024-04-14T15:13:06.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:06 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:06.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:06 smithi092 ceph-mon[33966]: from='client.25376 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi090-93998-4", "overlaypool": "test-rados-api-smithi090-93998-6"}]': finished 2024-04-14T15:13:06.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:06 smithi092 ceph-mon[33966]: from='client.25379 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi090-93582-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-04-14T15:13:06.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:06 smithi092 ceph-mon[33966]: from='client.25489 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi090-94522-2"}]': finished 2024-04-14T15:13:06.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:06 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/340734729' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi090-94522-2"}]: dispatch 2024-04-14T15:13:06.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:06 smithi092 ceph-mon[33966]: osdmap e118: 8 total, 8 up, 8 in 2024-04-14T15:13:06.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:06 smithi092 ceph-mon[33966]: from='client.25489 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi090-94522-2"}]: dispatch 2024-04-14T15:13:06.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:06 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1715119515' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi090-94662-2"}]: dispatch 2024-04-14T15:13:06.625 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:06 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3684793142' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi090-93582-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-04-14T15:13:06.625 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:06 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/274362209' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi090-93436-10"}]: dispatch 2024-04-14T15:13:06.625 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:06 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2105945719' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93998-6", "mode": "writeback"}]: dispatch 2024-04-14T15:13:06.625 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:06 smithi092 ceph-mon[33966]: from='client.25379 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi090-93582-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-04-14T15:13:06.625 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:06 smithi092 ceph-mon[33966]: from='client.25376 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93998-6", "mode": "writeback"}]: dispatch 2024-04-14T15:13:06.625 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:06 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/434516291' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi090-93709-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:06.625 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:06 smithi092 ceph-mon[33966]: from='client.25457 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi090-93709-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:06.625 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:06 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1713883211' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi090-93476-10"}]: dispatch 2024-04-14T15:13:06.625 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:06 smithi092 ceph-mon[33966]: from='client.25217 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi090-93476-10"}]: dispatch 2024-04-14T15:13:06.625 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:06 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3852633403' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi090-93930-12"}]: dispatch 2024-04-14T15:13:06.625 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:06 smithi092 ceph-mon[33966]: from='client.25636 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi090-93930-12"}]: dispatch 2024-04-14T15:13:06.626 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:06 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3852633403' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi090-93930-12"}]: dispatch 2024-04-14T15:13:06.626 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:06 smithi092 ceph-mon[33966]: from='client.25636 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi090-93930-12"}]: dispatch 2024-04-14T15:13:06.626 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:06 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3852633403' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi090-93930-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:13:06.626 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:06 smithi092 ceph-mon[33966]: from='client.25636 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi090-93930-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:13:06.697 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:13:06.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:06.721 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:06.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:13:06.803 INFO:tasks.workunit.client.0.smithi090.stdout: ec_io: Running main() from gmock_main.cc 2024-04-14T15:13:06.803 INFO:tasks.workunit.client.0.smithi090.stdout: ec_io: [==========] Running 2 tests from 1 test suite. 2024-04-14T15:13:06.803 INFO:tasks.workunit.client.0.smithi090.stdout: ec_io: [----------] Global test environment set-up. 2024-04-14T15:13:06.803 INFO:tasks.workunit.client.0.smithi090.stdout: ec_io: [----------] 2 tests from NeoRadosECIo 2024-04-14T15:13:06.803 INFO:tasks.workunit.client.0.smithi090.stdout: ec_io: [ RUN ] NeoRadosECIo.SimpleWrite 2024-04-14T15:13:06.804 INFO:tasks.workunit.client.0.smithi090.stdout: ec_io: [ OK ] NeoRadosECIo.SimpleWrite (5572 ms) 2024-04-14T15:13:06.804 INFO:tasks.workunit.client.0.smithi090.stdout: ec_io: [ RUN ] NeoRadosECIo.ReadOp 2024-04-14T15:13:06.804 INFO:tasks.workunit.client.0.smithi090.stdout: ec_io: [ OK ] NeoRadosECIo.ReadOp (6692 ms) 2024-04-14T15:13:06.804 INFO:tasks.workunit.client.0.smithi090.stdout: ec_io: [----------] 2 tests from NeoRadosECIo (12264 ms total) 2024-04-14T15:13:06.804 INFO:tasks.workunit.client.0.smithi090.stdout: ec_io: 2024-04-14T15:13:06.804 INFO:tasks.workunit.client.0.smithi090.stdout: ec_io: [----------] Global test environment tear-down 2024-04-14T15:13:06.804 INFO:tasks.workunit.client.0.smithi090.stdout: ec_io: [==========] 2 tests from 1 test suite ran. (12264 ms total) 2024-04-14T15:13:06.804 INFO:tasks.workunit.client.0.smithi090.stdout: ec_io: [ PASSED ] 2 tests. 2024-04-14T15:13:06.822 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:13:06.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:06.883 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:06.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:13:06.984 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:13:07.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:07.009 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:07.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:13:07.110 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:13:07.134 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:07.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:07.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:13:07.236 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:13:07.260 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:07.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:07.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:13:07.362 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:13:07.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:07.386 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:07.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:13:07.487 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:13:07.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:07.512 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:07.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:13:07.613 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:13:07.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:07 smithi092 ceph-mon[33966]: 16.5 deep-scrub starts 2024-04-14T15:13:07.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:07 smithi092 ceph-mon[33966]: 16.5 deep-scrub ok 2024-04-14T15:13:07.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:07 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:07.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:07 smithi092 ceph-mon[33966]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:13:07.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:07 smithi092 ceph-mon[33966]: from='client.25489 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi090-94522-2"}]': finished 2024-04-14T15:13:07.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:07 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1715119515' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi090-94662-2"}]': finished 2024-04-14T15:13:07.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:07 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/274362209' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi090-93436-10"}]': finished 2024-04-14T15:13:07.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:07 smithi092 ceph-mon[33966]: from='client.25379 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi090-93582-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-04-14T15:13:07.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:07 smithi092 ceph-mon[33966]: from='client.25376 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93998-6", "mode": "writeback"}]': finished 2024-04-14T15:13:07.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:07 smithi092 ceph-mon[33966]: from='client.25457 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi090-93709-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:07.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:07 smithi092 ceph-mon[33966]: from='client.25217 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi090-93476-10"}]': finished 2024-04-14T15:13:07.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:07 smithi092 ceph-mon[33966]: from='client.25636 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi090-93930-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:13:07.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:07 smithi092 ceph-mon[33966]: osdmap e119: 8 total, 8 up, 8 in 2024-04-14T15:13:07.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:07 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1713883211' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi090-93476-10"}]: dispatch 2024-04-14T15:13:07.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:07 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3852633403' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi090-93930-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi090-93930-12"}]: dispatch 2024-04-14T15:13:07.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:07 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/793350958' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi090-93342-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:07.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:07 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/274362209' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi090-93436-10"}]: dispatch 2024-04-14T15:13:07.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:07 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3749675023' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:07.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:07 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1715119515' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi090-94662-2"}]: dispatch 2024-04-14T15:13:07.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:07 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3684793142' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi090-93582-1","app":"app1","key":"key1"}]: dispatch 2024-04-14T15:13:07.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:07 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1422064328' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:07.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:07 smithi092 ceph-mon[33966]: from='client.25636 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi090-93930-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi090-93930-12"}]: dispatch 2024-04-14T15:13:07.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:07 smithi092 ceph-mon[33966]: from='client.25217 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi090-93476-10"}]: dispatch 2024-04-14T15:13:07.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:07 smithi092 ceph-mon[33966]: from='client.25379 ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi090-93582-1","app":"app1","key":"key1"}]: dispatch 2024-04-14T15:13:07.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:07 smithi092 ceph-mon[33966]: from='client.25651 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi090-93342-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:07.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:07 smithi092 ceph-mon[33966]: from='client.25666 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:07.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:07 smithi092 ceph-mon[33966]: from='client.25639 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:07.636 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:07.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:07.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:13:07.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:07 smithi090 ceph-mon[41670]: 16.5 deep-scrub starts 2024-04-14T15:13:07.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:07 smithi090 ceph-mon[41670]: 16.5 deep-scrub ok 2024-04-14T15:13:07.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:07 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:07.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:07 smithi090 ceph-mon[41670]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:13:07.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:07 smithi090 ceph-mon[41670]: from='client.25489 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi090-94522-2"}]': finished 2024-04-14T15:13:07.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:07 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1715119515' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi090-94662-2"}]': finished 2024-04-14T15:13:07.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:07 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/274362209' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi090-93436-10"}]': finished 2024-04-14T15:13:07.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:07 smithi090 ceph-mon[41670]: from='client.25379 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi090-93582-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-04-14T15:13:07.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:07 smithi090 ceph-mon[41670]: from='client.25376 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93998-6", "mode": "writeback"}]': finished 2024-04-14T15:13:07.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:07 smithi090 ceph-mon[41670]: from='client.25457 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi090-93709-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:07.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:07 smithi090 ceph-mon[41670]: from='client.25217 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi090-93476-10"}]': finished 2024-04-14T15:13:07.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:07 smithi090 ceph-mon[41670]: from='client.25636 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi090-93930-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:13:07.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:07 smithi090 ceph-mon[41670]: osdmap e119: 8 total, 8 up, 8 in 2024-04-14T15:13:07.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:07 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1713883211' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi090-93476-10"}]: dispatch 2024-04-14T15:13:07.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:07 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3852633403' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi090-93930-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi090-93930-12"}]: dispatch 2024-04-14T15:13:07.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:07 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/793350958' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi090-93342-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:07.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:07 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/274362209' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi090-93436-10"}]: dispatch 2024-04-14T15:13:07.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:07 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3749675023' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:07.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:07 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1715119515' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi090-94662-2"}]: dispatch 2024-04-14T15:13:07.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:07 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3684793142' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi090-93582-1","app":"app1","key":"key1"}]: dispatch 2024-04-14T15:13:07.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:07 smithi090 ceph-mon[30399]: 16.5 deep-scrub starts 2024-04-14T15:13:07.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:07 smithi090 ceph-mon[30399]: 16.5 deep-scrub ok 2024-04-14T15:13:07.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:07 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:07.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:07 smithi090 ceph-mon[30399]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:13:07.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:07 smithi090 ceph-mon[30399]: from='client.25489 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi090-94522-2"}]': finished 2024-04-14T15:13:07.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:07 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1715119515' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi090-94662-2"}]': finished 2024-04-14T15:13:07.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:07 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/274362209' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi090-93436-10"}]': finished 2024-04-14T15:13:07.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:07 smithi090 ceph-mon[30399]: from='client.25379 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi090-93582-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-04-14T15:13:07.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:07 smithi090 ceph-mon[30399]: from='client.25376 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93998-6", "mode": "writeback"}]': finished 2024-04-14T15:13:07.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:07 smithi090 ceph-mon[30399]: from='client.25457 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi090-93709-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:07.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:07 smithi090 ceph-mon[30399]: from='client.25217 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi090-93476-10"}]': finished 2024-04-14T15:13:07.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:07 smithi090 ceph-mon[30399]: from='client.25636 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi090-93930-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:13:07.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:07 smithi090 ceph-mon[30399]: osdmap e119: 8 total, 8 up, 8 in 2024-04-14T15:13:07.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:07 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1713883211' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi090-93476-10"}]: dispatch 2024-04-14T15:13:07.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:07 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3852633403' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi090-93930-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi090-93930-12"}]: dispatch 2024-04-14T15:13:07.695 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:07 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/793350958' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi090-93342-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:07.695 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:07 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/274362209' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi090-93436-10"}]: dispatch 2024-04-14T15:13:07.695 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:07 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3749675023' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:07.695 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:07 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1715119515' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi090-94662-2"}]: dispatch 2024-04-14T15:13:07.695 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:07 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3684793142' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi090-93582-1","app":"app1","key":"key1"}]: dispatch 2024-04-14T15:13:07.695 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:07 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1422064328' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:07.695 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:07 smithi090 ceph-mon[41670]: from='client.25636 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi090-93930-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi090-93930-12"}]: dispatch 2024-04-14T15:13:07.695 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:07 smithi090 ceph-mon[41670]: from='client.25217 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi090-93476-10"}]: dispatch 2024-04-14T15:13:07.695 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:07 smithi090 ceph-mon[41670]: from='client.25379 ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi090-93582-1","app":"app1","key":"key1"}]: dispatch 2024-04-14T15:13:07.695 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:07 smithi090 ceph-mon[41670]: from='client.25651 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi090-93342-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:07.695 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:07 smithi090 ceph-mon[41670]: from='client.25666 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:07.696 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:07 smithi090 ceph-mon[41670]: from='client.25639 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:07.696 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:07 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1422064328' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:07.696 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:07 smithi090 ceph-mon[30399]: from='client.25636 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi090-93930-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi090-93930-12"}]: dispatch 2024-04-14T15:13:07.696 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:07 smithi090 ceph-mon[30399]: from='client.25217 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi090-93476-10"}]: dispatch 2024-04-14T15:13:07.696 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:07 smithi090 ceph-mon[30399]: from='client.25379 ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi090-93582-1","app":"app1","key":"key1"}]: dispatch 2024-04-14T15:13:07.696 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:07 smithi090 ceph-mon[30399]: from='client.25651 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi090-93342-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:07.696 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:07 smithi090 ceph-mon[30399]: from='client.25666 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:07.696 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:07 smithi090 ceph-mon[30399]: from='client.25639 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:07.739 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:13:07.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:07.767 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:07.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:13:07.818 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock_pp: Running main() from gmock_main.cc 2024-04-14T15:13:07.818 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2024-04-14T15:13:07.818 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock_pp: [----------] Global test environment set-up. 2024-04-14T15:13:07.818 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2024-04-14T15:13:07.819 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock_pp: seed 93476 2024-04-14T15:13:07.819 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2024-04-14T15:13:07.819 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (499 ms) 2024-04-14T15:13:07.819 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2024-04-14T15:13:07.819 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (5 ms) 2024-04-14T15:13:07.819 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2024-04-14T15:13:07.819 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (1053 ms) 2024-04-14T15:13:07.819 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2024-04-14T15:13:07.819 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (1005 ms) 2024-04-14T15:13:07.820 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2024-04-14T15:13:07.820 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (3 ms) 2024-04-14T15:13:07.820 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2024-04-14T15:13:07.820 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (49 ms) 2024-04-14T15:13:07.820 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2024-04-14T15:13:07.820 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (5 ms) 2024-04-14T15:13:07.820 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2024-04-14T15:13:07.820 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (3 ms) 2024-04-14T15:13:07.820 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (2622 ms total) 2024-04-14T15:13:07.820 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock_pp: 2024-04-14T15:13:07.820 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2024-04-14T15:13:07.820 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2024-04-14T15:13:07.820 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (1170 ms) 2024-04-14T15:13:07.821 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2024-04-14T15:13:07.821 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (4 ms) 2024-04-14T15:13:07.821 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2024-04-14T15:13:07.821 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (1023 ms) 2024-04-14T15:13:07.821 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2024-04-14T15:13:07.821 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (1065 ms) 2024-04-14T15:13:07.821 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2024-04-14T15:13:07.821 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (4 ms) 2024-04-14T15:13:07.821 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2024-04-14T15:13:07.821 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (3 ms) 2024-04-14T15:13:07.821 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2024-04-14T15:13:07.821 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (4 ms) 2024-04-14T15:13:07.822 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2024-04-14T15:13:07.822 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (3 ms) 2024-04-14T15:13:07.822 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (3276 ms total) 2024-04-14T15:13:07.822 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock_pp: 2024-04-14T15:13:07.822 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock_pp: [----------] Global test environment tear-down 2024-04-14T15:13:07.822 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (13611 ms total) 2024-04-14T15:13:07.822 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock_pp: [ PASSED ] 16 tests. 2024-04-14T15:13:07.834 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock: Running main() from gmock_main.cc 2024-04-14T15:13:07.834 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2024-04-14T15:13:07.834 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock: [----------] Global test environment set-up. 2024-04-14T15:13:07.834 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock: [----------] 8 tests from LibRadosLock 2024-04-14T15:13:07.834 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2024-04-14T15:13:07.835 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (476 ms) 2024-04-14T15:13:07.835 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2024-04-14T15:13:07.835 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock: [ OK ] LibRadosLock.LockShared (24 ms) 2024-04-14T15:13:07.835 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2024-04-14T15:13:07.835 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (1049 ms) 2024-04-14T15:13:07.835 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2024-04-14T15:13:07.835 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (1064 ms) 2024-04-14T15:13:07.835 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2024-04-14T15:13:07.835 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (4 ms) 2024-04-14T15:13:07.835 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2024-04-14T15:13:07.835 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock: [ OK ] LibRadosLock.Unlock (3 ms) 2024-04-14T15:13:07.835 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2024-04-14T15:13:07.835 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (5 ms) 2024-04-14T15:13:07.836 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2024-04-14T15:13:07.836 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (2 ms) 2024-04-14T15:13:07.836 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock: [----------] 8 tests from LibRadosLock (2627 ms total) 2024-04-14T15:13:07.836 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock: 2024-04-14T15:13:07.836 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2024-04-14T15:13:07.836 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2024-04-14T15:13:07.836 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (1155 ms) 2024-04-14T15:13:07.836 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2024-04-14T15:13:07.836 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (3 ms) 2024-04-14T15:13:07.836 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2024-04-14T15:13:07.836 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (1022 ms) 2024-04-14T15:13:07.836 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2024-04-14T15:13:07.836 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (1040 ms) 2024-04-14T15:13:07.837 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2024-04-14T15:13:07.837 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (3 ms) 2024-04-14T15:13:07.837 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2024-04-14T15:13:07.837 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (4 ms) 2024-04-14T15:13:07.837 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2024-04-14T15:13:07.838 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (5 ms) 2024-04-14T15:13:07.838 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2024-04-14T15:13:07.838 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (3 ms) 2024-04-14T15:13:07.838 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (3235 ms total) 2024-04-14T15:13:07.838 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock: 2024-04-14T15:13:07.838 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock: [----------] Global test environment tear-down 2024-04-14T15:13:07.838 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (13658 ms total) 2024-04-14T15:13:07.838 INFO:tasks.workunit.client.0.smithi090.stdout: api_lock: [ PASSED ] 16 tests. 2024-04-14T15:13:07.864 INFO:tasks.workunit.client.0.smithi090.stdout: pool: Running main() from gmock_main.cc 2024-04-14T15:13:07.864 INFO:tasks.workunit.client.0.smithi090.stdout: pool: [==========] Running 6 tests from 1 test suite. 2024-04-14T15:13:07.864 INFO:tasks.workunit.client.0.smithi090.stdout: pool: [----------] Global test environment set-up. 2024-04-14T15:13:07.864 INFO:tasks.workunit.client.0.smithi090.stdout: pool: [----------] 6 tests from NeoRadosPools 2024-04-14T15:13:07.864 INFO:tasks.workunit.client.0.smithi090.stdout: pool: [ RUN ] NeoRadosPools.PoolList 2024-04-14T15:13:07.864 INFO:tasks.workunit.client.0.smithi090.stdout: pool: [ OK ] NeoRadosPools.PoolList (1538 ms) 2024-04-14T15:13:07.864 INFO:tasks.workunit.client.0.smithi090.stdout: pool: [ RUN ] NeoRadosPools.PoolLookup 2024-04-14T15:13:07.864 INFO:tasks.workunit.client.0.smithi090.stdout: pool: [ OK ] NeoRadosPools.PoolLookup (1958 ms) 2024-04-14T15:13:07.864 INFO:tasks.workunit.client.0.smithi090.stdout: pool: [ RUN ] NeoRadosPools.PoolLookupOtherInstance 2024-04-14T15:13:07.864 INFO:tasks.workunit.client.0.smithi090.stdout: pool: [ OK ] NeoRadosPools.PoolLookupOtherInstance (2039 ms) 2024-04-14T15:13:07.865 INFO:tasks.workunit.client.0.smithi090.stdout: pool: [ RUN ] NeoRadosPools.PoolDelete 2024-04-14T15:13:07.865 INFO:tasks.workunit.client.0.smithi090.stdout: pool: [ OK ] NeoRadosPools.PoolDelete (3700 ms) 2024-04-14T15:13:07.865 INFO:tasks.workunit.client.0.smithi090.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateDelete 2024-04-14T15:13:07.865 INFO:tasks.workunit.client.0.smithi090.stdout: pool: [ OK ] NeoRadosPools.PoolCreateDelete (1988 ms) 2024-04-14T15:13:07.865 INFO:tasks.workunit.client.0.smithi090.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateWithCrushRule 2024-04-14T15:13:07.865 INFO:tasks.workunit.client.0.smithi090.stdout: pool: [ OK ] NeoRadosPools.PoolCreateWithCrushRule (2044 ms) 2024-04-14T15:13:07.865 INFO:tasks.workunit.client.0.smithi090.stdout: pool: [----------] 6 tests from NeoRadosPools (13267 ms total) 2024-04-14T15:13:07.865 INFO:tasks.workunit.client.0.smithi090.stdout: pool: 2024-04-14T15:13:07.865 INFO:tasks.workunit.client.0.smithi090.stdout: pool: [----------] Global test environment tear-down 2024-04-14T15:13:07.865 INFO:tasks.workunit.client.0.smithi090.stdout: pool: [==========] 6 tests from 1 test suite ran. (13267 ms total) 2024-04-14T15:13:07.865 INFO:tasks.workunit.client.0.smithi090.stdout: pool: [ PASSED ] 6 tests. 2024-04-14T15:13:07.868 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:13:07.901 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:07.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:07.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:13:08.003 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:13:08.050 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:08.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:08.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:13:08.151 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:13:08.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:08.177 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:08.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:13:08.278 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:13:08.301 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:08.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:08.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:13:08.403 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:13:08.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:08 smithi090 ceph-mon[30399]: pgmap v125: 712 pgs: 4 creating+peering, 288 unknown, 420 active+clean; 145 MiB data, 767 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 31 MiB/s wr, 6 op/s 2024-04-14T15:13:08.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:08 smithi090 ceph-mon[30399]: 16.0 deep-scrub starts 2024-04-14T15:13:08.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:08 smithi090 ceph-mon[30399]: 16.0 deep-scrub ok 2024-04-14T15:13:08.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:08 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:08.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:08 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/274362209' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi090-93436-10"}]': finished 2024-04-14T15:13:08.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:08 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1715119515' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi090-94662-2"}]': finished 2024-04-14T15:13:08.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:08 smithi090 ceph-mon[30399]: from='client.25217 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi090-93476-10"}]': finished 2024-04-14T15:13:08.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:08 smithi090 ceph-mon[30399]: from='client.25379 ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi090-93582-1","app":"app1","key":"key1"}]': finished 2024-04-14T15:13:08.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:08 smithi090 ceph-mon[30399]: from='client.25651 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi090-93342-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:08.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:08 smithi090 ceph-mon[30399]: from='client.25666 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:08.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:08 smithi090 ceph-mon[30399]: from='client.25639 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:08.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:08 smithi090 ceph-mon[30399]: osdmap e120: 8 total, 8 up, 8 in 2024-04-14T15:13:08.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:08 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2461174822' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi090-94662-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:13:08.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:08 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:13:08.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:08 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:13:08.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:08 smithi090 ceph-mon[41670]: pgmap v125: 712 pgs: 4 creating+peering, 288 unknown, 420 active+clean; 145 MiB data, 767 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 31 MiB/s wr, 6 op/s 2024-04-14T15:13:08.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:08 smithi090 ceph-mon[41670]: 16.0 deep-scrub starts 2024-04-14T15:13:08.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:08 smithi090 ceph-mon[41670]: 16.0 deep-scrub ok 2024-04-14T15:13:08.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:08 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:08.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:08 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/274362209' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi090-93436-10"}]': finished 2024-04-14T15:13:08.445 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:08 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1715119515' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi090-94662-2"}]': finished 2024-04-14T15:13:08.445 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:08 smithi090 ceph-mon[41670]: from='client.25217 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi090-93476-10"}]': finished 2024-04-14T15:13:08.445 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:08 smithi090 ceph-mon[41670]: from='client.25379 ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi090-93582-1","app":"app1","key":"key1"}]': finished 2024-04-14T15:13:08.445 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:08 smithi090 ceph-mon[41670]: from='client.25651 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi090-93342-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:08.445 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:08 smithi090 ceph-mon[41670]: from='client.25666 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:08.445 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:08 smithi090 ceph-mon[41670]: from='client.25639 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:08.445 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:08 smithi090 ceph-mon[41670]: osdmap e120: 8 total, 8 up, 8 in 2024-04-14T15:13:08.445 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:08 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2461174822' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi090-94662-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:13:08.445 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:08 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:13:08.445 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:08 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:13:08.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:08.486 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:08.486 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:13:08.587 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:13:08.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:08 smithi092 ceph-mon[33966]: pgmap v125: 712 pgs: 4 creating+peering, 288 unknown, 420 active+clean; 145 MiB data, 767 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 31 MiB/s wr, 6 op/s 2024-04-14T15:13:08.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:08 smithi092 ceph-mon[33966]: 16.0 deep-scrub starts 2024-04-14T15:13:08.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:08 smithi092 ceph-mon[33966]: 16.0 deep-scrub ok 2024-04-14T15:13:08.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:08 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:08.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:08 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/274362209' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi090-93436-10"}]': finished 2024-04-14T15:13:08.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:08 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1715119515' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi090-94662-2"}]': finished 2024-04-14T15:13:08.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:08 smithi092 ceph-mon[33966]: from='client.25217 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi090-93476-10"}]': finished 2024-04-14T15:13:08.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:08 smithi092 ceph-mon[33966]: from='client.25379 ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi090-93582-1","app":"app1","key":"key1"}]': finished 2024-04-14T15:13:08.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:08 smithi092 ceph-mon[33966]: from='client.25651 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi090-93342-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:08.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:08 smithi092 ceph-mon[33966]: from='client.25666 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:08.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:08 smithi092 ceph-mon[33966]: from='client.25639 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:08.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:08 smithi092 ceph-mon[33966]: osdmap e120: 8 total, 8 up, 8 in 2024-04-14T15:13:08.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:08 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2461174822' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi090-94662-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:13:08.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:08 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:13:08.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:08 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:13:08.633 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:08.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:08.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:13:08.736 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:13:08.858 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:08.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:08.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:13:08.960 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:13:09.033 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:09.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:09.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:13:09.135 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:13:09.207 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:09.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:09.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:13:09.309 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:13:09.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:09.370 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:09.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:13:09.471 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:13:09.524 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:09.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:09.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:13:09.546 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:09 smithi090 ceph-mon[41670]: 16.7 deep-scrub starts 2024-04-14T15:13:09.547 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:09 smithi090 ceph-mon[41670]: 16.7 deep-scrub ok 2024-04-14T15:13:09.547 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:09 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:09.547 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:09 smithi090 ceph-mon[41670]: from='client.25636 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi090-93930-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi090-93930-12"}]': finished 2024-04-14T15:13:09.547 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:09 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2461174822' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi090-94662-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:13:09.547 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:09 smithi090 ceph-mon[41670]: osdmap e121: 8 total, 8 up, 8 in 2024-04-14T15:13:09.547 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:09 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2461174822' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi090-94662-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi090-94662-3"}]: dispatch 2024-04-14T15:13:09.547 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:09 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:09.547 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:09 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:09.547 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:09 smithi090 ceph-mon[30399]: 16.7 deep-scrub starts 2024-04-14T15:13:09.548 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:09 smithi090 ceph-mon[30399]: 16.7 deep-scrub ok 2024-04-14T15:13:09.548 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:09 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:09.548 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:09 smithi090 ceph-mon[30399]: from='client.25636 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi090-93930-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi090-93930-12"}]': finished 2024-04-14T15:13:09.548 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:09 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2461174822' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi090-94662-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:13:09.548 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:09 smithi090 ceph-mon[30399]: osdmap e121: 8 total, 8 up, 8 in 2024-04-14T15:13:09.548 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:09 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2461174822' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi090-94662-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi090-94662-3"}]: dispatch 2024-04-14T15:13:09.548 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:09 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:09.548 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:09 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:09.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:09 smithi092 ceph-mon[33966]: 16.7 deep-scrub starts 2024-04-14T15:13:09.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:09 smithi092 ceph-mon[33966]: 16.7 deep-scrub ok 2024-04-14T15:13:09.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:09 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:09.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:09 smithi092 ceph-mon[33966]: from='client.25636 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi090-93930-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi090-93930-12"}]': finished 2024-04-14T15:13:09.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:09 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2461174822' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi090-94662-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:13:09.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:09 smithi092 ceph-mon[33966]: osdmap e121: 8 total, 8 up, 8 in 2024-04-14T15:13:09.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:09 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2461174822' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi090-94662-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi090-94662-3"}]: dispatch 2024-04-14T15:13:09.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:09 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:09.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:09 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:09.626 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:13:09.649 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:09.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:09.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:13:09.752 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:13:09.861 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:09.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:09.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:13:09.918 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: Running main() from gmock_main.cc 2024-04-14T15:13:09.918 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2024-04-14T15:13:09.918 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2024-04-14T15:13:09.918 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2024-04-14T15:13:09.919 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2024-04-14T15:13:09.919 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: notify 2024-04-14T15:13:09.919 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (1032 ms) 2024-04-14T15:13:09.919 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2024-04-14T15:13:09.919 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (5 ms) 2024-04-14T15:13:09.919 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (1037 ms total) 2024-04-14T15:13:09.919 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: 2024-04-14T15:13:09.919 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2024-04-14T15:13:09.919 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2024-04-14T15:13:09.919 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: notify 2024-04-14T15:13:09.919 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (173 ms) 2024-04-14T15:13:09.919 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2024-04-14T15:13:09.919 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: notify 2024-04-14T15:13:09.919 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (3901 ms) 2024-04-14T15:13:09.920 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2024-04-14T15:13:09.920 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (5 ms) 2024-04-14T15:13:09.920 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2024-04-14T15:13:09.920 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (4 ms) 2024-04-14T15:13:09.920 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2024-04-14T15:13:09.920 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: handle_notify cookie 94679947476416 notify_id 511101108228 notifier_gid 25376 2024-04-14T15:13:09.920 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (5 ms) 2024-04-14T15:13:09.920 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2024-04-14T15:13:09.920 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: handle_notify cookie 94679947476416 notify_id 511101108225 notifier_gid 25376 2024-04-14T15:13:09.920 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (6 ms) 2024-04-14T15:13:09.920 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2024-04-14T15:13:09.920 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: handle_notify cookie 94679947497840 notify_id 511101108229 notifier_gid 25376 2024-04-14T15:13:09.920 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (5 ms) 2024-04-14T15:13:09.920 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2024-04-14T15:13:09.921 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: handle_notify cookie 94679947497840 notify_id 511101108230 notifier_gid 25376 2024-04-14T15:13:09.921 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (5 ms) 2024-04-14T15:13:09.921 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2024-04-14T15:13:09.921 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: handle_notify cookie 94679947497840 notify_id 511101108227 notifier_gid 25376 2024-04-14T15:13:09.921 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (5 ms) 2024-04-14T15:13:09.921 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2024-04-14T15:13:09.921 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: handle_notify cookie 94679948733872 notify_id 511101108225 notifier_gid 25376 2024-04-14T15:13:09.921 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (5 ms) 2024-04-14T15:13:09.921 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2024-04-14T15:13:09.921 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: trying... 2024-04-14T15:13:09.921 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: handle_notify cookie 94679948733872 notify_id 511101108226 notifier_gid 25376 2024-04-14T15:13:09.921 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: timed out 2024-04-14T15:13:09.921 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: flushing 2024-04-14T15:13:09.963 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:13:09.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:09.989 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:09.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:13:10.090 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:13:10.153 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:10.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:10.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:13:10.256 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:13:10.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:10.314 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:10.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:13:10.415 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:13:10.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:10.440 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:10.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:13:10.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:10 smithi090 ceph-mon[41670]: pgmap v128: 496 pgs: 54 creating+peering, 3 creating+activating, 25 unknown, 414 active+clean; 145 MiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-04-14T15:13:10.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:10 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:10.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:10 smithi090 ceph-mon[41670]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:13:10.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:10 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:10.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:10 smithi090 ceph-mon[41670]: osdmap e122: 8 total, 8 up, 8 in 2024-04-14T15:13:10.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:10 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3962832909' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi090-93724-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:10.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:10 smithi090 ceph-mon[41670]: from='client.26497 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi090-93724-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:10.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:10 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1828211824' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:10.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:10 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3997712235' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi090-93342-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:10.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:10 smithi090 ceph-mon[41670]: from='client.26551 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi090-93342-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:10.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:10 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3311439454' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:10.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:10 smithi090 ceph-mon[41670]: from='client.26545 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:10.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:10 smithi090 ceph-mon[41670]: from='client.26324 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:10.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:10 smithi090 ceph-mon[30399]: pgmap v128: 496 pgs: 54 creating+peering, 3 creating+activating, 25 unknown, 414 active+clean; 145 MiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-04-14T15:13:10.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:10 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:10.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:10 smithi090 ceph-mon[30399]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:13:10.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:10 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:10.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:10 smithi090 ceph-mon[30399]: osdmap e122: 8 total, 8 up, 8 in 2024-04-14T15:13:10.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:10 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3962832909' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi090-93724-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:10.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:10 smithi090 ceph-mon[30399]: from='client.26497 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi090-93724-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:10.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:10 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1828211824' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:10.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:10 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3997712235' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi090-93342-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:10.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:10 smithi090 ceph-mon[30399]: from='client.26551 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi090-93342-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:10.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:10 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3311439454' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:10.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:10 smithi090 ceph-mon[30399]: from='client.26545 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:10.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:10 smithi090 ceph-mon[30399]: from='client.26324 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:10.541 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:13:10.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:10 smithi092 ceph-mon[33966]: pgmap v128: 496 pgs: 54 creating+peering, 3 creating+activating, 25 unknown, 414 active+clean; 145 MiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-04-14T15:13:10.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:10 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:10.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:10 smithi092 ceph-mon[33966]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:13:10.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:10 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:10.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:10 smithi092 ceph-mon[33966]: osdmap e122: 8 total, 8 up, 8 in 2024-04-14T15:13:10.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:10 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3962832909' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi090-93724-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:10.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:10 smithi092 ceph-mon[33966]: from='client.26497 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi090-93724-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:10.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:10 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1828211824' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:10.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:10 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3997712235' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi090-93342-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:10.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:10 smithi092 ceph-mon[33966]: from='client.26551 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi090-93342-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:10.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:10 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3311439454' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:10.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:10 smithi092 ceph-mon[33966]: from='client.26545 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:10.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:10 smithi092 ceph-mon[33966]: from='client.26324 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:10.680 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-14T15:13:10.680 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:13:10.683 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-14T15:13:10.694 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:10.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:10.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:13:10.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:10.754+0000 7fcc3715f700 1 -- 172.21.15.90:0/346119666 >> v1:172.21.15.90:6790/0 conn(0x7fcc30111fd0 legacy=0x7fcc301143c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:10.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:10.754+0000 7fcc3715f700 1 -- 172.21.15.90:0/346119666 shutdown_connections 2024-04-14T15:13:10.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:10.754+0000 7fcc3715f700 1 -- 172.21.15.90:0/346119666 >> 172.21.15.90:0/346119666 conn(0x7fcc30076aa0 msgr2=0x7fcc30076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:13:10.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:10.756+0000 7fcc3715f700 1 -- 172.21.15.90:0/346119666 shutdown_connections 2024-04-14T15:13:10.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:10.757+0000 7fcc3715f700 1 -- 172.21.15.90:0/346119666 wait complete. 2024-04-14T15:13:10.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:10.758+0000 7fcc3715f700 1 librados: starting msgr at 2024-04-14T15:13:10.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:10.758+0000 7fcc3715f700 1 librados: starting objecter 2024-04-14T15:13:10.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:10.758+0000 7fcc3715f700 1 Processor -- start 2024-04-14T15:13:10.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:10.758+0000 7fcc3715f700 1 -- start start 2024-04-14T15:13:10.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:10.758+0000 7fcc3715f700 1 librados: setting wanted keys 2024-04-14T15:13:10.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:10.758+0000 7fcc3715f700 1 librados: calling monclient init 2024-04-14T15:13:10.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:10.759+0000 7fcc356fc700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fcc30105e30 0x7fcc301ace10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:39356/0 (socket says 172.21.15.90:39356) 2024-04-14T15:13:10.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:10.759+0000 7fcc356fc700 1 -- 172.21.15.90:0/4274269783 learned_addr learned my addr 172.21.15.90:0/4274269783 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:13:10.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:10.759+0000 7fcc3715f700 1 -- 172.21.15.90:0/4274269783 --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fcc301ae600 con 0x7fcc30105e30 2024-04-14T15:13:10.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:10.759+0000 7fcc2dffb700 10 client.?.objecter ms_handle_connect 0x7fcc30105e30 2024-04-14T15:13:10.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:10.759+0000 7fcc2dffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:13:10.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:10.759+0000 7fcc3715f700 1 -- 172.21.15.90:0/4274269783 --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fcc301af820 con 0x7fcc30115ae0 2024-04-14T15:13:10.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:10.759+0000 7fcc3715f700 1 -- 172.21.15.90:0/4274269783 --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fcc301ae810 con 0x7fcc30101110 2024-04-14T15:13:10.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:10.759+0000 7fcc2dffb700 1 -- 172.21.15.90:0/4274269783 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1814690370 0 0) 0x7fcc301ae600 con 0x7fcc30105e30 2024-04-14T15:13:10.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:10.759+0000 7fcc2dffb700 1 -- 172.21.15.90:0/4274269783 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc180035b0 con 0x7fcc30105e30 2024-04-14T15:13:10.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:10.762+0000 7fcc2dffb700 1 -- 172.21.15.90:0/4274269783 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1744759839 0 0) 0x7fcc180035b0 con 0x7fcc30105e30 2024-04-14T15:13:10.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:10.762+0000 7fcc2dffb700 1 -- 172.21.15.90:0/4274269783 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcc301ae600 con 0x7fcc30105e30 2024-04-14T15:13:10.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:10.762+0000 7fcc2dffb700 1 -- 172.21.15.90:0/4274269783 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fcc200018b0 con 0x7fcc30105e30 2024-04-14T15:13:10.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:10.762+0000 7fcc2dffb700 1 -- 172.21.15.90:0/4274269783 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 819490785 0 0) 0x7fcc301ae600 con 0x7fcc30105e30 2024-04-14T15:13:10.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:10.762+0000 7fcc2dffb700 1 -- 172.21.15.90:0/4274269783 >> v1:172.21.15.90:6790/0 conn(0x7fcc30101110 legacy=0x7fcc30102f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:10.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:10.762+0000 7fcc2dffb700 1 -- 172.21.15.90:0/4274269783 >> v1:172.21.15.92:6789/0 conn(0x7fcc30115ae0 legacy=0x7fcc30103640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:10.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:10.762+0000 7fcc2dffb700 1 -- 172.21.15.90:0/4274269783 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc301aea20 con 0x7fcc30105e30 2024-04-14T15:13:10.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:10.762+0000 7fcc3715f700 1 -- 172.21.15.90:0/4274269783 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fcc301aed10 con 0x7fcc30105e30 2024-04-14T15:13:10.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:10.763+0000 7fcc3715f700 10 client.18222.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:13:10.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:10.763+0000 7fcc2dffb700 1 -- 172.21.15.90:0/4274269783 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fcc20001c70 con 0x7fcc30105e30 2024-04-14T15:13:10.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:10.763+0000 7fcc3715f700 1 -- 172.21.15.90:0/4274269783 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fcc301af1f0 con 0x7fcc30105e30 2024-04-14T15:13:10.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:10.763+0000 7fcc3715f700 1 librados: init done 2024-04-14T15:13:10.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:10.763+0000 7fcc2dffb700 1 -- 172.21.15.90:0/4274269783 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fcc20005db0 con 0x7fcc30105e30 2024-04-14T15:13:10.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:10.765+0000 7fcc2dffb700 1 -- 172.21.15.90:0/4274269783 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fcc200070d0 con 0x7fcc30105e30 2024-04-14T15:13:10.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:10.766+0000 7fcc3715f700 1 -- 172.21.15.90:0/4274269783 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc1c005120 con 0x7fcc30105e30 2024-04-14T15:13:10.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:10.769+0000 7fcc2dffb700 1 -- 172.21.15.90:0/4274269783 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(122..122 src has 51..122) ==== 14713+0+0 (unknown 2872186498 0 0) 0x7fcc2008c460 con 0x7fcc30105e30 2024-04-14T15:13:10.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:10.769+0000 7fcc2dffb700 10 client.18222.objecter ms_dispatch 0x7fcc30001550 osd_map(122..122 src has 51..122) 2024-04-14T15:13:10.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:10.769+0000 7fcc2dffb700 3 client.18222.objecter handle_osd_map got epochs [122,122] > 0 2024-04-14T15:13:10.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:10.769+0000 7fcc2dffb700 3 client.18222.objecter handle_osd_map decoding full epoch 122 2024-04-14T15:13:10.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:10.770+0000 7fcc2dffb700 20 client.18222.objecter dump_active .. 0 homeless 2024-04-14T15:13:10.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:10.773+0000 7fcc2dffb700 10 client.18222.objecter ms_handle_connect 0x7fcc18071320 2024-04-14T15:13:10.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:10.779+0000 7fcc2dffb700 1 -- 172.21.15.90:0/4274269783 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fcc200584e0 con 0x7fcc30105e30 2024-04-14T15:13:10.802 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:13:11.017 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:11.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:11.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:13:11.119 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:13:11.231 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:11.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:11.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:13:11.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.253+0000 7fcc3715f700 1 -- 172.21.15.90:0/4274269783 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd rm-pg-upmap-items", "pgid": "4.d"} v 0) -- 0x7fcc1c005410 con 0x7fcc30105e30 2024-04-14T15:13:11.334 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:13:11.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:11.358 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:11.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:13:11.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:11 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:13:11.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:11 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:13:11.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:11 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:11.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:11 smithi090 ceph-mon[41670]: pool 'PoolQuotaPP_smithi090-93363-3' is full (reached quota's max_bytes: 4 KiB) 2024-04-14T15:13:11.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:11 smithi090 ceph-mon[41670]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-04-14T15:13:11.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:11 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2461174822' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi090-94662-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi090-94662-3"}]': finished 2024-04-14T15:13:11.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:11 smithi090 ceph-mon[41670]: from='client.26497 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi090-93724-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:11.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:11 smithi090 ceph-mon[41670]: from='client.26551 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi090-93342-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:11.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:11 smithi090 ceph-mon[41670]: from='client.26545 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:11.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:11 smithi090 ceph-mon[41670]: from='client.26324 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:11.445 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:11 smithi090 ceph-mon[41670]: osdmap e123: 8 total, 8 up, 8 in 2024-04-14T15:13:11.445 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:11 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3852633403' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi090-93930-12"}]: dispatch 2024-04-14T15:13:11.445 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:11 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:11.445 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:11 smithi090 ceph-mon[41670]: from='client.25636 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi090-93930-12"}]: dispatch 2024-04-14T15:13:11.445 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:11 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:11.445 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:11 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/4274269783' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.d"}]: dispatch 2024-04-14T15:13:11.445 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:11 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:13:11.445 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:11 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:13:11.445 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:11 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:11.445 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:11 smithi090 ceph-mon[30399]: pool 'PoolQuotaPP_smithi090-93363-3' is full (reached quota's max_bytes: 4 KiB) 2024-04-14T15:13:11.445 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:11 smithi090 ceph-mon[30399]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-04-14T15:13:11.445 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:11 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2461174822' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi090-94662-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi090-94662-3"}]': finished 2024-04-14T15:13:11.446 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:11 smithi090 ceph-mon[30399]: from='client.26497 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi090-93724-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:11.446 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:11 smithi090 ceph-mon[30399]: from='client.26551 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi090-93342-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:11.446 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:11 smithi090 ceph-mon[30399]: from='client.26545 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:11.446 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:11 smithi090 ceph-mon[30399]: from='client.26324 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:11.446 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:11 smithi090 ceph-mon[30399]: osdmap e123: 8 total, 8 up, 8 in 2024-04-14T15:13:11.446 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:11 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3852633403' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi090-93930-12"}]: dispatch 2024-04-14T15:13:11.446 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:11 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:11.446 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:11 smithi090 ceph-mon[30399]: from='client.25636 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi090-93930-12"}]: dispatch 2024-04-14T15:13:11.446 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:11 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:11.446 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:11 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/4274269783' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.d"}]: dispatch 2024-04-14T15:13:11.459 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:13:11.483 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:11.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:11.484 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:13:11.584 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:13:11.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:11 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:13:11.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:11 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:13:11.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:11 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:11.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:11 smithi092 ceph-mon[33966]: pool 'PoolQuotaPP_smithi090-93363-3' is full (reached quota's max_bytes: 4 KiB) 2024-04-14T15:13:11.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:11 smithi092 ceph-mon[33966]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-04-14T15:13:11.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:11 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2461174822' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi090-94662-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi090-94662-3"}]': finished 2024-04-14T15:13:11.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:11 smithi092 ceph-mon[33966]: from='client.26497 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi090-93724-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:11.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:11 smithi092 ceph-mon[33966]: from='client.26551 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi090-93342-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:11.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:11 smithi092 ceph-mon[33966]: from='client.26545 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:11.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:11 smithi092 ceph-mon[33966]: from='client.26324 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:11.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:11 smithi092 ceph-mon[33966]: osdmap e123: 8 total, 8 up, 8 in 2024-04-14T15:13:11.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:11 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3852633403' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi090-93930-12"}]: dispatch 2024-04-14T15:13:11.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:11 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:11.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:11 smithi092 ceph-mon[33966]: from='client.25636 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi090-93930-12"}]: dispatch 2024-04-14T15:13:11.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:11 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:11.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:11 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/4274269783' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.d"}]: dispatch 2024-04-14T15:13:11.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.252+0000 7f5797fff700 1 -- 172.21.15.90:0/4140215309 >> v1:172.21.15.92:6789/0 conn(0x7f579810fba0 legacy=0x7f579810ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:11.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.254+0000 7f5797fff700 1 -- 172.21.15.90:0/4140215309 shutdown_connections 2024-04-14T15:13:11.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.254+0000 7f5797fff700 1 -- 172.21.15.90:0/4140215309 >> 172.21.15.90:0/4140215309 conn(0x7f579806d810 msgr2=0x7f579806dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:13:11.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.254+0000 7f5797fff700 1 -- 172.21.15.90:0/4140215309 shutdown_connections 2024-04-14T15:13:11.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.254+0000 7f5797fff700 1 -- 172.21.15.90:0/4140215309 wait complete. 2024-04-14T15:13:11.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.255+0000 7f5797fff700 1 librados: starting msgr at 2024-04-14T15:13:11.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.255+0000 7f5797fff700 1 librados: starting objecter 2024-04-14T15:13:11.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.255+0000 7f5797fff700 1 Processor -- start 2024-04-14T15:13:11.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.255+0000 7f5797fff700 1 -- start start 2024-04-14T15:13:11.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.255+0000 7f5797fff700 1 librados: setting wanted keys 2024-04-14T15:13:11.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.255+0000 7f5797fff700 1 librados: calling monclient init 2024-04-14T15:13:11.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.256+0000 7f5797fff700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f57981b6430 con 0x7f5798110690 2024-04-14T15:13:11.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.256+0000 7f5797fff700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f57981b7650 con 0x7f579811e130 2024-04-14T15:13:11.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.256+0000 7f5797fff700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f57981b8890 con 0x7f579810fba0 2024-04-14T15:13:11.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.256+0000 7f57967fc700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f5798110690 0x7f57981b13e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:40420/0 (socket says 172.21.15.90:40420) 2024-04-14T15:13:11.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.256+0000 7f57967fc700 1 -- 172.21.15.90:0/3058164676 learned_addr learned my addr 172.21.15.90:0/3058164676 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:13:11.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.256+0000 7f579ca17700 10 client.?.objecter ms_handle_connect 0x7f5798110690 2024-04-14T15:13:11.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.256+0000 7f579ca17700 10 client.?.objecter resend_mon_ops 2024-04-14T15:13:11.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.256+0000 7f579ca17700 10 client.?.objecter ms_handle_connect 0x7f579810fba0 2024-04-14T15:13:11.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.256+0000 7f579ca17700 10 client.?.objecter resend_mon_ops 2024-04-14T15:13:11.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.256+0000 7f579ca17700 10 client.?.objecter ms_handle_connect 0x7f579811e130 2024-04-14T15:13:11.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.256+0000 7f579ca17700 10 client.?.objecter resend_mon_ops 2024-04-14T15:13:11.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.256+0000 7f579ca17700 1 -- 172.21.15.90:0/3058164676 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3602615895 0 0) 0x7f57981b6430 con 0x7f5798110690 2024-04-14T15:13:11.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.256+0000 7f579ca17700 1 -- 172.21.15.90:0/3058164676 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f57800031f0 con 0x7f5798110690 2024-04-14T15:13:11.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.256+0000 7f579ca17700 1 -- 172.21.15.90:0/3058164676 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3456710641 0 0) 0x7f57800031f0 con 0x7f5798110690 2024-04-14T15:13:11.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.256+0000 7f579ca17700 1 -- 172.21.15.90:0/3058164676 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f57981b6430 con 0x7f5798110690 2024-04-14T15:13:11.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.256+0000 7f579ca17700 1 -- 172.21.15.90:0/3058164676 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f57880019b0 con 0x7f5798110690 2024-04-14T15:13:11.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.257+0000 7f579ca17700 1 -- 172.21.15.90:0/3058164676 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2599399212 0 0) 0x7f57981b6430 con 0x7f5798110690 2024-04-14T15:13:11.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.257+0000 7f579ca17700 1 -- 172.21.15.90:0/3058164676 >> v1:172.21.15.90:6790/0 conn(0x7f579810fba0 legacy=0x7f579810f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:11.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.257+0000 7f579ca17700 1 -- 172.21.15.90:0/3058164676 >> v1:172.21.15.92:6789/0 conn(0x7f579811e130 legacy=0x7f57981b4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:11.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.257+0000 7f579ca17700 1 -- 172.21.15.90:0/3058164676 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57981b9ad0 con 0x7f5798110690 2024-04-14T15:13:11.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.257+0000 7f5797fff700 1 -- 172.21.15.90:0/3058164676 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f57981b8ac0 con 0x7f5798110690 2024-04-14T15:13:11.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.257+0000 7f5797fff700 10 client.18510.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:13:11.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.257+0000 7f5797fff700 1 -- 172.21.15.90:0/3058164676 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f57981b8fc0 con 0x7f5798110690 2024-04-14T15:13:11.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.257+0000 7f5797fff700 1 librados: init done 2024-04-14T15:13:11.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.257+0000 7f579ca17700 1 -- 172.21.15.90:0/3058164676 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f5788001c70 con 0x7f5798110690 2024-04-14T15:13:11.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.257+0000 7f579ca17700 1 -- 172.21.15.90:0/3058164676 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f5788005d80 con 0x7f5798110690 2024-04-14T15:13:11.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.259+0000 7f579ca17700 1 -- 172.21.15.90:0/3058164676 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f578801c930 con 0x7f5798110690 2024-04-14T15:13:11.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.262+0000 7f579ca17700 1 -- 172.21.15.90:0/3058164676 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(123..123 src has 51..123) ==== 15521+0+0 (unknown 385656418 0 0) 0x7f578808c910 con 0x7f5798110690 2024-04-14T15:13:11.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.262+0000 7f579ca17700 10 client.18510.objecter ms_dispatch 0x7f5798001550 osd_map(123..123 src has 51..123) 2024-04-14T15:13:11.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.262+0000 7f579ca17700 3 client.18510.objecter handle_osd_map got epochs [123,123] > 0 2024-04-14T15:13:11.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.262+0000 7f579ca17700 3 client.18510.objecter handle_osd_map decoding full epoch 123 2024-04-14T15:13:11.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.262+0000 7f579ca17700 10 client.18510.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:13:11.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.262+0000 7f579ca17700 1 -- 172.21.15.90:0/3058164676 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=124}) -- 0x7f57981b6430 con 0x7f5798110690 2024-04-14T15:13:11.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.262+0000 7f579ca17700 20 client.18510.objecter dump_active .. 0 homeless 2024-04-14T15:13:11.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.262+0000 7f579ca17700 10 client.18510.objecter ms_handle_connect 0x7f57800713d0 2024-04-14T15:13:11.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.263+0000 7f5797fff700 1 -- 172.21.15.90:0/3058164676 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5784005120 con 0x7f5798110690 2024-04-14T15:13:11.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.272+0000 7f579ca17700 1 -- 172.21.15.90:0/3058164676 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f5788057660 con 0x7f5798110690 2024-04-14T15:13:11.717 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-14T15:13:11.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.719+0000 7f5797fff700 1 -- 172.21.15.90:0/3058164676 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7f5784005410 con 0x7f5798110690 2024-04-14T15:13:11.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.807+0000 7f579ca17700 1 -- 172.21.15.90:0/3058164676 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v124) ==== 91+0+0 (unknown 1025991510 0 0) 0x7f578805b310 con 0x7f5798110690 2024-04-14T15:13:11.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.807+0000 7fcc2dffb700 1 -- 172.21.15.90:0/4274269783 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd rm-pg-upmap-items", "pgid": "4.d"}]=0 clear 4.d pg_upmap_items mapping v124) ==== 116+0+0 (unknown 4226880474 0 0) 0x7fcc2005c190 con 0x7fcc30105e30 2024-04-14T15:13:11.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.811+0000 7fcc3715f700 10 librados: watch_flush enter 2024-04-14T15:13:11.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.811+0000 7fcc3715f700 10 librados: watch_flush exit 2024-04-14T15:13:11.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.811+0000 7fcc3715f700 20 client.18222.objecter shutdown clearing up homeless session... 2024-04-14T15:13:11.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.811+0000 7fcc3715f700 10 client.18222.objecter successfully canceled tick 2024-04-14T15:13:11.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.811+0000 7f5797fff700 10 librados: watch_flush enter 2024-04-14T15:13:11.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.811+0000 7f5797fff700 10 librados: watch_flush exit 2024-04-14T15:13:11.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.811+0000 7f5797fff700 20 client.18510.objecter shutdown clearing up homeless session... 2024-04-14T15:13:11.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.811+0000 7f5797fff700 10 client.18510.objecter successfully canceled tick 2024-04-14T15:13:11.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.811+0000 7f5797fff700 1 -- 172.21.15.90:0/3058164676 >> v1:172.21.15.90:6800/420935427 conn(0x7f57800713d0 legacy=0x7f5780073890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:11.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.811+0000 7f5797fff700 1 -- 172.21.15.90:0/3058164676 >> v1:172.21.15.90:6789/0 conn(0x7f5798110690 legacy=0x7f57981b13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:11.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.815+0000 7f5797fff700 1 -- 172.21.15.90:0/3058164676 shutdown_connections 2024-04-14T15:13:11.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.815+0000 7f5797fff700 1 -- 172.21.15.90:0/3058164676 >> 172.21.15.90:0/3058164676 conn(0x7f579806d810 msgr2=0x7f5798072ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:13:11.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.821+0000 7f5797fff700 1 -- 172.21.15.90:0/3058164676 shutdown_connections 2024-04-14T15:13:11.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.815+0000 7fcc3715f700 1 -- 172.21.15.90:0/4274269783 >> v1:172.21.15.90:6800/420935427 conn(0x7fcc18071320 legacy=0x7fcc180737e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:11.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.815+0000 7fcc3715f700 1 -- 172.21.15.90:0/4274269783 >> v1:172.21.15.90:6789/0 conn(0x7fcc30105e30 legacy=0x7fcc301ace10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:11.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.817+0000 7fcc3715f700 1 -- 172.21.15.90:0/4274269783 shutdown_connections 2024-04-14T15:13:11.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.817+0000 7fcc3715f700 1 -- 172.21.15.90:0/4274269783 >> 172.21.15.90:0/4274269783 conn(0x7fcc30076aa0 msgr2=0x7fcc30100070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:13:11.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.821+0000 7fcc3715f700 1 -- 172.21.15.90:0/4274269783 shutdown_connections 2024-04-14T15:13:11.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.821+0000 7fcc3715f700 1 -- 172.21.15.90:0/4274269783 wait complete. 2024-04-14T15:13:11.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.821+0000 7fcc3715f700 1 librados: shutdown 2024-04-14T15:13:11.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.824+0000 7f5797fff700 1 -- 172.21.15.90:0/3058164676 wait complete. 2024-04-14T15:13:11.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:11.824+0000 7f5797fff700 1 librados: shutdown 2024-04-14T15:13:11.851 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is unset 2024-04-14T15:13:11.851 INFO:teuthology.orchestra.run.smithi090.stderr:clear 4.d pg_upmap_items mapping 2024-04-14T15:13:11.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:11.852 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:11.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:13:11.953 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:13:11.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:11.979 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:11.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:13:12.080 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:13:12.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:12.164 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:12.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:13:12.265 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:13:12.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:12.354 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:12.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:13:12.456 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:13:12.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:12.551 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:12.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:13:12.575 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:12 smithi090 ceph-mon[30399]: pgmap v131: 752 pgs: 64 creating+peering, 296 unknown, 392 active+clean; 145 MiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.5 KiB/s wr, 4 op/s 2024-04-14T15:13:12.576 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:12 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:12.576 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:12 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3058164676' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:13:12.576 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:12 smithi090 ceph-mon[30399]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:13:12.576 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:12 smithi090 ceph-mon[30399]: from='client.25636 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi090-93930-12"}]': finished 2024-04-14T15:13:12.576 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:12 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:12.576 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:12 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/4274269783' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.d"}]': finished 2024-04-14T15:13:12.576 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:12 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3058164676' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-14T15:13:12.576 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:12 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3852633403' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi090-93930-12"}]: dispatch 2024-04-14T15:13:12.576 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:12 smithi090 ceph-mon[30399]: osdmap e124: 8 total, 8 up, 8 in 2024-04-14T15:13:12.576 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:12 smithi090 ceph-mon[30399]: from='client.25636 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi090-93930-12"}]: dispatch 2024-04-14T15:13:12.577 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:12 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:13:12.577 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:12 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:13:12.577 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:12 smithi090 ceph-mon[41670]: pgmap v131: 752 pgs: 64 creating+peering, 296 unknown, 392 active+clean; 145 MiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.5 KiB/s wr, 4 op/s 2024-04-14T15:13:12.577 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:12 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:12.577 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:12 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3058164676' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:13:12.577 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:12 smithi090 ceph-mon[41670]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:13:12.577 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:12 smithi090 ceph-mon[41670]: from='client.25636 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi090-93930-12"}]': finished 2024-04-14T15:13:12.577 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:12 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:12.577 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:12 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/4274269783' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.d"}]': finished 2024-04-14T15:13:12.577 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:12 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3058164676' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-14T15:13:12.577 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:12 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3852633403' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi090-93930-12"}]: dispatch 2024-04-14T15:13:12.578 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:12 smithi090 ceph-mon[41670]: osdmap e124: 8 total, 8 up, 8 in 2024-04-14T15:13:12.578 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:12 smithi090 ceph-mon[41670]: from='client.25636 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi090-93930-12"}]: dispatch 2024-04-14T15:13:12.578 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:12 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:13:12.578 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:12 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:13:12.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:12 smithi092 ceph-mon[33966]: pgmap v131: 752 pgs: 64 creating+peering, 296 unknown, 392 active+clean; 145 MiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.5 KiB/s wr, 4 op/s 2024-04-14T15:13:12.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:12 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:12.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:12 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3058164676' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:13:12.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:12 smithi092 ceph-mon[33966]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:13:12.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:12 smithi092 ceph-mon[33966]: from='client.25636 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi090-93930-12"}]': finished 2024-04-14T15:13:12.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:12 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:12.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:12 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/4274269783' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.d"}]': finished 2024-04-14T15:13:12.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:12 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3058164676' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-14T15:13:12.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:12 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3852633403' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi090-93930-12"}]: dispatch 2024-04-14T15:13:12.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:12 smithi092 ceph-mon[33966]: osdmap e124: 8 total, 8 up, 8 in 2024-04-14T15:13:12.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:12 smithi092 ceph-mon[33966]: from='client.25636 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi090-93930-12"}]: dispatch 2024-04-14T15:13:12.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:12 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:13:12.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:12 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:13:12.652 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:13:12.742 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:12.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:12.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:13:12.840 INFO:tasks.workunit.client.0.smithi090.stdout: api_watc api_watch_notify: Running main() from gmock_main.cc 2024-04-14T15:13:12.840 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2024-04-14T15:13:12.840 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify: [----------] Global test environment set-up. 2024-04-14T15:13:12.840 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2024-04-14T15:13:12.840 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2024-04-14T15:13:12.840 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify: watch_notify_test_cb 2024-04-14T15:13:12.841 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (514 ms) 2024-04-14T15:13:12.841 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2024-04-14T15:13:12.841 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify: watch_notify2_test_errcb cookie 93826558502160 err -107 2024-04-14T15:13:12.841 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-04-14T15:13:12.841 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (34 ms) 2024-04-14T15:13:12.841 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2024-04-14T15:13:12.841 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify: watch_notify2_test_errcb cookie 93826558504288 err -107 2024-04-14T15:13:12.841 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-04-14T15:13:12.841 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (12 ms) 2024-04-14T15:13:12.841 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2024-04-14T15:13:12.841 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify: watch_notify2_test_cb from 24889 notify_id 463856467969 cookie 93826558504288 2024-04-14T15:13:12.841 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (15 ms) 2024-04-14T15:13:12.841 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2024-04-14T15:13:12.841 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify: watch_notify2_test_cb from 24889 notify_id 463856467969 cookie 93826558555440 2024-04-14T15:13:12.842 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (11 ms) 2024-04-14T15:13:12.842 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2024-04-14T15:13:12.842 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify: watch_notify2_test_cb from 24889 notify_id 463856467968 cookie 93826558555440 2024-04-14T15:13:12.842 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (9 ms) 2024-04-14T15:13:12.842 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2024-04-14T15:13:12.842 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify: watch_notify2_test_cb from 24889 notify_id 463856467969 cookie 93826558529888 2024-04-14T15:13:12.842 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify: watch_notify2_test_cb from 24889 notify_id 463856467969 cookie 93826558555440 2024-04-14T15:13:12.842 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (15 ms) 2024-04-14T15:13:12.842 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2024-04-14T15:13:12.842 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify: watch_notify2_test_cb from 24889 notify_id 463856467970 cookie 93826558555440 2024-04-14T15:13:12.842 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify: watch_notify2_test_cb from 24889 notify_id 468151435267 cookie 93826558555440 2024-04-14T15:13:12.842 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (3006 ms) 2024-04-14T15:13:12.842 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2024-04-14T15:13:12.842 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2024-04-14T15:13:12.843 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify: watch_notify2_test_errcb cookie 93826558555440 err -107 2024-04-14T15:13:12.843 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify: watch_notify2_test_cb from 24889 notify_id 498216206339 cookie 93826558555440 2024-04-14T15:13:12.843 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (5007 ms) 2024-04-14T15:13:12.843 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2024-04-14T15:13:12.843 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2024-04-14T15:13:12.843 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify: watch_notify2_test_errcb cookie 93826558555440 err -107 2024-04-14T15:13:12.843 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (1005 ms) 2024-04-14T15:13:12.843 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (9628 ms total) 2024-04-14T15:13:12.843 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify: 2024-04-14T15:13:12.843 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2024-04-14T15:13:12.843 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2024-04-14T15:13:12.843 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify: watch_notify_test_cb 2024-04-14T15:13:12.843 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (1174 ms) 2024-04-14T15:13:12.843 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (1174 ms total) 2024-04-14T15:13:12.843 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify: 2024-04-14T15:13:12.844 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify: [----------] Global test environment tear-down 2024-04-14T15:13:12.844 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (18518 ms total) 2024-04-14T15:13:12.844 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:13:12.845 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify: [ PASSED ] 11 tests. 2024-04-14T15:13:13.022 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:13:13.032 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:13:13.056 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:13.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:13.058 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:13:13.158 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:13:13.182 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:13.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:13.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:13:13.284 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:13:13.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:13.311 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:13.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:13:13.412 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:13:13.458 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:13 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:13:13.458 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:13 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:13:13.458 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:13 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:13.458 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:13 smithi090 ceph-mon[30399]: from='client.25636 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi090-93930-12"}]': finished 2024-04-14T15:13:13.458 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:13 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-7", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:13:13.458 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:13 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-7"}]: dispatch 2024-04-14T15:13:13.458 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:13 smithi090 ceph-mon[30399]: osdmap e125: 8 total, 8 up, 8 in 2024-04-14T15:13:13.458 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:13 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/854221802' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi090-93342-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:13.459 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:13 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-7"}]: dispatch 2024-04-14T15:13:13.459 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:13 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1969980219' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:13.459 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:13 smithi090 ceph-mon[30399]: from='client.29440 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi090-93342-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:13.459 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:13 smithi090 ceph-mon[30399]: from='client.29461 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:13.459 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:13 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:13:13.459 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:13 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:13:13.459 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:13 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:13.459 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:13 smithi090 ceph-mon[41670]: from='client.25636 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi090-93930-12"}]': finished 2024-04-14T15:13:13.459 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:13 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-7", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:13:13.459 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:13 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-7"}]: dispatch 2024-04-14T15:13:13.459 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:13 smithi090 ceph-mon[41670]: osdmap e125: 8 total, 8 up, 8 in 2024-04-14T15:13:13.460 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:13 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/854221802' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi090-93342-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:13.460 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:13 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-7"}]: dispatch 2024-04-14T15:13:13.460 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:13 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1969980219' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:13.460 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:13 smithi090 ceph-mon[41670]: from='client.29440 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi090-93342-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:13.460 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:13 smithi090 ceph-mon[41670]: from='client.29461 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:13.507 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:13.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:13.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:13:13.609 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:13:13.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:13 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:13:13.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:13 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:13:13.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:13 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:13.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:13 smithi092 ceph-mon[33966]: from='client.25636 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi090-93930-12"}]': finished 2024-04-14T15:13:13.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:13 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-7", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:13:13.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:13 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-7"}]: dispatch 2024-04-14T15:13:13.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:13 smithi092 ceph-mon[33966]: osdmap e125: 8 total, 8 up, 8 in 2024-04-14T15:13:13.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:13 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/854221802' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi090-93342-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:13.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:13 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-7"}]: dispatch 2024-04-14T15:13:13.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:13 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1969980219' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:13.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:13 smithi092 ceph-mon[33966]: from='client.29440 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi090-93342-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:13.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:13 smithi092 ceph-mon[33966]: from='client.29461 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:13.698 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:13.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:13.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:13:13.801 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:13:13.828 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:13.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:13.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:13:13.930 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:13:13.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:13.958 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:13.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:13:14.059 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:13:14.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:14.212 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:14.212 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:13:14.290 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:14 smithi090 ceph-mon[30399]: 6.1e scrub starts 2024-04-14T15:13:14.290 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:14 smithi090 ceph-mon[30399]: 6.1e scrub ok 2024-04-14T15:13:14.290 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:14 smithi090 ceph-mon[30399]: pgmap v134: 688 pgs: 32 creating+peering, 264 unknown, 392 active+clean; 145 MiB data, 779 MiB used, 714 GiB / 715 GiB avail 2024-04-14T15:13:14.290 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:14 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:14.290 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:14 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-7"}]': finished 2024-04-14T15:13:14.290 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:14 smithi090 ceph-mon[30399]: from='client.29440 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi090-93342-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:14.290 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:14 smithi090 ceph-mon[30399]: from='client.29461 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:14.290 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:14 smithi090 ceph-mon[30399]: osdmap e126: 8 total, 8 up, 8 in 2024-04-14T15:13:14.290 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:14 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:13:14.290 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:14 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:13:14.291 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:13:13 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:13:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:13:14.291 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:14 smithi090 ceph-mon[41670]: 6.1e scrub starts 2024-04-14T15:13:14.291 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:14 smithi090 ceph-mon[41670]: 6.1e scrub ok 2024-04-14T15:13:14.291 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:14 smithi090 ceph-mon[41670]: pgmap v134: 688 pgs: 32 creating+peering, 264 unknown, 392 active+clean; 145 MiB data, 779 MiB used, 714 GiB / 715 GiB avail 2024-04-14T15:13:14.291 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:14 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:14.291 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:14 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-7"}]': finished 2024-04-14T15:13:14.291 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:14 smithi090 ceph-mon[41670]: from='client.29440 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi090-93342-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:14.292 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:14 smithi090 ceph-mon[41670]: from='client.29461 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:14.292 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:14 smithi090 ceph-mon[41670]: osdmap e126: 8 total, 8 up, 8 in 2024-04-14T15:13:14.292 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:14 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:13:14.292 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:14 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:13:14.316 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:13:14.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:14.394 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:14.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:13:14.495 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:13:14.520 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:14.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:14.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:13:14.621 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:13:14.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:14 smithi092 ceph-mon[33966]: 6.1e scrub starts 2024-04-14T15:13:14.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:14 smithi092 ceph-mon[33966]: 6.1e scrub ok 2024-04-14T15:13:14.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:14 smithi092 ceph-mon[33966]: pgmap v134: 688 pgs: 32 creating+peering, 264 unknown, 392 active+clean; 145 MiB data, 779 MiB used, 714 GiB / 715 GiB avail 2024-04-14T15:13:14.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:14 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:14.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:14 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-7"}]': finished 2024-04-14T15:13:14.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:14 smithi092 ceph-mon[33966]: from='client.29440 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi090-93342-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:14.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:14 smithi092 ceph-mon[33966]: from='client.29461 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-94066-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:14.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:14 smithi092 ceph-mon[33966]: osdmap e126: 8 total, 8 up, 8 in 2024-04-14T15:13:14.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:14 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:13:14.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:14 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:13:14.698 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:14.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:14.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:13:14.799 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:13:14.825 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:14.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:14.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:13:14.927 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:13:14.953 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:14.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:14.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:13:15.055 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:13:15.081 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:15.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:15.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:13:15.122 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:13:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:13:15.182 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:13:15.289 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:15 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:13:15.289 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:15 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:13:15.290 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:15 smithi090 ceph-mon[30399]: 5.17 scrub starts 2024-04-14T15:13:15.290 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:15 smithi090 ceph-mon[30399]: 5.17 scrub ok 2024-04-14T15:13:15.290 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:15 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:15.290 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:15 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:13:15.290 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:15 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:13:15.290 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:15 smithi090 ceph-mon[30399]: osdmap e127: 8 total, 8 up, 8 in 2024-04-14T15:13:15.290 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:15 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/4121512451' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:15.290 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:15 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-7"}]: dispatch 2024-04-14T15:13:15.290 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:15 smithi090 ceph-mon[30399]: from='client.31396 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:15.290 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:15 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-7"}]: dispatch 2024-04-14T15:13:15.290 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:15 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:13:15.290 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:15 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2461174822' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi090-94662-3"}]: dispatch 2024-04-14T15:13:15.291 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:15.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:15.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:13:15.393 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:13:15.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:15.555 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:15.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:13:15.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:15 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:13:15.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:15 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:13:15.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:15 smithi092 ceph-mon[33966]: 5.17 scrub starts 2024-04-14T15:13:15.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:15 smithi092 ceph-mon[33966]: 5.17 scrub ok 2024-04-14T15:13:15.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:15 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:15.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:15 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:13:15.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:15 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:13:15.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:15 smithi092 ceph-mon[33966]: osdmap e127: 8 total, 8 up, 8 in 2024-04-14T15:13:15.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:15 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/4121512451' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:15.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:15 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-7"}]: dispatch 2024-04-14T15:13:15.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:15 smithi092 ceph-mon[33966]: from='client.31396 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:15.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:15 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-7"}]: dispatch 2024-04-14T15:13:15.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:15 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:13:15.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:15 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2461174822' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi090-94662-3"}]: dispatch 2024-04-14T15:13:15.656 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:13:15.688 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-14T15:13:15.695 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:15 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:13:15.696 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:15 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:13:15.696 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:15 smithi090 ceph-mon[41670]: 5.17 scrub starts 2024-04-14T15:13:15.696 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:15 smithi090 ceph-mon[41670]: 5.17 scrub ok 2024-04-14T15:13:15.696 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:15 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:15.696 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:15 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:13:15.696 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:15 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:13:15.696 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:15 smithi090 ceph-mon[41670]: osdmap e127: 8 total, 8 up, 8 in 2024-04-14T15:13:15.696 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:15 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/4121512451' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:15.696 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:15 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-7"}]: dispatch 2024-04-14T15:13:15.696 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:15 smithi090 ceph-mon[41670]: from='client.31396 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:15.696 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:15 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-7"}]: dispatch 2024-04-14T15:13:15.696 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:15 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:13:15.696 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:15 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2461174822' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi090-94662-3"}]: dispatch 2024-04-14T15:13:15.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:15.742 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:15.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:13:15.843 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:13:15.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:15.994 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:15.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:13:16.095 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:13:16.213 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:16.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:16.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:13:16.295 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:16 smithi092 ceph-mon[33966]: 4.1b scrub starts 2024-04-14T15:13:16.295 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:16 smithi092 ceph-mon[33966]: 4.1b scrub ok 2024-04-14T15:13:16.295 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:16 smithi092 ceph-mon[33966]: pgmap v137: 648 pgs: 64 creating+peering, 11 creating+activating, 2 active+clean+snaptrim, 32 unknown, 539 active+clean; 145 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 3.7 KiB/s rd, 5.0 KiB/s wr, 7 op/s; 27 B/s, 1 objects/s recovering 2024-04-14T15:13:16.295 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:16 smithi092 ceph-mon[33966]: 2.1a scrub starts 2024-04-14T15:13:16.295 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:16 smithi092 ceph-mon[33966]: 2.1a scrub ok 2024-04-14T15:13:16.295 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:16 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:16.295 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:16 smithi092 ceph-mon[33966]: Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:13:16.295 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:16 smithi092 ceph-mon[33966]: from='client.31396 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:16.295 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:16 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-7"}]': finished 2024-04-14T15:13:16.296 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:16 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2461174822' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi090-94662-3"}]': finished 2024-04-14T15:13:16.296 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:16 smithi092 ceph-mon[33966]: osdmap e128: 8 total, 8 up, 8 in 2024-04-14T15:13:16.296 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:16 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2461174822' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi090-94662-3"}]: dispatch 2024-04-14T15:13:16.296 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:16 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3264268010' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi090-93342-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:16.296 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:16 smithi092 ceph-mon[33966]: from='client.32359 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi090-93342-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:16.314 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:13:16.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:16.470 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:16.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:13:16.571 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:13:16.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:16 smithi090 ceph-mon[30399]: 4.1b scrub starts 2024-04-14T15:13:16.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:16 smithi090 ceph-mon[30399]: 4.1b scrub ok 2024-04-14T15:13:16.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:16 smithi090 ceph-mon[30399]: pgmap v137: 648 pgs: 64 creating+peering, 11 creating+activating, 2 active+clean+snaptrim, 32 unknown, 539 active+clean; 145 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 3.7 KiB/s rd, 5.0 KiB/s wr, 7 op/s; 27 B/s, 1 objects/s recovering 2024-04-14T15:13:16.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:16 smithi090 ceph-mon[30399]: 2.1a scrub starts 2024-04-14T15:13:16.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:16 smithi090 ceph-mon[30399]: 2.1a scrub ok 2024-04-14T15:13:16.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:16 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:16.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:16 smithi090 ceph-mon[30399]: Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:13:16.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:16 smithi090 ceph-mon[30399]: from='client.31396 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:16.695 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:16 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-7"}]': finished 2024-04-14T15:13:16.695 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:16 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2461174822' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi090-94662-3"}]': finished 2024-04-14T15:13:16.695 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:16 smithi090 ceph-mon[30399]: osdmap e128: 8 total, 8 up, 8 in 2024-04-14T15:13:16.695 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:16 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2461174822' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi090-94662-3"}]: dispatch 2024-04-14T15:13:16.695 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:16 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3264268010' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi090-93342-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:16.695 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:16 smithi090 ceph-mon[30399]: from='client.32359 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi090-93342-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:16.695 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:16 smithi090 ceph-mon[41670]: 4.1b scrub starts 2024-04-14T15:13:16.695 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:16 smithi090 ceph-mon[41670]: 4.1b scrub ok 2024-04-14T15:13:16.695 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:16 smithi090 ceph-mon[41670]: pgmap v137: 648 pgs: 64 creating+peering, 11 creating+activating, 2 active+clean+snaptrim, 32 unknown, 539 active+clean; 145 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 3.7 KiB/s rd, 5.0 KiB/s wr, 7 op/s; 27 B/s, 1 objects/s recovering 2024-04-14T15:13:16.695 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:16 smithi090 ceph-mon[41670]: 2.1a scrub starts 2024-04-14T15:13:16.695 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:16 smithi090 ceph-mon[41670]: 2.1a scrub ok 2024-04-14T15:13:16.696 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:16 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:16.696 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:16 smithi090 ceph-mon[41670]: Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:13:16.696 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:16 smithi090 ceph-mon[41670]: from='client.31396 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:16.696 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:16 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-7"}]': finished 2024-04-14T15:13:16.696 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:16 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2461174822' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi090-94662-3"}]': finished 2024-04-14T15:13:16.696 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:16 smithi090 ceph-mon[41670]: osdmap e128: 8 total, 8 up, 8 in 2024-04-14T15:13:16.696 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:16 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2461174822' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi090-94662-3"}]: dispatch 2024-04-14T15:13:16.696 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:16 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3264268010' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi090-93342-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:16.696 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:16 smithi090 ceph-mon[41670]: from='client.32359 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi090-93342-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:16.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:16.794 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:16.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:13:16.830 INFO:tasks.workunit.client.0.smithi090.stdout: ec_list: Running main() from gmock_main.cc 2024-04-14T15:13:16.830 INFO:tasks.workunit.client.0.smithi090.stdout: ec_list: [==========] Running 3 tests from 1 test suite. 2024-04-14T15:13:16.830 INFO:tasks.workunit.client.0.smithi090.stdout: ec_list: [----------] Global test environment set-up. 2024-04-14T15:13:16.830 INFO:tasks.workunit.client.0.smithi090.stdout: ec_list: [----------] 3 tests from NeoradosECList 2024-04-14T15:13:16.830 INFO:tasks.workunit.client.0.smithi090.stdout: ec_list: [ RUN ] NeoradosECList.ListObjects 2024-04-14T15:13:16.830 INFO:tasks.workunit.client.0.smithi090.stdout: ec_list: [ OK ] NeoradosECList.ListObjects (6228 ms) 2024-04-14T15:13:16.830 INFO:tasks.workunit.client.0.smithi090.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsNS 2024-04-14T15:13:16.830 INFO:tasks.workunit.client.0.smithi090.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsNS (7006 ms) 2024-04-14T15:13:16.831 INFO:tasks.workunit.client.0.smithi090.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsMany 2024-04-14T15:13:16.831 INFO:tasks.workunit.client.0.smithi090.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsMany (9011 ms) 2024-04-14T15:13:16.831 INFO:tasks.workunit.client.0.smithi090.stdout: ec_list: [----------] 3 tests from NeoradosECList (22245 ms total) 2024-04-14T15:13:16.831 INFO:tasks.workunit.client.0.smithi090.stdout: ec_list: 2024-04-14T15:13:16.831 INFO:tasks.workunit.client.0.smithi090.stdout: ec_list: [----------] Global test environment tear-down 2024-04-14T15:13:16.831 INFO:tasks.workunit.client.0.smithi090.stdout: ec_list: [==========] 3 tests from 1 test suite ran. (22245 ms total) 2024-04-14T15:13:16.831 INFO:tasks.workunit.client.0.smithi090.stdout: ec_list: [ PASSED ] 3 tests. 2024-04-14T15:13:16.895 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:13:16.925 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:16.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:16.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:13:17.027 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:13:17.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:17.141 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:17.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:13:17.242 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:13:17.357 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:17.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:17.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:13:17.458 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:13:17.490 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:17 smithi090 ceph-mon[30399]: 2.5 scrub starts 2024-04-14T15:13:17.490 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:17 smithi090 ceph-mon[30399]: 2.5 scrub ok 2024-04-14T15:13:17.490 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:17 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:17.490 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:17 smithi090 ceph-mon[30399]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-04-14T15:13:17.491 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:17 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2461174822' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi090-94662-3"}]': finished 2024-04-14T15:13:17.491 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:17 smithi090 ceph-mon[30399]: from='client.32359 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi090-93342-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:17.491 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:17 smithi090 ceph-mon[30399]: osdmap e129: 8 total, 8 up, 8 in 2024-04-14T15:13:17.500 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:17 smithi090 ceph-mon[41670]: 2.5 scrub starts 2024-04-14T15:13:17.501 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:17 smithi090 ceph-mon[41670]: 2.5 scrub ok 2024-04-14T15:13:17.501 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:17 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:17.501 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:17 smithi090 ceph-mon[41670]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-04-14T15:13:17.501 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:17 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2461174822' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi090-94662-3"}]': finished 2024-04-14T15:13:17.501 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:17 smithi090 ceph-mon[41670]: from='client.32359 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi090-93342-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:17.501 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:17 smithi090 ceph-mon[41670]: osdmap e129: 8 total, 8 up, 8 in 2024-04-14T15:13:17.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:17.607 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:17.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:13:17.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:17 smithi092 ceph-mon[33966]: 2.5 scrub starts 2024-04-14T15:13:17.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:17 smithi092 ceph-mon[33966]: 2.5 scrub ok 2024-04-14T15:13:17.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:17 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:17.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:17 smithi092 ceph-mon[33966]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-04-14T15:13:17.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:17 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2461174822' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi090-94662-3"}]': finished 2024-04-14T15:13:17.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:17 smithi092 ceph-mon[33966]: from='client.32359 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi090-93342-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:17.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:17 smithi092 ceph-mon[33966]: osdmap e129: 8 total, 8 up, 8 in 2024-04-14T15:13:17.714 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:13:17.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:17.741 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:17.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:13:17.842 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:13:17.885 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:17.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:17.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:13:17.987 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:13:18.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:18.066 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:18.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:13:18.079 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-14T15:13:18.079 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-14T15:13:18.079 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-04-14T15:13:18.079 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-14T15:13:18.079 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-14T15:13:18.080 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.0 config set filestore_inject_stall 3 2024-04-14T15:13:18.167 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:13:18.246 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:18.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:18.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:13:18.347 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:13:18.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:18 smithi090 ceph-mon[30399]: 5.b scrub starts 2024-04-14T15:13:18.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:18 smithi090 ceph-mon[30399]: 5.b scrub ok 2024-04-14T15:13:18.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:18 smithi090 ceph-mon[30399]: pgmap v140: 712 pgs: 32 creating+peering, 11 creating+activating, 2 active+clean+snaptrim, 192 unknown, 475 active+clean; 145 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 0 B/s wr, 4 op/s; 27 B/s, 1 objects/s recovering 2024-04-14T15:13:18.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:18 smithi090 ceph-mon[30399]: 6.5 scrub starts 2024-04-14T15:13:18.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:18 smithi090 ceph-mon[30399]: 6.5 scrub ok 2024-04-14T15:13:18.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:18 smithi090 ceph-mon[30399]: 4.1e scrub starts 2024-04-14T15:13:18.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:18 smithi090 ceph-mon[30399]: 4.1e scrub ok 2024-04-14T15:13:18.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:18 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:18.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:18 smithi090 ceph-mon[30399]: osdmap e130: 8 total, 8 up, 8 in 2024-04-14T15:13:18.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:18 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:18.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:18 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:18.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:18 smithi090 ceph-mon[30399]: 6.1b scrub starts 2024-04-14T15:13:18.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:18 smithi090 ceph-mon[30399]: 6.1b scrub ok 2024-04-14T15:13:18.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:18 smithi090 ceph-mon[41670]: 5.b scrub starts 2024-04-14T15:13:18.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:18 smithi090 ceph-mon[41670]: 5.b scrub ok 2024-04-14T15:13:18.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:18 smithi090 ceph-mon[41670]: pgmap v140: 712 pgs: 32 creating+peering, 11 creating+activating, 2 active+clean+snaptrim, 192 unknown, 475 active+clean; 145 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 0 B/s wr, 4 op/s; 27 B/s, 1 objects/s recovering 2024-04-14T15:13:18.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:18 smithi090 ceph-mon[41670]: 6.5 scrub starts 2024-04-14T15:13:18.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:18 smithi090 ceph-mon[41670]: 6.5 scrub ok 2024-04-14T15:13:18.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:18 smithi090 ceph-mon[41670]: 4.1e scrub starts 2024-04-14T15:13:18.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:18 smithi090 ceph-mon[41670]: 4.1e scrub ok 2024-04-14T15:13:18.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:18 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:18.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:18 smithi090 ceph-mon[41670]: osdmap e130: 8 total, 8 up, 8 in 2024-04-14T15:13:18.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:18 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:18.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:18 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:18.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:18 smithi090 ceph-mon[41670]: 6.1b scrub starts 2024-04-14T15:13:18.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:18 smithi090 ceph-mon[41670]: 6.1b scrub ok 2024-04-14T15:13:18.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:18.497 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:18.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:13:18.598 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:13:18.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:18 smithi092 ceph-mon[33966]: 5.b scrub starts 2024-04-14T15:13:18.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:18 smithi092 ceph-mon[33966]: 5.b scrub ok 2024-04-14T15:13:18.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:18 smithi092 ceph-mon[33966]: pgmap v140: 712 pgs: 32 creating+peering, 11 creating+activating, 2 active+clean+snaptrim, 192 unknown, 475 active+clean; 145 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 0 B/s wr, 4 op/s; 27 B/s, 1 objects/s recovering 2024-04-14T15:13:18.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:18 smithi092 ceph-mon[33966]: 6.5 scrub starts 2024-04-14T15:13:18.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:18 smithi092 ceph-mon[33966]: 6.5 scrub ok 2024-04-14T15:13:18.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:18 smithi092 ceph-mon[33966]: 4.1e scrub starts 2024-04-14T15:13:18.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:18 smithi092 ceph-mon[33966]: 4.1e scrub ok 2024-04-14T15:13:18.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:18 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:18.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:18 smithi092 ceph-mon[33966]: osdmap e130: 8 total, 8 up, 8 in 2024-04-14T15:13:18.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:18 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:18.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:18 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:18.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:18 smithi092 ceph-mon[33966]: 6.1b scrub starts 2024-04-14T15:13:18.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:18 smithi092 ceph-mon[33966]: 6.1b scrub ok 2024-04-14T15:13:18.744 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:18.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:18.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:13:18.846 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:13:18.894 INFO:tasks.workunit.client.0.smithi090.stdout: write_operations: Running main() from gmock_main.cc 2024-04-14T15:13:18.894 INFO:tasks.workunit.client.0.smithi090.stdout: write_operations: [==========] Running 7 tests from 1 test suite. 2024-04-14T15:13:18.894 INFO:tasks.workunit.client.0.smithi090.stdout: write_operations: [----------] Global test environment set-up. 2024-04-14T15:13:18.894 INFO:tasks.workunit.client.0.smithi090.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps 2024-04-14T15:13:18.894 INFO:tasks.workunit.client.0.smithi090.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertExists 2024-04-14T15:13:18.894 INFO:tasks.workunit.client.0.smithi090.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertExists (2487 ms) 2024-04-14T15:13:18.894 INFO:tasks.workunit.client.0.smithi090.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertVersion 2024-04-14T15:13:18.894 INFO:tasks.workunit.client.0.smithi090.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertVersion (3017 ms) 2024-04-14T15:13:18.894 INFO:tasks.workunit.client.0.smithi090.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Xattrs 2024-04-14T15:13:18.894 INFO:tasks.workunit.client.0.smithi090.stdout: write_operations: [ OK ] NeoRadosWriteOps.Xattrs (2692 ms) 2024-04-14T15:13:18.895 INFO:tasks.workunit.client.0.smithi090.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Write 2024-04-14T15:13:18.895 INFO:tasks.workunit.client.0.smithi090.stdout: write_operations: [ OK ] NeoRadosWriteOps.Write (2993 ms) 2024-04-14T15:13:18.895 INFO:tasks.workunit.client.0.smithi090.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Exec 2024-04-14T15:13:18.895 INFO:tasks.workunit.client.0.smithi090.stdout: write_operations: [ OK ] NeoRadosWriteOps.Exec (3015 ms) 2024-04-14T15:13:18.895 INFO:tasks.workunit.client.0.smithi090.stdout: write_operations: [ RUN ] NeoRadosWriteOps.WriteSame 2024-04-14T15:13:18.895 INFO:tasks.workunit.client.0.smithi090.stdout: write_operations: [ OK ] NeoRadosWriteOps.WriteSame (3024 ms) 2024-04-14T15:13:18.895 INFO:tasks.workunit.client.0.smithi090.stdout: write_operations: [ RUN ] NeoRadosWriteOps.CmpExt 2024-04-14T15:13:18.895 INFO:tasks.workunit.client.0.smithi090.stdout: write_operations: [ OK ] NeoRadosWriteOps.CmpExt (7039 ms) 2024-04-14T15:13:18.895 INFO:tasks.workunit.client.0.smithi090.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps (24267 ms total) 2024-04-14T15:13:18.895 INFO:tasks.workunit.client.0.smithi090.stdout: write_operations: 2024-04-14T15:13:18.895 INFO:tasks.workunit.client.0.smithi090.stdout: write_operations: [----------] Global test environment tear-down 2024-04-14T15:13:18.895 INFO:tasks.workunit.client.0.smithi090.stdout: write_operations: [==========] 7 tests from 1 test suite ran. (24267 ms total) 2024-04-14T15:13:18.895 INFO:tasks.workunit.client.0.smithi090.stdout: write_operations: [ PASSED ] 7 tests. 2024-04-14T15:13:18.913 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_write_operations: Running main() from gmock_main.cc 2024-04-14T15:13:18.913 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2024-04-14T15:13:18.913 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_write_operations: [----------] Global test environment set-up. 2024-04-14T15:13:18.913 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2024-04-14T15:13:18.914 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2024-04-14T15:13:18.914 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2024-04-14T15:13:18.914 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2024-04-14T15:13:18.914 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_write_operations: 2024-04-14T15:13:18.914 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2024-04-14T15:13:18.914 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2024-04-14T15:13:18.914 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (2730 ms) 2024-04-14T15:13:18.915 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2024-04-14T15:13:18.915 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (3030 ms) 2024-04-14T15:13:18.915 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2024-04-14T15:13:18.915 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (2687 ms) 2024-04-14T15:13:18.915 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2024-04-14T15:13:18.916 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (2994 ms) 2024-04-14T15:13:18.916 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2024-04-14T15:13:18.916 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (3017 ms) 2024-04-14T15:13:18.916 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2024-04-14T15:13:18.916 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (3030 ms) 2024-04-14T15:13:18.916 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2024-04-14T15:13:18.916 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (7053 ms) 2024-04-14T15:13:18.916 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (24541 ms total) 2024-04-14T15:13:18.916 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_write_operations: 2024-04-14T15:13:18.917 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_write_operations: [----------] Global test environment tear-down 2024-04-14T15:13:18.917 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (24544 ms total) 2024-04-14T15:13:18.917 INFO:tasks.workunit.client.0.smithi090.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2024-04-14T15:13:19.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:19.022 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:19.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:13:19.123 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:13:19.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:19.147 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:19.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:13:19.248 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:13:19.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:19.273 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:19.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:13:19.374 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:13:19.467 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:19.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:19.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:13:19.569 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:13:19.584 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:19 smithi090 ceph-mon[41670]: 2.e scrub starts 2024-04-14T15:13:19.584 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:19 smithi090 ceph-mon[41670]: 2.e scrub ok 2024-04-14T15:13:19.584 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:19 smithi090 ceph-mon[41670]: 6.17 scrub starts 2024-04-14T15:13:19.585 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:19 smithi090 ceph-mon[41670]: 6.17 scrub ok 2024-04-14T15:13:19.585 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:19 smithi090 ceph-mon[41670]: 2.7 scrub starts 2024-04-14T15:13:19.585 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:19 smithi090 ceph-mon[41670]: 2.7 scrub ok 2024-04-14T15:13:19.585 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:19 smithi090 ceph-mon[41670]: 4.0 scrub starts 2024-04-14T15:13:19.585 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:19 smithi090 ceph-mon[41670]: 4.0 scrub ok 2024-04-14T15:13:19.585 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:19 smithi090 ceph-mon[41670]: 4.1d scrub starts 2024-04-14T15:13:19.585 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:19 smithi090 ceph-mon[41670]: 4.1d scrub ok 2024-04-14T15:13:19.585 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:19 smithi090 ceph-mon[41670]: 2.1f scrub starts 2024-04-14T15:13:19.585 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:19 smithi090 ceph-mon[41670]: 2.1f scrub ok 2024-04-14T15:13:19.585 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:19 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:19.585 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:19 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:19.585 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:19 smithi090 ceph-mon[41670]: osdmap e131: 8 total, 8 up, 8 in 2024-04-14T15:13:19.585 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:19 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/305000435' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi090-93342-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:19.585 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:19 smithi090 ceph-mon[41670]: from='client.35263 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi090-93342-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:19.586 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:19 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2615399438' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi090-93363-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:19.586 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:19 smithi090 ceph-mon[41670]: from='client.35236 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi090-93363-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:19.586 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:19 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:13:19.586 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:19 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:13:19.586 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:19 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2105945719' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi090-93998-4"}]: dispatch 2024-04-14T15:13:19.586 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:19 smithi090 ceph-mon[41670]: from='client.25376 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi090-93998-4"}]: dispatch 2024-04-14T15:13:19.586 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:19 smithi090 ceph-mon[30399]: 2.e scrub starts 2024-04-14T15:13:19.586 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:19 smithi090 ceph-mon[30399]: 2.e scrub ok 2024-04-14T15:13:19.586 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:19 smithi090 ceph-mon[30399]: 6.17 scrub starts 2024-04-14T15:13:19.587 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:19 smithi090 ceph-mon[30399]: 6.17 scrub ok 2024-04-14T15:13:19.587 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:19 smithi090 ceph-mon[30399]: 2.7 scrub starts 2024-04-14T15:13:19.587 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:19 smithi090 ceph-mon[30399]: 2.7 scrub ok 2024-04-14T15:13:19.587 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:19 smithi090 ceph-mon[30399]: 4.0 scrub starts 2024-04-14T15:13:19.587 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:19 smithi090 ceph-mon[30399]: 4.0 scrub ok 2024-04-14T15:13:19.587 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:19 smithi090 ceph-mon[30399]: 4.1d scrub starts 2024-04-14T15:13:19.587 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:19 smithi090 ceph-mon[30399]: 4.1d scrub ok 2024-04-14T15:13:19.587 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:19 smithi090 ceph-mon[30399]: 2.1f scrub starts 2024-04-14T15:13:19.587 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:19 smithi090 ceph-mon[30399]: 2.1f scrub ok 2024-04-14T15:13:19.587 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:19 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:19.587 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:19 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:19.587 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:19 smithi090 ceph-mon[30399]: osdmap e131: 8 total, 8 up, 8 in 2024-04-14T15:13:19.587 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:19 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/305000435' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi090-93342-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:19.587 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:19 smithi090 ceph-mon[30399]: from='client.35263 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi090-93342-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:19.587 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:19 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2615399438' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi090-93363-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:19.588 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:19 smithi090 ceph-mon[30399]: from='client.35236 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi090-93363-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:19.588 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:19 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:13:19.588 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:19 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:13:19.588 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:19 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2105945719' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi090-93998-4"}]: dispatch 2024-04-14T15:13:19.588 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:19 smithi090 ceph-mon[30399]: from='client.25376 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi090-93998-4"}]: dispatch 2024-04-14T15:13:19.592 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:19.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:19.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:13:19.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:19 smithi092 ceph-mon[33966]: 2.e scrub starts 2024-04-14T15:13:19.626 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:19 smithi092 ceph-mon[33966]: 2.e scrub ok 2024-04-14T15:13:19.626 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:19 smithi092 ceph-mon[33966]: 6.17 scrub starts 2024-04-14T15:13:19.626 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:19 smithi092 ceph-mon[33966]: 6.17 scrub ok 2024-04-14T15:13:19.626 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:19 smithi092 ceph-mon[33966]: 2.7 scrub starts 2024-04-14T15:13:19.626 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:19 smithi092 ceph-mon[33966]: 2.7 scrub ok 2024-04-14T15:13:19.626 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:19 smithi092 ceph-mon[33966]: 4.0 scrub starts 2024-04-14T15:13:19.627 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:19 smithi092 ceph-mon[33966]: 4.0 scrub ok 2024-04-14T15:13:19.627 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:19 smithi092 ceph-mon[33966]: 4.1d scrub starts 2024-04-14T15:13:19.627 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:19 smithi092 ceph-mon[33966]: 4.1d scrub ok 2024-04-14T15:13:19.627 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:19 smithi092 ceph-mon[33966]: 2.1f scrub starts 2024-04-14T15:13:19.627 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:19 smithi092 ceph-mon[33966]: 2.1f scrub ok 2024-04-14T15:13:19.627 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:19 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:19.627 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:19 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:19.627 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:19 smithi092 ceph-mon[33966]: osdmap e131: 8 total, 8 up, 8 in 2024-04-14T15:13:19.627 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:19 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/305000435' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi090-93342-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:19.627 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:19 smithi092 ceph-mon[33966]: from='client.35263 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi090-93342-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:19.627 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:19 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2615399438' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi090-93363-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:19.627 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:19 smithi092 ceph-mon[33966]: from='client.35236 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi090-93363-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:19.627 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:19 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:13:19.627 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:19 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:13:19.627 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:19 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2105945719' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi090-93998-4"}]: dispatch 2024-04-14T15:13:19.628 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:19 smithi092 ceph-mon[33966]: from='client.25376 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi090-93998-4"}]: dispatch 2024-04-14T15:13:19.696 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:13:19.980 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-14T15:13:19.980 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:13:20.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:20.026 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:20.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:13:20.030 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:13:20.128 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:13:20.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:20.153 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:20.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:13:20.254 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:13:20.316 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:20.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:20.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:13:20.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.142+0000 7f804d72a700 1 -- 172.21.15.90:0/1580400723 >> v1:172.21.15.90:6789/0 conn(0x7f804810e310 legacy=0x7f80481107d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:20.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.144+0000 7f804d72a700 1 -- 172.21.15.90:0/1580400723 shutdown_connections 2024-04-14T15:13:20.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.144+0000 7f804d72a700 1 -- 172.21.15.90:0/1580400723 >> 172.21.15.90:0/1580400723 conn(0x7f80480fd290 msgr2=0x7f80480ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:13:20.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.146+0000 7f804d72a700 1 -- 172.21.15.90:0/1580400723 shutdown_connections 2024-04-14T15:13:20.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.147+0000 7f804d72a700 1 -- 172.21.15.90:0/1580400723 wait complete. 2024-04-14T15:13:20.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.148+0000 7f804d72a700 1 librados: starting msgr at 2024-04-14T15:13:20.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.148+0000 7f804d72a700 1 librados: starting objecter 2024-04-14T15:13:20.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.148+0000 7f804d72a700 1 Processor -- start 2024-04-14T15:13:20.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.149+0000 7f804d72a700 1 -- start start 2024-04-14T15:13:20.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.149+0000 7f804d72a700 1 librados: setting wanted keys 2024-04-14T15:13:20.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.149+0000 7f804d72a700 1 librados: calling monclient init 2024-04-14T15:13:20.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.149+0000 7f804d72a700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f804810dfc0 con 0x7f804810a740 2024-04-14T15:13:20.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.149+0000 7f804d72a700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f80481a6f10 con 0x7f8048107890 2024-04-14T15:13:20.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.149+0000 7f804d72a700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f80481a80f0 con 0x7f804810e310 2024-04-14T15:13:20.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.149+0000 7f80477fe700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f804810e310 0x7f80481a47e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:33202/0 (socket says 172.21.15.90:33202) 2024-04-14T15:13:20.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.150+0000 7f80477fe700 1 -- 172.21.15.90:0/2429599462 learned_addr learned my addr 172.21.15.90:0/2429599462 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:13:20.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.150+0000 7f802ffff700 10 client.?.objecter ms_handle_connect 0x7f804810e310 2024-04-14T15:13:20.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.150+0000 7f802ffff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:13:20.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.150+0000 7f802ffff700 1 -- 172.21.15.90:0/2429599462 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 956700980 0 0) 0x7f80481a80f0 con 0x7f804810e310 2024-04-14T15:13:20.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.150+0000 7f802ffff700 1 -- 172.21.15.90:0/2429599462 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f80300035b0 con 0x7f804810e310 2024-04-14T15:13:20.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.151+0000 7f802ffff700 1 -- 172.21.15.90:0/2429599462 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2718886741 0 0) 0x7f80300035b0 con 0x7f804810e310 2024-04-14T15:13:20.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.151+0000 7f802ffff700 1 -- 172.21.15.90:0/2429599462 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f80481a80f0 con 0x7f804810e310 2024-04-14T15:13:20.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.151+0000 7f802ffff700 1 -- 172.21.15.90:0/2429599462 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f803c001aa0 con 0x7f804810e310 2024-04-14T15:13:20.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.151+0000 7f802ffff700 1 -- 172.21.15.90:0/2429599462 <== mon.2 v1:172.21.15.90:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2855759834 0 0) 0x7f80481a80f0 con 0x7f804810e310 2024-04-14T15:13:20.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.151+0000 7f802ffff700 1 -- 172.21.15.90:0/2429599462 >> v1:172.21.15.92:6789/0 conn(0x7f8048107890 legacy=0x7f804810ab20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-14T15:13:20.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.151+0000 7f802ffff700 1 -- 172.21.15.90:0/2429599462 >> v1:172.21.15.90:6789/0 conn(0x7f804810a740 legacy=0x7f804810b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:20.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.151+0000 7f802ffff700 1 -- 172.21.15.90:0/2429599462 --> v1:172.21.15.90:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80481a92d0 con 0x7f804810e310 2024-04-14T15:13:20.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.151+0000 7f804d72a700 1 -- 172.21.15.90:0/2429599462 --> v1:172.21.15.90:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f80481a70e0 con 0x7f804810e310 2024-04-14T15:13:20.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.151+0000 7f804d72a700 10 client.36011.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:13:20.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.151+0000 7f804d72a700 1 -- 172.21.15.90:0/2429599462 --> v1:172.21.15.90:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f80481a75a0 con 0x7f804810e310 2024-04-14T15:13:20.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.151+0000 7f804d72a700 1 librados: init done 2024-04-14T15:13:20.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.153+0000 7f804d72a700 1 -- 172.21.15.90:0/2429599462 --> v1:172.21.15.90:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f80480693d0 con 0x7f804810e310 2024-04-14T15:13:20.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.159+0000 7f802ffff700 1 -- 172.21.15.90:0/2429599462 <== mon.2 v1:172.21.15.90:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f803c0036a0 con 0x7f804810e310 2024-04-14T15:13:20.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.159+0000 7f802ffff700 1 -- 172.21.15.90:0/2429599462 <== mon.2 v1:172.21.15.90:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f803c004f00 con 0x7f804810e310 2024-04-14T15:13:20.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.159+0000 7f802ffff700 1 -- 172.21.15.90:0/2429599462 <== mon.2 v1:172.21.15.90:6790/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f803c005140 con 0x7f804810e310 2024-04-14T15:13:20.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.159+0000 7f802ffff700 1 -- 172.21.15.90:0/2429599462 <== mon.2 v1:172.21.15.90:6790/0 8 ==== osd_map(132..132 src has 51..132) ==== 12764+0+0 (unknown 722716976 0 0) 0x7f803c08cf10 con 0x7f804810e310 2024-04-14T15:13:20.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.159+0000 7f802ffff700 10 client.36011.objecter ms_dispatch 0x7f8048001550 osd_map(132..132 src has 51..132) 2024-04-14T15:13:20.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.159+0000 7f802ffff700 3 client.36011.objecter handle_osd_map got epochs [132,132] > 0 2024-04-14T15:13:20.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.159+0000 7f802ffff700 3 client.36011.objecter handle_osd_map decoding full epoch 132 2024-04-14T15:13:20.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.160+0000 7f802ffff700 20 client.36011.objecter dump_active .. 0 homeless 2024-04-14T15:13:20.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.161+0000 7f802ffff700 10 client.36011.objecter ms_handle_connect 0x7f8030071310 2024-04-14T15:13:20.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.161+0000 7f802ffff700 1 -- 172.21.15.90:0/2429599462 <== mon.2 v1:172.21.15.90:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v132) ==== 54+0+16 (unknown 919436239 0 2420000192) 0x7f803c0587a0 con 0x7f804810e310 2024-04-14T15:13:20.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.167+0000 7f804d72a700 10 client.36011.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:13:20.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.167+0000 7f804d72a700 15 client.36011.objecter _session_command_op_assign -1 1 2024-04-14T15:13:20.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.167+0000 7f804d72a700 20 client.36011.objecter _get_session s=0x7f8048114760 osd=0 3 2024-04-14T15:13:20.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.167+0000 7f804d72a700 20 client.36011.objecter put_session s=0x7f8048114760 osd=0 3 2024-04-14T15:13:20.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.167+0000 7f804d72a700 20 client.36011.objecter _get_session s=0x7f8048114760 osd=0 3 2024-04-14T15:13:20.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.167+0000 7f804d72a700 15 client.36011.objecter _session_command_op_remove -1 1 2024-04-14T15:13:20.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.167+0000 7f804d72a700 20 client.36011.objecter get_session s=0x7f8048114760 osd=0 3 2024-04-14T15:13:20.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.167+0000 7f804d72a700 15 client.36011.objecter _session_command_op_assign 0 1 2024-04-14T15:13:20.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.167+0000 7f804d72a700 20 client.36011.objecter put_session s=0x7f8048114760 osd=0 4 2024-04-14T15:13:20.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.167+0000 7f804d72a700 10 client.36011.objecter _send_command 1 2024-04-14T15:13:20.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.167+0000 7f804d72a700 1 -- 172.21.15.90:0/2429599462 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8048102ec0 con 0x7f80481a5f00 2024-04-14T15:13:20.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.169+0000 7f802ffff700 10 client.36011.objecter ms_handle_connect 0x7f80481a5f00 2024-04-14T15:13:20.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.173+0000 7f802ffff700 1 -- 172.21.15.90:0/2429599462 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f8048102ec0 con 0x7f80481a5f00 2024-04-14T15:13:20.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.173+0000 7f802ffff700 10 client.36011.objecter ms_dispatch 0x7f8048001550 command_reply(tid 1: 0 ) 2024-04-14T15:13:20.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.173+0000 7f802ffff700 10 client.36011.objecter _finish_command 1 = system:0 2024-04-14T15:13:20.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.173+0000 7f802ffff700 20 client.36011.objecter put_session s=0x7f8048114760 osd=0 4 2024-04-14T15:13:20.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.173+0000 7f802ffff700 15 client.36011.objecter _session_command_op_remove 0 1 2024-04-14T15:13:20.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.205+0000 7f804d72a700 10 client.36011.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:13:20.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.205+0000 7f804d72a700 15 client.36011.objecter _session_command_op_assign -1 2 2024-04-14T15:13:20.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.205+0000 7f804d72a700 20 client.36011.objecter _get_session s=0x7f8048114760 osd=0 3 2024-04-14T15:13:20.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.205+0000 7f804d72a700 20 client.36011.objecter put_session s=0x7f8048114760 osd=0 3 2024-04-14T15:13:20.377 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:13:20.378 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:13:20.378 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:13:20.378 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:13:20.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.205+0000 7f804d72a700 20 client.36011.objecter _get_session s=0x7f8048114760 osd=0 3 2024-04-14T15:13:20.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.205+0000 7f804d72a700 15 client.36011.objecter _session_command_op_remove -1 2 2024-04-14T15:13:20.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.205+0000 7f804d72a700 20 client.36011.objecter get_session s=0x7f8048114760 osd=0 3 2024-04-14T15:13:20.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.205+0000 7f804d72a700 15 client.36011.objecter _session_command_op_assign 0 2 2024-04-14T15:13:20.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.205+0000 7f804d72a700 20 client.36011.objecter put_session s=0x7f8048114760 osd=0 4 2024-04-14T15:13:20.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.205+0000 7f804d72a700 10 client.36011.objecter _send_command 2 2024-04-14T15:13:20.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.205+0000 7f804d72a700 1 -- 172.21.15.90:0/2429599462 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f80481a69f0 con 0x7f80481a5f00 2024-04-14T15:13:20.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.209+0000 7f802ffff700 1 -- 172.21.15.90:0/2429599462 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f80481a69f0 con 0x7f80481a5f00 2024-04-14T15:13:20.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.209+0000 7f802ffff700 10 client.36011.objecter ms_dispatch 0x7f8048001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:13:20.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.209+0000 7f802ffff700 10 client.36011.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:13:20.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.209+0000 7f802ffff700 20 client.36011.objecter put_session s=0x7f8048114760 osd=0 4 2024-04-14T15:13:20.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.209+0000 7f802ffff700 15 client.36011.objecter _session_command_op_remove 0 2 2024-04-14T15:13:20.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.212+0000 7f802dffb700 10 client.36011.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:13:20.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.212+0000 7f802dffb700 15 client.36011.objecter _session_command_op_assign -1 3 2024-04-14T15:13:20.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.212+0000 7f802dffb700 20 client.36011.objecter _get_session s=0x7f8048105bd0 osd=1 3 2024-04-14T15:13:20.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.212+0000 7f802dffb700 20 client.36011.objecter put_session s=0x7f8048105bd0 osd=1 3 2024-04-14T15:13:20.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.212+0000 7f802dffb700 20 client.36011.objecter _get_session s=0x7f8048105bd0 osd=1 3 2024-04-14T15:13:20.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.212+0000 7f802dffb700 15 client.36011.objecter _session_command_op_remove -1 3 2024-04-14T15:13:20.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.212+0000 7f802dffb700 20 client.36011.objecter get_session s=0x7f8048105bd0 osd=1 3 2024-04-14T15:13:20.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.212+0000 7f802dffb700 15 client.36011.objecter _session_command_op_assign 1 3 2024-04-14T15:13:20.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.212+0000 7f802dffb700 20 client.36011.objecter put_session s=0x7f8048105bd0 osd=1 4 2024-04-14T15:13:20.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.212+0000 7f802dffb700 10 client.36011.objecter _send_command 3 2024-04-14T15:13:20.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.212+0000 7f802dffb700 1 -- 172.21.15.90:0/2429599462 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f80481aeca0 con 0x7f8048063bd0 2024-04-14T15:13:20.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.212+0000 7f802ffff700 10 client.36011.objecter ms_handle_connect 0x7f8048063bd0 2024-04-14T15:13:20.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.214+0000 7f802ffff700 1 -- 172.21.15.90:0/2429599462 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f80481aeca0 con 0x7f8048063bd0 2024-04-14T15:13:20.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.214+0000 7f802ffff700 10 client.36011.objecter ms_dispatch 0x7f8048001550 command_reply(tid 3: 0 ) 2024-04-14T15:13:20.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.214+0000 7f802ffff700 10 client.36011.objecter _finish_command 3 = system:0 2024-04-14T15:13:20.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.214+0000 7f802ffff700 20 client.36011.objecter put_session s=0x7f8048105bd0 osd=1 4 2024-04-14T15:13:20.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.214+0000 7f802ffff700 15 client.36011.objecter _session_command_op_remove 1 3 2024-04-14T15:13:20.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.249+0000 7f802dffb700 10 client.36011.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:13:20.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.249+0000 7f802dffb700 15 client.36011.objecter _session_command_op_assign -1 4 2024-04-14T15:13:20.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.249+0000 7f802dffb700 20 client.36011.objecter _get_session s=0x7f8048105bd0 osd=1 3 2024-04-14T15:13:20.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.249+0000 7f802dffb700 20 client.36011.objecter put_session s=0x7f8048105bd0 osd=1 3 2024-04-14T15:13:20.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.250+0000 7f802dffb700 20 client.36011.objecter _get_session s=0x7f8048105bd0 osd=1 3 2024-04-14T15:13:20.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.250+0000 7f802dffb700 15 client.36011.objecter _session_command_op_remove -1 4 2024-04-14T15:13:20.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.250+0000 7f802dffb700 20 client.36011.objecter get_session s=0x7f8048105bd0 osd=1 3 2024-04-14T15:13:20.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.250+0000 7f802dffb700 15 client.36011.objecter _session_command_op_assign 1 4 2024-04-14T15:13:20.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.250+0000 7f802dffb700 20 client.36011.objecter put_session s=0x7f8048105bd0 osd=1 4 2024-04-14T15:13:20.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.250+0000 7f802dffb700 10 client.36011.objecter _send_command 4 2024-04-14T15:13:20.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.250+0000 7f802dffb700 1 -- 172.21.15.90:0/2429599462 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f8048063140 con 0x7f8048063bd0 2024-04-14T15:13:20.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.255+0000 7f802ffff700 1 -- 172.21.15.90:0/2429599462 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f8048063140 con 0x7f8048063bd0 2024-04-14T15:13:20.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.256+0000 7f802ffff700 10 client.36011.objecter ms_dispatch 0x7f8048001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:13:20.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.256+0000 7f802ffff700 10 client.36011.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:13:20.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.256+0000 7f802ffff700 20 client.36011.objecter put_session s=0x7f8048105bd0 osd=1 4 2024-04-14T15:13:20.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.256+0000 7f802ffff700 15 client.36011.objecter _session_command_op_remove 1 4 2024-04-14T15:13:20.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.257+0000 7f802dffb700 10 client.36011.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:13:20.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.257+0000 7f802dffb700 15 client.36011.objecter _session_command_op_assign -1 5 2024-04-14T15:13:20.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.257+0000 7f802dffb700 20 client.36011.objecter _get_session s=0x7f804810ef70 osd=2 3 2024-04-14T15:13:20.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.257+0000 7f802dffb700 20 client.36011.objecter put_session s=0x7f804810ef70 osd=2 3 2024-04-14T15:13:20.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.257+0000 7f802dffb700 20 client.36011.objecter _get_session s=0x7f804810ef70 osd=2 3 2024-04-14T15:13:20.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.257+0000 7f802dffb700 15 client.36011.objecter _session_command_op_remove -1 5 2024-04-14T15:13:20.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.257+0000 7f802dffb700 20 client.36011.objecter get_session s=0x7f804810ef70 osd=2 3 2024-04-14T15:13:20.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.257+0000 7f802dffb700 15 client.36011.objecter _session_command_op_assign 2 5 2024-04-14T15:13:20.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.257+0000 7f802dffb700 20 client.36011.objecter put_session s=0x7f804810ef70 osd=2 4 2024-04-14T15:13:20.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.257+0000 7f802dffb700 10 client.36011.objecter _send_command 5 2024-04-14T15:13:20.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.257+0000 7f802dffb700 1 -- 172.21.15.90:0/2429599462 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f804810f5f0 con 0x7f80481b0000 2024-04-14T15:13:20.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.257+0000 7f802ffff700 10 client.36011.objecter ms_handle_connect 0x7f80481b0000 2024-04-14T15:13:20.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.259+0000 7f802ffff700 1 -- 172.21.15.90:0/2429599462 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f804810f5f0 con 0x7f80481b0000 2024-04-14T15:13:20.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.259+0000 7f802ffff700 10 client.36011.objecter ms_dispatch 0x7f8048001550 command_reply(tid 5: 0 ) 2024-04-14T15:13:20.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.259+0000 7f802ffff700 10 client.36011.objecter _finish_command 5 = system:0 2024-04-14T15:13:20.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.259+0000 7f802ffff700 20 client.36011.objecter put_session s=0x7f804810ef70 osd=2 4 2024-04-14T15:13:20.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.259+0000 7f802ffff700 15 client.36011.objecter _session_command_op_remove 2 5 2024-04-14T15:13:20.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.297+0000 7f802dffb700 10 client.36011.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:13:20.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.297+0000 7f802dffb700 15 client.36011.objecter _session_command_op_assign -1 6 2024-04-14T15:13:20.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.297+0000 7f802dffb700 20 client.36011.objecter _get_session s=0x7f804810ef70 osd=2 3 2024-04-14T15:13:20.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.297+0000 7f802dffb700 20 client.36011.objecter put_session s=0x7f804810ef70 osd=2 3 2024-04-14T15:13:20.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.297+0000 7f802dffb700 20 client.36011.objecter _get_session s=0x7f804810ef70 osd=2 3 2024-04-14T15:13:20.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.297+0000 7f802dffb700 15 client.36011.objecter _session_command_op_remove -1 6 2024-04-14T15:13:20.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.297+0000 7f802dffb700 20 client.36011.objecter get_session s=0x7f804810ef70 osd=2 3 2024-04-14T15:13:20.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.297+0000 7f802dffb700 15 client.36011.objecter _session_command_op_assign 2 6 2024-04-14T15:13:20.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.297+0000 7f802dffb700 20 client.36011.objecter put_session s=0x7f804810ef70 osd=2 4 2024-04-14T15:13:20.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.297+0000 7f802dffb700 10 client.36011.objecter _send_command 6 2024-04-14T15:13:20.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.297+0000 7f802dffb700 1 -- 172.21.15.90:0/2429599462 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f804810e9f0 con 0x7f80481b0000 2024-04-14T15:13:20.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.300+0000 7f802ffff700 1 -- 172.21.15.90:0/2429599462 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f804810e9f0 con 0x7f80481b0000 2024-04-14T15:13:20.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.300+0000 7f802ffff700 10 client.36011.objecter ms_dispatch 0x7f8048001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:13:20.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.300+0000 7f802ffff700 10 client.36011.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:13:20.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.300+0000 7f802ffff700 20 client.36011.objecter put_session s=0x7f804810ef70 osd=2 4 2024-04-14T15:13:20.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.300+0000 7f802ffff700 15 client.36011.objecter _session_command_op_remove 2 6 2024-04-14T15:13:20.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.301+0000 7f802dffb700 10 client.36011.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:13:20.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.301+0000 7f802dffb700 15 client.36011.objecter _session_command_op_assign -1 7 2024-04-14T15:13:20.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.301+0000 7f802dffb700 20 client.36011.objecter _get_session s=0x7f80481af650 osd=3 3 2024-04-14T15:13:20.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.301+0000 7f802dffb700 20 client.36011.objecter put_session s=0x7f80481af650 osd=3 3 2024-04-14T15:13:20.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.301+0000 7f802dffb700 20 client.36011.objecter _get_session s=0x7f80481af650 osd=3 3 2024-04-14T15:13:20.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.301+0000 7f802dffb700 15 client.36011.objecter _session_command_op_remove -1 7 2024-04-14T15:13:20.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.301+0000 7f802dffb700 20 client.36011.objecter get_session s=0x7f80481af650 osd=3 3 2024-04-14T15:13:20.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.301+0000 7f802dffb700 15 client.36011.objecter _session_command_op_assign 3 7 2024-04-14T15:13:20.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.301+0000 7f802dffb700 20 client.36011.objecter put_session s=0x7f80481af650 osd=3 4 2024-04-14T15:13:20.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.301+0000 7f802dffb700 10 client.36011.objecter _send_command 7 2024-04-14T15:13:20.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.301+0000 7f802dffb700 1 -- 172.21.15.90:0/2429599462 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f80481afcd0 con 0x7f80481b4c90 2024-04-14T15:13:20.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.301+0000 7f802ffff700 10 client.36011.objecter ms_handle_connect 0x7f80481b4c90 2024-04-14T15:13:20.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.302+0000 7f802ffff700 1 -- 172.21.15.90:0/2429599462 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f80481afcd0 con 0x7f80481b4c90 2024-04-14T15:13:20.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.302+0000 7f802ffff700 10 client.36011.objecter ms_dispatch 0x7f8048001550 command_reply(tid 7: 0 ) 2024-04-14T15:13:20.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.302+0000 7f802ffff700 10 client.36011.objecter _finish_command 7 = system:0 2024-04-14T15:13:20.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.302+0000 7f802ffff700 20 client.36011.objecter put_session s=0x7f80481af650 osd=3 4 2024-04-14T15:13:20.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.302+0000 7f802ffff700 15 client.36011.objecter _session_command_op_remove 3 7 2024-04-14T15:13:20.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.338+0000 7f802dffb700 10 client.36011.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:13:20.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.338+0000 7f802dffb700 15 client.36011.objecter _session_command_op_assign -1 8 2024-04-14T15:13:20.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.338+0000 7f802dffb700 20 client.36011.objecter _get_session s=0x7f80481af650 osd=3 3 2024-04-14T15:13:20.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.338+0000 7f802dffb700 20 client.36011.objecter put_session s=0x7f80481af650 osd=3 3 2024-04-14T15:13:20.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.338+0000 7f802dffb700 20 client.36011.objecter _get_session s=0x7f80481af650 osd=3 3 2024-04-14T15:13:20.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.338+0000 7f802dffb700 15 client.36011.objecter _session_command_op_remove -1 8 2024-04-14T15:13:20.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.338+0000 7f802dffb700 20 client.36011.objecter get_session s=0x7f80481af650 osd=3 3 2024-04-14T15:13:20.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.338+0000 7f802dffb700 15 client.36011.objecter _session_command_op_assign 3 8 2024-04-14T15:13:20.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.338+0000 7f802dffb700 20 client.36011.objecter put_session s=0x7f80481af650 osd=3 4 2024-04-14T15:13:20.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.338+0000 7f802dffb700 10 client.36011.objecter _send_command 8 2024-04-14T15:13:20.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.338+0000 7f802dffb700 1 -- 172.21.15.90:0/2429599462 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f80481b7fb0 con 0x7f80481b4c90 2024-04-14T15:13:20.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.340+0000 7f802ffff700 1 -- 172.21.15.90:0/2429599462 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f80481b7fb0 con 0x7f80481b4c90 2024-04-14T15:13:20.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.340+0000 7f802ffff700 10 client.36011.objecter ms_dispatch 0x7f8048001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:13:20.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.340+0000 7f802ffff700 10 client.36011.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:13:20.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.341+0000 7f802ffff700 20 client.36011.objecter put_session s=0x7f80481af650 osd=3 4 2024-04-14T15:13:20.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.341+0000 7f802ffff700 15 client.36011.objecter _session_command_op_remove 3 8 2024-04-14T15:13:20.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.341+0000 7f802dffb700 10 client.36011.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:13:20.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.341+0000 7f802dffb700 15 client.36011.objecter _session_command_op_assign -1 9 2024-04-14T15:13:20.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.341+0000 7f802dffb700 20 client.36011.objecter _get_session s=0x7f80481afe80 osd=4 3 2024-04-14T15:13:20.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.341+0000 7f802dffb700 20 client.36011.objecter put_session s=0x7f80481afe80 osd=4 3 2024-04-14T15:13:20.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.341+0000 7f802dffb700 20 client.36011.objecter _get_session s=0x7f80481afe80 osd=4 3 2024-04-14T15:13:20.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.341+0000 7f802dffb700 15 client.36011.objecter _session_command_op_remove -1 9 2024-04-14T15:13:20.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.341+0000 7f802dffb700 20 client.36011.objecter get_session s=0x7f80481afe80 osd=4 3 2024-04-14T15:13:20.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.341+0000 7f802dffb700 15 client.36011.objecter _session_command_op_assign 4 9 2024-04-14T15:13:20.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.341+0000 7f802dffb700 20 client.36011.objecter put_session s=0x7f80481afe80 osd=4 4 2024-04-14T15:13:20.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.341+0000 7f802dffb700 10 client.36011.objecter _send_command 9 2024-04-14T15:13:20.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.341+0000 7f802dffb700 1 -- 172.21.15.90:0/2429599462 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f80481b81c0 con 0x7f80481db3c0 2024-04-14T15:13:20.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.342+0000 7f802ffff700 10 client.36011.objecter ms_handle_connect 0x7f80481db3c0 2024-04-14T15:13:20.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.343+0000 7f802ffff700 1 -- 172.21.15.90:0/2429599462 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f80481b81c0 con 0x7f80481db3c0 2024-04-14T15:13:20.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.343+0000 7f802ffff700 10 client.36011.objecter ms_dispatch 0x7f8048001550 command_reply(tid 9: 0 ) 2024-04-14T15:13:20.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.343+0000 7f802ffff700 10 client.36011.objecter _finish_command 9 = system:0 2024-04-14T15:13:20.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.343+0000 7f802ffff700 20 client.36011.objecter put_session s=0x7f80481afe80 osd=4 4 2024-04-14T15:13:20.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.343+0000 7f802ffff700 15 client.36011.objecter _session_command_op_remove 4 9 2024-04-14T15:13:20.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.376+0000 7f802dffb700 10 client.36011.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:13:20.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.376+0000 7f802dffb700 15 client.36011.objecter _session_command_op_assign -1 10 2024-04-14T15:13:20.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.376+0000 7f802dffb700 20 client.36011.objecter _get_session s=0x7f80481afe80 osd=4 3 2024-04-14T15:13:20.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.376+0000 7f802dffb700 20 client.36011.objecter put_session s=0x7f80481afe80 osd=4 3 2024-04-14T15:13:20.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.376+0000 7f802dffb700 20 client.36011.objecter _get_session s=0x7f80481afe80 osd=4 3 2024-04-14T15:13:20.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.376+0000 7f802dffb700 15 client.36011.objecter _session_command_op_remove -1 10 2024-04-14T15:13:20.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.376+0000 7f802dffb700 20 client.36011.objecter get_session s=0x7f80481afe80 osd=4 3 2024-04-14T15:13:20.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.376+0000 7f802dffb700 15 client.36011.objecter _session_command_op_assign 4 10 2024-04-14T15:13:20.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.376+0000 7f802dffb700 20 client.36011.objecter put_session s=0x7f80481afe80 osd=4 4 2024-04-14T15:13:20.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.376+0000 7f802dffb700 10 client.36011.objecter _send_command 10 2024-04-14T15:13:20.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.376+0000 7f802dffb700 1 -- 172.21.15.90:0/2429599462 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f80481deed0 con 0x7f80481db3c0 2024-04-14T15:13:20.395 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:13:20.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.379+0000 7f802ffff700 1 -- 172.21.15.90:0/2429599462 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f80481deed0 con 0x7f80481db3c0 2024-04-14T15:13:20.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.379+0000 7f802ffff700 10 client.36011.objecter ms_dispatch 0x7f8048001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:13:20.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.379+0000 7f802ffff700 10 client.36011.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:13:20.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.379+0000 7f802ffff700 20 client.36011.objecter put_session s=0x7f80481afe80 osd=4 4 2024-04-14T15:13:20.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.379+0000 7f802ffff700 15 client.36011.objecter _session_command_op_remove 4 10 2024-04-14T15:13:20.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.380+0000 7f802dffb700 10 client.36011.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:13:20.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.380+0000 7f802dffb700 15 client.36011.objecter _session_command_op_assign -1 11 2024-04-14T15:13:20.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.380+0000 7f802dffb700 20 client.36011.objecter _get_session s=0x7f80481deed0 osd=5 3 2024-04-14T15:13:20.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.380+0000 7f802dffb700 20 client.36011.objecter put_session s=0x7f80481deed0 osd=5 3 2024-04-14T15:13:20.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.380+0000 7f802dffb700 20 client.36011.objecter _get_session s=0x7f80481deed0 osd=5 3 2024-04-14T15:13:20.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.380+0000 7f802dffb700 15 client.36011.objecter _session_command_op_remove -1 11 2024-04-14T15:13:20.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.380+0000 7f802dffb700 20 client.36011.objecter get_session s=0x7f80481deed0 osd=5 3 2024-04-14T15:13:20.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.380+0000 7f802dffb700 15 client.36011.objecter _session_command_op_assign 5 11 2024-04-14T15:13:20.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.380+0000 7f802dffb700 20 client.36011.objecter put_session s=0x7f80481deed0 osd=5 4 2024-04-14T15:13:20.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.380+0000 7f802dffb700 10 client.36011.objecter _send_command 11 2024-04-14T15:13:20.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.380+0000 7f802dffb700 1 -- 172.21.15.90:0/2429599462 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f80481e3210 con 0x7f80481df5f0 2024-04-14T15:13:20.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.382+0000 7f802ffff700 10 client.36011.objecter ms_handle_connect 0x7f80481df5f0 2024-04-14T15:13:20.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.383+0000 7f802ffff700 1 -- 172.21.15.90:0/2429599462 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f80481e3210 con 0x7f80481df5f0 2024-04-14T15:13:20.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.383+0000 7f802ffff700 10 client.36011.objecter ms_dispatch 0x7f8048001550 command_reply(tid 11: 0 ) 2024-04-14T15:13:20.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.383+0000 7f802ffff700 10 client.36011.objecter _finish_command 11 = system:0 2024-04-14T15:13:20.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.383+0000 7f802ffff700 20 client.36011.objecter put_session s=0x7f80481deed0 osd=5 4 2024-04-14T15:13:20.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.383+0000 7f802ffff700 15 client.36011.objecter _session_command_op_remove 5 11 2024-04-14T15:13:20.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.416+0000 7f802dffb700 10 client.36011.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:13:20.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.416+0000 7f802dffb700 15 client.36011.objecter _session_command_op_assign -1 12 2024-04-14T15:13:20.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.416+0000 7f802dffb700 20 client.36011.objecter _get_session s=0x7f80481deed0 osd=5 3 2024-04-14T15:13:20.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.416+0000 7f802dffb700 20 client.36011.objecter put_session s=0x7f80481deed0 osd=5 3 2024-04-14T15:13:20.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.416+0000 7f802dffb700 20 client.36011.objecter _get_session s=0x7f80481deed0 osd=5 3 2024-04-14T15:13:20.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.416+0000 7f802dffb700 15 client.36011.objecter _session_command_op_remove -1 12 2024-04-14T15:13:20.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.416+0000 7f802dffb700 20 client.36011.objecter get_session s=0x7f80481deed0 osd=5 3 2024-04-14T15:13:20.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.416+0000 7f802dffb700 15 client.36011.objecter _session_command_op_assign 5 12 2024-04-14T15:13:20.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.416+0000 7f802dffb700 20 client.36011.objecter put_session s=0x7f80481deed0 osd=5 4 2024-04-14T15:13:20.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.416+0000 7f802dffb700 10 client.36011.objecter _send_command 12 2024-04-14T15:13:20.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.416+0000 7f802dffb700 1 -- 172.21.15.90:0/2429599462 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f80481e3210 con 0x7f80481df5f0 2024-04-14T15:13:20.419 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:13:20.420 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:13:20.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.419+0000 7f802ffff700 1 -- 172.21.15.90:0/2429599462 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f80481e3210 con 0x7f80481df5f0 2024-04-14T15:13:20.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.419+0000 7f802ffff700 10 client.36011.objecter ms_dispatch 0x7f8048001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:13:20.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.419+0000 7f802ffff700 10 client.36011.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:13:20.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.419+0000 7f802ffff700 20 client.36011.objecter put_session s=0x7f80481deed0 osd=5 4 2024-04-14T15:13:20.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.419+0000 7f802ffff700 15 client.36011.objecter _session_command_op_remove 5 12 2024-04-14T15:13:20.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.419+0000 7f802dffb700 10 client.36011.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:13:20.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.419+0000 7f802dffb700 15 client.36011.objecter _session_command_op_assign -1 13 2024-04-14T15:13:20.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.419+0000 7f802dffb700 20 client.36011.objecter _get_session s=0x7f80481e3210 osd=6 3 2024-04-14T15:13:20.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.419+0000 7f802dffb700 20 client.36011.objecter put_session s=0x7f80481e3210 osd=6 3 2024-04-14T15:13:20.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.419+0000 7f802dffb700 20 client.36011.objecter _get_session s=0x7f80481e3210 osd=6 3 2024-04-14T15:13:20.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.419+0000 7f802dffb700 15 client.36011.objecter _session_command_op_remove -1 13 2024-04-14T15:13:20.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.419+0000 7f802dffb700 20 client.36011.objecter get_session s=0x7f80481e3210 osd=6 3 2024-04-14T15:13:20.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.419+0000 7f802dffb700 15 client.36011.objecter _session_command_op_assign 6 13 2024-04-14T15:13:20.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.419+0000 7f802dffb700 20 client.36011.objecter put_session s=0x7f80481e3210 osd=6 4 2024-04-14T15:13:20.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.419+0000 7f802dffb700 10 client.36011.objecter _send_command 13 2024-04-14T15:13:20.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.419+0000 7f802dffb700 1 -- 172.21.15.90:0/2429599462 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f80481e4520 con 0x7f80481e51a0 2024-04-14T15:13:20.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.421+0000 7f802ffff700 10 client.36011.objecter ms_handle_connect 0x7f80481e51a0 2024-04-14T15:13:20.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.423+0000 7f802ffff700 1 -- 172.21.15.90:0/2429599462 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f80481e4520 con 0x7f80481e51a0 2024-04-14T15:13:20.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.423+0000 7f802ffff700 10 client.36011.objecter ms_dispatch 0x7f8048001550 command_reply(tid 13: 0 ) 2024-04-14T15:13:20.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.423+0000 7f802ffff700 10 client.36011.objecter _finish_command 13 = system:0 2024-04-14T15:13:20.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.423+0000 7f802ffff700 20 client.36011.objecter put_session s=0x7f80481e3210 osd=6 4 2024-04-14T15:13:20.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.423+0000 7f802ffff700 15 client.36011.objecter _session_command_op_remove 6 13 2024-04-14T15:13:20.487 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:20 smithi090 ceph-mon[30399]: 4.d scrub starts 2024-04-14T15:13:20.487 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:20 smithi090 ceph-mon[30399]: 4.d scrub ok 2024-04-14T15:13:20.487 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:20 smithi090 ceph-mon[30399]: pgmap v143: 584 pgs: 4 creating+activating, 84 creating+peering, 2 active+clean+snaptrim, 62 unknown, 432 active+clean; 145 MiB data, 786 MiB used, 714 GiB / 715 GiB avail 2024-04-14T15:13:20.487 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:20 smithi090 ceph-mon[30399]: 4.1f scrub starts 2024-04-14T15:13:20.487 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:20 smithi090 ceph-mon[30399]: 4.1f scrub ok 2024-04-14T15:13:20.487 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:20 smithi090 ceph-mon[30399]: 5.1a scrub starts 2024-04-14T15:13:20.487 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:20 smithi090 ceph-mon[30399]: 5.1a scrub ok 2024-04-14T15:13:20.488 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:20 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:20.488 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:20 smithi090 ceph-mon[30399]: from='client.35263 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi090-93342-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:20.488 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:20 smithi090 ceph-mon[30399]: from='client.35236 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi090-93363-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:20.488 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:20 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-9", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:13:20.488 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:20 smithi090 ceph-mon[30399]: from='client.25376 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi090-93998-4"}]': finished 2024-04-14T15:13:20.488 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:20 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-9"}]: dispatch 2024-04-14T15:13:20.488 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:20 smithi090 ceph-mon[30399]: osdmap e132: 8 total, 8 up, 8 in 2024-04-14T15:13:20.488 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:20 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2595677146' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:20.488 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:20 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-9"}]: dispatch 2024-04-14T15:13:20.488 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:20 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2105945719' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi090-93998-4", "tierpool": "test-rados-api-smithi090-93998-6"}]: dispatch 2024-04-14T15:13:20.488 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:20 smithi090 ceph-mon[30399]: from='client.25376 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi090-93998-4", "tierpool": "test-rados-api-smithi090-93998-6"}]: dispatch 2024-04-14T15:13:20.488 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:20 smithi090 ceph-mon[30399]: from='client.36175 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:20.488 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:20 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:13:20.488 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:20 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:13:20.489 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:20 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2429599462' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:13:20.489 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:20 smithi090 ceph-mon[41670]: 4.d scrub starts 2024-04-14T15:13:20.489 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:20 smithi090 ceph-mon[41670]: 4.d scrub ok 2024-04-14T15:13:20.489 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:20 smithi090 ceph-mon[41670]: pgmap v143: 584 pgs: 4 creating+activating, 84 creating+peering, 2 active+clean+snaptrim, 62 unknown, 432 active+clean; 145 MiB data, 786 MiB used, 714 GiB / 715 GiB avail 2024-04-14T15:13:20.489 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:20 smithi090 ceph-mon[41670]: 4.1f scrub starts 2024-04-14T15:13:20.489 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:20 smithi090 ceph-mon[41670]: 4.1f scrub ok 2024-04-14T15:13:20.489 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:20 smithi090 ceph-mon[41670]: 5.1a scrub starts 2024-04-14T15:13:20.489 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:20 smithi090 ceph-mon[41670]: 5.1a scrub ok 2024-04-14T15:13:20.489 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:20 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:20.489 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:20 smithi090 ceph-mon[41670]: from='client.35263 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi090-93342-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:20.490 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:20 smithi090 ceph-mon[41670]: from='client.35236 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi090-93363-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:20.490 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:20 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-9", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:13:20.490 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:20 smithi090 ceph-mon[41670]: from='client.25376 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi090-93998-4"}]': finished 2024-04-14T15:13:20.490 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:20 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-9"}]: dispatch 2024-04-14T15:13:20.490 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:20 smithi090 ceph-mon[41670]: osdmap e132: 8 total, 8 up, 8 in 2024-04-14T15:13:20.490 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:20 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2595677146' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:20.490 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:20 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-9"}]: dispatch 2024-04-14T15:13:20.490 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:20 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2105945719' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi090-93998-4", "tierpool": "test-rados-api-smithi090-93998-6"}]: dispatch 2024-04-14T15:13:20.490 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:20 smithi090 ceph-mon[41670]: from='client.25376 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi090-93998-4", "tierpool": "test-rados-api-smithi090-93998-6"}]: dispatch 2024-04-14T15:13:20.490 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:20 smithi090 ceph-mon[41670]: from='client.36175 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:20.490 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:20 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:13:20.490 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:20 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:13:20.490 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:20 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2429599462' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:13:20.491 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:13:20.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.461+0000 7f802dffb700 10 client.36011.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:13:20.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.461+0000 7f802dffb700 15 client.36011.objecter _session_command_op_assign -1 14 2024-04-14T15:13:20.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.461+0000 7f802dffb700 20 client.36011.objecter _get_session s=0x7f80481e3210 osd=6 3 2024-04-14T15:13:20.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.461+0000 7f802dffb700 20 client.36011.objecter put_session s=0x7f80481e3210 osd=6 3 2024-04-14T15:13:20.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.461+0000 7f802dffb700 20 client.36011.objecter _get_session s=0x7f80481e3210 osd=6 3 2024-04-14T15:13:20.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.461+0000 7f802dffb700 15 client.36011.objecter _session_command_op_remove -1 14 2024-04-14T15:13:20.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.461+0000 7f802dffb700 20 client.36011.objecter get_session s=0x7f80481e3210 osd=6 3 2024-04-14T15:13:20.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.461+0000 7f802dffb700 15 client.36011.objecter _session_command_op_assign 6 14 2024-04-14T15:13:20.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.461+0000 7f802dffb700 20 client.36011.objecter put_session s=0x7f80481e3210 osd=6 4 2024-04-14T15:13:20.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.461+0000 7f802dffb700 10 client.36011.objecter _send_command 14 2024-04-14T15:13:20.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.461+0000 7f802dffb700 1 -- 172.21.15.90:0/2429599462 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f80481e4780 con 0x7f80481e51a0 2024-04-14T15:13:20.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.463+0000 7f802ffff700 1 -- 172.21.15.90:0/2429599462 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f80481e4780 con 0x7f80481e51a0 2024-04-14T15:13:20.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.463+0000 7f802ffff700 10 client.36011.objecter ms_dispatch 0x7f8048001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:13:20.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.463+0000 7f802ffff700 10 client.36011.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:13:20.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.463+0000 7f802ffff700 20 client.36011.objecter put_session s=0x7f80481e3210 osd=6 4 2024-04-14T15:13:20.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.463+0000 7f802ffff700 15 client.36011.objecter _session_command_op_remove 6 14 2024-04-14T15:13:20.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.466+0000 7f802dffb700 10 client.36011.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:13:20.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.466+0000 7f802dffb700 15 client.36011.objecter _session_command_op_assign -1 15 2024-04-14T15:13:20.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.466+0000 7f802dffb700 20 client.36011.objecter _get_session s=0x7f80481e4780 osd=7 3 2024-04-14T15:13:20.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.466+0000 7f802dffb700 20 client.36011.objecter put_session s=0x7f80481e4780 osd=7 3 2024-04-14T15:13:20.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.466+0000 7f802dffb700 20 client.36011.objecter _get_session s=0x7f80481e4780 osd=7 3 2024-04-14T15:13:20.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.466+0000 7f802dffb700 15 client.36011.objecter _session_command_op_remove -1 15 2024-04-14T15:13:20.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.466+0000 7f802dffb700 20 client.36011.objecter get_session s=0x7f80481e4780 osd=7 3 2024-04-14T15:13:20.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.466+0000 7f802dffb700 15 client.36011.objecter _session_command_op_assign 7 15 2024-04-14T15:13:20.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.466+0000 7f802dffb700 20 client.36011.objecter put_session s=0x7f80481e4780 osd=7 4 2024-04-14T15:13:20.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.466+0000 7f802dffb700 10 client.36011.objecter _send_command 15 2024-04-14T15:13:20.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.466+0000 7f802dffb700 1 -- 172.21.15.90:0/2429599462 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f80481ebe10 con 0x7f80481e8300 2024-04-14T15:13:20.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.468+0000 7f802ffff700 10 client.36011.objecter ms_handle_connect 0x7f80481e8300 2024-04-14T15:13:20.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.468+0000 7f802ffff700 1 -- 172.21.15.90:0/2429599462 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f80481ebe10 con 0x7f80481e8300 2024-04-14T15:13:20.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.468+0000 7f802ffff700 10 client.36011.objecter ms_dispatch 0x7f8048001550 command_reply(tid 15: 0 ) 2024-04-14T15:13:20.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.468+0000 7f802ffff700 10 client.36011.objecter _finish_command 15 = system:0 2024-04-14T15:13:20.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.468+0000 7f802ffff700 20 client.36011.objecter put_session s=0x7f80481e4780 osd=7 4 2024-04-14T15:13:20.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.468+0000 7f802ffff700 15 client.36011.objecter _session_command_op_remove 7 15 2024-04-14T15:13:20.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.509+0000 7f802dffb700 10 client.36011.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:13:20.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.509+0000 7f802dffb700 15 client.36011.objecter _session_command_op_assign -1 16 2024-04-14T15:13:20.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.509+0000 7f802dffb700 20 client.36011.objecter _get_session s=0x7f80481e4780 osd=7 3 2024-04-14T15:13:20.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.509+0000 7f802dffb700 20 client.36011.objecter put_session s=0x7f80481e4780 osd=7 3 2024-04-14T15:13:20.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.509+0000 7f802dffb700 20 client.36011.objecter _get_session s=0x7f80481e4780 osd=7 3 2024-04-14T15:13:20.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.509+0000 7f802dffb700 15 client.36011.objecter _session_command_op_remove -1 16 2024-04-14T15:13:20.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.509+0000 7f802dffb700 20 client.36011.objecter get_session s=0x7f80481e4780 osd=7 3 2024-04-14T15:13:20.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.509+0000 7f802dffb700 15 client.36011.objecter _session_command_op_assign 7 16 2024-04-14T15:13:20.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.509+0000 7f802dffb700 20 client.36011.objecter put_session s=0x7f80481e4780 osd=7 4 2024-04-14T15:13:20.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.509+0000 7f802dffb700 10 client.36011.objecter _send_command 16 2024-04-14T15:13:20.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.509+0000 7f802dffb700 1 -- 172.21.15.90:0/2429599462 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f80481ebe10 con 0x7f80481e8300 2024-04-14T15:13:20.516 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:13:20.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.515+0000 7f802ffff700 1 -- 172.21.15.90:0/2429599462 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f80481ebe10 con 0x7f80481e8300 2024-04-14T15:13:20.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.515+0000 7f802ffff700 10 client.36011.objecter ms_dispatch 0x7f8048001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:13:20.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.515+0000 7f802ffff700 10 client.36011.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:13:20.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.516+0000 7f802ffff700 20 client.36011.objecter put_session s=0x7f80481e4780 osd=7 4 2024-04-14T15:13:20.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.516+0000 7f802ffff700 15 client.36011.objecter _session_command_op_remove 7 16 2024-04-14T15:13:20.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.518+0000 7f804d72a700 10 librados: watch_flush enter 2024-04-14T15:13:20.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.518+0000 7f804d72a700 10 librados: watch_flush exit 2024-04-14T15:13:20.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.518+0000 7f804d72a700 10 client.36011.objecter close_session for osd.0 2024-04-14T15:13:20.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.518+0000 7f804d72a700 1 -- 172.21.15.90:0/2429599462 >> v1:172.21.15.90:6801/3761806885 conn(0x7f80481a5f00 legacy=0x7f80481a62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:20.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.519+0000 7f804d72a700 20 client.36011.objecter put_session s=0x7f8048114760 osd=0 1 2024-04-14T15:13:20.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.519+0000 7f804d72a700 10 client.36011.objecter close_session for osd.1 2024-04-14T15:13:20.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.519+0000 7f804d72a700 1 -- 172.21.15.90:0/2429599462 >> v1:172.21.15.90:6805/4170361699 conn(0x7f8048063bd0 legacy=0x7f80481ad580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:20.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.519+0000 7f804d72a700 20 client.36011.objecter put_session s=0x7f8048105bd0 osd=1 1 2024-04-14T15:13:20.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.519+0000 7f804d72a700 10 client.36011.objecter close_session for osd.2 2024-04-14T15:13:20.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.519+0000 7f804d72a700 1 -- 172.21.15.90:0/2429599462 >> v1:172.21.15.90:6809/1686184251 conn(0x7f80481b0000 legacy=0x7f80481b23f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:20.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.519+0000 7f80477fe700 1 -- 172.21.15.90:0/2429599462 reap_dead start 2024-04-14T15:13:20.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.519+0000 7f804d72a700 20 client.36011.objecter put_session s=0x7f804810ef70 osd=2 1 2024-04-14T15:13:20.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.519+0000 7f804d72a700 10 client.36011.objecter close_session for osd.3 2024-04-14T15:13:20.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.519+0000 7f804d72a700 1 -- 172.21.15.90:0/2429599462 >> v1:172.21.15.90:6813/2270166774 conn(0x7f80481b4c90 legacy=0x7f80481b70d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:20.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.519+0000 7f804d72a700 20 client.36011.objecter put_session s=0x7f80481af650 osd=3 1 2024-04-14T15:13:20.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.519+0000 7f804d72a700 10 client.36011.objecter close_session for osd.4 2024-04-14T15:13:20.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.519+0000 7f804d72a700 1 -- 172.21.15.90:0/2429599462 >> v1:172.21.15.92:6800/3887549733 conn(0x7f80481db3c0 legacy=0x7f80481dd7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:20.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.519+0000 7f804d72a700 20 client.36011.objecter put_session s=0x7f80481afe80 osd=4 1 2024-04-14T15:13:20.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.519+0000 7f804d72a700 10 client.36011.objecter close_session for osd.5 2024-04-14T15:13:20.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.519+0000 7f804d72a700 1 -- 172.21.15.90:0/2429599462 >> v1:172.21.15.92:6804/3244561440 conn(0x7f80481df5f0 legacy=0x7f80481e19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:20.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.519+0000 7f804d72a700 20 client.36011.objecter put_session s=0x7f80481deed0 osd=5 1 2024-04-14T15:13:20.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.519+0000 7f804d72a700 10 client.36011.objecter close_session for osd.6 2024-04-14T15:13:20.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.519+0000 7f804d72a700 1 -- 172.21.15.90:0/2429599462 >> v1:172.21.15.92:6808/636125963 conn(0x7f80481e51a0 legacy=0x7f80481e5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:20.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.519+0000 7f804d72a700 20 client.36011.objecter put_session s=0x7f80481e3210 osd=6 1 2024-04-14T15:13:20.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.519+0000 7f804d72a700 10 client.36011.objecter close_session for osd.7 2024-04-14T15:13:20.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.519+0000 7f804d72a700 1 -- 172.21.15.90:0/2429599462 >> v1:172.21.15.92:6812/1773706716 conn(0x7f80481e8300 legacy=0x7f80481ea6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:20.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.520+0000 7f80477fe700 1 -- 172.21.15.90:0/2429599462 reap_dead start 2024-04-14T15:13:20.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.520+0000 7f804d72a700 20 client.36011.objecter put_session s=0x7f80481e4780 osd=7 1 2024-04-14T15:13:20.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.520+0000 7f804d72a700 20 client.36011.objecter shutdown clearing up homeless session... 2024-04-14T15:13:20.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.520+0000 7f804d72a700 10 client.36011.objecter successfully canceled tick 2024-04-14T15:13:20.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.520+0000 7f804d72a700 1 -- 172.21.15.90:0/2429599462 >> v1:172.21.15.90:6800/420935427 conn(0x7f8030071310 legacy=0x7f80300737d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:20.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.520+0000 7f804d72a700 1 -- 172.21.15.90:0/2429599462 >> v1:172.21.15.90:6790/0 conn(0x7f804810e310 legacy=0x7f80481a47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:20.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.520+0000 7f804d72a700 1 -- 172.21.15.90:0/2429599462 shutdown_connections 2024-04-14T15:13:20.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.520+0000 7f804d72a700 1 -- 172.21.15.90:0/2429599462 >> 172.21.15.90:0/2429599462 conn(0x7f80480fd290 msgr2=0x7f804806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:13:20.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.522+0000 7f804d72a700 1 -- 172.21.15.90:0/2429599462 shutdown_connections 2024-04-14T15:13:20.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.522+0000 7f804d72a700 1 -- 172.21.15.90:0/2429599462 wait complete. 2024-04-14T15:13:20.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:20.523+0000 7f804d72a700 1 librados: shutdown 2024-04-14T15:13:20.528 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:13:20.528 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:13:20.528 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:13:20.528 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:13:20.528 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:13:20.528 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:13:20.529 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:13:20.529 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:13:20.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:20.586 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:20.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:13:20.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:20 smithi092 ceph-mon[33966]: 4.d scrub starts 2024-04-14T15:13:20.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:20 smithi092 ceph-mon[33966]: 4.d scrub ok 2024-04-14T15:13:20.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:20 smithi092 ceph-mon[33966]: pgmap v143: 584 pgs: 4 creating+activating, 84 creating+peering, 2 active+clean+snaptrim, 62 unknown, 432 active+clean; 145 MiB data, 786 MiB used, 714 GiB / 715 GiB avail 2024-04-14T15:13:20.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:20 smithi092 ceph-mon[33966]: 4.1f scrub starts 2024-04-14T15:13:20.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:20 smithi092 ceph-mon[33966]: 4.1f scrub ok 2024-04-14T15:13:20.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:20 smithi092 ceph-mon[33966]: 5.1a scrub starts 2024-04-14T15:13:20.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:20 smithi092 ceph-mon[33966]: 5.1a scrub ok 2024-04-14T15:13:20.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:20 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:20.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:20 smithi092 ceph-mon[33966]: from='client.35263 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi090-93342-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:20.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:20 smithi092 ceph-mon[33966]: from='client.35236 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi090-93363-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:20.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:20 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-9", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:13:20.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:20 smithi092 ceph-mon[33966]: from='client.25376 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi090-93998-4"}]': finished 2024-04-14T15:13:20.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:20 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-9"}]: dispatch 2024-04-14T15:13:20.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:20 smithi092 ceph-mon[33966]: osdmap e132: 8 total, 8 up, 8 in 2024-04-14T15:13:20.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:20 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2595677146' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:20.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:20 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-9"}]: dispatch 2024-04-14T15:13:20.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:20 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2105945719' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi090-93998-4", "tierpool": "test-rados-api-smithi090-93998-6"}]: dispatch 2024-04-14T15:13:20.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:20 smithi092 ceph-mon[33966]: from='client.25376 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi090-93998-4", "tierpool": "test-rados-api-smithi090-93998-6"}]: dispatch 2024-04-14T15:13:20.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:20 smithi092 ceph-mon[33966]: from='client.36175 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:20.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:20 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:13:20.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:20 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:13:20.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:20 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2429599462' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:13:20.687 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:13:20.711 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:20.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:20.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:13:20.813 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:13:20.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:20.957 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:20.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:13:21.036 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-14T15:13:21.058 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:13:21.167 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:21.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:21.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:13:21.269 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:13:21.304 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:21 smithi090 ceph-mon[30399]: 4.1a scrub starts 2024-04-14T15:13:21.304 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:21 smithi090 ceph-mon[30399]: 4.1a scrub ok 2024-04-14T15:13:21.304 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:21 smithi090 ceph-mon[30399]: 2.b scrub starts 2024-04-14T15:13:21.304 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:21 smithi090 ceph-mon[30399]: 2.b scrub ok 2024-04-14T15:13:21.304 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:21 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:21.304 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:21 smithi090 ceph-mon[30399]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:13:21.304 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:21 smithi090 ceph-mon[30399]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:13:21.304 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:21 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-9"}]': finished 2024-04-14T15:13:21.305 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:21 smithi090 ceph-mon[30399]: from='client.25376 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi090-93998-4", "tierpool": "test-rados-api-smithi090-93998-6"}]': finished 2024-04-14T15:13:21.305 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:21 smithi090 ceph-mon[30399]: from='client.36175 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:21.305 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:21 smithi090 ceph-mon[30399]: osdmap e133: 8 total, 8 up, 8 in 2024-04-14T15:13:21.305 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:21 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-9", "mode": "writeback"}]: dispatch 2024-04-14T15:13:21.305 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:21 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2105945719' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi090-93998-6", "pool2": "test-rados-api-smithi090-93998-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-14T15:13:21.305 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:21 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-9", "mode": "writeback"}]: dispatch 2024-04-14T15:13:21.305 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:21 smithi090 ceph-mon[30399]: from='client.25376 ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi090-93998-6", "pool2": "test-rados-api-smithi090-93998-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-14T15:13:21.305 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:21 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:13:21.305 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:21 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:13:21.305 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:21 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2598250370' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi090-93709-10"}]: dispatch 2024-04-14T15:13:21.305 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:21 smithi090 ceph-mon[30399]: from='client.38086 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi090-93709-10"}]: dispatch 2024-04-14T15:13:21.305 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:21 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2598250370' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi090-93709-10"}]: dispatch 2024-04-14T15:13:21.305 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:21 smithi090 ceph-mon[30399]: from='client.38086 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi090-93709-10"}]: dispatch 2024-04-14T15:13:21.306 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:21 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2598250370' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi090-93709-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:13:21.306 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:21 smithi090 ceph-mon[30399]: from='client.38086 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi090-93709-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:13:21.358 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:21.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:21.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:13:21.460 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:13:21.484 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:21.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:21.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:13:21.586 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:13:21.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:21 smithi092 ceph-mon[33966]: 4.1a scrub starts 2024-04-14T15:13:21.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:21 smithi092 ceph-mon[33966]: 4.1a scrub ok 2024-04-14T15:13:21.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:21 smithi092 ceph-mon[33966]: 2.b scrub starts 2024-04-14T15:13:21.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:21 smithi092 ceph-mon[33966]: 2.b scrub ok 2024-04-14T15:13:21.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:21 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:21.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:21 smithi092 ceph-mon[33966]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:13:21.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:21 smithi092 ceph-mon[33966]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:13:21.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:21 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-9"}]': finished 2024-04-14T15:13:21.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:21 smithi092 ceph-mon[33966]: from='client.25376 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi090-93998-4", "tierpool": "test-rados-api-smithi090-93998-6"}]': finished 2024-04-14T15:13:21.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:21 smithi092 ceph-mon[33966]: from='client.36175 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:21.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:21 smithi092 ceph-mon[33966]: osdmap e133: 8 total, 8 up, 8 in 2024-04-14T15:13:21.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:21 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-9", "mode": "writeback"}]: dispatch 2024-04-14T15:13:21.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:21 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2105945719' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi090-93998-6", "pool2": "test-rados-api-smithi090-93998-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-14T15:13:21.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:21 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-9", "mode": "writeback"}]: dispatch 2024-04-14T15:13:21.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:21 smithi092 ceph-mon[33966]: from='client.25376 ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi090-93998-6", "pool2": "test-rados-api-smithi090-93998-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-14T15:13:21.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:21 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:13:21.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:21 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:13:21.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:21 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2598250370' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi090-93709-10"}]: dispatch 2024-04-14T15:13:21.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:21 smithi092 ceph-mon[33966]: from='client.38086 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi090-93709-10"}]: dispatch 2024-04-14T15:13:21.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:21 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2598250370' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi090-93709-10"}]: dispatch 2024-04-14T15:13:21.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:21 smithi092 ceph-mon[33966]: from='client.38086 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi090-93709-10"}]: dispatch 2024-04-14T15:13:21.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:21 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2598250370' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi090-93709-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:13:21.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:21 smithi092 ceph-mon[33966]: from='client.38086 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi090-93709-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:13:21.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:21 smithi090 ceph-mon[41670]: 4.1a scrub starts 2024-04-14T15:13:21.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:21 smithi090 ceph-mon[41670]: 4.1a scrub ok 2024-04-14T15:13:21.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:21 smithi090 ceph-mon[41670]: 2.b scrub starts 2024-04-14T15:13:21.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:21 smithi090 ceph-mon[41670]: 2.b scrub ok 2024-04-14T15:13:21.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:21 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:21.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:21 smithi090 ceph-mon[41670]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:13:21.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:21 smithi090 ceph-mon[41670]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:13:21.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:21 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-9"}]': finished 2024-04-14T15:13:21.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:21 smithi090 ceph-mon[41670]: from='client.25376 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi090-93998-4", "tierpool": "test-rados-api-smithi090-93998-6"}]': finished 2024-04-14T15:13:21.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:21 smithi090 ceph-mon[41670]: from='client.36175 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:21.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:21 smithi090 ceph-mon[41670]: osdmap e133: 8 total, 8 up, 8 in 2024-04-14T15:13:21.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:21 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-9", "mode": "writeback"}]: dispatch 2024-04-14T15:13:21.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:21 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2105945719' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi090-93998-6", "pool2": "test-rados-api-smithi090-93998-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-14T15:13:21.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:21 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-9", "mode": "writeback"}]: dispatch 2024-04-14T15:13:21.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:21 smithi090 ceph-mon[41670]: from='client.25376 ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi090-93998-6", "pool2": "test-rados-api-smithi090-93998-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-14T15:13:21.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:21 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:13:21.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:21 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:13:21.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:21 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2598250370' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi090-93709-10"}]: dispatch 2024-04-14T15:13:21.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:21 smithi090 ceph-mon[41670]: from='client.38086 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi090-93709-10"}]: dispatch 2024-04-14T15:13:21.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:21 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2598250370' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi090-93709-10"}]: dispatch 2024-04-14T15:13:21.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:21 smithi090 ceph-mon[41670]: from='client.38086 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi090-93709-10"}]: dispatch 2024-04-14T15:13:21.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:21 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2598250370' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi090-93709-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:13:21.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:21 smithi090 ceph-mon[41670]: from='client.38086 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi090-93709-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:13:22.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:22.062 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:22.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:13:22.124 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:13:22.163 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:13:22.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:22.289 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:22.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:13:22.390 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:13:22.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:22.415 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:22.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:13:22.515 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:13:22.539 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:22.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:22.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:13:22.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:22 smithi092 ceph-mon[33966]: 4.1 scrub starts 2024-04-14T15:13:22.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:22 smithi092 ceph-mon[33966]: 4.1 scrub ok 2024-04-14T15:13:22.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:22 smithi092 ceph-mon[33966]: pgmap v146: 488 pgs: 96 unknown, 4 creating+activating, 388 active+clean; 145 MiB data, 766 MiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 1 op/s 2024-04-14T15:13:22.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:22 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:22.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:22 smithi092 ceph-mon[33966]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:13:22.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:22 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-9", "mode": "writeback"}]': finished 2024-04-14T15:13:22.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:22 smithi092 ceph-mon[33966]: from='client.25376 ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi090-93998-6", "pool2": "test-rados-api-smithi090-93998-6", "yes_i_really_really_mean_it": true}]': finished 2024-04-14T15:13:22.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:22 smithi092 ceph-mon[33966]: from='client.38086 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi090-93709-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:13:22.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:22 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2598250370' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi090-93709-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi090-93709-10"}]: dispatch 2024-04-14T15:13:22.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:22 smithi092 ceph-mon[33966]: osdmap e134: 8 total, 8 up, 8 in 2024-04-14T15:13:22.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:22 smithi092 ceph-mon[33966]: from='client.38086 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi090-93709-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi090-93709-10"}]: dispatch 2024-04-14T15:13:22.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:22 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/902879548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi090-93342-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:22.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:22 smithi092 ceph-mon[33966]: from='client.36767 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi090-93342-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:22.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:22 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2599630366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi090-93363-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:22.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:22 smithi092 ceph-mon[33966]: from='client.38089 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi090-93363-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:22.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:22 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:13:22.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:22 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:13:22.641 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:13:22.666 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:22.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:22.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:13:22.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:22 smithi090 ceph-mon[41670]: 4.1 scrub starts 2024-04-14T15:13:22.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:22 smithi090 ceph-mon[41670]: 4.1 scrub ok 2024-04-14T15:13:22.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:22 smithi090 ceph-mon[41670]: pgmap v146: 488 pgs: 96 unknown, 4 creating+activating, 388 active+clean; 145 MiB data, 766 MiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 1 op/s 2024-04-14T15:13:22.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:22 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:22.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:22 smithi090 ceph-mon[41670]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:13:22.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:22 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-9", "mode": "writeback"}]': finished 2024-04-14T15:13:22.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:22 smithi090 ceph-mon[41670]: from='client.25376 ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi090-93998-6", "pool2": "test-rados-api-smithi090-93998-6", "yes_i_really_really_mean_it": true}]': finished 2024-04-14T15:13:22.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:22 smithi090 ceph-mon[41670]: from='client.38086 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi090-93709-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:13:22.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:22 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2598250370' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi090-93709-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi090-93709-10"}]: dispatch 2024-04-14T15:13:22.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:22 smithi090 ceph-mon[41670]: osdmap e134: 8 total, 8 up, 8 in 2024-04-14T15:13:22.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:22 smithi090 ceph-mon[41670]: from='client.38086 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi090-93709-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi090-93709-10"}]: dispatch 2024-04-14T15:13:22.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:22 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/902879548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi090-93342-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:22.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:22 smithi090 ceph-mon[41670]: from='client.36767 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi090-93342-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:22.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:22 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2599630366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi090-93363-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:22.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:22 smithi090 ceph-mon[41670]: from='client.38089 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi090-93363-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:22.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:22 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:13:22.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:22 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:13:22.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:22 smithi090 ceph-mon[30399]: 4.1 scrub starts 2024-04-14T15:13:22.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:22 smithi090 ceph-mon[30399]: 4.1 scrub ok 2024-04-14T15:13:22.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:22 smithi090 ceph-mon[30399]: pgmap v146: 488 pgs: 96 unknown, 4 creating+activating, 388 active+clean; 145 MiB data, 766 MiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 1 op/s 2024-04-14T15:13:22.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:22 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:22.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:22 smithi090 ceph-mon[30399]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:13:22.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:22 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-9", "mode": "writeback"}]': finished 2024-04-14T15:13:22.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:22 smithi090 ceph-mon[30399]: from='client.25376 ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi090-93998-6", "pool2": "test-rados-api-smithi090-93998-6", "yes_i_really_really_mean_it": true}]': finished 2024-04-14T15:13:22.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:22 smithi090 ceph-mon[30399]: from='client.38086 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi090-93709-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:13:22.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:22 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2598250370' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi090-93709-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi090-93709-10"}]: dispatch 2024-04-14T15:13:22.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:22 smithi090 ceph-mon[30399]: osdmap e134: 8 total, 8 up, 8 in 2024-04-14T15:13:22.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:22 smithi090 ceph-mon[30399]: from='client.38086 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi090-93709-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi090-93709-10"}]: dispatch 2024-04-14T15:13:22.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:22 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/902879548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi090-93342-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:22.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:22 smithi090 ceph-mon[30399]: from='client.36767 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi090-93342-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:22.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:22 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2599630366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi090-93363-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:22.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:22 smithi090 ceph-mon[30399]: from='client.38089 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi090-93363-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:22.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:22 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:13:22.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:22 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:13:22.768 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:13:22.791 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:22.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:22.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:13:22.893 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:13:22.907 INFO:tasks.workunit.client.0.smithi090.stdout:h_notify_pp: flushed 2024-04-14T15:13:22.908 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (3004 ms) 2024-04-14T15:13:22.908 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2024-04-14T15:13:22.908 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: trying... 2024-04-14T15:13:22.908 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: handle_notify cookie 94679948733792 notify_id 523986010115 notifier_gid 25376 2024-04-14T15:13:22.908 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: timed out 2024-04-14T15:13:22.908 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: flushing 2024-04-14T15:13:22.908 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: flushed 2024-04-14T15:13:22.908 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (3086 ms) 2024-04-14T15:13:22.908 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2024-04-14T15:13:22.908 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: List watches 2024-04-14T15:13:22.909 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: notify2 2024-04-14T15:13:22.909 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: handle_notify cookie 94679948750496 notify_id 536870912002 notifier_gid 25376 2024-04-14T15:13:22.909 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: notify2 done 2024-04-14T15:13:22.909 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: watch_check 2024-04-14T15:13:22.909 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: unwatch2 2024-04-14T15:13:22.909 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: flushing 2024-04-14T15:13:22.909 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: done 2024-04-14T15:13:22.909 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (3011 ms) 2024-04-14T15:13:22.909 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2024-04-14T15:13:22.909 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: List watches 2024-04-14T15:13:22.909 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: notify2 2024-04-14T15:13:22.909 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: handle_notify cookie 94679948733792 notify_id 549755813891 notifier_gid 25376 2024-04-14T15:13:22.909 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: notify2 done 2024-04-14T15:13:22.910 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: watch_check 2024-04-14T15:13:22.910 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: unwatch2 2024-04-14T15:13:22.910 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: flushing 2024-04-14T15:13:22.910 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: done 2024-04-14T15:13:22.910 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (3013 ms) 2024-04-14T15:13:22.910 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (16228 ms total) 2024-04-14T15:13:22.910 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: 2024-04-14T15:13:22.910 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2024-04-14T15:13:22.910 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (28549 ms total) 2024-04-14T15:13:22.910 INFO:tasks.workunit.client.0.smithi090.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2024-04-14T15:13:22.921 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:22.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:22.922 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:13:23.023 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:13:23.050 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:23.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:23.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:13:23.152 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:13:23.244 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:23.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:23.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:13:23.316 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:23 smithi090 ceph-mon[30399]: 5.1 scrub starts 2024-04-14T15:13:23.316 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:23 smithi090 ceph-mon[30399]: 5.1 scrub ok 2024-04-14T15:13:23.316 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:23 smithi090 ceph-mon[30399]: 4.8 scrub starts 2024-04-14T15:13:23.316 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:23 smithi090 ceph-mon[30399]: 4.8 scrub ok 2024-04-14T15:13:23.316 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:23 smithi090 ceph-mon[30399]: 6.1f scrub starts 2024-04-14T15:13:23.317 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:23 smithi090 ceph-mon[30399]: 6.1f scrub ok 2024-04-14T15:13:23.317 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:23 smithi090 ceph-mon[30399]: 4.2 scrub starts 2024-04-14T15:13:23.317 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:23 smithi090 ceph-mon[30399]: 4.2 scrub ok 2024-04-14T15:13:23.317 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:23 smithi090 ceph-mon[30399]: 6.0 scrub starts 2024-04-14T15:13:23.317 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:23 smithi090 ceph-mon[30399]: 6.0 scrub ok 2024-04-14T15:13:23.317 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:23 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:23.317 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:23 smithi090 ceph-mon[30399]: from='client.36767 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi090-93342-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:23.317 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:23 smithi090 ceph-mon[30399]: from='client.38089 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi090-93363-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:23.317 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:23 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:13:23.317 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:23 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-9"}]: dispatch 2024-04-14T15:13:23.317 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:23 smithi090 ceph-mon[30399]: osdmap e135: 8 total, 8 up, 8 in 2024-04-14T15:13:23.317 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:23 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-9"}]: dispatch 2024-04-14T15:13:23.317 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:23 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:13:23.347 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:13:23.459 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:23.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:23.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:13:23.561 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:13:23.580 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:23 smithi090 ceph-mon[41670]: 5.1 scrub starts 2024-04-14T15:13:23.581 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:23 smithi090 ceph-mon[41670]: 5.1 scrub ok 2024-04-14T15:13:23.581 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:23 smithi090 ceph-mon[41670]: 4.8 scrub starts 2024-04-14T15:13:23.581 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:23 smithi090 ceph-mon[41670]: 4.8 scrub ok 2024-04-14T15:13:23.581 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:23 smithi090 ceph-mon[41670]: 6.1f scrub starts 2024-04-14T15:13:23.581 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:23 smithi090 ceph-mon[41670]: 6.1f scrub ok 2024-04-14T15:13:23.581 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:23 smithi090 ceph-mon[41670]: 4.2 scrub starts 2024-04-14T15:13:23.581 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:23 smithi090 ceph-mon[41670]: 4.2 scrub ok 2024-04-14T15:13:23.581 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:23 smithi090 ceph-mon[41670]: 6.0 scrub starts 2024-04-14T15:13:23.581 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:23 smithi090 ceph-mon[41670]: 6.0 scrub ok 2024-04-14T15:13:23.581 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:23 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:23.581 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:23 smithi090 ceph-mon[41670]: from='client.36767 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi090-93342-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:23.581 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:23 smithi090 ceph-mon[41670]: from='client.38089 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi090-93363-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:23.582 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:23 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:13:23.582 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:23 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-9"}]: dispatch 2024-04-14T15:13:23.582 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:23 smithi090 ceph-mon[41670]: osdmap e135: 8 total, 8 up, 8 in 2024-04-14T15:13:23.582 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:23 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-9"}]: dispatch 2024-04-14T15:13:23.582 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:23 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:13:23.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:23 smithi092 ceph-mon[33966]: 5.1 scrub starts 2024-04-14T15:13:23.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:23 smithi092 ceph-mon[33966]: 5.1 scrub ok 2024-04-14T15:13:23.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:23 smithi092 ceph-mon[33966]: 4.8 scrub starts 2024-04-14T15:13:23.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:23 smithi092 ceph-mon[33966]: 4.8 scrub ok 2024-04-14T15:13:23.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:23 smithi092 ceph-mon[33966]: 6.1f scrub starts 2024-04-14T15:13:23.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:23 smithi092 ceph-mon[33966]: 6.1f scrub ok 2024-04-14T15:13:23.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:23 smithi092 ceph-mon[33966]: 4.2 scrub starts 2024-04-14T15:13:23.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:23 smithi092 ceph-mon[33966]: 4.2 scrub ok 2024-04-14T15:13:23.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:23 smithi092 ceph-mon[33966]: 6.0 scrub starts 2024-04-14T15:13:23.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:23 smithi092 ceph-mon[33966]: 6.0 scrub ok 2024-04-14T15:13:23.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:23 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:23.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:23 smithi092 ceph-mon[33966]: from='client.36767 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi090-93342-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:23.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:23 smithi092 ceph-mon[33966]: from='client.38089 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi090-93363-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:23.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:23 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:13:23.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:23 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-9"}]: dispatch 2024-04-14T15:13:23.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:23 smithi092 ceph-mon[33966]: osdmap e135: 8 total, 8 up, 8 in 2024-04-14T15:13:23.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:23 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-9"}]: dispatch 2024-04-14T15:13:23.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:23 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:13:23.651 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:23.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:23.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:13:23.691 INFO:tasks.workunit.client.0.smithi090.stdout:service: mgr: y(active, since 2m), standbys: x 2024-04-14T15:13:23.691 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: osd: 8 osds: 8 up (since 3m), 8 in (since 3m) 2024-04-14T15:13:23.691 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: flags noscrub,nodeep-scrub 2024-04-14T15:13:23.691 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2024-04-14T15:13:23.691 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: laundry: 1 daemon active (1 hosts) 2024-04-14T15:13:23.691 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-04-14T15:13:23.691 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: 2024-04-14T15:13:23.692 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: data: 2024-04-14T15:13:23.692 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: pools: 31 pools, 836 pgs 2024-04-14T15:13:23.692 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: objects: 249 objects, 25 MiB 2024-04-14T15:13:23.692 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: usage: 282 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:13:23.692 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: pgs: 45.933% pgs unknown 2024-04-14T15:13:23.692 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: 7.656% pgs not active 2024-04-14T15:13:23.692 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: 388 active+clean 2024-04-14T15:13:23.692 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: 384 unknown 2024-04-14T15:13:23.692 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: 64 creating+peering 2024-04-14T15:13:23.692 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: 2024-04-14T15:13:23.692 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: io: 2024-04-14T15:13:23.692 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: client: 4.1 KiB/s rd, 6.2 MiB/s wr, 260 op/s rd, 294 op/s wr 2024-04-14T15:13:23.692 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: 2024-04-14T15:13:23.693 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: 2024-04-14T15:13:23.693 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: [ OK ] LibRadosService.StatusFormat (4183 ms) 2024-04-14T15:13:23.693 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: [ RUN ] LibRadosService.Status 2024-04-14T15:13:23.693 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: [ OK ] LibRadosService.Status (20020 ms) 2024-04-14T15:13:23.693 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: [----------] 4 tests from LibRadosService (29284 ms total) 2024-04-14T15:13:23.693 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: 2024-04-14T15:13:23.693 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: [----------] Global test environment tear-down 2024-04-14T15:13:23.693 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: [==========] 4 tests from 1 test suite ran. (29284 ms total) 2024-04-14T15:13:23.693 INFO:tasks.workunit.client.0.smithi090.stdout: api_service: [ PASSED ] 4 tests. 2024-04-14T15:13:23.753 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:13:23.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:23.779 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:23.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:13:23.880 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:13:24.197 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:24.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:24.198 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:13:24.299 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:13:24.318 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:24 smithi090 ceph-mon[30399]: pgmap v149: 580 pgs: 224 unknown, 4 creating+activating, 352 active+clean; 145 MiB data, 766 MiB used, 714 GiB / 715 GiB avail 2024-04-14T15:13:24.318 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:24 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:24.318 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:24 smithi090 ceph-mon[30399]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:13:24.318 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:24 smithi090 ceph-mon[30399]: from='client.38086 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi090-93709-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi090-93709-10"}]': finished 2024-04-14T15:13:24.318 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:24 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-9"}]': finished 2024-04-14T15:13:24.318 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:24 smithi090 ceph-mon[30399]: osdmap e136: 8 total, 8 up, 8 in 2024-04-14T15:13:24.319 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:13:23 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:13:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:13:24.319 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:24 smithi090 ceph-mon[41670]: pgmap v149: 580 pgs: 224 unknown, 4 creating+activating, 352 active+clean; 145 MiB data, 766 MiB used, 714 GiB / 715 GiB avail 2024-04-14T15:13:24.319 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:24 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:24.319 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:24 smithi090 ceph-mon[41670]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:13:24.319 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:24 smithi090 ceph-mon[41670]: from='client.38086 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi090-93709-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi090-93709-10"}]': finished 2024-04-14T15:13:24.319 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:24 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-9"}]': finished 2024-04-14T15:13:24.319 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:24 smithi090 ceph-mon[41670]: osdmap e136: 8 total, 8 up, 8 in 2024-04-14T15:13:24.456 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:24.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:24.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:13:24.558 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:13:24.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:24.583 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:24.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:13:24.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:24 smithi092 ceph-mon[33966]: pgmap v149: 580 pgs: 224 unknown, 4 creating+activating, 352 active+clean; 145 MiB data, 766 MiB used, 714 GiB / 715 GiB avail 2024-04-14T15:13:24.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:24 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:24.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:24 smithi092 ceph-mon[33966]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:13:24.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:24 smithi092 ceph-mon[33966]: from='client.38086 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi090-93709-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi090-93709-10"}]': finished 2024-04-14T15:13:24.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:24 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-9"}]': finished 2024-04-14T15:13:24.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:24 smithi092 ceph-mon[33966]: osdmap e136: 8 total, 8 up, 8 in 2024-04-14T15:13:24.684 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:13:24.706 INFO:tasks.workunit.client.0.smithi090.stdout: api_service_pp: [==========] Running 4 tests from 1 test suite. 2024-04-14T15:13:24.706 INFO:tasks.workunit.client.0.smithi090.stdout: api_service_pp: [----------] Global test environment set-up. 2024-04-14T15:13:24.706 INFO:tasks.workunit.client.0.smithi090.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2024-04-14T15:13:24.706 INFO:tasks.workunit.client.0.smithi090.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2024-04-14T15:13:24.706 INFO:tasks.workunit.client.0.smithi090.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5071 ms) 2024-04-14T15:13:24.707 INFO:tasks.workunit.client.0.smithi090.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2024-04-14T15:13:24.707 INFO:tasks.workunit.client.0.smithi090.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (60 ms) 2024-04-14T15:13:24.707 INFO:tasks.workunit.client.0.smithi090.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2024-04-14T15:13:24.707 INFO:tasks.workunit.client.0.smithi090.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (20040 ms) 2024-04-14T15:13:24.707 INFO:tasks.workunit.client.0.smithi090.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2024-04-14T15:13:24.707 INFO:tasks.workunit.client.0.smithi090.stdout: api_service_pp: attempt 0 of 20 2024-04-14T15:13:24.707 INFO:tasks.workunit.client.0.smithi090.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (5132 ms) 2024-04-14T15:13:24.707 INFO:tasks.workunit.client.0.smithi090.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (30303 ms total) 2024-04-14T15:13:24.707 INFO:tasks.workunit.client.0.smithi090.stdout: api_service_pp: 2024-04-14T15:13:24.707 INFO:tasks.workunit.client.0.smithi090.stdout: api_service_pp: [----------] Global test environment tear-down 2024-04-14T15:13:24.707 INFO:tasks.workunit.client.0.smithi090.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (30303 ms total) 2024-04-14T15:13:24.707 INFO:tasks.workunit.client.0.smithi090.stdout: api_service_pp: [ PASSED ] 4 tests. 2024-04-14T15:13:24.708 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:24.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:24.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:13:24.810 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:13:24.930 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:24.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:24.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:13:25.032 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:13:25.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:25.057 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:25.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:13:25.122 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:13:24 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:13:25.158 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:13:25.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:25.183 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:25.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:13:25.184 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-14T15:13:25.285 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:13:25.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:25.309 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:25.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:13:25.315 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:25 smithi090 ceph-mon[30399]: 4.12 scrub starts 2024-04-14T15:13:25.316 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:25 smithi090 ceph-mon[30399]: 4.12 scrub ok 2024-04-14T15:13:25.316 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:25 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:25.316 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:25 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2133272360' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-04-14T15:13:25.316 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:25 smithi090 ceph-mon[30399]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-04-14T15:13:25.316 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:25 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:13:25.317 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:25 smithi090 ceph-mon[30399]: osdmap e137: 8 total, 8 up, 8 in 2024-04-14T15:13:25.317 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:25 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/903363490' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:25.317 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:25 smithi090 ceph-mon[30399]: from='client.40001 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:25.317 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:25 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/790649677' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi090-93363-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:25.317 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:25 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1865301891' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi090-93342-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:25.317 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:25 smithi090 ceph-mon[30399]: from='client.41506 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi090-93363-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:25.317 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:25 smithi090 ceph-mon[30399]: from='client.41515 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi090-93342-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:25.319 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:25 smithi090 ceph-mon[41670]: 4.12 scrub starts 2024-04-14T15:13:25.319 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:25 smithi090 ceph-mon[41670]: 4.12 scrub ok 2024-04-14T15:13:25.319 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:25 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:25.319 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:25 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2133272360' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-04-14T15:13:25.320 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:25 smithi090 ceph-mon[41670]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-04-14T15:13:25.320 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:25 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:13:25.320 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:25 smithi090 ceph-mon[41670]: osdmap e137: 8 total, 8 up, 8 in 2024-04-14T15:13:25.320 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:25 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/903363490' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:25.320 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:25 smithi090 ceph-mon[41670]: from='client.40001 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:25.320 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:25 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/790649677' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi090-93363-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:25.321 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:25 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1865301891' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi090-93342-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:25.321 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:25 smithi090 ceph-mon[41670]: from='client.41506 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi090-93363-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:25.321 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:25 smithi090 ceph-mon[41670]: from='client.41515 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi090-93342-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:25.411 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:13:25.488 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:25.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:25.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:13:25.589 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:13:25.613 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:25.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:25.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:13:25.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:25 smithi092 ceph-mon[33966]: 4.12 scrub starts 2024-04-14T15:13:25.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:25 smithi092 ceph-mon[33966]: 4.12 scrub ok 2024-04-14T15:13:25.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:25 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:25.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:25 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2133272360' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-04-14T15:13:25.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:25 smithi092 ceph-mon[33966]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-04-14T15:13:25.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:25 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:13:25.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:25 smithi092 ceph-mon[33966]: osdmap e137: 8 total, 8 up, 8 in 2024-04-14T15:13:25.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:25 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/903363490' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:25.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:25 smithi092 ceph-mon[33966]: from='client.40001 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:25.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:25 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/790649677' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi090-93363-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:25.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:25 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1865301891' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi090-93342-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:25.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:25 smithi092 ceph-mon[33966]: from='client.41506 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi090-93363-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:25.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:25 smithi092 ceph-mon[33966]: from='client.41515 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi090-93342-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:25.714 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:13:25.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:25.739 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:25.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:13:25.840 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:13:26.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:26.016 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:26.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:13:26.117 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:13:26.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:26.142 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:26.142 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:13:26.243 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:13:26.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:26.342 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:26.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:13:26.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:26 smithi090 ceph-mon[41670]: 4.18 scrub starts 2024-04-14T15:13:26.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:26 smithi090 ceph-mon[41670]: 4.18 scrub ok 2024-04-14T15:13:26.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:26 smithi090 ceph-mon[41670]: pgmap v152: 524 pgs: 1 active+clean+snaptrim, 192 unknown, 8 creating+peering, 323 active+clean; 145 MiB data, 789 MiB used, 714 GiB / 715 GiB avail 2024-04-14T15:13:26.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:26 smithi090 ceph-mon[41670]: 6.3 scrub starts 2024-04-14T15:13:26.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:26 smithi090 ceph-mon[41670]: 6.3 scrub ok 2024-04-14T15:13:26.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:26 smithi090 ceph-mon[41670]: 5.d scrub starts 2024-04-14T15:13:26.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:26 smithi090 ceph-mon[41670]: 5.d scrub ok 2024-04-14T15:13:26.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:26 smithi090 ceph-mon[41670]: 5.19 scrub starts 2024-04-14T15:13:26.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:26 smithi090 ceph-mon[41670]: 5.19 scrub ok 2024-04-14T15:13:26.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:26 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:26.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:26 smithi090 ceph-mon[41670]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:13:26.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:26 smithi090 ceph-mon[41670]: from='client.40001 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:26.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:26 smithi090 ceph-mon[41670]: from='client.41506 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi090-93363-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:26.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:26 smithi090 ceph-mon[41670]: from='client.41515 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi090-93342-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:26.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:26 smithi090 ceph-mon[41670]: osdmap e138: 8 total, 8 up, 8 in 2024-04-14T15:13:26.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:26 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:26.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:26 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:26.443 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:13:26.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:26 smithi090 ceph-mon[30399]: 4.18 scrub starts 2024-04-14T15:13:26.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:26 smithi090 ceph-mon[30399]: 4.18 scrub ok 2024-04-14T15:13:26.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:26 smithi090 ceph-mon[30399]: pgmap v152: 524 pgs: 1 active+clean+snaptrim, 192 unknown, 8 creating+peering, 323 active+clean; 145 MiB data, 789 MiB used, 714 GiB / 715 GiB avail 2024-04-14T15:13:26.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:26 smithi090 ceph-mon[30399]: 6.3 scrub starts 2024-04-14T15:13:26.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:26 smithi090 ceph-mon[30399]: 6.3 scrub ok 2024-04-14T15:13:26.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:26 smithi090 ceph-mon[30399]: 5.d scrub starts 2024-04-14T15:13:26.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:26 smithi090 ceph-mon[30399]: 5.d scrub ok 2024-04-14T15:13:26.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:26 smithi090 ceph-mon[30399]: 5.19 scrub starts 2024-04-14T15:13:26.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:26 smithi090 ceph-mon[30399]: 5.19 scrub ok 2024-04-14T15:13:26.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:26 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:26.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:26 smithi090 ceph-mon[30399]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:13:26.445 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:26 smithi090 ceph-mon[30399]: from='client.40001 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:26.445 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:26 smithi090 ceph-mon[30399]: from='client.41506 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi090-93363-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:26.445 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:26 smithi090 ceph-mon[30399]: from='client.41515 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi090-93342-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:26.445 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:26 smithi090 ceph-mon[30399]: osdmap e138: 8 total, 8 up, 8 in 2024-04-14T15:13:26.445 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:26 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:26.445 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:26 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:26.510 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:26.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:26.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:13:26.612 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:13:26.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:26 smithi092 ceph-mon[33966]: 4.18 scrub starts 2024-04-14T15:13:26.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:26 smithi092 ceph-mon[33966]: 4.18 scrub ok 2024-04-14T15:13:26.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:26 smithi092 ceph-mon[33966]: pgmap v152: 524 pgs: 1 active+clean+snaptrim, 192 unknown, 8 creating+peering, 323 active+clean; 145 MiB data, 789 MiB used, 714 GiB / 715 GiB avail 2024-04-14T15:13:26.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:26 smithi092 ceph-mon[33966]: 6.3 scrub starts 2024-04-14T15:13:26.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:26 smithi092 ceph-mon[33966]: 6.3 scrub ok 2024-04-14T15:13:26.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:26 smithi092 ceph-mon[33966]: 5.d scrub starts 2024-04-14T15:13:26.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:26 smithi092 ceph-mon[33966]: 5.d scrub ok 2024-04-14T15:13:26.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:26 smithi092 ceph-mon[33966]: 5.19 scrub starts 2024-04-14T15:13:26.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:26 smithi092 ceph-mon[33966]: 5.19 scrub ok 2024-04-14T15:13:26.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:26 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:26.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:26 smithi092 ceph-mon[33966]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:13:26.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:26 smithi092 ceph-mon[33966]: from='client.40001 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93656-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:26.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:26 smithi092 ceph-mon[33966]: from='client.41506 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi090-93363-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:26.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:26 smithi092 ceph-mon[33966]: from='client.41515 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi090-93342-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:26.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:26 smithi092 ceph-mon[33966]: osdmap e138: 8 total, 8 up, 8 in 2024-04-14T15:13:26.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:26 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:26.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:26 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:26.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:26.710 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:26.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:13:26.811 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:13:26.834 INFO:tasks.workunit.client.0.smithi090.stdout: misc: Running main() from gmock_main.cc 2024-04-14T15:13:26.834 INFO:tasks.workunit.client.0.smithi090.stdout: misc: [==========] Running 12 tests from 1 test suite. 2024-04-14T15:13:26.834 INFO:tasks.workunit.client.0.smithi090.stdout: misc: [----------] Global test environment set-up. 2024-04-14T15:13:26.834 INFO:tasks.workunit.client.0.smithi090.stdout: misc: [----------] 12 tests from NeoRadosMisc 2024-04-14T15:13:26.834 INFO:tasks.workunit.client.0.smithi090.stdout: misc: [ RUN ] NeoRadosMisc.Version 2024-04-14T15:13:26.834 INFO:tasks.workunit.client.0.smithi090.stdout: misc: [ OK ] NeoRadosMisc.Version (1521 ms) 2024-04-14T15:13:26.834 INFO:tasks.workunit.client.0.smithi090.stdout: misc: [ RUN ] NeoRadosMisc.WaitOSDMap 2024-04-14T15:13:26.834 INFO:tasks.workunit.client.0.smithi090.stdout: misc: [ OK ] NeoRadosMisc.WaitOSDMap (1995 ms) 2024-04-14T15:13:26.835 INFO:tasks.workunit.client.0.smithi090.stdout: misc: [ RUN ] NeoRadosMisc.LongName 2024-04-14T15:13:26.835 INFO:tasks.workunit.client.0.smithi090.stdout: misc: [ OK ] NeoRadosMisc.LongName (2738 ms) 2024-04-14T15:13:26.835 INFO:tasks.workunit.client.0.smithi090.stdout: misc: [ RUN ] NeoRadosMisc.LongLocator 2024-04-14T15:13:26.835 INFO:tasks.workunit.client.0.smithi090.stdout: misc: [ OK ] NeoRadosMisc.LongLocator (2999 ms) 2024-04-14T15:13:26.835 INFO:tasks.workunit.client.0.smithi090.stdout: misc: [ RUN ] NeoRadosMisc.LongNamespace 2024-04-14T15:13:26.835 INFO:tasks.workunit.client.0.smithi090.stdout: misc: [ OK ] NeoRadosMisc.LongNamespace (2984 ms) 2024-04-14T15:13:26.835 INFO:tasks.workunit.client.0.smithi090.stdout: misc: [ RUN ] NeoRadosMisc.LongAttrName 2024-04-14T15:13:26.835 INFO:tasks.workunit.client.0.smithi090.stdout: misc: [ OK ] NeoRadosMisc.LongAttrName (3053 ms) 2024-04-14T15:13:26.835 INFO:tasks.workunit.client.0.smithi090.stdout: misc: [ RUN ] NeoRadosMisc.Exec 2024-04-14T15:13:26.835 INFO:tasks.workunit.client.0.smithi090.stdout: misc: [ OK ] NeoRadosMisc.Exec (2980 ms) 2024-04-14T15:13:26.835 INFO:tasks.workunit.client.0.smithi090.stdout: misc: [ RUN ] NeoRadosMisc.Operate1 2024-04-14T15:13:26.835 INFO:tasks.workunit.client.0.smithi090.stdout: misc: [ OK ] NeoRadosMisc.Operate1 (2965 ms) 2024-04-14T15:13:26.835 INFO:tasks.workunit.client.0.smithi090.stdout: misc: [ RUN ] NeoRadosMisc.Operate2 2024-04-14T15:13:26.835 INFO:tasks.workunit.client.0.smithi090.stdout: misc: [ OK ] NeoRadosMisc.Operate2 (3043 ms) 2024-04-14T15:13:26.836 INFO:tasks.workunit.client.0.smithi090.stdout: misc: [ RUN ] NeoRadosMisc.BigObject 2024-04-14T15:13:26.836 INFO:tasks.workunit.client.0.smithi090.stdout: misc: [ OK ] NeoRadosMisc.BigObject (2970 ms) 2024-04-14T15:13:26.836 INFO:tasks.workunit.client.0.smithi090.stdout: misc: [ RUN ] NeoRadosMisc.BigAttr 2024-04-14T15:13:26.836 INFO:tasks.workunit.client.0.smithi090.stdout: misc: [ OK ] NeoRadosMisc.BigAttr (2021 ms) 2024-04-14T15:13:26.836 INFO:tasks.workunit.client.0.smithi090.stdout: misc: [ RUN ] NeoRadosMisc.WriteSame 2024-04-14T15:13:26.836 INFO:tasks.workunit.client.0.smithi090.stdout: misc: [ OK ] NeoRadosMisc.WriteSame (2976 ms) 2024-04-14T15:13:26.836 INFO:tasks.workunit.client.0.smithi090.stdout: misc: [----------] 12 tests from NeoRadosMisc (32246 ms total) 2024-04-14T15:13:26.836 INFO:tasks.workunit.client.0.smithi090.stdout: misc: 2024-04-14T15:13:26.836 INFO:tasks.workunit.client.0.smithi090.stdout: misc: [----------] Global test environment tear-down 2024-04-14T15:13:26.836 INFO:tasks.workunit.client.0.smithi090.stdout: misc: [==========] 12 tests from 1 test suite ran. (32246 ms total) 2024-04-14T15:13:26.836 INFO:tasks.workunit.client.0.smithi090.stdout: misc: [ PASSED ] 12 tests. 2024-04-14T15:13:26.854 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:26.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:26.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:13:26.956 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:13:27.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:27.043 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:27.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:13:27.144 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:13:27.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:27.091+0000 7fab5646a700 1 -- 172.21.15.90:0/4097334852 >> v1:172.21.15.90:6790/0 conn(0x7fab5010fba0 legacy=0x7fab5010ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:27.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:27.092+0000 7fab5646a700 1 -- 172.21.15.90:0/4097334852 shutdown_connections 2024-04-14T15:13:27.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:27.092+0000 7fab5646a700 1 -- 172.21.15.90:0/4097334852 >> 172.21.15.90:0/4097334852 conn(0x7fab5006d810 msgr2=0x7fab5006dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:13:27.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:27.095+0000 7fab5646a700 1 -- 172.21.15.90:0/4097334852 shutdown_connections 2024-04-14T15:13:27.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:27.095+0000 7fab5646a700 1 -- 172.21.15.90:0/4097334852 wait complete. 2024-04-14T15:13:27.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:27.096+0000 7fab5646a700 1 librados: starting msgr at 2024-04-14T15:13:27.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:27.096+0000 7fab5646a700 1 librados: starting objecter 2024-04-14T15:13:27.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:27.096+0000 7fab5646a700 1 Processor -- start 2024-04-14T15:13:27.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:27.096+0000 7fab5646a700 1 -- start start 2024-04-14T15:13:27.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:27.097+0000 7fab5646a700 1 librados: setting wanted keys 2024-04-14T15:13:27.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:27.097+0000 7fab5646a700 1 librados: calling monclient init 2024-04-14T15:13:27.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:27.097+0000 7fab55468700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fab5010fba0 0x7fab5010f160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.90:38476/0 (socket says 172.21.15.90:38476) 2024-04-14T15:13:27.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:27.097+0000 7fab55468700 1 -- 172.21.15.90:0/3644626639 learned_addr learned my addr 172.21.15.90:0/3644626639 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:13:27.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:27.097+0000 7fab5646a700 1 -- 172.21.15.90:0/3644626639 --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fab501b6400 con 0x7fab50110690 2024-04-14T15:13:27.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:27.097+0000 7fab5646a700 1 -- 172.21.15.90:0/3644626639 --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fab501b7620 con 0x7fab5010fba0 2024-04-14T15:13:27.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:27.098+0000 7fab5646a700 1 -- 172.21.15.90:0/3644626639 --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fab501b77f0 con 0x7fab5011e130 2024-04-14T15:13:27.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:27.098+0000 7fab467fc700 10 client.?.objecter ms_handle_connect 0x7fab5010fba0 2024-04-14T15:13:27.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:27.098+0000 7fab467fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:13:27.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:27.098+0000 7fab467fc700 1 -- 172.21.15.90:0/3644626639 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 12135424 0 0) 0x7fab501b7620 con 0x7fab5010fba0 2024-04-14T15:13:27.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:27.098+0000 7fab467fc700 1 -- 172.21.15.90:0/3644626639 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fab3c0035b0 con 0x7fab5010fba0 2024-04-14T15:13:27.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:27.099+0000 7fab467fc700 10 client.?.objecter ms_handle_connect 0x7fab50110690 2024-04-14T15:13:27.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:27.099+0000 7fab467fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:13:27.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:27.099+0000 7fab467fc700 10 client.?.objecter ms_handle_connect 0x7fab5011e130 2024-04-14T15:13:27.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:27.099+0000 7fab467fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:13:27.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:27.099+0000 7fab467fc700 1 -- 172.21.15.90:0/3644626639 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 215754493 0 0) 0x7fab3c0035b0 con 0x7fab5010fba0 2024-04-14T15:13:27.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:27.099+0000 7fab467fc700 1 -- 172.21.15.90:0/3644626639 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fab501b7620 con 0x7fab5010fba0 2024-04-14T15:13:27.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:27.100+0000 7fab467fc700 1 -- 172.21.15.90:0/3644626639 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1593568752 0 0) 0x7fab501b6400 con 0x7fab50110690 2024-04-14T15:13:27.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:27.100+0000 7fab467fc700 1 -- 172.21.15.90:0/3644626639 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fab3c0035b0 con 0x7fab50110690 2024-04-14T15:13:27.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:27.100+0000 7fab467fc700 1 -- 172.21.15.90:0/3644626639 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 425073935 0 0) 0x7fab501b77f0 con 0x7fab5011e130 2024-04-14T15:13:27.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:27.100+0000 7fab467fc700 1 -- 172.21.15.90:0/3644626639 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fab501b6400 con 0x7fab5011e130 2024-04-14T15:13:27.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:27.100+0000 7fab467fc700 1 -- 172.21.15.90:0/3644626639 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fab4c004a30 con 0x7fab5010fba0 2024-04-14T15:13:27.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:27.100+0000 7fab467fc700 1 -- 172.21.15.90:0/3644626639 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2914481180 0 0) 0x7fab501b7620 con 0x7fab5010fba0 2024-04-14T15:13:27.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:27.100+0000 7fab467fc700 1 -- 172.21.15.90:0/3644626639 >> v1:172.21.15.90:6790/0 conn(0x7fab5011e130 legacy=0x7fab501b4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:27.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:27.100+0000 7fab467fc700 1 -- 172.21.15.90:0/3644626639 >> v1:172.21.15.90:6789/0 conn(0x7fab50110690 legacy=0x7fab501b13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:27.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:27.100+0000 7fab467fc700 1 -- 172.21.15.90:0/3644626639 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab501b7a00 con 0x7fab5010fba0 2024-04-14T15:13:27.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:27.100+0000 7fab5646a700 1 -- 172.21.15.90:0/3644626639 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fab501b7cf0 con 0x7fab5010fba0 2024-04-14T15:13:27.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:27.101+0000 7fab5646a700 10 client.44446.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:13:27.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:27.101+0000 7fab5646a700 1 -- 172.21.15.90:0/3644626639 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fab501b81b0 con 0x7fab5010fba0 2024-04-14T15:13:27.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:27.101+0000 7fab5646a700 1 librados: init done 2024-04-14T15:13:27.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:27.101+0000 7fab467fc700 1 -- 172.21.15.90:0/3644626639 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fab4c001ec0 con 0x7fab5010fba0 2024-04-14T15:13:27.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:27.101+0000 7fab467fc700 1 -- 172.21.15.90:0/3644626639 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fab4c005550 con 0x7fab5010fba0 2024-04-14T15:13:27.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:27.102+0000 7fab467fc700 1 -- 172.21.15.90:0/3644626639 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fab4c0056f0 con 0x7fab5010fba0 2024-04-14T15:13:27.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:27.103+0000 7fab5646a700 1 -- 172.21.15.90:0/3644626639 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab34005120 con 0x7fab5010fba0 2024-04-14T15:13:27.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:27.103+0000 7fab467fc700 1 -- 172.21.15.90:0/3644626639 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(139..139 src has 51..139) ==== 11371+0+0 (unknown 3806958867 0 0) 0x7fab4c08cc30 con 0x7fab5010fba0 2024-04-14T15:13:27.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:27.103+0000 7fab467fc700 10 client.44446.objecter ms_dispatch 0x7fab50001550 osd_map(139..139 src has 51..139) 2024-04-14T15:13:27.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:27.103+0000 7fab467fc700 3 client.44446.objecter handle_osd_map got epochs [139,139] > 0 2024-04-14T15:13:27.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:27.103+0000 7fab467fc700 3 client.44446.objecter handle_osd_map decoding full epoch 139 2024-04-14T15:13:27.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:27.104+0000 7fab467fc700 20 client.44446.objecter dump_active .. 0 homeless 2024-04-14T15:13:27.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:27.106+0000 7fab467fc700 10 client.44446.objecter ms_handle_connect 0x7fab3c0710e0 2024-04-14T15:13:27.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:27.109+0000 7fab467fc700 1 -- 172.21.15.90:0/3644626639 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fab4c0589b0 con 0x7fab5010fba0 2024-04-14T15:13:27.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:27.218 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:27.218 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:13:27.319 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:13:27.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:27.414 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:27.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:13:27.425 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:27 smithi090 ceph-mon[30399]: 2.2 scrub starts 2024-04-14T15:13:27.425 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:27 smithi090 ceph-mon[30399]: 2.2 scrub ok 2024-04-14T15:13:27.425 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:27 smithi090 ceph-mon[30399]: 6.12 scrub starts 2024-04-14T15:13:27.426 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:27 smithi090 ceph-mon[30399]: 6.12 scrub ok 2024-04-14T15:13:27.426 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:27 smithi090 ceph-mon[30399]: 4.5 scrub starts 2024-04-14T15:13:27.426 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:27 smithi090 ceph-mon[30399]: 4.5 scrub ok 2024-04-14T15:13:27.426 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:27 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:27.426 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:27 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:27.426 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:27 smithi090 ceph-mon[30399]: osdmap e139: 8 total, 8 up, 8 in 2024-04-14T15:13:27.426 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:27 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/903363490' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93656-10", "tierpool":"test-rados-api-smithi090-93656-10-cache", "force_nonempty":""}]: dispatch 2024-04-14T15:13:27.426 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:27 smithi090 ceph-mon[30399]: from='client.40001 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93656-10", "tierpool":"test-rados-api-smithi090-93656-10-cache", "force_nonempty":""}]: dispatch 2024-04-14T15:13:27.426 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:27 smithi090 ceph-mon[41670]: 2.2 scrub starts 2024-04-14T15:13:27.426 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:27 smithi090 ceph-mon[41670]: 2.2 scrub ok 2024-04-14T15:13:27.427 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:27 smithi090 ceph-mon[41670]: 6.12 scrub starts 2024-04-14T15:13:27.427 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:27 smithi090 ceph-mon[41670]: 6.12 scrub ok 2024-04-14T15:13:27.427 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:27 smithi090 ceph-mon[41670]: 4.5 scrub starts 2024-04-14T15:13:27.427 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:27 smithi090 ceph-mon[41670]: 4.5 scrub ok 2024-04-14T15:13:27.427 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:27 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:27.427 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:27 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:27.427 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:27 smithi090 ceph-mon[41670]: osdmap e139: 8 total, 8 up, 8 in 2024-04-14T15:13:27.427 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:27 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/903363490' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93656-10", "tierpool":"test-rados-api-smithi090-93656-10-cache", "force_nonempty":""}]: dispatch 2024-04-14T15:13:27.427 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:27 smithi090 ceph-mon[41670]: from='client.40001 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93656-10", "tierpool":"test-rados-api-smithi090-93656-10-cache", "force_nonempty":""}]: dispatch 2024-04-14T15:13:27.450 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:13:27.515 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:13:27.516 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:13:27.529 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-14T15:13:27.530 INFO:teuthology.orchestra.run.smithi090.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-14T15:13:27.530 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-14T15:13:27.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:27.547+0000 7fab5646a700 1 -- 172.21.15.90:0/3644626639 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7fab34005410 con 0x7fab5010fba0 2024-04-14T15:13:27.621 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:27.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:27.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:13:27.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:27 smithi092 ceph-mon[33966]: 2.2 scrub starts 2024-04-14T15:13:27.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:27 smithi092 ceph-mon[33966]: 2.2 scrub ok 2024-04-14T15:13:27.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:27 smithi092 ceph-mon[33966]: 6.12 scrub starts 2024-04-14T15:13:27.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:27 smithi092 ceph-mon[33966]: 6.12 scrub ok 2024-04-14T15:13:27.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:27 smithi092 ceph-mon[33966]: 4.5 scrub starts 2024-04-14T15:13:27.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:27 smithi092 ceph-mon[33966]: 4.5 scrub ok 2024-04-14T15:13:27.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:27 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:27.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:27 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:27.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:27 smithi092 ceph-mon[33966]: osdmap e139: 8 total, 8 up, 8 in 2024-04-14T15:13:27.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:27 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/903363490' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93656-10", "tierpool":"test-rados-api-smithi090-93656-10-cache", "force_nonempty":""}]: dispatch 2024-04-14T15:13:27.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:27 smithi092 ceph-mon[33966]: from='client.40001 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93656-10", "tierpool":"test-rados-api-smithi090-93656-10-cache", "force_nonempty":""}]: dispatch 2024-04-14T15:13:27.722 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:13:27.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:27.747 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:27.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:13:27.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:27.821+0000 7fab467fc700 1 -- 172.21.15.90:0/3644626639 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v140) ==== 101+0+0 (unknown 1217501619 0 0) 0x7fab4c05c660 con 0x7fab5010fba0 2024-04-14T15:13:27.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:27.824+0000 7fab5646a700 10 librados: watch_flush enter 2024-04-14T15:13:27.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:27.827+0000 7fab5646a700 10 librados: watch_flush exit 2024-04-14T15:13:27.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:27.827+0000 7fab5646a700 20 client.44446.objecter shutdown clearing up homeless session... 2024-04-14T15:13:27.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:27.827+0000 7fab5646a700 10 client.44446.objecter successfully canceled tick 2024-04-14T15:13:27.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:27.828+0000 7fab5646a700 1 -- 172.21.15.90:0/3644626639 >> v1:172.21.15.90:6800/420935427 conn(0x7fab3c0710e0 legacy=0x7fab3c0735a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:27.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:27.828+0000 7fab5646a700 1 -- 172.21.15.90:0/3644626639 >> v1:172.21.15.92:6789/0 conn(0x7fab5010fba0 legacy=0x7fab5010f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:27.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:27.830+0000 7fab5646a700 1 -- 172.21.15.90:0/3644626639 shutdown_connections 2024-04-14T15:13:27.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:27.830+0000 7fab5646a700 1 -- 172.21.15.90:0/3644626639 >> 172.21.15.90:0/3644626639 conn(0x7fab5006d810 msgr2=0x7fab50072ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:13:27.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:27.833+0000 7fab5646a700 1 -- 172.21.15.90:0/3644626639 shutdown_connections 2024-04-14T15:13:27.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:27.833+0000 7fab5646a700 1 -- 172.21.15.90:0/3644626639 wait complete. 2024-04-14T15:13:27.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:27.833+0000 7fab5646a700 1 librados: shutdown 2024-04-14T15:13:27.835 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is unset 2024-04-14T15:13:27.848 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:13:27.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:27.950 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:27.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:13:28.051 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:13:28.075 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:28.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:28.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:13:28.176 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:13:28.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:28.262 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:28.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:13:28.363 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:13:28.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:28.495 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:28.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:13:28.596 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:13:28.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:28 smithi092 ceph-mon[33966]: 6.2 scrub starts 2024-04-14T15:13:28.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:28 smithi092 ceph-mon[33966]: 6.2 scrub ok 2024-04-14T15:13:28.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:28 smithi092 ceph-mon[33966]: pgmap v155: 460 pgs: 1 active+clean+snaptrim, 128 unknown, 8 creating+peering, 323 active+clean; 145 MiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.5 KiB/s wr, 5 op/s 2024-04-14T15:13:28.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:28 smithi092 ceph-mon[33966]: 2.15 scrub starts 2024-04-14T15:13:28.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:28 smithi092 ceph-mon[33966]: 2.15 scrub ok 2024-04-14T15:13:28.625 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:28 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3644626639' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:13:28.625 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:28 smithi092 ceph-mon[33966]: from='client.44446 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:13:28.625 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:28 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:28.626 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:28 smithi092 ceph-mon[33966]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-14T15:13:28.626 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:28 smithi092 ceph-mon[33966]: from='client.40001 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93656-10", "tierpool":"test-rados-api-smithi090-93656-10-cache", "force_nonempty":""}]': finished 2024-04-14T15:13:28.626 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:28 smithi092 ceph-mon[33966]: from='client.44446 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-14T15:13:28.626 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:28 smithi092 ceph-mon[33966]: osdmap e140: 8 total, 8 up, 8 in 2024-04-14T15:13:28.626 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:28 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/4039543108' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi090-93363-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:28.626 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:28 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/903363490' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93656-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:28.627 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:28 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/685063981' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi090-93342-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:28.627 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:28 smithi092 ceph-mon[33966]: from='client.44341 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi090-93363-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:28.627 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:28 smithi092 ceph-mon[33966]: from='client.40001 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93656-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:28.627 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:28 smithi092 ceph-mon[33966]: from='client.44350 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi090-93342-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:28.627 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:28 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:13:28.627 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:28 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:13:28.682 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:28.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:28.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:13:28.694 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:28 smithi090 ceph-mon[41670]: 6.2 scrub starts 2024-04-14T15:13:28.694 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:28 smithi090 ceph-mon[41670]: 6.2 scrub ok 2024-04-14T15:13:28.695 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:28 smithi090 ceph-mon[41670]: pgmap v155: 460 pgs: 1 active+clean+snaptrim, 128 unknown, 8 creating+peering, 323 active+clean; 145 MiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.5 KiB/s wr, 5 op/s 2024-04-14T15:13:28.695 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:28 smithi090 ceph-mon[41670]: 2.15 scrub starts 2024-04-14T15:13:28.695 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:28 smithi090 ceph-mon[41670]: 2.15 scrub ok 2024-04-14T15:13:28.695 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:28 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3644626639' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:13:28.695 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:28 smithi090 ceph-mon[41670]: from='client.44446 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:13:28.695 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:28 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:28.695 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:28 smithi090 ceph-mon[41670]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-14T15:13:28.695 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:28 smithi090 ceph-mon[41670]: from='client.40001 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93656-10", "tierpool":"test-rados-api-smithi090-93656-10-cache", "force_nonempty":""}]': finished 2024-04-14T15:13:28.695 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:28 smithi090 ceph-mon[41670]: from='client.44446 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-14T15:13:28.695 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:28 smithi090 ceph-mon[41670]: osdmap e140: 8 total, 8 up, 8 in 2024-04-14T15:13:28.695 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:28 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/4039543108' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi090-93363-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:28.695 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:28 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/903363490' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93656-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:28.695 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:28 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/685063981' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi090-93342-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:28.695 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:28 smithi090 ceph-mon[41670]: from='client.44341 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi090-93363-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:28.696 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:28 smithi090 ceph-mon[41670]: from='client.40001 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93656-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:28.696 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:28 smithi090 ceph-mon[41670]: from='client.44350 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi090-93342-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:28.696 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:28 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:13:28.696 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:28 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:13:28.696 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:28 smithi090 ceph-mon[30399]: 6.2 scrub starts 2024-04-14T15:13:28.696 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:28 smithi090 ceph-mon[30399]: 6.2 scrub ok 2024-04-14T15:13:28.696 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:28 smithi090 ceph-mon[30399]: pgmap v155: 460 pgs: 1 active+clean+snaptrim, 128 unknown, 8 creating+peering, 323 active+clean; 145 MiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.5 KiB/s wr, 5 op/s 2024-04-14T15:13:28.696 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:28 smithi090 ceph-mon[30399]: 2.15 scrub starts 2024-04-14T15:13:28.696 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:28 smithi090 ceph-mon[30399]: 2.15 scrub ok 2024-04-14T15:13:28.696 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:28 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3644626639' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:13:28.696 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:28 smithi090 ceph-mon[30399]: from='client.44446 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:13:28.697 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:28 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:28.697 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:28 smithi090 ceph-mon[30399]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-14T15:13:28.697 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:28 smithi090 ceph-mon[30399]: from='client.40001 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93656-10", "tierpool":"test-rados-api-smithi090-93656-10-cache", "force_nonempty":""}]': finished 2024-04-14T15:13:28.697 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:28 smithi090 ceph-mon[30399]: from='client.44446 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-14T15:13:28.697 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:28 smithi090 ceph-mon[30399]: osdmap e140: 8 total, 8 up, 8 in 2024-04-14T15:13:28.697 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:28 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/4039543108' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi090-93363-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:28.697 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:28 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/903363490' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93656-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:28.697 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:28 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/685063981' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi090-93342-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:28.697 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:28 smithi090 ceph-mon[30399]: from='client.44341 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi090-93363-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:28.697 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:28 smithi090 ceph-mon[30399]: from='client.40001 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93656-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:28.697 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:28 smithi090 ceph-mon[30399]: from='client.44350 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi090-93342-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:28.697 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:28 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:13:28.697 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:28 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:13:28.784 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:13:28.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:28.809 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:28.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:13:28.910 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:13:28.939 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:28.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:28.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:13:29.041 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:13:29.104 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:29.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:29.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:13:29.208 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:13:29.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:29.233 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:29.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:13:29.334 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:13:29.358 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:29.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:29.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:13:29.459 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:13:29.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:29.484 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:29.484 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:13:29.586 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:13:29.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:29.611 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:29.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:13:29.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:29 smithi092 ceph-mon[33966]: 5.1f scrub starts 2024-04-14T15:13:29.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:29 smithi092 ceph-mon[33966]: 5.1f scrub ok 2024-04-14T15:13:29.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:29 smithi092 ceph-mon[33966]: 5.8 scrub starts 2024-04-14T15:13:29.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:29 smithi092 ceph-mon[33966]: 5.8 scrub ok 2024-04-14T15:13:29.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:29 smithi092 ceph-mon[33966]: 4.7 scrub starts 2024-04-14T15:13:29.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:29 smithi092 ceph-mon[33966]: 4.7 scrub ok 2024-04-14T15:13:29.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:29 smithi092 ceph-mon[33966]: 5.1e scrub starts 2024-04-14T15:13:29.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:29 smithi092 ceph-mon[33966]: 5.1e scrub ok 2024-04-14T15:13:29.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:29 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:29.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:29 smithi092 ceph-mon[33966]: from='client.44341 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi090-93363-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:29.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:29 smithi092 ceph-mon[33966]: from='client.40001 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93656-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:29.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:29 smithi092 ceph-mon[33966]: from='client.44350 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi090-93342-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:29.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:29 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-11", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:13:29.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:29 smithi092 ceph-mon[33966]: osdmap e141: 8 total, 8 up, 8 in 2024-04-14T15:13:29.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:29 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-11"}]: dispatch 2024-04-14T15:13:29.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:29 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-11"}]: dispatch 2024-04-14T15:13:29.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:29 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/903363490' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93656-10", "tierpool":"test-rados-api-smithi090-93656-10-cache"}]: dispatch 2024-04-14T15:13:29.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:29 smithi092 ceph-mon[33966]: from='client.40001 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93656-10", "tierpool":"test-rados-api-smithi090-93656-10-cache"}]: dispatch 2024-04-14T15:13:29.699 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:29 smithi090 ceph-mon[30399]: 5.1f scrub starts 2024-04-14T15:13:29.699 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:29 smithi090 ceph-mon[30399]: 5.1f scrub ok 2024-04-14T15:13:29.699 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:29 smithi090 ceph-mon[30399]: 5.8 scrub starts 2024-04-14T15:13:29.699 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:29 smithi090 ceph-mon[30399]: 5.8 scrub ok 2024-04-14T15:13:29.699 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:29 smithi090 ceph-mon[30399]: 4.7 scrub starts 2024-04-14T15:13:29.699 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:29 smithi090 ceph-mon[30399]: 4.7 scrub ok 2024-04-14T15:13:29.699 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:29 smithi090 ceph-mon[30399]: 5.1e scrub starts 2024-04-14T15:13:29.699 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:29 smithi090 ceph-mon[30399]: 5.1e scrub ok 2024-04-14T15:13:29.700 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:29 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:29.700 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:29 smithi090 ceph-mon[30399]: from='client.44341 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi090-93363-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:29.700 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:29 smithi090 ceph-mon[30399]: from='client.40001 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93656-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:29.700 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:29 smithi090 ceph-mon[30399]: from='client.44350 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi090-93342-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:29.700 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:29 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-11", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:13:29.700 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:29 smithi090 ceph-mon[30399]: osdmap e141: 8 total, 8 up, 8 in 2024-04-14T15:13:29.700 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:29 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-11"}]: dispatch 2024-04-14T15:13:29.700 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:29 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-11"}]: dispatch 2024-04-14T15:13:29.700 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:29 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/903363490' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93656-10", "tierpool":"test-rados-api-smithi090-93656-10-cache"}]: dispatch 2024-04-14T15:13:29.700 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:29 smithi090 ceph-mon[30399]: from='client.40001 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93656-10", "tierpool":"test-rados-api-smithi090-93656-10-cache"}]: dispatch 2024-04-14T15:13:29.701 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:29 smithi090 ceph-mon[41670]: 5.1f scrub starts 2024-04-14T15:13:29.701 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:29 smithi090 ceph-mon[41670]: 5.1f scrub ok 2024-04-14T15:13:29.701 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:29 smithi090 ceph-mon[41670]: 5.8 scrub starts 2024-04-14T15:13:29.701 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:29 smithi090 ceph-mon[41670]: 5.8 scrub ok 2024-04-14T15:13:29.701 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:29 smithi090 ceph-mon[41670]: 4.7 scrub starts 2024-04-14T15:13:29.701 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:29 smithi090 ceph-mon[41670]: 4.7 scrub ok 2024-04-14T15:13:29.701 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:29 smithi090 ceph-mon[41670]: 5.1e scrub starts 2024-04-14T15:13:29.701 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:29 smithi090 ceph-mon[41670]: 5.1e scrub ok 2024-04-14T15:13:29.701 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:29 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:29.701 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:29 smithi090 ceph-mon[41670]: from='client.44341 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi090-93363-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:29.701 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:29 smithi090 ceph-mon[41670]: from='client.40001 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93656-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:29.701 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:29 smithi090 ceph-mon[41670]: from='client.44350 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi090-93342-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:29.701 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:29 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-11", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:13:29.702 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:29 smithi090 ceph-mon[41670]: osdmap e141: 8 total, 8 up, 8 in 2024-04-14T15:13:29.702 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:29 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-11"}]: dispatch 2024-04-14T15:13:29.702 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:29 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-11"}]: dispatch 2024-04-14T15:13:29.702 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:29 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/903363490' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93656-10", "tierpool":"test-rados-api-smithi090-93656-10-cache"}]: dispatch 2024-04-14T15:13:29.702 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:29 smithi090 ceph-mon[41670]: from='client.40001 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93656-10", "tierpool":"test-rados-api-smithi090-93656-10-cache"}]: dispatch 2024-04-14T15:13:29.712 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:13:29.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:29.738 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:29.738 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:13:29.839 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:13:29.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:29.863 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:29.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:13:29.964 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:13:29.988 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:29.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:29.989 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:13:30.090 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:13:30.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:30.215 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:30.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:13:30.316 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:13:30.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:30.342 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:30.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:13:30.443 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:13:30.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:30.467 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:30.467 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:13:30.568 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:13:30.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:30.591 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:30.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:13:30.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:30 smithi092 ceph-mon[33966]: pgmap v158: 588 pgs: 3 active+clean+snaptrim, 103 unknown, 5 creating+activating, 3 creating+peering, 474 active+clean; 145 MiB data, 794 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 259 KiB/s wr, 5 op/s 2024-04-14T15:13:30.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:30 smithi092 ceph-mon[33966]: 5.10 scrub starts 2024-04-14T15:13:30.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:30 smithi092 ceph-mon[33966]: 5.10 scrub ok 2024-04-14T15:13:30.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:30 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:30.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:30 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-11"}]': finished 2024-04-14T15:13:30.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:30 smithi092 ceph-mon[33966]: from='client.40001 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93656-10", "tierpool":"test-rados-api-smithi090-93656-10-cache"}]': finished 2024-04-14T15:13:30.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:30 smithi092 ceph-mon[33966]: osdmap e142: 8 total, 8 up, 8 in 2024-04-14T15:13:30.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:30 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-11", "mode": "writeback"}]: dispatch 2024-04-14T15:13:30.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:30 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-11", "mode": "writeback"}]: dispatch 2024-04-14T15:13:30.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:30 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.0"}]: dispatch 2024-04-14T15:13:30.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:30 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.1"}]: dispatch 2024-04-14T15:13:30.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:30 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.2"}]: dispatch 2024-04-14T15:13:30.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:30 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.3"}]: dispatch 2024-04-14T15:13:30.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:30 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.4"}]: dispatch 2024-04-14T15:13:30.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:30 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.5"}]: dispatch 2024-04-14T15:13:30.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:30 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.6"}]: dispatch 2024-04-14T15:13:30.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:30 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.7"}]: dispatch 2024-04-14T15:13:30.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:30 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.8"}]: dispatch 2024-04-14T15:13:30.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:30 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.9"}]: dispatch 2024-04-14T15:13:30.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:30 smithi090 ceph-mon[30399]: pgmap v158: 588 pgs: 3 active+clean+snaptrim, 103 unknown, 5 creating+activating, 3 creating+peering, 474 active+clean; 145 MiB data, 794 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 259 KiB/s wr, 5 op/s 2024-04-14T15:13:30.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:30 smithi090 ceph-mon[30399]: 5.10 scrub starts 2024-04-14T15:13:30.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:30 smithi090 ceph-mon[30399]: 5.10 scrub ok 2024-04-14T15:13:30.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:30 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:30.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:30 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-11"}]': finished 2024-04-14T15:13:30.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:30 smithi090 ceph-mon[30399]: from='client.40001 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93656-10", "tierpool":"test-rados-api-smithi090-93656-10-cache"}]': finished 2024-04-14T15:13:30.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:30 smithi090 ceph-mon[30399]: osdmap e142: 8 total, 8 up, 8 in 2024-04-14T15:13:30.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:30 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-11", "mode": "writeback"}]: dispatch 2024-04-14T15:13:30.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:30 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-11", "mode": "writeback"}]: dispatch 2024-04-14T15:13:30.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:30 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.0"}]: dispatch 2024-04-14T15:13:30.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:30 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.1"}]: dispatch 2024-04-14T15:13:30.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:30 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.2"}]: dispatch 2024-04-14T15:13:30.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:30 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.3"}]: dispatch 2024-04-14T15:13:30.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:30 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.4"}]: dispatch 2024-04-14T15:13:30.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:30 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.5"}]: dispatch 2024-04-14T15:13:30.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:30 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.6"}]: dispatch 2024-04-14T15:13:30.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:30 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.7"}]: dispatch 2024-04-14T15:13:30.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:30 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.8"}]: dispatch 2024-04-14T15:13:30.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:30 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.9"}]: dispatch 2024-04-14T15:13:30.694 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:30 smithi090 ceph-mon[41670]: pgmap v158: 588 pgs: 3 active+clean+snaptrim, 103 unknown, 5 creating+activating, 3 creating+peering, 474 active+clean; 145 MiB data, 794 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 259 KiB/s wr, 5 op/s 2024-04-14T15:13:30.694 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:30 smithi090 ceph-mon[41670]: 5.10 scrub starts 2024-04-14T15:13:30.694 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:30 smithi090 ceph-mon[41670]: 5.10 scrub ok 2024-04-14T15:13:30.694 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:30 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:30.694 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:30 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-11"}]': finished 2024-04-14T15:13:30.694 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:30 smithi090 ceph-mon[41670]: from='client.40001 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93656-10", "tierpool":"test-rados-api-smithi090-93656-10-cache"}]': finished 2024-04-14T15:13:30.694 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:30 smithi090 ceph-mon[41670]: osdmap e142: 8 total, 8 up, 8 in 2024-04-14T15:13:30.694 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:30 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-11", "mode": "writeback"}]: dispatch 2024-04-14T15:13:30.695 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:30 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-11", "mode": "writeback"}]: dispatch 2024-04-14T15:13:30.695 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:30 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.0"}]: dispatch 2024-04-14T15:13:30.695 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:30 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.1"}]: dispatch 2024-04-14T15:13:30.695 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:30 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.2"}]: dispatch 2024-04-14T15:13:30.695 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:30 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.3"}]: dispatch 2024-04-14T15:13:30.695 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:30 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.4"}]: dispatch 2024-04-14T15:13:30.695 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:30 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.5"}]: dispatch 2024-04-14T15:13:30.695 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:30 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.6"}]: dispatch 2024-04-14T15:13:30.695 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:30 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.7"}]: dispatch 2024-04-14T15:13:30.695 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:30 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.8"}]: dispatch 2024-04-14T15:13:30.695 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:30 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.9"}]: dispatch 2024-04-14T15:13:30.696 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:13:30.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:30.721 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:30.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:13:30.822 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:13:30.847 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc: [==========] Running 12 tests from 4 test suites. 2024-04-14T15:13:30.847 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc: [----------] Global test environment set-up. 2024-04-14T15:13:30.847 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2024-04-14T15:13:30.847 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2024-04-14T15:13:30.847 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2024-04-14T15:13:30.847 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-04-14T15:13:30.847 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc: 2024-04-14T15:13:30.848 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2024-04-14T15:13:30.848 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2024-04-14T15:13:30.848 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc: unable to get monitor info from DNS SRV with service name: ceph-mon 2024-04-14T15:13:30.848 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc: 2024-04-14T15:12:54.294+0000 7f1596645780 -1 failed for service _ceph-mon._tcp 2024-04-14T15:13:30.848 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc: 2024-04-14T15:12:54.294+0000 7f1596645780 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2024-04-14T15:13:30.848 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (65 ms) 2024-04-14T15:13:30.848 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2024-04-14T15:13:30.848 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (2015 ms) 2024-04-14T15:13:30.848 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (2080 ms total) 2024-04-14T15:13:30.848 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc: 2024-04-14T15:13:30.848 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2024-04-14T15:13:30.848 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2024-04-14T15:13:30.848 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc: started 0x55b05a2d4630 2024-04-14T15:13:30.848 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc: started 0x55b05a6400f0 2024-04-14T15:13:30.849 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc: started 2 aios 2024-04-14T15:13:30.849 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc: waiting 0x55b05a2d4630 2024-04-14T15:13:30.849 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc: waiting 0x55b05a6400f0 2024-04-14T15:13:30.849 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc: done. 2024-04-14T15:13:30.849 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (4530 ms) 2024-04-14T15:13:30.849 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (4530 ms total) 2024-04-14T15:13:30.849 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc: 2024-04-14T15:13:30.849 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2024-04-14T15:13:30.849 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2024-04-14T15:13:30.849 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2024-04-14T15:13:30.849 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2024-04-14T15:13:30.849 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc: [ OK ] LibRadosMisc.Exec (156 ms) 2024-04-14T15:13:30.849 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2024-04-14T15:13:30.849 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (19 ms) 2024-04-14T15:13:30.849 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2024-04-14T15:13:30.849 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (3 ms) 2024-04-14T15:13:30.850 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2024-04-14T15:13:30.850 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc: [ OK ] LibRadosMisc.Applications (4825 ms) 2024-04-14T15:13:30.850 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2024-04-14T15:13:30.850 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2024-04-14T15:13:30.850 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2024-04-14T15:13:30.850 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2024-04-14T15:13:30.850 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2024-04-14T15:13:30.850 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (22487 ms) 2024-04-14T15:13:30.850 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc: [----------] 8 tests from LibRadosMisc (27490 ms total) 2024-04-14T15:13:30.850 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc: 2024-04-14T15:13:30.850 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc: [----------] Global test environment tear-down 2024-04-14T15:13:30.850 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (36592 ms total) 2024-04-14T15:13:30.850 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc: [ PASSED ] 12 tests. 2024-04-14T15:13:30.863 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:30.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:30.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:13:30.965 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:13:30.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:30.990 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:30.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:13:31.091 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:13:31.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:31.116 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:31.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:13:31.217 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:13:31.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:31.241 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:31.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:13:31.342 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:13:31.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:31.366 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:31.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:13:31.467 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:13:31.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:31.492 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:31.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:13:31.593 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:13:31.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:31.617 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:31.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:13:31.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:31 smithi092 ceph-mon[33966]: 4.1c scrub starts 2024-04-14T15:13:31.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:31 smithi092 ceph-mon[33966]: 4.1c scrub ok 2024-04-14T15:13:31.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:31 smithi092 ceph-mon[33966]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.0"}]: dispatch 2024-04-14T15:13:31.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:31 smithi092 ceph-mon[33966]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.1"}]: dispatch 2024-04-14T15:13:31.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:31 smithi092 ceph-mon[33966]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.2"}]: dispatch 2024-04-14T15:13:31.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:31 smithi092 ceph-mon[33966]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.3"}]: dispatch 2024-04-14T15:13:31.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:31 smithi092 ceph-mon[33966]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.4"}]: dispatch 2024-04-14T15:13:31.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:31 smithi092 ceph-mon[33966]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.5"}]: dispatch 2024-04-14T15:13:31.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:31 smithi092 ceph-mon[33966]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.6"}]: dispatch 2024-04-14T15:13:31.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:31 smithi092 ceph-mon[33966]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.7"}]: dispatch 2024-04-14T15:13:31.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:31 smithi092 ceph-mon[33966]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.8"}]: dispatch 2024-04-14T15:13:31.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:31 smithi092 ceph-mon[33966]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.9"}]: dispatch 2024-04-14T15:13:31.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:31 smithi092 ceph-mon[33966]: 6.8 scrub starts 2024-04-14T15:13:31.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:31 smithi092 ceph-mon[33966]: 6.8 scrub ok 2024-04-14T15:13:31.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:31 smithi092 ceph-mon[33966]: 16.1 deep-scrub starts 2024-04-14T15:13:31.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:31 smithi092 ceph-mon[33966]: 16.9 deep-scrub starts 2024-04-14T15:13:31.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:31 smithi092 ceph-mon[33966]: 16.9 deep-scrub ok 2024-04-14T15:13:31.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:31 smithi092 ceph-mon[33966]: 16.8 deep-scrub starts 2024-04-14T15:13:31.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:31 smithi092 ceph-mon[33966]: 16.8 deep-scrub ok 2024-04-14T15:13:31.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:31 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:31.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:31 smithi092 ceph-mon[33966]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:13:31.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:31 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-11", "mode": "writeback"}]': finished 2024-04-14T15:13:31.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:31 smithi092 ceph-mon[33966]: osdmap e143: 8 total, 8 up, 8 in 2024-04-14T15:13:31.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:31 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3302263227' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi090-93363-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:31.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:31 smithi092 ceph-mon[33966]: from='client.47569 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi090-93363-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:31.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:31 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3859057618' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi090-93342-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:31.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:31 smithi092 ceph-mon[33966]: from='client.47545 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi090-93342-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:31.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:31 smithi092 ceph-mon[33966]: 4.10 scrub starts 2024-04-14T15:13:31.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:31 smithi092 ceph-mon[33966]: 4.10 scrub ok 2024-04-14T15:13:31.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:31 smithi090 ceph-mon[30399]: 4.1c scrub starts 2024-04-14T15:13:31.702 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:31 smithi090 ceph-mon[30399]: 4.1c scrub ok 2024-04-14T15:13:31.703 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:31 smithi090 ceph-mon[30399]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.0"}]: dispatch 2024-04-14T15:13:31.703 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:31 smithi090 ceph-mon[30399]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.1"}]: dispatch 2024-04-14T15:13:31.703 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:31 smithi090 ceph-mon[30399]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.2"}]: dispatch 2024-04-14T15:13:31.703 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:31 smithi090 ceph-mon[30399]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.3"}]: dispatch 2024-04-14T15:13:31.703 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:31 smithi090 ceph-mon[30399]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.4"}]: dispatch 2024-04-14T15:13:31.703 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:31 smithi090 ceph-mon[30399]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.5"}]: dispatch 2024-04-14T15:13:31.703 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:31 smithi090 ceph-mon[30399]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.6"}]: dispatch 2024-04-14T15:13:31.703 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:31 smithi090 ceph-mon[30399]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.7"}]: dispatch 2024-04-14T15:13:31.703 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:31 smithi090 ceph-mon[30399]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.8"}]: dispatch 2024-04-14T15:13:31.703 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:31 smithi090 ceph-mon[30399]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.9"}]: dispatch 2024-04-14T15:13:31.703 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:31 smithi090 ceph-mon[30399]: 6.8 scrub starts 2024-04-14T15:13:31.703 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:31 smithi090 ceph-mon[30399]: 6.8 scrub ok 2024-04-14T15:13:31.703 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:31 smithi090 ceph-mon[30399]: 16.1 deep-scrub starts 2024-04-14T15:13:31.703 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:31 smithi090 ceph-mon[30399]: 16.9 deep-scrub starts 2024-04-14T15:13:31.704 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:31 smithi090 ceph-mon[30399]: 16.9 deep-scrub ok 2024-04-14T15:13:31.704 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:31 smithi090 ceph-mon[30399]: 16.8 deep-scrub starts 2024-04-14T15:13:31.704 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:31 smithi090 ceph-mon[30399]: 16.8 deep-scrub ok 2024-04-14T15:13:31.704 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:31 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:31.704 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:31 smithi090 ceph-mon[30399]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:13:31.704 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:31 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-11", "mode": "writeback"}]': finished 2024-04-14T15:13:31.704 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:31 smithi090 ceph-mon[30399]: osdmap e143: 8 total, 8 up, 8 in 2024-04-14T15:13:31.704 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:31 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3302263227' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi090-93363-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:31.704 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:31 smithi090 ceph-mon[30399]: from='client.47569 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi090-93363-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:31.704 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:31 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3859057618' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi090-93342-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:31.704 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:31 smithi090 ceph-mon[30399]: from='client.47545 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi090-93342-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:31.704 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:31 smithi090 ceph-mon[30399]: 4.10 scrub starts 2024-04-14T15:13:31.704 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:31 smithi090 ceph-mon[30399]: 4.10 scrub ok 2024-04-14T15:13:31.705 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:31 smithi090 ceph-mon[41670]: 4.1c scrub starts 2024-04-14T15:13:31.705 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:31 smithi090 ceph-mon[41670]: 4.1c scrub ok 2024-04-14T15:13:31.705 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:31 smithi090 ceph-mon[41670]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.0"}]: dispatch 2024-04-14T15:13:31.705 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:31 smithi090 ceph-mon[41670]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.1"}]: dispatch 2024-04-14T15:13:31.705 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:31 smithi090 ceph-mon[41670]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.2"}]: dispatch 2024-04-14T15:13:31.705 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:31 smithi090 ceph-mon[41670]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.3"}]: dispatch 2024-04-14T15:13:31.705 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:31 smithi090 ceph-mon[41670]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.4"}]: dispatch 2024-04-14T15:13:31.705 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:31 smithi090 ceph-mon[41670]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.5"}]: dispatch 2024-04-14T15:13:31.705 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:31 smithi090 ceph-mon[41670]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.6"}]: dispatch 2024-04-14T15:13:31.705 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:31 smithi090 ceph-mon[41670]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.7"}]: dispatch 2024-04-14T15:13:31.705 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:31 smithi090 ceph-mon[41670]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.8"}]: dispatch 2024-04-14T15:13:31.705 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:31 smithi090 ceph-mon[41670]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "16.9"}]: dispatch 2024-04-14T15:13:31.705 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:31 smithi090 ceph-mon[41670]: 6.8 scrub starts 2024-04-14T15:13:31.705 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:31 smithi090 ceph-mon[41670]: 6.8 scrub ok 2024-04-14T15:13:31.706 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:31 smithi090 ceph-mon[41670]: 16.1 deep-scrub starts 2024-04-14T15:13:31.706 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:31 smithi090 ceph-mon[41670]: 16.9 deep-scrub starts 2024-04-14T15:13:31.706 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:31 smithi090 ceph-mon[41670]: 16.9 deep-scrub ok 2024-04-14T15:13:31.706 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:31 smithi090 ceph-mon[41670]: 16.8 deep-scrub starts 2024-04-14T15:13:31.706 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:31 smithi090 ceph-mon[41670]: 16.8 deep-scrub ok 2024-04-14T15:13:31.706 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:31 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:31.706 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:31 smithi090 ceph-mon[41670]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:13:31.706 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:31 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-11", "mode": "writeback"}]': finished 2024-04-14T15:13:31.706 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:31 smithi090 ceph-mon[41670]: osdmap e143: 8 total, 8 up, 8 in 2024-04-14T15:13:31.706 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:31 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3302263227' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi090-93363-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:31.706 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:31 smithi090 ceph-mon[41670]: from='client.47569 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi090-93363-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:31.706 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:31 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3859057618' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi090-93342-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:31.706 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:31 smithi090 ceph-mon[41670]: from='client.47545 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi090-93342-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:31.707 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:31 smithi090 ceph-mon[41670]: 4.10 scrub starts 2024-04-14T15:13:31.707 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:31 smithi090 ceph-mon[41670]: 4.10 scrub ok 2024-04-14T15:13:31.718 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:13:31.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:31.745 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:31.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:13:31.836 INFO:tasks.workunit.client.0.smithi090.stdout: api_pool: Running main() from gmock_main.cc 2024-04-14T15:13:31.836 INFO:tasks.workunit.client.0.smithi090.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2024-04-14T15:13:31.836 INFO:tasks.workunit.client.0.smithi090.stdout: api_pool: [----------] Global test environment set-up. 2024-04-14T15:13:31.836 INFO:tasks.workunit.client.0.smithi090.stdout: api_pool: [----------] 9 tests from LibRadosPools 2024-04-14T15:13:31.836 INFO:tasks.workunit.client.0.smithi090.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2024-04-14T15:13:31.836 INFO:tasks.workunit.client.0.smithi090.stdout: api_pool: [ OK ] LibRadosPools.PoolList (2876 ms) 2024-04-14T15:13:31.836 INFO:tasks.workunit.client.0.smithi090.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2024-04-14T15:13:31.836 INFO:tasks.workunit.client.0.smithi090.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (2997 ms) 2024-04-14T15:13:31.836 INFO:tasks.workunit.client.0.smithi090.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2024-04-14T15:13:31.836 INFO:tasks.workunit.client.0.smithi090.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (2705 ms) 2024-04-14T15:13:31.836 INFO:tasks.workunit.client.0.smithi090.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2024-04-14T15:13:31.836 INFO:tasks.workunit.client.0.smithi090.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (3012 ms) 2024-04-14T15:13:31.836 INFO:tasks.workunit.client.0.smithi090.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2024-04-14T15:13:31.837 INFO:tasks.workunit.client.0.smithi090.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (3015 ms) 2024-04-14T15:13:31.837 INFO:tasks.workunit.client.0.smithi090.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2024-04-14T15:13:31.837 INFO:tasks.workunit.client.0.smithi090.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (5005 ms) 2024-04-14T15:13:31.837 INFO:tasks.workunit.client.0.smithi090.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2024-04-14T15:13:31.837 INFO:tasks.workunit.client.0.smithi090.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (4991 ms) 2024-04-14T15:13:31.837 INFO:tasks.workunit.client.0.smithi090.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2024-04-14T15:13:31.837 INFO:tasks.workunit.client.0.smithi090.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (5009 ms) 2024-04-14T15:13:31.837 INFO:tasks.workunit.client.0.smithi090.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2024-04-14T15:13:31.837 INFO:tasks.workunit.client.0.smithi090.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (7983 ms) 2024-04-14T15:13:31.837 INFO:tasks.workunit.client.0.smithi090.stdout: api_pool: [----------] 9 tests from LibRadosPools (37593 ms total) 2024-04-14T15:13:31.837 INFO:tasks.workunit.client.0.smithi090.stdout: api_pool: 2024-04-14T15:13:31.837 INFO:tasks.workunit.client.0.smithi090.stdout: api_pool: [----------] Global test environment tear-down 2024-04-14T15:13:31.837 INFO:tasks.workunit.client.0.smithi090.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (37593 ms total) 2024-04-14T15:13:31.837 INFO:tasks.workunit.client.0.smithi090.stdout: api_pool: [ PASSED ] 9 tests. 2024-04-14T15:13:31.846 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:13:31.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:31.893 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:31.893 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:13:31.994 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:13:32.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:32.018 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:32.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:13:32.119 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:13:32.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:32.143 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:32.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:13:32.245 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:13:32.270 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:32.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:32.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:13:32.372 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:13:32.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:32.396 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:32.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:13:32.497 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:13:32.521 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:32.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:32.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:13:32.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:32 smithi092 ceph-mon[33966]: 16.1 deep-scrub ok 2024-04-14T15:13:32.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:32 smithi092 ceph-mon[33966]: 16.2 deep-scrub starts 2024-04-14T15:13:32.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:32 smithi092 ceph-mon[33966]: 16.2 deep-scrub ok 2024-04-14T15:13:32.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:32 smithi092 ceph-mon[33966]: 16.3 deep-scrub starts 2024-04-14T15:13:32.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:32 smithi092 ceph-mon[33966]: 16.3 deep-scrub ok 2024-04-14T15:13:32.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:32 smithi092 ceph-mon[33966]: pgmap v161: 492 pgs: 128 unknown, 1 active+clean+snaptrim, 363 active+clean; 145 MiB data, 824 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 257 KiB/s wr, 2 op/s 2024-04-14T15:13:32.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:32 smithi092 ceph-mon[33966]: 16.6 deep-scrub starts 2024-04-14T15:13:32.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:32 smithi092 ceph-mon[33966]: 16.6 deep-scrub ok 2024-04-14T15:13:32.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:32 smithi092 ceph-mon[33966]: 5.4 scrub starts 2024-04-14T15:13:32.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:32 smithi092 ceph-mon[33966]: 5.4 scrub ok 2024-04-14T15:13:32.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:32 smithi092 ceph-mon[33966]: 4.19 scrub starts 2024-04-14T15:13:32.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:32 smithi092 ceph-mon[33966]: 4.19 scrub ok 2024-04-14T15:13:32.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:32 smithi092 ceph-mon[33966]: 16.4 deep-scrub starts 2024-04-14T15:13:32.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:32 smithi092 ceph-mon[33966]: 16.4 deep-scrub ok 2024-04-14T15:13:32.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:32 smithi092 ceph-mon[33966]: 16.5 deep-scrub starts 2024-04-14T15:13:32.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:32 smithi092 ceph-mon[33966]: 16.5 deep-scrub ok 2024-04-14T15:13:32.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:32 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:32.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:32 smithi092 ceph-mon[33966]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:13:32.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:32 smithi092 ceph-mon[33966]: from='client.47569 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi090-93363-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:32.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:32 smithi092 ceph-mon[33966]: from='client.47545 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi090-93342-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:32.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:32 smithi092 ceph-mon[33966]: osdmap e144: 8 total, 8 up, 8 in 2024-04-14T15:13:32.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:32 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:13:32.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:32 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:13:32.625 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:13:32.629 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:32 smithi090 ceph-mon[41670]: 16.1 deep-scrub ok 2024-04-14T15:13:32.629 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:32 smithi090 ceph-mon[41670]: 16.2 deep-scrub starts 2024-04-14T15:13:32.629 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:32 smithi090 ceph-mon[41670]: 16.2 deep-scrub ok 2024-04-14T15:13:32.630 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:32 smithi090 ceph-mon[41670]: 16.3 deep-scrub starts 2024-04-14T15:13:32.630 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:32 smithi090 ceph-mon[41670]: 16.3 deep-scrub ok 2024-04-14T15:13:32.630 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:32 smithi090 ceph-mon[41670]: pgmap v161: 492 pgs: 128 unknown, 1 active+clean+snaptrim, 363 active+clean; 145 MiB data, 824 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 257 KiB/s wr, 2 op/s 2024-04-14T15:13:32.630 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:32 smithi090 ceph-mon[41670]: 16.6 deep-scrub starts 2024-04-14T15:13:32.630 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:32 smithi090 ceph-mon[41670]: 16.6 deep-scrub ok 2024-04-14T15:13:32.630 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:32 smithi090 ceph-mon[41670]: 5.4 scrub starts 2024-04-14T15:13:32.630 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:32 smithi090 ceph-mon[41670]: 5.4 scrub ok 2024-04-14T15:13:32.630 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:32 smithi090 ceph-mon[41670]: 4.19 scrub starts 2024-04-14T15:13:32.630 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:32 smithi090 ceph-mon[41670]: 4.19 scrub ok 2024-04-14T15:13:32.630 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:32 smithi090 ceph-mon[41670]: 16.4 deep-scrub starts 2024-04-14T15:13:32.630 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:32 smithi090 ceph-mon[41670]: 16.4 deep-scrub ok 2024-04-14T15:13:32.630 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:32 smithi090 ceph-mon[41670]: 16.5 deep-scrub starts 2024-04-14T15:13:32.630 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:32 smithi090 ceph-mon[41670]: 16.5 deep-scrub ok 2024-04-14T15:13:32.630 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:32 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:32.631 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:32 smithi090 ceph-mon[41670]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:13:32.631 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:32 smithi090 ceph-mon[41670]: from='client.47569 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi090-93363-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:32.631 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:32 smithi090 ceph-mon[41670]: from='client.47545 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi090-93342-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:32.631 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:32 smithi090 ceph-mon[41670]: osdmap e144: 8 total, 8 up, 8 in 2024-04-14T15:13:32.631 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:32 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:13:32.631 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:32 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:13:32.631 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:32 smithi090 ceph-mon[30399]: 16.1 deep-scrub ok 2024-04-14T15:13:32.631 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:32 smithi090 ceph-mon[30399]: 16.2 deep-scrub starts 2024-04-14T15:13:32.631 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:32 smithi090 ceph-mon[30399]: 16.2 deep-scrub ok 2024-04-14T15:13:32.631 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:32 smithi090 ceph-mon[30399]: 16.3 deep-scrub starts 2024-04-14T15:13:32.631 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:32 smithi090 ceph-mon[30399]: 16.3 deep-scrub ok 2024-04-14T15:13:32.632 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:32 smithi090 ceph-mon[30399]: pgmap v161: 492 pgs: 128 unknown, 1 active+clean+snaptrim, 363 active+clean; 145 MiB data, 824 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 257 KiB/s wr, 2 op/s 2024-04-14T15:13:32.632 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:32 smithi090 ceph-mon[30399]: 16.6 deep-scrub starts 2024-04-14T15:13:32.632 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:32 smithi090 ceph-mon[30399]: 16.6 deep-scrub ok 2024-04-14T15:13:32.632 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:32 smithi090 ceph-mon[30399]: 5.4 scrub starts 2024-04-14T15:13:32.632 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:32 smithi090 ceph-mon[30399]: 5.4 scrub ok 2024-04-14T15:13:32.632 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:32 smithi090 ceph-mon[30399]: 4.19 scrub starts 2024-04-14T15:13:32.632 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:32 smithi090 ceph-mon[30399]: 4.19 scrub ok 2024-04-14T15:13:32.632 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:32 smithi090 ceph-mon[30399]: 16.4 deep-scrub starts 2024-04-14T15:13:32.632 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:32 smithi090 ceph-mon[30399]: 16.4 deep-scrub ok 2024-04-14T15:13:32.633 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:32 smithi090 ceph-mon[30399]: 16.5 deep-scrub starts 2024-04-14T15:13:32.633 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:32 smithi090 ceph-mon[30399]: 16.5 deep-scrub ok 2024-04-14T15:13:32.633 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:32 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:32.633 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:32 smithi090 ceph-mon[30399]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:13:32.633 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:32 smithi090 ceph-mon[30399]: from='client.47569 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi090-93363-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:32.633 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:32 smithi090 ceph-mon[30399]: from='client.47545 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi090-93342-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:32.633 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:32 smithi090 ceph-mon[30399]: osdmap e144: 8 total, 8 up, 8 in 2024-04-14T15:13:32.633 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:32 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:13:32.633 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:32 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:13:32.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:32.649 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:32.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:13:32.750 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:13:32.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:32.776 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:32.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:13:32.877 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:13:32.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:32.902 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:32.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:13:32.970 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd set noscrub 2024-04-14T15:13:33.003 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:13:33.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:33.027 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:33.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:13:33.128 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:13:33.158 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:33.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:33.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:13:33.260 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:13:33.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:33.285 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:33.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:13:33.386 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:13:33.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:33.410 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:33.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:13:33.511 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:13:33.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:33.535 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:33.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:13:33.620 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:33 smithi090 ceph-mon[41670]: 16.0 deep-scrub starts 2024-04-14T15:13:33.621 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:33 smithi090 ceph-mon[41670]: 16.0 deep-scrub ok 2024-04-14T15:13:33.621 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:33 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:33.621 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:33 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:13:33.621 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:33 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-11"}]: dispatch 2024-04-14T15:13:33.621 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:33 smithi090 ceph-mon[41670]: osdmap e145: 8 total, 8 up, 8 in 2024-04-14T15:13:33.621 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:33 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-11"}]: dispatch 2024-04-14T15:13:33.621 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:33 smithi090 ceph-mon[30399]: 16.0 deep-scrub starts 2024-04-14T15:13:33.621 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:33 smithi090 ceph-mon[30399]: 16.0 deep-scrub ok 2024-04-14T15:13:33.621 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:33 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:33.621 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:33 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:13:33.621 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:33 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-11"}]: dispatch 2024-04-14T15:13:33.622 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:33 smithi090 ceph-mon[30399]: osdmap e145: 8 total, 8 up, 8 in 2024-04-14T15:13:33.622 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:33 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-11"}]: dispatch 2024-04-14T15:13:33.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:33 smithi092 ceph-mon[33966]: 16.0 deep-scrub starts 2024-04-14T15:13:33.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:33 smithi092 ceph-mon[33966]: 16.0 deep-scrub ok 2024-04-14T15:13:33.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:33 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:33.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:33 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:13:33.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:33 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-11"}]: dispatch 2024-04-14T15:13:33.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:33 smithi092 ceph-mon[33966]: osdmap e145: 8 total, 8 up, 8 in 2024-04-14T15:13:33.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:33 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-11"}]: dispatch 2024-04-14T15:13:33.637 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:13:33.660 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:33.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:33.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:13:33.762 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:13:33.787 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:33.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:33.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:13:33.891 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:13:33.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:33.916 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:33.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:13:33.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.879+0000 7f699191c700 1 -- 172.21.15.90:0/1698670684 >> v1:172.21.15.92:6789/0 conn(0x7f698c103a40 legacy=0x7f698c103e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:33.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.879+0000 7f699191c700 1 -- 172.21.15.90:0/1698670684 shutdown_connections 2024-04-14T15:13:33.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.879+0000 7f699191c700 1 -- 172.21.15.90:0/1698670684 >> 172.21.15.90:0/1698670684 conn(0x7f698c0fd290 msgr2=0x7f698c0ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:13:33.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.879+0000 7f699191c700 1 -- 172.21.15.90:0/1698670684 shutdown_connections 2024-04-14T15:13:33.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.879+0000 7f699191c700 1 -- 172.21.15.90:0/1698670684 wait complete. 2024-04-14T15:13:33.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.879+0000 7f699191c700 1 librados: starting msgr at 2024-04-14T15:13:33.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.879+0000 7f699191c700 1 librados: starting objecter 2024-04-14T15:13:33.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.880+0000 7f699191c700 1 Processor -- start 2024-04-14T15:13:33.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.880+0000 7f699191c700 1 -- start start 2024-04-14T15:13:33.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.880+0000 7f699191c700 1 librados: setting wanted keys 2024-04-14T15:13:33.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.880+0000 7f699191c700 1 librados: calling monclient init 2024-04-14T15:13:33.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.880+0000 7f699191c700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f698c10b5e0 con 0x7f698c103a40 2024-04-14T15:13:33.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.880+0000 7f699191c700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f698c1a9070 con 0x7f698c10b930 2024-04-14T15:13:33.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.880+0000 7f699191c700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f698c1aa250 con 0x7f698c10f4d0 2024-04-14T15:13:33.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.880+0000 7f698affd700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f698c103a40 0x7f698c108430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:34338/0 (socket says 172.21.15.90:34338) 2024-04-14T15:13:33.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.880+0000 7f698affd700 1 -- 172.21.15.90:0/2095658072 learned_addr learned my addr 172.21.15.90:0/2095658072 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:13:33.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.880+0000 7f699091a700 10 client.?.objecter ms_handle_connect 0x7f698c103a40 2024-04-14T15:13:33.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.880+0000 7f699091a700 10 client.?.objecter resend_mon_ops 2024-04-14T15:13:33.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.880+0000 7f699091a700 10 client.?.objecter ms_handle_connect 0x7f698c10f4d0 2024-04-14T15:13:33.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.880+0000 7f699091a700 10 client.?.objecter resend_mon_ops 2024-04-14T15:13:33.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.880+0000 7f699091a700 10 client.?.objecter ms_handle_connect 0x7f698c10b930 2024-04-14T15:13:33.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.880+0000 7f699091a700 10 client.?.objecter resend_mon_ops 2024-04-14T15:13:33.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.880+0000 7f699091a700 1 -- 172.21.15.90:0/2095658072 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1773791024 0 0) 0x7f698c10b5e0 con 0x7f698c103a40 2024-04-14T15:13:33.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.880+0000 7f699091a700 1 -- 172.21.15.90:0/2095658072 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f69740031f0 con 0x7f698c103a40 2024-04-14T15:13:33.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.880+0000 7f699091a700 1 -- 172.21.15.90:0/2095658072 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1117906704 0 0) 0x7f698c1aa250 con 0x7f698c10f4d0 2024-04-14T15:13:33.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.881+0000 7f699091a700 1 -- 172.21.15.90:0/2095658072 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f698c10b5e0 con 0x7f698c10f4d0 2024-04-14T15:13:33.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.881+0000 7f699091a700 1 -- 172.21.15.90:0/2095658072 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1961632070 0 0) 0x7f698c1a9070 con 0x7f698c10b930 2024-04-14T15:13:33.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.881+0000 7f699091a700 1 -- 172.21.15.90:0/2095658072 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f698c1aa250 con 0x7f698c10b930 2024-04-14T15:13:33.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.881+0000 7f699091a700 1 -- 172.21.15.90:0/2095658072 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1538542523 0 0) 0x7f69740031f0 con 0x7f698c103a40 2024-04-14T15:13:33.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.881+0000 7f699091a700 1 -- 172.21.15.90:0/2095658072 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f698c1a9070 con 0x7f698c103a40 2024-04-14T15:13:33.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.881+0000 7f699091a700 1 -- 172.21.15.90:0/2095658072 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f697c0015d0 con 0x7f698c103a40 2024-04-14T15:13:33.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.881+0000 7f699091a700 1 -- 172.21.15.90:0/2095658072 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4206779484 0 0) 0x7f698c1aa250 con 0x7f698c10b930 2024-04-14T15:13:33.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.881+0000 7f699091a700 1 -- 172.21.15.90:0/2095658072 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f69740031f0 con 0x7f698c10b930 2024-04-14T15:13:33.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.881+0000 7f699091a700 1 -- 172.21.15.90:0/2095658072 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f6980001830 con 0x7f698c10b930 2024-04-14T15:13:33.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.881+0000 7f699091a700 1 -- 172.21.15.90:0/2095658072 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3734896191 0 0) 0x7f698c10b5e0 con 0x7f698c10f4d0 2024-04-14T15:13:33.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.881+0000 7f699091a700 1 -- 172.21.15.90:0/2095658072 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f698c1aa250 con 0x7f698c10f4d0 2024-04-14T15:13:33.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.881+0000 7f699091a700 1 -- 172.21.15.90:0/2095658072 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f6984001b30 con 0x7f698c10f4d0 2024-04-14T15:13:33.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.881+0000 7f699091a700 1 -- 172.21.15.90:0/2095658072 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4076766524 0 0) 0x7f698c1a9070 con 0x7f698c103a40 2024-04-14T15:13:33.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.881+0000 7f699091a700 1 -- 172.21.15.90:0/2095658072 >> v1:172.21.15.90:6790/0 conn(0x7f698c10f4d0 legacy=0x7f698c1a6940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:33.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.881+0000 7f699091a700 1 -- 172.21.15.90:0/2095658072 >> v1:172.21.15.92:6789/0 conn(0x7f698c10b930 legacy=0x7f698c108b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:33.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.881+0000 7f699091a700 1 -- 172.21.15.90:0/2095658072 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f698c1ab430 con 0x7f698c103a40 2024-04-14T15:13:33.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.881+0000 7f699191c700 1 -- 172.21.15.90:0/2095658072 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f698c1aa480 con 0x7f698c103a40 2024-04-14T15:13:33.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.881+0000 7f699191c700 10 client.36258.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:13:33.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.881+0000 7f699191c700 1 -- 172.21.15.90:0/2095658072 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f698c1aa940 con 0x7f698c103a40 2024-04-14T15:13:33.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.881+0000 7f699191c700 1 librados: init done 2024-04-14T15:13:33.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.881+0000 7f699091a700 1 -- 172.21.15.90:0/2095658072 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f697c001c50 con 0x7f698c103a40 2024-04-14T15:13:33.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.881+0000 7f699091a700 1 -- 172.21.15.90:0/2095658072 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f697c005100 con 0x7f698c103a40 2024-04-14T15:13:33.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.882+0000 7f699191c700 1 -- 172.21.15.90:0/2095658072 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f698c06b5d0 con 0x7f698c103a40 2024-04-14T15:13:33.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.883+0000 7f699091a700 1 -- 172.21.15.90:0/2095658072 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f697c0050e0 con 0x7f698c103a40 2024-04-14T15:13:33.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.883+0000 7f699091a700 1 -- 172.21.15.90:0/2095658072 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(146..146 src has 51..146) ==== 11327+0+0 (unknown 1694684601 0 0) 0x7f697c08b7b0 con 0x7f698c103a40 2024-04-14T15:13:33.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.883+0000 7f699091a700 10 client.36258.objecter ms_dispatch 0x7f698c001550 osd_map(146..146 src has 51..146) 2024-04-14T15:13:33.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.883+0000 7f699091a700 3 client.36258.objecter handle_osd_map got epochs [146,146] > 0 2024-04-14T15:13:33.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.883+0000 7f699091a700 3 client.36258.objecter handle_osd_map decoding full epoch 146 2024-04-14T15:13:33.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.883+0000 7f699091a700 20 client.36258.objecter dump_active .. 0 homeless 2024-04-14T15:13:33.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.883+0000 7f699091a700 1 -- 172.21.15.90:0/2095658072 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v146) ==== 54+0+16 (unknown 2793121449 0 2420000192) 0x7f697c08bbf0 con 0x7f698c103a40 2024-04-14T15:13:33.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.884+0000 7f699091a700 10 client.36258.objecter ms_handle_connect 0x7f69740712d0 2024-04-14T15:13:33.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.884+0000 7f699191c700 10 client.36258.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:13:33.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.884+0000 7f699191c700 15 client.36258.objecter _session_command_op_assign -1 1 2024-04-14T15:13:33.971 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:13:33.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.884+0000 7f699191c700 20 client.36258.objecter _get_session s=0x7f698c113490 osd=0 3 2024-04-14T15:13:33.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.884+0000 7f699191c700 20 client.36258.objecter put_session s=0x7f698c113490 osd=0 3 2024-04-14T15:13:33.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.884+0000 7f699191c700 20 client.36258.objecter _get_session s=0x7f698c113490 osd=0 3 2024-04-14T15:13:33.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.884+0000 7f699191c700 15 client.36258.objecter _session_command_op_remove -1 1 2024-04-14T15:13:33.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.884+0000 7f699191c700 20 client.36258.objecter get_session s=0x7f698c113490 osd=0 3 2024-04-14T15:13:33.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.884+0000 7f699191c700 15 client.36258.objecter _session_command_op_assign 0 1 2024-04-14T15:13:33.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.884+0000 7f699191c700 20 client.36258.objecter put_session s=0x7f698c113490 osd=0 4 2024-04-14T15:13:33.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.884+0000 7f699191c700 10 client.36258.objecter _send_command 1 2024-04-14T15:13:33.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.884+0000 7f699191c700 1 -- 172.21.15.90:0/2095658072 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f698c1035e0 con 0x7f698c1a9750 2024-04-14T15:13:33.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.884+0000 7f699091a700 10 client.36258.objecter ms_handle_connect 0x7f698c1a9750 2024-04-14T15:13:33.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.885+0000 7f699091a700 1 -- 172.21.15.90:0/2095658072 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f698c1035e0 con 0x7f698c1a9750 2024-04-14T15:13:33.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.885+0000 7f699091a700 10 client.36258.objecter ms_dispatch 0x7f698c001550 command_reply(tid 1: 0 ) 2024-04-14T15:13:33.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.885+0000 7f699091a700 10 client.36258.objecter _finish_command 1 = system:0 2024-04-14T15:13:33.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.885+0000 7f699091a700 20 client.36258.objecter put_session s=0x7f698c113490 osd=0 4 2024-04-14T15:13:33.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.885+0000 7f699091a700 15 client.36258.objecter _session_command_op_remove 0 1 2024-04-14T15:13:33.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.902+0000 7f699191c700 10 client.36258.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:13:33.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.902+0000 7f699191c700 15 client.36258.objecter _session_command_op_assign -1 2 2024-04-14T15:13:33.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.902+0000 7f699191c700 20 client.36258.objecter _get_session s=0x7f698c113490 osd=0 3 2024-04-14T15:13:33.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.902+0000 7f699191c700 20 client.36258.objecter put_session s=0x7f698c113490 osd=0 3 2024-04-14T15:13:33.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.902+0000 7f699191c700 20 client.36258.objecter _get_session s=0x7f698c113490 osd=0 3 2024-04-14T15:13:33.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.902+0000 7f699191c700 15 client.36258.objecter _session_command_op_remove -1 2 2024-04-14T15:13:33.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.902+0000 7f699191c700 20 client.36258.objecter get_session s=0x7f698c113490 osd=0 3 2024-04-14T15:13:33.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.902+0000 7f699191c700 15 client.36258.objecter _session_command_op_assign 0 2 2024-04-14T15:13:33.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.902+0000 7f699191c700 20 client.36258.objecter put_session s=0x7f698c113490 osd=0 4 2024-04-14T15:13:33.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.902+0000 7f699191c700 10 client.36258.objecter _send_command 2 2024-04-14T15:13:33.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.902+0000 7f699191c700 1 -- 172.21.15.90:0/2095658072 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f698c0693d0 con 0x7f698c1a9750 2024-04-14T15:13:33.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.904+0000 7f699091a700 1 -- 172.21.15.90:0/2095658072 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f698c0693d0 con 0x7f698c1a9750 2024-04-14T15:13:33.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.904+0000 7f699091a700 10 client.36258.objecter ms_dispatch 0x7f698c001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:13:33.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.904+0000 7f699091a700 10 client.36258.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:13:33.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.904+0000 7f699091a700 20 client.36258.objecter put_session s=0x7f698c113490 osd=0 4 2024-04-14T15:13:33.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.904+0000 7f699091a700 15 client.36258.objecter _session_command_op_remove 0 2 2024-04-14T15:13:33.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.904+0000 7f699191c700 10 client.36258.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:13:33.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.904+0000 7f699191c700 15 client.36258.objecter _session_command_op_assign -1 3 2024-04-14T15:13:33.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.904+0000 7f699191c700 20 client.36258.objecter _get_session s=0x7f698c109960 osd=1 3 2024-04-14T15:13:33.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.904+0000 7f699191c700 20 client.36258.objecter put_session s=0x7f698c109960 osd=1 3 2024-04-14T15:13:33.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.904+0000 7f699191c700 20 client.36258.objecter _get_session s=0x7f698c109960 osd=1 3 2024-04-14T15:13:33.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.904+0000 7f699191c700 15 client.36258.objecter _session_command_op_remove -1 3 2024-04-14T15:13:33.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.904+0000 7f699191c700 20 client.36258.objecter get_session s=0x7f698c109960 osd=1 3 2024-04-14T15:13:33.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.904+0000 7f699191c700 15 client.36258.objecter _session_command_op_assign 1 3 2024-04-14T15:13:33.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.904+0000 7f699191c700 20 client.36258.objecter put_session s=0x7f698c109960 osd=1 4 2024-04-14T15:13:33.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.904+0000 7f699191c700 10 client.36258.objecter _send_command 3 2024-04-14T15:13:33.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.904+0000 7f699191c700 1 -- 172.21.15.90:0/2095658072 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f698c10d430 con 0x7f698c0636c0 2024-04-14T15:13:33.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.904+0000 7f699091a700 10 client.36258.objecter ms_handle_connect 0x7f698c0636c0 2024-04-14T15:13:33.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.905+0000 7f699091a700 1 -- 172.21.15.90:0/2095658072 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f698c10d430 con 0x7f698c0636c0 2024-04-14T15:13:33.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.905+0000 7f699091a700 10 client.36258.objecter ms_dispatch 0x7f698c001550 command_reply(tid 3: 0 ) 2024-04-14T15:13:33.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.905+0000 7f699091a700 10 client.36258.objecter _finish_command 3 = system:0 2024-04-14T15:13:33.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.905+0000 7f699091a700 20 client.36258.objecter put_session s=0x7f698c109960 osd=1 4 2024-04-14T15:13:33.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.905+0000 7f699091a700 15 client.36258.objecter _session_command_op_remove 1 3 2024-04-14T15:13:33.976 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:13:33.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.922+0000 7f699191c700 10 client.36258.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:13:33.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.922+0000 7f699191c700 15 client.36258.objecter _session_command_op_assign -1 4 2024-04-14T15:13:33.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.922+0000 7f699191c700 20 client.36258.objecter _get_session s=0x7f698c109960 osd=1 3 2024-04-14T15:13:33.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.922+0000 7f699191c700 20 client.36258.objecter put_session s=0x7f698c109960 osd=1 3 2024-04-14T15:13:33.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.922+0000 7f699191c700 20 client.36258.objecter _get_session s=0x7f698c109960 osd=1 3 2024-04-14T15:13:33.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.922+0000 7f699191c700 15 client.36258.objecter _session_command_op_remove -1 4 2024-04-14T15:13:33.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.922+0000 7f699191c700 20 client.36258.objecter get_session s=0x7f698c109960 osd=1 3 2024-04-14T15:13:33.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.922+0000 7f699191c700 15 client.36258.objecter _session_command_op_assign 1 4 2024-04-14T15:13:33.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.922+0000 7f699191c700 20 client.36258.objecter put_session s=0x7f698c109960 osd=1 4 2024-04-14T15:13:33.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.922+0000 7f699191c700 10 client.36258.objecter _send_command 4 2024-04-14T15:13:33.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.922+0000 7f699191c700 1 -- 172.21.15.90:0/2095658072 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f698c063b00 con 0x7f698c0636c0 2024-04-14T15:13:33.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.924+0000 7f699091a700 1 -- 172.21.15.90:0/2095658072 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f698c063b00 con 0x7f698c0636c0 2024-04-14T15:13:33.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.924+0000 7f699091a700 10 client.36258.objecter ms_dispatch 0x7f698c001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:13:33.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.924+0000 7f699091a700 10 client.36258.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:13:33.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.924+0000 7f699091a700 20 client.36258.objecter put_session s=0x7f698c109960 osd=1 4 2024-04-14T15:13:33.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.924+0000 7f699091a700 15 client.36258.objecter _session_command_op_remove 1 4 2024-04-14T15:13:33.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.924+0000 7f699191c700 10 client.36258.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:13:33.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.924+0000 7f699191c700 15 client.36258.objecter _session_command_op_assign -1 5 2024-04-14T15:13:33.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.924+0000 7f699191c700 20 client.36258.objecter _get_session s=0x7f698c063f00 osd=2 3 2024-04-14T15:13:33.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.924+0000 7f699191c700 20 client.36258.objecter put_session s=0x7f698c063f00 osd=2 3 2024-04-14T15:13:33.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.924+0000 7f699191c700 20 client.36258.objecter _get_session s=0x7f698c063f00 osd=2 3 2024-04-14T15:13:33.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.924+0000 7f699191c700 15 client.36258.objecter _session_command_op_remove -1 5 2024-04-14T15:13:33.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.924+0000 7f699191c700 20 client.36258.objecter get_session s=0x7f698c063f00 osd=2 3 2024-04-14T15:13:33.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.924+0000 7f699191c700 15 client.36258.objecter _session_command_op_assign 2 5 2024-04-14T15:13:33.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.924+0000 7f699191c700 20 client.36258.objecter put_session s=0x7f698c063f00 osd=2 4 2024-04-14T15:13:33.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.924+0000 7f699191c700 10 client.36258.objecter _send_command 5 2024-04-14T15:13:33.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.924+0000 7f699191c700 1 -- 172.21.15.90:0/2095658072 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f698c1b1690 con 0x7f698c1adb80 2024-04-14T15:13:33.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.924+0000 7f699091a700 10 client.36258.objecter ms_handle_connect 0x7f698c1adb80 2024-04-14T15:13:33.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.925+0000 7f699091a700 1 -- 172.21.15.90:0/2095658072 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f698c1b1690 con 0x7f698c1adb80 2024-04-14T15:13:33.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.925+0000 7f699091a700 10 client.36258.objecter ms_dispatch 0x7f698c001550 command_reply(tid 5: 0 ) 2024-04-14T15:13:33.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.925+0000 7f699091a700 10 client.36258.objecter _finish_command 5 = system:0 2024-04-14T15:13:33.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.925+0000 7f699091a700 20 client.36258.objecter put_session s=0x7f698c063f00 osd=2 4 2024-04-14T15:13:33.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.925+0000 7f699091a700 15 client.36258.objecter _session_command_op_remove 2 5 2024-04-14T15:13:33.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.952+0000 7f699191c700 10 client.36258.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:13:33.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.952+0000 7f699191c700 15 client.36258.objecter _session_command_op_assign -1 6 2024-04-14T15:13:33.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.952+0000 7f699191c700 20 client.36258.objecter _get_session s=0x7f698c063f00 osd=2 3 2024-04-14T15:13:33.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.952+0000 7f699191c700 20 client.36258.objecter put_session s=0x7f698c063f00 osd=2 3 2024-04-14T15:13:33.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.952+0000 7f699191c700 20 client.36258.objecter _get_session s=0x7f698c063f00 osd=2 3 2024-04-14T15:13:33.980 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:13:33.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.952+0000 7f699191c700 15 client.36258.objecter _session_command_op_remove -1 6 2024-04-14T15:13:33.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.952+0000 7f699191c700 20 client.36258.objecter get_session s=0x7f698c063f00 osd=2 3 2024-04-14T15:13:33.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.952+0000 7f699191c700 15 client.36258.objecter _session_command_op_assign 2 6 2024-04-14T15:13:33.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.952+0000 7f699191c700 20 client.36258.objecter put_session s=0x7f698c063f00 osd=2 4 2024-04-14T15:13:33.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.952+0000 7f699191c700 10 client.36258.objecter _send_command 6 2024-04-14T15:13:33.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.952+0000 7f699191c700 1 -- 172.21.15.90:0/2095658072 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f698c063140 con 0x7f698c1adb80 2024-04-14T15:13:33.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.954+0000 7f699091a700 1 -- 172.21.15.90:0/2095658072 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f698c063140 con 0x7f698c1adb80 2024-04-14T15:13:33.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.954+0000 7f699091a700 10 client.36258.objecter ms_dispatch 0x7f698c001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:13:33.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.954+0000 7f699091a700 10 client.36258.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:13:33.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.954+0000 7f699091a700 20 client.36258.objecter put_session s=0x7f698c063f00 osd=2 4 2024-04-14T15:13:33.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.954+0000 7f699091a700 15 client.36258.objecter _session_command_op_remove 2 6 2024-04-14T15:13:33.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.954+0000 7f699191c700 10 client.36258.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:13:33.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.954+0000 7f699191c700 15 client.36258.objecter _session_command_op_assign -1 7 2024-04-14T15:13:33.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.954+0000 7f699191c700 20 client.36258.objecter _get_session s=0x7f698c063540 osd=3 3 2024-04-14T15:13:33.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.954+0000 7f699191c700 20 client.36258.objecter put_session s=0x7f698c063540 osd=3 3 2024-04-14T15:13:33.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.954+0000 7f699191c700 20 client.36258.objecter _get_session s=0x7f698c063540 osd=3 3 2024-04-14T15:13:33.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.954+0000 7f699191c700 15 client.36258.objecter _session_command_op_remove -1 7 2024-04-14T15:13:33.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.954+0000 7f699191c700 20 client.36258.objecter get_session s=0x7f698c063540 osd=3 3 2024-04-14T15:13:33.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.954+0000 7f699191c700 15 client.36258.objecter _session_command_op_assign 3 7 2024-04-14T15:13:33.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.954+0000 7f699191c700 20 client.36258.objecter put_session s=0x7f698c063540 osd=3 4 2024-04-14T15:13:33.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.954+0000 7f699191c700 10 client.36258.objecter _send_command 7 2024-04-14T15:13:33.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.954+0000 7f699191c700 1 -- 172.21.15.90:0/2095658072 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f698c1b6200 con 0x7f698c02db00 2024-04-14T15:13:33.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.955+0000 7f699091a700 10 client.36258.objecter ms_handle_connect 0x7f698c02db00 2024-04-14T15:13:33.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.956+0000 7f699091a700 1 -- 172.21.15.90:0/2095658072 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f698c1b6200 con 0x7f698c02db00 2024-04-14T15:13:33.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.956+0000 7f699091a700 10 client.36258.objecter ms_dispatch 0x7f698c001550 command_reply(tid 7: 0 ) 2024-04-14T15:13:33.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.956+0000 7f699091a700 10 client.36258.objecter _finish_command 7 = system:0 2024-04-14T15:13:33.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.956+0000 7f699091a700 20 client.36258.objecter put_session s=0x7f698c063540 osd=3 4 2024-04-14T15:13:33.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.956+0000 7f699091a700 15 client.36258.objecter _session_command_op_remove 3 7 2024-04-14T15:13:33.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.975+0000 7f699191c700 10 client.36258.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:13:33.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.975+0000 7f699191c700 15 client.36258.objecter _session_command_op_assign -1 8 2024-04-14T15:13:33.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.975+0000 7f699191c700 20 client.36258.objecter _get_session s=0x7f698c063540 osd=3 3 2024-04-14T15:13:33.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.975+0000 7f699191c700 20 client.36258.objecter put_session s=0x7f698c063540 osd=3 3 2024-04-14T15:13:33.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.975+0000 7f699191c700 20 client.36258.objecter _get_session s=0x7f698c063540 osd=3 3 2024-04-14T15:13:33.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.975+0000 7f699191c700 15 client.36258.objecter _session_command_op_remove -1 8 2024-04-14T15:13:33.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.975+0000 7f699191c700 20 client.36258.objecter get_session s=0x7f698c063540 osd=3 3 2024-04-14T15:13:33.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.975+0000 7f699191c700 15 client.36258.objecter _session_command_op_assign 3 8 2024-04-14T15:13:33.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.975+0000 7f699191c700 20 client.36258.objecter put_session s=0x7f698c063540 osd=3 4 2024-04-14T15:13:33.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.975+0000 7f699191c700 10 client.36258.objecter _send_command 8 2024-04-14T15:13:33.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.975+0000 7f699191c700 1 -- 172.21.15.90:0/2095658072 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f698c02d190 con 0x7f698c02db00 2024-04-14T15:13:33.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.978+0000 7f699091a700 1 -- 172.21.15.90:0/2095658072 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f698c02d190 con 0x7f698c02db00 2024-04-14T15:13:33.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.978+0000 7f699091a700 10 client.36258.objecter ms_dispatch 0x7f698c001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:13:33.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.978+0000 7f699091a700 10 client.36258.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:13:33.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.978+0000 7f699091a700 20 client.36258.objecter put_session s=0x7f698c063540 osd=3 4 2024-04-14T15:13:33.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.978+0000 7f699091a700 15 client.36258.objecter _session_command_op_remove 3 8 2024-04-14T15:13:33.985 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:13:33.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.978+0000 7f699191c700 10 client.36258.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:13:33.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.978+0000 7f699191c700 15 client.36258.objecter _session_command_op_assign -1 9 2024-04-14T15:13:33.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.978+0000 7f699191c700 20 client.36258.objecter _get_session s=0x7f698c02d190 osd=4 3 2024-04-14T15:13:33.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.978+0000 7f699191c700 20 client.36258.objecter put_session s=0x7f698c02d190 osd=4 3 2024-04-14T15:13:33.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.978+0000 7f699191c700 20 client.36258.objecter _get_session s=0x7f698c02d190 osd=4 3 2024-04-14T15:13:33.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.978+0000 7f699191c700 15 client.36258.objecter _session_command_op_remove -1 9 2024-04-14T15:13:33.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.978+0000 7f699191c700 20 client.36258.objecter get_session s=0x7f698c02d190 osd=4 3 2024-04-14T15:13:33.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.978+0000 7f699191c700 15 client.36258.objecter _session_command_op_assign 4 9 2024-04-14T15:13:33.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.978+0000 7f699191c700 20 client.36258.objecter put_session s=0x7f698c02d190 osd=4 4 2024-04-14T15:13:33.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.978+0000 7f699191c700 10 client.36258.objecter _send_command 9 2024-04-14T15:13:33.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.978+0000 7f699191c700 1 -- 172.21.15.90:0/2095658072 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f698c1bad90 con 0x7f698c1b2520 2024-04-14T15:13:33.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.979+0000 7f699091a700 10 client.36258.objecter ms_handle_connect 0x7f698c1b2520 2024-04-14T15:13:33.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.983+0000 7f699091a700 1 -- 172.21.15.90:0/2095658072 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f698c1bad90 con 0x7f698c1b2520 2024-04-14T15:13:33.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.983+0000 7f699091a700 10 client.36258.objecter ms_dispatch 0x7f698c001550 command_reply(tid 9: 0 ) 2024-04-14T15:13:33.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.983+0000 7f699091a700 10 client.36258.objecter _finish_command 9 = system:0 2024-04-14T15:13:33.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.984+0000 7f699091a700 20 client.36258.objecter put_session s=0x7f698c02d190 osd=4 4 2024-04-14T15:13:33.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:33.984+0000 7f699091a700 15 client.36258.objecter _session_command_op_remove 4 9 2024-04-14T15:13:34.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.004+0000 7f699191c700 10 client.36258.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:13:34.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.004+0000 7f699191c700 15 client.36258.objecter _session_command_op_assign -1 10 2024-04-14T15:13:34.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.004+0000 7f699191c700 20 client.36258.objecter _get_session s=0x7f698c02d190 osd=4 3 2024-04-14T15:13:34.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.004+0000 7f699191c700 20 client.36258.objecter put_session s=0x7f698c02d190 osd=4 3 2024-04-14T15:13:34.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.004+0000 7f699191c700 20 client.36258.objecter _get_session s=0x7f698c02d190 osd=4 3 2024-04-14T15:13:34.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.004+0000 7f699191c700 15 client.36258.objecter _session_command_op_remove -1 10 2024-04-14T15:13:34.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.004+0000 7f699191c700 20 client.36258.objecter get_session s=0x7f698c02d190 osd=4 3 2024-04-14T15:13:34.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.004+0000 7f699191c700 15 client.36258.objecter _session_command_op_assign 4 10 2024-04-14T15:13:34.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.004+0000 7f699191c700 20 client.36258.objecter put_session s=0x7f698c02d190 osd=4 4 2024-04-14T15:13:34.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.004+0000 7f699191c700 10 client.36258.objecter _send_command 10 2024-04-14T15:13:34.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.004+0000 7f699191c700 1 -- 172.21.15.90:0/2095658072 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f698c1bad90 con 0x7f698c1b2520 2024-04-14T15:13:34.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.006+0000 7f699091a700 1 -- 172.21.15.90:0/2095658072 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f698c1bad90 con 0x7f698c1b2520 2024-04-14T15:13:34.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.006+0000 7f699091a700 10 client.36258.objecter ms_dispatch 0x7f698c001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:13:34.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.006+0000 7f699091a700 10 client.36258.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:13:34.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.006+0000 7f699091a700 20 client.36258.objecter put_session s=0x7f698c02d190 osd=4 4 2024-04-14T15:13:34.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.006+0000 7f699091a700 15 client.36258.objecter _session_command_op_remove 4 10 2024-04-14T15:13:34.007 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:13:34.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.006+0000 7f699191c700 10 client.36258.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:13:34.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.006+0000 7f699191c700 15 client.36258.objecter _session_command_op_assign -1 11 2024-04-14T15:13:34.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.006+0000 7f699191c700 20 client.36258.objecter _get_session s=0x7f698c1bad90 osd=5 3 2024-04-14T15:13:34.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.006+0000 7f699191c700 20 client.36258.objecter put_session s=0x7f698c1bad90 osd=5 3 2024-04-14T15:13:34.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.006+0000 7f699191c700 20 client.36258.objecter _get_session s=0x7f698c1bad90 osd=5 3 2024-04-14T15:13:34.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.006+0000 7f699191c700 15 client.36258.objecter _session_command_op_remove -1 11 2024-04-14T15:13:34.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.006+0000 7f699191c700 20 client.36258.objecter get_session s=0x7f698c1bad90 osd=5 3 2024-04-14T15:13:34.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.006+0000 7f699191c700 15 client.36258.objecter _session_command_op_assign 5 11 2024-04-14T15:13:34.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.006+0000 7f699191c700 20 client.36258.objecter put_session s=0x7f698c1bad90 osd=5 4 2024-04-14T15:13:34.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.006+0000 7f699191c700 10 client.36258.objecter _send_command 11 2024-04-14T15:13:34.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.006+0000 7f699191c700 1 -- 172.21.15.90:0/2095658072 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f698c1bf110 con 0x7f698c1bb410 2024-04-14T15:13:34.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.007+0000 7f699091a700 10 client.36258.objecter ms_handle_connect 0x7f698c1bb410 2024-04-14T15:13:34.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.007+0000 7f699091a700 1 -- 172.21.15.90:0/2095658072 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f698c1bf110 con 0x7f698c1bb410 2024-04-14T15:13:34.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.007+0000 7f699091a700 10 client.36258.objecter ms_dispatch 0x7f698c001550 command_reply(tid 11: 0 ) 2024-04-14T15:13:34.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.007+0000 7f699091a700 10 client.36258.objecter _finish_command 11 = system:0 2024-04-14T15:13:34.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.007+0000 7f699091a700 20 client.36258.objecter put_session s=0x7f698c1bad90 osd=5 4 2024-04-14T15:13:34.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.007+0000 7f699091a700 15 client.36258.objecter _session_command_op_remove 5 11 2024-04-14T15:13:34.018 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:13:34.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.024+0000 7f699191c700 10 client.36258.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:13:34.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.024+0000 7f699191c700 15 client.36258.objecter _session_command_op_assign -1 12 2024-04-14T15:13:34.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.024+0000 7f699191c700 20 client.36258.objecter _get_session s=0x7f698c1bad90 osd=5 3 2024-04-14T15:13:34.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.024+0000 7f699191c700 20 client.36258.objecter put_session s=0x7f698c1bad90 osd=5 3 2024-04-14T15:13:34.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.024+0000 7f699191c700 20 client.36258.objecter _get_session s=0x7f698c1bad90 osd=5 3 2024-04-14T15:13:34.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.024+0000 7f699191c700 15 client.36258.objecter _session_command_op_remove -1 12 2024-04-14T15:13:34.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.024+0000 7f699191c700 20 client.36258.objecter get_session s=0x7f698c1bad90 osd=5 3 2024-04-14T15:13:34.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.024+0000 7f699191c700 15 client.36258.objecter _session_command_op_assign 5 12 2024-04-14T15:13:34.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.024+0000 7f699191c700 20 client.36258.objecter put_session s=0x7f698c1bad90 osd=5 4 2024-04-14T15:13:34.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.024+0000 7f699191c700 10 client.36258.objecter _send_command 12 2024-04-14T15:13:34.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.024+0000 7f699191c700 1 -- 172.21.15.90:0/2095658072 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f698c1bf2f0 con 0x7f698c1bb410 2024-04-14T15:13:34.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.026+0000 7f699091a700 1 -- 172.21.15.90:0/2095658072 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f698c1bf2f0 con 0x7f698c1bb410 2024-04-14T15:13:34.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.026+0000 7f699091a700 10 client.36258.objecter ms_dispatch 0x7f698c001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:13:34.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.026+0000 7f699091a700 10 client.36258.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:13:34.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.026+0000 7f699091a700 20 client.36258.objecter put_session s=0x7f698c1bad90 osd=5 4 2024-04-14T15:13:34.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.026+0000 7f699091a700 15 client.36258.objecter _session_command_op_remove 5 12 2024-04-14T15:13:34.028 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:13:34.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.026+0000 7f699191c700 10 client.36258.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:13:34.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.026+0000 7f699191c700 15 client.36258.objecter _session_command_op_assign -1 13 2024-04-14T15:13:34.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.026+0000 7f699191c700 20 client.36258.objecter _get_session s=0x7f698c1bf2f0 osd=6 3 2024-04-14T15:13:34.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.026+0000 7f699191c700 20 client.36258.objecter put_session s=0x7f698c1bf2f0 osd=6 3 2024-04-14T15:13:34.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.026+0000 7f699191c700 20 client.36258.objecter _get_session s=0x7f698c1bf2f0 osd=6 3 2024-04-14T15:13:34.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.026+0000 7f699191c700 15 client.36258.objecter _session_command_op_remove -1 13 2024-04-14T15:13:34.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.026+0000 7f699191c700 20 client.36258.objecter get_session s=0x7f698c1bf2f0 osd=6 3 2024-04-14T15:13:34.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.026+0000 7f699191c700 15 client.36258.objecter _session_command_op_assign 6 13 2024-04-14T15:13:34.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.026+0000 7f699191c700 20 client.36258.objecter put_session s=0x7f698c1bf2f0 osd=6 4 2024-04-14T15:13:34.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.026+0000 7f699191c700 10 client.36258.objecter _send_command 13 2024-04-14T15:13:34.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.026+0000 7f699191c700 1 -- 172.21.15.90:0/2095658072 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f698c1c0620 con 0x7f698c1bfa10 2024-04-14T15:13:34.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.027+0000 7f699091a700 10 client.36258.objecter ms_handle_connect 0x7f698c1bfa10 2024-04-14T15:13:34.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.028+0000 7f699091a700 1 -- 172.21.15.90:0/2095658072 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f698c1c0620 con 0x7f698c1bfa10 2024-04-14T15:13:34.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.028+0000 7f699091a700 10 client.36258.objecter ms_dispatch 0x7f698c001550 command_reply(tid 13: 0 ) 2024-04-14T15:13:34.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.028+0000 7f699091a700 10 client.36258.objecter _finish_command 13 = system:0 2024-04-14T15:13:34.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.028+0000 7f699091a700 20 client.36258.objecter put_session s=0x7f698c1bf2f0 osd=6 4 2024-04-14T15:13:34.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.028+0000 7f699091a700 15 client.36258.objecter _session_command_op_remove 6 13 2024-04-14T15:13:34.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:34.045 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:34.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:13:34.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.045+0000 7f699191c700 10 client.36258.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:13:34.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.045+0000 7f699191c700 15 client.36258.objecter _session_command_op_assign -1 14 2024-04-14T15:13:34.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.045+0000 7f699191c700 20 client.36258.objecter _get_session s=0x7f698c1bf2f0 osd=6 3 2024-04-14T15:13:34.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.045+0000 7f699191c700 20 client.36258.objecter put_session s=0x7f698c1bf2f0 osd=6 3 2024-04-14T15:13:34.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.045+0000 7f699191c700 20 client.36258.objecter _get_session s=0x7f698c1bf2f0 osd=6 3 2024-04-14T15:13:34.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.045+0000 7f699191c700 15 client.36258.objecter _session_command_op_remove -1 14 2024-04-14T15:13:34.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.045+0000 7f699191c700 20 client.36258.objecter get_session s=0x7f698c1bf2f0 osd=6 3 2024-04-14T15:13:34.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.045+0000 7f699191c700 15 client.36258.objecter _session_command_op_assign 6 14 2024-04-14T15:13:34.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.045+0000 7f699191c700 20 client.36258.objecter put_session s=0x7f698c1bf2f0 osd=6 4 2024-04-14T15:13:34.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.045+0000 7f699191c700 10 client.36258.objecter _send_command 14 2024-04-14T15:13:34.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.045+0000 7f699191c700 1 -- 172.21.15.90:0/2095658072 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f698c1c0620 con 0x7f698c1bfa10 2024-04-14T15:13:34.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.047+0000 7f699091a700 1 -- 172.21.15.90:0/2095658072 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f698c1c0620 con 0x7f698c1bfa10 2024-04-14T15:13:34.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.047+0000 7f699091a700 10 client.36258.objecter ms_dispatch 0x7f698c001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:13:34.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.047+0000 7f699091a700 10 client.36258.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:13:34.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.047+0000 7f699091a700 20 client.36258.objecter put_session s=0x7f698c1bf2f0 osd=6 4 2024-04-14T15:13:34.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.047+0000 7f699091a700 15 client.36258.objecter _session_command_op_remove 6 14 2024-04-14T15:13:34.049 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:13:34.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.047+0000 7f699191c700 10 client.36258.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:13:34.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.047+0000 7f699191c700 15 client.36258.objecter _session_command_op_assign -1 15 2024-04-14T15:13:34.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.047+0000 7f699191c700 20 client.36258.objecter _get_session s=0x7f698c1c0620 osd=7 3 2024-04-14T15:13:34.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.047+0000 7f699191c700 20 client.36258.objecter put_session s=0x7f698c1c0620 osd=7 3 2024-04-14T15:13:34.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.047+0000 7f699191c700 20 client.36258.objecter _get_session s=0x7f698c1c0620 osd=7 3 2024-04-14T15:13:34.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.047+0000 7f699191c700 15 client.36258.objecter _session_command_op_remove -1 15 2024-04-14T15:13:34.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.047+0000 7f699191c700 20 client.36258.objecter get_session s=0x7f698c1c0620 osd=7 3 2024-04-14T15:13:34.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.047+0000 7f699191c700 15 client.36258.objecter _session_command_op_assign 7 15 2024-04-14T15:13:34.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.047+0000 7f699191c700 20 client.36258.objecter put_session s=0x7f698c1c0620 osd=7 4 2024-04-14T15:13:34.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.047+0000 7f699191c700 10 client.36258.objecter _send_command 15 2024-04-14T15:13:34.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.047+0000 7f699191c700 1 -- 172.21.15.90:0/2095658072 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f698c1c7b00 con 0x7f698c1c3ff0 2024-04-14T15:13:34.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.048+0000 7f699091a700 10 client.36258.objecter ms_handle_connect 0x7f698c1c3ff0 2024-04-14T15:13:34.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.048+0000 7f699091a700 1 -- 172.21.15.90:0/2095658072 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f698c1c7b00 con 0x7f698c1c3ff0 2024-04-14T15:13:34.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.048+0000 7f699091a700 10 client.36258.objecter ms_dispatch 0x7f698c001550 command_reply(tid 15: 0 ) 2024-04-14T15:13:34.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.048+0000 7f699091a700 10 client.36258.objecter _finish_command 15 = system:0 2024-04-14T15:13:34.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.048+0000 7f699091a700 20 client.36258.objecter put_session s=0x7f698c1c0620 osd=7 4 2024-04-14T15:13:34.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.048+0000 7f699091a700 15 client.36258.objecter _session_command_op_remove 7 15 2024-04-14T15:13:34.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.065+0000 7f699191c700 10 client.36258.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:13:34.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.065+0000 7f699191c700 15 client.36258.objecter _session_command_op_assign -1 16 2024-04-14T15:13:34.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.065+0000 7f699191c700 20 client.36258.objecter _get_session s=0x7f698c1c0620 osd=7 3 2024-04-14T15:13:34.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.065+0000 7f699191c700 20 client.36258.objecter put_session s=0x7f698c1c0620 osd=7 3 2024-04-14T15:13:34.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.065+0000 7f699191c700 20 client.36258.objecter _get_session s=0x7f698c1c0620 osd=7 3 2024-04-14T15:13:34.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.065+0000 7f699191c700 15 client.36258.objecter _session_command_op_remove -1 16 2024-04-14T15:13:34.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.065+0000 7f699191c700 20 client.36258.objecter get_session s=0x7f698c1c0620 osd=7 3 2024-04-14T15:13:34.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.065+0000 7f699191c700 15 client.36258.objecter _session_command_op_assign 7 16 2024-04-14T15:13:34.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.065+0000 7f699191c700 20 client.36258.objecter put_session s=0x7f698c1c0620 osd=7 4 2024-04-14T15:13:34.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.065+0000 7f699191c700 10 client.36258.objecter _send_command 16 2024-04-14T15:13:34.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.065+0000 7f699191c700 1 -- 172.21.15.90:0/2095658072 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f698c1b7b80 con 0x7f698c1c3ff0 2024-04-14T15:13:34.067 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-14T15:13:34.067 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:13:34.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.067+0000 7f699091a700 1 -- 172.21.15.90:0/2095658072 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f698c1b7b80 con 0x7f698c1c3ff0 2024-04-14T15:13:34.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.067+0000 7f699091a700 10 client.36258.objecter ms_dispatch 0x7f698c001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:13:34.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.067+0000 7f699091a700 10 client.36258.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:13:34.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.067+0000 7f699091a700 20 client.36258.objecter put_session s=0x7f698c1c0620 osd=7 4 2024-04-14T15:13:34.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.067+0000 7f699091a700 15 client.36258.objecter _session_command_op_remove 7 16 2024-04-14T15:13:34.068 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:13:34.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.068+0000 7f699191c700 10 librados: watch_flush enter 2024-04-14T15:13:34.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.068+0000 7f699191c700 10 librados: watch_flush exit 2024-04-14T15:13:34.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.068+0000 7f699191c700 10 client.36258.objecter close_session for osd.0 2024-04-14T15:13:34.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.068+0000 7f699191c700 1 -- 172.21.15.90:0/2095658072 >> v1:172.21.15.90:6801/3761806885 conn(0x7f698c1a9750 legacy=0x7f698c1a9b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:34.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.068+0000 7f699191c700 20 client.36258.objecter put_session s=0x7f698c113490 osd=0 1 2024-04-14T15:13:34.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.068+0000 7f699191c700 10 client.36258.objecter close_session for osd.1 2024-04-14T15:13:34.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.068+0000 7f699191c700 1 -- 172.21.15.90:0/2095658072 >> v1:172.21.15.90:6805/4170361699 conn(0x7f698c0636c0 legacy=0x7f698c10bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:34.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.068+0000 7f699191c700 20 client.36258.objecter put_session s=0x7f698c109960 osd=1 1 2024-04-14T15:13:34.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.068+0000 7f699191c700 10 client.36258.objecter close_session for osd.2 2024-04-14T15:13:34.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.068+0000 7f699191c700 1 -- 172.21.15.90:0/2095658072 >> v1:172.21.15.90:6809/1686184251 conn(0x7f698c1adb80 legacy=0x7f698c1aff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:34.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.068+0000 7f699191c700 20 client.36258.objecter put_session s=0x7f698c063f00 osd=2 1 2024-04-14T15:13:34.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.068+0000 7f699191c700 10 client.36258.objecter close_session for osd.3 2024-04-14T15:13:34.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.068+0000 7f699191c700 1 -- 172.21.15.90:0/2095658072 >> v1:172.21.15.90:6813/2270166774 conn(0x7f698c02db00 legacy=0x7f698c1b49d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:34.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.068+0000 7f699191c700 20 client.36258.objecter put_session s=0x7f698c063540 osd=3 1 2024-04-14T15:13:34.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.068+0000 7f699191c700 10 client.36258.objecter close_session for osd.4 2024-04-14T15:13:34.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.068+0000 7f699191c700 1 -- 172.21.15.90:0/2095658072 >> v1:172.21.15.92:6800/3887549733 conn(0x7f698c1b2520 legacy=0x7f698c1b63e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:34.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.068+0000 7f699191c700 20 client.36258.objecter put_session s=0x7f698c02d190 osd=4 1 2024-04-14T15:13:34.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.068+0000 7f699191c700 10 client.36258.objecter close_session for osd.5 2024-04-14T15:13:34.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.068+0000 7f698b7fe700 1 -- 172.21.15.90:0/2095658072 reap_dead start 2024-04-14T15:13:34.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.068+0000 7f699191c700 1 -- 172.21.15.90:0/2095658072 >> v1:172.21.15.92:6804/3244561440 conn(0x7f698c1bb410 legacy=0x7f698c1bd820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:34.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.068+0000 7f699191c700 20 client.36258.objecter put_session s=0x7f698c1bad90 osd=5 1 2024-04-14T15:13:34.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.068+0000 7f699191c700 10 client.36258.objecter close_session for osd.6 2024-04-14T15:13:34.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.068+0000 7f699191c700 1 -- 172.21.15.90:0/2095658072 >> v1:172.21.15.92:6808/636125963 conn(0x7f698c1bfa10 legacy=0x7f698c1bfe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:34.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.068+0000 7f699191c700 20 client.36258.objecter put_session s=0x7f698c1bf2f0 osd=6 1 2024-04-14T15:13:34.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.068+0000 7f699191c700 10 client.36258.objecter close_session for osd.7 2024-04-14T15:13:34.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.068+0000 7f699191c700 1 -- 172.21.15.90:0/2095658072 >> v1:172.21.15.92:6812/1773706716 conn(0x7f698c1c3ff0 legacy=0x7f698c1c63e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:34.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.068+0000 7f699191c700 20 client.36258.objecter put_session s=0x7f698c1c0620 osd=7 1 2024-04-14T15:13:34.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.068+0000 7f699191c700 20 client.36258.objecter shutdown clearing up homeless session... 2024-04-14T15:13:34.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.068+0000 7f699191c700 10 client.36258.objecter successfully canceled tick 2024-04-14T15:13:34.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.068+0000 7f698b7fe700 1 -- 172.21.15.90:0/2095658072 reap_dead start 2024-04-14T15:13:34.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.068+0000 7f698b7fe700 1 -- 172.21.15.90:0/2095658072 reap_dead start 2024-04-14T15:13:34.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.068+0000 7f699191c700 1 -- 172.21.15.90:0/2095658072 >> v1:172.21.15.90:6800/420935427 conn(0x7f69740712d0 legacy=0x7f6974073790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:34.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.068+0000 7f699191c700 1 -- 172.21.15.90:0/2095658072 >> v1:172.21.15.90:6789/0 conn(0x7f698c103a40 legacy=0x7f698c108430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:34.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.068+0000 7f699191c700 1 -- 172.21.15.90:0/2095658072 shutdown_connections 2024-04-14T15:13:34.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.068+0000 7f699191c700 1 -- 172.21.15.90:0/2095658072 >> 172.21.15.90:0/2095658072 conn(0x7f698c0fd290 msgr2=0x7f698c111930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:13:34.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.068+0000 7f699191c700 1 -- 172.21.15.90:0/2095658072 shutdown_connections 2024-04-14T15:13:34.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.068+0000 7f699191c700 1 -- 172.21.15.90:0/2095658072 wait complete. 2024-04-14T15:13:34.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:34.068+0000 7f699191c700 1 librados: shutdown 2024-04-14T15:13:34.073 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:13:34.073 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:13:34.073 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:13:34.073 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:13:34.073 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:13:34.073 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:13:34.073 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:13:34.073 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:13:34.146 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:13:34.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:34.171 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:34.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:13:34.192 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:13:33 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:13:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:13:34.272 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:13:34.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:34.298 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:34.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:13:34.399 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:13:34.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:34.429 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:34.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:13:34.530 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:13:34.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:34.560 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:34.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:13:34.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:34 smithi092 ceph-mon[33966]: 2.f scrub starts 2024-04-14T15:13:34.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:34 smithi092 ceph-mon[33966]: 2.f scrub ok 2024-04-14T15:13:34.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:34 smithi092 ceph-mon[33966]: 3.2 scrub starts 2024-04-14T15:13:34.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:34 smithi092 ceph-mon[33966]: 3.2 scrub ok 2024-04-14T15:13:34.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:34 smithi092 ceph-mon[33966]: pgmap v164: 364 pgs: 32 unknown, 1 active+clean+snaptrim, 331 active+clean; 145 MiB data, 824 MiB used, 714 GiB / 715 GiB avail 2024-04-14T15:13:34.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:34 smithi092 ceph-mon[33966]: 16.7 deep-scrub starts 2024-04-14T15:13:34.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:34 smithi092 ceph-mon[33966]: 16.7 deep-scrub ok 2024-04-14T15:13:34.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:34 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:34.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:34 smithi092 ceph-mon[33966]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:13:34.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:34 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-11"}]': finished 2024-04-14T15:13:34.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:34 smithi092 ceph-mon[33966]: osdmap e146: 8 total, 8 up, 8 in 2024-04-14T15:13:34.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:34 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2598250370' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi090-93709-10"}]: dispatch 2024-04-14T15:13:34.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:34 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1257184442' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi090-93363-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:34.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:34 smithi092 ceph-mon[33966]: from='client.38086 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi090-93709-10"}]: dispatch 2024-04-14T15:13:34.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:34 smithi092 ceph-mon[33966]: from='client.48049 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi090-93363-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:34.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:34 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2949828533' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi090-93342-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:34.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:34 smithi092 ceph-mon[33966]: from='client.46178 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi090-93342-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:34.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:34 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2095658072' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:13:34.662 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:13:34.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:34.687 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:34.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:13:34.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:34 smithi090 ceph-mon[30399]: 2.f scrub starts 2024-04-14T15:13:34.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:34 smithi090 ceph-mon[30399]: 2.f scrub ok 2024-04-14T15:13:34.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:34 smithi090 ceph-mon[30399]: 3.2 scrub starts 2024-04-14T15:13:34.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:34 smithi090 ceph-mon[30399]: 3.2 scrub ok 2024-04-14T15:13:34.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:34 smithi090 ceph-mon[30399]: pgmap v164: 364 pgs: 32 unknown, 1 active+clean+snaptrim, 331 active+clean; 145 MiB data, 824 MiB used, 714 GiB / 715 GiB avail 2024-04-14T15:13:34.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:34 smithi090 ceph-mon[30399]: 16.7 deep-scrub starts 2024-04-14T15:13:34.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:34 smithi090 ceph-mon[30399]: 16.7 deep-scrub ok 2024-04-14T15:13:34.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:34 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:34.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:34 smithi090 ceph-mon[30399]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:13:34.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:34 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-11"}]': finished 2024-04-14T15:13:34.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:34 smithi090 ceph-mon[30399]: osdmap e146: 8 total, 8 up, 8 in 2024-04-14T15:13:34.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:34 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2598250370' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi090-93709-10"}]: dispatch 2024-04-14T15:13:34.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:34 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1257184442' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi090-93363-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:34.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:34 smithi090 ceph-mon[30399]: from='client.38086 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi090-93709-10"}]: dispatch 2024-04-14T15:13:34.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:34 smithi090 ceph-mon[30399]: from='client.48049 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi090-93363-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:34.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:34 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2949828533' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi090-93342-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:34.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:34 smithi090 ceph-mon[30399]: from='client.46178 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi090-93342-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:34.695 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:34 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2095658072' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:13:34.695 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:34 smithi090 ceph-mon[41670]: 2.f scrub starts 2024-04-14T15:13:34.696 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:34 smithi090 ceph-mon[41670]: 2.f scrub ok 2024-04-14T15:13:34.696 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:34 smithi090 ceph-mon[41670]: 3.2 scrub starts 2024-04-14T15:13:34.696 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:34 smithi090 ceph-mon[41670]: 3.2 scrub ok 2024-04-14T15:13:34.696 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:34 smithi090 ceph-mon[41670]: pgmap v164: 364 pgs: 32 unknown, 1 active+clean+snaptrim, 331 active+clean; 145 MiB data, 824 MiB used, 714 GiB / 715 GiB avail 2024-04-14T15:13:34.696 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:34 smithi090 ceph-mon[41670]: 16.7 deep-scrub starts 2024-04-14T15:13:34.697 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:34 smithi090 ceph-mon[41670]: 16.7 deep-scrub ok 2024-04-14T15:13:34.697 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:34 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:34.697 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:34 smithi090 ceph-mon[41670]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:13:34.697 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:34 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-11"}]': finished 2024-04-14T15:13:34.697 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:34 smithi090 ceph-mon[41670]: osdmap e146: 8 total, 8 up, 8 in 2024-04-14T15:13:34.697 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:34 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2598250370' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi090-93709-10"}]: dispatch 2024-04-14T15:13:34.697 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:34 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1257184442' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi090-93363-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:34.698 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:34 smithi090 ceph-mon[41670]: from='client.38086 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi090-93709-10"}]: dispatch 2024-04-14T15:13:34.698 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:34 smithi090 ceph-mon[41670]: from='client.48049 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi090-93363-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:34.698 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:34 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2949828533' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi090-93342-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:34.698 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:34 smithi090 ceph-mon[41670]: from='client.46178 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi090-93342-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:34.698 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:34 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2095658072' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:13:34.774 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-14T15:13:34.788 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:13:34.862 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-14T15:13:34.863 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-14T15:13:34.863 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-04-14T15:13:34.863 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph pg dump --format=json 2024-04-14T15:13:34.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:34.881 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:34.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:13:34.983 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:13:35.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:35.010 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:35.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:13:35.111 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:13:35.122 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:13:34 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:13:35.141 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:35.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:35.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:13:35.244 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:13:35.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:35.273 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:35.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:13:35.374 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:13:35.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:35.402 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:35.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:13:35.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:35 smithi090 ceph-mon[30399]: 5.e scrub starts 2024-04-14T15:13:35.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:35 smithi090 ceph-mon[30399]: 5.e scrub ok 2024-04-14T15:13:35.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:35 smithi090 ceph-mon[30399]: 2.12 scrub starts 2024-04-14T15:13:35.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:35 smithi090 ceph-mon[30399]: 2.12 scrub ok 2024-04-14T15:13:35.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:35 smithi090 ceph-mon[30399]: 2.1b scrub starts 2024-04-14T15:13:35.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:35 smithi090 ceph-mon[30399]: 2.1b scrub ok 2024-04-14T15:13:35.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:35 smithi090 ceph-mon[30399]: 6.11 scrub starts 2024-04-14T15:13:35.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:35 smithi090 ceph-mon[30399]: 6.11 scrub ok 2024-04-14T15:13:35.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:35 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:35.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:35 smithi090 ceph-mon[30399]: from='client.38086 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi090-93709-10"}]': finished 2024-04-14T15:13:35.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:35 smithi090 ceph-mon[30399]: from='client.48049 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi090-93363-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:35.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:35 smithi090 ceph-mon[30399]: from='client.46178 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi090-93342-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:35.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:35 smithi090 ceph-mon[30399]: osdmap e147: 8 total, 8 up, 8 in 2024-04-14T15:13:35.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:35 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2598250370' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi090-93709-10"}]: dispatch 2024-04-14T15:13:35.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:35 smithi090 ceph-mon[30399]: from='client.38086 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi090-93709-10"}]: dispatch 2024-04-14T15:13:35.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:35 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:13:35.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:35 smithi090 ceph-mon[41670]: 5.e scrub starts 2024-04-14T15:13:35.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:35 smithi090 ceph-mon[41670]: 5.e scrub ok 2024-04-14T15:13:35.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:35 smithi090 ceph-mon[41670]: 2.12 scrub starts 2024-04-14T15:13:35.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:35 smithi090 ceph-mon[41670]: 2.12 scrub ok 2024-04-14T15:13:35.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:35 smithi090 ceph-mon[41670]: 2.1b scrub starts 2024-04-14T15:13:35.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:35 smithi090 ceph-mon[41670]: 2.1b scrub ok 2024-04-14T15:13:35.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:35 smithi090 ceph-mon[41670]: 6.11 scrub starts 2024-04-14T15:13:35.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:35 smithi090 ceph-mon[41670]: 6.11 scrub ok 2024-04-14T15:13:35.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:35 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:35.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:35 smithi090 ceph-mon[41670]: from='client.38086 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi090-93709-10"}]': finished 2024-04-14T15:13:35.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:35 smithi090 ceph-mon[41670]: from='client.48049 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi090-93363-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:35.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:35 smithi090 ceph-mon[41670]: from='client.46178 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi090-93342-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:35.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:35 smithi090 ceph-mon[41670]: osdmap e147: 8 total, 8 up, 8 in 2024-04-14T15:13:35.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:35 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2598250370' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi090-93709-10"}]: dispatch 2024-04-14T15:13:35.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:35 smithi090 ceph-mon[41670]: from='client.38086 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi090-93709-10"}]: dispatch 2024-04-14T15:13:35.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:35 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:13:35.481 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:13:35.502 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:13:35.528 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:35.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:35.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:13:35.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:35 smithi092 ceph-mon[33966]: 5.e scrub starts 2024-04-14T15:13:35.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:35 smithi092 ceph-mon[33966]: 5.e scrub ok 2024-04-14T15:13:35.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:35 smithi092 ceph-mon[33966]: 2.12 scrub starts 2024-04-14T15:13:35.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:35 smithi092 ceph-mon[33966]: 2.12 scrub ok 2024-04-14T15:13:35.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:35 smithi092 ceph-mon[33966]: 2.1b scrub starts 2024-04-14T15:13:35.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:35 smithi092 ceph-mon[33966]: 2.1b scrub ok 2024-04-14T15:13:35.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:35 smithi092 ceph-mon[33966]: 6.11 scrub starts 2024-04-14T15:13:35.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:35 smithi092 ceph-mon[33966]: 6.11 scrub ok 2024-04-14T15:13:35.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:35 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:35.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:35 smithi092 ceph-mon[33966]: from='client.38086 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi090-93709-10"}]': finished 2024-04-14T15:13:35.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:35 smithi092 ceph-mon[33966]: from='client.48049 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi090-93363-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:35.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:35 smithi092 ceph-mon[33966]: from='client.46178 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi090-93342-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:35.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:35 smithi092 ceph-mon[33966]: osdmap e147: 8 total, 8 up, 8 in 2024-04-14T15:13:35.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:35 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2598250370' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi090-93709-10"}]: dispatch 2024-04-14T15:13:35.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:35 smithi092 ceph-mon[33966]: from='client.38086 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi090-93709-10"}]: dispatch 2024-04-14T15:13:35.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:35 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:13:35.631 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:13:35.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:35.659 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:35.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:13:35.760 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:13:35.786 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:35.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:35.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:13:35.827 INFO:tasks.workunit.client.0.smithi090.stdout: read_operations: Running main() from gmock_main.cc 2024-04-14T15:13:35.827 INFO:tasks.workunit.client.0.smithi090.stdout: read_operations: [==========] Running 14 tests from 1 test suite. 2024-04-14T15:13:35.827 INFO:tasks.workunit.client.0.smithi090.stdout: read_operations: [----------] Global test environment set-up. 2024-04-14T15:13:35.827 INFO:tasks.workunit.client.0.smithi090.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps 2024-04-14T15:13:35.827 INFO:tasks.workunit.client.0.smithi090.stdout: read_operations: [ RUN ] NeoRadosReadOps.SetOpFlags 2024-04-14T15:13:35.827 INFO:tasks.workunit.client.0.smithi090.stdout: read_operations: [ OK ] NeoRadosReadOps.SetOpFlags (2493 ms) 2024-04-14T15:13:35.827 INFO:tasks.workunit.client.0.smithi090.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertExists 2024-04-14T15:13:35.827 INFO:tasks.workunit.client.0.smithi090.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertExists (3033 ms) 2024-04-14T15:13:35.827 INFO:tasks.workunit.client.0.smithi090.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertVersion 2024-04-14T15:13:35.827 INFO:tasks.workunit.client.0.smithi090.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertVersion (2703 ms) 2024-04-14T15:13:35.827 INFO:tasks.workunit.client.0.smithi090.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpXattr 2024-04-14T15:13:35.827 INFO:tasks.workunit.client.0.smithi090.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpXattr (2998 ms) 2024-04-14T15:13:35.828 INFO:tasks.workunit.client.0.smithi090.stdout: read_operations: [ RUN ] NeoRadosReadOps.Read 2024-04-14T15:13:35.828 INFO:tasks.workunit.client.0.smithi090.stdout: read_operations: [ OK ] NeoRadosReadOps.Read (3029 ms) 2024-04-14T15:13:35.828 INFO:tasks.workunit.client.0.smithi090.stdout: read_operations: [ RUN ] NeoRadosReadOps.Checksum 2024-04-14T15:13:35.828 INFO:tasks.workunit.client.0.smithi090.stdout: read_operations: [ OK ] NeoRadosReadOps.Checksum (2994 ms) 2024-04-14T15:13:35.828 INFO:tasks.workunit.client.0.smithi090.stdout: read_operations: [ RUN ] NeoRadosReadOps.RWOrderedRead 2024-04-14T15:13:35.828 INFO:tasks.workunit.client.0.smithi090.stdout: read_operations: [ OK ] NeoRadosReadOps.RWOrderedRead (3005 ms) 2024-04-14T15:13:35.828 INFO:tasks.workunit.client.0.smithi090.stdout: read_operations: [ RUN ] NeoRadosReadOps.ShortRead 2024-04-14T15:13:35.828 INFO:tasks.workunit.client.0.smithi090.stdout: read_operations: [ OK ] NeoRadosReadOps.ShortRead (2993 ms) 2024-04-14T15:13:35.828 INFO:tasks.workunit.client.0.smithi090.stdout: read_operations: [ RUN ] NeoRadosReadOps.Exec 2024-04-14T15:13:35.828 INFO:tasks.workunit.client.0.smithi090.stdout: read_operations: [ OK ] NeoRadosReadOps.Exec (2989 ms) 2024-04-14T15:13:35.828 INFO:tasks.workunit.client.0.smithi090.stdout: read_operations: [ RUN ] NeoRadosReadOps.Stat 2024-04-14T15:13:35.828 INFO:tasks.workunit.client.0.smithi090.stdout: read_operations: [ OK ] NeoRadosReadOps.Stat (3084 ms) 2024-04-14T15:13:35.828 INFO:tasks.workunit.client.0.smithi090.stdout: read_operations: [ RUN ] NeoRadosReadOps.Omap 2024-04-14T15:13:35.828 INFO:tasks.workunit.client.0.smithi090.stdout: read_operations: [ OK ] NeoRadosReadOps.Omap (2953 ms) 2024-04-14T15:13:35.828 INFO:tasks.workunit.client.0.smithi090.stdout: read_operations: [ RUN ] NeoRadosReadOps.OmapNuls 2024-04-14T15:13:35.829 INFO:tasks.workunit.client.0.smithi090.stdout: read_operations: [ OK ] NeoRadosReadOps.OmapNuls (3030 ms) 2024-04-14T15:13:35.829 INFO:tasks.workunit.client.0.smithi090.stdout: read_operations: [ RUN ] NeoRadosReadOps.GetXattrs 2024-04-14T15:13:35.829 INFO:tasks.workunit.client.0.smithi090.stdout: read_operations: [ OK ] NeoRadosReadOps.GetXattrs (2941 ms) 2024-04-14T15:13:35.829 INFO:tasks.workunit.client.0.smithi090.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpExt 2024-04-14T15:13:35.829 INFO:tasks.workunit.client.0.smithi090.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpExt (2993 ms) 2024-04-14T15:13:35.829 INFO:tasks.workunit.client.0.smithi090.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps (41238 ms total) 2024-04-14T15:13:35.829 INFO:tasks.workunit.client.0.smithi090.stdout: read_operations: 2024-04-14T15:13:35.829 INFO:tasks.workunit.client.0.smithi090.stdout: read_operations: [----------] Global test environment tear-down 2024-04-14T15:13:35.829 INFO:tasks.workunit.client.0.smithi090.stdout: read_operations: [==========] 14 tests from 1 test suite ran. (41238 ms total) 2024-04-14T15:13:35.829 INFO:tasks.workunit.client.0.smithi090.stdout: read_operations: [ PASSED ] 14 tests. 2024-04-14T15:13:35.835 INFO:tasks.workunit.client.0.smithi090.stdout: io: Running main() from gmock_main.cc 2024-04-14T15:13:35.835 INFO:tasks.workunit.client.0.smithi090.stdout: io: [==========] Running 14 tests from 1 test suite. 2024-04-14T15:13:35.835 INFO:tasks.workunit.client.0.smithi090.stdout: io: [----------] Global test environment set-up. 2024-04-14T15:13:35.835 INFO:tasks.workunit.client.0.smithi090.stdout: io: [----------] 14 tests from NeoRadosIo 2024-04-14T15:13:35.835 INFO:tasks.workunit.client.0.smithi090.stdout: io: [ RUN ] NeoRadosIo.Limits 2024-04-14T15:13:35.836 INFO:tasks.workunit.client.0.smithi090.stdout: io: [ OK ] NeoRadosIo.Limits (2568 ms) 2024-04-14T15:13:35.836 INFO:tasks.workunit.client.0.smithi090.stdout: io: [ RUN ] NeoRadosIo.SimpleWrite 2024-04-14T15:13:35.836 INFO:tasks.workunit.client.0.smithi090.stdout: io: [ OK ] NeoRadosIo.SimpleWrite (3029 ms) 2024-04-14T15:13:35.836 INFO:tasks.workunit.client.0.smithi090.stdout: io: [ RUN ] NeoRadosIo.ReadOp 2024-04-14T15:13:35.836 INFO:tasks.workunit.client.0.smithi090.stdout: io: [ OK ] NeoRadosIo.ReadOp (2707 ms) 2024-04-14T15:13:35.836 INFO:tasks.workunit.client.0.smithi090.stdout: io: [ RUN ] NeoRadosIo.SparseRead 2024-04-14T15:13:35.836 INFO:tasks.workunit.client.0.smithi090.stdout: io: [ OK ] NeoRadosIo.SparseRead (2987 ms) 2024-04-14T15:13:35.836 INFO:tasks.workunit.client.0.smithi090.stdout: io: [ RUN ] NeoRadosIo.RoundTrip 2024-04-14T15:13:35.836 INFO:tasks.workunit.client.0.smithi090.stdout: io: [ OK ] NeoRadosIo.RoundTrip (3019 ms) 2024-04-14T15:13:35.836 INFO:tasks.workunit.client.0.smithi090.stdout: io: [ RUN ] NeoRadosIo.ReadIntoBuufferlist 2024-04-14T15:13:35.836 INFO:tasks.workunit.client.0.smithi090.stdout: io: [ OK ] NeoRadosIo.ReadIntoBuufferlist (3028 ms) 2024-04-14T15:13:35.836 INFO:tasks.workunit.client.0.smithi090.stdout: io: [ RUN ] NeoRadosIo.OverlappingWriteRoundTrip 2024-04-14T15:13:35.836 INFO:tasks.workunit.client.0.smithi090.stdout: io: [ OK ] NeoRadosIo.OverlappingWriteRoundTrip (2994 ms) 2024-04-14T15:13:35.836 INFO:tasks.workunit.client.0.smithi090.stdout: io: [ RUN ] NeoRadosIo.WriteFullRoundTrip 2024-04-14T15:13:35.836 INFO:tasks.workunit.client.0.smithi090.stdout: io: [ OK ] NeoRadosIo.WriteFullRoundTrip (2979 ms) 2024-04-14T15:13:35.837 INFO:tasks.workunit.client.0.smithi090.stdout: io: [ RUN ] NeoRadosIo.AppendRoundTrip 2024-04-14T15:13:35.837 INFO:tasks.workunit.client.0.smithi090.stdout: io: [ OK ] NeoRadosIo.AppendRoundTrip (3000 ms) 2024-04-14T15:13:35.837 INFO:tasks.workunit.client.0.smithi090.stdout: io: [ RUN ] NeoRadosIo.Trunc 2024-04-14T15:13:35.837 INFO:tasks.workunit.client.0.smithi090.stdout: io: [ OK ] NeoRadosIo.Trunc (3090 ms) 2024-04-14T15:13:35.837 INFO:tasks.workunit.client.0.smithi090.stdout: io: [ RUN ] NeoRadosIo.Remove 2024-04-14T15:13:35.837 INFO:tasks.workunit.client.0.smithi090.stdout: io: [ OK ] NeoRadosIo.Remove (2911 ms) 2024-04-14T15:13:35.837 INFO:tasks.workunit.client.0.smithi090.stdout: io: [ RUN ] NeoRadosIo.XattrsRoundTrip 2024-04-14T15:13:35.837 INFO:tasks.workunit.client.0.smithi090.stdout: io: [ OK ] NeoRadosIo.XattrsRoundTrip (3055 ms) 2024-04-14T15:13:35.837 INFO:tasks.workunit.client.0.smithi090.stdout: io: [ RUN ] NeoRadosIo.RmXattr 2024-04-14T15:13:35.837 INFO:tasks.workunit.client.0.smithi090.stdout: io: [ OK ] NeoRadosIo.RmXattr (2947 ms) 2024-04-14T15:13:35.837 INFO:tasks.workunit.client.0.smithi090.stdout: io: [ RUN ] NeoRadosIo.GetXattrs 2024-04-14T15:13:35.837 INFO:tasks.workunit.client.0.smithi090.stdout: io: [ OK ] NeoRadosIo.GetXattrs (2993 ms) 2024-04-14T15:13:35.837 INFO:tasks.workunit.client.0.smithi090.stdout: io: [----------] 14 tests from NeoRadosIo (41308 ms total) 2024-04-14T15:13:35.837 INFO:tasks.workunit.client.0.smithi090.stdout: io: 2024-04-14T15:13:35.837 INFO:tasks.workunit.client.0.smithi090.stdout: io: [----------] Global test environment tear-down 2024-04-14T15:13:35.838 INFO:tasks.workunit.client.0.smithi090.stdout: io: [==========] 14 tests from 1 test suite ran. (41308 ms total) 2024-04-14T15:13:35.838 INFO:tasks.workunit.client.0.smithi090.stdout: io: [ PASSED ] 14 tests. 2024-04-14T15:13:35.888 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:13:35.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:35.913 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:35.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:13:36.014 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:13:36.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:36.039 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:36.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:13:36.140 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:13:36.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:36.165 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:36.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:13:36.266 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:13:36.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:36.295 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:36.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:13:36.395 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:13:36.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:36.421 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:36.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:13:36.523 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:13:36.534 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:36 smithi090 ceph-mon[41670]: 2.c deep-scrub starts 2024-04-14T15:13:36.535 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:36 smithi090 ceph-mon[41670]: 2.c deep-scrub ok 2024-04-14T15:13:36.535 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:36 smithi090 ceph-mon[41670]: pgmap v167: 452 pgs: 128 creating+peering, 324 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 18 MiB/s wr, 6 op/s 2024-04-14T15:13:36.535 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:36 smithi090 ceph-mon[41670]: 4.4 scrub starts 2024-04-14T15:13:36.535 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:36 smithi090 ceph-mon[41670]: 4.4 scrub ok 2024-04-14T15:13:36.535 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:36 smithi090 ceph-mon[41670]: 5.5 deep-scrub starts 2024-04-14T15:13:36.535 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:36 smithi090 ceph-mon[41670]: 5.5 deep-scrub ok 2024-04-14T15:13:36.535 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:36 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:36.535 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:36 smithi090 ceph-mon[41670]: from='client.38086 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi090-93709-10"}]': finished 2024-04-14T15:13:36.535 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:36 smithi090 ceph-mon[41670]: osdmap e148: 8 total, 8 up, 8 in 2024-04-14T15:13:36.535 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:36 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:36.535 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:36 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:36.535 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:36 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2918742503' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi090-93709-15"}]: dispatch 2024-04-14T15:13:36.536 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:36 smithi090 ceph-mon[41670]: from='client.48067 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi090-93709-15"}]: dispatch 2024-04-14T15:13:36.536 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:36 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2918742503' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi090-93709-15"}]: dispatch 2024-04-14T15:13:36.536 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:36 smithi090 ceph-mon[41670]: from='client.48067 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi090-93709-15"}]: dispatch 2024-04-14T15:13:36.536 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:36 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2918742503' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi090-93709-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:13:36.536 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:36 smithi090 ceph-mon[41670]: from='client.48067 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi090-93709-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:13:36.536 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:36 smithi090 ceph-mon[30399]: 2.c deep-scrub starts 2024-04-14T15:13:36.536 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:36 smithi090 ceph-mon[30399]: 2.c deep-scrub ok 2024-04-14T15:13:36.536 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:36 smithi090 ceph-mon[30399]: pgmap v167: 452 pgs: 128 creating+peering, 324 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 18 MiB/s wr, 6 op/s 2024-04-14T15:13:36.536 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:36 smithi090 ceph-mon[30399]: 4.4 scrub starts 2024-04-14T15:13:36.536 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:36 smithi090 ceph-mon[30399]: 4.4 scrub ok 2024-04-14T15:13:36.537 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:36 smithi090 ceph-mon[30399]: 5.5 deep-scrub starts 2024-04-14T15:13:36.537 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:36 smithi090 ceph-mon[30399]: 5.5 deep-scrub ok 2024-04-14T15:13:36.537 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:36 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:36.537 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:36 smithi090 ceph-mon[30399]: from='client.38086 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi090-93709-10"}]': finished 2024-04-14T15:13:36.537 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:36 smithi090 ceph-mon[30399]: osdmap e148: 8 total, 8 up, 8 in 2024-04-14T15:13:36.537 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:36 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:36.537 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:36 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:36.537 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:36 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2918742503' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi090-93709-15"}]: dispatch 2024-04-14T15:13:36.537 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:36 smithi090 ceph-mon[30399]: from='client.48067 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi090-93709-15"}]: dispatch 2024-04-14T15:13:36.537 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:36 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2918742503' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi090-93709-15"}]: dispatch 2024-04-14T15:13:36.537 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:36 smithi090 ceph-mon[30399]: from='client.48067 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi090-93709-15"}]: dispatch 2024-04-14T15:13:36.537 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:36 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2918742503' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi090-93709-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:13:36.537 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:36 smithi090 ceph-mon[30399]: from='client.48067 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi090-93709-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:13:36.546 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:36.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:36.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:13:36.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:36 smithi092 ceph-mon[33966]: 2.c deep-scrub starts 2024-04-14T15:13:36.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:36 smithi092 ceph-mon[33966]: 2.c deep-scrub ok 2024-04-14T15:13:36.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:36 smithi092 ceph-mon[33966]: pgmap v167: 452 pgs: 128 creating+peering, 324 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 18 MiB/s wr, 6 op/s 2024-04-14T15:13:36.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:36 smithi092 ceph-mon[33966]: 4.4 scrub starts 2024-04-14T15:13:36.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:36 smithi092 ceph-mon[33966]: 4.4 scrub ok 2024-04-14T15:13:36.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:36 smithi092 ceph-mon[33966]: 5.5 deep-scrub starts 2024-04-14T15:13:36.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:36 smithi092 ceph-mon[33966]: 5.5 deep-scrub ok 2024-04-14T15:13:36.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:36 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:36.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:36 smithi092 ceph-mon[33966]: from='client.38086 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi090-93709-10"}]': finished 2024-04-14T15:13:36.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:36 smithi092 ceph-mon[33966]: osdmap e148: 8 total, 8 up, 8 in 2024-04-14T15:13:36.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:36 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:36.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:36 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:36.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:36 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2918742503' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi090-93709-15"}]: dispatch 2024-04-14T15:13:36.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:36 smithi092 ceph-mon[33966]: from='client.48067 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi090-93709-15"}]: dispatch 2024-04-14T15:13:36.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:36 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2918742503' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi090-93709-15"}]: dispatch 2024-04-14T15:13:36.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:36 smithi092 ceph-mon[33966]: from='client.48067 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi090-93709-15"}]: dispatch 2024-04-14T15:13:36.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:36 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2918742503' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi090-93709-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:13:36.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:36 smithi092 ceph-mon[33966]: from='client.48067 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi090-93709-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:13:36.648 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:13:36.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:36.674 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:36.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:13:36.775 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:13:36.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:36.801 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:36.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:13:36.902 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:13:36.926 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:36.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:36.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:13:37.028 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:13:37.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:37.053 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:37.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:13:37.154 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:13:37.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:37.178 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:37.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:13:37.279 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:13:37.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:37.304 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:37.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:13:37.405 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:13:37.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:37.429 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:37.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:13:37.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:37 smithi090 ceph-mon[41670]: 4.13 scrub starts 2024-04-14T15:13:37.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:37 smithi090 ceph-mon[41670]: 4.13 scrub ok 2024-04-14T15:13:37.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:37 smithi090 ceph-mon[41670]: 6.15 scrub starts 2024-04-14T15:13:37.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:37 smithi090 ceph-mon[41670]: 6.15 scrub ok 2024-04-14T15:13:37.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:37 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:37.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:37 smithi090 ceph-mon[41670]: 6.a scrub starts 2024-04-14T15:13:37.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:37 smithi090 ceph-mon[41670]: 6.a scrub ok 2024-04-14T15:13:37.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:37 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:37.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:37 smithi090 ceph-mon[41670]: from='client.48067 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi090-93709-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:13:37.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:37 smithi090 ceph-mon[41670]: osdmap e149: 8 total, 8 up, 8 in 2024-04-14T15:13:37.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:37 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2514486846' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi090-93363-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:37.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:37 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2114226354' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi090-93342-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:37.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:37 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2918742503' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi090-93709-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi090-93709-15"}]: dispatch 2024-04-14T15:13:37.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:37 smithi090 ceph-mon[41670]: from='client.48067 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi090-93709-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi090-93709-15"}]: dispatch 2024-04-14T15:13:37.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:37 smithi090 ceph-mon[30399]: 4.13 scrub starts 2024-04-14T15:13:37.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:37 smithi090 ceph-mon[30399]: 4.13 scrub ok 2024-04-14T15:13:37.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:37 smithi090 ceph-mon[30399]: 6.15 scrub starts 2024-04-14T15:13:37.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:37 smithi090 ceph-mon[30399]: 6.15 scrub ok 2024-04-14T15:13:37.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:37 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:37.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:37 smithi090 ceph-mon[30399]: 6.a scrub starts 2024-04-14T15:13:37.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:37 smithi090 ceph-mon[30399]: 6.a scrub ok 2024-04-14T15:13:37.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:37 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:37.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:37 smithi090 ceph-mon[30399]: from='client.48067 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi090-93709-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:13:37.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:37 smithi090 ceph-mon[30399]: osdmap e149: 8 total, 8 up, 8 in 2024-04-14T15:13:37.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:37 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2514486846' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi090-93363-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:37.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:37 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2114226354' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi090-93342-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:37.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:37 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2918742503' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi090-93709-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi090-93709-15"}]: dispatch 2024-04-14T15:13:37.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:37 smithi090 ceph-mon[30399]: from='client.48067 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi090-93709-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi090-93709-15"}]: dispatch 2024-04-14T15:13:37.469 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-14T15:13:37.529 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:13:37.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:37.555 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:37.555 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:13:37.555 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:13:37.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:37 smithi092 ceph-mon[33966]: 4.13 scrub starts 2024-04-14T15:13:37.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:37 smithi092 ceph-mon[33966]: 4.13 scrub ok 2024-04-14T15:13:37.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:37 smithi092 ceph-mon[33966]: 6.15 scrub starts 2024-04-14T15:13:37.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:37 smithi092 ceph-mon[33966]: 6.15 scrub ok 2024-04-14T15:13:37.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:37 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:37.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:37 smithi092 ceph-mon[33966]: 6.a scrub starts 2024-04-14T15:13:37.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:37 smithi092 ceph-mon[33966]: 6.a scrub ok 2024-04-14T15:13:37.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:37 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:37.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:37 smithi092 ceph-mon[33966]: from='client.48067 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi090-93709-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:13:37.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:37 smithi092 ceph-mon[33966]: osdmap e149: 8 total, 8 up, 8 in 2024-04-14T15:13:37.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:37 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2514486846' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi090-93363-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:37.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:37 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2114226354' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi090-93342-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:37.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:37 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2918742503' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi090-93709-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi090-93709-15"}]: dispatch 2024-04-14T15:13:37.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:37 smithi092 ceph-mon[33966]: from='client.48067 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi090-93709-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi090-93709-15"}]: dispatch 2024-04-14T15:13:37.657 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:13:37.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:37.684 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:37.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:13:37.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:37.768+0000 7faf6fe0c700 1 -- 172.21.15.90:0/1278505200 >> v1:172.21.15.90:6789/0 conn(0x7faf6810b720 legacy=0x7faf6810db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:37.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:37.768+0000 7faf6fe0c700 1 -- 172.21.15.90:0/1278505200 shutdown_connections 2024-04-14T15:13:37.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:37.768+0000 7faf6fe0c700 1 -- 172.21.15.90:0/1278505200 >> 172.21.15.90:0/1278505200 conn(0x7faf680fe3a0 msgr2=0x7faf681007c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:13:37.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:37.769+0000 7faf6fe0c700 1 -- 172.21.15.90:0/1278505200 shutdown_connections 2024-04-14T15:13:37.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:37.769+0000 7faf6fe0c700 1 -- 172.21.15.90:0/1278505200 wait complete. 2024-04-14T15:13:37.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:37.769+0000 7faf6fe0c700 1 librados: starting msgr at 2024-04-14T15:13:37.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:37.769+0000 7faf6fe0c700 1 librados: starting objecter 2024-04-14T15:13:37.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:37.769+0000 7faf6fe0c700 1 Processor -- start 2024-04-14T15:13:37.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:37.769+0000 7faf6fe0c700 1 -- start start 2024-04-14T15:13:37.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:37.769+0000 7faf6fe0c700 1 librados: setting wanted keys 2024-04-14T15:13:37.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:37.769+0000 7faf6fe0c700 1 librados: calling monclient init 2024-04-14T15:13:37.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:37.769+0000 7faf6fe0c700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7faf681a79c0 con 0x7faf6810f2f0 2024-04-14T15:13:37.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:37.769+0000 7faf6fe0c700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7faf681a8be0 con 0x7faf68108870 2024-04-14T15:13:37.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:37.769+0000 7faf6fe0c700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7faf681a9e20 con 0x7faf6810b720 2024-04-14T15:13:37.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:37.769+0000 7faf6dba8700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7faf68108870 0x7faf6810eae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.90:51786/0 (socket says 172.21.15.90:51786) 2024-04-14T15:13:37.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:37.769+0000 7faf6dba8700 1 -- 172.21.15.90:0/134893663 learned_addr learned my addr 172.21.15.90:0/134893663 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:13:37.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:37.770+0000 7faf5effd700 10 client.?.objecter ms_handle_connect 0x7faf68108870 2024-04-14T15:13:37.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:37.770+0000 7faf5effd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:13:37.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:37.770+0000 7faf5effd700 10 client.?.objecter ms_handle_connect 0x7faf6810f2f0 2024-04-14T15:13:37.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:37.770+0000 7faf5effd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:13:37.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:37.770+0000 7faf5effd700 10 client.?.objecter ms_handle_connect 0x7faf6810b720 2024-04-14T15:13:37.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:37.770+0000 7faf5effd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:13:37.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:37.770+0000 7faf5effd700 1 -- 172.21.15.90:0/134893663 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 17902292 0 0) 0x7faf681a8be0 con 0x7faf68108870 2024-04-14T15:13:37.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:37.770+0000 7faf5effd700 1 -- 172.21.15.90:0/134893663 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faf540031f0 con 0x7faf68108870 2024-04-14T15:13:37.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:37.770+0000 7faf5effd700 1 -- 172.21.15.90:0/134893663 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 889370200 0 0) 0x7faf681a79c0 con 0x7faf6810f2f0 2024-04-14T15:13:37.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:37.770+0000 7faf5effd700 1 -- 172.21.15.90:0/134893663 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faf681a8be0 con 0x7faf6810f2f0 2024-04-14T15:13:37.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:37.770+0000 7faf5effd700 1 -- 172.21.15.90:0/134893663 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 803034561 0 0) 0x7faf681a9e20 con 0x7faf6810b720 2024-04-14T15:13:37.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:37.770+0000 7faf5effd700 1 -- 172.21.15.90:0/134893663 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faf681a79c0 con 0x7faf6810b720 2024-04-14T15:13:37.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:37.770+0000 7faf5effd700 1 -- 172.21.15.90:0/134893663 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1843725146 0 0) 0x7faf540031f0 con 0x7faf68108870 2024-04-14T15:13:37.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:37.770+0000 7faf5effd700 1 -- 172.21.15.90:0/134893663 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faf681a9e20 con 0x7faf68108870 2024-04-14T15:13:37.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:37.770+0000 7faf5effd700 1 -- 172.21.15.90:0/134893663 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7faf64001a30 con 0x7faf68108870 2024-04-14T15:13:37.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:37.770+0000 7faf5effd700 1 -- 172.21.15.90:0/134893663 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1244568998 0 0) 0x7faf681a8be0 con 0x7faf6810f2f0 2024-04-14T15:13:37.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:37.770+0000 7faf5effd700 1 -- 172.21.15.90:0/134893663 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faf540031f0 con 0x7faf6810f2f0 2024-04-14T15:13:37.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:37.770+0000 7faf5effd700 1 -- 172.21.15.90:0/134893663 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7faf60001b30 con 0x7faf6810f2f0 2024-04-14T15:13:37.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:37.770+0000 7faf5effd700 1 -- 172.21.15.90:0/134893663 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4270017906 0 0) 0x7faf681a79c0 con 0x7faf6810b720 2024-04-14T15:13:37.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:37.770+0000 7faf5effd700 1 -- 172.21.15.90:0/134893663 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faf681a8be0 con 0x7faf6810b720 2024-04-14T15:13:37.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:37.770+0000 7faf5effd700 1 -- 172.21.15.90:0/134893663 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7faf58004470 con 0x7faf6810b720 2024-04-14T15:13:37.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:37.770+0000 7faf5effd700 1 -- 172.21.15.90:0/134893663 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2580630960 0 0) 0x7faf681a9e20 con 0x7faf68108870 2024-04-14T15:13:37.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:37.770+0000 7faf5effd700 1 -- 172.21.15.90:0/134893663 >> v1:172.21.15.90:6790/0 conn(0x7faf6810b720 legacy=0x7faf681a27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:37.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:37.770+0000 7faf5effd700 1 -- 172.21.15.90:0/134893663 >> v1:172.21.15.90:6789/0 conn(0x7faf6810f2f0 legacy=0x7faf681a60c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:37.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:37.770+0000 7faf5effd700 1 -- 172.21.15.90:0/134893663 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf681ab060 con 0x7faf68108870 2024-04-14T15:13:37.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:37.770+0000 7faf6fe0c700 1 -- 172.21.15.90:0/134893663 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7faf681a7bf0 con 0x7faf68108870 2024-04-14T15:13:37.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:37.770+0000 7faf6fe0c700 10 client.48082.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:13:37.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:37.770+0000 7faf6fe0c700 1 -- 172.21.15.90:0/134893663 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7faf681a80b0 con 0x7faf68108870 2024-04-14T15:13:37.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:37.770+0000 7faf6fe0c700 1 librados: init done 2024-04-14T15:13:37.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:37.770+0000 7faf5effd700 1 -- 172.21.15.90:0/134893663 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7faf64001ce0 con 0x7faf68108870 2024-04-14T15:13:37.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:37.770+0000 7faf5effd700 1 -- 172.21.15.90:0/134893663 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7faf64005db0 con 0x7faf68108870 2024-04-14T15:13:37.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:37.771+0000 7faf6fe0c700 1 -- 172.21.15.90:0/134893663 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf4c005120 con 0x7faf68108870 2024-04-14T15:13:37.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:37.772+0000 7faf5effd700 1 -- 172.21.15.90:0/134893663 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7faf64003930 con 0x7faf68108870 2024-04-14T15:13:37.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:37.772+0000 7faf5effd700 1 -- 172.21.15.90:0/134893663 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(149..149 src has 51..149) ==== 10527+0+0 (unknown 2334321141 0 0) 0x7faf6408b520 con 0x7faf68108870 2024-04-14T15:13:37.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:37.772+0000 7faf5effd700 10 client.48082.objecter ms_dispatch 0x7faf68001550 osd_map(149..149 src has 51..149) 2024-04-14T15:13:37.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:37.772+0000 7faf5effd700 3 client.48082.objecter handle_osd_map got epochs [149,149] > 0 2024-04-14T15:13:37.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:37.772+0000 7faf5effd700 3 client.48082.objecter handle_osd_map decoding full epoch 149 2024-04-14T15:13:37.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:37.772+0000 7faf5effd700 20 client.48082.objecter dump_active .. 0 homeless 2024-04-14T15:13:37.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:37.772+0000 7faf5effd700 10 client.48082.objecter ms_handle_connect 0x7faf54071340 2024-04-14T15:13:37.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:37.777+0000 7faf5effd700 1 -- 172.21.15.90:0/134893663 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7faf640575f0 con 0x7faf68108870 2024-04-14T15:13:37.786 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:13:37.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:37.811 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:37.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:13:37.912 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:13:37.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:37.940 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:37.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:13:38.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:38.038+0000 7faf6fe0c700 1 -- 172.21.15.90:0/134893663 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7faf4c005410 con 0x7faf68108870 2024-04-14T15:13:38.042 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:13:38.065 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:38.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:38.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:13:38.169 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:13:38.196 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:13:38.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:38.197 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:38.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:13:38.201 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:13:38.297 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:13:38.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:38.322 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:38.322 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:13:38.423 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:13:38.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:38.448 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:38.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:13:38.549 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:13:38.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:38.574 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:38.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:13:38.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:38 smithi092 ceph-mon[33966]: pgmap v170: 420 pgs: 96 unknown, 324 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 18 MiB/s wr, 6 op/s 2024-04-14T15:13:38.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:38 smithi092 ceph-mon[33966]: 4.16 scrub starts 2024-04-14T15:13:38.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:38 smithi092 ceph-mon[33966]: 4.16 scrub ok 2024-04-14T15:13:38.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:38 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:38.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:38 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2514486846' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi090-93363-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:38.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:38 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2114226354' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi090-93342-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:38.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:38 smithi092 ceph-mon[33966]: osdmap e150: 8 total, 8 up, 8 in 2024-04-14T15:13:38.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:38 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/134893663' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:13:38.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:38 smithi092 ceph-mon[33966]: from='client.48082 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:13:38.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:38 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:13:38.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:38 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:13:38.641 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:38 smithi090 ceph-mon[41670]: pgmap v170: 420 pgs: 96 unknown, 324 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 18 MiB/s wr, 6 op/s 2024-04-14T15:13:38.641 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:38 smithi090 ceph-mon[41670]: 4.16 scrub starts 2024-04-14T15:13:38.641 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:38 smithi090 ceph-mon[41670]: 4.16 scrub ok 2024-04-14T15:13:38.641 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:38 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:38.642 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:38 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2514486846' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi090-93363-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:38.642 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:38 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2114226354' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi090-93342-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:38.642 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:38 smithi090 ceph-mon[41670]: osdmap e150: 8 total, 8 up, 8 in 2024-04-14T15:13:38.642 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:38 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/134893663' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:13:38.642 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:38 smithi090 ceph-mon[41670]: from='client.48082 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:13:38.642 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:38 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:13:38.642 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:38 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:13:38.642 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:38 smithi090 ceph-mon[30399]: pgmap v170: 420 pgs: 96 unknown, 324 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 18 MiB/s wr, 6 op/s 2024-04-14T15:13:38.642 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:38 smithi090 ceph-mon[30399]: 4.16 scrub starts 2024-04-14T15:13:38.642 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:38 smithi090 ceph-mon[30399]: 4.16 scrub ok 2024-04-14T15:13:38.642 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:38 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:38.643 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:38 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2514486846' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi090-93363-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:38.643 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:38 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2114226354' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi090-93342-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:38.643 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:38 smithi090 ceph-mon[30399]: osdmap e150: 8 total, 8 up, 8 in 2024-04-14T15:13:38.643 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:38 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/134893663' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:13:38.643 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:38 smithi090 ceph-mon[30399]: from='client.48082 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:13:38.643 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:38 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:13:38.643 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:38 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:13:38.675 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:13:38.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:38.700 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:38.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:13:38.801 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:13:38.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:38.825+0000 7faf5effd700 1 -- 172.21.15.90:0/134893663 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v151) ==== 87+0+0 (unknown 1087130217 0 0) 0x7faf6405b2a0 con 0x7faf68108870 2024-04-14T15:13:38.827 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:38.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:38.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:13:38.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:38.830+0000 7faf6fe0c700 10 librados: watch_flush enter 2024-04-14T15:13:38.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:38.831+0000 7faf6fe0c700 10 librados: watch_flush exit 2024-04-14T15:13:38.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:38.831+0000 7faf6fe0c700 20 client.48082.objecter shutdown clearing up homeless session... 2024-04-14T15:13:38.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:38.831+0000 7faf6fe0c700 10 client.48082.objecter successfully canceled tick 2024-04-14T15:13:38.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:38.832+0000 7faf6fe0c700 1 -- 172.21.15.90:0/134893663 >> v1:172.21.15.90:6800/420935427 conn(0x7faf54071340 legacy=0x7faf54073800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:38.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:38.832+0000 7faf6fe0c700 1 -- 172.21.15.90:0/134893663 >> v1:172.21.15.92:6789/0 conn(0x7faf68108870 legacy=0x7faf6810eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:38.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:38.833+0000 7faf6fe0c700 1 -- 172.21.15.90:0/134893663 shutdown_connections 2024-04-14T15:13:38.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:38.833+0000 7faf6fe0c700 1 -- 172.21.15.90:0/134893663 >> 172.21.15.90:0/134893663 conn(0x7faf680fe3a0 msgr2=0x7faf68101520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:13:38.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:38.843+0000 7faf6fe0c700 1 -- 172.21.15.90:0/134893663 shutdown_connections 2024-04-14T15:13:38.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:38.843+0000 7faf6fe0c700 1 -- 172.21.15.90:0/134893663 wait complete. 2024-04-14T15:13:38.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:38.843+0000 7faf6fe0c700 1 librados: shutdown 2024-04-14T15:13:38.846 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is set 2024-04-14T15:13:38.929 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:13:38.955 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:38.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:38.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:13:39.057 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:13:39.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:39.081 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:39.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:13:39.182 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:13:39.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:39.208 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:39.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:13:39.309 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:13:39.334 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:39.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:39.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:13:39.436 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:13:39.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:39.460 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:39.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:13:39.561 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:13:39.585 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:39.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:39.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:13:39.592 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:39 smithi090 ceph-mon[41670]: 5.11 scrub starts 2024-04-14T15:13:39.592 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:39 smithi090 ceph-mon[41670]: 5.11 scrub ok 2024-04-14T15:13:39.593 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:39 smithi090 ceph-mon[41670]: 5.0 scrub starts 2024-04-14T15:13:39.593 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:39 smithi090 ceph-mon[41670]: 5.0 scrub ok 2024-04-14T15:13:39.593 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:39 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:39.593 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:39 smithi090 ceph-mon[41670]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:13:39.593 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:39 smithi090 ceph-mon[41670]: from='client.48067 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi090-93709-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi090-93709-15"}]': finished 2024-04-14T15:13:39.593 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:39 smithi090 ceph-mon[41670]: from='client.48082 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-14T15:13:39.593 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:39 smithi090 ceph-mon[41670]: osdmap e151: 8 total, 8 up, 8 in 2024-04-14T15:13:39.593 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:39 smithi090 ceph-mon[30399]: 5.11 scrub starts 2024-04-14T15:13:39.593 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:39 smithi090 ceph-mon[30399]: 5.11 scrub ok 2024-04-14T15:13:39.593 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:39 smithi090 ceph-mon[30399]: 5.0 scrub starts 2024-04-14T15:13:39.593 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:39 smithi090 ceph-mon[30399]: 5.0 scrub ok 2024-04-14T15:13:39.593 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:39 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:39.593 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:39 smithi090 ceph-mon[30399]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:13:39.594 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:39 smithi090 ceph-mon[30399]: from='client.48067 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi090-93709-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi090-93709-15"}]': finished 2024-04-14T15:13:39.594 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:39 smithi090 ceph-mon[30399]: from='client.48082 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-14T15:13:39.594 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:39 smithi090 ceph-mon[30399]: osdmap e151: 8 total, 8 up, 8 in 2024-04-14T15:13:39.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:39 smithi092 ceph-mon[33966]: 5.11 scrub starts 2024-04-14T15:13:39.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:39 smithi092 ceph-mon[33966]: 5.11 scrub ok 2024-04-14T15:13:39.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:39 smithi092 ceph-mon[33966]: 5.0 scrub starts 2024-04-14T15:13:39.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:39 smithi092 ceph-mon[33966]: 5.0 scrub ok 2024-04-14T15:13:39.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:39 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:39.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:39 smithi092 ceph-mon[33966]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:13:39.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:39 smithi092 ceph-mon[33966]: from='client.48067 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi090-93709-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi090-93709-15"}]': finished 2024-04-14T15:13:39.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:39 smithi092 ceph-mon[33966]: from='client.48082 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-14T15:13:39.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:39 smithi092 ceph-mon[33966]: osdmap e151: 8 total, 8 up, 8 in 2024-04-14T15:13:39.687 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:13:39.711 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:39.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:39.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:13:39.812 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:13:39.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:39.845 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:39.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:13:39.946 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:13:39.970 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:39.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:39.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:13:40.072 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:13:40.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:40.100 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:40.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:13:40.201 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:13:40.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:40.228 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:40.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:13:40.330 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:13:40.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:40.357 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:40.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:13:40.458 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:13:40.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:40.483 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:40.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:13:40.527 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:40 smithi090 ceph-mon[30399]: pgmap v173: 332 pgs: 4 creating+peering, 17 unknown, 311 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 25 KiB/s wr, 25 op/s 2024-04-14T15:13:40.528 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:40 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:40.528 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:40 smithi090 ceph-mon[30399]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:13:40.528 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:40 smithi090 ceph-mon[30399]: osdmap e152: 8 total, 8 up, 8 in 2024-04-14T15:13:40.528 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:40 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1502470276' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi090-93342-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:40.528 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:40 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1648016781' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi090-93363-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:40.528 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:40 smithi090 ceph-mon[30399]: from='client.48094 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi090-93342-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:40.528 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:40 smithi090 ceph-mon[30399]: from='client.48097 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi090-93363-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:40.528 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:40 smithi090 ceph-mon[41670]: pgmap v173: 332 pgs: 4 creating+peering, 17 unknown, 311 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 25 KiB/s wr, 25 op/s 2024-04-14T15:13:40.528 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:40 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:40.528 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:40 smithi090 ceph-mon[41670]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:13:40.529 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:40 smithi090 ceph-mon[41670]: osdmap e152: 8 total, 8 up, 8 in 2024-04-14T15:13:40.529 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:40 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1502470276' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi090-93342-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:40.529 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:40 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1648016781' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi090-93363-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:40.529 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:40 smithi090 ceph-mon[41670]: from='client.48094 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi090-93342-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:40.529 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:40 smithi090 ceph-mon[41670]: from='client.48097 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi090-93363-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:40.585 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:13:40.609 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:40.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:40.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:13:40.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:40 smithi092 ceph-mon[33966]: pgmap v173: 332 pgs: 4 creating+peering, 17 unknown, 311 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 25 KiB/s wr, 25 op/s 2024-04-14T15:13:40.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:40 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:40.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:40 smithi092 ceph-mon[33966]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:13:40.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:40 smithi092 ceph-mon[33966]: osdmap e152: 8 total, 8 up, 8 in 2024-04-14T15:13:40.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:40 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1502470276' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi090-93342-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:40.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:40 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1648016781' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi090-93363-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:40.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:40 smithi092 ceph-mon[33966]: from='client.48094 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi090-93342-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:40.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:40 smithi092 ceph-mon[33966]: from='client.48097 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi090-93363-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:40.711 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:13:40.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:40.735 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:40.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:13:40.837 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:13:40.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:40.880 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:40.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:13:40.981 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:13:41.004 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:41.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:41.005 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:13:41.106 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:13:41.130 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:41.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:41.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:13:41.232 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:13:41.256 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:41.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:41.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:13:41.358 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:13:41.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:41.383 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:41.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:13:41.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:41 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:41.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:41 smithi090 ceph-mon[30399]: from='client.48094 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi090-93342-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:41.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:41 smithi090 ceph-mon[30399]: from='client.48097 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi090-93363-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:41.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:41 smithi090 ceph-mon[30399]: osdmap e153: 8 total, 8 up, 8 in 2024-04-14T15:13:41.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:41 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:13:41.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:41 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:13:41.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:41 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:13:41.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:41 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:13:41.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:41 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:41.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:41 smithi090 ceph-mon[41670]: from='client.48094 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi090-93342-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:41.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:41 smithi090 ceph-mon[41670]: from='client.48097 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi090-93363-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:41.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:41 smithi090 ceph-mon[41670]: osdmap e153: 8 total, 8 up, 8 in 2024-04-14T15:13:41.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:41 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:13:41.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:41 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:13:41.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:41 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:13:41.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:41 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:13:41.484 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:13:41.508 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:41.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:41.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:13:41.611 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:13:41.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:41 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:41.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:41 smithi092 ceph-mon[33966]: from='client.48094 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi090-93342-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:41.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:41 smithi092 ceph-mon[33966]: from='client.48097 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi090-93363-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:41.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:41 smithi092 ceph-mon[33966]: osdmap e153: 8 total, 8 up, 8 in 2024-04-14T15:13:41.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:41 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:13:41.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:41 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:13:41.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:41 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:13:41.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:41 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:13:41.634 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:41.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:41.636 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:13:41.736 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:13:41.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:41.761 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:41.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:13:41.862 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:13:41.886 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:41.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:41.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:13:41.988 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:13:42.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:42.012 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:42.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:13:42.113 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:13:42.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:42.137 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:42.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:13:42.219 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-14T15:13:42.238 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:13:42.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:42.263 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:42.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:13:42.364 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:13:42.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:42.389 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:42.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:13:42.490 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:13:42.501 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:42 smithi090 ceph-mon[41670]: pgmap v176: 428 pgs: 96 unknown, 8 creating+peering, 324 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 67 KiB/s wr, 68 op/s 2024-04-14T15:13:42.501 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:42 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:42.501 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:42 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:13:42.501 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:42 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:13:42.501 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:42 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:13:42.501 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:42 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-13", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:13:42.501 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:42 smithi090 ceph-mon[41670]: osdmap e154: 8 total, 8 up, 8 in 2024-04-14T15:13:42.501 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:42 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-13"}]: dispatch 2024-04-14T15:13:42.501 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:42 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-13"}]: dispatch 2024-04-14T15:13:42.502 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:42 smithi090 ceph-mon[30399]: pgmap v176: 428 pgs: 96 unknown, 8 creating+peering, 324 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 67 KiB/s wr, 68 op/s 2024-04-14T15:13:42.502 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:42 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:42.502 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:42 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:13:42.502 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:42 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:13:42.502 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:42 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:13:42.502 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:42 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-13", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:13:42.502 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:42 smithi090 ceph-mon[30399]: osdmap e154: 8 total, 8 up, 8 in 2024-04-14T15:13:42.502 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:42 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-13"}]: dispatch 2024-04-14T15:13:42.502 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:42 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-13"}]: dispatch 2024-04-14T15:13:42.516 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:42.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:42.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:13:42.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.606+0000 7f717f68a700 1 -- 172.21.15.90:0/1742243844 >> v1:172.21.15.90:6789/0 conn(0x7f717810a630 legacy=0x7f717810ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:42.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.606+0000 7f717f68a700 1 -- 172.21.15.90:0/1742243844 shutdown_connections 2024-04-14T15:13:42.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.606+0000 7f717f68a700 1 -- 172.21.15.90:0/1742243844 >> 172.21.15.90:0/1742243844 conn(0x7f71780fd2b0 msgr2=0x7f71780ff6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:13:42.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.607+0000 7f717f68a700 1 -- 172.21.15.90:0/1742243844 shutdown_connections 2024-04-14T15:13:42.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.607+0000 7f717f68a700 1 -- 172.21.15.90:0/1742243844 wait complete. 2024-04-14T15:13:42.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.607+0000 7f717f68a700 1 librados: starting msgr at 2024-04-14T15:13:42.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.607+0000 7f717f68a700 1 librados: starting objecter 2024-04-14T15:13:42.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.607+0000 7f717f68a700 1 Processor -- start 2024-04-14T15:13:42.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.607+0000 7f717f68a700 1 -- start start 2024-04-14T15:13:42.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.607+0000 7f717f68a700 1 librados: setting wanted keys 2024-04-14T15:13:42.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.607+0000 7f717f68a700 1 librados: calling monclient init 2024-04-14T15:13:42.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.607+0000 7f717f68a700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7178110540 con 0x7f717810a630 2024-04-14T15:13:42.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.607+0000 7f717f68a700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f71781ab210 con 0x7f7178107780 2024-04-14T15:13:42.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.607+0000 7f717f68a700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f71781ac3f0 con 0x7f717810e200 2024-04-14T15:13:42.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.607+0000 7f717cc25700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f717810a630 0x7f717810ee90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:41362/0 (socket says 172.21.15.90:41362) 2024-04-14T15:13:42.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.607+0000 7f717cc25700 1 -- 172.21.15.90:0/145892757 learned_addr learned my addr 172.21.15.90:0/145892757 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:13:42.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.608+0000 7f716e7fc700 10 client.?.objecter ms_handle_connect 0x7f717810a630 2024-04-14T15:13:42.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.608+0000 7f716e7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:13:42.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.608+0000 7f716e7fc700 10 client.?.objecter ms_handle_connect 0x7f717810e200 2024-04-14T15:13:42.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.608+0000 7f716e7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:13:42.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.608+0000 7f716e7fc700 10 client.?.objecter ms_handle_connect 0x7f7178107780 2024-04-14T15:13:42.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.608+0000 7f716e7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:13:42.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.608+0000 7f716e7fc700 1 -- 172.21.15.90:0/145892757 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 983961712 0 0) 0x7f7178110540 con 0x7f717810a630 2024-04-14T15:13:42.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.608+0000 7f716e7fc700 1 -- 172.21.15.90:0/145892757 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71640035b0 con 0x7f717810a630 2024-04-14T15:13:42.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.608+0000 7f716e7fc700 1 -- 172.21.15.90:0/145892757 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1936091308 0 0) 0x7f71781ac3f0 con 0x7f717810e200 2024-04-14T15:13:42.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.608+0000 7f716e7fc700 1 -- 172.21.15.90:0/145892757 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7178110540 con 0x7f717810e200 2024-04-14T15:13:42.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.608+0000 7f716e7fc700 1 -- 172.21.15.90:0/145892757 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3460805377 0 0) 0x7f71781ab210 con 0x7f7178107780 2024-04-14T15:13:42.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.608+0000 7f716e7fc700 1 -- 172.21.15.90:0/145892757 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71781ac3f0 con 0x7f7178107780 2024-04-14T15:13:42.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.608+0000 7f716e7fc700 1 -- 172.21.15.90:0/145892757 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 477367368 0 0) 0x7f71781ac3f0 con 0x7f7178107780 2024-04-14T15:13:42.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.608+0000 7f716e7fc700 1 -- 172.21.15.90:0/145892757 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f71781ab210 con 0x7f7178107780 2024-04-14T15:13:42.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.608+0000 7f716e7fc700 1 -- 172.21.15.90:0/145892757 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f71740018b0 con 0x7f7178107780 2024-04-14T15:13:42.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.608+0000 7f716e7fc700 1 -- 172.21.15.90:0/145892757 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1316569922 0 0) 0x7f71640035b0 con 0x7f717810a630 2024-04-14T15:13:42.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.608+0000 7f716e7fc700 1 -- 172.21.15.90:0/145892757 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f71781ac3f0 con 0x7f717810a630 2024-04-14T15:13:42.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.608+0000 7f716e7fc700 1 -- 172.21.15.90:0/145892757 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 89019490 0 0) 0x7f7178110540 con 0x7f717810e200 2024-04-14T15:13:42.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.608+0000 7f716e7fc700 1 -- 172.21.15.90:0/145892757 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f71640035b0 con 0x7f717810e200 2024-04-14T15:13:42.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.608+0000 7f716e7fc700 1 -- 172.21.15.90:0/145892757 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f71680015d0 con 0x7f717810a630 2024-04-14T15:13:42.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.608+0000 7f716e7fc700 1 -- 172.21.15.90:0/145892757 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f7170001bb0 con 0x7f717810e200 2024-04-14T15:13:42.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.608+0000 7f716e7fc700 1 -- 172.21.15.90:0/145892757 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4102536223 0 0) 0x7f71781ab210 con 0x7f7178107780 2024-04-14T15:13:42.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.608+0000 7f716e7fc700 1 -- 172.21.15.90:0/145892757 >> v1:172.21.15.90:6790/0 conn(0x7f717810e200 legacy=0x7f71781a8ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:42.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.608+0000 7f716e7fc700 1 -- 172.21.15.90:0/145892757 >> v1:172.21.15.90:6789/0 conn(0x7f717810a630 legacy=0x7f717810ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:42.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.608+0000 7f716e7fc700 1 -- 172.21.15.90:0/145892757 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71781ad5d0 con 0x7f7178107780 2024-04-14T15:13:42.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.608+0000 7f717f68a700 1 -- 172.21.15.90:0/145892757 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f71781ac620 con 0x7f7178107780 2024-04-14T15:13:42.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.608+0000 7f717f68a700 10 client.48112.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:13:42.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.608+0000 7f717f68a700 1 -- 172.21.15.90:0/145892757 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f71781acae0 con 0x7f7178107780 2024-04-14T15:13:42.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.608+0000 7f717f68a700 1 librados: init done 2024-04-14T15:13:42.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.608+0000 7f716e7fc700 1 -- 172.21.15.90:0/145892757 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f7174003b40 con 0x7f7178107780 2024-04-14T15:13:42.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.608+0000 7f716e7fc700 1 -- 172.21.15.90:0/145892757 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f7174004950 con 0x7f7178107780 2024-04-14T15:13:42.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.609+0000 7f717f68a700 1 -- 172.21.15.90:0/145892757 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f715c005150 con 0x7f7178107780 2024-04-14T15:13:42.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.610+0000 7f716e7fc700 1 -- 172.21.15.90:0/145892757 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f71740036a0 con 0x7f7178107780 2024-04-14T15:13:42.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.610+0000 7f716e7fc700 1 -- 172.21.15.90:0/145892757 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(154..154 src has 51..154) ==== 10320+0+0 (unknown 479422130 0 0) 0x7f7174089f90 con 0x7f7178107780 2024-04-14T15:13:42.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.610+0000 7f716e7fc700 10 client.48112.objecter ms_dispatch 0x7f7178001550 osd_map(154..154 src has 51..154) 2024-04-14T15:13:42.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.610+0000 7f716e7fc700 3 client.48112.objecter handle_osd_map got epochs [154,154] > 0 2024-04-14T15:13:42.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.610+0000 7f716e7fc700 3 client.48112.objecter handle_osd_map decoding full epoch 154 2024-04-14T15:13:42.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.611+0000 7f716e7fc700 20 client.48112.objecter dump_active .. 0 homeless 2024-04-14T15:13:42.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.611+0000 7f716e7fc700 10 client.48112.objecter ms_handle_connect 0x7f7164071480 2024-04-14T15:13:42.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.615+0000 7f716e7fc700 1 -- 172.21.15.90:0/145892757 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f7174056130 con 0x7f7178107780 2024-04-14T15:13:42.618 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:13:42.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:42 smithi092 ceph-mon[33966]: pgmap v176: 428 pgs: 96 unknown, 8 creating+peering, 324 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 67 KiB/s wr, 68 op/s 2024-04-14T15:13:42.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:42 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:42.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:42 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:13:42.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:42 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:13:42.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:42 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:13:42.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:42 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-13", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:13:42.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:42 smithi092 ceph-mon[33966]: osdmap e154: 8 total, 8 up, 8 in 2024-04-14T15:13:42.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:42 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-13"}]: dispatch 2024-04-14T15:13:42.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:42 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-13"}]: dispatch 2024-04-14T15:13:42.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:42.642 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:42.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:13:42.743 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:13:42.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:42.768 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:42.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:13:42.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.819+0000 7f717f68a700 1 -- 172.21.15.90:0/145892757 --> v1:172.21.15.90:6800/420935427 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f715c000bc0 con 0x7f7164071480 2024-04-14T15:13:42.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.858+0000 7f716e7fc700 1 -- 172.21.15.90:0/145892757 <== mgr.14610 v1:172.21.15.90:6800/420935427 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+931361 (unknown 2965378022 0 1100307475) 0x7f715c000bc0 con 0x7f7164071480 2024-04-14T15:13:42.870 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:13:42.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.864+0000 7f717f68a700 10 librados: watch_flush enter 2024-04-14T15:13:42.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.864+0000 7f717f68a700 10 librados: watch_flush exit 2024-04-14T15:13:42.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.864+0000 7f717f68a700 20 client.48112.objecter shutdown clearing up homeless session... 2024-04-14T15:13:42.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.864+0000 7f717f68a700 10 client.48112.objecter successfully canceled tick 2024-04-14T15:13:42.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.864+0000 7f717f68a700 1 -- 172.21.15.90:0/145892757 >> v1:172.21.15.90:6800/420935427 conn(0x7f7164071480 legacy=0x7f7164073940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:42.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.864+0000 7f717f68a700 1 -- 172.21.15.90:0/145892757 >> v1:172.21.15.92:6789/0 conn(0x7f7178107780 legacy=0x7f717810e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:42.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.864+0000 7f717f68a700 1 -- 172.21.15.90:0/145892757 shutdown_connections 2024-04-14T15:13:42.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.864+0000 7f717f68a700 1 -- 172.21.15.90:0/145892757 >> 172.21.15.90:0/145892757 conn(0x7f71780fd2b0 msgr2=0x7f7178111100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:13:42.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.864+0000 7f717f68a700 1 -- 172.21.15.90:0/145892757 shutdown_connections 2024-04-14T15:13:42.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.864+0000 7f717f68a700 1 -- 172.21.15.90:0/145892757 wait complete. 2024-04-14T15:13:42.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:42.864+0000 7f717f68a700 1 librados: shutdown 2024-04-14T15:13:42.889 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-14T15:13:42.889 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:13:42.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:42.913 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:42.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:13:43.015 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:13:43.040 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:43.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:43.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:13:43.142 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:13:43.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:43.165 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:43.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:13:43.266 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:13:43.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:43.291 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:43.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:13:43.392 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:13:43.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:43.418 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:43.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:13:43.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:43 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:43.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:43 smithi090 ceph-mon[30399]: from='client.48112 v1:172.21.15.90:0/145892757' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:13:43.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:43 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-13"}]': finished 2024-04-14T15:13:43.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:43 smithi090 ceph-mon[30399]: osdmap e155: 8 total, 8 up, 8 in 2024-04-14T15:13:43.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:43 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-13", "mode": "writeback"}]: dispatch 2024-04-14T15:13:43.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:43 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-13", "mode": "writeback"}]: dispatch 2024-04-14T15:13:43.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:43 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3598646543' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi090-93363-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:43.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:43 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/124194790' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi090-93342-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:43.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:43 smithi090 ceph-mon[30399]: from='client.48106 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi090-93342-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:43.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:43 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:43.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:43 smithi090 ceph-mon[41670]: from='client.48112 v1:172.21.15.90:0/145892757' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:13:43.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:43 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-13"}]': finished 2024-04-14T15:13:43.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:43 smithi090 ceph-mon[41670]: osdmap e155: 8 total, 8 up, 8 in 2024-04-14T15:13:43.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:43 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-13", "mode": "writeback"}]: dispatch 2024-04-14T15:13:43.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:43 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-13", "mode": "writeback"}]: dispatch 2024-04-14T15:13:43.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:43 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3598646543' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi090-93363-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:43.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:43 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/124194790' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi090-93342-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:43.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:43 smithi090 ceph-mon[41670]: from='client.48106 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi090-93342-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:43.519 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:13:43.543 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:43.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:43.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:13:43.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:43 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:43.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:43 smithi092 ceph-mon[33966]: from='client.48112 v1:172.21.15.90:0/145892757' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:13:43.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:43 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-13"}]': finished 2024-04-14T15:13:43.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:43 smithi092 ceph-mon[33966]: osdmap e155: 8 total, 8 up, 8 in 2024-04-14T15:13:43.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:43 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-13", "mode": "writeback"}]: dispatch 2024-04-14T15:13:43.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:43 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-13", "mode": "writeback"}]: dispatch 2024-04-14T15:13:43.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:43 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3598646543' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi090-93363-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:43.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:43 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/124194790' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi090-93342-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:43.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:43 smithi092 ceph-mon[33966]: from='client.48106 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi090-93342-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:43.645 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:13:43.669 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:43.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:43.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:13:43.771 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:13:43.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:43.796 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:43.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:13:43.897 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:13:43.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:43.923 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:43.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:13:43.981 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:13:44.026 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:13:44.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:44.051 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:44.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:13:44.152 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:13:44.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:44.177 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:44.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:13:44.263 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:13:43 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:13:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:13:44.278 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:13:44.302 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:44.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:44.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:13:44.404 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:13:44.432 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:44.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:44.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:13:44.516 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:44 smithi090 ceph-mon[41670]: pgmap v179: 428 pgs: 96 unknown, 8 creating+peering, 324 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 42 KiB/s wr, 43 op/s 2024-04-14T15:13:44.516 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:44 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:44.516 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:44 smithi090 ceph-mon[41670]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:13:44.516 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:44 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-13", "mode": "writeback"}]': finished 2024-04-14T15:13:44.517 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:44 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3598646543' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi090-93363-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:44.517 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:44 smithi090 ceph-mon[41670]: from='client.48106 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi090-93342-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:44.517 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:44 smithi090 ceph-mon[41670]: osdmap e156: 8 total, 8 up, 8 in 2024-04-14T15:13:44.517 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:44 smithi090 ceph-mon[30399]: pgmap v179: 428 pgs: 96 unknown, 8 creating+peering, 324 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 42 KiB/s wr, 43 op/s 2024-04-14T15:13:44.517 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:44 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:44.517 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:44 smithi090 ceph-mon[30399]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:13:44.517 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:44 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-13", "mode": "writeback"}]': finished 2024-04-14T15:13:44.517 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:44 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3598646543' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi090-93363-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:44.517 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:44 smithi090 ceph-mon[30399]: from='client.48106 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi090-93342-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:44.518 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:44 smithi090 ceph-mon[30399]: osdmap e156: 8 total, 8 up, 8 in 2024-04-14T15:13:44.534 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:13:44.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:44.558 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:44.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:13:44.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:44 smithi092 ceph-mon[33966]: pgmap v179: 428 pgs: 96 unknown, 8 creating+peering, 324 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 42 KiB/s wr, 43 op/s 2024-04-14T15:13:44.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:44 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:44.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:44 smithi092 ceph-mon[33966]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:13:44.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:44 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-13", "mode": "writeback"}]': finished 2024-04-14T15:13:44.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:44 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3598646543' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi090-93363-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:44.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:44 smithi092 ceph-mon[33966]: from='client.48106 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi090-93342-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:44.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:44 smithi092 ceph-mon[33966]: osdmap e156: 8 total, 8 up, 8 in 2024-04-14T15:13:44.660 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:13:44.684 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:44.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:44.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:13:44.785 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:13:44.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:44.810 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:44.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:13:44.911 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:13:44.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:44.935 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:44.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:13:45.036 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:13:45.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:45.060 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:45.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:13:45.122 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:13:44 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:13:45.161 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:13:45.165 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":177,"stamp":"2024-04-14T15:13:41.826547+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":227102562,"num_objects":538,"num_object_clones":181,"num_object_copies":1614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":538,"num_whiteouts":0,"num_read":2185,"num_read_kb":50169,"num_write":2087,"num_write_kb":230576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1771434,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2334,"ondisk_log_size":2334,"up":996,"acting":996,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":1222,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":1105604,"kb_used_data":686788,"kb_used_omap":12,"kb_used_meta":418675,"kb_avail":748855612,"statfs":{"total":767960285184,"available":766828146688,"internally_reserved":0,"allocated":703270912,"data_stored":690923835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":428723762},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":988,"num_objects":278,"num_object_clones":179,"num_object_copies":834,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":278,"num_whiteouts":0,"num_read":3,"num_read_kb":5,"num_write":270,"num_write_kb":270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.009350"},"pg_stats":[{"pgid":"176.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:13:39.819660+0000","last_change":"2024-04-14T15:13:39.819660+0000","last_active":"2024-04-14T15:13:39.819660+0000","last_peered":"2024-04-14T15:13:39.819660+0000","last_clean":"2024-04-14T15:13:39.819660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:13:39.819660+0000","last_undegraded":"2024-04-14T15:13:39.819660+0000","last_fullsized":"2024-04-14T15:13:39.819660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:39.819660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:39.819660+0000","last_clean_scrub_stamp":"2024-04-14T15:13:39.819660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"172.10","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.887594+0000","last_change":"2024-04-14T15:13:36.836370+0000","last_active":"2024-04-14T15:13:38.887594+0000","last_peered":"2024-04-14T15:13:38.887594+0000","last_clean":"2024-04-14T15:13:38.887594+0000","last_became_active":"2024-04-14T15:13:36.835930+0000","last_became_peered":"2024-04-14T15:13:36.835930+0000","last_unstale":"2024-04-14T15:13:38.887594+0000","last_undegraded":"2024-04-14T15:13:38.887594+0000","last_fullsized":"2024-04-14T15:13:38.887594+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_clean_scrub_stamp":"2024-04-14T15:13:35.805229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:52.345881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"176.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:13:39.819660+0000","last_change":"2024-04-14T15:13:39.819660+0000","last_active":"2024-04-14T15:13:39.819660+0000","last_peered":"2024-04-14T15:13:39.819660+0000","last_clean":"2024-04-14T15:13:39.819660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:13:39.819660+0000","last_undegraded":"2024-04-14T15:13:39.819660+0000","last_fullsized":"2024-04-14T15:13:39.819660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:39.819660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:39.819660+0000","last_clean_scrub_stamp":"2024-04-14T15:13:39.819660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"172.11","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.881188+0000","last_change":"2024-04-14T15:13:36.839799+0000","last_active":"2024-04-14T15:13:38.881188+0000","last_peered":"2024-04-14T15:13:38.881188+0000","last_clean":"2024-04-14T15:13:38.881188+0000","last_became_active":"2024-04-14T15:13:36.839495+0000","last_became_peered":"2024-04-14T15:13:36.839495+0000","last_unstale":"2024-04-14T15:13:38.881188+0000","last_undegraded":"2024-04-14T15:13:38.881188+0000","last_fullsized":"2024-04-14T15:13:38.881188+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_clean_scrub_stamp":"2024-04-14T15:13:35.805229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:39.782625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"176.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:13:39.819660+0000","last_change":"2024-04-14T15:13:39.819660+0000","last_active":"2024-04-14T15:13:39.819660+0000","last_peered":"2024-04-14T15:13:39.819660+0000","last_clean":"2024-04-14T15:13:39.819660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:13:39.819660+0000","last_undegraded":"2024-04-14T15:13:39.819660+0000","last_fullsized":"2024-04-14T15:13:39.819660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:39.819660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:39.819660+0000","last_clean_scrub_stamp":"2024-04-14T15:13:39.819660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"172.12","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.869821+0000","last_change":"2024-04-14T15:13:36.839422+0000","last_active":"2024-04-14T15:13:38.869821+0000","last_peered":"2024-04-14T15:13:38.869821+0000","last_clean":"2024-04-14T15:13:38.869821+0000","last_became_active":"2024-04-14T15:13:36.839140+0000","last_became_peered":"2024-04-14T15:13:36.839140+0000","last_unstale":"2024-04-14T15:13:38.869821+0000","last_undegraded":"2024-04-14T15:13:38.869821+0000","last_fullsized":"2024-04-14T15:13:38.869821+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_clean_scrub_stamp":"2024-04-14T15:13:35.805229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:59.291451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"176.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:13:39.819660+0000","last_change":"2024-04-14T15:13:39.819660+0000","last_active":"2024-04-14T15:13:39.819660+0000","last_peered":"2024-04-14T15:13:39.819660+0000","last_clean":"2024-04-14T15:13:39.819660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:13:39.819660+0000","last_undegraded":"2024-04-14T15:13:39.819660+0000","last_fullsized":"2024-04-14T15:13:39.819660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:39.819660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:39.819660+0000","last_clean_scrub_stamp":"2024-04-14T15:13:39.819660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"172.13","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.848463+0000","last_change":"2024-04-14T15:13:36.843534+0000","last_active":"2024-04-14T15:13:38.848463+0000","last_peered":"2024-04-14T15:13:38.848463+0000","last_clean":"2024-04-14T15:13:38.848463+0000","last_became_active":"2024-04-14T15:13:36.843254+0000","last_became_peered":"2024-04-14T15:13:36.843254+0000","last_unstale":"2024-04-14T15:13:38.848463+0000","last_undegraded":"2024-04-14T15:13:38.848463+0000","last_fullsized":"2024-04-14T15:13:38.848463+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_clean_scrub_stamp":"2024-04-14T15:13:35.805229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:56.921689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"176.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:13:39.819660+0000","last_change":"2024-04-14T15:13:39.819660+0000","last_active":"2024-04-14T15:13:39.819660+0000","last_peered":"2024-04-14T15:13:39.819660+0000","last_clean":"2024-04-14T15:13:39.819660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:13:39.819660+0000","last_undegraded":"2024-04-14T15:13:39.819660+0000","last_fullsized":"2024-04-14T15:13:39.819660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:39.819660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:39.819660+0000","last_clean_scrub_stamp":"2024-04-14T15:13:39.819660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"172.14","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.839032+0000","last_change":"2024-04-14T15:13:36.835895+0000","last_active":"2024-04-14T15:13:38.839032+0000","last_peered":"2024-04-14T15:13:38.839032+0000","last_clean":"2024-04-14T15:13:38.839032+0000","last_became_active":"2024-04-14T15:13:36.835613+0000","last_became_peered":"2024-04-14T15:13:36.835613+0000","last_unstale":"2024-04-14T15:13:38.839032+0000","last_undegraded":"2024-04-14T15:13:38.839032+0000","last_fullsized":"2024-04-14T15:13:38.839032+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_clean_scrub_stamp":"2024-04-14T15:13:35.805229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:39.526748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"176.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:13:39.819660+0000","last_change":"2024-04-14T15:13:39.819660+0000","last_active":"2024-04-14T15:13:39.819660+0000","last_peered":"2024-04-14T15:13:39.819660+0000","last_clean":"2024-04-14T15:13:39.819660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:13:39.819660+0000","last_undegraded":"2024-04-14T15:13:39.819660+0000","last_fullsized":"2024-04-14T15:13:39.819660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:39.819660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:39.819660+0000","last_clean_scrub_stamp":"2024-04-14T15:13:39.819660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"172.15","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.828796+0000","last_change":"2024-04-14T15:13:36.843835+0000","last_active":"2024-04-14T15:13:38.828796+0000","last_peered":"2024-04-14T15:13:38.828796+0000","last_clean":"2024-04-14T15:13:38.828796+0000","last_became_active":"2024-04-14T15:13:36.843523+0000","last_became_peered":"2024-04-14T15:13:36.843523+0000","last_unstale":"2024-04-14T15:13:38.828796+0000","last_undegraded":"2024-04-14T15:13:38.828796+0000","last_fullsized":"2024-04-14T15:13:38.828796+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_clean_scrub_stamp":"2024-04-14T15:13:35.805229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:45.972612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"176.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:13:39.819660+0000","last_change":"2024-04-14T15:13:39.819660+0000","last_active":"2024-04-14T15:13:39.819660+0000","last_peered":"2024-04-14T15:13:39.819660+0000","last_clean":"2024-04-14T15:13:39.819660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:13:39.819660+0000","last_undegraded":"2024-04-14T15:13:39.819660+0000","last_fullsized":"2024-04-14T15:13:39.819660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:39.819660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:39.819660+0000","last_clean_scrub_stamp":"2024-04-14T15:13:39.819660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"172.16","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.831930+0000","last_change":"2024-04-14T15:13:36.845354+0000","last_active":"2024-04-14T15:13:38.831930+0000","last_peered":"2024-04-14T15:13:38.831930+0000","last_clean":"2024-04-14T15:13:38.831930+0000","last_became_active":"2024-04-14T15:13:36.845013+0000","last_became_peered":"2024-04-14T15:13:36.845013+0000","last_unstale":"2024-04-14T15:13:38.831930+0000","last_undegraded":"2024-04-14T15:13:38.831930+0000","last_fullsized":"2024-04-14T15:13:38.831930+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_clean_scrub_stamp":"2024-04-14T15:13:35.805229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:15:03.147487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"176.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:13:39.819660+0000","last_change":"2024-04-14T15:13:39.819660+0000","last_active":"2024-04-14T15:13:39.819660+0000","last_peered":"2024-04-14T15:13:39.819660+0000","last_clean":"2024-04-14T15:13:39.819660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:13:39.819660+0000","last_undegraded":"2024-04-14T15:13:39.819660+0000","last_fullsized":"2024-04-14T15:13:39.819660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:39.819660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:39.819660+0000","last_clean_scrub_stamp":"2024-04-14T15:13:39.819660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"172.17","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.852535+0000","last_change":"2024-04-14T15:13:36.839345+0000","last_active":"2024-04-14T15:13:38.852535+0000","last_peered":"2024-04-14T15:13:38.852535+0000","last_clean":"2024-04-14T15:13:38.852535+0000","last_became_active":"2024-04-14T15:13:36.838969+0000","last_became_peered":"2024-04-14T15:13:36.838969+0000","last_unstale":"2024-04-14T15:13:38.852535+0000","last_undegraded":"2024-04-14T15:13:38.852535+0000","last_fullsized":"2024-04-14T15:13:38.852535+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_clean_scrub_stamp":"2024-04-14T15:13:35.805229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:42.478621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"176.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:13:39.819660+0000","last_change":"2024-04-14T15:13:39.819660+0000","last_active":"2024-04-14T15:13:39.819660+0000","last_peered":"2024-04-14T15:13:39.819660+0000","last_clean":"2024-04-14T15:13:39.819660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:13:39.819660+0000","last_undegraded":"2024-04-14T15:13:39.819660+0000","last_fullsized":"2024-04-14T15:13:39.819660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:39.819660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:39.819660+0000","last_clean_scrub_stamp":"2024-04-14T15:13:39.819660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"172.18","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.843787+0000","last_change":"2024-04-14T15:13:36.837744+0000","last_active":"2024-04-14T15:13:38.843787+0000","last_peered":"2024-04-14T15:13:38.843787+0000","last_clean":"2024-04-14T15:13:38.843787+0000","last_became_active":"2024-04-14T15:13:36.837290+0000","last_became_peered":"2024-04-14T15:13:36.837290+0000","last_unstale":"2024-04-14T15:13:38.843787+0000","last_undegraded":"2024-04-14T15:13:38.843787+0000","last_fullsized":"2024-04-14T15:13:38.843787+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_clean_scrub_stamp":"2024-04-14T15:13:35.805229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:15:05.338235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"176.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:13:39.819660+0000","last_change":"2024-04-14T15:13:39.819660+0000","last_active":"2024-04-14T15:13:39.819660+0000","last_peered":"2024-04-14T15:13:39.819660+0000","last_clean":"2024-04-14T15:13:39.819660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:13:39.819660+0000","last_undegraded":"2024-04-14T15:13:39.819660+0000","last_fullsized":"2024-04-14T15:13:39.819660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:39.819660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:39.819660+0000","last_clean_scrub_stamp":"2024-04-14T15:13:39.819660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"172.19","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.887547+0000","last_change":"2024-04-14T15:13:36.838918+0000","last_active":"2024-04-14T15:13:38.887547+0000","last_peered":"2024-04-14T15:13:38.887547+0000","last_clean":"2024-04-14T15:13:38.887547+0000","last_became_active":"2024-04-14T15:13:36.838575+0000","last_became_peered":"2024-04-14T15:13:36.838575+0000","last_unstale":"2024-04-14T15:13:38.887547+0000","last_undegraded":"2024-04-14T15:13:38.887547+0000","last_fullsized":"2024-04-14T15:13:38.887547+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_clean_scrub_stamp":"2024-04-14T15:13:35.805229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:15:04.094748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"176.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:13:39.819660+0000","last_change":"2024-04-14T15:13:39.819660+0000","last_active":"2024-04-14T15:13:39.819660+0000","last_peered":"2024-04-14T15:13:39.819660+0000","last_clean":"2024-04-14T15:13:39.819660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:13:39.819660+0000","last_undegraded":"2024-04-14T15:13:39.819660+0000","last_fullsized":"2024-04-14T15:13:39.819660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:39.819660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:39.819660+0000","last_clean_scrub_stamp":"2024-04-14T15:13:39.819660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"172.1a","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.829380+0000","last_change":"2024-04-14T15:13:36.827815+0000","last_active":"2024-04-14T15:13:38.829380+0000","last_peered":"2024-04-14T15:13:38.829380+0000","last_clean":"2024-04-14T15:13:38.829380+0000","last_became_active":"2024-04-14T15:13:36.827449+0000","last_became_peered":"2024-04-14T15:13:36.827449+0000","last_unstale":"2024-04-14T15:13:38.829380+0000","last_undegraded":"2024-04-14T15:13:38.829380+0000","last_fullsized":"2024-04-14T15:13:38.829380+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_clean_scrub_stamp":"2024-04-14T15:13:35.805229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:52.252232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"176.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:13:39.819660+0000","last_change":"2024-04-14T15:13:39.819660+0000","last_active":"2024-04-14T15:13:39.819660+0000","last_peered":"2024-04-14T15:13:39.819660+0000","last_clean":"2024-04-14T15:13:39.819660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:13:39.819660+0000","last_undegraded":"2024-04-14T15:13:39.819660+0000","last_fullsized":"2024-04-14T15:13:39.819660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:39.819660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:39.819660+0000","last_clean_scrub_stamp":"2024-04-14T15:13:39.819660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"172.1b","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.825615+0000","last_change":"2024-04-14T15:13:36.845268+0000","last_active":"2024-04-14T15:13:38.825615+0000","last_peered":"2024-04-14T15:13:38.825615+0000","last_clean":"2024-04-14T15:13:38.825615+0000","last_became_active":"2024-04-14T15:13:36.844904+0000","last_became_peered":"2024-04-14T15:13:36.844904+0000","last_unstale":"2024-04-14T15:13:38.825615+0000","last_undegraded":"2024-04-14T15:13:38.825615+0000","last_fullsized":"2024-04-14T15:13:38.825615+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_clean_scrub_stamp":"2024-04-14T15:13:35.805229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:40.760339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"176.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:13:39.819660+0000","last_change":"2024-04-14T15:13:39.819660+0000","last_active":"2024-04-14T15:13:39.819660+0000","last_peered":"2024-04-14T15:13:39.819660+0000","last_clean":"2024-04-14T15:13:39.819660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:13:39.819660+0000","last_undegraded":"2024-04-14T15:13:39.819660+0000","last_fullsized":"2024-04-14T15:13:39.819660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:39.819660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:39.819660+0000","last_clean_scrub_stamp":"2024-04-14T15:13:39.819660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"172.1c","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.842684+0000","last_change":"2024-04-14T15:13:36.841773+0000","last_active":"2024-04-14T15:13:38.842684+0000","last_peered":"2024-04-14T15:13:38.842684+0000","last_clean":"2024-04-14T15:13:38.842684+0000","last_became_active":"2024-04-14T15:13:36.841500+0000","last_became_peered":"2024-04-14T15:13:36.841500+0000","last_unstale":"2024-04-14T15:13:38.842684+0000","last_undegraded":"2024-04-14T15:13:38.842684+0000","last_fullsized":"2024-04-14T15:13:38.842684+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_clean_scrub_stamp":"2024-04-14T15:13:35.805229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:39.204440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"176.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:13:39.819660+0000","last_change":"2024-04-14T15:13:39.819660+0000","last_active":"2024-04-14T15:13:39.819660+0000","last_peered":"2024-04-14T15:13:39.819660+0000","last_clean":"2024-04-14T15:13:39.819660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:13:39.819660+0000","last_undegraded":"2024-04-14T15:13:39.819660+0000","last_fullsized":"2024-04-14T15:13:39.819660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:39.819660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:39.819660+0000","last_clean_scrub_stamp":"2024-04-14T15:13:39.819660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"172.1d","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.849058+0000","last_change":"2024-04-14T15:13:36.844380+0000","last_active":"2024-04-14T15:13:38.849058+0000","last_peered":"2024-04-14T15:13:38.849058+0000","last_clean":"2024-04-14T15:13:38.849058+0000","last_became_active":"2024-04-14T15:13:36.844056+0000","last_became_peered":"2024-04-14T15:13:36.844056+0000","last_unstale":"2024-04-14T15:13:38.849058+0000","last_undegraded":"2024-04-14T15:13:38.849058+0000","last_fullsized":"2024-04-14T15:13:38.849058+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_clean_scrub_stamp":"2024-04-14T15:13:35.805229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:53.851732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"176.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:13:39.819660+0000","last_change":"2024-04-14T15:13:39.819660+0000","last_active":"2024-04-14T15:13:39.819660+0000","last_peered":"2024-04-14T15:13:39.819660+0000","last_clean":"2024-04-14T15:13:39.819660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:13:39.819660+0000","last_undegraded":"2024-04-14T15:13:39.819660+0000","last_fullsized":"2024-04-14T15:13:39.819660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:39.819660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:39.819660+0000","last_clean_scrub_stamp":"2024-04-14T15:13:39.819660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"172.1e","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.838476+0000","last_change":"2024-04-14T15:13:36.836288+0000","last_active":"2024-04-14T15:13:38.838476+0000","last_peered":"2024-04-14T15:13:38.838476+0000","last_clean":"2024-04-14T15:13:38.838476+0000","last_became_active":"2024-04-14T15:13:36.835984+0000","last_became_peered":"2024-04-14T15:13:36.835984+0000","last_unstale":"2024-04-14T15:13:38.838476+0000","last_undegraded":"2024-04-14T15:13:38.838476+0000","last_fullsized":"2024-04-14T15:13:38.838476+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_clean_scrub_stamp":"2024-04-14T15:13:35.805229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:44.832898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"176.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:13:39.819660+0000","last_change":"2024-04-14T15:13:39.819660+0000","last_active":"2024-04-14T15:13:39.819660+0000","last_peered":"2024-04-14T15:13:39.819660+0000","last_clean":"2024-04-14T15:13:39.819660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:13:39.819660+0000","last_undegraded":"2024-04-14T15:13:39.819660+0000","last_fullsized":"2024-04-14T15:13:39.819660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:39.819660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:39.819660+0000","last_clean_scrub_stamp":"2024-04-14T15:13:39.819660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"172.1f","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.841881+0000","last_change":"2024-04-14T15:13:36.839078+0000","last_active":"2024-04-14T15:13:38.841881+0000","last_peered":"2024-04-14T15:13:38.841881+0000","last_clean":"2024-04-14T15:13:38.841881+0000","last_became_active":"2024-04-14T15:13:36.838471+0000","last_became_peered":"2024-04-14T15:13:36.838471+0000","last_unstale":"2024-04-14T15:13:38.841881+0000","last_undegraded":"2024-04-14T15:13:38.841881+0000","last_fullsized":"2024-04-14T15:13:38.841881+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_clean_scrub_stamp":"2024-04-14T15:13:35.805229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:15:01.510870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"176.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:13:39.819660+0000","last_change":"2024-04-14T15:13:39.819660+0000","last_active":"2024-04-14T15:13:39.819660+0000","last_peered":"2024-04-14T15:13:39.819660+0000","last_clean":"2024-04-14T15:13:39.819660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:13:39.819660+0000","last_undegraded":"2024-04-14T15:13:39.819660+0000","last_fullsized":"2024-04-14T15:13:39.819660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:39.819660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:39.819660+0000","last_clean_scrub_stamp":"2024-04-14T15:13:39.819660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"175.3","version":"0'0","reported_seq":1,"reported_epoch":151,"state":"creating+peering","last_fresh":"2024-04-14T15:13:38.836507+0000","last_change":"2024-04-14T15:13:38.836507+0000","last_active":"2024-04-14T15:13:38.815487+0000","last_peered":"2024-04-14T15:13:38.815487+0000","last_clean":"2024-04-14T15:13:38.815487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:13:38.836507+0000","last_undegraded":"2024-04-14T15:13:38.836507+0000","last_fullsized":"2024-04-14T15:13:38.836507+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:38.815487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:38.815487+0000","last_clean_scrub_stamp":"2024-04-14T15:13:38.815487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"172.0","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.879999+0000","last_change":"2024-04-14T15:13:36.840562+0000","last_active":"2024-04-14T15:13:38.879999+0000","last_peered":"2024-04-14T15:13:38.879999+0000","last_clean":"2024-04-14T15:13:38.879999+0000","last_became_active":"2024-04-14T15:13:36.840245+0000","last_became_peered":"2024-04-14T15:13:36.840245+0000","last_unstale":"2024-04-14T15:13:38.879999+0000","last_undegraded":"2024-04-14T15:13:38.879999+0000","last_fullsized":"2024-04-14T15:13:38.879999+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_clean_scrub_stamp":"2024-04-14T15:13:35.805229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:59.569967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"176.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:13:39.819660+0000","last_change":"2024-04-14T15:13:39.819660+0000","last_active":"2024-04-14T15:13:39.819660+0000","last_peered":"2024-04-14T15:13:39.819660+0000","last_clean":"2024-04-14T15:13:39.819660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:13:39.819660+0000","last_undegraded":"2024-04-14T15:13:39.819660+0000","last_fullsized":"2024-04-14T15:13:39.819660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:39.819660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:39.819660+0000","last_clean_scrub_stamp":"2024-04-14T15:13:39.819660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"175.2","version":"0'0","reported_seq":1,"reported_epoch":151,"state":"creating+peering","last_fresh":"2024-04-14T15:13:38.867965+0000","last_change":"2024-04-14T15:13:38.867965+0000","last_active":"2024-04-14T15:13:38.815487+0000","last_peered":"2024-04-14T15:13:38.815487+0000","last_clean":"2024-04-14T15:13:38.815487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:13:38.867965+0000","last_undegraded":"2024-04-14T15:13:38.867965+0000","last_fullsized":"2024-04-14T15:13:38.867965+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:38.815487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:38.815487+0000","last_clean_scrub_stamp":"2024-04-14T15:13:38.815487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"172.1","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.868893+0000","last_change":"2024-04-14T15:13:36.840321+0000","last_active":"2024-04-14T15:13:38.868893+0000","last_peered":"2024-04-14T15:13:38.868893+0000","last_clean":"2024-04-14T15:13:38.868893+0000","last_became_active":"2024-04-14T15:13:36.840020+0000","last_became_peered":"2024-04-14T15:13:36.840020+0000","last_unstale":"2024-04-14T15:13:38.868893+0000","last_undegraded":"2024-04-14T15:13:38.868893+0000","last_fullsized":"2024-04-14T15:13:38.868893+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_clean_scrub_stamp":"2024-04-14T15:13:35.805229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:54.795977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"176.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:13:39.819660+0000","last_change":"2024-04-14T15:13:39.819660+0000","last_active":"2024-04-14T15:13:39.819660+0000","last_peered":"2024-04-14T15:13:39.819660+0000","last_clean":"2024-04-14T15:13:39.819660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:13:39.819660+0000","last_undegraded":"2024-04-14T15:13:39.819660+0000","last_fullsized":"2024-04-14T15:13:39.819660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:39.819660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:39.819660+0000","last_clean_scrub_stamp":"2024-04-14T15:13:39.819660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"175.1","version":"0'0","reported_seq":1,"reported_epoch":151,"state":"creating+peering","last_fresh":"2024-04-14T15:13:38.835288+0000","last_change":"2024-04-14T15:13:38.835288+0000","last_active":"2024-04-14T15:13:38.815487+0000","last_peered":"2024-04-14T15:13:38.815487+0000","last_clean":"2024-04-14T15:13:38.815487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:13:38.835288+0000","last_undegraded":"2024-04-14T15:13:38.835288+0000","last_fullsized":"2024-04-14T15:13:38.835288+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:38.815487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:38.815487+0000","last_clean_scrub_stamp":"2024-04-14T15:13:38.815487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"172.2","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.829466+0000","last_change":"2024-04-14T15:13:36.842396+0000","last_active":"2024-04-14T15:13:38.829466+0000","last_peered":"2024-04-14T15:13:38.829466+0000","last_clean":"2024-04-14T15:13:38.829466+0000","last_became_active":"2024-04-14T15:13:36.841465+0000","last_became_peered":"2024-04-14T15:13:36.841465+0000","last_unstale":"2024-04-14T15:13:38.829466+0000","last_undegraded":"2024-04-14T15:13:38.829466+0000","last_fullsized":"2024-04-14T15:13:38.829466+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_clean_scrub_stamp":"2024-04-14T15:13:35.805229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:38.175326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"176.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:13:39.819660+0000","last_change":"2024-04-14T15:13:39.819660+0000","last_active":"2024-04-14T15:13:39.819660+0000","last_peered":"2024-04-14T15:13:39.819660+0000","last_clean":"2024-04-14T15:13:39.819660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:13:39.819660+0000","last_undegraded":"2024-04-14T15:13:39.819660+0000","last_fullsized":"2024-04-14T15:13:39.819660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:39.819660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:39.819660+0000","last_clean_scrub_stamp":"2024-04-14T15:13:39.819660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"175.0","version":"0'0","reported_seq":1,"reported_epoch":151,"state":"creating+peering","last_fresh":"2024-04-14T15:13:38.837088+0000","last_change":"2024-04-14T15:13:38.837088+0000","last_active":"2024-04-14T15:13:38.815487+0000","last_peered":"2024-04-14T15:13:38.815487+0000","last_clean":"2024-04-14T15:13:38.815487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:13:38.837088+0000","last_undegraded":"2024-04-14T15:13:38.837088+0000","last_fullsized":"2024-04-14T15:13:38.837088+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:38.815487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:38.815487+0000","last_clean_scrub_stamp":"2024-04-14T15:13:38.815487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"172.3","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.834059+0000","last_change":"2024-04-14T15:13:36.840020+0000","last_active":"2024-04-14T15:13:38.834059+0000","last_peered":"2024-04-14T15:13:38.834059+0000","last_clean":"2024-04-14T15:13:38.834059+0000","last_became_active":"2024-04-14T15:13:36.839724+0000","last_became_peered":"2024-04-14T15:13:36.839724+0000","last_unstale":"2024-04-14T15:13:38.834059+0000","last_undegraded":"2024-04-14T15:13:38.834059+0000","last_fullsized":"2024-04-14T15:13:38.834059+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_clean_scrub_stamp":"2024-04-14T15:13:35.805229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:39.166219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"176.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:13:39.819660+0000","last_change":"2024-04-14T15:13:39.819660+0000","last_active":"2024-04-14T15:13:39.819660+0000","last_peered":"2024-04-14T15:13:39.819660+0000","last_clean":"2024-04-14T15:13:39.819660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:13:39.819660+0000","last_undegraded":"2024-04-14T15:13:39.819660+0000","last_fullsized":"2024-04-14T15:13:39.819660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:39.819660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:39.819660+0000","last_clean_scrub_stamp":"2024-04-14T15:13:39.819660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"175.7","version":"0'0","reported_seq":1,"reported_epoch":151,"state":"creating+peering","last_fresh":"2024-04-14T15:13:38.834825+0000","last_change":"2024-04-14T15:13:38.834825+0000","last_active":"2024-04-14T15:13:38.815487+0000","last_peered":"2024-04-14T15:13:38.815487+0000","last_clean":"2024-04-14T15:13:38.815487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:13:38.834825+0000","last_undegraded":"2024-04-14T15:13:38.834825+0000","last_fullsized":"2024-04-14T15:13:38.834825+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:38.815487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:38.815487+0000","last_clean_scrub_stamp":"2024-04-14T15:13:38.815487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"172.4","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.828038+0000","last_change":"2024-04-14T15:13:36.845603+0000","last_active":"2024-04-14T15:13:38.828038+0000","last_peered":"2024-04-14T15:13:38.828038+0000","last_clean":"2024-04-14T15:13:38.828038+0000","last_became_active":"2024-04-14T15:13:36.844915+0000","last_became_peered":"2024-04-14T15:13:36.844915+0000","last_unstale":"2024-04-14T15:13:38.828038+0000","last_undegraded":"2024-04-14T15:13:38.828038+0000","last_fullsized":"2024-04-14T15:13:38.828038+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_clean_scrub_stamp":"2024-04-14T15:13:35.805229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:35.808497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"176.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:13:39.819660+0000","last_change":"2024-04-14T15:13:39.819660+0000","last_active":"2024-04-14T15:13:39.819660+0000","last_peered":"2024-04-14T15:13:39.819660+0000","last_clean":"2024-04-14T15:13:39.819660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:13:39.819660+0000","last_undegraded":"2024-04-14T15:13:39.819660+0000","last_fullsized":"2024-04-14T15:13:39.819660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:39.819660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:39.819660+0000","last_clean_scrub_stamp":"2024-04-14T15:13:39.819660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"175.6","version":"0'0","reported_seq":1,"reported_epoch":151,"state":"creating+peering","last_fresh":"2024-04-14T15:13:38.837587+0000","last_change":"2024-04-14T15:13:38.837587+0000","last_active":"2024-04-14T15:13:38.815487+0000","last_peered":"2024-04-14T15:13:38.815487+0000","last_clean":"2024-04-14T15:13:38.815487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:13:38.837587+0000","last_undegraded":"2024-04-14T15:13:38.837587+0000","last_fullsized":"2024-04-14T15:13:38.837587+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:38.815487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:38.815487+0000","last_clean_scrub_stamp":"2024-04-14T15:13:38.815487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"172.5","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.872115+0000","last_change":"2024-04-14T15:13:36.835519+0000","last_active":"2024-04-14T15:13:38.872115+0000","last_peered":"2024-04-14T15:13:38.872115+0000","last_clean":"2024-04-14T15:13:38.872115+0000","last_became_active":"2024-04-14T15:13:36.835226+0000","last_became_peered":"2024-04-14T15:13:36.835226+0000","last_unstale":"2024-04-14T15:13:38.872115+0000","last_undegraded":"2024-04-14T15:13:38.872115+0000","last_fullsized":"2024-04-14T15:13:38.872115+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_clean_scrub_stamp":"2024-04-14T15:13:35.805229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:40.598250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"176.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:13:39.819660+0000","last_change":"2024-04-14T15:13:39.819660+0000","last_active":"2024-04-14T15:13:39.819660+0000","last_peered":"2024-04-14T15:13:39.819660+0000","last_clean":"2024-04-14T15:13:39.819660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:13:39.819660+0000","last_undegraded":"2024-04-14T15:13:39.819660+0000","last_fullsized":"2024-04-14T15:13:39.819660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:39.819660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:39.819660+0000","last_clean_scrub_stamp":"2024-04-14T15:13:39.819660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"175.5","version":"0'0","reported_seq":1,"reported_epoch":151,"state":"creating+peering","last_fresh":"2024-04-14T15:13:38.824132+0000","last_change":"2024-04-14T15:13:38.824132+0000","last_active":"2024-04-14T15:13:38.815487+0000","last_peered":"2024-04-14T15:13:38.815487+0000","last_clean":"2024-04-14T15:13:38.815487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:13:38.824132+0000","last_undegraded":"2024-04-14T15:13:38.824132+0000","last_fullsized":"2024-04-14T15:13:38.824132+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:38.815487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:38.815487+0000","last_clean_scrub_stamp":"2024-04-14T15:13:38.815487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"172.6","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.873993+0000","last_change":"2024-04-14T15:13:36.841708+0000","last_active":"2024-04-14T15:13:38.873993+0000","last_peered":"2024-04-14T15:13:38.873993+0000","last_clean":"2024-04-14T15:13:38.873993+0000","last_became_active":"2024-04-14T15:13:36.841433+0000","last_became_peered":"2024-04-14T15:13:36.841433+0000","last_unstale":"2024-04-14T15:13:38.873993+0000","last_undegraded":"2024-04-14T15:13:38.873993+0000","last_fullsized":"2024-04-14T15:13:38.873993+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_clean_scrub_stamp":"2024-04-14T15:13:35.805229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:40.206530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"176.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:13:39.819660+0000","last_change":"2024-04-14T15:13:39.819660+0000","last_active":"2024-04-14T15:13:39.819660+0000","last_peered":"2024-04-14T15:13:39.819660+0000","last_clean":"2024-04-14T15:13:39.819660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:13:39.819660+0000","last_undegraded":"2024-04-14T15:13:39.819660+0000","last_fullsized":"2024-04-14T15:13:39.819660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:39.819660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:39.819660+0000","last_clean_scrub_stamp":"2024-04-14T15:13:39.819660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"175.4","version":"0'0","reported_seq":1,"reported_epoch":151,"state":"creating+peering","last_fresh":"2024-04-14T15:13:38.828655+0000","last_change":"2024-04-14T15:13:38.828655+0000","last_active":"2024-04-14T15:13:38.815487+0000","last_peered":"2024-04-14T15:13:38.815487+0000","last_clean":"2024-04-14T15:13:38.815487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:13:38.828655+0000","last_undegraded":"2024-04-14T15:13:38.828655+0000","last_fullsized":"2024-04-14T15:13:38.828655+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:38.815487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:38.815487+0000","last_clean_scrub_stamp":"2024-04-14T15:13:38.815487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"172.7","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.843815+0000","last_change":"2024-04-14T15:13:36.838657+0000","last_active":"2024-04-14T15:13:38.843815+0000","last_peered":"2024-04-14T15:13:38.843815+0000","last_clean":"2024-04-14T15:13:38.843815+0000","last_became_active":"2024-04-14T15:13:36.837803+0000","last_became_peered":"2024-04-14T15:13:36.837803+0000","last_unstale":"2024-04-14T15:13:38.843815+0000","last_undegraded":"2024-04-14T15:13:38.843815+0000","last_fullsized":"2024-04-14T15:13:38.843815+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_clean_scrub_stamp":"2024-04-14T15:13:35.805229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:49.940571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"176.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:13:39.819660+0000","last_change":"2024-04-14T15:13:39.819660+0000","last_active":"2024-04-14T15:13:39.819660+0000","last_peered":"2024-04-14T15:13:39.819660+0000","last_clean":"2024-04-14T15:13:39.819660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:13:39.819660+0000","last_undegraded":"2024-04-14T15:13:39.819660+0000","last_fullsized":"2024-04-14T15:13:39.819660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:39.819660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:39.819660+0000","last_clean_scrub_stamp":"2024-04-14T15:13:39.819660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"172.8","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.837182+0000","last_change":"2024-04-14T15:13:36.834406+0000","last_active":"2024-04-14T15:13:38.837182+0000","last_peered":"2024-04-14T15:13:38.837182+0000","last_clean":"2024-04-14T15:13:38.837182+0000","last_became_active":"2024-04-14T15:13:36.834104+0000","last_became_peered":"2024-04-14T15:13:36.834104+0000","last_unstale":"2024-04-14T15:13:38.837182+0000","last_undegraded":"2024-04-14T15:13:38.837182+0000","last_fullsized":"2024-04-14T15:13:38.837182+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_clean_scrub_stamp":"2024-04-14T15:13:35.805229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:45.226927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"176.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:13:39.819660+0000","last_change":"2024-04-14T15:13:39.819660+0000","last_active":"2024-04-14T15:13:39.819660+0000","last_peered":"2024-04-14T15:13:39.819660+0000","last_clean":"2024-04-14T15:13:39.819660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:13:39.819660+0000","last_undegraded":"2024-04-14T15:13:39.819660+0000","last_fullsized":"2024-04-14T15:13:39.819660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:39.819660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:39.819660+0000","last_clean_scrub_stamp":"2024-04-14T15:13:39.819660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"172.9","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.848744+0000","last_change":"2024-04-14T15:13:36.843861+0000","last_active":"2024-04-14T15:13:38.848744+0000","last_peered":"2024-04-14T15:13:38.848744+0000","last_clean":"2024-04-14T15:13:38.848744+0000","last_became_active":"2024-04-14T15:13:36.842423+0000","last_became_peered":"2024-04-14T15:13:36.842423+0000","last_unstale":"2024-04-14T15:13:38.848744+0000","last_undegraded":"2024-04-14T15:13:38.848744+0000","last_fullsized":"2024-04-14T15:13:38.848744+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_clean_scrub_stamp":"2024-04-14T15:13:35.805229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:15:01.476326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"176.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:13:39.819660+0000","last_change":"2024-04-14T15:13:39.819660+0000","last_active":"2024-04-14T15:13:39.819660+0000","last_peered":"2024-04-14T15:13:39.819660+0000","last_clean":"2024-04-14T15:13:39.819660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:13:39.819660+0000","last_undegraded":"2024-04-14T15:13:39.819660+0000","last_fullsized":"2024-04-14T15:13:39.819660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:39.819660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:39.819660+0000","last_clean_scrub_stamp":"2024-04-14T15:13:39.819660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"172.a","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.825236+0000","last_change":"2024-04-14T15:13:36.844165+0000","last_active":"2024-04-14T15:13:38.825236+0000","last_peered":"2024-04-14T15:13:38.825236+0000","last_clean":"2024-04-14T15:13:38.825236+0000","last_became_active":"2024-04-14T15:13:36.843868+0000","last_became_peered":"2024-04-14T15:13:36.843868+0000","last_unstale":"2024-04-14T15:13:38.825236+0000","last_undegraded":"2024-04-14T15:13:38.825236+0000","last_fullsized":"2024-04-14T15:13:38.825236+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_clean_scrub_stamp":"2024-04-14T15:13:35.805229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:52.771545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"176.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:13:39.819660+0000","last_change":"2024-04-14T15:13:39.819660+0000","last_active":"2024-04-14T15:13:39.819660+0000","last_peered":"2024-04-14T15:13:39.819660+0000","last_clean":"2024-04-14T15:13:39.819660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:13:39.819660+0000","last_undegraded":"2024-04-14T15:13:39.819660+0000","last_fullsized":"2024-04-14T15:13:39.819660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:39.819660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:39.819660+0000","last_clean_scrub_stamp":"2024-04-14T15:13:39.819660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"172.b","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.828334+0000","last_change":"2024-04-14T15:13:36.835434+0000","last_active":"2024-04-14T15:13:38.828334+0000","last_peered":"2024-04-14T15:13:38.828334+0000","last_clean":"2024-04-14T15:13:38.828334+0000","last_became_active":"2024-04-14T15:13:36.835157+0000","last_became_peered":"2024-04-14T15:13:36.835157+0000","last_unstale":"2024-04-14T15:13:38.828334+0000","last_undegraded":"2024-04-14T15:13:38.828334+0000","last_fullsized":"2024-04-14T15:13:38.828334+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_clean_scrub_stamp":"2024-04-14T15:13:35.805229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:42.435877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"176.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:13:39.819660+0000","last_change":"2024-04-14T15:13:39.819660+0000","last_active":"2024-04-14T15:13:39.819660+0000","last_peered":"2024-04-14T15:13:39.819660+0000","last_clean":"2024-04-14T15:13:39.819660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:13:39.819660+0000","last_undegraded":"2024-04-14T15:13:39.819660+0000","last_fullsized":"2024-04-14T15:13:39.819660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:39.819660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:39.819660+0000","last_clean_scrub_stamp":"2024-04-14T15:13:39.819660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"172.c","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.828203+0000","last_change":"2024-04-14T15:13:36.842907+0000","last_active":"2024-04-14T15:13:38.828203+0000","last_peered":"2024-04-14T15:13:38.828203+0000","last_clean":"2024-04-14T15:13:38.828203+0000","last_became_active":"2024-04-14T15:13:36.842608+0000","last_became_peered":"2024-04-14T15:13:36.842608+0000","last_unstale":"2024-04-14T15:13:38.828203+0000","last_undegraded":"2024-04-14T15:13:38.828203+0000","last_fullsized":"2024-04-14T15:13:38.828203+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_clean_scrub_stamp":"2024-04-14T15:13:35.805229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:54.979507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"176.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:13:39.819660+0000","last_change":"2024-04-14T15:13:39.819660+0000","last_active":"2024-04-14T15:13:39.819660+0000","last_peered":"2024-04-14T15:13:39.819660+0000","last_clean":"2024-04-14T15:13:39.819660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:13:39.819660+0000","last_undegraded":"2024-04-14T15:13:39.819660+0000","last_fullsized":"2024-04-14T15:13:39.819660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:39.819660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:39.819660+0000","last_clean_scrub_stamp":"2024-04-14T15:13:39.819660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"172.d","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.841022+0000","last_change":"2024-04-14T15:13:36.846714+0000","last_active":"2024-04-14T15:13:38.841022+0000","last_peered":"2024-04-14T15:13:38.841022+0000","last_clean":"2024-04-14T15:13:38.841022+0000","last_became_active":"2024-04-14T15:13:36.846467+0000","last_became_peered":"2024-04-14T15:13:36.846467+0000","last_unstale":"2024-04-14T15:13:38.841022+0000","last_undegraded":"2024-04-14T15:13:38.841022+0000","last_fullsized":"2024-04-14T15:13:38.841022+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_clean_scrub_stamp":"2024-04-14T15:13:35.805229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:41.430045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"176.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:13:39.819660+0000","last_change":"2024-04-14T15:13:39.819660+0000","last_active":"2024-04-14T15:13:39.819660+0000","last_peered":"2024-04-14T15:13:39.819660+0000","last_clean":"2024-04-14T15:13:39.819660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:13:39.819660+0000","last_undegraded":"2024-04-14T15:13:39.819660+0000","last_fullsized":"2024-04-14T15:13:39.819660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:39.819660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:39.819660+0000","last_clean_scrub_stamp":"2024-04-14T15:13:39.819660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"172.e","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.830058+0000","last_change":"2024-04-14T15:13:36.845458+0000","last_active":"2024-04-14T15:13:38.830058+0000","last_peered":"2024-04-14T15:13:38.830058+0000","last_clean":"2024-04-14T15:13:38.830058+0000","last_became_active":"2024-04-14T15:13:36.845186+0000","last_became_peered":"2024-04-14T15:13:36.845186+0000","last_unstale":"2024-04-14T15:13:38.830058+0000","last_undegraded":"2024-04-14T15:13:38.830058+0000","last_fullsized":"2024-04-14T15:13:38.830058+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_clean_scrub_stamp":"2024-04-14T15:13:35.805229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:47.527904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"176.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:13:39.819660+0000","last_change":"2024-04-14T15:13:39.819660+0000","last_active":"2024-04-14T15:13:39.819660+0000","last_peered":"2024-04-14T15:13:39.819660+0000","last_clean":"2024-04-14T15:13:39.819660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:13:39.819660+0000","last_undegraded":"2024-04-14T15:13:39.819660+0000","last_fullsized":"2024-04-14T15:13:39.819660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:39.819660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:39.819660+0000","last_clean_scrub_stamp":"2024-04-14T15:13:39.819660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"172.f","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.873767+0000","last_change":"2024-04-14T15:13:36.839072+0000","last_active":"2024-04-14T15:13:38.873767+0000","last_peered":"2024-04-14T15:13:38.873767+0000","last_clean":"2024-04-14T15:13:38.873767+0000","last_became_active":"2024-04-14T15:13:36.838790+0000","last_became_peered":"2024-04-14T15:13:36.838790+0000","last_unstale":"2024-04-14T15:13:38.873767+0000","last_undegraded":"2024-04-14T15:13:38.873767+0000","last_fullsized":"2024-04-14T15:13:38.873767+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_clean_scrub_stamp":"2024-04-14T15:13:35.805229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:39.272125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"112.f","version":"0'0","reported_seq":129,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.837073+0000","last_change":"2024-04-14T15:13:33.832036+0000","last_active":"2024-04-14T15:13:38.837073+0000","last_peered":"2024-04-14T15:13:38.837073+0000","last_clean":"2024-04-14T15:13:38.837073+0000","last_became_active":"2024-04-14T15:13:10.975405+0000","last_became_peered":"2024-04-14T15:13:10.975405+0000","last_unstale":"2024-04-14T15:13:38.837073+0000","last_undegraded":"2024-04-14T15:13:38.837073+0000","last_fullsized":"2024-04-14T15:13:38.837073+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:09.802949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:09.802949+0000","last_clean_scrub_stamp":"2024-04-14T15:13:09.802949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:39.784009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015391389999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"6","length":"5"},{"start":"e","length":"1"}]},{"pgid":"110.11","version":"151'10","reported_seq":95,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.915382+0000","last_change":"2024-04-14T15:13:31.858972+0000","last_active":"2024-04-14T15:13:38.915382+0000","last_peered":"2024-04-14T15:13:38.915382+0000","last_clean":"2024-04-14T15:13:38.915382+0000","last_became_active":"2024-04-14T15:13:09.936619+0000","last_became_peered":"2024-04-14T15:13:09.936619+0000","last_unstale":"2024-04-14T15:13:38.915382+0000","last_undegraded":"2024-04-14T15:13:38.915382+0000","last_fullsized":"2024-04-14T15:13:38.915382+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:13:08.799672+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:13.740971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.018553739,"stat_sum":{"num_bytes":60,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"112.e","version":"140'5","reported_seq":133,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.829261+0000","last_change":"2024-04-14T15:13:33.842428+0000","last_active":"2024-04-14T15:13:38.829261+0000","last_peered":"2024-04-14T15:13:38.829261+0000","last_clean":"2024-04-14T15:13:38.829261+0000","last_became_active":"2024-04-14T15:13:10.921373+0000","last_became_peered":"2024-04-14T15:13:10.921373+0000","last_unstale":"2024-04-14T15:13:38.829261+0000","last_undegraded":"2024-04-14T15:13:38.829261+0000","last_fullsized":"2024-04-14T15:13:38.829261+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:09.802949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:09.802949+0000","last_clean_scrub_stamp":"2024-04-14T15:13:09.802949+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:15.588771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0056673239999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":1024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"6","length":"5"},{"start":"e","length":"1"}]},{"pgid":"110.10","version":"151'31","reported_seq":109,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.949551+0000","last_change":"2024-04-14T15:13:31.846027+0000","last_active":"2024-04-14T15:13:38.949551+0000","last_peered":"2024-04-14T15:13:38.949551+0000","last_clean":"2024-04-14T15:13:38.949551+0000","last_became_active":"2024-04-14T15:13:09.928915+0000","last_became_peered":"2024-04-14T15:13:09.928915+0000","last_unstale":"2024-04-14T15:13:38.949551+0000","last_undegraded":"2024-04-14T15:13:38.949551+0000","last_fullsized":"2024-04-14T15:13:38.949551+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:13:08.799672+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:31.418027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.021680241999999999,"stat_sum":{"num_bytes":193,"num_objects":20,"num_object_clones":11,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"112.d","version":"0'0","reported_seq":129,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.841659+0000","last_change":"2024-04-14T15:13:33.841972+0000","last_active":"2024-04-14T15:13:38.841659+0000","last_peered":"2024-04-14T15:13:38.841659+0000","last_clean":"2024-04-14T15:13:38.841659+0000","last_became_active":"2024-04-14T15:13:10.933597+0000","last_became_peered":"2024-04-14T15:13:10.933597+0000","last_unstale":"2024-04-14T15:13:38.841659+0000","last_undegraded":"2024-04-14T15:13:38.841659+0000","last_fullsized":"2024-04-14T15:13:38.841659+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:09.802949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:09.802949+0000","last_clean_scrub_stamp":"2024-04-14T15:13:09.802949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:17.912180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.006612471,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"5"},{"start":"e","length":"1"}]},{"pgid":"110.13","version":"151'10","reported_seq":95,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.937222+0000","last_change":"2024-04-14T15:13:31.855977+0000","last_active":"2024-04-14T15:13:38.937222+0000","last_peered":"2024-04-14T15:13:38.937222+0000","last_clean":"2024-04-14T15:13:38.937222+0000","last_became_active":"2024-04-14T15:13:09.918152+0000","last_became_peered":"2024-04-14T15:13:09.918152+0000","last_unstale":"2024-04-14T15:13:38.937222+0000","last_undegraded":"2024-04-14T15:13:38.937222+0000","last_fullsized":"2024-04-14T15:13:38.937222+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:13:08.799672+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:09.671330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0051087459999999999,"stat_sum":{"num_bytes":60,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"112.c","version":"0'0","reported_seq":129,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.841393+0000","last_change":"2024-04-14T15:13:33.870296+0000","last_active":"2024-04-14T15:13:38.841393+0000","last_peered":"2024-04-14T15:13:38.841393+0000","last_clean":"2024-04-14T15:13:38.841393+0000","last_became_active":"2024-04-14T15:13:10.835315+0000","last_became_peered":"2024-04-14T15:13:10.835315+0000","last_unstale":"2024-04-14T15:13:38.841393+0000","last_undegraded":"2024-04-14T15:13:38.841393+0000","last_fullsized":"2024-04-14T15:13:38.841393+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:09.802949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:09.802949+0000","last_clean_scrub_stamp":"2024-04-14T15:13:09.802949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:23.073527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.037403893000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"6","length":"5"},{"start":"e","length":"1"}]},{"pgid":"110.12","version":"151'20","reported_seq":104,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.965415+0000","last_change":"2024-04-14T15:13:31.908947+0000","last_active":"2024-04-14T15:13:38.965415+0000","last_peered":"2024-04-14T15:13:38.965415+0000","last_clean":"2024-04-14T15:13:38.965415+0000","last_became_active":"2024-04-14T15:13:09.939566+0000","last_became_peered":"2024-04-14T15:13:09.939566+0000","last_unstale":"2024-04-14T15:13:38.965415+0000","last_undegraded":"2024-04-14T15:13:38.965415+0000","last_fullsized":"2024-04-14T15:13:38.965415+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:13:08.799672+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:33.258086+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.045537372999999999,"stat_sum":{"num_bytes":95,"num_objects":10,"num_object_clones":6,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"112.b","version":"0'0","reported_seq":129,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.836671+0000","last_change":"2024-04-14T15:13:33.869801+0000","last_active":"2024-04-14T15:13:38.836671+0000","last_peered":"2024-04-14T15:13:38.836671+0000","last_clean":"2024-04-14T15:13:38.836671+0000","last_became_active":"2024-04-14T15:13:10.883370+0000","last_became_peered":"2024-04-14T15:13:10.883370+0000","last_unstale":"2024-04-14T15:13:38.836671+0000","last_undegraded":"2024-04-14T15:13:38.836671+0000","last_fullsized":"2024-04-14T15:13:38.836671+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:09.802949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:09.802949+0000","last_clean_scrub_stamp":"2024-04-14T15:13:09.802949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:18.377043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.040310958000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"6","length":"5"},{"start":"e","length":"1"}]},{"pgid":"110.15","version":"151'10","reported_seq":95,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.894593+0000","last_change":"2024-04-14T15:13:31.871256+0000","last_active":"2024-04-14T15:13:38.894593+0000","last_peered":"2024-04-14T15:13:38.894593+0000","last_clean":"2024-04-14T15:13:38.894593+0000","last_became_active":"2024-04-14T15:13:09.960121+0000","last_became_peered":"2024-04-14T15:13:09.960121+0000","last_unstale":"2024-04-14T15:13:38.894593+0000","last_undegraded":"2024-04-14T15:13:38.894593+0000","last_fullsized":"2024-04-14T15:13:38.894593+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:13:08.799672+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:26.675011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015032871999999999,"stat_sum":{"num_bytes":60,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"112.a","version":"0'0","reported_seq":129,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.880592+0000","last_change":"2024-04-14T15:13:33.841033+0000","last_active":"2024-04-14T15:13:38.880592+0000","last_peered":"2024-04-14T15:13:38.880592+0000","last_clean":"2024-04-14T15:13:38.880592+0000","last_became_active":"2024-04-14T15:13:11.028413+0000","last_became_peered":"2024-04-14T15:13:11.028413+0000","last_unstale":"2024-04-14T15:13:38.880592+0000","last_undegraded":"2024-04-14T15:13:38.880592+0000","last_fullsized":"2024-04-14T15:13:38.880592+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:09.802949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:09.802949+0000","last_clean_scrub_stamp":"2024-04-14T15:13:09.802949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:37.070137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010970631999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"6","length":"5"},{"start":"e","length":"1"}]},{"pgid":"110.14","version":"151'25","reported_seq":104,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.981810+0000","last_change":"2024-04-14T15:13:31.856253+0000","last_active":"2024-04-14T15:13:38.981810+0000","last_peered":"2024-04-14T15:13:38.981810+0000","last_clean":"2024-04-14T15:13:38.981810+0000","last_became_active":"2024-04-14T15:13:09.899561+0000","last_became_peered":"2024-04-14T15:13:09.899561+0000","last_unstale":"2024-04-14T15:13:38.981810+0000","last_undegraded":"2024-04-14T15:13:38.981810+0000","last_fullsized":"2024-04-14T15:13:38.981810+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:13:08.799672+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:19.867296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015294199,"stat_sum":{"num_bytes":150,"num_objects":15,"num_object_clones":10,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"112.9","version":"0'0","reported_seq":129,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.825676+0000","last_change":"2024-04-14T15:13:33.829507+0000","last_active":"2024-04-14T15:13:38.825676+0000","last_peered":"2024-04-14T15:13:38.825676+0000","last_clean":"2024-04-14T15:13:38.825676+0000","last_became_active":"2024-04-14T15:13:10.852780+0000","last_became_peered":"2024-04-14T15:13:10.852780+0000","last_unstale":"2024-04-14T15:13:38.825676+0000","last_undegraded":"2024-04-14T15:13:38.825676+0000","last_fullsized":"2024-04-14T15:13:38.825676+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:09.802949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:09.802949+0000","last_clean_scrub_stamp":"2024-04-14T15:13:09.802949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:22.491629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00075555499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6","length":"5"},{"start":"e","length":"1"}]},{"pgid":"110.17","version":"131'1","reported_seq":92,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.890275+0000","last_change":"2024-04-14T15:13:31.864280+0000","last_active":"2024-04-14T15:13:38.890275+0000","last_peered":"2024-04-14T15:13:38.890275+0000","last_clean":"2024-04-14T15:13:38.890275+0000","last_became_active":"2024-04-14T15:13:09.898521+0000","last_became_peered":"2024-04-14T15:13:09.898521+0000","last_unstale":"2024-04-14T15:13:38.890275+0000","last_undegraded":"2024-04-14T15:13:38.890275+0000","last_fullsized":"2024-04-14T15:13:38.890275+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:13:08.799672+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:09.594286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016996144000000001,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"112.8","version":"0'0","reported_seq":129,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.825015+0000","last_change":"2024-04-14T15:13:33.837777+0000","last_active":"2024-04-14T15:13:38.825015+0000","last_peered":"2024-04-14T15:13:38.825015+0000","last_clean":"2024-04-14T15:13:38.825015+0000","last_became_active":"2024-04-14T15:13:10.898167+0000","last_became_peered":"2024-04-14T15:13:10.898167+0000","last_unstale":"2024-04-14T15:13:38.825015+0000","last_undegraded":"2024-04-14T15:13:38.825015+0000","last_fullsized":"2024-04-14T15:13:38.825015+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:09.802949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:09.802949+0000","last_clean_scrub_stamp":"2024-04-14T15:13:09.802949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:30.210301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0054405520000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6","length":"5"},{"start":"e","length":"1"}]},{"pgid":"110.16","version":"151'8","reported_seq":94,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.831382+0000","last_change":"2024-04-14T15:13:31.831733+0000","last_active":"2024-04-14T15:13:38.831382+0000","last_peered":"2024-04-14T15:13:38.831382+0000","last_clean":"2024-04-14T15:13:38.831382+0000","last_became_active":"2024-04-14T15:13:09.952918+0000","last_became_peered":"2024-04-14T15:13:09.952918+0000","last_unstale":"2024-04-14T15:13:38.831382+0000","last_undegraded":"2024-04-14T15:13:38.831382+0000","last_fullsized":"2024-04-14T15:13:38.831382+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:13:08.799672+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:37.088289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0067552690000000004,"stat_sum":{"num_bytes":49,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"112.7","version":"0'0","reported_seq":129,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.827356+0000","last_change":"2024-04-14T15:13:33.838716+0000","last_active":"2024-04-14T15:13:38.827356+0000","last_peered":"2024-04-14T15:13:38.827356+0000","last_clean":"2024-04-14T15:13:38.827356+0000","last_became_active":"2024-04-14T15:13:10.897247+0000","last_became_peered":"2024-04-14T15:13:10.897247+0000","last_unstale":"2024-04-14T15:13:38.827356+0000","last_undegraded":"2024-04-14T15:13:38.827356+0000","last_fullsized":"2024-04-14T15:13:38.827356+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:09.802949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:09.802949+0000","last_clean_scrub_stamp":"2024-04-14T15:13:09.802949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:23.208097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040367529999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6","length":"5"},{"start":"e","length":"1"}]},{"pgid":"110.19","version":"151'25","reported_seq":104,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.969880+0000","last_change":"2024-04-14T15:13:31.820264+0000","last_active":"2024-04-14T15:13:38.969880+0000","last_peered":"2024-04-14T15:13:38.969880+0000","last_clean":"2024-04-14T15:13:38.969880+0000","last_became_active":"2024-04-14T15:13:09.998139+0000","last_became_peered":"2024-04-14T15:13:09.998139+0000","last_unstale":"2024-04-14T15:13:38.969880+0000","last_undegraded":"2024-04-14T15:13:38.969880+0000","last_fullsized":"2024-04-14T15:13:38.969880+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:13:08.799672+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:28.496304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022159749999999998,"stat_sum":{"num_bytes":150,"num_objects":15,"num_object_clones":10,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"112.6","version":"144'1","reported_seq":130,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.883721+0000","last_change":"2024-04-14T15:13:33.847535+0000","last_active":"2024-04-14T15:13:38.883721+0000","last_peered":"2024-04-14T15:13:38.883721+0000","last_clean":"2024-04-14T15:13:38.883721+0000","last_became_active":"2024-04-14T15:13:11.066907+0000","last_became_peered":"2024-04-14T15:13:11.066907+0000","last_unstale":"2024-04-14T15:13:38.883721+0000","last_undegraded":"2024-04-14T15:13:38.883721+0000","last_fullsized":"2024-04-14T15:13:38.883721+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:09.802949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:09.802949+0000","last_clean_scrub_stamp":"2024-04-14T15:13:09.802949+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:34.299840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002918309,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"6","length":"5"},{"start":"e","length":"1"}]},{"pgid":"110.18","version":"151'25","reported_seq":104,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.994492+0000","last_change":"2024-04-14T15:13:31.819345+0000","last_active":"2024-04-14T15:13:38.994492+0000","last_peered":"2024-04-14T15:13:38.994492+0000","last_clean":"2024-04-14T15:13:38.994492+0000","last_became_active":"2024-04-14T15:13:09.998803+0000","last_became_peered":"2024-04-14T15:13:09.998803+0000","last_unstale":"2024-04-14T15:13:38.994492+0000","last_undegraded":"2024-04-14T15:13:38.994492+0000","last_fullsized":"2024-04-14T15:13:38.994492+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:13:08.799672+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:12.835503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025881530000000002,"stat_sum":{"num_bytes":150,"num_objects":15,"num_object_clones":10,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"112.5","version":"127'6","reported_seq":135,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.842725+0000","last_change":"2024-04-14T15:13:33.834033+0000","last_active":"2024-04-14T15:13:38.842725+0000","last_peered":"2024-04-14T15:13:38.842725+0000","last_clean":"2024-04-14T15:13:38.842725+0000","last_became_active":"2024-04-14T15:13:10.847609+0000","last_became_peered":"2024-04-14T15:13:10.847609+0000","last_unstale":"2024-04-14T15:13:38.842725+0000","last_undegraded":"2024-04-14T15:13:38.842725+0000","last_fullsized":"2024-04-14T15:13:38.842725+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:09.802949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:09.802949+0000","last_clean_scrub_stamp":"2024-04-14T15:13:09.802949+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:39.470037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044407799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"6","length":"5"},{"start":"e","length":"1"}]},{"pgid":"110.1b","version":"151'25","reported_seq":104,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.926178+0000","last_change":"2024-04-14T15:13:31.855228+0000","last_active":"2024-04-14T15:13:38.926178+0000","last_peered":"2024-04-14T15:13:38.926178+0000","last_clean":"2024-04-14T15:13:38.926178+0000","last_became_active":"2024-04-14T15:13:09.946919+0000","last_became_peered":"2024-04-14T15:13:09.946919+0000","last_unstale":"2024-04-14T15:13:38.926178+0000","last_undegraded":"2024-04-14T15:13:38.926178+0000","last_fullsized":"2024-04-14T15:13:38.926178+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:13:08.799672+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:34.314121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0043648070000000001,"stat_sum":{"num_bytes":150,"num_objects":15,"num_object_clones":10,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"112.4","version":"0'0","reported_seq":129,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.839613+0000","last_change":"2024-04-14T15:13:33.834186+0000","last_active":"2024-04-14T15:13:38.839613+0000","last_peered":"2024-04-14T15:13:38.839613+0000","last_clean":"2024-04-14T15:13:38.839613+0000","last_became_active":"2024-04-14T15:13:10.991454+0000","last_became_peered":"2024-04-14T15:13:10.991454+0000","last_unstale":"2024-04-14T15:13:38.839613+0000","last_undegraded":"2024-04-14T15:13:38.839613+0000","last_fullsized":"2024-04-14T15:13:38.839613+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:09.802949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:09.802949+0000","last_clean_scrub_stamp":"2024-04-14T15:13:09.802949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:18.800349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030264720000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"5"},{"start":"e","length":"1"}]},{"pgid":"110.1a","version":"151'8","reported_seq":94,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.874773+0000","last_change":"2024-04-14T15:13:31.828510+0000","last_active":"2024-04-14T15:13:38.874773+0000","last_peered":"2024-04-14T15:13:38.874773+0000","last_clean":"2024-04-14T15:13:38.874773+0000","last_became_active":"2024-04-14T15:13:09.960319+0000","last_became_peered":"2024-04-14T15:13:09.960319+0000","last_unstale":"2024-04-14T15:13:38.874773+0000","last_undegraded":"2024-04-14T15:13:38.874773+0000","last_fullsized":"2024-04-14T15:13:38.874773+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:13:08.799672+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:18.527366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0064645620000000001,"stat_sum":{"num_bytes":49,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"112.3","version":"0'0","reported_seq":129,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.826246+0000","last_change":"2024-04-14T15:13:33.840085+0000","last_active":"2024-04-14T15:13:38.826246+0000","last_peered":"2024-04-14T15:13:38.826246+0000","last_clean":"2024-04-14T15:13:38.826246+0000","last_became_active":"2024-04-14T15:13:10.956597+0000","last_became_peered":"2024-04-14T15:13:10.956597+0000","last_unstale":"2024-04-14T15:13:38.826246+0000","last_undegraded":"2024-04-14T15:13:38.826246+0000","last_fullsized":"2024-04-14T15:13:38.826246+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:09.802949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:09.802949+0000","last_clean_scrub_stamp":"2024-04-14T15:13:09.802949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:30.621098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001191037,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6","length":"5"},{"start":"e","length":"1"}]},{"pgid":"110.1d","version":"151'23","reported_seq":103,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.930160+0000","last_change":"2024-04-14T15:13:31.857816+0000","last_active":"2024-04-14T15:13:38.930160+0000","last_peered":"2024-04-14T15:13:38.930160+0000","last_clean":"2024-04-14T15:13:38.930160+0000","last_became_active":"2024-04-14T15:13:09.868082+0000","last_became_peered":"2024-04-14T15:13:09.868082+0000","last_unstale":"2024-04-14T15:13:38.930160+0000","last_undegraded":"2024-04-14T15:13:38.930160+0000","last_fullsized":"2024-04-14T15:13:38.930160+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:13:08.799672+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:28.689580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012743758000000001,"stat_sum":{"num_bytes":139,"num_objects":14,"num_object_clones":9,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"112.2","version":"0'0","reported_seq":129,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.837698+0000","last_change":"2024-04-14T15:13:33.837374+0000","last_active":"2024-04-14T15:13:38.837698+0000","last_peered":"2024-04-14T15:13:38.837698+0000","last_clean":"2024-04-14T15:13:38.837698+0000","last_became_active":"2024-04-14T15:13:10.985929+0000","last_became_peered":"2024-04-14T15:13:10.985929+0000","last_unstale":"2024-04-14T15:13:38.837698+0000","last_undegraded":"2024-04-14T15:13:38.837698+0000","last_fullsized":"2024-04-14T15:13:38.837698+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:09.802949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:09.802949+0000","last_clean_scrub_stamp":"2024-04-14T15:13:09.802949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:22.518907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027608039999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"6","length":"5"},{"start":"e","length":"1"}]},{"pgid":"110.1c","version":"151'25","reported_seq":108,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:39.006156+0000","last_change":"2024-04-14T15:13:31.906550+0000","last_active":"2024-04-14T15:13:39.006156+0000","last_peered":"2024-04-14T15:13:39.006156+0000","last_clean":"2024-04-14T15:13:39.006156+0000","last_became_active":"2024-04-14T15:13:09.954091+0000","last_became_peered":"2024-04-14T15:13:09.954091+0000","last_unstale":"2024-04-14T15:13:39.006156+0000","last_undegraded":"2024-04-14T15:13:39.006156+0000","last_fullsized":"2024-04-14T15:13:39.006156+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:13:08.799672+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:36.650663+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.018861645999999999,"stat_sum":{"num_bytes":125,"num_objects":13,"num_object_clones":8,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":14,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"112.1","version":"137'20","reported_seq":150,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.824547+0000","last_change":"2024-04-14T15:13:33.842265+0000","last_active":"2024-04-14T15:13:38.824547+0000","last_peered":"2024-04-14T15:13:38.824547+0000","last_clean":"2024-04-14T15:13:38.824547+0000","last_became_active":"2024-04-14T15:13:10.895848+0000","last_became_peered":"2024-04-14T15:13:10.895848+0000","last_unstale":"2024-04-14T15:13:38.824547+0000","last_undegraded":"2024-04-14T15:13:38.824547+0000","last_fullsized":"2024-04-14T15:13:38.824547+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:09.802949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:09.802949+0000","last_clean_scrub_stamp":"2024-04-14T15:13:09.802949+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:28.897586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0045513209999999997,"stat_sum":{"num_bytes":1280,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":14,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"5"},{"start":"e","length":"1"}]},{"pgid":"110.1f","version":"151'5","reported_seq":92,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.842337+0000","last_change":"2024-04-14T15:13:31.836380+0000","last_active":"2024-04-14T15:13:38.842337+0000","last_peered":"2024-04-14T15:13:38.842337+0000","last_clean":"2024-04-14T15:13:38.842337+0000","last_became_active":"2024-04-14T15:13:09.999629+0000","last_became_peered":"2024-04-14T15:13:09.999629+0000","last_unstale":"2024-04-14T15:13:38.842337+0000","last_undegraded":"2024-04-14T15:13:38.842337+0000","last_fullsized":"2024-04-14T15:13:38.842337+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:13:08.799672+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:33.796124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010555880000000001,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"112.0","version":"0'0","reported_seq":129,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.869277+0000","last_change":"2024-04-14T15:13:33.832453+0000","last_active":"2024-04-14T15:13:38.869277+0000","last_peered":"2024-04-14T15:13:38.869277+0000","last_clean":"2024-04-14T15:13:38.869277+0000","last_became_active":"2024-04-14T15:13:10.884321+0000","last_became_peered":"2024-04-14T15:13:10.884321+0000","last_unstale":"2024-04-14T15:13:38.869277+0000","last_undegraded":"2024-04-14T15:13:38.869277+0000","last_fullsized":"2024-04-14T15:13:38.869277+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:09.802949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:09.802949+0000","last_clean_scrub_stamp":"2024-04-14T15:13:09.802949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:21.024429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060199199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"6","length":"5"},{"start":"e","length":"1"}]},{"pgid":"110.1e","version":"151'10","reported_seq":98,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.885261+0000","last_change":"2024-04-14T15:13:31.892343+0000","last_active":"2024-04-14T15:13:38.885261+0000","last_peered":"2024-04-14T15:13:38.885261+0000","last_clean":"2024-04-14T15:13:38.885261+0000","last_became_active":"2024-04-14T15:13:09.955932+0000","last_became_peered":"2024-04-14T15:13:09.955932+0000","last_unstale":"2024-04-14T15:13:38.885261+0000","last_undegraded":"2024-04-14T15:13:38.885261+0000","last_fullsized":"2024-04-14T15:13:38.885261+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:13:08.799672+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:10.580624+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.044420435000000001,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"112.1f","version":"130'9","reported_seq":143,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.887863+0000","last_change":"2024-04-14T15:13:33.843409+0000","last_active":"2024-04-14T15:13:38.887863+0000","last_peered":"2024-04-14T15:13:38.887863+0000","last_clean":"2024-04-14T15:13:38.887863+0000","last_became_active":"2024-04-14T15:13:11.038479+0000","last_became_peered":"2024-04-14T15:13:11.038479+0000","last_unstale":"2024-04-14T15:13:38.887863+0000","last_undegraded":"2024-04-14T15:13:38.887863+0000","last_fullsized":"2024-04-14T15:13:38.887863+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:09.802949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:09.802949+0000","last_clean_scrub_stamp":"2024-04-14T15:13:09.802949+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:30.567881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0035799,"stat_sum":{"num_bytes":384,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"6","length":"5"},{"start":"e","length":"1"}]},{"pgid":"110.1","version":"151'5","reported_seq":92,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.843130+0000","last_change":"2024-04-14T15:13:31.840886+0000","last_active":"2024-04-14T15:13:38.843130+0000","last_peered":"2024-04-14T15:13:38.843130+0000","last_clean":"2024-04-14T15:13:38.843130+0000","last_became_active":"2024-04-14T15:13:09.929562+0000","last_became_peered":"2024-04-14T15:13:09.929562+0000","last_unstale":"2024-04-14T15:13:38.843130+0000","last_undegraded":"2024-04-14T15:13:38.843130+0000","last_fullsized":"2024-04-14T15:13:38.843130+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:13:08.799672+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:29.950445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011417548,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"112.1e","version":"0'0","reported_seq":129,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.876624+0000","last_change":"2024-04-14T15:13:33.842482+0000","last_active":"2024-04-14T15:13:38.876624+0000","last_peered":"2024-04-14T15:13:38.876624+0000","last_clean":"2024-04-14T15:13:38.876624+0000","last_became_active":"2024-04-14T15:13:10.897839+0000","last_became_peered":"2024-04-14T15:13:10.897839+0000","last_unstale":"2024-04-14T15:13:38.876624+0000","last_undegraded":"2024-04-14T15:13:38.876624+0000","last_fullsized":"2024-04-14T15:13:38.876624+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:09.802949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:09.802949+0000","last_clean_scrub_stamp":"2024-04-14T15:13:09.802949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:22.250763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091885299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"6","length":"5"},{"start":"e","length":"1"}]},{"pgid":"110.0","version":"151'19","reported_seq":101,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.901095+0000","last_change":"2024-04-14T15:13:31.846906+0000","last_active":"2024-04-14T15:13:38.901095+0000","last_peered":"2024-04-14T15:13:38.901095+0000","last_clean":"2024-04-14T15:13:38.901095+0000","last_became_active":"2024-04-14T15:13:09.856407+0000","last_became_peered":"2024-04-14T15:13:09.856407+0000","last_unstale":"2024-04-14T15:13:38.901095+0000","last_undegraded":"2024-04-14T15:13:38.901095+0000","last_fullsized":"2024-04-14T15:13:38.901095+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:13:08.799672+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:29.176845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.022357385,"stat_sum":{"num_bytes":117,"num_objects":12,"num_object_clones":7,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"112.1d","version":"0'0","reported_seq":129,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.872466+0000","last_change":"2024-04-14T15:13:33.836181+0000","last_active":"2024-04-14T15:13:38.872466+0000","last_peered":"2024-04-14T15:13:38.872466+0000","last_clean":"2024-04-14T15:13:38.872466+0000","last_became_active":"2024-04-14T15:13:10.895905+0000","last_became_peered":"2024-04-14T15:13:10.895905+0000","last_unstale":"2024-04-14T15:13:38.872466+0000","last_undegraded":"2024-04-14T15:13:38.872466+0000","last_fullsized":"2024-04-14T15:13:38.872466+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:09.802949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:09.802949+0000","last_clean_scrub_stamp":"2024-04-14T15:13:09.802949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:36.887921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017604610000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"6","length":"5"},{"start":"e","length":"1"}]},{"pgid":"110.3","version":"151'15","reported_seq":98,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.941720+0000","last_change":"2024-04-14T15:13:31.830947+0000","last_active":"2024-04-14T15:13:38.941720+0000","last_peered":"2024-04-14T15:13:38.941720+0000","last_clean":"2024-04-14T15:13:38.941720+0000","last_became_active":"2024-04-14T15:13:09.997208+0000","last_became_peered":"2024-04-14T15:13:09.997208+0000","last_unstale":"2024-04-14T15:13:38.941720+0000","last_undegraded":"2024-04-14T15:13:38.941720+0000","last_fullsized":"2024-04-14T15:13:38.941720+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:13:08.799672+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:26.698300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0084448869999999999,"stat_sum":{"num_bytes":90,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"112.1c","version":"0'0","reported_seq":129,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.839468+0000","last_change":"2024-04-14T15:13:33.833889+0000","last_active":"2024-04-14T15:13:38.839468+0000","last_peered":"2024-04-14T15:13:38.839468+0000","last_clean":"2024-04-14T15:13:38.839468+0000","last_became_active":"2024-04-14T15:13:10.972890+0000","last_became_peered":"2024-04-14T15:13:10.972890+0000","last_unstale":"2024-04-14T15:13:38.839468+0000","last_undegraded":"2024-04-14T15:13:38.839468+0000","last_fullsized":"2024-04-14T15:13:38.839468+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:09.802949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:09.802949+0000","last_clean_scrub_stamp":"2024-04-14T15:13:09.802949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:30.507598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029382919999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"5"},{"start":"e","length":"1"}]},{"pgid":"110.2","version":"151'15","reported_seq":98,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.932426+0000","last_change":"2024-04-14T15:13:31.822872+0000","last_active":"2024-04-14T15:13:38.932426+0000","last_peered":"2024-04-14T15:13:38.932426+0000","last_clean":"2024-04-14T15:13:38.932426+0000","last_became_active":"2024-04-14T15:13:09.957187+0000","last_became_peered":"2024-04-14T15:13:09.957187+0000","last_unstale":"2024-04-14T15:13:38.932426+0000","last_undegraded":"2024-04-14T15:13:38.932426+0000","last_fullsized":"2024-04-14T15:13:38.932426+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:13:08.799672+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:33.655553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016792300000000001,"stat_sum":{"num_bytes":90,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"112.1b","version":"0'0","reported_seq":129,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.834656+0000","last_change":"2024-04-14T15:13:33.841646+0000","last_active":"2024-04-14T15:13:38.834656+0000","last_peered":"2024-04-14T15:13:38.834656+0000","last_clean":"2024-04-14T15:13:38.834656+0000","last_became_active":"2024-04-14T15:13:11.014097+0000","last_became_peered":"2024-04-14T15:13:11.014097+0000","last_unstale":"2024-04-14T15:13:38.834656+0000","last_undegraded":"2024-04-14T15:13:38.834656+0000","last_fullsized":"2024-04-14T15:13:38.834656+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:09.802949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:09.802949+0000","last_clean_scrub_stamp":"2024-04-14T15:13:09.802949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:29.415316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048951979999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"5"},{"start":"e","length":"1"}]},{"pgid":"110.5","version":"151'17","reported_seq":105,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.849407+0000","last_change":"2024-04-14T15:13:33.868919+0000","last_active":"2024-04-14T15:13:38.849407+0000","last_peered":"2024-04-14T15:13:38.849407+0000","last_clean":"2024-04-14T15:13:38.849407+0000","last_became_active":"2024-04-14T15:13:09.853689+0000","last_became_peered":"2024-04-14T15:13:09.853689+0000","last_unstale":"2024-04-14T15:13:38.849407+0000","last_undegraded":"2024-04-14T15:13:38.849407+0000","last_fullsized":"2024-04-14T15:13:38.849407+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:13:08.799672+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:24.141141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.035385613000000003,"stat_sum":{"num_bytes":77,"num_objects":9,"num_object_clones":4,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"112.1a","version":"143'4","reported_seq":132,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.848216+0000","last_change":"2024-04-14T15:13:33.840263+0000","last_active":"2024-04-14T15:13:38.848216+0000","last_peered":"2024-04-14T15:13:38.848216+0000","last_clean":"2024-04-14T15:13:38.848216+0000","last_became_active":"2024-04-14T15:13:10.930090+0000","last_became_peered":"2024-04-14T15:13:10.930090+0000","last_unstale":"2024-04-14T15:13:38.848216+0000","last_undegraded":"2024-04-14T15:13:38.848216+0000","last_fullsized":"2024-04-14T15:13:38.848216+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:09.802949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:09.802949+0000","last_clean_scrub_stamp":"2024-04-14T15:13:09.802949+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:20.124846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025694070000000001,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"6","length":"5"},{"start":"e","length":"1"}]},{"pgid":"110.4","version":"151'15","reported_seq":98,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.908670+0000","last_change":"2024-04-14T15:13:31.856629+0000","last_active":"2024-04-14T15:13:38.908670+0000","last_peered":"2024-04-14T15:13:38.908670+0000","last_clean":"2024-04-14T15:13:38.908670+0000","last_became_active":"2024-04-14T15:13:09.958847+0000","last_became_peered":"2024-04-14T15:13:09.958847+0000","last_unstale":"2024-04-14T15:13:38.908670+0000","last_undegraded":"2024-04-14T15:13:38.908670+0000","last_fullsized":"2024-04-14T15:13:38.908670+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:13:08.799672+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:27.719691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015455567,"stat_sum":{"num_bytes":90,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"112.19","version":"0'0","reported_seq":129,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.885826+0000","last_change":"2024-04-14T15:13:33.841283+0000","last_active":"2024-04-14T15:13:38.885826+0000","last_peered":"2024-04-14T15:13:38.885826+0000","last_clean":"2024-04-14T15:13:38.885826+0000","last_became_active":"2024-04-14T15:13:11.027285+0000","last_became_peered":"2024-04-14T15:13:11.027285+0000","last_unstale":"2024-04-14T15:13:38.885826+0000","last_undegraded":"2024-04-14T15:13:38.885826+0000","last_fullsized":"2024-04-14T15:13:38.885826+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:09.802949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:09.802949+0000","last_clean_scrub_stamp":"2024-04-14T15:13:09.802949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:31.247937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0042288919999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"6","length":"5"},{"start":"e","length":"1"}]},{"pgid":"110.7","version":"151'5","reported_seq":92,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.874700+0000","last_change":"2024-04-14T15:13:31.858935+0000","last_active":"2024-04-14T15:13:38.874700+0000","last_peered":"2024-04-14T15:13:38.874700+0000","last_clean":"2024-04-14T15:13:38.874700+0000","last_became_active":"2024-04-14T15:13:09.936631+0000","last_became_peered":"2024-04-14T15:13:09.936631+0000","last_unstale":"2024-04-14T15:13:38.874700+0000","last_undegraded":"2024-04-14T15:13:38.874700+0000","last_fullsized":"2024-04-14T15:13:38.874700+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:13:08.799672+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:17.101938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016100272999999998,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"112.18","version":"0'0","reported_seq":129,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.844411+0000","last_change":"2024-04-14T15:13:33.870030+0000","last_active":"2024-04-14T15:13:38.844411+0000","last_peered":"2024-04-14T15:13:38.844411+0000","last_clean":"2024-04-14T15:13:38.844411+0000","last_became_active":"2024-04-14T15:13:10.877423+0000","last_became_peered":"2024-04-14T15:13:10.877423+0000","last_unstale":"2024-04-14T15:13:38.844411+0000","last_undegraded":"2024-04-14T15:13:38.844411+0000","last_fullsized":"2024-04-14T15:13:38.844411+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:09.802949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:09.802949+0000","last_clean_scrub_stamp":"2024-04-14T15:13:09.802949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:35.716094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.039454917999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"6","length":"5"},{"start":"e","length":"1"}]},{"pgid":"110.6","version":"151'18","reported_seq":100,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.925127+0000","last_change":"2024-04-14T15:13:31.857626+0000","last_active":"2024-04-14T15:13:38.925127+0000","last_peered":"2024-04-14T15:13:38.925127+0000","last_clean":"2024-04-14T15:13:38.925127+0000","last_became_active":"2024-04-14T15:13:09.930125+0000","last_became_peered":"2024-04-14T15:13:09.930125+0000","last_unstale":"2024-04-14T15:13:38.925127+0000","last_undegraded":"2024-04-14T15:13:38.925127+0000","last_fullsized":"2024-04-14T15:13:38.925127+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:13:08.799672+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:18.408268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015594758,"stat_sum":{"num_bytes":109,"num_objects":11,"num_object_clones":7,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"112.17","version":"0'0","reported_seq":129,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.871902+0000","last_change":"2024-04-14T15:13:33.840232+0000","last_active":"2024-04-14T15:13:38.871902+0000","last_peered":"2024-04-14T15:13:38.871902+0000","last_clean":"2024-04-14T15:13:38.871902+0000","last_became_active":"2024-04-14T15:13:10.973251+0000","last_became_peered":"2024-04-14T15:13:10.973251+0000","last_unstale":"2024-04-14T15:13:38.871902+0000","last_undegraded":"2024-04-14T15:13:38.871902+0000","last_fullsized":"2024-04-14T15:13:38.871902+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:09.802949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:09.802949+0000","last_clean_scrub_stamp":"2024-04-14T15:13:09.802949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:15.364675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0061194600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"6","length":"5"},{"start":"e","length":"1"}]},{"pgid":"110.9","version":"151'5","reported_seq":92,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.825217+0000","last_change":"2024-04-14T15:13:31.819949+0000","last_active":"2024-04-14T15:13:38.825217+0000","last_peered":"2024-04-14T15:13:38.825217+0000","last_clean":"2024-04-14T15:13:38.825217+0000","last_became_active":"2024-04-14T15:13:09.957324+0000","last_became_peered":"2024-04-14T15:13:09.957324+0000","last_unstale":"2024-04-14T15:13:38.825217+0000","last_undegraded":"2024-04-14T15:13:38.825217+0000","last_fullsized":"2024-04-14T15:13:38.825217+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:13:08.799672+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:22.353113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021015529999999999,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"112.16","version":"0'0","reported_seq":129,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.858489+0000","last_change":"2024-04-14T15:13:33.850444+0000","last_active":"2024-04-14T15:13:38.858489+0000","last_peered":"2024-04-14T15:13:38.858489+0000","last_clean":"2024-04-14T15:13:38.858489+0000","last_became_active":"2024-04-14T15:13:10.946086+0000","last_became_peered":"2024-04-14T15:13:10.946086+0000","last_unstale":"2024-04-14T15:13:38.858489+0000","last_undegraded":"2024-04-14T15:13:38.858489+0000","last_fullsized":"2024-04-14T15:13:38.858489+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:09.802949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:09.802949+0000","last_clean_scrub_stamp":"2024-04-14T15:13:09.802949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:38.652397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012149800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"6","length":"5"},{"start":"e","length":"1"}]},{"pgid":"110.8","version":"150'3","reported_seq":91,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.826023+0000","last_change":"2024-04-14T15:13:31.832777+0000","last_active":"2024-04-14T15:13:38.826023+0000","last_peered":"2024-04-14T15:13:38.826023+0000","last_clean":"2024-04-14T15:13:38.826023+0000","last_became_active":"2024-04-14T15:13:09.952046+0000","last_became_peered":"2024-04-14T15:13:09.952046+0000","last_unstale":"2024-04-14T15:13:38.826023+0000","last_undegraded":"2024-04-14T15:13:38.826023+0000","last_fullsized":"2024-04-14T15:13:38.826023+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:13:08.799672+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:25.996814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.005243274,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"112.15","version":"0'0","reported_seq":129,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.845818+0000","last_change":"2024-04-14T15:13:33.870617+0000","last_active":"2024-04-14T15:13:38.845818+0000","last_peered":"2024-04-14T15:13:38.845818+0000","last_clean":"2024-04-14T15:13:38.845818+0000","last_became_active":"2024-04-14T15:13:10.879703+0000","last_became_peered":"2024-04-14T15:13:10.879703+0000","last_unstale":"2024-04-14T15:13:38.845818+0000","last_undegraded":"2024-04-14T15:13:38.845818+0000","last_fullsized":"2024-04-14T15:13:38.845818+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:09.802949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:09.802949+0000","last_clean_scrub_stamp":"2024-04-14T15:13:09.802949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:16.407822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.037426234000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"6","length":"5"},{"start":"e","length":"1"}]},{"pgid":"110.b","version":"151'20","reported_seq":101,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.978267+0000","last_change":"2024-04-14T15:13:31.856715+0000","last_active":"2024-04-14T15:13:38.978267+0000","last_peered":"2024-04-14T15:13:38.978267+0000","last_clean":"2024-04-14T15:13:38.978267+0000","last_became_active":"2024-04-14T15:13:09.964629+0000","last_became_peered":"2024-04-14T15:13:09.964629+0000","last_unstale":"2024-04-14T15:13:38.978267+0000","last_undegraded":"2024-04-14T15:13:38.978267+0000","last_fullsized":"2024-04-14T15:13:38.978267+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:13:08.799672+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:31.803209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0053331009999999998,"stat_sum":{"num_bytes":120,"num_objects":12,"num_object_clones":8,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"112.14","version":"0'0","reported_seq":129,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.854372+0000","last_change":"2024-04-14T15:13:33.847630+0000","last_active":"2024-04-14T15:13:38.854372+0000","last_peered":"2024-04-14T15:13:38.854372+0000","last_clean":"2024-04-14T15:13:38.854372+0000","last_became_active":"2024-04-14T15:13:10.903383+0000","last_became_peered":"2024-04-14T15:13:10.903383+0000","last_unstale":"2024-04-14T15:13:38.854372+0000","last_undegraded":"2024-04-14T15:13:38.854372+0000","last_fullsized":"2024-04-14T15:13:38.854372+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:09.802949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:09.802949+0000","last_clean_scrub_stamp":"2024-04-14T15:13:09.802949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:11.958840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004250729,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"6","length":"5"},{"start":"e","length":"1"}]},{"pgid":"110.a","version":"151'18","reported_seq":100,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.914264+0000","last_change":"2024-04-14T15:13:31.828989+0000","last_active":"2024-04-14T15:13:38.914264+0000","last_peered":"2024-04-14T15:13:38.914264+0000","last_clean":"2024-04-14T15:13:38.914264+0000","last_became_active":"2024-04-14T15:13:09.909752+0000","last_became_peered":"2024-04-14T15:13:09.909752+0000","last_unstale":"2024-04-14T15:13:38.914264+0000","last_undegraded":"2024-04-14T15:13:38.914264+0000","last_fullsized":"2024-04-14T15:13:38.914264+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:13:08.799672+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:36.284090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.006744145,"stat_sum":{"num_bytes":109,"num_objects":11,"num_object_clones":7,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"112.13","version":"0'0","reported_seq":129,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.826223+0000","last_change":"2024-04-14T15:13:33.840170+0000","last_active":"2024-04-14T15:13:38.826223+0000","last_peered":"2024-04-14T15:13:38.826223+0000","last_clean":"2024-04-14T15:13:38.826223+0000","last_became_active":"2024-04-14T15:13:11.043548+0000","last_became_peered":"2024-04-14T15:13:11.043548+0000","last_unstale":"2024-04-14T15:13:38.826223+0000","last_undegraded":"2024-04-14T15:13:38.826223+0000","last_fullsized":"2024-04-14T15:13:38.826223+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:09.802949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:09.802949+0000","last_clean_scrub_stamp":"2024-04-14T15:13:09.802949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:12.197564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003364864,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6","length":"5"},{"start":"e","length":"1"}]},{"pgid":"110.d","version":"151'25","reported_seq":104,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.948224+0000","last_change":"2024-04-14T15:13:31.892572+0000","last_active":"2024-04-14T15:13:38.948224+0000","last_peered":"2024-04-14T15:13:38.948224+0000","last_clean":"2024-04-14T15:13:38.948224+0000","last_became_active":"2024-04-14T15:13:09.944223+0000","last_became_peered":"2024-04-14T15:13:09.944223+0000","last_unstale":"2024-04-14T15:13:38.948224+0000","last_undegraded":"2024-04-14T15:13:38.948224+0000","last_fullsized":"2024-04-14T15:13:38.948224+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:13:08.799672+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:33.476199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.020921474999999998,"stat_sum":{"num_bytes":150,"num_objects":15,"num_object_clones":10,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"112.12","version":"0'0","reported_seq":129,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.836635+0000","last_change":"2024-04-14T15:13:33.869547+0000","last_active":"2024-04-14T15:13:38.836635+0000","last_peered":"2024-04-14T15:13:38.836635+0000","last_clean":"2024-04-14T15:13:38.836635+0000","last_became_active":"2024-04-14T15:13:10.942342+0000","last_became_peered":"2024-04-14T15:13:10.942342+0000","last_unstale":"2024-04-14T15:13:38.836635+0000","last_undegraded":"2024-04-14T15:13:38.836635+0000","last_fullsized":"2024-04-14T15:13:38.836635+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:09.802949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:09.802949+0000","last_clean_scrub_stamp":"2024-04-14T15:13:09.802949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:21.913594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.041896640999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"6","length":"5"},{"start":"e","length":"1"}]},{"pgid":"110.c","version":"151'5","reported_seq":92,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.858531+0000","last_change":"2024-04-14T15:13:31.906885+0000","last_active":"2024-04-14T15:13:38.858531+0000","last_peered":"2024-04-14T15:13:38.858531+0000","last_clean":"2024-04-14T15:13:38.858531+0000","last_became_active":"2024-04-14T15:13:09.984609+0000","last_became_peered":"2024-04-14T15:13:09.984609+0000","last_unstale":"2024-04-14T15:13:38.858531+0000","last_undegraded":"2024-04-14T15:13:38.858531+0000","last_fullsized":"2024-04-14T15:13:38.858531+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:13:08.799672+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:18.599801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01712311,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"112.11","version":"0'0","reported_seq":129,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.836807+0000","last_change":"2024-04-14T15:13:33.869282+0000","last_active":"2024-04-14T15:13:38.836807+0000","last_peered":"2024-04-14T15:13:38.836807+0000","last_clean":"2024-04-14T15:13:38.836807+0000","last_became_active":"2024-04-14T15:13:10.877575+0000","last_became_peered":"2024-04-14T15:13:10.877575+0000","last_unstale":"2024-04-14T15:13:38.836807+0000","last_undegraded":"2024-04-14T15:13:38.836807+0000","last_fullsized":"2024-04-14T15:13:38.836807+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:09.802949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:09.802949+0000","last_clean_scrub_stamp":"2024-04-14T15:13:09.802949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:34.264575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.041907132999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"6","length":"5"},{"start":"e","length":"1"}]},{"pgid":"110.f","version":"151'15","reported_seq":98,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.954067+0000","last_change":"2024-04-14T15:13:31.898618+0000","last_active":"2024-04-14T15:13:38.954067+0000","last_peered":"2024-04-14T15:13:38.954067+0000","last_clean":"2024-04-14T15:13:38.954067+0000","last_became_active":"2024-04-14T15:13:09.964057+0000","last_became_peered":"2024-04-14T15:13:09.964057+0000","last_unstale":"2024-04-14T15:13:38.954067+0000","last_undegraded":"2024-04-14T15:13:38.954067+0000","last_fullsized":"2024-04-14T15:13:38.954067+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:13:08.799672+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:29.892049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011988861999999999,"stat_sum":{"num_bytes":90,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"112.10","version":"0'0","reported_seq":129,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.846049+0000","last_change":"2024-04-14T15:13:33.870930+0000","last_active":"2024-04-14T15:13:38.846049+0000","last_peered":"2024-04-14T15:13:38.846049+0000","last_clean":"2024-04-14T15:13:38.846049+0000","last_became_active":"2024-04-14T15:13:10.878080+0000","last_became_peered":"2024-04-14T15:13:10.878080+0000","last_unstale":"2024-04-14T15:13:38.846049+0000","last_undegraded":"2024-04-14T15:13:38.846049+0000","last_fullsized":"2024-04-14T15:13:38.846049+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:09.802949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:09.802949+0000","last_clean_scrub_stamp":"2024-04-14T15:13:09.802949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:11.820261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033582473000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"6","length":"5"},{"start":"e","length":"1"}]},{"pgid":"110.e","version":"151'21","reported_seq":102,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.931300+0000","last_change":"2024-04-14T15:13:31.853707+0000","last_active":"2024-04-14T15:13:38.931300+0000","last_peered":"2024-04-14T15:13:38.931300+0000","last_clean":"2024-04-14T15:13:38.931300+0000","last_became_active":"2024-04-14T15:13:09.984980+0000","last_became_peered":"2024-04-14T15:13:09.984980+0000","last_unstale":"2024-04-14T15:13:38.931300+0000","last_undegraded":"2024-04-14T15:13:38.931300+0000","last_fullsized":"2024-04-14T15:13:38.931300+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:13:08.799672+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:21.392437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012145575,"stat_sum":{"num_bytes":128,"num_objects":13,"num_object_clones":8,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"13.1","version":"109'5","reported_seq":102,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.870459+0000","last_change":"2024-04-14T15:12:56.415926+0000","last_active":"2024-04-14T15:13:38.870459+0000","last_peered":"2024-04-14T15:13:38.870459+0000","last_clean":"2024-04-14T15:13:38.870459+0000","last_became_active":"2024-04-14T15:12:56.404932+0000","last_became_peered":"2024-04-14T15:12:56.404932+0000","last_unstale":"2024-04-14T15:13:38.870459+0000","last_undegraded":"2024-04-14T15:13:38.870459+0000","last_fullsized":"2024-04-14T15:13:38.870459+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:55.380333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":97,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.828478+0000","last_change":"2024-04-14T15:12:56.247166+0000","last_active":"2024-04-14T15:13:38.828478+0000","last_peered":"2024-04-14T15:13:38.828478+0000","last_clean":"2024-04-14T15:13:38.828478+0000","last_became_active":"2024-04-14T15:12:56.246864+0000","last_became_peered":"2024-04-14T15:12:56.246864+0000","last_unstale":"2024-04-14T15:13:38.828478+0000","last_undegraded":"2024-04-14T15:13:38.828478+0000","last_fullsized":"2024-04-14T15:13:38.828478+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:20.494014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"85'9","reported_seq":249,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.842924+0000","last_change":"2024-04-14T15:12:37.019655+0000","last_active":"2024-04-14T15:13:38.842924+0000","last_peered":"2024-04-14T15:13:38.842924+0000","last_clean":"2024-04-14T15:13:38.842924+0000","last_became_active":"2024-04-14T15:09:47.045441+0000","last_became_peered":"2024-04-14T15:09:47.045441+0000","last_unstale":"2024-04-14T15:13:38.842924+0000","last_undegraded":"2024-04-14T15:13:38.842924+0000","last_fullsized":"2024-04-14T15:13:38.842924+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9","last_scrub_stamp":"2024-04-14T15:12:37.019536+0000","last_deep_scrub":"85'9","last_deep_scrub_stamp":"2024-04-14T15:10:51.950580+0000","last_clean_scrub_stamp":"2024-04-14T15:12:37.019536+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:00.217464+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"72'15","reported_seq":221,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.825318+0000","last_change":"2024-04-14T15:13:21.857209+0000","last_active":"2024-04-14T15:13:38.825318+0000","last_peered":"2024-04-14T15:13:38.825318+0000","last_clean":"2024-04-14T15:13:38.825318+0000","last_became_active":"2024-04-14T15:09:45.180160+0000","last_became_peered":"2024-04-14T15:09:45.180160+0000","last_unstale":"2024-04-14T15:13:38.825318+0000","last_undegraded":"2024-04-14T15:13:38.825318+0000","last_fullsized":"2024-04-14T15:13:38.825318+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-04-14T15:13:21.857048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:13:21.857048+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:29.401213+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":200,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.859254+0000","last_change":"2024-04-14T15:13:17.346045+0000","last_active":"2024-04-14T15:13:38.859254+0000","last_peered":"2024-04-14T15:13:38.859254+0000","last_clean":"2024-04-14T15:13:38.859254+0000","last_became_active":"2024-04-14T15:09:43.021460+0000","last_became_peered":"2024-04-14T15:09:43.021460+0000","last_unstale":"2024-04-14T15:13:38.859254+0000","last_undegraded":"2024-04-14T15:13:38.859254+0000","last_fullsized":"2024-04-14T15:13:38.859254+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:17.345909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:13:17.345909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:33.916250+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":188,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.826905+0000","last_change":"2024-04-14T15:13:36.719842+0000","last_active":"2024-04-14T15:13:38.826905+0000","last_peered":"2024-04-14T15:13:38.826905+0000","last_clean":"2024-04-14T15:13:38.826905+0000","last_became_active":"2024-04-14T15:09:49.128070+0000","last_became_peered":"2024-04-14T15:09:49.128070+0000","last_unstale":"2024-04-14T15:13:38.826905+0000","last_undegraded":"2024-04-14T15:13:38.826905+0000","last_fullsized":"2024-04-14T15:13:38.826905+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:36.719765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:13:36.719765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:40.986313+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1c","version":"108'8","reported_seq":106,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.858743+0000","last_change":"2024-04-14T15:12:56.364615+0000","last_active":"2024-04-14T15:13:38.858743+0000","last_peered":"2024-04-14T15:13:38.858743+0000","last_clean":"2024-04-14T15:13:38.858743+0000","last_became_active":"2024-04-14T15:12:56.364330+0000","last_became_peered":"2024-04-14T15:12:56.364330+0000","last_unstale":"2024-04-14T15:13:38.858743+0000","last_undegraded":"2024-04-14T15:13:38.858743+0000","last_fullsized":"2024-04-14T15:13:38.858743+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:19.300825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":97,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.826770+0000","last_change":"2024-04-14T15:12:56.325685+0000","last_active":"2024-04-14T15:13:38.826770+0000","last_peered":"2024-04-14T15:13:38.826770+0000","last_clean":"2024-04-14T15:13:38.826770+0000","last_became_active":"2024-04-14T15:12:56.319358+0000","last_became_peered":"2024-04-14T15:12:56.319358+0000","last_unstale":"2024-04-14T15:13:38.826770+0000","last_undegraded":"2024-04-14T15:13:38.826770+0000","last_fullsized":"2024-04-14T15:13:38.826770+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:19.234823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":97,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.847891+0000","last_change":"2024-04-14T15:12:56.363693+0000","last_active":"2024-04-14T15:13:38.847891+0000","last_peered":"2024-04-14T15:13:38.847891+0000","last_clean":"2024-04-14T15:13:38.847891+0000","last_became_active":"2024-04-14T15:12:56.355348+0000","last_became_peered":"2024-04-14T15:12:56.355348+0000","last_unstale":"2024-04-14T15:13:38.847891+0000","last_undegraded":"2024-04-14T15:13:38.847891+0000","last_fullsized":"2024-04-14T15:13:38.847891+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:05.657487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"72'9","reported_seq":212,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.885556+0000","last_change":"2024-04-14T15:12:28.969215+0000","last_active":"2024-04-14T15:13:38.885556+0000","last_peered":"2024-04-14T15:13:38.885556+0000","last_clean":"2024-04-14T15:13:38.885556+0000","last_became_active":"2024-04-14T15:09:45.037179+0000","last_became_peered":"2024-04-14T15:09:45.037179+0000","last_unstale":"2024-04-14T15:13:38.885556+0000","last_undegraded":"2024-04-14T15:13:38.885556+0000","last_fullsized":"2024-04-14T15:13:38.885556+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-04-14T15:12:28.969074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:12:28.969074+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":200,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.843057+0000","last_change":"2024-04-14T15:12:26.367861+0000","last_active":"2024-04-14T15:13:38.843057+0000","last_peered":"2024-04-14T15:13:38.843057+0000","last_clean":"2024-04-14T15:13:38.843057+0000","last_became_active":"2024-04-14T15:09:43.020044+0000","last_became_peered":"2024-04-14T15:09:43.020044+0000","last_unstale":"2024-04-14T15:13:38.843057+0000","last_undegraded":"2024-04-14T15:13:38.843057+0000","last_fullsized":"2024-04-14T15:13:38.843057+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:26.367741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:12:26.367741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:50.128645+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":192,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.849859+0000","last_change":"2024-04-14T15:12:38.441705+0000","last_active":"2024-04-14T15:13:38.849859+0000","last_peered":"2024-04-14T15:13:38.849859+0000","last_clean":"2024-04-14T15:13:38.849859+0000","last_became_active":"2024-04-14T15:09:47.046153+0000","last_became_peered":"2024-04-14T15:09:47.046153+0000","last_unstale":"2024-04-14T15:13:38.849859+0000","last_undegraded":"2024-04-14T15:13:38.849859+0000","last_fullsized":"2024-04-14T15:13:38.849859+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:38.441583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:12:38.441583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:58.485709+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":188,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.890531+0000","last_change":"2024-04-14T15:12:25.021726+0000","last_active":"2024-04-14T15:13:38.890531+0000","last_peered":"2024-04-14T15:13:38.890531+0000","last_clean":"2024-04-14T15:13:38.890531+0000","last_became_active":"2024-04-14T15:09:49.061495+0000","last_became_peered":"2024-04-14T15:09:49.061495+0000","last_unstale":"2024-04-14T15:13:38.890531+0000","last_undegraded":"2024-04-14T15:13:38.890531+0000","last_fullsized":"2024-04-14T15:13:38.890531+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:25.021630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:12:25.021630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:51.043752+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":97,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.828262+0000","last_change":"2024-04-14T15:12:56.363352+0000","last_active":"2024-04-14T15:13:38.828262+0000","last_peered":"2024-04-14T15:13:38.828262+0000","last_clean":"2024-04-14T15:13:38.828262+0000","last_became_active":"2024-04-14T15:12:56.359459+0000","last_became_peered":"2024-04-14T15:12:56.359459+0000","last_unstale":"2024-04-14T15:13:38.828262+0000","last_undegraded":"2024-04-14T15:13:38.828262+0000","last_fullsized":"2024-04-14T15:13:38.828262+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:24.879243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.3","version":"109'5","reported_seq":102,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.886089+0000","last_change":"2024-04-14T15:12:56.531318+0000","last_active":"2024-04-14T15:13:38.886089+0000","last_peered":"2024-04-14T15:13:38.886089+0000","last_clean":"2024-04-14T15:13:38.886089+0000","last_became_active":"2024-04-14T15:12:56.529451+0000","last_became_peered":"2024-04-14T15:12:56.529451+0000","last_unstale":"2024-04-14T15:13:38.886089+0000","last_undegraded":"2024-04-14T15:13:38.886089+0000","last_fullsized":"2024-04-14T15:13:38.886089+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:05.197700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":97,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.846260+0000","last_change":"2024-04-14T15:12:56.246956+0000","last_active":"2024-04-14T15:13:38.846260+0000","last_peered":"2024-04-14T15:13:38.846260+0000","last_clean":"2024-04-14T15:13:38.846260+0000","last_became_active":"2024-04-14T15:12:56.246062+0000","last_became_peered":"2024-04-14T15:12:56.246062+0000","last_unstale":"2024-04-14T15:13:38.846260+0000","last_undegraded":"2024-04-14T15:13:38.846260+0000","last_fullsized":"2024-04-14T15:13:38.846260+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:04.773310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"72'19","reported_seq":227,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.871982+0000","last_change":"2024-04-14T15:12:38.142096+0000","last_active":"2024-04-14T15:13:38.871982+0000","last_peered":"2024-04-14T15:13:38.871982+0000","last_clean":"2024-04-14T15:13:38.871982+0000","last_became_active":"2024-04-14T15:09:45.178782+0000","last_became_peered":"2024-04-14T15:09:45.178782+0000","last_unstale":"2024-04-14T15:13:38.871982+0000","last_undegraded":"2024-04-14T15:13:38.871982+0000","last_fullsized":"2024-04-14T15:13:38.871982+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'19","last_scrub_stamp":"2024-04-14T15:12:38.141982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:12:38.141982+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:02.486682+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":208,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.859152+0000","last_change":"2024-04-14T15:13:34.342920+0000","last_active":"2024-04-14T15:13:38.859152+0000","last_peered":"2024-04-14T15:13:38.859152+0000","last_clean":"2024-04-14T15:13:38.859152+0000","last_became_active":"2024-04-14T15:09:43.020268+0000","last_became_peered":"2024-04-14T15:09:43.020268+0000","last_unstale":"2024-04-14T15:13:38.859152+0000","last_undegraded":"2024-04-14T15:13:38.859152+0000","last_fullsized":"2024-04-14T15:13:38.859152+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:34.342793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:34.342793+0000","last_clean_scrub_stamp":"2024-04-14T15:13:34.342793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:42.349015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":192,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.862021+0000","last_change":"2024-04-14T15:13:16.363599+0000","last_active":"2024-04-14T15:13:38.862021+0000","last_peered":"2024-04-14T15:13:38.862021+0000","last_clean":"2024-04-14T15:13:38.862021+0000","last_became_active":"2024-04-14T15:09:47.045542+0000","last_became_peered":"2024-04-14T15:09:47.045542+0000","last_unstale":"2024-04-14T15:13:38.862021+0000","last_undegraded":"2024-04-14T15:13:38.862021+0000","last_fullsized":"2024-04-14T15:13:38.862021+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:16.363478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:13:16.363478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:24.041235+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":188,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.839643+0000","last_change":"2024-04-14T15:13:30.197647+0000","last_active":"2024-04-14T15:13:38.839643+0000","last_peered":"2024-04-14T15:13:38.839643+0000","last_clean":"2024-04-14T15:13:38.839643+0000","last_became_active":"2024-04-14T15:09:49.065397+0000","last_became_peered":"2024-04-14T15:09:49.065397+0000","last_unstale":"2024-04-14T15:13:38.839643+0000","last_undegraded":"2024-04-14T15:13:38.839643+0000","last_fullsized":"2024-04-14T15:13:38.839643+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:30.197500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:13:30.197500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:32.195058+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":97,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.848844+0000","last_change":"2024-04-14T15:12:56.200223+0000","last_active":"2024-04-14T15:13:38.848844+0000","last_peered":"2024-04-14T15:13:38.848844+0000","last_clean":"2024-04-14T15:13:38.848844+0000","last_became_active":"2024-04-14T15:12:56.183682+0000","last_became_peered":"2024-04-14T15:12:56.183682+0000","last_unstale":"2024-04-14T15:13:38.848844+0000","last_undegraded":"2024-04-14T15:13:38.848844+0000","last_fullsized":"2024-04-14T15:13:38.848844+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:17.915602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":97,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.872174+0000","last_change":"2024-04-14T15:12:56.407193+0000","last_active":"2024-04-14T15:13:38.872174+0000","last_peered":"2024-04-14T15:13:38.872174+0000","last_clean":"2024-04-14T15:13:38.872174+0000","last_became_active":"2024-04-14T15:12:56.400306+0000","last_became_peered":"2024-04-14T15:12:56.400306+0000","last_unstale":"2024-04-14T15:13:38.872174+0000","last_undegraded":"2024-04-14T15:13:38.872174+0000","last_fullsized":"2024-04-14T15:13:38.872174+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:24.979167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":97,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.849517+0000","last_change":"2024-04-14T15:12:56.477020+0000","last_active":"2024-04-14T15:13:38.849517+0000","last_peered":"2024-04-14T15:13:38.849517+0000","last_clean":"2024-04-14T15:13:38.849517+0000","last_became_active":"2024-04-14T15:12:56.474322+0000","last_became_peered":"2024-04-14T15:12:56.474322+0000","last_unstale":"2024-04-14T15:13:38.849517+0000","last_undegraded":"2024-04-14T15:13:38.849517+0000","last_fullsized":"2024-04-14T15:13:38.849517+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:55.169995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"72'17","reported_seq":272,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.850264+0000","last_change":"2024-04-14T15:13:18.575120+0000","last_active":"2024-04-14T15:13:38.850264+0000","last_peered":"2024-04-14T15:13:38.850264+0000","last_clean":"2024-04-14T15:13:38.850264+0000","last_became_active":"2024-04-14T15:13:12.933907+0000","last_became_peered":"2024-04-14T15:13:12.933907+0000","last_unstale":"2024-04-14T15:13:38.850264+0000","last_undegraded":"2024-04-14T15:13:38.850264+0000","last_fullsized":"2024-04-14T15:13:38.850264+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'17","last_scrub_stamp":"2024-04-14T15:13:18.574993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:13:18.574993+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:23.951036+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":200,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.837833+0000","last_change":"2024-04-14T15:13:20.212603+0000","last_active":"2024-04-14T15:13:38.837833+0000","last_peered":"2024-04-14T15:13:38.837833+0000","last_clean":"2024-04-14T15:13:38.837833+0000","last_became_active":"2024-04-14T15:09:43.022351+0000","last_became_peered":"2024-04-14T15:09:43.022351+0000","last_unstale":"2024-04-14T15:13:38.837833+0000","last_undegraded":"2024-04-14T15:13:38.837833+0000","last_fullsized":"2024-04-14T15:13:38.837833+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:20.212523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:13:20.212523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:44.109813+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":184,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.841478+0000","last_change":"2024-04-14T15:12:19.226834+0000","last_active":"2024-04-14T15:13:38.841478+0000","last_peered":"2024-04-14T15:13:38.841478+0000","last_clean":"2024-04-14T15:13:38.841478+0000","last_became_active":"2024-04-14T15:09:47.042810+0000","last_became_peered":"2024-04-14T15:09:47.042810+0000","last_unstale":"2024-04-14T15:13:38.841478+0000","last_undegraded":"2024-04-14T15:13:38.841478+0000","last_fullsized":"2024-04-14T15:13:38.841478+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:19.226701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:12:19.226701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":188,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.854157+0000","last_change":"2024-04-14T15:12:42.342665+0000","last_active":"2024-04-14T15:13:38.854157+0000","last_peered":"2024-04-14T15:13:38.854157+0000","last_clean":"2024-04-14T15:13:38.854157+0000","last_became_active":"2024-04-14T15:09:49.069594+0000","last_became_peered":"2024-04-14T15:09:49.069594+0000","last_unstale":"2024-04-14T15:13:38.854157+0000","last_undegraded":"2024-04-14T15:13:38.854157+0000","last_fullsized":"2024-04-14T15:13:38.854157+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:42.342544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:12:42.342544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:42.834751+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":97,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.837116+0000","last_change":"2024-04-14T15:12:56.476000+0000","last_active":"2024-04-14T15:13:38.837116+0000","last_peered":"2024-04-14T15:13:38.837116+0000","last_clean":"2024-04-14T15:13:38.837116+0000","last_became_active":"2024-04-14T15:12:56.470749+0000","last_became_peered":"2024-04-14T15:12:56.470749+0000","last_unstale":"2024-04-14T15:13:38.837116+0000","last_undegraded":"2024-04-14T15:13:38.837116+0000","last_fullsized":"2024-04-14T15:13:38.837116+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:00.938994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":97,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.850367+0000","last_change":"2024-04-14T15:12:56.371816+0000","last_active":"2024-04-14T15:13:38.850367+0000","last_peered":"2024-04-14T15:13:38.850367+0000","last_clean":"2024-04-14T15:13:38.850367+0000","last_became_active":"2024-04-14T15:12:56.368789+0000","last_became_peered":"2024-04-14T15:12:56.368789+0000","last_unstale":"2024-04-14T15:13:38.850367+0000","last_undegraded":"2024-04-14T15:13:38.850367+0000","last_fullsized":"2024-04-14T15:13:38.850367+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:19.455404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":97,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.870892+0000","last_change":"2024-04-14T15:12:56.290116+0000","last_active":"2024-04-14T15:13:38.870892+0000","last_peered":"2024-04-14T15:13:38.870892+0000","last_clean":"2024-04-14T15:13:38.870892+0000","last_became_active":"2024-04-14T15:12:56.289745+0000","last_became_peered":"2024-04-14T15:12:56.289745+0000","last_unstale":"2024-04-14T15:13:38.870892+0000","last_undegraded":"2024-04-14T15:13:38.870892+0000","last_fullsized":"2024-04-14T15:13:38.870892+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:03.824111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"72'10","reported_seq":203,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.842819+0000","last_change":"2024-04-14T15:11:42.802009+0000","last_active":"2024-04-14T15:13:38.842819+0000","last_peered":"2024-04-14T15:13:38.842819+0000","last_clean":"2024-04-14T15:13:38.842819+0000","last_became_active":"2024-04-14T15:09:45.029170+0000","last_became_peered":"2024-04-14T15:09:45.029170+0000","last_unstale":"2024-04-14T15:13:38.842819+0000","last_undegraded":"2024-04-14T15:13:38.842819+0000","last_fullsized":"2024-04-14T15:13:38.842819+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-04-14T15:11:42.801888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:42.801888+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":200,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.831613+0000","last_change":"2024-04-14T15:12:24.329867+0000","last_active":"2024-04-14T15:13:38.831613+0000","last_peered":"2024-04-14T15:13:38.831613+0000","last_clean":"2024-04-14T15:13:38.831613+0000","last_became_active":"2024-04-14T15:09:43.018871+0000","last_became_peered":"2024-04-14T15:09:43.018871+0000","last_unstale":"2024-04-14T15:13:38.831613+0000","last_undegraded":"2024-04-14T15:13:38.831613+0000","last_fullsized":"2024-04-14T15:13:38.831613+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:24.329791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:12:24.329791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"72'8","reported_seq":248,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.860662+0000","last_change":"2024-04-14T15:13:25.292158+0000","last_active":"2024-04-14T15:13:38.860662+0000","last_peered":"2024-04-14T15:13:38.860662+0000","last_clean":"2024-04-14T15:13:38.860662+0000","last_became_active":"2024-04-14T15:09:47.044085+0000","last_became_peered":"2024-04-14T15:09:47.044085+0000","last_unstale":"2024-04-14T15:13:38.860662+0000","last_undegraded":"2024-04-14T15:13:38.860662+0000","last_fullsized":"2024-04-14T15:13:38.860662+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:13:25.292003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:13:25.292003+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:34.820359+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":180,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.845441+0000","last_change":"2024-04-14T15:12:24.516311+0000","last_active":"2024-04-14T15:13:38.845441+0000","last_peered":"2024-04-14T15:13:38.845441+0000","last_clean":"2024-04-14T15:13:38.845441+0000","last_became_active":"2024-04-14T15:09:49.065274+0000","last_became_peered":"2024-04-14T15:09:49.065274+0000","last_unstale":"2024-04-14T15:13:38.845441+0000","last_undegraded":"2024-04-14T15:13:38.845441+0000","last_fullsized":"2024-04-14T15:13:38.845441+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:24.516200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:12:24.516200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:50.899006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":97,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.890295+0000","last_change":"2024-04-14T15:12:56.490848+0000","last_active":"2024-04-14T15:13:38.890295+0000","last_peered":"2024-04-14T15:13:38.890295+0000","last_clean":"2024-04-14T15:13:38.890295+0000","last_became_active":"2024-04-14T15:12:56.490582+0000","last_became_peered":"2024-04-14T15:12:56.490582+0000","last_unstale":"2024-04-14T15:13:38.890295+0000","last_undegraded":"2024-04-14T15:13:38.890295+0000","last_fullsized":"2024-04-14T15:13:38.890295+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:04.489665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"109'9","reported_seq":106,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.845229+0000","last_change":"2024-04-14T15:12:56.242032+0000","last_active":"2024-04-14T15:13:38.845229+0000","last_peered":"2024-04-14T15:13:38.845229+0000","last_clean":"2024-04-14T15:13:38.845229+0000","last_became_active":"2024-04-14T15:12:56.241684+0000","last_became_peered":"2024-04-14T15:12:56.241684+0000","last_unstale":"2024-04-14T15:13:38.845229+0000","last_undegraded":"2024-04-14T15:13:38.845229+0000","last_fullsized":"2024-04-14T15:13:38.845229+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:06.634537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":97,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.834772+0000","last_change":"2024-04-14T15:12:56.495092+0000","last_active":"2024-04-14T15:13:38.834772+0000","last_peered":"2024-04-14T15:13:38.834772+0000","last_clean":"2024-04-14T15:13:38.834772+0000","last_became_active":"2024-04-14T15:12:56.487615+0000","last_became_peered":"2024-04-14T15:12:56.487615+0000","last_unstale":"2024-04-14T15:13:38.834772+0000","last_undegraded":"2024-04-14T15:13:38.834772+0000","last_fullsized":"2024-04-14T15:13:38.834772+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:03.246475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"72'15","reported_seq":221,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.842559+0000","last_change":"2024-04-14T15:12:42.312510+0000","last_active":"2024-04-14T15:13:38.842559+0000","last_peered":"2024-04-14T15:13:38.842559+0000","last_clean":"2024-04-14T15:13:38.842559+0000","last_became_active":"2024-04-14T15:09:45.033331+0000","last_became_peered":"2024-04-14T15:09:45.033331+0000","last_unstale":"2024-04-14T15:13:38.842559+0000","last_undegraded":"2024-04-14T15:13:38.842559+0000","last_fullsized":"2024-04-14T15:13:38.842559+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-04-14T15:12:42.312355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:12:42.312355+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:43.040335+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":200,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.826209+0000","last_change":"2024-04-14T15:12:31.395583+0000","last_active":"2024-04-14T15:13:38.826209+0000","last_peered":"2024-04-14T15:13:38.826209+0000","last_clean":"2024-04-14T15:13:38.826209+0000","last_became_active":"2024-04-14T15:09:43.020878+0000","last_became_peered":"2024-04-14T15:09:43.020878+0000","last_unstale":"2024-04-14T15:13:38.826209+0000","last_undegraded":"2024-04-14T15:13:38.826209+0000","last_fullsized":"2024-04-14T15:13:38.826209+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:31.395502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:12:31.395502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:49.270399+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"72'8","reported_seq":248,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.845339+0000","last_change":"2024-04-14T15:13:33.568661+0000","last_active":"2024-04-14T15:13:38.845339+0000","last_peered":"2024-04-14T15:13:38.845339+0000","last_clean":"2024-04-14T15:13:38.845339+0000","last_became_active":"2024-04-14T15:09:47.050962+0000","last_became_peered":"2024-04-14T15:09:47.050962+0000","last_unstale":"2024-04-14T15:13:38.845339+0000","last_undegraded":"2024-04-14T15:13:38.845339+0000","last_fullsized":"2024-04-14T15:13:38.845339+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:13:33.568569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:13:33.568569+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:40.930108+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":180,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.831649+0000","last_change":"2024-04-14T15:11:25.638017+0000","last_active":"2024-04-14T15:13:38.831649+0000","last_peered":"2024-04-14T15:13:38.831649+0000","last_clean":"2024-04-14T15:13:38.831649+0000","last_became_active":"2024-04-14T15:09:49.058940+0000","last_became_peered":"2024-04-14T15:09:49.058940+0000","last_unstale":"2024-04-14T15:13:38.831649+0000","last_undegraded":"2024-04-14T15:13:38.831649+0000","last_fullsized":"2024-04-14T15:13:38.831649+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:25.637888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:25.637888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1b","version":"110'1001","reported_seq":2098,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.826949+0000","last_change":"2024-04-14T15:12:56.247499+0000","last_active":"2024-04-14T15:13:38.826949+0000","last_peered":"2024-04-14T15:13:38.826949+0000","last_clean":"2024-04-14T15:13:38.826949+0000","last_became_active":"2024-04-14T15:12:56.247212+0000","last_became_peered":"2024-04-14T15:12:56.247212+0000","last_unstale":"2024-04-14T15:13:38.826949+0000","last_undegraded":"2024-04-14T15:13:38.826949+0000","last_fullsized":"2024-04-14T15:13:38.826949+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":1001,"log_dups_size":0,"ondisk_log_size":1001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:58.359310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1000,"num_read_kb":0,"num_write":1000,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":97,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.837316+0000","last_change":"2024-04-14T15:12:56.190468+0000","last_active":"2024-04-14T15:13:38.837316+0000","last_peered":"2024-04-14T15:13:38.837316+0000","last_clean":"2024-04-14T15:13:38.837316+0000","last_became_active":"2024-04-14T15:12:56.188780+0000","last_became_peered":"2024-04-14T15:12:56.188780+0000","last_unstale":"2024-04-14T15:13:38.837316+0000","last_undegraded":"2024-04-14T15:13:38.837316+0000","last_fullsized":"2024-04-14T15:13:38.837316+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:17.525051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1","version":"108'7","reported_seq":110,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.873327+0000","last_change":"2024-04-14T15:12:56.385652+0000","last_active":"2024-04-14T15:13:38.873327+0000","last_peered":"2024-04-14T15:13:38.873327+0000","last_clean":"2024-04-14T15:13:38.873327+0000","last_became_active":"2024-04-14T15:12:56.381026+0000","last_became_peered":"2024-04-14T15:12:56.381026+0000","last_unstale":"2024-04-14T15:13:38.873327+0000","last_undegraded":"2024-04-14T15:13:38.873327+0000","last_fullsized":"2024-04-14T15:13:38.873327+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:07.570579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"85'16","reported_seq":220,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.841411+0000","last_change":"2024-04-14T15:12:40.495261+0000","last_active":"2024-04-14T15:13:38.841411+0000","last_peered":"2024-04-14T15:13:38.841411+0000","last_clean":"2024-04-14T15:13:38.841411+0000","last_became_active":"2024-04-14T15:09:45.028212+0000","last_became_peered":"2024-04-14T15:09:45.028212+0000","last_unstale":"2024-04-14T15:13:38.841411+0000","last_undegraded":"2024-04-14T15:13:38.841411+0000","last_fullsized":"2024-04-14T15:13:38.841411+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-14T15:12:40.495139+0000","last_deep_scrub":"85'16","last_deep_scrub_stamp":"2024-04-14T15:10:47.456169+0000","last_clean_scrub_stamp":"2024-04-14T15:12:40.495139+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:00.218258+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":200,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.838775+0000","last_change":"2024-04-14T15:12:43.093032+0000","last_active":"2024-04-14T15:13:38.838775+0000","last_peered":"2024-04-14T15:13:38.838775+0000","last_clean":"2024-04-14T15:13:38.838775+0000","last_became_active":"2024-04-14T15:09:43.018601+0000","last_became_peered":"2024-04-14T15:09:43.018601+0000","last_unstale":"2024-04-14T15:13:38.838775+0000","last_undegraded":"2024-04-14T15:13:38.838775+0000","last_fullsized":"2024-04-14T15:13:38.838775+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:43.092895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:12:43.092895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:45.016272+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":192,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.828827+0000","last_change":"2024-04-14T15:12:27.761412+0000","last_active":"2024-04-14T15:13:38.828827+0000","last_peered":"2024-04-14T15:13:38.828827+0000","last_clean":"2024-04-14T15:13:38.828827+0000","last_became_active":"2024-04-14T15:09:47.043021+0000","last_became_peered":"2024-04-14T15:09:47.043021+0000","last_unstale":"2024-04-14T15:13:38.828827+0000","last_undegraded":"2024-04-14T15:13:38.828827+0000","last_fullsized":"2024-04-14T15:13:38.828827+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:27.761311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:12:27.761311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:54.632804+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":127,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.872584+0000","last_change":"2024-04-14T15:12:34.084996+0000","last_active":"2024-04-14T15:13:38.872584+0000","last_peered":"2024-04-14T15:13:38.872584+0000","last_clean":"2024-04-14T15:13:38.872584+0000","last_became_active":"2024-04-14T15:11:57.120861+0000","last_became_peered":"2024-04-14T15:11:57.120861+0000","last_unstale":"2024-04-14T15:13:38.872584+0000","last_undegraded":"2024-04-14T15:13:38.872584+0000","last_fullsized":"2024-04-14T15:13:38.872584+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:34.084920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:12:34.084920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:01.190876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1a","version":"108'1","reported_seq":99,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.873581+0000","last_change":"2024-04-14T15:12:56.310424+0000","last_active":"2024-04-14T15:13:38.873581+0000","last_peered":"2024-04-14T15:13:38.873581+0000","last_clean":"2024-04-14T15:13:38.873581+0000","last_became_active":"2024-04-14T15:12:56.310046+0000","last_became_peered":"2024-04-14T15:12:56.310046+0000","last_unstale":"2024-04-14T15:13:38.873581+0000","last_undegraded":"2024-04-14T15:13:38.873581+0000","last_fullsized":"2024-04-14T15:13:38.873581+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:16.150890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1024,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.8","version":"108'2","reported_seq":99,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.846037+0000","last_change":"2024-04-14T15:12:56.340353+0000","last_active":"2024-04-14T15:13:38.846037+0000","last_peered":"2024-04-14T15:13:38.846037+0000","last_clean":"2024-04-14T15:13:38.846037+0000","last_became_active":"2024-04-14T15:12:56.339672+0000","last_became_peered":"2024-04-14T15:12:56.339672+0000","last_unstale":"2024-04-14T15:13:38.846037+0000","last_undegraded":"2024-04-14T15:13:38.846037+0000","last_fullsized":"2024-04-14T15:13:38.846037+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:22.826093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":97,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.841319+0000","last_change":"2024-04-14T15:12:56.427581+0000","last_active":"2024-04-14T15:13:38.841319+0000","last_peered":"2024-04-14T15:13:38.841319+0000","last_clean":"2024-04-14T15:13:38.841319+0000","last_became_active":"2024-04-14T15:12:56.427276+0000","last_became_peered":"2024-04-14T15:12:56.427276+0000","last_unstale":"2024-04-14T15:13:38.841319+0000","last_undegraded":"2024-04-14T15:13:38.841319+0000","last_fullsized":"2024-04-14T15:13:38.841319+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:09.993135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"98'9","reported_seq":172,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.832075+0000","last_change":"2024-04-14T15:13:38.414555+0000","last_active":"2024-04-14T15:13:38.832075+0000","last_peered":"2024-04-14T15:13:38.832075+0000","last_clean":"2024-04-14T15:13:38.832075+0000","last_became_active":"2024-04-14T15:11:57.118068+0000","last_became_peered":"2024-04-14T15:11:57.118068+0000","last_unstale":"2024-04-14T15:13:38.832075+0000","last_undegraded":"2024-04-14T15:13:38.832075+0000","last_fullsized":"2024-04-14T15:13:38.832075+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-04-14T15:13:38.414471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:13:38.414471+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:49.479034+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"72'14","reported_seq":217,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.837398+0000","last_change":"2024-04-14T15:13:20.574920+0000","last_active":"2024-04-14T15:13:38.837398+0000","last_peered":"2024-04-14T15:13:38.837398+0000","last_clean":"2024-04-14T15:13:38.837398+0000","last_became_active":"2024-04-14T15:09:45.025673+0000","last_became_peered":"2024-04-14T15:09:45.025673+0000","last_unstale":"2024-04-14T15:13:38.837398+0000","last_undegraded":"2024-04-14T15:13:38.837398+0000","last_fullsized":"2024-04-14T15:13:38.837398+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'14","last_scrub_stamp":"2024-04-14T15:13:20.574826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:13:20.574826+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:22.739206+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":200,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.873823+0000","last_change":"2024-04-14T15:13:18.161888+0000","last_active":"2024-04-14T15:13:38.873823+0000","last_peered":"2024-04-14T15:13:38.873823+0000","last_clean":"2024-04-14T15:13:38.873823+0000","last_became_active":"2024-04-14T15:09:43.020503+0000","last_became_peered":"2024-04-14T15:09:43.020503+0000","last_unstale":"2024-04-14T15:13:38.873823+0000","last_undegraded":"2024-04-14T15:13:38.873823+0000","last_fullsized":"2024-04-14T15:13:38.873823+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:18.161799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:13:18.161799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:23.917967+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":188,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.838409+0000","last_change":"2024-04-14T15:13:25.116624+0000","last_active":"2024-04-14T15:13:38.838409+0000","last_peered":"2024-04-14T15:13:38.838409+0000","last_clean":"2024-04-14T15:13:38.838409+0000","last_became_active":"2024-04-14T15:09:49.129610+0000","last_became_peered":"2024-04-14T15:09:49.129610+0000","last_unstale":"2024-04-14T15:13:38.838409+0000","last_undegraded":"2024-04-14T15:13:38.838409+0000","last_fullsized":"2024-04-14T15:13:38.838409+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:25.116541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:13:25.116541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:51.711645+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":97,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.884100+0000","last_change":"2024-04-14T15:12:56.458443+0000","last_active":"2024-04-14T15:13:38.884100+0000","last_peered":"2024-04-14T15:13:38.884100+0000","last_clean":"2024-04-14T15:13:38.884100+0000","last_became_active":"2024-04-14T15:12:56.458177+0000","last_became_peered":"2024-04-14T15:12:56.458177+0000","last_unstale":"2024-04-14T15:13:38.884100+0000","last_undegraded":"2024-04-14T15:13:38.884100+0000","last_fullsized":"2024-04-14T15:13:38.884100+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:05.601441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":97,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.825849+0000","last_change":"2024-04-14T15:12:56.403204+0000","last_active":"2024-04-14T15:13:38.825849+0000","last_peered":"2024-04-14T15:13:38.825849+0000","last_clean":"2024-04-14T15:13:38.825849+0000","last_became_active":"2024-04-14T15:12:56.402900+0000","last_became_peered":"2024-04-14T15:12:56.402900+0000","last_unstale":"2024-04-14T15:13:38.825849+0000","last_undegraded":"2024-04-14T15:13:38.825849+0000","last_fullsized":"2024-04-14T15:13:38.825849+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:03.604731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":97,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.842461+0000","last_change":"2024-04-14T15:12:56.363969+0000","last_active":"2024-04-14T15:13:38.842461+0000","last_peered":"2024-04-14T15:13:38.842461+0000","last_clean":"2024-04-14T15:13:38.842461+0000","last_became_active":"2024-04-14T15:12:56.355254+0000","last_became_peered":"2024-04-14T15:12:56.355254+0000","last_unstale":"2024-04-14T15:13:38.842461+0000","last_undegraded":"2024-04-14T15:13:38.842461+0000","last_fullsized":"2024-04-14T15:13:38.842461+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:03.500100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":192,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.838111+0000","last_change":"2024-04-14T15:13:21.526922+0000","last_active":"2024-04-14T15:13:38.838111+0000","last_peered":"2024-04-14T15:13:38.838111+0000","last_clean":"2024-04-14T15:13:38.838111+0000","last_became_active":"2024-04-14T15:09:47.044610+0000","last_became_peered":"2024-04-14T15:09:47.044610+0000","last_unstale":"2024-04-14T15:13:38.838111+0000","last_undegraded":"2024-04-14T15:13:38.838111+0000","last_fullsized":"2024-04-14T15:13:38.838111+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:21.526839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:13:21.526839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:40.438748+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"72'18","reported_seq":177,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.838261+0000","last_change":"2024-04-14T15:13:18.170741+0000","last_active":"2024-04-14T15:13:38.838261+0000","last_peered":"2024-04-14T15:13:38.838261+0000","last_clean":"2024-04-14T15:13:38.838261+0000","last_became_active":"2024-04-14T15:11:57.118744+0000","last_became_peered":"2024-04-14T15:11:57.118744+0000","last_unstale":"2024-04-14T15:13:38.838261+0000","last_undegraded":"2024-04-14T15:13:38.838261+0000","last_fullsized":"2024-04-14T15:13:38.838261+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'18","last_scrub_stamp":"2024-04-14T15:13:18.170570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:13:18.170570+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:25.248483+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":33,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"91'2","reported_seq":194,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.839319+0000","last_change":"2024-04-14T15:11:37.974863+0000","last_active":"2024-04-14T15:13:38.839319+0000","last_peered":"2024-04-14T15:13:38.839319+0000","last_clean":"2024-04-14T15:13:38.839319+0000","last_became_active":"2024-04-14T15:09:43.018478+0000","last_became_peered":"2024-04-14T15:09:43.018478+0000","last_unstale":"2024-04-14T15:13:38.839319+0000","last_undegraded":"2024-04-14T15:13:38.839319+0000","last_fullsized":"2024-04-14T15:13:38.839319+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'2","last_scrub_stamp":"2024-04-14T15:11:37.974770+0000","last_deep_scrub":"91'2","last_deep_scrub_stamp":"2024-04-14T15:11:37.974770+0000","last_clean_scrub_stamp":"2024-04-14T15:11:37.974770+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":188,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.843486+0000","last_change":"2024-04-14T15:13:26.563376+0000","last_active":"2024-04-14T15:13:38.843486+0000","last_peered":"2024-04-14T15:13:38.843486+0000","last_clean":"2024-04-14T15:13:38.843486+0000","last_became_active":"2024-04-14T15:09:49.065684+0000","last_became_peered":"2024-04-14T15:09:49.065684+0000","last_unstale":"2024-04-14T15:13:38.843486+0000","last_undegraded":"2024-04-14T15:13:38.843486+0000","last_fullsized":"2024-04-14T15:13:38.843486+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:26.563298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:13:26.563298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:52.535208+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.14","version":"108'1","reported_seq":99,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.893182+0000","last_change":"2024-04-14T15:12:56.484486+0000","last_active":"2024-04-14T15:13:38.893182+0000","last_peered":"2024-04-14T15:13:38.893182+0000","last_clean":"2024-04-14T15:13:38.893182+0000","last_became_active":"2024-04-14T15:12:56.484244+0000","last_became_peered":"2024-04-14T15:12:56.484244+0000","last_unstale":"2024-04-14T15:13:38.893182+0000","last_undegraded":"2024-04-14T15:13:38.893182+0000","last_fullsized":"2024-04-14T15:13:38.893182+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:03.557774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"109'10","reported_seq":107,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.849431+0000","last_change":"2024-04-14T15:12:56.405375+0000","last_active":"2024-04-14T15:13:38.849431+0000","last_peered":"2024-04-14T15:13:38.849431+0000","last_clean":"2024-04-14T15:13:38.849431+0000","last_became_active":"2024-04-14T15:12:56.405078+0000","last_became_peered":"2024-04-14T15:12:56.405078+0000","last_unstale":"2024-04-14T15:13:38.849431+0000","last_undegraded":"2024-04-14T15:13:38.849431+0000","last_fullsized":"2024-04-14T15:13:38.849431+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:09.673883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":97,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.841767+0000","last_change":"2024-04-14T15:12:56.407793+0000","last_active":"2024-04-14T15:13:38.841767+0000","last_peered":"2024-04-14T15:13:38.841767+0000","last_clean":"2024-04-14T15:13:38.841767+0000","last_became_active":"2024-04-14T15:12:56.402023+0000","last_became_peered":"2024-04-14T15:12:56.402023+0000","last_unstale":"2024-04-14T15:13:38.841767+0000","last_undegraded":"2024-04-14T15:13:38.841767+0000","last_fullsized":"2024-04-14T15:13:38.841767+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:13.313190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":192,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.873342+0000","last_change":"2024-04-14T15:12:24.013345+0000","last_active":"2024-04-14T15:13:38.873342+0000","last_peered":"2024-04-14T15:13:38.873342+0000","last_clean":"2024-04-14T15:13:38.873342+0000","last_became_active":"2024-04-14T15:09:47.048427+0000","last_became_peered":"2024-04-14T15:09:47.048427+0000","last_unstale":"2024-04-14T15:13:38.873342+0000","last_undegraded":"2024-04-14T15:13:38.873342+0000","last_fullsized":"2024-04-14T15:13:38.873342+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:24.013261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:12:24.013261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:50.223688+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"102'27","reported_seq":240,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.885745+0000","last_change":"2024-04-14T15:12:30.928373+0000","last_active":"2024-04-14T15:13:38.885745+0000","last_peered":"2024-04-14T15:13:38.885745+0000","last_clean":"2024-04-14T15:13:38.885745+0000","last_became_active":"2024-04-14T15:09:45.179182+0000","last_became_peered":"2024-04-14T15:09:45.179182+0000","last_unstale":"2024-04-14T15:13:38.885745+0000","last_undegraded":"2024-04-14T15:13:38.885745+0000","last_fullsized":"2024-04-14T15:13:38.885745+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'27","last_scrub_stamp":"2024-04-14T15:12:30.928216+0000","last_deep_scrub":"102'27","last_deep_scrub_stamp":"2024-04-14T15:12:30.928216+0000","last_clean_scrub_stamp":"2024-04-14T15:12:30.928216+0000","objects_scrubbed":8,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:45.708881+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":200,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.845961+0000","last_change":"2024-04-14T15:13:16.122436+0000","last_active":"2024-04-14T15:13:38.845961+0000","last_peered":"2024-04-14T15:13:38.845961+0000","last_clean":"2024-04-14T15:13:38.845961+0000","last_became_active":"2024-04-14T15:09:43.021006+0000","last_became_peered":"2024-04-14T15:09:43.021006+0000","last_unstale":"2024-04-14T15:13:38.845961+0000","last_undegraded":"2024-04-14T15:13:38.845961+0000","last_fullsized":"2024-04-14T15:13:38.845961+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:16.122352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:13:16.122352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:46.102116+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":180,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.826281+0000","last_change":"2024-04-14T15:11:35.993757+0000","last_active":"2024-04-14T15:13:38.826281+0000","last_peered":"2024-04-14T15:13:38.826281+0000","last_clean":"2024-04-14T15:13:38.826281+0000","last_became_active":"2024-04-14T15:09:49.128136+0000","last_became_peered":"2024-04-14T15:09:49.128136+0000","last_unstale":"2024-04-14T15:13:38.826281+0000","last_undegraded":"2024-04-14T15:13:38.826281+0000","last_fullsized":"2024-04-14T15:13:38.826281+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:35.993681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:35.993681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.17","version":"108'1","reported_seq":98,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.853537+0000","last_change":"2024-04-14T15:12:56.473773+0000","last_active":"2024-04-14T15:13:38.853537+0000","last_peered":"2024-04-14T15:13:38.853537+0000","last_clean":"2024-04-14T15:13:38.853537+0000","last_became_active":"2024-04-14T15:12:56.469133+0000","last_became_peered":"2024-04-14T15:12:56.469133+0000","last_unstale":"2024-04-14T15:13:38.853537+0000","last_undegraded":"2024-04-14T15:13:38.853537+0000","last_fullsized":"2024-04-14T15:13:38.853537+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:56.511528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":97,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.872471+0000","last_change":"2024-04-14T15:12:56.402757+0000","last_active":"2024-04-14T15:13:38.872471+0000","last_peered":"2024-04-14T15:13:38.872471+0000","last_clean":"2024-04-14T15:13:38.872471+0000","last_became_active":"2024-04-14T15:12:56.383575+0000","last_became_peered":"2024-04-14T15:12:56.383575+0000","last_unstale":"2024-04-14T15:13:38.872471+0000","last_undegraded":"2024-04-14T15:13:38.872471+0000","last_fullsized":"2024-04-14T15:13:38.872471+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:11.621424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":192,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.829970+0000","last_change":"2024-04-14T15:12:36.818501+0000","last_active":"2024-04-14T15:13:38.829970+0000","last_peered":"2024-04-14T15:13:38.829970+0000","last_clean":"2024-04-14T15:13:38.829970+0000","last_became_active":"2024-04-14T15:09:47.036696+0000","last_became_peered":"2024-04-14T15:09:47.036696+0000","last_unstale":"2024-04-14T15:13:38.829970+0000","last_undegraded":"2024-04-14T15:13:38.829970+0000","last_fullsized":"2024-04-14T15:13:38.829970+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:36.818402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:12:36.818402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:43.044105+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"72'12","reported_seq":214,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.885878+0000","last_change":"2024-04-14T15:12:22.083844+0000","last_active":"2024-04-14T15:13:38.885878+0000","last_peered":"2024-04-14T15:13:38.885878+0000","last_clean":"2024-04-14T15:13:38.885878+0000","last_became_active":"2024-04-14T15:09:45.037231+0000","last_became_peered":"2024-04-14T15:09:45.037231+0000","last_unstale":"2024-04-14T15:13:38.885878+0000","last_undegraded":"2024-04-14T15:13:38.885878+0000","last_fullsized":"2024-04-14T15:13:38.885878+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'12","last_scrub_stamp":"2024-04-14T15:12:22.083718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:12:22.083718+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:41.128151+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":200,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.838981+0000","last_change":"2024-04-14T15:12:41.108387+0000","last_active":"2024-04-14T15:13:38.838981+0000","last_peered":"2024-04-14T15:13:38.838981+0000","last_clean":"2024-04-14T15:13:38.838981+0000","last_became_active":"2024-04-14T15:09:43.025095+0000","last_became_peered":"2024-04-14T15:09:43.025095+0000","last_unstale":"2024-04-14T15:13:38.838981+0000","last_undegraded":"2024-04-14T15:13:38.838981+0000","last_fullsized":"2024-04-14T15:13:38.838981+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:41.108288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:12:41.108288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:57.955243+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":207,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.887253+0000","last_change":"2024-04-14T15:12:39.104506+0000","last_active":"2024-04-14T15:13:38.887253+0000","last_peered":"2024-04-14T15:13:38.887253+0000","last_clean":"2024-04-14T15:13:38.887253+0000","last_became_active":"2024-04-14T15:09:43.018509+0000","last_became_peered":"2024-04-14T15:09:43.018509+0000","last_unstale":"2024-04-14T15:13:38.887253+0000","last_undegraded":"2024-04-14T15:13:38.887253+0000","last_fullsized":"2024-04-14T15:13:38.887253+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:39.104391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:12:39.104391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:51.570145+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0018592680000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"72'1","reported_seq":181,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.841561+0000","last_change":"2024-04-14T15:11:25.095270+0000","last_active":"2024-04-14T15:13:38.841561+0000","last_peered":"2024-04-14T15:13:38.841561+0000","last_clean":"2024-04-14T15:13:38.841561+0000","last_became_active":"2024-04-14T15:09:49.065243+0000","last_became_peered":"2024-04-14T15:09:49.065243+0000","last_unstale":"2024-04-14T15:13:38.841561+0000","last_undegraded":"2024-04-14T15:13:38.841561+0000","last_fullsized":"2024-04-14T15:13:38.841561+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-04-14T15:11:25.095182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:25.095182+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":97,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.873485+0000","last_change":"2024-04-14T15:12:56.411802+0000","last_active":"2024-04-14T15:13:38.873485+0000","last_peered":"2024-04-14T15:13:38.873485+0000","last_clean":"2024-04-14T15:13:38.873485+0000","last_became_active":"2024-04-14T15:12:56.406556+0000","last_became_peered":"2024-04-14T15:12:56.406556+0000","last_unstale":"2024-04-14T15:13:38.873485+0000","last_undegraded":"2024-04-14T15:13:38.873485+0000","last_fullsized":"2024-04-14T15:13:38.873485+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:08.622711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.12","version":"111'10","reported_seq":111,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.873480+0000","last_change":"2024-04-14T15:12:56.370347+0000","last_active":"2024-04-14T15:13:38.873480+0000","last_peered":"2024-04-14T15:13:38.873480+0000","last_clean":"2024-04-14T15:13:38.873480+0000","last_became_active":"2024-04-14T15:12:56.368444+0000","last_became_peered":"2024-04-14T15:12:56.368444+0000","last_unstale":"2024-04-14T15:13:38.873480+0000","last_undegraded":"2024-04-14T15:13:38.873480+0000","last_fullsized":"2024-04-14T15:13:38.873480+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:03.097649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165835,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":27326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":97,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.870909+0000","last_change":"2024-04-14T15:12:56.311510+0000","last_active":"2024-04-14T15:13:38.870909+0000","last_peered":"2024-04-14T15:13:38.870909+0000","last_clean":"2024-04-14T15:13:38.870909+0000","last_became_active":"2024-04-14T15:12:56.310799+0000","last_became_peered":"2024-04-14T15:12:56.310799+0000","last_unstale":"2024-04-14T15:13:38.870909+0000","last_undegraded":"2024-04-14T15:13:38.870909+0000","last_fullsized":"2024-04-14T15:13:38.870909+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:55.801206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":192,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.848217+0000","last_change":"2024-04-14T15:13:31.226741+0000","last_active":"2024-04-14T15:13:38.848217+0000","last_peered":"2024-04-14T15:13:38.848217+0000","last_clean":"2024-04-14T15:13:38.848217+0000","last_became_active":"2024-04-14T15:09:47.047201+0000","last_became_peered":"2024-04-14T15:09:47.047201+0000","last_unstale":"2024-04-14T15:13:38.848217+0000","last_undegraded":"2024-04-14T15:13:38.848217+0000","last_fullsized":"2024-04-14T15:13:38.848217+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:31.226659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:13:31.226659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:53.711821+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"90'24","reported_seq":274,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.874798+0000","last_change":"2024-04-14T15:13:26.147772+0000","last_active":"2024-04-14T15:13:38.874798+0000","last_peered":"2024-04-14T15:13:38.874798+0000","last_clean":"2024-04-14T15:13:38.874798+0000","last_became_active":"2024-04-14T15:09:45.179978+0000","last_became_peered":"2024-04-14T15:09:45.179978+0000","last_unstale":"2024-04-14T15:13:38.874798+0000","last_undegraded":"2024-04-14T15:13:38.874798+0000","last_fullsized":"2024-04-14T15:13:38.874798+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'24","last_scrub_stamp":"2024-04-14T15:13:26.147631+0000","last_deep_scrub":"90'24","last_deep_scrub_stamp":"2024-04-14T15:11:25.120692+0000","last_clean_scrub_stamp":"2024-04-14T15:13:26.147631+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:54.996017+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"91'130","reported_seq":445,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.838931+0000","last_change":"2024-04-14T15:11:28.875662+0000","last_active":"2024-04-14T15:13:38.838931+0000","last_peered":"2024-04-14T15:13:38.838931+0000","last_clean":"2024-04-14T15:13:38.838931+0000","last_became_active":"2024-04-14T15:09:33.753828+0000","last_became_peered":"2024-04-14T15:09:33.753828+0000","last_unstale":"2024-04-14T15:13:38.838931+0000","last_undegraded":"2024-04-14T15:13:38.838931+0000","last_fullsized":"2024-04-14T15:13:38.838931+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:11:28.875570+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":200,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.829926+0000","last_change":"2024-04-14T15:11:47.627424+0000","last_active":"2024-04-14T15:13:38.829926+0000","last_peered":"2024-04-14T15:13:38.829926+0000","last_clean":"2024-04-14T15:13:38.829926+0000","last_became_active":"2024-04-14T15:09:43.025421+0000","last_became_peered":"2024-04-14T15:09:43.025421+0000","last_unstale":"2024-04-14T15:13:38.829926+0000","last_undegraded":"2024-04-14T15:13:38.829926+0000","last_fullsized":"2024-04-14T15:13:38.829926+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:47.627305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:47.627305+0000","last_clean_scrub_stamp":"2024-04-14T15:11:47.627305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"69'2","reported_seq":151,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.828333+0000","last_change":"2024-04-14T15:13:32.805011+0000","last_active":"2024-04-14T15:13:38.828333+0000","last_peered":"2024-04-14T15:13:38.828333+0000","last_clean":"2024-04-14T15:13:38.828333+0000","last_became_active":"2024-04-14T15:11:57.121809+0000","last_became_peered":"2024-04-14T15:11:57.121809+0000","last_unstale":"2024-04-14T15:13:38.828333+0000","last_undegraded":"2024-04-14T15:13:38.828333+0000","last_fullsized":"2024-04-14T15:13:38.828333+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-04-14T15:13:32.804904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:13:32.804904+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:53.419472+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":180,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.829999+0000","last_change":"2024-04-14T15:11:32.672340+0000","last_active":"2024-04-14T15:13:38.829999+0000","last_peered":"2024-04-14T15:13:38.829999+0000","last_clean":"2024-04-14T15:13:38.829999+0000","last_became_active":"2024-04-14T15:09:49.067031+0000","last_became_peered":"2024-04-14T15:09:49.067031+0000","last_unstale":"2024-04-14T15:13:38.829999+0000","last_undegraded":"2024-04-14T15:13:38.829999+0000","last_fullsized":"2024-04-14T15:13:38.829999+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:32.672255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:32.672255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":97,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.840112+0000","last_change":"2024-04-14T15:12:56.407518+0000","last_active":"2024-04-14T15:13:38.840112+0000","last_peered":"2024-04-14T15:13:38.840112+0000","last_clean":"2024-04-14T15:13:38.840112+0000","last_became_active":"2024-04-14T15:12:56.401661+0000","last_became_peered":"2024-04-14T15:12:56.401661+0000","last_unstale":"2024-04-14T15:13:38.840112+0000","last_undegraded":"2024-04-14T15:13:38.840112+0000","last_fullsized":"2024-04-14T15:13:38.840112+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:22.223998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.11","version":"141'4","reported_seq":109,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.825794+0000","last_change":"2024-04-14T15:12:56.402927+0000","last_active":"2024-04-14T15:13:38.825794+0000","last_peered":"2024-04-14T15:13:38.825794+0000","last_clean":"2024-04-14T15:13:38.825794+0000","last_became_active":"2024-04-14T15:12:56.402343+0000","last_became_peered":"2024-04-14T15:12:56.402343+0000","last_unstale":"2024-04-14T15:13:38.825794+0000","last_undegraded":"2024-04-14T15:13:38.825794+0000","last_fullsized":"2024-04-14T15:13:38.825794+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:00.892629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165844,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":24579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":97,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.870605+0000","last_change":"2024-04-14T15:12:56.322582+0000","last_active":"2024-04-14T15:13:38.870605+0000","last_peered":"2024-04-14T15:13:38.870605+0000","last_clean":"2024-04-14T15:13:38.870605+0000","last_became_active":"2024-04-14T15:12:56.315249+0000","last_became_peered":"2024-04-14T15:12:56.315249+0000","last_unstale":"2024-04-14T15:13:38.870605+0000","last_undegraded":"2024-04-14T15:13:38.870605+0000","last_fullsized":"2024-04-14T15:13:38.870605+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:04.341211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":192,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.859900+0000","last_change":"2024-04-14T15:12:39.402534+0000","last_active":"2024-04-14T15:13:38.859900+0000","last_peered":"2024-04-14T15:13:38.859900+0000","last_clean":"2024-04-14T15:13:38.859900+0000","last_became_active":"2024-04-14T15:09:47.049457+0000","last_became_peered":"2024-04-14T15:09:47.049457+0000","last_unstale":"2024-04-14T15:13:38.859900+0000","last_undegraded":"2024-04-14T15:13:38.859900+0000","last_fullsized":"2024-04-14T15:13:38.859900+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:39.402404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:12:39.402404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:07.385043+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"72'13","reported_seq":223,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.842837+0000","last_change":"2024-04-14T15:13:28.340992+0000","last_active":"2024-04-14T15:13:38.842837+0000","last_peered":"2024-04-14T15:13:38.842837+0000","last_clean":"2024-04-14T15:13:38.842837+0000","last_became_active":"2024-04-14T15:09:45.033883+0000","last_became_peered":"2024-04-14T15:09:45.033883+0000","last_unstale":"2024-04-14T15:13:38.842837+0000","last_undegraded":"2024-04-14T15:13:38.842837+0000","last_fullsized":"2024-04-14T15:13:38.842837+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'13","last_scrub_stamp":"2024-04-14T15:13:28.340838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:13:28.340838+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:33.889823+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":200,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.848621+0000","last_change":"2024-04-14T15:12:37.457119+0000","last_active":"2024-04-14T15:13:38.848621+0000","last_peered":"2024-04-14T15:13:38.848621+0000","last_clean":"2024-04-14T15:13:38.848621+0000","last_became_active":"2024-04-14T15:09:43.021188+0000","last_became_peered":"2024-04-14T15:09:43.021188+0000","last_unstale":"2024-04-14T15:13:38.848621+0000","last_undegraded":"2024-04-14T15:13:38.848621+0000","last_fullsized":"2024-04-14T15:13:38.848621+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:37.456999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:12:37.456999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:45.565006+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"73'5","reported_seq":439,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.834856+0000","last_change":"2024-04-14T15:12:41.335732+0000","last_active":"2024-04-14T15:13:38.834856+0000","last_peered":"2024-04-14T15:13:38.834856+0000","last_clean":"2024-04-14T15:13:38.834856+0000","last_became_active":"2024-04-14T15:09:43.016544+0000","last_became_peered":"2024-04-14T15:09:43.016544+0000","last_unstale":"2024-04-14T15:13:38.834856+0000","last_undegraded":"2024-04-14T15:13:38.834856+0000","last_fullsized":"2024-04-14T15:13:38.834856+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-04-14T15:12:41.335600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:12:41.335600+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:57.445938+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00056542099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":229,"num_read_kb":224,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":188,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.846058+0000","last_change":"2024-04-14T15:13:17.120281+0000","last_active":"2024-04-14T15:13:38.846058+0000","last_peered":"2024-04-14T15:13:38.846058+0000","last_clean":"2024-04-14T15:13:38.846058+0000","last_became_active":"2024-04-14T15:09:49.130307+0000","last_became_peered":"2024-04-14T15:09:49.130307+0000","last_unstale":"2024-04-14T15:13:38.846058+0000","last_undegraded":"2024-04-14T15:13:38.846058+0000","last_fullsized":"2024-04-14T15:13:38.846058+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:17.120174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:13:17.120174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:46.362976+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":97,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.849676+0000","last_change":"2024-04-14T15:12:56.200883+0000","last_active":"2024-04-14T15:13:38.849676+0000","last_peered":"2024-04-14T15:13:38.849676+0000","last_clean":"2024-04-14T15:13:38.849676+0000","last_became_active":"2024-04-14T15:12:56.185179+0000","last_became_peered":"2024-04-14T15:12:56.185179+0000","last_unstale":"2024-04-14T15:13:38.849676+0000","last_undegraded":"2024-04-14T15:13:38.849676+0000","last_fullsized":"2024-04-14T15:13:38.849676+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:16.900454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":97,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.840176+0000","last_change":"2024-04-14T15:12:56.233291+0000","last_active":"2024-04-14T15:13:38.840176+0000","last_peered":"2024-04-14T15:13:38.840176+0000","last_clean":"2024-04-14T15:13:38.840176+0000","last_became_active":"2024-04-14T15:12:56.219428+0000","last_became_peered":"2024-04-14T15:12:56.219428+0000","last_unstale":"2024-04-14T15:13:38.840176+0000","last_undegraded":"2024-04-14T15:13:38.840176+0000","last_fullsized":"2024-04-14T15:13:38.840176+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:01.478401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":97,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.840259+0000","last_change":"2024-04-14T15:12:56.447607+0000","last_active":"2024-04-14T15:13:38.840259+0000","last_peered":"2024-04-14T15:13:38.840259+0000","last_clean":"2024-04-14T15:13:38.840259+0000","last_became_active":"2024-04-14T15:12:56.430017+0000","last_became_peered":"2024-04-14T15:12:56.430017+0000","last_unstale":"2024-04-14T15:13:38.840259+0000","last_undegraded":"2024-04-14T15:13:38.840259+0000","last_fullsized":"2024-04-14T15:13:38.840259+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:56.812126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":192,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.888004+0000","last_change":"2024-04-14T15:13:35.587741+0000","last_active":"2024-04-14T15:13:38.888004+0000","last_peered":"2024-04-14T15:13:38.888004+0000","last_clean":"2024-04-14T15:13:38.888004+0000","last_became_active":"2024-04-14T15:09:47.040185+0000","last_became_peered":"2024-04-14T15:09:47.040185+0000","last_unstale":"2024-04-14T15:13:38.888004+0000","last_undegraded":"2024-04-14T15:13:38.888004+0000","last_fullsized":"2024-04-14T15:13:38.888004+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:35.587624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.587624+0000","last_clean_scrub_stamp":"2024-04-14T15:13:35.587624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:44.743195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"72'28","reported_seq":292,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.840745+0000","last_change":"2024-04-14T15:13:35.350892+0000","last_active":"2024-04-14T15:13:38.840745+0000","last_peered":"2024-04-14T15:13:38.840745+0000","last_clean":"2024-04-14T15:13:38.840745+0000","last_became_active":"2024-04-14T15:09:45.034168+0000","last_became_peered":"2024-04-14T15:09:45.034168+0000","last_unstale":"2024-04-14T15:13:38.840745+0000","last_undegraded":"2024-04-14T15:13:38.840745+0000","last_fullsized":"2024-04-14T15:13:38.840745+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'28","last_scrub_stamp":"2024-04-14T15:13:35.350790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:13:35.350790+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:47.137871+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":200,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.828085+0000","last_change":"2024-04-14T15:13:25.908505+0000","last_active":"2024-04-14T15:13:38.828085+0000","last_peered":"2024-04-14T15:13:38.828085+0000","last_clean":"2024-04-14T15:13:38.828085+0000","last_became_active":"2024-04-14T15:09:43.028362+0000","last_became_peered":"2024-04-14T15:09:43.028362+0000","last_unstale":"2024-04-14T15:13:38.828085+0000","last_undegraded":"2024-04-14T15:13:38.828085+0000","last_fullsized":"2024-04-14T15:13:38.828085+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:25.908435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:13:25.908435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:52.100586+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":127,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.845509+0000","last_change":"2024-04-14T15:12:35.494479+0000","last_active":"2024-04-14T15:13:38.845509+0000","last_peered":"2024-04-14T15:13:38.845509+0000","last_clean":"2024-04-14T15:13:38.845509+0000","last_became_active":"2024-04-14T15:11:57.117900+0000","last_became_peered":"2024-04-14T15:11:57.117900+0000","last_unstale":"2024-04-14T15:13:38.845509+0000","last_undegraded":"2024-04-14T15:13:38.845509+0000","last_fullsized":"2024-04-14T15:13:38.845509+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:35.494389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:12:35.494389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:47.788465+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"109'5","reported_seq":102,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.849606+0000","last_change":"2024-04-14T15:12:56.320602+0000","last_active":"2024-04-14T15:13:38.849606+0000","last_peered":"2024-04-14T15:13:38.849606+0000","last_clean":"2024-04-14T15:13:38.849606+0000","last_became_active":"2024-04-14T15:12:56.318375+0000","last_became_peered":"2024-04-14T15:12:56.318375+0000","last_unstale":"2024-04-14T15:13:38.849606+0000","last_undegraded":"2024-04-14T15:13:38.849606+0000","last_fullsized":"2024-04-14T15:13:38.849606+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:09.778606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.10","version":"108'1","reported_seq":98,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.845926+0000","last_change":"2024-04-14T15:12:56.441371+0000","last_active":"2024-04-14T15:13:38.845926+0000","last_peered":"2024-04-14T15:13:38.845926+0000","last_clean":"2024-04-14T15:13:38.845926+0000","last_became_active":"2024-04-14T15:12:56.439533+0000","last_became_peered":"2024-04-14T15:12:56.439533+0000","last_unstale":"2024-04-14T15:13:38.845926+0000","last_undegraded":"2024-04-14T15:13:38.845926+0000","last_fullsized":"2024-04-14T15:13:38.845926+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:18.464342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.b","version":"108'2","reported_seq":99,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.831599+0000","last_change":"2024-04-14T15:12:56.419895+0000","last_active":"2024-04-14T15:13:38.831599+0000","last_peered":"2024-04-14T15:13:38.831599+0000","last_clean":"2024-04-14T15:13:38.831599+0000","last_became_active":"2024-04-14T15:12:56.417723+0000","last_became_peered":"2024-04-14T15:12:56.417723+0000","last_unstale":"2024-04-14T15:13:38.831599+0000","last_undegraded":"2024-04-14T15:13:38.831599+0000","last_fullsized":"2024-04-14T15:13:38.831599+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:12.195150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":97,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.860824+0000","last_change":"2024-04-14T15:12:56.351081+0000","last_active":"2024-04-14T15:13:38.860824+0000","last_peered":"2024-04-14T15:13:38.860824+0000","last_clean":"2024-04-14T15:13:38.860824+0000","last_became_active":"2024-04-14T15:12:56.350703+0000","last_became_peered":"2024-04-14T15:12:56.350703+0000","last_unstale":"2024-04-14T15:13:38.860824+0000","last_undegraded":"2024-04-14T15:13:38.860824+0000","last_fullsized":"2024-04-14T15:13:38.860824+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:14.515903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"72'8","reported_seq":248,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.884112+0000","last_change":"2024-04-14T15:12:20.089824+0000","last_active":"2024-04-14T15:13:38.884112+0000","last_peered":"2024-04-14T15:13:38.884112+0000","last_clean":"2024-04-14T15:13:38.884112+0000","last_became_active":"2024-04-14T15:09:47.047699+0000","last_became_peered":"2024-04-14T15:09:47.047699+0000","last_unstale":"2024-04-14T15:13:38.884112+0000","last_undegraded":"2024-04-14T15:13:38.884112+0000","last_fullsized":"2024-04-14T15:13:38.884112+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:12:20.089708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:12:20.089708+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"72'10","reported_seq":190,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.850076+0000","last_change":"2024-04-14T15:13:22.268455+0000","last_active":"2024-04-14T15:13:38.850076+0000","last_peered":"2024-04-14T15:13:38.850076+0000","last_clean":"2024-04-14T15:13:38.850076+0000","last_became_active":"2024-04-14T15:10:07.764042+0000","last_became_peered":"2024-04-14T15:10:07.764042+0000","last_unstale":"2024-04-14T15:13:38.850076+0000","last_undegraded":"2024-04-14T15:13:38.850076+0000","last_fullsized":"2024-04-14T15:13:38.850076+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-04-14T15:13:22.268301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:13:22.268301+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:40.226570+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":192,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.840658+0000","last_change":"2024-04-14T15:11:42.065911+0000","last_active":"2024-04-14T15:13:38.840658+0000","last_peered":"2024-04-14T15:13:38.840658+0000","last_clean":"2024-04-14T15:13:38.840658+0000","last_became_active":"2024-04-14T15:09:43.022597+0000","last_became_peered":"2024-04-14T15:09:43.022597+0000","last_unstale":"2024-04-14T15:13:38.840658+0000","last_undegraded":"2024-04-14T15:13:38.840658+0000","last_fullsized":"2024-04-14T15:13:38.840658+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:42.065812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:42.065812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":188,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.884140+0000","last_change":"2024-04-14T15:13:22.363505+0000","last_active":"2024-04-14T15:13:38.884140+0000","last_peered":"2024-04-14T15:13:38.884140+0000","last_clean":"2024-04-14T15:13:38.884140+0000","last_became_active":"2024-04-14T15:09:49.066648+0000","last_became_peered":"2024-04-14T15:09:49.066648+0000","last_unstale":"2024-04-14T15:13:38.884140+0000","last_undegraded":"2024-04-14T15:13:38.884140+0000","last_fullsized":"2024-04-14T15:13:38.884140+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:22.363370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:13:22.363370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:28.560720+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.16","version":"108'1","reported_seq":98,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.876874+0000","last_change":"2024-04-14T15:12:56.379069+0000","last_active":"2024-04-14T15:13:38.876874+0000","last_peered":"2024-04-14T15:13:38.876874+0000","last_clean":"2024-04-14T15:13:38.876874+0000","last_became_active":"2024-04-14T15:12:56.378696+0000","last_became_peered":"2024-04-14T15:12:56.378696+0000","last_unstale":"2024-04-14T15:13:38.876874+0000","last_undegraded":"2024-04-14T15:13:38.876874+0000","last_fullsized":"2024-04-14T15:13:38.876874+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:03.117831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.0","version":"109'5","reported_seq":102,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.839827+0000","last_change":"2024-04-14T15:12:56.204809+0000","last_active":"2024-04-14T15:13:38.839827+0000","last_peered":"2024-04-14T15:13:38.839827+0000","last_clean":"2024-04-14T15:13:38.839827+0000","last_became_active":"2024-04-14T15:12:56.204478+0000","last_became_peered":"2024-04-14T15:12:56.204478+0000","last_unstale":"2024-04-14T15:13:38.839827+0000","last_undegraded":"2024-04-14T15:13:38.839827+0000","last_fullsized":"2024-04-14T15:13:38.839827+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:56.114996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":97,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.885397+0000","last_change":"2024-04-14T15:12:56.564619+0000","last_active":"2024-04-14T15:13:38.885397+0000","last_peered":"2024-04-14T15:13:38.885397+0000","last_clean":"2024-04-14T15:13:38.885397+0000","last_became_active":"2024-04-14T15:12:56.563074+0000","last_became_peered":"2024-04-14T15:12:56.563074+0000","last_unstale":"2024-04-14T15:13:38.885397+0000","last_undegraded":"2024-04-14T15:13:38.885397+0000","last_fullsized":"2024-04-14T15:13:38.885397+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:06.308554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":192,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.846438+0000","last_change":"2024-04-14T15:13:28.305324+0000","last_active":"2024-04-14T15:13:38.846438+0000","last_peered":"2024-04-14T15:13:38.846438+0000","last_clean":"2024-04-14T15:13:38.846438+0000","last_became_active":"2024-04-14T15:09:47.049005+0000","last_became_peered":"2024-04-14T15:09:47.049005+0000","last_unstale":"2024-04-14T15:13:38.846438+0000","last_undegraded":"2024-04-14T15:13:38.846438+0000","last_fullsized":"2024-04-14T15:13:38.846438+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:28.305189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:13:28.305189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:33.710433+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"72'12","reported_seq":219,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.837633+0000","last_change":"2024-04-14T15:12:42.459812+0000","last_active":"2024-04-14T15:13:38.837633+0000","last_peered":"2024-04-14T15:13:38.837633+0000","last_clean":"2024-04-14T15:13:38.837633+0000","last_became_active":"2024-04-14T15:09:45.032892+0000","last_became_peered":"2024-04-14T15:09:45.032892+0000","last_unstale":"2024-04-14T15:13:38.837633+0000","last_undegraded":"2024-04-14T15:13:38.837633+0000","last_fullsized":"2024-04-14T15:13:38.837633+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'12","last_scrub_stamp":"2024-04-14T15:12:42.459703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:12:42.459703+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:46.045473+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"65'3","reported_seq":218,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.843860+0000","last_change":"2024-04-14T15:13:32.537671+0000","last_active":"2024-04-14T15:13:38.843860+0000","last_peered":"2024-04-14T15:13:38.843860+0000","last_clean":"2024-04-14T15:13:38.843860+0000","last_became_active":"2024-04-14T15:09:43.020490+0000","last_became_peered":"2024-04-14T15:09:43.020490+0000","last_unstale":"2024-04-14T15:13:38.843860+0000","last_undegraded":"2024-04-14T15:13:38.843860+0000","last_fullsized":"2024-04-14T15:13:38.843860+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3","last_scrub_stamp":"2024-04-14T15:13:32.537579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:13:32.537579+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:48.761569+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":528,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":127,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.838088+0000","last_change":"2024-04-14T15:12:26.063496+0000","last_active":"2024-04-14T15:13:38.838088+0000","last_peered":"2024-04-14T15:13:38.838088+0000","last_clean":"2024-04-14T15:13:38.838088+0000","last_became_active":"2024-04-14T15:11:57.118796+0000","last_became_peered":"2024-04-14T15:11:57.118796+0000","last_unstale":"2024-04-14T15:13:38.838088+0000","last_undegraded":"2024-04-14T15:13:38.838088+0000","last_fullsized":"2024-04-14T15:13:38.838088+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:26.063417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:12:26.063417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:44.294587+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1d","version":"110'2","reported_seq":102,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.861061+0000","last_change":"2024-04-14T15:12:56.388959+0000","last_active":"2024-04-14T15:13:38.861061+0000","last_peered":"2024-04-14T15:13:38.861061+0000","last_clean":"2024-04-14T15:13:38.861061+0000","last_became_active":"2024-04-14T15:12:56.380053+0000","last_became_peered":"2024-04-14T15:12:56.380053+0000","last_unstale":"2024-04-14T15:13:38.861061+0000","last_undegraded":"2024-04-14T15:13:38.861061+0000","last_fullsized":"2024-04-14T15:13:38.861061+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:01.878750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":97,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.885990+0000","last_change":"2024-04-14T15:12:56.553547+0000","last_active":"2024-04-14T15:13:38.885990+0000","last_peered":"2024-04-14T15:13:38.885990+0000","last_clean":"2024-04-14T15:13:38.885990+0000","last_became_active":"2024-04-14T15:12:56.551177+0000","last_became_peered":"2024-04-14T15:12:56.551177+0000","last_unstale":"2024-04-14T15:13:38.885990+0000","last_undegraded":"2024-04-14T15:13:38.885990+0000","last_fullsized":"2024-04-14T15:13:38.885990+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:04.301369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"72'9","reported_seq":212,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.885983+0000","last_change":"2024-04-14T15:13:37.516095+0000","last_active":"2024-04-14T15:13:38.885983+0000","last_peered":"2024-04-14T15:13:38.885983+0000","last_clean":"2024-04-14T15:13:38.885983+0000","last_became_active":"2024-04-14T15:09:45.180375+0000","last_became_peered":"2024-04-14T15:09:45.180375+0000","last_unstale":"2024-04-14T15:13:38.885983+0000","last_undegraded":"2024-04-14T15:13:38.885983+0000","last_fullsized":"2024-04-14T15:13:38.885983+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-04-14T15:13:37.515984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:13:37.515984+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:15:02.208462+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":200,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.846281+0000","last_change":"2024-04-14T15:12:30.462527+0000","last_active":"2024-04-14T15:13:38.846281+0000","last_peered":"2024-04-14T15:13:38.846281+0000","last_clean":"2024-04-14T15:13:38.846281+0000","last_became_active":"2024-04-14T15:09:43.016788+0000","last_became_peered":"2024-04-14T15:09:43.016788+0000","last_unstale":"2024-04-14T15:13:38.846281+0000","last_undegraded":"2024-04-14T15:13:38.846281+0000","last_fullsized":"2024-04-14T15:13:38.846281+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:30.462398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:12:30.462398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:54.452936+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":126,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.842743+0000","last_change":"2024-04-14T15:13:14.401268+0000","last_active":"2024-04-14T15:13:38.842743+0000","last_peered":"2024-04-14T15:13:38.842743+0000","last_clean":"2024-04-14T15:13:38.842743+0000","last_became_active":"2024-04-14T15:11:57.119468+0000","last_became_peered":"2024-04-14T15:11:57.119468+0000","last_unstale":"2024-04-14T15:13:38.842743+0000","last_undegraded":"2024-04-14T15:13:38.842743+0000","last_fullsized":"2024-04-14T15:13:38.842743+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:14.401142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:13:14.401142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:23.973809+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":188,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.859129+0000","last_change":"2024-04-14T15:12:41.389002+0000","last_active":"2024-04-14T15:13:38.859129+0000","last_peered":"2024-04-14T15:13:38.859129+0000","last_clean":"2024-04-14T15:13:38.859129+0000","last_became_active":"2024-04-14T15:09:49.064296+0000","last_became_peered":"2024-04-14T15:09:49.064296+0000","last_unstale":"2024-04-14T15:13:38.859129+0000","last_undegraded":"2024-04-14T15:13:38.859129+0000","last_fullsized":"2024-04-14T15:13:38.859129+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:41.388867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:12:41.388867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:10.014704+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.1f","version":"109'5","reported_seq":102,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.873326+0000","last_change":"2024-04-14T15:12:56.381181+0000","last_active":"2024-04-14T15:13:38.873326+0000","last_peered":"2024-04-14T15:13:38.873326+0000","last_clean":"2024-04-14T15:13:38.873326+0000","last_became_active":"2024-04-14T15:12:56.375790+0000","last_became_peered":"2024-04-14T15:12:56.375790+0000","last_unstale":"2024-04-14T15:13:38.873326+0000","last_undegraded":"2024-04-14T15:13:38.873326+0000","last_fullsized":"2024-04-14T15:13:38.873326+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:24.373609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":115,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.841745+0000","last_change":"2024-04-14T15:13:30.531320+0000","last_active":"2024-04-14T15:13:38.841745+0000","last_peered":"2024-04-14T15:13:38.841745+0000","last_clean":"2024-04-14T15:13:38.841745+0000","last_became_active":"2024-04-14T15:12:56.321327+0000","last_became_peered":"2024-04-14T15:12:56.321327+0000","last_unstale":"2024-04-14T15:13:38.841745+0000","last_undegraded":"2024-04-14T15:13:38.841745+0000","last_fullsized":"2024-04-14T15:13:38.841745+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:30.531240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:30.531240+0000","last_clean_scrub_stamp":"2024-04-14T15:13:30.531240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:58.104309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":97,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.846130+0000","last_change":"2024-04-14T15:12:56.340638+0000","last_active":"2024-04-14T15:13:38.846130+0000","last_peered":"2024-04-14T15:13:38.846130+0000","last_clean":"2024-04-14T15:13:38.846130+0000","last_became_active":"2024-04-14T15:12:56.340066+0000","last_became_peered":"2024-04-14T15:12:56.340066+0000","last_unstale":"2024-04-14T15:13:38.846130+0000","last_undegraded":"2024-04-14T15:13:38.846130+0000","last_fullsized":"2024-04-14T15:13:38.846130+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:58.829045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.17","version":"72'6","reported_seq":205,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.832614+0000","last_change":"2024-04-14T15:12:32.412850+0000","last_active":"2024-04-14T15:13:38.832614+0000","last_peered":"2024-04-14T15:13:38.832614+0000","last_clean":"2024-04-14T15:13:38.832614+0000","last_became_active":"2024-04-14T15:09:45.180037+0000","last_became_peered":"2024-04-14T15:09:45.180037+0000","last_unstale":"2024-04-14T15:13:38.832614+0000","last_undegraded":"2024-04-14T15:13:38.832614+0000","last_fullsized":"2024-04-14T15:13:38.832614+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'6","last_scrub_stamp":"2024-04-14T15:12:32.412700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:12:32.412700+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:52.270276+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":200,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.872364+0000","last_change":"2024-04-14T15:12:37.098768+0000","last_active":"2024-04-14T15:13:38.872364+0000","last_peered":"2024-04-14T15:13:38.872364+0000","last_clean":"2024-04-14T15:13:38.872364+0000","last_became_active":"2024-04-14T15:09:43.018987+0000","last_became_peered":"2024-04-14T15:09:43.018987+0000","last_unstale":"2024-04-14T15:13:38.872364+0000","last_undegraded":"2024-04-14T15:13:38.872364+0000","last_fullsized":"2024-04-14T15:13:38.872364+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:37.098686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:12:37.098686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:44.694811+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":192,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.831891+0000","last_change":"2024-04-14T15:12:29.760123+0000","last_active":"2024-04-14T15:13:38.831891+0000","last_peered":"2024-04-14T15:13:38.831891+0000","last_clean":"2024-04-14T15:13:38.831891+0000","last_became_active":"2024-04-14T15:09:47.045349+0000","last_became_peered":"2024-04-14T15:09:47.045349+0000","last_unstale":"2024-04-14T15:13:38.831891+0000","last_undegraded":"2024-04-14T15:13:38.831891+0000","last_fullsized":"2024-04-14T15:13:38.831891+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:29.759997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:10:53.900878+0000","last_clean_scrub_stamp":"2024-04-14T15:12:29.759997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:50.085831+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":188,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.846353+0000","last_change":"2024-04-14T15:13:36.173906+0000","last_active":"2024-04-14T15:13:38.846353+0000","last_peered":"2024-04-14T15:13:38.846353+0000","last_clean":"2024-04-14T15:13:38.846353+0000","last_became_active":"2024-04-14T15:09:49.130438+0000","last_became_peered":"2024-04-14T15:09:49.130438+0000","last_unstale":"2024-04-14T15:13:38.846353+0000","last_undegraded":"2024-04-14T15:13:38.846353+0000","last_fullsized":"2024-04-14T15:13:38.846353+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:36.173813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:13:36.173813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:15:03.922490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":97,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.841219+0000","last_change":"2024-04-14T15:12:56.267286+0000","last_active":"2024-04-14T15:13:38.841219+0000","last_peered":"2024-04-14T15:13:38.841219+0000","last_clean":"2024-04-14T15:13:38.841219+0000","last_became_active":"2024-04-14T15:12:56.265693+0000","last_became_peered":"2024-04-14T15:12:56.265693+0000","last_unstale":"2024-04-14T15:13:38.841219+0000","last_undegraded":"2024-04-14T15:13:38.841219+0000","last_fullsized":"2024-04-14T15:13:38.841219+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:19.787655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":115,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.826719+0000","last_change":"2024-04-14T15:13:30.875464+0000","last_active":"2024-04-14T15:13:38.826719+0000","last_peered":"2024-04-14T15:13:38.826719+0000","last_clean":"2024-04-14T15:13:38.826719+0000","last_became_active":"2024-04-14T15:12:56.230594+0000","last_became_peered":"2024-04-14T15:12:56.230594+0000","last_unstale":"2024-04-14T15:13:38.826719+0000","last_undegraded":"2024-04-14T15:13:38.826719+0000","last_fullsized":"2024-04-14T15:13:38.826719+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:30.875353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:30.875353+0000","last_clean_scrub_stamp":"2024-04-14T15:13:30.875353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:35.399282+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":97,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.836929+0000","last_change":"2024-04-14T15:12:56.206105+0000","last_active":"2024-04-14T15:13:38.836929+0000","last_peered":"2024-04-14T15:13:38.836929+0000","last_clean":"2024-04-14T15:13:38.836929+0000","last_became_active":"2024-04-14T15:12:56.203577+0000","last_became_peered":"2024-04-14T15:12:56.203577+0000","last_unstale":"2024-04-14T15:13:38.836929+0000","last_undegraded":"2024-04-14T15:13:38.836929+0000","last_fullsized":"2024-04-14T15:13:38.836929+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:14.215791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"72'10","reported_seq":160,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.841328+0000","last_change":"2024-04-14T15:12:34.297256+0000","last_active":"2024-04-14T15:13:38.841328+0000","last_peered":"2024-04-14T15:13:38.841328+0000","last_clean":"2024-04-14T15:13:38.841328+0000","last_became_active":"2024-04-14T15:11:57.126453+0000","last_became_peered":"2024-04-14T15:11:57.126453+0000","last_unstale":"2024-04-14T15:13:38.841328+0000","last_undegraded":"2024-04-14T15:13:38.841328+0000","last_fullsized":"2024-04-14T15:13:38.841328+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-04-14T15:12:34.297111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:12:34.297111+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:57.599915+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":200,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.827022+0000","last_change":"2024-04-14T15:13:33.770490+0000","last_active":"2024-04-14T15:13:38.827022+0000","last_peered":"2024-04-14T15:13:38.827022+0000","last_clean":"2024-04-14T15:13:38.827022+0000","last_became_active":"2024-04-14T15:09:43.027776+0000","last_became_peered":"2024-04-14T15:09:43.027776+0000","last_unstale":"2024-04-14T15:13:38.827022+0000","last_undegraded":"2024-04-14T15:13:38.827022+0000","last_fullsized":"2024-04-14T15:13:38.827022+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:33.770408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:13:33.770408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:53.137349+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"72'8","reported_seq":248,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.831469+0000","last_change":"2024-04-14T15:12:40.796148+0000","last_active":"2024-04-14T15:13:38.831469+0000","last_peered":"2024-04-14T15:13:38.831469+0000","last_clean":"2024-04-14T15:13:38.831469+0000","last_became_active":"2024-04-14T15:09:47.036688+0000","last_became_peered":"2024-04-14T15:09:47.036688+0000","last_unstale":"2024-04-14T15:13:38.831469+0000","last_undegraded":"2024-04-14T15:13:38.831469+0000","last_fullsized":"2024-04-14T15:13:38.831469+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:12:40.796043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:12:40.796043+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:52.859244+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":180,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.885448+0000","last_change":"2024-04-14T15:11:40.922318+0000","last_active":"2024-04-14T15:13:38.885448+0000","last_peered":"2024-04-14T15:13:38.885448+0000","last_clean":"2024-04-14T15:13:38.885448+0000","last_became_active":"2024-04-14T15:09:49.066068+0000","last_became_peered":"2024-04-14T15:09:49.066068+0000","last_unstale":"2024-04-14T15:13:38.885448+0000","last_undegraded":"2024-04-14T15:13:38.885448+0000","last_fullsized":"2024-04-14T15:13:38.885448+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:40.922231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:40.922231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"109'7","reported_seq":104,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.846539+0000","last_change":"2024-04-14T15:12:56.240388+0000","last_active":"2024-04-14T15:13:38.846539+0000","last_peered":"2024-04-14T15:13:38.846539+0000","last_clean":"2024-04-14T15:13:38.846539+0000","last_became_active":"2024-04-14T15:12:56.238786+0000","last_became_peered":"2024-04-14T15:12:56.238786+0000","last_unstale":"2024-04-14T15:13:38.846539+0000","last_undegraded":"2024-04-14T15:13:38.846539+0000","last_fullsized":"2024-04-14T15:13:38.846539+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:18.200488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":115,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.883810+0000","last_change":"2024-04-14T15:13:32.537830+0000","last_active":"2024-04-14T15:13:38.883810+0000","last_peered":"2024-04-14T15:13:38.883810+0000","last_clean":"2024-04-14T15:13:38.883810+0000","last_became_active":"2024-04-14T15:12:56.546929+0000","last_became_peered":"2024-04-14T15:12:56.546929+0000","last_unstale":"2024-04-14T15:13:38.883810+0000","last_undegraded":"2024-04-14T15:13:38.883810+0000","last_fullsized":"2024-04-14T15:13:38.883810+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:32.537745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:32.537745+0000","last_clean_scrub_stamp":"2024-04-14T15:13:32.537745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:38.124845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":97,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.827989+0000","last_change":"2024-04-14T15:12:56.375745+0000","last_active":"2024-04-14T15:13:38.827989+0000","last_peered":"2024-04-14T15:13:38.827989+0000","last_clean":"2024-04-14T15:13:38.827989+0000","last_became_active":"2024-04-14T15:12:56.358025+0000","last_became_peered":"2024-04-14T15:12:56.358025+0000","last_unstale":"2024-04-14T15:13:38.827989+0000","last_undegraded":"2024-04-14T15:13:38.827989+0000","last_fullsized":"2024-04-14T15:13:38.827989+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:10.090074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"72'9","reported_seq":212,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.830294+0000","last_change":"2024-04-14T15:12:32.762712+0000","last_active":"2024-04-14T15:13:38.830294+0000","last_peered":"2024-04-14T15:13:38.830294+0000","last_clean":"2024-04-14T15:13:38.830294+0000","last_became_active":"2024-04-14T15:09:45.178857+0000","last_became_peered":"2024-04-14T15:09:45.178857+0000","last_unstale":"2024-04-14T15:13:38.830294+0000","last_undegraded":"2024-04-14T15:13:38.830294+0000","last_fullsized":"2024-04-14T15:13:38.830294+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-04-14T15:12:32.762612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:12:32.762612+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:42.295699+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":200,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.885659+0000","last_change":"2024-04-14T15:12:34.037607+0000","last_active":"2024-04-14T15:13:38.885659+0000","last_peered":"2024-04-14T15:13:38.885659+0000","last_clean":"2024-04-14T15:13:38.885659+0000","last_became_active":"2024-04-14T15:09:43.019301+0000","last_became_peered":"2024-04-14T15:09:43.019301+0000","last_unstale":"2024-04-14T15:13:38.885659+0000","last_undegraded":"2024-04-14T15:13:38.885659+0000","last_fullsized":"2024-04-14T15:13:38.885659+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:34.037482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:12:34.037482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:43.093527+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"98'9","reported_seq":175,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.848324+0000","last_change":"2024-04-14T15:12:39.046563+0000","last_active":"2024-04-14T15:13:38.848324+0000","last_peered":"2024-04-14T15:13:38.848324+0000","last_clean":"2024-04-14T15:13:38.848324+0000","last_became_active":"2024-04-14T15:11:57.124325+0000","last_became_peered":"2024-04-14T15:11:57.124325+0000","last_unstale":"2024-04-14T15:13:38.848324+0000","last_undegraded":"2024-04-14T15:13:38.848324+0000","last_fullsized":"2024-04-14T15:13:38.848324+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-04-14T15:12:39.046477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:12:39.046477+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:43.566900+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":188,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.845115+0000","last_change":"2024-04-14T15:13:17.622241+0000","last_active":"2024-04-14T15:13:38.845115+0000","last_peered":"2024-04-14T15:13:38.845115+0000","last_clean":"2024-04-14T15:13:38.845115+0000","last_became_active":"2024-04-14T15:09:49.065242+0000","last_became_peered":"2024-04-14T15:09:49.065242+0000","last_unstale":"2024-04-14T15:13:38.845115+0000","last_undegraded":"2024-04-14T15:13:38.845115+0000","last_fullsized":"2024-04-14T15:13:38.845115+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:17.622129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:13:17.622129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:32.752780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":97,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.872079+0000","last_change":"2024-04-14T15:12:56.411418+0000","last_active":"2024-04-14T15:13:38.872079+0000","last_peered":"2024-04-14T15:13:38.872079+0000","last_clean":"2024-04-14T15:13:38.872079+0000","last_became_active":"2024-04-14T15:12:56.404918+0000","last_became_peered":"2024-04-14T15:12:56.404918+0000","last_unstale":"2024-04-14T15:13:38.872079+0000","last_undegraded":"2024-04-14T15:13:38.872079+0000","last_fullsized":"2024-04-14T15:13:38.872079+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:19.701234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1","version":"142'13","reported_seq":139,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.848215+0000","last_change":"2024-04-14T15:13:30.361747+0000","last_active":"2024-04-14T15:13:38.848215+0000","last_peered":"2024-04-14T15:13:38.848215+0000","last_clean":"2024-04-14T15:13:38.848215+0000","last_became_active":"2024-04-14T15:12:56.367401+0000","last_became_peered":"2024-04-14T15:12:56.367401+0000","last_unstale":"2024-04-14T15:13:38.848215+0000","last_undegraded":"2024-04-14T15:13:38.848215+0000","last_fullsized":"2024-04-14T15:13:38.848215+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"142'13","last_scrub_stamp":"2024-04-14T15:13:30.361654+0000","last_deep_scrub":"142'13","last_deep_scrub_stamp":"2024-04-14T15:13:30.361654+0000","last_clean_scrub_stamp":"2024-04-14T15:13:30.361654+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:41.128987+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331735,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":49156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":97,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.846448+0000","last_change":"2024-04-14T15:12:56.389498+0000","last_active":"2024-04-14T15:13:38.846448+0000","last_peered":"2024-04-14T15:13:38.846448+0000","last_clean":"2024-04-14T15:13:38.846448+0000","last_became_active":"2024-04-14T15:12:56.389204+0000","last_became_peered":"2024-04-14T15:12:56.389204+0000","last_unstale":"2024-04-14T15:13:38.846448+0000","last_undegraded":"2024-04-14T15:13:38.846448+0000","last_fullsized":"2024-04-14T15:13:38.846448+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:15.258772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"72'9","reported_seq":212,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.834211+0000","last_change":"2024-04-14T15:13:24.378688+0000","last_active":"2024-04-14T15:13:38.834211+0000","last_peered":"2024-04-14T15:13:38.834211+0000","last_clean":"2024-04-14T15:13:38.834211+0000","last_became_active":"2024-04-14T15:09:45.026363+0000","last_became_peered":"2024-04-14T15:09:45.026363+0000","last_unstale":"2024-04-14T15:13:38.834211+0000","last_undegraded":"2024-04-14T15:13:38.834211+0000","last_fullsized":"2024-04-14T15:13:38.834211+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-04-14T15:13:24.378555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:13:24.378555+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:49.661349+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":200,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.872502+0000","last_change":"2024-04-14T15:12:36.065034+0000","last_active":"2024-04-14T15:13:38.872502+0000","last_peered":"2024-04-14T15:13:38.872502+0000","last_clean":"2024-04-14T15:13:38.872502+0000","last_became_active":"2024-04-14T15:09:43.022480+0000","last_became_peered":"2024-04-14T15:09:43.022480+0000","last_unstale":"2024-04-14T15:13:38.872502+0000","last_undegraded":"2024-04-14T15:13:38.872502+0000","last_fullsized":"2024-04-14T15:13:38.872502+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:36.064950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:12:36.064950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:47.576737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":126,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.885874+0000","last_change":"2024-04-14T15:12:36.029185+0000","last_active":"2024-04-14T15:13:38.885874+0000","last_peered":"2024-04-14T15:13:38.885874+0000","last_clean":"2024-04-14T15:13:38.885874+0000","last_became_active":"2024-04-14T15:11:57.121871+0000","last_became_peered":"2024-04-14T15:11:57.121871+0000","last_unstale":"2024-04-14T15:13:38.885874+0000","last_undegraded":"2024-04-14T15:13:38.885874+0000","last_fullsized":"2024-04-14T15:13:38.885874+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:36.029063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:12:36.029063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:56.653231+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":188,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.883954+0000","last_change":"2024-04-14T15:12:31.969664+0000","last_active":"2024-04-14T15:13:38.883954+0000","last_peered":"2024-04-14T15:13:38.883954+0000","last_clean":"2024-04-14T15:13:38.883954+0000","last_became_active":"2024-04-14T15:09:49.059711+0000","last_became_peered":"2024-04-14T15:09:49.059711+0000","last_unstale":"2024-04-14T15:13:38.883954+0000","last_undegraded":"2024-04-14T15:13:38.883954+0000","last_fullsized":"2024-04-14T15:13:38.883954+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:31.969587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:12:31.969587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:55.245372+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1b","version":"109'10","reported_seq":107,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.836822+0000","last_change":"2024-04-14T15:12:56.261118+0000","last_active":"2024-04-14T15:13:38.836822+0000","last_peered":"2024-04-14T15:13:38.836822+0000","last_clean":"2024-04-14T15:13:38.836822+0000","last_became_active":"2024-04-14T15:12:56.259380+0000","last_became_peered":"2024-04-14T15:12:56.259380+0000","last_unstale":"2024-04-14T15:13:38.836822+0000","last_undegraded":"2024-04-14T15:13:38.836822+0000","last_fullsized":"2024-04-14T15:13:38.836822+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:59.334287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.6","version":"143'4","reported_seq":119,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.876903+0000","last_change":"2024-04-14T15:13:31.016293+0000","last_active":"2024-04-14T15:13:38.876903+0000","last_peered":"2024-04-14T15:13:38.876903+0000","last_clean":"2024-04-14T15:13:38.876903+0000","last_became_active":"2024-04-14T15:12:56.418987+0000","last_became_peered":"2024-04-14T15:12:56.418987+0000","last_unstale":"2024-04-14T15:13:38.876903+0000","last_undegraded":"2024-04-14T15:13:38.876903+0000","last_fullsized":"2024-04-14T15:13:38.876903+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'4","last_scrub_stamp":"2024-04-14T15:13:31.016207+0000","last_deep_scrub":"143'4","last_deep_scrub_stamp":"2024-04-14T15:13:31.016207+0000","last_clean_scrub_stamp":"2024-04-14T15:13:31.016207+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:55.026110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":97,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.872274+0000","last_change":"2024-04-14T15:12:56.406594+0000","last_active":"2024-04-14T15:13:38.872274+0000","last_peered":"2024-04-14T15:13:38.872274+0000","last_clean":"2024-04-14T15:13:38.872274+0000","last_became_active":"2024-04-14T15:12:56.396294+0000","last_became_peered":"2024-04-14T15:12:56.396294+0000","last_unstale":"2024-04-14T15:13:38.872274+0000","last_undegraded":"2024-04-14T15:13:38.872274+0000","last_fullsized":"2024-04-14T15:13:38.872274+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:13.139188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"72'11","reported_seq":215,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.837642+0000","last_change":"2024-04-14T15:13:35.599422+0000","last_active":"2024-04-14T15:13:38.837642+0000","last_peered":"2024-04-14T15:13:38.837642+0000","last_clean":"2024-04-14T15:13:38.837642+0000","last_became_active":"2024-04-14T15:09:45.026065+0000","last_became_peered":"2024-04-14T15:09:45.026065+0000","last_unstale":"2024-04-14T15:13:38.837642+0000","last_undegraded":"2024-04-14T15:13:38.837642+0000","last_fullsized":"2024-04-14T15:13:38.837642+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-04-14T15:13:35.599327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:13:35.599327+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:47.281371+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":200,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.842958+0000","last_change":"2024-04-14T15:13:27.378656+0000","last_active":"2024-04-14T15:13:38.842958+0000","last_peered":"2024-04-14T15:13:38.842958+0000","last_clean":"2024-04-14T15:13:38.842958+0000","last_became_active":"2024-04-14T15:09:43.023583+0000","last_became_peered":"2024-04-14T15:09:43.023583+0000","last_unstale":"2024-04-14T15:13:38.842958+0000","last_undegraded":"2024-04-14T15:13:38.842958+0000","last_fullsized":"2024-04-14T15:13:38.842958+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:27.378503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:13:27.378503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:32.810609+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":184,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.834432+0000","last_change":"2024-04-14T15:11:35.007456+0000","last_active":"2024-04-14T15:13:38.834432+0000","last_peered":"2024-04-14T15:13:38.834432+0000","last_clean":"2024-04-14T15:13:38.834432+0000","last_became_active":"2024-04-14T15:09:47.041683+0000","last_became_peered":"2024-04-14T15:09:47.041683+0000","last_unstale":"2024-04-14T15:13:38.834432+0000","last_undegraded":"2024-04-14T15:13:38.834432+0000","last_fullsized":"2024-04-14T15:13:38.834432+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:35.007228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:35.007228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":127,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.887602+0000","last_change":"2024-04-14T15:13:34.547742+0000","last_active":"2024-04-14T15:13:38.887602+0000","last_peered":"2024-04-14T15:13:38.887602+0000","last_clean":"2024-04-14T15:13:38.887602+0000","last_became_active":"2024-04-14T15:11:57.118363+0000","last_became_peered":"2024-04-14T15:11:57.118363+0000","last_unstale":"2024-04-14T15:13:38.887602+0000","last_undegraded":"2024-04-14T15:13:38.887602+0000","last_fullsized":"2024-04-14T15:13:38.887602+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:34.547654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:13:34.547654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:15:03.799838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"109'5","reported_seq":102,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.834323+0000","last_change":"2024-04-14T15:12:56.548707+0000","last_active":"2024-04-14T15:13:38.834323+0000","last_peered":"2024-04-14T15:13:38.834323+0000","last_clean":"2024-04-14T15:13:38.834323+0000","last_became_active":"2024-04-14T15:12:56.547630+0000","last_became_peered":"2024-04-14T15:12:56.547630+0000","last_unstale":"2024-04-14T15:13:38.834323+0000","last_undegraded":"2024-04-14T15:13:38.834323+0000","last_fullsized":"2024-04-14T15:13:38.834323+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:22.520179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":115,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.892508+0000","last_change":"2024-04-14T15:13:33.556959+0000","last_active":"2024-04-14T15:13:38.892508+0000","last_peered":"2024-04-14T15:13:38.892508+0000","last_clean":"2024-04-14T15:13:38.892508+0000","last_became_active":"2024-04-14T15:12:56.567611+0000","last_became_peered":"2024-04-14T15:12:56.567611+0000","last_unstale":"2024-04-14T15:13:38.892508+0000","last_undegraded":"2024-04-14T15:13:38.892508+0000","last_fullsized":"2024-04-14T15:13:38.892508+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:33.556876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:33.556876+0000","last_clean_scrub_stamp":"2024-04-14T15:13:33.556876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:57.827376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":97,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.853344+0000","last_change":"2024-04-14T15:12:56.397674+0000","last_active":"2024-04-14T15:13:38.853344+0000","last_peered":"2024-04-14T15:13:38.853344+0000","last_clean":"2024-04-14T15:13:38.853344+0000","last_became_active":"2024-04-14T15:12:56.387723+0000","last_became_peered":"2024-04-14T15:12:56.387723+0000","last_unstale":"2024-04-14T15:13:38.853344+0000","last_undegraded":"2024-04-14T15:13:38.853344+0000","last_fullsized":"2024-04-14T15:13:38.853344+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:02.309403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.10","version":"72'4","reported_seq":202,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.826398+0000","last_change":"2024-04-14T15:13:31.218945+0000","last_active":"2024-04-14T15:13:38.826398+0000","last_peered":"2024-04-14T15:13:38.826398+0000","last_clean":"2024-04-14T15:13:38.826398+0000","last_became_active":"2024-04-14T15:09:45.038815+0000","last_became_peered":"2024-04-14T15:09:45.038815+0000","last_unstale":"2024-04-14T15:13:38.826398+0000","last_undegraded":"2024-04-14T15:13:38.826398+0000","last_fullsized":"2024-04-14T15:13:38.826398+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4","last_scrub_stamp":"2024-04-14T15:13:31.218855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:13:31.218855+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:31.655479+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":200,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.831515+0000","last_change":"2024-04-14T15:12:28.767429+0000","last_active":"2024-04-14T15:13:38.831515+0000","last_peered":"2024-04-14T15:13:38.831515+0000","last_clean":"2024-04-14T15:13:38.831515+0000","last_became_active":"2024-04-14T15:09:43.026786+0000","last_became_peered":"2024-04-14T15:09:43.026786+0000","last_unstale":"2024-04-14T15:13:38.831515+0000","last_undegraded":"2024-04-14T15:13:38.831515+0000","last_fullsized":"2024-04-14T15:13:38.831515+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:28.767299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:08.712289+0000","last_clean_scrub_stamp":"2024-04-14T15:12:28.767299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:58.477876+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":200,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.872513+0000","last_change":"2024-04-14T15:13:37.910269+0000","last_active":"2024-04-14T15:13:38.872513+0000","last_peered":"2024-04-14T15:13:38.872513+0000","last_clean":"2024-04-14T15:13:38.872513+0000","last_became_active":"2024-04-14T15:09:47.041074+0000","last_became_peered":"2024-04-14T15:09:47.041074+0000","last_unstale":"2024-04-14T15:13:38.872513+0000","last_undegraded":"2024-04-14T15:13:38.872513+0000","last_fullsized":"2024-04-14T15:13:38.872513+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:37.910180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:13:37.910180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:49.017818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":188,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.838812+0000","last_change":"2024-04-14T15:13:26.142140+0000","last_active":"2024-04-14T15:13:38.838812+0000","last_peered":"2024-04-14T15:13:38.838812+0000","last_clean":"2024-04-14T15:13:38.838812+0000","last_became_active":"2024-04-14T15:09:49.061359+0000","last_became_peered":"2024-04-14T15:09:49.061359+0000","last_unstale":"2024-04-14T15:13:38.838812+0000","last_undegraded":"2024-04-14T15:13:38.838812+0000","last_fullsized":"2024-04-14T15:13:38.838812+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:26.142060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:13:26.142060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:50.720229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":97,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.843020+0000","last_change":"2024-04-14T15:12:56.478218+0000","last_active":"2024-04-14T15:13:38.843020+0000","last_peered":"2024-04-14T15:13:38.843020+0000","last_clean":"2024-04-14T15:13:38.843020+0000","last_became_active":"2024-04-14T15:12:56.472810+0000","last_became_peered":"2024-04-14T15:12:56.472810+0000","last_unstale":"2024-04-14T15:13:38.843020+0000","last_undegraded":"2024-04-14T15:13:38.843020+0000","last_fullsized":"2024-04-14T15:13:38.843020+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:03.926999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.4","version":"143'5","reported_seq":122,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.840023+0000","last_change":"2024-04-14T15:13:31.287529+0000","last_active":"2024-04-14T15:13:38.840023+0000","last_peered":"2024-04-14T15:13:38.840023+0000","last_clean":"2024-04-14T15:13:38.840023+0000","last_became_active":"2024-04-14T15:12:56.423605+0000","last_became_peered":"2024-04-14T15:12:56.423605+0000","last_unstale":"2024-04-14T15:13:38.840023+0000","last_undegraded":"2024-04-14T15:13:38.840023+0000","last_fullsized":"2024-04-14T15:13:38.840023+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'5","last_scrub_stamp":"2024-04-14T15:13:31.287443+0000","last_deep_scrub":"143'5","last_deep_scrub_stamp":"2024-04-14T15:13:31.287443+0000","last_clean_scrub_stamp":"2024-04-14T15:13:31.287443+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:55.319665+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2800,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":97,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.849243+0000","last_change":"2024-04-14T15:12:56.352827+0000","last_active":"2024-04-14T15:13:38.849243+0000","last_peered":"2024-04-14T15:13:38.849243+0000","last_clean":"2024-04-14T15:13:38.849243+0000","last_became_active":"2024-04-14T15:12:56.352528+0000","last_became_peered":"2024-04-14T15:12:56.352528+0000","last_unstale":"2024-04-14T15:13:38.849243+0000","last_undegraded":"2024-04-14T15:13:38.849243+0000","last_fullsized":"2024-04-14T15:13:38.849243+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:19.096680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"102'16","reported_seq":169,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.842775+0000","last_change":"2024-04-14T15:12:35.999465+0000","last_active":"2024-04-14T15:13:38.842775+0000","last_peered":"2024-04-14T15:13:38.842775+0000","last_clean":"2024-04-14T15:13:38.842775+0000","last_became_active":"2024-04-14T15:11:57.118033+0000","last_became_peered":"2024-04-14T15:11:57.118033+0000","last_unstale":"2024-04-14T15:13:38.842775+0000","last_undegraded":"2024-04-14T15:13:38.842775+0000","last_fullsized":"2024-04-14T15:13:38.842775+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'16","last_scrub_stamp":"2024-04-14T15:12:35.999326+0000","last_deep_scrub":"102'16","last_deep_scrub_stamp":"2024-04-14T15:12:35.999326+0000","last_clean_scrub_stamp":"2024-04-14T15:12:35.999326+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":200,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.873643+0000","last_change":"2024-04-14T15:12:28.117794+0000","last_active":"2024-04-14T15:13:38.873643+0000","last_peered":"2024-04-14T15:13:38.873643+0000","last_clean":"2024-04-14T15:13:38.873643+0000","last_became_active":"2024-04-14T15:09:43.023220+0000","last_became_peered":"2024-04-14T15:09:43.023220+0000","last_unstale":"2024-04-14T15:13:38.873643+0000","last_undegraded":"2024-04-14T15:13:38.873643+0000","last_fullsized":"2024-04-14T15:13:38.873643+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:28.117703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:12:28.117703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:54.952704+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":192,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.838363+0000","last_change":"2024-04-14T15:13:29.219729+0000","last_active":"2024-04-14T15:13:38.838363+0000","last_peered":"2024-04-14T15:13:38.838363+0000","last_clean":"2024-04-14T15:13:38.838363+0000","last_became_active":"2024-04-14T15:09:47.044665+0000","last_became_peered":"2024-04-14T15:09:47.044665+0000","last_unstale":"2024-04-14T15:13:38.838363+0000","last_undegraded":"2024-04-14T15:13:38.838363+0000","last_fullsized":"2024-04-14T15:13:38.838363+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:29.219589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:13:29.219589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:53.096780+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":127,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.887754+0000","last_change":"2024-04-14T15:12:42.042180+0000","last_active":"2024-04-14T15:13:38.887754+0000","last_peered":"2024-04-14T15:13:38.887754+0000","last_clean":"2024-04-14T15:13:38.887754+0000","last_became_active":"2024-04-14T15:11:57.122266+0000","last_became_peered":"2024-04-14T15:11:57.122266+0000","last_unstale":"2024-04-14T15:13:38.887754+0000","last_undegraded":"2024-04-14T15:13:38.887754+0000","last_fullsized":"2024-04-14T15:13:38.887754+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:42.042065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:12:42.042065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:59.693385+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.18","version":"108'2","reported_seq":99,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.838456+0000","last_change":"2024-04-14T15:12:56.396161+0000","last_active":"2024-04-14T15:13:38.838456+0000","last_peered":"2024-04-14T15:13:38.838456+0000","last_clean":"2024-04-14T15:13:38.838456+0000","last_became_active":"2024-04-14T15:12:56.395867+0000","last_became_peered":"2024-04-14T15:12:56.395867+0000","last_unstale":"2024-04-14T15:13:38.838456+0000","last_undegraded":"2024-04-14T15:13:38.838456+0000","last_fullsized":"2024-04-14T15:13:38.838456+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:21.075072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.5","version":"143'4","reported_seq":123,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.885409+0000","last_change":"2024-04-14T15:13:31.592480+0000","last_active":"2024-04-14T15:13:38.885409+0000","last_peered":"2024-04-14T15:13:38.885409+0000","last_clean":"2024-04-14T15:13:38.885409+0000","last_became_active":"2024-04-14T15:12:56.450678+0000","last_became_peered":"2024-04-14T15:12:56.450678+0000","last_unstale":"2024-04-14T15:13:38.885409+0000","last_undegraded":"2024-04-14T15:13:38.885409+0000","last_fullsized":"2024-04-14T15:13:38.885409+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'4","last_scrub_stamp":"2024-04-14T15:13:31.592386+0000","last_deep_scrub":"143'4","last_deep_scrub_stamp":"2024-04-14T15:13:31.592386+0000","last_clean_scrub_stamp":"2024-04-14T15:13:31.592386+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:15:01.588772+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165843,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":24577,"num_write":2,"num_write_kb":24578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":97,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.848335+0000","last_change":"2024-04-14T15:12:56.336958+0000","last_active":"2024-04-14T15:13:38.848335+0000","last_peered":"2024-04-14T15:13:38.848335+0000","last_clean":"2024-04-14T15:13:38.848335+0000","last_became_active":"2024-04-14T15:12:56.335230+0000","last_became_peered":"2024-04-14T15:12:56.335230+0000","last_unstale":"2024-04-14T15:13:38.848335+0000","last_undegraded":"2024-04-14T15:13:38.848335+0000","last_fullsized":"2024-04-14T15:13:38.848335+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:22.112968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"72'10","reported_seq":211,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.887825+0000","last_change":"2024-04-14T15:13:17.411319+0000","last_active":"2024-04-14T15:13:38.887825+0000","last_peered":"2024-04-14T15:13:38.887825+0000","last_clean":"2024-04-14T15:13:38.887825+0000","last_became_active":"2024-04-14T15:09:45.181456+0000","last_became_peered":"2024-04-14T15:09:45.181456+0000","last_unstale":"2024-04-14T15:13:38.887825+0000","last_undegraded":"2024-04-14T15:13:38.887825+0000","last_fullsized":"2024-04-14T15:13:38.887825+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-04-14T15:13:17.411175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:13:17.411175+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:36.912024+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":200,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.825438+0000","last_change":"2024-04-14T15:12:22.784729+0000","last_active":"2024-04-14T15:13:38.825438+0000","last_peered":"2024-04-14T15:13:38.825438+0000","last_clean":"2024-04-14T15:13:38.825438+0000","last_became_active":"2024-04-14T15:09:43.023257+0000","last_became_peered":"2024-04-14T15:09:43.023257+0000","last_unstale":"2024-04-14T15:13:38.825438+0000","last_undegraded":"2024-04-14T15:13:38.825438+0000","last_fullsized":"2024-04-14T15:13:38.825438+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:22.784606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:12:22.784606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"72'8","reported_seq":259,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.874764+0000","last_change":"2024-04-14T15:13:28.103421+0000","last_active":"2024-04-14T15:13:38.874764+0000","last_peered":"2024-04-14T15:13:38.874764+0000","last_clean":"2024-04-14T15:13:38.874764+0000","last_became_active":"2024-04-14T15:09:47.048216+0000","last_became_peered":"2024-04-14T15:09:47.048216+0000","last_unstale":"2024-04-14T15:13:38.874764+0000","last_undegraded":"2024-04-14T15:13:38.874764+0000","last_fullsized":"2024-04-14T15:13:38.874764+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:13:28.103336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:13:28.103336+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:44.872504+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"72'1","reported_seq":189,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.848550+0000","last_change":"2024-04-14T15:12:29.041410+0000","last_active":"2024-04-14T15:13:38.848550+0000","last_peered":"2024-04-14T15:13:38.848550+0000","last_clean":"2024-04-14T15:13:38.848550+0000","last_became_active":"2024-04-14T15:09:49.062051+0000","last_became_peered":"2024-04-14T15:09:49.062051+0000","last_unstale":"2024-04-14T15:13:38.848550+0000","last_undegraded":"2024-04-14T15:13:38.848550+0000","last_fullsized":"2024-04-14T15:13:38.848550+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-04-14T15:12:29.041314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:12:29.041314+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:51.150317+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.17","version":"109'5","reported_seq":102,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.828370+0000","last_change":"2024-04-14T15:12:56.397560+0000","last_active":"2024-04-14T15:13:38.828370+0000","last_peered":"2024-04-14T15:13:38.828370+0000","last_clean":"2024-04-14T15:13:38.828370+0000","last_became_active":"2024-04-14T15:12:56.393400+0000","last_became_peered":"2024-04-14T15:12:56.393400+0000","last_unstale":"2024-04-14T15:13:38.828370+0000","last_undegraded":"2024-04-14T15:13:38.828370+0000","last_fullsized":"2024-04-14T15:13:38.828370+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:18.713249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.a","version":"108'1","reported_seq":98,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.849605+0000","last_change":"2024-04-14T15:12:56.397225+0000","last_active":"2024-04-14T15:13:38.849605+0000","last_peered":"2024-04-14T15:13:38.849605+0000","last_clean":"2024-04-14T15:13:38.849605+0000","last_became_active":"2024-04-14T15:12:56.388044+0000","last_became_peered":"2024-04-14T15:12:56.388044+0000","last_unstale":"2024-04-14T15:13:38.849605+0000","last_undegraded":"2024-04-14T15:13:38.849605+0000","last_fullsized":"2024-04-14T15:13:38.849605+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:04.086604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":97,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.886115+0000","last_change":"2024-04-14T15:12:56.548050+0000","last_active":"2024-04-14T15:13:38.886115+0000","last_peered":"2024-04-14T15:13:38.886115+0000","last_clean":"2024-04-14T15:13:38.886115+0000","last_became_active":"2024-04-14T15:12:56.543793+0000","last_became_peered":"2024-04-14T15:12:56.543793+0000","last_unstale":"2024-04-14T15:13:38.886115+0000","last_undegraded":"2024-04-14T15:13:38.886115+0000","last_fullsized":"2024-04-14T15:13:38.886115+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:24.008675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"72'11","reported_seq":233,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.874848+0000","last_change":"2024-04-14T15:13:19.130181+0000","last_active":"2024-04-14T15:13:38.874848+0000","last_peered":"2024-04-14T15:13:38.874848+0000","last_clean":"2024-04-14T15:13:38.874848+0000","last_became_active":"2024-04-14T15:10:07.764183+0000","last_became_peered":"2024-04-14T15:10:07.764183+0000","last_unstale":"2024-04-14T15:13:38.874848+0000","last_undegraded":"2024-04-14T15:13:38.874848+0000","last_fullsized":"2024-04-14T15:13:38.874848+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-04-14T15:13:19.130044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:13:19.130044+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:35.463752+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"65'2","reported_seq":135,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.873223+0000","last_change":"2024-04-14T15:12:26.105766+0000","last_active":"2024-04-14T15:13:38.873223+0000","last_peered":"2024-04-14T15:13:38.873223+0000","last_clean":"2024-04-14T15:13:38.873223+0000","last_became_active":"2024-04-14T15:11:57.118202+0000","last_became_peered":"2024-04-14T15:11:57.118202+0000","last_unstale":"2024-04-14T15:13:38.873223+0000","last_undegraded":"2024-04-14T15:13:38.873223+0000","last_fullsized":"2024-04-14T15:13:38.873223+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-04-14T15:12:26.105658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:12:26.105658+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:41.346966+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":192,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.887961+0000","last_change":"2024-04-14T15:13:28.464940+0000","last_active":"2024-04-14T15:13:38.887961+0000","last_peered":"2024-04-14T15:13:38.887961+0000","last_clean":"2024-04-14T15:13:38.887961+0000","last_became_active":"2024-04-14T15:09:47.045730+0000","last_became_peered":"2024-04-14T15:09:47.045730+0000","last_unstale":"2024-04-14T15:13:38.887961+0000","last_undegraded":"2024-04-14T15:13:38.887961+0000","last_fullsized":"2024-04-14T15:13:38.887961+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:28.464811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:13:28.464811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:38.103587+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":180,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.842341+0000","last_change":"2024-04-14T15:11:21.247809+0000","last_active":"2024-04-14T15:13:38.842341+0000","last_peered":"2024-04-14T15:13:38.842341+0000","last_clean":"2024-04-14T15:13:38.842341+0000","last_became_active":"2024-04-14T15:09:49.057545+0000","last_became_peered":"2024-04-14T15:09:49.057545+0000","last_unstale":"2024-04-14T15:13:38.842341+0000","last_undegraded":"2024-04-14T15:13:38.842341+0000","last_fullsized":"2024-04-14T15:13:38.842341+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:21.247695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:21.247695+0000","last_clean_scrub_stamp":"2024-04-14T15:11:21.247695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"109'5","reported_seq":102,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.839797+0000","last_change":"2024-04-14T15:12:56.494071+0000","last_active":"2024-04-14T15:13:38.839797+0000","last_peered":"2024-04-14T15:13:38.839797+0000","last_clean":"2024-04-14T15:13:38.839797+0000","last_became_active":"2024-04-14T15:12:56.472861+0000","last_became_peered":"2024-04-14T15:12:56.472861+0000","last_unstale":"2024-04-14T15:13:38.839797+0000","last_undegraded":"2024-04-14T15:13:38.839797+0000","last_fullsized":"2024-04-14T15:13:38.839797+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:02.739302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":97,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.840365+0000","last_change":"2024-04-14T15:12:56.367623+0000","last_active":"2024-04-14T15:13:38.840365+0000","last_peered":"2024-04-14T15:13:38.840365+0000","last_clean":"2024-04-14T15:13:38.840365+0000","last_became_active":"2024-04-14T15:12:56.366632+0000","last_became_peered":"2024-04-14T15:12:56.366632+0000","last_unstale":"2024-04-14T15:13:38.840365+0000","last_undegraded":"2024-04-14T15:13:38.840365+0000","last_fullsized":"2024-04-14T15:13:38.840365+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:10.249889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":97,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.829974+0000","last_change":"2024-04-14T15:12:56.407322+0000","last_active":"2024-04-14T15:13:38.829974+0000","last_peered":"2024-04-14T15:13:38.829974+0000","last_clean":"2024-04-14T15:13:38.829974+0000","last_became_active":"2024-04-14T15:12:56.398379+0000","last_became_peered":"2024-04-14T15:12:56.398379+0000","last_unstale":"2024-04-14T15:13:38.829974+0000","last_undegraded":"2024-04-14T15:13:38.829974+0000","last_fullsized":"2024-04-14T15:13:38.829974+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:11.149569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"72'15","reported_seq":221,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.859221+0000","last_change":"2024-04-14T15:13:29.335174+0000","last_active":"2024-04-14T15:13:38.859221+0000","last_peered":"2024-04-14T15:13:38.859221+0000","last_clean":"2024-04-14T15:13:38.859221+0000","last_became_active":"2024-04-14T15:09:45.032887+0000","last_became_peered":"2024-04-14T15:09:45.032887+0000","last_unstale":"2024-04-14T15:13:38.859221+0000","last_undegraded":"2024-04-14T15:13:38.859221+0000","last_fullsized":"2024-04-14T15:13:38.859221+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-04-14T15:13:29.335026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:13:29.335026+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:46.261973+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":203,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.872245+0000","last_change":"2024-04-14T15:13:15.102843+0000","last_active":"2024-04-14T15:13:38.872245+0000","last_peered":"2024-04-14T15:13:38.872245+0000","last_clean":"2024-04-14T15:13:38.872245+0000","last_became_active":"2024-04-14T15:09:43.021494+0000","last_became_peered":"2024-04-14T15:09:43.021494+0000","last_unstale":"2024-04-14T15:13:38.872245+0000","last_undegraded":"2024-04-14T15:13:38.872245+0000","last_fullsized":"2024-04-14T15:13:38.872245+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:15.102763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:13:15.102763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:42.170242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":192,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.842866+0000","last_change":"2024-04-14T15:12:32.347056+0000","last_active":"2024-04-14T15:13:38.842866+0000","last_peered":"2024-04-14T15:13:38.842866+0000","last_clean":"2024-04-14T15:13:38.842866+0000","last_became_active":"2024-04-14T15:09:47.042733+0000","last_became_peered":"2024-04-14T15:09:47.042733+0000","last_unstale":"2024-04-14T15:13:38.842866+0000","last_undegraded":"2024-04-14T15:13:38.842866+0000","last_fullsized":"2024-04-14T15:13:38.842866+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:32.346921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:12:32.346921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":188,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.845079+0000","last_change":"2024-04-14T15:13:12.610073+0000","last_active":"2024-04-14T15:13:38.845079+0000","last_peered":"2024-04-14T15:13:38.845079+0000","last_clean":"2024-04-14T15:13:38.845079+0000","last_became_active":"2024-04-14T15:09:49.127979+0000","last_became_peered":"2024-04-14T15:09:49.127979+0000","last_unstale":"2024-04-14T15:13:38.845079+0000","last_undegraded":"2024-04-14T15:13:38.845079+0000","last_fullsized":"2024-04-14T15:13:38.845079+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:12.609958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:13:12.609958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:19.729963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":97,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.849739+0000","last_change":"2024-04-14T15:12:56.372771+0000","last_active":"2024-04-14T15:13:38.849739+0000","last_peered":"2024-04-14T15:13:38.849739+0000","last_clean":"2024-04-14T15:13:38.849739+0000","last_became_active":"2024-04-14T15:12:56.372456+0000","last_became_peered":"2024-04-14T15:12:56.372456+0000","last_unstale":"2024-04-14T15:13:38.849739+0000","last_undegraded":"2024-04-14T15:13:38.849739+0000","last_fullsized":"2024-04-14T15:13:38.849739+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:11.582238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":115,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.887789+0000","last_change":"2024-04-14T15:13:30.509411+0000","last_active":"2024-04-14T15:13:38.887789+0000","last_peered":"2024-04-14T15:13:38.887789+0000","last_clean":"2024-04-14T15:13:38.887789+0000","last_became_active":"2024-04-14T15:12:56.368566+0000","last_became_peered":"2024-04-14T15:12:56.368566+0000","last_unstale":"2024-04-14T15:13:38.887789+0000","last_undegraded":"2024-04-14T15:13:38.887789+0000","last_fullsized":"2024-04-14T15:13:38.887789+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:30.509291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:30.509291+0000","last_clean_scrub_stamp":"2024-04-14T15:13:30.509291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:57.120611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":97,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.837911+0000","last_change":"2024-04-14T15:12:56.392612+0000","last_active":"2024-04-14T15:13:38.837911+0000","last_peered":"2024-04-14T15:13:38.837911+0000","last_clean":"2024-04-14T15:13:38.837911+0000","last_became_active":"2024-04-14T15:12:56.392352+0000","last_became_peered":"2024-04-14T15:12:56.392352+0000","last_unstale":"2024-04-14T15:13:38.837911+0000","last_undegraded":"2024-04-14T15:13:38.837911+0000","last_fullsized":"2024-04-14T15:13:38.837911+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:04.806971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"72'12","reported_seq":168,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.843141+0000","last_change":"2024-04-14T15:13:18.392665+0000","last_active":"2024-04-14T15:13:38.843141+0000","last_peered":"2024-04-14T15:13:38.843141+0000","last_clean":"2024-04-14T15:13:38.843141+0000","last_became_active":"2024-04-14T15:11:57.118631+0000","last_became_peered":"2024-04-14T15:11:57.118631+0000","last_unstale":"2024-04-14T15:13:38.843141+0000","last_undegraded":"2024-04-14T15:13:38.843141+0000","last_fullsized":"2024-04-14T15:13:38.843141+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'12","last_scrub_stamp":"2024-04-14T15:13:18.392496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:13:18.392496+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:44.659931+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":126,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.838253+0000","last_change":"2024-04-14T15:13:34.177897+0000","last_active":"2024-04-14T15:13:38.838253+0000","last_peered":"2024-04-14T15:13:38.838253+0000","last_clean":"2024-04-14T15:13:38.838253+0000","last_became_active":"2024-04-14T15:11:57.117958+0000","last_became_peered":"2024-04-14T15:11:57.117958+0000","last_unstale":"2024-04-14T15:13:38.838253+0000","last_undegraded":"2024-04-14T15:13:38.838253+0000","last_fullsized":"2024-04-14T15:13:38.838253+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:34.177820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:13:34.177820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:52.828992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":192,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.842911+0000","last_change":"2024-04-14T15:12:39.489602+0000","last_active":"2024-04-14T15:13:38.842911+0000","last_peered":"2024-04-14T15:13:38.842911+0000","last_clean":"2024-04-14T15:13:38.842911+0000","last_became_active":"2024-04-14T15:09:47.048045+0000","last_became_peered":"2024-04-14T15:09:47.048045+0000","last_unstale":"2024-04-14T15:13:38.842911+0000","last_undegraded":"2024-04-14T15:13:38.842911+0000","last_fullsized":"2024-04-14T15:13:38.842911+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:39.489493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:12:39.489493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:04.760526+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":127,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.874986+0000","last_change":"2024-04-14T15:13:22.168812+0000","last_active":"2024-04-14T15:13:38.874986+0000","last_peered":"2024-04-14T15:13:38.874986+0000","last_clean":"2024-04-14T15:13:38.874986+0000","last_became_active":"2024-04-14T15:11:57.118128+0000","last_became_peered":"2024-04-14T15:11:57.118128+0000","last_unstale":"2024-04-14T15:13:38.874986+0000","last_undegraded":"2024-04-14T15:13:38.874986+0000","last_fullsized":"2024-04-14T15:13:38.874986+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:22.168737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:13:22.168737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:44.182086+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"108'2","reported_seq":99,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.892655+0000","last_change":"2024-04-14T15:12:56.419171+0000","last_active":"2024-04-14T15:13:38.892655+0000","last_peered":"2024-04-14T15:13:38.892655+0000","last_clean":"2024-04-14T15:13:38.892655+0000","last_became_active":"2024-04-14T15:12:56.418876+0000","last_became_peered":"2024-04-14T15:12:56.418876+0000","last_unstale":"2024-04-14T15:13:38.892655+0000","last_undegraded":"2024-04-14T15:13:38.892655+0000","last_fullsized":"2024-04-14T15:13:38.892655+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:02.550570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":115,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.825836+0000","last_change":"2024-04-14T15:13:30.306425+0000","last_active":"2024-04-14T15:13:38.825836+0000","last_peered":"2024-04-14T15:13:38.825836+0000","last_clean":"2024-04-14T15:13:38.825836+0000","last_became_active":"2024-04-14T15:12:56.284639+0000","last_became_peered":"2024-04-14T15:12:56.284639+0000","last_unstale":"2024-04-14T15:13:38.825836+0000","last_undegraded":"2024-04-14T15:13:38.825836+0000","last_fullsized":"2024-04-14T15:13:38.825836+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:30.306332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:30.306332+0000","last_clean_scrub_stamp":"2024-04-14T15:13:30.306332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:54.172014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":97,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.860971+0000","last_change":"2024-04-14T15:12:56.388503+0000","last_active":"2024-04-14T15:13:38.860971+0000","last_peered":"2024-04-14T15:13:38.860971+0000","last_clean":"2024-04-14T15:13:38.860971+0000","last_became_active":"2024-04-14T15:12:56.375278+0000","last_became_peered":"2024-04-14T15:12:56.375278+0000","last_unstale":"2024-04-14T15:13:38.860971+0000","last_undegraded":"2024-04-14T15:13:38.860971+0000","last_fullsized":"2024-04-14T15:13:38.860971+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:58.514674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"72'9","reported_seq":212,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.842016+0000","last_change":"2024-04-14T15:13:19.550185+0000","last_active":"2024-04-14T15:13:38.842016+0000","last_peered":"2024-04-14T15:13:38.842016+0000","last_clean":"2024-04-14T15:13:38.842016+0000","last_became_active":"2024-04-14T15:09:45.030093+0000","last_became_peered":"2024-04-14T15:09:45.030093+0000","last_unstale":"2024-04-14T15:13:38.842016+0000","last_undegraded":"2024-04-14T15:13:38.842016+0000","last_fullsized":"2024-04-14T15:13:38.842016+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-04-14T15:13:19.550061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:13:19.550061+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:29.082439+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":170,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.843036+0000","last_change":"2024-04-14T15:11:45.791911+0000","last_active":"2024-04-14T15:13:38.843036+0000","last_peered":"2024-04-14T15:13:38.843036+0000","last_clean":"2024-04-14T15:13:38.843036+0000","last_became_active":"2024-04-14T15:10:07.764086+0000","last_became_peered":"2024-04-14T15:10:07.764086+0000","last_unstale":"2024-04-14T15:13:38.843036+0000","last_undegraded":"2024-04-14T15:13:38.843036+0000","last_fullsized":"2024-04-14T15:13:38.843036+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:45.791810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:45.791810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":192,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.828562+0000","last_change":"2024-04-14T15:12:38.748661+0000","last_active":"2024-04-14T15:13:38.828562+0000","last_peered":"2024-04-14T15:13:38.828562+0000","last_clean":"2024-04-14T15:13:38.828562+0000","last_became_active":"2024-04-14T15:09:47.042384+0000","last_became_peered":"2024-04-14T15:09:47.042384+0000","last_unstale":"2024-04-14T15:13:38.828562+0000","last_undegraded":"2024-04-14T15:13:38.828562+0000","last_fullsized":"2024-04-14T15:13:38.828562+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:38.748520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:12:38.748520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:40.190249+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":188,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.887645+0000","last_change":"2024-04-14T15:12:26.053933+0000","last_active":"2024-04-14T15:13:38.887645+0000","last_peered":"2024-04-14T15:13:38.887645+0000","last_clean":"2024-04-14T15:13:38.887645+0000","last_became_active":"2024-04-14T15:09:49.061756+0000","last_became_peered":"2024-04-14T15:09:49.061756+0000","last_unstale":"2024-04-14T15:13:38.887645+0000","last_undegraded":"2024-04-14T15:13:38.887645+0000","last_fullsized":"2024-04-14T15:13:38.887645+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:26.053821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:12:26.053821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.13","version":"109'9","reported_seq":106,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.837930+0000","last_change":"2024-04-14T15:12:56.395805+0000","last_active":"2024-04-14T15:13:38.837930+0000","last_peered":"2024-04-14T15:13:38.837930+0000","last_clean":"2024-04-14T15:13:38.837930+0000","last_became_active":"2024-04-14T15:12:56.395510+0000","last_became_peered":"2024-04-14T15:12:56.395510+0000","last_unstale":"2024-04-14T15:13:38.837930+0000","last_undegraded":"2024-04-14T15:13:38.837930+0000","last_fullsized":"2024-04-14T15:13:38.837930+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:16.924678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.e","version":"142'3","reported_seq":111,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.859096+0000","last_change":"2024-04-14T15:12:56.459419+0000","last_active":"2024-04-14T15:13:38.859096+0000","last_peered":"2024-04-14T15:13:38.859096+0000","last_clean":"2024-04-14T15:13:38.859096+0000","last_became_active":"2024-04-14T15:12:56.451958+0000","last_became_peered":"2024-04-14T15:12:56.451958+0000","last_unstale":"2024-04-14T15:13:38.859096+0000","last_undegraded":"2024-04-14T15:13:38.859096+0000","last_fullsized":"2024-04-14T15:13:38.859096+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:18.857251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497492,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":24577,"num_write":3,"num_write_kb":73731,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":97,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.873359+0000","last_change":"2024-04-14T15:12:56.233385+0000","last_active":"2024-04-14T15:13:38.873359+0000","last_peered":"2024-04-14T15:13:38.873359+0000","last_clean":"2024-04-14T15:13:38.873359+0000","last_became_active":"2024-04-14T15:12:56.232831+0000","last_became_peered":"2024-04-14T15:12:56.232831+0000","last_unstale":"2024-04-14T15:13:38.873359+0000","last_undegraded":"2024-04-14T15:13:38.873359+0000","last_fullsized":"2024-04-14T15:13:38.873359+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:59.183825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1b","version":"87'7","reported_seq":208,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.837875+0000","last_change":"2024-04-14T15:13:14.599768+0000","last_active":"2024-04-14T15:13:38.837875+0000","last_peered":"2024-04-14T15:13:38.837875+0000","last_clean":"2024-04-14T15:13:38.837875+0000","last_became_active":"2024-04-14T15:09:45.028667+0000","last_became_peered":"2024-04-14T15:09:45.028667+0000","last_unstale":"2024-04-14T15:13:38.837875+0000","last_undegraded":"2024-04-14T15:13:38.837875+0000","last_fullsized":"2024-04-14T15:13:38.837875+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'7","last_scrub_stamp":"2024-04-14T15:13:14.599663+0000","last_deep_scrub":"87'7","last_deep_scrub_stamp":"2024-04-14T15:11:10.510785+0000","last_clean_scrub_stamp":"2024-04-14T15:13:14.599663+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:31.959592+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":200,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.846161+0000","last_change":"2024-04-14T15:12:28.044001+0000","last_active":"2024-04-14T15:13:38.846161+0000","last_peered":"2024-04-14T15:13:38.846161+0000","last_clean":"2024-04-14T15:13:38.846161+0000","last_became_active":"2024-04-14T15:09:43.021986+0000","last_became_peered":"2024-04-14T15:09:43.021986+0000","last_unstale":"2024-04-14T15:13:38.846161+0000","last_undegraded":"2024-04-14T15:13:38.846161+0000","last_fullsized":"2024-04-14T15:13:38.846161+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:28.043899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:10:43.084820+0000","last_clean_scrub_stamp":"2024-04-14T15:12:28.043899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":192,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.838667+0000","last_change":"2024-04-14T15:13:19.196559+0000","last_active":"2024-04-14T15:13:38.838667+0000","last_peered":"2024-04-14T15:13:38.838667+0000","last_clean":"2024-04-14T15:13:38.838667+0000","last_became_active":"2024-04-14T15:09:47.046517+0000","last_became_peered":"2024-04-14T15:09:47.046517+0000","last_unstale":"2024-04-14T15:13:38.838667+0000","last_undegraded":"2024-04-14T15:13:38.838667+0000","last_fullsized":"2024-04-14T15:13:38.838667+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:19.196454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:13:19.196454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:44.555952+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":188,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.826476+0000","last_change":"2024-04-14T15:12:37.782377+0000","last_active":"2024-04-14T15:13:38.826476+0000","last_peered":"2024-04-14T15:13:38.826476+0000","last_clean":"2024-04-14T15:13:38.826476+0000","last_became_active":"2024-04-14T15:09:49.069590+0000","last_became_peered":"2024-04-14T15:09:49.069590+0000","last_unstale":"2024-04-14T15:13:38.826476+0000","last_undegraded":"2024-04-14T15:13:38.826476+0000","last_fullsized":"2024-04-14T15:13:38.826476+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:37.782277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:12:37.782277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:57.975901+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":97,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.883360+0000","last_change":"2024-04-14T15:12:56.361131+0000","last_active":"2024-04-14T15:13:38.883360+0000","last_peered":"2024-04-14T15:13:38.883360+0000","last_clean":"2024-04-14T15:13:38.883360+0000","last_became_active":"2024-04-14T15:12:56.360314+0000","last_became_peered":"2024-04-14T15:12:56.360314+0000","last_unstale":"2024-04-14T15:13:38.883360+0000","last_undegraded":"2024-04-14T15:13:38.883360+0000","last_fullsized":"2024-04-14T15:13:38.883360+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:18.166639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":97,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.890984+0000","last_change":"2024-04-14T15:12:56.467201+0000","last_active":"2024-04-14T15:13:38.890984+0000","last_peered":"2024-04-14T15:13:38.890984+0000","last_clean":"2024-04-14T15:13:38.890984+0000","last_became_active":"2024-04-14T15:12:56.466934+0000","last_became_peered":"2024-04-14T15:12:56.466934+0000","last_unstale":"2024-04-14T15:13:38.890984+0000","last_undegraded":"2024-04-14T15:13:38.890984+0000","last_fullsized":"2024-04-14T15:13:38.890984+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:01.475377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":97,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.839648+0000","last_change":"2024-04-14T15:12:56.190716+0000","last_active":"2024-04-14T15:13:38.839648+0000","last_peered":"2024-04-14T15:13:38.839648+0000","last_clean":"2024-04-14T15:13:38.839648+0000","last_became_active":"2024-04-14T15:12:56.189117+0000","last_became_peered":"2024-04-14T15:12:56.189117+0000","last_unstale":"2024-04-14T15:13:38.839648+0000","last_undegraded":"2024-04-14T15:13:38.839648+0000","last_fullsized":"2024-04-14T15:13:38.839648+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:17.897219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.18","version":"72'9","reported_seq":212,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.846174+0000","last_change":"2024-04-14T15:13:24.538800+0000","last_active":"2024-04-14T15:13:38.846174+0000","last_peered":"2024-04-14T15:13:38.846174+0000","last_clean":"2024-04-14T15:13:38.846174+0000","last_became_active":"2024-04-14T15:09:45.032079+0000","last_became_peered":"2024-04-14T15:09:45.032079+0000","last_unstale":"2024-04-14T15:13:38.846174+0000","last_undegraded":"2024-04-14T15:13:38.846174+0000","last_fullsized":"2024-04-14T15:13:38.846174+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-04-14T15:13:24.538684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:13:24.538684+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:48.098124+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":118,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.886088+0000","last_change":"2024-04-14T15:11:57.122422+0000","last_active":"2024-04-14T15:13:38.886088+0000","last_peered":"2024-04-14T15:13:38.886088+0000","last_clean":"2024-04-14T15:13:38.886088+0000","last_became_active":"2024-04-14T15:11:57.122120+0000","last_became_peered":"2024-04-14T15:11:57.122120+0000","last_unstale":"2024-04-14T15:13:38.886088+0000","last_undegraded":"2024-04-14T15:13:38.886088+0000","last_fullsized":"2024-04-14T15:13:38.886088+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:43.589625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:43.589625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":192,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.826140+0000","last_change":"2024-04-14T15:13:25.370811+0000","last_active":"2024-04-14T15:13:38.826140+0000","last_peered":"2024-04-14T15:13:38.826140+0000","last_clean":"2024-04-14T15:13:38.826140+0000","last_became_active":"2024-04-14T15:09:47.041486+0000","last_became_peered":"2024-04-14T15:09:47.041486+0000","last_unstale":"2024-04-14T15:13:38.826140+0000","last_undegraded":"2024-04-14T15:13:38.826140+0000","last_fullsized":"2024-04-14T15:13:38.826140+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:25.370679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:13:25.370679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:54.308548+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":180,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.841914+0000","last_change":"2024-04-14T15:12:37.453882+0000","last_active":"2024-04-14T15:13:38.841914+0000","last_peered":"2024-04-14T15:13:38.841914+0000","last_clean":"2024-04-14T15:13:38.841914+0000","last_became_active":"2024-04-14T15:09:49.058904+0000","last_became_peered":"2024-04-14T15:09:49.058904+0000","last_unstale":"2024-04-14T15:13:38.841914+0000","last_undegraded":"2024-04-14T15:13:38.841914+0000","last_fullsized":"2024-04-14T15:13:38.841914+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:37.453778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:12:37.453778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:58.162338+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.11","version":"109'7","reported_seq":104,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.826061+0000","last_change":"2024-04-14T15:12:56.227471+0000","last_active":"2024-04-14T15:13:38.826061+0000","last_peered":"2024-04-14T15:13:38.826061+0000","last_clean":"2024-04-14T15:13:38.826061+0000","last_became_active":"2024-04-14T15:12:56.227047+0000","last_became_peered":"2024-04-14T15:12:56.227047+0000","last_unstale":"2024-04-14T15:13:38.826061+0000","last_undegraded":"2024-04-14T15:13:38.826061+0000","last_fullsized":"2024-04-14T15:13:38.826061+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:12.039101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.c","version":"141'5","reported_seq":106,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.826928+0000","last_change":"2024-04-14T15:12:56.396530+0000","last_active":"2024-04-14T15:13:38.826928+0000","last_peered":"2024-04-14T15:13:38.826928+0000","last_clean":"2024-04-14T15:13:38.826928+0000","last_became_active":"2024-04-14T15:12:56.396129+0000","last_became_peered":"2024-04-14T15:12:56.396129+0000","last_unstale":"2024-04-14T15:13:38.826928+0000","last_undegraded":"2024-04-14T15:13:38.826928+0000","last_fullsized":"2024-04-14T15:13:38.826928+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:03.476973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25168635,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":27329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":97,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.831982+0000","last_change":"2024-04-14T15:12:56.422596+0000","last_active":"2024-04-14T15:13:38.831982+0000","last_peered":"2024-04-14T15:13:38.831982+0000","last_clean":"2024-04-14T15:13:38.831982+0000","last_became_active":"2024-04-14T15:12:56.415471+0000","last_became_peered":"2024-04-14T15:12:56.415471+0000","last_unstale":"2024-04-14T15:13:38.831982+0000","last_undegraded":"2024-04-14T15:13:38.831982+0000","last_fullsized":"2024-04-14T15:13:38.831982+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:06.099234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"72'15","reported_seq":170,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.848423+0000","last_change":"2024-04-14T15:13:31.248407+0000","last_active":"2024-04-14T15:13:38.848423+0000","last_peered":"2024-04-14T15:13:38.848423+0000","last_clean":"2024-04-14T15:13:38.848423+0000","last_became_active":"2024-04-14T15:11:57.120945+0000","last_became_peered":"2024-04-14T15:11:57.120945+0000","last_unstale":"2024-04-14T15:13:38.848423+0000","last_undegraded":"2024-04-14T15:13:38.848423+0000","last_fullsized":"2024-04-14T15:13:38.848423+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-04-14T15:13:31.248315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:13:31.248315+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:48.969632+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":200,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.892401+0000","last_change":"2024-04-14T15:13:18.431688+0000","last_active":"2024-04-14T15:13:38.892401+0000","last_peered":"2024-04-14T15:13:38.892401+0000","last_clean":"2024-04-14T15:13:38.892401+0000","last_became_active":"2024-04-14T15:09:43.017351+0000","last_became_peered":"2024-04-14T15:09:43.017351+0000","last_unstale":"2024-04-14T15:13:38.892401+0000","last_undegraded":"2024-04-14T15:13:38.892401+0000","last_fullsized":"2024-04-14T15:13:38.892401+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:18.431571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:13:18.431571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:26.379954+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":184,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.847779+0000","last_change":"2024-04-14T15:12:20.610268+0000","last_active":"2024-04-14T15:13:38.847779+0000","last_peered":"2024-04-14T15:13:38.847779+0000","last_clean":"2024-04-14T15:13:38.847779+0000","last_became_active":"2024-04-14T15:09:47.042538+0000","last_became_peered":"2024-04-14T15:09:47.042538+0000","last_unstale":"2024-04-14T15:13:38.847779+0000","last_undegraded":"2024-04-14T15:13:38.847779+0000","last_fullsized":"2024-04-14T15:13:38.847779+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:20.610159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:12:20.610159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:42.966823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":188,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.832993+0000","last_change":"2024-04-14T15:13:18.143802+0000","last_active":"2024-04-14T15:13:38.832993+0000","last_peered":"2024-04-14T15:13:38.832993+0000","last_clean":"2024-04-14T15:13:38.832993+0000","last_became_active":"2024-04-14T15:09:49.129636+0000","last_became_peered":"2024-04-14T15:09:49.129636+0000","last_unstale":"2024-04-14T15:13:38.832993+0000","last_undegraded":"2024-04-14T15:13:38.832993+0000","last_fullsized":"2024-04-14T15:13:38.832993+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:18.143666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:13:18.143666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:37.337036+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":97,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.834604+0000","last_change":"2024-04-14T15:12:56.392284+0000","last_active":"2024-04-14T15:13:38.834604+0000","last_peered":"2024-04-14T15:13:38.834604+0000","last_clean":"2024-04-14T15:13:38.834604+0000","last_became_active":"2024-04-14T15:12:56.390819+0000","last_became_peered":"2024-04-14T15:12:56.390819+0000","last_unstale":"2024-04-14T15:13:38.834604+0000","last_undegraded":"2024-04-14T15:13:38.834604+0000","last_fullsized":"2024-04-14T15:13:38.834604+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:14.835938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.d","version":"108'1","reported_seq":98,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-14T15:13:38.874691+0000","last_change":"2024-04-14T15:12:56.396639+0000","last_active":"2024-04-14T15:13:38.874691+0000","last_peered":"2024-04-14T15:13:38.874691+0000","last_clean":"2024-04-14T15:13:38.874691+0000","last_became_active":"2024-04-14T15:12:56.393878+0000","last_became_peered":"2024-04-14T15:12:56.393878+0000","last_unstale":"2024-04-14T15:13:38.874691+0000","last_undegraded":"2024-04-14T15:13:38.874691+0000","last_fullsized":"2024-04-14T15:13:38.874691+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:07.801168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":176,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":175,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":5},{"poolid":172,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":112,"num_pg":32,"stat_sum":{"num_bytes":2048,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":11,"num_read_kb":4,"num_write":28,"num_write_kb":1049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":6144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45,"ondisk_log_size":45,"up":96,"acting":96,"num_store_stats":8},{"poolid":110,"num_pg":32,"stat_sum":{"num_bytes":2807,"num_objects":285,"num_object_clones":180,"num_object_copies":855,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":285,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":290,"num_write_kb":289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3502080,"data_stored":8421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":482,"ondisk_log_size":482,"up":96,"acting":96,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":226499262,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":1011,"num_read_kb":49161,"num_write":1047,"num_write_kb":226712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":679809024,"data_stored":679497786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1066,"ondisk_log_size":1066,"up":96,"acting":96,"num_store_stats":8},{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":2048,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100,"num_write_kb":58,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":6144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":96,"acting":96,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":229,"num_read_kb":224,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":693,"num_read_kb":450,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":421,"ondisk_log_size":421,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67,"ondisk_log_size":67,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103221,"num_pgs":124,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":38516,"kb_used_data":3180,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93706636,"statfs":{"total":95995035648,"available":95955595264,"internally_reserved":0,"allocated":3256320,"data_stored":1797306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397560,"num_pgs":170,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":159152,"kb_used_data":101600,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93586000,"statfs":{"total":95995035648,"available":95832064000,"internally_reserved":0,"allocated":104038400,"data_stored":102471529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574930,"num_pgs":180,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":147996,"kb_used_data":76556,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93597156,"statfs":{"total":95995035648,"available":95843487744,"internally_reserved":0,"allocated":78393344,"data_stored":76706995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738455,"num_pgs":148,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":224532,"kb_used_data":175292,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93520620,"statfs":{"total":95995035648,"available":95765114880,"internally_reserved":0,"allocated":179499008,"data_stored":177958995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411406,"num_pgs":142,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":174868,"kb_used_data":125636,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93570284,"statfs":{"total":95995035648,"available":95815970816,"internally_reserved":0,"allocated":128651264,"data_stored":127033811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182472,"num_pgs":155,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":29648,"kb_used_data":2624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93715504,"statfs":{"total":95995035648,"available":95964676096,"internally_reserved":0,"allocated":2686976,"data_stored":1211475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986242,"num_pgs":136,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":172392,"kb_used_data":100952,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93572760,"statfs":{"total":95995035648,"available":95818506240,"internally_reserved":0,"allocated":103374848,"data_stored":101871543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659199,"num_pgs":167,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":158500,"kb_used_data":100948,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93586652,"statfs":{"total":95995035648,"available":95832731648,"internally_reserved":0,"allocated":103370752,"data_stored":101872181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":176222208,"data_stored":176163663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":75554816,"data_stored":75500419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":125886464,"data_stored":125829277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":100687872,"data_stored":100666127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":100716544,"data_stored":100667195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":100708352,"data_stored":100664448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":479232,"data_stored":1158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":614400,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":610304,"data_stored":1480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":372736,"data_stored":883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":364544,"data_stored":864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":364544,"data_stored":867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":112,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":112,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":112,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":112,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":112,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":112,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":112,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":112,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":172,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":172,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":172,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":172,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":172,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":172,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":172,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":172,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":175,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":175,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":175,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":175,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":175,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-14T15:13:45.186 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:45.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:45.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:13:45.288 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:13:45.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:45.312 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:45.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:13:45.394 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:45 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "172.0"}]: dispatch 2024-04-14T15:13:45.394 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:45 smithi090 ceph-mon[30399]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "172.0"}]: dispatch 2024-04-14T15:13:45.394 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:45 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "172.1"}]: dispatch 2024-04-14T15:13:45.394 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:45 smithi090 ceph-mon[30399]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "172.1"}]: dispatch 2024-04-14T15:13:45.394 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:45 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "172.2"}]: dispatch 2024-04-14T15:13:45.394 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:45 smithi090 ceph-mon[30399]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "172.2"}]: dispatch 2024-04-14T15:13:45.394 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:45 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "172.3"}]: dispatch 2024-04-14T15:13:45.394 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:45 smithi090 ceph-mon[30399]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "172.3"}]: dispatch 2024-04-14T15:13:45.394 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:45 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "172.4"}]: dispatch 2024-04-14T15:13:45.395 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:45 smithi090 ceph-mon[30399]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "172.4"}]: dispatch 2024-04-14T15:13:45.395 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:45 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "172.5"}]: dispatch 2024-04-14T15:13:45.395 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:45 smithi090 ceph-mon[30399]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "172.5"}]: dispatch 2024-04-14T15:13:45.395 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:45 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "172.6"}]: dispatch 2024-04-14T15:13:45.395 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:45 smithi090 ceph-mon[30399]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "172.6"}]: dispatch 2024-04-14T15:13:45.395 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:45 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "172.7"}]: dispatch 2024-04-14T15:13:45.395 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:45 smithi090 ceph-mon[30399]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "172.7"}]: dispatch 2024-04-14T15:13:45.395 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:45 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "172.8"}]: dispatch 2024-04-14T15:13:45.395 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:45 smithi090 ceph-mon[30399]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "172.8"}]: dispatch 2024-04-14T15:13:45.395 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:45 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "172.9"}]: dispatch 2024-04-14T15:13:45.395 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:45 smithi090 ceph-mon[30399]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "172.9"}]: dispatch 2024-04-14T15:13:45.395 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:45 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:45.395 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:45 smithi090 ceph-mon[30399]: 172.4 scrub starts 2024-04-14T15:13:45.395 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:45 smithi090 ceph-mon[30399]: 172.4 scrub ok 2024-04-14T15:13:45.396 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:45 smithi090 ceph-mon[30399]: osdmap e157: 8 total, 8 up, 8 in 2024-04-14T15:13:45.396 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:45 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:13:45.396 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:45 smithi090 ceph-mon[30399]: 172.9 scrub starts 2024-04-14T15:13:45.396 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:45 smithi090 ceph-mon[30399]: 172.9 scrub ok 2024-04-14T15:13:45.396 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:45 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "172.0"}]: dispatch 2024-04-14T15:13:45.396 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:45 smithi090 ceph-mon[41670]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "172.0"}]: dispatch 2024-04-14T15:13:45.396 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:45 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "172.1"}]: dispatch 2024-04-14T15:13:45.396 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:45 smithi090 ceph-mon[41670]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "172.1"}]: dispatch 2024-04-14T15:13:45.396 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:45 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "172.2"}]: dispatch 2024-04-14T15:13:45.396 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:45 smithi090 ceph-mon[41670]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "172.2"}]: dispatch 2024-04-14T15:13:45.396 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:45 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "172.3"}]: dispatch 2024-04-14T15:13:45.396 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:45 smithi090 ceph-mon[41670]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "172.3"}]: dispatch 2024-04-14T15:13:45.397 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:45 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "172.4"}]: dispatch 2024-04-14T15:13:45.397 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:45 smithi090 ceph-mon[41670]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "172.4"}]: dispatch 2024-04-14T15:13:45.397 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:45 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "172.5"}]: dispatch 2024-04-14T15:13:45.397 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:45 smithi090 ceph-mon[41670]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "172.5"}]: dispatch 2024-04-14T15:13:45.397 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:45 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "172.6"}]: dispatch 2024-04-14T15:13:45.397 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:45 smithi090 ceph-mon[41670]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "172.6"}]: dispatch 2024-04-14T15:13:45.397 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:45 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "172.7"}]: dispatch 2024-04-14T15:13:45.397 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:45 smithi090 ceph-mon[41670]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "172.7"}]: dispatch 2024-04-14T15:13:45.397 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:45 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "172.8"}]: dispatch 2024-04-14T15:13:45.397 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:45 smithi090 ceph-mon[41670]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "172.8"}]: dispatch 2024-04-14T15:13:45.397 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:45 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "172.9"}]: dispatch 2024-04-14T15:13:45.397 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:45 smithi090 ceph-mon[41670]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "172.9"}]: dispatch 2024-04-14T15:13:45.397 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:45 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:45.397 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:45 smithi090 ceph-mon[41670]: 172.4 scrub starts 2024-04-14T15:13:45.397 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:45 smithi090 ceph-mon[41670]: 172.4 scrub ok 2024-04-14T15:13:45.398 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:45 smithi090 ceph-mon[41670]: osdmap e157: 8 total, 8 up, 8 in 2024-04-14T15:13:45.398 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:45 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:13:45.398 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:45 smithi090 ceph-mon[41670]: 172.9 scrub starts 2024-04-14T15:13:45.398 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:45 smithi090 ceph-mon[41670]: 172.9 scrub ok 2024-04-14T15:13:45.413 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:13:45.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:45.438 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:45.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:13:45.539 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:13:45.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:45.564 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:45.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:13:45.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:45 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "172.0"}]: dispatch 2024-04-14T15:13:45.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:45 smithi092 ceph-mon[33966]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "172.0"}]: dispatch 2024-04-14T15:13:45.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:45 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "172.1"}]: dispatch 2024-04-14T15:13:45.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:45 smithi092 ceph-mon[33966]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "172.1"}]: dispatch 2024-04-14T15:13:45.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:45 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "172.2"}]: dispatch 2024-04-14T15:13:45.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:45 smithi092 ceph-mon[33966]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "172.2"}]: dispatch 2024-04-14T15:13:45.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:45 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "172.3"}]: dispatch 2024-04-14T15:13:45.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:45 smithi092 ceph-mon[33966]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "172.3"}]: dispatch 2024-04-14T15:13:45.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:45 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "172.4"}]: dispatch 2024-04-14T15:13:45.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:45 smithi092 ceph-mon[33966]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "172.4"}]: dispatch 2024-04-14T15:13:45.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:45 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "172.5"}]: dispatch 2024-04-14T15:13:45.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:45 smithi092 ceph-mon[33966]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "172.5"}]: dispatch 2024-04-14T15:13:45.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:45 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "172.6"}]: dispatch 2024-04-14T15:13:45.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:45 smithi092 ceph-mon[33966]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "172.6"}]: dispatch 2024-04-14T15:13:45.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:45 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "172.7"}]: dispatch 2024-04-14T15:13:45.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:45 smithi092 ceph-mon[33966]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "172.7"}]: dispatch 2024-04-14T15:13:45.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:45 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "172.8"}]: dispatch 2024-04-14T15:13:45.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:45 smithi092 ceph-mon[33966]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "172.8"}]: dispatch 2024-04-14T15:13:45.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:45 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "172.9"}]: dispatch 2024-04-14T15:13:45.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:45 smithi092 ceph-mon[33966]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "172.9"}]: dispatch 2024-04-14T15:13:45.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:45 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:45.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:45 smithi092 ceph-mon[33966]: 172.4 scrub starts 2024-04-14T15:13:45.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:45 smithi092 ceph-mon[33966]: 172.4 scrub ok 2024-04-14T15:13:45.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:45 smithi092 ceph-mon[33966]: osdmap e157: 8 total, 8 up, 8 in 2024-04-14T15:13:45.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:45 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:13:45.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:45 smithi092 ceph-mon[33966]: 172.9 scrub starts 2024-04-14T15:13:45.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:45 smithi092 ceph-mon[33966]: 172.9 scrub ok 2024-04-14T15:13:45.665 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:13:45.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:45.690 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:45.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:13:45.791 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:13:45.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:45.816 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:45.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:13:45.918 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:13:45.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:45.943 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:45.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:13:46.044 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:13:46.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:46.070 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:46.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:13:46.171 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:13:46.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:46.196 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:46.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:13:46.276 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-14T15:13:46.276 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:13:46.297 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:13:46.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:46.322 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:46.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:13:46.423 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:13:46.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:46.448 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:46.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:13:46.549 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:13:46.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:46.574 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:46.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:13:46.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:46 smithi092 ceph-mon[33966]: 172.7 scrub starts 2024-04-14T15:13:46.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:46 smithi092 ceph-mon[33966]: 172.7 scrub ok 2024-04-14T15:13:46.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:46 smithi092 ceph-mon[33966]: 172.5 scrub starts 2024-04-14T15:13:46.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:46 smithi092 ceph-mon[33966]: 172.5 scrub ok 2024-04-14T15:13:46.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:46 smithi092 ceph-mon[33966]: pgmap v182: 332 pgs: 332 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 61 KiB/s wr, 73 op/s 2024-04-14T15:13:46.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:46 smithi092 ceph-mon[33966]: 172.8 scrub starts 2024-04-14T15:13:46.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:46 smithi092 ceph-mon[33966]: 172.8 scrub ok 2024-04-14T15:13:46.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:46 smithi092 ceph-mon[33966]: 172.0 scrub starts 2024-04-14T15:13:46.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:46 smithi092 ceph-mon[33966]: 172.0 scrub ok 2024-04-14T15:13:46.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:46 smithi092 ceph-mon[33966]: 172.3 scrub starts 2024-04-14T15:13:46.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:46 smithi092 ceph-mon[33966]: 172.3 scrub ok 2024-04-14T15:13:46.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:46 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:46.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:46 smithi092 ceph-mon[33966]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:13:46.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:46 smithi092 ceph-mon[33966]: osdmap e158: 8 total, 8 up, 8 in 2024-04-14T15:13:46.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:46 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3974633471' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi090-93342-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:46.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:46 smithi092 ceph-mon[33966]: from='client.48127 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi090-93342-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:46.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:46 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/11481467' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi090-93363-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:46.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:46 smithi092 ceph-mon[33966]: from='client.48130 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi090-93363-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:46.660 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:46 smithi090 ceph-mon[41670]: 172.7 scrub starts 2024-04-14T15:13:46.660 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:46 smithi090 ceph-mon[41670]: 172.7 scrub ok 2024-04-14T15:13:46.660 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:46 smithi090 ceph-mon[41670]: 172.5 scrub starts 2024-04-14T15:13:46.660 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:46 smithi090 ceph-mon[41670]: 172.5 scrub ok 2024-04-14T15:13:46.661 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:46 smithi090 ceph-mon[41670]: pgmap v182: 332 pgs: 332 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 61 KiB/s wr, 73 op/s 2024-04-14T15:13:46.661 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:46 smithi090 ceph-mon[41670]: 172.8 scrub starts 2024-04-14T15:13:46.661 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:46 smithi090 ceph-mon[41670]: 172.8 scrub ok 2024-04-14T15:13:46.661 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:46 smithi090 ceph-mon[41670]: 172.0 scrub starts 2024-04-14T15:13:46.661 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:46 smithi090 ceph-mon[41670]: 172.0 scrub ok 2024-04-14T15:13:46.661 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:46 smithi090 ceph-mon[41670]: 172.3 scrub starts 2024-04-14T15:13:46.661 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:46 smithi090 ceph-mon[41670]: 172.3 scrub ok 2024-04-14T15:13:46.661 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:46 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:46.661 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:46 smithi090 ceph-mon[41670]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:13:46.661 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:46 smithi090 ceph-mon[41670]: osdmap e158: 8 total, 8 up, 8 in 2024-04-14T15:13:46.661 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:46 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3974633471' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi090-93342-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:46.661 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:46 smithi090 ceph-mon[41670]: from='client.48127 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi090-93342-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:46.661 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:46 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/11481467' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi090-93363-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:46.661 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:46 smithi090 ceph-mon[41670]: from='client.48130 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi090-93363-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:46.662 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:46 smithi090 ceph-mon[30399]: 172.7 scrub starts 2024-04-14T15:13:46.662 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:46 smithi090 ceph-mon[30399]: 172.7 scrub ok 2024-04-14T15:13:46.662 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:46 smithi090 ceph-mon[30399]: 172.5 scrub starts 2024-04-14T15:13:46.662 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:46 smithi090 ceph-mon[30399]: 172.5 scrub ok 2024-04-14T15:13:46.662 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:46 smithi090 ceph-mon[30399]: pgmap v182: 332 pgs: 332 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 61 KiB/s wr, 73 op/s 2024-04-14T15:13:46.662 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:46 smithi090 ceph-mon[30399]: 172.8 scrub starts 2024-04-14T15:13:46.662 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:46 smithi090 ceph-mon[30399]: 172.8 scrub ok 2024-04-14T15:13:46.662 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:46 smithi090 ceph-mon[30399]: 172.0 scrub starts 2024-04-14T15:13:46.662 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:46 smithi090 ceph-mon[30399]: 172.0 scrub ok 2024-04-14T15:13:46.662 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:46 smithi090 ceph-mon[30399]: 172.3 scrub starts 2024-04-14T15:13:46.663 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:46 smithi090 ceph-mon[30399]: 172.3 scrub ok 2024-04-14T15:13:46.663 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:46 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:46.663 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:46 smithi090 ceph-mon[30399]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:13:46.663 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:46 smithi090 ceph-mon[30399]: osdmap e158: 8 total, 8 up, 8 in 2024-04-14T15:13:46.663 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:46 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3974633471' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi090-93342-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:46.663 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:46 smithi090 ceph-mon[30399]: from='client.48127 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi090-93342-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:46.663 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:46 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/11481467' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi090-93363-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:46.663 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:46 smithi090 ceph-mon[30399]: from='client.48130 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi090-93363-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:46.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.440+0000 7fe2dac9e700 1 -- 172.21.15.90:0/2896385282 >> v1:172.21.15.90:6789/0 conn(0x7fe2d410e360 legacy=0x7fe2d4110820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:46.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.440+0000 7fe2dac9e700 1 -- 172.21.15.90:0/2896385282 shutdown_connections 2024-04-14T15:13:46.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.440+0000 7fe2dac9e700 1 -- 172.21.15.90:0/2896385282 >> 172.21.15.90:0/2896385282 conn(0x7fe2d40fd2c0 msgr2=0x7fe2d40ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:13:46.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.440+0000 7fe2dac9e700 1 -- 172.21.15.90:0/2896385282 shutdown_connections 2024-04-14T15:13:46.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.440+0000 7fe2dac9e700 1 -- 172.21.15.90:0/2896385282 wait complete. 2024-04-14T15:13:46.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.440+0000 7fe2dac9e700 1 librados: starting msgr at 2024-04-14T15:13:46.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.440+0000 7fe2dac9e700 1 librados: starting objecter 2024-04-14T15:13:46.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.440+0000 7fe2dac9e700 1 Processor -- start 2024-04-14T15:13:46.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.440+0000 7fe2dac9e700 1 -- start start 2024-04-14T15:13:46.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.440+0000 7fe2dac9e700 1 librados: setting wanted keys 2024-04-14T15:13:46.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.440+0000 7fe2dac9e700 1 librados: calling monclient init 2024-04-14T15:13:46.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.441+0000 7fe2dac9e700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe2d410e160 con 0x7fe2d41078e0 2024-04-14T15:13:46.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.441+0000 7fe2dac9e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe2d41a8fc0 con 0x7fe2d410e360 2024-04-14T15:13:46.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.441+0000 7fe2dac9e700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe2d41aa200 con 0x7fe2d410a790 2024-04-14T15:13:46.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.441+0000 7fe2d8a3a700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fe2d41078e0 0x7fe2d406ae70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:41418/0 (socket says 172.21.15.90:41418) 2024-04-14T15:13:46.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.441+0000 7fe2d8a3a700 1 -- 172.21.15.90:0/250991494 learned_addr learned my addr 172.21.15.90:0/250991494 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:13:46.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.441+0000 7fe2d1ffb700 10 client.?.objecter ms_handle_connect 0x7fe2d41078e0 2024-04-14T15:13:46.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.441+0000 7fe2d1ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:13:46.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.441+0000 7fe2d1ffb700 10 client.?.objecter ms_handle_connect 0x7fe2d410a790 2024-04-14T15:13:46.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.441+0000 7fe2d1ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:13:46.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.441+0000 7fe2d1ffb700 10 client.?.objecter ms_handle_connect 0x7fe2d410e360 2024-04-14T15:13:46.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.441+0000 7fe2d1ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:13:46.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.441+0000 7fe2d1ffb700 1 -- 172.21.15.90:0/250991494 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1995079173 0 0) 0x7fe2d41a8fc0 con 0x7fe2d410e360 2024-04-14T15:13:46.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.441+0000 7fe2d1ffb700 1 -- 172.21.15.90:0/250991494 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe2c40035b0 con 0x7fe2d410e360 2024-04-14T15:13:46.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.441+0000 7fe2d1ffb700 1 -- 172.21.15.90:0/250991494 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3079068740 0 0) 0x7fe2d410e160 con 0x7fe2d41078e0 2024-04-14T15:13:46.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.441+0000 7fe2d1ffb700 1 -- 172.21.15.90:0/250991494 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe2d41a8fc0 con 0x7fe2d41078e0 2024-04-14T15:13:46.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.441+0000 7fe2d1ffb700 1 -- 172.21.15.90:0/250991494 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1142500553 0 0) 0x7fe2d41aa200 con 0x7fe2d410a790 2024-04-14T15:13:46.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.441+0000 7fe2d1ffb700 1 -- 172.21.15.90:0/250991494 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe2d410e160 con 0x7fe2d410a790 2024-04-14T15:13:46.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.441+0000 7fe2d1ffb700 1 -- 172.21.15.90:0/250991494 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1524013736 0 0) 0x7fe2c40035b0 con 0x7fe2d410e360 2024-04-14T15:13:46.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.441+0000 7fe2d1ffb700 1 -- 172.21.15.90:0/250991494 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe2d41aa200 con 0x7fe2d410e360 2024-04-14T15:13:46.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.441+0000 7fe2d1ffb700 1 -- 172.21.15.90:0/250991494 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fe2cc0017b0 con 0x7fe2d410e360 2024-04-14T15:13:46.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.441+0000 7fe2d1ffb700 1 -- 172.21.15.90:0/250991494 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3747917750 0 0) 0x7fe2d41a8fc0 con 0x7fe2d41078e0 2024-04-14T15:13:46.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.441+0000 7fe2d1ffb700 1 -- 172.21.15.90:0/250991494 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe2c40035b0 con 0x7fe2d41078e0 2024-04-14T15:13:46.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.441+0000 7fe2d1ffb700 1 -- 172.21.15.90:0/250991494 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fe2c80018b0 con 0x7fe2d41078e0 2024-04-14T15:13:46.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.441+0000 7fe2d1ffb700 1 -- 172.21.15.90:0/250991494 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1176606559 0 0) 0x7fe2d410e160 con 0x7fe2d410a790 2024-04-14T15:13:46.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.441+0000 7fe2d1ffb700 1 -- 172.21.15.90:0/250991494 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe2d41a8fc0 con 0x7fe2d410a790 2024-04-14T15:13:46.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.441+0000 7fe2d1ffb700 1 -- 172.21.15.90:0/250991494 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1784592570 0 0) 0x7fe2d41aa200 con 0x7fe2d410e360 2024-04-14T15:13:46.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.441+0000 7fe2d1ffb700 1 -- 172.21.15.90:0/250991494 >> v1:172.21.15.90:6790/0 conn(0x7fe2d410a790 legacy=0x7fe2d406b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:46.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.441+0000 7fe2d1ffb700 1 -- 172.21.15.90:0/250991494 >> v1:172.21.15.90:6789/0 conn(0x7fe2d41078e0 legacy=0x7fe2d406ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:46.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.442+0000 7fe2d1ffb700 1 -- 172.21.15.90:0/250991494 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2d41ab440 con 0x7fe2d410e360 2024-04-14T15:13:46.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.442+0000 7fe2dac9e700 1 -- 172.21.15.90:0/250991494 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe2d41aa430 con 0x7fe2d410e360 2024-04-14T15:13:46.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.442+0000 7fe2d1ffb700 1 -- 172.21.15.90:0/250991494 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fe2cc001ef0 con 0x7fe2d410e360 2024-04-14T15:13:46.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.442+0000 7fe2d1ffb700 1 -- 172.21.15.90:0/250991494 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fe2cc0052a0 con 0x7fe2d410e360 2024-04-14T15:13:46.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.442+0000 7fe2dac9e700 10 client.48136.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:13:46.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.442+0000 7fe2dac9e700 1 -- 172.21.15.90:0/250991494 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe2d41aa8f0 con 0x7fe2d410e360 2024-04-14T15:13:46.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.442+0000 7fe2dac9e700 1 librados: init done 2024-04-14T15:13:46.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.443+0000 7fe2dac9e700 1 -- 172.21.15.90:0/250991494 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2d40693d0 con 0x7fe2d410e360 2024-04-14T15:13:46.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.443+0000 7fe2d1ffb700 1 -- 172.21.15.90:0/250991494 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fe2cc01bf20 con 0x7fe2d410e360 2024-04-14T15:13:46.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.443+0000 7fe2d1ffb700 1 -- 172.21.15.90:0/250991494 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(158..158 src has 51..158) ==== 11113+0+0 (unknown 2298989900 0 0) 0x7fe2cc08a800 con 0x7fe2d410e360 2024-04-14T15:13:46.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.443+0000 7fe2d1ffb700 10 client.48136.objecter ms_dispatch 0x7fe2d4001550 osd_map(158..158 src has 51..158) 2024-04-14T15:13:46.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.443+0000 7fe2d1ffb700 3 client.48136.objecter handle_osd_map got epochs [158,158] > 0 2024-04-14T15:13:46.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.443+0000 7fe2d1ffb700 3 client.48136.objecter handle_osd_map decoding full epoch 158 2024-04-14T15:13:46.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.444+0000 7fe2d1ffb700 20 client.48136.objecter dump_active .. 0 homeless 2024-04-14T15:13:46.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.444+0000 7fe2d1ffb700 10 client.48136.objecter ms_handle_connect 0x7fe2c40713d0 2024-04-14T15:13:46.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.448+0000 7fe2d1ffb700 1 -- 172.21.15.90:0/250991494 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fe2cc056680 con 0x7fe2d410e360 2024-04-14T15:13:46.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.659+0000 7fe2dac9e700 1 -- 172.21.15.90:0/250991494 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7fe2d410d460 con 0x7fe2d410e360 2024-04-14T15:13:46.675 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:13:46.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:46.700 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:46.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:13:46.802 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:13:46.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.820+0000 7fe2d1ffb700 1 -- 172.21.15.90:0/250991494 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v159) ==== 97+0+0 (unknown 95529550 0 0) 0x7fe2cc05a330 con 0x7fe2d410e360 2024-04-14T15:13:46.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.823+0000 7fe2dac9e700 10 librados: watch_flush enter 2024-04-14T15:13:46.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.824+0000 7fe2dac9e700 10 librados: watch_flush exit 2024-04-14T15:13:46.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.824+0000 7fe2dac9e700 20 client.48136.objecter shutdown clearing up homeless session... 2024-04-14T15:13:46.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.824+0000 7fe2dac9e700 10 client.48136.objecter successfully canceled tick 2024-04-14T15:13:46.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.824+0000 7fe2dac9e700 1 -- 172.21.15.90:0/250991494 >> v1:172.21.15.90:6800/420935427 conn(0x7fe2c40713d0 legacy=0x7fe2c4073890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:46.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.824+0000 7fe2dac9e700 1 -- 172.21.15.90:0/250991494 >> v1:172.21.15.92:6789/0 conn(0x7fe2d410e360 legacy=0x7fe2d41a6850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:46.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.825+0000 7fe2dac9e700 1 -- 172.21.15.90:0/250991494 shutdown_connections 2024-04-14T15:13:46.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.825+0000 7fe2dac9e700 1 -- 172.21.15.90:0/250991494 >> 172.21.15.90:0/250991494 conn(0x7fe2d40fd2c0 msgr2=0x7fe2d4111860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:13:46.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.829+0000 7fe2dac9e700 1 -- 172.21.15.90:0/250991494 shutdown_connections 2024-04-14T15:13:46.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.829+0000 7fe2dac9e700 1 -- 172.21.15.90:0/250991494 wait complete. 2024-04-14T15:13:46.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:46.830+0000 7fe2dac9e700 1 librados: shutdown 2024-04-14T15:13:46.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:46.833 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:46.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:13:46.834 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is set 2024-04-14T15:13:46.934 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:13:46.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:46.958 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:46.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:13:47.059 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:13:47.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:47.083 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:47.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:13:47.144 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-14T15:13:47.185 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:13:47.211 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:47.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:47.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:13:47.313 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:13:47.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:47.338 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:47.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:13:47.386 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:13:47.439 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:13:47.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:47 smithi090 ceph-mon[30399]: 172.1 scrub starts 2024-04-14T15:13:47.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:47 smithi090 ceph-mon[30399]: 172.1 scrub ok 2024-04-14T15:13:47.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:47 smithi090 ceph-mon[30399]: 172.2 scrub starts 2024-04-14T15:13:47.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:47 smithi090 ceph-mon[30399]: 172.2 scrub ok 2024-04-14T15:13:47.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:47 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/250991494' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:13:47.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:47 smithi090 ceph-mon[30399]: from='client.48136 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:13:47.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:47 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:47.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:47 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi090-93444-1","var":"pg_num","val":"11"}]: dispatch 2024-04-14T15:13:47.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:47 smithi090 ceph-mon[30399]: from='client.24829 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi090-93444-1","var":"pg_num","val":"11"}]: dispatch 2024-04-14T15:13:47.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:47 smithi090 ceph-mon[30399]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:13:47.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:47 smithi090 ceph-mon[30399]: from='client.48127 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi090-93342-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:47.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:47 smithi090 ceph-mon[30399]: from='client.48130 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi090-93363-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:47.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:47 smithi090 ceph-mon[30399]: from='client.48136 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-14T15:13:47.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:47 smithi090 ceph-mon[30399]: from='client.24829 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi090-93444-1","var":"pg_num","val":"11"}]': finished 2024-04-14T15:13:47.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:47 smithi090 ceph-mon[30399]: osdmap e159: 8 total, 8 up, 8 in 2024-04-14T15:13:47.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:47 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:47.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:47 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:47.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:47 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi090-93444-1","var":"pgp_num","val":"11"}]: dispatch 2024-04-14T15:13:47.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:47 smithi090 ceph-mon[30399]: from='client.24829 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi090-93444-1","var":"pgp_num","val":"11"}]: dispatch 2024-04-14T15:13:47.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:47 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-04-14T15:13:47.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:47 smithi090 ceph-mon[41670]: 172.1 scrub starts 2024-04-14T15:13:47.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:47 smithi090 ceph-mon[41670]: 172.1 scrub ok 2024-04-14T15:13:47.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:47 smithi090 ceph-mon[41670]: 172.2 scrub starts 2024-04-14T15:13:47.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:47 smithi090 ceph-mon[41670]: 172.2 scrub ok 2024-04-14T15:13:47.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:47 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/250991494' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:13:47.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:47 smithi090 ceph-mon[41670]: from='client.48136 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:13:47.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:47 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:47.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:47 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi090-93444-1","var":"pg_num","val":"11"}]: dispatch 2024-04-14T15:13:47.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:47 smithi090 ceph-mon[41670]: from='client.24829 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi090-93444-1","var":"pg_num","val":"11"}]: dispatch 2024-04-14T15:13:47.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:47 smithi090 ceph-mon[41670]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:13:47.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:47 smithi090 ceph-mon[41670]: from='client.48127 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi090-93342-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:47.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:47 smithi090 ceph-mon[41670]: from='client.48130 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi090-93363-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:47.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:47 smithi090 ceph-mon[41670]: from='client.48136 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-14T15:13:47.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:47 smithi090 ceph-mon[41670]: from='client.24829 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi090-93444-1","var":"pg_num","val":"11"}]': finished 2024-04-14T15:13:47.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:47 smithi090 ceph-mon[41670]: osdmap e159: 8 total, 8 up, 8 in 2024-04-14T15:13:47.445 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:47 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:47.445 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:47 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:47.445 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:47 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi090-93444-1","var":"pgp_num","val":"11"}]: dispatch 2024-04-14T15:13:47.445 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:47 smithi090 ceph-mon[41670]: from='client.24829 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi090-93444-1","var":"pgp_num","val":"11"}]: dispatch 2024-04-14T15:13:47.445 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:47 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-04-14T15:13:47.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:47.463 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:47.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:13:47.507 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:47 smithi092 ceph-mon[33966]: 172.1 scrub starts 2024-04-14T15:13:47.507 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:47 smithi092 ceph-mon[33966]: 172.1 scrub ok 2024-04-14T15:13:47.507 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:47 smithi092 ceph-mon[33966]: 172.2 scrub starts 2024-04-14T15:13:47.508 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:47 smithi092 ceph-mon[33966]: 172.2 scrub ok 2024-04-14T15:13:47.508 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:47 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/250991494' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:13:47.508 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:47 smithi092 ceph-mon[33966]: from='client.48136 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:13:47.508 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:47 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:47.508 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:47 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi090-93444-1","var":"pg_num","val":"11"}]: dispatch 2024-04-14T15:13:47.508 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:47 smithi092 ceph-mon[33966]: from='client.24829 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi090-93444-1","var":"pg_num","val":"11"}]: dispatch 2024-04-14T15:13:47.508 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:47 smithi092 ceph-mon[33966]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:13:47.508 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:47 smithi092 ceph-mon[33966]: from='client.48127 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi090-93342-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:47.508 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:47 smithi092 ceph-mon[33966]: from='client.48130 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi090-93363-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:47.508 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:47 smithi092 ceph-mon[33966]: from='client.48136 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-14T15:13:47.508 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:47 smithi092 ceph-mon[33966]: from='client.24829 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi090-93444-1","var":"pg_num","val":"11"}]': finished 2024-04-14T15:13:47.508 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:47 smithi092 ceph-mon[33966]: osdmap e159: 8 total, 8 up, 8 in 2024-04-14T15:13:47.508 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:47 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:47.508 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:47 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:47.509 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:47 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi090-93444-1","var":"pgp_num","val":"11"}]: dispatch 2024-04-14T15:13:47.509 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:47 smithi092 ceph-mon[33966]: from='client.24829 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi090-93444-1","var":"pgp_num","val":"11"}]: dispatch 2024-04-14T15:13:47.509 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:47 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-04-14T15:13:47.564 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:13:47.588 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:47.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:47.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:13:47.690 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:13:47.714 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:47.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:47.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:13:47.816 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:13:47.852 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:47.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:47.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:13:47.954 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:13:47.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:47.978 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:47.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:13:48.081 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:13:48.105 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:48.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:48.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:13:48.207 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:13:48.231 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:48.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:48.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:13:48.333 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:13:48.357 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:48.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:48.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:13:48.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:48 smithi092 ceph-mon[33966]: pgmap v185: 428 pgs: 96 unknown, 332 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 61 KiB/s wr, 73 op/s 2024-04-14T15:13:48.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:48 smithi092 ceph-mon[33966]: 172.6 scrub starts 2024-04-14T15:13:48.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:48 smithi092 ceph-mon[33966]: 172.6 scrub ok 2024-04-14T15:13:48.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:48 smithi092 ceph-mon[33966]: from='client.24829 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi090-93444-1","var":"pgp_num","val":"11"}]': finished 2024-04-14T15:13:48.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:48 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pgp_num_actual", "val": "11"}]': finished 2024-04-14T15:13:48.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:48 smithi092 ceph-mon[33966]: osdmap e160: 8 total, 8 up, 8 in 2024-04-14T15:13:48.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:48 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2918742503' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi090-93709-15"}]: dispatch 2024-04-14T15:13:48.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:48 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:48.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:48 smithi092 ceph-mon[33966]: from='client.48067 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi090-93709-15"}]: dispatch 2024-04-14T15:13:48.458 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:13:48.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:48.482 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:48.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:13:48.583 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:13:48.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:48.608 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:48.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:13:48.676 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:48 smithi090 ceph-mon[30399]: pgmap v185: 428 pgs: 96 unknown, 332 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 61 KiB/s wr, 73 op/s 2024-04-14T15:13:48.676 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:48 smithi090 ceph-mon[30399]: 172.6 scrub starts 2024-04-14T15:13:48.676 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:48 smithi090 ceph-mon[30399]: 172.6 scrub ok 2024-04-14T15:13:48.676 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:48 smithi090 ceph-mon[30399]: from='client.24829 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi090-93444-1","var":"pgp_num","val":"11"}]': finished 2024-04-14T15:13:48.676 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:48 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pgp_num_actual", "val": "11"}]': finished 2024-04-14T15:13:48.676 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:48 smithi090 ceph-mon[30399]: osdmap e160: 8 total, 8 up, 8 in 2024-04-14T15:13:48.676 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:48 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2918742503' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi090-93709-15"}]: dispatch 2024-04-14T15:13:48.676 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:48 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:48.677 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:48 smithi090 ceph-mon[30399]: from='client.48067 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi090-93709-15"}]: dispatch 2024-04-14T15:13:48.677 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:48 smithi090 ceph-mon[41670]: pgmap v185: 428 pgs: 96 unknown, 332 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 61 KiB/s wr, 73 op/s 2024-04-14T15:13:48.677 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:48 smithi090 ceph-mon[41670]: 172.6 scrub starts 2024-04-14T15:13:48.677 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:48 smithi090 ceph-mon[41670]: 172.6 scrub ok 2024-04-14T15:13:48.677 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:48 smithi090 ceph-mon[41670]: from='client.24829 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi090-93444-1","var":"pgp_num","val":"11"}]': finished 2024-04-14T15:13:48.677 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:48 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pgp_num_actual", "val": "11"}]': finished 2024-04-14T15:13:48.677 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:48 smithi090 ceph-mon[41670]: osdmap e160: 8 total, 8 up, 8 in 2024-04-14T15:13:48.677 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:48 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2918742503' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi090-93709-15"}]: dispatch 2024-04-14T15:13:48.677 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:48 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:48.677 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:48 smithi090 ceph-mon[41670]: from='client.48067 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi090-93709-15"}]: dispatch 2024-04-14T15:13:48.710 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:13:48.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:48.735 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:48.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:13:48.836 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:13:48.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:48.861 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:48.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:13:48.962 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:13:48.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:48.987 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:48.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:13:49.088 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:13:49.111 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:49.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:49.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:13:49.154 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-04-14T15:13:49.154 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:13:49.213 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:13:49.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:49.239 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:49.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:13:49.341 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:13:49.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:49.366 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:49.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:13:49.467 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:13:49.491 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:49.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:49.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:13:49.593 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:13:49.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:49.619 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:49.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:13:49.720 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:13:49.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.582+0000 7f3c5ea17700 1 -- 172.21.15.90:0/713075790 >> v1:172.21.15.90:6790/0 conn(0x7f3c58101630 legacy=0x7f3c58101a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:49.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.582+0000 7f3c5ea17700 1 -- 172.21.15.90:0/713075790 shutdown_connections 2024-04-14T15:13:49.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.582+0000 7f3c5ea17700 1 -- 172.21.15.90:0/713075790 >> 172.21.15.90:0/713075790 conn(0x7f3c580fd2b0 msgr2=0x7f3c580ff6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:13:49.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.582+0000 7f3c5ea17700 1 -- 172.21.15.90:0/713075790 shutdown_connections 2024-04-14T15:13:49.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.582+0000 7f3c5ea17700 1 -- 172.21.15.90:0/713075790 wait complete. 2024-04-14T15:13:49.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.583+0000 7f3c5ea17700 1 librados: starting msgr at 2024-04-14T15:13:49.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.583+0000 7f3c5ea17700 1 librados: starting objecter 2024-04-14T15:13:49.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.583+0000 7f3c5ea17700 1 Processor -- start 2024-04-14T15:13:49.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.583+0000 7f3c5ea17700 1 -- start start 2024-04-14T15:13:49.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.583+0000 7f3c5ea17700 1 librados: setting wanted keys 2024-04-14T15:13:49.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.583+0000 7f3c5ea17700 1 librados: calling monclient init 2024-04-14T15:13:49.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.583+0000 7f3c5ea17700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3c581a9b70 con 0x7f3c58102210 2024-04-14T15:13:49.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.583+0000 7f3c5ea17700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3c581aad90 con 0x7f3c58101630 2024-04-14T15:13:49.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.583+0000 7f3c5ea17700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3c581abfd0 con 0x7f3c58111530 2024-04-14T15:13:49.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.583+0000 7f3c5cfb4700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f3c58111530 0x7f3c581a8230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:35636/0 (socket says 172.21.15.90:35636) 2024-04-14T15:13:49.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.583+0000 7f3c577fe700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f3c58102210 0x7f3c581a4990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:41506/0 (socket says 172.21.15.90:41506) 2024-04-14T15:13:49.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.583+0000 7f3c5cfb4700 1 -- 172.21.15.90:0/2640457140 learned_addr learned my addr 172.21.15.90:0/2640457140 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:13:49.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.583+0000 7f3c557fa700 10 client.?.objecter ms_handle_connect 0x7f3c58102210 2024-04-14T15:13:49.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.583+0000 7f3c557fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:13:49.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.583+0000 7f3c557fa700 10 client.?.objecter ms_handle_connect 0x7f3c58111530 2024-04-14T15:13:49.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.583+0000 7f3c557fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:13:49.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.583+0000 7f3c557fa700 10 client.?.objecter ms_handle_connect 0x7f3c58101630 2024-04-14T15:13:49.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.583+0000 7f3c557fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:13:49.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.584+0000 7f3c557fa700 1 -- 172.21.15.90:0/2640457140 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3994253288 0 0) 0x7f3c581a9b70 con 0x7f3c58102210 2024-04-14T15:13:49.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.584+0000 7f3c557fa700 1 -- 172.21.15.90:0/2640457140 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3c400031f0 con 0x7f3c58102210 2024-04-14T15:13:49.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.584+0000 7f3c557fa700 1 -- 172.21.15.90:0/2640457140 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 228818672 0 0) 0x7f3c581aad90 con 0x7f3c58101630 2024-04-14T15:13:49.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.584+0000 7f3c557fa700 1 -- 172.21.15.90:0/2640457140 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3c581a9b70 con 0x7f3c58101630 2024-04-14T15:13:49.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.584+0000 7f3c557fa700 1 -- 172.21.15.90:0/2640457140 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 986050335 0 0) 0x7f3c581abfd0 con 0x7f3c58111530 2024-04-14T15:13:49.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.584+0000 7f3c557fa700 1 -- 172.21.15.90:0/2640457140 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3c581aad90 con 0x7f3c58111530 2024-04-14T15:13:49.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.584+0000 7f3c557fa700 1 -- 172.21.15.90:0/2640457140 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3497344287 0 0) 0x7f3c581a9b70 con 0x7f3c58101630 2024-04-14T15:13:49.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.584+0000 7f3c557fa700 1 -- 172.21.15.90:0/2640457140 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3c581abfd0 con 0x7f3c58101630 2024-04-14T15:13:49.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.584+0000 7f3c557fa700 1 -- 172.21.15.90:0/2640457140 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 542975529 0 0) 0x7f3c581aad90 con 0x7f3c58111530 2024-04-14T15:13:49.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.584+0000 7f3c557fa700 1 -- 172.21.15.90:0/2640457140 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3c581a9b70 con 0x7f3c58111530 2024-04-14T15:13:49.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.584+0000 7f3c557fa700 1 -- 172.21.15.90:0/2640457140 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f3c48004470 con 0x7f3c58101630 2024-04-14T15:13:49.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.584+0000 7f3c557fa700 1 -- 172.21.15.90:0/2640457140 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2947419490 0 0) 0x7f3c400031f0 con 0x7f3c58102210 2024-04-14T15:13:49.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.584+0000 7f3c557fa700 1 -- 172.21.15.90:0/2640457140 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3c581aad90 con 0x7f3c58102210 2024-04-14T15:13:49.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.584+0000 7f3c557fa700 1 -- 172.21.15.90:0/2640457140 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f3c50001c10 con 0x7f3c58111530 2024-04-14T15:13:49.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.584+0000 7f3c557fa700 1 -- 172.21.15.90:0/2640457140 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f3c4c001870 con 0x7f3c58102210 2024-04-14T15:13:49.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.584+0000 7f3c557fa700 1 -- 172.21.15.90:0/2640457140 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2078582784 0 0) 0x7f3c581abfd0 con 0x7f3c58101630 2024-04-14T15:13:49.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.584+0000 7f3c557fa700 1 -- 172.21.15.90:0/2640457140 >> v1:172.21.15.90:6790/0 conn(0x7f3c58111530 legacy=0x7f3c581a8230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:49.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.584+0000 7f3c557fa700 1 -- 172.21.15.90:0/2640457140 >> v1:172.21.15.90:6789/0 conn(0x7f3c58102210 legacy=0x7f3c581a4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:49.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.584+0000 7f3c557fa700 1 -- 172.21.15.90:0/2640457140 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c581ad210 con 0x7f3c58101630 2024-04-14T15:13:49.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.584+0000 7f3c5ea17700 1 -- 172.21.15.90:0/2640457140 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3c581ac200 con 0x7f3c58101630 2024-04-14T15:13:49.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.584+0000 7f3c5ea17700 10 client.48157.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:13:49.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.584+0000 7f3c5ea17700 1 -- 172.21.15.90:0/2640457140 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f3c581ac6c0 con 0x7f3c58101630 2024-04-14T15:13:49.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.584+0000 7f3c5ea17700 1 librados: init done 2024-04-14T15:13:49.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.584+0000 7f3c557fa700 1 -- 172.21.15.90:0/2640457140 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f3c48001ae0 con 0x7f3c58101630 2024-04-14T15:13:49.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.584+0000 7f3c557fa700 1 -- 172.21.15.90:0/2640457140 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f3c48004d10 con 0x7f3c58101630 2024-04-14T15:13:49.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.585+0000 7f3c5ea17700 1 -- 172.21.15.90:0/2640457140 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f3c44005100 con 0x7f3c58101630 2024-04-14T15:13:49.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.586+0000 7f3c557fa700 1 -- 172.21.15.90:0/2640457140 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f3c480037f0 con 0x7f3c58101630 2024-04-14T15:13:49.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.586+0000 7f3c557fa700 1 -- 172.21.15.90:0/2640457140 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(161..161 src has 51..161) ==== 9895+0+0 (unknown 4128360227 0 0) 0x7f3c4808a0b0 con 0x7f3c58101630 2024-04-14T15:13:49.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.586+0000 7f3c557fa700 10 client.48157.objecter ms_dispatch 0x7f3c58001550 osd_map(161..161 src has 51..161) 2024-04-14T15:13:49.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.586+0000 7f3c557fa700 3 client.48157.objecter handle_osd_map got epochs [161,161] > 0 2024-04-14T15:13:49.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.586+0000 7f3c557fa700 3 client.48157.objecter handle_osd_map decoding full epoch 161 2024-04-14T15:13:49.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.586+0000 7f3c557fa700 20 client.48157.objecter dump_active .. 0 homeless 2024-04-14T15:13:49.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.586+0000 7f3c557fa700 10 client.48157.objecter ms_handle_connect 0x7f3c40071580 2024-04-14T15:13:49.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.586+0000 7f3c557fa700 1 -- 172.21.15.90:0/2640457140 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v161) ==== 54+0+16 (unknown 1338815507 0 2420000192) 0x7f3c4808a4b0 con 0x7f3c58101630 2024-04-14T15:13:49.726 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:13:49.726 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:13:49.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.586+0000 7f3c5ea17700 10 client.48157.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:13:49.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.586+0000 7f3c5ea17700 15 client.48157.objecter _session_command_op_assign -1 1 2024-04-14T15:13:49.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.586+0000 7f3c5ea17700 20 client.48157.objecter _get_session s=0x7f3c440053e0 osd=0 3 2024-04-14T15:13:49.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.586+0000 7f3c5ea17700 20 client.48157.objecter put_session s=0x7f3c440053e0 osd=0 3 2024-04-14T15:13:49.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.586+0000 7f3c5ea17700 20 client.48157.objecter _get_session s=0x7f3c440053e0 osd=0 3 2024-04-14T15:13:49.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.586+0000 7f3c5ea17700 15 client.48157.objecter _session_command_op_remove -1 1 2024-04-14T15:13:49.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.586+0000 7f3c5ea17700 20 client.48157.objecter get_session s=0x7f3c440053e0 osd=0 3 2024-04-14T15:13:49.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.586+0000 7f3c5ea17700 15 client.48157.objecter _session_command_op_assign 0 1 2024-04-14T15:13:49.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.586+0000 7f3c5ea17700 20 client.48157.objecter put_session s=0x7f3c440053e0 osd=0 4 2024-04-14T15:13:49.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.586+0000 7f3c5ea17700 10 client.48157.objecter _send_command 1 2024-04-14T15:13:49.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.586+0000 7f3c5ea17700 1 -- 172.21.15.90:0/2640457140 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3c44004b50 con 0x7f3c44004730 2024-04-14T15:13:49.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.587+0000 7f3c557fa700 10 client.48157.objecter ms_handle_connect 0x7f3c44004730 2024-04-14T15:13:49.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.587+0000 7f3c557fa700 1 -- 172.21.15.90:0/2640457140 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f3c44004b50 con 0x7f3c44004730 2024-04-14T15:13:49.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.587+0000 7f3c557fa700 10 client.48157.objecter ms_dispatch 0x7f3c58001550 command_reply(tid 1: 0 ) 2024-04-14T15:13:49.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.587+0000 7f3c557fa700 10 client.48157.objecter _finish_command 1 = system:0 2024-04-14T15:13:49.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.587+0000 7f3c557fa700 20 client.48157.objecter put_session s=0x7f3c440053e0 osd=0 4 2024-04-14T15:13:49.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.587+0000 7f3c557fa700 15 client.48157.objecter _session_command_op_remove 0 1 2024-04-14T15:13:49.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.604+0000 7f3c5ea17700 10 client.48157.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:13:49.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.604+0000 7f3c5ea17700 15 client.48157.objecter _session_command_op_assign -1 2 2024-04-14T15:13:49.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.604+0000 7f3c5ea17700 20 client.48157.objecter _get_session s=0x7f3c440053e0 osd=0 3 2024-04-14T15:13:49.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.604+0000 7f3c5ea17700 20 client.48157.objecter put_session s=0x7f3c440053e0 osd=0 3 2024-04-14T15:13:49.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.604+0000 7f3c5ea17700 20 client.48157.objecter _get_session s=0x7f3c440053e0 osd=0 3 2024-04-14T15:13:49.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.604+0000 7f3c5ea17700 15 client.48157.objecter _session_command_op_remove -1 2 2024-04-14T15:13:49.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.604+0000 7f3c5ea17700 20 client.48157.objecter get_session s=0x7f3c440053e0 osd=0 3 2024-04-14T15:13:49.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.604+0000 7f3c5ea17700 15 client.48157.objecter _session_command_op_assign 0 2 2024-04-14T15:13:49.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.604+0000 7f3c5ea17700 20 client.48157.objecter put_session s=0x7f3c440053e0 osd=0 4 2024-04-14T15:13:49.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.604+0000 7f3c5ea17700 10 client.48157.objecter _send_command 2 2024-04-14T15:13:49.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.604+0000 7f3c5ea17700 1 -- 172.21.15.90:0/2640457140 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f3c440079f0 con 0x7f3c44004730 2024-04-14T15:13:49.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.605+0000 7f3c557fa700 1 -- 172.21.15.90:0/2640457140 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f3c440079f0 con 0x7f3c44004730 2024-04-14T15:13:49.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.605+0000 7f3c557fa700 10 client.48157.objecter ms_dispatch 0x7f3c58001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:13:49.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.605+0000 7f3c557fa700 10 client.48157.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:13:49.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.605+0000 7f3c557fa700 20 client.48157.objecter put_session s=0x7f3c440053e0 osd=0 4 2024-04-14T15:13:49.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.605+0000 7f3c557fa700 15 client.48157.objecter _session_command_op_remove 0 2 2024-04-14T15:13:49.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.606+0000 7f3c5ea17700 10 client.48157.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:13:49.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.606+0000 7f3c5ea17700 15 client.48157.objecter _session_command_op_assign -1 3 2024-04-14T15:13:49.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.606+0000 7f3c5ea17700 20 client.48157.objecter _get_session s=0x7f3c44004110 osd=1 3 2024-04-14T15:13:49.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.606+0000 7f3c5ea17700 20 client.48157.objecter put_session s=0x7f3c44004110 osd=1 3 2024-04-14T15:13:49.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.606+0000 7f3c5ea17700 20 client.48157.objecter _get_session s=0x7f3c44004110 osd=1 3 2024-04-14T15:13:49.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.606+0000 7f3c5ea17700 15 client.48157.objecter _session_command_op_remove -1 3 2024-04-14T15:13:49.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.606+0000 7f3c5ea17700 20 client.48157.objecter get_session s=0x7f3c44004110 osd=1 3 2024-04-14T15:13:49.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.606+0000 7f3c5ea17700 15 client.48157.objecter _session_command_op_assign 1 3 2024-04-14T15:13:49.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.606+0000 7f3c5ea17700 20 client.48157.objecter put_session s=0x7f3c44004110 osd=1 4 2024-04-14T15:13:49.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.606+0000 7f3c5ea17700 10 client.48157.objecter _send_command 3 2024-04-14T15:13:49.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.606+0000 7f3c5ea17700 1 -- 172.21.15.90:0/2640457140 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f3c44002180 con 0x7f3c44004280 2024-04-14T15:13:49.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.606+0000 7f3c557fa700 10 client.48157.objecter ms_handle_connect 0x7f3c44004280 2024-04-14T15:13:49.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.607+0000 7f3c557fa700 1 -- 172.21.15.90:0/2640457140 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f3c44002180 con 0x7f3c44004280 2024-04-14T15:13:49.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.607+0000 7f3c557fa700 10 client.48157.objecter ms_dispatch 0x7f3c58001550 command_reply(tid 3: 0 ) 2024-04-14T15:13:49.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.607+0000 7f3c557fa700 10 client.48157.objecter _finish_command 3 = system:0 2024-04-14T15:13:49.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.607+0000 7f3c557fa700 20 client.48157.objecter put_session s=0x7f3c44004110 osd=1 4 2024-04-14T15:13:49.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.607+0000 7f3c557fa700 15 client.48157.objecter _session_command_op_remove 1 3 2024-04-14T15:13:49.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.623+0000 7f3c5ea17700 10 client.48157.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:13:49.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.623+0000 7f3c5ea17700 15 client.48157.objecter _session_command_op_assign -1 4 2024-04-14T15:13:49.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.623+0000 7f3c5ea17700 20 client.48157.objecter _get_session s=0x7f3c44004110 osd=1 3 2024-04-14T15:13:49.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.623+0000 7f3c5ea17700 20 client.48157.objecter put_session s=0x7f3c44004110 osd=1 3 2024-04-14T15:13:49.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.623+0000 7f3c5ea17700 20 client.48157.objecter _get_session s=0x7f3c44004110 osd=1 3 2024-04-14T15:13:49.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.623+0000 7f3c5ea17700 15 client.48157.objecter _session_command_op_remove -1 4 2024-04-14T15:13:49.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.623+0000 7f3c5ea17700 20 client.48157.objecter get_session s=0x7f3c44004110 osd=1 3 2024-04-14T15:13:49.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.623+0000 7f3c5ea17700 15 client.48157.objecter _session_command_op_assign 1 4 2024-04-14T15:13:49.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.623+0000 7f3c5ea17700 20 client.48157.objecter put_session s=0x7f3c44004110 osd=1 4 2024-04-14T15:13:49.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.623+0000 7f3c5ea17700 10 client.48157.objecter _send_command 4 2024-04-14T15:13:49.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.623+0000 7f3c5ea17700 1 -- 172.21.15.90:0/2640457140 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f3c44001600 con 0x7f3c44004280 2024-04-14T15:13:49.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.624+0000 7f3c557fa700 1 -- 172.21.15.90:0/2640457140 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f3c44001600 con 0x7f3c44004280 2024-04-14T15:13:49.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.625+0000 7f3c557fa700 10 client.48157.objecter ms_dispatch 0x7f3c58001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:13:49.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.625+0000 7f3c557fa700 10 client.48157.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:13:49.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.625+0000 7f3c557fa700 20 client.48157.objecter put_session s=0x7f3c44004110 osd=1 4 2024-04-14T15:13:49.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.625+0000 7f3c557fa700 15 client.48157.objecter _session_command_op_remove 1 4 2024-04-14T15:13:49.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.625+0000 7f3c5ea17700 10 client.48157.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:13:49.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.625+0000 7f3c5ea17700 15 client.48157.objecter _session_command_op_assign -1 5 2024-04-14T15:13:49.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.625+0000 7f3c5ea17700 20 client.48157.objecter _get_session s=0x7f3c44001a00 osd=2 3 2024-04-14T15:13:49.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.625+0000 7f3c5ea17700 20 client.48157.objecter put_session s=0x7f3c44001a00 osd=2 3 2024-04-14T15:13:49.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.625+0000 7f3c5ea17700 20 client.48157.objecter _get_session s=0x7f3c44001a00 osd=2 3 2024-04-14T15:13:49.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.625+0000 7f3c5ea17700 15 client.48157.objecter _session_command_op_remove -1 5 2024-04-14T15:13:49.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.625+0000 7f3c5ea17700 20 client.48157.objecter get_session s=0x7f3c44001a00 osd=2 3 2024-04-14T15:13:49.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.625+0000 7f3c5ea17700 15 client.48157.objecter _session_command_op_assign 2 5 2024-04-14T15:13:49.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.625+0000 7f3c5ea17700 20 client.48157.objecter put_session s=0x7f3c44001a00 osd=2 4 2024-04-14T15:13:49.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.625+0000 7f3c5ea17700 10 client.48157.objecter _send_command 5 2024-04-14T15:13:49.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.625+0000 7f3c5ea17700 1 -- 172.21.15.90:0/2640457140 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f3c44010f20 con 0x7f3c44008880 2024-04-14T15:13:49.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.625+0000 7f3c557fa700 10 client.48157.objecter ms_handle_connect 0x7f3c44008880 2024-04-14T15:13:49.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.626+0000 7f3c557fa700 1 -- 172.21.15.90:0/2640457140 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f3c44010f20 con 0x7f3c44008880 2024-04-14T15:13:49.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.626+0000 7f3c557fa700 10 client.48157.objecter ms_dispatch 0x7f3c58001550 command_reply(tid 5: 0 ) 2024-04-14T15:13:49.732 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:13:49.733 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:13:49.733 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:13:49.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.626+0000 7f3c557fa700 10 client.48157.objecter _finish_command 5 = system:0 2024-04-14T15:13:49.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.626+0000 7f3c557fa700 20 client.48157.objecter put_session s=0x7f3c44001a00 osd=2 4 2024-04-14T15:13:49.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.626+0000 7f3c557fa700 15 client.48157.objecter _session_command_op_remove 2 5 2024-04-14T15:13:49.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.642+0000 7f3c5ea17700 10 client.48157.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:13:49.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.642+0000 7f3c5ea17700 15 client.48157.objecter _session_command_op_assign -1 6 2024-04-14T15:13:49.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.642+0000 7f3c5ea17700 20 client.48157.objecter _get_session s=0x7f3c44001a00 osd=2 3 2024-04-14T15:13:49.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.642+0000 7f3c5ea17700 20 client.48157.objecter put_session s=0x7f3c44001a00 osd=2 3 2024-04-14T15:13:49.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.642+0000 7f3c5ea17700 20 client.48157.objecter _get_session s=0x7f3c44001a00 osd=2 3 2024-04-14T15:13:49.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.642+0000 7f3c5ea17700 15 client.48157.objecter _session_command_op_remove -1 6 2024-04-14T15:13:49.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.642+0000 7f3c5ea17700 20 client.48157.objecter get_session s=0x7f3c44001a00 osd=2 3 2024-04-14T15:13:49.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.642+0000 7f3c5ea17700 15 client.48157.objecter _session_command_op_assign 2 6 2024-04-14T15:13:49.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.642+0000 7f3c5ea17700 20 client.48157.objecter put_session s=0x7f3c44001a00 osd=2 4 2024-04-14T15:13:49.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.642+0000 7f3c5ea17700 10 client.48157.objecter _send_command 6 2024-04-14T15:13:49.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.642+0000 7f3c5ea17700 1 -- 172.21.15.90:0/2640457140 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f3c44007df0 con 0x7f3c44008880 2024-04-14T15:13:49.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.644+0000 7f3c557fa700 1 -- 172.21.15.90:0/2640457140 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f3c44007df0 con 0x7f3c44008880 2024-04-14T15:13:49.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.644+0000 7f3c557fa700 10 client.48157.objecter ms_dispatch 0x7f3c58001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:13:49.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.644+0000 7f3c557fa700 10 client.48157.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:13:49.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.644+0000 7f3c557fa700 20 client.48157.objecter put_session s=0x7f3c44001a00 osd=2 4 2024-04-14T15:13:49.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.644+0000 7f3c557fa700 15 client.48157.objecter _session_command_op_remove 2 6 2024-04-14T15:13:49.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.644+0000 7f3c5ea17700 10 client.48157.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:13:49.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.644+0000 7f3c5ea17700 15 client.48157.objecter _session_command_op_assign -1 7 2024-04-14T15:13:49.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.644+0000 7f3c5ea17700 20 client.48157.objecter _get_session s=0x7f3c44007df0 osd=3 3 2024-04-14T15:13:49.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.644+0000 7f3c5ea17700 20 client.48157.objecter put_session s=0x7f3c44007df0 osd=3 3 2024-04-14T15:13:49.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.644+0000 7f3c5ea17700 20 client.48157.objecter _get_session s=0x7f3c44007df0 osd=3 3 2024-04-14T15:13:49.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.644+0000 7f3c5ea17700 15 client.48157.objecter _session_command_op_remove -1 7 2024-04-14T15:13:49.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.644+0000 7f3c5ea17700 20 client.48157.objecter get_session s=0x7f3c44007df0 osd=3 3 2024-04-14T15:13:49.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.644+0000 7f3c5ea17700 15 client.48157.objecter _session_command_op_assign 3 7 2024-04-14T15:13:49.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.644+0000 7f3c5ea17700 20 client.48157.objecter put_session s=0x7f3c44007df0 osd=3 4 2024-04-14T15:13:49.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.644+0000 7f3c5ea17700 10 client.48157.objecter _send_command 7 2024-04-14T15:13:49.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.644+0000 7f3c5ea17700 1 -- 172.21.15.90:0/2640457140 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f3c44015ad0 con 0x7f3c4400d130 2024-04-14T15:13:49.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.644+0000 7f3c557fa700 10 client.48157.objecter ms_handle_connect 0x7f3c4400d130 2024-04-14T15:13:49.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.645+0000 7f3c557fa700 1 -- 172.21.15.90:0/2640457140 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f3c44015ad0 con 0x7f3c4400d130 2024-04-14T15:13:49.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.645+0000 7f3c557fa700 10 client.48157.objecter ms_dispatch 0x7f3c58001550 command_reply(tid 7: 0 ) 2024-04-14T15:13:49.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.645+0000 7f3c557fa700 10 client.48157.objecter _finish_command 7 = system:0 2024-04-14T15:13:49.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.645+0000 7f3c557fa700 20 client.48157.objecter put_session s=0x7f3c44007df0 osd=3 4 2024-04-14T15:13:49.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.645+0000 7f3c557fa700 15 client.48157.objecter _session_command_op_remove 3 7 2024-04-14T15:13:49.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.661+0000 7f3c5ea17700 10 client.48157.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:13:49.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.661+0000 7f3c5ea17700 15 client.48157.objecter _session_command_op_assign -1 8 2024-04-14T15:13:49.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.661+0000 7f3c5ea17700 20 client.48157.objecter _get_session s=0x7f3c44007df0 osd=3 3 2024-04-14T15:13:49.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.661+0000 7f3c5ea17700 20 client.48157.objecter put_session s=0x7f3c44007df0 osd=3 3 2024-04-14T15:13:49.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.661+0000 7f3c5ea17700 20 client.48157.objecter _get_session s=0x7f3c44007df0 osd=3 3 2024-04-14T15:13:49.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.661+0000 7f3c5ea17700 15 client.48157.objecter _session_command_op_remove -1 8 2024-04-14T15:13:49.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.661+0000 7f3c5ea17700 20 client.48157.objecter get_session s=0x7f3c44007df0 osd=3 3 2024-04-14T15:13:49.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.661+0000 7f3c5ea17700 15 client.48157.objecter _session_command_op_assign 3 8 2024-04-14T15:13:49.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.661+0000 7f3c5ea17700 20 client.48157.objecter put_session s=0x7f3c44007df0 osd=3 4 2024-04-14T15:13:49.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.661+0000 7f3c5ea17700 10 client.48157.objecter _send_command 8 2024-04-14T15:13:49.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.661+0000 7f3c5ea17700 1 -- 172.21.15.90:0/2640457140 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f3c4400c6a0 con 0x7f3c4400d130 2024-04-14T15:13:49.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.663+0000 7f3c557fa700 1 -- 172.21.15.90:0/2640457140 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f3c4400c6a0 con 0x7f3c4400d130 2024-04-14T15:13:49.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.663+0000 7f3c557fa700 10 client.48157.objecter ms_dispatch 0x7f3c58001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:13:49.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.663+0000 7f3c557fa700 10 client.48157.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:13:49.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.663+0000 7f3c557fa700 20 client.48157.objecter put_session s=0x7f3c44007df0 osd=3 4 2024-04-14T15:13:49.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.663+0000 7f3c557fa700 15 client.48157.objecter _session_command_op_remove 3 8 2024-04-14T15:13:49.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.663+0000 7f3c5ea17700 10 client.48157.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:13:49.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.663+0000 7f3c5ea17700 15 client.48157.objecter _session_command_op_assign -1 9 2024-04-14T15:13:49.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.663+0000 7f3c5ea17700 20 client.48157.objecter _get_session s=0x7f3c4400c450 osd=4 3 2024-04-14T15:13:49.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.663+0000 7f3c5ea17700 20 client.48157.objecter put_session s=0x7f3c4400c450 osd=4 3 2024-04-14T15:13:49.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.663+0000 7f3c5ea17700 20 client.48157.objecter _get_session s=0x7f3c4400c450 osd=4 3 2024-04-14T15:13:49.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.663+0000 7f3c5ea17700 15 client.48157.objecter _session_command_op_remove -1 9 2024-04-14T15:13:49.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.663+0000 7f3c5ea17700 20 client.48157.objecter get_session s=0x7f3c4400c450 osd=4 3 2024-04-14T15:13:49.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.663+0000 7f3c5ea17700 15 client.48157.objecter _session_command_op_assign 4 9 2024-04-14T15:13:49.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.663+0000 7f3c5ea17700 20 client.48157.objecter put_session s=0x7f3c4400c450 osd=4 4 2024-04-14T15:13:49.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.663+0000 7f3c5ea17700 10 client.48157.objecter _send_command 9 2024-04-14T15:13:49.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.663+0000 7f3c5ea17700 1 -- 172.21.15.90:0/2640457140 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f3c4400c5c0 con 0x7f3c440135b0 2024-04-14T15:13:49.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.664+0000 7f3c557fa700 10 client.48157.objecter ms_handle_connect 0x7f3c440135b0 2024-04-14T15:13:49.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.664+0000 7f3c557fa700 1 -- 172.21.15.90:0/2640457140 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f3c4400c5c0 con 0x7f3c440135b0 2024-04-14T15:13:49.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.664+0000 7f3c557fa700 10 client.48157.objecter ms_dispatch 0x7f3c58001550 command_reply(tid 9: 0 ) 2024-04-14T15:13:49.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.664+0000 7f3c557fa700 10 client.48157.objecter _finish_command 9 = system:0 2024-04-14T15:13:49.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.664+0000 7f3c557fa700 20 client.48157.objecter put_session s=0x7f3c4400c450 osd=4 4 2024-04-14T15:13:49.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.664+0000 7f3c557fa700 15 client.48157.objecter _session_command_op_remove 4 9 2024-04-14T15:13:49.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.680+0000 7f3c5ea17700 10 client.48157.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:13:49.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.680+0000 7f3c5ea17700 15 client.48157.objecter _session_command_op_assign -1 10 2024-04-14T15:13:49.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.680+0000 7f3c5ea17700 20 client.48157.objecter _get_session s=0x7f3c4400c450 osd=4 3 2024-04-14T15:13:49.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.680+0000 7f3c5ea17700 20 client.48157.objecter put_session s=0x7f3c4400c450 osd=4 3 2024-04-14T15:13:49.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.680+0000 7f3c5ea17700 20 client.48157.objecter _get_session s=0x7f3c4400c450 osd=4 3 2024-04-14T15:13:49.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.680+0000 7f3c5ea17700 15 client.48157.objecter _session_command_op_remove -1 10 2024-04-14T15:13:49.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.680+0000 7f3c5ea17700 20 client.48157.objecter get_session s=0x7f3c4400c450 osd=4 3 2024-04-14T15:13:49.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.680+0000 7f3c5ea17700 15 client.48157.objecter _session_command_op_assign 4 10 2024-04-14T15:13:49.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.680+0000 7f3c5ea17700 20 client.48157.objecter put_session s=0x7f3c4400c450 osd=4 4 2024-04-14T15:13:49.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.680+0000 7f3c5ea17700 10 client.48157.objecter _send_command 10 2024-04-14T15:13:49.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.680+0000 7f3c5ea17700 1 -- 172.21.15.90:0/2640457140 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f3c4401a660 con 0x7f3c440135b0 2024-04-14T15:13:49.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.682+0000 7f3c557fa700 1 -- 172.21.15.90:0/2640457140 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f3c4401a660 con 0x7f3c440135b0 2024-04-14T15:13:49.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.682+0000 7f3c557fa700 10 client.48157.objecter ms_dispatch 0x7f3c58001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:13:49.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.682+0000 7f3c557fa700 10 client.48157.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:13:49.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.682+0000 7f3c557fa700 20 client.48157.objecter put_session s=0x7f3c4400c450 osd=4 4 2024-04-14T15:13:49.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.682+0000 7f3c557fa700 15 client.48157.objecter _session_command_op_remove 4 10 2024-04-14T15:13:49.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.682+0000 7f3c5ea17700 10 client.48157.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:13:49.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.682+0000 7f3c5ea17700 15 client.48157.objecter _session_command_op_assign -1 11 2024-04-14T15:13:49.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.682+0000 7f3c5ea17700 20 client.48157.objecter _get_session s=0x7f3c4400ca70 osd=5 3 2024-04-14T15:13:49.739 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:13:49.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.682+0000 7f3c5ea17700 20 client.48157.objecter put_session s=0x7f3c4400ca70 osd=5 3 2024-04-14T15:13:49.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.682+0000 7f3c5ea17700 20 client.48157.objecter _get_session s=0x7f3c4400ca70 osd=5 3 2024-04-14T15:13:49.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.682+0000 7f3c5ea17700 15 client.48157.objecter _session_command_op_remove -1 11 2024-04-14T15:13:49.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.682+0000 7f3c5ea17700 20 client.48157.objecter get_session s=0x7f3c4400ca70 osd=5 3 2024-04-14T15:13:49.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.682+0000 7f3c5ea17700 15 client.48157.objecter _session_command_op_assign 5 11 2024-04-14T15:13:49.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.682+0000 7f3c5ea17700 20 client.48157.objecter put_session s=0x7f3c4400ca70 osd=5 4 2024-04-14T15:13:49.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.682+0000 7f3c5ea17700 10 client.48157.objecter _send_command 11 2024-04-14T15:13:49.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.682+0000 7f3c5ea17700 1 -- 172.21.15.90:0/2640457140 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f3c4401e6f0 con 0x7f3c4401ab70 2024-04-14T15:13:49.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.683+0000 7f3c557fa700 10 client.48157.objecter ms_handle_connect 0x7f3c4401ab70 2024-04-14T15:13:49.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.684+0000 7f3c557fa700 1 -- 172.21.15.90:0/2640457140 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f3c4401e6f0 con 0x7f3c4401ab70 2024-04-14T15:13:49.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.684+0000 7f3c557fa700 10 client.48157.objecter ms_dispatch 0x7f3c58001550 command_reply(tid 11: 0 ) 2024-04-14T15:13:49.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.684+0000 7f3c557fa700 10 client.48157.objecter _finish_command 11 = system:0 2024-04-14T15:13:49.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.684+0000 7f3c557fa700 20 client.48157.objecter put_session s=0x7f3c4400ca70 osd=5 4 2024-04-14T15:13:49.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.684+0000 7f3c557fa700 15 client.48157.objecter _session_command_op_remove 5 11 2024-04-14T15:13:49.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.700+0000 7f3c5ea17700 10 client.48157.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:13:49.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.700+0000 7f3c5ea17700 15 client.48157.objecter _session_command_op_assign -1 12 2024-04-14T15:13:49.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.700+0000 7f3c5ea17700 20 client.48157.objecter _get_session s=0x7f3c4400ca70 osd=5 3 2024-04-14T15:13:49.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.700+0000 7f3c5ea17700 20 client.48157.objecter put_session s=0x7f3c4400ca70 osd=5 3 2024-04-14T15:13:49.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.700+0000 7f3c5ea17700 20 client.48157.objecter _get_session s=0x7f3c4400ca70 osd=5 3 2024-04-14T15:13:49.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.700+0000 7f3c5ea17700 15 client.48157.objecter _session_command_op_remove -1 12 2024-04-14T15:13:49.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.700+0000 7f3c5ea17700 20 client.48157.objecter get_session s=0x7f3c4400ca70 osd=5 3 2024-04-14T15:13:49.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.700+0000 7f3c5ea17700 15 client.48157.objecter _session_command_op_assign 5 12 2024-04-14T15:13:49.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.700+0000 7f3c5ea17700 20 client.48157.objecter put_session s=0x7f3c4400ca70 osd=5 4 2024-04-14T15:13:49.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.700+0000 7f3c5ea17700 10 client.48157.objecter _send_command 12 2024-04-14T15:13:49.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.700+0000 7f3c5ea17700 1 -- 172.21.15.90:0/2640457140 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f3c440200a0 con 0x7f3c4401ab70 2024-04-14T15:13:49.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.702+0000 7f3c557fa700 1 -- 172.21.15.90:0/2640457140 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f3c440200a0 con 0x7f3c4401ab70 2024-04-14T15:13:49.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.702+0000 7f3c557fa700 10 client.48157.objecter ms_dispatch 0x7f3c58001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:13:49.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.702+0000 7f3c557fa700 10 client.48157.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:13:49.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.702+0000 7f3c557fa700 20 client.48157.objecter put_session s=0x7f3c4400ca70 osd=5 4 2024-04-14T15:13:49.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.702+0000 7f3c557fa700 15 client.48157.objecter _session_command_op_remove 5 12 2024-04-14T15:13:49.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.702+0000 7f3c5ea17700 10 client.48157.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:13:49.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.702+0000 7f3c5ea17700 15 client.48157.objecter _session_command_op_assign -1 13 2024-04-14T15:13:49.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.702+0000 7f3c5ea17700 20 client.48157.objecter _get_session s=0x7f3c440200a0 osd=6 3 2024-04-14T15:13:49.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.702+0000 7f3c5ea17700 20 client.48157.objecter put_session s=0x7f3c440200a0 osd=6 3 2024-04-14T15:13:49.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.702+0000 7f3c5ea17700 20 client.48157.objecter _get_session s=0x7f3c440200a0 osd=6 3 2024-04-14T15:13:49.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.702+0000 7f3c5ea17700 15 client.48157.objecter _session_command_op_remove -1 13 2024-04-14T15:13:49.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.702+0000 7f3c5ea17700 20 client.48157.objecter get_session s=0x7f3c440200a0 osd=6 3 2024-04-14T15:13:49.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.702+0000 7f3c5ea17700 15 client.48157.objecter _session_command_op_assign 6 13 2024-04-14T15:13:49.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.702+0000 7f3c5ea17700 20 client.48157.objecter put_session s=0x7f3c440200a0 osd=6 4 2024-04-14T15:13:49.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.702+0000 7f3c5ea17700 10 client.48157.objecter _send_command 13 2024-04-14T15:13:49.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.702+0000 7f3c5ea17700 1 -- 172.21.15.90:0/2640457140 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f3c44020ce0 con 0x7f3c440207c0 2024-04-14T15:13:49.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.703+0000 7f3c557fa700 10 client.48157.objecter ms_handle_connect 0x7f3c440207c0 2024-04-14T15:13:49.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.703+0000 7f3c557fa700 1 -- 172.21.15.90:0/2640457140 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f3c44020ce0 con 0x7f3c440207c0 2024-04-14T15:13:49.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.703+0000 7f3c557fa700 10 client.48157.objecter ms_dispatch 0x7f3c58001550 command_reply(tid 13: 0 ) 2024-04-14T15:13:49.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.703+0000 7f3c557fa700 10 client.48157.objecter _finish_command 13 = system:0 2024-04-14T15:13:49.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.703+0000 7f3c557fa700 20 client.48157.objecter put_session s=0x7f3c440200a0 osd=6 4 2024-04-14T15:13:49.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.703+0000 7f3c557fa700 15 client.48157.objecter _session_command_op_remove 6 13 2024-04-14T15:13:49.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.719+0000 7f3c5ea17700 10 client.48157.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:13:49.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.719+0000 7f3c5ea17700 15 client.48157.objecter _session_command_op_assign -1 14 2024-04-14T15:13:49.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.719+0000 7f3c5ea17700 20 client.48157.objecter _get_session s=0x7f3c440200a0 osd=6 3 2024-04-14T15:13:49.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.719+0000 7f3c5ea17700 20 client.48157.objecter put_session s=0x7f3c440200a0 osd=6 3 2024-04-14T15:13:49.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.719+0000 7f3c5ea17700 20 client.48157.objecter _get_session s=0x7f3c440200a0 osd=6 3 2024-04-14T15:13:49.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.719+0000 7f3c5ea17700 15 client.48157.objecter _session_command_op_remove -1 14 2024-04-14T15:13:49.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.719+0000 7f3c5ea17700 20 client.48157.objecter get_session s=0x7f3c440200a0 osd=6 3 2024-04-14T15:13:49.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.719+0000 7f3c5ea17700 15 client.48157.objecter _session_command_op_assign 6 14 2024-04-14T15:13:49.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.719+0000 7f3c5ea17700 20 client.48157.objecter put_session s=0x7f3c440200a0 osd=6 4 2024-04-14T15:13:49.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.719+0000 7f3c5ea17700 10 client.48157.objecter _send_command 14 2024-04-14T15:13:49.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.719+0000 7f3c5ea17700 1 -- 172.21.15.90:0/2640457140 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f3c44020ce0 con 0x7f3c440207c0 2024-04-14T15:13:49.746 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:13:49.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.721+0000 7f3c557fa700 1 -- 172.21.15.90:0/2640457140 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f3c44020ce0 con 0x7f3c440207c0 2024-04-14T15:13:49.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.721+0000 7f3c557fa700 10 client.48157.objecter ms_dispatch 0x7f3c58001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:13:49.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.721+0000 7f3c557fa700 10 client.48157.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:13:49.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.721+0000 7f3c557fa700 20 client.48157.objecter put_session s=0x7f3c440200a0 osd=6 4 2024-04-14T15:13:49.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.721+0000 7f3c557fa700 15 client.48157.objecter _session_command_op_remove 6 14 2024-04-14T15:13:49.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.721+0000 7f3c5ea17700 10 client.48157.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:13:49.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.721+0000 7f3c5ea17700 15 client.48157.objecter _session_command_op_assign -1 15 2024-04-14T15:13:49.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.721+0000 7f3c5ea17700 20 client.48157.objecter _get_session s=0x7f3c44020ce0 osd=7 3 2024-04-14T15:13:49.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.721+0000 7f3c5ea17700 20 client.48157.objecter put_session s=0x7f3c44020ce0 osd=7 3 2024-04-14T15:13:49.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.721+0000 7f3c5ea17700 20 client.48157.objecter _get_session s=0x7f3c44020ce0 osd=7 3 2024-04-14T15:13:49.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.721+0000 7f3c5ea17700 15 client.48157.objecter _session_command_op_remove -1 15 2024-04-14T15:13:49.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.721+0000 7f3c5ea17700 20 client.48157.objecter get_session s=0x7f3c44020ce0 osd=7 3 2024-04-14T15:13:49.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.721+0000 7f3c5ea17700 15 client.48157.objecter _session_command_op_assign 7 15 2024-04-14T15:13:49.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.721+0000 7f3c5ea17700 20 client.48157.objecter put_session s=0x7f3c44020ce0 osd=7 4 2024-04-14T15:13:49.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.721+0000 7f3c5ea17700 10 client.48157.objecter _send_command 15 2024-04-14T15:13:49.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.721+0000 7f3c5ea17700 1 -- 172.21.15.90:0/2640457140 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f3c440272c0 con 0x7f3c440237b0 2024-04-14T15:13:49.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.722+0000 7f3c557fa700 10 client.48157.objecter ms_handle_connect 0x7f3c440237b0 2024-04-14T15:13:49.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.723+0000 7f3c557fa700 1 -- 172.21.15.90:0/2640457140 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f3c440272c0 con 0x7f3c440237b0 2024-04-14T15:13:49.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.723+0000 7f3c557fa700 10 client.48157.objecter ms_dispatch 0x7f3c58001550 command_reply(tid 15: 0 ) 2024-04-14T15:13:49.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.723+0000 7f3c557fa700 10 client.48157.objecter _finish_command 15 = system:0 2024-04-14T15:13:49.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.723+0000 7f3c557fa700 20 client.48157.objecter put_session s=0x7f3c44020ce0 osd=7 4 2024-04-14T15:13:49.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.723+0000 7f3c557fa700 15 client.48157.objecter _session_command_op_remove 7 15 2024-04-14T15:13:49.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.739+0000 7f3c5ea17700 10 client.48157.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:13:49.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.739+0000 7f3c5ea17700 15 client.48157.objecter _session_command_op_assign -1 16 2024-04-14T15:13:49.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.739+0000 7f3c5ea17700 20 client.48157.objecter _get_session s=0x7f3c44020ce0 osd=7 3 2024-04-14T15:13:49.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.739+0000 7f3c5ea17700 20 client.48157.objecter put_session s=0x7f3c44020ce0 osd=7 3 2024-04-14T15:13:49.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.739+0000 7f3c5ea17700 20 client.48157.objecter _get_session s=0x7f3c44020ce0 osd=7 3 2024-04-14T15:13:49.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.739+0000 7f3c5ea17700 15 client.48157.objecter _session_command_op_remove -1 16 2024-04-14T15:13:49.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.739+0000 7f3c5ea17700 20 client.48157.objecter get_session s=0x7f3c44020ce0 osd=7 3 2024-04-14T15:13:49.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.739+0000 7f3c5ea17700 15 client.48157.objecter _session_command_op_assign 7 16 2024-04-14T15:13:49.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.739+0000 7f3c5ea17700 20 client.48157.objecter put_session s=0x7f3c44020ce0 osd=7 4 2024-04-14T15:13:49.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.739+0000 7f3c5ea17700 10 client.48157.objecter _send_command 16 2024-04-14T15:13:49.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.739+0000 7f3c5ea17700 1 -- 172.21.15.90:0/2640457140 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f3c440272c0 con 0x7f3c440237b0 2024-04-14T15:13:49.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.740+0000 7f3c557fa700 1 -- 172.21.15.90:0/2640457140 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f3c440272c0 con 0x7f3c440237b0 2024-04-14T15:13:49.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.740+0000 7f3c557fa700 10 client.48157.objecter ms_dispatch 0x7f3c58001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:13:49.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.740+0000 7f3c557fa700 10 client.48157.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:13:49.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.740+0000 7f3c557fa700 20 client.48157.objecter put_session s=0x7f3c44020ce0 osd=7 4 2024-04-14T15:13:49.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.740+0000 7f3c557fa700 15 client.48157.objecter _session_command_op_remove 7 16 2024-04-14T15:13:49.750 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:13:49.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.741+0000 7f3c5ea17700 10 librados: watch_flush enter 2024-04-14T15:13:49.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.741+0000 7f3c5ea17700 10 librados: watch_flush exit 2024-04-14T15:13:49.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.741+0000 7f3c5ea17700 10 client.48157.objecter close_session for osd.0 2024-04-14T15:13:49.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.741+0000 7f3c5ea17700 1 -- 172.21.15.90:0/2640457140 >> v1:172.21.15.90:6801/3761806885 conn(0x7f3c44004730 legacy=0x7f3c44002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:49.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.741+0000 7f3c5ea17700 20 client.48157.objecter put_session s=0x7f3c440053e0 osd=0 1 2024-04-14T15:13:49.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.741+0000 7f3c5ea17700 10 client.48157.objecter close_session for osd.1 2024-04-14T15:13:49.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.741+0000 7f3c5ea17700 1 -- 172.21.15.90:0/2640457140 >> v1:172.21.15.90:6805/4170361699 conn(0x7f3c44004280 legacy=0x7f3c4400ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:49.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.741+0000 7f3c5ea17700 20 client.48157.objecter put_session s=0x7f3c44004110 osd=1 1 2024-04-14T15:13:49.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.741+0000 7f3c5ea17700 10 client.48157.objecter close_session for osd.2 2024-04-14T15:13:49.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.741+0000 7f3c5ea17700 1 -- 172.21.15.90:0/2640457140 >> v1:172.21.15.90:6809/1686184251 conn(0x7f3c44008880 legacy=0x7f3c4400f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:49.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.741+0000 7f3c5ea17700 20 client.48157.objecter put_session s=0x7f3c44001a00 osd=2 1 2024-04-14T15:13:49.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.741+0000 7f3c5ea17700 10 client.48157.objecter close_session for osd.3 2024-04-14T15:13:49.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.741+0000 7f3c5ea17700 1 -- 172.21.15.90:0/2640457140 >> v1:172.21.15.90:6813/2270166774 conn(0x7f3c4400d130 legacy=0x7f3c44011100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:49.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.741+0000 7f3c5ea17700 20 client.48157.objecter put_session s=0x7f3c44007df0 osd=3 1 2024-04-14T15:13:49.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.741+0000 7f3c5ea17700 10 client.48157.objecter close_session for osd.4 2024-04-14T15:13:49.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.741+0000 7f3c5ea17700 1 -- 172.21.15.90:0/2640457140 >> v1:172.21.15.92:6800/3887549733 conn(0x7f3c440135b0 legacy=0x7f3c44015cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:49.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.741+0000 7f3c5ea17700 20 client.48157.objecter put_session s=0x7f3c4400c450 osd=4 1 2024-04-14T15:13:49.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.741+0000 7f3c5ea17700 10 client.48157.objecter close_session for osd.5 2024-04-14T15:13:49.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.741+0000 7f3c5ea17700 1 -- 172.21.15.90:0/2640457140 >> v1:172.21.15.92:6804/3244561440 conn(0x7f3c4401ab70 legacy=0x7f3c4401cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:49.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.741+0000 7f3c5cfb4700 1 -- 172.21.15.90:0/2640457140 reap_dead start 2024-04-14T15:13:49.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.741+0000 7f3c5ea17700 20 client.48157.objecter put_session s=0x7f3c4400ca70 osd=5 1 2024-04-14T15:13:49.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.741+0000 7f3c5ea17700 10 client.48157.objecter close_session for osd.6 2024-04-14T15:13:49.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.741+0000 7f3c5ea17700 1 -- 172.21.15.90:0/2640457140 >> v1:172.21.15.92:6808/636125963 conn(0x7f3c440207c0 legacy=0x7f3c4401e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:49.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.741+0000 7f3c5cfb4700 1 -- 172.21.15.90:0/2640457140 reap_dead start 2024-04-14T15:13:49.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.741+0000 7f3c5cfb4700 1 -- 172.21.15.90:0/2640457140 reap_dead start 2024-04-14T15:13:49.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.741+0000 7f3c5cfb4700 1 -- 172.21.15.90:0/2640457140 reap_dead start 2024-04-14T15:13:49.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.741+0000 7f3c5ea17700 20 client.48157.objecter put_session s=0x7f3c440200a0 osd=6 1 2024-04-14T15:13:49.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.741+0000 7f3c5ea17700 10 client.48157.objecter close_session for osd.7 2024-04-14T15:13:49.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.741+0000 7f3c5ea17700 1 -- 172.21.15.90:0/2640457140 >> v1:172.21.15.92:6812/1773706716 conn(0x7f3c440237b0 legacy=0x7f3c44025ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:49.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.741+0000 7f3c5ea17700 20 client.48157.objecter put_session s=0x7f3c44020ce0 osd=7 1 2024-04-14T15:13:49.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.741+0000 7f3c5ea17700 20 client.48157.objecter shutdown clearing up homeless session... 2024-04-14T15:13:49.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.741+0000 7f3c5ea17700 10 client.48157.objecter successfully canceled tick 2024-04-14T15:13:49.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.741+0000 7f3c5ea17700 1 -- 172.21.15.90:0/2640457140 >> v1:172.21.15.90:6800/420935427 conn(0x7f3c40071580 legacy=0x7f3c40073a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:49.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.741+0000 7f3c5ea17700 1 -- 172.21.15.90:0/2640457140 >> v1:172.21.15.92:6789/0 conn(0x7f3c58101630 legacy=0x7f3c5810fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:49.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.741+0000 7f3c5ea17700 1 -- 172.21.15.90:0/2640457140 shutdown_connections 2024-04-14T15:13:49.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.741+0000 7f3c5ea17700 1 -- 172.21.15.90:0/2640457140 >> 172.21.15.90:0/2640457140 conn(0x7f3c580fd2b0 msgr2=0x7f3c580ff6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:13:49.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.741+0000 7f3c5ea17700 1 -- 172.21.15.90:0/2640457140 shutdown_connections 2024-04-14T15:13:49.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.741+0000 7f3c5ea17700 1 -- 172.21.15.90:0/2640457140 wait complete. 2024-04-14T15:13:49.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:49.741+0000 7f3c5ea17700 1 librados: shutdown 2024-04-14T15:13:49.754 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:13:49.754 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:13:49.755 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:13:49.755 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:13:49.755 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:13:49.755 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:13:49.755 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:13:49.755 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:13:49.768 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:49.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:49.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:13:49.848 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots: Running main() from gmock_main.cc 2024-04-14T15:13:49.848 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots: [==========] Running 13 tests from 4 test suites. 2024-04-14T15:13:49.848 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots: [----------] Global test environment set-up. 2024-04-14T15:13:49.848 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2024-04-14T15:13:49.848 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2024-04-14T15:13:49.848 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (2009 ms) 2024-04-14T15:13:49.848 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2024-04-14T15:13:49.848 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (2024 ms) 2024-04-14T15:13:49.849 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2024-04-14T15:13:49.849 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (1709 ms) 2024-04-14T15:13:49.849 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2024-04-14T15:13:49.849 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (2005 ms) 2024-04-14T15:13:49.849 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (7747 ms total) 2024-04-14T15:13:49.849 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots: 2024-04-14T15:13:49.849 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged 2024-04-14T15:13:49.849 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2024-04-14T15:13:49.849 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (4096 ms) 2024-04-14T15:13:49.849 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2024-04-14T15:13:49.849 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (4078 ms) 2024-04-14T15:13:49.849 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.FutureSnapRollback 2024-04-14T15:13:49.849 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.FutureSnapRollback (4943 ms) 2024-04-14T15:13:49.849 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged (13117 ms total) 2024-04-14T15:13:49.850 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots: 2024-04-14T15:13:49.850 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2024-04-14T15:13:49.850 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2024-04-14T15:13:49.850 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (2990 ms) 2024-04-14T15:13:49.850 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2024-04-14T15:13:49.850 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (2030 ms) 2024-04-14T15:13:49.850 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2024-04-14T15:13:49.850 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (1973 ms) 2024-04-14T15:13:49.850 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2024-04-14T15:13:49.850 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (2009 ms) 2024-04-14T15:13:49.850 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (9002 ms total) 2024-04-14T15:13:49.850 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots: 2024-04-14T15:13:49.850 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2024-04-14T15:13:49.850 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2024-04-14T15:13:49.851 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (4019 ms) 2024-04-14T15:13:49.851 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2024-04-14T15:13:49.851 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (4011 ms) 2024-04-14T15:13:49.851 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (8030 ms total) 2024-04-14T15:13:49.851 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots: 2024-04-14T15:13:49.851 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots: [----------] Global test environment tear-down 2024-04-14T15:13:49.851 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots: [==========] 13 tests from 4 test suites ran. (55583 ms total) 2024-04-14T15:13:49.851 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots: [ PASSED ] 13 tests. 2024-04-14T15:13:49.870 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:13:49.902 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-14T15:13:49.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:49.904 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:49.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:13:50.005 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:13:50.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:50.030 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:50.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:13:50.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:49 smithi092 ceph-mon[33966]: from='client.48067 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi090-93709-15"}]': finished 2024-04-14T15:13:50.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:49 smithi092 ceph-mon[33966]: osdmap e161: 8 total, 8 up, 8 in 2024-04-14T15:13:50.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:49 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:50.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:49 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2918742503' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi090-93709-15"}]: dispatch 2024-04-14T15:13:50.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:49 smithi092 ceph-mon[33966]: from='client.48067 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi090-93709-15"}]: dispatch 2024-04-14T15:13:50.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:49 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3120752525' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi090-93363-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:50.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:49 smithi092 ceph-mon[33966]: from='client.48145 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi090-93363-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:50.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:49 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/290003185' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi090-93342-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:50.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:49 smithi092 ceph-mon[33966]: from='client.48148 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi090-93342-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:50.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:49 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-04-14T15:13:50.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:49 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:13:50.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:49 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:13:50.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:49 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2640457140' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:13:50.132 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:13:50.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:50.157 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:50.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:13:50.182 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-14T15:13:50.182 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-14T15:13:50.182 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-04-14T15:13:50.183 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd dump -f json-pretty 2024-04-14T15:13:50.186 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:13:50.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:49 smithi090 ceph-mon[41670]: from='client.48067 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi090-93709-15"}]': finished 2024-04-14T15:13:50.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:49 smithi090 ceph-mon[41670]: osdmap e161: 8 total, 8 up, 8 in 2024-04-14T15:13:50.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:49 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:50.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:49 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2918742503' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi090-93709-15"}]: dispatch 2024-04-14T15:13:50.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:49 smithi090 ceph-mon[41670]: from='client.48067 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi090-93709-15"}]: dispatch 2024-04-14T15:13:50.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:49 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3120752525' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi090-93363-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:50.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:49 smithi090 ceph-mon[41670]: from='client.48145 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi090-93363-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:50.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:49 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/290003185' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi090-93342-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:50.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:49 smithi090 ceph-mon[41670]: from='client.48148 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi090-93342-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:50.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:49 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-04-14T15:13:50.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:49 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:13:50.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:49 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:13:50.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:49 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2640457140' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:13:50.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:49 smithi090 ceph-mon[30399]: from='client.48067 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi090-93709-15"}]': finished 2024-04-14T15:13:50.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:49 smithi090 ceph-mon[30399]: osdmap e161: 8 total, 8 up, 8 in 2024-04-14T15:13:50.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:49 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:50.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:49 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2918742503' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi090-93709-15"}]: dispatch 2024-04-14T15:13:50.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:49 smithi090 ceph-mon[30399]: from='client.48067 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi090-93709-15"}]: dispatch 2024-04-14T15:13:50.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:49 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3120752525' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi090-93363-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:50.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:49 smithi090 ceph-mon[30399]: from='client.48145 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi090-93363-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:50.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:49 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/290003185' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi090-93342-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:50.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:49 smithi090 ceph-mon[30399]: from='client.48148 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi090-93342-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:50.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:49 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-04-14T15:13:50.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:49 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:13:50.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:49 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:13:50.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:49 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2640457140' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:13:50.259 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:13:50.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:50.284 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:50.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:13:50.385 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:13:50.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:50.409 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:50.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:13:50.510 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:13:50.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:50.535 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:50.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:13:50.605 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:13:50.636 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:13:50.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:50.661 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:50.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:13:50.702 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd unset noscrub 2024-04-14T15:13:50.762 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:13:50.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:50.786 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:50.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:13:50.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:50 smithi092 ceph-mon[33966]: pgmap v188: 388 pgs: 37 creating+peering, 27 unknown, 324 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 27 KiB/s rd, 34 op/s; 191 B/s, 2 objects/s recovering 2024-04-14T15:13:50.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:50 smithi092 ceph-mon[33966]: from='client.48067 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi090-93709-15"}]': finished 2024-04-14T15:13:50.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:50 smithi092 ceph-mon[33966]: from='client.48145 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi090-93363-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:50.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:50 smithi092 ceph-mon[33966]: from='client.48148 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi090-93342-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:50.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:50 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "31"}]': finished 2024-04-14T15:13:50.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:50 smithi092 ceph-mon[33966]: osdmap e162: 8 total, 8 up, 8 in 2024-04-14T15:13:50.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:50 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:50.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:50 smithi092 ceph-mon[33966]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:13:50.887 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:13:50.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:50.912 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:50.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:13:51.012 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:13:51.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:51.039 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:51.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:13:51.132 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:50 smithi090 ceph-mon[41670]: pgmap v188: 388 pgs: 37 creating+peering, 27 unknown, 324 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 27 KiB/s rd, 34 op/s; 191 B/s, 2 objects/s recovering 2024-04-14T15:13:51.132 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:50 smithi090 ceph-mon[41670]: from='client.48067 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi090-93709-15"}]': finished 2024-04-14T15:13:51.132 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:50 smithi090 ceph-mon[41670]: from='client.48145 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi090-93363-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:51.132 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:50 smithi090 ceph-mon[41670]: from='client.48148 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi090-93342-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:51.133 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:50 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "31"}]': finished 2024-04-14T15:13:51.133 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:50 smithi090 ceph-mon[41670]: osdmap e162: 8 total, 8 up, 8 in 2024-04-14T15:13:51.133 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:50 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:51.133 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:50 smithi090 ceph-mon[41670]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:13:51.133 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:50 smithi090 ceph-mon[30399]: pgmap v188: 388 pgs: 37 creating+peering, 27 unknown, 324 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 27 KiB/s rd, 34 op/s; 191 B/s, 2 objects/s recovering 2024-04-14T15:13:51.133 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:50 smithi090 ceph-mon[30399]: from='client.48067 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi090-93709-15"}]': finished 2024-04-14T15:13:51.133 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:50 smithi090 ceph-mon[30399]: from='client.48145 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi090-93363-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:51.133 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:50 smithi090 ceph-mon[30399]: from='client.48148 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi090-93342-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:51.133 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:50 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "31"}]': finished 2024-04-14T15:13:51.133 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:50 smithi090 ceph-mon[30399]: osdmap e162: 8 total, 8 up, 8 in 2024-04-14T15:13:51.133 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:50 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:51.133 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:50 smithi090 ceph-mon[30399]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:13:51.140 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:13:51.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:51.164 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:51.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:13:51.265 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:13:51.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:51.290 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:51.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:13:51.391 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:13:51.415 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:51.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:51.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:13:51.517 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:13:51.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:51.543 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:51.543 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:13:51.644 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:13:51.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:51.669 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:51.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:13:51.770 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:13:51.794 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:51.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:51.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:13:51.896 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:13:51.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:51.920 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:51.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:13:51.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:51 smithi090 ceph-mon[41670]: osdmap e163: 8 total, 8 up, 8 in 2024-04-14T15:13:51.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:51 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:51.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:51 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1563685096' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93724-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:51.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:51 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1563685096' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93724-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:51.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:51 smithi090 ceph-mon[41670]: osdmap e164: 8 total, 8 up, 8 in 2024-04-14T15:13:51.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:51 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:51.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:51 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3379463312' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi090-93342-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:51.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:51 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3040281885' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi090-93363-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:51.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:51 smithi090 ceph-mon[41670]: from='client.48169 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi090-93342-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:51.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:51 smithi090 ceph-mon[41670]: from='client.48175 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi090-93363-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:51.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:51 smithi090 ceph-mon[30399]: osdmap e163: 8 total, 8 up, 8 in 2024-04-14T15:13:51.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:51 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:51.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:51 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1563685096' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93724-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:51.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:51 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1563685096' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93724-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:51.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:51 smithi090 ceph-mon[30399]: osdmap e164: 8 total, 8 up, 8 in 2024-04-14T15:13:51.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:51 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:51.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:51 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3379463312' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi090-93342-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:51.944 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:51 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3040281885' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi090-93363-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:51.944 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:51 smithi090 ceph-mon[30399]: from='client.48169 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi090-93342-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:51.944 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:51 smithi090 ceph-mon[30399]: from='client.48175 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi090-93363-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:51.999 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-04-14T15:13:51.999 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-04-14T15:13:51.999 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-04-14T15:13:52.000 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-04-14T15:13:52.000 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-04-14T15:13:52.021 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:13:52.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:52.048 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:52.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:13:52.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:51 smithi092 ceph-mon[33966]: osdmap e163: 8 total, 8 up, 8 in 2024-04-14T15:13:52.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:51 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:52.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:51 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1563685096' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93724-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:52.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:51 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1563685096' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93724-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:52.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:51 smithi092 ceph-mon[33966]: osdmap e164: 8 total, 8 up, 8 in 2024-04-14T15:13:52.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:51 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:52.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:51 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3379463312' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi090-93342-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:52.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:51 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3040281885' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi090-93363-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:52.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:51 smithi092 ceph-mon[33966]: from='client.48169 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi090-93342-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:52.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:51 smithi092 ceph-mon[33966]: from='client.48175 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi090-93363-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:52.149 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:13:52.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:52.174 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:52.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:13:52.275 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:13:52.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:52.299 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:52.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:13:52.400 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:13:52.424 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:52.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:52.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:13:52.526 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:13:52.528 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-14T15:13:52.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.531+0000 7fb69d34c700 1 -- 172.21.15.90:0/2433764046 >> v1:172.21.15.90:6789/0 conn(0x7fb69810b890 legacy=0x7fb69810dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:52.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.531+0000 7fb69d34c700 1 -- 172.21.15.90:0/2433764046 shutdown_connections 2024-04-14T15:13:52.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.531+0000 7fb69d34c700 1 -- 172.21.15.90:0/2433764046 >> 172.21.15.90:0/2433764046 conn(0x7fb6980fd290 msgr2=0x7fb6980ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:13:52.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.531+0000 7fb69d34c700 1 -- 172.21.15.90:0/2433764046 shutdown_connections 2024-04-14T15:13:52.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.531+0000 7fb69d34c700 1 -- 172.21.15.90:0/2433764046 wait complete. 2024-04-14T15:13:52.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.531+0000 7fb69d34c700 1 librados: starting msgr at 2024-04-14T15:13:52.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.531+0000 7fb69d34c700 1 librados: starting objecter 2024-04-14T15:13:52.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.532+0000 7fb69d34c700 1 Processor -- start 2024-04-14T15:13:52.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.532+0000 7fb69d34c700 1 -- start start 2024-04-14T15:13:52.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.532+0000 7fb69d34c700 1 librados: setting wanted keys 2024-04-14T15:13:52.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.532+0000 7fb69d34c700 1 librados: calling monclient init 2024-04-14T15:13:52.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.532+0000 7fb69d34c700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb6981a77d0 con 0x7fb69810f3a0 2024-04-14T15:13:52.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.532+0000 7fb69d34c700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb6981a89f0 con 0x7fb698103a80 2024-04-14T15:13:52.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.532+0000 7fb69d34c700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb6981a9c10 con 0x7fb69810b890 2024-04-14T15:13:52.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.532+0000 7fb6967fc700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7fb69810b890 0x7fb6981a2780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:53832/0 (socket says 172.21.15.90:53832) 2024-04-14T15:13:52.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.532+0000 7fb6967fc700 1 -- 172.21.15.90:0/2370564494 learned_addr learned my addr 172.21.15.90:0/2370564494 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:13:52.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.532+0000 7fb68ffff700 10 client.?.objecter ms_handle_connect 0x7fb69810f3a0 2024-04-14T15:13:52.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.532+0000 7fb68ffff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:13:52.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.532+0000 7fb68ffff700 10 client.?.objecter ms_handle_connect 0x7fb69810b890 2024-04-14T15:13:52.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.532+0000 7fb68ffff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:13:52.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.532+0000 7fb68ffff700 10 client.?.objecter ms_handle_connect 0x7fb698103a80 2024-04-14T15:13:52.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.532+0000 7fb68ffff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:13:52.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.533+0000 7fb68ffff700 1 -- 172.21.15.90:0/2370564494 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 901101735 0 0) 0x7fb6981a77d0 con 0x7fb69810f3a0 2024-04-14T15:13:52.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.533+0000 7fb68ffff700 1 -- 172.21.15.90:0/2370564494 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6840031f0 con 0x7fb69810f3a0 2024-04-14T15:13:52.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.533+0000 7fb68ffff700 1 -- 172.21.15.90:0/2370564494 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3461743993 0 0) 0x7fb6981a89f0 con 0x7fb698103a80 2024-04-14T15:13:52.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.533+0000 7fb68ffff700 1 -- 172.21.15.90:0/2370564494 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6981a77d0 con 0x7fb698103a80 2024-04-14T15:13:52.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.533+0000 7fb68ffff700 1 -- 172.21.15.90:0/2370564494 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1893164763 0 0) 0x7fb6981a9c10 con 0x7fb69810b890 2024-04-14T15:13:52.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.533+0000 7fb68ffff700 1 -- 172.21.15.90:0/2370564494 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6981a89f0 con 0x7fb69810b890 2024-04-14T15:13:52.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.533+0000 7fb68ffff700 1 -- 172.21.15.90:0/2370564494 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1839407105 0 0) 0x7fb6981a77d0 con 0x7fb698103a80 2024-04-14T15:13:52.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.533+0000 7fb68ffff700 1 -- 172.21.15.90:0/2370564494 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb6981a9c10 con 0x7fb698103a80 2024-04-14T15:13:52.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.533+0000 7fb68ffff700 1 -- 172.21.15.90:0/2370564494 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fb688001bf0 con 0x7fb698103a80 2024-04-14T15:13:52.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.533+0000 7fb68ffff700 1 -- 172.21.15.90:0/2370564494 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3288857832 0 0) 0x7fb6840031f0 con 0x7fb69810f3a0 2024-04-14T15:13:52.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.533+0000 7fb68ffff700 1 -- 172.21.15.90:0/2370564494 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb6981a77d0 con 0x7fb69810f3a0 2024-04-14T15:13:52.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.533+0000 7fb68ffff700 1 -- 172.21.15.90:0/2370564494 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fb690001070 con 0x7fb69810f3a0 2024-04-14T15:13:52.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.533+0000 7fb68ffff700 1 -- 172.21.15.90:0/2370564494 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3582704450 0 0) 0x7fb6981a89f0 con 0x7fb69810b890 2024-04-14T15:13:52.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.533+0000 7fb68ffff700 1 -- 172.21.15.90:0/2370564494 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb6840031f0 con 0x7fb69810b890 2024-04-14T15:13:52.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.533+0000 7fb68ffff700 1 -- 172.21.15.90:0/2370564494 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fb680001830 con 0x7fb69810b890 2024-04-14T15:13:52.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.533+0000 7fb68ffff700 1 -- 172.21.15.90:0/2370564494 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2497958141 0 0) 0x7fb6981a9c10 con 0x7fb698103a80 2024-04-14T15:13:52.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.533+0000 7fb68ffff700 1 -- 172.21.15.90:0/2370564494 >> v1:172.21.15.90:6790/0 conn(0x7fb69810b890 legacy=0x7fb6981a2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:52.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.533+0000 7fb68ffff700 1 -- 172.21.15.90:0/2370564494 >> v1:172.21.15.90:6789/0 conn(0x7fb69810f3a0 legacy=0x7fb6981a5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:52.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.533+0000 7fb68ffff700 1 -- 172.21.15.90:0/2370564494 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6981aae50 con 0x7fb698103a80 2024-04-14T15:13:52.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.533+0000 7fb69d34c700 1 -- 172.21.15.90:0/2370564494 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb6981a8c20 con 0x7fb698103a80 2024-04-14T15:13:52.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.533+0000 7fb69d34c700 10 client.48187.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:13:52.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.533+0000 7fb69d34c700 1 -- 172.21.15.90:0/2370564494 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb6981a90e0 con 0x7fb698103a80 2024-04-14T15:13:52.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.533+0000 7fb69d34c700 1 librados: init done 2024-04-14T15:13:52.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.533+0000 7fb68ffff700 1 -- 172.21.15.90:0/2370564494 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fb688003ea0 con 0x7fb698103a80 2024-04-14T15:13:52.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.533+0000 7fb68ffff700 1 -- 172.21.15.90:0/2370564494 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fb688004e50 con 0x7fb698103a80 2024-04-14T15:13:52.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.534+0000 7fb69d34c700 1 -- 172.21.15.90:0/2370564494 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb69806ace0 con 0x7fb698103a80 2024-04-14T15:13:52.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.535+0000 7fb68ffff700 1 -- 172.21.15.90:0/2370564494 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fb688003a00 con 0x7fb698103a80 2024-04-14T15:13:52.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.535+0000 7fb68ffff700 1 -- 172.21.15.90:0/2370564494 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(164..164 src has 51..164) ==== 10134+0+0 (unknown 3135705362 0 0) 0x7fb68808a4c0 con 0x7fb698103a80 2024-04-14T15:13:52.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.535+0000 7fb68ffff700 10 client.48187.objecter ms_dispatch 0x7fb698001550 osd_map(164..164 src has 51..164) 2024-04-14T15:13:52.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.535+0000 7fb68ffff700 3 client.48187.objecter handle_osd_map got epochs [164,164] > 0 2024-04-14T15:13:52.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.535+0000 7fb68ffff700 3 client.48187.objecter handle_osd_map decoding full epoch 164 2024-04-14T15:13:52.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.535+0000 7fb68ffff700 20 client.48187.objecter dump_active .. 0 homeless 2024-04-14T15:13:52.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.535+0000 7fb68ffff700 10 client.48187.objecter ms_handle_connect 0x7fb684075880 2024-04-14T15:13:52.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.540+0000 7fb68ffff700 1 -- 172.21.15.90:0/2370564494 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fb688056720 con 0x7fb698103a80 2024-04-14T15:13:52.548 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:13:52.549 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:52.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:52.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:13:52.651 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:13:52.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:52.675 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:52.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:13:52.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.754+0000 7fb69d34c700 1 -- 172.21.15.90:0/2370564494 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) -- 0x7fb698069400 con 0x7fb698103a80 2024-04-14T15:13:52.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.758+0000 7fb68ffff700 1 -- 172.21.15.90:0/2370564494 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v164) ==== 81+0+69678 (unknown 3401422657 0 2993348601) 0x7fb698069400 con 0x7fb698103a80 2024-04-14T15:13:52.759 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:13:52.759 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-14T15:13:52.759 INFO:teuthology.orchestra.run.smithi090.stdout: "epoch": 164, 2024-04-14T15:13:52.759 INFO:teuthology.orchestra.run.smithi090.stdout: "fsid": "0e288098-fa70-11ee-bc8e-c7b262605968", 2024-04-14T15:13:52.759 INFO:teuthology.orchestra.run.smithi090.stdout: "created": "2024-04-14T15:04:05.808313+0000", 2024-04-14T15:13:52.759 INFO:teuthology.orchestra.run.smithi090.stdout: "modified": "2024-04-14T15:13:51.830791+0000", 2024-04-14T15:13:52.760 INFO:teuthology.orchestra.run.smithi090.stdout: "last_up_change": "2024-04-14T15:09:29.769352+0000", 2024-04-14T15:13:52.760 INFO:teuthology.orchestra.run.smithi090.stdout: "last_in_change": "2024-04-14T15:09:11.803164+0000", 2024-04-14T15:13:52.760 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-14T15:13:52.760 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_num": 5806080, 2024-04-14T15:13:52.760 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_set": [ 2024-04-14T15:13:52.760 INFO:teuthology.orchestra.run.smithi090.stdout: "nodeep-scrub", 2024-04-14T15:13:52.760 INFO:teuthology.orchestra.run.smithi090.stdout: "noscrub", 2024-04-14T15:13:52.760 INFO:teuthology.orchestra.run.smithi090.stdout: "pglog_hardlimit", 2024-04-14T15:13:52.760 INFO:teuthology.orchestra.run.smithi090.stdout: "purged_snapdirs", 2024-04-14T15:13:52.760 INFO:teuthology.orchestra.run.smithi090.stdout: "recovery_deletes", 2024-04-14T15:13:52.760 INFO:teuthology.orchestra.run.smithi090.stdout: "sortbitwise" 2024-04-14T15:13:52.760 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-14T15:13:52.761 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_version": 34, 2024-04-14T15:13:52.761 INFO:teuthology.orchestra.run.smithi090.stdout: "full_ratio": 0.94999998807907104, 2024-04-14T15:13:52.761 INFO:teuthology.orchestra.run.smithi090.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-14T15:13:52.761 INFO:teuthology.orchestra.run.smithi090.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-14T15:13:52.761 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_snapshot": "", 2024-04-14T15:13:52.761 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_max": 189, 2024-04-14T15:13:52.761 INFO:teuthology.orchestra.run.smithi090.stdout: "max_osd": 8, 2024-04-14T15:13:52.761 INFO:teuthology.orchestra.run.smithi090.stdout: "require_min_compat_client": "luminous", 2024-04-14T15:13:52.761 INFO:teuthology.orchestra.run.smithi090.stdout: "min_compat_client": "luminous", 2024-04-14T15:13:52.761 INFO:teuthology.orchestra.run.smithi090.stdout: "require_osd_release": "squid", 2024-04-14T15:13:52.761 INFO:teuthology.orchestra.run.smithi090.stdout: "allow_crimson": false, 2024-04-14T15:13:52.762 INFO:teuthology.orchestra.run.smithi090.stdout: "pools": [ 2024-04-14T15:13:52.762 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.762 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 1, 2024-04-14T15:13:52.762 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": ".mgr", 2024-04-14T15:13:52.762 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:07:23.959628+0000", 2024-04-14T15:13:52.762 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-14T15:13:52.762 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool", 2024-04-14T15:13:52.762 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:13:52.762 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:13:52.762 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:13:52.762 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:13:52.762 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:13:52.763 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:13:52.763 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:13:52.763 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:13:52.763 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:13:52.763 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:13:52.763 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 1, 2024-04-14T15:13:52.763 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 1, 2024-04-14T15:13:52.763 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 1, 2024-04-14T15:13:52.763 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 1, 2024-04-14T15:13:52.763 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 1, 2024-04-14T15:13:52.763 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:13:52.763 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:13:52.764 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:13:52.764 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:13:52.764 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:13:52.764 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:13:52.764 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:13:52.764 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.764 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "104", 2024-04-14T15:13:52.764 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:13:52.764 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:13:52.765 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:13:52.765 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:13:52.765 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:13:52.765 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:13:52.765 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:13:52.766 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:13:52.766 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:13:52.766 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:13:52.766 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:13:52.766 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:13:52.766 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:13:52.766 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:13:52.766 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:13:52.767 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:13:52.767 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:13:52.767 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:13:52.767 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:13:52.767 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:13:52.767 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:13:52.767 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:13:52.767 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:13:52.767 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:13:52.767 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:13:52.767 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:13:52.767 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.768 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:13:52.768 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:13:52.768 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:13:52.768 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:13:52.768 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:13:52.768 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:13:52.768 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:13:52.768 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:13:52.768 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:13:52.768 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:13:52.768 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:13:52.768 INFO:teuthology.orchestra.run.smithi090.stdout: "options": { 2024-04-14T15:13:52.769 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_max": 32, 2024-04-14T15:13:52.769 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_min": 1 2024-04-14T15:13:52.769 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.769 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:13:52.769 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": {} 2024-04-14T15:13:52.769 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.769 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:13:52.769 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:13:52.769 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 7.8899998664855957, 2024-04-14T15:13:52.769 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 7.8899998664855957, 2024-04-14T15:13:52.770 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 0.37999999523162842, 2024-04-14T15:13:52.770 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 3, 2024-04-14T15:13:52.770 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 3, 2024-04-14T15:13:52.770 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.90223121643066406, 2024-04-14T15:13:52.770 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 0.89999997615814209, 2024-04-14T15:13:52.770 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-04-14T15:13:52.770 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.770 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.770 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.770 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 2, 2024-04-14T15:13:52.770 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": ".rgw.root", 2024-04-14T15:13:52.770 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:09:41.608420+0000", 2024-04-14T15:13:52.771 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-14T15:13:52.771 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool", 2024-04-14T15:13:52.771 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:13:52.771 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:13:52.771 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:13:52.771 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:13:52.771 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:13:52.771 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:13:52.771 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:13:52.771 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:13:52.771 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:13:52.771 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:13:52.772 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:13:52.772 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:13:52.772 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:13:52.772 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:13:52.772 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:13:52.772 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:13:52.772 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:13:52.772 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:13:52.772 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:13:52.772 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:13:52.772 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:13:52.772 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:13:52.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.773 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "66", 2024-04-14T15:13:52.773 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:13:52.773 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:13:52.773 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:13:52.773 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:13:52.773 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:13:52.773 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:13:52.773 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:13:52.773 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:13:52.773 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:13:52.773 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:13:52.774 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:13:52.774 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:13:52.774 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:13:52.774 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:13:52.774 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:13:52.774 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:13:52.774 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:13:52.774 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:13:52.774 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:13:52.774 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:13:52.774 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:13:52.774 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:13:52.775 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:13:52.775 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:13:52.775 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:13:52.775 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:13:52.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.775 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:13:52.775 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:13:52.775 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:13:52.775 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:13:52.775 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:13:52.775 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:13:52.775 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:13:52.775 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:13:52.776 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:13:52.776 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:13:52.776 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:13:52.776 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:13:52.776 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:13:52.776 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw": {} 2024-04-14T15:13:52.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.777 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:13:52.777 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:13:52.777 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 1.5, 2024-04-14T15:13:52.777 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 1.5, 2024-04-14T15:13:52.777 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:13:52.777 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 1.5, 2024-04-14T15:13:52.777 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 1.5, 2024-04-14T15:13:52.777 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.90223121643066406, 2024-04-14T15:13:52.777 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 0.89999997615814209, 2024-04-14T15:13:52.777 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-04-14T15:13:52.777 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.778 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.778 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 3, 2024-04-14T15:13:52.778 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "datapool", 2024-04-14T15:13:52.778 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:09:41.744391+0000", 2024-04-14T15:13:52.778 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 8193, 2024-04-14T15:13:52.778 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-14T15:13:52.778 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:13:52.778 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:13:52.778 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:13:52.778 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:13:52.778 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:13:52.779 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:13:52.779 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:13:52.779 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:13:52.779 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:13:52.779 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:13:52.779 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 3, 2024-04-14T15:13:52.779 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 3, 2024-04-14T15:13:52.779 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 3, 2024-04-14T15:13:52.779 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 3, 2024-04-14T15:13:52.779 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 3, 2024-04-14T15:13:52.780 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:13:52.780 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:13:52.780 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:13:52.780 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:13:52.780 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:13:52.780 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:13:52.780 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:13:52.782 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:13:52.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.782 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "69", 2024-04-14T15:13:52.782 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:13:52.782 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:13:52.782 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:13:52.783 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:13:52.783 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:13:52.783 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 2, 2024-04-14T15:13:52.783 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 69, 2024-04-14T15:13:52.783 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:13:52.783 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:13:52.783 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:13:52.783 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:13:52.783 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:13:52.783 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:13:52.783 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:13:52.784 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:13:52.784 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:13:52.784 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:13:52.784 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:13:52.784 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:13:52.784 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:13:52.784 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:13:52.784 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:13:52.784 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:13:52.784 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:13:52.785 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:13:52.785 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:13:52.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.785 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:13:52.785 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:13:52.785 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:13:52.786 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:13:52.786 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:13:52.786 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:13:52.786 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:13:52.786 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:13:52.787 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:13:52.787 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:13:52.787 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:13:52.787 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:13:52.787 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:13:52.787 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd": {} 2024-04-14T15:13:52.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.787 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:13:52.788 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:13:52.788 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 2.6500000953674316, 2024-04-14T15:13:52.788 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 2.6500000953674316, 2024-04-14T15:13:52.788 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 0.87999999523162842, 2024-04-14T15:13:52.788 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 2.3299999237060547, 2024-04-14T15:13:52.788 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 2.3299999237060547, 2024-04-14T15:13:52.788 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.90223121643066406, 2024-04-14T15:13:52.788 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 0.89999997615814209, 2024-04-14T15:13:52.789 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-04-14T15:13:52.789 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.789 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 4, 2024-04-14T15:13:52.789 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "default.rgw.log", 2024-04-14T15:13:52.789 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:09:43.088157+0000", 2024-04-14T15:13:52.789 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-14T15:13:52.789 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool", 2024-04-14T15:13:52.789 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:13:52.789 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:13:52.789 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:13:52.790 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:13:52.790 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:13:52.790 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:13:52.790 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:13:52.790 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:13:52.790 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:13:52.790 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:13:52.790 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:13:52.790 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:13:52.791 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:13:52.791 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:13:52.791 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:13:52.791 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:13:52.791 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:13:52.792 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:13:52.792 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:13:52.792 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:13:52.792 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:13:52.792 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:13:52.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.792 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "68", 2024-04-14T15:13:52.792 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:13:52.792 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:13:52.792 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:13:52.792 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:13:52.793 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:13:52.793 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:13:52.793 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:13:52.793 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:13:52.793 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:13:52.793 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:13:52.793 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:13:52.793 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:13:52.793 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:13:52.795 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:13:52.795 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:13:52.795 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:13:52.795 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:13:52.796 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:13:52.796 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:13:52.796 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:13:52.796 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:13:52.796 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:13:52.796 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:13:52.797 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:13:52.797 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:13:52.797 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:13:52.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.797 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:13:52.797 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:13:52.797 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:13:52.797 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:13:52.797 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:13:52.797 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:13:52.797 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:13:52.797 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:13:52.797 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:13:52.797 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:13:52.797 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:13:52.798 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:13:52.798 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:13:52.798 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw": {} 2024-04-14T15:13:52.798 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.798 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:13:52.798 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:13:52.798 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 2.25, 2024-04-14T15:13:52.798 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 2.25, 2024-04-14T15:13:52.798 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:13:52.799 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 2.25, 2024-04-14T15:13:52.799 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 2.25, 2024-04-14T15:13:52.799 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.90223121643066406, 2024-04-14T15:13:52.799 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 0.89999997615814209, 2024-04-14T15:13:52.799 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-04-14T15:13:52.799 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.799 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 5, 2024-04-14T15:13:52.799 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "default.rgw.control", 2024-04-14T15:13:52.799 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:09:45.042722+0000", 2024-04-14T15:13:52.800 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-14T15:13:52.800 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool", 2024-04-14T15:13:52.800 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:13:52.800 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:13:52.800 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:13:52.800 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:13:52.800 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:13:52.800 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:13:52.800 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:13:52.800 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:13:52.800 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:13:52.800 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:13:52.800 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:13:52.800 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:13:52.801 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:13:52.801 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:13:52.801 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:13:52.801 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:13:52.801 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:13:52.801 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:13:52.801 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:13:52.801 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:13:52.801 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:13:52.801 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:13:52.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.801 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "70", 2024-04-14T15:13:52.801 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:13:52.801 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:13:52.801 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:13:52.802 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:13:52.802 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:13:52.802 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:13:52.802 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:13:52.802 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:13:52.802 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:13:52.802 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:13:52.802 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:13:52.802 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:13:52.802 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:13:52.803 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:13:52.803 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:13:52.803 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:13:52.803 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:13:52.803 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:13:52.803 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:13:52.803 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:13:52.803 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:13:52.803 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:13:52.803 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:13:52.803 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:13:52.803 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:13:52.803 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:13:52.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.804 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:13:52.804 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:13:52.804 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:13:52.804 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:13:52.804 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:13:52.804 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:13:52.804 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:13:52.804 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:13:52.804 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:13:52.805 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:13:52.805 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:13:52.805 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:13:52.805 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:13:52.805 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw": {} 2024-04-14T15:13:52.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.805 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:13:52.806 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:13:52.806 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 1.5, 2024-04-14T15:13:52.806 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 1.5, 2024-04-14T15:13:52.806 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:13:52.806 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 1.5, 2024-04-14T15:13:52.806 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 1.5, 2024-04-14T15:13:52.806 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.90223121643066406, 2024-04-14T15:13:52.806 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 0.89999997615814209, 2024-04-14T15:13:52.806 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-04-14T15:13:52.806 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.806 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.806 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.806 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 6, 2024-04-14T15:13:52.807 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "default.rgw.meta", 2024-04-14T15:13:52.807 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:09:47.111336+0000", 2024-04-14T15:13:52.807 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-14T15:13:52.807 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool", 2024-04-14T15:13:52.807 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:13:52.807 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:13:52.807 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:13:52.807 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:13:52.807 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:13:52.807 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:13:52.807 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:13:52.807 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:13:52.807 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:13:52.807 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:13:52.808 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:13:52.808 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:13:52.808 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:13:52.808 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:13:52.808 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:13:52.808 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:13:52.808 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:13:52.808 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:13:52.808 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:13:52.808 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:13:52.808 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:13:52.808 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:13:52.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.808 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "72", 2024-04-14T15:13:52.808 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:13:52.809 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:13:52.809 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:13:52.809 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:13:52.809 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:13:52.809 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:13:52.809 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:13:52.809 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:13:52.809 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:13:52.809 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:13:52.809 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:13:52.809 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:13:52.809 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:13:52.809 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:13:52.809 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:13:52.809 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:13:52.810 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:13:52.810 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:13:52.810 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:13:52.810 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:13:52.810 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:13:52.810 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:13:52.810 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:13:52.810 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:13:52.810 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:13:52.810 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:13:52.810 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.810 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:13:52.810 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:13:52.810 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:13:52.811 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:13:52.811 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:13:52.811 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:13:52.811 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:13:52.811 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:13:52.811 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:13:52.811 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:13:52.811 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:13:52.811 INFO:teuthology.orchestra.run.smithi090.stdout: "options": { 2024-04-14T15:13:52.813 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_bias": 4 2024-04-14T15:13:52.813 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.814 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:13:52.814 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw": {} 2024-04-14T15:13:52.814 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.814 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:13:52.814 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:13:52.814 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 1.75, 2024-04-14T15:13:52.814 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 1.75, 2024-04-14T15:13:52.814 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:13:52.814 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 1.75, 2024-04-14T15:13:52.815 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 1.75, 2024-04-14T15:13:52.815 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.90223121643066406, 2024-04-14T15:13:52.815 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 0.89999997615814209, 2024-04-14T15:13:52.815 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-04-14T15:13:52.815 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.815 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 11, 2024-04-14T15:13:52.815 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "ceph_test_rados_api_asio", 2024-04-14T15:13:52.815 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:12:54.225671+0000", 2024-04-14T15:13:52.815 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-14T15:13:52.816 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool", 2024-04-14T15:13:52.816 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:13:52.816 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:13:52.816 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:13:52.816 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:13:52.816 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:13:52.816 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:13:52.816 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:13:52.817 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:13:52.817 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:13:52.817 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:13:52.817 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:13:52.817 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:13:52.817 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:13:52.817 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:13:52.817 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:13:52.817 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:13:52.817 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:13:52.817 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:13:52.817 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:13:52.817 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:13:52.817 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:13:52.818 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:13:52.818 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.818 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "109", 2024-04-14T15:13:52.818 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:13:52.818 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:13:52.818 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:13:52.818 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:13:52.818 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:13:52.818 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:13:52.818 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:13:52.818 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:13:52.818 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:13:52.818 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:13:52.818 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:13:52.818 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:13:52.819 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:13:52.819 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:13:52.819 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:13:52.819 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:13:52.819 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:13:52.819 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:13:52.819 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:13:52.819 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:13:52.819 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:13:52.819 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:13:52.819 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:13:52.820 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:13:52.820 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:13:52.820 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:13:52.820 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.820 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:13:52.820 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:13:52.820 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:13:52.820 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:13:52.820 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:13:52.820 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:13:52.820 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:13:52.820 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:13:52.820 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:13:52.820 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:13:52.821 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:13:52.821 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:13:52.821 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": {}, 2024-04-14T15:13:52.821 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:13:52.821 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:13:52.821 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 1.75, 2024-04-14T15:13:52.821 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 1.75, 2024-04-14T15:13:52.821 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:13:52.821 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 1.75, 2024-04-14T15:13:52.821 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 1.75, 2024-04-14T15:13:52.821 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.90223121643066406, 2024-04-14T15:13:52.821 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 0.89999997615814209, 2024-04-14T15:13:52.821 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-04-14T15:13:52.821 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.821 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.822 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 13, 2024-04-14T15:13:52.822 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "LibRadosList_smithi090-93444-1", 2024-04-14T15:13:52.822 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:12:54.244611+0000", 2024-04-14T15:13:52.822 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-14T15:13:52.822 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool", 2024-04-14T15:13:52.822 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:13:52.822 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:13:52.822 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:13:52.822 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:13:52.822 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:13:52.822 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:13:52.822 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:13:52.822 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:13:52.823 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:13:52.823 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:13:52.823 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 31, 2024-04-14T15:13:52.823 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 11, 2024-04-14T15:13:52.823 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 11, 2024-04-14T15:13:52.823 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 11, 2024-04-14T15:13:52.823 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 31, 2024-04-14T15:13:52.823 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:13:52.823 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "13.1f", 2024-04-14T15:13:52.823 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 164, 2024-04-14T15:13:52.823 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 163, 2024-04-14T15:13:52.823 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 163, 2024-04-14T15:13:52.823 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "109'5", 2024-04-14T15:13:52.823 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:13:52.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.824 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "164", 2024-04-14T15:13:52.824 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:13:52.824 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "164", 2024-04-14T15:13:52.824 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "162", 2024-04-14T15:13:52.824 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:13:52.824 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:13:52.824 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:13:52.824 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:13:52.824 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:13:52.824 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:13:52.824 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:13:52.824 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:13:52.824 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:13:52.824 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:13:52.824 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:13:52.825 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:13:52.825 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:13:52.825 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:13:52.825 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:13:52.825 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:13:52.826 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:13:52.826 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:13:52.826 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:13:52.826 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:13:52.826 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:13:52.826 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:13:52.826 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:13:52.826 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.826 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:13:52.826 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:13:52.827 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:13:52.827 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:13:52.827 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:13:52.827 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:13:52.827 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:13:52.827 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:13:52.827 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:13:52.827 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:13:52.827 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:13:52.827 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:13:52.828 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:13:52.828 INFO:teuthology.orchestra.run.smithi090.stdout: "rados": {} 2024-04-14T15:13:52.828 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.828 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:13:52.828 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:13:52.828 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 2.059999942779541, 2024-04-14T15:13:52.828 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 2.059999942779541, 2024-04-14T15:13:52.828 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:13:52.828 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 2.059999942779541, 2024-04-14T15:13:52.828 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 2.059999942779541, 2024-04-14T15:13:52.828 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.90223121643066406, 2024-04-14T15:13:52.828 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 0.89999997615814209, 2024-04-14T15:13:52.828 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-04-14T15:13:52.828 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.828 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.829 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.829 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 16, 2024-04-14T15:13:52.829 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "LibRadosMiscPP_smithi090-93601-1", 2024-04-14T15:13:52.829 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:12:54.300251+0000", 2024-04-14T15:13:52.829 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-14T15:13:52.829 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool", 2024-04-14T15:13:52.829 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:13:52.829 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:13:52.829 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:13:52.829 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:13:52.829 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:13:52.829 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:13:52.829 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:13:52.829 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:13:52.830 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:13:52.830 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:13:52.830 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:13:52.830 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:13:52.830 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:13:52.830 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:13:52.830 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:13:52.830 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:13:52.830 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:13:52.830 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:13:52.830 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:13:52.830 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:13:52.830 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:13:52.830 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:13:52.831 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.831 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "109", 2024-04-14T15:13:52.831 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:13:52.831 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:13:52.831 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:13:52.831 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:13:52.831 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:13:52.831 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:13:52.831 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:13:52.831 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:13:52.831 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:13:52.831 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:13:52.831 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:13:52.831 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:13:52.831 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:13:52.832 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:13:52.832 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:13:52.832 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:13:52.832 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:13:52.832 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:13:52.832 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:13:52.832 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:13:52.832 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:13:52.832 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:13:52.832 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:13:52.832 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:13:52.833 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:13:52.833 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:13:52.833 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.833 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:13:52.833 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:13:52.833 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:13:52.833 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:13:52.833 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:13:52.833 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:13:52.833 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:13:52.833 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:13:52.833 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:13:52.833 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:13:52.834 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:13:52.834 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:13:52.834 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:13:52.834 INFO:teuthology.orchestra.run.smithi090.stdout: "rados": {} 2024-04-14T15:13:52.834 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.834 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:13:52.834 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:13:52.834 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 2, 2024-04-14T15:13:52.834 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 2, 2024-04-14T15:13:52.834 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:13:52.835 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 2, 2024-04-14T15:13:52.835 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 2, 2024-04-14T15:13:52.835 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.90223121643066406, 2024-04-14T15:13:52.835 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 0.89999997615814209, 2024-04-14T15:13:52.835 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-04-14T15:13:52.836 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.836 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.836 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.836 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 110, 2024-04-14T15:13:52.836 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "test-rados-api-smithi090-93652-6", 2024-04-14T15:13:52.836 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:13:07.960794+0000", 2024-04-14T15:13:52.836 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 8193, 2024-04-14T15:13:52.836 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-14T15:13:52.836 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:13:52.836 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:13:52.836 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:13:52.836 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:13:52.836 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:13:52.837 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:13:52.837 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:13:52.837 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:13:52.838 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:13:52.838 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:13:52.838 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:13:52.838 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:13:52.838 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:13:52.838 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:13:52.838 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:13:52.838 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:13:52.838 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:13:52.838 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:13:52.838 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:13:52.838 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:13:52.838 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:13:52.839 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:13:52.839 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.839 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "155", 2024-04-14T15:13:52.839 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "155", 2024-04-14T15:13:52.839 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "155", 2024-04-14T15:13:52.839 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "155", 2024-04-14T15:13:52.839 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:13:52.839 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:13:52.839 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 6, 2024-04-14T15:13:52.839 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 154, 2024-04-14T15:13:52.839 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:13:52.839 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:13:52.839 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:13:52.839 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:13:52.839 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [ 2024-04-14T15:13:52.840 INFO:teuthology.orchestra.run.smithi090.stdout: 172 2024-04-14T15:13:52.840 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-14T15:13:52.840 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:13:52.840 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": 172, 2024-04-14T15:13:52.840 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": 172, 2024-04-14T15:13:52.840 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:13:52.840 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:13:52.840 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:13:52.840 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:13:52.840 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:13:52.840 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:13:52.840 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:13:52.841 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:13:52.841 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:13:52.841 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:13:52.841 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:13:52.841 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.841 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:13:52.841 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:13:52.841 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:13:52.841 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:13:52.841 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:13:52.841 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:13:52.841 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:13:52.841 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:13:52.841 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:13:52.841 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:13:52.842 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:13:52.842 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:13:52.842 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:13:52.842 INFO:teuthology.orchestra.run.smithi090.stdout: "rados": {} 2024-04-14T15:13:52.842 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.842 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:13:52.842 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:13:52.842 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 2, 2024-04-14T15:13:52.842 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 2, 2024-04-14T15:13:52.842 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:13:52.842 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 2, 2024-04-14T15:13:52.842 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 2, 2024-04-14T15:13:52.842 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.90223121643066406, 2024-04-14T15:13:52.842 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 0.89999997615814209, 2024-04-14T15:13:52.842 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-04-14T15:13:52.843 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.843 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.843 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.843 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 172, 2024-04-14T15:13:52.843 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "test-rados-api-smithi090-93652-13", 2024-04-14T15:13:52.843 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:13:34.849790+0000", 2024-04-14T15:13:52.843 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 8201, 2024-04-14T15:13:52.843 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,incomplete_clones,selfmanaged_snaps", 2024-04-14T15:13:52.843 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:13:52.843 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:13:52.843 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:13:52.843 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:13:52.843 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:13:52.843 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:13:52.844 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:13:52.844 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:13:52.844 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:13:52.844 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:13:52.844 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:13:52.844 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:13:52.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.761+0000 7fb69d34c700 10 librados: watch_flush enter 2024-04-14T15:13:52.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.761+0000 7fb69d34c700 10 librados: watch_flush exit 2024-04-14T15:13:52.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.761+0000 7fb69d34c700 20 client.48187.objecter shutdown clearing up homeless session... 2024-04-14T15:13:52.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.761+0000 7fb69d34c700 10 client.48187.objecter successfully canceled tick 2024-04-14T15:13:52.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.761+0000 7fb69d34c700 1 -- 172.21.15.90:0/2370564494 >> v1:172.21.15.90:6800/420935427 conn(0x7fb684075880 legacy=0x7fb684077d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:52.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.761+0000 7fb69d34c700 1 -- 172.21.15.90:0/2370564494 >> v1:172.21.15.92:6789/0 conn(0x7fb698103a80 legacy=0x7fb69810af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:52.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.761+0000 7fb69d34c700 1 -- 172.21.15.90:0/2370564494 shutdown_connections 2024-04-14T15:13:52.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.761+0000 7fb69d34c700 1 -- 172.21.15.90:0/2370564494 >> 172.21.15.90:0/2370564494 conn(0x7fb6980fd290 msgr2=0x7fb6980ff680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:13:52.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.761+0000 7fb69d34c700 1 -- 172.21.15.90:0/2370564494 shutdown_connections 2024-04-14T15:13:52.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.761+0000 7fb69d34c700 1 -- 172.21.15.90:0/2370564494 wait complete. 2024-04-14T15:13:52.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:52.761+0000 7fb69d34c700 1 librados: shutdown 2024-04-14T15:13:52.846 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:13:52.846 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:13:52.846 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:13:52.846 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:13:52.846 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:13:52.846 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:13:52.846 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:13:52.847 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:13:52.847 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:13:52.847 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:13:52.847 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.847 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "156", 2024-04-14T15:13:52.847 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "155", 2024-04-14T15:13:52.847 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "155", 2024-04-14T15:13:52.847 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "155", 2024-04-14T15:13:52.847 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:13:52.847 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:13:52.847 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 6, 2024-04-14T15:13:52.847 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 154, 2024-04-14T15:13:52.847 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:13:52.848 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:13:52.848 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:13:52.848 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:13:52.848 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:13:52.848 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": 110, 2024-04-14T15:13:52.848 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:13:52.848 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:13:52.848 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "writeback", 2024-04-14T15:13:52.848 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:13:52.848 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:13:52.848 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:13:52.848 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:13:52.848 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:13:52.848 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:13:52.849 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:13:52.849 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:13:52.849 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:13:52.849 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:13:52.849 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.849 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:13:52.849 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:13:52.849 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:13:52.849 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:13:52.849 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:13:52.849 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:13:52.849 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:13:52.849 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:13:52.849 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:13:52.850 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:13:52.850 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:13:52.850 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:13:52.850 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:13:52.850 INFO:teuthology.orchestra.run.smithi090.stdout: "rados": {} 2024-04-14T15:13:52.850 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.850 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:13:52.850 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:13:52.850 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 1.5, 2024-04-14T15:13:52.850 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 1.5, 2024-04-14T15:13:52.850 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:13:52.851 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 1.5, 2024-04-14T15:13:52.851 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 1.5, 2024-04-14T15:13:52.851 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.90223121643066406, 2024-04-14T15:13:52.851 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 0.89999997615814209, 2024-04-14T15:13:52.851 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-04-14T15:13:52.851 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.851 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.851 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.851 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 186, 2024-04-14T15:13:52.851 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "ReusePurgedSnapsmithi090-94770-11", 2024-04-14T15:13:52.851 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:13:48.846489+0000", 2024-04-14T15:13:52.851 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 8193, 2024-04-14T15:13:52.852 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-14T15:13:52.852 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:13:52.852 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:13:52.852 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:13:52.852 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:13:52.852 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:13:52.852 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:13:52.852 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:13:52.852 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:13:52.852 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:13:52.852 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:13:52.852 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:13:52.852 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:13:52.852 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:13:52.853 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:13:52.853 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:13:52.853 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:13:52.853 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:13:52.853 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:13:52.853 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:13:52.853 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:13:52.853 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:13:52.853 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:13:52.853 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.853 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "164", 2024-04-14T15:13:52.853 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:13:52.853 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:13:52.853 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:13:52.853 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:13:52.854 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:13:52.854 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 3, 2024-04-14T15:13:52.854 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 164, 2024-04-14T15:13:52.854 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:13:52.854 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:13:52.854 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:13:52.854 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:13:52.854 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:13:52.854 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:13:52.854 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:13:52.854 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:13:52.854 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:13:52.854 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:13:52.854 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:13:52.855 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:13:52.855 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:13:52.855 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:13:52.855 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:13:52.855 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:13:52.855 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:13:52.855 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:13:52.855 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:13:52.855 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.855 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:13:52.855 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:13:52.855 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:13:52.855 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:13:52.855 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:13:52.855 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:13:52.856 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:13:52.856 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:13:52.856 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:13:52.856 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:13:52.856 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:13:52.856 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:13:52.856 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": {}, 2024-04-14T15:13:52.856 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:13:52.856 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:13:52.856 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 1.5, 2024-04-14T15:13:52.856 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 1.5, 2024-04-14T15:13:52.856 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:13:52.856 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 1.5, 2024-04-14T15:13:52.856 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 1.5, 2024-04-14T15:13:52.856 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.90223121643066406, 2024-04-14T15:13:52.857 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 0.89999997615814209, 2024-04-14T15:13:52.857 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-04-14T15:13:52.857 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.857 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.857 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.857 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 187, 2024-04-14T15:13:52.857 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "test-rados-api-smithi090-93724-15", 2024-04-14T15:13:52.857 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:13:49.891851+0000", 2024-04-14T15:13:52.857 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 32769, 2024-04-14T15:13:52.857 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,creating", 2024-04-14T15:13:52.857 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:13:52.857 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:13:52.857 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:13:52.857 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:13:52.858 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:13:52.858 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:13:52.858 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:13:52.858 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:13:52.858 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:13:52.858 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:13:52.858 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:13:52.858 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:13:52.858 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:13:52.858 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:13:52.858 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:13:52.858 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:13:52.858 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:13:52.858 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:13:52.858 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:13:52.859 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:13:52.859 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:13:52.859 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:13:52.859 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.859 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "164", 2024-04-14T15:13:52.859 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:13:52.859 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:13:52.859 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:13:52.859 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:13:52.859 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:13:52.859 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:13:52.859 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:13:52.859 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:13:52.860 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:13:52.860 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:13:52.860 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:13:52.860 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:13:52.860 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:13:52.860 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:13:52.860 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:13:52.860 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:13:52.860 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:13:52.860 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:13:52.860 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:13:52.860 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:13:52.860 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:13:52.861 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:13:52.861 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:13:52.861 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:13:52.861 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:13:52.861 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:13:52.861 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.861 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:13:52.861 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:13:52.861 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:13:52.861 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:13:52.861 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:13:52.861 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:13:52.861 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:13:52.861 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:13:52.861 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:13:52.862 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:13:52.862 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:13:52.862 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:13:52.862 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:13:52.862 INFO:teuthology.orchestra.run.smithi090.stdout: "rados": {} 2024-04-14T15:13:52.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.862 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:13:52.862 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:13:52.862 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 2, 2024-04-14T15:13:52.862 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 2, 2024-04-14T15:13:52.862 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:13:52.862 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 2, 2024-04-14T15:13:52.862 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 2, 2024-04-14T15:13:52.862 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.90223121643066406, 2024-04-14T15:13:52.863 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 0.89999997615814209, 2024-04-14T15:13:52.863 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-04-14T15:13:52.863 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.863 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.863 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.863 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 188, 2024-04-14T15:13:52.863 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "OperateMtime_smithi090-93342-20", 2024-04-14T15:13:52.863 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:13:50.904113+0000", 2024-04-14T15:13:52.863 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 32769, 2024-04-14T15:13:52.863 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,creating", 2024-04-14T15:13:52.863 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:13:52.863 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:13:52.863 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:13:52.864 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:13:52.864 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:13:52.864 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:13:52.864 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:13:52.864 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:13:52.864 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:13:52.864 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:13:52.864 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:13:52.864 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:13:52.864 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:13:52.864 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:13:52.864 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:13:52.864 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:13:52.864 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:13:52.864 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:13:52.865 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:13:52.865 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:13:52.865 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:13:52.865 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:13:52.865 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.865 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "164", 2024-04-14T15:13:52.865 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:13:52.865 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:13:52.865 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:13:52.865 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:13:52.865 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:13:52.865 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:13:52.865 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:13:52.865 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:13:52.866 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:13:52.866 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:13:52.866 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:13:52.866 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:13:52.866 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:13:52.866 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:13:52.866 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:13:52.866 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:13:52.866 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:13:52.866 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:13:52.866 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:13:52.866 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:13:52.866 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:13:52.867 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:13:52.867 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:13:52.867 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:13:52.867 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:13:52.867 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:13:52.867 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.867 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:13:52.867 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:13:52.867 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:13:52.867 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:13:52.867 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:13:52.867 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:13:52.867 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:13:52.867 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:13:52.867 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:13:52.868 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:13:52.868 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:13:52.868 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:13:52.868 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": {}, 2024-04-14T15:13:52.868 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:13:52.868 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:13:52.868 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 2.5, 2024-04-14T15:13:52.868 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 2.5, 2024-04-14T15:13:52.868 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:13:52.868 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 2.5, 2024-04-14T15:13:52.868 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 2.5, 2024-04-14T15:13:52.868 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.90223121643066406, 2024-04-14T15:13:52.868 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 0.89999997615814209, 2024-04-14T15:13:52.868 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-04-14T15:13:52.868 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.869 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.869 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.869 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 189, 2024-04-14T15:13:52.869 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "FlushAsyncPP_smithi090-93363-15", 2024-04-14T15:13:52.869 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:13:50.909692+0000", 2024-04-14T15:13:52.869 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 32769, 2024-04-14T15:13:52.869 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,creating", 2024-04-14T15:13:52.869 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:13:52.869 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:13:52.869 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:13:52.869 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:13:52.869 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:13:52.869 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:13:52.869 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:13:52.869 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:13:52.870 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:13:52.870 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:13:52.870 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:13:52.870 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:13:52.870 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:13:52.870 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:13:52.870 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:13:52.870 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:13:52.870 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:13:52.870 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:13:52.870 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:13:52.870 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:13:52.870 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:13:52.870 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:13:52.871 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.871 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "164", 2024-04-14T15:13:52.871 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:13:52.871 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:13:52.871 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:13:52.871 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:13:52.871 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:13:52.871 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:13:52.871 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:13:52.871 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:13:52.871 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:13:52.871 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:13:52.871 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:13:52.871 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:13:52.872 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:13:52.872 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:13:52.872 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:13:52.872 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:13:52.872 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:13:52.872 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:13:52.872 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:13:52.872 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:13:52.872 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:13:52.872 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:13:52.872 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:13:52.872 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:13:52.872 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:13:52.872 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:13:52.872 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.873 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:13:52.873 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:13:52.873 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:13:52.873 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:13:52.873 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:13:52.873 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:13:52.873 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:13:52.873 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:13:52.873 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:13:52.873 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:13:52.873 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:13:52.873 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:13:52.873 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": {}, 2024-04-14T15:13:52.873 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:13:52.873 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:13:52.874 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 1.75, 2024-04-14T15:13:52.874 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 1.75, 2024-04-14T15:13:52.874 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:13:52.874 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 1.75, 2024-04-14T15:13:52.874 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 1.75, 2024-04-14T15:13:52.874 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.90223121643066406, 2024-04-14T15:13:52.874 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 0.89999997615814209, 2024-04-14T15:13:52.874 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-04-14T15:13:52.874 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.874 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.874 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-14T15:13:52.874 INFO:teuthology.orchestra.run.smithi090.stdout: "osds": [ 2024-04-14T15:13:52.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.875 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 0, 2024-04-14T15:13:52.875 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "76772cfc-89d1-4013-b617-32284ed8dfd9", 2024-04-14T15:13:52.875 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-14T15:13:52.875 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-14T15:13:52.875 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-14T15:13:52.875 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-14T15:13:52.875 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-14T15:13:52.875 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-14T15:13:52.876 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 8, 2024-04-14T15:13:52.876 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 163, 2024-04-14T15:13:52.876 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-14T15:13:52.876 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-14T15:13:52.876 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-14T15:13:52.876 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:13:52.876 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.877 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:13:52.877 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6801", 2024-04-14T15:13:52.877 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3761806885 2024-04-14T15:13:52.877 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.877 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:13:52.877 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.877 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-14T15:13:52.877 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:13:52.877 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.877 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:13:52.878 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6802", 2024-04-14T15:13:52.878 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3761806885 2024-04-14T15:13:52.878 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.878 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:13:52.878 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.878 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-14T15:13:52.878 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:13:52.878 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.878 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:13:52.878 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6804", 2024-04-14T15:13:52.878 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3761806885 2024-04-14T15:13:52.878 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.879 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:13:52.879 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.879 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-14T15:13:52.879 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:13:52.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.879 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:13:52.879 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6803", 2024-04-14T15:13:52.879 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3761806885 2024-04-14T15:13:52.879 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.879 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:13:52.879 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.879 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6801/3761806885", 2024-04-14T15:13:52.880 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6802/3761806885", 2024-04-14T15:13:52.880 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6804/3761806885", 2024-04-14T15:13:52.880 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6803/3761806885", 2024-04-14T15:13:52.880 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-14T15:13:52.880 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-14T15:13:52.880 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-14T15:13:52.880 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:13:52.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.880 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.880 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 1, 2024-04-14T15:13:52.880 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "de4a7fa2-7d57-4ff3-a4fa-28c0cde8ef6c", 2024-04-14T15:13:52.880 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-14T15:13:52.880 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-14T15:13:52.880 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-14T15:13:52.881 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-14T15:13:52.881 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-14T15:13:52.881 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-14T15:13:52.881 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2024-04-14T15:13:52.881 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 163, 2024-04-14T15:13:52.881 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-14T15:13:52.881 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-14T15:13:52.881 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-14T15:13:52.881 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:13:52.881 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.881 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:13:52.882 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6805", 2024-04-14T15:13:52.882 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 4170361699 2024-04-14T15:13:52.882 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.882 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:13:52.882 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.882 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-14T15:13:52.882 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:13:52.882 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.883 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:13:52.883 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6806", 2024-04-14T15:13:52.883 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 4170361699 2024-04-14T15:13:52.883 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.883 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:13:52.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.883 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-14T15:13:52.883 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:13:52.883 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.883 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:13:52.883 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6808", 2024-04-14T15:13:52.884 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 4170361699 2024-04-14T15:13:52.884 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.884 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:13:52.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.884 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-14T15:13:52.884 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:13:52.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.884 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:13:52.884 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6807", 2024-04-14T15:13:52.884 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 4170361699 2024-04-14T15:13:52.884 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.884 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:13:52.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.884 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6805/4170361699", 2024-04-14T15:13:52.885 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6806/4170361699", 2024-04-14T15:13:52.885 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6808/4170361699", 2024-04-14T15:13:52.885 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6807/4170361699", 2024-04-14T15:13:52.885 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-14T15:13:52.885 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-14T15:13:52.885 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-14T15:13:52.885 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:13:52.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.885 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.885 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 2, 2024-04-14T15:13:52.885 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "9c39b5ff-6175-46ba-9d71-370581c6ff3d", 2024-04-14T15:13:52.885 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-14T15:13:52.885 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-14T15:13:52.885 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-14T15:13:52.886 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-14T15:13:52.886 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-14T15:13:52.886 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-14T15:13:52.886 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 18, 2024-04-14T15:13:52.886 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 163, 2024-04-14T15:13:52.886 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-14T15:13:52.886 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-14T15:13:52.886 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-14T15:13:52.886 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:13:52.886 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.886 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:13:52.886 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6809", 2024-04-14T15:13:52.886 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 1686184251 2024-04-14T15:13:52.886 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.887 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:13:52.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.887 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-14T15:13:52.887 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:13:52.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.887 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:13:52.887 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6810", 2024-04-14T15:13:52.887 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 1686184251 2024-04-14T15:13:52.887 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.887 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:13:52.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.887 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-14T15:13:52.887 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:13:52.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.888 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:13:52.888 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6812", 2024-04-14T15:13:52.888 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 1686184251 2024-04-14T15:13:52.888 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.888 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:13:52.888 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.888 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-14T15:13:52.888 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:13:52.888 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.888 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:13:52.888 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6811", 2024-04-14T15:13:52.888 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 1686184251 2024-04-14T15:13:52.888 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.888 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:13:52.888 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.889 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6809/1686184251", 2024-04-14T15:13:52.889 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6810/1686184251", 2024-04-14T15:13:52.889 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6812/1686184251", 2024-04-14T15:13:52.889 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6811/1686184251", 2024-04-14T15:13:52.889 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-14T15:13:52.889 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-14T15:13:52.889 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-14T15:13:52.889 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:13:52.889 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.889 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.889 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 3, 2024-04-14T15:13:52.889 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "efd2ca45-5b80-4013-a571-499e2cb5b2e8", 2024-04-14T15:13:52.889 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-14T15:13:52.889 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-14T15:13:52.889 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-14T15:13:52.890 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 0.2178497314453125, 2024-04-14T15:13:52.890 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-14T15:13:52.890 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-14T15:13:52.890 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 25, 2024-04-14T15:13:52.890 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 163, 2024-04-14T15:13:52.890 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-14T15:13:52.890 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-14T15:13:52.890 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-14T15:13:52.890 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:13:52.890 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.890 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:13:52.890 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6813", 2024-04-14T15:13:52.890 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 2270166774 2024-04-14T15:13:52.891 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.891 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:13:52.891 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.891 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-14T15:13:52.891 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:13:52.891 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.891 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:13:52.891 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6814", 2024-04-14T15:13:52.891 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 2270166774 2024-04-14T15:13:52.891 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.891 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:13:52.891 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.891 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-14T15:13:52.891 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:13:52.891 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.892 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:13:52.892 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6816", 2024-04-14T15:13:52.892 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 2270166774 2024-04-14T15:13:52.892 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.892 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:13:52.892 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.892 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-14T15:13:52.892 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:13:52.892 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.892 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:13:52.892 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6815", 2024-04-14T15:13:52.892 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 2270166774 2024-04-14T15:13:52.892 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.892 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:13:52.892 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.893 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6813/2270166774", 2024-04-14T15:13:52.893 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6814/2270166774", 2024-04-14T15:13:52.893 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6816/2270166774", 2024-04-14T15:13:52.893 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6815/2270166774", 2024-04-14T15:13:52.893 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-14T15:13:52.893 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-14T15:13:52.893 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-14T15:13:52.893 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:13:52.893 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.893 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.893 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 4, 2024-04-14T15:13:52.893 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "e3d9013d-5eb6-433c-96b3-3fc38ba7ac8d", 2024-04-14T15:13:52.893 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-14T15:13:52.893 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-14T15:13:52.894 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-14T15:13:52.894 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-14T15:13:52.894 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-14T15:13:52.894 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-14T15:13:52.894 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 31, 2024-04-14T15:13:52.894 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 163, 2024-04-14T15:13:52.894 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-14T15:13:52.894 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-14T15:13:52.894 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-14T15:13:52.894 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:13:52.894 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.894 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:13:52.894 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6800", 2024-04-14T15:13:52.894 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3887549733 2024-04-14T15:13:52.894 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.895 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:13:52.895 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.895 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-14T15:13:52.895 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:13:52.895 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.895 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:13:52.895 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6801", 2024-04-14T15:13:52.895 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3887549733 2024-04-14T15:13:52.895 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.895 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:13:52.895 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.895 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-14T15:13:52.895 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:13:52.896 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.896 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:13:52.896 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6803", 2024-04-14T15:13:52.896 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3887549733 2024-04-14T15:13:52.896 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.896 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:13:52.896 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.896 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-14T15:13:52.896 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:13:52.896 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.896 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:13:52.896 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6802", 2024-04-14T15:13:52.896 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3887549733 2024-04-14T15:13:52.896 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.897 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:13:52.897 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.897 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.92:6800/3887549733", 2024-04-14T15:13:52.897 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.92:6801/3887549733", 2024-04-14T15:13:52.897 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.92:6803/3887549733", 2024-04-14T15:13:52.897 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.92:6802/3887549733", 2024-04-14T15:13:52.897 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-14T15:13:52.897 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-14T15:13:52.897 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-14T15:13:52.897 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:13:52.897 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.897 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.897 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 5, 2024-04-14T15:13:52.897 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "1ec41805-6f82-4139-81bc-75305ba9e958", 2024-04-14T15:13:52.897 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-14T15:13:52.897 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-14T15:13:52.898 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-14T15:13:52.898 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-14T15:13:52.898 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-14T15:13:52.898 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-14T15:13:52.898 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 41, 2024-04-14T15:13:52.898 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 163, 2024-04-14T15:13:52.898 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-14T15:13:52.898 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-14T15:13:52.898 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-14T15:13:52.898 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:13:52.898 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.898 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:13:52.898 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6804", 2024-04-14T15:13:52.898 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3244561440 2024-04-14T15:13:52.899 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.899 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:13:52.899 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.899 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-14T15:13:52.899 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:13:52.899 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.899 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:13:52.899 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6805", 2024-04-14T15:13:52.899 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3244561440 2024-04-14T15:13:52.899 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.899 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:13:52.899 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.899 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-14T15:13:52.899 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:13:52.899 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.900 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:13:52.900 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6807", 2024-04-14T15:13:52.900 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3244561440 2024-04-14T15:13:52.900 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.900 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:13:52.900 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.900 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-14T15:13:52.900 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:13:52.900 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.900 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:13:52.900 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6806", 2024-04-14T15:13:52.900 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3244561440 2024-04-14T15:13:52.900 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.900 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:13:52.901 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.901 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.92:6804/3244561440", 2024-04-14T15:13:52.901 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.92:6805/3244561440", 2024-04-14T15:13:52.901 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.92:6807/3244561440", 2024-04-14T15:13:52.901 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.92:6806/3244561440", 2024-04-14T15:13:52.901 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-14T15:13:52.901 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-14T15:13:52.901 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-14T15:13:52.901 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:13:52.901 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.901 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.901 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 6, 2024-04-14T15:13:52.901 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "ca977ab2-bf8d-4b3a-ae5e-0efa5bc7a3ad", 2024-04-14T15:13:52.901 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-14T15:13:52.901 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-14T15:13:52.902 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-14T15:13:52.902 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-14T15:13:52.902 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-14T15:13:52.902 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-14T15:13:52.902 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 49, 2024-04-14T15:13:52.902 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 163, 2024-04-14T15:13:52.902 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-14T15:13:52.902 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-14T15:13:52.902 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-14T15:13:52.902 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:13:52.902 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.902 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:13:52.902 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6808", 2024-04-14T15:13:52.903 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 636125963 2024-04-14T15:13:52.903 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.903 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:13:52.903 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.903 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-14T15:13:52.903 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:13:52.903 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.903 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:13:52.903 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6809", 2024-04-14T15:13:52.903 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 636125963 2024-04-14T15:13:52.903 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.903 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:13:52.903 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.903 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-14T15:13:52.903 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:13:52.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.904 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:13:52.904 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6811", 2024-04-14T15:13:52.904 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 636125963 2024-04-14T15:13:52.904 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.904 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:13:52.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.904 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-14T15:13:52.904 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:13:52.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.904 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:13:52.904 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6810", 2024-04-14T15:13:52.904 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 636125963 2024-04-14T15:13:52.904 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.904 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:13:52.905 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.905 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.92:6808/636125963", 2024-04-14T15:13:52.905 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.92:6809/636125963", 2024-04-14T15:13:52.905 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.92:6811/636125963", 2024-04-14T15:13:52.905 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.92:6810/636125963", 2024-04-14T15:13:52.905 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-14T15:13:52.905 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-14T15:13:52.905 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-14T15:13:52.905 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:13:52.905 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.905 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.905 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 7, 2024-04-14T15:13:52.905 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "e5354645-79ec-4d4c-b6e5-f9b0653915ee", 2024-04-14T15:13:52.905 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-14T15:13:52.906 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-14T15:13:52.906 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-14T15:13:52.906 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-14T15:13:52.906 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-14T15:13:52.906 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-14T15:13:52.906 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 58, 2024-04-14T15:13:52.906 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 163, 2024-04-14T15:13:52.906 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-14T15:13:52.906 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-14T15:13:52.906 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-14T15:13:52.906 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:13:52.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.907 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:13:52.907 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6812", 2024-04-14T15:13:52.907 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 1773706716 2024-04-14T15:13:52.907 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.907 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:13:52.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.907 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-14T15:13:52.907 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:13:52.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.907 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:13:52.907 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6813", 2024-04-14T15:13:52.908 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 1773706716 2024-04-14T15:13:52.908 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.908 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:13:52.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.908 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-14T15:13:52.908 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:13:52.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.908 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:13:52.908 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6815", 2024-04-14T15:13:52.908 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 1773706716 2024-04-14T15:13:52.908 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.908 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:13:52.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.908 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-14T15:13:52.909 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:13:52.909 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.909 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:13:52.909 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6814", 2024-04-14T15:13:52.909 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 1773706716 2024-04-14T15:13:52.909 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.909 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:13:52.909 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.909 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.92:6812/1773706716", 2024-04-14T15:13:52.909 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.92:6813/1773706716", 2024-04-14T15:13:52.909 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.92:6815/1773706716", 2024-04-14T15:13:52.909 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.92:6814/1773706716", 2024-04-14T15:13:52.909 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-14T15:13:52.909 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-14T15:13:52.910 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-14T15:13:52.910 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:13:52.910 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.910 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-14T15:13:52.910 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_xinfo": [ 2024-04-14T15:13:52.910 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.910 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 0, 2024-04-14T15:13:52.910 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-14T15:13:52.910 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-14T15:13:52.910 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-14T15:13:52.910 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540701547738038271, 2024-04-14T15:13:52.910 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-14T15:13:52.910 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-14T15:13:12.372581+0000", 2024-04-14T15:13:52.910 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-14T15:13:52.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.911 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 1, 2024-04-14T15:13:52.911 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-14T15:13:52.911 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-14T15:13:52.911 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-14T15:13:52.911 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540701547738038271, 2024-04-14T15:13:52.911 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-14T15:13:52.911 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-14T15:12:44.215601+0000", 2024-04-14T15:13:52.911 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-14T15:13:52.911 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.911 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 2, 2024-04-14T15:13:52.911 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-14T15:13:52.911 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-14T15:13:52.911 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-14T15:13:52.912 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540701547738038271, 2024-04-14T15:13:52.912 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-14T15:13:52.912 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-14T15:13:13.389324+0000", 2024-04-14T15:13:52.912 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-14T15:13:52.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.912 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 3, 2024-04-14T15:13:52.912 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-14T15:13:52.912 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-14T15:13:52.912 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-14T15:13:52.912 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540701547738038271, 2024-04-14T15:13:52.912 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-14T15:13:52.912 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-14T15:13:20.899272+0000", 2024-04-14T15:13:52.912 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-14T15:13:52.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.913 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 4, 2024-04-14T15:13:52.913 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-14T15:13:52.913 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-14T15:13:52.913 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-14T15:13:52.913 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540701547738038271, 2024-04-14T15:13:52.913 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-14T15:13:52.913 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-14T15:13:09.817510+0000", 2024-04-14T15:13:52.913 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-14T15:13:52.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.913 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 5, 2024-04-14T15:13:52.914 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-14T15:13:52.914 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-14T15:13:52.914 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-14T15:13:52.914 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540701547738038271, 2024-04-14T15:13:52.914 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-14T15:13:52.914 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-14T15:13:49.210268+0000", 2024-04-14T15:13:52.914 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-14T15:13:52.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.914 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 6, 2024-04-14T15:13:52.914 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-14T15:13:52.914 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-14T15:13:52.914 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-14T15:13:52.914 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540701547738038271, 2024-04-14T15:13:52.915 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-14T15:13:52.915 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-14T15:12:41.863374+0000", 2024-04-14T15:13:52.915 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-14T15:13:52.915 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.915 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.915 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 7, 2024-04-14T15:13:52.915 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-14T15:13:52.915 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-14T15:13:52.915 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-14T15:13:52.915 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540701547738038271, 2024-04-14T15:13:52.915 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-14T15:13:52.915 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-14T15:13:05.127218+0000", 2024-04-14T15:13:52.915 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-14T15:13:52.915 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.916 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-14T15:13:52.916 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_upmap": [], 2024-04-14T15:13:52.916 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_upmap_items": [ 2024-04-14T15:13:52.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.916 INFO:teuthology.orchestra.run.smithi090.stdout: "pgid": "2.1c", 2024-04-14T15:13:52.916 INFO:teuthology.orchestra.run.smithi090.stdout: "mappings": [ 2024-04-14T15:13:52.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.916 INFO:teuthology.orchestra.run.smithi090.stdout: "from": 7, 2024-04-14T15:13:52.916 INFO:teuthology.orchestra.run.smithi090.stdout: "to": 4 2024-04-14T15:13:52.916 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.916 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:13:52.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.916 INFO:teuthology.orchestra.run.smithi090.stdout: "pgid": "4.2", 2024-04-14T15:13:52.916 INFO:teuthology.orchestra.run.smithi090.stdout: "mappings": [ 2024-04-14T15:13:52.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.917 INFO:teuthology.orchestra.run.smithi090.stdout: "from": 1, 2024-04-14T15:13:52.917 INFO:teuthology.orchestra.run.smithi090.stdout: "to": 2 2024-04-14T15:13:52.917 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.917 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:13:52.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.917 INFO:teuthology.orchestra.run.smithi090.stdout: "pgid": "4.1f", 2024-04-14T15:13:52.917 INFO:teuthology.orchestra.run.smithi090.stdout: "mappings": [ 2024-04-14T15:13:52.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.917 INFO:teuthology.orchestra.run.smithi090.stdout: "from": 1, 2024-04-14T15:13:52.917 INFO:teuthology.orchestra.run.smithi090.stdout: "to": 2 2024-04-14T15:13:52.917 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.918 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:13:52.918 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.918 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-14T15:13:52.918 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_upmap_primaries": [], 2024-04-14T15:13:52.918 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_temp": [], 2024-04-14T15:13:52.918 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_temp": [], 2024-04-14T15:13:52.918 INFO:teuthology.orchestra.run.smithi090.stdout: "blocklist": { 2024-04-14T15:13:52.918 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:0/508768479": "2024-04-15T15:10:22.918674+0000", 2024-04-14T15:13:52.918 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:0/1823072952": "2024-04-15T15:10:22.918674+0000", 2024-04-14T15:13:52.918 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:6800/491071118": "2024-04-15T15:10:22.918674+0000", 2024-04-14T15:13:52.918 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:0/1337064035": "2024-04-15T15:10:22.918674+0000", 2024-04-14T15:13:52.918 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:0/769462306": "2024-04-15T15:10:22.918674+0000", 2024-04-14T15:13:52.918 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:0/1139868610": "2024-04-15T15:05:05.882361+0000", 2024-04-14T15:13:52.918 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:0/93763408": "2024-04-15T15:10:22.918674+0000", 2024-04-14T15:13:52.919 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:0/977818204": "2024-04-15T15:04:39.251711+0000", 2024-04-14T15:13:52.919 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:6800/941091171": "2024-04-15T15:05:05.882361+0000", 2024-04-14T15:13:52.919 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:0/3671453561": "2024-04-15T15:05:05.882361+0000", 2024-04-14T15:13:52.919 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:0/2611856200": "2024-04-15T15:05:05.882361+0000", 2024-04-14T15:13:52.919 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:6800/2306258962": "2024-04-15T15:04:39.251711+0000", 2024-04-14T15:13:52.919 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:0/4051692380": "2024-04-15T15:04:39.251711+0000", 2024-04-14T15:13:52.919 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:0/2681502561": "2024-04-15T15:04:39.251711+0000" 2024-04-14T15:13:52.919 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.919 INFO:teuthology.orchestra.run.smithi090.stdout: "range_blocklist": {}, 2024-04-14T15:13:52.919 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profiles": { 2024-04-14T15:13:52.919 INFO:teuthology.orchestra.run.smithi090.stdout: "default": { 2024-04-14T15:13:52.919 INFO:teuthology.orchestra.run.smithi090.stdout: "crush-failure-domain": "osd", 2024-04-14T15:13:52.919 INFO:teuthology.orchestra.run.smithi090.stdout: "k": "2", 2024-04-14T15:13:52.919 INFO:teuthology.orchestra.run.smithi090.stdout: "m": "1", 2024-04-14T15:13:52.920 INFO:teuthology.orchestra.run.smithi090.stdout: "plugin": "jerasure", 2024-04-14T15:13:52.920 INFO:teuthology.orchestra.run.smithi090.stdout: "technique": "reed_sol_van" 2024-04-14T15:13:52.920 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.920 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.920 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps_queue": [ 2024-04-14T15:13:52.920 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.920 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 17, 2024-04-14T15:13:52.920 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:13:52.920 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.920 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 2, 2024-04-14T15:13:52.920 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 5 2024-04-14T15:13:52.920 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.920 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:13:52.920 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.920 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.921 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 18, 2024-04-14T15:13:52.921 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:13:52.921 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.921 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 2, 2024-04-14T15:13:52.921 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 3 2024-04-14T15:13:52.921 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.921 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:13:52.921 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.921 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.921 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 39, 2024-04-14T15:13:52.921 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:13:52.921 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.921 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 1, 2024-04-14T15:13:52.921 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2024-04-14T15:13:52.922 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.922 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:13:52.922 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.922 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.922 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 72, 2024-04-14T15:13:52.922 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:13:52.922 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.922 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 1, 2024-04-14T15:13:52.922 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2024-04-14T15:13:52.922 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.922 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:13:52.922 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.922 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.922 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 99, 2024-04-14T15:13:52.923 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:13:52.923 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.923 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 4, 2024-04-14T15:13:52.923 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 2 2024-04-14T15:13:52.923 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.923 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.923 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 7, 2024-04-14T15:13:52.923 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 2 2024-04-14T15:13:52.923 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.923 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:13:52.923 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.923 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.923 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 109, 2024-04-14T15:13:52.923 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:13:52.923 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.924 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 1, 2024-04-14T15:13:52.924 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2024-04-14T15:13:52.924 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.924 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:13:52.924 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.924 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.924 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 128, 2024-04-14T15:13:52.924 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:13:52.924 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.924 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 1, 2024-04-14T15:13:52.924 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 2 2024-04-14T15:13:52.924 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.924 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:13:52.924 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.925 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 142, 2024-04-14T15:13:52.925 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:13:52.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.925 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 3, 2024-04-14T15:13:52.925 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2024-04-14T15:13:52.925 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.925 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:13:52.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.925 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 149, 2024-04-14T15:13:52.925 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:13:52.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.925 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 1, 2024-04-14T15:13:52.925 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 4 2024-04-14T15:13:52.926 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.926 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:13:52.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.926 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 156, 2024-04-14T15:13:52.926 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:13:52.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.926 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 2, 2024-04-14T15:13:52.926 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2024-04-14T15:13:52.926 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.926 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:13:52.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.927 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 157, 2024-04-14T15:13:52.927 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:13:52.927 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.927 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 2, 2024-04-14T15:13:52.927 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 2 2024-04-14T15:13:52.927 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.927 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:13:52.927 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.927 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.927 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 167, 2024-04-14T15:13:52.927 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:13:52.927 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.927 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 2, 2024-04-14T15:13:52.927 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 3 2024-04-14T15:13:52.927 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.928 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:13:52.928 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.928 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.928 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 175, 2024-04-14T15:13:52.928 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:13:52.928 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.928 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 2, 2024-04-14T15:13:52.928 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 4 2024-04-14T15:13:52.928 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.928 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:13:52.928 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:13:52.928 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.928 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 176, 2024-04-14T15:13:52.928 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:13:52.928 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:13:52.929 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 2, 2024-04-14T15:13:52.929 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 2 2024-04-14T15:13:52.929 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.929 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:13:52.929 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.929 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-14T15:13:52.929 INFO:teuthology.orchestra.run.smithi090.stdout: "new_removed_snaps": [], 2024-04-14T15:13:52.929 INFO:teuthology.orchestra.run.smithi090.stdout: "new_purged_snaps": [], 2024-04-14T15:13:52.929 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_node_flags": {}, 2024-04-14T15:13:52.929 INFO:teuthology.orchestra.run.smithi090.stdout: "device_class_flags": {}, 2024-04-14T15:13:52.929 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_mode": { 2024-04-14T15:13:52.929 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_mode_enabled": false, 2024-04-14T15:13:52.930 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_bucket_count": 0, 2024-04-14T15:13:52.930 INFO:teuthology.orchestra.run.smithi090.stdout: "degraded_stretch_mode": 0, 2024-04-14T15:13:52.930 INFO:teuthology.orchestra.run.smithi090.stdout: "recovering_stretch_mode": 0, 2024-04-14T15:13:52.930 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_mode_bucket": 0 2024-04-14T15:13:52.930 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:13:52.930 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-14T15:13:52.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:52.932 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:52.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:13:52.933 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-14T15:13:53.033 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:13:53.056 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:53.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:53.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:13:53.062 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:52 smithi090 ceph-mon[41670]: pgmap v191: 356 pgs: 64 unknown, 292 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 43 KiB/s rd, 57 op/s; 319 B/s, 4 objects/s recovering 2024-04-14T15:13:53.062 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:52 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2370564494' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-14T15:13:53.062 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:52 smithi090 ceph-mon[41670]: from='client.48169 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi090-93342-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:53.062 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:52 smithi090 ceph-mon[41670]: from='client.48175 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi090-93363-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:53.062 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:52 smithi090 ceph-mon[41670]: osdmap e165: 8 total, 8 up, 8 in 2024-04-14T15:13:53.062 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:52 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:53.063 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:52 smithi090 ceph-mon[30399]: pgmap v191: 356 pgs: 64 unknown, 292 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 43 KiB/s rd, 57 op/s; 319 B/s, 4 objects/s recovering 2024-04-14T15:13:53.063 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:52 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2370564494' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-14T15:13:53.063 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:52 smithi090 ceph-mon[30399]: from='client.48169 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi090-93342-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:53.063 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:52 smithi090 ceph-mon[30399]: from='client.48175 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi090-93363-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:53.063 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:52 smithi090 ceph-mon[30399]: osdmap e165: 8 total, 8 up, 8 in 2024-04-14T15:13:53.063 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:52 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:53.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:52 smithi092 ceph-mon[33966]: pgmap v191: 356 pgs: 64 unknown, 292 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 43 KiB/s rd, 57 op/s; 319 B/s, 4 objects/s recovering 2024-04-14T15:13:53.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:52 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2370564494' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-14T15:13:53.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:52 smithi092 ceph-mon[33966]: from='client.48169 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi090-93342-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:53.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:52 smithi092 ceph-mon[33966]: from='client.48175 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi090-93363-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:53.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:52 smithi092 ceph-mon[33966]: osdmap e165: 8 total, 8 up, 8 in 2024-04-14T15:13:53.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:52 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:53.158 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:13:53.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:53.183 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:53.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:13:53.215 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:13:53.284 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:13:53.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:53.309 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:53.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:13:53.410 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:13:53.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:53.434 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:53.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:13:53.535 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:13:53.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:53.561 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:53.562 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:13:53.663 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:13:53.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:53.689 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:53.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:13:53.790 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:13:53.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:53.816 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:53.816 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:13:53.917 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:13:53.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:53.942 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:53.946 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:13:54.047 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:13:54.070 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:53 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-04-14T15:13:54.070 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:53 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:13:54.070 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:53 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:13:54.070 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:53 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "30"}]': finished 2024-04-14T15:13:54.070 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:53 smithi092 ceph-mon[33966]: osdmap e166: 8 total, 8 up, 8 in 2024-04-14T15:13:54.070 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:53 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:54.071 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:54.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:54.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:13:54.088 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:13:53 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:13:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:13:54.089 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:53 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-04-14T15:13:54.089 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:53 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:13:54.089 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:53 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:13:54.089 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:53 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "30"}]': finished 2024-04-14T15:13:54.089 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:53 smithi090 ceph-mon[41670]: osdmap e166: 8 total, 8 up, 8 in 2024-04-14T15:13:54.089 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:53 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:54.089 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:53 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-04-14T15:13:54.089 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:53 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:13:54.090 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:53 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:13:54.090 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:53 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "30"}]': finished 2024-04-14T15:13:54.090 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:53 smithi090 ceph-mon[30399]: osdmap e166: 8 total, 8 up, 8 in 2024-04-14T15:13:54.090 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:53 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:54.174 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:13:54.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:54.199 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:54.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:13:54.300 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:13:54.329 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:54.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:54.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:13:54.431 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:13:54.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:54.456 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:54.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:13:54.557 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:13:54.569 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:13:54.582 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:54.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:54.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:13:54.683 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:13:54.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:54.708 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:54.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:13:54.809 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:13:54.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:54.834 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:54.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:13:54.935 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:13:54.941 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:54 smithi090 ceph-mon[30399]: pgmap v194: 419 pgs: 128 unknown, 291 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 22 op/s; 95 B/s, 1 objects/s recovering 2024-04-14T15:13:54.941 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:54 smithi090 ceph-mon[30399]: osdmap e167: 8 total, 8 up, 8 in 2024-04-14T15:13:54.941 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:54 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3142294623' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi090-93363-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:54.941 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:54 smithi090 ceph-mon[30399]: from='client.46325 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi090-93363-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:54.941 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:54 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:54.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:54 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1048914816' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi090-93342-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:54.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:54 smithi090 ceph-mon[30399]: from='client.48202 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi090-93342-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:54.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:54 smithi090 ceph-mon[41670]: pgmap v194: 419 pgs: 128 unknown, 291 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 22 op/s; 95 B/s, 1 objects/s recovering 2024-04-14T15:13:54.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:54 smithi090 ceph-mon[41670]: osdmap e167: 8 total, 8 up, 8 in 2024-04-14T15:13:54.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:54 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3142294623' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi090-93363-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:54.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:54 smithi090 ceph-mon[41670]: from='client.46325 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi090-93363-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:54.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:54 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:54.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:54 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1048914816' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi090-93342-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:54.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:54 smithi090 ceph-mon[41670]: from='client.48202 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi090-93342-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:54.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:54.960 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:54.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:13:54.975 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:54 smithi092 ceph-mon[33966]: pgmap v194: 419 pgs: 128 unknown, 291 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 22 op/s; 95 B/s, 1 objects/s recovering 2024-04-14T15:13:54.975 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:54 smithi092 ceph-mon[33966]: osdmap e167: 8 total, 8 up, 8 in 2024-04-14T15:13:54.975 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:54 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3142294623' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi090-93363-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:54.975 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:54 smithi092 ceph-mon[33966]: from='client.46325 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi090-93363-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:54.975 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:54 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:54.975 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:54 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1048914816' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi090-93342-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:54.975 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:54 smithi092 ceph-mon[33966]: from='client.48202 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi090-93342-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:54.975 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:13:54 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:13:55.061 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:13:55.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:55.085 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:55.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:13:55.109 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-04-14T15:13:55.109 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-04-14T15:13:55.109 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-04-14T15:13:55.109 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-04-14T15:13:55.110 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.110 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25028.0:2 40.6 40:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-04-14T15:13:55.110 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:55.118640+0000", 2024-04-14T15:13:55.110 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 59.988884984999999, 2024-04-14T15:13:55.110 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.188111707, 2024-04-14T15:13:55.110 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:13:55.110 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:13:55.110 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:13:55.110 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:13:55.110 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25028", 2024-04-14T15:13:55.110 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/2033704495", 2024-04-14T15:13:55.110 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 2 2024-04-14T15:13:55.111 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.111 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:13:55.111 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.111 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:13:55.111 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.118640+0000", 2024-04-14T15:13:55.111 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:13:55.111 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.111 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.111 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:13:55.111 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.118640+0000", 2024-04-14T15:13:55.111 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:13:55.111 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.111 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.111 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:13:55.112 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.118642+0000", 2024-04-14T15:13:55.112 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0899999999999999e-06 2024-04-14T15:13:55.112 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.112 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.112 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:13:55.112 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.118647+0000", 2024-04-14T15:13:55.112 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6690000000000003e-06 2024-04-14T15:13:55.112 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.112 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.112 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:13:55.112 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.118656+0000", 2024-04-14T15:13:55.112 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.4279999999999996e-06 2024-04-14T15:13:55.112 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.112 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.113 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:13:55.113 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.118667+0000", 2024-04-14T15:13:55.113 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0723000000000001e-05 2024-04-14T15:13:55.113 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.113 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.113 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:13:55.113 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.118857+0000", 2024-04-14T15:13:55.113 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00019035600000000001 2024-04-14T15:13:55.113 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.113 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.113 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:13:55.113 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.118865+0000", 2024-04-14T15:13:55.113 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.7559999999999999e-06 2024-04-14T15:13:55.113 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.114 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.114 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:13:55.114 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.210788+0000", 2024-04-14T15:13:55.114 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.091922904 2024-04-14T15:13:55.114 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.114 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.114 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:13:55.114 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.210898+0000", 2024-04-14T15:13:55.114 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000110659 2024-04-14T15:13:55.114 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.114 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.114 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 2,5", 2024-04-14T15:13:55.114 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.211021+0000", 2024-04-14T15:13:55.115 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000122753 2024-04-14T15:13:55.115 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.115 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.115 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:13:55.115 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.213735+0000", 2024-04-14T15:13:55.115 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002713764 2024-04-14T15:13:55.115 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.115 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.115 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:13:55.115 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.250150+0000", 2024-04-14T15:13:55.115 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.036414806000000001 2024-04-14T15:13:55.115 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.115 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.116 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:13:55.116 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.306654+0000", 2024-04-14T15:13:55.116 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.056504293999999997 2024-04-14T15:13:55.116 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.116 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.116 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:13:55.116 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.306706+0000", 2024-04-14T15:13:55.116 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2346000000000003e-05 2024-04-14T15:13:55.116 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.116 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.116 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:13:55.116 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.306751+0000", 2024-04-14T15:13:55.116 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5158999999999999e-05 2024-04-14T15:13:55.116 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:13:55.117 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:13:55.117 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:13:55.117 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.117 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.117 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.24829.0:2 13.6 13:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-04-14T15:13:55.117 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:56.068832+0000", 2024-04-14T15:13:55.117 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 59.038693002999999, 2024-04-14T15:13:55.117 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.43249991799999998, 2024-04-14T15:13:55.117 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:13:55.117 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:13:55.117 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:13:55.117 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:13:55.117 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.24829", 2024-04-14T15:13:55.117 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/218717313", 2024-04-14T15:13:55.118 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 2 2024-04-14T15:13:55.118 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.118 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:13:55.118 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.118 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:13:55.118 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.068832+0000", 2024-04-14T15:13:55.118 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:13:55.118 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.118 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.118 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:13:55.118 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.068832+0000", 2024-04-14T15:13:55.118 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:13:55.118 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.118 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.119 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:13:55.119 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.068834+0000", 2024-04-14T15:13:55.119 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.323e-06 2024-04-14T15:13:55.119 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.119 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.119 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:13:55.119 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.068842+0000", 2024-04-14T15:13:55.119 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3809999999999999e-06 2024-04-14T15:13:55.119 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.119 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.119 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:13:55.119 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.068860+0000", 2024-04-14T15:13:55.119 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7620000000000001e-05 2024-04-14T15:13:55.119 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.120 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.120 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:13:55.120 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.068872+0000", 2024-04-14T15:13:55.120 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2106999999999999e-05 2024-04-14T15:13:55.120 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.120 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.120 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:13:55.120 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.068976+0000", 2024-04-14T15:13:55.120 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010345599999999999 2024-04-14T15:13:55.120 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.120 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.120 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op must wait for map", 2024-04-14T15:13:55.120 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.068982+0000", 2024-04-14T15:13:55.120 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.7530000000000004e-06 2024-04-14T15:13:55.121 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.121 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.121 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:13:55.121 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.146226+0000", 2024-04-14T15:13:55.121 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.077243952000000005 2024-04-14T15:13:55.121 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.121 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.121 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:13:55.121 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.146240+0000", 2024-04-14T15:13:55.121 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4015999999999999e-05 2024-04-14T15:13:55.121 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.121 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.121 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:13:55.121 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.245970+0000", 2024-04-14T15:13:55.122 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.099729313999999999 2024-04-14T15:13:55.122 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.122 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.122 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:13:55.122 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.246073+0000", 2024-04-14T15:13:55.122 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000103532 2024-04-14T15:13:55.122 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.122 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.122 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,5", 2024-04-14T15:13:55.122 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.246207+0000", 2024-04-14T15:13:55.122 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000134036 2024-04-14T15:13:55.122 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.122 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.122 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:13:55.123 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.248168+0000", 2024-04-14T15:13:55.123 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0019606049999999998 2024-04-14T15:13:55.123 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.123 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.123 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:13:55.123 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.267428+0000", 2024-04-14T15:13:55.123 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.01926025 2024-04-14T15:13:55.123 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.123 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.123 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:13:55.123 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.501199+0000", 2024-04-14T15:13:55.123 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.23377057900000001 2024-04-14T15:13:55.123 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.124 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.124 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:13:55.124 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.501285+0000", 2024-04-14T15:13:55.124 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6007000000000005e-05 2024-04-14T15:13:55.124 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.124 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.124 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:13:55.124 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.501332+0000", 2024-04-14T15:13:55.124 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6987000000000003e-05 2024-04-14T15:13:55.124 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:13:55.124 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:13:55.124 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:13:55.124 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.124 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.125 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.24896.0:2 23.15 23:ad551702:::test:head [stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-04-14T15:13:55.125 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:56.072291+0000", 2024-04-14T15:13:55.125 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 59.035234027000001, 2024-04-14T15:13:55.125 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.24892146700000001, 2024-04-14T15:13:55.125 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:13:55.125 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:13:55.125 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-04-14T15:13:55.125 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:13:55.125 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.24896", 2024-04-14T15:13:55.125 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/861388440", 2024-04-14T15:13:55.125 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 2 2024-04-14T15:13:55.125 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.125 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:13:55.126 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.126 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:13:55.126 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.072291+0000", 2024-04-14T15:13:55.126 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:13:55.126 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.126 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.126 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:13:55.126 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.072291+0000", 2024-04-14T15:13:55.126 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:13:55.126 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.126 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.127 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:13:55.127 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.072292+0000", 2024-04-14T15:13:55.127 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.812e-06 2024-04-14T15:13:55.127 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.127 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.127 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:13:55.127 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.072298+0000", 2024-04-14T15:13:55.127 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.631e-06 2024-04-14T15:13:55.127 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.127 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.127 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:13:55.127 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.072309+0000", 2024-04-14T15:13:55.128 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0934000000000001e-05 2024-04-14T15:13:55.128 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.128 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.128 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:13:55.128 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.072318+0000", 2024-04-14T15:13:55.128 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.9800000000000004e-06 2024-04-14T15:13:55.128 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.128 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.128 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:13:55.129 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.072374+0000", 2024-04-14T15:13:55.129 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6221000000000003e-05 2024-04-14T15:13:55.129 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.129 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.129 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op must wait for map", 2024-04-14T15:13:55.129 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.072381+0000", 2024-04-14T15:13:55.129 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3169999999999998e-06 2024-04-14T15:13:55.129 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.129 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.129 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:13:55.129 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.120345+0000", 2024-04-14T15:13:55.129 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.047964917000000003 2024-04-14T15:13:55.129 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.129 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.130 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:13:55.130 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.120355+0000", 2024-04-14T15:13:55.130 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.2909999999999996e-06 2024-04-14T15:13:55.130 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.130 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.130 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:13:55.130 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.321061+0000", 2024-04-14T15:13:55.130 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.20070617099999999 2024-04-14T15:13:55.130 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.130 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.130 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:13:55.130 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.321212+0000", 2024-04-14T15:13:55.130 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015119299999999999 2024-04-14T15:13:55.131 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:13:55.131 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:13:55.131 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:13:55.131 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.131 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.131 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.24802.0:2 8.14 8:2a134577:test-rados-api-smithi090-93379-2::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-04-14T15:13:55.131 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:56.073174+0000", 2024-04-14T15:13:55.131 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 59.034350963000001, 2024-04-14T15:13:55.131 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.34713522000000002, 2024-04-14T15:13:55.131 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:13:55.131 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:13:55.131 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:13:55.131 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:13:55.132 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.24802", 2024-04-14T15:13:55.132 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/1106328377", 2024-04-14T15:13:55.132 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 2 2024-04-14T15:13:55.132 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.132 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:13:55.132 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.132 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:13:55.132 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.073174+0000", 2024-04-14T15:13:55.132 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:13:55.132 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.132 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.132 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:13:55.132 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.073174+0000", 2024-04-14T15:13:55.132 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:13:55.133 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.133 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.133 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:13:55.133 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.073176+0000", 2024-04-14T15:13:55.133 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0640000000000001e-06 2024-04-14T15:13:55.133 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.133 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.133 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:13:55.133 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.073181+0000", 2024-04-14T15:13:55.133 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3399999999999997e-06 2024-04-14T15:13:55.133 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.133 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.133 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:13:55.134 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.073195+0000", 2024-04-14T15:13:55.134 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3471e-05 2024-04-14T15:13:55.134 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.134 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.134 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:13:55.134 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.073203+0000", 2024-04-14T15:13:55.134 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.123e-06 2024-04-14T15:13:55.134 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.134 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.134 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:13:55.134 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.073254+0000", 2024-04-14T15:13:55.134 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.1255000000000003e-05 2024-04-14T15:13:55.134 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.135 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.135 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op must wait for map", 2024-04-14T15:13:55.135 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.073261+0000", 2024-04-14T15:13:55.135 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2069999999999998e-06 2024-04-14T15:13:55.135 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.135 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.135 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:13:55.135 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.151898+0000", 2024-04-14T15:13:55.135 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.078636629 2024-04-14T15:13:55.135 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.135 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.135 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:13:55.135 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.151906+0000", 2024-04-14T15:13:55.135 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2640000000000008e-06 2024-04-14T15:13:55.136 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.136 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.136 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:13:55.136 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.294003+0000", 2024-04-14T15:13:55.136 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.14209672500000001 2024-04-14T15:13:55.136 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.136 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.136 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:13:55.136 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.294129+0000", 2024-04-14T15:13:55.136 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000125969 2024-04-14T15:13:55.136 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.136 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.136 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,2", 2024-04-14T15:13:55.136 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.294286+0000", 2024-04-14T15:13:55.137 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000157436 2024-04-14T15:13:55.137 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.137 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.137 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:13:55.137 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.295784+0000", 2024-04-14T15:13:55.137 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0014976600000000001 2024-04-14T15:13:55.137 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.137 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.137 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:13:55.137 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.357451+0000", 2024-04-14T15:13:55.137 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.061667551000000001 2024-04-14T15:13:55.137 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.137 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.138 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:13:55.138 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.419349+0000", 2024-04-14T15:13:55.138 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.061897887999999998 2024-04-14T15:13:55.138 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.138 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.138 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:13:55.138 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.420262+0000", 2024-04-14T15:13:55.138 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00091295099999999995 2024-04-14T15:13:55.138 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.138 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.140 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:13:55.140 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.420309+0000", 2024-04-14T15:13:55.140 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6687000000000002e-05 2024-04-14T15:13:55.140 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:13:55.140 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:13:55.140 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:13:55.140 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.140 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.140 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25028.0:3 40.6 40:602f83fe:::foo:head [watch watch cookie 94006567794064] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-04-14T15:13:55.140 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:56.307396+0000", 2024-04-14T15:13:55.140 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 58.800128407000003, 2024-04-14T15:13:55.140 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.053731905000000003, 2024-04-14T15:13:55.141 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:13:55.141 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:13:55.141 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:13:55.141 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:13:55.141 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25028", 2024-04-14T15:13:55.141 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/2033704495", 2024-04-14T15:13:55.141 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 3 2024-04-14T15:13:55.141 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.141 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:13:55.141 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.141 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:13:55.141 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.307396+0000", 2024-04-14T15:13:55.141 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:13:55.142 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.142 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.142 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:13:55.142 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.307396+0000", 2024-04-14T15:13:55.142 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:13:55.142 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.142 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.142 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:13:55.142 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.307398+0000", 2024-04-14T15:13:55.142 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.539e-06 2024-04-14T15:13:55.142 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.142 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.142 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:13:55.142 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.307403+0000", 2024-04-14T15:13:55.143 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.1039999999999998e-06 2024-04-14T15:13:55.143 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.143 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.143 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:13:55.143 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.307413+0000", 2024-04-14T15:13:55.143 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0212e-05 2024-04-14T15:13:55.143 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.143 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.143 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:13:55.143 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.307424+0000", 2024-04-14T15:13:55.143 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0964999999999999e-05 2024-04-14T15:13:55.143 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.143 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.143 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:13:55.144 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.308351+0000", 2024-04-14T15:13:55.144 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00092658400000000004 2024-04-14T15:13:55.144 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.144 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.144 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:13:55.144 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.308438+0000", 2024-04-14T15:13:55.144 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.7077000000000007e-05 2024-04-14T15:13:55.144 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.144 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.145 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 2,5", 2024-04-14T15:13:55.145 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.308709+0000", 2024-04-14T15:13:55.145 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00027113400000000002 2024-04-14T15:13:55.145 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.145 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.145 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:13:55.145 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.309380+0000", 2024-04-14T15:13:55.145 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00067086999999999995 2024-04-14T15:13:55.145 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.146 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.146 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:13:55.146 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.309489+0000", 2024-04-14T15:13:55.146 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000109181 2024-04-14T15:13:55.146 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.146 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.146 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:13:55.146 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.360999+0000", 2024-04-14T15:13:55.146 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.051509961999999999 2024-04-14T15:13:55.146 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.146 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.146 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:13:55.146 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.361055+0000", 2024-04-14T15:13:55.147 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6017000000000003e-05 2024-04-14T15:13:55.147 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.147 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.147 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:13:55.147 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.361128+0000", 2024-04-14T15:13:55.147 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.326e-05 2024-04-14T15:13:55.147 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:13:55.147 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:13:55.147 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:13:55.147 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.147 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.147 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25028.0:7 40.6 40:602f83fe:::foo:head [watch unwatch cookie 94006567794064] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-04-14T15:13:55.147 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:56.376539+0000", 2024-04-14T15:13:55.147 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 58.730985425999997, 2024-04-14T15:13:55.148 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.10311740799999999, 2024-04-14T15:13:55.148 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:13:55.148 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:13:55.148 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:13:55.148 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:13:55.148 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25028", 2024-04-14T15:13:55.148 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/2033704495", 2024-04-14T15:13:55.148 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 7 2024-04-14T15:13:55.148 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.148 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:13:55.148 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.148 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:13:55.148 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.376539+0000", 2024-04-14T15:13:55.148 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:13:55.149 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.149 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.149 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:13:55.149 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.376539+0000", 2024-04-14T15:13:55.149 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:13:55.149 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.149 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.149 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:13:55.149 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.376541+0000", 2024-04-14T15:13:55.149 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9829999999999999e-06 2024-04-14T15:13:55.149 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.149 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.149 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:13:55.149 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.376561+0000", 2024-04-14T15:13:55.150 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9961000000000001e-05 2024-04-14T15:13:55.150 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.150 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.150 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:13:55.150 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.376572+0000", 2024-04-14T15:13:55.150 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0813e-05 2024-04-14T15:13:55.150 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.150 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.150 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:13:55.150 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.376583+0000", 2024-04-14T15:13:55.150 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0529e-05 2024-04-14T15:13:55.150 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.150 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.151 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:13:55.151 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.376646+0000", 2024-04-14T15:13:55.151 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3589000000000004e-05 2024-04-14T15:13:55.151 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.151 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.151 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:13:55.151 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.376745+0000", 2024-04-14T15:13:55.151 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.9146000000000002e-05 2024-04-14T15:13:55.151 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.151 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.151 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 2,5", 2024-04-14T15:13:55.151 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.376916+0000", 2024-04-14T15:13:55.151 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00017041499999999999 2024-04-14T15:13:55.152 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.152 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.152 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:13:55.152 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.377764+0000", 2024-04-14T15:13:55.152 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00084804300000000004 2024-04-14T15:13:55.152 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.152 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.152 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:13:55.152 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.411348+0000", 2024-04-14T15:13:55.152 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.033583871000000001 2024-04-14T15:13:55.152 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.152 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.152 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:13:55.152 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.479526+0000", 2024-04-14T15:13:55.153 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.068178424000000001 2024-04-14T15:13:55.153 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.153 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.153 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:13:55.153 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.479586+0000", 2024-04-14T15:13:55.153 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0093000000000002e-05 2024-04-14T15:13:55.153 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.153 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.153 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:13:55.153 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.479657+0000", 2024-04-14T15:13:55.153 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0541000000000001e-05 2024-04-14T15:13:55.153 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:13:55.153 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:13:55.153 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:13:55.154 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.154 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.154 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.24829.0:828 13.15 13:a8000000::::head [pgnls start_epoch 108] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e108)", 2024-04-14T15:13:55.154 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:57.081514+0000", 2024-04-14T15:13:55.154 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 58.026011142999998, 2024-04-14T15:13:55.154 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.067804700999999995, 2024-04-14T15:13:55.154 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:13:55.154 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:13:55.154 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-04-14T15:13:55.154 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:13:55.154 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.24829", 2024-04-14T15:13:55.154 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/218717313", 2024-04-14T15:13:55.154 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 828 2024-04-14T15:13:55.155 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.155 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:13:55.155 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.155 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:13:55.155 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.081514+0000", 2024-04-14T15:13:55.155 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:13:55.155 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.155 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.155 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:13:55.155 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.081514+0000", 2024-04-14T15:13:55.155 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:13:55.155 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.155 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.155 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:13:55.156 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.081516+0000", 2024-04-14T15:13:55.156 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6180000000000002e-06 2024-04-14T15:13:55.156 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.156 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.156 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:13:55.156 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.081524+0000", 2024-04-14T15:13:55.156 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0479999999999999e-06 2024-04-14T15:13:55.156 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.156 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.156 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:13:55.156 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.092114+0000", 2024-04-14T15:13:55.156 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.010590123999999999 2024-04-14T15:13:55.156 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.157 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.157 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:13:55.157 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.092128+0000", 2024-04-14T15:13:55.157 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4064e-05 2024-04-14T15:13:55.157 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.157 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.157 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:13:55.157 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.149196+0000", 2024-04-14T15:13:55.157 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.057067179000000003 2024-04-14T15:13:55.157 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.157 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.157 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:13:55.157 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.149234+0000", 2024-04-14T15:13:55.157 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8847999999999998e-05 2024-04-14T15:13:55.158 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.158 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.158 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:13:55.158 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.149318+0000", 2024-04-14T15:13:55.158 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3819999999999994e-05 2024-04-14T15:13:55.158 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:13:55.158 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:13:55.158 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:13:55.158 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.158 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.158 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25198.0:2 55.1a 55:5c6776db:::0.obj:head [write 0~103] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-04-14T15:13:55.159 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:58.081881+0000", 2024-04-14T15:13:55.159 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 57.025643973000001, 2024-04-14T15:13:55.159 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2238229890000001, 2024-04-14T15:13:55.159 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:13:55.159 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:13:55.159 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:13:55.159 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:13:55.159 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25198", 2024-04-14T15:13:55.159 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/1077085148", 2024-04-14T15:13:55.159 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 2 2024-04-14T15:13:55.159 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.159 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:13:55.159 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.159 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:13:55.160 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.081881+0000", 2024-04-14T15:13:55.160 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:13:55.160 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.160 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.160 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:13:55.160 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.081881+0000", 2024-04-14T15:13:55.160 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:13:55.160 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.160 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.160 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:13:55.160 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.081883+0000", 2024-04-14T15:13:55.160 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2029999999999999e-06 2024-04-14T15:13:55.160 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.160 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.161 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:13:55.161 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.081893+0000", 2024-04-14T15:13:55.161 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0152e-05 2024-04-14T15:13:55.161 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.161 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.161 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:13:55.161 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.081903+0000", 2024-04-14T15:13:55.161 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0353000000000001e-05 2024-04-14T15:13:55.161 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.161 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.161 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:13:55.162 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.081912+0000", 2024-04-14T15:13:55.162 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.9199999999999993e-06 2024-04-14T15:13:55.162 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.162 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.162 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:13:55.162 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.103712+0000", 2024-04-14T15:13:55.162 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.021799163 2024-04-14T15:13:55.162 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.162 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.162 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:13:55.162 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.103720+0000", 2024-04-14T15:13:55.162 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.4470000000000006e-06 2024-04-14T15:13:55.162 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.163 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.163 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:13:55.163 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.204902+0000", 2024-04-14T15:13:55.163 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.101182071 2024-04-14T15:13:55.163 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.163 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.163 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:13:55.163 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.205015+0000", 2024-04-14T15:13:55.163 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011304800000000001 2024-04-14T15:13:55.163 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.163 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.163 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,3", 2024-04-14T15:13:55.163 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.205152+0000", 2024-04-14T15:13:55.163 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013653999999999999 2024-04-14T15:13:55.164 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.164 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.164 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:13:55.164 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.208940+0000", 2024-04-14T15:13:55.164 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0037881939999999999 2024-04-14T15:13:55.164 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.164 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.164 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:13:55.164 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.252540+0000", 2024-04-14T15:13:55.164 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.043599903000000002 2024-04-14T15:13:55.164 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.164 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.164 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:13:55.164 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.305601+0000", 2024-04-14T15:13:55.164 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.053060819000000002 2024-04-14T15:13:55.165 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.165 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.165 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:13:55.165 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.305655+0000", 2024-04-14T15:13:55.165 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4385999999999997e-05 2024-04-14T15:13:55.165 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.165 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.165 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:13:55.165 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.305704+0000", 2024-04-14T15:13:55.165 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8789999999999999e-05 2024-04-14T15:13:55.165 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:13:55.165 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:13:55.165 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:13:55.166 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.166 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.166 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.24868.0:5 17.2 17:412fddd1:test-rados-api-smithi090-93724-3::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-04-14T15:13:55.166 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:58.125668+0000", 2024-04-14T15:13:55.166 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 56.981857024, 2024-04-14T15:13:55.166 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.055211019, 2024-04-14T15:13:55.166 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:13:55.166 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:13:55.166 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:13:55.166 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:13:55.167 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.24868", 2024-04-14T15:13:55.167 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/1684644655", 2024-04-14T15:13:55.167 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 5 2024-04-14T15:13:55.167 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.167 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:13:55.167 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.167 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:13:55.167 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.125668+0000", 2024-04-14T15:13:55.167 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:13:55.167 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.167 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.167 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:13:55.167 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.125668+0000", 2024-04-14T15:13:55.167 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:13:55.168 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.168 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.168 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:13:55.168 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.125670+0000", 2024-04-14T15:13:55.168 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.891e-06 2024-04-14T15:13:55.168 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.168 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.168 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:13:55.168 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.125678+0000", 2024-04-14T15:13:55.168 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1100000000000003e-06 2024-04-14T15:13:55.168 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.168 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.168 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:13:55.168 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.125688+0000", 2024-04-14T15:13:55.169 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0685e-05 2024-04-14T15:13:55.169 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.169 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.169 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:13:55.169 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.125699+0000", 2024-04-14T15:13:55.169 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0411000000000001e-05 2024-04-14T15:13:55.169 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.169 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.169 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:13:55.169 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.125776+0000", 2024-04-14T15:13:55.169 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.6784000000000002e-05 2024-04-14T15:13:55.169 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.169 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.169 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:13:55.170 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.125893+0000", 2024-04-14T15:13:55.170 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000117534 2024-04-14T15:13:55.170 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.170 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.170 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,5", 2024-04-14T15:13:55.170 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.126048+0000", 2024-04-14T15:13:55.170 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000155086 2024-04-14T15:13:55.170 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.170 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.170 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:13:55.170 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.126561+0000", 2024-04-14T15:13:55.170 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00051295599999999996 2024-04-14T15:13:55.170 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.171 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.171 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:13:55.171 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.158390+0000", 2024-04-14T15:13:55.171 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.031829059999999999 2024-04-14T15:13:55.171 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.171 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.171 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:13:55.171 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.180774+0000", 2024-04-14T15:13:55.171 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.022384035 2024-04-14T15:13:55.171 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.171 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.171 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:13:55.172 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.180833+0000", 2024-04-14T15:13:55.172 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.8841000000000001e-05 2024-04-14T15:13:55.172 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.172 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.172 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:13:55.172 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.180879+0000", 2024-04-14T15:13:55.172 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5626e-05 2024-04-14T15:13:55.172 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:13:55.172 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:13:55.172 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:13:55.172 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.172 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.172 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25210.0:2 59.13 59:cd2ae97d:::foo1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-04-14T15:13:55.173 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:58.126711+0000", 2024-04-14T15:13:55.173 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 56.980813961000003, 2024-04-14T15:13:55.173 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.19058456, 2024-04-14T15:13:55.173 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:13:55.173 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:13:55.173 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:13:55.173 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:13:55.173 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25210", 2024-04-14T15:13:55.173 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/4158655387", 2024-04-14T15:13:55.173 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 2 2024-04-14T15:13:55.173 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.173 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:13:55.174 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.174 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:13:55.174 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.126711+0000", 2024-04-14T15:13:55.174 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:13:55.174 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.174 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.174 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:13:55.174 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.126711+0000", 2024-04-14T15:13:55.174 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:13:55.174 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.174 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.174 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:13:55.175 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.126712+0000", 2024-04-14T15:13:55.175 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2470000000000001e-06 2024-04-14T15:13:55.175 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.175 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.176 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:13:55.176 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.126717+0000", 2024-04-14T15:13:55.176 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.0520000000000001e-06 2024-04-14T15:13:55.176 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.176 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.176 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:13:55.176 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.126727+0000", 2024-04-14T15:13:55.176 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0189e-05 2024-04-14T15:13:55.176 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.176 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.176 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:13:55.176 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.126736+0000", 2024-04-14T15:13:55.176 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3119999999999994e-06 2024-04-14T15:13:55.176 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.177 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.177 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:13:55.177 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.126796+0000", 2024-04-14T15:13:55.177 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0158000000000002e-05 2024-04-14T15:13:55.177 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.178 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.178 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:13:55.178 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.126804+0000", 2024-04-14T15:13:55.178 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1899999999999995e-06 2024-04-14T15:13:55.178 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.178 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.178 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:13:55.178 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.295771+0000", 2024-04-14T15:13:55.178 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1689666919999999 2024-04-14T15:13:55.178 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.178 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.178 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:13:55.178 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.295875+0000", 2024-04-14T15:13:55.179 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000104329 2024-04-14T15:13:55.179 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.179 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.179 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,6", 2024-04-14T15:13:55.179 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.295967+0000", 2024-04-14T15:13:55.179 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.1788999999999998e-05 2024-04-14T15:13:55.179 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.179 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.179 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:13:55.179 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.296429+0000", 2024-04-14T15:13:55.179 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000462764 2024-04-14T15:13:55.179 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.179 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.179 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:13:55.180 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.302107+0000", 2024-04-14T15:13:55.180 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0056771829999999997 2024-04-14T15:13:55.180 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.180 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.180 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:13:55.180 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.317204+0000", 2024-04-14T15:13:55.180 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.015097203 2024-04-14T15:13:55.180 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.180 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.180 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:13:55.180 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.317244+0000", 2024-04-14T15:13:55.180 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9978000000000002e-05 2024-04-14T15:13:55.180 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.180 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.181 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:13:55.181 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.317295+0000", 2024-04-14T15:13:55.181 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.1474000000000002e-05 2024-04-14T15:13:55.181 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:13:55.181 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:13:55.181 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:13:55.181 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.181 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.181 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.24853.0:2069 16.e e111/107 16:70021a00:test-rados-api-smithi090-93601-17::big:head v 111'2, mlcod=0'0)", 2024-04-14T15:13:55.181 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:59.377958+0000", 2024-04-14T15:13:55.181 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 55.729566499000001, 2024-04-14T15:13:55.181 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.050254007000000003, 2024-04-14T15:13:55.181 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:13:55.181 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:13:55.182 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:13:55.182 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:13:55.182 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.182 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:13:55.182 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.377958+0000", 2024-04-14T15:13:55.182 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:13:55.182 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.182 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.182 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:13:55.182 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.377958+0000", 2024-04-14T15:13:55.182 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:13:55.182 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.182 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.182 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:13:55.183 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.377961+0000", 2024-04-14T15:13:55.183 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5299999999999999e-06 2024-04-14T15:13:55.183 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.183 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.183 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:13:55.183 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.401281+0000", 2024-04-14T15:13:55.183 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0233206 2024-04-14T15:13:55.183 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.183 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.183 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:13:55.183 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.401294+0000", 2024-04-14T15:13:55.183 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2187e-05 2024-04-14T15:13:55.183 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.183 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.184 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:13:55.184 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.401301+0000", 2024-04-14T15:13:55.184 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.4120000000000004e-06 2024-04-14T15:13:55.184 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.184 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.184 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:13:55.184 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.401345+0000", 2024-04-14T15:13:55.184 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.4450000000000003e-05 2024-04-14T15:13:55.184 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.184 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.184 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:13:55.184 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.401362+0000", 2024-04-14T15:13:55.184 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7042999999999999e-05 2024-04-14T15:13:55.184 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.185 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.185 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:13:55.185 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.428165+0000", 2024-04-14T15:13:55.185 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.026802277999999999 2024-04-14T15:13:55.185 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.185 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.185 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:13:55.185 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.428212+0000", 2024-04-14T15:13:55.185 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7506999999999998e-05 2024-04-14T15:13:55.185 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:13:55.185 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:13:55.185 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:13:55.185 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.186 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.186 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.24853.0:2074 16.c e111/107 16:31d79ce4:test-rados-api-smithi090-93601-17::big3:head v 111'2, mlcod=0'0)", 2024-04-14T15:13:55.186 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:59.574768+0000", 2024-04-14T15:13:55.186 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 55.532756362999997, 2024-04-14T15:13:55.186 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.062724147999999993, 2024-04-14T15:13:55.186 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:13:55.186 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:13:55.186 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:13:55.186 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:13:55.186 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.186 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:13:55.186 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.574768+0000", 2024-04-14T15:13:55.186 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:13:55.187 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.187 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.187 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:13:55.187 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.574768+0000", 2024-04-14T15:13:55.187 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:13:55.187 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.187 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.187 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:13:55.187 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.574772+0000", 2024-04-14T15:13:55.187 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.3409999999999999e-06 2024-04-14T15:13:55.187 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.187 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.187 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:13:55.187 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.584618+0000", 2024-04-14T15:13:55.188 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00984639 2024-04-14T15:13:55.188 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.188 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.188 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:13:55.188 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.584639+0000", 2024-04-14T15:13:55.188 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1053999999999999e-05 2024-04-14T15:13:55.188 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.188 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.188 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:13:55.188 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.584651+0000", 2024-04-14T15:13:55.188 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2354000000000001e-05 2024-04-14T15:13:55.188 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.188 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.188 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:13:55.189 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.584726+0000", 2024-04-14T15:13:55.189 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.4559000000000002e-05 2024-04-14T15:13:55.189 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.189 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.189 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:13:55.189 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.584745+0000", 2024-04-14T15:13:55.189 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8997000000000001e-05 2024-04-14T15:13:55.189 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.189 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.189 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:13:55.190 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.637426+0000", 2024-04-14T15:13:55.190 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.052681095999999997 2024-04-14T15:13:55.190 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.190 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.190 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:13:55.190 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.637492+0000", 2024-04-14T15:13:55.190 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.6357000000000002e-05 2024-04-14T15:13:55.190 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:13:55.190 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:13:55.190 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:13:55.190 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.191 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.191 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25354.0:2 72.0 72:0384ab8e:::oid:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-04-14T15:13:55.191 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:13:00.119235+0000", 2024-04-14T15:13:55.191 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 54.988289272999999, 2024-04-14T15:13:55.191 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.81347187600000004, 2024-04-14T15:13:55.191 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:13:55.191 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:13:55.191 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:13:55.191 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:13:55.191 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25354", 2024-04-14T15:13:55.191 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/3758511316", 2024-04-14T15:13:55.191 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 2 2024-04-14T15:13:55.191 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.192 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:13:55.192 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.192 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:13:55.192 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.119235+0000", 2024-04-14T15:13:55.192 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:13:55.192 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.192 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.192 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:13:55.192 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.119235+0000", 2024-04-14T15:13:55.192 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:13:55.192 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.192 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.192 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:13:55.193 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.119237+0000", 2024-04-14T15:13:55.193 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0810000000000001e-06 2024-04-14T15:13:55.193 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.193 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.193 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:13:55.193 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.119243+0000", 2024-04-14T15:13:55.193 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9680000000000003e-06 2024-04-14T15:13:55.193 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.193 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.193 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:13:55.193 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.126617+0000", 2024-04-14T15:13:55.193 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0073734839999999996 2024-04-14T15:13:55.193 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.193 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.194 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:13:55.194 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.126643+0000", 2024-04-14T15:13:55.194 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6350999999999999e-05 2024-04-14T15:13:55.194 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.194 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.194 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:13:55.194 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.127309+0000", 2024-04-14T15:13:55.194 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00066571200000000005 2024-04-14T15:13:55.194 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.194 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.194 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:13:55.194 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.127317+0000", 2024-04-14T15:13:55.194 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3219999999999993e-06 2024-04-14T15:13:55.195 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.195 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.195 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:13:55.195 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.891898+0000", 2024-04-14T15:13:55.195 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.76458101700000003 2024-04-14T15:13:55.195 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.195 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.195 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:13:55.195 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.892011+0000", 2024-04-14T15:13:55.195 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000112891 2024-04-14T15:13:55.195 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.195 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.196 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1,7", 2024-04-14T15:13:55.196 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.892135+0000", 2024-04-14T15:13:55.196 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012421700000000001 2024-04-14T15:13:55.196 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.196 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.196 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:13:55.196 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.892414+0000", 2024-04-14T15:13:55.196 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00027872999999999998 2024-04-14T15:13:55.196 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.196 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.196 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:13:55.196 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.897424+0000", 2024-04-14T15:13:55.196 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0050095629999999999 2024-04-14T15:13:55.197 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.197 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.197 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:13:55.197 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.932615+0000", 2024-04-14T15:13:55.197 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.035191564000000002 2024-04-14T15:13:55.197 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.197 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.197 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:13:55.197 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.932666+0000", 2024-04-14T15:13:55.197 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.0560999999999999e-05 2024-04-14T15:13:55.197 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.197 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.197 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:13:55.198 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.932707+0000", 2024-04-14T15:13:55.198 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1415000000000002e-05 2024-04-14T15:13:55.198 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:13:55.198 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:13:55.198 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:13:55.198 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.198 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.198 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25684.0:110 135.17 135:eb4a7a6a:test-rados-api-smithi090-93652-10::foo:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e134)", 2024-04-14T15:13:55.198 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:13:21.824891+0000", 2024-04-14T15:13:55.198 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 33.282633976, 2024-04-14T15:13:55.198 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.14413627700000001, 2024-04-14T15:13:55.198 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:13:55.198 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:13:55.199 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-04-14T15:13:55.199 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:13:55.199 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25684", 2024-04-14T15:13:55.199 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/1109245683", 2024-04-14T15:13:55.199 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 110 2024-04-14T15:13:55.199 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.199 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:13:55.199 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.199 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:13:55.200 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:21.824891+0000", 2024-04-14T15:13:55.200 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:13:55.200 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.200 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.200 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:13:55.200 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:21.824891+0000", 2024-04-14T15:13:55.200 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:13:55.200 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.200 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.200 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:13:55.200 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:21.824892+0000", 2024-04-14T15:13:55.200 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.404e-06 2024-04-14T15:13:55.200 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.201 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.201 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:13:55.201 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:21.824898+0000", 2024-04-14T15:13:55.201 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5029999999999997e-06 2024-04-14T15:13:55.201 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.201 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.201 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:13:55.201 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:21.824906+0000", 2024-04-14T15:13:55.201 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8589999999999994e-06 2024-04-14T15:13:55.201 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.201 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.201 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:13:55.201 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:21.824916+0000", 2024-04-14T15:13:55.201 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.6479999999999998e-06 2024-04-14T15:13:55.202 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.202 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.202 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:13:55.202 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:21.824971+0000", 2024-04-14T15:13:55.202 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4805999999999997e-05 2024-04-14T15:13:55.202 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.202 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.202 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for blocked object", 2024-04-14T15:13:55.202 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:21.825199+0000", 2024-04-14T15:13:55.202 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00022818299999999999 2024-04-14T15:13:55.202 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.202 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.202 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:13:55.202 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:21.905705+0000", 2024-04-14T15:13:55.203 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.080505987000000001 2024-04-14T15:13:55.203 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.203 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.203 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for rw locks", 2024-04-14T15:13:55.203 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:21.905814+0000", 2024-04-14T15:13:55.203 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010869 2024-04-14T15:13:55.203 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.203 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.203 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:13:55.203 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:21.968732+0000", 2024-04-14T15:13:55.203 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.062918098000000006 2024-04-14T15:13:55.203 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.203 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.204 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:13:55.204 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:21.968825+0000", 2024-04-14T15:13:55.204 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.2849000000000005e-05 2024-04-14T15:13:55.204 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.204 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.204 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:13:55.204 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:21.969027+0000", 2024-04-14T15:13:55.204 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00020225000000000001 2024-04-14T15:13:55.204 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:13:55.204 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:13:55.204 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:13:55.204 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.204 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.204 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.36767.0:2 147.6 147:602f83fe:::foo:head [append 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-04-14T15:13:55.205 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:13:22.868465+0000", 2024-04-14T15:13:55.205 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 32.239059396999998, 2024-04-14T15:13:55.205 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.10357522299999999, 2024-04-14T15:13:55.205 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:13:55.205 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:13:55.205 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:13:55.205 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:13:55.205 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.36767", 2024-04-14T15:13:55.205 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/902879548", 2024-04-14T15:13:55.205 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 2 2024-04-14T15:13:55.205 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.205 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:13:55.205 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.205 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:13:55.206 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:22.868465+0000", 2024-04-14T15:13:55.206 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:13:55.206 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.206 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.206 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:13:55.206 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:22.868465+0000", 2024-04-14T15:13:55.206 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:13:55.206 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.206 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.206 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:13:55.206 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:22.868467+0000", 2024-04-14T15:13:55.206 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9130000000000001e-06 2024-04-14T15:13:55.206 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.207 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.207 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:13:55.207 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:22.868474+0000", 2024-04-14T15:13:55.207 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2899999999999997e-06 2024-04-14T15:13:55.207 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.207 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.207 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:13:55.207 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:22.868486+0000", 2024-04-14T15:13:55.207 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1823e-05 2024-04-14T15:13:55.207 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.207 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.207 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:13:55.207 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:22.868497+0000", 2024-04-14T15:13:55.207 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0927000000000001e-05 2024-04-14T15:13:55.208 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.208 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.208 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:13:55.208 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:22.868557+0000", 2024-04-14T15:13:55.208 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9996e-05 2024-04-14T15:13:55.208 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.208 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.208 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:13:55.208 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:22.868566+0000", 2024-04-14T15:13:55.208 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3720000000000005e-06 2024-04-14T15:13:55.208 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.208 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.208 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:13:55.209 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:22.917084+0000", 2024-04-14T15:13:55.209 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.048518110000000003 2024-04-14T15:13:55.209 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.209 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.209 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:13:55.209 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:22.917206+0000", 2024-04-14T15:13:55.209 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012209699999999999 2024-04-14T15:13:55.209 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.209 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.209 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 2,6", 2024-04-14T15:13:55.209 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:22.917361+0000", 2024-04-14T15:13:55.209 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000155511 2024-04-14T15:13:55.209 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.209 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.209 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:13:55.210 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:22.917927+0000", 2024-04-14T15:13:55.210 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00056553500000000004 2024-04-14T15:13:55.210 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.210 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.210 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:13:55.210 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:22.918182+0000", 2024-04-14T15:13:55.210 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00025522199999999999 2024-04-14T15:13:55.210 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.210 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.210 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:13:55.210 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:22.971948+0000", 2024-04-14T15:13:55.210 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.053765598999999997 2024-04-14T15:13:55.210 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.211 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.211 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:13:55.211 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:22.971997+0000", 2024-04-14T15:13:55.211 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8860000000000003e-05 2024-04-14T15:13:55.211 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.211 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.211 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:13:55.211 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:22.972041+0000", 2024-04-14T15:13:55.211 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3968000000000003e-05 2024-04-14T15:13:55.211 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:13:55.211 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:13:55.211 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:13:55.211 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.211 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.212 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.41476.0:2 151.0 151:0384ab8e:::oid:head [zero 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-04-14T15:13:55.212 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:13:24.859716+0000", 2024-04-14T15:13:55.212 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 30.247808550999999, 2024-04-14T15:13:55.212 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1201019210000001, 2024-04-14T15:13:55.212 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:13:55.212 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:13:55.212 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-04-14T15:13:55.212 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:13:55.212 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.41476", 2024-04-14T15:13:55.212 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/1075568278", 2024-04-14T15:13:55.212 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 2 2024-04-14T15:13:55.212 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.212 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:13:55.212 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.213 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:13:55.213 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:24.859716+0000", 2024-04-14T15:13:55.213 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:13:55.213 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.213 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.213 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:13:55.213 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:24.859716+0000", 2024-04-14T15:13:55.213 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:13:55.213 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.213 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.213 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:13:55.213 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:24.859718+0000", 2024-04-14T15:13:55.213 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.683e-06 2024-04-14T15:13:55.213 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.214 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.214 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:13:55.214 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:24.859723+0000", 2024-04-14T15:13:55.214 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3279999999999996e-06 2024-04-14T15:13:55.214 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.214 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.214 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:13:55.214 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:24.859732+0000", 2024-04-14T15:13:55.214 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.1810000000000007e-06 2024-04-14T15:13:55.214 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.214 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.215 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:13:55.216 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:13:55.216 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:24.859742+0000", 2024-04-14T15:13:55.216 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.2709999999999998e-06 2024-04-14T15:13:55.216 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.216 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.216 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:13:55.216 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:24.859790+0000", 2024-04-14T15:13:55.216 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8547000000000001e-05 2024-04-14T15:13:55.216 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.217 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.217 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:13:55.217 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:24.859799+0000", 2024-04-14T15:13:55.217 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.3109999999999995e-06 2024-04-14T15:13:55.217 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.217 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.217 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:13:55.217 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.887558+0000", 2024-04-14T15:13:55.217 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.027758723 2024-04-14T15:13:55.217 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.217 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.217 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:13:55.217 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.887660+0000", 2024-04-14T15:13:55.217 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010152799999999999 2024-04-14T15:13:55.218 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.218 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.218 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:13:55.218 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.979818+0000", 2024-04-14T15:13:55.218 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.092158349000000001 2024-04-14T15:13:55.218 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:13:55.218 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:13:55.218 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:13:55.218 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.218 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.218 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.41515.0:2 154.6 154:602f83fe:::foo:head [append 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-04-14T15:13:55.218 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:13:25.843012+0000", 2024-04-14T15:13:55.218 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 29.264512302, 2024-04-14T15:13:55.219 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.136860596, 2024-04-14T15:13:55.219 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:13:55.219 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:13:55.219 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:13:55.219 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:13:55.219 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.41515", 2024-04-14T15:13:55.219 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/1865301891", 2024-04-14T15:13:55.219 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 2 2024-04-14T15:13:55.219 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.219 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:13:55.219 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.219 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:13:55.219 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.843012+0000", 2024-04-14T15:13:55.219 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:13:55.219 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.220 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.220 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:13:55.220 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.843012+0000", 2024-04-14T15:13:55.220 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:13:55.220 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.220 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.220 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:13:55.220 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.843015+0000", 2024-04-14T15:13:55.220 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.159e-06 2024-04-14T15:13:55.220 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.220 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.220 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:13:55.220 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.843024+0000", 2024-04-14T15:13:55.221 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.1330000000000005e-06 2024-04-14T15:13:55.221 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.221 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.221 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:13:55.221 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.843033+0000", 2024-04-14T15:13:55.221 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.5149999999999995e-06 2024-04-14T15:13:55.221 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.221 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.221 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:13:55.221 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.843043+0000", 2024-04-14T15:13:55.221 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0268000000000001e-05 2024-04-14T15:13:55.221 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.221 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.221 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:13:55.222 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.843356+0000", 2024-04-14T15:13:55.222 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00031297200000000001 2024-04-14T15:13:55.222 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.222 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.222 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:13:55.222 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.843366+0000", 2024-04-14T15:13:55.222 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.2240000000000001e-06 2024-04-14T15:13:55.222 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.222 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.222 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:13:55.222 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.903587+0000", 2024-04-14T15:13:55.222 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.060221471999999998 2024-04-14T15:13:55.222 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.223 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.223 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:13:55.223 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.903709+0000", 2024-04-14T15:13:55.223 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012226500000000001 2024-04-14T15:13:55.223 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.223 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.223 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 2,3", 2024-04-14T15:13:55.223 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.903851+0000", 2024-04-14T15:13:55.223 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014190700000000001 2024-04-14T15:13:55.223 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.223 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.223 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:13:55.223 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.904351+0000", 2024-04-14T15:13:55.223 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00049993900000000003 2024-04-14T15:13:55.224 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.224 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.224 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:13:55.224 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.940363+0000", 2024-04-14T15:13:55.224 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.036011941999999998 2024-04-14T15:13:55.224 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.224 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.224 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:13:55.224 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.979696+0000", 2024-04-14T15:13:55.224 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.039332371999999997 2024-04-14T15:13:55.224 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.224 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.224 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:13:55.224 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.979838+0000", 2024-04-14T15:13:55.225 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014200000000000001 2024-04-14T15:13:55.225 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.225 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.225 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:13:55.225 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.979873+0000", 2024-04-14T15:13:55.225 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5428e-05 2024-04-14T15:13:55.225 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:13:55.225 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:13:55.225 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:13:55.225 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.225 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.225 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25684.0:247 110.5 110:a57cb8b9:test-rados-api-smithi090-93652-12::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-04-14T15:13:55.225 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:13:26.816857+0000", 2024-04-14T15:13:55.225 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 28.290667936999998, 2024-04-14T15:13:55.226 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.048269978999999998, 2024-04-14T15:13:55.226 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:13:55.226 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:13:55.226 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:13:55.226 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:13:55.226 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25684", 2024-04-14T15:13:55.226 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/1109245683", 2024-04-14T15:13:55.226 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 247 2024-04-14T15:13:55.226 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.226 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:13:55.227 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.227 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:13:55.227 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.816857+0000", 2024-04-14T15:13:55.227 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:13:55.227 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.227 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.227 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:13:55.227 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.816857+0000", 2024-04-14T15:13:55.227 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:13:55.227 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.227 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.227 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:13:55.227 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.816859+0000", 2024-04-14T15:13:55.227 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.413e-06 2024-04-14T15:13:55.228 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.228 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.228 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:13:55.228 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.816869+0000", 2024-04-14T15:13:55.228 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.9969999999999993e-06 2024-04-14T15:13:55.228 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.228 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.228 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:13:55.228 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.816881+0000", 2024-04-14T15:13:55.228 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2072999999999999e-05 2024-04-14T15:13:55.228 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.228 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.228 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:13:55.228 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.816892+0000", 2024-04-14T15:13:55.229 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0376e-05 2024-04-14T15:13:55.229 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.229 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.229 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:13:55.229 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.840701+0000", 2024-04-14T15:13:55.229 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.023809737000000001 2024-04-14T15:13:55.229 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.229 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.229 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:13:55.229 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.840836+0000", 2024-04-14T15:13:55.230 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013419500000000001 2024-04-14T15:13:55.230 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.230 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.230 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 5,6", 2024-04-14T15:13:55.230 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.840979+0000", 2024-04-14T15:13:55.230 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000143838 2024-04-14T15:13:55.230 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.230 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.230 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:13:55.230 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.847401+0000", 2024-04-14T15:13:55.230 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0064213810000000003 2024-04-14T15:13:55.230 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.230 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.231 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:13:55.231 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.857277+0000", 2024-04-14T15:13:55.231 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0098764170000000002 2024-04-14T15:13:55.231 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.231 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.231 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:13:55.231 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.864967+0000", 2024-04-14T15:13:55.231 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.007689825 2024-04-14T15:13:55.231 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.232 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.232 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:13:55.232 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.865069+0000", 2024-04-14T15:13:55.232 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010176800000000001 2024-04-14T15:13:55.232 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.232 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.232 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:13:55.232 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.865127+0000", 2024-04-14T15:13:55.232 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7958999999999999e-05 2024-04-14T15:13:55.232 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:13:55.232 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:13:55.232 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:13:55.232 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.233 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.233 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.42382.0:3 157.0 157:0384ab8e:::oid:head [write 0~128] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-04-14T15:13:55.233 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:13:26.844811+0000", 2024-04-14T15:13:55.233 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 28.262713777999998, 2024-04-14T15:13:55.233 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.152799729, 2024-04-14T15:13:55.233 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:13:55.233 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:13:55.233 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:13:55.233 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:13:55.233 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.42382", 2024-04-14T15:13:55.233 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/1837092947", 2024-04-14T15:13:55.233 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 3 2024-04-14T15:13:55.233 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.233 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:13:55.234 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.234 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:13:55.234 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.844811+0000", 2024-04-14T15:13:55.234 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:13:55.234 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.234 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.234 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:13:55.234 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.844811+0000", 2024-04-14T15:13:55.234 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:13:55.234 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.234 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.234 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:13:55.234 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.847180+0000", 2024-04-14T15:13:55.234 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0023693730000000001 2024-04-14T15:13:55.235 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.235 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.235 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:13:55.235 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.847195+0000", 2024-04-14T15:13:55.235 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.489e-05 2024-04-14T15:13:55.235 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.235 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.235 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:13:55.235 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.847214+0000", 2024-04-14T15:13:55.235 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8564e-05 2024-04-14T15:13:55.235 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.235 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.235 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:13:55.236 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.847225+0000", 2024-04-14T15:13:55.236 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0786e-05 2024-04-14T15:13:55.236 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.236 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.236 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:13:55.236 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.868351+0000", 2024-04-14T15:13:55.236 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.021126823999999999 2024-04-14T15:13:55.236 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.236 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.236 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:13:55.236 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.868370+0000", 2024-04-14T15:13:55.236 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8508e-05 2024-04-14T15:13:55.236 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.236 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.237 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:13:55.237 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.961204+0000", 2024-04-14T15:13:55.237 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.092834633999999999 2024-04-14T15:13:55.237 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.237 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.237 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:13:55.237 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.961327+0000", 2024-04-14T15:13:55.237 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012294199999999999 2024-04-14T15:13:55.237 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.237 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.237 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,6", 2024-04-14T15:13:55.237 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.961485+0000", 2024-04-14T15:13:55.237 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015743699999999999 2024-04-14T15:13:55.237 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.238 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.238 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:13:55.238 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.961970+0000", 2024-04-14T15:13:55.238 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00048496900000000003 2024-04-14T15:13:55.238 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.238 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.238 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:13:55.238 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.962400+0000", 2024-04-14T15:13:55.238 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00043008599999999999 2024-04-14T15:13:55.238 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.238 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.238 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:13:55.238 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.997503+0000", 2024-04-14T15:13:55.239 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.035103500000000003 2024-04-14T15:13:55.239 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.239 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.239 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:13:55.239 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.997559+0000", 2024-04-14T15:13:55.239 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5831000000000001e-05 2024-04-14T15:13:55.239 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.239 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.239 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:13:55.239 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.997611+0000", 2024-04-14T15:13:55.239 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.1384999999999997e-05 2024-04-14T15:13:55.239 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:13:55.239 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:13:55.239 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:13:55.240 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.240 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.240 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.42382.0:8 157.0 157:0384ab8e:::oid:head [write 128~128] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-04-14T15:13:55.240 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:13:27.847441+0000", 2024-04-14T15:13:55.240 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 27.260084105000001, 2024-04-14T15:13:55.240 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.059003879000000002, 2024-04-14T15:13:55.240 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:13:55.240 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:13:55.240 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:13:55.240 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:13:55.241 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.42382", 2024-04-14T15:13:55.241 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/1837092947", 2024-04-14T15:13:55.241 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 8 2024-04-14T15:13:55.241 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.241 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:13:55.241 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.241 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:13:55.241 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:27.847441+0000", 2024-04-14T15:13:55.241 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:13:55.241 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.241 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.241 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:13:55.241 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:27.847441+0000", 2024-04-14T15:13:55.241 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:13:55.242 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.242 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.242 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:13:55.242 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:27.847443+0000", 2024-04-14T15:13:55.242 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0590000000000001e-06 2024-04-14T15:13:55.242 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.242 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.242 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:13:55.242 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:27.847449+0000", 2024-04-14T15:13:55.242 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.7129999999999999e-06 2024-04-14T15:13:55.242 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.242 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.242 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:13:55.243 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:27.847460+0000", 2024-04-14T15:13:55.243 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0359e-05 2024-04-14T15:13:55.243 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.243 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.243 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:13:55.243 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:27.847472+0000", 2024-04-14T15:13:55.243 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.184e-05 2024-04-14T15:13:55.243 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.243 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.243 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:13:55.243 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:27.854881+0000", 2024-04-14T15:13:55.243 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0074090730000000004 2024-04-14T15:13:55.243 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.243 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.243 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:13:55.244 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:27.854956+0000", 2024-04-14T15:13:55.244 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5054000000000003e-05 2024-04-14T15:13:55.244 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.244 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.244 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,6", 2024-04-14T15:13:55.244 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:27.856348+0000", 2024-04-14T15:13:55.244 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001392643 2024-04-14T15:13:55.244 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.244 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.244 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:13:55.244 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:27.857818+0000", 2024-04-14T15:13:55.244 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0014697 2024-04-14T15:13:55.244 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.245 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.245 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:13:55.245 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:27.858388+0000", 2024-04-14T15:13:55.245 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00057031499999999995 2024-04-14T15:13:55.245 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.245 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.245 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:13:55.245 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:27.906371+0000", 2024-04-14T15:13:55.245 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.047982524999999998 2024-04-14T15:13:55.245 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.245 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.245 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:13:55.245 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:27.906406+0000", 2024-04-14T15:13:55.245 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5475999999999997e-05 2024-04-14T15:13:55.246 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:13:55.246 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:13:55.246 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:13:55.246 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:27.906444+0000", 2024-04-14T15:13:55.246 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8121999999999999e-05 2024-04-14T15:13:55.246 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:13:55.246 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:13:55.246 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:13:55.246 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:13:55.246 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:13:55.246 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-04-14T15:13:55.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:55.273 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:55.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:13:55.373 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:13:55.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:55.398 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:55.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:13:55.499 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:13:55.517 INFO:tasks.thrashosds.thrasher:j is {'epoch': 164, 'fsid': '0e288098-fa70-11ee-bc8e-c7b262605968', 'created': '2024-04-14T15:04:05.808313+0000', 'modified': '2024-04-14T15:13:51.830791+0000', 'last_up_change': '2024-04-14T15:09:29.769352+0000', 'last_in_change': '2024-04-14T15:09:11.803164+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 34, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 189, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-14T15:07:23.959628+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '104', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9022312164306641, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-04-14T15:09:41.608420+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9022312164306641, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 3, 'pool_name': 'datapool', 'create_time': '2024-04-14T15:09:41.744391+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 69, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 0.9022312164306641, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-04-14T15:09:43.088157+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9022312164306641, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-04-14T15:09:45.042722+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9022312164306641, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-04-14T15:09:47.111336+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.9022312164306641, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 11, 'pool_name': 'ceph_test_rados_api_asio', 'create_time': '2024-04-14T15:12:54.225671+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.9022312164306641, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 13, 'pool_name': 'LibRadosList_smithi090-93444-1', 'create_time': '2024-04-14T15:12:54.244611+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '13.1f', 'ready_epoch': 164, 'last_epoch_started': 163, 'last_epoch_clean': 163, 'source_version': "109'5", 'target_version': "0'0"}, 'last_change': '164', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '164', 'last_force_op_resend_preluminous': '162', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.059999942779541, 'score_stable': 2.059999942779541, 'optimal_score': 1, 'raw_score_acting': 2.059999942779541, 'raw_score_stable': 2.059999942779541, 'primary_affinity_weighted': 0.9022312164306641, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 16, 'pool_name': 'LibRadosMiscPP_smithi090-93601-1', 'create_time': '2024-04-14T15:12:54.300251+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9022312164306641, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 110, 'pool_name': 'test-rados-api-smithi090-93652-6', 'create_time': '2024-04-14T15:13:07.960794+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '155', 'last_force_op_resend': '155', 'last_force_op_resend_prenautilus': '155', 'last_force_op_resend_preluminous': '155', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 6, 'snap_epoch': 154, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [172], 'tier_of': -1, 'read_tier': 172, 'write_tier': 172, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9022312164306641, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 172, 'pool_name': 'test-rados-api-smithi090-93652-13', 'create_time': '2024-04-14T15:13:34.849790+0000', 'flags': 8201, 'flags_names': 'hashpspool,incomplete_clones,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '156', 'last_force_op_resend': '155', 'last_force_op_resend_prenautilus': '155', 'last_force_op_resend_preluminous': '155', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 6, 'snap_epoch': 154, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': 110, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'writeback', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9022312164306641, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 186, 'pool_name': 'ReusePurgedSnapsmithi090-94770-11', 'create_time': '2024-04-14T15:13:48.846489+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '164', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 164, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9022312164306641, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 187, 'pool_name': 'test-rados-api-smithi090-93724-15', 'create_time': '2024-04-14T15:13:49.891851+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '164', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9022312164306641, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 188, 'pool_name': 'OperateMtime_smithi090-93342-20', 'create_time': '2024-04-14T15:13:50.904113+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '164', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 0.9022312164306641, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 189, 'pool_name': 'FlushAsyncPP_smithi090-93363-15', 'create_time': '2024-04-14T15:13:50.909692+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '164', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.9022312164306641, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}], 'osds': [{'osd': 0, 'uuid': '76772cfc-89d1-4013-b617-32284ed8dfd9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 163, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6801', 'nonce': 3761806885}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6802', 'nonce': 3761806885}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6804', 'nonce': 3761806885}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6803', 'nonce': 3761806885}]}, 'public_addr': '172.21.15.90:6801/3761806885', 'cluster_addr': '172.21.15.90:6802/3761806885', 'heartbeat_back_addr': '172.21.15.90:6804/3761806885', 'heartbeat_front_addr': '172.21.15.90:6803/3761806885', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'de4a7fa2-7d57-4ff3-a4fa-28c0cde8ef6c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 163, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6805', 'nonce': 4170361699}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6806', 'nonce': 4170361699}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6808', 'nonce': 4170361699}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6807', 'nonce': 4170361699}]}, 'public_addr': '172.21.15.90:6805/4170361699', 'cluster_addr': '172.21.15.90:6806/4170361699', 'heartbeat_back_addr': '172.21.15.90:6808/4170361699', 'heartbeat_front_addr': '172.21.15.90:6807/4170361699', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '9c39b5ff-6175-46ba-9d71-370581c6ff3d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 163, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6809', 'nonce': 1686184251}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6810', 'nonce': 1686184251}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6812', 'nonce': 1686184251}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6811', 'nonce': 1686184251}]}, 'public_addr': '172.21.15.90:6809/1686184251', 'cluster_addr': '172.21.15.90:6810/1686184251', 'heartbeat_back_addr': '172.21.15.90:6812/1686184251', 'heartbeat_front_addr': '172.21.15.90:6811/1686184251', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'efd2ca45-5b80-4013-a571-499e2cb5b2e8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.2178497314453125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 163, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6813', 'nonce': 2270166774}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6814', 'nonce': 2270166774}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6816', 'nonce': 2270166774}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6815', 'nonce': 2270166774}]}, 'public_addr': '172.21.15.90:6813/2270166774', 'cluster_addr': '172.21.15.90:6814/2270166774', 'heartbeat_back_addr': '172.21.15.90:6816/2270166774', 'heartbeat_front_addr': '172.21.15.90:6815/2270166774', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'e3d9013d-5eb6-433c-96b3-3fc38ba7ac8d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 163, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6800', 'nonce': 3887549733}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6801', 'nonce': 3887549733}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6803', 'nonce': 3887549733}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6802', 'nonce': 3887549733}]}, 'public_addr': '172.21.15.92:6800/3887549733', 'cluster_addr': '172.21.15.92:6801/3887549733', 'heartbeat_back_addr': '172.21.15.92:6803/3887549733', 'heartbeat_front_addr': '172.21.15.92:6802/3887549733', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '1ec41805-6f82-4139-81bc-75305ba9e958', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 41, 'up_thru': 163, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6804', 'nonce': 3244561440}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6805', 'nonce': 3244561440}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6807', 'nonce': 3244561440}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6806', 'nonce': 3244561440}]}, 'public_addr': '172.21.15.92:6804/3244561440', 'cluster_addr': '172.21.15.92:6805/3244561440', 'heartbeat_back_addr': '172.21.15.92:6807/3244561440', 'heartbeat_front_addr': '172.21.15.92:6806/3244561440', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'ca977ab2-bf8d-4b3a-ae5e-0efa5bc7a3ad', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 49, 'up_thru': 163, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6808', 'nonce': 636125963}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6809', 'nonce': 636125963}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6811', 'nonce': 636125963}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6810', 'nonce': 636125963}]}, 'public_addr': '172.21.15.92:6808/636125963', 'cluster_addr': '172.21.15.92:6809/636125963', 'heartbeat_back_addr': '172.21.15.92:6811/636125963', 'heartbeat_front_addr': '172.21.15.92:6810/636125963', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'e5354645-79ec-4d4c-b6e5-f9b0653915ee', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 58, 'up_thru': 163, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6812', 'nonce': 1773706716}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6813', 'nonce': 1773706716}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6815', 'nonce': 1773706716}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6814', 'nonce': 1773706716}]}, 'public_addr': '172.21.15.92:6812/1773706716', 'cluster_addr': '172.21.15.92:6813/1773706716', 'heartbeat_back_addr': '172.21.15.92:6815/1773706716', 'heartbeat_front_addr': '172.21.15.92:6814/1773706716', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-14T15:13:12.372581+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-14T15:12:44.215601+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-14T15:13:13.389324+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-14T15:13:20.899272+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-14T15:13:09.817510+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-14T15:13:49.210268+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-14T15:12:41.863374+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-14T15:13:05.127218+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '2.1c', 'mappings': [{'from': 7, 'to': 4}]}, {'pgid': '4.2', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1f', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.90:0/508768479': '2024-04-15T15:10:22.918674+0000', '172.21.15.90:0/1823072952': '2024-04-15T15:10:22.918674+0000', '172.21.15.90:6800/491071118': '2024-04-15T15:10:22.918674+0000', '172.21.15.90:0/1337064035': '2024-04-15T15:10:22.918674+0000', '172.21.15.90:0/769462306': '2024-04-15T15:10:22.918674+0000', '172.21.15.90:0/1139868610': '2024-04-15T15:05:05.882361+0000', '172.21.15.90:0/93763408': '2024-04-15T15:10:22.918674+0000', '172.21.15.90:0/977818204': '2024-04-15T15:04:39.251711+0000', '172.21.15.90:6800/941091171': '2024-04-15T15:05:05.882361+0000', '172.21.15.90:0/3671453561': '2024-04-15T15:05:05.882361+0000', '172.21.15.90:0/2611856200': '2024-04-15T15:05:05.882361+0000', '172.21.15.90:6800/2306258962': '2024-04-15T15:04:39.251711+0000', '172.21.15.90:0/4051692380': '2024-04-15T15:04:39.251711+0000', '172.21.15.90:0/2681502561': '2024-04-15T15:04:39.251711+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 17, 'snaps': [{'begin': 2, 'length': 5}]}, {'pool': 18, 'snaps': [{'begin': 2, 'length': 3}]}, {'pool': 39, 'snaps': [{'begin': 1, 'length': 1}]}, {'pool': 72, 'snaps': [{'begin': 1, 'length': 1}]}, {'pool': 99, 'snaps': [{'begin': 4, 'length': 2}, {'begin': 7, 'length': 2}]}, {'pool': 109, 'snaps': [{'begin': 1, 'length': 1}]}, {'pool': 128, 'snaps': [{'begin': 1, 'length': 2}]}, {'pool': 142, 'snaps': [{'begin': 3, 'length': 1}]}, {'pool': 149, 'snaps': [{'begin': 1, 'length': 4}]}, {'pool': 156, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 157, 'snaps': [{'begin': 2, 'length': 2}]}, {'pool': 167, 'snaps': [{'begin': 2, 'length': 3}]}, {'pool': 175, 'snaps': [{'begin': 2, 'length': 4}]}, {'pool': 176, 'snaps': [{'begin': 2, 'length': 2}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-14T15:13:55.517 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-04-14T15:13:55.526 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:55.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:55.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:13:55.628 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:13:55.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:55.655 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:55.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:13:55.756 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:13:55.781 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:55.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:55.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:13:55.883 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:13:55.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:55.908 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:55.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:13:56.009 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:13:56.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:56.033 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:56.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:13:56.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:55 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:13:56.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:55 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3868448150' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi090-93724-16"}]: dispatch 2024-04-14T15:13:56.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:55 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3868448150' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi090-93724-16"}]: dispatch 2024-04-14T15:13:56.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:55 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3868448150' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi090-93724-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:13:56.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:55 smithi090 ceph-mon[41670]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:13:56.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:55 smithi090 ceph-mon[41670]: from='client.46325 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi090-93363-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:56.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:55 smithi090 ceph-mon[41670]: from='client.48202 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi090-93342-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:56.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:55 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3868448150' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi090-93724-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:13:56.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:55 smithi090 ceph-mon[41670]: osdmap e168: 8 total, 8 up, 8 in 2024-04-14T15:13:56.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:55 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3868448150' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi090-93724-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi090-93724-16"}]: dispatch 2024-04-14T15:13:56.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:55 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:56.055 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:55 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:13:56.055 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:55 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3868448150' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi090-93724-16"}]: dispatch 2024-04-14T15:13:56.055 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:55 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3868448150' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi090-93724-16"}]: dispatch 2024-04-14T15:13:56.055 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:55 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3868448150' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi090-93724-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:13:56.055 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:55 smithi090 ceph-mon[30399]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:13:56.056 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:55 smithi090 ceph-mon[30399]: from='client.46325 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi090-93363-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:56.056 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:55 smithi090 ceph-mon[30399]: from='client.48202 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi090-93342-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:56.056 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:55 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3868448150' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi090-93724-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:13:56.056 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:55 smithi090 ceph-mon[30399]: osdmap e168: 8 total, 8 up, 8 in 2024-04-14T15:13:56.056 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:55 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3868448150' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi090-93724-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi090-93724-16"}]: dispatch 2024-04-14T15:13:56.056 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:55 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:56.134 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:13:56.157 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:56.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:56.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:13:56.227 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:55 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:13:56.227 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:55 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3868448150' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi090-93724-16"}]: dispatch 2024-04-14T15:13:56.227 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:55 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3868448150' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi090-93724-16"}]: dispatch 2024-04-14T15:13:56.227 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:55 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3868448150' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi090-93724-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:13:56.227 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:55 smithi092 ceph-mon[33966]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:13:56.227 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:55 smithi092 ceph-mon[33966]: from='client.46325 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi090-93363-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:56.227 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:55 smithi092 ceph-mon[33966]: from='client.48202 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi090-93342-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:56.227 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:55 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3868448150' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi090-93724-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:13:56.227 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:55 smithi092 ceph-mon[33966]: osdmap e168: 8 total, 8 up, 8 in 2024-04-14T15:13:56.227 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:55 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3868448150' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi090-93724-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi090-93724-16"}]: dispatch 2024-04-14T15:13:56.228 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:55 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:56.250 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-14T15:13:56.260 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:13:56.283 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:56.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:56.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:13:56.386 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:13:56.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:55.968+0000 7ff0af9f3700 1 -- 172.21.15.90:0/2357272914 >> v1:172.21.15.92:6789/0 conn(0x7ff0a810a6d0 legacy=0x7ff0a810cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:56.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:55.968+0000 7ff0af9f3700 1 -- 172.21.15.90:0/2357272914 shutdown_connections 2024-04-14T15:13:56.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:55.968+0000 7ff0af9f3700 1 -- 172.21.15.90:0/2357272914 >> 172.21.15.90:0/2357272914 conn(0x7ff0a8076aa0 msgr2=0x7ff0a8076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:13:56.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:55.968+0000 7ff0af9f3700 1 -- 172.21.15.90:0/2357272914 shutdown_connections 2024-04-14T15:13:56.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:55.968+0000 7ff0af9f3700 1 -- 172.21.15.90:0/2357272914 wait complete. 2024-04-14T15:13:56.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:55.969+0000 7ff0af9f3700 1 librados: starting msgr at 2024-04-14T15:13:56.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:55.969+0000 7ff0af9f3700 1 librados: starting objecter 2024-04-14T15:13:56.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:55.969+0000 7ff0af9f3700 1 Processor -- start 2024-04-14T15:13:56.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:55.969+0000 7ff0af9f3700 1 -- start start 2024-04-14T15:13:56.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:55.969+0000 7ff0af9f3700 1 librados: setting wanted keys 2024-04-14T15:13:56.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:55.969+0000 7ff0af9f3700 1 librados: calling monclient init 2024-04-14T15:13:56.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:55.969+0000 7ff0af9f3700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff0a81a9b90 con 0x7ff0a810a6d0 2024-04-14T15:13:56.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:55.969+0000 7ff0af9f3700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff0a81aadb0 con 0x7ff0a81023f0 2024-04-14T15:13:56.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:55.969+0000 7ff0af9f3700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff0a81abff0 con 0x7ff0a810e270 2024-04-14T15:13:56.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:55.969+0000 7ff0acf8e700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7ff0a810a6d0 0x7ff0a81a4ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:33808/0 (socket says 172.21.15.90:33808) 2024-04-14T15:13:56.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:55.969+0000 7ff0acf8e700 1 -- 172.21.15.90:0/2375375347 learned_addr learned my addr 172.21.15.90:0/2375375347 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:13:56.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:55.969+0000 7ff09e7fc700 10 client.?.objecter ms_handle_connect 0x7ff0a810a6d0 2024-04-14T15:13:56.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:55.969+0000 7ff09e7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:13:56.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:55.969+0000 7ff09e7fc700 10 client.?.objecter ms_handle_connect 0x7ff0a810e270 2024-04-14T15:13:56.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:55.969+0000 7ff09e7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:13:56.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:55.970+0000 7ff09e7fc700 10 client.?.objecter ms_handle_connect 0x7ff0a81023f0 2024-04-14T15:13:56.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:55.970+0000 7ff09e7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:13:56.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:55.970+0000 7ff09e7fc700 1 -- 172.21.15.90:0/2375375347 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2276303269 0 0) 0x7ff0a81a9b90 con 0x7ff0a810a6d0 2024-04-14T15:13:56.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:55.970+0000 7ff09e7fc700 1 -- 172.21.15.90:0/2375375347 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0940039c0 con 0x7ff0a810a6d0 2024-04-14T15:13:56.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:55.970+0000 7ff09e7fc700 1 -- 172.21.15.90:0/2375375347 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1381774839 0 0) 0x7ff0a81aadb0 con 0x7ff0a81023f0 2024-04-14T15:13:56.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:55.970+0000 7ff09e7fc700 1 -- 172.21.15.90:0/2375375347 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0a81a9b90 con 0x7ff0a81023f0 2024-04-14T15:13:56.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:55.970+0000 7ff09e7fc700 1 -- 172.21.15.90:0/2375375347 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1418990473 0 0) 0x7ff0a81abff0 con 0x7ff0a810e270 2024-04-14T15:13:56.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:55.970+0000 7ff09e7fc700 1 -- 172.21.15.90:0/2375375347 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0a81aadb0 con 0x7ff0a810e270 2024-04-14T15:13:56.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:55.970+0000 7ff09e7fc700 1 -- 172.21.15.90:0/2375375347 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1518123295 0 0) 0x7ff0a81a9b90 con 0x7ff0a81023f0 2024-04-14T15:13:56.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:55.970+0000 7ff09e7fc700 1 -- 172.21.15.90:0/2375375347 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0a81abff0 con 0x7ff0a81023f0 2024-04-14T15:13:56.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:55.970+0000 7ff09e7fc700 1 -- 172.21.15.90:0/2375375347 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7ff0a4001870 con 0x7ff0a81023f0 2024-04-14T15:13:56.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:55.970+0000 7ff09e7fc700 1 -- 172.21.15.90:0/2375375347 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3934720605 0 0) 0x7ff0940039c0 con 0x7ff0a810a6d0 2024-04-14T15:13:56.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:55.970+0000 7ff09e7fc700 1 -- 172.21.15.90:0/2375375347 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0a81a9b90 con 0x7ff0a810a6d0 2024-04-14T15:13:56.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:55.970+0000 7ff09e7fc700 1 -- 172.21.15.90:0/2375375347 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7ff0980043c0 con 0x7ff0a810a6d0 2024-04-14T15:13:56.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:55.970+0000 7ff09e7fc700 1 -- 172.21.15.90:0/2375375347 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2984656973 0 0) 0x7ff0a81aadb0 con 0x7ff0a810e270 2024-04-14T15:13:56.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:55.970+0000 7ff09e7fc700 1 -- 172.21.15.90:0/2375375347 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0940039c0 con 0x7ff0a810e270 2024-04-14T15:13:56.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:55.970+0000 7ff09e7fc700 1 -- 172.21.15.90:0/2375375347 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7ff0a0001470 con 0x7ff0a810e270 2024-04-14T15:13:56.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:55.970+0000 7ff09e7fc700 1 -- 172.21.15.90:0/2375375347 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4174160676 0 0) 0x7ff0a81abff0 con 0x7ff0a81023f0 2024-04-14T15:13:56.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:55.970+0000 7ff09e7fc700 1 -- 172.21.15.90:0/2375375347 >> v1:172.21.15.90:6790/0 conn(0x7ff0a810e270 legacy=0x7ff0a81a8250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:56.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:55.970+0000 7ff09e7fc700 1 -- 172.21.15.90:0/2375375347 >> v1:172.21.15.90:6789/0 conn(0x7ff0a810a6d0 legacy=0x7ff0a81a4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:56.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:55.970+0000 7ff09e7fc700 1 -- 172.21.15.90:0/2375375347 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0a81ad230 con 0x7ff0a81023f0 2024-04-14T15:13:56.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:55.970+0000 7ff0af9f3700 1 -- 172.21.15.90:0/2375375347 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff0a81ac220 con 0x7ff0a81023f0 2024-04-14T15:13:56.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:55.970+0000 7ff0af9f3700 10 client.48226.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:13:56.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:55.970+0000 7ff0af9f3700 1 -- 172.21.15.90:0/2375375347 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff0a81ac6e0 con 0x7ff0a81023f0 2024-04-14T15:13:56.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:55.970+0000 7ff0af9f3700 1 librados: init done 2024-04-14T15:13:56.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:55.970+0000 7ff09e7fc700 1 -- 172.21.15.90:0/2375375347 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7ff0a4003b40 con 0x7ff0a81023f0 2024-04-14T15:13:56.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:55.970+0000 7ff09e7fc700 1 -- 172.21.15.90:0/2375375347 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7ff0a4004be0 con 0x7ff0a81023f0 2024-04-14T15:13:56.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:55.971+0000 7ff0af9f3700 1 -- 172.21.15.90:0/2375375347 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff08c005120 con 0x7ff0a81023f0 2024-04-14T15:13:56.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:55.972+0000 7ff09e7fc700 1 -- 172.21.15.90:0/2375375347 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7ff0a401b870 con 0x7ff0a81023f0 2024-04-14T15:13:56.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:55.972+0000 7ff09e7fc700 1 -- 172.21.15.90:0/2375375347 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(168..168 src has 51..168) ==== 10176+0+0 (unknown 1032664547 0 0) 0x7ff0a408a030 con 0x7ff0a81023f0 2024-04-14T15:13:56.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:55.972+0000 7ff09e7fc700 10 client.48226.objecter ms_dispatch 0x7ff0a8001550 osd_map(168..168 src has 51..168) 2024-04-14T15:13:56.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:55.972+0000 7ff09e7fc700 3 client.48226.objecter handle_osd_map got epochs [168,168] > 0 2024-04-14T15:13:56.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:55.972+0000 7ff09e7fc700 3 client.48226.objecter handle_osd_map decoding full epoch 168 2024-04-14T15:13:56.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:55.972+0000 7ff09e7fc700 20 client.48226.objecter dump_active .. 0 homeless 2024-04-14T15:13:56.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:55.973+0000 7ff09e7fc700 10 client.48226.objecter ms_handle_connect 0x7ff094071390 2024-04-14T15:13:56.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:55.977+0000 7ff09e7fc700 1 -- 172.21.15.90:0/2375375347 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7ff0a4056260 con 0x7ff0a81023f0 2024-04-14T15:13:56.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:56.190+0000 7ff0af9f3700 1 -- 172.21.15.90:0/2375375347 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7ff08c005410 con 0x7ff0a81023f0 2024-04-14T15:13:56.409 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:56.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:56.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:13:56.488 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:13:56.511 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:13:56.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:56.536 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:56.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:13:56.636 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:13:56.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:56.660 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:56.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:13:56.762 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:13:56.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:56.788 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:56.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:13:56.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:56.821+0000 7ff09e7fc700 1 -- 172.21.15.90:0/2375375347 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v169) ==== 91+0+0 (unknown 3716300884 0 0) 0x7ff0a4059f10 con 0x7ff0a81023f0 2024-04-14T15:13:56.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:56.824+0000 7ff0af9f3700 10 librados: watch_flush enter 2024-04-14T15:13:56.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:56.827+0000 7ff0af9f3700 10 librados: watch_flush exit 2024-04-14T15:13:56.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:56.828+0000 7ff0af9f3700 20 client.48226.objecter shutdown clearing up homeless session... 2024-04-14T15:13:56.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:56.828+0000 7ff0af9f3700 10 client.48226.objecter successfully canceled tick 2024-04-14T15:13:56.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:56.829+0000 7ff0af9f3700 1 -- 172.21.15.90:0/2375375347 >> v1:172.21.15.90:6800/420935427 conn(0x7ff094071390 legacy=0x7ff094073850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:56.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:56.829+0000 7ff0af9f3700 1 -- 172.21.15.90:0/2375375347 >> v1:172.21.15.92:6789/0 conn(0x7ff0a81023f0 legacy=0x7ff0a8101890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:56.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:56.829+0000 7ff0af9f3700 1 -- 172.21.15.90:0/2375375347 shutdown_connections 2024-04-14T15:13:56.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:56.829+0000 7ff0af9f3700 1 -- 172.21.15.90:0/2375375347 >> 172.21.15.90:0/2375375347 conn(0x7ff0a8076aa0 msgr2=0x7ff0a810e6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:13:56.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:56.829+0000 7ff0af9f3700 1 -- 172.21.15.90:0/2375375347 shutdown_connections 2024-04-14T15:13:56.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:56.829+0000 7ff0af9f3700 1 -- 172.21.15.90:0/2375375347 wait complete. 2024-04-14T15:13:56.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:56.831+0000 7ff0af9f3700 1 librados: shutdown 2024-04-14T15:13:56.837 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is unset 2024-04-14T15:13:56.889 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:13:56.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:56.913 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:56.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:13:57.014 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:13:57.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:57.039 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:57.039 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:13:57.140 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:13:57.164 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:57.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:57.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:13:57.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:56 smithi090 ceph-mon[41670]: pgmap v197: 387 pgs: 2 peering, 64 unknown, 321 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-14T15:13:57.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:56 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2375375347' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:13:57.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:56 smithi090 ceph-mon[41670]: from='client.48226 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:13:57.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:56 smithi090 ceph-mon[41670]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:13:57.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:56 smithi090 ceph-mon[41670]: from='client.48226 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-14T15:13:57.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:56 smithi090 ceph-mon[41670]: osdmap e169: 8 total, 8 up, 8 in 2024-04-14T15:13:57.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:56 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:57.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:56 smithi090 ceph-mon[30399]: pgmap v197: 387 pgs: 2 peering, 64 unknown, 321 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-14T15:13:57.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:56 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2375375347' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:13:57.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:56 smithi090 ceph-mon[30399]: from='client.48226 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:13:57.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:56 smithi090 ceph-mon[30399]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:13:57.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:56 smithi090 ceph-mon[30399]: from='client.48226 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-14T15:13:57.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:56 smithi090 ceph-mon[30399]: osdmap e169: 8 total, 8 up, 8 in 2024-04-14T15:13:57.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:56 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:57.266 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:13:57.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:57.291 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:57.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:13:57.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:56 smithi092 ceph-mon[33966]: pgmap v197: 387 pgs: 2 peering, 64 unknown, 321 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-14T15:13:57.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:56 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2375375347' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:13:57.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:56 smithi092 ceph-mon[33966]: from='client.48226 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:13:57.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:56 smithi092 ceph-mon[33966]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:13:57.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:56 smithi092 ceph-mon[33966]: from='client.48226 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-14T15:13:57.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:56 smithi092 ceph-mon[33966]: osdmap e169: 8 total, 8 up, 8 in 2024-04-14T15:13:57.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:56 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:57.392 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:13:57.416 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:57.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:57.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:13:57.518 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:13:57.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:57.542 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:57.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:13:57.643 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:13:57.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:57.669 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:57.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:13:57.770 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:13:57.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:57.794 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:57.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:13:57.895 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:13:57.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:57.919 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:57.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:13:58.020 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:13:58.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.018+0000 7f4a35e71700 1 -- 172.21.15.90:0/3232740249 >> v1:172.21.15.90:6789/0 conn(0x7f4a3010a6c0 legacy=0x7f4a3010cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:58.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.018+0000 7f4a35e71700 1 -- 172.21.15.90:0/3232740249 shutdown_connections 2024-04-14T15:13:58.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.018+0000 7f4a35e71700 1 -- 172.21.15.90:0/3232740249 >> 172.21.15.90:0/3232740249 conn(0x7f4a30076a70 msgr2=0x7f4a30076e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:13:58.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.018+0000 7f4a35e71700 1 -- 172.21.15.90:0/3232740249 shutdown_connections 2024-04-14T15:13:58.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.018+0000 7f4a35e71700 1 -- 172.21.15.90:0/3232740249 wait complete. 2024-04-14T15:13:58.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.019+0000 7f4a35e71700 1 librados: starting msgr at 2024-04-14T15:13:58.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.019+0000 7f4a35e71700 1 librados: starting objecter 2024-04-14T15:13:58.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.019+0000 7f4a35e71700 1 Processor -- start 2024-04-14T15:13:58.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.019+0000 7f4a35e71700 1 -- start start 2024-04-14T15:13:58.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.019+0000 7f4a35e71700 1 librados: setting wanted keys 2024-04-14T15:13:58.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.019+0000 7f4a35e71700 1 librados: calling monclient init 2024-04-14T15:13:58.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.019+0000 7f4a35e71700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f4a30107030 con 0x7f4a3010e260 2024-04-14T15:13:58.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.019+0000 7f4a35e71700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f4a30107240 con 0x7f4a301023d0 2024-04-14T15:13:58.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.019+0000 7f4a35e71700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f4a301ac260 con 0x7f4a3010a6c0 2024-04-14T15:13:58.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.019+0000 7f4a2f7fe700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f4a301023d0 0x7f4a30101060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.90:51282/0 (socket says 172.21.15.90:51282) 2024-04-14T15:13:58.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.019+0000 7f4a2ffff700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f4a3010e260 0x7f4a301a8b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:33848/0 (socket says 172.21.15.90:33848) 2024-04-14T15:13:58.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.019+0000 7f4a2f7fe700 1 -- 172.21.15.90:0/3576255131 learned_addr learned my addr 172.21.15.90:0/3576255131 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:13:58.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.019+0000 7f4a2effd700 1 --1- 172.21.15.90:0/3576255131 >> v1:172.21.15.90:6790/0 conn(0x7f4a3010a6c0 0x7f4a30101770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:54046/0 (socket says 172.21.15.90:54046) 2024-04-14T15:13:58.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.020+0000 7f4a2cff9700 10 client.?.objecter ms_handle_connect 0x7f4a3010e260 2024-04-14T15:13:58.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.020+0000 7f4a2cff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:13:58.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.020+0000 7f4a2cff9700 10 client.?.objecter ms_handle_connect 0x7f4a3010a6c0 2024-04-14T15:13:58.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.020+0000 7f4a2cff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:13:58.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.020+0000 7f4a2cff9700 10 client.?.objecter ms_handle_connect 0x7f4a301023d0 2024-04-14T15:13:58.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.020+0000 7f4a2cff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:13:58.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.020+0000 7f4a2cff9700 1 -- 172.21.15.90:0/3576255131 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 594631054 0 0) 0x7f4a30107240 con 0x7f4a301023d0 2024-04-14T15:13:58.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.020+0000 7f4a2cff9700 1 -- 172.21.15.90:0/3576255131 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a180035b0 con 0x7f4a301023d0 2024-04-14T15:13:58.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.020+0000 7f4a2cff9700 1 -- 172.21.15.90:0/3576255131 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3411925172 0 0) 0x7f4a301ac260 con 0x7f4a3010a6c0 2024-04-14T15:13:58.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.020+0000 7f4a2cff9700 1 -- 172.21.15.90:0/3576255131 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a30107240 con 0x7f4a3010a6c0 2024-04-14T15:13:58.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.020+0000 7f4a2cff9700 1 -- 172.21.15.90:0/3576255131 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1856752137 0 0) 0x7f4a30107030 con 0x7f4a3010e260 2024-04-14T15:13:58.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.020+0000 7f4a2cff9700 1 -- 172.21.15.90:0/3576255131 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a301ac260 con 0x7f4a3010e260 2024-04-14T15:13:58.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.020+0000 7f4a2cff9700 1 -- 172.21.15.90:0/3576255131 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3972896188 0 0) 0x7f4a180035b0 con 0x7f4a301023d0 2024-04-14T15:13:58.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.020+0000 7f4a2cff9700 1 -- 172.21.15.90:0/3576255131 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4a30107030 con 0x7f4a301023d0 2024-04-14T15:13:58.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.020+0000 7f4a2cff9700 1 -- 172.21.15.90:0/3576255131 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f4a200018b0 con 0x7f4a301023d0 2024-04-14T15:13:58.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.020+0000 7f4a2cff9700 1 -- 172.21.15.90:0/3576255131 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 558187734 0 0) 0x7f4a30107240 con 0x7f4a3010a6c0 2024-04-14T15:13:58.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.020+0000 7f4a2cff9700 1 -- 172.21.15.90:0/3576255131 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4a180035b0 con 0x7f4a3010a6c0 2024-04-14T15:13:58.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.020+0000 7f4a2cff9700 1 -- 172.21.15.90:0/3576255131 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 663246410 0 0) 0x7f4a301ac260 con 0x7f4a3010e260 2024-04-14T15:13:58.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.020+0000 7f4a2cff9700 1 -- 172.21.15.90:0/3576255131 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4a30107240 con 0x7f4a3010e260 2024-04-14T15:13:58.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.020+0000 7f4a2cff9700 1 -- 172.21.15.90:0/3576255131 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f4a28001bb0 con 0x7f4a3010e260 2024-04-14T15:13:58.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.020+0000 7f4a2cff9700 1 -- 172.21.15.90:0/3576255131 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f4a240015d0 con 0x7f4a3010a6c0 2024-04-14T15:13:58.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.020+0000 7f4a2cff9700 1 -- 172.21.15.90:0/3576255131 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1836160252 0 0) 0x7f4a30107030 con 0x7f4a301023d0 2024-04-14T15:13:58.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.020+0000 7f4a2cff9700 1 -- 172.21.15.90:0/3576255131 >> v1:172.21.15.90:6790/0 conn(0x7f4a3010a6c0 legacy=0x7f4a30101770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:58.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.020+0000 7f4a2cff9700 1 -- 172.21.15.90:0/3576255131 >> v1:172.21.15.90:6789/0 conn(0x7f4a3010e260 legacy=0x7f4a301a8b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:58.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.020+0000 7f4a2cff9700 1 -- 172.21.15.90:0/3576255131 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a301ad440 con 0x7f4a301023d0 2024-04-14T15:13:58.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.020+0000 7f4a35e71700 1 -- 172.21.15.90:0/3576255131 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4a301ac490 con 0x7f4a301023d0 2024-04-14T15:13:58.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.020+0000 7f4a35e71700 10 client.48244.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:13:58.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.020+0000 7f4a35e71700 1 -- 172.21.15.90:0/3576255131 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f4a301ac950 con 0x7f4a301023d0 2024-04-14T15:13:58.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.020+0000 7f4a35e71700 1 librados: init done 2024-04-14T15:13:58.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.020+0000 7f4a2cff9700 1 -- 172.21.15.90:0/3576255131 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f4a20003b20 con 0x7f4a301023d0 2024-04-14T15:13:58.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.020+0000 7f4a2cff9700 1 -- 172.21.15.90:0/3576255131 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f4a20005c20 con 0x7f4a301023d0 2024-04-14T15:13:58.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.021+0000 7f4a35e71700 1 -- 172.21.15.90:0/3576255131 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f4a1c005100 con 0x7f4a301023d0 2024-04-14T15:13:58.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.022+0000 7f4a2cff9700 1 -- 172.21.15.90:0/3576255131 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f4a20003680 con 0x7f4a301023d0 2024-04-14T15:13:58.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.022+0000 7f4a2cff9700 1 -- 172.21.15.90:0/3576255131 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(170..170 src has 51..170) ==== 10697+0+0 (unknown 2871815051 0 0) 0x7f4a2008b2e0 con 0x7f4a301023d0 2024-04-14T15:13:58.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.022+0000 7f4a2cff9700 10 client.48244.objecter ms_dispatch 0x7f4a30001550 osd_map(170..170 src has 51..170) 2024-04-14T15:13:58.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.022+0000 7f4a2cff9700 3 client.48244.objecter handle_osd_map got epochs [170,170] > 0 2024-04-14T15:13:58.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.022+0000 7f4a2cff9700 3 client.48244.objecter handle_osd_map decoding full epoch 170 2024-04-14T15:13:58.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.022+0000 7f4a2cff9700 20 client.48244.objecter dump_active .. 0 homeless 2024-04-14T15:13:58.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.022+0000 7f4a2cff9700 1 -- 172.21.15.90:0/3576255131 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v170) ==== 54+0+16 (unknown 127392416 0 2420000192) 0x7f4a2008d710 con 0x7f4a301023d0 2024-04-14T15:13:58.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.022+0000 7f4a2cff9700 10 client.48244.objecter ms_handle_connect 0x7f4a18071550 2024-04-14T15:13:58.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.023+0000 7f4a35e71700 10 client.48244.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:13:58.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.023+0000 7f4a35e71700 15 client.48244.objecter _session_command_op_assign -1 1 2024-04-14T15:13:58.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.023+0000 7f4a35e71700 20 client.48244.objecter _get_session s=0x7f4a1c0053e0 osd=0 3 2024-04-14T15:13:58.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.023+0000 7f4a35e71700 20 client.48244.objecter put_session s=0x7f4a1c0053e0 osd=0 3 2024-04-14T15:13:58.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.023+0000 7f4a35e71700 20 client.48244.objecter _get_session s=0x7f4a1c0053e0 osd=0 3 2024-04-14T15:13:58.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.023+0000 7f4a35e71700 15 client.48244.objecter _session_command_op_remove -1 1 2024-04-14T15:13:58.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.023+0000 7f4a35e71700 20 client.48244.objecter get_session s=0x7f4a1c0053e0 osd=0 3 2024-04-14T15:13:58.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.023+0000 7f4a35e71700 15 client.48244.objecter _session_command_op_assign 0 1 2024-04-14T15:13:58.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.023+0000 7f4a35e71700 20 client.48244.objecter put_session s=0x7f4a1c0053e0 osd=0 4 2024-04-14T15:13:58.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.023+0000 7f4a35e71700 10 client.48244.objecter _send_command 1 2024-04-14T15:13:58.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.023+0000 7f4a35e71700 1 -- 172.21.15.90:0/3576255131 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4a1c004b50 con 0x7f4a1c004730 2024-04-14T15:13:58.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.023+0000 7f4a2cff9700 10 client.48244.objecter ms_handle_connect 0x7f4a1c004730 2024-04-14T15:13:58.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.024+0000 7f4a2cff9700 1 -- 172.21.15.90:0/3576255131 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f4a1c004b50 con 0x7f4a1c004730 2024-04-14T15:13:58.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.024+0000 7f4a2cff9700 10 client.48244.objecter ms_dispatch 0x7f4a30001550 command_reply(tid 1: 0 ) 2024-04-14T15:13:58.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.024+0000 7f4a2cff9700 10 client.48244.objecter _finish_command 1 = system:0 2024-04-14T15:13:58.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.024+0000 7f4a2cff9700 20 client.48244.objecter put_session s=0x7f4a1c0053e0 osd=0 4 2024-04-14T15:13:58.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.024+0000 7f4a2cff9700 15 client.48244.objecter _session_command_op_remove 0 1 2024-04-14T15:13:58.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.041+0000 7f4a35e71700 10 client.48244.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:13:58.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.041+0000 7f4a35e71700 15 client.48244.objecter _session_command_op_assign -1 2 2024-04-14T15:13:58.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.041+0000 7f4a35e71700 20 client.48244.objecter _get_session s=0x7f4a1c0053e0 osd=0 3 2024-04-14T15:13:58.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.041+0000 7f4a35e71700 20 client.48244.objecter put_session s=0x7f4a1c0053e0 osd=0 3 2024-04-14T15:13:58.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.041+0000 7f4a35e71700 20 client.48244.objecter _get_session s=0x7f4a1c0053e0 osd=0 3 2024-04-14T15:13:58.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.041+0000 7f4a35e71700 15 client.48244.objecter _session_command_op_remove -1 2 2024-04-14T15:13:58.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.041+0000 7f4a35e71700 20 client.48244.objecter get_session s=0x7f4a1c0053e0 osd=0 3 2024-04-14T15:13:58.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.041+0000 7f4a35e71700 15 client.48244.objecter _session_command_op_assign 0 2 2024-04-14T15:13:58.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.041+0000 7f4a35e71700 20 client.48244.objecter put_session s=0x7f4a1c0053e0 osd=0 4 2024-04-14T15:13:58.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.041+0000 7f4a35e71700 10 client.48244.objecter _send_command 2 2024-04-14T15:13:58.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.041+0000 7f4a35e71700 1 -- 172.21.15.90:0/3576255131 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4a1c007b00 con 0x7f4a1c004730 2024-04-14T15:13:58.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.042+0000 7f4a2cff9700 1 -- 172.21.15.90:0/3576255131 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f4a1c007b00 con 0x7f4a1c004730 2024-04-14T15:13:58.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.042+0000 7f4a2cff9700 10 client.48244.objecter ms_dispatch 0x7f4a30001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:13:58.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.042+0000 7f4a2cff9700 10 client.48244.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:13:58.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.042+0000 7f4a2cff9700 20 client.48244.objecter put_session s=0x7f4a1c0053e0 osd=0 4 2024-04-14T15:13:58.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.042+0000 7f4a2cff9700 15 client.48244.objecter _session_command_op_remove 0 2 2024-04-14T15:13:58.050 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:13:58.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.043+0000 7f4a35e71700 10 client.48244.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:13:58.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.043+0000 7f4a35e71700 15 client.48244.objecter _session_command_op_assign -1 3 2024-04-14T15:13:58.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.043+0000 7f4a35e71700 20 client.48244.objecter _get_session s=0x7f4a1c004180 osd=1 3 2024-04-14T15:13:58.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.043+0000 7f4a35e71700 20 client.48244.objecter put_session s=0x7f4a1c004180 osd=1 3 2024-04-14T15:13:58.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.043+0000 7f4a35e71700 20 client.48244.objecter _get_session s=0x7f4a1c004180 osd=1 3 2024-04-14T15:13:58.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.043+0000 7f4a35e71700 15 client.48244.objecter _session_command_op_remove -1 3 2024-04-14T15:13:58.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.043+0000 7f4a35e71700 20 client.48244.objecter get_session s=0x7f4a1c004180 osd=1 3 2024-04-14T15:13:58.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.043+0000 7f4a35e71700 15 client.48244.objecter _session_command_op_assign 1 3 2024-04-14T15:13:58.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.043+0000 7f4a35e71700 20 client.48244.objecter put_session s=0x7f4a1c004180 osd=1 4 2024-04-14T15:13:58.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.043+0000 7f4a35e71700 10 client.48244.objecter _send_command 3 2024-04-14T15:13:58.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.043+0000 7f4a35e71700 1 -- 172.21.15.90:0/3576255131 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f4a1c002200 con 0x7f4a1c0042f0 2024-04-14T15:13:58.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.043+0000 7f4a2cff9700 10 client.48244.objecter ms_handle_connect 0x7f4a1c0042f0 2024-04-14T15:13:58.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.043+0000 7f4a2cff9700 1 -- 172.21.15.90:0/3576255131 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f4a1c002200 con 0x7f4a1c0042f0 2024-04-14T15:13:58.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.043+0000 7f4a2cff9700 10 client.48244.objecter ms_dispatch 0x7f4a30001550 command_reply(tid 3: 0 ) 2024-04-14T15:13:58.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.043+0000 7f4a2cff9700 10 client.48244.objecter _finish_command 3 = system:0 2024-04-14T15:13:58.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.043+0000 7f4a2cff9700 20 client.48244.objecter put_session s=0x7f4a1c004180 osd=1 4 2024-04-14T15:13:58.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.043+0000 7f4a2cff9700 15 client.48244.objecter _session_command_op_remove 1 3 2024-04-14T15:13:58.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:58.053 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:58.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:13:58.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.061+0000 7f4a35e71700 10 client.48244.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:13:58.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.061+0000 7f4a35e71700 15 client.48244.objecter _session_command_op_assign -1 4 2024-04-14T15:13:58.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.061+0000 7f4a35e71700 20 client.48244.objecter _get_session s=0x7f4a1c004180 osd=1 3 2024-04-14T15:13:58.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.061+0000 7f4a35e71700 20 client.48244.objecter put_session s=0x7f4a1c004180 osd=1 3 2024-04-14T15:13:58.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.061+0000 7f4a35e71700 20 client.48244.objecter _get_session s=0x7f4a1c004180 osd=1 3 2024-04-14T15:13:58.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.061+0000 7f4a35e71700 15 client.48244.objecter _session_command_op_remove -1 4 2024-04-14T15:13:58.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.061+0000 7f4a35e71700 20 client.48244.objecter get_session s=0x7f4a1c004180 osd=1 3 2024-04-14T15:13:58.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.061+0000 7f4a35e71700 15 client.48244.objecter _session_command_op_assign 1 4 2024-04-14T15:13:58.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.061+0000 7f4a35e71700 20 client.48244.objecter put_session s=0x7f4a1c004180 osd=1 4 2024-04-14T15:13:58.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.061+0000 7f4a35e71700 10 client.48244.objecter _send_command 4 2024-04-14T15:13:58.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.061+0000 7f4a35e71700 1 -- 172.21.15.90:0/3576255131 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4a1c001600 con 0x7f4a1c0042f0 2024-04-14T15:13:58.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.062+0000 7f4a2cff9700 1 -- 172.21.15.90:0/3576255131 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f4a1c001600 con 0x7f4a1c0042f0 2024-04-14T15:13:58.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.062+0000 7f4a2cff9700 10 client.48244.objecter ms_dispatch 0x7f4a30001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:13:58.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.062+0000 7f4a2cff9700 10 client.48244.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:13:58.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.063+0000 7f4a2cff9700 20 client.48244.objecter put_session s=0x7f4a1c004180 osd=1 4 2024-04-14T15:13:58.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.063+0000 7f4a2cff9700 15 client.48244.objecter _session_command_op_remove 1 4 2024-04-14T15:13:58.064 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:13:58.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.063+0000 7f4a35e71700 10 client.48244.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:13:58.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.063+0000 7f4a35e71700 15 client.48244.objecter _session_command_op_assign -1 5 2024-04-14T15:13:58.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.063+0000 7f4a35e71700 20 client.48244.objecter _get_session s=0x7f4a1c008480 osd=2 3 2024-04-14T15:13:58.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.063+0000 7f4a35e71700 20 client.48244.objecter put_session s=0x7f4a1c008480 osd=2 3 2024-04-14T15:13:58.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.063+0000 7f4a35e71700 20 client.48244.objecter _get_session s=0x7f4a1c008480 osd=2 3 2024-04-14T15:13:58.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.063+0000 7f4a35e71700 15 client.48244.objecter _session_command_op_remove -1 5 2024-04-14T15:13:58.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.063+0000 7f4a35e71700 20 client.48244.objecter get_session s=0x7f4a1c008480 osd=2 3 2024-04-14T15:13:58.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.063+0000 7f4a35e71700 15 client.48244.objecter _session_command_op_assign 2 5 2024-04-14T15:13:58.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.063+0000 7f4a35e71700 20 client.48244.objecter put_session s=0x7f4a1c008480 osd=2 4 2024-04-14T15:13:58.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.063+0000 7f4a35e71700 10 client.48244.objecter _send_command 5 2024-04-14T15:13:58.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.063+0000 7f4a35e71700 1 -- 172.21.15.90:0/3576255131 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f4a1c0111f0 con 0x7f4a1c00d6e0 2024-04-14T15:13:58.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.063+0000 7f4a2cff9700 10 client.48244.objecter ms_handle_connect 0x7f4a1c00d6e0 2024-04-14T15:13:58.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.064+0000 7f4a2cff9700 1 -- 172.21.15.90:0/3576255131 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f4a1c0111f0 con 0x7f4a1c00d6e0 2024-04-14T15:13:58.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.064+0000 7f4a2cff9700 10 client.48244.objecter ms_dispatch 0x7f4a30001550 command_reply(tid 5: 0 ) 2024-04-14T15:13:58.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.064+0000 7f4a2cff9700 10 client.48244.objecter _finish_command 5 = system:0 2024-04-14T15:13:58.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.064+0000 7f4a2cff9700 20 client.48244.objecter put_session s=0x7f4a1c008480 osd=2 4 2024-04-14T15:13:58.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.064+0000 7f4a2cff9700 15 client.48244.objecter _session_command_op_remove 2 5 2024-04-14T15:13:58.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.080+0000 7f4a35e71700 10 client.48244.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:13:58.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.080+0000 7f4a35e71700 15 client.48244.objecter _session_command_op_assign -1 6 2024-04-14T15:13:58.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.080+0000 7f4a35e71700 20 client.48244.objecter _get_session s=0x7f4a1c008480 osd=2 3 2024-04-14T15:13:58.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.080+0000 7f4a35e71700 20 client.48244.objecter put_session s=0x7f4a1c008480 osd=2 3 2024-04-14T15:13:58.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.080+0000 7f4a35e71700 20 client.48244.objecter _get_session s=0x7f4a1c008480 osd=2 3 2024-04-14T15:13:58.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.080+0000 7f4a35e71700 15 client.48244.objecter _session_command_op_remove -1 6 2024-04-14T15:13:58.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.080+0000 7f4a35e71700 20 client.48244.objecter get_session s=0x7f4a1c008480 osd=2 3 2024-04-14T15:13:58.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.080+0000 7f4a35e71700 15 client.48244.objecter _session_command_op_assign 2 6 2024-04-14T15:13:58.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.080+0000 7f4a35e71700 20 client.48244.objecter put_session s=0x7f4a1c008480 osd=2 4 2024-04-14T15:13:58.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.080+0000 7f4a35e71700 10 client.48244.objecter _send_command 6 2024-04-14T15:13:58.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.080+0000 7f4a35e71700 1 -- 172.21.15.90:0/3576255131 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4a1c007f60 con 0x7f4a1c00d6e0 2024-04-14T15:13:58.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.082+0000 7f4a2cff9700 1 -- 172.21.15.90:0/3576255131 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f4a1c007f60 con 0x7f4a1c00d6e0 2024-04-14T15:13:58.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.082+0000 7f4a2cff9700 10 client.48244.objecter ms_dispatch 0x7f4a30001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:13:58.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.082+0000 7f4a2cff9700 10 client.48244.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:13:58.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.082+0000 7f4a2cff9700 20 client.48244.objecter put_session s=0x7f4a1c008480 osd=2 4 2024-04-14T15:13:58.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.082+0000 7f4a2cff9700 15 client.48244.objecter _session_command_op_remove 2 6 2024-04-14T15:13:58.083 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:13:58.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.082+0000 7f4a35e71700 10 client.48244.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:13:58.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.082+0000 7f4a35e71700 15 client.48244.objecter _session_command_op_assign -1 7 2024-04-14T15:13:58.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.082+0000 7f4a35e71700 20 client.48244.objecter _get_session s=0x7f4a1c00cd30 osd=3 3 2024-04-14T15:13:58.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.082+0000 7f4a35e71700 20 client.48244.objecter put_session s=0x7f4a1c00cd30 osd=3 3 2024-04-14T15:13:58.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.082+0000 7f4a35e71700 20 client.48244.objecter _get_session s=0x7f4a1c00cd30 osd=3 3 2024-04-14T15:13:58.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.082+0000 7f4a35e71700 15 client.48244.objecter _session_command_op_remove -1 7 2024-04-14T15:13:58.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.082+0000 7f4a35e71700 20 client.48244.objecter get_session s=0x7f4a1c00cd30 osd=3 3 2024-04-14T15:13:58.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.082+0000 7f4a35e71700 15 client.48244.objecter _session_command_op_assign 3 7 2024-04-14T15:13:58.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.082+0000 7f4a35e71700 20 client.48244.objecter put_session s=0x7f4a1c00cd30 osd=3 4 2024-04-14T15:13:58.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.082+0000 7f4a35e71700 10 client.48244.objecter _send_command 7 2024-04-14T15:13:58.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.082+0000 7f4a35e71700 1 -- 172.21.15.90:0/3576255131 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f4a1c016030 con 0x7f4a1c012520 2024-04-14T15:13:58.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.082+0000 7f4a2cff9700 10 client.48244.objecter ms_handle_connect 0x7f4a1c012520 2024-04-14T15:13:58.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.083+0000 7f4a2cff9700 1 -- 172.21.15.90:0/3576255131 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f4a1c016030 con 0x7f4a1c012520 2024-04-14T15:13:58.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.083+0000 7f4a2cff9700 10 client.48244.objecter ms_dispatch 0x7f4a30001550 command_reply(tid 7: 0 ) 2024-04-14T15:13:58.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.083+0000 7f4a2cff9700 10 client.48244.objecter _finish_command 7 = system:0 2024-04-14T15:13:58.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.083+0000 7f4a2cff9700 20 client.48244.objecter put_session s=0x7f4a1c00cd30 osd=3 4 2024-04-14T15:13:58.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.083+0000 7f4a2cff9700 15 client.48244.objecter _session_command_op_remove 3 7 2024-04-14T15:13:58.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.100+0000 7f4a35e71700 10 client.48244.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:13:58.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.100+0000 7f4a35e71700 15 client.48244.objecter _session_command_op_assign -1 8 2024-04-14T15:13:58.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.100+0000 7f4a35e71700 20 client.48244.objecter _get_session s=0x7f4a1c00cd30 osd=3 3 2024-04-14T15:13:58.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.100+0000 7f4a35e71700 20 client.48244.objecter put_session s=0x7f4a1c00cd30 osd=3 3 2024-04-14T15:13:58.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.100+0000 7f4a35e71700 20 client.48244.objecter _get_session s=0x7f4a1c00cd30 osd=3 3 2024-04-14T15:13:58.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.100+0000 7f4a35e71700 15 client.48244.objecter _session_command_op_remove -1 8 2024-04-14T15:13:58.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.100+0000 7f4a35e71700 20 client.48244.objecter get_session s=0x7f4a1c00cd30 osd=3 3 2024-04-14T15:13:58.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.100+0000 7f4a35e71700 15 client.48244.objecter _session_command_op_assign 3 8 2024-04-14T15:13:58.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.100+0000 7f4a35e71700 20 client.48244.objecter put_session s=0x7f4a1c00cd30 osd=3 4 2024-04-14T15:13:58.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.100+0000 7f4a35e71700 10 client.48244.objecter _send_command 8 2024-04-14T15:13:58.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.100+0000 7f4a35e71700 1 -- 172.21.15.90:0/3576255131 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4a1c00c7b0 con 0x7f4a1c012520 2024-04-14T15:13:58.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.102+0000 7f4a2cff9700 1 -- 172.21.15.90:0/3576255131 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f4a1c00c7b0 con 0x7f4a1c012520 2024-04-14T15:13:58.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.102+0000 7f4a2cff9700 10 client.48244.objecter ms_dispatch 0x7f4a30001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:13:58.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.102+0000 7f4a2cff9700 10 client.48244.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:13:58.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.102+0000 7f4a2cff9700 20 client.48244.objecter put_session s=0x7f4a1c00cd30 osd=3 4 2024-04-14T15:13:58.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.102+0000 7f4a2cff9700 15 client.48244.objecter _session_command_op_remove 3 8 2024-04-14T15:13:58.102 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:13:58.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.102+0000 7f4a35e71700 10 client.48244.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:13:58.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.102+0000 7f4a35e71700 15 client.48244.objecter _session_command_op_assign -1 9 2024-04-14T15:13:58.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.102+0000 7f4a35e71700 20 client.48244.objecter _get_session s=0x7f4a1c00c560 osd=4 3 2024-04-14T15:13:58.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.102+0000 7f4a35e71700 20 client.48244.objecter put_session s=0x7f4a1c00c560 osd=4 3 2024-04-14T15:13:58.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.102+0000 7f4a35e71700 20 client.48244.objecter _get_session s=0x7f4a1c00c560 osd=4 3 2024-04-14T15:13:58.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.102+0000 7f4a35e71700 15 client.48244.objecter _session_command_op_remove -1 9 2024-04-14T15:13:58.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.102+0000 7f4a35e71700 20 client.48244.objecter get_session s=0x7f4a1c00c560 osd=4 3 2024-04-14T15:13:58.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.102+0000 7f4a35e71700 15 client.48244.objecter _session_command_op_assign 4 9 2024-04-14T15:13:58.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.102+0000 7f4a35e71700 20 client.48244.objecter put_session s=0x7f4a1c00c560 osd=4 4 2024-04-14T15:13:58.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.102+0000 7f4a35e71700 10 client.48244.objecter _send_command 9 2024-04-14T15:13:58.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.102+0000 7f4a35e71700 1 -- 172.21.15.90:0/3576255131 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f4a1c00c700 con 0x7f4a1c012080 2024-04-14T15:13:58.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.102+0000 7f4a2cff9700 10 client.48244.objecter ms_handle_connect 0x7f4a1c012080 2024-04-14T15:13:58.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.103+0000 7f4a2cff9700 1 -- 172.21.15.90:0/3576255131 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f4a1c00c700 con 0x7f4a1c012080 2024-04-14T15:13:58.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.103+0000 7f4a2cff9700 10 client.48244.objecter ms_dispatch 0x7f4a30001550 command_reply(tid 9: 0 ) 2024-04-14T15:13:58.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.103+0000 7f4a2cff9700 10 client.48244.objecter _finish_command 9 = system:0 2024-04-14T15:13:58.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.103+0000 7f4a2cff9700 20 client.48244.objecter put_session s=0x7f4a1c00c560 osd=4 4 2024-04-14T15:13:58.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.103+0000 7f4a2cff9700 15 client.48244.objecter _session_command_op_remove 4 9 2024-04-14T15:13:58.118 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:57 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-04-14T15:13:58.118 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:57 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3868448150' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi090-93724-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi090-93724-16"}]': finished 2024-04-14T15:13:58.118 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:57 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "29"}]': finished 2024-04-14T15:13:58.118 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:57 smithi092 ceph-mon[33966]: osdmap e170: 8 total, 8 up, 8 in 2024-04-14T15:13:58.118 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:57 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/4250679047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi090-93363-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:58.119 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:57 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/901684184' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi090-93342-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:58.119 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:57 smithi092 ceph-mon[33966]: from='client.48238 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi090-93342-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:58.119 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:57 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:58.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.120+0000 7f4a35e71700 10 client.48244.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:13:58.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.120+0000 7f4a35e71700 15 client.48244.objecter _session_command_op_assign -1 10 2024-04-14T15:13:58.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.120+0000 7f4a35e71700 20 client.48244.objecter _get_session s=0x7f4a1c00c560 osd=4 3 2024-04-14T15:13:58.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.120+0000 7f4a35e71700 20 client.48244.objecter put_session s=0x7f4a1c00c560 osd=4 3 2024-04-14T15:13:58.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.120+0000 7f4a35e71700 20 client.48244.objecter _get_session s=0x7f4a1c00c560 osd=4 3 2024-04-14T15:13:58.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.120+0000 7f4a35e71700 15 client.48244.objecter _session_command_op_remove -1 10 2024-04-14T15:13:58.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.120+0000 7f4a35e71700 20 client.48244.objecter get_session s=0x7f4a1c00c560 osd=4 3 2024-04-14T15:13:58.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.120+0000 7f4a35e71700 15 client.48244.objecter _session_command_op_assign 4 10 2024-04-14T15:13:58.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.120+0000 7f4a35e71700 20 client.48244.objecter put_session s=0x7f4a1c00c560 osd=4 4 2024-04-14T15:13:58.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.120+0000 7f4a35e71700 10 client.48244.objecter _send_command 10 2024-04-14T15:13:58.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.120+0000 7f4a35e71700 1 -- 172.21.15.90:0/3576255131 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4a1c01aa90 con 0x7f4a1c012080 2024-04-14T15:13:58.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.121+0000 7f4a2cff9700 1 -- 172.21.15.90:0/3576255131 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f4a1c01aa90 con 0x7f4a1c012080 2024-04-14T15:13:58.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.121+0000 7f4a2cff9700 10 client.48244.objecter ms_dispatch 0x7f4a30001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:13:58.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.121+0000 7f4a2cff9700 10 client.48244.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:13:58.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.121+0000 7f4a2cff9700 20 client.48244.objecter put_session s=0x7f4a1c00c560 osd=4 4 2024-04-14T15:13:58.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.121+0000 7f4a2cff9700 15 client.48244.objecter _session_command_op_remove 4 10 2024-04-14T15:13:58.123 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:13:58.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.121+0000 7f4a35e71700 10 client.48244.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:13:58.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.121+0000 7f4a35e71700 15 client.48244.objecter _session_command_op_assign -1 11 2024-04-14T15:13:58.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.121+0000 7f4a35e71700 20 client.48244.objecter _get_session s=0x7f4a1c00c760 osd=5 3 2024-04-14T15:13:58.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.121+0000 7f4a35e71700 20 client.48244.objecter put_session s=0x7f4a1c00c760 osd=5 3 2024-04-14T15:13:58.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.121+0000 7f4a35e71700 20 client.48244.objecter _get_session s=0x7f4a1c00c760 osd=5 3 2024-04-14T15:13:58.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.121+0000 7f4a35e71700 15 client.48244.objecter _session_command_op_remove -1 11 2024-04-14T15:13:58.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.121+0000 7f4a35e71700 20 client.48244.objecter get_session s=0x7f4a1c00c760 osd=5 3 2024-04-14T15:13:58.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.121+0000 7f4a35e71700 15 client.48244.objecter _session_command_op_assign 5 11 2024-04-14T15:13:58.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.121+0000 7f4a35e71700 20 client.48244.objecter put_session s=0x7f4a1c00c760 osd=5 4 2024-04-14T15:13:58.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.121+0000 7f4a35e71700 10 client.48244.objecter _send_command 11 2024-04-14T15:13:58.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.121+0000 7f4a35e71700 1 -- 172.21.15.90:0/3576255131 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f4a1c01cd50 con 0x7f4a1c01b1b0 2024-04-14T15:13:58.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.122+0000 7f4a2cff9700 10 client.48244.objecter ms_handle_connect 0x7f4a1c01b1b0 2024-04-14T15:13:58.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.123+0000 7f4a2cff9700 1 -- 172.21.15.90:0/3576255131 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f4a1c01cd50 con 0x7f4a1c01b1b0 2024-04-14T15:13:58.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.123+0000 7f4a2cff9700 10 client.48244.objecter ms_dispatch 0x7f4a30001550 command_reply(tid 11: 0 ) 2024-04-14T15:13:58.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.123+0000 7f4a2cff9700 10 client.48244.objecter _finish_command 11 = system:0 2024-04-14T15:13:58.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.123+0000 7f4a2cff9700 20 client.48244.objecter put_session s=0x7f4a1c00c760 osd=5 4 2024-04-14T15:13:58.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.123+0000 7f4a2cff9700 15 client.48244.objecter _session_command_op_remove 5 11 2024-04-14T15:13:58.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.139+0000 7f4a35e71700 10 client.48244.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:13:58.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.139+0000 7f4a35e71700 15 client.48244.objecter _session_command_op_assign -1 12 2024-04-14T15:13:58.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.139+0000 7f4a35e71700 20 client.48244.objecter _get_session s=0x7f4a1c00c760 osd=5 3 2024-04-14T15:13:58.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.139+0000 7f4a35e71700 20 client.48244.objecter put_session s=0x7f4a1c00c760 osd=5 3 2024-04-14T15:13:58.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.139+0000 7f4a35e71700 20 client.48244.objecter _get_session s=0x7f4a1c00c760 osd=5 3 2024-04-14T15:13:58.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.139+0000 7f4a35e71700 15 client.48244.objecter _session_command_op_remove -1 12 2024-04-14T15:13:58.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.139+0000 7f4a35e71700 20 client.48244.objecter get_session s=0x7f4a1c00c760 osd=5 3 2024-04-14T15:13:58.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.139+0000 7f4a35e71700 15 client.48244.objecter _session_command_op_assign 5 12 2024-04-14T15:13:58.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.139+0000 7f4a35e71700 20 client.48244.objecter put_session s=0x7f4a1c00c760 osd=5 4 2024-04-14T15:13:58.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.139+0000 7f4a35e71700 10 client.48244.objecter _send_command 12 2024-04-14T15:13:58.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.139+0000 7f4a35e71700 1 -- 172.21.15.90:0/3576255131 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4a1c01cf80 con 0x7f4a1c01b1b0 2024-04-14T15:13:58.140 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:57 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-04-14T15:13:58.141 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:57 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3868448150' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi090-93724-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi090-93724-16"}]': finished 2024-04-14T15:13:58.141 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:57 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "29"}]': finished 2024-04-14T15:13:58.141 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:57 smithi090 ceph-mon[41670]: osdmap e170: 8 total, 8 up, 8 in 2024-04-14T15:13:58.141 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:57 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/4250679047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi090-93363-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:58.141 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:57 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/901684184' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi090-93342-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:58.141 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:57 smithi090 ceph-mon[41670]: from='client.48238 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi090-93342-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:58.141 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:57 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:58.141 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:57 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-04-14T15:13:58.141 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:57 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3868448150' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi090-93724-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi090-93724-16"}]': finished 2024-04-14T15:13:58.142 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:57 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "29"}]': finished 2024-04-14T15:13:58.142 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:57 smithi090 ceph-mon[30399]: osdmap e170: 8 total, 8 up, 8 in 2024-04-14T15:13:58.142 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:57 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/4250679047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi090-93363-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:58.142 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:57 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/901684184' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi090-93342-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:58.142 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:57 smithi090 ceph-mon[30399]: from='client.48238 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi090-93342-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:13:58.142 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:57 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:58.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.141+0000 7f4a2cff9700 1 -- 172.21.15.90:0/3576255131 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f4a1c01cf80 con 0x7f4a1c01b1b0 2024-04-14T15:13:58.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.141+0000 7f4a2cff9700 10 client.48244.objecter ms_dispatch 0x7f4a30001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:13:58.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.141+0000 7f4a2cff9700 10 client.48244.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:13:58.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.141+0000 7f4a2cff9700 20 client.48244.objecter put_session s=0x7f4a1c00c760 osd=5 4 2024-04-14T15:13:58.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.141+0000 7f4a2cff9700 15 client.48244.objecter _session_command_op_remove 5 12 2024-04-14T15:13:58.143 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:13:58.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.141+0000 7f4a35e71700 10 client.48244.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:13:58.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.141+0000 7f4a35e71700 15 client.48244.objecter _session_command_op_assign -1 13 2024-04-14T15:13:58.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.141+0000 7f4a35e71700 20 client.48244.objecter _get_session s=0x7f4a1c01cd50 osd=6 3 2024-04-14T15:13:58.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.141+0000 7f4a35e71700 20 client.48244.objecter put_session s=0x7f4a1c01cd50 osd=6 3 2024-04-14T15:13:58.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.141+0000 7f4a35e71700 20 client.48244.objecter _get_session s=0x7f4a1c01cd50 osd=6 3 2024-04-14T15:13:58.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.141+0000 7f4a35e71700 15 client.48244.objecter _session_command_op_remove -1 13 2024-04-14T15:13:58.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.141+0000 7f4a35e71700 20 client.48244.objecter get_session s=0x7f4a1c01cd50 osd=6 3 2024-04-14T15:13:58.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.141+0000 7f4a35e71700 15 client.48244.objecter _session_command_op_assign 6 13 2024-04-14T15:13:58.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.141+0000 7f4a35e71700 20 client.48244.objecter put_session s=0x7f4a1c01cd50 osd=6 4 2024-04-14T15:13:58.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.141+0000 7f4a35e71700 10 client.48244.objecter _send_command 13 2024-04-14T15:13:58.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.141+0000 7f4a35e71700 1 -- 172.21.15.90:0/3576255131 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f4a1c0231e0 con 0x7f4a1c01cf80 2024-04-14T15:13:58.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.141+0000 7f4a2cff9700 10 client.48244.objecter ms_handle_connect 0x7f4a1c01cf80 2024-04-14T15:13:58.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.142+0000 7f4a2cff9700 1 -- 172.21.15.90:0/3576255131 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f4a1c0231e0 con 0x7f4a1c01cf80 2024-04-14T15:13:58.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.142+0000 7f4a2cff9700 10 client.48244.objecter ms_dispatch 0x7f4a30001550 command_reply(tid 13: 0 ) 2024-04-14T15:13:58.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.142+0000 7f4a2cff9700 10 client.48244.objecter _finish_command 13 = system:0 2024-04-14T15:13:58.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.142+0000 7f4a2cff9700 20 client.48244.objecter put_session s=0x7f4a1c01cd50 osd=6 4 2024-04-14T15:13:58.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.142+0000 7f4a2cff9700 15 client.48244.objecter _session_command_op_remove 6 13 2024-04-14T15:13:58.154 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:13:58.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.158+0000 7f4a35e71700 10 client.48244.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:13:58.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.158+0000 7f4a35e71700 15 client.48244.objecter _session_command_op_assign -1 14 2024-04-14T15:13:58.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.158+0000 7f4a35e71700 20 client.48244.objecter _get_session s=0x7f4a1c01cd50 osd=6 3 2024-04-14T15:13:58.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.158+0000 7f4a35e71700 20 client.48244.objecter put_session s=0x7f4a1c01cd50 osd=6 3 2024-04-14T15:13:58.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.158+0000 7f4a35e71700 20 client.48244.objecter _get_session s=0x7f4a1c01cd50 osd=6 3 2024-04-14T15:13:58.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.158+0000 7f4a35e71700 15 client.48244.objecter _session_command_op_remove -1 14 2024-04-14T15:13:58.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.158+0000 7f4a35e71700 20 client.48244.objecter get_session s=0x7f4a1c01cd50 osd=6 3 2024-04-14T15:13:58.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.158+0000 7f4a35e71700 15 client.48244.objecter _session_command_op_assign 6 14 2024-04-14T15:13:58.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.158+0000 7f4a35e71700 20 client.48244.objecter put_session s=0x7f4a1c01cd50 osd=6 4 2024-04-14T15:13:58.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.158+0000 7f4a35e71700 10 client.48244.objecter _send_command 14 2024-04-14T15:13:58.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.158+0000 7f4a35e71700 1 -- 172.21.15.90:0/3576255131 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4a1c024b90 con 0x7f4a1c01cf80 2024-04-14T15:13:58.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.160+0000 7f4a2cff9700 1 -- 172.21.15.90:0/3576255131 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f4a1c024b90 con 0x7f4a1c01cf80 2024-04-14T15:13:58.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.160+0000 7f4a2cff9700 10 client.48244.objecter ms_dispatch 0x7f4a30001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:13:58.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.160+0000 7f4a2cff9700 10 client.48244.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:13:58.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.160+0000 7f4a2cff9700 20 client.48244.objecter put_session s=0x7f4a1c01cd50 osd=6 4 2024-04-14T15:13:58.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.160+0000 7f4a2cff9700 15 client.48244.objecter _session_command_op_remove 6 14 2024-04-14T15:13:58.161 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:13:58.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.160+0000 7f4a35e71700 10 client.48244.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:13:58.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.160+0000 7f4a35e71700 15 client.48244.objecter _session_command_op_assign -1 15 2024-04-14T15:13:58.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.160+0000 7f4a35e71700 20 client.48244.objecter _get_session s=0x7f4a1c024b90 osd=7 3 2024-04-14T15:13:58.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.160+0000 7f4a35e71700 20 client.48244.objecter put_session s=0x7f4a1c024b90 osd=7 3 2024-04-14T15:13:58.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.160+0000 7f4a35e71700 20 client.48244.objecter _get_session s=0x7f4a1c024b90 osd=7 3 2024-04-14T15:13:58.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.160+0000 7f4a35e71700 15 client.48244.objecter _session_command_op_remove -1 15 2024-04-14T15:13:58.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.160+0000 7f4a35e71700 20 client.48244.objecter get_session s=0x7f4a1c024b90 osd=7 3 2024-04-14T15:13:58.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.160+0000 7f4a35e71700 15 client.48244.objecter _session_command_op_assign 7 15 2024-04-14T15:13:58.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.160+0000 7f4a35e71700 20 client.48244.objecter put_session s=0x7f4a1c024b90 osd=7 4 2024-04-14T15:13:58.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.160+0000 7f4a35e71700 10 client.48244.objecter _send_command 15 2024-04-14T15:13:58.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.160+0000 7f4a35e71700 1 -- 172.21.15.90:0/3576255131 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f4a1c025710 con 0x7f4a1c0252b0 2024-04-14T15:13:58.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.161+0000 7f4a2cff9700 10 client.48244.objecter ms_handle_connect 0x7f4a1c0252b0 2024-04-14T15:13:58.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.162+0000 7f4a2cff9700 1 -- 172.21.15.90:0/3576255131 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f4a1c025710 con 0x7f4a1c0252b0 2024-04-14T15:13:58.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.162+0000 7f4a2cff9700 10 client.48244.objecter ms_dispatch 0x7f4a30001550 command_reply(tid 15: 0 ) 2024-04-14T15:13:58.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.162+0000 7f4a2cff9700 10 client.48244.objecter _finish_command 15 = system:0 2024-04-14T15:13:58.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.162+0000 7f4a2cff9700 20 client.48244.objecter put_session s=0x7f4a1c024b90 osd=7 4 2024-04-14T15:13:58.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.162+0000 7f4a2cff9700 15 client.48244.objecter _session_command_op_remove 7 15 2024-04-14T15:13:58.178 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:58.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.178+0000 7f4a35e71700 10 client.48244.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:13:58.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.178+0000 7f4a35e71700 15 client.48244.objecter _session_command_op_assign -1 16 2024-04-14T15:13:58.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.178+0000 7f4a35e71700 20 client.48244.objecter _get_session s=0x7f4a1c024b90 osd=7 3 2024-04-14T15:13:58.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.178+0000 7f4a35e71700 20 client.48244.objecter put_session s=0x7f4a1c024b90 osd=7 3 2024-04-14T15:13:58.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.178+0000 7f4a35e71700 20 client.48244.objecter _get_session s=0x7f4a1c024b90 osd=7 3 2024-04-14T15:13:58.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.178+0000 7f4a35e71700 15 client.48244.objecter _session_command_op_remove -1 16 2024-04-14T15:13:58.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.178+0000 7f4a35e71700 20 client.48244.objecter get_session s=0x7f4a1c024b90 osd=7 3 2024-04-14T15:13:58.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.178+0000 7f4a35e71700 15 client.48244.objecter _session_command_op_assign 7 16 2024-04-14T15:13:58.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.178+0000 7f4a35e71700 20 client.48244.objecter put_session s=0x7f4a1c024b90 osd=7 4 2024-04-14T15:13:58.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.178+0000 7f4a35e71700 10 client.48244.objecter _send_command 16 2024-04-14T15:13:58.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.178+0000 7f4a35e71700 1 -- 172.21.15.90:0/3576255131 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4a1c025710 con 0x7f4a1c0252b0 2024-04-14T15:13:58.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:58.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:13:58.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.180+0000 7f4a2cff9700 1 -- 172.21.15.90:0/3576255131 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f4a1c025710 con 0x7f4a1c0252b0 2024-04-14T15:13:58.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.180+0000 7f4a2cff9700 10 client.48244.objecter ms_dispatch 0x7f4a30001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:13:58.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.180+0000 7f4a2cff9700 10 client.48244.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:13:58.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.180+0000 7f4a2cff9700 20 client.48244.objecter put_session s=0x7f4a1c024b90 osd=7 4 2024-04-14T15:13:58.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.180+0000 7f4a2cff9700 15 client.48244.objecter _session_command_op_remove 7 16 2024-04-14T15:13:58.181 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:13:58.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.180+0000 7f4a35e71700 10 librados: watch_flush enter 2024-04-14T15:13:58.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.180+0000 7f4a35e71700 10 librados: watch_flush exit 2024-04-14T15:13:58.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.180+0000 7f4a35e71700 10 client.48244.objecter close_session for osd.0 2024-04-14T15:13:58.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.180+0000 7f4a35e71700 1 -- 172.21.15.90:0/3576255131 >> v1:172.21.15.90:6801/3761806885 conn(0x7f4a1c004730 legacy=0x7f4a1c002790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:58.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.180+0000 7f4a35e71700 20 client.48244.objecter put_session s=0x7f4a1c0053e0 osd=0 1 2024-04-14T15:13:58.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.180+0000 7f4a35e71700 10 client.48244.objecter close_session for osd.1 2024-04-14T15:13:58.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.180+0000 7f4a35e71700 1 -- 172.21.15.90:0/3576255131 >> v1:172.21.15.90:6805/4170361699 conn(0x7f4a1c0042f0 legacy=0x7f4a1c00ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:58.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.180+0000 7f4a35e71700 20 client.48244.objecter put_session s=0x7f4a1c004180 osd=1 1 2024-04-14T15:13:58.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.180+0000 7f4a35e71700 10 client.48244.objecter close_session for osd.2 2024-04-14T15:13:58.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.180+0000 7f4a35e71700 1 -- 172.21.15.90:0/3576255131 >> v1:172.21.15.90:6809/1686184251 conn(0x7f4a1c00d6e0 legacy=0x7f4a1c00fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:58.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.180+0000 7f4a35e71700 20 client.48244.objecter put_session s=0x7f4a1c008480 osd=2 1 2024-04-14T15:13:58.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.180+0000 7f4a35e71700 10 client.48244.objecter close_session for osd.3 2024-04-14T15:13:58.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.180+0000 7f4a35e71700 1 -- 172.21.15.90:0/3576255131 >> v1:172.21.15.90:6813/2270166774 conn(0x7f4a1c012520 legacy=0x7f4a1c014910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:58.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.180+0000 7f4a35e71700 20 client.48244.objecter put_session s=0x7f4a1c00cd30 osd=3 1 2024-04-14T15:13:58.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.180+0000 7f4a35e71700 10 client.48244.objecter close_session for osd.4 2024-04-14T15:13:58.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.180+0000 7f4a35e71700 1 -- 172.21.15.90:0/3576255131 >> v1:172.21.15.92:6800/3887549733 conn(0x7f4a1c012080 legacy=0x7f4a1c019370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:58.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.180+0000 7f4a2ffff700 1 -- 172.21.15.90:0/3576255131 reap_dead start 2024-04-14T15:13:58.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.180+0000 7f4a35e71700 20 client.48244.objecter put_session s=0x7f4a1c00c560 osd=4 1 2024-04-14T15:13:58.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.180+0000 7f4a35e71700 10 client.48244.objecter close_session for osd.5 2024-04-14T15:13:58.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.180+0000 7f4a35e71700 1 -- 172.21.15.90:0/3576255131 >> v1:172.21.15.92:6804/3244561440 conn(0x7f4a1c01b1b0 legacy=0x7f4a1c01b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:58.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.180+0000 7f4a2ffff700 1 -- 172.21.15.90:0/3576255131 reap_dead start 2024-04-14T15:13:58.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.180+0000 7f4a2ffff700 1 -- 172.21.15.90:0/3576255131 reap_dead start 2024-04-14T15:13:58.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.180+0000 7f4a35e71700 20 client.48244.objecter put_session s=0x7f4a1c00c760 osd=5 1 2024-04-14T15:13:58.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.180+0000 7f4a35e71700 10 client.48244.objecter close_session for osd.6 2024-04-14T15:13:58.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.180+0000 7f4a35e71700 1 -- 172.21.15.90:0/3576255131 >> v1:172.21.15.92:6808/636125963 conn(0x7f4a1c01cf80 legacy=0x7f4a1c021930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:58.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.180+0000 7f4a35e71700 20 client.48244.objecter put_session s=0x7f4a1c01cd50 osd=6 1 2024-04-14T15:13:58.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.180+0000 7f4a35e71700 10 client.48244.objecter close_session for osd.7 2024-04-14T15:13:58.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.180+0000 7f4a35e71700 1 -- 172.21.15.90:0/3576255131 >> v1:172.21.15.92:6812/1773706716 conn(0x7f4a1c0252b0 legacy=0x7f4a1c0233c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:58.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.180+0000 7f4a35e71700 20 client.48244.objecter put_session s=0x7f4a1c024b90 osd=7 1 2024-04-14T15:13:58.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.180+0000 7f4a35e71700 20 client.48244.objecter shutdown clearing up homeless session... 2024-04-14T15:13:58.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.180+0000 7f4a35e71700 10 client.48244.objecter successfully canceled tick 2024-04-14T15:13:58.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.180+0000 7f4a35e71700 1 -- 172.21.15.90:0/3576255131 >> v1:172.21.15.90:6800/420935427 conn(0x7f4a18071550 legacy=0x7f4a18073a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:58.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.180+0000 7f4a35e71700 1 -- 172.21.15.90:0/3576255131 >> v1:172.21.15.92:6789/0 conn(0x7f4a301023d0 legacy=0x7f4a30101060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:13:58.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.180+0000 7f4a2ffff700 1 -- 172.21.15.90:0/3576255131 reap_dead start 2024-04-14T15:13:58.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.180+0000 7f4a35e71700 1 -- 172.21.15.90:0/3576255131 shutdown_connections 2024-04-14T15:13:58.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.180+0000 7f4a35e71700 1 -- 172.21.15.90:0/3576255131 >> 172.21.15.90:0/3576255131 conn(0x7f4a30076a70 msgr2=0x7f4a30110700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:13:58.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.180+0000 7f4a35e71700 1 -- 172.21.15.90:0/3576255131 shutdown_connections 2024-04-14T15:13:58.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.181+0000 7f4a35e71700 1 -- 172.21.15.90:0/3576255131 wait complete. 2024-04-14T15:13:58.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:13:58.181+0000 7f4a35e71700 1 librados: shutdown 2024-04-14T15:13:58.185 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:13:58.185 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:13:58.185 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:13:58.185 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:13:58.185 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:13:58.185 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:13:58.186 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:13:58.186 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:13:58.281 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:13:58.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:58.306 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:58.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:13:58.358 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-04-14T15:13:58.358 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:13:58.407 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:13:58.430 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:58.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:58.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:13:58.532 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:13:58.556 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:58.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:58.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:13:58.658 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:13:58.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:58.683 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:58.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:13:58.785 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:13:58.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:58.809 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:58.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:13:58.911 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:13:58.935 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:58.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:58.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:13:59.036 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:13:59.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:59.062 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:59.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:13:59.079 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:58 smithi092 ceph-mon[33966]: pgmap v200: 322 pgs: 1 peering, 321 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s; 31 B/s, 0 objects/s recovering 2024-04-14T15:13:59.079 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:58 smithi092 ceph-mon[33966]: 1.0 scrub starts 2024-04-14T15:13:59.079 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:58 smithi092 ceph-mon[33966]: 1.0 scrub ok 2024-04-14T15:13:59.079 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:58 smithi092 ceph-mon[33966]: 6.7 scrub starts 2024-04-14T15:13:59.079 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:58 smithi092 ceph-mon[33966]: 6.7 scrub ok 2024-04-14T15:13:59.079 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:58 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3576255131' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:13:59.079 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:58 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/4250679047' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi090-93363-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:59.079 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:58 smithi092 ceph-mon[33966]: from='client.48238 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi090-93342-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:59.080 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:58 smithi092 ceph-mon[33966]: osdmap e171: 8 total, 8 up, 8 in 2024-04-14T15:13:59.080 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:58 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:59.105 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-14T15:13:59.136 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:58 smithi090 ceph-mon[41670]: pgmap v200: 322 pgs: 1 peering, 321 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s; 31 B/s, 0 objects/s recovering 2024-04-14T15:13:59.136 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:58 smithi090 ceph-mon[41670]: 1.0 scrub starts 2024-04-14T15:13:59.136 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:58 smithi090 ceph-mon[41670]: 1.0 scrub ok 2024-04-14T15:13:59.136 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:58 smithi090 ceph-mon[41670]: 6.7 scrub starts 2024-04-14T15:13:59.136 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:58 smithi090 ceph-mon[41670]: 6.7 scrub ok 2024-04-14T15:13:59.136 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:58 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3576255131' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:13:59.136 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:58 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/4250679047' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi090-93363-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:59.136 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:58 smithi090 ceph-mon[41670]: from='client.48238 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi090-93342-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:59.137 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:58 smithi090 ceph-mon[41670]: osdmap e171: 8 total, 8 up, 8 in 2024-04-14T15:13:59.137 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:58 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:59.137 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:58 smithi090 ceph-mon[30399]: pgmap v200: 322 pgs: 1 peering, 321 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s; 31 B/s, 0 objects/s recovering 2024-04-14T15:13:59.137 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:58 smithi090 ceph-mon[30399]: 1.0 scrub starts 2024-04-14T15:13:59.137 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:58 smithi090 ceph-mon[30399]: 1.0 scrub ok 2024-04-14T15:13:59.137 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:58 smithi090 ceph-mon[30399]: 6.7 scrub starts 2024-04-14T15:13:59.137 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:58 smithi090 ceph-mon[30399]: 6.7 scrub ok 2024-04-14T15:13:59.137 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:58 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3576255131' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:13:59.137 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:58 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/4250679047' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi090-93363-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:59.138 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:58 smithi090 ceph-mon[30399]: from='client.48238 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi090-93342-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:13:59.138 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:58 smithi090 ceph-mon[30399]: osdmap e171: 8 total, 8 up, 8 in 2024-04-14T15:13:59.138 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:58 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:13:59.163 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:13:59.213 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:59.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:59.214 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:13:59.315 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:13:59.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:59.344 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:59.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:13:59.384 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:13:59.445 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:13:59.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:59.474 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:59.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:13:59.576 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:13:59.600 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:59.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:59.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:13:59.702 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:13:59.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:59.727 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:59.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:13:59.828 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:13:59.859 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:13:59.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:59.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:13:59.961 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:13:59.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:13:59.986 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:13:59.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:14:00.087 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:14:00.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:00.270 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:00.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:14:00.271 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:59 smithi090 ceph-mon[41670]: 2.1e scrub starts 2024-04-14T15:14:00.271 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:59 smithi090 ceph-mon[41670]: 2.1e scrub ok 2024-04-14T15:14:00.271 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:59 smithi090 ceph-mon[41670]: osdmap e172: 8 total, 8 up, 8 in 2024-04-14T15:14:00.271 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:13:59 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:00.271 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:59 smithi090 ceph-mon[30399]: 2.1e scrub starts 2024-04-14T15:14:00.271 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:59 smithi090 ceph-mon[30399]: 2.1e scrub ok 2024-04-14T15:14:00.272 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:59 smithi090 ceph-mon[30399]: osdmap e172: 8 total, 8 up, 8 in 2024-04-14T15:14:00.272 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:13:59 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:00.272 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:59 smithi092 ceph-mon[33966]: 2.1e scrub starts 2024-04-14T15:14:00.272 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:59 smithi092 ceph-mon[33966]: 2.1e scrub ok 2024-04-14T15:14:00.272 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:59 smithi092 ceph-mon[33966]: osdmap e172: 8 total, 8 up, 8 in 2024-04-14T15:14:00.272 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:13:59 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:00.372 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:14:00.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:00.397 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:00.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:14:00.498 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:14:00.518 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-14T15:14:00.518 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-14T15:14:00.518 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-04-14T15:14:00.519 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph pg dump --format=json 2024-04-14T15:14:00.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:00.523 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:00.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:14:00.625 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:14:00.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:00.649 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:00.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:14:00.750 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:00.764 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:14:00.770 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-14T15:14:00.774 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:00.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:00.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:00.876 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:14:00.900 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:00.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:00.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:14:01.002 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:14:01.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:01.026 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:01.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:14:01.059 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:00 smithi090 ceph-mon[41670]: pgmap v203: 394 pgs: 33 unknown, 2 clean+premerge+peered, 359 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 511 B/s wr, 0 op/s 2024-04-14T15:14:01.059 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:00 smithi090 ceph-mon[41670]: 11.14 scrub starts 2024-04-14T15:14:01.059 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:00 smithi090 ceph-mon[41670]: 11.14 scrub ok 2024-04-14T15:14:01.059 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:00 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-14T15:14:01.059 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:00 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi090-93601-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:01.059 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:00 smithi090 ceph-mon[41670]: from='client.24853 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi090-93601-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:01.059 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:00 smithi090 ceph-mon[41670]: 6.16 scrub starts 2024-04-14T15:14:01.059 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:00 smithi090 ceph-mon[41670]: 6.16 scrub ok 2024-04-14T15:14:01.059 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:00 smithi090 ceph-mon[41670]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:14:01.059 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:00 smithi090 ceph-mon[41670]: from='client.24853 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi090-93601-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:01.059 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:00 smithi090 ceph-mon[41670]: osdmap e173: 8 total, 8 up, 8 in 2024-04-14T15:14:01.059 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:00 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi090-93601-1","app": "app2"}]: dispatch 2024-04-14T15:14:01.060 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:00 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1885308872' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi090-93363-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:01.060 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:00 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi090-93601-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:01.060 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:00 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3899035157' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi090-93342-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:01.060 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:00 smithi090 ceph-mon[41670]: from='client.24853 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi090-93601-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:01.060 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:00 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:01.060 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:00 smithi090 ceph-mon[30399]: pgmap v203: 394 pgs: 33 unknown, 2 clean+premerge+peered, 359 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 511 B/s wr, 0 op/s 2024-04-14T15:14:01.060 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:00 smithi090 ceph-mon[30399]: 11.14 scrub starts 2024-04-14T15:14:01.060 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:00 smithi090 ceph-mon[30399]: 11.14 scrub ok 2024-04-14T15:14:01.060 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:00 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-14T15:14:01.060 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:00 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi090-93601-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:01.060 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:00 smithi090 ceph-mon[30399]: from='client.24853 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi090-93601-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:01.061 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:00 smithi090 ceph-mon[30399]: 6.16 scrub starts 2024-04-14T15:14:01.061 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:00 smithi090 ceph-mon[30399]: 6.16 scrub ok 2024-04-14T15:14:01.061 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:00 smithi090 ceph-mon[30399]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:14:01.061 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:00 smithi090 ceph-mon[30399]: from='client.24853 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi090-93601-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:01.061 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:00 smithi090 ceph-mon[30399]: osdmap e173: 8 total, 8 up, 8 in 2024-04-14T15:14:01.061 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:00 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi090-93601-1","app": "app2"}]: dispatch 2024-04-14T15:14:01.061 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:00 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1885308872' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi090-93363-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:01.061 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:00 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi090-93601-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:01.061 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:00 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3899035157' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi090-93342-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:01.061 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:00 smithi090 ceph-mon[30399]: from='client.24853 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi090-93601-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:01.061 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:00 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:01.127 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:14:01.151 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:01.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:01.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:14:01.191 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:00 smithi092 ceph-mon[33966]: pgmap v203: 394 pgs: 33 unknown, 2 clean+premerge+peered, 359 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 511 B/s wr, 0 op/s 2024-04-14T15:14:01.191 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:00 smithi092 ceph-mon[33966]: 11.14 scrub starts 2024-04-14T15:14:01.191 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:00 smithi092 ceph-mon[33966]: 11.14 scrub ok 2024-04-14T15:14:01.191 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:00 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-14T15:14:01.191 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:00 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi090-93601-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:01.191 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:00 smithi092 ceph-mon[33966]: from='client.24853 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi090-93601-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:01.191 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:00 smithi092 ceph-mon[33966]: 6.16 scrub starts 2024-04-14T15:14:01.191 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:00 smithi092 ceph-mon[33966]: 6.16 scrub ok 2024-04-14T15:14:01.191 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:00 smithi092 ceph-mon[33966]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:14:01.192 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:00 smithi092 ceph-mon[33966]: from='client.24853 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi090-93601-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:01.192 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:00 smithi092 ceph-mon[33966]: osdmap e173: 8 total, 8 up, 8 in 2024-04-14T15:14:01.192 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:00 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi090-93601-1","app": "app2"}]: dispatch 2024-04-14T15:14:01.192 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:00 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1885308872' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi090-93363-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:01.192 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:00 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi090-93601-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:01.192 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:00 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3899035157' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi090-93342-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:01.192 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:00 smithi092 ceph-mon[33966]: from='client.24853 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi090-93601-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:01.192 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:00 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:01.223 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-04-14T15:14:01.223 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:14:01.253 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:01.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:01.277 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:01.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:01.378 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:14:01.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:01.407 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:01.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:14:01.478 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-14T15:14:01.508 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:14:01.513 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:14:01.534 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:01.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:01.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:14:01.635 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:14:01.659 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:01.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:01.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:14:01.761 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:14:01.785 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:01.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:01.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:14:01.887 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:14:01.904 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-14T15:14:01.911 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:01.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:01.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:14:02.013 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:14:02.036 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:02.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:02.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:14:02.138 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:14:02.144 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:14:02.162 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:02.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:02.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:14:02.176 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:01 smithi090 ceph-mon[41670]: 6.d scrub starts 2024-04-14T15:14:02.176 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:01 smithi090 ceph-mon[41670]: 6.d scrub ok 2024-04-14T15:14:02.176 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:01 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-04-14T15:14:02.176 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:01 smithi090 ceph-mon[41670]: 2.19 scrub starts 2024-04-14T15:14:02.176 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:01 smithi090 ceph-mon[41670]: 2.19 scrub ok 2024-04-14T15:14:02.176 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:01 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1885308872' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi090-93363-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:02.176 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:01 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3899035157' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi090-93342-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:02.176 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:01 smithi090 ceph-mon[41670]: from='client.24853 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi090-93601-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:02.177 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:01 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "28"}]': finished 2024-04-14T15:14:02.177 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:01 smithi090 ceph-mon[41670]: osdmap e174: 8 total, 8 up, 8 in 2024-04-14T15:14:02.177 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:01 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi090-93601-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-04-14T15:14:02.177 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:01 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi090-93601-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-04-14T15:14:02.177 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:01 smithi090 ceph-mon[41670]: from='client.24853 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi090-93601-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-04-14T15:14:02.177 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:01 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:02.177 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:01 smithi090 ceph-mon[30399]: 6.d scrub starts 2024-04-14T15:14:02.177 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:01 smithi090 ceph-mon[30399]: 6.d scrub ok 2024-04-14T15:14:02.177 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:01 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-04-14T15:14:02.177 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:01 smithi090 ceph-mon[30399]: 2.19 scrub starts 2024-04-14T15:14:02.177 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:01 smithi090 ceph-mon[30399]: 2.19 scrub ok 2024-04-14T15:14:02.177 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:01 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1885308872' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi090-93363-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:02.178 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:01 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3899035157' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi090-93342-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:02.178 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:01 smithi090 ceph-mon[30399]: from='client.24853 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi090-93601-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:02.178 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:01 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "28"}]': finished 2024-04-14T15:14:02.178 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:01 smithi090 ceph-mon[30399]: osdmap e174: 8 total, 8 up, 8 in 2024-04-14T15:14:02.178 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:01 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi090-93601-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-04-14T15:14:02.178 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:01 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi090-93601-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-04-14T15:14:02.178 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:01 smithi090 ceph-mon[30399]: from='client.24853 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi090-93601-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-04-14T15:14:02.178 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:01 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:02.265 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:02.266 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:01 smithi092 ceph-mon[33966]: 6.d scrub starts 2024-04-14T15:14:02.266 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:01 smithi092 ceph-mon[33966]: 6.d scrub ok 2024-04-14T15:14:02.266 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:01 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-04-14T15:14:02.266 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:01 smithi092 ceph-mon[33966]: 2.19 scrub starts 2024-04-14T15:14:02.266 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:01 smithi092 ceph-mon[33966]: 2.19 scrub ok 2024-04-14T15:14:02.266 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:01 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1885308872' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi090-93363-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:02.267 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:01 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3899035157' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi090-93342-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:02.267 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:01 smithi092 ceph-mon[33966]: from='client.24853 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi090-93601-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:02.267 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:01 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "28"}]': finished 2024-04-14T15:14:02.267 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:01 smithi092 ceph-mon[33966]: osdmap e174: 8 total, 8 up, 8 in 2024-04-14T15:14:02.267 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:01 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi090-93601-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-04-14T15:14:02.267 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:01 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi090-93601-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-04-14T15:14:02.267 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:01 smithi092 ceph-mon[33966]: from='client.24853 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi090-93601-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-04-14T15:14:02.267 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:01 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:02.290 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:02.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:02.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:02.392 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:14:02.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:02.417 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:02.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:14:02.518 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:14:02.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:02.543 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:02.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:14:02.644 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:14:02.654 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:14:02.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:02.669 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:02.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:14:02.770 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:14:02.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:02.797 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:02.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:14:02.828 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: Running main() from gmock_main.cc 2024-04-14T15:14:02.828 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: [==========] Running 21 tests from 5 test suites. 2024-04-14T15:14:02.828 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2024-04-14T15:14:02.828 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2024-04-14T15:14:02.828 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: seed 93724 2024-04-14T15:14:02.829 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2024-04-14T15:14:02.829 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (2010 ms) 2024-04-14T15:14:02.829 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2024-04-14T15:14:02.829 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (2033 ms) 2024-04-14T15:14:02.829 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2024-04-14T15:14:02.829 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (1698 ms) 2024-04-14T15:14:02.829 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2024-04-14T15:14:02.829 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (2007 ms) 2024-04-14T15:14:02.829 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2024-04-14T15:14:02.829 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (4000 ms) 2024-04-14T15:14:02.829 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (11749 ms total) 2024-04-14T15:14:02.829 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: 2024-04-14T15:14:02.829 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP 2024-04-14T15:14:02.829 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2024-04-14T15:14:02.829 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (4097 ms) 2024-04-14T15:14:02.830 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2024-04-14T15:14:02.830 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (3956 ms) 2024-04-14T15:14:02.830 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2024-04-14T15:14:02.830 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (5993 ms) 2024-04-14T15:14:02.830 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2024-04-14T15:14:02.830 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (3980 ms) 2024-04-14T15:14:02.830 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2024-04-14T15:14:02.830 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (2037 ms) 2024-04-14T15:14:02.830 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-04-14T15:14:02.830 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-2778-ga971fd5f/rpm/el9/BUILD/ceph-19.0.0-2778-ga971fd5f/src/test/librados/snapshots_cxx.cc:463: Skipped 2024-04-14T15:14:02.830 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: 2024-04-14T15:14:02.830 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback (0 ms) 2024-04-14T15:14:02.830 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2024-04-14T15:14:02.831 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: deleting snap 14 in pool LibRadosSnapshotsSelfManagedPP_smithi090-93724-7 2024-04-14T15:14:02.831 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: waiting for snaps to purge 2024-04-14T15:14:02.831 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (18029 ms) 2024-04-14T15:14:02.831 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP (38092 ms total) 2024-04-14T15:14:02.831 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: 2024-04-14T15:14:02.831 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode 2024-04-14T15:14:02.831 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected 2024-04-14T15:14:02.831 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected (9 ms) 2024-04-14T15:14:02.831 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance 2024-04-14T15:14:02.831 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance (5022 ms) 2024-04-14T15:14:02.831 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode (5031 ms total) 2024-04-14T15:14:02.831 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: 2024-04-14T15:14:02.831 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2024-04-14T15:14:02.831 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2024-04-14T15:14:02.832 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (2998 ms) 2024-04-14T15:14:02.832 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2024-04-14T15:14:02.898 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:14:02.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:02.926 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:02.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:14:03.027 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:14:03.036 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:02 smithi090 ceph-mon[41670]: pgmap v206: 393 pgs: 64 unknown, 1 clean+premerge+peered, 328 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:14:03.037 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:02 smithi090 ceph-mon[41670]: from='client.24853 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi090-93601-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-04-14T15:14:03.037 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:02 smithi090 ceph-mon[41670]: osdmap e175: 8 total, 8 up, 8 in 2024-04-14T15:14:03.037 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:02 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi090-93601-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-04-14T15:14:03.037 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:02 smithi090 ceph-mon[41670]: from='client.24853 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi090-93601-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-04-14T15:14:03.037 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:02 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:03.037 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:02 smithi090 ceph-mon[30399]: pgmap v206: 393 pgs: 64 unknown, 1 clean+premerge+peered, 328 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:14:03.037 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:02 smithi090 ceph-mon[30399]: from='client.24853 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi090-93601-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-04-14T15:14:03.037 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:02 smithi090 ceph-mon[30399]: osdmap e175: 8 total, 8 up, 8 in 2024-04-14T15:14:03.037 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:02 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi090-93601-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-04-14T15:14:03.037 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:02 smithi090 ceph-mon[30399]: from='client.24853 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi090-93601-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-04-14T15:14:03.038 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:02 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:03.052 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:03.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:03.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:14:03.154 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:14:03.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:03.179 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:03.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:14:03.281 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:14:03.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:03.306 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:03.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:14:03.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:02 smithi092 ceph-mon[33966]: pgmap v206: 393 pgs: 64 unknown, 1 clean+premerge+peered, 328 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:14:03.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:02 smithi092 ceph-mon[33966]: from='client.24853 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi090-93601-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-04-14T15:14:03.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:02 smithi092 ceph-mon[33966]: osdmap e175: 8 total, 8 up, 8 in 2024-04-14T15:14:03.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:02 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi090-93601-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-04-14T15:14:03.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:02 smithi092 ceph-mon[33966]: from='client.24853 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi090-93601-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-04-14T15:14:03.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:02 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:03.407 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:14:03.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:03.433 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:03.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:14:03.534 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:03.558 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:03.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:03.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:03.660 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:14:03.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:03.686 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:03.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:14:03.787 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:14:03.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:03.801+0000 7f1ec19c3700 1 -- 172.21.15.90:0/3658472754 >> v1:172.21.15.90:6789/0 conn(0x7f1ebc10b980 legacy=0x7f1ebc10dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:03.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:03.801+0000 7f1ec19c3700 1 -- 172.21.15.90:0/3658472754 shutdown_connections 2024-04-14T15:14:03.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:03.801+0000 7f1ec19c3700 1 -- 172.21.15.90:0/3658472754 >> 172.21.15.90:0/3658472754 conn(0x7f1ebc0fd2c0 msgr2=0x7f1ebc0ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:14:03.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:03.801+0000 7f1ec19c3700 1 -- 172.21.15.90:0/3658472754 shutdown_connections 2024-04-14T15:14:03.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:03.801+0000 7f1ec19c3700 1 -- 172.21.15.90:0/3658472754 wait complete. 2024-04-14T15:14:03.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:03.801+0000 7f1ec19c3700 1 librados: starting msgr at 2024-04-14T15:14:03.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:03.801+0000 7f1ec19c3700 1 librados: starting objecter 2024-04-14T15:14:03.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:03.802+0000 7f1ec19c3700 1 Processor -- start 2024-04-14T15:14:03.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:03.802+0000 7f1ec19c3700 1 -- start start 2024-04-14T15:14:03.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:03.802+0000 7f1ec19c3700 1 librados: setting wanted keys 2024-04-14T15:14:03.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:03.802+0000 7f1ec19c3700 1 librados: calling monclient init 2024-04-14T15:14:03.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:03.802+0000 7f1ec19c3700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1ebc1031f0 con 0x7f1ebc10b980 2024-04-14T15:14:03.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:03.802+0000 7f1ec19c3700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1ebc103400 con 0x7f1ebc10f520 2024-04-14T15:14:03.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:03.802+0000 7f1ec19c3700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1ebc103630 con 0x7f1ebc103a90 2024-04-14T15:14:03.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:03.802+0000 7f1ebaffd700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f1ebc103a90 0x7f1ebc1016f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:44198/0 (socket says 172.21.15.90:44198) 2024-04-14T15:14:03.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:03.802+0000 7f1ebaffd700 1 -- 172.21.15.90:0/1538160927 learned_addr learned my addr 172.21.15.90:0/1538160927 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:14:03.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:03.802+0000 7f1ec09c1700 10 client.?.objecter ms_handle_connect 0x7f1ebc103a90 2024-04-14T15:14:03.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:03.802+0000 7f1ec09c1700 10 client.?.objecter resend_mon_ops 2024-04-14T15:14:03.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:03.802+0000 7f1ec09c1700 10 client.?.objecter ms_handle_connect 0x7f1ebc10f520 2024-04-14T15:14:03.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:03.802+0000 7f1ec09c1700 10 client.?.objecter resend_mon_ops 2024-04-14T15:14:03.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:03.802+0000 7f1ec09c1700 10 client.?.objecter ms_handle_connect 0x7f1ebc10b980 2024-04-14T15:14:03.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:03.802+0000 7f1ec09c1700 10 client.?.objecter resend_mon_ops 2024-04-14T15:14:03.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:03.802+0000 7f1ec09c1700 1 -- 172.21.15.90:0/1538160927 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2020344634 0 0) 0x7f1ebc103400 con 0x7f1ebc10f520 2024-04-14T15:14:03.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:03.802+0000 7f1ec09c1700 1 -- 172.21.15.90:0/1538160927 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1ea40035b0 con 0x7f1ebc10f520 2024-04-14T15:14:03.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:03.802+0000 7f1ec09c1700 1 -- 172.21.15.90:0/1538160927 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3841597350 0 0) 0x7f1ebc103630 con 0x7f1ebc103a90 2024-04-14T15:14:03.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:03.802+0000 7f1ec09c1700 1 -- 172.21.15.90:0/1538160927 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1ebc103400 con 0x7f1ebc103a90 2024-04-14T15:14:03.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:03.802+0000 7f1ec09c1700 1 -- 172.21.15.90:0/1538160927 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3344051763 0 0) 0x7f1ebc1031f0 con 0x7f1ebc10b980 2024-04-14T15:14:03.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:03.802+0000 7f1ec09c1700 1 -- 172.21.15.90:0/1538160927 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1ebc103630 con 0x7f1ebc10b980 2024-04-14T15:14:03.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:03.803+0000 7f1ec09c1700 1 -- 172.21.15.90:0/1538160927 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1979363494 0 0) 0x7f1ea40035b0 con 0x7f1ebc10f520 2024-04-14T15:14:03.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:03.803+0000 7f1ec09c1700 1 -- 172.21.15.90:0/1538160927 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1ebc1031f0 con 0x7f1ebc10f520 2024-04-14T15:14:03.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:03.803+0000 7f1ec09c1700 1 -- 172.21.15.90:0/1538160927 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f1eb4001bb0 con 0x7f1ebc10f520 2024-04-14T15:14:03.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:03.803+0000 7f1ec09c1700 1 -- 172.21.15.90:0/1538160927 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 203795436 0 0) 0x7f1ebc103630 con 0x7f1ebc10b980 2024-04-14T15:14:03.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:03.803+0000 7f1ec09c1700 1 -- 172.21.15.90:0/1538160927 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1ea40035b0 con 0x7f1ebc10b980 2024-04-14T15:14:03.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:03.803+0000 7f1ec09c1700 1 -- 172.21.15.90:0/1538160927 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f1eb00015d0 con 0x7f1ebc10b980 2024-04-14T15:14:03.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:03.803+0000 7f1ec09c1700 1 -- 172.21.15.90:0/1538160927 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1254560337 0 0) 0x7f1ebc1031f0 con 0x7f1ebc10f520 2024-04-14T15:14:03.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:03.803+0000 7f1ec09c1700 1 -- 172.21.15.90:0/1538160927 >> v1:172.21.15.90:6790/0 conn(0x7f1ebc103a90 legacy=0x7f1ebc1016f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:03.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:03.803+0000 7f1ec09c1700 1 -- 172.21.15.90:0/1538160927 >> v1:172.21.15.90:6789/0 conn(0x7f1ebc10b980 legacy=0x7f1ebc101e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:03.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:03.803+0000 7f1ec09c1700 1 -- 172.21.15.90:0/1538160927 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ebc1ad510 con 0x7f1ebc10f520 2024-04-14T15:14:03.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:03.803+0000 7f1ec19c3700 1 -- 172.21.15.90:0/1538160927 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1ebc1aa500 con 0x7f1ebc10f520 2024-04-14T15:14:03.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:03.803+0000 7f1ec19c3700 10 client.48271.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:14:03.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:03.803+0000 7f1ec19c3700 1 -- 172.21.15.90:0/1538160927 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f1ebc1aa9c0 con 0x7f1ebc10f520 2024-04-14T15:14:03.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:03.803+0000 7f1ec19c3700 1 librados: init done 2024-04-14T15:14:03.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:03.803+0000 7f1ec09c1700 1 -- 172.21.15.90:0/1538160927 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f1eb4003ec0 con 0x7f1ebc10f520 2024-04-14T15:14:03.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:03.803+0000 7f1ec09c1700 1 -- 172.21.15.90:0/1538160927 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f1eb4004ef0 con 0x7f1ebc10f520 2024-04-14T15:14:03.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:03.804+0000 7f1ec19c3700 1 -- 172.21.15.90:0/1538160927 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ea8005120 con 0x7f1ebc10f520 2024-04-14T15:14:03.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:03.804+0000 7f1ec09c1700 1 -- 172.21.15.90:0/1538160927 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f1eb40051c0 con 0x7f1ebc10f520 2024-04-14T15:14:03.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:03.805+0000 7f1ec09c1700 1 -- 172.21.15.90:0/1538160927 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(175..175 src has 51..175) ==== 10018+0+0 (unknown 2879307384 0 0) 0x7f1eb408a2a0 con 0x7f1ebc10f520 2024-04-14T15:14:03.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:03.805+0000 7f1ec09c1700 10 client.48271.objecter ms_dispatch 0x7f1ebc001550 osd_map(175..175 src has 51..175) 2024-04-14T15:14:03.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:03.805+0000 7f1ec09c1700 3 client.48271.objecter handle_osd_map got epochs [175,175] > 0 2024-04-14T15:14:03.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:03.805+0000 7f1ec09c1700 3 client.48271.objecter handle_osd_map decoding full epoch 175 2024-04-14T15:14:03.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:03.805+0000 7f1ec09c1700 20 client.48271.objecter dump_active .. 0 homeless 2024-04-14T15:14:03.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:03.805+0000 7f1ec09c1700 10 client.48271.objecter ms_handle_connect 0x7f1ea4070fe0 2024-04-14T15:14:03.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:03.809+0000 7f1ec09c1700 1 -- 172.21.15.90:0/1538160927 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f1eb4056570 con 0x7f1ebc10f520 2024-04-14T15:14:03.812 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:03.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:03.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:14:03.914 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:03.937 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:03.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:03.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:03.951 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:03 smithi090 ceph-mon[41670]: 6.f scrub starts 2024-04-14T15:14:03.951 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:03 smithi090 ceph-mon[41670]: 6.f scrub ok 2024-04-14T15:14:03.951 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:03 smithi090 ceph-mon[41670]: 5.3 scrub starts 2024-04-14T15:14:03.951 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:03 smithi090 ceph-mon[41670]: 5.3 scrub ok 2024-04-14T15:14:03.951 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:03 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:14:03.952 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:03 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:14:03.952 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:03 smithi090 ceph-mon[41670]: from='client.24853 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi090-93601-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-04-14T15:14:03.952 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:03 smithi090 ceph-mon[41670]: osdmap e176: 8 total, 8 up, 8 in 2024-04-14T15:14:03.952 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:03 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi090-93601-1","app":"app1","key":"key1"}]: dispatch 2024-04-14T15:14:03.952 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:03 smithi090 ceph-mon[41670]: from='client.24853 ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi090-93601-1","app":"app1","key":"key1"}]: dispatch 2024-04-14T15:14:03.952 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:03 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1214404709' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi090-93342-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:03.952 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:03 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2741251536' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi090-93363-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:03.952 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:03 smithi090 ceph-mon[41670]: from='client.48265 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi090-93342-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:03.952 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:03 smithi090 ceph-mon[41670]: from='client.48259 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi090-93363-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:03.952 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:03 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:03.953 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:03 smithi090 ceph-mon[30399]: 6.f scrub starts 2024-04-14T15:14:03.953 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:03 smithi090 ceph-mon[30399]: 6.f scrub ok 2024-04-14T15:14:03.953 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:03 smithi090 ceph-mon[30399]: 5.3 scrub starts 2024-04-14T15:14:03.953 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:03 smithi090 ceph-mon[30399]: 5.3 scrub ok 2024-04-14T15:14:03.953 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:03 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:14:03.953 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:03 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:14:03.953 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:03 smithi090 ceph-mon[30399]: from='client.24853 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi090-93601-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-04-14T15:14:03.953 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:03 smithi090 ceph-mon[30399]: osdmap e176: 8 total, 8 up, 8 in 2024-04-14T15:14:03.953 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:03 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi090-93601-1","app":"app1","key":"key1"}]: dispatch 2024-04-14T15:14:03.954 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:03 smithi090 ceph-mon[30399]: from='client.24853 ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi090-93601-1","app":"app1","key":"key1"}]: dispatch 2024-04-14T15:14:03.954 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:03 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1214404709' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi090-93342-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:03.954 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:03 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2741251536' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi090-93363-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:03.954 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:03 smithi090 ceph-mon[30399]: from='client.48265 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi090-93342-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:03.954 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:03 smithi090 ceph-mon[30399]: from='client.48259 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi090-93363-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:03.954 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:03 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:04.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:04.028+0000 7f1ec19c3700 1 -- 172.21.15.90:0/1538160927 --> v1:172.21.15.90:6800/420935427 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1ea8000b90 con 0x7f1ea4070fe0 2024-04-14T15:14:04.039 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:14:04.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:04.053+0000 7f1ec09c1700 1 -- 172.21.15.90:0/1538160927 <== mgr.14610 v1:172.21.15.90:6800/420935427 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+995012 (unknown 2965378022 0 1858641572) 0x7f1ea8000b90 con 0x7f1ea4070fe0 2024-04-14T15:14:04.066 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:14:04.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:04.058+0000 7f1ec19c3700 10 librados: watch_flush enter 2024-04-14T15:14:04.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:04.058+0000 7f1ec19c3700 10 librados: watch_flush exit 2024-04-14T15:14:04.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:04.058+0000 7f1ec19c3700 20 client.48271.objecter shutdown clearing up homeless session... 2024-04-14T15:14:04.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:04.058+0000 7f1ec19c3700 10 client.48271.objecter successfully canceled tick 2024-04-14T15:14:04.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:04.058+0000 7f1ec19c3700 1 -- 172.21.15.90:0/1538160927 >> v1:172.21.15.90:6800/420935427 conn(0x7f1ea4070fe0 legacy=0x7f1ea40734a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:04.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:04.058+0000 7f1ec19c3700 1 -- 172.21.15.90:0/1538160927 >> v1:172.21.15.92:6789/0 conn(0x7f1ebc10f520 legacy=0x7f1ebc1a8d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:04.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:04.058+0000 7f1ec19c3700 1 -- 172.21.15.90:0/1538160927 shutdown_connections 2024-04-14T15:14:04.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:04.058+0000 7f1ec19c3700 1 -- 172.21.15.90:0/1538160927 >> 172.21.15.90:0/1538160927 conn(0x7f1ebc0fd2c0 msgr2=0x7f1ebc0ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:14:04.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:04.058+0000 7f1ec19c3700 1 -- 172.21.15.90:0/1538160927 shutdown_connections 2024-04-14T15:14:04.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:04.058+0000 7f1ec19c3700 1 -- 172.21.15.90:0/1538160927 wait complete. 2024-04-14T15:14:04.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:04.059+0000 7f1ec19c3700 1 librados: shutdown 2024-04-14T15:14:04.080 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-14T15:14:04.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:04.081 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:04.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:14:04.089 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-04-14T15:14:04.089 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:14:04.090 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:03 smithi092 ceph-mon[33966]: 6.f scrub starts 2024-04-14T15:14:04.090 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:03 smithi092 ceph-mon[33966]: 6.f scrub ok 2024-04-14T15:14:04.090 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:03 smithi092 ceph-mon[33966]: 5.3 scrub starts 2024-04-14T15:14:04.090 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:03 smithi092 ceph-mon[33966]: 5.3 scrub ok 2024-04-14T15:14:04.090 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:03 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:14:04.090 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:03 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:14:04.090 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:03 smithi092 ceph-mon[33966]: from='client.24853 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi090-93601-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-04-14T15:14:04.090 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:03 smithi092 ceph-mon[33966]: osdmap e176: 8 total, 8 up, 8 in 2024-04-14T15:14:04.090 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:03 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3851923713' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi090-93601-1","app":"app1","key":"key1"}]: dispatch 2024-04-14T15:14:04.090 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:03 smithi092 ceph-mon[33966]: from='client.24853 ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi090-93601-1","app":"app1","key":"key1"}]: dispatch 2024-04-14T15:14:04.090 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:03 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1214404709' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi090-93342-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:04.090 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:03 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2741251536' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi090-93363-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:04.090 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:03 smithi092 ceph-mon[33966]: from='client.48265 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi090-93342-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:04.091 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:03 smithi092 ceph-mon[33966]: from='client.48259 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi090-93363-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:04.091 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:03 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:04.182 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:04.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:04.207 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:04.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:04.308 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:14:04.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:04.332 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:04.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:14:04.434 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:14:04.442 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:14:03 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:14:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:14:04.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:04.458 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:04.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:14:04.559 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:14:04.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:04.584 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:04.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:14:04.685 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:14:04.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:04.709 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:04.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:14:04.811 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:14:04.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:04.840 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:04.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:14:04.903 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-14T15:14:04.911 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":210,"stamp":"2024-04-14T15:14:03.822256+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":227105343,"num_objects":975,"num_object_clones":541,"num_object_copies":2925,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":754,"num_whiteouts":0,"num_read":2496,"num_read_kb":50407,"num_write":2274,"num_write_kb":229745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1772586,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":221,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2940,"ondisk_log_size":2940,"up":984,"acting":984,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":1047,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":1248096,"kb_used_data":695936,"kb_used_omap":12,"kb_used_meta":551923,"kb_avail":748713120,"statfs":{"total":767960285184,"available":766682234880,"internally_reserved":0,"allocated":712638464,"data_stored":692884005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":565169714},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.997019"},"pg_stats":[{"pgid":"198.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:14:03.820054+0000","last_change":"2024-04-14T15:14:03.820054+0000","last_active":"2024-04-14T15:14:03.820054+0000","last_peered":"2024-04-14T15:14:03.820054+0000","last_clean":"2024-04-14T15:14:03.820054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:14:03.820054+0000","last_undegraded":"2024-04-14T15:14:03.820054+0000","last_fullsized":"2024-04-14T15:14:03.820054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.820054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"197.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:14:03.820054+0000","last_change":"2024-04-14T15:14:03.820054+0000","last_active":"2024-04-14T15:14:03.820054+0000","last_peered":"2024-04-14T15:14:03.820054+0000","last_clean":"2024-04-14T15:14:03.820054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:14:03.820054+0000","last_undegraded":"2024-04-14T15:14:03.820054+0000","last_fullsized":"2024-04-14T15:14:03.820054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.820054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"198.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:14:03.820054+0000","last_change":"2024-04-14T15:14:03.820054+0000","last_active":"2024-04-14T15:14:03.820054+0000","last_peered":"2024-04-14T15:14:03.820054+0000","last_clean":"2024-04-14T15:14:03.820054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:14:03.820054+0000","last_undegraded":"2024-04-14T15:14:03.820054+0000","last_fullsized":"2024-04-14T15:14:03.820054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.820054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"197.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:14:03.820054+0000","last_change":"2024-04-14T15:14:03.820054+0000","last_active":"2024-04-14T15:14:03.820054+0000","last_peered":"2024-04-14T15:14:03.820054+0000","last_clean":"2024-04-14T15:14:03.820054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:14:03.820054+0000","last_undegraded":"2024-04-14T15:14:03.820054+0000","last_fullsized":"2024-04-14T15:14:03.820054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.820054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"198.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:14:03.820054+0000","last_change":"2024-04-14T15:14:03.820054+0000","last_active":"2024-04-14T15:14:03.820054+0000","last_peered":"2024-04-14T15:14:03.820054+0000","last_clean":"2024-04-14T15:14:03.820054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:14:03.820054+0000","last_undegraded":"2024-04-14T15:14:03.820054+0000","last_fullsized":"2024-04-14T15:14:03.820054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.820054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"197.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:14:03.820054+0000","last_change":"2024-04-14T15:14:03.820054+0000","last_active":"2024-04-14T15:14:03.820054+0000","last_peered":"2024-04-14T15:14:03.820054+0000","last_clean":"2024-04-14T15:14:03.820054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:14:03.820054+0000","last_undegraded":"2024-04-14T15:14:03.820054+0000","last_fullsized":"2024-04-14T15:14:03.820054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.820054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"198.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:14:03.820054+0000","last_change":"2024-04-14T15:14:03.820054+0000","last_active":"2024-04-14T15:14:03.820054+0000","last_peered":"2024-04-14T15:14:03.820054+0000","last_clean":"2024-04-14T15:14:03.820054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:14:03.820054+0000","last_undegraded":"2024-04-14T15:14:03.820054+0000","last_fullsized":"2024-04-14T15:14:03.820054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.820054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"197.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:14:03.820054+0000","last_change":"2024-04-14T15:14:03.820054+0000","last_active":"2024-04-14T15:14:03.820054+0000","last_peered":"2024-04-14T15:14:03.820054+0000","last_clean":"2024-04-14T15:14:03.820054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:14:03.820054+0000","last_undegraded":"2024-04-14T15:14:03.820054+0000","last_fullsized":"2024-04-14T15:14:03.820054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.820054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"198.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:14:03.820054+0000","last_change":"2024-04-14T15:14:03.820054+0000","last_active":"2024-04-14T15:14:03.820054+0000","last_peered":"2024-04-14T15:14:03.820054+0000","last_clean":"2024-04-14T15:14:03.820054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:14:03.820054+0000","last_undegraded":"2024-04-14T15:14:03.820054+0000","last_fullsized":"2024-04-14T15:14:03.820054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.820054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"197.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:14:03.820054+0000","last_change":"2024-04-14T15:14:03.820054+0000","last_active":"2024-04-14T15:14:03.820054+0000","last_peered":"2024-04-14T15:14:03.820054+0000","last_clean":"2024-04-14T15:14:03.820054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:14:03.820054+0000","last_undegraded":"2024-04-14T15:14:03.820054+0000","last_fullsized":"2024-04-14T15:14:03.820054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.820054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"198.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:14:03.820054+0000","last_change":"2024-04-14T15:14:03.820054+0000","last_active":"2024-04-14T15:14:03.820054+0000","last_peered":"2024-04-14T15:14:03.820054+0000","last_clean":"2024-04-14T15:14:03.820054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:14:03.820054+0000","last_undegraded":"2024-04-14T15:14:03.820054+0000","last_fullsized":"2024-04-14T15:14:03.820054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.820054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"197.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:14:03.820054+0000","last_change":"2024-04-14T15:14:03.820054+0000","last_active":"2024-04-14T15:14:03.820054+0000","last_peered":"2024-04-14T15:14:03.820054+0000","last_clean":"2024-04-14T15:14:03.820054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:14:03.820054+0000","last_undegraded":"2024-04-14T15:14:03.820054+0000","last_fullsized":"2024-04-14T15:14:03.820054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.820054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"198.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:14:03.820054+0000","last_change":"2024-04-14T15:14:03.820054+0000","last_active":"2024-04-14T15:14:03.820054+0000","last_peered":"2024-04-14T15:14:03.820054+0000","last_clean":"2024-04-14T15:14:03.820054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:14:03.820054+0000","last_undegraded":"2024-04-14T15:14:03.820054+0000","last_fullsized":"2024-04-14T15:14:03.820054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.820054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"197.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:14:03.820054+0000","last_change":"2024-04-14T15:14:03.820054+0000","last_active":"2024-04-14T15:14:03.820054+0000","last_peered":"2024-04-14T15:14:03.820054+0000","last_clean":"2024-04-14T15:14:03.820054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:14:03.820054+0000","last_undegraded":"2024-04-14T15:14:03.820054+0000","last_fullsized":"2024-04-14T15:14:03.820054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.820054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"198.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:14:03.820054+0000","last_change":"2024-04-14T15:14:03.820054+0000","last_active":"2024-04-14T15:14:03.820054+0000","last_peered":"2024-04-14T15:14:03.820054+0000","last_clean":"2024-04-14T15:14:03.820054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:14:03.820054+0000","last_undegraded":"2024-04-14T15:14:03.820054+0000","last_fullsized":"2024-04-14T15:14:03.820054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.820054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"197.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:14:03.820054+0000","last_change":"2024-04-14T15:14:03.820054+0000","last_active":"2024-04-14T15:14:03.820054+0000","last_peered":"2024-04-14T15:14:03.820054+0000","last_clean":"2024-04-14T15:14:03.820054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:14:03.820054+0000","last_undegraded":"2024-04-14T15:14:03.820054+0000","last_fullsized":"2024-04-14T15:14:03.820054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.820054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"198.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:14:03.820054+0000","last_change":"2024-04-14T15:14:03.820054+0000","last_active":"2024-04-14T15:14:03.820054+0000","last_peered":"2024-04-14T15:14:03.820054+0000","last_clean":"2024-04-14T15:14:03.820054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:14:03.820054+0000","last_undegraded":"2024-04-14T15:14:03.820054+0000","last_fullsized":"2024-04-14T15:14:03.820054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.820054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"197.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:14:03.820054+0000","last_change":"2024-04-14T15:14:03.820054+0000","last_active":"2024-04-14T15:14:03.820054+0000","last_peered":"2024-04-14T15:14:03.820054+0000","last_clean":"2024-04-14T15:14:03.820054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:14:03.820054+0000","last_undegraded":"2024-04-14T15:14:03.820054+0000","last_fullsized":"2024-04-14T15:14:03.820054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.820054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"198.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:14:03.820054+0000","last_change":"2024-04-14T15:14:03.820054+0000","last_active":"2024-04-14T15:14:03.820054+0000","last_peered":"2024-04-14T15:14:03.820054+0000","last_clean":"2024-04-14T15:14:03.820054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:14:03.820054+0000","last_undegraded":"2024-04-14T15:14:03.820054+0000","last_fullsized":"2024-04-14T15:14:03.820054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.820054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"197.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:14:03.820054+0000","last_change":"2024-04-14T15:14:03.820054+0000","last_active":"2024-04-14T15:14:03.820054+0000","last_peered":"2024-04-14T15:14:03.820054+0000","last_clean":"2024-04-14T15:14:03.820054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:14:03.820054+0000","last_undegraded":"2024-04-14T15:14:03.820054+0000","last_fullsized":"2024-04-14T15:14:03.820054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.820054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"198.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:14:03.820054+0000","last_change":"2024-04-14T15:14:03.820054+0000","last_active":"2024-04-14T15:14:03.820054+0000","last_peered":"2024-04-14T15:14:03.820054+0000","last_clean":"2024-04-14T15:14:03.820054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:14:03.820054+0000","last_undegraded":"2024-04-14T15:14:03.820054+0000","last_fullsized":"2024-04-14T15:14:03.820054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.820054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"197.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:14:03.820054+0000","last_change":"2024-04-14T15:14:03.820054+0000","last_active":"2024-04-14T15:14:03.820054+0000","last_peered":"2024-04-14T15:14:03.820054+0000","last_clean":"2024-04-14T15:14:03.820054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:14:03.820054+0000","last_undegraded":"2024-04-14T15:14:03.820054+0000","last_fullsized":"2024-04-14T15:14:03.820054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.820054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"198.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:14:03.820054+0000","last_change":"2024-04-14T15:14:03.820054+0000","last_active":"2024-04-14T15:14:03.820054+0000","last_peered":"2024-04-14T15:14:03.820054+0000","last_clean":"2024-04-14T15:14:03.820054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:14:03.820054+0000","last_undegraded":"2024-04-14T15:14:03.820054+0000","last_fullsized":"2024-04-14T15:14:03.820054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.820054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"197.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:14:03.820054+0000","last_change":"2024-04-14T15:14:03.820054+0000","last_active":"2024-04-14T15:14:03.820054+0000","last_peered":"2024-04-14T15:14:03.820054+0000","last_clean":"2024-04-14T15:14:03.820054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:14:03.820054+0000","last_undegraded":"2024-04-14T15:14:03.820054+0000","last_fullsized":"2024-04-14T15:14:03.820054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.820054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"198.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:14:03.820054+0000","last_change":"2024-04-14T15:14:03.820054+0000","last_active":"2024-04-14T15:14:03.820054+0000","last_peered":"2024-04-14T15:14:03.820054+0000","last_clean":"2024-04-14T15:14:03.820054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:14:03.820054+0000","last_undegraded":"2024-04-14T15:14:03.820054+0000","last_fullsized":"2024-04-14T15:14:03.820054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.820054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"197.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:14:03.820054+0000","last_change":"2024-04-14T15:14:03.820054+0000","last_active":"2024-04-14T15:14:03.820054+0000","last_peered":"2024-04-14T15:14:03.820054+0000","last_clean":"2024-04-14T15:14:03.820054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:14:03.820054+0000","last_undegraded":"2024-04-14T15:14:03.820054+0000","last_fullsized":"2024-04-14T15:14:03.820054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.820054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"198.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:14:03.820054+0000","last_change":"2024-04-14T15:14:03.820054+0000","last_active":"2024-04-14T15:14:03.820054+0000","last_peered":"2024-04-14T15:14:03.820054+0000","last_clean":"2024-04-14T15:14:03.820054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:14:03.820054+0000","last_undegraded":"2024-04-14T15:14:03.820054+0000","last_fullsized":"2024-04-14T15:14:03.820054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.820054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"197.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:14:03.820054+0000","last_change":"2024-04-14T15:14:03.820054+0000","last_active":"2024-04-14T15:14:03.820054+0000","last_peered":"2024-04-14T15:14:03.820054+0000","last_clean":"2024-04-14T15:14:03.820054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:14:03.820054+0000","last_undegraded":"2024-04-14T15:14:03.820054+0000","last_fullsized":"2024-04-14T15:14:03.820054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.820054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"198.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:14:03.820054+0000","last_change":"2024-04-14T15:14:03.820054+0000","last_active":"2024-04-14T15:14:03.820054+0000","last_peered":"2024-04-14T15:14:03.820054+0000","last_clean":"2024-04-14T15:14:03.820054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:14:03.820054+0000","last_undegraded":"2024-04-14T15:14:03.820054+0000","last_fullsized":"2024-04-14T15:14:03.820054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.820054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"197.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:14:03.820054+0000","last_change":"2024-04-14T15:14:03.820054+0000","last_active":"2024-04-14T15:14:03.820054+0000","last_peered":"2024-04-14T15:14:03.820054+0000","last_clean":"2024-04-14T15:14:03.820054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:14:03.820054+0000","last_undegraded":"2024-04-14T15:14:03.820054+0000","last_fullsized":"2024-04-14T15:14:03.820054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.820054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"198.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:14:03.820054+0000","last_change":"2024-04-14T15:14:03.820054+0000","last_active":"2024-04-14T15:14:03.820054+0000","last_peered":"2024-04-14T15:14:03.820054+0000","last_clean":"2024-04-14T15:14:03.820054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:14:03.820054+0000","last_undegraded":"2024-04-14T15:14:03.820054+0000","last_fullsized":"2024-04-14T15:14:03.820054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.820054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"197.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:14:03.820054+0000","last_change":"2024-04-14T15:14:03.820054+0000","last_active":"2024-04-14T15:14:03.820054+0000","last_peered":"2024-04-14T15:14:03.820054+0000","last_clean":"2024-04-14T15:14:03.820054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:14:03.820054+0000","last_undegraded":"2024-04-14T15:14:03.820054+0000","last_fullsized":"2024-04-14T15:14:03.820054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.820054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"198.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:14:03.820054+0000","last_change":"2024-04-14T15:14:03.820054+0000","last_active":"2024-04-14T15:14:03.820054+0000","last_peered":"2024-04-14T15:14:03.820054+0000","last_clean":"2024-04-14T15:14:03.820054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:14:03.820054+0000","last_undegraded":"2024-04-14T15:14:03.820054+0000","last_fullsized":"2024-04-14T15:14:03.820054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.820054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"197.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:14:03.820054+0000","last_change":"2024-04-14T15:14:03.820054+0000","last_active":"2024-04-14T15:14:03.820054+0000","last_peered":"2024-04-14T15:14:03.820054+0000","last_clean":"2024-04-14T15:14:03.820054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:14:03.820054+0000","last_undegraded":"2024-04-14T15:14:03.820054+0000","last_fullsized":"2024-04-14T15:14:03.820054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.820054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"198.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:14:03.820054+0000","last_change":"2024-04-14T15:14:03.820054+0000","last_active":"2024-04-14T15:14:03.820054+0000","last_peered":"2024-04-14T15:14:03.820054+0000","last_clean":"2024-04-14T15:14:03.820054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:14:03.820054+0000","last_undegraded":"2024-04-14T15:14:03.820054+0000","last_fullsized":"2024-04-14T15:14:03.820054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.820054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"197.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:14:03.820054+0000","last_change":"2024-04-14T15:14:03.820054+0000","last_active":"2024-04-14T15:14:03.820054+0000","last_peered":"2024-04-14T15:14:03.820054+0000","last_clean":"2024-04-14T15:14:03.820054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:14:03.820054+0000","last_undegraded":"2024-04-14T15:14:03.820054+0000","last_fullsized":"2024-04-14T15:14:03.820054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.820054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"198.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:14:03.820054+0000","last_change":"2024-04-14T15:14:03.820054+0000","last_active":"2024-04-14T15:14:03.820054+0000","last_peered":"2024-04-14T15:14:03.820054+0000","last_clean":"2024-04-14T15:14:03.820054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:14:03.820054+0000","last_undegraded":"2024-04-14T15:14:03.820054+0000","last_fullsized":"2024-04-14T15:14:03.820054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.820054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"197.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:14:03.820054+0000","last_change":"2024-04-14T15:14:03.820054+0000","last_active":"2024-04-14T15:14:03.820054+0000","last_peered":"2024-04-14T15:14:03.820054+0000","last_clean":"2024-04-14T15:14:03.820054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:14:03.820054+0000","last_undegraded":"2024-04-14T15:14:03.820054+0000","last_fullsized":"2024-04-14T15:14:03.820054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.820054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"198.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:14:03.820054+0000","last_change":"2024-04-14T15:14:03.820054+0000","last_active":"2024-04-14T15:14:03.820054+0000","last_peered":"2024-04-14T15:14:03.820054+0000","last_clean":"2024-04-14T15:14:03.820054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:14:03.820054+0000","last_undegraded":"2024-04-14T15:14:03.820054+0000","last_fullsized":"2024-04-14T15:14:03.820054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.820054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"197.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:14:03.820054+0000","last_change":"2024-04-14T15:14:03.820054+0000","last_active":"2024-04-14T15:14:03.820054+0000","last_peered":"2024-04-14T15:14:03.820054+0000","last_clean":"2024-04-14T15:14:03.820054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:14:03.820054+0000","last_undegraded":"2024-04-14T15:14:03.820054+0000","last_fullsized":"2024-04-14T15:14:03.820054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.820054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"198.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:14:03.820054+0000","last_change":"2024-04-14T15:14:03.820054+0000","last_active":"2024-04-14T15:14:03.820054+0000","last_peered":"2024-04-14T15:14:03.820054+0000","last_clean":"2024-04-14T15:14:03.820054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:14:03.820054+0000","last_undegraded":"2024-04-14T15:14:03.820054+0000","last_fullsized":"2024-04-14T15:14:03.820054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.820054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"197.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:14:03.820054+0000","last_change":"2024-04-14T15:14:03.820054+0000","last_active":"2024-04-14T15:14:03.820054+0000","last_peered":"2024-04-14T15:14:03.820054+0000","last_clean":"2024-04-14T15:14:03.820054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:14:03.820054+0000","last_undegraded":"2024-04-14T15:14:03.820054+0000","last_fullsized":"2024-04-14T15:14:03.820054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.820054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"198.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:14:03.820054+0000","last_change":"2024-04-14T15:14:03.820054+0000","last_active":"2024-04-14T15:14:03.820054+0000","last_peered":"2024-04-14T15:14:03.820054+0000","last_clean":"2024-04-14T15:14:03.820054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:14:03.820054+0000","last_undegraded":"2024-04-14T15:14:03.820054+0000","last_fullsized":"2024-04-14T15:14:03.820054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.820054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"197.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:14:03.820054+0000","last_change":"2024-04-14T15:14:03.820054+0000","last_active":"2024-04-14T15:14:03.820054+0000","last_peered":"2024-04-14T15:14:03.820054+0000","last_clean":"2024-04-14T15:14:03.820054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:14:03.820054+0000","last_undegraded":"2024-04-14T15:14:03.820054+0000","last_fullsized":"2024-04-14T15:14:03.820054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.820054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"198.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:14:03.820054+0000","last_change":"2024-04-14T15:14:03.820054+0000","last_active":"2024-04-14T15:14:03.820054+0000","last_peered":"2024-04-14T15:14:03.820054+0000","last_clean":"2024-04-14T15:14:03.820054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:14:03.820054+0000","last_undegraded":"2024-04-14T15:14:03.820054+0000","last_fullsized":"2024-04-14T15:14:03.820054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.820054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"197.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:14:03.820054+0000","last_change":"2024-04-14T15:14:03.820054+0000","last_active":"2024-04-14T15:14:03.820054+0000","last_peered":"2024-04-14T15:14:03.820054+0000","last_clean":"2024-04-14T15:14:03.820054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:14:03.820054+0000","last_undegraded":"2024-04-14T15:14:03.820054+0000","last_fullsized":"2024-04-14T15:14:03.820054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.820054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"198.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:14:03.820054+0000","last_change":"2024-04-14T15:14:03.820054+0000","last_active":"2024-04-14T15:14:03.820054+0000","last_peered":"2024-04-14T15:14:03.820054+0000","last_clean":"2024-04-14T15:14:03.820054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:14:03.820054+0000","last_undegraded":"2024-04-14T15:14:03.820054+0000","last_fullsized":"2024-04-14T15:14:03.820054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.820054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"197.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:14:03.820054+0000","last_change":"2024-04-14T15:14:03.820054+0000","last_active":"2024-04-14T15:14:03.820054+0000","last_peered":"2024-04-14T15:14:03.820054+0000","last_clean":"2024-04-14T15:14:03.820054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:14:03.820054+0000","last_undegraded":"2024-04-14T15:14:03.820054+0000","last_fullsized":"2024-04-14T15:14:03.820054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.820054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"198.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:14:03.820054+0000","last_change":"2024-04-14T15:14:03.820054+0000","last_active":"2024-04-14T15:14:03.820054+0000","last_peered":"2024-04-14T15:14:03.820054+0000","last_clean":"2024-04-14T15:14:03.820054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:14:03.820054+0000","last_undegraded":"2024-04-14T15:14:03.820054+0000","last_fullsized":"2024-04-14T15:14:03.820054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.820054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"197.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:14:03.820054+0000","last_change":"2024-04-14T15:14:03.820054+0000","last_active":"2024-04-14T15:14:03.820054+0000","last_peered":"2024-04-14T15:14:03.820054+0000","last_clean":"2024-04-14T15:14:03.820054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:14:03.820054+0000","last_undegraded":"2024-04-14T15:14:03.820054+0000","last_fullsized":"2024-04-14T15:14:03.820054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.820054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"192.7","version":"0'0","reported_seq":10,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.860265+0000","last_change":"2024-04-14T15:13:58.859417+0000","last_active":"2024-04-14T15:13:58.860265+0000","last_peered":"2024-04-14T15:13:58.860265+0000","last_clean":"2024-04-14T15:13:58.860265+0000","last_became_active":"2024-04-14T15:13:58.859127+0000","last_became_peered":"2024-04-14T15:13:58.859127+0000","last_unstale":"2024-04-14T15:13:58.860265+0000","last_undegraded":"2024-04-14T15:13:58.860265+0000","last_fullsized":"2024-04-14T15:13:58.860265+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:57.819059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:57.819059+0000","last_clean_scrub_stamp":"2024-04-14T15:13:57.819059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:15:16.334669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"198.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:14:03.820054+0000","last_change":"2024-04-14T15:14:03.820054+0000","last_active":"2024-04-14T15:14:03.820054+0000","last_peered":"2024-04-14T15:14:03.820054+0000","last_clean":"2024-04-14T15:14:03.820054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:14:03.820054+0000","last_undegraded":"2024-04-14T15:14:03.820054+0000","last_fullsized":"2024-04-14T15:14:03.820054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.820054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"197.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:14:03.820054+0000","last_change":"2024-04-14T15:14:03.820054+0000","last_active":"2024-04-14T15:14:03.820054+0000","last_peered":"2024-04-14T15:14:03.820054+0000","last_clean":"2024-04-14T15:14:03.820054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:14:03.820054+0000","last_undegraded":"2024-04-14T15:14:03.820054+0000","last_fullsized":"2024-04-14T15:14:03.820054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.820054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"192.6","version":"171'1","reported_seq":11,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.866587+0000","last_change":"2024-04-14T15:13:58.859183+0000","last_active":"2024-04-14T15:13:58.866587+0000","last_peered":"2024-04-14T15:13:58.866587+0000","last_clean":"2024-04-14T15:13:58.866587+0000","last_became_active":"2024-04-14T15:13:58.858885+0000","last_became_peered":"2024-04-14T15:13:58.858885+0000","last_unstale":"2024-04-14T15:13:58.866587+0000","last_undegraded":"2024-04-14T15:13:58.866587+0000","last_fullsized":"2024-04-14T15:13:58.866587+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:57.819059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:57.819059+0000","last_clean_scrub_stamp":"2024-04-14T15:13:57.819059+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:15:00.797968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"198.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:14:03.820054+0000","last_change":"2024-04-14T15:14:03.820054+0000","last_active":"2024-04-14T15:14:03.820054+0000","last_peered":"2024-04-14T15:14:03.820054+0000","last_clean":"2024-04-14T15:14:03.820054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:14:03.820054+0000","last_undegraded":"2024-04-14T15:14:03.820054+0000","last_fullsized":"2024-04-14T15:14:03.820054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.820054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"197.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:14:03.820054+0000","last_change":"2024-04-14T15:14:03.820054+0000","last_active":"2024-04-14T15:14:03.820054+0000","last_peered":"2024-04-14T15:14:03.820054+0000","last_clean":"2024-04-14T15:14:03.820054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:14:03.820054+0000","last_undegraded":"2024-04-14T15:14:03.820054+0000","last_fullsized":"2024-04-14T15:14:03.820054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.820054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"192.5","version":"0'0","reported_seq":10,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.848175+0000","last_change":"2024-04-14T15:13:58.847450+0000","last_active":"2024-04-14T15:13:58.848175+0000","last_peered":"2024-04-14T15:13:58.848175+0000","last_clean":"2024-04-14T15:13:58.848175+0000","last_became_active":"2024-04-14T15:13:58.847033+0000","last_became_peered":"2024-04-14T15:13:58.847033+0000","last_unstale":"2024-04-14T15:13:58.848175+0000","last_undegraded":"2024-04-14T15:13:58.848175+0000","last_fullsized":"2024-04-14T15:13:58.848175+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:57.819059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:57.819059+0000","last_clean_scrub_stamp":"2024-04-14T15:13:57.819059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:15:12.936852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"198.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:14:03.820054+0000","last_change":"2024-04-14T15:14:03.820054+0000","last_active":"2024-04-14T15:14:03.820054+0000","last_peered":"2024-04-14T15:14:03.820054+0000","last_clean":"2024-04-14T15:14:03.820054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:14:03.820054+0000","last_undegraded":"2024-04-14T15:14:03.820054+0000","last_fullsized":"2024-04-14T15:14:03.820054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.820054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"197.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:14:03.820054+0000","last_change":"2024-04-14T15:14:03.820054+0000","last_active":"2024-04-14T15:14:03.820054+0000","last_peered":"2024-04-14T15:14:03.820054+0000","last_clean":"2024-04-14T15:14:03.820054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:14:03.820054+0000","last_undegraded":"2024-04-14T15:14:03.820054+0000","last_fullsized":"2024-04-14T15:14:03.820054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.820054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"192.4","version":"0'0","reported_seq":10,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.864826+0000","last_change":"2024-04-14T15:13:58.859948+0000","last_active":"2024-04-14T15:13:58.864826+0000","last_peered":"2024-04-14T15:13:58.864826+0000","last_clean":"2024-04-14T15:13:58.864826+0000","last_became_active":"2024-04-14T15:13:58.859643+0000","last_became_peered":"2024-04-14T15:13:58.859643+0000","last_unstale":"2024-04-14T15:13:58.864826+0000","last_undegraded":"2024-04-14T15:13:58.864826+0000","last_fullsized":"2024-04-14T15:13:58.864826+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:57.819059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:57.819059+0000","last_clean_scrub_stamp":"2024-04-14T15:13:57.819059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:15:27.727373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"198.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:14:03.820054+0000","last_change":"2024-04-14T15:14:03.820054+0000","last_active":"2024-04-14T15:14:03.820054+0000","last_peered":"2024-04-14T15:14:03.820054+0000","last_clean":"2024-04-14T15:14:03.820054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:14:03.820054+0000","last_undegraded":"2024-04-14T15:14:03.820054+0000","last_fullsized":"2024-04-14T15:14:03.820054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.820054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"197.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:14:03.820054+0000","last_change":"2024-04-14T15:14:03.820054+0000","last_active":"2024-04-14T15:14:03.820054+0000","last_peered":"2024-04-14T15:14:03.820054+0000","last_clean":"2024-04-14T15:14:03.820054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:14:03.820054+0000","last_undegraded":"2024-04-14T15:14:03.820054+0000","last_fullsized":"2024-04-14T15:14:03.820054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.820054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"192.3","version":"0'0","reported_seq":10,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.850879+0000","last_change":"2024-04-14T15:13:58.849979+0000","last_active":"2024-04-14T15:13:58.850879+0000","last_peered":"2024-04-14T15:13:58.850879+0000","last_clean":"2024-04-14T15:13:58.850879+0000","last_became_active":"2024-04-14T15:13:58.848586+0000","last_became_peered":"2024-04-14T15:13:58.848586+0000","last_unstale":"2024-04-14T15:13:58.850879+0000","last_undegraded":"2024-04-14T15:13:58.850879+0000","last_fullsized":"2024-04-14T15:13:58.850879+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:57.819059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:57.819059+0000","last_clean_scrub_stamp":"2024-04-14T15:13:57.819059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:15:24.212023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"198.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:14:03.820054+0000","last_change":"2024-04-14T15:14:03.820054+0000","last_active":"2024-04-14T15:14:03.820054+0000","last_peered":"2024-04-14T15:14:03.820054+0000","last_clean":"2024-04-14T15:14:03.820054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:14:03.820054+0000","last_undegraded":"2024-04-14T15:14:03.820054+0000","last_fullsized":"2024-04-14T15:14:03.820054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.820054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"197.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:14:03.820054+0000","last_change":"2024-04-14T15:14:03.820054+0000","last_active":"2024-04-14T15:14:03.820054+0000","last_peered":"2024-04-14T15:14:03.820054+0000","last_clean":"2024-04-14T15:14:03.820054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:14:03.820054+0000","last_undegraded":"2024-04-14T15:14:03.820054+0000","last_fullsized":"2024-04-14T15:14:03.820054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.820054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"192.2","version":"0'0","reported_seq":10,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.857424+0000","last_change":"2024-04-14T15:13:58.854920+0000","last_active":"2024-04-14T15:13:58.857424+0000","last_peered":"2024-04-14T15:13:58.857424+0000","last_clean":"2024-04-14T15:13:58.857424+0000","last_became_active":"2024-04-14T15:13:58.854536+0000","last_became_peered":"2024-04-14T15:13:58.854536+0000","last_unstale":"2024-04-14T15:13:58.857424+0000","last_undegraded":"2024-04-14T15:13:58.857424+0000","last_fullsized":"2024-04-14T15:13:58.857424+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:57.819059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:57.819059+0000","last_clean_scrub_stamp":"2024-04-14T15:13:57.819059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:15:17.040898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"198.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:14:03.820054+0000","last_change":"2024-04-14T15:14:03.820054+0000","last_active":"2024-04-14T15:14:03.820054+0000","last_peered":"2024-04-14T15:14:03.820054+0000","last_clean":"2024-04-14T15:14:03.820054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:14:03.820054+0000","last_undegraded":"2024-04-14T15:14:03.820054+0000","last_fullsized":"2024-04-14T15:14:03.820054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.820054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"197.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:14:03.820054+0000","last_change":"2024-04-14T15:14:03.820054+0000","last_active":"2024-04-14T15:14:03.820054+0000","last_peered":"2024-04-14T15:14:03.820054+0000","last_clean":"2024-04-14T15:14:03.820054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:14:03.820054+0000","last_undegraded":"2024-04-14T15:14:03.820054+0000","last_fullsized":"2024-04-14T15:14:03.820054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.820054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"192.1","version":"0'0","reported_seq":10,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.859583+0000","last_change":"2024-04-14T15:13:58.858278+0000","last_active":"2024-04-14T15:13:58.859583+0000","last_peered":"2024-04-14T15:13:58.859583+0000","last_clean":"2024-04-14T15:13:58.859583+0000","last_became_active":"2024-04-14T15:13:58.857939+0000","last_became_peered":"2024-04-14T15:13:58.857939+0000","last_unstale":"2024-04-14T15:13:58.859583+0000","last_undegraded":"2024-04-14T15:13:58.859583+0000","last_fullsized":"2024-04-14T15:13:58.859583+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:57.819059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:57.819059+0000","last_clean_scrub_stamp":"2024-04-14T15:13:57.819059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:15:08.008738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"198.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:14:03.820054+0000","last_change":"2024-04-14T15:14:03.820054+0000","last_active":"2024-04-14T15:14:03.820054+0000","last_peered":"2024-04-14T15:14:03.820054+0000","last_clean":"2024-04-14T15:14:03.820054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:14:03.820054+0000","last_undegraded":"2024-04-14T15:14:03.820054+0000","last_fullsized":"2024-04-14T15:14:03.820054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.820054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"197.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:14:03.820054+0000","last_change":"2024-04-14T15:14:03.820054+0000","last_active":"2024-04-14T15:14:03.820054+0000","last_peered":"2024-04-14T15:14:03.820054+0000","last_clean":"2024-04-14T15:14:03.820054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:14:03.820054+0000","last_undegraded":"2024-04-14T15:14:03.820054+0000","last_fullsized":"2024-04-14T15:14:03.820054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:03.820054+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.820054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"192.0","version":"0'0","reported_seq":10,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.858739+0000","last_change":"2024-04-14T15:13:58.856955+0000","last_active":"2024-04-14T15:13:58.858739+0000","last_peered":"2024-04-14T15:13:58.858739+0000","last_clean":"2024-04-14T15:13:58.858739+0000","last_became_active":"2024-04-14T15:13:58.856295+0000","last_became_peered":"2024-04-14T15:13:58.856295+0000","last_unstale":"2024-04-14T15:13:58.858739+0000","last_undegraded":"2024-04-14T15:13:58.858739+0000","last_fullsized":"2024-04-14T15:13:58.858739+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:57.819059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:57.819059+0000","last_clean_scrub_stamp":"2024-04-14T15:13:57.819059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:15:09.453152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"186.6","version":"0'0","reported_seq":32,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.834219+0000","last_change":"2024-04-14T15:13:52.858544+0000","last_active":"2024-04-14T15:13:58.834219+0000","last_peered":"2024-04-14T15:13:58.834219+0000","last_clean":"2024-04-14T15:13:58.834219+0000","last_became_active":"2024-04-14T15:13:50.889000+0000","last_became_peered":"2024-04-14T15:13:50.889000+0000","last_unstale":"2024-04-14T15:13:58.834219+0000","last_undegraded":"2024-04-14T15:13:58.834219+0000","last_fullsized":"2024-04-14T15:13:58.834219+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:49.823383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:49.823383+0000","last_clean_scrub_stamp":"2024-04-14T15:13:49.823383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:15:11.993570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0060028019999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"3","length":"1"}]},{"pgid":"172.10","version":"156'10","reported_seq":74,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.841993+0000","last_change":"2024-04-14T15:13:36.836370+0000","last_active":"2024-04-14T15:13:58.841993+0000","last_peered":"2024-04-14T15:13:58.841993+0000","last_clean":"2024-04-14T15:13:58.841993+0000","last_became_active":"2024-04-14T15:13:36.835930+0000","last_became_peered":"2024-04-14T15:13:36.835930+0000","last_unstale":"2024-04-14T15:13:58.841993+0000","last_undegraded":"2024-04-14T15:13:58.841993+0000","last_fullsized":"2024-04-14T15:13:58.841993+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_clean_scrub_stamp":"2024-04-14T15:13:35.805229+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:53.254262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101,"num_objects":10,"num_object_clones":6,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":10,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"186.7","version":"0'0","reported_seq":32,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.842310+0000","last_change":"2024-04-14T15:13:52.856284+0000","last_active":"2024-04-14T15:13:58.842310+0000","last_peered":"2024-04-14T15:13:58.842310+0000","last_clean":"2024-04-14T15:13:58.842310+0000","last_became_active":"2024-04-14T15:13:50.865292+0000","last_became_peered":"2024-04-14T15:13:50.865292+0000","last_unstale":"2024-04-14T15:13:58.842310+0000","last_undegraded":"2024-04-14T15:13:58.842310+0000","last_fullsized":"2024-04-14T15:13:58.842310+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:49.823383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:49.823383+0000","last_clean_scrub_stamp":"2024-04-14T15:13:49.823383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:56.695469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031340349999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"3","length":"1"}]},{"pgid":"172.11","version":"156'6","reported_seq":68,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.844871+0000","last_change":"2024-04-14T15:13:36.839799+0000","last_active":"2024-04-14T15:13:58.844871+0000","last_peered":"2024-04-14T15:13:58.844871+0000","last_clean":"2024-04-14T15:13:58.844871+0000","last_became_active":"2024-04-14T15:13:36.839495+0000","last_became_peered":"2024-04-14T15:13:36.839495+0000","last_unstale":"2024-04-14T15:13:58.844871+0000","last_undegraded":"2024-04-14T15:13:58.844871+0000","last_fullsized":"2024-04-14T15:13:58.844871+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_clean_scrub_stamp":"2024-04-14T15:13:35.805229+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:57.842840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":6,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"186.4","version":"0'0","reported_seq":32,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.831624+0000","last_change":"2024-04-14T15:13:52.850938+0000","last_active":"2024-04-14T15:13:58.831624+0000","last_peered":"2024-04-14T15:13:58.831624+0000","last_clean":"2024-04-14T15:13:58.831624+0000","last_became_active":"2024-04-14T15:13:50.863019+0000","last_became_peered":"2024-04-14T15:13:50.863019+0000","last_unstale":"2024-04-14T15:13:58.831624+0000","last_undegraded":"2024-04-14T15:13:58.831624+0000","last_fullsized":"2024-04-14T15:13:58.831624+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:49.823383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:49.823383+0000","last_clean_scrub_stamp":"2024-04-14T15:13:49.823383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:15:00.582157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0093935670000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"3","length":"1"}]},{"pgid":"172.12","version":"156'8","reported_seq":72,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.842144+0000","last_change":"2024-04-14T15:13:36.839422+0000","last_active":"2024-04-14T15:13:58.842144+0000","last_peered":"2024-04-14T15:13:58.842144+0000","last_clean":"2024-04-14T15:13:58.842144+0000","last_became_active":"2024-04-14T15:13:36.839140+0000","last_became_peered":"2024-04-14T15:13:36.839140+0000","last_unstale":"2024-04-14T15:13:58.842144+0000","last_undegraded":"2024-04-14T15:13:58.842144+0000","last_fullsized":"2024-04-14T15:13:58.842144+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_clean_scrub_stamp":"2024-04-14T15:13:35.805229+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:15:02.768401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":82,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":8,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"186.5","version":"0'0","reported_seq":32,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.836365+0000","last_change":"2024-04-14T15:13:52.854873+0000","last_active":"2024-04-14T15:13:58.836365+0000","last_peered":"2024-04-14T15:13:58.836365+0000","last_clean":"2024-04-14T15:13:58.836365+0000","last_became_active":"2024-04-14T15:13:50.867861+0000","last_became_peered":"2024-04-14T15:13:50.867861+0000","last_unstale":"2024-04-14T15:13:58.836365+0000","last_undegraded":"2024-04-14T15:13:58.836365+0000","last_fullsized":"2024-04-14T15:13:58.836365+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:49.823383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:49.823383+0000","last_clean_scrub_stamp":"2024-04-14T15:13:49.823383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:15:18.314031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0038106540000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"3","length":"1"}]},{"pgid":"172.13","version":"156'6","reported_seq":68,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.839089+0000","last_change":"2024-04-14T15:13:36.843534+0000","last_active":"2024-04-14T15:13:58.839089+0000","last_peered":"2024-04-14T15:13:58.839089+0000","last_clean":"2024-04-14T15:13:58.839089+0000","last_became_active":"2024-04-14T15:13:36.843254+0000","last_became_peered":"2024-04-14T15:13:36.843254+0000","last_unstale":"2024-04-14T15:13:58.839089+0000","last_undegraded":"2024-04-14T15:13:58.839089+0000","last_fullsized":"2024-04-14T15:13:58.839089+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_clean_scrub_stamp":"2024-04-14T15:13:35.805229+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:45.221751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":6,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"186.2","version":"0'0","reported_seq":32,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.828850+0000","last_change":"2024-04-14T15:13:52.855697+0000","last_active":"2024-04-14T15:13:58.828850+0000","last_peered":"2024-04-14T15:13:58.828850+0000","last_clean":"2024-04-14T15:13:58.828850+0000","last_became_active":"2024-04-14T15:13:50.888483+0000","last_became_peered":"2024-04-14T15:13:50.888483+0000","last_unstale":"2024-04-14T15:13:58.828850+0000","last_undegraded":"2024-04-14T15:13:58.828850+0000","last_fullsized":"2024-04-14T15:13:58.828850+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:49.823383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:49.823383+0000","last_clean_scrub_stamp":"2024-04-14T15:13:49.823383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:49.982106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001389291,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"3","length":"1"}]},{"pgid":"172.14","version":"156'13","reported_seq":80,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.833175+0000","last_change":"2024-04-14T15:13:36.835895+0000","last_active":"2024-04-14T15:13:58.833175+0000","last_peered":"2024-04-14T15:13:58.833175+0000","last_clean":"2024-04-14T15:13:58.833175+0000","last_became_active":"2024-04-14T15:13:36.835613+0000","last_became_peered":"2024-04-14T15:13:36.835613+0000","last_unstale":"2024-04-14T15:13:58.833175+0000","last_undegraded":"2024-04-14T15:13:58.833175+0000","last_fullsized":"2024-04-14T15:13:58.833175+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_clean_scrub_stamp":"2024-04-14T15:13:35.805229+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:46.796028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131,"num_objects":13,"num_object_clones":8,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":13,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"186.3","version":"0'0","reported_seq":32,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.847954+0000","last_change":"2024-04-14T15:13:52.860434+0000","last_active":"2024-04-14T15:13:58.847954+0000","last_peered":"2024-04-14T15:13:58.847954+0000","last_clean":"2024-04-14T15:13:58.847954+0000","last_became_active":"2024-04-14T15:13:50.892791+0000","last_became_peered":"2024-04-14T15:13:50.892791+0000","last_unstale":"2024-04-14T15:13:58.847954+0000","last_undegraded":"2024-04-14T15:13:58.847954+0000","last_fullsized":"2024-04-14T15:13:58.847954+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:49.823383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:49.823383+0000","last_clean_scrub_stamp":"2024-04-14T15:13:49.823383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:15:10.990491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057504050000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3","length":"1"}]},{"pgid":"172.15","version":"156'7","reported_seq":71,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.841972+0000","last_change":"2024-04-14T15:13:36.843835+0000","last_active":"2024-04-14T15:13:58.841972+0000","last_peered":"2024-04-14T15:13:58.841972+0000","last_clean":"2024-04-14T15:13:58.841972+0000","last_became_active":"2024-04-14T15:13:36.843523+0000","last_became_peered":"2024-04-14T15:13:36.843523+0000","last_unstale":"2024-04-14T15:13:58.841972+0000","last_undegraded":"2024-04-14T15:13:58.841972+0000","last_fullsized":"2024-04-14T15:13:58.841972+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_clean_scrub_stamp":"2024-04-14T15:13:35.805229+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:48.914790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":7,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"186.0","version":"163'1","reported_seq":33,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.833821+0000","last_change":"2024-04-14T15:13:52.858668+0000","last_active":"2024-04-14T15:13:58.833821+0000","last_peered":"2024-04-14T15:13:58.833821+0000","last_clean":"2024-04-14T15:13:58.833821+0000","last_became_active":"2024-04-14T15:13:50.882336+0000","last_became_peered":"2024-04-14T15:13:50.882336+0000","last_unstale":"2024-04-14T15:13:58.833821+0000","last_undegraded":"2024-04-14T15:13:58.833821+0000","last_fullsized":"2024-04-14T15:13:58.833821+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:49.823383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:49.823383+0000","last_clean_scrub_stamp":"2024-04-14T15:13:49.823383+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:15:18.889449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0061213980000000001,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3","length":"1"}]},{"pgid":"172.16","version":"156'7","reported_seq":122,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.840557+0000","last_change":"2024-04-14T15:13:36.845354+0000","last_active":"2024-04-14T15:13:58.840557+0000","last_peered":"2024-04-14T15:13:58.840557+0000","last_clean":"2024-04-14T15:13:58.840557+0000","last_became_active":"2024-04-14T15:13:36.845013+0000","last_became_peered":"2024-04-14T15:13:36.845013+0000","last_unstale":"2024-04-14T15:13:58.840557+0000","last_undegraded":"2024-04-14T15:13:58.840557+0000","last_fullsized":"2024-04-14T15:13:58.840557+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_clean_scrub_stamp":"2024-04-14T15:13:35.805229+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:38.495048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":57,"num_read_kb":57,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":7,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"186.1","version":"0'0","reported_seq":32,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.833270+0000","last_change":"2024-04-14T15:13:52.858447+0000","last_active":"2024-04-14T15:13:58.833270+0000","last_peered":"2024-04-14T15:13:58.833270+0000","last_clean":"2024-04-14T15:13:58.833270+0000","last_became_active":"2024-04-14T15:13:50.888202+0000","last_became_peered":"2024-04-14T15:13:50.888202+0000","last_unstale":"2024-04-14T15:13:58.833270+0000","last_undegraded":"2024-04-14T15:13:58.833270+0000","last_fullsized":"2024-04-14T15:13:58.833270+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:49.823383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:49.823383+0000","last_clean_scrub_stamp":"2024-04-14T15:13:49.823383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:15:02.343521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0064656169999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"3","length":"1"}]},{"pgid":"172.17","version":"0'0","reported_seq":58,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.832614+0000","last_change":"2024-04-14T15:13:36.839345+0000","last_active":"2024-04-14T15:13:58.832614+0000","last_peered":"2024-04-14T15:13:58.832614+0000","last_clean":"2024-04-14T15:13:58.832614+0000","last_became_active":"2024-04-14T15:13:36.838969+0000","last_became_peered":"2024-04-14T15:13:36.838969+0000","last_unstale":"2024-04-14T15:13:58.832614+0000","last_undegraded":"2024-04-14T15:13:58.832614+0000","last_fullsized":"2024-04-14T15:13:58.832614+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_clean_scrub_stamp":"2024-04-14T15:13:35.805229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:15:00.589457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"186.e","version":"0'0","reported_seq":32,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.839647+0000","last_change":"2024-04-14T15:13:52.856217+0000","last_active":"2024-04-14T15:13:58.839647+0000","last_peered":"2024-04-14T15:13:58.839647+0000","last_clean":"2024-04-14T15:13:58.839647+0000","last_became_active":"2024-04-14T15:13:50.877732+0000","last_became_peered":"2024-04-14T15:13:50.877732+0000","last_unstale":"2024-04-14T15:13:58.839647+0000","last_undegraded":"2024-04-14T15:13:58.839647+0000","last_fullsized":"2024-04-14T15:13:58.839647+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:49.823383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:49.823383+0000","last_clean_scrub_stamp":"2024-04-14T15:13:49.823383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:56.900212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015887399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"3","length":"1"}]},{"pgid":"172.18","version":"156'10","reported_seq":77,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.837486+0000","last_change":"2024-04-14T15:13:36.837744+0000","last_active":"2024-04-14T15:13:58.837486+0000","last_peered":"2024-04-14T15:13:58.837486+0000","last_clean":"2024-04-14T15:13:58.837486+0000","last_became_active":"2024-04-14T15:13:36.837290+0000","last_became_peered":"2024-04-14T15:13:36.837290+0000","last_unstale":"2024-04-14T15:13:58.837486+0000","last_undegraded":"2024-04-14T15:13:58.837486+0000","last_fullsized":"2024-04-14T15:13:58.837486+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_clean_scrub_stamp":"2024-04-14T15:13:35.805229+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:15:04.821731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":7,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":10,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"186.f","version":"0'0","reported_seq":32,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.834203+0000","last_change":"2024-04-14T15:13:52.853308+0000","last_active":"2024-04-14T15:13:58.834203+0000","last_peered":"2024-04-14T15:13:58.834203+0000","last_clean":"2024-04-14T15:13:58.834203+0000","last_became_active":"2024-04-14T15:13:50.895569+0000","last_became_peered":"2024-04-14T15:13:50.895569+0000","last_unstale":"2024-04-14T15:13:58.834203+0000","last_undegraded":"2024-04-14T15:13:58.834203+0000","last_fullsized":"2024-04-14T15:13:58.834203+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:49.823383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:49.823383+0000","last_clean_scrub_stamp":"2024-04-14T15:13:49.823383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:50.258423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002217779,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"3","length":"1"}]},{"pgid":"172.19","version":"156'18","reported_seq":93,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.845446+0000","last_change":"2024-04-14T15:13:36.838918+0000","last_active":"2024-04-14T15:13:58.845446+0000","last_peered":"2024-04-14T15:13:58.845446+0000","last_clean":"2024-04-14T15:13:58.845446+0000","last_became_active":"2024-04-14T15:13:36.838575+0000","last_became_peered":"2024-04-14T15:13:36.838575+0000","last_unstale":"2024-04-14T15:13:58.845446+0000","last_undegraded":"2024-04-14T15:13:58.845446+0000","last_fullsized":"2024-04-14T15:13:58.845446+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_clean_scrub_stamp":"2024-04-14T15:13:35.805229+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:36.191639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189,"num_objects":18,"num_object_clones":13,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":18,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"186.c","version":"0'0","reported_seq":32,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.828954+0000","last_change":"2024-04-14T15:13:52.854720+0000","last_active":"2024-04-14T15:13:58.828954+0000","last_peered":"2024-04-14T15:13:58.828954+0000","last_clean":"2024-04-14T15:13:58.828954+0000","last_became_active":"2024-04-14T15:13:50.855470+0000","last_became_peered":"2024-04-14T15:13:50.855470+0000","last_unstale":"2024-04-14T15:13:58.828954+0000","last_undegraded":"2024-04-14T15:13:58.828954+0000","last_fullsized":"2024-04-14T15:13:58.828954+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:49.823383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:49.823383+0000","last_clean_scrub_stamp":"2024-04-14T15:13:49.823383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:15:16.133400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0050883619999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"3","length":"1"}]},{"pgid":"172.1a","version":"156'5","reported_seq":66,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.831657+0000","last_change":"2024-04-14T15:13:36.827815+0000","last_active":"2024-04-14T15:13:58.831657+0000","last_peered":"2024-04-14T15:13:58.831657+0000","last_clean":"2024-04-14T15:13:58.831657+0000","last_became_active":"2024-04-14T15:13:36.827449+0000","last_became_peered":"2024-04-14T15:13:36.827449+0000","last_unstale":"2024-04-14T15:13:58.831657+0000","last_undegraded":"2024-04-14T15:13:58.831657+0000","last_fullsized":"2024-04-14T15:13:58.831657+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_clean_scrub_stamp":"2024-04-14T15:13:35.805229+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:38.029047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":5,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"186.d","version":"0'0","reported_seq":32,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.832475+0000","last_change":"2024-04-14T15:13:52.849170+0000","last_active":"2024-04-14T15:13:58.832475+0000","last_peered":"2024-04-14T15:13:58.832475+0000","last_clean":"2024-04-14T15:13:58.832475+0000","last_became_active":"2024-04-14T15:13:50.888226+0000","last_became_peered":"2024-04-14T15:13:50.888226+0000","last_unstale":"2024-04-14T15:13:58.832475+0000","last_undegraded":"2024-04-14T15:13:58.832475+0000","last_fullsized":"2024-04-14T15:13:58.832475+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:49.823383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:49.823383+0000","last_clean_scrub_stamp":"2024-04-14T15:13:49.823383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:15:19.170149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030188399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"3","length":"1"}]},{"pgid":"172.1b","version":"156'14","reported_seq":84,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.828416+0000","last_change":"2024-04-14T15:13:36.845268+0000","last_active":"2024-04-14T15:13:58.828416+0000","last_peered":"2024-04-14T15:13:58.828416+0000","last_clean":"2024-04-14T15:13:58.828416+0000","last_became_active":"2024-04-14T15:13:36.844904+0000","last_became_peered":"2024-04-14T15:13:36.844904+0000","last_unstale":"2024-04-14T15:13:58.828416+0000","last_undegraded":"2024-04-14T15:13:58.828416+0000","last_fullsized":"2024-04-14T15:13:58.828416+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_clean_scrub_stamp":"2024-04-14T15:13:35.805229+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:54.128362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":145,"num_objects":14,"num_object_clones":9,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":14,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"186.a","version":"0'0","reported_seq":32,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.832172+0000","last_change":"2024-04-14T15:13:52.849692+0000","last_active":"2024-04-14T15:13:58.832172+0000","last_peered":"2024-04-14T15:13:58.832172+0000","last_clean":"2024-04-14T15:13:58.832172+0000","last_became_active":"2024-04-14T15:13:50.888703+0000","last_became_peered":"2024-04-14T15:13:50.888703+0000","last_unstale":"2024-04-14T15:13:58.832172+0000","last_undegraded":"2024-04-14T15:13:58.832172+0000","last_fullsized":"2024-04-14T15:13:58.832172+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:49.823383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:49.823383+0000","last_clean_scrub_stamp":"2024-04-14T15:13:49.823383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:59.631571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0035217069999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"3","length":"1"}]},{"pgid":"172.1c","version":"156'5","reported_seq":69,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.831827+0000","last_change":"2024-04-14T15:13:36.841773+0000","last_active":"2024-04-14T15:13:58.831827+0000","last_peered":"2024-04-14T15:13:58.831827+0000","last_clean":"2024-04-14T15:13:58.831827+0000","last_became_active":"2024-04-14T15:13:36.841500+0000","last_became_peered":"2024-04-14T15:13:36.841500+0000","last_unstale":"2024-04-14T15:13:58.831827+0000","last_undegraded":"2024-04-14T15:13:58.831827+0000","last_fullsized":"2024-04-14T15:13:58.831827+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_clean_scrub_stamp":"2024-04-14T15:13:35.805229+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:39.117339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":5,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"186.b","version":"0'0","reported_seq":32,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.838919+0000","last_change":"2024-04-14T15:13:52.848707+0000","last_active":"2024-04-14T15:13:58.838919+0000","last_peered":"2024-04-14T15:13:58.838919+0000","last_clean":"2024-04-14T15:13:58.838919+0000","last_became_active":"2024-04-14T15:13:50.897682+0000","last_became_peered":"2024-04-14T15:13:50.897682+0000","last_unstale":"2024-04-14T15:13:58.838919+0000","last_undegraded":"2024-04-14T15:13:58.838919+0000","last_fullsized":"2024-04-14T15:13:58.838919+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:49.823383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:49.823383+0000","last_clean_scrub_stamp":"2024-04-14T15:13:49.823383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:15:14.619341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095489700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"3","length":"1"}]},{"pgid":"172.1d","version":"156'14","reported_seq":82,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.843868+0000","last_change":"2024-04-14T15:13:36.844380+0000","last_active":"2024-04-14T15:13:58.843868+0000","last_peered":"2024-04-14T15:13:58.843868+0000","last_clean":"2024-04-14T15:13:58.843868+0000","last_became_active":"2024-04-14T15:13:36.844056+0000","last_became_peered":"2024-04-14T15:13:36.844056+0000","last_unstale":"2024-04-14T15:13:58.843868+0000","last_undegraded":"2024-04-14T15:13:58.843868+0000","last_fullsized":"2024-04-14T15:13:58.843868+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_clean_scrub_stamp":"2024-04-14T15:13:35.805229+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:58.039301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":145,"num_objects":14,"num_object_clones":9,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":14,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"186.8","version":"0'0","reported_seq":32,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.837278+0000","last_change":"2024-04-14T15:13:52.841998+0000","last_active":"2024-04-14T15:13:58.837278+0000","last_peered":"2024-04-14T15:13:58.837278+0000","last_clean":"2024-04-14T15:13:58.837278+0000","last_became_active":"2024-04-14T15:13:50.867008+0000","last_became_peered":"2024-04-14T15:13:50.867008+0000","last_unstale":"2024-04-14T15:13:58.837278+0000","last_undegraded":"2024-04-14T15:13:58.837278+0000","last_fullsized":"2024-04-14T15:13:58.837278+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:49.823383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:49.823383+0000","last_clean_scrub_stamp":"2024-04-14T15:13:49.823383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:51.967617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002149991,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"3","length":"1"}]},{"pgid":"172.1e","version":"156'3","reported_seq":63,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.848042+0000","last_change":"2024-04-14T15:13:36.836288+0000","last_active":"2024-04-14T15:13:58.848042+0000","last_peered":"2024-04-14T15:13:58.848042+0000","last_clean":"2024-04-14T15:13:58.848042+0000","last_became_active":"2024-04-14T15:13:36.835984+0000","last_became_peered":"2024-04-14T15:13:36.835984+0000","last_unstale":"2024-04-14T15:13:58.848042+0000","last_undegraded":"2024-04-14T15:13:58.848042+0000","last_fullsized":"2024-04-14T15:13:58.848042+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_clean_scrub_stamp":"2024-04-14T15:13:35.805229+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:36.868982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"186.9","version":"0'0","reported_seq":32,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.827961+0000","last_change":"2024-04-14T15:13:52.851880+0000","last_active":"2024-04-14T15:13:58.827961+0000","last_peered":"2024-04-14T15:13:58.827961+0000","last_clean":"2024-04-14T15:13:58.827961+0000","last_became_active":"2024-04-14T15:13:50.856146+0000","last_became_peered":"2024-04-14T15:13:50.856146+0000","last_unstale":"2024-04-14T15:13:58.827961+0000","last_undegraded":"2024-04-14T15:13:58.827961+0000","last_fullsized":"2024-04-14T15:13:58.827961+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:49.823383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:49.823383+0000","last_clean_scrub_stamp":"2024-04-14T15:13:49.823383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:15:15.246461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031044829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"3","length":"1"}]},{"pgid":"172.1f","version":"156'3","reported_seq":63,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.844276+0000","last_change":"2024-04-14T15:13:36.839078+0000","last_active":"2024-04-14T15:13:58.844276+0000","last_peered":"2024-04-14T15:13:58.844276+0000","last_clean":"2024-04-14T15:13:58.844276+0000","last_became_active":"2024-04-14T15:13:36.838471+0000","last_became_peered":"2024-04-14T15:13:36.838471+0000","last_unstale":"2024-04-14T15:13:58.844276+0000","last_undegraded":"2024-04-14T15:13:58.844276+0000","last_fullsized":"2024-04-14T15:13:58.844276+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_clean_scrub_stamp":"2024-04-14T15:13:35.805229+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:15:01.441839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"186.16","version":"0'0","reported_seq":32,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.851383+0000","last_change":"2024-04-14T15:13:52.857331+0000","last_active":"2024-04-14T15:13:58.851383+0000","last_peered":"2024-04-14T15:13:58.851383+0000","last_clean":"2024-04-14T15:13:58.851383+0000","last_became_active":"2024-04-14T15:13:50.878896+0000","last_became_peered":"2024-04-14T15:13:50.878896+0000","last_unstale":"2024-04-14T15:13:58.851383+0000","last_undegraded":"2024-04-14T15:13:58.851383+0000","last_fullsized":"2024-04-14T15:13:58.851383+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:49.823383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:49.823383+0000","last_clean_scrub_stamp":"2024-04-14T15:13:49.823383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:53.147943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003736095,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"3","length":"1"}]},{"pgid":"172.0","version":"156'9","reported_seq":83,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.835070+0000","last_change":"2024-04-14T15:13:45.428596+0000","last_active":"2024-04-14T15:13:58.835070+0000","last_peered":"2024-04-14T15:13:58.835070+0000","last_clean":"2024-04-14T15:13:58.835070+0000","last_became_active":"2024-04-14T15:13:36.840245+0000","last_became_peered":"2024-04-14T15:13:36.840245+0000","last_unstale":"2024-04-14T15:13:58.835070+0000","last_undegraded":"2024-04-14T15:13:58.835070+0000","last_fullsized":"2024-04-14T15:13:58.835070+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"156'9","last_scrub_stamp":"2024-04-14T15:13:45.428502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_clean_scrub_stamp":"2024-04-14T15:13:45.428502+0000","objects_scrubbed":9,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:15:02.929374+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93,"num_objects":9,"num_object_clones":5,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":9,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"186.17","version":"0'0","reported_seq":32,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.832802+0000","last_change":"2024-04-14T15:13:52.853152+0000","last_active":"2024-04-14T15:13:58.832802+0000","last_peered":"2024-04-14T15:13:58.832802+0000","last_clean":"2024-04-14T15:13:58.832802+0000","last_became_active":"2024-04-14T15:13:50.890612+0000","last_became_peered":"2024-04-14T15:13:50.890612+0000","last_unstale":"2024-04-14T15:13:58.832802+0000","last_undegraded":"2024-04-14T15:13:58.832802+0000","last_fullsized":"2024-04-14T15:13:58.832802+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:49.823383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:49.823383+0000","last_clean_scrub_stamp":"2024-04-14T15:13:49.823383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:15:13.369108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023807759999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"3","length":"1"}]},{"pgid":"172.1","version":"156'3","reported_seq":72,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.836360+0000","last_change":"2024-04-14T15:13:45.994548+0000","last_active":"2024-04-14T15:13:58.836360+0000","last_peered":"2024-04-14T15:13:58.836360+0000","last_clean":"2024-04-14T15:13:58.836360+0000","last_became_active":"2024-04-14T15:13:36.840020+0000","last_became_peered":"2024-04-14T15:13:36.840020+0000","last_unstale":"2024-04-14T15:13:58.836360+0000","last_undegraded":"2024-04-14T15:13:58.836360+0000","last_fullsized":"2024-04-14T15:13:58.836360+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"156'3","last_scrub_stamp":"2024-04-14T15:13:45.994467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_clean_scrub_stamp":"2024-04-14T15:13:45.994467+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:54.830384+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"186.14","version":"0'0","reported_seq":32,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.845373+0000","last_change":"2024-04-14T15:13:52.857496+0000","last_active":"2024-04-14T15:13:58.845373+0000","last_peered":"2024-04-14T15:13:58.845373+0000","last_clean":"2024-04-14T15:13:58.845373+0000","last_became_active":"2024-04-14T15:13:50.882720+0000","last_became_peered":"2024-04-14T15:13:50.882720+0000","last_unstale":"2024-04-14T15:13:58.845373+0000","last_undegraded":"2024-04-14T15:13:58.845373+0000","last_fullsized":"2024-04-14T15:13:58.845373+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:49.823383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:49.823383+0000","last_clean_scrub_stamp":"2024-04-14T15:13:49.823383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:50.753539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013912966000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"3","length":"1"}]},{"pgid":"172.2","version":"156'7","reported_seq":80,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.831812+0000","last_change":"2024-04-14T15:13:46.503965+0000","last_active":"2024-04-14T15:13:58.831812+0000","last_peered":"2024-04-14T15:13:58.831812+0000","last_clean":"2024-04-14T15:13:58.831812+0000","last_became_active":"2024-04-14T15:13:36.841465+0000","last_became_peered":"2024-04-14T15:13:36.841465+0000","last_unstale":"2024-04-14T15:13:58.831812+0000","last_undegraded":"2024-04-14T15:13:58.831812+0000","last_fullsized":"2024-04-14T15:13:58.831812+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"156'7","last_scrub_stamp":"2024-04-14T15:13:46.503843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_clean_scrub_stamp":"2024-04-14T15:13:46.503843+0000","objects_scrubbed":7,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:15:10.892534+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":7,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"186.15","version":"0'0","reported_seq":32,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.831253+0000","last_change":"2024-04-14T15:13:52.848451+0000","last_active":"2024-04-14T15:13:58.831253+0000","last_peered":"2024-04-14T15:13:58.831253+0000","last_clean":"2024-04-14T15:13:58.831253+0000","last_became_active":"2024-04-14T15:13:50.879697+0000","last_became_peered":"2024-04-14T15:13:50.879697+0000","last_unstale":"2024-04-14T15:13:58.831253+0000","last_undegraded":"2024-04-14T15:13:58.831253+0000","last_fullsized":"2024-04-14T15:13:58.831253+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:49.823383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:49.823383+0000","last_clean_scrub_stamp":"2024-04-14T15:13:49.823383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:51.164728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003362613,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"3","length":"1"}]},{"pgid":"172.3","version":"156'3","reported_seq":72,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.834981+0000","last_change":"2024-04-14T15:13:45.497236+0000","last_active":"2024-04-14T15:13:58.834981+0000","last_peered":"2024-04-14T15:13:58.834981+0000","last_clean":"2024-04-14T15:13:58.834981+0000","last_became_active":"2024-04-14T15:13:36.839724+0000","last_became_peered":"2024-04-14T15:13:36.839724+0000","last_unstale":"2024-04-14T15:13:58.834981+0000","last_undegraded":"2024-04-14T15:13:58.834981+0000","last_fullsized":"2024-04-14T15:13:58.834981+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"156'3","last_scrub_stamp":"2024-04-14T15:13:45.497155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_clean_scrub_stamp":"2024-04-14T15:13:45.497155+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:15:04.687364+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"186.12","version":"0'0","reported_seq":32,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.832818+0000","last_change":"2024-04-14T15:13:52.845328+0000","last_active":"2024-04-14T15:13:58.832818+0000","last_peered":"2024-04-14T15:13:58.832818+0000","last_clean":"2024-04-14T15:13:58.832818+0000","last_became_active":"2024-04-14T15:13:50.869655+0000","last_became_peered":"2024-04-14T15:13:50.869655+0000","last_unstale":"2024-04-14T15:13:58.832818+0000","last_undegraded":"2024-04-14T15:13:58.832818+0000","last_fullsized":"2024-04-14T15:13:58.832818+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:49.823383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:49.823383+0000","last_clean_scrub_stamp":"2024-04-14T15:13:49.823383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:52.047996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0051647940000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"3","length":"1"}]},{"pgid":"172.4","version":"156'6","reported_seq":78,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.830454+0000","last_change":"2024-04-14T15:13:44.714443+0000","last_active":"2024-04-14T15:13:58.830454+0000","last_peered":"2024-04-14T15:13:58.830454+0000","last_clean":"2024-04-14T15:13:58.830454+0000","last_became_active":"2024-04-14T15:13:36.844915+0000","last_became_peered":"2024-04-14T15:13:36.844915+0000","last_unstale":"2024-04-14T15:13:58.830454+0000","last_undegraded":"2024-04-14T15:13:58.830454+0000","last_fullsized":"2024-04-14T15:13:58.830454+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"156'6","last_scrub_stamp":"2024-04-14T15:13:44.714347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_clean_scrub_stamp":"2024-04-14T15:13:44.714347+0000","objects_scrubbed":6,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:57.741886+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":6,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"186.13","version":"0'0","reported_seq":32,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.848021+0000","last_change":"2024-04-14T15:13:52.860627+0000","last_active":"2024-04-14T15:13:58.848021+0000","last_peered":"2024-04-14T15:13:58.848021+0000","last_clean":"2024-04-14T15:13:58.848021+0000","last_became_active":"2024-04-14T15:13:50.884863+0000","last_became_peered":"2024-04-14T15:13:50.884863+0000","last_unstale":"2024-04-14T15:13:58.848021+0000","last_undegraded":"2024-04-14T15:13:58.848021+0000","last_fullsized":"2024-04-14T15:13:58.848021+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:49.823383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:49.823383+0000","last_clean_scrub_stamp":"2024-04-14T15:13:49.823383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:15:03.831232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0061403179999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3","length":"1"}]},{"pgid":"172.5","version":"0'0","reported_seq":67,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.838475+0000","last_change":"2024-04-14T15:13:45.001351+0000","last_active":"2024-04-14T15:13:58.838475+0000","last_peered":"2024-04-14T15:13:58.838475+0000","last_clean":"2024-04-14T15:13:58.838475+0000","last_became_active":"2024-04-14T15:13:36.835226+0000","last_became_peered":"2024-04-14T15:13:36.835226+0000","last_unstale":"2024-04-14T15:13:58.838475+0000","last_undegraded":"2024-04-14T15:13:58.838475+0000","last_fullsized":"2024-04-14T15:13:58.838475+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:45.001243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_clean_scrub_stamp":"2024-04-14T15:13:45.001243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:49.350300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"186.10","version":"0'0","reported_seq":32,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.832100+0000","last_change":"2024-04-14T15:13:52.850054+0000","last_active":"2024-04-14T15:13:58.832100+0000","last_peered":"2024-04-14T15:13:58.832100+0000","last_clean":"2024-04-14T15:13:58.832100+0000","last_became_active":"2024-04-14T15:13:50.863737+0000","last_became_peered":"2024-04-14T15:13:50.863737+0000","last_unstale":"2024-04-14T15:13:58.832100+0000","last_undegraded":"2024-04-14T15:13:58.832100+0000","last_fullsized":"2024-04-14T15:13:58.832100+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:49.823383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:49.823383+0000","last_clean_scrub_stamp":"2024-04-14T15:13:49.823383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:53.439665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0068976000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"3","length":"1"}]},{"pgid":"172.6","version":"156'8","reported_seq":81,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.832905+0000","last_change":"2024-04-14T15:13:47.011768+0000","last_active":"2024-04-14T15:13:58.832905+0000","last_peered":"2024-04-14T15:13:58.832905+0000","last_clean":"2024-04-14T15:13:58.832905+0000","last_became_active":"2024-04-14T15:13:36.841433+0000","last_became_peered":"2024-04-14T15:13:36.841433+0000","last_unstale":"2024-04-14T15:13:58.832905+0000","last_undegraded":"2024-04-14T15:13:58.832905+0000","last_fullsized":"2024-04-14T15:13:58.832905+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"156'8","last_scrub_stamp":"2024-04-14T15:13:47.011685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_clean_scrub_stamp":"2024-04-14T15:13:47.011685+0000","objects_scrubbed":8,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:53.442333+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":82,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":8,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"186.11","version":"0'0","reported_seq":32,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.831704+0000","last_change":"2024-04-14T15:13:52.848452+0000","last_active":"2024-04-14T15:13:58.831704+0000","last_peered":"2024-04-14T15:13:58.831704+0000","last_clean":"2024-04-14T15:13:58.831704+0000","last_became_active":"2024-04-14T15:13:50.889668+0000","last_became_peered":"2024-04-14T15:13:50.889668+0000","last_unstale":"2024-04-14T15:13:58.831704+0000","last_undegraded":"2024-04-14T15:13:58.831704+0000","last_fullsized":"2024-04-14T15:13:58.831704+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:49.823383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:49.823383+0000","last_clean_scrub_stamp":"2024-04-14T15:13:49.823383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:51.603725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0080253700000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"3","length":"1"}]},{"pgid":"172.7","version":"156'2","reported_seq":71,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.844570+0000","last_change":"2024-04-14T15:13:44.601333+0000","last_active":"2024-04-14T15:13:58.844570+0000","last_peered":"2024-04-14T15:13:58.844570+0000","last_clean":"2024-04-14T15:13:58.844570+0000","last_became_active":"2024-04-14T15:13:36.837803+0000","last_became_peered":"2024-04-14T15:13:36.837803+0000","last_unstale":"2024-04-14T15:13:58.844570+0000","last_undegraded":"2024-04-14T15:13:58.844570+0000","last_fullsized":"2024-04-14T15:13:58.844570+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"156'2","last_scrub_stamp":"2024-04-14T15:13:44.601221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_clean_scrub_stamp":"2024-04-14T15:13:44.601221+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:45.101436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":2,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"186.1e","version":"0'0","reported_seq":32,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.834120+0000","last_change":"2024-04-14T15:13:52.853608+0000","last_active":"2024-04-14T15:13:58.834120+0000","last_peered":"2024-04-14T15:13:58.834120+0000","last_clean":"2024-04-14T15:13:58.834120+0000","last_became_active":"2024-04-14T15:13:50.878795+0000","last_became_peered":"2024-04-14T15:13:50.878795+0000","last_unstale":"2024-04-14T15:13:58.834120+0000","last_undegraded":"2024-04-14T15:13:58.834120+0000","last_fullsized":"2024-04-14T15:13:58.834120+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:49.823383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:49.823383+0000","last_clean_scrub_stamp":"2024-04-14T15:13:49.823383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:15:17.471800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001267493,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"3","length":"1"}]},{"pgid":"172.8","version":"156'1","reported_seq":69,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.832313+0000","last_change":"2024-04-14T15:13:45.258908+0000","last_active":"2024-04-14T15:13:58.832313+0000","last_peered":"2024-04-14T15:13:58.832313+0000","last_clean":"2024-04-14T15:13:58.832313+0000","last_became_active":"2024-04-14T15:13:36.834104+0000","last_became_peered":"2024-04-14T15:13:36.834104+0000","last_unstale":"2024-04-14T15:13:58.832313+0000","last_undegraded":"2024-04-14T15:13:58.832313+0000","last_fullsized":"2024-04-14T15:13:58.832313+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"156'1","last_scrub_stamp":"2024-04-14T15:13:45.258810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_clean_scrub_stamp":"2024-04-14T15:13:45.258810+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:15:06.473960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"186.1f","version":"0'0","reported_seq":32,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.852217+0000","last_change":"2024-04-14T15:13:52.857162+0000","last_active":"2024-04-14T15:13:58.852217+0000","last_peered":"2024-04-14T15:13:58.852217+0000","last_clean":"2024-04-14T15:13:58.852217+0000","last_became_active":"2024-04-14T15:13:50.893601+0000","last_became_peered":"2024-04-14T15:13:50.893601+0000","last_unstale":"2024-04-14T15:13:58.852217+0000","last_undegraded":"2024-04-14T15:13:58.852217+0000","last_fullsized":"2024-04-14T15:13:58.852217+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:49.823383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:49.823383+0000","last_clean_scrub_stamp":"2024-04-14T15:13:49.823383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:54.223752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0060000050000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3","length":"1"}]},{"pgid":"172.9","version":"156'2","reported_seq":71,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.841550+0000","last_change":"2024-04-14T15:13:45.211036+0000","last_active":"2024-04-14T15:13:58.841550+0000","last_peered":"2024-04-14T15:13:58.841550+0000","last_clean":"2024-04-14T15:13:58.841550+0000","last_became_active":"2024-04-14T15:13:36.842423+0000","last_became_peered":"2024-04-14T15:13:36.842423+0000","last_unstale":"2024-04-14T15:13:58.841550+0000","last_undegraded":"2024-04-14T15:13:58.841550+0000","last_fullsized":"2024-04-14T15:13:58.841550+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"156'2","last_scrub_stamp":"2024-04-14T15:13:45.210947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_clean_scrub_stamp":"2024-04-14T15:13:45.210947+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:15:05.299800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":2,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"186.1c","version":"0'0","reported_seq":32,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.832259+0000","last_change":"2024-04-14T15:13:52.851553+0000","last_active":"2024-04-14T15:13:58.832259+0000","last_peered":"2024-04-14T15:13:58.832259+0000","last_clean":"2024-04-14T15:13:58.832259+0000","last_became_active":"2024-04-14T15:13:50.894602+0000","last_became_peered":"2024-04-14T15:13:50.894602+0000","last_unstale":"2024-04-14T15:13:58.832259+0000","last_undegraded":"2024-04-14T15:13:58.832259+0000","last_fullsized":"2024-04-14T15:13:58.832259+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:49.823383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:49.823383+0000","last_clean_scrub_stamp":"2024-04-14T15:13:49.823383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:50.676724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001837359,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"3","length":"1"}]},{"pgid":"172.a","version":"156'8","reported_seq":72,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.838581+0000","last_change":"2024-04-14T15:13:36.844165+0000","last_active":"2024-04-14T15:13:58.838581+0000","last_peered":"2024-04-14T15:13:58.838581+0000","last_clean":"2024-04-14T15:13:58.838581+0000","last_became_active":"2024-04-14T15:13:36.843868+0000","last_became_peered":"2024-04-14T15:13:36.843868+0000","last_unstale":"2024-04-14T15:13:58.838581+0000","last_undegraded":"2024-04-14T15:13:58.838581+0000","last_fullsized":"2024-04-14T15:13:58.838581+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_clean_scrub_stamp":"2024-04-14T15:13:35.805229+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:57.340753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":8,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"186.1d","version":"0'0","reported_seq":32,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.835094+0000","last_change":"2024-04-14T15:13:52.850068+0000","last_active":"2024-04-14T15:13:58.835094+0000","last_peered":"2024-04-14T15:13:58.835094+0000","last_clean":"2024-04-14T15:13:58.835094+0000","last_became_active":"2024-04-14T15:13:50.878921+0000","last_became_peered":"2024-04-14T15:13:50.878921+0000","last_unstale":"2024-04-14T15:13:58.835094+0000","last_undegraded":"2024-04-14T15:13:58.835094+0000","last_fullsized":"2024-04-14T15:13:58.835094+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:49.823383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:49.823383+0000","last_clean_scrub_stamp":"2024-04-14T15:13:49.823383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:51.406778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074491200000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"3","length":"1"}]},{"pgid":"172.b","version":"156'11","reported_seq":77,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.834878+0000","last_change":"2024-04-14T15:13:36.835434+0000","last_active":"2024-04-14T15:13:58.834878+0000","last_peered":"2024-04-14T15:13:58.834878+0000","last_clean":"2024-04-14T15:13:58.834878+0000","last_became_active":"2024-04-14T15:13:36.835157+0000","last_became_peered":"2024-04-14T15:13:36.835157+0000","last_unstale":"2024-04-14T15:13:58.834878+0000","last_undegraded":"2024-04-14T15:13:58.834878+0000","last_fullsized":"2024-04-14T15:13:58.834878+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_clean_scrub_stamp":"2024-04-14T15:13:35.805229+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:44.065351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":11,"num_object_clones":8,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":11,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"186.1a","version":"0'0","reported_seq":32,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.848816+0000","last_change":"2024-04-14T15:13:52.851910+0000","last_active":"2024-04-14T15:13:58.848816+0000","last_peered":"2024-04-14T15:13:58.848816+0000","last_clean":"2024-04-14T15:13:58.848816+0000","last_became_active":"2024-04-14T15:13:50.883670+0000","last_became_peered":"2024-04-14T15:13:50.883670+0000","last_unstale":"2024-04-14T15:13:58.848816+0000","last_undegraded":"2024-04-14T15:13:58.848816+0000","last_fullsized":"2024-04-14T15:13:58.848816+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:49.823383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:49.823383+0000","last_clean_scrub_stamp":"2024-04-14T15:13:49.823383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:15:19.358852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064071899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3","length":"1"}]},{"pgid":"172.c","version":"156'2","reported_seq":61,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.830582+0000","last_change":"2024-04-14T15:13:36.842907+0000","last_active":"2024-04-14T15:13:58.830582+0000","last_peered":"2024-04-14T15:13:58.830582+0000","last_clean":"2024-04-14T15:13:58.830582+0000","last_became_active":"2024-04-14T15:13:36.842608+0000","last_became_peered":"2024-04-14T15:13:36.842608+0000","last_unstale":"2024-04-14T15:13:58.830582+0000","last_undegraded":"2024-04-14T15:13:58.830582+0000","last_fullsized":"2024-04-14T15:13:58.830582+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_clean_scrub_stamp":"2024-04-14T15:13:35.805229+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:36.549483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":2,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"186.1b","version":"0'0","reported_seq":32,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.842938+0000","last_change":"2024-04-14T15:13:52.845022+0000","last_active":"2024-04-14T15:13:58.842938+0000","last_peered":"2024-04-14T15:13:58.842938+0000","last_clean":"2024-04-14T15:13:58.842938+0000","last_became_active":"2024-04-14T15:13:50.885806+0000","last_became_peered":"2024-04-14T15:13:50.885806+0000","last_unstale":"2024-04-14T15:13:58.842938+0000","last_undegraded":"2024-04-14T15:13:58.842938+0000","last_fullsized":"2024-04-14T15:13:58.842938+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:49.823383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:49.823383+0000","last_clean_scrub_stamp":"2024-04-14T15:13:49.823383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:53.141646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0045537909999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"3","length":"1"}]},{"pgid":"172.d","version":"156'11","reported_seq":76,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.831407+0000","last_change":"2024-04-14T15:13:36.846714+0000","last_active":"2024-04-14T15:13:58.831407+0000","last_peered":"2024-04-14T15:13:58.831407+0000","last_clean":"2024-04-14T15:13:58.831407+0000","last_became_active":"2024-04-14T15:13:36.846467+0000","last_became_peered":"2024-04-14T15:13:36.846467+0000","last_unstale":"2024-04-14T15:13:58.831407+0000","last_undegraded":"2024-04-14T15:13:58.831407+0000","last_fullsized":"2024-04-14T15:13:58.831407+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_clean_scrub_stamp":"2024-04-14T15:13:35.805229+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:59.769915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":11,"num_object_clones":6,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":11,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"186.18","version":"0'0","reported_seq":32,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.836426+0000","last_change":"2024-04-14T15:13:52.852137+0000","last_active":"2024-04-14T15:13:58.836426+0000","last_peered":"2024-04-14T15:13:58.836426+0000","last_clean":"2024-04-14T15:13:58.836426+0000","last_became_active":"2024-04-14T15:13:50.876354+0000","last_became_peered":"2024-04-14T15:13:50.876354+0000","last_unstale":"2024-04-14T15:13:58.836426+0000","last_undegraded":"2024-04-14T15:13:58.836426+0000","last_fullsized":"2024-04-14T15:13:58.836426+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:49.823383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:49.823383+0000","last_clean_scrub_stamp":"2024-04-14T15:13:49.823383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:57.819858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0077672569999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"3","length":"1"}]},{"pgid":"172.e","version":"156'11","reported_seq":78,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.839794+0000","last_change":"2024-04-14T15:13:36.845458+0000","last_active":"2024-04-14T15:13:58.839794+0000","last_peered":"2024-04-14T15:13:58.839794+0000","last_clean":"2024-04-14T15:13:58.839794+0000","last_became_active":"2024-04-14T15:13:36.845186+0000","last_became_peered":"2024-04-14T15:13:36.845186+0000","last_unstale":"2024-04-14T15:13:58.839794+0000","last_undegraded":"2024-04-14T15:13:58.839794+0000","last_fullsized":"2024-04-14T15:13:58.839794+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_clean_scrub_stamp":"2024-04-14T15:13:35.805229+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:15:05.200010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":11,"num_object_clones":6,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":11,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"186.19","version":"0'0","reported_seq":32,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.835079+0000","last_change":"2024-04-14T15:13:52.842621+0000","last_active":"2024-04-14T15:13:58.835079+0000","last_peered":"2024-04-14T15:13:58.835079+0000","last_clean":"2024-04-14T15:13:58.835079+0000","last_became_active":"2024-04-14T15:13:50.881862+0000","last_became_peered":"2024-04-14T15:13:50.881862+0000","last_unstale":"2024-04-14T15:13:58.835079+0000","last_undegraded":"2024-04-14T15:13:58.835079+0000","last_fullsized":"2024-04-14T15:13:58.835079+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:49.823383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:49.823383+0000","last_clean_scrub_stamp":"2024-04-14T15:13:49.823383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:15:08.516309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001090639,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"3","length":"1"}]},{"pgid":"172.f","version":"156'8","reported_seq":71,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.841170+0000","last_change":"2024-04-14T15:13:36.839072+0000","last_active":"2024-04-14T15:13:58.841170+0000","last_peered":"2024-04-14T15:13:58.841170+0000","last_clean":"2024-04-14T15:13:58.841170+0000","last_became_active":"2024-04-14T15:13:36.838790+0000","last_became_peered":"2024-04-14T15:13:36.838790+0000","last_unstale":"2024-04-14T15:13:58.841170+0000","last_undegraded":"2024-04-14T15:13:58.841170+0000","last_fullsized":"2024-04-14T15:13:58.841170+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.805229+0000","last_clean_scrub_stamp":"2024-04-14T15:13:35.805229+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:47.707123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":8,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"110.11","version":"153'18","reported_seq":151,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.836535+0000","last_change":"2024-04-14T15:13:31.858972+0000","last_active":"2024-04-14T15:13:58.836535+0000","last_peered":"2024-04-14T15:13:58.836535+0000","last_clean":"2024-04-14T15:13:58.836535+0000","last_became_active":"2024-04-14T15:13:09.936619+0000","last_became_peered":"2024-04-14T15:13:09.936619+0000","last_unstale":"2024-04-14T15:13:58.836535+0000","last_undegraded":"2024-04-14T15:13:58.836535+0000","last_fullsized":"2024-04-14T15:13:58.836535+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:13:08.799672+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:21.376976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.018553739,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"110.10","version":"153'81","reported_seq":192,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.837606+0000","last_change":"2024-04-14T15:13:31.846027+0000","last_active":"2024-04-14T15:13:58.837606+0000","last_peered":"2024-04-14T15:13:58.837606+0000","last_clean":"2024-04-14T15:13:58.837606+0000","last_became_active":"2024-04-14T15:13:09.928915+0000","last_became_peered":"2024-04-14T15:13:09.928915+0000","last_unstale":"2024-04-14T15:13:58.837606+0000","last_undegraded":"2024-04-14T15:13:58.837606+0000","last_fullsized":"2024-04-14T15:13:58.837606+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:13:08.799672+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:19.024127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.021680241999999999,"stat_sum":{"num_bytes":468,"num_objects":45,"num_object_clones":36,"num_object_copies":135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":45,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"110.13","version":"153'18","reported_seq":151,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.839279+0000","last_change":"2024-04-14T15:13:31.855977+0000","last_active":"2024-04-14T15:13:58.839279+0000","last_peered":"2024-04-14T15:13:58.839279+0000","last_clean":"2024-04-14T15:13:58.839279+0000","last_became_active":"2024-04-14T15:13:09.918152+0000","last_became_peered":"2024-04-14T15:13:09.918152+0000","last_unstale":"2024-04-14T15:13:58.839279+0000","last_undegraded":"2024-04-14T15:13:58.839279+0000","last_fullsized":"2024-04-14T15:13:58.839279+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:13:08.799672+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:11.812186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0051087459999999999,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"110.12","version":"153'32","reported_seq":164,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.836763+0000","last_change":"2024-04-14T15:13:31.908947+0000","last_active":"2024-04-14T15:13:58.836763+0000","last_peered":"2024-04-14T15:13:58.836763+0000","last_clean":"2024-04-14T15:13:58.836763+0000","last_became_active":"2024-04-14T15:13:09.939566+0000","last_became_peered":"2024-04-14T15:13:09.939566+0000","last_unstale":"2024-04-14T15:13:58.836763+0000","last_undegraded":"2024-04-14T15:13:58.836763+0000","last_fullsized":"2024-04-14T15:13:58.836763+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:13:08.799672+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:19.900823+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.045537372999999999,"stat_sum":{"num_bytes":161,"num_objects":16,"num_object_clones":12,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"110.15","version":"153'18","reported_seq":152,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.848613+0000","last_change":"2024-04-14T15:13:31.871256+0000","last_active":"2024-04-14T15:13:58.848613+0000","last_peered":"2024-04-14T15:13:58.848613+0000","last_clean":"2024-04-14T15:13:58.848613+0000","last_became_active":"2024-04-14T15:13:09.960121+0000","last_became_peered":"2024-04-14T15:13:09.960121+0000","last_unstale":"2024-04-14T15:13:58.848613+0000","last_undegraded":"2024-04-14T15:13:58.848613+0000","last_fullsized":"2024-04-14T15:13:58.848613+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:13:08.799672+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:11.741418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015032871999999999,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"110.14","version":"153'45","reported_seq":176,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.846690+0000","last_change":"2024-04-14T15:13:31.856253+0000","last_active":"2024-04-14T15:13:58.846690+0000","last_peered":"2024-04-14T15:13:58.846690+0000","last_clean":"2024-04-14T15:13:58.846690+0000","last_became_active":"2024-04-14T15:13:09.899561+0000","last_became_peered":"2024-04-14T15:13:09.899561+0000","last_unstale":"2024-04-14T15:13:58.846690+0000","last_undegraded":"2024-04-14T15:13:58.846690+0000","last_fullsized":"2024-04-14T15:13:58.846690+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:13:08.799672+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:26.596004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015294199,"stat_sum":{"num_bytes":260,"num_objects":25,"num_object_clones":20,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"110.17","version":"131'1","reported_seq":136,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.852117+0000","last_change":"2024-04-14T15:13:31.864280+0000","last_active":"2024-04-14T15:13:58.852117+0000","last_peered":"2024-04-14T15:13:58.852117+0000","last_clean":"2024-04-14T15:13:58.852117+0000","last_became_active":"2024-04-14T15:13:09.898521+0000","last_became_peered":"2024-04-14T15:13:09.898521+0000","last_unstale":"2024-04-14T15:13:58.852117+0000","last_undegraded":"2024-04-14T15:13:58.852117+0000","last_fullsized":"2024-04-14T15:13:58.852117+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:13:08.799672+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:34.294592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016996144000000001,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"110.16","version":"153'18","reported_seq":152,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.839902+0000","last_change":"2024-04-14T15:13:31.831733+0000","last_active":"2024-04-14T15:13:58.839902+0000","last_peered":"2024-04-14T15:13:58.839902+0000","last_clean":"2024-04-14T15:13:58.839902+0000","last_became_active":"2024-04-14T15:13:09.952918+0000","last_became_peered":"2024-04-14T15:13:09.952918+0000","last_unstale":"2024-04-14T15:13:58.839902+0000","last_undegraded":"2024-04-14T15:13:58.839902+0000","last_fullsized":"2024-04-14T15:13:58.839902+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:13:08.799672+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:30.064474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0067552690000000004,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"110.19","version":"153'45","reported_seq":181,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.831277+0000","last_change":"2024-04-14T15:13:31.820264+0000","last_active":"2024-04-14T15:13:58.831277+0000","last_peered":"2024-04-14T15:13:58.831277+0000","last_clean":"2024-04-14T15:13:58.831277+0000","last_became_active":"2024-04-14T15:13:09.998139+0000","last_became_peered":"2024-04-14T15:13:09.998139+0000","last_unstale":"2024-04-14T15:13:58.831277+0000","last_undegraded":"2024-04-14T15:13:58.831277+0000","last_fullsized":"2024-04-14T15:13:58.831277+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:13:08.799672+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:15.557421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022159749999999998,"stat_sum":{"num_bytes":260,"num_objects":25,"num_object_clones":20,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"110.18","version":"153'45","reported_seq":171,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.827834+0000","last_change":"2024-04-14T15:13:31.819345+0000","last_active":"2024-04-14T15:13:58.827834+0000","last_peered":"2024-04-14T15:13:58.827834+0000","last_clean":"2024-04-14T15:13:58.827834+0000","last_became_active":"2024-04-14T15:13:09.998803+0000","last_became_peered":"2024-04-14T15:13:09.998803+0000","last_unstale":"2024-04-14T15:13:58.827834+0000","last_undegraded":"2024-04-14T15:13:58.827834+0000","last_fullsized":"2024-04-14T15:13:58.827834+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:13:08.799672+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:16.276049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025881530000000002,"stat_sum":{"num_bytes":260,"num_objects":25,"num_object_clones":20,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"110.1b","version":"153'45","reported_seq":177,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.839185+0000","last_change":"2024-04-14T15:13:31.855228+0000","last_active":"2024-04-14T15:13:58.839185+0000","last_peered":"2024-04-14T15:13:58.839185+0000","last_clean":"2024-04-14T15:13:58.839185+0000","last_became_active":"2024-04-14T15:13:09.946919+0000","last_became_peered":"2024-04-14T15:13:09.946919+0000","last_unstale":"2024-04-14T15:13:58.839185+0000","last_undegraded":"2024-04-14T15:13:58.839185+0000","last_fullsized":"2024-04-14T15:13:58.839185+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:13:08.799672+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:19.136903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0043648070000000001,"stat_sum":{"num_bytes":260,"num_objects":25,"num_object_clones":20,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"110.1a","version":"153'18","reported_seq":150,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.842726+0000","last_change":"2024-04-14T15:13:31.828510+0000","last_active":"2024-04-14T15:13:58.842726+0000","last_peered":"2024-04-14T15:13:58.842726+0000","last_clean":"2024-04-14T15:13:58.842726+0000","last_became_active":"2024-04-14T15:13:09.960319+0000","last_became_peered":"2024-04-14T15:13:09.960319+0000","last_unstale":"2024-04-14T15:13:58.842726+0000","last_undegraded":"2024-04-14T15:13:58.842726+0000","last_fullsized":"2024-04-14T15:13:58.842726+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:13:08.799672+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:33.254337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0064645620000000001,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"110.1d","version":"153'45","reported_seq":177,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.834317+0000","last_change":"2024-04-14T15:13:31.857816+0000","last_active":"2024-04-14T15:13:58.834317+0000","last_peered":"2024-04-14T15:13:58.834317+0000","last_clean":"2024-04-14T15:13:58.834317+0000","last_became_active":"2024-04-14T15:13:09.868082+0000","last_became_peered":"2024-04-14T15:13:09.868082+0000","last_unstale":"2024-04-14T15:13:58.834317+0000","last_undegraded":"2024-04-14T15:13:58.834317+0000","last_fullsized":"2024-04-14T15:13:58.834317+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:13:08.799672+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:23.916678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012743758000000001,"stat_sum":{"num_bytes":260,"num_objects":25,"num_object_clones":20,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"110.1c","version":"153'41","reported_seq":167,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.832351+0000","last_change":"2024-04-14T15:13:31.906550+0000","last_active":"2024-04-14T15:13:58.832351+0000","last_peered":"2024-04-14T15:13:58.832351+0000","last_clean":"2024-04-14T15:13:58.832351+0000","last_became_active":"2024-04-14T15:13:09.954091+0000","last_became_peered":"2024-04-14T15:13:09.954091+0000","last_unstale":"2024-04-14T15:13:58.832351+0000","last_undegraded":"2024-04-14T15:13:58.832351+0000","last_fullsized":"2024-04-14T15:13:58.832351+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:13:08.799672+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:34.595208+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.018861645999999999,"stat_sum":{"num_bytes":213,"num_objects":21,"num_object_clones":16,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"110.1f","version":"153'9","reported_seq":142,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.830959+0000","last_change":"2024-04-14T15:13:31.836380+0000","last_active":"2024-04-14T15:13:58.830959+0000","last_peered":"2024-04-14T15:13:58.830959+0000","last_clean":"2024-04-14T15:13:58.830959+0000","last_became_active":"2024-04-14T15:13:09.999629+0000","last_became_peered":"2024-04-14T15:13:09.999629+0000","last_unstale":"2024-04-14T15:13:58.830959+0000","last_undegraded":"2024-04-14T15:13:58.830959+0000","last_fullsized":"2024-04-14T15:13:58.830959+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:13:08.799672+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:38.588596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010555880000000001,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"110.1e","version":"153'14","reported_seq":148,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.848523+0000","last_change":"2024-04-14T15:13:31.892343+0000","last_active":"2024-04-14T15:13:58.848523+0000","last_peered":"2024-04-14T15:13:58.848523+0000","last_clean":"2024-04-14T15:13:58.848523+0000","last_became_active":"2024-04-14T15:13:09.955932+0000","last_became_peered":"2024-04-14T15:13:09.955932+0000","last_unstale":"2024-04-14T15:13:58.848523+0000","last_undegraded":"2024-04-14T15:13:58.848523+0000","last_fullsized":"2024-04-14T15:13:58.848523+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:13:08.799672+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:30.227279+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.044420435000000001,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"110.1","version":"153'9","reported_seq":142,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.833428+0000","last_change":"2024-04-14T15:13:31.840886+0000","last_active":"2024-04-14T15:13:58.833428+0000","last_peered":"2024-04-14T15:13:58.833428+0000","last_clean":"2024-04-14T15:13:58.833428+0000","last_became_active":"2024-04-14T15:13:09.929562+0000","last_became_peered":"2024-04-14T15:13:09.929562+0000","last_unstale":"2024-04-14T15:13:58.833428+0000","last_undegraded":"2024-04-14T15:13:58.833428+0000","last_fullsized":"2024-04-14T15:13:58.833428+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:13:08.799672+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:10.555817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011417548,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"110.0","version":"153'45","reported_seq":171,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.837707+0000","last_change":"2024-04-14T15:13:31.846906+0000","last_active":"2024-04-14T15:13:58.837707+0000","last_peered":"2024-04-14T15:13:58.837707+0000","last_clean":"2024-04-14T15:13:58.837707+0000","last_became_active":"2024-04-14T15:13:09.856407+0000","last_became_peered":"2024-04-14T15:13:09.856407+0000","last_unstale":"2024-04-14T15:13:58.837707+0000","last_undegraded":"2024-04-14T15:13:58.837707+0000","last_fullsized":"2024-04-14T15:13:58.837707+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:13:08.799672+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:29.923941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.022357385,"stat_sum":{"num_bytes":260,"num_objects":25,"num_object_clones":20,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"110.3","version":"153'27","reported_seq":152,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.835079+0000","last_change":"2024-04-14T15:13:31.830947+0000","last_active":"2024-04-14T15:13:58.835079+0000","last_peered":"2024-04-14T15:13:58.835079+0000","last_clean":"2024-04-14T15:13:58.835079+0000","last_became_active":"2024-04-14T15:13:09.997208+0000","last_became_peered":"2024-04-14T15:13:09.997208+0000","last_unstale":"2024-04-14T15:13:58.835079+0000","last_undegraded":"2024-04-14T15:13:58.835079+0000","last_fullsized":"2024-04-14T15:13:58.835079+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:13:08.799672+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:14.431524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0084448869999999999,"stat_sum":{"num_bytes":156,"num_objects":15,"num_object_clones":12,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"110.2","version":"153'27","reported_seq":158,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.831981+0000","last_change":"2024-04-14T15:13:31.822872+0000","last_active":"2024-04-14T15:13:58.831981+0000","last_peered":"2024-04-14T15:13:58.831981+0000","last_clean":"2024-04-14T15:13:58.831981+0000","last_became_active":"2024-04-14T15:13:09.957187+0000","last_became_peered":"2024-04-14T15:13:09.957187+0000","last_unstale":"2024-04-14T15:13:58.831981+0000","last_undegraded":"2024-04-14T15:13:58.831981+0000","last_fullsized":"2024-04-14T15:13:58.831981+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:13:08.799672+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:21.520626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016792300000000001,"stat_sum":{"num_bytes":156,"num_objects":15,"num_object_clones":12,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"110.5","version":"153'33","reported_seq":157,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.843551+0000","last_change":"2024-04-14T15:13:33.868919+0000","last_active":"2024-04-14T15:13:58.843551+0000","last_peered":"2024-04-14T15:13:58.843551+0000","last_clean":"2024-04-14T15:13:58.843551+0000","last_became_active":"2024-04-14T15:13:09.853689+0000","last_became_peered":"2024-04-14T15:13:09.853689+0000","last_unstale":"2024-04-14T15:13:58.843551+0000","last_undegraded":"2024-04-14T15:13:58.843551+0000","last_fullsized":"2024-04-14T15:13:58.843551+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:13:08.799672+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:19.853438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.035385613000000003,"stat_sum":{"num_bytes":165,"num_objects":17,"num_object_clones":12,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":18,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"110.4","version":"153'27","reported_seq":156,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.846712+0000","last_change":"2024-04-14T15:13:31.856629+0000","last_active":"2024-04-14T15:13:58.846712+0000","last_peered":"2024-04-14T15:13:58.846712+0000","last_clean":"2024-04-14T15:13:58.846712+0000","last_became_active":"2024-04-14T15:13:09.958847+0000","last_became_peered":"2024-04-14T15:13:09.958847+0000","last_unstale":"2024-04-14T15:13:58.846712+0000","last_undegraded":"2024-04-14T15:13:58.846712+0000","last_fullsized":"2024-04-14T15:13:58.846712+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:13:08.799672+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:25.256022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015455567,"stat_sum":{"num_bytes":156,"num_objects":15,"num_object_clones":12,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"110.7","version":"153'9","reported_seq":141,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.841511+0000","last_change":"2024-04-14T15:13:31.858935+0000","last_active":"2024-04-14T15:13:58.841511+0000","last_peered":"2024-04-14T15:13:58.841511+0000","last_clean":"2024-04-14T15:13:58.841511+0000","last_became_active":"2024-04-14T15:13:09.936631+0000","last_became_peered":"2024-04-14T15:13:09.936631+0000","last_unstale":"2024-04-14T15:13:58.841511+0000","last_undegraded":"2024-04-14T15:13:58.841511+0000","last_fullsized":"2024-04-14T15:13:58.841511+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:13:08.799672+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:29.071134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016100272999999998,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"110.6","version":"153'36","reported_seq":164,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.834603+0000","last_change":"2024-04-14T15:13:31.857626+0000","last_active":"2024-04-14T15:13:58.834603+0000","last_peered":"2024-04-14T15:13:58.834603+0000","last_clean":"2024-04-14T15:13:58.834603+0000","last_became_active":"2024-04-14T15:13:09.930125+0000","last_became_peered":"2024-04-14T15:13:09.930125+0000","last_unstale":"2024-04-14T15:13:58.834603+0000","last_undegraded":"2024-04-14T15:13:58.834603+0000","last_fullsized":"2024-04-14T15:13:58.834603+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:13:08.799672+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:38.446592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015594758,"stat_sum":{"num_bytes":208,"num_objects":20,"num_object_clones":16,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"110.9","version":"153'9","reported_seq":141,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.831172+0000","last_change":"2024-04-14T15:13:31.819949+0000","last_active":"2024-04-14T15:13:58.831172+0000","last_peered":"2024-04-14T15:13:58.831172+0000","last_clean":"2024-04-14T15:13:58.831172+0000","last_became_active":"2024-04-14T15:13:09.957324+0000","last_became_peered":"2024-04-14T15:13:09.957324+0000","last_unstale":"2024-04-14T15:13:58.831172+0000","last_undegraded":"2024-04-14T15:13:58.831172+0000","last_fullsized":"2024-04-14T15:13:58.831172+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:13:08.799672+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:13.998154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021015529999999999,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"110.8","version":"153'9","reported_seq":140,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.827991+0000","last_change":"2024-04-14T15:13:31.832777+0000","last_active":"2024-04-14T15:13:58.827991+0000","last_peered":"2024-04-14T15:13:58.827991+0000","last_clean":"2024-04-14T15:13:58.827991+0000","last_became_active":"2024-04-14T15:13:09.952046+0000","last_became_peered":"2024-04-14T15:13:09.952046+0000","last_unstale":"2024-04-14T15:13:58.827991+0000","last_undegraded":"2024-04-14T15:13:58.827991+0000","last_fullsized":"2024-04-14T15:13:58.827991+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:13:08.799672+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:15.142168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.005243274,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"110.b","version":"153'36","reported_seq":167,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.841331+0000","last_change":"2024-04-14T15:13:31.856715+0000","last_active":"2024-04-14T15:13:58.841331+0000","last_peered":"2024-04-14T15:13:58.841331+0000","last_clean":"2024-04-14T15:13:58.841331+0000","last_became_active":"2024-04-14T15:13:09.964629+0000","last_became_peered":"2024-04-14T15:13:09.964629+0000","last_unstale":"2024-04-14T15:13:58.841331+0000","last_undegraded":"2024-04-14T15:13:58.841331+0000","last_fullsized":"2024-04-14T15:13:58.841331+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:13:08.799672+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:26.452655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0053331009999999998,"stat_sum":{"num_bytes":208,"num_objects":20,"num_object_clones":16,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"110.a","version":"153'36","reported_seq":164,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.842837+0000","last_change":"2024-04-14T15:13:31.828989+0000","last_active":"2024-04-14T15:13:58.842837+0000","last_peered":"2024-04-14T15:13:58.842837+0000","last_clean":"2024-04-14T15:13:58.842837+0000","last_became_active":"2024-04-14T15:13:09.909752+0000","last_became_peered":"2024-04-14T15:13:09.909752+0000","last_unstale":"2024-04-14T15:13:58.842837+0000","last_undegraded":"2024-04-14T15:13:58.842837+0000","last_fullsized":"2024-04-14T15:13:58.842837+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:13:08.799672+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:20.474260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.006744145,"stat_sum":{"num_bytes":208,"num_objects":20,"num_object_clones":16,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"110.d","version":"153'45","reported_seq":174,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.831628+0000","last_change":"2024-04-14T15:13:31.892572+0000","last_active":"2024-04-14T15:13:58.831628+0000","last_peered":"2024-04-14T15:13:58.831628+0000","last_clean":"2024-04-14T15:13:58.831628+0000","last_became_active":"2024-04-14T15:13:09.944223+0000","last_became_peered":"2024-04-14T15:13:09.944223+0000","last_unstale":"2024-04-14T15:13:58.831628+0000","last_undegraded":"2024-04-14T15:13:58.831628+0000","last_fullsized":"2024-04-14T15:13:58.831628+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:13:08.799672+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:17.611091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.020921474999999998,"stat_sum":{"num_bytes":260,"num_objects":25,"num_object_clones":20,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"110.c","version":"153'9","reported_seq":141,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.832446+0000","last_change":"2024-04-14T15:13:31.906885+0000","last_active":"2024-04-14T15:13:58.832446+0000","last_peered":"2024-04-14T15:13:58.832446+0000","last_clean":"2024-04-14T15:13:58.832446+0000","last_became_active":"2024-04-14T15:13:09.984609+0000","last_became_peered":"2024-04-14T15:13:09.984609+0000","last_unstale":"2024-04-14T15:13:58.832446+0000","last_undegraded":"2024-04-14T15:13:58.832446+0000","last_fullsized":"2024-04-14T15:13:58.832446+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:13:08.799672+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:20.285517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01712311,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"110.f","version":"153'27","reported_seq":159,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.832705+0000","last_change":"2024-04-14T15:13:31.898618+0000","last_active":"2024-04-14T15:13:58.832705+0000","last_peered":"2024-04-14T15:13:58.832705+0000","last_clean":"2024-04-14T15:13:58.832705+0000","last_became_active":"2024-04-14T15:13:09.964057+0000","last_became_peered":"2024-04-14T15:13:09.964057+0000","last_unstale":"2024-04-14T15:13:58.832705+0000","last_undegraded":"2024-04-14T15:13:58.832705+0000","last_fullsized":"2024-04-14T15:13:58.832705+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:13:08.799672+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:37.178359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011988861999999999,"stat_sum":{"num_bytes":156,"num_objects":15,"num_object_clones":12,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"110.e","version":"153'45","reported_seq":174,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.834508+0000","last_change":"2024-04-14T15:13:31.853707+0000","last_active":"2024-04-14T15:13:58.834508+0000","last_peered":"2024-04-14T15:13:58.834508+0000","last_clean":"2024-04-14T15:13:58.834508+0000","last_became_active":"2024-04-14T15:13:09.984980+0000","last_became_peered":"2024-04-14T15:13:09.984980+0000","last_unstale":"2024-04-14T15:13:58.834508+0000","last_undegraded":"2024-04-14T15:13:58.834508+0000","last_fullsized":"2024-04-14T15:13:58.834508+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:13:08.799672+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:16.413851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012145575,"stat_sum":{"num_bytes":260,"num_objects":25,"num_object_clones":20,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.1","version":"109'5","reported_seq":149,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.838333+0000","last_change":"2024-04-14T15:12:56.415926+0000","last_active":"2024-04-14T15:13:58.838333+0000","last_peered":"2024-04-14T15:13:58.838333+0000","last_clean":"2024-04-14T15:13:58.838333+0000","last_became_active":"2024-04-14T15:12:56.404932+0000","last_became_peered":"2024-04-14T15:12:56.404932+0000","last_unstale":"2024-04-14T15:13:58.838333+0000","last_undegraded":"2024-04-14T15:13:58.838333+0000","last_fullsized":"2024-04-14T15:13:58.838333+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:02.595228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":137,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.842539+0000","last_change":"2024-04-14T15:12:56.247166+0000","last_active":"2024-04-14T15:13:58.842539+0000","last_peered":"2024-04-14T15:13:58.842539+0000","last_clean":"2024-04-14T15:13:58.842539+0000","last_became_active":"2024-04-14T15:12:56.246864+0000","last_became_peered":"2024-04-14T15:12:56.246864+0000","last_unstale":"2024-04-14T15:13:58.842539+0000","last_undegraded":"2024-04-14T15:13:58.842539+0000","last_fullsized":"2024-04-14T15:13:58.842539+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:20.494014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"85'9","reported_seq":293,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.833952+0000","last_change":"2024-04-14T15:12:37.019655+0000","last_active":"2024-04-14T15:13:58.833952+0000","last_peered":"2024-04-14T15:13:58.833952+0000","last_clean":"2024-04-14T15:13:58.833952+0000","last_became_active":"2024-04-14T15:09:47.045441+0000","last_became_peered":"2024-04-14T15:09:47.045441+0000","last_unstale":"2024-04-14T15:13:58.833952+0000","last_undegraded":"2024-04-14T15:13:58.833952+0000","last_fullsized":"2024-04-14T15:13:58.833952+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9","last_scrub_stamp":"2024-04-14T15:12:37.019536+0000","last_deep_scrub":"85'9","last_deep_scrub_stamp":"2024-04-14T15:10:51.950580+0000","last_clean_scrub_stamp":"2024-04-14T15:12:37.019536+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:00.217464+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"72'15","reported_seq":261,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.828451+0000","last_change":"2024-04-14T15:13:21.857209+0000","last_active":"2024-04-14T15:13:58.828451+0000","last_peered":"2024-04-14T15:13:58.828451+0000","last_clean":"2024-04-14T15:13:58.828451+0000","last_became_active":"2024-04-14T15:09:45.180160+0000","last_became_peered":"2024-04-14T15:09:45.180160+0000","last_unstale":"2024-04-14T15:13:58.828451+0000","last_undegraded":"2024-04-14T15:13:58.828451+0000","last_fullsized":"2024-04-14T15:13:58.828451+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-04-14T15:13:21.857048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:13:21.857048+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:29.401213+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":240,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.838578+0000","last_change":"2024-04-14T15:13:17.346045+0000","last_active":"2024-04-14T15:13:58.838578+0000","last_peered":"2024-04-14T15:13:58.838578+0000","last_clean":"2024-04-14T15:13:58.838578+0000","last_became_active":"2024-04-14T15:09:43.021460+0000","last_became_peered":"2024-04-14T15:09:43.021460+0000","last_unstale":"2024-04-14T15:13:58.838578+0000","last_undegraded":"2024-04-14T15:13:58.838578+0000","last_fullsized":"2024-04-14T15:13:58.838578+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:17.345909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:13:17.345909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:33.916250+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":228,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.829252+0000","last_change":"2024-04-14T15:13:36.719842+0000","last_active":"2024-04-14T15:13:58.829252+0000","last_peered":"2024-04-14T15:13:58.829252+0000","last_clean":"2024-04-14T15:13:58.829252+0000","last_became_active":"2024-04-14T15:09:49.128070+0000","last_became_peered":"2024-04-14T15:09:49.128070+0000","last_unstale":"2024-04-14T15:13:58.829252+0000","last_undegraded":"2024-04-14T15:13:58.829252+0000","last_fullsized":"2024-04-14T15:13:58.829252+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:36.719765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:13:36.719765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:40.986313+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1c","version":"108'8","reported_seq":146,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.833285+0000","last_change":"2024-04-14T15:12:56.364615+0000","last_active":"2024-04-14T15:13:58.833285+0000","last_peered":"2024-04-14T15:13:58.833285+0000","last_clean":"2024-04-14T15:13:58.833285+0000","last_became_active":"2024-04-14T15:12:56.364330+0000","last_became_peered":"2024-04-14T15:12:56.364330+0000","last_unstale":"2024-04-14T15:13:58.833285+0000","last_undegraded":"2024-04-14T15:13:58.833285+0000","last_fullsized":"2024-04-14T15:13:58.833285+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:19.300825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":144,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.829050+0000","last_change":"2024-04-14T15:12:56.325685+0000","last_active":"2024-04-14T15:13:58.829050+0000","last_peered":"2024-04-14T15:13:58.829050+0000","last_clean":"2024-04-14T15:13:58.829050+0000","last_became_active":"2024-04-14T15:12:56.319358+0000","last_became_peered":"2024-04-14T15:12:56.319358+0000","last_unstale":"2024-04-14T15:13:58.829050+0000","last_undegraded":"2024-04-14T15:13:58.829050+0000","last_fullsized":"2024-04-14T15:13:58.829050+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:11.858748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":137,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.835717+0000","last_change":"2024-04-14T15:12:56.363693+0000","last_active":"2024-04-14T15:13:58.835717+0000","last_peered":"2024-04-14T15:13:58.835717+0000","last_clean":"2024-04-14T15:13:58.835717+0000","last_became_active":"2024-04-14T15:12:56.355348+0000","last_became_peered":"2024-04-14T15:12:56.355348+0000","last_unstale":"2024-04-14T15:13:58.835717+0000","last_undegraded":"2024-04-14T15:13:58.835717+0000","last_fullsized":"2024-04-14T15:13:58.835717+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:05.657487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"72'9","reported_seq":252,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.853071+0000","last_change":"2024-04-14T15:12:28.969215+0000","last_active":"2024-04-14T15:13:58.853071+0000","last_peered":"2024-04-14T15:13:58.853071+0000","last_clean":"2024-04-14T15:13:58.853071+0000","last_became_active":"2024-04-14T15:09:45.037179+0000","last_became_peered":"2024-04-14T15:09:45.037179+0000","last_unstale":"2024-04-14T15:13:58.853071+0000","last_undegraded":"2024-04-14T15:13:58.853071+0000","last_fullsized":"2024-04-14T15:13:58.853071+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-04-14T15:12:28.969074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:12:28.969074+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":240,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.832384+0000","last_change":"2024-04-14T15:12:26.367861+0000","last_active":"2024-04-14T15:13:58.832384+0000","last_peered":"2024-04-14T15:13:58.832384+0000","last_clean":"2024-04-14T15:13:58.832384+0000","last_became_active":"2024-04-14T15:09:43.020044+0000","last_became_peered":"2024-04-14T15:09:43.020044+0000","last_unstale":"2024-04-14T15:13:58.832384+0000","last_undegraded":"2024-04-14T15:13:58.832384+0000","last_fullsized":"2024-04-14T15:13:58.832384+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:26.367741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:12:26.367741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":232,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.837264+0000","last_change":"2024-04-14T15:12:38.441705+0000","last_active":"2024-04-14T15:13:58.837264+0000","last_peered":"2024-04-14T15:13:58.837264+0000","last_clean":"2024-04-14T15:13:58.837264+0000","last_became_active":"2024-04-14T15:09:47.046153+0000","last_became_peered":"2024-04-14T15:09:47.046153+0000","last_unstale":"2024-04-14T15:13:58.837264+0000","last_undegraded":"2024-04-14T15:13:58.837264+0000","last_fullsized":"2024-04-14T15:13:58.837264+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:38.441583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:12:38.441583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":228,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.845925+0000","last_change":"2024-04-14T15:12:25.021726+0000","last_active":"2024-04-14T15:13:58.845925+0000","last_peered":"2024-04-14T15:13:58.845925+0000","last_clean":"2024-04-14T15:13:58.845925+0000","last_became_active":"2024-04-14T15:09:49.061495+0000","last_became_peered":"2024-04-14T15:09:49.061495+0000","last_unstale":"2024-04-14T15:13:58.845925+0000","last_undegraded":"2024-04-14T15:13:58.845925+0000","last_fullsized":"2024-04-14T15:13:58.845925+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:25.021630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:12:25.021630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":137,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.842449+0000","last_change":"2024-04-14T15:12:56.363352+0000","last_active":"2024-04-14T15:13:58.842449+0000","last_peered":"2024-04-14T15:13:58.842449+0000","last_clean":"2024-04-14T15:13:58.842449+0000","last_became_active":"2024-04-14T15:12:56.359459+0000","last_became_peered":"2024-04-14T15:12:56.359459+0000","last_unstale":"2024-04-14T15:13:58.842449+0000","last_undegraded":"2024-04-14T15:13:58.842449+0000","last_fullsized":"2024-04-14T15:13:58.842449+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:24.879243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.3","version":"109'5","reported_seq":149,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.856877+0000","last_change":"2024-04-14T15:12:56.531318+0000","last_active":"2024-04-14T15:13:58.856877+0000","last_peered":"2024-04-14T15:13:58.856877+0000","last_clean":"2024-04-14T15:13:58.856877+0000","last_became_active":"2024-04-14T15:12:56.529451+0000","last_became_peered":"2024-04-14T15:12:56.529451+0000","last_unstale":"2024-04-14T15:13:58.856877+0000","last_undegraded":"2024-04-14T15:13:58.856877+0000","last_fullsized":"2024-04-14T15:13:58.856877+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":137,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.834816+0000","last_change":"2024-04-14T15:12:56.246956+0000","last_active":"2024-04-14T15:13:58.834816+0000","last_peered":"2024-04-14T15:13:58.834816+0000","last_clean":"2024-04-14T15:13:58.834816+0000","last_became_active":"2024-04-14T15:12:56.246062+0000","last_became_peered":"2024-04-14T15:12:56.246062+0000","last_unstale":"2024-04-14T15:13:58.834816+0000","last_undegraded":"2024-04-14T15:13:58.834816+0000","last_fullsized":"2024-04-14T15:13:58.834816+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:04.773310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"72'19","reported_seq":267,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.843307+0000","last_change":"2024-04-14T15:12:38.142096+0000","last_active":"2024-04-14T15:13:58.843307+0000","last_peered":"2024-04-14T15:13:58.843307+0000","last_clean":"2024-04-14T15:13:58.843307+0000","last_became_active":"2024-04-14T15:09:45.178782+0000","last_became_peered":"2024-04-14T15:09:45.178782+0000","last_unstale":"2024-04-14T15:13:58.843307+0000","last_undegraded":"2024-04-14T15:13:58.843307+0000","last_fullsized":"2024-04-14T15:13:58.843307+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'19","last_scrub_stamp":"2024-04-14T15:12:38.141982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:12:38.141982+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:02.486682+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":248,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.833787+0000","last_change":"2024-04-14T15:13:34.342920+0000","last_active":"2024-04-14T15:13:58.833787+0000","last_peered":"2024-04-14T15:13:58.833787+0000","last_clean":"2024-04-14T15:13:58.833787+0000","last_became_active":"2024-04-14T15:09:43.020268+0000","last_became_peered":"2024-04-14T15:09:43.020268+0000","last_unstale":"2024-04-14T15:13:58.833787+0000","last_undegraded":"2024-04-14T15:13:58.833787+0000","last_fullsized":"2024-04-14T15:13:58.833787+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:34.342793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:34.342793+0000","last_clean_scrub_stamp":"2024-04-14T15:13:34.342793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:42.349015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":232,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.841420+0000","last_change":"2024-04-14T15:13:16.363599+0000","last_active":"2024-04-14T15:13:58.841420+0000","last_peered":"2024-04-14T15:13:58.841420+0000","last_clean":"2024-04-14T15:13:58.841420+0000","last_became_active":"2024-04-14T15:09:47.045542+0000","last_became_peered":"2024-04-14T15:09:47.045542+0000","last_unstale":"2024-04-14T15:13:58.841420+0000","last_undegraded":"2024-04-14T15:13:58.841420+0000","last_fullsized":"2024-04-14T15:13:58.841420+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:16.363478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:13:16.363478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:24.041235+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":228,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.833007+0000","last_change":"2024-04-14T15:13:30.197647+0000","last_active":"2024-04-14T15:13:58.833007+0000","last_peered":"2024-04-14T15:13:58.833007+0000","last_clean":"2024-04-14T15:13:58.833007+0000","last_became_active":"2024-04-14T15:09:49.065397+0000","last_became_peered":"2024-04-14T15:09:49.065397+0000","last_unstale":"2024-04-14T15:13:58.833007+0000","last_undegraded":"2024-04-14T15:13:58.833007+0000","last_fullsized":"2024-04-14T15:13:58.833007+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:30.197500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:13:30.197500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:32.195058+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":137,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.834776+0000","last_change":"2024-04-14T15:12:56.200223+0000","last_active":"2024-04-14T15:13:58.834776+0000","last_peered":"2024-04-14T15:13:58.834776+0000","last_clean":"2024-04-14T15:13:58.834776+0000","last_became_active":"2024-04-14T15:12:56.183682+0000","last_became_peered":"2024-04-14T15:12:56.183682+0000","last_unstale":"2024-04-14T15:13:58.834776+0000","last_undegraded":"2024-04-14T15:13:58.834776+0000","last_fullsized":"2024-04-14T15:13:58.834776+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:17.915602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":144,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.844500+0000","last_change":"2024-04-14T15:12:56.407193+0000","last_active":"2024-04-14T15:13:58.844500+0000","last_peered":"2024-04-14T15:13:58.844500+0000","last_clean":"2024-04-14T15:13:58.844500+0000","last_became_active":"2024-04-14T15:12:56.400306+0000","last_became_peered":"2024-04-14T15:12:56.400306+0000","last_unstale":"2024-04-14T15:13:58.844500+0000","last_undegraded":"2024-04-14T15:13:58.844500+0000","last_fullsized":"2024-04-14T15:13:58.844500+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:06.281523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":137,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.837026+0000","last_change":"2024-04-14T15:12:56.477020+0000","last_active":"2024-04-14T15:13:58.837026+0000","last_peered":"2024-04-14T15:13:58.837026+0000","last_clean":"2024-04-14T15:13:58.837026+0000","last_became_active":"2024-04-14T15:12:56.474322+0000","last_became_peered":"2024-04-14T15:12:56.474322+0000","last_unstale":"2024-04-14T15:13:58.837026+0000","last_undegraded":"2024-04-14T15:13:58.837026+0000","last_fullsized":"2024-04-14T15:13:58.837026+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"72'17","reported_seq":312,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.845868+0000","last_change":"2024-04-14T15:13:18.575120+0000","last_active":"2024-04-14T15:13:58.845868+0000","last_peered":"2024-04-14T15:13:58.845868+0000","last_clean":"2024-04-14T15:13:58.845868+0000","last_became_active":"2024-04-14T15:13:12.933907+0000","last_became_peered":"2024-04-14T15:13:12.933907+0000","last_unstale":"2024-04-14T15:13:58.845868+0000","last_undegraded":"2024-04-14T15:13:58.845868+0000","last_fullsized":"2024-04-14T15:13:58.845868+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'17","last_scrub_stamp":"2024-04-14T15:13:18.574993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:13:18.574993+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:23.951036+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":240,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.833602+0000","last_change":"2024-04-14T15:13:20.212603+0000","last_active":"2024-04-14T15:13:58.833602+0000","last_peered":"2024-04-14T15:13:58.833602+0000","last_clean":"2024-04-14T15:13:58.833602+0000","last_became_active":"2024-04-14T15:09:43.022351+0000","last_became_peered":"2024-04-14T15:09:43.022351+0000","last_unstale":"2024-04-14T15:13:58.833602+0000","last_undegraded":"2024-04-14T15:13:58.833602+0000","last_fullsized":"2024-04-14T15:13:58.833602+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:20.212523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:13:20.212523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:44.109813+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":224,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.834250+0000","last_change":"2024-04-14T15:12:19.226834+0000","last_active":"2024-04-14T15:13:58.834250+0000","last_peered":"2024-04-14T15:13:58.834250+0000","last_clean":"2024-04-14T15:13:58.834250+0000","last_became_active":"2024-04-14T15:09:47.042810+0000","last_became_peered":"2024-04-14T15:09:47.042810+0000","last_unstale":"2024-04-14T15:13:58.834250+0000","last_undegraded":"2024-04-14T15:13:58.834250+0000","last_fullsized":"2024-04-14T15:13:58.834250+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:19.226701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:12:19.226701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":228,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.834812+0000","last_change":"2024-04-14T15:12:42.342665+0000","last_active":"2024-04-14T15:13:58.834812+0000","last_peered":"2024-04-14T15:13:58.834812+0000","last_clean":"2024-04-14T15:13:58.834812+0000","last_became_active":"2024-04-14T15:09:49.069594+0000","last_became_peered":"2024-04-14T15:09:49.069594+0000","last_unstale":"2024-04-14T15:13:58.834812+0000","last_undegraded":"2024-04-14T15:13:58.834812+0000","last_fullsized":"2024-04-14T15:13:58.834812+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:42.342544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:12:42.342544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":137,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.837946+0000","last_change":"2024-04-14T15:12:56.476000+0000","last_active":"2024-04-14T15:13:58.837946+0000","last_peered":"2024-04-14T15:13:58.837946+0000","last_clean":"2024-04-14T15:13:58.837946+0000","last_became_active":"2024-04-14T15:12:56.470749+0000","last_became_peered":"2024-04-14T15:12:56.470749+0000","last_unstale":"2024-04-14T15:13:58.837946+0000","last_undegraded":"2024-04-14T15:13:58.837946+0000","last_fullsized":"2024-04-14T15:13:58.837946+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:00.938994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":144,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.846426+0000","last_change":"2024-04-14T15:12:56.371816+0000","last_active":"2024-04-14T15:13:58.846426+0000","last_peered":"2024-04-14T15:13:58.846426+0000","last_clean":"2024-04-14T15:13:58.846426+0000","last_became_active":"2024-04-14T15:12:56.368789+0000","last_became_peered":"2024-04-14T15:12:56.368789+0000","last_unstale":"2024-04-14T15:13:58.846426+0000","last_undegraded":"2024-04-14T15:13:58.846426+0000","last_fullsized":"2024-04-14T15:13:58.846426+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:05.650640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":137,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.833970+0000","last_change":"2024-04-14T15:12:56.290116+0000","last_active":"2024-04-14T15:13:58.833970+0000","last_peered":"2024-04-14T15:13:58.833970+0000","last_clean":"2024-04-14T15:13:58.833970+0000","last_became_active":"2024-04-14T15:12:56.289745+0000","last_became_peered":"2024-04-14T15:12:56.289745+0000","last_unstale":"2024-04-14T15:13:58.833970+0000","last_undegraded":"2024-04-14T15:13:58.833970+0000","last_fullsized":"2024-04-14T15:13:58.833970+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:03.824111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"72'10","reported_seq":243,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.831984+0000","last_change":"2024-04-14T15:11:42.802009+0000","last_active":"2024-04-14T15:13:58.831984+0000","last_peered":"2024-04-14T15:13:58.831984+0000","last_clean":"2024-04-14T15:13:58.831984+0000","last_became_active":"2024-04-14T15:09:45.029170+0000","last_became_peered":"2024-04-14T15:09:45.029170+0000","last_unstale":"2024-04-14T15:13:58.831984+0000","last_undegraded":"2024-04-14T15:13:58.831984+0000","last_fullsized":"2024-04-14T15:13:58.831984+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-04-14T15:11:42.801888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:11:42.801888+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":240,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.832558+0000","last_change":"2024-04-14T15:12:24.329867+0000","last_active":"2024-04-14T15:13:58.832558+0000","last_peered":"2024-04-14T15:13:58.832558+0000","last_clean":"2024-04-14T15:13:58.832558+0000","last_became_active":"2024-04-14T15:09:43.018871+0000","last_became_peered":"2024-04-14T15:09:43.018871+0000","last_unstale":"2024-04-14T15:13:58.832558+0000","last_undegraded":"2024-04-14T15:13:58.832558+0000","last_fullsized":"2024-04-14T15:13:58.832558+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:24.329791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:12:24.329791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"72'8","reported_seq":292,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.831788+0000","last_change":"2024-04-14T15:13:25.292158+0000","last_active":"2024-04-14T15:13:58.831788+0000","last_peered":"2024-04-14T15:13:58.831788+0000","last_clean":"2024-04-14T15:13:58.831788+0000","last_became_active":"2024-04-14T15:09:47.044085+0000","last_became_peered":"2024-04-14T15:09:47.044085+0000","last_unstale":"2024-04-14T15:13:58.831788+0000","last_undegraded":"2024-04-14T15:13:58.831788+0000","last_fullsized":"2024-04-14T15:13:58.831788+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:13:25.292003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:13:25.292003+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:34.820359+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":220,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.849357+0000","last_change":"2024-04-14T15:12:24.516311+0000","last_active":"2024-04-14T15:13:58.849357+0000","last_peered":"2024-04-14T15:13:58.849357+0000","last_clean":"2024-04-14T15:13:58.849357+0000","last_became_active":"2024-04-14T15:09:49.065274+0000","last_became_peered":"2024-04-14T15:09:49.065274+0000","last_unstale":"2024-04-14T15:13:58.849357+0000","last_undegraded":"2024-04-14T15:13:58.849357+0000","last_fullsized":"2024-04-14T15:13:58.849357+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:24.516200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:12:24.516200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:59.553945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":137,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.842282+0000","last_change":"2024-04-14T15:12:56.490848+0000","last_active":"2024-04-14T15:13:58.842282+0000","last_peered":"2024-04-14T15:13:58.842282+0000","last_clean":"2024-04-14T15:13:58.842282+0000","last_became_active":"2024-04-14T15:12:56.490582+0000","last_became_peered":"2024-04-14T15:12:56.490582+0000","last_unstale":"2024-04-14T15:13:58.842282+0000","last_undegraded":"2024-04-14T15:13:58.842282+0000","last_fullsized":"2024-04-14T15:13:58.842282+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:04.489665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"109'9","reported_seq":153,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.849132+0000","last_change":"2024-04-14T15:12:56.242032+0000","last_active":"2024-04-14T15:13:58.849132+0000","last_peered":"2024-04-14T15:13:58.849132+0000","last_clean":"2024-04-14T15:13:58.849132+0000","last_became_active":"2024-04-14T15:12:56.241684+0000","last_became_peered":"2024-04-14T15:12:56.241684+0000","last_unstale":"2024-04-14T15:13:58.849132+0000","last_undegraded":"2024-04-14T15:13:58.849132+0000","last_fullsized":"2024-04-14T15:13:58.849132+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:19.117444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":137,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.828833+0000","last_change":"2024-04-14T15:12:56.495092+0000","last_active":"2024-04-14T15:13:58.828833+0000","last_peered":"2024-04-14T15:13:58.828833+0000","last_clean":"2024-04-14T15:13:58.828833+0000","last_became_active":"2024-04-14T15:12:56.487615+0000","last_became_peered":"2024-04-14T15:12:56.487615+0000","last_unstale":"2024-04-14T15:13:58.828833+0000","last_undegraded":"2024-04-14T15:13:58.828833+0000","last_fullsized":"2024-04-14T15:13:58.828833+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:03.246475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"72'15","reported_seq":261,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.829903+0000","last_change":"2024-04-14T15:12:42.312510+0000","last_active":"2024-04-14T15:13:58.829903+0000","last_peered":"2024-04-14T15:13:58.829903+0000","last_clean":"2024-04-14T15:13:58.829903+0000","last_became_active":"2024-04-14T15:09:45.033331+0000","last_became_peered":"2024-04-14T15:09:45.033331+0000","last_unstale":"2024-04-14T15:13:58.829903+0000","last_undegraded":"2024-04-14T15:13:58.829903+0000","last_fullsized":"2024-04-14T15:13:58.829903+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-04-14T15:12:42.312355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:12:42.312355+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":240,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.831957+0000","last_change":"2024-04-14T15:12:31.395583+0000","last_active":"2024-04-14T15:13:58.831957+0000","last_peered":"2024-04-14T15:13:58.831957+0000","last_clean":"2024-04-14T15:13:58.831957+0000","last_became_active":"2024-04-14T15:09:43.020878+0000","last_became_peered":"2024-04-14T15:09:43.020878+0000","last_unstale":"2024-04-14T15:13:58.831957+0000","last_undegraded":"2024-04-14T15:13:58.831957+0000","last_fullsized":"2024-04-14T15:13:58.831957+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:31.395502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:12:31.395502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"72'8","reported_seq":292,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.849243+0000","last_change":"2024-04-14T15:13:33.568661+0000","last_active":"2024-04-14T15:13:58.849243+0000","last_peered":"2024-04-14T15:13:58.849243+0000","last_clean":"2024-04-14T15:13:58.849243+0000","last_became_active":"2024-04-14T15:09:47.050962+0000","last_became_peered":"2024-04-14T15:09:47.050962+0000","last_unstale":"2024-04-14T15:13:58.849243+0000","last_undegraded":"2024-04-14T15:13:58.849243+0000","last_fullsized":"2024-04-14T15:13:58.849243+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:13:33.568569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:13:33.568569+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:40.930108+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":220,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.840246+0000","last_change":"2024-04-14T15:11:25.638017+0000","last_active":"2024-04-14T15:13:58.840246+0000","last_peered":"2024-04-14T15:13:58.840246+0000","last_clean":"2024-04-14T15:13:58.840246+0000","last_became_active":"2024-04-14T15:09:49.058940+0000","last_became_peered":"2024-04-14T15:09:49.058940+0000","last_unstale":"2024-04-14T15:13:58.840246+0000","last_undegraded":"2024-04-14T15:13:58.840246+0000","last_fullsized":"2024-04-14T15:13:58.840246+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:25.637888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:25.637888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1b","version":"110'1001","reported_seq":2138,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.828704+0000","last_change":"2024-04-14T15:12:56.247499+0000","last_active":"2024-04-14T15:13:58.828704+0000","last_peered":"2024-04-14T15:13:58.828704+0000","last_clean":"2024-04-14T15:13:58.828704+0000","last_became_active":"2024-04-14T15:12:56.247212+0000","last_became_peered":"2024-04-14T15:12:56.247212+0000","last_unstale":"2024-04-14T15:13:58.828704+0000","last_undegraded":"2024-04-14T15:13:58.828704+0000","last_fullsized":"2024-04-14T15:13:58.828704+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":1001,"log_dups_size":0,"ondisk_log_size":1001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1000,"num_read_kb":0,"num_write":1000,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":144,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.838154+0000","last_change":"2024-04-14T15:12:56.190468+0000","last_active":"2024-04-14T15:13:58.838154+0000","last_peered":"2024-04-14T15:13:58.838154+0000","last_clean":"2024-04-14T15:13:58.838154+0000","last_became_active":"2024-04-14T15:12:56.188780+0000","last_became_peered":"2024-04-14T15:12:56.188780+0000","last_unstale":"2024-04-14T15:13:58.838154+0000","last_undegraded":"2024-04-14T15:13:58.838154+0000","last_fullsized":"2024-04-14T15:13:58.838154+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:15.601872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1","version":"108'7","reported_seq":150,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.838750+0000","last_change":"2024-04-14T15:12:56.385652+0000","last_active":"2024-04-14T15:13:58.838750+0000","last_peered":"2024-04-14T15:13:58.838750+0000","last_clean":"2024-04-14T15:13:58.838750+0000","last_became_active":"2024-04-14T15:12:56.381026+0000","last_became_peered":"2024-04-14T15:12:56.381026+0000","last_unstale":"2024-04-14T15:13:58.838750+0000","last_undegraded":"2024-04-14T15:13:58.838750+0000","last_fullsized":"2024-04-14T15:13:58.838750+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:07.570579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"85'16","reported_seq":260,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.848549+0000","last_change":"2024-04-14T15:12:40.495261+0000","last_active":"2024-04-14T15:13:58.848549+0000","last_peered":"2024-04-14T15:13:58.848549+0000","last_clean":"2024-04-14T15:13:58.848549+0000","last_became_active":"2024-04-14T15:09:45.028212+0000","last_became_peered":"2024-04-14T15:09:45.028212+0000","last_unstale":"2024-04-14T15:13:58.848549+0000","last_undegraded":"2024-04-14T15:13:58.848549+0000","last_fullsized":"2024-04-14T15:13:58.848549+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-14T15:12:40.495139+0000","last_deep_scrub":"85'16","last_deep_scrub_stamp":"2024-04-14T15:10:47.456169+0000","last_clean_scrub_stamp":"2024-04-14T15:12:40.495139+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:00.218258+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":240,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.832816+0000","last_change":"2024-04-14T15:12:43.093032+0000","last_active":"2024-04-14T15:13:58.832816+0000","last_peered":"2024-04-14T15:13:58.832816+0000","last_clean":"2024-04-14T15:13:58.832816+0000","last_became_active":"2024-04-14T15:09:43.018601+0000","last_became_peered":"2024-04-14T15:09:43.018601+0000","last_unstale":"2024-04-14T15:13:58.832816+0000","last_undegraded":"2024-04-14T15:13:58.832816+0000","last_fullsized":"2024-04-14T15:13:58.832816+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:43.092895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:12:43.092895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":232,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.842842+0000","last_change":"2024-04-14T15:12:27.761412+0000","last_active":"2024-04-14T15:13:58.842842+0000","last_peered":"2024-04-14T15:13:58.842842+0000","last_clean":"2024-04-14T15:13:58.842842+0000","last_became_active":"2024-04-14T15:09:47.043021+0000","last_became_peered":"2024-04-14T15:09:47.043021+0000","last_unstale":"2024-04-14T15:13:58.842842+0000","last_undegraded":"2024-04-14T15:13:58.842842+0000","last_fullsized":"2024-04-14T15:13:58.842842+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:27.761311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:12:27.761311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":167,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.845185+0000","last_change":"2024-04-14T15:12:34.084996+0000","last_active":"2024-04-14T15:13:58.845185+0000","last_peered":"2024-04-14T15:13:58.845185+0000","last_clean":"2024-04-14T15:13:58.845185+0000","last_became_active":"2024-04-14T15:11:57.120861+0000","last_became_peered":"2024-04-14T15:11:57.120861+0000","last_unstale":"2024-04-14T15:13:58.845185+0000","last_undegraded":"2024-04-14T15:13:58.845185+0000","last_fullsized":"2024-04-14T15:13:58.845185+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:34.084920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:12:34.084920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:01.190876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1a","version":"108'1","reported_seq":139,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.844150+0000","last_change":"2024-04-14T15:12:56.310424+0000","last_active":"2024-04-14T15:13:58.844150+0000","last_peered":"2024-04-14T15:13:58.844150+0000","last_clean":"2024-04-14T15:13:58.844150+0000","last_became_active":"2024-04-14T15:12:56.310046+0000","last_became_peered":"2024-04-14T15:12:56.310046+0000","last_unstale":"2024-04-14T15:13:58.844150+0000","last_undegraded":"2024-04-14T15:13:58.844150+0000","last_fullsized":"2024-04-14T15:13:58.844150+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:16.150890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1024,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.8","version":"108'2","reported_seq":146,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.839839+0000","last_change":"2024-04-14T15:12:56.340353+0000","last_active":"2024-04-14T15:13:58.839839+0000","last_peered":"2024-04-14T15:13:58.839839+0000","last_clean":"2024-04-14T15:13:58.839839+0000","last_became_active":"2024-04-14T15:12:56.339672+0000","last_became_peered":"2024-04-14T15:12:56.339672+0000","last_unstale":"2024-04-14T15:13:58.839839+0000","last_undegraded":"2024-04-14T15:13:58.839839+0000","last_fullsized":"2024-04-14T15:13:58.839839+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:13.587919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":137,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.848452+0000","last_change":"2024-04-14T15:12:56.427581+0000","last_active":"2024-04-14T15:13:58.848452+0000","last_peered":"2024-04-14T15:13:58.848452+0000","last_clean":"2024-04-14T15:13:58.848452+0000","last_became_active":"2024-04-14T15:12:56.427276+0000","last_became_peered":"2024-04-14T15:12:56.427276+0000","last_unstale":"2024-04-14T15:13:58.848452+0000","last_undegraded":"2024-04-14T15:13:58.848452+0000","last_fullsized":"2024-04-14T15:13:58.848452+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:09.993135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"98'9","reported_seq":217,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.828454+0000","last_change":"2024-04-14T15:13:38.414555+0000","last_active":"2024-04-14T15:13:58.828454+0000","last_peered":"2024-04-14T15:13:58.828454+0000","last_clean":"2024-04-14T15:13:58.828454+0000","last_became_active":"2024-04-14T15:11:57.118068+0000","last_became_peered":"2024-04-14T15:11:57.118068+0000","last_unstale":"2024-04-14T15:13:58.828454+0000","last_undegraded":"2024-04-14T15:13:58.828454+0000","last_fullsized":"2024-04-14T15:13:58.828454+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-04-14T15:13:38.414471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:13:38.414471+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:49.479034+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"72'14","reported_seq":258,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.838214+0000","last_change":"2024-04-14T15:13:20.574920+0000","last_active":"2024-04-14T15:13:58.838214+0000","last_peered":"2024-04-14T15:13:58.838214+0000","last_clean":"2024-04-14T15:13:58.838214+0000","last_became_active":"2024-04-14T15:09:45.025673+0000","last_became_peered":"2024-04-14T15:09:45.025673+0000","last_unstale":"2024-04-14T15:13:58.838214+0000","last_undegraded":"2024-04-14T15:13:58.838214+0000","last_fullsized":"2024-04-14T15:13:58.838214+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'14","last_scrub_stamp":"2024-04-14T15:13:20.574826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:13:20.574826+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:22.739206+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":240,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.841295+0000","last_change":"2024-04-14T15:13:18.161888+0000","last_active":"2024-04-14T15:13:58.841295+0000","last_peered":"2024-04-14T15:13:58.841295+0000","last_clean":"2024-04-14T15:13:58.841295+0000","last_became_active":"2024-04-14T15:09:43.020503+0000","last_became_peered":"2024-04-14T15:09:43.020503+0000","last_unstale":"2024-04-14T15:13:58.841295+0000","last_undegraded":"2024-04-14T15:13:58.841295+0000","last_fullsized":"2024-04-14T15:13:58.841295+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:18.161799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:13:18.161799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:23.917967+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":229,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.833941+0000","last_change":"2024-04-14T15:13:25.116624+0000","last_active":"2024-04-14T15:13:58.833941+0000","last_peered":"2024-04-14T15:13:58.833941+0000","last_clean":"2024-04-14T15:13:58.833941+0000","last_became_active":"2024-04-14T15:09:49.129610+0000","last_became_peered":"2024-04-14T15:09:49.129610+0000","last_unstale":"2024-04-14T15:13:58.833941+0000","last_undegraded":"2024-04-14T15:13:58.833941+0000","last_fullsized":"2024-04-14T15:13:58.833941+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:25.116541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:13:25.116541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:51.711645+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":137,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.848846+0000","last_change":"2024-04-14T15:12:56.458443+0000","last_active":"2024-04-14T15:13:58.848846+0000","last_peered":"2024-04-14T15:13:58.848846+0000","last_clean":"2024-04-14T15:13:58.848846+0000","last_became_active":"2024-04-14T15:12:56.458177+0000","last_became_peered":"2024-04-14T15:12:56.458177+0000","last_unstale":"2024-04-14T15:13:58.848846+0000","last_undegraded":"2024-04-14T15:13:58.848846+0000","last_fullsized":"2024-04-14T15:13:58.848846+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:05.601441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":144,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.828480+0000","last_change":"2024-04-14T15:12:56.403204+0000","last_active":"2024-04-14T15:13:58.828480+0000","last_peered":"2024-04-14T15:13:58.828480+0000","last_clean":"2024-04-14T15:13:58.828480+0000","last_became_active":"2024-04-14T15:12:56.402900+0000","last_became_peered":"2024-04-14T15:12:56.402900+0000","last_unstale":"2024-04-14T15:13:58.828480+0000","last_undegraded":"2024-04-14T15:13:58.828480+0000","last_fullsized":"2024-04-14T15:13:58.828480+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:11.956498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":137,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.838334+0000","last_change":"2024-04-14T15:12:56.363969+0000","last_active":"2024-04-14T15:13:58.838334+0000","last_peered":"2024-04-14T15:13:58.838334+0000","last_clean":"2024-04-14T15:13:58.838334+0000","last_became_active":"2024-04-14T15:12:56.355254+0000","last_became_peered":"2024-04-14T15:12:56.355254+0000","last_unstale":"2024-04-14T15:13:58.838334+0000","last_undegraded":"2024-04-14T15:13:58.838334+0000","last_fullsized":"2024-04-14T15:13:58.838334+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:03.500100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":233,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.838526+0000","last_change":"2024-04-14T15:13:21.526922+0000","last_active":"2024-04-14T15:13:58.838526+0000","last_peered":"2024-04-14T15:13:58.838526+0000","last_clean":"2024-04-14T15:13:58.838526+0000","last_became_active":"2024-04-14T15:09:47.044610+0000","last_became_peered":"2024-04-14T15:09:47.044610+0000","last_unstale":"2024-04-14T15:13:58.838526+0000","last_undegraded":"2024-04-14T15:13:58.838526+0000","last_fullsized":"2024-04-14T15:13:58.838526+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:21.526839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:13:21.526839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:40.438748+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"72'18","reported_seq":218,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.832526+0000","last_change":"2024-04-14T15:13:18.170741+0000","last_active":"2024-04-14T15:13:58.832526+0000","last_peered":"2024-04-14T15:13:58.832526+0000","last_clean":"2024-04-14T15:13:58.832526+0000","last_became_active":"2024-04-14T15:11:57.118744+0000","last_became_peered":"2024-04-14T15:11:57.118744+0000","last_unstale":"2024-04-14T15:13:58.832526+0000","last_undegraded":"2024-04-14T15:13:58.832526+0000","last_fullsized":"2024-04-14T15:13:58.832526+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'18","last_scrub_stamp":"2024-04-14T15:13:18.170570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:13:18.170570+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:25.248483+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":33,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"91'2","reported_seq":235,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.829335+0000","last_change":"2024-04-14T15:11:37.974863+0000","last_active":"2024-04-14T15:13:58.829335+0000","last_peered":"2024-04-14T15:13:58.829335+0000","last_clean":"2024-04-14T15:13:58.829335+0000","last_became_active":"2024-04-14T15:09:43.018478+0000","last_became_peered":"2024-04-14T15:09:43.018478+0000","last_unstale":"2024-04-14T15:13:58.829335+0000","last_undegraded":"2024-04-14T15:13:58.829335+0000","last_fullsized":"2024-04-14T15:13:58.829335+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'2","last_scrub_stamp":"2024-04-14T15:11:37.974770+0000","last_deep_scrub":"91'2","last_deep_scrub_stamp":"2024-04-14T15:11:37.974770+0000","last_clean_scrub_stamp":"2024-04-14T15:11:37.974770+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":228,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.833848+0000","last_change":"2024-04-14T15:13:26.563376+0000","last_active":"2024-04-14T15:13:58.833848+0000","last_peered":"2024-04-14T15:13:58.833848+0000","last_clean":"2024-04-14T15:13:58.833848+0000","last_became_active":"2024-04-14T15:09:49.065684+0000","last_became_peered":"2024-04-14T15:09:49.065684+0000","last_unstale":"2024-04-14T15:13:58.833848+0000","last_undegraded":"2024-04-14T15:13:58.833848+0000","last_fullsized":"2024-04-14T15:13:58.833848+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:26.563298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:13:26.563298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:52.535208+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.14","version":"108'1","reported_seq":139,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.848307+0000","last_change":"2024-04-14T15:12:56.484486+0000","last_active":"2024-04-14T15:13:58.848307+0000","last_peered":"2024-04-14T15:13:58.848307+0000","last_clean":"2024-04-14T15:13:58.848307+0000","last_became_active":"2024-04-14T15:12:56.484244+0000","last_became_peered":"2024-04-14T15:12:56.484244+0000","last_unstale":"2024-04-14T15:13:58.848307+0000","last_undegraded":"2024-04-14T15:13:58.848307+0000","last_fullsized":"2024-04-14T15:13:58.848307+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:03.557774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"109'10","reported_seq":154,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.836921+0000","last_change":"2024-04-14T15:12:56.405375+0000","last_active":"2024-04-14T15:13:58.836921+0000","last_peered":"2024-04-14T15:13:58.836921+0000","last_clean":"2024-04-14T15:13:58.836921+0000","last_became_active":"2024-04-14T15:12:56.405078+0000","last_became_peered":"2024-04-14T15:12:56.405078+0000","last_unstale":"2024-04-14T15:13:58.836921+0000","last_undegraded":"2024-04-14T15:13:58.836921+0000","last_fullsized":"2024-04-14T15:13:58.836921+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:06.264025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":137,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.833955+0000","last_change":"2024-04-14T15:12:56.407793+0000","last_active":"2024-04-14T15:13:58.833955+0000","last_peered":"2024-04-14T15:13:58.833955+0000","last_clean":"2024-04-14T15:13:58.833955+0000","last_became_active":"2024-04-14T15:12:56.402023+0000","last_became_peered":"2024-04-14T15:12:56.402023+0000","last_unstale":"2024-04-14T15:13:58.833955+0000","last_undegraded":"2024-04-14T15:13:58.833955+0000","last_fullsized":"2024-04-14T15:13:58.833955+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:13.313190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":233,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.843996+0000","last_change":"2024-04-14T15:12:24.013345+0000","last_active":"2024-04-14T15:13:58.843996+0000","last_peered":"2024-04-14T15:13:58.843996+0000","last_clean":"2024-04-14T15:13:58.843996+0000","last_became_active":"2024-04-14T15:09:47.048427+0000","last_became_peered":"2024-04-14T15:09:47.048427+0000","last_unstale":"2024-04-14T15:13:58.843996+0000","last_undegraded":"2024-04-14T15:13:58.843996+0000","last_fullsized":"2024-04-14T15:13:58.843996+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:24.013261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:12:24.013261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:01.096666+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"102'27","reported_seq":281,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.853310+0000","last_change":"2024-04-14T15:12:30.928373+0000","last_active":"2024-04-14T15:13:58.853310+0000","last_peered":"2024-04-14T15:13:58.853310+0000","last_clean":"2024-04-14T15:13:58.853310+0000","last_became_active":"2024-04-14T15:09:45.179182+0000","last_became_peered":"2024-04-14T15:09:45.179182+0000","last_unstale":"2024-04-14T15:13:58.853310+0000","last_undegraded":"2024-04-14T15:13:58.853310+0000","last_fullsized":"2024-04-14T15:13:58.853310+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'27","last_scrub_stamp":"2024-04-14T15:12:30.928216+0000","last_deep_scrub":"102'27","last_deep_scrub_stamp":"2024-04-14T15:12:30.928216+0000","last_clean_scrub_stamp":"2024-04-14T15:12:30.928216+0000","objects_scrubbed":8,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":241,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.834631+0000","last_change":"2024-04-14T15:13:16.122436+0000","last_active":"2024-04-14T15:13:58.834631+0000","last_peered":"2024-04-14T15:13:58.834631+0000","last_clean":"2024-04-14T15:13:58.834631+0000","last_became_active":"2024-04-14T15:09:43.021006+0000","last_became_peered":"2024-04-14T15:09:43.021006+0000","last_unstale":"2024-04-14T15:13:58.834631+0000","last_undegraded":"2024-04-14T15:13:58.834631+0000","last_fullsized":"2024-04-14T15:13:58.834631+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:16.122352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:13:16.122352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:46.102116+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":221,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.832099+0000","last_change":"2024-04-14T15:11:35.993757+0000","last_active":"2024-04-14T15:13:58.832099+0000","last_peered":"2024-04-14T15:13:58.832099+0000","last_clean":"2024-04-14T15:13:58.832099+0000","last_became_active":"2024-04-14T15:09:49.128136+0000","last_became_peered":"2024-04-14T15:09:49.128136+0000","last_unstale":"2024-04-14T15:13:58.832099+0000","last_undegraded":"2024-04-14T15:13:58.832099+0000","last_fullsized":"2024-04-14T15:13:58.832099+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:35.993681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:35.993681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.17","version":"108'1","reported_seq":138,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.834622+0000","last_change":"2024-04-14T15:12:56.473773+0000","last_active":"2024-04-14T15:13:58.834622+0000","last_peered":"2024-04-14T15:13:58.834622+0000","last_clean":"2024-04-14T15:13:58.834622+0000","last_became_active":"2024-04-14T15:12:56.469133+0000","last_became_peered":"2024-04-14T15:12:56.469133+0000","last_unstale":"2024-04-14T15:13:58.834622+0000","last_undegraded":"2024-04-14T15:13:58.834622+0000","last_fullsized":"2024-04-14T15:13:58.834622+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":137,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.838531+0000","last_change":"2024-04-14T15:12:56.402757+0000","last_active":"2024-04-14T15:13:58.838531+0000","last_peered":"2024-04-14T15:13:58.838531+0000","last_clean":"2024-04-14T15:13:58.838531+0000","last_became_active":"2024-04-14T15:12:56.383575+0000","last_became_peered":"2024-04-14T15:12:56.383575+0000","last_unstale":"2024-04-14T15:13:58.838531+0000","last_undegraded":"2024-04-14T15:13:58.838531+0000","last_fullsized":"2024-04-14T15:13:58.838531+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:11.621424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":232,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.842988+0000","last_change":"2024-04-14T15:12:36.818501+0000","last_active":"2024-04-14T15:13:58.842988+0000","last_peered":"2024-04-14T15:13:58.842988+0000","last_clean":"2024-04-14T15:13:58.842988+0000","last_became_active":"2024-04-14T15:09:47.036696+0000","last_became_peered":"2024-04-14T15:09:47.036696+0000","last_unstale":"2024-04-14T15:13:58.842988+0000","last_undegraded":"2024-04-14T15:13:58.842988+0000","last_fullsized":"2024-04-14T15:13:58.842988+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:36.818402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:12:36.818402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"72'12","reported_seq":254,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.848932+0000","last_change":"2024-04-14T15:12:22.083844+0000","last_active":"2024-04-14T15:13:58.848932+0000","last_peered":"2024-04-14T15:13:58.848932+0000","last_clean":"2024-04-14T15:13:58.848932+0000","last_became_active":"2024-04-14T15:09:45.037231+0000","last_became_peered":"2024-04-14T15:09:45.037231+0000","last_unstale":"2024-04-14T15:13:58.848932+0000","last_undegraded":"2024-04-14T15:13:58.848932+0000","last_fullsized":"2024-04-14T15:13:58.848932+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'12","last_scrub_stamp":"2024-04-14T15:12:22.083718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:12:22.083718+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:00.310067+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":241,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.832981+0000","last_change":"2024-04-14T15:12:41.108387+0000","last_active":"2024-04-14T15:13:58.832981+0000","last_peered":"2024-04-14T15:13:58.832981+0000","last_clean":"2024-04-14T15:13:58.832981+0000","last_became_active":"2024-04-14T15:09:43.025095+0000","last_became_peered":"2024-04-14T15:09:43.025095+0000","last_unstale":"2024-04-14T15:13:58.832981+0000","last_undegraded":"2024-04-14T15:13:58.832981+0000","last_fullsized":"2024-04-14T15:13:58.832981+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:41.108288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:12:41.108288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":248,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.845355+0000","last_change":"2024-04-14T15:12:39.104506+0000","last_active":"2024-04-14T15:13:58.845355+0000","last_peered":"2024-04-14T15:13:58.845355+0000","last_clean":"2024-04-14T15:13:58.845355+0000","last_became_active":"2024-04-14T15:09:43.018509+0000","last_became_peered":"2024-04-14T15:09:43.018509+0000","last_unstale":"2024-04-14T15:13:58.845355+0000","last_undegraded":"2024-04-14T15:13:58.845355+0000","last_fullsized":"2024-04-14T15:13:58.845355+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:39.104391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:12:39.104391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0018592680000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"72'1","reported_seq":221,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.834336+0000","last_change":"2024-04-14T15:11:25.095270+0000","last_active":"2024-04-14T15:13:58.834336+0000","last_peered":"2024-04-14T15:13:58.834336+0000","last_clean":"2024-04-14T15:13:58.834336+0000","last_became_active":"2024-04-14T15:09:49.065243+0000","last_became_peered":"2024-04-14T15:09:49.065243+0000","last_unstale":"2024-04-14T15:13:58.834336+0000","last_undegraded":"2024-04-14T15:13:58.834336+0000","last_fullsized":"2024-04-14T15:13:58.834336+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-04-14T15:11:25.095182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:25.095182+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.f","version":"109'5","reported_seq":47,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.839311+0000","last_change":"2024-04-14T15:13:52.871043+0000","last_active":"2024-04-14T15:13:58.839311+0000","last_peered":"2024-04-14T15:13:58.839311+0000","last_clean":"2024-04-14T15:13:58.839311+0000","last_became_active":"2024-04-14T15:13:52.870676+0000","last_became_peered":"2024-04-14T15:13:52.870676+0000","last_unstale":"2024-04-14T15:13:58.839311+0000","last_undegraded":"2024-04-14T15:13:58.839311+0000","last_fullsized":"2024-04-14T15:13:58.839311+0000","mapping_epoch":160,"log_start":"109'5","ondisk_log_start":"109'5","created":107,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":5,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:04.181372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.12","version":"111'10","reported_seq":151,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.844050+0000","last_change":"2024-04-14T15:12:56.370347+0000","last_active":"2024-04-14T15:13:58.844050+0000","last_peered":"2024-04-14T15:13:58.844050+0000","last_clean":"2024-04-14T15:13:58.844050+0000","last_became_active":"2024-04-14T15:12:56.368444+0000","last_became_peered":"2024-04-14T15:12:56.368444+0000","last_unstale":"2024-04-14T15:13:58.844050+0000","last_undegraded":"2024-04-14T15:13:58.844050+0000","last_fullsized":"2024-04-14T15:13:58.844050+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:03.097649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165835,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":27326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":137,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.843028+0000","last_change":"2024-04-14T15:12:56.311510+0000","last_active":"2024-04-14T15:13:58.843028+0000","last_peered":"2024-04-14T15:13:58.843028+0000","last_clean":"2024-04-14T15:13:58.843028+0000","last_became_active":"2024-04-14T15:12:56.310799+0000","last_became_peered":"2024-04-14T15:12:56.310799+0000","last_unstale":"2024-04-14T15:13:58.843028+0000","last_undegraded":"2024-04-14T15:13:58.843028+0000","last_fullsized":"2024-04-14T15:13:58.843028+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:01.096695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":232,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.836071+0000","last_change":"2024-04-14T15:13:31.226741+0000","last_active":"2024-04-14T15:13:58.836071+0000","last_peered":"2024-04-14T15:13:58.836071+0000","last_clean":"2024-04-14T15:13:58.836071+0000","last_became_active":"2024-04-14T15:09:47.047201+0000","last_became_peered":"2024-04-14T15:09:47.047201+0000","last_unstale":"2024-04-14T15:13:58.836071+0000","last_undegraded":"2024-04-14T15:13:58.836071+0000","last_fullsized":"2024-04-14T15:13:58.836071+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:31.226659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:13:31.226659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:53.711821+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"90'24","reported_seq":319,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.841060+0000","last_change":"2024-04-14T15:13:26.147772+0000","last_active":"2024-04-14T15:13:58.841060+0000","last_peered":"2024-04-14T15:13:58.841060+0000","last_clean":"2024-04-14T15:13:58.841060+0000","last_became_active":"2024-04-14T15:09:45.179978+0000","last_became_peered":"2024-04-14T15:09:45.179978+0000","last_unstale":"2024-04-14T15:13:58.841060+0000","last_undegraded":"2024-04-14T15:13:58.841060+0000","last_fullsized":"2024-04-14T15:13:58.841060+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'24","last_scrub_stamp":"2024-04-14T15:13:26.147631+0000","last_deep_scrub":"90'24","last_deep_scrub_stamp":"2024-04-14T15:11:25.120692+0000","last_clean_scrub_stamp":"2024-04-14T15:13:26.147631+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:54.996017+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"91'130","reported_seq":493,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.832910+0000","last_change":"2024-04-14T15:13:57.350498+0000","last_active":"2024-04-14T15:13:58.832910+0000","last_peered":"2024-04-14T15:13:58.832910+0000","last_clean":"2024-04-14T15:13:58.832910+0000","last_became_active":"2024-04-14T15:09:33.753828+0000","last_became_peered":"2024-04-14T15:09:33.753828+0000","last_unstale":"2024-04-14T15:13:58.832910+0000","last_undegraded":"2024-04-14T15:13:58.832910+0000","last_fullsized":"2024-04-14T15:13:58.832910+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:13:57.350407+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:13:57.350407+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:15:12.064995+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":241,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.831081+0000","last_change":"2024-04-14T15:11:47.627424+0000","last_active":"2024-04-14T15:13:58.831081+0000","last_peered":"2024-04-14T15:13:58.831081+0000","last_clean":"2024-04-14T15:13:58.831081+0000","last_became_active":"2024-04-14T15:09:43.025421+0000","last_became_peered":"2024-04-14T15:09:43.025421+0000","last_unstale":"2024-04-14T15:13:58.831081+0000","last_undegraded":"2024-04-14T15:13:58.831081+0000","last_fullsized":"2024-04-14T15:13:58.831081+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:47.627305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:47.627305+0000","last_clean_scrub_stamp":"2024-04-14T15:11:47.627305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"69'2","reported_seq":191,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.830839+0000","last_change":"2024-04-14T15:13:32.805011+0000","last_active":"2024-04-14T15:13:58.830839+0000","last_peered":"2024-04-14T15:13:58.830839+0000","last_clean":"2024-04-14T15:13:58.830839+0000","last_became_active":"2024-04-14T15:11:57.121809+0000","last_became_peered":"2024-04-14T15:11:57.121809+0000","last_unstale":"2024-04-14T15:13:58.830839+0000","last_undegraded":"2024-04-14T15:13:58.830839+0000","last_fullsized":"2024-04-14T15:13:58.830839+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-04-14T15:13:32.804904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:13:32.804904+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:53.419472+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":228,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.843076+0000","last_change":"2024-04-14T15:13:57.756800+0000","last_active":"2024-04-14T15:13:58.843076+0000","last_peered":"2024-04-14T15:13:58.843076+0000","last_clean":"2024-04-14T15:13:58.843076+0000","last_became_active":"2024-04-14T15:09:49.067031+0000","last_became_peered":"2024-04-14T15:09:49.067031+0000","last_unstale":"2024-04-14T15:13:58.843076+0000","last_undegraded":"2024-04-14T15:13:58.843076+0000","last_fullsized":"2024-04-14T15:13:58.843076+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:57.756721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:13:57.756721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:15:08.683241+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":36,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.844072+0000","last_change":"2024-04-14T15:13:48.891857+0000","last_active":"2024-04-14T15:13:58.844072+0000","last_peered":"2024-04-14T15:13:58.844072+0000","last_clean":"2024-04-14T15:13:58.844072+0000","last_became_active":"2024-04-14T15:13:48.891485+0000","last_became_peered":"2024-04-14T15:13:48.891485+0000","last_unstale":"2024-04-14T15:13:58.844072+0000","last_undegraded":"2024-04-14T15:13:58.844072+0000","last_fullsized":"2024-04-14T15:13:58.844072+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:06.982459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.11","version":"141'4","reported_seq":149,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.828432+0000","last_change":"2024-04-14T15:12:56.402927+0000","last_active":"2024-04-14T15:13:58.828432+0000","last_peered":"2024-04-14T15:13:58.828432+0000","last_clean":"2024-04-14T15:13:58.828432+0000","last_became_active":"2024-04-14T15:12:56.402343+0000","last_became_peered":"2024-04-14T15:12:56.402343+0000","last_unstale":"2024-04-14T15:13:58.828432+0000","last_undegraded":"2024-04-14T15:13:58.828432+0000","last_fullsized":"2024-04-14T15:13:58.828432+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:00.892629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165844,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":24579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":137,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.837799+0000","last_change":"2024-04-14T15:12:56.322582+0000","last_active":"2024-04-14T15:13:58.837799+0000","last_peered":"2024-04-14T15:13:58.837799+0000","last_clean":"2024-04-14T15:13:58.837799+0000","last_became_active":"2024-04-14T15:12:56.315249+0000","last_became_peered":"2024-04-14T15:12:56.315249+0000","last_unstale":"2024-04-14T15:13:58.837799+0000","last_undegraded":"2024-04-14T15:13:58.837799+0000","last_fullsized":"2024-04-14T15:13:58.837799+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:04.341211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":232,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.838782+0000","last_change":"2024-04-14T15:12:39.402534+0000","last_active":"2024-04-14T15:13:58.838782+0000","last_peered":"2024-04-14T15:13:58.838782+0000","last_clean":"2024-04-14T15:13:58.838782+0000","last_became_active":"2024-04-14T15:09:47.049457+0000","last_became_peered":"2024-04-14T15:09:47.049457+0000","last_unstale":"2024-04-14T15:13:58.838782+0000","last_undegraded":"2024-04-14T15:13:58.838782+0000","last_fullsized":"2024-04-14T15:13:58.838782+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:39.402404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:12:39.402404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:07.385043+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"72'13","reported_seq":263,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.830176+0000","last_change":"2024-04-14T15:13:28.340992+0000","last_active":"2024-04-14T15:13:58.830176+0000","last_peered":"2024-04-14T15:13:58.830176+0000","last_clean":"2024-04-14T15:13:58.830176+0000","last_became_active":"2024-04-14T15:09:45.033883+0000","last_became_peered":"2024-04-14T15:09:45.033883+0000","last_unstale":"2024-04-14T15:13:58.830176+0000","last_undegraded":"2024-04-14T15:13:58.830176+0000","last_fullsized":"2024-04-14T15:13:58.830176+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'13","last_scrub_stamp":"2024-04-14T15:13:28.340838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:13:28.340838+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:33.889823+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":241,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.841442+0000","last_change":"2024-04-14T15:12:37.457119+0000","last_active":"2024-04-14T15:13:58.841442+0000","last_peered":"2024-04-14T15:13:58.841442+0000","last_clean":"2024-04-14T15:13:58.841442+0000","last_became_active":"2024-04-14T15:09:43.021188+0000","last_became_peered":"2024-04-14T15:09:43.021188+0000","last_unstale":"2024-04-14T15:13:58.841442+0000","last_undegraded":"2024-04-14T15:13:58.841442+0000","last_fullsized":"2024-04-14T15:13:58.841442+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:37.456999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:12:37.456999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"73'5","reported_seq":500,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.829013+0000","last_change":"2024-04-14T15:12:41.335732+0000","last_active":"2024-04-14T15:13:58.829013+0000","last_peered":"2024-04-14T15:13:58.829013+0000","last_clean":"2024-04-14T15:13:58.829013+0000","last_became_active":"2024-04-14T15:09:43.016544+0000","last_became_peered":"2024-04-14T15:09:43.016544+0000","last_unstale":"2024-04-14T15:13:58.829013+0000","last_undegraded":"2024-04-14T15:13:58.829013+0000","last_fullsized":"2024-04-14T15:13:58.829013+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-04-14T15:12:41.335600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:12:41.335600+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00056542099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":249,"num_read_kb":244,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":228,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.834600+0000","last_change":"2024-04-14T15:13:17.120281+0000","last_active":"2024-04-14T15:13:58.834600+0000","last_peered":"2024-04-14T15:13:58.834600+0000","last_clean":"2024-04-14T15:13:58.834600+0000","last_became_active":"2024-04-14T15:09:49.130307+0000","last_became_peered":"2024-04-14T15:09:49.130307+0000","last_unstale":"2024-04-14T15:13:58.834600+0000","last_undegraded":"2024-04-14T15:13:58.834600+0000","last_fullsized":"2024-04-14T15:13:58.834600+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:17.120174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:13:17.120174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:46.362976+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":47,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.848360+0000","last_change":"2024-04-14T15:13:57.252500+0000","last_active":"2024-04-14T15:13:58.848360+0000","last_peered":"2024-04-14T15:13:58.848360+0000","last_clean":"2024-04-14T15:13:58.848360+0000","last_became_active":"2024-04-14T15:13:57.251818+0000","last_became_peered":"2024-04-14T15:13:57.251818+0000","last_unstale":"2024-04-14T15:13:58.848360+0000","last_undegraded":"2024-04-14T15:13:58.848360+0000","last_fullsized":"2024-04-14T15:13:58.848360+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:16.045505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":137,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.835276+0000","last_change":"2024-04-14T15:12:56.233291+0000","last_active":"2024-04-14T15:13:58.835276+0000","last_peered":"2024-04-14T15:13:58.835276+0000","last_clean":"2024-04-14T15:13:58.835276+0000","last_became_active":"2024-04-14T15:12:56.219428+0000","last_became_peered":"2024-04-14T15:12:56.219428+0000","last_unstale":"2024-04-14T15:13:58.835276+0000","last_undegraded":"2024-04-14T15:13:58.835276+0000","last_fullsized":"2024-04-14T15:13:58.835276+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:01.478401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":137,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.835362+0000","last_change":"2024-04-14T15:12:56.447607+0000","last_active":"2024-04-14T15:13:58.835362+0000","last_peered":"2024-04-14T15:13:58.835362+0000","last_clean":"2024-04-14T15:13:58.835362+0000","last_became_active":"2024-04-14T15:12:56.430017+0000","last_became_peered":"2024-04-14T15:12:56.430017+0000","last_unstale":"2024-04-14T15:13:58.835362+0000","last_undegraded":"2024-04-14T15:13:58.835362+0000","last_fullsized":"2024-04-14T15:13:58.835362+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":233,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.853125+0000","last_change":"2024-04-14T15:13:35.587741+0000","last_active":"2024-04-14T15:13:58.853125+0000","last_peered":"2024-04-14T15:13:58.853125+0000","last_clean":"2024-04-14T15:13:58.853125+0000","last_became_active":"2024-04-14T15:09:47.040185+0000","last_became_peered":"2024-04-14T15:09:47.040185+0000","last_unstale":"2024-04-14T15:13:58.853125+0000","last_undegraded":"2024-04-14T15:13:58.853125+0000","last_fullsized":"2024-04-14T15:13:58.853125+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:35.587624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.587624+0000","last_clean_scrub_stamp":"2024-04-14T15:13:35.587624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:44.743195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"72'28","reported_seq":337,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.833904+0000","last_change":"2024-04-14T15:13:35.350892+0000","last_active":"2024-04-14T15:13:58.833904+0000","last_peered":"2024-04-14T15:13:58.833904+0000","last_clean":"2024-04-14T15:13:58.833904+0000","last_became_active":"2024-04-14T15:09:45.034168+0000","last_became_peered":"2024-04-14T15:09:45.034168+0000","last_unstale":"2024-04-14T15:13:58.833904+0000","last_undegraded":"2024-04-14T15:13:58.833904+0000","last_fullsized":"2024-04-14T15:13:58.833904+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'28","last_scrub_stamp":"2024-04-14T15:13:35.350790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:13:35.350790+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:47.137871+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":241,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.830539+0000","last_change":"2024-04-14T15:13:25.908505+0000","last_active":"2024-04-14T15:13:58.830539+0000","last_peered":"2024-04-14T15:13:58.830539+0000","last_clean":"2024-04-14T15:13:58.830539+0000","last_became_active":"2024-04-14T15:09:43.028362+0000","last_became_peered":"2024-04-14T15:09:43.028362+0000","last_unstale":"2024-04-14T15:13:58.830539+0000","last_undegraded":"2024-04-14T15:13:58.830539+0000","last_fullsized":"2024-04-14T15:13:58.830539+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:25.908435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:13:25.908435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:52.100586+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":167,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.849445+0000","last_change":"2024-04-14T15:12:35.494479+0000","last_active":"2024-04-14T15:13:58.849445+0000","last_peered":"2024-04-14T15:13:58.849445+0000","last_clean":"2024-04-14T15:13:58.849445+0000","last_became_active":"2024-04-14T15:11:57.117900+0000","last_became_peered":"2024-04-14T15:11:57.117900+0000","last_unstale":"2024-04-14T15:13:58.849445+0000","last_undegraded":"2024-04-14T15:13:58.849445+0000","last_fullsized":"2024-04-14T15:13:58.849445+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:35.494389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:12:35.494389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"109'5","reported_seq":164,"reported_epoch":171,"state":"clean+premerge+peered","last_fresh":"2024-04-14T15:13:58.863537+0000","last_change":"2024-04-14T15:13:58.863537+0000","last_active":"2024-04-14T15:13:57.832922+0000","last_peered":"2024-04-14T15:13:58.863537+0000","last_clean":"2024-04-14T15:13:58.863537+0000","last_became_active":"2024-04-14T15:13:48.886403+0000","last_became_peered":"2024-04-14T15:13:58.863167+0000","last_unstale":"2024-04-14T15:13:58.863537+0000","last_undegraded":"2024-04-14T15:13:58.863537+0000","last_fullsized":"2024-04-14T15:13:58.863537+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:24.179974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.10","version":"108'1","reported_seq":138,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.839743+0000","last_change":"2024-04-14T15:12:56.441371+0000","last_active":"2024-04-14T15:13:58.839743+0000","last_peered":"2024-04-14T15:13:58.839743+0000","last_clean":"2024-04-14T15:13:58.839743+0000","last_became_active":"2024-04-14T15:12:56.439533+0000","last_became_peered":"2024-04-14T15:12:56.439533+0000","last_unstale":"2024-04-14T15:13:58.839743+0000","last_undegraded":"2024-04-14T15:13:58.839743+0000","last_fullsized":"2024-04-14T15:13:58.839743+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:18.464342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.b","version":"108'2","reported_seq":41,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.852854+0000","last_change":"2024-04-14T15:13:48.894871+0000","last_active":"2024-04-14T15:13:58.852854+0000","last_peered":"2024-04-14T15:13:58.852854+0000","last_clean":"2024-04-14T15:13:58.852854+0000","last_became_active":"2024-04-14T15:13:48.886255+0000","last_became_peered":"2024-04-14T15:13:48.886255+0000","last_unstale":"2024-04-14T15:13:58.852854+0000","last_undegraded":"2024-04-14T15:13:58.852854+0000","last_fullsized":"2024-04-14T15:13:58.852854+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:15.600990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":137,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.831970+0000","last_change":"2024-04-14T15:12:56.351081+0000","last_active":"2024-04-14T15:13:58.831970+0000","last_peered":"2024-04-14T15:13:58.831970+0000","last_clean":"2024-04-14T15:13:58.831970+0000","last_became_active":"2024-04-14T15:12:56.350703+0000","last_became_peered":"2024-04-14T15:12:56.350703+0000","last_unstale":"2024-04-14T15:13:58.831970+0000","last_undegraded":"2024-04-14T15:13:58.831970+0000","last_fullsized":"2024-04-14T15:13:58.831970+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:14.515903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"72'8","reported_seq":293,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.853014+0000","last_change":"2024-04-14T15:12:20.089824+0000","last_active":"2024-04-14T15:13:58.853014+0000","last_peered":"2024-04-14T15:13:58.853014+0000","last_clean":"2024-04-14T15:13:58.853014+0000","last_became_active":"2024-04-14T15:09:47.047699+0000","last_became_peered":"2024-04-14T15:09:47.047699+0000","last_unstale":"2024-04-14T15:13:58.853014+0000","last_undegraded":"2024-04-14T15:13:58.853014+0000","last_fullsized":"2024-04-14T15:13:58.853014+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:12:20.089708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:12:20.089708+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"72'10","reported_seq":230,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.837471+0000","last_change":"2024-04-14T15:13:22.268455+0000","last_active":"2024-04-14T15:13:58.837471+0000","last_peered":"2024-04-14T15:13:58.837471+0000","last_clean":"2024-04-14T15:13:58.837471+0000","last_became_active":"2024-04-14T15:10:07.764042+0000","last_became_peered":"2024-04-14T15:10:07.764042+0000","last_unstale":"2024-04-14T15:13:58.837471+0000","last_undegraded":"2024-04-14T15:13:58.837471+0000","last_fullsized":"2024-04-14T15:13:58.837471+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-04-14T15:13:22.268301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:13:22.268301+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:40.226570+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":233,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.833757+0000","last_change":"2024-04-14T15:11:42.065911+0000","last_active":"2024-04-14T15:13:58.833757+0000","last_peered":"2024-04-14T15:13:58.833757+0000","last_clean":"2024-04-14T15:13:58.833757+0000","last_became_active":"2024-04-14T15:09:43.022597+0000","last_became_peered":"2024-04-14T15:09:43.022597+0000","last_unstale":"2024-04-14T15:13:58.833757+0000","last_undegraded":"2024-04-14T15:13:58.833757+0000","last_fullsized":"2024-04-14T15:13:58.833757+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:42.065812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:42.065812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":229,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.841496+0000","last_change":"2024-04-14T15:13:22.363505+0000","last_active":"2024-04-14T15:13:58.841496+0000","last_peered":"2024-04-14T15:13:58.841496+0000","last_clean":"2024-04-14T15:13:58.841496+0000","last_became_active":"2024-04-14T15:09:49.066648+0000","last_became_peered":"2024-04-14T15:09:49.066648+0000","last_unstale":"2024-04-14T15:13:58.841496+0000","last_undegraded":"2024-04-14T15:13:58.841496+0000","last_fullsized":"2024-04-14T15:13:58.841496+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:22.363370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:13:22.363370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:28.560720+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.16","version":"108'1","reported_seq":138,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.833005+0000","last_change":"2024-04-14T15:12:56.379069+0000","last_active":"2024-04-14T15:13:58.833005+0000","last_peered":"2024-04-14T15:13:58.833005+0000","last_clean":"2024-04-14T15:13:58.833005+0000","last_became_active":"2024-04-14T15:12:56.378696+0000","last_became_peered":"2024-04-14T15:12:56.378696+0000","last_unstale":"2024-04-14T15:13:58.833005+0000","last_undegraded":"2024-04-14T15:13:58.833005+0000","last_fullsized":"2024-04-14T15:13:58.833005+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:03.117831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.0","version":"109'5","reported_seq":149,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.833101+0000","last_change":"2024-04-14T15:12:56.204809+0000","last_active":"2024-04-14T15:13:58.833101+0000","last_peered":"2024-04-14T15:13:58.833101+0000","last_clean":"2024-04-14T15:13:58.833101+0000","last_became_active":"2024-04-14T15:12:56.204478+0000","last_became_peered":"2024-04-14T15:12:56.204478+0000","last_unstale":"2024-04-14T15:13:58.833101+0000","last_undegraded":"2024-04-14T15:13:58.833101+0000","last_fullsized":"2024-04-14T15:13:58.833101+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:00.718658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":137,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.848744+0000","last_change":"2024-04-14T15:12:56.564619+0000","last_active":"2024-04-14T15:13:58.848744+0000","last_peered":"2024-04-14T15:13:58.848744+0000","last_clean":"2024-04-14T15:13:58.848744+0000","last_became_active":"2024-04-14T15:12:56.563074+0000","last_became_peered":"2024-04-14T15:12:56.563074+0000","last_unstale":"2024-04-14T15:13:58.848744+0000","last_undegraded":"2024-04-14T15:13:58.848744+0000","last_fullsized":"2024-04-14T15:13:58.848744+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:06.308554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":232,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.840477+0000","last_change":"2024-04-14T15:13:28.305324+0000","last_active":"2024-04-14T15:13:58.840477+0000","last_peered":"2024-04-14T15:13:58.840477+0000","last_clean":"2024-04-14T15:13:58.840477+0000","last_became_active":"2024-04-14T15:09:47.049005+0000","last_became_peered":"2024-04-14T15:09:47.049005+0000","last_unstale":"2024-04-14T15:13:58.840477+0000","last_undegraded":"2024-04-14T15:13:58.840477+0000","last_fullsized":"2024-04-14T15:13:58.840477+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:28.305189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:13:28.305189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:33.710433+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"72'12","reported_seq":259,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.838430+0000","last_change":"2024-04-14T15:12:42.459812+0000","last_active":"2024-04-14T15:13:58.838430+0000","last_peered":"2024-04-14T15:13:58.838430+0000","last_clean":"2024-04-14T15:13:58.838430+0000","last_became_active":"2024-04-14T15:09:45.032892+0000","last_became_peered":"2024-04-14T15:09:45.032892+0000","last_unstale":"2024-04-14T15:13:58.838430+0000","last_undegraded":"2024-04-14T15:13:58.838430+0000","last_fullsized":"2024-04-14T15:13:58.838430+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'12","last_scrub_stamp":"2024-04-14T15:12:42.459703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:12:42.459703+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"65'3","reported_seq":258,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.844692+0000","last_change":"2024-04-14T15:13:32.537671+0000","last_active":"2024-04-14T15:13:58.844692+0000","last_peered":"2024-04-14T15:13:58.844692+0000","last_clean":"2024-04-14T15:13:58.844692+0000","last_became_active":"2024-04-14T15:09:43.020490+0000","last_became_peered":"2024-04-14T15:09:43.020490+0000","last_unstale":"2024-04-14T15:13:58.844692+0000","last_undegraded":"2024-04-14T15:13:58.844692+0000","last_fullsized":"2024-04-14T15:13:58.844692+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3","last_scrub_stamp":"2024-04-14T15:13:32.537579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:13:32.537579+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:48.761569+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":528,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":167,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.833657+0000","last_change":"2024-04-14T15:12:26.063496+0000","last_active":"2024-04-14T15:13:58.833657+0000","last_peered":"2024-04-14T15:13:58.833657+0000","last_clean":"2024-04-14T15:13:58.833657+0000","last_became_active":"2024-04-14T15:11:57.118796+0000","last_became_peered":"2024-04-14T15:11:57.118796+0000","last_unstale":"2024-04-14T15:13:58.833657+0000","last_undegraded":"2024-04-14T15:13:58.833657+0000","last_fullsized":"2024-04-14T15:13:58.833657+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:26.063417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:12:26.063417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1d","version":"110'2","reported_seq":142,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.832147+0000","last_change":"2024-04-14T15:12:56.388959+0000","last_active":"2024-04-14T15:13:58.832147+0000","last_peered":"2024-04-14T15:13:58.832147+0000","last_clean":"2024-04-14T15:13:58.832147+0000","last_became_active":"2024-04-14T15:12:56.380053+0000","last_became_peered":"2024-04-14T15:12:56.380053+0000","last_unstale":"2024-04-14T15:13:58.832147+0000","last_undegraded":"2024-04-14T15:13:58.832147+0000","last_fullsized":"2024-04-14T15:13:58.832147+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:01.878750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":137,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.845213+0000","last_change":"2024-04-14T15:12:56.553547+0000","last_active":"2024-04-14T15:13:58.845213+0000","last_peered":"2024-04-14T15:13:58.845213+0000","last_clean":"2024-04-14T15:13:58.845213+0000","last_became_active":"2024-04-14T15:12:56.551177+0000","last_became_peered":"2024-04-14T15:12:56.551177+0000","last_unstale":"2024-04-14T15:13:58.845213+0000","last_undegraded":"2024-04-14T15:13:58.845213+0000","last_fullsized":"2024-04-14T15:13:58.845213+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:04.301369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"72'9","reported_seq":252,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.849041+0000","last_change":"2024-04-14T15:13:37.516095+0000","last_active":"2024-04-14T15:13:58.849041+0000","last_peered":"2024-04-14T15:13:58.849041+0000","last_clean":"2024-04-14T15:13:58.849041+0000","last_became_active":"2024-04-14T15:09:45.180375+0000","last_became_peered":"2024-04-14T15:09:45.180375+0000","last_unstale":"2024-04-14T15:13:58.849041+0000","last_undegraded":"2024-04-14T15:13:58.849041+0000","last_fullsized":"2024-04-14T15:13:58.849041+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-04-14T15:13:37.515984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:13:37.515984+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:15:02.208462+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":240,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.840322+0000","last_change":"2024-04-14T15:12:30.462527+0000","last_active":"2024-04-14T15:13:58.840322+0000","last_peered":"2024-04-14T15:13:58.840322+0000","last_clean":"2024-04-14T15:13:58.840322+0000","last_became_active":"2024-04-14T15:09:43.016788+0000","last_became_peered":"2024-04-14T15:09:43.016788+0000","last_unstale":"2024-04-14T15:13:58.840322+0000","last_undegraded":"2024-04-14T15:13:58.840322+0000","last_fullsized":"2024-04-14T15:13:58.840322+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:30.462398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:12:30.462398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":166,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.830092+0000","last_change":"2024-04-14T15:13:14.401268+0000","last_active":"2024-04-14T15:13:58.830092+0000","last_peered":"2024-04-14T15:13:58.830092+0000","last_clean":"2024-04-14T15:13:58.830092+0000","last_became_active":"2024-04-14T15:11:57.119468+0000","last_became_peered":"2024-04-14T15:11:57.119468+0000","last_unstale":"2024-04-14T15:13:58.830092+0000","last_undegraded":"2024-04-14T15:13:58.830092+0000","last_fullsized":"2024-04-14T15:13:58.830092+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:14.401142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:13:14.401142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:23.973809+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":228,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.833649+0000","last_change":"2024-04-14T15:12:41.389002+0000","last_active":"2024-04-14T15:13:58.833649+0000","last_peered":"2024-04-14T15:13:58.833649+0000","last_clean":"2024-04-14T15:13:58.833649+0000","last_became_active":"2024-04-14T15:09:49.064296+0000","last_became_peered":"2024-04-14T15:09:49.064296+0000","last_unstale":"2024-04-14T15:13:58.833649+0000","last_undegraded":"2024-04-14T15:13:58.833649+0000","last_fullsized":"2024-04-14T15:13:58.833649+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:41.388867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:12:41.388867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:10.014704+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":155,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.833221+0000","last_change":"2024-04-14T15:13:30.531320+0000","last_active":"2024-04-14T15:13:58.833221+0000","last_peered":"2024-04-14T15:13:58.833221+0000","last_clean":"2024-04-14T15:13:58.833221+0000","last_became_active":"2024-04-14T15:12:56.321327+0000","last_became_peered":"2024-04-14T15:12:56.321327+0000","last_unstale":"2024-04-14T15:13:58.833221+0000","last_undegraded":"2024-04-14T15:13:58.833221+0000","last_fullsized":"2024-04-14T15:13:58.833221+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:30.531240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:30.531240+0000","last_clean_scrub_stamp":"2024-04-14T15:13:30.531240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:58.104309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":137,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.839932+0000","last_change":"2024-04-14T15:12:56.340638+0000","last_active":"2024-04-14T15:13:58.839932+0000","last_peered":"2024-04-14T15:13:58.839932+0000","last_clean":"2024-04-14T15:13:58.839932+0000","last_became_active":"2024-04-14T15:12:56.340066+0000","last_became_peered":"2024-04-14T15:12:56.340066+0000","last_unstale":"2024-04-14T15:13:58.839932+0000","last_undegraded":"2024-04-14T15:13:58.839932+0000","last_fullsized":"2024-04-14T15:13:58.839932+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.17","version":"72'6","reported_seq":245,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.843151+0000","last_change":"2024-04-14T15:12:32.412850+0000","last_active":"2024-04-14T15:13:58.843151+0000","last_peered":"2024-04-14T15:13:58.843151+0000","last_clean":"2024-04-14T15:13:58.843151+0000","last_became_active":"2024-04-14T15:09:45.180037+0000","last_became_peered":"2024-04-14T15:09:45.180037+0000","last_unstale":"2024-04-14T15:13:58.843151+0000","last_undegraded":"2024-04-14T15:13:58.843151+0000","last_fullsized":"2024-04-14T15:13:58.843151+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'6","last_scrub_stamp":"2024-04-14T15:12:32.412700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:12:32.412700+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:00.334065+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":240,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.838445+0000","last_change":"2024-04-14T15:12:37.098768+0000","last_active":"2024-04-14T15:13:58.838445+0000","last_peered":"2024-04-14T15:13:58.838445+0000","last_clean":"2024-04-14T15:13:58.838445+0000","last_became_active":"2024-04-14T15:09:43.018987+0000","last_became_peered":"2024-04-14T15:09:43.018987+0000","last_unstale":"2024-04-14T15:13:58.838445+0000","last_undegraded":"2024-04-14T15:13:58.838445+0000","last_fullsized":"2024-04-14T15:13:58.838445+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:37.098686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:12:37.098686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:01.096637+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":232,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.840658+0000","last_change":"2024-04-14T15:12:29.760123+0000","last_active":"2024-04-14T15:13:58.840658+0000","last_peered":"2024-04-14T15:13:58.840658+0000","last_clean":"2024-04-14T15:13:58.840658+0000","last_became_active":"2024-04-14T15:09:47.045349+0000","last_became_peered":"2024-04-14T15:09:47.045349+0000","last_unstale":"2024-04-14T15:13:58.840658+0000","last_undegraded":"2024-04-14T15:13:58.840658+0000","last_fullsized":"2024-04-14T15:13:58.840658+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:29.759997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:10:53.900878+0000","last_clean_scrub_stamp":"2024-04-14T15:12:29.759997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":228,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.834909+0000","last_change":"2024-04-14T15:13:36.173906+0000","last_active":"2024-04-14T15:13:58.834909+0000","last_peered":"2024-04-14T15:13:58.834909+0000","last_clean":"2024-04-14T15:13:58.834909+0000","last_became_active":"2024-04-14T15:09:49.130438+0000","last_became_peered":"2024-04-14T15:09:49.130438+0000","last_unstale":"2024-04-14T15:13:58.834909+0000","last_undegraded":"2024-04-14T15:13:58.834909+0000","last_fullsized":"2024-04-14T15:13:58.834909+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:36.173813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:13:36.173813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:15:03.922490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":155,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.829167+0000","last_change":"2024-04-14T15:13:30.875464+0000","last_active":"2024-04-14T15:13:58.829167+0000","last_peered":"2024-04-14T15:13:58.829167+0000","last_clean":"2024-04-14T15:13:58.829167+0000","last_became_active":"2024-04-14T15:12:56.230594+0000","last_became_peered":"2024-04-14T15:12:56.230594+0000","last_unstale":"2024-04-14T15:13:58.829167+0000","last_undegraded":"2024-04-14T15:13:58.829167+0000","last_fullsized":"2024-04-14T15:13:58.829167+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:30.875353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:30.875353+0000","last_clean_scrub_stamp":"2024-04-14T15:13:30.875353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:35.399282+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":137,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.843439+0000","last_change":"2024-04-14T15:12:56.206105+0000","last_active":"2024-04-14T15:13:58.843439+0000","last_peered":"2024-04-14T15:13:58.843439+0000","last_clean":"2024-04-14T15:13:58.843439+0000","last_became_active":"2024-04-14T15:12:56.203577+0000","last_became_peered":"2024-04-14T15:12:56.203577+0000","last_unstale":"2024-04-14T15:13:58.843439+0000","last_undegraded":"2024-04-14T15:13:58.843439+0000","last_fullsized":"2024-04-14T15:13:58.843439+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:14.215791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"72'10","reported_seq":200,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.834108+0000","last_change":"2024-04-14T15:12:34.297256+0000","last_active":"2024-04-14T15:13:58.834108+0000","last_peered":"2024-04-14T15:13:58.834108+0000","last_clean":"2024-04-14T15:13:58.834108+0000","last_became_active":"2024-04-14T15:11:57.126453+0000","last_became_peered":"2024-04-14T15:11:57.126453+0000","last_unstale":"2024-04-14T15:13:58.834108+0000","last_undegraded":"2024-04-14T15:13:58.834108+0000","last_fullsized":"2024-04-14T15:13:58.834108+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-04-14T15:12:34.297111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:12:34.297111+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":240,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.829217+0000","last_change":"2024-04-14T15:13:33.770490+0000","last_active":"2024-04-14T15:13:58.829217+0000","last_peered":"2024-04-14T15:13:58.829217+0000","last_clean":"2024-04-14T15:13:58.829217+0000","last_became_active":"2024-04-14T15:09:43.027776+0000","last_became_peered":"2024-04-14T15:09:43.027776+0000","last_unstale":"2024-04-14T15:13:58.829217+0000","last_undegraded":"2024-04-14T15:13:58.829217+0000","last_fullsized":"2024-04-14T15:13:58.829217+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:33.770408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:13:33.770408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:53.137349+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"72'8","reported_seq":292,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.839953+0000","last_change":"2024-04-14T15:12:40.796148+0000","last_active":"2024-04-14T15:13:58.839953+0000","last_peered":"2024-04-14T15:13:58.839953+0000","last_clean":"2024-04-14T15:13:58.839953+0000","last_became_active":"2024-04-14T15:09:47.036688+0000","last_became_peered":"2024-04-14T15:09:47.036688+0000","last_unstale":"2024-04-14T15:13:58.839953+0000","last_undegraded":"2024-04-14T15:13:58.839953+0000","last_fullsized":"2024-04-14T15:13:58.839953+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:12:40.796043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:12:40.796043+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":220,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.848834+0000","last_change":"2024-04-14T15:11:40.922318+0000","last_active":"2024-04-14T15:13:58.848834+0000","last_peered":"2024-04-14T15:13:58.848834+0000","last_clean":"2024-04-14T15:13:58.848834+0000","last_became_active":"2024-04-14T15:09:49.066068+0000","last_became_peered":"2024-04-14T15:09:49.066068+0000","last_unstale":"2024-04-14T15:13:58.848834+0000","last_undegraded":"2024-04-14T15:13:58.848834+0000","last_fullsized":"2024-04-14T15:13:58.848834+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:40.922231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:11:40.922231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":155,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.841219+0000","last_change":"2024-04-14T15:13:32.537830+0000","last_active":"2024-04-14T15:13:58.841219+0000","last_peered":"2024-04-14T15:13:58.841219+0000","last_clean":"2024-04-14T15:13:58.841219+0000","last_became_active":"2024-04-14T15:12:56.546929+0000","last_became_peered":"2024-04-14T15:12:56.546929+0000","last_unstale":"2024-04-14T15:13:58.841219+0000","last_undegraded":"2024-04-14T15:13:58.841219+0000","last_fullsized":"2024-04-14T15:13:58.841219+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:32.537745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:32.537745+0000","last_clean_scrub_stamp":"2024-04-14T15:13:32.537745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:38.124845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":137,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.830674+0000","last_change":"2024-04-14T15:12:56.375745+0000","last_active":"2024-04-14T15:13:58.830674+0000","last_peered":"2024-04-14T15:13:58.830674+0000","last_clean":"2024-04-14T15:13:58.830674+0000","last_became_active":"2024-04-14T15:12:56.358025+0000","last_became_peered":"2024-04-14T15:12:56.358025+0000","last_unstale":"2024-04-14T15:13:58.830674+0000","last_undegraded":"2024-04-14T15:13:58.830674+0000","last_fullsized":"2024-04-14T15:13:58.830674+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:10.090074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"72'9","reported_seq":252,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.840100+0000","last_change":"2024-04-14T15:12:32.762712+0000","last_active":"2024-04-14T15:13:58.840100+0000","last_peered":"2024-04-14T15:13:58.840100+0000","last_clean":"2024-04-14T15:13:58.840100+0000","last_became_active":"2024-04-14T15:09:45.178857+0000","last_became_peered":"2024-04-14T15:09:45.178857+0000","last_unstale":"2024-04-14T15:13:58.840100+0000","last_undegraded":"2024-04-14T15:13:58.840100+0000","last_fullsized":"2024-04-14T15:13:58.840100+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-04-14T15:12:32.762612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:12:32.762612+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":240,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.853172+0000","last_change":"2024-04-14T15:12:34.037607+0000","last_active":"2024-04-14T15:13:58.853172+0000","last_peered":"2024-04-14T15:13:58.853172+0000","last_clean":"2024-04-14T15:13:58.853172+0000","last_became_active":"2024-04-14T15:09:43.019301+0000","last_became_peered":"2024-04-14T15:09:43.019301+0000","last_unstale":"2024-04-14T15:13:58.853172+0000","last_undegraded":"2024-04-14T15:13:58.853172+0000","last_fullsized":"2024-04-14T15:13:58.853172+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:34.037482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:12:34.037482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"98'9","reported_seq":219,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.836259+0000","last_change":"2024-04-14T15:12:39.046563+0000","last_active":"2024-04-14T15:13:58.836259+0000","last_peered":"2024-04-14T15:13:58.836259+0000","last_clean":"2024-04-14T15:13:58.836259+0000","last_became_active":"2024-04-14T15:11:57.124325+0000","last_became_peered":"2024-04-14T15:11:57.124325+0000","last_unstale":"2024-04-14T15:13:58.836259+0000","last_undegraded":"2024-04-14T15:13:58.836259+0000","last_fullsized":"2024-04-14T15:13:58.836259+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-04-14T15:12:39.046477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:12:39.046477+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":228,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.845356+0000","last_change":"2024-04-14T15:13:17.622241+0000","last_active":"2024-04-14T15:13:58.845356+0000","last_peered":"2024-04-14T15:13:58.845356+0000","last_clean":"2024-04-14T15:13:58.845356+0000","last_became_active":"2024-04-14T15:09:49.065242+0000","last_became_peered":"2024-04-14T15:09:49.065242+0000","last_unstale":"2024-04-14T15:13:58.845356+0000","last_undegraded":"2024-04-14T15:13:58.845356+0000","last_fullsized":"2024-04-14T15:13:58.845356+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:17.622129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:13:17.622129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:32.752780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1","version":"142'13","reported_seq":179,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.841049+0000","last_change":"2024-04-14T15:13:30.361747+0000","last_active":"2024-04-14T15:13:58.841049+0000","last_peered":"2024-04-14T15:13:58.841049+0000","last_clean":"2024-04-14T15:13:58.841049+0000","last_became_active":"2024-04-14T15:12:56.367401+0000","last_became_peered":"2024-04-14T15:12:56.367401+0000","last_unstale":"2024-04-14T15:13:58.841049+0000","last_undegraded":"2024-04-14T15:13:58.841049+0000","last_fullsized":"2024-04-14T15:13:58.841049+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"142'13","last_scrub_stamp":"2024-04-14T15:13:30.361654+0000","last_deep_scrub":"142'13","last_deep_scrub_stamp":"2024-04-14T15:13:30.361654+0000","last_clean_scrub_stamp":"2024-04-14T15:13:30.361654+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:41.128987+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331735,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":49156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":137,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.834990+0000","last_change":"2024-04-14T15:12:56.389498+0000","last_active":"2024-04-14T15:13:58.834990+0000","last_peered":"2024-04-14T15:13:58.834990+0000","last_clean":"2024-04-14T15:13:58.834990+0000","last_became_active":"2024-04-14T15:12:56.389204+0000","last_became_peered":"2024-04-14T15:12:56.389204+0000","last_unstale":"2024-04-14T15:13:58.834990+0000","last_undegraded":"2024-04-14T15:13:58.834990+0000","last_fullsized":"2024-04-14T15:13:58.834990+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:15.258772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"72'9","reported_seq":252,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.833172+0000","last_change":"2024-04-14T15:13:24.378688+0000","last_active":"2024-04-14T15:13:58.833172+0000","last_peered":"2024-04-14T15:13:58.833172+0000","last_clean":"2024-04-14T15:13:58.833172+0000","last_became_active":"2024-04-14T15:09:45.026363+0000","last_became_peered":"2024-04-14T15:09:45.026363+0000","last_unstale":"2024-04-14T15:13:58.833172+0000","last_undegraded":"2024-04-14T15:13:58.833172+0000","last_fullsized":"2024-04-14T15:13:58.833172+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-04-14T15:13:24.378555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:13:24.378555+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:49.661349+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":240,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.845088+0000","last_change":"2024-04-14T15:12:36.065034+0000","last_active":"2024-04-14T15:13:58.845088+0000","last_peered":"2024-04-14T15:13:58.845088+0000","last_clean":"2024-04-14T15:13:58.845088+0000","last_became_active":"2024-04-14T15:09:43.022480+0000","last_became_peered":"2024-04-14T15:09:43.022480+0000","last_unstale":"2024-04-14T15:13:58.845088+0000","last_undegraded":"2024-04-14T15:13:58.845088+0000","last_fullsized":"2024-04-14T15:13:58.845088+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:36.064950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:12:36.064950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:01.096652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":166,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.856736+0000","last_change":"2024-04-14T15:12:36.029185+0000","last_active":"2024-04-14T15:13:58.856736+0000","last_peered":"2024-04-14T15:13:58.856736+0000","last_clean":"2024-04-14T15:13:58.856736+0000","last_became_active":"2024-04-14T15:11:57.121871+0000","last_became_peered":"2024-04-14T15:11:57.121871+0000","last_unstale":"2024-04-14T15:13:58.856736+0000","last_undegraded":"2024-04-14T15:13:58.856736+0000","last_fullsized":"2024-04-14T15:13:58.856736+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:36.029063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:12:36.029063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":228,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.841325+0000","last_change":"2024-04-14T15:12:31.969664+0000","last_active":"2024-04-14T15:13:58.841325+0000","last_peered":"2024-04-14T15:13:58.841325+0000","last_clean":"2024-04-14T15:13:58.841325+0000","last_became_active":"2024-04-14T15:09:49.059711+0000","last_became_peered":"2024-04-14T15:09:49.059711+0000","last_unstale":"2024-04-14T15:13:58.841325+0000","last_undegraded":"2024-04-14T15:13:58.841325+0000","last_fullsized":"2024-04-14T15:13:58.841325+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:31.969587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:12:31.969587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:00.310087+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1b","version":"109'10","reported_seq":44,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.856641+0000","last_change":"2024-04-14T15:13:48.900074+0000","last_active":"2024-04-14T15:13:58.856641+0000","last_peered":"2024-04-14T15:13:58.856641+0000","last_clean":"2024-04-14T15:13:58.856641+0000","last_became_active":"2024-04-14T15:13:48.886357+0000","last_became_peered":"2024-04-14T15:13:48.886357+0000","last_unstale":"2024-04-14T15:13:58.856641+0000","last_undegraded":"2024-04-14T15:13:58.856641+0000","last_fullsized":"2024-04-14T15:13:58.856641+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:14.150979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"143'4","reported_seq":159,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.833100+0000","last_change":"2024-04-14T15:13:31.016293+0000","last_active":"2024-04-14T15:13:58.833100+0000","last_peered":"2024-04-14T15:13:58.833100+0000","last_clean":"2024-04-14T15:13:58.833100+0000","last_became_active":"2024-04-14T15:12:56.418987+0000","last_became_peered":"2024-04-14T15:12:56.418987+0000","last_unstale":"2024-04-14T15:13:58.833100+0000","last_undegraded":"2024-04-14T15:13:58.833100+0000","last_fullsized":"2024-04-14T15:13:58.833100+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'4","last_scrub_stamp":"2024-04-14T15:13:31.016207+0000","last_deep_scrub":"143'4","last_deep_scrub_stamp":"2024-04-14T15:13:31.016207+0000","last_clean_scrub_stamp":"2024-04-14T15:13:31.016207+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:55.026110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":137,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.844588+0000","last_change":"2024-04-14T15:12:56.406594+0000","last_active":"2024-04-14T15:13:58.844588+0000","last_peered":"2024-04-14T15:13:58.844588+0000","last_clean":"2024-04-14T15:13:58.844588+0000","last_became_active":"2024-04-14T15:12:56.396294+0000","last_became_peered":"2024-04-14T15:12:56.396294+0000","last_unstale":"2024-04-14T15:13:58.844588+0000","last_undegraded":"2024-04-14T15:13:58.844588+0000","last_fullsized":"2024-04-14T15:13:58.844588+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:13.139188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"72'11","reported_seq":255,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.843644+0000","last_change":"2024-04-14T15:13:35.599422+0000","last_active":"2024-04-14T15:13:58.843644+0000","last_peered":"2024-04-14T15:13:58.843644+0000","last_clean":"2024-04-14T15:13:58.843644+0000","last_became_active":"2024-04-14T15:09:45.026065+0000","last_became_peered":"2024-04-14T15:09:45.026065+0000","last_unstale":"2024-04-14T15:13:58.843644+0000","last_undegraded":"2024-04-14T15:13:58.843644+0000","last_fullsized":"2024-04-14T15:13:58.843644+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-04-14T15:13:35.599327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:13:35.599327+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:47.281371+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":240,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.832292+0000","last_change":"2024-04-14T15:13:27.378656+0000","last_active":"2024-04-14T15:13:58.832292+0000","last_peered":"2024-04-14T15:13:58.832292+0000","last_clean":"2024-04-14T15:13:58.832292+0000","last_became_active":"2024-04-14T15:09:43.023583+0000","last_became_peered":"2024-04-14T15:09:43.023583+0000","last_unstale":"2024-04-14T15:13:58.832292+0000","last_undegraded":"2024-04-14T15:13:58.832292+0000","last_fullsized":"2024-04-14T15:13:58.832292+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:27.378503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:13:27.378503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:32.810609+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":224,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.833332+0000","last_change":"2024-04-14T15:11:35.007456+0000","last_active":"2024-04-14T15:13:58.833332+0000","last_peered":"2024-04-14T15:13:58.833332+0000","last_clean":"2024-04-14T15:13:58.833332+0000","last_became_active":"2024-04-14T15:09:47.041683+0000","last_became_peered":"2024-04-14T15:09:47.041683+0000","last_unstale":"2024-04-14T15:13:58.833332+0000","last_undegraded":"2024-04-14T15:13:58.833332+0000","last_fullsized":"2024-04-14T15:13:58.833332+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:35.007228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:11:35.007228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":167,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.845541+0000","last_change":"2024-04-14T15:13:34.547742+0000","last_active":"2024-04-14T15:13:58.845541+0000","last_peered":"2024-04-14T15:13:58.845541+0000","last_clean":"2024-04-14T15:13:58.845541+0000","last_became_active":"2024-04-14T15:11:57.118363+0000","last_became_peered":"2024-04-14T15:11:57.118363+0000","last_unstale":"2024-04-14T15:13:58.845541+0000","last_undegraded":"2024-04-14T15:13:58.845541+0000","last_fullsized":"2024-04-14T15:13:58.845541+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:34.547654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:13:34.547654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:15:03.799838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"109'5","reported_seq":56,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.837630+0000","last_change":"2024-04-14T15:13:48.901184+0000","last_active":"2024-04-14T15:13:58.837630+0000","last_peered":"2024-04-14T15:13:58.837630+0000","last_clean":"2024-04-14T15:13:58.837630+0000","last_became_active":"2024-04-14T15:13:48.890997+0000","last_became_peered":"2024-04-14T15:13:48.890997+0000","last_unstale":"2024-04-14T15:13:58.837630+0000","last_undegraded":"2024-04-14T15:13:58.837630+0000","last_fullsized":"2024-04-14T15:13:58.837630+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:19.894394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":155,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.855161+0000","last_change":"2024-04-14T15:13:33.556959+0000","last_active":"2024-04-14T15:13:58.855161+0000","last_peered":"2024-04-14T15:13:58.855161+0000","last_clean":"2024-04-14T15:13:58.855161+0000","last_became_active":"2024-04-14T15:12:56.567611+0000","last_became_peered":"2024-04-14T15:12:56.567611+0000","last_unstale":"2024-04-14T15:13:58.855161+0000","last_undegraded":"2024-04-14T15:13:58.855161+0000","last_fullsized":"2024-04-14T15:13:58.855161+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:33.556876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:33.556876+0000","last_clean_scrub_stamp":"2024-04-14T15:13:33.556876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:57.827376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":137,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.834347+0000","last_change":"2024-04-14T15:12:56.397674+0000","last_active":"2024-04-14T15:13:58.834347+0000","last_peered":"2024-04-14T15:13:58.834347+0000","last_clean":"2024-04-14T15:13:58.834347+0000","last_became_active":"2024-04-14T15:12:56.387723+0000","last_became_peered":"2024-04-14T15:12:56.387723+0000","last_unstale":"2024-04-14T15:13:58.834347+0000","last_undegraded":"2024-04-14T15:13:58.834347+0000","last_fullsized":"2024-04-14T15:13:58.834347+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:02.309403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.10","version":"72'4","reported_seq":242,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.833466+0000","last_change":"2024-04-14T15:13:31.218945+0000","last_active":"2024-04-14T15:13:58.833466+0000","last_peered":"2024-04-14T15:13:58.833466+0000","last_clean":"2024-04-14T15:13:58.833466+0000","last_became_active":"2024-04-14T15:09:45.038815+0000","last_became_peered":"2024-04-14T15:09:45.038815+0000","last_unstale":"2024-04-14T15:13:58.833466+0000","last_undegraded":"2024-04-14T15:13:58.833466+0000","last_fullsized":"2024-04-14T15:13:58.833466+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4","last_scrub_stamp":"2024-04-14T15:13:31.218855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:13:31.218855+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:31.655479+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":240,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.840091+0000","last_change":"2024-04-14T15:12:28.767429+0000","last_active":"2024-04-14T15:13:58.840091+0000","last_peered":"2024-04-14T15:13:58.840091+0000","last_clean":"2024-04-14T15:13:58.840091+0000","last_became_active":"2024-04-14T15:09:43.026786+0000","last_became_peered":"2024-04-14T15:09:43.026786+0000","last_unstale":"2024-04-14T15:13:58.840091+0000","last_undegraded":"2024-04-14T15:13:58.840091+0000","last_fullsized":"2024-04-14T15:13:58.840091+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:28.767299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:08.712289+0000","last_clean_scrub_stamp":"2024-04-14T15:12:28.767299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":240,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.838532+0000","last_change":"2024-04-14T15:13:37.910269+0000","last_active":"2024-04-14T15:13:58.838532+0000","last_peered":"2024-04-14T15:13:58.838532+0000","last_clean":"2024-04-14T15:13:58.838532+0000","last_became_active":"2024-04-14T15:09:47.041074+0000","last_became_peered":"2024-04-14T15:09:47.041074+0000","last_unstale":"2024-04-14T15:13:58.838532+0000","last_undegraded":"2024-04-14T15:13:58.838532+0000","last_fullsized":"2024-04-14T15:13:58.838532+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:37.910180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:13:37.910180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:49.017818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":228,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.832726+0000","last_change":"2024-04-14T15:13:26.142140+0000","last_active":"2024-04-14T15:13:58.832726+0000","last_peered":"2024-04-14T15:13:58.832726+0000","last_clean":"2024-04-14T15:13:58.832726+0000","last_became_active":"2024-04-14T15:09:49.061359+0000","last_became_peered":"2024-04-14T15:09:49.061359+0000","last_unstale":"2024-04-14T15:13:58.832726+0000","last_undegraded":"2024-04-14T15:13:58.832726+0000","last_fullsized":"2024-04-14T15:13:58.832726+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:26.142060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:13:26.142060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:50.720229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":36,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.828066+0000","last_change":"2024-04-14T15:13:48.885990+0000","last_active":"2024-04-14T15:13:58.828066+0000","last_peered":"2024-04-14T15:13:58.828066+0000","last_clean":"2024-04-14T15:13:58.828066+0000","last_became_active":"2024-04-14T15:13:48.885618+0000","last_became_peered":"2024-04-14T15:13:48.885618+0000","last_unstale":"2024-04-14T15:13:58.828066+0000","last_undegraded":"2024-04-14T15:13:58.828066+0000","last_fullsized":"2024-04-14T15:13:58.828066+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.4","version":"143'5","reported_seq":162,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.833556+0000","last_change":"2024-04-14T15:13:31.287529+0000","last_active":"2024-04-14T15:13:58.833556+0000","last_peered":"2024-04-14T15:13:58.833556+0000","last_clean":"2024-04-14T15:13:58.833556+0000","last_became_active":"2024-04-14T15:12:56.423605+0000","last_became_peered":"2024-04-14T15:12:56.423605+0000","last_unstale":"2024-04-14T15:13:58.833556+0000","last_undegraded":"2024-04-14T15:13:58.833556+0000","last_fullsized":"2024-04-14T15:13:58.833556+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'5","last_scrub_stamp":"2024-04-14T15:13:31.287443+0000","last_deep_scrub":"143'5","last_deep_scrub_stamp":"2024-04-14T15:13:31.287443+0000","last_clean_scrub_stamp":"2024-04-14T15:13:31.287443+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:55.319665+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2800,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":137,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.834857+0000","last_change":"2024-04-14T15:12:56.352827+0000","last_active":"2024-04-14T15:13:58.834857+0000","last_peered":"2024-04-14T15:13:58.834857+0000","last_clean":"2024-04-14T15:13:58.834857+0000","last_became_active":"2024-04-14T15:12:56.352528+0000","last_became_peered":"2024-04-14T15:12:56.352528+0000","last_unstale":"2024-04-14T15:13:58.834857+0000","last_undegraded":"2024-04-14T15:13:58.834857+0000","last_fullsized":"2024-04-14T15:13:58.834857+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:19.096680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"102'16","reported_seq":209,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.833808+0000","last_change":"2024-04-14T15:12:35.999465+0000","last_active":"2024-04-14T15:13:58.833808+0000","last_peered":"2024-04-14T15:13:58.833808+0000","last_clean":"2024-04-14T15:13:58.833808+0000","last_became_active":"2024-04-14T15:11:57.118033+0000","last_became_peered":"2024-04-14T15:11:57.118033+0000","last_unstale":"2024-04-14T15:13:58.833808+0000","last_undegraded":"2024-04-14T15:13:58.833808+0000","last_fullsized":"2024-04-14T15:13:58.833808+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'16","last_scrub_stamp":"2024-04-14T15:12:35.999326+0000","last_deep_scrub":"102'16","last_deep_scrub_stamp":"2024-04-14T15:12:35.999326+0000","last_clean_scrub_stamp":"2024-04-14T15:12:35.999326+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":240,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.841476+0000","last_change":"2024-04-14T15:12:28.117794+0000","last_active":"2024-04-14T15:13:58.841476+0000","last_peered":"2024-04-14T15:13:58.841476+0000","last_clean":"2024-04-14T15:13:58.841476+0000","last_became_active":"2024-04-14T15:09:43.023220+0000","last_became_peered":"2024-04-14T15:09:43.023220+0000","last_unstale":"2024-04-14T15:13:58.841476+0000","last_undegraded":"2024-04-14T15:13:58.841476+0000","last_fullsized":"2024-04-14T15:13:58.841476+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:28.117703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:12:28.117703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:01.096680+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":232,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.832609+0000","last_change":"2024-04-14T15:13:29.219729+0000","last_active":"2024-04-14T15:13:58.832609+0000","last_peered":"2024-04-14T15:13:58.832609+0000","last_clean":"2024-04-14T15:13:58.832609+0000","last_became_active":"2024-04-14T15:09:47.044665+0000","last_became_peered":"2024-04-14T15:09:47.044665+0000","last_unstale":"2024-04-14T15:13:58.832609+0000","last_undegraded":"2024-04-14T15:13:58.832609+0000","last_fullsized":"2024-04-14T15:13:58.832609+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:29.219589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:13:29.219589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:53.096780+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":167,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.856978+0000","last_change":"2024-04-14T15:12:42.042180+0000","last_active":"2024-04-14T15:13:58.856978+0000","last_peered":"2024-04-14T15:13:58.856978+0000","last_clean":"2024-04-14T15:13:58.856978+0000","last_became_active":"2024-04-14T15:11:57.122266+0000","last_became_peered":"2024-04-14T15:11:57.122266+0000","last_unstale":"2024-04-14T15:13:58.856978+0000","last_undegraded":"2024-04-14T15:13:58.856978+0000","last_fullsized":"2024-04-14T15:13:58.856978+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:42.042065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:12:42.042065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:59.693385+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.18","version":"108'2","reported_seq":42,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.835065+0000","last_change":"2024-04-14T15:13:48.897092+0000","last_active":"2024-04-14T15:13:58.835065+0000","last_peered":"2024-04-14T15:13:58.835065+0000","last_clean":"2024-04-14T15:13:58.835065+0000","last_became_active":"2024-04-14T15:13:48.884844+0000","last_became_peered":"2024-04-14T15:13:48.884844+0000","last_unstale":"2024-04-14T15:13:58.835065+0000","last_undegraded":"2024-04-14T15:13:58.835065+0000","last_fullsized":"2024-04-14T15:13:58.835065+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:19.459354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.5","version":"143'4","reported_seq":163,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.848936+0000","last_change":"2024-04-14T15:13:31.592480+0000","last_active":"2024-04-14T15:13:58.848936+0000","last_peered":"2024-04-14T15:13:58.848936+0000","last_clean":"2024-04-14T15:13:58.848936+0000","last_became_active":"2024-04-14T15:12:56.450678+0000","last_became_peered":"2024-04-14T15:12:56.450678+0000","last_unstale":"2024-04-14T15:13:58.848936+0000","last_undegraded":"2024-04-14T15:13:58.848936+0000","last_fullsized":"2024-04-14T15:13:58.848936+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'4","last_scrub_stamp":"2024-04-14T15:13:31.592386+0000","last_deep_scrub":"143'4","last_deep_scrub_stamp":"2024-04-14T15:13:31.592386+0000","last_clean_scrub_stamp":"2024-04-14T15:13:31.592386+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:15:01.588772+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165843,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":24577,"num_write":2,"num_write_kb":24578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":137,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.841146+0000","last_change":"2024-04-14T15:12:56.336958+0000","last_active":"2024-04-14T15:13:58.841146+0000","last_peered":"2024-04-14T15:13:58.841146+0000","last_clean":"2024-04-14T15:13:58.841146+0000","last_became_active":"2024-04-14T15:12:56.335230+0000","last_became_peered":"2024-04-14T15:12:56.335230+0000","last_unstale":"2024-04-14T15:13:58.841146+0000","last_undegraded":"2024-04-14T15:13:58.841146+0000","last_fullsized":"2024-04-14T15:13:58.841146+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:22.112968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"72'10","reported_seq":252,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.858538+0000","last_change":"2024-04-14T15:13:17.411319+0000","last_active":"2024-04-14T15:13:58.858538+0000","last_peered":"2024-04-14T15:13:58.858538+0000","last_clean":"2024-04-14T15:13:58.858538+0000","last_became_active":"2024-04-14T15:09:45.181456+0000","last_became_peered":"2024-04-14T15:09:45.181456+0000","last_unstale":"2024-04-14T15:13:58.858538+0000","last_undegraded":"2024-04-14T15:13:58.858538+0000","last_fullsized":"2024-04-14T15:13:58.858538+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-04-14T15:13:17.411175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:13:17.411175+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:36.912024+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":240,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.828567+0000","last_change":"2024-04-14T15:12:22.784729+0000","last_active":"2024-04-14T15:13:58.828567+0000","last_peered":"2024-04-14T15:13:58.828567+0000","last_clean":"2024-04-14T15:13:58.828567+0000","last_became_active":"2024-04-14T15:09:43.023257+0000","last_became_peered":"2024-04-14T15:09:43.023257+0000","last_unstale":"2024-04-14T15:13:58.828567+0000","last_undegraded":"2024-04-14T15:13:58.828567+0000","last_fullsized":"2024-04-14T15:13:58.828567+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:22.784606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:12:22.784606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"72'8","reported_seq":304,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.841627+0000","last_change":"2024-04-14T15:13:28.103421+0000","last_active":"2024-04-14T15:13:58.841627+0000","last_peered":"2024-04-14T15:13:58.841627+0000","last_clean":"2024-04-14T15:13:58.841627+0000","last_became_active":"2024-04-14T15:09:47.048216+0000","last_became_peered":"2024-04-14T15:09:47.048216+0000","last_unstale":"2024-04-14T15:13:58.841627+0000","last_undegraded":"2024-04-14T15:13:58.841627+0000","last_fullsized":"2024-04-14T15:13:58.841627+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:13:28.103336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:13:28.103336+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:44.872504+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"72'1","reported_seq":229,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.836421+0000","last_change":"2024-04-14T15:12:29.041410+0000","last_active":"2024-04-14T15:13:58.836421+0000","last_peered":"2024-04-14T15:13:58.836421+0000","last_clean":"2024-04-14T15:13:58.836421+0000","last_became_active":"2024-04-14T15:09:49.062051+0000","last_became_peered":"2024-04-14T15:09:49.062051+0000","last_unstale":"2024-04-14T15:13:58.836421+0000","last_undegraded":"2024-04-14T15:13:58.836421+0000","last_fullsized":"2024-04-14T15:13:58.836421+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-04-14T15:12:29.041314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:12:29.041314+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.17","version":"109'5","reported_seq":43,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.838717+0000","last_change":"2024-04-14T15:13:48.893598+0000","last_active":"2024-04-14T15:13:58.838717+0000","last_peered":"2024-04-14T15:13:58.838717+0000","last_clean":"2024-04-14T15:13:58.838717+0000","last_became_active":"2024-04-14T15:13:48.850220+0000","last_became_peered":"2024-04-14T15:13:48.850220+0000","last_unstale":"2024-04-14T15:13:58.838717+0000","last_undegraded":"2024-04-14T15:13:58.838717+0000","last_fullsized":"2024-04-14T15:13:58.838717+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:06.951023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.a","version":"108'1","reported_seq":138,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.837166+0000","last_change":"2024-04-14T15:12:56.397225+0000","last_active":"2024-04-14T15:13:58.837166+0000","last_peered":"2024-04-14T15:13:58.837166+0000","last_clean":"2024-04-14T15:13:58.837166+0000","last_became_active":"2024-04-14T15:12:56.388044+0000","last_became_peered":"2024-04-14T15:12:56.388044+0000","last_unstale":"2024-04-14T15:13:58.837166+0000","last_undegraded":"2024-04-14T15:13:58.837166+0000","last_fullsized":"2024-04-14T15:13:58.837166+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:04.086604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":137,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.841587+0000","last_change":"2024-04-14T15:12:56.548050+0000","last_active":"2024-04-14T15:13:58.841587+0000","last_peered":"2024-04-14T15:13:58.841587+0000","last_clean":"2024-04-14T15:13:58.841587+0000","last_became_active":"2024-04-14T15:12:56.543793+0000","last_became_peered":"2024-04-14T15:12:56.543793+0000","last_unstale":"2024-04-14T15:13:58.841587+0000","last_undegraded":"2024-04-14T15:13:58.841587+0000","last_fullsized":"2024-04-14T15:13:58.841587+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:24.008675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"72'11","reported_seq":274,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.841659+0000","last_change":"2024-04-14T15:13:19.130181+0000","last_active":"2024-04-14T15:13:58.841659+0000","last_peered":"2024-04-14T15:13:58.841659+0000","last_clean":"2024-04-14T15:13:58.841659+0000","last_became_active":"2024-04-14T15:10:07.764183+0000","last_became_peered":"2024-04-14T15:10:07.764183+0000","last_unstale":"2024-04-14T15:13:58.841659+0000","last_undegraded":"2024-04-14T15:13:58.841659+0000","last_fullsized":"2024-04-14T15:13:58.841659+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-04-14T15:13:19.130044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:13:19.130044+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:35.463752+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"65'2","reported_seq":175,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.838646+0000","last_change":"2024-04-14T15:12:26.105766+0000","last_active":"2024-04-14T15:13:58.838646+0000","last_peered":"2024-04-14T15:13:58.838646+0000","last_clean":"2024-04-14T15:13:58.838646+0000","last_became_active":"2024-04-14T15:11:57.118202+0000","last_became_peered":"2024-04-14T15:11:57.118202+0000","last_unstale":"2024-04-14T15:13:58.838646+0000","last_undegraded":"2024-04-14T15:13:58.838646+0000","last_fullsized":"2024-04-14T15:13:58.838646+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-04-14T15:12:26.105658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:12:26.105658+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:01.096619+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":233,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.852077+0000","last_change":"2024-04-14T15:13:28.464940+0000","last_active":"2024-04-14T15:13:58.852077+0000","last_peered":"2024-04-14T15:13:58.852077+0000","last_clean":"2024-04-14T15:13:58.852077+0000","last_became_active":"2024-04-14T15:09:47.045730+0000","last_became_peered":"2024-04-14T15:09:47.045730+0000","last_unstale":"2024-04-14T15:13:58.852077+0000","last_undegraded":"2024-04-14T15:13:58.852077+0000","last_fullsized":"2024-04-14T15:13:58.852077+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:28.464811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:13:28.464811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:38.103587+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":220,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.831943+0000","last_change":"2024-04-14T15:11:21.247809+0000","last_active":"2024-04-14T15:13:58.831943+0000","last_peered":"2024-04-14T15:13:58.831943+0000","last_clean":"2024-04-14T15:13:58.831943+0000","last_became_active":"2024-04-14T15:09:49.057545+0000","last_became_peered":"2024-04-14T15:09:49.057545+0000","last_unstale":"2024-04-14T15:13:58.831943+0000","last_undegraded":"2024-04-14T15:13:58.831943+0000","last_fullsized":"2024-04-14T15:13:58.831943+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:21.247695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:21.247695+0000","last_clean_scrub_stamp":"2024-04-14T15:11:21.247695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"109'5","reported_seq":40,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.847941+0000","last_change":"2024-04-14T15:13:48.889118+0000","last_active":"2024-04-14T15:13:58.847941+0000","last_peered":"2024-04-14T15:13:58.847941+0000","last_clean":"2024-04-14T15:13:58.847941+0000","last_became_active":"2024-04-14T15:13:48.881644+0000","last_became_peered":"2024-04-14T15:13:48.881644+0000","last_unstale":"2024-04-14T15:13:58.847941+0000","last_undegraded":"2024-04-14T15:13:58.847941+0000","last_fullsized":"2024-04-14T15:13:58.847941+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":137,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.835467+0000","last_change":"2024-04-14T15:12:56.367623+0000","last_active":"2024-04-14T15:13:58.835467+0000","last_peered":"2024-04-14T15:13:58.835467+0000","last_clean":"2024-04-14T15:13:58.835467+0000","last_became_active":"2024-04-14T15:12:56.366632+0000","last_became_peered":"2024-04-14T15:12:56.366632+0000","last_unstale":"2024-04-14T15:13:58.835467+0000","last_undegraded":"2024-04-14T15:13:58.835467+0000","last_fullsized":"2024-04-14T15:13:58.835467+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:10.249889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":137,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.831181+0000","last_change":"2024-04-14T15:12:56.407322+0000","last_active":"2024-04-14T15:13:58.831181+0000","last_peered":"2024-04-14T15:13:58.831181+0000","last_clean":"2024-04-14T15:13:58.831181+0000","last_became_active":"2024-04-14T15:12:56.398379+0000","last_became_peered":"2024-04-14T15:12:56.398379+0000","last_unstale":"2024-04-14T15:13:58.831181+0000","last_undegraded":"2024-04-14T15:13:58.831181+0000","last_fullsized":"2024-04-14T15:13:58.831181+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:11.149569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"72'15","reported_seq":261,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.833890+0000","last_change":"2024-04-14T15:13:29.335174+0000","last_active":"2024-04-14T15:13:58.833890+0000","last_peered":"2024-04-14T15:13:58.833890+0000","last_clean":"2024-04-14T15:13:58.833890+0000","last_became_active":"2024-04-14T15:09:45.032887+0000","last_became_peered":"2024-04-14T15:09:45.032887+0000","last_unstale":"2024-04-14T15:13:58.833890+0000","last_undegraded":"2024-04-14T15:13:58.833890+0000","last_fullsized":"2024-04-14T15:13:58.833890+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-04-14T15:13:29.335026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:13:29.335026+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:46.261973+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":243,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.843874+0000","last_change":"2024-04-14T15:13:15.102843+0000","last_active":"2024-04-14T15:13:58.843874+0000","last_peered":"2024-04-14T15:13:58.843874+0000","last_clean":"2024-04-14T15:13:58.843874+0000","last_became_active":"2024-04-14T15:09:43.021494+0000","last_became_peered":"2024-04-14T15:09:43.021494+0000","last_unstale":"2024-04-14T15:13:58.843874+0000","last_undegraded":"2024-04-14T15:13:58.843874+0000","last_fullsized":"2024-04-14T15:13:58.843874+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:15.102763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:13:15.102763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:42.170242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":232,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.832199+0000","last_change":"2024-04-14T15:12:32.347056+0000","last_active":"2024-04-14T15:13:58.832199+0000","last_peered":"2024-04-14T15:13:58.832199+0000","last_clean":"2024-04-14T15:13:58.832199+0000","last_became_active":"2024-04-14T15:09:47.042733+0000","last_became_peered":"2024-04-14T15:09:47.042733+0000","last_unstale":"2024-04-14T15:13:58.832199+0000","last_undegraded":"2024-04-14T15:13:58.832199+0000","last_fullsized":"2024-04-14T15:13:58.832199+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:32.346921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:12:32.346921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":229,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.848650+0000","last_change":"2024-04-14T15:13:12.610073+0000","last_active":"2024-04-14T15:13:58.848650+0000","last_peered":"2024-04-14T15:13:58.848650+0000","last_clean":"2024-04-14T15:13:58.848650+0000","last_became_active":"2024-04-14T15:09:49.127979+0000","last_became_peered":"2024-04-14T15:09:49.127979+0000","last_unstale":"2024-04-14T15:13:58.848650+0000","last_undegraded":"2024-04-14T15:13:58.848650+0000","last_fullsized":"2024-04-14T15:13:58.848650+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:12.609958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:13:12.609958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:19.729963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":149,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.845076+0000","last_change":"2024-04-14T15:13:48.887205+0000","last_active":"2024-04-14T15:13:58.845076+0000","last_peered":"2024-04-14T15:13:58.845076+0000","last_clean":"2024-04-14T15:13:58.845076+0000","last_became_active":"2024-04-14T15:13:48.886877+0000","last_became_peered":"2024-04-14T15:13:48.886877+0000","last_unstale":"2024-04-14T15:13:58.845076+0000","last_undegraded":"2024-04-14T15:13:58.845076+0000","last_fullsized":"2024-04-14T15:13:58.845076+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:11.531176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":155,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.842195+0000","last_change":"2024-04-14T15:13:30.509411+0000","last_active":"2024-04-14T15:13:58.842195+0000","last_peered":"2024-04-14T15:13:58.842195+0000","last_clean":"2024-04-14T15:13:58.842195+0000","last_became_active":"2024-04-14T15:12:56.368566+0000","last_became_peered":"2024-04-14T15:12:56.368566+0000","last_unstale":"2024-04-14T15:13:58.842195+0000","last_undegraded":"2024-04-14T15:13:58.842195+0000","last_fullsized":"2024-04-14T15:13:58.842195+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:30.509291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:30.509291+0000","last_clean_scrub_stamp":"2024-04-14T15:13:30.509291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:57.120611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":137,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.832332+0000","last_change":"2024-04-14T15:12:56.392612+0000","last_active":"2024-04-14T15:13:58.832332+0000","last_peered":"2024-04-14T15:13:58.832332+0000","last_clean":"2024-04-14T15:13:58.832332+0000","last_became_active":"2024-04-14T15:12:56.392352+0000","last_became_peered":"2024-04-14T15:12:56.392352+0000","last_unstale":"2024-04-14T15:13:58.832332+0000","last_undegraded":"2024-04-14T15:13:58.832332+0000","last_fullsized":"2024-04-14T15:13:58.832332+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:04.806971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"72'12","reported_seq":208,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.832575+0000","last_change":"2024-04-14T15:13:18.392665+0000","last_active":"2024-04-14T15:13:58.832575+0000","last_peered":"2024-04-14T15:13:58.832575+0000","last_clean":"2024-04-14T15:13:58.832575+0000","last_became_active":"2024-04-14T15:11:57.118631+0000","last_became_peered":"2024-04-14T15:11:57.118631+0000","last_unstale":"2024-04-14T15:13:58.832575+0000","last_undegraded":"2024-04-14T15:13:58.832575+0000","last_fullsized":"2024-04-14T15:13:58.832575+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'12","last_scrub_stamp":"2024-04-14T15:13:18.392496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:13:18.392496+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:44.659931+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":166,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.833755+0000","last_change":"2024-04-14T15:13:34.177897+0000","last_active":"2024-04-14T15:13:58.833755+0000","last_peered":"2024-04-14T15:13:58.833755+0000","last_clean":"2024-04-14T15:13:58.833755+0000","last_became_active":"2024-04-14T15:11:57.117958+0000","last_became_peered":"2024-04-14T15:11:57.117958+0000","last_unstale":"2024-04-14T15:13:58.833755+0000","last_undegraded":"2024-04-14T15:13:58.833755+0000","last_fullsized":"2024-04-14T15:13:58.833755+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:34.177820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:13:34.177820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:52.828992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":232,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.832077+0000","last_change":"2024-04-14T15:12:39.489602+0000","last_active":"2024-04-14T15:13:58.832077+0000","last_peered":"2024-04-14T15:13:58.832077+0000","last_clean":"2024-04-14T15:13:58.832077+0000","last_became_active":"2024-04-14T15:09:47.048045+0000","last_became_peered":"2024-04-14T15:09:47.048045+0000","last_unstale":"2024-04-14T15:13:58.832077+0000","last_undegraded":"2024-04-14T15:13:58.832077+0000","last_fullsized":"2024-04-14T15:13:58.832077+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:39.489493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:12:39.489493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:04.760526+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":167,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.841738+0000","last_change":"2024-04-14T15:13:22.168812+0000","last_active":"2024-04-14T15:13:58.841738+0000","last_peered":"2024-04-14T15:13:58.841738+0000","last_clean":"2024-04-14T15:13:58.841738+0000","last_became_active":"2024-04-14T15:11:57.118128+0000","last_became_peered":"2024-04-14T15:11:57.118128+0000","last_unstale":"2024-04-14T15:13:58.841738+0000","last_undegraded":"2024-04-14T15:13:58.841738+0000","last_fullsized":"2024-04-14T15:13:58.841738+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:22.168737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:13:22.168737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:44.182086+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"108'2","reported_seq":51,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.844678+0000","last_change":"2024-04-14T15:13:48.896347+0000","last_active":"2024-04-14T15:13:58.844678+0000","last_peered":"2024-04-14T15:13:58.844678+0000","last_clean":"2024-04-14T15:13:58.844678+0000","last_became_active":"2024-04-14T15:13:48.892362+0000","last_became_peered":"2024-04-14T15:13:48.892362+0000","last_unstale":"2024-04-14T15:13:58.844678+0000","last_undegraded":"2024-04-14T15:13:58.844678+0000","last_fullsized":"2024-04-14T15:13:58.844678+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:03.870862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":155,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.831502+0000","last_change":"2024-04-14T15:13:30.306425+0000","last_active":"2024-04-14T15:13:58.831502+0000","last_peered":"2024-04-14T15:13:58.831502+0000","last_clean":"2024-04-14T15:13:58.831502+0000","last_became_active":"2024-04-14T15:12:56.284639+0000","last_became_peered":"2024-04-14T15:12:56.284639+0000","last_unstale":"2024-04-14T15:13:58.831502+0000","last_undegraded":"2024-04-14T15:13:58.831502+0000","last_fullsized":"2024-04-14T15:13:58.831502+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:30.306332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:30.306332+0000","last_clean_scrub_stamp":"2024-04-14T15:13:30.306332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:54.172014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":137,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.832060+0000","last_change":"2024-04-14T15:12:56.388503+0000","last_active":"2024-04-14T15:13:58.832060+0000","last_peered":"2024-04-14T15:13:58.832060+0000","last_clean":"2024-04-14T15:13:58.832060+0000","last_became_active":"2024-04-14T15:12:56.375278+0000","last_became_peered":"2024-04-14T15:12:56.375278+0000","last_unstale":"2024-04-14T15:13:58.832060+0000","last_undegraded":"2024-04-14T15:13:58.832060+0000","last_fullsized":"2024-04-14T15:13:58.832060+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"72'9","reported_seq":252,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.833448+0000","last_change":"2024-04-14T15:13:19.550185+0000","last_active":"2024-04-14T15:13:58.833448+0000","last_peered":"2024-04-14T15:13:58.833448+0000","last_clean":"2024-04-14T15:13:58.833448+0000","last_became_active":"2024-04-14T15:09:45.030093+0000","last_became_peered":"2024-04-14T15:09:45.030093+0000","last_unstale":"2024-04-14T15:13:58.833448+0000","last_undegraded":"2024-04-14T15:13:58.833448+0000","last_fullsized":"2024-04-14T15:13:58.833448+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-04-14T15:13:19.550061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:13:19.550061+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:29.082439+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":210,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.832218+0000","last_change":"2024-04-14T15:11:45.791911+0000","last_active":"2024-04-14T15:13:58.832218+0000","last_peered":"2024-04-14T15:13:58.832218+0000","last_clean":"2024-04-14T15:13:58.832218+0000","last_became_active":"2024-04-14T15:10:07.764086+0000","last_became_peered":"2024-04-14T15:10:07.764086+0000","last_unstale":"2024-04-14T15:13:58.832218+0000","last_undegraded":"2024-04-14T15:13:58.832218+0000","last_fullsized":"2024-04-14T15:13:58.832218+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:45.791810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:45.791810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":232,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.830997+0000","last_change":"2024-04-14T15:12:38.748661+0000","last_active":"2024-04-14T15:13:58.830997+0000","last_peered":"2024-04-14T15:13:58.830997+0000","last_clean":"2024-04-14T15:13:58.830997+0000","last_became_active":"2024-04-14T15:09:47.042384+0000","last_became_peered":"2024-04-14T15:09:47.042384+0000","last_unstale":"2024-04-14T15:13:58.830997+0000","last_undegraded":"2024-04-14T15:13:58.830997+0000","last_fullsized":"2024-04-14T15:13:58.830997+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:38.748520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:12:38.748520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":228,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.842088+0000","last_change":"2024-04-14T15:12:26.053933+0000","last_active":"2024-04-14T15:13:58.842088+0000","last_peered":"2024-04-14T15:13:58.842088+0000","last_clean":"2024-04-14T15:13:58.842088+0000","last_became_active":"2024-04-14T15:09:49.061756+0000","last_became_peered":"2024-04-14T15:09:49.061756+0000","last_unstale":"2024-04-14T15:13:58.842088+0000","last_undegraded":"2024-04-14T15:13:58.842088+0000","last_fullsized":"2024-04-14T15:13:58.842088+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:26.053821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:12:26.053821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.13","version":"109'9","reported_seq":67,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.856216+0000","last_change":"2024-04-14T15:13:48.899652+0000","last_active":"2024-04-14T15:13:58.856216+0000","last_peered":"2024-04-14T15:13:58.856216+0000","last_clean":"2024-04-14T15:13:58.856216+0000","last_became_active":"2024-04-14T15:13:48.890185+0000","last_became_peered":"2024-04-14T15:13:48.890185+0000","last_unstale":"2024-04-14T15:13:58.856216+0000","last_undegraded":"2024-04-14T15:13:58.856216+0000","last_fullsized":"2024-04-14T15:13:58.856216+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:15.459744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.e","version":"142'3","reported_seq":151,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.838426+0000","last_change":"2024-04-14T15:12:56.459419+0000","last_active":"2024-04-14T15:13:58.838426+0000","last_peered":"2024-04-14T15:13:58.838426+0000","last_clean":"2024-04-14T15:13:58.838426+0000","last_became_active":"2024-04-14T15:12:56.451958+0000","last_became_peered":"2024-04-14T15:12:56.451958+0000","last_unstale":"2024-04-14T15:13:58.838426+0000","last_undegraded":"2024-04-14T15:13:58.838426+0000","last_fullsized":"2024-04-14T15:13:58.838426+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:18.857251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497492,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":24577,"num_write":3,"num_write_kb":73731,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":137,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.845286+0000","last_change":"2024-04-14T15:12:56.233385+0000","last_active":"2024-04-14T15:13:58.845286+0000","last_peered":"2024-04-14T15:13:58.845286+0000","last_clean":"2024-04-14T15:13:58.845286+0000","last_became_active":"2024-04-14T15:12:56.232831+0000","last_became_peered":"2024-04-14T15:12:56.232831+0000","last_unstale":"2024-04-14T15:13:58.845286+0000","last_undegraded":"2024-04-14T15:13:58.845286+0000","last_fullsized":"2024-04-14T15:13:58.845286+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:13:59.183825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1b","version":"87'7","reported_seq":248,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.843881+0000","last_change":"2024-04-14T15:13:14.599768+0000","last_active":"2024-04-14T15:13:58.843881+0000","last_peered":"2024-04-14T15:13:58.843881+0000","last_clean":"2024-04-14T15:13:58.843881+0000","last_became_active":"2024-04-14T15:09:45.028667+0000","last_became_peered":"2024-04-14T15:09:45.028667+0000","last_unstale":"2024-04-14T15:13:58.843881+0000","last_undegraded":"2024-04-14T15:13:58.843881+0000","last_fullsized":"2024-04-14T15:13:58.843881+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'7","last_scrub_stamp":"2024-04-14T15:13:14.599663+0000","last_deep_scrub":"87'7","last_deep_scrub_stamp":"2024-04-14T15:11:10.510785+0000","last_clean_scrub_stamp":"2024-04-14T15:13:14.599663+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:31.959592+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":240,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.834713+0000","last_change":"2024-04-14T15:12:28.044001+0000","last_active":"2024-04-14T15:13:58.834713+0000","last_peered":"2024-04-14T15:13:58.834713+0000","last_clean":"2024-04-14T15:13:58.834713+0000","last_became_active":"2024-04-14T15:09:43.021986+0000","last_became_peered":"2024-04-14T15:09:43.021986+0000","last_unstale":"2024-04-14T15:13:58.834713+0000","last_undegraded":"2024-04-14T15:13:58.834713+0000","last_fullsized":"2024-04-14T15:13:58.834713+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:28.043899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:10:43.084820+0000","last_clean_scrub_stamp":"2024-04-14T15:12:28.043899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":232,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.832576+0000","last_change":"2024-04-14T15:13:19.196559+0000","last_active":"2024-04-14T15:13:58.832576+0000","last_peered":"2024-04-14T15:13:58.832576+0000","last_clean":"2024-04-14T15:13:58.832576+0000","last_became_active":"2024-04-14T15:09:47.046517+0000","last_became_peered":"2024-04-14T15:09:47.046517+0000","last_unstale":"2024-04-14T15:13:58.832576+0000","last_undegraded":"2024-04-14T15:13:58.832576+0000","last_fullsized":"2024-04-14T15:13:58.832576+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:19.196454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:13:19.196454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:44.555952+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":228,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.828638+0000","last_change":"2024-04-14T15:12:37.782377+0000","last_active":"2024-04-14T15:13:58.828638+0000","last_peered":"2024-04-14T15:13:58.828638+0000","last_clean":"2024-04-14T15:13:58.828638+0000","last_became_active":"2024-04-14T15:09:49.069590+0000","last_became_peered":"2024-04-14T15:09:49.069590+0000","last_unstale":"2024-04-14T15:13:58.828638+0000","last_undegraded":"2024-04-14T15:13:58.828638+0000","last_fullsized":"2024-04-14T15:13:58.828638+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:37.782277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:12:37.782277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":36,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.828882+0000","last_change":"2024-04-14T15:13:48.881903+0000","last_active":"2024-04-14T15:13:58.828882+0000","last_peered":"2024-04-14T15:13:58.828882+0000","last_clean":"2024-04-14T15:13:58.828882+0000","last_became_active":"2024-04-14T15:13:48.881502+0000","last_became_peered":"2024-04-14T15:13:48.881502+0000","last_unstale":"2024-04-14T15:13:58.828882+0000","last_undegraded":"2024-04-14T15:13:58.828882+0000","last_fullsized":"2024-04-14T15:13:58.828882+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":137,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.852436+0000","last_change":"2024-04-14T15:12:56.467201+0000","last_active":"2024-04-14T15:13:58.852436+0000","last_peered":"2024-04-14T15:13:58.852436+0000","last_clean":"2024-04-14T15:13:58.852436+0000","last_became_active":"2024-04-14T15:12:56.466934+0000","last_became_peered":"2024-04-14T15:12:56.466934+0000","last_unstale":"2024-04-14T15:13:58.852436+0000","last_undegraded":"2024-04-14T15:13:58.852436+0000","last_fullsized":"2024-04-14T15:13:58.852436+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:01.475377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":137,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.838251+0000","last_change":"2024-04-14T15:12:56.190716+0000","last_active":"2024-04-14T15:13:58.838251+0000","last_peered":"2024-04-14T15:13:58.838251+0000","last_clean":"2024-04-14T15:13:58.838251+0000","last_became_active":"2024-04-14T15:12:56.189117+0000","last_became_peered":"2024-04-14T15:12:56.189117+0000","last_unstale":"2024-04-14T15:13:58.838251+0000","last_undegraded":"2024-04-14T15:13:58.838251+0000","last_fullsized":"2024-04-14T15:13:58.838251+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:17.897219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.18","version":"72'9","reported_seq":252,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.837928+0000","last_change":"2024-04-14T15:13:24.538800+0000","last_active":"2024-04-14T15:13:58.837928+0000","last_peered":"2024-04-14T15:13:58.837928+0000","last_clean":"2024-04-14T15:13:58.837928+0000","last_became_active":"2024-04-14T15:09:45.032079+0000","last_became_peered":"2024-04-14T15:09:45.032079+0000","last_unstale":"2024-04-14T15:13:58.837928+0000","last_undegraded":"2024-04-14T15:13:58.837928+0000","last_fullsized":"2024-04-14T15:13:58.837928+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-04-14T15:13:24.538684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:13:24.538684+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:48.098124+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":159,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.851858+0000","last_change":"2024-04-14T15:11:57.122422+0000","last_active":"2024-04-14T15:13:58.851858+0000","last_peered":"2024-04-14T15:13:58.851858+0000","last_clean":"2024-04-14T15:13:58.851858+0000","last_became_active":"2024-04-14T15:11:57.122120+0000","last_became_peered":"2024-04-14T15:11:57.122120+0000","last_unstale":"2024-04-14T15:13:58.851858+0000","last_undegraded":"2024-04-14T15:13:58.851858+0000","last_fullsized":"2024-04-14T15:13:58.851858+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:11:43.589625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:11:43.589625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":232,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.831868+0000","last_change":"2024-04-14T15:13:25.370811+0000","last_active":"2024-04-14T15:13:58.831868+0000","last_peered":"2024-04-14T15:13:58.831868+0000","last_clean":"2024-04-14T15:13:58.831868+0000","last_became_active":"2024-04-14T15:09:47.041486+0000","last_became_peered":"2024-04-14T15:09:47.041486+0000","last_unstale":"2024-04-14T15:13:58.831868+0000","last_undegraded":"2024-04-14T15:13:58.831868+0000","last_fullsized":"2024-04-14T15:13:58.831868+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:25.370679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:13:25.370679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:54.308548+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":220,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.833362+0000","last_change":"2024-04-14T15:12:37.453882+0000","last_active":"2024-04-14T15:13:58.833362+0000","last_peered":"2024-04-14T15:13:58.833362+0000","last_clean":"2024-04-14T15:13:58.833362+0000","last_became_active":"2024-04-14T15:09:49.058904+0000","last_became_peered":"2024-04-14T15:09:49.058904+0000","last_unstale":"2024-04-14T15:13:58.833362+0000","last_undegraded":"2024-04-14T15:13:58.833362+0000","last_fullsized":"2024-04-14T15:13:58.833362+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:37.453778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:12:37.453778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.11","version":"109'7","reported_seq":62,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.838915+0000","last_change":"2024-04-14T15:13:48.899033+0000","last_active":"2024-04-14T15:13:58.838915+0000","last_peered":"2024-04-14T15:13:58.838915+0000","last_clean":"2024-04-14T15:13:58.838915+0000","last_became_active":"2024-04-14T15:13:48.889230+0000","last_became_peered":"2024-04-14T15:13:48.889230+0000","last_unstale":"2024-04-14T15:13:58.838915+0000","last_undegraded":"2024-04-14T15:13:58.838915+0000","last_fullsized":"2024-04-14T15:13:58.838915+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:05.127958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.c","version":"141'5","reported_seq":146,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.829113+0000","last_change":"2024-04-14T15:12:56.396530+0000","last_active":"2024-04-14T15:13:58.829113+0000","last_peered":"2024-04-14T15:13:58.829113+0000","last_clean":"2024-04-14T15:13:58.829113+0000","last_became_active":"2024-04-14T15:12:56.396129+0000","last_became_peered":"2024-04-14T15:12:56.396129+0000","last_unstale":"2024-04-14T15:13:58.829113+0000","last_undegraded":"2024-04-14T15:13:58.829113+0000","last_fullsized":"2024-04-14T15:13:58.829113+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:03.476973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25168635,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":27329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":137,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.840781+0000","last_change":"2024-04-14T15:12:56.422596+0000","last_active":"2024-04-14T15:13:58.840781+0000","last_peered":"2024-04-14T15:13:58.840781+0000","last_clean":"2024-04-14T15:13:58.840781+0000","last_became_active":"2024-04-14T15:12:56.415471+0000","last_became_peered":"2024-04-14T15:12:56.415471+0000","last_unstale":"2024-04-14T15:13:58.840781+0000","last_undegraded":"2024-04-14T15:13:58.840781+0000","last_fullsized":"2024-04-14T15:13:58.840781+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:06.099234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"72'15","reported_seq":210,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.841235+0000","last_change":"2024-04-14T15:13:31.248407+0000","last_active":"2024-04-14T15:13:58.841235+0000","last_peered":"2024-04-14T15:13:58.841235+0000","last_clean":"2024-04-14T15:13:58.841235+0000","last_became_active":"2024-04-14T15:11:57.120945+0000","last_became_peered":"2024-04-14T15:11:57.120945+0000","last_unstale":"2024-04-14T15:13:58.841235+0000","last_undegraded":"2024-04-14T15:13:58.841235+0000","last_fullsized":"2024-04-14T15:13:58.841235+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-04-14T15:13:31.248315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:13:31.248315+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:48.969632+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":241,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.852589+0000","last_change":"2024-04-14T15:13:18.431688+0000","last_active":"2024-04-14T15:13:58.852589+0000","last_peered":"2024-04-14T15:13:58.852589+0000","last_clean":"2024-04-14T15:13:58.852589+0000","last_became_active":"2024-04-14T15:09:43.017351+0000","last_became_peered":"2024-04-14T15:09:43.017351+0000","last_unstale":"2024-04-14T15:13:58.852589+0000","last_undegraded":"2024-04-14T15:13:58.852589+0000","last_fullsized":"2024-04-14T15:13:58.852589+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:18.431571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:13:18.431571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:26.379954+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":224,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.838184+0000","last_change":"2024-04-14T15:12:20.610268+0000","last_active":"2024-04-14T15:13:58.838184+0000","last_peered":"2024-04-14T15:13:58.838184+0000","last_clean":"2024-04-14T15:13:58.838184+0000","last_became_active":"2024-04-14T15:09:47.042538+0000","last_became_peered":"2024-04-14T15:09:47.042538+0000","last_unstale":"2024-04-14T15:13:58.838184+0000","last_undegraded":"2024-04-14T15:13:58.838184+0000","last_fullsized":"2024-04-14T15:13:58.838184+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:20.610159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:12:20.610159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":228,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.839334+0000","last_change":"2024-04-14T15:13:18.143802+0000","last_active":"2024-04-14T15:13:58.839334+0000","last_peered":"2024-04-14T15:13:58.839334+0000","last_clean":"2024-04-14T15:13:58.839334+0000","last_became_active":"2024-04-14T15:09:49.129636+0000","last_became_peered":"2024-04-14T15:09:49.129636+0000","last_unstale":"2024-04-14T15:13:58.839334+0000","last_undegraded":"2024-04-14T15:13:58.839334+0000","last_fullsized":"2024-04-14T15:13:58.839334+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:18.143666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:13:18.143666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:37.337036+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":36,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.832216+0000","last_change":"2024-04-14T15:13:48.889420+0000","last_active":"2024-04-14T15:13:58.832216+0000","last_peered":"2024-04-14T15:13:58.832216+0000","last_clean":"2024-04-14T15:13:58.832216+0000","last_became_active":"2024-04-14T15:13:48.889016+0000","last_became_peered":"2024-04-14T15:13:48.889016+0000","last_unstale":"2024-04-14T15:13:58.832216+0000","last_undegraded":"2024-04-14T15:13:58.832216+0000","last_fullsized":"2024-04-14T15:13:58.832216+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:21.980624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"108'1","reported_seq":138,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-14T15:13:58.838853+0000","last_change":"2024-04-14T15:12:56.396639+0000","last_active":"2024-04-14T15:13:58.838853+0000","last_peered":"2024-04-14T15:13:58.838853+0000","last_clean":"2024-04-14T15:13:58.838853+0000","last_became_active":"2024-04-14T15:12:56.393878+0000","last_became_peered":"2024-04-14T15:12:56.393878+0000","last_unstale":"2024-04-14T15:13:58.838853+0000","last_undegraded":"2024-04-14T15:13:58.838853+0000","last_fullsized":"2024-04-14T15:13:58.838853+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:14:07.801168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":198,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":197,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":192,"num_pg":8,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":12288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":24,"acting":24,"num_store_stats":8},{"poolid":186,"num_pg":32,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":96,"acting":96,"num_store_stats":8},{"poolid":172,"num_pg":32,"stat_sum":{"num_bytes":2281,"num_objects":221,"num_object_clones":141,"num_object_copies":663,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":222,"num_read_kb":222,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":221,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2715648,"data_stored":6843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":221,"ondisk_log_size":221,"up":96,"acting":96,"num_store_stats":8},{"poolid":110,"num_pg":32,"stat_sum":{"num_bytes":5227,"num_objects":505,"num_object_clones":400,"num_object_copies":1515,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":505,"num_whiteouts":0,"num_read":86,"num_read_kb":1,"num_write":510,"num_write_kb":509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6205440,"data_stored":15681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":922,"ondisk_log_size":922,"up":96,"acting":96,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":226499262,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":1011,"num_read_kb":49161,"num_write":1047,"num_write_kb":226712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":679809024,"data_stored":679497786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1066,"ondisk_log_size":1066,"up":96,"acting":96,"num_store_stats":8},{"poolid":13,"num_pg":28,"stat_sum":{"num_bytes":1920,"num_objects":15,"num_object_clones":0,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":93,"num_write_kb":54,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":1152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":6144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88,"ondisk_log_size":88,"up":84,"acting":84,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":249,"num_read_kb":244,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":693,"num_read_kb":450,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":421,"ondisk_log_size":421,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67,"ondisk_log_size":67,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103225,"num_pgs":128,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":57484,"kb_used_data":4084,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93687668,"statfs":{"total":95995035648,"available":95936172032,"internally_reserved":0,"allocated":4182016,"data_stored":2043813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":54655434},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397564,"num_pgs":114,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":178108,"kb_used_data":102500,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93567044,"statfs":{"total":95995035648,"available":95812653056,"internally_reserved":0,"allocated":104960000,"data_stored":102706072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574934,"num_pgs":148,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":153748,"kb_used_data":78132,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93591404,"statfs":{"total":95995035648,"available":95837597696,"internally_reserved":0,"allocated":80007168,"data_stored":76953823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738459,"num_pgs":130,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":248104,"kb_used_data":176664,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93497048,"statfs":{"total":95995035648,"available":95740977152,"internally_reserved":0,"allocated":180903936,"data_stored":178207052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411410,"num_pgs":130,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":198304,"kb_used_data":126848,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93546848,"statfs":{"total":95995035648,"available":95791972352,"internally_reserved":0,"allocated":129892352,"data_stored":127280910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182476,"num_pgs":135,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":52872,"kb_used_data":3624,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93692280,"statfs":{"total":95995035648,"available":95940894720,"internally_reserved":0,"allocated":3710976,"data_stored":1455893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986246,"num_pgs":134,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":177684,"kb_used_data":102068,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93567468,"statfs":{"total":95995035648,"available":95813087232,"internally_reserved":0,"allocated":104517632,"data_stored":102118804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659203,"num_pgs":128,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":181792,"kb_used_data":102016,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":93563360,"statfs":{"total":95995035648,"available":95808880640,"internally_reserved":0,"allocated":104464384,"data_stored":102117638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81656261},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":176222208,"data_stored":176163663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":75554816,"data_stored":75500419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":125886464,"data_stored":125829277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":100687872,"data_stored":100666127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":100716544,"data_stored":100667195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":100708352,"data_stored":100664448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":790528,"data_stored":1994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1024000,"data_stored":2600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1011712,"data_stored":2558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":1409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":724992,"data_stored":1829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":688128,"data_stored":1733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":684032,"data_stored":1725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":724992,"data_stored":1833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":172,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":524288,"data_stored":1324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":172,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":643072,"data_stored":1619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":172,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":253952,"data_stored":643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":172,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":344064,"data_stored":864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":172,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":299008,"data_stored":755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":172,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":172,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":225280,"data_stored":572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":172,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":186,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":186,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":186,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":186,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":186,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":186,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":186,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":186,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":192,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":192,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":192,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":192,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":192,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":192,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":192,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":192,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-14T15:14:04.927 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-04-14T15:14:04.928 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd pool set .mgr pg_num 11 2024-04-14T15:14:04.942 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:14:04.995 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:04.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:04.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:14:05.097 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:14:05.122 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:14:04 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:14:05.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:04 smithi092 ceph-mon[33966]: pgmap v209: 329 pgs: 1 clean+premerge+peered, 328 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-04-14T15:14:05.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:04 smithi092 ceph-mon[33966]: 6.1d scrub starts 2024-04-14T15:14:05.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:04 smithi092 ceph-mon[33966]: 6.1d scrub ok 2024-04-14T15:14:05.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:04 smithi092 ceph-mon[33966]: 6.18 scrub starts 2024-04-14T15:14:05.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:04 smithi092 ceph-mon[33966]: 6.18 scrub ok 2024-04-14T15:14:05.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:04 smithi092 ceph-mon[33966]: from='client.24853 ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi090-93601-1","app":"app1","key":"key1"}]': finished 2024-04-14T15:14:05.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:04 smithi092 ceph-mon[33966]: from='client.48265 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi090-93342-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:05.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:04 smithi092 ceph-mon[33966]: from='client.48259 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi090-93363-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:05.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:04 smithi092 ceph-mon[33966]: osdmap e177: 8 total, 8 up, 8 in 2024-04-14T15:14:05.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:04 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:05.123 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:05.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:05.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:14:05.147 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:14:05.172 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:04 smithi090 ceph-mon[30399]: pgmap v209: 329 pgs: 1 clean+premerge+peered, 328 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-04-14T15:14:05.172 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:04 smithi090 ceph-mon[30399]: 6.1d scrub starts 2024-04-14T15:14:05.172 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:04 smithi090 ceph-mon[30399]: 6.1d scrub ok 2024-04-14T15:14:05.172 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:04 smithi090 ceph-mon[30399]: 6.18 scrub starts 2024-04-14T15:14:05.172 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:04 smithi090 ceph-mon[30399]: 6.18 scrub ok 2024-04-14T15:14:05.172 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:04 smithi090 ceph-mon[30399]: from='client.24853 ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi090-93601-1","app":"app1","key":"key1"}]': finished 2024-04-14T15:14:05.172 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:04 smithi090 ceph-mon[30399]: from='client.48265 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi090-93342-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:05.172 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:04 smithi090 ceph-mon[30399]: from='client.48259 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi090-93363-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:05.173 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:04 smithi090 ceph-mon[30399]: osdmap e177: 8 total, 8 up, 8 in 2024-04-14T15:14:05.173 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:04 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:05.173 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:04 smithi090 ceph-mon[41670]: pgmap v209: 329 pgs: 1 clean+premerge+peered, 328 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-04-14T15:14:05.173 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:04 smithi090 ceph-mon[41670]: 6.1d scrub starts 2024-04-14T15:14:05.173 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:04 smithi090 ceph-mon[41670]: 6.1d scrub ok 2024-04-14T15:14:05.173 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:04 smithi090 ceph-mon[41670]: 6.18 scrub starts 2024-04-14T15:14:05.173 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:04 smithi090 ceph-mon[41670]: 6.18 scrub ok 2024-04-14T15:14:05.173 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:04 smithi090 ceph-mon[41670]: from='client.24853 ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi090-93601-1","app":"app1","key":"key1"}]': finished 2024-04-14T15:14:05.173 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:04 smithi090 ceph-mon[41670]: from='client.48265 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi090-93342-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:05.173 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:04 smithi090 ceph-mon[41670]: from='client.48259 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi090-93363-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:05.173 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:04 smithi090 ceph-mon[41670]: osdmap e177: 8 total, 8 up, 8 in 2024-04-14T15:14:05.174 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:04 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:05.226 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:14:05.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:05.251 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:05.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:14:05.353 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:14:05.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:05.379 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:05.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:14:05.480 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:05.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:05.509 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:05.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:05.610 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:14:05.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:05.638 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:05.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:14:05.739 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:14:05.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:05.766 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:05.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:14:05.828 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: [ OK ] LibRadosSnaps api_misc_pp: [==========] Running 31 tests from 7 test suites. 2024-04-14T15:14:05.828 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [----------] Global test environment set-up. 2024-04-14T15:14:05.828 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2024-04-14T15:14:05.828 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2024-04-14T15:14:05.828 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2024-04-14T15:14:05.829 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-04-14T15:14:05.829 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: 2024-04-14T15:14:05.829 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2024-04-14T15:14:05.829 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: seed 93601 2024-04-14T15:14:05.829 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2024-04-14T15:14:05.829 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (2 ms) 2024-04-14T15:14:05.829 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2024-04-14T15:14:05.829 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (543 ms) 2024-04-14T15:14:05.829 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2024-04-14T15:14:05.829 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (27 ms) 2024-04-14T15:14:05.829 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2024-04-14T15:14:05.829 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (9 ms) 2024-04-14T15:14:05.830 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2024-04-14T15:14:05.830 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (14 ms) 2024-04-14T15:14:05.830 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2024-04-14T15:14:05.830 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (4 ms) 2024-04-14T15:14:05.830 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2024-04-14T15:14:05.830 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (3 ms) 2024-04-14T15:14:05.830 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2024-04-14T15:14:05.830 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (7 ms) 2024-04-14T15:14:05.830 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2024-04-14T15:14:05.830 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (3 ms) 2024-04-14T15:14:05.830 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2024-04-14T15:14:05.830 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (13 ms) 2024-04-14T15:14:05.830 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2024-04-14T15:14:05.831 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (3 ms) 2024-04-14T15:14:05.831 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2024-04-14T15:14:05.831 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (8 ms) 2024-04-14T15:14:05.831 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2024-04-14T15:14:05.831 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (12 ms) 2024-04-14T15:14:05.831 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2024-04-14T15:14:05.831 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: osd_max_attr_size = 0 2024-04-14T15:14:05.831 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2024-04-14T15:14:05.831 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (2096 ms) 2024-04-14T15:14:05.831 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2024-04-14T15:14:05.831 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (540 ms) 2024-04-14T15:14:05.831 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2024-04-14T15:14:05.831 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: waiting for initial deep scrubs... 2024-04-14T15:14:05.832 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: done waiting, doing copies 2024-04-14T15:14:05.832 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: waiting for final deep scrubs... 2024-04-14T15:14:05.832 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: done waiting 2024-04-14T15:14:05.832 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (60808 ms) 2024-04-14T15:14:05.832 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2024-04-14T15:14:05.832 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (4 ms) 2024-04-14T15:14:05.832 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2024-04-14T15:14:05.832 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (3 ms) 2024-04-14T15:14:05.832 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2024-04-14T15:14:05.832 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (4670 ms) 2024-04-14T15:14:05.832 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2024-04-14T15:14:05.832 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2024-04-14T15:14:05.833 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2024-04-14T15:14:05.833 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2024-04-14T15:14:05.833 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2024-04-14T15:14:05.833 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (0 ms) 2024-04-14T15:14:05.869 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:14:05.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:05.868+0000 7ff252d95700 1 -- 172.21.15.90:0/17812388 >> v1:172.21.15.92:6789/0 conn(0x7ff24c10fba0 legacy=0x7ff24c10ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:05.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:05.868+0000 7ff250d91700 1 -- 172.21.15.90:0/17812388 <== mon.1 v1:172.21.15.92:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7ff23c0046d0 con 0x7ff24c10fba0 2024-04-14T15:14:05.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:05.871+0000 7ff252d95700 1 -- 172.21.15.90:0/17812388 shutdown_connections 2024-04-14T15:14:05.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:05.871+0000 7ff252d95700 1 -- 172.21.15.90:0/17812388 >> 172.21.15.90:0/17812388 conn(0x7ff24c06d810 msgr2=0x7ff24c06dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:14:05.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:05.877+0000 7ff252d95700 1 -- 172.21.15.90:0/17812388 shutdown_connections 2024-04-14T15:14:05.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:05.878+0000 7ff252d95700 1 -- 172.21.15.90:0/17812388 wait complete. 2024-04-14T15:14:05.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:05.878+0000 7ff252d95700 1 librados: starting msgr at 2024-04-14T15:14:05.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:05.878+0000 7ff252d95700 1 librados: starting objecter 2024-04-14T15:14:05.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:05.878+0000 7ff252d95700 1 Processor -- start 2024-04-14T15:14:05.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:05.879+0000 7ff252d95700 1 -- start start 2024-04-14T15:14:05.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:05.879+0000 7ff252d95700 1 librados: setting wanted keys 2024-04-14T15:14:05.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:05.879+0000 7ff252d95700 1 librados: calling monclient init 2024-04-14T15:14:05.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:05.879+0000 7ff252d95700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff24c1b6560 con 0x7ff24c11e130 2024-04-14T15:14:05.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:05.879+0000 7ff252d95700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff24c1b7780 con 0x7ff24c110690 2024-04-14T15:14:05.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:05.879+0000 7ff252d95700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff24c1b89c0 con 0x7ff24c10fba0 2024-04-14T15:14:05.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:05.881+0000 7ff251d93700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7ff24c10fba0 0x7ff24c10f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:44254/0 (socket says 172.21.15.90:44254) 2024-04-14T15:14:05.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:05.881+0000 7ff251d93700 1 -- 172.21.15.90:0/3425941736 learned_addr learned my addr 172.21.15.90:0/3425941736 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:14:05.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:05.881+0000 7ff242ffd700 10 client.?.objecter ms_handle_connect 0x7ff24c110690 2024-04-14T15:14:05.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:05.881+0000 7ff242ffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:14:05.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:05.881+0000 7ff242ffd700 1 -- 172.21.15.90:0/3425941736 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3278505867 0 0) 0x7ff24c1b7780 con 0x7ff24c110690 2024-04-14T15:14:05.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:05.881+0000 7ff242ffd700 1 -- 172.21.15.90:0/3425941736 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff238003780 con 0x7ff24c110690 2024-04-14T15:14:05.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:05.881+0000 7ff242ffd700 10 client.?.objecter ms_handle_connect 0x7ff24c11e130 2024-04-14T15:14:05.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:05.881+0000 7ff242ffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:14:05.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:05.882+0000 7ff242ffd700 1 -- 172.21.15.90:0/3425941736 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4086987040 0 0) 0x7ff238003780 con 0x7ff24c110690 2024-04-14T15:14:05.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:05.882+0000 7ff242ffd700 1 -- 172.21.15.90:0/3425941736 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff24c1b7780 con 0x7ff24c110690 2024-04-14T15:14:05.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:05.882+0000 7ff242ffd700 1 -- 172.21.15.90:0/3425941736 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7ff244001c50 con 0x7ff24c110690 2024-04-14T15:14:05.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:05.882+0000 7ff242ffd700 1 -- 172.21.15.90:0/3425941736 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 349701287 0 0) 0x7ff24c1b7780 con 0x7ff24c110690 2024-04-14T15:14:05.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:05.882+0000 7ff242ffd700 1 -- 172.21.15.90:0/3425941736 >> v1:172.21.15.90:6790/0 conn(0x7ff24c10fba0 legacy=0x7ff24c10f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:05.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:05.882+0000 7ff242ffd700 1 -- 172.21.15.90:0/3425941736 >> v1:172.21.15.90:6789/0 conn(0x7ff24c11e130 legacy=0x7ff24c1b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:05.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:05.882+0000 7ff242ffd700 1 -- 172.21.15.90:0/3425941736 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff24c1b9c00 con 0x7ff24c110690 2024-04-14T15:14:05.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:05.882+0000 7ff242ffd700 10 client.?.objecter ms_handle_connect 0x7ff24c10fba0 2024-04-14T15:14:05.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:05.882+0000 7ff242ffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:14:05.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:05.882+0000 7ff252d95700 1 -- 172.21.15.90:0/3425941736 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff24c1b6790 con 0x7ff24c110690 2024-04-14T15:14:05.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:05.882+0000 7ff252d95700 10 client.48292.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:14:05.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:05.882+0000 7ff252d95700 1 -- 172.21.15.90:0/3425941736 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff24c1b6c50 con 0x7ff24c110690 2024-04-14T15:14:05.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:05.882+0000 7ff252d95700 1 librados: init done 2024-04-14T15:14:05.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:05.883+0000 7ff242ffd700 1 -- 172.21.15.90:0/3425941736 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7ff244002010 con 0x7ff24c110690 2024-04-14T15:14:05.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:05.883+0000 7ff242ffd700 1 -- 172.21.15.90:0/3425941736 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7ff244004f00 con 0x7ff24c110690 2024-04-14T15:14:05.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:05.884+0000 7ff252d95700 1 -- 172.21.15.90:0/3425941736 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff230005120 con 0x7ff24c110690 2024-04-14T15:14:05.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:05.885+0000 7ff242ffd700 1 -- 172.21.15.90:0/3425941736 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7ff244003cd0 con 0x7ff24c110690 2024-04-14T15:14:05.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:05.886+0000 7ff242ffd700 1 -- 172.21.15.90:0/3425941736 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(178..178 src has 51..178) ==== 9605+0+0 (unknown 2512187588 0 0) 0x7ff24408a320 con 0x7ff24c110690 2024-04-14T15:14:05.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:05.886+0000 7ff242ffd700 10 client.48292.objecter ms_dispatch 0x7ff24c001550 osd_map(178..178 src has 51..178) 2024-04-14T15:14:05.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:05.886+0000 7ff242ffd700 3 client.48292.objecter handle_osd_map got epochs [178,178] > 0 2024-04-14T15:14:05.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:05.886+0000 7ff242ffd700 3 client.48292.objecter handle_osd_map decoding full epoch 178 2024-04-14T15:14:05.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:05.886+0000 7ff242ffd700 20 client.48292.objecter dump_active .. 0 homeless 2024-04-14T15:14:05.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:05.886+0000 7ff242ffd700 10 client.48292.objecter ms_handle_connect 0x7ff2380713f0 2024-04-14T15:14:05.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:05.890+0000 7ff242ffd700 1 -- 172.21.15.90:0/3425941736 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7ff244056790 con 0x7ff24c110690 2024-04-14T15:14:05.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:05.899 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:05.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:14:06.000 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:14:06.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:06.027 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:06.027 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:14:06.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.117+0000 7ff252d95700 1 -- 172.21.15.90:0/3425941736 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7ff230005410 con 0x7ff24c110690 2024-04-14T15:14:06.128 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:06.129 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:05 smithi090 ceph-mon[41670]: from='client.48271 v1:172.21.15.90:0/1538160927' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:14:06.129 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:05 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:14:06.129 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:05 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-04-14T15:14:06.129 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:05 smithi090 ceph-mon[41670]: 4.17 scrub starts 2024-04-14T15:14:06.129 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:05 smithi090 ceph-mon[41670]: 4.17 scrub ok 2024-04-14T15:14:06.129 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:05 smithi090 ceph-mon[41670]: 6.4 scrub starts 2024-04-14T15:14:06.129 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:05 smithi090 ceph-mon[41670]: 6.4 scrub ok 2024-04-14T15:14:06.130 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:05 smithi090 ceph-mon[41670]: 2.1c scrub starts 2024-04-14T15:14:06.130 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:05 smithi090 ceph-mon[41670]: 2.1c scrub ok 2024-04-14T15:14:06.130 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:05 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "27"}]': finished 2024-04-14T15:14:06.130 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:05 smithi090 ceph-mon[41670]: osdmap e178: 8 total, 8 up, 8 in 2024-04-14T15:14:06.130 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:05 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:06.130 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:05 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/844371530' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi090-93601-24"}]: dispatch 2024-04-14T15:14:06.130 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:05 smithi090 ceph-mon[41670]: from='client.46400 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi090-93601-24"}]: dispatch 2024-04-14T15:14:06.130 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:05 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/844371530' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi090-93601-24"}]: dispatch 2024-04-14T15:14:06.130 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:05 smithi090 ceph-mon[41670]: from='client.46400 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi090-93601-24"}]: dispatch 2024-04-14T15:14:06.130 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:05 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/844371530' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi090-93601-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:14:06.130 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:05 smithi090 ceph-mon[41670]: from='client.46400 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi090-93601-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:14:06.130 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:05 smithi090 ceph-mon[30399]: from='client.48271 v1:172.21.15.90:0/1538160927' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:14:06.131 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:05 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:14:06.131 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:05 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-04-14T15:14:06.131 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:05 smithi090 ceph-mon[30399]: 4.17 scrub starts 2024-04-14T15:14:06.131 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:05 smithi090 ceph-mon[30399]: 4.17 scrub ok 2024-04-14T15:14:06.131 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:05 smithi090 ceph-mon[30399]: 6.4 scrub starts 2024-04-14T15:14:06.131 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:05 smithi090 ceph-mon[30399]: 6.4 scrub ok 2024-04-14T15:14:06.131 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:05 smithi090 ceph-mon[30399]: 2.1c scrub starts 2024-04-14T15:14:06.131 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:05 smithi090 ceph-mon[30399]: 2.1c scrub ok 2024-04-14T15:14:06.131 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:05 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "27"}]': finished 2024-04-14T15:14:06.131 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:05 smithi090 ceph-mon[30399]: osdmap e178: 8 total, 8 up, 8 in 2024-04-14T15:14:06.131 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:05 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:06.131 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:05 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/844371530' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi090-93601-24"}]: dispatch 2024-04-14T15:14:06.131 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:05 smithi090 ceph-mon[30399]: from='client.46400 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi090-93601-24"}]: dispatch 2024-04-14T15:14:06.131 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:05 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/844371530' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi090-93601-24"}]: dispatch 2024-04-14T15:14:06.131 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:05 smithi090 ceph-mon[30399]: from='client.46400 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi090-93601-24"}]: dispatch 2024-04-14T15:14:06.132 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:05 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/844371530' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi090-93601-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:14:06.132 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:05 smithi090 ceph-mon[30399]: from='client.46400 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi090-93601-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:14:06.155 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:06.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:06.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:06.257 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:06.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:06.282 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:06.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:05 smithi092 ceph-mon[33966]: from='client.48271 v1:172.21.15.90:0/1538160927' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:14:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:05 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:14:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:05 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-04-14T15:14:06.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:05 smithi092 ceph-mon[33966]: 4.17 scrub starts 2024-04-14T15:14:06.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:05 smithi092 ceph-mon[33966]: 4.17 scrub ok 2024-04-14T15:14:06.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:05 smithi092 ceph-mon[33966]: 6.4 scrub starts 2024-04-14T15:14:06.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:05 smithi092 ceph-mon[33966]: 6.4 scrub ok 2024-04-14T15:14:06.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:05 smithi092 ceph-mon[33966]: 2.1c scrub starts 2024-04-14T15:14:06.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:05 smithi092 ceph-mon[33966]: 2.1c scrub ok 2024-04-14T15:14:06.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:05 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "27"}]': finished 2024-04-14T15:14:06.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:05 smithi092 ceph-mon[33966]: osdmap e178: 8 total, 8 up, 8 in 2024-04-14T15:14:06.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:05 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:06.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:05 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/844371530' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi090-93601-24"}]: dispatch 2024-04-14T15:14:06.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:05 smithi092 ceph-mon[33966]: from='client.46400 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi090-93601-24"}]: dispatch 2024-04-14T15:14:06.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:05 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/844371530' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi090-93601-24"}]: dispatch 2024-04-14T15:14:06.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:05 smithi092 ceph-mon[33966]: from='client.46400 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi090-93601-24"}]: dispatch 2024-04-14T15:14:06.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:05 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/844371530' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi090-93601-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:14:06.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:05 smithi092 ceph-mon[33966]: from='client.46400 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi090-93601-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:14:06.383 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:14:06.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:06.411 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:06.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:14:06.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.408+0000 7fdad2f78700 1 -- 172.21.15.90:0/973255024 >> v1:172.21.15.90:6789/0 conn(0x7fdacc108870 legacy=0x7fdacc108c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:06.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.408+0000 7fdad2f78700 1 -- 172.21.15.90:0/973255024 shutdown_connections 2024-04-14T15:14:06.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.408+0000 7fdad2f78700 1 -- 172.21.15.90:0/973255024 >> 172.21.15.90:0/973255024 conn(0x7fdacc0fe3a0 msgr2=0x7fdacc1007c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:14:06.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.408+0000 7fdad2f78700 1 -- 172.21.15.90:0/973255024 shutdown_connections 2024-04-14T15:14:06.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.408+0000 7fdad2f78700 1 -- 172.21.15.90:0/973255024 wait complete. 2024-04-14T15:14:06.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.409+0000 7fdad2f78700 1 librados: starting msgr at 2024-04-14T15:14:06.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.409+0000 7fdad2f78700 1 librados: starting objecter 2024-04-14T15:14:06.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.409+0000 7fdad2f78700 1 Processor -- start 2024-04-14T15:14:06.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.409+0000 7fdad2f78700 1 -- start start 2024-04-14T15:14:06.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.409+0000 7fdad2f78700 1 librados: setting wanted keys 2024-04-14T15:14:06.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.409+0000 7fdad2f78700 1 librados: calling monclient init 2024-04-14T15:14:06.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.409+0000 7fdad2f78700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdacc1a7ef0 con 0x7fdacc108870 2024-04-14T15:14:06.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.409+0000 7fdad2f78700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdacc1a90d0 con 0x7fdacc10f2f0 2024-04-14T15:14:06.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.409+0000 7fdad2f78700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdacc1aa2b0 con 0x7fdacc10b720 2024-04-14T15:14:06.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.409+0000 7fdad0d14700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fdacc108870 0x7fdacc10d9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:45340/0 (socket says 172.21.15.90:45340) 2024-04-14T15:14:06.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.409+0000 7fdad0d14700 1 -- 172.21.15.90:0/1451261414 learned_addr learned my addr 172.21.15.90:0/1451261414 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:14:06.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.409+0000 7fdac9ffb700 10 client.?.objecter ms_handle_connect 0x7fdacc108870 2024-04-14T15:14:06.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.409+0000 7fdac9ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:14:06.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.409+0000 7fdac9ffb700 10 client.?.objecter ms_handle_connect 0x7fdacc10b720 2024-04-14T15:14:06.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.409+0000 7fdac9ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:14:06.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.409+0000 7fdac9ffb700 10 client.?.objecter ms_handle_connect 0x7fdacc10f2f0 2024-04-14T15:14:06.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.409+0000 7fdac9ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:14:06.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.409+0000 7fdac9ffb700 1 -- 172.21.15.90:0/1451261414 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2895743776 0 0) 0x7fdacc1a7ef0 con 0x7fdacc108870 2024-04-14T15:14:06.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.409+0000 7fdac9ffb700 1 -- 172.21.15.90:0/1451261414 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdabc0035b0 con 0x7fdacc108870 2024-04-14T15:14:06.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.409+0000 7fdac9ffb700 1 -- 172.21.15.90:0/1451261414 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1277358321 0 0) 0x7fdacc1aa2b0 con 0x7fdacc10b720 2024-04-14T15:14:06.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.409+0000 7fdac9ffb700 1 -- 172.21.15.90:0/1451261414 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdacc1a7ef0 con 0x7fdacc10b720 2024-04-14T15:14:06.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.409+0000 7fdac9ffb700 1 -- 172.21.15.90:0/1451261414 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1034591333 0 0) 0x7fdacc1a90d0 con 0x7fdacc10f2f0 2024-04-14T15:14:06.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.409+0000 7fdac9ffb700 1 -- 172.21.15.90:0/1451261414 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdacc1aa2b0 con 0x7fdacc10f2f0 2024-04-14T15:14:06.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.410+0000 7fdac9ffb700 1 -- 172.21.15.90:0/1451261414 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 126907387 0 0) 0x7fdabc0035b0 con 0x7fdacc108870 2024-04-14T15:14:06.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.410+0000 7fdac9ffb700 1 -- 172.21.15.90:0/1451261414 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdacc1a90d0 con 0x7fdacc108870 2024-04-14T15:14:06.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.410+0000 7fdac9ffb700 1 -- 172.21.15.90:0/1451261414 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fdac00015d0 con 0x7fdacc108870 2024-04-14T15:14:06.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.410+0000 7fdac9ffb700 1 -- 172.21.15.90:0/1451261414 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 260125778 0 0) 0x7fdacc1aa2b0 con 0x7fdacc10f2f0 2024-04-14T15:14:06.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.410+0000 7fdac9ffb700 1 -- 172.21.15.90:0/1451261414 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdabc0035b0 con 0x7fdacc10f2f0 2024-04-14T15:14:06.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.410+0000 7fdac9ffb700 1 -- 172.21.15.90:0/1451261414 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fdac4001bb0 con 0x7fdacc10f2f0 2024-04-14T15:14:06.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.410+0000 7fdac9ffb700 1 -- 172.21.15.90:0/1451261414 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2007739004 0 0) 0x7fdacc1a7ef0 con 0x7fdacc10b720 2024-04-14T15:14:06.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.410+0000 7fdac9ffb700 1 -- 172.21.15.90:0/1451261414 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdacc1aa2b0 con 0x7fdacc10b720 2024-04-14T15:14:06.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.410+0000 7fdac9ffb700 1 -- 172.21.15.90:0/1451261414 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fdab8001830 con 0x7fdacc10b720 2024-04-14T15:14:06.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.410+0000 7fdac9ffb700 1 -- 172.21.15.90:0/1451261414 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 520333559 0 0) 0x7fdacc1a90d0 con 0x7fdacc108870 2024-04-14T15:14:06.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.410+0000 7fdac9ffb700 1 -- 172.21.15.90:0/1451261414 >> v1:172.21.15.90:6790/0 conn(0x7fdacc10b720 legacy=0x7fdacc10e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:06.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.410+0000 7fdac9ffb700 1 -- 172.21.15.90:0/1451261414 >> v1:172.21.15.92:6789/0 conn(0x7fdacc10f2f0 legacy=0x7fdacc1a67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:06.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.410+0000 7fdac9ffb700 1 -- 172.21.15.90:0/1451261414 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdacc1ab490 con 0x7fdacc108870 2024-04-14T15:14:06.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.410+0000 7fdad2f78700 1 -- 172.21.15.90:0/1451261414 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdacc1aa4e0 con 0x7fdacc108870 2024-04-14T15:14:06.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.410+0000 7fdac9ffb700 1 -- 172.21.15.90:0/1451261414 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fdac0001b70 con 0x7fdacc108870 2024-04-14T15:14:06.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.410+0000 7fdac9ffb700 1 -- 172.21.15.90:0/1451261414 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fdac0005020 con 0x7fdacc108870 2024-04-14T15:14:06.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.410+0000 7fdad2f78700 10 client.36459.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:14:06.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.410+0000 7fdad2f78700 1 -- 172.21.15.90:0/1451261414 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fdacc1aa9a0 con 0x7fdacc108870 2024-04-14T15:14:06.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.410+0000 7fdad2f78700 1 librados: init done 2024-04-14T15:14:06.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.411+0000 7fdad2f78700 1 -- 172.21.15.90:0/1451261414 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fdacc103fb0 con 0x7fdacc108870 2024-04-14T15:14:06.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.412+0000 7fdac9ffb700 1 -- 172.21.15.90:0/1451261414 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fdac00052a0 con 0x7fdacc108870 2024-04-14T15:14:06.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.412+0000 7fdac9ffb700 1 -- 172.21.15.90:0/1451261414 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(178..178 src has 51..178) ==== 9605+0+0 (unknown 2512187588 0 0) 0x7fdac008aff0 con 0x7fdacc108870 2024-04-14T15:14:06.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.412+0000 7fdac9ffb700 10 client.36459.objecter ms_dispatch 0x7fdacc001550 osd_map(178..178 src has 51..178) 2024-04-14T15:14:06.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.412+0000 7fdac9ffb700 3 client.36459.objecter handle_osd_map got epochs [178,178] > 0 2024-04-14T15:14:06.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.412+0000 7fdac9ffb700 3 client.36459.objecter handle_osd_map decoding full epoch 178 2024-04-14T15:14:06.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.412+0000 7fdac9ffb700 20 client.36459.objecter dump_active .. 0 homeless 2024-04-14T15:14:06.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.412+0000 7fdac9ffb700 1 -- 172.21.15.90:0/1451261414 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v178) ==== 54+0+16 (unknown 4263576952 0 2420000192) 0x7fdac008b420 con 0x7fdacc108870 2024-04-14T15:14:06.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.412+0000 7fdac9ffb700 10 client.36459.objecter ms_handle_connect 0x7fdabc071300 2024-04-14T15:14:06.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.412+0000 7fdad2f78700 10 client.36459.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:14:06.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.412+0000 7fdad2f78700 15 client.36459.objecter _session_command_op_assign -1 1 2024-04-14T15:14:06.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.412+0000 7fdad2f78700 20 client.36459.objecter _get_session s=0x7fdacc10d470 osd=0 3 2024-04-14T15:14:06.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.412+0000 7fdad2f78700 20 client.36459.objecter put_session s=0x7fdacc10d470 osd=0 3 2024-04-14T15:14:06.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.412+0000 7fdad2f78700 20 client.36459.objecter _get_session s=0x7fdacc10d470 osd=0 3 2024-04-14T15:14:06.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.412+0000 7fdad2f78700 15 client.36459.objecter _session_command_op_remove -1 1 2024-04-14T15:14:06.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.412+0000 7fdad2f78700 20 client.36459.objecter get_session s=0x7fdacc10d470 osd=0 3 2024-04-14T15:14:06.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.412+0000 7fdad2f78700 15 client.36459.objecter _session_command_op_assign 0 1 2024-04-14T15:14:06.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.412+0000 7fdad2f78700 20 client.36459.objecter put_session s=0x7fdacc10d470 osd=0 4 2024-04-14T15:14:06.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.412+0000 7fdad2f78700 10 client.36459.objecter _send_command 1 2024-04-14T15:14:06.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.412+0000 7fdad2f78700 1 -- 172.21.15.90:0/1451261414 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdacc0693d0 con 0x7fdacc1a97b0 2024-04-14T15:14:06.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.413+0000 7fdac9ffb700 10 client.36459.objecter ms_handle_connect 0x7fdacc1a97b0 2024-04-14T15:14:06.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.413+0000 7fdac9ffb700 1 -- 172.21.15.90:0/1451261414 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fdacc0693d0 con 0x7fdacc1a97b0 2024-04-14T15:14:06.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.413+0000 7fdac9ffb700 10 client.36459.objecter ms_dispatch 0x7fdacc001550 command_reply(tid 1: 0 ) 2024-04-14T15:14:06.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.413+0000 7fdac9ffb700 10 client.36459.objecter _finish_command 1 = system:0 2024-04-14T15:14:06.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.413+0000 7fdac9ffb700 20 client.36459.objecter put_session s=0x7fdacc10d470 osd=0 4 2024-04-14T15:14:06.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.413+0000 7fdac9ffb700 15 client.36459.objecter _session_command_op_remove 0 1 2024-04-14T15:14:06.423 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:14:06.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.433+0000 7fdad2f78700 10 client.36459.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:14:06.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.433+0000 7fdad2f78700 15 client.36459.objecter _session_command_op_assign -1 2 2024-04-14T15:14:06.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.433+0000 7fdad2f78700 20 client.36459.objecter _get_session s=0x7fdacc10d470 osd=0 3 2024-04-14T15:14:06.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.433+0000 7fdad2f78700 20 client.36459.objecter put_session s=0x7fdacc10d470 osd=0 3 2024-04-14T15:14:06.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.433+0000 7fdad2f78700 20 client.36459.objecter _get_session s=0x7fdacc10d470 osd=0 3 2024-04-14T15:14:06.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.433+0000 7fdad2f78700 15 client.36459.objecter _session_command_op_remove -1 2 2024-04-14T15:14:06.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.433+0000 7fdad2f78700 20 client.36459.objecter get_session s=0x7fdacc10d470 osd=0 3 2024-04-14T15:14:06.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.433+0000 7fdad2f78700 15 client.36459.objecter _session_command_op_assign 0 2 2024-04-14T15:14:06.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.433+0000 7fdad2f78700 20 client.36459.objecter put_session s=0x7fdacc10d470 osd=0 4 2024-04-14T15:14:06.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.433+0000 7fdad2f78700 10 client.36459.objecter _send_command 2 2024-04-14T15:14:06.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.433+0000 7fdad2f78700 1 -- 172.21.15.90:0/1451261414 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdacc1aaee0 con 0x7fdacc1a97b0 2024-04-14T15:14:06.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.435+0000 7fdac9ffb700 1 -- 172.21.15.90:0/1451261414 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fdacc1aaee0 con 0x7fdacc1a97b0 2024-04-14T15:14:06.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.435+0000 7fdac9ffb700 10 client.36459.objecter ms_dispatch 0x7fdacc001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:14:06.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.435+0000 7fdac9ffb700 10 client.36459.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:14:06.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.435+0000 7fdac9ffb700 20 client.36459.objecter put_session s=0x7fdacc10d470 osd=0 4 2024-04-14T15:14:06.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.435+0000 7fdac9ffb700 15 client.36459.objecter _session_command_op_remove 0 2 2024-04-14T15:14:06.436 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:14:06.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.435+0000 7fdad2f78700 10 client.36459.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:14:06.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.435+0000 7fdad2f78700 15 client.36459.objecter _session_command_op_assign -1 3 2024-04-14T15:14:06.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.435+0000 7fdad2f78700 20 client.36459.objecter _get_session s=0x7fdacc1ab2e0 osd=1 3 2024-04-14T15:14:06.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.435+0000 7fdad2f78700 20 client.36459.objecter put_session s=0x7fdacc1ab2e0 osd=1 3 2024-04-14T15:14:06.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.435+0000 7fdad2f78700 20 client.36459.objecter _get_session s=0x7fdacc1ab2e0 osd=1 3 2024-04-14T15:14:06.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.435+0000 7fdad2f78700 15 client.36459.objecter _session_command_op_remove -1 3 2024-04-14T15:14:06.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.435+0000 7fdad2f78700 20 client.36459.objecter get_session s=0x7fdacc1ab2e0 osd=1 3 2024-04-14T15:14:06.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.435+0000 7fdad2f78700 15 client.36459.objecter _session_command_op_assign 1 3 2024-04-14T15:14:06.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.435+0000 7fdad2f78700 20 client.36459.objecter put_session s=0x7fdacc1ab2e0 osd=1 4 2024-04-14T15:14:06.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.435+0000 7fdad2f78700 10 client.36459.objecter _send_command 3 2024-04-14T15:14:06.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.435+0000 7fdad2f78700 1 -- 172.21.15.90:0/1451261414 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fdacc10ae10 con 0x7fdacc0636c0 2024-04-14T15:14:06.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.435+0000 7fdac9ffb700 10 client.36459.objecter ms_handle_connect 0x7fdacc0636c0 2024-04-14T15:14:06.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.436+0000 7fdac9ffb700 1 -- 172.21.15.90:0/1451261414 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fdacc10ae10 con 0x7fdacc0636c0 2024-04-14T15:14:06.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.436+0000 7fdac9ffb700 10 client.36459.objecter ms_dispatch 0x7fdacc001550 command_reply(tid 3: 0 ) 2024-04-14T15:14:06.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.436+0000 7fdac9ffb700 10 client.36459.objecter _finish_command 3 = system:0 2024-04-14T15:14:06.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.436+0000 7fdac9ffb700 20 client.36459.objecter put_session s=0x7fdacc1ab2e0 osd=1 4 2024-04-14T15:14:06.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.436+0000 7fdac9ffb700 15 client.36459.objecter _session_command_op_remove 1 3 2024-04-14T15:14:06.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.460+0000 7fdad2f78700 10 client.36459.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:14:06.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.460+0000 7fdad2f78700 15 client.36459.objecter _session_command_op_assign -1 4 2024-04-14T15:14:06.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.460+0000 7fdad2f78700 20 client.36459.objecter _get_session s=0x7fdacc1ab2e0 osd=1 3 2024-04-14T15:14:06.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.460+0000 7fdad2f78700 20 client.36459.objecter put_session s=0x7fdacc1ab2e0 osd=1 3 2024-04-14T15:14:06.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.460+0000 7fdad2f78700 20 client.36459.objecter _get_session s=0x7fdacc1ab2e0 osd=1 3 2024-04-14T15:14:06.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.460+0000 7fdad2f78700 15 client.36459.objecter _session_command_op_remove -1 4 2024-04-14T15:14:06.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.460+0000 7fdad2f78700 20 client.36459.objecter get_session s=0x7fdacc1ab2e0 osd=1 3 2024-04-14T15:14:06.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.460+0000 7fdad2f78700 15 client.36459.objecter _session_command_op_assign 1 4 2024-04-14T15:14:06.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.460+0000 7fdad2f78700 20 client.36459.objecter put_session s=0x7fdacc1ab2e0 osd=1 4 2024-04-14T15:14:06.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.460+0000 7fdad2f78700 10 client.36459.objecter _send_command 4 2024-04-14T15:14:06.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.460+0000 7fdad2f78700 1 -- 172.21.15.90:0/1451261414 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdacc063d90 con 0x7fdacc0636c0 2024-04-14T15:14:06.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.462+0000 7fdac9ffb700 1 -- 172.21.15.90:0/1451261414 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fdacc063d90 con 0x7fdacc0636c0 2024-04-14T15:14:06.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.462+0000 7fdac9ffb700 10 client.36459.objecter ms_dispatch 0x7fdacc001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:14:06.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.462+0000 7fdac9ffb700 10 client.36459.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:14:06.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.462+0000 7fdac9ffb700 20 client.36459.objecter put_session s=0x7fdacc1ab2e0 osd=1 4 2024-04-14T15:14:06.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.462+0000 7fdac9ffb700 15 client.36459.objecter _session_command_op_remove 1 4 2024-04-14T15:14:06.463 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:14:06.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.462+0000 7fdad2f78700 10 client.36459.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:14:06.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.462+0000 7fdad2f78700 15 client.36459.objecter _session_command_op_assign -1 5 2024-04-14T15:14:06.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.462+0000 7fdad2f78700 20 client.36459.objecter _get_session s=0x7fdacc063140 osd=2 3 2024-04-14T15:14:06.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.462+0000 7fdad2f78700 20 client.36459.objecter put_session s=0x7fdacc063140 osd=2 3 2024-04-14T15:14:06.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.462+0000 7fdad2f78700 20 client.36459.objecter _get_session s=0x7fdacc063140 osd=2 3 2024-04-14T15:14:06.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.462+0000 7fdad2f78700 15 client.36459.objecter _session_command_op_remove -1 5 2024-04-14T15:14:06.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.462+0000 7fdad2f78700 20 client.36459.objecter get_session s=0x7fdacc063140 osd=2 3 2024-04-14T15:14:06.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.462+0000 7fdad2f78700 15 client.36459.objecter _session_command_op_assign 2 5 2024-04-14T15:14:06.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.462+0000 7fdad2f78700 20 client.36459.objecter put_session s=0x7fdacc063140 osd=2 4 2024-04-14T15:14:06.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.462+0000 7fdad2f78700 10 client.36459.objecter _send_command 5 2024-04-14T15:14:06.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.462+0000 7fdad2f78700 1 -- 172.21.15.90:0/1451261414 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fdacc02db00 con 0x7fdacc0632b0 2024-04-14T15:14:06.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.463+0000 7fdac9ffb700 10 client.36459.objecter ms_handle_connect 0x7fdacc0632b0 2024-04-14T15:14:06.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.463+0000 7fdac9ffb700 1 -- 172.21.15.90:0/1451261414 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fdacc02db00 con 0x7fdacc0632b0 2024-04-14T15:14:06.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.463+0000 7fdac9ffb700 10 client.36459.objecter ms_dispatch 0x7fdacc001550 command_reply(tid 5: 0 ) 2024-04-14T15:14:06.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.463+0000 7fdac9ffb700 10 client.36459.objecter _finish_command 5 = system:0 2024-04-14T15:14:06.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.463+0000 7fdac9ffb700 20 client.36459.objecter put_session s=0x7fdacc063140 osd=2 4 2024-04-14T15:14:06.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.463+0000 7fdac9ffb700 15 client.36459.objecter _session_command_op_remove 2 5 2024-04-14T15:14:06.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.479+0000 7fdad2f78700 10 client.36459.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:14:06.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.479+0000 7fdad2f78700 15 client.36459.objecter _session_command_op_assign -1 6 2024-04-14T15:14:06.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.479+0000 7fdad2f78700 20 client.36459.objecter _get_session s=0x7fdacc063140 osd=2 3 2024-04-14T15:14:06.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.479+0000 7fdad2f78700 20 client.36459.objecter put_session s=0x7fdacc063140 osd=2 3 2024-04-14T15:14:06.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.479+0000 7fdad2f78700 20 client.36459.objecter _get_session s=0x7fdacc063140 osd=2 3 2024-04-14T15:14:06.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.479+0000 7fdad2f78700 15 client.36459.objecter _session_command_op_remove -1 6 2024-04-14T15:14:06.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.479+0000 7fdad2f78700 20 client.36459.objecter get_session s=0x7fdacc063140 osd=2 3 2024-04-14T15:14:06.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.479+0000 7fdad2f78700 15 client.36459.objecter _session_command_op_assign 2 6 2024-04-14T15:14:06.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.479+0000 7fdad2f78700 20 client.36459.objecter put_session s=0x7fdacc063140 osd=2 4 2024-04-14T15:14:06.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.479+0000 7fdad2f78700 10 client.36459.objecter _send_command 6 2024-04-14T15:14:06.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.479+0000 7fdad2f78700 1 -- 172.21.15.90:0/1451261414 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdacc1b3f90 con 0x7fdacc0632b0 2024-04-14T15:14:06.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.481+0000 7fdac9ffb700 1 -- 172.21.15.90:0/1451261414 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fdacc1b3f90 con 0x7fdacc0632b0 2024-04-14T15:14:06.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.481+0000 7fdac9ffb700 10 client.36459.objecter ms_dispatch 0x7fdacc001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:14:06.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.481+0000 7fdac9ffb700 10 client.36459.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:14:06.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.481+0000 7fdac9ffb700 20 client.36459.objecter put_session s=0x7fdacc063140 osd=2 4 2024-04-14T15:14:06.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.481+0000 7fdac9ffb700 15 client.36459.objecter _session_command_op_remove 2 6 2024-04-14T15:14:06.484 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:14:06.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.481+0000 7fdad2f78700 10 client.36459.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:14:06.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.481+0000 7fdad2f78700 15 client.36459.objecter _session_command_op_assign -1 7 2024-04-14T15:14:06.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.481+0000 7fdad2f78700 20 client.36459.objecter _get_session s=0x7fdacc02d070 osd=3 3 2024-04-14T15:14:06.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.481+0000 7fdad2f78700 20 client.36459.objecter put_session s=0x7fdacc02d070 osd=3 3 2024-04-14T15:14:06.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.481+0000 7fdad2f78700 20 client.36459.objecter _get_session s=0x7fdacc02d070 osd=3 3 2024-04-14T15:14:06.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.481+0000 7fdad2f78700 15 client.36459.objecter _session_command_op_remove -1 7 2024-04-14T15:14:06.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.481+0000 7fdad2f78700 20 client.36459.objecter get_session s=0x7fdacc02d070 osd=3 3 2024-04-14T15:14:06.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.481+0000 7fdad2f78700 15 client.36459.objecter _session_command_op_assign 3 7 2024-04-14T15:14:06.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.481+0000 7fdad2f78700 20 client.36459.objecter put_session s=0x7fdacc02d070 osd=3 4 2024-04-14T15:14:06.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.481+0000 7fdad2f78700 10 client.36459.objecter _send_command 7 2024-04-14T15:14:06.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.481+0000 7fdad2f78700 1 -- 172.21.15.90:0/1451261414 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fdacc1b03c0 con 0x7fdacc02d1e0 2024-04-14T15:14:06.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.481+0000 7fdac9ffb700 10 client.36459.objecter ms_handle_connect 0x7fdacc02d1e0 2024-04-14T15:14:06.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.482+0000 7fdac9ffb700 1 -- 172.21.15.90:0/1451261414 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fdacc1b03c0 con 0x7fdacc02d1e0 2024-04-14T15:14:06.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.482+0000 7fdac9ffb700 10 client.36459.objecter ms_dispatch 0x7fdacc001550 command_reply(tid 7: 0 ) 2024-04-14T15:14:06.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.482+0000 7fdac9ffb700 10 client.36459.objecter _finish_command 7 = system:0 2024-04-14T15:14:06.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.482+0000 7fdac9ffb700 20 client.36459.objecter put_session s=0x7fdacc02d070 osd=3 4 2024-04-14T15:14:06.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.482+0000 7fdac9ffb700 15 client.36459.objecter _session_command_op_remove 3 7 2024-04-14T15:14:06.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.498+0000 7fdad2f78700 10 client.36459.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:14:06.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.498+0000 7fdad2f78700 15 client.36459.objecter _session_command_op_assign -1 8 2024-04-14T15:14:06.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.498+0000 7fdad2f78700 20 client.36459.objecter _get_session s=0x7fdacc02d070 osd=3 3 2024-04-14T15:14:06.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.498+0000 7fdad2f78700 20 client.36459.objecter put_session s=0x7fdacc02d070 osd=3 3 2024-04-14T15:14:06.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.498+0000 7fdad2f78700 20 client.36459.objecter _get_session s=0x7fdacc02d070 osd=3 3 2024-04-14T15:14:06.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.498+0000 7fdad2f78700 15 client.36459.objecter _session_command_op_remove -1 8 2024-04-14T15:14:06.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.498+0000 7fdad2f78700 20 client.36459.objecter get_session s=0x7fdacc02d070 osd=3 3 2024-04-14T15:14:06.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.498+0000 7fdad2f78700 15 client.36459.objecter _session_command_op_assign 3 8 2024-04-14T15:14:06.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.498+0000 7fdad2f78700 20 client.36459.objecter put_session s=0x7fdacc02d070 osd=3 4 2024-04-14T15:14:06.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.498+0000 7fdad2f78700 10 client.36459.objecter _send_command 8 2024-04-14T15:14:06.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.498+0000 7fdad2f78700 1 -- 172.21.15.90:0/1451261414 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdacc1b03c0 con 0x7fdacc02d1e0 2024-04-14T15:14:06.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.500+0000 7fdac9ffb700 1 -- 172.21.15.90:0/1451261414 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fdacc1b03c0 con 0x7fdacc02d1e0 2024-04-14T15:14:06.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.500+0000 7fdac9ffb700 10 client.36459.objecter ms_dispatch 0x7fdacc001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:14:06.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.500+0000 7fdac9ffb700 10 client.36459.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:14:06.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.500+0000 7fdac9ffb700 20 client.36459.objecter put_session s=0x7fdacc02d070 osd=3 4 2024-04-14T15:14:06.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.500+0000 7fdac9ffb700 15 client.36459.objecter _session_command_op_remove 3 8 2024-04-14T15:14:06.508 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:14:06.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.500+0000 7fdad2f78700 10 client.36459.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:14:06.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.500+0000 7fdad2f78700 15 client.36459.objecter _session_command_op_assign -1 9 2024-04-14T15:14:06.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.500+0000 7fdad2f78700 20 client.36459.objecter _get_session s=0x7fdacc1b03c0 osd=4 3 2024-04-14T15:14:06.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.500+0000 7fdad2f78700 20 client.36459.objecter put_session s=0x7fdacc1b03c0 osd=4 3 2024-04-14T15:14:06.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.500+0000 7fdad2f78700 20 client.36459.objecter _get_session s=0x7fdacc1b03c0 osd=4 3 2024-04-14T15:14:06.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.500+0000 7fdad2f78700 15 client.36459.objecter _session_command_op_remove -1 9 2024-04-14T15:14:06.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.500+0000 7fdad2f78700 20 client.36459.objecter get_session s=0x7fdacc1b03c0 osd=4 3 2024-04-14T15:14:06.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.500+0000 7fdad2f78700 15 client.36459.objecter _session_command_op_assign 4 9 2024-04-14T15:14:06.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.500+0000 7fdad2f78700 20 client.36459.objecter put_session s=0x7fdacc1b03c0 osd=4 4 2024-04-14T15:14:06.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.500+0000 7fdad2f78700 10 client.36459.objecter _send_command 9 2024-04-14T15:14:06.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.500+0000 7fdad2f78700 1 -- 172.21.15.90:0/1451261414 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fdacc1bee70 con 0x7fdacc0751f0 2024-04-14T15:14:06.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.501+0000 7fdac9ffb700 10 client.36459.objecter ms_handle_connect 0x7fdacc0751f0 2024-04-14T15:14:06.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.501+0000 7fdac9ffb700 1 -- 172.21.15.90:0/1451261414 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fdacc1bee70 con 0x7fdacc0751f0 2024-04-14T15:14:06.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.501+0000 7fdac9ffb700 10 client.36459.objecter ms_dispatch 0x7fdacc001550 command_reply(tid 9: 0 ) 2024-04-14T15:14:06.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.501+0000 7fdac9ffb700 10 client.36459.objecter _finish_command 9 = system:0 2024-04-14T15:14:06.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.501+0000 7fdac9ffb700 20 client.36459.objecter put_session s=0x7fdacc1b03c0 osd=4 4 2024-04-14T15:14:06.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.501+0000 7fdac9ffb700 15 client.36459.objecter _session_command_op_remove 4 9 2024-04-14T15:14:06.513 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:14:06.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.517+0000 7fdad2f78700 10 client.36459.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:14:06.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.518+0000 7fdad2f78700 15 client.36459.objecter _session_command_op_assign -1 10 2024-04-14T15:14:06.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.518+0000 7fdad2f78700 20 client.36459.objecter _get_session s=0x7fdacc1b03c0 osd=4 3 2024-04-14T15:14:06.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.518+0000 7fdad2f78700 20 client.36459.objecter put_session s=0x7fdacc1b03c0 osd=4 3 2024-04-14T15:14:06.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.518+0000 7fdad2f78700 20 client.36459.objecter _get_session s=0x7fdacc1b03c0 osd=4 3 2024-04-14T15:14:06.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.518+0000 7fdad2f78700 15 client.36459.objecter _session_command_op_remove -1 10 2024-04-14T15:14:06.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.518+0000 7fdad2f78700 20 client.36459.objecter get_session s=0x7fdacc1b03c0 osd=4 3 2024-04-14T15:14:06.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.518+0000 7fdad2f78700 15 client.36459.objecter _session_command_op_assign 4 10 2024-04-14T15:14:06.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.518+0000 7fdad2f78700 20 client.36459.objecter put_session s=0x7fdacc1b03c0 osd=4 4 2024-04-14T15:14:06.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.518+0000 7fdad2f78700 10 client.36459.objecter _send_command 10 2024-04-14T15:14:06.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.518+0000 7fdad2f78700 1 -- 172.21.15.90:0/1451261414 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdacc1af930 con 0x7fdacc0751f0 2024-04-14T15:14:06.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.520+0000 7fdac9ffb700 1 -- 172.21.15.90:0/1451261414 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fdacc1af930 con 0x7fdacc0751f0 2024-04-14T15:14:06.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.520+0000 7fdac9ffb700 10 client.36459.objecter ms_dispatch 0x7fdacc001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:14:06.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.520+0000 7fdac9ffb700 10 client.36459.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:14:06.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.520+0000 7fdac9ffb700 20 client.36459.objecter put_session s=0x7fdacc1b03c0 osd=4 4 2024-04-14T15:14:06.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.520+0000 7fdac9ffb700 15 client.36459.objecter _session_command_op_remove 4 10 2024-04-14T15:14:06.520 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:14:06.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.520+0000 7fdad2f78700 10 client.36459.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:14:06.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.520+0000 7fdad2f78700 15 client.36459.objecter _session_command_op_assign -1 11 2024-04-14T15:14:06.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.520+0000 7fdad2f78700 20 client.36459.objecter _get_session s=0x7fdacc1af6e0 osd=5 3 2024-04-14T15:14:06.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.520+0000 7fdad2f78700 20 client.36459.objecter put_session s=0x7fdacc1af6e0 osd=5 3 2024-04-14T15:14:06.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.520+0000 7fdad2f78700 20 client.36459.objecter _get_session s=0x7fdacc1af6e0 osd=5 3 2024-04-14T15:14:06.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.520+0000 7fdad2f78700 15 client.36459.objecter _session_command_op_remove -1 11 2024-04-14T15:14:06.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.520+0000 7fdad2f78700 20 client.36459.objecter get_session s=0x7fdacc1af6e0 osd=5 3 2024-04-14T15:14:06.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.520+0000 7fdad2f78700 15 client.36459.objecter _session_command_op_assign 5 11 2024-04-14T15:14:06.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.520+0000 7fdad2f78700 20 client.36459.objecter put_session s=0x7fdacc1af6e0 osd=5 4 2024-04-14T15:14:06.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.520+0000 7fdad2f78700 10 client.36459.objecter _send_command 11 2024-04-14T15:14:06.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.520+0000 7fdad2f78700 1 -- 172.21.15.90:0/1451261414 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fdacc1c1710 con 0x7fdacc1b4e20 2024-04-14T15:14:06.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.520+0000 7fdac9ffb700 10 client.36459.objecter ms_handle_connect 0x7fdacc1b4e20 2024-04-14T15:14:06.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.521+0000 7fdac9ffb700 1 -- 172.21.15.90:0/1451261414 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fdacc1c1710 con 0x7fdacc1b4e20 2024-04-14T15:14:06.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.521+0000 7fdac9ffb700 10 client.36459.objecter ms_dispatch 0x7fdacc001550 command_reply(tid 11: 0 ) 2024-04-14T15:14:06.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.521+0000 7fdac9ffb700 10 client.36459.objecter _finish_command 11 = system:0 2024-04-14T15:14:06.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.521+0000 7fdac9ffb700 20 client.36459.objecter put_session s=0x7fdacc1af6e0 osd=5 4 2024-04-14T15:14:06.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.521+0000 7fdac9ffb700 15 client.36459.objecter _session_command_op_remove 5 11 2024-04-14T15:14:06.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:06.538 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:06.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:14:06.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.538+0000 7fdad2f78700 10 client.36459.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:14:06.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.538+0000 7fdad2f78700 15 client.36459.objecter _session_command_op_assign -1 12 2024-04-14T15:14:06.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.538+0000 7fdad2f78700 20 client.36459.objecter _get_session s=0x7fdacc1af6e0 osd=5 3 2024-04-14T15:14:06.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.538+0000 7fdad2f78700 20 client.36459.objecter put_session s=0x7fdacc1af6e0 osd=5 3 2024-04-14T15:14:06.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.538+0000 7fdad2f78700 20 client.36459.objecter _get_session s=0x7fdacc1af6e0 osd=5 3 2024-04-14T15:14:06.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.538+0000 7fdad2f78700 15 client.36459.objecter _session_command_op_remove -1 12 2024-04-14T15:14:06.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.538+0000 7fdad2f78700 20 client.36459.objecter get_session s=0x7fdacc1af6e0 osd=5 3 2024-04-14T15:14:06.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.538+0000 7fdad2f78700 15 client.36459.objecter _session_command_op_assign 5 12 2024-04-14T15:14:06.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.538+0000 7fdad2f78700 20 client.36459.objecter put_session s=0x7fdacc1af6e0 osd=5 4 2024-04-14T15:14:06.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.538+0000 7fdad2f78700 10 client.36459.objecter _send_command 12 2024-04-14T15:14:06.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.538+0000 7fdad2f78700 1 -- 172.21.15.90:0/1451261414 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdacc1c1710 con 0x7fdacc1b4e20 2024-04-14T15:14:06.540 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:14:06.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.540+0000 7fdac9ffb700 1 -- 172.21.15.90:0/1451261414 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fdacc1c1710 con 0x7fdacc1b4e20 2024-04-14T15:14:06.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.540+0000 7fdac9ffb700 10 client.36459.objecter ms_dispatch 0x7fdacc001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:14:06.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.540+0000 7fdac9ffb700 10 client.36459.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:14:06.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.540+0000 7fdac9ffb700 20 client.36459.objecter put_session s=0x7fdacc1af6e0 osd=5 4 2024-04-14T15:14:06.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.540+0000 7fdac9ffb700 15 client.36459.objecter _session_command_op_remove 5 12 2024-04-14T15:14:06.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.540+0000 7fdad2f78700 10 client.36459.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:14:06.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.540+0000 7fdad2f78700 15 client.36459.objecter _session_command_op_assign -1 13 2024-04-14T15:14:06.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.540+0000 7fdad2f78700 20 client.36459.objecter _get_session s=0x7fdacc1c1710 osd=6 3 2024-04-14T15:14:06.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.540+0000 7fdad2f78700 20 client.36459.objecter put_session s=0x7fdacc1c1710 osd=6 3 2024-04-14T15:14:06.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.540+0000 7fdad2f78700 20 client.36459.objecter _get_session s=0x7fdacc1c1710 osd=6 3 2024-04-14T15:14:06.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.540+0000 7fdad2f78700 15 client.36459.objecter _session_command_op_remove -1 13 2024-04-14T15:14:06.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.540+0000 7fdad2f78700 20 client.36459.objecter get_session s=0x7fdacc1c1710 osd=6 3 2024-04-14T15:14:06.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.540+0000 7fdad2f78700 15 client.36459.objecter _session_command_op_assign 6 13 2024-04-14T15:14:06.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.540+0000 7fdad2f78700 20 client.36459.objecter put_session s=0x7fdacc1c1710 osd=6 4 2024-04-14T15:14:06.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.540+0000 7fdad2f78700 10 client.36459.objecter _send_command 13 2024-04-14T15:14:06.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.540+0000 7fdad2f78700 1 -- 172.21.15.90:0/1451261414 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fdacc1afb80 con 0x7fdacc1c3eb0 2024-04-14T15:14:06.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.541+0000 7fdac9ffb700 10 client.36459.objecter ms_handle_connect 0x7fdacc1c3eb0 2024-04-14T15:14:06.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.542+0000 7fdac9ffb700 1 -- 172.21.15.90:0/1451261414 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fdacc1afb80 con 0x7fdacc1c3eb0 2024-04-14T15:14:06.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.542+0000 7fdac9ffb700 10 client.36459.objecter ms_dispatch 0x7fdacc001550 command_reply(tid 13: 0 ) 2024-04-14T15:14:06.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.542+0000 7fdac9ffb700 10 client.36459.objecter _finish_command 13 = system:0 2024-04-14T15:14:06.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.542+0000 7fdac9ffb700 20 client.36459.objecter put_session s=0x7fdacc1c1710 osd=6 4 2024-04-14T15:14:06.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.542+0000 7fdac9ffb700 15 client.36459.objecter _session_command_op_remove 6 13 2024-04-14T15:14:06.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.558+0000 7fdad2f78700 10 client.36459.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:14:06.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.558+0000 7fdad2f78700 15 client.36459.objecter _session_command_op_assign -1 14 2024-04-14T15:14:06.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.558+0000 7fdad2f78700 20 client.36459.objecter _get_session s=0x7fdacc1c1710 osd=6 3 2024-04-14T15:14:06.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.558+0000 7fdad2f78700 20 client.36459.objecter put_session s=0x7fdacc1c1710 osd=6 3 2024-04-14T15:14:06.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.558+0000 7fdad2f78700 20 client.36459.objecter _get_session s=0x7fdacc1c1710 osd=6 3 2024-04-14T15:14:06.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.558+0000 7fdad2f78700 15 client.36459.objecter _session_command_op_remove -1 14 2024-04-14T15:14:06.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.558+0000 7fdad2f78700 20 client.36459.objecter get_session s=0x7fdacc1c1710 osd=6 3 2024-04-14T15:14:06.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.558+0000 7fdad2f78700 15 client.36459.objecter _session_command_op_assign 6 14 2024-04-14T15:14:06.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.558+0000 7fdad2f78700 20 client.36459.objecter put_session s=0x7fdacc1c1710 osd=6 4 2024-04-14T15:14:06.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.558+0000 7fdad2f78700 10 client.36459.objecter _send_command 14 2024-04-14T15:14:06.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.558+0000 7fdad2f78700 1 -- 172.21.15.90:0/1451261414 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdacc1c79c0 con 0x7fdacc1c3eb0 2024-04-14T15:14:06.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.560+0000 7fdac9ffb700 1 -- 172.21.15.90:0/1451261414 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fdacc1c79c0 con 0x7fdacc1c3eb0 2024-04-14T15:14:06.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.560+0000 7fdac9ffb700 10 client.36459.objecter ms_dispatch 0x7fdacc001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:14:06.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.560+0000 7fdac9ffb700 10 client.36459.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:14:06.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.560+0000 7fdac9ffb700 20 client.36459.objecter put_session s=0x7fdacc1c1710 osd=6 4 2024-04-14T15:14:06.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.560+0000 7fdac9ffb700 15 client.36459.objecter _session_command_op_remove 6 14 2024-04-14T15:14:06.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.560+0000 7fdad2f78700 10 client.36459.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:14:06.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.560+0000 7fdad2f78700 15 client.36459.objecter _session_command_op_assign -1 15 2024-04-14T15:14:06.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.560+0000 7fdad2f78700 20 client.36459.objecter _get_session s=0x7fdacc1afbf0 osd=7 3 2024-04-14T15:14:06.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.560+0000 7fdad2f78700 20 client.36459.objecter put_session s=0x7fdacc1afbf0 osd=7 3 2024-04-14T15:14:06.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.560+0000 7fdad2f78700 20 client.36459.objecter _get_session s=0x7fdacc1afbf0 osd=7 3 2024-04-14T15:14:06.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.560+0000 7fdad2f78700 15 client.36459.objecter _session_command_op_remove -1 15 2024-04-14T15:14:06.564 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:14:06.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.560+0000 7fdad2f78700 20 client.36459.objecter get_session s=0x7fdacc1afbf0 osd=7 3 2024-04-14T15:14:06.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.560+0000 7fdad2f78700 15 client.36459.objecter _session_command_op_assign 7 15 2024-04-14T15:14:06.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.560+0000 7fdad2f78700 20 client.36459.objecter put_session s=0x7fdacc1afbf0 osd=7 4 2024-04-14T15:14:06.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.560+0000 7fdad2f78700 10 client.36459.objecter _send_command 15 2024-04-14T15:14:06.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.560+0000 7fdad2f78700 1 -- 172.21.15.90:0/1451261414 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fdacc1cbc70 con 0x7fdacc1c7f10 2024-04-14T15:14:06.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.561+0000 7fdac9ffb700 10 client.36459.objecter ms_handle_connect 0x7fdacc1c7f10 2024-04-14T15:14:06.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.561+0000 7fdac9ffb700 1 -- 172.21.15.90:0/1451261414 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fdacc1cbc70 con 0x7fdacc1c7f10 2024-04-14T15:14:06.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.561+0000 7fdac9ffb700 10 client.36459.objecter ms_dispatch 0x7fdacc001550 command_reply(tid 15: 0 ) 2024-04-14T15:14:06.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.561+0000 7fdac9ffb700 10 client.36459.objecter _finish_command 15 = system:0 2024-04-14T15:14:06.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.561+0000 7fdac9ffb700 20 client.36459.objecter put_session s=0x7fdacc1afbf0 osd=7 4 2024-04-14T15:14:06.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.561+0000 7fdac9ffb700 15 client.36459.objecter _session_command_op_remove 7 15 2024-04-14T15:14:06.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.578+0000 7fdad2f78700 10 client.36459.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:14:06.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.578+0000 7fdad2f78700 15 client.36459.objecter _session_command_op_assign -1 16 2024-04-14T15:14:06.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.578+0000 7fdad2f78700 20 client.36459.objecter _get_session s=0x7fdacc1afbf0 osd=7 3 2024-04-14T15:14:06.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.578+0000 7fdad2f78700 20 client.36459.objecter put_session s=0x7fdacc1afbf0 osd=7 3 2024-04-14T15:14:06.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.578+0000 7fdad2f78700 20 client.36459.objecter _get_session s=0x7fdacc1afbf0 osd=7 3 2024-04-14T15:14:06.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.578+0000 7fdad2f78700 15 client.36459.objecter _session_command_op_remove -1 16 2024-04-14T15:14:06.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.578+0000 7fdad2f78700 20 client.36459.objecter get_session s=0x7fdacc1afbf0 osd=7 3 2024-04-14T15:14:06.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.578+0000 7fdad2f78700 15 client.36459.objecter _session_command_op_assign 7 16 2024-04-14T15:14:06.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.578+0000 7fdad2f78700 20 client.36459.objecter put_session s=0x7fdacc1afbf0 osd=7 4 2024-04-14T15:14:06.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.578+0000 7fdad2f78700 10 client.36459.objecter _send_command 16 2024-04-14T15:14:06.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.578+0000 7fdad2f78700 1 -- 172.21.15.90:0/1451261414 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdacc1cbc30 con 0x7fdacc1c7f10 2024-04-14T15:14:06.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.580+0000 7fdac9ffb700 1 -- 172.21.15.90:0/1451261414 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fdacc1cbc30 con 0x7fdacc1c7f10 2024-04-14T15:14:06.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.580+0000 7fdac9ffb700 10 client.36459.objecter ms_dispatch 0x7fdacc001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:14:06.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.580+0000 7fdac9ffb700 10 client.36459.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:14:06.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.580+0000 7fdac9ffb700 20 client.36459.objecter put_session s=0x7fdacc1afbf0 osd=7 4 2024-04-14T15:14:06.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.580+0000 7fdac9ffb700 15 client.36459.objecter _session_command_op_remove 7 16 2024-04-14T15:14:06.581 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:14:06.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.580+0000 7fdad2f78700 10 librados: watch_flush enter 2024-04-14T15:14:06.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.580+0000 7fdad2f78700 10 librados: watch_flush exit 2024-04-14T15:14:06.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.580+0000 7fdad2f78700 10 client.36459.objecter close_session for osd.0 2024-04-14T15:14:06.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.580+0000 7fdad2f78700 1 -- 172.21.15.90:0/1451261414 >> v1:172.21.15.90:6801/3761806885 conn(0x7fdacc1a97b0 legacy=0x7fdacc1a9b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:06.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.580+0000 7fdad2f78700 20 client.36459.objecter put_session s=0x7fdacc10d470 osd=0 1 2024-04-14T15:14:06.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.580+0000 7fdad2f78700 10 client.36459.objecter close_session for osd.1 2024-04-14T15:14:06.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.580+0000 7fdad2f78700 1 -- 172.21.15.90:0/1451261414 >> v1:172.21.15.90:6805/4170361699 conn(0x7fdacc0636c0 legacy=0x7fdacc1096f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:06.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.580+0000 7fdad2f78700 20 client.36459.objecter put_session s=0x7fdacc1ab2e0 osd=1 1 2024-04-14T15:14:06.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.580+0000 7fdad2f78700 10 client.36459.objecter close_session for osd.2 2024-04-14T15:14:06.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.580+0000 7fdad2f78700 1 -- 172.21.15.90:0/1451261414 >> v1:172.21.15.90:6809/1686184251 conn(0x7fdacc0632b0 legacy=0x7fdacc1b2870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:06.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.580+0000 7fdad2f78700 20 client.36459.objecter put_session s=0x7fdacc063140 osd=2 1 2024-04-14T15:14:06.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.580+0000 7fdad2f78700 10 client.36459.objecter close_session for osd.3 2024-04-14T15:14:06.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.580+0000 7fdad2f78700 1 -- 172.21.15.90:0/1451261414 >> v1:172.21.15.90:6813/2270166774 conn(0x7fdacc02d1e0 legacy=0x7fdacc1b7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:06.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.580+0000 7fdad2f78700 20 client.36459.objecter put_session s=0x7fdacc02d070 osd=3 1 2024-04-14T15:14:06.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.580+0000 7fdad2f78700 10 client.36459.objecter close_session for osd.4 2024-04-14T15:14:06.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.580+0000 7fdad2f78700 1 -- 172.21.15.90:0/1451261414 >> v1:172.21.15.92:6800/3887549733 conn(0x7fdacc0751f0 legacy=0x7fdacc075630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:06.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.580+0000 7fdad2f78700 20 client.36459.objecter put_session s=0x7fdacc1b03c0 osd=4 1 2024-04-14T15:14:06.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.580+0000 7fdad2f78700 10 client.36459.objecter close_session for osd.5 2024-04-14T15:14:06.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.580+0000 7fdad2f78700 1 -- 172.21.15.90:0/1451261414 >> v1:172.21.15.92:6804/3244561440 conn(0x7fdacc1b4e20 legacy=0x7fdacc1bfff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:06.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.580+0000 7fdad2f78700 20 client.36459.objecter put_session s=0x7fdacc1af6e0 osd=5 1 2024-04-14T15:14:06.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.580+0000 7fdad2f78700 10 client.36459.objecter close_session for osd.6 2024-04-14T15:14:06.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.580+0000 7fdad1515700 1 -- 172.21.15.90:0/1451261414 reap_dead start 2024-04-14T15:14:06.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.580+0000 7fdad2f78700 1 -- 172.21.15.90:0/1451261414 >> v1:172.21.15.92:6808/636125963 conn(0x7fdacc1c3eb0 legacy=0x7fdacc1c62a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:06.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.580+0000 7fdad1515700 1 -- 172.21.15.90:0/1451261414 reap_dead start 2024-04-14T15:14:06.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.580+0000 7fdad2f78700 20 client.36459.objecter put_session s=0x7fdacc1c1710 osd=6 1 2024-04-14T15:14:06.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.580+0000 7fdad2f78700 10 client.36459.objecter close_session for osd.7 2024-04-14T15:14:06.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.580+0000 7fdad2f78700 1 -- 172.21.15.90:0/1451261414 >> v1:172.21.15.92:6812/1773706716 conn(0x7fdacc1c7f10 legacy=0x7fdacc1ca380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:06.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.580+0000 7fdad1515700 1 -- 172.21.15.90:0/1451261414 reap_dead start 2024-04-14T15:14:06.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.580+0000 7fdad1515700 1 -- 172.21.15.90:0/1451261414 reap_dead start 2024-04-14T15:14:06.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.580+0000 7fdad2f78700 20 client.36459.objecter put_session s=0x7fdacc1afbf0 osd=7 1 2024-04-14T15:14:06.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.580+0000 7fdad2f78700 20 client.36459.objecter shutdown clearing up homeless session... 2024-04-14T15:14:06.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.580+0000 7fdad2f78700 10 client.36459.objecter successfully canceled tick 2024-04-14T15:14:06.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.581+0000 7fdad2f78700 1 -- 172.21.15.90:0/1451261414 >> v1:172.21.15.90:6800/420935427 conn(0x7fdabc071300 legacy=0x7fdabc0737c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:06.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.581+0000 7fdad2f78700 1 -- 172.21.15.90:0/1451261414 >> v1:172.21.15.90:6789/0 conn(0x7fdacc108870 legacy=0x7fdacc10d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:06.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.581+0000 7fdad2f78700 1 -- 172.21.15.90:0/1451261414 shutdown_connections 2024-04-14T15:14:06.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.581+0000 7fdad2f78700 1 -- 172.21.15.90:0/1451261414 >> 172.21.15.90:0/1451261414 conn(0x7fdacc0fe3a0 msgr2=0x7fdacc102910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:14:06.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.581+0000 7fdad2f78700 1 -- 172.21.15.90:0/1451261414 shutdown_connections 2024-04-14T15:14:06.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.581+0000 7fdad2f78700 1 -- 172.21.15.90:0/1451261414 wait complete. 2024-04-14T15:14:06.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.581+0000 7fdad2f78700 1 librados: shutdown 2024-04-14T15:14:06.585 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:14:06.585 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:14:06.585 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:14:06.585 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:14:06.585 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:14:06.585 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:14:06.585 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:14:06.586 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:14:06.639 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:14:06.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:06.668 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:06.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:14:06.769 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:14:06.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:06.795 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:06.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:14:06.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.823+0000 7ff242ffd700 1 -- 172.21.15.90:0/3425941736 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v179) ==== 101+0+0 (unknown 1921841384 0 0) 0x7ff24405a440 con 0x7ff24c110690 2024-04-14T15:14:06.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.827+0000 7ff252d95700 10 librados: watch_flush enter 2024-04-14T15:14:06.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.827+0000 7ff252d95700 10 librados: watch_flush exit 2024-04-14T15:14:06.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.827+0000 7ff252d95700 20 client.48292.objecter shutdown clearing up homeless session... 2024-04-14T15:14:06.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.827+0000 7ff252d95700 10 client.48292.objecter successfully canceled tick 2024-04-14T15:14:06.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.828+0000 7ff252d95700 1 -- 172.21.15.90:0/3425941736 >> v1:172.21.15.90:6800/420935427 conn(0x7ff2380713f0 legacy=0x7ff2380738b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:06.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.828+0000 7ff252d95700 1 -- 172.21.15.90:0/3425941736 >> v1:172.21.15.92:6789/0 conn(0x7ff24c110690 legacy=0x7ff24c1b13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:06.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.829+0000 7ff252d95700 1 -- 172.21.15.90:0/3425941736 shutdown_connections 2024-04-14T15:14:06.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.829+0000 7ff252d95700 1 -- 172.21.15.90:0/3425941736 >> 172.21.15.90:0/3425941736 conn(0x7ff24c06d810 msgr2=0x7ff24c072ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:14:06.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.829+0000 7ff252d95700 1 -- 172.21.15.90:0/3425941736 shutdown_connections 2024-04-14T15:14:06.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.829+0000 7ff252d95700 1 -- 172.21.15.90:0/3425941736 wait complete. 2024-04-14T15:14:06.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:06.829+0000 7ff252d95700 1 librados: shutdown 2024-04-14T15:14:06.837 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is unset 2024-04-14T15:14:06.896 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:14:06.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:06.920 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:06.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:14:07.021 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:14:07.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:07.047 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:07.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:14:07.148 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:14:07.151 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:06 smithi090 ceph-mon[41670]: pgmap v212: 392 pgs: 64 creating+peering, 1 peering, 327 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 2.7 KiB/s wr, 5 op/s; 31 B/s, 0 objects/s recovering 2024-04-14T15:14:07.151 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:06 smithi090 ceph-mon[41670]: 2.1 scrub starts 2024-04-14T15:14:07.151 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:06 smithi090 ceph-mon[41670]: 2.1 scrub ok 2024-04-14T15:14:07.151 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:06 smithi090 ceph-mon[41670]: 2.11 scrub starts 2024-04-14T15:14:07.152 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:06 smithi090 ceph-mon[41670]: 2.11 scrub ok 2024-04-14T15:14:07.152 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:06 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3425941736' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:14:07.152 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:06 smithi090 ceph-mon[41670]: from='client.48292 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:14:07.152 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:06 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1451261414' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:14:07.152 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:06 smithi090 ceph-mon[41670]: 6.1 scrub starts 2024-04-14T15:14:07.152 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:06 smithi090 ceph-mon[41670]: 6.1 scrub ok 2024-04-14T15:14:07.152 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:06 smithi090 ceph-mon[41670]: 2.3 scrub starts 2024-04-14T15:14:07.152 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:06 smithi090 ceph-mon[41670]: 2.3 scrub ok 2024-04-14T15:14:07.152 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:06 smithi090 ceph-mon[41670]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-14T15:14:07.152 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:06 smithi090 ceph-mon[41670]: from='client.46400 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi090-93601-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:14:07.152 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:06 smithi090 ceph-mon[41670]: from='client.48292 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-14T15:14:07.152 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:06 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/844371530' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi090-93601-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi090-93601-24"}]: dispatch 2024-04-14T15:14:07.152 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:06 smithi090 ceph-mon[41670]: osdmap e179: 8 total, 8 up, 8 in 2024-04-14T15:14:07.152 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:06 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1582012881' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi090-93342-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:07.153 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:06 smithi090 ceph-mon[41670]: from='client.46400 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi090-93601-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi090-93601-24"}]: dispatch 2024-04-14T15:14:07.153 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:06 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1753841519' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi090-93363-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:07.153 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:06 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:07.153 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:06 smithi090 ceph-mon[30399]: pgmap v212: 392 pgs: 64 creating+peering, 1 peering, 327 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 2.7 KiB/s wr, 5 op/s; 31 B/s, 0 objects/s recovering 2024-04-14T15:14:07.153 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:06 smithi090 ceph-mon[30399]: 2.1 scrub starts 2024-04-14T15:14:07.153 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:06 smithi090 ceph-mon[30399]: 2.1 scrub ok 2024-04-14T15:14:07.153 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:06 smithi090 ceph-mon[30399]: 2.11 scrub starts 2024-04-14T15:14:07.153 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:06 smithi090 ceph-mon[30399]: 2.11 scrub ok 2024-04-14T15:14:07.153 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:06 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3425941736' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:14:07.153 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:06 smithi090 ceph-mon[30399]: from='client.48292 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:14:07.154 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:06 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1451261414' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:14:07.154 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:06 smithi090 ceph-mon[30399]: 6.1 scrub starts 2024-04-14T15:14:07.154 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:06 smithi090 ceph-mon[30399]: 6.1 scrub ok 2024-04-14T15:14:07.154 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:06 smithi090 ceph-mon[30399]: 2.3 scrub starts 2024-04-14T15:14:07.154 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:06 smithi090 ceph-mon[30399]: 2.3 scrub ok 2024-04-14T15:14:07.154 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:06 smithi090 ceph-mon[30399]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-14T15:14:07.154 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:06 smithi090 ceph-mon[30399]: from='client.46400 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi090-93601-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:14:07.154 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:06 smithi090 ceph-mon[30399]: from='client.48292 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-14T15:14:07.154 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:06 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/844371530' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi090-93601-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi090-93601-24"}]: dispatch 2024-04-14T15:14:07.154 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:06 smithi090 ceph-mon[30399]: osdmap e179: 8 total, 8 up, 8 in 2024-04-14T15:14:07.154 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:06 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1582012881' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi090-93342-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:07.154 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:06 smithi090 ceph-mon[30399]: from='client.46400 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi090-93601-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi090-93601-24"}]: dispatch 2024-04-14T15:14:07.155 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:06 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1753841519' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi090-93363-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:07.155 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:06 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:07.171 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:07.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:07.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:14:07.273 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:14:07.280 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:06 smithi092 ceph-mon[33966]: pgmap v212: 392 pgs: 64 creating+peering, 1 peering, 327 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 2.7 KiB/s wr, 5 op/s; 31 B/s, 0 objects/s recovering 2024-04-14T15:14:07.281 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:06 smithi092 ceph-mon[33966]: 2.1 scrub starts 2024-04-14T15:14:07.281 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:06 smithi092 ceph-mon[33966]: 2.1 scrub ok 2024-04-14T15:14:07.281 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:06 smithi092 ceph-mon[33966]: 2.11 scrub starts 2024-04-14T15:14:07.281 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:06 smithi092 ceph-mon[33966]: 2.11 scrub ok 2024-04-14T15:14:07.281 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:06 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3425941736' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:14:07.281 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:06 smithi092 ceph-mon[33966]: from='client.48292 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:14:07.281 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:06 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1451261414' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:14:07.281 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:06 smithi092 ceph-mon[33966]: 6.1 scrub starts 2024-04-14T15:14:07.281 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:06 smithi092 ceph-mon[33966]: 6.1 scrub ok 2024-04-14T15:14:07.281 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:06 smithi092 ceph-mon[33966]: 2.3 scrub starts 2024-04-14T15:14:07.281 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:06 smithi092 ceph-mon[33966]: 2.3 scrub ok 2024-04-14T15:14:07.281 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:06 smithi092 ceph-mon[33966]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-14T15:14:07.281 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:06 smithi092 ceph-mon[33966]: from='client.46400 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi090-93601-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:14:07.282 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:06 smithi092 ceph-mon[33966]: from='client.48292 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-14T15:14:07.282 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:06 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/844371530' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi090-93601-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi090-93601-24"}]: dispatch 2024-04-14T15:14:07.282 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:06 smithi092 ceph-mon[33966]: osdmap e179: 8 total, 8 up, 8 in 2024-04-14T15:14:07.282 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:06 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1582012881' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi090-93342-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:07.282 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:06 smithi092 ceph-mon[33966]: from='client.46400 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi090-93601-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi090-93601-24"}]: dispatch 2024-04-14T15:14:07.282 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:06 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1753841519' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi090-93363-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:07.282 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:06 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:07.298 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:07.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:07.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:14:07.400 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:14:07.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:07.425 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:07.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:14:07.526 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:14:07.551 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:07.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:07.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:14:07.645 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-04-14T15:14:07.646 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-04-14T15:14:07.646 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 0 2024-04-14T15:14:07.646 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-04-14T15:14:07.652 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:14:07.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:07.677 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:07.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:14:07.778 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:07.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:07.803 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:07.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:07.904 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:14:07.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:07.929 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:07.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:14:08.030 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:14:08.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:08.054 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:08.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:14:08.155 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:14:08.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:08.180 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:08.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:14:08.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:07 smithi090 ceph-mon[41670]: 4.c deep-scrub starts 2024-04-14T15:14:08.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:07 smithi090 ceph-mon[41670]: 4.c deep-scrub ok 2024-04-14T15:14:08.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:07 smithi090 ceph-mon[41670]: 2.18 scrub starts 2024-04-14T15:14:08.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:07 smithi090 ceph-mon[41670]: 2.18 scrub ok 2024-04-14T15:14:08.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:07 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1582012881' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi090-93342-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:08.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:07 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1753841519' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi090-93363-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:08.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:07 smithi090 ceph-mon[41670]: osdmap e180: 8 total, 8 up, 8 in 2024-04-14T15:14:08.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:07 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3868448150' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi090-93724-16"}]: dispatch 2024-04-14T15:14:08.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:07 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:08.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:07 smithi090 ceph-mon[30399]: 4.c deep-scrub starts 2024-04-14T15:14:08.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:07 smithi090 ceph-mon[30399]: 4.c deep-scrub ok 2024-04-14T15:14:08.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:07 smithi090 ceph-mon[30399]: 2.18 scrub starts 2024-04-14T15:14:08.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:07 smithi090 ceph-mon[30399]: 2.18 scrub ok 2024-04-14T15:14:08.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:07 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1582012881' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi090-93342-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:08.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:07 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1753841519' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi090-93363-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:08.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:07 smithi090 ceph-mon[30399]: osdmap e180: 8 total, 8 up, 8 in 2024-04-14T15:14:08.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:07 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3868448150' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi090-93724-16"}]: dispatch 2024-04-14T15:14:08.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:07 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:08.282 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:08.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:08.306 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:08.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:08.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:07 smithi092 ceph-mon[33966]: 4.c deep-scrub starts 2024-04-14T15:14:08.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:07 smithi092 ceph-mon[33966]: 4.c deep-scrub ok 2024-04-14T15:14:08.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:07 smithi092 ceph-mon[33966]: 2.18 scrub starts 2024-04-14T15:14:08.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:07 smithi092 ceph-mon[33966]: 2.18 scrub ok 2024-04-14T15:14:08.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:07 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1582012881' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi090-93342-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:08.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:07 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1753841519' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi090-93363-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:08.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:07 smithi092 ceph-mon[33966]: osdmap e180: 8 total, 8 up, 8 in 2024-04-14T15:14:08.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:07 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3868448150' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi090-93724-16"}]: dispatch 2024-04-14T15:14:08.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:07 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:08.407 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:14:08.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:08.432 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:08.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:14:08.533 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:14:08.558 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:08.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:08.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:14:08.562 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-14T15:14:08.660 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:14:08.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:08.684 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:08.684 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:14:08.785 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:08.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:08.809 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:08.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:08.838 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP snapshots: Running main() from gmock_main.cc 2024-04-14T15:14:08.838 INFO:tasks.workunit.client.0.smithi090.stdout: snapshots: [==========] Running 11 tests from 2 test suites. 2024-04-14T15:14:08.838 INFO:tasks.workunit.client.0.smithi090.stdout: snapshots: [----------] Global test environment set-up. 2024-04-14T15:14:08.838 INFO:tasks.workunit.client.0.smithi090.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots 2024-04-14T15:14:08.838 INFO:tasks.workunit.client.0.smithi090.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapList 2024-04-14T15:14:08.838 INFO:tasks.workunit.client.0.smithi090.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapList (4521 ms) 2024-04-14T15:14:08.838 INFO:tasks.workunit.client.0.smithi090.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapRemove 2024-04-14T15:14:08.839 INFO:tasks.workunit.client.0.smithi090.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapRemove (4688 ms) 2024-04-14T15:14:08.839 INFO:tasks.workunit.client.0.smithi090.stdout: snapshots: [ RUN ] NeoRadosSnapshots.Rollback 2024-04-14T15:14:08.839 INFO:tasks.workunit.client.0.smithi090.stdout: snapshots: [ OK ] NeoRadosSnapshots.Rollback (4018 ms) 2024-04-14T15:14:08.839 INFO:tasks.workunit.client.0.smithi090.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapGetName 2024-04-14T15:14:08.839 INFO:tasks.workunit.client.0.smithi090.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapGetName (5003 ms) 2024-04-14T15:14:08.839 INFO:tasks.workunit.client.0.smithi090.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapCreateRemove 2024-04-14T15:14:08.839 INFO:tasks.workunit.client.0.smithi090.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapCreateRemove (7021 ms) 2024-04-14T15:14:08.839 INFO:tasks.workunit.client.0.smithi090.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots (25251 ms total) 2024-04-14T15:14:08.839 INFO:tasks.workunit.client.0.smithi090.stdout: snapshots: 2024-04-14T15:14:08.839 INFO:tasks.workunit.client.0.smithi090.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps 2024-04-14T15:14:08.839 INFO:tasks.workunit.client.0.smithi090.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Snap 2024-04-14T15:14:08.840 INFO:tasks.workunit.client.0.smithi090.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Snap (5011 ms) 2024-04-14T15:14:08.840 INFO:tasks.workunit.client.0.smithi090.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Rollback 2024-04-14T15:14:08.840 INFO:tasks.workunit.client.0.smithi090.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Rollback (5950 ms) 2024-04-14T15:14:08.840 INFO:tasks.workunit.client.0.smithi090.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.SnapOverlap 2024-04-14T15:14:08.840 INFO:tasks.workunit.client.0.smithi090.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.SnapOverlap (8014 ms) 2024-04-14T15:14:08.840 INFO:tasks.workunit.client.0.smithi090.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Bug11677 2024-04-14T15:14:08.840 INFO:tasks.workunit.client.0.smithi090.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Bug11677 (6019 ms) 2024-04-14T15:14:08.840 INFO:tasks.workunit.client.0.smithi090.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.OrderSnap 2024-04-14T15:14:08.840 INFO:tasks.workunit.client.0.smithi090.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.OrderSnap (3979 ms) 2024-04-14T15:14:08.840 INFO:tasks.workunit.client.0.smithi090.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.ReusePurgedSnap 2024-04-14T15:14:08.840 INFO:tasks.workunit.client.0.smithi090.stdout: snapshots: Deleting snap 3 in pool ReusePurgedSnapsmithi090-94770-11. 2024-04-14T15:14:08.840 INFO:tasks.workunit.client.0.smithi090.stdout: snapshots: Waiting for snaps to purge. 2024-04-14T15:14:08.841 INFO:tasks.workunit.client.0.smithi090.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.ReusePurgedSnap (20003 ms) 2024-04-14T15:14:08.841 INFO:tasks.workunit.client.0.smithi090.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps (48976 ms total) 2024-04-14T15:14:08.841 INFO:tasks.workunit.client.0.smithi090.stdout: snapshots: 2024-04-14T15:14:08.841 INFO:tasks.workunit.client.0.smithi090.stdout: snapshots: [----------] Global test environment tear-down 2024-04-14T15:14:08.841 INFO:tasks.workunit.client.0.smithi090.stdout: snapshots: [==========] 11 tests from 2 test suites ran. (74227 ms total) 2024-04-14T15:14:08.841 INFO:tasks.workunit.client.0.smithi090.stdout: snapshots: [ PASSED ] 11 tests. 2024-04-14T15:14:08.858 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:14:08.910 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:14:08.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:08.935 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:08.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:14:09.037 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:14:09.062 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:09.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:09.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:14:09.164 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:14:09.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:09.189 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:09.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:14:09.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:08 smithi090 ceph-mon[41670]: pgmap v215: 360 pgs: 64 unknown, 1 peering, 295 active+clean; 592 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 31 B/s, 0 objects/s recovering 2024-04-14T15:14:09.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:08 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:14:09.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:08 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:14:09.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:08 smithi090 ceph-mon[41670]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:14:09.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:08 smithi090 ceph-mon[41670]: 2.14 scrub starts 2024-04-14T15:14:09.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:08 smithi090 ceph-mon[41670]: 2.14 scrub ok 2024-04-14T15:14:09.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:08 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:14:09.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:08 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:14:09.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:08 smithi090 ceph-mon[41670]: 2.1d deep-scrub starts 2024-04-14T15:14:09.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:08 smithi090 ceph-mon[41670]: 2.1d deep-scrub ok 2024-04-14T15:14:09.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:08 smithi090 ceph-mon[41670]: 4.b scrub starts 2024-04-14T15:14:09.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:08 smithi090 ceph-mon[41670]: 4.b scrub ok 2024-04-14T15:14:09.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:08 smithi090 ceph-mon[41670]: 5.12 deep-scrub starts 2024-04-14T15:14:09.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:08 smithi090 ceph-mon[41670]: 5.12 deep-scrub ok 2024-04-14T15:14:09.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:08 smithi090 ceph-mon[41670]: 5.18 deep-scrub starts 2024-04-14T15:14:09.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:08 smithi090 ceph-mon[41670]: 5.18 deep-scrub ok 2024-04-14T15:14:09.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:08 smithi090 ceph-mon[41670]: from='client.46400 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi090-93601-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi090-93601-24"}]': finished 2024-04-14T15:14:09.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:08 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3868448150' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi090-93724-16"}]': finished 2024-04-14T15:14:09.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:08 smithi090 ceph-mon[41670]: osdmap e181: 8 total, 8 up, 8 in 2024-04-14T15:14:09.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:08 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3868448150' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi090-93724-16"}]: dispatch 2024-04-14T15:14:09.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:08 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:09.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:08 smithi090 ceph-mon[30399]: pgmap v215: 360 pgs: 64 unknown, 1 peering, 295 active+clean; 592 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 31 B/s, 0 objects/s recovering 2024-04-14T15:14:09.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:08 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:14:09.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:08 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:14:09.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:08 smithi090 ceph-mon[30399]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:14:09.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:08 smithi090 ceph-mon[30399]: 2.14 scrub starts 2024-04-14T15:14:09.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:08 smithi090 ceph-mon[30399]: 2.14 scrub ok 2024-04-14T15:14:09.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:08 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:14:09.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:08 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:14:09.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:08 smithi090 ceph-mon[30399]: 2.1d deep-scrub starts 2024-04-14T15:14:09.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:08 smithi090 ceph-mon[30399]: 2.1d deep-scrub ok 2024-04-14T15:14:09.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:08 smithi090 ceph-mon[30399]: 4.b scrub starts 2024-04-14T15:14:09.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:08 smithi090 ceph-mon[30399]: 4.b scrub ok 2024-04-14T15:14:09.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:08 smithi090 ceph-mon[30399]: 5.12 deep-scrub starts 2024-04-14T15:14:09.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:08 smithi090 ceph-mon[30399]: 5.12 deep-scrub ok 2024-04-14T15:14:09.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:08 smithi090 ceph-mon[30399]: 5.18 deep-scrub starts 2024-04-14T15:14:09.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:08 smithi090 ceph-mon[30399]: 5.18 deep-scrub ok 2024-04-14T15:14:09.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:08 smithi090 ceph-mon[30399]: from='client.46400 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi090-93601-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi090-93601-24"}]': finished 2024-04-14T15:14:09.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:08 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3868448150' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi090-93724-16"}]': finished 2024-04-14T15:14:09.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:08 smithi090 ceph-mon[30399]: osdmap e181: 8 total, 8 up, 8 in 2024-04-14T15:14:09.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:08 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3868448150' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi090-93724-16"}]: dispatch 2024-04-14T15:14:09.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:08 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:09.290 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:14:09.295 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:08 smithi092 ceph-mon[33966]: pgmap v215: 360 pgs: 64 unknown, 1 peering, 295 active+clean; 592 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 31 B/s, 0 objects/s recovering 2024-04-14T15:14:09.295 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:08 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:14:09.295 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:08 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:14:09.295 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:08 smithi092 ceph-mon[33966]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:14:09.295 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:08 smithi092 ceph-mon[33966]: 2.14 scrub starts 2024-04-14T15:14:09.295 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:08 smithi092 ceph-mon[33966]: 2.14 scrub ok 2024-04-14T15:14:09.295 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:08 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:14:09.295 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:08 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:14:09.295 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:08 smithi092 ceph-mon[33966]: 2.1d deep-scrub starts 2024-04-14T15:14:09.295 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:08 smithi092 ceph-mon[33966]: 2.1d deep-scrub ok 2024-04-14T15:14:09.295 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:08 smithi092 ceph-mon[33966]: 4.b scrub starts 2024-04-14T15:14:09.296 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:08 smithi092 ceph-mon[33966]: 4.b scrub ok 2024-04-14T15:14:09.296 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:08 smithi092 ceph-mon[33966]: 5.12 deep-scrub starts 2024-04-14T15:14:09.296 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:08 smithi092 ceph-mon[33966]: 5.12 deep-scrub ok 2024-04-14T15:14:09.296 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:08 smithi092 ceph-mon[33966]: 5.18 deep-scrub starts 2024-04-14T15:14:09.296 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:08 smithi092 ceph-mon[33966]: 5.18 deep-scrub ok 2024-04-14T15:14:09.296 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:08 smithi092 ceph-mon[33966]: from='client.46400 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi090-93601-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi090-93601-24"}]': finished 2024-04-14T15:14:09.296 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:08 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3868448150' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi090-93724-16"}]': finished 2024-04-14T15:14:09.296 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:08 smithi092 ceph-mon[33966]: osdmap e181: 8 total, 8 up, 8 in 2024-04-14T15:14:09.296 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:08 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3868448150' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi090-93724-16"}]: dispatch 2024-04-14T15:14:09.296 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:08 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:09.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:09.316 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:09.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:14:09.419 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:09.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:09.443 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:09.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:09.477 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-14T15:14:09.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.530+0000 7f0fbf816700 1 -- 172.21.15.90:0/716899600 >> v1:172.21.15.92:6789/0 conn(0x7f0fb8107760 legacy=0x7f0fb8107b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:09.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.530+0000 7f0fbf816700 1 -- 172.21.15.90:0/716899600 shutdown_connections 2024-04-14T15:14:09.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.530+0000 7f0fbf816700 1 -- 172.21.15.90:0/716899600 >> 172.21.15.90:0/716899600 conn(0x7f0fb80fd290 msgr2=0x7f0fb80ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:14:09.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.530+0000 7f0fbf816700 1 -- 172.21.15.90:0/716899600 shutdown_connections 2024-04-14T15:14:09.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.530+0000 7f0fbf816700 1 -- 172.21.15.90:0/716899600 wait complete. 2024-04-14T15:14:09.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.530+0000 7f0fbf816700 1 librados: starting msgr at 2024-04-14T15:14:09.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.530+0000 7f0fbf816700 1 librados: starting objecter 2024-04-14T15:14:09.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.531+0000 7f0fbf816700 1 Processor -- start 2024-04-14T15:14:09.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.531+0000 7f0fbf816700 1 -- start start 2024-04-14T15:14:09.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.531+0000 7f0fbf816700 1 librados: setting wanted keys 2024-04-14T15:14:09.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.531+0000 7f0fbf816700 1 librados: calling monclient init 2024-04-14T15:14:09.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.531+0000 7f0fbf816700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f0fb81a7df0 con 0x7f0fb810e1e0 2024-04-14T15:14:09.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.531+0000 7f0fbf816700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f0fb81a8fd0 con 0x7f0fb810a610 2024-04-14T15:14:09.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.531+0000 7f0fbf816700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f0fb81aa1b0 con 0x7f0fb8107760 2024-04-14T15:14:09.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.531+0000 7f0fbd5b2700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f0fb8107760 0x7f0fb810be60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:44322/0 (socket says 172.21.15.90:44322) 2024-04-14T15:14:09.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.531+0000 7f0fbd5b2700 1 -- 172.21.15.90:0/4091838128 learned_addr learned my addr 172.21.15.90:0/4091838128 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:14:09.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.531+0000 7f0fae7fc700 10 client.?.objecter ms_handle_connect 0x7f0fb810e1e0 2024-04-14T15:14:09.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.531+0000 7f0fae7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:14:09.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.531+0000 7f0fae7fc700 10 client.?.objecter ms_handle_connect 0x7f0fb8107760 2024-04-14T15:14:09.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.531+0000 7f0fae7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:14:09.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.531+0000 7f0fae7fc700 10 client.?.objecter ms_handle_connect 0x7f0fb810a610 2024-04-14T15:14:09.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.531+0000 7f0fae7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:14:09.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.531+0000 7f0fae7fc700 1 -- 172.21.15.90:0/4091838128 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2477889422 0 0) 0x7f0fb81a7df0 con 0x7f0fb810e1e0 2024-04-14T15:14:09.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.531+0000 7f0fae7fc700 1 -- 172.21.15.90:0/4091838128 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0fa40031f0 con 0x7f0fb810e1e0 2024-04-14T15:14:09.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.531+0000 7f0fae7fc700 1 -- 172.21.15.90:0/4091838128 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4284363020 0 0) 0x7f0fb81aa1b0 con 0x7f0fb8107760 2024-04-14T15:14:09.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.531+0000 7f0fae7fc700 1 -- 172.21.15.90:0/4091838128 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0fb81a7df0 con 0x7f0fb8107760 2024-04-14T15:14:09.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.531+0000 7f0fae7fc700 1 -- 172.21.15.90:0/4091838128 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1504492386 0 0) 0x7f0fb81a8fd0 con 0x7f0fb810a610 2024-04-14T15:14:09.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.531+0000 7f0fae7fc700 1 -- 172.21.15.90:0/4091838128 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0fb81aa1b0 con 0x7f0fb810a610 2024-04-14T15:14:09.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.531+0000 7f0fae7fc700 1 -- 172.21.15.90:0/4091838128 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1627828209 0 0) 0x7f0fb81aa1b0 con 0x7f0fb810a610 2024-04-14T15:14:09.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.532+0000 7f0fae7fc700 1 -- 172.21.15.90:0/4091838128 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0fb81a8fd0 con 0x7f0fb810a610 2024-04-14T15:14:09.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.532+0000 7f0fae7fc700 1 -- 172.21.15.90:0/4091838128 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f0fa80017f0 con 0x7f0fb810a610 2024-04-14T15:14:09.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.532+0000 7f0fae7fc700 1 -- 172.21.15.90:0/4091838128 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2434903525 0 0) 0x7f0fa40031f0 con 0x7f0fb810e1e0 2024-04-14T15:14:09.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.532+0000 7f0fae7fc700 1 -- 172.21.15.90:0/4091838128 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0fb81aa1b0 con 0x7f0fb810e1e0 2024-04-14T15:14:09.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.532+0000 7f0fae7fc700 1 -- 172.21.15.90:0/4091838128 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 242641700 0 0) 0x7f0fb81a7df0 con 0x7f0fb8107760 2024-04-14T15:14:09.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.532+0000 7f0fae7fc700 1 -- 172.21.15.90:0/4091838128 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0fa40031f0 con 0x7f0fb8107760 2024-04-14T15:14:09.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.532+0000 7f0fae7fc700 1 -- 172.21.15.90:0/4091838128 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f0fb0001b30 con 0x7f0fb810e1e0 2024-04-14T15:14:09.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.532+0000 7f0fae7fc700 1 -- 172.21.15.90:0/4091838128 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f0fb40015d0 con 0x7f0fb8107760 2024-04-14T15:14:09.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.532+0000 7f0fae7fc700 1 -- 172.21.15.90:0/4091838128 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3825968333 0 0) 0x7f0fb81a8fd0 con 0x7f0fb810a610 2024-04-14T15:14:09.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.532+0000 7f0fae7fc700 1 -- 172.21.15.90:0/4091838128 >> v1:172.21.15.90:6790/0 conn(0x7f0fb8107760 legacy=0x7f0fb810be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:09.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.532+0000 7f0fae7fc700 1 -- 172.21.15.90:0/4091838128 >> v1:172.21.15.90:6789/0 conn(0x7f0fb810e1e0 legacy=0x7f0fb81a66d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:09.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.532+0000 7f0fae7fc700 1 -- 172.21.15.90:0/4091838128 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fb81ab390 con 0x7f0fb810a610 2024-04-14T15:14:09.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.532+0000 7f0fbf816700 1 -- 172.21.15.90:0/4091838128 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0fb81a8020 con 0x7f0fb810a610 2024-04-14T15:14:09.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.532+0000 7f0fbf816700 10 client.48316.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:14:09.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.532+0000 7f0fbf816700 1 -- 172.21.15.90:0/4091838128 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f0fb81a84e0 con 0x7f0fb810a610 2024-04-14T15:14:09.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.532+0000 7f0fbf816700 1 librados: init done 2024-04-14T15:14:09.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.532+0000 7f0fae7fc700 1 -- 172.21.15.90:0/4091838128 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f0fa8001d00 con 0x7f0fb810a610 2024-04-14T15:14:09.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.532+0000 7f0fae7fc700 1 -- 172.21.15.90:0/4091838128 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f0fa80048e0 con 0x7f0fb810a610 2024-04-14T15:14:09.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.533+0000 7f0fbf816700 1 -- 172.21.15.90:0/4091838128 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f9c005120 con 0x7f0fb810a610 2024-04-14T15:14:09.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.534+0000 7f0fae7fc700 1 -- 172.21.15.90:0/4091838128 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f0fa80036a0 con 0x7f0fb810a610 2024-04-14T15:14:09.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.534+0000 7f0fae7fc700 1 -- 172.21.15.90:0/4091838128 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(181..181 src has 51..181) ==== 9280+0+0 (unknown 577513725 0 0) 0x7f0fa8089980 con 0x7f0fb810a610 2024-04-14T15:14:09.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.534+0000 7f0fae7fc700 10 client.48316.objecter ms_dispatch 0x7f0fb8001550 osd_map(181..181 src has 51..181) 2024-04-14T15:14:09.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.534+0000 7f0fae7fc700 3 client.48316.objecter handle_osd_map got epochs [181,181] > 0 2024-04-14T15:14:09.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.534+0000 7f0fae7fc700 3 client.48316.objecter handle_osd_map decoding full epoch 181 2024-04-14T15:14:09.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.534+0000 7f0fae7fc700 20 client.48316.objecter dump_active .. 0 homeless 2024-04-14T15:14:09.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.535+0000 7f0fae7fc700 10 client.48316.objecter ms_handle_connect 0x7f0fa40757d0 2024-04-14T15:14:09.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.539+0000 7f0fae7fc700 1 -- 172.21.15.90:0/4091838128 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f0fa8055f30 con 0x7f0fb810a610 2024-04-14T15:14:09.546 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:14:09.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:09.570 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:09.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:14:09.671 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:14:09.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:09.697 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:09.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:14:09.736 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:14:09.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.755+0000 7f0fbf816700 1 -- 172.21.15.90:0/4091838128 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} v 0) -- 0x7f0f9c005410 con 0x7f0fb810a610 2024-04-14T15:14:09.798 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:14:09.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:09.822 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:09.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:14:09.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.830+0000 7f0fae7fc700 1 -- 172.21.15.90:0/4091838128 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]=0 set pool 1 pg_num to 11 v182) ==== 129+0+0 (unknown 1958648892 0 0) 0x7f0fa8059be0 con 0x7f0fb810a610 2024-04-14T15:14:09.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.833+0000 7f0fbf816700 10 librados: watch_flush enter 2024-04-14T15:14:09.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.833+0000 7f0fbf816700 10 librados: watch_flush exit 2024-04-14T15:14:09.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.833+0000 7f0fbf816700 20 client.48316.objecter shutdown clearing up homeless session... 2024-04-14T15:14:09.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.833+0000 7f0fbf816700 10 client.48316.objecter successfully canceled tick 2024-04-14T15:14:09.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.833+0000 7f0fbf816700 1 -- 172.21.15.90:0/4091838128 >> v1:172.21.15.90:6800/420935427 conn(0x7f0fa40757d0 legacy=0x7f0fa4077c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:09.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.833+0000 7f0fbf816700 1 -- 172.21.15.90:0/4091838128 >> v1:172.21.15.92:6789/0 conn(0x7f0fb810a610 legacy=0x7f0fb810c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:09.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.834+0000 7f0fbf816700 1 -- 172.21.15.90:0/4091838128 shutdown_connections 2024-04-14T15:14:09.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.834+0000 7f0fbf816700 1 -- 172.21.15.90:0/4091838128 >> 172.21.15.90:0/4091838128 conn(0x7f0fb80fd290 msgr2=0x7f0fb80fffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:14:09.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.834+0000 7f0fbf816700 1 -- 172.21.15.90:0/4091838128 shutdown_connections 2024-04-14T15:14:09.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.836+0000 7f0fbf816700 1 -- 172.21.15.90:0/4091838128 wait complete. 2024-04-14T15:14:09.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:09.836+0000 7f0fbf816700 1 librados: shutdown 2024-04-14T15:14:09.840 INFO:teuthology.orchestra.run.smithi090.stderr:set pool 1 pg_num to 11 2024-04-14T15:14:09.923 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:14:09.947 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:09.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:09.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:14:10.049 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:14:10.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:10.074 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:10.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:14:10.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:09 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-04-14T15:14:10.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:09 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/4091838128' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-04-14T15:14:10.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:09 smithi092 ceph-mon[33966]: from='client.48316 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-04-14T15:14:10.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:09 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3868448150' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi090-93724-16"}]': finished 2024-04-14T15:14:10.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:09 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "26"}]': finished 2024-04-14T15:14:10.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:09 smithi092 ceph-mon[33966]: from='client.48316 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-04-14T15:14:10.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:09 smithi092 ceph-mon[33966]: osdmap e182: 8 total, 8 up, 8 in 2024-04-14T15:14:10.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:09 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/844371530' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93601-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:10.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:09 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3546791986' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi090-93342-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:10.124 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:09 smithi092 ceph-mon[33966]: from='client.46400 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93601-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:10.124 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:09 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3307363110' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi090-93363-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:10.124 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:09 smithi092 ceph-mon[33966]: from='client.48307 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi090-93363-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:10.124 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:09 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:10.124 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:09 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1863705178' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi090-93724-21"}]: dispatch 2024-04-14T15:14:10.124 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:09 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1863705178' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi090-93724-21"}]: dispatch 2024-04-14T15:14:10.124 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:09 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1863705178' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi090-93724-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:14:10.168 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:09 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-04-14T15:14:10.168 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:09 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/4091838128' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-04-14T15:14:10.168 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:09 smithi090 ceph-mon[41670]: from='client.48316 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-04-14T15:14:10.168 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:09 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3868448150' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi090-93724-16"}]': finished 2024-04-14T15:14:10.168 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:09 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "26"}]': finished 2024-04-14T15:14:10.168 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:09 smithi090 ceph-mon[41670]: from='client.48316 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-04-14T15:14:10.169 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:09 smithi090 ceph-mon[41670]: osdmap e182: 8 total, 8 up, 8 in 2024-04-14T15:14:10.169 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:09 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/844371530' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93601-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:10.169 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:09 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3546791986' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi090-93342-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:10.169 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:09 smithi090 ceph-mon[41670]: from='client.46400 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93601-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:10.169 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:09 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3307363110' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi090-93363-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:10.169 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:09 smithi090 ceph-mon[41670]: from='client.48307 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi090-93363-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:10.169 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:09 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:10.169 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:09 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1863705178' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi090-93724-21"}]: dispatch 2024-04-14T15:14:10.169 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:09 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1863705178' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi090-93724-21"}]: dispatch 2024-04-14T15:14:10.169 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:09 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1863705178' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi090-93724-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:14:10.169 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:09 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-04-14T15:14:10.169 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:09 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/4091838128' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-04-14T15:14:10.170 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:09 smithi090 ceph-mon[30399]: from='client.48316 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-04-14T15:14:10.170 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:09 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3868448150' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi090-93724-16"}]': finished 2024-04-14T15:14:10.170 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:09 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "26"}]': finished 2024-04-14T15:14:10.170 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:09 smithi090 ceph-mon[30399]: from='client.48316 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-04-14T15:14:10.170 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:09 smithi090 ceph-mon[30399]: osdmap e182: 8 total, 8 up, 8 in 2024-04-14T15:14:10.170 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:09 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/844371530' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93601-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:10.170 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:09 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3546791986' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi090-93342-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:10.170 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:09 smithi090 ceph-mon[30399]: from='client.46400 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93601-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:10.170 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:09 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3307363110' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi090-93363-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:10.170 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:09 smithi090 ceph-mon[30399]: from='client.48307 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi090-93363-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:10.170 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:09 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:10.170 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:09 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1863705178' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi090-93724-21"}]: dispatch 2024-04-14T15:14:10.170 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:09 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1863705178' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi090-93724-21"}]: dispatch 2024-04-14T15:14:10.171 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:09 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1863705178' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi090-93724-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:14:10.176 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:14:10.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:10.200 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:10.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:14:10.301 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:14:10.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:10.326 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:10.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:14:10.428 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:10.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:10.453 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:10.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:10.538 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd set noscrub 2024-04-14T15:14:10.555 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:14:10.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:10.580 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:10.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:14:10.681 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:14:10.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:10.707 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:10.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:14:10.807 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:10.812 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-04-14T15:14:10.812 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-04-14T15:14:10.812 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-04-14T15:14:10.812 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-04-14T15:14:10.812 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-04-14T15:14:10.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:10.834 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:10.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:10.935 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:14:10.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:10.960 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:10.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:14:11.061 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:14:11.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:11.086 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:11.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:14:11.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:10 smithi092 ceph-mon[33966]: pgmap v218: 263 pgs: 7 creating+peering, 1 unknown, 255 active+clean; 592 KiB data, 908 MiB used, 714 GiB / 715 GiB avail 2024-04-14T15:14:11.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:10 smithi092 ceph-mon[33966]: 4.11 scrub starts 2024-04-14T15:14:11.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:10 smithi092 ceph-mon[33966]: 4.11 scrub ok 2024-04-14T15:14:11.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:10 smithi092 ceph-mon[33966]: 4.6 deep-scrub starts 2024-04-14T15:14:11.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:10 smithi092 ceph-mon[33966]: 4.6 deep-scrub ok 2024-04-14T15:14:11.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:10 smithi092 ceph-mon[33966]: 5.1b scrub starts 2024-04-14T15:14:11.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:10 smithi092 ceph-mon[33966]: 5.1b scrub ok 2024-04-14T15:14:11.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:10 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3546791986' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi090-93342-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:11.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:10 smithi092 ceph-mon[33966]: from='client.46400 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93601-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:11.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:10 smithi092 ceph-mon[33966]: from='client.48307 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi090-93363-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:11.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:10 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1863705178' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi090-93724-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:14:11.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:10 smithi092 ceph-mon[33966]: osdmap e183: 8 total, 8 up, 8 in 2024-04-14T15:14:11.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:10 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/844371530' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93601-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:11.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:10 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1863705178' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi090-93724-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi090-93724-21"}]: dispatch 2024-04-14T15:14:11.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:10 smithi092 ceph-mon[33966]: from='client.46400 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93601-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:11.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:10 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:11.187 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:14:11.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:10 smithi090 ceph-mon[41670]: pgmap v218: 263 pgs: 7 creating+peering, 1 unknown, 255 active+clean; 592 KiB data, 908 MiB used, 714 GiB / 715 GiB avail 2024-04-14T15:14:11.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:10 smithi090 ceph-mon[41670]: 4.11 scrub starts 2024-04-14T15:14:11.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:10 smithi090 ceph-mon[41670]: 4.11 scrub ok 2024-04-14T15:14:11.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:10 smithi090 ceph-mon[41670]: 4.6 deep-scrub starts 2024-04-14T15:14:11.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:10 smithi090 ceph-mon[41670]: 4.6 deep-scrub ok 2024-04-14T15:14:11.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:10 smithi090 ceph-mon[41670]: 5.1b scrub starts 2024-04-14T15:14:11.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:10 smithi090 ceph-mon[41670]: 5.1b scrub ok 2024-04-14T15:14:11.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:10 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3546791986' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi090-93342-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:11.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:10 smithi090 ceph-mon[41670]: from='client.46400 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93601-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:11.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:10 smithi090 ceph-mon[41670]: from='client.48307 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi090-93363-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:11.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:10 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1863705178' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi090-93724-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:14:11.195 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:10 smithi090 ceph-mon[41670]: osdmap e183: 8 total, 8 up, 8 in 2024-04-14T15:14:11.195 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:10 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/844371530' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93601-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:11.195 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:10 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1863705178' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi090-93724-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi090-93724-21"}]: dispatch 2024-04-14T15:14:11.195 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:10 smithi090 ceph-mon[41670]: from='client.46400 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93601-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:11.195 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:10 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:11.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:10 smithi090 ceph-mon[30399]: pgmap v218: 263 pgs: 7 creating+peering, 1 unknown, 255 active+clean; 592 KiB data, 908 MiB used, 714 GiB / 715 GiB avail 2024-04-14T15:14:11.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:10 smithi090 ceph-mon[30399]: 4.11 scrub starts 2024-04-14T15:14:11.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:10 smithi090 ceph-mon[30399]: 4.11 scrub ok 2024-04-14T15:14:11.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:10 smithi090 ceph-mon[30399]: 4.6 deep-scrub starts 2024-04-14T15:14:11.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:10 smithi090 ceph-mon[30399]: 4.6 deep-scrub ok 2024-04-14T15:14:11.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:10 smithi090 ceph-mon[30399]: 5.1b scrub starts 2024-04-14T15:14:11.196 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:10 smithi090 ceph-mon[30399]: 5.1b scrub ok 2024-04-14T15:14:11.196 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:10 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3546791986' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi090-93342-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:11.196 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:10 smithi090 ceph-mon[30399]: from='client.46400 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93601-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:11.196 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:10 smithi090 ceph-mon[30399]: from='client.48307 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi090-93363-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:11.196 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:10 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1863705178' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi090-93724-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:14:11.196 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:10 smithi090 ceph-mon[30399]: osdmap e183: 8 total, 8 up, 8 in 2024-04-14T15:14:11.196 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:10 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/844371530' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93601-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:11.196 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:10 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1863705178' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi090-93724-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi090-93724-21"}]: dispatch 2024-04-14T15:14:11.196 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:10 smithi090 ceph-mon[30399]: from='client.46400 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93601-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:11.196 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:10 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:11.214 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:11.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:11.215 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:14:11.218 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:14:11.316 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:11.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:11.341 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:11.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:11.442 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:14:11.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:11.466 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:11.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:14:11.567 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:14:11.592 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-14T15:14:11.594 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:11.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:11.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:14:11.698 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:14:11.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:11.723 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:11.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:14:11.824 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:14:11.838 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:14:11.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:11.865 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:11.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:14:11.966 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:14:11.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:11.991 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:11.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:14:12.093 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:14:12.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:12.119 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:12.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:14:12.130 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:11 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-04-14T15:14:12.130 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:11 smithi090 ceph-mon[41670]: 2.13 scrub starts 2024-04-14T15:14:12.130 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:11 smithi090 ceph-mon[41670]: 2.13 scrub ok 2024-04-14T15:14:12.130 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:11 smithi090 ceph-mon[41670]: from='client.46400 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93601-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:12.130 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:11 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-04-14T15:14:12.130 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:11 smithi090 ceph-mon[41670]: osdmap e184: 8 total, 8 up, 8 in 2024-04-14T15:14:12.131 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:11 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:12.131 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:11 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1240983831' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi090-93342-27"}]: dispatch 2024-04-14T15:14:12.131 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:11 smithi090 ceph-mon[41670]: from='client.48328 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi090-93342-27"}]: dispatch 2024-04-14T15:14:12.131 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:11 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1240983831' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi090-93342-27"}]: dispatch 2024-04-14T15:14:12.131 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:11 smithi090 ceph-mon[41670]: from='client.48328 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi090-93342-27"}]: dispatch 2024-04-14T15:14:12.131 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:11 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1240983831' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi090-93342-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:14:12.131 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:11 smithi090 ceph-mon[41670]: from='client.48328 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi090-93342-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:14:12.131 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:11 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-04-14T15:14:12.131 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:11 smithi090 ceph-mon[30399]: 2.13 scrub starts 2024-04-14T15:14:12.131 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:11 smithi090 ceph-mon[30399]: 2.13 scrub ok 2024-04-14T15:14:12.131 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:11 smithi090 ceph-mon[30399]: from='client.46400 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93601-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:12.131 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:11 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-04-14T15:14:12.132 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:11 smithi090 ceph-mon[30399]: osdmap e184: 8 total, 8 up, 8 in 2024-04-14T15:14:12.132 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:11 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:12.132 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:11 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1240983831' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi090-93342-27"}]: dispatch 2024-04-14T15:14:12.132 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:11 smithi090 ceph-mon[30399]: from='client.48328 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi090-93342-27"}]: dispatch 2024-04-14T15:14:12.132 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:11 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1240983831' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi090-93342-27"}]: dispatch 2024-04-14T15:14:12.132 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:11 smithi090 ceph-mon[30399]: from='client.48328 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi090-93342-27"}]: dispatch 2024-04-14T15:14:12.132 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:11 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1240983831' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi090-93342-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:14:12.132 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:11 smithi090 ceph-mon[30399]: from='client.48328 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi090-93342-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:14:12.221 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:12.245 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:12.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:12.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:12.293 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:11 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-04-14T15:14:12.293 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:11 smithi092 ceph-mon[33966]: 2.13 scrub starts 2024-04-14T15:14:12.293 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:11 smithi092 ceph-mon[33966]: 2.13 scrub ok 2024-04-14T15:14:12.293 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:11 smithi092 ceph-mon[33966]: from='client.46400 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93601-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:12.293 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:11 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-04-14T15:14:12.293 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:11 smithi092 ceph-mon[33966]: osdmap e184: 8 total, 8 up, 8 in 2024-04-14T15:14:12.293 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:11 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:12.293 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:11 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1240983831' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi090-93342-27"}]: dispatch 2024-04-14T15:14:12.293 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:11 smithi092 ceph-mon[33966]: from='client.48328 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi090-93342-27"}]: dispatch 2024-04-14T15:14:12.293 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:11 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1240983831' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi090-93342-27"}]: dispatch 2024-04-14T15:14:12.293 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:11 smithi092 ceph-mon[33966]: from='client.48328 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi090-93342-27"}]: dispatch 2024-04-14T15:14:12.294 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:11 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1240983831' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi090-93342-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:14:12.294 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:11 smithi092 ceph-mon[33966]: from='client.48328 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi090-93342-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:14:12.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.314+0000 7fd0326db700 1 -- 172.21.15.90:0/865306178 >> v1:172.21.15.90:6789/0 conn(0x7fd02c10a610 legacy=0x7fd02c10ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:12.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.314+0000 7fd0326db700 1 -- 172.21.15.90:0/865306178 shutdown_connections 2024-04-14T15:14:12.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.314+0000 7fd0326db700 1 -- 172.21.15.90:0/865306178 >> 172.21.15.90:0/865306178 conn(0x7fd02c0fd290 msgr2=0x7fd02c0ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:14:12.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.314+0000 7fd0326db700 1 -- 172.21.15.90:0/865306178 shutdown_connections 2024-04-14T15:14:12.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.314+0000 7fd0326db700 1 -- 172.21.15.90:0/865306178 wait complete. 2024-04-14T15:14:12.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.314+0000 7fd0326db700 1 librados: starting msgr at 2024-04-14T15:14:12.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.314+0000 7fd0326db700 1 librados: starting objecter 2024-04-14T15:14:12.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.314+0000 7fd0326db700 1 Processor -- start 2024-04-14T15:14:12.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.314+0000 7fd0326db700 1 -- start start 2024-04-14T15:14:12.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.314+0000 7fd0326db700 1 librados: setting wanted keys 2024-04-14T15:14:12.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.314+0000 7fd0326db700 1 librados: calling monclient init 2024-04-14T15:14:12.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.315+0000 7fd0326db700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd02c1a7990 con 0x7fd02c107760 2024-04-14T15:14:12.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.315+0000 7fd0326db700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd02c1a8bb0 con 0x7fd02c10a610 2024-04-14T15:14:12.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.315+0000 7fd0326db700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd02c1a9df0 con 0x7fd02c10e1e0 2024-04-14T15:14:12.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.315+0000 7fd02bfff700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fd02c107760 0x7fd02c10da70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:57228/0 (socket says 172.21.15.90:57228) 2024-04-14T15:14:12.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.315+0000 7fd02bfff700 1 -- 172.21.15.90:0/2439123785 learned_addr learned my addr 172.21.15.90:0/2439123785 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:14:12.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.315+0000 7fd0297fa700 10 client.?.objecter ms_handle_connect 0x7fd02c107760 2024-04-14T15:14:12.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.315+0000 7fd0297fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:14:12.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.315+0000 7fd0297fa700 10 client.?.objecter ms_handle_connect 0x7fd02c10e1e0 2024-04-14T15:14:12.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.315+0000 7fd0297fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:14:12.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.315+0000 7fd0297fa700 10 client.?.objecter ms_handle_connect 0x7fd02c10a610 2024-04-14T15:14:12.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.315+0000 7fd0297fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:14:12.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.315+0000 7fd0297fa700 1 -- 172.21.15.90:0/2439123785 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1578448476 0 0) 0x7fd02c1a7990 con 0x7fd02c107760 2024-04-14T15:14:12.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.315+0000 7fd0297fa700 1 -- 172.21.15.90:0/2439123785 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd0140035b0 con 0x7fd02c107760 2024-04-14T15:14:12.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.316+0000 7fd0297fa700 1 -- 172.21.15.90:0/2439123785 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1163644092 0 0) 0x7fd02c1a8bb0 con 0x7fd02c10a610 2024-04-14T15:14:12.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.316+0000 7fd0297fa700 1 -- 172.21.15.90:0/2439123785 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd02c1a7990 con 0x7fd02c10a610 2024-04-14T15:14:12.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.316+0000 7fd0297fa700 1 -- 172.21.15.90:0/2439123785 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1289707389 0 0) 0x7fd02c1a9df0 con 0x7fd02c10e1e0 2024-04-14T15:14:12.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.316+0000 7fd0297fa700 1 -- 172.21.15.90:0/2439123785 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd02c1a8bb0 con 0x7fd02c10e1e0 2024-04-14T15:14:12.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.316+0000 7fd0297fa700 1 -- 172.21.15.90:0/2439123785 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2966082549 0 0) 0x7fd0140035b0 con 0x7fd02c107760 2024-04-14T15:14:12.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.316+0000 7fd0297fa700 1 -- 172.21.15.90:0/2439123785 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd02c1a9df0 con 0x7fd02c107760 2024-04-14T15:14:12.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.316+0000 7fd0297fa700 1 -- 172.21.15.90:0/2439123785 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 305771227 0 0) 0x7fd02c1a7990 con 0x7fd02c10a610 2024-04-14T15:14:12.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.316+0000 7fd0297fa700 1 -- 172.21.15.90:0/2439123785 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd0140035b0 con 0x7fd02c10a610 2024-04-14T15:14:12.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.316+0000 7fd0297fa700 1 -- 172.21.15.90:0/2439123785 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 423007613 0 0) 0x7fd02c1a8bb0 con 0x7fd02c10e1e0 2024-04-14T15:14:12.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.316+0000 7fd0297fa700 1 -- 172.21.15.90:0/2439123785 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd02c1a7990 con 0x7fd02c10e1e0 2024-04-14T15:14:12.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.316+0000 7fd0297fa700 1 -- 172.21.15.90:0/2439123785 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fd01c0018f0 con 0x7fd02c107760 2024-04-14T15:14:12.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.316+0000 7fd0297fa700 1 -- 172.21.15.90:0/2439123785 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fd020004620 con 0x7fd02c10a610 2024-04-14T15:14:12.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.316+0000 7fd0297fa700 1 -- 172.21.15.90:0/2439123785 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fd024001b30 con 0x7fd02c10e1e0 2024-04-14T15:14:12.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.316+0000 7fd0297fa700 1 -- 172.21.15.90:0/2439123785 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1276367413 0 0) 0x7fd0140035b0 con 0x7fd02c10a610 2024-04-14T15:14:12.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.316+0000 7fd0297fa700 1 -- 172.21.15.90:0/2439123785 >> v1:172.21.15.90:6790/0 conn(0x7fd02c10e1e0 legacy=0x7fd02c1a6050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:12.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.316+0000 7fd0297fa700 1 -- 172.21.15.90:0/2439123785 >> v1:172.21.15.90:6789/0 conn(0x7fd02c107760 legacy=0x7fd02c10da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:12.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.316+0000 7fd0297fa700 1 -- 172.21.15.90:0/2439123785 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd02c1ab030 con 0x7fd02c10a610 2024-04-14T15:14:12.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.316+0000 7fd0326db700 1 -- 172.21.15.90:0/2439123785 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd02c1aa020 con 0x7fd02c10a610 2024-04-14T15:14:12.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.316+0000 7fd0326db700 10 client.48340.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:14:12.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.316+0000 7fd0326db700 1 -- 172.21.15.90:0/2439123785 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd02c1aa4e0 con 0x7fd02c10a610 2024-04-14T15:14:12.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.316+0000 7fd0326db700 1 librados: init done 2024-04-14T15:14:12.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.316+0000 7fd0297fa700 1 -- 172.21.15.90:0/2439123785 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fd0200011a0 con 0x7fd02c10a610 2024-04-14T15:14:12.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.316+0000 7fd0297fa700 1 -- 172.21.15.90:0/2439123785 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fd020004c90 con 0x7fd02c10a610 2024-04-14T15:14:12.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.317+0000 7fd0326db700 1 -- 172.21.15.90:0/2439123785 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fd018005100 con 0x7fd02c10a610 2024-04-14T15:14:12.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.318+0000 7fd0297fa700 1 -- 172.21.15.90:0/2439123785 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fd020003680 con 0x7fd02c10a610 2024-04-14T15:14:12.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.319+0000 7fd0297fa700 1 -- 172.21.15.90:0/2439123785 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(184..184 src has 51..184) ==== 10036+0+0 (unknown 52713893 0 0) 0x7fd020089f40 con 0x7fd02c10a610 2024-04-14T15:14:12.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.319+0000 7fd0297fa700 10 client.48340.objecter ms_dispatch 0x7fd02c001550 osd_map(184..184 src has 51..184) 2024-04-14T15:14:12.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.319+0000 7fd0297fa700 3 client.48340.objecter handle_osd_map got epochs [184,184] > 0 2024-04-14T15:14:12.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.319+0000 7fd0297fa700 3 client.48340.objecter handle_osd_map decoding full epoch 184 2024-04-14T15:14:12.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.319+0000 7fd0297fa700 20 client.48340.objecter dump_active .. 0 homeless 2024-04-14T15:14:12.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.319+0000 7fd0297fa700 1 -- 172.21.15.90:0/2439123785 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v184) ==== 54+0+16 (unknown 3164069218 0 2420000192) 0x7fd020056200 con 0x7fd02c10a610 2024-04-14T15:14:12.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.319+0000 7fd0297fa700 10 client.48340.objecter ms_handle_connect 0x7fd014075760 2024-04-14T15:14:12.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.319+0000 7fd0326db700 10 client.48340.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:14:12.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.319+0000 7fd0326db700 15 client.48340.objecter _session_command_op_assign -1 1 2024-04-14T15:14:12.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.319+0000 7fd0326db700 20 client.48340.objecter _get_session s=0x7fd0180053e0 osd=0 3 2024-04-14T15:14:12.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.319+0000 7fd0326db700 20 client.48340.objecter put_session s=0x7fd0180053e0 osd=0 3 2024-04-14T15:14:12.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.319+0000 7fd0326db700 20 client.48340.objecter _get_session s=0x7fd0180053e0 osd=0 3 2024-04-14T15:14:12.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.319+0000 7fd0326db700 15 client.48340.objecter _session_command_op_remove -1 1 2024-04-14T15:14:12.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.319+0000 7fd0326db700 20 client.48340.objecter get_session s=0x7fd0180053e0 osd=0 3 2024-04-14T15:14:12.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.319+0000 7fd0326db700 15 client.48340.objecter _session_command_op_assign 0 1 2024-04-14T15:14:12.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.319+0000 7fd0326db700 20 client.48340.objecter put_session s=0x7fd0180053e0 osd=0 4 2024-04-14T15:14:12.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.319+0000 7fd0326db700 10 client.48340.objecter _send_command 1 2024-04-14T15:14:12.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.319+0000 7fd0326db700 1 -- 172.21.15.90:0/2439123785 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd018004b50 con 0x7fd018004730 2024-04-14T15:14:12.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.319+0000 7fd0297fa700 10 client.48340.objecter ms_handle_connect 0x7fd018004730 2024-04-14T15:14:12.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.320+0000 7fd0297fa700 1 -- 172.21.15.90:0/2439123785 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd018004b50 con 0x7fd018004730 2024-04-14T15:14:12.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.320+0000 7fd0297fa700 10 client.48340.objecter ms_dispatch 0x7fd02c001550 command_reply(tid 1: 0 ) 2024-04-14T15:14:12.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.320+0000 7fd0297fa700 10 client.48340.objecter _finish_command 1 = system:0 2024-04-14T15:14:12.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.320+0000 7fd0297fa700 20 client.48340.objecter put_session s=0x7fd0180053e0 osd=0 4 2024-04-14T15:14:12.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.320+0000 7fd0297fa700 15 client.48340.objecter _session_command_op_remove 0 1 2024-04-14T15:14:12.348 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:14:12.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.337+0000 7fd0326db700 10 client.48340.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:14:12.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.337+0000 7fd0326db700 15 client.48340.objecter _session_command_op_assign -1 2 2024-04-14T15:14:12.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.337+0000 7fd0326db700 20 client.48340.objecter _get_session s=0x7fd0180053e0 osd=0 3 2024-04-14T15:14:12.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.337+0000 7fd0326db700 20 client.48340.objecter put_session s=0x7fd0180053e0 osd=0 3 2024-04-14T15:14:12.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.337+0000 7fd0326db700 20 client.48340.objecter _get_session s=0x7fd0180053e0 osd=0 3 2024-04-14T15:14:12.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.337+0000 7fd0326db700 15 client.48340.objecter _session_command_op_remove -1 2 2024-04-14T15:14:12.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.337+0000 7fd0326db700 20 client.48340.objecter get_session s=0x7fd0180053e0 osd=0 3 2024-04-14T15:14:12.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.337+0000 7fd0326db700 15 client.48340.objecter _session_command_op_assign 0 2 2024-04-14T15:14:12.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.337+0000 7fd0326db700 20 client.48340.objecter put_session s=0x7fd0180053e0 osd=0 4 2024-04-14T15:14:12.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.337+0000 7fd0326db700 10 client.48340.objecter _send_command 2 2024-04-14T15:14:12.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.337+0000 7fd0326db700 1 -- 172.21.15.90:0/2439123785 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd018007b00 con 0x7fd018004730 2024-04-14T15:14:12.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.338+0000 7fd0297fa700 1 -- 172.21.15.90:0/2439123785 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd018007b00 con 0x7fd018004730 2024-04-14T15:14:12.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.338+0000 7fd0297fa700 10 client.48340.objecter ms_dispatch 0x7fd02c001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:14:12.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.338+0000 7fd0297fa700 10 client.48340.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:14:12.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.338+0000 7fd0297fa700 20 client.48340.objecter put_session s=0x7fd0180053e0 osd=0 4 2024-04-14T15:14:12.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.338+0000 7fd0297fa700 15 client.48340.objecter _session_command_op_remove 0 2 2024-04-14T15:14:12.351 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:14:12.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.339+0000 7fd0326db700 10 client.48340.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:14:12.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.339+0000 7fd0326db700 15 client.48340.objecter _session_command_op_assign -1 3 2024-04-14T15:14:12.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.339+0000 7fd0326db700 20 client.48340.objecter _get_session s=0x7fd018004180 osd=1 3 2024-04-14T15:14:12.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.339+0000 7fd0326db700 20 client.48340.objecter put_session s=0x7fd018004180 osd=1 3 2024-04-14T15:14:12.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.339+0000 7fd0326db700 20 client.48340.objecter _get_session s=0x7fd018004180 osd=1 3 2024-04-14T15:14:12.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.339+0000 7fd0326db700 15 client.48340.objecter _session_command_op_remove -1 3 2024-04-14T15:14:12.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.339+0000 7fd0326db700 20 client.48340.objecter get_session s=0x7fd018004180 osd=1 3 2024-04-14T15:14:12.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.339+0000 7fd0326db700 15 client.48340.objecter _session_command_op_assign 1 3 2024-04-14T15:14:12.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.339+0000 7fd0326db700 20 client.48340.objecter put_session s=0x7fd018004180 osd=1 4 2024-04-14T15:14:12.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.339+0000 7fd0326db700 10 client.48340.objecter _send_command 3 2024-04-14T15:14:12.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.339+0000 7fd0326db700 1 -- 172.21.15.90:0/2439123785 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fd018002190 con 0x7fd0180042f0 2024-04-14T15:14:12.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.339+0000 7fd0297fa700 10 client.48340.objecter ms_handle_connect 0x7fd0180042f0 2024-04-14T15:14:12.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.340+0000 7fd0297fa700 1 -- 172.21.15.90:0/2439123785 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd018002190 con 0x7fd0180042f0 2024-04-14T15:14:12.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.340+0000 7fd0297fa700 10 client.48340.objecter ms_dispatch 0x7fd02c001550 command_reply(tid 3: 0 ) 2024-04-14T15:14:12.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.340+0000 7fd0297fa700 10 client.48340.objecter _finish_command 3 = system:0 2024-04-14T15:14:12.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.340+0000 7fd0297fa700 20 client.48340.objecter put_session s=0x7fd018004180 osd=1 4 2024-04-14T15:14:12.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.340+0000 7fd0297fa700 15 client.48340.objecter _session_command_op_remove 1 3 2024-04-14T15:14:12.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.356+0000 7fd0326db700 10 client.48340.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:14:12.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.356+0000 7fd0326db700 15 client.48340.objecter _session_command_op_assign -1 4 2024-04-14T15:14:12.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.356+0000 7fd0326db700 20 client.48340.objecter _get_session s=0x7fd018004180 osd=1 3 2024-04-14T15:14:12.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.356+0000 7fd0326db700 20 client.48340.objecter put_session s=0x7fd018004180 osd=1 3 2024-04-14T15:14:12.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.356+0000 7fd0326db700 20 client.48340.objecter _get_session s=0x7fd018004180 osd=1 3 2024-04-14T15:14:12.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.356+0000 7fd0326db700 15 client.48340.objecter _session_command_op_remove -1 4 2024-04-14T15:14:12.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.356+0000 7fd0326db700 20 client.48340.objecter get_session s=0x7fd018004180 osd=1 3 2024-04-14T15:14:12.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.356+0000 7fd0326db700 15 client.48340.objecter _session_command_op_assign 1 4 2024-04-14T15:14:12.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.356+0000 7fd0326db700 20 client.48340.objecter put_session s=0x7fd018004180 osd=1 4 2024-04-14T15:14:12.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.356+0000 7fd0326db700 10 client.48340.objecter _send_command 4 2024-04-14T15:14:12.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.356+0000 7fd0326db700 1 -- 172.21.15.90:0/2439123785 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd018001600 con 0x7fd0180042f0 2024-04-14T15:14:12.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.358+0000 7fd0297fa700 1 -- 172.21.15.90:0/2439123785 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd018001600 con 0x7fd0180042f0 2024-04-14T15:14:12.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.358+0000 7fd0297fa700 10 client.48340.objecter ms_dispatch 0x7fd02c001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:14:12.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.358+0000 7fd0297fa700 10 client.48340.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:14:12.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.358+0000 7fd0297fa700 20 client.48340.objecter put_session s=0x7fd018004180 osd=1 4 2024-04-14T15:14:12.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.358+0000 7fd0297fa700 15 client.48340.objecter _session_command_op_remove 1 4 2024-04-14T15:14:12.367 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:14:12.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.358+0000 7fd0326db700 10 client.48340.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:14:12.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.358+0000 7fd0326db700 15 client.48340.objecter _session_command_op_assign -1 5 2024-04-14T15:14:12.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.358+0000 7fd0326db700 20 client.48340.objecter _get_session s=0x7fd018008480 osd=2 3 2024-04-14T15:14:12.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.358+0000 7fd0326db700 20 client.48340.objecter put_session s=0x7fd018008480 osd=2 3 2024-04-14T15:14:12.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.358+0000 7fd0326db700 20 client.48340.objecter _get_session s=0x7fd018008480 osd=2 3 2024-04-14T15:14:12.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.358+0000 7fd0326db700 15 client.48340.objecter _session_command_op_remove -1 5 2024-04-14T15:14:12.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.358+0000 7fd0326db700 20 client.48340.objecter get_session s=0x7fd018008480 osd=2 3 2024-04-14T15:14:12.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.358+0000 7fd0326db700 15 client.48340.objecter _session_command_op_assign 2 5 2024-04-14T15:14:12.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.358+0000 7fd0326db700 20 client.48340.objecter put_session s=0x7fd018008480 osd=2 4 2024-04-14T15:14:12.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.358+0000 7fd0326db700 10 client.48340.objecter _send_command 5 2024-04-14T15:14:12.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.358+0000 7fd0326db700 1 -- 172.21.15.90:0/2439123785 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fd0180111f0 con 0x7fd01800d6e0 2024-04-14T15:14:12.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.358+0000 7fd0297fa700 10 client.48340.objecter ms_handle_connect 0x7fd01800d6e0 2024-04-14T15:14:12.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.359+0000 7fd0297fa700 1 -- 172.21.15.90:0/2439123785 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd0180111f0 con 0x7fd01800d6e0 2024-04-14T15:14:12.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.359+0000 7fd0297fa700 10 client.48340.objecter ms_dispatch 0x7fd02c001550 command_reply(tid 5: 0 ) 2024-04-14T15:14:12.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.359+0000 7fd0297fa700 10 client.48340.objecter _finish_command 5 = system:0 2024-04-14T15:14:12.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.359+0000 7fd0297fa700 20 client.48340.objecter put_session s=0x7fd018008480 osd=2 4 2024-04-14T15:14:12.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.359+0000 7fd0297fa700 15 client.48340.objecter _session_command_op_remove 2 5 2024-04-14T15:14:12.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.375+0000 7fd0326db700 10 client.48340.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:14:12.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.375+0000 7fd0326db700 15 client.48340.objecter _session_command_op_assign -1 6 2024-04-14T15:14:12.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.375+0000 7fd0326db700 20 client.48340.objecter _get_session s=0x7fd018008480 osd=2 3 2024-04-14T15:14:12.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.375+0000 7fd0326db700 20 client.48340.objecter put_session s=0x7fd018008480 osd=2 3 2024-04-14T15:14:12.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.375+0000 7fd0326db700 20 client.48340.objecter _get_session s=0x7fd018008480 osd=2 3 2024-04-14T15:14:12.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.375+0000 7fd0326db700 15 client.48340.objecter _session_command_op_remove -1 6 2024-04-14T15:14:12.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.375+0000 7fd0326db700 20 client.48340.objecter get_session s=0x7fd018008480 osd=2 3 2024-04-14T15:14:12.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.375+0000 7fd0326db700 15 client.48340.objecter _session_command_op_assign 2 6 2024-04-14T15:14:12.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.375+0000 7fd0326db700 20 client.48340.objecter put_session s=0x7fd018008480 osd=2 4 2024-04-14T15:14:12.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.375+0000 7fd0326db700 10 client.48340.objecter _send_command 6 2024-04-14T15:14:12.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.375+0000 7fd0326db700 1 -- 172.21.15.90:0/2439123785 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd018007fa0 con 0x7fd01800d6e0 2024-04-14T15:14:12.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.377+0000 7fd0297fa700 1 -- 172.21.15.90:0/2439123785 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd018007fa0 con 0x7fd01800d6e0 2024-04-14T15:14:12.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.377+0000 7fd0297fa700 10 client.48340.objecter ms_dispatch 0x7fd02c001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:14:12.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.377+0000 7fd0297fa700 10 client.48340.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:14:12.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.377+0000 7fd0297fa700 20 client.48340.objecter put_session s=0x7fd018008480 osd=2 4 2024-04-14T15:14:12.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.377+0000 7fd0297fa700 15 client.48340.objecter _session_command_op_remove 2 6 2024-04-14T15:14:12.378 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:14:12.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.377+0000 7fd0326db700 10 client.48340.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:14:12.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.377+0000 7fd0326db700 15 client.48340.objecter _session_command_op_assign -1 7 2024-04-14T15:14:12.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.377+0000 7fd0326db700 20 client.48340.objecter _get_session s=0x7fd01800cd30 osd=3 3 2024-04-14T15:14:12.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.377+0000 7fd0326db700 20 client.48340.objecter put_session s=0x7fd01800cd30 osd=3 3 2024-04-14T15:14:12.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.377+0000 7fd0326db700 20 client.48340.objecter _get_session s=0x7fd01800cd30 osd=3 3 2024-04-14T15:14:12.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.377+0000 7fd0326db700 15 client.48340.objecter _session_command_op_remove -1 7 2024-04-14T15:14:12.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.377+0000 7fd0326db700 20 client.48340.objecter get_session s=0x7fd01800cd30 osd=3 3 2024-04-14T15:14:12.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.377+0000 7fd0326db700 15 client.48340.objecter _session_command_op_assign 3 7 2024-04-14T15:14:12.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.377+0000 7fd0326db700 20 client.48340.objecter put_session s=0x7fd01800cd30 osd=3 4 2024-04-14T15:14:12.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.377+0000 7fd0326db700 10 client.48340.objecter _send_command 7 2024-04-14T15:14:12.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.377+0000 7fd0326db700 1 -- 172.21.15.90:0/2439123785 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fd01800d4e0 con 0x7fd018012520 2024-04-14T15:14:12.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.377+0000 7fd0297fa700 10 client.48340.objecter ms_handle_connect 0x7fd018012520 2024-04-14T15:14:12.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.378+0000 7fd0297fa700 1 -- 172.21.15.90:0/2439123785 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd01800d4e0 con 0x7fd018012520 2024-04-14T15:14:12.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.378+0000 7fd0297fa700 10 client.48340.objecter ms_dispatch 0x7fd02c001550 command_reply(tid 7: 0 ) 2024-04-14T15:14:12.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.378+0000 7fd0297fa700 10 client.48340.objecter _finish_command 7 = system:0 2024-04-14T15:14:12.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.378+0000 7fd0297fa700 20 client.48340.objecter put_session s=0x7fd01800cd30 osd=3 4 2024-04-14T15:14:12.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.378+0000 7fd0297fa700 15 client.48340.objecter _session_command_op_remove 3 7 2024-04-14T15:14:12.393 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:12.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:12.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:14:12.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.395+0000 7fd0326db700 10 client.48340.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:14:12.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.395+0000 7fd0326db700 15 client.48340.objecter _session_command_op_assign -1 8 2024-04-14T15:14:12.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.395+0000 7fd0326db700 20 client.48340.objecter _get_session s=0x7fd01800cd30 osd=3 3 2024-04-14T15:14:12.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.395+0000 7fd0326db700 20 client.48340.objecter put_session s=0x7fd01800cd30 osd=3 3 2024-04-14T15:14:12.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.395+0000 7fd0326db700 20 client.48340.objecter _get_session s=0x7fd01800cd30 osd=3 3 2024-04-14T15:14:12.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.395+0000 7fd0326db700 15 client.48340.objecter _session_command_op_remove -1 8 2024-04-14T15:14:12.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.395+0000 7fd0326db700 20 client.48340.objecter get_session s=0x7fd01800cd30 osd=3 3 2024-04-14T15:14:12.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.395+0000 7fd0326db700 15 client.48340.objecter _session_command_op_assign 3 8 2024-04-14T15:14:12.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.395+0000 7fd0326db700 20 client.48340.objecter put_session s=0x7fd01800cd30 osd=3 4 2024-04-14T15:14:12.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.395+0000 7fd0326db700 10 client.48340.objecter _send_command 8 2024-04-14T15:14:12.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.395+0000 7fd0326db700 1 -- 172.21.15.90:0/2439123785 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd01800c8d0 con 0x7fd018012520 2024-04-14T15:14:12.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.397+0000 7fd0297fa700 1 -- 172.21.15.90:0/2439123785 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd01800c8d0 con 0x7fd018012520 2024-04-14T15:14:12.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.397+0000 7fd0297fa700 10 client.48340.objecter ms_dispatch 0x7fd02c001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:14:12.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.397+0000 7fd0297fa700 10 client.48340.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:14:12.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.397+0000 7fd0297fa700 20 client.48340.objecter put_session s=0x7fd01800cd30 osd=3 4 2024-04-14T15:14:12.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.397+0000 7fd0297fa700 15 client.48340.objecter _session_command_op_remove 3 8 2024-04-14T15:14:12.397 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:14:12.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.397+0000 7fd0326db700 10 client.48340.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:14:12.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.397+0000 7fd0326db700 15 client.48340.objecter _session_command_op_assign -1 9 2024-04-14T15:14:12.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.397+0000 7fd0326db700 20 client.48340.objecter _get_session s=0x7fd01800c680 osd=4 3 2024-04-14T15:14:12.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.397+0000 7fd0326db700 20 client.48340.objecter put_session s=0x7fd01800c680 osd=4 3 2024-04-14T15:14:12.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.397+0000 7fd0326db700 20 client.48340.objecter _get_session s=0x7fd01800c680 osd=4 3 2024-04-14T15:14:12.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.397+0000 7fd0326db700 15 client.48340.objecter _session_command_op_remove -1 9 2024-04-14T15:14:12.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.397+0000 7fd0326db700 20 client.48340.objecter get_session s=0x7fd01800c680 osd=4 3 2024-04-14T15:14:12.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.397+0000 7fd0326db700 15 client.48340.objecter _session_command_op_assign 4 9 2024-04-14T15:14:12.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.397+0000 7fd0326db700 20 client.48340.objecter put_session s=0x7fd01800c680 osd=4 4 2024-04-14T15:14:12.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.397+0000 7fd0326db700 10 client.48340.objecter _send_command 9 2024-04-14T15:14:12.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.397+0000 7fd0326db700 1 -- 172.21.15.90:0/2439123785 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fd01800c8b0 con 0x7fd018012080 2024-04-14T15:14:12.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.397+0000 7fd0297fa700 10 client.48340.objecter ms_handle_connect 0x7fd018012080 2024-04-14T15:14:12.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.398+0000 7fd0297fa700 1 -- 172.21.15.90:0/2439123785 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd01800c8b0 con 0x7fd018012080 2024-04-14T15:14:12.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.398+0000 7fd0297fa700 10 client.48340.objecter ms_dispatch 0x7fd02c001550 command_reply(tid 9: 0 ) 2024-04-14T15:14:12.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.398+0000 7fd0297fa700 10 client.48340.objecter _finish_command 9 = system:0 2024-04-14T15:14:12.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.398+0000 7fd0297fa700 20 client.48340.objecter put_session s=0x7fd01800c680 osd=4 4 2024-04-14T15:14:12.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.398+0000 7fd0297fa700 15 client.48340.objecter _session_command_op_remove 4 9 2024-04-14T15:14:12.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.414+0000 7fd0326db700 10 client.48340.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:14:12.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.414+0000 7fd0326db700 15 client.48340.objecter _session_command_op_assign -1 10 2024-04-14T15:14:12.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.414+0000 7fd0326db700 20 client.48340.objecter _get_session s=0x7fd01800c680 osd=4 3 2024-04-14T15:14:12.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.414+0000 7fd0326db700 20 client.48340.objecter put_session s=0x7fd01800c680 osd=4 3 2024-04-14T15:14:12.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.414+0000 7fd0326db700 20 client.48340.objecter _get_session s=0x7fd01800c680 osd=4 3 2024-04-14T15:14:12.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.414+0000 7fd0326db700 15 client.48340.objecter _session_command_op_remove -1 10 2024-04-14T15:14:12.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.414+0000 7fd0326db700 20 client.48340.objecter get_session s=0x7fd01800c680 osd=4 3 2024-04-14T15:14:12.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.414+0000 7fd0326db700 15 client.48340.objecter _session_command_op_assign 4 10 2024-04-14T15:14:12.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.414+0000 7fd0326db700 20 client.48340.objecter put_session s=0x7fd01800c680 osd=4 4 2024-04-14T15:14:12.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.414+0000 7fd0326db700 10 client.48340.objecter _send_command 10 2024-04-14T15:14:12.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.414+0000 7fd0326db700 1 -- 172.21.15.90:0/2439123785 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd018017790 con 0x7fd018012080 2024-04-14T15:14:12.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.416+0000 7fd0297fa700 1 -- 172.21.15.90:0/2439123785 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd018017790 con 0x7fd018012080 2024-04-14T15:14:12.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.416+0000 7fd0297fa700 10 client.48340.objecter ms_dispatch 0x7fd02c001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:14:12.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.416+0000 7fd0297fa700 10 client.48340.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:14:12.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.416+0000 7fd0297fa700 20 client.48340.objecter put_session s=0x7fd01800c680 osd=4 4 2024-04-14T15:14:12.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.416+0000 7fd0297fa700 15 client.48340.objecter _session_command_op_remove 4 10 2024-04-14T15:14:12.417 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:14:12.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.416+0000 7fd0326db700 10 client.48340.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:14:12.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.416+0000 7fd0326db700 15 client.48340.objecter _session_command_op_assign -1 11 2024-04-14T15:14:12.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.416+0000 7fd0326db700 20 client.48340.objecter _get_session s=0x7fd01800c910 osd=5 3 2024-04-14T15:14:12.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.416+0000 7fd0326db700 20 client.48340.objecter put_session s=0x7fd01800c910 osd=5 3 2024-04-14T15:14:12.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.416+0000 7fd0326db700 20 client.48340.objecter _get_session s=0x7fd01800c910 osd=5 3 2024-04-14T15:14:12.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.416+0000 7fd0326db700 15 client.48340.objecter _session_command_op_remove -1 11 2024-04-14T15:14:12.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.416+0000 7fd0326db700 20 client.48340.objecter get_session s=0x7fd01800c910 osd=5 3 2024-04-14T15:14:12.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.416+0000 7fd0326db700 15 client.48340.objecter _session_command_op_assign 5 11 2024-04-14T15:14:12.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.417+0000 7fd0326db700 20 client.48340.objecter put_session s=0x7fd01800c910 osd=5 4 2024-04-14T15:14:12.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.417+0000 7fd0326db700 10 client.48340.objecter _send_command 11 2024-04-14T15:14:12.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.417+0000 7fd0326db700 1 -- 172.21.15.90:0/2439123785 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fd018018360 con 0x7fd018017eb0 2024-04-14T15:14:12.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.417+0000 7fd0297fa700 10 client.48340.objecter ms_handle_connect 0x7fd018017eb0 2024-04-14T15:14:12.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.418+0000 7fd0297fa700 1 -- 172.21.15.90:0/2439123785 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd018018360 con 0x7fd018017eb0 2024-04-14T15:14:12.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.418+0000 7fd0297fa700 10 client.48340.objecter ms_dispatch 0x7fd02c001550 command_reply(tid 11: 0 ) 2024-04-14T15:14:12.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.418+0000 7fd0297fa700 10 client.48340.objecter _finish_command 11 = system:0 2024-04-14T15:14:12.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.418+0000 7fd0297fa700 20 client.48340.objecter put_session s=0x7fd01800c910 osd=5 4 2024-04-14T15:14:12.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.418+0000 7fd0297fa700 15 client.48340.objecter _session_command_op_remove 5 11 2024-04-14T15:14:12.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.438+0000 7fd0326db700 10 client.48340.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:14:12.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.438+0000 7fd0326db700 15 client.48340.objecter _session_command_op_assign -1 12 2024-04-14T15:14:12.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.438+0000 7fd0326db700 20 client.48340.objecter _get_session s=0x7fd01800c910 osd=5 3 2024-04-14T15:14:12.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.438+0000 7fd0326db700 20 client.48340.objecter put_session s=0x7fd01800c910 osd=5 3 2024-04-14T15:14:12.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.438+0000 7fd0326db700 20 client.48340.objecter _get_session s=0x7fd01800c910 osd=5 3 2024-04-14T15:14:12.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.438+0000 7fd0326db700 15 client.48340.objecter _session_command_op_remove -1 12 2024-04-14T15:14:12.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.438+0000 7fd0326db700 20 client.48340.objecter get_session s=0x7fd01800c910 osd=5 3 2024-04-14T15:14:12.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.438+0000 7fd0326db700 15 client.48340.objecter _session_command_op_assign 5 12 2024-04-14T15:14:12.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.438+0000 7fd0326db700 20 client.48340.objecter put_session s=0x7fd01800c910 osd=5 4 2024-04-14T15:14:12.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.438+0000 7fd0326db700 10 client.48340.objecter _send_command 12 2024-04-14T15:14:12.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.438+0000 7fd0326db700 1 -- 172.21.15.90:0/2439123785 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd018018570 con 0x7fd018017eb0 2024-04-14T15:14:12.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.440+0000 7fd0297fa700 1 -- 172.21.15.90:0/2439123785 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd018018570 con 0x7fd018017eb0 2024-04-14T15:14:12.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.440+0000 7fd0297fa700 10 client.48340.objecter ms_dispatch 0x7fd02c001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:14:12.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.440+0000 7fd0297fa700 10 client.48340.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:14:12.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.440+0000 7fd0297fa700 20 client.48340.objecter put_session s=0x7fd01800c910 osd=5 4 2024-04-14T15:14:12.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.440+0000 7fd0297fa700 15 client.48340.objecter _session_command_op_remove 5 12 2024-04-14T15:14:12.441 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:14:12.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.440+0000 7fd0326db700 10 client.48340.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:14:12.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.440+0000 7fd0326db700 15 client.48340.objecter _session_command_op_assign -1 13 2024-04-14T15:14:12.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.440+0000 7fd0326db700 20 client.48340.objecter _get_session s=0x7fd018018360 osd=6 3 2024-04-14T15:14:12.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.440+0000 7fd0326db700 20 client.48340.objecter put_session s=0x7fd018018360 osd=6 3 2024-04-14T15:14:12.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.440+0000 7fd0326db700 20 client.48340.objecter _get_session s=0x7fd018018360 osd=6 3 2024-04-14T15:14:12.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.440+0000 7fd0326db700 15 client.48340.objecter _session_command_op_remove -1 13 2024-04-14T15:14:12.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.440+0000 7fd0326db700 20 client.48340.objecter get_session s=0x7fd018018360 osd=6 3 2024-04-14T15:14:12.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.440+0000 7fd0326db700 15 client.48340.objecter _session_command_op_assign 6 13 2024-04-14T15:14:12.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.440+0000 7fd0326db700 20 client.48340.objecter put_session s=0x7fd018018360 osd=6 4 2024-04-14T15:14:12.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.440+0000 7fd0326db700 10 client.48340.objecter _send_command 13 2024-04-14T15:14:12.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.440+0000 7fd0326db700 1 -- 172.21.15.90:0/2439123785 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fd018023150 con 0x7fd0180184d0 2024-04-14T15:14:12.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.440+0000 7fd0297fa700 10 client.48340.objecter ms_handle_connect 0x7fd0180184d0 2024-04-14T15:14:12.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.441+0000 7fd0297fa700 1 -- 172.21.15.90:0/2439123785 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd018023150 con 0x7fd0180184d0 2024-04-14T15:14:12.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.441+0000 7fd0297fa700 10 client.48340.objecter ms_dispatch 0x7fd02c001550 command_reply(tid 13: 0 ) 2024-04-14T15:14:12.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.441+0000 7fd0297fa700 10 client.48340.objecter _finish_command 13 = system:0 2024-04-14T15:14:12.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.441+0000 7fd0297fa700 20 client.48340.objecter put_session s=0x7fd018018360 osd=6 4 2024-04-14T15:14:12.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.441+0000 7fd0297fa700 15 client.48340.objecter _session_command_op_remove 6 13 2024-04-14T15:14:12.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.459+0000 7fd0326db700 10 client.48340.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:14:12.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.459+0000 7fd0326db700 15 client.48340.objecter _session_command_op_assign -1 14 2024-04-14T15:14:12.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.459+0000 7fd0326db700 20 client.48340.objecter _get_session s=0x7fd018018360 osd=6 3 2024-04-14T15:14:12.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.459+0000 7fd0326db700 20 client.48340.objecter put_session s=0x7fd018018360 osd=6 3 2024-04-14T15:14:12.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.459+0000 7fd0326db700 20 client.48340.objecter _get_session s=0x7fd018018360 osd=6 3 2024-04-14T15:14:12.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.459+0000 7fd0326db700 15 client.48340.objecter _session_command_op_remove -1 14 2024-04-14T15:14:12.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.459+0000 7fd0326db700 20 client.48340.objecter get_session s=0x7fd018018360 osd=6 3 2024-04-14T15:14:12.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.459+0000 7fd0326db700 15 client.48340.objecter _session_command_op_assign 6 14 2024-04-14T15:14:12.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.459+0000 7fd0326db700 20 client.48340.objecter put_session s=0x7fd018018360 osd=6 4 2024-04-14T15:14:12.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.459+0000 7fd0326db700 10 client.48340.objecter _send_command 14 2024-04-14T15:14:12.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.459+0000 7fd0326db700 1 -- 172.21.15.90:0/2439123785 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd018023330 con 0x7fd0180184d0 2024-04-14T15:14:12.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.461+0000 7fd0297fa700 1 -- 172.21.15.90:0/2439123785 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd018023330 con 0x7fd0180184d0 2024-04-14T15:14:12.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.461+0000 7fd0297fa700 10 client.48340.objecter ms_dispatch 0x7fd02c001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:14:12.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.461+0000 7fd0297fa700 10 client.48340.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:14:12.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.461+0000 7fd0297fa700 20 client.48340.objecter put_session s=0x7fd018018360 osd=6 4 2024-04-14T15:14:12.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.461+0000 7fd0297fa700 15 client.48340.objecter _session_command_op_remove 6 14 2024-04-14T15:14:12.461 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:14:12.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.461+0000 7fd0326db700 10 client.48340.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:14:12.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.461+0000 7fd0326db700 15 client.48340.objecter _session_command_op_assign -1 15 2024-04-14T15:14:12.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.461+0000 7fd0326db700 20 client.48340.objecter _get_session s=0x7fd018023330 osd=7 3 2024-04-14T15:14:12.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.461+0000 7fd0326db700 20 client.48340.objecter put_session s=0x7fd018023330 osd=7 3 2024-04-14T15:14:12.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.461+0000 7fd0326db700 20 client.48340.objecter _get_session s=0x7fd018023330 osd=7 3 2024-04-14T15:14:12.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.461+0000 7fd0326db700 15 client.48340.objecter _session_command_op_remove -1 15 2024-04-14T15:14:12.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.461+0000 7fd0326db700 20 client.48340.objecter get_session s=0x7fd018023330 osd=7 3 2024-04-14T15:14:12.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.461+0000 7fd0326db700 15 client.48340.objecter _session_command_op_assign 7 15 2024-04-14T15:14:12.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.461+0000 7fd0326db700 20 client.48340.objecter put_session s=0x7fd018023330 osd=7 4 2024-04-14T15:14:12.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.461+0000 7fd0326db700 10 client.48340.objecter _send_command 15 2024-04-14T15:14:12.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.461+0000 7fd0326db700 1 -- 172.21.15.90:0/2439123785 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fd018024650 con 0x7fd018023a50 2024-04-14T15:14:12.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.461+0000 7fd0297fa700 10 client.48340.objecter ms_handle_connect 0x7fd018023a50 2024-04-14T15:14:12.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.462+0000 7fd0297fa700 1 -- 172.21.15.90:0/2439123785 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd018024650 con 0x7fd018023a50 2024-04-14T15:14:12.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.462+0000 7fd0297fa700 10 client.48340.objecter ms_dispatch 0x7fd02c001550 command_reply(tid 15: 0 ) 2024-04-14T15:14:12.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.462+0000 7fd0297fa700 10 client.48340.objecter _finish_command 15 = system:0 2024-04-14T15:14:12.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.462+0000 7fd0297fa700 20 client.48340.objecter put_session s=0x7fd018023330 osd=7 4 2024-04-14T15:14:12.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.462+0000 7fd0297fa700 15 client.48340.objecter _session_command_op_remove 7 15 2024-04-14T15:14:12.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.479+0000 7fd0326db700 10 client.48340.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:14:12.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.479+0000 7fd0326db700 15 client.48340.objecter _session_command_op_assign -1 16 2024-04-14T15:14:12.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.479+0000 7fd0326db700 20 client.48340.objecter _get_session s=0x7fd018023330 osd=7 3 2024-04-14T15:14:12.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.479+0000 7fd0326db700 20 client.48340.objecter put_session s=0x7fd018023330 osd=7 3 2024-04-14T15:14:12.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.479+0000 7fd0326db700 20 client.48340.objecter _get_session s=0x7fd018023330 osd=7 3 2024-04-14T15:14:12.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.479+0000 7fd0326db700 15 client.48340.objecter _session_command_op_remove -1 16 2024-04-14T15:14:12.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.479+0000 7fd0326db700 20 client.48340.objecter get_session s=0x7fd018023330 osd=7 3 2024-04-14T15:14:12.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.479+0000 7fd0326db700 15 client.48340.objecter _session_command_op_assign 7 16 2024-04-14T15:14:12.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.479+0000 7fd0326db700 20 client.48340.objecter put_session s=0x7fd018023330 osd=7 4 2024-04-14T15:14:12.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.479+0000 7fd0326db700 10 client.48340.objecter _send_command 16 2024-04-14T15:14:12.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.479+0000 7fd0326db700 1 -- 172.21.15.90:0/2439123785 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd018024650 con 0x7fd018023a50 2024-04-14T15:14:12.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.480+0000 7fd0297fa700 1 -- 172.21.15.90:0/2439123785 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd018024650 con 0x7fd018023a50 2024-04-14T15:14:12.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.480+0000 7fd0297fa700 10 client.48340.objecter ms_dispatch 0x7fd02c001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:14:12.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.480+0000 7fd0297fa700 10 client.48340.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:14:12.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.480+0000 7fd0297fa700 20 client.48340.objecter put_session s=0x7fd018023330 osd=7 4 2024-04-14T15:14:12.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.480+0000 7fd0297fa700 15 client.48340.objecter _session_command_op_remove 7 16 2024-04-14T15:14:12.481 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:14:12.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.481+0000 7fd0326db700 10 librados: watch_flush enter 2024-04-14T15:14:12.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.481+0000 7fd0326db700 10 librados: watch_flush exit 2024-04-14T15:14:12.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.481+0000 7fd0326db700 10 client.48340.objecter close_session for osd.0 2024-04-14T15:14:12.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.481+0000 7fd0326db700 1 -- 172.21.15.90:0/2439123785 >> v1:172.21.15.90:6801/3761806885 conn(0x7fd018004730 legacy=0x7fd018002790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:12.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.481+0000 7fd0326db700 20 client.48340.objecter put_session s=0x7fd0180053e0 osd=0 1 2024-04-14T15:14:12.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.481+0000 7fd0326db700 10 client.48340.objecter close_session for osd.1 2024-04-14T15:14:12.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.481+0000 7fd0326db700 1 -- 172.21.15.90:0/2439123785 >> v1:172.21.15.90:6805/4170361699 conn(0x7fd0180042f0 legacy=0x7fd01800ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:12.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.481+0000 7fd0326db700 20 client.48340.objecter put_session s=0x7fd018004180 osd=1 1 2024-04-14T15:14:12.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.481+0000 7fd0326db700 10 client.48340.objecter close_session for osd.2 2024-04-14T15:14:12.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.481+0000 7fd0326db700 1 -- 172.21.15.90:0/2439123785 >> v1:172.21.15.90:6809/1686184251 conn(0x7fd01800d6e0 legacy=0x7fd01800fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:12.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.481+0000 7fd0326db700 20 client.48340.objecter put_session s=0x7fd018008480 osd=2 1 2024-04-14T15:14:12.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.481+0000 7fd0326db700 10 client.48340.objecter close_session for osd.3 2024-04-14T15:14:12.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.481+0000 7fd0326db700 1 -- 172.21.15.90:0/2439123785 >> v1:172.21.15.90:6813/2270166774 conn(0x7fd018012520 legacy=0x7fd018014910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:12.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.481+0000 7fd0326db700 20 client.48340.objecter put_session s=0x7fd01800cd30 osd=3 1 2024-04-14T15:14:12.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.481+0000 7fd0326db700 10 client.48340.objecter close_session for osd.4 2024-04-14T15:14:12.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.481+0000 7fd0326db700 1 -- 172.21.15.90:0/2439123785 >> v1:172.21.15.92:6800/3887549733 conn(0x7fd018012080 legacy=0x7fd018016030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:12.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.481+0000 7fd0326db700 20 client.48340.objecter put_session s=0x7fd01800c680 osd=4 1 2024-04-14T15:14:12.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.481+0000 7fd0326db700 10 client.48340.objecter close_session for osd.5 2024-04-14T15:14:12.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.481+0000 7fd0326db700 1 -- 172.21.15.90:0/2439123785 >> v1:172.21.15.92:6804/3244561440 conn(0x7fd018017eb0 legacy=0x7fd01801a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:12.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.481+0000 7fd0326db700 20 client.48340.objecter put_session s=0x7fd01800c910 osd=5 1 2024-04-14T15:14:12.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.481+0000 7fd0326db700 10 client.48340.objecter close_session for osd.6 2024-04-14T15:14:12.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.481+0000 7fd0326db700 1 -- 172.21.15.90:0/2439123785 >> v1:172.21.15.92:6808/636125963 conn(0x7fd0180184d0 legacy=0x7fd0180218b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:12.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.481+0000 7fd0326db700 20 client.48340.objecter put_session s=0x7fd018018360 osd=6 1 2024-04-14T15:14:12.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.481+0000 7fd0326db700 10 client.48340.objecter close_session for osd.7 2024-04-14T15:14:12.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.481+0000 7fd0326db700 1 -- 172.21.15.90:0/2439123785 >> v1:172.21.15.92:6812/1773706716 conn(0x7fd018023a50 legacy=0x7fd018023ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:12.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.481+0000 7fd0326db700 20 client.48340.objecter put_session s=0x7fd018023330 osd=7 1 2024-04-14T15:14:12.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.481+0000 7fd0326db700 20 client.48340.objecter shutdown clearing up homeless session... 2024-04-14T15:14:12.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.481+0000 7fd0326db700 10 client.48340.objecter successfully canceled tick 2024-04-14T15:14:12.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.481+0000 7fd030c78700 1 -- 172.21.15.90:0/2439123785 reap_dead start 2024-04-14T15:14:12.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.481+0000 7fd030c78700 1 -- 172.21.15.90:0/2439123785 reap_dead start 2024-04-14T15:14:12.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.481+0000 7fd030c78700 1 -- 172.21.15.90:0/2439123785 reap_dead start 2024-04-14T15:14:12.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.481+0000 7fd030c78700 1 -- 172.21.15.90:0/2439123785 reap_dead start 2024-04-14T15:14:12.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.481+0000 7fd0326db700 1 -- 172.21.15.90:0/2439123785 >> v1:172.21.15.90:6800/420935427 conn(0x7fd014075760 legacy=0x7fd014077c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:12.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.481+0000 7fd030c78700 1 -- 172.21.15.90:0/2439123785 reap_dead start 2024-04-14T15:14:12.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.481+0000 7fd0326db700 1 -- 172.21.15.90:0/2439123785 >> v1:172.21.15.92:6789/0 conn(0x7fd02c10a610 legacy=0x7fd02c1a26a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:12.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.481+0000 7fd0326db700 1 -- 172.21.15.90:0/2439123785 shutdown_connections 2024-04-14T15:14:12.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.481+0000 7fd0326db700 1 -- 172.21.15.90:0/2439123785 >> 172.21.15.90:0/2439123785 conn(0x7fd02c0fd290 msgr2=0x7fd02c1006f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:14:12.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.481+0000 7fd0326db700 1 -- 172.21.15.90:0/2439123785 shutdown_connections 2024-04-14T15:14:12.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.481+0000 7fd0326db700 1 -- 172.21.15.90:0/2439123785 wait complete. 2024-04-14T15:14:12.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:12.481+0000 7fd0326db700 1 librados: shutdown 2024-04-14T15:14:12.486 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:14:12.486 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:14:12.486 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:14:12.486 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:14:12.486 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:14:12.486 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:14:12.486 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:14:12.487 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:14:12.495 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:14:12.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:12.520 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:12.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:14:12.622 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:14:12.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:12.649 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:12.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:14:12.750 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:14:12.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:12.775 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:12.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:14:12.876 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:14:12.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:12.904 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:12.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:14:13.005 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:14:13.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:13.034 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:13.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:14:13.135 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:14:13.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:13.159 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:13.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:14:13.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:12 smithi090 ceph-mon[41670]: pgmap v221: 359 pgs: 96 unknown, 8 creating+peering, 255 active+clean; 592 KiB data, 548 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:14:13.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:12 smithi090 ceph-mon[41670]: 2.10 scrub starts 2024-04-14T15:14:13.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:12 smithi090 ceph-mon[41670]: 2.10 scrub ok 2024-04-14T15:14:13.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:12 smithi090 ceph-mon[41670]: 4.9 deep-scrub starts 2024-04-14T15:14:13.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:12 smithi090 ceph-mon[41670]: 4.9 deep-scrub ok 2024-04-14T15:14:13.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:12 smithi090 ceph-mon[41670]: 4.15 deep-scrub starts 2024-04-14T15:14:13.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:12 smithi090 ceph-mon[41670]: 4.15 deep-scrub ok 2024-04-14T15:14:13.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:12 smithi090 ceph-mon[41670]: 5.14 scrub starts 2024-04-14T15:14:13.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:12 smithi090 ceph-mon[41670]: 5.14 scrub ok 2024-04-14T15:14:13.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:12 smithi090 ceph-mon[41670]: 11.2 scrub starts 2024-04-14T15:14:13.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:12 smithi090 ceph-mon[41670]: 11.2 scrub ok 2024-04-14T15:14:13.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:12 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2439123785' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:14:13.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:12 smithi090 ceph-mon[41670]: 4.3 scrub starts 2024-04-14T15:14:13.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:12 smithi090 ceph-mon[41670]: 4.3 scrub ok 2024-04-14T15:14:13.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:12 smithi090 ceph-mon[41670]: 5.7 scrub starts 2024-04-14T15:14:13.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:12 smithi090 ceph-mon[41670]: 5.7 scrub ok 2024-04-14T15:14:13.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:12 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1863705178' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi090-93724-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi090-93724-21"}]': finished 2024-04-14T15:14:13.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:12 smithi090 ceph-mon[41670]: from='client.48328 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi090-93342-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:14:13.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:12 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1240983831' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi090-93342-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi090-93342-27"}]: dispatch 2024-04-14T15:14:13.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:12 smithi090 ceph-mon[41670]: osdmap e185: 8 total, 8 up, 8 in 2024-04-14T15:14:13.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:12 smithi090 ceph-mon[41670]: from='client.48328 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi090-93342-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi090-93342-27"}]: dispatch 2024-04-14T15:14:13.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:12 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2722044008' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi090-93363-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:13.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:12 smithi090 ceph-mon[41670]: from='client.48334 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi090-93363-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:13.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:12 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:13.194 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:12 smithi092 ceph-mon[33966]: pgmap v221: 359 pgs: 96 unknown, 8 creating+peering, 255 active+clean; 592 KiB data, 548 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:14:13.194 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:12 smithi092 ceph-mon[33966]: 2.10 scrub starts 2024-04-14T15:14:13.194 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:12 smithi092 ceph-mon[33966]: 2.10 scrub ok 2024-04-14T15:14:13.194 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:12 smithi092 ceph-mon[33966]: 4.9 deep-scrub starts 2024-04-14T15:14:13.194 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:12 smithi092 ceph-mon[33966]: 4.9 deep-scrub ok 2024-04-14T15:14:13.194 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:12 smithi092 ceph-mon[33966]: 4.15 deep-scrub starts 2024-04-14T15:14:13.194 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:12 smithi092 ceph-mon[33966]: 4.15 deep-scrub ok 2024-04-14T15:14:13.194 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:12 smithi092 ceph-mon[33966]: 5.14 scrub starts 2024-04-14T15:14:13.194 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:12 smithi092 ceph-mon[33966]: 5.14 scrub ok 2024-04-14T15:14:13.194 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:12 smithi092 ceph-mon[33966]: 11.2 scrub starts 2024-04-14T15:14:13.194 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:12 smithi092 ceph-mon[33966]: 11.2 scrub ok 2024-04-14T15:14:13.194 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:12 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2439123785' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:14:13.194 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:12 smithi092 ceph-mon[33966]: 4.3 scrub starts 2024-04-14T15:14:13.195 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:12 smithi092 ceph-mon[33966]: 4.3 scrub ok 2024-04-14T15:14:13.195 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:12 smithi092 ceph-mon[33966]: 5.7 scrub starts 2024-04-14T15:14:13.195 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:12 smithi092 ceph-mon[33966]: 5.7 scrub ok 2024-04-14T15:14:13.195 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:12 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1863705178' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi090-93724-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi090-93724-21"}]': finished 2024-04-14T15:14:13.195 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:12 smithi092 ceph-mon[33966]: from='client.48328 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi090-93342-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:14:13.195 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:12 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1240983831' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi090-93342-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi090-93342-27"}]: dispatch 2024-04-14T15:14:13.195 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:12 smithi092 ceph-mon[33966]: osdmap e185: 8 total, 8 up, 8 in 2024-04-14T15:14:13.195 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:12 smithi092 ceph-mon[33966]: from='client.48328 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi090-93342-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi090-93342-27"}]: dispatch 2024-04-14T15:14:13.195 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:12 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2722044008' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi090-93363-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:13.195 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:12 smithi092 ceph-mon[33966]: from='client.48334 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi090-93363-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:13.195 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:12 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:13.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:12 smithi090 ceph-mon[30399]: pgmap v221: 359 pgs: 96 unknown, 8 creating+peering, 255 active+clean; 592 KiB data, 548 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:14:13.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:12 smithi090 ceph-mon[30399]: 2.10 scrub starts 2024-04-14T15:14:13.196 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:12 smithi090 ceph-mon[30399]: 2.10 scrub ok 2024-04-14T15:14:13.196 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:12 smithi090 ceph-mon[30399]: 4.9 deep-scrub starts 2024-04-14T15:14:13.196 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:12 smithi090 ceph-mon[30399]: 4.9 deep-scrub ok 2024-04-14T15:14:13.196 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:12 smithi090 ceph-mon[30399]: 4.15 deep-scrub starts 2024-04-14T15:14:13.196 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:12 smithi090 ceph-mon[30399]: 4.15 deep-scrub ok 2024-04-14T15:14:13.196 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:12 smithi090 ceph-mon[30399]: 5.14 scrub starts 2024-04-14T15:14:13.196 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:12 smithi090 ceph-mon[30399]: 5.14 scrub ok 2024-04-14T15:14:13.196 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:12 smithi090 ceph-mon[30399]: 11.2 scrub starts 2024-04-14T15:14:13.196 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:12 smithi090 ceph-mon[30399]: 11.2 scrub ok 2024-04-14T15:14:13.196 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:12 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2439123785' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:14:13.196 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:12 smithi090 ceph-mon[30399]: 4.3 scrub starts 2024-04-14T15:14:13.196 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:12 smithi090 ceph-mon[30399]: 4.3 scrub ok 2024-04-14T15:14:13.196 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:12 smithi090 ceph-mon[30399]: 5.7 scrub starts 2024-04-14T15:14:13.196 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:12 smithi090 ceph-mon[30399]: 5.7 scrub ok 2024-04-14T15:14:13.196 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:12 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1863705178' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi090-93724-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi090-93724-21"}]': finished 2024-04-14T15:14:13.197 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:12 smithi090 ceph-mon[30399]: from='client.48328 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi090-93342-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:14:13.197 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:12 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1240983831' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi090-93342-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi090-93342-27"}]: dispatch 2024-04-14T15:14:13.197 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:12 smithi090 ceph-mon[30399]: osdmap e185: 8 total, 8 up, 8 in 2024-04-14T15:14:13.197 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:12 smithi090 ceph-mon[30399]: from='client.48328 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi090-93342-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi090-93342-27"}]: dispatch 2024-04-14T15:14:13.197 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:12 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2722044008' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi090-93363-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:13.197 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:12 smithi090 ceph-mon[30399]: from='client.48334 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi090-93363-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:13.197 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:12 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:13.260 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:14:13.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:13.286 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:13.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:14:13.387 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:14:13.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:13.412 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:13.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:14:13.513 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:14:13.537 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:13.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:13.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:14:13.640 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:14:13.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:13.666 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:13.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:14:13.767 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:14:13.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:13.791 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:13.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:14:13.892 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:14:13.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:13.919 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:13.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:14:14.020 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:14:14.032 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-04-14T15:14:14.032 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:14:14.032 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:13 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-04-14T15:14:14.032 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:13 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-04-14T15:14:14.032 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:13 smithi092 ceph-mon[33966]: 1.0 scrub starts 2024-04-14T15:14:14.032 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:13 smithi092 ceph-mon[33966]: 1.0 scrub ok 2024-04-14T15:14:14.033 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:13 smithi092 ceph-mon[33966]: 6.9 scrub starts 2024-04-14T15:14:14.033 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:13 smithi092 ceph-mon[33966]: 6.9 scrub ok 2024-04-14T15:14:14.033 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:13 smithi092 ceph-mon[33966]: 5.16 deep-scrub starts 2024-04-14T15:14:14.033 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:13 smithi092 ceph-mon[33966]: 5.16 deep-scrub ok 2024-04-14T15:14:14.033 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:13 smithi092 ceph-mon[33966]: from='client.48334 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi090-93363-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:14.033 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:13 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "25"}]': finished 2024-04-14T15:14:14.033 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:13 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-04-14T15:14:14.033 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:13 smithi092 ceph-mon[33966]: osdmap e186: 8 total, 8 up, 8 in 2024-04-14T15:14:14.033 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:13 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/844371530' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi090-93601-24"}]: dispatch 2024-04-14T15:14:14.033 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:13 smithi092 ceph-mon[33966]: from='client.46400 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi090-93601-24"}]: dispatch 2024-04-14T15:14:14.033 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:13 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:14.043 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:14.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:14.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:14:14.145 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:14:14.169 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:14.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:14.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:14:14.191 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:14:13 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:14:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:14:14.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:13 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-04-14T15:14:14.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:13 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-04-14T15:14:14.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:13 smithi090 ceph-mon[41670]: 1.0 scrub starts 2024-04-14T15:14:14.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:13 smithi090 ceph-mon[41670]: 1.0 scrub ok 2024-04-14T15:14:14.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:13 smithi090 ceph-mon[41670]: 6.9 scrub starts 2024-04-14T15:14:14.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:13 smithi090 ceph-mon[41670]: 6.9 scrub ok 2024-04-14T15:14:14.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:13 smithi090 ceph-mon[41670]: 5.16 deep-scrub starts 2024-04-14T15:14:14.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:13 smithi090 ceph-mon[41670]: 5.16 deep-scrub ok 2024-04-14T15:14:14.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:13 smithi090 ceph-mon[41670]: from='client.48334 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi090-93363-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:14.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:13 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "25"}]': finished 2024-04-14T15:14:14.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:13 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-04-14T15:14:14.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:13 smithi090 ceph-mon[41670]: osdmap e186: 8 total, 8 up, 8 in 2024-04-14T15:14:14.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:13 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/844371530' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi090-93601-24"}]: dispatch 2024-04-14T15:14:14.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:13 smithi090 ceph-mon[41670]: from='client.46400 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi090-93601-24"}]: dispatch 2024-04-14T15:14:14.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:13 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:14.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:13 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-04-14T15:14:14.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:13 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-04-14T15:14:14.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:13 smithi090 ceph-mon[30399]: 1.0 scrub starts 2024-04-14T15:14:14.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:13 smithi090 ceph-mon[30399]: 1.0 scrub ok 2024-04-14T15:14:14.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:13 smithi090 ceph-mon[30399]: 6.9 scrub starts 2024-04-14T15:14:14.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:13 smithi090 ceph-mon[30399]: 6.9 scrub ok 2024-04-14T15:14:14.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:13 smithi090 ceph-mon[30399]: 5.16 deep-scrub starts 2024-04-14T15:14:14.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:13 smithi090 ceph-mon[30399]: 5.16 deep-scrub ok 2024-04-14T15:14:14.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:13 smithi090 ceph-mon[30399]: from='client.48334 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi090-93363-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:14.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:13 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "25"}]': finished 2024-04-14T15:14:14.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:13 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-04-14T15:14:14.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:13 smithi090 ceph-mon[30399]: osdmap e186: 8 total, 8 up, 8 in 2024-04-14T15:14:14.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:13 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/844371530' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi090-93601-24"}]: dispatch 2024-04-14T15:14:14.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:13 smithi090 ceph-mon[30399]: from='client.46400 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi090-93601-24"}]: dispatch 2024-04-14T15:14:14.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:13 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:14.270 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:14:14.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:14.296 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:14.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:14:14.397 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:14:14.421 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:14.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:14.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:14:14.523 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:14:14.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:14.550 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:14.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:14:14.651 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:14:14.675 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:14.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:14.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:14:14.777 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:14:14.791 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-14T15:14:14.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:14.800+0000 7fbeafad5700 1 -- 172.21.15.90:0/2158413817 >> v1:172.21.15.90:6789/0 conn(0x7fbea8109da0 legacy=0x7fbea810c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:14.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:14.800+0000 7fbeafad5700 1 -- 172.21.15.90:0/2158413817 shutdown_connections 2024-04-14T15:14:14.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:14.800+0000 7fbeafad5700 1 -- 172.21.15.90:0/2158413817 >> 172.21.15.90:0/2158413817 conn(0x7fbea8076aa0 msgr2=0x7fbea8076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:14:14.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:14.800+0000 7fbeafad5700 1 -- 172.21.15.90:0/2158413817 shutdown_connections 2024-04-14T15:14:14.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:14.800+0000 7fbeafad5700 1 -- 172.21.15.90:0/2158413817 wait complete. 2024-04-14T15:14:14.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:14.801+0000 7fbeafad5700 1 librados: starting msgr at 2024-04-14T15:14:14.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:14.801+0000 7fbeafad5700 1 librados: starting objecter 2024-04-14T15:14:14.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:14.801+0000 7fbeafad5700 1 Processor -- start 2024-04-14T15:14:14.802 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:14.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:14.801+0000 7fbeafad5700 1 -- start start 2024-04-14T15:14:14.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:14.801+0000 7fbeafad5700 1 librados: setting wanted keys 2024-04-14T15:14:14.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:14.801+0000 7fbeafad5700 1 librados: calling monclient init 2024-04-14T15:14:14.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:14.801+0000 7fbeafad5700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbea81a9cd0 con 0x7fbea8106f80 2024-04-14T15:14:14.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:14.801+0000 7fbeafad5700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbea81aaef0 con 0x7fbea810d940 2024-04-14T15:14:14.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:14.801+0000 7fbeafad5700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbea81ac130 con 0x7fbea8109da0 2024-04-14T15:14:14.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:14.801+0000 7fbead871700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fbea8106f80 0x7fbea80fe560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:57252/0 (socket says 172.21.15.90:57252) 2024-04-14T15:14:14.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:14.801+0000 7fbead871700 1 -- 172.21.15.90:0/671814371 learned_addr learned my addr 172.21.15.90:0/671814371 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:14:14.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:14.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:14:14.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:14.801+0000 7fbe9effd700 10 client.?.objecter ms_handle_connect 0x7fbea8106f80 2024-04-14T15:14:14.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:14.801+0000 7fbe9effd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:14:14.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:14.801+0000 7fbe9effd700 10 client.?.objecter ms_handle_connect 0x7fbea810d940 2024-04-14T15:14:14.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:14.801+0000 7fbe9effd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:14:14.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:14.801+0000 7fbe9effd700 10 client.?.objecter ms_handle_connect 0x7fbea8109da0 2024-04-14T15:14:14.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:14.801+0000 7fbe9effd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:14:14.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:14.802+0000 7fbe9effd700 1 -- 172.21.15.90:0/671814371 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3570492652 0 0) 0x7fbea81aaef0 con 0x7fbea810d940 2024-04-14T15:14:14.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:14.802+0000 7fbe9effd700 1 -- 172.21.15.90:0/671814371 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbe94004370 con 0x7fbea810d940 2024-04-14T15:14:14.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:14.802+0000 7fbe9effd700 1 -- 172.21.15.90:0/671814371 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 404575628 0 0) 0x7fbea81a9cd0 con 0x7fbea8106f80 2024-04-14T15:14:14.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:14.802+0000 7fbe9effd700 1 -- 172.21.15.90:0/671814371 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbea81aaef0 con 0x7fbea8106f80 2024-04-14T15:14:14.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:14.802+0000 7fbe9effd700 1 -- 172.21.15.90:0/671814371 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1588978143 0 0) 0x7fbea81ac130 con 0x7fbea8109da0 2024-04-14T15:14:14.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:14.802+0000 7fbe9effd700 1 -- 172.21.15.90:0/671814371 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbea81a9cd0 con 0x7fbea8109da0 2024-04-14T15:14:14.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:14.802+0000 7fbe9effd700 1 -- 172.21.15.90:0/671814371 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 383952540 0 0) 0x7fbe94004370 con 0x7fbea810d940 2024-04-14T15:14:14.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:14.802+0000 7fbe9effd700 1 -- 172.21.15.90:0/671814371 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbea81ac130 con 0x7fbea810d940 2024-04-14T15:14:14.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:14.802+0000 7fbe9effd700 1 -- 172.21.15.90:0/671814371 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fbea0001c30 con 0x7fbea810d940 2024-04-14T15:14:14.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:14.802+0000 7fbe9effd700 1 -- 172.21.15.90:0/671814371 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3148751551 0 0) 0x7fbea81aaef0 con 0x7fbea8106f80 2024-04-14T15:14:14.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:14.802+0000 7fbe9effd700 1 -- 172.21.15.90:0/671814371 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbe94004370 con 0x7fbea8106f80 2024-04-14T15:14:14.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:14.802+0000 7fbe9effd700 1 -- 172.21.15.90:0/671814371 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2226190353 0 0) 0x7fbea81a9cd0 con 0x7fbea8109da0 2024-04-14T15:14:14.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:14.802+0000 7fbe9effd700 1 -- 172.21.15.90:0/671814371 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbea81aaef0 con 0x7fbea8109da0 2024-04-14T15:14:14.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:14.802+0000 7fbe9effd700 1 -- 172.21.15.90:0/671814371 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fbe98001430 con 0x7fbea8109da0 2024-04-14T15:14:14.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:14.802+0000 7fbe9effd700 1 -- 172.21.15.90:0/671814371 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fbea40018b0 con 0x7fbea8106f80 2024-04-14T15:14:14.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:14.802+0000 7fbe9effd700 1 -- 172.21.15.90:0/671814371 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2977007706 0 0) 0x7fbea81ac130 con 0x7fbea810d940 2024-04-14T15:14:14.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:14.802+0000 7fbe9effd700 1 -- 172.21.15.90:0/671814371 >> v1:172.21.15.90:6790/0 conn(0x7fbea8109da0 legacy=0x7fbea80fec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:14.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:14.802+0000 7fbe9effd700 1 -- 172.21.15.90:0/671814371 >> v1:172.21.15.90:6789/0 conn(0x7fbea8106f80 legacy=0x7fbea80fe560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:14.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:14.802+0000 7fbe9effd700 1 -- 172.21.15.90:0/671814371 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbea81ad370 con 0x7fbea810d940 2024-04-14T15:14:14.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:14.802+0000 7fbeafad5700 1 -- 172.21.15.90:0/671814371 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbea81ab120 con 0x7fbea810d940 2024-04-14T15:14:14.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:14.802+0000 7fbeafad5700 10 client.48346.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:14:14.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:14.802+0000 7fbeafad5700 1 -- 172.21.15.90:0/671814371 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fbea81ab5e0 con 0x7fbea810d940 2024-04-14T15:14:14.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:14.802+0000 7fbeafad5700 1 librados: init done 2024-04-14T15:14:14.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:14.802+0000 7fbe9effd700 1 -- 172.21.15.90:0/671814371 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fbea0003ec0 con 0x7fbea810d940 2024-04-14T15:14:14.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:14.802+0000 7fbe9effd700 1 -- 172.21.15.90:0/671814371 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fbea0004ed0 con 0x7fbea810d940 2024-04-14T15:14:14.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:14.803+0000 7fbeafad5700 1 -- 172.21.15.90:0/671814371 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe8c005120 con 0x7fbea810d940 2024-04-14T15:14:14.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:14.804+0000 7fbe9effd700 1 -- 172.21.15.90:0/671814371 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fbea0003a20 con 0x7fbea810d940 2024-04-14T15:14:14.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:14.804+0000 7fbe9effd700 1 -- 172.21.15.90:0/671814371 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(186..186 src has 51..186) ==== 10489+0+0 (unknown 1708831308 0 0) 0x7fbea008a500 con 0x7fbea810d940 2024-04-14T15:14:14.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:14.804+0000 7fbe9effd700 10 client.48346.objecter ms_dispatch 0x7fbea8001550 osd_map(186..186 src has 51..186) 2024-04-14T15:14:14.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:14.804+0000 7fbe9effd700 3 client.48346.objecter handle_osd_map got epochs [186,186] > 0 2024-04-14T15:14:14.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:14.804+0000 7fbe9effd700 3 client.48346.objecter handle_osd_map decoding full epoch 186 2024-04-14T15:14:14.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:14.804+0000 7fbe9effd700 20 client.48346.objecter dump_active .. 0 homeless 2024-04-14T15:14:14.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:14.804+0000 7fbe9effd700 10 client.48346.objecter ms_handle_connect 0x7fbe94071320 2024-04-14T15:14:14.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:14.809+0000 7fbe9effd700 1 -- 172.21.15.90:0/671814371 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fbea00565f0 con 0x7fbea810d940 2024-04-14T15:14:14.905 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:14:14.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:14.929 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:14.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:14:15.030 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:14:15.052 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:14:15.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:15.055+0000 7fbeafad5700 1 -- 172.21.15.90:0/671814371 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7fbe8c005410 con 0x7fbea810d940 2024-04-14T15:14:15.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:14 smithi090 ceph-mon[41670]: pgmap v224: 312 pgs: 50 unknown, 8 creating+peering, 254 active+clean; 592 KiB data, 548 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s; 31 B/s, 0 objects/s recovering 2024-04-14T15:14:15.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:14 smithi090 ceph-mon[41670]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:14:15.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:14 smithi090 ceph-mon[41670]: 5.2 scrub starts 2024-04-14T15:14:15.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:14 smithi090 ceph-mon[41670]: 5.2 scrub ok 2024-04-14T15:14:15.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:14 smithi090 ceph-mon[41670]: 6.b scrub starts 2024-04-14T15:14:15.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:14 smithi090 ceph-mon[41670]: 6.b scrub ok 2024-04-14T15:14:15.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:14 smithi090 ceph-mon[41670]: 2.4 scrub starts 2024-04-14T15:14:15.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:14 smithi090 ceph-mon[41670]: 2.4 scrub ok 2024-04-14T15:14:15.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:14 smithi090 ceph-mon[41670]: from='client.48328 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi090-93342-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi090-93342-27"}]': finished 2024-04-14T15:14:15.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:14 smithi090 ceph-mon[41670]: from='client.46400 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi090-93601-24"}]': finished 2024-04-14T15:14:15.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:14 smithi090 ceph-mon[41670]: osdmap e187: 8 total, 8 up, 8 in 2024-04-14T15:14:15.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:14 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/844371530' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi090-93601-24"}]: dispatch 2024-04-14T15:14:15.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:14 smithi090 ceph-mon[41670]: from='client.46400 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi090-93601-24"}]: dispatch 2024-04-14T15:14:15.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:14 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:15.057 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:14 smithi090 ceph-mon[30399]: pgmap v224: 312 pgs: 50 unknown, 8 creating+peering, 254 active+clean; 592 KiB data, 548 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s; 31 B/s, 0 objects/s recovering 2024-04-14T15:14:15.057 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:14 smithi090 ceph-mon[30399]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:14:15.057 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:14 smithi090 ceph-mon[30399]: 5.2 scrub starts 2024-04-14T15:14:15.057 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:14 smithi090 ceph-mon[30399]: 5.2 scrub ok 2024-04-14T15:14:15.057 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:14 smithi090 ceph-mon[30399]: 6.b scrub starts 2024-04-14T15:14:15.057 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:14 smithi090 ceph-mon[30399]: 6.b scrub ok 2024-04-14T15:14:15.057 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:14 smithi090 ceph-mon[30399]: 2.4 scrub starts 2024-04-14T15:14:15.057 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:14 smithi090 ceph-mon[30399]: 2.4 scrub ok 2024-04-14T15:14:15.057 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:14 smithi090 ceph-mon[30399]: from='client.48328 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi090-93342-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi090-93342-27"}]': finished 2024-04-14T15:14:15.057 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:14 smithi090 ceph-mon[30399]: from='client.46400 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi090-93601-24"}]': finished 2024-04-14T15:14:15.057 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:14 smithi090 ceph-mon[30399]: osdmap e187: 8 total, 8 up, 8 in 2024-04-14T15:14:15.057 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:14 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/844371530' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi090-93601-24"}]: dispatch 2024-04-14T15:14:15.057 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:14 smithi090 ceph-mon[30399]: from='client.46400 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi090-93601-24"}]: dispatch 2024-04-14T15:14:15.057 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:14 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:15.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:15.059 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:15.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:14:15.122 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:14:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:14:15.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:14 smithi092 ceph-mon[33966]: pgmap v224: 312 pgs: 50 unknown, 8 creating+peering, 254 active+clean; 592 KiB data, 548 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s; 31 B/s, 0 objects/s recovering 2024-04-14T15:14:15.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:14 smithi092 ceph-mon[33966]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:14:15.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:14 smithi092 ceph-mon[33966]: 5.2 scrub starts 2024-04-14T15:14:15.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:14 smithi092 ceph-mon[33966]: 5.2 scrub ok 2024-04-14T15:14:15.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:14 smithi092 ceph-mon[33966]: 6.b scrub starts 2024-04-14T15:14:15.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:14 smithi092 ceph-mon[33966]: 6.b scrub ok 2024-04-14T15:14:15.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:14 smithi092 ceph-mon[33966]: 2.4 scrub starts 2024-04-14T15:14:15.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:14 smithi092 ceph-mon[33966]: 2.4 scrub ok 2024-04-14T15:14:15.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:14 smithi092 ceph-mon[33966]: from='client.48328 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi090-93342-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi090-93342-27"}]': finished 2024-04-14T15:14:15.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:14 smithi092 ceph-mon[33966]: from='client.46400 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi090-93601-24"}]': finished 2024-04-14T15:14:15.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:14 smithi092 ceph-mon[33966]: osdmap e187: 8 total, 8 up, 8 in 2024-04-14T15:14:15.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:14 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/844371530' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi090-93601-24"}]: dispatch 2024-04-14T15:14:15.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:14 smithi092 ceph-mon[33966]: from='client.46400 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi090-93601-24"}]: dispatch 2024-04-14T15:14:15.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:14 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:15.161 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:14:15.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:15.185 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:15.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:14:15.287 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:14:15.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:15.312 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:15.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:14:15.413 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:14:15.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:15.439 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:15.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:14:15.521 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-14T15:14:15.521 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-14T15:14:15.521 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-04-14T15:14:15.522 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd dump -f json-pretty 2024-04-14T15:14:15.540 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:15.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:15.589 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:15.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:15.690 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:15.715 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:15.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:15.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:15.764 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:14:15.817 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:15.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:15.826+0000 7fbe9effd700 1 -- 172.21.15.90:0/671814371 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v188) ==== 87+0+0 (unknown 3133664441 0 0) 0x7fbea005a2a0 con 0x7fbea810d940 2024-04-14T15:14:15.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:15.830+0000 7fbeafad5700 10 librados: watch_flush enter 2024-04-14T15:14:15.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:15.830+0000 7fbeafad5700 10 librados: watch_flush exit 2024-04-14T15:14:15.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:15.830+0000 7fbeafad5700 20 client.48346.objecter shutdown clearing up homeless session... 2024-04-14T15:14:15.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:15.831+0000 7fbeafad5700 10 client.48346.objecter successfully canceled tick 2024-04-14T15:14:15.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:15.832+0000 7fbeafad5700 1 -- 172.21.15.90:0/671814371 >> v1:172.21.15.90:6800/420935427 conn(0x7fbe94071320 legacy=0x7fbe940737e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:15.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:15.833+0000 7fbeafad5700 1 -- 172.21.15.90:0/671814371 >> v1:172.21.15.92:6789/0 conn(0x7fbea810d940 legacy=0x7fbea81a8390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:15.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:15.833+0000 7fbeafad5700 1 -- 172.21.15.90:0/671814371 shutdown_connections 2024-04-14T15:14:15.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:15.833+0000 7fbeafad5700 1 -- 172.21.15.90:0/671814371 >> 172.21.15.90:0/671814371 conn(0x7fbea8076aa0 msgr2=0x7fbea8107c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:14:15.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:15.833+0000 7fbeafad5700 1 -- 172.21.15.90:0/671814371 shutdown_connections 2024-04-14T15:14:15.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:15.833+0000 7fbeafad5700 1 -- 172.21.15.90:0/671814371 wait complete. 2024-04-14T15:14:15.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:15.833+0000 7fbeafad5700 1 librados: shutdown 2024-04-14T15:14:15.836 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is set 2024-04-14T15:14:15.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:15.859 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:15.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:15.960 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:15.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:15.986 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:15.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:16.087 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:16.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:16.111 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:16.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:16.113 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:15 smithi092 ceph-mon[33966]: 5.a scrub starts 2024-04-14T15:14:16.113 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:15 smithi092 ceph-mon[33966]: 5.a scrub ok 2024-04-14T15:14:16.114 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:15 smithi092 ceph-mon[33966]: 6.6 scrub starts 2024-04-14T15:14:16.114 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:15 smithi092 ceph-mon[33966]: 6.6 scrub ok 2024-04-14T15:14:16.114 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:15 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:14:16.114 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:15 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/671814371' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:14:16.114 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:15 smithi092 ceph-mon[33966]: from='client.48346 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:14:16.114 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:15 smithi092 ceph-mon[33966]: 2.17 scrub starts 2024-04-14T15:14:16.114 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:15 smithi092 ceph-mon[33966]: 2.17 scrub ok 2024-04-14T15:14:16.114 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:15 smithi092 ceph-mon[33966]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:14:16.114 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:15 smithi092 ceph-mon[33966]: from='client.46400 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi090-93601-24"}]': finished 2024-04-14T15:14:16.114 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:15 smithi092 ceph-mon[33966]: from='client.48346 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-14T15:14:16.114 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:15 smithi092 ceph-mon[33966]: osdmap e188: 8 total, 8 up, 8 in 2024-04-14T15:14:16.114 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:15 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1390812480' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi090-93363-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:16.114 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:15 smithi092 ceph-mon[33966]: from='client.48352 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi090-93363-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:16.114 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:15 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:16.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:15 smithi090 ceph-mon[41670]: 5.a scrub starts 2024-04-14T15:14:16.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:15 smithi090 ceph-mon[41670]: 5.a scrub ok 2024-04-14T15:14:16.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:15 smithi090 ceph-mon[41670]: 6.6 scrub starts 2024-04-14T15:14:16.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:15 smithi090 ceph-mon[41670]: 6.6 scrub ok 2024-04-14T15:14:16.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:15 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:14:16.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:15 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/671814371' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:14:16.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:15 smithi090 ceph-mon[41670]: from='client.48346 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:14:16.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:15 smithi090 ceph-mon[41670]: 2.17 scrub starts 2024-04-14T15:14:16.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:15 smithi090 ceph-mon[41670]: 2.17 scrub ok 2024-04-14T15:14:16.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:15 smithi090 ceph-mon[41670]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:14:16.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:15 smithi090 ceph-mon[41670]: from='client.46400 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi090-93601-24"}]': finished 2024-04-14T15:14:16.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:15 smithi090 ceph-mon[41670]: from='client.48346 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-14T15:14:16.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:15 smithi090 ceph-mon[41670]: osdmap e188: 8 total, 8 up, 8 in 2024-04-14T15:14:16.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:15 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1390812480' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi090-93363-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:16.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:15 smithi090 ceph-mon[41670]: from='client.48352 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi090-93363-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:16.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:15 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:16.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:15 smithi090 ceph-mon[30399]: 5.a scrub starts 2024-04-14T15:14:16.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:15 smithi090 ceph-mon[30399]: 5.a scrub ok 2024-04-14T15:14:16.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:15 smithi090 ceph-mon[30399]: 6.6 scrub starts 2024-04-14T15:14:16.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:15 smithi090 ceph-mon[30399]: 6.6 scrub ok 2024-04-14T15:14:16.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:15 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:14:16.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:15 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/671814371' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:14:16.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:15 smithi090 ceph-mon[30399]: from='client.48346 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:14:16.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:15 smithi090 ceph-mon[30399]: 2.17 scrub starts 2024-04-14T15:14:16.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:15 smithi090 ceph-mon[30399]: 2.17 scrub ok 2024-04-14T15:14:16.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:15 smithi090 ceph-mon[30399]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:14:16.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:15 smithi090 ceph-mon[30399]: from='client.46400 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi090-93601-24"}]': finished 2024-04-14T15:14:16.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:15 smithi090 ceph-mon[30399]: from='client.48346 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-14T15:14:16.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:15 smithi090 ceph-mon[30399]: osdmap e188: 8 total, 8 up, 8 in 2024-04-14T15:14:16.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:15 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1390812480' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi090-93363-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:16.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:15 smithi090 ceph-mon[30399]: from='client.48352 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi090-93363-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:16.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:15 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:16.212 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:14:16.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:16.238 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:16.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:14:16.339 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:16.353 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-14T15:14:16.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:16.364 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:16.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:16.466 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:14:16.489 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:16.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:16.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:14:16.591 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:14:16.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:16.615 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:16.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:14:16.716 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:14:16.740 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:16.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:16.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:14:16.841 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:14:16.882 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:16.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:16.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:14:16.984 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:14:17.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:17.009 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:17.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:14:17.084 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:16 smithi092 ceph-mon[33966]: pgmap v227: 280 pgs: 2 peering, 18 unknown, 260 active+clean; 233 KiB data, 536 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:14:17.084 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:16 smithi092 ceph-mon[33966]: 6.e scrub starts 2024-04-14T15:14:17.084 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:16 smithi092 ceph-mon[33966]: 6.e scrub ok 2024-04-14T15:14:17.084 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:16 smithi092 ceph-mon[33966]: from='client.48352 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi090-93363-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:17.084 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:16 smithi092 ceph-mon[33966]: osdmap e189: 8 total, 8 up, 8 in 2024-04-14T15:14:17.085 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:16 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2303409867' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi090-93601-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:17.085 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:16 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1240983831' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi090-93342-27"}]: dispatch 2024-04-14T15:14:17.085 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:16 smithi092 ceph-mon[33966]: from='client.48358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi090-93601-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:17.085 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:16 smithi092 ceph-mon[33966]: from='client.48328 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi090-93342-27"}]: dispatch 2024-04-14T15:14:17.085 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:16 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:17.105 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-04-14T15:14:17.105 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:14:17.110 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:14:17.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:17.136 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:17.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:14:17.149 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:16 smithi090 ceph-mon[30399]: pgmap v227: 280 pgs: 2 peering, 18 unknown, 260 active+clean; 233 KiB data, 536 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:14:17.149 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:16 smithi090 ceph-mon[30399]: 6.e scrub starts 2024-04-14T15:14:17.149 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:16 smithi090 ceph-mon[30399]: 6.e scrub ok 2024-04-14T15:14:17.149 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:16 smithi090 ceph-mon[30399]: from='client.48352 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi090-93363-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:17.149 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:16 smithi090 ceph-mon[30399]: osdmap e189: 8 total, 8 up, 8 in 2024-04-14T15:14:17.149 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:16 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2303409867' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi090-93601-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:17.150 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:16 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1240983831' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi090-93342-27"}]: dispatch 2024-04-14T15:14:17.150 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:16 smithi090 ceph-mon[30399]: from='client.48358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi090-93601-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:17.150 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:16 smithi090 ceph-mon[30399]: from='client.48328 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi090-93342-27"}]: dispatch 2024-04-14T15:14:17.150 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:16 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:17.150 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:16 smithi090 ceph-mon[41670]: pgmap v227: 280 pgs: 2 peering, 18 unknown, 260 active+clean; 233 KiB data, 536 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:14:17.150 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:16 smithi090 ceph-mon[41670]: 6.e scrub starts 2024-04-14T15:14:17.150 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:16 smithi090 ceph-mon[41670]: 6.e scrub ok 2024-04-14T15:14:17.150 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:16 smithi090 ceph-mon[41670]: from='client.48352 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi090-93363-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:17.150 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:16 smithi090 ceph-mon[41670]: osdmap e189: 8 total, 8 up, 8 in 2024-04-14T15:14:17.150 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:16 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2303409867' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi090-93601-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:17.150 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:16 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1240983831' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi090-93342-27"}]: dispatch 2024-04-14T15:14:17.151 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:16 smithi090 ceph-mon[41670]: from='client.48358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi090-93601-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:17.151 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:16 smithi090 ceph-mon[41670]: from='client.48328 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi090-93342-27"}]: dispatch 2024-04-14T15:14:17.151 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:16 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:17.236 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:14:17.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:17.260 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:17.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:14:17.361 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:14:17.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:17.387 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:17.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:14:17.488 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:14:17.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:17.512 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:17.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:14:17.613 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:14:17.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:17.638 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:17.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:14:17.739 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:14:17.764 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:17.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:17.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:14:17.816 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:14:17.866 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:14:17.897 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:17.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:17.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:14:17.919 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-14T15:14:17.999 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:14:18.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:18.024 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:18.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:14:18.125 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:14:18.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:18.150 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:18.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:14:18.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:17 smithi090 ceph-mon[30399]: pgmap v230: 335 pgs: 1 peering, 74 unknown, 260 active+clean; 233 KiB data, 536 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:14:18.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:17 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-04-14T15:14:18.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:17 smithi090 ceph-mon[30399]: from='client.48358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi090-93601-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:18.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:17 smithi090 ceph-mon[30399]: from='client.48328 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi090-93342-27"}]': finished 2024-04-14T15:14:18.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:17 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "24"}]': finished 2024-04-14T15:14:18.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:17 smithi090 ceph-mon[30399]: osdmap e190: 8 total, 8 up, 8 in 2024-04-14T15:14:18.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:17 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1240983831' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi090-93342-27"}]: dispatch 2024-04-14T15:14:18.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:17 smithi090 ceph-mon[30399]: from='client.48328 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi090-93342-27"}]: dispatch 2024-04-14T15:14:18.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:17 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:18.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:17 smithi090 ceph-mon[41670]: pgmap v230: 335 pgs: 1 peering, 74 unknown, 260 active+clean; 233 KiB data, 536 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:14:18.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:17 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-04-14T15:14:18.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:17 smithi090 ceph-mon[41670]: from='client.48358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi090-93601-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:18.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:17 smithi090 ceph-mon[41670]: from='client.48328 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi090-93342-27"}]': finished 2024-04-14T15:14:18.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:17 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "24"}]': finished 2024-04-14T15:14:18.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:17 smithi090 ceph-mon[41670]: osdmap e190: 8 total, 8 up, 8 in 2024-04-14T15:14:18.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:17 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1240983831' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi090-93342-27"}]: dispatch 2024-04-14T15:14:18.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:17 smithi090 ceph-mon[41670]: from='client.48328 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi090-93342-27"}]: dispatch 2024-04-14T15:14:18.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:17 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:18.199 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:14:18.251 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:14:18.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:18.275 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:18.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:14:18.320 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:17 smithi092 ceph-mon[33966]: pgmap v230: 335 pgs: 1 peering, 74 unknown, 260 active+clean; 233 KiB data, 536 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:14:18.321 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:17 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-04-14T15:14:18.321 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:17 smithi092 ceph-mon[33966]: from='client.48358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi090-93601-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:18.321 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:17 smithi092 ceph-mon[33966]: from='client.48328 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi090-93342-27"}]': finished 2024-04-14T15:14:18.321 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:17 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "24"}]': finished 2024-04-14T15:14:18.321 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:17 smithi092 ceph-mon[33966]: osdmap e190: 8 total, 8 up, 8 in 2024-04-14T15:14:18.321 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:17 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1240983831' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi090-93342-27"}]: dispatch 2024-04-14T15:14:18.321 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:17 smithi092 ceph-mon[33966]: from='client.48328 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi090-93342-27"}]: dispatch 2024-04-14T15:14:18.321 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:17 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:18.377 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:18.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:18.402 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:18.403 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:18.503 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:14:18.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:18.528 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:18.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:14:18.629 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:14:18.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:18.653 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:18.653 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:14:18.754 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:14:18.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:18.779 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:18.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:14:18.848 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: Running main() from gmock_main.cc 2024-04-14T15:14:18.848 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [==========] Running 42 tests from 2 test suites. 2024-04-14T15:14:18.848 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [----------] Global test environment set-up. 2024-04-14T15:14:18.848 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [----------] 26 tests from LibRadosAio 2024-04-14T15:14:18.848 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2024-04-14T15:14:18.848 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ OK ] LibRadosAio.TooBig (2940 ms) 2024-04-14T15:14:18.848 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2024-04-14T15:14:18.849 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (3025 ms) 2024-04-14T15:14:18.849 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2024-04-14T15:14:18.849 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (2705 ms) 2024-04-14T15:14:18.849 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2024-04-14T15:14:18.849 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (3007 ms) 2024-04-14T15:14:18.849 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2024-04-14T15:14:18.849 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (3027 ms) 2024-04-14T15:14:18.849 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2024-04-14T15:14:18.849 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (2987 ms) 2024-04-14T15:14:18.849 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2024-04-14T15:14:18.849 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (3026 ms) 2024-04-14T15:14:18.849 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2024-04-14T15:14:18.849 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (2967 ms) 2024-04-14T15:14:18.849 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2024-04-14T15:14:18.849 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (3004 ms) 2024-04-14T15:14:18.850 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2024-04-14T15:14:18.850 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (3070 ms) 2024-04-14T15:14:18.850 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2024-04-14T15:14:18.850 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (2956 ms) 2024-04-14T15:14:18.850 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2024-04-14T15:14:18.850 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (3032 ms) 2024-04-14T15:14:18.850 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2024-04-14T15:14:18.850 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (2948 ms) 2024-04-14T15:14:18.850 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2024-04-14T15:14:18.850 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (2986 ms) 2024-04-14T15:14:18.850 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2024-04-14T15:14:18.850 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ OK ] LibRadosAio.Flush (3012 ms) 2024-04-14T15:14:18.850 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2024-04-14T15:14:18.851 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (3000 ms) 2024-04-14T15:14:18.851 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2024-04-14T15:14:18.851 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (3015 ms) 2024-04-14T15:14:18.851 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2024-04-14T15:14:18.851 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (2983 ms) 2024-04-14T15:14:18.851 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2024-04-14T15:14:18.851 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (3020 ms) 2024-04-14T15:14:18.851 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2024-04-14T15:14:18.851 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (2993 ms) 2024-04-14T15:14:18.851 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ RUN ] LibRadosAio.Operate2Mtime 2024-04-14T15:14:18.851 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ OK ] LibRadosAio.Operate2Mtime (2991 ms) 2024-04-14T15:14:18.851 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2024-04-14T15:14:18.851 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (2996 ms) 2024-04-14T15:14:18.852 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2024-04-14T15:14:18.852 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (3000 ms) 2024-04-14T15:14:18.852 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2024-04-14T15:14:18.852 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (3004 ms) 2024-04-14T15:14:18.852 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2024-04-14T15:14:18.852 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (2999 ms) 2024-04-14T15:14:18.853 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2024-04-14T15:14:18.853 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (3017 ms) 2024-04-14T15:14:18.853 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [----------] 26 tests from LibRadosAio (77710 ms total) 2024-04-14T15:14:18.853 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: 2024-04-14T15:14:18.853 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2024-04-14T15:14:18.853 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleWrite 2024-04-14T15:14:18.880 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:14:18.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:18.904 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:18.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:14:18.998 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:18 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:14:18.999 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:18 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:14:18.999 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:18 smithi090 ceph-mon[41670]: from='client.48328 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi090-93342-27"}]': finished 2024-04-14T15:14:18.999 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:18 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:14:18.999 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:18 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-13"}]: dispatch 2024-04-14T15:14:18.999 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:18 smithi090 ceph-mon[41670]: osdmap e191: 8 total, 8 up, 8 in 2024-04-14T15:14:18.999 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:18 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-13"}]: dispatch 2024-04-14T15:14:18.999 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:18 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/4059441647' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi090-93363-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:18.999 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:18 smithi090 ceph-mon[41670]: from='client.48364 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi090-93363-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:18.999 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:18 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1487099185' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi090-93342-28"}]: dispatch 2024-04-14T15:14:18.999 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:18 smithi090 ceph-mon[41670]: from='client.48370 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi090-93342-28"}]: dispatch 2024-04-14T15:14:18.999 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:18 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1487099185' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi090-93342-28"}]: dispatch 2024-04-14T15:14:18.999 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:18 smithi090 ceph-mon[41670]: from='client.48370 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi090-93342-28"}]: dispatch 2024-04-14T15:14:19.000 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:18 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1487099185' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi090-93342-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:14:19.000 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:18 smithi090 ceph-mon[41670]: from='client.48370 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi090-93342-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:14:19.000 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:18 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:19.000 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:18 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:14:19.000 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:18 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:14:19.000 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:18 smithi090 ceph-mon[30399]: from='client.48328 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi090-93342-27"}]': finished 2024-04-14T15:14:19.000 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:18 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:14:19.000 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:18 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-13"}]: dispatch 2024-04-14T15:14:19.000 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:18 smithi090 ceph-mon[30399]: osdmap e191: 8 total, 8 up, 8 in 2024-04-14T15:14:19.000 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:18 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-13"}]: dispatch 2024-04-14T15:14:19.000 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:18 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/4059441647' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi090-93363-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:19.000 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:18 smithi090 ceph-mon[30399]: from='client.48364 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi090-93363-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:19.001 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:18 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1487099185' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi090-93342-28"}]: dispatch 2024-04-14T15:14:19.001 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:18 smithi090 ceph-mon[30399]: from='client.48370 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi090-93342-28"}]: dispatch 2024-04-14T15:14:19.001 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:18 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1487099185' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi090-93342-28"}]: dispatch 2024-04-14T15:14:19.001 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:18 smithi090 ceph-mon[30399]: from='client.48370 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi090-93342-28"}]: dispatch 2024-04-14T15:14:19.001 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:18 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1487099185' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi090-93342-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:14:19.001 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:18 smithi090 ceph-mon[30399]: from='client.48370 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi090-93342-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:14:19.001 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:18 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:19.004 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:19.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:19.030 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:19.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:19.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:18 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:14:19.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:18 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:14:19.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:18 smithi092 ceph-mon[33966]: from='client.48328 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi090-93342-27"}]': finished 2024-04-14T15:14:19.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:18 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:14:19.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:18 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-13"}]: dispatch 2024-04-14T15:14:19.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:18 smithi092 ceph-mon[33966]: osdmap e191: 8 total, 8 up, 8 in 2024-04-14T15:14:19.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:18 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-13"}]: dispatch 2024-04-14T15:14:19.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:18 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/4059441647' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi090-93363-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:19.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:18 smithi092 ceph-mon[33966]: from='client.48364 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi090-93363-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:19.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:18 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1487099185' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi090-93342-28"}]: dispatch 2024-04-14T15:14:19.124 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:18 smithi092 ceph-mon[33966]: from='client.48370 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi090-93342-28"}]: dispatch 2024-04-14T15:14:19.124 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:18 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1487099185' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi090-93342-28"}]: dispatch 2024-04-14T15:14:19.124 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:18 smithi092 ceph-mon[33966]: from='client.48370 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi090-93342-28"}]: dispatch 2024-04-14T15:14:19.124 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:18 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1487099185' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi090-93342-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:14:19.124 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:18 smithi092 ceph-mon[33966]: from='client.48370 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi090-93342-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:14:19.124 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:18 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:19.130 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:14:19.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:19.155 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:19.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:14:19.175 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-14T15:14:19.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.220+0000 7fb2546f7700 1 -- 172.21.15.90:0/105922966 >> v1:172.21.15.90:6790/0 conn(0x7fb24c108880 legacy=0x7fb24c108c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:19.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.220+0000 7fb2546f7700 1 -- 172.21.15.90:0/105922966 shutdown_connections 2024-04-14T15:14:19.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.220+0000 7fb2546f7700 1 -- 172.21.15.90:0/105922966 >> 172.21.15.90:0/105922966 conn(0x7fb24c0fe390 msgr2=0x7fb24c1007b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:14:19.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.220+0000 7fb2546f7700 1 -- 172.21.15.90:0/105922966 shutdown_connections 2024-04-14T15:14:19.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.220+0000 7fb2546f7700 1 -- 172.21.15.90:0/105922966 wait complete. 2024-04-14T15:14:19.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.221+0000 7fb2546f7700 1 librados: starting msgr at 2024-04-14T15:14:19.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.221+0000 7fb2546f7700 1 librados: starting objecter 2024-04-14T15:14:19.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.221+0000 7fb2546f7700 1 Processor -- start 2024-04-14T15:14:19.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.221+0000 7fb2546f7700 1 -- start start 2024-04-14T15:14:19.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.221+0000 7fb2546f7700 1 librados: setting wanted keys 2024-04-14T15:14:19.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.221+0000 7fb2546f7700 1 librados: calling monclient init 2024-04-14T15:14:19.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.221+0000 7fb2546f7700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb24c1aa070 con 0x7fb24c10b730 2024-04-14T15:14:19.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.221+0000 7fb2546f7700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb24c1ab290 con 0x7fb24c10f300 2024-04-14T15:14:19.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.221+0000 7fb2546f7700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb24c1ac4d0 con 0x7fb24c1a62c0 2024-04-14T15:14:19.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.221+0000 7fb252493700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fb24c10b730 0x7fb24c0766c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:57386/0 (socket says 172.21.15.90:57386) 2024-04-14T15:14:19.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.221+0000 7fb252493700 1 -- 172.21.15.90:0/3536155582 learned_addr learned my addr 172.21.15.90:0/3536155582 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:14:19.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.221+0000 7fb2437fe700 10 client.?.objecter ms_handle_connect 0x7fb24c1a62c0 2024-04-14T15:14:19.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.221+0000 7fb2437fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:14:19.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.221+0000 7fb2437fe700 10 client.?.objecter ms_handle_connect 0x7fb24c10b730 2024-04-14T15:14:19.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.221+0000 7fb2437fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:14:19.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.221+0000 7fb2437fe700 10 client.?.objecter ms_handle_connect 0x7fb24c10f300 2024-04-14T15:14:19.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.221+0000 7fb2437fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:14:19.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.221+0000 7fb2437fe700 1 -- 172.21.15.90:0/3536155582 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3865376390 0 0) 0x7fb24c1ac4d0 con 0x7fb24c1a62c0 2024-04-14T15:14:19.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.221+0000 7fb2437fe700 1 -- 172.21.15.90:0/3536155582 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb238003aa0 con 0x7fb24c1a62c0 2024-04-14T15:14:19.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.221+0000 7fb2437fe700 1 -- 172.21.15.90:0/3536155582 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3634702638 0 0) 0x7fb24c1ab290 con 0x7fb24c10f300 2024-04-14T15:14:19.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.221+0000 7fb2437fe700 1 -- 172.21.15.90:0/3536155582 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb24c1ac4d0 con 0x7fb24c10f300 2024-04-14T15:14:19.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.221+0000 7fb2437fe700 1 -- 172.21.15.90:0/3536155582 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1808112983 0 0) 0x7fb24c1aa070 con 0x7fb24c10b730 2024-04-14T15:14:19.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.222+0000 7fb2437fe700 1 -- 172.21.15.90:0/3536155582 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb24c1ab290 con 0x7fb24c10b730 2024-04-14T15:14:19.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.222+0000 7fb2437fe700 1 -- 172.21.15.90:0/3536155582 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2897013570 0 0) 0x7fb24c1ac4d0 con 0x7fb24c10f300 2024-04-14T15:14:19.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.222+0000 7fb2437fe700 1 -- 172.21.15.90:0/3536155582 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb24c1aa070 con 0x7fb24c10f300 2024-04-14T15:14:19.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.222+0000 7fb2437fe700 1 -- 172.21.15.90:0/3536155582 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fb23c0010b0 con 0x7fb24c10f300 2024-04-14T15:14:19.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.222+0000 7fb2437fe700 1 -- 172.21.15.90:0/3536155582 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1133620224 0 0) 0x7fb24c1ab290 con 0x7fb24c10b730 2024-04-14T15:14:19.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.222+0000 7fb2437fe700 1 -- 172.21.15.90:0/3536155582 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb24c1ac4d0 con 0x7fb24c10b730 2024-04-14T15:14:19.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.222+0000 7fb2437fe700 1 -- 172.21.15.90:0/3536155582 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3227445318 0 0) 0x7fb238003aa0 con 0x7fb24c1a62c0 2024-04-14T15:14:19.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.222+0000 7fb2437fe700 1 -- 172.21.15.90:0/3536155582 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb24c1ab290 con 0x7fb24c1a62c0 2024-04-14T15:14:19.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.222+0000 7fb2437fe700 1 -- 172.21.15.90:0/3536155582 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fb248003820 con 0x7fb24c10b730 2024-04-14T15:14:19.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.222+0000 7fb2437fe700 1 -- 172.21.15.90:0/3536155582 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fb244001af0 con 0x7fb24c1a62c0 2024-04-14T15:14:19.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.222+0000 7fb2437fe700 1 -- 172.21.15.90:0/3536155582 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1511652006 0 0) 0x7fb24c1aa070 con 0x7fb24c10f300 2024-04-14T15:14:19.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.222+0000 7fb2437fe700 1 -- 172.21.15.90:0/3536155582 >> v1:172.21.15.90:6790/0 conn(0x7fb24c1a62c0 legacy=0x7fb24c1a8730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:19.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.222+0000 7fb2437fe700 1 -- 172.21.15.90:0/3536155582 >> v1:172.21.15.90:6789/0 conn(0x7fb24c10b730 legacy=0x7fb24c0766c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:19.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.222+0000 7fb2437fe700 1 -- 172.21.15.90:0/3536155582 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb24c1ad710 con 0x7fb24c10f300 2024-04-14T15:14:19.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.222+0000 7fb2546f7700 1 -- 172.21.15.90:0/3536155582 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb24c1aa2a0 con 0x7fb24c10f300 2024-04-14T15:14:19.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.222+0000 7fb2546f7700 10 client.48382.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:14:19.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.222+0000 7fb2546f7700 1 -- 172.21.15.90:0/3536155582 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb24c1aa760 con 0x7fb24c10f300 2024-04-14T15:14:19.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.222+0000 7fb2546f7700 1 librados: init done 2024-04-14T15:14:19.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.222+0000 7fb2437fe700 1 -- 172.21.15.90:0/3536155582 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fb23c003b20 con 0x7fb24c10f300 2024-04-14T15:14:19.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.222+0000 7fb2437fe700 1 -- 172.21.15.90:0/3536155582 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fb23c0049a0 con 0x7fb24c10f300 2024-04-14T15:14:19.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.223+0000 7fb2546f7700 1 -- 172.21.15.90:0/3536155582 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb230005150 con 0x7fb24c10f300 2024-04-14T15:14:19.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.224+0000 7fb2437fe700 1 -- 172.21.15.90:0/3536155582 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fb23c01b650 con 0x7fb24c10f300 2024-04-14T15:14:19.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.225+0000 7fb2437fe700 1 -- 172.21.15.90:0/3536155582 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(191..191 src has 51..191) ==== 9690+0+0 (unknown 1224328198 0 0) 0x7fb23c089d70 con 0x7fb24c10f300 2024-04-14T15:14:19.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.225+0000 7fb2437fe700 10 client.48382.objecter ms_dispatch 0x7fb24c001550 osd_map(191..191 src has 51..191) 2024-04-14T15:14:19.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.225+0000 7fb2437fe700 3 client.48382.objecter handle_osd_map got epochs [191,191] > 0 2024-04-14T15:14:19.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.225+0000 7fb2437fe700 3 client.48382.objecter handle_osd_map decoding full epoch 191 2024-04-14T15:14:19.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.225+0000 7fb2437fe700 20 client.48382.objecter dump_active .. 0 homeless 2024-04-14T15:14:19.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.225+0000 7fb2437fe700 10 client.48382.objecter ms_handle_connect 0x7fb2380713d0 2024-04-14T15:14:19.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.229+0000 7fb2437fe700 1 -- 172.21.15.90:0/3536155582 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fb23c05e6f0 con 0x7fb24c10f300 2024-04-14T15:14:19.256 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:14:19.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:19.281 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:19.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:14:19.382 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:14:19.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:19.407 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:19.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:14:19.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.440+0000 7fb2546f7700 1 -- 172.21.15.90:0/3536155582 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) -- 0x7fb230005440 con 0x7fb24c10f300 2024-04-14T15:14:19.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.443+0000 7fb2437fe700 1 -- 172.21.15.90:0/3536155582 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v191) ==== 81+0+61425 (unknown 2673198947 0 100336650) 0x7fb23c053f40 con 0x7fb24c10f300 2024-04-14T15:14:19.444 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:14:19.444 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-14T15:14:19.445 INFO:teuthology.orchestra.run.smithi090.stdout: "epoch": 191, 2024-04-14T15:14:19.445 INFO:teuthology.orchestra.run.smithi090.stdout: "fsid": "0e288098-fa70-11ee-bc8e-c7b262605968", 2024-04-14T15:14:19.445 INFO:teuthology.orchestra.run.smithi090.stdout: "created": "2024-04-14T15:04:05.808313+0000", 2024-04-14T15:14:19.445 INFO:teuthology.orchestra.run.smithi090.stdout: "modified": "2024-04-14T15:14:18.825285+0000", 2024-04-14T15:14:19.445 INFO:teuthology.orchestra.run.smithi090.stdout: "last_up_change": "2024-04-14T15:09:29.769352+0000", 2024-04-14T15:14:19.445 INFO:teuthology.orchestra.run.smithi090.stdout: "last_in_change": "2024-04-14T15:09:11.803164+0000", 2024-04-14T15:14:19.445 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-14T15:14:19.445 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_num": 5801984, 2024-04-14T15:14:19.445 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_set": [ 2024-04-14T15:14:19.445 INFO:teuthology.orchestra.run.smithi090.stdout: "noscrub", 2024-04-14T15:14:19.445 INFO:teuthology.orchestra.run.smithi090.stdout: "pglog_hardlimit", 2024-04-14T15:14:19.446 INFO:teuthology.orchestra.run.smithi090.stdout: "purged_snapdirs", 2024-04-14T15:14:19.446 INFO:teuthology.orchestra.run.smithi090.stdout: "recovery_deletes", 2024-04-14T15:14:19.446 INFO:teuthology.orchestra.run.smithi090.stdout: "sortbitwise" 2024-04-14T15:14:19.446 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-14T15:14:19.446 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_version": 41, 2024-04-14T15:14:19.446 INFO:teuthology.orchestra.run.smithi090.stdout: "full_ratio": 0.94999998807907104, 2024-04-14T15:14:19.446 INFO:teuthology.orchestra.run.smithi090.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-14T15:14:19.446 INFO:teuthology.orchestra.run.smithi090.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-14T15:14:19.446 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_snapshot": "", 2024-04-14T15:14:19.446 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_max": 210, 2024-04-14T15:14:19.446 INFO:teuthology.orchestra.run.smithi090.stdout: "max_osd": 8, 2024-04-14T15:14:19.446 INFO:teuthology.orchestra.run.smithi090.stdout: "require_min_compat_client": "luminous", 2024-04-14T15:14:19.446 INFO:teuthology.orchestra.run.smithi090.stdout: "min_compat_client": "luminous", 2024-04-14T15:14:19.446 INFO:teuthology.orchestra.run.smithi090.stdout: "require_osd_release": "squid", 2024-04-14T15:14:19.447 INFO:teuthology.orchestra.run.smithi090.stdout: "allow_crimson": false, 2024-04-14T15:14:19.447 INFO:teuthology.orchestra.run.smithi090.stdout: "pools": [ 2024-04-14T15:14:19.447 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.447 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 1, 2024-04-14T15:14:19.447 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": ".mgr", 2024-04-14T15:14:19.447 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:07:23.959628+0000", 2024-04-14T15:14:19.447 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-14T15:14:19.447 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool", 2024-04-14T15:14:19.448 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:14:19.448 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:14:19.448 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:14:19.448 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:14:19.448 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:14:19.448 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:14:19.448 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:14:19.448 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:14:19.448 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:14:19.448 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:14:19.448 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 11, 2024-04-14T15:14:19.448 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 11, 2024-04-14T15:14:19.448 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 11, 2024-04-14T15:14:19.448 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 11, 2024-04-14T15:14:19.449 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 11, 2024-04-14T15:14:19.449 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:14:19.449 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:14:19.449 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:14:19.449 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:14:19.449 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:14:19.449 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:14:19.449 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:14:19.449 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.449 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "186", 2024-04-14T15:14:19.449 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:14:19.449 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:14:19.449 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "184", 2024-04-14T15:14:19.449 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:14:19.449 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:14:19.450 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:14:19.450 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:14:19.450 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:14:19.450 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:14:19.450 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:14:19.450 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:14:19.450 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:14:19.450 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:14:19.450 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:14:19.450 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:14:19.450 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:14:19.450 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:14:19.450 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:14:19.450 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:14:19.451 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:14:19.451 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:14:19.451 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:14:19.451 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:14:19.451 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:14:19.451 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:14:19.451 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:14:19.451 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.451 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:14:19.451 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:14:19.451 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:14:19.451 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:14:19.451 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:14:19.451 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:14:19.452 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:14:19.452 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:14:19.452 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:14:19.452 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:14:19.452 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:14:19.452 INFO:teuthology.orchestra.run.smithi090.stdout: "options": { 2024-04-14T15:14:19.452 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_max": 32, 2024-04-14T15:14:19.452 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_min": 1 2024-04-14T15:14:19.452 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.452 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:14:19.452 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": {} 2024-04-14T15:14:19.452 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.452 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:14:19.452 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:14:19.453 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 2.1800000667572021, 2024-04-14T15:14:19.453 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 2.1800000667572021, 2024-04-14T15:14:19.453 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:14:19.453 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 2.1800000667572021, 2024-04-14T15:14:19.453 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 2.1800000667572021, 2024-04-14T15:14:19.453 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.90223121643066406, 2024-04-14T15:14:19.453 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 0.89999997615814209, 2024-04-14T15:14:19.453 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-04-14T15:14:19.453 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.453 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.453 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.453 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 2, 2024-04-14T15:14:19.453 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": ".rgw.root", 2024-04-14T15:14:19.453 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:09:41.608420+0000", 2024-04-14T15:14:19.453 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-14T15:14:19.454 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool", 2024-04-14T15:14:19.454 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:14:19.454 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:14:19.454 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:14:19.454 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:14:19.454 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:14:19.454 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:14:19.454 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:14:19.454 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:14:19.454 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:14:19.454 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:14:19.454 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:14:19.454 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:14:19.455 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:14:19.455 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:14:19.455 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:14:19.455 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:14:19.455 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:14:19.455 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:14:19.455 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:14:19.455 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:14:19.455 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:14:19.455 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:14:19.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.455 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "66", 2024-04-14T15:14:19.455 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:14:19.455 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:14:19.455 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:14:19.456 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:14:19.456 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:14:19.456 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:14:19.456 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:14:19.456 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:14:19.456 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:14:19.456 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:14:19.456 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:14:19.456 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:14:19.456 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:14:19.456 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:14:19.456 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:14:19.456 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:14:19.456 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:14:19.456 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:14:19.456 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:14:19.457 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:14:19.457 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:14:19.457 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:14:19.457 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:14:19.457 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:14:19.457 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:14:19.457 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:14:19.457 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.457 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:14:19.458 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:14:19.458 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:14:19.458 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:14:19.458 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:14:19.458 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:14:19.458 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:14:19.458 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:14:19.458 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:14:19.458 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:14:19.458 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:14:19.458 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:14:19.458 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:14:19.458 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw": {} 2024-04-14T15:14:19.459 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.459 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:14:19.459 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:14:19.459 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 1.5, 2024-04-14T15:14:19.459 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 1.5, 2024-04-14T15:14:19.459 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:14:19.459 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 1.5, 2024-04-14T15:14:19.459 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 1.5, 2024-04-14T15:14:19.459 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.90223121643066406, 2024-04-14T15:14:19.459 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 0.89999997615814209, 2024-04-14T15:14:19.459 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-04-14T15:14:19.459 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.459 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.459 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.459 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 3, 2024-04-14T15:14:19.459 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "datapool", 2024-04-14T15:14:19.460 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:09:41.744391+0000", 2024-04-14T15:14:19.460 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 8193, 2024-04-14T15:14:19.460 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-14T15:14:19.460 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:14:19.460 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:14:19.460 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:14:19.460 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:14:19.460 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:14:19.460 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:14:19.460 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:14:19.460 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:14:19.460 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:14:19.460 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:14:19.460 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 3, 2024-04-14T15:14:19.460 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 3, 2024-04-14T15:14:19.461 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 3, 2024-04-14T15:14:19.461 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 3, 2024-04-14T15:14:19.461 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 3, 2024-04-14T15:14:19.461 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:14:19.461 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:14:19.461 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:14:19.461 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:14:19.461 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:14:19.461 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:14:19.462 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:14:19.462 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.462 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "69", 2024-04-14T15:14:19.462 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:14:19.462 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:14:19.462 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:14:19.463 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:14:19.463 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:14:19.463 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 2, 2024-04-14T15:14:19.463 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 69, 2024-04-14T15:14:19.463 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:14:19.463 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:14:19.463 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:14:19.463 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:14:19.463 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:14:19.463 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:14:19.463 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:14:19.463 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:14:19.464 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:14:19.464 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:14:19.464 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:14:19.464 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:14:19.464 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:14:19.464 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:14:19.464 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:14:19.464 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:14:19.464 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:14:19.464 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:14:19.464 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:14:19.464 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.464 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:14:19.464 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:14:19.465 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:14:19.465 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:14:19.465 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:14:19.465 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:14:19.465 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:14:19.465 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:14:19.465 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:14:19.465 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:14:19.465 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:14:19.465 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:14:19.465 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:14:19.465 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd": {} 2024-04-14T15:14:19.466 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.466 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:14:19.466 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:14:19.466 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 2.6500000953674316, 2024-04-14T15:14:19.466 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 2.6500000953674316, 2024-04-14T15:14:19.466 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 0.87999999523162842, 2024-04-14T15:14:19.466 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 2.3299999237060547, 2024-04-14T15:14:19.466 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 2.3299999237060547, 2024-04-14T15:14:19.466 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.90223121643066406, 2024-04-14T15:14:19.467 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 0.89999997615814209, 2024-04-14T15:14:19.467 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-04-14T15:14:19.468 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.468 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.468 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.468 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 4, 2024-04-14T15:14:19.468 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "default.rgw.log", 2024-04-14T15:14:19.469 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:09:43.088157+0000", 2024-04-14T15:14:19.469 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-14T15:14:19.469 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool", 2024-04-14T15:14:19.469 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:14:19.469 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:14:19.469 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:14:19.469 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:14:19.469 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:14:19.469 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:14:19.469 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:14:19.470 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:14:19.470 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:14:19.470 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:14:19.470 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:14:19.470 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:14:19.470 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:14:19.470 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:14:19.470 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:14:19.470 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:14:19.470 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:14:19.471 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:14:19.471 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:14:19.471 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:14:19.471 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:14:19.471 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:14:19.471 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.471 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "68", 2024-04-14T15:14:19.471 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:14:19.471 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:14:19.471 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:14:19.471 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:14:19.471 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:14:19.472 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:14:19.472 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:14:19.472 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:14:19.472 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:14:19.472 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:14:19.472 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:14:19.472 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:14:19.472 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:14:19.472 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:14:19.472 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:14:19.472 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:14:19.472 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:14:19.472 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:14:19.472 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:14:19.473 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:14:19.473 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:14:19.473 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:14:19.473 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:14:19.474 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:14:19.474 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:14:19.474 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:14:19.474 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.474 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:14:19.474 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:14:19.475 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:14:19.475 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:14:19.475 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:14:19.475 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:14:19.475 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:14:19.475 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:14:19.475 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:14:19.475 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:14:19.475 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:14:19.475 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:14:19.475 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:14:19.475 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw": {} 2024-04-14T15:14:19.476 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.476 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:14:19.476 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:14:19.476 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 2.25, 2024-04-14T15:14:19.476 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 2.25, 2024-04-14T15:14:19.476 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:14:19.476 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 2.25, 2024-04-14T15:14:19.476 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 2.25, 2024-04-14T15:14:19.476 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.90223121643066406, 2024-04-14T15:14:19.476 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 0.89999997615814209, 2024-04-14T15:14:19.476 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-04-14T15:14:19.476 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.476 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.477 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.477 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 5, 2024-04-14T15:14:19.477 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "default.rgw.control", 2024-04-14T15:14:19.477 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:09:45.042722+0000", 2024-04-14T15:14:19.477 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-14T15:14:19.477 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool", 2024-04-14T15:14:19.477 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:14:19.477 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:14:19.477 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:14:19.478 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:14:19.478 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:14:19.478 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:14:19.478 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:14:19.478 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:14:19.478 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:14:19.478 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:14:19.478 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:14:19.479 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:14:19.479 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:14:19.479 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:14:19.479 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:14:19.479 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:14:19.479 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:14:19.479 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:14:19.479 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:14:19.479 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:14:19.479 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:14:19.480 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:14:19.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.480 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "70", 2024-04-14T15:14:19.480 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:14:19.480 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:14:19.480 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:14:19.480 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:14:19.480 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:14:19.480 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:14:19.480 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:14:19.480 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:14:19.480 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:14:19.480 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:14:19.480 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:14:19.481 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:14:19.481 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:14:19.481 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:14:19.481 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:14:19.481 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:14:19.481 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:14:19.481 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:14:19.481 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:14:19.481 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:14:19.481 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:14:19.481 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:14:19.481 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:14:19.482 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:14:19.482 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:14:19.482 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:14:19.482 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.482 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:14:19.482 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:14:19.482 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:14:19.482 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:14:19.482 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:14:19.483 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:14:19.483 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:14:19.483 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:14:19.483 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:14:19.484 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:14:19.484 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:14:19.484 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:14:19.484 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:14:19.486 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw": {} 2024-04-14T15:14:19.486 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.486 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:14:19.486 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:14:19.486 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 1.5, 2024-04-14T15:14:19.486 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 1.5, 2024-04-14T15:14:19.487 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:14:19.487 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 1.5, 2024-04-14T15:14:19.487 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 1.5, 2024-04-14T15:14:19.487 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.90223121643066406, 2024-04-14T15:14:19.487 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 0.89999997615814209, 2024-04-14T15:14:19.487 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-04-14T15:14:19.487 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.488 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 6, 2024-04-14T15:14:19.488 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "default.rgw.meta", 2024-04-14T15:14:19.488 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:09:47.111336+0000", 2024-04-14T15:14:19.488 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-14T15:14:19.488 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool", 2024-04-14T15:14:19.488 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:14:19.488 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:14:19.489 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:14:19.489 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:14:19.489 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:14:19.489 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:14:19.490 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:14:19.490 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:14:19.490 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:14:19.490 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:14:19.490 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:14:19.490 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:14:19.490 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:14:19.491 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:14:19.491 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:14:19.491 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:14:19.491 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:14:19.491 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:14:19.491 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:14:19.492 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:14:19.492 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:14:19.492 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:14:19.492 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.492 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "72", 2024-04-14T15:14:19.492 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:14:19.492 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:14:19.493 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:14:19.493 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:14:19.493 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:14:19.493 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:14:19.493 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:14:19.493 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:14:19.493 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:14:19.494 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:14:19.494 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:14:19.494 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:14:19.494 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:14:19.494 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:14:19.495 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:14:19.495 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:14:19.495 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:14:19.496 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:14:19.496 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:14:19.496 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:14:19.496 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:14:19.496 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:14:19.496 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:14:19.496 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:14:19.497 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:14:19.497 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:14:19.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.497 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:14:19.497 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:14:19.497 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:14:19.497 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:14:19.497 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:14:19.498 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:14:19.498 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:14:19.498 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:14:19.498 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:14:19.498 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:14:19.498 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:14:19.498 INFO:teuthology.orchestra.run.smithi090.stdout: "options": { 2024-04-14T15:14:19.498 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_bias": 4 2024-04-14T15:14:19.499 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.499 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:14:19.499 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw": {} 2024-04-14T15:14:19.499 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.499 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:14:19.499 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:14:19.499 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 1.75, 2024-04-14T15:14:19.500 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 1.75, 2024-04-14T15:14:19.500 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:14:19.500 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 1.75, 2024-04-14T15:14:19.500 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 1.75, 2024-04-14T15:14:19.501 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.90223121643066406, 2024-04-14T15:14:19.501 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 0.89999997615814209, 2024-04-14T15:14:19.501 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-04-14T15:14:19.501 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.501 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.501 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 11, 2024-04-14T15:14:19.501 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "ceph_test_rados_api_asio", 2024-04-14T15:14:19.501 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:12:54.225671+0000", 2024-04-14T15:14:19.502 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-14T15:14:19.502 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool", 2024-04-14T15:14:19.502 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:14:19.502 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:14:19.502 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:14:19.502 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:14:19.502 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:14:19.502 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:14:19.503 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:14:19.503 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:14:19.503 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:14:19.503 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:14:19.503 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:14:19.503 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:14:19.503 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:14:19.503 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:14:19.504 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:14:19.504 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:14:19.504 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:14:19.504 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:14:19.504 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:14:19.504 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:14:19.504 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:14:19.504 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:14:19.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.505 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "109", 2024-04-14T15:14:19.505 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:14:19.505 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:14:19.505 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:14:19.505 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:14:19.505 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:14:19.505 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:14:19.505 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:14:19.505 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:14:19.506 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:14:19.506 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:14:19.506 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:14:19.506 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:14:19.506 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:14:19.506 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:14:19.506 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:14:19.506 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:14:19.506 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:14:19.506 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:14:19.506 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:14:19.506 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:14:19.506 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:14:19.507 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:14:19.507 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:14:19.507 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:14:19.507 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:14:19.507 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:14:19.507 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.507 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:14:19.507 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:14:19.507 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:14:19.508 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:14:19.508 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:14:19.508 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:14:19.508 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:14:19.508 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:14:19.508 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:14:19.508 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:14:19.508 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:14:19.508 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:14:19.509 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": {}, 2024-04-14T15:14:19.509 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:14:19.509 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:14:19.509 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 1.75, 2024-04-14T15:14:19.509 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 1.75, 2024-04-14T15:14:19.509 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:14:19.509 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 1.75, 2024-04-14T15:14:19.509 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 1.75, 2024-04-14T15:14:19.509 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.90223121643066406, 2024-04-14T15:14:19.510 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 0.89999997615814209, 2024-04-14T15:14:19.510 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-04-14T15:14:19.510 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.510 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 13, 2024-04-14T15:14:19.510 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "LibRadosList_smithi090-93444-1", 2024-04-14T15:14:19.511 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:12:54.244611+0000", 2024-04-14T15:14:19.511 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-14T15:14:19.511 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool", 2024-04-14T15:14:19.511 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:14:19.511 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:14:19.511 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:14:19.511 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:14:19.511 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:14:19.511 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:14:19.512 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:14:19.512 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:14:19.512 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:14:19.512 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:14:19.512 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 25, 2024-04-14T15:14:19.512 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 11, 2024-04-14T15:14:19.512 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 11, 2024-04-14T15:14:19.512 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 11, 2024-04-14T15:14:19.513 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 24, 2024-04-14T15:14:19.513 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:14:19.513 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "13.19", 2024-04-14T15:14:19.513 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 188, 2024-04-14T15:14:19.513 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 187, 2024-04-14T15:14:19.513 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 187, 2024-04-14T15:14:19.513 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:14:19.513 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:14:19.514 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.514 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "190", 2024-04-14T15:14:19.514 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:14:19.514 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "190", 2024-04-14T15:14:19.514 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "190", 2024-04-14T15:14:19.514 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:14:19.514 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:14:19.514 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:14:19.515 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:14:19.515 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:14:19.515 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:14:19.515 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:14:19.515 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:14:19.515 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:14:19.515 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:14:19.515 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:14:19.516 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:14:19.516 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:14:19.516 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:14:19.516 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:14:19.516 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:14:19.516 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:14:19.516 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:14:19.516 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:14:19.517 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:14:19.517 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:14:19.517 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:14:19.517 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:14:19.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.517 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:14:19.517 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:14:19.518 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:14:19.518 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:14:19.518 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:14:19.518 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:14:19.518 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:14:19.518 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:14:19.518 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:14:19.518 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:14:19.519 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:14:19.519 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:14:19.519 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:14:19.519 INFO:teuthology.orchestra.run.smithi090.stdout: "rados": {} 2024-04-14T15:14:19.519 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.519 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:14:19.519 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:14:19.520 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 1.9199999570846558, 2024-04-14T15:14:19.520 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 1.9199999570846558, 2024-04-14T15:14:19.520 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:14:19.520 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 1.9199999570846558, 2024-04-14T15:14:19.520 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 1.9199999570846558, 2024-04-14T15:14:19.520 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.90223121643066406, 2024-04-14T15:14:19.520 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 0.89999997615814209, 2024-04-14T15:14:19.520 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-04-14T15:14:19.521 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.521 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 110, 2024-04-14T15:14:19.521 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "test-rados-api-smithi090-93652-6", 2024-04-14T15:14:19.521 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:13:07.960794+0000", 2024-04-14T15:14:19.521 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 8193, 2024-04-14T15:14:19.521 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-14T15:14:19.522 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:14:19.522 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:14:19.522 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:14:19.522 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:14:19.522 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:14:19.522 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:14:19.522 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:14:19.522 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:14:19.522 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:14:19.522 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:14:19.523 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:14:19.523 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:14:19.523 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:14:19.523 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:14:19.523 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:14:19.523 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:14:19.523 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:14:19.523 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:14:19.523 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:14:19.523 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:14:19.523 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:14:19.523 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:14:19.523 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.524 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "191", 2024-04-14T15:14:19.524 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "191", 2024-04-14T15:14:19.524 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "191", 2024-04-14T15:14:19.524 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "191", 2024-04-14T15:14:19.524 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:14:19.524 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:14:19.524 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 6, 2024-04-14T15:14:19.524 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 190, 2024-04-14T15:14:19.524 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:14:19.524 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:14:19.525 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:14:19.525 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:14:19.525 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [ 2024-04-14T15:14:19.525 INFO:teuthology.orchestra.run.smithi090.stdout: 172 2024-04-14T15:14:19.525 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-14T15:14:19.525 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:14:19.525 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:14:19.525 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:14:19.525 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:14:19.525 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:14:19.525 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:14:19.525 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:14:19.525 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:14:19.525 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:14:19.525 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:14:19.526 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:14:19.526 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:14:19.526 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:14:19.526 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:14:19.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.526 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:14:19.526 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:14:19.526 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:14:19.526 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:14:19.526 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:14:19.526 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:14:19.526 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:14:19.527 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:14:19.527 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:14:19.527 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:14:19.527 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:14:19.527 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:14:19.527 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:14:19.527 INFO:teuthology.orchestra.run.smithi090.stdout: "rados": {} 2024-04-14T15:14:19.527 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.527 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:14:19.527 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:14:19.527 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 2, 2024-04-14T15:14:19.527 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 2, 2024-04-14T15:14:19.527 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:14:19.528 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 2, 2024-04-14T15:14:19.528 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 2, 2024-04-14T15:14:19.528 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.90223121643066406, 2024-04-14T15:14:19.528 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 0.89999997615814209, 2024-04-14T15:14:19.528 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-04-14T15:14:19.528 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.528 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.528 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.528 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 172, 2024-04-14T15:14:19.528 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "test-rados-api-smithi090-93652-13", 2024-04-14T15:14:19.528 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:13:34.849790+0000", 2024-04-14T15:14:19.528 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 8201, 2024-04-14T15:14:19.528 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,incomplete_clones,selfmanaged_snaps", 2024-04-14T15:14:19.528 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:14:19.529 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:14:19.529 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:14:19.529 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:14:19.529 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:14:19.529 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:14:19.529 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:14:19.529 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:14:19.529 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:14:19.529 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:14:19.529 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:14:19.529 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:14:19.529 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:14:19.529 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:14:19.529 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:14:19.530 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:14:19.530 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:14:19.530 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:14:19.530 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:14:19.530 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:14:19.530 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:14:19.530 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:14:19.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.530 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "191", 2024-04-14T15:14:19.530 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "191", 2024-04-14T15:14:19.530 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "191", 2024-04-14T15:14:19.530 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "191", 2024-04-14T15:14:19.530 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:14:19.531 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:14:19.531 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 6, 2024-04-14T15:14:19.531 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 190, 2024-04-14T15:14:19.531 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:14:19.531 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:14:19.531 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:14:19.531 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:14:19.531 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:14:19.531 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": 110, 2024-04-14T15:14:19.532 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:14:19.532 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:14:19.532 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "writeback", 2024-04-14T15:14:19.532 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:14:19.532 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:14:19.532 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:14:19.532 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:14:19.532 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:14:19.532 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:14:19.532 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:14:19.532 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:14:19.533 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:14:19.533 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:14:19.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.533 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:14:19.533 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:14:19.533 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:14:19.534 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:14:19.534 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:14:19.534 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:14:19.534 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:14:19.534 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:14:19.534 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:14:19.534 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:14:19.534 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:14:19.534 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:14:19.534 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:14:19.534 INFO:teuthology.orchestra.run.smithi090.stdout: "rados": {} 2024-04-14T15:14:19.534 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.534 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:14:19.535 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:14:19.535 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 1.5, 2024-04-14T15:14:19.535 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 1.5, 2024-04-14T15:14:19.535 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:14:19.535 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 1.5, 2024-04-14T15:14:19.535 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 1.5, 2024-04-14T15:14:19.535 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.90223121643066406, 2024-04-14T15:14:19.535 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 0.89999997615814209, 2024-04-14T15:14:19.535 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-04-14T15:14:19.536 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.536 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.536 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.536 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 205, 2024-04-14T15:14:19.536 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "LibRadosSnapshotsSelfManagedECPP_smithi090-93724-21", 2024-04-14T15:14:19.536 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:14:11.845038+0000", 2024-04-14T15:14:19.536 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 8193, 2024-04-14T15:14:19.536 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-14T15:14:19.536 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 3, 2024-04-14T15:14:19.536 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:14:19.536 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:14:19.537 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 1, 2024-04-14T15:14:19.537 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:14:19.537 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:14:19.537 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:14:19.537 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:14:19.537 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:14:19.537 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:14:19.537 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 8, 2024-04-14T15:14:19.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.446+0000 7fb2546f7700 10 librados: watch_flush enter 2024-04-14T15:14:19.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.446+0000 7fb2546f7700 10 librados: watch_flush exit 2024-04-14T15:14:19.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.446+0000 7fb2546f7700 20 client.48382.objecter shutdown clearing up homeless session... 2024-04-14T15:14:19.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.446+0000 7fb2546f7700 10 client.48382.objecter successfully canceled tick 2024-04-14T15:14:19.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.446+0000 7fb2546f7700 1 -- 172.21.15.90:0/3536155582 >> v1:172.21.15.90:6800/420935427 conn(0x7fb2380713d0 legacy=0x7fb238073890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:19.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.446+0000 7fb2546f7700 1 -- 172.21.15.90:0/3536155582 >> v1:172.21.15.92:6789/0 conn(0x7fb24c10f300 legacy=0x7fb24c1a4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:19.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.446+0000 7fb2546f7700 1 -- 172.21.15.90:0/3536155582 shutdown_connections 2024-04-14T15:14:19.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.446+0000 7fb2546f7700 1 -- 172.21.15.90:0/3536155582 >> 172.21.15.90:0/3536155582 conn(0x7fb24c0fe390 msgr2=0x7fb24c110e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:14:19.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.446+0000 7fb2546f7700 1 -- 172.21.15.90:0/3536155582 shutdown_connections 2024-04-14T15:14:19.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.446+0000 7fb2546f7700 1 -- 172.21.15.90:0/3536155582 wait complete. 2024-04-14T15:14:19.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:19.446+0000 7fb2546f7700 1 librados: shutdown 2024-04-14T15:14:19.539 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 8, 2024-04-14T15:14:19.539 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 8, 2024-04-14T15:14:19.539 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 8, 2024-04-14T15:14:19.539 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 8, 2024-04-14T15:14:19.540 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:14:19.540 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:14:19.540 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:14:19.540 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:14:19.540 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:14:19.540 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:14:19.540 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:14:19.540 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.540 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "191", 2024-04-14T15:14:19.540 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:14:19.540 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:14:19.540 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:14:19.540 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:14:19.540 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:14:19.541 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 5, 2024-04-14T15:14:19.541 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 191, 2024-04-14T15:14:19.541 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:14:19.541 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:14:19.541 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:14:19.541 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:14:19.541 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:14:19.541 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:14:19.541 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:14:19.541 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:14:19.541 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:14:19.541 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:14:19.541 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:14:19.541 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:14:19.542 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:14:19.542 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:14:19.542 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:14:19.542 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:14:19.542 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi090-93724-21", 2024-04-14T15:14:19.542 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:14:19.542 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:14:19.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.543 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:14:19.543 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:14:19.543 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:14:19.543 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:14:19.543 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:14:19.543 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:14:19.543 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:14:19.543 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:14:19.543 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 8192, 2024-04-14T15:14:19.543 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:14:19.543 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:14:19.543 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:14:19.543 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": {} 2024-04-14T15:14:19.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.544 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.544 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 210, 2024-04-14T15:14:19.544 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "StatRemovePP_smithi090-93363-24", 2024-04-14T15:14:19.544 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:14:17.898201+0000", 2024-04-14T15:14:19.544 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 32769, 2024-04-14T15:14:19.544 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,creating", 2024-04-14T15:14:19.544 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:14:19.544 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:14:19.544 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:14:19.544 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:14:19.544 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:14:19.544 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:14:19.544 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:14:19.544 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:14:19.545 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:14:19.545 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:14:19.545 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:14:19.545 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:14:19.545 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:14:19.545 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:14:19.546 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:14:19.546 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:14:19.546 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:14:19.546 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:14:19.546 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:14:19.546 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:14:19.547 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:14:19.547 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:14:19.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.547 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "191", 2024-04-14T15:14:19.547 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:14:19.547 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:14:19.547 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:14:19.547 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:14:19.547 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:14:19.547 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:14:19.547 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:14:19.547 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:14:19.547 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:14:19.548 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:14:19.548 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:14:19.548 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:14:19.548 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:14:19.548 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:14:19.548 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:14:19.548 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:14:19.548 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:14:19.548 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:14:19.548 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:14:19.548 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:14:19.548 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:14:19.549 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:14:19.549 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:14:19.549 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:14:19.549 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:14:19.549 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:14:19.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.549 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:14:19.549 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:14:19.549 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:14:19.550 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:14:19.550 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:14:19.550 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:14:19.550 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:14:19.550 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:14:19.550 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:14:19.550 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:14:19.550 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:14:19.550 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:14:19.550 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": {}, 2024-04-14T15:14:19.551 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:14:19.551 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:14:19.551 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 1.5, 2024-04-14T15:14:19.551 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 1.5, 2024-04-14T15:14:19.551 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:14:19.551 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 1.5, 2024-04-14T15:14:19.551 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 1.5, 2024-04-14T15:14:19.552 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.90223121643066406, 2024-04-14T15:14:19.552 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 0.89999997615814209, 2024-04-14T15:14:19.552 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-04-14T15:14:19.552 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.552 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.552 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-14T15:14:19.552 INFO:teuthology.orchestra.run.smithi090.stdout: "osds": [ 2024-04-14T15:14:19.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.553 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 0, 2024-04-14T15:14:19.553 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "76772cfc-89d1-4013-b617-32284ed8dfd9", 2024-04-14T15:14:19.553 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-14T15:14:19.553 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-14T15:14:19.553 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-14T15:14:19.553 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-14T15:14:19.553 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-14T15:14:19.554 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-14T15:14:19.554 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 8, 2024-04-14T15:14:19.554 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 189, 2024-04-14T15:14:19.554 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-14T15:14:19.554 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-14T15:14:19.554 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-14T15:14:19.554 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:14:19.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.555 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:14:19.555 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6801", 2024-04-14T15:14:19.555 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3761806885 2024-04-14T15:14:19.555 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.555 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:14:19.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.555 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-14T15:14:19.555 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:14:19.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.556 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:14:19.556 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6802", 2024-04-14T15:14:19.556 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3761806885 2024-04-14T15:14:19.556 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.556 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:14:19.556 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.556 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-14T15:14:19.557 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:14:19.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.557 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:14:19.557 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6804", 2024-04-14T15:14:19.557 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3761806885 2024-04-14T15:14:19.557 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.557 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:14:19.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.558 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-14T15:14:19.558 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:14:19.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.558 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:14:19.558 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6803", 2024-04-14T15:14:19.558 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3761806885 2024-04-14T15:14:19.558 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.558 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:14:19.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.559 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6801/3761806885", 2024-04-14T15:14:19.559 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6802/3761806885", 2024-04-14T15:14:19.559 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6804/3761806885", 2024-04-14T15:14:19.559 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6803/3761806885", 2024-04-14T15:14:19.559 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-14T15:14:19.559 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-14T15:14:19.559 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-14T15:14:19.560 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:14:19.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.560 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.560 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 1, 2024-04-14T15:14:19.560 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "de4a7fa2-7d57-4ff3-a4fa-28c0cde8ef6c", 2024-04-14T15:14:19.560 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-14T15:14:19.560 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-14T15:14:19.560 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-14T15:14:19.561 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-14T15:14:19.561 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-14T15:14:19.561 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-14T15:14:19.561 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2024-04-14T15:14:19.561 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 189, 2024-04-14T15:14:19.561 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-14T15:14:19.561 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-14T15:14:19.561 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-14T15:14:19.561 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:14:19.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.562 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:14:19.562 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6805", 2024-04-14T15:14:19.562 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 4170361699 2024-04-14T15:14:19.562 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.562 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:14:19.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.562 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-14T15:14:19.562 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:14:19.562 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.562 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:14:19.562 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6806", 2024-04-14T15:14:19.562 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 4170361699 2024-04-14T15:14:19.563 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.563 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:14:19.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.563 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-14T15:14:19.563 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:14:19.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.563 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:14:19.564 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6808", 2024-04-14T15:14:19.564 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 4170361699 2024-04-14T15:14:19.564 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.564 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:14:19.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.564 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-14T15:14:19.564 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:14:19.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.565 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:14:19.565 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6807", 2024-04-14T15:14:19.565 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 4170361699 2024-04-14T15:14:19.565 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.565 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:14:19.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.566 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6805/4170361699", 2024-04-14T15:14:19.566 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6806/4170361699", 2024-04-14T15:14:19.566 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6808/4170361699", 2024-04-14T15:14:19.566 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6807/4170361699", 2024-04-14T15:14:19.566 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-14T15:14:19.566 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-14T15:14:19.567 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-14T15:14:19.567 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:14:19.567 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.567 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.567 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 2, 2024-04-14T15:14:19.567 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "9c39b5ff-6175-46ba-9d71-370581c6ff3d", 2024-04-14T15:14:19.568 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-14T15:14:19.568 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-14T15:14:19.568 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-14T15:14:19.568 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-14T15:14:19.568 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-14T15:14:19.568 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-14T15:14:19.568 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 18, 2024-04-14T15:14:19.569 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 190, 2024-04-14T15:14:19.569 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-14T15:14:19.569 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-14T15:14:19.569 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-14T15:14:19.569 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:14:19.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.569 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:14:19.570 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6809", 2024-04-14T15:14:19.570 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 1686184251 2024-04-14T15:14:19.570 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.570 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:14:19.570 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.570 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-14T15:14:19.570 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:14:19.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.571 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:14:19.571 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6810", 2024-04-14T15:14:19.571 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 1686184251 2024-04-14T15:14:19.571 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.571 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:14:19.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.572 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-14T15:14:19.572 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:14:19.572 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.572 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:14:19.572 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6812", 2024-04-14T15:14:19.573 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 1686184251 2024-04-14T15:14:19.573 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.573 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:14:19.573 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.573 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-14T15:14:19.573 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:14:19.573 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.574 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:14:19.574 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6811", 2024-04-14T15:14:19.574 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 1686184251 2024-04-14T15:14:19.574 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.574 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:14:19.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.574 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6809/1686184251", 2024-04-14T15:14:19.574 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6810/1686184251", 2024-04-14T15:14:19.574 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6812/1686184251", 2024-04-14T15:14:19.574 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6811/1686184251", 2024-04-14T15:14:19.574 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-14T15:14:19.575 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-14T15:14:19.575 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-14T15:14:19.575 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:14:19.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.575 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 3, 2024-04-14T15:14:19.575 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "efd2ca45-5b80-4013-a571-499e2cb5b2e8", 2024-04-14T15:14:19.575 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-14T15:14:19.575 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-14T15:14:19.575 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-14T15:14:19.576 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 0.2178497314453125, 2024-04-14T15:14:19.576 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-14T15:14:19.576 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-14T15:14:19.576 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 25, 2024-04-14T15:14:19.576 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 189, 2024-04-14T15:14:19.576 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-14T15:14:19.576 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-14T15:14:19.576 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-14T15:14:19.576 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:14:19.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.576 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:14:19.576 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6813", 2024-04-14T15:14:19.576 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 2270166774 2024-04-14T15:14:19.577 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.577 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:14:19.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.577 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-14T15:14:19.577 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:14:19.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.577 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:14:19.577 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6814", 2024-04-14T15:14:19.577 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 2270166774 2024-04-14T15:14:19.577 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.577 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:14:19.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.577 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-14T15:14:19.577 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:14:19.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.578 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:14:19.578 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6816", 2024-04-14T15:14:19.578 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 2270166774 2024-04-14T15:14:19.578 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.578 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:14:19.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.578 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-14T15:14:19.578 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:14:19.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.578 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:14:19.579 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6815", 2024-04-14T15:14:19.579 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 2270166774 2024-04-14T15:14:19.579 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.579 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:14:19.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.579 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6813/2270166774", 2024-04-14T15:14:19.579 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6814/2270166774", 2024-04-14T15:14:19.579 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6816/2270166774", 2024-04-14T15:14:19.579 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6815/2270166774", 2024-04-14T15:14:19.579 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-14T15:14:19.579 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-14T15:14:19.579 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-14T15:14:19.581 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:14:19.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.582 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 4, 2024-04-14T15:14:19.582 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "e3d9013d-5eb6-433c-96b3-3fc38ba7ac8d", 2024-04-14T15:14:19.582 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-14T15:14:19.582 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-14T15:14:19.582 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-14T15:14:19.582 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-14T15:14:19.582 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-14T15:14:19.582 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-14T15:14:19.582 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 31, 2024-04-14T15:14:19.582 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 189, 2024-04-14T15:14:19.582 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-14T15:14:19.583 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-14T15:14:19.583 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-14T15:14:19.583 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:14:19.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.583 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:14:19.583 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6800", 2024-04-14T15:14:19.583 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3887549733 2024-04-14T15:14:19.583 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.583 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:14:19.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.583 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-14T15:14:19.583 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:14:19.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.584 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:14:19.584 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6801", 2024-04-14T15:14:19.584 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3887549733 2024-04-14T15:14:19.584 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.584 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:14:19.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.584 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-14T15:14:19.584 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:14:19.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.585 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:14:19.585 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6803", 2024-04-14T15:14:19.585 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3887549733 2024-04-14T15:14:19.585 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.586 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:14:19.586 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.586 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-14T15:14:19.586 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:14:19.586 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.586 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:14:19.586 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6802", 2024-04-14T15:14:19.586 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3887549733 2024-04-14T15:14:19.586 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.586 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:14:19.586 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.586 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.92:6800/3887549733", 2024-04-14T15:14:19.586 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.92:6801/3887549733", 2024-04-14T15:14:19.587 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.92:6803/3887549733", 2024-04-14T15:14:19.587 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.92:6802/3887549733", 2024-04-14T15:14:19.587 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-14T15:14:19.587 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-14T15:14:19.587 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-14T15:14:19.587 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:14:19.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.587 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 5, 2024-04-14T15:14:19.587 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "1ec41805-6f82-4139-81bc-75305ba9e958", 2024-04-14T15:14:19.587 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-14T15:14:19.587 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-14T15:14:19.587 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-14T15:14:19.587 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-14T15:14:19.588 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-14T15:14:19.588 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-14T15:14:19.588 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 41, 2024-04-14T15:14:19.588 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 189, 2024-04-14T15:14:19.588 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-14T15:14:19.588 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-14T15:14:19.588 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-14T15:14:19.588 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:14:19.588 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.588 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:14:19.588 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6804", 2024-04-14T15:14:19.588 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3244561440 2024-04-14T15:14:19.588 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.588 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:14:19.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.589 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-14T15:14:19.589 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:14:19.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.589 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:14:19.589 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6805", 2024-04-14T15:14:19.589 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3244561440 2024-04-14T15:14:19.589 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.589 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:14:19.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.589 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-14T15:14:19.589 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:14:19.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.589 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:14:19.590 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6807", 2024-04-14T15:14:19.590 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3244561440 2024-04-14T15:14:19.590 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.590 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:14:19.590 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.590 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-14T15:14:19.590 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:14:19.590 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.590 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:14:19.590 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6806", 2024-04-14T15:14:19.590 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3244561440 2024-04-14T15:14:19.590 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.590 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:14:19.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.591 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.92:6804/3244561440", 2024-04-14T15:14:19.591 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.92:6805/3244561440", 2024-04-14T15:14:19.591 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.92:6807/3244561440", 2024-04-14T15:14:19.591 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.92:6806/3244561440", 2024-04-14T15:14:19.591 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-14T15:14:19.591 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-14T15:14:19.591 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-14T15:14:19.591 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:14:19.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.591 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.591 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 6, 2024-04-14T15:14:19.592 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "ca977ab2-bf8d-4b3a-ae5e-0efa5bc7a3ad", 2024-04-14T15:14:19.592 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-14T15:14:19.592 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-14T15:14:19.592 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-14T15:14:19.592 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-14T15:14:19.592 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-14T15:14:19.592 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-14T15:14:19.592 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 49, 2024-04-14T15:14:19.592 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 189, 2024-04-14T15:14:19.592 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-14T15:14:19.592 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-14T15:14:19.592 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-14T15:14:19.593 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:14:19.593 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.593 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:14:19.593 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6808", 2024-04-14T15:14:19.593 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 636125963 2024-04-14T15:14:19.593 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.593 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:14:19.593 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.593 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-14T15:14:19.593 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:14:19.593 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.593 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:14:19.593 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6809", 2024-04-14T15:14:19.594 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 636125963 2024-04-14T15:14:19.594 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.594 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:14:19.594 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.594 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-14T15:14:19.594 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:14:19.594 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.594 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:14:19.594 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6811", 2024-04-14T15:14:19.594 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 636125963 2024-04-14T15:14:19.594 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.595 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:14:19.595 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.595 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-14T15:14:19.595 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:14:19.595 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.595 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:14:19.595 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6810", 2024-04-14T15:14:19.595 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 636125963 2024-04-14T15:14:19.595 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.595 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:14:19.595 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.596 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.92:6808/636125963", 2024-04-14T15:14:19.596 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.92:6809/636125963", 2024-04-14T15:14:19.596 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.92:6811/636125963", 2024-04-14T15:14:19.596 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.92:6810/636125963", 2024-04-14T15:14:19.596 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-14T15:14:19.596 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-14T15:14:19.596 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-14T15:14:19.596 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:14:19.596 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.596 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.596 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 7, 2024-04-14T15:14:19.596 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "e5354645-79ec-4d4c-b6e5-f9b0653915ee", 2024-04-14T15:14:19.596 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-14T15:14:19.597 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-14T15:14:19.597 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-14T15:14:19.597 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-14T15:14:19.597 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-14T15:14:19.597 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-14T15:14:19.597 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 58, 2024-04-14T15:14:19.597 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 189, 2024-04-14T15:14:19.597 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-14T15:14:19.597 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-14T15:14:19.597 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-14T15:14:19.597 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:14:19.597 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.597 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:14:19.597 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6812", 2024-04-14T15:14:19.598 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 1773706716 2024-04-14T15:14:19.598 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.598 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:14:19.598 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.598 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-14T15:14:19.598 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:14:19.598 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.598 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:14:19.598 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6813", 2024-04-14T15:14:19.598 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 1773706716 2024-04-14T15:14:19.598 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.598 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:14:19.598 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.599 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-14T15:14:19.599 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:14:19.599 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.599 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:14:19.599 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6815", 2024-04-14T15:14:19.599 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 1773706716 2024-04-14T15:14:19.599 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.599 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:14:19.599 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.599 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-14T15:14:19.599 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:14:19.599 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.600 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:14:19.600 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6814", 2024-04-14T15:14:19.600 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 1773706716 2024-04-14T15:14:19.601 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.601 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:14:19.601 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.601 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.92:6812/1773706716", 2024-04-14T15:14:19.601 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.92:6813/1773706716", 2024-04-14T15:14:19.601 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.92:6815/1773706716", 2024-04-14T15:14:19.601 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.92:6814/1773706716", 2024-04-14T15:14:19.601 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-14T15:14:19.601 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-14T15:14:19.601 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-14T15:14:19.601 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:14:19.601 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.601 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-14T15:14:19.601 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_xinfo": [ 2024-04-14T15:14:19.602 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.602 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 0, 2024-04-14T15:14:19.602 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-14T15:14:19.602 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-14T15:14:19.602 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-14T15:14:19.602 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540701547738038271, 2024-04-14T15:14:19.602 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-14T15:14:19.602 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-14T15:13:12.372581+0000", 2024-04-14T15:14:19.602 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-14T15:14:19.602 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.602 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.602 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 1, 2024-04-14T15:14:19.602 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-14T15:14:19.603 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-14T15:14:19.603 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-14T15:14:19.603 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540701547738038271, 2024-04-14T15:14:19.603 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-14T15:14:19.603 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-14T15:14:03.393903+0000", 2024-04-14T15:14:19.603 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-14T15:14:19.603 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.603 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.603 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 2, 2024-04-14T15:14:19.603 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-14T15:14:19.603 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-14T15:14:19.603 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-14T15:14:19.603 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540701547738038271, 2024-04-14T15:14:19.604 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-14T15:14:19.604 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-14T15:13:13.389324+0000", 2024-04-14T15:14:19.604 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-14T15:14:19.604 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.604 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.604 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 3, 2024-04-14T15:14:19.604 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-14T15:14:19.604 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-14T15:14:19.604 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-14T15:14:19.604 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540701547738038271, 2024-04-14T15:14:19.605 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-14T15:14:19.605 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-14T15:13:20.899272+0000", 2024-04-14T15:14:19.605 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-14T15:14:19.605 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.605 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.605 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 4, 2024-04-14T15:14:19.605 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-14T15:14:19.605 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-14T15:14:19.605 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-14T15:14:19.605 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540701547738038271, 2024-04-14T15:14:19.605 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-14T15:14:19.605 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-14T15:13:09.817510+0000", 2024-04-14T15:14:19.605 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-14T15:14:19.606 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.606 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.606 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 5, 2024-04-14T15:14:19.606 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-14T15:14:19.606 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-14T15:14:19.606 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-14T15:14:19.606 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540701547738038271, 2024-04-14T15:14:19.606 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-14T15:14:19.606 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-14T15:13:49.210268+0000", 2024-04-14T15:14:19.606 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-14T15:14:19.606 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.606 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.606 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 6, 2024-04-14T15:14:19.606 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-14T15:14:19.607 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-14T15:14:19.607 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-14T15:14:19.607 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540701547738038271, 2024-04-14T15:14:19.607 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-14T15:14:19.607 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-14T15:14:07.913295+0000", 2024-04-14T15:14:19.607 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-14T15:14:19.607 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.607 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.607 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 7, 2024-04-14T15:14:19.607 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-14T15:14:19.607 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-14T15:14:19.607 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-14T15:14:19.607 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540701547738038271, 2024-04-14T15:14:19.608 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-14T15:14:19.608 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-14T15:13:05.127218+0000", 2024-04-14T15:14:19.608 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-14T15:14:19.608 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.608 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-14T15:14:19.608 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_upmap": [], 2024-04-14T15:14:19.608 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_upmap_items": [ 2024-04-14T15:14:19.608 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.608 INFO:teuthology.orchestra.run.smithi090.stdout: "pgid": "2.1c", 2024-04-14T15:14:19.608 INFO:teuthology.orchestra.run.smithi090.stdout: "mappings": [ 2024-04-14T15:14:19.608 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.608 INFO:teuthology.orchestra.run.smithi090.stdout: "from": 7, 2024-04-14T15:14:19.608 INFO:teuthology.orchestra.run.smithi090.stdout: "to": 4 2024-04-14T15:14:19.608 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.608 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:14:19.609 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.609 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.609 INFO:teuthology.orchestra.run.smithi090.stdout: "pgid": "4.2", 2024-04-14T15:14:19.609 INFO:teuthology.orchestra.run.smithi090.stdout: "mappings": [ 2024-04-14T15:14:19.609 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.609 INFO:teuthology.orchestra.run.smithi090.stdout: "from": 1, 2024-04-14T15:14:19.609 INFO:teuthology.orchestra.run.smithi090.stdout: "to": 2 2024-04-14T15:14:19.609 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.609 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:14:19.609 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.609 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.609 INFO:teuthology.orchestra.run.smithi090.stdout: "pgid": "4.1f", 2024-04-14T15:14:19.609 INFO:teuthology.orchestra.run.smithi090.stdout: "mappings": [ 2024-04-14T15:14:19.609 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.610 INFO:teuthology.orchestra.run.smithi090.stdout: "from": 1, 2024-04-14T15:14:19.610 INFO:teuthology.orchestra.run.smithi090.stdout: "to": 2 2024-04-14T15:14:19.610 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.610 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:14:19.610 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.610 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-14T15:14:19.610 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_upmap_primaries": [], 2024-04-14T15:14:19.610 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_temp": [], 2024-04-14T15:14:19.610 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_temp": [], 2024-04-14T15:14:19.610 INFO:teuthology.orchestra.run.smithi090.stdout: "blocklist": { 2024-04-14T15:14:19.610 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:0/508768479": "2024-04-15T15:10:22.918674+0000", 2024-04-14T15:14:19.610 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:0/1823072952": "2024-04-15T15:10:22.918674+0000", 2024-04-14T15:14:19.610 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:6800/491071118": "2024-04-15T15:10:22.918674+0000", 2024-04-14T15:14:19.610 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:0/1337064035": "2024-04-15T15:10:22.918674+0000", 2024-04-14T15:14:19.611 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:0/769462306": "2024-04-15T15:10:22.918674+0000", 2024-04-14T15:14:19.611 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:0/1139868610": "2024-04-15T15:05:05.882361+0000", 2024-04-14T15:14:19.611 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:0/93763408": "2024-04-15T15:10:22.918674+0000", 2024-04-14T15:14:19.611 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:0/977818204": "2024-04-15T15:04:39.251711+0000", 2024-04-14T15:14:19.611 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:6800/941091171": "2024-04-15T15:05:05.882361+0000", 2024-04-14T15:14:19.611 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:0/3671453561": "2024-04-15T15:05:05.882361+0000", 2024-04-14T15:14:19.611 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:0/2611856200": "2024-04-15T15:05:05.882361+0000", 2024-04-14T15:14:19.611 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:6800/2306258962": "2024-04-15T15:04:39.251711+0000", 2024-04-14T15:14:19.611 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:0/4051692380": "2024-04-15T15:04:39.251711+0000", 2024-04-14T15:14:19.611 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:0/2681502561": "2024-04-15T15:04:39.251711+0000" 2024-04-14T15:14:19.611 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.611 INFO:teuthology.orchestra.run.smithi090.stdout: "range_blocklist": {}, 2024-04-14T15:14:19.611 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profiles": { 2024-04-14T15:14:19.611 INFO:teuthology.orchestra.run.smithi090.stdout: "default": { 2024-04-14T15:14:19.612 INFO:teuthology.orchestra.run.smithi090.stdout: "crush-failure-domain": "osd", 2024-04-14T15:14:19.612 INFO:teuthology.orchestra.run.smithi090.stdout: "k": "2", 2024-04-14T15:14:19.612 INFO:teuthology.orchestra.run.smithi090.stdout: "m": "1", 2024-04-14T15:14:19.612 INFO:teuthology.orchestra.run.smithi090.stdout: "plugin": "jerasure", 2024-04-14T15:14:19.612 INFO:teuthology.orchestra.run.smithi090.stdout: "technique": "reed_sol_van" 2024-04-14T15:14:19.612 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.612 INFO:teuthology.orchestra.run.smithi090.stdout: "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi090-93724-21": { 2024-04-14T15:14:19.612 INFO:teuthology.orchestra.run.smithi090.stdout: "crush-device-class": "", 2024-04-14T15:14:19.612 INFO:teuthology.orchestra.run.smithi090.stdout: "crush-failure-domain": "osd", 2024-04-14T15:14:19.612 INFO:teuthology.orchestra.run.smithi090.stdout: "crush-num-failure-domains": "0", 2024-04-14T15:14:19.612 INFO:teuthology.orchestra.run.smithi090.stdout: "crush-osds-per-failure-domain": "0", 2024-04-14T15:14:19.612 INFO:teuthology.orchestra.run.smithi090.stdout: "crush-root": "default", 2024-04-14T15:14:19.612 INFO:teuthology.orchestra.run.smithi090.stdout: "jerasure-per-chunk-alignment": "false", 2024-04-14T15:14:19.613 INFO:teuthology.orchestra.run.smithi090.stdout: "k": "2", 2024-04-14T15:14:19.613 INFO:teuthology.orchestra.run.smithi090.stdout: "m": "1", 2024-04-14T15:14:19.613 INFO:teuthology.orchestra.run.smithi090.stdout: "plugin": "jerasure", 2024-04-14T15:14:19.613 INFO:teuthology.orchestra.run.smithi090.stdout: "technique": "reed_sol_van", 2024-04-14T15:14:19.613 INFO:teuthology.orchestra.run.smithi090.stdout: "w": "8" 2024-04-14T15:14:19.613 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.613 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.613 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps_queue": [ 2024-04-14T15:14:19.613 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.613 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 17, 2024-04-14T15:14:19.613 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:14:19.613 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.613 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 2, 2024-04-14T15:14:19.613 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 5 2024-04-14T15:14:19.614 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.614 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:14:19.614 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.614 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.614 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 18, 2024-04-14T15:14:19.614 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:14:19.614 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.614 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 2, 2024-04-14T15:14:19.614 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 3 2024-04-14T15:14:19.614 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.614 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:14:19.614 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.614 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.615 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 39, 2024-04-14T15:14:19.615 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:14:19.615 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.615 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 1, 2024-04-14T15:14:19.615 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2024-04-14T15:14:19.615 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.615 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:14:19.615 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.615 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.615 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 72, 2024-04-14T15:14:19.615 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:14:19.615 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.615 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 1, 2024-04-14T15:14:19.615 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2024-04-14T15:14:19.616 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.616 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:14:19.616 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.616 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.616 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 99, 2024-04-14T15:14:19.616 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:14:19.616 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.616 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 4, 2024-04-14T15:14:19.616 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 2 2024-04-14T15:14:19.616 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.616 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.616 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 7, 2024-04-14T15:14:19.616 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 2 2024-04-14T15:14:19.616 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.616 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:14:19.617 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.617 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.617 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 109, 2024-04-14T15:14:19.617 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:14:19.617 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.617 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 1, 2024-04-14T15:14:19.617 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2024-04-14T15:14:19.617 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.617 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:14:19.617 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.617 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.617 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 110, 2024-04-14T15:14:19.617 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:14:19.617 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.617 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 3, 2024-04-14T15:14:19.618 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 4 2024-04-14T15:14:19.618 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.618 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:14:19.618 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.618 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.618 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 128, 2024-04-14T15:14:19.618 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:14:19.618 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.618 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 1, 2024-04-14T15:14:19.618 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 2 2024-04-14T15:14:19.618 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.618 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:14:19.618 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.619 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.619 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 142, 2024-04-14T15:14:19.619 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:14:19.619 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.619 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 3, 2024-04-14T15:14:19.619 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2024-04-14T15:14:19.619 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.619 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:14:19.619 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.619 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.619 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 149, 2024-04-14T15:14:19.619 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:14:19.619 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.620 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 1, 2024-04-14T15:14:19.620 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 4 2024-04-14T15:14:19.620 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.620 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:14:19.620 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.620 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.620 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 156, 2024-04-14T15:14:19.620 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:14:19.620 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.620 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 2, 2024-04-14T15:14:19.620 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2024-04-14T15:14:19.620 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.620 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:14:19.620 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.621 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.621 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 157, 2024-04-14T15:14:19.621 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:14:19.621 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.621 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 2, 2024-04-14T15:14:19.621 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 2 2024-04-14T15:14:19.621 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.621 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:14:19.621 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.621 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.621 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 167, 2024-04-14T15:14:19.621 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:14:19.621 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.621 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 2, 2024-04-14T15:14:19.621 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 3 2024-04-14T15:14:19.622 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.622 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:14:19.622 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.622 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.622 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 172, 2024-04-14T15:14:19.622 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:14:19.622 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.622 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 3, 2024-04-14T15:14:19.622 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 4 2024-04-14T15:14:19.622 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.622 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:14:19.622 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.622 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.623 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 175, 2024-04-14T15:14:19.623 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:14:19.623 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.623 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 2, 2024-04-14T15:14:19.623 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 4 2024-04-14T15:14:19.623 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.623 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:14:19.623 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.623 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.623 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 176, 2024-04-14T15:14:19.623 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:14:19.623 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.623 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 2, 2024-04-14T15:14:19.623 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 2 2024-04-14T15:14:19.623 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.624 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:14:19.624 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.624 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.624 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 187, 2024-04-14T15:14:19.624 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:14:19.624 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.624 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 2, 2024-04-14T15:14:19.624 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2024-04-14T15:14:19.624 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.624 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:14:19.624 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.624 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.624 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 192, 2024-04-14T15:14:19.624 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:14:19.625 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.625 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 1, 2024-04-14T15:14:19.625 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 4 2024-04-14T15:14:19.625 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.625 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:14:19.625 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:14:19.625 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.625 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 205, 2024-04-14T15:14:19.625 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:14:19.625 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:14:19.625 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 2, 2024-04-14T15:14:19.625 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 2 2024-04-14T15:14:19.625 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.625 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:14:19.625 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.626 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-14T15:14:19.626 INFO:teuthology.orchestra.run.smithi090.stdout: "new_removed_snaps": [], 2024-04-14T15:14:19.626 INFO:teuthology.orchestra.run.smithi090.stdout: "new_purged_snaps": [], 2024-04-14T15:14:19.626 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_node_flags": {}, 2024-04-14T15:14:19.626 INFO:teuthology.orchestra.run.smithi090.stdout: "device_class_flags": {}, 2024-04-14T15:14:19.626 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_mode": { 2024-04-14T15:14:19.626 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_mode_enabled": false, 2024-04-14T15:14:19.626 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_bucket_count": 0, 2024-04-14T15:14:19.626 INFO:teuthology.orchestra.run.smithi090.stdout: "degraded_stretch_mode": 0, 2024-04-14T15:14:19.626 INFO:teuthology.orchestra.run.smithi090.stdout: "recovering_stretch_mode": 0, 2024-04-14T15:14:19.626 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_mode_bucket": 0 2024-04-14T15:14:19.626 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:14:19.626 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-14T15:14:19.627 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:14:19.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:19.651 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:19.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:14:19.753 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:19.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:19.778 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:19.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:19.879 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:14:19.904 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:19.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:19.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:14:20.006 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:14:20.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:20.031 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:20.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:14:20.132 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:14:20.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:20.160 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:20.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:14:20.174 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-04-14T15:14:20.174 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:14:20.174 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:19 smithi092 ceph-mon[33966]: pgmap v233: 303 pgs: 16 creating+peering, 8 active+clean+snaptrim, 20 active+clean+snaptrim_wait, 17 unknown, 242 active+clean; 518 KiB data, 535 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 282 KiB/s wr, 51 op/s; 341 KiB/s, 1 objects/s recovering 2024-04-14T15:14:20.174 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:19 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3536155582' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-14T15:14:20.174 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:19 smithi092 ceph-mon[33966]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:14:20.174 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:19 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-13"}]': finished 2024-04-14T15:14:20.174 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:19 smithi092 ceph-mon[33966]: from='client.48364 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi090-93363-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:20.175 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:19 smithi092 ceph-mon[33966]: from='client.48370 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi090-93342-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:14:20.175 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:19 smithi092 ceph-mon[33966]: osdmap e192: 8 total, 8 up, 8 in 2024-04-14T15:14:20.175 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:19 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1487099185' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi090-93342-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi090-93342-28"}]: dispatch 2024-04-14T15:14:20.175 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:19 smithi092 ceph-mon[33966]: from='client.48370 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi090-93342-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi090-93342-28"}]: dispatch 2024-04-14T15:14:20.175 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:19 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/414434713' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi090-93601-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:20.175 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:19 smithi092 ceph-mon[33966]: from='client.48376 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi090-93601-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:20.175 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:19 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:20.262 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:20.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:20.287 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:20.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:20.388 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:14:20.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:20.413 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:20.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:14:20.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:19 smithi090 ceph-mon[30399]: pgmap v233: 303 pgs: 16 creating+peering, 8 active+clean+snaptrim, 20 active+clean+snaptrim_wait, 17 unknown, 242 active+clean; 518 KiB data, 535 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 282 KiB/s wr, 51 op/s; 341 KiB/s, 1 objects/s recovering 2024-04-14T15:14:20.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:19 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3536155582' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-14T15:14:20.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:19 smithi090 ceph-mon[30399]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:14:20.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:19 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-13"}]': finished 2024-04-14T15:14:20.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:19 smithi090 ceph-mon[30399]: from='client.48364 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi090-93363-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:20.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:19 smithi090 ceph-mon[30399]: from='client.48370 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi090-93342-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:14:20.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:19 smithi090 ceph-mon[30399]: osdmap e192: 8 total, 8 up, 8 in 2024-04-14T15:14:20.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:19 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1487099185' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi090-93342-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi090-93342-28"}]: dispatch 2024-04-14T15:14:20.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:19 smithi090 ceph-mon[30399]: from='client.48370 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi090-93342-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi090-93342-28"}]: dispatch 2024-04-14T15:14:20.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:19 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/414434713' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi090-93601-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:20.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:19 smithi090 ceph-mon[30399]: from='client.48376 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi090-93601-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:20.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:19 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:20.445 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:19 smithi090 ceph-mon[41670]: pgmap v233: 303 pgs: 16 creating+peering, 8 active+clean+snaptrim, 20 active+clean+snaptrim_wait, 17 unknown, 242 active+clean; 518 KiB data, 535 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 282 KiB/s wr, 51 op/s; 341 KiB/s, 1 objects/s recovering 2024-04-14T15:14:20.445 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:19 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3536155582' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-14T15:14:20.445 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:19 smithi090 ceph-mon[41670]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:14:20.445 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:19 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-13"}]': finished 2024-04-14T15:14:20.445 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:19 smithi090 ceph-mon[41670]: from='client.48364 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi090-93363-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:20.445 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:19 smithi090 ceph-mon[41670]: from='client.48370 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi090-93342-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:14:20.445 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:19 smithi090 ceph-mon[41670]: osdmap e192: 8 total, 8 up, 8 in 2024-04-14T15:14:20.446 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:19 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1487099185' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi090-93342-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi090-93342-28"}]: dispatch 2024-04-14T15:14:20.446 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:19 smithi090 ceph-mon[41670]: from='client.48370 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi090-93342-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi090-93342-28"}]: dispatch 2024-04-14T15:14:20.446 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:19 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/414434713' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi090-93601-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:20.446 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:19 smithi090 ceph-mon[41670]: from='client.48376 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi090-93601-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:20.446 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:19 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:20.514 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:14:20.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:20.540 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:20.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:14:20.641 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:14:20.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:20.665 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:20.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:14:20.766 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:14:20.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:20.793 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:20.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:14:20.894 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:14:20.911 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-14T15:14:20.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:20.919 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:20.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:14:21.021 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:14:21.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:21.046 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:21.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:14:21.147 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:14:21.159 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:14:21.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:21.171 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:21.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:14:21.272 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:21.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:21.298 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:21.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:21.399 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:14:21.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:21.423 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:21.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:14:21.524 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:14:21.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:21.549 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:21.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:14:21.650 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:14:21.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:21.676 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:21.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:14:21.777 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:14:21.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:21.801 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:21.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:14:21.844 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:21 smithi090 ceph-mon[30399]: from='client.48376 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi090-93601-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:21.844 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:21 smithi090 ceph-mon[30399]: osdmap e193: 8 total, 8 up, 8 in 2024-04-14T15:14:21.844 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:21 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:21.844 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:21 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-04-14T15:14:21.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:21 smithi092 ceph-mon[33966]: from='client.48376 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi090-93601-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:21.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:21 smithi092 ceph-mon[33966]: osdmap e193: 8 total, 8 up, 8 in 2024-04-14T15:14:21.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:21 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:21.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:21 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-04-14T15:14:21.903 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:14:21.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:21.927 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:21.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:14:22.028 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:14:22.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:22.052 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:22.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:14:22.153 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:14:22.169 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:21 smithi090 ceph-mon[41670]: from='client.48376 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi090-93601-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:22.169 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:21 smithi090 ceph-mon[41670]: osdmap e193: 8 total, 8 up, 8 in 2024-04-14T15:14:22.170 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:21 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:22.170 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:21 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-04-14T15:14:22.177 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:22.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:22.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:14:22.197 INFO:tasks.thrashosds.thrasher:j is {'epoch': 191, 'fsid': '0e288098-fa70-11ee-bc8e-c7b262605968', 'created': '2024-04-14T15:04:05.808313+0000', 'modified': '2024-04-14T15:14:18.825285+0000', 'last_up_change': '2024-04-14T15:09:29.769352+0000', 'last_in_change': '2024-04-14T15:09:11.803164+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 41, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 210, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-14T15:07:23.959628+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '186', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '184', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.180000066757202, 'score_stable': 2.180000066757202, 'optimal_score': 1, 'raw_score_acting': 2.180000066757202, 'raw_score_stable': 2.180000066757202, 'primary_affinity_weighted': 0.9022312164306641, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-04-14T15:09:41.608420+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9022312164306641, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 3, 'pool_name': 'datapool', 'create_time': '2024-04-14T15:09:41.744391+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 69, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 0.9022312164306641, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-04-14T15:09:43.088157+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9022312164306641, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-04-14T15:09:45.042722+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9022312164306641, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-04-14T15:09:47.111336+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.9022312164306641, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 11, 'pool_name': 'ceph_test_rados_api_asio', 'create_time': '2024-04-14T15:12:54.225671+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.9022312164306641, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 13, 'pool_name': 'LibRadosList_smithi090-93444-1', 'create_time': '2024-04-14T15:12:54.244611+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 25, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 24, 'last_pg_merge_meta': {'source_pgid': '13.19', 'ready_epoch': 188, 'last_epoch_started': 187, 'last_epoch_clean': 187, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '190', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '190', 'last_force_op_resend_preluminous': '190', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9199999570846558, 'score_stable': 1.9199999570846558, 'optimal_score': 1, 'raw_score_acting': 1.9199999570846558, 'raw_score_stable': 1.9199999570846558, 'primary_affinity_weighted': 0.9022312164306641, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 110, 'pool_name': 'test-rados-api-smithi090-93652-6', 'create_time': '2024-04-14T15:13:07.960794+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '191', 'last_force_op_resend': '191', 'last_force_op_resend_prenautilus': '191', 'last_force_op_resend_preluminous': '191', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 6, 'snap_epoch': 190, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [172], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9022312164306641, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 172, 'pool_name': 'test-rados-api-smithi090-93652-13', 'create_time': '2024-04-14T15:13:34.849790+0000', 'flags': 8201, 'flags_names': 'hashpspool,incomplete_clones,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '191', 'last_force_op_resend': '191', 'last_force_op_resend_prenautilus': '191', 'last_force_op_resend_preluminous': '191', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 6, 'snap_epoch': 190, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': 110, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'writeback', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9022312164306641, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 205, 'pool_name': 'LibRadosSnapshotsSelfManagedECPP_smithi090-93724-21', 'create_time': '2024-04-14T15:14:11.845038+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 3, 'size': 3, 'min_size': 2, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '191', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 5, 'snap_epoch': 191, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'testprofile-LibRadosSnapshotsSelfManagedECPP_smithi090-93724-21', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}, {'pool': 210, 'pool_name': 'StatRemovePP_smithi090-93363-24', 'create_time': '2024-04-14T15:14:17.898201+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '191', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9022312164306641, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}], 'osds': [{'osd': 0, 'uuid': '76772cfc-89d1-4013-b617-32284ed8dfd9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 189, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6801', 'nonce': 3761806885}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6802', 'nonce': 3761806885}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6804', 'nonce': 3761806885}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6803', 'nonce': 3761806885}]}, 'public_addr': '172.21.15.90:6801/3761806885', 'cluster_addr': '172.21.15.90:6802/3761806885', 'heartbeat_back_addr': '172.21.15.90:6804/3761806885', 'heartbeat_front_addr': '172.21.15.90:6803/3761806885', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'de4a7fa2-7d57-4ff3-a4fa-28c0cde8ef6c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 189, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6805', 'nonce': 4170361699}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6806', 'nonce': 4170361699}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6808', 'nonce': 4170361699}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6807', 'nonce': 4170361699}]}, 'public_addr': '172.21.15.90:6805/4170361699', 'cluster_addr': '172.21.15.90:6806/4170361699', 'heartbeat_back_addr': '172.21.15.90:6808/4170361699', 'heartbeat_front_addr': '172.21.15.90:6807/4170361699', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '9c39b5ff-6175-46ba-9d71-370581c6ff3d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 190, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6809', 'nonce': 1686184251}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6810', 'nonce': 1686184251}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6812', 'nonce': 1686184251}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6811', 'nonce': 1686184251}]}, 'public_addr': '172.21.15.90:6809/1686184251', 'cluster_addr': '172.21.15.90:6810/1686184251', 'heartbeat_back_addr': '172.21.15.90:6812/1686184251', 'heartbeat_front_addr': '172.21.15.90:6811/1686184251', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'efd2ca45-5b80-4013-a571-499e2cb5b2e8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.2178497314453125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 189, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6813', 'nonce': 2270166774}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6814', 'nonce': 2270166774}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6816', 'nonce': 2270166774}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6815', 'nonce': 2270166774}]}, 'public_addr': '172.21.15.90:6813/2270166774', 'cluster_addr': '172.21.15.90:6814/2270166774', 'heartbeat_back_addr': '172.21.15.90:6816/2270166774', 'heartbeat_front_addr': '172.21.15.90:6815/2270166774', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'e3d9013d-5eb6-433c-96b3-3fc38ba7ac8d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 189, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6800', 'nonce': 3887549733}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6801', 'nonce': 3887549733}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6803', 'nonce': 3887549733}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6802', 'nonce': 3887549733}]}, 'public_addr': '172.21.15.92:6800/3887549733', 'cluster_addr': '172.21.15.92:6801/3887549733', 'heartbeat_back_addr': '172.21.15.92:6803/3887549733', 'heartbeat_front_addr': '172.21.15.92:6802/3887549733', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '1ec41805-6f82-4139-81bc-75305ba9e958', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 41, 'up_thru': 189, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6804', 'nonce': 3244561440}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6805', 'nonce': 3244561440}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6807', 'nonce': 3244561440}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6806', 'nonce': 3244561440}]}, 'public_addr': '172.21.15.92:6804/3244561440', 'cluster_addr': '172.21.15.92:6805/3244561440', 'heartbeat_back_addr': '172.21.15.92:6807/3244561440', 'heartbeat_front_addr': '172.21.15.92:6806/3244561440', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'ca977ab2-bf8d-4b3a-ae5e-0efa5bc7a3ad', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 49, 'up_thru': 189, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6808', 'nonce': 636125963}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6809', 'nonce': 636125963}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6811', 'nonce': 636125963}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6810', 'nonce': 636125963}]}, 'public_addr': '172.21.15.92:6808/636125963', 'cluster_addr': '172.21.15.92:6809/636125963', 'heartbeat_back_addr': '172.21.15.92:6811/636125963', 'heartbeat_front_addr': '172.21.15.92:6810/636125963', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'e5354645-79ec-4d4c-b6e5-f9b0653915ee', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 58, 'up_thru': 189, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6812', 'nonce': 1773706716}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6813', 'nonce': 1773706716}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6815', 'nonce': 1773706716}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6814', 'nonce': 1773706716}]}, 'public_addr': '172.21.15.92:6812/1773706716', 'cluster_addr': '172.21.15.92:6813/1773706716', 'heartbeat_back_addr': '172.21.15.92:6815/1773706716', 'heartbeat_front_addr': '172.21.15.92:6814/1773706716', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-14T15:13:12.372581+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-14T15:14:03.393903+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-14T15:13:13.389324+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-14T15:13:20.899272+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-14T15:13:09.817510+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-14T15:13:49.210268+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-14T15:14:07.913295+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-14T15:13:05.127218+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '2.1c', 'mappings': [{'from': 7, 'to': 4}]}, {'pgid': '4.2', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1f', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.90:0/508768479': '2024-04-15T15:10:22.918674+0000', '172.21.15.90:0/1823072952': '2024-04-15T15:10:22.918674+0000', '172.21.15.90:6800/491071118': '2024-04-15T15:10:22.918674+0000', '172.21.15.90:0/1337064035': '2024-04-15T15:10:22.918674+0000', '172.21.15.90:0/769462306': '2024-04-15T15:10:22.918674+0000', '172.21.15.90:0/1139868610': '2024-04-15T15:05:05.882361+0000', '172.21.15.90:0/93763408': '2024-04-15T15:10:22.918674+0000', '172.21.15.90:0/977818204': '2024-04-15T15:04:39.251711+0000', '172.21.15.90:6800/941091171': '2024-04-15T15:05:05.882361+0000', '172.21.15.90:0/3671453561': '2024-04-15T15:05:05.882361+0000', '172.21.15.90:0/2611856200': '2024-04-15T15:05:05.882361+0000', '172.21.15.90:6800/2306258962': '2024-04-15T15:04:39.251711+0000', '172.21.15.90:0/4051692380': '2024-04-15T15:04:39.251711+0000', '172.21.15.90:0/2681502561': '2024-04-15T15:04:39.251711+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}, 'testprofile-LibRadosSnapshotsSelfManagedECPP_smithi090-93724-21': {'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-num-failure-domains': '0', 'crush-osds-per-failure-domain': '0', 'crush-root': 'default', 'jerasure-per-chunk-alignment': 'false', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van', 'w': '8'}}, 'removed_snaps_queue': [{'pool': 17, 'snaps': [{'begin': 2, 'length': 5}]}, {'pool': 18, 'snaps': [{'begin': 2, 'length': 3}]}, {'pool': 39, 'snaps': [{'begin': 1, 'length': 1}]}, {'pool': 72, 'snaps': [{'begin': 1, 'length': 1}]}, {'pool': 99, 'snaps': [{'begin': 4, 'length': 2}, {'begin': 7, 'length': 2}]}, {'pool': 109, 'snaps': [{'begin': 1, 'length': 1}]}, {'pool': 110, 'snaps': [{'begin': 3, 'length': 4}]}, {'pool': 128, 'snaps': [{'begin': 1, 'length': 2}]}, {'pool': 142, 'snaps': [{'begin': 3, 'length': 1}]}, {'pool': 149, 'snaps': [{'begin': 1, 'length': 4}]}, {'pool': 156, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 157, 'snaps': [{'begin': 2, 'length': 2}]}, {'pool': 167, 'snaps': [{'begin': 2, 'length': 3}]}, {'pool': 172, 'snaps': [{'begin': 3, 'length': 4}]}, {'pool': 175, 'snaps': [{'begin': 2, 'length': 4}]}, {'pool': 176, 'snaps': [{'begin': 2, 'length': 2}]}, {'pool': 187, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 192, 'snaps': [{'begin': 1, 'length': 4}]}, {'pool': 205, 'snaps': [{'begin': 2, 'length': 2}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-14T15:14:22.197 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 2.1c 2024-04-14T15:14:22.197 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd rm-pg-upmap-items 2.1c 2024-04-14T15:14:22.279 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:14:22.303 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:22.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:22.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:14:22.385 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:14:22.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.393+0000 7f7e83c6c700 1 -- 172.21.15.90:0/2320322432 >> v1:172.21.15.90:6789/0 conn(0x7f7e7c1023f0 legacy=0x7f7e7c1027d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:22.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.394+0000 7f7e83c6c700 1 -- 172.21.15.90:0/2320322432 shutdown_connections 2024-04-14T15:14:22.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.394+0000 7f7e83c6c700 1 -- 172.21.15.90:0/2320322432 >> 172.21.15.90:0/2320322432 conn(0x7f7e7c076aa0 msgr2=0x7f7e7c076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:14:22.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.394+0000 7f7e83c6c700 1 -- 172.21.15.90:0/2320322432 shutdown_connections 2024-04-14T15:14:22.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.394+0000 7f7e83c6c700 1 -- 172.21.15.90:0/2320322432 wait complete. 2024-04-14T15:14:22.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.394+0000 7f7e83c6c700 1 librados: starting msgr at 2024-04-14T15:14:22.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.394+0000 7f7e83c6c700 1 librados: starting objecter 2024-04-14T15:14:22.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.394+0000 7f7e83c6c700 1 Processor -- start 2024-04-14T15:14:22.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.395+0000 7f7e83c6c700 1 -- start start 2024-04-14T15:14:22.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.395+0000 7f7e83c6c700 1 librados: setting wanted keys 2024-04-14T15:14:22.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.395+0000 7f7e83c6c700 1 librados: calling monclient init 2024-04-14T15:14:22.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.395+0000 7f7e83c6c700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7e7c1a5910 con 0x7f7e7c10a6d0 2024-04-14T15:14:22.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.395+0000 7f7e83c6c700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7e7c1a6b30 con 0x7f7e7c1023f0 2024-04-14T15:14:22.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.395+0000 7f7e83c6c700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7e7c1a7d70 con 0x7f7e7c10e270 2024-04-14T15:14:22.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.395+0000 7f7e81207700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f7e7c10a6d0 0x7f7e7c1a06d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:40178/0 (socket says 172.21.15.90:40178) 2024-04-14T15:14:22.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.395+0000 7f7e81207700 1 -- 172.21.15.90:0/1108240145 learned_addr learned my addr 172.21.15.90:0/1108240145 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:14:22.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.395+0000 7f7e72ffd700 10 client.?.objecter ms_handle_connect 0x7f7e7c10e270 2024-04-14T15:14:22.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.395+0000 7f7e72ffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:14:22.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.395+0000 7f7e72ffd700 10 client.?.objecter ms_handle_connect 0x7f7e7c1023f0 2024-04-14T15:14:22.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.395+0000 7f7e72ffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:14:22.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.395+0000 7f7e72ffd700 10 client.?.objecter ms_handle_connect 0x7f7e7c10a6d0 2024-04-14T15:14:22.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.395+0000 7f7e72ffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:14:22.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.396+0000 7f7e72ffd700 1 -- 172.21.15.90:0/1108240145 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1061490703 0 0) 0x7f7e7c1a6b30 con 0x7f7e7c1023f0 2024-04-14T15:14:22.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.396+0000 7f7e72ffd700 1 -- 172.21.15.90:0/1108240145 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e680035b0 con 0x7f7e7c1023f0 2024-04-14T15:14:22.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.396+0000 7f7e72ffd700 1 -- 172.21.15.90:0/1108240145 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2009461211 0 0) 0x7f7e7c1a5910 con 0x7f7e7c10a6d0 2024-04-14T15:14:22.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.396+0000 7f7e72ffd700 1 -- 172.21.15.90:0/1108240145 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e7c1a6b30 con 0x7f7e7c10a6d0 2024-04-14T15:14:22.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.396+0000 7f7e72ffd700 1 -- 172.21.15.90:0/1108240145 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 446528819 0 0) 0x7f7e7c1a7d70 con 0x7f7e7c10e270 2024-04-14T15:14:22.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.396+0000 7f7e72ffd700 1 -- 172.21.15.90:0/1108240145 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e7c1a5910 con 0x7f7e7c10e270 2024-04-14T15:14:22.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.396+0000 7f7e72ffd700 1 -- 172.21.15.90:0/1108240145 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3528165098 0 0) 0x7f7e680035b0 con 0x7f7e7c1023f0 2024-04-14T15:14:22.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.396+0000 7f7e72ffd700 1 -- 172.21.15.90:0/1108240145 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7e7c1a7d70 con 0x7f7e7c1023f0 2024-04-14T15:14:22.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.396+0000 7f7e72ffd700 1 -- 172.21.15.90:0/1108240145 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f7e780015d0 con 0x7f7e7c1023f0 2024-04-14T15:14:22.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.396+0000 7f7e72ffd700 1 -- 172.21.15.90:0/1108240145 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3429054625 0 0) 0x7f7e7c1a5910 con 0x7f7e7c10e270 2024-04-14T15:14:22.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.396+0000 7f7e72ffd700 1 -- 172.21.15.90:0/1108240145 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7e680035b0 con 0x7f7e7c10e270 2024-04-14T15:14:22.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.396+0000 7f7e72ffd700 1 -- 172.21.15.90:0/1108240145 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f7e74001bb0 con 0x7f7e7c10e270 2024-04-14T15:14:22.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.396+0000 7f7e72ffd700 1 -- 172.21.15.90:0/1108240145 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3938895321 0 0) 0x7f7e7c1a6b30 con 0x7f7e7c10a6d0 2024-04-14T15:14:22.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.396+0000 7f7e72ffd700 1 -- 172.21.15.90:0/1108240145 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7e7c1a5910 con 0x7f7e7c10a6d0 2024-04-14T15:14:22.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.396+0000 7f7e72ffd700 1 -- 172.21.15.90:0/1108240145 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f7e6c001870 con 0x7f7e7c10a6d0 2024-04-14T15:14:22.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.396+0000 7f7e72ffd700 1 -- 172.21.15.90:0/1108240145 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2451932163 0 0) 0x7f7e7c1a7d70 con 0x7f7e7c1023f0 2024-04-14T15:14:22.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.396+0000 7f7e72ffd700 1 -- 172.21.15.90:0/1108240145 >> v1:172.21.15.90:6790/0 conn(0x7f7e7c10e270 legacy=0x7f7e7c1a4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:22.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.396+0000 7f7e72ffd700 1 -- 172.21.15.90:0/1108240145 >> v1:172.21.15.90:6789/0 conn(0x7f7e7c10a6d0 legacy=0x7f7e7c1a06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:22.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.396+0000 7f7e72ffd700 1 -- 172.21.15.90:0/1108240145 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e7c1a5b20 con 0x7f7e7c1023f0 2024-04-14T15:14:22.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.396+0000 7f7e83c6c700 1 -- 172.21.15.90:0/1108240145 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7e7c1a5e10 con 0x7f7e7c1023f0 2024-04-14T15:14:22.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.397+0000 7f7e83c6c700 10 client.48400.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:14:22.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.397+0000 7f7e83c6c700 1 -- 172.21.15.90:0/1108240145 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f7e7c1a62d0 con 0x7f7e7c1023f0 2024-04-14T15:14:22.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.397+0000 7f7e83c6c700 1 librados: init done 2024-04-14T15:14:22.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.397+0000 7f7e72ffd700 1 -- 172.21.15.90:0/1108240145 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f7e78003680 con 0x7f7e7c1023f0 2024-04-14T15:14:22.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.397+0000 7f7e72ffd700 1 -- 172.21.15.90:0/1108240145 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f7e78004cc0 con 0x7f7e7c1023f0 2024-04-14T15:14:22.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.398+0000 7f7e83c6c700 1 -- 172.21.15.90:0/1108240145 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f7e60005100 con 0x7f7e7c1023f0 2024-04-14T15:14:22.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.398+0000 7f7e72ffd700 1 -- 172.21.15.90:0/1108240145 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f7e7801b6b0 con 0x7f7e7c1023f0 2024-04-14T15:14:22.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.399+0000 7f7e72ffd700 1 -- 172.21.15.90:0/1108240145 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(194..194 src has 51..194) ==== 10530+0+0 (unknown 2001918057 0 0) 0x7f7e7808a090 con 0x7f7e7c1023f0 2024-04-14T15:14:22.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.399+0000 7f7e72ffd700 10 client.48400.objecter ms_dispatch 0x7f7e7c001550 osd_map(194..194 src has 51..194) 2024-04-14T15:14:22.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.399+0000 7f7e72ffd700 3 client.48400.objecter handle_osd_map got epochs [194,194] > 0 2024-04-14T15:14:22.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.399+0000 7f7e72ffd700 3 client.48400.objecter handle_osd_map decoding full epoch 194 2024-04-14T15:14:22.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.400+0000 7f7e72ffd700 20 client.48400.objecter dump_active .. 0 homeless 2024-04-14T15:14:22.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.400+0000 7f7e72ffd700 10 client.48400.objecter ms_handle_connect 0x7f7e68073720 2024-04-14T15:14:22.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.400+0000 7f7e72ffd700 1 -- 172.21.15.90:0/1108240145 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v194) ==== 54+0+16 (unknown 3088419739 0 2420000192) 0x7f7e7808c4c0 con 0x7f7e7c1023f0 2024-04-14T15:14:22.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.400+0000 7f7e83c6c700 10 client.48400.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:14:22.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.400+0000 7f7e83c6c700 15 client.48400.objecter _session_command_op_assign -1 1 2024-04-14T15:14:22.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.400+0000 7f7e83c6c700 20 client.48400.objecter _get_session s=0x7f7e600053e0 osd=0 3 2024-04-14T15:14:22.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.400+0000 7f7e83c6c700 20 client.48400.objecter put_session s=0x7f7e600053e0 osd=0 3 2024-04-14T15:14:22.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.400+0000 7f7e83c6c700 20 client.48400.objecter _get_session s=0x7f7e600053e0 osd=0 3 2024-04-14T15:14:22.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.400+0000 7f7e83c6c700 15 client.48400.objecter _session_command_op_remove -1 1 2024-04-14T15:14:22.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.400+0000 7f7e83c6c700 20 client.48400.objecter get_session s=0x7f7e600053e0 osd=0 3 2024-04-14T15:14:22.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.400+0000 7f7e83c6c700 15 client.48400.objecter _session_command_op_assign 0 1 2024-04-14T15:14:22.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.400+0000 7f7e83c6c700 20 client.48400.objecter put_session s=0x7f7e600053e0 osd=0 4 2024-04-14T15:14:22.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.400+0000 7f7e83c6c700 10 client.48400.objecter _send_command 1 2024-04-14T15:14:22.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.400+0000 7f7e83c6c700 1 -- 172.21.15.90:0/1108240145 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7e60004b50 con 0x7f7e60004730 2024-04-14T15:14:22.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.400+0000 7f7e72ffd700 10 client.48400.objecter ms_handle_connect 0x7f7e60004730 2024-04-14T15:14:22.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.401+0000 7f7e72ffd700 1 -- 172.21.15.90:0/1108240145 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f7e60004b50 con 0x7f7e60004730 2024-04-14T15:14:22.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.401+0000 7f7e72ffd700 10 client.48400.objecter ms_dispatch 0x7f7e7c001550 command_reply(tid 1: 0 ) 2024-04-14T15:14:22.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.401+0000 7f7e72ffd700 10 client.48400.objecter _finish_command 1 = system:0 2024-04-14T15:14:22.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.401+0000 7f7e72ffd700 20 client.48400.objecter put_session s=0x7f7e600053e0 osd=0 4 2024-04-14T15:14:22.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.401+0000 7f7e72ffd700 15 client.48400.objecter _session_command_op_remove 0 1 2024-04-14T15:14:22.405 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:14:22.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.427+0000 7f7e83c6c700 10 client.48400.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:14:22.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.427+0000 7f7e83c6c700 15 client.48400.objecter _session_command_op_assign -1 2 2024-04-14T15:14:22.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.427+0000 7f7e83c6c700 20 client.48400.objecter _get_session s=0x7f7e600053e0 osd=0 3 2024-04-14T15:14:22.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.427+0000 7f7e83c6c700 20 client.48400.objecter put_session s=0x7f7e600053e0 osd=0 3 2024-04-14T15:14:22.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.427+0000 7f7e83c6c700 20 client.48400.objecter _get_session s=0x7f7e600053e0 osd=0 3 2024-04-14T15:14:22.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.427+0000 7f7e83c6c700 15 client.48400.objecter _session_command_op_remove -1 2 2024-04-14T15:14:22.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.427+0000 7f7e83c6c700 20 client.48400.objecter get_session s=0x7f7e600053e0 osd=0 3 2024-04-14T15:14:22.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.427+0000 7f7e83c6c700 15 client.48400.objecter _session_command_op_assign 0 2 2024-04-14T15:14:22.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.427+0000 7f7e83c6c700 20 client.48400.objecter put_session s=0x7f7e600053e0 osd=0 4 2024-04-14T15:14:22.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.427+0000 7f7e83c6c700 10 client.48400.objecter _send_command 2 2024-04-14T15:14:22.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.427+0000 7f7e83c6c700 1 -- 172.21.15.90:0/1108240145 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f7e60007b00 con 0x7f7e60004730 2024-04-14T15:14:22.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.428+0000 7f7e72ffd700 1 -- 172.21.15.90:0/1108240145 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f7e60007b00 con 0x7f7e60004730 2024-04-14T15:14:22.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.428+0000 7f7e72ffd700 10 client.48400.objecter ms_dispatch 0x7f7e7c001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:14:22.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.428+0000 7f7e72ffd700 10 client.48400.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:14:22.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.428+0000 7f7e72ffd700 20 client.48400.objecter put_session s=0x7f7e600053e0 osd=0 4 2024-04-14T15:14:22.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.428+0000 7f7e72ffd700 15 client.48400.objecter _session_command_op_remove 0 2 2024-04-14T15:14:22.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:22.430 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:14:22.433 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:22.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:14:22.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.429+0000 7f7e83c6c700 10 client.48400.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:14:22.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.429+0000 7f7e83c6c700 15 client.48400.objecter _session_command_op_assign -1 3 2024-04-14T15:14:22.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.429+0000 7f7e83c6c700 20 client.48400.objecter _get_session s=0x7f7e60004180 osd=1 3 2024-04-14T15:14:22.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.429+0000 7f7e83c6c700 20 client.48400.objecter put_session s=0x7f7e60004180 osd=1 3 2024-04-14T15:14:22.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.429+0000 7f7e83c6c700 20 client.48400.objecter _get_session s=0x7f7e60004180 osd=1 3 2024-04-14T15:14:22.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.429+0000 7f7e83c6c700 15 client.48400.objecter _session_command_op_remove -1 3 2024-04-14T15:14:22.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.429+0000 7f7e83c6c700 20 client.48400.objecter get_session s=0x7f7e60004180 osd=1 3 2024-04-14T15:14:22.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.429+0000 7f7e83c6c700 15 client.48400.objecter _session_command_op_assign 1 3 2024-04-14T15:14:22.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.429+0000 7f7e83c6c700 20 client.48400.objecter put_session s=0x7f7e60004180 osd=1 4 2024-04-14T15:14:22.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.429+0000 7f7e83c6c700 10 client.48400.objecter _send_command 3 2024-04-14T15:14:22.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.429+0000 7f7e83c6c700 1 -- 172.21.15.90:0/1108240145 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f7e60002200 con 0x7f7e600042f0 2024-04-14T15:14:22.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.429+0000 7f7e72ffd700 10 client.48400.objecter ms_handle_connect 0x7f7e600042f0 2024-04-14T15:14:22.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.429+0000 7f7e72ffd700 1 -- 172.21.15.90:0/1108240145 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f7e60002200 con 0x7f7e600042f0 2024-04-14T15:14:22.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.429+0000 7f7e72ffd700 10 client.48400.objecter ms_dispatch 0x7f7e7c001550 command_reply(tid 3: 0 ) 2024-04-14T15:14:22.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.429+0000 7f7e72ffd700 10 client.48400.objecter _finish_command 3 = system:0 2024-04-14T15:14:22.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.430+0000 7f7e72ffd700 20 client.48400.objecter put_session s=0x7f7e60004180 osd=1 4 2024-04-14T15:14:22.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.430+0000 7f7e72ffd700 15 client.48400.objecter _session_command_op_remove 1 3 2024-04-14T15:14:22.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.446+0000 7f7e83c6c700 10 client.48400.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:14:22.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.446+0000 7f7e83c6c700 15 client.48400.objecter _session_command_op_assign -1 4 2024-04-14T15:14:22.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.446+0000 7f7e83c6c700 20 client.48400.objecter _get_session s=0x7f7e60004180 osd=1 3 2024-04-14T15:14:22.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.446+0000 7f7e83c6c700 20 client.48400.objecter put_session s=0x7f7e60004180 osd=1 3 2024-04-14T15:14:22.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.446+0000 7f7e83c6c700 20 client.48400.objecter _get_session s=0x7f7e60004180 osd=1 3 2024-04-14T15:14:22.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.446+0000 7f7e83c6c700 15 client.48400.objecter _session_command_op_remove -1 4 2024-04-14T15:14:22.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.446+0000 7f7e83c6c700 20 client.48400.objecter get_session s=0x7f7e60004180 osd=1 3 2024-04-14T15:14:22.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.446+0000 7f7e83c6c700 15 client.48400.objecter _session_command_op_assign 1 4 2024-04-14T15:14:22.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.446+0000 7f7e83c6c700 20 client.48400.objecter put_session s=0x7f7e60004180 osd=1 4 2024-04-14T15:14:22.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.446+0000 7f7e83c6c700 10 client.48400.objecter _send_command 4 2024-04-14T15:14:22.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.446+0000 7f7e83c6c700 1 -- 172.21.15.90:0/1108240145 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f7e60001600 con 0x7f7e600042f0 2024-04-14T15:14:22.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.448+0000 7f7e72ffd700 1 -- 172.21.15.90:0/1108240145 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f7e60001600 con 0x7f7e600042f0 2024-04-14T15:14:22.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.448+0000 7f7e72ffd700 10 client.48400.objecter ms_dispatch 0x7f7e7c001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:14:22.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.448+0000 7f7e72ffd700 10 client.48400.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:14:22.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.448+0000 7f7e72ffd700 20 client.48400.objecter put_session s=0x7f7e60004180 osd=1 4 2024-04-14T15:14:22.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.448+0000 7f7e72ffd700 15 client.48400.objecter _session_command_op_remove 1 4 2024-04-14T15:14:22.450 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:14:22.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.448+0000 7f7e83c6c700 10 client.48400.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:14:22.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.448+0000 7f7e83c6c700 15 client.48400.objecter _session_command_op_assign -1 5 2024-04-14T15:14:22.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.448+0000 7f7e83c6c700 20 client.48400.objecter _get_session s=0x7f7e60008480 osd=2 3 2024-04-14T15:14:22.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.448+0000 7f7e83c6c700 20 client.48400.objecter put_session s=0x7f7e60008480 osd=2 3 2024-04-14T15:14:22.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.448+0000 7f7e83c6c700 20 client.48400.objecter _get_session s=0x7f7e60008480 osd=2 3 2024-04-14T15:14:22.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.448+0000 7f7e83c6c700 15 client.48400.objecter _session_command_op_remove -1 5 2024-04-14T15:14:22.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.448+0000 7f7e83c6c700 20 client.48400.objecter get_session s=0x7f7e60008480 osd=2 3 2024-04-14T15:14:22.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.448+0000 7f7e83c6c700 15 client.48400.objecter _session_command_op_assign 2 5 2024-04-14T15:14:22.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.448+0000 7f7e83c6c700 20 client.48400.objecter put_session s=0x7f7e60008480 osd=2 4 2024-04-14T15:14:22.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.448+0000 7f7e83c6c700 10 client.48400.objecter _send_command 5 2024-04-14T15:14:22.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.448+0000 7f7e83c6c700 1 -- 172.21.15.90:0/1108240145 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f7e600111f0 con 0x7f7e6000d6e0 2024-04-14T15:14:22.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.448+0000 7f7e72ffd700 10 client.48400.objecter ms_handle_connect 0x7f7e6000d6e0 2024-04-14T15:14:22.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.449+0000 7f7e72ffd700 1 -- 172.21.15.90:0/1108240145 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f7e600111f0 con 0x7f7e6000d6e0 2024-04-14T15:14:22.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.449+0000 7f7e72ffd700 10 client.48400.objecter ms_dispatch 0x7f7e7c001550 command_reply(tid 5: 0 ) 2024-04-14T15:14:22.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.449+0000 7f7e72ffd700 10 client.48400.objecter _finish_command 5 = system:0 2024-04-14T15:14:22.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.449+0000 7f7e72ffd700 20 client.48400.objecter put_session s=0x7f7e60008480 osd=2 4 2024-04-14T15:14:22.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.449+0000 7f7e72ffd700 15 client.48400.objecter _session_command_op_remove 2 5 2024-04-14T15:14:22.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.466+0000 7f7e83c6c700 10 client.48400.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:14:22.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.466+0000 7f7e83c6c700 15 client.48400.objecter _session_command_op_assign -1 6 2024-04-14T15:14:22.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.466+0000 7f7e83c6c700 20 client.48400.objecter _get_session s=0x7f7e60008480 osd=2 3 2024-04-14T15:14:22.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.466+0000 7f7e83c6c700 20 client.48400.objecter put_session s=0x7f7e60008480 osd=2 3 2024-04-14T15:14:22.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.466+0000 7f7e83c6c700 20 client.48400.objecter _get_session s=0x7f7e60008480 osd=2 3 2024-04-14T15:14:22.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.466+0000 7f7e83c6c700 15 client.48400.objecter _session_command_op_remove -1 6 2024-04-14T15:14:22.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.466+0000 7f7e83c6c700 20 client.48400.objecter get_session s=0x7f7e60008480 osd=2 3 2024-04-14T15:14:22.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.466+0000 7f7e83c6c700 15 client.48400.objecter _session_command_op_assign 2 6 2024-04-14T15:14:22.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.466+0000 7f7e83c6c700 20 client.48400.objecter put_session s=0x7f7e60008480 osd=2 4 2024-04-14T15:14:22.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.466+0000 7f7e83c6c700 10 client.48400.objecter _send_command 6 2024-04-14T15:14:22.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.466+0000 7f7e83c6c700 1 -- 172.21.15.90:0/1108240145 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f7e60007fc0 con 0x7f7e6000d6e0 2024-04-14T15:14:22.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.467+0000 7f7e72ffd700 1 -- 172.21.15.90:0/1108240145 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f7e60007fc0 con 0x7f7e6000d6e0 2024-04-14T15:14:22.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.467+0000 7f7e72ffd700 10 client.48400.objecter ms_dispatch 0x7f7e7c001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:14:22.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.467+0000 7f7e72ffd700 10 client.48400.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:14:22.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.467+0000 7f7e72ffd700 20 client.48400.objecter put_session s=0x7f7e60008480 osd=2 4 2024-04-14T15:14:22.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.467+0000 7f7e72ffd700 15 client.48400.objecter _session_command_op_remove 2 6 2024-04-14T15:14:22.469 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:14:22.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.467+0000 7f7e83c6c700 10 client.48400.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:14:22.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.467+0000 7f7e83c6c700 15 client.48400.objecter _session_command_op_assign -1 7 2024-04-14T15:14:22.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.467+0000 7f7e83c6c700 20 client.48400.objecter _get_session s=0x7f7e60008250 osd=3 3 2024-04-14T15:14:22.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.467+0000 7f7e83c6c700 20 client.48400.objecter put_session s=0x7f7e60008250 osd=3 3 2024-04-14T15:14:22.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.467+0000 7f7e83c6c700 20 client.48400.objecter _get_session s=0x7f7e60008250 osd=3 3 2024-04-14T15:14:22.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.467+0000 7f7e83c6c700 15 client.48400.objecter _session_command_op_remove -1 7 2024-04-14T15:14:22.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.467+0000 7f7e83c6c700 20 client.48400.objecter get_session s=0x7f7e60008250 osd=3 3 2024-04-14T15:14:22.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.467+0000 7f7e83c6c700 15 client.48400.objecter _session_command_op_assign 3 7 2024-04-14T15:14:22.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.467+0000 7f7e83c6c700 20 client.48400.objecter put_session s=0x7f7e60008250 osd=3 4 2024-04-14T15:14:22.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.467+0000 7f7e83c6c700 10 client.48400.objecter _send_command 7 2024-04-14T15:14:22.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.467+0000 7f7e83c6c700 1 -- 172.21.15.90:0/1108240145 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f7e60016030 con 0x7f7e60012520 2024-04-14T15:14:22.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.468+0000 7f7e72ffd700 10 client.48400.objecter ms_handle_connect 0x7f7e60012520 2024-04-14T15:14:22.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.468+0000 7f7e72ffd700 1 -- 172.21.15.90:0/1108240145 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f7e60016030 con 0x7f7e60012520 2024-04-14T15:14:22.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.468+0000 7f7e72ffd700 10 client.48400.objecter ms_dispatch 0x7f7e7c001550 command_reply(tid 7: 0 ) 2024-04-14T15:14:22.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.468+0000 7f7e72ffd700 10 client.48400.objecter _finish_command 7 = system:0 2024-04-14T15:14:22.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.468+0000 7f7e72ffd700 20 client.48400.objecter put_session s=0x7f7e60008250 osd=3 4 2024-04-14T15:14:22.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.468+0000 7f7e72ffd700 15 client.48400.objecter _session_command_op_remove 3 7 2024-04-14T15:14:22.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.485+0000 7f7e83c6c700 10 client.48400.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:14:22.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.485+0000 7f7e83c6c700 15 client.48400.objecter _session_command_op_assign -1 8 2024-04-14T15:14:22.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.485+0000 7f7e83c6c700 20 client.48400.objecter _get_session s=0x7f7e60008250 osd=3 3 2024-04-14T15:14:22.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.485+0000 7f7e83c6c700 20 client.48400.objecter put_session s=0x7f7e60008250 osd=3 3 2024-04-14T15:14:22.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.485+0000 7f7e83c6c700 20 client.48400.objecter _get_session s=0x7f7e60008250 osd=3 3 2024-04-14T15:14:22.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.485+0000 7f7e83c6c700 15 client.48400.objecter _session_command_op_remove -1 8 2024-04-14T15:14:22.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.485+0000 7f7e83c6c700 20 client.48400.objecter get_session s=0x7f7e60008250 osd=3 3 2024-04-14T15:14:22.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.485+0000 7f7e83c6c700 15 client.48400.objecter _session_command_op_assign 3 8 2024-04-14T15:14:22.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.485+0000 7f7e83c6c700 20 client.48400.objecter put_session s=0x7f7e60008250 osd=3 4 2024-04-14T15:14:22.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.485+0000 7f7e83c6c700 10 client.48400.objecter _send_command 8 2024-04-14T15:14:22.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.485+0000 7f7e83c6c700 1 -- 172.21.15.90:0/1108240145 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f7e6000ca60 con 0x7f7e60012520 2024-04-14T15:14:22.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.487+0000 7f7e72ffd700 1 -- 172.21.15.90:0/1108240145 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f7e6000ca60 con 0x7f7e60012520 2024-04-14T15:14:22.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.487+0000 7f7e72ffd700 10 client.48400.objecter ms_dispatch 0x7f7e7c001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:14:22.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.487+0000 7f7e72ffd700 10 client.48400.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:14:22.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.487+0000 7f7e72ffd700 20 client.48400.objecter put_session s=0x7f7e60008250 osd=3 4 2024-04-14T15:14:22.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.487+0000 7f7e72ffd700 15 client.48400.objecter _session_command_op_remove 3 8 2024-04-14T15:14:22.488 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:14:22.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.487+0000 7f7e83c6c700 10 client.48400.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:14:22.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.487+0000 7f7e83c6c700 15 client.48400.objecter _session_command_op_assign -1 9 2024-04-14T15:14:22.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.487+0000 7f7e83c6c700 20 client.48400.objecter _get_session s=0x7f7e6000c810 osd=4 3 2024-04-14T15:14:22.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.487+0000 7f7e83c6c700 20 client.48400.objecter put_session s=0x7f7e6000c810 osd=4 3 2024-04-14T15:14:22.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.487+0000 7f7e83c6c700 20 client.48400.objecter _get_session s=0x7f7e6000c810 osd=4 3 2024-04-14T15:14:22.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.487+0000 7f7e83c6c700 15 client.48400.objecter _session_command_op_remove -1 9 2024-04-14T15:14:22.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.487+0000 7f7e83c6c700 20 client.48400.objecter get_session s=0x7f7e6000c810 osd=4 3 2024-04-14T15:14:22.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.487+0000 7f7e83c6c700 15 client.48400.objecter _session_command_op_assign 4 9 2024-04-14T15:14:22.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.487+0000 7f7e83c6c700 20 client.48400.objecter put_session s=0x7f7e6000c810 osd=4 4 2024-04-14T15:14:22.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.487+0000 7f7e83c6c700 10 client.48400.objecter _send_command 9 2024-04-14T15:14:22.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.487+0000 7f7e83c6c700 1 -- 172.21.15.90:0/1108240145 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f7e6001aa90 con 0x7f7e60012080 2024-04-14T15:14:22.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.488+0000 7f7e72ffd700 10 client.48400.objecter ms_handle_connect 0x7f7e60012080 2024-04-14T15:14:22.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.489+0000 7f7e72ffd700 1 -- 172.21.15.90:0/1108240145 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f7e6001aa90 con 0x7f7e60012080 2024-04-14T15:14:22.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.489+0000 7f7e72ffd700 10 client.48400.objecter ms_dispatch 0x7f7e7c001550 command_reply(tid 9: 0 ) 2024-04-14T15:14:22.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.489+0000 7f7e72ffd700 10 client.48400.objecter _finish_command 9 = system:0 2024-04-14T15:14:22.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.489+0000 7f7e72ffd700 20 client.48400.objecter put_session s=0x7f7e6000c810 osd=4 4 2024-04-14T15:14:22.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.489+0000 7f7e72ffd700 15 client.48400.objecter _session_command_op_remove 4 9 2024-04-14T15:14:22.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.505+0000 7f7e83c6c700 10 client.48400.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:14:22.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.505+0000 7f7e83c6c700 15 client.48400.objecter _session_command_op_assign -1 10 2024-04-14T15:14:22.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.505+0000 7f7e83c6c700 20 client.48400.objecter _get_session s=0x7f7e6000c810 osd=4 3 2024-04-14T15:14:22.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.505+0000 7f7e83c6c700 20 client.48400.objecter put_session s=0x7f7e6000c810 osd=4 3 2024-04-14T15:14:22.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.505+0000 7f7e83c6c700 20 client.48400.objecter _get_session s=0x7f7e6000c810 osd=4 3 2024-04-14T15:14:22.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.505+0000 7f7e83c6c700 15 client.48400.objecter _session_command_op_remove -1 10 2024-04-14T15:14:22.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.505+0000 7f7e83c6c700 20 client.48400.objecter get_session s=0x7f7e6000c810 osd=4 3 2024-04-14T15:14:22.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.505+0000 7f7e83c6c700 15 client.48400.objecter _session_command_op_assign 4 10 2024-04-14T15:14:22.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.505+0000 7f7e83c6c700 20 client.48400.objecter put_session s=0x7f7e6000c810 osd=4 4 2024-04-14T15:14:22.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.505+0000 7f7e83c6c700 10 client.48400.objecter _send_command 10 2024-04-14T15:14:22.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.505+0000 7f7e83c6c700 1 -- 172.21.15.90:0/1108240145 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f7e6001aa90 con 0x7f7e60012080 2024-04-14T15:14:22.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.507+0000 7f7e72ffd700 1 -- 172.21.15.90:0/1108240145 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f7e6001aa90 con 0x7f7e60012080 2024-04-14T15:14:22.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.507+0000 7f7e72ffd700 10 client.48400.objecter ms_dispatch 0x7f7e7c001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:14:22.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.507+0000 7f7e72ffd700 10 client.48400.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:14:22.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.507+0000 7f7e72ffd700 20 client.48400.objecter put_session s=0x7f7e6000c810 osd=4 4 2024-04-14T15:14:22.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.507+0000 7f7e72ffd700 15 client.48400.objecter _session_command_op_remove 4 10 2024-04-14T15:14:22.508 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:14:22.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.507+0000 7f7e83c6c700 10 client.48400.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:14:22.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.507+0000 7f7e83c6c700 15 client.48400.objecter _session_command_op_assign -1 11 2024-04-14T15:14:22.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.507+0000 7f7e83c6c700 20 client.48400.objecter _get_session s=0x7f7e6001aa90 osd=5 3 2024-04-14T15:14:22.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.507+0000 7f7e83c6c700 20 client.48400.objecter put_session s=0x7f7e6001aa90 osd=5 3 2024-04-14T15:14:22.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.507+0000 7f7e83c6c700 20 client.48400.objecter _get_session s=0x7f7e6001aa90 osd=5 3 2024-04-14T15:14:22.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.507+0000 7f7e83c6c700 15 client.48400.objecter _session_command_op_remove -1 11 2024-04-14T15:14:22.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.507+0000 7f7e83c6c700 20 client.48400.objecter get_session s=0x7f7e6001aa90 osd=5 3 2024-04-14T15:14:22.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.507+0000 7f7e83c6c700 15 client.48400.objecter _session_command_op_assign 5 11 2024-04-14T15:14:22.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.507+0000 7f7e83c6c700 20 client.48400.objecter put_session s=0x7f7e6001aa90 osd=5 4 2024-04-14T15:14:22.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.507+0000 7f7e83c6c700 10 client.48400.objecter _send_command 11 2024-04-14T15:14:22.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.507+0000 7f7e83c6c700 1 -- 172.21.15.90:0/1108240145 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f7e6001ed30 con 0x7f7e6001b110 2024-04-14T15:14:22.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.508+0000 7f7e72ffd700 10 client.48400.objecter ms_handle_connect 0x7f7e6001b110 2024-04-14T15:14:22.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.508+0000 7f7e72ffd700 1 -- 172.21.15.90:0/1108240145 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f7e6001ed30 con 0x7f7e6001b110 2024-04-14T15:14:22.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.508+0000 7f7e72ffd700 10 client.48400.objecter ms_dispatch 0x7f7e7c001550 command_reply(tid 11: 0 ) 2024-04-14T15:14:22.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.508+0000 7f7e72ffd700 10 client.48400.objecter _finish_command 11 = system:0 2024-04-14T15:14:22.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.508+0000 7f7e72ffd700 20 client.48400.objecter put_session s=0x7f7e6001aa90 osd=5 4 2024-04-14T15:14:22.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.508+0000 7f7e72ffd700 15 client.48400.objecter _session_command_op_remove 5 11 2024-04-14T15:14:22.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.526+0000 7f7e83c6c700 10 client.48400.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:14:22.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.526+0000 7f7e83c6c700 15 client.48400.objecter _session_command_op_assign -1 12 2024-04-14T15:14:22.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.526+0000 7f7e83c6c700 20 client.48400.objecter _get_session s=0x7f7e6001aa90 osd=5 3 2024-04-14T15:14:22.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.526+0000 7f7e83c6c700 20 client.48400.objecter put_session s=0x7f7e6001aa90 osd=5 3 2024-04-14T15:14:22.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.526+0000 7f7e83c6c700 20 client.48400.objecter _get_session s=0x7f7e6001aa90 osd=5 3 2024-04-14T15:14:22.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.526+0000 7f7e83c6c700 15 client.48400.objecter _session_command_op_remove -1 12 2024-04-14T15:14:22.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.526+0000 7f7e83c6c700 20 client.48400.objecter get_session s=0x7f7e6001aa90 osd=5 3 2024-04-14T15:14:22.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.526+0000 7f7e83c6c700 15 client.48400.objecter _session_command_op_assign 5 12 2024-04-14T15:14:22.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.526+0000 7f7e83c6c700 20 client.48400.objecter put_session s=0x7f7e6001aa90 osd=5 4 2024-04-14T15:14:22.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.526+0000 7f7e83c6c700 10 client.48400.objecter _send_command 12 2024-04-14T15:14:22.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.526+0000 7f7e83c6c700 1 -- 172.21.15.90:0/1108240145 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f7e600206e0 con 0x7f7e6001b110 2024-04-14T15:14:22.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.527+0000 7f7e72ffd700 1 -- 172.21.15.90:0/1108240145 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f7e600206e0 con 0x7f7e6001b110 2024-04-14T15:14:22.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.527+0000 7f7e72ffd700 10 client.48400.objecter ms_dispatch 0x7f7e7c001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:14:22.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.527+0000 7f7e72ffd700 10 client.48400.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:14:22.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.527+0000 7f7e72ffd700 20 client.48400.objecter put_session s=0x7f7e6001aa90 osd=5 4 2024-04-14T15:14:22.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.527+0000 7f7e72ffd700 15 client.48400.objecter _session_command_op_remove 5 12 2024-04-14T15:14:22.528 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:14:22.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.528+0000 7f7e83c6c700 10 client.48400.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:14:22.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.528+0000 7f7e83c6c700 15 client.48400.objecter _session_command_op_assign -1 13 2024-04-14T15:14:22.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.528+0000 7f7e83c6c700 20 client.48400.objecter _get_session s=0x7f7e600206e0 osd=6 3 2024-04-14T15:14:22.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.528+0000 7f7e83c6c700 20 client.48400.objecter put_session s=0x7f7e600206e0 osd=6 3 2024-04-14T15:14:22.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.528+0000 7f7e83c6c700 20 client.48400.objecter _get_session s=0x7f7e600206e0 osd=6 3 2024-04-14T15:14:22.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.528+0000 7f7e83c6c700 15 client.48400.objecter _session_command_op_remove -1 13 2024-04-14T15:14:22.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.528+0000 7f7e83c6c700 20 client.48400.objecter get_session s=0x7f7e600206e0 osd=6 3 2024-04-14T15:14:22.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.528+0000 7f7e83c6c700 15 client.48400.objecter _session_command_op_assign 6 13 2024-04-14T15:14:22.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.528+0000 7f7e83c6c700 20 client.48400.objecter put_session s=0x7f7e600206e0 osd=6 4 2024-04-14T15:14:22.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.528+0000 7f7e83c6c700 10 client.48400.objecter _send_command 13 2024-04-14T15:14:22.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.528+0000 7f7e83c6c700 1 -- 172.21.15.90:0/1108240145 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f7e600212b0 con 0x7f7e60020e00 2024-04-14T15:14:22.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.528+0000 7f7e72ffd700 10 client.48400.objecter ms_handle_connect 0x7f7e60020e00 2024-04-14T15:14:22.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.529+0000 7f7e72ffd700 1 -- 172.21.15.90:0/1108240145 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f7e600212b0 con 0x7f7e60020e00 2024-04-14T15:14:22.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.529+0000 7f7e72ffd700 10 client.48400.objecter ms_dispatch 0x7f7e7c001550 command_reply(tid 13: 0 ) 2024-04-14T15:14:22.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.529+0000 7f7e72ffd700 10 client.48400.objecter _finish_command 13 = system:0 2024-04-14T15:14:22.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.529+0000 7f7e72ffd700 20 client.48400.objecter put_session s=0x7f7e600206e0 osd=6 4 2024-04-14T15:14:22.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.529+0000 7f7e72ffd700 15 client.48400.objecter _session_command_op_remove 6 13 2024-04-14T15:14:22.534 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:14:22.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.545+0000 7f7e83c6c700 10 client.48400.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:14:22.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.545+0000 7f7e83c6c700 15 client.48400.objecter _session_command_op_assign -1 14 2024-04-14T15:14:22.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.545+0000 7f7e83c6c700 20 client.48400.objecter _get_session s=0x7f7e600206e0 osd=6 3 2024-04-14T15:14:22.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.545+0000 7f7e83c6c700 20 client.48400.objecter put_session s=0x7f7e600206e0 osd=6 3 2024-04-14T15:14:22.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.545+0000 7f7e83c6c700 20 client.48400.objecter _get_session s=0x7f7e600206e0 osd=6 3 2024-04-14T15:14:22.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.545+0000 7f7e83c6c700 15 client.48400.objecter _session_command_op_remove -1 14 2024-04-14T15:14:22.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.545+0000 7f7e83c6c700 20 client.48400.objecter get_session s=0x7f7e600206e0 osd=6 3 2024-04-14T15:14:22.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.545+0000 7f7e83c6c700 15 client.48400.objecter _session_command_op_assign 6 14 2024-04-14T15:14:22.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.545+0000 7f7e83c6c700 20 client.48400.objecter put_session s=0x7f7e600206e0 osd=6 4 2024-04-14T15:14:22.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.545+0000 7f7e83c6c700 10 client.48400.objecter _send_command 14 2024-04-14T15:14:22.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.545+0000 7f7e83c6c700 1 -- 172.21.15.90:0/1108240145 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f7e600212b0 con 0x7f7e60020e00 2024-04-14T15:14:22.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.547+0000 7f7e72ffd700 1 -- 172.21.15.90:0/1108240145 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f7e600212b0 con 0x7f7e60020e00 2024-04-14T15:14:22.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.547+0000 7f7e72ffd700 10 client.48400.objecter ms_dispatch 0x7f7e7c001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:14:22.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.547+0000 7f7e72ffd700 10 client.48400.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:14:22.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.547+0000 7f7e72ffd700 20 client.48400.objecter put_session s=0x7f7e600206e0 osd=6 4 2024-04-14T15:14:22.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.547+0000 7f7e72ffd700 15 client.48400.objecter _session_command_op_remove 6 14 2024-04-14T15:14:22.548 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:14:22.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.547+0000 7f7e83c6c700 10 client.48400.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:14:22.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.547+0000 7f7e83c6c700 15 client.48400.objecter _session_command_op_assign -1 15 2024-04-14T15:14:22.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.547+0000 7f7e83c6c700 20 client.48400.objecter _get_session s=0x7f7e600212b0 osd=7 3 2024-04-14T15:14:22.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.547+0000 7f7e83c6c700 20 client.48400.objecter put_session s=0x7f7e600212b0 osd=7 3 2024-04-14T15:14:22.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.547+0000 7f7e83c6c700 20 client.48400.objecter _get_session s=0x7f7e600212b0 osd=7 3 2024-04-14T15:14:22.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.547+0000 7f7e83c6c700 15 client.48400.objecter _session_command_op_remove -1 15 2024-04-14T15:14:22.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.547+0000 7f7e83c6c700 20 client.48400.objecter get_session s=0x7f7e600212b0 osd=7 3 2024-04-14T15:14:22.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.547+0000 7f7e83c6c700 15 client.48400.objecter _session_command_op_assign 7 15 2024-04-14T15:14:22.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.547+0000 7f7e83c6c700 20 client.48400.objecter put_session s=0x7f7e600212b0 osd=7 4 2024-04-14T15:14:22.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.547+0000 7f7e83c6c700 10 client.48400.objecter _send_command 15 2024-04-14T15:14:22.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.547+0000 7f7e83c6c700 1 -- 172.21.15.90:0/1108240145 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f7e60027900 con 0x7f7e60023df0 2024-04-14T15:14:22.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.548+0000 7f7e72ffd700 10 client.48400.objecter ms_handle_connect 0x7f7e60023df0 2024-04-14T15:14:22.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.549+0000 7f7e72ffd700 1 -- 172.21.15.90:0/1108240145 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f7e60027900 con 0x7f7e60023df0 2024-04-14T15:14:22.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.549+0000 7f7e72ffd700 10 client.48400.objecter ms_dispatch 0x7f7e7c001550 command_reply(tid 15: 0 ) 2024-04-14T15:14:22.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.549+0000 7f7e72ffd700 10 client.48400.objecter _finish_command 15 = system:0 2024-04-14T15:14:22.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.549+0000 7f7e72ffd700 20 client.48400.objecter put_session s=0x7f7e600212b0 osd=7 4 2024-04-14T15:14:22.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.549+0000 7f7e72ffd700 15 client.48400.objecter _session_command_op_remove 7 15 2024-04-14T15:14:22.558 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:22.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:22.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:14:22.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.565+0000 7f7e83c6c700 10 client.48400.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:14:22.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.565+0000 7f7e83c6c700 15 client.48400.objecter _session_command_op_assign -1 16 2024-04-14T15:14:22.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.565+0000 7f7e83c6c700 20 client.48400.objecter _get_session s=0x7f7e600212b0 osd=7 3 2024-04-14T15:14:22.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.565+0000 7f7e83c6c700 20 client.48400.objecter put_session s=0x7f7e600212b0 osd=7 3 2024-04-14T15:14:22.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.565+0000 7f7e83c6c700 20 client.48400.objecter _get_session s=0x7f7e600212b0 osd=7 3 2024-04-14T15:14:22.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.565+0000 7f7e83c6c700 15 client.48400.objecter _session_command_op_remove -1 16 2024-04-14T15:14:22.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.565+0000 7f7e83c6c700 20 client.48400.objecter get_session s=0x7f7e600212b0 osd=7 3 2024-04-14T15:14:22.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.565+0000 7f7e83c6c700 15 client.48400.objecter _session_command_op_assign 7 16 2024-04-14T15:14:22.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.565+0000 7f7e83c6c700 20 client.48400.objecter put_session s=0x7f7e600212b0 osd=7 4 2024-04-14T15:14:22.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.565+0000 7f7e83c6c700 10 client.48400.objecter _send_command 16 2024-04-14T15:14:22.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.565+0000 7f7e83c6c700 1 -- 172.21.15.90:0/1108240145 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f7e60027900 con 0x7f7e60023df0 2024-04-14T15:14:22.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.567+0000 7f7e72ffd700 1 -- 172.21.15.90:0/1108240145 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f7e60027900 con 0x7f7e60023df0 2024-04-14T15:14:22.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.567+0000 7f7e72ffd700 10 client.48400.objecter ms_dispatch 0x7f7e7c001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:14:22.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.567+0000 7f7e72ffd700 10 client.48400.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:14:22.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.567+0000 7f7e72ffd700 20 client.48400.objecter put_session s=0x7f7e600212b0 osd=7 4 2024-04-14T15:14:22.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.567+0000 7f7e72ffd700 15 client.48400.objecter _session_command_op_remove 7 16 2024-04-14T15:14:22.568 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:14:22.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.568+0000 7f7e83c6c700 10 librados: watch_flush enter 2024-04-14T15:14:22.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.568+0000 7f7e83c6c700 10 librados: watch_flush exit 2024-04-14T15:14:22.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.568+0000 7f7e83c6c700 10 client.48400.objecter close_session for osd.0 2024-04-14T15:14:22.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.568+0000 7f7e83c6c700 1 -- 172.21.15.90:0/1108240145 >> v1:172.21.15.90:6801/3761806885 conn(0x7f7e60004730 legacy=0x7f7e60002790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:22.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.568+0000 7f7e83c6c700 20 client.48400.objecter put_session s=0x7f7e600053e0 osd=0 1 2024-04-14T15:14:22.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.568+0000 7f7e83c6c700 10 client.48400.objecter close_session for osd.1 2024-04-14T15:14:22.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.568+0000 7f7e83c6c700 1 -- 172.21.15.90:0/1108240145 >> v1:172.21.15.90:6805/4170361699 conn(0x7f7e600042f0 legacy=0x7f7e6000ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:22.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.568+0000 7f7e83c6c700 20 client.48400.objecter put_session s=0x7f7e60004180 osd=1 1 2024-04-14T15:14:22.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.568+0000 7f7e83c6c700 10 client.48400.objecter close_session for osd.2 2024-04-14T15:14:22.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.568+0000 7f7e83c6c700 1 -- 172.21.15.90:0/1108240145 >> v1:172.21.15.90:6809/1686184251 conn(0x7f7e6000d6e0 legacy=0x7f7e6000fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:22.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.568+0000 7f7e83c6c700 20 client.48400.objecter put_session s=0x7f7e60008480 osd=2 1 2024-04-14T15:14:22.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.568+0000 7f7e83c6c700 10 client.48400.objecter close_session for osd.3 2024-04-14T15:14:22.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.568+0000 7f7e83c6c700 1 -- 172.21.15.90:0/1108240145 >> v1:172.21.15.90:6813/2270166774 conn(0x7f7e60012520 legacy=0x7f7e60014910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:22.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.568+0000 7f7e83c6c700 20 client.48400.objecter put_session s=0x7f7e60008250 osd=3 1 2024-04-14T15:14:22.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.568+0000 7f7e83c6c700 10 client.48400.objecter close_session for osd.4 2024-04-14T15:14:22.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.568+0000 7f7e83c6c700 1 -- 172.21.15.90:0/1108240145 >> v1:172.21.15.92:6800/3887549733 conn(0x7f7e60012080 legacy=0x7f7e60019370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:22.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.568+0000 7f7e83c6c700 20 client.48400.objecter put_session s=0x7f7e6000c810 osd=4 1 2024-04-14T15:14:22.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.568+0000 7f7e83c6c700 10 client.48400.objecter close_session for osd.5 2024-04-14T15:14:22.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.568+0000 7f7e83c6c700 1 -- 172.21.15.90:0/1108240145 >> v1:172.21.15.92:6804/3244561440 conn(0x7f7e6001b110 legacy=0x7f7e6001d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:22.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.568+0000 7f7e82209700 1 -- 172.21.15.90:0/1108240145 reap_dead start 2024-04-14T15:14:22.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.568+0000 7f7e83c6c700 20 client.48400.objecter put_session s=0x7f7e6001aa90 osd=5 1 2024-04-14T15:14:22.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.568+0000 7f7e83c6c700 10 client.48400.objecter close_session for osd.6 2024-04-14T15:14:22.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.568+0000 7f7e83c6c700 1 -- 172.21.15.90:0/1108240145 >> v1:172.21.15.92:6808/636125963 conn(0x7f7e60020e00 legacy=0x7f7e6001ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:22.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.568+0000 7f7e82209700 1 -- 172.21.15.90:0/1108240145 reap_dead start 2024-04-14T15:14:22.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.568+0000 7f7e83c6c700 20 client.48400.objecter put_session s=0x7f7e600206e0 osd=6 1 2024-04-14T15:14:22.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.568+0000 7f7e83c6c700 10 client.48400.objecter close_session for osd.7 2024-04-14T15:14:22.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.568+0000 7f7e83c6c700 1 -- 172.21.15.90:0/1108240145 >> v1:172.21.15.92:6812/1773706716 conn(0x7f7e60023df0 legacy=0x7f7e600261e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:22.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.568+0000 7f7e83c6c700 20 client.48400.objecter put_session s=0x7f7e600212b0 osd=7 1 2024-04-14T15:14:22.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.568+0000 7f7e83c6c700 20 client.48400.objecter shutdown clearing up homeless session... 2024-04-14T15:14:22.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.568+0000 7f7e83c6c700 10 client.48400.objecter successfully canceled tick 2024-04-14T15:14:22.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.568+0000 7f7e83c6c700 1 -- 172.21.15.90:0/1108240145 >> v1:172.21.15.90:6800/420935427 conn(0x7f7e68073720 legacy=0x7f7e68075be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:22.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.568+0000 7f7e83c6c700 1 -- 172.21.15.90:0/1108240145 >> v1:172.21.15.92:6789/0 conn(0x7f7e7c1023f0 legacy=0x7f7e7c101970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:22.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.568+0000 7f7e82209700 1 -- 172.21.15.90:0/1108240145 reap_dead start 2024-04-14T15:14:22.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.568+0000 7f7e82209700 1 -- 172.21.15.90:0/1108240145 reap_dead start 2024-04-14T15:14:22.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.568+0000 7f7e83c6c700 1 -- 172.21.15.90:0/1108240145 shutdown_connections 2024-04-14T15:14:22.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.568+0000 7f7e83c6c700 1 -- 172.21.15.90:0/1108240145 >> 172.21.15.90:0/1108240145 conn(0x7f7e7c076aa0 msgr2=0x7f7e7c109560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:14:22.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.568+0000 7f7e83c6c700 1 -- 172.21.15.90:0/1108240145 shutdown_connections 2024-04-14T15:14:22.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.568+0000 7f7e83c6c700 1 -- 172.21.15.90:0/1108240145 wait complete. 2024-04-14T15:14:22.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:22.568+0000 7f7e83c6c700 1 librados: shutdown 2024-04-14T15:14:22.572 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:14:22.572 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:14:22.572 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:14:22.572 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:14:22.572 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:14:22.573 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:14:22.573 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:14:22.573 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:14:22.659 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:14:22.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:22.684 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:22.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:14:22.785 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:14:22.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:22.809 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:22.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:14:22.817 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-04-14T15:14:22.817 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-04-14T15:14:22.818 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-04-14T15:14:22.818 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-04-14T15:14:22.818 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.818 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.24811.0:2 10.c 10:344c75f9:test-rados-api-smithi090-93398-3::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-04-14T15:14:22.818 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:56.072078+0000", 2024-04-14T15:14:22.818 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 86.744083613000001, 2024-04-14T15:14:22.818 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37243854900000001, 2024-04-14T15:14:22.818 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:14:22.818 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:14:22.818 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:14:22.818 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:14:22.818 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.24811", 2024-04-14T15:14:22.818 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/473680876", 2024-04-14T15:14:22.819 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 2 2024-04-14T15:14:22.819 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.819 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:14:22.819 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.819 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:14:22.819 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.072078+0000", 2024-04-14T15:14:22.819 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:14:22.819 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.819 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.819 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:14:22.819 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.072078+0000", 2024-04-14T15:14:22.819 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:14:22.819 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.820 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.820 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:14:22.820 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.072081+0000", 2024-04-14T15:14:22.820 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1629999999999999e-06 2024-04-14T15:14:22.820 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.820 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.820 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:14:22.820 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.072088+0000", 2024-04-14T15:14:22.820 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.6000000000000001e-06 2024-04-14T15:14:22.820 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.820 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.820 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:14:22.820 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.072098+0000", 2024-04-14T15:14:22.820 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.6339999999999996e-06 2024-04-14T15:14:22.821 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.821 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.821 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:14:22.821 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.072108+0000", 2024-04-14T15:14:22.821 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.91e-06 2024-04-14T15:14:22.821 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.821 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.821 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:14:22.821 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.094370+0000", 2024-04-14T15:14:22.821 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.022262684000000001 2024-04-14T15:14:22.821 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.821 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.821 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:14:22.822 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.094379+0000", 2024-04-14T15:14:22.822 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.884e-06 2024-04-14T15:14:22.822 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.822 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.822 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:14:22.822 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.390946+0000", 2024-04-14T15:14:22.822 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.29656687500000001 2024-04-14T15:14:22.822 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.822 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.823 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:14:22.823 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.391053+0000", 2024-04-14T15:14:22.823 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0001065 2024-04-14T15:14:22.823 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.823 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.823 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,4", 2024-04-14T15:14:22.823 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.391181+0000", 2024-04-14T15:14:22.823 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000128274 2024-04-14T15:14:22.823 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.823 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.823 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:14:22.823 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.392837+0000", 2024-04-14T15:14:22.823 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001655753 2024-04-14T15:14:22.823 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.824 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.824 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:14:22.824 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.407847+0000", 2024-04-14T15:14:22.824 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.015010449 2024-04-14T15:14:22.824 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.824 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.824 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:14:22.824 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.444405+0000", 2024-04-14T15:14:22.824 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.036557943000000002 2024-04-14T15:14:22.824 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.824 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.824 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:14:22.824 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.444466+0000", 2024-04-14T15:14:22.824 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0498999999999998e-05 2024-04-14T15:14:22.825 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.825 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.825 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:14:22.825 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.444517+0000", 2024-04-14T15:14:22.825 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.1381000000000002e-05 2024-04-14T15:14:22.825 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:14:22.825 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:14:22.825 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:14:22.825 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.825 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.825 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.24835.0:2 14.15 14:aca66623:test-rados-api-smithi090-93476-2::foo:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-04-14T15:14:22.825 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:56.073264+0000", 2024-04-14T15:14:22.825 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 86.742897654999993, 2024-04-14T15:14:22.825 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.48725632200000002, 2024-04-14T15:14:22.826 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:14:22.826 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:14:22.826 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:14:22.826 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:14:22.826 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.24835", 2024-04-14T15:14:22.826 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/1412026529", 2024-04-14T15:14:22.826 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 2 2024-04-14T15:14:22.826 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.826 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:14:22.826 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.826 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:14:22.826 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.073264+0000", 2024-04-14T15:14:22.826 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:14:22.826 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.827 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.827 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:14:22.827 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.073264+0000", 2024-04-14T15:14:22.827 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:14:22.827 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.827 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.827 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:14:22.827 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.073266+0000", 2024-04-14T15:14:22.827 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0389999999999999e-06 2024-04-14T15:14:22.827 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.827 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.827 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:14:22.827 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.073273+0000", 2024-04-14T15:14:22.827 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.6969999999999996e-06 2024-04-14T15:14:22.828 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.828 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.828 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:14:22.828 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.073397+0000", 2024-04-14T15:14:22.828 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012349099999999999 2024-04-14T15:14:22.828 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.828 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.828 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:14:22.828 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.073413+0000", 2024-04-14T15:14:22.828 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6010000000000001e-05 2024-04-14T15:14:22.828 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.828 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.828 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:14:22.828 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.123284+0000", 2024-04-14T15:14:22.829 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.049871086000000002 2024-04-14T15:14:22.829 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.829 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.829 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:14:22.829 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.123294+0000", 2024-04-14T15:14:22.829 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.7529999999999996e-06 2024-04-14T15:14:22.829 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.829 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.829 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:14:22.829 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.486557+0000", 2024-04-14T15:14:22.829 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.36326328099999999 2024-04-14T15:14:22.829 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.829 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.829 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:14:22.829 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.486683+0000", 2024-04-14T15:14:22.830 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012594900000000001 2024-04-14T15:14:22.830 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.830 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.830 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,4", 2024-04-14T15:14:22.830 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.486827+0000", 2024-04-14T15:14:22.830 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014408199999999999 2024-04-14T15:14:22.830 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.830 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.830 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:14:22.830 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.487112+0000", 2024-04-14T15:14:22.830 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00028515600000000001 2024-04-14T15:14:22.830 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.830 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.830 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:14:22.831 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.487348+0000", 2024-04-14T15:14:22.831 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000236154 2024-04-14T15:14:22.831 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.831 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.831 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:14:22.831 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.560418+0000", 2024-04-14T15:14:22.831 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.073069633999999994 2024-04-14T15:14:22.831 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.831 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.831 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:14:22.831 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.560460+0000", 2024-04-14T15:14:22.831 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2009999999999999e-05 2024-04-14T15:14:22.831 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.831 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.833 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:14:22.833 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.560521+0000", 2024-04-14T15:14:22.833 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0980000000000002e-05 2024-04-14T15:14:22.833 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:14:22.833 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:14:22.833 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:14:22.833 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.833 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.833 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.14949.0:2 20.1f 20:fc8130a0:test-rados-api-smithi090-93652-2::foo:head [undirty] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-04-14T15:14:22.834 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:56.175333+0000", 2024-04-14T15:14:22.834 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 86.640829159000006, 2024-04-14T15:14:22.834 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.36689643300000002, 2024-04-14T15:14:22.834 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:14:22.834 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:14:22.834 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-04-14T15:14:22.834 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:14:22.834 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.14949", 2024-04-14T15:14:22.834 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/495867689", 2024-04-14T15:14:22.834 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 2 2024-04-14T15:14:22.834 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.834 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:14:22.834 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.834 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:14:22.835 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.175333+0000", 2024-04-14T15:14:22.835 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:14:22.835 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.835 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.835 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:14:22.835 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.175333+0000", 2024-04-14T15:14:22.835 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:14:22.835 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.835 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.835 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:14:22.836 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.175335+0000", 2024-04-14T15:14:22.836 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.57e-06 2024-04-14T15:14:22.836 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.836 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.836 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:14:22.836 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.175340+0000", 2024-04-14T15:14:22.836 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2089999999999996e-06 2024-04-14T15:14:22.836 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.836 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.836 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:14:22.836 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.175349+0000", 2024-04-14T15:14:22.836 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.6770000000000007e-06 2024-04-14T15:14:22.836 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.836 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.837 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:14:22.837 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.175360+0000", 2024-04-14T15:14:22.837 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0579e-05 2024-04-14T15:14:22.837 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.837 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.837 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:14:22.837 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.175414+0000", 2024-04-14T15:14:22.837 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4216999999999999e-05 2024-04-14T15:14:22.837 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.837 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.837 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:14:22.837 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.175423+0000", 2024-04-14T15:14:22.837 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.2529999999999993e-06 2024-04-14T15:14:22.837 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.838 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.838 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:14:22.838 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.369598+0000", 2024-04-14T15:14:22.838 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.19417422100000001 2024-04-14T15:14:22.838 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.838 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.838 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:14:22.838 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.369694+0000", 2024-04-14T15:14:22.838 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.6237999999999996e-05 2024-04-14T15:14:22.838 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.838 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.838 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:14:22.838 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.542229+0000", 2024-04-14T15:14:22.838 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.172535469 2024-04-14T15:14:22.839 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:14:22.839 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:14:22.839 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:14:22.839 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.839 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.839 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.24889.0:2 22.15 e108/107 22:ac175842:test-rados-api-smithi090-93930-2::foo:head v 108'1, mlcod=0'0)", 2024-04-14T15:14:22.839 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:56.360343+0000", 2024-04-14T15:14:22.839 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 86.455818917000002, 2024-04-14T15:14:22.839 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.017111601000000001, 2024-04-14T15:14:22.839 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:14:22.839 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:14:22.839 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:14:22.839 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:14:22.839 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.840 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:14:22.840 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.360343+0000", 2024-04-14T15:14:22.840 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:14:22.840 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.840 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.840 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:14:22.840 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.360343+0000", 2024-04-14T15:14:22.840 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:14:22.840 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.841 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.841 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:14:22.841 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.360343+0000", 2024-04-14T15:14:22.841 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8599999999999999e-07 2024-04-14T15:14:22.841 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.842 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.842 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:14:22.842 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.360348+0000", 2024-04-14T15:14:22.842 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6099999999999999e-06 2024-04-14T15:14:22.842 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.842 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.842 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:14:22.842 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.360353+0000", 2024-04-14T15:14:22.842 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7530000000000001e-06 2024-04-14T15:14:22.842 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.842 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.842 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:14:22.842 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.360360+0000", 2024-04-14T15:14:22.843 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.861e-06 2024-04-14T15:14:22.843 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.843 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.843 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:14:22.843 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.374936+0000", 2024-04-14T15:14:22.843 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.014575832 2024-04-14T15:14:22.843 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.843 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.843 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:14:22.843 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.374955+0000", 2024-04-14T15:14:22.843 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9868999999999999e-05 2024-04-14T15:14:22.843 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.843 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.844 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:14:22.844 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.377424+0000", 2024-04-14T15:14:22.844 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0024681159999999998 2024-04-14T15:14:22.844 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.844 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.844 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:14:22.844 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.377455+0000", 2024-04-14T15:14:22.844 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1273999999999999e-05 2024-04-14T15:14:22.844 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:14:22.844 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:14:22.844 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:14:22.844 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.844 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.845 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.24860.0:5 21.e 21:76b36fd7:test-rados-api-smithi090-93864-3::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-04-14T15:14:22.845 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:56.499357+0000", 2024-04-14T15:14:22.845 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 86.316804908999998, 2024-04-14T15:14:22.845 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.042914895000000002, 2024-04-14T15:14:22.845 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:14:22.845 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:14:22.845 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:14:22.845 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:14:22.845 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.24860", 2024-04-14T15:14:22.845 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/766142007", 2024-04-14T15:14:22.845 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 5 2024-04-14T15:14:22.845 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.845 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:14:22.845 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.846 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:14:22.846 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.499357+0000", 2024-04-14T15:14:22.846 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:14:22.846 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.846 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.846 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:14:22.846 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.499357+0000", 2024-04-14T15:14:22.846 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:14:22.846 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.846 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.846 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:14:22.846 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.499359+0000", 2024-04-14T15:14:22.846 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.468e-06 2024-04-14T15:14:22.847 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.847 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.847 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:14:22.847 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.499365+0000", 2024-04-14T15:14:22.847 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3409999999999999e-06 2024-04-14T15:14:22.847 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.847 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.847 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:14:22.847 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.499373+0000", 2024-04-14T15:14:22.847 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.9999999999999996e-06 2024-04-14T15:14:22.847 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.847 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.847 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:14:22.847 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.499379+0000", 2024-04-14T15:14:22.847 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.553e-06 2024-04-14T15:14:22.848 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.848 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.848 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:14:22.848 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.499418+0000", 2024-04-14T15:14:22.848 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9793999999999999e-05 2024-04-14T15:14:22.848 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.848 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.848 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:14:22.848 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.499497+0000", 2024-04-14T15:14:22.848 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.8158999999999995e-05 2024-04-14T15:14:22.848 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.848 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.848 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1,2", 2024-04-14T15:14:22.848 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.499590+0000", 2024-04-14T15:14:22.849 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.3486000000000006e-05 2024-04-14T15:14:22.849 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.849 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.849 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:14:22.849 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.499881+0000", 2024-04-14T15:14:22.849 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00029056299999999998 2024-04-14T15:14:22.849 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.849 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.849 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:14:22.849 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.514940+0000", 2024-04-14T15:14:22.849 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.015059348 2024-04-14T15:14:22.849 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.849 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.849 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:14:22.850 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.542164+0000", 2024-04-14T15:14:22.850 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.027223798 2024-04-14T15:14:22.850 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.850 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.850 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:14:22.850 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.542232+0000", 2024-04-14T15:14:22.850 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.7900999999999999e-05 2024-04-14T15:14:22.850 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.850 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.850 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:14:22.850 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.542272+0000", 2024-04-14T15:14:22.850 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0484000000000001e-05 2024-04-14T15:14:22.850 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:14:22.851 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:14:22.851 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:14:22.851 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.851 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.851 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.14949.0:3 20.1f 20:fc8130a0:test-rados-api-smithi090-93652-2::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-04-14T15:14:22.851 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:56.545665+0000", 2024-04-14T15:14:22.851 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 86.270496875999996, 2024-04-14T15:14:22.851 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.024140885000000001, 2024-04-14T15:14:22.851 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:14:22.851 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:14:22.851 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:14:22.851 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:14:22.851 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.14949", 2024-04-14T15:14:22.852 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/495867689", 2024-04-14T15:14:22.852 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 3 2024-04-14T15:14:22.852 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.852 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:14:22.852 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.853 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:14:22.853 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.545665+0000", 2024-04-14T15:14:22.854 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:14:22.854 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.854 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.854 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:14:22.854 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.545665+0000", 2024-04-14T15:14:22.854 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:14:22.854 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.854 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.854 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:14:22.854 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.545667+0000", 2024-04-14T15:14:22.854 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.562e-06 2024-04-14T15:14:22.854 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.854 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.855 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:14:22.855 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.545672+0000", 2024-04-14T15:14:22.855 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6019999999999999e-06 2024-04-14T15:14:22.855 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.855 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.855 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:14:22.855 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.545685+0000", 2024-04-14T15:14:22.855 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2238999999999999e-05 2024-04-14T15:14:22.855 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.855 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.855 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:14:22.855 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.545693+0000", 2024-04-14T15:14:22.855 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3979999999999999e-06 2024-04-14T15:14:22.856 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.856 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.856 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:14:22.856 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.545759+0000", 2024-04-14T15:14:22.856 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5983999999999997e-05 2024-04-14T15:14:22.856 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.856 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.856 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:14:22.856 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.545831+0000", 2024-04-14T15:14:22.856 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1712999999999996e-05 2024-04-14T15:14:22.856 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.856 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.857 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1,2", 2024-04-14T15:14:22.857 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.545976+0000", 2024-04-14T15:14:22.857 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014565499999999999 2024-04-14T15:14:22.857 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.857 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.857 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:14:22.857 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.546414+0000", 2024-04-14T15:14:22.857 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00043759000000000002 2024-04-14T15:14:22.857 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.857 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.857 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:14:22.857 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.557021+0000", 2024-04-14T15:14:22.857 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.010606595 2024-04-14T15:14:22.857 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.858 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.858 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:14:22.858 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.569731+0000", 2024-04-14T15:14:22.858 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.012710749 2024-04-14T15:14:22.858 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.858 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.858 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:14:22.858 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.569773+0000", 2024-04-14T15:14:22.858 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1363999999999999e-05 2024-04-14T15:14:22.858 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.858 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.858 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:14:22.858 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.569806+0000", 2024-04-14T15:14:22.859 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.3433999999999999e-05 2024-04-14T15:14:22.859 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:14:22.859 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:14:22.859 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:14:22.859 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.859 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.859 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.24965.0:5 30.12 30:4a6d8757:::3.obj:head [write 0~115] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-04-14T15:14:22.859 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:56.556271+0000", 2024-04-14T15:14:22.859 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 86.259891326000002, 2024-04-14T15:14:22.859 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.013576438, 2024-04-14T15:14:22.859 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:14:22.860 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:14:22.860 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:14:22.860 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:14:22.860 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.24965", 2024-04-14T15:14:22.860 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/1840281572", 2024-04-14T15:14:22.860 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 5 2024-04-14T15:14:22.860 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.860 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:14:22.860 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.860 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:14:22.860 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.556271+0000", 2024-04-14T15:14:22.860 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:14:22.860 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.861 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.861 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:14:22.861 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.556271+0000", 2024-04-14T15:14:22.861 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:14:22.861 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.861 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.861 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:14:22.861 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.556272+0000", 2024-04-14T15:14:22.861 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.5000000000000001e-07 2024-04-14T15:14:22.861 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.861 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.861 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:14:22.861 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.556276+0000", 2024-04-14T15:14:22.862 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0060000000000003e-06 2024-04-14T15:14:22.862 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.862 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.862 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:14:22.862 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.556283+0000", 2024-04-14T15:14:22.862 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2309999999999999e-06 2024-04-14T15:14:22.862 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.862 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.862 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:14:22.862 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.556288+0000", 2024-04-14T15:14:22.862 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5099999999999998e-06 2024-04-14T15:14:22.862 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.862 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.863 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:14:22.863 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.556331+0000", 2024-04-14T15:14:22.863 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2735999999999999e-05 2024-04-14T15:14:22.863 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.863 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.863 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:14:22.863 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.556440+0000", 2024-04-14T15:14:22.863 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010840000000000001 2024-04-14T15:14:22.863 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.863 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.863 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 2,3", 2024-04-14T15:14:22.863 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.556568+0000", 2024-04-14T15:14:22.863 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000128639 2024-04-14T15:14:22.863 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.864 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.864 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:14:22.864 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.556887+0000", 2024-04-14T15:14:22.864 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00031888199999999999 2024-04-14T15:14:22.864 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.864 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.864 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:14:22.864 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.564915+0000", 2024-04-14T15:14:22.864 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0080279530000000009 2024-04-14T15:14:22.864 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.864 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.864 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:14:22.864 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.569789+0000", 2024-04-14T15:14:22.864 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00487354 2024-04-14T15:14:22.865 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.865 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.865 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:14:22.865 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.569821+0000", 2024-04-14T15:14:22.865 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2138000000000003e-05 2024-04-14T15:14:22.865 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.865 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.865 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:14:22.865 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.569847+0000", 2024-04-14T15:14:22.865 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6452999999999999e-05 2024-04-14T15:14:22.865 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:14:22.865 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:14:22.865 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:14:22.865 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.866 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.866 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25144.0:16 24.10 24:09666ec2:::28.obj2:head [write 0~154] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-04-14T15:14:22.866 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:57.069924+0000", 2024-04-14T15:14:22.866 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 85.746238266000006, 2024-04-14T15:14:22.866 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.26734572499999998, 2024-04-14T15:14:22.866 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:14:22.866 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:14:22.866 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:14:22.866 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:14:22.866 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25144", 2024-04-14T15:14:22.866 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/790563004", 2024-04-14T15:14:22.866 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 16 2024-04-14T15:14:22.866 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.866 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:14:22.867 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.867 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:14:22.867 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.069924+0000", 2024-04-14T15:14:22.867 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:14:22.867 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.867 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.867 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:14:22.867 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.069924+0000", 2024-04-14T15:14:22.867 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:14:22.867 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.867 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.867 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:14:22.867 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.069926+0000", 2024-04-14T15:14:22.867 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.294e-06 2024-04-14T15:14:22.868 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.868 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.868 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:14:22.868 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.070071+0000", 2024-04-14T15:14:22.868 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014495299999999999 2024-04-14T15:14:22.868 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.868 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.868 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:14:22.868 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.070632+0000", 2024-04-14T15:14:22.868 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00056131300000000002 2024-04-14T15:14:22.868 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.869 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.869 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:14:22.869 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.070642+0000", 2024-04-14T15:14:22.869 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.3980000000000005e-06 2024-04-14T15:14:22.869 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.869 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.869 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:14:22.869 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.073309+0000", 2024-04-14T15:14:22.869 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002666917 2024-04-14T15:14:22.869 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.869 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.869 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:14:22.869 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.073566+0000", 2024-04-14T15:14:22.869 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00025740399999999998 2024-04-14T15:14:22.869 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.870 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.870 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,4", 2024-04-14T15:14:22.870 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.076377+0000", 2024-04-14T15:14:22.870 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0028108629999999998 2024-04-14T15:14:22.870 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.870 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.870 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:14:22.870 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.133777+0000", 2024-04-14T15:14:22.870 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.057399618999999999 2024-04-14T15:14:22.870 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.870 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.870 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:14:22.870 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.197851+0000", 2024-04-14T15:14:22.871 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.064074318000000005 2024-04-14T15:14:22.871 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.871 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.871 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:14:22.871 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.337201+0000", 2024-04-14T15:14:22.871 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.13934986099999999 2024-04-14T15:14:22.871 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.871 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.871 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:14:22.871 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.337238+0000", 2024-04-14T15:14:22.871 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.7283000000000002e-05 2024-04-14T15:14:22.872 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.872 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.872 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:14:22.872 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.337270+0000", 2024-04-14T15:14:22.872 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1501999999999998e-05 2024-04-14T15:14:22.872 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:14:22.872 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:14:22.872 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:14:22.872 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.872 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.872 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.24829.0:830 13.1d 13:b8000000::::head [pgnls start_epoch 109] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e109)", 2024-04-14T15:14:22.872 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:57.162358+0000", 2024-04-14T15:14:22.873 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 85.653804179999995, 2024-04-14T15:14:22.873 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.015254693, 2024-04-14T15:14:22.873 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:14:22.873 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:14:22.873 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-04-14T15:14:22.873 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:14:22.873 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.24829", 2024-04-14T15:14:22.873 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/218717313", 2024-04-14T15:14:22.873 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 830 2024-04-14T15:14:22.873 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.873 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:14:22.873 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.873 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:14:22.873 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.162358+0000", 2024-04-14T15:14:22.873 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:14:22.874 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.874 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.874 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:14:22.874 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.162358+0000", 2024-04-14T15:14:22.874 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:14:22.874 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.874 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.874 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:14:22.874 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.162360+0000", 2024-04-14T15:14:22.875 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1210000000000001e-06 2024-04-14T15:14:22.875 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.875 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.875 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:14:22.875 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.162366+0000", 2024-04-14T15:14:22.875 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2740000000000004e-06 2024-04-14T15:14:22.875 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.875 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.875 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:14:22.875 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.168995+0000", 2024-04-14T15:14:22.875 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0066289230000000001 2024-04-14T15:14:22.876 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.876 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.876 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:14:22.876 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.169012+0000", 2024-04-14T15:14:22.876 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6531000000000001e-05 2024-04-14T15:14:22.876 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.876 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.876 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:14:22.876 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.177479+0000", 2024-04-14T15:14:22.876 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0084670560000000006 2024-04-14T15:14:22.876 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.876 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.876 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:14:22.876 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.177519+0000", 2024-04-14T15:14:22.876 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0059999999999999e-05 2024-04-14T15:14:22.877 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.877 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.877 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:14:22.877 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.177613+0000", 2024-04-14T15:14:22.877 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.3727999999999994e-05 2024-04-14T15:14:22.877 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:14:22.877 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:14:22.877 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:14:22.877 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.877 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.877 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.25144.0:16 24.10 e109/107 ondisk, result = 0)", 2024-04-14T15:14:22.877 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:57.179164+0000", 2024-04-14T15:14:22.877 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 85.636998086000006, 2024-04-14T15:14:22.877 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.018695868000000001, 2024-04-14T15:14:22.877 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:14:22.878 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:14:22.878 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-04-14T15:14:22.878 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:14:22.878 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.878 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:14:22.878 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.179164+0000", 2024-04-14T15:14:22.878 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:14:22.878 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.878 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.878 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:14:22.878 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.179164+0000", 2024-04-14T15:14:22.878 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:14:22.879 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.879 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.879 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:14:22.879 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.179166+0000", 2024-04-14T15:14:22.879 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6700000000000001e-06 2024-04-14T15:14:22.879 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.879 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.879 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:14:22.879 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.179173+0000", 2024-04-14T15:14:22.879 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3139999999999998e-06 2024-04-14T15:14:22.879 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.879 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.879 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:14:22.879 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.180020+0000", 2024-04-14T15:14:22.880 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00084736399999999997 2024-04-14T15:14:22.880 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.880 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.880 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:14:22.880 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.180031+0000", 2024-04-14T15:14:22.880 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0451e-05 2024-04-14T15:14:22.880 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.880 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.880 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:14:22.880 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.197835+0000", 2024-04-14T15:14:22.880 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.017803785999999999 2024-04-14T15:14:22.880 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.880 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.881 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:14:22.881 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.197844+0000", 2024-04-14T15:14:22.881 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.4630000000000007e-06 2024-04-14T15:14:22.881 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.881 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.881 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:14:22.881 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.197860+0000", 2024-04-14T15:14:22.881 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5820000000000001e-05 2024-04-14T15:14:22.881 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:14:22.881 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:14:22.881 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:14:22.881 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.881 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.881 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25255.0:2 62.6 62:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-04-14T15:14:22.882 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:58.078816+0000", 2024-04-14T15:14:22.882 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 84.737345699000002, 2024-04-14T15:14:22.882 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2444698199999999, 2024-04-14T15:14:22.882 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:14:22.882 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:14:22.882 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:14:22.882 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:14:22.882 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25255", 2024-04-14T15:14:22.882 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/574211416", 2024-04-14T15:14:22.882 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 2 2024-04-14T15:14:22.882 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.882 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:14:22.882 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.883 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:14:22.883 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.078816+0000", 2024-04-14T15:14:22.883 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:14:22.883 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.883 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.883 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:14:22.883 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.078816+0000", 2024-04-14T15:14:22.883 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:14:22.883 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.883 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.883 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:14:22.883 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.078818+0000", 2024-04-14T15:14:22.883 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.001e-06 2024-04-14T15:14:22.883 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.883 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.884 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:14:22.884 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.078824+0000", 2024-04-14T15:14:22.884 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9009999999999999e-06 2024-04-14T15:14:22.884 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.884 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.884 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:14:22.884 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.078844+0000", 2024-04-14T15:14:22.884 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9224000000000001e-05 2024-04-14T15:14:22.884 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.884 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.884 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:14:22.884 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.078856+0000", 2024-04-14T15:14:22.884 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2815e-05 2024-04-14T15:14:22.884 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.884 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.885 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:14:22.885 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.090814+0000", 2024-04-14T15:14:22.885 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.011957756 2024-04-14T15:14:22.885 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.885 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.885 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:14:22.885 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.090823+0000", 2024-04-14T15:14:22.885 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8799999999999997e-06 2024-04-14T15:14:22.885 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.885 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.885 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:14:22.885 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.292320+0000", 2024-04-14T15:14:22.885 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.201496535 2024-04-14T15:14:22.885 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.886 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.886 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:14:22.886 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.292408+0000", 2024-04-14T15:14:22.886 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8769000000000003e-05 2024-04-14T15:14:22.886 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.886 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.886 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 2,4", 2024-04-14T15:14:22.886 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.292509+0000", 2024-04-14T15:14:22.886 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000100987 2024-04-14T15:14:22.886 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.886 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.886 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:14:22.887 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.294694+0000", 2024-04-14T15:14:22.887 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0021848530000000001 2024-04-14T15:14:22.887 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.887 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.887 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:14:22.887 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.299561+0000", 2024-04-14T15:14:22.887 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0048669450000000001 2024-04-14T15:14:22.887 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.887 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.887 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:14:22.887 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.323203+0000", 2024-04-14T15:14:22.887 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.023641639999999998 2024-04-14T15:14:22.887 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.888 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.888 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:14:22.888 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.323249+0000", 2024-04-14T15:14:22.888 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6211999999999997e-05 2024-04-14T15:14:22.888 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.888 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.888 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:14:22.888 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.323286+0000", 2024-04-14T15:14:22.888 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.7302000000000003e-05 2024-04-14T15:14:22.888 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:14:22.888 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:14:22.888 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:14:22.888 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.888 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.889 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25177.0:2 52.15 52:ad551702:::test:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-04-14T15:14:22.889 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:58.078901+0000", 2024-04-14T15:14:22.889 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 84.737261431999997, 2024-04-14T15:14:22.889 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.244162038, 2024-04-14T15:14:22.889 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:14:22.889 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:14:22.889 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:14:22.889 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:14:22.889 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25177", 2024-04-14T15:14:22.889 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/1748400265", 2024-04-14T15:14:22.889 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 2 2024-04-14T15:14:22.889 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.889 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:14:22.889 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.889 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:14:22.890 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.078901+0000", 2024-04-14T15:14:22.890 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:14:22.890 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.890 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.890 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:14:22.890 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.078901+0000", 2024-04-14T15:14:22.890 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:14:22.890 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.890 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.890 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:14:22.890 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.078902+0000", 2024-04-14T15:14:22.891 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.113e-06 2024-04-14T15:14:22.891 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.891 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.891 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:14:22.891 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.078906+0000", 2024-04-14T15:14:22.891 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2200000000000003e-06 2024-04-14T15:14:22.891 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.891 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.891 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:14:22.891 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.078914+0000", 2024-04-14T15:14:22.891 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0190000000000006e-06 2024-04-14T15:14:22.891 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.891 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.891 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:14:22.892 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.078922+0000", 2024-04-14T15:14:22.892 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5549999999999997e-06 2024-04-14T15:14:22.892 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.892 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.892 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:14:22.892 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.100239+0000", 2024-04-14T15:14:22.892 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.021317499 2024-04-14T15:14:22.892 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.892 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.892 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:14:22.892 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.100247+0000", 2024-04-14T15:14:22.892 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.8469999999999996e-06 2024-04-14T15:14:22.892 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.892 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.893 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:14:22.893 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.290831+0000", 2024-04-14T15:14:22.893 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1905837370000001 2024-04-14T15:14:22.893 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.893 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.893 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:14:22.893 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.292076+0000", 2024-04-14T15:14:22.893 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001245777 2024-04-14T15:14:22.893 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.893 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.893 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 2,4", 2024-04-14T15:14:22.893 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.292205+0000", 2024-04-14T15:14:22.893 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012861 2024-04-14T15:14:22.893 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.895 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.895 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:14:22.895 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.294671+0000", 2024-04-14T15:14:22.895 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0024663110000000001 2024-04-14T15:14:22.895 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.895 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.895 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:14:22.895 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.301523+0000", 2024-04-14T15:14:22.895 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.006851345 2024-04-14T15:14:22.896 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.896 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.896 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:14:22.896 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.322975+0000", 2024-04-14T15:14:22.896 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.021452123 2024-04-14T15:14:22.896 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.896 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.896 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:14:22.896 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.323023+0000", 2024-04-14T15:14:22.896 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8661e-05 2024-04-14T15:14:22.896 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.896 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.896 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:14:22.896 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.323063+0000", 2024-04-14T15:14:22.897 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9221000000000003e-05 2024-04-14T15:14:22.897 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:14:22.897 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:14:22.897 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:14:22.897 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.897 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.897 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25192.0:2 54.6 54:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-14T15:14:22.897 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:59.092683+0000", 2024-04-14T15:14:22.897 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 83.723479482000002, 2024-04-14T15:14:22.897 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.23136773699999999, 2024-04-14T15:14:22.897 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:14:22.897 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:14:22.897 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:14:22.897 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:14:22.898 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25192", 2024-04-14T15:14:22.898 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/246759086", 2024-04-14T15:14:22.898 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 2 2024-04-14T15:14:22.898 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.898 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:14:22.898 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.898 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:14:22.898 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.092683+0000", 2024-04-14T15:14:22.898 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:14:22.898 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.898 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.898 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:14:22.898 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.092683+0000", 2024-04-14T15:14:22.899 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:14:22.899 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.899 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.899 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:14:22.899 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.092685+0000", 2024-04-14T15:14:22.899 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.176e-06 2024-04-14T15:14:22.899 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.899 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.899 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:14:22.899 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.092692+0000", 2024-04-14T15:14:22.899 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3470000000000001e-06 2024-04-14T15:14:22.899 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.900 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.900 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:14:22.900 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.092708+0000", 2024-04-14T15:14:22.900 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6137000000000002e-05 2024-04-14T15:14:22.900 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.900 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.900 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:14:22.900 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.092719+0000", 2024-04-14T15:14:22.900 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0703000000000001e-05 2024-04-14T15:14:22.900 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.900 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.900 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:14:22.900 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.092781+0000", 2024-04-14T15:14:22.900 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2409999999999994e-05 2024-04-14T15:14:22.901 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.901 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.901 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op must wait for map", 2024-04-14T15:14:22.901 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.092788+0000", 2024-04-14T15:14:22.901 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9800000000000001e-06 2024-04-14T15:14:22.901 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.901 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.901 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:14:22.901 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.148615+0000", 2024-04-14T15:14:22.901 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.055826133999999999 2024-04-14T15:14:22.901 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.901 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.901 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:14:22.901 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.148624+0000", 2024-04-14T15:14:22.901 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.4010000000000003e-06 2024-04-14T15:14:22.902 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.902 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.902 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:14:22.902 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.290897+0000", 2024-04-14T15:14:22.902 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.14227313799999999 2024-04-14T15:14:22.902 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.902 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.902 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:14:22.902 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.292381+0000", 2024-04-14T15:14:22.902 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0014842799999999999 2024-04-14T15:14:22.902 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.902 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.902 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 2,5", 2024-04-14T15:14:22.903 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.292499+0000", 2024-04-14T15:14:22.903 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011736299999999999 2024-04-14T15:14:22.903 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.903 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.903 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:14:22.903 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.294699+0000", 2024-04-14T15:14:22.903 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0022005520000000002 2024-04-14T15:14:22.903 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.903 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.903 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:14:22.903 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.303496+0000", 2024-04-14T15:14:22.903 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0087963390000000002 2024-04-14T15:14:22.904 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.904 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.904 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:14:22.904 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.323957+0000", 2024-04-14T15:14:22.904 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.02046156 2024-04-14T15:14:22.904 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.904 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.904 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:14:22.904 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.324008+0000", 2024-04-14T15:14:22.904 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.1146999999999997e-05 2024-04-14T15:14:22.904 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.904 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.904 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:14:22.904 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.324050+0000", 2024-04-14T15:14:22.905 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2070000000000002e-05 2024-04-14T15:14:22.905 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:14:22.905 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:14:22.905 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:14:22.905 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.905 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.905 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25264.0:3 61.15 61:ab97ba35:::nonexistent:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-14T15:14:22.905 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:59.278987+0000", 2024-04-14T15:14:22.905 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 83.537175039000005, 2024-04-14T15:14:22.905 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.019926803, 2024-04-14T15:14:22.905 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:14:22.905 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:14:22.905 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-04-14T15:14:22.906 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:14:22.906 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25264", 2024-04-14T15:14:22.906 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/4109623542", 2024-04-14T15:14:22.906 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 3 2024-04-14T15:14:22.906 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.906 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:14:22.906 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.906 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:14:22.906 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.278987+0000", 2024-04-14T15:14:22.906 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:14:22.906 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.906 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.906 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:14:22.906 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.278987+0000", 2024-04-14T15:14:22.907 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:14:22.907 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.907 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.907 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:14:22.907 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.278989+0000", 2024-04-14T15:14:22.907 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5549999999999999e-06 2024-04-14T15:14:22.907 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.907 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.907 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:14:22.907 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.278993+0000", 2024-04-14T15:14:22.907 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1590000000000003e-06 2024-04-14T15:14:22.907 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.907 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.907 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:14:22.907 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.279001+0000", 2024-04-14T15:14:22.908 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.7600000000000002e-06 2024-04-14T15:14:22.908 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.908 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.908 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:14:22.908 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.279008+0000", 2024-04-14T15:14:22.908 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.4850000000000003e-06 2024-04-14T15:14:22.908 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.908 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.908 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:14:22.908 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.279055+0000", 2024-04-14T15:14:22.908 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7202e-05 2024-04-14T15:14:22.909 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.909 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.909 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:14:22.909 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.279063+0000", 2024-04-14T15:14:22.909 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.4070000000000004e-06 2024-04-14T15:14:22.909 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.909 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.909 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:14:22.909 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.298830+0000", 2024-04-14T15:14:22.909 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.019767429 2024-04-14T15:14:22.909 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.909 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.909 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:14:22.910 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.298914+0000", 2024-04-14T15:14:22.910 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3806000000000004e-05 2024-04-14T15:14:22.910 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:14:22.910 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:14:22.910 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:14:22.910 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.910 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.910 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25249.0:1 70.0s0 70:16c0e1b8:test-rados-api-smithi090-93731-8::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-04-14T15:14:22.910 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:13:00.113664+0000", 2024-04-14T15:14:22.910 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 82.702497700999999, 2024-04-14T15:14:22.910 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.83956185500000002, 2024-04-14T15:14:22.910 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:14:22.910 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:14:22.910 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:14:22.911 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:14:22.911 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25249", 2024-04-14T15:14:22.911 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/3274182089", 2024-04-14T15:14:22.911 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 1 2024-04-14T15:14:22.911 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.911 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:14:22.911 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.911 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:14:22.911 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.113664+0000", 2024-04-14T15:14:22.911 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:14:22.911 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.911 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.911 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:14:22.911 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.113664+0000", 2024-04-14T15:14:22.912 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:14:22.912 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.912 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.912 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:14:22.912 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.113667+0000", 2024-04-14T15:14:22.912 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3410000000000001e-06 2024-04-14T15:14:22.912 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.912 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.912 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:14:22.912 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.113676+0000", 2024-04-14T15:14:22.912 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.9570000000000008e-06 2024-04-14T15:14:22.912 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.912 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.912 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:14:22.912 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.114092+0000", 2024-04-14T15:14:22.913 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00041648800000000001 2024-04-14T15:14:22.913 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.913 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.913 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:14:22.913 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.114106+0000", 2024-04-14T15:14:22.913 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4071e-05 2024-04-14T15:14:22.913 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.913 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.913 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:14:22.913 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.178348+0000", 2024-04-14T15:14:22.913 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.064242219000000003 2024-04-14T15:14:22.913 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.913 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.914 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:14:22.914 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.178353+0000", 2024-04-14T15:14:22.914 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2579999999999997e-06 2024-04-14T15:14:22.914 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.914 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.914 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:14:22.914 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.950009+0000", 2024-04-14T15:14:22.914 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.77165668899999995 2024-04-14T15:14:22.914 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.914 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.915 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:14:22.915 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.950122+0000", 2024-04-14T15:14:22.915 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000112571 2024-04-14T15:14:22.915 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.915 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.915 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_started", 2024-04-14T15:14:22.915 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.950521+0000", 2024-04-14T15:14:22.915 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00039894300000000002 2024-04-14T15:14:22.915 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.915 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.915 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_committed", 2024-04-14T15:14:22.915 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.951090+0000", 2024-04-14T15:14:22.915 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00056921700000000001 2024-04-14T15:14:22.915 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.915 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.916 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:14:22.916 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.953177+0000", 2024-04-14T15:14:22.916 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002086999 2024-04-14T15:14:22.916 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.916 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.916 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:14:22.916 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.953226+0000", 2024-04-14T15:14:22.916 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.9101999999999998e-05 2024-04-14T15:14:22.916 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:14:22.916 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:14:22.916 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:14:22.916 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.916 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.916 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.24868.0:8 17.1d 17:bc2e7061:test-rados-api-smithi090-93724-4::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-04-14T15:14:22.917 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:13:00.135108+0000", 2024-04-14T15:14:22.917 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 82.681053994999999, 2024-04-14T15:14:22.917 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.16196240200000001, 2024-04-14T15:14:22.917 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:14:22.917 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:14:22.917 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:14:22.917 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:14:22.917 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.24868", 2024-04-14T15:14:22.917 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/1684644655", 2024-04-14T15:14:22.917 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 8 2024-04-14T15:14:22.917 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.917 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:14:22.917 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.917 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:14:22.918 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.135108+0000", 2024-04-14T15:14:22.918 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:14:22.918 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.918 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.918 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:14:22.918 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.135108+0000", 2024-04-14T15:14:22.918 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:14:22.918 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.918 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.918 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:14:22.918 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.135110+0000", 2024-04-14T15:14:22.918 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.08e-06 2024-04-14T15:14:22.918 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.919 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.919 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:14:22.919 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.135118+0000", 2024-04-14T15:14:22.925 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.8369999999999997e-06 2024-04-14T15:14:22.925 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.925 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.926 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:14:22.926 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.138414+0000", 2024-04-14T15:14:22.926 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0032960609999999999 2024-04-14T15:14:22.926 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.926 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.926 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:14:22.926 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.138431+0000", 2024-04-14T15:14:22.926 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6481999999999999e-05 2024-04-14T15:14:22.926 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.926 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.926 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:14:22.926 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.184836+0000", 2024-04-14T15:14:22.927 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.046405710000000003 2024-04-14T15:14:22.927 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.927 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.927 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:14:22.927 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.184962+0000", 2024-04-14T15:14:22.927 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012590600000000001 2024-04-14T15:14:22.927 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.928 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.928 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 2,3", 2024-04-14T15:14:22.928 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.185123+0000", 2024-04-14T15:14:22.928 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016045400000000001 2024-04-14T15:14:22.928 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.928 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.928 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:14:22.928 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.194908+0000", 2024-04-14T15:14:22.928 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00978563 2024-04-14T15:14:22.928 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.928 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.929 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:14:22.929 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.252165+0000", 2024-04-14T15:14:22.929 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.057256970999999997 2024-04-14T15:14:22.929 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.929 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.929 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:14:22.929 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.296544+0000", 2024-04-14T15:14:22.929 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.044378784999999997 2024-04-14T15:14:22.929 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.929 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.929 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:14:22.929 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.297018+0000", 2024-04-14T15:14:22.929 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00047406299999999999 2024-04-14T15:14:22.929 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.930 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.930 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:14:22.930 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.297071+0000", 2024-04-14T15:14:22.930 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2423000000000003e-05 2024-04-14T15:14:22.930 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:14:22.930 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:14:22.930 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:14:22.930 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.930 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.930 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.24868.0:10 17.1d 17:bc2e7061:test-rados-api-smithi090-93724-4::foo:head [writefull 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-04-14T15:14:22.930 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:13:00.846041+0000", 2024-04-14T15:14:22.930 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 81.970120659000003, 2024-04-14T15:14:22.930 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.114722243, 2024-04-14T15:14:22.930 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:14:22.931 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:14:22.931 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:14:22.931 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:14:22.931 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.24868", 2024-04-14T15:14:22.931 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/1684644655", 2024-04-14T15:14:22.931 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 10 2024-04-14T15:14:22.931 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.931 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:14:22.931 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.931 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:14:22.931 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.846041+0000", 2024-04-14T15:14:22.931 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:14:22.931 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.931 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.932 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:14:22.932 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.846041+0000", 2024-04-14T15:14:22.932 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:14:22.932 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.932 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.932 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:14:22.932 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.846043+0000", 2024-04-14T15:14:22.932 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.691e-06 2024-04-14T15:14:22.932 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.932 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.932 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:14:22.932 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.846049+0000", 2024-04-14T15:14:22.933 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.8769999999999998e-06 2024-04-14T15:14:22.933 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.933 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.933 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:14:22.933 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.846057+0000", 2024-04-14T15:14:22.933 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2810000000000008e-06 2024-04-14T15:14:22.933 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.933 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.933 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:14:22.933 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.846067+0000", 2024-04-14T15:14:22.933 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.3409999999999992e-06 2024-04-14T15:14:22.933 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.933 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.933 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:14:22.933 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.853559+0000", 2024-04-14T15:14:22.934 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0074928039999999996 2024-04-14T15:14:22.934 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.934 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.934 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op must wait for map", 2024-04-14T15:14:22.934 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.853567+0000", 2024-04-14T15:14:22.934 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.4780000000000002e-06 2024-04-14T15:14:22.934 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.934 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.934 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:14:22.934 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.896296+0000", 2024-04-14T15:14:22.934 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.042729023999999997 2024-04-14T15:14:22.934 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.934 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.934 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:14:22.935 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.896481+0000", 2024-04-14T15:14:22.935 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000185484 2024-04-14T15:14:22.935 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.935 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.935 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 2,3", 2024-04-14T15:14:22.935 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.896606+0000", 2024-04-14T15:14:22.935 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012460300000000001 2024-04-14T15:14:22.935 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.935 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.935 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:14:22.935 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.897735+0000", 2024-04-14T15:14:22.935 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0011286239999999999 2024-04-14T15:14:22.935 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.936 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.936 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:14:22.936 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.930420+0000", 2024-04-14T15:14:22.936 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.032685487999999999 2024-04-14T15:14:22.936 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.936 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.936 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:14:22.936 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.960575+0000", 2024-04-14T15:14:22.936 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.030155011999999998 2024-04-14T15:14:22.936 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.936 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.936 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:14:22.936 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.960718+0000", 2024-04-14T15:14:22.936 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014259799999999999 2024-04-14T15:14:22.937 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.937 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.937 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:14:22.937 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.960764+0000", 2024-04-14T15:14:22.937 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5938e-05 2024-04-14T15:14:22.937 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:14:22.937 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:14:22.937 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:14:22.937 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.937 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.937 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.44350.0:2 162.6 162:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-14T15:14:22.937 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:13:28.858744+0000", 2024-04-14T15:14:22.937 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 53.957417673000002, 2024-04-14T15:14:22.938 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.096871834000000004, 2024-04-14T15:14:22.938 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:14:22.938 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:14:22.938 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:14:22.938 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:14:22.938 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.44350", 2024-04-14T15:14:22.938 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/685063981", 2024-04-14T15:14:22.938 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 2 2024-04-14T15:14:22.938 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.938 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:14:22.939 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.939 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:14:22.939 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.858744+0000", 2024-04-14T15:14:22.939 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:14:22.939 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.939 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.939 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:14:22.939 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.858744+0000", 2024-04-14T15:14:22.939 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:14:22.939 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.939 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.939 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:14:22.939 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.858747+0000", 2024-04-14T15:14:22.939 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2639999999999998e-06 2024-04-14T15:14:22.940 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.940 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.940 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:14:22.940 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.858753+0000", 2024-04-14T15:14:22.940 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2049999999999999e-06 2024-04-14T15:14:22.940 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.940 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.940 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:14:22.940 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.858765+0000", 2024-04-14T15:14:22.940 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2009e-05 2024-04-14T15:14:22.940 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.940 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.940 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:14:22.941 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.858774+0000", 2024-04-14T15:14:22.941 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.3360000000000001e-06 2024-04-14T15:14:22.941 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.941 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.941 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:14:22.941 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.858831+0000", 2024-04-14T15:14:22.941 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6548999999999997e-05 2024-04-14T15:14:22.941 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.941 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.941 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:14:22.941 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.858840+0000", 2024-04-14T15:14:22.941 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8000000000000004e-06 2024-04-14T15:14:22.941 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.941 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.942 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:14:22.942 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.920124+0000", 2024-04-14T15:14:22.942 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.061284813 2024-04-14T15:14:22.942 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.942 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.942 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:14:22.942 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.920263+0000", 2024-04-14T15:14:22.942 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000138209 2024-04-14T15:14:22.942 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.942 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.942 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1,5", 2024-04-14T15:14:22.942 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.920410+0000", 2024-04-14T15:14:22.942 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000147207 2024-04-14T15:14:22.942 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.943 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.943 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:14:22.943 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.924400+0000", 2024-04-14T15:14:22.943 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0039906450000000001 2024-04-14T15:14:22.943 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.943 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.943 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:14:22.943 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.924948+0000", 2024-04-14T15:14:22.943 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000547099 2024-04-14T15:14:22.943 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.943 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.943 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:14:22.943 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.955500+0000", 2024-04-14T15:14:22.943 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.030552240000000001 2024-04-14T15:14:22.943 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.944 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.944 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:14:22.944 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.955561+0000", 2024-04-14T15:14:22.944 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1462000000000002e-05 2024-04-14T15:14:22.944 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.944 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.944 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:14:22.944 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.955616+0000", 2024-04-14T15:14:22.944 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4996e-05 2024-04-14T15:14:22.944 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:14:22.944 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:14:22.944 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:14:22.944 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.944 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.944 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.48145.0:2 184.d 184:b5a28c71:::oid_FlushPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-04-14T15:14:22.945 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:13:49.857587+0000", 2024-04-14T15:14:22.945 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 32.958575430000003, 2024-04-14T15:14:22.945 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.027962643999999998, 2024-04-14T15:14:22.945 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:14:22.945 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:14:22.945 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:14:22.945 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:14:22.945 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.48145", 2024-04-14T15:14:22.945 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/3120752525", 2024-04-14T15:14:22.945 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 2 2024-04-14T15:14:22.945 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.945 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:14:22.945 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.945 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:14:22.945 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:49.857587+0000", 2024-04-14T15:14:22.946 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:14:22.946 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.946 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.946 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:14:22.946 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:49.857587+0000", 2024-04-14T15:14:22.946 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:14:22.946 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.946 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.946 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:14:22.946 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:49.857589+0000", 2024-04-14T15:14:22.946 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6730000000000001e-06 2024-04-14T15:14:22.946 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.946 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.946 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:14:22.947 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:49.857597+0000", 2024-04-14T15:14:22.947 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3239999999999997e-06 2024-04-14T15:14:22.947 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.947 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.947 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:14:22.947 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:49.860459+0000", 2024-04-14T15:14:22.947 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002862337 2024-04-14T15:14:22.947 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.947 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.947 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:14:22.947 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:49.860472+0000", 2024-04-14T15:14:22.947 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2948999999999999e-05 2024-04-14T15:14:22.947 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.947 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.947 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:14:22.948 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:49.865618+0000", 2024-04-14T15:14:22.948 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0051455449999999996 2024-04-14T15:14:22.948 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.948 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.948 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:14:22.948 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:49.865627+0000", 2024-04-14T15:14:22.948 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.1519999999999998e-06 2024-04-14T15:14:22.948 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.948 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.949 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:14:22.949 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:49.873327+0000", 2024-04-14T15:14:22.949 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0076999570000000003 2024-04-14T15:14:22.949 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.949 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.949 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:14:22.949 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:49.873435+0000", 2024-04-14T15:14:22.949 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000108412 2024-04-14T15:14:22.949 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.949 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.949 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 2,5", 2024-04-14T15:14:22.949 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:49.873562+0000", 2024-04-14T15:14:22.949 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012723899999999999 2024-04-14T15:14:22.949 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.949 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.950 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:14:22.950 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:49.875260+0000", 2024-04-14T15:14:22.950 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0016974119999999999 2024-04-14T15:14:22.950 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.950 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.950 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:14:22.950 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:49.880846+0000", 2024-04-14T15:14:22.950 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0055863869999999999 2024-04-14T15:14:22.950 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.950 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.950 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:14:22.950 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:49.885357+0000", 2024-04-14T15:14:22.950 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0045112019999999997 2024-04-14T15:14:22.950 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.950 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.951 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:14:22.951 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:49.885416+0000", 2024-04-14T15:14:22.951 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9178000000000002e-05 2024-04-14T15:14:22.951 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.951 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.951 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:14:22.951 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:49.885549+0000", 2024-04-14T15:14:22.951 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000132877 2024-04-14T15:14:22.951 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:14:22.951 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:14:22.951 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:14:22.951 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.951 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.951 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.46400.0:2 202.11 e184/182 202:8de73fe4:test-rados-api-smithi090-93601-26::foo:head v 184'1, mlcod=0'0)", 2024-04-14T15:14:22.951 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:14:11.855244+0000", 2024-04-14T15:14:22.952 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 10.960917873, 2024-04-14T15:14:22.952 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.03739924, 2024-04-14T15:14:22.952 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:14:22.952 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:14:22.952 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:14:22.952 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:14:22.952 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.952 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:14:22.952 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:11.855244+0000", 2024-04-14T15:14:22.952 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:14:22.952 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.952 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.952 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:14:22.952 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:11.855244+0000", 2024-04-14T15:14:22.953 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:14:22.953 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.953 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.953 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:14:22.953 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:11.855245+0000", 2024-04-14T15:14:22.953 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1899999999999998e-07 2024-04-14T15:14:22.953 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.953 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.953 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:14:22.953 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:11.876856+0000", 2024-04-14T15:14:22.953 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.021610917 2024-04-14T15:14:22.953 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.953 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.953 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:14:22.953 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:11.876872+0000", 2024-04-14T15:14:22.954 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6345e-05 2024-04-14T15:14:22.954 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.954 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.954 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:14:22.954 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:11.876879+0000", 2024-04-14T15:14:22.954 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.4139999999999997e-06 2024-04-14T15:14:22.954 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.954 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.954 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:14:22.954 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:11.876933+0000", 2024-04-14T15:14:22.954 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3208000000000003e-05 2024-04-14T15:14:22.954 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.954 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.954 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:14:22.955 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:11.876949+0000", 2024-04-14T15:14:22.955 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6016999999999999e-05 2024-04-14T15:14:22.955 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.955 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.955 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:14:22.955 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:11.892596+0000", 2024-04-14T15:14:22.955 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.015647431 2024-04-14T15:14:22.955 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:14:22.955 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:14:22.955 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:14:22.955 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:11.892643+0000", 2024-04-14T15:14:22.955 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7488999999999999e-05 2024-04-14T15:14:22.955 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:14:22.955 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:14:22.955 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:14:22.956 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:14:22.956 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:14:22.956 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-04-14T15:14:22.956 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:14:22.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:22.980 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:22.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:14:23.081 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:14:23.101 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:22 smithi092 ceph-mon[33966]: pgmap v236: 270 pgs: 1 clean+premerge+peered, 7 active+clean+snaptrim, 32 unknown, 15 active+clean+snaptrim_wait, 215 active+clean; 559 KiB data, 534 MiB used, 715 GiB / 715 GiB avail; 339 KiB/s wr, 25 op/s; 380 KiB/s, 1 objects/s recovering 2024-04-14T15:14:23.101 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:22 smithi092 ceph-mon[33966]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:14:23.101 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:22 smithi092 ceph-mon[33966]: from='client.48370 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi090-93342-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi090-93342-28"}]': finished 2024-04-14T15:14:23.101 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:22 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "23"}]': finished 2024-04-14T15:14:23.101 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:22 smithi092 ceph-mon[33966]: osdmap e194: 8 total, 8 up, 8 in 2024-04-14T15:14:23.101 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:22 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/679904944' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi090-93363-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:23.101 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:22 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:23.101 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:22 smithi092 ceph-mon[33966]: from='client.48388 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi090-93363-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:23.101 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:22 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:23.101 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:22 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:23.101 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:22 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1108240145' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:14:23.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:23.106 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:23.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:14:23.137 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:22 smithi090 ceph-mon[41670]: pgmap v236: 270 pgs: 1 clean+premerge+peered, 7 active+clean+snaptrim, 32 unknown, 15 active+clean+snaptrim_wait, 215 active+clean; 559 KiB data, 534 MiB used, 715 GiB / 715 GiB avail; 339 KiB/s wr, 25 op/s; 380 KiB/s, 1 objects/s recovering 2024-04-14T15:14:23.137 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:22 smithi090 ceph-mon[41670]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:14:23.137 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:22 smithi090 ceph-mon[41670]: from='client.48370 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi090-93342-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi090-93342-28"}]': finished 2024-04-14T15:14:23.137 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:22 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "23"}]': finished 2024-04-14T15:14:23.137 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:22 smithi090 ceph-mon[41670]: osdmap e194: 8 total, 8 up, 8 in 2024-04-14T15:14:23.137 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:22 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/679904944' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi090-93363-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:23.137 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:22 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:23.137 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:22 smithi090 ceph-mon[41670]: from='client.48388 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi090-93363-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:23.138 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:22 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:23.138 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:22 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:23.138 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:22 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1108240145' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:14:23.138 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:22 smithi090 ceph-mon[30399]: pgmap v236: 270 pgs: 1 clean+premerge+peered, 7 active+clean+snaptrim, 32 unknown, 15 active+clean+snaptrim_wait, 215 active+clean; 559 KiB data, 534 MiB used, 715 GiB / 715 GiB avail; 339 KiB/s wr, 25 op/s; 380 KiB/s, 1 objects/s recovering 2024-04-14T15:14:23.138 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:22 smithi090 ceph-mon[30399]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:14:23.138 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:22 smithi090 ceph-mon[30399]: from='client.48370 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi090-93342-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi090-93342-28"}]': finished 2024-04-14T15:14:23.138 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:22 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "23"}]': finished 2024-04-14T15:14:23.138 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:22 smithi090 ceph-mon[30399]: osdmap e194: 8 total, 8 up, 8 in 2024-04-14T15:14:23.138 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:22 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/679904944' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi090-93363-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:23.138 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:22 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:23.138 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:22 smithi090 ceph-mon[30399]: from='client.48388 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi090-93363-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:23.138 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:22 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:23.139 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:22 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:23.139 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:22 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1108240145' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:14:23.208 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:14:23.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:23.232 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:23.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:14:23.333 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:14:23.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:23.359 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:23.359 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:14:23.460 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:23.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:23.485 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:23.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:23.586 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:14:23.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:23.610 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:23.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:14:23.710 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:23.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:23.735 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:23.735 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:23.814 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-14T15:14:23.836 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:23.909 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:23.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:23.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:24.011 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:14:24.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:24.036 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:24.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:14:24.058 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:14:23 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:14:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:14:24.059 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:23 smithi090 ceph-mon[41670]: from='client.48388 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi090-93363-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:24.059 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:23 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:24.059 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:23 smithi090 ceph-mon[41670]: osdmap e195: 8 total, 8 up, 8 in 2024-04-14T15:14:24.059 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:23 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/60043059' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi090-93601-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:24.059 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:23 smithi090 ceph-mon[41670]: from='client.46496 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi090-93601-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:24.059 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:23 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:24.059 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:23 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:14:24.059 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:23 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:14:24.059 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:23 smithi090 ceph-mon[41670]: from='client.46496 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi090-93601-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:24.059 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:23 smithi090 ceph-mon[41670]: osdmap e196: 8 total, 8 up, 8 in 2024-04-14T15:14:24.060 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:23 smithi090 ceph-mon[30399]: from='client.48388 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi090-93363-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:24.060 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:23 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:24.060 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:23 smithi090 ceph-mon[30399]: osdmap e195: 8 total, 8 up, 8 in 2024-04-14T15:14:24.060 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:23 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/60043059' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi090-93601-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:24.060 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:23 smithi090 ceph-mon[30399]: from='client.46496 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi090-93601-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:24.060 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:23 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:24.060 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:23 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:14:24.060 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:23 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:14:24.060 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:23 smithi090 ceph-mon[30399]: from='client.46496 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi090-93601-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:24.060 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:23 smithi090 ceph-mon[30399]: osdmap e196: 8 total, 8 up, 8 in 2024-04-14T15:14:24.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:23 smithi092 ceph-mon[33966]: from='client.48388 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi090-93363-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:24.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:23 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:24.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:23 smithi092 ceph-mon[33966]: osdmap e195: 8 total, 8 up, 8 in 2024-04-14T15:14:24.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:23 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/60043059' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi090-93601-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:24.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:23 smithi092 ceph-mon[33966]: from='client.46496 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi090-93601-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:24.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:23 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:24.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:23 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:14:24.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:23 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:14:24.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:23 smithi092 ceph-mon[33966]: from='client.46496 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi090-93601-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:24.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:23 smithi092 ceph-mon[33966]: osdmap e196: 8 total, 8 up, 8 in 2024-04-14T15:14:24.137 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:14:24.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:24.162 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:24.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:14:24.263 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:14:24.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:24.290 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:24.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:14:24.391 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:14:24.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:24.415 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:24.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:14:24.516 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:24.540 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:24.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:24.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:24.642 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:14:24.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:24.666 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:24.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:14:24.768 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:14:24.793 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:24.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:24.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:14:24.895 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:14:24.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:24.921 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:24.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:14:24.941 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:24 smithi090 ceph-mon[41670]: pgmap v239: 342 pgs: 1 clean+premerge+peered, 7 active+clean+snaptrim, 104 unknown, 15 active+clean+snaptrim_wait, 215 active+clean; 559 KiB data, 534 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s, 0 objects/s recovering 2024-04-14T15:14:24.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:24 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1487099185' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi090-93342-28"}]: dispatch 2024-04-14T15:14:24.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:24 smithi090 ceph-mon[41670]: from='client.48370 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi090-93342-28"}]: dispatch 2024-04-14T15:14:24.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:24 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:24.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:24 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:14:24.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:24 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:14:24.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:24 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:14:24.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:24 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:14:24.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:24 smithi090 ceph-mon[41670]: from='client.48370 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi090-93342-28"}]': finished 2024-04-14T15:14:24.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:24 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-15", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:14:24.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:24 smithi090 ceph-mon[41670]: osdmap e197: 8 total, 8 up, 8 in 2024-04-14T15:14:24.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:24 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-15"}]: dispatch 2024-04-14T15:14:24.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:24 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1487099185' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi090-93342-28"}]: dispatch 2024-04-14T15:14:24.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:24 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-15"}]: dispatch 2024-04-14T15:14:24.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:24 smithi090 ceph-mon[41670]: from='client.48370 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi090-93342-28"}]: dispatch 2024-04-14T15:14:24.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:24 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/747438472' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi090-93363-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:24.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:24 smithi090 ceph-mon[41670]: from='client.48406 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi090-93363-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:24.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:24 smithi090 ceph-mon[30399]: pgmap v239: 342 pgs: 1 clean+premerge+peered, 7 active+clean+snaptrim, 104 unknown, 15 active+clean+snaptrim_wait, 215 active+clean; 559 KiB data, 534 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s, 0 objects/s recovering 2024-04-14T15:14:24.944 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:24 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1487099185' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi090-93342-28"}]: dispatch 2024-04-14T15:14:24.944 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:24 smithi090 ceph-mon[30399]: from='client.48370 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi090-93342-28"}]: dispatch 2024-04-14T15:14:24.944 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:24 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:24.944 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:24 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:14:24.944 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:24 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:14:24.944 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:24 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:14:24.945 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:24 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:14:24.945 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:24 smithi090 ceph-mon[30399]: from='client.48370 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi090-93342-28"}]': finished 2024-04-14T15:14:24.945 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:24 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-15", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:14:24.945 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:24 smithi090 ceph-mon[30399]: osdmap e197: 8 total, 8 up, 8 in 2024-04-14T15:14:24.945 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:24 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-15"}]: dispatch 2024-04-14T15:14:24.945 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:24 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1487099185' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi090-93342-28"}]: dispatch 2024-04-14T15:14:24.945 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:24 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-15"}]: dispatch 2024-04-14T15:14:24.945 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:24 smithi090 ceph-mon[30399]: from='client.48370 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi090-93342-28"}]: dispatch 2024-04-14T15:14:24.945 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:24 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/747438472' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi090-93363-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:24.945 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:24 smithi090 ceph-mon[30399]: from='client.48406 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi090-93363-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:25.002 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:14:25.002 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:14:25.022 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:14:25.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:25.049 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:25.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:14:25.122 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:14:24 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:14:25.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:24 smithi092 ceph-mon[33966]: pgmap v239: 342 pgs: 1 clean+premerge+peered, 7 active+clean+snaptrim, 104 unknown, 15 active+clean+snaptrim_wait, 215 active+clean; 559 KiB data, 534 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s, 0 objects/s recovering 2024-04-14T15:14:25.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:24 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1487099185' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi090-93342-28"}]: dispatch 2024-04-14T15:14:25.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:24 smithi092 ceph-mon[33966]: from='client.48370 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi090-93342-28"}]: dispatch 2024-04-14T15:14:25.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:24 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:25.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:24 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:14:25.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:24 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:14:25.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:24 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:14:25.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:24 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:14:25.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:24 smithi092 ceph-mon[33966]: from='client.48370 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi090-93342-28"}]': finished 2024-04-14T15:14:25.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:24 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-15", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:14:25.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:24 smithi092 ceph-mon[33966]: osdmap e197: 8 total, 8 up, 8 in 2024-04-14T15:14:25.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:24 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-15"}]: dispatch 2024-04-14T15:14:25.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:24 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1487099185' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi090-93342-28"}]: dispatch 2024-04-14T15:14:25.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:24 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-15"}]: dispatch 2024-04-14T15:14:25.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:24 smithi092 ceph-mon[33966]: from='client.48370 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi090-93342-28"}]: dispatch 2024-04-14T15:14:25.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:24 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/747438472' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi090-93363-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:25.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:24 smithi092 ceph-mon[33966]: from='client.48406 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi090-93363-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:25.150 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:14:25.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:25.174 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:25.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:14:25.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.240+0000 7f3de213f700 1 -- 172.21.15.90:0/3629789691 >> v1:172.21.15.90:6789/0 conn(0x7f3ddc103a40 legacy=0x7f3ddc103e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:25.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.240+0000 7f3de213f700 1 -- 172.21.15.90:0/3629789691 shutdown_connections 2024-04-14T15:14:25.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.240+0000 7f3de213f700 1 -- 172.21.15.90:0/3629789691 >> 172.21.15.90:0/3629789691 conn(0x7f3ddc0fd2b0 msgr2=0x7f3ddc0ff6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:14:25.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.240+0000 7f3de213f700 1 -- 172.21.15.90:0/3629789691 shutdown_connections 2024-04-14T15:14:25.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.240+0000 7f3de213f700 1 -- 172.21.15.90:0/3629789691 wait complete. 2024-04-14T15:14:25.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.240+0000 7f3de213f700 1 librados: starting msgr at 2024-04-14T15:14:25.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.240+0000 7f3de213f700 1 librados: starting objecter 2024-04-14T15:14:25.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.241+0000 7f3de213f700 1 Processor -- start 2024-04-14T15:14:25.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.241+0000 7f3de213f700 1 -- start start 2024-04-14T15:14:25.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.241+0000 7f3de213f700 1 librados: setting wanted keys 2024-04-14T15:14:25.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.241+0000 7f3de213f700 1 librados: calling monclient init 2024-04-14T15:14:25.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.241+0000 7f3de213f700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3ddc1ae170 con 0x7f3ddc10f360 2024-04-14T15:14:25.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.241+0000 7f3de213f700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3ddc1af390 con 0x7f3ddc10b850 2024-04-14T15:14:25.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.241+0000 7f3de213f700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3ddc1b05d0 con 0x7f3ddc103a40 2024-04-14T15:14:25.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.241+0000 7f3ddb7fe700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f3ddc103a40 0x7f3ddc10b090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:47516/0 (socket says 172.21.15.90:47516) 2024-04-14T15:14:25.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.241+0000 7f3ddb7fe700 1 -- 172.21.15.90:0/1361902499 learned_addr learned my addr 172.21.15.90:0/1361902499 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:14:25.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.241+0000 7f3dd8ff9700 10 client.?.objecter ms_handle_connect 0x7f3ddc10f360 2024-04-14T15:14:25.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.241+0000 7f3dd8ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:14:25.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.241+0000 7f3dd8ff9700 10 client.?.objecter ms_handle_connect 0x7f3ddc10b850 2024-04-14T15:14:25.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.241+0000 7f3dd8ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:14:25.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.241+0000 7f3dd8ff9700 10 client.?.objecter ms_handle_connect 0x7f3ddc103a40 2024-04-14T15:14:25.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.241+0000 7f3dd8ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:14:25.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.242+0000 7f3dd8ff9700 1 -- 172.21.15.90:0/1361902499 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2768895745 0 0) 0x7f3ddc1af390 con 0x7f3ddc10b850 2024-04-14T15:14:25.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.242+0000 7f3dd8ff9700 1 -- 172.21.15.90:0/1361902499 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3dc40035b0 con 0x7f3ddc10b850 2024-04-14T15:14:25.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.242+0000 7f3dd8ff9700 1 -- 172.21.15.90:0/1361902499 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2157263311 0 0) 0x7f3ddc1ae170 con 0x7f3ddc10f360 2024-04-14T15:14:25.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.242+0000 7f3dd8ff9700 1 -- 172.21.15.90:0/1361902499 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3ddc1af390 con 0x7f3ddc10f360 2024-04-14T15:14:25.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.242+0000 7f3dd8ff9700 1 -- 172.21.15.90:0/1361902499 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1484085315 0 0) 0x7f3ddc1b05d0 con 0x7f3ddc103a40 2024-04-14T15:14:25.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.242+0000 7f3dd8ff9700 1 -- 172.21.15.90:0/1361902499 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3ddc1ae170 con 0x7f3ddc103a40 2024-04-14T15:14:25.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.242+0000 7f3dd8ff9700 1 -- 172.21.15.90:0/1361902499 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2676448795 0 0) 0x7f3dc40035b0 con 0x7f3ddc10b850 2024-04-14T15:14:25.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.242+0000 7f3dd8ff9700 1 -- 172.21.15.90:0/1361902499 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3ddc1b05d0 con 0x7f3ddc10b850 2024-04-14T15:14:25.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.242+0000 7f3dd8ff9700 1 -- 172.21.15.90:0/1361902499 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f3dd00017b0 con 0x7f3ddc10b850 2024-04-14T15:14:25.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.242+0000 7f3dd8ff9700 1 -- 172.21.15.90:0/1361902499 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3873670577 0 0) 0x7f3ddc1ae170 con 0x7f3ddc103a40 2024-04-14T15:14:25.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.242+0000 7f3dd8ff9700 1 -- 172.21.15.90:0/1361902499 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3dc40035b0 con 0x7f3ddc103a40 2024-04-14T15:14:25.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.242+0000 7f3dd8ff9700 1 -- 172.21.15.90:0/1361902499 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1221284519 0 0) 0x7f3ddc1af390 con 0x7f3ddc10f360 2024-04-14T15:14:25.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.242+0000 7f3dd8ff9700 1 -- 172.21.15.90:0/1361902499 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3ddc1ae170 con 0x7f3ddc10f360 2024-04-14T15:14:25.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.242+0000 7f3dd8ff9700 1 -- 172.21.15.90:0/1361902499 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f3dcc001430 con 0x7f3ddc103a40 2024-04-14T15:14:25.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.242+0000 7f3dd8ff9700 1 -- 172.21.15.90:0/1361902499 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f3dd4001bd0 con 0x7f3ddc10f360 2024-04-14T15:14:25.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.242+0000 7f3dd8ff9700 1 -- 172.21.15.90:0/1361902499 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1484130711 0 0) 0x7f3ddc1b05d0 con 0x7f3ddc10b850 2024-04-14T15:14:25.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.242+0000 7f3dd8ff9700 1 -- 172.21.15.90:0/1361902499 >> v1:172.21.15.90:6790/0 conn(0x7f3ddc103a40 legacy=0x7f3ddc10b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:25.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.242+0000 7f3dd8ff9700 1 -- 172.21.15.90:0/1361902499 >> v1:172.21.15.90:6789/0 conn(0x7f3ddc10f360 legacy=0x7f3ddc1ac830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:25.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.242+0000 7f3dd8ff9700 1 -- 172.21.15.90:0/1361902499 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ddc1b1810 con 0x7f3ddc10b850 2024-04-14T15:14:25.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.242+0000 7f3de213f700 1 -- 172.21.15.90:0/1361902499 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3ddc1af5c0 con 0x7f3ddc10b850 2024-04-14T15:14:25.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.242+0000 7f3de213f700 10 client.48418.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:14:25.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.242+0000 7f3dd8ff9700 1 -- 172.21.15.90:0/1361902499 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f3dd0003b40 con 0x7f3ddc10b850 2024-04-14T15:14:25.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.242+0000 7f3de213f700 1 -- 172.21.15.90:0/1361902499 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f3ddc1afa80 con 0x7f3ddc10b850 2024-04-14T15:14:25.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.242+0000 7f3de213f700 1 librados: init done 2024-04-14T15:14:25.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.242+0000 7f3dd8ff9700 1 -- 172.21.15.90:0/1361902499 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f3dd0004a20 con 0x7f3ddc10b850 2024-04-14T15:14:25.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.243+0000 7f3de213f700 1 -- 172.21.15.90:0/1361902499 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ddc06b730 con 0x7f3ddc10b850 2024-04-14T15:14:25.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.244+0000 7f3dd8ff9700 1 -- 172.21.15.90:0/1361902499 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f3dd00036a0 con 0x7f3ddc10b850 2024-04-14T15:14:25.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.244+0000 7f3dd8ff9700 1 -- 172.21.15.90:0/1361902499 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(197..197 src has 51..197) ==== 9822+0+0 (unknown 2377230797 0 0) 0x7f3dd0089ee0 con 0x7f3ddc10b850 2024-04-14T15:14:25.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.244+0000 7f3dd8ff9700 10 client.48418.objecter ms_dispatch 0x7f3ddc001550 osd_map(197..197 src has 51..197) 2024-04-14T15:14:25.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.244+0000 7f3dd8ff9700 3 client.48418.objecter handle_osd_map got epochs [197,197] > 0 2024-04-14T15:14:25.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.244+0000 7f3dd8ff9700 3 client.48418.objecter handle_osd_map decoding full epoch 197 2024-04-14T15:14:25.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.244+0000 7f3dd8ff9700 20 client.48418.objecter dump_active .. 0 homeless 2024-04-14T15:14:25.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.244+0000 7f3dd8ff9700 10 client.48418.objecter ms_handle_connect 0x7f3dc40714d0 2024-04-14T15:14:25.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.249+0000 7f3dd8ff9700 1 -- 172.21.15.90:0/1361902499 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f3dd0056270 con 0x7f3ddc10b850 2024-04-14T15:14:25.275 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:14:25.298 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:25.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:25.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:14:25.400 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:25.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:25.424 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:25.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:25.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.469+0000 7f3de213f700 1 -- 172.21.15.90:0/1361902499 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7f3ddc0693d0 con 0x7f3ddc10b850 2024-04-14T15:14:25.525 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:14:25.553 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:25.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:25.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:14:25.655 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:14:25.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:25.679 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:25.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:14:25.780 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:14:25.807 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:25.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:25.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:14:25.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.828+0000 7f3dd8ff9700 1 -- 172.21.15.90:0/1361902499 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v198) ==== 97+0+0 (unknown 2199682155 0 0) 0x7f3dd0059f20 con 0x7f3ddc10b850 2024-04-14T15:14:25.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.832+0000 7f3de213f700 10 librados: watch_flush enter 2024-04-14T15:14:25.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.832+0000 7f3de213f700 10 librados: watch_flush exit 2024-04-14T15:14:25.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.833+0000 7f3de213f700 20 client.48418.objecter shutdown clearing up homeless session... 2024-04-14T15:14:25.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.833+0000 7f3de213f700 10 client.48418.objecter successfully canceled tick 2024-04-14T15:14:25.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.834+0000 7f3de213f700 1 -- 172.21.15.90:0/1361902499 >> v1:172.21.15.90:6800/420935427 conn(0x7f3dc40714d0 legacy=0x7f3dc4073990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:25.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.834+0000 7f3de213f700 1 -- 172.21.15.90:0/1361902499 >> v1:172.21.15.92:6789/0 conn(0x7f3ddc10b850 legacy=0x7f3ddc1a8f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:25.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.835+0000 7f3de213f700 1 -- 172.21.15.90:0/1361902499 shutdown_connections 2024-04-14T15:14:25.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.835+0000 7f3de213f700 1 -- 172.21.15.90:0/1361902499 >> 172.21.15.90:0/1361902499 conn(0x7f3ddc0fd2b0 msgr2=0x7f3ddc0ff0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:14:25.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.835+0000 7f3de213f700 1 -- 172.21.15.90:0/1361902499 shutdown_connections 2024-04-14T15:14:25.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.835+0000 7f3de213f700 1 -- 172.21.15.90:0/1361902499 wait complete. 2024-04-14T15:14:25.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:25.835+0000 7f3de213f700 1 librados: shutdown 2024-04-14T15:14:25.838 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is set 2024-04-14T15:14:25.909 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:14:25.934 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:25.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:25.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:14:26.036 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:26.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:26.061 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:26.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:26.162 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:14:26.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:26.186 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:26.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:14:26.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:25 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3214396702' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi090-93601-36"}]: dispatch 2024-04-14T15:14:26.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:25 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3214396702' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi090-93601-36"}]: dispatch 2024-04-14T15:14:26.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:25 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3214396702' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi090-93601-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:14:26.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:25 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:26.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:25 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:14:26.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:25 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-04-14T15:14:26.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:25 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1361902499' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:14:26.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:25 smithi090 ceph-mon[41670]: from='client.48418 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:14:26.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:25 smithi090 ceph-mon[41670]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:14:26.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:25 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-15"}]': finished 2024-04-14T15:14:26.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:25 smithi090 ceph-mon[41670]: from='client.48370 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi090-93342-28"}]': finished 2024-04-14T15:14:26.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:25 smithi090 ceph-mon[41670]: from='client.48406 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi090-93363-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:26.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:25 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3214396702' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi090-93601-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:14:26.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:25 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "22"}]': finished 2024-04-14T15:14:26.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:25 smithi090 ceph-mon[41670]: from='client.48418 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-14T15:14:26.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:25 smithi090 ceph-mon[41670]: osdmap e198: 8 total, 8 up, 8 in 2024-04-14T15:14:26.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:25 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3214396702' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi090-93601-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi090-93601-36"}]: dispatch 2024-04-14T15:14:26.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:25 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-15", "mode": "writeback"}]: dispatch 2024-04-14T15:14:26.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:25 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1863705178' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi090-93724-21"}]: dispatch 2024-04-14T15:14:26.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:25 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-15", "mode": "writeback"}]: dispatch 2024-04-14T15:14:26.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:25 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3214396702' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi090-93601-36"}]: dispatch 2024-04-14T15:14:26.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:25 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3214396702' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi090-93601-36"}]: dispatch 2024-04-14T15:14:26.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:25 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3214396702' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi090-93601-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:14:26.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:25 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:26.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:25 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:14:26.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:25 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-04-14T15:14:26.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:25 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1361902499' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:14:26.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:25 smithi090 ceph-mon[30399]: from='client.48418 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:14:26.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:25 smithi090 ceph-mon[30399]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:14:26.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:25 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-15"}]': finished 2024-04-14T15:14:26.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:25 smithi090 ceph-mon[30399]: from='client.48370 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi090-93342-28"}]': finished 2024-04-14T15:14:26.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:25 smithi090 ceph-mon[30399]: from='client.48406 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi090-93363-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:26.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:25 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3214396702' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi090-93601-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:14:26.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:25 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "22"}]': finished 2024-04-14T15:14:26.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:25 smithi090 ceph-mon[30399]: from='client.48418 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-14T15:14:26.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:25 smithi090 ceph-mon[30399]: osdmap e198: 8 total, 8 up, 8 in 2024-04-14T15:14:26.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:25 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3214396702' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi090-93601-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi090-93601-36"}]: dispatch 2024-04-14T15:14:26.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:25 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-15", "mode": "writeback"}]: dispatch 2024-04-14T15:14:26.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:25 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1863705178' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi090-93724-21"}]: dispatch 2024-04-14T15:14:26.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:25 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-15", "mode": "writeback"}]: dispatch 2024-04-14T15:14:26.287 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:14:26.315 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:26.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:26.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:14:26.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:25 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3214396702' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi090-93601-36"}]: dispatch 2024-04-14T15:14:26.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:25 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3214396702' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi090-93601-36"}]: dispatch 2024-04-14T15:14:26.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:25 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3214396702' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi090-93601-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:14:26.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:25 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:26.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:25 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:14:26.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:25 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-04-14T15:14:26.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:25 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1361902499' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:14:26.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:25 smithi092 ceph-mon[33966]: from='client.48418 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:14:26.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:25 smithi092 ceph-mon[33966]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:14:26.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:25 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-15"}]': finished 2024-04-14T15:14:26.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:25 smithi092 ceph-mon[33966]: from='client.48370 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi090-93342-28"}]': finished 2024-04-14T15:14:26.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:25 smithi092 ceph-mon[33966]: from='client.48406 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi090-93363-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:26.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:25 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3214396702' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi090-93601-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:14:26.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:25 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "22"}]': finished 2024-04-14T15:14:26.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:25 smithi092 ceph-mon[33966]: from='client.48418 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-14T15:14:26.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:25 smithi092 ceph-mon[33966]: osdmap e198: 8 total, 8 up, 8 in 2024-04-14T15:14:26.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:25 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3214396702' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi090-93601-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi090-93601-36"}]: dispatch 2024-04-14T15:14:26.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:25 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-15", "mode": "writeback"}]: dispatch 2024-04-14T15:14:26.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:25 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1863705178' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi090-93724-21"}]: dispatch 2024-04-14T15:14:26.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:25 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-15", "mode": "writeback"}]: dispatch 2024-04-14T15:14:26.417 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:14:26.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:26.441 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:26.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:14:26.542 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:14:26.566 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:26.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:26.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:14:26.668 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:14:26.695 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:26.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:26.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:14:26.714 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-14T15:14:26.797 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:14:26.825 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:26.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:26.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:14:26.927 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:14:26.952 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:26.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:26.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:14:27.026 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:26 smithi090 ceph-mon[30399]: pgmap v242: 301 pgs: 1 peering, 8 active+clean+snaptrim, 32 unknown, 14 active+clean+snaptrim_wait, 246 active+clean; 567 KiB data, 528 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.7 KiB/s wr, 2 op/s; 0 B/s, 0 objects/s recovering 2024-04-14T15:14:27.026 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:26 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2359757960' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi090-93342-29"}]: dispatch 2024-04-14T15:14:27.026 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:26 smithi090 ceph-mon[30399]: from='client.48424 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi090-93342-29"}]: dispatch 2024-04-14T15:14:27.026 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:26 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2359757960' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi090-93342-29"}]: dispatch 2024-04-14T15:14:27.026 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:26 smithi090 ceph-mon[30399]: from='client.48424 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi090-93342-29"}]: dispatch 2024-04-14T15:14:27.026 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:26 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2359757960' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi090-93342-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:14:27.027 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:26 smithi090 ceph-mon[30399]: from='client.48424 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi090-93342-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:14:27.027 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:26 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:27.027 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:26 smithi090 ceph-mon[30399]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:14:27.027 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:26 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1863705178' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi090-93724-21"}]': finished 2024-04-14T15:14:27.027 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:26 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-15", "mode": "writeback"}]': finished 2024-04-14T15:14:27.027 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:26 smithi090 ceph-mon[30399]: from='client.48424 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi090-93342-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:14:27.027 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:26 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2359757960' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi090-93342-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi090-93342-29"}]: dispatch 2024-04-14T15:14:27.027 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:26 smithi090 ceph-mon[30399]: osdmap e199: 8 total, 8 up, 8 in 2024-04-14T15:14:27.027 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:26 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1863705178' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi090-93724-21"}]: dispatch 2024-04-14T15:14:27.027 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:26 smithi090 ceph-mon[30399]: from='client.48424 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi090-93342-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi090-93342-29"}]: dispatch 2024-04-14T15:14:27.027 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:26 smithi090 ceph-mon[41670]: pgmap v242: 301 pgs: 1 peering, 8 active+clean+snaptrim, 32 unknown, 14 active+clean+snaptrim_wait, 246 active+clean; 567 KiB data, 528 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.7 KiB/s wr, 2 op/s; 0 B/s, 0 objects/s recovering 2024-04-14T15:14:27.027 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:26 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2359757960' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi090-93342-29"}]: dispatch 2024-04-14T15:14:27.028 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:26 smithi090 ceph-mon[41670]: from='client.48424 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi090-93342-29"}]: dispatch 2024-04-14T15:14:27.028 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:26 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2359757960' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi090-93342-29"}]: dispatch 2024-04-14T15:14:27.028 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:26 smithi090 ceph-mon[41670]: from='client.48424 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi090-93342-29"}]: dispatch 2024-04-14T15:14:27.028 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:26 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2359757960' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi090-93342-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:14:27.028 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:26 smithi090 ceph-mon[41670]: from='client.48424 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi090-93342-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:14:27.028 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:26 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:27.028 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:26 smithi090 ceph-mon[41670]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:14:27.028 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:26 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1863705178' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi090-93724-21"}]': finished 2024-04-14T15:14:27.028 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:26 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-15", "mode": "writeback"}]': finished 2024-04-14T15:14:27.028 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:26 smithi090 ceph-mon[41670]: from='client.48424 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi090-93342-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:14:27.028 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:26 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2359757960' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi090-93342-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi090-93342-29"}]: dispatch 2024-04-14T15:14:27.028 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:26 smithi090 ceph-mon[41670]: osdmap e199: 8 total, 8 up, 8 in 2024-04-14T15:14:27.028 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:26 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1863705178' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi090-93724-21"}]: dispatch 2024-04-14T15:14:27.029 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:26 smithi090 ceph-mon[41670]: from='client.48424 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi090-93342-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi090-93342-29"}]: dispatch 2024-04-14T15:14:27.054 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:14:27.079 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:27.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:27.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:14:27.180 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:14:27.205 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:27.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:27.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:14:27.307 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:14:27.331 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:27.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:27.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:14:27.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:26 smithi092 ceph-mon[33966]: pgmap v242: 301 pgs: 1 peering, 8 active+clean+snaptrim, 32 unknown, 14 active+clean+snaptrim_wait, 246 active+clean; 567 KiB data, 528 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.7 KiB/s wr, 2 op/s; 0 B/s, 0 objects/s recovering 2024-04-14T15:14:27.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:26 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2359757960' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi090-93342-29"}]: dispatch 2024-04-14T15:14:27.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:26 smithi092 ceph-mon[33966]: from='client.48424 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi090-93342-29"}]: dispatch 2024-04-14T15:14:27.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:26 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2359757960' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi090-93342-29"}]: dispatch 2024-04-14T15:14:27.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:26 smithi092 ceph-mon[33966]: from='client.48424 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi090-93342-29"}]: dispatch 2024-04-14T15:14:27.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:26 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2359757960' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi090-93342-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:14:27.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:26 smithi092 ceph-mon[33966]: from='client.48424 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi090-93342-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:14:27.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:26 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:27.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:26 smithi092 ceph-mon[33966]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:14:27.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:26 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1863705178' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi090-93724-21"}]': finished 2024-04-14T15:14:27.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:26 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-15", "mode": "writeback"}]': finished 2024-04-14T15:14:27.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:26 smithi092 ceph-mon[33966]: from='client.48424 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi090-93342-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:14:27.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:26 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2359757960' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi090-93342-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi090-93342-29"}]: dispatch 2024-04-14T15:14:27.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:26 smithi092 ceph-mon[33966]: osdmap e199: 8 total, 8 up, 8 in 2024-04-14T15:14:27.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:26 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1863705178' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi090-93724-21"}]: dispatch 2024-04-14T15:14:27.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:26 smithi092 ceph-mon[33966]: from='client.48424 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi090-93342-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi090-93342-29"}]: dispatch 2024-04-14T15:14:27.434 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:14:27.458 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:27.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:27.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:14:27.560 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:14:27.585 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:27.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:27.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:14:27.687 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:27.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:27.715 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:27.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:27.816 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:14:27.853 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:27.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:27.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:14:27.854 INFO:tasks.workunit.client.0.smithi090.stdout: hotsECPP.SnapRemovePP (2000 ms) 2024-04-14T15:14:27.854 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2024-04-14T15:14:27.855 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (2007 ms) 2024-04-14T15:14:27.855 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2024-04-14T15:14:27.855 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (2005 ms) 2024-04-14T15:14:27.855 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (9010 ms total) 2024-04-14T15:14:27.855 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: 2024-04-14T15:14:27.855 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2024-04-14T15:14:27.855 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.SnapPP 2024-04-14T15:14:27.855 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (4030 ms) 2024-04-14T15:14:27.855 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2024-04-14T15:14:27.855 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (3989 ms) 2024-04-14T15:14:27.855 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2024-04-14T15:14:27.855 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (3997 ms) 2024-04-14T15:14:27.855 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (12016 ms total) 2024-04-14T15:14:27.855 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: 2024-04-14T15:14:27.856 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2024-04-14T15:14:27.856 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: [==========] 21 tests from 5 test suites ran. (93593 ms total) 2024-04-14T15:14:27.856 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: [ PASSED ] 20 tests. 2024-04-14T15:14:27.856 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: [ SKIPPED ] 1 test, listed below: 2024-04-14T15:14:27.856 INFO:tasks.workunit.client.0.smithi090.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-04-14T15:14:27.955 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:14:27.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:27.979 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:27.980 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:14:28.080 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:14:28.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:28.106 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:28.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:14:28.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:27 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:28.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:27 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3214396702' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi090-93601-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi090-93601-36"}]': finished 2024-04-14T15:14:28.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:27 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1863705178' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi090-93724-21"}]': finished 2024-04-14T15:14:28.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:27 smithi090 ceph-mon[41670]: osdmap e200: 8 total, 8 up, 8 in 2024-04-14T15:14:28.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:27 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1479183444' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi090-93363-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:28.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:27 smithi090 ceph-mon[41670]: from='client.48430 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi090-93363-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:28.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:27 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:14:28.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:27 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:14:28.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:27 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:28.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:27 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3214396702' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi090-93601-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi090-93601-36"}]': finished 2024-04-14T15:14:28.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:27 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1863705178' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi090-93724-21"}]': finished 2024-04-14T15:14:28.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:27 smithi090 ceph-mon[30399]: osdmap e200: 8 total, 8 up, 8 in 2024-04-14T15:14:28.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:27 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1479183444' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi090-93363-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:28.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:27 smithi090 ceph-mon[30399]: from='client.48430 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi090-93363-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:28.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:27 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:14:28.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:27 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:14:28.208 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:14:28.232 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:28.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:28.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:14:28.334 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:14:28.359 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:28.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:28.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:14:28.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:27 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:28.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:27 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3214396702' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi090-93601-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi090-93601-36"}]': finished 2024-04-14T15:14:28.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:27 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1863705178' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi090-93724-21"}]': finished 2024-04-14T15:14:28.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:27 smithi092 ceph-mon[33966]: osdmap e200: 8 total, 8 up, 8 in 2024-04-14T15:14:28.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:27 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1479183444' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi090-93363-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:28.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:27 smithi092 ceph-mon[33966]: from='client.48430 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi090-93363-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:28.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:27 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:14:28.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:27 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:14:28.461 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:28.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:28.487 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:28.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:28.590 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:14:28.617 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:28.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:28.618 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:14:28.719 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:14:28.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:28.747 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:28.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:14:28.848 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:14:28.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:28.882 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:28.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:14:28.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:28.978+0000 7f6c6fa33700 1 -- 172.21.15.90:0/3190194607 >> v1:172.21.15.90:6789/0 conn(0x7f6c6810e1f0 legacy=0x7f6c681106b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:28.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:28.978+0000 7f6c6fa33700 1 -- 172.21.15.90:0/3190194607 shutdown_connections 2024-04-14T15:14:28.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:28.978+0000 7f6c6fa33700 1 -- 172.21.15.90:0/3190194607 >> 172.21.15.90:0/3190194607 conn(0x7f6c680fd280 msgr2=0x7f6c680ff6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:14:28.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:28.978+0000 7f6c6fa33700 1 -- 172.21.15.90:0/3190194607 shutdown_connections 2024-04-14T15:14:28.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:28.978+0000 7f6c6fa33700 1 -- 172.21.15.90:0/3190194607 wait complete. 2024-04-14T15:14:28.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:28.978+0000 7f6c6fa33700 1 librados: starting msgr at 2024-04-14T15:14:28.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:28.978+0000 7f6c6fa33700 1 librados: starting objecter 2024-04-14T15:14:28.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:28.978+0000 7f6c6fa33700 1 Processor -- start 2024-04-14T15:14:28.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:28.978+0000 7f6c6fa33700 1 -- start start 2024-04-14T15:14:28.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:28.978+0000 7f6c6fa33700 1 librados: setting wanted keys 2024-04-14T15:14:28.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:28.978+0000 7f6c6fa33700 1 librados: calling monclient init 2024-04-14T15:14:28.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:28.979+0000 7f6c6fa33700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6c681a7e00 con 0x7f6c6810a620 2024-04-14T15:14:28.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:28.979+0000 7f6c6fa33700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6c681a8fe0 con 0x7f6c6810e1f0 2024-04-14T15:14:28.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:28.979+0000 7f6c6fa33700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6c681aa1c0 con 0x7f6c68107770 2024-04-14T15:14:28.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:28.979+0000 7f6c6cfce700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f6c6810a620 0x7f6c6810d520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:40280/0 (socket says 172.21.15.90:40280) 2024-04-14T15:14:28.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:28.979+0000 7f6c6cfce700 1 -- 172.21.15.90:0/1260749702 learned_addr learned my addr 172.21.15.90:0/1260749702 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:14:28.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:28.979+0000 7f6c5e7fc700 10 client.?.objecter ms_handle_connect 0x7f6c6810a620 2024-04-14T15:14:28.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:28.979+0000 7f6c5e7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:14:28.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:28.979+0000 7f6c5e7fc700 10 client.?.objecter ms_handle_connect 0x7f6c68107770 2024-04-14T15:14:28.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:28.979+0000 7f6c5e7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:14:28.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:28.979+0000 7f6c5e7fc700 10 client.?.objecter ms_handle_connect 0x7f6c6810e1f0 2024-04-14T15:14:28.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:28.979+0000 7f6c5e7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:14:28.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:28.979+0000 7f6c5e7fc700 1 -- 172.21.15.90:0/1260749702 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1990746979 0 0) 0x7f6c681a7e00 con 0x7f6c6810a620 2024-04-14T15:14:28.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:28.979+0000 7f6c5e7fc700 1 -- 172.21.15.90:0/1260749702 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c540035b0 con 0x7f6c6810a620 2024-04-14T15:14:28.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:28.979+0000 7f6c5e7fc700 1 -- 172.21.15.90:0/1260749702 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1195041065 0 0) 0x7f6c681a8fe0 con 0x7f6c6810e1f0 2024-04-14T15:14:28.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:28.979+0000 7f6c5e7fc700 1 -- 172.21.15.90:0/1260749702 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c681a7e00 con 0x7f6c6810e1f0 2024-04-14T15:14:28.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:28.979+0000 7f6c5e7fc700 1 -- 172.21.15.90:0/1260749702 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 790877284 0 0) 0x7f6c681aa1c0 con 0x7f6c68107770 2024-04-14T15:14:28.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:28.979+0000 7f6c5e7fc700 1 -- 172.21.15.90:0/1260749702 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c681a8fe0 con 0x7f6c68107770 2024-04-14T15:14:28.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:28.979+0000 7f6c5e7fc700 1 -- 172.21.15.90:0/1260749702 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1919472439 0 0) 0x7f6c681a7e00 con 0x7f6c6810e1f0 2024-04-14T15:14:28.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:28.979+0000 7f6c5e7fc700 1 -- 172.21.15.90:0/1260749702 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6c681aa1c0 con 0x7f6c6810e1f0 2024-04-14T15:14:28.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:28.979+0000 7f6c5e7fc700 1 -- 172.21.15.90:0/1260749702 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f6c600017b0 con 0x7f6c6810e1f0 2024-04-14T15:14:28.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:28.979+0000 7f6c5e7fc700 1 -- 172.21.15.90:0/1260749702 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3178514671 0 0) 0x7f6c540035b0 con 0x7f6c6810a620 2024-04-14T15:14:28.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:28.979+0000 7f6c5e7fc700 1 -- 172.21.15.90:0/1260749702 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6c681a7e00 con 0x7f6c6810a620 2024-04-14T15:14:28.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:28.979+0000 7f6c5e7fc700 1 -- 172.21.15.90:0/1260749702 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f6c580018b0 con 0x7f6c6810a620 2024-04-14T15:14:28.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:28.979+0000 7f6c5e7fc700 1 -- 172.21.15.90:0/1260749702 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 113293629 0 0) 0x7f6c681a8fe0 con 0x7f6c68107770 2024-04-14T15:14:28.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:28.979+0000 7f6c5e7fc700 1 -- 172.21.15.90:0/1260749702 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6c540035b0 con 0x7f6c68107770 2024-04-14T15:14:28.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:28.979+0000 7f6c5e7fc700 1 -- 172.21.15.90:0/1260749702 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f6c640017f0 con 0x7f6c68107770 2024-04-14T15:14:28.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:28.980+0000 7f6c5e7fc700 1 -- 172.21.15.90:0/1260749702 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 927613184 0 0) 0x7f6c681aa1c0 con 0x7f6c6810e1f0 2024-04-14T15:14:28.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:28.980+0000 7f6c5e7fc700 1 -- 172.21.15.90:0/1260749702 >> v1:172.21.15.90:6790/0 conn(0x7f6c68107770 legacy=0x7f6c6810ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:28.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:28.980+0000 7f6c5e7fc700 1 -- 172.21.15.90:0/1260749702 >> v1:172.21.15.90:6789/0 conn(0x7f6c6810a620 legacy=0x7f6c6810d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:28.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:28.980+0000 7f6c5e7fc700 1 -- 172.21.15.90:0/1260749702 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c681ab3a0 con 0x7f6c6810e1f0 2024-04-14T15:14:28.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:28.980+0000 7f6c6fa33700 1 -- 172.21.15.90:0/1260749702 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6c681aa3f0 con 0x7f6c6810e1f0 2024-04-14T15:14:28.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:28.980+0000 7f6c6fa33700 10 client.48436.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:14:28.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:28.980+0000 7f6c6fa33700 1 -- 172.21.15.90:0/1260749702 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f6c681aa8b0 con 0x7f6c6810e1f0 2024-04-14T15:14:28.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:28.980+0000 7f6c6fa33700 1 librados: init done 2024-04-14T15:14:28.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:28.980+0000 7f6c5e7fc700 1 -- 172.21.15.90:0/1260749702 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f6c60003a00 con 0x7f6c6810e1f0 2024-04-14T15:14:28.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:28.980+0000 7f6c5e7fc700 1 -- 172.21.15.90:0/1260749702 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f6c60005110 con 0x7f6c6810e1f0 2024-04-14T15:14:28.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:28.981+0000 7f6c6fa33700 1 -- 172.21.15.90:0/1260749702 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c4c005150 con 0x7f6c6810e1f0 2024-04-14T15:14:28.984 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:14:28.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:28.981+0000 7f6c5e7fc700 1 -- 172.21.15.90:0/1260749702 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f6c60001ef0 con 0x7f6c6810e1f0 2024-04-14T15:14:28.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:28.982+0000 7f6c5e7fc700 1 -- 172.21.15.90:0/1260749702 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(201..201 src has 51..201) ==== 10502+0+0 (unknown 1628222708 0 0) 0x7f6c6008a550 con 0x7f6c6810e1f0 2024-04-14T15:14:28.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:28.982+0000 7f6c5e7fc700 10 client.48436.objecter ms_dispatch 0x7f6c68001550 osd_map(201..201 src has 51..201) 2024-04-14T15:14:28.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:28.982+0000 7f6c5e7fc700 3 client.48436.objecter handle_osd_map got epochs [201,201] > 0 2024-04-14T15:14:28.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:28.982+0000 7f6c5e7fc700 3 client.48436.objecter handle_osd_map decoding full epoch 201 2024-04-14T15:14:28.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:28.982+0000 7f6c5e7fc700 20 client.48436.objecter dump_active .. 0 homeless 2024-04-14T15:14:28.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:28.982+0000 7f6c5e7fc700 10 client.48436.objecter ms_handle_connect 0x7f6c54071430 2024-04-14T15:14:28.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:28.986+0000 7f6c5e7fc700 1 -- 172.21.15.90:0/1260749702 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f6c60056640 con 0x7f6c6810e1f0 2024-04-14T15:14:29.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:29.011 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:29.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:14:29.104 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:28 smithi090 ceph-mon[30399]: pgmap v245: 261 pgs: 1 peering, 8 active+clean+snaptrim, 9 active+clean+snaptrim_wait, 243 active+clean; 535 KiB data, 528 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-04-14T15:14:29.104 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:28 smithi090 ceph-mon[30399]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:14:29.104 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:28 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:29.104 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:28 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:14:29.104 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:28 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:14:29.104 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:28 smithi090 ceph-mon[30399]: from='client.48424 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi090-93342-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi090-93342-29"}]': finished 2024-04-14T15:14:29.104 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:28 smithi090 ceph-mon[30399]: from='client.48430 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi090-93363-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:29.104 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:28 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:14:29.105 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:28 smithi090 ceph-mon[30399]: osdmap e201: 8 total, 8 up, 8 in 2024-04-14T15:14:29.105 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:28 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-15"}]: dispatch 2024-04-14T15:14:29.105 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:28 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-15"}]: dispatch 2024-04-14T15:14:29.105 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:28 smithi090 ceph-mon[41670]: pgmap v245: 261 pgs: 1 peering, 8 active+clean+snaptrim, 9 active+clean+snaptrim_wait, 243 active+clean; 535 KiB data, 528 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-04-14T15:14:29.105 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:28 smithi090 ceph-mon[41670]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:14:29.105 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:28 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:29.105 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:28 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:14:29.105 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:28 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:14:29.105 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:28 smithi090 ceph-mon[41670]: from='client.48424 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi090-93342-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi090-93342-29"}]': finished 2024-04-14T15:14:29.105 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:28 smithi090 ceph-mon[41670]: from='client.48430 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi090-93363-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:29.105 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:28 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:14:29.105 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:28 smithi090 ceph-mon[41670]: osdmap e201: 8 total, 8 up, 8 in 2024-04-14T15:14:29.106 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:28 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-15"}]: dispatch 2024-04-14T15:14:29.106 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:28 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-15"}]: dispatch 2024-04-14T15:14:29.113 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:29.138 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:29.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:29.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:29.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:29.211+0000 7f6c6fa33700 1 -- 172.21.15.90:0/1260749702 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd rm-pg-upmap-items", "pgid": "2.1c"} v 0) -- 0x7f6c4c005440 con 0x7f6c6810e1f0 2024-04-14T15:14:29.237 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-14T15:14:29.237 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-14T15:14:29.237 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-14T15:14:29.238 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-14T15:14:29.240 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:29.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:29.266 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:29.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:29.367 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:14:29.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:28 smithi092 ceph-mon[33966]: pgmap v245: 261 pgs: 1 peering, 8 active+clean+snaptrim, 9 active+clean+snaptrim_wait, 243 active+clean; 535 KiB data, 528 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-04-14T15:14:29.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:28 smithi092 ceph-mon[33966]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:14:29.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:28 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:29.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:28 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:14:29.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:28 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:14:29.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:28 smithi092 ceph-mon[33966]: from='client.48424 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi090-93342-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi090-93342-29"}]': finished 2024-04-14T15:14:29.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:28 smithi092 ceph-mon[33966]: from='client.48430 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi090-93363-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:29.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:28 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:14:29.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:28 smithi092 ceph-mon[33966]: osdmap e201: 8 total, 8 up, 8 in 2024-04-14T15:14:29.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:28 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-15"}]: dispatch 2024-04-14T15:14:29.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:28 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-15"}]: dispatch 2024-04-14T15:14:29.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:29.392 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:29.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:14:29.429 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:14:29.493 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:14:29.520 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:29.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:29.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:14:29.622 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:29.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:29.647 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:29.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:29.748 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:29.770 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd unset noscrub 2024-04-14T15:14:29.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:29.773 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:29.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:29.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:29.837+0000 7f6c5e7fc700 1 -- 172.21.15.90:0/1260749702 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd rm-pg-upmap-items", "pgid": "2.1c"}]=0 clear 2.1c pg_upmap_items mapping v202) ==== 118+0+0 (unknown 768709033 0 0) 0x7f6c6005a2f0 con 0x7f6c6810e1f0 2024-04-14T15:14:29.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:29.841+0000 7f6c6fa33700 10 librados: watch_flush enter 2024-04-14T15:14:29.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:29.841+0000 7f6c6fa33700 10 librados: watch_flush exit 2024-04-14T15:14:29.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:29.841+0000 7f6c6fa33700 20 client.48436.objecter shutdown clearing up homeless session... 2024-04-14T15:14:29.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:29.841+0000 7f6c6fa33700 10 client.48436.objecter successfully canceled tick 2024-04-14T15:14:29.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:29.841+0000 7f6c6fa33700 1 -- 172.21.15.90:0/1260749702 >> v1:172.21.15.90:6800/420935427 conn(0x7f6c54071430 legacy=0x7f6c540738f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:29.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:29.841+0000 7f6c6fa33700 1 -- 172.21.15.90:0/1260749702 >> v1:172.21.15.92:6789/0 conn(0x7f6c6810e1f0 legacy=0x7f6c681a66e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:29.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:29.841+0000 7f6c6fa33700 1 -- 172.21.15.90:0/1260749702 shutdown_connections 2024-04-14T15:14:29.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:29.841+0000 7f6c6fa33700 1 -- 172.21.15.90:0/1260749702 >> 172.21.15.90:0/1260749702 conn(0x7f6c680fd280 msgr2=0x7f6c68111770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:14:29.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:29.842+0000 7f6c6fa33700 1 -- 172.21.15.90:0/1260749702 shutdown_connections 2024-04-14T15:14:29.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:29.842+0000 7f6c6fa33700 1 -- 172.21.15.90:0/1260749702 wait complete. 2024-04-14T15:14:29.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:29.843+0000 7f6c6fa33700 1 librados: shutdown 2024-04-14T15:14:29.847 INFO:teuthology.orchestra.run.smithi090.stderr:clear 2.1c pg_upmap_items mapping 2024-04-14T15:14:29.875 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:14:29.908 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:29.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:29.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:14:29.963 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-14T15:14:30.012 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:14:30.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:30.036 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:30.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:14:30.137 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:14:30.163 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:30.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:30.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:14:30.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:29 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:30.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:29 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-04-14T15:14:30.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:29 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1260749702' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "2.1c"}]: dispatch 2024-04-14T15:14:30.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:29 smithi090 ceph-mon[41670]: from='client.48436 ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "2.1c"}]: dispatch 2024-04-14T15:14:30.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:29 smithi090 ceph-mon[41670]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:14:30.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:29 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-15"}]': finished 2024-04-14T15:14:30.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:29 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "21"}]': finished 2024-04-14T15:14:30.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:29 smithi090 ceph-mon[41670]: from='client.48436 ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "2.1c"}]': finished 2024-04-14T15:14:30.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:29 smithi090 ceph-mon[41670]: osdmap e202: 8 total, 8 up, 8 in 2024-04-14T15:14:30.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:29 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3214396702' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi090-93601-36"}]: dispatch 2024-04-14T15:14:30.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:29 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:30.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:29 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-04-14T15:14:30.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:29 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1260749702' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "2.1c"}]: dispatch 2024-04-14T15:14:30.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:29 smithi090 ceph-mon[30399]: from='client.48436 ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "2.1c"}]: dispatch 2024-04-14T15:14:30.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:29 smithi090 ceph-mon[30399]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:14:30.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:29 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-15"}]': finished 2024-04-14T15:14:30.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:29 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "21"}]': finished 2024-04-14T15:14:30.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:29 smithi090 ceph-mon[30399]: from='client.48436 ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "2.1c"}]': finished 2024-04-14T15:14:30.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:29 smithi090 ceph-mon[30399]: osdmap e202: 8 total, 8 up, 8 in 2024-04-14T15:14:30.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:29 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3214396702' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi090-93601-36"}]: dispatch 2024-04-14T15:14:30.265 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:14:30.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:30.291 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:30.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:14:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:29 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:29 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-04-14T15:14:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:29 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1260749702' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "2.1c"}]: dispatch 2024-04-14T15:14:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:29 smithi092 ceph-mon[33966]: from='client.48436 ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "2.1c"}]: dispatch 2024-04-14T15:14:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:29 smithi092 ceph-mon[33966]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:14:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:29 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-15"}]': finished 2024-04-14T15:14:30.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:29 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "21"}]': finished 2024-04-14T15:14:30.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:29 smithi092 ceph-mon[33966]: from='client.48436 ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "2.1c"}]': finished 2024-04-14T15:14:30.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:29 smithi092 ceph-mon[33966]: osdmap e202: 8 total, 8 up, 8 in 2024-04-14T15:14:30.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:29 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3214396702' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi090-93601-36"}]: dispatch 2024-04-14T15:14:30.392 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:14:30.417 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:30.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:30.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:14:30.519 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:30.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:30.544 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:30.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:30.645 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:14:30.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:30.670 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:30.670 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:14:30.770 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:14:30.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:30.798 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:30.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:14:30.900 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:14:30.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:30.927 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:30.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:14:31.027 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:14:31.051 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:31.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:31.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:14:31.153 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:14:31.203 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:30 smithi090 ceph-mon[41670]: pgmap v248: 308 pgs: 3 creating+peering, 8 active+clean+snaptrim, 23 unknown, 19 active+clean+snaptrim_wait, 255 active+clean; 535 KiB data, 529 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:14:31.203 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:30 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:31.203 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:30 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3214396702' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi090-93601-36"}]': finished 2024-04-14T15:14:31.203 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:30 smithi090 ceph-mon[41670]: osdmap e203: 8 total, 8 up, 8 in 2024-04-14T15:14:31.203 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:30 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3214396702' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi090-93601-36"}]: dispatch 2024-04-14T15:14:31.204 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:30 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2359757960' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi090-93342-29"}]: dispatch 2024-04-14T15:14:31.204 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:30 smithi090 ceph-mon[41670]: from='client.48424 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi090-93342-29"}]: dispatch 2024-04-14T15:14:31.204 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:30 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1708340624' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi090-93363-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:31.204 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:30 smithi090 ceph-mon[41670]: from='client.48442 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi090-93363-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:31.204 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:30 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:31.204 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:30 smithi090 ceph-mon[30399]: pgmap v248: 308 pgs: 3 creating+peering, 8 active+clean+snaptrim, 23 unknown, 19 active+clean+snaptrim_wait, 255 active+clean; 535 KiB data, 529 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:14:31.204 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:30 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:31.204 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:30 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3214396702' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi090-93601-36"}]': finished 2024-04-14T15:14:31.204 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:30 smithi090 ceph-mon[30399]: osdmap e203: 8 total, 8 up, 8 in 2024-04-14T15:14:31.204 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:30 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3214396702' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi090-93601-36"}]: dispatch 2024-04-14T15:14:31.204 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:30 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2359757960' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi090-93342-29"}]: dispatch 2024-04-14T15:14:31.205 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:30 smithi090 ceph-mon[30399]: from='client.48424 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi090-93342-29"}]: dispatch 2024-04-14T15:14:31.205 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:30 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1708340624' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi090-93363-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:31.205 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:30 smithi090 ceph-mon[30399]: from='client.48442 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi090-93363-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:31.205 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:30 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:31.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:31.231 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:31.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:14:31.332 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:14:31.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:31.362 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:31.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:14:31.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:30 smithi092 ceph-mon[33966]: pgmap v248: 308 pgs: 3 creating+peering, 8 active+clean+snaptrim, 23 unknown, 19 active+clean+snaptrim_wait, 255 active+clean; 535 KiB data, 529 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:14:31.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:30 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:31.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:30 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3214396702' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi090-93601-36"}]': finished 2024-04-14T15:14:31.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:30 smithi092 ceph-mon[33966]: osdmap e203: 8 total, 8 up, 8 in 2024-04-14T15:14:31.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:30 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3214396702' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi090-93601-36"}]: dispatch 2024-04-14T15:14:31.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:30 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2359757960' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi090-93342-29"}]: dispatch 2024-04-14T15:14:31.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:30 smithi092 ceph-mon[33966]: from='client.48424 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi090-93342-29"}]: dispatch 2024-04-14T15:14:31.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:30 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1708340624' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi090-93363-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:31.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:30 smithi092 ceph-mon[33966]: from='client.48442 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi090-93363-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:31.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:30 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:31.395 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:14:31.395 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:14:31.463 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:14:31.490 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:31.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:31.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:14:31.592 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:14:31.619 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:31.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:31.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:14:31.721 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:14:31.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:31.746 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:31.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:14:31.848 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:14:31.856 INFO:tasks.workunit.client.0.smithi090.stdout: (68769 ms total) 2024-04-14T15:14:31.856 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: 2024-04-14T15:14:31.857 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2024-04-14T15:14:31.857 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2024-04-14T15:14:31.857 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (124 ms) 2024-04-14T15:14:31.857 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (124 ms total) 2024-04-14T15:14:31.857 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: 2024-04-14T15:14:31.857 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2024-04-14T15:14:31.857 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2024-04-14T15:14:31.857 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (33 ms) 2024-04-14T15:14:31.857 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2024-04-14T15:14:31.857 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (3 ms) 2024-04-14T15:14:31.857 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (36 ms total) 2024-04-14T15:14:31.857 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: 2024-04-14T15:14:31.857 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2024-04-14T15:14:31.858 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2024-04-14T15:14:31.858 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (23 ms) 2024-04-14T15:14:31.858 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2024-04-14T15:14:31.858 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (16 ms) 2024-04-14T15:14:31.858 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (39 ms total) 2024-04-14T15:14:31.858 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: 2024-04-14T15:14:31.858 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2024-04-14T15:14:31.858 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2024-04-14T15:14:31.858 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (33 ms) 2024-04-14T15:14:31.858 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2024-04-14T15:14:31.858 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (12 ms) 2024-04-14T15:14:31.858 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (45 ms total) 2024-04-14T15:14:31.859 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: 2024-04-14T15:14:31.859 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2024-04-14T15:14:31.859 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2024-04-14T15:14:31.859 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (1010 ms) 2024-04-14T15:14:31.859 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (1010 ms total) 2024-04-14T15:14:31.859 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: 2024-04-14T15:14:31.859 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [----------] Global test environment tear-down 2024-04-14T15:14:31.859 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (97614 ms total) 2024-04-14T15:14:31.859 INFO:tasks.workunit.client.0.smithi090.stdout: api_misc_pp: [ PASSED ] 31 tests. 2024-04-14T15:14:31.883 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:31.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:31.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:14:31.985 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:14:32.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:32.009 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:32.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:14:32.110 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:14:32.138 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:32.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:32.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:14:32.240 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:14:32.266 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:32.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:32.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:14:32.368 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:14:32.395 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:32.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:32.396 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:14:32.496 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:14:32.523 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:32.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:32.524 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:14:32.625 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:14:32.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:32.651 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:32.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:14:32.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.738+0000 7f9adf57b700 1 -- 172.21.15.90:0/4225352330 >> v1:172.21.15.90:6789/0 conn(0x7f9ad8103aa0 legacy=0x7f9ad8103e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:32.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.738+0000 7f9adf57b700 1 -- 172.21.15.90:0/4225352330 shutdown_connections 2024-04-14T15:14:32.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.738+0000 7f9adf57b700 1 -- 172.21.15.90:0/4225352330 >> 172.21.15.90:0/4225352330 conn(0x7f9ad80fd290 msgr2=0x7f9ad80ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:14:32.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.738+0000 7f9adf57b700 1 -- 172.21.15.90:0/4225352330 shutdown_connections 2024-04-14T15:14:32.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.738+0000 7f9adf57b700 1 -- 172.21.15.90:0/4225352330 wait complete. 2024-04-14T15:14:32.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.738+0000 7f9adf57b700 1 librados: starting msgr at 2024-04-14T15:14:32.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.738+0000 7f9adf57b700 1 librados: starting objecter 2024-04-14T15:14:32.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.738+0000 7f9adf57b700 1 Processor -- start 2024-04-14T15:14:32.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.738+0000 7f9adf57b700 1 -- start start 2024-04-14T15:14:32.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.738+0000 7f9adf57b700 1 librados: setting wanted keys 2024-04-14T15:14:32.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.738+0000 7f9adf57b700 1 librados: calling monclient init 2024-04-14T15:14:32.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.739+0000 7f9adf57b700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f9ad81a9a30 con 0x7f9ad8103aa0 2024-04-14T15:14:32.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.739+0000 7f9adf57b700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f9ad81aac10 con 0x7f9ad810b8b0 2024-04-14T15:14:32.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.739+0000 7f9adf57b700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f9ad81abe30 con 0x7f9ad810f3c0 2024-04-14T15:14:32.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.739+0000 7f9add317700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f9ad8103aa0 0x7f9ad810aee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:54960/0 (socket says 172.21.15.90:54960) 2024-04-14T15:14:32.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.739+0000 7f9add317700 1 -- 172.21.15.90:0/1189631935 learned_addr learned my addr 172.21.15.90:0/1189631935 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:14:32.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.739+0000 7f9ace7fc700 10 client.?.objecter ms_handle_connect 0x7f9ad8103aa0 2024-04-14T15:14:32.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.739+0000 7f9ace7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:14:32.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.739+0000 7f9ace7fc700 10 client.?.objecter ms_handle_connect 0x7f9ad810b8b0 2024-04-14T15:14:32.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.739+0000 7f9ace7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:14:32.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.739+0000 7f9ace7fc700 10 client.?.objecter ms_handle_connect 0x7f9ad810f3c0 2024-04-14T15:14:32.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.739+0000 7f9ace7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:14:32.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.739+0000 7f9ace7fc700 1 -- 172.21.15.90:0/1189631935 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3531612381 0 0) 0x7f9ad81a9a30 con 0x7f9ad8103aa0 2024-04-14T15:14:32.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.739+0000 7f9ace7fc700 1 -- 172.21.15.90:0/1189631935 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9ac40035b0 con 0x7f9ad8103aa0 2024-04-14T15:14:32.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.739+0000 7f9ace7fc700 1 -- 172.21.15.90:0/1189631935 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 395539790 0 0) 0x7f9ad81aac10 con 0x7f9ad810b8b0 2024-04-14T15:14:32.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.739+0000 7f9ace7fc700 1 -- 172.21.15.90:0/1189631935 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9ad81a9a30 con 0x7f9ad810b8b0 2024-04-14T15:14:32.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.739+0000 7f9ace7fc700 1 -- 172.21.15.90:0/1189631935 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3146832554 0 0) 0x7f9ad81abe30 con 0x7f9ad810f3c0 2024-04-14T15:14:32.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.739+0000 7f9ace7fc700 1 -- 172.21.15.90:0/1189631935 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9ad81aac10 con 0x7f9ad810f3c0 2024-04-14T15:14:32.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.739+0000 7f9ace7fc700 1 -- 172.21.15.90:0/1189631935 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2400495376 0 0) 0x7f9ac40035b0 con 0x7f9ad8103aa0 2024-04-14T15:14:32.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.739+0000 7f9ace7fc700 1 -- 172.21.15.90:0/1189631935 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9ad81abe30 con 0x7f9ad8103aa0 2024-04-14T15:14:32.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.739+0000 7f9ace7fc700 1 -- 172.21.15.90:0/1189631935 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f9ad40046e0 con 0x7f9ad8103aa0 2024-04-14T15:14:32.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.739+0000 7f9ace7fc700 1 -- 172.21.15.90:0/1189631935 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 435561194 0 0) 0x7f9ad81a9a30 con 0x7f9ad810b8b0 2024-04-14T15:14:32.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.739+0000 7f9ace7fc700 1 -- 172.21.15.90:0/1189631935 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9ac40035b0 con 0x7f9ad810b8b0 2024-04-14T15:14:32.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.739+0000 7f9ace7fc700 1 -- 172.21.15.90:0/1189631935 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f9ac8001870 con 0x7f9ad810b8b0 2024-04-14T15:14:32.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.739+0000 7f9ace7fc700 1 -- 172.21.15.90:0/1189631935 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4191176094 0 0) 0x7f9ad81aac10 con 0x7f9ad810f3c0 2024-04-14T15:14:32.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.739+0000 7f9ace7fc700 1 -- 172.21.15.90:0/1189631935 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9ad81a9a30 con 0x7f9ad810f3c0 2024-04-14T15:14:32.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.739+0000 7f9ace7fc700 1 -- 172.21.15.90:0/1189631935 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f9ad0001b30 con 0x7f9ad810f3c0 2024-04-14T15:14:32.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.739+0000 7f9ace7fc700 1 -- 172.21.15.90:0/1189631935 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3844880570 0 0) 0x7f9ad81abe30 con 0x7f9ad8103aa0 2024-04-14T15:14:32.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.739+0000 7f9ace7fc700 1 -- 172.21.15.90:0/1189631935 >> v1:172.21.15.90:6790/0 conn(0x7f9ad810f3c0 legacy=0x7f9ad81a8280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:32.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.739+0000 7f9ace7fc700 1 -- 172.21.15.90:0/1189631935 >> v1:172.21.15.92:6789/0 conn(0x7f9ad810b8b0 legacy=0x7f9ad81a4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:32.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.739+0000 7f9ace7fc700 1 -- 172.21.15.90:0/1189631935 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ad81ad050 con 0x7f9ad8103aa0 2024-04-14T15:14:32.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.740+0000 7f9adf57b700 1 -- 172.21.15.90:0/1189631935 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9ad81aae40 con 0x7f9ad8103aa0 2024-04-14T15:14:32.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.740+0000 7f9adf57b700 10 client.36600.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:14:32.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.740+0000 7f9adf57b700 1 -- 172.21.15.90:0/1189631935 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f9ad81ab300 con 0x7f9ad8103aa0 2024-04-14T15:14:32.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.740+0000 7f9adf57b700 1 librados: init done 2024-04-14T15:14:32.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.740+0000 7f9ace7fc700 1 -- 172.21.15.90:0/1189631935 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f9ad4001b70 con 0x7f9ad8103aa0 2024-04-14T15:14:32.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.740+0000 7f9ace7fc700 1 -- 172.21.15.90:0/1189631935 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f9ad4005110 con 0x7f9ad8103aa0 2024-04-14T15:14:32.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.740+0000 7f9adf57b700 1 -- 172.21.15.90:0/1189631935 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f9abc005100 con 0x7f9ad8103aa0 2024-04-14T15:14:32.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.741+0000 7f9ace7fc700 1 -- 172.21.15.90:0/1189631935 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f9ad40054a0 con 0x7f9ad8103aa0 2024-04-14T15:14:32.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.742+0000 7f9ace7fc700 1 -- 172.21.15.90:0/1189631935 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(204..204 src has 51..204) ==== 8857+0+0 (unknown 1872180656 0 0) 0x7f9ad408af60 con 0x7f9ad8103aa0 2024-04-14T15:14:32.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.742+0000 7f9ace7fc700 10 client.36600.objecter ms_dispatch 0x7f9ad8001550 osd_map(204..204 src has 51..204) 2024-04-14T15:14:32.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.742+0000 7f9ace7fc700 3 client.36600.objecter handle_osd_map got epochs [204,204] > 0 2024-04-14T15:14:32.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.742+0000 7f9ace7fc700 3 client.36600.objecter handle_osd_map decoding full epoch 204 2024-04-14T15:14:32.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.742+0000 7f9ace7fc700 20 client.36600.objecter dump_active .. 0 homeless 2024-04-14T15:14:32.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.742+0000 7f9ace7fc700 1 -- 172.21.15.90:0/1189631935 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v204) ==== 54+0+16 (unknown 3507849509 0 2420000192) 0x7f9ad408b2e0 con 0x7f9ad8103aa0 2024-04-14T15:14:32.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.742+0000 7f9ace7fc700 10 client.36600.objecter ms_handle_connect 0x7f9ac4071350 2024-04-14T15:14:32.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.742+0000 7f9adf57b700 10 client.36600.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:14:32.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.742+0000 7f9adf57b700 15 client.36600.objecter _session_command_op_assign -1 1 2024-04-14T15:14:32.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.742+0000 7f9adf57b700 20 client.36600.objecter _get_session s=0x7f9abc0053e0 osd=0 3 2024-04-14T15:14:32.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.742+0000 7f9adf57b700 20 client.36600.objecter put_session s=0x7f9abc0053e0 osd=0 3 2024-04-14T15:14:32.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.742+0000 7f9adf57b700 20 client.36600.objecter _get_session s=0x7f9abc0053e0 osd=0 3 2024-04-14T15:14:32.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.742+0000 7f9adf57b700 15 client.36600.objecter _session_command_op_remove -1 1 2024-04-14T15:14:32.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.742+0000 7f9adf57b700 20 client.36600.objecter get_session s=0x7f9abc0053e0 osd=0 3 2024-04-14T15:14:32.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.742+0000 7f9adf57b700 15 client.36600.objecter _session_command_op_assign 0 1 2024-04-14T15:14:32.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.742+0000 7f9adf57b700 20 client.36600.objecter put_session s=0x7f9abc0053e0 osd=0 4 2024-04-14T15:14:32.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.742+0000 7f9adf57b700 10 client.36600.objecter _send_command 1 2024-04-14T15:14:32.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.742+0000 7f9adf57b700 1 -- 172.21.15.90:0/1189631935 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9abc004b50 con 0x7f9abc004730 2024-04-14T15:14:32.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.742+0000 7f9ace7fc700 10 client.36600.objecter ms_handle_connect 0x7f9abc004730 2024-04-14T15:14:32.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.743+0000 7f9ace7fc700 1 -- 172.21.15.90:0/1189631935 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f9abc004b50 con 0x7f9abc004730 2024-04-14T15:14:32.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.743+0000 7f9ace7fc700 10 client.36600.objecter ms_dispatch 0x7f9ad8001550 command_reply(tid 1: 0 ) 2024-04-14T15:14:32.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.743+0000 7f9ace7fc700 10 client.36600.objecter _finish_command 1 = system:0 2024-04-14T15:14:32.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.743+0000 7f9ace7fc700 20 client.36600.objecter put_session s=0x7f9abc0053e0 osd=0 4 2024-04-14T15:14:32.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.743+0000 7f9ace7fc700 15 client.36600.objecter _session_command_op_remove 0 1 2024-04-14T15:14:32.752 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:14:32.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.760+0000 7f9adf57b700 10 client.36600.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:14:32.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.760+0000 7f9adf57b700 15 client.36600.objecter _session_command_op_assign -1 2 2024-04-14T15:14:32.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.760+0000 7f9adf57b700 20 client.36600.objecter _get_session s=0x7f9abc0053e0 osd=0 3 2024-04-14T15:14:32.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.760+0000 7f9adf57b700 20 client.36600.objecter put_session s=0x7f9abc0053e0 osd=0 3 2024-04-14T15:14:32.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.760+0000 7f9adf57b700 20 client.36600.objecter _get_session s=0x7f9abc0053e0 osd=0 3 2024-04-14T15:14:32.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.760+0000 7f9adf57b700 15 client.36600.objecter _session_command_op_remove -1 2 2024-04-14T15:14:32.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.760+0000 7f9adf57b700 20 client.36600.objecter get_session s=0x7f9abc0053e0 osd=0 3 2024-04-14T15:14:32.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.760+0000 7f9adf57b700 15 client.36600.objecter _session_command_op_assign 0 2 2024-04-14T15:14:32.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.760+0000 7f9adf57b700 20 client.36600.objecter put_session s=0x7f9abc0053e0 osd=0 4 2024-04-14T15:14:32.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.760+0000 7f9adf57b700 10 client.36600.objecter _send_command 2 2024-04-14T15:14:32.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.760+0000 7f9adf57b700 1 -- 172.21.15.90:0/1189631935 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9abc0079f0 con 0x7f9abc004730 2024-04-14T15:14:32.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.761+0000 7f9ace7fc700 1 -- 172.21.15.90:0/1189631935 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f9abc0079f0 con 0x7f9abc004730 2024-04-14T15:14:32.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.761+0000 7f9ace7fc700 10 client.36600.objecter ms_dispatch 0x7f9ad8001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:14:32.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.761+0000 7f9ace7fc700 10 client.36600.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:14:32.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.761+0000 7f9ace7fc700 20 client.36600.objecter put_session s=0x7f9abc0053e0 osd=0 4 2024-04-14T15:14:32.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.761+0000 7f9ace7fc700 15 client.36600.objecter _session_command_op_remove 0 2 2024-04-14T15:14:32.762 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:14:32.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.762+0000 7f9adf57b700 10 client.36600.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:14:32.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.762+0000 7f9adf57b700 15 client.36600.objecter _session_command_op_assign -1 3 2024-04-14T15:14:32.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.762+0000 7f9adf57b700 20 client.36600.objecter _get_session s=0x7f9abc004110 osd=1 3 2024-04-14T15:14:32.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.762+0000 7f9adf57b700 20 client.36600.objecter put_session s=0x7f9abc004110 osd=1 3 2024-04-14T15:14:32.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.762+0000 7f9adf57b700 20 client.36600.objecter _get_session s=0x7f9abc004110 osd=1 3 2024-04-14T15:14:32.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.762+0000 7f9adf57b700 15 client.36600.objecter _session_command_op_remove -1 3 2024-04-14T15:14:32.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.762+0000 7f9adf57b700 20 client.36600.objecter get_session s=0x7f9abc004110 osd=1 3 2024-04-14T15:14:32.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.762+0000 7f9adf57b700 15 client.36600.objecter _session_command_op_assign 1 3 2024-04-14T15:14:32.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.762+0000 7f9adf57b700 20 client.36600.objecter put_session s=0x7f9abc004110 osd=1 4 2024-04-14T15:14:32.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.762+0000 7f9adf57b700 10 client.36600.objecter _send_command 3 2024-04-14T15:14:32.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.762+0000 7f9adf57b700 1 -- 172.21.15.90:0/1189631935 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f9abc002160 con 0x7f9abc004280 2024-04-14T15:14:32.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.762+0000 7f9ace7fc700 10 client.36600.objecter ms_handle_connect 0x7f9abc004280 2024-04-14T15:14:32.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.763+0000 7f9ace7fc700 1 -- 172.21.15.90:0/1189631935 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f9abc002160 con 0x7f9abc004280 2024-04-14T15:14:32.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.763+0000 7f9ace7fc700 10 client.36600.objecter ms_dispatch 0x7f9ad8001550 command_reply(tid 3: 0 ) 2024-04-14T15:14:32.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.763+0000 7f9ace7fc700 10 client.36600.objecter _finish_command 3 = system:0 2024-04-14T15:14:32.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.763+0000 7f9ace7fc700 20 client.36600.objecter put_session s=0x7f9abc004110 osd=1 4 2024-04-14T15:14:32.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.763+0000 7f9ace7fc700 15 client.36600.objecter _session_command_op_remove 1 3 2024-04-14T15:14:32.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:32.777 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:32.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:14:32.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.779+0000 7f9adf57b700 10 client.36600.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:14:32.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.779+0000 7f9adf57b700 15 client.36600.objecter _session_command_op_assign -1 4 2024-04-14T15:14:32.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.779+0000 7f9adf57b700 20 client.36600.objecter _get_session s=0x7f9abc004110 osd=1 3 2024-04-14T15:14:32.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.779+0000 7f9adf57b700 20 client.36600.objecter put_session s=0x7f9abc004110 osd=1 3 2024-04-14T15:14:32.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.779+0000 7f9adf57b700 20 client.36600.objecter _get_session s=0x7f9abc004110 osd=1 3 2024-04-14T15:14:32.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.779+0000 7f9adf57b700 15 client.36600.objecter _session_command_op_remove -1 4 2024-04-14T15:14:32.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.779+0000 7f9adf57b700 20 client.36600.objecter get_session s=0x7f9abc004110 osd=1 3 2024-04-14T15:14:32.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.779+0000 7f9adf57b700 15 client.36600.objecter _session_command_op_assign 1 4 2024-04-14T15:14:32.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.779+0000 7f9adf57b700 20 client.36600.objecter put_session s=0x7f9abc004110 osd=1 4 2024-04-14T15:14:32.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.779+0000 7f9adf57b700 10 client.36600.objecter _send_command 4 2024-04-14T15:14:32.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.779+0000 7f9adf57b700 1 -- 172.21.15.90:0/1189631935 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9abc001600 con 0x7f9abc004280 2024-04-14T15:14:32.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.781+0000 7f9ace7fc700 1 -- 172.21.15.90:0/1189631935 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f9abc001600 con 0x7f9abc004280 2024-04-14T15:14:32.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.781+0000 7f9ace7fc700 10 client.36600.objecter ms_dispatch 0x7f9ad8001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:14:32.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.781+0000 7f9ace7fc700 10 client.36600.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:14:32.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.781+0000 7f9ace7fc700 20 client.36600.objecter put_session s=0x7f9abc004110 osd=1 4 2024-04-14T15:14:32.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.781+0000 7f9ace7fc700 15 client.36600.objecter _session_command_op_remove 1 4 2024-04-14T15:14:32.782 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:14:32.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.781+0000 7f9adf57b700 10 client.36600.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:14:32.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.781+0000 7f9adf57b700 15 client.36600.objecter _session_command_op_assign -1 5 2024-04-14T15:14:32.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.781+0000 7f9adf57b700 20 client.36600.objecter _get_session s=0x7f9abc001a00 osd=2 3 2024-04-14T15:14:32.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.781+0000 7f9adf57b700 20 client.36600.objecter put_session s=0x7f9abc001a00 osd=2 3 2024-04-14T15:14:32.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.781+0000 7f9adf57b700 20 client.36600.objecter _get_session s=0x7f9abc001a00 osd=2 3 2024-04-14T15:14:32.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.781+0000 7f9adf57b700 15 client.36600.objecter _session_command_op_remove -1 5 2024-04-14T15:14:32.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.781+0000 7f9adf57b700 20 client.36600.objecter get_session s=0x7f9abc001a00 osd=2 3 2024-04-14T15:14:32.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.781+0000 7f9adf57b700 15 client.36600.objecter _session_command_op_assign 2 5 2024-04-14T15:14:32.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.781+0000 7f9adf57b700 20 client.36600.objecter put_session s=0x7f9abc001a00 osd=2 4 2024-04-14T15:14:32.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.781+0000 7f9adf57b700 10 client.36600.objecter _send_command 5 2024-04-14T15:14:32.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.781+0000 7f9adf57b700 1 -- 172.21.15.90:0/1189631935 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f9abc010f40 con 0x7f9abc008880 2024-04-14T15:14:32.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.782+0000 7f9ace7fc700 10 client.36600.objecter ms_handle_connect 0x7f9abc008880 2024-04-14T15:14:32.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.782+0000 7f9ace7fc700 1 -- 172.21.15.90:0/1189631935 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f9abc010f40 con 0x7f9abc008880 2024-04-14T15:14:32.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.782+0000 7f9ace7fc700 10 client.36600.objecter ms_dispatch 0x7f9ad8001550 command_reply(tid 5: 0 ) 2024-04-14T15:14:32.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.782+0000 7f9ace7fc700 10 client.36600.objecter _finish_command 5 = system:0 2024-04-14T15:14:32.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.782+0000 7f9ace7fc700 20 client.36600.objecter put_session s=0x7f9abc001a00 osd=2 4 2024-04-14T15:14:32.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.782+0000 7f9ace7fc700 15 client.36600.objecter _session_command_op_remove 2 5 2024-04-14T15:14:32.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.798+0000 7f9adf57b700 10 client.36600.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:14:32.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.798+0000 7f9adf57b700 15 client.36600.objecter _session_command_op_assign -1 6 2024-04-14T15:14:32.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.798+0000 7f9adf57b700 20 client.36600.objecter _get_session s=0x7f9abc001a00 osd=2 3 2024-04-14T15:14:32.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.798+0000 7f9adf57b700 20 client.36600.objecter put_session s=0x7f9abc001a00 osd=2 3 2024-04-14T15:14:32.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.798+0000 7f9adf57b700 20 client.36600.objecter _get_session s=0x7f9abc001a00 osd=2 3 2024-04-14T15:14:32.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.798+0000 7f9adf57b700 15 client.36600.objecter _session_command_op_remove -1 6 2024-04-14T15:14:32.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.798+0000 7f9adf57b700 20 client.36600.objecter get_session s=0x7f9abc001a00 osd=2 3 2024-04-14T15:14:32.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.798+0000 7f9adf57b700 15 client.36600.objecter _session_command_op_assign 2 6 2024-04-14T15:14:32.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.798+0000 7f9adf57b700 20 client.36600.objecter put_session s=0x7f9abc001a00 osd=2 4 2024-04-14T15:14:32.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.798+0000 7f9adf57b700 10 client.36600.objecter _send_command 6 2024-04-14T15:14:32.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.798+0000 7f9adf57b700 1 -- 172.21.15.90:0/1189631935 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9abc007eb0 con 0x7f9abc008880 2024-04-14T15:14:32.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.800+0000 7f9ace7fc700 1 -- 172.21.15.90:0/1189631935 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f9abc007eb0 con 0x7f9abc008880 2024-04-14T15:14:32.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.800+0000 7f9ace7fc700 10 client.36600.objecter ms_dispatch 0x7f9ad8001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:14:32.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.800+0000 7f9ace7fc700 10 client.36600.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:14:32.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.800+0000 7f9ace7fc700 20 client.36600.objecter put_session s=0x7f9abc001a00 osd=2 4 2024-04-14T15:14:32.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.800+0000 7f9ace7fc700 15 client.36600.objecter _session_command_op_remove 2 6 2024-04-14T15:14:32.801 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:14:32.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.800+0000 7f9adf57b700 10 client.36600.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:14:32.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.800+0000 7f9adf57b700 15 client.36600.objecter _session_command_op_assign -1 7 2024-04-14T15:14:32.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.800+0000 7f9adf57b700 20 client.36600.objecter _get_session s=0x7f9abc0081a0 osd=3 3 2024-04-14T15:14:32.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.800+0000 7f9adf57b700 20 client.36600.objecter put_session s=0x7f9abc0081a0 osd=3 3 2024-04-14T15:14:32.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.800+0000 7f9adf57b700 20 client.36600.objecter _get_session s=0x7f9abc0081a0 osd=3 3 2024-04-14T15:14:32.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.800+0000 7f9adf57b700 15 client.36600.objecter _session_command_op_remove -1 7 2024-04-14T15:14:32.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.800+0000 7f9adf57b700 20 client.36600.objecter get_session s=0x7f9abc0081a0 osd=3 3 2024-04-14T15:14:32.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.800+0000 7f9adf57b700 15 client.36600.objecter _session_command_op_assign 3 7 2024-04-14T15:14:32.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.800+0000 7f9adf57b700 20 client.36600.objecter put_session s=0x7f9abc0081a0 osd=3 4 2024-04-14T15:14:32.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.800+0000 7f9adf57b700 10 client.36600.objecter _send_command 7 2024-04-14T15:14:32.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.800+0000 7f9adf57b700 1 -- 172.21.15.90:0/1189631935 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f9abc00d350 con 0x7f9abc00ce70 2024-04-14T15:14:32.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.801+0000 7f9ace7fc700 10 client.36600.objecter ms_handle_connect 0x7f9abc00ce70 2024-04-14T15:14:32.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.801+0000 7f9ace7fc700 1 -- 172.21.15.90:0/1189631935 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f9abc00d350 con 0x7f9abc00ce70 2024-04-14T15:14:32.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.801+0000 7f9ace7fc700 10 client.36600.objecter ms_dispatch 0x7f9ad8001550 command_reply(tid 7: 0 ) 2024-04-14T15:14:32.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.801+0000 7f9ace7fc700 10 client.36600.objecter _finish_command 7 = system:0 2024-04-14T15:14:32.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.801+0000 7f9ace7fc700 20 client.36600.objecter put_session s=0x7f9abc0081a0 osd=3 4 2024-04-14T15:14:32.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.801+0000 7f9ace7fc700 15 client.36600.objecter _session_command_op_remove 3 7 2024-04-14T15:14:32.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.818+0000 7f9adf57b700 10 client.36600.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:14:32.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.818+0000 7f9adf57b700 15 client.36600.objecter _session_command_op_assign -1 8 2024-04-14T15:14:32.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.818+0000 7f9adf57b700 20 client.36600.objecter _get_session s=0x7f9abc0081a0 osd=3 3 2024-04-14T15:14:32.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.818+0000 7f9adf57b700 20 client.36600.objecter put_session s=0x7f9abc0081a0 osd=3 3 2024-04-14T15:14:32.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.818+0000 7f9adf57b700 20 client.36600.objecter _get_session s=0x7f9abc0081a0 osd=3 3 2024-04-14T15:14:32.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.818+0000 7f9adf57b700 15 client.36600.objecter _session_command_op_remove -1 8 2024-04-14T15:14:32.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.818+0000 7f9adf57b700 20 client.36600.objecter get_session s=0x7f9abc0081a0 osd=3 3 2024-04-14T15:14:32.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.818+0000 7f9adf57b700 15 client.36600.objecter _session_command_op_assign 3 8 2024-04-14T15:14:32.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.818+0000 7f9adf57b700 20 client.36600.objecter put_session s=0x7f9abc0081a0 osd=3 4 2024-04-14T15:14:32.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.818+0000 7f9adf57b700 10 client.36600.objecter _send_command 8 2024-04-14T15:14:32.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.818+0000 7f9adf57b700 1 -- 172.21.15.90:0/1189631935 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9abc00c9b0 con 0x7f9abc00ce70 2024-04-14T15:14:32.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.820+0000 7f9ace7fc700 1 -- 172.21.15.90:0/1189631935 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f9abc00c9b0 con 0x7f9abc00ce70 2024-04-14T15:14:32.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.820+0000 7f9ace7fc700 10 client.36600.objecter ms_dispatch 0x7f9ad8001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:14:32.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.820+0000 7f9ace7fc700 10 client.36600.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:14:32.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.820+0000 7f9ace7fc700 20 client.36600.objecter put_session s=0x7f9abc0081a0 osd=3 4 2024-04-14T15:14:32.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.820+0000 7f9ace7fc700 15 client.36600.objecter _session_command_op_remove 3 8 2024-04-14T15:14:32.820 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:14:32.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.820+0000 7f9adf57b700 10 client.36600.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:14:32.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.820+0000 7f9adf57b700 15 client.36600.objecter _session_command_op_assign -1 9 2024-04-14T15:14:32.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.820+0000 7f9adf57b700 20 client.36600.objecter _get_session s=0x7f9abc00c760 osd=4 3 2024-04-14T15:14:32.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.820+0000 7f9adf57b700 20 client.36600.objecter put_session s=0x7f9abc00c760 osd=4 3 2024-04-14T15:14:32.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.820+0000 7f9adf57b700 20 client.36600.objecter _get_session s=0x7f9abc00c760 osd=4 3 2024-04-14T15:14:32.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.820+0000 7f9adf57b700 15 client.36600.objecter _session_command_op_remove -1 9 2024-04-14T15:14:32.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.820+0000 7f9adf57b700 20 client.36600.objecter get_session s=0x7f9abc00c760 osd=4 3 2024-04-14T15:14:32.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.820+0000 7f9adf57b700 15 client.36600.objecter _session_command_op_assign 4 9 2024-04-14T15:14:32.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.820+0000 7f9adf57b700 20 client.36600.objecter put_session s=0x7f9abc00c760 osd=4 4 2024-04-14T15:14:32.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.820+0000 7f9adf57b700 10 client.36600.objecter _send_command 9 2024-04-14T15:14:32.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.820+0000 7f9adf57b700 1 -- 172.21.15.90:0/1189631935 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f9abc01a8d0 con 0x7f9abc011e00 2024-04-14T15:14:32.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.820+0000 7f9ace7fc700 10 client.36600.objecter ms_handle_connect 0x7f9abc011e00 2024-04-14T15:14:32.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.821+0000 7f9ace7fc700 1 -- 172.21.15.90:0/1189631935 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f9abc01a8d0 con 0x7f9abc011e00 2024-04-14T15:14:32.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.821+0000 7f9ace7fc700 10 client.36600.objecter ms_dispatch 0x7f9ad8001550 command_reply(tid 9: 0 ) 2024-04-14T15:14:32.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.821+0000 7f9ace7fc700 10 client.36600.objecter _finish_command 9 = system:0 2024-04-14T15:14:32.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.821+0000 7f9ace7fc700 20 client.36600.objecter put_session s=0x7f9abc00c760 osd=4 4 2024-04-14T15:14:32.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.821+0000 7f9ace7fc700 15 client.36600.objecter _session_command_op_remove 4 9 2024-04-14T15:14:32.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.837+0000 7f9adf57b700 10 client.36600.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:14:32.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.837+0000 7f9adf57b700 15 client.36600.objecter _session_command_op_assign -1 10 2024-04-14T15:14:32.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.837+0000 7f9adf57b700 20 client.36600.objecter _get_session s=0x7f9abc00c760 osd=4 3 2024-04-14T15:14:32.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.837+0000 7f9adf57b700 20 client.36600.objecter put_session s=0x7f9abc00c760 osd=4 3 2024-04-14T15:14:32.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.837+0000 7f9adf57b700 20 client.36600.objecter _get_session s=0x7f9abc00c760 osd=4 3 2024-04-14T15:14:32.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.837+0000 7f9adf57b700 15 client.36600.objecter _session_command_op_remove -1 10 2024-04-14T15:14:32.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.837+0000 7f9adf57b700 20 client.36600.objecter get_session s=0x7f9abc00c760 osd=4 3 2024-04-14T15:14:32.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.837+0000 7f9adf57b700 15 client.36600.objecter _session_command_op_assign 4 10 2024-04-14T15:14:32.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.837+0000 7f9adf57b700 20 client.36600.objecter put_session s=0x7f9abc00c760 osd=4 4 2024-04-14T15:14:32.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.837+0000 7f9adf57b700 10 client.36600.objecter _send_command 10 2024-04-14T15:14:32.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.837+0000 7f9adf57b700 1 -- 172.21.15.90:0/1189631935 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9abc01a8d0 con 0x7f9abc011e00 2024-04-14T15:14:32.840 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:14:32.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.840+0000 7f9ace7fc700 1 -- 172.21.15.90:0/1189631935 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f9abc01a8d0 con 0x7f9abc011e00 2024-04-14T15:14:32.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.840+0000 7f9ace7fc700 10 client.36600.objecter ms_dispatch 0x7f9ad8001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:14:32.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.840+0000 7f9ace7fc700 10 client.36600.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:14:32.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.840+0000 7f9ace7fc700 20 client.36600.objecter put_session s=0x7f9abc00c760 osd=4 4 2024-04-14T15:14:32.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.840+0000 7f9ace7fc700 15 client.36600.objecter _session_command_op_remove 4 10 2024-04-14T15:14:32.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.840+0000 7f9adf57b700 10 client.36600.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:14:32.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.840+0000 7f9adf57b700 15 client.36600.objecter _session_command_op_assign -1 11 2024-04-14T15:14:32.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.840+0000 7f9adf57b700 20 client.36600.objecter _get_session s=0x7f9abc00cd00 osd=5 3 2024-04-14T15:14:32.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.840+0000 7f9adf57b700 20 client.36600.objecter put_session s=0x7f9abc00cd00 osd=5 3 2024-04-14T15:14:32.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.840+0000 7f9adf57b700 20 client.36600.objecter _get_session s=0x7f9abc00cd00 osd=5 3 2024-04-14T15:14:32.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.840+0000 7f9adf57b700 15 client.36600.objecter _session_command_op_remove -1 11 2024-04-14T15:14:32.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.840+0000 7f9adf57b700 20 client.36600.objecter get_session s=0x7f9abc00cd00 osd=5 3 2024-04-14T15:14:32.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.840+0000 7f9adf57b700 15 client.36600.objecter _session_command_op_assign 5 11 2024-04-14T15:14:32.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.840+0000 7f9adf57b700 20 client.36600.objecter put_session s=0x7f9abc00cd00 osd=5 4 2024-04-14T15:14:32.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.840+0000 7f9adf57b700 10 client.36600.objecter _send_command 11 2024-04-14T15:14:32.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.840+0000 7f9adf57b700 1 -- 172.21.15.90:0/1189631935 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f9abc01eb60 con 0x7f9abc01ae50 2024-04-14T15:14:32.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.848+0000 7f9ace7fc700 10 client.36600.objecter ms_handle_connect 0x7f9abc01ae50 2024-04-14T15:14:32.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.863+0000 7f9ace7fc700 1 -- 172.21.15.90:0/1189631935 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f9abc01eb60 con 0x7f9abc01ae50 2024-04-14T15:14:32.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.863+0000 7f9ace7fc700 10 client.36600.objecter ms_dispatch 0x7f9ad8001550 command_reply(tid 11: 0 ) 2024-04-14T15:14:32.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.863+0000 7f9ace7fc700 10 client.36600.objecter _finish_command 11 = system:0 2024-04-14T15:14:32.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.863+0000 7f9ace7fc700 20 client.36600.objecter put_session s=0x7f9abc00cd00 osd=5 4 2024-04-14T15:14:32.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.863+0000 7f9ace7fc700 15 client.36600.objecter _session_command_op_remove 5 11 2024-04-14T15:14:32.878 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:14:32.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.897+0000 7f9adf57b700 10 client.36600.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:14:32.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.897+0000 7f9adf57b700 15 client.36600.objecter _session_command_op_assign -1 12 2024-04-14T15:14:32.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.897+0000 7f9adf57b700 20 client.36600.objecter _get_session s=0x7f9abc00cd00 osd=5 3 2024-04-14T15:14:32.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.897+0000 7f9adf57b700 20 client.36600.objecter put_session s=0x7f9abc00cd00 osd=5 3 2024-04-14T15:14:32.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.897+0000 7f9adf57b700 20 client.36600.objecter _get_session s=0x7f9abc00cd00 osd=5 3 2024-04-14T15:14:32.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.897+0000 7f9adf57b700 15 client.36600.objecter _session_command_op_remove -1 12 2024-04-14T15:14:32.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.897+0000 7f9adf57b700 20 client.36600.objecter get_session s=0x7f9abc00cd00 osd=5 3 2024-04-14T15:14:32.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.897+0000 7f9adf57b700 15 client.36600.objecter _session_command_op_assign 5 12 2024-04-14T15:14:32.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.897+0000 7f9adf57b700 20 client.36600.objecter put_session s=0x7f9abc00cd00 osd=5 4 2024-04-14T15:14:32.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.897+0000 7f9adf57b700 10 client.36600.objecter _send_command 12 2024-04-14T15:14:32.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.897+0000 7f9adf57b700 1 -- 172.21.15.90:0/1189631935 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9abc01eb60 con 0x7f9abc01ae50 2024-04-14T15:14:32.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.899+0000 7f9ace7fc700 1 -- 172.21.15.90:0/1189631935 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f9abc01eb60 con 0x7f9abc01ae50 2024-04-14T15:14:32.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.899+0000 7f9ace7fc700 10 client.36600.objecter ms_dispatch 0x7f9ad8001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:14:32.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.899+0000 7f9ace7fc700 10 client.36600.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:14:32.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.899+0000 7f9ace7fc700 20 client.36600.objecter put_session s=0x7f9abc00cd00 osd=5 4 2024-04-14T15:14:32.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.899+0000 7f9ace7fc700 15 client.36600.objecter _session_command_op_remove 5 12 2024-04-14T15:14:32.900 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:14:32.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.900+0000 7f9adf57b700 10 client.36600.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:14:32.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.900+0000 7f9adf57b700 15 client.36600.objecter _session_command_op_assign -1 13 2024-04-14T15:14:32.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.900+0000 7f9adf57b700 20 client.36600.objecter _get_session s=0x7f9abc01eba0 osd=6 3 2024-04-14T15:14:32.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.900+0000 7f9adf57b700 20 client.36600.objecter put_session s=0x7f9abc01eba0 osd=6 3 2024-04-14T15:14:32.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.900+0000 7f9adf57b700 20 client.36600.objecter _get_session s=0x7f9abc01eba0 osd=6 3 2024-04-14T15:14:32.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.900+0000 7f9adf57b700 15 client.36600.objecter _session_command_op_remove -1 13 2024-04-14T15:14:32.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.900+0000 7f9adf57b700 20 client.36600.objecter get_session s=0x7f9abc01eba0 osd=6 3 2024-04-14T15:14:32.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.900+0000 7f9adf57b700 15 client.36600.objecter _session_command_op_assign 6 13 2024-04-14T15:14:32.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.900+0000 7f9adf57b700 20 client.36600.objecter put_session s=0x7f9abc01eba0 osd=6 4 2024-04-14T15:14:32.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.900+0000 7f9adf57b700 10 client.36600.objecter _send_command 13 2024-04-14T15:14:32.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.900+0000 7f9adf57b700 1 -- 172.21.15.90:0/1189631935 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f9abc01fe80 con 0x7f9abc020b10 2024-04-14T15:14:32.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.900+0000 7f9ace7fc700 10 client.36600.objecter ms_handle_connect 0x7f9abc020b10 2024-04-14T15:14:32.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.901+0000 7f9ace7fc700 1 -- 172.21.15.90:0/1189631935 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f9abc01fe80 con 0x7f9abc020b10 2024-04-14T15:14:32.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.901+0000 7f9ace7fc700 10 client.36600.objecter ms_dispatch 0x7f9ad8001550 command_reply(tid 13: 0 ) 2024-04-14T15:14:32.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.901+0000 7f9ace7fc700 10 client.36600.objecter _finish_command 13 = system:0 2024-04-14T15:14:32.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.901+0000 7f9ace7fc700 20 client.36600.objecter put_session s=0x7f9abc01eba0 osd=6 4 2024-04-14T15:14:32.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.901+0000 7f9ace7fc700 15 client.36600.objecter _session_command_op_remove 6 13 2024-04-14T15:14:32.914 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:32.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:32.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:14:32.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.923+0000 7f9adf57b700 10 client.36600.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:14:32.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.923+0000 7f9adf57b700 15 client.36600.objecter _session_command_op_assign -1 14 2024-04-14T15:14:32.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.923+0000 7f9adf57b700 20 client.36600.objecter _get_session s=0x7f9abc01eba0 osd=6 3 2024-04-14T15:14:32.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.923+0000 7f9adf57b700 20 client.36600.objecter put_session s=0x7f9abc01eba0 osd=6 3 2024-04-14T15:14:32.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.923+0000 7f9adf57b700 20 client.36600.objecter _get_session s=0x7f9abc01eba0 osd=6 3 2024-04-14T15:14:32.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.923+0000 7f9adf57b700 15 client.36600.objecter _session_command_op_remove -1 14 2024-04-14T15:14:32.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.923+0000 7f9adf57b700 20 client.36600.objecter get_session s=0x7f9abc01eba0 osd=6 3 2024-04-14T15:14:32.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.923+0000 7f9adf57b700 15 client.36600.objecter _session_command_op_assign 6 14 2024-04-14T15:14:32.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.923+0000 7f9adf57b700 20 client.36600.objecter put_session s=0x7f9abc01eba0 osd=6 4 2024-04-14T15:14:32.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.923+0000 7f9adf57b700 10 client.36600.objecter _send_command 14 2024-04-14T15:14:32.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.923+0000 7f9adf57b700 1 -- 172.21.15.90:0/1189631935 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9abc01fe80 con 0x7f9abc020b10 2024-04-14T15:14:32.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.925+0000 7f9ace7fc700 1 -- 172.21.15.90:0/1189631935 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f9abc01fe80 con 0x7f9abc020b10 2024-04-14T15:14:32.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.925+0000 7f9ace7fc700 10 client.36600.objecter ms_dispatch 0x7f9ad8001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:14:32.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.925+0000 7f9ace7fc700 10 client.36600.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:14:32.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.925+0000 7f9ace7fc700 20 client.36600.objecter put_session s=0x7f9abc01eba0 osd=6 4 2024-04-14T15:14:32.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.925+0000 7f9ace7fc700 15 client.36600.objecter _session_command_op_remove 6 14 2024-04-14T15:14:32.926 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:14:32.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.925+0000 7f9adf57b700 10 client.36600.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:14:32.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.925+0000 7f9adf57b700 15 client.36600.objecter _session_command_op_assign -1 15 2024-04-14T15:14:32.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.925+0000 7f9adf57b700 20 client.36600.objecter _get_session s=0x7f9abc01fe80 osd=7 3 2024-04-14T15:14:32.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.925+0000 7f9adf57b700 20 client.36600.objecter put_session s=0x7f9abc01fe80 osd=7 3 2024-04-14T15:14:32.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.925+0000 7f9adf57b700 20 client.36600.objecter _get_session s=0x7f9abc01fe80 osd=7 3 2024-04-14T15:14:32.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.925+0000 7f9adf57b700 15 client.36600.objecter _session_command_op_remove -1 15 2024-04-14T15:14:32.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.925+0000 7f9adf57b700 20 client.36600.objecter get_session s=0x7f9abc01fe80 osd=7 3 2024-04-14T15:14:32.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.925+0000 7f9adf57b700 15 client.36600.objecter _session_command_op_assign 7 15 2024-04-14T15:14:32.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.925+0000 7f9adf57b700 20 client.36600.objecter put_session s=0x7f9abc01fe80 osd=7 4 2024-04-14T15:14:32.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.925+0000 7f9adf57b700 10 client.36600.objecter _send_command 15 2024-04-14T15:14:32.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.925+0000 7f9adf57b700 1 -- 172.21.15.90:0/1189631935 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f9abc027500 con 0x7f9abc020150 2024-04-14T15:14:32.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.926+0000 7f9ace7fc700 10 client.36600.objecter ms_handle_connect 0x7f9abc020150 2024-04-14T15:14:32.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.927+0000 7f9ace7fc700 1 -- 172.21.15.90:0/1189631935 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f9abc027500 con 0x7f9abc020150 2024-04-14T15:14:32.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.927+0000 7f9ace7fc700 10 client.36600.objecter ms_dispatch 0x7f9ad8001550 command_reply(tid 15: 0 ) 2024-04-14T15:14:32.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.927+0000 7f9ace7fc700 10 client.36600.objecter _finish_command 15 = system:0 2024-04-14T15:14:32.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.927+0000 7f9ace7fc700 20 client.36600.objecter put_session s=0x7f9abc01fe80 osd=7 4 2024-04-14T15:14:32.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.927+0000 7f9ace7fc700 15 client.36600.objecter _session_command_op_remove 7 15 2024-04-14T15:14:32.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.943+0000 7f9adf57b700 10 client.36600.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:14:32.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.943+0000 7f9adf57b700 15 client.36600.objecter _session_command_op_assign -1 16 2024-04-14T15:14:32.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.943+0000 7f9adf57b700 20 client.36600.objecter _get_session s=0x7f9abc01fe80 osd=7 3 2024-04-14T15:14:32.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.943+0000 7f9adf57b700 20 client.36600.objecter put_session s=0x7f9abc01fe80 osd=7 3 2024-04-14T15:14:32.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.943+0000 7f9adf57b700 20 client.36600.objecter _get_session s=0x7f9abc01fe80 osd=7 3 2024-04-14T15:14:32.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.943+0000 7f9adf57b700 15 client.36600.objecter _session_command_op_remove -1 16 2024-04-14T15:14:32.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.943+0000 7f9adf57b700 20 client.36600.objecter get_session s=0x7f9abc01fe80 osd=7 3 2024-04-14T15:14:32.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.943+0000 7f9adf57b700 15 client.36600.objecter _session_command_op_assign 7 16 2024-04-14T15:14:32.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.943+0000 7f9adf57b700 20 client.36600.objecter put_session s=0x7f9abc01fe80 osd=7 4 2024-04-14T15:14:32.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.943+0000 7f9adf57b700 10 client.36600.objecter _send_command 16 2024-04-14T15:14:32.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.943+0000 7f9adf57b700 1 -- 172.21.15.90:0/1189631935 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9abc000990 con 0x7f9abc020150 2024-04-14T15:14:32.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.945+0000 7f9ace7fc700 1 -- 172.21.15.90:0/1189631935 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f9abc000990 con 0x7f9abc020150 2024-04-14T15:14:32.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.945+0000 7f9ace7fc700 10 client.36600.objecter ms_dispatch 0x7f9ad8001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:14:32.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.945+0000 7f9ace7fc700 10 client.36600.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:14:32.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.945+0000 7f9ace7fc700 20 client.36600.objecter put_session s=0x7f9abc01fe80 osd=7 4 2024-04-14T15:14:32.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.945+0000 7f9ace7fc700 15 client.36600.objecter _session_command_op_remove 7 16 2024-04-14T15:14:32.946 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:14:32.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.945+0000 7f9adf57b700 10 librados: watch_flush enter 2024-04-14T15:14:32.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.945+0000 7f9adf57b700 10 librados: watch_flush exit 2024-04-14T15:14:32.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.945+0000 7f9adf57b700 10 client.36600.objecter close_session for osd.0 2024-04-14T15:14:32.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.945+0000 7f9adf57b700 1 -- 172.21.15.90:0/1189631935 >> v1:172.21.15.90:6801/3761806885 conn(0x7f9abc004730 legacy=0x7f9abc002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:32.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.945+0000 7f9adf57b700 20 client.36600.objecter put_session s=0x7f9abc0053e0 osd=0 1 2024-04-14T15:14:32.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.945+0000 7f9adf57b700 10 client.36600.objecter close_session for osd.1 2024-04-14T15:14:32.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.945+0000 7f9adf57b700 1 -- 172.21.15.90:0/1189631935 >> v1:172.21.15.90:6805/4170361699 conn(0x7f9abc004280 legacy=0x7f9abc00ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:32.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.945+0000 7f9adf57b700 20 client.36600.objecter put_session s=0x7f9abc004110 osd=1 1 2024-04-14T15:14:32.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.945+0000 7f9adf57b700 10 client.36600.objecter close_session for osd.2 2024-04-14T15:14:32.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.945+0000 7f9adf57b700 1 -- 172.21.15.90:0/1189631935 >> v1:172.21.15.90:6809/1686184251 conn(0x7f9abc008880 legacy=0x7f9abc00f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:32.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.945+0000 7f9adf57b700 20 client.36600.objecter put_session s=0x7f9abc001a00 osd=2 1 2024-04-14T15:14:32.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.945+0000 7f9adf57b700 10 client.36600.objecter close_session for osd.3 2024-04-14T15:14:32.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.945+0000 7f9adf57b700 1 -- 172.21.15.90:0/1189631935 >> v1:172.21.15.90:6813/2270166774 conn(0x7f9abc00ce70 legacy=0x7f9abc014900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:32.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.945+0000 7f9adf57b700 20 client.36600.objecter put_session s=0x7f9abc0081a0 osd=3 1 2024-04-14T15:14:32.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.945+0000 7f9adf57b700 10 client.36600.objecter close_session for osd.4 2024-04-14T15:14:32.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.945+0000 7f9adf57b700 1 -- 172.21.15.90:0/1189631935 >> v1:172.21.15.92:6800/3887549733 conn(0x7f9abc011e00 legacy=0x7f9abc0191b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:32.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.945+0000 7f9adf57b700 20 client.36600.objecter put_session s=0x7f9abc00c760 osd=4 1 2024-04-14T15:14:32.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.945+0000 7f9adf57b700 10 client.36600.objecter close_session for osd.5 2024-04-14T15:14:32.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.945+0000 7f9adf57b700 1 -- 172.21.15.90:0/1189631935 >> v1:172.21.15.92:6804/3244561440 conn(0x7f9abc01ae50 legacy=0x7f9abc01d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:32.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.945+0000 7f9adf57b700 20 client.36600.objecter put_session s=0x7f9abc00cd00 osd=5 1 2024-04-14T15:14:32.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.945+0000 7f9adf57b700 10 client.36600.objecter close_session for osd.6 2024-04-14T15:14:32.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.945+0000 7f9adf57b700 1 -- 172.21.15.90:0/1189631935 >> v1:172.21.15.92:6808/636125963 conn(0x7f9abc020b10 legacy=0x7f9abc020f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:32.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.945+0000 7f9adf57b700 20 client.36600.objecter put_session s=0x7f9abc01eba0 osd=6 1 2024-04-14T15:14:32.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.945+0000 7f9adf57b700 10 client.36600.objecter close_session for osd.7 2024-04-14T15:14:32.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.945+0000 7f9adf57b700 1 -- 172.21.15.90:0/1189631935 >> v1:172.21.15.92:6812/1773706716 conn(0x7f9abc020150 legacy=0x7f9abc025c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:32.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.945+0000 7f9adf57b700 20 client.36600.objecter put_session s=0x7f9abc01fe80 osd=7 1 2024-04-14T15:14:32.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.945+0000 7f9adf57b700 20 client.36600.objecter shutdown clearing up homeless session... 2024-04-14T15:14:32.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.945+0000 7f9adf57b700 10 client.36600.objecter successfully canceled tick 2024-04-14T15:14:32.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.945+0000 7f9addb18700 1 -- 172.21.15.90:0/1189631935 reap_dead start 2024-04-14T15:14:32.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.945+0000 7f9addb18700 1 -- 172.21.15.90:0/1189631935 reap_dead start 2024-04-14T15:14:32.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.945+0000 7f9addb18700 1 -- 172.21.15.90:0/1189631935 reap_dead start 2024-04-14T15:14:32.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.945+0000 7f9addb18700 1 -- 172.21.15.90:0/1189631935 reap_dead start 2024-04-14T15:14:32.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.945+0000 7f9adf57b700 1 -- 172.21.15.90:0/1189631935 >> v1:172.21.15.90:6800/420935427 conn(0x7f9ac4071350 legacy=0x7f9ac4073810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:32.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.945+0000 7f9addb18700 1 -- 172.21.15.90:0/1189631935 reap_dead start 2024-04-14T15:14:32.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.945+0000 7f9adf57b700 1 -- 172.21.15.90:0/1189631935 >> v1:172.21.15.90:6789/0 conn(0x7f9ad8103aa0 legacy=0x7f9ad810aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:32.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.945+0000 7f9adf57b700 1 -- 172.21.15.90:0/1189631935 shutdown_connections 2024-04-14T15:14:32.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.945+0000 7f9adf57b700 1 -- 172.21.15.90:0/1189631935 >> 172.21.15.90:0/1189631935 conn(0x7f9ad80fd290 msgr2=0x7f9ad80ff680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:14:32.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.946+0000 7f9adf57b700 1 -- 172.21.15.90:0/1189631935 shutdown_connections 2024-04-14T15:14:32.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.946+0000 7f9adf57b700 1 -- 172.21.15.90:0/1189631935 wait complete. 2024-04-14T15:14:32.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:32.946+0000 7f9adf57b700 1 librados: shutdown 2024-04-14T15:14:32.951 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:14:32.951 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:14:32.951 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:14:32.951 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:14:32.951 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:14:32.951 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:14:32.952 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:14:32.952 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:14:33.017 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:14:33.042 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:33.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:33.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:14:33.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:32 smithi092 ceph-mon[33966]: pgmap v251: 260 pgs: 8 active+clean+snaptrim, 32 unknown, 9 active+clean+snaptrim_wait, 211 active+clean; 535 KiB data, 533 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:14:33.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:32 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:14:33.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:32 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:14:33.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:32 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3214396702' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi090-93601-36"}]': finished 2024-04-14T15:14:33.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:32 smithi092 ceph-mon[33966]: from='client.48424 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi090-93342-29"}]': finished 2024-04-14T15:14:33.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:32 smithi092 ceph-mon[33966]: from='client.48442 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi090-93363-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:33.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:32 smithi092 ceph-mon[33966]: osdmap e204: 8 total, 8 up, 8 in 2024-04-14T15:14:33.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:32 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2359757960' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi090-93342-29"}]: dispatch 2024-04-14T15:14:33.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:32 smithi092 ceph-mon[33966]: from='client.48424 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi090-93342-29"}]: dispatch 2024-04-14T15:14:33.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:32 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:33.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:32 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:33.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:32 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:33.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:32 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1189631935' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:14:33.145 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:14:33.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:33.171 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:33.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:14:33.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:32 smithi090 ceph-mon[30399]: pgmap v251: 260 pgs: 8 active+clean+snaptrim, 32 unknown, 9 active+clean+snaptrim_wait, 211 active+clean; 535 KiB data, 533 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:14:33.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:32 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:14:33.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:32 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:14:33.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:32 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3214396702' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi090-93601-36"}]': finished 2024-04-14T15:14:33.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:32 smithi090 ceph-mon[30399]: from='client.48424 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi090-93342-29"}]': finished 2024-04-14T15:14:33.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:32 smithi090 ceph-mon[30399]: from='client.48442 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi090-93363-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:33.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:32 smithi090 ceph-mon[30399]: osdmap e204: 8 total, 8 up, 8 in 2024-04-14T15:14:33.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:32 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2359757960' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi090-93342-29"}]: dispatch 2024-04-14T15:14:33.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:32 smithi090 ceph-mon[30399]: from='client.48424 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi090-93342-29"}]: dispatch 2024-04-14T15:14:33.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:32 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:33.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:32 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:33.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:32 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:33.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:32 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1189631935' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:14:33.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:32 smithi090 ceph-mon[41670]: pgmap v251: 260 pgs: 8 active+clean+snaptrim, 32 unknown, 9 active+clean+snaptrim_wait, 211 active+clean; 535 KiB data, 533 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:14:33.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:32 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:14:33.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:32 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:14:33.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:32 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3214396702' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi090-93601-36"}]': finished 2024-04-14T15:14:33.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:32 smithi090 ceph-mon[41670]: from='client.48424 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi090-93342-29"}]': finished 2024-04-14T15:14:33.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:32 smithi090 ceph-mon[41670]: from='client.48442 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi090-93363-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:33.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:32 smithi090 ceph-mon[41670]: osdmap e204: 8 total, 8 up, 8 in 2024-04-14T15:14:33.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:32 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2359757960' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi090-93342-29"}]: dispatch 2024-04-14T15:14:33.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:32 smithi090 ceph-mon[41670]: from='client.48424 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi090-93342-29"}]: dispatch 2024-04-14T15:14:33.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:32 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:33.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:32 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:33.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:32 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:33.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:32 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1189631935' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:14:33.272 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:14:33.297 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:33.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:33.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:14:33.398 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:14:33.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:33.423 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:33.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:14:33.524 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:14:33.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:33.549 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:33.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:14:33.650 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:33.675 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:33.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:33.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:33.777 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:14:33.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:33.801 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:33.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:14:33.902 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:14:33.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:33.927 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:33.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:14:34.028 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:14:34.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:34.052 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:34.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:14:34.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:33 smithi092 ceph-mon[33966]: from='client.48424 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi090-93342-29"}]': finished 2024-04-14T15:14:34.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:33 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:34.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:33 smithi092 ceph-mon[33966]: osdmap e205: 8 total, 8 up, 8 in 2024-04-14T15:14:34.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:33 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:14:34.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:33 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:14:34.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:33 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2834621764' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi090-93342-30"}]: dispatch 2024-04-14T15:14:34.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:33 smithi092 ceph-mon[33966]: from='client.48454 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi090-93342-30"}]: dispatch 2024-04-14T15:14:34.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:33 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2834621764' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi090-93342-30"}]: dispatch 2024-04-14T15:14:34.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:33 smithi092 ceph-mon[33966]: from='client.48454 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi090-93342-30"}]: dispatch 2024-04-14T15:14:34.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:33 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2834621764' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi090-93342-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:14:34.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:33 smithi092 ceph-mon[33966]: from='client.48454 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi090-93342-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:14:34.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:33 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:34.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:33 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-04-14T15:14:34.154 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:14:34.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:34.179 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:34.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:14:34.182 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:14:33 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:14:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:14:34.183 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:33 smithi090 ceph-mon[41670]: from='client.48424 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi090-93342-29"}]': finished 2024-04-14T15:14:34.183 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:33 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:34.183 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:33 smithi090 ceph-mon[41670]: osdmap e205: 8 total, 8 up, 8 in 2024-04-14T15:14:34.183 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:33 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:14:34.183 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:33 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:14:34.183 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:33 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2834621764' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi090-93342-30"}]: dispatch 2024-04-14T15:14:34.183 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:33 smithi090 ceph-mon[41670]: from='client.48454 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi090-93342-30"}]: dispatch 2024-04-14T15:14:34.183 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:33 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2834621764' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi090-93342-30"}]: dispatch 2024-04-14T15:14:34.183 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:33 smithi090 ceph-mon[41670]: from='client.48454 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi090-93342-30"}]: dispatch 2024-04-14T15:14:34.183 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:33 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2834621764' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi090-93342-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:14:34.183 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:33 smithi090 ceph-mon[41670]: from='client.48454 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi090-93342-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:14:34.184 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:33 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:34.184 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:33 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-04-14T15:14:34.184 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:33 smithi090 ceph-mon[30399]: from='client.48424 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi090-93342-29"}]': finished 2024-04-14T15:14:34.184 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:33 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:34.184 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:33 smithi090 ceph-mon[30399]: osdmap e205: 8 total, 8 up, 8 in 2024-04-14T15:14:34.184 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:33 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:14:34.184 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:33 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:14:34.184 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:33 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2834621764' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi090-93342-30"}]: dispatch 2024-04-14T15:14:34.184 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:33 smithi090 ceph-mon[30399]: from='client.48454 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi090-93342-30"}]: dispatch 2024-04-14T15:14:34.184 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:33 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2834621764' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi090-93342-30"}]: dispatch 2024-04-14T15:14:34.184 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:33 smithi090 ceph-mon[30399]: from='client.48454 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi090-93342-30"}]: dispatch 2024-04-14T15:14:34.184 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:33 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2834621764' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi090-93342-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:14:34.185 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:33 smithi090 ceph-mon[30399]: from='client.48454 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi090-93342-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:14:34.185 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:33 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:34.185 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:33 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-04-14T15:14:34.280 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:14:34.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:34.304 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:34.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:14:34.405 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:14:34.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:34.429 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:34.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:14:34.530 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:34.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:34.556 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:34.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:34.657 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:34.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:34.682 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:34.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:34.783 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:14:34.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:34.808 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:34.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:14:34.909 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:14:34.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:34.935 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:34.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:14:34.941 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:34 smithi090 ceph-mon[30399]: pgmap v254: 259 pgs: 8 active+clean+snaptrim, 32 unknown, 9 active+clean+snaptrim_wait, 210 active+clean; 535 KiB data, 533 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:14:34.941 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:34 smithi090 ceph-mon[30399]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:14:34.941 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:34 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-17", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:14:34.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:34 smithi090 ceph-mon[30399]: from='client.48454 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi090-93342-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:14:34.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:34 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "20"}]': finished 2024-04-14T15:14:34.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:34 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2834621764' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi090-93342-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi090-93342-30"}]: dispatch 2024-04-14T15:14:34.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:34 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-17"}]: dispatch 2024-04-14T15:14:34.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:34 smithi090 ceph-mon[30399]: osdmap e206: 8 total, 8 up, 8 in 2024-04-14T15:14:34.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:34 smithi090 ceph-mon[30399]: from='client.48454 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi090-93342-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi090-93342-30"}]: dispatch 2024-04-14T15:14:34.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:34 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3155829937' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi090-93363-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:34.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:34 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-17"}]: dispatch 2024-04-14T15:14:34.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:34 smithi090 ceph-mon[30399]: from='client.48457 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi090-93363-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:34.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:34 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:34.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:34 smithi090 ceph-mon[41670]: pgmap v254: 259 pgs: 8 active+clean+snaptrim, 32 unknown, 9 active+clean+snaptrim_wait, 210 active+clean; 535 KiB data, 533 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:14:34.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:34 smithi090 ceph-mon[41670]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:14:34.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:34 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-17", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:14:34.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:34 smithi090 ceph-mon[41670]: from='client.48454 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi090-93342-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:14:34.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:34 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "20"}]': finished 2024-04-14T15:14:34.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:34 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2834621764' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi090-93342-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi090-93342-30"}]: dispatch 2024-04-14T15:14:34.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:34 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-17"}]: dispatch 2024-04-14T15:14:34.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:34 smithi090 ceph-mon[41670]: osdmap e206: 8 total, 8 up, 8 in 2024-04-14T15:14:34.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:34 smithi090 ceph-mon[41670]: from='client.48454 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi090-93342-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi090-93342-30"}]: dispatch 2024-04-14T15:14:34.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:34 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3155829937' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi090-93363-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:34.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:34 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-17"}]: dispatch 2024-04-14T15:14:34.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:34 smithi090 ceph-mon[41670]: from='client.48457 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi090-93363-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:34.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:34 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:35.037 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:14:35.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:35.063 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:35.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:14:35.164 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:14:35.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:35.189 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:35.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:14:35.290 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:14:35.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:35.314 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:35.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:14:35.372 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:14:34 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:14:35.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:34 smithi092 ceph-mon[33966]: pgmap v254: 259 pgs: 8 active+clean+snaptrim, 32 unknown, 9 active+clean+snaptrim_wait, 210 active+clean; 535 KiB data, 533 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:14:35.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:34 smithi092 ceph-mon[33966]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:14:35.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:34 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-17", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:14:35.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:34 smithi092 ceph-mon[33966]: from='client.48454 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi090-93342-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:14:35.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:34 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "20"}]': finished 2024-04-14T15:14:35.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:34 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2834621764' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi090-93342-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi090-93342-30"}]: dispatch 2024-04-14T15:14:35.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:34 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-17"}]: dispatch 2024-04-14T15:14:35.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:34 smithi092 ceph-mon[33966]: osdmap e206: 8 total, 8 up, 8 in 2024-04-14T15:14:35.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:34 smithi092 ceph-mon[33966]: from='client.48454 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi090-93342-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi090-93342-30"}]: dispatch 2024-04-14T15:14:35.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:34 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3155829937' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi090-93363-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:35.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:34 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-17"}]: dispatch 2024-04-14T15:14:35.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:34 smithi092 ceph-mon[33966]: from='client.48457 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi090-93363-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:35.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:34 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:35.415 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:14:35.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:35.440 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:35.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:14:35.541 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:14:35.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:35.567 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:35.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:14:35.668 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:14:35.692 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:35.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:35.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:14:35.794 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:14:35.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:35.820 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:35.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:14:35.838 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: Running main() from gmock_main.cc 2024-04-14T15:14:35.838 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [==========] Running 57 tests from 4 test suites. 2024-04-14T15:14:35.838 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [----------] Global test environment set-up. 2024-04-14T15:14:35.839 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio 2024-04-14T15:14:35.839 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2024-04-14T15:14:35.839 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (2928 ms) 2024-04-14T15:14:35.839 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2024-04-14T15:14:35.839 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (20768 ms) 2024-04-14T15:14:35.839 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2024-04-14T15:14:35.839 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (6020 ms) 2024-04-14T15:14:35.839 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2024-04-14T15:14:35.839 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (2992 ms) 2024-04-14T15:14:35.839 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2024-04-14T15:14:35.839 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (3035 ms) 2024-04-14T15:14:35.839 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2024-04-14T15:14:35.839 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (2942 ms) 2024-04-14T15:14:35.839 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2024-04-14T15:14:35.840 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (3006 ms) 2024-04-14T15:14:35.840 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2024-04-14T15:14:35.840 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (2998 ms) 2024-04-14T15:14:35.840 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2024-04-14T15:14:35.840 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (2999 ms) 2024-04-14T15:14:35.840 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2024-04-14T15:14:35.840 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (3012 ms) 2024-04-14T15:14:35.840 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2024-04-14T15:14:35.840 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (2986 ms) 2024-04-14T15:14:35.840 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2024-04-14T15:14:35.840 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (3033 ms) 2024-04-14T15:14:35.840 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2024-04-14T15:14:35.840 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (2985 ms) 2024-04-14T15:14:35.840 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2024-04-14T15:14:35.841 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (2986 ms) 2024-04-14T15:14:35.841 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2024-04-14T15:14:35.841 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (2996 ms) 2024-04-14T15:14:35.841 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2024-04-14T15:14:35.841 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (2998 ms) 2024-04-14T15:14:35.841 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2024-04-14T15:14:35.841 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (3002 ms) 2024-04-14T15:14:35.841 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2024-04-14T15:14:35.841 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (3007 ms) 2024-04-14T15:14:35.841 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2024-04-14T15:14:35.841 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (3020 ms) 2024-04-14T15:14:35.841 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2024-04-14T15:14:35.841 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (2988 ms) 2024-04-14T15:14:35.841 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2024-04-14T15:14:35.841 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (3013 ms) 2024-04-14T15:14:35.842 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2024-04-14T15:14:35.842 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (3004 ms) 2024-04-14T15:14:35.842 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2024-04-14T15:14:35.842 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (3000 ms) 2024-04-14T15:14:35.842 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2024-04-14T15:14:35.842 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (2980 ms) 2024-04-14T15:14:35.842 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2024-04-14T15:14:35.842 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (3005 ms) 2024-04-14T15:14:35.842 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2024-04-14T15:14:35.842 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (3001 ms) 2024-04-14T15:14:35.842 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2024-04-14T15:14:35.921 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:14:35.928 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-14T15:14:35.928 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-14T15:14:35.928 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-04-14T15:14:35.928 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-14T15:14:35.930 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-14T15:14:35.930 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-04-14T15:14:35.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:35.945 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:35.946 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:14:35.964 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:35 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-17"}]': finished 2024-04-14T15:14:35.965 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:35 smithi090 ceph-mon[30399]: from='client.48457 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi090-93363-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:35.965 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:35 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-17", "mode": "writeback"}]: dispatch 2024-04-14T15:14:35.965 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:35 smithi090 ceph-mon[30399]: osdmap e207: 8 total, 8 up, 8 in 2024-04-14T15:14:35.965 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:35 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-17", "mode": "writeback"}]: dispatch 2024-04-14T15:14:35.965 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:35 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:35.965 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:35 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:14:35.965 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:35 smithi090 ceph-mon[30399]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:14:35.965 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:35 smithi090 ceph-mon[30399]: from='client.48454 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi090-93342-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi090-93342-30"}]': finished 2024-04-14T15:14:35.965 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:35 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-17", "mode": "writeback"}]': finished 2024-04-14T15:14:35.965 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:35 smithi090 ceph-mon[30399]: osdmap e208: 8 total, 8 up, 8 in 2024-04-14T15:14:35.965 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:35 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-17"}]': finished 2024-04-14T15:14:35.966 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:35 smithi090 ceph-mon[41670]: from='client.48457 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi090-93363-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:35.966 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:35 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-17", "mode": "writeback"}]: dispatch 2024-04-14T15:14:35.966 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:35 smithi090 ceph-mon[41670]: osdmap e207: 8 total, 8 up, 8 in 2024-04-14T15:14:35.966 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:35 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-17", "mode": "writeback"}]: dispatch 2024-04-14T15:14:35.966 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:35 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:35.966 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:35 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:14:35.966 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:35 smithi090 ceph-mon[41670]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:14:35.967 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:35 smithi090 ceph-mon[41670]: from='client.48454 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi090-93342-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi090-93342-30"}]': finished 2024-04-14T15:14:35.967 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:35 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-17", "mode": "writeback"}]': finished 2024-04-14T15:14:35.967 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:35 smithi090 ceph-mon[41670]: osdmap e208: 8 total, 8 up, 8 in 2024-04-14T15:14:36.046 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:14:36.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:36.071 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:36.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:14:36.172 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:14:36.183 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:14:36.197 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:36.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:36.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:14:36.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:36.179+0000 7f43728e7700 1 -- 172.21.15.90:0/1188644717 >> v1:172.21.15.90:6789/0 conn(0x7f436c106f80 legacy=0x7f436c107360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:36.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:36.180+0000 7f43728e7700 1 -- 172.21.15.90:0/1188644717 shutdown_connections 2024-04-14T15:14:36.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:36.180+0000 7f43728e7700 1 -- 172.21.15.90:0/1188644717 >> 172.21.15.90:0/1188644717 conn(0x7f436c076aa0 msgr2=0x7f436c076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:14:36.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:36.180+0000 7f43728e7700 1 -- 172.21.15.90:0/1188644717 shutdown_connections 2024-04-14T15:14:36.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:36.180+0000 7f43728e7700 1 -- 172.21.15.90:0/1188644717 wait complete. 2024-04-14T15:14:36.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:36.180+0000 7f43728e7700 1 librados: starting msgr at 2024-04-14T15:14:36.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:36.180+0000 7f43728e7700 1 librados: starting objecter 2024-04-14T15:14:36.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:36.180+0000 7f43728e7700 1 Processor -- start 2024-04-14T15:14:36.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:36.180+0000 7f43728e7700 1 -- start start 2024-04-14T15:14:36.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:36.180+0000 7f43728e7700 1 librados: setting wanted keys 2024-04-14T15:14:36.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:36.180+0000 7f43728e7700 1 librados: calling monclient init 2024-04-14T15:14:36.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:36.181+0000 7f43728e7700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f436c1a77f0 con 0x7f436c10d940 2024-04-14T15:14:36.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:36.181+0000 7f43728e7700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f436c1a8a10 con 0x7f436c109da0 2024-04-14T15:14:36.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:36.181+0000 7f43728e7700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f436c1a9c50 con 0x7f436c106f80 2024-04-14T15:14:36.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:36.181+0000 7f4370e84700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f436c10d940 0x7f436c1a5eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:55028/0 (socket says 172.21.15.90:55028) 2024-04-14T15:14:36.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:36.181+0000 7f436bfff700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f436c106f80 0x7f436c100270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:52146/0 (socket says 172.21.15.90:52146) 2024-04-14T15:14:36.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:36.181+0000 7f4370e84700 1 -- 172.21.15.90:0/1306966638 learned_addr learned my addr 172.21.15.90:0/1306966638 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:14:36.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:36.181+0000 7f43697fa700 10 client.?.objecter ms_handle_connect 0x7f436c109da0 2024-04-14T15:14:36.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:36.181+0000 7f43697fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:14:36.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:36.181+0000 7f43697fa700 10 client.?.objecter ms_handle_connect 0x7f436c106f80 2024-04-14T15:14:36.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:36.181+0000 7f43697fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:14:36.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:36.181+0000 7f43697fa700 10 client.?.objecter ms_handle_connect 0x7f436c10d940 2024-04-14T15:14:36.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:36.181+0000 7f43697fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:14:36.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:36.181+0000 7f43697fa700 1 -- 172.21.15.90:0/1306966638 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2853069494 0 0) 0x7f436c1a8a10 con 0x7f436c109da0 2024-04-14T15:14:36.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:36.181+0000 7f43697fa700 1 -- 172.21.15.90:0/1306966638 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f43540046c0 con 0x7f436c109da0 2024-04-14T15:14:36.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:36.181+0000 7f43697fa700 1 -- 172.21.15.90:0/1306966638 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3513092480 0 0) 0x7f436c1a9c50 con 0x7f436c106f80 2024-04-14T15:14:36.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:36.181+0000 7f43697fa700 1 -- 172.21.15.90:0/1306966638 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f436c1a8a10 con 0x7f436c106f80 2024-04-14T15:14:36.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:36.181+0000 7f43697fa700 1 -- 172.21.15.90:0/1306966638 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 253051399 0 0) 0x7f436c1a77f0 con 0x7f436c10d940 2024-04-14T15:14:36.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:36.181+0000 7f43697fa700 1 -- 172.21.15.90:0/1306966638 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f436c1a9c50 con 0x7f436c10d940 2024-04-14T15:14:36.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:36.181+0000 7f43697fa700 1 -- 172.21.15.90:0/1306966638 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1449338979 0 0) 0x7f43540046c0 con 0x7f436c109da0 2024-04-14T15:14:36.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:36.182+0000 7f43697fa700 1 -- 172.21.15.90:0/1306966638 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f436c1a77f0 con 0x7f436c109da0 2024-04-14T15:14:36.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:36.182+0000 7f43697fa700 1 -- 172.21.15.90:0/1306966638 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f4360001130 con 0x7f436c109da0 2024-04-14T15:14:36.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:36.182+0000 7f43697fa700 1 -- 172.21.15.90:0/1306966638 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1695310860 0 0) 0x7f436c1a8a10 con 0x7f436c106f80 2024-04-14T15:14:36.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:36.182+0000 7f43697fa700 1 -- 172.21.15.90:0/1306966638 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f43540046c0 con 0x7f436c106f80 2024-04-14T15:14:36.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:36.182+0000 7f43697fa700 1 -- 172.21.15.90:0/1306966638 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f435c004540 con 0x7f436c106f80 2024-04-14T15:14:36.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:36.182+0000 7f43697fa700 1 -- 172.21.15.90:0/1306966638 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2389018754 0 0) 0x7f436c1a77f0 con 0x7f436c109da0 2024-04-14T15:14:36.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:36.182+0000 7f43697fa700 1 -- 172.21.15.90:0/1306966638 >> v1:172.21.15.90:6790/0 conn(0x7f436c106f80 legacy=0x7f436c100270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:36.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:36.182+0000 7f43697fa700 1 -- 172.21.15.90:0/1306966638 >> v1:172.21.15.90:6789/0 conn(0x7f436c10d940 legacy=0x7f436c1a5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:36.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:36.182+0000 7f43697fa700 1 -- 172.21.15.90:0/1306966638 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f436c1aae90 con 0x7f436c109da0 2024-04-14T15:14:36.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:36.182+0000 7f43728e7700 1 -- 172.21.15.90:0/1306966638 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f436c1a9e80 con 0x7f436c109da0 2024-04-14T15:14:36.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:36.182+0000 7f43728e7700 10 client.48469.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:14:36.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:36.182+0000 7f43728e7700 1 -- 172.21.15.90:0/1306966638 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f436c1aa340 con 0x7f436c109da0 2024-04-14T15:14:36.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:36.182+0000 7f43728e7700 1 librados: init done 2024-04-14T15:14:36.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:36.182+0000 7f43697fa700 1 -- 172.21.15.90:0/1306966638 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f4360003b20 con 0x7f436c109da0 2024-04-14T15:14:36.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:36.182+0000 7f43697fa700 1 -- 172.21.15.90:0/1306966638 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f4360004a30 con 0x7f436c109da0 2024-04-14T15:14:36.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:36.183+0000 7f43728e7700 1 -- 172.21.15.90:0/1306966638 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f436c102630 con 0x7f436c109da0 2024-04-14T15:14:36.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:36.183+0000 7f43697fa700 1 -- 172.21.15.90:0/1306966638 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f4360003680 con 0x7f436c109da0 2024-04-14T15:14:36.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:36.184+0000 7f43697fa700 1 -- 172.21.15.90:0/1306966638 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(208..208 src has 51..208) ==== 9231+0+0 (unknown 3215815959 0 0) 0x7f4360089e40 con 0x7f436c109da0 2024-04-14T15:14:36.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:36.184+0000 7f43697fa700 10 client.48469.objecter ms_dispatch 0x7f436c001550 osd_map(208..208 src has 51..208) 2024-04-14T15:14:36.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:36.184+0000 7f43697fa700 3 client.48469.objecter handle_osd_map got epochs [208,208] > 0 2024-04-14T15:14:36.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:36.184+0000 7f43697fa700 3 client.48469.objecter handle_osd_map decoding full epoch 208 2024-04-14T15:14:36.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:36.184+0000 7f43697fa700 20 client.48469.objecter dump_active .. 0 homeless 2024-04-14T15:14:36.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:36.185+0000 7f43697fa700 10 client.48469.objecter ms_handle_connect 0x7f43540711e0 2024-04-14T15:14:36.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:36.189+0000 7f43697fa700 1 -- 172.21.15.90:0/1306966638 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f4360056420 con 0x7f436c109da0 2024-04-14T15:14:36.299 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:36.305 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:35 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-17"}]': finished 2024-04-14T15:14:36.305 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:35 smithi092 ceph-mon[33966]: from='client.48457 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi090-93363-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:36.305 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:35 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-17", "mode": "writeback"}]: dispatch 2024-04-14T15:14:36.305 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:35 smithi092 ceph-mon[33966]: osdmap e207: 8 total, 8 up, 8 in 2024-04-14T15:14:36.305 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:35 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-17", "mode": "writeback"}]: dispatch 2024-04-14T15:14:36.305 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:35 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:36.305 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:35 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:14:36.305 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:35 smithi092 ceph-mon[33966]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:14:36.305 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:35 smithi092 ceph-mon[33966]: from='client.48454 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi090-93342-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi090-93342-30"}]': finished 2024-04-14T15:14:36.305 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:35 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-17", "mode": "writeback"}]': finished 2024-04-14T15:14:36.305 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:35 smithi092 ceph-mon[33966]: osdmap e208: 8 total, 8 up, 8 in 2024-04-14T15:14:36.323 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:36.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:36.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:36.393 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-14T15:14:36.394 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:14:36.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:36.408+0000 7f43728e7700 1 -- 172.21.15.90:0/1306966638 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7f436c101690 con 0x7f436c109da0 2024-04-14T15:14:36.425 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:14:36.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:36.449 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:36.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:14:36.550 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:36.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:36.574 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:36.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:36.675 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:14:36.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:36.700 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:36.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:14:36.801 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:36.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:36.825 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:36.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:36.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:36.835+0000 7f43697fa700 1 -- 172.21.15.90:0/1306966638 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v209) ==== 91+0+0 (unknown 3582658995 0 0) 0x7f436005a0d0 con 0x7f436c109da0 2024-04-14T15:14:36.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:36.839+0000 7f43728e7700 10 librados: watch_flush enter 2024-04-14T15:14:36.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:36.839+0000 7f43728e7700 10 librados: watch_flush exit 2024-04-14T15:14:36.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:36.839+0000 7f43728e7700 20 client.48469.objecter shutdown clearing up homeless session... 2024-04-14T15:14:36.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:36.839+0000 7f43728e7700 10 client.48469.objecter successfully canceled tick 2024-04-14T15:14:36.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:36.839+0000 7f43728e7700 1 -- 172.21.15.90:0/1306966638 >> v1:172.21.15.90:6800/420935427 conn(0x7f43540711e0 legacy=0x7f43540736a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:36.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:36.839+0000 7f43728e7700 1 -- 172.21.15.90:0/1306966638 >> v1:172.21.15.92:6789/0 conn(0x7f436c109da0 legacy=0x7f436c100980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:36.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:36.839+0000 7f43728e7700 1 -- 172.21.15.90:0/1306966638 shutdown_connections 2024-04-14T15:14:36.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:36.839+0000 7f43728e7700 1 -- 172.21.15.90:0/1306966638 >> 172.21.15.90:0/1306966638 conn(0x7f436c076aa0 msgr2=0x7f436c110940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:14:36.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:36.841+0000 7f43728e7700 1 -- 172.21.15.90:0/1306966638 shutdown_connections 2024-04-14T15:14:36.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:36.841+0000 7f43728e7700 1 -- 172.21.15.90:0/1306966638 wait complete. 2024-04-14T15:14:36.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:36.841+0000 7f43728e7700 1 librados: shutdown 2024-04-14T15:14:36.844 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is unset 2024-04-14T15:14:36.927 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:14:36.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:36.953 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:36.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:14:37.054 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:14:37.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:37.079 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:37.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:14:37.180 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:14:37.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:36 smithi090 ceph-mon[30399]: pgmap v257: 291 pgs: 32 creating+peering, 2 peering, 8 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 241 active+clean; 534 KiB data, 541 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-14T15:14:37.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:36 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:14:37.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:36 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:14:37.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:36 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:37.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:36 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:14:37.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:36 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:14:37.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:36 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1306966638' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:14:37.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:36 smithi090 ceph-mon[30399]: from='client.48469 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:14:37.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:36 smithi090 ceph-mon[30399]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:14:37.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:36 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:14:37.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:36 smithi090 ceph-mon[30399]: from='client.48469 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-14T15:14:37.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:36 smithi090 ceph-mon[30399]: osdmap e209: 8 total, 8 up, 8 in 2024-04-14T15:14:37.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:36 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2377717316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi090-93363-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:37.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:36 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-17"}]: dispatch 2024-04-14T15:14:37.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:36 smithi090 ceph-mon[30399]: from='client.46568 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi090-93363-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:37.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:36 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-17"}]: dispatch 2024-04-14T15:14:37.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:36 smithi090 ceph-mon[41670]: pgmap v257: 291 pgs: 32 creating+peering, 2 peering, 8 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 241 active+clean; 534 KiB data, 541 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-14T15:14:37.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:36 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:14:37.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:36 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:14:37.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:36 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:37.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:36 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:14:37.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:36 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:14:37.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:36 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1306966638' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:14:37.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:36 smithi090 ceph-mon[41670]: from='client.48469 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:14:37.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:36 smithi090 ceph-mon[41670]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:14:37.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:36 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:14:37.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:36 smithi090 ceph-mon[41670]: from='client.48469 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-14T15:14:37.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:36 smithi090 ceph-mon[41670]: osdmap e209: 8 total, 8 up, 8 in 2024-04-14T15:14:37.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:36 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2377717316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi090-93363-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:37.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:36 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-17"}]: dispatch 2024-04-14T15:14:37.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:36 smithi090 ceph-mon[41670]: from='client.46568 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi090-93363-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:37.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:36 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-17"}]: dispatch 2024-04-14T15:14:37.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:37.207 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:37.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:14:37.219 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-14T15:14:37.308 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:14:37.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:37.333 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:37.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:14:37.369 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:36 smithi092 ceph-mon[33966]: pgmap v257: 291 pgs: 32 creating+peering, 2 peering, 8 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 241 active+clean; 534 KiB data, 541 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-14T15:14:37.370 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:36 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:14:37.370 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:36 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:14:37.370 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:36 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:37.370 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:36 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:14:37.370 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:36 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:14:37.370 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:36 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1306966638' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:14:37.370 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:36 smithi092 ceph-mon[33966]: from='client.48469 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:14:37.370 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:36 smithi092 ceph-mon[33966]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:14:37.370 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:36 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:14:37.370 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:36 smithi092 ceph-mon[33966]: from='client.48469 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-14T15:14:37.370 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:36 smithi092 ceph-mon[33966]: osdmap e209: 8 total, 8 up, 8 in 2024-04-14T15:14:37.370 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:36 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2377717316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi090-93363-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:37.370 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:36 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-17"}]: dispatch 2024-04-14T15:14:37.370 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:36 smithi092 ceph-mon[33966]: from='client.46568 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi090-93363-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:37.371 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:36 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-17"}]: dispatch 2024-04-14T15:14:37.434 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:14:37.459 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:37.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:37.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:14:37.467 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-14T15:14:37.561 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:37.585 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:37.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:37.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:37.687 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:14:37.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:37.715 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:37.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:14:37.816 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:14:37.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:37.849 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:37.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:14:37.882 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:14:37.882 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:14:37.950 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:14:37.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:37.975 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:37.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:14:37.980 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-04-14T15:14:37.980 INFO:teuthology.orchestra.run.smithi092.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-14T15:14:37.980 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-04-14T15:14:37.980 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:37 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:37.981 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:37 smithi092 ceph-mon[33966]: 2.8 scrub starts 2024-04-14T15:14:37.981 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:37 smithi092 ceph-mon[33966]: 2.8 scrub ok 2024-04-14T15:14:37.981 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:37 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-04-14T15:14:37.981 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:37 smithi092 ceph-mon[33966]: 5.15 scrub starts 2024-04-14T15:14:37.981 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:37 smithi092 ceph-mon[33966]: 5.15 scrub ok 2024-04-14T15:14:37.981 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:37 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:14:37.981 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:37 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:14:37.981 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:37 smithi092 ceph-mon[33966]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:14:37.981 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:37 smithi092 ceph-mon[33966]: from='client.46568 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi090-93363-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:37.981 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:37 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-17"}]': finished 2024-04-14T15:14:37.981 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:37 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "19"}]': finished 2024-04-14T15:14:37.981 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:37 smithi092 ceph-mon[33966]: osdmap e210: 8 total, 8 up, 8 in 2024-04-14T15:14:37.981 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:37 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2834621764' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi090-93342-30"}]: dispatch 2024-04-14T15:14:37.981 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:37 smithi092 ceph-mon[33966]: from='client.48454 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi090-93342-30"}]: dispatch 2024-04-14T15:14:38.032 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:37 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:38.032 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:37 smithi090 ceph-mon[41670]: 2.8 scrub starts 2024-04-14T15:14:38.032 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:37 smithi090 ceph-mon[41670]: 2.8 scrub ok 2024-04-14T15:14:38.032 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:37 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-04-14T15:14:38.032 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:37 smithi090 ceph-mon[41670]: 5.15 scrub starts 2024-04-14T15:14:38.032 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:37 smithi090 ceph-mon[41670]: 5.15 scrub ok 2024-04-14T15:14:38.032 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:37 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:14:38.032 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:37 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:14:38.032 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:37 smithi090 ceph-mon[41670]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:14:38.032 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:37 smithi090 ceph-mon[41670]: from='client.46568 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi090-93363-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:38.032 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:37 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-17"}]': finished 2024-04-14T15:14:38.033 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:37 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "19"}]': finished 2024-04-14T15:14:38.033 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:37 smithi090 ceph-mon[41670]: osdmap e210: 8 total, 8 up, 8 in 2024-04-14T15:14:38.033 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:37 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2834621764' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi090-93342-30"}]: dispatch 2024-04-14T15:14:38.033 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:37 smithi090 ceph-mon[41670]: from='client.48454 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi090-93342-30"}]: dispatch 2024-04-14T15:14:38.033 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:37 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:38.033 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:37 smithi090 ceph-mon[30399]: 2.8 scrub starts 2024-04-14T15:14:38.033 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:37 smithi090 ceph-mon[30399]: 2.8 scrub ok 2024-04-14T15:14:38.033 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:37 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-04-14T15:14:38.033 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:37 smithi090 ceph-mon[30399]: 5.15 scrub starts 2024-04-14T15:14:38.033 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:37 smithi090 ceph-mon[30399]: 5.15 scrub ok 2024-04-14T15:14:38.033 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:37 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:14:38.033 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:37 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:14:38.034 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:37 smithi090 ceph-mon[30399]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:14:38.034 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:37 smithi090 ceph-mon[30399]: from='client.46568 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi090-93363-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:38.034 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:37 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-17"}]': finished 2024-04-14T15:14:38.034 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:37 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "19"}]': finished 2024-04-14T15:14:38.034 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:37 smithi090 ceph-mon[30399]: osdmap e210: 8 total, 8 up, 8 in 2024-04-14T15:14:38.034 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:37 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2834621764' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi090-93342-30"}]: dispatch 2024-04-14T15:14:38.034 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:37 smithi090 ceph-mon[30399]: from='client.48454 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi090-93342-30"}]: dispatch 2024-04-14T15:14:38.076 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:14:38.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:38.101 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:38.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:14:38.202 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:14:38.226 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:38.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:38.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:14:38.331 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:14:38.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:38.358 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:38.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:14:38.459 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:14:38.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:38.485 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:38.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:14:38.586 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:14:38.614 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:38.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:38.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:14:38.716 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:14:38.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:38.745 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:38.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:14:38.846 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:14:38.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:38.879 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:38.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:14:38.941 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:38 smithi090 ceph-mon[41670]: pgmap v260: 298 pgs: 1 peering, 8 active+clean+snaptrim, 40 unknown, 8 active+clean+snaptrim_wait, 241 active+clean; 534 KiB data, 541 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:14:38.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:38 smithi090 ceph-mon[41670]: 6.1a scrub starts 2024-04-14T15:14:38.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:38 smithi090 ceph-mon[41670]: 6.1a scrub ok 2024-04-14T15:14:38.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:38 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:38.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:38 smithi090 ceph-mon[41670]: 6.1c scrub starts 2024-04-14T15:14:38.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:38 smithi090 ceph-mon[41670]: 6.1c scrub ok 2024-04-14T15:14:38.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:38 smithi090 ceph-mon[41670]: 6.1b scrub starts 2024-04-14T15:14:38.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:38 smithi090 ceph-mon[41670]: 6.1b scrub ok 2024-04-14T15:14:38.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:38 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:14:38.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:38 smithi090 ceph-mon[41670]: 2.6 scrub starts 2024-04-14T15:14:38.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:38 smithi090 ceph-mon[41670]: 2.6 scrub ok 2024-04-14T15:14:38.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:38 smithi090 ceph-mon[41670]: from='client.48454 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi090-93342-30"}]': finished 2024-04-14T15:14:38.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:38 smithi090 ceph-mon[41670]: osdmap e211: 8 total, 8 up, 8 in 2024-04-14T15:14:38.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:38 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2834621764' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi090-93342-30"}]: dispatch 2024-04-14T15:14:38.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:38 smithi090 ceph-mon[41670]: from='client.48454 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi090-93342-30"}]: dispatch 2024-04-14T15:14:38.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:38 smithi090 ceph-mon[30399]: pgmap v260: 298 pgs: 1 peering, 8 active+clean+snaptrim, 40 unknown, 8 active+clean+snaptrim_wait, 241 active+clean; 534 KiB data, 541 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:14:38.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:38 smithi090 ceph-mon[30399]: 6.1a scrub starts 2024-04-14T15:14:38.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:38 smithi090 ceph-mon[30399]: 6.1a scrub ok 2024-04-14T15:14:38.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:38 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:38.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:38 smithi090 ceph-mon[30399]: 6.1c scrub starts 2024-04-14T15:14:38.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:38 smithi090 ceph-mon[30399]: 6.1c scrub ok 2024-04-14T15:14:38.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:38 smithi090 ceph-mon[30399]: 6.1b scrub starts 2024-04-14T15:14:38.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:38 smithi090 ceph-mon[30399]: 6.1b scrub ok 2024-04-14T15:14:38.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:38 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:14:38.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:38 smithi090 ceph-mon[30399]: 2.6 scrub starts 2024-04-14T15:14:38.944 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:38 smithi090 ceph-mon[30399]: 2.6 scrub ok 2024-04-14T15:14:38.944 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:38 smithi090 ceph-mon[30399]: from='client.48454 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi090-93342-30"}]': finished 2024-04-14T15:14:38.944 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:38 smithi090 ceph-mon[30399]: osdmap e211: 8 total, 8 up, 8 in 2024-04-14T15:14:38.944 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:38 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2834621764' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi090-93342-30"}]: dispatch 2024-04-14T15:14:38.944 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:38 smithi090 ceph-mon[30399]: from='client.48454 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi090-93342-30"}]: dispatch 2024-04-14T15:14:38.981 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:14:39.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:39.007 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:39.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:14:39.108 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:14:39.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:38 smithi092 ceph-mon[33966]: pgmap v260: 298 pgs: 1 peering, 8 active+clean+snaptrim, 40 unknown, 8 active+clean+snaptrim_wait, 241 active+clean; 534 KiB data, 541 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:14:39.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:38 smithi092 ceph-mon[33966]: 6.1a scrub starts 2024-04-14T15:14:39.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:38 smithi092 ceph-mon[33966]: 6.1a scrub ok 2024-04-14T15:14:39.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:38 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:39.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:38 smithi092 ceph-mon[33966]: 6.1c scrub starts 2024-04-14T15:14:39.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:38 smithi092 ceph-mon[33966]: 6.1c scrub ok 2024-04-14T15:14:39.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:38 smithi092 ceph-mon[33966]: 6.1b scrub starts 2024-04-14T15:14:39.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:38 smithi092 ceph-mon[33966]: 6.1b scrub ok 2024-04-14T15:14:39.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:38 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:14:39.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:38 smithi092 ceph-mon[33966]: 2.6 scrub starts 2024-04-14T15:14:39.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:38 smithi092 ceph-mon[33966]: 2.6 scrub ok 2024-04-14T15:14:39.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:38 smithi092 ceph-mon[33966]: from='client.48454 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi090-93342-30"}]': finished 2024-04-14T15:14:39.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:38 smithi092 ceph-mon[33966]: osdmap e211: 8 total, 8 up, 8 in 2024-04-14T15:14:39.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:38 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2834621764' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi090-93342-30"}]: dispatch 2024-04-14T15:14:39.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:38 smithi092 ceph-mon[33966]: from='client.48454 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi090-93342-30"}]: dispatch 2024-04-14T15:14:39.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:39.133 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:39.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:14:39.235 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:14:39.261 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:39.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:39.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:14:39.363 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:14:39.389 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:39.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:39.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:14:39.492 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:14:39.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:39.519 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:39.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:14:39.620 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:39.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:39.644 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:39.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:39.745 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:14:39.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:39.773 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:39.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:14:39.815 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-14T15:14:39.874 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:14:39.899 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:39.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:39.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:14:40.001 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:14:40.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:40.025 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:40.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:14:40.126 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:14:40.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:40.153 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:40.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:14:40.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:39 smithi090 ceph-mon[41670]: 1.5 scrub starts 2024-04-14T15:14:40.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:39 smithi090 ceph-mon[41670]: 1.5 scrub ok 2024-04-14T15:14:40.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:39 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:40.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:39 smithi090 ceph-mon[41670]: 1.6 scrub starts 2024-04-14T15:14:40.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:39 smithi090 ceph-mon[41670]: 1.6 scrub ok 2024-04-14T15:14:40.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:39 smithi090 ceph-mon[41670]: 1.a scrub starts 2024-04-14T15:14:40.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:39 smithi090 ceph-mon[41670]: 1.a scrub ok 2024-04-14T15:14:40.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:39 smithi090 ceph-mon[41670]: from='client.48454 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi090-93342-30"}]': finished 2024-04-14T15:14:40.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:39 smithi090 ceph-mon[41670]: osdmap e212: 8 total, 8 up, 8 in 2024-04-14T15:14:40.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:39 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3183956142' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi090-93363-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:40.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:39 smithi090 ceph-mon[41670]: from='client.48475 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi090-93363-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:40.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:39 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:40.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:39 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:40.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:39 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3131004652' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi090-93342-31"}]: dispatch 2024-04-14T15:14:40.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:39 smithi090 ceph-mon[41670]: from='client.48481 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi090-93342-31"}]: dispatch 2024-04-14T15:14:40.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:39 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3131004652' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi090-93342-31"}]: dispatch 2024-04-14T15:14:40.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:39 smithi090 ceph-mon[41670]: from='client.48481 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi090-93342-31"}]: dispatch 2024-04-14T15:14:40.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:39 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3131004652' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi090-93342-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:14:40.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:39 smithi090 ceph-mon[41670]: from='client.48481 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi090-93342-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:14:40.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:39 smithi090 ceph-mon[30399]: 1.5 scrub starts 2024-04-14T15:14:40.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:39 smithi090 ceph-mon[30399]: 1.5 scrub ok 2024-04-14T15:14:40.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:39 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:40.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:39 smithi090 ceph-mon[30399]: 1.6 scrub starts 2024-04-14T15:14:40.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:39 smithi090 ceph-mon[30399]: 1.6 scrub ok 2024-04-14T15:14:40.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:39 smithi090 ceph-mon[30399]: 1.a scrub starts 2024-04-14T15:14:40.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:39 smithi090 ceph-mon[30399]: 1.a scrub ok 2024-04-14T15:14:40.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:39 smithi090 ceph-mon[30399]: from='client.48454 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi090-93342-30"}]': finished 2024-04-14T15:14:40.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:39 smithi090 ceph-mon[30399]: osdmap e212: 8 total, 8 up, 8 in 2024-04-14T15:14:40.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:39 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3183956142' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi090-93363-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:40.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:39 smithi090 ceph-mon[30399]: from='client.48475 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi090-93363-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:40.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:39 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:40.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:39 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:40.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:39 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3131004652' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi090-93342-31"}]: dispatch 2024-04-14T15:14:40.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:39 smithi090 ceph-mon[30399]: from='client.48481 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi090-93342-31"}]: dispatch 2024-04-14T15:14:40.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:39 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3131004652' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi090-93342-31"}]: dispatch 2024-04-14T15:14:40.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:39 smithi090 ceph-mon[30399]: from='client.48481 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi090-93342-31"}]: dispatch 2024-04-14T15:14:40.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:39 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3131004652' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi090-93342-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:14:40.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:39 smithi090 ceph-mon[30399]: from='client.48481 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi090-93342-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:14:40.254 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:14:40.279 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:40.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:40.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:14:40.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:39 smithi092 ceph-mon[33966]: 1.5 scrub starts 2024-04-14T15:14:40.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:39 smithi092 ceph-mon[33966]: 1.5 scrub ok 2024-04-14T15:14:40.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:39 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:40.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:39 smithi092 ceph-mon[33966]: 1.6 scrub starts 2024-04-14T15:14:40.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:39 smithi092 ceph-mon[33966]: 1.6 scrub ok 2024-04-14T15:14:40.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:39 smithi092 ceph-mon[33966]: 1.a scrub starts 2024-04-14T15:14:40.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:39 smithi092 ceph-mon[33966]: 1.a scrub ok 2024-04-14T15:14:40.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:39 smithi092 ceph-mon[33966]: from='client.48454 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi090-93342-30"}]': finished 2024-04-14T15:14:40.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:39 smithi092 ceph-mon[33966]: osdmap e212: 8 total, 8 up, 8 in 2024-04-14T15:14:40.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:39 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3183956142' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi090-93363-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:40.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:39 smithi092 ceph-mon[33966]: from='client.48475 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi090-93363-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:40.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:39 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:40.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:39 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:40.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:39 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3131004652' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi090-93342-31"}]: dispatch 2024-04-14T15:14:40.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:39 smithi092 ceph-mon[33966]: from='client.48481 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi090-93342-31"}]: dispatch 2024-04-14T15:14:40.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:39 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3131004652' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi090-93342-31"}]: dispatch 2024-04-14T15:14:40.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:39 smithi092 ceph-mon[33966]: from='client.48481 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi090-93342-31"}]: dispatch 2024-04-14T15:14:40.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:39 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3131004652' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi090-93342-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:14:40.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:39 smithi092 ceph-mon[33966]: from='client.48481 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi090-93342-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:14:40.385 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:40.409 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:40.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:40.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:40.511 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:14:40.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:40.537 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:40.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:14:40.638 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:14:40.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:40.664 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:40.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:14:40.765 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:14:40.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:40.791 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:40.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:14:40.834 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-14T15:14:40.834 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:14:40.892 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:14:40.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:40.918 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:40.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:14:41.019 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:14:41.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:41.045 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:41.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:14:41.146 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:14:41.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:41.170 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:41.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:14:41.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:40 smithi090 ceph-mon[41670]: pgmap v263: 226 pgs: 6 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 213 active+clean; 482 KiB data, 546 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:14:41.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:40 smithi090 ceph-mon[41670]: 13.8 scrub starts 2024-04-14T15:14:41.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:40 smithi090 ceph-mon[41670]: 13.8 scrub ok 2024-04-14T15:14:41.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:40 smithi090 ceph-mon[41670]: 13.6 scrub starts 2024-04-14T15:14:41.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:40 smithi090 ceph-mon[41670]: 13.6 scrub ok 2024-04-14T15:14:41.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:40 smithi090 ceph-mon[41670]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:14:41.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:40 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:41.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:40 smithi090 ceph-mon[41670]: 1.4 scrub starts 2024-04-14T15:14:41.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:40 smithi090 ceph-mon[41670]: 1.4 scrub ok 2024-04-14T15:14:41.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:40 smithi090 ceph-mon[41670]: 1.7 scrub starts 2024-04-14T15:14:41.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:40 smithi090 ceph-mon[41670]: 1.7 scrub ok 2024-04-14T15:14:41.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:40 smithi090 ceph-mon[41670]: 2.a scrub starts 2024-04-14T15:14:41.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:40 smithi090 ceph-mon[41670]: 2.a scrub ok 2024-04-14T15:14:41.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:40 smithi090 ceph-mon[41670]: 1.8 scrub starts 2024-04-14T15:14:41.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:40 smithi090 ceph-mon[41670]: 1.8 scrub ok 2024-04-14T15:14:41.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:40 smithi090 ceph-mon[41670]: from='client.48475 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi090-93363-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:41.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:40 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:41.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:40 smithi090 ceph-mon[41670]: from='client.48481 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi090-93342-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:14:41.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:40 smithi090 ceph-mon[41670]: osdmap e213: 8 total, 8 up, 8 in 2024-04-14T15:14:41.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:40 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3131004652' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi090-93342-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi090-93342-31"}]: dispatch 2024-04-14T15:14:41.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:40 smithi090 ceph-mon[41670]: from='client.48481 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi090-93342-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi090-93342-31"}]: dispatch 2024-04-14T15:14:41.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:40 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:14:41.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:40 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:14:41.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:40 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:41.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:40 smithi090 ceph-mon[30399]: pgmap v263: 226 pgs: 6 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 213 active+clean; 482 KiB data, 546 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:14:41.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:40 smithi090 ceph-mon[30399]: 13.8 scrub starts 2024-04-14T15:14:41.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:40 smithi090 ceph-mon[30399]: 13.8 scrub ok 2024-04-14T15:14:41.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:40 smithi090 ceph-mon[30399]: 13.6 scrub starts 2024-04-14T15:14:41.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:40 smithi090 ceph-mon[30399]: 13.6 scrub ok 2024-04-14T15:14:41.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:40 smithi090 ceph-mon[30399]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:14:41.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:40 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:41.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:40 smithi090 ceph-mon[30399]: 1.4 scrub starts 2024-04-14T15:14:41.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:40 smithi090 ceph-mon[30399]: 1.4 scrub ok 2024-04-14T15:14:41.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:40 smithi090 ceph-mon[30399]: 1.7 scrub starts 2024-04-14T15:14:41.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:40 smithi090 ceph-mon[30399]: 1.7 scrub ok 2024-04-14T15:14:41.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:40 smithi090 ceph-mon[30399]: 2.a scrub starts 2024-04-14T15:14:41.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:40 smithi090 ceph-mon[30399]: 2.a scrub ok 2024-04-14T15:14:41.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:40 smithi090 ceph-mon[30399]: 1.8 scrub starts 2024-04-14T15:14:41.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:40 smithi090 ceph-mon[30399]: 1.8 scrub ok 2024-04-14T15:14:41.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:40 smithi090 ceph-mon[30399]: from='client.48475 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi090-93363-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:41.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:40 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:41.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:40 smithi090 ceph-mon[30399]: from='client.48481 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi090-93342-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:14:41.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:40 smithi090 ceph-mon[30399]: osdmap e213: 8 total, 8 up, 8 in 2024-04-14T15:14:41.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:40 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3131004652' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi090-93342-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi090-93342-31"}]: dispatch 2024-04-14T15:14:41.196 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:40 smithi090 ceph-mon[30399]: from='client.48481 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi090-93342-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi090-93342-31"}]: dispatch 2024-04-14T15:14:41.196 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:40 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:14:41.196 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:40 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:14:41.196 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:40 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:41.271 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:41.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:41.296 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:41.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:41.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:40 smithi092 ceph-mon[33966]: pgmap v263: 226 pgs: 6 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 213 active+clean; 482 KiB data, 546 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:14:41.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:40 smithi092 ceph-mon[33966]: 13.8 scrub starts 2024-04-14T15:14:41.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:40 smithi092 ceph-mon[33966]: 13.8 scrub ok 2024-04-14T15:14:41.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:40 smithi092 ceph-mon[33966]: 13.6 scrub starts 2024-04-14T15:14:41.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:40 smithi092 ceph-mon[33966]: 13.6 scrub ok 2024-04-14T15:14:41.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:40 smithi092 ceph-mon[33966]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:14:41.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:40 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:41.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:40 smithi092 ceph-mon[33966]: 1.4 scrub starts 2024-04-14T15:14:41.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:40 smithi092 ceph-mon[33966]: 1.4 scrub ok 2024-04-14T15:14:41.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:40 smithi092 ceph-mon[33966]: 1.7 scrub starts 2024-04-14T15:14:41.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:40 smithi092 ceph-mon[33966]: 1.7 scrub ok 2024-04-14T15:14:41.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:40 smithi092 ceph-mon[33966]: 2.a scrub starts 2024-04-14T15:14:41.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:40 smithi092 ceph-mon[33966]: 2.a scrub ok 2024-04-14T15:14:41.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:40 smithi092 ceph-mon[33966]: 1.8 scrub starts 2024-04-14T15:14:41.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:40 smithi092 ceph-mon[33966]: 1.8 scrub ok 2024-04-14T15:14:41.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:40 smithi092 ceph-mon[33966]: from='client.48475 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi090-93363-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:41.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:40 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:41.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:40 smithi092 ceph-mon[33966]: from='client.48481 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi090-93342-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:14:41.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:40 smithi092 ceph-mon[33966]: osdmap e213: 8 total, 8 up, 8 in 2024-04-14T15:14:41.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:40 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3131004652' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi090-93342-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi090-93342-31"}]: dispatch 2024-04-14T15:14:41.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:40 smithi092 ceph-mon[33966]: from='client.48481 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi090-93342-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi090-93342-31"}]: dispatch 2024-04-14T15:14:41.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:40 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:14:41.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:40 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:14:41.376 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:40 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:41.397 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:14:41.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:41.428 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:41.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:14:41.529 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:14:41.555 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:41.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:41.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:14:41.621 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-14T15:14:41.657 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:14:41.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:41.683 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:41.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:14:41.784 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:41.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:41.810 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:41.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:41.911 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:14:41.936 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:41.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:41.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:14:42.038 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:14:42.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:42.062 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:42.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:14:42.163 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:14:42.183 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:41 smithi090 ceph-mon[41670]: 13.a scrub starts 2024-04-14T15:14:42.183 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:41 smithi090 ceph-mon[41670]: 13.a scrub ok 2024-04-14T15:14:42.183 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:41 smithi090 ceph-mon[41670]: 1.2 scrub starts 2024-04-14T15:14:42.183 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:41 smithi090 ceph-mon[41670]: 1.2 scrub ok 2024-04-14T15:14:42.183 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:41 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-04-14T15:14:42.183 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:41 smithi090 ceph-mon[41670]: 1.1 scrub starts 2024-04-14T15:14:42.184 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:41 smithi090 ceph-mon[41670]: 1.1 scrub ok 2024-04-14T15:14:42.184 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:41 smithi090 ceph-mon[41670]: 3.1 scrub starts 2024-04-14T15:14:42.184 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:41 smithi090 ceph-mon[41670]: 3.1 scrub ok 2024-04-14T15:14:42.184 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:41 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-19", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:14:42.184 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:41 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "18"}]': finished 2024-04-14T15:14:42.184 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:41 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-19"}]: dispatch 2024-04-14T15:14:42.184 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:41 smithi090 ceph-mon[41670]: osdmap e214: 8 total, 8 up, 8 in 2024-04-14T15:14:42.184 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:41 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-19"}]: dispatch 2024-04-14T15:14:42.184 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:41 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:42.184 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:41 smithi090 ceph-mon[30399]: 13.a scrub starts 2024-04-14T15:14:42.184 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:41 smithi090 ceph-mon[30399]: 13.a scrub ok 2024-04-14T15:14:42.184 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:41 smithi090 ceph-mon[30399]: 1.2 scrub starts 2024-04-14T15:14:42.184 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:41 smithi090 ceph-mon[30399]: 1.2 scrub ok 2024-04-14T15:14:42.185 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:41 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-04-14T15:14:42.185 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:41 smithi090 ceph-mon[30399]: 1.1 scrub starts 2024-04-14T15:14:42.185 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:41 smithi090 ceph-mon[30399]: 1.1 scrub ok 2024-04-14T15:14:42.185 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:41 smithi090 ceph-mon[30399]: 3.1 scrub starts 2024-04-14T15:14:42.185 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:41 smithi090 ceph-mon[30399]: 3.1 scrub ok 2024-04-14T15:14:42.185 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:41 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-19", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:14:42.185 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:41 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "18"}]': finished 2024-04-14T15:14:42.185 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:41 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-19"}]: dispatch 2024-04-14T15:14:42.185 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:41 smithi090 ceph-mon[30399]: osdmap e214: 8 total, 8 up, 8 in 2024-04-14T15:14:42.185 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:41 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-19"}]: dispatch 2024-04-14T15:14:42.185 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:41 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:42.187 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:42.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:42.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:14:42.289 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:14:42.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.287+0000 7ff76619b700 1 -- 172.21.15.90:0/1062353957 >> v1:172.21.15.90:6790/0 conn(0x7ff760107760 legacy=0x7ff760107b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:42.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.287+0000 7ff76619b700 1 -- 172.21.15.90:0/1062353957 shutdown_connections 2024-04-14T15:14:42.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.287+0000 7ff76619b700 1 -- 172.21.15.90:0/1062353957 >> 172.21.15.90:0/1062353957 conn(0x7ff7600fd290 msgr2=0x7ff7600ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:14:42.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.287+0000 7ff76619b700 1 -- 172.21.15.90:0/1062353957 shutdown_connections 2024-04-14T15:14:42.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.287+0000 7ff76619b700 1 -- 172.21.15.90:0/1062353957 wait complete. 2024-04-14T15:14:42.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.287+0000 7ff76619b700 1 librados: starting msgr at 2024-04-14T15:14:42.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.287+0000 7ff76619b700 1 librados: starting objecter 2024-04-14T15:14:42.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.288+0000 7ff76619b700 1 Processor -- start 2024-04-14T15:14:42.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.288+0000 7ff76619b700 1 -- start start 2024-04-14T15:14:42.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.288+0000 7ff76619b700 1 librados: setting wanted keys 2024-04-14T15:14:42.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.288+0000 7ff76619b700 1 librados: calling monclient init 2024-04-14T15:14:42.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.288+0000 7ff76619b700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff7601a9b60 con 0x7ff760107760 2024-04-14T15:14:42.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.288+0000 7ff76619b700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff7601aad80 con 0x7ff76010e1e0 2024-04-14T15:14:42.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.288+0000 7ff76619b700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff7601abfc0 con 0x7ff76010a610 2024-04-14T15:14:42.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.288+0000 7ff75f7fe700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7ff760107760 0x7ff76010d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:54460/0 (socket says 172.21.15.90:54460) 2024-04-14T15:14:42.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.288+0000 7ff75f7fe700 1 -- 172.21.15.90:0/2547188057 learned_addr learned my addr 172.21.15.90:0/2547188057 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:14:42.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.288+0000 7ff75cff9700 10 client.?.objecter ms_handle_connect 0x7ff76010a610 2024-04-14T15:14:42.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.288+0000 7ff75cff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:14:42.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.288+0000 7ff75cff9700 10 client.?.objecter ms_handle_connect 0x7ff760107760 2024-04-14T15:14:42.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.288+0000 7ff75cff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:14:42.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.288+0000 7ff75cff9700 10 client.?.objecter ms_handle_connect 0x7ff76010e1e0 2024-04-14T15:14:42.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.288+0000 7ff75cff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:14:42.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.288+0000 7ff75cff9700 1 -- 172.21.15.90:0/2547188057 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2462201183 0 0) 0x7ff7601abfc0 con 0x7ff76010a610 2024-04-14T15:14:42.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.288+0000 7ff75cff9700 1 -- 172.21.15.90:0/2547188057 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff7480033a0 con 0x7ff76010a610 2024-04-14T15:14:42.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.288+0000 7ff75cff9700 1 -- 172.21.15.90:0/2547188057 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1522945616 0 0) 0x7ff7601a9b60 con 0x7ff760107760 2024-04-14T15:14:42.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.288+0000 7ff75cff9700 1 -- 172.21.15.90:0/2547188057 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff7601abfc0 con 0x7ff760107760 2024-04-14T15:14:42.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.289+0000 7ff75cff9700 1 -- 172.21.15.90:0/2547188057 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 852505039 0 0) 0x7ff7601aad80 con 0x7ff76010e1e0 2024-04-14T15:14:42.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.289+0000 7ff75cff9700 1 -- 172.21.15.90:0/2547188057 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff7601a9b60 con 0x7ff76010e1e0 2024-04-14T15:14:42.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.289+0000 7ff75cff9700 1 -- 172.21.15.90:0/2547188057 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3669388377 0 0) 0x7ff7480033a0 con 0x7ff76010a610 2024-04-14T15:14:42.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.289+0000 7ff75cff9700 1 -- 172.21.15.90:0/2547188057 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff7601aad80 con 0x7ff76010a610 2024-04-14T15:14:42.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.289+0000 7ff75cff9700 1 -- 172.21.15.90:0/2547188057 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7ff7500018b0 con 0x7ff76010a610 2024-04-14T15:14:42.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.289+0000 7ff75cff9700 1 -- 172.21.15.90:0/2547188057 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 307273385 0 0) 0x7ff7601a9b60 con 0x7ff76010e1e0 2024-04-14T15:14:42.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.289+0000 7ff75cff9700 1 -- 172.21.15.90:0/2547188057 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff7480033a0 con 0x7ff76010e1e0 2024-04-14T15:14:42.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.289+0000 7ff75cff9700 1 -- 172.21.15.90:0/2547188057 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2824690289 0 0) 0x7ff7601abfc0 con 0x7ff760107760 2024-04-14T15:14:42.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.289+0000 7ff75cff9700 1 -- 172.21.15.90:0/2547188057 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff7601a9b60 con 0x7ff760107760 2024-04-14T15:14:42.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.289+0000 7ff75cff9700 1 -- 172.21.15.90:0/2547188057 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7ff758001590 con 0x7ff76010e1e0 2024-04-14T15:14:42.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.289+0000 7ff75cff9700 1 -- 172.21.15.90:0/2547188057 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7ff7540044c0 con 0x7ff760107760 2024-04-14T15:14:42.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.289+0000 7ff75cff9700 1 -- 172.21.15.90:0/2547188057 <== mon.2 v1:172.21.15.90:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4084131338 0 0) 0x7ff7601aad80 con 0x7ff76010a610 2024-04-14T15:14:42.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.289+0000 7ff75cff9700 1 -- 172.21.15.90:0/2547188057 >> v1:172.21.15.92:6789/0 conn(0x7ff76010e1e0 legacy=0x7ff7601a8220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:42.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.289+0000 7ff75cff9700 1 -- 172.21.15.90:0/2547188057 >> v1:172.21.15.90:6789/0 conn(0x7ff760107760 legacy=0x7ff76010d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:42.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.289+0000 7ff75cff9700 1 -- 172.21.15.90:0/2547188057 --> v1:172.21.15.90:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7601ad200 con 0x7ff76010a610 2024-04-14T15:14:42.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.289+0000 7ff76619b700 1 -- 172.21.15.90:0/2547188057 --> v1:172.21.15.90:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff7601a9d90 con 0x7ff76010a610 2024-04-14T15:14:42.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.289+0000 7ff76619b700 10 client.46598.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:14:42.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.289+0000 7ff76619b700 1 -- 172.21.15.90:0/2547188057 --> v1:172.21.15.90:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7ff7601aa250 con 0x7ff76010a610 2024-04-14T15:14:42.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.289+0000 7ff76619b700 1 librados: init done 2024-04-14T15:14:42.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.289+0000 7ff75cff9700 1 -- 172.21.15.90:0/2547188057 <== mon.2 v1:172.21.15.90:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7ff750001c60 con 0x7ff76010a610 2024-04-14T15:14:42.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.289+0000 7ff75cff9700 1 -- 172.21.15.90:0/2547188057 <== mon.2 v1:172.21.15.90:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7ff750004b40 con 0x7ff76010a610 2024-04-14T15:14:42.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.290+0000 7ff76619b700 1 -- 172.21.15.90:0/2547188057 --> v1:172.21.15.90:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7ff74c005130 con 0x7ff76010a610 2024-04-14T15:14:42.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.291+0000 7ff75cff9700 1 -- 172.21.15.90:0/2547188057 <== mon.2 v1:172.21.15.90:6790/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7ff75001b550 con 0x7ff76010a610 2024-04-14T15:14:42.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.291+0000 7ff75cff9700 1 -- 172.21.15.90:0/2547188057 <== mon.2 v1:172.21.15.90:6790/0 8 ==== osd_map(214..214 src has 51..214) ==== 8832+0+0 (unknown 764065836 0 0) 0x7ff7500899f0 con 0x7ff76010a610 2024-04-14T15:14:42.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.291+0000 7ff75cff9700 10 client.46598.objecter ms_dispatch 0x7ff760001550 osd_map(214..214 src has 51..214) 2024-04-14T15:14:42.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.291+0000 7ff75cff9700 3 client.46598.objecter handle_osd_map got epochs [214,214] > 0 2024-04-14T15:14:42.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.291+0000 7ff75cff9700 3 client.46598.objecter handle_osd_map decoding full epoch 214 2024-04-14T15:14:42.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.291+0000 7ff75cff9700 20 client.46598.objecter dump_active .. 0 homeless 2024-04-14T15:14:42.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.291+0000 7ff75cff9700 1 -- 172.21.15.90:0/2547188057 <== mon.2 v1:172.21.15.90:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v214) ==== 54+0+16 (unknown 1031289775 0 2420000192) 0x7ff750056180 con 0x7ff76010a610 2024-04-14T15:14:42.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.292+0000 7ff75cff9700 10 client.46598.objecter ms_handle_connect 0x7ff748071310 2024-04-14T15:14:42.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.292+0000 7ff76619b700 10 client.46598.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:14:42.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.292+0000 7ff76619b700 15 client.46598.objecter _session_command_op_assign -1 1 2024-04-14T15:14:42.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.292+0000 7ff76619b700 20 client.46598.objecter _get_session s=0x7ff74c005410 osd=0 3 2024-04-14T15:14:42.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.292+0000 7ff76619b700 20 client.46598.objecter put_session s=0x7ff74c005410 osd=0 3 2024-04-14T15:14:42.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.292+0000 7ff76619b700 20 client.46598.objecter _get_session s=0x7ff74c005410 osd=0 3 2024-04-14T15:14:42.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.292+0000 7ff76619b700 15 client.46598.objecter _session_command_op_remove -1 1 2024-04-14T15:14:42.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.292+0000 7ff76619b700 20 client.46598.objecter get_session s=0x7ff74c005410 osd=0 3 2024-04-14T15:14:42.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.292+0000 7ff76619b700 15 client.46598.objecter _session_command_op_assign 0 1 2024-04-14T15:14:42.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.292+0000 7ff76619b700 20 client.46598.objecter put_session s=0x7ff74c005410 osd=0 4 2024-04-14T15:14:42.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.292+0000 7ff76619b700 10 client.46598.objecter _send_command 1 2024-04-14T15:14:42.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.292+0000 7ff76619b700 1 -- 172.21.15.90:0/2547188057 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff74c004ba0 con 0x7ff74c004760 2024-04-14T15:14:42.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.292+0000 7ff75cff9700 10 client.46598.objecter ms_handle_connect 0x7ff74c004760 2024-04-14T15:14:42.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.293+0000 7ff75cff9700 1 -- 172.21.15.90:0/2547188057 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7ff74c004ba0 con 0x7ff74c004760 2024-04-14T15:14:42.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.293+0000 7ff75cff9700 10 client.46598.objecter ms_dispatch 0x7ff760001550 command_reply(tid 1: 0 ) 2024-04-14T15:14:42.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.293+0000 7ff75cff9700 10 client.46598.objecter _finish_command 1 = system:0 2024-04-14T15:14:42.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.293+0000 7ff75cff9700 20 client.46598.objecter put_session s=0x7ff74c005410 osd=0 4 2024-04-14T15:14:42.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.293+0000 7ff75cff9700 15 client.46598.objecter _session_command_op_remove 0 1 2024-04-14T15:14:42.312 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:42.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.311+0000 7ff76619b700 10 client.46598.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:14:42.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.311+0000 7ff76619b700 15 client.46598.objecter _session_command_op_assign -1 2 2024-04-14T15:14:42.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.311+0000 7ff76619b700 20 client.46598.objecter _get_session s=0x7ff74c005410 osd=0 3 2024-04-14T15:14:42.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.311+0000 7ff76619b700 20 client.46598.objecter put_session s=0x7ff74c005410 osd=0 3 2024-04-14T15:14:42.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.311+0000 7ff76619b700 20 client.46598.objecter _get_session s=0x7ff74c005410 osd=0 3 2024-04-14T15:14:42.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.311+0000 7ff76619b700 15 client.46598.objecter _session_command_op_remove -1 2 2024-04-14T15:14:42.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.311+0000 7ff76619b700 20 client.46598.objecter get_session s=0x7ff74c005410 osd=0 3 2024-04-14T15:14:42.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.311+0000 7ff76619b700 15 client.46598.objecter _session_command_op_assign 0 2 2024-04-14T15:14:42.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.311+0000 7ff76619b700 20 client.46598.objecter put_session s=0x7ff74c005410 osd=0 4 2024-04-14T15:14:42.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.311+0000 7ff76619b700 10 client.46598.objecter _send_command 2 2024-04-14T15:14:42.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.311+0000 7ff76619b700 1 -- 172.21.15.90:0/2547188057 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff74c007a20 con 0x7ff74c004760 2024-04-14T15:14:42.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:42.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:14:42.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.314+0000 7ff75cff9700 1 -- 172.21.15.90:0/2547188057 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff74c007a20 con 0x7ff74c004760 2024-04-14T15:14:42.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.314+0000 7ff75cff9700 10 client.46598.objecter ms_dispatch 0x7ff760001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:14:42.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.314+0000 7ff75cff9700 10 client.46598.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:14:42.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.314+0000 7ff75cff9700 20 client.46598.objecter put_session s=0x7ff74c005410 osd=0 4 2024-04-14T15:14:42.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.314+0000 7ff75cff9700 15 client.46598.objecter _session_command_op_remove 0 2 2024-04-14T15:14:42.315 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:14:42.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.314+0000 7ff76619b700 10 client.46598.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:14:42.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.314+0000 7ff76619b700 15 client.46598.objecter _session_command_op_assign -1 3 2024-04-14T15:14:42.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.314+0000 7ff76619b700 20 client.46598.objecter _get_session s=0x7ff74c004160 osd=1 3 2024-04-14T15:14:42.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.314+0000 7ff76619b700 20 client.46598.objecter put_session s=0x7ff74c004160 osd=1 3 2024-04-14T15:14:42.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.314+0000 7ff76619b700 20 client.46598.objecter _get_session s=0x7ff74c004160 osd=1 3 2024-04-14T15:14:42.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.314+0000 7ff76619b700 15 client.46598.objecter _session_command_op_remove -1 3 2024-04-14T15:14:42.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.314+0000 7ff76619b700 20 client.46598.objecter get_session s=0x7ff74c004160 osd=1 3 2024-04-14T15:14:42.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.314+0000 7ff76619b700 15 client.46598.objecter _session_command_op_assign 1 3 2024-04-14T15:14:42.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.314+0000 7ff76619b700 20 client.46598.objecter put_session s=0x7ff74c004160 osd=1 4 2024-04-14T15:14:42.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.314+0000 7ff76619b700 10 client.46598.objecter _send_command 3 2024-04-14T15:14:42.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.314+0000 7ff76619b700 1 -- 172.21.15.90:0/2547188057 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7ff74c0022d0 con 0x7ff74c004340 2024-04-14T15:14:42.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.315+0000 7ff75cff9700 10 client.46598.objecter ms_handle_connect 0x7ff74c004340 2024-04-14T15:14:42.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.315+0000 7ff75cff9700 1 -- 172.21.15.90:0/2547188057 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7ff74c0022d0 con 0x7ff74c004340 2024-04-14T15:14:42.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.315+0000 7ff75cff9700 10 client.46598.objecter ms_dispatch 0x7ff760001550 command_reply(tid 3: 0 ) 2024-04-14T15:14:42.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.315+0000 7ff75cff9700 10 client.46598.objecter _finish_command 3 = system:0 2024-04-14T15:14:42.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.315+0000 7ff75cff9700 20 client.46598.objecter put_session s=0x7ff74c004160 osd=1 4 2024-04-14T15:14:42.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.315+0000 7ff75cff9700 15 client.46598.objecter _session_command_op_remove 1 3 2024-04-14T15:14:42.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.334+0000 7ff76619b700 10 client.46598.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:14:42.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.334+0000 7ff76619b700 15 client.46598.objecter _session_command_op_assign -1 4 2024-04-14T15:14:42.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.334+0000 7ff76619b700 20 client.46598.objecter _get_session s=0x7ff74c004160 osd=1 3 2024-04-14T15:14:42.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.334+0000 7ff76619b700 20 client.46598.objecter put_session s=0x7ff74c004160 osd=1 3 2024-04-14T15:14:42.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.334+0000 7ff76619b700 20 client.46598.objecter _get_session s=0x7ff74c004160 osd=1 3 2024-04-14T15:14:42.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.334+0000 7ff76619b700 15 client.46598.objecter _session_command_op_remove -1 4 2024-04-14T15:14:42.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.334+0000 7ff76619b700 20 client.46598.objecter get_session s=0x7ff74c004160 osd=1 3 2024-04-14T15:14:42.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.334+0000 7ff76619b700 15 client.46598.objecter _session_command_op_assign 1 4 2024-04-14T15:14:42.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.334+0000 7ff76619b700 20 client.46598.objecter put_session s=0x7ff74c004160 osd=1 4 2024-04-14T15:14:42.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.334+0000 7ff76619b700 10 client.46598.objecter _send_command 4 2024-04-14T15:14:42.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.334+0000 7ff76619b700 1 -- 172.21.15.90:0/2547188057 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff74c001750 con 0x7ff74c004340 2024-04-14T15:14:42.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.336+0000 7ff75cff9700 1 -- 172.21.15.90:0/2547188057 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff74c001750 con 0x7ff74c004340 2024-04-14T15:14:42.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.336+0000 7ff75cff9700 10 client.46598.objecter ms_dispatch 0x7ff760001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:14:42.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.336+0000 7ff75cff9700 10 client.46598.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:14:42.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.336+0000 7ff75cff9700 20 client.46598.objecter put_session s=0x7ff74c004160 osd=1 4 2024-04-14T15:14:42.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.336+0000 7ff75cff9700 15 client.46598.objecter _session_command_op_remove 1 4 2024-04-14T15:14:42.338 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:14:42.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.337+0000 7ff76619b700 10 client.46598.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:14:42.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.337+0000 7ff76619b700 15 client.46598.objecter _session_command_op_assign -1 5 2024-04-14T15:14:42.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.337+0000 7ff76619b700 20 client.46598.objecter _get_session s=0x7ff74c0019b0 osd=2 3 2024-04-14T15:14:42.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.337+0000 7ff76619b700 20 client.46598.objecter put_session s=0x7ff74c0019b0 osd=2 3 2024-04-14T15:14:42.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.337+0000 7ff76619b700 20 client.46598.objecter _get_session s=0x7ff74c0019b0 osd=2 3 2024-04-14T15:14:42.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.337+0000 7ff76619b700 15 client.46598.objecter _session_command_op_remove -1 5 2024-04-14T15:14:42.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.337+0000 7ff76619b700 20 client.46598.objecter get_session s=0x7ff74c0019b0 osd=2 3 2024-04-14T15:14:42.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.337+0000 7ff76619b700 15 client.46598.objecter _session_command_op_assign 2 5 2024-04-14T15:14:42.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.337+0000 7ff76619b700 20 client.46598.objecter put_session s=0x7ff74c0019b0 osd=2 4 2024-04-14T15:14:42.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.337+0000 7ff76619b700 10 client.46598.objecter _send_command 5 2024-04-14T15:14:42.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.337+0000 7ff76619b700 1 -- 172.21.15.90:0/2547188057 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7ff74c0111a0 con 0x7ff74c00d600 2024-04-14T15:14:42.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.337+0000 7ff75cff9700 10 client.46598.objecter ms_handle_connect 0x7ff74c00d600 2024-04-14T15:14:42.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.338+0000 7ff75cff9700 1 -- 172.21.15.90:0/2547188057 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7ff74c0111a0 con 0x7ff74c00d600 2024-04-14T15:14:42.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.338+0000 7ff75cff9700 10 client.46598.objecter ms_dispatch 0x7ff760001550 command_reply(tid 5: 0 ) 2024-04-14T15:14:42.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.338+0000 7ff75cff9700 10 client.46598.objecter _finish_command 5 = system:0 2024-04-14T15:14:42.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.338+0000 7ff75cff9700 20 client.46598.objecter put_session s=0x7ff74c0019b0 osd=2 4 2024-04-14T15:14:42.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.338+0000 7ff75cff9700 15 client.46598.objecter _session_command_op_remove 2 5 2024-04-14T15:14:42.343 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:14:42.353 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:14:42.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.356+0000 7ff76619b700 10 client.46598.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:14:42.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.356+0000 7ff76619b700 15 client.46598.objecter _session_command_op_assign -1 6 2024-04-14T15:14:42.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.356+0000 7ff76619b700 20 client.46598.objecter _get_session s=0x7ff74c0019b0 osd=2 3 2024-04-14T15:14:42.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.356+0000 7ff76619b700 20 client.46598.objecter put_session s=0x7ff74c0019b0 osd=2 3 2024-04-14T15:14:42.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.356+0000 7ff76619b700 20 client.46598.objecter _get_session s=0x7ff74c0019b0 osd=2 3 2024-04-14T15:14:42.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.356+0000 7ff76619b700 15 client.46598.objecter _session_command_op_remove -1 6 2024-04-14T15:14:42.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.356+0000 7ff76619b700 20 client.46598.objecter get_session s=0x7ff74c0019b0 osd=2 3 2024-04-14T15:14:42.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.356+0000 7ff76619b700 15 client.46598.objecter _session_command_op_assign 2 6 2024-04-14T15:14:42.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.356+0000 7ff76619b700 20 client.46598.objecter put_session s=0x7ff74c0019b0 osd=2 4 2024-04-14T15:14:42.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.356+0000 7ff76619b700 10 client.46598.objecter _send_command 6 2024-04-14T15:14:42.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.356+0000 7ff76619b700 1 -- 172.21.15.90:0/2547188057 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff74c008130 con 0x7ff74c00d600 2024-04-14T15:14:42.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.358+0000 7ff75cff9700 1 -- 172.21.15.90:0/2547188057 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff74c008130 con 0x7ff74c00d600 2024-04-14T15:14:42.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.358+0000 7ff75cff9700 10 client.46598.objecter ms_dispatch 0x7ff760001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:14:42.360 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:14:42.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.358+0000 7ff75cff9700 10 client.46598.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:14:42.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.358+0000 7ff75cff9700 20 client.46598.objecter put_session s=0x7ff74c0019b0 osd=2 4 2024-04-14T15:14:42.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.358+0000 7ff75cff9700 15 client.46598.objecter _session_command_op_remove 2 6 2024-04-14T15:14:42.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.358+0000 7ff76619b700 10 client.46598.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:14:42.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.358+0000 7ff76619b700 15 client.46598.objecter _session_command_op_assign -1 7 2024-04-14T15:14:42.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.359+0000 7ff76619b700 20 client.46598.objecter _get_session s=0x7ff74c00cc50 osd=3 3 2024-04-14T15:14:42.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.359+0000 7ff76619b700 20 client.46598.objecter put_session s=0x7ff74c00cc50 osd=3 3 2024-04-14T15:14:42.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.359+0000 7ff76619b700 20 client.46598.objecter _get_session s=0x7ff74c00cc50 osd=3 3 2024-04-14T15:14:42.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.359+0000 7ff76619b700 15 client.46598.objecter _session_command_op_remove -1 7 2024-04-14T15:14:42.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.359+0000 7ff76619b700 20 client.46598.objecter get_session s=0x7ff74c00cc50 osd=3 3 2024-04-14T15:14:42.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.359+0000 7ff76619b700 15 client.46598.objecter _session_command_op_assign 3 7 2024-04-14T15:14:42.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.359+0000 7ff76619b700 20 client.46598.objecter put_session s=0x7ff74c00cc50 osd=3 4 2024-04-14T15:14:42.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.359+0000 7ff76619b700 10 client.46598.objecter _send_command 7 2024-04-14T15:14:42.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.359+0000 7ff76619b700 1 -- 172.21.15.90:0/2547188057 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7ff74c015fe0 con 0x7ff74c0124d0 2024-04-14T15:14:42.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.359+0000 7ff75cff9700 10 client.46598.objecter ms_handle_connect 0x7ff74c0124d0 2024-04-14T15:14:42.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.360+0000 7ff75cff9700 1 -- 172.21.15.90:0/2547188057 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7ff74c015fe0 con 0x7ff74c0124d0 2024-04-14T15:14:42.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.360+0000 7ff75cff9700 10 client.46598.objecter ms_dispatch 0x7ff760001550 command_reply(tid 7: 0 ) 2024-04-14T15:14:42.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.360+0000 7ff75cff9700 10 client.46598.objecter _finish_command 7 = system:0 2024-04-14T15:14:42.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.360+0000 7ff75cff9700 20 client.46598.objecter put_session s=0x7ff74c00cc50 osd=3 4 2024-04-14T15:14:42.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.360+0000 7ff75cff9700 15 client.46598.objecter _session_command_op_remove 3 7 2024-04-14T15:14:42.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:41 smithi092 ceph-mon[33966]: 13.a scrub starts 2024-04-14T15:14:42.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:41 smithi092 ceph-mon[33966]: 13.a scrub ok 2024-04-14T15:14:42.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:41 smithi092 ceph-mon[33966]: 1.2 scrub starts 2024-04-14T15:14:42.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:41 smithi092 ceph-mon[33966]: 1.2 scrub ok 2024-04-14T15:14:42.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:41 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-04-14T15:14:42.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:41 smithi092 ceph-mon[33966]: 1.1 scrub starts 2024-04-14T15:14:42.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:41 smithi092 ceph-mon[33966]: 1.1 scrub ok 2024-04-14T15:14:42.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:41 smithi092 ceph-mon[33966]: 3.1 scrub starts 2024-04-14T15:14:42.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:41 smithi092 ceph-mon[33966]: 3.1 scrub ok 2024-04-14T15:14:42.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:41 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-19", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:14:42.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:41 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "18"}]': finished 2024-04-14T15:14:42.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:41 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-19"}]: dispatch 2024-04-14T15:14:42.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:41 smithi092 ceph-mon[33966]: osdmap e214: 8 total, 8 up, 8 in 2024-04-14T15:14:42.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:41 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-19"}]: dispatch 2024-04-14T15:14:42.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:41 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:42.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.379+0000 7ff76619b700 10 client.46598.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:14:42.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.379+0000 7ff76619b700 15 client.46598.objecter _session_command_op_assign -1 8 2024-04-14T15:14:42.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.379+0000 7ff76619b700 20 client.46598.objecter _get_session s=0x7ff74c00cc50 osd=3 3 2024-04-14T15:14:42.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.379+0000 7ff76619b700 20 client.46598.objecter put_session s=0x7ff74c00cc50 osd=3 3 2024-04-14T15:14:42.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.379+0000 7ff76619b700 20 client.46598.objecter _get_session s=0x7ff74c00cc50 osd=3 3 2024-04-14T15:14:42.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.379+0000 7ff76619b700 15 client.46598.objecter _session_command_op_remove -1 8 2024-04-14T15:14:42.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.379+0000 7ff76619b700 20 client.46598.objecter get_session s=0x7ff74c00cc50 osd=3 3 2024-04-14T15:14:42.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.379+0000 7ff76619b700 15 client.46598.objecter _session_command_op_assign 3 8 2024-04-14T15:14:42.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.379+0000 7ff76619b700 20 client.46598.objecter put_session s=0x7ff74c00cc50 osd=3 4 2024-04-14T15:14:42.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.379+0000 7ff76619b700 10 client.46598.objecter _send_command 8 2024-04-14T15:14:42.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.379+0000 7ff76619b700 1 -- 172.21.15.90:0/2547188057 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff74c00c790 con 0x7ff74c0124d0 2024-04-14T15:14:42.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.381+0000 7ff75cff9700 1 -- 172.21.15.90:0/2547188057 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff74c00c790 con 0x7ff74c0124d0 2024-04-14T15:14:42.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.382+0000 7ff75cff9700 10 client.46598.objecter ms_dispatch 0x7ff760001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:14:42.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.382+0000 7ff75cff9700 10 client.46598.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:14:42.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.382+0000 7ff75cff9700 20 client.46598.objecter put_session s=0x7ff74c00cc50 osd=3 4 2024-04-14T15:14:42.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.382+0000 7ff75cff9700 15 client.46598.objecter _session_command_op_remove 3 8 2024-04-14T15:14:42.383 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:14:42.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.382+0000 7ff76619b700 10 client.46598.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:14:42.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.382+0000 7ff76619b700 15 client.46598.objecter _session_command_op_assign -1 9 2024-04-14T15:14:42.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.382+0000 7ff76619b700 20 client.46598.objecter _get_session s=0x7ff74c00ca00 osd=4 3 2024-04-14T15:14:42.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.382+0000 7ff76619b700 20 client.46598.objecter put_session s=0x7ff74c00ca00 osd=4 3 2024-04-14T15:14:42.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.382+0000 7ff76619b700 20 client.46598.objecter _get_session s=0x7ff74c00ca00 osd=4 3 2024-04-14T15:14:42.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.382+0000 7ff76619b700 15 client.46598.objecter _session_command_op_remove -1 9 2024-04-14T15:14:42.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.382+0000 7ff76619b700 20 client.46598.objecter get_session s=0x7ff74c00ca00 osd=4 3 2024-04-14T15:14:42.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.382+0000 7ff76619b700 15 client.46598.objecter _session_command_op_assign 4 9 2024-04-14T15:14:42.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.382+0000 7ff76619b700 20 client.46598.objecter put_session s=0x7ff74c00ca00 osd=4 4 2024-04-14T15:14:42.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.382+0000 7ff76619b700 10 client.46598.objecter _send_command 9 2024-04-14T15:14:42.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.382+0000 7ff76619b700 1 -- 172.21.15.90:0/2547188057 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7ff74c01aae0 con 0x7ff74c012030 2024-04-14T15:14:42.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.383+0000 7ff75cff9700 10 client.46598.objecter ms_handle_connect 0x7ff74c012030 2024-04-14T15:14:42.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.383+0000 7ff75cff9700 1 -- 172.21.15.90:0/2547188057 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7ff74c01aae0 con 0x7ff74c012030 2024-04-14T15:14:42.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.383+0000 7ff75cff9700 10 client.46598.objecter ms_dispatch 0x7ff760001550 command_reply(tid 9: 0 ) 2024-04-14T15:14:42.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.383+0000 7ff75cff9700 10 client.46598.objecter _finish_command 9 = system:0 2024-04-14T15:14:42.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.383+0000 7ff75cff9700 20 client.46598.objecter put_session s=0x7ff74c00ca00 osd=4 4 2024-04-14T15:14:42.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.383+0000 7ff75cff9700 15 client.46598.objecter _session_command_op_remove 4 9 2024-04-14T15:14:42.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.400+0000 7ff76619b700 10 client.46598.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:14:42.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.400+0000 7ff76619b700 15 client.46598.objecter _session_command_op_assign -1 10 2024-04-14T15:14:42.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.400+0000 7ff76619b700 20 client.46598.objecter _get_session s=0x7ff74c00ca00 osd=4 3 2024-04-14T15:14:42.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.400+0000 7ff76619b700 20 client.46598.objecter put_session s=0x7ff74c00ca00 osd=4 3 2024-04-14T15:14:42.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.400+0000 7ff76619b700 20 client.46598.objecter _get_session s=0x7ff74c00ca00 osd=4 3 2024-04-14T15:14:42.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.400+0000 7ff76619b700 15 client.46598.objecter _session_command_op_remove -1 10 2024-04-14T15:14:42.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.400+0000 7ff76619b700 20 client.46598.objecter get_session s=0x7ff74c00ca00 osd=4 3 2024-04-14T15:14:42.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.400+0000 7ff76619b700 15 client.46598.objecter _session_command_op_assign 4 10 2024-04-14T15:14:42.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.400+0000 7ff76619b700 20 client.46598.objecter put_session s=0x7ff74c00ca00 osd=4 4 2024-04-14T15:14:42.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.400+0000 7ff76619b700 10 client.46598.objecter _send_command 10 2024-04-14T15:14:42.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.400+0000 7ff76619b700 1 -- 172.21.15.90:0/2547188057 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff74c01c490 con 0x7ff74c012030 2024-04-14T15:14:42.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.402+0000 7ff75cff9700 1 -- 172.21.15.90:0/2547188057 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff74c01c490 con 0x7ff74c012030 2024-04-14T15:14:42.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.402+0000 7ff75cff9700 10 client.46598.objecter ms_dispatch 0x7ff760001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:14:42.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.402+0000 7ff75cff9700 10 client.46598.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:14:42.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.402+0000 7ff75cff9700 20 client.46598.objecter put_session s=0x7ff74c00ca00 osd=4 4 2024-04-14T15:14:42.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.402+0000 7ff75cff9700 15 client.46598.objecter _session_command_op_remove 4 10 2024-04-14T15:14:42.404 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:14:42.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.402+0000 7ff76619b700 10 client.46598.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:14:42.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.402+0000 7ff76619b700 15 client.46598.objecter _session_command_op_assign -1 11 2024-04-14T15:14:42.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.402+0000 7ff76619b700 20 client.46598.objecter _get_session s=0x7ff74c01c490 osd=5 3 2024-04-14T15:14:42.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.402+0000 7ff76619b700 20 client.46598.objecter put_session s=0x7ff74c01c490 osd=5 3 2024-04-14T15:14:42.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.402+0000 7ff76619b700 20 client.46598.objecter _get_session s=0x7ff74c01c490 osd=5 3 2024-04-14T15:14:42.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.402+0000 7ff76619b700 15 client.46598.objecter _session_command_op_remove -1 11 2024-04-14T15:14:42.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.402+0000 7ff76619b700 20 client.46598.objecter get_session s=0x7ff74c01c490 osd=5 3 2024-04-14T15:14:42.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.402+0000 7ff76619b700 15 client.46598.objecter _session_command_op_assign 5 11 2024-04-14T15:14:42.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.402+0000 7ff76619b700 20 client.46598.objecter put_session s=0x7ff74c01c490 osd=5 4 2024-04-14T15:14:42.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.402+0000 7ff76619b700 10 client.46598.objecter _send_command 11 2024-04-14T15:14:42.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.402+0000 7ff76619b700 1 -- 172.21.15.90:0/2547188057 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7ff74c01d040 con 0x7ff74c01cbb0 2024-04-14T15:14:42.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.402+0000 7ff75cff9700 10 client.46598.objecter ms_handle_connect 0x7ff74c01cbb0 2024-04-14T15:14:42.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.404+0000 7ff75cff9700 1 -- 172.21.15.90:0/2547188057 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7ff74c01d040 con 0x7ff74c01cbb0 2024-04-14T15:14:42.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.404+0000 7ff75cff9700 10 client.46598.objecter ms_dispatch 0x7ff760001550 command_reply(tid 11: 0 ) 2024-04-14T15:14:42.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.404+0000 7ff75cff9700 10 client.46598.objecter _finish_command 11 = system:0 2024-04-14T15:14:42.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.404+0000 7ff75cff9700 20 client.46598.objecter put_session s=0x7ff74c01c490 osd=5 4 2024-04-14T15:14:42.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.404+0000 7ff75cff9700 15 client.46598.objecter _session_command_op_remove 5 11 2024-04-14T15:14:42.415 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:14:42.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.420+0000 7ff76619b700 10 client.46598.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:14:42.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.420+0000 7ff76619b700 15 client.46598.objecter _session_command_op_assign -1 12 2024-04-14T15:14:42.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.420+0000 7ff76619b700 20 client.46598.objecter _get_session s=0x7ff74c01c490 osd=5 3 2024-04-14T15:14:42.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.420+0000 7ff76619b700 20 client.46598.objecter put_session s=0x7ff74c01c490 osd=5 3 2024-04-14T15:14:42.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.420+0000 7ff76619b700 20 client.46598.objecter _get_session s=0x7ff74c01c490 osd=5 3 2024-04-14T15:14:42.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.420+0000 7ff76619b700 15 client.46598.objecter _session_command_op_remove -1 12 2024-04-14T15:14:42.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.420+0000 7ff76619b700 20 client.46598.objecter get_session s=0x7ff74c01c490 osd=5 3 2024-04-14T15:14:42.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.420+0000 7ff76619b700 15 client.46598.objecter _session_command_op_assign 5 12 2024-04-14T15:14:42.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.420+0000 7ff76619b700 20 client.46598.objecter put_session s=0x7ff74c01c490 osd=5 4 2024-04-14T15:14:42.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.420+0000 7ff76619b700 10 client.46598.objecter _send_command 12 2024-04-14T15:14:42.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.420+0000 7ff76619b700 1 -- 172.21.15.90:0/2547188057 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff74c01d040 con 0x7ff74c01cbb0 2024-04-14T15:14:42.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.422+0000 7ff75cff9700 1 -- 172.21.15.90:0/2547188057 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff74c01d040 con 0x7ff74c01cbb0 2024-04-14T15:14:42.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.422+0000 7ff75cff9700 10 client.46598.objecter ms_dispatch 0x7ff760001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:14:42.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.422+0000 7ff75cff9700 10 client.46598.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:14:42.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.422+0000 7ff75cff9700 20 client.46598.objecter put_session s=0x7ff74c01c490 osd=5 4 2024-04-14T15:14:42.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.422+0000 7ff75cff9700 15 client.46598.objecter _session_command_op_remove 5 12 2024-04-14T15:14:42.424 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:14:42.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.422+0000 7ff76619b700 10 client.46598.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:14:42.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.422+0000 7ff76619b700 15 client.46598.objecter _session_command_op_assign -1 13 2024-04-14T15:14:42.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.422+0000 7ff76619b700 20 client.46598.objecter _get_session s=0x7ff74c01d040 osd=6 3 2024-04-14T15:14:42.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.422+0000 7ff76619b700 20 client.46598.objecter put_session s=0x7ff74c01d040 osd=6 3 2024-04-14T15:14:42.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.422+0000 7ff76619b700 20 client.46598.objecter _get_session s=0x7ff74c01d040 osd=6 3 2024-04-14T15:14:42.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.422+0000 7ff76619b700 15 client.46598.objecter _session_command_op_remove -1 13 2024-04-14T15:14:42.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.422+0000 7ff76619b700 20 client.46598.objecter get_session s=0x7ff74c01d040 osd=6 3 2024-04-14T15:14:42.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.422+0000 7ff76619b700 15 client.46598.objecter _session_command_op_assign 6 13 2024-04-14T15:14:42.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.422+0000 7ff76619b700 20 client.46598.objecter put_session s=0x7ff74c01d040 osd=6 4 2024-04-14T15:14:42.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.422+0000 7ff76619b700 10 client.46598.objecter _send_command 13 2024-04-14T15:14:42.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.422+0000 7ff76619b700 1 -- 172.21.15.90:0/2547188057 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7ff74c01d460 con 0x7ff74c01fba0 2024-04-14T15:14:42.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.423+0000 7ff75cff9700 10 client.46598.objecter ms_handle_connect 0x7ff74c01fba0 2024-04-14T15:14:42.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.424+0000 7ff75cff9700 1 -- 172.21.15.90:0/2547188057 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7ff74c01d460 con 0x7ff74c01fba0 2024-04-14T15:14:42.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.424+0000 7ff75cff9700 10 client.46598.objecter ms_dispatch 0x7ff760001550 command_reply(tid 13: 0 ) 2024-04-14T15:14:42.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.424+0000 7ff75cff9700 10 client.46598.objecter _finish_command 13 = system:0 2024-04-14T15:14:42.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.424+0000 7ff75cff9700 20 client.46598.objecter put_session s=0x7ff74c01d040 osd=6 4 2024-04-14T15:14:42.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.424+0000 7ff75cff9700 15 client.46598.objecter _session_command_op_remove 6 13 2024-04-14T15:14:42.438 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:42.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:42.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:14:42.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.442+0000 7ff76619b700 10 client.46598.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:14:42.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.442+0000 7ff76619b700 15 client.46598.objecter _session_command_op_assign -1 14 2024-04-14T15:14:42.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.442+0000 7ff76619b700 20 client.46598.objecter _get_session s=0x7ff74c01d040 osd=6 3 2024-04-14T15:14:42.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.442+0000 7ff76619b700 20 client.46598.objecter put_session s=0x7ff74c01d040 osd=6 3 2024-04-14T15:14:42.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.442+0000 7ff76619b700 20 client.46598.objecter _get_session s=0x7ff74c01d040 osd=6 3 2024-04-14T15:14:42.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.442+0000 7ff76619b700 15 client.46598.objecter _session_command_op_remove -1 14 2024-04-14T15:14:42.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.442+0000 7ff76619b700 20 client.46598.objecter get_session s=0x7ff74c01d040 osd=6 3 2024-04-14T15:14:42.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.442+0000 7ff76619b700 15 client.46598.objecter _session_command_op_assign 6 14 2024-04-14T15:14:42.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.442+0000 7ff76619b700 20 client.46598.objecter put_session s=0x7ff74c01d040 osd=6 4 2024-04-14T15:14:42.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.442+0000 7ff76619b700 10 client.46598.objecter _send_command 14 2024-04-14T15:14:42.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.442+0000 7ff76619b700 1 -- 172.21.15.90:0/2547188057 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff74c0236e0 con 0x7ff74c01fba0 2024-04-14T15:14:42.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.443+0000 7ff75cff9700 1 -- 172.21.15.90:0/2547188057 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff74c0236e0 con 0x7ff74c01fba0 2024-04-14T15:14:42.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.443+0000 7ff75cff9700 10 client.46598.objecter ms_dispatch 0x7ff760001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:14:42.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.443+0000 7ff75cff9700 10 client.46598.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:14:42.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.443+0000 7ff75cff9700 20 client.46598.objecter put_session s=0x7ff74c01d040 osd=6 4 2024-04-14T15:14:42.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.443+0000 7ff75cff9700 15 client.46598.objecter _session_command_op_remove 6 14 2024-04-14T15:14:42.445 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:14:42.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.444+0000 7ff76619b700 10 client.46598.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:14:42.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.444+0000 7ff76619b700 15 client.46598.objecter _session_command_op_assign -1 15 2024-04-14T15:14:42.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.444+0000 7ff76619b700 20 client.46598.objecter _get_session s=0x7ff74c0236e0 osd=7 3 2024-04-14T15:14:42.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.444+0000 7ff76619b700 20 client.46598.objecter put_session s=0x7ff74c0236e0 osd=7 3 2024-04-14T15:14:42.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.444+0000 7ff76619b700 20 client.46598.objecter _get_session s=0x7ff74c0236e0 osd=7 3 2024-04-14T15:14:42.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.444+0000 7ff76619b700 15 client.46598.objecter _session_command_op_remove -1 15 2024-04-14T15:14:42.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.444+0000 7ff76619b700 20 client.46598.objecter get_session s=0x7ff74c0236e0 osd=7 3 2024-04-14T15:14:42.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.444+0000 7ff76619b700 15 client.46598.objecter _session_command_op_assign 7 15 2024-04-14T15:14:42.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.444+0000 7ff76619b700 20 client.46598.objecter put_session s=0x7ff74c0236e0 osd=7 4 2024-04-14T15:14:42.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.444+0000 7ff76619b700 10 client.46598.objecter _send_command 15 2024-04-14T15:14:42.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.444+0000 7ff76619b700 1 -- 172.21.15.90:0/2547188057 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7ff74c0249c0 con 0x7ff74c023e00 2024-04-14T15:14:42.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.444+0000 7ff75cff9700 10 client.46598.objecter ms_handle_connect 0x7ff74c023e00 2024-04-14T15:14:42.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.445+0000 7ff75cff9700 1 -- 172.21.15.90:0/2547188057 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7ff74c0249c0 con 0x7ff74c023e00 2024-04-14T15:14:42.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.445+0000 7ff75cff9700 10 client.46598.objecter ms_dispatch 0x7ff760001550 command_reply(tid 15: 0 ) 2024-04-14T15:14:42.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.445+0000 7ff75cff9700 10 client.46598.objecter _finish_command 15 = system:0 2024-04-14T15:14:42.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.445+0000 7ff75cff9700 20 client.46598.objecter put_session s=0x7ff74c0236e0 osd=7 4 2024-04-14T15:14:42.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.445+0000 7ff75cff9700 15 client.46598.objecter _session_command_op_remove 7 15 2024-04-14T15:14:42.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.461+0000 7ff76619b700 10 client.46598.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:14:42.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.461+0000 7ff76619b700 15 client.46598.objecter _session_command_op_assign -1 16 2024-04-14T15:14:42.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.461+0000 7ff76619b700 20 client.46598.objecter _get_session s=0x7ff74c0236e0 osd=7 3 2024-04-14T15:14:42.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.461+0000 7ff76619b700 20 client.46598.objecter put_session s=0x7ff74c0236e0 osd=7 3 2024-04-14T15:14:42.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.461+0000 7ff76619b700 20 client.46598.objecter _get_session s=0x7ff74c0236e0 osd=7 3 2024-04-14T15:14:42.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.461+0000 7ff76619b700 15 client.46598.objecter _session_command_op_remove -1 16 2024-04-14T15:14:42.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.461+0000 7ff76619b700 20 client.46598.objecter get_session s=0x7ff74c0236e0 osd=7 3 2024-04-14T15:14:42.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.461+0000 7ff76619b700 15 client.46598.objecter _session_command_op_assign 7 16 2024-04-14T15:14:42.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.461+0000 7ff76619b700 20 client.46598.objecter put_session s=0x7ff74c0236e0 osd=7 4 2024-04-14T15:14:42.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.461+0000 7ff76619b700 10 client.46598.objecter _send_command 16 2024-04-14T15:14:42.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.461+0000 7ff76619b700 1 -- 172.21.15.90:0/2547188057 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff74c0249c0 con 0x7ff74c023e00 2024-04-14T15:14:42.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.463+0000 7ff75cff9700 1 -- 172.21.15.90:0/2547188057 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff74c0249c0 con 0x7ff74c023e00 2024-04-14T15:14:42.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.463+0000 7ff75cff9700 10 client.46598.objecter ms_dispatch 0x7ff760001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:14:42.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.463+0000 7ff75cff9700 10 client.46598.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:14:42.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.463+0000 7ff75cff9700 20 client.46598.objecter put_session s=0x7ff74c0236e0 osd=7 4 2024-04-14T15:14:42.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.463+0000 7ff75cff9700 15 client.46598.objecter _session_command_op_remove 7 16 2024-04-14T15:14:42.465 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:14:42.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.464+0000 7ff76619b700 10 librados: watch_flush enter 2024-04-14T15:14:42.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.464+0000 7ff76619b700 10 librados: watch_flush exit 2024-04-14T15:14:42.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.464+0000 7ff76619b700 10 client.46598.objecter close_session for osd.0 2024-04-14T15:14:42.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.464+0000 7ff76619b700 1 -- 172.21.15.90:0/2547188057 >> v1:172.21.15.90:6801/3761806885 conn(0x7ff74c004760 legacy=0x7ff74c002780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:42.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.464+0000 7ff76619b700 20 client.46598.objecter put_session s=0x7ff74c005410 osd=0 1 2024-04-14T15:14:42.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.464+0000 7ff76619b700 10 client.46598.objecter close_session for osd.1 2024-04-14T15:14:42.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.464+0000 7ff76619b700 1 -- 172.21.15.90:0/2547188057 >> v1:172.21.15.90:6805/4170361699 conn(0x7ff74c004340 legacy=0x7ff74c00ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:42.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.464+0000 7ff76619b700 20 client.46598.objecter put_session s=0x7ff74c004160 osd=1 1 2024-04-14T15:14:42.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.464+0000 7ff76619b700 10 client.46598.objecter close_session for osd.2 2024-04-14T15:14:42.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.464+0000 7ff76619b700 1 -- 172.21.15.90:0/2547188057 >> v1:172.21.15.90:6809/1686184251 conn(0x7ff74c00d600 legacy=0x7ff74c00f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:42.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.464+0000 7ff76619b700 20 client.46598.objecter put_session s=0x7ff74c0019b0 osd=2 1 2024-04-14T15:14:42.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.464+0000 7ff76619b700 10 client.46598.objecter close_session for osd.3 2024-04-14T15:14:42.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.464+0000 7ff76619b700 1 -- 172.21.15.90:0/2547188057 >> v1:172.21.15.90:6813/2270166774 conn(0x7ff74c0124d0 legacy=0x7ff74c0148c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:42.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.464+0000 7ff75ffff700 1 -- 172.21.15.90:0/2547188057 reap_dead start 2024-04-14T15:14:42.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.464+0000 7ff76619b700 20 client.46598.objecter put_session s=0x7ff74c00cc50 osd=3 1 2024-04-14T15:14:42.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.464+0000 7ff76619b700 10 client.46598.objecter close_session for osd.4 2024-04-14T15:14:42.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.464+0000 7ff76619b700 1 -- 172.21.15.90:0/2547188057 >> v1:172.21.15.92:6800/3887549733 conn(0x7ff74c012030 legacy=0x7ff74c019320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:42.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.464+0000 7ff75ffff700 1 -- 172.21.15.90:0/2547188057 reap_dead start 2024-04-14T15:14:42.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.464+0000 7ff76619b700 20 client.46598.objecter put_session s=0x7ff74c00ca00 osd=4 1 2024-04-14T15:14:42.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.464+0000 7ff76619b700 10 client.46598.objecter close_session for osd.5 2024-04-14T15:14:42.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.464+0000 7ff76619b700 1 -- 172.21.15.90:0/2547188057 >> v1:172.21.15.92:6804/3244561440 conn(0x7ff74c01cbb0 legacy=0x7ff74c01acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:42.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.464+0000 7ff76619b700 20 client.46598.objecter put_session s=0x7ff74c01c490 osd=5 1 2024-04-14T15:14:42.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.464+0000 7ff76619b700 10 client.46598.objecter close_session for osd.6 2024-04-14T15:14:42.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.464+0000 7ff76619b700 1 -- 172.21.15.90:0/2547188057 >> v1:172.21.15.92:6808/636125963 conn(0x7ff74c01fba0 legacy=0x7ff74c021f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:42.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.464+0000 7ff76619b700 20 client.46598.objecter put_session s=0x7ff74c01d040 osd=6 1 2024-04-14T15:14:42.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.464+0000 7ff76619b700 10 client.46598.objecter close_session for osd.7 2024-04-14T15:14:42.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.464+0000 7ff76619b700 1 -- 172.21.15.90:0/2547188057 >> v1:172.21.15.92:6812/1773706716 conn(0x7ff74c023e00 legacy=0x7ff74c024280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:42.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.464+0000 7ff76619b700 20 client.46598.objecter put_session s=0x7ff74c0236e0 osd=7 1 2024-04-14T15:14:42.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.464+0000 7ff76619b700 20 client.46598.objecter shutdown clearing up homeless session... 2024-04-14T15:14:42.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.464+0000 7ff76619b700 10 client.46598.objecter successfully canceled tick 2024-04-14T15:14:42.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.464+0000 7ff76619b700 1 -- 172.21.15.90:0/2547188057 >> v1:172.21.15.90:6800/420935427 conn(0x7ff748071310 legacy=0x7ff7480737d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:42.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.464+0000 7ff76619b700 1 -- 172.21.15.90:0/2547188057 >> v1:172.21.15.90:6790/0 conn(0x7ff76010a610 legacy=0x7ff7601a4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:42.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.464+0000 7ff75ffff700 1 -- 172.21.15.90:0/2547188057 reap_dead start 2024-04-14T15:14:42.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.464+0000 7ff76619b700 1 -- 172.21.15.90:0/2547188057 shutdown_connections 2024-04-14T15:14:42.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.464+0000 7ff76619b700 1 -- 172.21.15.90:0/2547188057 >> 172.21.15.90:0/2547188057 conn(0x7ff7600fd290 msgr2=0x7ff760111490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:14:42.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.464+0000 7ff76619b700 1 -- 172.21.15.90:0/2547188057 shutdown_connections 2024-04-14T15:14:42.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.464+0000 7ff76619b700 1 -- 172.21.15.90:0/2547188057 wait complete. 2024-04-14T15:14:42.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:42.464+0000 7ff76619b700 1 librados: shutdown 2024-04-14T15:14:42.469 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:14:42.469 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:14:42.469 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:14:42.469 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:14:42.469 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:14:42.469 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:14:42.469 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:14:42.470 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:14:42.539 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:42.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:42.565 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:42.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:42.666 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:14:42.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:42.690 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:42.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:14:42.792 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:14:42.816 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:42.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:42.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:14:42.918 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:14:42.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:42.943 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:42.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:14:43.044 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:14:43.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:43.069 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:43.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:14:43.170 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:43.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:42 smithi090 ceph-mon[41670]: pgmap v266: 289 pgs: 1 clean+premerge+peered, 6 active+clean+snaptrim, 64 unknown, 7 active+clean+snaptrim_wait, 211 active+clean; 481 KiB data, 546 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:14:43.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:42 smithi090 ceph-mon[41670]: 4.e scrub starts 2024-04-14T15:14:43.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:42 smithi090 ceph-mon[41670]: 4.e scrub ok 2024-04-14T15:14:43.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:42 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:14:43.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:42 smithi090 ceph-mon[41670]: 1.3 scrub starts 2024-04-14T15:14:43.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:42 smithi090 ceph-mon[41670]: 1.3 scrub ok 2024-04-14T15:14:43.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:42 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2547188057' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:14:43.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:42 smithi090 ceph-mon[41670]: from='client.48481 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi090-93342-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi090-93342-31"}]': finished 2024-04-14T15:14:43.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:42 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-19"}]': finished 2024-04-14T15:14:43.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:42 smithi090 ceph-mon[41670]: osdmap e215: 8 total, 8 up, 8 in 2024-04-14T15:14:43.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:42 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-19", "mode": "writeback"}]: dispatch 2024-04-14T15:14:43.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:42 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-19", "mode": "writeback"}]: dispatch 2024-04-14T15:14:43.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:42 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2979721795' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi090-93363-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:43.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:42 smithi090 ceph-mon[41670]: from='client.46592 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi090-93363-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:43.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:42 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:43.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:42 smithi090 ceph-mon[30399]: pgmap v266: 289 pgs: 1 clean+premerge+peered, 6 active+clean+snaptrim, 64 unknown, 7 active+clean+snaptrim_wait, 211 active+clean; 481 KiB data, 546 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:14:43.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:42 smithi090 ceph-mon[30399]: 4.e scrub starts 2024-04-14T15:14:43.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:42 smithi090 ceph-mon[30399]: 4.e scrub ok 2024-04-14T15:14:43.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:42 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:14:43.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:42 smithi090 ceph-mon[30399]: 1.3 scrub starts 2024-04-14T15:14:43.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:42 smithi090 ceph-mon[30399]: 1.3 scrub ok 2024-04-14T15:14:43.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:42 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2547188057' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:14:43.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:42 smithi090 ceph-mon[30399]: from='client.48481 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi090-93342-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi090-93342-31"}]': finished 2024-04-14T15:14:43.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:42 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-19"}]': finished 2024-04-14T15:14:43.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:42 smithi090 ceph-mon[30399]: osdmap e215: 8 total, 8 up, 8 in 2024-04-14T15:14:43.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:42 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-19", "mode": "writeback"}]: dispatch 2024-04-14T15:14:43.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:42 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-19", "mode": "writeback"}]: dispatch 2024-04-14T15:14:43.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:42 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2979721795' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi090-93363-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:43.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:42 smithi090 ceph-mon[30399]: from='client.46592 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi090-93363-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:43.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:42 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:43.194 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:43.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:43.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:43.296 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:14:43.320 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:43.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:43.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:14:43.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:42 smithi092 ceph-mon[33966]: pgmap v266: 289 pgs: 1 clean+premerge+peered, 6 active+clean+snaptrim, 64 unknown, 7 active+clean+snaptrim_wait, 211 active+clean; 481 KiB data, 546 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:14:43.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:42 smithi092 ceph-mon[33966]: 4.e scrub starts 2024-04-14T15:14:43.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:42 smithi092 ceph-mon[33966]: 4.e scrub ok 2024-04-14T15:14:43.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:42 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:14:43.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:42 smithi092 ceph-mon[33966]: 1.3 scrub starts 2024-04-14T15:14:43.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:42 smithi092 ceph-mon[33966]: 1.3 scrub ok 2024-04-14T15:14:43.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:42 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2547188057' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:14:43.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:42 smithi092 ceph-mon[33966]: from='client.48481 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi090-93342-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi090-93342-31"}]': finished 2024-04-14T15:14:43.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:42 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-19"}]': finished 2024-04-14T15:14:43.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:42 smithi092 ceph-mon[33966]: osdmap e215: 8 total, 8 up, 8 in 2024-04-14T15:14:43.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:42 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-19", "mode": "writeback"}]: dispatch 2024-04-14T15:14:43.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:42 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-19", "mode": "writeback"}]: dispatch 2024-04-14T15:14:43.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:42 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2979721795' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi090-93363-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:43.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:42 smithi092 ceph-mon[33966]: from='client.46592 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi090-93363-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:43.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:42 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:43.422 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:14:43.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:43.446 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:43.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:14:43.547 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:14:43.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:43.572 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:43.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:14:43.673 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:14:43.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:43.698 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:43.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:14:43.799 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:14:43.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:43.823 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:43.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:14:43.924 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:43.948 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:43.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:43.949 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:44.050 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:44.074 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:44.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:44.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:44.176 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:44.191 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:14:43 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:14:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:14:44.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:43 smithi090 ceph-mon[30399]: 11.15 scrub starts 2024-04-14T15:14:44.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:43 smithi090 ceph-mon[30399]: 11.15 scrub ok 2024-04-14T15:14:44.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:43 smithi090 ceph-mon[30399]: 5.f scrub starts 2024-04-14T15:14:44.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:43 smithi090 ceph-mon[30399]: 5.f scrub ok 2024-04-14T15:14:44.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:43 smithi090 ceph-mon[30399]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:14:44.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:43 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-19", "mode": "writeback"}]': finished 2024-04-14T15:14:44.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:43 smithi090 ceph-mon[30399]: from='client.46592 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi090-93363-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:44.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:43 smithi090 ceph-mon[30399]: osdmap e216: 8 total, 8 up, 8 in 2024-04-14T15:14:44.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:43 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:14:44.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:43 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:14:44.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:43 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:44.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:43 smithi090 ceph-mon[41670]: 11.15 scrub starts 2024-04-14T15:14:44.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:43 smithi090 ceph-mon[41670]: 11.15 scrub ok 2024-04-14T15:14:44.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:43 smithi090 ceph-mon[41670]: 5.f scrub starts 2024-04-14T15:14:44.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:43 smithi090 ceph-mon[41670]: 5.f scrub ok 2024-04-14T15:14:44.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:43 smithi090 ceph-mon[41670]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:14:44.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:43 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-19", "mode": "writeback"}]': finished 2024-04-14T15:14:44.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:43 smithi090 ceph-mon[41670]: from='client.46592 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi090-93363-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:44.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:43 smithi090 ceph-mon[41670]: osdmap e216: 8 total, 8 up, 8 in 2024-04-14T15:14:44.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:43 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:14:44.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:43 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:14:44.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:43 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:44.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:44.201 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:44.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:44.270 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-14T15:14:44.271 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-14T15:14:44.271 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-04-14T15:14:44.271 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-14T15:14:44.271 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-14T15:14:44.271 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-04-14T15:14:44.302 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:14:44.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:44.327 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:44.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:14:44.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:43 smithi092 ceph-mon[33966]: 11.15 scrub starts 2024-04-14T15:14:44.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:43 smithi092 ceph-mon[33966]: 11.15 scrub ok 2024-04-14T15:14:44.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:43 smithi092 ceph-mon[33966]: 5.f scrub starts 2024-04-14T15:14:44.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:43 smithi092 ceph-mon[33966]: 5.f scrub ok 2024-04-14T15:14:44.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:43 smithi092 ceph-mon[33966]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:14:44.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:43 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-19", "mode": "writeback"}]': finished 2024-04-14T15:14:44.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:43 smithi092 ceph-mon[33966]: from='client.46592 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi090-93363-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:44.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:43 smithi092 ceph-mon[33966]: osdmap e216: 8 total, 8 up, 8 in 2024-04-14T15:14:44.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:43 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:14:44.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:43 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:14:44.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:43 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:44.428 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:14:44.453 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:44.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:44.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:14:44.555 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:14:44.579 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:44.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:44.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:14:44.681 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:14:44.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:44.705 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:44.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:14:44.806 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:44.830 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:44.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:44.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:44.932 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:14:44.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:44.957 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:44.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:14:45.058 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:45.082 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:45.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:45.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:45.184 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:14:45.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:44 smithi090 ceph-mon[30399]: pgmap v269: 297 pgs: 1 clean+premerge+peered, 6 active+clean+snaptrim, 72 unknown, 7 active+clean+snaptrim_wait, 211 active+clean; 481 KiB data, 546 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:14:45.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:44 smithi090 ceph-mon[30399]: 11.18 scrub starts 2024-04-14T15:14:45.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:44 smithi090 ceph-mon[30399]: 11.18 scrub ok 2024-04-14T15:14:45.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:44 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:14:45.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:44 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:14:45.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:44 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:14:45.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:44 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:14:45.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:44 smithi090 ceph-mon[30399]: osdmap e217: 8 total, 8 up, 8 in 2024-04-14T15:14:45.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:44 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3131004652' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi090-93342-31"}]: dispatch 2024-04-14T15:14:45.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:44 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-19"}]: dispatch 2024-04-14T15:14:45.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:44 smithi090 ceph-mon[30399]: from='client.48481 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi090-93342-31"}]: dispatch 2024-04-14T15:14:45.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:44 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-19"}]: dispatch 2024-04-14T15:14:45.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:44 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:45.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:44 smithi090 ceph-mon[41670]: pgmap v269: 297 pgs: 1 clean+premerge+peered, 6 active+clean+snaptrim, 72 unknown, 7 active+clean+snaptrim_wait, 211 active+clean; 481 KiB data, 546 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:14:45.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:44 smithi090 ceph-mon[41670]: 11.18 scrub starts 2024-04-14T15:14:45.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:44 smithi090 ceph-mon[41670]: 11.18 scrub ok 2024-04-14T15:14:45.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:44 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:14:45.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:44 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:14:45.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:44 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:14:45.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:44 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:14:45.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:44 smithi090 ceph-mon[41670]: osdmap e217: 8 total, 8 up, 8 in 2024-04-14T15:14:45.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:44 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3131004652' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi090-93342-31"}]: dispatch 2024-04-14T15:14:45.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:44 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-19"}]: dispatch 2024-04-14T15:14:45.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:44 smithi090 ceph-mon[41670]: from='client.48481 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi090-93342-31"}]: dispatch 2024-04-14T15:14:45.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:44 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-19"}]: dispatch 2024-04-14T15:14:45.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:44 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:45.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:45.209 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:45.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:14:45.310 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:14:45.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:45.335 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:45.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:14:45.372 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:14:44 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:14:45.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:44 smithi092 ceph-mon[33966]: pgmap v269: 297 pgs: 1 clean+premerge+peered, 6 active+clean+snaptrim, 72 unknown, 7 active+clean+snaptrim_wait, 211 active+clean; 481 KiB data, 546 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:14:45.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:44 smithi092 ceph-mon[33966]: 11.18 scrub starts 2024-04-14T15:14:45.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:44 smithi092 ceph-mon[33966]: 11.18 scrub ok 2024-04-14T15:14:45.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:44 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:14:45.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:44 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:14:45.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:44 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:14:45.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:44 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:14:45.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:44 smithi092 ceph-mon[33966]: osdmap e217: 8 total, 8 up, 8 in 2024-04-14T15:14:45.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:44 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3131004652' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi090-93342-31"}]: dispatch 2024-04-14T15:14:45.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:44 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-19"}]: dispatch 2024-04-14T15:14:45.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:44 smithi092 ceph-mon[33966]: from='client.48481 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi090-93342-31"}]: dispatch 2024-04-14T15:14:45.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:44 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-19"}]: dispatch 2024-04-14T15:14:45.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:44 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:45.436 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:45.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:45.463 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:45.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:45.564 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:14:45.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:45.589 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:45.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:14:45.690 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:14:45.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:45.715 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:45.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:14:45.816 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:14:45.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:45.847 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:45.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:14:45.899 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-14T15:14:45.948 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:14:45.972 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:45.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:45.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:14:46.074 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:14:46.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:46.099 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:46.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:14:46.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:45 smithi090 ceph-mon[30399]: 11.1f scrub starts 2024-04-14T15:14:46.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:45 smithi090 ceph-mon[30399]: 11.1f scrub ok 2024-04-14T15:14:46.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:45 smithi090 ceph-mon[30399]: 2.0 scrub starts 2024-04-14T15:14:46.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:45 smithi090 ceph-mon[30399]: 2.0 scrub ok 2024-04-14T15:14:46.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:45 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:14:46.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:45 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-04-14T15:14:46.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:45 smithi090 ceph-mon[30399]: 4.10 scrub starts 2024-04-14T15:14:46.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:45 smithi090 ceph-mon[30399]: 4.10 scrub ok 2024-04-14T15:14:46.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:45 smithi090 ceph-mon[30399]: 11.a scrub starts 2024-04-14T15:14:46.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:45 smithi090 ceph-mon[30399]: 11.a scrub ok 2024-04-14T15:14:46.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:45 smithi090 ceph-mon[30399]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:14:46.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:45 smithi090 ceph-mon[30399]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:14:46.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:45 smithi090 ceph-mon[30399]: from='client.48481 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi090-93342-31"}]': finished 2024-04-14T15:14:46.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:45 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-19"}]': finished 2024-04-14T15:14:46.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:45 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "17"}]': finished 2024-04-14T15:14:46.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:45 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3131004652' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi090-93342-31"}]: dispatch 2024-04-14T15:14:46.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:45 smithi090 ceph-mon[30399]: osdmap e218: 8 total, 8 up, 8 in 2024-04-14T15:14:46.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:45 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2543828664' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi090-93363-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:46.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:45 smithi090 ceph-mon[30399]: from='client.48481 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi090-93342-31"}]: dispatch 2024-04-14T15:14:46.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:45 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:46.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:45 smithi090 ceph-mon[41670]: 11.1f scrub starts 2024-04-14T15:14:46.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:45 smithi090 ceph-mon[41670]: 11.1f scrub ok 2024-04-14T15:14:46.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:45 smithi090 ceph-mon[41670]: 2.0 scrub starts 2024-04-14T15:14:46.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:45 smithi090 ceph-mon[41670]: 2.0 scrub ok 2024-04-14T15:14:46.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:45 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:14:46.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:45 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-04-14T15:14:46.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:45 smithi090 ceph-mon[41670]: 4.10 scrub starts 2024-04-14T15:14:46.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:45 smithi090 ceph-mon[41670]: 4.10 scrub ok 2024-04-14T15:14:46.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:45 smithi090 ceph-mon[41670]: 11.a scrub starts 2024-04-14T15:14:46.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:45 smithi090 ceph-mon[41670]: 11.a scrub ok 2024-04-14T15:14:46.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:45 smithi090 ceph-mon[41670]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:14:46.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:45 smithi090 ceph-mon[41670]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:14:46.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:45 smithi090 ceph-mon[41670]: from='client.48481 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi090-93342-31"}]': finished 2024-04-14T15:14:46.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:45 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-19"}]': finished 2024-04-14T15:14:46.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:45 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "17"}]': finished 2024-04-14T15:14:46.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:45 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3131004652' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi090-93342-31"}]: dispatch 2024-04-14T15:14:46.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:45 smithi090 ceph-mon[41670]: osdmap e218: 8 total, 8 up, 8 in 2024-04-14T15:14:46.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:45 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2543828664' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi090-93363-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:46.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:45 smithi090 ceph-mon[41670]: from='client.48481 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi090-93342-31"}]: dispatch 2024-04-14T15:14:46.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:45 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:46.200 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:14:46.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:46.224 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:46.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:14:46.325 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:14:46.350 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:46.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:46.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:14:46.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:45 smithi092 ceph-mon[33966]: 11.1f scrub starts 2024-04-14T15:14:46.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:45 smithi092 ceph-mon[33966]: 11.1f scrub ok 2024-04-14T15:14:46.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:45 smithi092 ceph-mon[33966]: 2.0 scrub starts 2024-04-14T15:14:46.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:45 smithi092 ceph-mon[33966]: 2.0 scrub ok 2024-04-14T15:14:46.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:45 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:14:46.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:45 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-04-14T15:14:46.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:45 smithi092 ceph-mon[33966]: 4.10 scrub starts 2024-04-14T15:14:46.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:45 smithi092 ceph-mon[33966]: 4.10 scrub ok 2024-04-14T15:14:46.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:45 smithi092 ceph-mon[33966]: 11.a scrub starts 2024-04-14T15:14:46.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:45 smithi092 ceph-mon[33966]: 11.a scrub ok 2024-04-14T15:14:46.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:45 smithi092 ceph-mon[33966]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:14:46.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:45 smithi092 ceph-mon[33966]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:14:46.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:45 smithi092 ceph-mon[33966]: from='client.48481 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi090-93342-31"}]': finished 2024-04-14T15:14:46.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:45 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-19"}]': finished 2024-04-14T15:14:46.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:45 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "17"}]': finished 2024-04-14T15:14:46.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:45 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3131004652' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi090-93342-31"}]: dispatch 2024-04-14T15:14:46.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:45 smithi092 ceph-mon[33966]: osdmap e218: 8 total, 8 up, 8 in 2024-04-14T15:14:46.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:45 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2543828664' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi090-93363-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:46.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:45 smithi092 ceph-mon[33966]: from='client.48481 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi090-93342-31"}]: dispatch 2024-04-14T15:14:46.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:45 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:46.452 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:14:46.479 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:46.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:46.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:14:46.581 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:14:46.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:46.609 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:46.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:14:46.710 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:14:46.737 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:46.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:46.738 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:14:46.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:46.798+0000 7f08e6c6e700 1 -- 172.21.15.90:0/4157453291 >> v1:172.21.15.90:6789/0 conn(0x7f08e010e310 legacy=0x7f08e01107d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:46.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:46.798+0000 7f08e6c6e700 1 -- 172.21.15.90:0/4157453291 shutdown_connections 2024-04-14T15:14:46.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:46.798+0000 7f08e6c6e700 1 -- 172.21.15.90:0/4157453291 >> 172.21.15.90:0/4157453291 conn(0x7f08e00fd290 msgr2=0x7f08e00ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:14:46.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:46.798+0000 7f08e6c6e700 1 -- 172.21.15.90:0/4157453291 shutdown_connections 2024-04-14T15:14:46.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:46.798+0000 7f08e6c6e700 1 -- 172.21.15.90:0/4157453291 wait complete. 2024-04-14T15:14:46.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:46.798+0000 7f08e6c6e700 1 librados: starting msgr at 2024-04-14T15:14:46.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:46.798+0000 7f08e6c6e700 1 librados: starting objecter 2024-04-14T15:14:46.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:46.798+0000 7f08e6c6e700 1 Processor -- start 2024-04-14T15:14:46.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:46.798+0000 7f08e6c6e700 1 -- start start 2024-04-14T15:14:46.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:46.798+0000 7f08e6c6e700 1 librados: setting wanted keys 2024-04-14T15:14:46.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:46.798+0000 7f08e6c6e700 1 librados: calling monclient init 2024-04-14T15:14:46.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:46.799+0000 7f08e6c6e700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f08e01a7f20 con 0x7f08e010e310 2024-04-14T15:14:46.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:46.799+0000 7f08e6c6e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f08e01a9100 con 0x7f08e010a740 2024-04-14T15:14:46.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:46.799+0000 7f08e6c6e700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f08e01aa2e0 con 0x7f08e0107890 2024-04-14T15:14:46.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:46.799+0000 7f08e4a0a700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f08e0107890 0x7f08e010bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:33512/0 (socket says 172.21.15.90:33512) 2024-04-14T15:14:46.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:46.799+0000 7f08e4a0a700 1 -- 172.21.15.90:0/3837189474 learned_addr learned my addr 172.21.15.90:0/3837189474 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:14:46.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:46.799+0000 7f08ddffb700 10 client.?.objecter ms_handle_connect 0x7f08e010e310 2024-04-14T15:14:46.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:46.799+0000 7f08ddffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:14:46.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:46.799+0000 7f08ddffb700 10 client.?.objecter ms_handle_connect 0x7f08e0107890 2024-04-14T15:14:46.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:46.799+0000 7f08ddffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:14:46.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:46.799+0000 7f08ddffb700 10 client.?.objecter ms_handle_connect 0x7f08e010a740 2024-04-14T15:14:46.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:46.799+0000 7f08ddffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:14:46.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:46.799+0000 7f08ddffb700 1 -- 172.21.15.90:0/3837189474 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 213766669 0 0) 0x7f08e01a7f20 con 0x7f08e010e310 2024-04-14T15:14:46.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:46.799+0000 7f08ddffb700 1 -- 172.21.15.90:0/3837189474 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f08cc0035b0 con 0x7f08e010e310 2024-04-14T15:14:46.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:46.799+0000 7f08ddffb700 1 -- 172.21.15.90:0/3837189474 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4282540801 0 0) 0x7f08e01aa2e0 con 0x7f08e0107890 2024-04-14T15:14:46.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:46.799+0000 7f08ddffb700 1 -- 172.21.15.90:0/3837189474 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f08e01a7f20 con 0x7f08e0107890 2024-04-14T15:14:46.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:46.799+0000 7f08ddffb700 1 -- 172.21.15.90:0/3837189474 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4119679489 0 0) 0x7f08e01a9100 con 0x7f08e010a740 2024-04-14T15:14:46.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:46.799+0000 7f08ddffb700 1 -- 172.21.15.90:0/3837189474 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f08e01aa2e0 con 0x7f08e010a740 2024-04-14T15:14:46.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:46.799+0000 7f08ddffb700 1 -- 172.21.15.90:0/3837189474 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 999899186 0 0) 0x7f08e01a7f20 con 0x7f08e0107890 2024-04-14T15:14:46.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:46.799+0000 7f08ddffb700 1 -- 172.21.15.90:0/3837189474 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f08e01a9100 con 0x7f08e0107890 2024-04-14T15:14:46.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:46.799+0000 7f08ddffb700 1 -- 172.21.15.90:0/3837189474 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f08d8001bf0 con 0x7f08e0107890 2024-04-14T15:14:46.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:46.799+0000 7f08ddffb700 1 -- 172.21.15.90:0/3837189474 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2754914820 0 0) 0x7f08cc0035b0 con 0x7f08e010e310 2024-04-14T15:14:46.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:46.799+0000 7f08ddffb700 1 -- 172.21.15.90:0/3837189474 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f08e01a7f20 con 0x7f08e010e310 2024-04-14T15:14:46.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:46.799+0000 7f08ddffb700 1 -- 172.21.15.90:0/3837189474 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f08c8001430 con 0x7f08e010e310 2024-04-14T15:14:46.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:46.800+0000 7f08ddffb700 1 -- 172.21.15.90:0/3837189474 <== mon.2 v1:172.21.15.90:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2379370918 0 0) 0x7f08e01a9100 con 0x7f08e0107890 2024-04-14T15:14:46.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:46.800+0000 7f08ddffb700 1 -- 172.21.15.90:0/3837189474 >> v1:172.21.15.92:6789/0 conn(0x7f08e010a740 legacy=0x7f08e010c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:46.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:46.800+0000 7f08ddffb700 1 -- 172.21.15.90:0/3837189474 >> v1:172.21.15.90:6789/0 conn(0x7f08e010e310 legacy=0x7f08e01a6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:46.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:46.800+0000 7f08ddffb700 1 -- 172.21.15.90:0/3837189474 --> v1:172.21.15.90:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08e01ab4c0 con 0x7f08e0107890 2024-04-14T15:14:46.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:46.800+0000 7f08e6c6e700 1 -- 172.21.15.90:0/3837189474 --> v1:172.21.15.90:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f08e01aa510 con 0x7f08e0107890 2024-04-14T15:14:46.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:46.800+0000 7f08ddffb700 1 -- 172.21.15.90:0/3837189474 <== mon.2 v1:172.21.15.90:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f08d8003ec0 con 0x7f08e0107890 2024-04-14T15:14:46.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:46.800+0000 7f08ddffb700 1 -- 172.21.15.90:0/3837189474 <== mon.2 v1:172.21.15.90:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f08d80060f0 con 0x7f08e0107890 2024-04-14T15:14:46.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:46.800+0000 7f08e6c6e700 10 client.46610.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:14:46.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:46.800+0000 7f08e6c6e700 1 -- 172.21.15.90:0/3837189474 --> v1:172.21.15.90:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f08e01aa9d0 con 0x7f08e0107890 2024-04-14T15:14:46.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:46.800+0000 7f08e6c6e700 1 librados: init done 2024-04-14T15:14:46.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:46.801+0000 7f08e6c6e700 1 -- 172.21.15.90:0/3837189474 --> v1:172.21.15.90:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08e00693d0 con 0x7f08e0107890 2024-04-14T15:14:46.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:46.801+0000 7f08ddffb700 1 -- 172.21.15.90:0/3837189474 <== mon.2 v1:172.21.15.90:6790/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f08d8003a20 con 0x7f08e0107890 2024-04-14T15:14:46.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:46.802+0000 7f08ddffb700 1 -- 172.21.15.90:0/3837189474 <== mon.2 v1:172.21.15.90:6790/0 8 ==== osd_map(218..218 src has 51..218) ==== 8891+0+0 (unknown 4045986657 0 0) 0x7f08d808b100 con 0x7f08e0107890 2024-04-14T15:14:46.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:46.802+0000 7f08ddffb700 10 client.46610.objecter ms_dispatch 0x7f08e0001550 osd_map(218..218 src has 51..218) 2024-04-14T15:14:46.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:46.802+0000 7f08ddffb700 3 client.46610.objecter handle_osd_map got epochs [218,218] > 0 2024-04-14T15:14:46.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:46.802+0000 7f08ddffb700 3 client.46610.objecter handle_osd_map decoding full epoch 218 2024-04-14T15:14:46.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:46.802+0000 7f08ddffb700 20 client.46610.objecter dump_active .. 0 homeless 2024-04-14T15:14:46.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:46.802+0000 7f08ddffb700 10 client.46610.objecter ms_handle_connect 0x7f08cc0710d0 2024-04-14T15:14:46.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:46.807+0000 7f08ddffb700 1 -- 172.21.15.90:0/3837189474 <== mon.2 v1:172.21.15.90:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f08d8057850 con 0x7f08e0107890 2024-04-14T15:14:46.838 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:46.891 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:46.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:46.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:46.993 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:14:47.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:47.018 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:47.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:14:47.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:47.060+0000 7f08e6c6e700 1 -- 172.21.15.90:0/3837189474 --> v1:172.21.15.90:6790/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7f08e0114150 con 0x7f08e0107890 2024-04-14T15:14:47.119 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:14:47.143 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:47.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:47.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:14:47.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:46 smithi090 ceph-mon[41670]: pgmap v272: 256 pgs: 1 peering, 6 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 243 active+clean; 639 KiB data, 550 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:14:47.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:46 smithi090 ceph-mon[41670]: 5.6 scrub starts 2024-04-14T15:14:47.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:46 smithi090 ceph-mon[41670]: 5.6 scrub ok 2024-04-14T15:14:47.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:46 smithi090 ceph-mon[41670]: 13.5 scrub starts 2024-04-14T15:14:47.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:46 smithi090 ceph-mon[41670]: 13.5 scrub ok 2024-04-14T15:14:47.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:46 smithi090 ceph-mon[41670]: 5.9 scrub starts 2024-04-14T15:14:47.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:46 smithi090 ceph-mon[41670]: 5.9 scrub ok 2024-04-14T15:14:47.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:46 smithi090 ceph-mon[41670]: 5.c scrub starts 2024-04-14T15:14:47.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:46 smithi090 ceph-mon[41670]: 5.c scrub ok 2024-04-14T15:14:47.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:46 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2543828664' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi090-93363-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:47.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:46 smithi090 ceph-mon[41670]: from='client.48481 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi090-93342-31"}]': finished 2024-04-14T15:14:47.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:46 smithi090 ceph-mon[41670]: osdmap e219: 8 total, 8 up, 8 in 2024-04-14T15:14:47.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:46 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1251627847' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi090-93342-32"}]: dispatch 2024-04-14T15:14:47.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:46 smithi090 ceph-mon[41670]: from='client.48511 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi090-93342-32"}]: dispatch 2024-04-14T15:14:47.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:46 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1251627847' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi090-93342-32"}]: dispatch 2024-04-14T15:14:47.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:46 smithi090 ceph-mon[41670]: from='client.48511 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi090-93342-32"}]: dispatch 2024-04-14T15:14:47.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:46 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1251627847' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi090-93342-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:14:47.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:46 smithi090 ceph-mon[41670]: from='client.48511 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi090-93342-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:14:47.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:46 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:47.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:46 smithi090 ceph-mon[30399]: pgmap v272: 256 pgs: 1 peering, 6 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 243 active+clean; 639 KiB data, 550 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:14:47.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:46 smithi090 ceph-mon[30399]: 5.6 scrub starts 2024-04-14T15:14:47.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:46 smithi090 ceph-mon[30399]: 5.6 scrub ok 2024-04-14T15:14:47.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:46 smithi090 ceph-mon[30399]: 13.5 scrub starts 2024-04-14T15:14:47.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:46 smithi090 ceph-mon[30399]: 13.5 scrub ok 2024-04-14T15:14:47.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:46 smithi090 ceph-mon[30399]: 5.9 scrub starts 2024-04-14T15:14:47.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:46 smithi090 ceph-mon[30399]: 5.9 scrub ok 2024-04-14T15:14:47.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:46 smithi090 ceph-mon[30399]: 5.c scrub starts 2024-04-14T15:14:47.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:46 smithi090 ceph-mon[30399]: 5.c scrub ok 2024-04-14T15:14:47.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:46 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2543828664' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi090-93363-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:47.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:46 smithi090 ceph-mon[30399]: from='client.48481 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi090-93342-31"}]': finished 2024-04-14T15:14:47.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:46 smithi090 ceph-mon[30399]: osdmap e219: 8 total, 8 up, 8 in 2024-04-14T15:14:47.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:46 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1251627847' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi090-93342-32"}]: dispatch 2024-04-14T15:14:47.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:46 smithi090 ceph-mon[30399]: from='client.48511 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi090-93342-32"}]: dispatch 2024-04-14T15:14:47.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:46 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1251627847' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi090-93342-32"}]: dispatch 2024-04-14T15:14:47.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:46 smithi090 ceph-mon[30399]: from='client.48511 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi090-93342-32"}]: dispatch 2024-04-14T15:14:47.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:46 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1251627847' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi090-93342-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:14:47.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:46 smithi090 ceph-mon[30399]: from='client.48511 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi090-93342-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:14:47.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:46 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:47.245 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:14:47.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:47.270 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:47.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:14:47.371 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:14:47.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:46 smithi092 ceph-mon[33966]: pgmap v272: 256 pgs: 1 peering, 6 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 243 active+clean; 639 KiB data, 550 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:14:47.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:46 smithi092 ceph-mon[33966]: 5.6 scrub starts 2024-04-14T15:14:47.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:46 smithi092 ceph-mon[33966]: 5.6 scrub ok 2024-04-14T15:14:47.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:46 smithi092 ceph-mon[33966]: 13.5 scrub starts 2024-04-14T15:14:47.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:46 smithi092 ceph-mon[33966]: 13.5 scrub ok 2024-04-14T15:14:47.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:46 smithi092 ceph-mon[33966]: 5.9 scrub starts 2024-04-14T15:14:47.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:46 smithi092 ceph-mon[33966]: 5.9 scrub ok 2024-04-14T15:14:47.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:46 smithi092 ceph-mon[33966]: 5.c scrub starts 2024-04-14T15:14:47.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:46 smithi092 ceph-mon[33966]: 5.c scrub ok 2024-04-14T15:14:47.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:46 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2543828664' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi090-93363-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:47.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:46 smithi092 ceph-mon[33966]: from='client.48481 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi090-93342-31"}]': finished 2024-04-14T15:14:47.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:46 smithi092 ceph-mon[33966]: osdmap e219: 8 total, 8 up, 8 in 2024-04-14T15:14:47.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:46 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1251627847' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi090-93342-32"}]: dispatch 2024-04-14T15:14:47.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:46 smithi092 ceph-mon[33966]: from='client.48511 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi090-93342-32"}]: dispatch 2024-04-14T15:14:47.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:46 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1251627847' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi090-93342-32"}]: dispatch 2024-04-14T15:14:47.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:46 smithi092 ceph-mon[33966]: from='client.48511 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi090-93342-32"}]: dispatch 2024-04-14T15:14:47.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:46 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1251627847' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi090-93342-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:14:47.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:46 smithi092 ceph-mon[33966]: from='client.48511 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi090-93342-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:14:47.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:46 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:47.394 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:47.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:47.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:14:47.443 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:14:47.482 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:14:47.496 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:47.521 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:47.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:47.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:47.623 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:14:47.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:47.647 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:47.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:14:47.749 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:14:47.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:47.773 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:47.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:14:47.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:47.837+0000 7f08ddffb700 1 -- 172.21.15.90:0/3837189474 <== mon.2 v1:172.21.15.90:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v220) ==== 101+0+0 (unknown 3839916639 0 0) 0x7f08d805b500 con 0x7f08e0107890 2024-04-14T15:14:47.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:47.841+0000 7f08e6c6e700 10 librados: watch_flush enter 2024-04-14T15:14:47.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:47.841+0000 7f08e6c6e700 10 librados: watch_flush exit 2024-04-14T15:14:47.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:47.841+0000 7f08e6c6e700 20 client.46610.objecter shutdown clearing up homeless session... 2024-04-14T15:14:47.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:47.841+0000 7f08e6c6e700 10 client.46610.objecter successfully canceled tick 2024-04-14T15:14:47.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:47.841+0000 7f08e6c6e700 1 -- 172.21.15.90:0/3837189474 >> v1:172.21.15.90:6800/420935427 conn(0x7f08cc0710d0 legacy=0x7f08cc073590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:47.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:47.841+0000 7f08e6c6e700 1 -- 172.21.15.90:0/3837189474 >> v1:172.21.15.90:6790/0 conn(0x7f08e0107890 legacy=0x7f08e010bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:47.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:47.841+0000 7f08e6c6e700 1 -- 172.21.15.90:0/3837189474 shutdown_connections 2024-04-14T15:14:47.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:47.841+0000 7f08e6c6e700 1 -- 172.21.15.90:0/3837189474 >> 172.21.15.90:0/3837189474 conn(0x7f08e00fd290 msgr2=0x7f08e006b060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:14:47.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:47.842+0000 7f08e6c6e700 1 -- 172.21.15.90:0/3837189474 shutdown_connections 2024-04-14T15:14:47.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:47.842+0000 7f08e6c6e700 1 -- 172.21.15.90:0/3837189474 wait complete. 2024-04-14T15:14:47.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:47.842+0000 7f08e6c6e700 1 librados: shutdown 2024-04-14T15:14:47.843 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is unset 2024-04-14T15:14:47.874 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:14:47.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:47.899 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:47.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:14:48.000 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:48.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:48.024 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:48.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:48.126 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:14:48.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:48.151 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:48.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:14:48.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:47 smithi090 ceph-mon[41670]: 6.14 scrub starts 2024-04-14T15:14:48.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:47 smithi090 ceph-mon[41670]: 6.14 scrub ok 2024-04-14T15:14:48.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:47 smithi090 ceph-mon[41670]: 1.9 scrub starts 2024-04-14T15:14:48.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:47 smithi090 ceph-mon[41670]: 1.9 scrub ok 2024-04-14T15:14:48.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:47 smithi090 ceph-mon[41670]: 6.c scrub starts 2024-04-14T15:14:48.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:47 smithi090 ceph-mon[41670]: 6.c scrub ok 2024-04-14T15:14:48.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:47 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3837189474' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:14:48.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:47 smithi090 ceph-mon[41670]: from='client.46610 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:14:48.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:47 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2543828664' entity='client.admin' cmd=[{ 2024-04-14T15:14:48.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:47 smithi090 ceph-mon[41670]: "prefix": "osd pool set", 2024-04-14T15:14:48.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:47 smithi090 ceph-mon[41670]: "pool": "PoolEIOFlag_smithi090-93363-33", 2024-04-14T15:14:48.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:47 smithi090 ceph-mon[41670]: "var": "eio", 2024-04-14T15:14:48.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:47 smithi090 ceph-mon[41670]: "val": "true" 2024-04-14T15:14:48.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:47 smithi090 ceph-mon[41670]: }]: dispatch 2024-04-14T15:14:48.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:47 smithi090 ceph-mon[41670]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-14T15:14:48.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:47 smithi090 ceph-mon[41670]: from='client.48511 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi090-93342-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:14:48.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:47 smithi090 ceph-mon[41670]: from='client.46610 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-14T15:14:48.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:47 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2543828664' entity='client.admin' cmd='[{ 2024-04-14T15:14:48.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:47 smithi090 ceph-mon[41670]: "prefix": "osd pool set", 2024-04-14T15:14:48.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:47 smithi090 ceph-mon[41670]: "pool": "PoolEIOFlag_smithi090-93363-33", 2024-04-14T15:14:48.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:47 smithi090 ceph-mon[41670]: "var": "eio", 2024-04-14T15:14:48.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:47 smithi090 ceph-mon[41670]: "val": "true" 2024-04-14T15:14:48.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:47 smithi090 ceph-mon[41670]: }]': finished 2024-04-14T15:14:48.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:47 smithi090 ceph-mon[41670]: osdmap e220: 8 total, 8 up, 8 in 2024-04-14T15:14:48.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:47 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1251627847' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi090-93342-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi090-93342-32"}]: dispatch 2024-04-14T15:14:48.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:47 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:48.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:47 smithi090 ceph-mon[41670]: from='client.48511 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi090-93342-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi090-93342-32"}]: dispatch 2024-04-14T15:14:48.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:47 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:48.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:47 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:48.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:47 smithi090 ceph-mon[30399]: 6.14 scrub starts 2024-04-14T15:14:48.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:47 smithi090 ceph-mon[30399]: 6.14 scrub ok 2024-04-14T15:14:48.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:47 smithi090 ceph-mon[30399]: 1.9 scrub starts 2024-04-14T15:14:48.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:47 smithi090 ceph-mon[30399]: 1.9 scrub ok 2024-04-14T15:14:48.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:47 smithi090 ceph-mon[30399]: 6.c scrub starts 2024-04-14T15:14:48.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:47 smithi090 ceph-mon[30399]: 6.c scrub ok 2024-04-14T15:14:48.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:47 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3837189474' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:14:48.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:47 smithi090 ceph-mon[30399]: from='client.46610 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:14:48.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:47 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2543828664' entity='client.admin' cmd=[{ 2024-04-14T15:14:48.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:47 smithi090 ceph-mon[30399]: "prefix": "osd pool set", 2024-04-14T15:14:48.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:47 smithi090 ceph-mon[30399]: "pool": "PoolEIOFlag_smithi090-93363-33", 2024-04-14T15:14:48.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:47 smithi090 ceph-mon[30399]: "var": "eio", 2024-04-14T15:14:48.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:47 smithi090 ceph-mon[30399]: "val": "true" 2024-04-14T15:14:48.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:47 smithi090 ceph-mon[30399]: }]: dispatch 2024-04-14T15:14:48.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:47 smithi090 ceph-mon[30399]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-14T15:14:48.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:47 smithi090 ceph-mon[30399]: from='client.48511 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi090-93342-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:14:48.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:47 smithi090 ceph-mon[30399]: from='client.46610 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-14T15:14:48.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:47 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2543828664' entity='client.admin' cmd='[{ 2024-04-14T15:14:48.196 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:47 smithi090 ceph-mon[30399]: "prefix": "osd pool set", 2024-04-14T15:14:48.196 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:47 smithi090 ceph-mon[30399]: "pool": "PoolEIOFlag_smithi090-93363-33", 2024-04-14T15:14:48.196 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:47 smithi090 ceph-mon[30399]: "var": "eio", 2024-04-14T15:14:48.196 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:47 smithi090 ceph-mon[30399]: "val": "true" 2024-04-14T15:14:48.196 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:47 smithi090 ceph-mon[30399]: }]': finished 2024-04-14T15:14:48.196 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:47 smithi090 ceph-mon[30399]: osdmap e220: 8 total, 8 up, 8 in 2024-04-14T15:14:48.196 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:47 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1251627847' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi090-93342-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi090-93342-32"}]: dispatch 2024-04-14T15:14:48.196 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:47 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:48.196 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:47 smithi090 ceph-mon[30399]: from='client.48511 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi090-93342-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi090-93342-32"}]: dispatch 2024-04-14T15:14:48.196 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:47 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:48.196 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:47 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:48.253 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:14:48.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:48.277 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:48.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:14:48.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:47 smithi092 ceph-mon[33966]: 6.14 scrub starts 2024-04-14T15:14:48.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:47 smithi092 ceph-mon[33966]: 6.14 scrub ok 2024-04-14T15:14:48.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:47 smithi092 ceph-mon[33966]: 1.9 scrub starts 2024-04-14T15:14:48.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:47 smithi092 ceph-mon[33966]: 1.9 scrub ok 2024-04-14T15:14:48.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:47 smithi092 ceph-mon[33966]: 6.c scrub starts 2024-04-14T15:14:48.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:47 smithi092 ceph-mon[33966]: 6.c scrub ok 2024-04-14T15:14:48.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:47 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3837189474' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:14:48.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:47 smithi092 ceph-mon[33966]: from='client.46610 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:14:48.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:47 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2543828664' entity='client.admin' cmd=[{ 2024-04-14T15:14:48.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:47 smithi092 ceph-mon[33966]: "prefix": "osd pool set", 2024-04-14T15:14:48.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:47 smithi092 ceph-mon[33966]: "pool": "PoolEIOFlag_smithi090-93363-33", 2024-04-14T15:14:48.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:47 smithi092 ceph-mon[33966]: "var": "eio", 2024-04-14T15:14:48.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:47 smithi092 ceph-mon[33966]: "val": "true" 2024-04-14T15:14:48.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:47 smithi092 ceph-mon[33966]: }]: dispatch 2024-04-14T15:14:48.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:47 smithi092 ceph-mon[33966]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-14T15:14:48.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:47 smithi092 ceph-mon[33966]: from='client.48511 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi090-93342-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:14:48.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:47 smithi092 ceph-mon[33966]: from='client.46610 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-14T15:14:48.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:47 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2543828664' entity='client.admin' cmd='[{ 2024-04-14T15:14:48.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:47 smithi092 ceph-mon[33966]: "prefix": "osd pool set", 2024-04-14T15:14:48.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:47 smithi092 ceph-mon[33966]: "pool": "PoolEIOFlag_smithi090-93363-33", 2024-04-14T15:14:48.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:47 smithi092 ceph-mon[33966]: "var": "eio", 2024-04-14T15:14:48.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:47 smithi092 ceph-mon[33966]: "val": "true" 2024-04-14T15:14:48.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:47 smithi092 ceph-mon[33966]: }]': finished 2024-04-14T15:14:48.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:47 smithi092 ceph-mon[33966]: osdmap e220: 8 total, 8 up, 8 in 2024-04-14T15:14:48.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:47 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1251627847' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi090-93342-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi090-93342-32"}]: dispatch 2024-04-14T15:14:48.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:47 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:48.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:47 smithi092 ceph-mon[33966]: from='client.48511 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi090-93342-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi090-93342-32"}]: dispatch 2024-04-14T15:14:48.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:47 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:48.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:47 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:48.378 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:14:48.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:48.402 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:48.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:14:48.503 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:14:48.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:48.528 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:48.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:14:48.629 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:48.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:48.653 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:48.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:48.754 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:14:48.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:48.780 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:48.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:14:48.881 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:14:48.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:48.906 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:48.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:14:49.008 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:49.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:49.033 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:49.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:49.134 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:14:49.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:49.159 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:49.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:14:49.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:48 smithi090 ceph-mon[41670]: pgmap v275: 256 pgs: 1 peering, 6 active+clean+snaptrim, 32 unknown, 6 active+clean+snaptrim_wait, 211 active+clean; 639 KiB data, 550 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:14:49.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:48 smithi090 ceph-mon[41670]: 13.e scrub starts 2024-04-14T15:14:49.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:48 smithi090 ceph-mon[41670]: 13.e scrub ok 2024-04-14T15:14:49.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:48 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:49.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:48 smithi090 ceph-mon[41670]: osdmap e221: 8 total, 8 up, 8 in 2024-04-14T15:14:49.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:48 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:14:49.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:48 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:14:49.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:48 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:49.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:48 smithi090 ceph-mon[30399]: pgmap v275: 256 pgs: 1 peering, 6 active+clean+snaptrim, 32 unknown, 6 active+clean+snaptrim_wait, 211 active+clean; 639 KiB data, 550 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:14:49.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:48 smithi090 ceph-mon[30399]: 13.e scrub starts 2024-04-14T15:14:49.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:48 smithi090 ceph-mon[30399]: 13.e scrub ok 2024-04-14T15:14:49.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:48 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:49.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:48 smithi090 ceph-mon[30399]: osdmap e221: 8 total, 8 up, 8 in 2024-04-14T15:14:49.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:48 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:14:49.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:48 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:14:49.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:48 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:49.261 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:14:49.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:49.284 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:49.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:14:49.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:48 smithi092 ceph-mon[33966]: pgmap v275: 256 pgs: 1 peering, 6 active+clean+snaptrim, 32 unknown, 6 active+clean+snaptrim_wait, 211 active+clean; 639 KiB data, 550 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:14:49.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:48 smithi092 ceph-mon[33966]: 13.e scrub starts 2024-04-14T15:14:49.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:48 smithi092 ceph-mon[33966]: 13.e scrub ok 2024-04-14T15:14:49.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:48 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:49.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:48 smithi092 ceph-mon[33966]: osdmap e221: 8 total, 8 up, 8 in 2024-04-14T15:14:49.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:48 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:14:49.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:48 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:14:49.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:48 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:49.386 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:14:49.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:49.411 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:49.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:14:49.512 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:14:49.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:49.540 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:49.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:14:49.641 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:14:49.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:49.668 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:49.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:14:49.769 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:14:49.794 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:49.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:49.795 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:14:49.896 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:14:49.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:49.923 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:49.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:14:50.024 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:14:50.049 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:50.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:50.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:14:50.151 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:50.173 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:49 smithi090 ceph-mon[41670]: 110.1c scrub starts 2024-04-14T15:14:50.173 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:49 smithi090 ceph-mon[41670]: 110.1c scrub ok 2024-04-14T15:14:50.173 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:49 smithi090 ceph-mon[41670]: 13.d scrub starts 2024-04-14T15:14:50.173 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:49 smithi090 ceph-mon[41670]: 13.d scrub ok 2024-04-14T15:14:50.173 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:49 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-04-14T15:14:50.173 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:49 smithi090 ceph-mon[41670]: from='client.48511 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi090-93342-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi090-93342-32"}]': finished 2024-04-14T15:14:50.174 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:49 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-21", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:14:50.174 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:49 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "16"}]': finished 2024-04-14T15:14:50.174 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:49 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-21"}]: dispatch 2024-04-14T15:14:50.174 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:49 smithi090 ceph-mon[41670]: osdmap e222: 8 total, 8 up, 8 in 2024-04-14T15:14:50.174 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:49 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-21"}]: dispatch 2024-04-14T15:14:50.174 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:49 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/4069049515' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi090-93363-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:50.174 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:49 smithi090 ceph-mon[41670]: from='client.48517 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi090-93363-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:50.174 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:49 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:50.174 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:49 smithi090 ceph-mon[30399]: 110.1c scrub starts 2024-04-14T15:14:50.174 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:49 smithi090 ceph-mon[30399]: 110.1c scrub ok 2024-04-14T15:14:50.175 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:49 smithi090 ceph-mon[30399]: 13.d scrub starts 2024-04-14T15:14:50.175 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:49 smithi090 ceph-mon[30399]: 13.d scrub ok 2024-04-14T15:14:50.175 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:49 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-04-14T15:14:50.175 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:49 smithi090 ceph-mon[30399]: from='client.48511 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi090-93342-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi090-93342-32"}]': finished 2024-04-14T15:14:50.175 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:49 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-21", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:14:50.175 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:49 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "16"}]': finished 2024-04-14T15:14:50.175 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:49 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-21"}]: dispatch 2024-04-14T15:14:50.175 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:49 smithi090 ceph-mon[30399]: osdmap e222: 8 total, 8 up, 8 in 2024-04-14T15:14:50.175 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:49 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-21"}]: dispatch 2024-04-14T15:14:50.175 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:49 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/4069049515' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi090-93363-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:50.175 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:49 smithi090 ceph-mon[30399]: from='client.48517 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi090-93363-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:50.175 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:49 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:50.180 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:50.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:50.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:50.282 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:14:50.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:50.309 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:50.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:14:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:49 smithi092 ceph-mon[33966]: 110.1c scrub starts 2024-04-14T15:14:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:49 smithi092 ceph-mon[33966]: 110.1c scrub ok 2024-04-14T15:14:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:49 smithi092 ceph-mon[33966]: 13.d scrub starts 2024-04-14T15:14:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:49 smithi092 ceph-mon[33966]: 13.d scrub ok 2024-04-14T15:14:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:49 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-04-14T15:14:50.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:49 smithi092 ceph-mon[33966]: from='client.48511 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi090-93342-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi090-93342-32"}]': finished 2024-04-14T15:14:50.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:49 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-21", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:14:50.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:49 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "16"}]': finished 2024-04-14T15:14:50.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:49 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-21"}]: dispatch 2024-04-14T15:14:50.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:49 smithi092 ceph-mon[33966]: osdmap e222: 8 total, 8 up, 8 in 2024-04-14T15:14:50.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:49 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-21"}]: dispatch 2024-04-14T15:14:50.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:49 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/4069049515' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi090-93363-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:50.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:49 smithi092 ceph-mon[33966]: from='client.48517 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi090-93363-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:50.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:49 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:50.411 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:14:50.436 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:50.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:50.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:14:50.538 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:14:50.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:50.566 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:50.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:14:50.667 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:14:50.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:50.695 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:50.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:14:50.796 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:14:50.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:50.823 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:50.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:14:50.924 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:14:50.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:50.950 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:50.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:14:51.051 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:14:51.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:51.077 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:51.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:14:51.178 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:14:51.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:50 smithi090 ceph-mon[41670]: pgmap v278: 255 pgs: 4 active+clean+snaptrim, 10 unknown, 4 active+clean+snaptrim_wait, 237 active+clean; 586 KiB data, 551 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:14:51.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:50 smithi090 ceph-mon[41670]: 11.d scrub starts 2024-04-14T15:14:51.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:50 smithi090 ceph-mon[41670]: 11.d scrub ok 2024-04-14T15:14:51.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:50 smithi090 ceph-mon[41670]: 4.a deep-scrub starts 2024-04-14T15:14:51.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:50 smithi090 ceph-mon[41670]: 4.a deep-scrub ok 2024-04-14T15:14:51.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:50 smithi090 ceph-mon[41670]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:14:51.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:50 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-21"}]': finished 2024-04-14T15:14:51.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:50 smithi090 ceph-mon[41670]: from='client.48517 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi090-93363-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:51.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:50 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-21", "mode": "writeback"}]: dispatch 2024-04-14T15:14:51.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:50 smithi090 ceph-mon[41670]: osdmap e223: 8 total, 8 up, 8 in 2024-04-14T15:14:51.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:50 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-21", "mode": "writeback"}]: dispatch 2024-04-14T15:14:51.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:50 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:51.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:50 smithi090 ceph-mon[30399]: pgmap v278: 255 pgs: 4 active+clean+snaptrim, 10 unknown, 4 active+clean+snaptrim_wait, 237 active+clean; 586 KiB data, 551 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:14:51.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:50 smithi090 ceph-mon[30399]: 11.d scrub starts 2024-04-14T15:14:51.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:50 smithi090 ceph-mon[30399]: 11.d scrub ok 2024-04-14T15:14:51.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:50 smithi090 ceph-mon[30399]: 4.a deep-scrub starts 2024-04-14T15:14:51.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:50 smithi090 ceph-mon[30399]: 4.a deep-scrub ok 2024-04-14T15:14:51.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:50 smithi090 ceph-mon[30399]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:14:51.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:50 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-21"}]': finished 2024-04-14T15:14:51.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:50 smithi090 ceph-mon[30399]: from='client.48517 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi090-93363-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:51.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:50 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-21", "mode": "writeback"}]: dispatch 2024-04-14T15:14:51.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:50 smithi090 ceph-mon[30399]: osdmap e223: 8 total, 8 up, 8 in 2024-04-14T15:14:51.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:50 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-21", "mode": "writeback"}]: dispatch 2024-04-14T15:14:51.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:50 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:51.203 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:51.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:51.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:14:51.305 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:14:51.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:51.330 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:51.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:14:51.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:50 smithi092 ceph-mon[33966]: pgmap v278: 255 pgs: 4 active+clean+snaptrim, 10 unknown, 4 active+clean+snaptrim_wait, 237 active+clean; 586 KiB data, 551 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:14:51.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:50 smithi092 ceph-mon[33966]: 11.d scrub starts 2024-04-14T15:14:51.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:50 smithi092 ceph-mon[33966]: 11.d scrub ok 2024-04-14T15:14:51.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:50 smithi092 ceph-mon[33966]: 4.a deep-scrub starts 2024-04-14T15:14:51.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:50 smithi092 ceph-mon[33966]: 4.a deep-scrub ok 2024-04-14T15:14:51.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:50 smithi092 ceph-mon[33966]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:14:51.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:50 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-21"}]': finished 2024-04-14T15:14:51.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:50 smithi092 ceph-mon[33966]: from='client.48517 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi090-93363-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:51.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:50 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-21", "mode": "writeback"}]: dispatch 2024-04-14T15:14:51.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:50 smithi092 ceph-mon[33966]: osdmap e223: 8 total, 8 up, 8 in 2024-04-14T15:14:51.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:50 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-21", "mode": "writeback"}]: dispatch 2024-04-14T15:14:51.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:50 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:51.432 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:14:51.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:51.457 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:51.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:14:51.558 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:14:51.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:51.583 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:51.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:14:51.684 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:51.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:51.709 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:51.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:51.810 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:14:51.835 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:51.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:51.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:14:51.883 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-14T15:14:51.883 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-14T15:14:51.883 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-14T15:14:51.883 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-14T15:14:51.937 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:14:51.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:51.963 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:51.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:14:52.064 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:14:52.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:52.094 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:52.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:14:52.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:51 smithi090 ceph-mon[41670]: 13.f scrub starts 2024-04-14T15:14:52.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:51 smithi090 ceph-mon[41670]: 13.f scrub ok 2024-04-14T15:14:52.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:51 smithi090 ceph-mon[41670]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:14:52.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:51 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-21", "mode": "writeback"}]': finished 2024-04-14T15:14:52.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:51 smithi090 ceph-mon[41670]: osdmap e224: 8 total, 8 up, 8 in 2024-04-14T15:14:52.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:51 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1251627847' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi090-93342-32"}]: dispatch 2024-04-14T15:14:52.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:51 smithi090 ceph-mon[41670]: from='client.48511 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi090-93342-32"}]: dispatch 2024-04-14T15:14:52.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:51 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:14:52.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:51 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:14:52.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:51 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:52.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:51 smithi090 ceph-mon[30399]: 13.f scrub starts 2024-04-14T15:14:52.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:51 smithi090 ceph-mon[30399]: 13.f scrub ok 2024-04-14T15:14:52.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:51 smithi090 ceph-mon[30399]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:14:52.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:51 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-21", "mode": "writeback"}]': finished 2024-04-14T15:14:52.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:51 smithi090 ceph-mon[30399]: osdmap e224: 8 total, 8 up, 8 in 2024-04-14T15:14:52.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:51 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1251627847' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi090-93342-32"}]: dispatch 2024-04-14T15:14:52.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:51 smithi090 ceph-mon[30399]: from='client.48511 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi090-93342-32"}]: dispatch 2024-04-14T15:14:52.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:51 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:14:52.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:51 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:14:52.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:51 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:52.197 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:14:52.205 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd set noscrub 2024-04-14T15:14:52.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:52.221 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:52.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:14:52.273 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:51 smithi092 ceph-mon[33966]: 13.f scrub starts 2024-04-14T15:14:52.273 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:51 smithi092 ceph-mon[33966]: 13.f scrub ok 2024-04-14T15:14:52.274 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:51 smithi092 ceph-mon[33966]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:14:52.274 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:51 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-21", "mode": "writeback"}]': finished 2024-04-14T15:14:52.274 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:51 smithi092 ceph-mon[33966]: osdmap e224: 8 total, 8 up, 8 in 2024-04-14T15:14:52.274 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:51 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1251627847' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi090-93342-32"}]: dispatch 2024-04-14T15:14:52.274 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:51 smithi092 ceph-mon[33966]: from='client.48511 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi090-93342-32"}]: dispatch 2024-04-14T15:14:52.274 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:51 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:14:52.274 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:51 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:14:52.274 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:51 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:52.323 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:14:52.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:52.347 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:52.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:14:52.448 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:14:52.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:52.474 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:52.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:14:52.576 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:52.599 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-14T15:14:52.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:52.602 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:52.602 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:52.704 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:14:52.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:52.730 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:52.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:14:52.832 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:14:52.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:52.874 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:52.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:14:52.975 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:52.986 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:52 smithi092 ceph-mon[33966]: pgmap v281: 295 pgs: 4 active+clean+snaptrim, 40 unknown, 4 active+clean+snaptrim_wait, 247 active+clean; 586 KiB data, 551 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:14:52.986 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:52 smithi092 ceph-mon[33966]: 110.b scrub starts 2024-04-14T15:14:52.986 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:52 smithi092 ceph-mon[33966]: 110.b scrub ok 2024-04-14T15:14:52.986 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:52 smithi092 ceph-mon[33966]: 13.7 deep-scrub starts 2024-04-14T15:14:52.986 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:52 smithi092 ceph-mon[33966]: 13.7 deep-scrub ok 2024-04-14T15:14:52.986 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:52 smithi092 ceph-mon[33966]: 11.3 scrub starts 2024-04-14T15:14:52.986 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:52 smithi092 ceph-mon[33966]: 11.3 scrub ok 2024-04-14T15:14:52.987 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:52 smithi092 ceph-mon[33966]: 13.9 scrub starts 2024-04-14T15:14:52.987 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:52 smithi092 ceph-mon[33966]: 13.9 scrub ok 2024-04-14T15:14:52.987 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:52 smithi092 ceph-mon[33966]: from='client.48511 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi090-93342-32"}]': finished 2024-04-14T15:14:52.987 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:52 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:14:52.987 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:52 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-21"}]: dispatch 2024-04-14T15:14:52.987 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:52 smithi092 ceph-mon[33966]: osdmap e225: 8 total, 8 up, 8 in 2024-04-14T15:14:52.987 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:52 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-21"}]: dispatch 2024-04-14T15:14:52.987 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:52 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1058124934' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi090-93363-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:52.987 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:52 smithi092 ceph-mon[33966]: from='client.48523 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi090-93363-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:52.987 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:52 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1251627847' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi090-93342-32"}]: dispatch 2024-04-14T15:14:52.987 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:52 smithi092 ceph-mon[33966]: from='client.48511 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi090-93342-32"}]: dispatch 2024-04-14T15:14:52.987 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:52 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:52.999 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:53.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:53.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:53.094 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:52 smithi090 ceph-mon[41670]: pgmap v281: 295 pgs: 4 active+clean+snaptrim, 40 unknown, 4 active+clean+snaptrim_wait, 247 active+clean; 586 KiB data, 551 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:14:53.094 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:52 smithi090 ceph-mon[41670]: 110.b scrub starts 2024-04-14T15:14:53.094 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:52 smithi090 ceph-mon[41670]: 110.b scrub ok 2024-04-14T15:14:53.094 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:52 smithi090 ceph-mon[41670]: 13.7 deep-scrub starts 2024-04-14T15:14:53.094 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:52 smithi090 ceph-mon[41670]: 13.7 deep-scrub ok 2024-04-14T15:14:53.094 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:52 smithi090 ceph-mon[41670]: 11.3 scrub starts 2024-04-14T15:14:53.094 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:52 smithi090 ceph-mon[41670]: 11.3 scrub ok 2024-04-14T15:14:53.094 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:52 smithi090 ceph-mon[41670]: 13.9 scrub starts 2024-04-14T15:14:53.094 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:52 smithi090 ceph-mon[41670]: 13.9 scrub ok 2024-04-14T15:14:53.095 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:52 smithi090 ceph-mon[41670]: from='client.48511 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi090-93342-32"}]': finished 2024-04-14T15:14:53.095 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:52 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:14:53.095 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:52 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-21"}]: dispatch 2024-04-14T15:14:53.095 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:52 smithi090 ceph-mon[41670]: osdmap e225: 8 total, 8 up, 8 in 2024-04-14T15:14:53.095 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:52 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-21"}]: dispatch 2024-04-14T15:14:53.095 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:52 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1058124934' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi090-93363-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:53.095 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:52 smithi090 ceph-mon[41670]: from='client.48523 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi090-93363-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:53.095 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:52 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1251627847' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi090-93342-32"}]: dispatch 2024-04-14T15:14:53.095 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:52 smithi090 ceph-mon[41670]: from='client.48511 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi090-93342-32"}]: dispatch 2024-04-14T15:14:53.095 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:52 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:53.095 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:52 smithi090 ceph-mon[30399]: pgmap v281: 295 pgs: 4 active+clean+snaptrim, 40 unknown, 4 active+clean+snaptrim_wait, 247 active+clean; 586 KiB data, 551 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:14:53.095 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:52 smithi090 ceph-mon[30399]: 110.b scrub starts 2024-04-14T15:14:53.096 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:52 smithi090 ceph-mon[30399]: 110.b scrub ok 2024-04-14T15:14:53.096 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:52 smithi090 ceph-mon[30399]: 13.7 deep-scrub starts 2024-04-14T15:14:53.096 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:52 smithi090 ceph-mon[30399]: 13.7 deep-scrub ok 2024-04-14T15:14:53.096 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:52 smithi090 ceph-mon[30399]: 11.3 scrub starts 2024-04-14T15:14:53.096 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:52 smithi090 ceph-mon[30399]: 11.3 scrub ok 2024-04-14T15:14:53.096 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:52 smithi090 ceph-mon[30399]: 13.9 scrub starts 2024-04-14T15:14:53.096 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:52 smithi090 ceph-mon[30399]: 13.9 scrub ok 2024-04-14T15:14:53.096 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:52 smithi090 ceph-mon[30399]: from='client.48511 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi090-93342-32"}]': finished 2024-04-14T15:14:53.096 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:52 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:14:53.096 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:52 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-21"}]: dispatch 2024-04-14T15:14:53.096 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:52 smithi090 ceph-mon[30399]: osdmap e225: 8 total, 8 up, 8 in 2024-04-14T15:14:53.096 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:52 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-21"}]: dispatch 2024-04-14T15:14:53.096 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:52 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1058124934' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi090-93363-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:53.096 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:52 smithi090 ceph-mon[30399]: from='client.48523 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi090-93363-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:53.096 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:52 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1251627847' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi090-93342-32"}]: dispatch 2024-04-14T15:14:53.097 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:52 smithi090 ceph-mon[30399]: from='client.48511 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi090-93342-32"}]: dispatch 2024-04-14T15:14:53.097 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:52 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:53.102 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:14:53.126 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:53.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:53.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:14:53.228 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:14:53.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.093+0000 7fd285580700 1 -- 172.21.15.90:0/4252644741 >> v1:172.21.15.90:6789/0 conn(0x7fd28010e270 legacy=0x7fd280110710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:53.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.093+0000 7fd285580700 1 -- 172.21.15.90:0/4252644741 shutdown_connections 2024-04-14T15:14:53.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.093+0000 7fd285580700 1 -- 172.21.15.90:0/4252644741 >> 172.21.15.90:0/4252644741 conn(0x7fd280076aa0 msgr2=0x7fd280076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:14:53.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.093+0000 7fd285580700 1 -- 172.21.15.90:0/4252644741 shutdown_connections 2024-04-14T15:14:53.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.093+0000 7fd285580700 1 -- 172.21.15.90:0/4252644741 wait complete. 2024-04-14T15:14:53.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.094+0000 7fd285580700 1 librados: starting msgr at 2024-04-14T15:14:53.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.094+0000 7fd285580700 1 librados: starting objecter 2024-04-14T15:14:53.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.094+0000 7fd285580700 1 Processor -- start 2024-04-14T15:14:53.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.094+0000 7fd285580700 1 -- start start 2024-04-14T15:14:53.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.094+0000 7fd285580700 1 librados: setting wanted keys 2024-04-14T15:14:53.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.094+0000 7fd285580700 1 librados: calling monclient init 2024-04-14T15:14:53.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.094+0000 7fd285580700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd2801a7e40 con 0x7fd28010a6d0 2024-04-14T15:14:53.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.094+0000 7fd285580700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd2801a9020 con 0x7fd2801023f0 2024-04-14T15:14:53.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.094+0000 7fd285580700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd2801aa200 con 0x7fd28010e270 2024-04-14T15:14:53.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.094+0000 7fd27e7fc700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fd28010a6d0 0x7fd280101810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:32890/0 (socket says 172.21.15.90:32890) 2024-04-14T15:14:53.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.094+0000 7fd27e7fc700 1 -- 172.21.15.90:0/3515548345 learned_addr learned my addr 172.21.15.90:0/3515548345 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:14:53.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.094+0000 7fd27f7fe700 1 --1- 172.21.15.90:0/3515548345 >> v1:172.21.15.90:6790/0 conn(0x7fd28010e270 0x7fd2801a6720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:52546/0 (socket says 172.21.15.90:52546) 2024-04-14T15:14:53.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.095+0000 7fd267fff700 10 client.?.objecter ms_handle_connect 0x7fd28010a6d0 2024-04-14T15:14:53.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.095+0000 7fd267fff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:14:53.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.095+0000 7fd267fff700 10 client.?.objecter ms_handle_connect 0x7fd28010e270 2024-04-14T15:14:53.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.095+0000 7fd267fff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:14:53.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.095+0000 7fd267fff700 10 client.?.objecter ms_handle_connect 0x7fd2801023f0 2024-04-14T15:14:53.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.095+0000 7fd267fff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:14:53.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.095+0000 7fd267fff700 1 -- 172.21.15.90:0/3515548345 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3809848393 0 0) 0x7fd2801a7e40 con 0x7fd28010a6d0 2024-04-14T15:14:53.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.095+0000 7fd267fff700 1 -- 172.21.15.90:0/3515548345 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd2680031f0 con 0x7fd28010a6d0 2024-04-14T15:14:53.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.095+0000 7fd267fff700 1 -- 172.21.15.90:0/3515548345 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3855702765 0 0) 0x7fd2801a9020 con 0x7fd2801023f0 2024-04-14T15:14:53.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.095+0000 7fd267fff700 1 -- 172.21.15.90:0/3515548345 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd2801a7e40 con 0x7fd2801023f0 2024-04-14T15:14:53.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.095+0000 7fd267fff700 1 -- 172.21.15.90:0/3515548345 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3983406523 0 0) 0x7fd2801aa200 con 0x7fd28010e270 2024-04-14T15:14:53.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.095+0000 7fd267fff700 1 -- 172.21.15.90:0/3515548345 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd2801a9020 con 0x7fd28010e270 2024-04-14T15:14:53.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.095+0000 7fd267fff700 1 -- 172.21.15.90:0/3515548345 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 575625336 0 0) 0x7fd2801a7e40 con 0x7fd2801023f0 2024-04-14T15:14:53.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.095+0000 7fd267fff700 1 -- 172.21.15.90:0/3515548345 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd2801aa200 con 0x7fd2801023f0 2024-04-14T15:14:53.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.095+0000 7fd267fff700 1 -- 172.21.15.90:0/3515548345 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fd2740018b0 con 0x7fd2801023f0 2024-04-14T15:14:53.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.095+0000 7fd267fff700 1 -- 172.21.15.90:0/3515548345 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2359995508 0 0) 0x7fd2680031f0 con 0x7fd28010a6d0 2024-04-14T15:14:53.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.095+0000 7fd267fff700 1 -- 172.21.15.90:0/3515548345 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd2801a7e40 con 0x7fd28010a6d0 2024-04-14T15:14:53.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.095+0000 7fd267fff700 1 -- 172.21.15.90:0/3515548345 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fd270001930 con 0x7fd28010a6d0 2024-04-14T15:14:53.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.095+0000 7fd267fff700 1 -- 172.21.15.90:0/3515548345 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1358198629 0 0) 0x7fd2801a9020 con 0x7fd28010e270 2024-04-14T15:14:53.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.095+0000 7fd267fff700 1 -- 172.21.15.90:0/3515548345 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd2680031f0 con 0x7fd28010e270 2024-04-14T15:14:53.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.095+0000 7fd267fff700 1 -- 172.21.15.90:0/3515548345 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fd2780017b0 con 0x7fd28010e270 2024-04-14T15:14:53.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.095+0000 7fd267fff700 1 -- 172.21.15.90:0/3515548345 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3570277466 0 0) 0x7fd2801aa200 con 0x7fd2801023f0 2024-04-14T15:14:53.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.095+0000 7fd267fff700 1 -- 172.21.15.90:0/3515548345 >> v1:172.21.15.90:6790/0 conn(0x7fd28010e270 legacy=0x7fd2801a6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:53.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.095+0000 7fd267fff700 1 -- 172.21.15.90:0/3515548345 >> v1:172.21.15.90:6789/0 conn(0x7fd28010a6d0 legacy=0x7fd280101810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:53.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.095+0000 7fd267fff700 1 -- 172.21.15.90:0/3515548345 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2801ab3e0 con 0x7fd2801023f0 2024-04-14T15:14:53.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.095+0000 7fd285580700 1 -- 172.21.15.90:0/3515548345 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd2801aa430 con 0x7fd2801023f0 2024-04-14T15:14:53.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.095+0000 7fd285580700 10 client.48529.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:14:53.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.095+0000 7fd267fff700 1 -- 172.21.15.90:0/3515548345 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fd274003b40 con 0x7fd2801023f0 2024-04-14T15:14:53.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.095+0000 7fd285580700 1 -- 172.21.15.90:0/3515548345 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd2801aa8f0 con 0x7fd2801023f0 2024-04-14T15:14:53.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.095+0000 7fd285580700 1 librados: init done 2024-04-14T15:14:53.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.095+0000 7fd267fff700 1 -- 172.21.15.90:0/3515548345 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fd274004a50 con 0x7fd2801023f0 2024-04-14T15:14:53.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.096+0000 7fd285580700 1 -- 172.21.15.90:0/3515548345 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fd28006a170 con 0x7fd2801023f0 2024-04-14T15:14:53.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.097+0000 7fd267fff700 1 -- 172.21.15.90:0/3515548345 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fd2740036a0 con 0x7fd2801023f0 2024-04-14T15:14:53.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.097+0000 7fd267fff700 1 -- 172.21.15.90:0/3515548345 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(225..225 src has 51..225) ==== 8873+0+0 (unknown 1850078076 0 0) 0x7fd274089a30 con 0x7fd2801023f0 2024-04-14T15:14:53.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.097+0000 7fd267fff700 10 client.48529.objecter ms_dispatch 0x7fd280001550 osd_map(225..225 src has 51..225) 2024-04-14T15:14:53.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.097+0000 7fd267fff700 3 client.48529.objecter handle_osd_map got epochs [225,225] > 0 2024-04-14T15:14:53.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.097+0000 7fd267fff700 3 client.48529.objecter handle_osd_map decoding full epoch 225 2024-04-14T15:14:53.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.097+0000 7fd267fff700 20 client.48529.objecter dump_active .. 0 homeless 2024-04-14T15:14:53.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.097+0000 7fd267fff700 1 -- 172.21.15.90:0/3515548345 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v225) ==== 54+0+16 (unknown 4286182321 0 2420000192) 0x7fd27408a060 con 0x7fd2801023f0 2024-04-14T15:14:53.238 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:14:53.238 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:14:53.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.097+0000 7fd267fff700 10 client.48529.objecter ms_handle_connect 0x7fd268071460 2024-04-14T15:14:53.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.098+0000 7fd285580700 10 client.48529.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:14:53.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.098+0000 7fd285580700 15 client.48529.objecter _session_command_op_assign -1 1 2024-04-14T15:14:53.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.098+0000 7fd285580700 20 client.48529.objecter _get_session s=0x7fd280106580 osd=0 3 2024-04-14T15:14:53.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.098+0000 7fd285580700 20 client.48529.objecter put_session s=0x7fd280106580 osd=0 3 2024-04-14T15:14:53.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.098+0000 7fd285580700 20 client.48529.objecter _get_session s=0x7fd280106580 osd=0 3 2024-04-14T15:14:53.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.098+0000 7fd285580700 15 client.48529.objecter _session_command_op_remove -1 1 2024-04-14T15:14:53.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.098+0000 7fd285580700 20 client.48529.objecter get_session s=0x7fd280106580 osd=0 3 2024-04-14T15:14:53.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.098+0000 7fd285580700 15 client.48529.objecter _session_command_op_assign 0 1 2024-04-14T15:14:53.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.098+0000 7fd285580700 20 client.48529.objecter put_session s=0x7fd280106580 osd=0 4 2024-04-14T15:14:53.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.098+0000 7fd285580700 10 client.48529.objecter _send_command 1 2024-04-14T15:14:53.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.098+0000 7fd285580700 1 -- 172.21.15.90:0/3515548345 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd2800693d0 con 0x7fd2801a91f0 2024-04-14T15:14:53.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.098+0000 7fd267fff700 10 client.48529.objecter ms_handle_connect 0x7fd2801a91f0 2024-04-14T15:14:53.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.099+0000 7fd267fff700 1 -- 172.21.15.90:0/3515548345 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd2800693d0 con 0x7fd2801a91f0 2024-04-14T15:14:53.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.099+0000 7fd267fff700 10 client.48529.objecter ms_dispatch 0x7fd280001550 command_reply(tid 1: 0 ) 2024-04-14T15:14:53.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.099+0000 7fd267fff700 10 client.48529.objecter _finish_command 1 = system:0 2024-04-14T15:14:53.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.099+0000 7fd267fff700 20 client.48529.objecter put_session s=0x7fd280106580 osd=0 4 2024-04-14T15:14:53.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.099+0000 7fd267fff700 15 client.48529.objecter _session_command_op_remove 0 1 2024-04-14T15:14:53.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.115+0000 7fd285580700 10 client.48529.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:14:53.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.115+0000 7fd285580700 15 client.48529.objecter _session_command_op_assign -1 2 2024-04-14T15:14:53.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.115+0000 7fd285580700 20 client.48529.objecter _get_session s=0x7fd280106580 osd=0 3 2024-04-14T15:14:53.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.115+0000 7fd285580700 20 client.48529.objecter put_session s=0x7fd280106580 osd=0 3 2024-04-14T15:14:53.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.115+0000 7fd285580700 20 client.48529.objecter _get_session s=0x7fd280106580 osd=0 3 2024-04-14T15:14:53.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.115+0000 7fd285580700 15 client.48529.objecter _session_command_op_remove -1 2 2024-04-14T15:14:53.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.115+0000 7fd285580700 20 client.48529.objecter get_session s=0x7fd280106580 osd=0 3 2024-04-14T15:14:53.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.115+0000 7fd285580700 15 client.48529.objecter _session_command_op_assign 0 2 2024-04-14T15:14:53.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.115+0000 7fd285580700 20 client.48529.objecter put_session s=0x7fd280106580 osd=0 4 2024-04-14T15:14:53.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.115+0000 7fd285580700 10 client.48529.objecter _send_command 2 2024-04-14T15:14:53.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.115+0000 7fd285580700 1 -- 172.21.15.90:0/3515548345 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd2801a9ce0 con 0x7fd2801a91f0 2024-04-14T15:14:53.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.117+0000 7fd267fff700 1 -- 172.21.15.90:0/3515548345 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd2801a9ce0 con 0x7fd2801a91f0 2024-04-14T15:14:53.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.117+0000 7fd267fff700 10 client.48529.objecter ms_dispatch 0x7fd280001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:14:53.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.117+0000 7fd267fff700 10 client.48529.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:14:53.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.117+0000 7fd267fff700 20 client.48529.objecter put_session s=0x7fd280106580 osd=0 4 2024-04-14T15:14:53.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.117+0000 7fd267fff700 15 client.48529.objecter _session_command_op_remove 0 2 2024-04-14T15:14:53.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.117+0000 7fd285580700 10 client.48529.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:14:53.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.117+0000 7fd285580700 15 client.48529.objecter _session_command_op_assign -1 3 2024-04-14T15:14:53.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.117+0000 7fd285580700 20 client.48529.objecter _get_session s=0x7fd280107550 osd=1 3 2024-04-14T15:14:53.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.117+0000 7fd285580700 20 client.48529.objecter put_session s=0x7fd280107550 osd=1 3 2024-04-14T15:14:53.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.117+0000 7fd285580700 20 client.48529.objecter _get_session s=0x7fd280107550 osd=1 3 2024-04-14T15:14:53.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.117+0000 7fd285580700 15 client.48529.objecter _session_command_op_remove -1 3 2024-04-14T15:14:53.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.117+0000 7fd285580700 20 client.48529.objecter get_session s=0x7fd280107550 osd=1 3 2024-04-14T15:14:53.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.117+0000 7fd285580700 15 client.48529.objecter _session_command_op_assign 1 3 2024-04-14T15:14:53.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.117+0000 7fd285580700 20 client.48529.objecter put_session s=0x7fd280107550 osd=1 4 2024-04-14T15:14:53.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.117+0000 7fd285580700 10 client.48529.objecter _send_command 3 2024-04-14T15:14:53.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.117+0000 7fd285580700 1 -- 172.21.15.90:0/3515548345 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fd280107ff0 con 0x7fd280107bd0 2024-04-14T15:14:53.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.118+0000 7fd267fff700 10 client.48529.objecter ms_handle_connect 0x7fd280107bd0 2024-04-14T15:14:53.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.119+0000 7fd267fff700 1 -- 172.21.15.90:0/3515548345 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd280107ff0 con 0x7fd280107bd0 2024-04-14T15:14:53.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.119+0000 7fd267fff700 10 client.48529.objecter ms_dispatch 0x7fd280001550 command_reply(tid 3: 0 ) 2024-04-14T15:14:53.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.119+0000 7fd267fff700 10 client.48529.objecter _finish_command 3 = system:0 2024-04-14T15:14:53.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.119+0000 7fd267fff700 20 client.48529.objecter put_session s=0x7fd280107550 osd=1 4 2024-04-14T15:14:53.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.119+0000 7fd267fff700 15 client.48529.objecter _session_command_op_remove 1 3 2024-04-14T15:14:53.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.138+0000 7fd285580700 10 client.48529.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:14:53.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.138+0000 7fd285580700 15 client.48529.objecter _session_command_op_assign -1 4 2024-04-14T15:14:53.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.138+0000 7fd285580700 20 client.48529.objecter _get_session s=0x7fd280107550 osd=1 3 2024-04-14T15:14:53.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.138+0000 7fd285580700 20 client.48529.objecter put_session s=0x7fd280107550 osd=1 3 2024-04-14T15:14:53.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.138+0000 7fd285580700 20 client.48529.objecter _get_session s=0x7fd280107550 osd=1 3 2024-04-14T15:14:53.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.138+0000 7fd285580700 15 client.48529.objecter _session_command_op_remove -1 4 2024-04-14T15:14:53.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.138+0000 7fd285580700 20 client.48529.objecter get_session s=0x7fd280107550 osd=1 3 2024-04-14T15:14:53.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.138+0000 7fd285580700 15 client.48529.objecter _session_command_op_assign 1 4 2024-04-14T15:14:53.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.138+0000 7fd285580700 20 client.48529.objecter put_session s=0x7fd280107550 osd=1 4 2024-04-14T15:14:53.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.138+0000 7fd285580700 10 client.48529.objecter _send_command 4 2024-04-14T15:14:53.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.138+0000 7fd285580700 1 -- 172.21.15.90:0/3515548345 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd2801083f0 con 0x7fd280107bd0 2024-04-14T15:14:53.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.139+0000 7fd267fff700 1 -- 172.21.15.90:0/3515548345 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd2801083f0 con 0x7fd280107bd0 2024-04-14T15:14:53.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.139+0000 7fd267fff700 10 client.48529.objecter ms_dispatch 0x7fd280001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:14:53.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.139+0000 7fd267fff700 10 client.48529.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:14:53.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.139+0000 7fd267fff700 20 client.48529.objecter put_session s=0x7fd280107550 osd=1 4 2024-04-14T15:14:53.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.139+0000 7fd267fff700 15 client.48529.objecter _session_command_op_remove 1 4 2024-04-14T15:14:53.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.139+0000 7fd285580700 10 client.48529.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:14:53.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.139+0000 7fd285580700 15 client.48529.objecter _session_command_op_assign -1 5 2024-04-14T15:14:53.244 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:14:53.244 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:14:53.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.139+0000 7fd285580700 20 client.48529.objecter _get_session s=0x7fd2801009d0 osd=2 3 2024-04-14T15:14:53.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.139+0000 7fd285580700 20 client.48529.objecter put_session s=0x7fd2801009d0 osd=2 3 2024-04-14T15:14:53.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.139+0000 7fd285580700 20 client.48529.objecter _get_session s=0x7fd2801009d0 osd=2 3 2024-04-14T15:14:53.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.139+0000 7fd285580700 15 client.48529.objecter _session_command_op_remove -1 5 2024-04-14T15:14:53.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.139+0000 7fd285580700 20 client.48529.objecter get_session s=0x7fd2801009d0 osd=2 3 2024-04-14T15:14:53.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.139+0000 7fd285580700 15 client.48529.objecter _session_command_op_assign 2 5 2024-04-14T15:14:53.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.139+0000 7fd285580700 20 client.48529.objecter put_session s=0x7fd2801009d0 osd=2 4 2024-04-14T15:14:53.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.139+0000 7fd285580700 10 client.48529.objecter _send_command 5 2024-04-14T15:14:53.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.139+0000 7fd285580700 1 -- 172.21.15.90:0/3515548345 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fd2801b1d50 con 0x7fd280100b40 2024-04-14T15:14:53.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.140+0000 7fd267fff700 10 client.48529.objecter ms_handle_connect 0x7fd280100b40 2024-04-14T15:14:53.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.140+0000 7fd267fff700 1 -- 172.21.15.90:0/3515548345 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd2801b1d50 con 0x7fd280100b40 2024-04-14T15:14:53.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.140+0000 7fd267fff700 10 client.48529.objecter ms_dispatch 0x7fd280001550 command_reply(tid 5: 0 ) 2024-04-14T15:14:53.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.140+0000 7fd267fff700 10 client.48529.objecter _finish_command 5 = system:0 2024-04-14T15:14:53.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.140+0000 7fd267fff700 20 client.48529.objecter put_session s=0x7fd2801009d0 osd=2 4 2024-04-14T15:14:53.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.140+0000 7fd267fff700 15 client.48529.objecter _session_command_op_remove 2 5 2024-04-14T15:14:53.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.156+0000 7fd285580700 10 client.48529.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:14:53.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.156+0000 7fd285580700 15 client.48529.objecter _session_command_op_assign -1 6 2024-04-14T15:14:53.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.156+0000 7fd285580700 20 client.48529.objecter _get_session s=0x7fd2801009d0 osd=2 3 2024-04-14T15:14:53.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.156+0000 7fd285580700 20 client.48529.objecter put_session s=0x7fd2801009d0 osd=2 3 2024-04-14T15:14:53.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.156+0000 7fd285580700 20 client.48529.objecter _get_session s=0x7fd2801009d0 osd=2 3 2024-04-14T15:14:53.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.157+0000 7fd285580700 15 client.48529.objecter _session_command_op_remove -1 6 2024-04-14T15:14:53.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.157+0000 7fd285580700 20 client.48529.objecter get_session s=0x7fd2801009d0 osd=2 3 2024-04-14T15:14:53.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.157+0000 7fd285580700 15 client.48529.objecter _session_command_op_assign 2 6 2024-04-14T15:14:53.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.157+0000 7fd285580700 20 client.48529.objecter put_session s=0x7fd2801009d0 osd=2 4 2024-04-14T15:14:53.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.157+0000 7fd285580700 10 client.48529.objecter _send_command 6 2024-04-14T15:14:53.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.157+0000 7fd285580700 1 -- 172.21.15.90:0/3515548345 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd280063e20 con 0x7fd280100b40 2024-04-14T15:14:53.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.158+0000 7fd267fff700 1 -- 172.21.15.90:0/3515548345 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd280063e20 con 0x7fd280100b40 2024-04-14T15:14:53.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.158+0000 7fd267fff700 10 client.48529.objecter ms_dispatch 0x7fd280001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:14:53.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.158+0000 7fd267fff700 10 client.48529.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:14:53.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.158+0000 7fd267fff700 20 client.48529.objecter put_session s=0x7fd2801009d0 osd=2 4 2024-04-14T15:14:53.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.158+0000 7fd267fff700 15 client.48529.objecter _session_command_op_remove 2 6 2024-04-14T15:14:53.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.158+0000 7fd285580700 10 client.48529.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:14:53.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.158+0000 7fd285580700 15 client.48529.objecter _session_command_op_assign -1 7 2024-04-14T15:14:53.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.158+0000 7fd285580700 20 client.48529.objecter _get_session s=0x7fd280063140 osd=3 3 2024-04-14T15:14:53.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.158+0000 7fd285580700 20 client.48529.objecter put_session s=0x7fd280063140 osd=3 3 2024-04-14T15:14:53.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.158+0000 7fd285580700 20 client.48529.objecter _get_session s=0x7fd280063140 osd=3 3 2024-04-14T15:14:53.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.158+0000 7fd285580700 15 client.48529.objecter _session_command_op_remove -1 7 2024-04-14T15:14:53.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.158+0000 7fd285580700 20 client.48529.objecter get_session s=0x7fd280063140 osd=3 3 2024-04-14T15:14:53.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.158+0000 7fd285580700 15 client.48529.objecter _session_command_op_assign 3 7 2024-04-14T15:14:53.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.158+0000 7fd285580700 20 client.48529.objecter put_session s=0x7fd280063140 osd=3 4 2024-04-14T15:14:53.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.158+0000 7fd285580700 10 client.48529.objecter _send_command 7 2024-04-14T15:14:53.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.158+0000 7fd285580700 1 -- 172.21.15.90:0/3515548345 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fd280104bb0 con 0x7fd2800632b0 2024-04-14T15:14:53.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.159+0000 7fd267fff700 10 client.48529.objecter ms_handle_connect 0x7fd2800632b0 2024-04-14T15:14:53.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.159+0000 7fd267fff700 1 -- 172.21.15.90:0/3515548345 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd280104bb0 con 0x7fd2800632b0 2024-04-14T15:14:53.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.159+0000 7fd267fff700 10 client.48529.objecter ms_dispatch 0x7fd280001550 command_reply(tid 7: 0 ) 2024-04-14T15:14:53.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.159+0000 7fd267fff700 10 client.48529.objecter _finish_command 7 = system:0 2024-04-14T15:14:53.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.159+0000 7fd267fff700 20 client.48529.objecter put_session s=0x7fd280063140 osd=3 4 2024-04-14T15:14:53.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.159+0000 7fd267fff700 15 client.48529.objecter _session_command_op_remove 3 7 2024-04-14T15:14:53.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.176+0000 7fd285580700 10 client.48529.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:14:53.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.176+0000 7fd285580700 15 client.48529.objecter _session_command_op_assign -1 8 2024-04-14T15:14:53.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.176+0000 7fd285580700 20 client.48529.objecter _get_session s=0x7fd280063140 osd=3 3 2024-04-14T15:14:53.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.176+0000 7fd285580700 20 client.48529.objecter put_session s=0x7fd280063140 osd=3 3 2024-04-14T15:14:53.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.176+0000 7fd285580700 20 client.48529.objecter _get_session s=0x7fd280063140 osd=3 3 2024-04-14T15:14:53.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.176+0000 7fd285580700 15 client.48529.objecter _session_command_op_remove -1 8 2024-04-14T15:14:53.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.176+0000 7fd285580700 20 client.48529.objecter get_session s=0x7fd280063140 osd=3 3 2024-04-14T15:14:53.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.176+0000 7fd285580700 15 client.48529.objecter _session_command_op_assign 3 8 2024-04-14T15:14:53.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.176+0000 7fd285580700 20 client.48529.objecter put_session s=0x7fd280063140 osd=3 4 2024-04-14T15:14:53.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.176+0000 7fd285580700 10 client.48529.objecter _send_command 8 2024-04-14T15:14:53.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.176+0000 7fd285580700 1 -- 172.21.15.90:0/3515548345 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd2801bac10 con 0x7fd2800632b0 2024-04-14T15:14:53.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.177+0000 7fd267fff700 1 -- 172.21.15.90:0/3515548345 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd2801bac10 con 0x7fd2800632b0 2024-04-14T15:14:53.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.177+0000 7fd267fff700 10 client.48529.objecter ms_dispatch 0x7fd280001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:14:53.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.177+0000 7fd267fff700 10 client.48529.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:14:53.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.177+0000 7fd267fff700 20 client.48529.objecter put_session s=0x7fd280063140 osd=3 4 2024-04-14T15:14:53.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.177+0000 7fd267fff700 15 client.48529.objecter _session_command_op_remove 3 8 2024-04-14T15:14:53.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.177+0000 7fd285580700 10 client.48529.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:14:53.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.177+0000 7fd285580700 15 client.48529.objecter _session_command_op_assign -1 9 2024-04-14T15:14:53.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.177+0000 7fd285580700 20 client.48529.objecter _get_session s=0x7fd2801bac10 osd=4 3 2024-04-14T15:14:53.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.177+0000 7fd285580700 20 client.48529.objecter put_session s=0x7fd2801bac10 osd=4 3 2024-04-14T15:14:53.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.177+0000 7fd285580700 20 client.48529.objecter _get_session s=0x7fd2801bac10 osd=4 3 2024-04-14T15:14:53.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.177+0000 7fd285580700 15 client.48529.objecter _session_command_op_remove -1 9 2024-04-14T15:14:53.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.177+0000 7fd285580700 20 client.48529.objecter get_session s=0x7fd2801bac10 osd=4 3 2024-04-14T15:14:53.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.177+0000 7fd285580700 15 client.48529.objecter _session_command_op_assign 4 9 2024-04-14T15:14:53.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.177+0000 7fd285580700 20 client.48529.objecter put_session s=0x7fd2801bac10 osd=4 4 2024-04-14T15:14:53.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.177+0000 7fd285580700 10 client.48529.objecter _send_command 9 2024-04-14T15:14:53.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.177+0000 7fd285580700 1 -- 172.21.15.90:0/3515548345 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fd2801bee40 con 0x7fd2801bb290 2024-04-14T15:14:53.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.178+0000 7fd267fff700 10 client.48529.objecter ms_handle_connect 0x7fd2801bb290 2024-04-14T15:14:53.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.179+0000 7fd267fff700 1 -- 172.21.15.90:0/3515548345 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd2801bee40 con 0x7fd2801bb290 2024-04-14T15:14:53.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.179+0000 7fd267fff700 10 client.48529.objecter ms_dispatch 0x7fd280001550 command_reply(tid 9: 0 ) 2024-04-14T15:14:53.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.179+0000 7fd267fff700 10 client.48529.objecter _finish_command 9 = system:0 2024-04-14T15:14:53.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.179+0000 7fd267fff700 20 client.48529.objecter put_session s=0x7fd2801bac10 osd=4 4 2024-04-14T15:14:53.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.179+0000 7fd267fff700 15 client.48529.objecter _session_command_op_remove 4 9 2024-04-14T15:14:53.251 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:14:53.251 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:14:53.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.195+0000 7fd285580700 10 client.48529.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:14:53.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.195+0000 7fd285580700 15 client.48529.objecter _session_command_op_assign -1 10 2024-04-14T15:14:53.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.195+0000 7fd285580700 20 client.48529.objecter _get_session s=0x7fd2801bac10 osd=4 3 2024-04-14T15:14:53.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.195+0000 7fd285580700 20 client.48529.objecter put_session s=0x7fd2801bac10 osd=4 3 2024-04-14T15:14:53.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.195+0000 7fd285580700 20 client.48529.objecter _get_session s=0x7fd2801bac10 osd=4 3 2024-04-14T15:14:53.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.195+0000 7fd285580700 15 client.48529.objecter _session_command_op_remove -1 10 2024-04-14T15:14:53.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.195+0000 7fd285580700 20 client.48529.objecter get_session s=0x7fd2801bac10 osd=4 3 2024-04-14T15:14:53.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.195+0000 7fd285580700 15 client.48529.objecter _session_command_op_assign 4 10 2024-04-14T15:14:53.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.195+0000 7fd285580700 20 client.48529.objecter put_session s=0x7fd2801bac10 osd=4 4 2024-04-14T15:14:53.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.195+0000 7fd285580700 10 client.48529.objecter _send_command 10 2024-04-14T15:14:53.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.195+0000 7fd285580700 1 -- 172.21.15.90:0/3515548345 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd280104dc0 con 0x7fd2801bb290 2024-04-14T15:14:53.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.198+0000 7fd267fff700 1 -- 172.21.15.90:0/3515548345 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd280104dc0 con 0x7fd2801bb290 2024-04-14T15:14:53.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.198+0000 7fd267fff700 10 client.48529.objecter ms_dispatch 0x7fd280001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:14:53.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.198+0000 7fd267fff700 10 client.48529.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:14:53.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.198+0000 7fd267fff700 20 client.48529.objecter put_session s=0x7fd2801bac10 osd=4 4 2024-04-14T15:14:53.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.198+0000 7fd267fff700 15 client.48529.objecter _session_command_op_remove 4 10 2024-04-14T15:14:53.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.198+0000 7fd285580700 10 client.48529.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:14:53.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.198+0000 7fd285580700 15 client.48529.objecter _session_command_op_assign -1 11 2024-04-14T15:14:53.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.198+0000 7fd285580700 20 client.48529.objecter _get_session s=0x7fd280104dc0 osd=5 3 2024-04-14T15:14:53.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.198+0000 7fd285580700 20 client.48529.objecter put_session s=0x7fd280104dc0 osd=5 3 2024-04-14T15:14:53.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.198+0000 7fd285580700 20 client.48529.objecter _get_session s=0x7fd280104dc0 osd=5 3 2024-04-14T15:14:53.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.198+0000 7fd285580700 15 client.48529.objecter _session_command_op_remove -1 11 2024-04-14T15:14:53.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.198+0000 7fd285580700 20 client.48529.objecter get_session s=0x7fd280104dc0 osd=5 3 2024-04-14T15:14:53.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.198+0000 7fd285580700 15 client.48529.objecter _session_command_op_assign 5 11 2024-04-14T15:14:53.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.198+0000 7fd285580700 20 client.48529.objecter put_session s=0x7fd280104dc0 osd=5 4 2024-04-14T15:14:53.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.198+0000 7fd285580700 10 client.48529.objecter _send_command 11 2024-04-14T15:14:53.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.198+0000 7fd285580700 1 -- 172.21.15.90:0/3515548345 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fd2801c0600 con 0x7fd2801b6b30 2024-04-14T15:14:53.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.198+0000 7fd267fff700 10 client.48529.objecter ms_handle_connect 0x7fd2801b6b30 2024-04-14T15:14:53.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.199+0000 7fd267fff700 1 -- 172.21.15.90:0/3515548345 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd27000c900 con 0x7fd2801b6b30 2024-04-14T15:14:53.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.199+0000 7fd267fff700 10 client.48529.objecter ms_dispatch 0x7fd280001550 command_reply(tid 11: 0 ) 2024-04-14T15:14:53.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.199+0000 7fd267fff700 10 client.48529.objecter _finish_command 11 = system:0 2024-04-14T15:14:53.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.199+0000 7fd267fff700 20 client.48529.objecter put_session s=0x7fd280104dc0 osd=5 4 2024-04-14T15:14:53.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.199+0000 7fd267fff700 15 client.48529.objecter _session_command_op_remove 5 11 2024-04-14T15:14:53.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.215+0000 7fd285580700 10 client.48529.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:14:53.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.215+0000 7fd285580700 15 client.48529.objecter _session_command_op_assign -1 12 2024-04-14T15:14:53.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.215+0000 7fd285580700 20 client.48529.objecter _get_session s=0x7fd280104dc0 osd=5 3 2024-04-14T15:14:53.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.215+0000 7fd285580700 20 client.48529.objecter put_session s=0x7fd280104dc0 osd=5 3 2024-04-14T15:14:53.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.215+0000 7fd285580700 20 client.48529.objecter _get_session s=0x7fd280104dc0 osd=5 3 2024-04-14T15:14:53.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.215+0000 7fd285580700 15 client.48529.objecter _session_command_op_remove -1 12 2024-04-14T15:14:53.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.215+0000 7fd285580700 20 client.48529.objecter get_session s=0x7fd280104dc0 osd=5 3 2024-04-14T15:14:53.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.215+0000 7fd285580700 15 client.48529.objecter _session_command_op_assign 5 12 2024-04-14T15:14:53.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.215+0000 7fd285580700 20 client.48529.objecter put_session s=0x7fd280104dc0 osd=5 4 2024-04-14T15:14:53.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.215+0000 7fd285580700 10 client.48529.objecter _send_command 12 2024-04-14T15:14:53.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.215+0000 7fd285580700 1 -- 172.21.15.90:0/3515548345 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd2801b7130 con 0x7fd2801b6b30 2024-04-14T15:14:53.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.217+0000 7fd267fff700 1 -- 172.21.15.90:0/3515548345 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd2801b7130 con 0x7fd2801b6b30 2024-04-14T15:14:53.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.217+0000 7fd267fff700 10 client.48529.objecter ms_dispatch 0x7fd280001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:14:53.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.217+0000 7fd267fff700 10 client.48529.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:14:53.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.217+0000 7fd267fff700 20 client.48529.objecter put_session s=0x7fd280104dc0 osd=5 4 2024-04-14T15:14:53.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.217+0000 7fd267fff700 15 client.48529.objecter _session_command_op_remove 5 12 2024-04-14T15:14:53.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.217+0000 7fd285580700 10 client.48529.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:14:53.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.217+0000 7fd285580700 15 client.48529.objecter _session_command_op_assign -1 13 2024-04-14T15:14:53.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.217+0000 7fd285580700 20 client.48529.objecter _get_session s=0x7fd2801b7130 osd=6 3 2024-04-14T15:14:53.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.217+0000 7fd285580700 20 client.48529.objecter put_session s=0x7fd2801b7130 osd=6 3 2024-04-14T15:14:53.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.217+0000 7fd285580700 20 client.48529.objecter _get_session s=0x7fd2801b7130 osd=6 3 2024-04-14T15:14:53.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.217+0000 7fd285580700 15 client.48529.objecter _session_command_op_remove -1 13 2024-04-14T15:14:53.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.217+0000 7fd285580700 20 client.48529.objecter get_session s=0x7fd2801b7130 osd=6 3 2024-04-14T15:14:53.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.217+0000 7fd285580700 15 client.48529.objecter _session_command_op_assign 6 13 2024-04-14T15:14:53.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.217+0000 7fd285580700 20 client.48529.objecter put_session s=0x7fd2801b7130 osd=6 4 2024-04-14T15:14:53.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.217+0000 7fd285580700 10 client.48529.objecter _send_command 13 2024-04-14T15:14:53.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.217+0000 7fd285580700 1 -- 172.21.15.90:0/3515548345 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fd2801c7a50 con 0x7fd2801c3f00 2024-04-14T15:14:53.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.218+0000 7fd267fff700 10 client.48529.objecter ms_handle_connect 0x7fd2801c3f00 2024-04-14T15:14:53.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.218+0000 7fd267fff700 1 -- 172.21.15.90:0/3515548345 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd2801c7a50 con 0x7fd2801c3f00 2024-04-14T15:14:53.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.218+0000 7fd267fff700 10 client.48529.objecter ms_dispatch 0x7fd280001550 command_reply(tid 13: 0 ) 2024-04-14T15:14:53.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.218+0000 7fd267fff700 10 client.48529.objecter _finish_command 13 = system:0 2024-04-14T15:14:53.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.219+0000 7fd267fff700 20 client.48529.objecter put_session s=0x7fd2801b7130 osd=6 4 2024-04-14T15:14:53.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.219+0000 7fd267fff700 15 client.48529.objecter _session_command_op_remove 6 13 2024-04-14T15:14:53.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.237+0000 7fd285580700 10 client.48529.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:14:53.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.237+0000 7fd285580700 15 client.48529.objecter _session_command_op_assign -1 14 2024-04-14T15:14:53.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.237+0000 7fd285580700 20 client.48529.objecter _get_session s=0x7fd2801b7130 osd=6 3 2024-04-14T15:14:53.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.237+0000 7fd285580700 20 client.48529.objecter put_session s=0x7fd2801b7130 osd=6 3 2024-04-14T15:14:53.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.237+0000 7fd285580700 20 client.48529.objecter _get_session s=0x7fd2801b7130 osd=6 3 2024-04-14T15:14:53.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.237+0000 7fd285580700 15 client.48529.objecter _session_command_op_remove -1 14 2024-04-14T15:14:53.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.237+0000 7fd285580700 20 client.48529.objecter get_session s=0x7fd2801b7130 osd=6 3 2024-04-14T15:14:53.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.237+0000 7fd285580700 15 client.48529.objecter _session_command_op_assign 6 14 2024-04-14T15:14:53.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.237+0000 7fd285580700 20 client.48529.objecter put_session s=0x7fd2801b7130 osd=6 4 2024-04-14T15:14:53.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.237+0000 7fd285580700 10 client.48529.objecter _send_command 14 2024-04-14T15:14:53.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.237+0000 7fd285580700 1 -- 172.21.15.90:0/3515548345 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd2801b65b0 con 0x7fd2801c3f00 2024-04-14T15:14:53.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.239+0000 7fd267fff700 1 -- 172.21.15.90:0/3515548345 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd2801b65b0 con 0x7fd2801c3f00 2024-04-14T15:14:53.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.239+0000 7fd267fff700 10 client.48529.objecter ms_dispatch 0x7fd280001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:14:53.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.239+0000 7fd267fff700 10 client.48529.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:14:53.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.239+0000 7fd267fff700 20 client.48529.objecter put_session s=0x7fd2801b7130 osd=6 4 2024-04-14T15:14:53.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.239+0000 7fd267fff700 15 client.48529.objecter _session_command_op_remove 6 14 2024-04-14T15:14:53.258 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:14:53.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.239+0000 7fd285580700 10 client.48529.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:14:53.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.239+0000 7fd285580700 15 client.48529.objecter _session_command_op_assign -1 15 2024-04-14T15:14:53.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.239+0000 7fd285580700 20 client.48529.objecter _get_session s=0x7fd280103f90 osd=7 3 2024-04-14T15:14:53.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.239+0000 7fd285580700 20 client.48529.objecter put_session s=0x7fd280103f90 osd=7 3 2024-04-14T15:14:53.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.239+0000 7fd285580700 20 client.48529.objecter _get_session s=0x7fd280103f90 osd=7 3 2024-04-14T15:14:53.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.239+0000 7fd285580700 15 client.48529.objecter _session_command_op_remove -1 15 2024-04-14T15:14:53.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.239+0000 7fd285580700 20 client.48529.objecter get_session s=0x7fd280103f90 osd=7 3 2024-04-14T15:14:53.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.239+0000 7fd285580700 15 client.48529.objecter _session_command_op_assign 7 15 2024-04-14T15:14:53.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.239+0000 7fd285580700 20 client.48529.objecter put_session s=0x7fd280103f90 osd=7 4 2024-04-14T15:14:53.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.239+0000 7fd285580700 10 client.48529.objecter _send_command 15 2024-04-14T15:14:53.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.239+0000 7fd285580700 1 -- 172.21.15.90:0/3515548345 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fd2801c1790 con 0x7fd2801b6360 2024-04-14T15:14:53.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.240+0000 7fd267fff700 10 client.48529.objecter ms_handle_connect 0x7fd2801b6360 2024-04-14T15:14:53.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.241+0000 7fd267fff700 1 -- 172.21.15.90:0/3515548345 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd2801c1790 con 0x7fd2801b6360 2024-04-14T15:14:53.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.241+0000 7fd267fff700 10 client.48529.objecter ms_dispatch 0x7fd280001550 command_reply(tid 15: 0 ) 2024-04-14T15:14:53.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.241+0000 7fd267fff700 10 client.48529.objecter _finish_command 15 = system:0 2024-04-14T15:14:53.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.241+0000 7fd267fff700 20 client.48529.objecter put_session s=0x7fd280103f90 osd=7 4 2024-04-14T15:14:53.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.241+0000 7fd267fff700 15 client.48529.objecter _session_command_op_remove 7 15 2024-04-14T15:14:53.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.259+0000 7fd285580700 10 client.48529.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:14:53.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.259+0000 7fd285580700 15 client.48529.objecter _session_command_op_assign -1 16 2024-04-14T15:14:53.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.259+0000 7fd285580700 20 client.48529.objecter _get_session s=0x7fd280103f90 osd=7 3 2024-04-14T15:14:53.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.259+0000 7fd285580700 20 client.48529.objecter put_session s=0x7fd280103f90 osd=7 3 2024-04-14T15:14:53.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.259+0000 7fd285580700 20 client.48529.objecter _get_session s=0x7fd280103f90 osd=7 3 2024-04-14T15:14:53.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.259+0000 7fd285580700 15 client.48529.objecter _session_command_op_remove -1 16 2024-04-14T15:14:53.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.259+0000 7fd285580700 20 client.48529.objecter get_session s=0x7fd280103f90 osd=7 3 2024-04-14T15:14:53.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.259+0000 7fd285580700 15 client.48529.objecter _session_command_op_assign 7 16 2024-04-14T15:14:53.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.259+0000 7fd285580700 20 client.48529.objecter put_session s=0x7fd280103f90 osd=7 4 2024-04-14T15:14:53.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.259+0000 7fd285580700 10 client.48529.objecter _send_command 16 2024-04-14T15:14:53.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.259+0000 7fd285580700 1 -- 172.21.15.90:0/3515548345 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd2801c1790 con 0x7fd2801b6360 2024-04-14T15:14:53.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:53.262 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:53.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:14:53.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.261+0000 7fd267fff700 1 -- 172.21.15.90:0/3515548345 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd2801c1790 con 0x7fd2801b6360 2024-04-14T15:14:53.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.261+0000 7fd267fff700 10 client.48529.objecter ms_dispatch 0x7fd280001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:14:53.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.261+0000 7fd267fff700 10 client.48529.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:14:53.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.261+0000 7fd267fff700 20 client.48529.objecter put_session s=0x7fd280103f90 osd=7 4 2024-04-14T15:14:53.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.261+0000 7fd267fff700 15 client.48529.objecter _session_command_op_remove 7 16 2024-04-14T15:14:53.263 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:14:53.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.261+0000 7fd285580700 10 librados: watch_flush enter 2024-04-14T15:14:53.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.261+0000 7fd285580700 10 librados: watch_flush exit 2024-04-14T15:14:53.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.261+0000 7fd285580700 10 client.48529.objecter close_session for osd.0 2024-04-14T15:14:53.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.261+0000 7fd285580700 1 -- 172.21.15.90:0/3515548345 >> v1:172.21.15.90:6801/3761806885 conn(0x7fd2801a91f0 legacy=0x7fd2801a95d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:53.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.261+0000 7fd285580700 20 client.48529.objecter put_session s=0x7fd280106580 osd=0 1 2024-04-14T15:14:53.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.261+0000 7fd285580700 10 client.48529.objecter close_session for osd.1 2024-04-14T15:14:53.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.261+0000 7fd285580700 1 -- 172.21.15.90:0/3515548345 >> v1:172.21.15.90:6805/4170361699 conn(0x7fd280107bd0 legacy=0x7fd28002ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:53.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.262+0000 7fd285580700 20 client.48529.objecter put_session s=0x7fd280107550 osd=1 1 2024-04-14T15:14:53.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.262+0000 7fd285580700 10 client.48529.objecter close_session for osd.2 2024-04-14T15:14:53.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.262+0000 7fd285580700 1 -- 172.21.15.90:0/3515548345 >> v1:172.21.15.90:6809/1686184251 conn(0x7fd280100b40 legacy=0x7fd2801b1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:53.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.262+0000 7fd285580700 20 client.48529.objecter put_session s=0x7fd2801009d0 osd=2 1 2024-04-14T15:14:53.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.262+0000 7fd285580700 10 client.48529.objecter close_session for osd.3 2024-04-14T15:14:53.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.262+0000 7fd285580700 1 -- 172.21.15.90:0/3515548345 >> v1:172.21.15.90:6813/2270166774 conn(0x7fd2800632b0 legacy=0x7fd2801b94f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:53.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.262+0000 7fd285580700 20 client.48529.objecter put_session s=0x7fd280063140 osd=3 1 2024-04-14T15:14:53.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.262+0000 7fd285580700 10 client.48529.objecter close_session for osd.4 2024-04-14T15:14:53.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.262+0000 7fd285580700 1 -- 172.21.15.90:0/3515548345 >> v1:172.21.15.92:6800/3887549733 conn(0x7fd2801bb290 legacy=0x7fd2801bd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:53.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.262+0000 7fd285580700 20 client.48529.objecter put_session s=0x7fd2801bac10 osd=4 1 2024-04-14T15:14:53.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.262+0000 7fd285580700 10 client.48529.objecter close_session for osd.5 2024-04-14T15:14:53.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.262+0000 7fd285580700 1 -- 172.21.15.90:0/3515548345 >> v1:172.21.15.92:6804/3244561440 conn(0x7fd2801b6b30 legacy=0x7fd2801bee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:53.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.262+0000 7fd285580700 20 client.48529.objecter put_session s=0x7fd280104dc0 osd=5 1 2024-04-14T15:14:53.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.262+0000 7fd285580700 10 client.48529.objecter close_session for osd.6 2024-04-14T15:14:53.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.262+0000 7fd285580700 1 -- 172.21.15.90:0/3515548345 >> v1:172.21.15.92:6808/636125963 conn(0x7fd2801c3f00 legacy=0x7fd2801c62f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:53.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.262+0000 7fd285580700 20 client.48529.objecter put_session s=0x7fd2801b7130 osd=6 1 2024-04-14T15:14:53.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.262+0000 7fd285580700 10 client.48529.objecter close_session for osd.7 2024-04-14T15:14:53.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.262+0000 7fd285580700 1 -- 172.21.15.90:0/3515548345 >> v1:172.21.15.92:6812/1773706716 conn(0x7fd2801b6360 legacy=0x7fd2801c0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:53.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.262+0000 7fd285580700 20 client.48529.objecter put_session s=0x7fd280103f90 osd=7 1 2024-04-14T15:14:53.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.262+0000 7fd285580700 20 client.48529.objecter shutdown clearing up homeless session... 2024-04-14T15:14:53.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.262+0000 7fd285580700 10 client.48529.objecter successfully canceled tick 2024-04-14T15:14:53.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.262+0000 7fd27f7fe700 1 -- 172.21.15.90:0/3515548345 reap_dead start 2024-04-14T15:14:53.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.262+0000 7fd27f7fe700 1 -- 172.21.15.90:0/3515548345 reap_dead start 2024-04-14T15:14:53.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.262+0000 7fd27f7fe700 1 -- 172.21.15.90:0/3515548345 reap_dead start 2024-04-14T15:14:53.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.262+0000 7fd27f7fe700 1 -- 172.21.15.90:0/3515548345 reap_dead start 2024-04-14T15:14:53.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.262+0000 7fd27f7fe700 1 -- 172.21.15.90:0/3515548345 reap_dead start 2024-04-14T15:14:53.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.262+0000 7fd285580700 1 -- 172.21.15.90:0/3515548345 >> v1:172.21.15.90:6800/420935427 conn(0x7fd268071460 legacy=0x7fd268073920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:53.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.262+0000 7fd285580700 1 -- 172.21.15.90:0/3515548345 >> v1:172.21.15.92:6789/0 conn(0x7fd2801023f0 legacy=0x7fd280101100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:14:53.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.262+0000 7fd285580700 1 -- 172.21.15.90:0/3515548345 shutdown_connections 2024-04-14T15:14:53.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.262+0000 7fd285580700 1 -- 172.21.15.90:0/3515548345 >> 172.21.15.90:0/3515548345 conn(0x7fd280076aa0 msgr2=0x7fd280100070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:14:53.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.262+0000 7fd285580700 1 -- 172.21.15.90:0/3515548345 shutdown_connections 2024-04-14T15:14:53.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.262+0000 7fd285580700 1 -- 172.21.15.90:0/3515548345 wait complete. 2024-04-14T15:14:53.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:14:53.262+0000 7fd285580700 1 librados: shutdown 2024-04-14T15:14:53.268 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:14:53.268 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:14:53.268 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:14:53.268 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:14:53.268 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:14:53.268 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:14:53.268 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:14:53.268 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:14:53.363 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:14:53.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:53.387 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:53.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:14:53.488 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:14:53.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:53.512 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:53.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:14:53.613 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:14:53.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:53.638 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:53.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:14:53.740 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:14:53.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:53.764 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:53.764 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:14:53.865 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:14:53.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:53.893 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:53.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:14:53.994 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:14:54.018 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:54.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:54.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:14:54.120 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:14:54.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:54.145 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:54.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:14:54.246 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:54.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:54.270 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:54.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:54.301 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:14:53 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:14:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:14:54.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:53 smithi090 ceph-mon[41670]: 11.f scrub starts 2024-04-14T15:14:54.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:53 smithi090 ceph-mon[41670]: 11.f scrub ok 2024-04-14T15:14:54.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:53 smithi090 ceph-mon[41670]: pgmap v284: 286 pgs: 4 active+clean+snaptrim, 32 unknown, 4 active+clean+snaptrim_wait, 246 active+clean; 586 KiB data, 551 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:14:54.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:53 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-04-14T15:14:54.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:53 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3515548345' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:14:54.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:53 smithi090 ceph-mon[41670]: 6.10 scrub starts 2024-04-14T15:14:54.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:53 smithi090 ceph-mon[41670]: 6.10 scrub ok 2024-04-14T15:14:54.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:53 smithi090 ceph-mon[41670]: 110.12 scrub starts 2024-04-14T15:14:54.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:53 smithi090 ceph-mon[41670]: 110.12 scrub ok 2024-04-14T15:14:54.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:53 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:14:54.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:53 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:14:54.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:53 smithi090 ceph-mon[41670]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:14:54.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:53 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-21"}]': finished 2024-04-14T15:14:54.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:53 smithi090 ceph-mon[41670]: from='client.48523 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi090-93363-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:54.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:53 smithi090 ceph-mon[41670]: from='client.48511 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi090-93342-32"}]': finished 2024-04-14T15:14:54.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:53 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "15"}]': finished 2024-04-14T15:14:54.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:53 smithi090 ceph-mon[41670]: osdmap e226: 8 total, 8 up, 8 in 2024-04-14T15:14:54.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:53 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1194319830' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi090-93342-33"}]: dispatch 2024-04-14T15:14:54.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:53 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1194319830' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi090-93342-33"}]: dispatch 2024-04-14T15:14:54.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:53 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1194319830' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi090-93342-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:14:54.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:53 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:54.304 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:53 smithi090 ceph-mon[30399]: 11.f scrub starts 2024-04-14T15:14:54.304 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:53 smithi090 ceph-mon[30399]: 11.f scrub ok 2024-04-14T15:14:54.304 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:53 smithi090 ceph-mon[30399]: pgmap v284: 286 pgs: 4 active+clean+snaptrim, 32 unknown, 4 active+clean+snaptrim_wait, 246 active+clean; 586 KiB data, 551 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:14:54.304 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:53 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-04-14T15:14:54.304 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:53 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3515548345' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:14:54.304 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:53 smithi090 ceph-mon[30399]: 6.10 scrub starts 2024-04-14T15:14:54.304 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:53 smithi090 ceph-mon[30399]: 6.10 scrub ok 2024-04-14T15:14:54.304 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:53 smithi090 ceph-mon[30399]: 110.12 scrub starts 2024-04-14T15:14:54.304 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:53 smithi090 ceph-mon[30399]: 110.12 scrub ok 2024-04-14T15:14:54.304 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:53 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:14:54.304 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:53 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:14:54.305 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:53 smithi090 ceph-mon[30399]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:14:54.305 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:53 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-21"}]': finished 2024-04-14T15:14:54.305 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:53 smithi090 ceph-mon[30399]: from='client.48523 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi090-93363-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:54.305 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:53 smithi090 ceph-mon[30399]: from='client.48511 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi090-93342-32"}]': finished 2024-04-14T15:14:54.305 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:53 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "15"}]': finished 2024-04-14T15:14:54.305 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:53 smithi090 ceph-mon[30399]: osdmap e226: 8 total, 8 up, 8 in 2024-04-14T15:14:54.305 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:53 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1194319830' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi090-93342-33"}]: dispatch 2024-04-14T15:14:54.305 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:53 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1194319830' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi090-93342-33"}]: dispatch 2024-04-14T15:14:54.305 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:53 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1194319830' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi090-93342-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:14:54.305 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:53 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:54.372 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:14:54.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:53 smithi092 ceph-mon[33966]: 11.f scrub starts 2024-04-14T15:14:54.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:53 smithi092 ceph-mon[33966]: 11.f scrub ok 2024-04-14T15:14:54.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:53 smithi092 ceph-mon[33966]: pgmap v284: 286 pgs: 4 active+clean+snaptrim, 32 unknown, 4 active+clean+snaptrim_wait, 246 active+clean; 586 KiB data, 551 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:14:54.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:53 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-04-14T15:14:54.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:53 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3515548345' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:14:54.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:53 smithi092 ceph-mon[33966]: 6.10 scrub starts 2024-04-14T15:14:54.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:53 smithi092 ceph-mon[33966]: 6.10 scrub ok 2024-04-14T15:14:54.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:53 smithi092 ceph-mon[33966]: 110.12 scrub starts 2024-04-14T15:14:54.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:53 smithi092 ceph-mon[33966]: 110.12 scrub ok 2024-04-14T15:14:54.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:53 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:14:54.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:53 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:14:54.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:53 smithi092 ceph-mon[33966]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:14:54.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:53 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-21"}]': finished 2024-04-14T15:14:54.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:53 smithi092 ceph-mon[33966]: from='client.48523 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi090-93363-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:54.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:53 smithi092 ceph-mon[33966]: from='client.48511 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi090-93342-32"}]': finished 2024-04-14T15:14:54.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:53 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "15"}]': finished 2024-04-14T15:14:54.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:53 smithi092 ceph-mon[33966]: osdmap e226: 8 total, 8 up, 8 in 2024-04-14T15:14:54.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:53 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1194319830' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi090-93342-33"}]: dispatch 2024-04-14T15:14:54.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:53 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1194319830' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi090-93342-33"}]: dispatch 2024-04-14T15:14:54.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:53 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1194319830' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi090-93342-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:14:54.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:53 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:54.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:54.400 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:54.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:14:54.438 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:14:54.501 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:14:54.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:54.525 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:54.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:14:54.626 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:14:54.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:54.651 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:54.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:14:54.752 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:14:54.775 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:54.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:54.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:14:54.877 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:14:54.901 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:54.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:54.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:14:55.003 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:14:55.022 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:14:55.027 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:55.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:55.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:14:55.128 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:14:55.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:55.152 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:55.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:14:55.253 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:14:55.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:55.280 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:55.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:14:55.286 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:54 smithi090 ceph-mon[41670]: 11.8 scrub starts 2024-04-14T15:14:55.286 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:54 smithi090 ceph-mon[41670]: 11.8 scrub ok 2024-04-14T15:14:55.286 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:54 smithi090 ceph-mon[41670]: 5.13 scrub starts 2024-04-14T15:14:55.287 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:54 smithi090 ceph-mon[41670]: 5.13 scrub ok 2024-04-14T15:14:55.287 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:54 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1194319830' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi090-93342-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:14:55.287 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:54 smithi090 ceph-mon[41670]: osdmap e227: 8 total, 8 up, 8 in 2024-04-14T15:14:55.287 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:54 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1194319830' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi090-93342-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi090-93342-33"}]: dispatch 2024-04-14T15:14:55.287 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:54 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:14:55.287 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:54 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:55.287 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:54 smithi090 ceph-mon[30399]: 11.8 scrub starts 2024-04-14T15:14:55.287 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:54 smithi090 ceph-mon[30399]: 11.8 scrub ok 2024-04-14T15:14:55.287 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:54 smithi090 ceph-mon[30399]: 5.13 scrub starts 2024-04-14T15:14:55.287 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:54 smithi090 ceph-mon[30399]: 5.13 scrub ok 2024-04-14T15:14:55.287 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:54 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1194319830' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi090-93342-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:14:55.288 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:54 smithi090 ceph-mon[30399]: osdmap e227: 8 total, 8 up, 8 in 2024-04-14T15:14:55.288 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:54 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1194319830' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi090-93342-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi090-93342-33"}]: dispatch 2024-04-14T15:14:55.288 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:54 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:14:55.288 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:54 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:55.372 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:14:54 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:14:55.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:54 smithi092 ceph-mon[33966]: 11.8 scrub starts 2024-04-14T15:14:55.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:54 smithi092 ceph-mon[33966]: 11.8 scrub ok 2024-04-14T15:14:55.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:54 smithi092 ceph-mon[33966]: 5.13 scrub starts 2024-04-14T15:14:55.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:54 smithi092 ceph-mon[33966]: 5.13 scrub ok 2024-04-14T15:14:55.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:54 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1194319830' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi090-93342-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:14:55.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:54 smithi092 ceph-mon[33966]: osdmap e227: 8 total, 8 up, 8 in 2024-04-14T15:14:55.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:54 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1194319830' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi090-93342-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi090-93342-33"}]: dispatch 2024-04-14T15:14:55.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:54 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:14:55.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:54 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:55.381 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:14:55.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:55.406 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:55.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:14:55.507 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:14:55.526 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-14T15:14:55.526 INFO:teuthology.orchestra.run.smithi090.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-14T15:14:55.526 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-14T15:14:55.531 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:55.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:55.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:14:55.633 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:14:55.657 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:55.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:55.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:14:55.759 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:14:55.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:55.784 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:55.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:14:55.885 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:14:55.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:55.909 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:55.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:14:56.010 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:14:56.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:56.035 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:56.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:14:56.136 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:14:56.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:56.163 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:56.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:14:56.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:55 smithi090 ceph-mon[41670]: 13.0 scrub starts 2024-04-14T15:14:56.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:55 smithi090 ceph-mon[41670]: 13.0 scrub ok 2024-04-14T15:14:56.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:55 smithi090 ceph-mon[41670]: pgmap v287: 222 pgs: 2 peering, 4 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 212 active+clean; 586 KiB data, 539 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-14T15:14:56.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:55 smithi090 ceph-mon[41670]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:14:56.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:55 smithi090 ceph-mon[41670]: osdmap e228: 8 total, 8 up, 8 in 2024-04-14T15:14:56.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:55 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:56.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:55 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1177353462' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi090-93363-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:56.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:55 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:56.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:55 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:56.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:55 smithi090 ceph-mon[30399]: 13.0 scrub starts 2024-04-14T15:14:56.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:55 smithi090 ceph-mon[30399]: 13.0 scrub ok 2024-04-14T15:14:56.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:55 smithi090 ceph-mon[30399]: pgmap v287: 222 pgs: 2 peering, 4 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 212 active+clean; 586 KiB data, 539 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-14T15:14:56.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:55 smithi090 ceph-mon[30399]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:14:56.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:55 smithi090 ceph-mon[30399]: osdmap e228: 8 total, 8 up, 8 in 2024-04-14T15:14:56.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:55 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:56.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:55 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1177353462' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi090-93363-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:56.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:55 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:56.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:55 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:56.264 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:14:56.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:56.291 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:56.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:14:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:55 smithi092 ceph-mon[33966]: 13.0 scrub starts 2024-04-14T15:14:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:55 smithi092 ceph-mon[33966]: 13.0 scrub ok 2024-04-14T15:14:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:55 smithi092 ceph-mon[33966]: pgmap v287: 222 pgs: 2 peering, 4 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 212 active+clean; 586 KiB data, 539 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-14T15:14:56.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:55 smithi092 ceph-mon[33966]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:14:56.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:55 smithi092 ceph-mon[33966]: osdmap e228: 8 total, 8 up, 8 in 2024-04-14T15:14:56.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:55 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:56.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:55 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1177353462' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi090-93363-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:56.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:55 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:56.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:55 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:56.391 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:14:56.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:56.418 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:56.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:14:56.519 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:14:56.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:56.544 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:56.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:14:56.645 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:14:56.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:56.674 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:56.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:14:56.775 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:14:56.802 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:56.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:56.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:14:56.904 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:14:56.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:56.930 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:56.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:14:56.967 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:56 smithi090 ceph-mon[41670]: 11.5 scrub starts 2024-04-14T15:14:56.967 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:56 smithi090 ceph-mon[41670]: 11.5 scrub ok 2024-04-14T15:14:56.967 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:56 smithi090 ceph-mon[41670]: 13.4 scrub starts 2024-04-14T15:14:56.967 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:56 smithi090 ceph-mon[41670]: 13.4 scrub ok 2024-04-14T15:14:56.967 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:56 smithi090 ceph-mon[41670]: 5.1d scrub starts 2024-04-14T15:14:56.967 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:56 smithi090 ceph-mon[41670]: 5.1d scrub ok 2024-04-14T15:14:56.967 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:56 smithi090 ceph-mon[41670]: 6.19 deep-scrub starts 2024-04-14T15:14:56.967 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:56 smithi090 ceph-mon[41670]: 6.19 deep-scrub ok 2024-04-14T15:14:56.967 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:56 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1194319830' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi090-93342-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi090-93342-33"}]': finished 2024-04-14T15:14:56.967 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:56 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1177353462' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi090-93363-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:56.967 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:56 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:56.967 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:56 smithi090 ceph-mon[41670]: osdmap e229: 8 total, 8 up, 8 in 2024-04-14T15:14:56.967 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:56 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:57.031 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:14:57.050 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-14T15:14:57.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:57.056 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:57.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:14:57.156 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:14:57.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:57.181 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:57.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:14:57.282 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:57.305 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:57.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:57.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:57.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:56 smithi092 ceph-mon[33966]: 11.5 scrub starts 2024-04-14T15:14:57.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:56 smithi092 ceph-mon[33966]: 11.5 scrub ok 2024-04-14T15:14:57.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:56 smithi092 ceph-mon[33966]: 13.4 scrub starts 2024-04-14T15:14:57.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:56 smithi092 ceph-mon[33966]: 13.4 scrub ok 2024-04-14T15:14:57.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:56 smithi092 ceph-mon[33966]: 5.1d scrub starts 2024-04-14T15:14:57.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:56 smithi092 ceph-mon[33966]: 5.1d scrub ok 2024-04-14T15:14:57.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:56 smithi092 ceph-mon[33966]: 6.19 deep-scrub starts 2024-04-14T15:14:57.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:56 smithi092 ceph-mon[33966]: 6.19 deep-scrub ok 2024-04-14T15:14:57.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:56 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1194319830' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi090-93342-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi090-93342-33"}]': finished 2024-04-14T15:14:57.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:56 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1177353462' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi090-93363-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:57.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:56 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:57.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:56 smithi092 ceph-mon[33966]: osdmap e229: 8 total, 8 up, 8 in 2024-04-14T15:14:57.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:56 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:57.407 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:14:57.431 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:57.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:57.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:14:57.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:56 smithi090 ceph-mon[30399]: 11.5 scrub starts 2024-04-14T15:14:57.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:56 smithi090 ceph-mon[30399]: 11.5 scrub ok 2024-04-14T15:14:57.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:56 smithi090 ceph-mon[30399]: 13.4 scrub starts 2024-04-14T15:14:57.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:56 smithi090 ceph-mon[30399]: 13.4 scrub ok 2024-04-14T15:14:57.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:56 smithi090 ceph-mon[30399]: 5.1d scrub starts 2024-04-14T15:14:57.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:56 smithi090 ceph-mon[30399]: 5.1d scrub ok 2024-04-14T15:14:57.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:56 smithi090 ceph-mon[30399]: 6.19 deep-scrub starts 2024-04-14T15:14:57.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:56 smithi090 ceph-mon[30399]: 6.19 deep-scrub ok 2024-04-14T15:14:57.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:56 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1194319830' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi090-93342-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi090-93342-33"}]': finished 2024-04-14T15:14:57.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:56 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1177353462' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi090-93363-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:57.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:56 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:57.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:56 smithi090 ceph-mon[30399]: osdmap e229: 8 total, 8 up, 8 in 2024-04-14T15:14:57.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:56 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:57.533 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:14:57.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:57.560 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:57.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:14:57.661 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:57.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:57.685 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:57.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:57.786 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:14:57.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:57.813 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:57.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:14:57.914 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:14:57.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:57.939 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:57.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:14:58.040 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:14:58.064 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:58.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:58.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:14:58.166 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:14:58.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:58.192 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:58.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:14:58.293 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:58.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:58.317 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:58.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:58.376 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:57 smithi092 ceph-mon[33966]: 5.1c scrub starts 2024-04-14T15:14:58.376 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:57 smithi092 ceph-mon[33966]: 5.1c scrub ok 2024-04-14T15:14:58.376 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:57 smithi092 ceph-mon[33966]: pgmap v290: 293 pgs: 1 peering, 4 active+clean+snaptrim, 72 unknown, 4 active+clean+snaptrim_wait, 212 active+clean; 586 KiB data, 539 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:14:58.376 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:57 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-04-14T15:14:58.376 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:57 smithi092 ceph-mon[33966]: 2.16 scrub starts 2024-04-14T15:14:58.377 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:57 smithi092 ceph-mon[33966]: 2.16 scrub ok 2024-04-14T15:14:58.377 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:57 smithi092 ceph-mon[33966]: 6.13 scrub starts 2024-04-14T15:14:58.377 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:57 smithi092 ceph-mon[33966]: 6.13 scrub ok 2024-04-14T15:14:58.377 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:57 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "14"}]': finished 2024-04-14T15:14:58.377 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:57 smithi092 ceph-mon[33966]: osdmap e230: 8 total, 8 up, 8 in 2024-04-14T15:14:58.377 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:57 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1399148291' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi090-93363-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:58.377 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:57 smithi092 ceph-mon[33966]: from='client.46652 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi090-93363-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:58.377 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:57 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:14:58.377 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:57 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:14:58.377 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:57 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:58.418 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:14:58.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:57 smithi090 ceph-mon[41670]: 5.1c scrub starts 2024-04-14T15:14:58.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:57 smithi090 ceph-mon[41670]: 5.1c scrub ok 2024-04-14T15:14:58.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:57 smithi090 ceph-mon[41670]: pgmap v290: 293 pgs: 1 peering, 4 active+clean+snaptrim, 72 unknown, 4 active+clean+snaptrim_wait, 212 active+clean; 586 KiB data, 539 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:14:58.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:57 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-04-14T15:14:58.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:57 smithi090 ceph-mon[41670]: 2.16 scrub starts 2024-04-14T15:14:58.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:57 smithi090 ceph-mon[41670]: 2.16 scrub ok 2024-04-14T15:14:58.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:57 smithi090 ceph-mon[41670]: 6.13 scrub starts 2024-04-14T15:14:58.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:57 smithi090 ceph-mon[41670]: 6.13 scrub ok 2024-04-14T15:14:58.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:57 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "14"}]': finished 2024-04-14T15:14:58.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:57 smithi090 ceph-mon[41670]: osdmap e230: 8 total, 8 up, 8 in 2024-04-14T15:14:58.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:57 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1399148291' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi090-93363-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:58.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:57 smithi090 ceph-mon[41670]: from='client.46652 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi090-93363-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:58.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:57 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:14:58.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:57 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:14:58.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:57 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:58.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:57 smithi090 ceph-mon[30399]: 5.1c scrub starts 2024-04-14T15:14:58.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:57 smithi090 ceph-mon[30399]: 5.1c scrub ok 2024-04-14T15:14:58.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:57 smithi090 ceph-mon[30399]: pgmap v290: 293 pgs: 1 peering, 4 active+clean+snaptrim, 72 unknown, 4 active+clean+snaptrim_wait, 212 active+clean; 586 KiB data, 539 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:14:58.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:57 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-04-14T15:14:58.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:57 smithi090 ceph-mon[30399]: 2.16 scrub starts 2024-04-14T15:14:58.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:57 smithi090 ceph-mon[30399]: 2.16 scrub ok 2024-04-14T15:14:58.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:57 smithi090 ceph-mon[30399]: 6.13 scrub starts 2024-04-14T15:14:58.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:57 smithi090 ceph-mon[30399]: 6.13 scrub ok 2024-04-14T15:14:58.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:57 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "14"}]': finished 2024-04-14T15:14:58.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:57 smithi090 ceph-mon[30399]: osdmap e230: 8 total, 8 up, 8 in 2024-04-14T15:14:58.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:57 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1399148291' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi090-93363-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:58.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:57 smithi090 ceph-mon[30399]: from='client.46652 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi090-93363-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:14:58.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:57 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:14:58.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:57 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:14:58.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:57 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:58.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:58.445 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:58.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:14:58.546 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:14:58.572 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:58.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:58.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:14:58.674 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:14:58.701 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:58.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:58.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:14:58.802 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:14:58.829 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:58.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:58.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:14:58.931 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:14:58.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:58.956 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:58.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:14:59.057 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:14:59.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:59.081 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:59.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:14:59.182 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:14:59.208 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:59.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:59.209 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:14:59.225 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:58 smithi090 ceph-mon[30399]: 4.f scrub starts 2024-04-14T15:14:59.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:58 smithi090 ceph-mon[30399]: 4.f scrub ok 2024-04-14T15:14:59.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:58 smithi090 ceph-mon[30399]: from='client.46652 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi090-93363-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:59.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:58 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-23", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:14:59.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:58 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-23"}]: dispatch 2024-04-14T15:14:59.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:58 smithi090 ceph-mon[30399]: osdmap e231: 8 total, 8 up, 8 in 2024-04-14T15:14:59.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:58 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-23"}]: dispatch 2024-04-14T15:14:59.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:58 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1194319830' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi090-93342-33"}]: dispatch 2024-04-14T15:14:59.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:58 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:59.226 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:58 smithi090 ceph-mon[41670]: 4.f scrub starts 2024-04-14T15:14:59.226 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:58 smithi090 ceph-mon[41670]: 4.f scrub ok 2024-04-14T15:14:59.227 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:58 smithi090 ceph-mon[41670]: from='client.46652 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi090-93363-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:59.227 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:58 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-23", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:14:59.227 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:58 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-23"}]: dispatch 2024-04-14T15:14:59.227 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:58 smithi090 ceph-mon[41670]: osdmap e231: 8 total, 8 up, 8 in 2024-04-14T15:14:59.227 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:58 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-23"}]: dispatch 2024-04-14T15:14:59.227 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:58 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1194319830' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi090-93342-33"}]: dispatch 2024-04-14T15:14:59.227 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:58 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:59.311 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:14:59.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:59.337 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:59.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:14:59.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:58 smithi092 ceph-mon[33966]: 4.f scrub starts 2024-04-14T15:14:59.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:58 smithi092 ceph-mon[33966]: 4.f scrub ok 2024-04-14T15:14:59.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:58 smithi092 ceph-mon[33966]: from='client.46652 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi090-93363-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:14:59.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:58 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-23", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:14:59.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:58 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-23"}]: dispatch 2024-04-14T15:14:59.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:58 smithi092 ceph-mon[33966]: osdmap e231: 8 total, 8 up, 8 in 2024-04-14T15:14:59.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:58 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-23"}]: dispatch 2024-04-14T15:14:59.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:58 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1194319830' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi090-93342-33"}]: dispatch 2024-04-14T15:14:59.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:58 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:14:59.438 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:14:59.463 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:59.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:59.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:14:59.565 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:14:59.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:59.591 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:59.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:14:59.692 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:14:59.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:59.716 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:14:59.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:14:59.817 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:14:59.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:59.849 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:59.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:14:59.950 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:14:59.978 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:14:59.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:59.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:15:00.041 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-14T15:15:00.041 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:15:00.079 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:15:00.104 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:00.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:00.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:15:00.206 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:15:00.234 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:00.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:00.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:15:00.336 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:15:00.364 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:00.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:00.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:15:00.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:59 smithi092 ceph-mon[33966]: pgmap v293: 317 pgs: 2 clean+premerge+peered, 3 active+clean+snaptrim, 26 unknown, 4 active+clean+snaptrim_wait, 282 active+clean; 586 KiB data, 540 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-04-14T15:15:00.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:59 smithi092 ceph-mon[33966]: 13.1 scrub starts 2024-04-14T15:15:00.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:59 smithi092 ceph-mon[33966]: 13.1 scrub ok 2024-04-14T15:15:00.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:59 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-23"}]': finished 2024-04-14T15:15:00.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:59 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1194319830' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi090-93342-33"}]': finished 2024-04-14T15:15:00.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:59 smithi092 ceph-mon[33966]: osdmap e232: 8 total, 8 up, 8 in 2024-04-14T15:15:00.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:59 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-23", "mode": "writeback"}]: dispatch 2024-04-14T15:15:00.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:59 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1194319830' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi090-93342-33"}]: dispatch 2024-04-14T15:15:00.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:59 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/343825698' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi090-93363-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:00.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:59 smithi092 ceph-mon[33966]: from='client.46658 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi090-93363-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:00.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:59 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-23", "mode": "writeback"}]: dispatch 2024-04-14T15:15:00.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:14:59 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:00.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:59 smithi090 ceph-mon[30399]: pgmap v293: 317 pgs: 2 clean+premerge+peered, 3 active+clean+snaptrim, 26 unknown, 4 active+clean+snaptrim_wait, 282 active+clean; 586 KiB data, 540 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-04-14T15:15:00.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:59 smithi090 ceph-mon[30399]: 13.1 scrub starts 2024-04-14T15:15:00.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:59 smithi090 ceph-mon[30399]: 13.1 scrub ok 2024-04-14T15:15:00.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:59 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-23"}]': finished 2024-04-14T15:15:00.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:59 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1194319830' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi090-93342-33"}]': finished 2024-04-14T15:15:00.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:59 smithi090 ceph-mon[30399]: osdmap e232: 8 total, 8 up, 8 in 2024-04-14T15:15:00.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:59 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-23", "mode": "writeback"}]: dispatch 2024-04-14T15:15:00.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:59 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1194319830' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi090-93342-33"}]: dispatch 2024-04-14T15:15:00.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:59 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/343825698' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi090-93363-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:00.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:59 smithi090 ceph-mon[30399]: from='client.46658 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi090-93363-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:00.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:59 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-23", "mode": "writeback"}]: dispatch 2024-04-14T15:15:00.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:14:59 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:00.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:59 smithi090 ceph-mon[41670]: pgmap v293: 317 pgs: 2 clean+premerge+peered, 3 active+clean+snaptrim, 26 unknown, 4 active+clean+snaptrim_wait, 282 active+clean; 586 KiB data, 540 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-04-14T15:15:00.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:59 smithi090 ceph-mon[41670]: 13.1 scrub starts 2024-04-14T15:15:00.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:59 smithi090 ceph-mon[41670]: 13.1 scrub ok 2024-04-14T15:15:00.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:59 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-23"}]': finished 2024-04-14T15:15:00.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:59 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1194319830' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi090-93342-33"}]': finished 2024-04-14T15:15:00.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:59 smithi090 ceph-mon[41670]: osdmap e232: 8 total, 8 up, 8 in 2024-04-14T15:15:00.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:59 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-23", "mode": "writeback"}]: dispatch 2024-04-14T15:15:00.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:59 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1194319830' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi090-93342-33"}]: dispatch 2024-04-14T15:15:00.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:59 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/343825698' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi090-93363-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:00.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:59 smithi090 ceph-mon[41670]: from='client.46658 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi090-93363-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:00.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:59 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-23", "mode": "writeback"}]: dispatch 2024-04-14T15:15:00.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:14:59 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:00.466 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:15:00.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:00.491 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:00.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:15:00.593 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:15:00.620 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:00.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:00.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:15:00.624 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:15:00.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:00.639+0000 7fc0c5322700 1 -- 172.21.15.90:0/3042553803 >> v1:172.21.15.90:6789/0 conn(0x7fc0c010b730 legacy=0x7fc0c010db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:00.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:00.639+0000 7fc0c5322700 1 -- 172.21.15.90:0/3042553803 shutdown_connections 2024-04-14T15:15:00.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:00.639+0000 7fc0c5322700 1 -- 172.21.15.90:0/3042553803 >> 172.21.15.90:0/3042553803 conn(0x7fc0c00fe390 msgr2=0x7fc0c01007b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:15:00.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:00.639+0000 7fc0c5322700 1 -- 172.21.15.90:0/3042553803 shutdown_connections 2024-04-14T15:15:00.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:00.639+0000 7fc0c5322700 1 -- 172.21.15.90:0/3042553803 wait complete. 2024-04-14T15:15:00.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:00.640+0000 7fc0c5322700 1 librados: starting msgr at 2024-04-14T15:15:00.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:00.640+0000 7fc0c5322700 1 librados: starting objecter 2024-04-14T15:15:00.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:00.640+0000 7fc0c5322700 1 Processor -- start 2024-04-14T15:15:00.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:00.640+0000 7fc0c5322700 1 -- start start 2024-04-14T15:15:00.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:00.640+0000 7fc0c5322700 1 librados: setting wanted keys 2024-04-14T15:15:00.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:00.640+0000 7fc0c5322700 1 librados: calling monclient init 2024-04-14T15:15:00.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:00.640+0000 7fc0c5322700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc0c010f060 con 0x7fc0c010f300 2024-04-14T15:15:00.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:00.640+0000 7fc0c5322700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc0c01a8f10 con 0x7fc0c010b730 2024-04-14T15:15:00.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:00.640+0000 7fc0c5322700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc0c01aa0f0 con 0x7fc0c0108880 2024-04-14T15:15:00.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:00.640+0000 7fc0beffd700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7fc0c0108880 0x7fc0c010d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:40650/0 (socket says 172.21.15.90:40650) 2024-04-14T15:15:00.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:00.640+0000 7fc0beffd700 1 -- 172.21.15.90:0/71004710 learned_addr learned my addr 172.21.15.90:0/71004710 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:15:00.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:00.641+0000 7fc0a7fff700 10 client.?.objecter ms_handle_connect 0x7fc0c010f300 2024-04-14T15:15:00.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:00.641+0000 7fc0a7fff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:15:00.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:00.641+0000 7fc0a7fff700 10 client.?.objecter ms_handle_connect 0x7fc0c010b730 2024-04-14T15:15:00.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:00.641+0000 7fc0a7fff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:15:00.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:00.641+0000 7fc0a7fff700 10 client.?.objecter ms_handle_connect 0x7fc0c0108880 2024-04-14T15:15:00.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:00.641+0000 7fc0a7fff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:15:00.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:00.641+0000 7fc0a7fff700 1 -- 172.21.15.90:0/71004710 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1272167676 0 0) 0x7fc0c01a8f10 con 0x7fc0c010b730 2024-04-14T15:15:00.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:00.641+0000 7fc0a7fff700 1 -- 172.21.15.90:0/71004710 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc0a80035b0 con 0x7fc0c010b730 2024-04-14T15:15:00.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:00.641+0000 7fc0a7fff700 1 -- 172.21.15.90:0/71004710 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3193228418 0 0) 0x7fc0c010f060 con 0x7fc0c010f300 2024-04-14T15:15:00.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:00.641+0000 7fc0a7fff700 1 -- 172.21.15.90:0/71004710 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc0c01a8f10 con 0x7fc0c010f300 2024-04-14T15:15:00.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:00.641+0000 7fc0a7fff700 1 -- 172.21.15.90:0/71004710 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1684789325 0 0) 0x7fc0c01aa0f0 con 0x7fc0c0108880 2024-04-14T15:15:00.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:00.641+0000 7fc0a7fff700 1 -- 172.21.15.90:0/71004710 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc0c010f060 con 0x7fc0c0108880 2024-04-14T15:15:00.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:00.641+0000 7fc0a7fff700 1 -- 172.21.15.90:0/71004710 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 303526627 0 0) 0x7fc0a80035b0 con 0x7fc0c010b730 2024-04-14T15:15:00.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:00.641+0000 7fc0a7fff700 1 -- 172.21.15.90:0/71004710 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc0c01aa0f0 con 0x7fc0c010b730 2024-04-14T15:15:00.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:00.641+0000 7fc0a7fff700 1 -- 172.21.15.90:0/71004710 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fc0b40015d0 con 0x7fc0c010b730 2024-04-14T15:15:00.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:00.641+0000 7fc0a7fff700 1 -- 172.21.15.90:0/71004710 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2593340448 0 0) 0x7fc0c01a8f10 con 0x7fc0c010f300 2024-04-14T15:15:00.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:00.641+0000 7fc0a7fff700 1 -- 172.21.15.90:0/71004710 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc0a80035b0 con 0x7fc0c010f300 2024-04-14T15:15:00.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:00.641+0000 7fc0a7fff700 1 -- 172.21.15.90:0/71004710 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4276697994 0 0) 0x7fc0c010f060 con 0x7fc0c0108880 2024-04-14T15:15:00.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:00.641+0000 7fc0a7fff700 1 -- 172.21.15.90:0/71004710 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc0c01a8f10 con 0x7fc0c0108880 2024-04-14T15:15:00.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:00.641+0000 7fc0a7fff700 1 -- 172.21.15.90:0/71004710 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fc0b0001830 con 0x7fc0c0108880 2024-04-14T15:15:00.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:00.641+0000 7fc0a7fff700 1 -- 172.21.15.90:0/71004710 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3548041487 0 0) 0x7fc0c01aa0f0 con 0x7fc0c010b730 2024-04-14T15:15:00.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:00.641+0000 7fc0a7fff700 1 -- 172.21.15.90:0/71004710 >> v1:172.21.15.90:6790/0 conn(0x7fc0c0108880 legacy=0x7fc0c010d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:00.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:00.641+0000 7fc0a7fff700 1 -- 172.21.15.90:0/71004710 >> v1:172.21.15.90:6789/0 conn(0x7fc0c010f300 legacy=0x7fc0c01a67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:00.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:00.641+0000 7fc0a7fff700 1 -- 172.21.15.90:0/71004710 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0c01ab330 con 0x7fc0c010b730 2024-04-14T15:15:00.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:00.641+0000 7fc0c5322700 1 -- 172.21.15.90:0/71004710 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc0c01a9140 con 0x7fc0c010b730 2024-04-14T15:15:00.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:00.641+0000 7fc0c5322700 10 client.48559.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:15:00.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:00.641+0000 7fc0c5322700 1 -- 172.21.15.90:0/71004710 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc0c01a9600 con 0x7fc0c010b730 2024-04-14T15:15:00.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:00.641+0000 7fc0c5322700 1 librados: init done 2024-04-14T15:15:00.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:00.641+0000 7fc0a7fff700 1 -- 172.21.15.90:0/71004710 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fc0b4001b70 con 0x7fc0c010b730 2024-04-14T15:15:00.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:00.641+0000 7fc0a7fff700 1 -- 172.21.15.90:0/71004710 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fc0b4004f90 con 0x7fc0c010b730 2024-04-14T15:15:00.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:00.643+0000 7fc0c5322700 1 -- 172.21.15.90:0/71004710 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0c0103fc0 con 0x7fc0c010b730 2024-04-14T15:15:00.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:00.643+0000 7fc0a7fff700 1 -- 172.21.15.90:0/71004710 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fc0b401bbc0 con 0x7fc0c010b730 2024-04-14T15:15:00.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:00.644+0000 7fc0a7fff700 1 -- 172.21.15.90:0/71004710 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(232..232 src has 51..232) ==== 9644+0+0 (unknown 1664532676 0 0) 0x7fc0b4089ee0 con 0x7fc0c010b730 2024-04-14T15:15:00.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:00.644+0000 7fc0a7fff700 10 client.48559.objecter ms_dispatch 0x7fc0c0001550 osd_map(232..232 src has 51..232) 2024-04-14T15:15:00.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:00.644+0000 7fc0a7fff700 3 client.48559.objecter handle_osd_map got epochs [232,232] > 0 2024-04-14T15:15:00.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:00.644+0000 7fc0a7fff700 3 client.48559.objecter handle_osd_map decoding full epoch 232 2024-04-14T15:15:00.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:00.644+0000 7fc0a7fff700 20 client.48559.objecter dump_active .. 0 homeless 2024-04-14T15:15:00.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:00.644+0000 7fc0a7fff700 10 client.48559.objecter ms_handle_connect 0x7fc0a8071360 2024-04-14T15:15:00.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:00.648+0000 7fc0a7fff700 1 -- 172.21.15.90:0/71004710 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fc0b405e890 con 0x7fc0c010b730 2024-04-14T15:15:00.722 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:15:00.749 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:00.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:00.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:15:00.851 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:15:00.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:00.888+0000 7fc0c5322700 1 -- 172.21.15.90:0/71004710 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7fc0c010d3c0 con 0x7fc0c010b730 2024-04-14T15:15:00.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:00.904 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:00.905 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:15:01.006 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:15:01.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:01.032 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:01.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:15:01.133 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:15:01.157 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:00 smithi090 ceph-mon[41670]: 11.e scrub starts 2024-04-14T15:15:01.157 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:00 smithi090 ceph-mon[41670]: 11.e scrub ok 2024-04-14T15:15:01.158 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:00 smithi090 ceph-mon[41670]: 2.9 scrub starts 2024-04-14T15:15:01.158 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:00 smithi090 ceph-mon[41670]: 2.9 scrub ok 2024-04-14T15:15:01.158 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:00 smithi090 ceph-mon[41670]: 11.16 scrub starts 2024-04-14T15:15:01.158 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:00 smithi090 ceph-mon[41670]: 11.16 scrub ok 2024-04-14T15:15:01.159 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:00 smithi090 ceph-mon[41670]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:15:01.159 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:00 smithi090 ceph-mon[41670]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:15:01.159 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:00 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1194319830' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi090-93342-33"}]': finished 2024-04-14T15:15:01.159 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:00 smithi090 ceph-mon[41670]: from='client.46658 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi090-93363-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:15:01.159 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:00 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-23", "mode": "writeback"}]': finished 2024-04-14T15:15:01.160 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:00 smithi090 ceph-mon[41670]: osdmap e233: 8 total, 8 up, 8 in 2024-04-14T15:15:01.160 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:00 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3767512710' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi090-93342-34"}]: dispatch 2024-04-14T15:15:01.160 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:00 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/71004710' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:15:01.160 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:00 smithi090 ceph-mon[41670]: from='client.48565 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi090-93342-34"}]: dispatch 2024-04-14T15:15:01.160 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:00 smithi090 ceph-mon[41670]: from='client.48559 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:15:01.160 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:00 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3767512710' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi090-93342-34"}]: dispatch 2024-04-14T15:15:01.160 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:00 smithi090 ceph-mon[41670]: from='client.48565 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi090-93342-34"}]: dispatch 2024-04-14T15:15:01.160 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:00 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3767512710' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi090-93342-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:01.160 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:00 smithi090 ceph-mon[41670]: from='client.48565 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi090-93342-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:01.160 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:00 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:01.161 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:00 smithi090 ceph-mon[30399]: 11.e scrub starts 2024-04-14T15:15:01.161 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:00 smithi090 ceph-mon[30399]: 11.e scrub ok 2024-04-14T15:15:01.161 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:00 smithi090 ceph-mon[30399]: 2.9 scrub starts 2024-04-14T15:15:01.161 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:00 smithi090 ceph-mon[30399]: 2.9 scrub ok 2024-04-14T15:15:01.161 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:00 smithi090 ceph-mon[30399]: 11.16 scrub starts 2024-04-14T15:15:01.161 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:00 smithi090 ceph-mon[30399]: 11.16 scrub ok 2024-04-14T15:15:01.161 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:00 smithi090 ceph-mon[30399]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:15:01.161 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:00 smithi090 ceph-mon[30399]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:15:01.161 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:00 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1194319830' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi090-93342-33"}]': finished 2024-04-14T15:15:01.161 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:00 smithi090 ceph-mon[30399]: from='client.46658 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi090-93363-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:15:01.161 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:00 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-23", "mode": "writeback"}]': finished 2024-04-14T15:15:01.162 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:00 smithi090 ceph-mon[30399]: osdmap e233: 8 total, 8 up, 8 in 2024-04-14T15:15:01.162 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:00 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3767512710' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi090-93342-34"}]: dispatch 2024-04-14T15:15:01.162 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:00 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/71004710' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:15:01.162 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:00 smithi090 ceph-mon[30399]: from='client.48565 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi090-93342-34"}]: dispatch 2024-04-14T15:15:01.162 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:00 smithi090 ceph-mon[30399]: from='client.48559 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:15:01.162 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:00 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3767512710' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi090-93342-34"}]: dispatch 2024-04-14T15:15:01.162 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:00 smithi090 ceph-mon[30399]: from='client.48565 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi090-93342-34"}]: dispatch 2024-04-14T15:15:01.162 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:00 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3767512710' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi090-93342-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:01.162 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:00 smithi090 ceph-mon[30399]: from='client.48565 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi090-93342-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:01.163 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:00 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:01.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:01.165 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:01.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:15:01.266 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:15:01.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:01.294 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:01.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:15:01.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:00 smithi092 ceph-mon[33966]: 11.e scrub starts 2024-04-14T15:15:01.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:00 smithi092 ceph-mon[33966]: 11.e scrub ok 2024-04-14T15:15:01.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:00 smithi092 ceph-mon[33966]: 2.9 scrub starts 2024-04-14T15:15:01.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:00 smithi092 ceph-mon[33966]: 2.9 scrub ok 2024-04-14T15:15:01.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:00 smithi092 ceph-mon[33966]: 11.16 scrub starts 2024-04-14T15:15:01.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:00 smithi092 ceph-mon[33966]: 11.16 scrub ok 2024-04-14T15:15:01.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:00 smithi092 ceph-mon[33966]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:15:01.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:00 smithi092 ceph-mon[33966]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:15:01.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:00 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1194319830' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi090-93342-33"}]': finished 2024-04-14T15:15:01.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:00 smithi092 ceph-mon[33966]: from='client.46658 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi090-93363-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:15:01.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:00 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-23", "mode": "writeback"}]': finished 2024-04-14T15:15:01.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:00 smithi092 ceph-mon[33966]: osdmap e233: 8 total, 8 up, 8 in 2024-04-14T15:15:01.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:00 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3767512710' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi090-93342-34"}]: dispatch 2024-04-14T15:15:01.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:00 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/71004710' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:15:01.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:00 smithi092 ceph-mon[33966]: from='client.48565 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi090-93342-34"}]: dispatch 2024-04-14T15:15:01.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:00 smithi092 ceph-mon[33966]: from='client.48559 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:15:01.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:00 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3767512710' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi090-93342-34"}]: dispatch 2024-04-14T15:15:01.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:00 smithi092 ceph-mon[33966]: from='client.48565 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi090-93342-34"}]: dispatch 2024-04-14T15:15:01.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:00 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3767512710' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi090-93342-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:01.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:00 smithi092 ceph-mon[33966]: from='client.48565 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi090-93342-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:01.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:00 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:01.394 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:15:01.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:01.422 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:01.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:15:01.523 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:15:01.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:01.550 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:01.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:15:01.651 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:15:01.675 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:01.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:01.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:15:01.738 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-14T15:15:01.738 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-14T15:15:01.738 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-04-14T15:15:01.738 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-14T15:15:01.739 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-14T15:15:01.739 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-04-14T15:15:01.776 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:15:01.809 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:01.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:01.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:01.852+0000 7fc0a7fff700 1 -- 172.21.15.90:0/71004710 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v234) ==== 87+0+0 (unknown 1262278632 0 0) 0x7fc0b4087550 con 0x7fc0c010b730 2024-04-14T15:15:01.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:01.855+0000 7fc0c5322700 10 librados: watch_flush enter 2024-04-14T15:15:01.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:01.855+0000 7fc0c5322700 10 librados: watch_flush exit 2024-04-14T15:15:01.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:01.855+0000 7fc0c5322700 20 client.48559.objecter shutdown clearing up homeless session... 2024-04-14T15:15:01.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:01.855+0000 7fc0c5322700 10 client.48559.objecter successfully canceled tick 2024-04-14T15:15:01.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:01.856+0000 7fc0c5322700 1 -- 172.21.15.90:0/71004710 >> v1:172.21.15.90:6800/420935427 conn(0x7fc0a8071360 legacy=0x7fc0a8073820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:01.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:01.856+0000 7fc0c5322700 1 -- 172.21.15.90:0/71004710 >> v1:172.21.15.92:6789/0 conn(0x7fc0c010b730 legacy=0x7fc0c010e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:01.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:01.856+0000 7fc0c5322700 1 -- 172.21.15.90:0/71004710 shutdown_connections 2024-04-14T15:15:01.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:01.856+0000 7fc0c5322700 1 -- 172.21.15.90:0/71004710 >> 172.21.15.90:0/71004710 conn(0x7fc0c00fe390 msgr2=0x7fc0c01127e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:15:01.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:01.858+0000 7fc0c5322700 1 -- 172.21.15.90:0/71004710 shutdown_connections 2024-04-14T15:15:01.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:15:01.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:01.859+0000 7fc0c5322700 1 -- 172.21.15.90:0/71004710 wait complete. 2024-04-14T15:15:01.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:01.859+0000 7fc0c5322700 1 librados: shutdown 2024-04-14T15:15:01.862 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is set 2024-04-14T15:15:01.961 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:15:01.985 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:01.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:01.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:15:02.015 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:15:02.087 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:15:02.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:02.111 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:02.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:15:02.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:01 smithi092 ceph-mon[33966]: pgmap v296: 348 pgs: 1 clean+premerge+peered, 3 active+clean+snaptrim, 32 unknown, 4 active+clean+snaptrim_wait, 308 active+clean; 586 KiB data, 540 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.7 KiB/s wr, 4 op/s; 31 B/s, 0 objects/s recovering 2024-04-14T15:15:02.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:01 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-04-14T15:15:02.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:01 smithi092 ceph-mon[33966]: 110.13 scrub starts 2024-04-14T15:15:02.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:01 smithi092 ceph-mon[33966]: 110.13 scrub ok 2024-04-14T15:15:02.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:01 smithi092 ceph-mon[33966]: 11.1 deep-scrub starts 2024-04-14T15:15:02.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:01 smithi092 ceph-mon[33966]: 11.1 deep-scrub ok 2024-04-14T15:15:02.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:01 smithi092 ceph-mon[33966]: 11.1a scrub starts 2024-04-14T15:15:02.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:01 smithi092 ceph-mon[33966]: 11.1a scrub ok 2024-04-14T15:15:02.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:01 smithi092 ceph-mon[33966]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:15:02.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:01 smithi092 ceph-mon[33966]: from='client.48559 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-14T15:15:02.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:01 smithi092 ceph-mon[33966]: from='client.48565 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi090-93342-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:15:02.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:01 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "13"}]': finished 2024-04-14T15:15:02.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:01 smithi092 ceph-mon[33966]: osdmap e234: 8 total, 8 up, 8 in 2024-04-14T15:15:02.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:01 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3767512710' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi090-93342-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi090-93342-34"}]: dispatch 2024-04-14T15:15:02.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:01 smithi092 ceph-mon[33966]: from='client.48565 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi090-93342-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi090-93342-34"}]: dispatch 2024-04-14T15:15:02.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:01 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:15:02.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:01 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:15:02.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:01 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:02.212 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:15:02.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:02.238 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:02.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:15:02.266 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:01 smithi090 ceph-mon[41670]: pgmap v296: 348 pgs: 1 clean+premerge+peered, 3 active+clean+snaptrim, 32 unknown, 4 active+clean+snaptrim_wait, 308 active+clean; 586 KiB data, 540 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.7 KiB/s wr, 4 op/s; 31 B/s, 0 objects/s recovering 2024-04-14T15:15:02.266 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:01 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-04-14T15:15:02.266 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:01 smithi090 ceph-mon[41670]: 110.13 scrub starts 2024-04-14T15:15:02.266 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:01 smithi090 ceph-mon[41670]: 110.13 scrub ok 2024-04-14T15:15:02.266 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:01 smithi090 ceph-mon[41670]: 11.1 deep-scrub starts 2024-04-14T15:15:02.266 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:01 smithi090 ceph-mon[41670]: 11.1 deep-scrub ok 2024-04-14T15:15:02.266 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:01 smithi090 ceph-mon[41670]: 11.1a scrub starts 2024-04-14T15:15:02.266 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:01 smithi090 ceph-mon[41670]: 11.1a scrub ok 2024-04-14T15:15:02.266 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:01 smithi090 ceph-mon[41670]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:15:02.267 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:01 smithi090 ceph-mon[41670]: from='client.48559 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-14T15:15:02.267 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:01 smithi090 ceph-mon[41670]: from='client.48565 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi090-93342-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:15:02.267 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:01 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "13"}]': finished 2024-04-14T15:15:02.267 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:01 smithi090 ceph-mon[41670]: osdmap e234: 8 total, 8 up, 8 in 2024-04-14T15:15:02.267 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:01 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3767512710' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi090-93342-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi090-93342-34"}]: dispatch 2024-04-14T15:15:02.267 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:01 smithi090 ceph-mon[41670]: from='client.48565 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi090-93342-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi090-93342-34"}]: dispatch 2024-04-14T15:15:02.267 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:01 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:15:02.267 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:01 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:15:02.267 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:01 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:02.267 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:01 smithi090 ceph-mon[30399]: pgmap v296: 348 pgs: 1 clean+premerge+peered, 3 active+clean+snaptrim, 32 unknown, 4 active+clean+snaptrim_wait, 308 active+clean; 586 KiB data, 540 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.7 KiB/s wr, 4 op/s; 31 B/s, 0 objects/s recovering 2024-04-14T15:15:02.267 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:01 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-04-14T15:15:02.268 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:01 smithi090 ceph-mon[30399]: 110.13 scrub starts 2024-04-14T15:15:02.268 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:01 smithi090 ceph-mon[30399]: 110.13 scrub ok 2024-04-14T15:15:02.268 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:01 smithi090 ceph-mon[30399]: 11.1 deep-scrub starts 2024-04-14T15:15:02.268 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:01 smithi090 ceph-mon[30399]: 11.1 deep-scrub ok 2024-04-14T15:15:02.268 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:01 smithi090 ceph-mon[30399]: 11.1a scrub starts 2024-04-14T15:15:02.268 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:01 smithi090 ceph-mon[30399]: 11.1a scrub ok 2024-04-14T15:15:02.268 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:01 smithi090 ceph-mon[30399]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:15:02.268 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:01 smithi090 ceph-mon[30399]: from='client.48559 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-14T15:15:02.268 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:01 smithi090 ceph-mon[30399]: from='client.48565 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi090-93342-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:15:02.268 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:01 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "13"}]': finished 2024-04-14T15:15:02.268 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:01 smithi090 ceph-mon[30399]: osdmap e234: 8 total, 8 up, 8 in 2024-04-14T15:15:02.268 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:01 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3767512710' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi090-93342-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi090-93342-34"}]: dispatch 2024-04-14T15:15:02.268 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:01 smithi090 ceph-mon[30399]: from='client.48565 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi090-93342-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi090-93342-34"}]: dispatch 2024-04-14T15:15:02.269 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:01 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:15:02.269 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:01 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:15:02.269 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:01 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:02.339 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:15:02.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:02.365 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:02.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:15:02.466 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:15:02.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:02.490 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:02.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:15:02.591 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:15:02.615 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:02.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:02.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:15:02.634 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-14T15:15:02.717 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:15:02.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:02.742 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:02.742 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:15:02.843 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:15:02.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:02.873 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:02.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:15:02.974 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:15:02.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:02.999 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:02.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:15:03.100 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:15:03.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:03.124 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:03.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:15:03.225 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:15:03.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:03.250 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:03.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:15:03.351 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:15:03.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:03.376 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:03.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:15:03.477 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:15:03.500 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:03.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:03.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:15:03.603 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:15:03.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:03.628 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:03.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:15:03.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.684+0000 7fca67b74700 1 -- 172.21.15.90:0/2388582284 >> v1:172.21.15.90:6789/0 conn(0x7fca601117a0 legacy=0x7fca60113b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:03.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.684+0000 7fca67b74700 1 -- 172.21.15.90:0/2388582284 shutdown_connections 2024-04-14T15:15:03.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.684+0000 7fca67b74700 1 -- 172.21.15.90:0/2388582284 >> 172.21.15.90:0/2388582284 conn(0x7fca600fd2c0 msgr2=0x7fca600ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:15:03.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.684+0000 7fca67b74700 1 -- 172.21.15.90:0/2388582284 shutdown_connections 2024-04-14T15:15:03.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.684+0000 7fca67b74700 1 -- 172.21.15.90:0/2388582284 wait complete. 2024-04-14T15:15:03.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.684+0000 7fca67b74700 1 librados: starting msgr at 2024-04-14T15:15:03.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.684+0000 7fca67b74700 1 librados: starting objecter 2024-04-14T15:15:03.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.684+0000 7fca67b74700 1 Processor -- start 2024-04-14T15:15:03.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.684+0000 7fca67b74700 1 -- start start 2024-04-14T15:15:03.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.684+0000 7fca67b74700 1 librados: setting wanted keys 2024-04-14T15:15:03.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.684+0000 7fca67b74700 1 librados: calling monclient init 2024-04-14T15:15:03.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.685+0000 7fca67b74700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fca601a9d50 con 0x7fca601117a0 2024-04-14T15:15:03.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.685+0000 7fca67b74700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fca601aaf70 con 0x7fca60101880 2024-04-14T15:15:03.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.685+0000 7fca67b74700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fca601ac1b0 con 0x7fca6010dc90 2024-04-14T15:15:03.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.685+0000 7fca6510f700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7fca6010dc90 0x7fca601a4b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:40660/0 (socket says 172.21.15.90:40660) 2024-04-14T15:15:03.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.685+0000 7fca6510f700 1 -- 172.21.15.90:0/3774280749 learned_addr learned my addr 172.21.15.90:0/3774280749 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:15:03.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.685+0000 7fca56ffd700 10 client.?.objecter ms_handle_connect 0x7fca6010dc90 2024-04-14T15:15:03.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.685+0000 7fca56ffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:15:03.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.685+0000 7fca56ffd700 10 client.?.objecter ms_handle_connect 0x7fca601117a0 2024-04-14T15:15:03.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.685+0000 7fca56ffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:15:03.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.685+0000 7fca56ffd700 10 client.?.objecter ms_handle_connect 0x7fca60101880 2024-04-14T15:15:03.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.685+0000 7fca56ffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:15:03.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.685+0000 7fca56ffd700 1 -- 172.21.15.90:0/3774280749 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2396942294 0 0) 0x7fca601ac1b0 con 0x7fca6010dc90 2024-04-14T15:15:03.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.685+0000 7fca56ffd700 1 -- 172.21.15.90:0/3774280749 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fca4c0031f0 con 0x7fca6010dc90 2024-04-14T15:15:03.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.685+0000 7fca56ffd700 1 -- 172.21.15.90:0/3774280749 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3425376355 0 0) 0x7fca601a9d50 con 0x7fca601117a0 2024-04-14T15:15:03.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.685+0000 7fca56ffd700 1 -- 172.21.15.90:0/3774280749 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fca601ac1b0 con 0x7fca601117a0 2024-04-14T15:15:03.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.685+0000 7fca56ffd700 1 -- 172.21.15.90:0/3774280749 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2612053425 0 0) 0x7fca601aaf70 con 0x7fca60101880 2024-04-14T15:15:03.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.685+0000 7fca56ffd700 1 -- 172.21.15.90:0/3774280749 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fca601a9d50 con 0x7fca60101880 2024-04-14T15:15:03.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.685+0000 7fca56ffd700 1 -- 172.21.15.90:0/3774280749 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2786923971 0 0) 0x7fca4c0031f0 con 0x7fca6010dc90 2024-04-14T15:15:03.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.685+0000 7fca56ffd700 1 -- 172.21.15.90:0/3774280749 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fca601aaf70 con 0x7fca6010dc90 2024-04-14T15:15:03.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.685+0000 7fca56ffd700 1 -- 172.21.15.90:0/3774280749 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3755172160 0 0) 0x7fca601ac1b0 con 0x7fca601117a0 2024-04-14T15:15:03.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.685+0000 7fca56ffd700 1 -- 172.21.15.90:0/3774280749 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fca4c0031f0 con 0x7fca601117a0 2024-04-14T15:15:03.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.685+0000 7fca56ffd700 1 -- 172.21.15.90:0/3774280749 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fca50001230 con 0x7fca6010dc90 2024-04-14T15:15:03.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.685+0000 7fca56ffd700 1 -- 172.21.15.90:0/3774280749 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fca5c001b80 con 0x7fca601117a0 2024-04-14T15:15:03.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.685+0000 7fca56ffd700 1 -- 172.21.15.90:0/3774280749 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3504118195 0 0) 0x7fca601a9d50 con 0x7fca60101880 2024-04-14T15:15:03.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.685+0000 7fca56ffd700 1 -- 172.21.15.90:0/3774280749 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fca601ac1b0 con 0x7fca60101880 2024-04-14T15:15:03.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.685+0000 7fca56ffd700 1 -- 172.21.15.90:0/3774280749 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fca58001830 con 0x7fca60101880 2024-04-14T15:15:03.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.686+0000 7fca56ffd700 1 -- 172.21.15.90:0/3774280749 <== mon.2 v1:172.21.15.90:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4283919844 0 0) 0x7fca601aaf70 con 0x7fca6010dc90 2024-04-14T15:15:03.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.686+0000 7fca56ffd700 1 -- 172.21.15.90:0/3774280749 >> v1:172.21.15.92:6789/0 conn(0x7fca60101880 legacy=0x7fca6010d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:03.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.686+0000 7fca56ffd700 1 -- 172.21.15.90:0/3774280749 >> v1:172.21.15.90:6789/0 conn(0x7fca601117a0 legacy=0x7fca601a8410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:03.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.686+0000 7fca56ffd700 1 -- 172.21.15.90:0/3774280749 --> v1:172.21.15.90:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca601ad3f0 con 0x7fca6010dc90 2024-04-14T15:15:03.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.686+0000 7fca67b74700 1 -- 172.21.15.90:0/3774280749 --> v1:172.21.15.90:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fca601a9f80 con 0x7fca6010dc90 2024-04-14T15:15:03.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.686+0000 7fca67b74700 10 client.46676.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:15:03.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.686+0000 7fca67b74700 1 -- 172.21.15.90:0/3774280749 --> v1:172.21.15.90:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fca601aa440 con 0x7fca6010dc90 2024-04-14T15:15:03.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.686+0000 7fca67b74700 1 librados: init done 2024-04-14T15:15:03.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.686+0000 7fca56ffd700 1 -- 172.21.15.90:0/3774280749 <== mon.2 v1:172.21.15.90:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fca50001600 con 0x7fca6010dc90 2024-04-14T15:15:03.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.686+0000 7fca56ffd700 1 -- 172.21.15.90:0/3774280749 <== mon.2 v1:172.21.15.90:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fca50005cd0 con 0x7fca6010dc90 2024-04-14T15:15:03.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.687+0000 7fca67b74700 1 -- 172.21.15.90:0/3774280749 --> v1:172.21.15.90:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fca44005100 con 0x7fca6010dc90 2024-04-14T15:15:03.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.687+0000 7fca56ffd700 1 -- 172.21.15.90:0/3774280749 <== mon.2 v1:172.21.15.90:6790/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fca50003930 con 0x7fca6010dc90 2024-04-14T15:15:03.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.688+0000 7fca56ffd700 1 -- 172.21.15.90:0/3774280749 <== mon.2 v1:172.21.15.90:6790/0 8 ==== osd_map(235..235 src has 51..235) ==== 9240+0+0 (unknown 3027475103 0 0) 0x7fca5008afd0 con 0x7fca6010dc90 2024-04-14T15:15:03.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.688+0000 7fca56ffd700 10 client.46676.objecter ms_dispatch 0x7fca60001550 osd_map(235..235 src has 51..235) 2024-04-14T15:15:03.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.688+0000 7fca56ffd700 3 client.46676.objecter handle_osd_map got epochs [235,235] > 0 2024-04-14T15:15:03.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.688+0000 7fca56ffd700 3 client.46676.objecter handle_osd_map decoding full epoch 235 2024-04-14T15:15:03.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.688+0000 7fca56ffd700 20 client.46676.objecter dump_active .. 0 homeless 2024-04-14T15:15:03.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.688+0000 7fca56ffd700 1 -- 172.21.15.90:0/3774280749 <== mon.2 v1:172.21.15.90:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v235) ==== 54+0+16 (unknown 3184470955 0 2420000192) 0x7fca5008d430 con 0x7fca6010dc90 2024-04-14T15:15:03.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.688+0000 7fca56ffd700 10 client.46676.objecter ms_handle_connect 0x7fca4c0714d0 2024-04-14T15:15:03.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.688+0000 7fca67b74700 10 client.46676.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:15:03.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.688+0000 7fca67b74700 15 client.46676.objecter _session_command_op_assign -1 1 2024-04-14T15:15:03.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.688+0000 7fca67b74700 20 client.46676.objecter _get_session s=0x7fca440053e0 osd=0 3 2024-04-14T15:15:03.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.688+0000 7fca67b74700 20 client.46676.objecter put_session s=0x7fca440053e0 osd=0 3 2024-04-14T15:15:03.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.688+0000 7fca67b74700 20 client.46676.objecter _get_session s=0x7fca440053e0 osd=0 3 2024-04-14T15:15:03.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.688+0000 7fca67b74700 15 client.46676.objecter _session_command_op_remove -1 1 2024-04-14T15:15:03.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.688+0000 7fca67b74700 20 client.46676.objecter get_session s=0x7fca440053e0 osd=0 3 2024-04-14T15:15:03.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.688+0000 7fca67b74700 15 client.46676.objecter _session_command_op_assign 0 1 2024-04-14T15:15:03.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.688+0000 7fca67b74700 20 client.46676.objecter put_session s=0x7fca440053e0 osd=0 4 2024-04-14T15:15:03.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.688+0000 7fca67b74700 10 client.46676.objecter _send_command 1 2024-04-14T15:15:03.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.688+0000 7fca67b74700 1 -- 172.21.15.90:0/3774280749 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fca44004b70 con 0x7fca44004730 2024-04-14T15:15:03.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.688+0000 7fca56ffd700 10 client.46676.objecter ms_handle_connect 0x7fca44004730 2024-04-14T15:15:03.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.689+0000 7fca56ffd700 1 -- 172.21.15.90:0/3774280749 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fca44004b70 con 0x7fca44004730 2024-04-14T15:15:03.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.689+0000 7fca56ffd700 10 client.46676.objecter ms_dispatch 0x7fca60001550 command_reply(tid 1: 0 ) 2024-04-14T15:15:03.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.689+0000 7fca56ffd700 10 client.46676.objecter _finish_command 1 = system:0 2024-04-14T15:15:03.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.689+0000 7fca56ffd700 20 client.46676.objecter put_session s=0x7fca440053e0 osd=0 4 2024-04-14T15:15:03.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.689+0000 7fca56ffd700 15 client.46676.objecter _session_command_op_remove 0 1 2024-04-14T15:15:03.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.706+0000 7fca67b74700 10 client.46676.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:15:03.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.706+0000 7fca67b74700 15 client.46676.objecter _session_command_op_assign -1 2 2024-04-14T15:15:03.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.706+0000 7fca67b74700 20 client.46676.objecter _get_session s=0x7fca440053e0 osd=0 3 2024-04-14T15:15:03.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.706+0000 7fca67b74700 20 client.46676.objecter put_session s=0x7fca440053e0 osd=0 3 2024-04-14T15:15:03.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.706+0000 7fca67b74700 20 client.46676.objecter _get_session s=0x7fca440053e0 osd=0 3 2024-04-14T15:15:03.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.706+0000 7fca67b74700 15 client.46676.objecter _session_command_op_remove -1 2 2024-04-14T15:15:03.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.706+0000 7fca67b74700 20 client.46676.objecter get_session s=0x7fca440053e0 osd=0 3 2024-04-14T15:15:03.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.706+0000 7fca67b74700 15 client.46676.objecter _session_command_op_assign 0 2 2024-04-14T15:15:03.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.706+0000 7fca67b74700 20 client.46676.objecter put_session s=0x7fca440053e0 osd=0 4 2024-04-14T15:15:03.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.706+0000 7fca67b74700 10 client.46676.objecter _send_command 2 2024-04-14T15:15:03.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.706+0000 7fca67b74700 1 -- 172.21.15.90:0/3774280749 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fca440079f0 con 0x7fca44004730 2024-04-14T15:15:03.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.708+0000 7fca56ffd700 1 -- 172.21.15.90:0/3774280749 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fca440079f0 con 0x7fca44004730 2024-04-14T15:15:03.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.708+0000 7fca56ffd700 10 client.46676.objecter ms_dispatch 0x7fca60001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:15:03.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.708+0000 7fca56ffd700 10 client.46676.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:15:03.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.708+0000 7fca56ffd700 20 client.46676.objecter put_session s=0x7fca440053e0 osd=0 4 2024-04-14T15:15:03.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.708+0000 7fca56ffd700 15 client.46676.objecter _session_command_op_remove 0 2 2024-04-14T15:15:03.710 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:15:03.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.708+0000 7fca67b74700 10 client.46676.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:15:03.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.708+0000 7fca67b74700 15 client.46676.objecter _session_command_op_assign -1 3 2024-04-14T15:15:03.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.708+0000 7fca67b74700 20 client.46676.objecter _get_session s=0x7fca44004130 osd=1 3 2024-04-14T15:15:03.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.708+0000 7fca67b74700 20 client.46676.objecter put_session s=0x7fca44004130 osd=1 3 2024-04-14T15:15:03.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.708+0000 7fca67b74700 20 client.46676.objecter _get_session s=0x7fca44004130 osd=1 3 2024-04-14T15:15:03.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.708+0000 7fca67b74700 15 client.46676.objecter _session_command_op_remove -1 3 2024-04-14T15:15:03.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.708+0000 7fca67b74700 20 client.46676.objecter get_session s=0x7fca44004130 osd=1 3 2024-04-14T15:15:03.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.708+0000 7fca67b74700 15 client.46676.objecter _session_command_op_assign 1 3 2024-04-14T15:15:03.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.708+0000 7fca67b74700 20 client.46676.objecter put_session s=0x7fca44004130 osd=1 4 2024-04-14T15:15:03.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.708+0000 7fca67b74700 10 client.46676.objecter _send_command 3 2024-04-14T15:15:03.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.708+0000 7fca67b74700 1 -- 172.21.15.90:0/3774280749 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fca440022a0 con 0x7fca44004310 2024-04-14T15:15:03.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.709+0000 7fca56ffd700 10 client.46676.objecter ms_handle_connect 0x7fca44004310 2024-04-14T15:15:03.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.709+0000 7fca56ffd700 1 -- 172.21.15.90:0/3774280749 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fca440022a0 con 0x7fca44004310 2024-04-14T15:15:03.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.709+0000 7fca56ffd700 10 client.46676.objecter ms_dispatch 0x7fca60001550 command_reply(tid 3: 0 ) 2024-04-14T15:15:03.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.709+0000 7fca56ffd700 10 client.46676.objecter _finish_command 3 = system:0 2024-04-14T15:15:03.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.709+0000 7fca56ffd700 20 client.46676.objecter put_session s=0x7fca44004130 osd=1 4 2024-04-14T15:15:03.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.709+0000 7fca56ffd700 15 client.46676.objecter _session_command_op_remove 1 3 2024-04-14T15:15:03.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.727+0000 7fca67b74700 10 client.46676.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:15:03.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.727+0000 7fca67b74700 15 client.46676.objecter _session_command_op_assign -1 4 2024-04-14T15:15:03.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.727+0000 7fca67b74700 20 client.46676.objecter _get_session s=0x7fca44004130 osd=1 3 2024-04-14T15:15:03.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.727+0000 7fca67b74700 20 client.46676.objecter put_session s=0x7fca44004130 osd=1 3 2024-04-14T15:15:03.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.727+0000 7fca67b74700 20 client.46676.objecter _get_session s=0x7fca44004130 osd=1 3 2024-04-14T15:15:03.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.727+0000 7fca67b74700 15 client.46676.objecter _session_command_op_remove -1 4 2024-04-14T15:15:03.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.727+0000 7fca67b74700 20 client.46676.objecter get_session s=0x7fca44004130 osd=1 3 2024-04-14T15:15:03.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.727+0000 7fca67b74700 15 client.46676.objecter _session_command_op_assign 1 4 2024-04-14T15:15:03.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.727+0000 7fca67b74700 20 client.46676.objecter put_session s=0x7fca44004130 osd=1 4 2024-04-14T15:15:03.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.727+0000 7fca67b74700 10 client.46676.objecter _send_command 4 2024-04-14T15:15:03.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.727+0000 7fca67b74700 1 -- 172.21.15.90:0/3774280749 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fca44001720 con 0x7fca44004310 2024-04-14T15:15:03.729 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:15:03.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.730+0000 7fca56ffd700 1 -- 172.21.15.90:0/3774280749 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fca44001720 con 0x7fca44004310 2024-04-14T15:15:03.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.730+0000 7fca56ffd700 10 client.46676.objecter ms_dispatch 0x7fca60001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:15:03.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.730+0000 7fca56ffd700 10 client.46676.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:15:03.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.730+0000 7fca56ffd700 20 client.46676.objecter put_session s=0x7fca44004130 osd=1 4 2024-04-14T15:15:03.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.730+0000 7fca56ffd700 15 client.46676.objecter _session_command_op_remove 1 4 2024-04-14T15:15:03.731 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:15:03.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.730+0000 7fca67b74700 10 client.46676.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:15:03.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.730+0000 7fca67b74700 15 client.46676.objecter _session_command_op_assign -1 5 2024-04-14T15:15:03.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.730+0000 7fca67b74700 20 client.46676.objecter _get_session s=0x7fca44001980 osd=2 3 2024-04-14T15:15:03.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.730+0000 7fca67b74700 20 client.46676.objecter put_session s=0x7fca44001980 osd=2 3 2024-04-14T15:15:03.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.730+0000 7fca67b74700 20 client.46676.objecter _get_session s=0x7fca44001980 osd=2 3 2024-04-14T15:15:03.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.730+0000 7fca67b74700 15 client.46676.objecter _session_command_op_remove -1 5 2024-04-14T15:15:03.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.730+0000 7fca67b74700 20 client.46676.objecter get_session s=0x7fca44001980 osd=2 3 2024-04-14T15:15:03.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.730+0000 7fca67b74700 15 client.46676.objecter _session_command_op_assign 2 5 2024-04-14T15:15:03.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.730+0000 7fca67b74700 20 client.46676.objecter put_session s=0x7fca44001980 osd=2 4 2024-04-14T15:15:03.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.730+0000 7fca67b74700 10 client.46676.objecter _send_command 5 2024-04-14T15:15:03.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.730+0000 7fca67b74700 1 -- 172.21.15.90:0/3774280749 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fca44011170 con 0x7fca4400d5d0 2024-04-14T15:15:03.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.730+0000 7fca56ffd700 10 client.46676.objecter ms_handle_connect 0x7fca4400d5d0 2024-04-14T15:15:03.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.731+0000 7fca56ffd700 1 -- 172.21.15.90:0/3774280749 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fca44011170 con 0x7fca4400d5d0 2024-04-14T15:15:03.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.731+0000 7fca56ffd700 10 client.46676.objecter ms_dispatch 0x7fca60001550 command_reply(tid 5: 0 ) 2024-04-14T15:15:03.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.731+0000 7fca56ffd700 10 client.46676.objecter _finish_command 5 = system:0 2024-04-14T15:15:03.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.731+0000 7fca56ffd700 20 client.46676.objecter put_session s=0x7fca44001980 osd=2 4 2024-04-14T15:15:03.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.731+0000 7fca56ffd700 15 client.46676.objecter _session_command_op_remove 2 5 2024-04-14T15:15:03.734 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:15:03.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.748+0000 7fca67b74700 10 client.46676.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:15:03.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.748+0000 7fca67b74700 15 client.46676.objecter _session_command_op_assign -1 6 2024-04-14T15:15:03.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.748+0000 7fca67b74700 20 client.46676.objecter _get_session s=0x7fca44001980 osd=2 3 2024-04-14T15:15:03.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.748+0000 7fca67b74700 20 client.46676.objecter put_session s=0x7fca44001980 osd=2 3 2024-04-14T15:15:03.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.748+0000 7fca67b74700 20 client.46676.objecter _get_session s=0x7fca44001980 osd=2 3 2024-04-14T15:15:03.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.748+0000 7fca67b74700 15 client.46676.objecter _session_command_op_remove -1 6 2024-04-14T15:15:03.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.748+0000 7fca67b74700 20 client.46676.objecter get_session s=0x7fca44001980 osd=2 3 2024-04-14T15:15:03.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.748+0000 7fca67b74700 15 client.46676.objecter _session_command_op_assign 2 6 2024-04-14T15:15:03.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.748+0000 7fca67b74700 20 client.46676.objecter put_session s=0x7fca44001980 osd=2 4 2024-04-14T15:15:03.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.748+0000 7fca67b74700 10 client.46676.objecter _send_command 6 2024-04-14T15:15:03.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.748+0000 7fca67b74700 1 -- 172.21.15.90:0/3774280749 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fca44008100 con 0x7fca4400d5d0 2024-04-14T15:15:03.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.750+0000 7fca56ffd700 1 -- 172.21.15.90:0/3774280749 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fca44008100 con 0x7fca4400d5d0 2024-04-14T15:15:03.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.750+0000 7fca56ffd700 10 client.46676.objecter ms_dispatch 0x7fca60001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:15:03.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.750+0000 7fca56ffd700 10 client.46676.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:15:03.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.750+0000 7fca56ffd700 20 client.46676.objecter put_session s=0x7fca44001980 osd=2 4 2024-04-14T15:15:03.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.750+0000 7fca56ffd700 15 client.46676.objecter _session_command_op_remove 2 6 2024-04-14T15:15:03.751 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:15:03.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.750+0000 7fca67b74700 10 client.46676.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:15:03.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.750+0000 7fca67b74700 15 client.46676.objecter _session_command_op_assign -1 7 2024-04-14T15:15:03.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.750+0000 7fca67b74700 20 client.46676.objecter _get_session s=0x7fca44008390 osd=3 3 2024-04-14T15:15:03.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.750+0000 7fca67b74700 20 client.46676.objecter put_session s=0x7fca44008390 osd=3 3 2024-04-14T15:15:03.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.750+0000 7fca67b74700 20 client.46676.objecter _get_session s=0x7fca44008390 osd=3 3 2024-04-14T15:15:03.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.750+0000 7fca67b74700 15 client.46676.objecter _session_command_op_remove -1 7 2024-04-14T15:15:03.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.750+0000 7fca67b74700 20 client.46676.objecter get_session s=0x7fca44008390 osd=3 3 2024-04-14T15:15:03.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.750+0000 7fca67b74700 15 client.46676.objecter _session_command_op_assign 3 7 2024-04-14T15:15:03.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.750+0000 7fca67b74700 20 client.46676.objecter put_session s=0x7fca44008390 osd=3 4 2024-04-14T15:15:03.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.750+0000 7fca67b74700 10 client.46676.objecter _send_command 7 2024-04-14T15:15:03.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.750+0000 7fca67b74700 1 -- 172.21.15.90:0/3774280749 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fca4400d380 con 0x7fca440124a0 2024-04-14T15:15:03.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.751+0000 7fca56ffd700 10 client.46676.objecter ms_handle_connect 0x7fca440124a0 2024-04-14T15:15:03.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.751+0000 7fca56ffd700 1 -- 172.21.15.90:0/3774280749 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fca4400d380 con 0x7fca440124a0 2024-04-14T15:15:03.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.751+0000 7fca56ffd700 10 client.46676.objecter ms_dispatch 0x7fca60001550 command_reply(tid 7: 0 ) 2024-04-14T15:15:03.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.751+0000 7fca56ffd700 10 client.46676.objecter _finish_command 7 = system:0 2024-04-14T15:15:03.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.751+0000 7fca56ffd700 20 client.46676.objecter put_session s=0x7fca44008390 osd=3 4 2024-04-14T15:15:03.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.751+0000 7fca56ffd700 15 client.46676.objecter _session_command_op_remove 3 7 2024-04-14T15:15:03.757 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:03.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:03.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:15:03.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.770+0000 7fca67b74700 10 client.46676.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:15:03.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.770+0000 7fca67b74700 15 client.46676.objecter _session_command_op_assign -1 8 2024-04-14T15:15:03.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.770+0000 7fca67b74700 20 client.46676.objecter _get_session s=0x7fca44008390 osd=3 3 2024-04-14T15:15:03.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.770+0000 7fca67b74700 20 client.46676.objecter put_session s=0x7fca44008390 osd=3 3 2024-04-14T15:15:03.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.770+0000 7fca67b74700 20 client.46676.objecter _get_session s=0x7fca44008390 osd=3 3 2024-04-14T15:15:03.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.770+0000 7fca67b74700 15 client.46676.objecter _session_command_op_remove -1 8 2024-04-14T15:15:03.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.770+0000 7fca67b74700 20 client.46676.objecter get_session s=0x7fca44008390 osd=3 3 2024-04-14T15:15:03.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.770+0000 7fca67b74700 15 client.46676.objecter _session_command_op_assign 3 8 2024-04-14T15:15:03.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.770+0000 7fca67b74700 20 client.46676.objecter put_session s=0x7fca44008390 osd=3 4 2024-04-14T15:15:03.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.770+0000 7fca67b74700 10 client.46676.objecter _send_command 8 2024-04-14T15:15:03.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.770+0000 7fca67b74700 1 -- 172.21.15.90:0/3774280749 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fca4400c8f0 con 0x7fca440124a0 2024-04-14T15:15:03.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.771+0000 7fca56ffd700 1 -- 172.21.15.90:0/3774280749 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fca4400c8f0 con 0x7fca440124a0 2024-04-14T15:15:03.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.771+0000 7fca56ffd700 10 client.46676.objecter ms_dispatch 0x7fca60001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:15:03.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.771+0000 7fca56ffd700 10 client.46676.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:15:03.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.771+0000 7fca56ffd700 20 client.46676.objecter put_session s=0x7fca44008390 osd=3 4 2024-04-14T15:15:03.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.771+0000 7fca56ffd700 15 client.46676.objecter _session_command_op_remove 3 8 2024-04-14T15:15:03.772 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:15:03.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.771+0000 7fca67b74700 10 client.46676.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:15:03.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.771+0000 7fca67b74700 15 client.46676.objecter _session_command_op_assign -1 9 2024-04-14T15:15:03.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.772+0000 7fca67b74700 20 client.46676.objecter _get_session s=0x7fca4400c8f0 osd=4 3 2024-04-14T15:15:03.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.772+0000 7fca67b74700 20 client.46676.objecter put_session s=0x7fca4400c8f0 osd=4 3 2024-04-14T15:15:03.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.772+0000 7fca67b74700 20 client.46676.objecter _get_session s=0x7fca4400c8f0 osd=4 3 2024-04-14T15:15:03.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.772+0000 7fca67b74700 15 client.46676.objecter _session_command_op_remove -1 9 2024-04-14T15:15:03.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.772+0000 7fca67b74700 20 client.46676.objecter get_session s=0x7fca4400c8f0 osd=4 3 2024-04-14T15:15:03.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.772+0000 7fca67b74700 15 client.46676.objecter _session_command_op_assign 4 9 2024-04-14T15:15:03.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.772+0000 7fca67b74700 20 client.46676.objecter put_session s=0x7fca4400c8f0 osd=4 4 2024-04-14T15:15:03.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.772+0000 7fca67b74700 10 client.46676.objecter _send_command 9 2024-04-14T15:15:03.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.772+0000 7fca67b74700 1 -- 172.21.15.90:0/3774280749 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fca44012170 con 0x7fca4400ca60 2024-04-14T15:15:03.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.772+0000 7fca56ffd700 10 client.46676.objecter ms_handle_connect 0x7fca4400ca60 2024-04-14T15:15:03.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.773+0000 7fca56ffd700 1 -- 172.21.15.90:0/3774280749 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fca44012170 con 0x7fca4400ca60 2024-04-14T15:15:03.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.773+0000 7fca56ffd700 10 client.46676.objecter ms_dispatch 0x7fca60001550 command_reply(tid 9: 0 ) 2024-04-14T15:15:03.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.773+0000 7fca56ffd700 10 client.46676.objecter _finish_command 9 = system:0 2024-04-14T15:15:03.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.773+0000 7fca56ffd700 20 client.46676.objecter put_session s=0x7fca4400c8f0 osd=4 4 2024-04-14T15:15:03.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.773+0000 7fca56ffd700 15 client.46676.objecter _session_command_op_remove 4 9 2024-04-14T15:15:03.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.789+0000 7fca67b74700 10 client.46676.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:15:03.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.789+0000 7fca67b74700 15 client.46676.objecter _session_command_op_assign -1 10 2024-04-14T15:15:03.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.789+0000 7fca67b74700 20 client.46676.objecter _get_session s=0x7fca4400c8f0 osd=4 3 2024-04-14T15:15:03.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.789+0000 7fca67b74700 20 client.46676.objecter put_session s=0x7fca4400c8f0 osd=4 3 2024-04-14T15:15:03.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.789+0000 7fca67b74700 20 client.46676.objecter _get_session s=0x7fca4400c8f0 osd=4 3 2024-04-14T15:15:03.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.789+0000 7fca67b74700 15 client.46676.objecter _session_command_op_remove -1 10 2024-04-14T15:15:03.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.789+0000 7fca67b74700 20 client.46676.objecter get_session s=0x7fca4400c8f0 osd=4 3 2024-04-14T15:15:03.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.789+0000 7fca67b74700 15 client.46676.objecter _session_command_op_assign 4 10 2024-04-14T15:15:03.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.789+0000 7fca67b74700 20 client.46676.objecter put_session s=0x7fca4400c8f0 osd=4 4 2024-04-14T15:15:03.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.789+0000 7fca67b74700 10 client.46676.objecter _send_command 10 2024-04-14T15:15:03.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.789+0000 7fca67b74700 1 -- 172.21.15.90:0/3774280749 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fca4401a860 con 0x7fca4400ca60 2024-04-14T15:15:03.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.791+0000 7fca56ffd700 1 -- 172.21.15.90:0/3774280749 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fca4401a860 con 0x7fca4400ca60 2024-04-14T15:15:03.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.791+0000 7fca56ffd700 10 client.46676.objecter ms_dispatch 0x7fca60001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:15:03.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.791+0000 7fca56ffd700 10 client.46676.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:15:03.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.791+0000 7fca56ffd700 20 client.46676.objecter put_session s=0x7fca4400c8f0 osd=4 4 2024-04-14T15:15:03.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.791+0000 7fca56ffd700 15 client.46676.objecter _session_command_op_remove 4 10 2024-04-14T15:15:03.792 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:15:03.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.792+0000 7fca67b74700 10 client.46676.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:15:03.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.792+0000 7fca67b74700 15 client.46676.objecter _session_command_op_assign -1 11 2024-04-14T15:15:03.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.792+0000 7fca67b74700 20 client.46676.objecter _get_session s=0x7fca44012170 osd=5 3 2024-04-14T15:15:03.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.792+0000 7fca67b74700 20 client.46676.objecter put_session s=0x7fca44012170 osd=5 3 2024-04-14T15:15:03.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.792+0000 7fca67b74700 20 client.46676.objecter _get_session s=0x7fca44012170 osd=5 3 2024-04-14T15:15:03.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.792+0000 7fca67b74700 15 client.46676.objecter _session_command_op_remove -1 11 2024-04-14T15:15:03.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.792+0000 7fca67b74700 20 client.46676.objecter get_session s=0x7fca44012170 osd=5 3 2024-04-14T15:15:03.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.792+0000 7fca67b74700 15 client.46676.objecter _session_command_op_assign 5 11 2024-04-14T15:15:03.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.792+0000 7fca67b74700 20 client.46676.objecter put_session s=0x7fca44012170 osd=5 4 2024-04-14T15:15:03.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.792+0000 7fca67b74700 10 client.46676.objecter _send_command 11 2024-04-14T15:15:03.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.792+0000 7fca67b74700 1 -- 172.21.15.90:0/3774280749 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fca4401ea30 con 0x7fca4401ad70 2024-04-14T15:15:03.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.792+0000 7fca56ffd700 10 client.46676.objecter ms_handle_connect 0x7fca4401ad70 2024-04-14T15:15:03.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.793+0000 7fca56ffd700 1 -- 172.21.15.90:0/3774280749 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fca4401ea30 con 0x7fca4401ad70 2024-04-14T15:15:03.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.793+0000 7fca56ffd700 10 client.46676.objecter ms_dispatch 0x7fca60001550 command_reply(tid 11: 0 ) 2024-04-14T15:15:03.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.793+0000 7fca56ffd700 10 client.46676.objecter _finish_command 11 = system:0 2024-04-14T15:15:03.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.793+0000 7fca56ffd700 20 client.46676.objecter put_session s=0x7fca44012170 osd=5 4 2024-04-14T15:15:03.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.793+0000 7fca56ffd700 15 client.46676.objecter _session_command_op_remove 5 11 2024-04-14T15:15:03.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.809+0000 7fca67b74700 10 client.46676.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:15:03.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.809+0000 7fca67b74700 15 client.46676.objecter _session_command_op_assign -1 12 2024-04-14T15:15:03.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.809+0000 7fca67b74700 20 client.46676.objecter _get_session s=0x7fca44012170 osd=5 3 2024-04-14T15:15:03.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.809+0000 7fca67b74700 20 client.46676.objecter put_session s=0x7fca44012170 osd=5 3 2024-04-14T15:15:03.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.809+0000 7fca67b74700 20 client.46676.objecter _get_session s=0x7fca44012170 osd=5 3 2024-04-14T15:15:03.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.809+0000 7fca67b74700 15 client.46676.objecter _session_command_op_remove -1 12 2024-04-14T15:15:03.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.809+0000 7fca67b74700 20 client.46676.objecter get_session s=0x7fca44012170 osd=5 3 2024-04-14T15:15:03.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.809+0000 7fca67b74700 15 client.46676.objecter _session_command_op_assign 5 12 2024-04-14T15:15:03.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.809+0000 7fca67b74700 20 client.46676.objecter put_session s=0x7fca44012170 osd=5 4 2024-04-14T15:15:03.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.809+0000 7fca67b74700 10 client.46676.objecter _send_command 12 2024-04-14T15:15:03.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.809+0000 7fca67b74700 1 -- 172.21.15.90:0/3774280749 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fca4400c700 con 0x7fca4401ad70 2024-04-14T15:15:03.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.811+0000 7fca56ffd700 1 -- 172.21.15.90:0/3774280749 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fca4400c700 con 0x7fca4401ad70 2024-04-14T15:15:03.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.811+0000 7fca56ffd700 10 client.46676.objecter ms_dispatch 0x7fca60001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:15:03.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.811+0000 7fca56ffd700 10 client.46676.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:15:03.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.811+0000 7fca56ffd700 20 client.46676.objecter put_session s=0x7fca44012170 osd=5 4 2024-04-14T15:15:03.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.811+0000 7fca56ffd700 15 client.46676.objecter _session_command_op_remove 5 12 2024-04-14T15:15:03.811 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:15:03.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.811+0000 7fca67b74700 10 client.46676.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:15:03.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.811+0000 7fca67b74700 15 client.46676.objecter _session_command_op_assign -1 13 2024-04-14T15:15:03.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.811+0000 7fca67b74700 20 client.46676.objecter _get_session s=0x7fca4400c700 osd=6 3 2024-04-14T15:15:03.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.811+0000 7fca67b74700 20 client.46676.objecter put_session s=0x7fca4400c700 osd=6 3 2024-04-14T15:15:03.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.811+0000 7fca67b74700 20 client.46676.objecter _get_session s=0x7fca4400c700 osd=6 3 2024-04-14T15:15:03.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.811+0000 7fca67b74700 15 client.46676.objecter _session_command_op_remove -1 13 2024-04-14T15:15:03.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.811+0000 7fca67b74700 20 client.46676.objecter get_session s=0x7fca4400c700 osd=6 3 2024-04-14T15:15:03.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.811+0000 7fca67b74700 15 client.46676.objecter _session_command_op_assign 6 13 2024-04-14T15:15:03.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.811+0000 7fca67b74700 20 client.46676.objecter put_session s=0x7fca4400c700 osd=6 4 2024-04-14T15:15:03.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.811+0000 7fca67b74700 10 client.46676.objecter _send_command 13 2024-04-14T15:15:03.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.811+0000 7fca67b74700 1 -- 172.21.15.90:0/3774280749 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fca440201f0 con 0x7fca44016780 2024-04-14T15:15:03.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.811+0000 7fca56ffd700 10 client.46676.objecter ms_handle_connect 0x7fca44016780 2024-04-14T15:15:03.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.812+0000 7fca56ffd700 1 -- 172.21.15.90:0/3774280749 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fca5c004170 con 0x7fca44016780 2024-04-14T15:15:03.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.812+0000 7fca56ffd700 10 client.46676.objecter ms_dispatch 0x7fca60001550 command_reply(tid 13: 0 ) 2024-04-14T15:15:03.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.812+0000 7fca56ffd700 10 client.46676.objecter _finish_command 13 = system:0 2024-04-14T15:15:03.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.812+0000 7fca56ffd700 20 client.46676.objecter put_session s=0x7fca4400c700 osd=6 4 2024-04-14T15:15:03.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.812+0000 7fca56ffd700 15 client.46676.objecter _session_command_op_remove 6 13 2024-04-14T15:15:03.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.828+0000 7fca67b74700 10 client.46676.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:15:03.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.828+0000 7fca67b74700 15 client.46676.objecter _session_command_op_assign -1 14 2024-04-14T15:15:03.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.828+0000 7fca67b74700 20 client.46676.objecter _get_session s=0x7fca4400c700 osd=6 3 2024-04-14T15:15:03.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.828+0000 7fca67b74700 20 client.46676.objecter put_session s=0x7fca4400c700 osd=6 3 2024-04-14T15:15:03.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.828+0000 7fca67b74700 20 client.46676.objecter _get_session s=0x7fca4400c700 osd=6 3 2024-04-14T15:15:03.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.828+0000 7fca67b74700 15 client.46676.objecter _session_command_op_remove -1 14 2024-04-14T15:15:03.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.828+0000 7fca67b74700 20 client.46676.objecter get_session s=0x7fca4400c700 osd=6 3 2024-04-14T15:15:03.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.828+0000 7fca67b74700 15 client.46676.objecter _session_command_op_assign 6 14 2024-04-14T15:15:03.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.828+0000 7fca67b74700 20 client.46676.objecter put_session s=0x7fca4400c700 osd=6 4 2024-04-14T15:15:03.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.828+0000 7fca67b74700 10 client.46676.objecter _send_command 14 2024-04-14T15:15:03.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.828+0000 7fca67b74700 1 -- 172.21.15.90:0/3774280749 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fca44016d70 con 0x7fca44016780 2024-04-14T15:15:03.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.830+0000 7fca56ffd700 1 -- 172.21.15.90:0/3774280749 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fca44016d70 con 0x7fca44016780 2024-04-14T15:15:03.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.830+0000 7fca56ffd700 10 client.46676.objecter ms_dispatch 0x7fca60001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:15:03.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.830+0000 7fca56ffd700 10 client.46676.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:15:03.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.830+0000 7fca56ffd700 20 client.46676.objecter put_session s=0x7fca4400c700 osd=6 4 2024-04-14T15:15:03.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.830+0000 7fca56ffd700 15 client.46676.objecter _session_command_op_remove 6 14 2024-04-14T15:15:03.832 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:15:03.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.830+0000 7fca67b74700 10 client.46676.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:15:03.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.830+0000 7fca67b74700 15 client.46676.objecter _session_command_op_assign -1 15 2024-04-14T15:15:03.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.830+0000 7fca67b74700 20 client.46676.objecter _get_session s=0x7fca44016d70 osd=7 3 2024-04-14T15:15:03.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.830+0000 7fca67b74700 20 client.46676.objecter put_session s=0x7fca44016d70 osd=7 3 2024-04-14T15:15:03.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.830+0000 7fca67b74700 20 client.46676.objecter _get_session s=0x7fca44016d70 osd=7 3 2024-04-14T15:15:03.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.830+0000 7fca67b74700 15 client.46676.objecter _session_command_op_remove -1 15 2024-04-14T15:15:03.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.830+0000 7fca67b74700 20 client.46676.objecter get_session s=0x7fca44016d70 osd=7 3 2024-04-14T15:15:03.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.830+0000 7fca67b74700 15 client.46676.objecter _session_command_op_assign 7 15 2024-04-14T15:15:03.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.830+0000 7fca67b74700 20 client.46676.objecter put_session s=0x7fca44016d70 osd=7 4 2024-04-14T15:15:03.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.830+0000 7fca67b74700 10 client.46676.objecter _send_command 15 2024-04-14T15:15:03.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.830+0000 7fca67b74700 1 -- 172.21.15.90:0/3774280749 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fca44027600 con 0x7fca44023af0 2024-04-14T15:15:03.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.831+0000 7fca56ffd700 10 client.46676.objecter ms_handle_connect 0x7fca44023af0 2024-04-14T15:15:03.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.831+0000 7fca56ffd700 1 -- 172.21.15.90:0/3774280749 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fca44027600 con 0x7fca44023af0 2024-04-14T15:15:03.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.832+0000 7fca56ffd700 10 client.46676.objecter ms_dispatch 0x7fca60001550 command_reply(tid 15: 0 ) 2024-04-14T15:15:03.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.832+0000 7fca56ffd700 10 client.46676.objecter _finish_command 15 = system:0 2024-04-14T15:15:03.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.832+0000 7fca56ffd700 20 client.46676.objecter put_session s=0x7fca44016d70 osd=7 4 2024-04-14T15:15:03.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.832+0000 7fca56ffd700 15 client.46676.objecter _session_command_op_remove 7 15 2024-04-14T15:15:03.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.848+0000 7fca67b74700 10 client.46676.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:15:03.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.848+0000 7fca67b74700 15 client.46676.objecter _session_command_op_assign -1 16 2024-04-14T15:15:03.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.848+0000 7fca67b74700 20 client.46676.objecter _get_session s=0x7fca44016d70 osd=7 3 2024-04-14T15:15:03.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.848+0000 7fca67b74700 20 client.46676.objecter put_session s=0x7fca44016d70 osd=7 3 2024-04-14T15:15:03.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.848+0000 7fca67b74700 20 client.46676.objecter _get_session s=0x7fca44016d70 osd=7 3 2024-04-14T15:15:03.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.848+0000 7fca67b74700 15 client.46676.objecter _session_command_op_remove -1 16 2024-04-14T15:15:03.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.848+0000 7fca67b74700 20 client.46676.objecter get_session s=0x7fca44016d70 osd=7 3 2024-04-14T15:15:03.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.848+0000 7fca67b74700 15 client.46676.objecter _session_command_op_assign 7 16 2024-04-14T15:15:03.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.848+0000 7fca67b74700 20 client.46676.objecter put_session s=0x7fca44016d70 osd=7 4 2024-04-14T15:15:03.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.848+0000 7fca67b74700 10 client.46676.objecter _send_command 16 2024-04-14T15:15:03.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.848+0000 7fca67b74700 1 -- 172.21.15.90:0/3774280749 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fca44016200 con 0x7fca44023af0 2024-04-14T15:15:03.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.849+0000 7fca56ffd700 1 -- 172.21.15.90:0/3774280749 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fca44016200 con 0x7fca44023af0 2024-04-14T15:15:03.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.849+0000 7fca56ffd700 10 client.46676.objecter ms_dispatch 0x7fca60001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:15:03.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.849+0000 7fca56ffd700 10 client.46676.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:15:03.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.849+0000 7fca56ffd700 20 client.46676.objecter put_session s=0x7fca44016d70 osd=7 4 2024-04-14T15:15:03.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.849+0000 7fca56ffd700 15 client.46676.objecter _session_command_op_remove 7 16 2024-04-14T15:15:03.850 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:15:03.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.850+0000 7fca67b74700 10 librados: watch_flush enter 2024-04-14T15:15:03.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.850+0000 7fca67b74700 10 librados: watch_flush exit 2024-04-14T15:15:03.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.850+0000 7fca67b74700 10 client.46676.objecter close_session for osd.0 2024-04-14T15:15:03.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.850+0000 7fca67b74700 1 -- 172.21.15.90:0/3774280749 >> v1:172.21.15.90:6801/3761806885 conn(0x7fca44004730 legacy=0x7fca44002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:03.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.850+0000 7fca67b74700 20 client.46676.objecter put_session s=0x7fca440053e0 osd=0 1 2024-04-14T15:15:03.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.850+0000 7fca67b74700 10 client.46676.objecter close_session for osd.1 2024-04-14T15:15:03.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.850+0000 7fca67b74700 1 -- 172.21.15.90:0/3774280749 >> v1:172.21.15.90:6805/4170361699 conn(0x7fca44004310 legacy=0x7fca4400ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:03.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.850+0000 7fca67b74700 20 client.46676.objecter put_session s=0x7fca44004130 osd=1 1 2024-04-14T15:15:03.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.850+0000 7fca67b74700 10 client.46676.objecter close_session for osd.2 2024-04-14T15:15:03.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.850+0000 7fca67b74700 1 -- 172.21.15.90:0/3774280749 >> v1:172.21.15.90:6809/1686184251 conn(0x7fca4400d5d0 legacy=0x7fca4400f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:03.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.850+0000 7fca67b74700 20 client.46676.objecter put_session s=0x7fca44001980 osd=2 1 2024-04-14T15:15:03.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.850+0000 7fca67b74700 10 client.46676.objecter close_session for osd.3 2024-04-14T15:15:03.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.850+0000 7fca67b74700 1 -- 172.21.15.90:0/3774280749 >> v1:172.21.15.90:6813/2270166774 conn(0x7fca440124a0 legacy=0x7fca44014890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:03.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.850+0000 7fca66111700 1 -- 172.21.15.90:0/3774280749 reap_dead start 2024-04-14T15:15:03.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.850+0000 7fca67b74700 20 client.46676.objecter put_session s=0x7fca44008390 osd=3 1 2024-04-14T15:15:03.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.850+0000 7fca67b74700 10 client.46676.objecter close_session for osd.4 2024-04-14T15:15:03.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.850+0000 7fca67b74700 1 -- 172.21.15.90:0/3774280749 >> v1:172.21.15.92:6800/3887549733 conn(0x7fca4400ca60 legacy=0x7fca44019140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:03.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.850+0000 7fca67b74700 20 client.46676.objecter put_session s=0x7fca4400c8f0 osd=4 1 2024-04-14T15:15:03.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.850+0000 7fca67b74700 10 client.46676.objecter close_session for osd.5 2024-04-14T15:15:03.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.850+0000 7fca66111700 1 -- 172.21.15.90:0/3774280749 reap_dead start 2024-04-14T15:15:03.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.850+0000 7fca67b74700 1 -- 172.21.15.90:0/3774280749 >> v1:172.21.15.92:6804/3244561440 conn(0x7fca4401ad70 legacy=0x7fca4401d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:03.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.850+0000 7fca67b74700 20 client.46676.objecter put_session s=0x7fca44012170 osd=5 1 2024-04-14T15:15:03.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.850+0000 7fca67b74700 10 client.46676.objecter close_session for osd.6 2024-04-14T15:15:03.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.850+0000 7fca67b74700 1 -- 172.21.15.90:0/3774280749 >> v1:172.21.15.92:6808/636125963 conn(0x7fca44016780 legacy=0x7fca4401ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:03.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.850+0000 7fca67b74700 20 client.46676.objecter put_session s=0x7fca4400c700 osd=6 1 2024-04-14T15:15:03.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.850+0000 7fca67b74700 10 client.46676.objecter close_session for osd.7 2024-04-14T15:15:03.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.850+0000 7fca67b74700 1 -- 172.21.15.90:0/3774280749 >> v1:172.21.15.92:6812/1773706716 conn(0x7fca44023af0 legacy=0x7fca44025ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:03.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.850+0000 7fca67b74700 20 client.46676.objecter put_session s=0x7fca44016d70 osd=7 1 2024-04-14T15:15:03.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.850+0000 7fca67b74700 20 client.46676.objecter shutdown clearing up homeless session... 2024-04-14T15:15:03.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.850+0000 7fca67b74700 10 client.46676.objecter successfully canceled tick 2024-04-14T15:15:03.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.850+0000 7fca67b74700 1 -- 172.21.15.90:0/3774280749 >> v1:172.21.15.90:6800/420935427 conn(0x7fca4c0714d0 legacy=0x7fca4c073990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:03.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.850+0000 7fca67b74700 1 -- 172.21.15.90:0/3774280749 >> v1:172.21.15.90:6790/0 conn(0x7fca6010dc90 legacy=0x7fca601a4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:03.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.850+0000 7fca66111700 1 -- 172.21.15.90:0/3774280749 reap_dead start 2024-04-14T15:15:03.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.850+0000 7fca67b74700 1 -- 172.21.15.90:0/3774280749 shutdown_connections 2024-04-14T15:15:03.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.850+0000 7fca67b74700 1 -- 172.21.15.90:0/3774280749 >> 172.21.15.90:0/3774280749 conn(0x7fca600fd2c0 msgr2=0x7fca60113b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:15:03.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.850+0000 7fca67b74700 1 -- 172.21.15.90:0/3774280749 shutdown_connections 2024-04-14T15:15:03.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.850+0000 7fca67b74700 1 -- 172.21.15.90:0/3774280749 wait complete. 2024-04-14T15:15:03.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:03.850+0000 7fca67b74700 1 librados: shutdown 2024-04-14T15:15:03.855 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:15:03.855 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:15:03.855 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:15:03.855 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:15:03.856 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:15:03.856 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:15:03.856 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:15:03.856 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:15:03.859 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:15:03.888 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:03.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:03.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:15:03.990 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:15:03.996 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:03 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:15:03.996 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:03 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-23"}]: dispatch 2024-04-14T15:15:03.996 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:03 smithi092 ceph-mon[33966]: osdmap e235: 8 total, 8 up, 8 in 2024-04-14T15:15:03.996 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:03 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-23"}]: dispatch 2024-04-14T15:15:03.996 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:03 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:03.996 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:03 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3774280749' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:15:04.015 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:04.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:04.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:15:04.021 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-04-14T15:15:04.021 INFO:teuthology.orchestra.run.smithi092.stdout: "success": "" 2024-04-14T15:15:04.021 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-04-14T15:15:04.118 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:15:04.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:04.143 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:04.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:15:04.191 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:15:03 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:15:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:15:04.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:03 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:15:04.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:03 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-23"}]: dispatch 2024-04-14T15:15:04.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:03 smithi090 ceph-mon[30399]: osdmap e235: 8 total, 8 up, 8 in 2024-04-14T15:15:04.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:03 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-23"}]: dispatch 2024-04-14T15:15:04.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:03 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:04.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:03 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3774280749' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:15:04.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:03 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:15:04.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:03 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-23"}]: dispatch 2024-04-14T15:15:04.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:03 smithi090 ceph-mon[41670]: osdmap e235: 8 total, 8 up, 8 in 2024-04-14T15:15:04.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:03 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-23"}]: dispatch 2024-04-14T15:15:04.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:03 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:04.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:03 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3774280749' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:15:04.244 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:15:04.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:04.269 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:04.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:15:04.370 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:15:04.394 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:04.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:04.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:15:04.495 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:15:04.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:04.521 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:04.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:15:04.622 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:15:04.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:04.647 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:04.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:15:04.749 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:15:04.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:04.774 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:04.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:15:04.875 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:15:04.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:04.907 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:04.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:15:04.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:04 smithi090 ceph-mon[41670]: pgmap v299: 284 pgs: 1 clean+premerge+peered, 3 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 276 active+clean; 586 KiB data, 540 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 1 op/s 2024-04-14T15:15:04.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:04 smithi090 ceph-mon[41670]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:15:04.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:04 smithi090 ceph-mon[41670]: from='client.48565 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi090-93342-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi090-93342-34"}]': finished 2024-04-14T15:15:04.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:04 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-23"}]': finished 2024-04-14T15:15:04.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:04 smithi090 ceph-mon[41670]: osdmap e236: 8 total, 8 up, 8 in 2024-04-14T15:15:04.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:04 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:04.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:04 smithi090 ceph-mon[30399]: pgmap v299: 284 pgs: 1 clean+premerge+peered, 3 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 276 active+clean; 586 KiB data, 540 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 1 op/s 2024-04-14T15:15:04.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:04 smithi090 ceph-mon[30399]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:15:04.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:04 smithi090 ceph-mon[30399]: from='client.48565 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi090-93342-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi090-93342-34"}]': finished 2024-04-14T15:15:04.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:04 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-23"}]': finished 2024-04-14T15:15:04.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:04 smithi090 ceph-mon[30399]: osdmap e236: 8 total, 8 up, 8 in 2024-04-14T15:15:04.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:04 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:05.008 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:15:05.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:05.033 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:05.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:15:05.122 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:15:04 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:15:05.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:04 smithi092 ceph-mon[33966]: pgmap v299: 284 pgs: 1 clean+premerge+peered, 3 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 276 active+clean; 586 KiB data, 540 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 1 op/s 2024-04-14T15:15:05.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:04 smithi092 ceph-mon[33966]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:15:05.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:04 smithi092 ceph-mon[33966]: from='client.48565 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi090-93342-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi090-93342-34"}]': finished 2024-04-14T15:15:05.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:04 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-23"}]': finished 2024-04-14T15:15:05.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:04 smithi092 ceph-mon[33966]: osdmap e236: 8 total, 8 up, 8 in 2024-04-14T15:15:05.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:04 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:05.134 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:15:05.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:05.159 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:05.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:15:05.221 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-14T15:15:05.260 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:15:05.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:05.284 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:05.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:15:05.386 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:15:05.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:05.412 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:05.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:15:05.513 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:15:05.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:05.537 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:05.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:15:05.638 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:15:05.663 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:05.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:05.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:15:05.765 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:15:05.790 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:05.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:05.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:15:05.892 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:15:05.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:05.916 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:05.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:15:06.017 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:15:06.039 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:05 smithi090 ceph-mon[41670]: osdmap e237: 8 total, 8 up, 8 in 2024-04-14T15:15:06.039 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:05 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/32297493' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi090-93363-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:06.039 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:05 smithi090 ceph-mon[41670]: from='client.48577 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi090-93363-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:06.039 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:05 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:15:06.039 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:05 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:06.040 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:05 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-04-14T15:15:06.040 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:05 smithi090 ceph-mon[41670]: from='client.48577 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi090-93363-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:15:06.040 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:05 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "12"}]': finished 2024-04-14T15:15:06.040 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:05 smithi090 ceph-mon[41670]: osdmap e238: 8 total, 8 up, 8 in 2024-04-14T15:15:06.040 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:05 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:06.040 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:05 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:06.040 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:05 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3767512710' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi090-93342-34"}]: dispatch 2024-04-14T15:15:06.041 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:05 smithi090 ceph-mon[41670]: from='client.48565 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi090-93342-34"}]: dispatch 2024-04-14T15:15:06.041 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:05 smithi090 ceph-mon[30399]: osdmap e237: 8 total, 8 up, 8 in 2024-04-14T15:15:06.041 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:05 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/32297493' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi090-93363-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:06.041 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:05 smithi090 ceph-mon[30399]: from='client.48577 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi090-93363-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:06.042 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:05 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:15:06.042 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:05 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:06.042 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:05 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-04-14T15:15:06.042 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:05 smithi090 ceph-mon[30399]: from='client.48577 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi090-93363-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:15:06.042 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:05 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "12"}]': finished 2024-04-14T15:15:06.042 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:05 smithi090 ceph-mon[30399]: osdmap e238: 8 total, 8 up, 8 in 2024-04-14T15:15:06.043 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:05 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:06.043 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:05 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:06.043 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:05 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3767512710' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi090-93342-34"}]: dispatch 2024-04-14T15:15:06.043 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:05 smithi090 ceph-mon[30399]: from='client.48565 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi090-93342-34"}]: dispatch 2024-04-14T15:15:06.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:06.045 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:06.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:15:06.146 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:15:06.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:06.172 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:06.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:15:06.273 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:15:06.280 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:15:06.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:06.299 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:06.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:15:06.306 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-14T15:15:06.307 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-14T15:15:06.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-14T15:15:06.307 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-14T15:15:06.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.307 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.14154.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-04-14T15:15:06.307 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:07:25.616096+0000", 2024-04-14T15:15:06.307 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 460.68759518799999, 2024-04-14T15:15:06.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.28991978400000001, 2024-04-14T15:15:06.307 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:15:06.307 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:15:06.307 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:15:06.307 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:15:06.308 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.14154", 2024-04-14T15:15:06.308 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/508768479", 2024-04-14T15:15:06.308 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1 2024-04-14T15:15:06.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.308 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:15:06.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.308 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:15:06.308 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:07:25.616096+0000", 2024-04-14T15:15:06.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:15:06.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.308 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:15:06.308 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:07:25.616096+0000", 2024-04-14T15:15:06.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:15:06.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.309 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:15:06.309 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:07:25.616098+0000", 2024-04-14T15:15:06.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2740000000000002e-06 2024-04-14T15:15:06.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.309 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:15:06.309 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:07:25.616108+0000", 2024-04-14T15:15:06.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9059999999999997e-06 2024-04-14T15:15:06.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.309 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:15:06.309 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:07:25.616119+0000", 2024-04-14T15:15:06.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0844e-05 2024-04-14T15:15:06.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.310 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.310 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:15:06.310 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:07:25.616129+0000", 2024-04-14T15:15:06.310 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8190000000000003e-06 2024-04-14T15:15:06.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.310 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.310 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:15:06.310 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:07:25.616189+0000", 2024-04-14T15:15:06.310 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0136999999999997e-05 2024-04-14T15:15:06.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.310 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.310 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:15:06.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:07:25.616197+0000", 2024-04-14T15:15:06.311 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9039999999999992e-06 2024-04-14T15:15:06.311 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:15:06.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:07:25.904871+0000", 2024-04-14T15:15:06.311 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.28867436800000001 2024-04-14T15:15:06.311 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:15:06.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:07:25.904985+0000", 2024-04-14T15:15:06.311 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011321 2024-04-14T15:15:06.311 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,2", 2024-04-14T15:15:06.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:07:25.905132+0000", 2024-04-14T15:15:06.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014694299999999999 2024-04-14T15:15:06.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:15:06.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:07:25.905772+0000", 2024-04-14T15:15:06.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064086600000000003 2024-04-14T15:15:06.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:15:06.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:07:25.905854+0000", 2024-04-14T15:15:06.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2088000000000004e-05 2024-04-14T15:15:06.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.313 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:15:06.313 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:07:25.905907+0000", 2024-04-14T15:15:06.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2966000000000001e-05 2024-04-14T15:15:06.313 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.313 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.313 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:15:06.313 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:07:25.905966+0000", 2024-04-14T15:15:06.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8570000000000003e-05 2024-04-14T15:15:06.313 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.313 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.313 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:15:06.313 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:07:25.906016+0000", 2024-04-14T15:15:06.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9889000000000003e-05 2024-04-14T15:15:06.313 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:15:06.313 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:15:06.314 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:15:06.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.314 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24928.0:2 24.1a 24:5c6776db:::0.obj:head [write 0~103] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-04-14T15:15:06.314 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:12:55.060209+0000", 2024-04-14T15:15:06.314 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 131.24348207200001, 2024-04-14T15:15:06.314 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.508190039, 2024-04-14T15:15:06.314 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:15:06.314 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:15:06.314 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:15:06.314 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:15:06.314 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24928", 2024-04-14T15:15:06.314 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/346751907", 2024-04-14T15:15:06.314 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2 2024-04-14T15:15:06.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.315 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:15:06.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:15:06.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.060209+0000", 2024-04-14T15:15:06.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:15:06.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:15:06.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.060209+0000", 2024-04-14T15:15:06.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:15:06.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.316 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:15:06.316 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.060211+0000", 2024-04-14T15:15:06.316 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0090000000000002e-06 2024-04-14T15:15:06.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.317 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:15:06.317 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.060220+0000", 2024-04-14T15:15:06.317 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7870000000000007e-06 2024-04-14T15:15:06.317 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.317 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:15:06.317 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.060231+0000", 2024-04-14T15:15:06.317 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1317e-05 2024-04-14T15:15:06.317 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.317 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:15:06.317 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.060243+0000", 2024-04-14T15:15:06.317 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.13e-05 2024-04-14T15:15:06.317 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.318 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:15:06.318 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.107072+0000", 2024-04-14T15:15:06.318 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.046829279000000001 2024-04-14T15:15:06.318 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.318 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.318 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:15:06.318 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.107080+0000", 2024-04-14T15:15:06.318 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4400000000000005e-06 2024-04-14T15:15:06.318 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.318 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.318 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:15:06.318 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.502476+0000", 2024-04-14T15:15:06.318 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.395395623 2024-04-14T15:15:06.318 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.319 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:15:06.319 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.502603+0000", 2024-04-14T15:15:06.319 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000127508 2024-04-14T15:15:06.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.319 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,3", 2024-04-14T15:15:06.319 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.502746+0000", 2024-04-14T15:15:06.319 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000142817 2024-04-14T15:15:06.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.319 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:15:06.319 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.514091+0000", 2024-04-14T15:15:06.319 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011344468999999999 2024-04-14T15:15:06.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.320 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:15:06.320 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.559343+0000", 2024-04-14T15:15:06.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.045251922999999999 2024-04-14T15:15:06.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.320 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:15:06.320 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.568307+0000", 2024-04-14T15:15:06.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0089640250000000005 2024-04-14T15:15:06.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.320 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:15:06.320 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.568354+0000", 2024-04-14T15:15:06.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7246000000000001e-05 2024-04-14T15:15:06.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.321 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:15:06.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.568399+0000", 2024-04-14T15:15:06.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5296000000000001e-05 2024-04-14T15:15:06.321 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:15:06.321 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:15:06.321 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:15:06.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.321 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24965.0:2 30.1a 30:5c6776db:::0.obj:head [write 0~103] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-04-14T15:15:06.321 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:12:55.063803+0000", 2024-04-14T15:15:06.321 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 131.23988866100001, 2024-04-14T15:15:06.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4453814549999999, 2024-04-14T15:15:06.321 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:15:06.322 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:15:06.322 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:15:06.322 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:15:06.322 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24965", 2024-04-14T15:15:06.322 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1840281572", 2024-04-14T15:15:06.322 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2 2024-04-14T15:15:06.322 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.322 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:15:06.322 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.322 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:15:06.322 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.063803+0000", 2024-04-14T15:15:06.322 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:15:06.322 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.322 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:15:06.323 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.063803+0000", 2024-04-14T15:15:06.323 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:15:06.323 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.323 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:15:06.323 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.063805+0000", 2024-04-14T15:15:06.323 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4990000000000001e-06 2024-04-14T15:15:06.323 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.323 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:15:06.323 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.063813+0000", 2024-04-14T15:15:06.323 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0299999999999994e-06 2024-04-14T15:15:06.323 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.323 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.324 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:15:06.324 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.063826+0000", 2024-04-14T15:15:06.324 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2480000000000001e-05 2024-04-14T15:15:06.324 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.324 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.324 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:15:06.324 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.063837+0000", 2024-04-14T15:15:06.324 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.119e-05 2024-04-14T15:15:06.324 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.324 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.324 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:15:06.324 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.120883+0000", 2024-04-14T15:15:06.324 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.057046615000000002 2024-04-14T15:15:06.324 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.324 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.325 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:15:06.325 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.120892+0000", 2024-04-14T15:15:06.325 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2530000000000004e-06 2024-04-14T15:15:06.325 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.325 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.325 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:15:06.325 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.446585+0000", 2024-04-14T15:15:06.325 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.325693403 2024-04-14T15:15:06.325 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.325 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.325 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:15:06.325 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.446706+0000", 2024-04-14T15:15:06.325 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000121396 2024-04-14T15:15:06.325 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.325 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.326 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,5", 2024-04-14T15:15:06.326 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.446847+0000", 2024-04-14T15:15:06.326 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014026300000000001 2024-04-14T15:15:06.326 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.326 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.326 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:15:06.326 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.479690+0000", 2024-04-14T15:15:06.326 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.032843226000000003 2024-04-14T15:15:06.326 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.326 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.326 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:15:06.326 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.493251+0000", 2024-04-14T15:15:06.326 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013560588 2024-04-14T15:15:06.327 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.327 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:15:06.327 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.509099+0000", 2024-04-14T15:15:06.327 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015848240999999999 2024-04-14T15:15:06.327 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.327 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:15:06.327 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.509147+0000", 2024-04-14T15:15:06.327 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8028999999999997e-05 2024-04-14T15:15:06.327 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.327 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:15:06.327 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.509184+0000", 2024-04-14T15:15:06.327 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7242e-05 2024-04-14T15:15:06.328 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:15:06.328 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:15:06.328 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:15:06.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.328 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24860.0:2 21.d 21:b2a9e82d:test-rados-api-smithi090-93864-2::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-04-14T15:15:06.328 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:12:56.070980+0000", 2024-04-14T15:15:06.328 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 130.23271121900001, 2024-04-14T15:15:06.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.42151502099999999, 2024-04-14T15:15:06.328 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:15:06.328 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:15:06.328 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:15:06.328 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:15:06.328 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24860", 2024-04-14T15:15:06.328 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/766142007", 2024-04-14T15:15:06.329 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2 2024-04-14T15:15:06.329 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.329 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:15:06.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.329 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:15:06.329 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.070980+0000", 2024-04-14T15:15:06.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:15:06.329 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.329 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:15:06.329 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.070980+0000", 2024-04-14T15:15:06.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:15:06.329 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.329 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:15:06.330 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.070982+0000", 2024-04-14T15:15:06.330 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7099999999999999e-06 2024-04-14T15:15:06.330 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.330 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:15:06.330 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.071000+0000", 2024-04-14T15:15:06.330 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8391999999999999e-05 2024-04-14T15:15:06.330 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.330 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:15:06.330 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.071010+0000", 2024-04-14T15:15:06.330 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4020000000000008e-06 2024-04-14T15:15:06.330 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.330 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:15:06.331 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.071027+0000", 2024-04-14T15:15:06.331 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7200000000000001e-05 2024-04-14T15:15:06.331 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.331 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.331 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:15:06.331 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.187255+0000", 2024-04-14T15:15:06.332 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.116227975 2024-04-14T15:15:06.332 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.332 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.332 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:15:06.332 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.187264+0000", 2024-04-14T15:15:06.332 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0750000000000004e-06 2024-04-14T15:15:06.332 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.332 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:15:06.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.464111+0000", 2024-04-14T15:15:06.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.27684678200000001 2024-04-14T15:15:06.333 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:15:06.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.464239+0000", 2024-04-14T15:15:06.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012847999999999999 2024-04-14T15:15:06.333 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,6", 2024-04-14T15:15:06.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.464380+0000", 2024-04-14T15:15:06.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000141209 2024-04-14T15:15:06.333 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.334 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:15:06.334 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.469522+0000", 2024-04-14T15:15:06.334 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0051416109999999999 2024-04-14T15:15:06.334 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.334 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.334 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:15:06.334 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.473583+0000", 2024-04-14T15:15:06.334 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040609499999999998 2024-04-14T15:15:06.334 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.334 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.334 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:15:06.334 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.492397+0000", 2024-04-14T15:15:06.336 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.018814000000000001 2024-04-14T15:15:06.336 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.336 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.336 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:15:06.336 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.492454+0000", 2024-04-14T15:15:06.336 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7510000000000003e-05 2024-04-14T15:15:06.336 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.336 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.336 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:15:06.336 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.492495+0000", 2024-04-14T15:15:06.336 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0725000000000001e-05 2024-04-14T15:15:06.336 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:15:06.336 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:15:06.337 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:15:06.337 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.337 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.337 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.14955.0:2 19.1 19:8584e8fe:test-rados-api-smithi090-93731-2::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-04-14T15:15:06.337 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:12:56.109199+0000", 2024-04-14T15:15:06.337 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 130.19449202999999, 2024-04-14T15:15:06.337 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.44945317499999998, 2024-04-14T15:15:06.337 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:15:06.337 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:15:06.337 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:15:06.337 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:15:06.337 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.14955", 2024-04-14T15:15:06.337 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2763816171", 2024-04-14T15:15:06.337 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2 2024-04-14T15:15:06.337 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.338 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:15:06.338 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.338 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:15:06.338 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.109199+0000", 2024-04-14T15:15:06.338 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:15:06.338 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.338 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.338 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:15:06.338 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.109199+0000", 2024-04-14T15:15:06.338 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:15:06.338 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.338 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.338 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:15:06.338 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.109201+0000", 2024-04-14T15:15:06.339 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8679999999999999e-06 2024-04-14T15:15:06.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.339 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.339 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:15:06.339 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.109209+0000", 2024-04-14T15:15:06.339 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1820000000000006e-06 2024-04-14T15:15:06.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.339 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.340 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:15:06.340 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.109223+0000", 2024-04-14T15:15:06.340 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4149e-05 2024-04-14T15:15:06.340 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.341 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.341 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:15:06.341 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.109234+0000", 2024-04-14T15:15:06.341 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0883999999999999e-05 2024-04-14T15:15:06.341 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.341 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.341 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:15:06.341 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.109301+0000", 2024-04-14T15:15:06.341 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7136000000000002e-05 2024-04-14T15:15:06.341 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.341 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.341 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:15:06.341 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.109311+0000", 2024-04-14T15:15:06.341 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4269999999999997e-06 2024-04-14T15:15:06.342 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.342 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.342 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:15:06.342 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.476527+0000", 2024-04-14T15:15:06.342 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.36721647699999999 2024-04-14T15:15:06.342 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.342 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.342 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:15:06.342 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.476630+0000", 2024-04-14T15:15:06.342 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000102748 2024-04-14T15:15:06.342 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.342 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.342 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,7", 2024-04-14T15:15:06.342 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.476756+0000", 2024-04-14T15:15:06.343 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000125489 2024-04-14T15:15:06.343 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.343 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.343 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:15:06.343 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.493558+0000", 2024-04-14T15:15:06.343 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016802699000000001 2024-04-14T15:15:06.343 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.343 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.343 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:15:06.343 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.495245+0000", 2024-04-14T15:15:06.343 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016863609999999999 2024-04-14T15:15:06.343 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.343 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.343 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:15:06.343 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.558554+0000", 2024-04-14T15:15:06.344 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.063309655000000006 2024-04-14T15:15:06.344 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.344 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.344 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:15:06.344 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.558606+0000", 2024-04-14T15:15:06.344 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2160000000000002e-05 2024-04-14T15:15:06.344 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.344 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.344 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:15:06.344 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.558652+0000", 2024-04-14T15:15:06.345 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5939999999999997e-05 2024-04-14T15:15:06.345 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:15:06.345 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:15:06.345 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:15:06.345 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.345 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.346 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.25030.0:2 35.6 e108/107 35:602f83fe:::foo:head v 108'1, mlcod=0'0)", 2024-04-14T15:15:06.346 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:12:56.452112+0000", 2024-04-14T15:15:06.346 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 129.85157964499999, 2024-04-14T15:15:06.346 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.061498231, 2024-04-14T15:15:06.346 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:15:06.346 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:15:06.346 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:15:06.346 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:15:06.346 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.346 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:15:06.346 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.452112+0000", 2024-04-14T15:15:06.346 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:15:06.346 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.346 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.347 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:15:06.347 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.452112+0000", 2024-04-14T15:15:06.347 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:15:06.347 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.347 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.347 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:15:06.347 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.452112+0000", 2024-04-14T15:15:06.347 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5100000000000001e-07 2024-04-14T15:15:06.347 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.347 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.347 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:15:06.347 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.452118+0000", 2024-04-14T15:15:06.347 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7339999999999996e-06 2024-04-14T15:15:06.347 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.347 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.348 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:15:06.348 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.452123+0000", 2024-04-14T15:15:06.348 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1379999999999998e-06 2024-04-14T15:15:06.348 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.348 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.348 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:15:06.348 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.452131+0000", 2024-04-14T15:15:06.348 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4290000000000001e-06 2024-04-14T15:15:06.348 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.348 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.348 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:15:06.348 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.497771+0000", 2024-04-14T15:15:06.348 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.045639800000000001 2024-04-14T15:15:06.348 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.348 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.349 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:15:06.349 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.497790+0000", 2024-04-14T15:15:06.349 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9224000000000001e-05 2024-04-14T15:15:06.349 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.349 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.349 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:15:06.349 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.513569+0000", 2024-04-14T15:15:06.349 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015779202999999999 2024-04-14T15:15:06.349 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.349 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.349 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:15:06.349 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.513610+0000", 2024-04-14T15:15:06.349 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0451999999999998e-05 2024-04-14T15:15:06.349 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:15:06.349 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:15:06.350 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:15:06.350 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.350 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.350 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24955.0:3 26.f 26:f7483bc4:::src_object.2:head [writefull 0~4096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-04-14T15:15:06.350 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:12:56.472685+0000", 2024-04-14T15:15:06.350 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 129.83100611, 2024-04-14T15:15:06.350 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.086134901999999999, 2024-04-14T15:15:06.350 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:15:06.350 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:15:06.350 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:15:06.350 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:15:06.350 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24955", 2024-04-14T15:15:06.350 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/117387996", 2024-04-14T15:15:06.350 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3 2024-04-14T15:15:06.350 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.351 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:15:06.351 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.351 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:15:06.351 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.472685+0000", 2024-04-14T15:15:06.351 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:15:06.351 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.351 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.351 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:15:06.351 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.472685+0000", 2024-04-14T15:15:06.351 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:15:06.351 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.351 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.351 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:15:06.351 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.472687+0000", 2024-04-14T15:15:06.352 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8300000000000001e-06 2024-04-14T15:15:06.352 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.352 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.352 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:15:06.352 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.472697+0000", 2024-04-14T15:15:06.352 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8190000000000003e-06 2024-04-14T15:15:06.352 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.352 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.352 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:15:06.352 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.472705+0000", 2024-04-14T15:15:06.352 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8299999999999996e-06 2024-04-14T15:15:06.352 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.352 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.352 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:15:06.353 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.472711+0000", 2024-04-14T15:15:06.353 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5409999999999997e-06 2024-04-14T15:15:06.353 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.353 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.353 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:15:06.353 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.475204+0000", 2024-04-14T15:15:06.353 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0024926380000000001 2024-04-14T15:15:06.353 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.353 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.353 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:15:06.353 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.475324+0000", 2024-04-14T15:15:06.353 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000119669 2024-04-14T15:15:06.353 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.353 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.354 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,7", 2024-04-14T15:15:06.354 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.475458+0000", 2024-04-14T15:15:06.354 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013436200000000001 2024-04-14T15:15:06.354 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.354 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.354 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:15:06.354 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.486202+0000", 2024-04-14T15:15:06.354 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010744448 2024-04-14T15:15:06.354 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.354 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.354 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:15:06.354 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.494871+0000", 2024-04-14T15:15:06.354 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0086686590000000004 2024-04-14T15:15:06.355 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.355 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.355 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:15:06.355 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.558724+0000", 2024-04-14T15:15:06.355 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.063852781999999997 2024-04-14T15:15:06.355 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.355 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.355 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:15:06.355 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.558777+0000", 2024-04-14T15:15:06.355 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2805e-05 2024-04-14T15:15:06.355 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.355 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.355 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:15:06.355 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.558820+0000", 2024-04-14T15:15:06.355 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3519000000000001e-05 2024-04-14T15:15:06.356 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:15:06.356 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:15:06.356 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:15:06.356 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.356 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.356 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.24853.0:433 16.1b e108/107 16:dbd443e1:test-rados-api-smithi090-93601-15::foo:head v 108'194, mlcod=108'192)", 2024-04-14T15:15:06.356 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:12:57.109500+0000", 2024-04-14T15:15:06.356 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 129.19419176700001, 2024-04-14T15:15:06.356 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.097327853000000006, 2024-04-14T15:15:06.356 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:15:06.356 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:15:06.356 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:15:06.356 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:15:06.356 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.356 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:15:06.357 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.109500+0000", 2024-04-14T15:15:06.357 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:15:06.357 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.357 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.357 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:15:06.357 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.109500+0000", 2024-04-14T15:15:06.357 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:15:06.357 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.357 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.357 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:15:06.357 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.109500+0000", 2024-04-14T15:15:06.357 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1199999999999999e-07 2024-04-14T15:15:06.357 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.357 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.358 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:15:06.358 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.109506+0000", 2024-04-14T15:15:06.358 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0179999999999998e-06 2024-04-14T15:15:06.358 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.358 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.358 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:15:06.358 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.109511+0000", 2024-04-14T15:15:06.358 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7609999999999998e-06 2024-04-14T15:15:06.358 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.358 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.358 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:15:06.358 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.109520+0000", 2024-04-14T15:15:06.358 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9449999999999999e-06 2024-04-14T15:15:06.358 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.359 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.359 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:15:06.359 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.109898+0000", 2024-04-14T15:15:06.359 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037795899999999998 2024-04-14T15:15:06.359 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.359 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.359 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:15:06.359 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.109922+0000", 2024-04-14T15:15:06.359 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4073000000000001e-05 2024-04-14T15:15:06.359 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.359 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.359 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:15:06.359 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.206793+0000", 2024-04-14T15:15:06.359 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.096871041000000005 2024-04-14T15:15:06.360 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.360 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.360 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:15:06.360 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.206827+0000", 2024-04-14T15:15:06.360 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4743999999999998e-05 2024-04-14T15:15:06.360 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:15:06.360 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:15:06.360 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:15:06.360 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.360 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.360 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.14973.0:1 42.1s0 42:8981422e:test-rados-api-smithi090-93998-2::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-04-14T15:15:06.360 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:12:57.328942+0000", 2024-04-14T15:15:06.360 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 128.974749618, 2024-04-14T15:15:06.360 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.87141143300000001, 2024-04-14T15:15:06.360 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:15:06.361 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:15:06.361 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:15:06.361 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:15:06.361 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.14973", 2024-04-14T15:15:06.361 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/309942895", 2024-04-14T15:15:06.361 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1 2024-04-14T15:15:06.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.361 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:15:06.361 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.361 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:15:06.361 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.328942+0000", 2024-04-14T15:15:06.361 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:15:06.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.361 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.361 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:15:06.362 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.328942+0000", 2024-04-14T15:15:06.362 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:15:06.362 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.362 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.362 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:15:06.362 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.328943+0000", 2024-04-14T15:15:06.362 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6819999999999999e-06 2024-04-14T15:15:06.362 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.362 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.362 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:15:06.362 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.328949+0000", 2024-04-14T15:15:06.362 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6470000000000003e-06 2024-04-14T15:15:06.362 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.362 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.363 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:15:06.363 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.328957+0000", 2024-04-14T15:15:06.363 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2439999999999993e-06 2024-04-14T15:15:06.363 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.363 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.363 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:15:06.363 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.328965+0000", 2024-04-14T15:15:06.363 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7989999999999994e-06 2024-04-14T15:15:06.363 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.363 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.363 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:15:06.363 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.329261+0000", 2024-04-14T15:15:06.363 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029585899999999999 2024-04-14T15:15:06.363 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.363 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.364 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:15:06.364 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.329265+0000", 2024-04-14T15:15:06.364 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9339999999999999e-06 2024-04-14T15:15:06.364 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.364 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.364 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:15:06.364 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.196185+0000", 2024-04-14T15:15:06.364 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.86692026799999999 2024-04-14T15:15:06.364 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.364 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.364 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:15:06.364 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.196319+0000", 2024-04-14T15:15:06.364 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013423299999999999 2024-04-14T15:15:06.364 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.365 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.365 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2024-04-14T15:15:06.365 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.197772+0000", 2024-04-14T15:15:06.365 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001452395 2024-04-14T15:15:06.365 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.365 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.365 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2024-04-14T15:15:06.365 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.198662+0000", 2024-04-14T15:15:06.365 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00089075099999999996 2024-04-14T15:15:06.365 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.365 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.365 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:15:06.365 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.200283+0000", 2024-04-14T15:15:06.366 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016206040000000001 2024-04-14T15:15:06.366 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.366 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.366 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:15:06.366 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.200353+0000", 2024-04-14T15:15:06.366 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0017000000000004e-05 2024-04-14T15:15:06.366 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:15:06.366 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:15:06.366 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:15:06.366 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.366 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.366 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.24853.0:1853 16.1b e110/107 16:dbd443e1:test-rados-api-smithi090-93601-15::foo:head v 110'904, mlcod=110'902)", 2024-04-14T15:15:06.366 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:12:58.610337+0000", 2024-04-14T15:15:06.366 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 127.693354324, 2024-04-14T15:15:06.366 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.067447942999999996, 2024-04-14T15:15:06.367 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:15:06.367 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:15:06.367 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:15:06.367 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:15:06.367 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.367 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:15:06.367 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.610337+0000", 2024-04-14T15:15:06.367 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:15:06.367 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.367 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.367 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:15:06.367 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.610337+0000", 2024-04-14T15:15:06.367 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:15:06.367 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.368 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.368 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:15:06.368 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.610338+0000", 2024-04-14T15:15:06.368 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9700000000000005e-07 2024-04-14T15:15:06.368 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.368 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.368 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:15:06.368 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.610342+0000", 2024-04-14T15:15:06.368 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1280000000000001e-06 2024-04-14T15:15:06.368 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.368 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.368 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:15:06.368 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.610348+0000", 2024-04-14T15:15:06.368 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4679999999999997e-06 2024-04-14T15:15:06.369 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.369 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.369 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:15:06.369 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.610352+0000", 2024-04-14T15:15:06.369 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8809999999999998e-06 2024-04-14T15:15:06.369 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.369 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.369 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:15:06.369 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.610377+0000", 2024-04-14T15:15:06.369 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5080999999999999e-05 2024-04-14T15:15:06.369 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.369 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.369 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:15:06.369 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.610388+0000", 2024-04-14T15:15:06.369 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0139000000000001e-05 2024-04-14T15:15:06.370 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.370 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.370 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:15:06.370 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.610725+0000", 2024-04-14T15:15:06.370 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000337502 2024-04-14T15:15:06.370 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.370 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.370 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:15:06.370 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.677785+0000", 2024-04-14T15:15:06.370 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.067059847000000006 2024-04-14T15:15:06.370 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:15:06.370 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:15:06.370 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:15:06.370 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.371 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.371 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.24853.0:2059 16.1 e110/107 16:85b90e3d:test-rados-api-smithi090-93601-16::big:head v 110'7, mlcod=108'5)", 2024-04-14T15:15:06.371 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:12:58.846275+0000", 2024-04-14T15:15:06.371 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 127.45741639000001, 2024-04-14T15:15:06.371 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.060809294, 2024-04-14T15:15:06.371 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:15:06.371 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:15:06.371 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:15:06.371 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:15:06.371 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.371 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:15:06.371 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.846275+0000", 2024-04-14T15:15:06.371 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:15:06.371 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.372 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.372 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:15:06.372 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.846275+0000", 2024-04-14T15:15:06.372 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:15:06.372 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.372 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.372 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:15:06.372 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.846278+0000", 2024-04-14T15:15:06.372 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9840000000000001e-06 2024-04-14T15:15:06.372 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.372 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.372 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:15:06.372 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.860763+0000", 2024-04-14T15:15:06.372 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014485345 2024-04-14T15:15:06.373 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.373 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.373 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:15:06.373 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.860787+0000", 2024-04-14T15:15:06.373 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3428999999999999e-05 2024-04-14T15:15:06.373 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.373 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.373 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:15:06.373 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.860797+0000", 2024-04-14T15:15:06.373 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8500000000000006e-06 2024-04-14T15:15:06.373 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.373 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.373 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:15:06.373 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.860864+0000", 2024-04-14T15:15:06.374 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7710000000000001e-05 2024-04-14T15:15:06.374 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.374 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.374 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:15:06.374 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.860885+0000", 2024-04-14T15:15:06.374 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0696999999999999e-05 2024-04-14T15:15:06.374 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.374 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.374 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:15:06.374 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.907037+0000", 2024-04-14T15:15:06.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.046151615 2024-04-14T15:15:06.376 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.376 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.376 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:15:06.376 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.907084+0000", 2024-04-14T15:15:06.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7664000000000003e-05 2024-04-14T15:15:06.376 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:15:06.376 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:15:06.376 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:15:06.376 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.376 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.376 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25258.0:2 58.15 58:ad551702:::test:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-14T15:15:06.377 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:12:59.091942+0000", 2024-04-14T15:15:06.377 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 127.21174898300001, 2024-04-14T15:15:06.377 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.178728945, 2024-04-14T15:15:06.377 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:15:06.377 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:15:06.377 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:15:06.377 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:15:06.377 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25258", 2024-04-14T15:15:06.377 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1067094721", 2024-04-14T15:15:06.377 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2 2024-04-14T15:15:06.377 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.377 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:15:06.377 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.377 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:15:06.378 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.091942+0000", 2024-04-14T15:15:06.378 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:15:06.378 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.378 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.378 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:15:06.378 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.091942+0000", 2024-04-14T15:15:06.378 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:15:06.378 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.378 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.378 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:15:06.378 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.091944+0000", 2024-04-14T15:15:06.378 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.784e-06 2024-04-14T15:15:06.378 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.378 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.378 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:15:06.379 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.091949+0000", 2024-04-14T15:15:06.379 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2510000000000002e-06 2024-04-14T15:15:06.379 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.379 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.379 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:15:06.379 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.091960+0000", 2024-04-14T15:15:06.379 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0958000000000001e-05 2024-04-14T15:15:06.379 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.379 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.379 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:15:06.379 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.091967+0000", 2024-04-14T15:15:06.379 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8199999999999999e-06 2024-04-14T15:15:06.379 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.379 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.380 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:15:06.380 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.092028+0000", 2024-04-14T15:15:06.380 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0875000000000002e-05 2024-04-14T15:15:06.380 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.380 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.380 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2024-04-14T15:15:06.380 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.092036+0000", 2024-04-14T15:15:06.380 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9410000000000007e-06 2024-04-14T15:15:06.380 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.380 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.380 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:15:06.380 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.119696+0000", 2024-04-14T15:15:06.380 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.027659777 2024-04-14T15:15:06.380 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.380 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.381 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:15:06.381 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.119705+0000", 2024-04-14T15:15:06.381 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0280000000000006e-06 2024-04-14T15:15:06.381 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.381 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.381 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:15:06.381 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.252689+0000", 2024-04-14T15:15:06.381 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.132983924 2024-04-14T15:15:06.381 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.381 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.381 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:15:06.381 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.252798+0000", 2024-04-14T15:15:06.381 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000109275 2024-04-14T15:15:06.381 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.382 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.382 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4,5", 2024-04-14T15:15:06.382 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.252921+0000", 2024-04-14T15:15:06.382 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000122583 2024-04-14T15:15:06.382 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.382 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.382 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:15:06.382 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.257824+0000", 2024-04-14T15:15:06.382 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0049038320000000003 2024-04-14T15:15:06.382 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.382 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.382 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:15:06.382 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.269059+0000", 2024-04-14T15:15:06.382 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011234545 2024-04-14T15:15:06.383 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.383 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.383 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:15:06.383 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.270587+0000", 2024-04-14T15:15:06.383 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001528223 2024-04-14T15:15:06.383 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.383 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.383 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:15:06.383 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.270631+0000", 2024-04-14T15:15:06.383 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3869999999999998e-05 2024-04-14T15:15:06.383 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.383 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.383 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:15:06.383 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.270671+0000", 2024-04-14T15:15:06.384 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0259000000000002e-05 2024-04-14T15:15:06.384 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:15:06.384 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:15:06.384 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:15:06.384 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.384 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.384 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.15264.0:3 64.14 64:2a647cd4:::obj0:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-14T15:15:06.384 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:12:59.285619+0000", 2024-04-14T15:15:06.384 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 127.018071972, 2024-04-14T15:15:06.384 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99102734800000003, 2024-04-14T15:15:06.384 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:15:06.384 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:15:06.384 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-04-14T15:15:06.384 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:15:06.385 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.15264", 2024-04-14T15:15:06.385 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1337739815", 2024-04-14T15:15:06.385 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3 2024-04-14T15:15:06.385 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.385 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:15:06.385 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.385 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:15:06.385 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.285619+0000", 2024-04-14T15:15:06.385 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:15:06.385 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.385 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.385 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:15:06.385 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.285619+0000", 2024-04-14T15:15:06.385 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:15:06.386 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.386 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.386 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:15:06.386 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.285621+0000", 2024-04-14T15:15:06.386 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.339e-06 2024-04-14T15:15:06.386 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.386 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.386 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:15:06.386 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.285625+0000", 2024-04-14T15:15:06.386 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4429999999999996e-06 2024-04-14T15:15:06.386 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.386 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.386 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:15:06.386 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.285636+0000", 2024-04-14T15:15:06.387 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0769e-05 2024-04-14T15:15:06.387 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.387 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.387 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:15:06.387 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.285644+0000", 2024-04-14T15:15:06.387 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5029999999999997e-06 2024-04-14T15:15:06.387 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.387 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.387 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:15:06.387 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.286065+0000", 2024-04-14T15:15:06.387 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00042017999999999998 2024-04-14T15:15:06.387 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.387 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.387 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:15:06.388 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.286072+0000", 2024-04-14T15:15:06.388 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6000000000000001e-06 2024-04-14T15:15:06.388 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.388 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.388 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:15:06.388 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.276569+0000", 2024-04-14T15:15:06.388 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99049650199999995 2024-04-14T15:15:06.388 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.388 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.388 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:15:06.388 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.276647+0000", 2024-04-14T15:15:06.388 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8011999999999998e-05 2024-04-14T15:15:06.388 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:15:06.388 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:15:06.389 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:15:06.389 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.389 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.389 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24871.0:8 18.1c 18:3cd39c19:test-rados-api-smithi090-93709-4::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-04-14T15:15:06.389 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:13:00.205084+0000", 2024-04-14T15:15:06.389 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 126.09860696600001, 2024-04-14T15:15:06.389 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.071432922999999995, 2024-04-14T15:15:06.389 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:15:06.389 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:15:06.389 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:15:06.389 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:15:06.389 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24871", 2024-04-14T15:15:06.389 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1680088077", 2024-04-14T15:15:06.389 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 8 2024-04-14T15:15:06.390 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.390 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:15:06.390 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.390 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:15:06.390 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.205084+0000", 2024-04-14T15:15:06.390 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:15:06.390 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.390 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.390 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:15:06.390 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.205084+0000", 2024-04-14T15:15:06.390 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:15:06.390 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.390 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.390 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:15:06.390 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.205087+0000", 2024-04-14T15:15:06.391 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3829999999999999e-06 2024-04-14T15:15:06.391 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.391 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.391 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:15:06.391 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.205096+0000", 2024-04-14T15:15:06.391 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5259999999999999e-06 2024-04-14T15:15:06.391 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.391 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.391 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:15:06.391 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.205109+0000", 2024-04-14T15:15:06.391 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2473000000000001e-05 2024-04-14T15:15:06.391 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.391 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.391 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:15:06.391 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.205119+0000", 2024-04-14T15:15:06.392 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.077e-05 2024-04-14T15:15:06.392 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.392 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.392 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:15:06.392 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.206813+0000", 2024-04-14T15:15:06.392 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001693684 2024-04-14T15:15:06.392 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.392 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.392 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:15:06.392 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.206943+0000", 2024-04-14T15:15:06.392 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129667 2024-04-14T15:15:06.392 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.392 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.392 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,7", 2024-04-14T15:15:06.392 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.210425+0000", 2024-04-14T15:15:06.393 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0034817469999999999 2024-04-14T15:15:06.393 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.393 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.393 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:15:06.393 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.212326+0000", 2024-04-14T15:15:06.393 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0019013420000000001 2024-04-14T15:15:06.393 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.393 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.393 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:15:06.393 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.255071+0000", 2024-04-14T15:15:06.393 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.042745584000000003 2024-04-14T15:15:06.393 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.393 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.394 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:15:06.394 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.276425+0000", 2024-04-14T15:15:06.394 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.021353556999999999 2024-04-14T15:15:06.394 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.394 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.394 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:15:06.394 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.276476+0000", 2024-04-14T15:15:06.394 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1041000000000001e-05 2024-04-14T15:15:06.394 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.394 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.394 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:15:06.394 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.276517+0000", 2024-04-14T15:15:06.394 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1149000000000002e-05 2024-04-14T15:15:06.395 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:15:06.395 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:15:06.395 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:15:06.395 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.395 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.395 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24871.0:10 18.1c 18:3cd39c19:test-rados-api-smithi090-93709-4::foo:head [writefull 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-04-14T15:15:06.395 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:13:00.870154+0000", 2024-04-14T15:15:06.395 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 125.433536948, 2024-04-14T15:15:06.395 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.067196143, 2024-04-14T15:15:06.395 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:15:06.395 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:15:06.395 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:15:06.395 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:15:06.395 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24871", 2024-04-14T15:15:06.396 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1680088077", 2024-04-14T15:15:06.396 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 10 2024-04-14T15:15:06.396 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.396 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:15:06.396 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.396 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:15:06.396 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.870154+0000", 2024-04-14T15:15:06.396 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:15:06.396 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.396 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.396 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:15:06.396 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.870154+0000", 2024-04-14T15:15:06.396 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:15:06.396 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.396 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.397 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:15:06.397 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.870156+0000", 2024-04-14T15:15:06.397 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.75e-06 2024-04-14T15:15:06.397 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.397 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.397 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:15:06.397 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.870165+0000", 2024-04-14T15:15:06.397 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7490000000000004e-06 2024-04-14T15:15:06.397 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.397 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.397 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:15:06.397 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.870177+0000", 2024-04-14T15:15:06.397 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1686e-05 2024-04-14T15:15:06.397 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.397 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.398 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:15:06.398 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.870188+0000", 2024-04-14T15:15:06.398 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1654000000000001e-05 2024-04-14T15:15:06.398 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.398 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.398 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:15:06.398 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.880646+0000", 2024-04-14T15:15:06.398 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010457664 2024-04-14T15:15:06.398 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.398 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.398 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:15:06.398 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.880744+0000", 2024-04-14T15:15:06.398 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7684999999999997e-05 2024-04-14T15:15:06.398 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.399 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.399 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,7", 2024-04-14T15:15:06.399 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.880916+0000", 2024-04-14T15:15:06.399 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000172518 2024-04-14T15:15:06.399 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.399 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.399 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:15:06.399 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.902443+0000", 2024-04-14T15:15:06.399 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.021526851999999999 2024-04-14T15:15:06.399 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.399 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.399 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:15:06.399 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.913208+0000", 2024-04-14T15:15:06.399 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010764785000000001 2024-04-14T15:15:06.400 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.400 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.400 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:15:06.400 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.937249+0000", 2024-04-14T15:15:06.400 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.024041565000000001 2024-04-14T15:15:06.400 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.400 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.400 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:15:06.400 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.937305+0000", 2024-04-14T15:15:06.400 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5503e-05 2024-04-14T15:15:06.400 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.400 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.400 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:15:06.400 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.937350+0000", 2024-04-14T15:15:06.400 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5732000000000002e-05 2024-04-14T15:15:06.401 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:15:06.401 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:15:06.401 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:15:06.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.401 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.401 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.26497.0:35 112.1 112:8133d882:test-rados-api-smithi090-93724-10::foo:head [write 128~128] snapc 7=[7,6] ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-04-14T15:15:06.401 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:13:20.864703+0000", 2024-04-14T15:15:06.401 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 105.43898783500001, 2024-04-14T15:15:06.401 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.080590538000000003, 2024-04-14T15:15:06.401 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:15:06.401 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:15:06.401 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:15:06.401 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:15:06.401 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.26497", 2024-04-14T15:15:06.402 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3962832909", 2024-04-14T15:15:06.402 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 35 2024-04-14T15:15:06.402 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.402 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:15:06.402 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.402 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:15:06.402 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.864703+0000", 2024-04-14T15:15:06.402 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:15:06.402 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.402 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.402 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:15:06.402 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.864703+0000", 2024-04-14T15:15:06.402 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:15:06.402 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.403 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.403 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:15:06.403 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.864706+0000", 2024-04-14T15:15:06.403 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2139999999999999e-06 2024-04-14T15:15:06.403 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.403 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.403 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:15:06.403 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.864714+0000", 2024-04-14T15:15:06.403 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4610000000000008e-06 2024-04-14T15:15:06.403 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.403 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.403 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:15:06.403 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.864726+0000", 2024-04-14T15:15:06.403 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1395e-05 2024-04-14T15:15:06.403 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.404 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.404 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:15:06.404 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.864736+0000", 2024-04-14T15:15:06.404 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0335e-05 2024-04-14T15:15:06.404 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.404 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.404 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:15:06.404 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.869283+0000", 2024-04-14T15:15:06.404 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0045468080000000003 2024-04-14T15:15:06.404 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.404 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.404 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:15:06.404 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.869389+0000", 2024-04-14T15:15:06.404 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010623799999999999 2024-04-14T15:15:06.404 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.405 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.405 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,5", 2024-04-14T15:15:06.405 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.869590+0000", 2024-04-14T15:15:06.405 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020072800000000001 2024-04-14T15:15:06.405 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.405 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.405 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:15:06.405 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.878172+0000", 2024-04-14T15:15:06.405 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0085828139999999994 2024-04-14T15:15:06.405 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.405 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.405 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:15:06.405 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.897188+0000", 2024-04-14T15:15:06.405 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019015964 2024-04-14T15:15:06.405 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.406 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.406 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:15:06.406 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.945179+0000", 2024-04-14T15:15:06.406 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.047990789999999998 2024-04-14T15:15:06.406 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.406 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.406 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:15:06.406 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.945239+0000", 2024-04-14T15:15:06.406 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9553999999999999e-05 2024-04-14T15:15:06.406 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.406 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.406 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:15:06.407 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:15:06.408 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:05 smithi092 ceph-mon[33966]: osdmap e237: 8 total, 8 up, 8 in 2024-04-14T15:15:06.408 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:05 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/32297493' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi090-93363-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:06.408 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:05 smithi092 ceph-mon[33966]: from='client.48577 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi090-93363-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:06.408 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:05 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:15:06.408 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:05 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:06.408 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:05 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-04-14T15:15:06.408 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:05 smithi092 ceph-mon[33966]: from='client.48577 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi090-93363-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:15:06.408 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:05 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "12"}]': finished 2024-04-14T15:15:06.408 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:05 smithi092 ceph-mon[33966]: osdmap e238: 8 total, 8 up, 8 in 2024-04-14T15:15:06.408 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:05 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:06.408 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:05 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:06.408 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:05 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3767512710' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi090-93342-34"}]: dispatch 2024-04-14T15:15:06.409 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:05 smithi092 ceph-mon[33966]: from='client.48565 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi090-93342-34"}]: dispatch 2024-04-14T15:15:06.409 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.945294+0000", 2024-04-14T15:15:06.409 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5237e-05 2024-04-14T15:15:06.409 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:15:06.409 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:15:06.409 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:15:06.409 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.409 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.409 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.38086.0:1 149.4s0 149:2d0198b2:test-rados-api-smithi090-93709-11::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-04-14T15:15:06.409 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:13:23.915322+0000", 2024-04-14T15:15:06.410 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 102.388369023, 2024-04-14T15:15:06.410 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.98550727800000004, 2024-04-14T15:15:06.410 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:15:06.410 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:15:06.410 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:15:06.410 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:15:06.410 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.38086", 2024-04-14T15:15:06.410 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2598250370", 2024-04-14T15:15:06.410 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1 2024-04-14T15:15:06.410 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.410 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:15:06.410 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.410 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:15:06.410 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:23.915322+0000", 2024-04-14T15:15:06.411 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:15:06.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.411 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.411 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:15:06.411 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:23.915322+0000", 2024-04-14T15:15:06.411 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:15:06.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.411 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.411 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:15:06.411 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:23.915330+0000", 2024-04-14T15:15:06.411 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3350000000000001e-06 2024-04-14T15:15:06.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.411 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.411 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:15:06.412 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:23.915345+0000", 2024-04-14T15:15:06.412 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4956e-05 2024-04-14T15:15:06.412 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.412 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.412 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:15:06.412 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:23.915361+0000", 2024-04-14T15:15:06.412 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6750999999999999e-05 2024-04-14T15:15:06.412 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.412 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.412 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:15:06.412 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:23.915374+0000", 2024-04-14T15:15:06.412 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.309e-05 2024-04-14T15:15:06.412 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.412 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.412 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:15:06.413 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:23.933322+0000", 2024-04-14T15:15:06.413 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017947874999999999 2024-04-14T15:15:06.413 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.413 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.413 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:15:06.413 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:23.933327+0000", 2024-04-14T15:15:06.413 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5399999999999997e-06 2024-04-14T15:15:06.413 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.413 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.413 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:15:06.413 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:24.887691+0000", 2024-04-14T15:15:06.413 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.95436462 2024-04-14T15:15:06.413 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.413 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.414 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:15:06.414 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:24.887825+0000", 2024-04-14T15:15:06.414 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133387 2024-04-14T15:15:06.414 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.414 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.414 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2024-04-14T15:15:06.414 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:24.888636+0000", 2024-04-14T15:15:06.414 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00081147700000000001 2024-04-14T15:15:06.414 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.414 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.414 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2024-04-14T15:15:06.414 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:24.896179+0000", 2024-04-14T15:15:06.414 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0075423469999999996 2024-04-14T15:15:06.414 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.415 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.415 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:15:06.415 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:24.900761+0000", 2024-04-14T15:15:06.415 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0045823230000000001 2024-04-14T15:15:06.415 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.415 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.415 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:15:06.415 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:24.900830+0000", 2024-04-14T15:15:06.415 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8577000000000004e-05 2024-04-14T15:15:06.415 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:15:06.415 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:15:06.415 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:15:06.415 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.415 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.415 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.42758.0:2 160.1b 160:d96a01e0:::testobj:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-04-14T15:15:06.416 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:13:27.843225+0000", 2024-04-14T15:15:06.416 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 98.460465876000001, 2024-04-14T15:15:06.416 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.150888965, 2024-04-14T15:15:06.416 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:15:06.416 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:15:06.416 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:15:06.416 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:15:06.416 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.42758", 2024-04-14T15:15:06.416 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2966476037", 2024-04-14T15:15:06.416 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2 2024-04-14T15:15:06.416 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.416 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:15:06.416 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.416 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:15:06.417 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:27.843225+0000", 2024-04-14T15:15:06.417 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:15:06.417 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.417 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.417 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:15:06.417 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:27.843225+0000", 2024-04-14T15:15:06.417 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:15:06.417 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.417 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.417 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:15:06.417 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:27.843228+0000", 2024-04-14T15:15:06.417 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3140000000000002e-06 2024-04-14T15:15:06.417 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.417 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.417 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:15:06.418 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:27.843237+0000", 2024-04-14T15:15:06.418 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9099999999999994e-06 2024-04-14T15:15:06.418 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.418 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.418 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:15:06.418 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:27.843248+0000", 2024-04-14T15:15:06.418 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1494999999999999e-05 2024-04-14T15:15:06.418 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.418 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.418 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:15:06.418 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:27.843261+0000", 2024-04-14T15:15:06.418 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3067e-05 2024-04-14T15:15:06.418 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.418 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.418 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:15:06.419 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:27.878933+0000", 2024-04-14T15:15:06.419 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.035672263000000003 2024-04-14T15:15:06.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.419 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.419 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:15:06.419 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:27.878942+0000", 2024-04-14T15:15:06.419 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3860000000000007e-06 2024-04-14T15:15:06.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.419 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.419 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:15:06.419 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.961457+0000", 2024-04-14T15:15:06.419 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0825154450000001 2024-04-14T15:15:06.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.419 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.419 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:15:06.420 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.961594+0000", 2024-04-14T15:15:06.420 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000136687 2024-04-14T15:15:06.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.420 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2024-04-14T15:15:06.420 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.961729+0000", 2024-04-14T15:15:06.420 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000134914 2024-04-14T15:15:06.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.420 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:15:06.420 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.962866+0000", 2024-04-14T15:15:06.420 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011373519999999999 2024-04-14T15:15:06.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.420 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:15:06.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.968752+0000", 2024-04-14T15:15:06.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0058854370000000003 2024-04-14T15:15:06.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:15:06.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.994007+0000", 2024-04-14T15:15:06.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.025255248000000001 2024-04-14T15:15:06.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:15:06.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.994064+0000", 2024-04-14T15:15:06.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6789000000000001e-05 2024-04-14T15:15:06.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:15:06.422 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.994114+0000", 2024-04-14T15:15:06.422 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0658000000000002e-05 2024-04-14T15:15:06.422 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:15:06.423 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:15:06.423 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:15:06.423 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.423 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.423 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.44332.0:2 159.0 159:0384ab8e:::oid:head [append 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-04-14T15:15:06.424 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:13:27.899663+0000", 2024-04-14T15:15:06.424 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 98.404027976999998, 2024-04-14T15:15:06.424 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0793927670000001, 2024-04-14T15:15:06.424 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:15:06.424 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:15:06.424 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:15:06.424 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:15:06.424 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.44332", 2024-04-14T15:15:06.424 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/4025981533", 2024-04-14T15:15:06.424 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2 2024-04-14T15:15:06.424 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.424 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:15:06.424 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.424 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:15:06.425 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:27.899663+0000", 2024-04-14T15:15:06.425 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:15:06.425 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.425 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.425 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:15:06.425 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:27.899663+0000", 2024-04-14T15:15:06.425 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:15:06.425 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.425 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.425 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:15:06.425 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:27.899665+0000", 2024-04-14T15:15:06.425 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.44e-06 2024-04-14T15:15:06.425 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.426 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.426 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:15:06.426 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:27.899670+0000", 2024-04-14T15:15:06.426 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7200000000000003e-06 2024-04-14T15:15:06.426 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.426 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.426 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:15:06.426 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:27.899680+0000", 2024-04-14T15:15:06.426 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7049999999999994e-06 2024-04-14T15:15:06.426 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.426 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.426 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:15:06.427 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:27.899691+0000", 2024-04-14T15:15:06.427 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0336999999999999e-05 2024-04-14T15:15:06.427 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.427 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.427 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:15:06.427 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:27.899995+0000", 2024-04-14T15:15:06.427 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030434700000000001 2024-04-14T15:15:06.427 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.427 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.427 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:15:06.427 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:27.900004+0000", 2024-04-14T15:15:06.427 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2420000000000006e-06 2024-04-14T15:15:06.427 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.428 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.428 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:15:06.428 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.969216+0000", 2024-04-14T15:15:06.428 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.069211793 2024-04-14T15:15:06.428 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.428 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.428 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:15:06.428 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.969339+0000", 2024-04-14T15:15:06.428 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012265 2024-04-14T15:15:06.428 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.428 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.428 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,5", 2024-04-14T15:15:06.428 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.969485+0000", 2024-04-14T15:15:06.429 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014675699999999999 2024-04-14T15:15:06.429 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.429 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.429 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:15:06.429 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.971325+0000", 2024-04-14T15:15:06.429 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0018392720000000001 2024-04-14T15:15:06.429 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.429 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.429 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:15:06.429 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.971713+0000", 2024-04-14T15:15:06.429 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038802299999999998 2024-04-14T15:15:06.429 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.429 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.429 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:15:06.430 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.978938+0000", 2024-04-14T15:15:06.430 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0072256710000000004 2024-04-14T15:15:06.430 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.430 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.430 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:15:06.430 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.979007+0000", 2024-04-14T15:15:06.430 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9220000000000005e-05 2024-04-14T15:15:06.430 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.430 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.430 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:15:06.430 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.979056+0000", 2024-04-14T15:15:06.430 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8590000000000001e-05 2024-04-14T15:15:06.431 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:15:06.431 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:15:06.431 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:15:06.431 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.431 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.431 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.35916.0:2 164.1b 164:d96a01e0:::testobj:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-04-14T15:15:06.431 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:13:30.848008+0000", 2024-04-14T15:15:06.431 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 95.455683691999994, 2024-04-14T15:15:06.431 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.012766968, 2024-04-14T15:15:06.431 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:15:06.431 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:15:06.431 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:15:06.431 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:15:06.432 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.35916", 2024-04-14T15:15:06.432 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2231042766", 2024-04-14T15:15:06.432 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2 2024-04-14T15:15:06.432 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.432 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:15:06.432 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.432 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:15:06.432 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:30.848008+0000", 2024-04-14T15:15:06.432 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:15:06.432 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.432 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.432 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:15:06.432 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:30.848008+0000", 2024-04-14T15:15:06.432 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:15:06.433 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.433 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.433 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:15:06.433 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:30.848009+0000", 2024-04-14T15:15:06.433 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5e-06 2024-04-14T15:15:06.433 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.433 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.434 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:15:06.434 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:30.848015+0000", 2024-04-14T15:15:06.434 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2439999999999998e-06 2024-04-14T15:15:06.434 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.434 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.434 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:15:06.434 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:30.848024+0000", 2024-04-14T15:15:06.434 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2560000000000002e-06 2024-04-14T15:15:06.434 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.434 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.434 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:15:06.434 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:30.848032+0000", 2024-04-14T15:15:06.434 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0630000000000006e-06 2024-04-14T15:15:06.435 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.435 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.435 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:15:06.435 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:30.848081+0000", 2024-04-14T15:15:06.435 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9628e-05 2024-04-14T15:15:06.435 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.435 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.435 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:15:06.435 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:30.848089+0000", 2024-04-14T15:15:06.435 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.362e-06 2024-04-14T15:15:06.435 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.435 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.435 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:15:06.436 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:31.850888+0000", 2024-04-14T15:15:06.436 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.002799182 2024-04-14T15:15:06.436 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.436 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.436 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:15:06.436 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:31.850999+0000", 2024-04-14T15:15:06.436 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000111294 2024-04-14T15:15:06.436 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.436 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.436 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,6", 2024-04-14T15:15:06.436 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:31.851133+0000", 2024-04-14T15:15:06.436 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133678 2024-04-14T15:15:06.436 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.436 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.437 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:15:06.437 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:31.852376+0000", 2024-04-14T15:15:06.437 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0012427479999999999 2024-04-14T15:15:06.437 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.437 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.437 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:15:06.437 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:31.859603+0000", 2024-04-14T15:15:06.437 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0072276409999999999 2024-04-14T15:15:06.437 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.437 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.437 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:15:06.437 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:31.860679+0000", 2024-04-14T15:15:06.437 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001076123 2024-04-14T15:15:06.437 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.438 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.438 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:15:06.438 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:31.860733+0000", 2024-04-14T15:15:06.438 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3341999999999999e-05 2024-04-14T15:15:06.438 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:15:06.438 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:15:06.438 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:15:06.438 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:31.860775+0000", 2024-04-14T15:15:06.438 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1906999999999997e-05 2024-04-14T15:15:06.438 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:15:06.438 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:15:06.438 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:15:06.438 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:15:06.439 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:15:06.439 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-14T15:15:06.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:06.462 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:06.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:15:06.563 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:15:06.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:06.591 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:06.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:15:06.692 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:15:06.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:06.716 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:06.716 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:15:06.817 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:15:06.840 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:06.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:06.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:15:06.942 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:15:06.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:06.968 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:06.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:15:07.070 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:15:07.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:07.094 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:07.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:15:07.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:06 smithi090 ceph-mon[41670]: pgmap v302: 259 pgs: 32 unknown, 8 creating+peering, 1 active+clean+snaptrim, 1 peering, 217 active+clean; 586 KiB data, 545 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:15:07.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:06 smithi090 ceph-mon[41670]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:15:07.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:06 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:07.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:06 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:15:07.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:06 smithi090 ceph-mon[41670]: from='client.48565 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi090-93342-34"}]': finished 2024-04-14T15:15:07.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:06 smithi090 ceph-mon[41670]: osdmap e239: 8 total, 8 up, 8 in 2024-04-14T15:15:07.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:06 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3767512710' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi090-93342-34"}]: dispatch 2024-04-14T15:15:07.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:06 smithi090 ceph-mon[41670]: from='client.48565 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi090-93342-34"}]: dispatch 2024-04-14T15:15:07.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:06 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3698586579' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi090-93363-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:07.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:06 smithi090 ceph-mon[30399]: pgmap v302: 259 pgs: 32 unknown, 8 creating+peering, 1 active+clean+snaptrim, 1 peering, 217 active+clean; 586 KiB data, 545 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:15:07.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:06 smithi090 ceph-mon[30399]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:15:07.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:06 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:07.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:06 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:15:07.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:06 smithi090 ceph-mon[30399]: from='client.48565 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi090-93342-34"}]': finished 2024-04-14T15:15:07.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:06 smithi090 ceph-mon[30399]: osdmap e239: 8 total, 8 up, 8 in 2024-04-14T15:15:07.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:06 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3767512710' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi090-93342-34"}]: dispatch 2024-04-14T15:15:07.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:06 smithi090 ceph-mon[30399]: from='client.48565 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi090-93342-34"}]: dispatch 2024-04-14T15:15:07.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:06 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3698586579' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi090-93363-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:07.196 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:15:07.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:07.219 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:07.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:15:07.320 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:15:07.334 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-14T15:15:07.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:07.344 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:07.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:15:07.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:06 smithi092 ceph-mon[33966]: pgmap v302: 259 pgs: 32 unknown, 8 creating+peering, 1 active+clean+snaptrim, 1 peering, 217 active+clean; 586 KiB data, 545 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:15:07.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:06 smithi092 ceph-mon[33966]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:15:07.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:06 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:07.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:06 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:15:07.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:06 smithi092 ceph-mon[33966]: from='client.48565 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi090-93342-34"}]': finished 2024-04-14T15:15:07.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:06 smithi092 ceph-mon[33966]: osdmap e239: 8 total, 8 up, 8 in 2024-04-14T15:15:07.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:06 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3767512710' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi090-93342-34"}]: dispatch 2024-04-14T15:15:07.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:06 smithi092 ceph-mon[33966]: from='client.48565 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi090-93342-34"}]: dispatch 2024-04-14T15:15:07.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:06 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3698586579' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi090-93363-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:07.445 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:15:07.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:07.473 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:07.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:15:07.574 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:15:07.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:07.599 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:07.600 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:15:07.700 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:15:07.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:07.725 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:07.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:15:07.826 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:15:07.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:07.861 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:07.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:15:07.941 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:07 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:07.941 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:07 smithi090 ceph-mon[41670]: from='client.48565 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi090-93342-34"}]': finished 2024-04-14T15:15:07.941 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:07 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3698586579' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi090-93363-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:15:07.941 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:07 smithi090 ceph-mon[41670]: osdmap e240: 8 total, 8 up, 8 in 2024-04-14T15:15:07.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:07 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2466564286' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi090-93342-35"}]: dispatch 2024-04-14T15:15:07.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:07 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2466564286' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi090-93342-35"}]: dispatch 2024-04-14T15:15:07.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:07 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2466564286' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi090-93342-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:07.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:07 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:15:07.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:07 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:15:07.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:07 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:07.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:07 smithi090 ceph-mon[30399]: from='client.48565 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi090-93342-34"}]': finished 2024-04-14T15:15:07.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:07 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3698586579' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi090-93363-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:15:07.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:07 smithi090 ceph-mon[30399]: osdmap e240: 8 total, 8 up, 8 in 2024-04-14T15:15:07.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:07 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2466564286' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi090-93342-35"}]: dispatch 2024-04-14T15:15:07.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:07 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2466564286' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi090-93342-35"}]: dispatch 2024-04-14T15:15:07.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:07 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2466564286' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi090-93342-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:07.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:07 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:15:07.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:07 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:15:07.962 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:15:07.991 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:07.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:07.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:15:08.093 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:15:08.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:08.118 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:08.118 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:15:08.219 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:15:08.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:08.243 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:08.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:15:08.344 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:15:08.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:07 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:08.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:07 smithi092 ceph-mon[33966]: from='client.48565 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi090-93342-34"}]': finished 2024-04-14T15:15:08.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:07 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3698586579' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi090-93363-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:15:08.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:07 smithi092 ceph-mon[33966]: osdmap e240: 8 total, 8 up, 8 in 2024-04-14T15:15:08.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:07 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2466564286' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi090-93342-35"}]: dispatch 2024-04-14T15:15:08.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:07 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2466564286' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi090-93342-35"}]: dispatch 2024-04-14T15:15:08.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:07 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2466564286' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi090-93342-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:08.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:07 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:15:08.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:07 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:15:08.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:08.374 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:08.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:15:08.429 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-14T15:15:08.474 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:15:08.501 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:08.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:08.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:15:08.603 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:15:08.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:08.627 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:08.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:15:08.728 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:15:08.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:08.752 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:08.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:15:08.853 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:15:08.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:08.880 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:08.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:15:08.981 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:15:09.005 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:09.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:09.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:15:09.065 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:15:09.107 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:15:09.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:09.138 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:09.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:15:09.167 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:08 smithi090 ceph-mon[41670]: pgmap v305: 315 pgs: 96 unknown, 1 active+clean+snaptrim, 1 peering, 217 active+clean; 586 KiB data, 545 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:15:09.168 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:08 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:09.168 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:08 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:15:09.168 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:08 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:15:09.168 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:08 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2466564286' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi090-93342-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:15:09.168 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:08 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-25", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:15:09.168 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:08 smithi090 ceph-mon[41670]: osdmap e241: 8 total, 8 up, 8 in 2024-04-14T15:15:09.168 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:08 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2466564286' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi090-93342-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi090-93342-35"}]: dispatch 2024-04-14T15:15:09.168 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:08 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-25"}]: dispatch 2024-04-14T15:15:09.168 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:08 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-25"}]: dispatch 2024-04-14T15:15:09.168 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:08 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2940164857' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi090-93363-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:09.169 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:08 smithi090 ceph-mon[30399]: pgmap v305: 315 pgs: 96 unknown, 1 active+clean+snaptrim, 1 peering, 217 active+clean; 586 KiB data, 545 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:15:09.169 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:08 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:09.169 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:08 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:15:09.169 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:08 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:15:09.169 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:08 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2466564286' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi090-93342-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:15:09.169 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:08 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-25", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:15:09.169 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:08 smithi090 ceph-mon[30399]: osdmap e241: 8 total, 8 up, 8 in 2024-04-14T15:15:09.169 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:08 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2466564286' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi090-93342-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi090-93342-35"}]: dispatch 2024-04-14T15:15:09.169 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:08 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-25"}]: dispatch 2024-04-14T15:15:09.169 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:08 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-25"}]: dispatch 2024-04-14T15:15:09.169 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:08 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2940164857' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi090-93363-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:09.239 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:15:09.267 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:09.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:09.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:15:09.369 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:15:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:08 smithi092 ceph-mon[33966]: pgmap v305: 315 pgs: 96 unknown, 1 active+clean+snaptrim, 1 peering, 217 active+clean; 586 KiB data, 545 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:15:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:08 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:08 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:15:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:08 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:15:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:08 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2466564286' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi090-93342-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:15:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:08 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-25", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:15:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:08 smithi092 ceph-mon[33966]: osdmap e241: 8 total, 8 up, 8 in 2024-04-14T15:15:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:08 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2466564286' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi090-93342-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi090-93342-35"}]: dispatch 2024-04-14T15:15:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:08 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-25"}]: dispatch 2024-04-14T15:15:09.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:08 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-25"}]: dispatch 2024-04-14T15:15:09.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:08 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2940164857' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi090-93363-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:09.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:09.396 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:09.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:15:09.498 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:15:09.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:09.522 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:09.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:15:09.623 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:15:09.648 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:09.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:09.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:15:09.750 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:15:09.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:09.774 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:09.774 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:15:09.875 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:15:09.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:09.904 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:09.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:15:09.952 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-14T15:15:09.952 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-14T15:15:09.952 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-04-14T15:15:09.952 INFO:tasks.thrashosds.thrasher:Setting osd 3 primary_affinity to 1.000000 2024-04-14T15:15:09.953 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd primary-affinity 3 1 2024-04-14T15:15:09.975 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:09 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:09.975 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:09 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-04-14T15:15:09.975 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:09 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-25"}]': finished 2024-04-14T15:15:09.975 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:09 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2940164857' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi090-93363-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:15:09.975 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:09 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "11"}]': finished 2024-04-14T15:15:09.975 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:09 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-25", "mode": "writeback"}]: dispatch 2024-04-14T15:15:09.976 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:09 smithi090 ceph-mon[41670]: osdmap e242: 8 total, 8 up, 8 in 2024-04-14T15:15:09.976 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:09 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-25", "mode": "writeback"}]: dispatch 2024-04-14T15:15:09.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:09 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:09.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:09 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-04-14T15:15:09.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:09 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-25"}]': finished 2024-04-14T15:15:09.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:09 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2940164857' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi090-93363-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:15:09.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:09 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "11"}]': finished 2024-04-14T15:15:09.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:09 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-25", "mode": "writeback"}]: dispatch 2024-04-14T15:15:09.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:09 smithi090 ceph-mon[30399]: osdmap e242: 8 total, 8 up, 8 in 2024-04-14T15:15:09.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:09 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-25", "mode": "writeback"}]: dispatch 2024-04-14T15:15:10.006 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:15:10.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:10.031 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:10.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:15:10.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:09.718+0000 7f1dd33f4700 1 -- 172.21.15.90:0/364894960 >> v1:172.21.15.90:6789/0 conn(0x7f1dcc10e310 legacy=0x7f1dcc1107d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:10.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:09.718+0000 7f1dd33f4700 1 -- 172.21.15.90:0/364894960 shutdown_connections 2024-04-14T15:15:10.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:09.718+0000 7f1dd33f4700 1 -- 172.21.15.90:0/364894960 >> 172.21.15.90:0/364894960 conn(0x7f1dcc0fd290 msgr2=0x7f1dcc0ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:15:10.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:09.718+0000 7f1dd33f4700 1 -- 172.21.15.90:0/364894960 shutdown_connections 2024-04-14T15:15:10.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:09.718+0000 7f1dd33f4700 1 -- 172.21.15.90:0/364894960 wait complete. 2024-04-14T15:15:10.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:09.719+0000 7f1dd33f4700 1 librados: starting msgr at 2024-04-14T15:15:10.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:09.719+0000 7f1dd33f4700 1 librados: starting objecter 2024-04-14T15:15:10.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:09.719+0000 7f1dd33f4700 1 Processor -- start 2024-04-14T15:15:10.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:09.719+0000 7f1dd33f4700 1 -- start start 2024-04-14T15:15:10.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:09.719+0000 7f1dd33f4700 1 librados: setting wanted keys 2024-04-14T15:15:10.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:09.719+0000 7f1dd33f4700 1 librados: calling monclient init 2024-04-14T15:15:10.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:09.719+0000 7f1dd33f4700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1dcc1a9be0 con 0x7f1dcc10a740 2024-04-14T15:15:10.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:09.719+0000 7f1dd33f4700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1dcc1aae00 con 0x7f1dcc10e310 2024-04-14T15:15:10.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:09.719+0000 7f1dd33f4700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1dcc1ac020 con 0x7f1dcc107890 2024-04-14T15:15:10.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:09.719+0000 7f1dd1190700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f1dcc107890 0x7f1dcc10d930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:40742/0 (socket says 172.21.15.90:40742) 2024-04-14T15:15:10.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:09.719+0000 7f1dd1190700 1 -- 172.21.15.90:0/3283301722 learned_addr learned my addr 172.21.15.90:0/3283301722 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:15:10.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:09.720+0000 7f1dc27fc700 10 client.?.objecter ms_handle_connect 0x7f1dcc10a740 2024-04-14T15:15:10.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:09.720+0000 7f1dc27fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:15:10.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:09.720+0000 7f1dc27fc700 10 client.?.objecter ms_handle_connect 0x7f1dcc107890 2024-04-14T15:15:10.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:09.720+0000 7f1dc27fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:15:10.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:09.720+0000 7f1dc27fc700 1 -- 172.21.15.90:0/3283301722 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 618666843 0 0) 0x7f1dcc1a9be0 con 0x7f1dcc10a740 2024-04-14T15:15:10.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:09.720+0000 7f1dc27fc700 1 -- 172.21.15.90:0/3283301722 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1db80035b0 con 0x7f1dcc10a740 2024-04-14T15:15:10.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:09.720+0000 7f1dc27fc700 10 client.?.objecter ms_handle_connect 0x7f1dcc10e310 2024-04-14T15:15:10.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:09.720+0000 7f1dc27fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:15:10.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:09.720+0000 7f1dc27fc700 1 -- 172.21.15.90:0/3283301722 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2859572122 0 0) 0x7f1dcc1aae00 con 0x7f1dcc10e310 2024-04-14T15:15:10.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:09.720+0000 7f1dc27fc700 1 -- 172.21.15.90:0/3283301722 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1dcc1a9be0 con 0x7f1dcc10e310 2024-04-14T15:15:10.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:09.720+0000 7f1dc27fc700 1 -- 172.21.15.90:0/3283301722 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2978252601 0 0) 0x7f1db80035b0 con 0x7f1dcc10a740 2024-04-14T15:15:10.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:09.720+0000 7f1dc27fc700 1 -- 172.21.15.90:0/3283301722 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1dcc1aae00 con 0x7f1dcc10a740 2024-04-14T15:15:10.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:09.720+0000 7f1dc27fc700 1 -- 172.21.15.90:0/3283301722 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f1dbc001a30 con 0x7f1dcc10a740 2024-04-14T15:15:10.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:09.720+0000 7f1dc27fc700 1 -- 172.21.15.90:0/3283301722 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3991466428 0 0) 0x7f1dcc1ac020 con 0x7f1dcc107890 2024-04-14T15:15:10.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:09.720+0000 7f1dc27fc700 1 -- 172.21.15.90:0/3283301722 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1db80035b0 con 0x7f1dcc107890 2024-04-14T15:15:10.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:09.720+0000 7f1dc27fc700 1 -- 172.21.15.90:0/3283301722 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3311179128 0 0) 0x7f1dcc1a9be0 con 0x7f1dcc10e310 2024-04-14T15:15:10.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:09.720+0000 7f1dc27fc700 1 -- 172.21.15.90:0/3283301722 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1dcc1ac020 con 0x7f1dcc10e310 2024-04-14T15:15:10.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:09.720+0000 7f1dc27fc700 1 -- 172.21.15.90:0/3283301722 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f1dc40047f0 con 0x7f1dcc10e310 2024-04-14T15:15:10.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:09.720+0000 7f1dc27fc700 1 -- 172.21.15.90:0/3283301722 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2047609831 0 0) 0x7f1dcc1aae00 con 0x7f1dcc10a740 2024-04-14T15:15:10.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:09.720+0000 7f1dc27fc700 1 -- 172.21.15.90:0/3283301722 >> v1:172.21.15.90:6790/0 conn(0x7f1dcc107890 legacy=0x7f1dcc10d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:10.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:09.720+0000 7f1dc27fc700 1 -- 172.21.15.90:0/3283301722 >> v1:172.21.15.92:6789/0 conn(0x7f1dcc10e310 legacy=0x7f1dcc1a82f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:10.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:09.720+0000 7f1dc27fc700 1 -- 172.21.15.90:0/3283301722 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dcc1ad260 con 0x7f1dcc10a740 2024-04-14T15:15:10.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:09.720+0000 7f1dd33f4700 1 -- 172.21.15.90:0/3283301722 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1dcc1ac250 con 0x7f1dcc10a740 2024-04-14T15:15:10.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:09.720+0000 7f1dd33f4700 10 client.36750.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:15:10.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:09.720+0000 7f1dd33f4700 1 -- 172.21.15.90:0/3283301722 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f1dcc1ac710 con 0x7f1dcc10a740 2024-04-14T15:15:10.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:09.721+0000 7f1dd33f4700 1 librados: init done 2024-04-14T15:15:10.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:09.721+0000 7f1dc27fc700 1 -- 172.21.15.90:0/3283301722 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f1dbc001ce0 con 0x7f1dcc10a740 2024-04-14T15:15:10.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:09.721+0000 7f1dc27fc700 1 -- 172.21.15.90:0/3283301722 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f1dbc0049d0 con 0x7f1dcc10a740 2024-04-14T15:15:10.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:09.722+0000 7f1dd33f4700 1 -- 172.21.15.90:0/3283301722 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1db0005120 con 0x7f1dcc10a740 2024-04-14T15:15:10.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:09.722+0000 7f1dc27fc700 1 -- 172.21.15.90:0/3283301722 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f1dbc01b660 con 0x7f1dcc10a740 2024-04-14T15:15:10.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:09.723+0000 7f1dc27fc700 1 -- 172.21.15.90:0/3283301722 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(241..241 src has 51..241) ==== 9867+0+0 (unknown 603882919 0 0) 0x7f1dbc089f20 con 0x7f1dcc10a740 2024-04-14T15:15:10.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:09.723+0000 7f1dc27fc700 10 client.36750.objecter ms_dispatch 0x7f1dcc001550 osd_map(241..241 src has 51..241) 2024-04-14T15:15:10.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:09.723+0000 7f1dc27fc700 3 client.36750.objecter handle_osd_map got epochs [241,241] > 0 2024-04-14T15:15:10.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:09.723+0000 7f1dc27fc700 3 client.36750.objecter handle_osd_map decoding full epoch 241 2024-04-14T15:15:10.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:09.723+0000 7f1dc27fc700 20 client.36750.objecter dump_active .. 0 homeless 2024-04-14T15:15:10.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:09.723+0000 7f1dc27fc700 10 client.36750.objecter ms_handle_connect 0x7f1db8071470 2024-04-14T15:15:10.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:09.728+0000 7f1dc27fc700 1 -- 172.21.15.90:0/3283301722 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f1dbc0562a0 con 0x7f1dcc10a740 2024-04-14T15:15:10.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:09.974+0000 7f1dd33f4700 1 -- 172.21.15.90:0/3283301722 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7f1db0005410 con 0x7f1dcc10a740 2024-04-14T15:15:10.132 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:15:10.153 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:15:10.159 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:10.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:10.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:15:10.260 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:15:10.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:10.285 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:10.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:15:10.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:09 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:10.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:09 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-04-14T15:15:10.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:09 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-25"}]': finished 2024-04-14T15:15:10.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:09 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2940164857' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi090-93363-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:15:10.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:09 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi090-93444-1", "var": "pg_num_actual", "val": "11"}]': finished 2024-04-14T15:15:10.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:09 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-25", "mode": "writeback"}]: dispatch 2024-04-14T15:15:10.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:09 smithi092 ceph-mon[33966]: osdmap e242: 8 total, 8 up, 8 in 2024-04-14T15:15:10.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:09 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-25", "mode": "writeback"}]: dispatch 2024-04-14T15:15:10.385 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:15:10.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:10.414 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:10.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:15:10.515 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:15:10.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:10.543 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:10.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:15:10.644 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:15:10.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:10.669 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:10.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:15:10.778 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:15:10.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:10.806 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:10.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:15:10.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:10.838+0000 7f1dc27fc700 1 -- 172.21.15.90:0/3283301722 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v243) ==== 97+0+0 (unknown 1818213210 0 0) 0x7f1dbc059f50 con 0x7f1dcc10a740 2024-04-14T15:15:10.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:10.840+0000 7f1dd33f4700 10 librados: watch_flush enter 2024-04-14T15:15:10.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:10.840+0000 7f1dd33f4700 10 librados: watch_flush exit 2024-04-14T15:15:10.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:10.840+0000 7f1dd33f4700 20 client.36750.objecter shutdown clearing up homeless session... 2024-04-14T15:15:10.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:10.840+0000 7f1dd33f4700 10 client.36750.objecter successfully canceled tick 2024-04-14T15:15:10.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:10.840+0000 7f1dd33f4700 1 -- 172.21.15.90:0/3283301722 >> v1:172.21.15.90:6800/420935427 conn(0x7f1db8071470 legacy=0x7f1db8073930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:10.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:10.840+0000 7f1dd33f4700 1 -- 172.21.15.90:0/3283301722 >> v1:172.21.15.90:6789/0 conn(0x7f1dcc10a740 legacy=0x7f1dcc1a4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:10.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:10.840+0000 7f1dd33f4700 1 -- 172.21.15.90:0/3283301722 shutdown_connections 2024-04-14T15:15:10.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:10.840+0000 7f1dd33f4700 1 -- 172.21.15.90:0/3283301722 >> 172.21.15.90:0/3283301722 conn(0x7f1dcc0fd290 msgr2=0x7f1dcc1006f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:15:10.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:10.840+0000 7f1dd33f4700 1 -- 172.21.15.90:0/3283301722 shutdown_connections 2024-04-14T15:15:10.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:10.841+0000 7f1dd33f4700 1 -- 172.21.15.90:0/3283301722 wait complete. 2024-04-14T15:15:10.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:10.841+0000 7f1dd33f4700 1 librados: shutdown 2024-04-14T15:15:10.842 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is set 2024-04-14T15:15:10.908 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:15:10.933 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:10.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:10.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:15:11.035 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:15:11.059 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:11.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:11.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:15:11.161 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:15:11.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:11.187 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:11.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:15:11.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:10 smithi090 ceph-mon[30399]: pgmap v308: 346 pgs: 23 creating+peering, 34 unknown, 289 active+clean; 586 KiB data, 545 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-04-14T15:15:11.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:10 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:11.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:10 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3283301722' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:15:11.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:10 smithi090 ceph-mon[30399]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:15:11.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:10 smithi090 ceph-mon[30399]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:15:11.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:10 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2466564286' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi090-93342-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi090-93342-35"}]': finished 2024-04-14T15:15:11.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:10 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-25", "mode": "writeback"}]': finished 2024-04-14T15:15:11.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:10 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3283301722' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-14T15:15:11.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:10 smithi090 ceph-mon[30399]: osdmap e243: 8 total, 8 up, 8 in 2024-04-14T15:15:11.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:10 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/548115492' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi090-93363-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:11.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:10 smithi090 ceph-mon[30399]: from='client.46709 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi090-93363-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:11.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:10 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:15:11.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:10 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:15:11.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:10 smithi090 ceph-mon[41670]: pgmap v308: 346 pgs: 23 creating+peering, 34 unknown, 289 active+clean; 586 KiB data, 545 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-04-14T15:15:11.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:10 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:11.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:10 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3283301722' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:15:11.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:10 smithi090 ceph-mon[41670]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:15:11.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:10 smithi090 ceph-mon[41670]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:15:11.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:10 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2466564286' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi090-93342-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi090-93342-35"}]': finished 2024-04-14T15:15:11.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:10 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-25", "mode": "writeback"}]': finished 2024-04-14T15:15:11.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:10 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3283301722' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-14T15:15:11.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:10 smithi090 ceph-mon[41670]: osdmap e243: 8 total, 8 up, 8 in 2024-04-14T15:15:11.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:10 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/548115492' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi090-93363-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:11.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:10 smithi090 ceph-mon[41670]: from='client.46709 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi090-93363-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:11.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:10 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:15:11.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:10 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:15:11.289 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:15:11.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:11.314 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:11.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:15:11.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:10 smithi092 ceph-mon[33966]: pgmap v308: 346 pgs: 23 creating+peering, 34 unknown, 289 active+clean; 586 KiB data, 545 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-04-14T15:15:11.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:10 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:11.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:10 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3283301722' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:15:11.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:10 smithi092 ceph-mon[33966]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:15:11.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:10 smithi092 ceph-mon[33966]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:15:11.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:10 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2466564286' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi090-93342-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi090-93342-35"}]': finished 2024-04-14T15:15:11.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:10 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-25", "mode": "writeback"}]': finished 2024-04-14T15:15:11.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:10 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3283301722' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-14T15:15:11.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:10 smithi092 ceph-mon[33966]: osdmap e243: 8 total, 8 up, 8 in 2024-04-14T15:15:11.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:10 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/548115492' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi090-93363-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:11.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:10 smithi092 ceph-mon[33966]: from='client.46709 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi090-93363-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:11.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:10 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:15:11.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:10 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:15:11.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.390+0000 7f69ceac1700 1 -- 172.21.15.90:0/1133453910 >> v1:172.21.15.90:6789/0 conn(0x7f69c8108880 legacy=0x7f69c8108c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:11.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.390+0000 7f69ceac1700 1 -- 172.21.15.90:0/1133453910 shutdown_connections 2024-04-14T15:15:11.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.390+0000 7f69ceac1700 1 -- 172.21.15.90:0/1133453910 >> 172.21.15.90:0/1133453910 conn(0x7f69c80fe3d0 msgr2=0x7f69c81007f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:15:11.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.390+0000 7f69ceac1700 1 -- 172.21.15.90:0/1133453910 shutdown_connections 2024-04-14T15:15:11.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.390+0000 7f69ceac1700 1 -- 172.21.15.90:0/1133453910 wait complete. 2024-04-14T15:15:11.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.391+0000 7f69ceac1700 1 librados: starting msgr at 2024-04-14T15:15:11.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.391+0000 7f69ceac1700 1 librados: starting objecter 2024-04-14T15:15:11.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.391+0000 7f69ceac1700 1 Processor -- start 2024-04-14T15:15:11.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.391+0000 7f69ceac1700 1 -- start start 2024-04-14T15:15:11.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.391+0000 7f69ceac1700 1 librados: setting wanted keys 2024-04-14T15:15:11.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.391+0000 7f69ceac1700 1 librados: calling monclient init 2024-04-14T15:15:11.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.391+0000 7f69ceac1700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f69c81a9d80 con 0x7f69c810b730 2024-04-14T15:15:11.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.391+0000 7f69ceac1700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f69c81aafa0 con 0x7f69c8108880 2024-04-14T15:15:11.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.391+0000 7f69ceac1700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f69c81ac1e0 con 0x7f69c810f300 2024-04-14T15:15:11.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.391+0000 7f69c7fff700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f69c810b730 0x7f69c81a4ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:59216/0 (socket says 172.21.15.90:59216) 2024-04-14T15:15:11.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.391+0000 7f69c7fff700 1 -- 172.21.15.90:0/3652623365 learned_addr learned my addr 172.21.15.90:0/3652623365 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:15:11.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.391+0000 7f69c5ffb700 10 client.?.objecter ms_handle_connect 0x7f69c810f300 2024-04-14T15:15:11.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.391+0000 7f69c5ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:15:11.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.391+0000 7f69c5ffb700 10 client.?.objecter ms_handle_connect 0x7f69c810b730 2024-04-14T15:15:11.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.391+0000 7f69c5ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:15:11.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.391+0000 7f69c5ffb700 10 client.?.objecter ms_handle_connect 0x7f69c8108880 2024-04-14T15:15:11.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.391+0000 7f69c5ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:15:11.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.391+0000 7f69c5ffb700 1 -- 172.21.15.90:0/3652623365 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1480295667 0 0) 0x7f69c81ac1e0 con 0x7f69c810f300 2024-04-14T15:15:11.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.391+0000 7f69c5ffb700 1 -- 172.21.15.90:0/3652623365 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f69b80035b0 con 0x7f69c810f300 2024-04-14T15:15:11.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.391+0000 7f69c5ffb700 1 -- 172.21.15.90:0/3652623365 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1748724691 0 0) 0x7f69c81a9d80 con 0x7f69c810b730 2024-04-14T15:15:11.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.391+0000 7f69c5ffb700 1 -- 172.21.15.90:0/3652623365 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f69c81ac1e0 con 0x7f69c810b730 2024-04-14T15:15:11.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.391+0000 7f69c5ffb700 1 -- 172.21.15.90:0/3652623365 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1268550592 0 0) 0x7f69c81aafa0 con 0x7f69c8108880 2024-04-14T15:15:11.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.391+0000 7f69c5ffb700 1 -- 172.21.15.90:0/3652623365 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f69c81a9d80 con 0x7f69c8108880 2024-04-14T15:15:11.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.391+0000 7f69c5ffb700 1 -- 172.21.15.90:0/3652623365 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1822808293 0 0) 0x7f69b80035b0 con 0x7f69c810f300 2024-04-14T15:15:11.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.392+0000 7f69c5ffb700 1 -- 172.21.15.90:0/3652623365 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f69c81aafa0 con 0x7f69c810f300 2024-04-14T15:15:11.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.392+0000 7f69c5ffb700 1 -- 172.21.15.90:0/3652623365 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f69c0001c30 con 0x7f69c810f300 2024-04-14T15:15:11.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.392+0000 7f69c5ffb700 1 -- 172.21.15.90:0/3652623365 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2866648762 0 0) 0x7f69c81a9d80 con 0x7f69c8108880 2024-04-14T15:15:11.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.392+0000 7f69c5ffb700 1 -- 172.21.15.90:0/3652623365 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f69b80035b0 con 0x7f69c8108880 2024-04-14T15:15:11.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.392+0000 7f69c5ffb700 1 -- 172.21.15.90:0/3652623365 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2706102880 0 0) 0x7f69c81ac1e0 con 0x7f69c810b730 2024-04-14T15:15:11.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.392+0000 7f69c5ffb700 1 -- 172.21.15.90:0/3652623365 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f69c81a9d80 con 0x7f69c810b730 2024-04-14T15:15:11.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.392+0000 7f69c5ffb700 1 -- 172.21.15.90:0/3652623365 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f69bc0019d0 con 0x7f69c8108880 2024-04-14T15:15:11.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.392+0000 7f69c5ffb700 1 -- 172.21.15.90:0/3652623365 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f69b4001870 con 0x7f69c810b730 2024-04-14T15:15:11.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.392+0000 7f69c5ffb700 1 -- 172.21.15.90:0/3652623365 <== mon.2 v1:172.21.15.90:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1151109559 0 0) 0x7f69c81aafa0 con 0x7f69c810f300 2024-04-14T15:15:11.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.392+0000 7f69c5ffb700 1 -- 172.21.15.90:0/3652623365 >> v1:172.21.15.92:6789/0 conn(0x7f69c8108880 legacy=0x7f69c80767e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:11.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.392+0000 7f69c5ffb700 1 -- 172.21.15.90:0/3652623365 >> v1:172.21.15.90:6789/0 conn(0x7f69c810b730 legacy=0x7f69c81a4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:11.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.392+0000 7f69c5ffb700 1 -- 172.21.15.90:0/3652623365 --> v1:172.21.15.90:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69c81ad420 con 0x7f69c810f300 2024-04-14T15:15:11.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.392+0000 7f69ceac1700 1 -- 172.21.15.90:0/3652623365 --> v1:172.21.15.90:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f69c81a9fb0 con 0x7f69c810f300 2024-04-14T15:15:11.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.392+0000 7f69c5ffb700 1 -- 172.21.15.90:0/3652623365 <== mon.2 v1:172.21.15.90:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f69c0003ec0 con 0x7f69c810f300 2024-04-14T15:15:11.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.392+0000 7f69c5ffb700 1 -- 172.21.15.90:0/3652623365 <== mon.2 v1:172.21.15.90:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f69c0004eb0 con 0x7f69c810f300 2024-04-14T15:15:11.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.392+0000 7f69ceac1700 10 client.46715.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:15:11.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.392+0000 7f69ceac1700 1 -- 172.21.15.90:0/3652623365 --> v1:172.21.15.90:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f69c81aa470 con 0x7f69c810f300 2024-04-14T15:15:11.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.392+0000 7f69ceac1700 1 librados: init done 2024-04-14T15:15:11.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.393+0000 7f69ceac1700 1 -- 172.21.15.90:0/3652623365 --> v1:172.21.15.90:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f69c8103fc0 con 0x7f69c810f300 2024-04-14T15:15:11.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.393+0000 7f69c5ffb700 1 -- 172.21.15.90:0/3652623365 <== mon.2 v1:172.21.15.90:6790/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f69c0004f00 con 0x7f69c810f300 2024-04-14T15:15:11.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.394+0000 7f69c5ffb700 1 -- 172.21.15.90:0/3652623365 <== mon.2 v1:172.21.15.90:6790/0 8 ==== osd_map(243..243 src has 51..243) ==== 10746+0+0 (unknown 848047230 0 0) 0x7f69c008a630 con 0x7f69c810f300 2024-04-14T15:15:11.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.394+0000 7f69c5ffb700 10 client.46715.objecter ms_dispatch 0x7f69c8001550 osd_map(243..243 src has 51..243) 2024-04-14T15:15:11.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.394+0000 7f69c5ffb700 3 client.46715.objecter handle_osd_map got epochs [243,243] > 0 2024-04-14T15:15:11.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.394+0000 7f69c5ffb700 3 client.46715.objecter handle_osd_map decoding full epoch 243 2024-04-14T15:15:11.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.394+0000 7f69c5ffb700 20 client.46715.objecter dump_active .. 0 homeless 2024-04-14T15:15:11.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.394+0000 7f69c5ffb700 10 client.46715.objecter ms_handle_connect 0x7f69b8071400 2024-04-14T15:15:11.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.394+0000 7f69c5ffb700 1 -- 172.21.15.90:0/3652623365 <== mon.2 v1:172.21.15.90:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v243) ==== 54+0+16 (unknown 1148843123 0 2420000192) 0x7f69c008ca00 con 0x7f69c810f300 2024-04-14T15:15:11.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.394+0000 7f69ceac1700 10 client.46715.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:15:11.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.394+0000 7f69ceac1700 15 client.46715.objecter _session_command_op_assign -1 1 2024-04-14T15:15:11.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.394+0000 7f69ceac1700 20 client.46715.objecter _get_session s=0x7f69c81132d0 osd=0 3 2024-04-14T15:15:11.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.394+0000 7f69ceac1700 20 client.46715.objecter put_session s=0x7f69c81132d0 osd=0 3 2024-04-14T15:15:11.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.394+0000 7f69ceac1700 20 client.46715.objecter _get_session s=0x7f69c81132d0 osd=0 3 2024-04-14T15:15:11.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.394+0000 7f69ceac1700 15 client.46715.objecter _session_command_op_remove -1 1 2024-04-14T15:15:11.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.394+0000 7f69ceac1700 20 client.46715.objecter get_session s=0x7f69c81132d0 osd=0 3 2024-04-14T15:15:11.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.394+0000 7f69ceac1700 15 client.46715.objecter _session_command_op_assign 0 1 2024-04-14T15:15:11.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.394+0000 7f69ceac1700 20 client.46715.objecter put_session s=0x7f69c81132d0 osd=0 4 2024-04-14T15:15:11.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.394+0000 7f69ceac1700 10 client.46715.objecter _send_command 1 2024-04-14T15:15:11.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.394+0000 7f69ceac1700 1 -- 172.21.15.90:0/3652623365 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f69c80762a0 con 0x7f69c81ab680 2024-04-14T15:15:11.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.394+0000 7f69c5ffb700 10 client.46715.objecter ms_handle_connect 0x7f69c81ab680 2024-04-14T15:15:11.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.395+0000 7f69c5ffb700 1 -- 172.21.15.90:0/3652623365 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f69c80762a0 con 0x7f69c81ab680 2024-04-14T15:15:11.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.395+0000 7f69c5ffb700 10 client.46715.objecter ms_dispatch 0x7f69c8001550 command_reply(tid 1: 0 ) 2024-04-14T15:15:11.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.395+0000 7f69c5ffb700 10 client.46715.objecter _finish_command 1 = system:0 2024-04-14T15:15:11.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.395+0000 7f69c5ffb700 20 client.46715.objecter put_session s=0x7f69c81132d0 osd=0 4 2024-04-14T15:15:11.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.395+0000 7f69c5ffb700 15 client.46715.objecter _session_command_op_remove 0 1 2024-04-14T15:15:11.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.412+0000 7f69ceac1700 10 client.46715.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:15:11.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.412+0000 7f69ceac1700 15 client.46715.objecter _session_command_op_assign -1 2 2024-04-14T15:15:11.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.412+0000 7f69ceac1700 20 client.46715.objecter _get_session s=0x7f69c81132d0 osd=0 3 2024-04-14T15:15:11.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.412+0000 7f69ceac1700 20 client.46715.objecter put_session s=0x7f69c81132d0 osd=0 3 2024-04-14T15:15:11.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.412+0000 7f69ceac1700 20 client.46715.objecter _get_session s=0x7f69c81132d0 osd=0 3 2024-04-14T15:15:11.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.412+0000 7f69ceac1700 15 client.46715.objecter _session_command_op_remove -1 2 2024-04-14T15:15:11.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.412+0000 7f69ceac1700 20 client.46715.objecter get_session s=0x7f69c81132d0 osd=0 3 2024-04-14T15:15:11.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.412+0000 7f69ceac1700 15 client.46715.objecter _session_command_op_assign 0 2 2024-04-14T15:15:11.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.412+0000 7f69ceac1700 20 client.46715.objecter put_session s=0x7f69c81132d0 osd=0 4 2024-04-14T15:15:11.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.412+0000 7f69ceac1700 10 client.46715.objecter _send_command 2 2024-04-14T15:15:11.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.412+0000 7f69ceac1700 1 -- 172.21.15.90:0/3652623365 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f69c80693d0 con 0x7f69c81ab680 2024-04-14T15:15:11.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.414+0000 7f69c5ffb700 1 -- 172.21.15.90:0/3652623365 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f69c80693d0 con 0x7f69c81ab680 2024-04-14T15:15:11.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.414+0000 7f69c5ffb700 10 client.46715.objecter ms_dispatch 0x7f69c8001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:15:11.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.414+0000 7f69c5ffb700 10 client.46715.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:15:11.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.414+0000 7f69c5ffb700 20 client.46715.objecter put_session s=0x7f69c81132d0 osd=0 4 2024-04-14T15:15:11.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.414+0000 7f69c5ffb700 15 client.46715.objecter _session_command_op_remove 0 2 2024-04-14T15:15:11.415 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:15:11.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.414+0000 7f69ceac1700 10 client.46715.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:15:11.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.414+0000 7f69ceac1700 15 client.46715.objecter _session_command_op_assign -1 3 2024-04-14T15:15:11.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.414+0000 7f69ceac1700 20 client.46715.objecter _get_session s=0x7f69c81aac00 osd=1 3 2024-04-14T15:15:11.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.414+0000 7f69ceac1700 20 client.46715.objecter put_session s=0x7f69c81aac00 osd=1 3 2024-04-14T15:15:11.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.414+0000 7f69ceac1700 20 client.46715.objecter _get_session s=0x7f69c81aac00 osd=1 3 2024-04-14T15:15:11.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.414+0000 7f69ceac1700 15 client.46715.objecter _session_command_op_remove -1 3 2024-04-14T15:15:11.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.414+0000 7f69ceac1700 20 client.46715.objecter get_session s=0x7f69c81aac00 osd=1 3 2024-04-14T15:15:11.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.414+0000 7f69ceac1700 15 client.46715.objecter _session_command_op_assign 1 3 2024-04-14T15:15:11.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.414+0000 7f69ceac1700 20 client.46715.objecter put_session s=0x7f69c81aac00 osd=1 4 2024-04-14T15:15:11.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.414+0000 7f69ceac1700 10 client.46715.objecter _send_command 3 2024-04-14T15:15:11.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.414+0000 7f69ceac1700 1 -- 172.21.15.90:0/3652623365 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f69c81aad70 con 0x7f69c80636c0 2024-04-14T15:15:11.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.414+0000 7f69c5ffb700 10 client.46715.objecter ms_handle_connect 0x7f69c80636c0 2024-04-14T15:15:11.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.415+0000 7f69c5ffb700 1 -- 172.21.15.90:0/3652623365 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f69bc003810 con 0x7f69c80636c0 2024-04-14T15:15:11.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.415+0000 7f69c5ffb700 10 client.46715.objecter ms_dispatch 0x7f69c8001550 command_reply(tid 3: 0 ) 2024-04-14T15:15:11.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.415+0000 7f69c5ffb700 10 client.46715.objecter _finish_command 3 = system:0 2024-04-14T15:15:11.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.415+0000 7f69c5ffb700 20 client.46715.objecter put_session s=0x7f69c81aac00 osd=1 4 2024-04-14T15:15:11.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.415+0000 7f69c5ffb700 15 client.46715.objecter _session_command_op_remove 1 3 2024-04-14T15:15:11.417 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:15:11.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.437+0000 7f69ceac1700 10 client.46715.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:15:11.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.437+0000 7f69ceac1700 15 client.46715.objecter _session_command_op_assign -1 4 2024-04-14T15:15:11.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.437+0000 7f69ceac1700 20 client.46715.objecter _get_session s=0x7f69c81aac00 osd=1 3 2024-04-14T15:15:11.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.437+0000 7f69ceac1700 20 client.46715.objecter put_session s=0x7f69c81aac00 osd=1 3 2024-04-14T15:15:11.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.437+0000 7f69ceac1700 20 client.46715.objecter _get_session s=0x7f69c81aac00 osd=1 3 2024-04-14T15:15:11.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.437+0000 7f69ceac1700 15 client.46715.objecter _session_command_op_remove -1 4 2024-04-14T15:15:11.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.437+0000 7f69ceac1700 20 client.46715.objecter get_session s=0x7f69c81aac00 osd=1 3 2024-04-14T15:15:11.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.437+0000 7f69ceac1700 15 client.46715.objecter _session_command_op_assign 1 4 2024-04-14T15:15:11.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.437+0000 7f69ceac1700 20 client.46715.objecter put_session s=0x7f69c81aac00 osd=1 4 2024-04-14T15:15:11.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.437+0000 7f69ceac1700 10 client.46715.objecter _send_command 4 2024-04-14T15:15:11.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.437+0000 7f69ceac1700 1 -- 172.21.15.90:0/3652623365 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f69c810ae20 con 0x7f69c80636c0 2024-04-14T15:15:11.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.438+0000 7f69c5ffb700 1 -- 172.21.15.90:0/3652623365 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f69c810ae20 con 0x7f69c80636c0 2024-04-14T15:15:11.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.438+0000 7f69c5ffb700 10 client.46715.objecter ms_dispatch 0x7f69c8001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:15:11.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.438+0000 7f69c5ffb700 10 client.46715.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:15:11.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.438+0000 7f69c5ffb700 20 client.46715.objecter put_session s=0x7f69c81aac00 osd=1 4 2024-04-14T15:15:11.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.438+0000 7f69c5ffb700 15 client.46715.objecter _session_command_op_remove 1 4 2024-04-14T15:15:11.439 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:15:11.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.439+0000 7f69ceac1700 10 client.46715.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:15:11.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.439+0000 7f69ceac1700 15 client.46715.objecter _session_command_op_assign -1 5 2024-04-14T15:15:11.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.439+0000 7f69ceac1700 20 client.46715.objecter _get_session s=0x7f69c81b16d0 osd=2 3 2024-04-14T15:15:11.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.439+0000 7f69ceac1700 20 client.46715.objecter put_session s=0x7f69c81b16d0 osd=2 3 2024-04-14T15:15:11.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.439+0000 7f69ceac1700 20 client.46715.objecter _get_session s=0x7f69c81b16d0 osd=2 3 2024-04-14T15:15:11.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.439+0000 7f69ceac1700 15 client.46715.objecter _session_command_op_remove -1 5 2024-04-14T15:15:11.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.439+0000 7f69ceac1700 20 client.46715.objecter get_session s=0x7f69c81b16d0 osd=2 3 2024-04-14T15:15:11.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.439+0000 7f69ceac1700 15 client.46715.objecter _session_command_op_assign 2 5 2024-04-14T15:15:11.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.439+0000 7f69ceac1700 20 client.46715.objecter put_session s=0x7f69c81b16d0 osd=2 4 2024-04-14T15:15:11.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.439+0000 7f69ceac1700 10 client.46715.objecter _send_command 5 2024-04-14T15:15:11.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.439+0000 7f69ceac1700 1 -- 172.21.15.90:0/3652623365 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f69c81b5960 con 0x7f69c81b1d50 2024-04-14T15:15:11.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.439+0000 7f69c5ffb700 10 client.46715.objecter ms_handle_connect 0x7f69c81b1d50 2024-04-14T15:15:11.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.440+0000 7f69c5ffb700 1 -- 172.21.15.90:0/3652623365 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f69c81b5960 con 0x7f69c81b1d50 2024-04-14T15:15:11.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.440+0000 7f69c5ffb700 10 client.46715.objecter ms_dispatch 0x7f69c8001550 command_reply(tid 5: 0 ) 2024-04-14T15:15:11.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.440+0000 7f69c5ffb700 10 client.46715.objecter _finish_command 5 = system:0 2024-04-14T15:15:11.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.440+0000 7f69c5ffb700 20 client.46715.objecter put_session s=0x7f69c81b16d0 osd=2 4 2024-04-14T15:15:11.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.440+0000 7f69c5ffb700 15 client.46715.objecter _session_command_op_remove 2 5 2024-04-14T15:15:11.442 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:11.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:11.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:15:11.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.456+0000 7f69ceac1700 10 client.46715.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:15:11.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.456+0000 7f69ceac1700 15 client.46715.objecter _session_command_op_assign -1 6 2024-04-14T15:15:11.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.456+0000 7f69ceac1700 20 client.46715.objecter _get_session s=0x7f69c81b16d0 osd=2 3 2024-04-14T15:15:11.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.456+0000 7f69ceac1700 20 client.46715.objecter put_session s=0x7f69c81b16d0 osd=2 3 2024-04-14T15:15:11.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.456+0000 7f69ceac1700 20 client.46715.objecter _get_session s=0x7f69c81b16d0 osd=2 3 2024-04-14T15:15:11.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.456+0000 7f69ceac1700 15 client.46715.objecter _session_command_op_remove -1 6 2024-04-14T15:15:11.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.456+0000 7f69ceac1700 20 client.46715.objecter get_session s=0x7f69c81b16d0 osd=2 3 2024-04-14T15:15:11.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.456+0000 7f69ceac1700 15 client.46715.objecter _session_command_op_assign 2 6 2024-04-14T15:15:11.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.456+0000 7f69ceac1700 20 client.46715.objecter put_session s=0x7f69c81b16d0 osd=2 4 2024-04-14T15:15:11.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.456+0000 7f69ceac1700 10 client.46715.objecter _send_command 6 2024-04-14T15:15:11.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.456+0000 7f69ceac1700 1 -- 172.21.15.90:0/3652623365 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f69c80631a0 con 0x7f69c81b1d50 2024-04-14T15:15:11.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.458+0000 7f69c5ffb700 1 -- 172.21.15.90:0/3652623365 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f69c80631a0 con 0x7f69c81b1d50 2024-04-14T15:15:11.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.458+0000 7f69c5ffb700 10 client.46715.objecter ms_dispatch 0x7f69c8001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:15:11.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.458+0000 7f69c5ffb700 10 client.46715.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:15:11.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.458+0000 7f69c5ffb700 20 client.46715.objecter put_session s=0x7f69c81b16d0 osd=2 4 2024-04-14T15:15:11.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.458+0000 7f69c5ffb700 15 client.46715.objecter _session_command_op_remove 2 6 2024-04-14T15:15:11.459 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:15:11.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.458+0000 7f69ceac1700 10 client.46715.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:15:11.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.458+0000 7f69ceac1700 15 client.46715.objecter _session_command_op_assign -1 7 2024-04-14T15:15:11.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.458+0000 7f69ceac1700 20 client.46715.objecter _get_session s=0x7f69c81b6cf0 osd=3 3 2024-04-14T15:15:11.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.458+0000 7f69ceac1700 20 client.46715.objecter put_session s=0x7f69c81b6cf0 osd=3 3 2024-04-14T15:15:11.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.458+0000 7f69ceac1700 20 client.46715.objecter _get_session s=0x7f69c81b6cf0 osd=3 3 2024-04-14T15:15:11.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.458+0000 7f69ceac1700 15 client.46715.objecter _session_command_op_remove -1 7 2024-04-14T15:15:11.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.458+0000 7f69ceac1700 20 client.46715.objecter get_session s=0x7f69c81b6cf0 osd=3 3 2024-04-14T15:15:11.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.458+0000 7f69ceac1700 15 client.46715.objecter _session_command_op_assign 3 7 2024-04-14T15:15:11.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.458+0000 7f69ceac1700 20 client.46715.objecter put_session s=0x7f69c81b6cf0 osd=3 4 2024-04-14T15:15:11.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.458+0000 7f69ceac1700 10 client.46715.objecter _send_command 7 2024-04-14T15:15:11.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.458+0000 7f69ceac1700 1 -- 172.21.15.90:0/3652623365 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f69c81b6f20 con 0x7f69c802db00 2024-04-14T15:15:11.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.459+0000 7f69c5ffb700 10 client.46715.objecter ms_handle_connect 0x7f69c802db00 2024-04-14T15:15:11.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.459+0000 7f69c5ffb700 1 -- 172.21.15.90:0/3652623365 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f69c81b6f20 con 0x7f69c802db00 2024-04-14T15:15:11.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.459+0000 7f69c5ffb700 10 client.46715.objecter ms_dispatch 0x7f69c8001550 command_reply(tid 7: 0 ) 2024-04-14T15:15:11.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.459+0000 7f69c5ffb700 10 client.46715.objecter _finish_command 7 = system:0 2024-04-14T15:15:11.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.459+0000 7f69c5ffb700 20 client.46715.objecter put_session s=0x7f69c81b6cf0 osd=3 4 2024-04-14T15:15:11.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.459+0000 7f69c5ffb700 15 client.46715.objecter _session_command_op_remove 3 7 2024-04-14T15:15:11.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.476+0000 7f69ceac1700 10 client.46715.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:15:11.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.476+0000 7f69ceac1700 15 client.46715.objecter _session_command_op_assign -1 8 2024-04-14T15:15:11.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.476+0000 7f69ceac1700 20 client.46715.objecter _get_session s=0x7f69c81b6cf0 osd=3 3 2024-04-14T15:15:11.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.476+0000 7f69ceac1700 20 client.46715.objecter put_session s=0x7f69c81b6cf0 osd=3 3 2024-04-14T15:15:11.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.476+0000 7f69ceac1700 20 client.46715.objecter _get_session s=0x7f69c81b6cf0 osd=3 3 2024-04-14T15:15:11.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.476+0000 7f69ceac1700 15 client.46715.objecter _session_command_op_remove -1 8 2024-04-14T15:15:11.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.476+0000 7f69ceac1700 20 client.46715.objecter get_session s=0x7f69c81b6cf0 osd=3 3 2024-04-14T15:15:11.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.476+0000 7f69ceac1700 15 client.46715.objecter _session_command_op_assign 3 8 2024-04-14T15:15:11.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.476+0000 7f69ceac1700 20 client.46715.objecter put_session s=0x7f69c81b6cf0 osd=3 4 2024-04-14T15:15:11.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.476+0000 7f69ceac1700 10 client.46715.objecter _send_command 8 2024-04-14T15:15:11.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.476+0000 7f69ceac1700 1 -- 172.21.15.90:0/3652623365 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f69c802d070 con 0x7f69c802db00 2024-04-14T15:15:11.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.478+0000 7f69c5ffb700 1 -- 172.21.15.90:0/3652623365 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f69c802d070 con 0x7f69c802db00 2024-04-14T15:15:11.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.478+0000 7f69c5ffb700 10 client.46715.objecter ms_dispatch 0x7f69c8001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:15:11.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.478+0000 7f69c5ffb700 10 client.46715.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:15:11.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.478+0000 7f69c5ffb700 20 client.46715.objecter put_session s=0x7f69c81b6cf0 osd=3 4 2024-04-14T15:15:11.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.478+0000 7f69c5ffb700 15 client.46715.objecter _session_command_op_remove 3 8 2024-04-14T15:15:11.479 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:15:11.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.478+0000 7f69ceac1700 10 client.46715.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:15:11.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.478+0000 7f69ceac1700 15 client.46715.objecter _session_command_op_assign -1 9 2024-04-14T15:15:11.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.478+0000 7f69ceac1700 20 client.46715.objecter _get_session s=0x7f69c802d070 osd=4 3 2024-04-14T15:15:11.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.478+0000 7f69ceac1700 20 client.46715.objecter put_session s=0x7f69c802d070 osd=4 3 2024-04-14T15:15:11.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.478+0000 7f69ceac1700 20 client.46715.objecter _get_session s=0x7f69c802d070 osd=4 3 2024-04-14T15:15:11.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.478+0000 7f69ceac1700 15 client.46715.objecter _session_command_op_remove -1 9 2024-04-14T15:15:11.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.478+0000 7f69ceac1700 20 client.46715.objecter get_session s=0x7f69c802d070 osd=4 3 2024-04-14T15:15:11.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.478+0000 7f69ceac1700 15 client.46715.objecter _session_command_op_assign 4 9 2024-04-14T15:15:11.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.478+0000 7f69ceac1700 20 client.46715.objecter put_session s=0x7f69c802d070 osd=4 4 2024-04-14T15:15:11.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.478+0000 7f69ceac1700 10 client.46715.objecter _send_command 9 2024-04-14T15:15:11.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.478+0000 7f69ceac1700 1 -- 172.21.15.90:0/3652623365 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f69c81b6980 con 0x7f69c802d1e0 2024-04-14T15:15:11.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.478+0000 7f69c5ffb700 10 client.46715.objecter ms_handle_connect 0x7f69c802d1e0 2024-04-14T15:15:11.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.479+0000 7f69c5ffb700 1 -- 172.21.15.90:0/3652623365 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f69c81b6980 con 0x7f69c802d1e0 2024-04-14T15:15:11.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.479+0000 7f69c5ffb700 10 client.46715.objecter ms_dispatch 0x7f69c8001550 command_reply(tid 9: 0 ) 2024-04-14T15:15:11.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.479+0000 7f69c5ffb700 10 client.46715.objecter _finish_command 9 = system:0 2024-04-14T15:15:11.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.479+0000 7f69c5ffb700 20 client.46715.objecter put_session s=0x7f69c802d070 osd=4 4 2024-04-14T15:15:11.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.479+0000 7f69c5ffb700 15 client.46715.objecter _session_command_op_remove 4 9 2024-04-14T15:15:11.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.496+0000 7f69ceac1700 10 client.46715.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:15:11.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.496+0000 7f69ceac1700 15 client.46715.objecter _session_command_op_assign -1 10 2024-04-14T15:15:11.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.496+0000 7f69ceac1700 20 client.46715.objecter _get_session s=0x7f69c802d070 osd=4 3 2024-04-14T15:15:11.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.496+0000 7f69ceac1700 20 client.46715.objecter put_session s=0x7f69c802d070 osd=4 3 2024-04-14T15:15:11.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.496+0000 7f69ceac1700 20 client.46715.objecter _get_session s=0x7f69c802d070 osd=4 3 2024-04-14T15:15:11.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.496+0000 7f69ceac1700 15 client.46715.objecter _session_command_op_remove -1 10 2024-04-14T15:15:11.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.496+0000 7f69ceac1700 20 client.46715.objecter get_session s=0x7f69c802d070 osd=4 3 2024-04-14T15:15:11.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.496+0000 7f69ceac1700 15 client.46715.objecter _session_command_op_assign 4 10 2024-04-14T15:15:11.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.496+0000 7f69ceac1700 20 client.46715.objecter put_session s=0x7f69c802d070 osd=4 4 2024-04-14T15:15:11.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.496+0000 7f69ceac1700 10 client.46715.objecter _send_command 10 2024-04-14T15:15:11.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.496+0000 7f69ceac1700 1 -- 172.21.15.90:0/3652623365 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f69c81b6980 con 0x7f69c802d1e0 2024-04-14T15:15:11.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.498+0000 7f69c5ffb700 1 -- 172.21.15.90:0/3652623365 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f69c81b6980 con 0x7f69c802d1e0 2024-04-14T15:15:11.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.498+0000 7f69c5ffb700 10 client.46715.objecter ms_dispatch 0x7f69c8001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:15:11.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.498+0000 7f69c5ffb700 10 client.46715.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:15:11.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.498+0000 7f69c5ffb700 20 client.46715.objecter put_session s=0x7f69c802d070 osd=4 4 2024-04-14T15:15:11.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.498+0000 7f69c5ffb700 15 client.46715.objecter _session_command_op_remove 4 10 2024-04-14T15:15:11.499 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:15:11.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.498+0000 7f69ceac1700 10 client.46715.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:15:11.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.498+0000 7f69ceac1700 15 client.46715.objecter _session_command_op_assign -1 11 2024-04-14T15:15:11.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.498+0000 7f69ceac1700 20 client.46715.objecter _get_session s=0x7f69c81b6980 osd=5 3 2024-04-14T15:15:11.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.498+0000 7f69ceac1700 20 client.46715.objecter put_session s=0x7f69c81b6980 osd=5 3 2024-04-14T15:15:11.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.498+0000 7f69ceac1700 20 client.46715.objecter _get_session s=0x7f69c81b6980 osd=5 3 2024-04-14T15:15:11.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.498+0000 7f69ceac1700 15 client.46715.objecter _session_command_op_remove -1 11 2024-04-14T15:15:11.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.498+0000 7f69ceac1700 20 client.46715.objecter get_session s=0x7f69c81b6980 osd=5 3 2024-04-14T15:15:11.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.498+0000 7f69ceac1700 15 client.46715.objecter _session_command_op_assign 5 11 2024-04-14T15:15:11.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.498+0000 7f69ceac1700 20 client.46715.objecter put_session s=0x7f69c81b6980 osd=5 4 2024-04-14T15:15:11.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.498+0000 7f69ceac1700 10 client.46715.objecter _send_command 11 2024-04-14T15:15:11.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.498+0000 7f69ceac1700 1 -- 172.21.15.90:0/3652623365 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f69c81c1520 con 0x7f69c81bf890 2024-04-14T15:15:11.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.499+0000 7f69c5ffb700 10 client.46715.objecter ms_handle_connect 0x7f69c81bf890 2024-04-14T15:15:11.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.499+0000 7f69c5ffb700 1 -- 172.21.15.90:0/3652623365 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f69c81c1520 con 0x7f69c81bf890 2024-04-14T15:15:11.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.499+0000 7f69c5ffb700 10 client.46715.objecter ms_dispatch 0x7f69c8001550 command_reply(tid 11: 0 ) 2024-04-14T15:15:11.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.499+0000 7f69c5ffb700 10 client.46715.objecter _finish_command 11 = system:0 2024-04-14T15:15:11.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.499+0000 7f69c5ffb700 20 client.46715.objecter put_session s=0x7f69c81b6980 osd=5 4 2024-04-14T15:15:11.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.499+0000 7f69c5ffb700 15 client.46715.objecter _session_command_op_remove 5 11 2024-04-14T15:15:11.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.516+0000 7f69ceac1700 10 client.46715.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:15:11.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.516+0000 7f69ceac1700 15 client.46715.objecter _session_command_op_assign -1 12 2024-04-14T15:15:11.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.516+0000 7f69ceac1700 20 client.46715.objecter _get_session s=0x7f69c81b6980 osd=5 3 2024-04-14T15:15:11.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.516+0000 7f69ceac1700 20 client.46715.objecter put_session s=0x7f69c81b6980 osd=5 3 2024-04-14T15:15:11.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.516+0000 7f69ceac1700 20 client.46715.objecter _get_session s=0x7f69c81b6980 osd=5 3 2024-04-14T15:15:11.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.516+0000 7f69ceac1700 15 client.46715.objecter _session_command_op_remove -1 12 2024-04-14T15:15:11.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.516+0000 7f69ceac1700 20 client.46715.objecter get_session s=0x7f69c81b6980 osd=5 3 2024-04-14T15:15:11.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.516+0000 7f69ceac1700 15 client.46715.objecter _session_command_op_assign 5 12 2024-04-14T15:15:11.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.516+0000 7f69ceac1700 20 client.46715.objecter put_session s=0x7f69c81b6980 osd=5 4 2024-04-14T15:15:11.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.516+0000 7f69ceac1700 10 client.46715.objecter _send_command 12 2024-04-14T15:15:11.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.516+0000 7f69ceac1700 1 -- 172.21.15.90:0/3652623365 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f69c81c1520 con 0x7f69c81bf890 2024-04-14T15:15:11.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.518+0000 7f69c5ffb700 1 -- 172.21.15.90:0/3652623365 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f69c81c1520 con 0x7f69c81bf890 2024-04-14T15:15:11.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.518+0000 7f69c5ffb700 10 client.46715.objecter ms_dispatch 0x7f69c8001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:15:11.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.518+0000 7f69c5ffb700 10 client.46715.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:15:11.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.518+0000 7f69c5ffb700 20 client.46715.objecter put_session s=0x7f69c81b6980 osd=5 4 2024-04-14T15:15:11.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.518+0000 7f69c5ffb700 15 client.46715.objecter _session_command_op_remove 5 12 2024-04-14T15:15:11.519 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:15:11.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.518+0000 7f69ceac1700 10 client.46715.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:15:11.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.518+0000 7f69ceac1700 15 client.46715.objecter _session_command_op_assign -1 13 2024-04-14T15:15:11.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.518+0000 7f69ceac1700 20 client.46715.objecter _get_session s=0x7f69c81c1520 osd=6 3 2024-04-14T15:15:11.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.518+0000 7f69ceac1700 20 client.46715.objecter put_session s=0x7f69c81c1520 osd=6 3 2024-04-14T15:15:11.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.518+0000 7f69ceac1700 20 client.46715.objecter _get_session s=0x7f69c81c1520 osd=6 3 2024-04-14T15:15:11.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.518+0000 7f69ceac1700 15 client.46715.objecter _session_command_op_remove -1 13 2024-04-14T15:15:11.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.518+0000 7f69ceac1700 20 client.46715.objecter get_session s=0x7f69c81c1520 osd=6 3 2024-04-14T15:15:11.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.518+0000 7f69ceac1700 15 client.46715.objecter _session_command_op_assign 6 13 2024-04-14T15:15:11.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.518+0000 7f69ceac1700 20 client.46715.objecter put_session s=0x7f69c81c1520 osd=6 4 2024-04-14T15:15:11.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.518+0000 7f69ceac1700 10 client.46715.objecter _send_command 13 2024-04-14T15:15:11.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.518+0000 7f69ceac1700 1 -- 172.21.15.90:0/3652623365 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f69c81c7a20 con 0x7f69c81c17a0 2024-04-14T15:15:11.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.518+0000 7f69c5ffb700 10 client.46715.objecter ms_handle_connect 0x7f69c81c17a0 2024-04-14T15:15:11.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.519+0000 7f69c5ffb700 1 -- 172.21.15.90:0/3652623365 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f69c81c7a20 con 0x7f69c81c17a0 2024-04-14T15:15:11.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.519+0000 7f69c5ffb700 10 client.46715.objecter ms_dispatch 0x7f69c8001550 command_reply(tid 13: 0 ) 2024-04-14T15:15:11.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.519+0000 7f69c5ffb700 10 client.46715.objecter _finish_command 13 = system:0 2024-04-14T15:15:11.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.519+0000 7f69c5ffb700 20 client.46715.objecter put_session s=0x7f69c81c1520 osd=6 4 2024-04-14T15:15:11.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.519+0000 7f69c5ffb700 15 client.46715.objecter _session_command_op_remove 6 13 2024-04-14T15:15:11.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.538+0000 7f69ceac1700 10 client.46715.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:15:11.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.538+0000 7f69ceac1700 15 client.46715.objecter _session_command_op_assign -1 14 2024-04-14T15:15:11.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.538+0000 7f69ceac1700 20 client.46715.objecter _get_session s=0x7f69c81c1520 osd=6 3 2024-04-14T15:15:11.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.538+0000 7f69ceac1700 20 client.46715.objecter put_session s=0x7f69c81c1520 osd=6 3 2024-04-14T15:15:11.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.538+0000 7f69ceac1700 20 client.46715.objecter _get_session s=0x7f69c81c1520 osd=6 3 2024-04-14T15:15:11.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.538+0000 7f69ceac1700 15 client.46715.objecter _session_command_op_remove -1 14 2024-04-14T15:15:11.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.538+0000 7f69ceac1700 20 client.46715.objecter get_session s=0x7f69c81c1520 osd=6 3 2024-04-14T15:15:11.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.538+0000 7f69ceac1700 15 client.46715.objecter _session_command_op_assign 6 14 2024-04-14T15:15:11.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.538+0000 7f69ceac1700 20 client.46715.objecter put_session s=0x7f69c81c1520 osd=6 4 2024-04-14T15:15:11.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.538+0000 7f69ceac1700 10 client.46715.objecter _send_command 14 2024-04-14T15:15:11.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.538+0000 7f69ceac1700 1 -- 172.21.15.90:0/3652623365 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f69c802ce80 con 0x7f69c81c17a0 2024-04-14T15:15:11.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.540+0000 7f69c5ffb700 1 -- 172.21.15.90:0/3652623365 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f69c802ce80 con 0x7f69c81c17a0 2024-04-14T15:15:11.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.540+0000 7f69c5ffb700 10 client.46715.objecter ms_dispatch 0x7f69c8001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:15:11.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.540+0000 7f69c5ffb700 10 client.46715.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:15:11.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.540+0000 7f69c5ffb700 20 client.46715.objecter put_session s=0x7f69c81c1520 osd=6 4 2024-04-14T15:15:11.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.540+0000 7f69c5ffb700 15 client.46715.objecter _session_command_op_remove 6 14 2024-04-14T15:15:11.541 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:15:11.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.540+0000 7f69ceac1700 10 client.46715.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:15:11.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.540+0000 7f69ceac1700 15 client.46715.objecter _session_command_op_assign -1 15 2024-04-14T15:15:11.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.540+0000 7f69ceac1700 20 client.46715.objecter _get_session s=0x7f69c802cec0 osd=7 3 2024-04-14T15:15:11.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.540+0000 7f69ceac1700 20 client.46715.objecter put_session s=0x7f69c802cec0 osd=7 3 2024-04-14T15:15:11.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.540+0000 7f69ceac1700 20 client.46715.objecter _get_session s=0x7f69c802cec0 osd=7 3 2024-04-14T15:15:11.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.540+0000 7f69ceac1700 15 client.46715.objecter _session_command_op_remove -1 15 2024-04-14T15:15:11.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.540+0000 7f69ceac1700 20 client.46715.objecter get_session s=0x7f69c802cec0 osd=7 3 2024-04-14T15:15:11.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.540+0000 7f69ceac1700 15 client.46715.objecter _session_command_op_assign 7 15 2024-04-14T15:15:11.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.540+0000 7f69ceac1700 20 client.46715.objecter put_session s=0x7f69c802cec0 osd=7 4 2024-04-14T15:15:11.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.540+0000 7f69ceac1700 10 client.46715.objecter _send_command 15 2024-04-14T15:15:11.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.540+0000 7f69ceac1700 1 -- 172.21.15.90:0/3652623365 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f69c81c8fc0 con 0x7f69c81c8b60 2024-04-14T15:15:11.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.540+0000 7f69c5ffb700 10 client.46715.objecter ms_handle_connect 0x7f69c81c8b60 2024-04-14T15:15:11.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.541+0000 7f69c5ffb700 1 -- 172.21.15.90:0/3652623365 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f69c81c8fc0 con 0x7f69c81c8b60 2024-04-14T15:15:11.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.541+0000 7f69c5ffb700 10 client.46715.objecter ms_dispatch 0x7f69c8001550 command_reply(tid 15: 0 ) 2024-04-14T15:15:11.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.541+0000 7f69c5ffb700 10 client.46715.objecter _finish_command 15 = system:0 2024-04-14T15:15:11.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.541+0000 7f69c5ffb700 20 client.46715.objecter put_session s=0x7f69c802cec0 osd=7 4 2024-04-14T15:15:11.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.541+0000 7f69c5ffb700 15 client.46715.objecter _session_command_op_remove 7 15 2024-04-14T15:15:11.544 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:15:11.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.558+0000 7f69ceac1700 10 client.46715.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:15:11.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.558+0000 7f69ceac1700 15 client.46715.objecter _session_command_op_assign -1 16 2024-04-14T15:15:11.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.558+0000 7f69ceac1700 20 client.46715.objecter _get_session s=0x7f69c802cec0 osd=7 3 2024-04-14T15:15:11.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.558+0000 7f69ceac1700 20 client.46715.objecter put_session s=0x7f69c802cec0 osd=7 3 2024-04-14T15:15:11.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.558+0000 7f69ceac1700 20 client.46715.objecter _get_session s=0x7f69c802cec0 osd=7 3 2024-04-14T15:15:11.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.558+0000 7f69ceac1700 15 client.46715.objecter _session_command_op_remove -1 16 2024-04-14T15:15:11.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.558+0000 7f69ceac1700 20 client.46715.objecter get_session s=0x7f69c802cec0 osd=7 3 2024-04-14T15:15:11.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.558+0000 7f69ceac1700 15 client.46715.objecter _session_command_op_assign 7 16 2024-04-14T15:15:11.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.558+0000 7f69ceac1700 20 client.46715.objecter put_session s=0x7f69c802cec0 osd=7 4 2024-04-14T15:15:11.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.558+0000 7f69ceac1700 10 client.46715.objecter _send_command 16 2024-04-14T15:15:11.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.558+0000 7f69ceac1700 1 -- 172.21.15.90:0/3652623365 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f69c81c8fc0 con 0x7f69c81c8b60 2024-04-14T15:15:11.561 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:15:11.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.559+0000 7f69c5ffb700 1 -- 172.21.15.90:0/3652623365 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f69c81c8fc0 con 0x7f69c81c8b60 2024-04-14T15:15:11.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.559+0000 7f69c5ffb700 10 client.46715.objecter ms_dispatch 0x7f69c8001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:15:11.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.559+0000 7f69c5ffb700 10 client.46715.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:15:11.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.559+0000 7f69c5ffb700 20 client.46715.objecter put_session s=0x7f69c802cec0 osd=7 4 2024-04-14T15:15:11.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.559+0000 7f69c5ffb700 15 client.46715.objecter _session_command_op_remove 7 16 2024-04-14T15:15:11.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.560+0000 7f69ceac1700 10 librados: watch_flush enter 2024-04-14T15:15:11.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.560+0000 7f69ceac1700 10 librados: watch_flush exit 2024-04-14T15:15:11.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.560+0000 7f69ceac1700 10 client.46715.objecter close_session for osd.0 2024-04-14T15:15:11.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.560+0000 7f69ceac1700 1 -- 172.21.15.90:0/3652623365 >> v1:172.21.15.90:6801/3761806885 conn(0x7f69c81ab680 legacy=0x7f69c81aba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:11.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.560+0000 7f69ceac1700 20 client.46715.objecter put_session s=0x7f69c81132d0 osd=0 1 2024-04-14T15:15:11.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.560+0000 7f69ceac1700 10 client.46715.objecter close_session for osd.1 2024-04-14T15:15:11.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.560+0000 7f69ceac1700 1 -- 172.21.15.90:0/3652623365 >> v1:172.21.15.90:6805/4170361699 conn(0x7f69c80636c0 legacy=0x7f69c8109700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:11.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.560+0000 7f69ceac1700 20 client.46715.objecter put_session s=0x7f69c81aac00 osd=1 1 2024-04-14T15:15:11.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.560+0000 7f69ceac1700 10 client.46715.objecter close_session for osd.2 2024-04-14T15:15:11.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.560+0000 7f69ceac1700 1 -- 172.21.15.90:0/3652623365 >> v1:172.21.15.90:6809/1686184251 conn(0x7f69c81b1d50 legacy=0x7f69c81b4140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:11.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.560+0000 7f69cd05e700 1 -- 172.21.15.90:0/3652623365 reap_dead start 2024-04-14T15:15:11.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.560+0000 7f69ceac1700 20 client.46715.objecter put_session s=0x7f69c81b16d0 osd=2 1 2024-04-14T15:15:11.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.560+0000 7f69ceac1700 10 client.46715.objecter close_session for osd.3 2024-04-14T15:15:11.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.560+0000 7f69ceac1700 1 -- 172.21.15.90:0/3652623365 >> v1:172.21.15.90:6813/2270166774 conn(0x7f69c802db00 legacy=0x7f69c81b9350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:11.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.560+0000 7f69ceac1700 20 client.46715.objecter put_session s=0x7f69c81b6cf0 osd=3 1 2024-04-14T15:15:11.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.560+0000 7f69ceac1700 10 client.46715.objecter close_session for osd.4 2024-04-14T15:15:11.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.560+0000 7f69ceac1700 1 -- 172.21.15.90:0/3652623365 >> v1:172.21.15.92:6800/3887549733 conn(0x7f69c802d1e0 legacy=0x7f69c81bdc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:11.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.560+0000 7f69ceac1700 20 client.46715.objecter put_session s=0x7f69c802d070 osd=4 1 2024-04-14T15:15:11.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.560+0000 7f69ceac1700 10 client.46715.objecter close_session for osd.5 2024-04-14T15:15:11.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.560+0000 7f69ceac1700 1 -- 172.21.15.90:0/3652623365 >> v1:172.21.15.92:6804/3244561440 conn(0x7f69c81bf890 legacy=0x7f69c81bfcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:11.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.560+0000 7f69ceac1700 20 client.46715.objecter put_session s=0x7f69c81b6980 osd=5 1 2024-04-14T15:15:11.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.560+0000 7f69ceac1700 10 client.46715.objecter close_session for osd.6 2024-04-14T15:15:11.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.560+0000 7f69ceac1700 1 -- 172.21.15.90:0/3652623365 >> v1:172.21.15.92:6808/636125963 conn(0x7f69c81c17a0 legacy=0x7f69c81c6220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:11.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.560+0000 7f69ceac1700 20 client.46715.objecter put_session s=0x7f69c81c1520 osd=6 1 2024-04-14T15:15:11.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.560+0000 7f69ceac1700 10 client.46715.objecter close_session for osd.7 2024-04-14T15:15:11.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.560+0000 7f69ceac1700 1 -- 172.21.15.90:0/3652623365 >> v1:172.21.15.92:6812/1773706716 conn(0x7f69c81c8b60 legacy=0x7f69c81bb240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:11.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.560+0000 7f69cd05e700 1 -- 172.21.15.90:0/3652623365 reap_dead start 2024-04-14T15:15:11.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.560+0000 7f69ceac1700 20 client.46715.objecter put_session s=0x7f69c802cec0 osd=7 1 2024-04-14T15:15:11.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.560+0000 7f69ceac1700 20 client.46715.objecter shutdown clearing up homeless session... 2024-04-14T15:15:11.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.560+0000 7f69ceac1700 10 client.46715.objecter successfully canceled tick 2024-04-14T15:15:11.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.560+0000 7f69ceac1700 1 -- 172.21.15.90:0/3652623365 >> v1:172.21.15.90:6800/420935427 conn(0x7f69b8071400 legacy=0x7f69b80738c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:11.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.560+0000 7f69ceac1700 1 -- 172.21.15.90:0/3652623365 >> v1:172.21.15.90:6790/0 conn(0x7f69c810f300 legacy=0x7f69c81a8440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:11.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.560+0000 7f69ceac1700 1 -- 172.21.15.90:0/3652623365 shutdown_connections 2024-04-14T15:15:11.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.560+0000 7f69ceac1700 1 -- 172.21.15.90:0/3652623365 >> 172.21.15.90:0/3652623365 conn(0x7f69c80fe3d0 msgr2=0x7f69c8102880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:15:11.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.560+0000 7f69ceac1700 1 -- 172.21.15.90:0/3652623365 shutdown_connections 2024-04-14T15:15:11.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.560+0000 7f69ceac1700 1 -- 172.21.15.90:0/3652623365 wait complete. 2024-04-14T15:15:11.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:11.560+0000 7f69ceac1700 1 librados: shutdown 2024-04-14T15:15:11.567 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:15:11.568 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:15:11.568 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:15:11.568 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:15:11.568 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:15:11.568 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:15:11.568 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:15:11.568 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:15:11.569 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:11.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:11.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:15:11.671 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:15:11.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:11.696 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:11.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:15:11.797 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:15:11.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:11.823 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:11.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:15:11.924 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:15:11.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:11.949 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:11.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:15:12.053 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:15:12.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:12.079 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:12.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:15:12.180 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:15:12.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:11 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:12.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:11 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3652623365' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:15:12.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:11 smithi090 ceph-mon[41670]: from='client.46709 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi090-93363-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:15:12.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:11 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:15:12.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:11 smithi090 ceph-mon[41670]: osdmap e244: 8 total, 8 up, 8 in 2024-04-14T15:15:12.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:11 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-25"}]: dispatch 2024-04-14T15:15:12.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:11 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-25"}]: dispatch 2024-04-14T15:15:12.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:11 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:12.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:11 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3652623365' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:15:12.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:11 smithi090 ceph-mon[30399]: from='client.46709 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi090-93363-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:15:12.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:11 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:15:12.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:11 smithi090 ceph-mon[30399]: osdmap e244: 8 total, 8 up, 8 in 2024-04-14T15:15:12.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:11 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-25"}]: dispatch 2024-04-14T15:15:12.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:11 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-25"}]: dispatch 2024-04-14T15:15:12.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:12.206 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:12.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:15:12.307 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:15:12.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:12.333 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:12.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:15:12.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:11 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:12.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:11 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3652623365' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:15:12.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:11 smithi092 ceph-mon[33966]: from='client.46709 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi090-93363-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:15:12.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:11 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:15:12.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:11 smithi092 ceph-mon[33966]: osdmap e244: 8 total, 8 up, 8 in 2024-04-14T15:15:12.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:11 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-25"}]: dispatch 2024-04-14T15:15:12.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:11 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-25"}]: dispatch 2024-04-14T15:15:12.434 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:15:12.458 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:12.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:12.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:15:12.560 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:15:12.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:12.585 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:12.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:15:12.686 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:15:12.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:12.711 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:12.711 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:15:12.812 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:15:12.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:12.837 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:12.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:15:12.938 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:15:12.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:12.963 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:12.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:15:13.064 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:15:13.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:13.090 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:13.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:15:13.171 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:12 smithi090 ceph-mon[41670]: pgmap v311: 386 pgs: 40 unknown, 32 creating+peering, 314 active+clean; 586 KiB data, 545 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-04-14T15:15:13.171 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:12 smithi090 ceph-mon[41670]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:15:13.171 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:12 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:13.171 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:12 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:15:13.171 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:12 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:15:13.171 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:12 smithi090 ceph-mon[41670]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:15:13.171 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:12 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-25"}]': finished 2024-04-14T15:15:13.171 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:12 smithi090 ceph-mon[41670]: osdmap e245: 8 total, 8 up, 8 in 2024-04-14T15:15:13.171 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:12 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2466564286' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi090-93342-35"}]: dispatch 2024-04-14T15:15:13.171 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:12 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2045583521' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi090-93363-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:13.172 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:12 smithi090 ceph-mon[30399]: pgmap v311: 386 pgs: 40 unknown, 32 creating+peering, 314 active+clean; 586 KiB data, 545 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-04-14T15:15:13.172 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:12 smithi090 ceph-mon[30399]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:15:13.172 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:12 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:13.172 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:12 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:15:13.172 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:12 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:15:13.172 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:12 smithi090 ceph-mon[30399]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:15:13.172 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:12 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-25"}]': finished 2024-04-14T15:15:13.172 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:12 smithi090 ceph-mon[30399]: osdmap e245: 8 total, 8 up, 8 in 2024-04-14T15:15:13.172 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:12 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2466564286' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi090-93342-35"}]: dispatch 2024-04-14T15:15:13.172 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:12 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2045583521' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi090-93363-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:13.191 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:15:13.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:13.215 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:13.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:15:13.316 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:15:13.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:13.341 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:13.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:15:13.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:12 smithi092 ceph-mon[33966]: pgmap v311: 386 pgs: 40 unknown, 32 creating+peering, 314 active+clean; 586 KiB data, 545 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-04-14T15:15:13.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:12 smithi092 ceph-mon[33966]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:15:13.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:12 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:13.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:12 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:15:13.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:12 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:15:13.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:12 smithi092 ceph-mon[33966]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:15:13.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:12 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-25"}]': finished 2024-04-14T15:15:13.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:12 smithi092 ceph-mon[33966]: osdmap e245: 8 total, 8 up, 8 in 2024-04-14T15:15:13.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:12 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2466564286' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi090-93342-35"}]: dispatch 2024-04-14T15:15:13.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:12 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2045583521' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi090-93363-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:13.441 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:15:13.465 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:13.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:13.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:15:13.567 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:15:13.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:13.592 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:13.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:15:13.694 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:15:13.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:13.719 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:13.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:15:13.820 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:15:13.844 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:13.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:13.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:15:13.946 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:15:13.968 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:15:13.973 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:13.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:13.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:15:14.074 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:15:14.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:14.099 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:14.100 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:15:14.191 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:15:13 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:15:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:15:14.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:13 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:14.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:13 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2466564286' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi090-93342-35"}]': finished 2024-04-14T15:15:14.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:13 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2045583521' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi090-93363-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:15:14.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:13 smithi090 ceph-mon[41670]: osdmap e246: 8 total, 8 up, 8 in 2024-04-14T15:15:14.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:13 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2466564286' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi090-93342-35"}]: dispatch 2024-04-14T15:15:14.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:13 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:14.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:13 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2466564286' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi090-93342-35"}]': finished 2024-04-14T15:15:14.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:13 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2045583521' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi090-93363-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:15:14.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:13 smithi090 ceph-mon[30399]: osdmap e246: 8 total, 8 up, 8 in 2024-04-14T15:15:14.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:13 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2466564286' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi090-93342-35"}]: dispatch 2024-04-14T15:15:14.200 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:15:14.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:14.227 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:14.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:15:14.328 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:15:14.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:14.354 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:14.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:15:14.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:13 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:14.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:13 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2466564286' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi090-93342-35"}]': finished 2024-04-14T15:15:14.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:13 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2045583521' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi090-93363-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:15:14.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:13 smithi092 ceph-mon[33966]: osdmap e246: 8 total, 8 up, 8 in 2024-04-14T15:15:14.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:13 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2466564286' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi090-93342-35"}]: dispatch 2024-04-14T15:15:14.456 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:15:14.481 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:14.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:14.483 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:15:14.503 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd unset noscrub 2024-04-14T15:15:14.583 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:15:14.601 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-14T15:15:14.601 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:15:14.607 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:14.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:14.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:15:14.708 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:15:14.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:14.733 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:14.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:15:14.834 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:15:14.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:14.870 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:14.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:15:14.944 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-14T15:15:14.971 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:15:14.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:14.996 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:14.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:15:15.097 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:15:15.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:15.123 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:15.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:15:15.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:14 smithi090 ceph-mon[41670]: pgmap v314: 409 pgs: 64 unknown, 32 creating+peering, 313 active+clean; 585 KiB data, 545 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:15:15.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:14 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:15.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:14 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2466564286' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi090-93342-35"}]': finished 2024-04-14T15:15:15.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:14 smithi090 ceph-mon[41670]: osdmap e247: 8 total, 8 up, 8 in 2024-04-14T15:15:15.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:14 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:15.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:14 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:15.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:14 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/926874511' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi090-93342-36"}]: dispatch 2024-04-14T15:15:15.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:14 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/926874511' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi090-93342-36"}]: dispatch 2024-04-14T15:15:15.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:14 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/926874511' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi090-93342-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:15.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:14 smithi090 ceph-mon[30399]: pgmap v314: 409 pgs: 64 unknown, 32 creating+peering, 313 active+clean; 585 KiB data, 545 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:15:15.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:14 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:15.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:14 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2466564286' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi090-93342-35"}]': finished 2024-04-14T15:15:15.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:14 smithi090 ceph-mon[30399]: osdmap e247: 8 total, 8 up, 8 in 2024-04-14T15:15:15.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:14 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:15.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:14 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:15.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:14 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/926874511' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi090-93342-36"}]: dispatch 2024-04-14T15:15:15.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:14 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/926874511' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi090-93342-36"}]: dispatch 2024-04-14T15:15:15.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:14 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/926874511' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi090-93342-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:15.225 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:15:15.238 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-14T15:15:15.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:15.252 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:15.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:15:15.353 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:15:15.374 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:15:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:15:15.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:14 smithi092 ceph-mon[33966]: pgmap v314: 409 pgs: 64 unknown, 32 creating+peering, 313 active+clean; 585 KiB data, 545 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:15:15.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:14 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:15.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:14 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2466564286' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi090-93342-35"}]': finished 2024-04-14T15:15:15.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:14 smithi092 ceph-mon[33966]: osdmap e247: 8 total, 8 up, 8 in 2024-04-14T15:15:15.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:14 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:15.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:14 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:15.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:14 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/926874511' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi090-93342-36"}]: dispatch 2024-04-14T15:15:15.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:14 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/926874511' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi090-93342-36"}]: dispatch 2024-04-14T15:15:15.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:14 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/926874511' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi090-93342-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:15.377 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:15.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:15.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:15:15.478 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:15:15.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:15.506 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:15.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:15:15.607 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:15:15.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:15.632 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:15.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:15:15.733 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:15:15.760 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:15.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:15.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:15:15.863 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:15:15.897 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:15.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:15.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:15:15.989 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:15:15.989 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:15:15.999 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:15:16.027 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:16.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:16.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:15:16.029 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:15:16.119 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:15 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:15:16.120 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:15 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:16.120 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:15 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:15:16.120 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:15 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/926874511' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi090-93342-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:15:16.120 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:15 smithi090 ceph-mon[41670]: osdmap e248: 8 total, 8 up, 8 in 2024-04-14T15:15:16.120 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:15 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/926874511' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi090-93342-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi090-93342-36"}]: dispatch 2024-04-14T15:15:16.120 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:15 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:15:16.120 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:15 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:16.120 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:15 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:15:16.120 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:15 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/926874511' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi090-93342-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:15:16.120 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:15 smithi090 ceph-mon[30399]: osdmap e248: 8 total, 8 up, 8 in 2024-04-14T15:15:16.121 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:15 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/926874511' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi090-93342-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi090-93342-36"}]: dispatch 2024-04-14T15:15:16.130 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:15:16.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:16.154 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:16.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:15:16.255 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:15:16.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:16.281 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:16.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:15:16.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:15 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:15:16.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:15 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:16.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:15 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:15:16.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:15 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/926874511' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi090-93342-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:15:16.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:15 smithi092 ceph-mon[33966]: osdmap e248: 8 total, 8 up, 8 in 2024-04-14T15:15:16.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:15 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/926874511' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi090-93342-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi090-93342-36"}]: dispatch 2024-04-14T15:15:16.382 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:15:16.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:16.410 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:16.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:15:16.511 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:15:16.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:16.538 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:16.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:15:16.640 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:15:16.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:16.665 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:16.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:15:16.766 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:15:16.793 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:16.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:16.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:15:16.895 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:15:16.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:16.923 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:16.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:15:17.001 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:16 smithi090 ceph-mon[41670]: pgmap v317: 377 pgs: 32 unknown, 345 active+clean; 586 KiB data, 546 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-14T15:15:17.002 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:16 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:17.002 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:16 smithi090 ceph-mon[41670]: osdmap e249: 8 total, 8 up, 8 in 2024-04-14T15:15:17.002 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:16 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:15:17.002 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:16 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:15:17.002 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:16 smithi090 ceph-mon[30399]: pgmap v317: 377 pgs: 32 unknown, 345 active+clean; 586 KiB data, 546 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-14T15:15:17.002 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:16 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:17.002 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:16 smithi090 ceph-mon[30399]: osdmap e249: 8 total, 8 up, 8 in 2024-04-14T15:15:17.002 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:16 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:15:17.002 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:16 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:15:17.024 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:15:17.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:17.049 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:17.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:15:17.150 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:15:17.176 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:17.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:17.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:15:17.278 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:15:17.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:17.306 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:17.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:15:17.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:16 smithi092 ceph-mon[33966]: pgmap v317: 377 pgs: 32 unknown, 345 active+clean; 586 KiB data, 546 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-14T15:15:17.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:16 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:17.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:16 smithi092 ceph-mon[33966]: osdmap e249: 8 total, 8 up, 8 in 2024-04-14T15:15:17.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:16 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:15:17.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:16 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:15:17.407 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:15:17.433 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:17.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:17.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:15:17.535 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:15:17.559 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:17.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:17.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:15:17.660 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:15:17.684 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:17.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:17.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:15:17.786 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:15:17.810 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:17.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:17.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:15:17.912 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:15:17.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:17.940 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:17.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:15:18.041 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:15:18.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:18.065 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:18.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:15:18.166 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:15:18.190 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:18.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:18.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:15:18.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:17 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:18.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:17 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/926874511' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi090-93342-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi090-93342-36"}]': finished 2024-04-14T15:15:18.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:17 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-27", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:15:18.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:17 smithi090 ceph-mon[30399]: osdmap e250: 8 total, 8 up, 8 in 2024-04-14T15:15:18.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:17 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-27"}]: dispatch 2024-04-14T15:15:18.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:17 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-27"}]: dispatch 2024-04-14T15:15:18.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:17 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:18.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:17 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/926874511' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi090-93342-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi090-93342-36"}]': finished 2024-04-14T15:15:18.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:17 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-27", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:15:18.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:17 smithi090 ceph-mon[41670]: osdmap e250: 8 total, 8 up, 8 in 2024-04-14T15:15:18.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:17 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-27"}]: dispatch 2024-04-14T15:15:18.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:17 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-27"}]: dispatch 2024-04-14T15:15:18.293 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:15:18.320 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:18.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:18.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:15:18.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:17 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:18.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:17 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/926874511' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi090-93342-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi090-93342-36"}]': finished 2024-04-14T15:15:18.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:17 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-27", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:15:18.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:17 smithi092 ceph-mon[33966]: osdmap e250: 8 total, 8 up, 8 in 2024-04-14T15:15:18.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:17 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-27"}]: dispatch 2024-04-14T15:15:18.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:17 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-27"}]: dispatch 2024-04-14T15:15:18.422 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:15:18.448 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:18.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:18.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:15:18.550 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:15:18.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:18.575 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:18.576 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:15:18.676 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:15:18.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:18.700 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:18.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:15:18.801 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:15:18.829 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:18.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:18.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:15:18.931 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:15:18.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:18.956 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:18.956 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:15:19.057 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:15:19.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:19.082 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:19.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:15:19.183 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:15:19.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:18 smithi090 ceph-mon[30399]: pgmap v320: 313 pgs: 32 unknown, 281 active+clean; 586 KiB data, 546 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 5 op/s; 63 B/s, 0 objects/s recovering 2024-04-14T15:15:19.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:18 smithi090 ceph-mon[30399]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:15:19.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:18 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:19.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:18 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-27"}]': finished 2024-04-14T15:15:19.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:18 smithi090 ceph-mon[30399]: osdmap e251: 8 total, 8 up, 8 in 2024-04-14T15:15:19.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:18 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-27", "mode": "writeback"}]: dispatch 2024-04-14T15:15:19.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:18 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-27", "mode": "writeback"}]: dispatch 2024-04-14T15:15:19.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:18 smithi090 ceph-mon[41670]: pgmap v320: 313 pgs: 32 unknown, 281 active+clean; 586 KiB data, 546 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 5 op/s; 63 B/s, 0 objects/s recovering 2024-04-14T15:15:19.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:18 smithi090 ceph-mon[41670]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:15:19.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:18 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:19.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:18 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-27"}]': finished 2024-04-14T15:15:19.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:18 smithi090 ceph-mon[41670]: osdmap e251: 8 total, 8 up, 8 in 2024-04-14T15:15:19.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:18 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-27", "mode": "writeback"}]: dispatch 2024-04-14T15:15:19.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:18 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-27", "mode": "writeback"}]: dispatch 2024-04-14T15:15:19.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:19.208 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:19.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:15:19.309 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:15:19.337 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:19.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:19.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:15:19.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:18 smithi092 ceph-mon[33966]: pgmap v320: 313 pgs: 32 unknown, 281 active+clean; 586 KiB data, 546 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 5 op/s; 63 B/s, 0 objects/s recovering 2024-04-14T15:15:19.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:18 smithi092 ceph-mon[33966]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:15:19.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:18 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:19.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:18 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-27"}]': finished 2024-04-14T15:15:19.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:18 smithi092 ceph-mon[33966]: osdmap e251: 8 total, 8 up, 8 in 2024-04-14T15:15:19.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:18 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-27", "mode": "writeback"}]: dispatch 2024-04-14T15:15:19.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:18 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-27", "mode": "writeback"}]: dispatch 2024-04-14T15:15:19.439 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:15:19.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:19.464 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:19.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:15:19.565 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:15:19.593 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:19.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:19.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:15:19.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.591+0000 7f76e46b5700 1 -- 172.21.15.90:0/2092875237 >> v1:172.21.15.90:6789/0 conn(0x7f76dc10b730 legacy=0x7f76dc10db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:19.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.591+0000 7f76e46b5700 1 -- 172.21.15.90:0/2092875237 shutdown_connections 2024-04-14T15:15:19.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.591+0000 7f76e46b5700 1 -- 172.21.15.90:0/2092875237 >> 172.21.15.90:0/2092875237 conn(0x7f76dc0fe3d0 msgr2=0x7f76dc1007f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:15:19.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.591+0000 7f76e46b5700 1 -- 172.21.15.90:0/2092875237 shutdown_connections 2024-04-14T15:15:19.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.592+0000 7f76e46b5700 1 -- 172.21.15.90:0/2092875237 wait complete. 2024-04-14T15:15:19.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.592+0000 7f76e46b5700 1 librados: starting msgr at 2024-04-14T15:15:19.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.592+0000 7f76e46b5700 1 librados: starting objecter 2024-04-14T15:15:19.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.592+0000 7f76e46b5700 1 Processor -- start 2024-04-14T15:15:19.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.592+0000 7f76e46b5700 1 -- start start 2024-04-14T15:15:19.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.592+0000 7f76e46b5700 1 librados: setting wanted keys 2024-04-14T15:15:19.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.592+0000 7f76e46b5700 1 librados: calling monclient init 2024-04-14T15:15:19.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.593+0000 7f76e46b5700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f76dc1a7910 con 0x7f76dc10b730 2024-04-14T15:15:19.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.593+0000 7f76e46b5700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f76dc1a8b30 con 0x7f76dc10f300 2024-04-14T15:15:19.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.593+0000 7f76e46b5700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f76dc1a9d70 con 0x7f76dc108880 2024-04-14T15:15:19.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.593+0000 7f76e1c50700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f76dc10b730 0x7f76dc1a27b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:59310/0 (socket says 172.21.15.90:59310) 2024-04-14T15:15:19.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.593+0000 7f76e1c50700 1 -- 172.21.15.90:0/3126162787 learned_addr learned my addr 172.21.15.90:0/3126162787 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:15:19.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.593+0000 7f76e2451700 1 --1- 172.21.15.90:0/3126162787 >> v1:172.21.15.90:6790/0 conn(0x7f76dc108880 0x7f76dc10eab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:52442/0 (socket says 172.21.15.90:52442) 2024-04-14T15:15:19.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.593+0000 7f76db7fe700 10 client.?.objecter ms_handle_connect 0x7f76dc108880 2024-04-14T15:15:19.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.593+0000 7f76db7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:15:19.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.593+0000 7f76db7fe700 10 client.?.objecter ms_handle_connect 0x7f76dc10b730 2024-04-14T15:15:19.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.593+0000 7f76db7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:15:19.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.593+0000 7f76db7fe700 10 client.?.objecter ms_handle_connect 0x7f76dc10f300 2024-04-14T15:15:19.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.593+0000 7f76db7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:15:19.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.593+0000 7f76db7fe700 1 -- 172.21.15.90:0/3126162787 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 148881168 0 0) 0x7f76dc1a9d70 con 0x7f76dc108880 2024-04-14T15:15:19.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.593+0000 7f76db7fe700 1 -- 172.21.15.90:0/3126162787 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f76c40035b0 con 0x7f76dc108880 2024-04-14T15:15:19.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.593+0000 7f76db7fe700 1 -- 172.21.15.90:0/3126162787 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1081423101 0 0) 0x7f76dc1a7910 con 0x7f76dc10b730 2024-04-14T15:15:19.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.593+0000 7f76db7fe700 1 -- 172.21.15.90:0/3126162787 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f76dc1a9d70 con 0x7f76dc10b730 2024-04-14T15:15:19.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.593+0000 7f76db7fe700 1 -- 172.21.15.90:0/3126162787 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1063604728 0 0) 0x7f76dc1a8b30 con 0x7f76dc10f300 2024-04-14T15:15:19.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.593+0000 7f76db7fe700 1 -- 172.21.15.90:0/3126162787 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f76dc1a7910 con 0x7f76dc10f300 2024-04-14T15:15:19.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.593+0000 7f76db7fe700 1 -- 172.21.15.90:0/3126162787 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 581844611 0 0) 0x7f76c40035b0 con 0x7f76dc108880 2024-04-14T15:15:19.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.593+0000 7f76db7fe700 1 -- 172.21.15.90:0/3126162787 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f76dc1a8b30 con 0x7f76dc108880 2024-04-14T15:15:19.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.593+0000 7f76db7fe700 1 -- 172.21.15.90:0/3126162787 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 733471980 0 0) 0x7f76dc1a9d70 con 0x7f76dc10b730 2024-04-14T15:15:19.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.593+0000 7f76db7fe700 1 -- 172.21.15.90:0/3126162787 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f76c40035b0 con 0x7f76dc10b730 2024-04-14T15:15:19.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.593+0000 7f76db7fe700 1 -- 172.21.15.90:0/3126162787 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f76cc001d90 con 0x7f76dc108880 2024-04-14T15:15:19.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.593+0000 7f76db7fe700 1 -- 172.21.15.90:0/3126162787 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f76d0004410 con 0x7f76dc10b730 2024-04-14T15:15:19.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.593+0000 7f76db7fe700 1 -- 172.21.15.90:0/3126162787 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3007668957 0 0) 0x7f76dc1a7910 con 0x7f76dc10f300 2024-04-14T15:15:19.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.593+0000 7f76db7fe700 1 -- 172.21.15.90:0/3126162787 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f76dc1a9d70 con 0x7f76dc10f300 2024-04-14T15:15:19.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.593+0000 7f76db7fe700 1 -- 172.21.15.90:0/3126162787 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f76d4001830 con 0x7f76dc10f300 2024-04-14T15:15:19.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.593+0000 7f76db7fe700 1 -- 172.21.15.90:0/3126162787 <== mon.2 v1:172.21.15.90:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3183720195 0 0) 0x7f76dc1a8b30 con 0x7f76dc108880 2024-04-14T15:15:19.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.593+0000 7f76db7fe700 1 -- 172.21.15.90:0/3126162787 >> v1:172.21.15.92:6789/0 conn(0x7f76dc10f300 legacy=0x7f76dc1a5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:19.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.593+0000 7f76db7fe700 1 -- 172.21.15.90:0/3126162787 >> v1:172.21.15.90:6789/0 conn(0x7f76dc10b730 legacy=0x7f76dc1a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:19.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.593+0000 7f76db7fe700 1 -- 172.21.15.90:0/3126162787 --> v1:172.21.15.90:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76dc1aafb0 con 0x7f76dc108880 2024-04-14T15:15:19.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.593+0000 7f76e46b5700 1 -- 172.21.15.90:0/3126162787 --> v1:172.21.15.90:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f76dc1a9fa0 con 0x7f76dc108880 2024-04-14T15:15:19.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.593+0000 7f76e46b5700 10 client.46739.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:15:19.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.593+0000 7f76e46b5700 1 -- 172.21.15.90:0/3126162787 --> v1:172.21.15.90:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f76dc1aa460 con 0x7f76dc108880 2024-04-14T15:15:19.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.593+0000 7f76e46b5700 1 librados: init done 2024-04-14T15:15:19.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.594+0000 7f76db7fe700 1 -- 172.21.15.90:0/3126162787 <== mon.2 v1:172.21.15.90:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f76cc001ff0 con 0x7f76dc108880 2024-04-14T15:15:19.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.594+0000 7f76db7fe700 1 -- 172.21.15.90:0/3126162787 <== mon.2 v1:172.21.15.90:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f76cc006050 con 0x7f76dc108880 2024-04-14T15:15:19.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.594+0000 7f76e46b5700 1 -- 172.21.15.90:0/3126162787 --> v1:172.21.15.90:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f76c8005130 con 0x7f76dc108880 2024-04-14T15:15:19.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.595+0000 7f76db7fe700 1 -- 172.21.15.90:0/3126162787 <== mon.2 v1:172.21.15.90:6790/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f76cc01ca60 con 0x7f76dc108880 2024-04-14T15:15:19.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.595+0000 7f76db7fe700 1 -- 172.21.15.90:0/3126162787 <== mon.2 v1:172.21.15.90:6790/0 8 ==== osd_map(251..251 src has 51..251) ==== 9231+0+0 (unknown 5503253 0 0) 0x7f76cc08b080 con 0x7f76dc108880 2024-04-14T15:15:19.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.595+0000 7f76db7fe700 10 client.46739.objecter ms_dispatch 0x7f76dc001550 osd_map(251..251 src has 51..251) 2024-04-14T15:15:19.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.595+0000 7f76db7fe700 3 client.46739.objecter handle_osd_map got epochs [251,251] > 0 2024-04-14T15:15:19.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.595+0000 7f76db7fe700 3 client.46739.objecter handle_osd_map decoding full epoch 251 2024-04-14T15:15:19.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.596+0000 7f76db7fe700 20 client.46739.objecter dump_active .. 0 homeless 2024-04-14T15:15:19.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.596+0000 7f76db7fe700 1 -- 172.21.15.90:0/3126162787 <== mon.2 v1:172.21.15.90:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v251) ==== 54+0+16 (unknown 320094523 0 2420000192) 0x7f76cc08d4e0 con 0x7f76dc108880 2024-04-14T15:15:19.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.596+0000 7f76db7fe700 10 client.46739.objecter ms_handle_connect 0x7f76c40713d0 2024-04-14T15:15:19.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.596+0000 7f76e46b5700 10 client.46739.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:15:19.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.596+0000 7f76e46b5700 15 client.46739.objecter _session_command_op_assign -1 1 2024-04-14T15:15:19.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.596+0000 7f76e46b5700 20 client.46739.objecter _get_session s=0x7f76c8005410 osd=0 3 2024-04-14T15:15:19.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.596+0000 7f76e46b5700 20 client.46739.objecter put_session s=0x7f76c8005410 osd=0 3 2024-04-14T15:15:19.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.596+0000 7f76e46b5700 20 client.46739.objecter _get_session s=0x7f76c8005410 osd=0 3 2024-04-14T15:15:19.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.596+0000 7f76e46b5700 15 client.46739.objecter _session_command_op_remove -1 1 2024-04-14T15:15:19.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.596+0000 7f76e46b5700 20 client.46739.objecter get_session s=0x7f76c8005410 osd=0 3 2024-04-14T15:15:19.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.596+0000 7f76e46b5700 15 client.46739.objecter _session_command_op_assign 0 1 2024-04-14T15:15:19.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.596+0000 7f76e46b5700 20 client.46739.objecter put_session s=0x7f76c8005410 osd=0 4 2024-04-14T15:15:19.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.596+0000 7f76e46b5700 10 client.46739.objecter _send_command 1 2024-04-14T15:15:19.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.596+0000 7f76e46b5700 1 -- 172.21.15.90:0/3126162787 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f76c8004b80 con 0x7f76c8004760 2024-04-14T15:15:19.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.596+0000 7f76db7fe700 10 client.46739.objecter ms_handle_connect 0x7f76c8004760 2024-04-14T15:15:19.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.597+0000 7f76db7fe700 1 -- 172.21.15.90:0/3126162787 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f76c8004b80 con 0x7f76c8004760 2024-04-14T15:15:19.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.597+0000 7f76db7fe700 10 client.46739.objecter ms_dispatch 0x7f76dc001550 command_reply(tid 1: 0 ) 2024-04-14T15:15:19.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.597+0000 7f76db7fe700 10 client.46739.objecter _finish_command 1 = system:0 2024-04-14T15:15:19.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.597+0000 7f76db7fe700 20 client.46739.objecter put_session s=0x7f76c8005410 osd=0 4 2024-04-14T15:15:19.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.597+0000 7f76db7fe700 15 client.46739.objecter _session_command_op_remove 0 1 2024-04-14T15:15:19.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.613+0000 7f76e46b5700 10 client.46739.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:15:19.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.613+0000 7f76e46b5700 15 client.46739.objecter _session_command_op_assign -1 2 2024-04-14T15:15:19.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.613+0000 7f76e46b5700 20 client.46739.objecter _get_session s=0x7f76c8005410 osd=0 3 2024-04-14T15:15:19.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.613+0000 7f76e46b5700 20 client.46739.objecter put_session s=0x7f76c8005410 osd=0 3 2024-04-14T15:15:19.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.613+0000 7f76e46b5700 20 client.46739.objecter _get_session s=0x7f76c8005410 osd=0 3 2024-04-14T15:15:19.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.613+0000 7f76e46b5700 15 client.46739.objecter _session_command_op_remove -1 2 2024-04-14T15:15:19.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.613+0000 7f76e46b5700 20 client.46739.objecter get_session s=0x7f76c8005410 osd=0 3 2024-04-14T15:15:19.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.613+0000 7f76e46b5700 15 client.46739.objecter _session_command_op_assign 0 2 2024-04-14T15:15:19.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.613+0000 7f76e46b5700 20 client.46739.objecter put_session s=0x7f76c8005410 osd=0 4 2024-04-14T15:15:19.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.613+0000 7f76e46b5700 10 client.46739.objecter _send_command 2 2024-04-14T15:15:19.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.613+0000 7f76e46b5700 1 -- 172.21.15.90:0/3126162787 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f76c8007b30 con 0x7f76c8004760 2024-04-14T15:15:19.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.616+0000 7f76db7fe700 1 -- 172.21.15.90:0/3126162787 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f76c8007b30 con 0x7f76c8004760 2024-04-14T15:15:19.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.616+0000 7f76db7fe700 10 client.46739.objecter ms_dispatch 0x7f76dc001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:15:19.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.616+0000 7f76db7fe700 10 client.46739.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:15:19.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.616+0000 7f76db7fe700 20 client.46739.objecter put_session s=0x7f76c8005410 osd=0 4 2024-04-14T15:15:19.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.616+0000 7f76db7fe700 15 client.46739.objecter _session_command_op_remove 0 2 2024-04-14T15:15:19.617 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:15:19.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.616+0000 7f76e46b5700 10 client.46739.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:15:19.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.616+0000 7f76e46b5700 15 client.46739.objecter _session_command_op_assign -1 3 2024-04-14T15:15:19.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.616+0000 7f76e46b5700 20 client.46739.objecter _get_session s=0x7f76c8004160 osd=1 3 2024-04-14T15:15:19.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.616+0000 7f76e46b5700 20 client.46739.objecter put_session s=0x7f76c8004160 osd=1 3 2024-04-14T15:15:19.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.616+0000 7f76e46b5700 20 client.46739.objecter _get_session s=0x7f76c8004160 osd=1 3 2024-04-14T15:15:19.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.616+0000 7f76e46b5700 15 client.46739.objecter _session_command_op_remove -1 3 2024-04-14T15:15:19.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.616+0000 7f76e46b5700 20 client.46739.objecter get_session s=0x7f76c8004160 osd=1 3 2024-04-14T15:15:19.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.616+0000 7f76e46b5700 15 client.46739.objecter _session_command_op_assign 1 3 2024-04-14T15:15:19.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.616+0000 7f76e46b5700 20 client.46739.objecter put_session s=0x7f76c8004160 osd=1 4 2024-04-14T15:15:19.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.616+0000 7f76e46b5700 10 client.46739.objecter _send_command 3 2024-04-14T15:15:19.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.616+0000 7f76e46b5700 1 -- 172.21.15.90:0/3126162787 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f76c8002190 con 0x7f76c80042d0 2024-04-14T15:15:19.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.616+0000 7f76db7fe700 10 client.46739.objecter ms_handle_connect 0x7f76c80042d0 2024-04-14T15:15:19.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.617+0000 7f76db7fe700 1 -- 172.21.15.90:0/3126162787 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f76c8002190 con 0x7f76c80042d0 2024-04-14T15:15:19.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.617+0000 7f76db7fe700 10 client.46739.objecter ms_dispatch 0x7f76dc001550 command_reply(tid 3: 0 ) 2024-04-14T15:15:19.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.617+0000 7f76db7fe700 10 client.46739.objecter _finish_command 3 = system:0 2024-04-14T15:15:19.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.617+0000 7f76db7fe700 20 client.46739.objecter put_session s=0x7f76c8004160 osd=1 4 2024-04-14T15:15:19.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.617+0000 7f76db7fe700 15 client.46739.objecter _session_command_op_remove 1 3 2024-04-14T15:15:19.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.634+0000 7f76e46b5700 10 client.46739.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:15:19.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.634+0000 7f76e46b5700 15 client.46739.objecter _session_command_op_assign -1 4 2024-04-14T15:15:19.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.634+0000 7f76e46b5700 20 client.46739.objecter _get_session s=0x7f76c8004160 osd=1 3 2024-04-14T15:15:19.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.634+0000 7f76e46b5700 20 client.46739.objecter put_session s=0x7f76c8004160 osd=1 3 2024-04-14T15:15:19.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.634+0000 7f76e46b5700 20 client.46739.objecter _get_session s=0x7f76c8004160 osd=1 3 2024-04-14T15:15:19.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.634+0000 7f76e46b5700 15 client.46739.objecter _session_command_op_remove -1 4 2024-04-14T15:15:19.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.634+0000 7f76e46b5700 20 client.46739.objecter get_session s=0x7f76c8004160 osd=1 3 2024-04-14T15:15:19.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.634+0000 7f76e46b5700 15 client.46739.objecter _session_command_op_assign 1 4 2024-04-14T15:15:19.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.634+0000 7f76e46b5700 20 client.46739.objecter put_session s=0x7f76c8004160 osd=1 4 2024-04-14T15:15:19.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.634+0000 7f76e46b5700 10 client.46739.objecter _send_command 4 2024-04-14T15:15:19.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.634+0000 7f76e46b5700 1 -- 172.21.15.90:0/3126162787 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f76c8001630 con 0x7f76c80042d0 2024-04-14T15:15:19.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.635+0000 7f76db7fe700 1 -- 172.21.15.90:0/3126162787 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f76c8001630 con 0x7f76c80042d0 2024-04-14T15:15:19.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.635+0000 7f76db7fe700 10 client.46739.objecter ms_dispatch 0x7f76dc001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:15:19.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.635+0000 7f76db7fe700 10 client.46739.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:15:19.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.635+0000 7f76db7fe700 20 client.46739.objecter put_session s=0x7f76c8004160 osd=1 4 2024-04-14T15:15:19.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.635+0000 7f76db7fe700 15 client.46739.objecter _session_command_op_remove 1 4 2024-04-14T15:15:19.636 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:15:19.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.635+0000 7f76e46b5700 10 client.46739.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:15:19.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.635+0000 7f76e46b5700 15 client.46739.objecter _session_command_op_assign -1 5 2024-04-14T15:15:19.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.635+0000 7f76e46b5700 20 client.46739.objecter _get_session s=0x7f76c8001a30 osd=2 3 2024-04-14T15:15:19.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.635+0000 7f76e46b5700 20 client.46739.objecter put_session s=0x7f76c8001a30 osd=2 3 2024-04-14T15:15:19.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.635+0000 7f76e46b5700 20 client.46739.objecter _get_session s=0x7f76c8001a30 osd=2 3 2024-04-14T15:15:19.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.635+0000 7f76e46b5700 15 client.46739.objecter _session_command_op_remove -1 5 2024-04-14T15:15:19.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.635+0000 7f76e46b5700 20 client.46739.objecter get_session s=0x7f76c8001a30 osd=2 3 2024-04-14T15:15:19.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.635+0000 7f76e46b5700 15 client.46739.objecter _session_command_op_assign 2 5 2024-04-14T15:15:19.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.635+0000 7f76e46b5700 20 client.46739.objecter put_session s=0x7f76c8001a30 osd=2 4 2024-04-14T15:15:19.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.635+0000 7f76e46b5700 10 client.46739.objecter _send_command 5 2024-04-14T15:15:19.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.635+0000 7f76e46b5700 1 -- 172.21.15.90:0/3126162787 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f76c8010fe0 con 0x7f76c80089c0 2024-04-14T15:15:19.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.636+0000 7f76db7fe700 10 client.46739.objecter ms_handle_connect 0x7f76c80089c0 2024-04-14T15:15:19.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.636+0000 7f76db7fe700 1 -- 172.21.15.90:0/3126162787 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f76c8010fe0 con 0x7f76c80089c0 2024-04-14T15:15:19.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.636+0000 7f76db7fe700 10 client.46739.objecter ms_dispatch 0x7f76dc001550 command_reply(tid 5: 0 ) 2024-04-14T15:15:19.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.636+0000 7f76db7fe700 10 client.46739.objecter _finish_command 5 = system:0 2024-04-14T15:15:19.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.636+0000 7f76db7fe700 20 client.46739.objecter put_session s=0x7f76c8001a30 osd=2 4 2024-04-14T15:15:19.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.636+0000 7f76db7fe700 15 client.46739.objecter _session_command_op_remove 2 5 2024-04-14T15:15:19.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.652+0000 7f76e46b5700 10 client.46739.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:15:19.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.652+0000 7f76e46b5700 15 client.46739.objecter _session_command_op_assign -1 6 2024-04-14T15:15:19.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.652+0000 7f76e46b5700 20 client.46739.objecter _get_session s=0x7f76c8001a30 osd=2 3 2024-04-14T15:15:19.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.652+0000 7f76e46b5700 20 client.46739.objecter put_session s=0x7f76c8001a30 osd=2 3 2024-04-14T15:15:19.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.652+0000 7f76e46b5700 20 client.46739.objecter _get_session s=0x7f76c8001a30 osd=2 3 2024-04-14T15:15:19.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.652+0000 7f76e46b5700 15 client.46739.objecter _session_command_op_remove -1 6 2024-04-14T15:15:19.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.652+0000 7f76e46b5700 20 client.46739.objecter get_session s=0x7f76c8001a30 osd=2 3 2024-04-14T15:15:19.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.652+0000 7f76e46b5700 15 client.46739.objecter _session_command_op_assign 2 6 2024-04-14T15:15:19.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.652+0000 7f76e46b5700 20 client.46739.objecter put_session s=0x7f76c8001a30 osd=2 4 2024-04-14T15:15:19.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.652+0000 7f76e46b5700 10 client.46739.objecter _send_command 6 2024-04-14T15:15:19.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.652+0000 7f76e46b5700 1 -- 172.21.15.90:0/3126162787 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f76c8008050 con 0x7f76c80089c0 2024-04-14T15:15:19.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.654+0000 7f76db7fe700 1 -- 172.21.15.90:0/3126162787 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f76c8008050 con 0x7f76c80089c0 2024-04-14T15:15:19.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.654+0000 7f76db7fe700 10 client.46739.objecter ms_dispatch 0x7f76dc001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:15:19.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.654+0000 7f76db7fe700 10 client.46739.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:15:19.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.654+0000 7f76db7fe700 20 client.46739.objecter put_session s=0x7f76c8001a30 osd=2 4 2024-04-14T15:15:19.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.654+0000 7f76db7fe700 15 client.46739.objecter _session_command_op_remove 2 6 2024-04-14T15:15:19.655 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:15:19.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.654+0000 7f76e46b5700 10 client.46739.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:15:19.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.654+0000 7f76e46b5700 15 client.46739.objecter _session_command_op_assign -1 7 2024-04-14T15:15:19.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.654+0000 7f76e46b5700 20 client.46739.objecter _get_session s=0x7f76c8008320 osd=3 3 2024-04-14T15:15:19.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.654+0000 7f76e46b5700 20 client.46739.objecter put_session s=0x7f76c8008320 osd=3 3 2024-04-14T15:15:19.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.654+0000 7f76e46b5700 20 client.46739.objecter _get_session s=0x7f76c8008320 osd=3 3 2024-04-14T15:15:19.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.654+0000 7f76e46b5700 15 client.46739.objecter _session_command_op_remove -1 7 2024-04-14T15:15:19.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.654+0000 7f76e46b5700 20 client.46739.objecter get_session s=0x7f76c8008320 osd=3 3 2024-04-14T15:15:19.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.654+0000 7f76e46b5700 15 client.46739.objecter _session_command_op_assign 3 7 2024-04-14T15:15:19.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.654+0000 7f76e46b5700 20 client.46739.objecter put_session s=0x7f76c8008320 osd=3 4 2024-04-14T15:15:19.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.654+0000 7f76e46b5700 10 client.46739.objecter _send_command 7 2024-04-14T15:15:19.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.654+0000 7f76e46b5700 1 -- 172.21.15.90:0/3126162787 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f76c8011d10 con 0x7f76c80111c0 2024-04-14T15:15:19.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.654+0000 7f76db7fe700 10 client.46739.objecter ms_handle_connect 0x7f76c80111c0 2024-04-14T15:15:19.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.655+0000 7f76db7fe700 1 -- 172.21.15.90:0/3126162787 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f76c8011d10 con 0x7f76c80111c0 2024-04-14T15:15:19.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.655+0000 7f76db7fe700 10 client.46739.objecter ms_dispatch 0x7f76dc001550 command_reply(tid 7: 0 ) 2024-04-14T15:15:19.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.655+0000 7f76db7fe700 10 client.46739.objecter _finish_command 7 = system:0 2024-04-14T15:15:19.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.655+0000 7f76db7fe700 20 client.46739.objecter put_session s=0x7f76c8008320 osd=3 4 2024-04-14T15:15:19.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.655+0000 7f76db7fe700 15 client.46739.objecter _session_command_op_remove 3 7 2024-04-14T15:15:19.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.671+0000 7f76e46b5700 10 client.46739.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:15:19.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.671+0000 7f76e46b5700 15 client.46739.objecter _session_command_op_assign -1 8 2024-04-14T15:15:19.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.671+0000 7f76e46b5700 20 client.46739.objecter _get_session s=0x7f76c8008320 osd=3 3 2024-04-14T15:15:19.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.671+0000 7f76e46b5700 20 client.46739.objecter put_session s=0x7f76c8008320 osd=3 3 2024-04-14T15:15:19.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.671+0000 7f76e46b5700 20 client.46739.objecter _get_session s=0x7f76c8008320 osd=3 3 2024-04-14T15:15:19.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.671+0000 7f76e46b5700 15 client.46739.objecter _session_command_op_remove -1 8 2024-04-14T15:15:19.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.671+0000 7f76e46b5700 20 client.46739.objecter get_session s=0x7f76c8008320 osd=3 3 2024-04-14T15:15:19.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.671+0000 7f76e46b5700 15 client.46739.objecter _session_command_op_assign 3 8 2024-04-14T15:15:19.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.671+0000 7f76e46b5700 20 client.46739.objecter put_session s=0x7f76c8008320 osd=3 4 2024-04-14T15:15:19.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.671+0000 7f76e46b5700 10 client.46739.objecter _send_command 8 2024-04-14T15:15:19.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.671+0000 7f76e46b5700 1 -- 172.21.15.90:0/3126162787 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f76c80126c0 con 0x7f76c80111c0 2024-04-14T15:15:19.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.673+0000 7f76db7fe700 1 -- 172.21.15.90:0/3126162787 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f76c80126c0 con 0x7f76c80111c0 2024-04-14T15:15:19.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.673+0000 7f76db7fe700 10 client.46739.objecter ms_dispatch 0x7f76dc001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:15:19.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.673+0000 7f76db7fe700 10 client.46739.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:15:19.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.673+0000 7f76db7fe700 20 client.46739.objecter put_session s=0x7f76c8008320 osd=3 4 2024-04-14T15:15:19.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.673+0000 7f76db7fe700 15 client.46739.objecter _session_command_op_remove 3 8 2024-04-14T15:15:19.678 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:15:19.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.673+0000 7f76e46b5700 10 client.46739.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:15:19.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.673+0000 7f76e46b5700 15 client.46739.objecter _session_command_op_assign -1 9 2024-04-14T15:15:19.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.673+0000 7f76e46b5700 20 client.46739.objecter _get_session s=0x7f76c8013840 osd=4 3 2024-04-14T15:15:19.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.673+0000 7f76e46b5700 20 client.46739.objecter put_session s=0x7f76c8013840 osd=4 3 2024-04-14T15:15:19.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.673+0000 7f76e46b5700 20 client.46739.objecter _get_session s=0x7f76c8013840 osd=4 3 2024-04-14T15:15:19.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.673+0000 7f76e46b5700 15 client.46739.objecter _session_command_op_remove -1 9 2024-04-14T15:15:19.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.673+0000 7f76e46b5700 20 client.46739.objecter get_session s=0x7f76c8013840 osd=4 3 2024-04-14T15:15:19.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.673+0000 7f76e46b5700 15 client.46739.objecter _session_command_op_assign 4 9 2024-04-14T15:15:19.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.673+0000 7f76e46b5700 20 client.46739.objecter put_session s=0x7f76c8013840 osd=4 4 2024-04-14T15:15:19.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.673+0000 7f76e46b5700 10 client.46739.objecter _send_command 9 2024-04-14T15:15:19.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.673+0000 7f76e46b5700 1 -- 172.21.15.90:0/3126162787 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f76c801a7c0 con 0x7f76c8016ba0 2024-04-14T15:15:19.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.674+0000 7f76db7fe700 10 client.46739.objecter ms_handle_connect 0x7f76c8016ba0 2024-04-14T15:15:19.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.675+0000 7f76db7fe700 1 -- 172.21.15.90:0/3126162787 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f76c801a7c0 con 0x7f76c8016ba0 2024-04-14T15:15:19.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.675+0000 7f76db7fe700 10 client.46739.objecter ms_dispatch 0x7f76dc001550 command_reply(tid 9: 0 ) 2024-04-14T15:15:19.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.675+0000 7f76db7fe700 10 client.46739.objecter _finish_command 9 = system:0 2024-04-14T15:15:19.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.675+0000 7f76db7fe700 20 client.46739.objecter put_session s=0x7f76c8013840 osd=4 4 2024-04-14T15:15:19.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.675+0000 7f76db7fe700 15 client.46739.objecter _session_command_op_remove 4 9 2024-04-14T15:15:19.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.692+0000 7f76e46b5700 10 client.46739.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:15:19.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.692+0000 7f76e46b5700 15 client.46739.objecter _session_command_op_assign -1 10 2024-04-14T15:15:19.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.692+0000 7f76e46b5700 20 client.46739.objecter _get_session s=0x7f76c8013840 osd=4 3 2024-04-14T15:15:19.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.692+0000 7f76e46b5700 20 client.46739.objecter put_session s=0x7f76c8013840 osd=4 3 2024-04-14T15:15:19.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.692+0000 7f76e46b5700 20 client.46739.objecter _get_session s=0x7f76c8013840 osd=4 3 2024-04-14T15:15:19.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.692+0000 7f76e46b5700 15 client.46739.objecter _session_command_op_remove -1 10 2024-04-14T15:15:19.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.692+0000 7f76e46b5700 20 client.46739.objecter get_session s=0x7f76c8013840 osd=4 3 2024-04-14T15:15:19.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.692+0000 7f76e46b5700 15 client.46739.objecter _session_command_op_assign 4 10 2024-04-14T15:15:19.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.692+0000 7f76e46b5700 20 client.46739.objecter put_session s=0x7f76c8013840 osd=4 4 2024-04-14T15:15:19.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.692+0000 7f76e46b5700 10 client.46739.objecter _send_command 10 2024-04-14T15:15:19.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.692+0000 7f76e46b5700 1 -- 172.21.15.90:0/3126162787 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f76c800c8a0 con 0x7f76c8016ba0 2024-04-14T15:15:19.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.694+0000 7f76db7fe700 1 -- 172.21.15.90:0/3126162787 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f76c800c8a0 con 0x7f76c8016ba0 2024-04-14T15:15:19.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.694+0000 7f76db7fe700 10 client.46739.objecter ms_dispatch 0x7f76dc001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:15:19.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.694+0000 7f76db7fe700 10 client.46739.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:15:19.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.694+0000 7f76db7fe700 20 client.46739.objecter put_session s=0x7f76c8013840 osd=4 4 2024-04-14T15:15:19.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.694+0000 7f76db7fe700 15 client.46739.objecter _session_command_op_remove 4 10 2024-04-14T15:15:19.695 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:15:19.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.694+0000 7f76e46b5700 10 client.46739.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:15:19.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.694+0000 7f76e46b5700 15 client.46739.objecter _session_command_op_assign -1 11 2024-04-14T15:15:19.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.694+0000 7f76e46b5700 20 client.46739.objecter _get_session s=0x7f76c801bfb0 osd=5 3 2024-04-14T15:15:19.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.694+0000 7f76e46b5700 20 client.46739.objecter put_session s=0x7f76c801bfb0 osd=5 3 2024-04-14T15:15:19.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.694+0000 7f76e46b5700 20 client.46739.objecter _get_session s=0x7f76c801bfb0 osd=5 3 2024-04-14T15:15:19.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.694+0000 7f76e46b5700 15 client.46739.objecter _session_command_op_remove -1 11 2024-04-14T15:15:19.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.694+0000 7f76e46b5700 20 client.46739.objecter get_session s=0x7f76c801bfb0 osd=5 3 2024-04-14T15:15:19.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.694+0000 7f76e46b5700 15 client.46739.objecter _session_command_op_assign 5 11 2024-04-14T15:15:19.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.694+0000 7f76e46b5700 20 client.46739.objecter put_session s=0x7f76c801bfb0 osd=5 4 2024-04-14T15:15:19.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.694+0000 7f76e46b5700 10 client.46739.objecter _send_command 11 2024-04-14T15:15:19.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.694+0000 7f76e46b5700 1 -- 172.21.15.90:0/3126162787 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f76c800c8d0 con 0x7f76c801ce40 2024-04-14T15:15:19.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.695+0000 7f76db7fe700 10 client.46739.objecter ms_handle_connect 0x7f76c801ce40 2024-04-14T15:15:19.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.696+0000 7f76db7fe700 1 -- 172.21.15.90:0/3126162787 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f76d0003c60 con 0x7f76c801ce40 2024-04-14T15:15:19.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.696+0000 7f76db7fe700 10 client.46739.objecter ms_dispatch 0x7f76dc001550 command_reply(tid 11: 0 ) 2024-04-14T15:15:19.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.696+0000 7f76db7fe700 10 client.46739.objecter _finish_command 11 = system:0 2024-04-14T15:15:19.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.696+0000 7f76db7fe700 20 client.46739.objecter put_session s=0x7f76c801bfb0 osd=5 4 2024-04-14T15:15:19.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.696+0000 7f76db7fe700 15 client.46739.objecter _session_command_op_remove 5 11 2024-04-14T15:15:19.697 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:15:19.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.712+0000 7f76e46b5700 10 client.46739.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:15:19.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.712+0000 7f76e46b5700 15 client.46739.objecter _session_command_op_assign -1 12 2024-04-14T15:15:19.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.712+0000 7f76e46b5700 20 client.46739.objecter _get_session s=0x7f76c801bfb0 osd=5 3 2024-04-14T15:15:19.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.712+0000 7f76e46b5700 20 client.46739.objecter put_session s=0x7f76c801bfb0 osd=5 3 2024-04-14T15:15:19.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.712+0000 7f76e46b5700 20 client.46739.objecter _get_session s=0x7f76c801bfb0 osd=5 3 2024-04-14T15:15:19.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.712+0000 7f76e46b5700 15 client.46739.objecter _session_command_op_remove -1 12 2024-04-14T15:15:19.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.712+0000 7f76e46b5700 20 client.46739.objecter get_session s=0x7f76c801bfb0 osd=5 3 2024-04-14T15:15:19.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.712+0000 7f76e46b5700 15 client.46739.objecter _session_command_op_assign 5 12 2024-04-14T15:15:19.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.712+0000 7f76e46b5700 20 client.46739.objecter put_session s=0x7f76c801bfb0 osd=5 4 2024-04-14T15:15:19.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.712+0000 7f76e46b5700 10 client.46739.objecter _send_command 12 2024-04-14T15:15:19.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.712+0000 7f76e46b5700 1 -- 172.21.15.90:0/3126162787 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f76c801c940 con 0x7f76c801ce40 2024-04-14T15:15:19.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.714+0000 7f76db7fe700 1 -- 172.21.15.90:0/3126162787 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f76c801c940 con 0x7f76c801ce40 2024-04-14T15:15:19.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.714+0000 7f76db7fe700 10 client.46739.objecter ms_dispatch 0x7f76dc001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:15:19.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.714+0000 7f76db7fe700 10 client.46739.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:15:19.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.714+0000 7f76db7fe700 20 client.46739.objecter put_session s=0x7f76c801bfb0 osd=5 4 2024-04-14T15:15:19.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.714+0000 7f76db7fe700 15 client.46739.objecter _session_command_op_remove 5 12 2024-04-14T15:15:19.714 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:15:19.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.714+0000 7f76e46b5700 10 client.46739.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:15:19.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.714+0000 7f76e46b5700 15 client.46739.objecter _session_command_op_assign -1 13 2024-04-14T15:15:19.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.714+0000 7f76e46b5700 20 client.46739.objecter _get_session s=0x7f76c801cb60 osd=6 3 2024-04-14T15:15:19.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.714+0000 7f76e46b5700 20 client.46739.objecter put_session s=0x7f76c801cb60 osd=6 3 2024-04-14T15:15:19.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.714+0000 7f76e46b5700 20 client.46739.objecter _get_session s=0x7f76c801cb60 osd=6 3 2024-04-14T15:15:19.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.714+0000 7f76e46b5700 15 client.46739.objecter _session_command_op_remove -1 13 2024-04-14T15:15:19.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.714+0000 7f76e46b5700 20 client.46739.objecter get_session s=0x7f76c801cb60 osd=6 3 2024-04-14T15:15:19.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.714+0000 7f76e46b5700 15 client.46739.objecter _session_command_op_assign 6 13 2024-04-14T15:15:19.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.714+0000 7f76e46b5700 20 client.46739.objecter put_session s=0x7f76c801cb60 osd=6 4 2024-04-14T15:15:19.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.714+0000 7f76e46b5700 10 client.46739.objecter _send_command 13 2024-04-14T15:15:19.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.714+0000 7f76e46b5700 1 -- 172.21.15.90:0/3126162787 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f76c8023390 con 0x7f76c801f880 2024-04-14T15:15:19.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.714+0000 7f76db7fe700 10 client.46739.objecter ms_handle_connect 0x7f76c801f880 2024-04-14T15:15:19.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.715+0000 7f76db7fe700 1 -- 172.21.15.90:0/3126162787 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f76c8023390 con 0x7f76c801f880 2024-04-14T15:15:19.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.715+0000 7f76db7fe700 10 client.46739.objecter ms_dispatch 0x7f76dc001550 command_reply(tid 13: 0 ) 2024-04-14T15:15:19.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.715+0000 7f76db7fe700 10 client.46739.objecter _finish_command 13 = system:0 2024-04-14T15:15:19.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.715+0000 7f76db7fe700 20 client.46739.objecter put_session s=0x7f76c801cb60 osd=6 4 2024-04-14T15:15:19.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.715+0000 7f76db7fe700 15 client.46739.objecter _session_command_op_remove 6 13 2024-04-14T15:15:19.725 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:19.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:19.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:15:19.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.732+0000 7f76e46b5700 10 client.46739.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:15:19.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.732+0000 7f76e46b5700 15 client.46739.objecter _session_command_op_assign -1 14 2024-04-14T15:15:19.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.732+0000 7f76e46b5700 20 client.46739.objecter _get_session s=0x7f76c801cb60 osd=6 3 2024-04-14T15:15:19.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.732+0000 7f76e46b5700 20 client.46739.objecter put_session s=0x7f76c801cb60 osd=6 3 2024-04-14T15:15:19.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.732+0000 7f76e46b5700 20 client.46739.objecter _get_session s=0x7f76c801cb60 osd=6 3 2024-04-14T15:15:19.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.732+0000 7f76e46b5700 15 client.46739.objecter _session_command_op_remove -1 14 2024-04-14T15:15:19.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.732+0000 7f76e46b5700 20 client.46739.objecter get_session s=0x7f76c801cb60 osd=6 3 2024-04-14T15:15:19.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.732+0000 7f76e46b5700 15 client.46739.objecter _session_command_op_assign 6 14 2024-04-14T15:15:19.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.732+0000 7f76e46b5700 20 client.46739.objecter put_session s=0x7f76c801cb60 osd=6 4 2024-04-14T15:15:19.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.732+0000 7f76e46b5700 10 client.46739.objecter _send_command 14 2024-04-14T15:15:19.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.732+0000 7f76e46b5700 1 -- 172.21.15.90:0/3126162787 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f76c8024d40 con 0x7f76c801f880 2024-04-14T15:15:19.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.734+0000 7f76db7fe700 1 -- 172.21.15.90:0/3126162787 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f76c8024d40 con 0x7f76c801f880 2024-04-14T15:15:19.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.734+0000 7f76db7fe700 10 client.46739.objecter ms_dispatch 0x7f76dc001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:15:19.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.734+0000 7f76db7fe700 10 client.46739.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:15:19.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.734+0000 7f76db7fe700 20 client.46739.objecter put_session s=0x7f76c801cb60 osd=6 4 2024-04-14T15:15:19.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.734+0000 7f76db7fe700 15 client.46739.objecter _session_command_op_remove 6 14 2024-04-14T15:15:19.735 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:15:19.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.734+0000 7f76e46b5700 10 client.46739.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:15:19.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.734+0000 7f76e46b5700 15 client.46739.objecter _session_command_op_assign -1 15 2024-04-14T15:15:19.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.734+0000 7f76e46b5700 20 client.46739.objecter _get_session s=0x7f76c8024d40 osd=7 3 2024-04-14T15:15:19.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.734+0000 7f76e46b5700 20 client.46739.objecter put_session s=0x7f76c8024d40 osd=7 3 2024-04-14T15:15:19.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.734+0000 7f76e46b5700 20 client.46739.objecter _get_session s=0x7f76c8024d40 osd=7 3 2024-04-14T15:15:19.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.734+0000 7f76e46b5700 15 client.46739.objecter _session_command_op_remove -1 15 2024-04-14T15:15:19.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.734+0000 7f76e46b5700 20 client.46739.objecter get_session s=0x7f76c8024d40 osd=7 3 2024-04-14T15:15:19.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.734+0000 7f76e46b5700 15 client.46739.objecter _session_command_op_assign 7 15 2024-04-14T15:15:19.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.734+0000 7f76e46b5700 20 client.46739.objecter put_session s=0x7f76c8024d40 osd=7 4 2024-04-14T15:15:19.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.734+0000 7f76e46b5700 10 client.46739.objecter _send_command 15 2024-04-14T15:15:19.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.734+0000 7f76e46b5700 1 -- 172.21.15.90:0/3126162787 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f76c80258c0 con 0x7f76c8025460 2024-04-14T15:15:19.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.735+0000 7f76db7fe700 10 client.46739.objecter ms_handle_connect 0x7f76c8025460 2024-04-14T15:15:19.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.736+0000 7f76db7fe700 1 -- 172.21.15.90:0/3126162787 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f76c80258c0 con 0x7f76c8025460 2024-04-14T15:15:19.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.736+0000 7f76db7fe700 10 client.46739.objecter ms_dispatch 0x7f76dc001550 command_reply(tid 15: 0 ) 2024-04-14T15:15:19.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.736+0000 7f76db7fe700 10 client.46739.objecter _finish_command 15 = system:0 2024-04-14T15:15:19.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.736+0000 7f76db7fe700 20 client.46739.objecter put_session s=0x7f76c8024d40 osd=7 4 2024-04-14T15:15:19.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.736+0000 7f76db7fe700 15 client.46739.objecter _session_command_op_remove 7 15 2024-04-14T15:15:19.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.753+0000 7f76e46b5700 10 client.46739.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:15:19.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.753+0000 7f76e46b5700 15 client.46739.objecter _session_command_op_assign -1 16 2024-04-14T15:15:19.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.753+0000 7f76e46b5700 20 client.46739.objecter _get_session s=0x7f76c8024d40 osd=7 3 2024-04-14T15:15:19.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.753+0000 7f76e46b5700 20 client.46739.objecter put_session s=0x7f76c8024d40 osd=7 3 2024-04-14T15:15:19.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.753+0000 7f76e46b5700 20 client.46739.objecter _get_session s=0x7f76c8024d40 osd=7 3 2024-04-14T15:15:19.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.753+0000 7f76e46b5700 15 client.46739.objecter _session_command_op_remove -1 16 2024-04-14T15:15:19.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.753+0000 7f76e46b5700 20 client.46739.objecter get_session s=0x7f76c8024d40 osd=7 3 2024-04-14T15:15:19.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.753+0000 7f76e46b5700 15 client.46739.objecter _session_command_op_assign 7 16 2024-04-14T15:15:19.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.753+0000 7f76e46b5700 20 client.46739.objecter put_session s=0x7f76c8024d40 osd=7 4 2024-04-14T15:15:19.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.753+0000 7f76e46b5700 10 client.46739.objecter _send_command 16 2024-04-14T15:15:19.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.753+0000 7f76e46b5700 1 -- 172.21.15.90:0/3126162787 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f76c801c910 con 0x7f76c8025460 2024-04-14T15:15:19.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.755+0000 7f76db7fe700 1 -- 172.21.15.90:0/3126162787 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f76c801c910 con 0x7f76c8025460 2024-04-14T15:15:19.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.755+0000 7f76db7fe700 10 client.46739.objecter ms_dispatch 0x7f76dc001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:15:19.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.755+0000 7f76db7fe700 10 client.46739.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:15:19.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.755+0000 7f76db7fe700 20 client.46739.objecter put_session s=0x7f76c8024d40 osd=7 4 2024-04-14T15:15:19.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.755+0000 7f76db7fe700 15 client.46739.objecter _session_command_op_remove 7 16 2024-04-14T15:15:19.755 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:15:19.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.755+0000 7f76e46b5700 10 librados: watch_flush enter 2024-04-14T15:15:19.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.755+0000 7f76e46b5700 10 librados: watch_flush exit 2024-04-14T15:15:19.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.755+0000 7f76e46b5700 10 client.46739.objecter close_session for osd.0 2024-04-14T15:15:19.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.755+0000 7f76e46b5700 1 -- 172.21.15.90:0/3126162787 >> v1:172.21.15.90:6801/3761806885 conn(0x7f76c8004760 legacy=0x7f76c8002780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:19.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.755+0000 7f76e46b5700 20 client.46739.objecter put_session s=0x7f76c8005410 osd=0 1 2024-04-14T15:15:19.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.755+0000 7f76e46b5700 10 client.46739.objecter close_session for osd.1 2024-04-14T15:15:19.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.755+0000 7f76e46b5700 1 -- 172.21.15.90:0/3126162787 >> v1:172.21.15.90:6805/4170361699 conn(0x7f76c80042d0 legacy=0x7f76c800ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:19.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.755+0000 7f76e46b5700 20 client.46739.objecter put_session s=0x7f76c8004160 osd=1 1 2024-04-14T15:15:19.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.755+0000 7f76e46b5700 10 client.46739.objecter close_session for osd.2 2024-04-14T15:15:19.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.755+0000 7f76e46b5700 1 -- 172.21.15.90:0/3126162787 >> v1:172.21.15.90:6809/1686184251 conn(0x7f76c80089c0 legacy=0x7f76c800f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:19.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.755+0000 7f76e46b5700 20 client.46739.objecter put_session s=0x7f76c8001a30 osd=2 1 2024-04-14T15:15:19.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.755+0000 7f76e46b5700 10 client.46739.objecter close_session for osd.3 2024-04-14T15:15:19.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.755+0000 7f76e46b5700 1 -- 172.21.15.90:0/3126162787 >> v1:172.21.15.90:6813/2270166774 conn(0x7f76c80111c0 legacy=0x7f76c8011600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:19.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.755+0000 7f76e46b5700 20 client.46739.objecter put_session s=0x7f76c8008320 osd=3 1 2024-04-14T15:15:19.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.755+0000 7f76e46b5700 10 client.46739.objecter close_session for osd.4 2024-04-14T15:15:19.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.755+0000 7f76e46b5700 1 -- 172.21.15.90:0/3126162787 >> v1:172.21.15.92:6800/3887549733 conn(0x7f76c8016ba0 legacy=0x7f76c8018f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:19.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.755+0000 7f76e46b5700 20 client.46739.objecter put_session s=0x7f76c8013840 osd=4 1 2024-04-14T15:15:19.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.755+0000 7f76e46b5700 10 client.46739.objecter close_session for osd.5 2024-04-14T15:15:19.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.755+0000 7f76e46b5700 1 -- 172.21.15.90:0/3126162787 >> v1:172.21.15.92:6804/3244561440 conn(0x7f76c801ce40 legacy=0x7f76c8011ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:19.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.755+0000 7f76e46b5700 20 client.46739.objecter put_session s=0x7f76c801bfb0 osd=5 1 2024-04-14T15:15:19.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.755+0000 7f76e46b5700 10 client.46739.objecter close_session for osd.6 2024-04-14T15:15:19.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.755+0000 7f76e46b5700 1 -- 172.21.15.90:0/3126162787 >> v1:172.21.15.92:6808/636125963 conn(0x7f76c801f880 legacy=0x7f76c8021c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:19.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.755+0000 7f76e46b5700 20 client.46739.objecter put_session s=0x7f76c801cb60 osd=6 1 2024-04-14T15:15:19.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.755+0000 7f76e46b5700 10 client.46739.objecter close_session for osd.7 2024-04-14T15:15:19.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.755+0000 7f76e46b5700 1 -- 172.21.15.90:0/3126162787 >> v1:172.21.15.92:6812/1773706716 conn(0x7f76c8025460 legacy=0x7f76c8023570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:19.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.755+0000 7f76e46b5700 20 client.46739.objecter put_session s=0x7f76c8024d40 osd=7 1 2024-04-14T15:15:19.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.755+0000 7f76e46b5700 20 client.46739.objecter shutdown clearing up homeless session... 2024-04-14T15:15:19.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.755+0000 7f76e46b5700 10 client.46739.objecter successfully canceled tick 2024-04-14T15:15:19.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.755+0000 7f76e2c52700 1 -- 172.21.15.90:0/3126162787 reap_dead start 2024-04-14T15:15:19.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.755+0000 7f76e2c52700 1 -- 172.21.15.90:0/3126162787 reap_dead start 2024-04-14T15:15:19.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.755+0000 7f76e2c52700 1 -- 172.21.15.90:0/3126162787 reap_dead start 2024-04-14T15:15:19.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.755+0000 7f76e46b5700 1 -- 172.21.15.90:0/3126162787 >> v1:172.21.15.90:6800/420935427 conn(0x7f76c40713d0 legacy=0x7f76c4073890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:19.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.755+0000 7f76e2c52700 1 -- 172.21.15.90:0/3126162787 reap_dead start 2024-04-14T15:15:19.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.755+0000 7f76e2c52700 1 -- 172.21.15.90:0/3126162787 reap_dead start 2024-04-14T15:15:19.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.755+0000 7f76e46b5700 1 -- 172.21.15.90:0/3126162787 >> v1:172.21.15.90:6790/0 conn(0x7f76dc108880 legacy=0x7f76dc10eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:19.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.755+0000 7f76e46b5700 1 -- 172.21.15.90:0/3126162787 shutdown_connections 2024-04-14T15:15:19.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.755+0000 7f76e46b5700 1 -- 172.21.15.90:0/3126162787 >> 172.21.15.90:0/3126162787 conn(0x7f76dc0fe3d0 msgr2=0x7f76dc111fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:15:19.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.756+0000 7f76e46b5700 1 -- 172.21.15.90:0/3126162787 shutdown_connections 2024-04-14T15:15:19.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.756+0000 7f76e46b5700 1 -- 172.21.15.90:0/3126162787 wait complete. 2024-04-14T15:15:19.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.756+0000 7f76e46b5700 1 librados: shutdown 2024-04-14T15:15:19.759 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:15:19.759 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:15:19.760 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:15:19.760 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:15:19.760 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:15:19.760 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:15:19.760 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:15:19.760 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:15:19.826 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:15:19.852 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:19.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:19.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:15:19.954 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:15:19.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:19.983 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:19.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:15:20.084 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:15:20.111 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:20.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:20.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:15:20.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:19 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:20.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:19 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3126162787' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:15:20.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:19 smithi090 ceph-mon[30399]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:15:20.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:19 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-27", "mode": "writeback"}]': finished 2024-04-14T15:15:20.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:19 smithi090 ceph-mon[30399]: osdmap e252: 8 total, 8 up, 8 in 2024-04-14T15:15:20.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:19 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/926874511' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi090-93342-36"}]: dispatch 2024-04-14T15:15:20.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:19 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/289305254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi090-93363-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:20.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:19 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:20.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:19 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3126162787' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:15:20.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:19 smithi090 ceph-mon[41670]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:15:20.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:19 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-27", "mode": "writeback"}]': finished 2024-04-14T15:15:20.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:19 smithi090 ceph-mon[41670]: osdmap e252: 8 total, 8 up, 8 in 2024-04-14T15:15:20.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:19 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/926874511' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi090-93342-36"}]: dispatch 2024-04-14T15:15:20.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:19 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/289305254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi090-93363-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:20.212 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:15:20.239 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:20.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:20.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:15:20.341 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:15:20.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:20.368 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:20.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:15:20.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:19 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:20.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:19 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3126162787' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:15:20.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:19 smithi092 ceph-mon[33966]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:15:20.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:19 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-27", "mode": "writeback"}]': finished 2024-04-14T15:15:20.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:19 smithi092 ceph-mon[33966]: osdmap e252: 8 total, 8 up, 8 in 2024-04-14T15:15:20.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:19 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/926874511' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi090-93342-36"}]: dispatch 2024-04-14T15:15:20.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:19 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/289305254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi090-93363-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:20.469 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:15:20.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:20.497 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:20.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:15:20.598 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:15:20.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:20.625 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:20.625 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:15:20.726 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:15:20.750 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:20.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:20.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:15:20.853 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:15:20.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:20.896 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:20.897 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:15:20.998 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:15:21.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:21.025 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:21.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:15:21.127 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:15:21.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:21.152 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:21.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:15:21.226 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:20 smithi090 ceph-mon[41670]: pgmap v323: 257 pgs: 14 unknown, 243 active+clean; 586 KiB data, 555 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:15:21.227 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:20 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:21.227 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:20 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3797385793' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-04-14T15:15:21.227 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:20 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/926874511' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi090-93342-36"}]': finished 2024-04-14T15:15:21.227 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:20 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/289305254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi090-93363-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:15:21.227 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:20 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3797385793' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-04-14T15:15:21.227 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:20 smithi090 ceph-mon[41670]: osdmap e253: 8 total, 8 up, 8 in 2024-04-14T15:15:21.227 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:20 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/926874511' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi090-93342-36"}]: dispatch 2024-04-14T15:15:21.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:20 smithi090 ceph-mon[30399]: pgmap v323: 257 pgs: 14 unknown, 243 active+clean; 586 KiB data, 555 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:15:21.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:20 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:21.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:20 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3797385793' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-04-14T15:15:21.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:20 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/926874511' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi090-93342-36"}]': finished 2024-04-14T15:15:21.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:20 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/289305254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi090-93363-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:15:21.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:20 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3797385793' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-04-14T15:15:21.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:20 smithi090 ceph-mon[30399]: osdmap e253: 8 total, 8 up, 8 in 2024-04-14T15:15:21.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:20 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/926874511' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi090-93342-36"}]: dispatch 2024-04-14T15:15:21.253 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:15:21.279 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:21.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:21.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:15:21.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:20 smithi092 ceph-mon[33966]: pgmap v323: 257 pgs: 14 unknown, 243 active+clean; 586 KiB data, 555 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:15:21.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:20 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:21.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:20 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3797385793' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-04-14T15:15:21.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:20 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/926874511' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi090-93342-36"}]': finished 2024-04-14T15:15:21.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:20 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/289305254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi090-93363-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:15:21.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:20 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3797385793' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-04-14T15:15:21.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:20 smithi092 ceph-mon[33966]: osdmap e253: 8 total, 8 up, 8 in 2024-04-14T15:15:21.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:20 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/926874511' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi090-93342-36"}]: dispatch 2024-04-14T15:15:21.381 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:15:21.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:21.406 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:21.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:15:21.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.913+0000 7f89ccb66700 1 -- 172.21.15.90:0/1414102847 >> v1:172.21.15.90:6789/0 conn(0x7f89c810b740 legacy=0x7f89c810db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:21.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.913+0000 7f89ccb66700 1 -- 172.21.15.90:0/1414102847 shutdown_connections 2024-04-14T15:15:21.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.913+0000 7f89ccb66700 1 -- 172.21.15.90:0/1414102847 >> 172.21.15.90:0/1414102847 conn(0x7f89c80fe380 msgr2=0x7f89c81007a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:15:21.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.913+0000 7f89ccb66700 1 -- 172.21.15.90:0/1414102847 shutdown_connections 2024-04-14T15:15:21.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.913+0000 7f89ccb66700 1 -- 172.21.15.90:0/1414102847 wait complete. 2024-04-14T15:15:21.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.914+0000 7f89ccb66700 1 librados: starting msgr at 2024-04-14T15:15:21.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.914+0000 7f89ccb66700 1 librados: starting objecter 2024-04-14T15:15:21.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.914+0000 7f89ccb66700 1 Processor -- start 2024-04-14T15:15:21.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.914+0000 7f89ccb66700 1 -- start start 2024-04-14T15:15:21.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.914+0000 7f89ccb66700 1 librados: setting wanted keys 2024-04-14T15:15:21.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.914+0000 7f89ccb66700 1 librados: calling monclient init 2024-04-14T15:15:21.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.914+0000 7f89ccb66700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f89c81a79e0 con 0x7f89c810f310 2024-04-14T15:15:21.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.914+0000 7f89ccb66700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f89c81a8c00 con 0x7f89c8108890 2024-04-14T15:15:21.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.914+0000 7f89ccb66700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f89c81a9e40 con 0x7f89c810b740 2024-04-14T15:15:21.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.914+0000 7f89c6d9d700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f89c810f310 0x7f89c81a60a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:59328/0 (socket says 172.21.15.90:59328) 2024-04-14T15:15:21.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.914+0000 7f89c5d9b700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f89c810b740 0x7f89c81a27c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:52456/0 (socket says 172.21.15.90:52456) 2024-04-14T15:15:21.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.914+0000 7f89c6d9d700 1 -- 172.21.15.90:0/3797385793 learned_addr learned my addr 172.21.15.90:0/3797385793 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:15:21.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.914+0000 7f89b77fe700 10 client.?.objecter ms_handle_connect 0x7f89c810f310 2024-04-14T15:15:21.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.914+0000 7f89b77fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:15:21.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.914+0000 7f89b77fe700 10 client.?.objecter ms_handle_connect 0x7f89c810b740 2024-04-14T15:15:21.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.914+0000 7f89b77fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:15:21.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.914+0000 7f89b77fe700 10 client.?.objecter ms_handle_connect 0x7f89c8108890 2024-04-14T15:15:21.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.914+0000 7f89b77fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:15:21.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.914+0000 7f89b77fe700 1 -- 172.21.15.90:0/3797385793 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 635807869 0 0) 0x7f89c81a79e0 con 0x7f89c810f310 2024-04-14T15:15:21.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.915+0000 7f89b77fe700 1 -- 172.21.15.90:0/3797385793 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89b00031f0 con 0x7f89c810f310 2024-04-14T15:15:21.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.915+0000 7f89b77fe700 1 -- 172.21.15.90:0/3797385793 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2993403516 0 0) 0x7f89c81a9e40 con 0x7f89c810b740 2024-04-14T15:15:21.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.915+0000 7f89b77fe700 1 -- 172.21.15.90:0/3797385793 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89c81a79e0 con 0x7f89c810b740 2024-04-14T15:15:21.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.915+0000 7f89b77fe700 1 -- 172.21.15.90:0/3797385793 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1366481678 0 0) 0x7f89c81a8c00 con 0x7f89c8108890 2024-04-14T15:15:21.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.915+0000 7f89b77fe700 1 -- 172.21.15.90:0/3797385793 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89c81a9e40 con 0x7f89c8108890 2024-04-14T15:15:21.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.915+0000 7f89b77fe700 1 -- 172.21.15.90:0/3797385793 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3237286458 0 0) 0x7f89b00031f0 con 0x7f89c810f310 2024-04-14T15:15:21.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.915+0000 7f89b77fe700 1 -- 172.21.15.90:0/3797385793 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f89c81a8c00 con 0x7f89c810f310 2024-04-14T15:15:21.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.915+0000 7f89b77fe700 1 -- 172.21.15.90:0/3797385793 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f89b80013d0 con 0x7f89c810f310 2024-04-14T15:15:21.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.915+0000 7f89b77fe700 1 -- 172.21.15.90:0/3797385793 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2614770355 0 0) 0x7f89c81a79e0 con 0x7f89c810b740 2024-04-14T15:15:21.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.915+0000 7f89b77fe700 1 -- 172.21.15.90:0/3797385793 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f89b00031f0 con 0x7f89c810b740 2024-04-14T15:15:21.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.915+0000 7f89b77fe700 1 -- 172.21.15.90:0/3797385793 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f89bc004550 con 0x7f89c810b740 2024-04-14T15:15:21.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.915+0000 7f89b77fe700 1 -- 172.21.15.90:0/3797385793 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2738439936 0 0) 0x7f89c81a9e40 con 0x7f89c8108890 2024-04-14T15:15:21.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.915+0000 7f89b77fe700 1 -- 172.21.15.90:0/3797385793 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f89c81a79e0 con 0x7f89c8108890 2024-04-14T15:15:21.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.915+0000 7f89b77fe700 1 -- 172.21.15.90:0/3797385793 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f89c0001bb0 con 0x7f89c8108890 2024-04-14T15:15:21.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.915+0000 7f89b77fe700 1 -- 172.21.15.90:0/3797385793 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 270502639 0 0) 0x7f89c81a8c00 con 0x7f89c810f310 2024-04-14T15:15:21.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.915+0000 7f89b77fe700 1 -- 172.21.15.90:0/3797385793 >> v1:172.21.15.90:6790/0 conn(0x7f89c810b740 legacy=0x7f89c81a27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:21.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.915+0000 7f89b77fe700 1 -- 172.21.15.90:0/3797385793 >> v1:172.21.15.92:6789/0 conn(0x7f89c8108890 legacy=0x7f89c810eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:21.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.915+0000 7f89b77fe700 1 -- 172.21.15.90:0/3797385793 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89c81ab080 con 0x7f89c810f310 2024-04-14T15:15:21.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.915+0000 7f89ccb66700 1 -- 172.21.15.90:0/3797385793 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f89c81aa070 con 0x7f89c810f310 2024-04-14T15:15:21.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.915+0000 7f89ccb66700 10 client.36792.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:15:21.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.915+0000 7f89ccb66700 1 -- 172.21.15.90:0/3797385793 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f89c81aa530 con 0x7f89c810f310 2024-04-14T15:15:21.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.915+0000 7f89ccb66700 1 librados: init done 2024-04-14T15:15:21.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.915+0000 7f89b77fe700 1 -- 172.21.15.90:0/3797385793 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f89b8001630 con 0x7f89c810f310 2024-04-14T15:15:21.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.915+0000 7f89b77fe700 1 -- 172.21.15.90:0/3797385793 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f89b8005c50 con 0x7f89c810f310 2024-04-14T15:15:21.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.916+0000 7f89ccb66700 1 -- 172.21.15.90:0/3797385793 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89a8005120 con 0x7f89c810f310 2024-04-14T15:15:21.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.917+0000 7f89b77fe700 1 -- 172.21.15.90:0/3797385793 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f89b801c640 con 0x7f89c810f310 2024-04-14T15:15:21.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.917+0000 7f89b77fe700 1 -- 172.21.15.90:0/3797385793 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(252..252 src has 51..252) ==== 9192+0+0 (unknown 1599297662 0 0) 0x7f89b808adc0 con 0x7f89c810f310 2024-04-14T15:15:21.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.917+0000 7f89b77fe700 10 client.36792.objecter ms_dispatch 0x7f89c8001550 osd_map(252..252 src has 51..252) 2024-04-14T15:15:21.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.917+0000 7f89b77fe700 3 client.36792.objecter handle_osd_map got epochs [252,252] > 0 2024-04-14T15:15:21.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.917+0000 7f89b77fe700 3 client.36792.objecter handle_osd_map decoding full epoch 252 2024-04-14T15:15:21.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.917+0000 7f89b77fe700 20 client.36792.objecter dump_active .. 0 homeless 2024-04-14T15:15:21.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.917+0000 7f89b77fe700 10 client.36792.objecter ms_handle_connect 0x7f89b0071490 2024-04-14T15:15:21.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:19.922+0000 7f89b77fe700 1 -- 172.21.15.90:0/3797385793 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f89b80573f0 con 0x7f89c810f310 2024-04-14T15:15:21.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:20.139+0000 7f89ccb66700 1 -- 172.21.15.90:0/3797385793 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} v 0) -- 0x7f89a8005c20 con 0x7f89c810f310 2024-04-14T15:15:21.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:20.851+0000 7f89b77fe700 1 -- 172.21.15.90:0/3797385793 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]=0 set osd.3 primary-affinity to 1 (8655362) v253) ==== 133+0+0 (unknown 1161506327 0 0) 0x7f89b805b0a0 con 0x7f89c810f310 2024-04-14T15:15:21.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:20.855+0000 7f89ccb66700 10 librados: watch_flush enter 2024-04-14T15:15:21.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:20.855+0000 7f89ccb66700 10 librados: watch_flush exit 2024-04-14T15:15:21.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:20.855+0000 7f89ccb66700 20 client.36792.objecter shutdown clearing up homeless session... 2024-04-14T15:15:21.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:20.855+0000 7f89ccb66700 10 client.36792.objecter successfully canceled tick 2024-04-14T15:15:21.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:20.855+0000 7f89ccb66700 1 -- 172.21.15.90:0/3797385793 >> v1:172.21.15.90:6800/420935427 conn(0x7f89b0071490 legacy=0x7f89b0073950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:21.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:20.855+0000 7f89ccb66700 1 -- 172.21.15.90:0/3797385793 >> v1:172.21.15.90:6789/0 conn(0x7f89c810f310 legacy=0x7f89c81a60a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:21.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:20.855+0000 7f89ccb66700 1 -- 172.21.15.90:0/3797385793 shutdown_connections 2024-04-14T15:15:21.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:20.855+0000 7f89ccb66700 1 -- 172.21.15.90:0/3797385793 >> 172.21.15.90:0/3797385793 conn(0x7f89c80fe380 msgr2=0x7f89c81126d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:15:21.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:20.855+0000 7f89ccb66700 1 -- 172.21.15.90:0/3797385793 shutdown_connections 2024-04-14T15:15:21.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:20.855+0000 7f89ccb66700 1 -- 172.21.15.90:0/3797385793 wait complete. 2024-04-14T15:15:21.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:20.855+0000 7f89ccb66700 1 librados: shutdown 2024-04-14T15:15:21.484 INFO:teuthology.orchestra.run.smithi090.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-04-14T15:15:21.507 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:15:21.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:21.532 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:21.532 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:15:21.633 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:15:21.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:21.662 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:21.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:15:21.763 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:15:21.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:21.788 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:21.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:15:21.890 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:15:21.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:21.917 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:21.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:15:22.018 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:15:22.034 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:21 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:22.035 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:21 smithi090 ceph-mon[41670]: pgmap v326: 281 pgs: 32 unknown, 249 active+clean; 586 KiB data, 563 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 1 op/s 2024-04-14T15:15:22.035 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:21 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/926874511' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi090-93342-36"}]': finished 2024-04-14T15:15:22.035 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:21 smithi090 ceph-mon[41670]: osdmap e254: 8 total, 8 up, 8 in 2024-04-14T15:15:22.035 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:21 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:15:22.035 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:21 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:15:22.035 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:21 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/747873163' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi090-93342-37"}]: dispatch 2024-04-14T15:15:22.035 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:21 smithi090 ceph-mon[41670]: from='client.48652 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi090-93342-37"}]: dispatch 2024-04-14T15:15:22.035 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:21 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/747873163' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi090-93342-37"}]: dispatch 2024-04-14T15:15:22.035 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:21 smithi090 ceph-mon[41670]: from='client.48652 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi090-93342-37"}]: dispatch 2024-04-14T15:15:22.035 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:21 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/747873163' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi090-93342-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:22.035 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:21 smithi090 ceph-mon[41670]: from='client.48652 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi090-93342-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:22.035 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:21 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:22.036 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:21 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:22.036 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:21 smithi090 ceph-mon[30399]: pgmap v326: 281 pgs: 32 unknown, 249 active+clean; 586 KiB data, 563 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 1 op/s 2024-04-14T15:15:22.036 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:21 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/926874511' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi090-93342-36"}]': finished 2024-04-14T15:15:22.036 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:21 smithi090 ceph-mon[30399]: osdmap e254: 8 total, 8 up, 8 in 2024-04-14T15:15:22.036 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:21 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:15:22.036 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:21 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:15:22.036 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:21 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/747873163' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi090-93342-37"}]: dispatch 2024-04-14T15:15:22.036 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:21 smithi090 ceph-mon[30399]: from='client.48652 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi090-93342-37"}]: dispatch 2024-04-14T15:15:22.036 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:21 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/747873163' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi090-93342-37"}]: dispatch 2024-04-14T15:15:22.036 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:21 smithi090 ceph-mon[30399]: from='client.48652 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi090-93342-37"}]: dispatch 2024-04-14T15:15:22.036 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:21 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/747873163' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi090-93342-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:22.036 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:21 smithi090 ceph-mon[30399]: from='client.48652 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi090-93342-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:22.036 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:21 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:22.042 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:22.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:22.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:15:22.144 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:15:22.170 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:22.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:22.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:15:22.272 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:15:22.273 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:21 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:22.273 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:21 smithi092 ceph-mon[33966]: pgmap v326: 281 pgs: 32 unknown, 249 active+clean; 586 KiB data, 563 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 1 op/s 2024-04-14T15:15:22.274 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:21 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/926874511' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi090-93342-36"}]': finished 2024-04-14T15:15:22.274 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:21 smithi092 ceph-mon[33966]: osdmap e254: 8 total, 8 up, 8 in 2024-04-14T15:15:22.274 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:21 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:15:22.274 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:21 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:15:22.274 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:21 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/747873163' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi090-93342-37"}]: dispatch 2024-04-14T15:15:22.274 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:21 smithi092 ceph-mon[33966]: from='client.48652 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi090-93342-37"}]: dispatch 2024-04-14T15:15:22.274 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:21 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/747873163' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi090-93342-37"}]: dispatch 2024-04-14T15:15:22.274 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:21 smithi092 ceph-mon[33966]: from='client.48652 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi090-93342-37"}]: dispatch 2024-04-14T15:15:22.274 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:21 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/747873163' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi090-93342-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:22.274 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:21 smithi092 ceph-mon[33966]: from='client.48652 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi090-93342-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:22.275 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:21 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:22.297 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:22.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:22.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:15:22.399 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:15:22.423 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:22.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:22.424 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:15:22.525 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:15:22.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:22.552 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:22.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:15:22.653 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:15:22.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:22.678 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:22.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:15:22.779 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:15:22.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:22.804 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:22.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:15:22.905 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:15:22.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:22.931 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:22.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:15:23.032 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:15:23.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:22.952+0000 7f579e5c4700 1 -- 172.21.15.90:0/4237847331 >> v1:172.21.15.90:6789/0 conn(0x7f579810e200 legacy=0x7f57981106c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:23.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:22.952+0000 7f579e5c4700 1 -- 172.21.15.90:0/4237847331 shutdown_connections 2024-04-14T15:15:23.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:22.952+0000 7f579e5c4700 1 -- 172.21.15.90:0/4237847331 >> 172.21.15.90:0/4237847331 conn(0x7f57980fd290 msgr2=0x7f57980ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:15:23.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:22.952+0000 7f579e5c4700 1 -- 172.21.15.90:0/4237847331 shutdown_connections 2024-04-14T15:15:23.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:22.952+0000 7f579e5c4700 1 -- 172.21.15.90:0/4237847331 wait complete. 2024-04-14T15:15:23.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:22.953+0000 7f579e5c4700 1 librados: starting msgr at 2024-04-14T15:15:23.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:22.953+0000 7f579e5c4700 1 librados: starting objecter 2024-04-14T15:15:23.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:22.953+0000 7f579e5c4700 1 Processor -- start 2024-04-14T15:15:23.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:22.953+0000 7f579e5c4700 1 -- start start 2024-04-14T15:15:23.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:22.953+0000 7f579e5c4700 1 librados: setting wanted keys 2024-04-14T15:15:23.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:22.953+0000 7f579e5c4700 1 librados: calling monclient init 2024-04-14T15:15:23.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:22.953+0000 7f579e5c4700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f579810dfc0 con 0x7f5798107780 2024-04-14T15:15:23.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:22.953+0000 7f579e5c4700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f57981a8e20 con 0x7f579810a630 2024-04-14T15:15:23.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:22.953+0000 7f579e5c4700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f57981aa060 con 0x7f579810e200 2024-04-14T15:15:23.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:22.953+0000 7f5797fff700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f5798107780 0x7f579806aed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:51636/0 (socket says 172.21.15.90:51636) 2024-04-14T15:15:23.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:22.953+0000 7f5797fff700 1 -- 172.21.15.90:0/3622376841 learned_addr learned my addr 172.21.15.90:0/3622376841 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:15:23.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:22.953+0000 7f57957fa700 10 client.?.objecter ms_handle_connect 0x7f5798107780 2024-04-14T15:15:23.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:22.953+0000 7f57957fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:15:23.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:22.953+0000 7f57957fa700 10 client.?.objecter ms_handle_connect 0x7f579810e200 2024-04-14T15:15:23.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:22.953+0000 7f57957fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:15:23.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:22.953+0000 7f57957fa700 10 client.?.objecter ms_handle_connect 0x7f579810a630 2024-04-14T15:15:23.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:22.953+0000 7f57957fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:15:23.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:22.953+0000 7f57957fa700 1 -- 172.21.15.90:0/3622376841 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1586958931 0 0) 0x7f579810dfc0 con 0x7f5798107780 2024-04-14T15:15:23.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:22.954+0000 7f57957fa700 1 -- 172.21.15.90:0/3622376841 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f57800031f0 con 0x7f5798107780 2024-04-14T15:15:23.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:22.954+0000 7f57957fa700 1 -- 172.21.15.90:0/3622376841 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1833984944 0 0) 0x7f57981aa060 con 0x7f579810e200 2024-04-14T15:15:23.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:22.954+0000 7f57957fa700 1 -- 172.21.15.90:0/3622376841 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f579810dfc0 con 0x7f579810e200 2024-04-14T15:15:23.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:22.954+0000 7f57957fa700 1 -- 172.21.15.90:0/3622376841 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1523656114 0 0) 0x7f57981a8e20 con 0x7f579810a630 2024-04-14T15:15:23.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:22.954+0000 7f57957fa700 1 -- 172.21.15.90:0/3622376841 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f57981aa060 con 0x7f579810a630 2024-04-14T15:15:23.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:22.954+0000 7f57957fa700 1 -- 172.21.15.90:0/3622376841 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2945173926 0 0) 0x7f57800031f0 con 0x7f5798107780 2024-04-14T15:15:23.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:22.954+0000 7f57957fa700 1 -- 172.21.15.90:0/3622376841 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f57981a8e20 con 0x7f5798107780 2024-04-14T15:15:23.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:22.954+0000 7f57957fa700 1 -- 172.21.15.90:0/3622376841 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f57880018b0 con 0x7f5798107780 2024-04-14T15:15:23.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:22.954+0000 7f57957fa700 1 -- 172.21.15.90:0/3622376841 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1294733409 0 0) 0x7f57981aa060 con 0x7f579810a630 2024-04-14T15:15:23.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:22.954+0000 7f57957fa700 1 -- 172.21.15.90:0/3622376841 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f57800031f0 con 0x7f579810a630 2024-04-14T15:15:23.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:22.954+0000 7f57957fa700 1 -- 172.21.15.90:0/3622376841 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2755594068 0 0) 0x7f579810dfc0 con 0x7f579810e200 2024-04-14T15:15:23.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:22.954+0000 7f57957fa700 1 -- 172.21.15.90:0/3622376841 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f57981aa060 con 0x7f579810e200 2024-04-14T15:15:23.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:22.954+0000 7f57957fa700 1 -- 172.21.15.90:0/3622376841 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f578c0017f0 con 0x7f579810a630 2024-04-14T15:15:23.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:22.954+0000 7f57957fa700 1 -- 172.21.15.90:0/3622376841 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f57900017b0 con 0x7f579810e200 2024-04-14T15:15:23.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:22.954+0000 7f57957fa700 1 -- 172.21.15.90:0/3622376841 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 527338137 0 0) 0x7f57981a8e20 con 0x7f5798107780 2024-04-14T15:15:23.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:22.954+0000 7f57957fa700 1 -- 172.21.15.90:0/3622376841 >> v1:172.21.15.90:6790/0 conn(0x7f579810e200 legacy=0x7f57981a66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:23.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:22.954+0000 7f57957fa700 1 -- 172.21.15.90:0/3622376841 >> v1:172.21.15.92:6789/0 conn(0x7f579810a630 legacy=0x7f579806b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:23.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:22.954+0000 7f57957fa700 1 -- 172.21.15.90:0/3622376841 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57981ab2a0 con 0x7f5798107780 2024-04-14T15:15:23.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:22.954+0000 7f579e5c4700 1 -- 172.21.15.90:0/3622376841 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f57981aa290 con 0x7f5798107780 2024-04-14T15:15:23.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:22.954+0000 7f579e5c4700 10 client.36810.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:15:23.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:22.954+0000 7f579e5c4700 1 -- 172.21.15.90:0/3622376841 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f57981aa750 con 0x7f5798107780 2024-04-14T15:15:23.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:22.954+0000 7f579e5c4700 1 librados: init done 2024-04-14T15:15:23.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:22.954+0000 7f57957fa700 1 -- 172.21.15.90:0/3622376841 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f5788001a50 con 0x7f5798107780 2024-04-14T15:15:23.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:22.954+0000 7f57957fa700 1 -- 172.21.15.90:0/3622376841 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f5788005c60 con 0x7f5798107780 2024-04-14T15:15:23.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:22.955+0000 7f579e5c4700 1 -- 172.21.15.90:0/3622376841 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5784005150 con 0x7f5798107780 2024-04-14T15:15:23.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:22.956+0000 7f57957fa700 1 -- 172.21.15.90:0/3622376841 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f5788003680 con 0x7f5798107780 2024-04-14T15:15:23.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:22.956+0000 7f57957fa700 1 -- 172.21.15.90:0/3622376841 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(255..255 src has 51..255) ==== 9150+0+0 (unknown 2932370694 0 0) 0x7f578808acf0 con 0x7f5798107780 2024-04-14T15:15:23.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:22.956+0000 7f57957fa700 10 client.36810.objecter ms_dispatch 0x7f5798001550 osd_map(255..255 src has 51..255) 2024-04-14T15:15:23.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:22.956+0000 7f57957fa700 3 client.36810.objecter handle_osd_map got epochs [255,255] > 0 2024-04-14T15:15:23.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:22.956+0000 7f57957fa700 3 client.36810.objecter handle_osd_map decoding full epoch 255 2024-04-14T15:15:23.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:22.956+0000 7f57957fa700 20 client.36810.objecter dump_active .. 0 homeless 2024-04-14T15:15:23.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:22.957+0000 7f57957fa700 10 client.36810.objecter ms_handle_connect 0x7f5780071340 2024-04-14T15:15:23.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:22.961+0000 7f57957fa700 1 -- 172.21.15.90:0/3622376841 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f5788057320 con 0x7f5798107780 2024-04-14T15:15:23.057 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:23.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:23.058 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:15:23.102 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-14T15:15:23.159 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:15:23.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:23.183 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:23.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:15:23.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:23.182+0000 7f579e5c4700 1 -- 172.21.15.90:0/3622376841 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7f5784005440 con 0x7f5798107780 2024-04-14T15:15:23.284 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:15:23.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:23.310 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:23.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:15:23.411 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:15:23.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:23.438 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:23.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:15:23.539 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:15:23.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:23.567 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:23.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:15:23.668 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:15:23.693 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:23.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:23.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:15:23.796 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:15:23.820 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:23.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:23.821 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:15:23.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:23.860+0000 7f57957fa700 1 -- 172.21.15.90:0/3622376841 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v256) ==== 91+0+0 (unknown 2215888437 0 0) 0x7f578805afd0 con 0x7f5798107780 2024-04-14T15:15:23.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:23.868+0000 7f579e5c4700 10 librados: watch_flush enter 2024-04-14T15:15:23.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:23.869+0000 7f579e5c4700 10 librados: watch_flush exit 2024-04-14T15:15:23.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:23.869+0000 7f579e5c4700 20 client.36810.objecter shutdown clearing up homeless session... 2024-04-14T15:15:23.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:23.869+0000 7f579e5c4700 10 client.36810.objecter successfully canceled tick 2024-04-14T15:15:23.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:23.870+0000 7f579e5c4700 1 -- 172.21.15.90:0/3622376841 >> v1:172.21.15.90:6800/420935427 conn(0x7f5780071340 legacy=0x7f5780073800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:23.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:23.870+0000 7f579e5c4700 1 -- 172.21.15.90:0/3622376841 >> v1:172.21.15.90:6789/0 conn(0x7f5798107780 legacy=0x7f579806aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:23.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:23.870+0000 7f579e5c4700 1 -- 172.21.15.90:0/3622376841 shutdown_connections 2024-04-14T15:15:23.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:23.870+0000 7f579e5c4700 1 -- 172.21.15.90:0/3622376841 >> 172.21.15.90:0/3622376841 conn(0x7f57980fd290 msgr2=0x7f57981116a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:15:23.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:23.872+0000 7f579e5c4700 1 -- 172.21.15.90:0/3622376841 shutdown_connections 2024-04-14T15:15:23.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:23.873+0000 7f579e5c4700 1 -- 172.21.15.90:0/3622376841 wait complete. 2024-04-14T15:15:23.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:23.875+0000 7f579e5c4700 1 librados: shutdown 2024-04-14T15:15:23.878 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is unset 2024-04-14T15:15:23.922 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:15:23.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:23.947 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:23.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:15:24.048 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:15:24.071 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:24.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:24.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:15:24.108 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:15:23 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:15:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:15:24.108 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:23 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:15:24.108 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:23 smithi090 ceph-mon[30399]: from='client.48652 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi090-93342-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:15:24.108 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:23 smithi090 ceph-mon[30399]: osdmap e255: 8 total, 8 up, 8 in 2024-04-14T15:15:24.109 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:23 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-27"}]: dispatch 2024-04-14T15:15:24.109 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:23 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-27"}]: dispatch 2024-04-14T15:15:24.109 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:23 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/505613538' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi090-93363-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:24.109 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:23 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/747873163' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi090-93342-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi090-93342-37"}]: dispatch 2024-04-14T15:15:24.109 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:23 smithi090 ceph-mon[30399]: from='client.48649 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi090-93363-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:24.109 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:23 smithi090 ceph-mon[30399]: from='client.48652 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi090-93342-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi090-93342-37"}]: dispatch 2024-04-14T15:15:24.109 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:23 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:24.109 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:23 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3622376841' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:15:24.109 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:23 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:15:24.109 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:23 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:15:24.109 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:23 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:15:24.110 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:23 smithi090 ceph-mon[41670]: from='client.48652 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi090-93342-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:15:24.110 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:23 smithi090 ceph-mon[41670]: osdmap e255: 8 total, 8 up, 8 in 2024-04-14T15:15:24.110 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:23 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-27"}]: dispatch 2024-04-14T15:15:24.110 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:23 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-27"}]: dispatch 2024-04-14T15:15:24.110 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:23 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/505613538' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi090-93363-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:24.110 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:23 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/747873163' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi090-93342-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi090-93342-37"}]: dispatch 2024-04-14T15:15:24.110 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:23 smithi090 ceph-mon[41670]: from='client.48649 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi090-93363-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:24.110 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:23 smithi090 ceph-mon[41670]: from='client.48652 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi090-93342-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi090-93342-37"}]: dispatch 2024-04-14T15:15:24.110 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:23 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:24.110 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:23 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3622376841' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:15:24.110 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:23 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:15:24.110 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:23 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:15:24.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:23 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:15:24.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:23 smithi092 ceph-mon[33966]: from='client.48652 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi090-93342-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:15:24.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:23 smithi092 ceph-mon[33966]: osdmap e255: 8 total, 8 up, 8 in 2024-04-14T15:15:24.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:23 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-27"}]: dispatch 2024-04-14T15:15:24.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:23 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-27"}]: dispatch 2024-04-14T15:15:24.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:23 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/505613538' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi090-93363-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:24.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:23 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/747873163' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi090-93342-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi090-93342-37"}]: dispatch 2024-04-14T15:15:24.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:23 smithi092 ceph-mon[33966]: from='client.48649 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi090-93363-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:24.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:23 smithi092 ceph-mon[33966]: from='client.48652 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi090-93342-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi090-93342-37"}]: dispatch 2024-04-14T15:15:24.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:23 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:24.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:23 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3622376841' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:15:24.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:23 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:15:24.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:23 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:15:24.136 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-14T15:15:24.136 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-14T15:15:24.137 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-14T15:15:24.137 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-14T15:15:24.137 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-14T15:15:24.173 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:15:24.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:24.197 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:24.198 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:15:24.299 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:15:24.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:24.324 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:24.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:15:24.425 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:15:24.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:24.450 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:24.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:15:24.551 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:15:24.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:24.577 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:24.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:15:24.678 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:15:24.706 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:24.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:24.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:15:24.808 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:15:24.835 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:24.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:24.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:15:24.937 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:15:24.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:24.962 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:24.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:15:25.063 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:15:25.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:25.088 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:25.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:15:25.090 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:24 smithi090 ceph-mon[30399]: pgmap v329: 281 pgs: 32 unknown, 249 active+clean; 586 KiB data, 563 MiB used, 715 GiB / 715 GiB avail; 1023 B/s wr, 0 op/s 2024-04-14T15:15:25.090 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:24 smithi090 ceph-mon[30399]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:15:25.091 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:24 smithi090 ceph-mon[30399]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:15:25.091 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:24 smithi090 ceph-mon[30399]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:15:25.091 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:24 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-27"}]': finished 2024-04-14T15:15:25.091 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:24 smithi090 ceph-mon[30399]: from='client.48649 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi090-93363-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:15:25.091 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:24 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3622376841' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-14T15:15:25.091 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:24 smithi090 ceph-mon[30399]: osdmap e256: 8 total, 8 up, 8 in 2024-04-14T15:15:25.091 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:24 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:25.092 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:24 smithi090 ceph-mon[41670]: pgmap v329: 281 pgs: 32 unknown, 249 active+clean; 586 KiB data, 563 MiB used, 715 GiB / 715 GiB avail; 1023 B/s wr, 0 op/s 2024-04-14T15:15:25.092 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:24 smithi090 ceph-mon[41670]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:15:25.092 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:24 smithi090 ceph-mon[41670]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:15:25.092 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:24 smithi090 ceph-mon[41670]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:15:25.092 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:24 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-27"}]': finished 2024-04-14T15:15:25.092 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:24 smithi090 ceph-mon[41670]: from='client.48649 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi090-93363-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:15:25.092 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:24 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3622376841' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-14T15:15:25.092 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:24 smithi090 ceph-mon[41670]: osdmap e256: 8 total, 8 up, 8 in 2024-04-14T15:15:25.092 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:24 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:25.096 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:15:25.122 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:15:24 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:15:25.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:24 smithi092 ceph-mon[33966]: pgmap v329: 281 pgs: 32 unknown, 249 active+clean; 586 KiB data, 563 MiB used, 715 GiB / 715 GiB avail; 1023 B/s wr, 0 op/s 2024-04-14T15:15:25.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:24 smithi092 ceph-mon[33966]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:15:25.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:24 smithi092 ceph-mon[33966]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:15:25.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:24 smithi092 ceph-mon[33966]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:15:25.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:24 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-27"}]': finished 2024-04-14T15:15:25.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:24 smithi092 ceph-mon[33966]: from='client.48649 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi090-93363-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:15:25.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:24 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3622376841' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-14T15:15:25.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:24 smithi092 ceph-mon[33966]: osdmap e256: 8 total, 8 up, 8 in 2024-04-14T15:15:25.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:24 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:25.124 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-14T15:15:25.189 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:15:25.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:25.231 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:25.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:15:25.332 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:15:25.356 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:25.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:25.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:15:25.458 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:15:25.487 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:25.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:25.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:15:25.588 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:15:25.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:25.617 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:25.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:15:25.718 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:15:25.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:25.746 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:25.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:15:25.847 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:15:25.879 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:25.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:25.881 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:15:25.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:25 smithi090 ceph-mon[30399]: 110.c scrub starts 2024-04-14T15:15:25.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:25 smithi090 ceph-mon[30399]: 110.c scrub ok 2024-04-14T15:15:25.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:25 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:15:25.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:25 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:15:25.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:25 smithi090 ceph-mon[30399]: 110.e scrub starts 2024-04-14T15:15:25.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:25 smithi090 ceph-mon[30399]: 110.e scrub ok 2024-04-14T15:15:25.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:25 smithi090 ceph-mon[30399]: 110.16 scrub starts 2024-04-14T15:15:25.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:25 smithi090 ceph-mon[30399]: 110.16 scrub ok 2024-04-14T15:15:25.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:25 smithi090 ceph-mon[30399]: from='client.48652 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi090-93342-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi090-93342-37"}]': finished 2024-04-14T15:15:25.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:25 smithi090 ceph-mon[30399]: osdmap e257: 8 total, 8 up, 8 in 2024-04-14T15:15:25.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:25 smithi090 ceph-mon[30399]: 11.1b scrub starts 2024-04-14T15:15:25.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:25 smithi090 ceph-mon[30399]: 11.1b scrub ok 2024-04-14T15:15:25.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:25 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2597116422' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi090-93363-46"}]: dispatch 2024-04-14T15:15:25.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:25 smithi090 ceph-mon[30399]: from='client.46769 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi090-93363-46"}]: dispatch 2024-04-14T15:15:25.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:25 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2597116422' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi090-93363-46"}]: dispatch 2024-04-14T15:15:25.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:25 smithi090 ceph-mon[30399]: from='client.46769 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi090-93363-46"}]: dispatch 2024-04-14T15:15:25.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:25 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2597116422' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi090-93363-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:25.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:25 smithi090 ceph-mon[30399]: from='client.46769 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi090-93363-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:25.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:25 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:15:25.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:25 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:25.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:25 smithi090 ceph-mon[30399]: 2.d scrub starts 2024-04-14T15:15:25.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:25 smithi090 ceph-mon[30399]: 2.d scrub ok 2024-04-14T15:15:25.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:25 smithi090 ceph-mon[30399]: 110.8 scrub starts 2024-04-14T15:15:25.944 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:25 smithi090 ceph-mon[30399]: 110.8 scrub ok 2024-04-14T15:15:25.944 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:25 smithi090 ceph-mon[30399]: from='client.46769 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi090-93363-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:15:25.944 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:25 smithi090 ceph-mon[30399]: osdmap e258: 8 total, 8 up, 8 in 2024-04-14T15:15:25.944 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:25 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:25.944 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:25 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:25.944 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:25 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2597116422' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi090-93363-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi090-93363-46"}]: dispatch 2024-04-14T15:15:25.944 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:25 smithi090 ceph-mon[30399]: from='client.46769 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi090-93363-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi090-93363-46"}]: dispatch 2024-04-14T15:15:25.944 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:25 smithi090 ceph-mon[41670]: 110.c scrub starts 2024-04-14T15:15:25.944 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:25 smithi090 ceph-mon[41670]: 110.c scrub ok 2024-04-14T15:15:25.944 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:25 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:15:25.944 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:25 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:15:25.945 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:25 smithi090 ceph-mon[41670]: 110.e scrub starts 2024-04-14T15:15:25.945 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:25 smithi090 ceph-mon[41670]: 110.e scrub ok 2024-04-14T15:15:25.945 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:25 smithi090 ceph-mon[41670]: 110.16 scrub starts 2024-04-14T15:15:25.945 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:25 smithi090 ceph-mon[41670]: 110.16 scrub ok 2024-04-14T15:15:25.945 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:25 smithi090 ceph-mon[41670]: from='client.48652 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi090-93342-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi090-93342-37"}]': finished 2024-04-14T15:15:25.945 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:25 smithi090 ceph-mon[41670]: osdmap e257: 8 total, 8 up, 8 in 2024-04-14T15:15:25.945 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:25 smithi090 ceph-mon[41670]: 11.1b scrub starts 2024-04-14T15:15:25.945 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:25 smithi090 ceph-mon[41670]: 11.1b scrub ok 2024-04-14T15:15:25.945 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:25 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2597116422' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi090-93363-46"}]: dispatch 2024-04-14T15:15:25.945 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:25 smithi090 ceph-mon[41670]: from='client.46769 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi090-93363-46"}]: dispatch 2024-04-14T15:15:25.945 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:25 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2597116422' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi090-93363-46"}]: dispatch 2024-04-14T15:15:25.946 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:25 smithi090 ceph-mon[41670]: from='client.46769 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi090-93363-46"}]: dispatch 2024-04-14T15:15:25.946 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:25 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2597116422' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi090-93363-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:25.946 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:25 smithi090 ceph-mon[41670]: from='client.46769 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi090-93363-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:25.946 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:25 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:15:25.946 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:25 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:25.946 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:25 smithi090 ceph-mon[41670]: 2.d scrub starts 2024-04-14T15:15:25.946 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:25 smithi090 ceph-mon[41670]: 2.d scrub ok 2024-04-14T15:15:25.946 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:25 smithi090 ceph-mon[41670]: 110.8 scrub starts 2024-04-14T15:15:25.946 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:25 smithi090 ceph-mon[41670]: 110.8 scrub ok 2024-04-14T15:15:25.946 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:25 smithi090 ceph-mon[41670]: from='client.46769 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi090-93363-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:15:25.947 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:25 smithi090 ceph-mon[41670]: osdmap e258: 8 total, 8 up, 8 in 2024-04-14T15:15:25.947 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:25 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:25.947 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:25 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:25.947 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:25 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2597116422' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi090-93363-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi090-93363-46"}]: dispatch 2024-04-14T15:15:25.947 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:25 smithi090 ceph-mon[41670]: from='client.46769 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi090-93363-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi090-93363-46"}]: dispatch 2024-04-14T15:15:25.981 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:15:26.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:26.008 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:26.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:15:26.109 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:15:26.133 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:26.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:26.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:15:26.235 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:15:26.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:26.260 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:26.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:15:26.360 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:15:26.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:25 smithi092 ceph-mon[33966]: 110.c scrub starts 2024-04-14T15:15:26.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:25 smithi092 ceph-mon[33966]: 110.c scrub ok 2024-04-14T15:15:26.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:25 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:15:26.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:25 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:15:26.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:25 smithi092 ceph-mon[33966]: 110.e scrub starts 2024-04-14T15:15:26.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:25 smithi092 ceph-mon[33966]: 110.e scrub ok 2024-04-14T15:15:26.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:25 smithi092 ceph-mon[33966]: 110.16 scrub starts 2024-04-14T15:15:26.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:25 smithi092 ceph-mon[33966]: 110.16 scrub ok 2024-04-14T15:15:26.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:25 smithi092 ceph-mon[33966]: from='client.48652 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi090-93342-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi090-93342-37"}]': finished 2024-04-14T15:15:26.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:25 smithi092 ceph-mon[33966]: osdmap e257: 8 total, 8 up, 8 in 2024-04-14T15:15:26.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:25 smithi092 ceph-mon[33966]: 11.1b scrub starts 2024-04-14T15:15:26.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:25 smithi092 ceph-mon[33966]: 11.1b scrub ok 2024-04-14T15:15:26.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:25 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2597116422' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi090-93363-46"}]: dispatch 2024-04-14T15:15:26.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:25 smithi092 ceph-mon[33966]: from='client.46769 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi090-93363-46"}]: dispatch 2024-04-14T15:15:26.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:25 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2597116422' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi090-93363-46"}]: dispatch 2024-04-14T15:15:26.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:25 smithi092 ceph-mon[33966]: from='client.46769 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi090-93363-46"}]: dispatch 2024-04-14T15:15:26.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:25 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2597116422' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi090-93363-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:26.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:25 smithi092 ceph-mon[33966]: from='client.46769 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi090-93363-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:26.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:25 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:15:26.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:25 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:26.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:25 smithi092 ceph-mon[33966]: 2.d scrub starts 2024-04-14T15:15:26.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:25 smithi092 ceph-mon[33966]: 2.d scrub ok 2024-04-14T15:15:26.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:25 smithi092 ceph-mon[33966]: 110.8 scrub starts 2024-04-14T15:15:26.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:25 smithi092 ceph-mon[33966]: 110.8 scrub ok 2024-04-14T15:15:26.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:25 smithi092 ceph-mon[33966]: from='client.46769 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi090-93363-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:15:26.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:25 smithi092 ceph-mon[33966]: osdmap e258: 8 total, 8 up, 8 in 2024-04-14T15:15:26.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:25 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:26.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:25 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:26.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:25 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2597116422' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi090-93363-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi090-93363-46"}]: dispatch 2024-04-14T15:15:26.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:25 smithi092 ceph-mon[33966]: from='client.46769 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi090-93363-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi090-93363-46"}]: dispatch 2024-04-14T15:15:26.387 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:26.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:26.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:15:26.489 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:15:26.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:26.516 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:26.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:15:26.617 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:15:26.645 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:26.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:26.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:15:26.746 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:15:26.773 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:26.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:26.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:15:26.826 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:15:26.875 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:15:26.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:26.906 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:26.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:15:27.007 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:15:27.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:27.032 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:27.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:15:27.059 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-14T15:15:27.059 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-14T15:15:27.059 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-04-14T15:15:27.059 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-14T15:15:27.059 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-14T15:15:27.060 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-04-14T15:15:27.069 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:26 smithi090 ceph-mon[41670]: pgmap v332: 225 pgs: 8 unknown, 217 active+clean; 586 KiB data, 547 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 3 op/s 2024-04-14T15:15:27.069 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:26 smithi090 ceph-mon[41670]: 110.f scrub starts 2024-04-14T15:15:27.069 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:26 smithi090 ceph-mon[41670]: 110.f scrub ok 2024-04-14T15:15:27.069 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:26 smithi090 ceph-mon[41670]: 110.0 scrub starts 2024-04-14T15:15:27.069 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:26 smithi090 ceph-mon[41670]: 110.0 scrub ok 2024-04-14T15:15:27.069 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:26 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:27.069 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:26 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:15:27.069 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:26 smithi090 ceph-mon[41670]: osdmap e259: 8 total, 8 up, 8 in 2024-04-14T15:15:27.069 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:26 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/747873163' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi090-93342-37"}]: dispatch 2024-04-14T15:15:27.069 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:26 smithi090 ceph-mon[41670]: from='client.48652 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi090-93342-37"}]: dispatch 2024-04-14T15:15:27.070 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:26 smithi090 ceph-mon[30399]: pgmap v332: 225 pgs: 8 unknown, 217 active+clean; 586 KiB data, 547 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 3 op/s 2024-04-14T15:15:27.070 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:26 smithi090 ceph-mon[30399]: 110.f scrub starts 2024-04-14T15:15:27.070 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:26 smithi090 ceph-mon[30399]: 110.f scrub ok 2024-04-14T15:15:27.070 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:26 smithi090 ceph-mon[30399]: 110.0 scrub starts 2024-04-14T15:15:27.070 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:26 smithi090 ceph-mon[30399]: 110.0 scrub ok 2024-04-14T15:15:27.070 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:26 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:27.070 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:26 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:15:27.070 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:26 smithi090 ceph-mon[30399]: osdmap e259: 8 total, 8 up, 8 in 2024-04-14T15:15:27.070 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:26 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/747873163' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi090-93342-37"}]: dispatch 2024-04-14T15:15:27.071 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:26 smithi090 ceph-mon[30399]: from='client.48652 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi090-93342-37"}]: dispatch 2024-04-14T15:15:27.134 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:15:27.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:27.160 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:27.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:15:27.262 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:15:27.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:27.289 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:27.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:15:27.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:26 smithi092 ceph-mon[33966]: pgmap v332: 225 pgs: 8 unknown, 217 active+clean; 586 KiB data, 547 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 3 op/s 2024-04-14T15:15:27.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:26 smithi092 ceph-mon[33966]: 110.f scrub starts 2024-04-14T15:15:27.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:26 smithi092 ceph-mon[33966]: 110.f scrub ok 2024-04-14T15:15:27.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:26 smithi092 ceph-mon[33966]: 110.0 scrub starts 2024-04-14T15:15:27.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:26 smithi092 ceph-mon[33966]: 110.0 scrub ok 2024-04-14T15:15:27.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:26 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:27.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:26 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:15:27.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:26 smithi092 ceph-mon[33966]: osdmap e259: 8 total, 8 up, 8 in 2024-04-14T15:15:27.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:26 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/747873163' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi090-93342-37"}]: dispatch 2024-04-14T15:15:27.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:26 smithi092 ceph-mon[33966]: from='client.48652 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi090-93342-37"}]: dispatch 2024-04-14T15:15:27.390 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:15:27.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:27.452 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:27.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:15:27.553 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:15:27.580 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:27.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:27.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:15:27.682 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:15:27.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:27.707 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:27.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:15:27.808 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:15:27.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:27.832 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:27.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:15:27.933 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:15:27.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:27.958 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:27.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:15:28.059 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:15:28.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:28.087 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:28.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:15:28.188 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:15:28.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:27 smithi090 ceph-mon[41670]: 110.1d scrub starts 2024-04-14T15:15:28.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:27 smithi090 ceph-mon[41670]: 110.1d scrub ok 2024-04-14T15:15:28.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:27 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:28.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:27 smithi090 ceph-mon[41670]: 11.1c scrub starts 2024-04-14T15:15:28.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:27 smithi090 ceph-mon[41670]: 11.1c scrub ok 2024-04-14T15:15:28.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:27 smithi090 ceph-mon[41670]: from='client.46769 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi090-93363-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi090-93363-46"}]': finished 2024-04-14T15:15:28.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:27 smithi090 ceph-mon[41670]: from='client.48652 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi090-93342-37"}]': finished 2024-04-14T15:15:28.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:27 smithi090 ceph-mon[41670]: osdmap e260: 8 total, 8 up, 8 in 2024-04-14T15:15:28.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:27 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/747873163' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi090-93342-37"}]: dispatch 2024-04-14T15:15:28.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:27 smithi090 ceph-mon[41670]: from='client.48652 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi090-93342-37"}]: dispatch 2024-04-14T15:15:28.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:27 smithi090 ceph-mon[30399]: 110.1d scrub starts 2024-04-14T15:15:28.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:27 smithi090 ceph-mon[30399]: 110.1d scrub ok 2024-04-14T15:15:28.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:27 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:28.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:27 smithi090 ceph-mon[30399]: 11.1c scrub starts 2024-04-14T15:15:28.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:27 smithi090 ceph-mon[30399]: 11.1c scrub ok 2024-04-14T15:15:28.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:27 smithi090 ceph-mon[30399]: from='client.46769 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi090-93363-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi090-93363-46"}]': finished 2024-04-14T15:15:28.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:27 smithi090 ceph-mon[30399]: from='client.48652 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi090-93342-37"}]': finished 2024-04-14T15:15:28.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:27 smithi090 ceph-mon[30399]: osdmap e260: 8 total, 8 up, 8 in 2024-04-14T15:15:28.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:27 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/747873163' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi090-93342-37"}]: dispatch 2024-04-14T15:15:28.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:27 smithi090 ceph-mon[30399]: from='client.48652 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi090-93342-37"}]: dispatch 2024-04-14T15:15:28.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:28.214 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:28.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:15:28.315 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:15:28.343 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:28.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:28.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:15:28.367 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-14T15:15:28.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:27 smithi092 ceph-mon[33966]: 110.1d scrub starts 2024-04-14T15:15:28.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:27 smithi092 ceph-mon[33966]: 110.1d scrub ok 2024-04-14T15:15:28.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:27 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:28.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:27 smithi092 ceph-mon[33966]: 11.1c scrub starts 2024-04-14T15:15:28.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:27 smithi092 ceph-mon[33966]: 11.1c scrub ok 2024-04-14T15:15:28.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:27 smithi092 ceph-mon[33966]: from='client.46769 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi090-93363-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi090-93363-46"}]': finished 2024-04-14T15:15:28.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:27 smithi092 ceph-mon[33966]: from='client.48652 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi090-93342-37"}]': finished 2024-04-14T15:15:28.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:27 smithi092 ceph-mon[33966]: osdmap e260: 8 total, 8 up, 8 in 2024-04-14T15:15:28.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:27 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/747873163' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi090-93342-37"}]: dispatch 2024-04-14T15:15:28.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:27 smithi092 ceph-mon[33966]: from='client.48652 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi090-93342-37"}]: dispatch 2024-04-14T15:15:28.445 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:15:28.472 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:28.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:28.474 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:15:28.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.491+0000 7f4b30626700 1 -- 172.21.15.90:0/1981977995 >> v1:172.21.15.90:6789/0 conn(0x7f4b2810b750 legacy=0x7f4b2810db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:28.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.492+0000 7f4b30626700 1 -- 172.21.15.90:0/1981977995 shutdown_connections 2024-04-14T15:15:28.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.492+0000 7f4b30626700 1 -- 172.21.15.90:0/1981977995 >> 172.21.15.90:0/1981977995 conn(0x7f4b280fe3d0 msgr2=0x7f4b281007f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:15:28.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.492+0000 7f4b30626700 1 -- 172.21.15.90:0/1981977995 shutdown_connections 2024-04-14T15:15:28.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.492+0000 7f4b30626700 1 -- 172.21.15.90:0/1981977995 wait complete. 2024-04-14T15:15:28.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.492+0000 7f4b30626700 1 librados: starting msgr at 2024-04-14T15:15:28.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.492+0000 7f4b30626700 1 librados: starting objecter 2024-04-14T15:15:28.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.492+0000 7f4b30626700 1 Processor -- start 2024-04-14T15:15:28.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.492+0000 7f4b30626700 1 -- start start 2024-04-14T15:15:28.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.492+0000 7f4b30626700 1 librados: setting wanted keys 2024-04-14T15:15:28.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.492+0000 7f4b30626700 1 librados: calling monclient init 2024-04-14T15:15:28.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.493+0000 7f4b30626700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f4b2810f010 con 0x7f4b2810f320 2024-04-14T15:15:28.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.493+0000 7f4b30626700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f4b281a8f30 con 0x7f4b2810b750 2024-04-14T15:15:28.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.493+0000 7f4b2ebc3700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f4b2810f320 0x7f4b281a6800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:51666/0 (socket says 172.21.15.90:51666) 2024-04-14T15:15:28.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.493+0000 7f4b2ebc3700 1 -- 172.21.15.90:0/442819077 learned_addr learned my addr 172.21.15.90:0/442819077 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:15:28.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.493+0000 7f4b30626700 1 -- 172.21.15.90:0/442819077 --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f4b281aa110 con 0x7f4b281088a0 2024-04-14T15:15:28.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.493+0000 7f4b1f7fe700 10 client.?.objecter ms_handle_connect 0x7f4b2810f320 2024-04-14T15:15:28.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.493+0000 7f4b1f7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:15:28.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.493+0000 7f4b1f7fe700 10 client.?.objecter ms_handle_connect 0x7f4b281088a0 2024-04-14T15:15:28.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.493+0000 7f4b1f7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:15:28.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.493+0000 7f4b1f7fe700 1 -- 172.21.15.90:0/442819077 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 862440624 0 0) 0x7f4b2810f010 con 0x7f4b2810f320 2024-04-14T15:15:28.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.493+0000 7f4b1f7fe700 1 -- 172.21.15.90:0/442819077 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b140035b0 con 0x7f4b2810f320 2024-04-14T15:15:28.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.493+0000 7f4b1f7fe700 1 -- 172.21.15.90:0/442819077 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1762432446 0 0) 0x7f4b281aa110 con 0x7f4b281088a0 2024-04-14T15:15:28.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.493+0000 7f4b1f7fe700 1 -- 172.21.15.90:0/442819077 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b2810f010 con 0x7f4b281088a0 2024-04-14T15:15:28.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.493+0000 7f4b1f7fe700 10 client.?.objecter ms_handle_connect 0x7f4b2810b750 2024-04-14T15:15:28.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.493+0000 7f4b1f7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:15:28.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.494+0000 7f4b1f7fe700 1 -- 172.21.15.90:0/442819077 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 902038052 0 0) 0x7f4b140035b0 con 0x7f4b2810f320 2024-04-14T15:15:28.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.494+0000 7f4b1f7fe700 1 -- 172.21.15.90:0/442819077 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4b281aa110 con 0x7f4b2810f320 2024-04-14T15:15:28.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.494+0000 7f4b1f7fe700 1 -- 172.21.15.90:0/442819077 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f4b20001c30 con 0x7f4b2810f320 2024-04-14T15:15:28.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.494+0000 7f4b1f7fe700 1 -- 172.21.15.90:0/442819077 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2315488099 0 0) 0x7f4b281a8f30 con 0x7f4b2810b750 2024-04-14T15:15:28.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.494+0000 7f4b1f7fe700 1 -- 172.21.15.90:0/442819077 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b140035b0 con 0x7f4b2810b750 2024-04-14T15:15:28.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.494+0000 7f4b1f7fe700 1 -- 172.21.15.90:0/442819077 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1829300709 0 0) 0x7f4b2810f010 con 0x7f4b281088a0 2024-04-14T15:15:28.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.494+0000 7f4b1f7fe700 1 -- 172.21.15.90:0/442819077 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4b281a8f30 con 0x7f4b281088a0 2024-04-14T15:15:28.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.494+0000 7f4b1f7fe700 1 -- 172.21.15.90:0/442819077 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f4b18001830 con 0x7f4b281088a0 2024-04-14T15:15:28.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.494+0000 7f4b1f7fe700 1 -- 172.21.15.90:0/442819077 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 170146680 0 0) 0x7f4b140035b0 con 0x7f4b2810b750 2024-04-14T15:15:28.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.494+0000 7f4b1f7fe700 1 -- 172.21.15.90:0/442819077 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4b2810f010 con 0x7f4b2810b750 2024-04-14T15:15:28.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.494+0000 7f4b1f7fe700 1 -- 172.21.15.90:0/442819077 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1579199428 0 0) 0x7f4b281aa110 con 0x7f4b2810f320 2024-04-14T15:15:28.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.494+0000 7f4b1f7fe700 1 -- 172.21.15.90:0/442819077 >> v1:172.21.15.90:6790/0 conn(0x7f4b281088a0 legacy=0x7f4b2810bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:28.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.494+0000 7f4b1f7fe700 1 -- 172.21.15.90:0/442819077 >> v1:172.21.15.92:6789/0 conn(0x7f4b2810b750 legacy=0x7f4b2810c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:28.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.494+0000 7f4b1f7fe700 1 -- 172.21.15.90:0/442819077 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b281ab2f0 con 0x7f4b2810f320 2024-04-14T15:15:28.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.494+0000 7f4b1f7fe700 1 -- 172.21.15.90:0/442819077 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f4b20003ea0 con 0x7f4b2810f320 2024-04-14T15:15:28.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.494+0000 7f4b1f7fe700 1 -- 172.21.15.90:0/442819077 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f4b20006020 con 0x7f4b2810f320 2024-04-14T15:15:28.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.494+0000 7f4b30626700 1 -- 172.21.15.90:0/442819077 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4b281aa340 con 0x7f4b2810f320 2024-04-14T15:15:28.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.494+0000 7f4b30626700 10 client.36822.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:15:28.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.494+0000 7f4b30626700 1 -- 172.21.15.90:0/442819077 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f4b281aa800 con 0x7f4b2810f320 2024-04-14T15:15:28.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.494+0000 7f4b30626700 1 librados: init done 2024-04-14T15:15:28.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.496+0000 7f4b30626700 1 -- 172.21.15.90:0/442819077 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f4b0c005130 con 0x7f4b2810f320 2024-04-14T15:15:28.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.497+0000 7f4b1f7fe700 1 -- 172.21.15.90:0/442819077 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f4b20003a00 con 0x7f4b2810f320 2024-04-14T15:15:28.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.497+0000 7f4b1f7fe700 1 -- 172.21.15.90:0/442819077 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(260..260 src has 51..260) ==== 9404+0+0 (unknown 906408286 0 0) 0x7f4b2008b0a0 con 0x7f4b2810f320 2024-04-14T15:15:28.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.497+0000 7f4b1f7fe700 10 client.36822.objecter ms_dispatch 0x7f4b28001550 osd_map(260..260 src has 51..260) 2024-04-14T15:15:28.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.497+0000 7f4b1f7fe700 3 client.36822.objecter handle_osd_map got epochs [260,260] > 0 2024-04-14T15:15:28.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.497+0000 7f4b1f7fe700 3 client.36822.objecter handle_osd_map decoding full epoch 260 2024-04-14T15:15:28.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.498+0000 7f4b1f7fe700 20 client.36822.objecter dump_active .. 0 homeless 2024-04-14T15:15:28.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.498+0000 7f4b1f7fe700 1 -- 172.21.15.90:0/442819077 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v260) ==== 54+0+16 (unknown 2605084944 0 2420000192) 0x7f4b2008d500 con 0x7f4b2810f320 2024-04-14T15:15:28.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.498+0000 7f4b1f7fe700 10 client.36822.objecter ms_handle_connect 0x7f4b140758b0 2024-04-14T15:15:28.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.498+0000 7f4b30626700 10 client.36822.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:15:28.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.498+0000 7f4b30626700 15 client.36822.objecter _session_command_op_assign -1 1 2024-04-14T15:15:28.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.498+0000 7f4b30626700 20 client.36822.objecter _get_session s=0x7f4b0c005410 osd=0 3 2024-04-14T15:15:28.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.498+0000 7f4b30626700 20 client.36822.objecter put_session s=0x7f4b0c005410 osd=0 3 2024-04-14T15:15:28.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.498+0000 7f4b30626700 20 client.36822.objecter _get_session s=0x7f4b0c005410 osd=0 3 2024-04-14T15:15:28.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.498+0000 7f4b30626700 15 client.36822.objecter _session_command_op_remove -1 1 2024-04-14T15:15:28.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.498+0000 7f4b30626700 20 client.36822.objecter get_session s=0x7f4b0c005410 osd=0 3 2024-04-14T15:15:28.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.498+0000 7f4b30626700 15 client.36822.objecter _session_command_op_assign 0 1 2024-04-14T15:15:28.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.498+0000 7f4b30626700 20 client.36822.objecter put_session s=0x7f4b0c005410 osd=0 4 2024-04-14T15:15:28.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.498+0000 7f4b30626700 10 client.36822.objecter _send_command 1 2024-04-14T15:15:28.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.498+0000 7f4b30626700 1 -- 172.21.15.90:0/442819077 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4b0c004b80 con 0x7f4b0c004760 2024-04-14T15:15:28.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.498+0000 7f4b1f7fe700 10 client.36822.objecter ms_handle_connect 0x7f4b0c004760 2024-04-14T15:15:28.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.499+0000 7f4b1f7fe700 1 -- 172.21.15.90:0/442819077 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f4b0c004b80 con 0x7f4b0c004760 2024-04-14T15:15:28.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.499+0000 7f4b1f7fe700 10 client.36822.objecter ms_dispatch 0x7f4b28001550 command_reply(tid 1: 0 ) 2024-04-14T15:15:28.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.499+0000 7f4b1f7fe700 10 client.36822.objecter _finish_command 1 = system:0 2024-04-14T15:15:28.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.499+0000 7f4b1f7fe700 20 client.36822.objecter put_session s=0x7f4b0c005410 osd=0 4 2024-04-14T15:15:28.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.499+0000 7f4b1f7fe700 15 client.36822.objecter _session_command_op_remove 0 1 2024-04-14T15:15:28.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.519+0000 7f4b30626700 10 client.36822.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:15:28.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.519+0000 7f4b30626700 15 client.36822.objecter _session_command_op_assign -1 2 2024-04-14T15:15:28.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.519+0000 7f4b30626700 20 client.36822.objecter _get_session s=0x7f4b0c005410 osd=0 3 2024-04-14T15:15:28.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.519+0000 7f4b30626700 20 client.36822.objecter put_session s=0x7f4b0c005410 osd=0 3 2024-04-14T15:15:28.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.519+0000 7f4b30626700 20 client.36822.objecter _get_session s=0x7f4b0c005410 osd=0 3 2024-04-14T15:15:28.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.519+0000 7f4b30626700 15 client.36822.objecter _session_command_op_remove -1 2 2024-04-14T15:15:28.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.519+0000 7f4b30626700 20 client.36822.objecter get_session s=0x7f4b0c005410 osd=0 3 2024-04-14T15:15:28.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.519+0000 7f4b30626700 15 client.36822.objecter _session_command_op_assign 0 2 2024-04-14T15:15:28.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.519+0000 7f4b30626700 20 client.36822.objecter put_session s=0x7f4b0c005410 osd=0 4 2024-04-14T15:15:28.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.519+0000 7f4b30626700 10 client.36822.objecter _send_command 2 2024-04-14T15:15:28.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.519+0000 7f4b30626700 1 -- 172.21.15.90:0/442819077 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4b0c007b30 con 0x7f4b0c004760 2024-04-14T15:15:28.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.521+0000 7f4b1f7fe700 1 -- 172.21.15.90:0/442819077 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f4b0c007b30 con 0x7f4b0c004760 2024-04-14T15:15:28.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.521+0000 7f4b1f7fe700 10 client.36822.objecter ms_dispatch 0x7f4b28001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:15:28.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.521+0000 7f4b1f7fe700 10 client.36822.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:15:28.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.521+0000 7f4b1f7fe700 20 client.36822.objecter put_session s=0x7f4b0c005410 osd=0 4 2024-04-14T15:15:28.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.521+0000 7f4b1f7fe700 15 client.36822.objecter _session_command_op_remove 0 2 2024-04-14T15:15:28.522 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:15:28.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.522+0000 7f4b30626700 10 client.36822.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:15:28.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.522+0000 7f4b30626700 15 client.36822.objecter _session_command_op_assign -1 3 2024-04-14T15:15:28.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.522+0000 7f4b30626700 20 client.36822.objecter _get_session s=0x7f4b0c004fc0 osd=1 3 2024-04-14T15:15:28.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.522+0000 7f4b30626700 20 client.36822.objecter put_session s=0x7f4b0c004fc0 osd=1 3 2024-04-14T15:15:28.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.522+0000 7f4b30626700 20 client.36822.objecter _get_session s=0x7f4b0c004fc0 osd=1 3 2024-04-14T15:15:28.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.522+0000 7f4b30626700 15 client.36822.objecter _session_command_op_remove -1 3 2024-04-14T15:15:28.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.522+0000 7f4b30626700 20 client.36822.objecter get_session s=0x7f4b0c004fc0 osd=1 3 2024-04-14T15:15:28.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.522+0000 7f4b30626700 15 client.36822.objecter _session_command_op_assign 1 3 2024-04-14T15:15:28.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.522+0000 7f4b30626700 20 client.36822.objecter put_session s=0x7f4b0c004fc0 osd=1 4 2024-04-14T15:15:28.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.522+0000 7f4b30626700 10 client.36822.objecter _send_command 3 2024-04-14T15:15:28.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.522+0000 7f4b30626700 1 -- 172.21.15.90:0/442819077 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f4b0c002130 con 0x7f4b0c001bb0 2024-04-14T15:15:28.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.522+0000 7f4b1f7fe700 10 client.36822.objecter ms_handle_connect 0x7f4b0c001bb0 2024-04-14T15:15:28.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.523+0000 7f4b1f7fe700 1 -- 172.21.15.90:0/442819077 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f4b0c002130 con 0x7f4b0c001bb0 2024-04-14T15:15:28.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.523+0000 7f4b1f7fe700 10 client.36822.objecter ms_dispatch 0x7f4b28001550 command_reply(tid 3: 0 ) 2024-04-14T15:15:28.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.523+0000 7f4b1f7fe700 10 client.36822.objecter _finish_command 3 = system:0 2024-04-14T15:15:28.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.523+0000 7f4b1f7fe700 20 client.36822.objecter put_session s=0x7f4b0c004fc0 osd=1 4 2024-04-14T15:15:28.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.523+0000 7f4b1f7fe700 15 client.36822.objecter _session_command_op_remove 1 3 2024-04-14T15:15:28.539 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:15:28.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.548+0000 7f4b30626700 10 client.36822.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:15:28.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.548+0000 7f4b30626700 15 client.36822.objecter _session_command_op_assign -1 4 2024-04-14T15:15:28.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.548+0000 7f4b30626700 20 client.36822.objecter _get_session s=0x7f4b0c004fc0 osd=1 3 2024-04-14T15:15:28.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.548+0000 7f4b30626700 20 client.36822.objecter put_session s=0x7f4b0c004fc0 osd=1 3 2024-04-14T15:15:28.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.548+0000 7f4b30626700 20 client.36822.objecter _get_session s=0x7f4b0c004fc0 osd=1 3 2024-04-14T15:15:28.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.548+0000 7f4b30626700 15 client.36822.objecter _session_command_op_remove -1 4 2024-04-14T15:15:28.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.548+0000 7f4b30626700 20 client.36822.objecter get_session s=0x7f4b0c004fc0 osd=1 3 2024-04-14T15:15:28.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.548+0000 7f4b30626700 15 client.36822.objecter _session_command_op_assign 1 4 2024-04-14T15:15:28.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.548+0000 7f4b30626700 20 client.36822.objecter put_session s=0x7f4b0c004fc0 osd=1 4 2024-04-14T15:15:28.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.548+0000 7f4b30626700 10 client.36822.objecter _send_command 4 2024-04-14T15:15:28.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.548+0000 7f4b30626700 1 -- 172.21.15.90:0/442819077 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4b0c001630 con 0x7f4b0c001bb0 2024-04-14T15:15:28.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.550+0000 7f4b1f7fe700 1 -- 172.21.15.90:0/442819077 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f4b0c001630 con 0x7f4b0c001bb0 2024-04-14T15:15:28.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.550+0000 7f4b1f7fe700 10 client.36822.objecter ms_dispatch 0x7f4b28001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:15:28.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.550+0000 7f4b1f7fe700 10 client.36822.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:15:28.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.550+0000 7f4b1f7fe700 20 client.36822.objecter put_session s=0x7f4b0c004fc0 osd=1 4 2024-04-14T15:15:28.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.550+0000 7f4b1f7fe700 15 client.36822.objecter _session_command_op_remove 1 4 2024-04-14T15:15:28.551 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:15:28.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.550+0000 7f4b30626700 10 client.36822.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:15:28.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.550+0000 7f4b30626700 15 client.36822.objecter _session_command_op_assign -1 5 2024-04-14T15:15:28.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.551+0000 7f4b30626700 20 client.36822.objecter _get_session s=0x7f4b0c001a30 osd=2 3 2024-04-14T15:15:28.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.551+0000 7f4b30626700 20 client.36822.objecter put_session s=0x7f4b0c001a30 osd=2 3 2024-04-14T15:15:28.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.551+0000 7f4b30626700 20 client.36822.objecter _get_session s=0x7f4b0c001a30 osd=2 3 2024-04-14T15:15:28.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.551+0000 7f4b30626700 15 client.36822.objecter _session_command_op_remove -1 5 2024-04-14T15:15:28.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.551+0000 7f4b30626700 20 client.36822.objecter get_session s=0x7f4b0c001a30 osd=2 3 2024-04-14T15:15:28.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.551+0000 7f4b30626700 15 client.36822.objecter _session_command_op_assign 2 5 2024-04-14T15:15:28.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.551+0000 7f4b30626700 20 client.36822.objecter put_session s=0x7f4b0c001a30 osd=2 4 2024-04-14T15:15:28.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.551+0000 7f4b30626700 10 client.36822.objecter _send_command 5 2024-04-14T15:15:28.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.551+0000 7f4b30626700 1 -- 172.21.15.90:0/442819077 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f4b0c010fe0 con 0x7f4b0c0089c0 2024-04-14T15:15:28.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.551+0000 7f4b1f7fe700 10 client.36822.objecter ms_handle_connect 0x7f4b0c0089c0 2024-04-14T15:15:28.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.552+0000 7f4b1f7fe700 1 -- 172.21.15.90:0/442819077 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f4b0c010fe0 con 0x7f4b0c0089c0 2024-04-14T15:15:28.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.552+0000 7f4b1f7fe700 10 client.36822.objecter ms_dispatch 0x7f4b28001550 command_reply(tid 5: 0 ) 2024-04-14T15:15:28.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.552+0000 7f4b1f7fe700 10 client.36822.objecter _finish_command 5 = system:0 2024-04-14T15:15:28.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.552+0000 7f4b1f7fe700 20 client.36822.objecter put_session s=0x7f4b0c001a30 osd=2 4 2024-04-14T15:15:28.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.552+0000 7f4b1f7fe700 15 client.36822.objecter _session_command_op_remove 2 5 2024-04-14T15:15:28.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.571+0000 7f4b30626700 10 client.36822.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:15:28.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.571+0000 7f4b30626700 15 client.36822.objecter _session_command_op_assign -1 6 2024-04-14T15:15:28.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.571+0000 7f4b30626700 20 client.36822.objecter _get_session s=0x7f4b0c001a30 osd=2 3 2024-04-14T15:15:28.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.571+0000 7f4b30626700 20 client.36822.objecter put_session s=0x7f4b0c001a30 osd=2 3 2024-04-14T15:15:28.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.571+0000 7f4b30626700 20 client.36822.objecter _get_session s=0x7f4b0c001a30 osd=2 3 2024-04-14T15:15:28.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.571+0000 7f4b30626700 15 client.36822.objecter _session_command_op_remove -1 6 2024-04-14T15:15:28.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.571+0000 7f4b30626700 20 client.36822.objecter get_session s=0x7f4b0c001a30 osd=2 3 2024-04-14T15:15:28.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.571+0000 7f4b30626700 15 client.36822.objecter _session_command_op_assign 2 6 2024-04-14T15:15:28.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.571+0000 7f4b30626700 20 client.36822.objecter put_session s=0x7f4b0c001a30 osd=2 4 2024-04-14T15:15:28.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.571+0000 7f4b30626700 10 client.36822.objecter _send_command 6 2024-04-14T15:15:28.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.571+0000 7f4b30626700 1 -- 172.21.15.90:0/442819077 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4b0c007f30 con 0x7f4b0c0089c0 2024-04-14T15:15:28.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.573+0000 7f4b1f7fe700 1 -- 172.21.15.90:0/442819077 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f4b0c007f30 con 0x7f4b0c0089c0 2024-04-14T15:15:28.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.573+0000 7f4b1f7fe700 10 client.36822.objecter ms_dispatch 0x7f4b28001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:15:28.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.573+0000 7f4b1f7fe700 10 client.36822.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:15:28.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.573+0000 7f4b1f7fe700 20 client.36822.objecter put_session s=0x7f4b0c001a30 osd=2 4 2024-04-14T15:15:28.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.573+0000 7f4b1f7fe700 15 client.36822.objecter _session_command_op_remove 2 6 2024-04-14T15:15:28.574 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:15:28.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.573+0000 7f4b30626700 10 client.36822.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:15:28.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.573+0000 7f4b30626700 15 client.36822.objecter _session_command_op_assign -1 7 2024-04-14T15:15:28.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.573+0000 7f4b30626700 20 client.36822.objecter _get_session s=0x7f4b0c00cd60 osd=3 3 2024-04-14T15:15:28.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.573+0000 7f4b30626700 20 client.36822.objecter put_session s=0x7f4b0c00cd60 osd=3 3 2024-04-14T15:15:28.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.573+0000 7f4b30626700 20 client.36822.objecter _get_session s=0x7f4b0c00cd60 osd=3 3 2024-04-14T15:15:28.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.573+0000 7f4b30626700 15 client.36822.objecter _session_command_op_remove -1 7 2024-04-14T15:15:28.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.573+0000 7f4b30626700 20 client.36822.objecter get_session s=0x7f4b0c00cd60 osd=3 3 2024-04-14T15:15:28.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.573+0000 7f4b30626700 15 client.36822.objecter _session_command_op_assign 3 7 2024-04-14T15:15:28.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.573+0000 7f4b30626700 20 client.36822.objecter put_session s=0x7f4b0c00cd60 osd=3 4 2024-04-14T15:15:28.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.573+0000 7f4b30626700 10 client.36822.objecter _send_command 7 2024-04-14T15:15:28.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.573+0000 7f4b30626700 1 -- 172.21.15.90:0/442819077 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f4b0c00d550 con 0x7f4b0c011190 2024-04-14T15:15:28.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.574+0000 7f4b1f7fe700 10 client.36822.objecter ms_handle_connect 0x7f4b0c011190 2024-04-14T15:15:28.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.574+0000 7f4b1f7fe700 1 -- 172.21.15.90:0/442819077 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f4b200575f0 con 0x7f4b0c011190 2024-04-14T15:15:28.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.574+0000 7f4b1f7fe700 10 client.36822.objecter ms_dispatch 0x7f4b28001550 command_reply(tid 7: 0 ) 2024-04-14T15:15:28.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.574+0000 7f4b1f7fe700 10 client.36822.objecter _finish_command 7 = system:0 2024-04-14T15:15:28.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.574+0000 7f4b1f7fe700 20 client.36822.objecter put_session s=0x7f4b0c00cd60 osd=3 4 2024-04-14T15:15:28.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.574+0000 7f4b1f7fe700 15 client.36822.objecter _session_command_op_remove 3 7 2024-04-14T15:15:28.576 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:15:28.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.591+0000 7f4b30626700 10 client.36822.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:15:28.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.591+0000 7f4b30626700 15 client.36822.objecter _session_command_op_assign -1 8 2024-04-14T15:15:28.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.591+0000 7f4b30626700 20 client.36822.objecter _get_session s=0x7f4b0c00cd60 osd=3 3 2024-04-14T15:15:28.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.591+0000 7f4b30626700 20 client.36822.objecter put_session s=0x7f4b0c00cd60 osd=3 3 2024-04-14T15:15:28.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.591+0000 7f4b30626700 20 client.36822.objecter _get_session s=0x7f4b0c00cd60 osd=3 3 2024-04-14T15:15:28.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.591+0000 7f4b30626700 15 client.36822.objecter _session_command_op_remove -1 8 2024-04-14T15:15:28.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.591+0000 7f4b30626700 20 client.36822.objecter get_session s=0x7f4b0c00cd60 osd=3 3 2024-04-14T15:15:28.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.591+0000 7f4b30626700 15 client.36822.objecter _session_command_op_assign 3 8 2024-04-14T15:15:28.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.591+0000 7f4b30626700 20 client.36822.objecter put_session s=0x7f4b0c00cd60 osd=3 4 2024-04-14T15:15:28.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.591+0000 7f4b30626700 10 client.36822.objecter _send_command 8 2024-04-14T15:15:28.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.591+0000 7f4b30626700 1 -- 172.21.15.90:0/442819077 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4b0c012510 con 0x7f4b0c011190 2024-04-14T15:15:28.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.593+0000 7f4b1f7fe700 1 -- 172.21.15.90:0/442819077 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f4b0c012510 con 0x7f4b0c011190 2024-04-14T15:15:28.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.593+0000 7f4b1f7fe700 10 client.36822.objecter ms_dispatch 0x7f4b28001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:15:28.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.593+0000 7f4b1f7fe700 10 client.36822.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:15:28.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.593+0000 7f4b1f7fe700 20 client.36822.objecter put_session s=0x7f4b0c00cd60 osd=3 4 2024-04-14T15:15:28.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.593+0000 7f4b1f7fe700 15 client.36822.objecter _session_command_op_remove 3 8 2024-04-14T15:15:28.594 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:15:28.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.593+0000 7f4b30626700 10 client.36822.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:15:28.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.593+0000 7f4b30626700 15 client.36822.objecter _session_command_op_assign -1 9 2024-04-14T15:15:28.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.593+0000 7f4b30626700 20 client.36822.objecter _get_session s=0x7f4b0c013980 osd=4 3 2024-04-14T15:15:28.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.593+0000 7f4b30626700 20 client.36822.objecter put_session s=0x7f4b0c013980 osd=4 3 2024-04-14T15:15:28.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.593+0000 7f4b30626700 20 client.36822.objecter _get_session s=0x7f4b0c013980 osd=4 3 2024-04-14T15:15:28.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.593+0000 7f4b30626700 15 client.36822.objecter _session_command_op_remove -1 9 2024-04-14T15:15:28.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.593+0000 7f4b30626700 20 client.36822.objecter get_session s=0x7f4b0c013980 osd=4 3 2024-04-14T15:15:28.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.593+0000 7f4b30626700 15 client.36822.objecter _session_command_op_assign 4 9 2024-04-14T15:15:28.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.593+0000 7f4b30626700 20 client.36822.objecter put_session s=0x7f4b0c013980 osd=4 4 2024-04-14T15:15:28.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.593+0000 7f4b30626700 10 client.36822.objecter _send_command 9 2024-04-14T15:15:28.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.593+0000 7f4b30626700 1 -- 172.21.15.90:0/442819077 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f4b0c01a7e0 con 0x7f4b0c00c900 2024-04-14T15:15:28.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.594+0000 7f4b1f7fe700 10 client.36822.objecter ms_handle_connect 0x7f4b0c00c900 2024-04-14T15:15:28.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.595+0000 7f4b1f7fe700 1 -- 172.21.15.90:0/442819077 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f4b0c01a7e0 con 0x7f4b0c00c900 2024-04-14T15:15:28.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.595+0000 7f4b1f7fe700 10 client.36822.objecter ms_dispatch 0x7f4b28001550 command_reply(tid 9: 0 ) 2024-04-14T15:15:28.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.595+0000 7f4b1f7fe700 10 client.36822.objecter _finish_command 9 = system:0 2024-04-14T15:15:28.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.595+0000 7f4b1f7fe700 20 client.36822.objecter put_session s=0x7f4b0c013980 osd=4 4 2024-04-14T15:15:28.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.595+0000 7f4b1f7fe700 15 client.36822.objecter _session_command_op_remove 4 9 2024-04-14T15:15:28.601 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:28.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:28.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:15:28.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.611+0000 7f4b30626700 10 client.36822.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:15:28.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.611+0000 7f4b30626700 15 client.36822.objecter _session_command_op_assign -1 10 2024-04-14T15:15:28.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.611+0000 7f4b30626700 20 client.36822.objecter _get_session s=0x7f4b0c013980 osd=4 3 2024-04-14T15:15:28.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.611+0000 7f4b30626700 20 client.36822.objecter put_session s=0x7f4b0c013980 osd=4 3 2024-04-14T15:15:28.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.611+0000 7f4b30626700 20 client.36822.objecter _get_session s=0x7f4b0c013980 osd=4 3 2024-04-14T15:15:28.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.611+0000 7f4b30626700 15 client.36822.objecter _session_command_op_remove -1 10 2024-04-14T15:15:28.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.611+0000 7f4b30626700 20 client.36822.objecter get_session s=0x7f4b0c013980 osd=4 3 2024-04-14T15:15:28.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.611+0000 7f4b30626700 15 client.36822.objecter _session_command_op_assign 4 10 2024-04-14T15:15:28.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.611+0000 7f4b30626700 20 client.36822.objecter put_session s=0x7f4b0c013980 osd=4 4 2024-04-14T15:15:28.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.611+0000 7f4b30626700 10 client.36822.objecter _send_command 10 2024-04-14T15:15:28.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.611+0000 7f4b30626700 1 -- 172.21.15.90:0/442819077 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4b0c011f30 con 0x7f4b0c00c900 2024-04-14T15:15:28.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.613+0000 7f4b1f7fe700 1 -- 172.21.15.90:0/442819077 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f4b0c011f30 con 0x7f4b0c00c900 2024-04-14T15:15:28.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.613+0000 7f4b1f7fe700 10 client.36822.objecter ms_dispatch 0x7f4b28001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:15:28.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.613+0000 7f4b1f7fe700 10 client.36822.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:15:28.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.613+0000 7f4b1f7fe700 20 client.36822.objecter put_session s=0x7f4b0c013980 osd=4 4 2024-04-14T15:15:28.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.613+0000 7f4b1f7fe700 15 client.36822.objecter _session_command_op_remove 4 10 2024-04-14T15:15:28.614 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:15:28.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.613+0000 7f4b30626700 10 client.36822.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:15:28.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.613+0000 7f4b30626700 15 client.36822.objecter _session_command_op_assign -1 11 2024-04-14T15:15:28.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.613+0000 7f4b30626700 20 client.36822.objecter _get_session s=0x7f4b0c011ce0 osd=5 3 2024-04-14T15:15:28.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.613+0000 7f4b30626700 20 client.36822.objecter put_session s=0x7f4b0c011ce0 osd=5 3 2024-04-14T15:15:28.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.613+0000 7f4b30626700 20 client.36822.objecter _get_session s=0x7f4b0c011ce0 osd=5 3 2024-04-14T15:15:28.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.613+0000 7f4b30626700 15 client.36822.objecter _session_command_op_remove -1 11 2024-04-14T15:15:28.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.613+0000 7f4b30626700 20 client.36822.objecter get_session s=0x7f4b0c011ce0 osd=5 3 2024-04-14T15:15:28.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.613+0000 7f4b30626700 15 client.36822.objecter _session_command_op_assign 5 11 2024-04-14T15:15:28.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.613+0000 7f4b30626700 20 client.36822.objecter put_session s=0x7f4b0c011ce0 osd=5 4 2024-04-14T15:15:28.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.613+0000 7f4b30626700 10 client.36822.objecter _send_command 11 2024-04-14T15:15:28.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.613+0000 7f4b30626700 1 -- 172.21.15.90:0/442819077 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f4b0c01d080 con 0x7f4b0c0134a0 2024-04-14T15:15:28.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.614+0000 7f4b1f7fe700 10 client.36822.objecter ms_handle_connect 0x7f4b0c0134a0 2024-04-14T15:15:28.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.615+0000 7f4b1f7fe700 1 -- 172.21.15.90:0/442819077 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f4b0c01d080 con 0x7f4b0c0134a0 2024-04-14T15:15:28.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.615+0000 7f4b1f7fe700 10 client.36822.objecter ms_dispatch 0x7f4b28001550 command_reply(tid 11: 0 ) 2024-04-14T15:15:28.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.615+0000 7f4b1f7fe700 10 client.36822.objecter _finish_command 11 = system:0 2024-04-14T15:15:28.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.615+0000 7f4b1f7fe700 20 client.36822.objecter put_session s=0x7f4b0c011ce0 osd=5 4 2024-04-14T15:15:28.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.615+0000 7f4b1f7fe700 15 client.36822.objecter _session_command_op_remove 5 11 2024-04-14T15:15:28.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.631+0000 7f4b30626700 10 client.36822.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:15:28.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.631+0000 7f4b30626700 15 client.36822.objecter _session_command_op_assign -1 12 2024-04-14T15:15:28.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.631+0000 7f4b30626700 20 client.36822.objecter _get_session s=0x7f4b0c011ce0 osd=5 3 2024-04-14T15:15:28.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.631+0000 7f4b30626700 20 client.36822.objecter put_session s=0x7f4b0c011ce0 osd=5 3 2024-04-14T15:15:28.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.631+0000 7f4b30626700 20 client.36822.objecter _get_session s=0x7f4b0c011ce0 osd=5 3 2024-04-14T15:15:28.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.631+0000 7f4b30626700 15 client.36822.objecter _session_command_op_remove -1 12 2024-04-14T15:15:28.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.631+0000 7f4b30626700 20 client.36822.objecter get_session s=0x7f4b0c011ce0 osd=5 3 2024-04-14T15:15:28.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.631+0000 7f4b30626700 15 client.36822.objecter _session_command_op_assign 5 12 2024-04-14T15:15:28.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.631+0000 7f4b30626700 20 client.36822.objecter put_session s=0x7f4b0c011ce0 osd=5 4 2024-04-14T15:15:28.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.631+0000 7f4b30626700 10 client.36822.objecter _send_command 12 2024-04-14T15:15:28.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.631+0000 7f4b30626700 1 -- 172.21.15.90:0/442819077 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4b0c01d080 con 0x7f4b0c0134a0 2024-04-14T15:15:28.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.633+0000 7f4b1f7fe700 1 -- 172.21.15.90:0/442819077 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f4b0c01d080 con 0x7f4b0c0134a0 2024-04-14T15:15:28.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.633+0000 7f4b1f7fe700 10 client.36822.objecter ms_dispatch 0x7f4b28001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:15:28.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.633+0000 7f4b1f7fe700 10 client.36822.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:15:28.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.633+0000 7f4b1f7fe700 20 client.36822.objecter put_session s=0x7f4b0c011ce0 osd=5 4 2024-04-14T15:15:28.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.633+0000 7f4b1f7fe700 15 client.36822.objecter _session_command_op_remove 5 12 2024-04-14T15:15:28.634 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:15:28.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.633+0000 7f4b30626700 10 client.36822.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:15:28.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.633+0000 7f4b30626700 15 client.36822.objecter _session_command_op_assign -1 13 2024-04-14T15:15:28.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.633+0000 7f4b30626700 20 client.36822.objecter _get_session s=0x7f4b0c01d080 osd=6 3 2024-04-14T15:15:28.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.633+0000 7f4b30626700 20 client.36822.objecter put_session s=0x7f4b0c01d080 osd=6 3 2024-04-14T15:15:28.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.633+0000 7f4b30626700 20 client.36822.objecter _get_session s=0x7f4b0c01d080 osd=6 3 2024-04-14T15:15:28.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.633+0000 7f4b30626700 15 client.36822.objecter _session_command_op_remove -1 13 2024-04-14T15:15:28.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.633+0000 7f4b30626700 20 client.36822.objecter get_session s=0x7f4b0c01d080 osd=6 3 2024-04-14T15:15:28.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.633+0000 7f4b30626700 15 client.36822.objecter _session_command_op_assign 6 13 2024-04-14T15:15:28.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.633+0000 7f4b30626700 20 client.36822.objecter put_session s=0x7f4b0c01d080 osd=6 4 2024-04-14T15:15:28.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.633+0000 7f4b30626700 10 client.36822.objecter _send_command 13 2024-04-14T15:15:28.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.633+0000 7f4b30626700 1 -- 172.21.15.90:0/442819077 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f4b0c0121d0 con 0x7f4b0c01f820 2024-04-14T15:15:28.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.634+0000 7f4b1f7fe700 10 client.36822.objecter ms_handle_connect 0x7f4b0c01f820 2024-04-14T15:15:28.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.635+0000 7f4b1f7fe700 1 -- 172.21.15.90:0/442819077 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f4b2005b2a0 con 0x7f4b0c01f820 2024-04-14T15:15:28.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.635+0000 7f4b1f7fe700 10 client.36822.objecter ms_dispatch 0x7f4b28001550 command_reply(tid 13: 0 ) 2024-04-14T15:15:28.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.635+0000 7f4b1f7fe700 10 client.36822.objecter _finish_command 13 = system:0 2024-04-14T15:15:28.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.635+0000 7f4b1f7fe700 20 client.36822.objecter put_session s=0x7f4b0c01d080 osd=6 4 2024-04-14T15:15:28.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.635+0000 7f4b1f7fe700 15 client.36822.objecter _session_command_op_remove 6 13 2024-04-14T15:15:28.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.651+0000 7f4b30626700 10 client.36822.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:15:28.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.651+0000 7f4b30626700 15 client.36822.objecter _session_command_op_assign -1 14 2024-04-14T15:15:28.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.651+0000 7f4b30626700 20 client.36822.objecter _get_session s=0x7f4b0c01d080 osd=6 3 2024-04-14T15:15:28.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.651+0000 7f4b30626700 20 client.36822.objecter put_session s=0x7f4b0c01d080 osd=6 3 2024-04-14T15:15:28.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.651+0000 7f4b30626700 20 client.36822.objecter _get_session s=0x7f4b0c01d080 osd=6 3 2024-04-14T15:15:28.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.651+0000 7f4b30626700 15 client.36822.objecter _session_command_op_remove -1 14 2024-04-14T15:15:28.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.651+0000 7f4b30626700 20 client.36822.objecter get_session s=0x7f4b0c01d080 osd=6 3 2024-04-14T15:15:28.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.651+0000 7f4b30626700 15 client.36822.objecter _session_command_op_assign 6 14 2024-04-14T15:15:28.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.651+0000 7f4b30626700 20 client.36822.objecter put_session s=0x7f4b0c01d080 osd=6 4 2024-04-14T15:15:28.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.651+0000 7f4b30626700 10 client.36822.objecter _send_command 14 2024-04-14T15:15:28.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.651+0000 7f4b30626700 1 -- 172.21.15.90:0/442819077 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4b0c0129c0 con 0x7f4b0c01f820 2024-04-14T15:15:28.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.653+0000 7f4b1f7fe700 1 -- 172.21.15.90:0/442819077 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f4b0c0129c0 con 0x7f4b0c01f820 2024-04-14T15:15:28.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.653+0000 7f4b1f7fe700 10 client.36822.objecter ms_dispatch 0x7f4b28001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:15:28.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.653+0000 7f4b1f7fe700 10 client.36822.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:15:28.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.653+0000 7f4b1f7fe700 20 client.36822.objecter put_session s=0x7f4b0c01d080 osd=6 4 2024-04-14T15:15:28.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.653+0000 7f4b1f7fe700 15 client.36822.objecter _session_command_op_remove 6 14 2024-04-14T15:15:28.654 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:15:28.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.653+0000 7f4b30626700 10 client.36822.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:15:28.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.653+0000 7f4b30626700 15 client.36822.objecter _session_command_op_assign -1 15 2024-04-14T15:15:28.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.653+0000 7f4b30626700 20 client.36822.objecter _get_session s=0x7f4b0c012e20 osd=7 3 2024-04-14T15:15:28.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.653+0000 7f4b30626700 20 client.36822.objecter put_session s=0x7f4b0c012e20 osd=7 3 2024-04-14T15:15:28.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.653+0000 7f4b30626700 20 client.36822.objecter _get_session s=0x7f4b0c012e20 osd=7 3 2024-04-14T15:15:28.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.653+0000 7f4b30626700 15 client.36822.objecter _session_command_op_remove -1 15 2024-04-14T15:15:28.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.653+0000 7f4b30626700 20 client.36822.objecter get_session s=0x7f4b0c012e20 osd=7 3 2024-04-14T15:15:28.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.653+0000 7f4b30626700 15 client.36822.objecter _session_command_op_assign 7 15 2024-04-14T15:15:28.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.653+0000 7f4b30626700 20 client.36822.objecter put_session s=0x7f4b0c012e20 osd=7 4 2024-04-14T15:15:28.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.653+0000 7f4b30626700 10 client.36822.objecter _send_command 15 2024-04-14T15:15:28.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.653+0000 7f4b30626700 1 -- 172.21.15.90:0/442819077 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f4b0c01b4c0 con 0x7f4b0c012770 2024-04-14T15:15:28.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.654+0000 7f4b1f7fe700 10 client.36822.objecter ms_handle_connect 0x7f4b0c012770 2024-04-14T15:15:28.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.655+0000 7f4b1f7fe700 1 -- 172.21.15.90:0/442819077 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f4b0c01b4c0 con 0x7f4b0c012770 2024-04-14T15:15:28.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.655+0000 7f4b1f7fe700 10 client.36822.objecter ms_dispatch 0x7f4b28001550 command_reply(tid 15: 0 ) 2024-04-14T15:15:28.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.655+0000 7f4b1f7fe700 10 client.36822.objecter _finish_command 15 = system:0 2024-04-14T15:15:28.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.655+0000 7f4b1f7fe700 20 client.36822.objecter put_session s=0x7f4b0c012e20 osd=7 4 2024-04-14T15:15:28.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.655+0000 7f4b1f7fe700 15 client.36822.objecter _session_command_op_remove 7 15 2024-04-14T15:15:28.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.671+0000 7f4b30626700 10 client.36822.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:15:28.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.671+0000 7f4b30626700 15 client.36822.objecter _session_command_op_assign -1 16 2024-04-14T15:15:28.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.671+0000 7f4b30626700 20 client.36822.objecter _get_session s=0x7f4b0c012e20 osd=7 3 2024-04-14T15:15:28.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.671+0000 7f4b30626700 20 client.36822.objecter put_session s=0x7f4b0c012e20 osd=7 3 2024-04-14T15:15:28.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.671+0000 7f4b30626700 20 client.36822.objecter _get_session s=0x7f4b0c012e20 osd=7 3 2024-04-14T15:15:28.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.671+0000 7f4b30626700 15 client.36822.objecter _session_command_op_remove -1 16 2024-04-14T15:15:28.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.671+0000 7f4b30626700 20 client.36822.objecter get_session s=0x7f4b0c012e20 osd=7 3 2024-04-14T15:15:28.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.671+0000 7f4b30626700 15 client.36822.objecter _session_command_op_assign 7 16 2024-04-14T15:15:28.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.671+0000 7f4b30626700 20 client.36822.objecter put_session s=0x7f4b0c012e20 osd=7 4 2024-04-14T15:15:28.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.671+0000 7f4b30626700 10 client.36822.objecter _send_command 16 2024-04-14T15:15:28.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.671+0000 7f4b30626700 1 -- 172.21.15.90:0/442819077 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4b0c01b4c0 con 0x7f4b0c012770 2024-04-14T15:15:28.673 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:15:28.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.673+0000 7f4b1f7fe700 1 -- 172.21.15.90:0/442819077 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f4b0c01b4c0 con 0x7f4b0c012770 2024-04-14T15:15:28.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.673+0000 7f4b1f7fe700 10 client.36822.objecter ms_dispatch 0x7f4b28001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:15:28.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.673+0000 7f4b1f7fe700 10 client.36822.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:15:28.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.673+0000 7f4b1f7fe700 20 client.36822.objecter put_session s=0x7f4b0c012e20 osd=7 4 2024-04-14T15:15:28.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.673+0000 7f4b1f7fe700 15 client.36822.objecter _session_command_op_remove 7 16 2024-04-14T15:15:28.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.674+0000 7f4b30626700 10 librados: watch_flush enter 2024-04-14T15:15:28.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.674+0000 7f4b30626700 10 librados: watch_flush exit 2024-04-14T15:15:28.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.674+0000 7f4b30626700 10 client.36822.objecter close_session for osd.0 2024-04-14T15:15:28.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.674+0000 7f4b30626700 1 -- 172.21.15.90:0/442819077 >> v1:172.21.15.90:6801/3761806885 conn(0x7f4b0c004760 legacy=0x7f4b0c002780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:28.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.674+0000 7f4b30626700 20 client.36822.objecter put_session s=0x7f4b0c005410 osd=0 1 2024-04-14T15:15:28.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.674+0000 7f4b30626700 10 client.36822.objecter close_session for osd.1 2024-04-14T15:15:28.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.674+0000 7f4b30626700 1 -- 172.21.15.90:0/442819077 >> v1:172.21.15.90:6805/4170361699 conn(0x7f4b0c001bb0 legacy=0x7f4b0c00ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:28.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.674+0000 7f4b30626700 20 client.36822.objecter put_session s=0x7f4b0c004fc0 osd=1 1 2024-04-14T15:15:28.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.674+0000 7f4b30626700 10 client.36822.objecter close_session for osd.2 2024-04-14T15:15:28.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.674+0000 7f4b30626700 1 -- 172.21.15.90:0/442819077 >> v1:172.21.15.90:6809/1686184251 conn(0x7f4b0c0089c0 legacy=0x7f4b0c00f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:28.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.674+0000 7f4b30626700 20 client.36822.objecter put_session s=0x7f4b0c001a30 osd=2 1 2024-04-14T15:15:28.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.674+0000 7f4b30626700 10 client.36822.objecter close_session for osd.3 2024-04-14T15:15:28.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.674+0000 7f4b30626700 1 -- 172.21.15.90:0/442819077 >> v1:172.21.15.90:6813/2270166774 conn(0x7f4b0c011190 legacy=0x7f4b0c0115d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:28.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.674+0000 7f4b30626700 20 client.36822.objecter put_session s=0x7f4b0c00cd60 osd=3 1 2024-04-14T15:15:28.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.674+0000 7f4b30626700 10 client.36822.objecter close_session for osd.4 2024-04-14T15:15:28.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.674+0000 7f4b30626700 1 -- 172.21.15.90:0/442819077 >> v1:172.21.15.92:6800/3887549733 conn(0x7f4b0c00c900 legacy=0x7f4b0c0190c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:28.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.674+0000 7f4b2ebc3700 1 -- 172.21.15.90:0/442819077 reap_dead start 2024-04-14T15:15:28.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.674+0000 7f4b30626700 20 client.36822.objecter put_session s=0x7f4b0c013980 osd=4 1 2024-04-14T15:15:28.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.674+0000 7f4b30626700 10 client.36822.objecter close_session for osd.5 2024-04-14T15:15:28.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.674+0000 7f4b30626700 1 -- 172.21.15.90:0/442819077 >> v1:172.21.15.92:6804/3244561440 conn(0x7f4b0c0134a0 legacy=0x7f4b0c01b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:28.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.674+0000 7f4b30626700 20 client.36822.objecter put_session s=0x7f4b0c011ce0 osd=5 1 2024-04-14T15:15:28.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.674+0000 7f4b30626700 10 client.36822.objecter close_session for osd.6 2024-04-14T15:15:28.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.674+0000 7f4b30626700 1 -- 172.21.15.90:0/442819077 >> v1:172.21.15.92:6808/636125963 conn(0x7f4b0c01f820 legacy=0x7f4b0c021c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:28.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.674+0000 7f4b30626700 20 client.36822.objecter put_session s=0x7f4b0c01d080 osd=6 1 2024-04-14T15:15:28.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.674+0000 7f4b30626700 10 client.36822.objecter close_session for osd.7 2024-04-14T15:15:28.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.674+0000 7f4b30626700 1 -- 172.21.15.90:0/442819077 >> v1:172.21.15.92:6812/1773706716 conn(0x7f4b0c012770 legacy=0x7f4b0c023330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:28.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.674+0000 7f4b30626700 20 client.36822.objecter put_session s=0x7f4b0c012e20 osd=7 1 2024-04-14T15:15:28.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.674+0000 7f4b30626700 20 client.36822.objecter shutdown clearing up homeless session... 2024-04-14T15:15:28.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.674+0000 7f4b30626700 10 client.36822.objecter successfully canceled tick 2024-04-14T15:15:28.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.674+0000 7f4b2ebc3700 1 -- 172.21.15.90:0/442819077 reap_dead start 2024-04-14T15:15:28.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.674+0000 7f4b2ebc3700 1 -- 172.21.15.90:0/442819077 reap_dead start 2024-04-14T15:15:28.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.674+0000 7f4b30626700 1 -- 172.21.15.90:0/442819077 >> v1:172.21.15.90:6800/420935427 conn(0x7f4b140758b0 legacy=0x7f4b14077d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:28.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.674+0000 7f4b2ebc3700 1 -- 172.21.15.90:0/442819077 reap_dead start 2024-04-14T15:15:28.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.674+0000 7f4b30626700 1 -- 172.21.15.90:0/442819077 >> v1:172.21.15.90:6789/0 conn(0x7f4b2810f320 legacy=0x7f4b281a6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:28.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.674+0000 7f4b2ebc3700 1 -- 172.21.15.90:0/442819077 reap_dead start 2024-04-14T15:15:28.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.674+0000 7f4b30626700 1 -- 172.21.15.90:0/442819077 shutdown_connections 2024-04-14T15:15:28.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.674+0000 7f4b30626700 1 -- 172.21.15.90:0/442819077 >> 172.21.15.90:0/442819077 conn(0x7f4b280fe3d0 msgr2=0x7f4b28112870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:15:28.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.674+0000 7f4b30626700 1 -- 172.21.15.90:0/442819077 shutdown_connections 2024-04-14T15:15:28.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.674+0000 7f4b30626700 1 -- 172.21.15.90:0/442819077 wait complete. 2024-04-14T15:15:28.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:28.674+0000 7f4b30626700 1 librados: shutdown 2024-04-14T15:15:28.678 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:15:28.678 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:15:28.678 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:15:28.678 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:15:28.678 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:15:28.678 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:15:28.679 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:15:28.679 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:15:28.703 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:15:28.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:28.728 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:28.728 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:15:28.829 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:15:28.855 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:28.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:28.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:15:28.957 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:15:28.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:28.986 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:28.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:15:29.087 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:15:29.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:29.111 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:29.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:15:29.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:28 smithi090 ceph-mon[30399]: pgmap v335: 249 pgs: 32 unknown, 217 active+clean; 586 KiB data, 547 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 3 op/s 2024-04-14T15:15:29.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:28 smithi090 ceph-mon[30399]: 110.1b scrub starts 2024-04-14T15:15:29.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:28 smithi090 ceph-mon[30399]: 110.1b scrub ok 2024-04-14T15:15:29.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:28 smithi090 ceph-mon[30399]: 110.5 scrub starts 2024-04-14T15:15:29.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:28 smithi090 ceph-mon[30399]: 110.5 scrub ok 2024-04-14T15:15:29.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:28 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:29.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:28 smithi090 ceph-mon[30399]: 3.0 scrub starts 2024-04-14T15:15:29.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:28 smithi090 ceph-mon[30399]: 3.0 scrub ok 2024-04-14T15:15:29.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:28 smithi090 ceph-mon[30399]: 11.13 scrub starts 2024-04-14T15:15:29.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:28 smithi090 ceph-mon[30399]: 11.13 scrub ok 2024-04-14T15:15:29.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:28 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/442819077' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:15:29.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:28 smithi090 ceph-mon[30399]: from='client.48652 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi090-93342-37"}]': finished 2024-04-14T15:15:29.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:28 smithi090 ceph-mon[30399]: osdmap e261: 8 total, 8 up, 8 in 2024-04-14T15:15:29.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:28 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:15:29.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:28 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:15:29.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:28 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/327060375' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi090-93342-38"}]: dispatch 2024-04-14T15:15:29.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:28 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/327060375' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi090-93342-38"}]: dispatch 2024-04-14T15:15:29.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:28 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/327060375' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi090-93342-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:29.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:28 smithi090 ceph-mon[41670]: pgmap v335: 249 pgs: 32 unknown, 217 active+clean; 586 KiB data, 547 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 3 op/s 2024-04-14T15:15:29.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:28 smithi090 ceph-mon[41670]: 110.1b scrub starts 2024-04-14T15:15:29.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:28 smithi090 ceph-mon[41670]: 110.1b scrub ok 2024-04-14T15:15:29.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:28 smithi090 ceph-mon[41670]: 110.5 scrub starts 2024-04-14T15:15:29.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:28 smithi090 ceph-mon[41670]: 110.5 scrub ok 2024-04-14T15:15:29.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:28 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:29.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:28 smithi090 ceph-mon[41670]: 3.0 scrub starts 2024-04-14T15:15:29.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:28 smithi090 ceph-mon[41670]: 3.0 scrub ok 2024-04-14T15:15:29.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:28 smithi090 ceph-mon[41670]: 11.13 scrub starts 2024-04-14T15:15:29.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:28 smithi090 ceph-mon[41670]: 11.13 scrub ok 2024-04-14T15:15:29.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:28 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/442819077' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:15:29.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:28 smithi090 ceph-mon[41670]: from='client.48652 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi090-93342-37"}]': finished 2024-04-14T15:15:29.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:28 smithi090 ceph-mon[41670]: osdmap e261: 8 total, 8 up, 8 in 2024-04-14T15:15:29.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:28 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:15:29.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:28 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:15:29.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:28 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/327060375' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi090-93342-38"}]: dispatch 2024-04-14T15:15:29.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:28 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/327060375' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi090-93342-38"}]: dispatch 2024-04-14T15:15:29.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:28 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/327060375' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi090-93342-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:29.213 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:15:29.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:29.241 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:29.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:15:29.343 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:15:29.367 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:29.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:29.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:15:29.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:28 smithi092 ceph-mon[33966]: pgmap v335: 249 pgs: 32 unknown, 217 active+clean; 586 KiB data, 547 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 3 op/s 2024-04-14T15:15:29.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:28 smithi092 ceph-mon[33966]: 110.1b scrub starts 2024-04-14T15:15:29.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:28 smithi092 ceph-mon[33966]: 110.1b scrub ok 2024-04-14T15:15:29.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:28 smithi092 ceph-mon[33966]: 110.5 scrub starts 2024-04-14T15:15:29.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:28 smithi092 ceph-mon[33966]: 110.5 scrub ok 2024-04-14T15:15:29.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:28 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:29.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:28 smithi092 ceph-mon[33966]: 3.0 scrub starts 2024-04-14T15:15:29.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:28 smithi092 ceph-mon[33966]: 3.0 scrub ok 2024-04-14T15:15:29.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:28 smithi092 ceph-mon[33966]: 11.13 scrub starts 2024-04-14T15:15:29.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:28 smithi092 ceph-mon[33966]: 11.13 scrub ok 2024-04-14T15:15:29.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:28 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/442819077' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:15:29.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:28 smithi092 ceph-mon[33966]: from='client.48652 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi090-93342-37"}]': finished 2024-04-14T15:15:29.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:28 smithi092 ceph-mon[33966]: osdmap e261: 8 total, 8 up, 8 in 2024-04-14T15:15:29.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:28 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:15:29.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:28 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:15:29.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:28 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/327060375' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi090-93342-38"}]: dispatch 2024-04-14T15:15:29.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:28 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/327060375' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi090-93342-38"}]: dispatch 2024-04-14T15:15:29.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:28 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/327060375' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi090-93342-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:29.469 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:15:29.497 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:29.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:29.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:15:29.599 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:15:29.626 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:29.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:29.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:15:29.728 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:15:29.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:29.753 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:29.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:15:29.854 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:15:29.883 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:29.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:29.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:15:29.985 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:15:30.012 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:30.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:30.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:15:30.114 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:15:30.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:30.138 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:30.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:15:30.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:29 smithi090 ceph-mon[30399]: 11.11 scrub starts 2024-04-14T15:15:30.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:29 smithi090 ceph-mon[30399]: 11.11 scrub ok 2024-04-14T15:15:30.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:29 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:30.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:29 smithi090 ceph-mon[30399]: 11.0 scrub starts 2024-04-14T15:15:30.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:29 smithi090 ceph-mon[30399]: 11.0 scrub ok 2024-04-14T15:15:30.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:29 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-29", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:15:30.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:29 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/327060375' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi090-93342-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:15:30.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:29 smithi090 ceph-mon[30399]: osdmap e262: 8 total, 8 up, 8 in 2024-04-14T15:15:30.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:29 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/327060375' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi090-93342-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi090-93342-38"}]: dispatch 2024-04-14T15:15:30.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:29 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2597116422' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi090-93363-46"}]: dispatch 2024-04-14T15:15:30.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:29 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-29"}]: dispatch 2024-04-14T15:15:30.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:29 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-29"}]: dispatch 2024-04-14T15:15:30.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:29 smithi090 ceph-mon[30399]: from='client.46769 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi090-93363-46"}]: dispatch 2024-04-14T15:15:30.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:29 smithi090 ceph-mon[41670]: 11.11 scrub starts 2024-04-14T15:15:30.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:29 smithi090 ceph-mon[41670]: 11.11 scrub ok 2024-04-14T15:15:30.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:29 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:30.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:29 smithi090 ceph-mon[41670]: 11.0 scrub starts 2024-04-14T15:15:30.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:29 smithi090 ceph-mon[41670]: 11.0 scrub ok 2024-04-14T15:15:30.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:29 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-29", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:15:30.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:29 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/327060375' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi090-93342-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:15:30.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:29 smithi090 ceph-mon[41670]: osdmap e262: 8 total, 8 up, 8 in 2024-04-14T15:15:30.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:29 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/327060375' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi090-93342-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi090-93342-38"}]: dispatch 2024-04-14T15:15:30.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:29 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2597116422' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi090-93363-46"}]: dispatch 2024-04-14T15:15:30.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:29 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-29"}]: dispatch 2024-04-14T15:15:30.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:29 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-29"}]: dispatch 2024-04-14T15:15:30.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:29 smithi090 ceph-mon[41670]: from='client.46769 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi090-93363-46"}]: dispatch 2024-04-14T15:15:30.239 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:15:30.266 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:30.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:30.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:15:30.318 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-14T15:15:30.319 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:15:30.368 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:15:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:29 smithi092 ceph-mon[33966]: 11.11 scrub starts 2024-04-14T15:15:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:29 smithi092 ceph-mon[33966]: 11.11 scrub ok 2024-04-14T15:15:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:29 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:29 smithi092 ceph-mon[33966]: 11.0 scrub starts 2024-04-14T15:15:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:29 smithi092 ceph-mon[33966]: 11.0 scrub ok 2024-04-14T15:15:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:29 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-29", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:15:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:29 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/327060375' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi090-93342-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:15:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:29 smithi092 ceph-mon[33966]: osdmap e262: 8 total, 8 up, 8 in 2024-04-14T15:15:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:29 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/327060375' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi090-93342-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi090-93342-38"}]: dispatch 2024-04-14T15:15:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:29 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2597116422' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi090-93363-46"}]: dispatch 2024-04-14T15:15:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:29 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-29"}]: dispatch 2024-04-14T15:15:30.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:29 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-29"}]: dispatch 2024-04-14T15:15:30.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:29 smithi092 ceph-mon[33966]: from='client.46769 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi090-93363-46"}]: dispatch 2024-04-14T15:15:30.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:30.395 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:30.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:15:30.496 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:15:30.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:30.521 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:30.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:15:30.622 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:15:30.649 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:30.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:30.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:15:30.751 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:15:30.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:30.779 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:30.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:15:30.880 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:15:30.907 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:30.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:30.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:15:31.009 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:15:31.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:31.034 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:31.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:15:31.135 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:15:31.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:31.162 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:31.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:15:31.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:30 smithi090 ceph-mon[41670]: pgmap v338: 257 pgs: 2 creating+activating, 14 unknown, 241 active+clean; 4.6 MiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.0 MiB/s wr, 1 op/s 2024-04-14T15:15:31.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:30 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:15:31.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:30 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:15:31.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:30 smithi090 ceph-mon[41670]: 110.10 scrub starts 2024-04-14T15:15:31.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:30 smithi090 ceph-mon[41670]: 110.10 scrub ok 2024-04-14T15:15:31.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:30 smithi090 ceph-mon[41670]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:15:31.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:30 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:31.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:30 smithi090 ceph-mon[41670]: 110.7 scrub starts 2024-04-14T15:15:31.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:30 smithi090 ceph-mon[41670]: 110.7 scrub ok 2024-04-14T15:15:31.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:30 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-29"}]': finished 2024-04-14T15:15:31.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:30 smithi090 ceph-mon[41670]: from='client.46769 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi090-93363-46"}]': finished 2024-04-14T15:15:31.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:30 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2597116422' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi090-93363-46"}]: dispatch 2024-04-14T15:15:31.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:30 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-29", "mode": "writeback"}]: dispatch 2024-04-14T15:15:31.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:30 smithi090 ceph-mon[41670]: osdmap e263: 8 total, 8 up, 8 in 2024-04-14T15:15:31.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:30 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-29", "mode": "writeback"}]: dispatch 2024-04-14T15:15:31.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:30 smithi090 ceph-mon[41670]: from='client.46769 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi090-93363-46"}]: dispatch 2024-04-14T15:15:31.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:30 smithi090 ceph-mon[30399]: pgmap v338: 257 pgs: 2 creating+activating, 14 unknown, 241 active+clean; 4.6 MiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.0 MiB/s wr, 1 op/s 2024-04-14T15:15:31.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:30 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:15:31.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:30 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:15:31.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:30 smithi090 ceph-mon[30399]: 110.10 scrub starts 2024-04-14T15:15:31.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:30 smithi090 ceph-mon[30399]: 110.10 scrub ok 2024-04-14T15:15:31.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:30 smithi090 ceph-mon[30399]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:15:31.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:30 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:31.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:30 smithi090 ceph-mon[30399]: 110.7 scrub starts 2024-04-14T15:15:31.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:30 smithi090 ceph-mon[30399]: 110.7 scrub ok 2024-04-14T15:15:31.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:30 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-29"}]': finished 2024-04-14T15:15:31.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:30 smithi090 ceph-mon[30399]: from='client.46769 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi090-93363-46"}]': finished 2024-04-14T15:15:31.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:30 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2597116422' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi090-93363-46"}]: dispatch 2024-04-14T15:15:31.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:30 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-29", "mode": "writeback"}]: dispatch 2024-04-14T15:15:31.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:30 smithi090 ceph-mon[30399]: osdmap e263: 8 total, 8 up, 8 in 2024-04-14T15:15:31.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:30 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-29", "mode": "writeback"}]: dispatch 2024-04-14T15:15:31.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:30 smithi090 ceph-mon[30399]: from='client.46769 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi090-93363-46"}]: dispatch 2024-04-14T15:15:31.263 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:15:31.289 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:31.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:31.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:15:31.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:30 smithi092 ceph-mon[33966]: pgmap v338: 257 pgs: 2 creating+activating, 14 unknown, 241 active+clean; 4.6 MiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.0 MiB/s wr, 1 op/s 2024-04-14T15:15:31.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:30 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:15:31.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:30 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:15:31.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:30 smithi092 ceph-mon[33966]: 110.10 scrub starts 2024-04-14T15:15:31.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:30 smithi092 ceph-mon[33966]: 110.10 scrub ok 2024-04-14T15:15:31.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:30 smithi092 ceph-mon[33966]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:15:31.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:30 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:31.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:30 smithi092 ceph-mon[33966]: 110.7 scrub starts 2024-04-14T15:15:31.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:30 smithi092 ceph-mon[33966]: 110.7 scrub ok 2024-04-14T15:15:31.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:30 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-29"}]': finished 2024-04-14T15:15:31.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:30 smithi092 ceph-mon[33966]: from='client.46769 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi090-93363-46"}]': finished 2024-04-14T15:15:31.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:30 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2597116422' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi090-93363-46"}]: dispatch 2024-04-14T15:15:31.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:30 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-29", "mode": "writeback"}]: dispatch 2024-04-14T15:15:31.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:30 smithi092 ceph-mon[33966]: osdmap e263: 8 total, 8 up, 8 in 2024-04-14T15:15:31.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:30 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-29", "mode": "writeback"}]: dispatch 2024-04-14T15:15:31.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:30 smithi092 ceph-mon[33966]: from='client.46769 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi090-93363-46"}]: dispatch 2024-04-14T15:15:31.391 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:15:31.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:31.417 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:31.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:15:31.518 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:15:31.529 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-14T15:15:31.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:31.543 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:31.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:15:31.644 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:15:31.669 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:31.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:31.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:15:31.771 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:15:31.796 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:31.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:31.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:15:31.865 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-14T15:15:31.897 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:15:31.935 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:31.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:31.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:15:31.979 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:15:32.037 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:15:32.062 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:32.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:32.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:15:32.166 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:15:32.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:31 smithi090 ceph-mon[30399]: 5.b scrub starts 2024-04-14T15:15:32.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:31 smithi090 ceph-mon[30399]: 5.b scrub ok 2024-04-14T15:15:32.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:31 smithi090 ceph-mon[30399]: 4.14 scrub starts 2024-04-14T15:15:32.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:31 smithi090 ceph-mon[30399]: 4.14 scrub ok 2024-04-14T15:15:32.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:31 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:32.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:31 smithi090 ceph-mon[30399]: 11.c scrub starts 2024-04-14T15:15:32.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:31 smithi090 ceph-mon[30399]: 11.c scrub ok 2024-04-14T15:15:32.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:31 smithi090 ceph-mon[30399]: 13.2 scrub starts 2024-04-14T15:15:32.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:31 smithi090 ceph-mon[30399]: 13.2 scrub ok 2024-04-14T15:15:32.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:31 smithi090 ceph-mon[30399]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:15:32.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:31 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/327060375' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi090-93342-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi090-93342-38"}]': finished 2024-04-14T15:15:32.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:31 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-29", "mode": "writeback"}]': finished 2024-04-14T15:15:32.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:31 smithi090 ceph-mon[30399]: from='client.46769 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi090-93363-46"}]': finished 2024-04-14T15:15:32.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:31 smithi090 ceph-mon[30399]: osdmap e264: 8 total, 8 up, 8 in 2024-04-14T15:15:32.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:31 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3884299353' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi090-93363-47"}]: dispatch 2024-04-14T15:15:32.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:31 smithi090 ceph-mon[30399]: from='client.46787 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi090-93363-47"}]: dispatch 2024-04-14T15:15:32.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:31 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3884299353' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi090-93363-47"}]: dispatch 2024-04-14T15:15:32.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:31 smithi090 ceph-mon[30399]: from='client.46787 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi090-93363-47"}]: dispatch 2024-04-14T15:15:32.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:31 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3884299353' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi090-93363-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:32.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:31 smithi090 ceph-mon[30399]: from='client.46787 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi090-93363-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:32.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:31 smithi090 ceph-mon[41670]: 5.b scrub starts 2024-04-14T15:15:32.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:31 smithi090 ceph-mon[41670]: 5.b scrub ok 2024-04-14T15:15:32.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:31 smithi090 ceph-mon[41670]: 4.14 scrub starts 2024-04-14T15:15:32.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:31 smithi090 ceph-mon[41670]: 4.14 scrub ok 2024-04-14T15:15:32.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:31 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:32.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:31 smithi090 ceph-mon[41670]: 11.c scrub starts 2024-04-14T15:15:32.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:31 smithi090 ceph-mon[41670]: 11.c scrub ok 2024-04-14T15:15:32.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:31 smithi090 ceph-mon[41670]: 13.2 scrub starts 2024-04-14T15:15:32.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:31 smithi090 ceph-mon[41670]: 13.2 scrub ok 2024-04-14T15:15:32.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:31 smithi090 ceph-mon[41670]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:15:32.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:31 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/327060375' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi090-93342-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi090-93342-38"}]': finished 2024-04-14T15:15:32.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:31 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-29", "mode": "writeback"}]': finished 2024-04-14T15:15:32.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:31 smithi090 ceph-mon[41670]: from='client.46769 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi090-93363-46"}]': finished 2024-04-14T15:15:32.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:31 smithi090 ceph-mon[41670]: osdmap e264: 8 total, 8 up, 8 in 2024-04-14T15:15:32.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:31 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3884299353' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi090-93363-47"}]: dispatch 2024-04-14T15:15:32.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:31 smithi090 ceph-mon[41670]: from='client.46787 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi090-93363-47"}]: dispatch 2024-04-14T15:15:32.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:31 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3884299353' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi090-93363-47"}]: dispatch 2024-04-14T15:15:32.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:31 smithi090 ceph-mon[41670]: from='client.46787 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi090-93363-47"}]: dispatch 2024-04-14T15:15:32.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:31 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3884299353' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi090-93363-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:32.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:31 smithi090 ceph-mon[41670]: from='client.46787 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi090-93363-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:32.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:32.195 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:32.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:15:32.296 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:15:32.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:32.322 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:32.322 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:15:32.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:31 smithi092 ceph-mon[33966]: 5.b scrub starts 2024-04-14T15:15:32.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:31 smithi092 ceph-mon[33966]: 5.b scrub ok 2024-04-14T15:15:32.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:31 smithi092 ceph-mon[33966]: 4.14 scrub starts 2024-04-14T15:15:32.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:31 smithi092 ceph-mon[33966]: 4.14 scrub ok 2024-04-14T15:15:32.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:31 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:32.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:31 smithi092 ceph-mon[33966]: 11.c scrub starts 2024-04-14T15:15:32.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:31 smithi092 ceph-mon[33966]: 11.c scrub ok 2024-04-14T15:15:32.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:31 smithi092 ceph-mon[33966]: 13.2 scrub starts 2024-04-14T15:15:32.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:31 smithi092 ceph-mon[33966]: 13.2 scrub ok 2024-04-14T15:15:32.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:31 smithi092 ceph-mon[33966]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:15:32.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:31 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/327060375' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi090-93342-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi090-93342-38"}]': finished 2024-04-14T15:15:32.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:31 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-29", "mode": "writeback"}]': finished 2024-04-14T15:15:32.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:31 smithi092 ceph-mon[33966]: from='client.46769 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi090-93363-46"}]': finished 2024-04-14T15:15:32.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:31 smithi092 ceph-mon[33966]: osdmap e264: 8 total, 8 up, 8 in 2024-04-14T15:15:32.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:31 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3884299353' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi090-93363-47"}]: dispatch 2024-04-14T15:15:32.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:31 smithi092 ceph-mon[33966]: from='client.46787 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi090-93363-47"}]: dispatch 2024-04-14T15:15:32.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:31 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3884299353' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi090-93363-47"}]: dispatch 2024-04-14T15:15:32.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:31 smithi092 ceph-mon[33966]: from='client.46787 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi090-93363-47"}]: dispatch 2024-04-14T15:15:32.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:31 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3884299353' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi090-93363-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:32.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:31 smithi092 ceph-mon[33966]: from='client.46787 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi090-93363-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:32.422 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:15:32.447 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:32.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:32.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:15:32.549 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:15:32.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:32.578 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:32.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:15:32.679 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:15:32.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:32.708 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:32.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:15:32.809 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:15:32.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:32.837 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:32.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:15:32.938 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:15:32.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:32.964 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:32.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:15:33.065 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:15:33.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:33.089 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:33.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:15:33.134 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-14T15:15:33.134 INFO:teuthology.orchestra.run.smithi090.stdout: "success": "" 2024-04-14T15:15:33.134 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-14T15:15:33.135 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:32 smithi090 ceph-mon[41670]: pgmap v341: 249 pgs: 249 active+clean; 4.6 MiB data, 552 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-04-14T15:15:33.135 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:32 smithi090 ceph-mon[41670]: 11.12 scrub starts 2024-04-14T15:15:33.135 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:32 smithi090 ceph-mon[41670]: 11.12 scrub ok 2024-04-14T15:15:33.135 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:32 smithi090 ceph-mon[41670]: 11.9 scrub starts 2024-04-14T15:15:33.135 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:32 smithi090 ceph-mon[41670]: 11.9 scrub ok 2024-04-14T15:15:33.135 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:32 smithi090 ceph-mon[41670]: 6.1e scrub starts 2024-04-14T15:15:33.136 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:32 smithi090 ceph-mon[41670]: 6.1e scrub ok 2024-04-14T15:15:33.136 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:32 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:33.136 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:32 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:33.136 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:32 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi090-93444-1","var":"pg_num","val":"11"}]: dispatch 2024-04-14T15:15:33.136 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:32 smithi090 ceph-mon[41670]: from='client.24829 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi090-93444-1","var":"pg_num","val":"11"}]: dispatch 2024-04-14T15:15:33.136 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:32 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:15:33.136 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:32 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:15:33.136 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:32 smithi090 ceph-mon[41670]: 11.7 scrub starts 2024-04-14T15:15:33.136 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:32 smithi090 ceph-mon[41670]: 11.7 scrub ok 2024-04-14T15:15:33.136 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:32 smithi090 ceph-mon[41670]: from='client.46787 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi090-93363-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:15:33.136 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:32 smithi090 ceph-mon[41670]: from='client.24829 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi090-93444-1","var":"pg_num","val":"11"}]': finished 2024-04-14T15:15:33.136 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:32 smithi090 ceph-mon[41670]: osdmap e265: 8 total, 8 up, 8 in 2024-04-14T15:15:33.136 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:32 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3884299353' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi090-93363-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi090-93363-47"}]: dispatch 2024-04-14T15:15:33.136 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:32 smithi090 ceph-mon[41670]: from='client.46787 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi090-93363-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi090-93363-47"}]: dispatch 2024-04-14T15:15:33.136 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:32 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:33.137 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:32 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:33.137 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:32 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi090-93444-1","var":"pgp_num","val":"11"}]: dispatch 2024-04-14T15:15:33.137 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:32 smithi090 ceph-mon[41670]: from='client.24829 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi090-93444-1","var":"pgp_num","val":"11"}]: dispatch 2024-04-14T15:15:33.137 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:32 smithi090 ceph-mon[30399]: pgmap v341: 249 pgs: 249 active+clean; 4.6 MiB data, 552 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-04-14T15:15:33.137 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:32 smithi090 ceph-mon[30399]: 11.12 scrub starts 2024-04-14T15:15:33.137 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:32 smithi090 ceph-mon[30399]: 11.12 scrub ok 2024-04-14T15:15:33.137 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:32 smithi090 ceph-mon[30399]: 11.9 scrub starts 2024-04-14T15:15:33.137 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:32 smithi090 ceph-mon[30399]: 11.9 scrub ok 2024-04-14T15:15:33.137 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:32 smithi090 ceph-mon[30399]: 6.1e scrub starts 2024-04-14T15:15:33.137 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:32 smithi090 ceph-mon[30399]: 6.1e scrub ok 2024-04-14T15:15:33.137 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:32 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:33.137 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:32 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:33.138 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:32 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi090-93444-1","var":"pg_num","val":"11"}]: dispatch 2024-04-14T15:15:33.138 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:32 smithi090 ceph-mon[30399]: from='client.24829 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi090-93444-1","var":"pg_num","val":"11"}]: dispatch 2024-04-14T15:15:33.138 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:32 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:15:33.138 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:32 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:15:33.138 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:32 smithi090 ceph-mon[30399]: 11.7 scrub starts 2024-04-14T15:15:33.138 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:32 smithi090 ceph-mon[30399]: 11.7 scrub ok 2024-04-14T15:15:33.138 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:32 smithi090 ceph-mon[30399]: from='client.46787 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi090-93363-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:15:33.138 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:32 smithi090 ceph-mon[30399]: from='client.24829 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi090-93444-1","var":"pg_num","val":"11"}]': finished 2024-04-14T15:15:33.138 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:32 smithi090 ceph-mon[30399]: osdmap e265: 8 total, 8 up, 8 in 2024-04-14T15:15:33.138 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:32 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3884299353' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi090-93363-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi090-93363-47"}]: dispatch 2024-04-14T15:15:33.138 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:32 smithi090 ceph-mon[30399]: from='client.46787 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi090-93363-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi090-93363-47"}]: dispatch 2024-04-14T15:15:33.138 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:32 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:33.138 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:32 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:33.138 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:32 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi090-93444-1","var":"pgp_num","val":"11"}]: dispatch 2024-04-14T15:15:33.139 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:32 smithi090 ceph-mon[30399]: from='client.24829 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi090-93444-1","var":"pgp_num","val":"11"}]: dispatch 2024-04-14T15:15:33.190 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:15:33.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:33.217 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:33.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:15:33.317 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:15:33.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:33.342 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:33.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:15:33.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:32 smithi092 ceph-mon[33966]: pgmap v341: 249 pgs: 249 active+clean; 4.6 MiB data, 552 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-04-14T15:15:33.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:32 smithi092 ceph-mon[33966]: 11.12 scrub starts 2024-04-14T15:15:33.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:32 smithi092 ceph-mon[33966]: 11.12 scrub ok 2024-04-14T15:15:33.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:32 smithi092 ceph-mon[33966]: 11.9 scrub starts 2024-04-14T15:15:33.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:32 smithi092 ceph-mon[33966]: 11.9 scrub ok 2024-04-14T15:15:33.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:32 smithi092 ceph-mon[33966]: 6.1e scrub starts 2024-04-14T15:15:33.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:32 smithi092 ceph-mon[33966]: 6.1e scrub ok 2024-04-14T15:15:33.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:32 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:33.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:32 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:33.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:32 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi090-93444-1","var":"pg_num","val":"11"}]: dispatch 2024-04-14T15:15:33.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:32 smithi092 ceph-mon[33966]: from='client.24829 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi090-93444-1","var":"pg_num","val":"11"}]: dispatch 2024-04-14T15:15:33.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:32 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:15:33.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:32 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:15:33.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:32 smithi092 ceph-mon[33966]: 11.7 scrub starts 2024-04-14T15:15:33.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:32 smithi092 ceph-mon[33966]: 11.7 scrub ok 2024-04-14T15:15:33.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:32 smithi092 ceph-mon[33966]: from='client.46787 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi090-93363-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:15:33.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:32 smithi092 ceph-mon[33966]: from='client.24829 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi090-93444-1","var":"pg_num","val":"11"}]': finished 2024-04-14T15:15:33.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:32 smithi092 ceph-mon[33966]: osdmap e265: 8 total, 8 up, 8 in 2024-04-14T15:15:33.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:32 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3884299353' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi090-93363-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi090-93363-47"}]: dispatch 2024-04-14T15:15:33.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:32 smithi092 ceph-mon[33966]: from='client.46787 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi090-93363-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi090-93363-47"}]: dispatch 2024-04-14T15:15:33.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:32 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:33.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:32 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:33.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:32 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi090-93444-1","var":"pgp_num","val":"11"}]: dispatch 2024-04-14T15:15:33.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:32 smithi092 ceph-mon[33966]: from='client.24829 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi090-93444-1","var":"pgp_num","val":"11"}]: dispatch 2024-04-14T15:15:33.443 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:15:33.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:33.468 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:33.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:15:33.569 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:15:33.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:33.594 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:33.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:15:33.695 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:15:33.719 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:33.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:33.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:15:33.821 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:15:33.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:33.845 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:33.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:15:33.947 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:15:33.973 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:33.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:33.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:15:34.075 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:15:34.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:34.099 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:34.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:15:34.191 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:15:33 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:15:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:15:34.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:33 smithi090 ceph-mon[41670]: 11.4 scrub starts 2024-04-14T15:15:34.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:33 smithi090 ceph-mon[41670]: 11.4 scrub ok 2024-04-14T15:15:34.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:33 smithi090 ceph-mon[41670]: 11.b scrub starts 2024-04-14T15:15:34.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:33 smithi090 ceph-mon[41670]: 11.b scrub ok 2024-04-14T15:15:34.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:33 smithi090 ceph-mon[41670]: from='client.24829 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi090-93444-1","var":"pgp_num","val":"11"}]': finished 2024-04-14T15:15:34.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:33 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:34.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:33 smithi090 ceph-mon[41670]: osdmap e266: 8 total, 8 up, 8 in 2024-04-14T15:15:34.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:33 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/327060375' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi090-93342-38"}]: dispatch 2024-04-14T15:15:34.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:33 smithi090 ceph-mon[30399]: 11.4 scrub starts 2024-04-14T15:15:34.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:33 smithi090 ceph-mon[30399]: 11.4 scrub ok 2024-04-14T15:15:34.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:33 smithi090 ceph-mon[30399]: 11.b scrub starts 2024-04-14T15:15:34.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:33 smithi090 ceph-mon[30399]: 11.b scrub ok 2024-04-14T15:15:34.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:33 smithi090 ceph-mon[30399]: from='client.24829 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi090-93444-1","var":"pgp_num","val":"11"}]': finished 2024-04-14T15:15:34.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:33 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:34.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:33 smithi090 ceph-mon[30399]: osdmap e266: 8 total, 8 up, 8 in 2024-04-14T15:15:34.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:33 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/327060375' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi090-93342-38"}]: dispatch 2024-04-14T15:15:34.200 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:15:34.224 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:34.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:34.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:15:34.326 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:15:34.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:34.350 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:34.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:15:34.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:33 smithi092 ceph-mon[33966]: 11.4 scrub starts 2024-04-14T15:15:34.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:33 smithi092 ceph-mon[33966]: 11.4 scrub ok 2024-04-14T15:15:34.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:33 smithi092 ceph-mon[33966]: 11.b scrub starts 2024-04-14T15:15:34.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:33 smithi092 ceph-mon[33966]: 11.b scrub ok 2024-04-14T15:15:34.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:33 smithi092 ceph-mon[33966]: from='client.24829 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi090-93444-1","var":"pgp_num","val":"11"}]': finished 2024-04-14T15:15:34.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:33 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/218717313' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-14T15:15:34.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:33 smithi092 ceph-mon[33966]: osdmap e266: 8 total, 8 up, 8 in 2024-04-14T15:15:34.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:33 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/327060375' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi090-93342-38"}]: dispatch 2024-04-14T15:15:34.416 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:15:34.451 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:15:34.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:34.476 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:34.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:15:34.577 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:15:34.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:34.601 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:34.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:15:34.702 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:15:34.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:34.726 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:34.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:15:34.827 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:15:34.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:34.852 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:34.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:15:34.953 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:15:34.977 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:34.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:34.978 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:15:35.079 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:15:35.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:35.104 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:35.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:15:35.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:34 smithi090 ceph-mon[41670]: pgmap v344: 257 pgs: 8 unknown, 249 active+clean; 4.6 MiB data, 552 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:15:35.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:34 smithi090 ceph-mon[41670]: 5.8 scrub starts 2024-04-14T15:15:35.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:34 smithi090 ceph-mon[41670]: 5.8 scrub ok 2024-04-14T15:15:35.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:34 smithi090 ceph-mon[41670]: 110.1a scrub starts 2024-04-14T15:15:35.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:34 smithi090 ceph-mon[41670]: 110.1a scrub ok 2024-04-14T15:15:35.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:34 smithi090 ceph-mon[41670]: 11.19 scrub starts 2024-04-14T15:15:35.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:34 smithi090 ceph-mon[41670]: 11.19 scrub ok 2024-04-14T15:15:35.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:34 smithi090 ceph-mon[41670]: from='client.46787 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi090-93363-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi090-93363-47"}]': finished 2024-04-14T15:15:35.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:34 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/327060375' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi090-93342-38"}]': finished 2024-04-14T15:15:35.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:34 smithi090 ceph-mon[41670]: osdmap e267: 8 total, 8 up, 8 in 2024-04-14T15:15:35.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:34 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/327060375' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi090-93342-38"}]: dispatch 2024-04-14T15:15:35.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:34 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2015849731' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi090-93444-2"}]: dispatch 2024-04-14T15:15:35.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:34 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2015849731' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi090-93444-2"}]: dispatch 2024-04-14T15:15:35.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:34 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2015849731' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi090-93444-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:35.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:34 smithi090 ceph-mon[30399]: pgmap v344: 257 pgs: 8 unknown, 249 active+clean; 4.6 MiB data, 552 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:15:35.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:34 smithi090 ceph-mon[30399]: 5.8 scrub starts 2024-04-14T15:15:35.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:34 smithi090 ceph-mon[30399]: 5.8 scrub ok 2024-04-14T15:15:35.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:34 smithi090 ceph-mon[30399]: 110.1a scrub starts 2024-04-14T15:15:35.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:34 smithi090 ceph-mon[30399]: 110.1a scrub ok 2024-04-14T15:15:35.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:34 smithi090 ceph-mon[30399]: 11.19 scrub starts 2024-04-14T15:15:35.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:34 smithi090 ceph-mon[30399]: 11.19 scrub ok 2024-04-14T15:15:35.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:34 smithi090 ceph-mon[30399]: from='client.46787 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi090-93363-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi090-93363-47"}]': finished 2024-04-14T15:15:35.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:34 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/327060375' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi090-93342-38"}]': finished 2024-04-14T15:15:35.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:34 smithi090 ceph-mon[30399]: osdmap e267: 8 total, 8 up, 8 in 2024-04-14T15:15:35.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:34 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/327060375' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi090-93342-38"}]: dispatch 2024-04-14T15:15:35.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:34 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2015849731' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi090-93444-2"}]: dispatch 2024-04-14T15:15:35.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:34 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2015849731' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi090-93444-2"}]: dispatch 2024-04-14T15:15:35.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:34 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2015849731' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi090-93444-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:35.205 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:15:35.228 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:35.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:35.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:15:35.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.299+0000 7ff5f033f700 1 -- 172.21.15.90:0/3152624260 >> v1:172.21.15.90:6789/0 conn(0x7ff5e8108880 legacy=0x7ff5e8108c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:35.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.299+0000 7ff5f033f700 1 -- 172.21.15.90:0/3152624260 shutdown_connections 2024-04-14T15:15:35.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.299+0000 7ff5f033f700 1 -- 172.21.15.90:0/3152624260 >> 172.21.15.90:0/3152624260 conn(0x7ff5e80fe3d0 msgr2=0x7ff5e81007f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:15:35.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.299+0000 7ff5f033f700 1 -- 172.21.15.90:0/3152624260 shutdown_connections 2024-04-14T15:15:35.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.299+0000 7ff5f033f700 1 -- 172.21.15.90:0/3152624260 wait complete. 2024-04-14T15:15:35.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.300+0000 7ff5f033f700 1 librados: starting msgr at 2024-04-14T15:15:35.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.300+0000 7ff5f033f700 1 librados: starting objecter 2024-04-14T15:15:35.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.300+0000 7ff5f033f700 1 Processor -- start 2024-04-14T15:15:35.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.300+0000 7ff5f033f700 1 -- start start 2024-04-14T15:15:35.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.300+0000 7ff5f033f700 1 librados: setting wanted keys 2024-04-14T15:15:35.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.300+0000 7ff5f033f700 1 librados: calling monclient init 2024-04-14T15:15:35.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.300+0000 7ff5f033f700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff5e810ef60 con 0x7ff5e810b730 2024-04-14T15:15:35.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.300+0000 7ff5f033f700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff5e81a8f10 con 0x7ff5e810f300 2024-04-14T15:15:35.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.300+0000 7ff5f033f700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff5e81aa0f0 con 0x7ff5e8108880 2024-04-14T15:15:35.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.300+0000 7ff5ee0db700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7ff5e8108880 0x7ff5e810be00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:36906/0 (socket says 172.21.15.90:36906) 2024-04-14T15:15:35.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.300+0000 7ff5ed8da700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7ff5e810b730 0x7ff5e810c510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:41840/0 (socket says 172.21.15.90:41840) 2024-04-14T15:15:35.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.300+0000 7ff5ee0db700 1 -- 172.21.15.90:0/913796516 learned_addr learned my addr 172.21.15.90:0/913796516 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:15:35.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.300+0000 7ff5df7fe700 10 client.?.objecter ms_handle_connect 0x7ff5e810b730 2024-04-14T15:15:35.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.300+0000 7ff5df7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:15:35.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.300+0000 7ff5df7fe700 10 client.?.objecter ms_handle_connect 0x7ff5e8108880 2024-04-14T15:15:35.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.300+0000 7ff5df7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:15:35.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.300+0000 7ff5df7fe700 10 client.?.objecter ms_handle_connect 0x7ff5e810f300 2024-04-14T15:15:35.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.300+0000 7ff5df7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:15:35.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.300+0000 7ff5df7fe700 1 -- 172.21.15.90:0/913796516 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2690525141 0 0) 0x7ff5e810ef60 con 0x7ff5e810b730 2024-04-14T15:15:35.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.300+0000 7ff5df7fe700 1 -- 172.21.15.90:0/913796516 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff5d40035b0 con 0x7ff5e810b730 2024-04-14T15:15:35.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.300+0000 7ff5df7fe700 1 -- 172.21.15.90:0/913796516 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4100321938 0 0) 0x7ff5e81aa0f0 con 0x7ff5e8108880 2024-04-14T15:15:35.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.300+0000 7ff5df7fe700 1 -- 172.21.15.90:0/913796516 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff5e810ef60 con 0x7ff5e8108880 2024-04-14T15:15:35.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.300+0000 7ff5df7fe700 1 -- 172.21.15.90:0/913796516 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 19279729 0 0) 0x7ff5e81a8f10 con 0x7ff5e810f300 2024-04-14T15:15:35.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.300+0000 7ff5df7fe700 1 -- 172.21.15.90:0/913796516 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff5e81aa0f0 con 0x7ff5e810f300 2024-04-14T15:15:35.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.301+0000 7ff5df7fe700 1 -- 172.21.15.90:0/913796516 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2655570849 0 0) 0x7ff5d40035b0 con 0x7ff5e810b730 2024-04-14T15:15:35.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.301+0000 7ff5df7fe700 1 -- 172.21.15.90:0/913796516 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff5e81a8f10 con 0x7ff5e810b730 2024-04-14T15:15:35.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.301+0000 7ff5df7fe700 1 -- 172.21.15.90:0/913796516 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7ff5d80018d0 con 0x7ff5e810b730 2024-04-14T15:15:35.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.301+0000 7ff5df7fe700 1 -- 172.21.15.90:0/913796516 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 378039151 0 0) 0x7ff5e81aa0f0 con 0x7ff5e810f300 2024-04-14T15:15:35.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.301+0000 7ff5df7fe700 1 -- 172.21.15.90:0/913796516 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff5d40035b0 con 0x7ff5e810f300 2024-04-14T15:15:35.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.301+0000 7ff5df7fe700 1 -- 172.21.15.90:0/913796516 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3284305232 0 0) 0x7ff5e810ef60 con 0x7ff5e8108880 2024-04-14T15:15:35.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.301+0000 7ff5df7fe700 1 -- 172.21.15.90:0/913796516 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff5e81aa0f0 con 0x7ff5e8108880 2024-04-14T15:15:35.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.301+0000 7ff5df7fe700 1 -- 172.21.15.90:0/913796516 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7ff5e0001af0 con 0x7ff5e810f300 2024-04-14T15:15:35.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.301+0000 7ff5df7fe700 1 -- 172.21.15.90:0/913796516 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7ff5e4001430 con 0x7ff5e8108880 2024-04-14T15:15:35.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.301+0000 7ff5df7fe700 1 -- 172.21.15.90:0/913796516 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2235741959 0 0) 0x7ff5e81a8f10 con 0x7ff5e810b730 2024-04-14T15:15:35.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.301+0000 7ff5df7fe700 1 -- 172.21.15.90:0/913796516 >> v1:172.21.15.90:6790/0 conn(0x7ff5e8108880 legacy=0x7ff5e810be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:35.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.301+0000 7ff5df7fe700 1 -- 172.21.15.90:0/913796516 >> v1:172.21.15.92:6789/0 conn(0x7ff5e810f300 legacy=0x7ff5e81a67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:35.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.301+0000 7ff5df7fe700 1 -- 172.21.15.90:0/913796516 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5e81ab2d0 con 0x7ff5e810b730 2024-04-14T15:15:35.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.301+0000 7ff5f033f700 1 -- 172.21.15.90:0/913796516 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff5e81aa320 con 0x7ff5e810b730 2024-04-14T15:15:35.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.301+0000 7ff5f033f700 10 client.36846.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:15:35.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.301+0000 7ff5f033f700 1 -- 172.21.15.90:0/913796516 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff5e81aa7e0 con 0x7ff5e810b730 2024-04-14T15:15:35.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.301+0000 7ff5f033f700 1 librados: init done 2024-04-14T15:15:35.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.301+0000 7ff5df7fe700 1 -- 172.21.15.90:0/913796516 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7ff5d8003b40 con 0x7ff5e810b730 2024-04-14T15:15:35.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.301+0000 7ff5df7fe700 1 -- 172.21.15.90:0/913796516 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7ff5d8005ba0 con 0x7ff5e810b730 2024-04-14T15:15:35.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.302+0000 7ff5f033f700 1 -- 172.21.15.90:0/913796516 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5cc005120 con 0x7ff5e810b730 2024-04-14T15:15:35.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.302+0000 7ff5df7fe700 1 -- 172.21.15.90:0/913796516 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7ff5d80036a0 con 0x7ff5e810b730 2024-04-14T15:15:35.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.303+0000 7ff5df7fe700 1 -- 172.21.15.90:0/913796516 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(267..267 src has 51..267) ==== 9106+0+0 (unknown 750311948 0 0) 0x7ff5d808acc0 con 0x7ff5e810b730 2024-04-14T15:15:35.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.303+0000 7ff5df7fe700 10 client.36846.objecter ms_dispatch 0x7ff5e8001550 osd_map(267..267 src has 51..267) 2024-04-14T15:15:35.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.303+0000 7ff5df7fe700 3 client.36846.objecter handle_osd_map got epochs [267,267] > 0 2024-04-14T15:15:35.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.303+0000 7ff5df7fe700 3 client.36846.objecter handle_osd_map decoding full epoch 267 2024-04-14T15:15:35.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.303+0000 7ff5df7fe700 20 client.36846.objecter dump_active .. 0 homeless 2024-04-14T15:15:35.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.303+0000 7ff5df7fe700 10 client.36846.objecter ms_handle_connect 0x7ff5d40713a0 2024-04-14T15:15:35.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.308+0000 7ff5df7fe700 1 -- 172.21.15.90:0/913796516 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7ff5d8057320 con 0x7ff5e810b730 2024-04-14T15:15:35.323 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:15:35.330 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:15:35.355 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:35.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:35.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:15:35.372 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:15:34 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:15:35.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:34 smithi092 ceph-mon[33966]: pgmap v344: 257 pgs: 8 unknown, 249 active+clean; 4.6 MiB data, 552 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:15:35.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:34 smithi092 ceph-mon[33966]: 5.8 scrub starts 2024-04-14T15:15:35.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:34 smithi092 ceph-mon[33966]: 5.8 scrub ok 2024-04-14T15:15:35.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:34 smithi092 ceph-mon[33966]: 110.1a scrub starts 2024-04-14T15:15:35.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:34 smithi092 ceph-mon[33966]: 110.1a scrub ok 2024-04-14T15:15:35.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:34 smithi092 ceph-mon[33966]: 11.19 scrub starts 2024-04-14T15:15:35.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:34 smithi092 ceph-mon[33966]: 11.19 scrub ok 2024-04-14T15:15:35.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:34 smithi092 ceph-mon[33966]: from='client.46787 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi090-93363-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi090-93363-47"}]': finished 2024-04-14T15:15:35.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:34 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/327060375' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi090-93342-38"}]': finished 2024-04-14T15:15:35.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:34 smithi092 ceph-mon[33966]: osdmap e267: 8 total, 8 up, 8 in 2024-04-14T15:15:35.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:34 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/327060375' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi090-93342-38"}]: dispatch 2024-04-14T15:15:35.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:34 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2015849731' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi090-93444-2"}]: dispatch 2024-04-14T15:15:35.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:34 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2015849731' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi090-93444-2"}]: dispatch 2024-04-14T15:15:35.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:34 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2015849731' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi090-93444-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:35.457 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:15:35.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:35.482 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:35.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:15:35.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.526+0000 7ff5f033f700 1 -- 172.21.15.90:0/913796516 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7ff5cc005410 con 0x7ff5e810b730 2024-04-14T15:15:35.584 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:15:35.608 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:35.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:35.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:15:35.710 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:15:35.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:35.735 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:35.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:15:35.836 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:15:35.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.846+0000 7ff5df7fe700 1 -- 172.21.15.90:0/913796516 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v268) ==== 101+0+0 (unknown 1742731136 0 0) 0x7ff5d805afd0 con 0x7ff5e810b730 2024-04-14T15:15:35.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.850+0000 7ff5f033f700 10 librados: watch_flush enter 2024-04-14T15:15:35.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.852+0000 7ff5f033f700 10 librados: watch_flush exit 2024-04-14T15:15:35.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.852+0000 7ff5f033f700 20 client.36846.objecter shutdown clearing up homeless session... 2024-04-14T15:15:35.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.852+0000 7ff5f033f700 10 client.36846.objecter successfully canceled tick 2024-04-14T15:15:35.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.852+0000 7ff5f033f700 1 -- 172.21.15.90:0/913796516 >> v1:172.21.15.90:6800/420935427 conn(0x7ff5d40713a0 legacy=0x7ff5d4073860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:35.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.852+0000 7ff5f033f700 1 -- 172.21.15.90:0/913796516 >> v1:172.21.15.90:6789/0 conn(0x7ff5e810b730 legacy=0x7ff5e810c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:35.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.853+0000 7ff5f033f700 1 -- 172.21.15.90:0/913796516 shutdown_connections 2024-04-14T15:15:35.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.853+0000 7ff5f033f700 1 -- 172.21.15.90:0/913796516 >> 172.21.15.90:0/913796516 conn(0x7ff5e80fe3d0 msgr2=0x7ff5e8112820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:15:35.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.854+0000 7ff5f033f700 1 -- 172.21.15.90:0/913796516 shutdown_connections 2024-04-14T15:15:35.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.854+0000 7ff5f033f700 1 -- 172.21.15.90:0/913796516 wait complete. 2024-04-14T15:15:35.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:35.854+0000 7ff5f033f700 1 librados: shutdown 2024-04-14T15:15:35.859 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is unset 2024-04-14T15:15:35.866 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:35.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:35.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:15:35.968 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:15:35.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:35.993 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:35.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:15:36.094 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:15:36.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:36.119 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:36.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:15:36.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:35 smithi090 ceph-mon[30399]: 11.1d scrub starts 2024-04-14T15:15:36.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:35 smithi090 ceph-mon[30399]: 11.1d scrub ok 2024-04-14T15:15:36.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:35 smithi090 ceph-mon[30399]: 110.a scrub starts 2024-04-14T15:15:36.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:35 smithi090 ceph-mon[30399]: 110.a scrub ok 2024-04-14T15:15:36.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:35 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:15:36.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:35 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:15:36.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:35 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:15:36.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:35 smithi090 ceph-mon[30399]: 110.18 scrub starts 2024-04-14T15:15:36.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:35 smithi090 ceph-mon[30399]: 110.18 scrub ok 2024-04-14T15:15:36.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:35 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/913796516' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:15:36.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:35 smithi090 ceph-mon[30399]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-14T15:15:36.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:35 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/327060375' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi090-93342-38"}]': finished 2024-04-14T15:15:36.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:35 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2015849731' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi090-93444-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:15:36.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:35 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:15:36.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:35 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/913796516' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-14T15:15:36.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:35 smithi090 ceph-mon[30399]: osdmap e268: 8 total, 8 up, 8 in 2024-04-14T15:15:36.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:35 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-29"}]: dispatch 2024-04-14T15:15:36.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:35 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2015849731' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi090-93444-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi090-93444-2"}]: dispatch 2024-04-14T15:15:36.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:35 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-29"}]: dispatch 2024-04-14T15:15:36.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:35 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3211270109' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi090-93342-39"}]: dispatch 2024-04-14T15:15:36.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:35 smithi090 ceph-mon[30399]: from='client.48691 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi090-93342-39"}]: dispatch 2024-04-14T15:15:36.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:35 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3211270109' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi090-93342-39"}]: dispatch 2024-04-14T15:15:36.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:35 smithi090 ceph-mon[30399]: from='client.48691 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi090-93342-39"}]: dispatch 2024-04-14T15:15:36.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:35 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3211270109' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi090-93342-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:36.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:35 smithi090 ceph-mon[30399]: from='client.48691 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi090-93342-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:36.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:35 smithi090 ceph-mon[41670]: 11.1d scrub starts 2024-04-14T15:15:36.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:35 smithi090 ceph-mon[41670]: 11.1d scrub ok 2024-04-14T15:15:36.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:35 smithi090 ceph-mon[41670]: 110.a scrub starts 2024-04-14T15:15:36.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:35 smithi090 ceph-mon[41670]: 110.a scrub ok 2024-04-14T15:15:36.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:35 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:15:36.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:35 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:15:36.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:35 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:15:36.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:35 smithi090 ceph-mon[41670]: 110.18 scrub starts 2024-04-14T15:15:36.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:35 smithi090 ceph-mon[41670]: 110.18 scrub ok 2024-04-14T15:15:36.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:35 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/913796516' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:15:36.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:35 smithi090 ceph-mon[41670]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-14T15:15:36.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:35 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/327060375' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi090-93342-38"}]': finished 2024-04-14T15:15:36.195 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:35 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2015849731' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi090-93444-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:15:36.195 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:35 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:15:36.195 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:35 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/913796516' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-14T15:15:36.195 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:35 smithi090 ceph-mon[41670]: osdmap e268: 8 total, 8 up, 8 in 2024-04-14T15:15:36.195 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:35 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-29"}]: dispatch 2024-04-14T15:15:36.195 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:35 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2015849731' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi090-93444-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi090-93444-2"}]: dispatch 2024-04-14T15:15:36.195 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:35 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-29"}]: dispatch 2024-04-14T15:15:36.195 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:35 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3211270109' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi090-93342-39"}]: dispatch 2024-04-14T15:15:36.195 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:35 smithi090 ceph-mon[41670]: from='client.48691 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi090-93342-39"}]: dispatch 2024-04-14T15:15:36.195 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:35 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3211270109' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi090-93342-39"}]: dispatch 2024-04-14T15:15:36.195 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:35 smithi090 ceph-mon[41670]: from='client.48691 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi090-93342-39"}]: dispatch 2024-04-14T15:15:36.195 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:35 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3211270109' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi090-93342-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:36.195 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:35 smithi090 ceph-mon[41670]: from='client.48691 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi090-93342-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:36.220 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:15:36.244 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:36.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:36.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:15:36.346 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:15:36.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:36.370 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:36.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:15:36.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:35 smithi092 ceph-mon[33966]: 11.1d scrub starts 2024-04-14T15:15:36.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:35 smithi092 ceph-mon[33966]: 11.1d scrub ok 2024-04-14T15:15:36.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:35 smithi092 ceph-mon[33966]: 110.a scrub starts 2024-04-14T15:15:36.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:35 smithi092 ceph-mon[33966]: 110.a scrub ok 2024-04-14T15:15:36.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:35 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:15:36.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:35 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:15:36.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:35 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:15:36.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:35 smithi092 ceph-mon[33966]: 110.18 scrub starts 2024-04-14T15:15:36.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:35 smithi092 ceph-mon[33966]: 110.18 scrub ok 2024-04-14T15:15:36.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:35 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/913796516' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:15:36.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:35 smithi092 ceph-mon[33966]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-14T15:15:36.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:35 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/327060375' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi090-93342-38"}]': finished 2024-04-14T15:15:36.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:35 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2015849731' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi090-93444-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:15:36.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:35 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:15:36.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:35 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/913796516' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-14T15:15:36.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:35 smithi092 ceph-mon[33966]: osdmap e268: 8 total, 8 up, 8 in 2024-04-14T15:15:36.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:35 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-29"}]: dispatch 2024-04-14T15:15:36.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:35 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2015849731' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi090-93444-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi090-93444-2"}]: dispatch 2024-04-14T15:15:36.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:35 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-29"}]: dispatch 2024-04-14T15:15:36.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:35 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3211270109' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi090-93342-39"}]: dispatch 2024-04-14T15:15:36.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:35 smithi092 ceph-mon[33966]: from='client.48691 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi090-93342-39"}]: dispatch 2024-04-14T15:15:36.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:35 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3211270109' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi090-93342-39"}]: dispatch 2024-04-14T15:15:36.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:35 smithi092 ceph-mon[33966]: from='client.48691 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi090-93342-39"}]: dispatch 2024-04-14T15:15:36.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:35 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3211270109' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi090-93342-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:36.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:35 smithi092 ceph-mon[33966]: from='client.48691 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi090-93342-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:36.472 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:15:36.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:36.498 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:36.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:15:36.599 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:15:36.623 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:36.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:36.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:15:36.725 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:15:36.751 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:36.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:36.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:15:36.853 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:15:36.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:36.884 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:36.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:15:36.986 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:15:37.011 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:37.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:37.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:15:37.113 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:15:37.136 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:37.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:37.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:15:37.162 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:36 smithi090 ceph-mon[30399]: pgmap v347: 246 pgs: 8 unknown, 1 active+clean+snaptrim, 237 active+clean; 8.6 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 14 op/s 2024-04-14T15:15:37.162 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:36 smithi090 ceph-mon[30399]: 2.e scrub starts 2024-04-14T15:15:37.162 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:36 smithi090 ceph-mon[30399]: 2.e scrub ok 2024-04-14T15:15:37.162 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:36 smithi090 ceph-mon[30399]: 110.1 scrub starts 2024-04-14T15:15:37.162 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:36 smithi090 ceph-mon[30399]: 110.1 scrub ok 2024-04-14T15:15:37.162 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:36 smithi090 ceph-mon[30399]: 11.6 scrub starts 2024-04-14T15:15:37.162 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:36 smithi090 ceph-mon[30399]: 11.6 scrub ok 2024-04-14T15:15:37.162 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:36 smithi090 ceph-mon[30399]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:15:37.162 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:36 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-29"}]': finished 2024-04-14T15:15:37.163 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:36 smithi090 ceph-mon[30399]: from='client.48691 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi090-93342-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:15:37.163 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:36 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3211270109' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi090-93342-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi090-93342-39"}]: dispatch 2024-04-14T15:15:37.163 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:36 smithi090 ceph-mon[30399]: osdmap e269: 8 total, 8 up, 8 in 2024-04-14T15:15:37.163 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:36 smithi090 ceph-mon[30399]: from='client.48691 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi090-93342-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi090-93342-39"}]: dispatch 2024-04-14T15:15:37.163 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:36 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3884299353' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi090-93363-47"}]: dispatch 2024-04-14T15:15:37.163 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:36 smithi090 ceph-mon[30399]: from='client.46787 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi090-93363-47"}]: dispatch 2024-04-14T15:15:37.163 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:36 smithi090 ceph-mon[41670]: pgmap v347: 246 pgs: 8 unknown, 1 active+clean+snaptrim, 237 active+clean; 8.6 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 14 op/s 2024-04-14T15:15:37.163 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:36 smithi090 ceph-mon[41670]: 2.e scrub starts 2024-04-14T15:15:37.163 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:36 smithi090 ceph-mon[41670]: 2.e scrub ok 2024-04-14T15:15:37.163 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:36 smithi090 ceph-mon[41670]: 110.1 scrub starts 2024-04-14T15:15:37.163 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:36 smithi090 ceph-mon[41670]: 110.1 scrub ok 2024-04-14T15:15:37.164 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:36 smithi090 ceph-mon[41670]: 11.6 scrub starts 2024-04-14T15:15:37.164 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:36 smithi090 ceph-mon[41670]: 11.6 scrub ok 2024-04-14T15:15:37.164 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:36 smithi090 ceph-mon[41670]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:15:37.164 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:36 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-29"}]': finished 2024-04-14T15:15:37.164 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:36 smithi090 ceph-mon[41670]: from='client.48691 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi090-93342-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:15:37.164 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:36 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3211270109' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi090-93342-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi090-93342-39"}]: dispatch 2024-04-14T15:15:37.164 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:36 smithi090 ceph-mon[41670]: osdmap e269: 8 total, 8 up, 8 in 2024-04-14T15:15:37.164 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:36 smithi090 ceph-mon[41670]: from='client.48691 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi090-93342-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi090-93342-39"}]: dispatch 2024-04-14T15:15:37.164 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:36 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3884299353' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi090-93363-47"}]: dispatch 2024-04-14T15:15:37.164 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:36 smithi090 ceph-mon[41670]: from='client.46787 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi090-93363-47"}]: dispatch 2024-04-14T15:15:37.238 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:15:37.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:37.265 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:37.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:15:37.366 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:15:37.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:36 smithi092 ceph-mon[33966]: pgmap v347: 246 pgs: 8 unknown, 1 active+clean+snaptrim, 237 active+clean; 8.6 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 14 op/s 2024-04-14T15:15:37.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:36 smithi092 ceph-mon[33966]: 2.e scrub starts 2024-04-14T15:15:37.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:36 smithi092 ceph-mon[33966]: 2.e scrub ok 2024-04-14T15:15:37.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:36 smithi092 ceph-mon[33966]: 110.1 scrub starts 2024-04-14T15:15:37.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:36 smithi092 ceph-mon[33966]: 110.1 scrub ok 2024-04-14T15:15:37.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:36 smithi092 ceph-mon[33966]: 11.6 scrub starts 2024-04-14T15:15:37.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:36 smithi092 ceph-mon[33966]: 11.6 scrub ok 2024-04-14T15:15:37.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:36 smithi092 ceph-mon[33966]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:15:37.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:36 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-29"}]': finished 2024-04-14T15:15:37.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:36 smithi092 ceph-mon[33966]: from='client.48691 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi090-93342-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:15:37.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:36 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3211270109' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi090-93342-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi090-93342-39"}]: dispatch 2024-04-14T15:15:37.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:36 smithi092 ceph-mon[33966]: osdmap e269: 8 total, 8 up, 8 in 2024-04-14T15:15:37.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:36 smithi092 ceph-mon[33966]: from='client.48691 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi090-93342-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi090-93342-39"}]: dispatch 2024-04-14T15:15:37.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:36 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3884299353' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi090-93363-47"}]: dispatch 2024-04-14T15:15:37.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:36 smithi092 ceph-mon[33966]: from='client.46787 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi090-93363-47"}]: dispatch 2024-04-14T15:15:37.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:37.392 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:37.392 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:15:37.493 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:15:37.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:37.518 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:37.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:15:37.619 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:15:37.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.630+0000 7ff90e4d2700 1 -- 172.21.15.90:0/2486968577 >> v1:172.21.15.90:6789/0 conn(0x7ff908108880 legacy=0x7ff908108c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:37.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.630+0000 7ff90e4d2700 1 -- 172.21.15.90:0/2486968577 shutdown_connections 2024-04-14T15:15:37.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.630+0000 7ff90e4d2700 1 -- 172.21.15.90:0/2486968577 >> 172.21.15.90:0/2486968577 conn(0x7ff9080fe3d0 msgr2=0x7ff9081007f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:15:37.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.630+0000 7ff90e4d2700 1 -- 172.21.15.90:0/2486968577 shutdown_connections 2024-04-14T15:15:37.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.630+0000 7ff90e4d2700 1 -- 172.21.15.90:0/2486968577 wait complete. 2024-04-14T15:15:37.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.630+0000 7ff90e4d2700 1 librados: starting msgr at 2024-04-14T15:15:37.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.630+0000 7ff90e4d2700 1 librados: starting objecter 2024-04-14T15:15:37.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.630+0000 7ff90e4d2700 1 Processor -- start 2024-04-14T15:15:37.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.630+0000 7ff90e4d2700 1 -- start start 2024-04-14T15:15:37.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.630+0000 7ff90e4d2700 1 librados: setting wanted keys 2024-04-14T15:15:37.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.630+0000 7ff90e4d2700 1 librados: calling monclient init 2024-04-14T15:15:37.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.631+0000 7ff90e4d2700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff9081a78f0 con 0x7ff90810b730 2024-04-14T15:15:37.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.631+0000 7ff90e4d2700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff9081a8b10 con 0x7ff908108880 2024-04-14T15:15:37.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.631+0000 7ff90e4d2700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff9081a9d50 con 0x7ff90810f300 2024-04-14T15:15:37.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.631+0000 7ff9077fe700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7ff90810b730 0x7ff9081a27b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:41878/0 (socket says 172.21.15.90:41878) 2024-04-14T15:15:37.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.631+0000 7ff9077fe700 1 -- 172.21.15.90:0/2914038849 learned_addr learned my addr 172.21.15.90:0/2914038849 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:15:37.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.631+0000 7ff9057fa700 10 client.?.objecter ms_handle_connect 0x7ff90810b730 2024-04-14T15:15:37.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.631+0000 7ff9057fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:15:37.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.631+0000 7ff9057fa700 10 client.?.objecter ms_handle_connect 0x7ff90810f300 2024-04-14T15:15:37.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.631+0000 7ff9057fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:15:37.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.631+0000 7ff9057fa700 10 client.?.objecter ms_handle_connect 0x7ff908108880 2024-04-14T15:15:37.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.631+0000 7ff9057fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:15:37.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.631+0000 7ff9057fa700 1 -- 172.21.15.90:0/2914038849 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2017364125 0 0) 0x7ff9081a9d50 con 0x7ff90810f300 2024-04-14T15:15:37.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.631+0000 7ff9057fa700 1 -- 172.21.15.90:0/2914038849 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff8f00031f0 con 0x7ff90810f300 2024-04-14T15:15:37.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.631+0000 7ff9057fa700 1 -- 172.21.15.90:0/2914038849 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2497678514 0 0) 0x7ff9081a78f0 con 0x7ff90810b730 2024-04-14T15:15:37.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.631+0000 7ff9057fa700 1 -- 172.21.15.90:0/2914038849 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff9081a9d50 con 0x7ff90810b730 2024-04-14T15:15:37.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.631+0000 7ff9057fa700 1 -- 172.21.15.90:0/2914038849 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3702672710 0 0) 0x7ff9081a8b10 con 0x7ff908108880 2024-04-14T15:15:37.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.631+0000 7ff9057fa700 1 -- 172.21.15.90:0/2914038849 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff9081a78f0 con 0x7ff908108880 2024-04-14T15:15:37.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.631+0000 7ff9057fa700 1 -- 172.21.15.90:0/2914038849 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 542848120 0 0) 0x7ff8f00031f0 con 0x7ff90810f300 2024-04-14T15:15:37.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.631+0000 7ff9057fa700 1 -- 172.21.15.90:0/2914038849 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff9081a8b10 con 0x7ff90810f300 2024-04-14T15:15:37.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.631+0000 7ff9057fa700 1 -- 172.21.15.90:0/2914038849 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7ff900001b30 con 0x7ff90810f300 2024-04-14T15:15:37.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.631+0000 7ff9057fa700 1 -- 172.21.15.90:0/2914038849 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1953080955 0 0) 0x7ff9081a78f0 con 0x7ff908108880 2024-04-14T15:15:37.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.631+0000 7ff9057fa700 1 -- 172.21.15.90:0/2914038849 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff8f00031f0 con 0x7ff908108880 2024-04-14T15:15:37.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.631+0000 7ff9057fa700 1 -- 172.21.15.90:0/2914038849 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7ff8f80019e0 con 0x7ff908108880 2024-04-14T15:15:37.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.632+0000 7ff9057fa700 1 -- 172.21.15.90:0/2914038849 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 986771169 0 0) 0x7ff9081a9d50 con 0x7ff90810b730 2024-04-14T15:15:37.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.632+0000 7ff9057fa700 1 -- 172.21.15.90:0/2914038849 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff9081a78f0 con 0x7ff90810b730 2024-04-14T15:15:37.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.632+0000 7ff9057fa700 1 -- 172.21.15.90:0/2914038849 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7ff8fc001370 con 0x7ff90810b730 2024-04-14T15:15:37.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.632+0000 7ff9057fa700 1 -- 172.21.15.90:0/2914038849 <== mon.2 v1:172.21.15.90:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3009847644 0 0) 0x7ff9081a8b10 con 0x7ff90810f300 2024-04-14T15:15:37.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.632+0000 7ff9057fa700 1 -- 172.21.15.90:0/2914038849 >> v1:172.21.15.92:6789/0 conn(0x7ff908108880 legacy=0x7ff90810e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:37.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.632+0000 7ff9057fa700 1 -- 172.21.15.90:0/2914038849 >> v1:172.21.15.90:6789/0 conn(0x7ff90810b730 legacy=0x7ff9081a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:37.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.632+0000 7ff9057fa700 1 -- 172.21.15.90:0/2914038849 --> v1:172.21.15.90:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9081aaf90 con 0x7ff90810f300 2024-04-14T15:15:37.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.632+0000 7ff90e4d2700 1 -- 172.21.15.90:0/2914038849 --> v1:172.21.15.90:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff9081a9f80 con 0x7ff90810f300 2024-04-14T15:15:37.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.632+0000 7ff90e4d2700 10 client.46811.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:15:37.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.632+0000 7ff90e4d2700 1 -- 172.21.15.90:0/2914038849 --> v1:172.21.15.90:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7ff9081aa440 con 0x7ff90810f300 2024-04-14T15:15:37.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.632+0000 7ff90e4d2700 1 librados: init done 2024-04-14T15:15:37.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.632+0000 7ff9057fa700 1 -- 172.21.15.90:0/2914038849 <== mon.2 v1:172.21.15.90:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7ff900003ec0 con 0x7ff90810f300 2024-04-14T15:15:37.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.632+0000 7ff9057fa700 1 -- 172.21.15.90:0/2914038849 <== mon.2 v1:172.21.15.90:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7ff900004d20 con 0x7ff90810f300 2024-04-14T15:15:37.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.633+0000 7ff90e4d2700 1 -- 172.21.15.90:0/2914038849 --> v1:172.21.15.90:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7ff8f4005130 con 0x7ff90810f300 2024-04-14T15:15:37.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.633+0000 7ff9057fa700 1 -- 172.21.15.90:0/2914038849 <== mon.2 v1:172.21.15.90:6790/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7ff900003a20 con 0x7ff90810f300 2024-04-14T15:15:37.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.634+0000 7ff9057fa700 1 -- 172.21.15.90:0/2914038849 <== mon.2 v1:172.21.15.90:6790/0 8 ==== osd_map(269..269 src has 51..269) ==== 9285+0+0 (unknown 1483780693 0 0) 0x7ff900089e00 con 0x7ff90810f300 2024-04-14T15:15:37.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.634+0000 7ff9057fa700 10 client.46811.objecter ms_dispatch 0x7ff908001550 osd_map(269..269 src has 51..269) 2024-04-14T15:15:37.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.634+0000 7ff9057fa700 3 client.46811.objecter handle_osd_map got epochs [269,269] > 0 2024-04-14T15:15:37.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.634+0000 7ff9057fa700 3 client.46811.objecter handle_osd_map decoding full epoch 269 2024-04-14T15:15:37.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.634+0000 7ff9057fa700 20 client.46811.objecter dump_active .. 0 homeless 2024-04-14T15:15:37.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.634+0000 7ff9057fa700 1 -- 172.21.15.90:0/2914038849 <== mon.2 v1:172.21.15.90:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v269) ==== 54+0+16 (unknown 3334894321 0 2420000192) 0x7ff90008c1e0 con 0x7ff90810f300 2024-04-14T15:15:37.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.634+0000 7ff9057fa700 10 client.46811.objecter ms_handle_connect 0x7ff8f0071390 2024-04-14T15:15:37.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.634+0000 7ff90e4d2700 10 client.46811.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:15:37.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.634+0000 7ff90e4d2700 15 client.46811.objecter _session_command_op_assign -1 1 2024-04-14T15:15:37.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.634+0000 7ff90e4d2700 20 client.46811.objecter _get_session s=0x7ff8f4005410 osd=0 3 2024-04-14T15:15:37.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.634+0000 7ff90e4d2700 20 client.46811.objecter put_session s=0x7ff8f4005410 osd=0 3 2024-04-14T15:15:37.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.634+0000 7ff90e4d2700 20 client.46811.objecter _get_session s=0x7ff8f4005410 osd=0 3 2024-04-14T15:15:37.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.634+0000 7ff90e4d2700 15 client.46811.objecter _session_command_op_remove -1 1 2024-04-14T15:15:37.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.634+0000 7ff90e4d2700 20 client.46811.objecter get_session s=0x7ff8f4005410 osd=0 3 2024-04-14T15:15:37.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.634+0000 7ff90e4d2700 15 client.46811.objecter _session_command_op_assign 0 1 2024-04-14T15:15:37.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.634+0000 7ff90e4d2700 20 client.46811.objecter put_session s=0x7ff8f4005410 osd=0 4 2024-04-14T15:15:37.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.634+0000 7ff90e4d2700 10 client.46811.objecter _send_command 1 2024-04-14T15:15:37.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.634+0000 7ff90e4d2700 1 -- 172.21.15.90:0/2914038849 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff8f4004b80 con 0x7ff8f4004760 2024-04-14T15:15:37.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.635+0000 7ff9057fa700 10 client.46811.objecter ms_handle_connect 0x7ff8f4004760 2024-04-14T15:15:37.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.635+0000 7ff9057fa700 1 -- 172.21.15.90:0/2914038849 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7ff8f4004b80 con 0x7ff8f4004760 2024-04-14T15:15:37.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.635+0000 7ff9057fa700 10 client.46811.objecter ms_dispatch 0x7ff908001550 command_reply(tid 1: 0 ) 2024-04-14T15:15:37.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.635+0000 7ff9057fa700 10 client.46811.objecter _finish_command 1 = system:0 2024-04-14T15:15:37.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.635+0000 7ff9057fa700 20 client.46811.objecter put_session s=0x7ff8f4005410 osd=0 4 2024-04-14T15:15:37.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.635+0000 7ff9057fa700 15 client.46811.objecter _session_command_op_remove 0 1 2024-04-14T15:15:37.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:37.647 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:37.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:15:37.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.652+0000 7ff90e4d2700 10 client.46811.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:15:37.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.652+0000 7ff90e4d2700 15 client.46811.objecter _session_command_op_assign -1 2 2024-04-14T15:15:37.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.652+0000 7ff90e4d2700 20 client.46811.objecter _get_session s=0x7ff8f4005410 osd=0 3 2024-04-14T15:15:37.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.652+0000 7ff90e4d2700 20 client.46811.objecter put_session s=0x7ff8f4005410 osd=0 3 2024-04-14T15:15:37.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.652+0000 7ff90e4d2700 20 client.46811.objecter _get_session s=0x7ff8f4005410 osd=0 3 2024-04-14T15:15:37.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.652+0000 7ff90e4d2700 15 client.46811.objecter _session_command_op_remove -1 2 2024-04-14T15:15:37.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.652+0000 7ff90e4d2700 20 client.46811.objecter get_session s=0x7ff8f4005410 osd=0 3 2024-04-14T15:15:37.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.652+0000 7ff90e4d2700 15 client.46811.objecter _session_command_op_assign 0 2 2024-04-14T15:15:37.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.652+0000 7ff90e4d2700 20 client.46811.objecter put_session s=0x7ff8f4005410 osd=0 4 2024-04-14T15:15:37.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.652+0000 7ff90e4d2700 10 client.46811.objecter _send_command 2 2024-04-14T15:15:37.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.652+0000 7ff90e4d2700 1 -- 172.21.15.90:0/2914038849 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff8f4007b30 con 0x7ff8f4004760 2024-04-14T15:15:37.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.653+0000 7ff9057fa700 1 -- 172.21.15.90:0/2914038849 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff8f4007b30 con 0x7ff8f4004760 2024-04-14T15:15:37.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.653+0000 7ff9057fa700 10 client.46811.objecter ms_dispatch 0x7ff908001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:15:37.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.653+0000 7ff9057fa700 10 client.46811.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:15:37.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.653+0000 7ff9057fa700 20 client.46811.objecter put_session s=0x7ff8f4005410 osd=0 4 2024-04-14T15:15:37.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.653+0000 7ff9057fa700 15 client.46811.objecter _session_command_op_remove 0 2 2024-04-14T15:15:37.654 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:15:37.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.654+0000 7ff90e4d2700 10 client.46811.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:15:37.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.654+0000 7ff90e4d2700 15 client.46811.objecter _session_command_op_assign -1 3 2024-04-14T15:15:37.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.654+0000 7ff90e4d2700 20 client.46811.objecter _get_session s=0x7ff8f4004160 osd=1 3 2024-04-14T15:15:37.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.654+0000 7ff90e4d2700 20 client.46811.objecter put_session s=0x7ff8f4004160 osd=1 3 2024-04-14T15:15:37.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.654+0000 7ff90e4d2700 20 client.46811.objecter _get_session s=0x7ff8f4004160 osd=1 3 2024-04-14T15:15:37.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.654+0000 7ff90e4d2700 15 client.46811.objecter _session_command_op_remove -1 3 2024-04-14T15:15:37.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.654+0000 7ff90e4d2700 20 client.46811.objecter get_session s=0x7ff8f4004160 osd=1 3 2024-04-14T15:15:37.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.654+0000 7ff90e4d2700 15 client.46811.objecter _session_command_op_assign 1 3 2024-04-14T15:15:37.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.654+0000 7ff90e4d2700 20 client.46811.objecter put_session s=0x7ff8f4004160 osd=1 4 2024-04-14T15:15:37.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.654+0000 7ff90e4d2700 10 client.46811.objecter _send_command 3 2024-04-14T15:15:37.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.654+0000 7ff90e4d2700 1 -- 172.21.15.90:0/2914038849 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7ff8f4002190 con 0x7ff8f40042d0 2024-04-14T15:15:37.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.654+0000 7ff9057fa700 10 client.46811.objecter ms_handle_connect 0x7ff8f40042d0 2024-04-14T15:15:37.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.654+0000 7ff9057fa700 1 -- 172.21.15.90:0/2914038849 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7ff8f4002190 con 0x7ff8f40042d0 2024-04-14T15:15:37.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.654+0000 7ff9057fa700 10 client.46811.objecter ms_dispatch 0x7ff908001550 command_reply(tid 3: 0 ) 2024-04-14T15:15:37.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.654+0000 7ff9057fa700 10 client.46811.objecter _finish_command 3 = system:0 2024-04-14T15:15:37.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.654+0000 7ff9057fa700 20 client.46811.objecter put_session s=0x7ff8f4004160 osd=1 4 2024-04-14T15:15:37.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.654+0000 7ff9057fa700 15 client.46811.objecter _session_command_op_remove 1 3 2024-04-14T15:15:37.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.671+0000 7ff90e4d2700 10 client.46811.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:15:37.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.671+0000 7ff90e4d2700 15 client.46811.objecter _session_command_op_assign -1 4 2024-04-14T15:15:37.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.671+0000 7ff90e4d2700 20 client.46811.objecter _get_session s=0x7ff8f4004160 osd=1 3 2024-04-14T15:15:37.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.671+0000 7ff90e4d2700 20 client.46811.objecter put_session s=0x7ff8f4004160 osd=1 3 2024-04-14T15:15:37.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.671+0000 7ff90e4d2700 20 client.46811.objecter _get_session s=0x7ff8f4004160 osd=1 3 2024-04-14T15:15:37.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.671+0000 7ff90e4d2700 15 client.46811.objecter _session_command_op_remove -1 4 2024-04-14T15:15:37.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.671+0000 7ff90e4d2700 20 client.46811.objecter get_session s=0x7ff8f4004160 osd=1 3 2024-04-14T15:15:37.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.671+0000 7ff90e4d2700 15 client.46811.objecter _session_command_op_assign 1 4 2024-04-14T15:15:37.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.671+0000 7ff90e4d2700 20 client.46811.objecter put_session s=0x7ff8f4004160 osd=1 4 2024-04-14T15:15:37.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.671+0000 7ff90e4d2700 10 client.46811.objecter _send_command 4 2024-04-14T15:15:37.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.671+0000 7ff90e4d2700 1 -- 172.21.15.90:0/2914038849 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff8f4001630 con 0x7ff8f40042d0 2024-04-14T15:15:37.673 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:15:37.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.673+0000 7ff9057fa700 1 -- 172.21.15.90:0/2914038849 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff8f4001630 con 0x7ff8f40042d0 2024-04-14T15:15:37.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.673+0000 7ff9057fa700 10 client.46811.objecter ms_dispatch 0x7ff908001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:15:37.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.673+0000 7ff9057fa700 10 client.46811.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:15:37.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.673+0000 7ff9057fa700 20 client.46811.objecter put_session s=0x7ff8f4004160 osd=1 4 2024-04-14T15:15:37.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.673+0000 7ff9057fa700 15 client.46811.objecter _session_command_op_remove 1 4 2024-04-14T15:15:37.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.673+0000 7ff90e4d2700 10 client.46811.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:15:37.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.673+0000 7ff90e4d2700 15 client.46811.objecter _session_command_op_assign -1 5 2024-04-14T15:15:37.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.673+0000 7ff90e4d2700 20 client.46811.objecter _get_session s=0x7ff8f40084b0 osd=2 3 2024-04-14T15:15:37.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.673+0000 7ff90e4d2700 20 client.46811.objecter put_session s=0x7ff8f40084b0 osd=2 3 2024-04-14T15:15:37.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.673+0000 7ff90e4d2700 20 client.46811.objecter _get_session s=0x7ff8f40084b0 osd=2 3 2024-04-14T15:15:37.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.673+0000 7ff90e4d2700 15 client.46811.objecter _session_command_op_remove -1 5 2024-04-14T15:15:37.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.673+0000 7ff90e4d2700 20 client.46811.objecter get_session s=0x7ff8f40084b0 osd=2 3 2024-04-14T15:15:37.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.673+0000 7ff90e4d2700 15 client.46811.objecter _session_command_op_assign 2 5 2024-04-14T15:15:37.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.673+0000 7ff90e4d2700 20 client.46811.objecter put_session s=0x7ff8f40084b0 osd=2 4 2024-04-14T15:15:37.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.673+0000 7ff90e4d2700 10 client.46811.objecter _send_command 5 2024-04-14T15:15:37.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.673+0000 7ff90e4d2700 1 -- 172.21.15.90:0/2914038849 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7ff8f4015630 con 0x7ff8f4011b20 2024-04-14T15:15:37.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.673+0000 7ff9057fa700 10 client.46811.objecter ms_handle_connect 0x7ff8f4011b20 2024-04-14T15:15:37.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.674+0000 7ff9057fa700 1 -- 172.21.15.90:0/2914038849 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7ff8f4015630 con 0x7ff8f4011b20 2024-04-14T15:15:37.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.674+0000 7ff9057fa700 10 client.46811.objecter ms_dispatch 0x7ff908001550 command_reply(tid 5: 0 ) 2024-04-14T15:15:37.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.674+0000 7ff9057fa700 10 client.46811.objecter _finish_command 5 = system:0 2024-04-14T15:15:37.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.674+0000 7ff9057fa700 20 client.46811.objecter put_session s=0x7ff8f40084b0 osd=2 4 2024-04-14T15:15:37.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.674+0000 7ff9057fa700 15 client.46811.objecter _session_command_op_remove 2 5 2024-04-14T15:15:37.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.697+0000 7ff90e4d2700 10 client.46811.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:15:37.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.697+0000 7ff90e4d2700 15 client.46811.objecter _session_command_op_assign -1 6 2024-04-14T15:15:37.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.697+0000 7ff90e4d2700 20 client.46811.objecter _get_session s=0x7ff8f40084b0 osd=2 3 2024-04-14T15:15:37.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.697+0000 7ff90e4d2700 20 client.46811.objecter put_session s=0x7ff8f40084b0 osd=2 3 2024-04-14T15:15:37.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.697+0000 7ff90e4d2700 20 client.46811.objecter _get_session s=0x7ff8f40084b0 osd=2 3 2024-04-14T15:15:37.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.697+0000 7ff90e4d2700 15 client.46811.objecter _session_command_op_remove -1 6 2024-04-14T15:15:37.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.697+0000 7ff90e4d2700 20 client.46811.objecter get_session s=0x7ff8f40084b0 osd=2 3 2024-04-14T15:15:37.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.697+0000 7ff90e4d2700 15 client.46811.objecter _session_command_op_assign 2 6 2024-04-14T15:15:37.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.697+0000 7ff90e4d2700 20 client.46811.objecter put_session s=0x7ff8f40084b0 osd=2 4 2024-04-14T15:15:37.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.697+0000 7ff90e4d2700 10 client.46811.objecter _send_command 6 2024-04-14T15:15:37.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.697+0000 7ff90e4d2700 1 -- 172.21.15.90:0/2914038849 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff8f4007f30 con 0x7ff8f4011b20 2024-04-14T15:15:37.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.699+0000 7ff9057fa700 1 -- 172.21.15.90:0/2914038849 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff8f4007f30 con 0x7ff8f4011b20 2024-04-14T15:15:37.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.699+0000 7ff9057fa700 10 client.46811.objecter ms_dispatch 0x7ff908001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:15:37.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.699+0000 7ff9057fa700 10 client.46811.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:15:37.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.699+0000 7ff9057fa700 20 client.46811.objecter put_session s=0x7ff8f40084b0 osd=2 4 2024-04-14T15:15:37.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.699+0000 7ff9057fa700 15 client.46811.objecter _session_command_op_remove 2 6 2024-04-14T15:15:37.700 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:15:37.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.699+0000 7ff90e4d2700 10 client.46811.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:15:37.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.699+0000 7ff90e4d2700 15 client.46811.objecter _session_command_op_assign -1 7 2024-04-14T15:15:37.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.699+0000 7ff90e4d2700 20 client.46811.objecter _get_session s=0x7ff8f4008330 osd=3 3 2024-04-14T15:15:37.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.699+0000 7ff90e4d2700 20 client.46811.objecter put_session s=0x7ff8f4008330 osd=3 3 2024-04-14T15:15:37.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.699+0000 7ff90e4d2700 20 client.46811.objecter _get_session s=0x7ff8f4008330 osd=3 3 2024-04-14T15:15:37.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.699+0000 7ff90e4d2700 15 client.46811.objecter _session_command_op_remove -1 7 2024-04-14T15:15:37.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.699+0000 7ff90e4d2700 20 client.46811.objecter get_session s=0x7ff8f4008330 osd=3 3 2024-04-14T15:15:37.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.699+0000 7ff90e4d2700 15 client.46811.objecter _session_command_op_assign 3 7 2024-04-14T15:15:37.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.699+0000 7ff90e4d2700 20 client.46811.objecter put_session s=0x7ff8f4008330 osd=3 4 2024-04-14T15:15:37.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.699+0000 7ff90e4d2700 10 client.46811.objecter _send_command 7 2024-04-14T15:15:37.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.699+0000 7ff90e4d2700 1 -- 172.21.15.90:0/2914038849 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7ff8f401a1a0 con 0x7ff8f400d270 2024-04-14T15:15:37.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.700+0000 7ff9057fa700 10 client.46811.objecter ms_handle_connect 0x7ff8f400d270 2024-04-14T15:15:37.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.700+0000 7ff9057fa700 1 -- 172.21.15.90:0/2914038849 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7ff8f401a1a0 con 0x7ff8f400d270 2024-04-14T15:15:37.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.700+0000 7ff9057fa700 10 client.46811.objecter ms_dispatch 0x7ff908001550 command_reply(tid 7: 0 ) 2024-04-14T15:15:37.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.700+0000 7ff9057fa700 10 client.46811.objecter _finish_command 7 = system:0 2024-04-14T15:15:37.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.700+0000 7ff9057fa700 20 client.46811.objecter put_session s=0x7ff8f4008330 osd=3 4 2024-04-14T15:15:37.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.700+0000 7ff9057fa700 15 client.46811.objecter _session_command_op_remove 3 7 2024-04-14T15:15:37.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.717+0000 7ff90e4d2700 10 client.46811.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:15:37.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.717+0000 7ff90e4d2700 15 client.46811.objecter _session_command_op_assign -1 8 2024-04-14T15:15:37.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.717+0000 7ff90e4d2700 20 client.46811.objecter _get_session s=0x7ff8f4008330 osd=3 3 2024-04-14T15:15:37.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.717+0000 7ff90e4d2700 20 client.46811.objecter put_session s=0x7ff8f4008330 osd=3 3 2024-04-14T15:15:37.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.717+0000 7ff90e4d2700 20 client.46811.objecter _get_session s=0x7ff8f4008330 osd=3 3 2024-04-14T15:15:37.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.717+0000 7ff90e4d2700 15 client.46811.objecter _session_command_op_remove -1 8 2024-04-14T15:15:37.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.717+0000 7ff90e4d2700 20 client.46811.objecter get_session s=0x7ff8f4008330 osd=3 3 2024-04-14T15:15:37.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.717+0000 7ff90e4d2700 15 client.46811.objecter _session_command_op_assign 3 8 2024-04-14T15:15:37.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.717+0000 7ff90e4d2700 20 client.46811.objecter put_session s=0x7ff8f4008330 osd=3 4 2024-04-14T15:15:37.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.717+0000 7ff90e4d2700 10 client.46811.objecter _send_command 8 2024-04-14T15:15:37.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.717+0000 7ff90e4d2700 1 -- 172.21.15.90:0/2914038849 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff8f400c900 con 0x7ff8f400d270 2024-04-14T15:15:37.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.719+0000 7ff9057fa700 1 -- 172.21.15.90:0/2914038849 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff8f400c900 con 0x7ff8f400d270 2024-04-14T15:15:37.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.719+0000 7ff9057fa700 10 client.46811.objecter ms_dispatch 0x7ff908001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:15:37.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.719+0000 7ff9057fa700 10 client.46811.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:15:37.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.719+0000 7ff9057fa700 20 client.46811.objecter put_session s=0x7ff8f4008330 osd=3 4 2024-04-14T15:15:37.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.719+0000 7ff9057fa700 15 client.46811.objecter _session_command_op_remove 3 8 2024-04-14T15:15:37.720 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:15:37.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.719+0000 7ff90e4d2700 10 client.46811.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:15:37.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.719+0000 7ff90e4d2700 15 client.46811.objecter _session_command_op_assign -1 9 2024-04-14T15:15:37.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.719+0000 7ff90e4d2700 20 client.46811.objecter _get_session s=0x7ff8f400c900 osd=4 3 2024-04-14T15:15:37.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.719+0000 7ff90e4d2700 20 client.46811.objecter put_session s=0x7ff8f400c900 osd=4 3 2024-04-14T15:15:37.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.719+0000 7ff90e4d2700 20 client.46811.objecter _get_session s=0x7ff8f400c900 osd=4 3 2024-04-14T15:15:37.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.719+0000 7ff90e4d2700 15 client.46811.objecter _session_command_op_remove -1 9 2024-04-14T15:15:37.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.719+0000 7ff90e4d2700 20 client.46811.objecter get_session s=0x7ff8f400c900 osd=4 3 2024-04-14T15:15:37.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.719+0000 7ff90e4d2700 15 client.46811.objecter _session_command_op_assign 4 9 2024-04-14T15:15:37.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.719+0000 7ff90e4d2700 20 client.46811.objecter put_session s=0x7ff8f400c900 osd=4 4 2024-04-14T15:15:37.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.719+0000 7ff90e4d2700 10 client.46811.objecter _send_command 9 2024-04-14T15:15:37.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.719+0000 7ff90e4d2700 1 -- 172.21.15.90:0/2914038849 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7ff8f401ed50 con 0x7ff8f40164c0 2024-04-14T15:15:37.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.720+0000 7ff9057fa700 10 client.46811.objecter ms_handle_connect 0x7ff8f40164c0 2024-04-14T15:15:37.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.720+0000 7ff9057fa700 1 -- 172.21.15.90:0/2914038849 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7ff8f401ed50 con 0x7ff8f40164c0 2024-04-14T15:15:37.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.720+0000 7ff9057fa700 10 client.46811.objecter ms_dispatch 0x7ff908001550 command_reply(tid 9: 0 ) 2024-04-14T15:15:37.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.720+0000 7ff9057fa700 10 client.46811.objecter _finish_command 9 = system:0 2024-04-14T15:15:37.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.720+0000 7ff9057fa700 20 client.46811.objecter put_session s=0x7ff8f400c900 osd=4 4 2024-04-14T15:15:37.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.720+0000 7ff9057fa700 15 client.46811.objecter _session_command_op_remove 4 9 2024-04-14T15:15:37.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.738+0000 7ff90e4d2700 10 client.46811.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:15:37.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.738+0000 7ff90e4d2700 15 client.46811.objecter _session_command_op_assign -1 10 2024-04-14T15:15:37.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.738+0000 7ff90e4d2700 20 client.46811.objecter _get_session s=0x7ff8f400c900 osd=4 3 2024-04-14T15:15:37.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.738+0000 7ff90e4d2700 20 client.46811.objecter put_session s=0x7ff8f400c900 osd=4 3 2024-04-14T15:15:37.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.738+0000 7ff90e4d2700 20 client.46811.objecter _get_session s=0x7ff8f400c900 osd=4 3 2024-04-14T15:15:37.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.738+0000 7ff90e4d2700 15 client.46811.objecter _session_command_op_remove -1 10 2024-04-14T15:15:37.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.738+0000 7ff90e4d2700 20 client.46811.objecter get_session s=0x7ff8f400c900 osd=4 3 2024-04-14T15:15:37.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.738+0000 7ff90e4d2700 15 client.46811.objecter _session_command_op_assign 4 10 2024-04-14T15:15:37.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.738+0000 7ff90e4d2700 20 client.46811.objecter put_session s=0x7ff8f400c900 osd=4 4 2024-04-14T15:15:37.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.738+0000 7ff90e4d2700 10 client.46811.objecter _send_command 10 2024-04-14T15:15:37.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.738+0000 7ff90e4d2700 1 -- 172.21.15.90:0/2914038849 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff8f401ed50 con 0x7ff8f40164c0 2024-04-14T15:15:37.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.740+0000 7ff9057fa700 1 -- 172.21.15.90:0/2914038849 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff8f401ed50 con 0x7ff8f40164c0 2024-04-14T15:15:37.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.740+0000 7ff9057fa700 10 client.46811.objecter ms_dispatch 0x7ff908001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:15:37.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.740+0000 7ff9057fa700 10 client.46811.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:15:37.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.740+0000 7ff9057fa700 20 client.46811.objecter put_session s=0x7ff8f400c900 osd=4 4 2024-04-14T15:15:37.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.740+0000 7ff9057fa700 15 client.46811.objecter _session_command_op_remove 4 10 2024-04-14T15:15:37.742 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:15:37.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.740+0000 7ff90e4d2700 10 client.46811.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:15:37.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.740+0000 7ff90e4d2700 15 client.46811.objecter _session_command_op_assign -1 11 2024-04-14T15:15:37.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.740+0000 7ff90e4d2700 20 client.46811.objecter _get_session s=0x7ff8f401ed50 osd=5 3 2024-04-14T15:15:37.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.740+0000 7ff90e4d2700 20 client.46811.objecter put_session s=0x7ff8f401ed50 osd=5 3 2024-04-14T15:15:37.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.740+0000 7ff90e4d2700 20 client.46811.objecter _get_session s=0x7ff8f401ed50 osd=5 3 2024-04-14T15:15:37.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.740+0000 7ff90e4d2700 15 client.46811.objecter _session_command_op_remove -1 11 2024-04-14T15:15:37.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.740+0000 7ff90e4d2700 20 client.46811.objecter get_session s=0x7ff8f401ed50 osd=5 3 2024-04-14T15:15:37.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.740+0000 7ff90e4d2700 15 client.46811.objecter _session_command_op_assign 5 11 2024-04-14T15:15:37.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.740+0000 7ff90e4d2700 20 client.46811.objecter put_session s=0x7ff8f401ed50 osd=5 4 2024-04-14T15:15:37.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.740+0000 7ff90e4d2700 10 client.46811.objecter _send_command 11 2024-04-14T15:15:37.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.740+0000 7ff90e4d2700 1 -- 172.21.15.90:0/2914038849 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7ff8f4023120 con 0x7ff8f401f470 2024-04-14T15:15:37.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.740+0000 7ff9057fa700 10 client.46811.objecter ms_handle_connect 0x7ff8f401f470 2024-04-14T15:15:37.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.741+0000 7ff9057fa700 1 -- 172.21.15.90:0/2914038849 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7ff8f4023120 con 0x7ff8f401f470 2024-04-14T15:15:37.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.741+0000 7ff9057fa700 10 client.46811.objecter ms_dispatch 0x7ff908001550 command_reply(tid 11: 0 ) 2024-04-14T15:15:37.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.741+0000 7ff9057fa700 10 client.46811.objecter _finish_command 11 = system:0 2024-04-14T15:15:37.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.741+0000 7ff9057fa700 20 client.46811.objecter put_session s=0x7ff8f401ed50 osd=5 4 2024-04-14T15:15:37.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.741+0000 7ff9057fa700 15 client.46811.objecter _session_command_op_remove 5 11 2024-04-14T15:15:37.748 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:15:37.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.758+0000 7ff90e4d2700 10 client.46811.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:15:37.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.758+0000 7ff90e4d2700 15 client.46811.objecter _session_command_op_assign -1 12 2024-04-14T15:15:37.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.758+0000 7ff90e4d2700 20 client.46811.objecter _get_session s=0x7ff8f401ed50 osd=5 3 2024-04-14T15:15:37.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.758+0000 7ff90e4d2700 20 client.46811.objecter put_session s=0x7ff8f401ed50 osd=5 3 2024-04-14T15:15:37.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.758+0000 7ff90e4d2700 20 client.46811.objecter _get_session s=0x7ff8f401ed50 osd=5 3 2024-04-14T15:15:37.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.758+0000 7ff90e4d2700 15 client.46811.objecter _session_command_op_remove -1 12 2024-04-14T15:15:37.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.758+0000 7ff90e4d2700 20 client.46811.objecter get_session s=0x7ff8f401ed50 osd=5 3 2024-04-14T15:15:37.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.758+0000 7ff90e4d2700 15 client.46811.objecter _session_command_op_assign 5 12 2024-04-14T15:15:37.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.758+0000 7ff90e4d2700 20 client.46811.objecter put_session s=0x7ff8f401ed50 osd=5 4 2024-04-14T15:15:37.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.758+0000 7ff90e4d2700 10 client.46811.objecter _send_command 12 2024-04-14T15:15:37.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.758+0000 7ff90e4d2700 1 -- 172.21.15.90:0/2914038849 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff8f4024ad0 con 0x7ff8f401f470 2024-04-14T15:15:37.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.759+0000 7ff9057fa700 1 -- 172.21.15.90:0/2914038849 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff8f4024ad0 con 0x7ff8f401f470 2024-04-14T15:15:37.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.760+0000 7ff9057fa700 10 client.46811.objecter ms_dispatch 0x7ff908001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:15:37.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.760+0000 7ff9057fa700 10 client.46811.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:15:37.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.760+0000 7ff9057fa700 20 client.46811.objecter put_session s=0x7ff8f401ed50 osd=5 4 2024-04-14T15:15:37.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.760+0000 7ff9057fa700 15 client.46811.objecter _session_command_op_remove 5 12 2024-04-14T15:15:37.760 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:15:37.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.760+0000 7ff90e4d2700 10 client.46811.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:15:37.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.760+0000 7ff90e4d2700 15 client.46811.objecter _session_command_op_assign -1 13 2024-04-14T15:15:37.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.760+0000 7ff90e4d2700 20 client.46811.objecter _get_session s=0x7ff8f4024ad0 osd=6 3 2024-04-14T15:15:37.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.760+0000 7ff90e4d2700 20 client.46811.objecter put_session s=0x7ff8f4024ad0 osd=6 3 2024-04-14T15:15:37.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.760+0000 7ff90e4d2700 20 client.46811.objecter _get_session s=0x7ff8f4024ad0 osd=6 3 2024-04-14T15:15:37.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.760+0000 7ff90e4d2700 15 client.46811.objecter _session_command_op_remove -1 13 2024-04-14T15:15:37.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.760+0000 7ff90e4d2700 20 client.46811.objecter get_session s=0x7ff8f4024ad0 osd=6 3 2024-04-14T15:15:37.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.760+0000 7ff90e4d2700 15 client.46811.objecter _session_command_op_assign 6 13 2024-04-14T15:15:37.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.760+0000 7ff90e4d2700 20 client.46811.objecter put_session s=0x7ff8f4024ad0 osd=6 4 2024-04-14T15:15:37.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.760+0000 7ff90e4d2700 10 client.46811.objecter _send_command 13 2024-04-14T15:15:37.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.760+0000 7ff90e4d2700 1 -- 172.21.15.90:0/2914038849 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7ff8f4025670 con 0x7ff8f40251f0 2024-04-14T15:15:37.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.760+0000 7ff9057fa700 10 client.46811.objecter ms_handle_connect 0x7ff8f40251f0 2024-04-14T15:15:37.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.761+0000 7ff9057fa700 1 -- 172.21.15.90:0/2914038849 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7ff8f4025670 con 0x7ff8f40251f0 2024-04-14T15:15:37.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.761+0000 7ff9057fa700 10 client.46811.objecter ms_dispatch 0x7ff908001550 command_reply(tid 13: 0 ) 2024-04-14T15:15:37.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.761+0000 7ff9057fa700 10 client.46811.objecter _finish_command 13 = system:0 2024-04-14T15:15:37.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.761+0000 7ff9057fa700 20 client.46811.objecter put_session s=0x7ff8f4024ad0 osd=6 4 2024-04-14T15:15:37.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.761+0000 7ff9057fa700 15 client.46811.objecter _session_command_op_remove 6 13 2024-04-14T15:15:37.774 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:37.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:37.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:15:37.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.777+0000 7ff90e4d2700 10 client.46811.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:15:37.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.777+0000 7ff90e4d2700 15 client.46811.objecter _session_command_op_assign -1 14 2024-04-14T15:15:37.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.777+0000 7ff90e4d2700 20 client.46811.objecter _get_session s=0x7ff8f4024ad0 osd=6 3 2024-04-14T15:15:37.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.777+0000 7ff90e4d2700 20 client.46811.objecter put_session s=0x7ff8f4024ad0 osd=6 3 2024-04-14T15:15:37.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.777+0000 7ff90e4d2700 20 client.46811.objecter _get_session s=0x7ff8f4024ad0 osd=6 3 2024-04-14T15:15:37.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.777+0000 7ff90e4d2700 15 client.46811.objecter _session_command_op_remove -1 14 2024-04-14T15:15:37.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.777+0000 7ff90e4d2700 20 client.46811.objecter get_session s=0x7ff8f4024ad0 osd=6 3 2024-04-14T15:15:37.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.777+0000 7ff90e4d2700 15 client.46811.objecter _session_command_op_assign 6 14 2024-04-14T15:15:37.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.777+0000 7ff90e4d2700 20 client.46811.objecter put_session s=0x7ff8f4024ad0 osd=6 4 2024-04-14T15:15:37.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.777+0000 7ff90e4d2700 10 client.46811.objecter _send_command 14 2024-04-14T15:15:37.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.777+0000 7ff90e4d2700 1 -- 172.21.15.90:0/2914038849 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff8f4025670 con 0x7ff8f40251f0 2024-04-14T15:15:37.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.779+0000 7ff9057fa700 1 -- 172.21.15.90:0/2914038849 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff8f4025670 con 0x7ff8f40251f0 2024-04-14T15:15:37.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.779+0000 7ff9057fa700 10 client.46811.objecter ms_dispatch 0x7ff908001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:15:37.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.779+0000 7ff9057fa700 10 client.46811.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:15:37.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.779+0000 7ff9057fa700 20 client.46811.objecter put_session s=0x7ff8f4024ad0 osd=6 4 2024-04-14T15:15:37.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.779+0000 7ff9057fa700 15 client.46811.objecter _session_command_op_remove 6 14 2024-04-14T15:15:37.780 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:15:37.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.780+0000 7ff90e4d2700 10 client.46811.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:15:37.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.780+0000 7ff90e4d2700 15 client.46811.objecter _session_command_op_assign -1 15 2024-04-14T15:15:37.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.780+0000 7ff90e4d2700 20 client.46811.objecter _get_session s=0x7ff8f4025670 osd=7 3 2024-04-14T15:15:37.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.780+0000 7ff90e4d2700 20 client.46811.objecter put_session s=0x7ff8f4025670 osd=7 3 2024-04-14T15:15:37.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.780+0000 7ff90e4d2700 20 client.46811.objecter _get_session s=0x7ff8f4025670 osd=7 3 2024-04-14T15:15:37.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.780+0000 7ff90e4d2700 15 client.46811.objecter _session_command_op_remove -1 15 2024-04-14T15:15:37.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.780+0000 7ff90e4d2700 20 client.46811.objecter get_session s=0x7ff8f4025670 osd=7 3 2024-04-14T15:15:37.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.780+0000 7ff90e4d2700 15 client.46811.objecter _session_command_op_assign 7 15 2024-04-14T15:15:37.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.780+0000 7ff90e4d2700 20 client.46811.objecter put_session s=0x7ff8f4025670 osd=7 4 2024-04-14T15:15:37.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.780+0000 7ff90e4d2700 10 client.46811.objecter _send_command 15 2024-04-14T15:15:37.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.780+0000 7ff90e4d2700 1 -- 172.21.15.90:0/2914038849 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7ff8f402ba80 con 0x7ff8f40257e0 2024-04-14T15:15:37.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.780+0000 7ff9057fa700 10 client.46811.objecter ms_handle_connect 0x7ff8f40257e0 2024-04-14T15:15:37.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.781+0000 7ff9057fa700 1 -- 172.21.15.90:0/2914038849 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7ff8f402ba80 con 0x7ff8f40257e0 2024-04-14T15:15:37.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.781+0000 7ff9057fa700 10 client.46811.objecter ms_dispatch 0x7ff908001550 command_reply(tid 15: 0 ) 2024-04-14T15:15:37.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.781+0000 7ff9057fa700 10 client.46811.objecter _finish_command 15 = system:0 2024-04-14T15:15:37.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.781+0000 7ff9057fa700 20 client.46811.objecter put_session s=0x7ff8f4025670 osd=7 4 2024-04-14T15:15:37.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.781+0000 7ff9057fa700 15 client.46811.objecter _session_command_op_remove 7 15 2024-04-14T15:15:37.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.799+0000 7ff90e4d2700 10 client.46811.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:15:37.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.799+0000 7ff90e4d2700 15 client.46811.objecter _session_command_op_assign -1 16 2024-04-14T15:15:37.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.799+0000 7ff90e4d2700 20 client.46811.objecter _get_session s=0x7ff8f4025670 osd=7 3 2024-04-14T15:15:37.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.799+0000 7ff90e4d2700 20 client.46811.objecter put_session s=0x7ff8f4025670 osd=7 3 2024-04-14T15:15:37.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.799+0000 7ff90e4d2700 20 client.46811.objecter _get_session s=0x7ff8f4025670 osd=7 3 2024-04-14T15:15:37.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.799+0000 7ff90e4d2700 15 client.46811.objecter _session_command_op_remove -1 16 2024-04-14T15:15:37.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.799+0000 7ff90e4d2700 20 client.46811.objecter get_session s=0x7ff8f4025670 osd=7 3 2024-04-14T15:15:37.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.799+0000 7ff90e4d2700 15 client.46811.objecter _session_command_op_assign 7 16 2024-04-14T15:15:37.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.799+0000 7ff90e4d2700 20 client.46811.objecter put_session s=0x7ff8f4025670 osd=7 4 2024-04-14T15:15:37.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.799+0000 7ff90e4d2700 10 client.46811.objecter _send_command 16 2024-04-14T15:15:37.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.799+0000 7ff90e4d2700 1 -- 172.21.15.90:0/2914038849 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff8f402ba80 con 0x7ff8f40257e0 2024-04-14T15:15:37.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.801+0000 7ff9057fa700 1 -- 172.21.15.90:0/2914038849 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff8f402ba80 con 0x7ff8f40257e0 2024-04-14T15:15:37.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.801+0000 7ff9057fa700 10 client.46811.objecter ms_dispatch 0x7ff908001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:15:37.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.801+0000 7ff9057fa700 10 client.46811.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:15:37.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.801+0000 7ff9057fa700 20 client.46811.objecter put_session s=0x7ff8f4025670 osd=7 4 2024-04-14T15:15:37.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.801+0000 7ff9057fa700 15 client.46811.objecter _session_command_op_remove 7 16 2024-04-14T15:15:37.802 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:15:37.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.801+0000 7ff90e4d2700 10 librados: watch_flush enter 2024-04-14T15:15:37.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.801+0000 7ff90e4d2700 10 librados: watch_flush exit 2024-04-14T15:15:37.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.801+0000 7ff90e4d2700 10 client.46811.objecter close_session for osd.0 2024-04-14T15:15:37.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.801+0000 7ff90e4d2700 1 -- 172.21.15.90:0/2914038849 >> v1:172.21.15.90:6801/3761806885 conn(0x7ff8f4004760 legacy=0x7ff8f4002780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:37.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.801+0000 7ff90e4d2700 20 client.46811.objecter put_session s=0x7ff8f4005410 osd=0 1 2024-04-14T15:15:37.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.801+0000 7ff90e4d2700 10 client.46811.objecter close_session for osd.1 2024-04-14T15:15:37.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.801+0000 7ff90e4d2700 1 -- 172.21.15.90:0/2914038849 >> v1:172.21.15.90:6805/4170361699 conn(0x7ff8f40042d0 legacy=0x7ff8f400ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:37.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.801+0000 7ff90e4d2700 20 client.46811.objecter put_session s=0x7ff8f4004160 osd=1 1 2024-04-14T15:15:37.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.801+0000 7ff90e4d2700 10 client.46811.objecter close_session for osd.2 2024-04-14T15:15:37.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.801+0000 7ff90e4d2700 1 -- 172.21.15.90:0/2914038849 >> v1:172.21.15.90:6809/1686184251 conn(0x7ff8f4011b20 legacy=0x7ff8f4013f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:37.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.801+0000 7ff90ca6f700 1 -- 172.21.15.90:0/2914038849 reap_dead start 2024-04-14T15:15:37.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.801+0000 7ff90e4d2700 20 client.46811.objecter put_session s=0x7ff8f40084b0 osd=2 1 2024-04-14T15:15:37.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.801+0000 7ff90e4d2700 10 client.46811.objecter close_session for osd.3 2024-04-14T15:15:37.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.801+0000 7ff90e4d2700 1 -- 172.21.15.90:0/2914038849 >> v1:172.21.15.90:6813/2270166774 conn(0x7ff8f400d270 legacy=0x7ff8f4018970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:37.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.801+0000 7ff90e4d2700 20 client.46811.objecter put_session s=0x7ff8f4008330 osd=3 1 2024-04-14T15:15:37.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.801+0000 7ff90e4d2700 10 client.46811.objecter close_session for osd.4 2024-04-14T15:15:37.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.801+0000 7ff90e4d2700 1 -- 172.21.15.90:0/2914038849 >> v1:172.21.15.92:6800/3887549733 conn(0x7ff8f40164c0 legacy=0x7ff8f401a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:37.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.802+0000 7ff90e4d2700 20 client.46811.objecter put_session s=0x7ff8f400c900 osd=4 1 2024-04-14T15:15:37.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.802+0000 7ff90e4d2700 10 client.46811.objecter close_session for osd.5 2024-04-14T15:15:37.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.802+0000 7ff90e4d2700 1 -- 172.21.15.90:0/2914038849 >> v1:172.21.15.92:6804/3244561440 conn(0x7ff8f401f470 legacy=0x7ff8f40218a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:37.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.802+0000 7ff90e4d2700 20 client.46811.objecter put_session s=0x7ff8f401ed50 osd=5 1 2024-04-14T15:15:37.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.802+0000 7ff90e4d2700 10 client.46811.objecter close_session for osd.6 2024-04-14T15:15:37.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.802+0000 7ff90e4d2700 1 -- 172.21.15.90:0/2914038849 >> v1:172.21.15.92:6808/636125963 conn(0x7ff8f40251f0 legacy=0x7ff8f4023300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:37.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.802+0000 7ff90e4d2700 20 client.46811.objecter put_session s=0x7ff8f4024ad0 osd=6 1 2024-04-14T15:15:37.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.802+0000 7ff90e4d2700 10 client.46811.objecter close_session for osd.7 2024-04-14T15:15:37.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.802+0000 7ff90e4d2700 1 -- 172.21.15.90:0/2914038849 >> v1:172.21.15.92:6812/1773706716 conn(0x7ff8f40257e0 legacy=0x7ff8f402a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:37.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.802+0000 7ff90ca6f700 1 -- 172.21.15.90:0/2914038849 reap_dead start 2024-04-14T15:15:37.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.802+0000 7ff90e4d2700 20 client.46811.objecter put_session s=0x7ff8f4025670 osd=7 1 2024-04-14T15:15:37.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.802+0000 7ff90e4d2700 20 client.46811.objecter shutdown clearing up homeless session... 2024-04-14T15:15:37.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.802+0000 7ff90e4d2700 10 client.46811.objecter successfully canceled tick 2024-04-14T15:15:37.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.802+0000 7ff90e4d2700 1 -- 172.21.15.90:0/2914038849 >> v1:172.21.15.90:6800/420935427 conn(0x7ff8f0071390 legacy=0x7ff8f0073850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:37.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.802+0000 7ff90e4d2700 1 -- 172.21.15.90:0/2914038849 >> v1:172.21.15.90:6790/0 conn(0x7ff90810f300 legacy=0x7ff9081a5ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:37.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.802+0000 7ff90e4d2700 1 -- 172.21.15.90:0/2914038849 shutdown_connections 2024-04-14T15:15:37.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.802+0000 7ff90e4d2700 1 -- 172.21.15.90:0/2914038849 >> 172.21.15.90:0/2914038849 conn(0x7ff9080fe3d0 msgr2=0x7ff908111f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:15:37.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.802+0000 7ff90e4d2700 1 -- 172.21.15.90:0/2914038849 shutdown_connections 2024-04-14T15:15:37.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.802+0000 7ff90e4d2700 1 -- 172.21.15.90:0/2914038849 wait complete. 2024-04-14T15:15:37.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:37.802+0000 7ff90e4d2700 1 librados: shutdown 2024-04-14T15:15:37.806 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:15:37.806 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:15:37.806 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:15:37.806 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:15:37.806 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:15:37.806 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:15:37.806 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:15:37.806 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:15:37.876 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:15:37.901 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:37.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:37.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:15:38.003 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:15:38.028 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:38.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:38.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:15:38.130 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:15:38.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:38.155 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:38.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:15:38.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:37 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2914038849' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:15:38.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:37 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2015849731' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi090-93444-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi090-93444-2"}]': finished 2024-04-14T15:15:38.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:37 smithi090 ceph-mon[41670]: from='client.46787 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi090-93363-47"}]': finished 2024-04-14T15:15:38.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:37 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3884299353' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi090-93363-47"}]: dispatch 2024-04-14T15:15:38.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:37 smithi090 ceph-mon[41670]: osdmap e270: 8 total, 8 up, 8 in 2024-04-14T15:15:38.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:37 smithi090 ceph-mon[41670]: from='client.46787 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi090-93363-47"}]: dispatch 2024-04-14T15:15:38.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:37 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2914038849' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:15:38.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:37 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2015849731' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi090-93444-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi090-93444-2"}]': finished 2024-04-14T15:15:38.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:37 smithi090 ceph-mon[30399]: from='client.46787 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi090-93363-47"}]': finished 2024-04-14T15:15:38.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:37 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3884299353' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi090-93363-47"}]: dispatch 2024-04-14T15:15:38.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:37 smithi090 ceph-mon[30399]: osdmap e270: 8 total, 8 up, 8 in 2024-04-14T15:15:38.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:37 smithi090 ceph-mon[30399]: from='client.46787 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi090-93363-47"}]: dispatch 2024-04-14T15:15:38.256 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:15:38.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:38.281 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:38.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:15:38.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:37 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2914038849' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:15:38.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:37 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2015849731' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi090-93444-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi090-93444-2"}]': finished 2024-04-14T15:15:38.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:37 smithi092 ceph-mon[33966]: from='client.46787 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi090-93363-47"}]': finished 2024-04-14T15:15:38.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:37 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3884299353' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi090-93363-47"}]: dispatch 2024-04-14T15:15:38.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:37 smithi092 ceph-mon[33966]: osdmap e270: 8 total, 8 up, 8 in 2024-04-14T15:15:38.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:37 smithi092 ceph-mon[33966]: from='client.46787 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi090-93363-47"}]: dispatch 2024-04-14T15:15:38.382 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:15:38.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:38.406 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:38.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:15:38.507 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:15:38.535 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:38.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:38.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:15:38.636 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:15:38.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:38.664 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:38.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:15:38.765 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:15:38.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:38.790 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:38.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:15:38.814 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-14T15:15:38.814 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-14T15:15:38.814 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-04-14T15:15:38.814 INFO:tasks.thrashosds.thrasher:Setting osd 5 primary_affinity to 0.000000 2024-04-14T15:15:38.815 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd primary-affinity 5 0 2024-04-14T15:15:38.892 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:15:38.918 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:38.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:38.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:15:38.925 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ OK ] LibRadosAiotry=14 expected=14 2024-04-14T15:15:38.925 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : seek to 13:566253c9:::13:head 2024-04-14T15:15:38.925 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : cursor()=13:566253c9:::13:head expected=13:566253c9:::13:head 2024-04-14T15:15:38.925 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: > 13:566253c9:::13:head -> 13 2024-04-14T15:15:38.925 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : entry=13 expected=13 2024-04-14T15:15:38.925 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : seek to 13:52ea6a34:::10:head 2024-04-14T15:15:38.925 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : cursor()=13:52ea6a34:::10:head expected=13:52ea6a34:::10:head 2024-04-14T15:15:38.925 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: > 13:52ea6a34:::10:head -> 10 2024-04-14T15:15:38.925 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : entry=10 expected=10 2024-04-14T15:15:38.926 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : seek to 13:02547ec2:::1:head 2024-04-14T15:15:38.926 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : cursor()=13:02547ec2:::1:head expected=13:02547ec2:::1:head 2024-04-14T15:15:38.926 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: > 13:02547ec2:::1:head -> 1 2024-04-14T15:15:38.926 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : entry=1 expected=1 2024-04-14T15:15:38.926 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : seek to 13:e960b815:::9:head 2024-04-14T15:15:38.926 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : cursor()=13:e960b815:::9:head expected=13:e960b815:::9:head 2024-04-14T15:15:38.926 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: > 13:e960b815:::9:head -> 9 2024-04-14T15:15:38.926 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : entry=9 expected=9 2024-04-14T15:15:38.926 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : seek to 13:de5d7c5f:::12:head 2024-04-14T15:15:38.926 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : cursor()=13:de5d7c5f:::12:head expected=13:de5d7c5f:::12:head 2024-04-14T15:15:38.926 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: > 13:de5d7c5f:::12:head -> 12 2024-04-14T15:15:38.926 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : entry=12 expected=12 2024-04-14T15:15:38.926 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : seek to 13:c4fdafeb:::6:head 2024-04-14T15:15:38.926 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : cursor()=13:c4fdafeb:::6:head expected=13:c4fdafeb:::6:head 2024-04-14T15:15:38.927 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: > 13:c4fdafeb:::6:head -> 6 2024-04-14T15:15:38.927 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : entry=6 expected=6 2024-04-14T15:15:38.927 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : seek to 13:f905c69b:::2:head 2024-04-14T15:15:38.927 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : cursor()=13:f905c69b:::2:head expected=13:f905c69b:::2:head 2024-04-14T15:15:38.927 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: > 13:f905c69b:::2:head -> 2 2024-04-14T15:15:38.927 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : entry=2 expected=2 2024-04-14T15:15:38.927 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : seek to 13:bd63b0f1:::8:head 2024-04-14T15:15:38.927 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : cursor()=13:bd63b0f1:::8:head expected=13:bd63b0f1:::8:head 2024-04-14T15:15:38.927 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: > 13:bd63b0f1:::8:head -> 8 2024-04-14T15:15:38.927 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: : entry=8 expected=8 2024-04-14T15:15:38.927 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (685 ms) 2024-04-14T15:15:38.927 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2024-04-14T15:15:38.927 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (154435 ms) 2024-04-14T15:15:38.927 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2024-04-14T15:15:38.927 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: split 0/5 -> MIN 13:33333333::::head 2024-04-14T15:15:38.928 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: split 1/5 -> 13:33333333::::head 13:66666666::::head 2024-04-14T15:15:38.928 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: split 2/5 -> 13:66666666::::head 13:99999999::::head 2024-04-14T15:15:38.928 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: split 3/5 -> 13:99999999::::head 13:cccccccc::::head 2024-04-14T15:15:38.928 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: split 4/5 -> 13:cccccccc::::head MAX 2024-04-14T15:15:38.928 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (1903 ms) 2024-04-14T15:15:38.928 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: [----------] 7 tests from LibRadosList (157848 ms total) 2024-04-14T15:15:38.928 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: 2024-04-14T15:15:38.928 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: [----------] 3 tests from LibRadosListEC 2024-04-14T15:15:38.928 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2024-04-14T15:15:38.928 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (1012 ms) 2024-04-14T15:15:38.928 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2024-04-14T15:15:38.928 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: myset foo1,foo2,foo3 2024-04-14T15:15:38.928 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: foo1 2024-04-14T15:15:38.928 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: foo2 2024-04-14T15:15:38.929 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: foo3 2024-04-14T15:15:38.929 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: myset foo1,foo4,foo5 2024-04-14T15:15:38.929 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: foo4 2024-04-14T15:15:38.929 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: foo5 2024-04-14T15:15:38.929 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: foo1 2024-04-14T15:15:38.929 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: myset foo6,foo7 2024-04-14T15:15:38.929 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: foo7 2024-04-14T15:15:38.929 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: foo6 2024-04-14T15:15:38.929 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-04-14T15:15:38.929 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: ns1:foo4 2024-04-14T15:15:38.929 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: ns1:foo5 2024-04-14T15:15:38.929 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: ns2:foo7 2024-04-14T15:15:38.929 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: ns2:foo6 2024-04-14T15:15:38.929 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: ns1:foo1 2024-04-14T15:15:38.929 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: :foo1 2024-04-14T15:15:38.930 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: :foo2 2024-04-14T15:15:38.930 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: :foo3 2024-04-14T15:15:38.930 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (38 ms) 2024-04-14T15:15:38.939 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd set noscrub 2024-04-14T15:15:39.020 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:15:39.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:39.045 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:39.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:15:39.146 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:15:39.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:39.171 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:39.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:15:39.172 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:38 smithi090 ceph-mon[41670]: pgmap v350: 238 pgs: 1 active+clean+snaptrim, 237 active+clean; 8.6 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 14 op/s 2024-04-14T15:15:39.172 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:38 smithi090 ceph-mon[41670]: 11.17 scrub starts 2024-04-14T15:15:39.172 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:38 smithi090 ceph-mon[41670]: 11.17 scrub ok 2024-04-14T15:15:39.172 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:38 smithi090 ceph-mon[41670]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:15:39.172 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:38 smithi090 ceph-mon[41670]: 110.11 scrub starts 2024-04-14T15:15:39.172 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:38 smithi090 ceph-mon[41670]: 110.11 scrub ok 2024-04-14T15:15:39.173 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:38 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:15:39.173 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:38 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:15:39.173 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:38 smithi090 ceph-mon[41670]: from='client.48691 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi090-93342-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi090-93342-39"}]': finished 2024-04-14T15:15:39.173 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:38 smithi090 ceph-mon[41670]: from='client.46787 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi090-93363-47"}]': finished 2024-04-14T15:15:39.173 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:38 smithi090 ceph-mon[41670]: osdmap e271: 8 total, 8 up, 8 in 2024-04-14T15:15:39.173 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:38 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:39.173 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:38 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:39.173 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:38 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3688286685' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi090-93363-48"}]: dispatch 2024-04-14T15:15:39.173 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:38 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3688286685' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi090-93363-48"}]: dispatch 2024-04-14T15:15:39.173 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:38 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3688286685' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi090-93363-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:39.174 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:38 smithi090 ceph-mon[30399]: pgmap v350: 238 pgs: 1 active+clean+snaptrim, 237 active+clean; 8.6 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 14 op/s 2024-04-14T15:15:39.174 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:38 smithi090 ceph-mon[30399]: 11.17 scrub starts 2024-04-14T15:15:39.174 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:38 smithi090 ceph-mon[30399]: 11.17 scrub ok 2024-04-14T15:15:39.174 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:38 smithi090 ceph-mon[30399]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:15:39.174 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:38 smithi090 ceph-mon[30399]: 110.11 scrub starts 2024-04-14T15:15:39.174 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:38 smithi090 ceph-mon[30399]: 110.11 scrub ok 2024-04-14T15:15:39.174 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:38 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:15:39.174 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:38 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:15:39.174 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:38 smithi090 ceph-mon[30399]: from='client.48691 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi090-93342-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi090-93342-39"}]': finished 2024-04-14T15:15:39.174 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:38 smithi090 ceph-mon[30399]: from='client.46787 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi090-93363-47"}]': finished 2024-04-14T15:15:39.174 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:38 smithi090 ceph-mon[30399]: osdmap e271: 8 total, 8 up, 8 in 2024-04-14T15:15:39.175 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:38 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:39.175 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:38 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:39.175 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:38 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3688286685' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi090-93363-48"}]: dispatch 2024-04-14T15:15:39.175 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:38 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3688286685' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi090-93363-48"}]: dispatch 2024-04-14T15:15:39.175 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:38 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3688286685' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi090-93363-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:39.272 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:15:39.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:39.297 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:39.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:15:39.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:38 smithi092 ceph-mon[33966]: pgmap v350: 238 pgs: 1 active+clean+snaptrim, 237 active+clean; 8.6 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 14 op/s 2024-04-14T15:15:39.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:38 smithi092 ceph-mon[33966]: 11.17 scrub starts 2024-04-14T15:15:39.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:38 smithi092 ceph-mon[33966]: 11.17 scrub ok 2024-04-14T15:15:39.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:38 smithi092 ceph-mon[33966]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:15:39.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:38 smithi092 ceph-mon[33966]: 110.11 scrub starts 2024-04-14T15:15:39.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:38 smithi092 ceph-mon[33966]: 110.11 scrub ok 2024-04-14T15:15:39.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:38 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:15:39.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:38 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:15:39.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:38 smithi092 ceph-mon[33966]: from='client.48691 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi090-93342-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi090-93342-39"}]': finished 2024-04-14T15:15:39.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:38 smithi092 ceph-mon[33966]: from='client.46787 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi090-93363-47"}]': finished 2024-04-14T15:15:39.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:38 smithi092 ceph-mon[33966]: osdmap e271: 8 total, 8 up, 8 in 2024-04-14T15:15:39.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:38 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:39.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:38 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:39.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:38 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3688286685' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi090-93363-48"}]: dispatch 2024-04-14T15:15:39.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:38 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3688286685' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi090-93363-48"}]: dispatch 2024-04-14T15:15:39.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:38 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3688286685' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi090-93363-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:39.399 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:15:39.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:39.424 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:39.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:15:39.524 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:15:39.548 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:39.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:39.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:15:39.650 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:15:39.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:39.678 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:39.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:15:39.779 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:15:39.805 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:39.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:39.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:15:39.829 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:15:39.906 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:15:39.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:39.933 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:39.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:15:40.034 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:15:40.037 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:39 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:15:40.037 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:39 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:15:40.038 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:39 smithi090 ceph-mon[30399]: 110.d scrub starts 2024-04-14T15:15:40.038 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:39 smithi090 ceph-mon[30399]: 110.d scrub ok 2024-04-14T15:15:40.038 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:39 smithi090 ceph-mon[30399]: 4.1 scrub starts 2024-04-14T15:15:40.038 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:39 smithi090 ceph-mon[30399]: 4.1 scrub ok 2024-04-14T15:15:40.038 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:39 smithi090 ceph-mon[30399]: 2.7 scrub starts 2024-04-14T15:15:40.038 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:39 smithi090 ceph-mon[30399]: 2.7 scrub ok 2024-04-14T15:15:40.038 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:39 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:15:40.038 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:39 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:15:40.038 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:39 smithi090 ceph-mon[30399]: 4.8 scrub starts 2024-04-14T15:15:40.038 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:39 smithi090 ceph-mon[30399]: 4.8 scrub ok 2024-04-14T15:15:40.038 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:39 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:15:40.038 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:39 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3688286685' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi090-93363-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:15:40.038 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:39 smithi090 ceph-mon[30399]: osdmap e272: 8 total, 8 up, 8 in 2024-04-14T15:15:40.039 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:39 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3688286685' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi090-93363-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi090-93363-48"}]: dispatch 2024-04-14T15:15:40.039 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:39 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2015849731' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi090-93444-2"}]: dispatch 2024-04-14T15:15:40.039 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:39 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:15:40.039 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:39 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:15:40.039 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:39 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:15:40.039 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:39 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:15:40.039 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:39 smithi090 ceph-mon[41670]: 110.d scrub starts 2024-04-14T15:15:40.039 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:39 smithi090 ceph-mon[41670]: 110.d scrub ok 2024-04-14T15:15:40.040 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:39 smithi090 ceph-mon[41670]: 4.1 scrub starts 2024-04-14T15:15:40.040 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:39 smithi090 ceph-mon[41670]: 4.1 scrub ok 2024-04-14T15:15:40.040 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:39 smithi090 ceph-mon[41670]: 2.7 scrub starts 2024-04-14T15:15:40.040 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:39 smithi090 ceph-mon[41670]: 2.7 scrub ok 2024-04-14T15:15:40.040 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:39 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:15:40.040 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:39 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:15:40.040 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:39 smithi090 ceph-mon[41670]: 4.8 scrub starts 2024-04-14T15:15:40.040 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:39 smithi090 ceph-mon[41670]: 4.8 scrub ok 2024-04-14T15:15:40.040 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:39 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:15:40.040 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:39 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3688286685' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi090-93363-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:15:40.040 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:39 smithi090 ceph-mon[41670]: osdmap e272: 8 total, 8 up, 8 in 2024-04-14T15:15:40.040 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:39 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3688286685' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi090-93363-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi090-93363-48"}]: dispatch 2024-04-14T15:15:40.040 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:39 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2015849731' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi090-93444-2"}]: dispatch 2024-04-14T15:15:40.040 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:39 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:15:40.041 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:39 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:15:40.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:40.061 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:40.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:15:40.153 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:15:40.163 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:15:40.170 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-14T15:15:40.170 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-14T15:15:40.170 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-14T15:15:40.170 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-14T15:15:40.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:40.188 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:40.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:15:40.289 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:15:40.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:40.316 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:40.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:15:40.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:39 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:15:40.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:39 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:15:40.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:39 smithi092 ceph-mon[33966]: 110.d scrub starts 2024-04-14T15:15:40.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:39 smithi092 ceph-mon[33966]: 110.d scrub ok 2024-04-14T15:15:40.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:39 smithi092 ceph-mon[33966]: 4.1 scrub starts 2024-04-14T15:15:40.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:39 smithi092 ceph-mon[33966]: 4.1 scrub ok 2024-04-14T15:15:40.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:39 smithi092 ceph-mon[33966]: 2.7 scrub starts 2024-04-14T15:15:40.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:39 smithi092 ceph-mon[33966]: 2.7 scrub ok 2024-04-14T15:15:40.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:39 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:15:40.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:39 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:15:40.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:39 smithi092 ceph-mon[33966]: 4.8 scrub starts 2024-04-14T15:15:40.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:39 smithi092 ceph-mon[33966]: 4.8 scrub ok 2024-04-14T15:15:40.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:39 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:15:40.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:39 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3688286685' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi090-93363-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:15:40.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:39 smithi092 ceph-mon[33966]: osdmap e272: 8 total, 8 up, 8 in 2024-04-14T15:15:40.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:39 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3688286685' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi090-93363-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi090-93363-48"}]: dispatch 2024-04-14T15:15:40.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:39 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2015849731' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi090-93444-2"}]: dispatch 2024-04-14T15:15:40.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:39 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:15:40.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:39 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:15:40.417 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:15:40.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:40.443 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:40.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:15:40.544 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:15:40.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:40.570 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:40.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:15:40.671 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:15:40.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:40.696 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:40.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:15:40.798 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:15:40.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:40.823 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:40.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:15:40.924 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:15:40.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:40.950 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:40.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:15:41.009 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-14T15:15:41.050 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:15:41.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:41.076 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:41.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:15:41.177 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:15:41.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:40 smithi090 ceph-mon[41670]: pgmap v353: 254 pgs: 32 creating+peering, 10 unknown, 212 active+clean; 4.6 MiB data, 557 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:15:41.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:40 smithi090 ceph-mon[41670]: 4.d scrub starts 2024-04-14T15:15:41.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:40 smithi090 ceph-mon[41670]: 4.d scrub ok 2024-04-14T15:15:41.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:40 smithi090 ceph-mon[41670]: 6.a deep-scrub starts 2024-04-14T15:15:41.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:40 smithi090 ceph-mon[41670]: 6.a deep-scrub ok 2024-04-14T15:15:41.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:40 smithi090 ceph-mon[41670]: 110.2 scrub starts 2024-04-14T15:15:41.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:40 smithi090 ceph-mon[41670]: 110.2 scrub ok 2024-04-14T15:15:41.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:40 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2015849731' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi090-93444-2"}]': finished 2024-04-14T15:15:41.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:40 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-31", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:15:41.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:40 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-31"}]: dispatch 2024-04-14T15:15:41.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:40 smithi090 ceph-mon[41670]: osdmap e273: 8 total, 8 up, 8 in 2024-04-14T15:15:41.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:40 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2015849731' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi090-93444-2"}]: dispatch 2024-04-14T15:15:41.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:40 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3211270109' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi090-93342-39"}]: dispatch 2024-04-14T15:15:41.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:40 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-31"}]: dispatch 2024-04-14T15:15:41.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:40 smithi090 ceph-mon[41670]: from='client.48691 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi090-93342-39"}]: dispatch 2024-04-14T15:15:41.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:40 smithi090 ceph-mon[30399]: pgmap v353: 254 pgs: 32 creating+peering, 10 unknown, 212 active+clean; 4.6 MiB data, 557 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:15:41.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:40 smithi090 ceph-mon[30399]: 4.d scrub starts 2024-04-14T15:15:41.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:40 smithi090 ceph-mon[30399]: 4.d scrub ok 2024-04-14T15:15:41.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:40 smithi090 ceph-mon[30399]: 6.a deep-scrub starts 2024-04-14T15:15:41.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:40 smithi090 ceph-mon[30399]: 6.a deep-scrub ok 2024-04-14T15:15:41.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:40 smithi090 ceph-mon[30399]: 110.2 scrub starts 2024-04-14T15:15:41.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:40 smithi090 ceph-mon[30399]: 110.2 scrub ok 2024-04-14T15:15:41.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:40 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2015849731' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi090-93444-2"}]': finished 2024-04-14T15:15:41.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:40 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-31", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:15:41.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:40 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-31"}]: dispatch 2024-04-14T15:15:41.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:40 smithi090 ceph-mon[30399]: osdmap e273: 8 total, 8 up, 8 in 2024-04-14T15:15:41.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:40 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2015849731' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi090-93444-2"}]: dispatch 2024-04-14T15:15:41.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:40 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3211270109' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi090-93342-39"}]: dispatch 2024-04-14T15:15:41.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:40 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-31"}]: dispatch 2024-04-14T15:15:41.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:40 smithi090 ceph-mon[30399]: from='client.48691 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi090-93342-39"}]: dispatch 2024-04-14T15:15:41.203 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:41.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:41.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:15:41.304 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:15:41.329 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:41.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:41.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:15:41.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:40 smithi092 ceph-mon[33966]: pgmap v353: 254 pgs: 32 creating+peering, 10 unknown, 212 active+clean; 4.6 MiB data, 557 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:15:41.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:40 smithi092 ceph-mon[33966]: 4.d scrub starts 2024-04-14T15:15:41.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:40 smithi092 ceph-mon[33966]: 4.d scrub ok 2024-04-14T15:15:41.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:40 smithi092 ceph-mon[33966]: 6.a deep-scrub starts 2024-04-14T15:15:41.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:40 smithi092 ceph-mon[33966]: 6.a deep-scrub ok 2024-04-14T15:15:41.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:40 smithi092 ceph-mon[33966]: 110.2 scrub starts 2024-04-14T15:15:41.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:40 smithi092 ceph-mon[33966]: 110.2 scrub ok 2024-04-14T15:15:41.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:40 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2015849731' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi090-93444-2"}]': finished 2024-04-14T15:15:41.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:40 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-31", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:15:41.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:40 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-31"}]: dispatch 2024-04-14T15:15:41.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:40 smithi092 ceph-mon[33966]: osdmap e273: 8 total, 8 up, 8 in 2024-04-14T15:15:41.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:40 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2015849731' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi090-93444-2"}]: dispatch 2024-04-14T15:15:41.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:40 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3211270109' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi090-93342-39"}]: dispatch 2024-04-14T15:15:41.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:40 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-31"}]: dispatch 2024-04-14T15:15:41.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:40 smithi092 ceph-mon[33966]: from='client.48691 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi090-93342-39"}]: dispatch 2024-04-14T15:15:41.431 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:15:41.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:41.456 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:41.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:15:41.557 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:15:41.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:41.582 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:41.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:15:41.683 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:15:41.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:41.708 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:41.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:15:41.809 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:15:41.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:41.833 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:41.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:15:41.934 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:15:41.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:41.959 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:41.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:15:42.060 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:15:42.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:42.084 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:42.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:15:42.152 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:41 smithi090 ceph-mon[41670]: 5.d scrub starts 2024-04-14T15:15:42.152 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:41 smithi090 ceph-mon[41670]: 5.d scrub ok 2024-04-14T15:15:42.152 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:41 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3688286685' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi090-93363-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi090-93363-48"}]': finished 2024-04-14T15:15:42.152 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:41 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2015849731' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi090-93444-2"}]': finished 2024-04-14T15:15:42.153 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:41 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-31"}]': finished 2024-04-14T15:15:42.153 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:41 smithi090 ceph-mon[41670]: from='client.48691 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi090-93342-39"}]': finished 2024-04-14T15:15:42.153 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:41 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-31", "mode": "writeback"}]: dispatch 2024-04-14T15:15:42.153 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:41 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3211270109' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi090-93342-39"}]: dispatch 2024-04-14T15:15:42.153 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:41 smithi090 ceph-mon[41670]: osdmap e274: 8 total, 8 up, 8 in 2024-04-14T15:15:42.153 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:41 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-31", "mode": "writeback"}]: dispatch 2024-04-14T15:15:42.153 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:41 smithi090 ceph-mon[41670]: from='client.48691 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi090-93342-39"}]: dispatch 2024-04-14T15:15:42.153 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:41 smithi090 ceph-mon[30399]: 5.d scrub starts 2024-04-14T15:15:42.153 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:41 smithi090 ceph-mon[30399]: 5.d scrub ok 2024-04-14T15:15:42.153 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:41 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3688286685' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi090-93363-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi090-93363-48"}]': finished 2024-04-14T15:15:42.153 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:41 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2015849731' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi090-93444-2"}]': finished 2024-04-14T15:15:42.153 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:41 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-31"}]': finished 2024-04-14T15:15:42.154 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:41 smithi090 ceph-mon[30399]: from='client.48691 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi090-93342-39"}]': finished 2024-04-14T15:15:42.154 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:41 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-31", "mode": "writeback"}]: dispatch 2024-04-14T15:15:42.154 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:41 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3211270109' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi090-93342-39"}]: dispatch 2024-04-14T15:15:42.154 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:41 smithi090 ceph-mon[30399]: osdmap e274: 8 total, 8 up, 8 in 2024-04-14T15:15:42.154 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:41 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-31", "mode": "writeback"}]: dispatch 2024-04-14T15:15:42.154 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:41 smithi090 ceph-mon[30399]: from='client.48691 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi090-93342-39"}]: dispatch 2024-04-14T15:15:42.184 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:15:42.208 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:42.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:42.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:15:42.310 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:15:42.334 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:42.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:42.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:15:42.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:41 smithi092 ceph-mon[33966]: 5.d scrub starts 2024-04-14T15:15:42.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:41 smithi092 ceph-mon[33966]: 5.d scrub ok 2024-04-14T15:15:42.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:41 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3688286685' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi090-93363-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi090-93363-48"}]': finished 2024-04-14T15:15:42.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:41 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2015849731' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi090-93444-2"}]': finished 2024-04-14T15:15:42.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:41 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-31"}]': finished 2024-04-14T15:15:42.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:41 smithi092 ceph-mon[33966]: from='client.48691 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi090-93342-39"}]': finished 2024-04-14T15:15:42.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:41 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-31", "mode": "writeback"}]: dispatch 2024-04-14T15:15:42.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:41 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3211270109' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi090-93342-39"}]: dispatch 2024-04-14T15:15:42.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:41 smithi092 ceph-mon[33966]: osdmap e274: 8 total, 8 up, 8 in 2024-04-14T15:15:42.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:41 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-31", "mode": "writeback"}]: dispatch 2024-04-14T15:15:42.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:41 smithi092 ceph-mon[33966]: from='client.48691 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi090-93342-39"}]: dispatch 2024-04-14T15:15:42.437 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:15:42.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:42.462 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:42.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:15:42.563 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:15:42.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:42.588 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:42.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:15:42.689 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:15:42.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:42.714 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:42.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:15:42.815 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:15:42.841 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:42.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:42.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:15:42.944 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:15:42.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:42.968 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:42.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:15:43.069 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:15:43.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:43.094 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:43.095 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:15:43.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:42 smithi090 ceph-mon[41670]: pgmap v356: 238 pgs: 32 creating+peering, 206 active+clean; 4.6 MiB data, 553 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:15:43.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:42 smithi090 ceph-mon[41670]: 4.2 scrub starts 2024-04-14T15:15:43.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:42 smithi090 ceph-mon[41670]: 4.2 scrub ok 2024-04-14T15:15:43.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:42 smithi090 ceph-mon[41670]: 110.14 scrub starts 2024-04-14T15:15:43.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:42 smithi090 ceph-mon[41670]: 110.14 scrub ok 2024-04-14T15:15:43.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:42 smithi090 ceph-mon[41670]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:15:43.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:42 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-31", "mode": "writeback"}]': finished 2024-04-14T15:15:43.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:42 smithi090 ceph-mon[41670]: from='client.48691 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi090-93342-39"}]': finished 2024-04-14T15:15:43.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:42 smithi090 ceph-mon[41670]: osdmap e275: 8 total, 8 up, 8 in 2024-04-14T15:15:43.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:42 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1438530465' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93444-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:43.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:42 smithi090 ceph-mon[41670]: from='client.48709 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93444-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:43.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:42 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1788001083' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi090-93342-40"}]: dispatch 2024-04-14T15:15:43.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:42 smithi090 ceph-mon[41670]: from='client.48715 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi090-93342-40"}]: dispatch 2024-04-14T15:15:43.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:42 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1788001083' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi090-93342-40"}]: dispatch 2024-04-14T15:15:43.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:42 smithi090 ceph-mon[41670]: from='client.48715 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi090-93342-40"}]: dispatch 2024-04-14T15:15:43.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:42 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1788001083' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi090-93342-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:43.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:42 smithi090 ceph-mon[41670]: from='client.48715 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi090-93342-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:43.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:42 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:15:43.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:42 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:15:43.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:42 smithi090 ceph-mon[30399]: pgmap v356: 238 pgs: 32 creating+peering, 206 active+clean; 4.6 MiB data, 553 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:15:43.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:42 smithi090 ceph-mon[30399]: 4.2 scrub starts 2024-04-14T15:15:43.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:42 smithi090 ceph-mon[30399]: 4.2 scrub ok 2024-04-14T15:15:43.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:42 smithi090 ceph-mon[30399]: 110.14 scrub starts 2024-04-14T15:15:43.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:42 smithi090 ceph-mon[30399]: 110.14 scrub ok 2024-04-14T15:15:43.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:42 smithi090 ceph-mon[30399]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:15:43.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:42 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-31", "mode": "writeback"}]': finished 2024-04-14T15:15:43.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:42 smithi090 ceph-mon[30399]: from='client.48691 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi090-93342-39"}]': finished 2024-04-14T15:15:43.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:42 smithi090 ceph-mon[30399]: osdmap e275: 8 total, 8 up, 8 in 2024-04-14T15:15:43.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:42 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1438530465' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93444-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:43.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:42 smithi090 ceph-mon[30399]: from='client.48709 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93444-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:43.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:42 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1788001083' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi090-93342-40"}]: dispatch 2024-04-14T15:15:43.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:42 smithi090 ceph-mon[30399]: from='client.48715 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi090-93342-40"}]: dispatch 2024-04-14T15:15:43.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:42 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1788001083' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi090-93342-40"}]: dispatch 2024-04-14T15:15:43.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:42 smithi090 ceph-mon[30399]: from='client.48715 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi090-93342-40"}]: dispatch 2024-04-14T15:15:43.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:42 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1788001083' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi090-93342-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:43.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:42 smithi090 ceph-mon[30399]: from='client.48715 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi090-93342-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:43.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:42 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:15:43.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:42 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:15:43.196 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:15:43.221 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:43.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:43.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:15:43.252 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-14T15:15:43.322 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:15:43.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:43.347 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:43.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:15:43.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:42 smithi092 ceph-mon[33966]: pgmap v356: 238 pgs: 32 creating+peering, 206 active+clean; 4.6 MiB data, 553 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:15:43.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:42 smithi092 ceph-mon[33966]: 4.2 scrub starts 2024-04-14T15:15:43.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:42 smithi092 ceph-mon[33966]: 4.2 scrub ok 2024-04-14T15:15:43.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:42 smithi092 ceph-mon[33966]: 110.14 scrub starts 2024-04-14T15:15:43.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:42 smithi092 ceph-mon[33966]: 110.14 scrub ok 2024-04-14T15:15:43.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:42 smithi092 ceph-mon[33966]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:15:43.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:42 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-31", "mode": "writeback"}]': finished 2024-04-14T15:15:43.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:42 smithi092 ceph-mon[33966]: from='client.48691 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi090-93342-39"}]': finished 2024-04-14T15:15:43.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:42 smithi092 ceph-mon[33966]: osdmap e275: 8 total, 8 up, 8 in 2024-04-14T15:15:43.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:42 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1438530465' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93444-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:43.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:42 smithi092 ceph-mon[33966]: from='client.48709 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93444-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:43.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:42 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1788001083' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi090-93342-40"}]: dispatch 2024-04-14T15:15:43.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:42 smithi092 ceph-mon[33966]: from='client.48715 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi090-93342-40"}]: dispatch 2024-04-14T15:15:43.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:42 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1788001083' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi090-93342-40"}]: dispatch 2024-04-14T15:15:43.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:42 smithi092 ceph-mon[33966]: from='client.48715 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi090-93342-40"}]: dispatch 2024-04-14T15:15:43.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:42 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1788001083' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi090-93342-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:43.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:42 smithi092 ceph-mon[33966]: from='client.48715 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi090-93342-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:43.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:42 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:15:43.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:42 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:15:43.448 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:15:43.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:43.473 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:43.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:15:43.574 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:15:43.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:43.599 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:43.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:15:43.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.479+0000 7fd2caad4700 1 -- 172.21.15.90:0/489433259 >> v1:172.21.15.90:6790/0 conn(0x7fd2c4108890 legacy=0x7fd2c4108c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:43.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.479+0000 7fd2caad4700 1 -- 172.21.15.90:0/489433259 shutdown_connections 2024-04-14T15:15:43.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.479+0000 7fd2caad4700 1 -- 172.21.15.90:0/489433259 >> 172.21.15.90:0/489433259 conn(0x7fd2c40fe380 msgr2=0x7fd2c41007a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:15:43.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.479+0000 7fd2caad4700 1 -- 172.21.15.90:0/489433259 shutdown_connections 2024-04-14T15:15:43.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.479+0000 7fd2caad4700 1 -- 172.21.15.90:0/489433259 wait complete. 2024-04-14T15:15:43.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.479+0000 7fd2caad4700 1 librados: starting msgr at 2024-04-14T15:15:43.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.479+0000 7fd2caad4700 1 librados: starting objecter 2024-04-14T15:15:43.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.480+0000 7fd2caad4700 1 Processor -- start 2024-04-14T15:15:43.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.480+0000 7fd2caad4700 1 -- start start 2024-04-14T15:15:43.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.480+0000 7fd2caad4700 1 librados: setting wanted keys 2024-04-14T15:15:43.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.480+0000 7fd2caad4700 1 librados: calling monclient init 2024-04-14T15:15:43.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.480+0000 7fd2caad4700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd2c410ee30 con 0x7fd2c410b740 2024-04-14T15:15:43.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.480+0000 7fd2caad4700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd2c410f000 con 0x7fd2c4108890 2024-04-14T15:15:43.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.480+0000 7fd2caad4700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd2c41b27d0 con 0x7fd2c410f310 2024-04-14T15:15:43.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.480+0000 7fd2c3fff700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fd2c410b740 0x7fd2c4074530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:45820/0 (socket says 172.21.15.90:45820) 2024-04-14T15:15:43.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.480+0000 7fd2c9071700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7fd2c410f310 0x7fd2c410c470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:38812/0 (socket says 172.21.15.90:38812) 2024-04-14T15:15:43.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.480+0000 7fd2c3fff700 1 -- 172.21.15.90:0/3396532154 learned_addr learned my addr 172.21.15.90:0/3396532154 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:15:43.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.480+0000 7fd2c1ffb700 10 client.?.objecter ms_handle_connect 0x7fd2c410f310 2024-04-14T15:15:43.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.480+0000 7fd2c1ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:15:43.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.480+0000 7fd2c1ffb700 10 client.?.objecter ms_handle_connect 0x7fd2c410b740 2024-04-14T15:15:43.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.480+0000 7fd2c1ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:15:43.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.480+0000 7fd2c1ffb700 10 client.?.objecter ms_handle_connect 0x7fd2c4108890 2024-04-14T15:15:43.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.480+0000 7fd2c1ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:15:43.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.480+0000 7fd2c1ffb700 1 -- 172.21.15.90:0/3396532154 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 224311337 0 0) 0x7fd2c410ee30 con 0x7fd2c410b740 2024-04-14T15:15:43.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.480+0000 7fd2c1ffb700 1 -- 172.21.15.90:0/3396532154 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd2b4003480 con 0x7fd2c410b740 2024-04-14T15:15:43.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.480+0000 7fd2c1ffb700 1 -- 172.21.15.90:0/3396532154 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1468411947 0 0) 0x7fd2c41b27d0 con 0x7fd2c410f310 2024-04-14T15:15:43.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.480+0000 7fd2c1ffb700 1 -- 172.21.15.90:0/3396532154 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd2c410ee30 con 0x7fd2c410f310 2024-04-14T15:15:43.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.480+0000 7fd2c1ffb700 1 -- 172.21.15.90:0/3396532154 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 99633363 0 0) 0x7fd2c410f000 con 0x7fd2c4108890 2024-04-14T15:15:43.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.480+0000 7fd2c1ffb700 1 -- 172.21.15.90:0/3396532154 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd2c41b27d0 con 0x7fd2c4108890 2024-04-14T15:15:43.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.480+0000 7fd2c1ffb700 1 -- 172.21.15.90:0/3396532154 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2948071885 0 0) 0x7fd2b4003480 con 0x7fd2c410b740 2024-04-14T15:15:43.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.480+0000 7fd2c1ffb700 1 -- 172.21.15.90:0/3396532154 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd2c410f000 con 0x7fd2c410b740 2024-04-14T15:15:43.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.480+0000 7fd2c1ffb700 1 -- 172.21.15.90:0/3396532154 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 809474203 0 0) 0x7fd2c41b27d0 con 0x7fd2c4108890 2024-04-14T15:15:43.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.480+0000 7fd2c1ffb700 1 -- 172.21.15.90:0/3396532154 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd2b4003480 con 0x7fd2c4108890 2024-04-14T15:15:43.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.481+0000 7fd2c1ffb700 1 -- 172.21.15.90:0/3396532154 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fd2b00018f0 con 0x7fd2c410b740 2024-04-14T15:15:43.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.481+0000 7fd2c1ffb700 1 -- 172.21.15.90:0/3396532154 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1405818868 0 0) 0x7fd2c410ee30 con 0x7fd2c410f310 2024-04-14T15:15:43.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.481+0000 7fd2c1ffb700 1 -- 172.21.15.90:0/3396532154 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd2c41b27d0 con 0x7fd2c410f310 2024-04-14T15:15:43.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.481+0000 7fd2c1ffb700 1 -- 172.21.15.90:0/3396532154 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fd2b80015d0 con 0x7fd2c4108890 2024-04-14T15:15:43.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.481+0000 7fd2c1ffb700 1 -- 172.21.15.90:0/3396532154 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fd2bc001c10 con 0x7fd2c410f310 2024-04-14T15:15:43.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.481+0000 7fd2c1ffb700 1 -- 172.21.15.90:0/3396532154 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3894634718 0 0) 0x7fd2b4003480 con 0x7fd2c4108890 2024-04-14T15:15:43.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.481+0000 7fd2c1ffb700 1 -- 172.21.15.90:0/3396532154 >> v1:172.21.15.90:6790/0 conn(0x7fd2c410f310 legacy=0x7fd2c410c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:43.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.481+0000 7fd2c1ffb700 1 -- 172.21.15.90:0/3396532154 >> v1:172.21.15.90:6789/0 conn(0x7fd2c410b740 legacy=0x7fd2c4074530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:43.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.481+0000 7fd2c1ffb700 1 -- 172.21.15.90:0/3396532154 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2c41b39b0 con 0x7fd2c4108890 2024-04-14T15:15:43.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.481+0000 7fd2caad4700 1 -- 172.21.15.90:0/3396532154 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd2c41b2a00 con 0x7fd2c4108890 2024-04-14T15:15:43.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.481+0000 7fd2c1ffb700 1 -- 172.21.15.90:0/3396532154 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fd2b8003bd0 con 0x7fd2c4108890 2024-04-14T15:15:43.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.481+0000 7fd2c1ffb700 1 -- 172.21.15.90:0/3396532154 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fd2b8004f30 con 0x7fd2c4108890 2024-04-14T15:15:43.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.481+0000 7fd2caad4700 10 client.48721.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:15:43.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.481+0000 7fd2caad4700 1 -- 172.21.15.90:0/3396532154 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd2c41b2ec0 con 0x7fd2c4108890 2024-04-14T15:15:43.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.481+0000 7fd2caad4700 1 librados: init done 2024-04-14T15:15:43.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.482+0000 7fd2caad4700 1 -- 172.21.15.90:0/3396532154 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2c4103fd0 con 0x7fd2c4108890 2024-04-14T15:15:43.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.482+0000 7fd2c1ffb700 1 -- 172.21.15.90:0/3396532154 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fd2b80044f0 con 0x7fd2c4108890 2024-04-14T15:15:43.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.483+0000 7fd2c1ffb700 1 -- 172.21.15.90:0/3396532154 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(275..275 src has 51..275) ==== 9316+0+0 (unknown 737518190 0 0) 0x7fd2b8089f20 con 0x7fd2c4108890 2024-04-14T15:15:43.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.483+0000 7fd2c1ffb700 10 client.48721.objecter ms_dispatch 0x7fd2c4001550 osd_map(275..275 src has 51..275) 2024-04-14T15:15:43.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.483+0000 7fd2c1ffb700 3 client.48721.objecter handle_osd_map got epochs [275,275] > 0 2024-04-14T15:15:43.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.483+0000 7fd2c1ffb700 3 client.48721.objecter handle_osd_map decoding full epoch 275 2024-04-14T15:15:43.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.483+0000 7fd2c1ffb700 20 client.48721.objecter dump_active .. 0 homeless 2024-04-14T15:15:43.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.483+0000 7fd2c1ffb700 10 client.48721.objecter ms_handle_connect 0x7fd2b4071340 2024-04-14T15:15:43.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.487+0000 7fd2c1ffb700 1 -- 172.21.15.90:0/3396532154 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fd2b80564b0 con 0x7fd2c4108890 2024-04-14T15:15:43.699 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:15:43.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.703+0000 7fd2caad4700 1 -- 172.21.15.90:0/3396532154 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd primary-affinity", "id": 5, "weight": 0.0} v 0) -- 0x7fd2c40693d0 con 0x7fd2c4108890 2024-04-14T15:15:43.722 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:43.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:43.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:15:43.824 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:15:43.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:43.849 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:43.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:15:43.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.859+0000 7fd2c1ffb700 1 -- 172.21.15.90:0/3396532154 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]=0 set osd.5 primary-affinity to 0 (802) v276) ==== 129+0+0 (unknown 743031519 0 0) 0x7fd2b805a160 con 0x7fd2c4108890 2024-04-14T15:15:43.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.863+0000 7fd2caad4700 10 librados: watch_flush enter 2024-04-14T15:15:43.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.863+0000 7fd2caad4700 10 librados: watch_flush exit 2024-04-14T15:15:43.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.863+0000 7fd2caad4700 20 client.48721.objecter shutdown clearing up homeless session... 2024-04-14T15:15:43.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.863+0000 7fd2caad4700 10 client.48721.objecter successfully canceled tick 2024-04-14T15:15:43.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.864+0000 7fd2caad4700 1 -- 172.21.15.90:0/3396532154 >> v1:172.21.15.90:6800/420935427 conn(0x7fd2b4071340 legacy=0x7fd2b4073800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:43.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.864+0000 7fd2caad4700 1 -- 172.21.15.90:0/3396532154 >> v1:172.21.15.92:6789/0 conn(0x7fd2c4108890 legacy=0x7fd2c410bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:43.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.865+0000 7fd2caad4700 1 -- 172.21.15.90:0/3396532154 shutdown_connections 2024-04-14T15:15:43.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.865+0000 7fd2caad4700 1 -- 172.21.15.90:0/3396532154 >> 172.21.15.90:0/3396532154 conn(0x7fd2c40fe380 msgr2=0x7fd2c4102890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:15:43.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.865+0000 7fd2caad4700 1 -- 172.21.15.90:0/3396532154 shutdown_connections 2024-04-14T15:15:43.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.865+0000 7fd2caad4700 1 -- 172.21.15.90:0/3396532154 wait complete. 2024-04-14T15:15:43.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:43.866+0000 7fd2caad4700 1 librados: shutdown 2024-04-14T15:15:43.868 INFO:teuthology.orchestra.run.smithi090.stderr:set osd.5 primary-affinity to 0 (802) 2024-04-14T15:15:43.950 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:15:43.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:43.975 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:43.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:15:44.076 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:15:44.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:44.101 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:44.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:15:44.117 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:15:43 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:15:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:15:44.118 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:43 smithi090 ceph-mon[41670]: 110.6 deep-scrub starts 2024-04-14T15:15:44.118 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:43 smithi090 ceph-mon[41670]: 110.6 deep-scrub ok 2024-04-14T15:15:44.118 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:43 smithi090 ceph-mon[41670]: 5.17 scrub starts 2024-04-14T15:15:44.118 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:43 smithi090 ceph-mon[41670]: 5.17 scrub ok 2024-04-14T15:15:44.118 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:43 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3396532154' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]: dispatch 2024-04-14T15:15:44.118 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:43 smithi090 ceph-mon[41670]: from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]: dispatch 2024-04-14T15:15:44.118 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:43 smithi090 ceph-mon[41670]: from='client.48709 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93444-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:15:44.118 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:43 smithi090 ceph-mon[41670]: from='client.48715 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi090-93342-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:15:44.118 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:43 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:15:44.118 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:43 smithi090 ceph-mon[41670]: from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]': finished 2024-04-14T15:15:44.118 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:43 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-31"}]: dispatch 2024-04-14T15:15:44.119 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:43 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1788001083' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi090-93342-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi090-93342-40"}]: dispatch 2024-04-14T15:15:44.119 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:43 smithi090 ceph-mon[41670]: osdmap e276: 8 total, 8 up, 8 in 2024-04-14T15:15:44.119 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:43 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-31"}]: dispatch 2024-04-14T15:15:44.119 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:43 smithi090 ceph-mon[41670]: from='client.48715 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi090-93342-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi090-93342-40"}]: dispatch 2024-04-14T15:15:44.119 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:43 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3688286685' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi090-93363-48"}]: dispatch 2024-04-14T15:15:44.119 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:43 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1438530465' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi090-93444-3","pool2":"test-rados-api-smithi090-93444-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-04-14T15:15:44.119 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:43 smithi090 ceph-mon[41670]: from='client.48709 ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi090-93444-3","pool2":"test-rados-api-smithi090-93444-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-04-14T15:15:44.119 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:43 smithi090 ceph-mon[30399]: 110.6 deep-scrub starts 2024-04-14T15:15:44.119 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:43 smithi090 ceph-mon[30399]: 110.6 deep-scrub ok 2024-04-14T15:15:44.119 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:43 smithi090 ceph-mon[30399]: 5.17 scrub starts 2024-04-14T15:15:44.119 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:43 smithi090 ceph-mon[30399]: 5.17 scrub ok 2024-04-14T15:15:44.119 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:43 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3396532154' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]: dispatch 2024-04-14T15:15:44.120 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:43 smithi090 ceph-mon[30399]: from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]: dispatch 2024-04-14T15:15:44.120 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:43 smithi090 ceph-mon[30399]: from='client.48709 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93444-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:15:44.120 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:43 smithi090 ceph-mon[30399]: from='client.48715 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi090-93342-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:15:44.120 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:43 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:15:44.120 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:43 smithi090 ceph-mon[30399]: from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]': finished 2024-04-14T15:15:44.120 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:43 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-31"}]: dispatch 2024-04-14T15:15:44.120 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:43 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1788001083' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi090-93342-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi090-93342-40"}]: dispatch 2024-04-14T15:15:44.120 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:43 smithi090 ceph-mon[30399]: osdmap e276: 8 total, 8 up, 8 in 2024-04-14T15:15:44.120 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:43 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-31"}]: dispatch 2024-04-14T15:15:44.120 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:43 smithi090 ceph-mon[30399]: from='client.48715 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi090-93342-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi090-93342-40"}]: dispatch 2024-04-14T15:15:44.120 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:43 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3688286685' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi090-93363-48"}]: dispatch 2024-04-14T15:15:44.120 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:43 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1438530465' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi090-93444-3","pool2":"test-rados-api-smithi090-93444-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-04-14T15:15:44.120 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:43 smithi090 ceph-mon[30399]: from='client.48709 ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi090-93444-3","pool2":"test-rados-api-smithi090-93444-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-04-14T15:15:44.202 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:15:44.226 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:44.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:44.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:15:44.328 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:15:44.352 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:44.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:44.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:15:44.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:43 smithi092 ceph-mon[33966]: 110.6 deep-scrub starts 2024-04-14T15:15:44.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:43 smithi092 ceph-mon[33966]: 110.6 deep-scrub ok 2024-04-14T15:15:44.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:43 smithi092 ceph-mon[33966]: 5.17 scrub starts 2024-04-14T15:15:44.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:43 smithi092 ceph-mon[33966]: 5.17 scrub ok 2024-04-14T15:15:44.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:43 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3396532154' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]: dispatch 2024-04-14T15:15:44.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:43 smithi092 ceph-mon[33966]: from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]: dispatch 2024-04-14T15:15:44.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:43 smithi092 ceph-mon[33966]: from='client.48709 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93444-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:15:44.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:43 smithi092 ceph-mon[33966]: from='client.48715 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi090-93342-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:15:44.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:43 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:15:44.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:43 smithi092 ceph-mon[33966]: from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]': finished 2024-04-14T15:15:44.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:43 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-31"}]: dispatch 2024-04-14T15:15:44.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:43 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1788001083' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi090-93342-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi090-93342-40"}]: dispatch 2024-04-14T15:15:44.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:43 smithi092 ceph-mon[33966]: osdmap e276: 8 total, 8 up, 8 in 2024-04-14T15:15:44.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:43 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-31"}]: dispatch 2024-04-14T15:15:44.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:43 smithi092 ceph-mon[33966]: from='client.48715 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi090-93342-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi090-93342-40"}]: dispatch 2024-04-14T15:15:44.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:43 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3688286685' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi090-93363-48"}]: dispatch 2024-04-14T15:15:44.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:43 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1438530465' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi090-93444-3","pool2":"test-rados-api-smithi090-93444-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-04-14T15:15:44.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:43 smithi092 ceph-mon[33966]: from='client.48709 ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi090-93444-3","pool2":"test-rados-api-smithi090-93444-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-04-14T15:15:44.454 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:15:44.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:44.478 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:44.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:15:44.579 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:15:44.595 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:15:44.603 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:44.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:44.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:15:44.705 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:15:44.729 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:44.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:44.730 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:15:44.831 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:15:44.856 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:44.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:44.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:15:44.873 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsStart 2024-04-14T15:15:44.873 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: 1 0 2024-04-14T15:15:44.873 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: 10 0 2024-04-14T15:15:44.873 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: 13 0 2024-04-14T15:15:44.874 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: 7 0 2024-04-14T15:15:44.874 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: 14 0 2024-04-14T15:15:44.874 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: 0 0 2024-04-14T15:15:44.874 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: 15 0 2024-04-14T15:15:44.874 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: 11 0 2024-04-14T15:15:44.875 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: 5 0 2024-04-14T15:15:44.875 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: 8 0 2024-04-14T15:15:44.875 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: 6 0 2024-04-14T15:15:44.875 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: 3 0 2024-04-14T15:15:44.875 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: 4 0 2024-04-14T15:15:44.875 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: 12 0 2024-04-14T15:15:44.875 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: 9 0 2024-04-14T15:15:44.876 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: 2 0 2024-04-14T15:15:44.876 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-04-14T15:15:44.876 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (51 ms) 2024-04-14T15:15:44.876 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: [----------] 3 tests from LibRadosListEC (1101 ms total) 2024-04-14T15:15:44.876 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: 2024-04-14T15:15:44.876 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: [----------] 1 test from LibRadosListNP 2024-04-14T15:15:44.876 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2024-04-14T15:15:44.877 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (3006 ms) 2024-04-14T15:15:44.877 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: [----------] 1 test from LibRadosListNP (3006 ms total) 2024-04-14T15:15:44.877 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: 2024-04-14T15:15:44.877 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: [----------] Global test environment tear-down 2024-04-14T15:15:44.877 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: [==========] 11 tests from 3 test suites ran. (170674 ms total) 2024-04-14T15:15:44.877 INFO:tasks.workunit.client.0.smithi090.stdout: api_list: [ PASSED ] 11 tests. 2024-04-14T15:15:44.959 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:15:44.982 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:44.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:44.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:15:45.084 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:15:45.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:45.109 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:45.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:15:45.203 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:44 smithi090 ceph-mon[30399]: pgmap v359: 278 pgs: 40 unknown, 32 creating+peering, 206 active+clean; 4.6 MiB data, 553 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:15:45.203 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:44 smithi090 ceph-mon[30399]: 2.c scrub starts 2024-04-14T15:15:45.204 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:44 smithi090 ceph-mon[30399]: 2.c scrub ok 2024-04-14T15:15:45.204 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:44 smithi090 ceph-mon[30399]: 11.1e deep-scrub starts 2024-04-14T15:15:45.204 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:44 smithi090 ceph-mon[30399]: 11.1e deep-scrub ok 2024-04-14T15:15:45.204 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:44 smithi090 ceph-mon[30399]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:15:45.204 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:44 smithi090 ceph-mon[30399]: 4.1a deep-scrub starts 2024-04-14T15:15:45.204 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:44 smithi090 ceph-mon[30399]: 4.1a deep-scrub ok 2024-04-14T15:15:45.204 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:44 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:15:45.205 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:44 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:15:45.205 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:44 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:15:45.205 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:44 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:15:45.205 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:44 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:15:45.205 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:44 smithi090 ceph-mon[30399]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:15:45.205 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:44 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-31"}]': finished 2024-04-14T15:15:45.205 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:44 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3688286685' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi090-93363-48"}]': finished 2024-04-14T15:15:45.206 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:44 smithi090 ceph-mon[30399]: from='client.48709 ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi090-93444-3","pool2":"test-rados-api-smithi090-93444-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-04-14T15:15:45.206 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:44 smithi090 ceph-mon[30399]: osdmap e277: 8 total, 8 up, 8 in 2024-04-14T15:15:45.206 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:44 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3688286685' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi090-93363-48"}]: dispatch 2024-04-14T15:15:45.206 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:44 smithi090 ceph-mon[41670]: pgmap v359: 278 pgs: 40 unknown, 32 creating+peering, 206 active+clean; 4.6 MiB data, 553 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:15:45.206 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:44 smithi090 ceph-mon[41670]: 2.c scrub starts 2024-04-14T15:15:45.206 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:44 smithi090 ceph-mon[41670]: 2.c scrub ok 2024-04-14T15:15:45.207 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:44 smithi090 ceph-mon[41670]: 11.1e deep-scrub starts 2024-04-14T15:15:45.207 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:44 smithi090 ceph-mon[41670]: 11.1e deep-scrub ok 2024-04-14T15:15:45.207 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:44 smithi090 ceph-mon[41670]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:15:45.207 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:44 smithi090 ceph-mon[41670]: 4.1a deep-scrub starts 2024-04-14T15:15:45.207 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:44 smithi090 ceph-mon[41670]: 4.1a deep-scrub ok 2024-04-14T15:15:45.207 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:44 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:15:45.207 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:44 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:15:45.207 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:44 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:15:45.207 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:44 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:15:45.207 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:44 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:15:45.208 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:44 smithi090 ceph-mon[41670]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:15:45.208 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:44 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-31"}]': finished 2024-04-14T15:15:45.208 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:44 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3688286685' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi090-93363-48"}]': finished 2024-04-14T15:15:45.208 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:44 smithi090 ceph-mon[41670]: from='client.48709 ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi090-93444-3","pool2":"test-rados-api-smithi090-93444-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-04-14T15:15:45.208 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:44 smithi090 ceph-mon[41670]: osdmap e277: 8 total, 8 up, 8 in 2024-04-14T15:15:45.208 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:44 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3688286685' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi090-93363-48"}]: dispatch 2024-04-14T15:15:45.210 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:15:45.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:45.236 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:45.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:15:45.314 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:15:45.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.315+0000 7f800a018700 1 -- 172.21.15.90:0/2521583158 >> v1:172.21.15.90:6789/0 conn(0x7f800410a6d0 legacy=0x7f800410cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:45.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.315+0000 7f800a018700 1 -- 172.21.15.90:0/2521583158 shutdown_connections 2024-04-14T15:15:45.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.315+0000 7f800a018700 1 -- 172.21.15.90:0/2521583158 >> 172.21.15.90:0/2521583158 conn(0x7f8004076aa0 msgr2=0x7f8004076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:15:45.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.315+0000 7f800a018700 1 -- 172.21.15.90:0/2521583158 shutdown_connections 2024-04-14T15:15:45.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.316+0000 7f800a018700 1 -- 172.21.15.90:0/2521583158 wait complete. 2024-04-14T15:15:45.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.317+0000 7f800a018700 1 librados: starting msgr at 2024-04-14T15:15:45.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.317+0000 7f800a018700 1 librados: starting objecter 2024-04-14T15:15:45.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.317+0000 7f800a018700 1 Processor -- start 2024-04-14T15:15:45.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.317+0000 7f800a018700 1 -- start start 2024-04-14T15:15:45.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.317+0000 7f800a018700 1 librados: setting wanted keys 2024-04-14T15:15:45.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.317+0000 7f800a018700 1 librados: calling monclient init 2024-04-14T15:15:45.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.317+0000 7f800a018700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f8004101dd0 con 0x7f80041023f0 2024-04-14T15:15:45.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.318+0000 7f800a018700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f80041a8e50 con 0x7f800410e270 2024-04-14T15:15:45.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.318+0000 7f80037fe700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f80041023f0 0x7f8004105e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:45844/0 (socket says 172.21.15.90:45844) 2024-04-14T15:15:45.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.318+0000 7f80037fe700 1 -- 172.21.15.90:0/18312551 learned_addr learned my addr 172.21.15.90:0/18312551 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:15:45.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.318+0000 7f8000ff9700 10 client.?.objecter ms_handle_connect 0x7f80041023f0 2024-04-14T15:15:45.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.318+0000 7f8000ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:15:45.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.318+0000 7f8000ff9700 1 -- 172.21.15.90:0/18312551 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3486197665 0 0) 0x7f8004101dd0 con 0x7f80041023f0 2024-04-14T15:15:45.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.318+0000 7f800a018700 1 -- 172.21.15.90:0/18312551 --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f80041a7e40 con 0x7f800410a6d0 2024-04-14T15:15:45.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.318+0000 7f8000ff9700 1 -- 172.21.15.90:0/18312551 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7fec0035b0 con 0x7f80041023f0 2024-04-14T15:15:45.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.318+0000 7f8000ff9700 10 client.?.objecter ms_handle_connect 0x7f800410a6d0 2024-04-14T15:15:45.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.318+0000 7f8000ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:15:45.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.318+0000 7f8000ff9700 10 client.?.objecter ms_handle_connect 0x7f800410e270 2024-04-14T15:15:45.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.318+0000 7f8000ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:15:45.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.318+0000 7f8000ff9700 1 -- 172.21.15.90:0/18312551 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2728043932 0 0) 0x7f80041a7e40 con 0x7f800410a6d0 2024-04-14T15:15:45.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.318+0000 7f8000ff9700 1 -- 172.21.15.90:0/18312551 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8004101dd0 con 0x7f800410a6d0 2024-04-14T15:15:45.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.318+0000 7f8000ff9700 1 -- 172.21.15.90:0/18312551 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1635925167 0 0) 0x7f80041a8e50 con 0x7f800410e270 2024-04-14T15:15:45.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.318+0000 7f8000ff9700 1 -- 172.21.15.90:0/18312551 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f80041a7e40 con 0x7f800410e270 2024-04-14T15:15:45.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.318+0000 7f8000ff9700 1 -- 172.21.15.90:0/18312551 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1755708352 0 0) 0x7f7fec0035b0 con 0x7f80041023f0 2024-04-14T15:15:45.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.319+0000 7f8000ff9700 1 -- 172.21.15.90:0/18312551 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f80041a8e50 con 0x7f80041023f0 2024-04-14T15:15:45.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.319+0000 7f8000ff9700 1 -- 172.21.15.90:0/18312551 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f7ff8001930 con 0x7f80041023f0 2024-04-14T15:15:45.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.319+0000 7f8000ff9700 1 -- 172.21.15.90:0/18312551 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3113241673 0 0) 0x7f80041a7e40 con 0x7f800410e270 2024-04-14T15:15:45.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.319+0000 7f8000ff9700 1 -- 172.21.15.90:0/18312551 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7fec0035b0 con 0x7f800410e270 2024-04-14T15:15:45.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.319+0000 7f8000ff9700 1 -- 172.21.15.90:0/18312551 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f7ffc0015d0 con 0x7f800410e270 2024-04-14T15:15:45.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.319+0000 7f8000ff9700 1 -- 172.21.15.90:0/18312551 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3022985638 0 0) 0x7f8004101dd0 con 0x7f800410a6d0 2024-04-14T15:15:45.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.319+0000 7f8000ff9700 1 -- 172.21.15.90:0/18312551 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f80041a7e40 con 0x7f800410a6d0 2024-04-14T15:15:45.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.319+0000 7f8000ff9700 1 -- 172.21.15.90:0/18312551 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f7ff40044b0 con 0x7f800410a6d0 2024-04-14T15:15:45.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.319+0000 7f8000ff9700 1 -- 172.21.15.90:0/18312551 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3126314673 0 0) 0x7f7fec0035b0 con 0x7f800410e270 2024-04-14T15:15:45.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.319+0000 7f8000ff9700 1 -- 172.21.15.90:0/18312551 >> v1:172.21.15.90:6790/0 conn(0x7f800410a6d0 legacy=0x7f8004106580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:45.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.319+0000 7f8000ff9700 1 -- 172.21.15.90:0/18312551 >> v1:172.21.15.90:6789/0 conn(0x7f80041023f0 legacy=0x7f8004105e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:45.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.319+0000 7f8000ff9700 1 -- 172.21.15.90:0/18312551 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80041a8010 con 0x7f800410e270 2024-04-14T15:15:45.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.319+0000 7f800a018700 1 -- 172.21.15.90:0/18312551 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f80041a8300 con 0x7f800410e270 2024-04-14T15:15:45.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.319+0000 7f8000ff9700 1 -- 172.21.15.90:0/18312551 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f7ffc003ea0 con 0x7f800410e270 2024-04-14T15:15:45.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.319+0000 7f8000ff9700 1 -- 172.21.15.90:0/18312551 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f7ffc004e00 con 0x7f800410e270 2024-04-14T15:15:45.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.319+0000 7f800a018700 10 client.48727.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:15:45.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.319+0000 7f800a018700 1 -- 172.21.15.90:0/18312551 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f80041a87c0 con 0x7f800410e270 2024-04-14T15:15:45.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.319+0000 7f800a018700 1 librados: init done 2024-04-14T15:15:45.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.320+0000 7f800a018700 1 -- 172.21.15.90:0/18312551 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f800406a170 con 0x7f800410e270 2024-04-14T15:15:45.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.321+0000 7f8000ff9700 1 -- 172.21.15.90:0/18312551 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f7ffc01bae0 con 0x7f800410e270 2024-04-14T15:15:45.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.321+0000 7f8000ff9700 1 -- 172.21.15.90:0/18312551 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(277..277 src has 51..277) ==== 8624+0+0 (unknown 2681495200 0 0) 0x7f7ffc089de0 con 0x7f800410e270 2024-04-14T15:15:45.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.321+0000 7f8000ff9700 10 client.48727.objecter ms_dispatch 0x7f8004001550 osd_map(277..277 src has 51..277) 2024-04-14T15:15:45.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.321+0000 7f8000ff9700 3 client.48727.objecter handle_osd_map got epochs [277,277] > 0 2024-04-14T15:15:45.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.321+0000 7f8000ff9700 3 client.48727.objecter handle_osd_map decoding full epoch 277 2024-04-14T15:15:45.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.321+0000 7f8000ff9700 20 client.48727.objecter dump_active .. 0 homeless 2024-04-14T15:15:45.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.322+0000 7f8000ff9700 10 client.48727.objecter ms_handle_connect 0x7f7fec071360 2024-04-14T15:15:45.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.326+0000 7f8000ff9700 1 -- 172.21.15.90:0/18312551 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f7ffc056620 con 0x7f800410e270 2024-04-14T15:15:45.336 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:15:45.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:45.361 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:45.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:15:45.372 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:15:44 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:15:45.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:44 smithi092 ceph-mon[33966]: pgmap v359: 278 pgs: 40 unknown, 32 creating+peering, 206 active+clean; 4.6 MiB data, 553 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:15:45.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:44 smithi092 ceph-mon[33966]: 2.c scrub starts 2024-04-14T15:15:45.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:44 smithi092 ceph-mon[33966]: 2.c scrub ok 2024-04-14T15:15:45.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:44 smithi092 ceph-mon[33966]: 11.1e deep-scrub starts 2024-04-14T15:15:45.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:44 smithi092 ceph-mon[33966]: 11.1e deep-scrub ok 2024-04-14T15:15:45.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:44 smithi092 ceph-mon[33966]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:15:45.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:44 smithi092 ceph-mon[33966]: 4.1a deep-scrub starts 2024-04-14T15:15:45.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:44 smithi092 ceph-mon[33966]: 4.1a deep-scrub ok 2024-04-14T15:15:45.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:44 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:15:45.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:44 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:15:45.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:44 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:15:45.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:44 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:15:45.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:44 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:15:45.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:44 smithi092 ceph-mon[33966]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:15:45.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:44 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-31"}]': finished 2024-04-14T15:15:45.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:44 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3688286685' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi090-93363-48"}]': finished 2024-04-14T15:15:45.376 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:44 smithi092 ceph-mon[33966]: from='client.48709 ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi090-93444-3","pool2":"test-rados-api-smithi090-93444-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-04-14T15:15:45.376 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:44 smithi092 ceph-mon[33966]: osdmap e277: 8 total, 8 up, 8 in 2024-04-14T15:15:45.376 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:44 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3688286685' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi090-93363-48"}]: dispatch 2024-04-14T15:15:45.463 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:15:45.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:45.488 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:45.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:15:45.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.560+0000 7f800a018700 1 -- 172.21.15.90:0/18312551 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7f80040693d0 con 0x7f800410e270 2024-04-14T15:15:45.589 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:15:45.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:45.614 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:45.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:15:45.716 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:15:45.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:45.741 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:45.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:15:45.842 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:15:45.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.854+0000 7f8000ff9700 1 -- 172.21.15.90:0/18312551 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v278) ==== 87+0+0 (unknown 111558484 0 0) 0x7f7ffc05a2d0 con 0x7f800410e270 2024-04-14T15:15:45.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.858+0000 7f800a018700 10 librados: watch_flush enter 2024-04-14T15:15:45.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.858+0000 7f800a018700 10 librados: watch_flush exit 2024-04-14T15:15:45.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.858+0000 7f800a018700 20 client.48727.objecter shutdown clearing up homeless session... 2024-04-14T15:15:45.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.858+0000 7f800a018700 10 client.48727.objecter successfully canceled tick 2024-04-14T15:15:45.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.861+0000 7f800a018700 1 -- 172.21.15.90:0/18312551 >> v1:172.21.15.90:6800/420935427 conn(0x7f7fec071360 legacy=0x7f7fec073820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:45.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.861+0000 7f800a018700 1 -- 172.21.15.90:0/18312551 >> v1:172.21.15.92:6789/0 conn(0x7f800410e270 legacy=0x7f80041a6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:45.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.866+0000 7f800a018700 1 -- 172.21.15.90:0/18312551 shutdown_connections 2024-04-14T15:15:45.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.866+0000 7f800a018700 1 -- 172.21.15.90:0/18312551 >> 172.21.15.90:0/18312551 conn(0x7f8004076aa0 msgr2=0x7f8004111a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:15:45.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.869+0000 7f800a018700 1 -- 172.21.15.90:0/18312551 shutdown_connections 2024-04-14T15:15:45.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.869+0000 7f800a018700 1 -- 172.21.15.90:0/18312551 wait complete. 2024-04-14T15:15:45.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:45.870+0000 7f800a018700 1 librados: shutdown 2024-04-14T15:15:45.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:45.874 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:45.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:15:45.874 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is set 2024-04-14T15:15:45.975 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:15:45.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:45.999 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:46.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:15:46.100 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:15:46.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:46.125 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:46.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:15:46.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:45 smithi090 ceph-mon[41670]: 4.1c scrub starts 2024-04-14T15:15:46.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:45 smithi090 ceph-mon[41670]: 4.1c scrub ok 2024-04-14T15:15:46.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:45 smithi090 ceph-mon[41670]: 4.1b scrub starts 2024-04-14T15:15:46.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:45 smithi090 ceph-mon[41670]: 4.1b scrub ok 2024-04-14T15:15:46.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:45 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:15:46.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:45 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:15:46.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:45 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:15:46.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:45 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:15:46.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:45 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:15:46.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:45 smithi090 ceph-mon[41670]: 1.1 scrub starts 2024-04-14T15:15:46.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:45 smithi090 ceph-mon[41670]: 1.1 scrub ok 2024-04-14T15:15:46.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:45 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:15:46.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:45 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:15:46.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:45 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:15:46.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:45 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/18312551' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:15:46.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:45 smithi090 ceph-mon[41670]: from='client.48727 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:15:46.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:45 smithi090 ceph-mon[41670]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:15:46.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:45 smithi090 ceph-mon[41670]: from='client.48715 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi090-93342-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi090-93342-40"}]': finished 2024-04-14T15:15:46.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:45 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3688286685' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi090-93363-48"}]': finished 2024-04-14T15:15:46.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:45 smithi090 ceph-mon[41670]: from='client.48727 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-14T15:15:46.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:45 smithi090 ceph-mon[41670]: osdmap e278: 8 total, 8 up, 8 in 2024-04-14T15:15:46.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:45 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2058889066' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi090-93363-49"}]: dispatch 2024-04-14T15:15:46.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:45 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2058889066' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi090-93363-49"}]: dispatch 2024-04-14T15:15:46.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:45 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2058889066' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi090-93363-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:46.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:45 smithi090 ceph-mon[30399]: 4.1c scrub starts 2024-04-14T15:15:46.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:45 smithi090 ceph-mon[30399]: 4.1c scrub ok 2024-04-14T15:15:46.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:45 smithi090 ceph-mon[30399]: 4.1b scrub starts 2024-04-14T15:15:46.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:45 smithi090 ceph-mon[30399]: 4.1b scrub ok 2024-04-14T15:15:46.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:45 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:15:46.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:45 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:15:46.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:45 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:15:46.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:45 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:15:46.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:45 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:15:46.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:45 smithi090 ceph-mon[30399]: 1.1 scrub starts 2024-04-14T15:15:46.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:45 smithi090 ceph-mon[30399]: 1.1 scrub ok 2024-04-14T15:15:46.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:45 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:15:46.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:45 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:15:46.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:45 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:15:46.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:45 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/18312551' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:15:46.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:45 smithi090 ceph-mon[30399]: from='client.48727 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:15:46.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:45 smithi090 ceph-mon[30399]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:15:46.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:45 smithi090 ceph-mon[30399]: from='client.48715 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi090-93342-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi090-93342-40"}]': finished 2024-04-14T15:15:46.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:45 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3688286685' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi090-93363-48"}]': finished 2024-04-14T15:15:46.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:45 smithi090 ceph-mon[30399]: from='client.48727 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-14T15:15:46.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:45 smithi090 ceph-mon[30399]: osdmap e278: 8 total, 8 up, 8 in 2024-04-14T15:15:46.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:45 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2058889066' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi090-93363-49"}]: dispatch 2024-04-14T15:15:46.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:45 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2058889066' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi090-93363-49"}]: dispatch 2024-04-14T15:15:46.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:45 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2058889066' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi090-93363-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:46.227 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:15:46.251 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:46.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:46.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:15:46.352 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:15:46.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:45 smithi092 ceph-mon[33966]: 4.1c scrub starts 2024-04-14T15:15:46.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:45 smithi092 ceph-mon[33966]: 4.1c scrub ok 2024-04-14T15:15:46.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:45 smithi092 ceph-mon[33966]: 4.1b scrub starts 2024-04-14T15:15:46.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:45 smithi092 ceph-mon[33966]: 4.1b scrub ok 2024-04-14T15:15:46.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:45 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:15:46.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:45 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:15:46.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:45 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:15:46.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:45 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:15:46.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:45 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:15:46.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:45 smithi092 ceph-mon[33966]: 1.1 scrub starts 2024-04-14T15:15:46.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:45 smithi092 ceph-mon[33966]: 1.1 scrub ok 2024-04-14T15:15:46.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:45 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:15:46.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:45 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:15:46.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:45 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:15:46.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:45 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/18312551' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:15:46.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:45 smithi092 ceph-mon[33966]: from='client.48727 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:15:46.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:45 smithi092 ceph-mon[33966]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:15:46.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:45 smithi092 ceph-mon[33966]: from='client.48715 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi090-93342-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi090-93342-40"}]': finished 2024-04-14T15:15:46.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:45 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3688286685' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi090-93363-48"}]': finished 2024-04-14T15:15:46.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:45 smithi092 ceph-mon[33966]: from='client.48727 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-14T15:15:46.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:45 smithi092 ceph-mon[33966]: osdmap e278: 8 total, 8 up, 8 in 2024-04-14T15:15:46.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:45 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2058889066' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi090-93363-49"}]: dispatch 2024-04-14T15:15:46.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:45 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2058889066' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi090-93363-49"}]: dispatch 2024-04-14T15:15:46.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:45 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2058889066' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi090-93363-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:46.376 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:46.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:46.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:15:46.477 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:15:46.500 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:46.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:46.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:15:46.602 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:15:46.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:46.626 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:46.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:15:46.727 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:15:46.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:46.753 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:46.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:15:46.854 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:15:46.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:46.888 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:46.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:15:46.989 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:15:47.013 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:47.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:47.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:15:47.115 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:15:47.138 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:47.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:47.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:15:47.240 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:15:47.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:47.265 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:47.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:15:47.355 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:46 smithi090 ceph-mon[30399]: pgmap v362: 238 pgs: 27 peering, 211 active+clean; 4.5 MiB data, 558 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 165 KiB/s wr, 29 op/s; 118 KiB/s, 0 objects/s recovering 2024-04-14T15:15:47.355 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:46 smithi090 ceph-mon[30399]: 1.a scrub starts 2024-04-14T15:15:47.355 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:46 smithi090 ceph-mon[30399]: 1.a scrub ok 2024-04-14T15:15:47.355 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:46 smithi090 ceph-mon[30399]: Health check failed: Reduced data availability: 19 pgs inactive, 24 pgs peering (PG_AVAILABILITY) 2024-04-14T15:15:47.355 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:46 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2058889066' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi090-93363-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:15:47.355 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:46 smithi090 ceph-mon[30399]: osdmap e279: 8 total, 8 up, 8 in 2024-04-14T15:15:47.355 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:46 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2058889066' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi090-93363-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi090-93363-49"}]: dispatch 2024-04-14T15:15:47.356 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:46 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:47.356 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:46 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:47.356 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:46 smithi090 ceph-mon[41670]: pgmap v362: 238 pgs: 27 peering, 211 active+clean; 4.5 MiB data, 558 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 165 KiB/s wr, 29 op/s; 118 KiB/s, 0 objects/s recovering 2024-04-14T15:15:47.356 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:46 smithi090 ceph-mon[41670]: 1.a scrub starts 2024-04-14T15:15:47.356 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:46 smithi090 ceph-mon[41670]: 1.a scrub ok 2024-04-14T15:15:47.356 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:46 smithi090 ceph-mon[41670]: Health check failed: Reduced data availability: 19 pgs inactive, 24 pgs peering (PG_AVAILABILITY) 2024-04-14T15:15:47.356 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:46 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2058889066' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi090-93363-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:15:47.356 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:46 smithi090 ceph-mon[41670]: osdmap e279: 8 total, 8 up, 8 in 2024-04-14T15:15:47.356 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:46 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2058889066' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi090-93363-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi090-93363-49"}]: dispatch 2024-04-14T15:15:47.356 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:46 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:47.356 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:46 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:47.366 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:15:47.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:46 smithi092 ceph-mon[33966]: pgmap v362: 238 pgs: 27 peering, 211 active+clean; 4.5 MiB data, 558 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 165 KiB/s wr, 29 op/s; 118 KiB/s, 0 objects/s recovering 2024-04-14T15:15:47.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:46 smithi092 ceph-mon[33966]: 1.a scrub starts 2024-04-14T15:15:47.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:46 smithi092 ceph-mon[33966]: 1.a scrub ok 2024-04-14T15:15:47.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:46 smithi092 ceph-mon[33966]: Health check failed: Reduced data availability: 19 pgs inactive, 24 pgs peering (PG_AVAILABILITY) 2024-04-14T15:15:47.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:46 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2058889066' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi090-93363-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:15:47.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:46 smithi092 ceph-mon[33966]: osdmap e279: 8 total, 8 up, 8 in 2024-04-14T15:15:47.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:46 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2058889066' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi090-93363-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi090-93363-49"}]: dispatch 2024-04-14T15:15:47.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:46 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:47.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:46 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:47.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:47.390 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:47.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:15:47.491 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:15:47.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:47.517 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:47.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:15:47.618 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:15:47.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:47.643 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:47.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:15:47.744 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:15:47.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:47.770 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:47.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:15:47.872 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:15:47.939 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:47.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:47.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:15:48.040 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:15:48.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:48.065 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:48.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:15:48.166 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:15:48.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:48.190 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:48.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:15:48.291 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:15:48.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:48.316 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:48.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:15:48.417 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:15:48.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:48.442 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:48.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:15:48.543 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:15:48.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:48.568 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:48.568 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:15:48.669 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:15:48.694 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:48.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:48.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:15:48.796 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:15:48.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:48.822 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:48.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:15:48.923 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:15:48.941 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:48 smithi090 ceph-mon[41670]: pgmap v365: 246 pgs: 40 unknown, 24 peering, 182 active+clean; 4.5 MiB data, 558 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 165 KiB/s wr, 27 op/s; 118 KiB/s, 0 objects/s recovering 2024-04-14T15:15:48.941 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:48 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:15:48.941 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:48 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:15:48.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:48 smithi090 ceph-mon[41670]: osdmap e280: 8 total, 8 up, 8 in 2024-04-14T15:15:48.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:48 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:15:48.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:48 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1788001083' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi090-93342-40"}]: dispatch 2024-04-14T15:15:48.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:48 smithi090 ceph-mon[41670]: from='client.48715 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi090-93342-40"}]: dispatch 2024-04-14T15:15:48.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:48 smithi090 ceph-mon[30399]: pgmap v365: 246 pgs: 40 unknown, 24 peering, 182 active+clean; 4.5 MiB data, 558 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 165 KiB/s wr, 27 op/s; 118 KiB/s, 0 objects/s recovering 2024-04-14T15:15:48.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:48 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:15:48.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:48 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:15:48.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:48 smithi090 ceph-mon[30399]: osdmap e280: 8 total, 8 up, 8 in 2024-04-14T15:15:48.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:48 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:15:48.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:48 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1788001083' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi090-93342-40"}]: dispatch 2024-04-14T15:15:48.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:48 smithi090 ceph-mon[30399]: from='client.48715 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi090-93342-40"}]: dispatch 2024-04-14T15:15:48.948 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:48.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:48.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:15:48.983 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-14T15:15:49.050 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:15:49.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:49.076 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:49.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:15:49.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:48 smithi092 ceph-mon[33966]: pgmap v365: 246 pgs: 40 unknown, 24 peering, 182 active+clean; 4.5 MiB data, 558 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 165 KiB/s wr, 27 op/s; 118 KiB/s, 0 objects/s recovering 2024-04-14T15:15:49.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:48 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:15:49.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:48 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:15:49.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:48 smithi092 ceph-mon[33966]: osdmap e280: 8 total, 8 up, 8 in 2024-04-14T15:15:49.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:48 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:15:49.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:48 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1788001083' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi090-93342-40"}]: dispatch 2024-04-14T15:15:49.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:48 smithi092 ceph-mon[33966]: from='client.48715 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi090-93342-40"}]: dispatch 2024-04-14T15:15:49.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:48.996+0000 7f2bfe7a5700 1 -- 172.21.15.90:0/3010673422 >> v1:172.21.15.90:6789/0 conn(0x7f2bf8103a20 legacy=0x7f2bf8103e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:49.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:48.996+0000 7f2bfe7a5700 1 -- 172.21.15.90:0/3010673422 shutdown_connections 2024-04-14T15:15:49.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:48.996+0000 7f2bfe7a5700 1 -- 172.21.15.90:0/3010673422 >> 172.21.15.90:0/3010673422 conn(0x7f2bf80fd290 msgr2=0x7f2bf80ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:15:49.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:48.996+0000 7f2bfe7a5700 1 -- 172.21.15.90:0/3010673422 shutdown_connections 2024-04-14T15:15:49.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:48.996+0000 7f2bfe7a5700 1 -- 172.21.15.90:0/3010673422 wait complete. 2024-04-14T15:15:49.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:48.997+0000 7f2bfe7a5700 1 librados: starting msgr at 2024-04-14T15:15:49.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:48.997+0000 7f2bfe7a5700 1 librados: starting objecter 2024-04-14T15:15:49.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:48.997+0000 7f2bfe7a5700 1 Processor -- start 2024-04-14T15:15:49.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:48.997+0000 7f2bfe7a5700 1 -- start start 2024-04-14T15:15:49.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:48.997+0000 7f2bfe7a5700 1 librados: setting wanted keys 2024-04-14T15:15:49.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:48.997+0000 7f2bfe7a5700 1 librados: calling monclient init 2024-04-14T15:15:49.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:48.997+0000 7f2bfe7a5700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f2bf81a78e0 con 0x7f2bf8103a20 2024-04-14T15:15:49.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:48.998+0000 7f2bfe7a5700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f2bf81a8b00 con 0x7f2bf810b830 2024-04-14T15:15:49.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:48.998+0000 7f2bfe7a5700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f2bf81a9d40 con 0x7f2bf810f340 2024-04-14T15:15:49.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:48.998+0000 7f2bf7fff700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f2bf8103a20 0x7f2bf810b000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:45886/0 (socket says 172.21.15.90:45886) 2024-04-14T15:15:49.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:48.998+0000 7f2bf7fff700 1 -- 172.21.15.90:0/4219049636 learned_addr learned my addr 172.21.15.90:0/4219049636 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:15:49.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:48.998+0000 7f2bf57fa700 10 client.?.objecter ms_handle_connect 0x7f2bf8103a20 2024-04-14T15:15:49.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:48.998+0000 7f2bf57fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:15:49.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:48.998+0000 7f2bf57fa700 10 client.?.objecter ms_handle_connect 0x7f2bf810b830 2024-04-14T15:15:49.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:48.998+0000 7f2bf57fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:15:49.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:48.998+0000 7f2bf57fa700 1 -- 172.21.15.90:0/4219049636 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1771719340 0 0) 0x7f2bf81a8b00 con 0x7f2bf810b830 2024-04-14T15:15:49.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:48.998+0000 7f2bf57fa700 1 -- 172.21.15.90:0/4219049636 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2be00035b0 con 0x7f2bf810b830 2024-04-14T15:15:49.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:48.998+0000 7f2bf57fa700 10 client.?.objecter ms_handle_connect 0x7f2bf810f340 2024-04-14T15:15:49.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:48.998+0000 7f2bf57fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:15:49.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:48.998+0000 7f2bf57fa700 1 -- 172.21.15.90:0/4219049636 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3728604849 0 0) 0x7f2bf81a78e0 con 0x7f2bf8103a20 2024-04-14T15:15:49.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:48.998+0000 7f2bf57fa700 1 -- 172.21.15.90:0/4219049636 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2bf81a8b00 con 0x7f2bf8103a20 2024-04-14T15:15:49.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:48.998+0000 7f2bf57fa700 1 -- 172.21.15.90:0/4219049636 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3606473778 0 0) 0x7f2bf81a9d40 con 0x7f2bf810f340 2024-04-14T15:15:49.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:48.998+0000 7f2bf57fa700 1 -- 172.21.15.90:0/4219049636 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2bf81a78e0 con 0x7f2bf810f340 2024-04-14T15:15:49.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:48.998+0000 7f2bf57fa700 1 -- 172.21.15.90:0/4219049636 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3364779446 0 0) 0x7f2be00035b0 con 0x7f2bf810b830 2024-04-14T15:15:49.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:48.998+0000 7f2bf57fa700 1 -- 172.21.15.90:0/4219049636 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2bf81a9d40 con 0x7f2bf810b830 2024-04-14T15:15:49.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:48.998+0000 7f2bf57fa700 1 -- 172.21.15.90:0/4219049636 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f2bec0018b0 con 0x7f2bf810b830 2024-04-14T15:15:49.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:48.998+0000 7f2bf57fa700 1 -- 172.21.15.90:0/4219049636 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 895484728 0 0) 0x7f2bf81a8b00 con 0x7f2bf8103a20 2024-04-14T15:15:49.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:48.998+0000 7f2bf57fa700 1 -- 172.21.15.90:0/4219049636 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2be00035b0 con 0x7f2bf8103a20 2024-04-14T15:15:49.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:48.998+0000 7f2bf57fa700 1 -- 172.21.15.90:0/4219049636 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f2be8004550 con 0x7f2bf8103a20 2024-04-14T15:15:49.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:48.998+0000 7f2bf57fa700 1 -- 172.21.15.90:0/4219049636 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2689639877 0 0) 0x7f2bf81a78e0 con 0x7f2bf810f340 2024-04-14T15:15:49.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:48.999+0000 7f2bf57fa700 1 -- 172.21.15.90:0/4219049636 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2bf81a8b00 con 0x7f2bf810f340 2024-04-14T15:15:49.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:48.999+0000 7f2bf57fa700 1 -- 172.21.15.90:0/4219049636 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f2bf0001b70 con 0x7f2bf810f340 2024-04-14T15:15:49.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:48.999+0000 7f2bf57fa700 1 -- 172.21.15.90:0/4219049636 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3405688089 0 0) 0x7f2be00035b0 con 0x7f2bf8103a20 2024-04-14T15:15:49.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:48.999+0000 7f2bf57fa700 1 -- 172.21.15.90:0/4219049636 >> v1:172.21.15.90:6790/0 conn(0x7f2bf810f340 legacy=0x7f2bf81a5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:49.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:48.999+0000 7f2bf57fa700 1 -- 172.21.15.90:0/4219049636 >> v1:172.21.15.92:6789/0 conn(0x7f2bf810b830 legacy=0x7f2bf81a2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:49.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:48.999+0000 7f2bf57fa700 1 -- 172.21.15.90:0/4219049636 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bf81aaf80 con 0x7f2bf8103a20 2024-04-14T15:15:49.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:48.999+0000 7f2bfe7a5700 1 -- 172.21.15.90:0/4219049636 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2bf81a8cd0 con 0x7f2bf8103a20 2024-04-14T15:15:49.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:48.999+0000 7f2bfe7a5700 10 client.36900.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:15:49.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:48.999+0000 7f2bfe7a5700 1 -- 172.21.15.90:0/4219049636 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f2bf81a9190 con 0x7f2bf8103a20 2024-04-14T15:15:49.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:48.999+0000 7f2bfe7a5700 1 librados: init done 2024-04-14T15:15:49.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:48.999+0000 7f2bf57fa700 1 -- 172.21.15.90:0/4219049636 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f2be8001ae0 con 0x7f2bf8103a20 2024-04-14T15:15:49.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:48.999+0000 7f2bf57fa700 1 -- 172.21.15.90:0/4219049636 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f2be8004fd0 con 0x7f2bf8103a20 2024-04-14T15:15:49.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.000+0000 7f2bfe7a5700 1 -- 172.21.15.90:0/4219049636 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f2be4005100 con 0x7f2bf8103a20 2024-04-14T15:15:49.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.000+0000 7f2bf57fa700 1 -- 172.21.15.90:0/4219049636 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f2be8004fd0 con 0x7f2bf8103a20 2024-04-14T15:15:49.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.001+0000 7f2bf57fa700 1 -- 172.21.15.90:0/4219049636 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(281..281 src has 51..281) ==== 9014+0+0 (unknown 3801588390 0 0) 0x7f2be808ae40 con 0x7f2bf8103a20 2024-04-14T15:15:49.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.001+0000 7f2bf57fa700 10 client.36900.objecter ms_dispatch 0x7f2bf8001550 osd_map(281..281 src has 51..281) 2024-04-14T15:15:49.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.001+0000 7f2bf57fa700 3 client.36900.objecter handle_osd_map got epochs [281,281] > 0 2024-04-14T15:15:49.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.001+0000 7f2bf57fa700 3 client.36900.objecter handle_osd_map decoding full epoch 281 2024-04-14T15:15:49.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.001+0000 7f2bf57fa700 20 client.36900.objecter dump_active .. 0 homeless 2024-04-14T15:15:49.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.001+0000 7f2bf57fa700 1 -- 172.21.15.90:0/4219049636 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v281) ==== 54+0+16 (unknown 1135329989 0 2420000192) 0x7f2be808d2a0 con 0x7f2bf8103a20 2024-04-14T15:15:49.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.001+0000 7f2bfe7a5700 10 client.36900.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:15:49.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.001+0000 7f2bfe7a5700 15 client.36900.objecter _session_command_op_assign -1 1 2024-04-14T15:15:49.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.002+0000 7f2bfe7a5700 20 client.36900.objecter _get_session s=0x7f2be40053e0 osd=0 3 2024-04-14T15:15:49.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.002+0000 7f2bfe7a5700 20 client.36900.objecter put_session s=0x7f2be40053e0 osd=0 3 2024-04-14T15:15:49.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.002+0000 7f2bfe7a5700 20 client.36900.objecter _get_session s=0x7f2be40053e0 osd=0 3 2024-04-14T15:15:49.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.002+0000 7f2bfe7a5700 15 client.36900.objecter _session_command_op_remove -1 1 2024-04-14T15:15:49.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.002+0000 7f2bfe7a5700 20 client.36900.objecter get_session s=0x7f2be40053e0 osd=0 3 2024-04-14T15:15:49.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.002+0000 7f2bfe7a5700 15 client.36900.objecter _session_command_op_assign 0 1 2024-04-14T15:15:49.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.002+0000 7f2bfe7a5700 20 client.36900.objecter put_session s=0x7f2be40053e0 osd=0 4 2024-04-14T15:15:49.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.002+0000 7f2bfe7a5700 10 client.36900.objecter _send_command 1 2024-04-14T15:15:49.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.002+0000 7f2bfe7a5700 1 -- 172.21.15.90:0/4219049636 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2be4004b50 con 0x7f2be4004730 2024-04-14T15:15:49.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.002+0000 7f2bf57fa700 10 client.36900.objecter ms_handle_connect 0x7f2be00757f0 2024-04-14T15:15:49.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.002+0000 7f2bf57fa700 10 client.36900.objecter ms_handle_connect 0x7f2be4004730 2024-04-14T15:15:49.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.003+0000 7f2bf57fa700 1 -- 172.21.15.90:0/4219049636 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2be4004b50 con 0x7f2be4004730 2024-04-14T15:15:49.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.003+0000 7f2bf57fa700 10 client.36900.objecter ms_dispatch 0x7f2bf8001550 command_reply(tid 1: 0 ) 2024-04-14T15:15:49.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.003+0000 7f2bf57fa700 10 client.36900.objecter _finish_command 1 = system:0 2024-04-14T15:15:49.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.003+0000 7f2bf57fa700 20 client.36900.objecter put_session s=0x7f2be40053e0 osd=0 4 2024-04-14T15:15:49.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.003+0000 7f2bf57fa700 15 client.36900.objecter _session_command_op_remove 0 1 2024-04-14T15:15:49.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.020+0000 7f2bfe7a5700 10 client.36900.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:15:49.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.020+0000 7f2bfe7a5700 15 client.36900.objecter _session_command_op_assign -1 2 2024-04-14T15:15:49.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.020+0000 7f2bfe7a5700 20 client.36900.objecter _get_session s=0x7f2be40053e0 osd=0 3 2024-04-14T15:15:49.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.020+0000 7f2bfe7a5700 20 client.36900.objecter put_session s=0x7f2be40053e0 osd=0 3 2024-04-14T15:15:49.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.020+0000 7f2bfe7a5700 20 client.36900.objecter _get_session s=0x7f2be40053e0 osd=0 3 2024-04-14T15:15:49.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.020+0000 7f2bfe7a5700 15 client.36900.objecter _session_command_op_remove -1 2 2024-04-14T15:15:49.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.020+0000 7f2bfe7a5700 20 client.36900.objecter get_session s=0x7f2be40053e0 osd=0 3 2024-04-14T15:15:49.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.020+0000 7f2bfe7a5700 15 client.36900.objecter _session_command_op_assign 0 2 2024-04-14T15:15:49.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.020+0000 7f2bfe7a5700 20 client.36900.objecter put_session s=0x7f2be40053e0 osd=0 4 2024-04-14T15:15:49.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.020+0000 7f2bfe7a5700 10 client.36900.objecter _send_command 2 2024-04-14T15:15:49.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.020+0000 7f2bfe7a5700 1 -- 172.21.15.90:0/4219049636 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f2be4007b00 con 0x7f2be4004730 2024-04-14T15:15:49.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.022+0000 7f2bf57fa700 1 -- 172.21.15.90:0/4219049636 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2be4007b00 con 0x7f2be4004730 2024-04-14T15:15:49.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.022+0000 7f2bf57fa700 10 client.36900.objecter ms_dispatch 0x7f2bf8001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:15:49.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.022+0000 7f2bf57fa700 10 client.36900.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:15:49.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.022+0000 7f2bf57fa700 20 client.36900.objecter put_session s=0x7f2be40053e0 osd=0 4 2024-04-14T15:15:49.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.022+0000 7f2bf57fa700 15 client.36900.objecter _session_command_op_remove 0 2 2024-04-14T15:15:49.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.023+0000 7f2bfe7a5700 10 client.36900.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:15:49.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.023+0000 7f2bfe7a5700 15 client.36900.objecter _session_command_op_assign -1 3 2024-04-14T15:15:49.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.023+0000 7f2bfe7a5700 20 client.36900.objecter _get_session s=0x7f2be4004130 osd=1 3 2024-04-14T15:15:49.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.023+0000 7f2bfe7a5700 20 client.36900.objecter put_session s=0x7f2be4004130 osd=1 3 2024-04-14T15:15:49.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.023+0000 7f2bfe7a5700 20 client.36900.objecter _get_session s=0x7f2be4004130 osd=1 3 2024-04-14T15:15:49.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.023+0000 7f2bfe7a5700 15 client.36900.objecter _session_command_op_remove -1 3 2024-04-14T15:15:49.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.023+0000 7f2bfe7a5700 20 client.36900.objecter get_session s=0x7f2be4004130 osd=1 3 2024-04-14T15:15:49.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.023+0000 7f2bfe7a5700 15 client.36900.objecter _session_command_op_assign 1 3 2024-04-14T15:15:49.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.023+0000 7f2bfe7a5700 20 client.36900.objecter put_session s=0x7f2be4004130 osd=1 4 2024-04-14T15:15:49.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.023+0000 7f2bfe7a5700 10 client.36900.objecter _send_command 3 2024-04-14T15:15:49.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.023+0000 7f2bfe7a5700 1 -- 172.21.15.90:0/4219049636 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f2be4002160 con 0x7f2be40042a0 2024-04-14T15:15:49.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.023+0000 7f2bf57fa700 10 client.36900.objecter ms_handle_connect 0x7f2be40042a0 2024-04-14T15:15:49.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.024+0000 7f2bf57fa700 1 -- 172.21.15.90:0/4219049636 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2be4002160 con 0x7f2be40042a0 2024-04-14T15:15:49.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.024+0000 7f2bf57fa700 10 client.36900.objecter ms_dispatch 0x7f2bf8001550 command_reply(tid 3: 0 ) 2024-04-14T15:15:49.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.024+0000 7f2bf57fa700 10 client.36900.objecter _finish_command 3 = system:0 2024-04-14T15:15:49.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.024+0000 7f2bf57fa700 20 client.36900.objecter put_session s=0x7f2be4004130 osd=1 4 2024-04-14T15:15:49.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.024+0000 7f2bf57fa700 15 client.36900.objecter _session_command_op_remove 1 3 2024-04-14T15:15:49.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.042+0000 7f2bfe7a5700 10 client.36900.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:15:49.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.042+0000 7f2bfe7a5700 15 client.36900.objecter _session_command_op_assign -1 4 2024-04-14T15:15:49.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.042+0000 7f2bfe7a5700 20 client.36900.objecter _get_session s=0x7f2be4004130 osd=1 3 2024-04-14T15:15:49.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.042+0000 7f2bfe7a5700 20 client.36900.objecter put_session s=0x7f2be4004130 osd=1 3 2024-04-14T15:15:49.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.042+0000 7f2bfe7a5700 20 client.36900.objecter _get_session s=0x7f2be4004130 osd=1 3 2024-04-14T15:15:49.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.042+0000 7f2bfe7a5700 15 client.36900.objecter _session_command_op_remove -1 4 2024-04-14T15:15:49.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.042+0000 7f2bfe7a5700 20 client.36900.objecter get_session s=0x7f2be4004130 osd=1 3 2024-04-14T15:15:49.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.042+0000 7f2bfe7a5700 15 client.36900.objecter _session_command_op_assign 1 4 2024-04-14T15:15:49.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.042+0000 7f2bfe7a5700 20 client.36900.objecter put_session s=0x7f2be4004130 osd=1 4 2024-04-14T15:15:49.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.042+0000 7f2bfe7a5700 10 client.36900.objecter _send_command 4 2024-04-14T15:15:49.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.042+0000 7f2bfe7a5700 1 -- 172.21.15.90:0/4219049636 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f2be4001600 con 0x7f2be40042a0 2024-04-14T15:15:49.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.044+0000 7f2bf57fa700 1 -- 172.21.15.90:0/4219049636 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2be4001600 con 0x7f2be40042a0 2024-04-14T15:15:49.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.044+0000 7f2bf57fa700 10 client.36900.objecter ms_dispatch 0x7f2bf8001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:15:49.157 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:15:49.158 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:15:49.158 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:15:49.158 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:15:49.158 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:15:49.158 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:15:49.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.044+0000 7f2bf57fa700 10 client.36900.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:15:49.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.044+0000 7f2bf57fa700 20 client.36900.objecter put_session s=0x7f2be4004130 osd=1 4 2024-04-14T15:15:49.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.044+0000 7f2bf57fa700 15 client.36900.objecter _session_command_op_remove 1 4 2024-04-14T15:15:49.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.044+0000 7f2bfe7a5700 10 client.36900.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:15:49.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.044+0000 7f2bfe7a5700 15 client.36900.objecter _session_command_op_assign -1 5 2024-04-14T15:15:49.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.044+0000 7f2bfe7a5700 20 client.36900.objecter _get_session s=0x7f2be4001a00 osd=2 3 2024-04-14T15:15:49.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.044+0000 7f2bfe7a5700 20 client.36900.objecter put_session s=0x7f2be4001a00 osd=2 3 2024-04-14T15:15:49.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.044+0000 7f2bfe7a5700 20 client.36900.objecter _get_session s=0x7f2be4001a00 osd=2 3 2024-04-14T15:15:49.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.044+0000 7f2bfe7a5700 15 client.36900.objecter _session_command_op_remove -1 5 2024-04-14T15:15:49.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.044+0000 7f2bfe7a5700 20 client.36900.objecter get_session s=0x7f2be4001a00 osd=2 3 2024-04-14T15:15:49.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.044+0000 7f2bfe7a5700 15 client.36900.objecter _session_command_op_assign 2 5 2024-04-14T15:15:49.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.044+0000 7f2bfe7a5700 20 client.36900.objecter put_session s=0x7f2be4001a00 osd=2 4 2024-04-14T15:15:49.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.044+0000 7f2bfe7a5700 10 client.36900.objecter _send_command 5 2024-04-14T15:15:49.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.044+0000 7f2bfe7a5700 1 -- 172.21.15.90:0/4219049636 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f2be4010fb0 con 0x7f2be4008990 2024-04-14T15:15:49.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.045+0000 7f2bf57fa700 10 client.36900.objecter ms_handle_connect 0x7f2be4008990 2024-04-14T15:15:49.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.045+0000 7f2bf57fa700 1 -- 172.21.15.90:0/4219049636 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2be4010fb0 con 0x7f2be4008990 2024-04-14T15:15:49.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.045+0000 7f2bf57fa700 10 client.36900.objecter ms_dispatch 0x7f2bf8001550 command_reply(tid 5: 0 ) 2024-04-14T15:15:49.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.045+0000 7f2bf57fa700 10 client.36900.objecter _finish_command 5 = system:0 2024-04-14T15:15:49.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.045+0000 7f2bf57fa700 20 client.36900.objecter put_session s=0x7f2be4001a00 osd=2 4 2024-04-14T15:15:49.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.045+0000 7f2bf57fa700 15 client.36900.objecter _session_command_op_remove 2 5 2024-04-14T15:15:49.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.062+0000 7f2bfe7a5700 10 client.36900.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:15:49.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.062+0000 7f2bfe7a5700 15 client.36900.objecter _session_command_op_assign -1 6 2024-04-14T15:15:49.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.062+0000 7f2bfe7a5700 20 client.36900.objecter _get_session s=0x7f2be4001a00 osd=2 3 2024-04-14T15:15:49.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.062+0000 7f2bfe7a5700 20 client.36900.objecter put_session s=0x7f2be4001a00 osd=2 3 2024-04-14T15:15:49.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.062+0000 7f2bfe7a5700 20 client.36900.objecter _get_session s=0x7f2be4001a00 osd=2 3 2024-04-14T15:15:49.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.062+0000 7f2bfe7a5700 15 client.36900.objecter _session_command_op_remove -1 6 2024-04-14T15:15:49.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.062+0000 7f2bfe7a5700 20 client.36900.objecter get_session s=0x7f2be4001a00 osd=2 3 2024-04-14T15:15:49.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.062+0000 7f2bfe7a5700 15 client.36900.objecter _session_command_op_assign 2 6 2024-04-14T15:15:49.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.062+0000 7f2bfe7a5700 20 client.36900.objecter put_session s=0x7f2be4001a00 osd=2 4 2024-04-14T15:15:49.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.062+0000 7f2bfe7a5700 10 client.36900.objecter _send_command 6 2024-04-14T15:15:49.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.062+0000 7f2bfe7a5700 1 -- 172.21.15.90:0/4219049636 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f2be4008020 con 0x7f2be4008990 2024-04-14T15:15:49.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.064+0000 7f2bf57fa700 1 -- 172.21.15.90:0/4219049636 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2be4008020 con 0x7f2be4008990 2024-04-14T15:15:49.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.064+0000 7f2bf57fa700 10 client.36900.objecter ms_dispatch 0x7f2bf8001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:15:49.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.064+0000 7f2bf57fa700 10 client.36900.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:15:49.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.064+0000 7f2bf57fa700 20 client.36900.objecter put_session s=0x7f2be4001a00 osd=2 4 2024-04-14T15:15:49.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.064+0000 7f2bf57fa700 15 client.36900.objecter _session_command_op_remove 2 6 2024-04-14T15:15:49.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.065+0000 7f2bfe7a5700 10 client.36900.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:15:49.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.065+0000 7f2bfe7a5700 15 client.36900.objecter _session_command_op_assign -1 7 2024-04-14T15:15:49.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.065+0000 7f2bfe7a5700 20 client.36900.objecter _get_session s=0x7f2be40082f0 osd=3 3 2024-04-14T15:15:49.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.065+0000 7f2bfe7a5700 20 client.36900.objecter put_session s=0x7f2be40082f0 osd=3 3 2024-04-14T15:15:49.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.065+0000 7f2bfe7a5700 20 client.36900.objecter _get_session s=0x7f2be40082f0 osd=3 3 2024-04-14T15:15:49.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.065+0000 7f2bfe7a5700 15 client.36900.objecter _session_command_op_remove -1 7 2024-04-14T15:15:49.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.065+0000 7f2bfe7a5700 20 client.36900.objecter get_session s=0x7f2be40082f0 osd=3 3 2024-04-14T15:15:49.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.065+0000 7f2bfe7a5700 15 client.36900.objecter _session_command_op_assign 3 7 2024-04-14T15:15:49.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.065+0000 7f2bfe7a5700 20 client.36900.objecter put_session s=0x7f2be40082f0 osd=3 4 2024-04-14T15:15:49.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.065+0000 7f2bfe7a5700 10 client.36900.objecter _send_command 7 2024-04-14T15:15:49.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.065+0000 7f2bfe7a5700 1 -- 172.21.15.90:0/4219049636 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f2be4011ce0 con 0x7f2be4011190 2024-04-14T15:15:49.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.065+0000 7f2bf57fa700 10 client.36900.objecter ms_handle_connect 0x7f2be4011190 2024-04-14T15:15:49.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.066+0000 7f2bf57fa700 1 -- 172.21.15.90:0/4219049636 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2be4011ce0 con 0x7f2be4011190 2024-04-14T15:15:49.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.066+0000 7f2bf57fa700 10 client.36900.objecter ms_dispatch 0x7f2bf8001550 command_reply(tid 7: 0 ) 2024-04-14T15:15:49.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.066+0000 7f2bf57fa700 10 client.36900.objecter _finish_command 7 = system:0 2024-04-14T15:15:49.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.066+0000 7f2bf57fa700 20 client.36900.objecter put_session s=0x7f2be40082f0 osd=3 4 2024-04-14T15:15:49.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.066+0000 7f2bf57fa700 15 client.36900.objecter _session_command_op_remove 3 7 2024-04-14T15:15:49.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.084+0000 7f2bfe7a5700 10 client.36900.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:15:49.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.084+0000 7f2bfe7a5700 15 client.36900.objecter _session_command_op_assign -1 8 2024-04-14T15:15:49.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.084+0000 7f2bfe7a5700 20 client.36900.objecter _get_session s=0x7f2be40082f0 osd=3 3 2024-04-14T15:15:49.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.084+0000 7f2bfe7a5700 20 client.36900.objecter put_session s=0x7f2be40082f0 osd=3 3 2024-04-14T15:15:49.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.084+0000 7f2bfe7a5700 20 client.36900.objecter _get_session s=0x7f2be40082f0 osd=3 3 2024-04-14T15:15:49.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.084+0000 7f2bfe7a5700 15 client.36900.objecter _session_command_op_remove -1 8 2024-04-14T15:15:49.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.084+0000 7f2bfe7a5700 20 client.36900.objecter get_session s=0x7f2be40082f0 osd=3 3 2024-04-14T15:15:49.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.084+0000 7f2bfe7a5700 15 client.36900.objecter _session_command_op_assign 3 8 2024-04-14T15:15:49.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.084+0000 7f2bfe7a5700 20 client.36900.objecter put_session s=0x7f2be40082f0 osd=3 4 2024-04-14T15:15:49.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.084+0000 7f2bfe7a5700 10 client.36900.objecter _send_command 8 2024-04-14T15:15:49.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.084+0000 7f2bfe7a5700 1 -- 172.21.15.90:0/4219049636 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f2be4012690 con 0x7f2be4011190 2024-04-14T15:15:49.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.086+0000 7f2bf57fa700 1 -- 172.21.15.90:0/4219049636 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2be4012690 con 0x7f2be4011190 2024-04-14T15:15:49.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.086+0000 7f2bf57fa700 10 client.36900.objecter ms_dispatch 0x7f2bf8001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:15:49.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.086+0000 7f2bf57fa700 10 client.36900.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:15:49.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.086+0000 7f2bf57fa700 20 client.36900.objecter put_session s=0x7f2be40082f0 osd=3 4 2024-04-14T15:15:49.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.086+0000 7f2bf57fa700 15 client.36900.objecter _session_command_op_remove 3 8 2024-04-14T15:15:49.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.086+0000 7f2bfe7a5700 10 client.36900.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:15:49.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.086+0000 7f2bfe7a5700 15 client.36900.objecter _session_command_op_assign -1 9 2024-04-14T15:15:49.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.086+0000 7f2bfe7a5700 20 client.36900.objecter _get_session s=0x7f2be4013810 osd=4 3 2024-04-14T15:15:49.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.086+0000 7f2bfe7a5700 20 client.36900.objecter put_session s=0x7f2be4013810 osd=4 3 2024-04-14T15:15:49.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.086+0000 7f2bfe7a5700 20 client.36900.objecter _get_session s=0x7f2be4013810 osd=4 3 2024-04-14T15:15:49.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.086+0000 7f2bfe7a5700 15 client.36900.objecter _session_command_op_remove -1 9 2024-04-14T15:15:49.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.086+0000 7f2bfe7a5700 20 client.36900.objecter get_session s=0x7f2be4013810 osd=4 3 2024-04-14T15:15:49.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.086+0000 7f2bfe7a5700 15 client.36900.objecter _session_command_op_assign 4 9 2024-04-14T15:15:49.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.086+0000 7f2bfe7a5700 20 client.36900.objecter put_session s=0x7f2be4013810 osd=4 4 2024-04-14T15:15:49.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.086+0000 7f2bfe7a5700 10 client.36900.objecter _send_command 9 2024-04-14T15:15:49.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.086+0000 7f2bfe7a5700 1 -- 172.21.15.90:0/4219049636 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f2be401a790 con 0x7f2be4016b70 2024-04-14T15:15:49.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.087+0000 7f2bf57fa700 10 client.36900.objecter ms_handle_connect 0x7f2be4016b70 2024-04-14T15:15:49.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.087+0000 7f2bf57fa700 1 -- 172.21.15.90:0/4219049636 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2be401a790 con 0x7f2be4016b70 2024-04-14T15:15:49.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.087+0000 7f2bf57fa700 10 client.36900.objecter ms_dispatch 0x7f2bf8001550 command_reply(tid 9: 0 ) 2024-04-14T15:15:49.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.087+0000 7f2bf57fa700 10 client.36900.objecter _finish_command 9 = system:0 2024-04-14T15:15:49.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.087+0000 7f2bf57fa700 20 client.36900.objecter put_session s=0x7f2be4013810 osd=4 4 2024-04-14T15:15:49.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.087+0000 7f2bf57fa700 15 client.36900.objecter _session_command_op_remove 4 9 2024-04-14T15:15:49.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.104+0000 7f2bfe7a5700 10 client.36900.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:15:49.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.104+0000 7f2bfe7a5700 15 client.36900.objecter _session_command_op_assign -1 10 2024-04-14T15:15:49.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.104+0000 7f2bfe7a5700 20 client.36900.objecter _get_session s=0x7f2be4013810 osd=4 3 2024-04-14T15:15:49.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.104+0000 7f2bfe7a5700 20 client.36900.objecter put_session s=0x7f2be4013810 osd=4 3 2024-04-14T15:15:49.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.104+0000 7f2bfe7a5700 20 client.36900.objecter _get_session s=0x7f2be4013810 osd=4 3 2024-04-14T15:15:49.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.104+0000 7f2bfe7a5700 15 client.36900.objecter _session_command_op_remove -1 10 2024-04-14T15:15:49.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.104+0000 7f2bfe7a5700 20 client.36900.objecter get_session s=0x7f2be4013810 osd=4 3 2024-04-14T15:15:49.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.104+0000 7f2bfe7a5700 15 client.36900.objecter _session_command_op_assign 4 10 2024-04-14T15:15:49.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.104+0000 7f2bfe7a5700 20 client.36900.objecter put_session s=0x7f2be4013810 osd=4 4 2024-04-14T15:15:49.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.104+0000 7f2bfe7a5700 10 client.36900.objecter _send_command 10 2024-04-14T15:15:49.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.104+0000 7f2bfe7a5700 1 -- 172.21.15.90:0/4219049636 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f2be400c870 con 0x7f2be4016b70 2024-04-14T15:15:49.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.106+0000 7f2bf57fa700 1 -- 172.21.15.90:0/4219049636 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2be400c870 con 0x7f2be4016b70 2024-04-14T15:15:49.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.106+0000 7f2bf57fa700 10 client.36900.objecter ms_dispatch 0x7f2bf8001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:15:49.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.106+0000 7f2bf57fa700 10 client.36900.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:15:49.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.106+0000 7f2bf57fa700 20 client.36900.objecter put_session s=0x7f2be4013810 osd=4 4 2024-04-14T15:15:49.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.106+0000 7f2bf57fa700 15 client.36900.objecter _session_command_op_remove 4 10 2024-04-14T15:15:49.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.106+0000 7f2bfe7a5700 10 client.36900.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:15:49.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.106+0000 7f2bfe7a5700 15 client.36900.objecter _session_command_op_assign -1 11 2024-04-14T15:15:49.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.106+0000 7f2bfe7a5700 20 client.36900.objecter _get_session s=0x7f2be401bf80 osd=5 3 2024-04-14T15:15:49.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.106+0000 7f2bfe7a5700 20 client.36900.objecter put_session s=0x7f2be401bf80 osd=5 3 2024-04-14T15:15:49.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.106+0000 7f2bfe7a5700 20 client.36900.objecter _get_session s=0x7f2be401bf80 osd=5 3 2024-04-14T15:15:49.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.106+0000 7f2bfe7a5700 15 client.36900.objecter _session_command_op_remove -1 11 2024-04-14T15:15:49.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.106+0000 7f2bfe7a5700 20 client.36900.objecter get_session s=0x7f2be401bf80 osd=5 3 2024-04-14T15:15:49.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.106+0000 7f2bfe7a5700 15 client.36900.objecter _session_command_op_assign 5 11 2024-04-14T15:15:49.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.106+0000 7f2bfe7a5700 20 client.36900.objecter put_session s=0x7f2be401bf80 osd=5 4 2024-04-14T15:15:49.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.106+0000 7f2bfe7a5700 10 client.36900.objecter _send_command 11 2024-04-14T15:15:49.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.106+0000 7f2bfe7a5700 1 -- 172.21.15.90:0/4219049636 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f2be400c8a0 con 0x7f2be401ce10 2024-04-14T15:15:49.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.107+0000 7f2bf57fa700 10 client.36900.objecter ms_handle_connect 0x7f2be401ce10 2024-04-14T15:15:49.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.108+0000 7f2bf57fa700 1 -- 172.21.15.90:0/4219049636 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2bec00c9c0 con 0x7f2be401ce10 2024-04-14T15:15:49.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.108+0000 7f2bf57fa700 10 client.36900.objecter ms_dispatch 0x7f2bf8001550 command_reply(tid 11: 0 ) 2024-04-14T15:15:49.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.108+0000 7f2bf57fa700 10 client.36900.objecter _finish_command 11 = system:0 2024-04-14T15:15:49.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.108+0000 7f2bf57fa700 20 client.36900.objecter put_session s=0x7f2be401bf80 osd=5 4 2024-04-14T15:15:49.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.108+0000 7f2bf57fa700 15 client.36900.objecter _session_command_op_remove 5 11 2024-04-14T15:15:49.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.143+0000 7f2bfe7a5700 10 client.36900.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:15:49.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.143+0000 7f2bfe7a5700 15 client.36900.objecter _session_command_op_assign -1 12 2024-04-14T15:15:49.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.143+0000 7f2bfe7a5700 20 client.36900.objecter _get_session s=0x7f2be401bf80 osd=5 3 2024-04-14T15:15:49.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.143+0000 7f2bfe7a5700 20 client.36900.objecter put_session s=0x7f2be401bf80 osd=5 3 2024-04-14T15:15:49.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.143+0000 7f2bfe7a5700 20 client.36900.objecter _get_session s=0x7f2be401bf80 osd=5 3 2024-04-14T15:15:49.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.143+0000 7f2bfe7a5700 15 client.36900.objecter _session_command_op_remove -1 12 2024-04-14T15:15:49.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.143+0000 7f2bfe7a5700 20 client.36900.objecter get_session s=0x7f2be401bf80 osd=5 3 2024-04-14T15:15:49.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.143+0000 7f2bfe7a5700 15 client.36900.objecter _session_command_op_assign 5 12 2024-04-14T15:15:49.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.143+0000 7f2bfe7a5700 20 client.36900.objecter put_session s=0x7f2be401bf80 osd=5 4 2024-04-14T15:15:49.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.143+0000 7f2bfe7a5700 10 client.36900.objecter _send_command 12 2024-04-14T15:15:49.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.143+0000 7f2bfe7a5700 1 -- 172.21.15.90:0/4219049636 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f2be401c910 con 0x7f2be401ce10 2024-04-14T15:15:49.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.145+0000 7f2bf57fa700 1 -- 172.21.15.90:0/4219049636 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2be401c910 con 0x7f2be401ce10 2024-04-14T15:15:49.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.145+0000 7f2bf57fa700 10 client.36900.objecter ms_dispatch 0x7f2bf8001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:15:49.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.145+0000 7f2bf57fa700 10 client.36900.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:15:49.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.145+0000 7f2bf57fa700 20 client.36900.objecter put_session s=0x7f2be401bf80 osd=5 4 2024-04-14T15:15:49.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.145+0000 7f2bf57fa700 15 client.36900.objecter _session_command_op_remove 5 12 2024-04-14T15:15:49.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.145+0000 7f2bfe7a5700 10 client.36900.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:15:49.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.145+0000 7f2bfe7a5700 15 client.36900.objecter _session_command_op_assign -1 13 2024-04-14T15:15:49.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.145+0000 7f2bfe7a5700 20 client.36900.objecter _get_session s=0x7f2be401cb30 osd=6 3 2024-04-14T15:15:49.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.145+0000 7f2bfe7a5700 20 client.36900.objecter put_session s=0x7f2be401cb30 osd=6 3 2024-04-14T15:15:49.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.145+0000 7f2bfe7a5700 20 client.36900.objecter _get_session s=0x7f2be401cb30 osd=6 3 2024-04-14T15:15:49.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.145+0000 7f2bfe7a5700 15 client.36900.objecter _session_command_op_remove -1 13 2024-04-14T15:15:49.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.145+0000 7f2bfe7a5700 20 client.36900.objecter get_session s=0x7f2be401cb30 osd=6 3 2024-04-14T15:15:49.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.145+0000 7f2bfe7a5700 15 client.36900.objecter _session_command_op_assign 6 13 2024-04-14T15:15:49.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.145+0000 7f2bfe7a5700 20 client.36900.objecter put_session s=0x7f2be401cb30 osd=6 4 2024-04-14T15:15:49.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.145+0000 7f2bfe7a5700 10 client.36900.objecter _send_command 13 2024-04-14T15:15:49.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.145+0000 7f2bfe7a5700 1 -- 172.21.15.90:0/4219049636 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f2be4023360 con 0x7f2be401f850 2024-04-14T15:15:49.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.146+0000 7f2bf57fa700 10 client.36900.objecter ms_handle_connect 0x7f2be401f850 2024-04-14T15:15:49.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.146+0000 7f2bf57fa700 1 -- 172.21.15.90:0/4219049636 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2be4023360 con 0x7f2be401f850 2024-04-14T15:15:49.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.146+0000 7f2bf57fa700 10 client.36900.objecter ms_dispatch 0x7f2bf8001550 command_reply(tid 13: 0 ) 2024-04-14T15:15:49.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.146+0000 7f2bf57fa700 10 client.36900.objecter _finish_command 13 = system:0 2024-04-14T15:15:49.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.146+0000 7f2bf57fa700 20 client.36900.objecter put_session s=0x7f2be401cb30 osd=6 4 2024-04-14T15:15:49.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.146+0000 7f2bf57fa700 15 client.36900.objecter _session_command_op_remove 6 13 2024-04-14T15:15:49.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.163+0000 7f2bfe7a5700 10 client.36900.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:15:49.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.163+0000 7f2bfe7a5700 15 client.36900.objecter _session_command_op_assign -1 14 2024-04-14T15:15:49.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.163+0000 7f2bfe7a5700 20 client.36900.objecter _get_session s=0x7f2be401cb30 osd=6 3 2024-04-14T15:15:49.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.163+0000 7f2bfe7a5700 20 client.36900.objecter put_session s=0x7f2be401cb30 osd=6 3 2024-04-14T15:15:49.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.163+0000 7f2bfe7a5700 20 client.36900.objecter _get_session s=0x7f2be401cb30 osd=6 3 2024-04-14T15:15:49.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.163+0000 7f2bfe7a5700 15 client.36900.objecter _session_command_op_remove -1 14 2024-04-14T15:15:49.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.163+0000 7f2bfe7a5700 20 client.36900.objecter get_session s=0x7f2be401cb30 osd=6 3 2024-04-14T15:15:49.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.163+0000 7f2bfe7a5700 15 client.36900.objecter _session_command_op_assign 6 14 2024-04-14T15:15:49.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.163+0000 7f2bfe7a5700 20 client.36900.objecter put_session s=0x7f2be401cb30 osd=6 4 2024-04-14T15:15:49.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.163+0000 7f2bfe7a5700 10 client.36900.objecter _send_command 14 2024-04-14T15:15:49.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.163+0000 7f2bfe7a5700 1 -- 172.21.15.90:0/4219049636 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f2be401c420 con 0x7f2be401f850 2024-04-14T15:15:49.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.165+0000 7f2bf57fa700 1 -- 172.21.15.90:0/4219049636 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2be401c420 con 0x7f2be401f850 2024-04-14T15:15:49.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.165+0000 7f2bf57fa700 10 client.36900.objecter ms_dispatch 0x7f2bf8001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:15:49.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.165+0000 7f2bf57fa700 10 client.36900.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:15:49.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.165+0000 7f2bf57fa700 20 client.36900.objecter put_session s=0x7f2be401cb30 osd=6 4 2024-04-14T15:15:49.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.165+0000 7f2bf57fa700 15 client.36900.objecter _session_command_op_remove 6 14 2024-04-14T15:15:49.173 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:15:49.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.165+0000 7f2bfe7a5700 10 client.36900.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:15:49.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.165+0000 7f2bfe7a5700 15 client.36900.objecter _session_command_op_assign -1 15 2024-04-14T15:15:49.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.165+0000 7f2bfe7a5700 20 client.36900.objecter _get_session s=0x7f2be401c420 osd=7 3 2024-04-14T15:15:49.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.165+0000 7f2bfe7a5700 20 client.36900.objecter put_session s=0x7f2be401c420 osd=7 3 2024-04-14T15:15:49.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.165+0000 7f2bfe7a5700 20 client.36900.objecter _get_session s=0x7f2be401c420 osd=7 3 2024-04-14T15:15:49.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.165+0000 7f2bfe7a5700 15 client.36900.objecter _session_command_op_remove -1 15 2024-04-14T15:15:49.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.165+0000 7f2bfe7a5700 20 client.36900.objecter get_session s=0x7f2be401c420 osd=7 3 2024-04-14T15:15:49.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.165+0000 7f2bfe7a5700 15 client.36900.objecter _session_command_op_assign 7 15 2024-04-14T15:15:49.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.165+0000 7f2bfe7a5700 20 client.36900.objecter put_session s=0x7f2be401c420 osd=7 4 2024-04-14T15:15:49.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.165+0000 7f2bfe7a5700 10 client.36900.objecter _send_command 15 2024-04-14T15:15:49.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.165+0000 7f2bfe7a5700 1 -- 172.21.15.90:0/4219049636 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f2be401c630 con 0x7f2be401b470 2024-04-14T15:15:49.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.166+0000 7f2bf57fa700 10 client.36900.objecter ms_handle_connect 0x7f2be401b470 2024-04-14T15:15:49.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.166+0000 7f2bf57fa700 1 -- 172.21.15.90:0/4219049636 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2be401c630 con 0x7f2be401b470 2024-04-14T15:15:49.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.166+0000 7f2bf57fa700 10 client.36900.objecter ms_dispatch 0x7f2bf8001550 command_reply(tid 15: 0 ) 2024-04-14T15:15:49.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.166+0000 7f2bf57fa700 10 client.36900.objecter _finish_command 15 = system:0 2024-04-14T15:15:49.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.166+0000 7f2bf57fa700 20 client.36900.objecter put_session s=0x7f2be401c420 osd=7 4 2024-04-14T15:15:49.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.166+0000 7f2bf57fa700 15 client.36900.objecter _session_command_op_remove 7 15 2024-04-14T15:15:49.176 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:15:49.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.183+0000 7f2bfe7a5700 10 client.36900.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:15:49.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.183+0000 7f2bfe7a5700 15 client.36900.objecter _session_command_op_assign -1 16 2024-04-14T15:15:49.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.183+0000 7f2bfe7a5700 20 client.36900.objecter _get_session s=0x7f2be401c420 osd=7 3 2024-04-14T15:15:49.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.183+0000 7f2bfe7a5700 20 client.36900.objecter put_session s=0x7f2be401c420 osd=7 3 2024-04-14T15:15:49.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.183+0000 7f2bfe7a5700 20 client.36900.objecter _get_session s=0x7f2be401c420 osd=7 3 2024-04-14T15:15:49.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.183+0000 7f2bfe7a5700 15 client.36900.objecter _session_command_op_remove -1 16 2024-04-14T15:15:49.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.183+0000 7f2bfe7a5700 20 client.36900.objecter get_session s=0x7f2be401c420 osd=7 3 2024-04-14T15:15:49.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.183+0000 7f2bfe7a5700 15 client.36900.objecter _session_command_op_assign 7 16 2024-04-14T15:15:49.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.183+0000 7f2bfe7a5700 20 client.36900.objecter put_session s=0x7f2be401c420 osd=7 4 2024-04-14T15:15:49.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.183+0000 7f2bfe7a5700 10 client.36900.objecter _send_command 16 2024-04-14T15:15:49.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.183+0000 7f2bfe7a5700 1 -- 172.21.15.90:0/4219049636 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f2be401c630 con 0x7f2be401b470 2024-04-14T15:15:49.186 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:15:49.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.185+0000 7f2bf57fa700 1 -- 172.21.15.90:0/4219049636 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2be401c630 con 0x7f2be401b470 2024-04-14T15:15:49.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.185+0000 7f2bf57fa700 10 client.36900.objecter ms_dispatch 0x7f2bf8001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:15:49.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.185+0000 7f2bf57fa700 10 client.36900.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:15:49.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.185+0000 7f2bf57fa700 20 client.36900.objecter put_session s=0x7f2be401c420 osd=7 4 2024-04-14T15:15:49.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.185+0000 7f2bf57fa700 15 client.36900.objecter _session_command_op_remove 7 16 2024-04-14T15:15:49.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.186+0000 7f2bfe7a5700 10 librados: watch_flush enter 2024-04-14T15:15:49.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.186+0000 7f2bfe7a5700 10 librados: watch_flush exit 2024-04-14T15:15:49.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.186+0000 7f2bfe7a5700 10 client.36900.objecter close_session for osd.0 2024-04-14T15:15:49.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.186+0000 7f2bfe7a5700 1 -- 172.21.15.90:0/4219049636 >> v1:172.21.15.90:6801/3761806885 conn(0x7f2be4004730 legacy=0x7f2be4002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:49.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.186+0000 7f2bfe7a5700 20 client.36900.objecter put_session s=0x7f2be40053e0 osd=0 1 2024-04-14T15:15:49.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.186+0000 7f2bfe7a5700 10 client.36900.objecter close_session for osd.1 2024-04-14T15:15:49.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.186+0000 7f2bfe7a5700 1 -- 172.21.15.90:0/4219049636 >> v1:172.21.15.90:6805/4170361699 conn(0x7f2be40042a0 legacy=0x7f2be400ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:49.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.186+0000 7f2bfe7a5700 20 client.36900.objecter put_session s=0x7f2be4004130 osd=1 1 2024-04-14T15:15:49.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.186+0000 7f2bfe7a5700 10 client.36900.objecter close_session for osd.2 2024-04-14T15:15:49.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.186+0000 7f2bfe7a5700 1 -- 172.21.15.90:0/4219049636 >> v1:172.21.15.90:6809/1686184251 conn(0x7f2be4008990 legacy=0x7f2be400f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:49.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.186+0000 7f2bfcd42700 1 -- 172.21.15.90:0/4219049636 reap_dead start 2024-04-14T15:15:49.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.186+0000 7f2bfe7a5700 20 client.36900.objecter put_session s=0x7f2be4001a00 osd=2 1 2024-04-14T15:15:49.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.186+0000 7f2bfe7a5700 10 client.36900.objecter close_session for osd.3 2024-04-14T15:15:49.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.186+0000 7f2bfe7a5700 1 -- 172.21.15.90:0/4219049636 >> v1:172.21.15.90:6813/2270166774 conn(0x7f2be4011190 legacy=0x7f2be40115d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:49.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.186+0000 7f2bfe7a5700 20 client.36900.objecter put_session s=0x7f2be40082f0 osd=3 1 2024-04-14T15:15:49.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.186+0000 7f2bfe7a5700 10 client.36900.objecter close_session for osd.4 2024-04-14T15:15:49.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.186+0000 7f2bfe7a5700 1 -- 172.21.15.90:0/4219049636 >> v1:172.21.15.92:6800/3887549733 conn(0x7f2be4016b70 legacy=0x7f2be4018f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:49.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.186+0000 7f2bfe7a5700 20 client.36900.objecter put_session s=0x7f2be4013810 osd=4 1 2024-04-14T15:15:49.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.186+0000 7f2bfe7a5700 10 client.36900.objecter close_session for osd.5 2024-04-14T15:15:49.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.186+0000 7f2bfe7a5700 1 -- 172.21.15.90:0/4219049636 >> v1:172.21.15.92:6804/3244561440 conn(0x7f2be401ce10 legacy=0x7f2be4011ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:49.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.186+0000 7f2bfe7a5700 20 client.36900.objecter put_session s=0x7f2be401bf80 osd=5 1 2024-04-14T15:15:49.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.186+0000 7f2bfe7a5700 10 client.36900.objecter close_session for osd.6 2024-04-14T15:15:49.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.186+0000 7f2bfe7a5700 1 -- 172.21.15.90:0/4219049636 >> v1:172.21.15.92:6808/636125963 conn(0x7f2be401f850 legacy=0x7f2be4021c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:49.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.186+0000 7f2bfe7a5700 20 client.36900.objecter put_session s=0x7f2be401cb30 osd=6 1 2024-04-14T15:15:49.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.186+0000 7f2bfe7a5700 10 client.36900.objecter close_session for osd.7 2024-04-14T15:15:49.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.186+0000 7f2bfe7a5700 1 -- 172.21.15.90:0/4219049636 >> v1:172.21.15.92:6812/1773706716 conn(0x7f2be401b470 legacy=0x7f2be4023360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:49.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.186+0000 7f2bfe7a5700 20 client.36900.objecter put_session s=0x7f2be401c420 osd=7 1 2024-04-14T15:15:49.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.186+0000 7f2bfe7a5700 20 client.36900.objecter shutdown clearing up homeless session... 2024-04-14T15:15:49.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.186+0000 7f2bfe7a5700 10 client.36900.objecter successfully canceled tick 2024-04-14T15:15:49.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.186+0000 7f2bfcd42700 1 -- 172.21.15.90:0/4219049636 reap_dead start 2024-04-14T15:15:49.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.186+0000 7f2bfe7a5700 1 -- 172.21.15.90:0/4219049636 >> v1:172.21.15.90:6800/420935427 conn(0x7f2be00757f0 legacy=0x7f2be0077cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:49.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.186+0000 7f2bfe7a5700 1 -- 172.21.15.90:0/4219049636 >> v1:172.21.15.90:6789/0 conn(0x7f2bf8103a20 legacy=0x7f2bf810b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:49.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.186+0000 7f2bfe7a5700 1 -- 172.21.15.90:0/4219049636 shutdown_connections 2024-04-14T15:15:49.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.186+0000 7f2bfe7a5700 1 -- 172.21.15.90:0/4219049636 >> 172.21.15.90:0/4219049636 conn(0x7f2bf80fd290 msgr2=0x7f2bf80ff680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:15:49.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.186+0000 7f2bfe7a5700 1 -- 172.21.15.90:0/4219049636 shutdown_connections 2024-04-14T15:15:49.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.186+0000 7f2bfe7a5700 1 -- 172.21.15.90:0/4219049636 wait complete. 2024-04-14T15:15:49.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:49.186+0000 7f2bfe7a5700 1 librados: shutdown 2024-04-14T15:15:49.191 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:15:49.191 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:15:49.191 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:15:49.191 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:15:49.191 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:15:49.191 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:15:49.191 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:15:49.191 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:15:49.201 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:49.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:49.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:15:49.303 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:15:49.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:49.328 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:49.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:15:49.429 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:15:49.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:49.454 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:49.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:15:49.462 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-14T15:15:49.462 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:15:49.555 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:15:49.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:49.579 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:49.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:15:49.680 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:15:49.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:49.704 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:49.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:15:49.805 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:15:49.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:49.830 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:49.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:15:49.931 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:15:49.957 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:49.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:49.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:15:50.059 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:15:50.062 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:49 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2058889066' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi090-93363-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi090-93363-49"}]': finished 2024-04-14T15:15:50.062 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:49 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-33", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:15:50.063 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:49 smithi090 ceph-mon[41670]: from='client.48715 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi090-93342-40"}]': finished 2024-04-14T15:15:50.063 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:49 smithi090 ceph-mon[41670]: osdmap e281: 8 total, 8 up, 8 in 2024-04-14T15:15:50.063 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:49 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-33"}]: dispatch 2024-04-14T15:15:50.063 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:49 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1788001083' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi090-93342-40"}]: dispatch 2024-04-14T15:15:50.063 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:49 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-33"}]: dispatch 2024-04-14T15:15:50.063 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:49 smithi090 ceph-mon[41670]: from='client.48715 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi090-93342-40"}]: dispatch 2024-04-14T15:15:50.063 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:49 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/4219049636' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:15:50.063 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:49 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2058889066' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi090-93363-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi090-93363-49"}]': finished 2024-04-14T15:15:50.063 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:49 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-33", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:15:50.063 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:49 smithi090 ceph-mon[30399]: from='client.48715 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi090-93342-40"}]': finished 2024-04-14T15:15:50.063 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:49 smithi090 ceph-mon[30399]: osdmap e281: 8 total, 8 up, 8 in 2024-04-14T15:15:50.064 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:49 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-33"}]: dispatch 2024-04-14T15:15:50.064 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:49 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1788001083' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi090-93342-40"}]: dispatch 2024-04-14T15:15:50.064 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:49 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-33"}]: dispatch 2024-04-14T15:15:50.064 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:49 smithi090 ceph-mon[30399]: from='client.48715 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi090-93342-40"}]: dispatch 2024-04-14T15:15:50.064 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:49 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/4219049636' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:15:50.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:50.085 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:50.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:15:50.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:49 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2058889066' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi090-93363-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi090-93363-49"}]': finished 2024-04-14T15:15:50.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:49 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-33", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:15:50.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:49 smithi092 ceph-mon[33966]: from='client.48715 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi090-93342-40"}]': finished 2024-04-14T15:15:50.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:49 smithi092 ceph-mon[33966]: osdmap e281: 8 total, 8 up, 8 in 2024-04-14T15:15:50.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:49 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-33"}]: dispatch 2024-04-14T15:15:50.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:49 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1788001083' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi090-93342-40"}]: dispatch 2024-04-14T15:15:50.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:49 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-33"}]: dispatch 2024-04-14T15:15:50.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:49 smithi092 ceph-mon[33966]: from='client.48715 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi090-93342-40"}]: dispatch 2024-04-14T15:15:50.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:49 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/4219049636' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:15:50.130 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-14T15:15:50.130 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-14T15:15:50.130 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-04-14T15:15:50.130 INFO:tasks.thrashosds.thrasher:Setting osd 6 primary_affinity to 1.000000 2024-04-14T15:15:50.130 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd primary-affinity 6 1 2024-04-14T15:15:50.186 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:15:50.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:50.210 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:50.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:15:50.311 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:15:50.335 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:50.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:50.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:15:50.437 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:15:50.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:50.462 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:50.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:15:50.504 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-14T15:15:50.563 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:15:50.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:50.587 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:50.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:15:50.688 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:15:50.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:50.713 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:50.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:15:50.814 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:15:50.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:50.838 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:50.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:15:50.939 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:15:50.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:50.964 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:50.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:15:51.065 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:15:51.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:51.090 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:51.091 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:15:51.117 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:15:51.191 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:15:51.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:50 smithi090 ceph-mon[30399]: pgmap v368: 246 pgs: 7 creating+peering, 8 unknown, 3 peering, 228 active+clean; 4.4 MiB data, 559 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:15:51.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:50 smithi090 ceph-mon[30399]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:15:51.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:50 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-33"}]': finished 2024-04-14T15:15:51.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:50 smithi090 ceph-mon[30399]: from='client.48715 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi090-93342-40"}]': finished 2024-04-14T15:15:51.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:50 smithi090 ceph-mon[30399]: osdmap e282: 8 total, 8 up, 8 in 2024-04-14T15:15:51.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:50 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-33", "mode": "writeback"}]: dispatch 2024-04-14T15:15:51.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:50 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-33", "mode": "writeback"}]: dispatch 2024-04-14T15:15:51.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:50 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3501572542' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi090-93342-41"}]: dispatch 2024-04-14T15:15:51.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:50 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3501572542' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi090-93342-41"}]: dispatch 2024-04-14T15:15:51.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:50 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3501572542' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi090-93342-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:51.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:50 smithi090 ceph-mon[30399]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:15:51.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:50 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-33", "mode": "writeback"}]': finished 2024-04-14T15:15:51.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:50 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3501572542' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi090-93342-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:15:51.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:50 smithi090 ceph-mon[30399]: osdmap e283: 8 total, 8 up, 8 in 2024-04-14T15:15:51.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:50 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3501572542' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi090-93342-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi090-93342-41"}]: dispatch 2024-04-14T15:15:51.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:50 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2058889066' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi090-93363-49"}]: dispatch 2024-04-14T15:15:51.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:50 smithi090 ceph-mon[41670]: pgmap v368: 246 pgs: 7 creating+peering, 8 unknown, 3 peering, 228 active+clean; 4.4 MiB data, 559 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:15:51.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:50 smithi090 ceph-mon[41670]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:15:51.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:50 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-33"}]': finished 2024-04-14T15:15:51.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:50 smithi090 ceph-mon[41670]: from='client.48715 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi090-93342-40"}]': finished 2024-04-14T15:15:51.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:50 smithi090 ceph-mon[41670]: osdmap e282: 8 total, 8 up, 8 in 2024-04-14T15:15:51.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:50 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-33", "mode": "writeback"}]: dispatch 2024-04-14T15:15:51.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:50 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-33", "mode": "writeback"}]: dispatch 2024-04-14T15:15:51.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:50 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3501572542' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi090-93342-41"}]: dispatch 2024-04-14T15:15:51.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:50 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3501572542' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi090-93342-41"}]: dispatch 2024-04-14T15:15:51.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:50 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3501572542' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi090-93342-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:51.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:50 smithi090 ceph-mon[41670]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:15:51.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:50 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-33", "mode": "writeback"}]': finished 2024-04-14T15:15:51.195 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:50 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3501572542' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi090-93342-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:15:51.195 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:50 smithi090 ceph-mon[41670]: osdmap e283: 8 total, 8 up, 8 in 2024-04-14T15:15:51.195 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:50 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3501572542' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi090-93342-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi090-93342-41"}]: dispatch 2024-04-14T15:15:51.195 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:50 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2058889066' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi090-93363-49"}]: dispatch 2024-04-14T15:15:51.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:51.218 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:51.218 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:15:51.319 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:15:51.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:51.343 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:51.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:15:51.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:50 smithi092 ceph-mon[33966]: pgmap v368: 246 pgs: 7 creating+peering, 8 unknown, 3 peering, 228 active+clean; 4.4 MiB data, 559 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:15:51.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:50 smithi092 ceph-mon[33966]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:15:51.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:50 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-33"}]': finished 2024-04-14T15:15:51.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:50 smithi092 ceph-mon[33966]: from='client.48715 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi090-93342-40"}]': finished 2024-04-14T15:15:51.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:50 smithi092 ceph-mon[33966]: osdmap e282: 8 total, 8 up, 8 in 2024-04-14T15:15:51.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:50 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-33", "mode": "writeback"}]: dispatch 2024-04-14T15:15:51.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:50 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-33", "mode": "writeback"}]: dispatch 2024-04-14T15:15:51.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:50 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3501572542' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi090-93342-41"}]: dispatch 2024-04-14T15:15:51.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:50 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3501572542' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi090-93342-41"}]: dispatch 2024-04-14T15:15:51.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:50 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3501572542' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi090-93342-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:51.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:50 smithi092 ceph-mon[33966]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:15:51.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:50 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-33", "mode": "writeback"}]': finished 2024-04-14T15:15:51.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:50 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3501572542' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi090-93342-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:15:51.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:50 smithi092 ceph-mon[33966]: osdmap e283: 8 total, 8 up, 8 in 2024-04-14T15:15:51.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:50 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3501572542' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi090-93342-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi090-93342-41"}]: dispatch 2024-04-14T15:15:51.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:50 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2058889066' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi090-93363-49"}]: dispatch 2024-04-14T15:15:51.445 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:15:51.468 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:51.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:51.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:15:51.570 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:15:51.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:51.594 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:51.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:15:51.696 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:15:51.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:51.720 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:51.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:15:51.821 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:15:51.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:51.846 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:51.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:15:51.947 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:15:51.971 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:51.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:51.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:15:51.981 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:15:52.014 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:15:52.072 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:15:52.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:52.097 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:52.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:15:52.105 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:51 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:15:52.105 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:51 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:15:52.105 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:51 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2058889066' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi090-93363-49"}]': finished 2024-04-14T15:15:52.106 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:51 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:15:52.106 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:51 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:15:52.106 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:51 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2058889066' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi090-93363-49"}]': finished 2024-04-14T15:15:52.199 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:15:52.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:52.224 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:52.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:15:52.273 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:51 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:15:52.273 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:51 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:15:52.273 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:51 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2058889066' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi090-93363-49"}]': finished 2024-04-14T15:15:52.325 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:15:52.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:52.351 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:52.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:15:52.452 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:15:52.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:52.477 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:52.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:15:52.578 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:15:52.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:52.602 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:52.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:15:52.703 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:15:52.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:52.728 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:52.728 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:15:52.829 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:15:52.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:52.856 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:52.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:15:52.956 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:15:52.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:52.982 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:52.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:15:53.084 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:15:53.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:53.108 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:53.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:15:53.111 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-14T15:15:53.172 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:52 smithi090 ceph-mon[30399]: pgmap v371: 238 pgs: 238 active+clean; 4.4 MiB data, 559 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:15:53.172 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:52 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:15:53.173 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:52 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-33"}]: dispatch 2024-04-14T15:15:53.173 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:52 smithi090 ceph-mon[30399]: osdmap e284: 8 total, 8 up, 8 in 2024-04-14T15:15:53.173 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:52 smithi090 ceph-mon[30399]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2024-04-14T15:15:53.173 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:52 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2058889066' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi090-93363-49"}]: dispatch 2024-04-14T15:15:53.173 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:52 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-33"}]: dispatch 2024-04-14T15:15:53.173 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:52 smithi090 ceph-mon[30399]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:15:53.173 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:52 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3501572542' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi090-93342-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi090-93342-41"}]': finished 2024-04-14T15:15:53.173 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:52 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2058889066' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi090-93363-49"}]': finished 2024-04-14T15:15:53.173 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:52 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-33"}]': finished 2024-04-14T15:15:53.173 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:52 smithi090 ceph-mon[30399]: osdmap e285: 8 total, 8 up, 8 in 2024-04-14T15:15:53.173 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:52 smithi090 ceph-mon[41670]: pgmap v371: 238 pgs: 238 active+clean; 4.4 MiB data, 559 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:15:53.173 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:52 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:15:53.174 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:52 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-33"}]: dispatch 2024-04-14T15:15:53.174 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:52 smithi090 ceph-mon[41670]: osdmap e284: 8 total, 8 up, 8 in 2024-04-14T15:15:53.174 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:52 smithi090 ceph-mon[41670]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2024-04-14T15:15:53.174 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:52 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2058889066' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi090-93363-49"}]: dispatch 2024-04-14T15:15:53.174 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:52 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-33"}]: dispatch 2024-04-14T15:15:53.174 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:52 smithi090 ceph-mon[41670]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:15:53.174 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:52 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3501572542' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi090-93342-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi090-93342-41"}]': finished 2024-04-14T15:15:53.174 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:52 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2058889066' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi090-93363-49"}]': finished 2024-04-14T15:15:53.174 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:52 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-33"}]': finished 2024-04-14T15:15:53.174 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:52 smithi090 ceph-mon[41670]: osdmap e285: 8 total, 8 up, 8 in 2024-04-14T15:15:53.209 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:15:53.233 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:53.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:53.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:15:53.335 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:15:53.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:53.359 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:53.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:15:53.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:52 smithi092 ceph-mon[33966]: pgmap v371: 238 pgs: 238 active+clean; 4.4 MiB data, 559 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:15:53.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:52 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:15:53.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:52 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-33"}]: dispatch 2024-04-14T15:15:53.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:52 smithi092 ceph-mon[33966]: osdmap e284: 8 total, 8 up, 8 in 2024-04-14T15:15:53.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:52 smithi092 ceph-mon[33966]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2024-04-14T15:15:53.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:52 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2058889066' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi090-93363-49"}]: dispatch 2024-04-14T15:15:53.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:52 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-33"}]: dispatch 2024-04-14T15:15:53.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:52 smithi092 ceph-mon[33966]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:15:53.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:52 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3501572542' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi090-93342-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi090-93342-41"}]': finished 2024-04-14T15:15:53.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:52 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2058889066' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi090-93363-49"}]': finished 2024-04-14T15:15:53.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:52 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-33"}]': finished 2024-04-14T15:15:53.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:52 smithi092 ceph-mon[33966]: osdmap e285: 8 total, 8 up, 8 in 2024-04-14T15:15:53.460 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:15:53.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:53.486 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:53.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:15:53.587 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:15:53.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:53.612 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:53.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:15:53.713 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:15:53.738 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:53.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:53.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:15:53.840 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:15:53.864 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:53.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:53.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:15:53.966 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:15:53.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:53.992 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:53.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:15:54.093 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:15:54.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:54.117 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:54.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:15:54.191 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:15:53 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:15:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:15:54.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:53 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3032682252' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi090-93363-50"}]: dispatch 2024-04-14T15:15:54.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:53 smithi090 ceph-mon[41670]: from='client.46865 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi090-93363-50"}]: dispatch 2024-04-14T15:15:54.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:53 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3032682252' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi090-93363-50"}]: dispatch 2024-04-14T15:15:54.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:53 smithi090 ceph-mon[41670]: from='client.46865 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi090-93363-50"}]: dispatch 2024-04-14T15:15:54.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:53 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3032682252' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi090-93363-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:54.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:53 smithi090 ceph-mon[41670]: from='client.46865 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi090-93363-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:54.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:53 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:15:54.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:53 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:15:54.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:53 smithi090 ceph-mon[41670]: from='client.46865 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi090-93363-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:15:54.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:53 smithi090 ceph-mon[41670]: osdmap e286: 8 total, 8 up, 8 in 2024-04-14T15:15:54.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:53 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3032682252' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi090-93363-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi090-93363-50"}]: dispatch 2024-04-14T15:15:54.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:53 smithi090 ceph-mon[41670]: from='client.46865 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi090-93363-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi090-93363-50"}]: dispatch 2024-04-14T15:15:54.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:53 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3032682252' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi090-93363-50"}]: dispatch 2024-04-14T15:15:54.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:53 smithi090 ceph-mon[30399]: from='client.46865 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi090-93363-50"}]: dispatch 2024-04-14T15:15:54.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:53 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3032682252' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi090-93363-50"}]: dispatch 2024-04-14T15:15:54.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:53 smithi090 ceph-mon[30399]: from='client.46865 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi090-93363-50"}]: dispatch 2024-04-14T15:15:54.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:53 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3032682252' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi090-93363-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:54.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:53 smithi090 ceph-mon[30399]: from='client.46865 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi090-93363-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:54.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:53 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:15:54.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:53 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:15:54.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:53 smithi090 ceph-mon[30399]: from='client.46865 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi090-93363-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:15:54.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:53 smithi090 ceph-mon[30399]: osdmap e286: 8 total, 8 up, 8 in 2024-04-14T15:15:54.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:53 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3032682252' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi090-93363-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi090-93363-50"}]: dispatch 2024-04-14T15:15:54.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:53 smithi090 ceph-mon[30399]: from='client.46865 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi090-93363-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi090-93363-50"}]: dispatch 2024-04-14T15:15:54.218 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:15:54.242 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:54.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:54.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:15:54.344 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:15:54.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:54.370 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:54.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:15:54.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:53 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3032682252' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi090-93363-50"}]: dispatch 2024-04-14T15:15:54.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:53 smithi092 ceph-mon[33966]: from='client.46865 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi090-93363-50"}]: dispatch 2024-04-14T15:15:54.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:53 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3032682252' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi090-93363-50"}]: dispatch 2024-04-14T15:15:54.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:53 smithi092 ceph-mon[33966]: from='client.46865 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi090-93363-50"}]: dispatch 2024-04-14T15:15:54.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:53 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3032682252' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi090-93363-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:54.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:53 smithi092 ceph-mon[33966]: from='client.46865 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi090-93363-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:54.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:53 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:15:54.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:53 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:15:54.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:53 smithi092 ceph-mon[33966]: from='client.46865 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi090-93363-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:15:54.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:53 smithi092 ceph-mon[33966]: osdmap e286: 8 total, 8 up, 8 in 2024-04-14T15:15:54.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:53 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3032682252' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi090-93363-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi090-93363-50"}]: dispatch 2024-04-14T15:15:54.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:53 smithi092 ceph-mon[33966]: from='client.46865 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi090-93363-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi090-93363-50"}]: dispatch 2024-04-14T15:15:54.471 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:15:54.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:54.496 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:54.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:15:54.597 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:15:54.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:54.622 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:54.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:15:54.723 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:15:54.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:54.751 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:54.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:15:54.852 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:15:54.878 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:54.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:54.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:15:54.981 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:15:55.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:55.007 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:55.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:15:55.108 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:15:55.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:55.135 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:55.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:15:55.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:54 smithi090 ceph-mon[41670]: pgmap v374: 246 pgs: 8 unknown, 238 active+clean; 4.4 MiB data, 559 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:15:55.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:54 smithi090 ceph-mon[41670]: osdmap e287: 8 total, 8 up, 8 in 2024-04-14T15:15:55.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:54 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:55.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:54 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3501572542' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi090-93342-41"}]: dispatch 2024-04-14T15:15:55.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:54 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:55.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:54 smithi090 ceph-mon[30399]: pgmap v374: 246 pgs: 8 unknown, 238 active+clean; 4.4 MiB data, 559 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:15:55.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:54 smithi090 ceph-mon[30399]: osdmap e287: 8 total, 8 up, 8 in 2024-04-14T15:15:55.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:54 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:55.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:54 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3501572542' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi090-93342-41"}]: dispatch 2024-04-14T15:15:55.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:54 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:55.236 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:15:55.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:55.261 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:55.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:15:55.362 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:15:55.372 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:15:54 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:15:55.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:54 smithi092 ceph-mon[33966]: pgmap v374: 246 pgs: 8 unknown, 238 active+clean; 4.4 MiB data, 559 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:15:55.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:54 smithi092 ceph-mon[33966]: osdmap e287: 8 total, 8 up, 8 in 2024-04-14T15:15:55.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:54 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:55.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:54 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3501572542' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi090-93342-41"}]: dispatch 2024-04-14T15:15:55.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:54 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:15:55.390 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:55.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:55.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:15:55.492 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:15:55.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:55.516 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:55.517 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:15:55.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:55.489+0000 7f54eaba9700 1 -- 172.21.15.90:0/3488484312 >> v1:172.21.15.90:6789/0 conn(0x7f54e410b930 legacy=0x7f54e410dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:55.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:55.489+0000 7f54eaba9700 1 -- 172.21.15.90:0/3488484312 shutdown_connections 2024-04-14T15:15:55.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:55.489+0000 7f54eaba9700 1 -- 172.21.15.90:0/3488484312 >> 172.21.15.90:0/3488484312 conn(0x7f54e40fd290 msgr2=0x7f54e40ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:15:55.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:55.489+0000 7f54eaba9700 1 -- 172.21.15.90:0/3488484312 shutdown_connections 2024-04-14T15:15:55.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:55.489+0000 7f54eaba9700 1 -- 172.21.15.90:0/3488484312 wait complete. 2024-04-14T15:15:55.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:55.489+0000 7f54eaba9700 1 librados: starting msgr at 2024-04-14T15:15:55.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:55.489+0000 7f54eaba9700 1 librados: starting objecter 2024-04-14T15:15:55.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:55.490+0000 7f54eaba9700 1 Processor -- start 2024-04-14T15:15:55.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:55.490+0000 7f54eaba9700 1 -- start start 2024-04-14T15:15:55.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:55.490+0000 7f54eaba9700 1 librados: setting wanted keys 2024-04-14T15:15:55.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:55.490+0000 7f54eaba9700 1 librados: calling monclient init 2024-04-14T15:15:55.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:55.490+0000 7f54eaba9700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f54e4103240 con 0x7f54e410b930 2024-04-14T15:15:55.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:55.490+0000 7f54eaba9700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f54e4103450 con 0x7f54e4103a40 2024-04-14T15:15:55.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:55.490+0000 7f54eaba9700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f54e41ac470 con 0x7f54e410f4d0 2024-04-14T15:15:55.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:55.490+0000 7f54dbfff700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f54e410b930 0x7f54e4101db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:48420/0 (socket says 172.21.15.90:48420) 2024-04-14T15:15:55.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:55.490+0000 7f54e9146700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f54e410f4d0 0x7f54e41a8d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:36044/0 (socket says 172.21.15.90:36044) 2024-04-14T15:15:55.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:55.490+0000 7f54dbfff700 1 -- 172.21.15.90:0/2688309294 learned_addr learned my addr 172.21.15.90:0/2688309294 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:15:55.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:55.490+0000 7f54d9ffb700 10 client.?.objecter ms_handle_connect 0x7f54e410b930 2024-04-14T15:15:55.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:55.490+0000 7f54d9ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:15:55.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:55.490+0000 7f54d9ffb700 10 client.?.objecter ms_handle_connect 0x7f54e410f4d0 2024-04-14T15:15:55.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:55.490+0000 7f54d9ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:15:55.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:55.490+0000 7f54d9ffb700 10 client.?.objecter ms_handle_connect 0x7f54e4103a40 2024-04-14T15:15:55.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:55.490+0000 7f54d9ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:15:55.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:55.490+0000 7f54d9ffb700 1 -- 172.21.15.90:0/2688309294 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2200843709 0 0) 0x7f54e4103240 con 0x7f54e410b930 2024-04-14T15:15:55.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:55.490+0000 7f54d9ffb700 1 -- 172.21.15.90:0/2688309294 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f54cc0031f0 con 0x7f54e410b930 2024-04-14T15:15:55.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:55.490+0000 7f54d9ffb700 1 -- 172.21.15.90:0/2688309294 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3800166491 0 0) 0x7f54e41ac470 con 0x7f54e410f4d0 2024-04-14T15:15:55.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:55.490+0000 7f54d9ffb700 1 -- 172.21.15.90:0/2688309294 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f54e4103240 con 0x7f54e410f4d0 2024-04-14T15:15:55.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:55.490+0000 7f54d9ffb700 1 -- 172.21.15.90:0/2688309294 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3513761743 0 0) 0x7f54e4103450 con 0x7f54e4103a40 2024-04-14T15:15:55.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:55.490+0000 7f54d9ffb700 1 -- 172.21.15.90:0/2688309294 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f54e41ac470 con 0x7f54e4103a40 2024-04-14T15:15:55.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:55.490+0000 7f54d9ffb700 1 -- 172.21.15.90:0/2688309294 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2886242175 0 0) 0x7f54cc0031f0 con 0x7f54e410b930 2024-04-14T15:15:55.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:55.491+0000 7f54d9ffb700 1 -- 172.21.15.90:0/2688309294 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f54e4103450 con 0x7f54e410b930 2024-04-14T15:15:55.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:55.491+0000 7f54d9ffb700 1 -- 172.21.15.90:0/2688309294 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f54d4004730 con 0x7f54e410b930 2024-04-14T15:15:55.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:55.491+0000 7f54d9ffb700 1 -- 172.21.15.90:0/2688309294 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2210406505 0 0) 0x7f54e4103240 con 0x7f54e410f4d0 2024-04-14T15:15:55.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:55.491+0000 7f54d9ffb700 1 -- 172.21.15.90:0/2688309294 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f54cc0031f0 con 0x7f54e410f4d0 2024-04-14T15:15:55.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:55.491+0000 7f54d9ffb700 1 -- 172.21.15.90:0/2688309294 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f54dc001bd0 con 0x7f54e410f4d0 2024-04-14T15:15:55.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:55.491+0000 7f54d9ffb700 1 -- 172.21.15.90:0/2688309294 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2385298085 0 0) 0x7f54e4103450 con 0x7f54e410b930 2024-04-14T15:15:55.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:55.491+0000 7f54d9ffb700 1 -- 172.21.15.90:0/2688309294 >> v1:172.21.15.90:6790/0 conn(0x7f54e410f4d0 legacy=0x7f54e41a8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:55.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:55.491+0000 7f54d9ffb700 1 -- 172.21.15.90:0/2688309294 >> v1:172.21.15.92:6789/0 conn(0x7f54e4103a40 legacy=0x7f54e41016a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:55.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:55.491+0000 7f54d9ffb700 1 -- 172.21.15.90:0/2688309294 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54e41ad650 con 0x7f54e410b930 2024-04-14T15:15:55.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:55.491+0000 7f54eaba9700 1 -- 172.21.15.90:0/2688309294 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f54e41ac6a0 con 0x7f54e410b930 2024-04-14T15:15:55.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:55.491+0000 7f54d9ffb700 1 -- 172.21.15.90:0/2688309294 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f54d4001ae0 con 0x7f54e410b930 2024-04-14T15:15:55.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:55.491+0000 7f54d9ffb700 1 -- 172.21.15.90:0/2688309294 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f54d40051f0 con 0x7f54e410b930 2024-04-14T15:15:55.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:55.491+0000 7f54eaba9700 10 client.36918.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:15:55.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:55.491+0000 7f54eaba9700 1 -- 172.21.15.90:0/2688309294 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f54e41acb60 con 0x7f54e410b930 2024-04-14T15:15:55.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:55.491+0000 7f54eaba9700 1 librados: init done 2024-04-14T15:15:55.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:55.492+0000 7f54eaba9700 1 -- 172.21.15.90:0/2688309294 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54e406b5d0 con 0x7f54e410b930 2024-04-14T15:15:55.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:55.492+0000 7f54d9ffb700 1 -- 172.21.15.90:0/2688309294 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f54d4005390 con 0x7f54e410b930 2024-04-14T15:15:55.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:55.493+0000 7f54d9ffb700 1 -- 172.21.15.90:0/2688309294 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(287..287 src has 51..287) ==== 8913+0+0 (unknown 1851210724 0 0) 0x7f54d408b0f0 con 0x7f54e410b930 2024-04-14T15:15:55.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:55.493+0000 7f54d9ffb700 10 client.36918.objecter ms_dispatch 0x7f54e4001550 osd_map(287..287 src has 51..287) 2024-04-14T15:15:55.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:55.493+0000 7f54d9ffb700 3 client.36918.objecter handle_osd_map got epochs [287,287] > 0 2024-04-14T15:15:55.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:55.493+0000 7f54d9ffb700 3 client.36918.objecter handle_osd_map decoding full epoch 287 2024-04-14T15:15:55.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:55.493+0000 7f54d9ffb700 20 client.36918.objecter dump_active .. 0 homeless 2024-04-14T15:15:55.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:55.493+0000 7f54d9ffb700 10 client.36918.objecter ms_handle_connect 0x7f54cc0712b0 2024-04-14T15:15:55.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:55.498+0000 7f54d9ffb700 1 -- 172.21.15.90:0/2688309294 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f54d4057830 con 0x7f54e410b930 2024-04-14T15:15:55.618 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:15:55.645 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:55.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:55.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:15:55.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:55.712+0000 7f54eaba9700 1 -- 172.21.15.90:0/2688309294 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7f54e40693d0 con 0x7f54e410b930 2024-04-14T15:15:55.747 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:15:55.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:55.772 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:55.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:15:55.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:55.853+0000 7f54d9ffb700 1 -- 172.21.15.90:0/2688309294 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v288) ==== 97+0+0 (unknown 481751723 0 0) 0x7f54d405b4e0 con 0x7f54e410b930 2024-04-14T15:15:55.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:55.856+0000 7f54eaba9700 10 librados: watch_flush enter 2024-04-14T15:15:55.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:55.857+0000 7f54eaba9700 10 librados: watch_flush exit 2024-04-14T15:15:55.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:55.857+0000 7f54eaba9700 20 client.36918.objecter shutdown clearing up homeless session... 2024-04-14T15:15:55.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:55.857+0000 7f54eaba9700 10 client.36918.objecter successfully canceled tick 2024-04-14T15:15:55.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:55.858+0000 7f54eaba9700 1 -- 172.21.15.90:0/2688309294 >> v1:172.21.15.90:6800/420935427 conn(0x7f54cc0712b0 legacy=0x7f54cc073770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:55.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:55.858+0000 7f54eaba9700 1 -- 172.21.15.90:0/2688309294 >> v1:172.21.15.90:6789/0 conn(0x7f54e410b930 legacy=0x7f54e4101db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:55.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:55.863+0000 7f54eaba9700 1 -- 172.21.15.90:0/2688309294 shutdown_connections 2024-04-14T15:15:55.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:55.864+0000 7f54eaba9700 1 -- 172.21.15.90:0/2688309294 >> 172.21.15.90:0/2688309294 conn(0x7f54e40fd290 msgr2=0x7f54e4111930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:15:55.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:55.864+0000 7f54eaba9700 1 -- 172.21.15.90:0/2688309294 shutdown_connections 2024-04-14T15:15:55.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:55.864+0000 7f54eaba9700 1 -- 172.21.15.90:0/2688309294 wait complete. 2024-04-14T15:15:55.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:55.867+0000 7f54eaba9700 1 librados: shutdown 2024-04-14T15:15:55.870 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is set 2024-04-14T15:15:55.873 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:15:55.900 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:55.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:55.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:15:56.001 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:15:56.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:56.025 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:56.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:15:56.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:56.035+0000 7f476b748700 1 -- 172.21.15.90:0/753400880 >> v1:172.21.15.90:6789/0 conn(0x7f476410a6d0 legacy=0x7f476410cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:56.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:56.035+0000 7f476b748700 1 -- 172.21.15.90:0/753400880 shutdown_connections 2024-04-14T15:15:56.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:56.035+0000 7f476b748700 1 -- 172.21.15.90:0/753400880 >> 172.21.15.90:0/753400880 conn(0x7f4764076aa0 msgr2=0x7f4764076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:15:56.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:56.035+0000 7f476b748700 1 -- 172.21.15.90:0/753400880 shutdown_connections 2024-04-14T15:15:56.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:56.035+0000 7f476b748700 1 -- 172.21.15.90:0/753400880 wait complete. 2024-04-14T15:15:56.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:56.035+0000 7f476b748700 1 librados: starting msgr at 2024-04-14T15:15:56.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:56.035+0000 7f476b748700 1 librados: starting objecter 2024-04-14T15:15:56.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:56.036+0000 7f476b748700 1 Processor -- start 2024-04-14T15:15:56.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:56.036+0000 7f476b748700 1 -- start start 2024-04-14T15:15:56.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:56.036+0000 7f476b748700 1 librados: setting wanted keys 2024-04-14T15:15:56.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:56.036+0000 7f476b748700 1 librados: calling monclient init 2024-04-14T15:15:56.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:56.036+0000 7f476b748700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f47641070c0 con 0x7f476410a6d0 2024-04-14T15:15:56.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:56.036+0000 7f476b748700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f47641a8e50 con 0x7f47641023f0 2024-04-14T15:15:56.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:56.036+0000 7f476b748700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f47641aa030 con 0x7f476410e270 2024-04-14T15:15:56.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:56.036+0000 7f4769ce5700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f476410e270 0x7f47641a6720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:36060/0 (socket says 172.21.15.90:36060) 2024-04-14T15:15:56.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:56.036+0000 7f4768ce3700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f476410a6d0 0x7f4764101810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:48442/0 (socket says 172.21.15.90:48442) 2024-04-14T15:15:56.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:56.036+0000 7f4769ce5700 1 -- 172.21.15.90:0/1095192300 learned_addr learned my addr 172.21.15.90:0/1095192300 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:15:56.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:56.036+0000 7f475a7fc700 10 client.?.objecter ms_handle_connect 0x7f476410e270 2024-04-14T15:15:56.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:56.036+0000 7f475a7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:15:56.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:56.036+0000 7f475a7fc700 10 client.?.objecter ms_handle_connect 0x7f476410a6d0 2024-04-14T15:15:56.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:56.036+0000 7f475a7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:15:56.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:56.036+0000 7f475a7fc700 1 -- 172.21.15.90:0/1095192300 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 840231629 0 0) 0x7f47641aa030 con 0x7f476410e270 2024-04-14T15:15:56.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:56.036+0000 7f475a7fc700 1 -- 172.21.15.90:0/1095192300 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f47500031f0 con 0x7f476410e270 2024-04-14T15:15:56.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:56.036+0000 7f475a7fc700 1 -- 172.21.15.90:0/1095192300 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 642472477 0 0) 0x7f47641070c0 con 0x7f476410a6d0 2024-04-14T15:15:56.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:56.036+0000 7f475a7fc700 1 -- 172.21.15.90:0/1095192300 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f47641aa030 con 0x7f476410a6d0 2024-04-14T15:15:56.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:56.036+0000 7f475a7fc700 10 client.?.objecter ms_handle_connect 0x7f47641023f0 2024-04-14T15:15:56.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:56.036+0000 7f475a7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:15:56.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:56.036+0000 7f475a7fc700 1 -- 172.21.15.90:0/1095192300 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3376108142 0 0) 0x7f47641aa030 con 0x7f476410a6d0 2024-04-14T15:15:56.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:56.037+0000 7f475a7fc700 1 -- 172.21.15.90:0/1095192300 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f47641070c0 con 0x7f476410a6d0 2024-04-14T15:15:56.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:56.037+0000 7f475a7fc700 1 -- 172.21.15.90:0/1095192300 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2500861051 0 0) 0x7f47641a8e50 con 0x7f47641023f0 2024-04-14T15:15:56.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:56.037+0000 7f475a7fc700 1 -- 172.21.15.90:0/1095192300 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f47641aa030 con 0x7f47641023f0 2024-04-14T15:15:56.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:56.037+0000 7f475a7fc700 1 -- 172.21.15.90:0/1095192300 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f4760004500 con 0x7f476410a6d0 2024-04-14T15:15:56.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:56.037+0000 7f475a7fc700 1 -- 172.21.15.90:0/1095192300 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2883211080 0 0) 0x7f47500031f0 con 0x7f476410e270 2024-04-14T15:15:56.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:56.037+0000 7f475a7fc700 1 -- 172.21.15.90:0/1095192300 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f47641a8e50 con 0x7f476410e270 2024-04-14T15:15:56.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:56.037+0000 7f475a7fc700 1 -- 172.21.15.90:0/1095192300 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f475c001bf0 con 0x7f476410e270 2024-04-14T15:15:56.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:56.037+0000 7f475a7fc700 1 -- 172.21.15.90:0/1095192300 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2815484153 0 0) 0x7f47641070c0 con 0x7f476410a6d0 2024-04-14T15:15:56.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:56.037+0000 7f475a7fc700 1 -- 172.21.15.90:0/1095192300 >> v1:172.21.15.90:6790/0 conn(0x7f476410e270 legacy=0x7f47641a6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:56.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:56.037+0000 7f475a7fc700 1 -- 172.21.15.90:0/1095192300 >> v1:172.21.15.92:6789/0 conn(0x7f47641023f0 legacy=0x7f4764101100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:56.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:56.037+0000 7f475a7fc700 1 -- 172.21.15.90:0/1095192300 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47641ab210 con 0x7f476410a6d0 2024-04-14T15:15:56.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:56.037+0000 7f476b748700 1 -- 172.21.15.90:0/1095192300 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f47641a7e40 con 0x7f476410a6d0 2024-04-14T15:15:56.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:56.037+0000 7f476b748700 10 client.36924.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:15:56.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:56.037+0000 7f476b748700 1 -- 172.21.15.90:0/1095192300 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f47641a8300 con 0x7f476410a6d0 2024-04-14T15:15:56.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:56.037+0000 7f476b748700 1 librados: init done 2024-04-14T15:15:56.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:56.037+0000 7f475a7fc700 1 -- 172.21.15.90:0/1095192300 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f4760001b30 con 0x7f476410a6d0 2024-04-14T15:15:56.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:56.037+0000 7f475a7fc700 1 -- 172.21.15.90:0/1095192300 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f4760005050 con 0x7f476410a6d0 2024-04-14T15:15:56.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:56.038+0000 7f476b748700 1 -- 172.21.15.90:0/1095192300 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f476406a170 con 0x7f476410a6d0 2024-04-14T15:15:56.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:56.039+0000 7f475a7fc700 1 -- 172.21.15.90:0/1095192300 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f4760005290 con 0x7f476410a6d0 2024-04-14T15:15:56.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:56.039+0000 7f475a7fc700 1 -- 172.21.15.90:0/1095192300 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(288..288 src has 51..288) ==== 9012+0+0 (unknown 780467465 0 0) 0x7f476008afe0 con 0x7f476410a6d0 2024-04-14T15:15:56.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:56.039+0000 7f475a7fc700 10 client.36924.objecter ms_dispatch 0x7f4764001550 osd_map(288..288 src has 51..288) 2024-04-14T15:15:56.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:56.039+0000 7f475a7fc700 3 client.36924.objecter handle_osd_map got epochs [288,288] > 0 2024-04-14T15:15:56.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:56.039+0000 7f475a7fc700 3 client.36924.objecter handle_osd_map decoding full epoch 288 2024-04-14T15:15:56.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:56.039+0000 7f475a7fc700 20 client.36924.objecter dump_active .. 0 homeless 2024-04-14T15:15:56.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:56.039+0000 7f475a7fc700 10 client.36924.objecter ms_handle_connect 0x7f4750071150 2024-04-14T15:15:56.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:56.044+0000 7f475a7fc700 1 -- 172.21.15.90:0/1095192300 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f47600576c0 con 0x7f476410a6d0 2024-04-14T15:15:56.127 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:15:56.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:56.152 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:56.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:15:56.164 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:55 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:15:56.164 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:55 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2688309294' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:15:56.164 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:55 smithi090 ceph-mon[41670]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:15:56.164 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:55 smithi090 ceph-mon[41670]: from='client.46865 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi090-93363-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi090-93363-50"}]': finished 2024-04-14T15:15:56.164 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:55 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3501572542' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi090-93342-41"}]': finished 2024-04-14T15:15:56.164 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:55 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:15:56.164 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:55 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2688309294' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-14T15:15:56.164 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:55 smithi090 ceph-mon[41670]: osdmap e288: 8 total, 8 up, 8 in 2024-04-14T15:15:56.165 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:55 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3501572542' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi090-93342-41"}]: dispatch 2024-04-14T15:15:56.165 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:55 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:15:56.165 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:55 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:15:56.165 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:55 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:15:56.165 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:55 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2688309294' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:15:56.165 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:55 smithi090 ceph-mon[30399]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:15:56.165 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:55 smithi090 ceph-mon[30399]: from='client.46865 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi090-93363-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi090-93363-50"}]': finished 2024-04-14T15:15:56.166 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:55 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3501572542' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi090-93342-41"}]': finished 2024-04-14T15:15:56.166 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:55 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:15:56.166 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:55 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2688309294' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-14T15:15:56.166 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:55 smithi090 ceph-mon[30399]: osdmap e288: 8 total, 8 up, 8 in 2024-04-14T15:15:56.166 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:55 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3501572542' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi090-93342-41"}]: dispatch 2024-04-14T15:15:56.166 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:55 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:15:56.166 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:55 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:15:56.253 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:15:56.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:56.266+0000 7f476b748700 1 -- 172.21.15.90:0/1095192300 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} v 0) -- 0x7f47641063c0 con 0x7f476410a6d0 2024-04-14T15:15:56.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:56.281 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:56.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:15:56.339 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-14T15:15:56.339 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:15:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:55 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:15:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:55 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2688309294' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:15:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:55 smithi092 ceph-mon[33966]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:15:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:55 smithi092 ceph-mon[33966]: from='client.46865 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi090-93363-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi090-93363-50"}]': finished 2024-04-14T15:15:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:55 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3501572542' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi090-93342-41"}]': finished 2024-04-14T15:15:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:55 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:15:56.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:55 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2688309294' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-14T15:15:56.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:55 smithi092 ceph-mon[33966]: osdmap e288: 8 total, 8 up, 8 in 2024-04-14T15:15:56.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:55 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3501572542' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi090-93342-41"}]: dispatch 2024-04-14T15:15:56.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:55 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:15:56.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:55 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:15:56.382 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:15:56.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:56.407 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:56.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:15:56.508 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:15:56.536 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:56.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:56.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:15:56.638 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:15:56.665 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:56.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:56.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:15:56.767 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:15:56.794 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:56.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:56.795 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:15:56.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:56.855+0000 7f475a7fc700 1 -- 172.21.15.90:0/1095192300 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]=0 set osd.6 primary-affinity to 1 (8655362) v289) ==== 133+0+0 (unknown 1344053466 0 0) 0x7f476005b370 con 0x7f476410a6d0 2024-04-14T15:15:56.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:56.858+0000 7f476b748700 10 librados: watch_flush enter 2024-04-14T15:15:56.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:56.859+0000 7f476b748700 10 librados: watch_flush exit 2024-04-14T15:15:56.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:56.859+0000 7f476b748700 20 client.36924.objecter shutdown clearing up homeless session... 2024-04-14T15:15:56.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:56.859+0000 7f476b748700 10 client.36924.objecter successfully canceled tick 2024-04-14T15:15:56.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:56.860+0000 7f476b748700 1 -- 172.21.15.90:0/1095192300 >> v1:172.21.15.90:6800/420935427 conn(0x7f4750071150 legacy=0x7f4750073610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:56.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:56.860+0000 7f476b748700 1 -- 172.21.15.90:0/1095192300 >> v1:172.21.15.90:6789/0 conn(0x7f476410a6d0 legacy=0x7f4764101810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:15:56.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:56.860+0000 7f476b748700 1 -- 172.21.15.90:0/1095192300 shutdown_connections 2024-04-14T15:15:56.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:56.860+0000 7f476b748700 1 -- 172.21.15.90:0/1095192300 >> 172.21.15.90:0/1095192300 conn(0x7f4764076aa0 msgr2=0x7f4764100070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:15:56.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:56.861+0000 7f476b748700 1 -- 172.21.15.90:0/1095192300 shutdown_connections 2024-04-14T15:15:56.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:56.862+0000 7f476b748700 1 -- 172.21.15.90:0/1095192300 wait complete. 2024-04-14T15:15:56.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:15:56.862+0000 7f476b748700 1 librados: shutdown 2024-04-14T15:15:56.866 INFO:teuthology.orchestra.run.smithi090.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-04-14T15:15:56.896 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:15:56.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:56.925 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:56.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:15:57.016 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-14T15:15:57.026 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:15:57.085 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:57.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:57.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:15:57.189 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:15:57.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:56 smithi090 ceph-mon[41670]: pgmap v377: 238 pgs: 32 unknown, 206 active+clean; 4.4 MiB data, 559 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-14T15:15:57.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:56 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1095192300' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-04-14T15:15:57.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:56 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3501572542' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi090-93342-41"}]': finished 2024-04-14T15:15:57.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:56 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-35", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:15:57.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:56 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1095192300' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-04-14T15:15:57.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:56 smithi090 ceph-mon[41670]: osdmap e289: 8 total, 8 up, 8 in 2024-04-14T15:15:57.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:56 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-35"}]: dispatch 2024-04-14T15:15:57.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:56 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-35"}]: dispatch 2024-04-14T15:15:57.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:56 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2635358863' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi090-93342-42"}]: dispatch 2024-04-14T15:15:57.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:56 smithi090 ceph-mon[41670]: from='client.48769 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi090-93342-42"}]: dispatch 2024-04-14T15:15:57.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:56 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2635358863' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi090-93342-42"}]: dispatch 2024-04-14T15:15:57.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:56 smithi090 ceph-mon[41670]: from='client.48769 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi090-93342-42"}]: dispatch 2024-04-14T15:15:57.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:56 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2635358863' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi090-93342-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:57.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:56 smithi090 ceph-mon[41670]: from='client.48769 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi090-93342-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:57.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:56 smithi090 ceph-mon[30399]: pgmap v377: 238 pgs: 32 unknown, 206 active+clean; 4.4 MiB data, 559 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-14T15:15:57.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:56 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1095192300' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-04-14T15:15:57.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:56 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3501572542' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi090-93342-41"}]': finished 2024-04-14T15:15:57.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:56 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-35", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:15:57.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:56 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1095192300' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-04-14T15:15:57.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:56 smithi090 ceph-mon[30399]: osdmap e289: 8 total, 8 up, 8 in 2024-04-14T15:15:57.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:56 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-35"}]: dispatch 2024-04-14T15:15:57.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:56 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-35"}]: dispatch 2024-04-14T15:15:57.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:56 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2635358863' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi090-93342-42"}]: dispatch 2024-04-14T15:15:57.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:56 smithi090 ceph-mon[30399]: from='client.48769 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi090-93342-42"}]: dispatch 2024-04-14T15:15:57.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:56 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2635358863' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi090-93342-42"}]: dispatch 2024-04-14T15:15:57.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:56 smithi090 ceph-mon[30399]: from='client.48769 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi090-93342-42"}]: dispatch 2024-04-14T15:15:57.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:56 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2635358863' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi090-93342-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:57.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:56 smithi090 ceph-mon[30399]: from='client.48769 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi090-93342-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:57.215 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:57.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:57.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:15:57.270 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:15:57.316 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:15:57.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:57.341 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:57.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:15:57.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:56 smithi092 ceph-mon[33966]: pgmap v377: 238 pgs: 32 unknown, 206 active+clean; 4.4 MiB data, 559 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-14T15:15:57.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:56 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1095192300' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-04-14T15:15:57.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:56 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3501572542' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi090-93342-41"}]': finished 2024-04-14T15:15:57.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:56 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-35", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:15:57.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:56 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1095192300' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-04-14T15:15:57.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:56 smithi092 ceph-mon[33966]: osdmap e289: 8 total, 8 up, 8 in 2024-04-14T15:15:57.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:56 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-35"}]: dispatch 2024-04-14T15:15:57.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:56 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-35"}]: dispatch 2024-04-14T15:15:57.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:56 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2635358863' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi090-93342-42"}]: dispatch 2024-04-14T15:15:57.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:56 smithi092 ceph-mon[33966]: from='client.48769 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi090-93342-42"}]: dispatch 2024-04-14T15:15:57.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:56 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2635358863' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi090-93342-42"}]: dispatch 2024-04-14T15:15:57.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:56 smithi092 ceph-mon[33966]: from='client.48769 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi090-93342-42"}]: dispatch 2024-04-14T15:15:57.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:56 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2635358863' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi090-93342-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:57.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:56 smithi092 ceph-mon[33966]: from='client.48769 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi090-93342-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:15:57.442 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:15:57.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:57.469 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:57.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:15:57.569 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:15:57.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:57.595 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:57.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:15:57.696 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:15:57.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:57.722 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:57.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:15:57.823 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:15:57.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:57.850 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:57.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:15:57.915 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:15:57.951 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:15:57.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:57.976 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:57.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:15:58.077 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:15:58.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:58.102 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:58.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:15:58.203 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:15:58.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:58.227 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:58.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:15:58.328 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:15:58.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:58.355 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:58.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:15:58.456 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:15:58.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:58.481 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:58.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:15:58.582 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:15:58.606 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:58.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:58.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:15:58.707 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:15:58.732 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:58.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:58.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:15:58.834 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:15:58.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:58.858 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:58.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:15:58.959 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:15:58.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:58.985 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:58.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:15:59.086 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:15:59.111 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:59.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:59.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:15:59.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:58 smithi092 ceph-mon[33966]: pgmap v380: 246 pgs: 40 unknown, 206 active+clean; 4.4 MiB data, 559 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-14T15:15:59.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:58 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-35"}]': finished 2024-04-14T15:15:59.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:58 smithi092 ceph-mon[33966]: from='client.48769 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi090-93342-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:15:59.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:58 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-35", "mode": "writeback"}]: dispatch 2024-04-14T15:15:59.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:58 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2635358863' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi090-93342-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi090-93342-42"}]: dispatch 2024-04-14T15:15:59.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:58 smithi092 ceph-mon[33966]: osdmap e290: 8 total, 8 up, 8 in 2024-04-14T15:15:59.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:58 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-35", "mode": "writeback"}]: dispatch 2024-04-14T15:15:59.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:58 smithi092 ceph-mon[33966]: from='client.48769 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi090-93342-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi090-93342-42"}]: dispatch 2024-04-14T15:15:59.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:58 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3032682252' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi090-93363-50"}]: dispatch 2024-04-14T15:15:59.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:58 smithi092 ceph-mon[33966]: from='client.46865 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi090-93363-50"}]: dispatch 2024-04-14T15:15:59.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:58 smithi090 ceph-mon[30399]: pgmap v380: 246 pgs: 40 unknown, 206 active+clean; 4.4 MiB data, 559 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-14T15:15:59.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:58 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-35"}]': finished 2024-04-14T15:15:59.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:58 smithi090 ceph-mon[30399]: from='client.48769 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi090-93342-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:15:59.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:58 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-35", "mode": "writeback"}]: dispatch 2024-04-14T15:15:59.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:58 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2635358863' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi090-93342-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi090-93342-42"}]: dispatch 2024-04-14T15:15:59.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:58 smithi090 ceph-mon[30399]: osdmap e290: 8 total, 8 up, 8 in 2024-04-14T15:15:59.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:58 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-35", "mode": "writeback"}]: dispatch 2024-04-14T15:15:59.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:58 smithi090 ceph-mon[30399]: from='client.48769 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi090-93342-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi090-93342-42"}]: dispatch 2024-04-14T15:15:59.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:58 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3032682252' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi090-93363-50"}]: dispatch 2024-04-14T15:15:59.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:58 smithi090 ceph-mon[30399]: from='client.46865 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi090-93363-50"}]: dispatch 2024-04-14T15:15:59.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:58 smithi090 ceph-mon[41670]: pgmap v380: 246 pgs: 40 unknown, 206 active+clean; 4.4 MiB data, 559 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-14T15:15:59.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:58 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-35"}]': finished 2024-04-14T15:15:59.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:58 smithi090 ceph-mon[41670]: from='client.48769 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi090-93342-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:15:59.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:58 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-35", "mode": "writeback"}]: dispatch 2024-04-14T15:15:59.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:58 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2635358863' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi090-93342-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi090-93342-42"}]: dispatch 2024-04-14T15:15:59.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:58 smithi090 ceph-mon[41670]: osdmap e290: 8 total, 8 up, 8 in 2024-04-14T15:15:59.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:58 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-35", "mode": "writeback"}]: dispatch 2024-04-14T15:15:59.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:58 smithi090 ceph-mon[41670]: from='client.48769 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi090-93342-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi090-93342-42"}]: dispatch 2024-04-14T15:15:59.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:58 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3032682252' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi090-93363-50"}]: dispatch 2024-04-14T15:15:59.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:58 smithi090 ceph-mon[41670]: from='client.46865 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi090-93363-50"}]: dispatch 2024-04-14T15:15:59.213 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:15:59.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:59.237 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:59.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:15:59.266 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-04-14T15:15:59.266 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:15:59.338 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:15:59.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:59.362 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:59.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:15:59.463 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:15:59.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:59.489 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:59.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:15:59.591 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:15:59.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:59.617 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:59.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:15:59.719 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:15:59.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:59.745 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:15:59.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:15:59.846 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:15:59.854 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-14T15:15:59.873 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:15:59.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:15:59.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:15:59.975 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:16:00.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:00.001 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:00.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:16:00.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.087+0000 7f1d636b8700 1 -- 172.21.15.90:0/2022801342 >> v1:172.21.15.90:6789/0 conn(0x7f1d5c107760 legacy=0x7f1d5c107b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:00.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.087+0000 7f1d636b8700 1 -- 172.21.15.90:0/2022801342 shutdown_connections 2024-04-14T15:16:00.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.087+0000 7f1d636b8700 1 -- 172.21.15.90:0/2022801342 >> 172.21.15.90:0/2022801342 conn(0x7f1d5c0fd290 msgr2=0x7f1d5c0ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:16:00.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.087+0000 7f1d636b8700 1 -- 172.21.15.90:0/2022801342 shutdown_connections 2024-04-14T15:16:00.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.087+0000 7f1d636b8700 1 -- 172.21.15.90:0/2022801342 wait complete. 2024-04-14T15:16:00.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.088+0000 7f1d636b8700 1 librados: starting msgr at 2024-04-14T15:16:00.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.088+0000 7f1d636b8700 1 librados: starting objecter 2024-04-14T15:16:00.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.088+0000 7f1d636b8700 1 Processor -- start 2024-04-14T15:16:00.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.088+0000 7f1d636b8700 1 -- start start 2024-04-14T15:16:00.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.088+0000 7f1d636b8700 1 librados: setting wanted keys 2024-04-14T15:16:00.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.088+0000 7f1d636b8700 1 librados: calling monclient init 2024-04-14T15:16:00.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.088+0000 7f1d636b8700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1d5c1a6780 con 0x7f1d5c10a610 2024-04-14T15:16:00.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.088+0000 7f1d636b8700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1d5c1a79a0 con 0x7f1d5c107760 2024-04-14T15:16:00.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.088+0000 7f1d636b8700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1d5c1a8be0 con 0x7f1d5c10e1e0 2024-04-14T15:16:00.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.088+0000 7f1d61c55700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f1d5c10e1e0 0x7f1d5c1a4e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:36106/0 (socket says 172.21.15.90:36106) 2024-04-14T15:16:00.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.088+0000 7f1d61c55700 1 -- 172.21.15.90:0/1055788082 learned_addr learned my addr 172.21.15.90:0/1055788082 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:16:00.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.088+0000 7f1d527fc700 10 client.?.objecter ms_handle_connect 0x7f1d5c10a610 2024-04-14T15:16:00.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.088+0000 7f1d527fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:16:00.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.088+0000 7f1d527fc700 10 client.?.objecter ms_handle_connect 0x7f1d5c10e1e0 2024-04-14T15:16:00.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.088+0000 7f1d527fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:16:00.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.088+0000 7f1d527fc700 10 client.?.objecter ms_handle_connect 0x7f1d5c107760 2024-04-14T15:16:00.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.088+0000 7f1d527fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:16:00.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.088+0000 7f1d527fc700 1 -- 172.21.15.90:0/1055788082 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3239473772 0 0) 0x7f1d5c1a6780 con 0x7f1d5c10a610 2024-04-14T15:16:00.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.089+0000 7f1d527fc700 1 -- 172.21.15.90:0/1055788082 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1d480031f0 con 0x7f1d5c10a610 2024-04-14T15:16:00.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.089+0000 7f1d527fc700 1 -- 172.21.15.90:0/1055788082 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 889924594 0 0) 0x7f1d5c1a8be0 con 0x7f1d5c10e1e0 2024-04-14T15:16:00.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.089+0000 7f1d527fc700 1 -- 172.21.15.90:0/1055788082 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1d5c1a6780 con 0x7f1d5c10e1e0 2024-04-14T15:16:00.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.089+0000 7f1d527fc700 1 -- 172.21.15.90:0/1055788082 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1036603040 0 0) 0x7f1d5c1a79a0 con 0x7f1d5c107760 2024-04-14T15:16:00.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.089+0000 7f1d527fc700 1 -- 172.21.15.90:0/1055788082 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1d5c1a8be0 con 0x7f1d5c107760 2024-04-14T15:16:00.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.089+0000 7f1d527fc700 1 -- 172.21.15.90:0/1055788082 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 435968062 0 0) 0x7f1d480031f0 con 0x7f1d5c10a610 2024-04-14T15:16:00.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.089+0000 7f1d527fc700 1 -- 172.21.15.90:0/1055788082 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1d5c1a79a0 con 0x7f1d5c10a610 2024-04-14T15:16:00.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.089+0000 7f1d527fc700 1 -- 172.21.15.90:0/1055788082 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f1d58001070 con 0x7f1d5c10a610 2024-04-14T15:16:00.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.089+0000 7f1d527fc700 1 -- 172.21.15.90:0/1055788082 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 705852398 0 0) 0x7f1d5c1a6780 con 0x7f1d5c10e1e0 2024-04-14T15:16:00.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.089+0000 7f1d527fc700 1 -- 172.21.15.90:0/1055788082 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1d480031f0 con 0x7f1d5c10e1e0 2024-04-14T15:16:00.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.089+0000 7f1d527fc700 1 -- 172.21.15.90:0/1055788082 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 721050590 0 0) 0x7f1d5c1a8be0 con 0x7f1d5c107760 2024-04-14T15:16:00.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.089+0000 7f1d527fc700 1 -- 172.21.15.90:0/1055788082 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1d5c1a6780 con 0x7f1d5c107760 2024-04-14T15:16:00.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.089+0000 7f1d527fc700 1 -- 172.21.15.90:0/1055788082 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f1d540014b0 con 0x7f1d5c10e1e0 2024-04-14T15:16:00.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.089+0000 7f1d527fc700 1 -- 172.21.15.90:0/1055788082 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f1d4c001840 con 0x7f1d5c107760 2024-04-14T15:16:00.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.089+0000 7f1d527fc700 1 -- 172.21.15.90:0/1055788082 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3757364088 0 0) 0x7f1d5c1a79a0 con 0x7f1d5c10a610 2024-04-14T15:16:00.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.089+0000 7f1d527fc700 1 -- 172.21.15.90:0/1055788082 >> v1:172.21.15.90:6790/0 conn(0x7f1d5c10e1e0 legacy=0x7f1d5c1a4e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:00.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.089+0000 7f1d527fc700 1 -- 172.21.15.90:0/1055788082 >> v1:172.21.15.92:6789/0 conn(0x7f1d5c107760 legacy=0x7f1d5c10d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:00.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.089+0000 7f1d527fc700 1 -- 172.21.15.90:0/1055788082 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d5c1a9e20 con 0x7f1d5c10a610 2024-04-14T15:16:00.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.089+0000 7f1d636b8700 1 -- 172.21.15.90:0/1055788082 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1d5c1a8e10 con 0x7f1d5c10a610 2024-04-14T15:16:00.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.089+0000 7f1d636b8700 10 client.36942.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:16:00.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.089+0000 7f1d636b8700 1 -- 172.21.15.90:0/1055788082 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f1d5c1a92d0 con 0x7f1d5c10a610 2024-04-14T15:16:00.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.089+0000 7f1d636b8700 1 librados: init done 2024-04-14T15:16:00.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.089+0000 7f1d527fc700 1 -- 172.21.15.90:0/1055788082 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f1d58003b20 con 0x7f1d5c10a610 2024-04-14T15:16:00.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.089+0000 7f1d527fc700 1 -- 172.21.15.90:0/1055788082 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f1d58004af0 con 0x7f1d5c10a610 2024-04-14T15:16:00.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.090+0000 7f1d636b8700 1 -- 172.21.15.90:0/1055788082 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f1d40005130 con 0x7f1d5c10a610 2024-04-14T15:16:00.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.091+0000 7f1d527fc700 1 -- 172.21.15.90:0/1055788082 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f1d58003680 con 0x7f1d5c10a610 2024-04-14T15:16:00.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.091+0000 7f1d527fc700 1 -- 172.21.15.90:0/1055788082 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(292..292 src has 51..292) ==== 8905+0+0 (unknown 424862730 0 0) 0x7f1d58089a60 con 0x7f1d5c10a610 2024-04-14T15:16:00.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.091+0000 7f1d527fc700 10 client.36942.objecter ms_dispatch 0x7f1d5c001550 osd_map(292..292 src has 51..292) 2024-04-14T15:16:00.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.092+0000 7f1d527fc700 3 client.36942.objecter handle_osd_map got epochs [292,292] > 0 2024-04-14T15:16:00.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.092+0000 7f1d527fc700 3 client.36942.objecter handle_osd_map decoding full epoch 292 2024-04-14T15:16:00.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.092+0000 7f1d527fc700 20 client.36942.objecter dump_active .. 0 homeless 2024-04-14T15:16:00.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.092+0000 7f1d527fc700 10 client.36942.objecter ms_handle_connect 0x7f1d48071310 2024-04-14T15:16:00.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.092+0000 7f1d527fc700 1 -- 172.21.15.90:0/1055788082 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v292) ==== 54+0+16 (unknown 3273446081 0 2420000192) 0x7f1d58089e20 con 0x7f1d5c10a610 2024-04-14T15:16:00.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.092+0000 7f1d636b8700 10 client.36942.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:16:00.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.092+0000 7f1d636b8700 15 client.36942.objecter _session_command_op_assign -1 1 2024-04-14T15:16:00.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.092+0000 7f1d636b8700 20 client.36942.objecter _get_session s=0x7f1d40005410 osd=0 3 2024-04-14T15:16:00.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.092+0000 7f1d636b8700 20 client.36942.objecter put_session s=0x7f1d40005410 osd=0 3 2024-04-14T15:16:00.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.092+0000 7f1d636b8700 20 client.36942.objecter _get_session s=0x7f1d40005410 osd=0 3 2024-04-14T15:16:00.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.092+0000 7f1d636b8700 15 client.36942.objecter _session_command_op_remove -1 1 2024-04-14T15:16:00.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.092+0000 7f1d636b8700 20 client.36942.objecter get_session s=0x7f1d40005410 osd=0 3 2024-04-14T15:16:00.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.092+0000 7f1d636b8700 15 client.36942.objecter _session_command_op_assign 0 1 2024-04-14T15:16:00.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.092+0000 7f1d636b8700 20 client.36942.objecter put_session s=0x7f1d40005410 osd=0 4 2024-04-14T15:16:00.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.092+0000 7f1d636b8700 10 client.36942.objecter _send_command 1 2024-04-14T15:16:00.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.092+0000 7f1d636b8700 1 -- 172.21.15.90:0/1055788082 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1d40004b80 con 0x7f1d40004760 2024-04-14T15:16:00.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.092+0000 7f1d527fc700 10 client.36942.objecter ms_handle_connect 0x7f1d40004760 2024-04-14T15:16:00.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.093+0000 7f1d527fc700 1 -- 172.21.15.90:0/1055788082 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1d40004b80 con 0x7f1d40004760 2024-04-14T15:16:00.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.093+0000 7f1d527fc700 10 client.36942.objecter ms_dispatch 0x7f1d5c001550 command_reply(tid 1: 0 ) 2024-04-14T15:16:00.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.093+0000 7f1d527fc700 10 client.36942.objecter _finish_command 1 = system:0 2024-04-14T15:16:00.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.093+0000 7f1d527fc700 20 client.36942.objecter put_session s=0x7f1d40005410 osd=0 4 2024-04-14T15:16:00.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.093+0000 7f1d527fc700 15 client.36942.objecter _session_command_op_remove 0 1 2024-04-14T15:16:00.103 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:16:00.110 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:16:00.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.110+0000 7f1d636b8700 10 client.36942.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:16:00.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.110+0000 7f1d636b8700 15 client.36942.objecter _session_command_op_assign -1 2 2024-04-14T15:16:00.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.110+0000 7f1d636b8700 20 client.36942.objecter _get_session s=0x7f1d40005410 osd=0 3 2024-04-14T15:16:00.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.110+0000 7f1d636b8700 20 client.36942.objecter put_session s=0x7f1d40005410 osd=0 3 2024-04-14T15:16:00.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.110+0000 7f1d636b8700 20 client.36942.objecter _get_session s=0x7f1d40005410 osd=0 3 2024-04-14T15:16:00.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.110+0000 7f1d636b8700 15 client.36942.objecter _session_command_op_remove -1 2 2024-04-14T15:16:00.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.110+0000 7f1d636b8700 20 client.36942.objecter get_session s=0x7f1d40005410 osd=0 3 2024-04-14T15:16:00.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.110+0000 7f1d636b8700 15 client.36942.objecter _session_command_op_assign 0 2 2024-04-14T15:16:00.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.110+0000 7f1d636b8700 20 client.36942.objecter put_session s=0x7f1d40005410 osd=0 4 2024-04-14T15:16:00.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.110+0000 7f1d636b8700 10 client.36942.objecter _send_command 2 2024-04-14T15:16:00.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.110+0000 7f1d636b8700 1 -- 172.21.15.90:0/1055788082 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f1d40009c30 con 0x7f1d40004760 2024-04-14T15:16:00.114 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:59 smithi090 ceph-mon[41670]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:16:00.114 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:59 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-35", "mode": "writeback"}]': finished 2024-04-14T15:16:00.114 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:59 smithi090 ceph-mon[41670]: from='client.46865 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi090-93363-50"}]': finished 2024-04-14T15:16:00.114 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:59 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3032682252' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi090-93363-50"}]: dispatch 2024-04-14T15:16:00.115 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:59 smithi090 ceph-mon[41670]: osdmap e291: 8 total, 8 up, 8 in 2024-04-14T15:16:00.115 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:59 smithi090 ceph-mon[41670]: from='client.46865 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi090-93363-50"}]: dispatch 2024-04-14T15:16:00.115 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:59 smithi090 ceph-mon[41670]: from='client.48769 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi090-93342-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi090-93342-42"}]': finished 2024-04-14T15:16:00.115 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:59 smithi090 ceph-mon[41670]: from='client.46865 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi090-93363-50"}]': finished 2024-04-14T15:16:00.115 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:15:59 smithi090 ceph-mon[41670]: osdmap e292: 8 total, 8 up, 8 in 2024-04-14T15:16:00.116 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:59 smithi090 ceph-mon[30399]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:16:00.116 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:59 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-35", "mode": "writeback"}]': finished 2024-04-14T15:16:00.116 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:59 smithi090 ceph-mon[30399]: from='client.46865 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi090-93363-50"}]': finished 2024-04-14T15:16:00.116 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:59 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3032682252' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi090-93363-50"}]: dispatch 2024-04-14T15:16:00.116 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:59 smithi090 ceph-mon[30399]: osdmap e291: 8 total, 8 up, 8 in 2024-04-14T15:16:00.116 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:59 smithi090 ceph-mon[30399]: from='client.46865 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi090-93363-50"}]: dispatch 2024-04-14T15:16:00.116 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:59 smithi090 ceph-mon[30399]: from='client.48769 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi090-93342-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi090-93342-42"}]': finished 2024-04-14T15:16:00.116 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:59 smithi090 ceph-mon[30399]: from='client.46865 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi090-93363-50"}]': finished 2024-04-14T15:16:00.116 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:15:59 smithi090 ceph-mon[30399]: osdmap e292: 8 total, 8 up, 8 in 2024-04-14T15:16:00.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.112+0000 7f1d527fc700 1 -- 172.21.15.90:0/1055788082 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1d40009c30 con 0x7f1d40004760 2024-04-14T15:16:00.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.112+0000 7f1d527fc700 10 client.36942.objecter ms_dispatch 0x7f1d5c001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:16:00.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.112+0000 7f1d527fc700 10 client.36942.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:16:00.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.112+0000 7f1d527fc700 20 client.36942.objecter put_session s=0x7f1d40005410 osd=0 4 2024-04-14T15:16:00.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.112+0000 7f1d527fc700 15 client.36942.objecter _session_command_op_remove 0 2 2024-04-14T15:16:00.117 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:16:00.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.112+0000 7f1d636b8700 10 client.36942.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:16:00.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.112+0000 7f1d636b8700 15 client.36942.objecter _session_command_op_assign -1 3 2024-04-14T15:16:00.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.112+0000 7f1d636b8700 20 client.36942.objecter _get_session s=0x7f1d40004140 osd=1 3 2024-04-14T15:16:00.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.112+0000 7f1d636b8700 20 client.36942.objecter put_session s=0x7f1d40004140 osd=1 3 2024-04-14T15:16:00.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.112+0000 7f1d636b8700 20 client.36942.objecter _get_session s=0x7f1d40004140 osd=1 3 2024-04-14T15:16:00.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.112+0000 7f1d636b8700 15 client.36942.objecter _session_command_op_remove -1 3 2024-04-14T15:16:00.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.112+0000 7f1d636b8700 20 client.36942.objecter get_session s=0x7f1d40004140 osd=1 3 2024-04-14T15:16:00.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.112+0000 7f1d636b8700 15 client.36942.objecter _session_command_op_assign 1 3 2024-04-14T15:16:00.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.112+0000 7f1d636b8700 20 client.36942.objecter put_session s=0x7f1d40004140 osd=1 4 2024-04-14T15:16:00.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.112+0000 7f1d636b8700 10 client.36942.objecter _send_command 3 2024-04-14T15:16:00.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.112+0000 7f1d636b8700 1 -- 172.21.15.90:0/1055788082 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f1d400021b0 con 0x7f1d400042b0 2024-04-14T15:16:00.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.113+0000 7f1d527fc700 10 client.36942.objecter ms_handle_connect 0x7f1d400042b0 2024-04-14T15:16:00.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.113+0000 7f1d527fc700 1 -- 172.21.15.90:0/1055788082 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1d400021b0 con 0x7f1d400042b0 2024-04-14T15:16:00.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.113+0000 7f1d527fc700 10 client.36942.objecter ms_dispatch 0x7f1d5c001550 command_reply(tid 3: 0 ) 2024-04-14T15:16:00.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.113+0000 7f1d527fc700 10 client.36942.objecter _finish_command 3 = system:0 2024-04-14T15:16:00.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.113+0000 7f1d527fc700 20 client.36942.objecter put_session s=0x7f1d40004140 osd=1 4 2024-04-14T15:16:00.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.113+0000 7f1d527fc700 15 client.36942.objecter _session_command_op_remove 1 3 2024-04-14T15:16:00.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:59 smithi092 ceph-mon[33966]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:16:00.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:59 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-35", "mode": "writeback"}]': finished 2024-04-14T15:16:00.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:59 smithi092 ceph-mon[33966]: from='client.46865 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi090-93363-50"}]': finished 2024-04-14T15:16:00.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:59 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3032682252' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi090-93363-50"}]: dispatch 2024-04-14T15:16:00.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:59 smithi092 ceph-mon[33966]: osdmap e291: 8 total, 8 up, 8 in 2024-04-14T15:16:00.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:59 smithi092 ceph-mon[33966]: from='client.46865 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi090-93363-50"}]: dispatch 2024-04-14T15:16:00.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:59 smithi092 ceph-mon[33966]: from='client.48769 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi090-93342-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi090-93342-42"}]': finished 2024-04-14T15:16:00.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:59 smithi092 ceph-mon[33966]: from='client.46865 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi090-93363-50"}]': finished 2024-04-14T15:16:00.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:15:59 smithi092 ceph-mon[33966]: osdmap e292: 8 total, 8 up, 8 in 2024-04-14T15:16:00.126 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:00.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:00.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:16:00.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.131+0000 7f1d636b8700 10 client.36942.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:16:00.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.131+0000 7f1d636b8700 15 client.36942.objecter _session_command_op_assign -1 4 2024-04-14T15:16:00.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.131+0000 7f1d636b8700 20 client.36942.objecter _get_session s=0x7f1d40004140 osd=1 3 2024-04-14T15:16:00.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.131+0000 7f1d636b8700 20 client.36942.objecter put_session s=0x7f1d40004140 osd=1 3 2024-04-14T15:16:00.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.131+0000 7f1d636b8700 20 client.36942.objecter _get_session s=0x7f1d40004140 osd=1 3 2024-04-14T15:16:00.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.131+0000 7f1d636b8700 15 client.36942.objecter _session_command_op_remove -1 4 2024-04-14T15:16:00.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.131+0000 7f1d636b8700 20 client.36942.objecter get_session s=0x7f1d40004140 osd=1 3 2024-04-14T15:16:00.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.131+0000 7f1d636b8700 15 client.36942.objecter _session_command_op_assign 1 4 2024-04-14T15:16:00.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.131+0000 7f1d636b8700 20 client.36942.objecter put_session s=0x7f1d40004140 osd=1 4 2024-04-14T15:16:00.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.131+0000 7f1d636b8700 10 client.36942.objecter _send_command 4 2024-04-14T15:16:00.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.131+0000 7f1d636b8700 1 -- 172.21.15.90:0/1055788082 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f1d40001630 con 0x7f1d400042b0 2024-04-14T15:16:00.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.133+0000 7f1d527fc700 1 -- 172.21.15.90:0/1055788082 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1d40001630 con 0x7f1d400042b0 2024-04-14T15:16:00.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.133+0000 7f1d527fc700 10 client.36942.objecter ms_dispatch 0x7f1d5c001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:16:00.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.133+0000 7f1d527fc700 10 client.36942.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:16:00.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.133+0000 7f1d527fc700 20 client.36942.objecter put_session s=0x7f1d40004140 osd=1 4 2024-04-14T15:16:00.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.133+0000 7f1d527fc700 15 client.36942.objecter _session_command_op_remove 1 4 2024-04-14T15:16:00.134 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:16:00.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.133+0000 7f1d636b8700 10 client.36942.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:16:00.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.133+0000 7f1d636b8700 15 client.36942.objecter _session_command_op_assign -1 5 2024-04-14T15:16:00.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.133+0000 7f1d636b8700 20 client.36942.objecter _get_session s=0x7f1d40001a30 osd=2 3 2024-04-14T15:16:00.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.133+0000 7f1d636b8700 20 client.36942.objecter put_session s=0x7f1d40001a30 osd=2 3 2024-04-14T15:16:00.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.133+0000 7f1d636b8700 20 client.36942.objecter _get_session s=0x7f1d40001a30 osd=2 3 2024-04-14T15:16:00.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.133+0000 7f1d636b8700 15 client.36942.objecter _session_command_op_remove -1 5 2024-04-14T15:16:00.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.133+0000 7f1d636b8700 20 client.36942.objecter get_session s=0x7f1d40001a30 osd=2 3 2024-04-14T15:16:00.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.133+0000 7f1d636b8700 15 client.36942.objecter _session_command_op_assign 2 5 2024-04-14T15:16:00.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.133+0000 7f1d636b8700 20 client.36942.objecter put_session s=0x7f1d40001a30 osd=2 4 2024-04-14T15:16:00.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.133+0000 7f1d636b8700 10 client.36942.objecter _send_command 5 2024-04-14T15:16:00.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.133+0000 7f1d636b8700 1 -- 172.21.15.90:0/1055788082 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f1d40013160 con 0x7f1d4000aac0 2024-04-14T15:16:00.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.134+0000 7f1d527fc700 10 client.36942.objecter ms_handle_connect 0x7f1d4000aac0 2024-04-14T15:16:00.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.134+0000 7f1d527fc700 1 -- 172.21.15.90:0/1055788082 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1d40013160 con 0x7f1d4000aac0 2024-04-14T15:16:00.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.134+0000 7f1d527fc700 10 client.36942.objecter ms_dispatch 0x7f1d5c001550 command_reply(tid 5: 0 ) 2024-04-14T15:16:00.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.134+0000 7f1d527fc700 10 client.36942.objecter _finish_command 5 = system:0 2024-04-14T15:16:00.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.134+0000 7f1d527fc700 20 client.36942.objecter put_session s=0x7f1d40001a30 osd=2 4 2024-04-14T15:16:00.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.134+0000 7f1d527fc700 15 client.36942.objecter _session_command_op_remove 2 5 2024-04-14T15:16:00.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.151+0000 7f1d636b8700 10 client.36942.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:16:00.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.151+0000 7f1d636b8700 15 client.36942.objecter _session_command_op_assign -1 6 2024-04-14T15:16:00.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.151+0000 7f1d636b8700 20 client.36942.objecter _get_session s=0x7f1d40001a30 osd=2 3 2024-04-14T15:16:00.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.151+0000 7f1d636b8700 20 client.36942.objecter put_session s=0x7f1d40001a30 osd=2 3 2024-04-14T15:16:00.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.151+0000 7f1d636b8700 20 client.36942.objecter _get_session s=0x7f1d40001a30 osd=2 3 2024-04-14T15:16:00.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.151+0000 7f1d636b8700 15 client.36942.objecter _session_command_op_remove -1 6 2024-04-14T15:16:00.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.151+0000 7f1d636b8700 20 client.36942.objecter get_session s=0x7f1d40001a30 osd=2 3 2024-04-14T15:16:00.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.151+0000 7f1d636b8700 15 client.36942.objecter _session_command_op_assign 2 6 2024-04-14T15:16:00.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.151+0000 7f1d636b8700 20 client.36942.objecter put_session s=0x7f1d40001a30 osd=2 4 2024-04-14T15:16:00.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.151+0000 7f1d636b8700 10 client.36942.objecter _send_command 6 2024-04-14T15:16:00.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.151+0000 7f1d636b8700 1 -- 172.21.15.90:0/1055788082 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f1d4000a150 con 0x7f1d4000aac0 2024-04-14T15:16:00.153 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:16:00.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.152+0000 7f1d527fc700 1 -- 172.21.15.90:0/1055788082 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1d4000a150 con 0x7f1d4000aac0 2024-04-14T15:16:00.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.152+0000 7f1d527fc700 10 client.36942.objecter ms_dispatch 0x7f1d5c001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:16:00.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.152+0000 7f1d527fc700 10 client.36942.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:16:00.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.152+0000 7f1d527fc700 20 client.36942.objecter put_session s=0x7f1d40001a30 osd=2 4 2024-04-14T15:16:00.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.152+0000 7f1d527fc700 15 client.36942.objecter _session_command_op_remove 2 6 2024-04-14T15:16:00.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.152+0000 7f1d636b8700 10 client.36942.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:16:00.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.152+0000 7f1d636b8700 15 client.36942.objecter _session_command_op_assign -1 7 2024-04-14T15:16:00.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.153+0000 7f1d636b8700 20 client.36942.objecter _get_session s=0x7f1d4000a3b0 osd=3 3 2024-04-14T15:16:00.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.153+0000 7f1d636b8700 20 client.36942.objecter put_session s=0x7f1d4000a3b0 osd=3 3 2024-04-14T15:16:00.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.153+0000 7f1d636b8700 20 client.36942.objecter _get_session s=0x7f1d4000a3b0 osd=3 3 2024-04-14T15:16:00.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.153+0000 7f1d636b8700 15 client.36942.objecter _session_command_op_remove -1 7 2024-04-14T15:16:00.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.153+0000 7f1d636b8700 20 client.36942.objecter get_session s=0x7f1d4000a3b0 osd=3 3 2024-04-14T15:16:00.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.153+0000 7f1d636b8700 15 client.36942.objecter _session_command_op_assign 3 7 2024-04-14T15:16:00.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.153+0000 7f1d636b8700 20 client.36942.objecter put_session s=0x7f1d4000a3b0 osd=3 4 2024-04-14T15:16:00.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.153+0000 7f1d636b8700 10 client.36942.objecter _send_command 7 2024-04-14T15:16:00.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.153+0000 7f1d636b8700 1 -- 172.21.15.90:0/1055788082 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f1d40013eb0 con 0x7f1d40013310 2024-04-14T15:16:00.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.153+0000 7f1d527fc700 10 client.36942.objecter ms_handle_connect 0x7f1d40013310 2024-04-14T15:16:00.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.153+0000 7f1d527fc700 1 -- 172.21.15.90:0/1055788082 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1d40013eb0 con 0x7f1d40013310 2024-04-14T15:16:00.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.153+0000 7f1d527fc700 10 client.36942.objecter ms_dispatch 0x7f1d5c001550 command_reply(tid 7: 0 ) 2024-04-14T15:16:00.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.153+0000 7f1d527fc700 10 client.36942.objecter _finish_command 7 = system:0 2024-04-14T15:16:00.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.153+0000 7f1d527fc700 20 client.36942.objecter put_session s=0x7f1d4000a3b0 osd=3 4 2024-04-14T15:16:00.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.153+0000 7f1d527fc700 15 client.36942.objecter _session_command_op_remove 3 7 2024-04-14T15:16:00.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.170+0000 7f1d636b8700 10 client.36942.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:16:00.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.170+0000 7f1d636b8700 15 client.36942.objecter _session_command_op_assign -1 8 2024-04-14T15:16:00.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.170+0000 7f1d636b8700 20 client.36942.objecter _get_session s=0x7f1d4000a3b0 osd=3 3 2024-04-14T15:16:00.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.170+0000 7f1d636b8700 20 client.36942.objecter put_session s=0x7f1d4000a3b0 osd=3 3 2024-04-14T15:16:00.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.170+0000 7f1d636b8700 20 client.36942.objecter _get_session s=0x7f1d4000a3b0 osd=3 3 2024-04-14T15:16:00.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.170+0000 7f1d636b8700 15 client.36942.objecter _session_command_op_remove -1 8 2024-04-14T15:16:00.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.170+0000 7f1d636b8700 20 client.36942.objecter get_session s=0x7f1d4000a3b0 osd=3 3 2024-04-14T15:16:00.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.170+0000 7f1d636b8700 15 client.36942.objecter _session_command_op_assign 3 8 2024-04-14T15:16:00.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.170+0000 7f1d636b8700 20 client.36942.objecter put_session s=0x7f1d4000a3b0 osd=3 4 2024-04-14T15:16:00.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.170+0000 7f1d636b8700 10 client.36942.objecter _send_command 8 2024-04-14T15:16:00.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.170+0000 7f1d636b8700 1 -- 172.21.15.90:0/1055788082 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f1d40014860 con 0x7f1d40013310 2024-04-14T15:16:00.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.171+0000 7f1d527fc700 1 -- 172.21.15.90:0/1055788082 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1d40014860 con 0x7f1d40013310 2024-04-14T15:16:00.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.171+0000 7f1d527fc700 10 client.36942.objecter ms_dispatch 0x7f1d5c001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:16:00.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.171+0000 7f1d527fc700 10 client.36942.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:16:00.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.171+0000 7f1d527fc700 20 client.36942.objecter put_session s=0x7f1d4000a3b0 osd=3 4 2024-04-14T15:16:00.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.171+0000 7f1d527fc700 15 client.36942.objecter _session_command_op_remove 3 8 2024-04-14T15:16:00.173 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:16:00.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.172+0000 7f1d636b8700 10 client.36942.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:16:00.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.172+0000 7f1d636b8700 15 client.36942.objecter _session_command_op_assign -1 9 2024-04-14T15:16:00.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.172+0000 7f1d636b8700 20 client.36942.objecter _get_session s=0x7f1d400159e0 osd=4 3 2024-04-14T15:16:00.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.172+0000 7f1d636b8700 20 client.36942.objecter put_session s=0x7f1d400159e0 osd=4 3 2024-04-14T15:16:00.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.172+0000 7f1d636b8700 20 client.36942.objecter _get_session s=0x7f1d400159e0 osd=4 3 2024-04-14T15:16:00.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.172+0000 7f1d636b8700 15 client.36942.objecter _session_command_op_remove -1 9 2024-04-14T15:16:00.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.172+0000 7f1d636b8700 20 client.36942.objecter get_session s=0x7f1d400159e0 osd=4 3 2024-04-14T15:16:00.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.172+0000 7f1d636b8700 15 client.36942.objecter _session_command_op_assign 4 9 2024-04-14T15:16:00.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.172+0000 7f1d636b8700 20 client.36942.objecter put_session s=0x7f1d400159e0 osd=4 4 2024-04-14T15:16:00.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.172+0000 7f1d636b8700 10 client.36942.objecter _send_command 9 2024-04-14T15:16:00.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.172+0000 7f1d636b8700 1 -- 172.21.15.90:0/1055788082 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f1d4001c960 con 0x7f1d40018d20 2024-04-14T15:16:00.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.172+0000 7f1d527fc700 10 client.36942.objecter ms_handle_connect 0x7f1d40018d20 2024-04-14T15:16:00.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.173+0000 7f1d527fc700 1 -- 172.21.15.90:0/1055788082 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1d4001c960 con 0x7f1d40018d20 2024-04-14T15:16:00.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.173+0000 7f1d527fc700 10 client.36942.objecter ms_dispatch 0x7f1d5c001550 command_reply(tid 9: 0 ) 2024-04-14T15:16:00.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.173+0000 7f1d527fc700 10 client.36942.objecter _finish_command 9 = system:0 2024-04-14T15:16:00.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.173+0000 7f1d527fc700 20 client.36942.objecter put_session s=0x7f1d400159e0 osd=4 4 2024-04-14T15:16:00.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.173+0000 7f1d527fc700 15 client.36942.objecter _session_command_op_remove 4 9 2024-04-14T15:16:00.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.189+0000 7f1d636b8700 10 client.36942.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:16:00.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.189+0000 7f1d636b8700 15 client.36942.objecter _session_command_op_assign -1 10 2024-04-14T15:16:00.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.189+0000 7f1d636b8700 20 client.36942.objecter _get_session s=0x7f1d400159e0 osd=4 3 2024-04-14T15:16:00.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.189+0000 7f1d636b8700 20 client.36942.objecter put_session s=0x7f1d400159e0 osd=4 3 2024-04-14T15:16:00.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.189+0000 7f1d636b8700 20 client.36942.objecter _get_session s=0x7f1d400159e0 osd=4 3 2024-04-14T15:16:00.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.189+0000 7f1d636b8700 15 client.36942.objecter _session_command_op_remove -1 10 2024-04-14T15:16:00.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.189+0000 7f1d636b8700 20 client.36942.objecter get_session s=0x7f1d400159e0 osd=4 3 2024-04-14T15:16:00.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.189+0000 7f1d636b8700 15 client.36942.objecter _session_command_op_assign 4 10 2024-04-14T15:16:00.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.189+0000 7f1d636b8700 20 client.36942.objecter put_session s=0x7f1d400159e0 osd=4 4 2024-04-14T15:16:00.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.189+0000 7f1d636b8700 10 client.36942.objecter _send_command 10 2024-04-14T15:16:00.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.189+0000 7f1d636b8700 1 -- 172.21.15.90:0/1055788082 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f1d4001c920 con 0x7f1d40018d20 2024-04-14T15:16:00.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.191+0000 7f1d527fc700 1 -- 172.21.15.90:0/1055788082 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1d4001c920 con 0x7f1d40018d20 2024-04-14T15:16:00.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.191+0000 7f1d527fc700 10 client.36942.objecter ms_dispatch 0x7f1d5c001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:16:00.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.191+0000 7f1d527fc700 10 client.36942.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:16:00.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.191+0000 7f1d527fc700 20 client.36942.objecter put_session s=0x7f1d400159e0 osd=4 4 2024-04-14T15:16:00.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.191+0000 7f1d527fc700 15 client.36942.objecter _session_command_op_remove 4 10 2024-04-14T15:16:00.192 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:16:00.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.192+0000 7f1d636b8700 10 client.36942.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:16:00.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.192+0000 7f1d636b8700 15 client.36942.objecter _session_command_op_assign -1 11 2024-04-14T15:16:00.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.192+0000 7f1d636b8700 20 client.36942.objecter _get_session s=0x7f1d4000e9a0 osd=5 3 2024-04-14T15:16:00.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.192+0000 7f1d636b8700 20 client.36942.objecter put_session s=0x7f1d4000e9a0 osd=5 3 2024-04-14T15:16:00.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.192+0000 7f1d636b8700 20 client.36942.objecter _get_session s=0x7f1d4000e9a0 osd=5 3 2024-04-14T15:16:00.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.192+0000 7f1d636b8700 15 client.36942.objecter _session_command_op_remove -1 11 2024-04-14T15:16:00.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.192+0000 7f1d636b8700 20 client.36942.objecter get_session s=0x7f1d4000e9a0 osd=5 3 2024-04-14T15:16:00.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.192+0000 7f1d636b8700 15 client.36942.objecter _session_command_op_assign 5 11 2024-04-14T15:16:00.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.192+0000 7f1d636b8700 20 client.36942.objecter put_session s=0x7f1d4000e9a0 osd=5 4 2024-04-14T15:16:00.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.192+0000 7f1d636b8700 10 client.36942.objecter _send_command 11 2024-04-14T15:16:00.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.192+0000 7f1d636b8700 1 -- 172.21.15.90:0/1055788082 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f1d40009a60 con 0x7f1d4001d300 2024-04-14T15:16:00.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.192+0000 7f1d527fc700 10 client.36942.objecter ms_handle_connect 0x7f1d4001d300 2024-04-14T15:16:00.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.193+0000 7f1d527fc700 1 -- 172.21.15.90:0/1055788082 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1d40009a60 con 0x7f1d4001d300 2024-04-14T15:16:00.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.193+0000 7f1d527fc700 10 client.36942.objecter ms_dispatch 0x7f1d5c001550 command_reply(tid 11: 0 ) 2024-04-14T15:16:00.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.193+0000 7f1d527fc700 10 client.36942.objecter _finish_command 11 = system:0 2024-04-14T15:16:00.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.193+0000 7f1d527fc700 20 client.36942.objecter put_session s=0x7f1d4000e9a0 osd=5 4 2024-04-14T15:16:00.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.193+0000 7f1d527fc700 15 client.36942.objecter _session_command_op_remove 5 11 2024-04-14T15:16:00.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.209+0000 7f1d636b8700 10 client.36942.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:16:00.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.209+0000 7f1d636b8700 15 client.36942.objecter _session_command_op_assign -1 12 2024-04-14T15:16:00.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.209+0000 7f1d636b8700 20 client.36942.objecter _get_session s=0x7f1d4000e9a0 osd=5 3 2024-04-14T15:16:00.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.209+0000 7f1d636b8700 20 client.36942.objecter put_session s=0x7f1d4000e9a0 osd=5 3 2024-04-14T15:16:00.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.209+0000 7f1d636b8700 20 client.36942.objecter _get_session s=0x7f1d4000e9a0 osd=5 3 2024-04-14T15:16:00.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.209+0000 7f1d636b8700 15 client.36942.objecter _session_command_op_remove -1 12 2024-04-14T15:16:00.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.209+0000 7f1d636b8700 20 client.36942.objecter get_session s=0x7f1d4000e9a0 osd=5 3 2024-04-14T15:16:00.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.209+0000 7f1d636b8700 15 client.36942.objecter _session_command_op_assign 5 12 2024-04-14T15:16:00.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.209+0000 7f1d636b8700 20 client.36942.objecter put_session s=0x7f1d4000e9a0 osd=5 4 2024-04-14T15:16:00.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.209+0000 7f1d636b8700 10 client.36942.objecter _send_command 12 2024-04-14T15:16:00.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.209+0000 7f1d636b8700 1 -- 172.21.15.90:0/1055788082 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f1d40009a60 con 0x7f1d4001d300 2024-04-14T15:16:00.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.211+0000 7f1d527fc700 1 -- 172.21.15.90:0/1055788082 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1d40009a60 con 0x7f1d4001d300 2024-04-14T15:16:00.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.211+0000 7f1d527fc700 10 client.36942.objecter ms_dispatch 0x7f1d5c001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:16:00.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.211+0000 7f1d527fc700 10 client.36942.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:16:00.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.211+0000 7f1d527fc700 20 client.36942.objecter put_session s=0x7f1d4000e9a0 osd=5 4 2024-04-14T15:16:00.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.211+0000 7f1d527fc700 15 client.36942.objecter _session_command_op_remove 5 12 2024-04-14T15:16:00.213 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:16:00.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.211+0000 7f1d636b8700 10 client.36942.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:16:00.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.211+0000 7f1d636b8700 15 client.36942.objecter _session_command_op_assign -1 13 2024-04-14T15:16:00.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.211+0000 7f1d636b8700 20 client.36942.objecter _get_session s=0x7f1d40009a60 osd=6 3 2024-04-14T15:16:00.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.211+0000 7f1d636b8700 20 client.36942.objecter put_session s=0x7f1d40009a60 osd=6 3 2024-04-14T15:16:00.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.211+0000 7f1d636b8700 20 client.36942.objecter _get_session s=0x7f1d40009a60 osd=6 3 2024-04-14T15:16:00.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.211+0000 7f1d636b8700 15 client.36942.objecter _session_command_op_remove -1 13 2024-04-14T15:16:00.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.211+0000 7f1d636b8700 20 client.36942.objecter get_session s=0x7f1d40009a60 osd=6 3 2024-04-14T15:16:00.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.211+0000 7f1d636b8700 15 client.36942.objecter _session_command_op_assign 6 13 2024-04-14T15:16:00.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.211+0000 7f1d636b8700 20 client.36942.objecter put_session s=0x7f1d40009a60 osd=6 4 2024-04-14T15:16:00.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.211+0000 7f1d636b8700 10 client.36942.objecter _send_command 13 2024-04-14T15:16:00.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.211+0000 7f1d636b8700 1 -- 172.21.15.90:0/1055788082 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f1d400275c0 con 0x7f1d4001e0a0 2024-04-14T15:16:00.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.212+0000 7f1d527fc700 10 client.36942.objecter ms_handle_connect 0x7f1d4001e0a0 2024-04-14T15:16:00.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.213+0000 7f1d527fc700 1 -- 172.21.15.90:0/1055788082 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1d400275c0 con 0x7f1d4001e0a0 2024-04-14T15:16:00.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.213+0000 7f1d527fc700 10 client.36942.objecter ms_dispatch 0x7f1d5c001550 command_reply(tid 13: 0 ) 2024-04-14T15:16:00.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.213+0000 7f1d527fc700 10 client.36942.objecter _finish_command 13 = system:0 2024-04-14T15:16:00.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.213+0000 7f1d527fc700 20 client.36942.objecter put_session s=0x7f1d40009a60 osd=6 4 2024-04-14T15:16:00.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.213+0000 7f1d527fc700 15 client.36942.objecter _session_command_op_remove 6 13 2024-04-14T15:16:00.228 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:16:00.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.229+0000 7f1d636b8700 10 client.36942.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:16:00.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.229+0000 7f1d636b8700 15 client.36942.objecter _session_command_op_assign -1 14 2024-04-14T15:16:00.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.229+0000 7f1d636b8700 20 client.36942.objecter _get_session s=0x7f1d40009a60 osd=6 3 2024-04-14T15:16:00.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.229+0000 7f1d636b8700 20 client.36942.objecter put_session s=0x7f1d40009a60 osd=6 3 2024-04-14T15:16:00.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.229+0000 7f1d636b8700 20 client.36942.objecter _get_session s=0x7f1d40009a60 osd=6 3 2024-04-14T15:16:00.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.229+0000 7f1d636b8700 15 client.36942.objecter _session_command_op_remove -1 14 2024-04-14T15:16:00.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.229+0000 7f1d636b8700 20 client.36942.objecter get_session s=0x7f1d40009a60 osd=6 3 2024-04-14T15:16:00.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.229+0000 7f1d636b8700 15 client.36942.objecter _session_command_op_assign 6 14 2024-04-14T15:16:00.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.229+0000 7f1d636b8700 20 client.36942.objecter put_session s=0x7f1d40009a60 osd=6 4 2024-04-14T15:16:00.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.229+0000 7f1d636b8700 10 client.36942.objecter _send_command 14 2024-04-14T15:16:00.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.229+0000 7f1d636b8700 1 -- 172.21.15.90:0/1055788082 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f1d400275c0 con 0x7f1d4001e0a0 2024-04-14T15:16:00.232 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:16:00.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.231+0000 7f1d527fc700 1 -- 172.21.15.90:0/1055788082 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1d400275c0 con 0x7f1d4001e0a0 2024-04-14T15:16:00.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.231+0000 7f1d527fc700 10 client.36942.objecter ms_dispatch 0x7f1d5c001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:16:00.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.231+0000 7f1d527fc700 10 client.36942.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:16:00.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.231+0000 7f1d527fc700 20 client.36942.objecter put_session s=0x7f1d40009a60 osd=6 4 2024-04-14T15:16:00.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.231+0000 7f1d527fc700 15 client.36942.objecter _session_command_op_remove 6 14 2024-04-14T15:16:00.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.231+0000 7f1d636b8700 10 client.36942.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:16:00.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.231+0000 7f1d636b8700 15 client.36942.objecter _session_command_op_assign -1 15 2024-04-14T15:16:00.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.231+0000 7f1d636b8700 20 client.36942.objecter _get_session s=0x7f1d400275c0 osd=7 3 2024-04-14T15:16:00.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.231+0000 7f1d636b8700 20 client.36942.objecter put_session s=0x7f1d400275c0 osd=7 3 2024-04-14T15:16:00.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.231+0000 7f1d636b8700 20 client.36942.objecter _get_session s=0x7f1d400275c0 osd=7 3 2024-04-14T15:16:00.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.231+0000 7f1d636b8700 15 client.36942.objecter _session_command_op_remove -1 15 2024-04-14T15:16:00.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.231+0000 7f1d636b8700 20 client.36942.objecter get_session s=0x7f1d400275c0 osd=7 3 2024-04-14T15:16:00.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.231+0000 7f1d636b8700 15 client.36942.objecter _session_command_op_assign 7 15 2024-04-14T15:16:00.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.231+0000 7f1d636b8700 20 client.36942.objecter put_session s=0x7f1d400275c0 osd=7 4 2024-04-14T15:16:00.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.231+0000 7f1d636b8700 10 client.36942.objecter _send_command 15 2024-04-14T15:16:00.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.231+0000 7f1d636b8700 1 -- 172.21.15.90:0/1055788082 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f1d400299c0 con 0x7f1d40027ce0 2024-04-14T15:16:00.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.231+0000 7f1d527fc700 10 client.36942.objecter ms_handle_connect 0x7f1d40027ce0 2024-04-14T15:16:00.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.232+0000 7f1d527fc700 1 -- 172.21.15.90:0/1055788082 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1d400299c0 con 0x7f1d40027ce0 2024-04-14T15:16:00.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.232+0000 7f1d527fc700 10 client.36942.objecter ms_dispatch 0x7f1d5c001550 command_reply(tid 15: 0 ) 2024-04-14T15:16:00.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.232+0000 7f1d527fc700 10 client.36942.objecter _finish_command 15 = system:0 2024-04-14T15:16:00.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.232+0000 7f1d527fc700 20 client.36942.objecter put_session s=0x7f1d400275c0 osd=7 4 2024-04-14T15:16:00.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.232+0000 7f1d527fc700 15 client.36942.objecter _session_command_op_remove 7 15 2024-04-14T15:16:00.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.251+0000 7f1d636b8700 10 client.36942.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:16:00.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.251+0000 7f1d636b8700 15 client.36942.objecter _session_command_op_assign -1 16 2024-04-14T15:16:00.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.251+0000 7f1d636b8700 20 client.36942.objecter _get_session s=0x7f1d400275c0 osd=7 3 2024-04-14T15:16:00.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.251+0000 7f1d636b8700 20 client.36942.objecter put_session s=0x7f1d400275c0 osd=7 3 2024-04-14T15:16:00.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.251+0000 7f1d636b8700 20 client.36942.objecter _get_session s=0x7f1d400275c0 osd=7 3 2024-04-14T15:16:00.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.251+0000 7f1d636b8700 15 client.36942.objecter _session_command_op_remove -1 16 2024-04-14T15:16:00.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.251+0000 7f1d636b8700 20 client.36942.objecter get_session s=0x7f1d400275c0 osd=7 3 2024-04-14T15:16:00.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.251+0000 7f1d636b8700 15 client.36942.objecter _session_command_op_assign 7 16 2024-04-14T15:16:00.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.251+0000 7f1d636b8700 20 client.36942.objecter put_session s=0x7f1d400275c0 osd=7 4 2024-04-14T15:16:00.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.251+0000 7f1d636b8700 10 client.36942.objecter _send_command 16 2024-04-14T15:16:00.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.251+0000 7f1d636b8700 1 -- 172.21.15.90:0/1055788082 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f1d400299c0 con 0x7f1d40027ce0 2024-04-14T15:16:00.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.253+0000 7f1d527fc700 1 -- 172.21.15.90:0/1055788082 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1d400299c0 con 0x7f1d40027ce0 2024-04-14T15:16:00.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.253+0000 7f1d527fc700 10 client.36942.objecter ms_dispatch 0x7f1d5c001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:16:00.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.253+0000 7f1d527fc700 10 client.36942.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:16:00.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.253+0000 7f1d527fc700 20 client.36942.objecter put_session s=0x7f1d400275c0 osd=7 4 2024-04-14T15:16:00.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.253+0000 7f1d527fc700 15 client.36942.objecter _session_command_op_remove 7 16 2024-04-14T15:16:00.254 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:16:00.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.253+0000 7f1d636b8700 10 librados: watch_flush enter 2024-04-14T15:16:00.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.253+0000 7f1d636b8700 10 librados: watch_flush exit 2024-04-14T15:16:00.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.253+0000 7f1d636b8700 10 client.36942.objecter close_session for osd.0 2024-04-14T15:16:00.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.253+0000 7f1d636b8700 1 -- 172.21.15.90:0/1055788082 >> v1:172.21.15.90:6801/3761806885 conn(0x7f1d40004760 legacy=0x7f1d40002780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:00.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.253+0000 7f1d636b8700 20 client.36942.objecter put_session s=0x7f1d40005410 osd=0 1 2024-04-14T15:16:00.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.253+0000 7f1d636b8700 10 client.36942.objecter close_session for osd.1 2024-04-14T15:16:00.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.253+0000 7f1d636b8700 1 -- 172.21.15.90:0/1055788082 >> v1:172.21.15.90:6805/4170361699 conn(0x7f1d400042b0 legacy=0x7f1d4000cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:00.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.253+0000 7f1d636b8700 20 client.36942.objecter put_session s=0x7f1d40004140 osd=1 1 2024-04-14T15:16:00.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.253+0000 7f1d636b8700 10 client.36942.objecter close_session for osd.2 2024-04-14T15:16:00.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.253+0000 7f1d636b8700 1 -- 172.21.15.90:0/1055788082 >> v1:172.21.15.90:6809/1686184251 conn(0x7f1d4000aac0 legacy=0x7f1d40011820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:00.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.253+0000 7f1d636b8700 20 client.36942.objecter put_session s=0x7f1d40001a30 osd=2 1 2024-04-14T15:16:00.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.253+0000 7f1d636b8700 10 client.36942.objecter close_session for osd.3 2024-04-14T15:16:00.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.253+0000 7f1d636b8700 1 -- 172.21.15.90:0/1055788082 >> v1:172.21.15.90:6813/2270166774 conn(0x7f1d40013310 legacy=0x7f1d400137a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:00.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.253+0000 7f1d636b8700 20 client.36942.objecter put_session s=0x7f1d4000a3b0 osd=3 1 2024-04-14T15:16:00.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.253+0000 7f1d636b8700 10 client.36942.objecter close_session for osd.4 2024-04-14T15:16:00.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.253+0000 7f1d636b8700 1 -- 172.21.15.90:0/1055788082 >> v1:172.21.15.92:6800/3887549733 conn(0x7f1d40018d20 legacy=0x7f1d4001b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:00.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.253+0000 7f1d636b8700 20 client.36942.objecter put_session s=0x7f1d400159e0 osd=4 1 2024-04-14T15:16:00.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.253+0000 7f1d636b8700 10 client.36942.objecter close_session for osd.5 2024-04-14T15:16:00.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.253+0000 7f1d636b8700 1 -- 172.21.15.90:0/1055788082 >> v1:172.21.15.92:6804/3244561440 conn(0x7f1d4001d300 legacy=0x7f1d4001d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:00.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.253+0000 7f1d636b8700 20 client.36942.objecter put_session s=0x7f1d4000e9a0 osd=5 1 2024-04-14T15:16:00.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.253+0000 7f1d636b8700 10 client.36942.objecter close_session for osd.6 2024-04-14T15:16:00.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.253+0000 7f1d636b8700 1 -- 172.21.15.90:0/1055788082 >> v1:172.21.15.92:6808/636125963 conn(0x7f1d4001e0a0 legacy=0x7f1d40025ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:00.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.253+0000 7f1d636b8700 20 client.36942.objecter put_session s=0x7f1d40009a60 osd=6 1 2024-04-14T15:16:00.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.253+0000 7f1d636b8700 10 client.36942.objecter close_session for osd.7 2024-04-14T15:16:00.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.253+0000 7f1d636b8700 1 -- 172.21.15.90:0/1055788082 >> v1:172.21.15.92:6812/1773706716 conn(0x7f1d40027ce0 legacy=0x7f1d40028140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:00.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.253+0000 7f1d636b8700 20 client.36942.objecter put_session s=0x7f1d400275c0 osd=7 1 2024-04-14T15:16:00.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.253+0000 7f1d636b8700 20 client.36942.objecter shutdown clearing up homeless session... 2024-04-14T15:16:00.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.253+0000 7f1d636b8700 10 client.36942.objecter successfully canceled tick 2024-04-14T15:16:00.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.253+0000 7f1d636b8700 1 -- 172.21.15.90:0/1055788082 >> v1:172.21.15.90:6800/420935427 conn(0x7f1d48071310 legacy=0x7f1d480737d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:00.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.253+0000 7f1d636b8700 1 -- 172.21.15.90:0/1055788082 >> v1:172.21.15.90:6789/0 conn(0x7f1d5c10a610 legacy=0x7f1d5c1a1690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:00.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.254+0000 7f1d61c55700 1 -- 172.21.15.90:0/1055788082 reap_dead start 2024-04-14T15:16:00.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.254+0000 7f1d61c55700 1 -- 172.21.15.90:0/1055788082 reap_dead start 2024-04-14T15:16:00.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.254+0000 7f1d61c55700 1 -- 172.21.15.90:0/1055788082 reap_dead start 2024-04-14T15:16:00.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.254+0000 7f1d61c55700 1 -- 172.21.15.90:0/1055788082 reap_dead start 2024-04-14T15:16:00.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.254+0000 7f1d61c55700 1 -- 172.21.15.90:0/1055788082 reap_dead start 2024-04-14T15:16:00.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.254+0000 7f1d636b8700 1 -- 172.21.15.90:0/1055788082 shutdown_connections 2024-04-14T15:16:00.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.254+0000 7f1d636b8700 1 -- 172.21.15.90:0/1055788082 >> 172.21.15.90:0/1055788082 conn(0x7f1d5c0fd290 msgr2=0x7f1d5c110e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:16:00.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.254+0000 7f1d636b8700 1 -- 172.21.15.90:0/1055788082 shutdown_connections 2024-04-14T15:16:00.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.254+0000 7f1d636b8700 1 -- 172.21.15.90:0/1055788082 wait complete. 2024-04-14T15:16:00.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:00.254+0000 7f1d636b8700 1 librados: shutdown 2024-04-14T15:16:00.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:00.260 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:00.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:16:00.260 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:16:00.260 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:16:00.260 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:16:00.260 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:16:00.260 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:16:00.261 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:16:00.261 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:16:00.261 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:16:00.360 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:16:00.386 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:00.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:00.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:16:00.488 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:16:00.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:00.512 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:00.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:16:00.613 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:16:00.636 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd unset noscrub 2024-04-14T15:16:00.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:00.638 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:00.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:16:00.740 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:16:00.763 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:00.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:00.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:16:00.865 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:16:00.898 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:00.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:00.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:16:00.972 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:16:01.000 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:01.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:01.025 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:01.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:01.117 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:00 smithi092 ceph-mon[33966]: pgmap v383: 238 pgs: 5 unknown, 233 active+clean; 4.4 MiB data, 564 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:16:01.118 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:00 smithi092 ceph-mon[33966]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:16:01.118 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:00 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1055788082' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:16:01.118 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:00 smithi092 ceph-mon[33966]: osdmap e293: 8 total, 8 up, 8 in 2024-04-14T15:16:01.118 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:00 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1358676595' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi090-93363-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:16:01.118 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:00 smithi092 ceph-mon[33966]: from='client.48775 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi090-93363-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:16:01.126 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:01.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:01.151 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:01.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:01.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:00 smithi090 ceph-mon[41670]: pgmap v383: 238 pgs: 5 unknown, 233 active+clean; 4.4 MiB data, 564 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:16:01.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:00 smithi090 ceph-mon[41670]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:16:01.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:00 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1055788082' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:16:01.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:00 smithi090 ceph-mon[41670]: osdmap e293: 8 total, 8 up, 8 in 2024-04-14T15:16:01.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:00 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1358676595' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi090-93363-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:16:01.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:00 smithi090 ceph-mon[41670]: from='client.48775 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi090-93363-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:16:01.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:00 smithi090 ceph-mon[30399]: pgmap v383: 238 pgs: 5 unknown, 233 active+clean; 4.4 MiB data, 564 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:16:01.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:00 smithi090 ceph-mon[30399]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:16:01.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:00 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1055788082' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:16:01.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:00 smithi090 ceph-mon[30399]: osdmap e293: 8 total, 8 up, 8 in 2024-04-14T15:16:01.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:00 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1358676595' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi090-93363-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:16:01.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:00 smithi090 ceph-mon[30399]: from='client.48775 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi090-93363-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:16:01.252 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:16:01.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:01.276 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:01.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:16:01.377 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:16:01.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:01.402 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:01.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:16:01.503 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:16:01.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:01.528 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:01.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:16:01.629 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:16:01.652 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:01.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:01.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:16:01.754 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:16:01.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:01.779 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:01.779 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:16:01.880 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:01.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:01.909 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:01.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:01.962 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-04-14T15:16:01.962 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-04-14T15:16:01.962 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-04-14T15:16:01.962 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-04-14T15:16:01.962 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-04-14T15:16:01.963 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:01 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:16:01.963 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:01 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:16:01.963 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:01 smithi092 ceph-mon[33966]: from='client.48775 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi090-93363-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:16:01.963 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:01 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:16:01.963 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:01 smithi092 ceph-mon[33966]: osdmap e294: 8 total, 8 up, 8 in 2024-04-14T15:16:01.963 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:01 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2635358863' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi090-93342-42"}]: dispatch 2024-04-14T15:16:01.963 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:01 smithi092 ceph-mon[33966]: from='client.48769 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi090-93342-42"}]: dispatch 2024-04-14T15:16:01.963 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:01 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-35"}]: dispatch 2024-04-14T15:16:01.963 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:01 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-35"}]: dispatch 2024-04-14T15:16:02.011 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:02.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:02.035 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:02.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:02.136 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:16:02.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:02.161 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:02.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:16:02.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:01 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:16:02.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:01 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:16:02.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:01 smithi090 ceph-mon[30399]: from='client.48775 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi090-93363-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:16:02.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:01 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:16:02.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:01 smithi090 ceph-mon[30399]: osdmap e294: 8 total, 8 up, 8 in 2024-04-14T15:16:02.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:01 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2635358863' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi090-93342-42"}]: dispatch 2024-04-14T15:16:02.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:01 smithi090 ceph-mon[30399]: from='client.48769 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi090-93342-42"}]: dispatch 2024-04-14T15:16:02.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:01 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-35"}]: dispatch 2024-04-14T15:16:02.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:01 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-35"}]: dispatch 2024-04-14T15:16:02.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:01 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:16:02.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:01 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:16:02.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:01 smithi090 ceph-mon[41670]: from='client.48775 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi090-93363-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:16:02.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:01 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:16:02.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:01 smithi090 ceph-mon[41670]: osdmap e294: 8 total, 8 up, 8 in 2024-04-14T15:16:02.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:01 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2635358863' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi090-93342-42"}]: dispatch 2024-04-14T15:16:02.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:01 smithi090 ceph-mon[41670]: from='client.48769 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi090-93342-42"}]: dispatch 2024-04-14T15:16:02.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:01 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-35"}]: dispatch 2024-04-14T15:16:02.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:01 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-35"}]: dispatch 2024-04-14T15:16:02.263 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:02.286 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:02.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:02.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:02.388 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:16:02.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:02.413 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:02.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:16:02.514 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:16:02.539 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:02.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:02.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:16:02.641 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:02.665 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:02.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:02.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:02.670 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-14T15:16:02.670 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-14T15:16:02.670 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-04-14T15:16:02.670 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd dump -f json-pretty 2024-04-14T15:16:02.701 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-14T15:16:02.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:02.756+0000 7fef95e4d700 1 -- 172.21.15.90:0/493568946 >> v1:172.21.15.90:6789/0 conn(0x7fef90108880 legacy=0x7fef90108c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:02.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:02.756+0000 7fef95e4d700 1 -- 172.21.15.90:0/493568946 shutdown_connections 2024-04-14T15:16:02.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:02.756+0000 7fef95e4d700 1 -- 172.21.15.90:0/493568946 >> 172.21.15.90:0/493568946 conn(0x7fef900fe3d0 msgr2=0x7fef901007f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:16:02.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:02.756+0000 7fef95e4d700 1 -- 172.21.15.90:0/493568946 shutdown_connections 2024-04-14T15:16:02.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:02.756+0000 7fef95e4d700 1 -- 172.21.15.90:0/493568946 wait complete. 2024-04-14T15:16:02.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:02.756+0000 7fef95e4d700 1 librados: starting msgr at 2024-04-14T15:16:02.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:02.756+0000 7fef95e4d700 1 librados: starting objecter 2024-04-14T15:16:02.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:02.756+0000 7fef95e4d700 1 Processor -- start 2024-04-14T15:16:02.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:02.757+0000 7fef95e4d700 1 -- start start 2024-04-14T15:16:02.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:02.757+0000 7fef95e4d700 1 librados: setting wanted keys 2024-04-14T15:16:02.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:02.757+0000 7fef95e4d700 1 librados: calling monclient init 2024-04-14T15:16:02.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:02.757+0000 7fef95e4d700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fef901a79a0 con 0x7fef90108880 2024-04-14T15:16:02.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:02.757+0000 7fef95e4d700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fef901a8be0 con 0x7fef9010b730 2024-04-14T15:16:02.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:02.757+0000 7fef95e4d700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fef901a9e20 con 0x7fef9010f300 2024-04-14T15:16:02.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:02.757+0000 7fef8f7fe700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fef90108880 0x7fef9010ea90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:35834/0 (socket says 172.21.15.90:35834) 2024-04-14T15:16:02.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:02.757+0000 7fef8f7fe700 1 -- 172.21.15.90:0/2054244879 learned_addr learned my addr 172.21.15.90:0/2054244879 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:16:02.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:02.757+0000 7fef8cff9700 10 client.?.objecter ms_handle_connect 0x7fef9010f300 2024-04-14T15:16:02.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:02.757+0000 7fef8cff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:16:02.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:02.757+0000 7fef8cff9700 10 client.?.objecter ms_handle_connect 0x7fef9010b730 2024-04-14T15:16:02.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:02.757+0000 7fef8cff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:16:02.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:02.757+0000 7fef8cff9700 10 client.?.objecter ms_handle_connect 0x7fef90108880 2024-04-14T15:16:02.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:02.757+0000 7fef8cff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:16:02.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:02.757+0000 7fef8cff9700 1 -- 172.21.15.90:0/2054244879 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3834116717 0 0) 0x7fef901a9e20 con 0x7fef9010f300 2024-04-14T15:16:02.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:02.757+0000 7fef8cff9700 1 -- 172.21.15.90:0/2054244879 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef780035b0 con 0x7fef9010f300 2024-04-14T15:16:02.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:02.757+0000 7fef8cff9700 1 -- 172.21.15.90:0/2054244879 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3943545165 0 0) 0x7fef901a8be0 con 0x7fef9010b730 2024-04-14T15:16:02.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:02.757+0000 7fef8cff9700 1 -- 172.21.15.90:0/2054244879 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef901a9e20 con 0x7fef9010b730 2024-04-14T15:16:02.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:02.757+0000 7fef8cff9700 1 -- 172.21.15.90:0/2054244879 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3283062423 0 0) 0x7fef901a79a0 con 0x7fef90108880 2024-04-14T15:16:02.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:02.758+0000 7fef8cff9700 1 -- 172.21.15.90:0/2054244879 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef901a8be0 con 0x7fef90108880 2024-04-14T15:16:02.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:02.758+0000 7fef8cff9700 1 -- 172.21.15.90:0/2054244879 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2374161748 0 0) 0x7fef901a9e20 con 0x7fef9010b730 2024-04-14T15:16:02.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:02.758+0000 7fef8cff9700 1 -- 172.21.15.90:0/2054244879 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fef901a79a0 con 0x7fef9010b730 2024-04-14T15:16:02.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:02.758+0000 7fef8cff9700 1 -- 172.21.15.90:0/2054244879 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fef84001110 con 0x7fef9010b730 2024-04-14T15:16:02.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:02.758+0000 7fef8cff9700 1 -- 172.21.15.90:0/2054244879 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3765695276 0 0) 0x7fef780035b0 con 0x7fef9010f300 2024-04-14T15:16:02.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:02.758+0000 7fef8cff9700 1 -- 172.21.15.90:0/2054244879 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fef901a9e20 con 0x7fef9010f300 2024-04-14T15:16:02.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:02.758+0000 7fef8cff9700 1 -- 172.21.15.90:0/2054244879 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4006036633 0 0) 0x7fef901a8be0 con 0x7fef90108880 2024-04-14T15:16:02.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:02.758+0000 7fef8cff9700 1 -- 172.21.15.90:0/2054244879 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fef780035b0 con 0x7fef90108880 2024-04-14T15:16:02.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:02.758+0000 7fef8cff9700 1 -- 172.21.15.90:0/2054244879 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fef88001bb0 con 0x7fef9010f300 2024-04-14T15:16:02.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:02.758+0000 7fef8cff9700 1 -- 172.21.15.90:0/2054244879 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fef80004540 con 0x7fef90108880 2024-04-14T15:16:02.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:02.758+0000 7fef8cff9700 1 -- 172.21.15.90:0/2054244879 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1987003267 0 0) 0x7fef901a79a0 con 0x7fef9010b730 2024-04-14T15:16:02.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:02.758+0000 7fef8cff9700 1 -- 172.21.15.90:0/2054244879 >> v1:172.21.15.90:6790/0 conn(0x7fef9010f300 legacy=0x7fef901a60a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:02.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:02.758+0000 7fef8cff9700 1 -- 172.21.15.90:0/2054244879 >> v1:172.21.15.90:6789/0 conn(0x7fef90108880 legacy=0x7fef9010ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:02.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:02.758+0000 7fef8cff9700 1 -- 172.21.15.90:0/2054244879 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef901ab060 con 0x7fef9010b730 2024-04-14T15:16:02.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:02.758+0000 7fef95e4d700 1 -- 172.21.15.90:0/2054244879 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fef901a8e10 con 0x7fef9010b730 2024-04-14T15:16:02.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:02.758+0000 7fef95e4d700 10 client.48787.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:16:02.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:02.758+0000 7fef95e4d700 1 -- 172.21.15.90:0/2054244879 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fef901a92d0 con 0x7fef9010b730 2024-04-14T15:16:02.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:02.758+0000 7fef95e4d700 1 librados: init done 2024-04-14T15:16:02.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:02.758+0000 7fef8cff9700 1 -- 172.21.15.90:0/2054244879 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fef84003b20 con 0x7fef9010b730 2024-04-14T15:16:02.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:02.758+0000 7fef8cff9700 1 -- 172.21.15.90:0/2054244879 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fef84004a20 con 0x7fef9010b730 2024-04-14T15:16:02.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:02.759+0000 7fef95e4d700 1 -- 172.21.15.90:0/2054244879 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef7c005120 con 0x7fef9010b730 2024-04-14T15:16:02.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:02.760+0000 7fef8cff9700 1 -- 172.21.15.90:0/2054244879 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fef84003680 con 0x7fef9010b730 2024-04-14T15:16:02.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:02.760+0000 7fef8cff9700 1 -- 172.21.15.90:0/2054244879 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(294..294 src has 51..294) ==== 8879+0+0 (unknown 3144606938 0 0) 0x7fef84089b80 con 0x7fef9010b730 2024-04-14T15:16:02.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:02.760+0000 7fef8cff9700 10 client.48787.objecter ms_dispatch 0x7fef90001550 osd_map(294..294 src has 51..294) 2024-04-14T15:16:02.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:02.760+0000 7fef8cff9700 3 client.48787.objecter handle_osd_map got epochs [294,294] > 0 2024-04-14T15:16:02.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:02.760+0000 7fef8cff9700 3 client.48787.objecter handle_osd_map decoding full epoch 294 2024-04-14T15:16:02.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:02.760+0000 7fef8cff9700 20 client.48787.objecter dump_active .. 0 homeless 2024-04-14T15:16:02.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:02.760+0000 7fef8cff9700 10 client.48787.objecter ms_handle_connect 0x7fef780714a0 2024-04-14T15:16:02.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:02.765+0000 7fef8cff9700 1 -- 172.21.15.90:0/2054244879 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fef840562c0 con 0x7fef9010b730 2024-04-14T15:16:02.767 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:16:02.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:02.791 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:02.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:16:02.892 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:02.896 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:02 smithi090 ceph-mon[30399]: pgmap v386: 278 pgs: 40 unknown, 238 active+clean; 4.4 MiB data, 564 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-14T15:16:02.897 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:02 smithi090 ceph-mon[30399]: from='client.48769 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi090-93342-42"}]': finished 2024-04-14T15:16:02.897 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:02 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-35"}]': finished 2024-04-14T15:16:02.897 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:02 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2635358863' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi090-93342-42"}]: dispatch 2024-04-14T15:16:02.897 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:02 smithi090 ceph-mon[30399]: osdmap e295: 8 total, 8 up, 8 in 2024-04-14T15:16:02.897 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:02 smithi090 ceph-mon[30399]: from='client.48769 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi090-93342-42"}]: dispatch 2024-04-14T15:16:02.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:02.918 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:02.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:02.926 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-14T15:16:02.934 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:16:02.985 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:16:03.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:03.002+0000 7fef95e4d700 1 -- 172.21.15.90:0/2054244879 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7fef7c005410 con 0x7fef9010b730 2024-04-14T15:16:03.020 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:03.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:03.045 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:03.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:03.107 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:02 smithi092 ceph-mon[33966]: pgmap v386: 278 pgs: 40 unknown, 238 active+clean; 4.4 MiB data, 564 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-14T15:16:03.107 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:02 smithi092 ceph-mon[33966]: from='client.48769 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi090-93342-42"}]': finished 2024-04-14T15:16:03.107 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:02 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-35"}]': finished 2024-04-14T15:16:03.108 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:02 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2635358863' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi090-93342-42"}]: dispatch 2024-04-14T15:16:03.108 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:02 smithi092 ceph-mon[33966]: osdmap e295: 8 total, 8 up, 8 in 2024-04-14T15:16:03.108 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:02 smithi092 ceph-mon[33966]: from='client.48769 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi090-93342-42"}]: dispatch 2024-04-14T15:16:03.146 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:16:03.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:03.172 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:03.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:16:03.187 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:02 smithi090 ceph-mon[41670]: pgmap v386: 278 pgs: 40 unknown, 238 active+clean; 4.4 MiB data, 564 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-14T15:16:03.188 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:02 smithi090 ceph-mon[41670]: from='client.48769 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi090-93342-42"}]': finished 2024-04-14T15:16:03.188 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:02 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-35"}]': finished 2024-04-14T15:16:03.188 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:02 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2635358863' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi090-93342-42"}]: dispatch 2024-04-14T15:16:03.188 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:02 smithi090 ceph-mon[41670]: osdmap e295: 8 total, 8 up, 8 in 2024-04-14T15:16:03.188 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:02 smithi090 ceph-mon[41670]: from='client.48769 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi090-93342-42"}]: dispatch 2024-04-14T15:16:03.274 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:16:03.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:03.298 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:03.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:16:03.400 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:16:03.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:03.424 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:03.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:16:03.525 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:16:03.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:03.549 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:03.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:16:03.650 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:16:03.673 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:16:03.675 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:03.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:03.676 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:16:03.777 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:16:03.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:03.802 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:03.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:16:03.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:03.874+0000 7fef8cff9700 1 -- 172.21.15.90:0/2054244879 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v296) ==== 91+0+0 (unknown 2213216616 0 0) 0x7fef84059f70 con 0x7fef9010b730 2024-04-14T15:16:03.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:03.877+0000 7fef95e4d700 10 librados: watch_flush enter 2024-04-14T15:16:03.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:03.878+0000 7fef95e4d700 10 librados: watch_flush exit 2024-04-14T15:16:03.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:03.878+0000 7fef95e4d700 20 client.48787.objecter shutdown clearing up homeless session... 2024-04-14T15:16:03.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:03.878+0000 7fef95e4d700 10 client.48787.objecter successfully canceled tick 2024-04-14T15:16:03.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:03.878+0000 7fef95e4d700 1 -- 172.21.15.90:0/2054244879 >> v1:172.21.15.90:6800/420935427 conn(0x7fef780714a0 legacy=0x7fef78073960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:03.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:03.878+0000 7fef95e4d700 1 -- 172.21.15.90:0/2054244879 >> v1:172.21.15.92:6789/0 conn(0x7fef9010b730 legacy=0x7fef901a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:03.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:03.878+0000 7fef95e4d700 1 -- 172.21.15.90:0/2054244879 shutdown_connections 2024-04-14T15:16:03.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:03.878+0000 7fef95e4d700 1 -- 172.21.15.90:0/2054244879 >> 172.21.15.90:0/2054244879 conn(0x7fef900fe3d0 msgr2=0x7fef90100fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:16:03.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:03.879+0000 7fef95e4d700 1 -- 172.21.15.90:0/2054244879 shutdown_connections 2024-04-14T15:16:03.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:03.879+0000 7fef95e4d700 1 -- 172.21.15.90:0/2054244879 wait complete. 2024-04-14T15:16:03.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:03.879+0000 7fef95e4d700 1 librados: shutdown 2024-04-14T15:16:03.881 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleWrite (6991 ms) 2024-04-14T15:16:03.881 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplete 2024-04-14T15:16:03.881 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (6992 ms) 2024-04-14T15:16:03.881 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2024-04-14T15:16:03.881 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (7013 ms) 2024-04-14T15:16:03.881 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2024-04-14T15:16:03.881 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (7001 ms) 2024-04-14T15:16:03.881 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2024-04-14T15:16:03.881 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (6993 ms) 2024-04-14T15:16:03.882 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2024-04-14T15:16:03.882 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (7011 ms) 2024-04-14T15:16:03.882 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2024-04-14T15:16:03.882 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (6992 ms) 2024-04-14T15:16:03.882 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2024-04-14T15:16:03.882 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (7006 ms) 2024-04-14T15:16:03.882 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2024-04-14T15:16:03.882 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (7006 ms) 2024-04-14T15:16:03.882 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2024-04-14T15:16:03.882 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (7007 ms) 2024-04-14T15:16:03.882 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2024-04-14T15:16:03.882 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (6995 ms) 2024-04-14T15:16:03.882 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2024-04-14T15:16:03.883 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (7004 ms) 2024-04-14T15:16:03.883 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2024-04-14T15:16:03.883 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (7001 ms) 2024-04-14T15:16:03.883 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2024-04-14T15:16:03.883 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (7005 ms) 2024-04-14T15:16:03.883 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2024-04-14T15:16:03.883 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (6995 ms) 2024-04-14T15:16:03.883 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2024-04-14T15:16:03.883 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (7013 ms) 2024-04-14T15:16:03.883 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (112026 ms total) 2024-04-14T15:16:03.883 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: 2024-04-14T15:16:03.883 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [----------] Global test environment tear-down 2024-04-14T15:16:03.883 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [==========] 42 tests from 2 test suites ran. (189736 ms total) 2024-04-14T15:16:03.883 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio: [ PASSED ] 42 tests. 2024-04-14T15:16:03.884 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is unset 2024-04-14T15:16:03.903 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:16:03.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:03.928 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:03.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:16:04.029 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:16:04.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:04.055 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:04.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:16:04.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:03 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3028951022' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi090-93363-52"}]: dispatch 2024-04-14T15:16:04.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:03 smithi092 ceph-mon[33966]: from='client.48793 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi090-93363-52"}]: dispatch 2024-04-14T15:16:04.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:03 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3028951022' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi090-93363-52"}]: dispatch 2024-04-14T15:16:04.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:03 smithi092 ceph-mon[33966]: from='client.48793 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi090-93363-52"}]: dispatch 2024-04-14T15:16:04.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:03 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3028951022' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi090-93363-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:16:04.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:03 smithi092 ceph-mon[33966]: from='client.48793 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi090-93363-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:16:04.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:03 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2054244879' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:16:04.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:03 smithi092 ceph-mon[33966]: from='client.48787 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:16:04.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:03 smithi092 ceph-mon[33966]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:16:04.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:03 smithi092 ceph-mon[33966]: from='client.48769 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi090-93342-42"}]': finished 2024-04-14T15:16:04.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:03 smithi092 ceph-mon[33966]: from='client.48793 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi090-93363-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:16:04.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:03 smithi092 ceph-mon[33966]: from='client.48787 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-14T15:16:04.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:03 smithi092 ceph-mon[33966]: osdmap e296: 8 total, 8 up, 8 in 2024-04-14T15:16:04.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:03 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3028951022' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi090-93363-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi090-93363-52"}]: dispatch 2024-04-14T15:16:04.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:03 smithi092 ceph-mon[33966]: from='client.48793 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi090-93363-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi090-93363-52"}]: dispatch 2024-04-14T15:16:04.156 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:04.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:04.180 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:04.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:04.198 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:16:03 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:16:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:16:04.199 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:03 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3028951022' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi090-93363-52"}]: dispatch 2024-04-14T15:16:04.199 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:03 smithi090 ceph-mon[41670]: from='client.48793 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi090-93363-52"}]: dispatch 2024-04-14T15:16:04.199 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:03 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3028951022' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi090-93363-52"}]: dispatch 2024-04-14T15:16:04.199 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:03 smithi090 ceph-mon[41670]: from='client.48793 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi090-93363-52"}]: dispatch 2024-04-14T15:16:04.199 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:03 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3028951022' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi090-93363-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:16:04.199 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:03 smithi090 ceph-mon[41670]: from='client.48793 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi090-93363-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:16:04.199 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:03 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2054244879' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:16:04.199 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:03 smithi090 ceph-mon[41670]: from='client.48787 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:16:04.199 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:03 smithi090 ceph-mon[41670]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:16:04.199 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:03 smithi090 ceph-mon[41670]: from='client.48769 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi090-93342-42"}]': finished 2024-04-14T15:16:04.199 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:03 smithi090 ceph-mon[41670]: from='client.48793 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi090-93363-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:16:04.199 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:03 smithi090 ceph-mon[41670]: from='client.48787 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-14T15:16:04.200 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:03 smithi090 ceph-mon[41670]: osdmap e296: 8 total, 8 up, 8 in 2024-04-14T15:16:04.200 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:03 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3028951022' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi090-93363-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi090-93363-52"}]: dispatch 2024-04-14T15:16:04.200 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:03 smithi090 ceph-mon[41670]: from='client.48793 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi090-93363-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi090-93363-52"}]: dispatch 2024-04-14T15:16:04.200 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:03 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3028951022' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi090-93363-52"}]: dispatch 2024-04-14T15:16:04.200 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:03 smithi090 ceph-mon[30399]: from='client.48793 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi090-93363-52"}]: dispatch 2024-04-14T15:16:04.200 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:03 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3028951022' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi090-93363-52"}]: dispatch 2024-04-14T15:16:04.200 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:03 smithi090 ceph-mon[30399]: from='client.48793 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi090-93363-52"}]: dispatch 2024-04-14T15:16:04.200 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:03 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3028951022' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi090-93363-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:16:04.200 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:03 smithi090 ceph-mon[30399]: from='client.48793 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi090-93363-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:16:04.200 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:03 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2054244879' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:16:04.200 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:03 smithi090 ceph-mon[30399]: from='client.48787 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:16:04.200 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:03 smithi090 ceph-mon[30399]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:16:04.201 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:03 smithi090 ceph-mon[30399]: from='client.48769 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi090-93342-42"}]': finished 2024-04-14T15:16:04.201 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:03 smithi090 ceph-mon[30399]: from='client.48793 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi090-93363-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:16:04.201 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:03 smithi090 ceph-mon[30399]: from='client.48787 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-14T15:16:04.201 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:03 smithi090 ceph-mon[30399]: osdmap e296: 8 total, 8 up, 8 in 2024-04-14T15:16:04.201 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:03 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3028951022' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi090-93363-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi090-93363-52"}]: dispatch 2024-04-14T15:16:04.201 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:03 smithi090 ceph-mon[30399]: from='client.48793 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi090-93363-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi090-93363-52"}]: dispatch 2024-04-14T15:16:04.281 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:16:04.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:04.306 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:04.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:16:04.407 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:04.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:04.435 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:04.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:04.536 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:16:04.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:04.561 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:04.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:16:04.661 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:16:04.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:04.688 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:04.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:16:04.764 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-04-14T15:16:04.764 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-04-14T15:16:04.764 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-04-14T15:16:04.764 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-04-14T15:16:04.764 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.764 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25028.0:2 40.6 40:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-04-14T15:16:04.764 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:55.118640+0000", 2024-04-14T15:16:04.764 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 189.64314968100001, 2024-04-14T15:16:04.765 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.188111707, 2024-04-14T15:16:04.765 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:16:04.765 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:16:04.765 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:16:04.765 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:16:04.765 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25028", 2024-04-14T15:16:04.765 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/2033704495", 2024-04-14T15:16:04.765 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 2 2024-04-14T15:16:04.765 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.765 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:16:04.765 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.765 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:16:04.765 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.118640+0000", 2024-04-14T15:16:04.766 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:04.766 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.766 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.766 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:16:04.766 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.118640+0000", 2024-04-14T15:16:04.766 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:04.766 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.766 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.766 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:16:04.766 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.118642+0000", 2024-04-14T15:16:04.766 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0899999999999999e-06 2024-04-14T15:16:04.767 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.767 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.767 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:16:04.767 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.118647+0000", 2024-04-14T15:16:04.767 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6690000000000003e-06 2024-04-14T15:16:04.767 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.767 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.767 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:16:04.767 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.118656+0000", 2024-04-14T15:16:04.767 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.4279999999999996e-06 2024-04-14T15:16:04.768 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.768 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.768 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:16:04.768 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.118667+0000", 2024-04-14T15:16:04.768 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0723000000000001e-05 2024-04-14T15:16:04.768 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.768 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.768 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:04.768 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.118857+0000", 2024-04-14T15:16:04.769 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00019035600000000001 2024-04-14T15:16:04.769 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.769 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.769 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:16:04.769 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.118865+0000", 2024-04-14T15:16:04.769 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.7559999999999999e-06 2024-04-14T15:16:04.769 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.769 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.769 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:04.769 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.210788+0000", 2024-04-14T15:16:04.769 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.091922904 2024-04-14T15:16:04.769 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.769 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.769 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:16:04.770 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.210898+0000", 2024-04-14T15:16:04.770 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000110659 2024-04-14T15:16:04.770 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.770 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.770 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 2,5", 2024-04-14T15:16:04.770 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.211021+0000", 2024-04-14T15:16:04.770 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000122753 2024-04-14T15:16:04.770 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.770 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.770 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:16:04.771 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.213735+0000", 2024-04-14T15:16:04.771 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002713764 2024-04-14T15:16:04.771 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.771 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.771 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:16:04.771 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.250150+0000", 2024-04-14T15:16:04.771 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.036414806000000001 2024-04-14T15:16:04.771 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.771 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.771 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:16:04.771 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.306654+0000", 2024-04-14T15:16:04.771 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.056504293999999997 2024-04-14T15:16:04.772 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.772 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.772 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:16:04.772 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.306706+0000", 2024-04-14T15:16:04.772 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2346000000000003e-05 2024-04-14T15:16:04.772 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.772 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.772 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:16:04.772 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.306751+0000", 2024-04-14T15:16:04.772 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5158999999999999e-05 2024-04-14T15:16:04.772 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:04.772 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:16:04.773 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:04.773 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.773 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.773 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.24829.0:2 13.6 13:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-04-14T15:16:04.773 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:56.068832+0000", 2024-04-14T15:16:04.773 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 188.692957699, 2024-04-14T15:16:04.774 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.43249991799999998, 2024-04-14T15:16:04.774 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:16:04.774 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:16:04.774 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:16:04.774 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:16:04.774 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.24829", 2024-04-14T15:16:04.774 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/218717313", 2024-04-14T15:16:04.775 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 2 2024-04-14T15:16:04.775 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.775 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:16:04.775 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.775 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:16:04.775 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.068832+0000", 2024-04-14T15:16:04.775 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:04.775 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.775 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.775 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:16:04.775 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.068832+0000", 2024-04-14T15:16:04.775 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:04.775 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.775 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.776 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:16:04.776 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.068834+0000", 2024-04-14T15:16:04.776 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.323e-06 2024-04-14T15:16:04.776 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.776 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.776 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:16:04.776 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.068842+0000", 2024-04-14T15:16:04.776 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3809999999999999e-06 2024-04-14T15:16:04.776 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.776 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.776 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:16:04.777 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.068860+0000", 2024-04-14T15:16:04.777 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7620000000000001e-05 2024-04-14T15:16:04.777 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.777 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.777 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:16:04.777 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.068872+0000", 2024-04-14T15:16:04.777 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2106999999999999e-05 2024-04-14T15:16:04.777 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.777 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.777 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:04.777 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.068976+0000", 2024-04-14T15:16:04.777 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010345599999999999 2024-04-14T15:16:04.777 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.777 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.778 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op must wait for map", 2024-04-14T15:16:04.778 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.068982+0000", 2024-04-14T15:16:04.778 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.7530000000000004e-06 2024-04-14T15:16:04.778 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.778 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.778 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:04.778 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.146226+0000", 2024-04-14T15:16:04.778 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.077243952000000005 2024-04-14T15:16:04.778 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.778 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.778 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:16:04.778 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.146240+0000", 2024-04-14T15:16:04.779 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4015999999999999e-05 2024-04-14T15:16:04.779 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.779 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.779 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:04.779 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.245970+0000", 2024-04-14T15:16:04.779 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.099729313999999999 2024-04-14T15:16:04.779 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.779 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.779 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:16:04.779 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.246073+0000", 2024-04-14T15:16:04.779 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000103532 2024-04-14T15:16:04.780 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.780 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.780 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,5", 2024-04-14T15:16:04.780 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.246207+0000", 2024-04-14T15:16:04.780 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000134036 2024-04-14T15:16:04.780 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.780 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.780 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:16:04.780 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.248168+0000", 2024-04-14T15:16:04.780 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0019606049999999998 2024-04-14T15:16:04.780 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.780 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.780 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:16:04.781 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.267428+0000", 2024-04-14T15:16:04.781 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.01926025 2024-04-14T15:16:04.781 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.781 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.781 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:16:04.781 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.501199+0000", 2024-04-14T15:16:04.781 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.23377057900000001 2024-04-14T15:16:04.781 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.781 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.781 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:16:04.781 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.501285+0000", 2024-04-14T15:16:04.781 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6007000000000005e-05 2024-04-14T15:16:04.781 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.781 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.782 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:16:04.782 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.501332+0000", 2024-04-14T15:16:04.782 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6987000000000003e-05 2024-04-14T15:16:04.782 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:04.782 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:16:04.782 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:04.782 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.782 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.783 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.24896.0:2 23.15 23:ad551702:::test:head [stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-04-14T15:16:04.783 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:56.072291+0000", 2024-04-14T15:16:04.783 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 188.68949872300001, 2024-04-14T15:16:04.783 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.24892146700000001, 2024-04-14T15:16:04.783 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:16:04.783 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:16:04.783 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-04-14T15:16:04.783 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:16:04.783 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.24896", 2024-04-14T15:16:04.783 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/861388440", 2024-04-14T15:16:04.783 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 2 2024-04-14T15:16:04.783 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.784 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:16:04.784 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.784 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:16:04.784 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.072291+0000", 2024-04-14T15:16:04.784 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:04.784 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.784 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.784 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:16:04.784 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.072291+0000", 2024-04-14T15:16:04.784 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:04.784 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.784 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.785 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:16:04.785 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.072292+0000", 2024-04-14T15:16:04.785 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.812e-06 2024-04-14T15:16:04.785 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.786 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.786 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:16:04.786 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.072298+0000", 2024-04-14T15:16:04.786 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.631e-06 2024-04-14T15:16:04.786 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.786 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.786 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:16:04.787 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.072309+0000", 2024-04-14T15:16:04.787 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0934000000000001e-05 2024-04-14T15:16:04.787 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.787 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.787 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:16:04.787 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.072318+0000", 2024-04-14T15:16:04.787 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.9800000000000004e-06 2024-04-14T15:16:04.787 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.787 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.788 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:04.788 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.072374+0000", 2024-04-14T15:16:04.788 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6221000000000003e-05 2024-04-14T15:16:04.788 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.788 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.788 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op must wait for map", 2024-04-14T15:16:04.788 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.072381+0000", 2024-04-14T15:16:04.788 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3169999999999998e-06 2024-04-14T15:16:04.788 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.788 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.788 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:04.788 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.120345+0000", 2024-04-14T15:16:04.788 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.047964917000000003 2024-04-14T15:16:04.788 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.789 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.789 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:16:04.789 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.120355+0000", 2024-04-14T15:16:04.789 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.2909999999999996e-06 2024-04-14T15:16:04.789 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.789 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.789 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:04.789 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.321061+0000", 2024-04-14T15:16:04.789 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.20070617099999999 2024-04-14T15:16:04.789 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.789 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.790 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:16:04.790 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.321212+0000", 2024-04-14T15:16:04.790 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015119299999999999 2024-04-14T15:16:04.790 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:04.790 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:16:04.790 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:04.790 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.790 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.791 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.24802.0:2 8.14 8:2a134577:test-rados-api-smithi090-93379-2::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-04-14T15:16:04.791 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:56.073174+0000", 2024-04-14T15:16:04.791 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 188.68861565899999, 2024-04-14T15:16:04.791 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.34713522000000002, 2024-04-14T15:16:04.791 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:16:04.791 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:16:04.791 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:16:04.791 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:16:04.791 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.24802", 2024-04-14T15:16:04.791 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/1106328377", 2024-04-14T15:16:04.791 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 2 2024-04-14T15:16:04.791 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.791 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:16:04.792 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.792 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:16:04.792 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.073174+0000", 2024-04-14T15:16:04.792 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:04.792 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.792 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.792 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:16:04.792 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.073174+0000", 2024-04-14T15:16:04.792 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:04.792 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.792 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.792 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:16:04.792 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.073176+0000", 2024-04-14T15:16:04.792 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0640000000000001e-06 2024-04-14T15:16:04.792 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.793 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.793 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:16:04.793 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.073181+0000", 2024-04-14T15:16:04.793 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3399999999999997e-06 2024-04-14T15:16:04.793 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.793 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.793 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:16:04.793 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.073195+0000", 2024-04-14T15:16:04.793 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3471e-05 2024-04-14T15:16:04.793 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.793 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.793 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:16:04.793 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.073203+0000", 2024-04-14T15:16:04.793 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.123e-06 2024-04-14T15:16:04.793 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.794 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.794 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:04.794 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.073254+0000", 2024-04-14T15:16:04.794 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.1255000000000003e-05 2024-04-14T15:16:04.794 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.794 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.794 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op must wait for map", 2024-04-14T15:16:04.794 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.073261+0000", 2024-04-14T15:16:04.794 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2069999999999998e-06 2024-04-14T15:16:04.794 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.794 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.794 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:04.794 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.151898+0000", 2024-04-14T15:16:04.794 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.078636629 2024-04-14T15:16:04.794 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.795 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.795 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:16:04.795 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.151906+0000", 2024-04-14T15:16:04.795 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2640000000000008e-06 2024-04-14T15:16:04.795 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.795 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.795 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:04.795 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.294003+0000", 2024-04-14T15:16:04.795 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.14209672500000001 2024-04-14T15:16:04.795 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.795 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.795 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:16:04.795 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.294129+0000", 2024-04-14T15:16:04.795 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000125969 2024-04-14T15:16:04.795 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.796 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.796 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,2", 2024-04-14T15:16:04.796 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.294286+0000", 2024-04-14T15:16:04.796 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000157436 2024-04-14T15:16:04.796 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.796 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.796 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:16:04.796 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.295784+0000", 2024-04-14T15:16:04.796 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0014976600000000001 2024-04-14T15:16:04.796 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.796 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.796 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:16:04.796 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.357451+0000", 2024-04-14T15:16:04.796 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.061667551000000001 2024-04-14T15:16:04.796 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.797 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.797 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:16:04.797 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.419349+0000", 2024-04-14T15:16:04.797 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.061897887999999998 2024-04-14T15:16:04.797 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.797 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.797 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:16:04.797 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.420262+0000", 2024-04-14T15:16:04.797 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00091295099999999995 2024-04-14T15:16:04.797 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.797 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.798 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:16:04.800 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:16:04.800 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.420309+0000", 2024-04-14T15:16:04.800 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6687000000000002e-05 2024-04-14T15:16:04.800 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:04.800 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:16:04.800 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:04.800 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.800 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.801 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25028.0:3 40.6 40:602f83fe:::foo:head [watch watch cookie 94006567794064] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-04-14T15:16:04.801 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:56.307396+0000", 2024-04-14T15:16:04.801 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 188.454393103, 2024-04-14T15:16:04.801 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.053731905000000003, 2024-04-14T15:16:04.801 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:16:04.801 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:16:04.801 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:16:04.801 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:16:04.801 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25028", 2024-04-14T15:16:04.801 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/2033704495", 2024-04-14T15:16:04.801 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 3 2024-04-14T15:16:04.801 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.801 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:16:04.801 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.802 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:16:04.802 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.307396+0000", 2024-04-14T15:16:04.802 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:04.802 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.802 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.802 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:16:04.802 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.307396+0000", 2024-04-14T15:16:04.802 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:04.802 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.802 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.802 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:16:04.802 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.307398+0000", 2024-04-14T15:16:04.802 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.539e-06 2024-04-14T15:16:04.802 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.803 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.803 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:16:04.803 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.307403+0000", 2024-04-14T15:16:04.803 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.1039999999999998e-06 2024-04-14T15:16:04.803 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.803 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.803 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:16:04.803 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.307413+0000", 2024-04-14T15:16:04.803 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0212e-05 2024-04-14T15:16:04.803 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.803 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.804 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:16:04.804 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.307424+0000", 2024-04-14T15:16:04.804 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0964999999999999e-05 2024-04-14T15:16:04.804 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.804 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.804 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:04.804 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.308351+0000", 2024-04-14T15:16:04.804 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00092658400000000004 2024-04-14T15:16:04.804 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.804 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.805 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:16:04.805 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.308438+0000", 2024-04-14T15:16:04.805 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.7077000000000007e-05 2024-04-14T15:16:04.805 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.805 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.805 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 2,5", 2024-04-14T15:16:04.805 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.308709+0000", 2024-04-14T15:16:04.805 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00027113400000000002 2024-04-14T15:16:04.805 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.805 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.805 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:16:04.805 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.309380+0000", 2024-04-14T15:16:04.805 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00067086999999999995 2024-04-14T15:16:04.805 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.805 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.806 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:16:04.806 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.309489+0000", 2024-04-14T15:16:04.806 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000109181 2024-04-14T15:16:04.806 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.806 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.806 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:16:04.806 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.360999+0000", 2024-04-14T15:16:04.806 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.051509961999999999 2024-04-14T15:16:04.806 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.806 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.806 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:16:04.806 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.361055+0000", 2024-04-14T15:16:04.806 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6017000000000003e-05 2024-04-14T15:16:04.806 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.807 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.807 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:16:04.807 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.361128+0000", 2024-04-14T15:16:04.807 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.326e-05 2024-04-14T15:16:04.807 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:04.807 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:16:04.807 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:04.807 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.807 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.807 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25028.0:7 40.6 40:602f83fe:::foo:head [watch unwatch cookie 94006567794064] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-04-14T15:16:04.807 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:56.376539+0000", 2024-04-14T15:16:04.807 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 188.385250122, 2024-04-14T15:16:04.807 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.10311740799999999, 2024-04-14T15:16:04.807 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:16:04.808 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:16:04.808 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:16:04.808 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:16:04.808 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25028", 2024-04-14T15:16:04.808 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/2033704495", 2024-04-14T15:16:04.808 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 7 2024-04-14T15:16:04.808 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.808 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:16:04.808 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.808 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:16:04.808 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.376539+0000", 2024-04-14T15:16:04.808 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:04.808 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.808 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.808 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:16:04.809 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.376539+0000", 2024-04-14T15:16:04.809 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:04.809 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.809 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.809 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:16:04.809 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.376541+0000", 2024-04-14T15:16:04.809 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9829999999999999e-06 2024-04-14T15:16:04.809 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.809 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.809 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:16:04.809 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.376561+0000", 2024-04-14T15:16:04.809 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9961000000000001e-05 2024-04-14T15:16:04.809 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.809 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.809 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:16:04.810 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.376572+0000", 2024-04-14T15:16:04.810 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0813e-05 2024-04-14T15:16:04.810 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.810 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.810 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:16:04.810 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.376583+0000", 2024-04-14T15:16:04.810 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0529e-05 2024-04-14T15:16:04.810 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.810 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.810 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:04.810 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.376646+0000", 2024-04-14T15:16:04.810 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3589000000000004e-05 2024-04-14T15:16:04.810 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.811 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.811 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:16:04.811 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.376745+0000", 2024-04-14T15:16:04.811 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.9146000000000002e-05 2024-04-14T15:16:04.811 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.811 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.811 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 2,5", 2024-04-14T15:16:04.811 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.376916+0000", 2024-04-14T15:16:04.812 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00017041499999999999 2024-04-14T15:16:04.812 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.812 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.812 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:16:04.812 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.377764+0000", 2024-04-14T15:16:04.812 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00084804300000000004 2024-04-14T15:16:04.812 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.812 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.812 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:16:04.812 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.411348+0000", 2024-04-14T15:16:04.812 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.033583871000000001 2024-04-14T15:16:04.812 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.812 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.812 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:16:04.812 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.479526+0000", 2024-04-14T15:16:04.813 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.068178424000000001 2024-04-14T15:16:04.813 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.813 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.813 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:16:04.813 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.479586+0000", 2024-04-14T15:16:04.813 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0093000000000002e-05 2024-04-14T15:16:04.813 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.813 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.813 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:16:04.813 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.479657+0000", 2024-04-14T15:16:04.813 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0541000000000001e-05 2024-04-14T15:16:04.813 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:04.813 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:16:04.813 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:04.813 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.814 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.814 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.24829.0:828 13.15 13:a8000000::::head [pgnls start_epoch 108] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e108)", 2024-04-14T15:16:04.814 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:57.081514+0000", 2024-04-14T15:16:04.814 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 187.68027583899999, 2024-04-14T15:16:04.814 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.067804700999999995, 2024-04-14T15:16:04.814 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:16:04.814 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:16:04.814 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-04-14T15:16:04.814 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:16:04.814 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.24829", 2024-04-14T15:16:04.814 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/218717313", 2024-04-14T15:16:04.814 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 828 2024-04-14T15:16:04.814 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.814 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:16:04.815 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.815 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:16:04.815 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.081514+0000", 2024-04-14T15:16:04.815 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:04.815 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.815 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.815 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:16:04.815 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.081514+0000", 2024-04-14T15:16:04.815 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:04.815 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.815 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.815 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:16:04.815 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.081516+0000", 2024-04-14T15:16:04.815 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6180000000000002e-06 2024-04-14T15:16:04.815 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.816 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.816 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:16:04.816 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.081524+0000", 2024-04-14T15:16:04.816 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0479999999999999e-06 2024-04-14T15:16:04.816 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.816 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.816 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:16:04.816 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.092114+0000", 2024-04-14T15:16:04.816 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.010590123999999999 2024-04-14T15:16:04.816 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.816 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.816 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:16:04.816 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.092128+0000", 2024-04-14T15:16:04.816 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4064e-05 2024-04-14T15:16:04.816 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.817 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.817 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:04.817 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.149196+0000", 2024-04-14T15:16:04.817 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.057067179000000003 2024-04-14T15:16:04.817 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.817 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.817 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:16:04.817 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.149234+0000", 2024-04-14T15:16:04.817 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8847999999999998e-05 2024-04-14T15:16:04.817 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.817 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.817 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:16:04.817 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.149318+0000", 2024-04-14T15:16:04.817 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3819999999999994e-05 2024-04-14T15:16:04.817 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:04.818 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:16:04.818 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:04.818 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.818 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.818 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25198.0:2 55.1a 55:5c6776db:::0.obj:head [write 0~103] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-04-14T15:16:04.818 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:58.081881+0000", 2024-04-14T15:16:04.818 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 186.67990866900001, 2024-04-14T15:16:04.818 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2238229890000001, 2024-04-14T15:16:04.818 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:16:04.818 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:16:04.818 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:16:04.818 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:16:04.818 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25198", 2024-04-14T15:16:04.818 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/1077085148", 2024-04-14T15:16:04.819 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 2 2024-04-14T15:16:04.819 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.819 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:16:04.819 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.819 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:16:04.819 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.081881+0000", 2024-04-14T15:16:04.819 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:04.819 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.819 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.819 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:16:04.819 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.081881+0000", 2024-04-14T15:16:04.819 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:04.819 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.819 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.820 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:16:04.820 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.081883+0000", 2024-04-14T15:16:04.820 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2029999999999999e-06 2024-04-14T15:16:04.820 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.820 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.820 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:16:04.820 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.081893+0000", 2024-04-14T15:16:04.820 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0152e-05 2024-04-14T15:16:04.820 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.820 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.820 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:16:04.820 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.081903+0000", 2024-04-14T15:16:04.820 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0353000000000001e-05 2024-04-14T15:16:04.820 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.820 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.821 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:16:04.821 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.081912+0000", 2024-04-14T15:16:04.821 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.9199999999999993e-06 2024-04-14T15:16:04.821 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.821 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.821 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:04.821 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.103712+0000", 2024-04-14T15:16:04.821 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.021799163 2024-04-14T15:16:04.822 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.822 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.822 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:16:04.822 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.103720+0000", 2024-04-14T15:16:04.822 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.4470000000000006e-06 2024-04-14T15:16:04.822 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.822 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.822 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:04.822 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.204902+0000", 2024-04-14T15:16:04.822 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.101182071 2024-04-14T15:16:04.822 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.822 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.822 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:16:04.822 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.205015+0000", 2024-04-14T15:16:04.823 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011304800000000001 2024-04-14T15:16:04.823 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.823 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.823 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,3", 2024-04-14T15:16:04.823 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.205152+0000", 2024-04-14T15:16:04.823 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013653999999999999 2024-04-14T15:16:04.823 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.823 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.823 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:16:04.823 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.208940+0000", 2024-04-14T15:16:04.823 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0037881939999999999 2024-04-14T15:16:04.824 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.824 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.824 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:16:04.824 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.252540+0000", 2024-04-14T15:16:04.824 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.043599903000000002 2024-04-14T15:16:04.824 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.824 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.824 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:16:04.824 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.305601+0000", 2024-04-14T15:16:04.824 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.053060819000000002 2024-04-14T15:16:04.824 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.824 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.824 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:16:04.824 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.305655+0000", 2024-04-14T15:16:04.824 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4385999999999997e-05 2024-04-14T15:16:04.825 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.825 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.825 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:16:04.825 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.305704+0000", 2024-04-14T15:16:04.825 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8789999999999999e-05 2024-04-14T15:16:04.825 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:04.825 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:16:04.825 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:04.825 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.825 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.825 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.24868.0:5 17.2 17:412fddd1:test-rados-api-smithi090-93724-3::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-04-14T15:16:04.825 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:58.125668+0000", 2024-04-14T15:16:04.825 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 186.63612172000001, 2024-04-14T15:16:04.825 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.055211019, 2024-04-14T15:16:04.826 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:16:04.826 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:16:04.826 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:16:04.826 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:16:04.826 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.24868", 2024-04-14T15:16:04.826 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/1684644655", 2024-04-14T15:16:04.826 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 5 2024-04-14T15:16:04.826 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.826 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:16:04.826 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.826 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:16:04.826 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.125668+0000", 2024-04-14T15:16:04.826 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:04.826 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.826 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.827 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:16:04.827 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.125668+0000", 2024-04-14T15:16:04.827 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:04.827 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.827 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.827 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:16:04.827 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.125670+0000", 2024-04-14T15:16:04.827 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.891e-06 2024-04-14T15:16:04.827 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.827 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.827 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:16:04.827 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.125678+0000", 2024-04-14T15:16:04.827 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1100000000000003e-06 2024-04-14T15:16:04.827 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.827 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.828 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:16:04.828 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.125688+0000", 2024-04-14T15:16:04.828 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0685e-05 2024-04-14T15:16:04.828 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.828 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.828 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:16:04.828 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.125699+0000", 2024-04-14T15:16:04.828 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0411000000000001e-05 2024-04-14T15:16:04.828 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.828 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.828 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:04.828 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.125776+0000", 2024-04-14T15:16:04.828 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.6784000000000002e-05 2024-04-14T15:16:04.828 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.829 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.829 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:16:04.829 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.125893+0000", 2024-04-14T15:16:04.829 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000117534 2024-04-14T15:16:04.829 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.829 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.829 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,5", 2024-04-14T15:16:04.829 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.126048+0000", 2024-04-14T15:16:04.829 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000155086 2024-04-14T15:16:04.829 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.829 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.829 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:16:04.829 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.126561+0000", 2024-04-14T15:16:04.829 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00051295599999999996 2024-04-14T15:16:04.829 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.830 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.830 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:16:04.830 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.158390+0000", 2024-04-14T15:16:04.830 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.031829059999999999 2024-04-14T15:16:04.830 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.830 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.830 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:16:04.830 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.180774+0000", 2024-04-14T15:16:04.830 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.022384035 2024-04-14T15:16:04.830 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.830 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.830 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:16:04.830 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.180833+0000", 2024-04-14T15:16:04.830 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.8841000000000001e-05 2024-04-14T15:16:04.831 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.831 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.831 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:16:04.831 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.180879+0000", 2024-04-14T15:16:04.831 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5626e-05 2024-04-14T15:16:04.831 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:04.831 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:16:04.831 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:04.831 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.831 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.831 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25210.0:2 59.13 59:cd2ae97d:::foo1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-04-14T15:16:04.832 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:58.126711+0000", 2024-04-14T15:16:04.832 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 186.63507865700001, 2024-04-14T15:16:04.832 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.19058456, 2024-04-14T15:16:04.832 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:16:04.832 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:16:04.832 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:16:04.832 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:16:04.832 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25210", 2024-04-14T15:16:04.833 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/4158655387", 2024-04-14T15:16:04.833 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 2 2024-04-14T15:16:04.833 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.833 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:16:04.833 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.833 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:16:04.833 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.126711+0000", 2024-04-14T15:16:04.833 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:04.833 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.833 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.833 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:16:04.833 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.126711+0000", 2024-04-14T15:16:04.834 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:04.834 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.834 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.834 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:16:04.834 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.126712+0000", 2024-04-14T15:16:04.834 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2470000000000001e-06 2024-04-14T15:16:04.834 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.834 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.834 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:16:04.834 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.126717+0000", 2024-04-14T15:16:04.834 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.0520000000000001e-06 2024-04-14T15:16:04.834 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.834 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.834 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:16:04.835 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.126727+0000", 2024-04-14T15:16:04.835 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0189e-05 2024-04-14T15:16:04.835 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.835 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.835 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:16:04.835 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.126736+0000", 2024-04-14T15:16:04.835 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3119999999999994e-06 2024-04-14T15:16:04.835 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.835 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.835 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:04.835 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.126796+0000", 2024-04-14T15:16:04.835 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0158000000000002e-05 2024-04-14T15:16:04.835 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.835 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.836 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:16:04.836 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.126804+0000", 2024-04-14T15:16:04.836 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1899999999999995e-06 2024-04-14T15:16:04.836 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.836 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.836 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:04.836 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.295771+0000", 2024-04-14T15:16:04.836 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1689666919999999 2024-04-14T15:16:04.836 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.836 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.836 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:16:04.836 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.295875+0000", 2024-04-14T15:16:04.836 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000104329 2024-04-14T15:16:04.836 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.837 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.837 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,6", 2024-04-14T15:16:04.837 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.295967+0000", 2024-04-14T15:16:04.837 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.1788999999999998e-05 2024-04-14T15:16:04.837 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.837 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.837 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:16:04.837 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.296429+0000", 2024-04-14T15:16:04.837 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000462764 2024-04-14T15:16:04.838 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.838 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.838 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:16:04.838 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.302107+0000", 2024-04-14T15:16:04.838 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0056771829999999997 2024-04-14T15:16:04.838 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.838 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.838 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:16:04.838 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.317204+0000", 2024-04-14T15:16:04.838 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.015097203 2024-04-14T15:16:04.838 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.838 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.838 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:16:04.841 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.317244+0000", 2024-04-14T15:16:04.842 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9978000000000002e-05 2024-04-14T15:16:04.842 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.842 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.842 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:16:04.842 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.317295+0000", 2024-04-14T15:16:04.842 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.1474000000000002e-05 2024-04-14T15:16:04.842 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:04.842 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:16:04.842 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:04.842 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.842 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.842 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.24853.0:2069 16.e e111/107 16:70021a00:test-rados-api-smithi090-93601-17::big:head v 111'2, mlcod=0'0)", 2024-04-14T15:16:04.842 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:59.377958+0000", 2024-04-14T15:16:04.843 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 185.383831195, 2024-04-14T15:16:04.843 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.050254007000000003, 2024-04-14T15:16:04.843 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:16:04.843 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:16:04.843 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:16:04.843 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:16:04.843 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.843 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:16:04.843 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.377958+0000", 2024-04-14T15:16:04.843 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:04.843 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.843 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.843 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:16:04.843 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.377958+0000", 2024-04-14T15:16:04.843 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:04.844 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.844 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.844 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:16:04.844 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.377961+0000", 2024-04-14T15:16:04.844 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5299999999999999e-06 2024-04-14T15:16:04.844 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.844 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.844 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:16:04.844 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.401281+0000", 2024-04-14T15:16:04.844 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0233206 2024-04-14T15:16:04.844 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.844 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.844 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:16:04.844 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.401294+0000", 2024-04-14T15:16:04.844 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2187e-05 2024-04-14T15:16:04.845 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.845 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.845 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:16:04.845 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.401301+0000", 2024-04-14T15:16:04.845 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.4120000000000004e-06 2024-04-14T15:16:04.845 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.845 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.845 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:04.845 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.401345+0000", 2024-04-14T15:16:04.845 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.4450000000000003e-05 2024-04-14T15:16:04.845 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.845 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.845 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:16:04.845 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.401362+0000", 2024-04-14T15:16:04.845 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7042999999999999e-05 2024-04-14T15:16:04.846 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.846 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.846 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:16:04.846 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.428165+0000", 2024-04-14T15:16:04.846 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.026802277999999999 2024-04-14T15:16:04.846 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.846 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.846 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:16:04.846 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.428212+0000", 2024-04-14T15:16:04.846 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7506999999999998e-05 2024-04-14T15:16:04.846 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:04.846 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:16:04.846 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:04.846 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.847 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.847 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.24853.0:2074 16.c e111/107 16:31d79ce4:test-rados-api-smithi090-93601-17::big3:head v 111'2, mlcod=0'0)", 2024-04-14T15:16:04.847 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:59.574768+0000", 2024-04-14T15:16:04.847 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 185.18702105899999, 2024-04-14T15:16:04.847 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.062724147999999993, 2024-04-14T15:16:04.847 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:16:04.847 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:16:04.847 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:16:04.847 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:16:04.847 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.847 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:16:04.847 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.574768+0000", 2024-04-14T15:16:04.847 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:04.847 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.847 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.848 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:16:04.848 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.574768+0000", 2024-04-14T15:16:04.848 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:04.848 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.848 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.848 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:16:04.848 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.574772+0000", 2024-04-14T15:16:04.848 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.3409999999999999e-06 2024-04-14T15:16:04.848 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.848 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.848 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:16:04.848 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.584618+0000", 2024-04-14T15:16:04.848 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00984639 2024-04-14T15:16:04.848 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.849 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.849 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:16:04.849 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.584639+0000", 2024-04-14T15:16:04.849 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1053999999999999e-05 2024-04-14T15:16:04.849 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.849 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.849 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:16:04.849 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.584651+0000", 2024-04-14T15:16:04.849 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2354000000000001e-05 2024-04-14T15:16:04.849 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.849 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.849 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:04.849 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.584726+0000", 2024-04-14T15:16:04.849 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.4559000000000002e-05 2024-04-14T15:16:04.849 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.850 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.850 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:16:04.850 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.584745+0000", 2024-04-14T15:16:04.850 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8997000000000001e-05 2024-04-14T15:16:04.850 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.850 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.850 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:16:04.850 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.637426+0000", 2024-04-14T15:16:04.850 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.052681095999999997 2024-04-14T15:16:04.850 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.850 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.850 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:16:04.850 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.637492+0000", 2024-04-14T15:16:04.852 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.6357000000000002e-05 2024-04-14T15:16:04.852 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:04.852 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:16:04.852 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:04.852 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.852 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.852 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25354.0:2 72.0 72:0384ab8e:::oid:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-04-14T15:16:04.852 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:13:00.119235+0000", 2024-04-14T15:16:04.852 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 184.64255396900001, 2024-04-14T15:16:04.852 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.81347187600000004, 2024-04-14T15:16:04.853 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:16:04.853 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:16:04.853 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:16:04.853 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:16:04.853 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25354", 2024-04-14T15:16:04.853 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/3758511316", 2024-04-14T15:16:04.853 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 2 2024-04-14T15:16:04.853 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.853 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:16:04.853 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.853 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:16:04.853 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.119235+0000", 2024-04-14T15:16:04.853 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:04.853 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.853 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.854 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:16:04.854 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.119235+0000", 2024-04-14T15:16:04.854 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:04.854 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.854 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.854 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:16:04.854 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.119237+0000", 2024-04-14T15:16:04.854 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0810000000000001e-06 2024-04-14T15:16:04.854 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.854 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.854 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:16:04.854 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.119243+0000", 2024-04-14T15:16:04.854 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9680000000000003e-06 2024-04-14T15:16:04.854 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.855 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.855 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:16:04.855 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.126617+0000", 2024-04-14T15:16:04.855 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0073734839999999996 2024-04-14T15:16:04.855 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.855 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.855 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:16:04.855 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.126643+0000", 2024-04-14T15:16:04.855 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6350999999999999e-05 2024-04-14T15:16:04.855 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.855 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.855 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:04.855 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.127309+0000", 2024-04-14T15:16:04.855 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00066571200000000005 2024-04-14T15:16:04.855 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.856 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.856 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:16:04.856 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.127317+0000", 2024-04-14T15:16:04.856 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3219999999999993e-06 2024-04-14T15:16:04.856 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.856 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.856 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:04.856 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.891898+0000", 2024-04-14T15:16:04.856 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.76458101700000003 2024-04-14T15:16:04.856 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.856 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.856 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:16:04.856 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.892011+0000", 2024-04-14T15:16:04.856 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000112891 2024-04-14T15:16:04.856 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.857 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.857 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1,7", 2024-04-14T15:16:04.857 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.892135+0000", 2024-04-14T15:16:04.857 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012421700000000001 2024-04-14T15:16:04.857 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.857 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.857 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:16:04.857 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.892414+0000", 2024-04-14T15:16:04.857 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00027872999999999998 2024-04-14T15:16:04.857 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.857 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.857 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:16:04.857 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.897424+0000", 2024-04-14T15:16:04.857 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0050095629999999999 2024-04-14T15:16:04.857 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.858 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.858 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:16:04.858 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.932615+0000", 2024-04-14T15:16:04.858 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.035191564000000002 2024-04-14T15:16:04.858 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.858 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.858 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:16:04.858 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.932666+0000", 2024-04-14T15:16:04.858 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.0560999999999999e-05 2024-04-14T15:16:04.858 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.859 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.859 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:16:04.859 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.932707+0000", 2024-04-14T15:16:04.859 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1415000000000002e-05 2024-04-14T15:16:04.859 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:04.859 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:16:04.859 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:04.859 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.859 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.859 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25684.0:110 135.17 135:eb4a7a6a:test-rados-api-smithi090-93652-10::foo:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e134)", 2024-04-14T15:16:04.859 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:13:21.824891+0000", 2024-04-14T15:16:04.859 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 162.93689867200001, 2024-04-14T15:16:04.859 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.14413627700000001, 2024-04-14T15:16:04.859 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:16:04.860 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:16:04.860 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-04-14T15:16:04.860 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:16:04.860 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25684", 2024-04-14T15:16:04.860 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/1109245683", 2024-04-14T15:16:04.860 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 110 2024-04-14T15:16:04.860 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.860 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:16:04.860 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.860 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:16:04.860 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:21.824891+0000", 2024-04-14T15:16:04.860 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:04.860 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.860 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.860 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:16:04.861 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:21.824891+0000", 2024-04-14T15:16:04.861 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:04.861 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.861 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.861 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:16:04.861 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:21.824892+0000", 2024-04-14T15:16:04.861 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.404e-06 2024-04-14T15:16:04.861 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.861 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.861 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:16:04.861 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:21.824898+0000", 2024-04-14T15:16:04.861 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5029999999999997e-06 2024-04-14T15:16:04.861 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.861 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.862 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:16:04.862 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:21.824906+0000", 2024-04-14T15:16:04.862 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8589999999999994e-06 2024-04-14T15:16:04.862 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.862 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.862 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:16:04.862 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:21.824916+0000", 2024-04-14T15:16:04.862 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.6479999999999998e-06 2024-04-14T15:16:04.862 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.862 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.862 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:04.862 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:21.824971+0000", 2024-04-14T15:16:04.862 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4805999999999997e-05 2024-04-14T15:16:04.862 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.863 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.863 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for blocked object", 2024-04-14T15:16:04.863 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:21.825199+0000", 2024-04-14T15:16:04.863 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00022818299999999999 2024-04-14T15:16:04.863 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.863 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.863 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:04.863 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:21.905705+0000", 2024-04-14T15:16:04.863 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.080505987000000001 2024-04-14T15:16:04.863 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.863 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.863 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for rw locks", 2024-04-14T15:16:04.863 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:21.905814+0000", 2024-04-14T15:16:04.863 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010869 2024-04-14T15:16:04.863 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.864 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.864 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:04.864 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:21.968732+0000", 2024-04-14T15:16:04.864 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.062918098000000006 2024-04-14T15:16:04.864 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.864 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.864 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:16:04.864 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:21.968825+0000", 2024-04-14T15:16:04.864 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.2849000000000005e-05 2024-04-14T15:16:04.864 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.864 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.864 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:16:04.864 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:21.969027+0000", 2024-04-14T15:16:04.864 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00020225000000000001 2024-04-14T15:16:04.865 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:04.865 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:16:04.865 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:04.865 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.865 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.865 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.36767.0:2 147.6 147:602f83fe:::foo:head [append 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-04-14T15:16:04.865 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:13:22.868465+0000", 2024-04-14T15:16:04.865 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 161.89332409299999, 2024-04-14T15:16:04.865 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.10357522299999999, 2024-04-14T15:16:04.865 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:16:04.865 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:16:04.865 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:16:04.865 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:16:04.865 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.36767", 2024-04-14T15:16:04.865 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/902879548", 2024-04-14T15:16:04.866 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 2 2024-04-14T15:16:04.866 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.866 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:16:04.866 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.866 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:16:04.866 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:22.868465+0000", 2024-04-14T15:16:04.866 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:04.866 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.866 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.866 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:16:04.866 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:22.868465+0000", 2024-04-14T15:16:04.866 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:04.866 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.866 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.867 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:16:04.867 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:22.868467+0000", 2024-04-14T15:16:04.867 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9130000000000001e-06 2024-04-14T15:16:04.867 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.867 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.867 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:16:04.867 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:22.868474+0000", 2024-04-14T15:16:04.867 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2899999999999997e-06 2024-04-14T15:16:04.867 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.867 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.867 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:16:04.867 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:22.868486+0000", 2024-04-14T15:16:04.867 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1823e-05 2024-04-14T15:16:04.867 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.868 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.868 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:16:04.868 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:22.868497+0000", 2024-04-14T15:16:04.868 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0927000000000001e-05 2024-04-14T15:16:04.868 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.868 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.868 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:04.868 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:22.868557+0000", 2024-04-14T15:16:04.868 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9996e-05 2024-04-14T15:16:04.868 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.868 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.868 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:16:04.868 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:22.868566+0000", 2024-04-14T15:16:04.868 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3720000000000005e-06 2024-04-14T15:16:04.868 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.869 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.869 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:04.869 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:22.917084+0000", 2024-04-14T15:16:04.869 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.048518110000000003 2024-04-14T15:16:04.869 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.869 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.869 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:16:04.869 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:22.917206+0000", 2024-04-14T15:16:04.869 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012209699999999999 2024-04-14T15:16:04.869 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.869 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.869 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 2,6", 2024-04-14T15:16:04.869 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:22.917361+0000", 2024-04-14T15:16:04.869 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000155511 2024-04-14T15:16:04.869 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.870 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.870 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:16:04.870 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:22.917927+0000", 2024-04-14T15:16:04.870 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00056553500000000004 2024-04-14T15:16:04.870 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.870 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.870 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:16:04.870 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:22.918182+0000", 2024-04-14T15:16:04.870 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00025522199999999999 2024-04-14T15:16:04.870 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.870 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.870 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:16:04.870 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:22.971948+0000", 2024-04-14T15:16:04.870 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.053765598999999997 2024-04-14T15:16:04.871 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.871 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.871 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:16:04.871 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:22.971997+0000", 2024-04-14T15:16:04.871 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8860000000000003e-05 2024-04-14T15:16:04.871 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.871 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.871 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:16:04.871 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:22.972041+0000", 2024-04-14T15:16:04.871 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3968000000000003e-05 2024-04-14T15:16:04.871 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:04.871 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:16:04.871 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:04.871 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.871 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.872 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.41476.0:2 151.0 151:0384ab8e:::oid:head [zero 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-04-14T15:16:04.872 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:13:24.859716+0000", 2024-04-14T15:16:04.872 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 159.902073247, 2024-04-14T15:16:04.872 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1201019210000001, 2024-04-14T15:16:04.872 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:16:04.872 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:16:04.872 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-04-14T15:16:04.872 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:16:04.872 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.41476", 2024-04-14T15:16:04.872 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/1075568278", 2024-04-14T15:16:04.872 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 2 2024-04-14T15:16:04.872 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.872 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:16:04.872 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.872 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:16:04.873 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:24.859716+0000", 2024-04-14T15:16:04.873 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:04.873 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.873 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.873 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:16:04.873 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:24.859716+0000", 2024-04-14T15:16:04.873 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:04.873 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.873 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.873 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:16:04.873 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:24.859718+0000", 2024-04-14T15:16:04.873 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.683e-06 2024-04-14T15:16:04.873 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.873 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.873 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:16:04.874 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:24.859723+0000", 2024-04-14T15:16:04.874 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3279999999999996e-06 2024-04-14T15:16:04.874 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.874 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.874 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:16:04.874 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:24.859732+0000", 2024-04-14T15:16:04.874 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.1810000000000007e-06 2024-04-14T15:16:04.874 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.874 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.874 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:16:04.874 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:24.859742+0000", 2024-04-14T15:16:04.874 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.2709999999999998e-06 2024-04-14T15:16:04.874 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.875 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.875 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:04.875 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:24.859790+0000", 2024-04-14T15:16:04.875 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8547000000000001e-05 2024-04-14T15:16:04.875 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.875 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.875 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:16:04.875 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:24.859799+0000", 2024-04-14T15:16:04.875 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.3109999999999995e-06 2024-04-14T15:16:04.875 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.875 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.875 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:04.875 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.887558+0000", 2024-04-14T15:16:04.875 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.027758723 2024-04-14T15:16:04.876 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.876 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.876 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:16:04.876 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.887660+0000", 2024-04-14T15:16:04.876 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010152799999999999 2024-04-14T15:16:04.876 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.876 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.876 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:16:04.876 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.979818+0000", 2024-04-14T15:16:04.876 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.092158349000000001 2024-04-14T15:16:04.876 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:04.876 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:16:04.876 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:04.876 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.876 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.877 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.41515.0:2 154.6 154:602f83fe:::foo:head [append 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-04-14T15:16:04.877 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:13:25.843012+0000", 2024-04-14T15:16:04.877 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 158.918776998, 2024-04-14T15:16:04.877 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.136860596, 2024-04-14T15:16:04.877 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:16:04.877 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:16:04.877 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:16:04.877 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:16:04.877 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.41515", 2024-04-14T15:16:04.877 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/1865301891", 2024-04-14T15:16:04.877 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 2 2024-04-14T15:16:04.877 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.877 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:16:04.877 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.878 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:16:04.878 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.843012+0000", 2024-04-14T15:16:04.878 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:04.878 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.878 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.878 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:16:04.878 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.843012+0000", 2024-04-14T15:16:04.878 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:04.878 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.878 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.878 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:16:04.878 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.843015+0000", 2024-04-14T15:16:04.878 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.159e-06 2024-04-14T15:16:04.879 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.879 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.879 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:16:04.879 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.843024+0000", 2024-04-14T15:16:04.879 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.1330000000000005e-06 2024-04-14T15:16:04.879 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.879 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.879 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:16:04.879 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.843033+0000", 2024-04-14T15:16:04.879 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.5149999999999995e-06 2024-04-14T15:16:04.879 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.879 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.879 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:16:04.879 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.843043+0000", 2024-04-14T15:16:04.879 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0268000000000001e-05 2024-04-14T15:16:04.880 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.880 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.880 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:04.880 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.843356+0000", 2024-04-14T15:16:04.880 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00031297200000000001 2024-04-14T15:16:04.880 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.880 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.880 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:16:04.880 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.843366+0000", 2024-04-14T15:16:04.880 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.2240000000000001e-06 2024-04-14T15:16:04.880 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.880 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.880 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:04.880 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.903587+0000", 2024-04-14T15:16:04.880 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.060221471999999998 2024-04-14T15:16:04.881 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.881 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.881 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:16:04.881 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.903709+0000", 2024-04-14T15:16:04.881 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012226500000000001 2024-04-14T15:16:04.881 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.881 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.881 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 2,3", 2024-04-14T15:16:04.881 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.903851+0000", 2024-04-14T15:16:04.881 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014190700000000001 2024-04-14T15:16:04.881 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.881 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.881 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:16:04.881 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.904351+0000", 2024-04-14T15:16:04.881 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00049993900000000003 2024-04-14T15:16:04.882 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.882 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.882 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:16:04.882 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.940363+0000", 2024-04-14T15:16:04.882 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.036011941999999998 2024-04-14T15:16:04.882 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.882 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.882 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:16:04.882 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.979696+0000", 2024-04-14T15:16:04.882 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.039332371999999997 2024-04-14T15:16:04.882 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.882 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.882 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:16:04.882 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.979838+0000", 2024-04-14T15:16:04.883 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014200000000000001 2024-04-14T15:16:04.883 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.883 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.883 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:16:04.883 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.979873+0000", 2024-04-14T15:16:04.883 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5428e-05 2024-04-14T15:16:04.883 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:04.883 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:16:04.883 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:04.883 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.883 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.883 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.42382.0:3 157.0 157:0384ab8e:::oid:head [write 0~128] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-04-14T15:16:04.883 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:13:26.844811+0000", 2024-04-14T15:16:04.883 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 157.91697847399999, 2024-04-14T15:16:04.884 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.152799729, 2024-04-14T15:16:04.884 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:16:04.884 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:16:04.884 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:16:04.884 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:16:04.884 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.42382", 2024-04-14T15:16:04.884 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/1837092947", 2024-04-14T15:16:04.884 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 3 2024-04-14T15:16:04.884 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.884 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:16:04.884 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.884 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:16:04.884 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.844811+0000", 2024-04-14T15:16:04.884 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:04.884 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.885 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.885 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:16:04.885 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.844811+0000", 2024-04-14T15:16:04.885 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:04.885 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.885 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.885 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:16:04.885 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.847180+0000", 2024-04-14T15:16:04.885 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0023693730000000001 2024-04-14T15:16:04.885 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.885 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.885 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:16:04.885 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.847195+0000", 2024-04-14T15:16:04.885 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.489e-05 2024-04-14T15:16:04.885 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.886 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.886 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:16:04.886 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.847214+0000", 2024-04-14T15:16:04.886 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8564e-05 2024-04-14T15:16:04.886 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.886 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.886 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:16:04.886 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.847225+0000", 2024-04-14T15:16:04.886 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0786e-05 2024-04-14T15:16:04.886 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.886 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.886 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:04.886 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.868351+0000", 2024-04-14T15:16:04.886 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.021126823999999999 2024-04-14T15:16:04.886 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.887 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.887 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:16:04.887 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.868370+0000", 2024-04-14T15:16:04.887 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8508e-05 2024-04-14T15:16:04.887 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.887 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.887 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:04.887 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.961204+0000", 2024-04-14T15:16:04.887 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.092834633999999999 2024-04-14T15:16:04.887 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.887 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.887 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:16:04.887 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.961327+0000", 2024-04-14T15:16:04.887 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012294199999999999 2024-04-14T15:16:04.887 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.888 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.888 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,6", 2024-04-14T15:16:04.889 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.961485+0000", 2024-04-14T15:16:04.889 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015743699999999999 2024-04-14T15:16:04.889 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.889 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.889 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:16:04.889 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.961970+0000", 2024-04-14T15:16:04.889 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00048496900000000003 2024-04-14T15:16:04.889 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.889 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.889 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:16:04.889 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.962400+0000", 2024-04-14T15:16:04.889 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00043008599999999999 2024-04-14T15:16:04.889 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.889 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.890 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:16:04.890 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.997503+0000", 2024-04-14T15:16:04.890 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.035103500000000003 2024-04-14T15:16:04.890 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.890 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.890 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:16:04.890 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.997559+0000", 2024-04-14T15:16:04.890 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5831000000000001e-05 2024-04-14T15:16:04.890 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.890 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.890 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:16:04.890 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.997611+0000", 2024-04-14T15:16:04.890 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.1384999999999997e-05 2024-04-14T15:16:04.890 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:04.891 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:16:04.891 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:04.891 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.891 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.891 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.42382.0:8 157.0 157:0384ab8e:::oid:head [write 128~128] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-04-14T15:16:04.891 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:13:27.847441+0000", 2024-04-14T15:16:04.891 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 156.91434880099999, 2024-04-14T15:16:04.891 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.059003879000000002, 2024-04-14T15:16:04.891 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:16:04.891 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:16:04.891 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:16:04.891 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:16:04.891 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.42382", 2024-04-14T15:16:04.891 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/1837092947", 2024-04-14T15:16:04.891 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 8 2024-04-14T15:16:04.892 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.892 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:16:04.892 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.892 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:16:04.892 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:27.847441+0000", 2024-04-14T15:16:04.892 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:04.892 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.892 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.892 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:16:04.892 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:27.847441+0000", 2024-04-14T15:16:04.892 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:04.892 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.892 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.892 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:16:04.893 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:27.847443+0000", 2024-04-14T15:16:04.893 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0590000000000001e-06 2024-04-14T15:16:04.893 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.893 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.893 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:16:04.893 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:27.847449+0000", 2024-04-14T15:16:04.893 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.7129999999999999e-06 2024-04-14T15:16:04.893 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.893 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.893 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:16:04.893 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:27.847460+0000", 2024-04-14T15:16:04.893 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0359e-05 2024-04-14T15:16:04.893 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.893 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.894 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:16:04.894 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:27.847472+0000", 2024-04-14T15:16:04.894 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.184e-05 2024-04-14T15:16:04.894 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.894 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.894 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:04.894 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:27.854881+0000", 2024-04-14T15:16:04.894 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0074090730000000004 2024-04-14T15:16:04.894 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.894 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.894 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:16:04.894 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:27.854956+0000", 2024-04-14T15:16:04.894 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5054000000000003e-05 2024-04-14T15:16:04.894 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.894 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.895 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,6", 2024-04-14T15:16:04.895 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:27.856348+0000", 2024-04-14T15:16:04.895 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001392643 2024-04-14T15:16:04.895 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.895 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.895 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:16:04.895 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:27.857818+0000", 2024-04-14T15:16:04.895 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0014697 2024-04-14T15:16:04.895 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.895 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.895 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:16:04.895 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:27.858388+0000", 2024-04-14T15:16:04.895 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00057031499999999995 2024-04-14T15:16:04.895 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.896 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.896 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:16:04.896 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:27.906371+0000", 2024-04-14T15:16:04.896 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.047982524999999998 2024-04-14T15:16:04.896 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.896 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.896 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:16:04.896 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:27.906406+0000", 2024-04-14T15:16:04.896 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5475999999999997e-05 2024-04-14T15:16:04.896 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.896 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.896 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:16:04.896 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:27.906444+0000", 2024-04-14T15:16:04.896 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8121999999999999e-05 2024-04-14T15:16:04.896 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:04.897 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:16:04.897 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:04.897 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.897 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.897 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.36864.0:1 259.7s0 259:e732be06:::oid_WaitForSafePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e274)", 2024-04-14T15:16:04.897 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:15:41.861974+0000", 2024-04-14T15:16:04.897 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 22.899814899999999, 2024-04-14T15:16:04.897 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.009675004, 2024-04-14T15:16:04.897 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:16:04.897 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:16:04.897 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:16:04.897 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:16:04.897 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.36864", 2024-04-14T15:16:04.897 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/3688286685", 2024-04-14T15:16:04.898 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 1 2024-04-14T15:16:04.898 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.898 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:16:04.898 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.898 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:16:04.898 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:41.861974+0000", 2024-04-14T15:16:04.898 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:04.898 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.898 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.898 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:16:04.898 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:41.861974+0000", 2024-04-14T15:16:04.898 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:04.898 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.898 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.898 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:16:04.899 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:41.861976+0000", 2024-04-14T15:16:04.899 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5090000000000001e-06 2024-04-14T15:16:04.899 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.899 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.899 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:16:04.899 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:41.861982+0000", 2024-04-14T15:16:04.899 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6659999999999996e-06 2024-04-14T15:16:04.899 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.899 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.899 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:16:04.899 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:41.861990+0000", 2024-04-14T15:16:04.899 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2789999999999998e-06 2024-04-14T15:16:04.899 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.899 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.899 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:16:04.900 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:41.861999+0000", 2024-04-14T15:16:04.900 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0040000000000005e-06 2024-04-14T15:16:04.900 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.900 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.900 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:04.900 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:41.862045+0000", 2024-04-14T15:16:04.900 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5679000000000001e-05 2024-04-14T15:16:04.900 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.900 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.900 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:16:04.900 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:41.862048+0000", 2024-04-14T15:16:04.900 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0639999999999998e-06 2024-04-14T15:16:04.900 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.900 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.901 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:04.901 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:42.869700+0000", 2024-04-14T15:16:04.901 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0076524060000001 2024-04-14T15:16:04.901 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.901 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.901 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:16:04.901 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:42.869825+0000", 2024-04-14T15:16:04.901 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012442599999999999 2024-04-14T15:16:04.901 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.901 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.901 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_started", 2024-04-14T15:16:04.901 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:42.870232+0000", 2024-04-14T15:16:04.901 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00040733700000000002 2024-04-14T15:16:04.901 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.901 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.901 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_committed", 2024-04-14T15:16:04.902 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:42.870790+0000", 2024-04-14T15:16:04.902 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00055832199999999998 2024-04-14T15:16:04.902 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.902 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.902 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:16:04.902 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:42.871585+0000", 2024-04-14T15:16:04.902 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00079456100000000005 2024-04-14T15:16:04.902 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:04.902 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:04.902 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:16:04.902 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:42.871649+0000", 2024-04-14T15:16:04.902 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4751000000000004e-05 2024-04-14T15:16:04.902 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:04.902 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:16:04.903 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:04.903 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:04.903 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:16:04.903 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-04-14T15:16:04.911 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:04.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:04.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:16:04.941 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:04 smithi090 ceph-mon[41670]: pgmap v389: 238 pgs: 238 active+clean; 4.4 MiB data, 564 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-04-14T15:16:04.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:04 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:16:04.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:04 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:16:04.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:04 smithi090 ceph-mon[41670]: 6.17 scrub starts 2024-04-14T15:16:04.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:04 smithi090 ceph-mon[41670]: 6.17 scrub ok 2024-04-14T15:16:04.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:04 smithi090 ceph-mon[41670]: 110.19 scrub starts 2024-04-14T15:16:04.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:04 smithi090 ceph-mon[41670]: 110.19 scrub ok 2024-04-14T15:16:04.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:04 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:16:04.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:04 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-35"}]: dispatch 2024-04-14T15:16:04.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:04 smithi090 ceph-mon[41670]: osdmap e297: 8 total, 8 up, 8 in 2024-04-14T15:16:04.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:04 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-35"}]: dispatch 2024-04-14T15:16:04.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:04 smithi090 ceph-mon[30399]: pgmap v389: 238 pgs: 238 active+clean; 4.4 MiB data, 564 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-04-14T15:16:04.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:04 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:16:04.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:04 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:16:04.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:04 smithi090 ceph-mon[30399]: 6.17 scrub starts 2024-04-14T15:16:04.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:04 smithi090 ceph-mon[30399]: 6.17 scrub ok 2024-04-14T15:16:04.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:04 smithi090 ceph-mon[30399]: 110.19 scrub starts 2024-04-14T15:16:04.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:04 smithi090 ceph-mon[30399]: 110.19 scrub ok 2024-04-14T15:16:04.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:04 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:16:04.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:04 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-35"}]: dispatch 2024-04-14T15:16:04.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:04 smithi090 ceph-mon[30399]: osdmap e297: 8 total, 8 up, 8 in 2024-04-14T15:16:04.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:04 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-35"}]: dispatch 2024-04-14T15:16:05.014 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:05.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:05.038 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:05.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:05.122 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:16:04 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:16:05.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:04 smithi092 ceph-mon[33966]: pgmap v389: 238 pgs: 238 active+clean; 4.4 MiB data, 564 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-04-14T15:16:05.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:04 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:16:05.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:04 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:16:05.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:04 smithi092 ceph-mon[33966]: 6.17 scrub starts 2024-04-14T15:16:05.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:04 smithi092 ceph-mon[33966]: 6.17 scrub ok 2024-04-14T15:16:05.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:04 smithi092 ceph-mon[33966]: 110.19 scrub starts 2024-04-14T15:16:05.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:04 smithi092 ceph-mon[33966]: 110.19 scrub ok 2024-04-14T15:16:05.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:04 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:16:05.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:04 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-35"}]: dispatch 2024-04-14T15:16:05.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:04 smithi092 ceph-mon[33966]: osdmap e297: 8 total, 8 up, 8 in 2024-04-14T15:16:05.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:04 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-35"}]: dispatch 2024-04-14T15:16:05.138 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:16:05.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:05.168 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:05.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:16:05.269 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:16:05.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:05.293 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:05.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:16:05.394 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:16:05.419 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:05.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:05.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:16:05.521 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:16:05.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:05.546 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:05.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:16:05.565 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-14T15:16:05.647 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:16:05.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:05.673 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:05.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:16:05.774 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:16:05.798 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:05.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:05.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:16:05.801 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:16:05.899 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:16:05.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:05.925 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:05.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:16:06.026 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:06.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:06.050 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:06.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:06.152 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:16:06.175 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:06.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:06.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:16:06.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:05 smithi090 ceph-mon[41670]: 110.1c scrub starts 2024-04-14T15:16:06.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:05 smithi090 ceph-mon[41670]: 110.1c scrub ok 2024-04-14T15:16:06.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:05 smithi090 ceph-mon[41670]: 110.4 scrub starts 2024-04-14T15:16:06.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:05 smithi090 ceph-mon[41670]: 110.4 scrub ok 2024-04-14T15:16:06.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:05 smithi090 ceph-mon[41670]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:16:06.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:05 smithi090 ceph-mon[41670]: from='client.48793 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi090-93363-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi090-93363-52"}]': finished 2024-04-14T15:16:06.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:05 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-35"}]': finished 2024-04-14T15:16:06.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:05 smithi090 ceph-mon[41670]: osdmap e298: 8 total, 8 up, 8 in 2024-04-14T15:16:06.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:05 smithi090 ceph-mon[30399]: 110.1c scrub starts 2024-04-14T15:16:06.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:05 smithi090 ceph-mon[30399]: 110.1c scrub ok 2024-04-14T15:16:06.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:05 smithi090 ceph-mon[30399]: 110.4 scrub starts 2024-04-14T15:16:06.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:05 smithi090 ceph-mon[30399]: 110.4 scrub ok 2024-04-14T15:16:06.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:05 smithi090 ceph-mon[30399]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:16:06.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:05 smithi090 ceph-mon[30399]: from='client.48793 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi090-93363-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi090-93363-52"}]': finished 2024-04-14T15:16:06.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:05 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-35"}]': finished 2024-04-14T15:16:06.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:05 smithi090 ceph-mon[30399]: osdmap e298: 8 total, 8 up, 8 in 2024-04-14T15:16:06.193 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:05 smithi092 ceph-mon[33966]: 110.1c scrub starts 2024-04-14T15:16:06.193 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:05 smithi092 ceph-mon[33966]: 110.1c scrub ok 2024-04-14T15:16:06.194 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:05 smithi092 ceph-mon[33966]: 110.4 scrub starts 2024-04-14T15:16:06.194 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:05 smithi092 ceph-mon[33966]: 110.4 scrub ok 2024-04-14T15:16:06.194 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:05 smithi092 ceph-mon[33966]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:16:06.194 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:05 smithi092 ceph-mon[33966]: from='client.48793 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi090-93363-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi090-93363-52"}]': finished 2024-04-14T15:16:06.194 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:05 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-35"}]': finished 2024-04-14T15:16:06.194 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:05 smithi092 ceph-mon[33966]: osdmap e298: 8 total, 8 up, 8 in 2024-04-14T15:16:06.278 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:06.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:06.304 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:06.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:06.405 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:16:06.428 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:06.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:06.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:16:06.530 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:06.555 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:06.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:06.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:06.657 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:16:06.681 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:06.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:06.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:16:06.783 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:16:06.807 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:06.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:06.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:16:06.909 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:06.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:06.921+0000 7f8377419700 1 -- 172.21.15.90:0/4037467935 >> v1:172.21.15.90:6789/0 conn(0x7f837010f350 legacy=0x7f8370111810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:06.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:06.922+0000 7f8377419700 1 -- 172.21.15.90:0/4037467935 shutdown_connections 2024-04-14T15:16:06.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:06.922+0000 7f8377419700 1 -- 172.21.15.90:0/4037467935 >> 172.21.15.90:0/4037467935 conn(0x7f83700fe3c0 msgr2=0x7f83701007e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:16:06.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:06.922+0000 7f8377419700 1 -- 172.21.15.90:0/4037467935 shutdown_connections 2024-04-14T15:16:06.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:06.922+0000 7f8377419700 1 -- 172.21.15.90:0/4037467935 wait complete. 2024-04-14T15:16:06.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:06.922+0000 7f8377419700 1 librados: starting msgr at 2024-04-14T15:16:06.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:06.922+0000 7f8377419700 1 librados: starting objecter 2024-04-14T15:16:06.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:06.922+0000 7f8377419700 1 Processor -- start 2024-04-14T15:16:06.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:06.922+0000 7f8377419700 1 -- start start 2024-04-14T15:16:06.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:06.922+0000 7f8377419700 1 librados: setting wanted keys 2024-04-14T15:16:06.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:06.922+0000 7f8377419700 1 librados: calling monclient init 2024-04-14T15:16:06.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:06.922+0000 7f8377419700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f837010f010 con 0x7f83701088d0 2024-04-14T15:16:06.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:06.922+0000 7f8377419700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f83701a8f60 con 0x7f837010f350 2024-04-14T15:16:06.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:06.923+0000 7f8377419700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f83701aa140 con 0x7f837010b780 2024-04-14T15:16:06.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:06.923+0000 7f83751b5700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f83701088d0 0x7f837010be60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:35864/0 (socket says 172.21.15.90:35864) 2024-04-14T15:16:06.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:06.923+0000 7f83751b5700 1 -- 172.21.15.90:0/1167609710 learned_addr learned my addr 172.21.15.90:0/1167609710 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:16:06.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:06.923+0000 7f83749b4700 1 --1- 172.21.15.90:0/1167609710 >> v1:172.21.15.90:6790/0 conn(0x7f837010b780 0x7f837010c570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:52970/0 (socket says 172.21.15.90:52970) 2024-04-14T15:16:06.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:06.923+0000 7f83667fc700 10 client.?.objecter ms_handle_connect 0x7f83701088d0 2024-04-14T15:16:06.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:06.923+0000 7f83667fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:16:06.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:06.923+0000 7f83667fc700 10 client.?.objecter ms_handle_connect 0x7f837010b780 2024-04-14T15:16:06.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:06.923+0000 7f83667fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:16:06.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:06.923+0000 7f83667fc700 10 client.?.objecter ms_handle_connect 0x7f837010f350 2024-04-14T15:16:06.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:06.923+0000 7f83667fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:16:06.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:06.923+0000 7f83667fc700 1 -- 172.21.15.90:0/1167609710 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 688658361 0 0) 0x7f83701aa140 con 0x7f837010b780 2024-04-14T15:16:06.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:06.923+0000 7f83667fc700 1 -- 172.21.15.90:0/1167609710 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f835c0031f0 con 0x7f837010b780 2024-04-14T15:16:06.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:06.923+0000 7f83667fc700 1 -- 172.21.15.90:0/1167609710 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 731893861 0 0) 0x7f83701a8f60 con 0x7f837010f350 2024-04-14T15:16:06.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:06.923+0000 7f83667fc700 1 -- 172.21.15.90:0/1167609710 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f83701aa140 con 0x7f837010f350 2024-04-14T15:16:06.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:06.923+0000 7f83667fc700 1 -- 172.21.15.90:0/1167609710 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1778180917 0 0) 0x7f837010f010 con 0x7f83701088d0 2024-04-14T15:16:06.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:06.923+0000 7f83667fc700 1 -- 172.21.15.90:0/1167609710 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f83701a8f60 con 0x7f83701088d0 2024-04-14T15:16:06.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:06.923+0000 7f83667fc700 1 -- 172.21.15.90:0/1167609710 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1642326787 0 0) 0x7f83701aa140 con 0x7f837010f350 2024-04-14T15:16:06.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:06.923+0000 7f83667fc700 1 -- 172.21.15.90:0/1167609710 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f837010f010 con 0x7f837010f350 2024-04-14T15:16:06.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:06.923+0000 7f83667fc700 1 -- 172.21.15.90:0/1167609710 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3490112838 0 0) 0x7f83701a8f60 con 0x7f83701088d0 2024-04-14T15:16:06.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:06.923+0000 7f83667fc700 1 -- 172.21.15.90:0/1167609710 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f83701aa140 con 0x7f83701088d0 2024-04-14T15:16:06.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:06.923+0000 7f83667fc700 1 -- 172.21.15.90:0/1167609710 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2449859809 0 0) 0x7f835c0031f0 con 0x7f837010b780 2024-04-14T15:16:06.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:06.923+0000 7f83667fc700 1 -- 172.21.15.90:0/1167609710 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f83701a8f60 con 0x7f837010b780 2024-04-14T15:16:06.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:06.923+0000 7f83667fc700 1 -- 172.21.15.90:0/1167609710 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f83680017b0 con 0x7f837010f350 2024-04-14T15:16:06.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:06.923+0000 7f83667fc700 1 -- 172.21.15.90:0/1167609710 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f836c001760 con 0x7f83701088d0 2024-04-14T15:16:06.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:06.923+0000 7f83667fc700 1 -- 172.21.15.90:0/1167609710 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f8360001830 con 0x7f837010b780 2024-04-14T15:16:06.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:06.923+0000 7f83667fc700 1 -- 172.21.15.90:0/1167609710 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1034518219 0 0) 0x7f837010f010 con 0x7f837010f350 2024-04-14T15:16:06.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:06.924+0000 7f83667fc700 1 -- 172.21.15.90:0/1167609710 >> v1:172.21.15.90:6790/0 conn(0x7f837010b780 legacy=0x7f837010c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:06.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:06.924+0000 7f83667fc700 1 -- 172.21.15.90:0/1167609710 >> v1:172.21.15.90:6789/0 conn(0x7f83701088d0 legacy=0x7f837010be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:06.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:06.924+0000 7f83667fc700 1 -- 172.21.15.90:0/1167609710 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83701ab320 con 0x7f837010f350 2024-04-14T15:16:06.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:06.924+0000 7f8377419700 1 -- 172.21.15.90:0/1167609710 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f83701aa370 con 0x7f837010f350 2024-04-14T15:16:06.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:06.924+0000 7f8377419700 10 client.48799.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:16:06.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:06.924+0000 7f8377419700 1 -- 172.21.15.90:0/1167609710 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f83701aa830 con 0x7f837010f350 2024-04-14T15:16:06.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:06.924+0000 7f8377419700 1 librados: init done 2024-04-14T15:16:06.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:06.924+0000 7f83667fc700 1 -- 172.21.15.90:0/1167609710 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f8368001a10 con 0x7f837010f350 2024-04-14T15:16:06.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:06.924+0000 7f83667fc700 1 -- 172.21.15.90:0/1167609710 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f8368004f40 con 0x7f837010f350 2024-04-14T15:16:06.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:06.925+0000 7f8377419700 1 -- 172.21.15.90:0/1167609710 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8354005150 con 0x7f837010f350 2024-04-14T15:16:06.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:06.925+0000 7f83667fc700 1 -- 172.21.15.90:0/1167609710 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f8368003e30 con 0x7f837010f350 2024-04-14T15:16:06.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:06.926+0000 7f83667fc700 1 -- 172.21.15.90:0/1167609710 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(299..299 src has 51..299) ==== 8612+0+0 (unknown 2481288328 0 0) 0x7f8368089b80 con 0x7f837010f350 2024-04-14T15:16:06.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:06.926+0000 7f83667fc700 10 client.48799.objecter ms_dispatch 0x7f8370001550 osd_map(299..299 src has 51..299) 2024-04-14T15:16:06.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:06.926+0000 7f83667fc700 3 client.48799.objecter handle_osd_map got epochs [299,299] > 0 2024-04-14T15:16:06.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:06.926+0000 7f83667fc700 3 client.48799.objecter handle_osd_map decoding full epoch 299 2024-04-14T15:16:06.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:06.926+0000 7f83667fc700 20 client.48799.objecter dump_active .. 0 homeless 2024-04-14T15:16:06.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:06.926+0000 7f83667fc700 10 client.48799.objecter ms_handle_connect 0x7f835c071510 2024-04-14T15:16:06.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:06.931+0000 7f83667fc700 1 -- 172.21.15.90:0/1167609710 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f83680563d0 con 0x7f837010f350 2024-04-14T15:16:06.933 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:06.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:06.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:06.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:06 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:16:06.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:06 smithi090 ceph-mon[41670]: pgmap v392: 238 pgs: 1 active+clean+snaptrim, 237 active+clean; 4.4 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-04-14T15:16:06.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:06 smithi090 ceph-mon[41670]: 6.13 scrub starts 2024-04-14T15:16:06.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:06 smithi090 ceph-mon[41670]: 6.13 scrub ok 2024-04-14T15:16:06.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:06 smithi090 ceph-mon[41670]: 5.1 scrub starts 2024-04-14T15:16:06.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:06 smithi090 ceph-mon[41670]: 5.1 scrub ok 2024-04-14T15:16:06.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:06 smithi090 ceph-mon[41670]: 110.3 scrub starts 2024-04-14T15:16:06.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:06 smithi090 ceph-mon[41670]: 110.3 scrub ok 2024-04-14T15:16:06.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:06 smithi090 ceph-mon[41670]: 110.15 scrub starts 2024-04-14T15:16:06.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:06 smithi090 ceph-mon[41670]: 110.15 scrub ok 2024-04-14T15:16:06.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:06 smithi090 ceph-mon[41670]: osdmap e299: 8 total, 8 up, 8 in 2024-04-14T15:16:06.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:06 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:16:06.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:06 smithi090 ceph-mon[30399]: pgmap v392: 238 pgs: 1 active+clean+snaptrim, 237 active+clean; 4.4 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-04-14T15:16:06.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:06 smithi090 ceph-mon[30399]: 6.13 scrub starts 2024-04-14T15:16:06.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:06 smithi090 ceph-mon[30399]: 6.13 scrub ok 2024-04-14T15:16:06.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:06 smithi090 ceph-mon[30399]: 5.1 scrub starts 2024-04-14T15:16:06.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:06 smithi090 ceph-mon[30399]: 5.1 scrub ok 2024-04-14T15:16:06.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:06 smithi090 ceph-mon[30399]: 110.3 scrub starts 2024-04-14T15:16:06.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:06 smithi090 ceph-mon[30399]: 110.3 scrub ok 2024-04-14T15:16:06.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:06 smithi090 ceph-mon[30399]: 110.15 scrub starts 2024-04-14T15:16:06.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:06 smithi090 ceph-mon[30399]: 110.15 scrub ok 2024-04-14T15:16:06.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:06 smithi090 ceph-mon[30399]: osdmap e299: 8 total, 8 up, 8 in 2024-04-14T15:16:07.035 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:16:07.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:07.061 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:07.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:16:07.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:07.139+0000 7f8377419700 1 -- 172.21.15.90:0/1167609710 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) -- 0x7f8354005440 con 0x7f837010f350 2024-04-14T15:16:07.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:07.141+0000 7f83667fc700 1 -- 172.21.15.90:0/1167609710 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v299) ==== 81+0+52683 (unknown 3796469153 0 1575902048) 0x7f836805a080 con 0x7f837010f350 2024-04-14T15:16:07.142 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:16:07.142 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-14T15:16:07.142 INFO:teuthology.orchestra.run.smithi090.stdout: "epoch": 299, 2024-04-14T15:16:07.142 INFO:teuthology.orchestra.run.smithi090.stdout: "fsid": "0e288098-fa70-11ee-bc8e-c7b262605968", 2024-04-14T15:16:07.143 INFO:teuthology.orchestra.run.smithi090.stdout: "created": "2024-04-14T15:04:05.808313+0000", 2024-04-14T15:16:07.143 INFO:teuthology.orchestra.run.smithi090.stdout: "modified": "2024-04-14T15:16:06.858863+0000", 2024-04-14T15:16:07.143 INFO:teuthology.orchestra.run.smithi090.stdout: "last_up_change": "2024-04-14T15:09:29.769352+0000", 2024-04-14T15:16:07.143 INFO:teuthology.orchestra.run.smithi090.stdout: "last_in_change": "2024-04-14T15:09:11.803164+0000", 2024-04-14T15:16:07.143 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-14T15:16:07.143 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_num": 5804032, 2024-04-14T15:16:07.143 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_set": [ 2024-04-14T15:16:07.143 INFO:teuthology.orchestra.run.smithi090.stdout: "nodeep-scrub", 2024-04-14T15:16:07.143 INFO:teuthology.orchestra.run.smithi090.stdout: "pglog_hardlimit", 2024-04-14T15:16:07.143 INFO:teuthology.orchestra.run.smithi090.stdout: "purged_snapdirs", 2024-04-14T15:16:07.143 INFO:teuthology.orchestra.run.smithi090.stdout: "recovery_deletes", 2024-04-14T15:16:07.143 INFO:teuthology.orchestra.run.smithi090.stdout: "sortbitwise" 2024-04-14T15:16:07.143 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-14T15:16:07.144 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_version": 86, 2024-04-14T15:16:07.144 INFO:teuthology.orchestra.run.smithi090.stdout: "full_ratio": 0.94999998807907104, 2024-04-14T15:16:07.144 INFO:teuthology.orchestra.run.smithi090.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-14T15:16:07.144 INFO:teuthology.orchestra.run.smithi090.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-14T15:16:07.144 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_snapshot": "", 2024-04-14T15:16:07.144 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_max": 269, 2024-04-14T15:16:07.144 INFO:teuthology.orchestra.run.smithi090.stdout: "max_osd": 8, 2024-04-14T15:16:07.144 INFO:teuthology.orchestra.run.smithi090.stdout: "require_min_compat_client": "luminous", 2024-04-14T15:16:07.144 INFO:teuthology.orchestra.run.smithi090.stdout: "min_compat_client": "luminous", 2024-04-14T15:16:07.144 INFO:teuthology.orchestra.run.smithi090.stdout: "require_osd_release": "squid", 2024-04-14T15:16:07.144 INFO:teuthology.orchestra.run.smithi090.stdout: "allow_crimson": false, 2024-04-14T15:16:07.144 INFO:teuthology.orchestra.run.smithi090.stdout: "pools": [ 2024-04-14T15:16:07.144 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.144 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 1, 2024-04-14T15:16:07.144 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": ".mgr", 2024-04-14T15:16:07.145 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:07:23.959628+0000", 2024-04-14T15:16:07.145 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-14T15:16:07.145 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool", 2024-04-14T15:16:07.145 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:16:07.145 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:16:07.145 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:16:07.145 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:16:07.145 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:16:07.145 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:16:07.145 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:16:07.145 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:16:07.145 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:16:07.145 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:16:07.145 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 11, 2024-04-14T15:16:07.146 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 11, 2024-04-14T15:16:07.146 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 11, 2024-04-14T15:16:07.146 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 11, 2024-04-14T15:16:07.146 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 11, 2024-04-14T15:16:07.146 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:16:07.146 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:16:07.146 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:16:07.146 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:16:07.146 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:16:07.146 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:16:07.146 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:16:07.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.146 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "186", 2024-04-14T15:16:07.147 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:16:07.147 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:16:07.147 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "184", 2024-04-14T15:16:07.147 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:16:07.147 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:16:07.147 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:16:07.147 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:16:07.147 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:16:07.147 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:16:07.147 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:16:07.147 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:16:07.147 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:16:07.147 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:16:07.147 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:16:07.147 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:16:07.148 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:16:07.148 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:16:07.148 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:16:07.148 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:16:07.148 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:16:07.148 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:16:07.148 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:16:07.148 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:16:07.148 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:16:07.148 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:16:07.148 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:16:07.148 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.148 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:16:07.148 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:16:07.148 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:16:07.149 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:16:07.149 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:16:07.149 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:16:07.149 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:16:07.149 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:16:07.149 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:16:07.149 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:16:07.149 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:16:07.149 INFO:teuthology.orchestra.run.smithi090.stdout: "options": { 2024-04-14T15:16:07.149 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_max": 32, 2024-04-14T15:16:07.149 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_min": 1 2024-04-14T15:16:07.149 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.149 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:16:07.149 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": {} 2024-04-14T15:16:07.150 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.150 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:16:07.150 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:16:07.150 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 1.9099999666213989, 2024-04-14T15:16:07.150 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 1.9099999666213989, 2024-04-14T15:16:07.150 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1.1399999856948853, 2024-04-14T15:16:07.150 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 2.1800000667572021, 2024-04-14T15:16:07.150 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 2.1800000667572021, 2024-04-14T15:16:07.150 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.875, 2024-04-14T15:16:07.150 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 1, 2024-04-14T15:16:07.150 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-04-14T15:16:07.150 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.150 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.150 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.151 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 2, 2024-04-14T15:16:07.151 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": ".rgw.root", 2024-04-14T15:16:07.151 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:09:41.608420+0000", 2024-04-14T15:16:07.151 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-14T15:16:07.151 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool", 2024-04-14T15:16:07.151 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:16:07.151 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:16:07.151 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:16:07.151 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:16:07.151 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:16:07.151 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:16:07.151 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:16:07.151 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:16:07.151 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:16:07.151 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:16:07.152 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:16:07.152 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:16:07.152 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:16:07.152 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:16:07.152 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:16:07.152 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:16:07.152 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:16:07.152 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:16:07.152 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:16:07.152 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:16:07.152 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:16:07.152 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:16:07.152 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.152 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "66", 2024-04-14T15:16:07.152 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:16:07.153 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:16:07.153 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:16:07.153 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:16:07.153 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:16:07.153 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:16:07.153 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:16:07.153 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:16:07.153 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:16:07.153 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:16:07.153 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:16:07.153 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:16:07.153 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:16:07.153 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:16:07.153 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:16:07.154 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:16:07.154 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:16:07.154 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:16:07.154 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:16:07.154 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:16:07.154 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:16:07.154 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:16:07.154 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:16:07.154 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:16:07.154 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:16:07.154 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:16:07.154 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.154 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:16:07.154 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:16:07.155 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:16:07.155 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:16:07.155 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:16:07.155 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:16:07.155 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:16:07.155 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:16:07.155 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:16:07.155 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:16:07.155 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:16:07.155 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:16:07.155 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:16:07.155 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw": {} 2024-04-14T15:16:07.155 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.155 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:16:07.155 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:16:07.156 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 1.5399999618530273, 2024-04-14T15:16:07.156 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 1.5399999618530273, 2024-04-14T15:16:07.156 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1.1399999856948853, 2024-04-14T15:16:07.156 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 1.75, 2024-04-14T15:16:07.156 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 1.75, 2024-04-14T15:16:07.156 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.875, 2024-04-14T15:16:07.156 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 1, 2024-04-14T15:16:07.156 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-04-14T15:16:07.156 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.156 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.156 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.156 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 3, 2024-04-14T15:16:07.156 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "datapool", 2024-04-14T15:16:07.156 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:09:41.744391+0000", 2024-04-14T15:16:07.156 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 8193, 2024-04-14T15:16:07.157 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-14T15:16:07.157 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:16:07.157 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:16:07.157 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:16:07.157 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:16:07.157 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:16:07.157 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:16:07.157 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:16:07.157 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:16:07.157 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:16:07.157 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:16:07.157 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 3, 2024-04-14T15:16:07.157 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 3, 2024-04-14T15:16:07.157 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 3, 2024-04-14T15:16:07.157 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 3, 2024-04-14T15:16:07.158 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 3, 2024-04-14T15:16:07.158 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:16:07.158 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:16:07.158 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:16:07.158 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:16:07.158 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:16:07.158 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:16:07.159 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:16:07.159 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.159 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "69", 2024-04-14T15:16:07.159 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:16:07.159 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:16:07.159 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:16:07.159 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:16:07.159 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:16:07.159 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 2, 2024-04-14T15:16:07.160 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 69, 2024-04-14T15:16:07.160 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:16:07.160 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:16:07.160 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:16:07.160 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:16:07.160 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:16:07.160 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:16:07.160 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:16:07.160 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:16:07.160 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:16:07.160 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:16:07.160 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:16:07.161 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:16:07.161 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:16:07.161 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:16:07.161 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:16:07.161 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:16:07.161 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:16:07.161 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:16:07.161 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:16:07.161 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.161 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:16:07.161 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:16:07.162 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:16:07.162 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:16:07.162 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:16:07.162 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:16:07.162 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:16:07.162 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:16:07.162 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:16:07.162 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:16:07.162 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:16:07.162 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:16:07.162 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:16:07.162 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd": {} 2024-04-14T15:16:07.162 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.162 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:16:07.163 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:16:07.163 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 2.3299999237060547, 2024-04-14T15:16:07.163 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 2.3299999237060547, 2024-04-14T15:16:07.163 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:16:07.163 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 2.3299999237060547, 2024-04-14T15:16:07.163 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 2.3299999237060547, 2024-04-14T15:16:07.163 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.875, 2024-04-14T15:16:07.163 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 1, 2024-04-14T15:16:07.163 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-04-14T15:16:07.163 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.163 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.163 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.163 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 4, 2024-04-14T15:16:07.163 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "default.rgw.log", 2024-04-14T15:16:07.164 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:09:43.088157+0000", 2024-04-14T15:16:07.164 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-14T15:16:07.164 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool", 2024-04-14T15:16:07.164 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:16:07.164 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:16:07.164 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:16:07.164 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:16:07.164 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:16:07.164 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:16:07.164 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:16:07.164 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:16:07.164 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:16:07.164 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:16:07.164 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:16:07.165 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:16:07.165 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:16:07.165 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:16:07.165 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:16:07.165 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:16:07.165 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:16:07.165 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:16:07.165 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:16:07.165 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:16:07.165 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:16:07.165 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:16:07.165 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.165 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "68", 2024-04-14T15:16:07.165 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:16:07.165 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:16:07.166 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:16:07.166 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:16:07.166 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:16:07.166 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:16:07.166 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:16:07.166 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:16:07.166 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:16:07.166 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:16:07.166 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:16:07.166 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:16:07.167 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:16:07.167 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:16:07.167 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:16:07.167 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:16:07.167 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:16:07.167 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:16:07.167 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:16:07.167 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:16:07.167 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:16:07.167 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:16:07.167 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:16:07.168 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:16:07.168 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:16:07.168 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:16:07.168 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.168 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:16:07.169 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:16:07.169 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:16:07.169 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:16:07.169 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:16:07.169 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:16:07.169 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:16:07.169 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:16:07.169 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:16:07.169 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:16:07.169 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:16:07.169 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:16:07.169 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:16:07.169 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw": {} 2024-04-14T15:16:07.169 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.170 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:16:07.170 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:16:07.170 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 1.9700000286102295, 2024-04-14T15:16:07.170 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 1.9700000286102295, 2024-04-14T15:16:07.170 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1.1399999856948853, 2024-04-14T15:16:07.170 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 2.25, 2024-04-14T15:16:07.170 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 2.25, 2024-04-14T15:16:07.170 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.875, 2024-04-14T15:16:07.170 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 1, 2024-04-14T15:16:07.170 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-04-14T15:16:07.170 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.170 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.170 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.170 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 5, 2024-04-14T15:16:07.171 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "default.rgw.control", 2024-04-14T15:16:07.171 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:09:45.042722+0000", 2024-04-14T15:16:07.171 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-14T15:16:07.171 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool", 2024-04-14T15:16:07.171 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:16:07.171 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:16:07.171 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:16:07.171 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:16:07.171 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:16:07.171 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:16:07.171 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:16:07.171 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:16:07.171 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:16:07.171 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:16:07.172 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:16:07.172 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:16:07.172 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:16:07.172 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:16:07.172 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:16:07.172 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:16:07.172 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:16:07.172 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:16:07.172 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:16:07.172 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:16:07.172 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:16:07.172 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:16:07.172 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.172 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "70", 2024-04-14T15:16:07.173 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:16:07.173 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:16:07.173 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:16:07.173 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:16:07.173 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:16:07.173 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:16:07.173 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:16:07.173 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:16:07.173 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:16:07.173 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:16:07.173 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:16:07.173 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:16:07.173 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:16:07.173 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:16:07.174 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:16:07.174 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:16:07.174 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:16:07.174 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:16:07.174 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:16:07.174 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:16:07.174 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:16:07.174 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:16:07.174 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:16:07.174 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:16:07.174 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:16:07.174 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:16:07.174 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.174 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:16:07.175 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:16:07.175 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:16:07.175 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:16:07.175 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:16:07.175 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:16:07.175 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:16:07.175 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:16:07.175 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:16:07.175 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:16:07.175 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:16:07.175 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:16:07.175 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:16:07.175 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw": {} 2024-04-14T15:16:07.175 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.175 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:16:07.176 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:16:07.176 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 1.3200000524520874, 2024-04-14T15:16:07.176 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 1.3200000524520874, 2024-04-14T15:16:07.176 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1.1399999856948853, 2024-04-14T15:16:07.176 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 1.5, 2024-04-14T15:16:07.176 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 1.5, 2024-04-14T15:16:07.176 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.875, 2024-04-14T15:16:07.176 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 1, 2024-04-14T15:16:07.176 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-04-14T15:16:07.176 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.176 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.176 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.176 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 6, 2024-04-14T15:16:07.177 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "default.rgw.meta", 2024-04-14T15:16:07.177 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:09:47.111336+0000", 2024-04-14T15:16:07.177 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-14T15:16:07.177 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool", 2024-04-14T15:16:07.177 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:16:07.177 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:16:07.177 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:16:07.177 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:16:07.177 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:16:07.177 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:16:07.177 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:16:07.177 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:16:07.177 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:16:07.177 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:16:07.177 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:16:07.178 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:16:07.178 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:16:07.178 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:16:07.178 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:16:07.178 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:16:07.178 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:16:07.178 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:16:07.178 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:16:07.178 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:16:07.178 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:16:07.178 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:16:07.178 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.178 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "72", 2024-04-14T15:16:07.179 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:16:07.179 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:16:07.179 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:16:07.179 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:16:07.179 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:16:07.179 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:16:07.179 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:16:07.179 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:16:07.179 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:16:07.179 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:16:07.179 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:16:07.179 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:16:07.179 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:16:07.179 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:16:07.179 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:16:07.180 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:16:07.180 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:16:07.180 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:16:07.180 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:16:07.180 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:16:07.180 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:16:07.180 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:16:07.180 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:16:07.180 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:16:07.180 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:16:07.180 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:16:07.180 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.180 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:16:07.180 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:16:07.180 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:16:07.181 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:16:07.181 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:16:07.181 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:16:07.181 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:16:07.181 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:16:07.181 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:16:07.181 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:16:07.181 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:16:07.181 INFO:teuthology.orchestra.run.smithi090.stdout: "options": { 2024-04-14T15:16:07.181 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_bias": 4 2024-04-14T15:16:07.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.181 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:16:07.181 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw": {} 2024-04-14T15:16:07.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.182 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:16:07.182 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:16:07.182 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 1.75, 2024-04-14T15:16:07.182 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 1.75, 2024-04-14T15:16:07.182 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1.1399999856948853, 2024-04-14T15:16:07.182 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 2, 2024-04-14T15:16:07.182 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 2, 2024-04-14T15:16:07.182 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.875, 2024-04-14T15:16:07.182 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 1, 2024-04-14T15:16:07.182 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-04-14T15:16:07.182 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.183 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 11, 2024-04-14T15:16:07.183 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "ceph_test_rados_api_asio", 2024-04-14T15:16:07.183 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:12:54.225671+0000", 2024-04-14T15:16:07.183 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-14T15:16:07.183 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool", 2024-04-14T15:16:07.183 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:16:07.183 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:16:07.183 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:16:07.183 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:16:07.183 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:16:07.183 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:16:07.183 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:16:07.183 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:16:07.183 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:16:07.184 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:16:07.184 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:16:07.184 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:16:07.184 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:16:07.184 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:16:07.184 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:16:07.184 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:16:07.184 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:16:07.184 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:16:07.184 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:16:07.184 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:16:07.184 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:16:07.184 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:16:07.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.184 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "109", 2024-04-14T15:16:07.185 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:16:07.185 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:16:07.185 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:16:07.185 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:16:07.185 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:16:07.185 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:16:07.185 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:16:07.185 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:16:07.185 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:16:07.185 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:16:07.185 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:16:07.185 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:16:07.185 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:16:07.185 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:16:07.185 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:16:07.186 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:16:07.186 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:16:07.186 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:16:07.186 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:16:07.186 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:16:07.186 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:16:07.186 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:16:07.186 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:16:07.186 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:16:07.186 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:16:07.186 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:16:07.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.186 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:16:07.186 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:16:07.187 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:16:07.187 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:16:07.187 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:16:07.187 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:16:07.187 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:16:07.187 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:16:07.187 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:16:07.187 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:16:07.187 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:16:07.187 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:16:07.187 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": {}, 2024-04-14T15:16:07.187 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:16:07.187 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:16:07.187 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 1.75, 2024-04-14T15:16:07.188 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 1.75, 2024-04-14T15:16:07.188 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1.1399999856948853, 2024-04-14T15:16:07.188 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 2, 2024-04-14T15:16:07.188 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 2, 2024-04-14T15:16:07.188 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.875, 2024-04-14T15:16:07.188 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 1, 2024-04-14T15:16:07.188 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-04-14T15:16:07.188 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.188 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 110, 2024-04-14T15:16:07.188 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "test-rados-api-smithi090-93652-6", 2024-04-14T15:16:07.188 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:13:07.960794+0000", 2024-04-14T15:16:07.188 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 8193, 2024-04-14T15:16:07.189 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-14T15:16:07.189 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:16:07.189 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:16:07.189 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:16:07.189 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:16:07.189 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:16:07.189 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:16:07.189 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:16:07.189 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:16:07.189 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:16:07.189 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:16:07.189 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:16:07.189 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:16:07.189 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:16:07.189 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:16:07.190 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:16:07.190 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:16:07.190 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:16:07.190 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:16:07.190 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:16:07.190 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:16:07.190 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:16:07.190 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:16:07.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.190 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "298", 2024-04-14T15:16:07.190 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "297", 2024-04-14T15:16:07.190 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "297", 2024-04-14T15:16:07.190 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "297", 2024-04-14T15:16:07.190 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:16:07.191 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:16:07.191 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 14, 2024-04-14T15:16:07.191 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 296, 2024-04-14T15:16:07.191 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:16:07.191 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:16:07.191 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:16:07.191 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:16:07.191 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:16:07.191 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:16:07.191 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:16:07.191 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:16:07.191 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:16:07.191 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:16:07.191 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:16:07.191 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:16:07.192 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:16:07.192 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:16:07.192 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:16:07.192 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:16:07.192 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:16:07.192 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:16:07.192 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:16:07.192 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.192 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:16:07.192 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:16:07.192 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:16:07.192 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:16:07.193 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:16:07.193 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:16:07.193 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:16:07.193 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:16:07.193 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:16:07.193 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:16:07.193 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:16:07.193 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:16:07.193 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:16:07.194 INFO:teuthology.orchestra.run.smithi090.stdout: "rados": {} 2024-04-14T15:16:07.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.194 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:16:07.194 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:16:07.194 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 1.5399999618530273, 2024-04-14T15:16:07.194 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 1.5399999618530273, 2024-04-14T15:16:07.194 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1.1399999856948853, 2024-04-14T15:16:07.194 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 1.75, 2024-04-14T15:16:07.195 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 1.75, 2024-04-14T15:16:07.195 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.875, 2024-04-14T15:16:07.195 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 1, 2024-04-14T15:16:07.195 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-04-14T15:16:07.195 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.195 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 269, 2024-04-14T15:16:07.195 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "RoundTripSparseReadPP_smithi090-93363-52", 2024-04-14T15:16:07.195 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:16:04.882616+0000", 2024-04-14T15:16:07.195 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 32769, 2024-04-14T15:16:07.195 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,creating", 2024-04-14T15:16:07.196 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 3, 2024-04-14T15:16:07.196 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:16:07.196 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:16:07.196 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 1, 2024-04-14T15:16:07.196 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:16:07.196 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:16:07.196 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:16:07.196 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:16:07.196 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:16:07.196 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:16:07.196 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 8, 2024-04-14T15:16:07.196 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 8, 2024-04-14T15:16:07.196 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 8, 2024-04-14T15:16:07.196 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 8, 2024-04-14T15:16:07.196 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 8, 2024-04-14T15:16:07.197 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:16:07.197 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:16:07.197 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:16:07.197 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:16:07.197 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:16:07.197 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:16:07.197 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:16:07.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.197 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "298", 2024-04-14T15:16:07.197 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:16:07.197 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:16:07.197 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:16:07.197 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:16:07.198 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:16:07.198 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:16:07.198 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:16:07.198 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:16:07.198 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:16:07.198 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:16:07.198 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:16:07.198 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:16:07.198 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:16:07.198 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:16:07.198 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:16:07.198 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:16:07.198 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:16:07.198 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:16:07.199 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:16:07.199 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:16:07.199 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:16:07.199 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:16:07.199 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:16:07.199 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "testprofile-RoundTripSparseReadPP_smithi090-93363-52", 2024-04-14T15:16:07.199 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:16:07.199 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:16:07.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.199 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:16:07.199 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:16:07.199 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:16:07.199 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:16:07.199 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:16:07.200 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:16:07.200 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:16:07.200 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:16:07.200 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 8192, 2024-04-14T15:16:07.200 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:16:07.200 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:16:07.200 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:16:07.200 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": {} 2024-04-14T15:16:07.200 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.200 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-14T15:16:07.200 INFO:teuthology.orchestra.run.smithi090.stdout: "osds": [ 2024-04-14T15:16:07.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.200 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 0, 2024-04-14T15:16:07.200 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "76772cfc-89d1-4013-b617-32284ed8dfd9", 2024-04-14T15:16:07.201 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-14T15:16:07.201 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-14T15:16:07.201 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-14T15:16:07.201 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-14T15:16:07.201 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-14T15:16:07.201 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-14T15:16:07.201 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 8, 2024-04-14T15:16:07.201 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 298, 2024-04-14T15:16:07.201 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-14T15:16:07.201 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-14T15:16:07.201 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-14T15:16:07.201 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:16:07.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.201 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:16:07.201 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6801", 2024-04-14T15:16:07.202 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3761806885 2024-04-14T15:16:07.202 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.202 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:16:07.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.202 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-14T15:16:07.202 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:16:07.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.202 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:16:07.202 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6802", 2024-04-14T15:16:07.202 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3761806885 2024-04-14T15:16:07.202 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.202 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:16:07.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.202 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-14T15:16:07.203 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:16:07.203 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.203 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:16:07.203 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6804", 2024-04-14T15:16:07.203 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3761806885 2024-04-14T15:16:07.203 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.203 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:16:07.203 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.203 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-14T15:16:07.203 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:16:07.203 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.203 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:16:07.203 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6803", 2024-04-14T15:16:07.203 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3761806885 2024-04-14T15:16:07.204 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.204 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:16:07.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.204 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6801/3761806885", 2024-04-14T15:16:07.204 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6802/3761806885", 2024-04-14T15:16:07.204 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6804/3761806885", 2024-04-14T15:16:07.204 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6803/3761806885", 2024-04-14T15:16:07.204 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-14T15:16:07.204 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-14T15:16:07.204 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-14T15:16:07.204 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:16:07.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.204 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.204 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 1, 2024-04-14T15:16:07.204 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "de4a7fa2-7d57-4ff3-a4fa-28c0cde8ef6c", 2024-04-14T15:16:07.205 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-14T15:16:07.205 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-14T15:16:07.205 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-14T15:16:07.205 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-14T15:16:07.205 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-14T15:16:07.205 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-14T15:16:07.205 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2024-04-14T15:16:07.205 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 298, 2024-04-14T15:16:07.205 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-14T15:16:07.205 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-14T15:16:07.205 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-14T15:16:07.205 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:16:07.205 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.205 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:16:07.206 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6805", 2024-04-14T15:16:07.206 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 4170361699 2024-04-14T15:16:07.206 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.206 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:16:07.206 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.206 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-14T15:16:07.206 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:16:07.206 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.206 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:16:07.206 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6806", 2024-04-14T15:16:07.206 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 4170361699 2024-04-14T15:16:07.206 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.206 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:16:07.206 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.207 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-14T15:16:07.207 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:16:07.207 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.207 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:16:07.207 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6808", 2024-04-14T15:16:07.207 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 4170361699 2024-04-14T15:16:07.207 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.207 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:16:07.207 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.207 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-14T15:16:07.207 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:16:07.207 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.207 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:16:07.207 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6807", 2024-04-14T15:16:07.207 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 4170361699 2024-04-14T15:16:07.208 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.208 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:16:07.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.208 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6805/4170361699", 2024-04-14T15:16:07.208 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6806/4170361699", 2024-04-14T15:16:07.208 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6808/4170361699", 2024-04-14T15:16:07.208 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6807/4170361699", 2024-04-14T15:16:07.208 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-14T15:16:07.208 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-14T15:16:07.208 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-14T15:16:07.208 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:16:07.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.208 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.208 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 2, 2024-04-14T15:16:07.208 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "9c39b5ff-6175-46ba-9d71-370581c6ff3d", 2024-04-14T15:16:07.209 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-14T15:16:07.209 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-14T15:16:07.209 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-14T15:16:07.209 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-14T15:16:07.209 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-14T15:16:07.209 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-14T15:16:07.209 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 18, 2024-04-14T15:16:07.209 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 298, 2024-04-14T15:16:07.209 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-14T15:16:07.209 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-14T15:16:07.209 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-14T15:16:07.209 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:16:07.209 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.209 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:16:07.209 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6809", 2024-04-14T15:16:07.210 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 1686184251 2024-04-14T15:16:07.210 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.210 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:16:07.210 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.210 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-14T15:16:07.210 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:16:07.210 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:07.143+0000 7f8377419700 10 librados: watch_flush enter 2024-04-14T15:16:07.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:07.143+0000 7f8377419700 10 librados: watch_flush exit 2024-04-14T15:16:07.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:07.143+0000 7f8377419700 20 client.48799.objecter shutdown clearing up homeless session... 2024-04-14T15:16:07.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:07.143+0000 7f8377419700 10 client.48799.objecter successfully canceled tick 2024-04-14T15:16:07.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:07.143+0000 7f8377419700 1 -- 172.21.15.90:0/1167609710 >> v1:172.21.15.90:6800/420935427 conn(0x7f835c071510 legacy=0x7f835c0739d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:07.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:07.143+0000 7f8377419700 1 -- 172.21.15.90:0/1167609710 >> v1:172.21.15.92:6789/0 conn(0x7f837010f350 legacy=0x7f83701a6830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:07.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:07.143+0000 7f8377419700 1 -- 172.21.15.90:0/1167609710 shutdown_connections 2024-04-14T15:16:07.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:07.143+0000 7f8377419700 1 -- 172.21.15.90:0/1167609710 >> 172.21.15.90:0/1167609710 conn(0x7f83700fe3c0 msgr2=0x7f8370112860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:16:07.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:07.143+0000 7f8377419700 1 -- 172.21.15.90:0/1167609710 shutdown_connections 2024-04-14T15:16:07.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:07.143+0000 7f8377419700 1 -- 172.21.15.90:0/1167609710 wait complete. 2024-04-14T15:16:07.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:07.143+0000 7f8377419700 1 librados: shutdown 2024-04-14T15:16:07.212 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:16:07.212 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6810", 2024-04-14T15:16:07.212 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 1686184251 2024-04-14T15:16:07.212 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.212 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:16:07.212 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.212 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-14T15:16:07.212 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:16:07.212 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.212 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:16:07.212 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6812", 2024-04-14T15:16:07.213 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 1686184251 2024-04-14T15:16:07.213 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.213 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:16:07.213 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.213 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-14T15:16:07.213 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:16:07.213 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.213 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:16:07.213 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6811", 2024-04-14T15:16:07.213 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 1686184251 2024-04-14T15:16:07.213 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.213 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:16:07.213 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.213 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6809/1686184251", 2024-04-14T15:16:07.214 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6810/1686184251", 2024-04-14T15:16:07.214 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6812/1686184251", 2024-04-14T15:16:07.214 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6811/1686184251", 2024-04-14T15:16:07.214 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-14T15:16:07.214 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-14T15:16:07.214 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-14T15:16:07.214 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:16:07.214 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.214 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.214 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 3, 2024-04-14T15:16:07.214 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "efd2ca45-5b80-4013-a571-499e2cb5b2e8", 2024-04-14T15:16:07.214 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-14T15:16:07.214 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-14T15:16:07.214 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-14T15:16:07.215 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-14T15:16:07.215 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-14T15:16:07.215 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-14T15:16:07.215 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 25, 2024-04-14T15:16:07.215 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 293, 2024-04-14T15:16:07.215 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-14T15:16:07.215 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-14T15:16:07.215 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-14T15:16:07.215 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:16:07.215 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.215 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:16:07.215 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6813", 2024-04-14T15:16:07.215 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 2270166774 2024-04-14T15:16:07.215 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.216 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:16:07.216 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.216 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-14T15:16:07.216 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:16:07.216 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.216 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:16:07.216 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6814", 2024-04-14T15:16:07.216 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 2270166774 2024-04-14T15:16:07.216 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.216 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:16:07.216 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.216 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-14T15:16:07.216 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:16:07.216 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.216 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:16:07.217 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6816", 2024-04-14T15:16:07.217 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 2270166774 2024-04-14T15:16:07.217 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.217 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:16:07.217 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.217 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-14T15:16:07.217 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:16:07.217 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.217 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:16:07.217 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6815", 2024-04-14T15:16:07.217 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 2270166774 2024-04-14T15:16:07.217 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.217 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:16:07.217 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.218 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6813/2270166774", 2024-04-14T15:16:07.218 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6814/2270166774", 2024-04-14T15:16:07.218 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6816/2270166774", 2024-04-14T15:16:07.218 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6815/2270166774", 2024-04-14T15:16:07.218 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-14T15:16:07.218 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-14T15:16:07.218 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-14T15:16:07.218 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:16:07.218 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.218 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.218 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 4, 2024-04-14T15:16:07.218 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "e3d9013d-5eb6-433c-96b3-3fc38ba7ac8d", 2024-04-14T15:16:07.218 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-14T15:16:07.218 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-14T15:16:07.219 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-14T15:16:07.219 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-14T15:16:07.219 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-14T15:16:07.219 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-14T15:16:07.219 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 31, 2024-04-14T15:16:07.219 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 298, 2024-04-14T15:16:07.219 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-14T15:16:07.219 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-14T15:16:07.219 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-14T15:16:07.219 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:16:07.219 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.219 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:16:07.219 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6800", 2024-04-14T15:16:07.219 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3887549733 2024-04-14T15:16:07.219 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.220 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:16:07.220 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.220 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-14T15:16:07.220 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:16:07.220 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.220 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:16:07.220 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6801", 2024-04-14T15:16:07.220 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3887549733 2024-04-14T15:16:07.220 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.220 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:16:07.220 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.220 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-14T15:16:07.220 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:16:07.220 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.221 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:16:07.221 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6803", 2024-04-14T15:16:07.221 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3887549733 2024-04-14T15:16:07.221 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.221 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:16:07.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.221 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-14T15:16:07.221 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:16:07.221 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.221 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:16:07.221 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6802", 2024-04-14T15:16:07.221 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3887549733 2024-04-14T15:16:07.221 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.221 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:16:07.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.222 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.92:6800/3887549733", 2024-04-14T15:16:07.222 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.92:6801/3887549733", 2024-04-14T15:16:07.222 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.92:6803/3887549733", 2024-04-14T15:16:07.222 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.92:6802/3887549733", 2024-04-14T15:16:07.222 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-14T15:16:07.222 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-14T15:16:07.222 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-14T15:16:07.222 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:16:07.222 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.222 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.222 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 5, 2024-04-14T15:16:07.222 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "1ec41805-6f82-4139-81bc-75305ba9e958", 2024-04-14T15:16:07.223 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-14T15:16:07.223 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-14T15:16:07.223 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-14T15:16:07.223 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 0, 2024-04-14T15:16:07.223 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-14T15:16:07.223 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-14T15:16:07.223 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 41, 2024-04-14T15:16:07.223 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 271, 2024-04-14T15:16:07.223 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-14T15:16:07.223 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-14T15:16:07.223 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-14T15:16:07.223 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:16:07.223 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.223 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:16:07.223 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6804", 2024-04-14T15:16:07.224 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3244561440 2024-04-14T15:16:07.224 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.224 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:16:07.224 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.224 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-14T15:16:07.224 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:16:07.224 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.224 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:16:07.224 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6805", 2024-04-14T15:16:07.224 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3244561440 2024-04-14T15:16:07.224 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.224 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:16:07.224 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.224 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-14T15:16:07.224 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:16:07.225 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.225 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:16:07.225 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6807", 2024-04-14T15:16:07.225 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3244561440 2024-04-14T15:16:07.225 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.225 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:16:07.225 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.225 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-14T15:16:07.225 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:16:07.225 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.225 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:16:07.225 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6806", 2024-04-14T15:16:07.225 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3244561440 2024-04-14T15:16:07.225 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.225 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:16:07.226 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.226 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.92:6804/3244561440", 2024-04-14T15:16:07.226 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.92:6805/3244561440", 2024-04-14T15:16:07.226 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.92:6807/3244561440", 2024-04-14T15:16:07.226 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.92:6806/3244561440", 2024-04-14T15:16:07.226 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-14T15:16:07.226 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-14T15:16:07.226 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-14T15:16:07.226 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:16:07.226 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.226 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.226 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 6, 2024-04-14T15:16:07.226 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "ca977ab2-bf8d-4b3a-ae5e-0efa5bc7a3ad", 2024-04-14T15:16:07.226 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-14T15:16:07.227 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-14T15:16:07.227 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-14T15:16:07.227 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-14T15:16:07.227 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-14T15:16:07.227 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-14T15:16:07.227 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 49, 2024-04-14T15:16:07.227 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 298, 2024-04-14T15:16:07.227 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-14T15:16:07.227 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-14T15:16:07.227 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-14T15:16:07.227 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:16:07.227 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.227 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:16:07.227 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6808", 2024-04-14T15:16:07.228 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 636125963 2024-04-14T15:16:07.228 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.228 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:16:07.228 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.228 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-14T15:16:07.228 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:16:07.228 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.228 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:16:07.228 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6809", 2024-04-14T15:16:07.228 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 636125963 2024-04-14T15:16:07.228 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.228 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:16:07.228 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.228 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-14T15:16:07.228 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:16:07.229 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.229 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:16:07.229 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6811", 2024-04-14T15:16:07.229 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 636125963 2024-04-14T15:16:07.229 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.229 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:16:07.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.229 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-14T15:16:07.229 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:16:07.229 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.229 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:16:07.229 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6810", 2024-04-14T15:16:07.229 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 636125963 2024-04-14T15:16:07.229 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.229 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:16:07.230 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.230 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.92:6808/636125963", 2024-04-14T15:16:07.230 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.92:6809/636125963", 2024-04-14T15:16:07.230 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.92:6811/636125963", 2024-04-14T15:16:07.230 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.92:6810/636125963", 2024-04-14T15:16:07.230 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-14T15:16:07.230 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-14T15:16:07.230 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-14T15:16:07.230 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:16:07.230 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.230 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.230 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 7, 2024-04-14T15:16:07.230 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "e5354645-79ec-4d4c-b6e5-f9b0653915ee", 2024-04-14T15:16:07.230 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-14T15:16:07.231 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-14T15:16:07.231 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-14T15:16:07.231 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-14T15:16:07.231 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-14T15:16:07.231 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-14T15:16:07.231 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 58, 2024-04-14T15:16:07.231 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 298, 2024-04-14T15:16:07.231 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-14T15:16:07.231 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-14T15:16:07.231 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-14T15:16:07.231 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:16:07.231 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.231 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:16:07.231 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6812", 2024-04-14T15:16:07.231 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 1773706716 2024-04-14T15:16:07.232 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.232 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:16:07.232 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.232 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-14T15:16:07.232 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:16:07.232 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.232 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:16:07.232 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6813", 2024-04-14T15:16:07.232 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 1773706716 2024-04-14T15:16:07.232 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.232 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:16:07.232 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.232 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-14T15:16:07.232 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:16:07.233 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.233 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:16:07.233 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6815", 2024-04-14T15:16:07.233 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 1773706716 2024-04-14T15:16:07.233 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.233 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:16:07.233 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.233 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-14T15:16:07.233 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:16:07.233 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.233 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:16:07.233 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6814", 2024-04-14T15:16:07.233 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 1773706716 2024-04-14T15:16:07.233 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.233 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:16:07.234 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.234 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.92:6812/1773706716", 2024-04-14T15:16:07.234 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.92:6813/1773706716", 2024-04-14T15:16:07.234 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.92:6815/1773706716", 2024-04-14T15:16:07.234 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.92:6814/1773706716", 2024-04-14T15:16:07.234 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-14T15:16:07.234 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-14T15:16:07.234 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-14T15:16:07.234 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:16:07.234 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.234 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-14T15:16:07.234 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_xinfo": [ 2024-04-14T15:16:07.234 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.234 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 0, 2024-04-14T15:16:07.235 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-14T15:16:07.235 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-14T15:16:07.235 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-14T15:16:07.235 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540701547738038271, 2024-04-14T15:16:07.235 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-14T15:16:07.235 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-14T15:15:39.318766+0000", 2024-04-14T15:16:07.235 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-14T15:16:07.235 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.235 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.235 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 1, 2024-04-14T15:16:07.235 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-14T15:16:07.235 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-14T15:16:07.235 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-14T15:16:07.235 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540701547738038271, 2024-04-14T15:16:07.236 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-14T15:16:07.236 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-14T15:15:29.773262+0000", 2024-04-14T15:16:07.236 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-14T15:16:07.236 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.236 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.236 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 2, 2024-04-14T15:16:07.236 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-14T15:16:07.236 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-14T15:16:07.236 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-14T15:16:07.236 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540701547738038271, 2024-04-14T15:16:07.236 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-14T15:16:07.236 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-14T15:15:38.442760+0000", 2024-04-14T15:16:07.236 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-14T15:16:07.236 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.236 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.237 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 3, 2024-04-14T15:16:07.237 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-14T15:16:07.237 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-14T15:16:07.237 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-14T15:16:07.237 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540701547738038271, 2024-04-14T15:16:07.237 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-14T15:16:07.237 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-14T15:15:32.198726+0000", 2024-04-14T15:16:07.237 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-14T15:16:07.237 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.237 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.237 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 4, 2024-04-14T15:16:07.237 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-14T15:16:07.237 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-14T15:16:07.237 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-14T15:16:07.237 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540701547738038271, 2024-04-14T15:16:07.238 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-14T15:16:07.238 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-14T15:15:45.008171+0000", 2024-04-14T15:16:07.238 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-14T15:16:07.238 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.238 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.238 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 5, 2024-04-14T15:16:07.238 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-14T15:16:07.238 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-14T15:16:07.238 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-14T15:16:07.238 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540701547738038271, 2024-04-14T15:16:07.238 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-14T15:16:07.238 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-14T15:15:12.105372+0000", 2024-04-14T15:16:07.238 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-14T15:16:07.238 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.239 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.239 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 6, 2024-04-14T15:16:07.239 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-14T15:16:07.239 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-14T15:16:07.239 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-14T15:16:07.239 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540701547738038271, 2024-04-14T15:16:07.239 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-14T15:16:07.239 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-14T15:15:24.397005+0000", 2024-04-14T15:16:07.239 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-14T15:16:07.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.239 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.239 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 7, 2024-04-14T15:16:07.239 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-14T15:16:07.239 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-14T15:16:07.239 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-14T15:16:07.240 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540701547738038271, 2024-04-14T15:16:07.240 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-14T15:16:07.240 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-14T15:15:45.036409+0000", 2024-04-14T15:16:07.240 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-14T15:16:07.240 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.240 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-14T15:16:07.240 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_upmap": [], 2024-04-14T15:16:07.240 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_upmap_items": [ 2024-04-14T15:16:07.240 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.240 INFO:teuthology.orchestra.run.smithi090.stdout: "pgid": "4.2", 2024-04-14T15:16:07.240 INFO:teuthology.orchestra.run.smithi090.stdout: "mappings": [ 2024-04-14T15:16:07.240 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.240 INFO:teuthology.orchestra.run.smithi090.stdout: "from": 1, 2024-04-14T15:16:07.240 INFO:teuthology.orchestra.run.smithi090.stdout: "to": 2 2024-04-14T15:16:07.240 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.241 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:16:07.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.241 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.241 INFO:teuthology.orchestra.run.smithi090.stdout: "pgid": "4.1f", 2024-04-14T15:16:07.241 INFO:teuthology.orchestra.run.smithi090.stdout: "mappings": [ 2024-04-14T15:16:07.241 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.241 INFO:teuthology.orchestra.run.smithi090.stdout: "from": 1, 2024-04-14T15:16:07.241 INFO:teuthology.orchestra.run.smithi090.stdout: "to": 2 2024-04-14T15:16:07.241 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.241 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:16:07.241 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.241 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-14T15:16:07.241 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_upmap_primaries": [], 2024-04-14T15:16:07.241 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_temp": [], 2024-04-14T15:16:07.241 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_temp": [], 2024-04-14T15:16:07.242 INFO:teuthology.orchestra.run.smithi090.stdout: "blocklist": { 2024-04-14T15:16:07.242 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:0/508768479": "2024-04-15T15:10:22.918674+0000", 2024-04-14T15:16:07.242 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:0/1823072952": "2024-04-15T15:10:22.918674+0000", 2024-04-14T15:16:07.242 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:6800/491071118": "2024-04-15T15:10:22.918674+0000", 2024-04-14T15:16:07.242 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:0/1337064035": "2024-04-15T15:10:22.918674+0000", 2024-04-14T15:16:07.242 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:0/769462306": "2024-04-15T15:10:22.918674+0000", 2024-04-14T15:16:07.242 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:0/1139868610": "2024-04-15T15:05:05.882361+0000", 2024-04-14T15:16:07.242 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:0/93763408": "2024-04-15T15:10:22.918674+0000", 2024-04-14T15:16:07.242 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:0/977818204": "2024-04-15T15:04:39.251711+0000", 2024-04-14T15:16:07.242 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:6800/941091171": "2024-04-15T15:05:05.882361+0000", 2024-04-14T15:16:07.242 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:0/3671453561": "2024-04-15T15:05:05.882361+0000", 2024-04-14T15:16:07.242 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:0/2611856200": "2024-04-15T15:05:05.882361+0000", 2024-04-14T15:16:07.242 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:6800/2306258962": "2024-04-15T15:04:39.251711+0000", 2024-04-14T15:16:07.242 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:0/4051692380": "2024-04-15T15:04:39.251711+0000", 2024-04-14T15:16:07.243 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:0/2681502561": "2024-04-15T15:04:39.251711+0000" 2024-04-14T15:16:07.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.243 INFO:teuthology.orchestra.run.smithi090.stdout: "range_blocklist": {}, 2024-04-14T15:16:07.243 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profiles": { 2024-04-14T15:16:07.243 INFO:teuthology.orchestra.run.smithi090.stdout: "default": { 2024-04-14T15:16:07.243 INFO:teuthology.orchestra.run.smithi090.stdout: "crush-failure-domain": "osd", 2024-04-14T15:16:07.243 INFO:teuthology.orchestra.run.smithi090.stdout: "k": "2", 2024-04-14T15:16:07.243 INFO:teuthology.orchestra.run.smithi090.stdout: "m": "1", 2024-04-14T15:16:07.243 INFO:teuthology.orchestra.run.smithi090.stdout: "plugin": "jerasure", 2024-04-14T15:16:07.243 INFO:teuthology.orchestra.run.smithi090.stdout: "technique": "reed_sol_van" 2024-04-14T15:16:07.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.243 INFO:teuthology.orchestra.run.smithi090.stdout: "testprofile-RoundTripSparseReadPP_smithi090-93363-52": { 2024-04-14T15:16:07.243 INFO:teuthology.orchestra.run.smithi090.stdout: "crush-device-class": "", 2024-04-14T15:16:07.243 INFO:teuthology.orchestra.run.smithi090.stdout: "crush-failure-domain": "osd", 2024-04-14T15:16:07.243 INFO:teuthology.orchestra.run.smithi090.stdout: "crush-num-failure-domains": "0", 2024-04-14T15:16:07.244 INFO:teuthology.orchestra.run.smithi090.stdout: "crush-osds-per-failure-domain": "0", 2024-04-14T15:16:07.244 INFO:teuthology.orchestra.run.smithi090.stdout: "crush-root": "default", 2024-04-14T15:16:07.244 INFO:teuthology.orchestra.run.smithi090.stdout: "jerasure-per-chunk-alignment": "false", 2024-04-14T15:16:07.244 INFO:teuthology.orchestra.run.smithi090.stdout: "k": "2", 2024-04-14T15:16:07.244 INFO:teuthology.orchestra.run.smithi090.stdout: "m": "1", 2024-04-14T15:16:07.244 INFO:teuthology.orchestra.run.smithi090.stdout: "plugin": "jerasure", 2024-04-14T15:16:07.244 INFO:teuthology.orchestra.run.smithi090.stdout: "technique": "reed_sol_van", 2024-04-14T15:16:07.244 INFO:teuthology.orchestra.run.smithi090.stdout: "w": "8" 2024-04-14T15:16:07.244 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.244 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps_queue": [ 2024-04-14T15:16:07.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.244 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 17, 2024-04-14T15:16:07.244 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:16:07.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.245 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 2, 2024-04-14T15:16:07.245 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 5 2024-04-14T15:16:07.245 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.245 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:16:07.245 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.245 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.245 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 18, 2024-04-14T15:16:07.245 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:16:07.245 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.245 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 2, 2024-04-14T15:16:07.245 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 3 2024-04-14T15:16:07.245 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.245 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:16:07.245 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.245 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.246 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 39, 2024-04-14T15:16:07.246 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:16:07.246 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.246 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 1, 2024-04-14T15:16:07.246 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2024-04-14T15:16:07.246 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.246 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:16:07.246 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.246 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.246 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 72, 2024-04-14T15:16:07.246 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:16:07.246 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.246 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 1, 2024-04-14T15:16:07.246 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2024-04-14T15:16:07.247 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.247 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:16:07.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.247 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 99, 2024-04-14T15:16:07.247 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:16:07.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.247 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 4, 2024-04-14T15:16:07.247 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 2 2024-04-14T15:16:07.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.247 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 7, 2024-04-14T15:16:07.247 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 2 2024-04-14T15:16:07.247 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.247 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:16:07.248 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.248 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.248 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 109, 2024-04-14T15:16:07.248 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:16:07.248 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.248 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 1, 2024-04-14T15:16:07.248 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2024-04-14T15:16:07.248 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.248 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:16:07.248 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.248 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.248 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 110, 2024-04-14T15:16:07.248 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:16:07.248 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.248 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 14, 2024-04-14T15:16:07.249 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2024-04-14T15:16:07.249 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.249 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:16:07.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.249 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.249 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 128, 2024-04-14T15:16:07.249 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:16:07.249 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.249 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 1, 2024-04-14T15:16:07.249 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 2 2024-04-14T15:16:07.249 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.249 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:16:07.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.249 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.250 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 142, 2024-04-14T15:16:07.250 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:16:07.250 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.250 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 3, 2024-04-14T15:16:07.250 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2024-04-14T15:16:07.250 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.250 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:16:07.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.250 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.250 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 149, 2024-04-14T15:16:07.250 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:16:07.250 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.250 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 1, 2024-04-14T15:16:07.250 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 4 2024-04-14T15:16:07.250 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.251 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:16:07.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.251 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 156, 2024-04-14T15:16:07.251 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:16:07.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.251 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 2, 2024-04-14T15:16:07.251 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2024-04-14T15:16:07.251 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.251 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:16:07.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.251 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 157, 2024-04-14T15:16:07.251 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:16:07.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.252 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 2, 2024-04-14T15:16:07.252 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 2 2024-04-14T15:16:07.252 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.252 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:16:07.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.252 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 167, 2024-04-14T15:16:07.252 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:16:07.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.252 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 2, 2024-04-14T15:16:07.252 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 3 2024-04-14T15:16:07.252 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.252 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:16:07.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.253 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.253 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 172, 2024-04-14T15:16:07.253 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:16:07.253 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.253 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 3, 2024-04-14T15:16:07.253 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 4 2024-04-14T15:16:07.253 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.253 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:16:07.253 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.253 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.253 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 175, 2024-04-14T15:16:07.253 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:16:07.253 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.253 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 2, 2024-04-14T15:16:07.253 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 4 2024-04-14T15:16:07.254 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.254 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:16:07.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.254 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.254 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 176, 2024-04-14T15:16:07.254 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:16:07.254 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.254 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 2, 2024-04-14T15:16:07.254 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 2 2024-04-14T15:16:07.254 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.254 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:16:07.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.254 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.254 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 187, 2024-04-14T15:16:07.255 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:16:07.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.255 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 2, 2024-04-14T15:16:07.255 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2024-04-14T15:16:07.255 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.255 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:16:07.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.255 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 192, 2024-04-14T15:16:07.255 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:16:07.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.255 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 1, 2024-04-14T15:16:07.255 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 4 2024-04-14T15:16:07.255 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.255 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:16:07.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.256 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 205, 2024-04-14T15:16:07.256 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:16:07.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.256 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 2, 2024-04-14T15:16:07.256 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 6 2024-04-14T15:16:07.256 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.256 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:16:07.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.256 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 213, 2024-04-14T15:16:07.256 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:16:07.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.256 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 7, 2024-04-14T15:16:07.256 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2024-04-14T15:16:07.257 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.257 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:16:07.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.257 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 234, 2024-04-14T15:16:07.257 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:16:07.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.257 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 8, 2024-04-14T15:16:07.257 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2024-04-14T15:16:07.257 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.257 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:16:07.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.257 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 247, 2024-04-14T15:16:07.258 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:16:07.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.258 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 10, 2024-04-14T15:16:07.258 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2024-04-14T15:16:07.258 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.258 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:16:07.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.258 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 252, 2024-04-14T15:16:07.258 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:16:07.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.258 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 11, 2024-04-14T15:16:07.258 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 2 2024-04-14T15:16:07.258 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.258 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:16:07.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:16:07.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.259 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 265, 2024-04-14T15:16:07.259 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:16:07.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:16:07.259 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 14, 2024-04-14T15:16:07.259 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2024-04-14T15:16:07.259 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.259 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:16:07.259 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.259 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-14T15:16:07.259 INFO:teuthology.orchestra.run.smithi090.stdout: "new_removed_snaps": [], 2024-04-14T15:16:07.259 INFO:teuthology.orchestra.run.smithi090.stdout: "new_purged_snaps": [], 2024-04-14T15:16:07.259 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_node_flags": {}, 2024-04-14T15:16:07.259 INFO:teuthology.orchestra.run.smithi090.stdout: "device_class_flags": {}, 2024-04-14T15:16:07.260 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_mode": { 2024-04-14T15:16:07.260 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_mode_enabled": false, 2024-04-14T15:16:07.260 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_bucket_count": 0, 2024-04-14T15:16:07.260 INFO:teuthology.orchestra.run.smithi090.stdout: "degraded_stretch_mode": 0, 2024-04-14T15:16:07.260 INFO:teuthology.orchestra.run.smithi090.stdout: "recovering_stretch_mode": 0, 2024-04-14T15:16:07.260 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_mode_bucket": 0 2024-04-14T15:16:07.260 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:16:07.260 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-14T15:16:07.260 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:16:07.261 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:06 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:16:07.261 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:06 smithi092 ceph-mon[33966]: pgmap v392: 238 pgs: 1 active+clean+snaptrim, 237 active+clean; 4.4 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-04-14T15:16:07.261 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:06 smithi092 ceph-mon[33966]: 6.13 scrub starts 2024-04-14T15:16:07.261 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:06 smithi092 ceph-mon[33966]: 6.13 scrub ok 2024-04-14T15:16:07.261 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:06 smithi092 ceph-mon[33966]: 5.1 scrub starts 2024-04-14T15:16:07.262 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:06 smithi092 ceph-mon[33966]: 5.1 scrub ok 2024-04-14T15:16:07.262 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:06 smithi092 ceph-mon[33966]: 110.3 scrub starts 2024-04-14T15:16:07.262 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:06 smithi092 ceph-mon[33966]: 110.3 scrub ok 2024-04-14T15:16:07.262 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:06 smithi092 ceph-mon[33966]: 110.15 scrub starts 2024-04-14T15:16:07.262 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:06 smithi092 ceph-mon[33966]: 110.15 scrub ok 2024-04-14T15:16:07.262 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:06 smithi092 ceph-mon[33966]: osdmap e299: 8 total, 8 up, 8 in 2024-04-14T15:16:07.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:07.286 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:07.286 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:16:07.387 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:16:07.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:07.411 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:07.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:16:07.448 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-04-14T15:16:07.448 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-04-14T15:16:07.448 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 0 2024-04-14T15:16:07.448 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-04-14T15:16:07.514 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:16:07.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:07.538 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:07.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:16:07.639 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:16:07.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:07.664 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:07.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:16:07.764 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:16:07.789 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:07.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:07.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:16:07.891 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:16:07.914 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:07.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:07.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:16:08.016 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:16:08.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:08.041 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:08.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:16:08.142 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:08.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:08.166 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:08.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:08.172 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:07 smithi090 ceph-mon[41670]: 4.1f scrub starts 2024-04-14T15:16:08.172 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:07 smithi090 ceph-mon[41670]: 4.1f scrub ok 2024-04-14T15:16:08.172 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:07 smithi090 ceph-mon[41670]: 5.e scrub starts 2024-04-14T15:16:08.172 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:07 smithi090 ceph-mon[41670]: 5.e scrub ok 2024-04-14T15:16:08.172 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:07 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1167609710' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-14T15:16:08.172 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:07 smithi090 ceph-mon[41670]: 110.17 scrub starts 2024-04-14T15:16:08.172 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:07 smithi090 ceph-mon[41670]: 110.17 scrub ok 2024-04-14T15:16:08.172 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:07 smithi090 ceph-mon[41670]: osdmap e300: 8 total, 8 up, 8 in 2024-04-14T15:16:08.172 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:07 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3028951022' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi090-93363-52"}]: dispatch 2024-04-14T15:16:08.173 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:07 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:16:08.173 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:07 smithi090 ceph-mon[41670]: from='client.48793 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi090-93363-52"}]: dispatch 2024-04-14T15:16:08.173 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:07 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:16:08.173 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:07 smithi090 ceph-mon[30399]: 4.1f scrub starts 2024-04-14T15:16:08.173 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:07 smithi090 ceph-mon[30399]: 4.1f scrub ok 2024-04-14T15:16:08.173 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:07 smithi090 ceph-mon[30399]: 5.e scrub starts 2024-04-14T15:16:08.173 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:07 smithi090 ceph-mon[30399]: 5.e scrub ok 2024-04-14T15:16:08.173 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:07 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1167609710' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-14T15:16:08.173 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:07 smithi090 ceph-mon[30399]: 110.17 scrub starts 2024-04-14T15:16:08.173 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:07 smithi090 ceph-mon[30399]: 110.17 scrub ok 2024-04-14T15:16:08.173 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:07 smithi090 ceph-mon[30399]: osdmap e300: 8 total, 8 up, 8 in 2024-04-14T15:16:08.173 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:07 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3028951022' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi090-93363-52"}]: dispatch 2024-04-14T15:16:08.174 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:07 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:16:08.174 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:07 smithi090 ceph-mon[30399]: from='client.48793 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi090-93363-52"}]: dispatch 2024-04-14T15:16:08.174 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:07 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:16:08.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.110+0000 7f660cce8700 1 -- 172.21.15.90:0/4098248127 >> v1:172.21.15.90:6789/0 conn(0x7f660810e260 legacy=0x7f6608110700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:08.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.110+0000 7f660cce8700 1 -- 172.21.15.90:0/4098248127 shutdown_connections 2024-04-14T15:16:08.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.110+0000 7f660cce8700 1 -- 172.21.15.90:0/4098248127 >> 172.21.15.90:0/4098248127 conn(0x7f6608076a70 msgr2=0x7f6608076e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:16:08.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.110+0000 7f660cce8700 1 -- 172.21.15.90:0/4098248127 shutdown_connections 2024-04-14T15:16:08.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.110+0000 7f660cce8700 1 -- 172.21.15.90:0/4098248127 wait complete. 2024-04-14T15:16:08.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.110+0000 7f660cce8700 1 librados: starting msgr at 2024-04-14T15:16:08.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.110+0000 7f660cce8700 1 librados: starting objecter 2024-04-14T15:16:08.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.111+0000 7f660cce8700 1 Processor -- start 2024-04-14T15:16:08.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.111+0000 7f660cce8700 1 -- start start 2024-04-14T15:16:08.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.111+0000 7f660cce8700 1 librados: setting wanted keys 2024-04-14T15:16:08.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.111+0000 7f660cce8700 1 librados: calling monclient init 2024-04-14T15:16:08.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.111+0000 7f660cce8700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6608107060 con 0x7f66081023d0 2024-04-14T15:16:08.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.111+0000 7f660cce8700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6608107270 con 0x7f660810e260 2024-04-14T15:16:08.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.111+0000 7f660cce8700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f66081ac260 con 0x7f660810a6c0 2024-04-14T15:16:08.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.111+0000 7f660659c700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f66081023d0 0x7f6608101060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:35880/0 (socket says 172.21.15.90:35880) 2024-04-14T15:16:08.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.111+0000 7f660659c700 1 -- 172.21.15.90:0/2464858537 learned_addr learned my addr 172.21.15.90:0/2464858537 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:16:08.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.111+0000 7f65f77fe700 10 client.?.objecter ms_handle_connect 0x7f66081023d0 2024-04-14T15:16:08.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.111+0000 7f65f77fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:16:08.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.111+0000 7f65f77fe700 10 client.?.objecter ms_handle_connect 0x7f660810a6c0 2024-04-14T15:16:08.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.111+0000 7f65f77fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:16:08.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.111+0000 7f65f77fe700 1 -- 172.21.15.90:0/2464858537 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3938799685 0 0) 0x7f66081ac260 con 0x7f660810a6c0 2024-04-14T15:16:08.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.111+0000 7f65f77fe700 1 -- 172.21.15.90:0/2464858537 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f65f00035b0 con 0x7f660810a6c0 2024-04-14T15:16:08.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.111+0000 7f65f77fe700 10 client.?.objecter ms_handle_connect 0x7f660810e260 2024-04-14T15:16:08.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.111+0000 7f65f77fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:16:08.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.111+0000 7f65f77fe700 1 -- 172.21.15.90:0/2464858537 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4287668251 0 0) 0x7f6608107060 con 0x7f66081023d0 2024-04-14T15:16:08.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.111+0000 7f65f77fe700 1 -- 172.21.15.90:0/2464858537 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f66081ac260 con 0x7f66081023d0 2024-04-14T15:16:08.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.111+0000 7f65f77fe700 1 -- 172.21.15.90:0/2464858537 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2352008224 0 0) 0x7f6608107270 con 0x7f660810e260 2024-04-14T15:16:08.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.111+0000 7f65f77fe700 1 -- 172.21.15.90:0/2464858537 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6608107060 con 0x7f660810e260 2024-04-14T15:16:08.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.111+0000 7f65f77fe700 1 -- 172.21.15.90:0/2464858537 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1787972700 0 0) 0x7f65f00035b0 con 0x7f660810a6c0 2024-04-14T15:16:08.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.111+0000 7f65f77fe700 1 -- 172.21.15.90:0/2464858537 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6608107270 con 0x7f660810a6c0 2024-04-14T15:16:08.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.111+0000 7f65f77fe700 1 -- 172.21.15.90:0/2464858537 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3657314460 0 0) 0x7f66081ac260 con 0x7f66081023d0 2024-04-14T15:16:08.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.112+0000 7f65f77fe700 1 -- 172.21.15.90:0/2464858537 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f65f00035b0 con 0x7f66081023d0 2024-04-14T15:16:08.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.112+0000 7f65f77fe700 1 -- 172.21.15.90:0/2464858537 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f65fc0018b0 con 0x7f660810a6c0 2024-04-14T15:16:08.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.112+0000 7f65f77fe700 1 -- 172.21.15.90:0/2464858537 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f65f80018b0 con 0x7f66081023d0 2024-04-14T15:16:08.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.112+0000 7f65f77fe700 1 -- 172.21.15.90:0/2464858537 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2693625901 0 0) 0x7f6608107060 con 0x7f660810e260 2024-04-14T15:16:08.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.112+0000 7f65f77fe700 1 -- 172.21.15.90:0/2464858537 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f66081ac260 con 0x7f660810e260 2024-04-14T15:16:08.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.112+0000 7f65f77fe700 1 -- 172.21.15.90:0/2464858537 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f66000017b0 con 0x7f660810e260 2024-04-14T15:16:08.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.112+0000 7f65f77fe700 1 -- 172.21.15.90:0/2464858537 <== mon.2 v1:172.21.15.90:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1591120046 0 0) 0x7f6608107270 con 0x7f660810a6c0 2024-04-14T15:16:08.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.112+0000 7f65f77fe700 1 -- 172.21.15.90:0/2464858537 >> v1:172.21.15.92:6789/0 conn(0x7f660810e260 legacy=0x7f66081a8b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:08.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.112+0000 7f65f77fe700 1 -- 172.21.15.90:0/2464858537 >> v1:172.21.15.90:6789/0 conn(0x7f66081023d0 legacy=0x7f6608101060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:08.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.112+0000 7f65f77fe700 1 -- 172.21.15.90:0/2464858537 --> v1:172.21.15.90:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66081ad480 con 0x7f660810a6c0 2024-04-14T15:16:08.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.112+0000 7f660cce8700 1 -- 172.21.15.90:0/2464858537 --> v1:172.21.15.90:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f66081ac490 con 0x7f660810a6c0 2024-04-14T15:16:08.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.112+0000 7f660cce8700 10 client.46919.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:16:08.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.112+0000 7f660cce8700 1 -- 172.21.15.90:0/2464858537 --> v1:172.21.15.90:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f66081ac950 con 0x7f660810a6c0 2024-04-14T15:16:08.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.112+0000 7f660cce8700 1 librados: init done 2024-04-14T15:16:08.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.112+0000 7f65f77fe700 1 -- 172.21.15.90:0/2464858537 <== mon.2 v1:172.21.15.90:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f65fc003b40 con 0x7f660810a6c0 2024-04-14T15:16:08.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.112+0000 7f65f77fe700 1 -- 172.21.15.90:0/2464858537 <== mon.2 v1:172.21.15.90:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f65fc0049d0 con 0x7f660810a6c0 2024-04-14T15:16:08.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.113+0000 7f660cce8700 1 -- 172.21.15.90:0/2464858537 --> v1:172.21.15.90:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f65e8005100 con 0x7f660810a6c0 2024-04-14T15:16:08.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.113+0000 7f65f77fe700 1 -- 172.21.15.90:0/2464858537 <== mon.2 v1:172.21.15.90:6790/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f65fc01b660 con 0x7f660810a6c0 2024-04-14T15:16:08.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.114+0000 7f65f77fe700 1 -- 172.21.15.90:0/2464858537 <== mon.2 v1:172.21.15.90:6790/0 8 ==== osd_map(300..300 src has 51..300) ==== 8553+0+0 (unknown 2259838249 0 0) 0x7f65fc089900 con 0x7f660810a6c0 2024-04-14T15:16:08.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.114+0000 7f65f77fe700 10 client.46919.objecter ms_dispatch 0x7f6608001550 osd_map(300..300 src has 51..300) 2024-04-14T15:16:08.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.114+0000 7f65f77fe700 3 client.46919.objecter handle_osd_map got epochs [300,300] > 0 2024-04-14T15:16:08.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.114+0000 7f65f77fe700 3 client.46919.objecter handle_osd_map decoding full epoch 300 2024-04-14T15:16:08.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.114+0000 7f65f77fe700 20 client.46919.objecter dump_active .. 0 homeless 2024-04-14T15:16:08.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.114+0000 7f65f77fe700 1 -- 172.21.15.90:0/2464858537 <== mon.2 v1:172.21.15.90:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v300) ==== 54+0+16 (unknown 2490460041 0 2420000192) 0x7f65fc089cf0 con 0x7f660810a6c0 2024-04-14T15:16:08.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.114+0000 7f65f77fe700 10 client.46919.objecter ms_handle_connect 0x7f65f0071350 2024-04-14T15:16:08.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.114+0000 7f660cce8700 10 client.46919.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:16:08.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.114+0000 7f660cce8700 15 client.46919.objecter _session_command_op_assign -1 1 2024-04-14T15:16:08.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.114+0000 7f660cce8700 20 client.46919.objecter _get_session s=0x7f65e80053e0 osd=0 3 2024-04-14T15:16:08.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.114+0000 7f660cce8700 20 client.46919.objecter put_session s=0x7f65e80053e0 osd=0 3 2024-04-14T15:16:08.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.114+0000 7f660cce8700 20 client.46919.objecter _get_session s=0x7f65e80053e0 osd=0 3 2024-04-14T15:16:08.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.114+0000 7f660cce8700 15 client.46919.objecter _session_command_op_remove -1 1 2024-04-14T15:16:08.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.114+0000 7f660cce8700 20 client.46919.objecter get_session s=0x7f65e80053e0 osd=0 3 2024-04-14T15:16:08.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.114+0000 7f660cce8700 15 client.46919.objecter _session_command_op_assign 0 1 2024-04-14T15:16:08.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.114+0000 7f660cce8700 20 client.46919.objecter put_session s=0x7f65e80053e0 osd=0 4 2024-04-14T15:16:08.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.114+0000 7f660cce8700 10 client.46919.objecter _send_command 1 2024-04-14T15:16:08.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.114+0000 7f660cce8700 1 -- 172.21.15.90:0/2464858537 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f65e8004b50 con 0x7f65e8004730 2024-04-14T15:16:08.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.114+0000 7f65f77fe700 10 client.46919.objecter ms_handle_connect 0x7f65e8004730 2024-04-14T15:16:08.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.115+0000 7f65f77fe700 1 -- 172.21.15.90:0/2464858537 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f65e8004b50 con 0x7f65e8004730 2024-04-14T15:16:08.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.115+0000 7f65f77fe700 10 client.46919.objecter ms_dispatch 0x7f6608001550 command_reply(tid 1: 0 ) 2024-04-14T15:16:08.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.115+0000 7f65f77fe700 10 client.46919.objecter _finish_command 1 = system:0 2024-04-14T15:16:08.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.115+0000 7f65f77fe700 20 client.46919.objecter put_session s=0x7f65e80053e0 osd=0 4 2024-04-14T15:16:08.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.115+0000 7f65f77fe700 15 client.46919.objecter _session_command_op_remove 0 1 2024-04-14T15:16:08.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.132+0000 7f660cce8700 10 client.46919.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:16:08.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.132+0000 7f660cce8700 15 client.46919.objecter _session_command_op_assign -1 2 2024-04-14T15:16:08.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.132+0000 7f660cce8700 20 client.46919.objecter _get_session s=0x7f65e80053e0 osd=0 3 2024-04-14T15:16:08.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.132+0000 7f660cce8700 20 client.46919.objecter put_session s=0x7f65e80053e0 osd=0 3 2024-04-14T15:16:08.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.132+0000 7f660cce8700 20 client.46919.objecter _get_session s=0x7f65e80053e0 osd=0 3 2024-04-14T15:16:08.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.132+0000 7f660cce8700 15 client.46919.objecter _session_command_op_remove -1 2 2024-04-14T15:16:08.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.132+0000 7f660cce8700 20 client.46919.objecter get_session s=0x7f65e80053e0 osd=0 3 2024-04-14T15:16:08.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.132+0000 7f660cce8700 15 client.46919.objecter _session_command_op_assign 0 2 2024-04-14T15:16:08.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.132+0000 7f660cce8700 20 client.46919.objecter put_session s=0x7f65e80053e0 osd=0 4 2024-04-14T15:16:08.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.132+0000 7f660cce8700 10 client.46919.objecter _send_command 2 2024-04-14T15:16:08.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.132+0000 7f660cce8700 1 -- 172.21.15.90:0/2464858537 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f65e8007b00 con 0x7f65e8004730 2024-04-14T15:16:08.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.134+0000 7f65f77fe700 1 -- 172.21.15.90:0/2464858537 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f65e8007b00 con 0x7f65e8004730 2024-04-14T15:16:08.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.134+0000 7f65f77fe700 10 client.46919.objecter ms_dispatch 0x7f6608001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:16:08.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.134+0000 7f65f77fe700 10 client.46919.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:16:08.234 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:16:08.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.134+0000 7f65f77fe700 20 client.46919.objecter put_session s=0x7f65e80053e0 osd=0 4 2024-04-14T15:16:08.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.134+0000 7f65f77fe700 15 client.46919.objecter _session_command_op_remove 0 2 2024-04-14T15:16:08.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.134+0000 7f660cce8700 10 client.46919.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:16:08.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.134+0000 7f660cce8700 15 client.46919.objecter _session_command_op_assign -1 3 2024-04-14T15:16:08.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.134+0000 7f660cce8700 20 client.46919.objecter _get_session s=0x7f65e8004130 osd=1 3 2024-04-14T15:16:08.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.134+0000 7f660cce8700 20 client.46919.objecter put_session s=0x7f65e8004130 osd=1 3 2024-04-14T15:16:08.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.134+0000 7f660cce8700 20 client.46919.objecter _get_session s=0x7f65e8004130 osd=1 3 2024-04-14T15:16:08.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.134+0000 7f660cce8700 15 client.46919.objecter _session_command_op_remove -1 3 2024-04-14T15:16:08.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.134+0000 7f660cce8700 20 client.46919.objecter get_session s=0x7f65e8004130 osd=1 3 2024-04-14T15:16:08.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.134+0000 7f660cce8700 15 client.46919.objecter _session_command_op_assign 1 3 2024-04-14T15:16:08.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.134+0000 7f660cce8700 20 client.46919.objecter put_session s=0x7f65e8004130 osd=1 4 2024-04-14T15:16:08.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.134+0000 7f660cce8700 10 client.46919.objecter _send_command 3 2024-04-14T15:16:08.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.134+0000 7f660cce8700 1 -- 172.21.15.90:0/2464858537 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f65e8002130 con 0x7f65e80042a0 2024-04-14T15:16:08.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.134+0000 7f65f77fe700 10 client.46919.objecter ms_handle_connect 0x7f65e80042a0 2024-04-14T15:16:08.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.135+0000 7f65f77fe700 1 -- 172.21.15.90:0/2464858537 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f65e8002130 con 0x7f65e80042a0 2024-04-14T15:16:08.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.135+0000 7f65f77fe700 10 client.46919.objecter ms_dispatch 0x7f6608001550 command_reply(tid 3: 0 ) 2024-04-14T15:16:08.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.135+0000 7f65f77fe700 10 client.46919.objecter _finish_command 3 = system:0 2024-04-14T15:16:08.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.135+0000 7f65f77fe700 20 client.46919.objecter put_session s=0x7f65e8004130 osd=1 4 2024-04-14T15:16:08.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.135+0000 7f65f77fe700 15 client.46919.objecter _session_command_op_remove 1 3 2024-04-14T15:16:08.236 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:16:08.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.152+0000 7f660cce8700 10 client.46919.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:16:08.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.152+0000 7f660cce8700 15 client.46919.objecter _session_command_op_assign -1 4 2024-04-14T15:16:08.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.152+0000 7f660cce8700 20 client.46919.objecter _get_session s=0x7f65e8004130 osd=1 3 2024-04-14T15:16:08.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.152+0000 7f660cce8700 20 client.46919.objecter put_session s=0x7f65e8004130 osd=1 3 2024-04-14T15:16:08.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.152+0000 7f660cce8700 20 client.46919.objecter _get_session s=0x7f65e8004130 osd=1 3 2024-04-14T15:16:08.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.152+0000 7f660cce8700 15 client.46919.objecter _session_command_op_remove -1 4 2024-04-14T15:16:08.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.152+0000 7f660cce8700 20 client.46919.objecter get_session s=0x7f65e8004130 osd=1 3 2024-04-14T15:16:08.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.152+0000 7f660cce8700 15 client.46919.objecter _session_command_op_assign 1 4 2024-04-14T15:16:08.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.152+0000 7f660cce8700 20 client.46919.objecter put_session s=0x7f65e8004130 osd=1 4 2024-04-14T15:16:08.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.152+0000 7f660cce8700 10 client.46919.objecter _send_command 4 2024-04-14T15:16:08.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.152+0000 7f660cce8700 1 -- 172.21.15.90:0/2464858537 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f65e8001600 con 0x7f65e80042a0 2024-04-14T15:16:08.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.154+0000 7f65f77fe700 1 -- 172.21.15.90:0/2464858537 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f65e8001600 con 0x7f65e80042a0 2024-04-14T15:16:08.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.154+0000 7f65f77fe700 10 client.46919.objecter ms_dispatch 0x7f6608001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:16:08.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.154+0000 7f65f77fe700 10 client.46919.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:16:08.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.154+0000 7f65f77fe700 20 client.46919.objecter put_session s=0x7f65e8004130 osd=1 4 2024-04-14T15:16:08.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.154+0000 7f65f77fe700 15 client.46919.objecter _session_command_op_remove 1 4 2024-04-14T15:16:08.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.154+0000 7f660cce8700 10 client.46919.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:16:08.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.154+0000 7f660cce8700 15 client.46919.objecter _session_command_op_assign -1 5 2024-04-14T15:16:08.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.154+0000 7f660cce8700 20 client.46919.objecter _get_session s=0x7f65e8001a00 osd=2 3 2024-04-14T15:16:08.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.154+0000 7f660cce8700 20 client.46919.objecter put_session s=0x7f65e8001a00 osd=2 3 2024-04-14T15:16:08.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.154+0000 7f660cce8700 20 client.46919.objecter _get_session s=0x7f65e8001a00 osd=2 3 2024-04-14T15:16:08.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.154+0000 7f660cce8700 15 client.46919.objecter _session_command_op_remove -1 5 2024-04-14T15:16:08.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.154+0000 7f660cce8700 20 client.46919.objecter get_session s=0x7f65e8001a00 osd=2 3 2024-04-14T15:16:08.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.154+0000 7f660cce8700 15 client.46919.objecter _session_command_op_assign 2 5 2024-04-14T15:16:08.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.154+0000 7f660cce8700 20 client.46919.objecter put_session s=0x7f65e8001a00 osd=2 4 2024-04-14T15:16:08.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.154+0000 7f660cce8700 10 client.46919.objecter _send_command 5 2024-04-14T15:16:08.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.154+0000 7f660cce8700 1 -- 172.21.15.90:0/2464858537 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f65e8010fe0 con 0x7f65e8008990 2024-04-14T15:16:08.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.154+0000 7f65f77fe700 10 client.46919.objecter ms_handle_connect 0x7f65e8008990 2024-04-14T15:16:08.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.155+0000 7f65f77fe700 1 -- 172.21.15.90:0/2464858537 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f65e8010fe0 con 0x7f65e8008990 2024-04-14T15:16:08.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.155+0000 7f65f77fe700 10 client.46919.objecter ms_dispatch 0x7f6608001550 command_reply(tid 5: 0 ) 2024-04-14T15:16:08.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.155+0000 7f65f77fe700 10 client.46919.objecter _finish_command 5 = system:0 2024-04-14T15:16:08.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.155+0000 7f65f77fe700 20 client.46919.objecter put_session s=0x7f65e8001a00 osd=2 4 2024-04-14T15:16:08.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.155+0000 7f65f77fe700 15 client.46919.objecter _session_command_op_remove 2 5 2024-04-14T15:16:08.240 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:16:08.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.171+0000 7f660cce8700 10 client.46919.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:16:08.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.171+0000 7f660cce8700 15 client.46919.objecter _session_command_op_assign -1 6 2024-04-14T15:16:08.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.171+0000 7f660cce8700 20 client.46919.objecter _get_session s=0x7f65e8001a00 osd=2 3 2024-04-14T15:16:08.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.171+0000 7f660cce8700 20 client.46919.objecter put_session s=0x7f65e8001a00 osd=2 3 2024-04-14T15:16:08.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.171+0000 7f660cce8700 20 client.46919.objecter _get_session s=0x7f65e8001a00 osd=2 3 2024-04-14T15:16:08.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.171+0000 7f660cce8700 15 client.46919.objecter _session_command_op_remove -1 6 2024-04-14T15:16:08.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.171+0000 7f660cce8700 20 client.46919.objecter get_session s=0x7f65e8001a00 osd=2 3 2024-04-14T15:16:08.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.171+0000 7f660cce8700 15 client.46919.objecter _session_command_op_assign 2 6 2024-04-14T15:16:08.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.171+0000 7f660cce8700 20 client.46919.objecter put_session s=0x7f65e8001a00 osd=2 4 2024-04-14T15:16:08.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.171+0000 7f660cce8700 10 client.46919.objecter _send_command 6 2024-04-14T15:16:08.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.171+0000 7f660cce8700 1 -- 172.21.15.90:0/2464858537 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f65e8007fa0 con 0x7f65e8008990 2024-04-14T15:16:08.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.173+0000 7f65f77fe700 1 -- 172.21.15.90:0/2464858537 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f65e8007fa0 con 0x7f65e8008990 2024-04-14T15:16:08.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.173+0000 7f65f77fe700 10 client.46919.objecter ms_dispatch 0x7f6608001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:16:08.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.173+0000 7f65f77fe700 10 client.46919.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:16:08.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.173+0000 7f65f77fe700 20 client.46919.objecter put_session s=0x7f65e8001a00 osd=2 4 2024-04-14T15:16:08.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.173+0000 7f65f77fe700 15 client.46919.objecter _session_command_op_remove 2 6 2024-04-14T15:16:08.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.173+0000 7f660cce8700 10 client.46919.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:16:08.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.173+0000 7f660cce8700 15 client.46919.objecter _session_command_op_assign -1 7 2024-04-14T15:16:08.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.173+0000 7f660cce8700 20 client.46919.objecter _get_session s=0x7f65e8008200 osd=3 3 2024-04-14T15:16:08.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.173+0000 7f660cce8700 20 client.46919.objecter put_session s=0x7f65e8008200 osd=3 3 2024-04-14T15:16:08.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.173+0000 7f660cce8700 20 client.46919.objecter _get_session s=0x7f65e8008200 osd=3 3 2024-04-14T15:16:08.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.173+0000 7f660cce8700 15 client.46919.objecter _session_command_op_remove -1 7 2024-04-14T15:16:08.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.173+0000 7f660cce8700 20 client.46919.objecter get_session s=0x7f65e8008200 osd=3 3 2024-04-14T15:16:08.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.173+0000 7f660cce8700 15 client.46919.objecter _session_command_op_assign 3 7 2024-04-14T15:16:08.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.173+0000 7f660cce8700 20 client.46919.objecter put_session s=0x7f65e8008200 osd=3 4 2024-04-14T15:16:08.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.173+0000 7f660cce8700 10 client.46919.objecter _send_command 7 2024-04-14T15:16:08.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.173+0000 7f660cce8700 1 -- 172.21.15.90:0/2464858537 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f65e800d500 con 0x7f65e8011190 2024-04-14T15:16:08.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.173+0000 7f65f77fe700 10 client.46919.objecter ms_handle_connect 0x7f65e8011190 2024-04-14T15:16:08.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.174+0000 7f65f77fe700 1 -- 172.21.15.90:0/2464858537 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f65e800d500 con 0x7f65e8011190 2024-04-14T15:16:08.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.174+0000 7f65f77fe700 10 client.46919.objecter ms_dispatch 0x7f6608001550 command_reply(tid 7: 0 ) 2024-04-14T15:16:08.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.174+0000 7f65f77fe700 10 client.46919.objecter _finish_command 7 = system:0 2024-04-14T15:16:08.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.174+0000 7f65f77fe700 20 client.46919.objecter put_session s=0x7f65e8008200 osd=3 4 2024-04-14T15:16:08.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.174+0000 7f65f77fe700 15 client.46919.objecter _session_command_op_remove 3 7 2024-04-14T15:16:08.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.190+0000 7f660cce8700 10 client.46919.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:16:08.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.190+0000 7f660cce8700 15 client.46919.objecter _session_command_op_assign -1 8 2024-04-14T15:16:08.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.190+0000 7f660cce8700 20 client.46919.objecter _get_session s=0x7f65e8008200 osd=3 3 2024-04-14T15:16:08.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.190+0000 7f660cce8700 20 client.46919.objecter put_session s=0x7f65e8008200 osd=3 3 2024-04-14T15:16:08.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.190+0000 7f660cce8700 20 client.46919.objecter _get_session s=0x7f65e8008200 osd=3 3 2024-04-14T15:16:08.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.190+0000 7f660cce8700 15 client.46919.objecter _session_command_op_remove -1 8 2024-04-14T15:16:08.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.190+0000 7f660cce8700 20 client.46919.objecter get_session s=0x7f65e8008200 osd=3 3 2024-04-14T15:16:08.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.190+0000 7f660cce8700 15 client.46919.objecter _session_command_op_assign 3 8 2024-04-14T15:16:08.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.190+0000 7f660cce8700 20 client.46919.objecter put_session s=0x7f65e8008200 osd=3 4 2024-04-14T15:16:08.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.190+0000 7f660cce8700 10 client.46919.objecter _send_command 8 2024-04-14T15:16:08.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.190+0000 7f660cce8700 1 -- 172.21.15.90:0/2464858537 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f65e8012570 con 0x7f65e8011190 2024-04-14T15:16:08.243 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:16:08.243 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:16:08.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.192+0000 7f65f77fe700 1 -- 172.21.15.90:0/2464858537 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f65e8012570 con 0x7f65e8011190 2024-04-14T15:16:08.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.192+0000 7f65f77fe700 10 client.46919.objecter ms_dispatch 0x7f6608001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:16:08.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.192+0000 7f65f77fe700 10 client.46919.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:16:08.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.192+0000 7f65f77fe700 20 client.46919.objecter put_session s=0x7f65e8008200 osd=3 4 2024-04-14T15:16:08.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.192+0000 7f65f77fe700 15 client.46919.objecter _session_command_op_remove 3 8 2024-04-14T15:16:08.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.192+0000 7f660cce8700 10 client.46919.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:16:08.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.192+0000 7f660cce8700 15 client.46919.objecter _session_command_op_assign -1 9 2024-04-14T15:16:08.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.192+0000 7f660cce8700 20 client.46919.objecter _get_session s=0x7f65e80136f0 osd=4 3 2024-04-14T15:16:08.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.192+0000 7f660cce8700 20 client.46919.objecter put_session s=0x7f65e80136f0 osd=4 3 2024-04-14T15:16:08.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.192+0000 7f660cce8700 20 client.46919.objecter _get_session s=0x7f65e80136f0 osd=4 3 2024-04-14T15:16:08.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.192+0000 7f660cce8700 15 client.46919.objecter _session_command_op_remove -1 9 2024-04-14T15:16:08.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.192+0000 7f660cce8700 20 client.46919.objecter get_session s=0x7f65e80136f0 osd=4 3 2024-04-14T15:16:08.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.192+0000 7f660cce8700 15 client.46919.objecter _session_command_op_assign 4 9 2024-04-14T15:16:08.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.192+0000 7f660cce8700 20 client.46919.objecter put_session s=0x7f65e80136f0 osd=4 4 2024-04-14T15:16:08.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.192+0000 7f660cce8700 10 client.46919.objecter _send_command 9 2024-04-14T15:16:08.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.192+0000 7f660cce8700 1 -- 172.21.15.90:0/2464858537 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f65e801a7e0 con 0x7f65e800cb20 2024-04-14T15:16:08.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.193+0000 7f65f77fe700 10 client.46919.objecter ms_handle_connect 0x7f65e800cb20 2024-04-14T15:16:08.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.193+0000 7f65f77fe700 1 -- 172.21.15.90:0/2464858537 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f65e801a7e0 con 0x7f65e800cb20 2024-04-14T15:16:08.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.194+0000 7f65f77fe700 10 client.46919.objecter ms_dispatch 0x7f6608001550 command_reply(tid 9: 0 ) 2024-04-14T15:16:08.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.194+0000 7f65f77fe700 10 client.46919.objecter _finish_command 9 = system:0 2024-04-14T15:16:08.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.194+0000 7f65f77fe700 20 client.46919.objecter put_session s=0x7f65e80136f0 osd=4 4 2024-04-14T15:16:08.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.194+0000 7f65f77fe700 15 client.46919.objecter _session_command_op_remove 4 9 2024-04-14T15:16:08.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.210+0000 7f660cce8700 10 client.46919.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:16:08.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.210+0000 7f660cce8700 15 client.46919.objecter _session_command_op_assign -1 10 2024-04-14T15:16:08.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.210+0000 7f660cce8700 20 client.46919.objecter _get_session s=0x7f65e80136f0 osd=4 3 2024-04-14T15:16:08.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.210+0000 7f660cce8700 20 client.46919.objecter put_session s=0x7f65e80136f0 osd=4 3 2024-04-14T15:16:08.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.210+0000 7f660cce8700 20 client.46919.objecter _get_session s=0x7f65e80136f0 osd=4 3 2024-04-14T15:16:08.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.210+0000 7f660cce8700 15 client.46919.objecter _session_command_op_remove -1 10 2024-04-14T15:16:08.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.210+0000 7f660cce8700 20 client.46919.objecter get_session s=0x7f65e80136f0 osd=4 3 2024-04-14T15:16:08.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.210+0000 7f660cce8700 15 client.46919.objecter _session_command_op_assign 4 10 2024-04-14T15:16:08.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.210+0000 7f660cce8700 20 client.46919.objecter put_session s=0x7f65e80136f0 osd=4 4 2024-04-14T15:16:08.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.210+0000 7f660cce8700 10 client.46919.objecter _send_command 10 2024-04-14T15:16:08.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.210+0000 7f660cce8700 1 -- 172.21.15.90:0/2464858537 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f65e80127c0 con 0x7f65e800cb20 2024-04-14T15:16:08.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.212+0000 7f65f77fe700 1 -- 172.21.15.90:0/2464858537 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f65e80127c0 con 0x7f65e800cb20 2024-04-14T15:16:08.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.212+0000 7f65f77fe700 10 client.46919.objecter ms_dispatch 0x7f6608001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:16:08.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.212+0000 7f65f77fe700 10 client.46919.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:16:08.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.212+0000 7f65f77fe700 20 client.46919.objecter put_session s=0x7f65e80136f0 osd=4 4 2024-04-14T15:16:08.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.212+0000 7f65f77fe700 15 client.46919.objecter _session_command_op_remove 4 10 2024-04-14T15:16:08.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.212+0000 7f660cce8700 10 client.46919.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:16:08.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.212+0000 7f660cce8700 15 client.46919.objecter _session_command_op_assign -1 11 2024-04-14T15:16:08.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.212+0000 7f660cce8700 20 client.46919.objecter _get_session s=0x7f65e800c930 osd=5 3 2024-04-14T15:16:08.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.212+0000 7f660cce8700 20 client.46919.objecter put_session s=0x7f65e800c930 osd=5 3 2024-04-14T15:16:08.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.212+0000 7f660cce8700 20 client.46919.objecter _get_session s=0x7f65e800c930 osd=5 3 2024-04-14T15:16:08.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.212+0000 7f660cce8700 15 client.46919.objecter _session_command_op_remove -1 11 2024-04-14T15:16:08.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.212+0000 7f660cce8700 20 client.46919.objecter get_session s=0x7f65e800c930 osd=5 3 2024-04-14T15:16:08.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.212+0000 7f660cce8700 15 client.46919.objecter _session_command_op_assign 5 11 2024-04-14T15:16:08.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.212+0000 7f660cce8700 20 client.46919.objecter put_session s=0x7f65e800c930 osd=5 4 2024-04-14T15:16:08.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.212+0000 7f660cce8700 10 client.46919.objecter _send_command 11 2024-04-14T15:16:08.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.212+0000 7f660cce8700 1 -- 172.21.15.90:0/2464858537 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f65e8013290 con 0x7f65e8012cd0 2024-04-14T15:16:08.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.212+0000 7f65f77fe700 10 client.46919.objecter ms_handle_connect 0x7f65e8012cd0 2024-04-14T15:16:08.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.213+0000 7f65f77fe700 1 -- 172.21.15.90:0/2464858537 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f65e8013290 con 0x7f65e8012cd0 2024-04-14T15:16:08.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.213+0000 7f65f77fe700 10 client.46919.objecter ms_dispatch 0x7f6608001550 command_reply(tid 11: 0 ) 2024-04-14T15:16:08.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.213+0000 7f65f77fe700 10 client.46919.objecter _finish_command 11 = system:0 2024-04-14T15:16:08.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.213+0000 7f65f77fe700 20 client.46919.objecter put_session s=0x7f65e800c930 osd=5 4 2024-04-14T15:16:08.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.213+0000 7f65f77fe700 15 client.46919.objecter _session_command_op_remove 5 11 2024-04-14T15:16:08.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.231+0000 7f660cce8700 10 client.46919.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:16:08.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.231+0000 7f660cce8700 15 client.46919.objecter _session_command_op_assign -1 12 2024-04-14T15:16:08.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.231+0000 7f660cce8700 20 client.46919.objecter _get_session s=0x7f65e800c930 osd=5 3 2024-04-14T15:16:08.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.231+0000 7f660cce8700 20 client.46919.objecter put_session s=0x7f65e800c930 osd=5 3 2024-04-14T15:16:08.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.231+0000 7f660cce8700 20 client.46919.objecter _get_session s=0x7f65e800c930 osd=5 3 2024-04-14T15:16:08.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.231+0000 7f660cce8700 15 client.46919.objecter _session_command_op_remove -1 12 2024-04-14T15:16:08.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.231+0000 7f660cce8700 20 client.46919.objecter get_session s=0x7f65e800c930 osd=5 3 2024-04-14T15:16:08.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.231+0000 7f660cce8700 15 client.46919.objecter _session_command_op_assign 5 12 2024-04-14T15:16:08.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.231+0000 7f660cce8700 20 client.46919.objecter put_session s=0x7f65e800c930 osd=5 4 2024-04-14T15:16:08.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.231+0000 7f660cce8700 10 client.46919.objecter _send_command 12 2024-04-14T15:16:08.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.231+0000 7f660cce8700 1 -- 172.21.15.90:0/2464858537 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f65e80134f0 con 0x7f65e8012cd0 2024-04-14T15:16:08.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.233+0000 7f65f77fe700 1 -- 172.21.15.90:0/2464858537 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f65e80134f0 con 0x7f65e8012cd0 2024-04-14T15:16:08.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.233+0000 7f65f77fe700 10 client.46919.objecter ms_dispatch 0x7f6608001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:16:08.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.233+0000 7f65f77fe700 10 client.46919.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:16:08.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.233+0000 7f65f77fe700 20 client.46919.objecter put_session s=0x7f65e800c930 osd=5 4 2024-04-14T15:16:08.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.233+0000 7f65f77fe700 15 client.46919.objecter _session_command_op_remove 5 12 2024-04-14T15:16:08.250 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:16:08.250 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:07 smithi092 ceph-mon[33966]: 4.1f scrub starts 2024-04-14T15:16:08.250 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:07 smithi092 ceph-mon[33966]: 4.1f scrub ok 2024-04-14T15:16:08.250 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:07 smithi092 ceph-mon[33966]: 5.e scrub starts 2024-04-14T15:16:08.250 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:07 smithi092 ceph-mon[33966]: 5.e scrub ok 2024-04-14T15:16:08.250 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:07 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1167609710' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-14T15:16:08.251 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:07 smithi092 ceph-mon[33966]: 110.17 scrub starts 2024-04-14T15:16:08.251 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:07 smithi092 ceph-mon[33966]: 110.17 scrub ok 2024-04-14T15:16:08.251 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:07 smithi092 ceph-mon[33966]: osdmap e300: 8 total, 8 up, 8 in 2024-04-14T15:16:08.251 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:07 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3028951022' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi090-93363-52"}]: dispatch 2024-04-14T15:16:08.251 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:07 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:16:08.251 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:07 smithi092 ceph-mon[33966]: from='client.48793 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi090-93363-52"}]: dispatch 2024-04-14T15:16:08.251 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:07 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:16:08.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.233+0000 7f660cce8700 10 client.46919.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:16:08.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.233+0000 7f660cce8700 15 client.46919.objecter _session_command_op_assign -1 13 2024-04-14T15:16:08.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.233+0000 7f660cce8700 20 client.46919.objecter _get_session s=0x7f65e80134f0 osd=6 3 2024-04-14T15:16:08.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.233+0000 7f660cce8700 20 client.46919.objecter put_session s=0x7f65e80134f0 osd=6 3 2024-04-14T15:16:08.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.233+0000 7f660cce8700 20 client.46919.objecter _get_session s=0x7f65e80134f0 osd=6 3 2024-04-14T15:16:08.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.233+0000 7f660cce8700 15 client.46919.objecter _session_command_op_remove -1 13 2024-04-14T15:16:08.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.233+0000 7f660cce8700 20 client.46919.objecter get_session s=0x7f65e80134f0 osd=6 3 2024-04-14T15:16:08.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.233+0000 7f660cce8700 15 client.46919.objecter _session_command_op_assign 6 13 2024-04-14T15:16:08.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.233+0000 7f660cce8700 20 client.46919.objecter put_session s=0x7f65e80134f0 osd=6 4 2024-04-14T15:16:08.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.233+0000 7f660cce8700 10 client.46919.objecter _send_command 13 2024-04-14T15:16:08.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.233+0000 7f660cce8700 1 -- 172.21.15.90:0/2464858537 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f65e80231c0 con 0x7f65e801f5a0 2024-04-14T15:16:08.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.233+0000 7f65f77fe700 10 client.46919.objecter ms_handle_connect 0x7f65e801f5a0 2024-04-14T15:16:08.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.234+0000 7f65f77fe700 1 -- 172.21.15.90:0/2464858537 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f65e80231c0 con 0x7f65e801f5a0 2024-04-14T15:16:08.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.234+0000 7f65f77fe700 10 client.46919.objecter ms_dispatch 0x7f6608001550 command_reply(tid 13: 0 ) 2024-04-14T15:16:08.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.234+0000 7f65f77fe700 10 client.46919.objecter _finish_command 13 = system:0 2024-04-14T15:16:08.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.234+0000 7f65f77fe700 20 client.46919.objecter put_session s=0x7f65e80134f0 osd=6 4 2024-04-14T15:16:08.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.234+0000 7f65f77fe700 15 client.46919.objecter _session_command_op_remove 6 13 2024-04-14T15:16:08.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.251+0000 7f660cce8700 10 client.46919.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:16:08.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.251+0000 7f660cce8700 15 client.46919.objecter _session_command_op_assign -1 14 2024-04-14T15:16:08.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.251+0000 7f660cce8700 20 client.46919.objecter _get_session s=0x7f65e80134f0 osd=6 3 2024-04-14T15:16:08.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.251+0000 7f660cce8700 20 client.46919.objecter put_session s=0x7f65e80134f0 osd=6 3 2024-04-14T15:16:08.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.251+0000 7f660cce8700 20 client.46919.objecter _get_session s=0x7f65e80134f0 osd=6 3 2024-04-14T15:16:08.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.251+0000 7f660cce8700 15 client.46919.objecter _session_command_op_remove -1 14 2024-04-14T15:16:08.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.251+0000 7f660cce8700 20 client.46919.objecter get_session s=0x7f65e80134f0 osd=6 3 2024-04-14T15:16:08.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.251+0000 7f660cce8700 15 client.46919.objecter _session_command_op_assign 6 14 2024-04-14T15:16:08.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.251+0000 7f660cce8700 20 client.46919.objecter put_session s=0x7f65e80134f0 osd=6 4 2024-04-14T15:16:08.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.251+0000 7f660cce8700 10 client.46919.objecter _send_command 14 2024-04-14T15:16:08.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.251+0000 7f660cce8700 1 -- 172.21.15.90:0/2464858537 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f65e80231c0 con 0x7f65e801f5a0 2024-04-14T15:16:08.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.252+0000 7f65f77fe700 1 -- 172.21.15.90:0/2464858537 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f65e80231c0 con 0x7f65e801f5a0 2024-04-14T15:16:08.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.252+0000 7f65f77fe700 10 client.46919.objecter ms_dispatch 0x7f6608001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:16:08.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.253+0000 7f65f77fe700 10 client.46919.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:16:08.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.253+0000 7f65f77fe700 20 client.46919.objecter put_session s=0x7f65e80134f0 osd=6 4 2024-04-14T15:16:08.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.253+0000 7f65f77fe700 15 client.46919.objecter _session_command_op_remove 6 14 2024-04-14T15:16:08.255 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:16:08.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.253+0000 7f660cce8700 10 client.46919.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:16:08.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.253+0000 7f660cce8700 15 client.46919.objecter _session_command_op_assign -1 15 2024-04-14T15:16:08.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.253+0000 7f660cce8700 20 client.46919.objecter _get_session s=0x7f65e80231c0 osd=7 3 2024-04-14T15:16:08.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.253+0000 7f660cce8700 20 client.46919.objecter put_session s=0x7f65e80231c0 osd=7 3 2024-04-14T15:16:08.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.253+0000 7f660cce8700 20 client.46919.objecter _get_session s=0x7f65e80231c0 osd=7 3 2024-04-14T15:16:08.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.253+0000 7f660cce8700 15 client.46919.objecter _session_command_op_remove -1 15 2024-04-14T15:16:08.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.253+0000 7f660cce8700 20 client.46919.objecter get_session s=0x7f65e80231c0 osd=7 3 2024-04-14T15:16:08.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.253+0000 7f660cce8700 15 client.46919.objecter _session_command_op_assign 7 15 2024-04-14T15:16:08.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.253+0000 7f660cce8700 20 client.46919.objecter put_session s=0x7f65e80231c0 osd=7 4 2024-04-14T15:16:08.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.253+0000 7f660cce8700 10 client.46919.objecter _send_command 15 2024-04-14T15:16:08.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.253+0000 7f660cce8700 1 -- 172.21.15.90:0/2464858537 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f65e8024420 con 0x7f65e80250e0 2024-04-14T15:16:08.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.253+0000 7f65f77fe700 10 client.46919.objecter ms_handle_connect 0x7f65e80250e0 2024-04-14T15:16:08.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.254+0000 7f65f77fe700 1 -- 172.21.15.90:0/2464858537 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f65e8024420 con 0x7f65e80250e0 2024-04-14T15:16:08.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.254+0000 7f65f77fe700 10 client.46919.objecter ms_dispatch 0x7f6608001550 command_reply(tid 15: 0 ) 2024-04-14T15:16:08.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.254+0000 7f65f77fe700 10 client.46919.objecter _finish_command 15 = system:0 2024-04-14T15:16:08.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.254+0000 7f65f77fe700 20 client.46919.objecter put_session s=0x7f65e80231c0 osd=7 4 2024-04-14T15:16:08.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.254+0000 7f65f77fe700 15 client.46919.objecter _session_command_op_remove 7 15 2024-04-14T15:16:08.267 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:08.270 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-14T15:16:08.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.270+0000 7f660cce8700 10 client.46919.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:16:08.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.270+0000 7f660cce8700 15 client.46919.objecter _session_command_op_assign -1 16 2024-04-14T15:16:08.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.270+0000 7f660cce8700 20 client.46919.objecter _get_session s=0x7f65e80231c0 osd=7 3 2024-04-14T15:16:08.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.270+0000 7f660cce8700 20 client.46919.objecter put_session s=0x7f65e80231c0 osd=7 3 2024-04-14T15:16:08.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.270+0000 7f660cce8700 20 client.46919.objecter _get_session s=0x7f65e80231c0 osd=7 3 2024-04-14T15:16:08.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.270+0000 7f660cce8700 15 client.46919.objecter _session_command_op_remove -1 16 2024-04-14T15:16:08.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.270+0000 7f660cce8700 20 client.46919.objecter get_session s=0x7f65e80231c0 osd=7 3 2024-04-14T15:16:08.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.270+0000 7f660cce8700 15 client.46919.objecter _session_command_op_assign 7 16 2024-04-14T15:16:08.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.270+0000 7f660cce8700 20 client.46919.objecter put_session s=0x7f65e80231c0 osd=7 4 2024-04-14T15:16:08.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.270+0000 7f660cce8700 10 client.46919.objecter _send_command 16 2024-04-14T15:16:08.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.270+0000 7f660cce8700 1 -- 172.21.15.90:0/2464858537 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f65e8024420 con 0x7f65e80250e0 2024-04-14T15:16:08.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.272+0000 7f65f77fe700 1 -- 172.21.15.90:0/2464858537 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f65e8024420 con 0x7f65e80250e0 2024-04-14T15:16:08.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.272+0000 7f65f77fe700 10 client.46919.objecter ms_dispatch 0x7f6608001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:16:08.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.272+0000 7f65f77fe700 10 client.46919.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:16:08.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.272+0000 7f65f77fe700 20 client.46919.objecter put_session s=0x7f65e80231c0 osd=7 4 2024-04-14T15:16:08.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.272+0000 7f65f77fe700 15 client.46919.objecter _session_command_op_remove 7 16 2024-04-14T15:16:08.273 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:16:08.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.272+0000 7f660cce8700 10 librados: watch_flush enter 2024-04-14T15:16:08.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.272+0000 7f660cce8700 10 librados: watch_flush exit 2024-04-14T15:16:08.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.272+0000 7f660cce8700 10 client.46919.objecter close_session for osd.0 2024-04-14T15:16:08.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.272+0000 7f660cce8700 1 -- 172.21.15.90:0/2464858537 >> v1:172.21.15.90:6801/3761806885 conn(0x7f65e8004730 legacy=0x7f65e8002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:08.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.272+0000 7f660cce8700 20 client.46919.objecter put_session s=0x7f65e80053e0 osd=0 1 2024-04-14T15:16:08.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.272+0000 7f660cce8700 10 client.46919.objecter close_session for osd.1 2024-04-14T15:16:08.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.272+0000 7f660cce8700 1 -- 172.21.15.90:0/2464858537 >> v1:172.21.15.90:6805/4170361699 conn(0x7f65e80042a0 legacy=0x7f65e800ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:08.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.272+0000 7f660cce8700 20 client.46919.objecter put_session s=0x7f65e8004130 osd=1 1 2024-04-14T15:16:08.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.272+0000 7f660cce8700 10 client.46919.objecter close_session for osd.2 2024-04-14T15:16:08.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.272+0000 7f660cce8700 1 -- 172.21.15.90:0/2464858537 >> v1:172.21.15.90:6809/1686184251 conn(0x7f65e8008990 legacy=0x7f65e800f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:08.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.272+0000 7f660cce8700 20 client.46919.objecter put_session s=0x7f65e8001a00 osd=2 1 2024-04-14T15:16:08.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.272+0000 7f660cce8700 10 client.46919.objecter close_session for osd.3 2024-04-14T15:16:08.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.272+0000 7f660cce8700 1 -- 172.21.15.90:0/2464858537 >> v1:172.21.15.90:6813/2270166774 conn(0x7f65e8011190 legacy=0x7f65e80115f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:08.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.272+0000 7f660cce8700 20 client.46919.objecter put_session s=0x7f65e8008200 osd=3 1 2024-04-14T15:16:08.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.272+0000 7f660cce8700 10 client.46919.objecter close_session for osd.4 2024-04-14T15:16:08.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.272+0000 7f660cce8700 1 -- 172.21.15.90:0/2464858537 >> v1:172.21.15.92:6800/3887549733 conn(0x7f65e800cb20 legacy=0x7f65e80190c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:08.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.272+0000 7f660cce8700 20 client.46919.objecter put_session s=0x7f65e80136f0 osd=4 1 2024-04-14T15:16:08.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.272+0000 7f660cce8700 10 client.46919.objecter close_session for osd.5 2024-04-14T15:16:08.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.272+0000 7f660cce8700 1 -- 172.21.15.90:0/2464858537 >> v1:172.21.15.92:6804/3244561440 conn(0x7f65e8012cd0 legacy=0x7f65e801d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:08.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.272+0000 7f6606d9d700 1 -- 172.21.15.90:0/2464858537 reap_dead start 2024-04-14T15:16:08.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.272+0000 7f660cce8700 20 client.46919.objecter put_session s=0x7f65e800c930 osd=5 1 2024-04-14T15:16:08.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.272+0000 7f660cce8700 10 client.46919.objecter close_session for osd.6 2024-04-14T15:16:08.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.272+0000 7f660cce8700 1 -- 172.21.15.90:0/2464858537 >> v1:172.21.15.92:6808/636125963 conn(0x7f65e801f5a0 legacy=0x7f65e8021990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:08.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.272+0000 7f6606d9d700 1 -- 172.21.15.90:0/2464858537 reap_dead start 2024-04-14T15:16:08.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.272+0000 7f660cce8700 20 client.46919.objecter put_session s=0x7f65e80134f0 osd=6 1 2024-04-14T15:16:08.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.272+0000 7f660cce8700 10 client.46919.objecter close_session for osd.7 2024-04-14T15:16:08.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.272+0000 7f660cce8700 1 -- 172.21.15.90:0/2464858537 >> v1:172.21.15.92:6812/1773706716 conn(0x7f65e80250e0 legacy=0x7f65e8025560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:08.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.272+0000 7f660cce8700 20 client.46919.objecter put_session s=0x7f65e80231c0 osd=7 1 2024-04-14T15:16:08.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.272+0000 7f660cce8700 20 client.46919.objecter shutdown clearing up homeless session... 2024-04-14T15:16:08.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.272+0000 7f660cce8700 10 client.46919.objecter successfully canceled tick 2024-04-14T15:16:08.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.272+0000 7f6606d9d700 1 -- 172.21.15.90:0/2464858537 reap_dead start 2024-04-14T15:16:08.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.272+0000 7f6606d9d700 1 -- 172.21.15.90:0/2464858537 reap_dead start 2024-04-14T15:16:08.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.272+0000 7f660cce8700 1 -- 172.21.15.90:0/2464858537 >> v1:172.21.15.90:6800/420935427 conn(0x7f65f0071350 legacy=0x7f65f0073810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:08.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.272+0000 7f660cce8700 1 -- 172.21.15.90:0/2464858537 >> v1:172.21.15.90:6790/0 conn(0x7f660810a6c0 legacy=0x7f6608101770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:08.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.272+0000 7f660cce8700 1 -- 172.21.15.90:0/2464858537 shutdown_connections 2024-04-14T15:16:08.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.272+0000 7f660cce8700 1 -- 172.21.15.90:0/2464858537 >> 172.21.15.90:0/2464858537 conn(0x7f6608076a70 msgr2=0x7f66081106c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:16:08.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.273+0000 7f660cce8700 1 -- 172.21.15.90:0/2464858537 shutdown_connections 2024-04-14T15:16:08.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.273+0000 7f660cce8700 1 -- 172.21.15.90:0/2464858537 wait complete. 2024-04-14T15:16:08.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:08.273+0000 7f660cce8700 1 librados: shutdown 2024-04-14T15:16:08.277 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:16:08.277 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:16:08.277 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:16:08.277 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:16:08.277 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:16:08.277 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:16:08.277 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:16:08.277 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:16:08.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:08.291 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:08.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:08.393 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:08.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:08.427 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:08.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:08.528 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:16:08.533 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:16:08.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:08.553 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:08.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:16:08.654 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:16:08.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:08.678 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:08.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:16:08.714 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-14T15:16:08.779 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:08.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:08.805 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:08.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:08.906 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:16:08.931 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:08.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:08.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:16:09.033 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:09.057 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:09.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:09.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:09.156 INFO:tasks.thrashosds.thrasher:j is {'epoch': 299, 'fsid': '0e288098-fa70-11ee-bc8e-c7b262605968', 'created': '2024-04-14T15:04:05.808313+0000', 'modified': '2024-04-14T15:16:06.858863+0000', 'last_up_change': '2024-04-14T15:09:29.769352+0000', 'last_in_change': '2024-04-14T15:09:11.803164+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 86, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 269, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-14T15:07:23.959628+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '186', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '184', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.909999966621399, 'score_stable': 1.909999966621399, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 2.180000066757202, 'raw_score_stable': 2.180000066757202, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-04-14T15:09:41.608420+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5399999618530273, 'score_stable': 1.5399999618530273, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 3, 'pool_name': 'datapool', 'create_time': '2024-04-14T15:09:41.744391+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 69, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.3299999237060547, 'score_stable': 2.3299999237060547, 'optimal_score': 1, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-04-14T15:09:43.088157+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9700000286102295, 'score_stable': 1.9700000286102295, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-04-14T15:09:45.042722+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3200000524520874, 'score_stable': 1.3200000524520874, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-04-14T15:09:47.111336+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 11, 'pool_name': 'ceph_test_rados_api_asio', 'create_time': '2024-04-14T15:12:54.225671+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 110, 'pool_name': 'test-rados-api-smithi090-93652-6', 'create_time': '2024-04-14T15:13:07.960794+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '298', 'last_force_op_resend': '297', 'last_force_op_resend_prenautilus': '297', 'last_force_op_resend_preluminous': '297', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 14, 'snap_epoch': 296, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5399999618530273, 'score_stable': 1.5399999618530273, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 269, 'pool_name': 'RoundTripSparseReadPP_smithi090-93363-52', 'create_time': '2024-04-14T15:16:04.882616+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 3, 'size': 3, 'min_size': 2, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '298', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'testprofile-RoundTripSparseReadPP_smithi090-93363-52', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}], 'osds': [{'osd': 0, 'uuid': '76772cfc-89d1-4013-b617-32284ed8dfd9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 298, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6801', 'nonce': 3761806885}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6802', 'nonce': 3761806885}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6804', 'nonce': 3761806885}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6803', 'nonce': 3761806885}]}, 'public_addr': '172.21.15.90:6801/3761806885', 'cluster_addr': '172.21.15.90:6802/3761806885', 'heartbeat_back_addr': '172.21.15.90:6804/3761806885', 'heartbeat_front_addr': '172.21.15.90:6803/3761806885', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'de4a7fa2-7d57-4ff3-a4fa-28c0cde8ef6c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 298, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6805', 'nonce': 4170361699}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6806', 'nonce': 4170361699}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6808', 'nonce': 4170361699}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6807', 'nonce': 4170361699}]}, 'public_addr': '172.21.15.90:6805/4170361699', 'cluster_addr': '172.21.15.90:6806/4170361699', 'heartbeat_back_addr': '172.21.15.90:6808/4170361699', 'heartbeat_front_addr': '172.21.15.90:6807/4170361699', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '9c39b5ff-6175-46ba-9d71-370581c6ff3d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 298, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6809', 'nonce': 1686184251}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6810', 'nonce': 1686184251}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6812', 'nonce': 1686184251}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6811', 'nonce': 1686184251}]}, 'public_addr': '172.21.15.90:6809/1686184251', 'cluster_addr': '172.21.15.90:6810/1686184251', 'heartbeat_back_addr': '172.21.15.90:6812/1686184251', 'heartbeat_front_addr': '172.21.15.90:6811/1686184251', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'efd2ca45-5b80-4013-a571-499e2cb5b2e8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 293, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6813', 'nonce': 2270166774}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6814', 'nonce': 2270166774}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6816', 'nonce': 2270166774}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6815', 'nonce': 2270166774}]}, 'public_addr': '172.21.15.90:6813/2270166774', 'cluster_addr': '172.21.15.90:6814/2270166774', 'heartbeat_back_addr': '172.21.15.90:6816/2270166774', 'heartbeat_front_addr': '172.21.15.90:6815/2270166774', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'e3d9013d-5eb6-433c-96b3-3fc38ba7ac8d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 298, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6800', 'nonce': 3887549733}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6801', 'nonce': 3887549733}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6803', 'nonce': 3887549733}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6802', 'nonce': 3887549733}]}, 'public_addr': '172.21.15.92:6800/3887549733', 'cluster_addr': '172.21.15.92:6801/3887549733', 'heartbeat_back_addr': '172.21.15.92:6803/3887549733', 'heartbeat_front_addr': '172.21.15.92:6802/3887549733', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '1ec41805-6f82-4139-81bc-75305ba9e958', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 41, 'up_thru': 271, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6804', 'nonce': 3244561440}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6805', 'nonce': 3244561440}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6807', 'nonce': 3244561440}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6806', 'nonce': 3244561440}]}, 'public_addr': '172.21.15.92:6804/3244561440', 'cluster_addr': '172.21.15.92:6805/3244561440', 'heartbeat_back_addr': '172.21.15.92:6807/3244561440', 'heartbeat_front_addr': '172.21.15.92:6806/3244561440', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'ca977ab2-bf8d-4b3a-ae5e-0efa5bc7a3ad', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 49, 'up_thru': 298, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6808', 'nonce': 636125963}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6809', 'nonce': 636125963}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6811', 'nonce': 636125963}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6810', 'nonce': 636125963}]}, 'public_addr': '172.21.15.92:6808/636125963', 'cluster_addr': '172.21.15.92:6809/636125963', 'heartbeat_back_addr': '172.21.15.92:6811/636125963', 'heartbeat_front_addr': '172.21.15.92:6810/636125963', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'e5354645-79ec-4d4c-b6e5-f9b0653915ee', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 58, 'up_thru': 298, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6812', 'nonce': 1773706716}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6813', 'nonce': 1773706716}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6815', 'nonce': 1773706716}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6814', 'nonce': 1773706716}]}, 'public_addr': '172.21.15.92:6812/1773706716', 'cluster_addr': '172.21.15.92:6813/1773706716', 'heartbeat_back_addr': '172.21.15.92:6815/1773706716', 'heartbeat_front_addr': '172.21.15.92:6814/1773706716', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-14T15:15:39.318766+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-14T15:15:29.773262+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-14T15:15:38.442760+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-14T15:15:32.198726+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-14T15:15:45.008171+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-14T15:15:12.105372+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-14T15:15:24.397005+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-14T15:15:45.036409+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.2', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1f', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.90:0/508768479': '2024-04-15T15:10:22.918674+0000', '172.21.15.90:0/1823072952': '2024-04-15T15:10:22.918674+0000', '172.21.15.90:6800/491071118': '2024-04-15T15:10:22.918674+0000', '172.21.15.90:0/1337064035': '2024-04-15T15:10:22.918674+0000', '172.21.15.90:0/769462306': '2024-04-15T15:10:22.918674+0000', '172.21.15.90:0/1139868610': '2024-04-15T15:05:05.882361+0000', '172.21.15.90:0/93763408': '2024-04-15T15:10:22.918674+0000', '172.21.15.90:0/977818204': '2024-04-15T15:04:39.251711+0000', '172.21.15.90:6800/941091171': '2024-04-15T15:05:05.882361+0000', '172.21.15.90:0/3671453561': '2024-04-15T15:05:05.882361+0000', '172.21.15.90:0/2611856200': '2024-04-15T15:05:05.882361+0000', '172.21.15.90:6800/2306258962': '2024-04-15T15:04:39.251711+0000', '172.21.15.90:0/4051692380': '2024-04-15T15:04:39.251711+0000', '172.21.15.90:0/2681502561': '2024-04-15T15:04:39.251711+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}, 'testprofile-RoundTripSparseReadPP_smithi090-93363-52': {'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-num-failure-domains': '0', 'crush-osds-per-failure-domain': '0', 'crush-root': 'default', 'jerasure-per-chunk-alignment': 'false', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van', 'w': '8'}}, 'removed_snaps_queue': [{'pool': 17, 'snaps': [{'begin': 2, 'length': 5}]}, {'pool': 18, 'snaps': [{'begin': 2, 'length': 3}]}, {'pool': 39, 'snaps': [{'begin': 1, 'length': 1}]}, {'pool': 72, 'snaps': [{'begin': 1, 'length': 1}]}, {'pool': 99, 'snaps': [{'begin': 4, 'length': 2}, {'begin': 7, 'length': 2}]}, {'pool': 109, 'snaps': [{'begin': 1, 'length': 1}]}, {'pool': 110, 'snaps': [{'begin': 14, 'length': 1}]}, {'pool': 128, 'snaps': [{'begin': 1, 'length': 2}]}, {'pool': 142, 'snaps': [{'begin': 3, 'length': 1}]}, {'pool': 149, 'snaps': [{'begin': 1, 'length': 4}]}, {'pool': 156, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 157, 'snaps': [{'begin': 2, 'length': 2}]}, {'pool': 167, 'snaps': [{'begin': 2, 'length': 3}]}, {'pool': 172, 'snaps': [{'begin': 3, 'length': 4}]}, {'pool': 175, 'snaps': [{'begin': 2, 'length': 4}]}, {'pool': 176, 'snaps': [{'begin': 2, 'length': 2}]}, {'pool': 187, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 192, 'snaps': [{'begin': 1, 'length': 4}]}, {'pool': 205, 'snaps': [{'begin': 2, 'length': 6}]}, {'pool': 213, 'snaps': [{'begin': 7, 'length': 1}]}, {'pool': 234, 'snaps': [{'begin': 8, 'length': 1}]}, {'pool': 247, 'snaps': [{'begin': 10, 'length': 1}]}, {'pool': 252, 'snaps': [{'begin': 11, 'length': 2}]}, {'pool': 265, 'snaps': [{'begin': 14, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-14T15:16:09.156 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph pg dump --format=json 2024-04-14T15:16:09.159 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:16:09.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:09.183 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:09.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:16:09.184 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:08 smithi090 ceph-mon[41670]: pgmap v395: 214 pgs: 8 unknown, 1 active+clean+snaptrim, 205 active+clean; 4.4 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-14T15:16:09.184 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:08 smithi090 ceph-mon[41670]: 1.8 scrub starts 2024-04-14T15:16:09.184 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:08 smithi090 ceph-mon[41670]: 1.8 scrub ok 2024-04-14T15:16:09.184 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:08 smithi090 ceph-mon[41670]: 6.1f scrub starts 2024-04-14T15:16:09.184 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:08 smithi090 ceph-mon[41670]: 6.1f scrub ok 2024-04-14T15:16:09.184 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:08 smithi090 ceph-mon[41670]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:16:09.184 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:08 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2464858537' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:16:09.184 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:08 smithi090 ceph-mon[41670]: 110.1f scrub starts 2024-04-14T15:16:09.185 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:08 smithi090 ceph-mon[41670]: 110.1f scrub ok 2024-04-14T15:16:09.185 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:08 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:16:09.185 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:08 smithi090 ceph-mon[41670]: from='client.48793 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi090-93363-52"}]': finished 2024-04-14T15:16:09.185 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:08 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:16:09.185 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:08 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3028951022' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi090-93363-52"}]: dispatch 2024-04-14T15:16:09.185 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:08 smithi090 ceph-mon[41670]: osdmap e301: 8 total, 8 up, 8 in 2024-04-14T15:16:09.185 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:08 smithi090 ceph-mon[41670]: from='client.48793 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi090-93363-52"}]: dispatch 2024-04-14T15:16:09.185 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:08 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:16:09.185 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:08 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:16:09.185 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:08 smithi090 ceph-mon[30399]: pgmap v395: 214 pgs: 8 unknown, 1 active+clean+snaptrim, 205 active+clean; 4.4 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-14T15:16:09.185 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:08 smithi090 ceph-mon[30399]: 1.8 scrub starts 2024-04-14T15:16:09.185 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:08 smithi090 ceph-mon[30399]: 1.8 scrub ok 2024-04-14T15:16:09.186 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:08 smithi090 ceph-mon[30399]: 6.1f scrub starts 2024-04-14T15:16:09.186 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:08 smithi090 ceph-mon[30399]: 6.1f scrub ok 2024-04-14T15:16:09.186 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:08 smithi090 ceph-mon[30399]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:16:09.186 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:08 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2464858537' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:16:09.186 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:08 smithi090 ceph-mon[30399]: 110.1f scrub starts 2024-04-14T15:16:09.186 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:08 smithi090 ceph-mon[30399]: 110.1f scrub ok 2024-04-14T15:16:09.186 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:08 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:16:09.186 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:08 smithi090 ceph-mon[30399]: from='client.48793 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi090-93363-52"}]': finished 2024-04-14T15:16:09.186 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:08 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:16:09.186 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:08 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3028951022' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi090-93363-52"}]: dispatch 2024-04-14T15:16:09.186 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:08 smithi090 ceph-mon[30399]: osdmap e301: 8 total, 8 up, 8 in 2024-04-14T15:16:09.186 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:08 smithi090 ceph-mon[30399]: from='client.48793 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi090-93363-52"}]: dispatch 2024-04-14T15:16:09.186 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:08 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:16:09.186 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:08 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:16:09.284 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:09.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:09.308 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:09.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:08 smithi092 ceph-mon[33966]: pgmap v395: 214 pgs: 8 unknown, 1 active+clean+snaptrim, 205 active+clean; 4.4 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-14T15:16:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:08 smithi092 ceph-mon[33966]: 1.8 scrub starts 2024-04-14T15:16:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:08 smithi092 ceph-mon[33966]: 1.8 scrub ok 2024-04-14T15:16:09.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:08 smithi092 ceph-mon[33966]: 6.1f scrub starts 2024-04-14T15:16:09.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:08 smithi092 ceph-mon[33966]: 6.1f scrub ok 2024-04-14T15:16:09.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:08 smithi092 ceph-mon[33966]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:16:09.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:08 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2464858537' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:16:09.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:08 smithi092 ceph-mon[33966]: 110.1f scrub starts 2024-04-14T15:16:09.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:08 smithi092 ceph-mon[33966]: 110.1f scrub ok 2024-04-14T15:16:09.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:08 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:16:09.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:08 smithi092 ceph-mon[33966]: from='client.48793 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi090-93363-52"}]': finished 2024-04-14T15:16:09.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:08 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:16:09.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:08 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3028951022' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi090-93363-52"}]: dispatch 2024-04-14T15:16:09.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:08 smithi092 ceph-mon[33966]: osdmap e301: 8 total, 8 up, 8 in 2024-04-14T15:16:09.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:08 smithi092 ceph-mon[33966]: from='client.48793 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi090-93363-52"}]: dispatch 2024-04-14T15:16:09.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:08 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:16:09.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:08 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:16:09.409 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:16:09.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:09.433 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:09.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:16:09.534 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:09.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:09.559 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:09.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:09.660 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:16:09.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:09.686 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:09.686 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:16:09.742 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:16:09.787 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:09.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:09.811 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:09.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:09.912 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:16:09.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:09.938 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:09.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:16:09.975 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:09 smithi092 ceph-mon[33966]: 6.8 scrub starts 2024-04-14T15:16:09.975 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:09 smithi092 ceph-mon[33966]: 6.8 scrub ok 2024-04-14T15:16:09.975 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:09 smithi092 ceph-mon[33966]: 4.13 scrub starts 2024-04-14T15:16:09.975 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:09 smithi092 ceph-mon[33966]: 4.13 scrub ok 2024-04-14T15:16:09.975 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:09 smithi092 ceph-mon[33966]: 110.9 scrub starts 2024-04-14T15:16:09.975 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:09 smithi092 ceph-mon[33966]: 110.9 scrub ok 2024-04-14T15:16:09.975 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:09 smithi092 ceph-mon[33966]: 11.10 scrub starts 2024-04-14T15:16:09.975 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:09 smithi092 ceph-mon[33966]: 11.10 scrub ok 2024-04-14T15:16:09.976 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:09 smithi092 ceph-mon[33966]: from='client.48793 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi090-93363-52"}]': finished 2024-04-14T15:16:09.976 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:09 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-37", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:16:09.976 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:09 smithi092 ceph-mon[33966]: osdmap e302: 8 total, 8 up, 8 in 2024-04-14T15:16:09.976 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:09 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-37"}]: dispatch 2024-04-14T15:16:09.976 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:09 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-37"}]: dispatch 2024-04-14T15:16:09.976 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:09 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/442343645' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi090-93363-53"}]: dispatch 2024-04-14T15:16:09.976 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:09 smithi092 ceph-mon[33966]: from='client.48811 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi090-93363-53"}]: dispatch 2024-04-14T15:16:09.976 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:09 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/442343645' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi090-93363-53"}]: dispatch 2024-04-14T15:16:09.976 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:09 smithi092 ceph-mon[33966]: from='client.48811 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi090-93363-53"}]: dispatch 2024-04-14T15:16:09.976 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:09 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/442343645' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi090-93363-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:16:09.976 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:09 smithi092 ceph-mon[33966]: from='client.48811 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi090-93363-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:16:10.039 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:10.062 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:10.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:10.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:10.148 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:09 smithi090 ceph-mon[41670]: 6.8 scrub starts 2024-04-14T15:16:10.148 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:09 smithi090 ceph-mon[41670]: 6.8 scrub ok 2024-04-14T15:16:10.148 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:09 smithi090 ceph-mon[41670]: 4.13 scrub starts 2024-04-14T15:16:10.148 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:09 smithi090 ceph-mon[41670]: 4.13 scrub ok 2024-04-14T15:16:10.148 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:09 smithi090 ceph-mon[41670]: 110.9 scrub starts 2024-04-14T15:16:10.148 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:09 smithi090 ceph-mon[41670]: 110.9 scrub ok 2024-04-14T15:16:10.148 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:09 smithi090 ceph-mon[41670]: 11.10 scrub starts 2024-04-14T15:16:10.148 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:09 smithi090 ceph-mon[41670]: 11.10 scrub ok 2024-04-14T15:16:10.148 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:09 smithi090 ceph-mon[41670]: from='client.48793 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi090-93363-52"}]': finished 2024-04-14T15:16:10.149 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:09 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-37", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:16:10.149 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:09 smithi090 ceph-mon[41670]: osdmap e302: 8 total, 8 up, 8 in 2024-04-14T15:16:10.149 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:09 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-37"}]: dispatch 2024-04-14T15:16:10.149 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:09 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-37"}]: dispatch 2024-04-14T15:16:10.149 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:09 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/442343645' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi090-93363-53"}]: dispatch 2024-04-14T15:16:10.149 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:09 smithi090 ceph-mon[41670]: from='client.48811 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi090-93363-53"}]: dispatch 2024-04-14T15:16:10.149 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:09 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/442343645' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi090-93363-53"}]: dispatch 2024-04-14T15:16:10.149 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:09 smithi090 ceph-mon[41670]: from='client.48811 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi090-93363-53"}]: dispatch 2024-04-14T15:16:10.149 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:09 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/442343645' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi090-93363-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:16:10.149 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:09 smithi090 ceph-mon[41670]: from='client.48811 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi090-93363-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:16:10.149 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:09 smithi090 ceph-mon[30399]: 6.8 scrub starts 2024-04-14T15:16:10.149 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:09 smithi090 ceph-mon[30399]: 6.8 scrub ok 2024-04-14T15:16:10.150 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:09 smithi090 ceph-mon[30399]: 4.13 scrub starts 2024-04-14T15:16:10.150 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:09 smithi090 ceph-mon[30399]: 4.13 scrub ok 2024-04-14T15:16:10.150 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:09 smithi090 ceph-mon[30399]: 110.9 scrub starts 2024-04-14T15:16:10.150 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:09 smithi090 ceph-mon[30399]: 110.9 scrub ok 2024-04-14T15:16:10.150 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:09 smithi090 ceph-mon[30399]: 11.10 scrub starts 2024-04-14T15:16:10.150 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:09 smithi090 ceph-mon[30399]: 11.10 scrub ok 2024-04-14T15:16:10.150 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:09 smithi090 ceph-mon[30399]: from='client.48793 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi090-93363-52"}]': finished 2024-04-14T15:16:10.150 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:09 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-37", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:16:10.150 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:09 smithi090 ceph-mon[30399]: osdmap e302: 8 total, 8 up, 8 in 2024-04-14T15:16:10.150 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:09 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-37"}]: dispatch 2024-04-14T15:16:10.150 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:09 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-37"}]: dispatch 2024-04-14T15:16:10.150 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:09 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/442343645' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi090-93363-53"}]: dispatch 2024-04-14T15:16:10.150 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:09 smithi090 ceph-mon[30399]: from='client.48811 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi090-93363-53"}]: dispatch 2024-04-14T15:16:10.150 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:09 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/442343645' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi090-93363-53"}]: dispatch 2024-04-14T15:16:10.151 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:09 smithi090 ceph-mon[30399]: from='client.48811 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi090-93363-53"}]: dispatch 2024-04-14T15:16:10.151 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:09 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/442343645' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi090-93363-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:16:10.151 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:09 smithi090 ceph-mon[30399]: from='client.48811 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi090-93363-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:16:10.164 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:10.187 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:10.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:10.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:10.204 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-04-14T15:16:10.204 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:16:10.290 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:16:10.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:10.315 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:10.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:16:10.416 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:10.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:10.441 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:10.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:10.460 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:16:10.542 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:16:10.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:10.568 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:10.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:16:10.669 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:16:10.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:10.694 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:10.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:16:10.796 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:16:10.821 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-14T15:16:10.822 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:10.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:10.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:16:10.924 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:16:10.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:10.950 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:10.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:16:11.051 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:16:11.077 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:11.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:11.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:16:11.109 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:16:11.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:10 smithi092 ceph-mon[33966]: pgmap v398: 238 pgs: 10 creating+activating, 3 unknown, 225 active+clean; 4.4 MiB data, 570 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-04-14T15:16:11.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:10 smithi092 ceph-mon[33966]: 2.1a scrub starts 2024-04-14T15:16:11.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:10 smithi092 ceph-mon[33966]: 2.1a scrub ok 2024-04-14T15:16:11.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:10 smithi092 ceph-mon[33966]: 4.18 scrub starts 2024-04-14T15:16:11.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:10 smithi092 ceph-mon[33966]: 4.18 scrub ok 2024-04-14T15:16:11.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:10 smithi092 ceph-mon[33966]: 110.1e scrub starts 2024-04-14T15:16:11.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:10 smithi092 ceph-mon[33966]: 110.1e scrub ok 2024-04-14T15:16:11.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:10 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-37"}]': finished 2024-04-14T15:16:11.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:10 smithi092 ceph-mon[33966]: from='client.48811 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi090-93363-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:16:11.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:10 smithi092 ceph-mon[33966]: osdmap e303: 8 total, 8 up, 8 in 2024-04-14T15:16:11.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:10 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/442343645' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi090-93363-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi090-93363-53"}]: dispatch 2024-04-14T15:16:11.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:10 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-37", "mode": "writeback"}]: dispatch 2024-04-14T15:16:11.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:10 smithi092 ceph-mon[33966]: from='client.48811 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi090-93363-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi090-93363-53"}]: dispatch 2024-04-14T15:16:11.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:10 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-37", "mode": "writeback"}]: dispatch 2024-04-14T15:16:11.139 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:10 smithi090 ceph-mon[30399]: pgmap v398: 238 pgs: 10 creating+activating, 3 unknown, 225 active+clean; 4.4 MiB data, 570 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-04-14T15:16:11.139 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:10 smithi090 ceph-mon[30399]: 2.1a scrub starts 2024-04-14T15:16:11.140 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:10 smithi090 ceph-mon[30399]: 2.1a scrub ok 2024-04-14T15:16:11.140 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:10 smithi090 ceph-mon[30399]: 4.18 scrub starts 2024-04-14T15:16:11.140 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:10 smithi090 ceph-mon[30399]: 4.18 scrub ok 2024-04-14T15:16:11.140 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:10 smithi090 ceph-mon[30399]: 110.1e scrub starts 2024-04-14T15:16:11.140 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:10 smithi090 ceph-mon[30399]: 110.1e scrub ok 2024-04-14T15:16:11.140 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:10 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-37"}]': finished 2024-04-14T15:16:11.140 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:10 smithi090 ceph-mon[30399]: from='client.48811 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi090-93363-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:16:11.140 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:10 smithi090 ceph-mon[30399]: osdmap e303: 8 total, 8 up, 8 in 2024-04-14T15:16:11.140 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:10 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/442343645' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi090-93363-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi090-93363-53"}]: dispatch 2024-04-14T15:16:11.140 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:10 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-37", "mode": "writeback"}]: dispatch 2024-04-14T15:16:11.140 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:10 smithi090 ceph-mon[30399]: from='client.48811 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi090-93363-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi090-93363-53"}]: dispatch 2024-04-14T15:16:11.140 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:10 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-37", "mode": "writeback"}]: dispatch 2024-04-14T15:16:11.141 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:10 smithi090 ceph-mon[41670]: pgmap v398: 238 pgs: 10 creating+activating, 3 unknown, 225 active+clean; 4.4 MiB data, 570 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-04-14T15:16:11.141 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:10 smithi090 ceph-mon[41670]: 2.1a scrub starts 2024-04-14T15:16:11.141 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:10 smithi090 ceph-mon[41670]: 2.1a scrub ok 2024-04-14T15:16:11.141 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:10 smithi090 ceph-mon[41670]: 4.18 scrub starts 2024-04-14T15:16:11.141 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:10 smithi090 ceph-mon[41670]: 4.18 scrub ok 2024-04-14T15:16:11.141 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:10 smithi090 ceph-mon[41670]: 110.1e scrub starts 2024-04-14T15:16:11.141 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:10 smithi090 ceph-mon[41670]: 110.1e scrub ok 2024-04-14T15:16:11.141 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:10 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-37"}]': finished 2024-04-14T15:16:11.141 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:10 smithi090 ceph-mon[41670]: from='client.48811 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi090-93363-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:16:11.141 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:10 smithi090 ceph-mon[41670]: osdmap e303: 8 total, 8 up, 8 in 2024-04-14T15:16:11.141 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:10 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/442343645' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi090-93363-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi090-93363-53"}]: dispatch 2024-04-14T15:16:11.141 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:10 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-37", "mode": "writeback"}]: dispatch 2024-04-14T15:16:11.141 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:10 smithi090 ceph-mon[41670]: from='client.48811 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi090-93363-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi090-93363-53"}]: dispatch 2024-04-14T15:16:11.141 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:10 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-37", "mode": "writeback"}]: dispatch 2024-04-14T15:16:11.179 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:16:11.203 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:11.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:11.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:16:11.305 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:16:11.328 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:11.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:11.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:16:11.430 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:16:11.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:11.454 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:11.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:16:11.555 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:16:11.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:11.580 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:11.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:16:11.681 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:11.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:11.705 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:11.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:11.730 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-14T15:16:11.806 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:16:11.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:11.830 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:11.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:16:11.931 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:11.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:11.957 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:11.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:11.985 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:11 smithi090 ceph-mon[41670]: 2.3 scrub starts 2024-04-14T15:16:11.985 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:11 smithi090 ceph-mon[41670]: 2.3 scrub ok 2024-04-14T15:16:11.985 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:11 smithi090 ceph-mon[41670]: 2.15 scrub starts 2024-04-14T15:16:11.985 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:11 smithi090 ceph-mon[41670]: 2.15 scrub ok 2024-04-14T15:16:11.985 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:11 smithi090 ceph-mon[41670]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:16:11.985 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:11 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-37", "mode": "writeback"}]': finished 2024-04-14T15:16:11.985 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:11 smithi090 ceph-mon[41670]: osdmap e304: 8 total, 8 up, 8 in 2024-04-14T15:16:11.985 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:11 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:16:11.985 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:11 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:16:11.986 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:11 smithi090 ceph-mon[30399]: 2.3 scrub starts 2024-04-14T15:16:11.986 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:11 smithi090 ceph-mon[30399]: 2.3 scrub ok 2024-04-14T15:16:11.986 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:11 smithi090 ceph-mon[30399]: 2.15 scrub starts 2024-04-14T15:16:11.986 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:11 smithi090 ceph-mon[30399]: 2.15 scrub ok 2024-04-14T15:16:11.986 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:11 smithi090 ceph-mon[30399]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:16:11.986 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:11 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-37", "mode": "writeback"}]': finished 2024-04-14T15:16:11.986 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:11 smithi090 ceph-mon[30399]: osdmap e304: 8 total, 8 up, 8 in 2024-04-14T15:16:11.986 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:11 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:16:11.986 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:11 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:16:12.058 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:16:12.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:12.082 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:12.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:16:12.183 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:16:12.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:12.207 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:12.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:16:12.308 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:16:12.316 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:11 smithi092 ceph-mon[33966]: 2.3 scrub starts 2024-04-14T15:16:12.316 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:11 smithi092 ceph-mon[33966]: 2.3 scrub ok 2024-04-14T15:16:12.316 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:11 smithi092 ceph-mon[33966]: 2.15 scrub starts 2024-04-14T15:16:12.316 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:11 smithi092 ceph-mon[33966]: 2.15 scrub ok 2024-04-14T15:16:12.316 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:11 smithi092 ceph-mon[33966]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:16:12.316 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:11 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-37", "mode": "writeback"}]': finished 2024-04-14T15:16:12.316 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:11 smithi092 ceph-mon[33966]: osdmap e304: 8 total, 8 up, 8 in 2024-04-14T15:16:12.317 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:11 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:16:12.317 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:11 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:16:12.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:12.333 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:12.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:16:12.435 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:12.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:12.459 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:12.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:12.560 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:12.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:12.587 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:12.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:12.688 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:16:12.712 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:12.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:12.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:16:12.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:12.733+0000 7f272505a700 1 -- 172.21.15.90:0/2760217207 >> v1:172.21.15.90:6789/0 conn(0x7f272010b820 legacy=0x7f272010dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:12.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:12.733+0000 7f272505a700 1 -- 172.21.15.90:0/2760217207 shutdown_connections 2024-04-14T15:16:12.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:12.733+0000 7f272505a700 1 -- 172.21.15.90:0/2760217207 >> 172.21.15.90:0/2760217207 conn(0x7f2720076aa0 msgr2=0x7f2720076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:16:12.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:12.733+0000 7f272505a700 1 -- 172.21.15.90:0/2760217207 shutdown_connections 2024-04-14T15:16:12.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:12.733+0000 7f272505a700 1 -- 172.21.15.90:0/2760217207 wait complete. 2024-04-14T15:16:12.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:12.734+0000 7f272505a700 1 librados: starting msgr at 2024-04-14T15:16:12.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:12.734+0000 7f272505a700 1 librados: starting objecter 2024-04-14T15:16:12.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:12.734+0000 7f272505a700 1 Processor -- start 2024-04-14T15:16:12.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:12.734+0000 7f272505a700 1 -- start start 2024-04-14T15:16:12.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:12.734+0000 7f272505a700 1 librados: setting wanted keys 2024-04-14T15:16:12.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:12.734+0000 7f272505a700 1 librados: calling monclient init 2024-04-14T15:16:12.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:12.734+0000 7f272505a700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f27201a7a30 con 0x7f2720108a20 2024-04-14T15:16:12.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:12.734+0000 7f272505a700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f27201a8c50 con 0x7f272010b820 2024-04-14T15:16:12.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:12.734+0000 7f272505a700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f27201a9e90 con 0x7f272010f420 2024-04-14T15:16:12.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:12.734+0000 7f271ed9d700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f2720108a20 0x7f2720101850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:51962/0 (socket says 172.21.15.90:51962) 2024-04-14T15:16:12.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:12.734+0000 7f271ed9d700 1 -- 172.21.15.90:0/3103189505 learned_addr learned my addr 172.21.15.90:0/3103189505 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:16:12.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:12.735+0000 7f2707fff700 10 client.?.objecter ms_handle_connect 0x7f2720108a20 2024-04-14T15:16:12.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:12.735+0000 7f2707fff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:16:12.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:12.735+0000 7f2707fff700 10 client.?.objecter ms_handle_connect 0x7f272010f420 2024-04-14T15:16:12.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:12.735+0000 7f2707fff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:16:12.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:12.735+0000 7f2707fff700 1 -- 172.21.15.90:0/3103189505 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1235519515 0 0) 0x7f27201a9e90 con 0x7f272010f420 2024-04-14T15:16:12.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:12.735+0000 7f2707fff700 1 -- 172.21.15.90:0/3103189505 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f27080031f0 con 0x7f272010f420 2024-04-14T15:16:12.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:12.735+0000 7f2707fff700 10 client.?.objecter ms_handle_connect 0x7f272010b820 2024-04-14T15:16:12.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:12.735+0000 7f2707fff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:16:12.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:12.735+0000 7f2707fff700 1 -- 172.21.15.90:0/3103189505 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3810215139 0 0) 0x7f27201a7a30 con 0x7f2720108a20 2024-04-14T15:16:12.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:12.735+0000 7f2707fff700 1 -- 172.21.15.90:0/3103189505 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f27201a9e90 con 0x7f2720108a20 2024-04-14T15:16:12.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:12.735+0000 7f2707fff700 1 -- 172.21.15.90:0/3103189505 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 391623824 0 0) 0x7f27201a8c50 con 0x7f272010b820 2024-04-14T15:16:12.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:12.735+0000 7f2707fff700 1 -- 172.21.15.90:0/3103189505 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f27201a7a30 con 0x7f272010b820 2024-04-14T15:16:12.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:12.735+0000 7f2707fff700 1 -- 172.21.15.90:0/3103189505 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 416863153 0 0) 0x7f27080031f0 con 0x7f272010f420 2024-04-14T15:16:12.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:12.735+0000 7f2707fff700 1 -- 172.21.15.90:0/3103189505 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f27201a8c50 con 0x7f272010f420 2024-04-14T15:16:12.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:12.735+0000 7f2707fff700 1 -- 172.21.15.90:0/3103189505 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f2718001c30 con 0x7f272010f420 2024-04-14T15:16:12.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:12.735+0000 7f2707fff700 1 -- 172.21.15.90:0/3103189505 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1470324001 0 0) 0x7f27201a7a30 con 0x7f272010b820 2024-04-14T15:16:12.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:12.735+0000 7f2707fff700 1 -- 172.21.15.90:0/3103189505 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f27080031f0 con 0x7f272010b820 2024-04-14T15:16:12.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:12.735+0000 7f2707fff700 1 -- 172.21.15.90:0/3103189505 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f2714001440 con 0x7f272010b820 2024-04-14T15:16:12.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:12.735+0000 7f2707fff700 1 -- 172.21.15.90:0/3103189505 <== mon.2 v1:172.21.15.90:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3465649676 0 0) 0x7f27201a8c50 con 0x7f272010f420 2024-04-14T15:16:12.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:12.735+0000 7f2707fff700 1 -- 172.21.15.90:0/3103189505 >> v1:172.21.15.92:6789/0 conn(0x7f272010b820 legacy=0x7f27201a28b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:12.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:12.735+0000 7f2707fff700 1 -- 172.21.15.90:0/3103189505 >> v1:172.21.15.90:6789/0 conn(0x7f2720108a20 legacy=0x7f2720101850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:12.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:12.735+0000 7f2707fff700 1 -- 172.21.15.90:0/3103189505 --> v1:172.21.15.90:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27201ab0d0 con 0x7f272010f420 2024-04-14T15:16:12.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:12.736+0000 7f272505a700 1 -- 172.21.15.90:0/3103189505 --> v1:172.21.15.90:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f27201aa0c0 con 0x7f272010f420 2024-04-14T15:16:12.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:12.736+0000 7f272505a700 10 client.46931.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:16:12.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:12.736+0000 7f272505a700 1 -- 172.21.15.90:0/3103189505 --> v1:172.21.15.90:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f27201aa580 con 0x7f272010f420 2024-04-14T15:16:12.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:12.736+0000 7f272505a700 1 librados: init done 2024-04-14T15:16:12.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:12.736+0000 7f2707fff700 1 -- 172.21.15.90:0/3103189505 <== mon.2 v1:172.21.15.90:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f2718003ec0 con 0x7f272010f420 2024-04-14T15:16:12.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:12.736+0000 7f2707fff700 1 -- 172.21.15.90:0/3103189505 <== mon.2 v1:172.21.15.90:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f2718004e40 con 0x7f272010f420 2024-04-14T15:16:12.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:12.737+0000 7f272505a700 1 -- 172.21.15.90:0/3103189505 --> v1:172.21.15.90:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f270c005150 con 0x7f272010f420 2024-04-14T15:16:12.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:12.738+0000 7f2707fff700 1 -- 172.21.15.90:0/3103189505 <== mon.2 v1:172.21.15.90:6790/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f271801b9f0 con 0x7f272010f420 2024-04-14T15:16:12.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:12.738+0000 7f2707fff700 1 -- 172.21.15.90:0/3103189505 <== mon.2 v1:172.21.15.90:6790/0 8 ==== osd_map(304..304 src has 51..304) ==== 8538+0+0 (unknown 1093769472 0 0) 0x7f2718089dc0 con 0x7f272010f420 2024-04-14T15:16:12.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:12.738+0000 7f2707fff700 10 client.46931.objecter ms_dispatch 0x7f2720001550 osd_map(304..304 src has 51..304) 2024-04-14T15:16:12.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:12.738+0000 7f2707fff700 3 client.46931.objecter handle_osd_map got epochs [304,304] > 0 2024-04-14T15:16:12.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:12.738+0000 7f2707fff700 3 client.46931.objecter handle_osd_map decoding full epoch 304 2024-04-14T15:16:12.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:12.738+0000 7f2707fff700 20 client.46931.objecter dump_active .. 0 homeless 2024-04-14T15:16:12.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:12.739+0000 7f2707fff700 10 client.46931.objecter ms_handle_connect 0x7f2708071140 2024-04-14T15:16:12.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:12.743+0000 7f2707fff700 1 -- 172.21.15.90:0/3103189505 <== mon.2 v1:172.21.15.90:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f2718056670 con 0x7f272010f420 2024-04-14T15:16:12.814 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:16:12.830 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-04-14T15:16:12.830 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:16:12.838 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:12.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:12.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:16:12.940 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:16:12.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:12.962+0000 7f272505a700 1 -- 172.21.15.90:0/3103189505 --> v1:172.21.15.90:6790/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7f270c005440 con 0x7f272010f420 2024-04-14T15:16:12.963 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:12.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:12.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:16:12.984 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:12 smithi090 ceph-mon[41670]: pgmap v401: 238 pgs: 10 creating+activating, 228 active+clean; 4.6 MiB data, 570 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:16:12.985 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:12 smithi090 ceph-mon[41670]: 4.1d scrub starts 2024-04-14T15:16:12.985 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:12 smithi090 ceph-mon[41670]: 4.1d scrub ok 2024-04-14T15:16:12.985 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:12 smithi090 ceph-mon[41670]: 2.f scrub starts 2024-04-14T15:16:12.985 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:12 smithi090 ceph-mon[41670]: 2.f scrub ok 2024-04-14T15:16:12.985 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:12 smithi090 ceph-mon[41670]: from='client.48811 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi090-93363-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi090-93363-53"}]': finished 2024-04-14T15:16:12.985 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:12 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:16:12.985 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:12 smithi090 ceph-mon[41670]: osdmap e305: 8 total, 8 up, 8 in 2024-04-14T15:16:12.985 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:12 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-37"}]: dispatch 2024-04-14T15:16:12.985 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:12 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-37"}]: dispatch 2024-04-14T15:16:12.985 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:12 smithi090 ceph-mon[30399]: pgmap v401: 238 pgs: 10 creating+activating, 228 active+clean; 4.6 MiB data, 570 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:16:12.985 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:12 smithi090 ceph-mon[30399]: 4.1d scrub starts 2024-04-14T15:16:12.986 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:12 smithi090 ceph-mon[30399]: 4.1d scrub ok 2024-04-14T15:16:12.986 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:12 smithi090 ceph-mon[30399]: 2.f scrub starts 2024-04-14T15:16:12.986 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:12 smithi090 ceph-mon[30399]: 2.f scrub ok 2024-04-14T15:16:12.986 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:12 smithi090 ceph-mon[30399]: from='client.48811 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi090-93363-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi090-93363-53"}]': finished 2024-04-14T15:16:12.986 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:12 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:16:12.986 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:12 smithi090 ceph-mon[30399]: osdmap e305: 8 total, 8 up, 8 in 2024-04-14T15:16:12.986 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:12 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-37"}]: dispatch 2024-04-14T15:16:12.986 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:12 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-37"}]: dispatch 2024-04-14T15:16:13.066 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:16:13.090 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:13.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:13.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:16:13.192 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:16:13.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.192+0000 7f3f72abe700 1 -- 172.21.15.90:0/3525806861 >> v1:172.21.15.90:6789/0 conn(0x7f3f6c10aaa0 legacy=0x7f3f6c10ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:13.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.192+0000 7f3f72abe700 1 -- 172.21.15.90:0/3525806861 shutdown_connections 2024-04-14T15:16:13.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.192+0000 7f3f72abe700 1 -- 172.21.15.90:0/3525806861 >> 172.21.15.90:0/3525806861 conn(0x7f3f6c100550 msgr2=0x7f3f6c102970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:16:13.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.192+0000 7f3f72abe700 1 -- 172.21.15.90:0/3525806861 shutdown_connections 2024-04-14T15:16:13.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.192+0000 7f3f72abe700 1 -- 172.21.15.90:0/3525806861 wait complete. 2024-04-14T15:16:13.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.193+0000 7f3f72abe700 1 librados: starting msgr at 2024-04-14T15:16:13.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.193+0000 7f3f72abe700 1 librados: starting objecter 2024-04-14T15:16:13.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.193+0000 7f3f72abe700 1 Processor -- start 2024-04-14T15:16:13.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.193+0000 7f3f72abe700 1 -- start start 2024-04-14T15:16:13.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.193+0000 7f3f72abe700 1 librados: setting wanted keys 2024-04-14T15:16:13.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.193+0000 7f3f72abe700 1 librados: calling monclient init 2024-04-14T15:16:13.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.193+0000 7f3f72abe700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3f6c1a9b30 con 0x7f3f6c111520 2024-04-14T15:16:13.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.193+0000 7f3f72abe700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3f6c1aad50 con 0x7f3f6c10d950 2024-04-14T15:16:13.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.193+0000 7f3f72abe700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3f6c1abf90 con 0x7f3f6c10aaa0 2024-04-14T15:16:13.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.193+0000 7f3f7085a700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f3f6c10aaa0 0x7f3f6c110cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:54686/0 (socket says 172.21.15.90:54686) 2024-04-14T15:16:13.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.193+0000 7f3f7105b700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f3f6c111520 0x7f3f6c1a81f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:51980/0 (socket says 172.21.15.90:51980) 2024-04-14T15:16:13.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.193+0000 7f3f7085a700 1 -- 172.21.15.90:0/3537572389 learned_addr learned my addr 172.21.15.90:0/3537572389 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:16:13.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.193+0000 7f3f69ffb700 10 client.?.objecter ms_handle_connect 0x7f3f6c111520 2024-04-14T15:16:13.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.193+0000 7f3f69ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:16:13.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.193+0000 7f3f69ffb700 10 client.?.objecter ms_handle_connect 0x7f3f6c10d950 2024-04-14T15:16:13.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.193+0000 7f3f69ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:16:13.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.194+0000 7f3f69ffb700 10 client.?.objecter ms_handle_connect 0x7f3f6c10aaa0 2024-04-14T15:16:13.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.194+0000 7f3f69ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:16:13.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.194+0000 7f3f69ffb700 1 -- 172.21.15.90:0/3537572389 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4122997162 0 0) 0x7f3f6c1aad50 con 0x7f3f6c10d950 2024-04-14T15:16:13.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.194+0000 7f3f69ffb700 1 -- 172.21.15.90:0/3537572389 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f5c0035b0 con 0x7f3f6c10d950 2024-04-14T15:16:13.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.194+0000 7f3f69ffb700 1 -- 172.21.15.90:0/3537572389 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2610054273 0 0) 0x7f3f6c1a9b30 con 0x7f3f6c111520 2024-04-14T15:16:13.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.194+0000 7f3f69ffb700 1 -- 172.21.15.90:0/3537572389 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f6c1aad50 con 0x7f3f6c111520 2024-04-14T15:16:13.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.194+0000 7f3f69ffb700 1 -- 172.21.15.90:0/3537572389 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3656155421 0 0) 0x7f3f6c1abf90 con 0x7f3f6c10aaa0 2024-04-14T15:16:13.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.194+0000 7f3f69ffb700 1 -- 172.21.15.90:0/3537572389 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f6c1a9b30 con 0x7f3f6c10aaa0 2024-04-14T15:16:13.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.194+0000 7f3f69ffb700 1 -- 172.21.15.90:0/3537572389 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3061558215 0 0) 0x7f3f5c0035b0 con 0x7f3f6c10d950 2024-04-14T15:16:13.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.194+0000 7f3f69ffb700 1 -- 172.21.15.90:0/3537572389 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3f6c1abf90 con 0x7f3f6c10d950 2024-04-14T15:16:13.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.194+0000 7f3f69ffb700 1 -- 172.21.15.90:0/3537572389 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f3f580018d0 con 0x7f3f6c10d950 2024-04-14T15:16:13.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.194+0000 7f3f69ffb700 1 -- 172.21.15.90:0/3537572389 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1251544978 0 0) 0x7f3f6c1aad50 con 0x7f3f6c111520 2024-04-14T15:16:13.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.194+0000 7f3f69ffb700 1 -- 172.21.15.90:0/3537572389 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3f5c0035b0 con 0x7f3f6c111520 2024-04-14T15:16:13.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.194+0000 7f3f69ffb700 1 -- 172.21.15.90:0/3537572389 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f3f64001c10 con 0x7f3f6c111520 2024-04-14T15:16:13.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.194+0000 7f3f69ffb700 1 -- 172.21.15.90:0/3537572389 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1595770039 0 0) 0x7f3f6c1abf90 con 0x7f3f6c10d950 2024-04-14T15:16:13.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.194+0000 7f3f69ffb700 1 -- 172.21.15.90:0/3537572389 >> v1:172.21.15.90:6790/0 conn(0x7f3f6c10aaa0 legacy=0x7f3f6c110cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:13.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.194+0000 7f3f69ffb700 1 -- 172.21.15.90:0/3537572389 >> v1:172.21.15.90:6789/0 conn(0x7f3f6c111520 legacy=0x7f3f6c1a81f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:13.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.194+0000 7f3f69ffb700 1 -- 172.21.15.90:0/3537572389 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f6c1ad1d0 con 0x7f3f6c10d950 2024-04-14T15:16:13.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.194+0000 7f3f72abe700 1 -- 172.21.15.90:0/3537572389 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3f6c1aaf80 con 0x7f3f6c10d950 2024-04-14T15:16:13.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.194+0000 7f3f72abe700 10 client.48823.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:16:13.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.194+0000 7f3f72abe700 1 -- 172.21.15.90:0/3537572389 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f3f6c1ab440 con 0x7f3f6c10d950 2024-04-14T15:16:13.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.194+0000 7f3f72abe700 1 librados: init done 2024-04-14T15:16:13.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.194+0000 7f3f69ffb700 1 -- 172.21.15.90:0/3537572389 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f3f58003b40 con 0x7f3f6c10d950 2024-04-14T15:16:13.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.194+0000 7f3f69ffb700 1 -- 172.21.15.90:0/3537572389 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f3f58004a70 con 0x7f3f6c10d950 2024-04-14T15:16:13.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.196+0000 7f3f72abe700 1 -- 172.21.15.90:0/3537572389 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f50005120 con 0x7f3f6c10d950 2024-04-14T15:16:13.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.196+0000 7f3f69ffb700 1 -- 172.21.15.90:0/3537572389 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f3f58004dd0 con 0x7f3f6c10d950 2024-04-14T15:16:13.198 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:16:13.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.196+0000 7f3f69ffb700 1 -- 172.21.15.90:0/3537572389 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(305..305 src has 51..305) ==== 8961+0+0 (unknown 2017160465 0 0) 0x7f3f58089cb0 con 0x7f3f6c10d950 2024-04-14T15:16:13.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.196+0000 7f3f69ffb700 10 client.48823.objecter ms_dispatch 0x7f3f6c001550 osd_map(305..305 src has 51..305) 2024-04-14T15:16:13.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.196+0000 7f3f69ffb700 3 client.48823.objecter handle_osd_map got epochs [305,305] > 0 2024-04-14T15:16:13.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.196+0000 7f3f69ffb700 3 client.48823.objecter handle_osd_map decoding full epoch 305 2024-04-14T15:16:13.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.197+0000 7f3f69ffb700 20 client.48823.objecter dump_active .. 0 homeless 2024-04-14T15:16:13.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.197+0000 7f3f69ffb700 10 client.48823.objecter ms_handle_connect 0x7f3f5c071050 2024-04-14T15:16:13.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.201+0000 7f3f69ffb700 1 -- 172.21.15.90:0/3537572389 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f3f580563a0 con 0x7f3f6c10d950 2024-04-14T15:16:13.217 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:13.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:13.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:16:13.320 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:16:13.345 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:13.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:13.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:16:13.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:12 smithi092 ceph-mon[33966]: pgmap v401: 238 pgs: 10 creating+activating, 228 active+clean; 4.6 MiB data, 570 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:16:13.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:12 smithi092 ceph-mon[33966]: 4.1d scrub starts 2024-04-14T15:16:13.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:12 smithi092 ceph-mon[33966]: 4.1d scrub ok 2024-04-14T15:16:13.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:12 smithi092 ceph-mon[33966]: 2.f scrub starts 2024-04-14T15:16:13.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:12 smithi092 ceph-mon[33966]: 2.f scrub ok 2024-04-14T15:16:13.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:12 smithi092 ceph-mon[33966]: from='client.48811 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi090-93363-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi090-93363-53"}]': finished 2024-04-14T15:16:13.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:12 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:16:13.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:12 smithi092 ceph-mon[33966]: osdmap e305: 8 total, 8 up, 8 in 2024-04-14T15:16:13.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:12 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-37"}]: dispatch 2024-04-14T15:16:13.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:12 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-37"}]: dispatch 2024-04-14T15:16:13.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.408+0000 7f3f72abe700 1 -- 172.21.15.90:0/3537572389 --> v1:172.21.15.90:6800/420935427 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3f50000b90 con 0x7f3f5c071050 2024-04-14T15:16:13.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.424+0000 7f3f69ffb700 1 -- 172.21.15.90:0/3537572389 <== mgr.14610 v1:172.21.15.90:6800/420935427 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+630631 (unknown 2965378022 0 3637962564) 0x7f3f50000b90 con 0x7f3f5c071050 2024-04-14T15:16:13.434 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:16:13.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.429+0000 7f3f72abe700 10 librados: watch_flush enter 2024-04-14T15:16:13.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.429+0000 7f3f72abe700 10 librados: watch_flush exit 2024-04-14T15:16:13.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.429+0000 7f3f72abe700 20 client.48823.objecter shutdown clearing up homeless session... 2024-04-14T15:16:13.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.429+0000 7f3f72abe700 10 client.48823.objecter successfully canceled tick 2024-04-14T15:16:13.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.429+0000 7f3f72abe700 1 -- 172.21.15.90:0/3537572389 >> v1:172.21.15.90:6800/420935427 conn(0x7f3f5c071050 legacy=0x7f3f5c073510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:13.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.429+0000 7f3f72abe700 1 -- 172.21.15.90:0/3537572389 >> v1:172.21.15.92:6789/0 conn(0x7f3f6c10d950 legacy=0x7f3f6c1a49d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:13.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.429+0000 7f3f72abe700 1 -- 172.21.15.90:0/3537572389 shutdown_connections 2024-04-14T15:16:13.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.429+0000 7f3f72abe700 1 -- 172.21.15.90:0/3537572389 >> 172.21.15.90:0/3537572389 conn(0x7f3f6c100550 msgr2=0x7f3f6c104aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:16:13.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.429+0000 7f3f72abe700 1 -- 172.21.15.90:0/3537572389 shutdown_connections 2024-04-14T15:16:13.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.429+0000 7f3f72abe700 1 -- 172.21.15.90:0/3537572389 wait complete. 2024-04-14T15:16:13.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.429+0000 7f3f72abe700 1 librados: shutdown 2024-04-14T15:16:13.445 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-14T15:16:13.448 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:16:13.449 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-14T15:16:13.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:13.474 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:13.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:16:13.575 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:16:13.599 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:13.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:13.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:16:13.701 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:16:13.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:13.725 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:13.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:16:13.729 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:16:13.826 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:16:13.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:13.851 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:13.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:16:13.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.875+0000 7f2707fff700 1 -- 172.21.15.90:0/3103189505 <== mon.2 v1:172.21.15.90:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v306) ==== 101+0+0 (unknown 1624624873 0 0) 0x7f271805a320 con 0x7f272010f420 2024-04-14T15:16:13.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.878+0000 7f272505a700 10 librados: watch_flush enter 2024-04-14T15:16:13.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.878+0000 7f272505a700 10 librados: watch_flush exit 2024-04-14T15:16:13.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.878+0000 7f272505a700 20 client.46931.objecter shutdown clearing up homeless session... 2024-04-14T15:16:13.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.878+0000 7f272505a700 10 client.46931.objecter successfully canceled tick 2024-04-14T15:16:13.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.878+0000 7f272505a700 1 -- 172.21.15.90:0/3103189505 >> v1:172.21.15.90:6800/420935427 conn(0x7f2708071140 legacy=0x7f2708073600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:13.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.878+0000 7f272505a700 1 -- 172.21.15.90:0/3103189505 >> v1:172.21.15.90:6790/0 conn(0x7f272010f420 legacy=0x7f27201a60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:13.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.878+0000 7f272505a700 1 -- 172.21.15.90:0/3103189505 shutdown_connections 2024-04-14T15:16:13.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.878+0000 7f272505a700 1 -- 172.21.15.90:0/3103189505 >> 172.21.15.90:0/3103189505 conn(0x7f2720076aa0 msgr2=0x7f2720112b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:16:13.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.879+0000 7f272505a700 1 -- 172.21.15.90:0/3103189505 shutdown_connections 2024-04-14T15:16:13.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.879+0000 7f272505a700 1 -- 172.21.15.90:0/3103189505 wait complete. 2024-04-14T15:16:13.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:13.880+0000 7f272505a700 1 librados: shutdown 2024-04-14T15:16:13.884 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is unset 2024-04-14T15:16:13.952 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:16:13.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:13.977 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:13.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:16:14.078 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:14.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:14.103 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:14.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:14.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:13 smithi092 ceph-mon[33966]: 5.1f scrub starts 2024-04-14T15:16:14.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:13 smithi092 ceph-mon[33966]: 5.1f scrub ok 2024-04-14T15:16:14.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:13 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3103189505' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:16:14.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:13 smithi092 ceph-mon[33966]: from='client.46931 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:16:14.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:13 smithi092 ceph-mon[33966]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:16:14.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:13 smithi092 ceph-mon[33966]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-14T15:16:14.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:13 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-37"}]': finished 2024-04-14T15:16:14.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:13 smithi092 ceph-mon[33966]: from='client.46931 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-14T15:16:14.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:13 smithi092 ceph-mon[33966]: osdmap e306: 8 total, 8 up, 8 in 2024-04-14T15:16:14.204 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:16:14.228 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:14.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:14.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:16:14.285 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:16:13 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:16:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:16:14.285 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:13 smithi090 ceph-mon[41670]: 5.1f scrub starts 2024-04-14T15:16:14.285 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:13 smithi090 ceph-mon[41670]: 5.1f scrub ok 2024-04-14T15:16:14.285 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:13 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3103189505' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:16:14.285 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:13 smithi090 ceph-mon[41670]: from='client.46931 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:16:14.285 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:13 smithi090 ceph-mon[41670]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:16:14.285 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:13 smithi090 ceph-mon[41670]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-14T15:16:14.285 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:13 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-37"}]': finished 2024-04-14T15:16:14.286 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:13 smithi090 ceph-mon[41670]: from='client.46931 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-14T15:16:14.286 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:13 smithi090 ceph-mon[41670]: osdmap e306: 8 total, 8 up, 8 in 2024-04-14T15:16:14.286 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:13 smithi090 ceph-mon[30399]: 5.1f scrub starts 2024-04-14T15:16:14.286 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:13 smithi090 ceph-mon[30399]: 5.1f scrub ok 2024-04-14T15:16:14.286 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:13 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3103189505' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:16:14.286 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:13 smithi090 ceph-mon[30399]: from='client.46931 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:16:14.286 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:13 smithi090 ceph-mon[30399]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:16:14.286 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:13 smithi090 ceph-mon[30399]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-14T15:16:14.286 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:13 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-37"}]': finished 2024-04-14T15:16:14.286 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:13 smithi090 ceph-mon[30399]: from='client.46931 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-14T15:16:14.286 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:13 smithi090 ceph-mon[30399]: osdmap e306: 8 total, 8 up, 8 in 2024-04-14T15:16:14.330 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:14.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:14.355 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:14.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:14.456 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:16:14.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:14.481 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:14.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:16:14.582 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:16:14.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:14.609 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:14.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:16:14.710 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:14.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:14.735 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:14.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:14.836 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:16:14.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:14.860 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:14.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:16:14.961 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:16:14.971 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:14 smithi092 ceph-mon[33966]: pgmap v404: 246 pgs: 8 unknown, 10 creating+activating, 228 active+clean; 4.6 MiB data, 570 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:16:14.971 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:14 smithi092 ceph-mon[33966]: from='client.48823 v1:172.21.15.90:0/3537572389' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:16:14.971 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:14 smithi092 ceph-mon[33966]: 4.19 deep-scrub starts 2024-04-14T15:16:14.971 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:14 smithi092 ceph-mon[33966]: 4.19 deep-scrub ok 2024-04-14T15:16:14.971 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:14 smithi092 ceph-mon[33966]: 5.11 scrub starts 2024-04-14T15:16:14.971 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:14 smithi092 ceph-mon[33966]: 5.11 scrub ok 2024-04-14T15:16:14.972 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:14 smithi092 ceph-mon[33966]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:16:14.972 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:14 smithi092 ceph-mon[33966]: 6.2 scrub starts 2024-04-14T15:16:14.972 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:14 smithi092 ceph-mon[33966]: 6.2 scrub ok 2024-04-14T15:16:14.972 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:14 smithi092 ceph-mon[33966]: 2.1f scrub starts 2024-04-14T15:16:14.972 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:14 smithi092 ceph-mon[33966]: 2.1f scrub ok 2024-04-14T15:16:14.972 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:14 smithi092 ceph-mon[33966]: osdmap e307: 8 total, 8 up, 8 in 2024-04-14T15:16:14.972 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:14 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/442343645' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi090-93363-53"}]: dispatch 2024-04-14T15:16:14.972 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:14 smithi092 ceph-mon[33966]: from='client.48811 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi090-93363-53"}]: dispatch 2024-04-14T15:16:14.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:14.986 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:14.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:16:15.087 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:16:15.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:15.112 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:15.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:16:15.149 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:14 smithi090 ceph-mon[41670]: pgmap v404: 246 pgs: 8 unknown, 10 creating+activating, 228 active+clean; 4.6 MiB data, 570 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:16:15.150 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:14 smithi090 ceph-mon[41670]: from='client.48823 v1:172.21.15.90:0/3537572389' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:16:15.150 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:14 smithi090 ceph-mon[41670]: 4.19 deep-scrub starts 2024-04-14T15:16:15.150 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:14 smithi090 ceph-mon[41670]: 4.19 deep-scrub ok 2024-04-14T15:16:15.150 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:14 smithi090 ceph-mon[41670]: 5.11 scrub starts 2024-04-14T15:16:15.150 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:14 smithi090 ceph-mon[41670]: 5.11 scrub ok 2024-04-14T15:16:15.150 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:14 smithi090 ceph-mon[41670]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:16:15.150 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:14 smithi090 ceph-mon[41670]: 6.2 scrub starts 2024-04-14T15:16:15.150 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:14 smithi090 ceph-mon[41670]: 6.2 scrub ok 2024-04-14T15:16:15.150 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:14 smithi090 ceph-mon[41670]: 2.1f scrub starts 2024-04-14T15:16:15.150 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:14 smithi090 ceph-mon[41670]: 2.1f scrub ok 2024-04-14T15:16:15.150 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:14 smithi090 ceph-mon[41670]: osdmap e307: 8 total, 8 up, 8 in 2024-04-14T15:16:15.150 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:14 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/442343645' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi090-93363-53"}]: dispatch 2024-04-14T15:16:15.151 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:14 smithi090 ceph-mon[41670]: from='client.48811 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi090-93363-53"}]: dispatch 2024-04-14T15:16:15.151 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:14 smithi090 ceph-mon[30399]: pgmap v404: 246 pgs: 8 unknown, 10 creating+activating, 228 active+clean; 4.6 MiB data, 570 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:16:15.151 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:14 smithi090 ceph-mon[30399]: from='client.48823 v1:172.21.15.90:0/3537572389' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:16:15.151 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:14 smithi090 ceph-mon[30399]: 4.19 deep-scrub starts 2024-04-14T15:16:15.151 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:14 smithi090 ceph-mon[30399]: 4.19 deep-scrub ok 2024-04-14T15:16:15.151 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:14 smithi090 ceph-mon[30399]: 5.11 scrub starts 2024-04-14T15:16:15.151 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:14 smithi090 ceph-mon[30399]: 5.11 scrub ok 2024-04-14T15:16:15.151 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:14 smithi090 ceph-mon[30399]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:16:15.151 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:14 smithi090 ceph-mon[30399]: 6.2 scrub starts 2024-04-14T15:16:15.151 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:14 smithi090 ceph-mon[30399]: 6.2 scrub ok 2024-04-14T15:16:15.151 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:14 smithi090 ceph-mon[30399]: 2.1f scrub starts 2024-04-14T15:16:15.151 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:14 smithi090 ceph-mon[30399]: 2.1f scrub ok 2024-04-14T15:16:15.152 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:14 smithi090 ceph-mon[30399]: osdmap e307: 8 total, 8 up, 8 in 2024-04-14T15:16:15.152 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:14 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/442343645' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi090-93363-53"}]: dispatch 2024-04-14T15:16:15.152 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:14 smithi090 ceph-mon[30399]: from='client.48811 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi090-93363-53"}]: dispatch 2024-04-14T15:16:15.212 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:16:15.237 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:15.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:15.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:16:15.283 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:16:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:16:15.339 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:16:15.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:15.364 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:15.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:16:15.465 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:16:15.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:15.490 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:15.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:16:15.528 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-04-14T15:16:15.528 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:16:15.592 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:16:15.615 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:15.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:15.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:16:15.717 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:16:15.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:15.742 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:15.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:16:15.843 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:16:15.870 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:15.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:15.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:16:15.972 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:16:15.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:15.996 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:15.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:16:16.076 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":404,"stamp":"2024-04-14T15:16:13.042058+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4791905,"num_objects":321,"num_object_clones":2,"num_object_copies":963,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":321,"num_whiteouts":0,"num_read":1203,"num_read_kb":861,"num_write":988,"num_write_kb":8782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3778,"ondisk_log_size":3778,"up":714,"acting":714,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":677,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":583372,"kb_used_data":56128,"kb_used_omap":12,"kb_used_meta":526963,"kb_avail":749377844,"statfs":{"total":767960285184,"available":767362912256,"internally_reserved":0,"allocated":57475072,"data_stored":37888527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":539610674},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":209018,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":-19,"num_read_kb":-28,"num_write":-16,"num_write_kb":-219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":-161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001366"},"pg_stats":[{"pgid":"270.1f","version":"0'0","reported_seq":10,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:09.078400+0000","last_change":"2024-04-14T15:16:09.078126+0000","last_active":"2024-04-14T15:16:09.078400+0000","last_peered":"2024-04-14T15:16:09.078400+0000","last_clean":"2024-04-14T15:16:09.078400+0000","last_became_active":"2024-04-14T15:16:09.077851+0000","last_became_peered":"2024-04-14T15:16:09.077851+0000","last_unstale":"2024-04-14T15:16:09.078400+0000","last_undegraded":"2024-04-14T15:16:09.078400+0000","last_fullsized":"2024-04-14T15:16:09.078400+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:07.860882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:07.860882+0000","last_clean_scrub_stamp":"2024-04-14T15:16:07.860882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:11.500481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"270.1e","version":"0'0","reported_seq":4,"reported_epoch":301,"state":"creating+activating","last_fresh":"2024-04-14T15:16:08.889088+0000","last_change":"2024-04-14T15:16:08.884137+0000","last_active":"2024-04-14T15:16:07.860882+0000","last_peered":"2024-04-14T15:16:07.860882+0000","last_clean":"2024-04-14T15:16:07.860882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:16:08.889088+0000","last_undegraded":"2024-04-14T15:16:08.889088+0000","last_fullsized":"2024-04-14T15:16:08.889088+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:07.860882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:07.860882+0000","last_clean_scrub_stamp":"2024-04-14T15:16:07.860882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":["7","0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"270.1d","version":"0'0","reported_seq":10,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.880552+0000","last_change":"2024-04-14T15:16:08.879834+0000","last_active":"2024-04-14T15:16:08.880552+0000","last_peered":"2024-04-14T15:16:08.880552+0000","last_clean":"2024-04-14T15:16:08.880552+0000","last_became_active":"2024-04-14T15:16:08.879551+0000","last_became_peered":"2024-04-14T15:16:08.879551+0000","last_unstale":"2024-04-14T15:16:08.880552+0000","last_undegraded":"2024-04-14T15:16:08.880552+0000","last_fullsized":"2024-04-14T15:16:08.880552+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:07.860882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:07.860882+0000","last_clean_scrub_stamp":"2024-04-14T15:16:07.860882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:09.927308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"270.1c","version":"0'0","reported_seq":4,"reported_epoch":301,"state":"creating+activating","last_fresh":"2024-04-14T15:16:08.886971+0000","last_change":"2024-04-14T15:16:08.880285+0000","last_active":"2024-04-14T15:16:07.860882+0000","last_peered":"2024-04-14T15:16:07.860882+0000","last_clean":"2024-04-14T15:16:07.860882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:16:08.886971+0000","last_undegraded":"2024-04-14T15:16:08.886971+0000","last_fullsized":"2024-04-14T15:16:08.886971+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:07.860882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:07.860882+0000","last_clean_scrub_stamp":"2024-04-14T15:16:07.860882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":["7","2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"270.1b","version":"0'0","reported_seq":4,"reported_epoch":301,"state":"creating+activating","last_fresh":"2024-04-14T15:16:08.879045+0000","last_change":"2024-04-14T15:16:08.874818+0000","last_active":"2024-04-14T15:16:07.860882+0000","last_peered":"2024-04-14T15:16:07.860882+0000","last_clean":"2024-04-14T15:16:07.860882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:16:08.879045+0000","last_undegraded":"2024-04-14T15:16:08.879045+0000","last_fullsized":"2024-04-14T15:16:08.879045+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:07.860882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:07.860882+0000","last_clean_scrub_stamp":"2024-04-14T15:16:07.860882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":["6","4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"270.1a","version":"0'0","reported_seq":4,"reported_epoch":301,"state":"creating+activating","last_fresh":"2024-04-14T15:16:08.879910+0000","last_change":"2024-04-14T15:16:08.872473+0000","last_active":"2024-04-14T15:16:07.860882+0000","last_peered":"2024-04-14T15:16:07.860882+0000","last_clean":"2024-04-14T15:16:07.860882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:16:08.879910+0000","last_undegraded":"2024-04-14T15:16:08.879910+0000","last_fullsized":"2024-04-14T15:16:08.879910+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:07.860882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:07.860882+0000","last_clean_scrub_stamp":"2024-04-14T15:16:07.860882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":["1","2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"270.19","version":"0'0","reported_seq":10,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.880907+0000","last_change":"2024-04-14T15:16:08.880504+0000","last_active":"2024-04-14T15:16:08.880907+0000","last_peered":"2024-04-14T15:16:08.880907+0000","last_clean":"2024-04-14T15:16:08.880907+0000","last_became_active":"2024-04-14T15:16:08.880219+0000","last_became_peered":"2024-04-14T15:16:08.880219+0000","last_unstale":"2024-04-14T15:16:08.880907+0000","last_undegraded":"2024-04-14T15:16:08.880907+0000","last_fullsized":"2024-04-14T15:16:08.880907+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:07.860882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:07.860882+0000","last_clean_scrub_stamp":"2024-04-14T15:16:07.860882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:17.363735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"270.18","version":"0'0","reported_seq":10,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.882011+0000","last_change":"2024-04-14T15:16:08.881568+0000","last_active":"2024-04-14T15:16:08.882011+0000","last_peered":"2024-04-14T15:16:08.882011+0000","last_clean":"2024-04-14T15:16:08.882011+0000","last_became_active":"2024-04-14T15:16:08.881077+0000","last_became_peered":"2024-04-14T15:16:08.881077+0000","last_unstale":"2024-04-14T15:16:08.882011+0000","last_undegraded":"2024-04-14T15:16:08.882011+0000","last_fullsized":"2024-04-14T15:16:08.882011+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:07.860882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:07.860882+0000","last_clean_scrub_stamp":"2024-04-14T15:16:07.860882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:26.927332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"270.17","version":"0'0","reported_seq":10,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.883077+0000","last_change":"2024-04-14T15:16:08.882747+0000","last_active":"2024-04-14T15:16:08.883077+0000","last_peered":"2024-04-14T15:16:08.883077+0000","last_clean":"2024-04-14T15:16:08.883077+0000","last_became_active":"2024-04-14T15:16:08.882406+0000","last_became_peered":"2024-04-14T15:16:08.882406+0000","last_unstale":"2024-04-14T15:16:08.883077+0000","last_undegraded":"2024-04-14T15:16:08.883077+0000","last_fullsized":"2024-04-14T15:16:08.883077+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:07.860882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:07.860882+0000","last_clean_scrub_stamp":"2024-04-14T15:16:07.860882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:32.404445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"270.16","version":"0'0","reported_seq":10,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.887233+0000","last_change":"2024-04-14T15:16:08.885277+0000","last_active":"2024-04-14T15:16:08.887233+0000","last_peered":"2024-04-14T15:16:08.887233+0000","last_clean":"2024-04-14T15:16:08.887233+0000","last_became_active":"2024-04-14T15:16:08.884406+0000","last_became_peered":"2024-04-14T15:16:08.884406+0000","last_unstale":"2024-04-14T15:16:08.887233+0000","last_undegraded":"2024-04-14T15:16:08.887233+0000","last_fullsized":"2024-04-14T15:16:08.887233+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:07.860882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:07.860882+0000","last_clean_scrub_stamp":"2024-04-14T15:16:07.860882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:16.957257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"270.15","version":"0'0","reported_seq":10,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.883312+0000","last_change":"2024-04-14T15:16:08.882794+0000","last_active":"2024-04-14T15:16:08.883312+0000","last_peered":"2024-04-14T15:16:08.883312+0000","last_clean":"2024-04-14T15:16:08.883312+0000","last_became_active":"2024-04-14T15:16:08.882475+0000","last_became_peered":"2024-04-14T15:16:08.882475+0000","last_unstale":"2024-04-14T15:16:08.883312+0000","last_undegraded":"2024-04-14T15:16:08.883312+0000","last_fullsized":"2024-04-14T15:16:08.883312+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:07.860882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:07.860882+0000","last_clean_scrub_stamp":"2024-04-14T15:16:07.860882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:12.827225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"270.14","version":"0'0","reported_seq":10,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.891362+0000","last_change":"2024-04-14T15:16:08.890981+0000","last_active":"2024-04-14T15:16:08.891362+0000","last_peered":"2024-04-14T15:16:08.891362+0000","last_clean":"2024-04-14T15:16:08.891362+0000","last_became_active":"2024-04-14T15:16:08.890772+0000","last_became_peered":"2024-04-14T15:16:08.890772+0000","last_unstale":"2024-04-14T15:16:08.891362+0000","last_undegraded":"2024-04-14T15:16:08.891362+0000","last_fullsized":"2024-04-14T15:16:08.891362+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:07.860882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:07.860882+0000","last_clean_scrub_stamp":"2024-04-14T15:16:07.860882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:19.378366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"270.13","version":"0'0","reported_seq":10,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.891214+0000","last_change":"2024-04-14T15:16:08.890511+0000","last_active":"2024-04-14T15:16:08.891214+0000","last_peered":"2024-04-14T15:16:08.891214+0000","last_clean":"2024-04-14T15:16:08.891214+0000","last_became_active":"2024-04-14T15:16:08.890215+0000","last_became_peered":"2024-04-14T15:16:08.890215+0000","last_unstale":"2024-04-14T15:16:08.891214+0000","last_undegraded":"2024-04-14T15:16:08.891214+0000","last_fullsized":"2024-04-14T15:16:08.891214+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:07.860882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:07.860882+0000","last_clean_scrub_stamp":"2024-04-14T15:16:07.860882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:16.175907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"270.12","version":"0'0","reported_seq":10,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.884233+0000","last_change":"2024-04-14T15:16:08.883874+0000","last_active":"2024-04-14T15:16:08.884233+0000","last_peered":"2024-04-14T15:16:08.884233+0000","last_clean":"2024-04-14T15:16:08.884233+0000","last_became_active":"2024-04-14T15:16:08.883217+0000","last_became_peered":"2024-04-14T15:16:08.883217+0000","last_unstale":"2024-04-14T15:16:08.884233+0000","last_undegraded":"2024-04-14T15:16:08.884233+0000","last_fullsized":"2024-04-14T15:16:08.884233+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:07.860882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:07.860882+0000","last_clean_scrub_stamp":"2024-04-14T15:16:07.860882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:13.606150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"270.11","version":"0'0","reported_seq":10,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.889294+0000","last_change":"2024-04-14T15:16:08.888730+0000","last_active":"2024-04-14T15:16:08.889294+0000","last_peered":"2024-04-14T15:16:08.889294+0000","last_clean":"2024-04-14T15:16:08.889294+0000","last_became_active":"2024-04-14T15:16:08.888402+0000","last_became_peered":"2024-04-14T15:16:08.888402+0000","last_unstale":"2024-04-14T15:16:08.889294+0000","last_undegraded":"2024-04-14T15:16:08.889294+0000","last_fullsized":"2024-04-14T15:16:08.889294+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:07.860882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:07.860882+0000","last_clean_scrub_stamp":"2024-04-14T15:16:07.860882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:30.927046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"270.10","version":"0'0","reported_seq":10,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.889201+0000","last_change":"2024-04-14T15:16:08.888187+0000","last_active":"2024-04-14T15:16:08.889201+0000","last_peered":"2024-04-14T15:16:08.889201+0000","last_clean":"2024-04-14T15:16:08.889201+0000","last_became_active":"2024-04-14T15:16:08.887883+0000","last_became_peered":"2024-04-14T15:16:08.887883+0000","last_unstale":"2024-04-14T15:16:08.889201+0000","last_undegraded":"2024-04-14T15:16:08.889201+0000","last_fullsized":"2024-04-14T15:16:08.889201+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:07.860882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:07.860882+0000","last_clean_scrub_stamp":"2024-04-14T15:16:07.860882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:33.811604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"270.f","version":"0'0","reported_seq":10,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.886112+0000","last_change":"2024-04-14T15:16:08.884434+0000","last_active":"2024-04-14T15:16:08.886112+0000","last_peered":"2024-04-14T15:16:08.886112+0000","last_clean":"2024-04-14T15:16:08.886112+0000","last_became_active":"2024-04-14T15:16:08.884160+0000","last_became_peered":"2024-04-14T15:16:08.884160+0000","last_unstale":"2024-04-14T15:16:08.886112+0000","last_undegraded":"2024-04-14T15:16:08.886112+0000","last_fullsized":"2024-04-14T15:16:08.886112+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:07.860882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:07.860882+0000","last_clean_scrub_stamp":"2024-04-14T15:16:07.860882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:33.931366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"270.e","version":"0'0","reported_seq":4,"reported_epoch":301,"state":"creating+activating","last_fresh":"2024-04-14T15:16:08.887673+0000","last_change":"2024-04-14T15:16:08.883431+0000","last_active":"2024-04-14T15:16:07.860882+0000","last_peered":"2024-04-14T15:16:07.860882+0000","last_clean":"2024-04-14T15:16:07.860882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:16:08.887673+0000","last_undegraded":"2024-04-14T15:16:08.887673+0000","last_fullsized":"2024-04-14T15:16:08.887673+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:07.860882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:07.860882+0000","last_clean_scrub_stamp":"2024-04-14T15:16:07.860882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":["0","5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"270.d","version":"0'0","reported_seq":10,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.891581+0000","last_change":"2024-04-14T15:16:08.891219+0000","last_active":"2024-04-14T15:16:08.891581+0000","last_peered":"2024-04-14T15:16:08.891581+0000","last_clean":"2024-04-14T15:16:08.891581+0000","last_became_active":"2024-04-14T15:16:08.890926+0000","last_became_peered":"2024-04-14T15:16:08.890926+0000","last_unstale":"2024-04-14T15:16:08.891581+0000","last_undegraded":"2024-04-14T15:16:08.891581+0000","last_fullsized":"2024-04-14T15:16:08.891581+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:07.860882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:07.860882+0000","last_clean_scrub_stamp":"2024-04-14T15:16:07.860882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:22.162799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"270.c","version":"0'0","reported_seq":10,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.885029+0000","last_change":"2024-04-14T15:16:08.881532+0000","last_active":"2024-04-14T15:16:08.885029+0000","last_peered":"2024-04-14T15:16:08.885029+0000","last_clean":"2024-04-14T15:16:08.885029+0000","last_became_active":"2024-04-14T15:16:08.881246+0000","last_became_peered":"2024-04-14T15:16:08.881246+0000","last_unstale":"2024-04-14T15:16:08.885029+0000","last_undegraded":"2024-04-14T15:16:08.885029+0000","last_fullsized":"2024-04-14T15:16:08.885029+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:07.860882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:07.860882+0000","last_clean_scrub_stamp":"2024-04-14T15:16:07.860882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:12.547489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"270.b","version":"0'0","reported_seq":10,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.881547+0000","last_change":"2024-04-14T15:16:08.880629+0000","last_active":"2024-04-14T15:16:08.881547+0000","last_peered":"2024-04-14T15:16:08.881547+0000","last_clean":"2024-04-14T15:16:08.881547+0000","last_became_active":"2024-04-14T15:16:08.880238+0000","last_became_peered":"2024-04-14T15:16:08.880238+0000","last_unstale":"2024-04-14T15:16:08.881547+0000","last_undegraded":"2024-04-14T15:16:08.881547+0000","last_fullsized":"2024-04-14T15:16:08.881547+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:07.860882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:07.860882+0000","last_clean_scrub_stamp":"2024-04-14T15:16:07.860882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:15.719017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"270.a","version":"0'0","reported_seq":4,"reported_epoch":301,"state":"creating+activating","last_fresh":"2024-04-14T15:16:08.879151+0000","last_change":"2024-04-14T15:16:08.868817+0000","last_active":"2024-04-14T15:16:07.860882+0000","last_peered":"2024-04-14T15:16:07.860882+0000","last_clean":"2024-04-14T15:16:07.860882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:16:08.879151+0000","last_undegraded":"2024-04-14T15:16:08.879151+0000","last_fullsized":"2024-04-14T15:16:08.879151+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:07.860882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:07.860882+0000","last_clean_scrub_stamp":"2024-04-14T15:16:07.860882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":["4","5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"270.9","version":"0'0","reported_seq":10,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.886651+0000","last_change":"2024-04-14T15:16:08.886027+0000","last_active":"2024-04-14T15:16:08.886651+0000","last_peered":"2024-04-14T15:16:08.886651+0000","last_clean":"2024-04-14T15:16:08.886651+0000","last_became_active":"2024-04-14T15:16:08.885740+0000","last_became_peered":"2024-04-14T15:16:08.885740+0000","last_unstale":"2024-04-14T15:16:08.886651+0000","last_undegraded":"2024-04-14T15:16:08.886651+0000","last_fullsized":"2024-04-14T15:16:08.886651+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:07.860882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:07.860882+0000","last_clean_scrub_stamp":"2024-04-14T15:16:07.860882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:16.669141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"270.8","version":"0'0","reported_seq":4,"reported_epoch":301,"state":"creating+activating","last_fresh":"2024-04-14T15:16:08.878600+0000","last_change":"2024-04-14T15:16:08.875565+0000","last_active":"2024-04-14T15:16:07.860882+0000","last_peered":"2024-04-14T15:16:07.860882+0000","last_clean":"2024-04-14T15:16:07.860882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:16:08.878600+0000","last_undegraded":"2024-04-14T15:16:08.878600+0000","last_fullsized":"2024-04-14T15:16:08.878600+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:07.860882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:07.860882+0000","last_clean_scrub_stamp":"2024-04-14T15:16:07.860882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":["7","5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"271.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:16:12.865685+0000","last_change":"2024-04-14T15:16:12.865685+0000","last_active":"2024-04-14T15:16:12.865685+0000","last_peered":"2024-04-14T15:16:12.865685+0000","last_clean":"2024-04-14T15:16:12.865685+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:16:12.865685+0000","last_undegraded":"2024-04-14T15:16:12.865685+0000","last_fullsized":"2024-04-14T15:16:12.865685+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:12.865685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:12.865685+0000","last_clean_scrub_stamp":"2024-04-14T15:16:12.865685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"270.7","version":"0'0","reported_seq":4,"reported_epoch":301,"state":"creating+activating","last_fresh":"2024-04-14T15:16:08.878805+0000","last_change":"2024-04-14T15:16:08.869212+0000","last_active":"2024-04-14T15:16:07.860882+0000","last_peered":"2024-04-14T15:16:07.860882+0000","last_clean":"2024-04-14T15:16:07.860882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:16:08.878805+0000","last_undegraded":"2024-04-14T15:16:08.878805+0000","last_fullsized":"2024-04-14T15:16:08.878805+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:07.860882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:07.860882+0000","last_clean_scrub_stamp":"2024-04-14T15:16:07.860882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":["4","1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"271.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:16:12.865685+0000","last_change":"2024-04-14T15:16:12.865685+0000","last_active":"2024-04-14T15:16:12.865685+0000","last_peered":"2024-04-14T15:16:12.865685+0000","last_clean":"2024-04-14T15:16:12.865685+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:16:12.865685+0000","last_undegraded":"2024-04-14T15:16:12.865685+0000","last_fullsized":"2024-04-14T15:16:12.865685+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:12.865685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:12.865685+0000","last_clean_scrub_stamp":"2024-04-14T15:16:12.865685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"270.6","version":"0'0","reported_seq":4,"reported_epoch":301,"state":"creating+activating","last_fresh":"2024-04-14T15:16:08.888509+0000","last_change":"2024-04-14T15:16:08.883852+0000","last_active":"2024-04-14T15:16:07.860882+0000","last_peered":"2024-04-14T15:16:07.860882+0000","last_clean":"2024-04-14T15:16:07.860882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:16:08.888509+0000","last_undegraded":"2024-04-14T15:16:08.888509+0000","last_fullsized":"2024-04-14T15:16:08.888509+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:07.860882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:07.860882+0000","last_clean_scrub_stamp":"2024-04-14T15:16:07.860882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":["0","5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"271.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:16:12.865685+0000","last_change":"2024-04-14T15:16:12.865685+0000","last_active":"2024-04-14T15:16:12.865685+0000","last_peered":"2024-04-14T15:16:12.865685+0000","last_clean":"2024-04-14T15:16:12.865685+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:16:12.865685+0000","last_undegraded":"2024-04-14T15:16:12.865685+0000","last_fullsized":"2024-04-14T15:16:12.865685+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:12.865685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:12.865685+0000","last_clean_scrub_stamp":"2024-04-14T15:16:12.865685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"270.5","version":"0'0","reported_seq":4,"reported_epoch":301,"state":"creating+activating","last_fresh":"2024-04-14T15:16:08.881807+0000","last_change":"2024-04-14T15:16:08.874491+0000","last_active":"2024-04-14T15:16:07.860882+0000","last_peered":"2024-04-14T15:16:07.860882+0000","last_clean":"2024-04-14T15:16:07.860882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:16:08.881807+0000","last_undegraded":"2024-04-14T15:16:08.881807+0000","last_fullsized":"2024-04-14T15:16:08.881807+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:07.860882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:07.860882+0000","last_clean_scrub_stamp":"2024-04-14T15:16:07.860882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":["1","3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"271.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:16:12.865685+0000","last_change":"2024-04-14T15:16:12.865685+0000","last_active":"2024-04-14T15:16:12.865685+0000","last_peered":"2024-04-14T15:16:12.865685+0000","last_clean":"2024-04-14T15:16:12.865685+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:16:12.865685+0000","last_undegraded":"2024-04-14T15:16:12.865685+0000","last_fullsized":"2024-04-14T15:16:12.865685+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:12.865685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:12.865685+0000","last_clean_scrub_stamp":"2024-04-14T15:16:12.865685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"270.4","version":"0'0","reported_seq":10,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.889625+0000","last_change":"2024-04-14T15:16:08.888619+0000","last_active":"2024-04-14T15:16:08.889625+0000","last_peered":"2024-04-14T15:16:08.889625+0000","last_clean":"2024-04-14T15:16:08.889625+0000","last_became_active":"2024-04-14T15:16:08.888324+0000","last_became_peered":"2024-04-14T15:16:08.888324+0000","last_unstale":"2024-04-14T15:16:08.889625+0000","last_undegraded":"2024-04-14T15:16:08.889625+0000","last_fullsized":"2024-04-14T15:16:08.889625+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:07.860882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:07.860882+0000","last_clean_scrub_stamp":"2024-04-14T15:16:07.860882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:34.113946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"271.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:16:12.865685+0000","last_change":"2024-04-14T15:16:12.865685+0000","last_active":"2024-04-14T15:16:12.865685+0000","last_peered":"2024-04-14T15:16:12.865685+0000","last_clean":"2024-04-14T15:16:12.865685+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:16:12.865685+0000","last_undegraded":"2024-04-14T15:16:12.865685+0000","last_fullsized":"2024-04-14T15:16:12.865685+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:12.865685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:12.865685+0000","last_clean_scrub_stamp":"2024-04-14T15:16:12.865685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"270.3","version":"0'0","reported_seq":10,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.887797+0000","last_change":"2024-04-14T15:16:08.887469+0000","last_active":"2024-04-14T15:16:08.887797+0000","last_peered":"2024-04-14T15:16:08.887797+0000","last_clean":"2024-04-14T15:16:08.887797+0000","last_became_active":"2024-04-14T15:16:08.887108+0000","last_became_peered":"2024-04-14T15:16:08.887108+0000","last_unstale":"2024-04-14T15:16:08.887797+0000","last_undegraded":"2024-04-14T15:16:08.887797+0000","last_fullsized":"2024-04-14T15:16:08.887797+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:07.860882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:07.860882+0000","last_clean_scrub_stamp":"2024-04-14T15:16:07.860882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:12.368992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"271.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:16:12.865685+0000","last_change":"2024-04-14T15:16:12.865685+0000","last_active":"2024-04-14T15:16:12.865685+0000","last_peered":"2024-04-14T15:16:12.865685+0000","last_clean":"2024-04-14T15:16:12.865685+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:16:12.865685+0000","last_undegraded":"2024-04-14T15:16:12.865685+0000","last_fullsized":"2024-04-14T15:16:12.865685+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:12.865685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:12.865685+0000","last_clean_scrub_stamp":"2024-04-14T15:16:12.865685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"270.2","version":"0'0","reported_seq":10,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.883756+0000","last_change":"2024-04-14T15:16:08.880535+0000","last_active":"2024-04-14T15:16:08.883756+0000","last_peered":"2024-04-14T15:16:08.883756+0000","last_clean":"2024-04-14T15:16:08.883756+0000","last_became_active":"2024-04-14T15:16:08.880229+0000","last_became_peered":"2024-04-14T15:16:08.880229+0000","last_unstale":"2024-04-14T15:16:08.883756+0000","last_undegraded":"2024-04-14T15:16:08.883756+0000","last_fullsized":"2024-04-14T15:16:08.883756+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:07.860882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:07.860882+0000","last_clean_scrub_stamp":"2024-04-14T15:16:07.860882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:14.418243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"271.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:16:12.865685+0000","last_change":"2024-04-14T15:16:12.865685+0000","last_active":"2024-04-14T15:16:12.865685+0000","last_peered":"2024-04-14T15:16:12.865685+0000","last_clean":"2024-04-14T15:16:12.865685+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:16:12.865685+0000","last_undegraded":"2024-04-14T15:16:12.865685+0000","last_fullsized":"2024-04-14T15:16:12.865685+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:12.865685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:12.865685+0000","last_clean_scrub_stamp":"2024-04-14T15:16:12.865685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"270.1","version":"0'0","reported_seq":10,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.889892+0000","last_change":"2024-04-14T15:16:08.889594+0000","last_active":"2024-04-14T15:16:08.889892+0000","last_peered":"2024-04-14T15:16:08.889892+0000","last_clean":"2024-04-14T15:16:08.889892+0000","last_became_active":"2024-04-14T15:16:08.889250+0000","last_became_peered":"2024-04-14T15:16:08.889250+0000","last_unstale":"2024-04-14T15:16:08.889892+0000","last_undegraded":"2024-04-14T15:16:08.889892+0000","last_fullsized":"2024-04-14T15:16:08.889892+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:07.860882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:07.860882+0000","last_clean_scrub_stamp":"2024-04-14T15:16:07.860882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:17.833018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"271.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:16:12.865685+0000","last_change":"2024-04-14T15:16:12.865685+0000","last_active":"2024-04-14T15:16:12.865685+0000","last_peered":"2024-04-14T15:16:12.865685+0000","last_clean":"2024-04-14T15:16:12.865685+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:16:12.865685+0000","last_undegraded":"2024-04-14T15:16:12.865685+0000","last_fullsized":"2024-04-14T15:16:12.865685+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:12.865685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:12.865685+0000","last_clean_scrub_stamp":"2024-04-14T15:16:12.865685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"270.0","version":"0'0","reported_seq":10,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.883297+0000","last_change":"2024-04-14T15:16:08.882859+0000","last_active":"2024-04-14T15:16:08.883297+0000","last_peered":"2024-04-14T15:16:08.883297+0000","last_clean":"2024-04-14T15:16:08.883297+0000","last_became_active":"2024-04-14T15:16:08.882575+0000","last_became_peered":"2024-04-14T15:16:08.882575+0000","last_unstale":"2024-04-14T15:16:08.883297+0000","last_undegraded":"2024-04-14T15:16:08.883297+0000","last_fullsized":"2024-04-14T15:16:08.883297+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:07.860882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:07.860882+0000","last_clean_scrub_stamp":"2024-04-14T15:16:07.860882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:33.630995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"110.11","version":"253'39","reported_seq":542,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.874787+0000","last_change":"2024-04-14T15:16:03.875072+0000","last_active":"2024-04-14T15:16:08.874787+0000","last_peered":"2024-04-14T15:16:08.874787+0000","last_clean":"2024-04-14T15:16:08.874787+0000","last_became_active":"2024-04-14T15:13:09.936619+0000","last_became_peered":"2024-04-14T15:13:09.936619+0000","last_unstale":"2024-04-14T15:16:08.874787+0000","last_undegraded":"2024-04-14T15:16:08.874787+0000","last_fullsized":"2024-04-14T15:16:08.874787+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"253'39","last_scrub_stamp":"2024-04-14T15:15:38.146569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:15:38.146569+0000","objects_scrubbed":2,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:54.274848+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0016290829999999999,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"110.10","version":"253'158","reported_seq":627,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.869888+0000","last_change":"2024-04-14T15:16:03.881102+0000","last_active":"2024-04-14T15:16:08.869888+0000","last_peered":"2024-04-14T15:16:08.869888+0000","last_clean":"2024-04-14T15:16:08.869888+0000","last_became_active":"2024-04-14T15:13:09.928915+0000","last_became_peered":"2024-04-14T15:13:09.928915+0000","last_unstale":"2024-04-14T15:16:08.869888+0000","last_undegraded":"2024-04-14T15:16:08.869888+0000","last_fullsized":"2024-04-14T15:16:08.869888+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"253'158","last_scrub_stamp":"2024-04-14T15:15:29.853735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:15:29.853735+0000","objects_scrubbed":10,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:48.721712+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0018074390000000001,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":47,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"110.13","version":"234'39","reported_seq":545,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.877515+0000","last_change":"2024-04-14T15:16:03.896377+0000","last_active":"2024-04-14T15:16:08.877515+0000","last_peered":"2024-04-14T15:16:08.877515+0000","last_clean":"2024-04-14T15:16:08.877515+0000","last_became_active":"2024-04-14T15:13:09.918152+0000","last_became_peered":"2024-04-14T15:13:09.918152+0000","last_unstale":"2024-04-14T15:16:08.877515+0000","last_undegraded":"2024-04-14T15:16:08.877515+0000","last_fullsized":"2024-04-14T15:16:08.877515+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'37","last_scrub_stamp":"2024-04-14T15:15:01.282258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:15:01.282258+0000","objects_scrubbed":4,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:09.318992+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0047971690000000004,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"110.12","version":"253'66","reported_seq":569,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.876266+0000","last_change":"2024-04-14T15:16:03.894781+0000","last_active":"2024-04-14T15:16:08.876266+0000","last_peered":"2024-04-14T15:16:08.876266+0000","last_clean":"2024-04-14T15:16:08.876266+0000","last_became_active":"2024-04-14T15:13:09.939566+0000","last_became_peered":"2024-04-14T15:13:09.939566+0000","last_unstale":"2024-04-14T15:16:08.876266+0000","last_undegraded":"2024-04-14T15:16:08.876266+0000","last_fullsized":"2024-04-14T15:16:08.876266+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'56","last_scrub_stamp":"2024-04-14T15:14:53.271937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:14:53.271937+0000","objects_scrubbed":4,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.003844564,"stat_sum":{"num_bytes":48,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":21,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"110.15","version":"190'34","reported_seq":539,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.885672+0000","last_change":"2024-04-14T15:16:06.494156+0000","last_active":"2024-04-14T15:16:08.885672+0000","last_peered":"2024-04-14T15:16:08.885672+0000","last_clean":"2024-04-14T15:16:08.885672+0000","last_became_active":"2024-04-14T15:13:09.960121+0000","last_became_peered":"2024-04-14T15:13:09.960121+0000","last_unstale":"2024-04-14T15:16:08.885672+0000","last_undegraded":"2024-04-14T15:16:08.885672+0000","last_fullsized":"2024-04-14T15:16:08.885672+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'34","last_scrub_stamp":"2024-04-14T15:16:06.494065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:16:06.494065+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:08.963732+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0020885510000000001,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"110.14","version":"275'86","reported_seq":584,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.886909+0000","last_change":"2024-04-14T15:16:03.883165+0000","last_active":"2024-04-14T15:16:08.886909+0000","last_peered":"2024-04-14T15:16:08.886909+0000","last_clean":"2024-04-14T15:16:08.886909+0000","last_became_active":"2024-04-14T15:13:09.899561+0000","last_became_peered":"2024-04-14T15:13:09.899561+0000","last_unstale":"2024-04-14T15:16:08.886909+0000","last_undegraded":"2024-04-14T15:16:08.886909+0000","last_fullsized":"2024-04-14T15:16:08.886909+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'85","last_scrub_stamp":"2024-04-14T15:15:42.409029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:15:42.409029+0000","objects_scrubbed":5,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:11.940212+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0039978469999999997,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":26,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"110.17","version":"234'6","reported_seq":518,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.881390+0000","last_change":"2024-04-14T15:16:07.459851+0000","last_active":"2024-04-14T15:16:08.881390+0000","last_peered":"2024-04-14T15:16:08.881390+0000","last_clean":"2024-04-14T15:16:08.881390+0000","last_became_active":"2024-04-14T15:13:09.898521+0000","last_became_peered":"2024-04-14T15:13:09.898521+0000","last_unstale":"2024-04-14T15:16:08.881390+0000","last_undegraded":"2024-04-14T15:16:08.881390+0000","last_fullsized":"2024-04-14T15:16:08.881390+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'6","last_scrub_stamp":"2024-04-14T15:16:07.459718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:16:07.459718+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:22.451012+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0019949540000000002,"stat_sum":{"num_bytes":13,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"110.16","version":"190'34","reported_seq":374,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.878906+0000","last_change":"2024-04-14T15:16:03.887023+0000","last_active":"2024-04-14T15:16:08.878906+0000","last_peered":"2024-04-14T15:16:08.878906+0000","last_clean":"2024-04-14T15:16:08.878906+0000","last_became_active":"2024-04-14T15:15:21.889783+0000","last_became_peered":"2024-04-14T15:15:21.889783+0000","last_unstale":"2024-04-14T15:16:08.878906+0000","last_undegraded":"2024-04-14T15:16:08.878906+0000","last_fullsized":"2024-04-14T15:16:08.878906+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'34","last_scrub_stamp":"2024-04-14T15:15:24.848169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:15:24.848169+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:42.474031+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.002398131,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"110.19","version":"237'85","reported_seq":588,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.871710+0000","last_change":"2024-04-14T15:16:04.177194+0000","last_active":"2024-04-14T15:16:08.871710+0000","last_peered":"2024-04-14T15:16:08.871710+0000","last_clean":"2024-04-14T15:16:08.871710+0000","last_became_active":"2024-04-14T15:13:09.998139+0000","last_became_peered":"2024-04-14T15:13:09.998139+0000","last_unstale":"2024-04-14T15:16:08.871710+0000","last_undegraded":"2024-04-14T15:16:08.871710+0000","last_fullsized":"2024-04-14T15:16:08.871710+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'85","last_scrub_stamp":"2024-04-14T15:16:04.177097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:16:04.177097+0000","objects_scrubbed":5,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:28.884049+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0060025429999999999,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"110.18","version":"296'93","reported_seq":587,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.871348+0000","last_change":"2024-04-14T15:16:05.887645+0000","last_active":"2024-04-14T15:16:08.871348+0000","last_peered":"2024-04-14T15:16:08.871348+0000","last_clean":"2024-04-14T15:16:08.871348+0000","last_became_active":"2024-04-14T15:13:09.998803+0000","last_became_peered":"2024-04-14T15:13:09.998803+0000","last_unstale":"2024-04-14T15:16:08.871348+0000","last_undegraded":"2024-04-14T15:16:08.871348+0000","last_fullsized":"2024-04-14T15:16:08.871348+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'85","last_scrub_stamp":"2024-04-14T15:15:35.308199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:15:35.308199+0000","objects_scrubbed":5,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:38.133629+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.024348310000000001,"stat_sum":{"num_bytes":57,"num_objects":7,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":28,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"110.1b","version":"234'90","reported_seq":588,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.877408+0000","last_change":"2024-04-14T15:16:03.895253+0000","last_active":"2024-04-14T15:16:08.877408+0000","last_peered":"2024-04-14T15:16:08.877408+0000","last_clean":"2024-04-14T15:16:08.877408+0000","last_became_active":"2024-04-14T15:13:09.946919+0000","last_became_peered":"2024-04-14T15:13:09.946919+0000","last_unstale":"2024-04-14T15:16:08.877408+0000","last_undegraded":"2024-04-14T15:16:08.877408+0000","last_fullsized":"2024-04-14T15:16:08.877408+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'90","last_scrub_stamp":"2024-04-14T15:15:27.424479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:15:27.424479+0000","objects_scrubbed":5,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:46.832268+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0037919410000000001,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":27,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"110.1a","version":"203'34","reported_seq":537,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.877282+0000","last_change":"2024-04-14T15:16:03.875517+0000","last_active":"2024-04-14T15:16:08.877282+0000","last_peered":"2024-04-14T15:16:08.877282+0000","last_clean":"2024-04-14T15:16:08.877282+0000","last_became_active":"2024-04-14T15:13:09.960319+0000","last_became_peered":"2024-04-14T15:13:09.960319+0000","last_unstale":"2024-04-14T15:16:08.877282+0000","last_undegraded":"2024-04-14T15:16:08.877282+0000","last_fullsized":"2024-04-14T15:16:08.877282+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'34","last_scrub_stamp":"2024-04-14T15:15:34.212376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:15:34.212376+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:35.852150+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0017807330000000001,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"110.1d","version":"225'85","reported_seq":584,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.881718+0000","last_change":"2024-04-14T15:16:03.896420+0000","last_active":"2024-04-14T15:16:08.881718+0000","last_peered":"2024-04-14T15:16:08.881718+0000","last_clean":"2024-04-14T15:16:08.881718+0000","last_became_active":"2024-04-14T15:13:09.868082+0000","last_became_peered":"2024-04-14T15:13:09.868082+0000","last_unstale":"2024-04-14T15:16:08.881718+0000","last_undegraded":"2024-04-14T15:16:08.881718+0000","last_fullsized":"2024-04-14T15:16:08.881718+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"225'85","last_scrub_stamp":"2024-04-14T15:15:26.704550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:15:26.704550+0000","objects_scrubbed":5,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:40.836177+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00141866,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"110.1c","version":"190'73","reported_seq":574,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.877820+0000","last_change":"2024-04-14T15:16:04.615669+0000","last_active":"2024-04-14T15:16:08.877820+0000","last_peered":"2024-04-14T15:16:08.877820+0000","last_clean":"2024-04-14T15:16:08.877820+0000","last_became_active":"2024-04-14T15:13:09.954091+0000","last_became_peered":"2024-04-14T15:13:09.954091+0000","last_unstale":"2024-04-14T15:16:08.877820+0000","last_undegraded":"2024-04-14T15:16:08.877820+0000","last_fullsized":"2024-04-14T15:16:08.877820+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'73","last_scrub_stamp":"2024-04-14T15:16:04.615572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:16:04.615572+0000","objects_scrubbed":5,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:20.831295+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0056190420000000003,"stat_sum":{"num_bytes":49,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"110.1f","version":"268'25","reported_seq":584,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.875064+0000","last_change":"2024-04-14T15:16:08.237015+0000","last_active":"2024-04-14T15:16:08.875064+0000","last_peered":"2024-04-14T15:16:08.875064+0000","last_clean":"2024-04-14T15:16:08.875064+0000","last_became_active":"2024-04-14T15:13:09.999629+0000","last_became_peered":"2024-04-14T15:13:09.999629+0000","last_unstale":"2024-04-14T15:16:08.875064+0000","last_undegraded":"2024-04-14T15:16:08.875064+0000","last_fullsized":"2024-04-14T15:16:08.875064+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'25","last_scrub_stamp":"2024-04-14T15:16:08.236881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:16:08.236881+0000","objects_scrubbed":2,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:11.141492+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0061123380000000001,"stat_sum":{"num_bytes":4194315,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":8,"num_write_kb":8198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"110.1e","version":"190'22","reported_seq":523,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.883711+0000","last_change":"2024-04-14T15:16:03.885392+0000","last_active":"2024-04-14T15:16:08.883711+0000","last_peered":"2024-04-14T15:16:08.883711+0000","last_clean":"2024-04-14T15:16:08.883711+0000","last_became_active":"2024-04-14T15:13:09.955932+0000","last_became_peered":"2024-04-14T15:13:09.955932+0000","last_unstale":"2024-04-14T15:16:08.883711+0000","last_undegraded":"2024-04-14T15:16:08.883711+0000","last_fullsized":"2024-04-14T15:16:08.883711+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:13:08.799672+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003436284,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"110.1","version":"219'24","reported_seq":536,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.879598+0000","last_change":"2024-04-14T15:16:03.895140+0000","last_active":"2024-04-14T15:16:08.879598+0000","last_peered":"2024-04-14T15:16:08.879598+0000","last_clean":"2024-04-14T15:16:08.879598+0000","last_became_active":"2024-04-14T15:13:09.929562+0000","last_became_peered":"2024-04-14T15:13:09.929562+0000","last_unstale":"2024-04-14T15:16:08.879598+0000","last_undegraded":"2024-04-14T15:16:08.879598+0000","last_fullsized":"2024-04-14T15:16:08.879598+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"219'24","last_scrub_stamp":"2024-04-14T15:15:35.591006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:15:35.591006+0000","objects_scrubbed":3,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:42.003847+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00097444399999999998,"stat_sum":{"num_bytes":24,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":9,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"110.0","version":"219'85","reported_seq":578,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.869995+0000","last_change":"2024-04-14T15:16:03.881458+0000","last_active":"2024-04-14T15:16:08.869995+0000","last_peered":"2024-04-14T15:16:08.869995+0000","last_clean":"2024-04-14T15:16:08.869995+0000","last_became_active":"2024-04-14T15:13:09.856407+0000","last_became_peered":"2024-04-14T15:13:09.856407+0000","last_unstale":"2024-04-14T15:16:08.869995+0000","last_undegraded":"2024-04-14T15:16:08.869995+0000","last_fullsized":"2024-04-14T15:16:08.869995+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"219'85","last_scrub_stamp":"2024-04-14T15:15:25.872949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:15:25.872949+0000","objects_scrubbed":5,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:34.991073+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0.0020747539999999998,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"110.3","version":"223'51","reported_seq":547,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.871819+0000","last_change":"2024-04-14T15:16:06.149432+0000","last_active":"2024-04-14T15:16:08.871819+0000","last_peered":"2024-04-14T15:16:08.871819+0000","last_clean":"2024-04-14T15:16:08.871819+0000","last_became_active":"2024-04-14T15:13:09.997208+0000","last_became_peered":"2024-04-14T15:13:09.997208+0000","last_unstale":"2024-04-14T15:16:08.871819+0000","last_undegraded":"2024-04-14T15:16:08.871819+0000","last_fullsized":"2024-04-14T15:16:08.871819+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"223'51","last_scrub_stamp":"2024-04-14T15:16:06.149271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:16:06.149271+0000","objects_scrubbed":3,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:08.570271+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0068076480000000003,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"110.2","version":"240'55","reported_seq":558,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.872455+0000","last_change":"2024-04-14T15:16:03.880577+0000","last_active":"2024-04-14T15:16:08.872455+0000","last_peered":"2024-04-14T15:16:08.872455+0000","last_clean":"2024-04-14T15:16:08.872455+0000","last_became_active":"2024-04-14T15:13:09.957187+0000","last_became_peered":"2024-04-14T15:13:09.957187+0000","last_unstale":"2024-04-14T15:16:08.872455+0000","last_undegraded":"2024-04-14T15:16:08.872455+0000","last_fullsized":"2024-04-14T15:16:08.872455+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'55","last_scrub_stamp":"2024-04-14T15:15:40.376133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:15:40.376133+0000","objects_scrubbed":6,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:05.879250+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0066344619999999998,"stat_sum":{"num_bytes":49,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":18,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"110.5","version":"283'59","reported_seq":556,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.869370+0000","last_change":"2024-04-14T15:16:03.882342+0000","last_active":"2024-04-14T15:16:08.869370+0000","last_peered":"2024-04-14T15:16:08.869370+0000","last_clean":"2024-04-14T15:16:08.869370+0000","last_became_active":"2024-04-14T15:13:09.853689+0000","last_became_peered":"2024-04-14T15:13:09.853689+0000","last_unstale":"2024-04-14T15:16:08.869370+0000","last_undegraded":"2024-04-14T15:16:08.869370+0000","last_fullsized":"2024-04-14T15:16:08.869370+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'57","last_scrub_stamp":"2024-04-14T15:15:27.862702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:15:27.862702+0000","objects_scrubbed":5,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:34.297023+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.001407202,"stat_sum":{"num_bytes":42,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":20,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"110.4","version":"190'51","reported_seq":551,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.887025+0000","last_change":"2024-04-14T15:16:05.487931+0000","last_active":"2024-04-14T15:16:08.887025+0000","last_peered":"2024-04-14T15:16:08.887025+0000","last_clean":"2024-04-14T15:16:08.887025+0000","last_became_active":"2024-04-14T15:13:09.958847+0000","last_became_peered":"2024-04-14T15:13:09.958847+0000","last_unstale":"2024-04-14T15:16:08.887025+0000","last_undegraded":"2024-04-14T15:16:08.887025+0000","last_fullsized":"2024-04-14T15:16:08.887025+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'51","last_scrub_stamp":"2024-04-14T15:16:05.487850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:16:05.487850+0000","objects_scrubbed":3,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:34.289111+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0048323250000000002,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"110.7","version":"201'17","reported_seq":524,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.878833+0000","last_change":"2024-04-14T15:16:03.876877+0000","last_active":"2024-04-14T15:16:08.878833+0000","last_peered":"2024-04-14T15:16:08.878833+0000","last_clean":"2024-04-14T15:16:08.878833+0000","last_became_active":"2024-04-14T15:13:09.936631+0000","last_became_peered":"2024-04-14T15:13:09.936631+0000","last_unstale":"2024-04-14T15:16:08.878833+0000","last_undegraded":"2024-04-14T15:16:08.878833+0000","last_fullsized":"2024-04-14T15:16:08.878833+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"201'17","last_scrub_stamp":"2024-04-14T15:15:30.308163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:15:30.308163+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:35.177195+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001103808,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"110.6","version":"274'74","reported_seq":569,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.884598+0000","last_change":"2024-04-14T15:16:03.897478+0000","last_active":"2024-04-14T15:16:08.884598+0000","last_peered":"2024-04-14T15:16:08.884598+0000","last_clean":"2024-04-14T15:16:08.884598+0000","last_became_active":"2024-04-14T15:13:09.930125+0000","last_became_peered":"2024-04-14T15:13:09.930125+0000","last_unstale":"2024-04-14T15:16:08.884598+0000","last_undegraded":"2024-04-14T15:16:08.884598+0000","last_fullsized":"2024-04-14T15:16:08.884598+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'74","last_scrub_stamp":"2024-04-14T15:15:42.641684+0000","last_deep_scrub":"274'74","last_deep_scrub_stamp":"2024-04-14T15:15:42.641684+0000","last_clean_scrub_stamp":"2024-04-14T15:15:42.641684+0000","objects_scrubbed":5,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:56.397899+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001879475,"stat_sum":{"num_bytes":44,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":21,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"110.9","version":"225'17","reported_seq":516,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.871596+0000","last_change":"2024-04-14T15:16:03.879083+0000","last_active":"2024-04-14T15:16:08.871596+0000","last_peered":"2024-04-14T15:16:08.871596+0000","last_clean":"2024-04-14T15:16:08.871596+0000","last_became_active":"2024-04-14T15:13:09.957324+0000","last_became_peered":"2024-04-14T15:13:09.957324+0000","last_unstale":"2024-04-14T15:16:08.871596+0000","last_undegraded":"2024-04-14T15:16:08.871596+0000","last_fullsized":"2024-04-14T15:16:08.871596+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:13:08.799672+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0055950490000000004,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"110.8","version":"190'17","reported_seq":279,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.877515+0000","last_change":"2024-04-14T15:16:03.889092+0000","last_active":"2024-04-14T15:16:08.877515+0000","last_peered":"2024-04-14T15:16:08.877515+0000","last_clean":"2024-04-14T15:16:08.877515+0000","last_became_active":"2024-04-14T15:15:44.890057+0000","last_became_peered":"2024-04-14T15:15:44.890057+0000","last_unstale":"2024-04-14T15:16:08.877515+0000","last_undegraded":"2024-04-14T15:16:08.877515+0000","last_fullsized":"2024-04-14T15:16:08.877515+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'17","last_scrub_stamp":"2024-04-14T15:15:25.478924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:15:25.478924+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:25.835562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00271433,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"110.b","version":"190'68","reported_seq":566,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.877674+0000","last_change":"2024-04-14T15:16:03.898209+0000","last_active":"2024-04-14T15:16:08.877674+0000","last_peered":"2024-04-14T15:16:08.877674+0000","last_clean":"2024-04-14T15:16:08.877674+0000","last_became_active":"2024-04-14T15:13:09.964629+0000","last_became_peered":"2024-04-14T15:13:09.964629+0000","last_unstale":"2024-04-14T15:16:08.877674+0000","last_undegraded":"2024-04-14T15:16:08.877674+0000","last_fullsized":"2024-04-14T15:16:08.877674+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'68","last_scrub_stamp":"2024-04-14T15:14:51.349107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:14:51.349107+0000","objects_scrubbed":4,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0049053020000000003,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"110.a","version":"224'70","reported_seq":495,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.877597+0000","last_change":"2024-04-14T15:16:03.888790+0000","last_active":"2024-04-14T15:16:08.877597+0000","last_peered":"2024-04-14T15:16:08.877597+0000","last_clean":"2024-04-14T15:16:08.877597+0000","last_became_active":"2024-04-14T15:15:21.885346+0000","last_became_peered":"2024-04-14T15:15:21.885346+0000","last_unstale":"2024-04-14T15:16:08.877597+0000","last_undegraded":"2024-04-14T15:16:08.877597+0000","last_fullsized":"2024-04-14T15:16:08.877597+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'70","last_scrub_stamp":"2024-04-14T15:15:34.881731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:15:34.881731+0000","objects_scrubbed":5,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:55.068792+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0.00502967,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"110.d","version":"190'85","reported_seq":417,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.878910+0000","last_change":"2024-04-14T15:16:03.886262+0000","last_active":"2024-04-14T15:16:08.878910+0000","last_peered":"2024-04-14T15:16:08.878910+0000","last_clean":"2024-04-14T15:16:08.878910+0000","last_became_active":"2024-04-14T15:15:21.877073+0000","last_became_peered":"2024-04-14T15:15:21.877073+0000","last_unstale":"2024-04-14T15:16:08.878910+0000","last_undegraded":"2024-04-14T15:16:08.878910+0000","last_fullsized":"2024-04-14T15:16:08.878910+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'85","last_scrub_stamp":"2024-04-14T15:15:38.773256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:15:38.773256+0000","objects_scrubbed":5,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:02.793486+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0018858709999999999,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"110.c","version":"190'17","reported_seq":524,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.877941+0000","last_change":"2024-04-14T15:16:03.899855+0000","last_active":"2024-04-14T15:16:08.877941+0000","last_peered":"2024-04-14T15:16:08.877941+0000","last_clean":"2024-04-14T15:16:08.877941+0000","last_became_active":"2024-04-14T15:13:09.984609+0000","last_became_peered":"2024-04-14T15:13:09.984609+0000","last_unstale":"2024-04-14T15:16:08.877941+0000","last_undegraded":"2024-04-14T15:16:08.877941+0000","last_fullsized":"2024-04-14T15:16:08.877941+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'17","last_scrub_stamp":"2024-04-14T15:15:24.368920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:15:24.368920+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:27.443433+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0057052739999999998,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"110.f","version":"190'51","reported_seq":554,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.878418+0000","last_change":"2024-04-14T15:16:03.898253+0000","last_active":"2024-04-14T15:16:08.878418+0000","last_peered":"2024-04-14T15:16:08.878418+0000","last_clean":"2024-04-14T15:16:08.878418+0000","last_became_active":"2024-04-14T15:13:09.964057+0000","last_became_peered":"2024-04-14T15:13:09.964057+0000","last_unstale":"2024-04-14T15:16:08.878418+0000","last_undegraded":"2024-04-14T15:16:08.878418+0000","last_fullsized":"2024-04-14T15:16:08.878418+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'51","last_scrub_stamp":"2024-04-14T15:15:25.399677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:15:25.399677+0000","objects_scrubbed":3,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:43.537422+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0052446230000000003,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"110.e","version":"234'90","reported_seq":587,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.884517+0000","last_change":"2024-04-14T15:16:03.897032+0000","last_active":"2024-04-14T15:16:08.884517+0000","last_peered":"2024-04-14T15:16:08.884517+0000","last_clean":"2024-04-14T15:16:08.884517+0000","last_became_active":"2024-04-14T15:13:09.984980+0000","last_became_peered":"2024-04-14T15:13:09.984980+0000","last_unstale":"2024-04-14T15:16:08.884517+0000","last_undegraded":"2024-04-14T15:16:08.884517+0000","last_fullsized":"2024-04-14T15:16:08.884517+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'90","last_scrub_stamp":"2024-04-14T15:15:24.699013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:15:24.699013+0000","objects_scrubbed":6,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:29.188077+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0016350589999999999,"stat_sum":{"num_bytes":60,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"11.7","version":"0'0","reported_seq":60,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.875154+0000","last_change":"2024-04-14T15:15:44.883275+0000","last_active":"2024-04-14T15:16:08.875154+0000","last_peered":"2024-04-14T15:16:08.875154+0000","last_clean":"2024-04-14T15:16:08.875154+0000","last_became_active":"2024-04-14T15:15:44.882941+0000","last_became_peered":"2024-04-14T15:15:44.882941+0000","last_unstale":"2024-04-14T15:16:08.875154+0000","last_undegraded":"2024-04-14T15:16:08.875154+0000","last_fullsized":"2024-04-14T15:16:08.875154+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:32.465858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:32.465858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:53.128842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.9","version":"85'9","reported_seq":588,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.880305+0000","last_change":"2024-04-14T15:14:45.904003+0000","last_active":"2024-04-14T15:16:08.880305+0000","last_peered":"2024-04-14T15:16:08.880305+0000","last_clean":"2024-04-14T15:16:08.880305+0000","last_became_active":"2024-04-14T15:09:47.045441+0000","last_became_peered":"2024-04-14T15:09:47.045441+0000","last_unstale":"2024-04-14T15:16:08.880305+0000","last_undegraded":"2024-04-14T15:16:08.880305+0000","last_fullsized":"2024-04-14T15:16:08.880305+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9","last_scrub_stamp":"2024-04-14T15:14:45.903923+0000","last_deep_scrub":"85'9","last_deep_scrub_stamp":"2024-04-14T15:10:51.950580+0000","last_clean_scrub_stamp":"2024-04-14T15:14:45.903923+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:10.087549+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"72'15","reported_seq":104,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.876375+0000","last_change":"2024-04-14T15:15:44.885742+0000","last_active":"2024-04-14T15:16:08.876375+0000","last_peered":"2024-04-14T15:16:08.876375+0000","last_clean":"2024-04-14T15:16:08.876375+0000","last_became_active":"2024-04-14T15:15:44.885409+0000","last_became_peered":"2024-04-14T15:15:44.885409+0000","last_unstale":"2024-04-14T15:16:08.876375+0000","last_undegraded":"2024-04-14T15:16:08.876375+0000","last_fullsized":"2024-04-14T15:16:08.876375+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-04-14T15:15:39.325129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:15:39.325129+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:00.619852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":509,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.878724+0000","last_change":"2024-04-14T15:15:35.560140+0000","last_active":"2024-04-14T15:16:08.878724+0000","last_peered":"2024-04-14T15:16:08.878724+0000","last_clean":"2024-04-14T15:16:08.878724+0000","last_became_active":"2024-04-14T15:09:43.021460+0000","last_became_peered":"2024-04-14T15:09:43.021460+0000","last_unstale":"2024-04-14T15:16:08.878724+0000","last_undegraded":"2024-04-14T15:16:08.878724+0000","last_fullsized":"2024-04-14T15:16:08.878724+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:35.560063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:15:35.560063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:41.037947+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":61,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.877088+0000","last_change":"2024-04-14T15:15:44.888612+0000","last_active":"2024-04-14T15:16:08.877088+0000","last_peered":"2024-04-14T15:16:08.877088+0000","last_clean":"2024-04-14T15:16:08.877088+0000","last_became_active":"2024-04-14T15:15:44.888154+0000","last_became_peered":"2024-04-14T15:15:44.888154+0000","last_unstale":"2024-04-14T15:16:08.877088+0000","last_undegraded":"2024-04-14T15:16:08.877088+0000","last_fullsized":"2024-04-14T15:16:08.877088+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:40.368219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:15:40.368219+0000","last_clean_scrub_stamp":"2024-04-14T15:15:40.368219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:00.403221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":406,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.880860+0000","last_change":"2024-04-14T15:15:32.596167+0000","last_active":"2024-04-14T15:16:08.880860+0000","last_peered":"2024-04-14T15:16:08.880860+0000","last_clean":"2024-04-14T15:16:08.880860+0000","last_became_active":"2024-04-14T15:12:56.355348+0000","last_became_peered":"2024-04-14T15:12:56.355348+0000","last_unstale":"2024-04-14T15:16:08.880860+0000","last_undegraded":"2024-04-14T15:16:08.880860+0000","last_fullsized":"2024-04-14T15:16:08.880860+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:32.596096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:32.596096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:53.341960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"72'9","reported_seq":521,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.886545+0000","last_change":"2024-04-14T15:14:08.367575+0000","last_active":"2024-04-14T15:16:08.886545+0000","last_peered":"2024-04-14T15:16:08.886545+0000","last_clean":"2024-04-14T15:16:08.886545+0000","last_became_active":"2024-04-14T15:09:45.037179+0000","last_became_peered":"2024-04-14T15:09:45.037179+0000","last_unstale":"2024-04-14T15:16:08.886545+0000","last_undegraded":"2024-04-14T15:16:08.886545+0000","last_fullsized":"2024-04-14T15:16:08.886545+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-04-14T15:14:08.367484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:14:08.367484+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":509,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.875146+0000","last_change":"2024-04-14T15:15:25.311260+0000","last_active":"2024-04-14T15:16:08.875146+0000","last_peered":"2024-04-14T15:16:08.875146+0000","last_clean":"2024-04-14T15:16:08.875146+0000","last_became_active":"2024-04-14T15:09:43.020044+0000","last_became_peered":"2024-04-14T15:09:43.020044+0000","last_unstale":"2024-04-14T15:16:08.875146+0000","last_undegraded":"2024-04-14T15:16:08.875146+0000","last_fullsized":"2024-04-14T15:16:08.875146+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:25.311184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:15:25.311184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:51.092544+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":501,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.876523+0000","last_change":"2024-04-14T15:14:14.354053+0000","last_active":"2024-04-14T15:16:08.876523+0000","last_peered":"2024-04-14T15:16:08.876523+0000","last_clean":"2024-04-14T15:16:08.876523+0000","last_became_active":"2024-04-14T15:09:47.046153+0000","last_became_peered":"2024-04-14T15:09:47.046153+0000","last_unstale":"2024-04-14T15:16:08.876523+0000","last_undegraded":"2024-04-14T15:16:08.876523+0000","last_fullsized":"2024-04-14T15:16:08.876523+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:14.353970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:14:14.353970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":497,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.885858+0000","last_change":"2024-04-14T15:14:13.382977+0000","last_active":"2024-04-14T15:16:08.885858+0000","last_peered":"2024-04-14T15:16:08.885858+0000","last_clean":"2024-04-14T15:16:08.885858+0000","last_became_active":"2024-04-14T15:09:49.061495+0000","last_became_peered":"2024-04-14T15:09:49.061495+0000","last_unstale":"2024-04-14T15:16:08.885858+0000","last_undegraded":"2024-04-14T15:16:08.885858+0000","last_fullsized":"2024-04-14T15:16:08.885858+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:13.382879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:14:13.382879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":406,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.883180+0000","last_change":"2024-04-14T15:14:55.989719+0000","last_active":"2024-04-14T15:16:08.883180+0000","last_peered":"2024-04-14T15:16:08.883180+0000","last_clean":"2024-04-14T15:16:08.883180+0000","last_became_active":"2024-04-14T15:12:56.246062+0000","last_became_peered":"2024-04-14T15:12:56.246062+0000","last_unstale":"2024-04-14T15:16:08.883180+0000","last_undegraded":"2024-04-14T15:16:08.883180+0000","last_fullsized":"2024-04-14T15:16:08.883180+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:55.989627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:14:55.989627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:12.636250+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"222'28","reported_seq":545,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.877408+0000","last_change":"2024-04-14T15:14:50.055093+0000","last_active":"2024-04-14T15:16:08.877408+0000","last_peered":"2024-04-14T15:16:08.877408+0000","last_clean":"2024-04-14T15:16:08.877408+0000","last_became_active":"2024-04-14T15:09:45.178782+0000","last_became_peered":"2024-04-14T15:09:45.178782+0000","last_unstale":"2024-04-14T15:16:08.877408+0000","last_undegraded":"2024-04-14T15:16:08.877408+0000","last_fullsized":"2024-04-14T15:16:08.877408+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'28","last_scrub_stamp":"2024-04-14T15:14:50.054931+0000","last_deep_scrub":"222'28","last_deep_scrub_stamp":"2024-04-14T15:14:50.054931+0000","last_clean_scrub_stamp":"2024-04-14T15:14:50.054931+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":517,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.879829+0000","last_change":"2024-04-14T15:15:43.569812+0000","last_active":"2024-04-14T15:16:08.879829+0000","last_peered":"2024-04-14T15:16:08.879829+0000","last_clean":"2024-04-14T15:16:08.879829+0000","last_became_active":"2024-04-14T15:09:43.020268+0000","last_became_peered":"2024-04-14T15:09:43.020268+0000","last_unstale":"2024-04-14T15:16:08.879829+0000","last_undegraded":"2024-04-14T15:16:08.879829+0000","last_fullsized":"2024-04-14T15:16:08.879829+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:43.569735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:34.342793+0000","last_clean_scrub_stamp":"2024-04-14T15:15:43.569735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:04.129563+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":501,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.877770+0000","last_change":"2024-04-14T15:15:30.488857+0000","last_active":"2024-04-14T15:16:08.877770+0000","last_peered":"2024-04-14T15:16:08.877770+0000","last_clean":"2024-04-14T15:16:08.877770+0000","last_became_active":"2024-04-14T15:09:47.045542+0000","last_became_peered":"2024-04-14T15:09:47.045542+0000","last_unstale":"2024-04-14T15:16:08.877770+0000","last_undegraded":"2024-04-14T15:16:08.877770+0000","last_fullsized":"2024-04-14T15:16:08.877770+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:30.488781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:15:30.488781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:56.619373+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":497,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.876615+0000","last_change":"2024-04-14T15:16:08.857476+0000","last_active":"2024-04-14T15:16:08.876615+0000","last_peered":"2024-04-14T15:16:08.876615+0000","last_clean":"2024-04-14T15:16:08.876615+0000","last_became_active":"2024-04-14T15:09:49.065397+0000","last_became_peered":"2024-04-14T15:09:49.065397+0000","last_unstale":"2024-04-14T15:16:08.876615+0000","last_undegraded":"2024-04-14T15:16:08.876615+0000","last_fullsized":"2024-04-14T15:16:08.876615+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:08.857397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:08.857397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:26.230364+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"91'130","reported_seq":385,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.874624+0000","last_change":"2024-04-14T15:16:07.609240+0000","last_active":"2024-04-14T15:16:08.874624+0000","last_peered":"2024-04-14T15:16:08.874624+0000","last_clean":"2024-04-14T15:16:08.874624+0000","last_became_active":"2024-04-14T15:15:44.886240+0000","last_became_peered":"2024-04-14T15:15:44.886240+0000","last_unstale":"2024-04-14T15:16:08.874624+0000","last_undegraded":"2024-04-14T15:16:08.874624+0000","last_fullsized":"2024-04-14T15:16:08.874624+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:16:07.609119+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:16:07.609119+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:27.856801+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":406,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.876438+0000","last_change":"2024-04-14T15:14:12.319235+0000","last_active":"2024-04-14T15:16:08.876438+0000","last_peered":"2024-04-14T15:16:08.876438+0000","last_clean":"2024-04-14T15:16:08.876438+0000","last_became_active":"2024-04-14T15:12:56.474322+0000","last_became_peered":"2024-04-14T15:12:56.474322+0000","last_unstale":"2024-04-14T15:16:08.876438+0000","last_undegraded":"2024-04-14T15:16:08.876438+0000","last_fullsized":"2024-04-14T15:16:08.876438+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:12.319153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:14:12.319153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"72'17","reported_seq":581,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.870289+0000","last_change":"2024-04-14T15:15:39.992484+0000","last_active":"2024-04-14T15:16:08.870289+0000","last_peered":"2024-04-14T15:16:08.870289+0000","last_clean":"2024-04-14T15:16:08.870289+0000","last_became_active":"2024-04-14T15:13:12.933907+0000","last_became_peered":"2024-04-14T15:13:12.933907+0000","last_unstale":"2024-04-14T15:16:08.870289+0000","last_undegraded":"2024-04-14T15:16:08.870289+0000","last_fullsized":"2024-04-14T15:16:08.870289+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'17","last_scrub_stamp":"2024-04-14T15:15:39.992385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:15:39.992385+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:48.777854+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":501,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.882004+0000","last_change":"2024-04-14T15:13:20.212603+0000","last_active":"2024-04-14T15:16:08.882004+0000","last_peered":"2024-04-14T15:16:08.882004+0000","last_clean":"2024-04-14T15:16:08.882004+0000","last_became_active":"2024-04-14T15:09:43.022351+0000","last_became_peered":"2024-04-14T15:09:43.022351+0000","last_unstale":"2024-04-14T15:16:08.882004+0000","last_undegraded":"2024-04-14T15:16:08.882004+0000","last_fullsized":"2024-04-14T15:16:08.882004+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:20.212523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:13:20.212523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":493,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.874245+0000","last_change":"2024-04-14T15:14:46.208558+0000","last_active":"2024-04-14T15:16:08.874245+0000","last_peered":"2024-04-14T15:16:08.874245+0000","last_clean":"2024-04-14T15:16:08.874245+0000","last_became_active":"2024-04-14T15:09:47.042810+0000","last_became_peered":"2024-04-14T15:09:47.042810+0000","last_unstale":"2024-04-14T15:16:08.874245+0000","last_undegraded":"2024-04-14T15:16:08.874245+0000","last_fullsized":"2024-04-14T15:16:08.874245+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:46.208455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:14:46.208455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:13.100718+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":497,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.878710+0000","last_change":"2024-04-14T15:14:03.309357+0000","last_active":"2024-04-14T15:16:08.878710+0000","last_peered":"2024-04-14T15:16:08.878710+0000","last_clean":"2024-04-14T15:16:08.878710+0000","last_became_active":"2024-04-14T15:09:49.069594+0000","last_became_peered":"2024-04-14T15:09:49.069594+0000","last_unstale":"2024-04-14T15:16:08.878710+0000","last_undegraded":"2024-04-14T15:16:08.878710+0000","last_fullsized":"2024-04-14T15:16:08.878710+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.309280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.309280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"91'130","reported_seq":778,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.875897+0000","last_change":"2024-04-14T15:14:46.903338+0000","last_active":"2024-04-14T15:16:08.875897+0000","last_peered":"2024-04-14T15:16:08.875897+0000","last_clean":"2024-04-14T15:16:08.875897+0000","last_became_active":"2024-04-14T15:14:17.873403+0000","last_became_peered":"2024-04-14T15:14:17.873403+0000","last_unstale":"2024-04-14T15:16:08.875897+0000","last_undegraded":"2024-04-14T15:16:08.875897+0000","last_fullsized":"2024-04-14T15:16:08.875897+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:14:46.903248+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:14:46.903248+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":406,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.878112+0000","last_change":"2024-04-14T15:14:52.110153+0000","last_active":"2024-04-14T15:16:08.878112+0000","last_peered":"2024-04-14T15:16:08.878112+0000","last_clean":"2024-04-14T15:16:08.878112+0000","last_became_active":"2024-04-14T15:12:56.289745+0000","last_became_peered":"2024-04-14T15:12:56.289745+0000","last_unstale":"2024-04-14T15:16:08.878112+0000","last_undegraded":"2024-04-14T15:16:08.878112+0000","last_fullsized":"2024-04-14T15:16:08.878112+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:52.110075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:14:52.110075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"179'15","reported_seq":517,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.869111+0000","last_change":"2024-04-14T15:14:07.488544+0000","last_active":"2024-04-14T15:16:08.869111+0000","last_peered":"2024-04-14T15:16:08.869111+0000","last_clean":"2024-04-14T15:16:08.869111+0000","last_became_active":"2024-04-14T15:09:45.029170+0000","last_became_peered":"2024-04-14T15:09:45.029170+0000","last_unstale":"2024-04-14T15:16:08.869111+0000","last_undegraded":"2024-04-14T15:16:08.869111+0000","last_fullsized":"2024-04-14T15:16:08.869111+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'15","last_scrub_stamp":"2024-04-14T15:14:07.488422+0000","last_deep_scrub":"179'15","last_deep_scrub_stamp":"2024-04-14T15:14:07.488422+0000","last_clean_scrub_stamp":"2024-04-14T15:14:07.488422+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":509,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.872828+0000","last_change":"2024-04-14T15:14:40.222949+0000","last_active":"2024-04-14T15:16:08.872828+0000","last_peered":"2024-04-14T15:16:08.872828+0000","last_clean":"2024-04-14T15:16:08.872828+0000","last_became_active":"2024-04-14T15:09:43.018871+0000","last_became_peered":"2024-04-14T15:09:43.018871+0000","last_unstale":"2024-04-14T15:16:08.872828+0000","last_undegraded":"2024-04-14T15:16:08.872828+0000","last_fullsized":"2024-04-14T15:16:08.872828+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:40.222873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:14:40.222873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"72'8","reported_seq":587,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.877502+0000","last_change":"2024-04-14T15:15:40.617542+0000","last_active":"2024-04-14T15:16:08.877502+0000","last_peered":"2024-04-14T15:16:08.877502+0000","last_clean":"2024-04-14T15:16:08.877502+0000","last_became_active":"2024-04-14T15:09:47.044085+0000","last_became_peered":"2024-04-14T15:09:47.044085+0000","last_unstale":"2024-04-14T15:16:08.877502+0000","last_undegraded":"2024-04-14T15:16:08.877502+0000","last_fullsized":"2024-04-14T15:16:08.877502+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:15:40.617433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:15:40.617433+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:51.220120+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":489,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.870488+0000","last_change":"2024-04-14T15:14:15.629311+0000","last_active":"2024-04-14T15:16:08.870488+0000","last_peered":"2024-04-14T15:16:08.870488+0000","last_clean":"2024-04-14T15:16:08.870488+0000","last_became_active":"2024-04-14T15:09:49.065274+0000","last_became_peered":"2024-04-14T15:09:49.065274+0000","last_unstale":"2024-04-14T15:16:08.870488+0000","last_undegraded":"2024-04-14T15:16:08.870488+0000","last_fullsized":"2024-04-14T15:16:08.870488+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:15.629225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:14:15.629225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"91'130","reported_seq":385,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.876095+0000","last_change":"2024-04-14T15:15:45.589217+0000","last_active":"2024-04-14T15:16:08.876095+0000","last_peered":"2024-04-14T15:16:08.876095+0000","last_clean":"2024-04-14T15:16:08.876095+0000","last_became_active":"2024-04-14T15:15:44.887114+0000","last_became_peered":"2024-04-14T15:15:44.887114+0000","last_unstale":"2024-04-14T15:16:08.876095+0000","last_undegraded":"2024-04-14T15:16:08.876095+0000","last_fullsized":"2024-04-14T15:16:08.876095+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:15:45.589132+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:15:45.589132+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:13.065344+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":406,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.871980+0000","last_change":"2024-04-14T15:15:29.366895+0000","last_active":"2024-04-14T15:16:08.871980+0000","last_peered":"2024-04-14T15:16:08.871980+0000","last_clean":"2024-04-14T15:16:08.871980+0000","last_became_active":"2024-04-14T15:12:56.487615+0000","last_became_peered":"2024-04-14T15:12:56.487615+0000","last_unstale":"2024-04-14T15:16:08.871980+0000","last_undegraded":"2024-04-14T15:16:08.871980+0000","last_fullsized":"2024-04-14T15:16:08.871980+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:29.366820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:29.366820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:55.079805+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"72'15","reported_seq":530,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.875278+0000","last_change":"2024-04-14T15:14:58.216134+0000","last_active":"2024-04-14T15:16:08.875278+0000","last_peered":"2024-04-14T15:16:08.875278+0000","last_clean":"2024-04-14T15:16:08.875278+0000","last_became_active":"2024-04-14T15:09:45.033331+0000","last_became_peered":"2024-04-14T15:09:45.033331+0000","last_unstale":"2024-04-14T15:16:08.875278+0000","last_undegraded":"2024-04-14T15:16:08.875278+0000","last_fullsized":"2024-04-14T15:16:08.875278+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-04-14T15:14:58.215973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:14:58.215973+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":509,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.872073+0000","last_change":"2024-04-14T15:15:00.203399+0000","last_active":"2024-04-14T15:16:08.872073+0000","last_peered":"2024-04-14T15:16:08.872073+0000","last_clean":"2024-04-14T15:16:08.872073+0000","last_became_active":"2024-04-14T15:09:43.020878+0000","last_became_peered":"2024-04-14T15:09:43.020878+0000","last_unstale":"2024-04-14T15:16:08.872073+0000","last_undegraded":"2024-04-14T15:16:08.872073+0000","last_fullsized":"2024-04-14T15:16:08.872073+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:00.203310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:15:00.203310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:25.756347+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"72'8","reported_seq":587,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.870475+0000","last_change":"2024-04-14T15:16:06.933663+0000","last_active":"2024-04-14T15:16:08.870475+0000","last_peered":"2024-04-14T15:16:08.870475+0000","last_clean":"2024-04-14T15:16:08.870475+0000","last_became_active":"2024-04-14T15:09:47.050962+0000","last_became_peered":"2024-04-14T15:09:47.050962+0000","last_unstale":"2024-04-14T15:16:08.870475+0000","last_undegraded":"2024-04-14T15:16:08.870475+0000","last_fullsized":"2024-04-14T15:16:08.870475+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:16:06.933489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:06.933489+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:32.343513+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":61,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.874667+0000","last_change":"2024-04-14T15:15:44.880052+0000","last_active":"2024-04-14T15:16:08.874667+0000","last_peered":"2024-04-14T15:16:08.874667+0000","last_clean":"2024-04-14T15:16:08.874667+0000","last_became_active":"2024-04-14T15:15:44.879699+0000","last_became_peered":"2024-04-14T15:15:44.879699+0000","last_unstale":"2024-04-14T15:16:08.874667+0000","last_undegraded":"2024-04-14T15:16:08.874667+0000","last_fullsized":"2024-04-14T15:16:08.874667+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:00.789082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:14:00.789082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"233'8","reported_seq":420,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.875295+0000","last_change":"2024-04-14T15:15:01.295884+0000","last_active":"2024-04-14T15:16:08.875295+0000","last_peered":"2024-04-14T15:16:08.875295+0000","last_clean":"2024-04-14T15:16:08.875295+0000","last_became_active":"2024-04-14T15:12:56.381026+0000","last_became_peered":"2024-04-14T15:12:56.381026+0000","last_unstale":"2024-04-14T15:16:08.875295+0000","last_undegraded":"2024-04-14T15:16:08.875295+0000","last_fullsized":"2024-04-14T15:16:08.875295+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'8","last_scrub_stamp":"2024-04-14T15:15:01.295800+0000","last_deep_scrub":"233'8","last_deep_scrub_stamp":"2024-04-14T15:15:01.295800+0000","last_clean_scrub_stamp":"2024-04-14T15:15:01.295800+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:26.600713+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"85'16","reported_seq":529,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.870269+0000","last_change":"2024-04-14T15:14:41.852173+0000","last_active":"2024-04-14T15:16:08.870269+0000","last_peered":"2024-04-14T15:16:08.870269+0000","last_clean":"2024-04-14T15:16:08.870269+0000","last_became_active":"2024-04-14T15:09:45.028212+0000","last_became_peered":"2024-04-14T15:09:45.028212+0000","last_unstale":"2024-04-14T15:16:08.870269+0000","last_undegraded":"2024-04-14T15:16:08.870269+0000","last_fullsized":"2024-04-14T15:16:08.870269+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-14T15:14:41.851725+0000","last_deep_scrub":"85'16","last_deep_scrub_stamp":"2024-04-14T15:10:47.456169+0000","last_clean_scrub_stamp":"2024-04-14T15:14:41.851725+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":509,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.876500+0000","last_change":"2024-04-14T15:14:36.968608+0000","last_active":"2024-04-14T15:16:08.876500+0000","last_peered":"2024-04-14T15:16:08.876500+0000","last_clean":"2024-04-14T15:16:08.876500+0000","last_became_active":"2024-04-14T15:09:43.018601+0000","last_became_peered":"2024-04-14T15:09:43.018601+0000","last_unstale":"2024-04-14T15:16:08.876500+0000","last_undegraded":"2024-04-14T15:16:08.876500+0000","last_fullsized":"2024-04-14T15:16:08.876500+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:36.968530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:14:36.968530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":60,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.870115+0000","last_change":"2024-04-14T15:15:44.887157+0000","last_active":"2024-04-14T15:16:08.870115+0000","last_peered":"2024-04-14T15:16:08.870115+0000","last_clean":"2024-04-14T15:16:08.870115+0000","last_became_active":"2024-04-14T15:15:44.886834+0000","last_became_peered":"2024-04-14T15:15:44.886834+0000","last_unstale":"2024-04-14T15:16:08.870115+0000","last_undegraded":"2024-04-14T15:16:08.870115+0000","last_fullsized":"2024-04-14T15:16:08.870115+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:43.288325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:14:43.288325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":174,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.880523+0000","last_change":"2024-04-14T15:15:21.888865+0000","last_active":"2024-04-14T15:16:08.880523+0000","last_peered":"2024-04-14T15:16:08.880523+0000","last_clean":"2024-04-14T15:16:08.880523+0000","last_became_active":"2024-04-14T15:15:21.888587+0000","last_became_peered":"2024-04-14T15:15:21.888587+0000","last_unstale":"2024-04-14T15:16:08.880523+0000","last_undegraded":"2024-04-14T15:16:08.880523+0000","last_fullsized":"2024-04-14T15:16:08.880523+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:47.061794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:14:47.061794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"91'130","reported_seq":559,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.879140+0000","last_change":"2024-04-14T15:14:40.097282+0000","last_active":"2024-04-14T15:16:08.879140+0000","last_peered":"2024-04-14T15:16:08.879140+0000","last_clean":"2024-04-14T15:16:08.879140+0000","last_became_active":"2024-04-14T15:14:17.863503+0000","last_became_peered":"2024-04-14T15:14:17.863503+0000","last_unstale":"2024-04-14T15:16:08.879140+0000","last_undegraded":"2024-04-14T15:16:08.879140+0000","last_fullsized":"2024-04-14T15:16:08.879140+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:14:40.097172+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:14:40.097172+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":406,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.870165+0000","last_change":"2024-04-14T15:14:59.876005+0000","last_active":"2024-04-14T15:16:08.870165+0000","last_peered":"2024-04-14T15:16:08.870165+0000","last_clean":"2024-04-14T15:16:08.870165+0000","last_became_active":"2024-04-14T15:12:56.427276+0000","last_became_peered":"2024-04-14T15:12:56.427276+0000","last_unstale":"2024-04-14T15:16:08.870165+0000","last_undegraded":"2024-04-14T15:16:08.870165+0000","last_fullsized":"2024-04-14T15:16:08.870165+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:59.875917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:14:59.875917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:09.325316+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"255'10","reported_seq":225,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.877941+0000","last_change":"2024-04-14T15:15:21.874078+0000","last_active":"2024-04-14T15:16:08.877941+0000","last_peered":"2024-04-14T15:16:08.877941+0000","last_clean":"2024-04-14T15:16:08.877941+0000","last_became_active":"2024-04-14T15:15:21.873747+0000","last_became_peered":"2024-04-14T15:15:21.873747+0000","last_unstale":"2024-04-14T15:16:08.877941+0000","last_undegraded":"2024-04-14T15:16:08.877941+0000","last_fullsized":"2024-04-14T15:16:08.877941+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-04-14T15:13:38.414471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:13:38.414471+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"72'14","reported_seq":527,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.869700+0000","last_change":"2024-04-14T15:15:38.962791+0000","last_active":"2024-04-14T15:16:08.869700+0000","last_peered":"2024-04-14T15:16:08.869700+0000","last_clean":"2024-04-14T15:16:08.869700+0000","last_became_active":"2024-04-14T15:09:45.025673+0000","last_became_peered":"2024-04-14T15:09:45.025673+0000","last_unstale":"2024-04-14T15:16:08.869700+0000","last_undegraded":"2024-04-14T15:16:08.869700+0000","last_fullsized":"2024-04-14T15:16:08.869700+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'14","last_scrub_stamp":"2024-04-14T15:15:38.962680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:15:38.962680+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:54.305088+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":509,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.878744+0000","last_change":"2024-04-14T15:15:39.123305+0000","last_active":"2024-04-14T15:16:08.878744+0000","last_peered":"2024-04-14T15:16:08.878744+0000","last_clean":"2024-04-14T15:16:08.878744+0000","last_became_active":"2024-04-14T15:09:43.020503+0000","last_became_peered":"2024-04-14T15:09:43.020503+0000","last_unstale":"2024-04-14T15:16:08.878744+0000","last_undegraded":"2024-04-14T15:16:08.878744+0000","last_fullsized":"2024-04-14T15:16:08.878744+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:39.123227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:15:39.123227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:07.799320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":490,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.882379+0000","last_change":"2024-04-14T15:13:25.116624+0000","last_active":"2024-04-14T15:16:08.882379+0000","last_peered":"2024-04-14T15:16:08.882379+0000","last_clean":"2024-04-14T15:16:08.882379+0000","last_became_active":"2024-04-14T15:09:49.129610+0000","last_became_peered":"2024-04-14T15:09:49.129610+0000","last_unstale":"2024-04-14T15:16:08.882379+0000","last_undegraded":"2024-04-14T15:16:08.882379+0000","last_fullsized":"2024-04-14T15:16:08.882379+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:25.116541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:13:25.116541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"91'130","reported_seq":559,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.869130+0000","last_change":"2024-04-14T15:14:38.832437+0000","last_active":"2024-04-14T15:16:08.869130+0000","last_peered":"2024-04-14T15:16:08.869130+0000","last_clean":"2024-04-14T15:16:08.869130+0000","last_became_active":"2024-04-14T15:14:17.871922+0000","last_became_peered":"2024-04-14T15:14:17.871922+0000","last_unstale":"2024-04-14T15:16:08.869130+0000","last_undegraded":"2024-04-14T15:16:08.869130+0000","last_fullsized":"2024-04-14T15:16:08.869130+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:14:38.832326+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:14:38.832326+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":406,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.885067+0000","last_change":"2024-04-14T15:14:52.954282+0000","last_active":"2024-04-14T15:16:08.885067+0000","last_peered":"2024-04-14T15:16:08.885067+0000","last_clean":"2024-04-14T15:16:08.885067+0000","last_became_active":"2024-04-14T15:12:56.355254+0000","last_became_peered":"2024-04-14T15:12:56.355254+0000","last_unstale":"2024-04-14T15:16:08.885067+0000","last_undegraded":"2024-04-14T15:16:08.885067+0000","last_fullsized":"2024-04-14T15:16:08.885067+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:52.954163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:14:52.954163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":502,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.870501+0000","last_change":"2024-04-14T15:16:05.917909+0000","last_active":"2024-04-14T15:16:08.870501+0000","last_peered":"2024-04-14T15:16:08.870501+0000","last_clean":"2024-04-14T15:16:08.870501+0000","last_became_active":"2024-04-14T15:09:47.044610+0000","last_became_peered":"2024-04-14T15:09:47.044610+0000","last_unstale":"2024-04-14T15:16:08.870501+0000","last_undegraded":"2024-04-14T15:16:08.870501+0000","last_fullsized":"2024-04-14T15:16:08.870501+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:05.917804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:05.917804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:15.789677+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"72'18","reported_seq":215,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.877913+0000","last_change":"2024-04-14T15:15:21.880835+0000","last_active":"2024-04-14T15:16:08.877913+0000","last_peered":"2024-04-14T15:16:08.877913+0000","last_clean":"2024-04-14T15:16:08.877913+0000","last_became_active":"2024-04-14T15:15:21.880565+0000","last_became_peered":"2024-04-14T15:15:21.880565+0000","last_unstale":"2024-04-14T15:16:08.877913+0000","last_undegraded":"2024-04-14T15:16:08.877913+0000","last_fullsized":"2024-04-14T15:16:08.877913+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'18","last_scrub_stamp":"2024-04-14T15:13:18.170570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:13:18.170570+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'2","reported_seq":504,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.873285+0000","last_change":"2024-04-14T15:14:38.291600+0000","last_active":"2024-04-14T15:16:08.873285+0000","last_peered":"2024-04-14T15:16:08.873285+0000","last_clean":"2024-04-14T15:16:08.873285+0000","last_became_active":"2024-04-14T15:09:43.018478+0000","last_became_peered":"2024-04-14T15:09:43.018478+0000","last_unstale":"2024-04-14T15:16:08.873285+0000","last_undegraded":"2024-04-14T15:16:08.873285+0000","last_fullsized":"2024-04-14T15:16:08.873285+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'2","last_scrub_stamp":"2024-04-14T15:14:38.291510+0000","last_deep_scrub":"91'2","last_deep_scrub_stamp":"2024-04-14T15:11:37.974770+0000","last_clean_scrub_stamp":"2024-04-14T15:14:38.291510+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":490,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.869917+0000","last_change":"2024-04-14T15:13:26.563376+0000","last_active":"2024-04-14T15:16:08.869917+0000","last_peered":"2024-04-14T15:16:08.869917+0000","last_clean":"2024-04-14T15:16:08.869917+0000","last_became_active":"2024-04-14T15:09:49.065684+0000","last_became_peered":"2024-04-14T15:09:49.065684+0000","last_unstale":"2024-04-14T15:16:08.869917+0000","last_undegraded":"2024-04-14T15:16:08.869917+0000","last_fullsized":"2024-04-14T15:16:08.869917+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:26.563298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:13:26.563298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"91'130","reported_seq":566,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.878422+0000","last_change":"2024-04-14T15:14:39.146276+0000","last_active":"2024-04-14T15:16:08.878422+0000","last_peered":"2024-04-14T15:16:08.878422+0000","last_clean":"2024-04-14T15:16:08.878422+0000","last_became_active":"2024-04-14T15:14:17.853143+0000","last_became_peered":"2024-04-14T15:14:17.853143+0000","last_unstale":"2024-04-14T15:16:08.878422+0000","last_undegraded":"2024-04-14T15:16:08.878422+0000","last_fullsized":"2024-04-14T15:16:08.878422+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:14:39.146174+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:14:39.146174+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":406,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.874060+0000","last_change":"2024-04-14T15:15:31.403165+0000","last_active":"2024-04-14T15:16:08.874060+0000","last_peered":"2024-04-14T15:16:08.874060+0000","last_clean":"2024-04-14T15:16:08.874060+0000","last_became_active":"2024-04-14T15:12:56.402023+0000","last_became_peered":"2024-04-14T15:12:56.402023+0000","last_unstale":"2024-04-14T15:16:08.874060+0000","last_undegraded":"2024-04-14T15:16:08.874060+0000","last_fullsized":"2024-04-14T15:16:08.874060+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:31.403086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:31.403086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:40.387194+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":502,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.877607+0000","last_change":"2024-04-14T15:14:14.070145+0000","last_active":"2024-04-14T15:16:08.877607+0000","last_peered":"2024-04-14T15:16:08.877607+0000","last_clean":"2024-04-14T15:16:08.877607+0000","last_became_active":"2024-04-14T15:09:47.048427+0000","last_became_peered":"2024-04-14T15:09:47.048427+0000","last_unstale":"2024-04-14T15:16:08.877607+0000","last_undegraded":"2024-04-14T15:16:08.877607+0000","last_fullsized":"2024-04-14T15:16:08.877607+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:14.070046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:14:14.070046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"102'27","reported_seq":550,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.886729+0000","last_change":"2024-04-14T15:14:12.391631+0000","last_active":"2024-04-14T15:16:08.886729+0000","last_peered":"2024-04-14T15:16:08.886729+0000","last_clean":"2024-04-14T15:16:08.886729+0000","last_became_active":"2024-04-14T15:09:45.179182+0000","last_became_peered":"2024-04-14T15:09:45.179182+0000","last_unstale":"2024-04-14T15:16:08.886729+0000","last_undegraded":"2024-04-14T15:16:08.886729+0000","last_fullsized":"2024-04-14T15:16:08.886729+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'27","last_scrub_stamp":"2024-04-14T15:14:12.391537+0000","last_deep_scrub":"102'27","last_deep_scrub_stamp":"2024-04-14T15:12:30.928216+0000","last_clean_scrub_stamp":"2024-04-14T15:14:12.391537+0000","objects_scrubbed":8,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":502,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.882737+0000","last_change":"2024-04-14T15:13:16.122436+0000","last_active":"2024-04-14T15:16:08.882737+0000","last_peered":"2024-04-14T15:16:08.882737+0000","last_clean":"2024-04-14T15:16:08.882737+0000","last_became_active":"2024-04-14T15:09:43.021006+0000","last_became_peered":"2024-04-14T15:09:43.021006+0000","last_unstale":"2024-04-14T15:16:08.882737+0000","last_undegraded":"2024-04-14T15:16:08.882737+0000","last_fullsized":"2024-04-14T15:16:08.882737+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:16.122352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:13:16.122352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":490,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.874453+0000","last_change":"2024-04-14T15:14:06.454685+0000","last_active":"2024-04-14T15:16:08.874453+0000","last_peered":"2024-04-14T15:16:08.874453+0000","last_clean":"2024-04-14T15:16:08.874453+0000","last_became_active":"2024-04-14T15:09:49.128136+0000","last_became_peered":"2024-04-14T15:09:49.128136+0000","last_unstale":"2024-04-14T15:16:08.874453+0000","last_undegraded":"2024-04-14T15:16:08.874453+0000","last_fullsized":"2024-04-14T15:16:08.874453+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:06.454602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:14:06.454602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"91'130","reported_seq":559,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.871719+0000","last_change":"2024-04-14T15:14:42.213499+0000","last_active":"2024-04-14T15:16:08.871719+0000","last_peered":"2024-04-14T15:16:08.871719+0000","last_clean":"2024-04-14T15:16:08.871719+0000","last_became_active":"2024-04-14T15:14:17.856694+0000","last_became_peered":"2024-04-14T15:14:17.856694+0000","last_unstale":"2024-04-14T15:16:08.871719+0000","last_undegraded":"2024-04-14T15:16:08.871719+0000","last_fullsized":"2024-04-14T15:16:08.871719+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:14:42.213346+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:14:42.213346+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":114,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.878585+0000","last_change":"2024-04-14T15:15:31.812967+0000","last_active":"2024-04-14T15:16:08.878585+0000","last_peered":"2024-04-14T15:16:08.878585+0000","last_clean":"2024-04-14T15:16:08.878585+0000","last_became_active":"2024-04-14T15:15:21.887769+0000","last_became_peered":"2024-04-14T15:15:21.887769+0000","last_unstale":"2024-04-14T15:16:08.878585+0000","last_undegraded":"2024-04-14T15:16:08.878585+0000","last_fullsized":"2024-04-14T15:16:08.878585+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:31.812888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:31.812888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:40.709024+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":60,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.875605+0000","last_change":"2024-04-14T15:15:44.884167+0000","last_active":"2024-04-14T15:16:08.875605+0000","last_peered":"2024-04-14T15:16:08.875605+0000","last_clean":"2024-04-14T15:16:08.875605+0000","last_became_active":"2024-04-14T15:15:44.883694+0000","last_became_peered":"2024-04-14T15:15:44.883694+0000","last_unstale":"2024-04-14T15:16:08.875605+0000","last_undegraded":"2024-04-14T15:16:08.875605+0000","last_fullsized":"2024-04-14T15:16:08.875605+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:12.743355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:14:12.743355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"182'18","reported_seq":529,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.884271+0000","last_change":"2024-04-14T15:14:10.384002+0000","last_active":"2024-04-14T15:16:08.884271+0000","last_peered":"2024-04-14T15:16:08.884271+0000","last_clean":"2024-04-14T15:16:08.884271+0000","last_became_active":"2024-04-14T15:09:45.037231+0000","last_became_peered":"2024-04-14T15:09:45.037231+0000","last_unstale":"2024-04-14T15:16:08.884271+0000","last_undegraded":"2024-04-14T15:16:08.884271+0000","last_fullsized":"2024-04-14T15:16:08.884271+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'18","last_scrub_stamp":"2024-04-14T15:14:10.383906+0000","last_deep_scrub":"182'18","last_deep_scrub_stamp":"2024-04-14T15:14:10.383906+0000","last_clean_scrub_stamp":"2024-04-14T15:14:10.383906+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":510,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.876590+0000","last_change":"2024-04-14T15:14:44.864919+0000","last_active":"2024-04-14T15:16:08.876590+0000","last_peered":"2024-04-14T15:16:08.876590+0000","last_clean":"2024-04-14T15:16:08.876590+0000","last_became_active":"2024-04-14T15:09:43.025095+0000","last_became_peered":"2024-04-14T15:09:43.025095+0000","last_unstale":"2024-04-14T15:16:08.876590+0000","last_undegraded":"2024-04-14T15:16:08.876590+0000","last_fullsized":"2024-04-14T15:16:08.876590+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:44.864789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:14:44.864789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":517,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.885477+0000","last_change":"2024-04-14T15:14:41.295132+0000","last_active":"2024-04-14T15:16:08.885477+0000","last_peered":"2024-04-14T15:16:08.885477+0000","last_clean":"2024-04-14T15:16:08.885477+0000","last_became_active":"2024-04-14T15:09:43.018509+0000","last_became_peered":"2024-04-14T15:09:43.018509+0000","last_unstale":"2024-04-14T15:16:08.885477+0000","last_undegraded":"2024-04-14T15:16:08.885477+0000","last_fullsized":"2024-04-14T15:16:08.885477+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:41.295049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:14:41.295049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0018592680000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"72'1","reported_seq":491,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.874376+0000","last_change":"2024-04-14T15:14:05.500796+0000","last_active":"2024-04-14T15:16:08.874376+0000","last_peered":"2024-04-14T15:16:08.874376+0000","last_clean":"2024-04-14T15:16:08.874376+0000","last_became_active":"2024-04-14T15:09:49.065243+0000","last_became_peered":"2024-04-14T15:09:49.065243+0000","last_unstale":"2024-04-14T15:16:08.874376+0000","last_undegraded":"2024-04-14T15:16:08.874376+0000","last_fullsized":"2024-04-14T15:16:08.874376+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-04-14T15:14:05.500685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:14:05.500685+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":106,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.877873+0000","last_change":"2024-04-14T15:15:21.890855+0000","last_active":"2024-04-14T15:16:08.877873+0000","last_peered":"2024-04-14T15:16:08.877873+0000","last_clean":"2024-04-14T15:16:08.877873+0000","last_became_active":"2024-04-14T15:15:21.890115+0000","last_became_peered":"2024-04-14T15:15:21.890115+0000","last_unstale":"2024-04-14T15:16:08.877873+0000","last_undegraded":"2024-04-14T15:16:08.877873+0000","last_fullsized":"2024-04-14T15:16:08.877873+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:45.148342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:14:45.148342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:11.624351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":494,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.881166+0000","last_change":"2024-04-14T15:13:31.226741+0000","last_active":"2024-04-14T15:16:08.881166+0000","last_peered":"2024-04-14T15:16:08.881166+0000","last_clean":"2024-04-14T15:16:08.881166+0000","last_became_active":"2024-04-14T15:09:47.047201+0000","last_became_peered":"2024-04-14T15:09:47.047201+0000","last_unstale":"2024-04-14T15:16:08.881166+0000","last_undegraded":"2024-04-14T15:16:08.881166+0000","last_fullsized":"2024-04-14T15:16:08.881166+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:31.226659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:13:31.226659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"90'24","reported_seq":606,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.877866+0000","last_change":"2024-04-14T15:13:26.147772+0000","last_active":"2024-04-14T15:16:08.877866+0000","last_peered":"2024-04-14T15:16:08.877866+0000","last_clean":"2024-04-14T15:16:08.877866+0000","last_became_active":"2024-04-14T15:09:45.179978+0000","last_became_peered":"2024-04-14T15:09:45.179978+0000","last_unstale":"2024-04-14T15:16:08.877866+0000","last_undegraded":"2024-04-14T15:16:08.877866+0000","last_fullsized":"2024-04-14T15:16:08.877866+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'24","last_scrub_stamp":"2024-04-14T15:13:26.147631+0000","last_deep_scrub":"90'24","last_deep_scrub_stamp":"2024-04-14T15:11:25.120692+0000","last_clean_scrub_stamp":"2024-04-14T15:13:26.147631+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"91'130","reported_seq":771,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.876543+0000","last_change":"2024-04-14T15:14:13.215896+0000","last_active":"2024-04-14T15:16:08.876543+0000","last_peered":"2024-04-14T15:16:08.876543+0000","last_clean":"2024-04-14T15:16:08.876543+0000","last_became_active":"2024-04-14T15:14:12.858953+0000","last_became_peered":"2024-04-14T15:14:12.858953+0000","last_unstale":"2024-04-14T15:16:08.876543+0000","last_undegraded":"2024-04-14T15:16:08.876543+0000","last_fullsized":"2024-04-14T15:16:08.876543+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:14:13.215812+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:14:13.215812+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":60,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.877857+0000","last_change":"2024-04-14T15:15:44.885612+0000","last_active":"2024-04-14T15:16:08.877857+0000","last_peered":"2024-04-14T15:16:08.877857+0000","last_clean":"2024-04-14T15:16:08.877857+0000","last_became_active":"2024-04-14T15:15:44.885315+0000","last_became_peered":"2024-04-14T15:15:44.885315+0000","last_unstale":"2024-04-14T15:16:08.877857+0000","last_undegraded":"2024-04-14T15:16:08.877857+0000","last_fullsized":"2024-04-14T15:16:08.877857+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:06.676174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:47.627305+0000","last_clean_scrub_stamp":"2024-04-14T15:14:06.676174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"69'2","reported_seq":196,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.877971+0000","last_change":"2024-04-14T15:15:21.890189+0000","last_active":"2024-04-14T15:16:08.877971+0000","last_peered":"2024-04-14T15:16:08.877971+0000","last_clean":"2024-04-14T15:16:08.877971+0000","last_became_active":"2024-04-14T15:15:21.889851+0000","last_became_peered":"2024-04-14T15:15:21.889851+0000","last_unstale":"2024-04-14T15:16:08.877971+0000","last_undegraded":"2024-04-14T15:16:08.877971+0000","last_fullsized":"2024-04-14T15:16:08.877971+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-04-14T15:13:32.804904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:13:32.804904+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0058046169999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":61,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.879432+0000","last_change":"2024-04-14T15:15:44.888223+0000","last_active":"2024-04-14T15:16:08.879432+0000","last_peered":"2024-04-14T15:16:08.879432+0000","last_clean":"2024-04-14T15:16:08.879432+0000","last_became_active":"2024-04-14T15:15:44.887934+0000","last_became_peered":"2024-04-14T15:15:44.887934+0000","last_unstale":"2024-04-14T15:16:08.879432+0000","last_undegraded":"2024-04-14T15:16:08.879432+0000","last_fullsized":"2024-04-14T15:16:08.879432+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:57.756721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:13:57.756721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"91'130","reported_seq":778,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.879061+0000","last_change":"2024-04-14T15:14:40.906412+0000","last_active":"2024-04-14T15:16:08.879061+0000","last_peered":"2024-04-14T15:16:08.879061+0000","last_clean":"2024-04-14T15:16:08.879061+0000","last_became_active":"2024-04-14T15:14:17.858029+0000","last_became_peered":"2024-04-14T15:14:17.858029+0000","last_unstale":"2024-04-14T15:16:08.879061+0000","last_undegraded":"2024-04-14T15:16:08.879061+0000","last_fullsized":"2024-04-14T15:16:08.879061+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:14:40.906329+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:14:40.906329+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":106,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.877782+0000","last_change":"2024-04-14T15:15:21.890274+0000","last_active":"2024-04-14T15:16:08.877782+0000","last_peered":"2024-04-14T15:16:08.877782+0000","last_clean":"2024-04-14T15:16:08.877782+0000","last_became_active":"2024-04-14T15:15:21.889970+0000","last_became_peered":"2024-04-14T15:15:21.889970+0000","last_unstale":"2024-04-14T15:16:08.877782+0000","last_undegraded":"2024-04-14T15:16:08.877782+0000","last_fullsized":"2024-04-14T15:16:08.877782+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:54.156421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:14:54.156421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":501,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.879506+0000","last_change":"2024-04-14T15:14:45.465021+0000","last_active":"2024-04-14T15:16:08.879506+0000","last_peered":"2024-04-14T15:16:08.879506+0000","last_clean":"2024-04-14T15:16:08.879506+0000","last_became_active":"2024-04-14T15:09:47.049457+0000","last_became_peered":"2024-04-14T15:09:47.049457+0000","last_unstale":"2024-04-14T15:16:08.879506+0000","last_undegraded":"2024-04-14T15:16:08.879506+0000","last_fullsized":"2024-04-14T15:16:08.879506+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:45.464940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:14:45.464940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"72'13","reported_seq":524,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.875484+0000","last_change":"2024-04-14T15:13:28.340992+0000","last_active":"2024-04-14T15:16:08.875484+0000","last_peered":"2024-04-14T15:16:08.875484+0000","last_clean":"2024-04-14T15:16:08.875484+0000","last_became_active":"2024-04-14T15:09:45.033883+0000","last_became_peered":"2024-04-14T15:09:45.033883+0000","last_unstale":"2024-04-14T15:16:08.875484+0000","last_undegraded":"2024-04-14T15:16:08.875484+0000","last_fullsized":"2024-04-14T15:16:08.875484+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'13","last_scrub_stamp":"2024-04-14T15:13:28.340838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:13:28.340838+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":510,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.875716+0000","last_change":"2024-04-14T15:14:05.378718+0000","last_active":"2024-04-14T15:16:08.875716+0000","last_peered":"2024-04-14T15:16:08.875716+0000","last_clean":"2024-04-14T15:16:08.875716+0000","last_became_active":"2024-04-14T15:09:43.021188+0000","last_became_peered":"2024-04-14T15:09:43.021188+0000","last_unstale":"2024-04-14T15:16:08.875716+0000","last_undegraded":"2024-04-14T15:16:08.875716+0000","last_fullsized":"2024-04-14T15:16:08.875716+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:05.378634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:14:05.378634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"73'5","reported_seq":899,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.872162+0000","last_change":"2024-04-14T15:15:28.328150+0000","last_active":"2024-04-14T15:16:08.872162+0000","last_peered":"2024-04-14T15:16:08.872162+0000","last_clean":"2024-04-14T15:16:08.872162+0000","last_became_active":"2024-04-14T15:09:43.016544+0000","last_became_peered":"2024-04-14T15:09:43.016544+0000","last_unstale":"2024-04-14T15:16:08.872162+0000","last_undegraded":"2024-04-14T15:16:08.872162+0000","last_fullsized":"2024-04-14T15:16:08.872162+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-04-14T15:15:28.328050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:15:28.328050+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:35.661053+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00056542099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":379,"num_read_kb":374,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":490,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.882852+0000","last_change":"2024-04-14T15:13:17.120281+0000","last_active":"2024-04-14T15:16:08.882852+0000","last_peered":"2024-04-14T15:16:08.882852+0000","last_clean":"2024-04-14T15:16:08.882852+0000","last_became_active":"2024-04-14T15:09:49.130307+0000","last_became_peered":"2024-04-14T15:09:49.130307+0000","last_unstale":"2024-04-14T15:16:08.882852+0000","last_undegraded":"2024-04-14T15:16:08.882852+0000","last_fullsized":"2024-04-14T15:16:08.882852+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:17.120174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:13:17.120174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"91'130","reported_seq":391,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.883563+0000","last_change":"2024-04-14T15:15:45.362962+0000","last_active":"2024-04-14T15:16:08.883563+0000","last_peered":"2024-04-14T15:16:08.883563+0000","last_clean":"2024-04-14T15:16:08.883563+0000","last_became_active":"2024-04-14T15:15:44.884623+0000","last_became_peered":"2024-04-14T15:15:44.884623+0000","last_unstale":"2024-04-14T15:16:08.883563+0000","last_undegraded":"2024-04-14T15:16:08.883563+0000","last_fullsized":"2024-04-14T15:16:08.883563+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:15:45.362860+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:15:45.362860+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:54.355120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":114,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.881216+0000","last_change":"2024-04-14T15:15:32.839576+0000","last_active":"2024-04-14T15:16:08.881216+0000","last_peered":"2024-04-14T15:16:08.881216+0000","last_clean":"2024-04-14T15:16:08.881216+0000","last_became_active":"2024-04-14T15:15:21.883717+0000","last_became_peered":"2024-04-14T15:15:21.883717+0000","last_unstale":"2024-04-14T15:16:08.881216+0000","last_undegraded":"2024-04-14T15:16:08.881216+0000","last_fullsized":"2024-04-14T15:16:08.881216+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:32.839491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:32.839491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:59.110217+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":494,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.886180+0000","last_change":"2024-04-14T15:13:35.587741+0000","last_active":"2024-04-14T15:16:08.886180+0000","last_peered":"2024-04-14T15:16:08.886180+0000","last_clean":"2024-04-14T15:16:08.886180+0000","last_became_active":"2024-04-14T15:09:47.040185+0000","last_became_peered":"2024-04-14T15:09:47.040185+0000","last_unstale":"2024-04-14T15:16:08.886180+0000","last_undegraded":"2024-04-14T15:16:08.886180+0000","last_fullsized":"2024-04-14T15:16:08.886180+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:35.587624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.587624+0000","last_clean_scrub_stamp":"2024-04-14T15:13:35.587624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"72'28","reported_seq":624,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.874014+0000","last_change":"2024-04-14T15:13:35.350892+0000","last_active":"2024-04-14T15:16:08.874014+0000","last_peered":"2024-04-14T15:16:08.874014+0000","last_clean":"2024-04-14T15:16:08.874014+0000","last_became_active":"2024-04-14T15:09:45.034168+0000","last_became_peered":"2024-04-14T15:09:45.034168+0000","last_unstale":"2024-04-14T15:16:08.874014+0000","last_undegraded":"2024-04-14T15:16:08.874014+0000","last_fullsized":"2024-04-14T15:16:08.874014+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'28","last_scrub_stamp":"2024-04-14T15:13:35.350790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:13:35.350790+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":60,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.872626+0000","last_change":"2024-04-14T15:15:44.885595+0000","last_active":"2024-04-14T15:16:08.872626+0000","last_peered":"2024-04-14T15:16:08.872626+0000","last_clean":"2024-04-14T15:16:08.872626+0000","last_became_active":"2024-04-14T15:15:44.885315+0000","last_became_peered":"2024-04-14T15:15:44.885315+0000","last_unstale":"2024-04-14T15:16:08.872626+0000","last_undegraded":"2024-04-14T15:16:08.872626+0000","last_fullsized":"2024-04-14T15:16:08.872626+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:25.908435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:13:25.908435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":166,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.880648+0000","last_change":"2024-04-14T15:15:21.875075+0000","last_active":"2024-04-14T15:16:08.880648+0000","last_peered":"2024-04-14T15:16:08.880648+0000","last_clean":"2024-04-14T15:16:08.880648+0000","last_became_active":"2024-04-14T15:15:21.874742+0000","last_became_peered":"2024-04-14T15:15:21.874742+0000","last_unstale":"2024-04-14T15:16:08.880648+0000","last_undegraded":"2024-04-14T15:16:08.880648+0000","last_fullsized":"2024-04-14T15:16:08.880648+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:14.594652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:14:14.594652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"91'130","reported_seq":566,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.878505+0000","last_change":"2024-04-14T15:14:40.134739+0000","last_active":"2024-04-14T15:16:08.878505+0000","last_peered":"2024-04-14T15:16:08.878505+0000","last_clean":"2024-04-14T15:16:08.878505+0000","last_became_active":"2024-04-14T15:14:17.859910+0000","last_became_peered":"2024-04-14T15:14:17.859910+0000","last_unstale":"2024-04-14T15:16:08.878505+0000","last_undegraded":"2024-04-14T15:16:08.878505+0000","last_fullsized":"2024-04-14T15:16:08.878505+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:14:40.134645+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:14:40.134645+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":406,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.877583+0000","last_change":"2024-04-14T15:14:49.421861+0000","last_active":"2024-04-14T15:16:08.877583+0000","last_peered":"2024-04-14T15:16:08.877583+0000","last_clean":"2024-04-14T15:16:08.877583+0000","last_became_active":"2024-04-14T15:12:56.350703+0000","last_became_peered":"2024-04-14T15:12:56.350703+0000","last_unstale":"2024-04-14T15:16:08.877583+0000","last_undegraded":"2024-04-14T15:16:08.877583+0000","last_fullsized":"2024-04-14T15:16:08.877583+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:49.421768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:14:49.421768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"72'8","reported_seq":588,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.886451+0000","last_change":"2024-04-14T15:14:03.352331+0000","last_active":"2024-04-14T15:16:08.886451+0000","last_peered":"2024-04-14T15:16:08.886451+0000","last_clean":"2024-04-14T15:16:08.886451+0000","last_became_active":"2024-04-14T15:09:47.047699+0000","last_became_peered":"2024-04-14T15:09:47.047699+0000","last_unstale":"2024-04-14T15:16:08.886451+0000","last_undegraded":"2024-04-14T15:16:08.886451+0000","last_fullsized":"2024-04-14T15:16:08.886451+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:14:03.352195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.352195+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"72'10","reported_seq":499,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.876717+0000","last_change":"2024-04-14T15:15:41.610385+0000","last_active":"2024-04-14T15:16:08.876717+0000","last_peered":"2024-04-14T15:16:08.876717+0000","last_clean":"2024-04-14T15:16:08.876717+0000","last_became_active":"2024-04-14T15:10:07.764042+0000","last_became_peered":"2024-04-14T15:10:07.764042+0000","last_unstale":"2024-04-14T15:16:08.876717+0000","last_undegraded":"2024-04-14T15:16:08.876717+0000","last_fullsized":"2024-04-14T15:16:08.876717+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-04-14T15:15:41.610279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:15:41.610279+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:08.216750+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":502,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.873969+0000","last_change":"2024-04-14T15:14:14.462940+0000","last_active":"2024-04-14T15:16:08.873969+0000","last_peered":"2024-04-14T15:16:08.873969+0000","last_clean":"2024-04-14T15:16:08.873969+0000","last_became_active":"2024-04-14T15:09:43.022597+0000","last_became_peered":"2024-04-14T15:09:43.022597+0000","last_unstale":"2024-04-14T15:16:08.873969+0000","last_undegraded":"2024-04-14T15:16:08.873969+0000","last_fullsized":"2024-04-14T15:16:08.873969+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:14.462847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:14:14.462847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":490,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.880807+0000","last_change":"2024-04-14T15:13:22.363505+0000","last_active":"2024-04-14T15:16:08.880807+0000","last_peered":"2024-04-14T15:16:08.880807+0000","last_clean":"2024-04-14T15:16:08.880807+0000","last_became_active":"2024-04-14T15:09:49.066648+0000","last_became_peered":"2024-04-14T15:09:49.066648+0000","last_unstale":"2024-04-14T15:16:08.880807+0000","last_undegraded":"2024-04-14T15:16:08.880807+0000","last_fullsized":"2024-04-14T15:16:08.880807+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:22.363370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:13:22.363370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":406,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.884009+0000","last_change":"2024-04-14T15:15:36.403919+0000","last_active":"2024-04-14T15:16:08.884009+0000","last_peered":"2024-04-14T15:16:08.884009+0000","last_clean":"2024-04-14T15:16:08.884009+0000","last_became_active":"2024-04-14T15:12:56.563074+0000","last_became_peered":"2024-04-14T15:12:56.563074+0000","last_unstale":"2024-04-14T15:16:08.884009+0000","last_undegraded":"2024-04-14T15:16:08.884009+0000","last_fullsized":"2024-04-14T15:16:08.884009+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:36.403843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:36.403843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:36.624784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":501,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.875099+0000","last_change":"2024-04-14T15:15:33.526229+0000","last_active":"2024-04-14T15:16:08.875099+0000","last_peered":"2024-04-14T15:16:08.875099+0000","last_clean":"2024-04-14T15:16:08.875099+0000","last_became_active":"2024-04-14T15:09:47.049005+0000","last_became_peered":"2024-04-14T15:09:47.049005+0000","last_unstale":"2024-04-14T15:16:08.875099+0000","last_undegraded":"2024-04-14T15:16:08.875099+0000","last_fullsized":"2024-04-14T15:16:08.875099+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:33.526146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:15:33.526146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:53.685917+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"183'17","reported_seq":533,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.870451+0000","last_change":"2024-04-14T15:14:11.577765+0000","last_active":"2024-04-14T15:16:08.870451+0000","last_peered":"2024-04-14T15:16:08.870451+0000","last_clean":"2024-04-14T15:16:08.870451+0000","last_became_active":"2024-04-14T15:09:45.032892+0000","last_became_peered":"2024-04-14T15:09:45.032892+0000","last_unstale":"2024-04-14T15:16:08.870451+0000","last_undegraded":"2024-04-14T15:16:08.870451+0000","last_fullsized":"2024-04-14T15:16:08.870451+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'17","last_scrub_stamp":"2024-04-14T15:14:11.577574+0000","last_deep_scrub":"183'17","last_deep_scrub_stamp":"2024-04-14T15:14:11.577574+0000","last_clean_scrub_stamp":"2024-04-14T15:14:11.577574+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"65'3","reported_seq":519,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.870007+0000","last_change":"2024-04-14T15:13:32.537671+0000","last_active":"2024-04-14T15:16:08.870007+0000","last_peered":"2024-04-14T15:16:08.870007+0000","last_clean":"2024-04-14T15:16:08.870007+0000","last_became_active":"2024-04-14T15:09:43.020490+0000","last_became_peered":"2024-04-14T15:09:43.020490+0000","last_unstale":"2024-04-14T15:16:08.870007+0000","last_undegraded":"2024-04-14T15:16:08.870007+0000","last_fullsized":"2024-04-14T15:16:08.870007+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3","last_scrub_stamp":"2024-04-14T15:13:32.537579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:13:32.537579+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":528,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":166,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.881589+0000","last_change":"2024-04-14T15:15:21.878709+0000","last_active":"2024-04-14T15:16:08.881589+0000","last_peered":"2024-04-14T15:16:08.881589+0000","last_clean":"2024-04-14T15:16:08.881589+0000","last_became_active":"2024-04-14T15:15:21.876410+0000","last_became_peered":"2024-04-14T15:15:21.876410+0000","last_unstale":"2024-04-14T15:16:08.881589+0000","last_undegraded":"2024-04-14T15:16:08.881589+0000","last_fullsized":"2024-04-14T15:16:08.881589+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:14.171627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:14:14.171627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":406,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.885350+0000","last_change":"2024-04-14T15:15:34.382014+0000","last_active":"2024-04-14T15:16:08.885350+0000","last_peered":"2024-04-14T15:16:08.885350+0000","last_clean":"2024-04-14T15:16:08.885350+0000","last_became_active":"2024-04-14T15:12:56.551177+0000","last_became_peered":"2024-04-14T15:12:56.551177+0000","last_unstale":"2024-04-14T15:16:08.885350+0000","last_undegraded":"2024-04-14T15:16:08.885350+0000","last_fullsized":"2024-04-14T15:16:08.885350+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:34.381865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:34.381865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:42.325831+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"72'9","reported_seq":513,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.884367+0000","last_change":"2024-04-14T15:13:37.516095+0000","last_active":"2024-04-14T15:16:08.884367+0000","last_peered":"2024-04-14T15:16:08.884367+0000","last_clean":"2024-04-14T15:16:08.884367+0000","last_became_active":"2024-04-14T15:09:45.180375+0000","last_became_peered":"2024-04-14T15:09:45.180375+0000","last_unstale":"2024-04-14T15:16:08.884367+0000","last_undegraded":"2024-04-14T15:16:08.884367+0000","last_fullsized":"2024-04-14T15:16:08.884367+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-04-14T15:13:37.515984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:13:37.515984+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":509,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.875004+0000","last_change":"2024-04-14T15:14:11.340243+0000","last_active":"2024-04-14T15:16:08.875004+0000","last_peered":"2024-04-14T15:16:08.875004+0000","last_clean":"2024-04-14T15:16:08.875004+0000","last_became_active":"2024-04-14T15:09:43.016788+0000","last_became_peered":"2024-04-14T15:09:43.016788+0000","last_unstale":"2024-04-14T15:16:08.875004+0000","last_undegraded":"2024-04-14T15:16:08.875004+0000","last_fullsized":"2024-04-14T15:16:08.875004+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:11.340164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:14:11.340164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":186,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.879371+0000","last_change":"2024-04-14T15:15:42.769452+0000","last_active":"2024-04-14T15:16:08.879371+0000","last_peered":"2024-04-14T15:16:08.879371+0000","last_clean":"2024-04-14T15:16:08.879371+0000","last_became_active":"2024-04-14T15:15:21.873580+0000","last_became_peered":"2024-04-14T15:15:21.873580+0000","last_unstale":"2024-04-14T15:16:08.879371+0000","last_undegraded":"2024-04-14T15:16:08.879371+0000","last_fullsized":"2024-04-14T15:16:08.879371+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:42.769375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:15:42.769375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:50.977618+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":497,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.879598+0000","last_change":"2024-04-14T15:14:46.461336+0000","last_active":"2024-04-14T15:16:08.879598+0000","last_peered":"2024-04-14T15:16:08.879598+0000","last_clean":"2024-04-14T15:16:08.879598+0000","last_became_active":"2024-04-14T15:09:49.064296+0000","last_became_peered":"2024-04-14T15:09:49.064296+0000","last_unstale":"2024-04-14T15:16:08.879598+0000","last_undegraded":"2024-04-14T15:16:08.879598+0000","last_fullsized":"2024-04-14T15:16:08.879598+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:46.461257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:14:46.461257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":406,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.874802+0000","last_change":"2024-04-14T15:14:43.531950+0000","last_active":"2024-04-14T15:16:08.874802+0000","last_peered":"2024-04-14T15:16:08.874802+0000","last_clean":"2024-04-14T15:16:08.874802+0000","last_became_active":"2024-04-14T15:12:56.340066+0000","last_became_peered":"2024-04-14T15:12:56.340066+0000","last_unstale":"2024-04-14T15:16:08.874802+0000","last_undegraded":"2024-04-14T15:16:08.874802+0000","last_fullsized":"2024-04-14T15:16:08.874802+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:43.531866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:14:43.531866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:09.892772+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.17","version":"72'6","reported_seq":514,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.879267+0000","last_change":"2024-04-14T15:14:05.104367+0000","last_active":"2024-04-14T15:16:08.879267+0000","last_peered":"2024-04-14T15:16:08.879267+0000","last_clean":"2024-04-14T15:16:08.879267+0000","last_became_active":"2024-04-14T15:09:45.180037+0000","last_became_peered":"2024-04-14T15:09:45.180037+0000","last_unstale":"2024-04-14T15:16:08.879267+0000","last_undegraded":"2024-04-14T15:16:08.879267+0000","last_fullsized":"2024-04-14T15:16:08.879267+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'6","last_scrub_stamp":"2024-04-14T15:14:05.104204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:14:05.104204+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":509,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.875482+0000","last_change":"2024-04-14T15:14:06.048273+0000","last_active":"2024-04-14T15:16:08.875482+0000","last_peered":"2024-04-14T15:16:08.875482+0000","last_clean":"2024-04-14T15:16:08.875482+0000","last_became_active":"2024-04-14T15:09:43.018987+0000","last_became_peered":"2024-04-14T15:09:43.018987+0000","last_unstale":"2024-04-14T15:16:08.875482+0000","last_undegraded":"2024-04-14T15:16:08.875482+0000","last_fullsized":"2024-04-14T15:16:08.875482+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:06.048196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:14:06.048196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":60,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.879295+0000","last_change":"2024-04-14T15:15:44.890140+0000","last_active":"2024-04-14T15:16:08.879295+0000","last_peered":"2024-04-14T15:16:08.879295+0000","last_clean":"2024-04-14T15:16:08.879295+0000","last_became_active":"2024-04-14T15:15:44.889786+0000","last_became_peered":"2024-04-14T15:15:44.889786+0000","last_unstale":"2024-04-14T15:16:08.879295+0000","last_undegraded":"2024-04-14T15:16:08.879295+0000","last_fullsized":"2024-04-14T15:16:08.879295+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:13.735024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:13.735024+0000","last_clean_scrub_stamp":"2024-04-14T15:14:13.735024+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":489,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.883309+0000","last_change":"2024-04-14T15:13:36.173906+0000","last_active":"2024-04-14T15:16:08.883309+0000","last_peered":"2024-04-14T15:16:08.883309+0000","last_clean":"2024-04-14T15:16:08.883309+0000","last_became_active":"2024-04-14T15:09:49.130438+0000","last_became_peered":"2024-04-14T15:09:49.130438+0000","last_unstale":"2024-04-14T15:16:08.883309+0000","last_undegraded":"2024-04-14T15:16:08.883309+0000","last_fullsized":"2024-04-14T15:16:08.883309+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:36.173813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:13:36.173813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":406,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.869203+0000","last_change":"2024-04-14T15:15:24.902253+0000","last_active":"2024-04-14T15:16:08.869203+0000","last_peered":"2024-04-14T15:16:08.869203+0000","last_clean":"2024-04-14T15:16:08.869203+0000","last_became_active":"2024-04-14T15:12:56.203577+0000","last_became_peered":"2024-04-14T15:12:56.203577+0000","last_unstale":"2024-04-14T15:16:08.869203+0000","last_undegraded":"2024-04-14T15:16:08.869203+0000","last_fullsized":"2024-04-14T15:16:08.869203+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:24.902134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:24.902134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:39.833144+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"72'10","reported_seq":205,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.880630+0000","last_change":"2024-04-14T15:15:30.806041+0000","last_active":"2024-04-14T15:16:08.880630+0000","last_peered":"2024-04-14T15:16:08.880630+0000","last_clean":"2024-04-14T15:16:08.880630+0000","last_became_active":"2024-04-14T15:15:21.877667+0000","last_became_peered":"2024-04-14T15:15:21.877667+0000","last_unstale":"2024-04-14T15:16:08.880630+0000","last_undegraded":"2024-04-14T15:16:08.880630+0000","last_fullsized":"2024-04-14T15:16:08.880630+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-04-14T15:15:30.805923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:15:30.805923+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:31.513697+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":60,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.878289+0000","last_change":"2024-04-14T15:15:44.888478+0000","last_active":"2024-04-14T15:16:08.878289+0000","last_peered":"2024-04-14T15:16:08.878289+0000","last_clean":"2024-04-14T15:16:08.878289+0000","last_became_active":"2024-04-14T15:15:44.887452+0000","last_became_peered":"2024-04-14T15:15:44.887452+0000","last_unstale":"2024-04-14T15:16:08.878289+0000","last_undegraded":"2024-04-14T15:16:08.878289+0000","last_fullsized":"2024-04-14T15:16:08.878289+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:33.770408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:13:33.770408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"280'9","reported_seq":90,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.874886+0000","last_change":"2024-04-14T15:15:44.883512+0000","last_active":"2024-04-14T15:16:08.874886+0000","last_peered":"2024-04-14T15:16:08.874886+0000","last_clean":"2024-04-14T15:16:08.874886+0000","last_became_active":"2024-04-14T15:15:44.882916+0000","last_became_peered":"2024-04-14T15:15:44.882916+0000","last_unstale":"2024-04-14T15:16:08.874886+0000","last_undegraded":"2024-04-14T15:16:08.874886+0000","last_fullsized":"2024-04-14T15:16:08.874886+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:14:37.162603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:14:37.162603+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":489,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.884169+0000","last_change":"2024-04-14T15:14:00.319415+0000","last_active":"2024-04-14T15:16:08.884169+0000","last_peered":"2024-04-14T15:16:08.884169+0000","last_clean":"2024-04-14T15:16:08.884169+0000","last_became_active":"2024-04-14T15:09:49.066068+0000","last_became_peered":"2024-04-14T15:09:49.066068+0000","last_unstale":"2024-04-14T15:16:08.884169+0000","last_undegraded":"2024-04-14T15:16:08.884169+0000","last_fullsized":"2024-04-14T15:16:08.884169+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:00.319334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:14:00.319334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":60,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.877715+0000","last_change":"2024-04-14T15:15:44.889470+0000","last_active":"2024-04-14T15:16:08.877715+0000","last_peered":"2024-04-14T15:16:08.877715+0000","last_clean":"2024-04-14T15:16:08.877715+0000","last_became_active":"2024-04-14T15:15:44.889106+0000","last_became_peered":"2024-04-14T15:15:44.889106+0000","last_unstale":"2024-04-14T15:16:08.877715+0000","last_undegraded":"2024-04-14T15:16:08.877715+0000","last_fullsized":"2024-04-14T15:16:08.877715+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:01.356001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:01.356001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:09.002602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"183'13","reported_seq":339,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.882953+0000","last_change":"2024-04-14T15:15:44.884579+0000","last_active":"2024-04-14T15:16:08.882953+0000","last_peered":"2024-04-14T15:16:08.882953+0000","last_clean":"2024-04-14T15:16:08.882953+0000","last_became_active":"2024-04-14T15:15:44.884211+0000","last_became_peered":"2024-04-14T15:15:44.884211+0000","last_unstale":"2024-04-14T15:16:08.882953+0000","last_undegraded":"2024-04-14T15:16:08.882953+0000","last_fullsized":"2024-04-14T15:16:08.882953+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'13","last_scrub_stamp":"2024-04-14T15:14:11.755252+0000","last_deep_scrub":"183'13","last_deep_scrub_stamp":"2024-04-14T15:14:11.755252+0000","last_clean_scrub_stamp":"2024-04-14T15:14:11.755252+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":509,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.886637+0000","last_change":"2024-04-14T15:14:11.348546+0000","last_active":"2024-04-14T15:16:08.886637+0000","last_peered":"2024-04-14T15:16:08.886637+0000","last_clean":"2024-04-14T15:16:08.886637+0000","last_became_active":"2024-04-14T15:09:43.019301+0000","last_became_peered":"2024-04-14T15:09:43.019301+0000","last_unstale":"2024-04-14T15:16:08.886637+0000","last_undegraded":"2024-04-14T15:16:08.886637+0000","last_fullsized":"2024-04-14T15:16:08.886637+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:11.348478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:14:11.348478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"255'10","reported_seq":227,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.880910+0000","last_change":"2024-04-14T15:15:21.884815+0000","last_active":"2024-04-14T15:16:08.880910+0000","last_peered":"2024-04-14T15:16:08.880910+0000","last_clean":"2024-04-14T15:16:08.880910+0000","last_became_active":"2024-04-14T15:15:21.884468+0000","last_became_peered":"2024-04-14T15:15:21.884468+0000","last_unstale":"2024-04-14T15:16:08.880910+0000","last_undegraded":"2024-04-14T15:16:08.880910+0000","last_fullsized":"2024-04-14T15:16:08.880910+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-04-14T15:14:12.169628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:14:12.169628+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":497,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.870335+0000","last_change":"2024-04-14T15:16:03.948252+0000","last_active":"2024-04-14T15:16:08.870335+0000","last_peered":"2024-04-14T15:16:08.870335+0000","last_clean":"2024-04-14T15:16:08.870335+0000","last_became_active":"2024-04-14T15:09:49.065242+0000","last_became_peered":"2024-04-14T15:09:49.065242+0000","last_unstale":"2024-04-14T15:16:08.870335+0000","last_undegraded":"2024-04-14T15:16:08.870335+0000","last_fullsized":"2024-04-14T15:16:08.870335+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:03.948140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:03.948140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:17.120471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":406,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.883416+0000","last_change":"2024-04-14T15:15:34.555810+0000","last_active":"2024-04-14T15:16:08.883416+0000","last_peered":"2024-04-14T15:16:08.883416+0000","last_clean":"2024-04-14T15:16:08.883416+0000","last_became_active":"2024-04-14T15:12:56.389204+0000","last_became_peered":"2024-04-14T15:12:56.389204+0000","last_unstale":"2024-04-14T15:16:08.883416+0000","last_undegraded":"2024-04-14T15:16:08.883416+0000","last_fullsized":"2024-04-14T15:16:08.883416+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:34.555728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:34.555728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:02.062574+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"72'9","reported_seq":513,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.872917+0000","last_change":"2024-04-14T15:13:24.378688+0000","last_active":"2024-04-14T15:16:08.872917+0000","last_peered":"2024-04-14T15:16:08.872917+0000","last_clean":"2024-04-14T15:16:08.872917+0000","last_became_active":"2024-04-14T15:09:45.026363+0000","last_became_peered":"2024-04-14T15:09:45.026363+0000","last_unstale":"2024-04-14T15:16:08.872917+0000","last_undegraded":"2024-04-14T15:16:08.872917+0000","last_fullsized":"2024-04-14T15:16:08.872917+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-04-14T15:13:24.378555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:13:24.378555+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":509,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.878021+0000","last_change":"2024-04-14T15:14:08.010682+0000","last_active":"2024-04-14T15:16:08.878021+0000","last_peered":"2024-04-14T15:16:08.878021+0000","last_clean":"2024-04-14T15:16:08.878021+0000","last_became_active":"2024-04-14T15:09:43.022480+0000","last_became_peered":"2024-04-14T15:09:43.022480+0000","last_unstale":"2024-04-14T15:16:08.878021+0000","last_undegraded":"2024-04-14T15:16:08.878021+0000","last_fullsized":"2024-04-14T15:16:08.878021+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:08.010603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:14:08.010603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":178,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.881416+0000","last_change":"2024-04-14T15:15:21.885176+0000","last_active":"2024-04-14T15:16:08.881416+0000","last_peered":"2024-04-14T15:16:08.881416+0000","last_clean":"2024-04-14T15:16:08.881416+0000","last_became_active":"2024-04-14T15:15:21.882643+0000","last_became_peered":"2024-04-14T15:15:21.882643+0000","last_unstale":"2024-04-14T15:16:08.881416+0000","last_undegraded":"2024-04-14T15:16:08.881416+0000","last_fullsized":"2024-04-14T15:16:08.881416+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:54.252608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:14:54.252608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:18.110931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":497,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.880710+0000","last_change":"2024-04-14T15:14:53.219773+0000","last_active":"2024-04-14T15:16:08.880710+0000","last_peered":"2024-04-14T15:16:08.880710+0000","last_clean":"2024-04-14T15:16:08.880710+0000","last_became_active":"2024-04-14T15:09:49.059711+0000","last_became_peered":"2024-04-14T15:09:49.059711+0000","last_unstale":"2024-04-14T15:16:08.880710+0000","last_undegraded":"2024-04-14T15:16:08.880710+0000","last_fullsized":"2024-04-14T15:16:08.880710+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:53.219686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:14:53.219686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":406,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.877884+0000","last_change":"2024-04-14T15:15:27.330424+0000","last_active":"2024-04-14T15:16:08.877884+0000","last_peered":"2024-04-14T15:16:08.877884+0000","last_clean":"2024-04-14T15:16:08.877884+0000","last_became_active":"2024-04-14T15:12:56.396294+0000","last_became_peered":"2024-04-14T15:12:56.396294+0000","last_unstale":"2024-04-14T15:16:08.877884+0000","last_undegraded":"2024-04-14T15:16:08.877884+0000","last_fullsized":"2024-04-14T15:16:08.877884+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:27.330328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:27.330328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:40.634678+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"72'11","reported_seq":524,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.917153+0000","last_change":"2024-04-14T15:16:08.917185+0000","last_active":"2024-04-14T15:16:08.917153+0000","last_peered":"2024-04-14T15:16:08.917153+0000","last_clean":"2024-04-14T15:16:08.917153+0000","last_became_active":"2024-04-14T15:09:45.026065+0000","last_became_peered":"2024-04-14T15:09:45.026065+0000","last_unstale":"2024-04-14T15:16:08.917153+0000","last_undegraded":"2024-04-14T15:16:08.917153+0000","last_fullsized":"2024-04-14T15:16:08.917153+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-04-14T15:16:08.917092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:16:08.917092+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:29.947559+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":501,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.875069+0000","last_change":"2024-04-14T15:13:27.378656+0000","last_active":"2024-04-14T15:16:08.875069+0000","last_peered":"2024-04-14T15:16:08.875069+0000","last_clean":"2024-04-14T15:16:08.875069+0000","last_became_active":"2024-04-14T15:09:43.023583+0000","last_became_peered":"2024-04-14T15:09:43.023583+0000","last_unstale":"2024-04-14T15:16:08.875069+0000","last_undegraded":"2024-04-14T15:16:08.875069+0000","last_fullsized":"2024-04-14T15:16:08.875069+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:27.378503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:13:27.378503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":493,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.872971+0000","last_change":"2024-04-14T15:14:08.486305+0000","last_active":"2024-04-14T15:16:08.872971+0000","last_peered":"2024-04-14T15:16:08.872971+0000","last_clean":"2024-04-14T15:16:08.872971+0000","last_became_active":"2024-04-14T15:09:47.041683+0000","last_became_peered":"2024-04-14T15:09:47.041683+0000","last_unstale":"2024-04-14T15:16:08.872971+0000","last_undegraded":"2024-04-14T15:16:08.872971+0000","last_fullsized":"2024-04-14T15:16:08.872971+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:08.486217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:08.486217+0000","last_clean_scrub_stamp":"2024-04-14T15:14:08.486217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":174,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.878249+0000","last_change":"2024-04-14T15:15:21.890353+0000","last_active":"2024-04-14T15:16:08.878249+0000","last_peered":"2024-04-14T15:16:08.878249+0000","last_clean":"2024-04-14T15:16:08.878249+0000","last_became_active":"2024-04-14T15:15:21.889567+0000","last_became_peered":"2024-04-14T15:15:21.889567+0000","last_unstale":"2024-04-14T15:16:08.878249+0000","last_undegraded":"2024-04-14T15:16:08.878249+0000","last_fullsized":"2024-04-14T15:16:08.878249+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:34.547654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:13:34.547654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":406,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.878571+0000","last_change":"2024-04-14T15:14:44.506694+0000","last_active":"2024-04-14T15:16:08.878571+0000","last_peered":"2024-04-14T15:16:08.878571+0000","last_clean":"2024-04-14T15:16:08.878571+0000","last_became_active":"2024-04-14T15:12:56.387723+0000","last_became_peered":"2024-04-14T15:12:56.387723+0000","last_unstale":"2024-04-14T15:16:08.878571+0000","last_undegraded":"2024-04-14T15:16:08.878571+0000","last_fullsized":"2024-04-14T15:16:08.878571+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:44.506614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:14:44.506614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.10","version":"72'4","reported_seq":511,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.877677+0000","last_change":"2024-04-14T15:14:45.052235+0000","last_active":"2024-04-14T15:16:08.877677+0000","last_peered":"2024-04-14T15:16:08.877677+0000","last_clean":"2024-04-14T15:16:08.877677+0000","last_became_active":"2024-04-14T15:09:45.038815+0000","last_became_peered":"2024-04-14T15:09:45.038815+0000","last_unstale":"2024-04-14T15:16:08.877677+0000","last_undegraded":"2024-04-14T15:16:08.877677+0000","last_fullsized":"2024-04-14T15:16:08.877677+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4","last_scrub_stamp":"2024-04-14T15:14:45.052074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:14:45.052074+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":60,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.878609+0000","last_change":"2024-04-14T15:15:44.885024+0000","last_active":"2024-04-14T15:16:08.878609+0000","last_peered":"2024-04-14T15:16:08.878609+0000","last_clean":"2024-04-14T15:16:08.878609+0000","last_became_active":"2024-04-14T15:15:44.884730+0000","last_became_peered":"2024-04-14T15:15:44.884730+0000","last_unstale":"2024-04-14T15:16:08.878609+0000","last_undegraded":"2024-04-14T15:16:08.878609+0000","last_fullsized":"2024-04-14T15:16:08.878609+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:57.260939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:08.712289+0000","last_clean_scrub_stamp":"2024-04-14T15:14:57.260939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":501,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.875146+0000","last_change":"2024-04-14T15:13:37.910269+0000","last_active":"2024-04-14T15:16:08.875146+0000","last_peered":"2024-04-14T15:16:08.875146+0000","last_clean":"2024-04-14T15:16:08.875146+0000","last_became_active":"2024-04-14T15:09:47.041074+0000","last_became_peered":"2024-04-14T15:09:47.041074+0000","last_unstale":"2024-04-14T15:16:08.875146+0000","last_undegraded":"2024-04-14T15:16:08.875146+0000","last_fullsized":"2024-04-14T15:16:08.875146+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:37.910180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:13:37.910180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":489,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.879705+0000","last_change":"2024-04-14T15:13:26.142140+0000","last_active":"2024-04-14T15:16:08.879705+0000","last_peered":"2024-04-14T15:16:08.879705+0000","last_clean":"2024-04-14T15:16:08.879705+0000","last_became_active":"2024-04-14T15:09:49.061359+0000","last_became_peered":"2024-04-14T15:09:49.061359+0000","last_unstale":"2024-04-14T15:16:08.879705+0000","last_undegraded":"2024-04-14T15:16:08.879705+0000","last_fullsized":"2024-04-14T15:16:08.879705+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:26.142060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:13:26.142060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":406,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.884730+0000","last_change":"2024-04-14T15:15:43.680936+0000","last_active":"2024-04-14T15:16:08.884730+0000","last_peered":"2024-04-14T15:16:08.884730+0000","last_clean":"2024-04-14T15:16:08.884730+0000","last_became_active":"2024-04-14T15:12:56.352528+0000","last_became_peered":"2024-04-14T15:12:56.352528+0000","last_unstale":"2024-04-14T15:16:08.884730+0000","last_undegraded":"2024-04-14T15:16:08.884730+0000","last_fullsized":"2024-04-14T15:16:08.884730+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:43.680852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:15:43.680852+0000","last_clean_scrub_stamp":"2024-04-14T15:15:43.680852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:51.024108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"102'16","reported_seq":167,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.879299+0000","last_change":"2024-04-14T15:15:21.886770+0000","last_active":"2024-04-14T15:16:08.879299+0000","last_peered":"2024-04-14T15:16:08.879299+0000","last_clean":"2024-04-14T15:16:08.879299+0000","last_became_active":"2024-04-14T15:15:21.884783+0000","last_became_peered":"2024-04-14T15:15:21.884783+0000","last_unstale":"2024-04-14T15:16:08.879299+0000","last_undegraded":"2024-04-14T15:16:08.879299+0000","last_fullsized":"2024-04-14T15:16:08.879299+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'16","last_scrub_stamp":"2024-04-14T15:14:10.211163+0000","last_deep_scrub":"102'16","last_deep_scrub_stamp":"2024-04-14T15:12:35.999326+0000","last_clean_scrub_stamp":"2024-04-14T15:14:10.211163+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":509,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.878659+0000","last_change":"2024-04-14T15:14:15.075691+0000","last_active":"2024-04-14T15:16:08.878659+0000","last_peered":"2024-04-14T15:16:08.878659+0000","last_clean":"2024-04-14T15:16:08.878659+0000","last_became_active":"2024-04-14T15:09:43.023220+0000","last_became_peered":"2024-04-14T15:09:43.023220+0000","last_unstale":"2024-04-14T15:16:08.878659+0000","last_undegraded":"2024-04-14T15:16:08.878659+0000","last_fullsized":"2024-04-14T15:16:08.878659+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:15.075612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:14:15.075612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":493,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.876265+0000","last_change":"2024-04-14T15:13:29.219729+0000","last_active":"2024-04-14T15:16:08.876265+0000","last_peered":"2024-04-14T15:16:08.876265+0000","last_clean":"2024-04-14T15:16:08.876265+0000","last_became_active":"2024-04-14T15:09:47.044665+0000","last_became_peered":"2024-04-14T15:09:47.044665+0000","last_unstale":"2024-04-14T15:16:08.876265+0000","last_undegraded":"2024-04-14T15:16:08.876265+0000","last_fullsized":"2024-04-14T15:16:08.876265+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:29.219589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:13:29.219589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":182,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.881679+0000","last_change":"2024-04-14T15:16:05.885527+0000","last_active":"2024-04-14T15:16:08.881679+0000","last_peered":"2024-04-14T15:16:08.881679+0000","last_clean":"2024-04-14T15:16:08.881679+0000","last_became_active":"2024-04-14T15:15:21.883940+0000","last_became_peered":"2024-04-14T15:15:21.883940+0000","last_unstale":"2024-04-14T15:16:08.881679+0000","last_undegraded":"2024-04-14T15:16:08.881679+0000","last_fullsized":"2024-04-14T15:16:08.881679+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:05.885438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:05.885438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:27.063364+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":406,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.875481+0000","last_change":"2024-04-14T15:15:28.474035+0000","last_active":"2024-04-14T15:16:08.875481+0000","last_peered":"2024-04-14T15:16:08.875481+0000","last_clean":"2024-04-14T15:16:08.875481+0000","last_became_active":"2024-04-14T15:12:56.335230+0000","last_became_peered":"2024-04-14T15:12:56.335230+0000","last_unstale":"2024-04-14T15:16:08.875481+0000","last_undegraded":"2024-04-14T15:16:08.875481+0000","last_fullsized":"2024-04-14T15:16:08.875481+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:28.473899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:28.473899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:55.001475+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"72'10","reported_seq":513,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.886297+0000","last_change":"2024-04-14T15:13:17.411319+0000","last_active":"2024-04-14T15:16:08.886297+0000","last_peered":"2024-04-14T15:16:08.886297+0000","last_clean":"2024-04-14T15:16:08.886297+0000","last_became_active":"2024-04-14T15:09:45.181456+0000","last_became_peered":"2024-04-14T15:09:45.181456+0000","last_unstale":"2024-04-14T15:16:08.886297+0000","last_undegraded":"2024-04-14T15:16:08.886297+0000","last_fullsized":"2024-04-14T15:16:08.886297+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-04-14T15:13:17.411175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:13:17.411175+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":60,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.870171+0000","last_change":"2024-04-14T15:15:44.878646+0000","last_active":"2024-04-14T15:16:08.870171+0000","last_peered":"2024-04-14T15:16:08.870171+0000","last_clean":"2024-04-14T15:16:08.870171+0000","last_became_active":"2024-04-14T15:15:44.878336+0000","last_became_peered":"2024-04-14T15:15:44.878336+0000","last_unstale":"2024-04-14T15:16:08.870171+0000","last_undegraded":"2024-04-14T15:16:08.870171+0000","last_fullsized":"2024-04-14T15:16:08.870171+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:07.680508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:14:07.680508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"72'8","reported_seq":591,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.878937+0000","last_change":"2024-04-14T15:13:28.103421+0000","last_active":"2024-04-14T15:16:08.878937+0000","last_peered":"2024-04-14T15:16:08.878937+0000","last_clean":"2024-04-14T15:16:08.878937+0000","last_became_active":"2024-04-14T15:09:47.048216+0000","last_became_peered":"2024-04-14T15:09:47.048216+0000","last_unstale":"2024-04-14T15:16:08.878937+0000","last_undegraded":"2024-04-14T15:16:08.878937+0000","last_fullsized":"2024-04-14T15:16:08.878937+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:13:28.103336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:13:28.103336+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"72'1","reported_seq":498,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.881571+0000","last_change":"2024-04-14T15:14:37.932503+0000","last_active":"2024-04-14T15:16:08.881571+0000","last_peered":"2024-04-14T15:16:08.881571+0000","last_clean":"2024-04-14T15:16:08.881571+0000","last_became_active":"2024-04-14T15:09:49.062051+0000","last_became_peered":"2024-04-14T15:09:49.062051+0000","last_unstale":"2024-04-14T15:16:08.881571+0000","last_undegraded":"2024-04-14T15:16:08.881571+0000","last_fullsized":"2024-04-14T15:16:08.881571+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-04-14T15:14:37.932420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:14:37.932420+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":398,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.880896+0000","last_change":"2024-04-14T15:12:56.548050+0000","last_active":"2024-04-14T15:16:08.880896+0000","last_peered":"2024-04-14T15:16:08.880896+0000","last_clean":"2024-04-14T15:16:08.880896+0000","last_became_active":"2024-04-14T15:12:56.543793+0000","last_became_peered":"2024-04-14T15:12:56.543793+0000","last_unstale":"2024-04-14T15:16:08.880896+0000","last_undegraded":"2024-04-14T15:16:08.880896+0000","last_fullsized":"2024-04-14T15:16:08.880896+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:12:55.034316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"72'11","reported_seq":543,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.879040+0000","last_change":"2024-04-14T15:16:06.886451+0000","last_active":"2024-04-14T15:16:08.879040+0000","last_peered":"2024-04-14T15:16:08.879040+0000","last_clean":"2024-04-14T15:16:08.879040+0000","last_became_active":"2024-04-14T15:10:07.764183+0000","last_became_peered":"2024-04-14T15:10:07.764183+0000","last_unstale":"2024-04-14T15:16:08.879040+0000","last_undegraded":"2024-04-14T15:16:08.879040+0000","last_fullsized":"2024-04-14T15:16:08.879040+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-04-14T15:16:06.886325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:16:06.886325+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:30.942226+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"65'2","reported_seq":193,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.878480+0000","last_change":"2024-04-14T15:15:21.887031+0000","last_active":"2024-04-14T15:16:08.878480+0000","last_peered":"2024-04-14T15:16:08.878480+0000","last_clean":"2024-04-14T15:16:08.878480+0000","last_became_active":"2024-04-14T15:15:21.885343+0000","last_became_peered":"2024-04-14T15:15:21.885343+0000","last_unstale":"2024-04-14T15:16:08.878480+0000","last_undegraded":"2024-04-14T15:16:08.878480+0000","last_fullsized":"2024-04-14T15:16:08.878480+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-04-14T15:14:01.114004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:14:01.114004+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":494,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.886409+0000","last_change":"2024-04-14T15:13:28.464940+0000","last_active":"2024-04-14T15:16:08.886409+0000","last_peered":"2024-04-14T15:16:08.886409+0000","last_clean":"2024-04-14T15:16:08.886409+0000","last_became_active":"2024-04-14T15:09:47.045730+0000","last_became_peered":"2024-04-14T15:09:47.045730+0000","last_unstale":"2024-04-14T15:16:08.886409+0000","last_undegraded":"2024-04-14T15:16:08.886409+0000","last_fullsized":"2024-04-14T15:16:08.886409+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:28.464811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:13:28.464811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":489,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.874750+0000","last_change":"2024-04-14T15:14:03.551214+0000","last_active":"2024-04-14T15:16:08.874750+0000","last_peered":"2024-04-14T15:16:08.874750+0000","last_clean":"2024-04-14T15:16:08.874750+0000","last_became_active":"2024-04-14T15:09:49.057545+0000","last_became_peered":"2024-04-14T15:09:49.057545+0000","last_unstale":"2024-04-14T15:16:08.874750+0000","last_undegraded":"2024-04-14T15:16:08.874750+0000","last_fullsized":"2024-04-14T15:16:08.874750+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.551112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:21.247695+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.551112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":60,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.869139+0000","last_change":"2024-04-14T15:15:44.882920+0000","last_active":"2024-04-14T15:16:08.869139+0000","last_peered":"2024-04-14T15:16:08.869139+0000","last_clean":"2024-04-14T15:16:08.869139+0000","last_became_active":"2024-04-14T15:15:44.882604+0000","last_became_peered":"2024-04-14T15:15:44.882604+0000","last_unstale":"2024-04-14T15:16:08.869139+0000","last_undegraded":"2024-04-14T15:16:08.869139+0000","last_fullsized":"2024-04-14T15:16:08.869139+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:28.483617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:28.483617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:29.345071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"72'15","reported_seq":530,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.879718+0000","last_change":"2024-04-14T15:15:44.609525+0000","last_active":"2024-04-14T15:16:08.879718+0000","last_peered":"2024-04-14T15:16:08.879718+0000","last_clean":"2024-04-14T15:16:08.879718+0000","last_became_active":"2024-04-14T15:09:45.032887+0000","last_became_peered":"2024-04-14T15:09:45.032887+0000","last_unstale":"2024-04-14T15:16:08.879718+0000","last_undegraded":"2024-04-14T15:16:08.879718+0000","last_fullsized":"2024-04-14T15:16:08.879718+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-04-14T15:15:44.609332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:15:44.609332+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:57.512618+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":504,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.877521+0000","last_change":"2024-04-14T15:13:15.102843+0000","last_active":"2024-04-14T15:16:08.877521+0000","last_peered":"2024-04-14T15:16:08.877521+0000","last_clean":"2024-04-14T15:16:08.877521+0000","last_became_active":"2024-04-14T15:09:43.021494+0000","last_became_peered":"2024-04-14T15:09:43.021494+0000","last_unstale":"2024-04-14T15:16:08.877521+0000","last_undegraded":"2024-04-14T15:16:08.877521+0000","last_fullsized":"2024-04-14T15:16:08.877521+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:15.102763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:13:15.102763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":501,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.874977+0000","last_change":"2024-04-14T15:14:56.199840+0000","last_active":"2024-04-14T15:16:08.874977+0000","last_peered":"2024-04-14T15:16:08.874977+0000","last_clean":"2024-04-14T15:16:08.874977+0000","last_became_active":"2024-04-14T15:09:47.042733+0000","last_became_peered":"2024-04-14T15:09:47.042733+0000","last_unstale":"2024-04-14T15:16:08.874977+0000","last_undegraded":"2024-04-14T15:16:08.874977+0000","last_fullsized":"2024-04-14T15:16:08.874977+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:56.199756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:14:56.199756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:16.563165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":498,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.870402+0000","last_change":"2024-04-14T15:15:31.874372+0000","last_active":"2024-04-14T15:16:08.870402+0000","last_peered":"2024-04-14T15:16:08.870402+0000","last_clean":"2024-04-14T15:16:08.870402+0000","last_became_active":"2024-04-14T15:09:49.127979+0000","last_became_peered":"2024-04-14T15:09:49.127979+0000","last_unstale":"2024-04-14T15:16:08.870402+0000","last_undegraded":"2024-04-14T15:16:08.870402+0000","last_fullsized":"2024-04-14T15:16:08.870402+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:31.874232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:15:31.874232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:53.658126+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":406,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.879259+0000","last_change":"2024-04-14T15:15:31.635164+0000","last_active":"2024-04-14T15:16:08.879259+0000","last_peered":"2024-04-14T15:16:08.879259+0000","last_clean":"2024-04-14T15:16:08.879259+0000","last_became_active":"2024-04-14T15:12:56.392352+0000","last_became_peered":"2024-04-14T15:12:56.392352+0000","last_unstale":"2024-04-14T15:16:08.879259+0000","last_undegraded":"2024-04-14T15:16:08.879259+0000","last_fullsized":"2024-04-14T15:16:08.879259+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:31.635060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:31.635060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:37.237891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"72'12","reported_seq":205,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.879034+0000","last_change":"2024-04-14T15:15:21.878502+0000","last_active":"2024-04-14T15:16:08.879034+0000","last_peered":"2024-04-14T15:16:08.879034+0000","last_clean":"2024-04-14T15:16:08.879034+0000","last_became_active":"2024-04-14T15:15:21.877838+0000","last_became_peered":"2024-04-14T15:15:21.877838+0000","last_unstale":"2024-04-14T15:16:08.879034+0000","last_undegraded":"2024-04-14T15:16:08.879034+0000","last_fullsized":"2024-04-14T15:16:08.879034+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'12","last_scrub_stamp":"2024-04-14T15:13:18.392496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:13:18.392496+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":186,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.881506+0000","last_change":"2024-04-14T15:15:21.886300+0000","last_active":"2024-04-14T15:16:08.881506+0000","last_peered":"2024-04-14T15:16:08.881506+0000","last_clean":"2024-04-14T15:16:08.881506+0000","last_became_active":"2024-04-14T15:15:21.883544+0000","last_became_peered":"2024-04-14T15:15:21.883544+0000","last_unstale":"2024-04-14T15:16:08.881506+0000","last_undegraded":"2024-04-14T15:16:08.881506+0000","last_fullsized":"2024-04-14T15:16:08.881506+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:34.177820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:13:34.177820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":501,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.869233+0000","last_change":"2024-04-14T15:14:56.868446+0000","last_active":"2024-04-14T15:16:08.869233+0000","last_peered":"2024-04-14T15:16:08.869233+0000","last_clean":"2024-04-14T15:16:08.869233+0000","last_became_active":"2024-04-14T15:09:47.048045+0000","last_became_peered":"2024-04-14T15:09:47.048045+0000","last_unstale":"2024-04-14T15:16:08.869233+0000","last_undegraded":"2024-04-14T15:16:08.869233+0000","last_fullsized":"2024-04-14T15:16:08.869233+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:56.868328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:14:56.868328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:19.257648+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":183,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.879165+0000","last_change":"2024-04-14T15:16:07.879394+0000","last_active":"2024-04-14T15:16:08.879165+0000","last_peered":"2024-04-14T15:16:08.879165+0000","last_clean":"2024-04-14T15:16:08.879165+0000","last_became_active":"2024-04-14T15:15:21.889691+0000","last_became_peered":"2024-04-14T15:15:21.889691+0000","last_unstale":"2024-04-14T15:16:08.879165+0000","last_undegraded":"2024-04-14T15:16:08.879165+0000","last_fullsized":"2024-04-14T15:16:08.879165+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:07.879280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:07.879280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:23.294021+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":406,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.877605+0000","last_change":"2024-04-14T15:14:42.527202+0000","last_active":"2024-04-14T15:16:08.877605+0000","last_peered":"2024-04-14T15:16:08.877605+0000","last_clean":"2024-04-14T15:16:08.877605+0000","last_became_active":"2024-04-14T15:12:56.375278+0000","last_became_peered":"2024-04-14T15:12:56.375278+0000","last_unstale":"2024-04-14T15:16:08.877605+0000","last_undegraded":"2024-04-14T15:16:08.877605+0000","last_fullsized":"2024-04-14T15:16:08.877605+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:42.527099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:14:42.527099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:11.706762+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"276'13","reported_seq":525,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.869429+0000","last_change":"2024-04-14T15:15:43.969860+0000","last_active":"2024-04-14T15:16:08.869429+0000","last_peered":"2024-04-14T15:16:08.869429+0000","last_clean":"2024-04-14T15:16:08.869429+0000","last_became_active":"2024-04-14T15:09:45.030093+0000","last_became_peered":"2024-04-14T15:09:45.030093+0000","last_unstale":"2024-04-14T15:16:08.869429+0000","last_undegraded":"2024-04-14T15:16:08.869429+0000","last_fullsized":"2024-04-14T15:16:08.869429+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'13","last_scrub_stamp":"2024-04-14T15:15:43.969775+0000","last_deep_scrub":"276'13","last_deep_scrub_stamp":"2024-04-14T15:15:43.969775+0000","last_clean_scrub_stamp":"2024-04-14T15:15:43.969775+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:52.603875+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":207,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.880461+0000","last_change":"2024-04-14T15:14:30.858015+0000","last_active":"2024-04-14T15:16:08.880461+0000","last_peered":"2024-04-14T15:16:08.880461+0000","last_clean":"2024-04-14T15:16:08.880461+0000","last_became_active":"2024-04-14T15:14:30.857727+0000","last_became_peered":"2024-04-14T15:14:30.857727+0000","last_unstale":"2024-04-14T15:16:08.880461+0000","last_undegraded":"2024-04-14T15:16:08.880461+0000","last_fullsized":"2024-04-14T15:16:08.880461+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:05.518321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:14:05.518321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":60,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.886829+0000","last_change":"2024-04-14T15:15:44.883445+0000","last_active":"2024-04-14T15:16:08.886829+0000","last_peered":"2024-04-14T15:16:08.886829+0000","last_clean":"2024-04-14T15:16:08.886829+0000","last_became_active":"2024-04-14T15:15:44.883121+0000","last_became_peered":"2024-04-14T15:15:44.883121+0000","last_unstale":"2024-04-14T15:16:08.886829+0000","last_undegraded":"2024-04-14T15:16:08.886829+0000","last_fullsized":"2024-04-14T15:16:08.886829+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:10.728836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:14:10.728836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":497,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.881267+0000","last_change":"2024-04-14T15:14:04.346959+0000","last_active":"2024-04-14T15:16:08.881267+0000","last_peered":"2024-04-14T15:16:08.881267+0000","last_clean":"2024-04-14T15:16:08.881267+0000","last_became_active":"2024-04-14T15:09:49.061756+0000","last_became_peered":"2024-04-14T15:09:49.061756+0000","last_unstale":"2024-04-14T15:16:08.881267+0000","last_undegraded":"2024-04-14T15:16:08.881267+0000","last_fullsized":"2024-04-14T15:16:08.881267+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:04.346872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:14:04.346872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":406,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.878162+0000","last_change":"2024-04-14T15:14:00.109792+0000","last_active":"2024-04-14T15:16:08.878162+0000","last_peered":"2024-04-14T15:16:08.878162+0000","last_clean":"2024-04-14T15:16:08.878162+0000","last_became_active":"2024-04-14T15:12:56.232831+0000","last_became_peered":"2024-04-14T15:12:56.232831+0000","last_unstale":"2024-04-14T15:16:08.878162+0000","last_undegraded":"2024-04-14T15:16:08.878162+0000","last_fullsized":"2024-04-14T15:16:08.878162+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:00.109696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:14:00.109696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1b","version":"87'7","reported_seq":517,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.869860+0000","last_change":"2024-04-14T15:15:44.927867+0000","last_active":"2024-04-14T15:16:08.869860+0000","last_peered":"2024-04-14T15:16:08.869860+0000","last_clean":"2024-04-14T15:16:08.869860+0000","last_became_active":"2024-04-14T15:09:45.028667+0000","last_became_peered":"2024-04-14T15:09:45.028667+0000","last_unstale":"2024-04-14T15:16:08.869860+0000","last_undegraded":"2024-04-14T15:16:08.869860+0000","last_fullsized":"2024-04-14T15:16:08.869860+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'7","last_scrub_stamp":"2024-04-14T15:15:44.927776+0000","last_deep_scrub":"87'7","last_deep_scrub_stamp":"2024-04-14T15:11:10.510785+0000","last_clean_scrub_stamp":"2024-04-14T15:15:44.927776+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:09.960055+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":509,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.883080+0000","last_change":"2024-04-14T15:14:08.254685+0000","last_active":"2024-04-14T15:16:08.883080+0000","last_peered":"2024-04-14T15:16:08.883080+0000","last_clean":"2024-04-14T15:16:08.883080+0000","last_became_active":"2024-04-14T15:09:43.021986+0000","last_became_peered":"2024-04-14T15:09:43.021986+0000","last_unstale":"2024-04-14T15:16:08.883080+0000","last_undegraded":"2024-04-14T15:16:08.883080+0000","last_fullsized":"2024-04-14T15:16:08.883080+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:08.254601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:08.254601+0000","last_clean_scrub_stamp":"2024-04-14T15:14:08.254601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":493,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.879631+0000","last_change":"2024-04-14T15:13:19.196559+0000","last_active":"2024-04-14T15:16:08.879631+0000","last_peered":"2024-04-14T15:16:08.879631+0000","last_clean":"2024-04-14T15:16:08.879631+0000","last_became_active":"2024-04-14T15:09:47.046517+0000","last_became_peered":"2024-04-14T15:09:47.046517+0000","last_unstale":"2024-04-14T15:16:08.879631+0000","last_undegraded":"2024-04-14T15:16:08.879631+0000","last_fullsized":"2024-04-14T15:16:08.879631+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:19.196454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:13:19.196454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":61,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.874571+0000","last_change":"2024-04-14T15:15:44.887620+0000","last_active":"2024-04-14T15:16:08.874571+0000","last_peered":"2024-04-14T15:16:08.874571+0000","last_clean":"2024-04-14T15:16:08.874571+0000","last_became_active":"2024-04-14T15:15:44.887227+0000","last_became_peered":"2024-04-14T15:15:44.887227+0000","last_unstale":"2024-04-14T15:16:08.874571+0000","last_undegraded":"2024-04-14T15:16:08.874571+0000","last_fullsized":"2024-04-14T15:16:08.874571+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:56.268086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:56.268086+0000","last_clean_scrub_stamp":"2024-04-14T15:14:56.268086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:14.092981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":406,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.884981+0000","last_change":"2024-04-14T15:15:37.629813+0000","last_active":"2024-04-14T15:16:08.884981+0000","last_peered":"2024-04-14T15:16:08.884981+0000","last_clean":"2024-04-14T15:16:08.884981+0000","last_became_active":"2024-04-14T15:12:56.189117+0000","last_became_peered":"2024-04-14T15:12:56.189117+0000","last_unstale":"2024-04-14T15:16:08.884981+0000","last_undegraded":"2024-04-14T15:16:08.884981+0000","last_fullsized":"2024-04-14T15:16:08.884981+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:37.629718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:37.629718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:48.883517+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.18","version":"72'9","reported_seq":513,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.870070+0000","last_change":"2024-04-14T15:13:24.538800+0000","last_active":"2024-04-14T15:16:08.870070+0000","last_peered":"2024-04-14T15:16:08.870070+0000","last_clean":"2024-04-14T15:16:08.870070+0000","last_became_active":"2024-04-14T15:09:45.032079+0000","last_became_peered":"2024-04-14T15:09:45.032079+0000","last_unstale":"2024-04-14T15:16:08.870070+0000","last_undegraded":"2024-04-14T15:16:08.870070+0000","last_fullsized":"2024-04-14T15:16:08.870070+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-04-14T15:13:24.538684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:13:24.538684+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":187,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.880558+0000","last_change":"2024-04-14T15:15:21.892804+0000","last_active":"2024-04-14T15:16:08.880558+0000","last_peered":"2024-04-14T15:16:08.880558+0000","last_clean":"2024-04-14T15:16:08.880558+0000","last_became_active":"2024-04-14T15:15:21.891926+0000","last_became_peered":"2024-04-14T15:15:21.891926+0000","last_unstale":"2024-04-14T15:16:08.880558+0000","last_undegraded":"2024-04-14T15:16:08.880558+0000","last_fullsized":"2024-04-14T15:16:08.880558+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:59.366623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:13:59.366623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":493,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.871956+0000","last_change":"2024-04-14T15:13:25.370811+0000","last_active":"2024-04-14T15:16:08.871956+0000","last_peered":"2024-04-14T15:16:08.871956+0000","last_clean":"2024-04-14T15:16:08.871956+0000","last_became_active":"2024-04-14T15:09:47.041486+0000","last_became_peered":"2024-04-14T15:09:47.041486+0000","last_unstale":"2024-04-14T15:16:08.871956+0000","last_undegraded":"2024-04-14T15:16:08.871956+0000","last_fullsized":"2024-04-14T15:16:08.871956+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:25.370679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:13:25.370679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":489,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.869203+0000","last_change":"2024-04-14T15:14:37.812472+0000","last_active":"2024-04-14T15:16:08.869203+0000","last_peered":"2024-04-14T15:16:08.869203+0000","last_clean":"2024-04-14T15:16:08.869203+0000","last_became_active":"2024-04-14T15:09:49.058904+0000","last_became_peered":"2024-04-14T15:09:49.058904+0000","last_unstale":"2024-04-14T15:16:08.869203+0000","last_undegraded":"2024-04-14T15:16:08.869203+0000","last_fullsized":"2024-04-14T15:16:08.869203+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:37.812396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:14:37.812396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":60,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.884611+0000","last_change":"2024-04-14T15:15:44.889891+0000","last_active":"2024-04-14T15:16:08.884611+0000","last_peered":"2024-04-14T15:16:08.884611+0000","last_clean":"2024-04-14T15:16:08.884611+0000","last_became_active":"2024-04-14T15:15:44.889605+0000","last_became_peered":"2024-04-14T15:15:44.889605+0000","last_unstale":"2024-04-14T15:16:08.884611+0000","last_undegraded":"2024-04-14T15:16:08.884611+0000","last_fullsized":"2024-04-14T15:16:08.884611+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:00.363827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:00.363827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:09.050267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"72'15","reported_seq":207,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.879186+0000","last_change":"2024-04-14T15:15:21.886521+0000","last_active":"2024-04-14T15:16:08.879186+0000","last_peered":"2024-04-14T15:16:08.879186+0000","last_clean":"2024-04-14T15:16:08.879186+0000","last_became_active":"2024-04-14T15:15:21.884310+0000","last_became_peered":"2024-04-14T15:15:21.884310+0000","last_unstale":"2024-04-14T15:16:08.879186+0000","last_undegraded":"2024-04-14T15:16:08.879186+0000","last_fullsized":"2024-04-14T15:16:08.879186+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-04-14T15:13:31.248315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:13:31.248315+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":502,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.881558+0000","last_change":"2024-04-14T15:13:18.431688+0000","last_active":"2024-04-14T15:16:08.881558+0000","last_peered":"2024-04-14T15:16:08.881558+0000","last_clean":"2024-04-14T15:16:08.881558+0000","last_became_active":"2024-04-14T15:09:43.017351+0000","last_became_peered":"2024-04-14T15:09:43.017351+0000","last_unstale":"2024-04-14T15:16:08.881558+0000","last_undegraded":"2024-04-14T15:16:08.881558+0000","last_fullsized":"2024-04-14T15:16:08.881558+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:18.431571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:13:18.431571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":493,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.870239+0000","last_change":"2024-04-14T15:14:08.512221+0000","last_active":"2024-04-14T15:16:08.870239+0000","last_peered":"2024-04-14T15:16:08.870239+0000","last_clean":"2024-04-14T15:16:08.870239+0000","last_became_active":"2024-04-14T15:09:47.042538+0000","last_became_peered":"2024-04-14T15:09:47.042538+0000","last_unstale":"2024-04-14T15:16:08.870239+0000","last_undegraded":"2024-04-14T15:16:08.870239+0000","last_fullsized":"2024-04-14T15:16:08.870239+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:08.512130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:08.512130+0000","last_clean_scrub_stamp":"2024-04-14T15:14:08.512130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":497,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-14T15:16:08.881123+0000","last_change":"2024-04-14T15:14:38.023392+0000","last_active":"2024-04-14T15:16:08.881123+0000","last_peered":"2024-04-14T15:16:08.881123+0000","last_clean":"2024-04-14T15:16:08.881123+0000","last_became_active":"2024-04-14T15:09:49.129636+0000","last_became_peered":"2024-04-14T15:09:49.129636+0000","last_unstale":"2024-04-14T15:16:08.881123+0000","last_undegraded":"2024-04-14T15:16:08.881123+0000","last_fullsized":"2024-04-14T15:16:08.881123+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:38.023315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:14:38.023315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":271,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":270,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":7},{"poolid":110,"num_pg":32,"stat_sum":{"num_bytes":4195508,"num_objects":122,"num_object_clones":2,"num_object_copies":366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":122,"num_whiteouts":0,"num_read":98,"num_read_kb":4,"num_write":545,"num_write_kb":8730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14057472,"data_stored":12586524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1800,"ondisk_log_size":1800,"up":96,"acting":96,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1430,"ondisk_log_size":1430,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":379,"num_read_kb":374,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":696,"num_read_kb":453,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":454,"ondisk_log_size":454,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70,"ondisk_log_size":70,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103251,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":58924,"kb_used_data":9664,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93686228,"statfs":{"total":95995035648,"available":95934697472,"internally_reserved":0,"allocated":9895936,"data_stored":7461453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397590,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":76700,"kb_used_data":5232,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93668452,"statfs":{"total":95995035648,"available":95916494848,"internally_reserved":0,"allocated":5357568,"data_stored":2939735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574960,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":80888,"kb_used_data":9432,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93664264,"statfs":{"total":95995035648,"available":95912206336,"internally_reserved":0,"allocated":9658368,"data_stored":7134439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738485,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":72568,"kb_used_data":5284,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93672584,"statfs":{"total":95995035648,"available":95920726016,"internally_reserved":0,"allocated":5410816,"data_stored":2940170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411436,"num_pgs":88,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":68672,"kb_used_data":5548,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93676480,"statfs":{"total":95995035648,"available":95924715520,"internally_reserved":0,"allocated":5681152,"data_stored":3201826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182502,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":63392,"kb_used_data":5812,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93681760,"statfs":{"total":95995035648,"available":95930122240,"internally_reserved":0,"allocated":5951488,"data_stored":3539333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986272,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":76716,"kb_used_data":5260,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93668436,"statfs":{"total":95995035648,"available":95916478464,"internally_reserved":0,"allocated":5386240,"data_stored":2948231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659229,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":85512,"kb_used_data":9896,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93659640,"statfs":{"total":95995035648,"available":95907471360,"internally_reserved":0,"allocated":10133504,"data_stored":7723340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4427776,"data_stored":4194882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":172032,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4358144,"data_stored":4194699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4374528,"data_stored":4194740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":270,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":270,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":270,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":270,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":270,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":270,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":270,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-14T15:16:16.086 INFO:tasks.thrashosds.thrasher:Setting 110.b to [7, 2, 6, 5, 3, 0] 2024-04-14T15:16:16.086 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '110.b', '7', '2', '6', '5', '3', '0'] 2024-04-14T15:16:16.086 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd pg-upmap-items 110.b 7 2 6 5 3 0 2024-04-14T15:16:16.098 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:16:16.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:16.153 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:16.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:16:16.215 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-14T15:16:16.254 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:16:16.280 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:16.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:16.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:16:16.382 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:16:16.407 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:16.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:16.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:16:16.454 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:16:16.509 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:16:16.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:16.534 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:16.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:16:16.636 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:16:16.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:16.661 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:16.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:16:16.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.698+0000 7f1c71be1700 1 -- 172.21.15.90:0/1727309107 >> v1:172.21.15.90:6789/0 conn(0x7f1c6c10a740 legacy=0x7f1c6c10cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:16.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.698+0000 7f1c71be1700 1 -- 172.21.15.90:0/1727309107 shutdown_connections 2024-04-14T15:16:16.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.698+0000 7f1c71be1700 1 -- 172.21.15.90:0/1727309107 >> 172.21.15.90:0/1727309107 conn(0x7f1c6c0fd290 msgr2=0x7f1c6c0ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:16:16.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.698+0000 7f1c71be1700 1 -- 172.21.15.90:0/1727309107 shutdown_connections 2024-04-14T15:16:16.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.698+0000 7f1c71be1700 1 -- 172.21.15.90:0/1727309107 wait complete. 2024-04-14T15:16:16.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.699+0000 7f1c71be1700 1 librados: starting msgr at 2024-04-14T15:16:16.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.699+0000 7f1c71be1700 1 librados: starting objecter 2024-04-14T15:16:16.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.699+0000 7f1c71be1700 1 Processor -- start 2024-04-14T15:16:16.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.699+0000 7f1c71be1700 1 -- start start 2024-04-14T15:16:16.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.699+0000 7f1c71be1700 1 librados: setting wanted keys 2024-04-14T15:16:16.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.699+0000 7f1c71be1700 1 librados: calling monclient init 2024-04-14T15:16:16.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.699+0000 7f1c71be1700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1c6c1a7f20 con 0x7f1c6c10a740 2024-04-14T15:16:16.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.699+0000 7f1c71be1700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1c6c1a9100 con 0x7f1c6c10e310 2024-04-14T15:16:16.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.699+0000 7f1c71be1700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1c6c1aa2e0 con 0x7f1c6c107890 2024-04-14T15:16:16.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.700+0000 7f1c6b7fe700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f1c6c107890 0x7f1c6c10bff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:54704/0 (socket says 172.21.15.90:54704) 2024-04-14T15:16:16.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.700+0000 7f1c6b7fe700 1 -- 172.21.15.90:0/3322422450 learned_addr learned my addr 172.21.15.90:0/3322422450 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:16:16.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.700+0000 7f1c6affd700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f1c6c10a740 0x7f1c6c10c700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:52008/0 (socket says 172.21.15.90:52008) 2024-04-14T15:16:16.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.700+0000 7f1c68ff9700 10 client.?.objecter ms_handle_connect 0x7f1c6c10e310 2024-04-14T15:16:16.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.700+0000 7f1c68ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:16:16.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.700+0000 7f1c68ff9700 10 client.?.objecter ms_handle_connect 0x7f1c6c10a740 2024-04-14T15:16:16.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.700+0000 7f1c68ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:16:16.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.700+0000 7f1c68ff9700 10 client.?.objecter ms_handle_connect 0x7f1c6c107890 2024-04-14T15:16:16.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.700+0000 7f1c68ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:16:16.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.700+0000 7f1c68ff9700 1 -- 172.21.15.90:0/3322422450 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1225146959 0 0) 0x7f1c6c1a9100 con 0x7f1c6c10e310 2024-04-14T15:16:16.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.700+0000 7f1c68ff9700 1 -- 172.21.15.90:0/3322422450 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1c540035b0 con 0x7f1c6c10e310 2024-04-14T15:16:16.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.700+0000 7f1c68ff9700 1 -- 172.21.15.90:0/3322422450 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3758958939 0 0) 0x7f1c6c1aa2e0 con 0x7f1c6c107890 2024-04-14T15:16:16.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.700+0000 7f1c68ff9700 1 -- 172.21.15.90:0/3322422450 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1c6c1a9100 con 0x7f1c6c107890 2024-04-14T15:16:16.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.700+0000 7f1c68ff9700 1 -- 172.21.15.90:0/3322422450 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 461423178 0 0) 0x7f1c6c1a7f20 con 0x7f1c6c10a740 2024-04-14T15:16:16.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.700+0000 7f1c68ff9700 1 -- 172.21.15.90:0/3322422450 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1c6c1aa2e0 con 0x7f1c6c10a740 2024-04-14T15:16:16.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.700+0000 7f1c68ff9700 1 -- 172.21.15.90:0/3322422450 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2997980945 0 0) 0x7f1c540035b0 con 0x7f1c6c10e310 2024-04-14T15:16:16.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.700+0000 7f1c68ff9700 1 -- 172.21.15.90:0/3322422450 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1c6c1a7f20 con 0x7f1c6c10e310 2024-04-14T15:16:16.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.700+0000 7f1c68ff9700 1 -- 172.21.15.90:0/3322422450 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f1c64001c30 con 0x7f1c6c10e310 2024-04-14T15:16:16.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.700+0000 7f1c68ff9700 1 -- 172.21.15.90:0/3322422450 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1112523364 0 0) 0x7f1c6c1aa2e0 con 0x7f1c6c10a740 2024-04-14T15:16:16.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.700+0000 7f1c68ff9700 1 -- 172.21.15.90:0/3322422450 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1c540035b0 con 0x7f1c6c10a740 2024-04-14T15:16:16.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.700+0000 7f1c68ff9700 1 -- 172.21.15.90:0/3322422450 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3576258457 0 0) 0x7f1c6c1a9100 con 0x7f1c6c107890 2024-04-14T15:16:16.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.700+0000 7f1c68ff9700 1 -- 172.21.15.90:0/3322422450 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1c6c1aa2e0 con 0x7f1c6c107890 2024-04-14T15:16:16.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.700+0000 7f1c68ff9700 1 -- 172.21.15.90:0/3322422450 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f1c60001430 con 0x7f1c6c10a740 2024-04-14T15:16:16.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.700+0000 7f1c68ff9700 1 -- 172.21.15.90:0/3322422450 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2344389477 0 0) 0x7f1c6c1a7f20 con 0x7f1c6c10e310 2024-04-14T15:16:16.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.700+0000 7f1c68ff9700 1 -- 172.21.15.90:0/3322422450 >> v1:172.21.15.90:6790/0 conn(0x7f1c6c107890 legacy=0x7f1c6c10bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:16.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.700+0000 7f1c68ff9700 1 -- 172.21.15.90:0/3322422450 >> v1:172.21.15.90:6789/0 conn(0x7f1c6c10a740 legacy=0x7f1c6c10c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:16.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.700+0000 7f1c68ff9700 1 -- 172.21.15.90:0/3322422450 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c6c1ab500 con 0x7f1c6c10e310 2024-04-14T15:16:16.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.700+0000 7f1c71be1700 1 -- 172.21.15.90:0/3322422450 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1c6c1a8150 con 0x7f1c6c10e310 2024-04-14T15:16:16.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.700+0000 7f1c71be1700 10 client.48829.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:16:16.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.701+0000 7f1c71be1700 1 -- 172.21.15.90:0/3322422450 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f1c6c1a8610 con 0x7f1c6c10e310 2024-04-14T15:16:16.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.701+0000 7f1c71be1700 1 librados: init done 2024-04-14T15:16:16.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.701+0000 7f1c68ff9700 1 -- 172.21.15.90:0/3322422450 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f1c64003ec0 con 0x7f1c6c10e310 2024-04-14T15:16:16.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.701+0000 7f1c68ff9700 1 -- 172.21.15.90:0/3322422450 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f1c64004d70 con 0x7f1c6c10e310 2024-04-14T15:16:16.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.702+0000 7f1c71be1700 1 -- 172.21.15.90:0/3322422450 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f1c58005130 con 0x7f1c6c10e310 2024-04-14T15:16:16.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.702+0000 7f1c68ff9700 1 -- 172.21.15.90:0/3322422450 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f1c6401ba00 con 0x7f1c6c10e310 2024-04-14T15:16:16.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.703+0000 7f1c68ff9700 1 -- 172.21.15.90:0/3322422450 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(308..308 src has 51..308) ==== 8185+0+0 (unknown 431893661 0 0) 0x7f1c64089b10 con 0x7f1c6c10e310 2024-04-14T15:16:16.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.703+0000 7f1c68ff9700 10 client.48829.objecter ms_dispatch 0x7f1c6c001550 osd_map(308..308 src has 51..308) 2024-04-14T15:16:16.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.703+0000 7f1c68ff9700 3 client.48829.objecter handle_osd_map got epochs [308,308] > 0 2024-04-14T15:16:16.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.703+0000 7f1c68ff9700 3 client.48829.objecter handle_osd_map decoding full epoch 308 2024-04-14T15:16:16.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.703+0000 7f1c68ff9700 20 client.48829.objecter dump_active .. 0 homeless 2024-04-14T15:16:16.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.703+0000 7f1c68ff9700 1 -- 172.21.15.90:0/3322422450 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v308) ==== 54+0+16 (unknown 1841805393 0 2420000192) 0x7f1c64089f00 con 0x7f1c6c10e310 2024-04-14T15:16:16.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.703+0000 7f1c68ff9700 10 client.48829.objecter ms_handle_connect 0x7f1c540713f0 2024-04-14T15:16:16.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.703+0000 7f1c71be1700 10 client.48829.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:16:16.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.703+0000 7f1c71be1700 15 client.48829.objecter _session_command_op_assign -1 1 2024-04-14T15:16:16.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.703+0000 7f1c71be1700 20 client.48829.objecter _get_session s=0x7f1c58005410 osd=0 3 2024-04-14T15:16:16.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.703+0000 7f1c71be1700 20 client.48829.objecter put_session s=0x7f1c58005410 osd=0 3 2024-04-14T15:16:16.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.703+0000 7f1c71be1700 20 client.48829.objecter _get_session s=0x7f1c58005410 osd=0 3 2024-04-14T15:16:16.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.703+0000 7f1c71be1700 15 client.48829.objecter _session_command_op_remove -1 1 2024-04-14T15:16:16.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.703+0000 7f1c71be1700 20 client.48829.objecter get_session s=0x7f1c58005410 osd=0 3 2024-04-14T15:16:16.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.703+0000 7f1c71be1700 15 client.48829.objecter _session_command_op_assign 0 1 2024-04-14T15:16:16.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.703+0000 7f1c71be1700 20 client.48829.objecter put_session s=0x7f1c58005410 osd=0 4 2024-04-14T15:16:16.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.703+0000 7f1c71be1700 10 client.48829.objecter _send_command 1 2024-04-14T15:16:16.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.703+0000 7f1c71be1700 1 -- 172.21.15.90:0/3322422450 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1c58004ba0 con 0x7f1c58004760 2024-04-14T15:16:16.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.703+0000 7f1c68ff9700 10 client.48829.objecter ms_handle_connect 0x7f1c58004760 2024-04-14T15:16:16.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.704+0000 7f1c68ff9700 1 -- 172.21.15.90:0/3322422450 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1c58004ba0 con 0x7f1c58004760 2024-04-14T15:16:16.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.704+0000 7f1c68ff9700 10 client.48829.objecter ms_dispatch 0x7f1c6c001550 command_reply(tid 1: 0 ) 2024-04-14T15:16:16.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.704+0000 7f1c68ff9700 10 client.48829.objecter _finish_command 1 = system:0 2024-04-14T15:16:16.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.704+0000 7f1c68ff9700 20 client.48829.objecter put_session s=0x7f1c58005410 osd=0 4 2024-04-14T15:16:16.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.704+0000 7f1c68ff9700 15 client.48829.objecter _session_command_op_remove 0 1 2024-04-14T15:16:16.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.726+0000 7f1c71be1700 10 client.48829.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:16:16.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.726+0000 7f1c71be1700 15 client.48829.objecter _session_command_op_assign -1 2 2024-04-14T15:16:16.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.726+0000 7f1c71be1700 20 client.48829.objecter _get_session s=0x7f1c58005410 osd=0 3 2024-04-14T15:16:16.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.726+0000 7f1c71be1700 20 client.48829.objecter put_session s=0x7f1c58005410 osd=0 3 2024-04-14T15:16:16.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.726+0000 7f1c71be1700 20 client.48829.objecter _get_session s=0x7f1c58005410 osd=0 3 2024-04-14T15:16:16.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.726+0000 7f1c71be1700 15 client.48829.objecter _session_command_op_remove -1 2 2024-04-14T15:16:16.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.726+0000 7f1c71be1700 20 client.48829.objecter get_session s=0x7f1c58005410 osd=0 3 2024-04-14T15:16:16.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.726+0000 7f1c71be1700 15 client.48829.objecter _session_command_op_assign 0 2 2024-04-14T15:16:16.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.726+0000 7f1c71be1700 20 client.48829.objecter put_session s=0x7f1c58005410 osd=0 4 2024-04-14T15:16:16.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.726+0000 7f1c71be1700 10 client.48829.objecter _send_command 2 2024-04-14T15:16:16.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.726+0000 7f1c71be1700 1 -- 172.21.15.90:0/3322422450 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f1c58007a20 con 0x7f1c58004760 2024-04-14T15:16:16.728 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:16:16.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.728+0000 7f1c68ff9700 1 -- 172.21.15.90:0/3322422450 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1c58007a20 con 0x7f1c58004760 2024-04-14T15:16:16.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.728+0000 7f1c68ff9700 10 client.48829.objecter ms_dispatch 0x7f1c6c001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:16:16.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.728+0000 7f1c68ff9700 10 client.48829.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:16:16.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.728+0000 7f1c68ff9700 20 client.48829.objecter put_session s=0x7f1c58005410 osd=0 4 2024-04-14T15:16:16.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.728+0000 7f1c68ff9700 15 client.48829.objecter _session_command_op_remove 0 2 2024-04-14T15:16:16.729 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:16:16.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.728+0000 7f1c71be1700 10 client.48829.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:16:16.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.728+0000 7f1c71be1700 15 client.48829.objecter _session_command_op_assign -1 3 2024-04-14T15:16:16.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.728+0000 7f1c71be1700 20 client.48829.objecter _get_session s=0x7f1c58004120 osd=1 3 2024-04-14T15:16:16.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.728+0000 7f1c71be1700 20 client.48829.objecter put_session s=0x7f1c58004120 osd=1 3 2024-04-14T15:16:16.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.728+0000 7f1c71be1700 20 client.48829.objecter _get_session s=0x7f1c58004120 osd=1 3 2024-04-14T15:16:16.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.728+0000 7f1c71be1700 15 client.48829.objecter _session_command_op_remove -1 3 2024-04-14T15:16:16.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.728+0000 7f1c71be1700 20 client.48829.objecter get_session s=0x7f1c58004120 osd=1 3 2024-04-14T15:16:16.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.728+0000 7f1c71be1700 15 client.48829.objecter _session_command_op_assign 1 3 2024-04-14T15:16:16.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.728+0000 7f1c71be1700 20 client.48829.objecter put_session s=0x7f1c58004120 osd=1 4 2024-04-14T15:16:16.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.728+0000 7f1c71be1700 10 client.48829.objecter _send_command 3 2024-04-14T15:16:16.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.728+0000 7f1c71be1700 1 -- 172.21.15.90:0/3322422450 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f1c580043f0 con 0x7f1c580020c0 2024-04-14T15:16:16.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.728+0000 7f1c68ff9700 10 client.48829.objecter ms_handle_connect 0x7f1c580020c0 2024-04-14T15:16:16.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.729+0000 7f1c68ff9700 1 -- 172.21.15.90:0/3322422450 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1c580043f0 con 0x7f1c580020c0 2024-04-14T15:16:16.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.729+0000 7f1c68ff9700 10 client.48829.objecter ms_dispatch 0x7f1c6c001550 command_reply(tid 3: 0 ) 2024-04-14T15:16:16.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.729+0000 7f1c68ff9700 10 client.48829.objecter _finish_command 3 = system:0 2024-04-14T15:16:16.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.729+0000 7f1c68ff9700 20 client.48829.objecter put_session s=0x7f1c58004120 osd=1 4 2024-04-14T15:16:16.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.729+0000 7f1c68ff9700 15 client.48829.objecter _session_command_op_remove 1 3 2024-04-14T15:16:16.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.757+0000 7f1c71be1700 10 client.48829.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:16:16.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.757+0000 7f1c71be1700 15 client.48829.objecter _session_command_op_assign -1 4 2024-04-14T15:16:16.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.757+0000 7f1c71be1700 20 client.48829.objecter _get_session s=0x7f1c58004120 osd=1 3 2024-04-14T15:16:16.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.757+0000 7f1c71be1700 20 client.48829.objecter put_session s=0x7f1c58004120 osd=1 3 2024-04-14T15:16:16.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.757+0000 7f1c71be1700 20 client.48829.objecter _get_session s=0x7f1c58004120 osd=1 3 2024-04-14T15:16:16.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.757+0000 7f1c71be1700 15 client.48829.objecter _session_command_op_remove -1 4 2024-04-14T15:16:16.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.757+0000 7f1c71be1700 20 client.48829.objecter get_session s=0x7f1c58004120 osd=1 3 2024-04-14T15:16:16.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.757+0000 7f1c71be1700 15 client.48829.objecter _session_command_op_assign 1 4 2024-04-14T15:16:16.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.757+0000 7f1c71be1700 20 client.48829.objecter put_session s=0x7f1c58004120 osd=1 4 2024-04-14T15:16:16.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.757+0000 7f1c71be1700 10 client.48829.objecter _send_command 4 2024-04-14T15:16:16.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.757+0000 7f1c71be1700 1 -- 172.21.15.90:0/3322422450 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f1c58001690 con 0x7f1c580020c0 2024-04-14T15:16:16.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.759+0000 7f1c68ff9700 1 -- 172.21.15.90:0/3322422450 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1c58001690 con 0x7f1c580020c0 2024-04-14T15:16:16.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.759+0000 7f1c68ff9700 10 client.48829.objecter ms_dispatch 0x7f1c6c001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:16:16.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.759+0000 7f1c68ff9700 10 client.48829.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:16:16.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.759+0000 7f1c68ff9700 20 client.48829.objecter put_session s=0x7f1c58004120 osd=1 4 2024-04-14T15:16:16.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.759+0000 7f1c68ff9700 15 client.48829.objecter _session_command_op_remove 1 4 2024-04-14T15:16:16.760 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:16:16.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.759+0000 7f1c71be1700 10 client.48829.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:16:16.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.759+0000 7f1c71be1700 15 client.48829.objecter _session_command_op_assign -1 5 2024-04-14T15:16:16.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.759+0000 7f1c71be1700 20 client.48829.objecter _get_session s=0x7f1c580083a0 osd=2 3 2024-04-14T15:16:16.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.759+0000 7f1c71be1700 20 client.48829.objecter put_session s=0x7f1c580083a0 osd=2 3 2024-04-14T15:16:16.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.759+0000 7f1c71be1700 20 client.48829.objecter _get_session s=0x7f1c580083a0 osd=2 3 2024-04-14T15:16:16.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.759+0000 7f1c71be1700 15 client.48829.objecter _session_command_op_remove -1 5 2024-04-14T15:16:16.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.759+0000 7f1c71be1700 20 client.48829.objecter get_session s=0x7f1c580083a0 osd=2 3 2024-04-14T15:16:16.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.759+0000 7f1c71be1700 15 client.48829.objecter _session_command_op_assign 2 5 2024-04-14T15:16:16.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.759+0000 7f1c71be1700 20 client.48829.objecter put_session s=0x7f1c580083a0 osd=2 4 2024-04-14T15:16:16.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.759+0000 7f1c71be1700 10 client.48829.objecter _send_command 5 2024-04-14T15:16:16.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.759+0000 7f1c71be1700 1 -- 172.21.15.90:0/3322422450 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f1c58011110 con 0x7f1c5800d600 2024-04-14T15:16:16.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.759+0000 7f1c68ff9700 10 client.48829.objecter ms_handle_connect 0x7f1c5800d600 2024-04-14T15:16:16.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.760+0000 7f1c68ff9700 1 -- 172.21.15.90:0/3322422450 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1c58011110 con 0x7f1c5800d600 2024-04-14T15:16:16.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.760+0000 7f1c68ff9700 10 client.48829.objecter ms_dispatch 0x7f1c6c001550 command_reply(tid 5: 0 ) 2024-04-14T15:16:16.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.760+0000 7f1c68ff9700 10 client.48829.objecter _finish_command 5 = system:0 2024-04-14T15:16:16.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.760+0000 7f1c68ff9700 20 client.48829.objecter put_session s=0x7f1c580083a0 osd=2 4 2024-04-14T15:16:16.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.760+0000 7f1c68ff9700 15 client.48829.objecter _session_command_op_remove 2 5 2024-04-14T15:16:16.762 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:16:16.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.777+0000 7f1c71be1700 10 client.48829.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:16:16.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.777+0000 7f1c71be1700 15 client.48829.objecter _session_command_op_assign -1 6 2024-04-14T15:16:16.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.777+0000 7f1c71be1700 20 client.48829.objecter _get_session s=0x7f1c580083a0 osd=2 3 2024-04-14T15:16:16.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.777+0000 7f1c71be1700 20 client.48829.objecter put_session s=0x7f1c580083a0 osd=2 3 2024-04-14T15:16:16.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.777+0000 7f1c71be1700 20 client.48829.objecter _get_session s=0x7f1c580083a0 osd=2 3 2024-04-14T15:16:16.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.777+0000 7f1c71be1700 15 client.48829.objecter _session_command_op_remove -1 6 2024-04-14T15:16:16.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.777+0000 7f1c71be1700 20 client.48829.objecter get_session s=0x7f1c580083a0 osd=2 3 2024-04-14T15:16:16.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.777+0000 7f1c71be1700 15 client.48829.objecter _session_command_op_assign 2 6 2024-04-14T15:16:16.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.777+0000 7f1c71be1700 20 client.48829.objecter put_session s=0x7f1c580083a0 osd=2 4 2024-04-14T15:16:16.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.777+0000 7f1c71be1700 10 client.48829.objecter _send_command 6 2024-04-14T15:16:16.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.777+0000 7f1c71be1700 1 -- 172.21.15.90:0/3322422450 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f1c58007ee0 con 0x7f1c5800d600 2024-04-14T15:16:16.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.779+0000 7f1c68ff9700 1 -- 172.21.15.90:0/3322422450 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1c58007ee0 con 0x7f1c5800d600 2024-04-14T15:16:16.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.779+0000 7f1c68ff9700 10 client.48829.objecter ms_dispatch 0x7f1c6c001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:16:16.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.779+0000 7f1c68ff9700 10 client.48829.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:16:16.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.779+0000 7f1c68ff9700 20 client.48829.objecter put_session s=0x7f1c580083a0 osd=2 4 2024-04-14T15:16:16.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.779+0000 7f1c68ff9700 15 client.48829.objecter _session_command_op_remove 2 6 2024-04-14T15:16:16.780 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:16:16.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.779+0000 7f1c71be1700 10 client.48829.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:16:16.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.779+0000 7f1c71be1700 15 client.48829.objecter _session_command_op_assign -1 7 2024-04-14T15:16:16.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.779+0000 7f1c71be1700 20 client.48829.objecter _get_session s=0x7f1c5800cc50 osd=3 3 2024-04-14T15:16:16.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.779+0000 7f1c71be1700 20 client.48829.objecter put_session s=0x7f1c5800cc50 osd=3 3 2024-04-14T15:16:16.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.779+0000 7f1c71be1700 20 client.48829.objecter _get_session s=0x7f1c5800cc50 osd=3 3 2024-04-14T15:16:16.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.779+0000 7f1c71be1700 15 client.48829.objecter _session_command_op_remove -1 7 2024-04-14T15:16:16.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.779+0000 7f1c71be1700 20 client.48829.objecter get_session s=0x7f1c5800cc50 osd=3 3 2024-04-14T15:16:16.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.779+0000 7f1c71be1700 15 client.48829.objecter _session_command_op_assign 3 7 2024-04-14T15:16:16.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.779+0000 7f1c71be1700 20 client.48829.objecter put_session s=0x7f1c5800cc50 osd=3 4 2024-04-14T15:16:16.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.779+0000 7f1c71be1700 10 client.48829.objecter _send_command 7 2024-04-14T15:16:16.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.779+0000 7f1c71be1700 1 -- 172.21.15.90:0/3322422450 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f1c58015f50 con 0x7f1c58012440 2024-04-14T15:16:16.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.779+0000 7f1c68ff9700 10 client.48829.objecter ms_handle_connect 0x7f1c58012440 2024-04-14T15:16:16.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.780+0000 7f1c68ff9700 1 -- 172.21.15.90:0/3322422450 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1c58015f50 con 0x7f1c58012440 2024-04-14T15:16:16.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.780+0000 7f1c68ff9700 10 client.48829.objecter ms_dispatch 0x7f1c6c001550 command_reply(tid 7: 0 ) 2024-04-14T15:16:16.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.780+0000 7f1c68ff9700 10 client.48829.objecter _finish_command 7 = system:0 2024-04-14T15:16:16.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.780+0000 7f1c68ff9700 20 client.48829.objecter put_session s=0x7f1c5800cc50 osd=3 4 2024-04-14T15:16:16.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.780+0000 7f1c68ff9700 15 client.48829.objecter _session_command_op_remove 3 7 2024-04-14T15:16:16.785 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:16.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:16.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:16:16.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.797+0000 7f1c71be1700 10 client.48829.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:16:16.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.797+0000 7f1c71be1700 15 client.48829.objecter _session_command_op_assign -1 8 2024-04-14T15:16:16.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.797+0000 7f1c71be1700 20 client.48829.objecter _get_session s=0x7f1c5800cc50 osd=3 3 2024-04-14T15:16:16.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.797+0000 7f1c71be1700 20 client.48829.objecter put_session s=0x7f1c5800cc50 osd=3 3 2024-04-14T15:16:16.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.797+0000 7f1c71be1700 20 client.48829.objecter _get_session s=0x7f1c5800cc50 osd=3 3 2024-04-14T15:16:16.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.797+0000 7f1c71be1700 15 client.48829.objecter _session_command_op_remove -1 8 2024-04-14T15:16:16.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.797+0000 7f1c71be1700 20 client.48829.objecter get_session s=0x7f1c5800cc50 osd=3 3 2024-04-14T15:16:16.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.797+0000 7f1c71be1700 15 client.48829.objecter _session_command_op_assign 3 8 2024-04-14T15:16:16.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.797+0000 7f1c71be1700 20 client.48829.objecter put_session s=0x7f1c5800cc50 osd=3 4 2024-04-14T15:16:16.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.797+0000 7f1c71be1700 10 client.48829.objecter _send_command 8 2024-04-14T15:16:16.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.797+0000 7f1c71be1700 1 -- 172.21.15.90:0/3322422450 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f1c5800c6d0 con 0x7f1c58012440 2024-04-14T15:16:16.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.799+0000 7f1c68ff9700 1 -- 172.21.15.90:0/3322422450 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1c5800c6d0 con 0x7f1c58012440 2024-04-14T15:16:16.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.799+0000 7f1c68ff9700 10 client.48829.objecter ms_dispatch 0x7f1c6c001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:16:16.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.799+0000 7f1c68ff9700 10 client.48829.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:16:16.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.799+0000 7f1c68ff9700 20 client.48829.objecter put_session s=0x7f1c5800cc50 osd=3 4 2024-04-14T15:16:16.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.799+0000 7f1c68ff9700 15 client.48829.objecter _session_command_op_remove 3 8 2024-04-14T15:16:16.799 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:16:16.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.799+0000 7f1c71be1700 10 client.48829.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:16:16.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.799+0000 7f1c71be1700 15 client.48829.objecter _session_command_op_assign -1 9 2024-04-14T15:16:16.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.799+0000 7f1c71be1700 20 client.48829.objecter _get_session s=0x7f1c5800c480 osd=4 3 2024-04-14T15:16:16.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.799+0000 7f1c71be1700 20 client.48829.objecter put_session s=0x7f1c5800c480 osd=4 3 2024-04-14T15:16:16.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.799+0000 7f1c71be1700 20 client.48829.objecter _get_session s=0x7f1c5800c480 osd=4 3 2024-04-14T15:16:16.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.799+0000 7f1c71be1700 15 client.48829.objecter _session_command_op_remove -1 9 2024-04-14T15:16:16.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.799+0000 7f1c71be1700 20 client.48829.objecter get_session s=0x7f1c5800c480 osd=4 3 2024-04-14T15:16:16.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.799+0000 7f1c71be1700 15 client.48829.objecter _session_command_op_assign 4 9 2024-04-14T15:16:16.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.799+0000 7f1c71be1700 20 client.48829.objecter put_session s=0x7f1c5800c480 osd=4 4 2024-04-14T15:16:16.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.799+0000 7f1c71be1700 10 client.48829.objecter _send_command 9 2024-04-14T15:16:16.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.799+0000 7f1c71be1700 1 -- 172.21.15.90:0/3322422450 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f1c5801a9b0 con 0x7f1c58011fa0 2024-04-14T15:16:16.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.799+0000 7f1c68ff9700 10 client.48829.objecter ms_handle_connect 0x7f1c58011fa0 2024-04-14T15:16:16.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.800+0000 7f1c68ff9700 1 -- 172.21.15.90:0/3322422450 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1c5801a9b0 con 0x7f1c58011fa0 2024-04-14T15:16:16.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.800+0000 7f1c68ff9700 10 client.48829.objecter ms_dispatch 0x7f1c6c001550 command_reply(tid 9: 0 ) 2024-04-14T15:16:16.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.800+0000 7f1c68ff9700 10 client.48829.objecter _finish_command 9 = system:0 2024-04-14T15:16:16.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.800+0000 7f1c68ff9700 20 client.48829.objecter put_session s=0x7f1c5800c480 osd=4 4 2024-04-14T15:16:16.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.800+0000 7f1c68ff9700 15 client.48829.objecter _session_command_op_remove 4 9 2024-04-14T15:16:16.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.817+0000 7f1c71be1700 10 client.48829.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:16:16.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.817+0000 7f1c71be1700 15 client.48829.objecter _session_command_op_assign -1 10 2024-04-14T15:16:16.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.817+0000 7f1c71be1700 20 client.48829.objecter _get_session s=0x7f1c5800c480 osd=4 3 2024-04-14T15:16:16.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.817+0000 7f1c71be1700 20 client.48829.objecter put_session s=0x7f1c5800c480 osd=4 3 2024-04-14T15:16:16.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.817+0000 7f1c71be1700 20 client.48829.objecter _get_session s=0x7f1c5800c480 osd=4 3 2024-04-14T15:16:16.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.817+0000 7f1c71be1700 15 client.48829.objecter _session_command_op_remove -1 10 2024-04-14T15:16:16.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.817+0000 7f1c71be1700 20 client.48829.objecter get_session s=0x7f1c5800c480 osd=4 3 2024-04-14T15:16:16.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.817+0000 7f1c71be1700 15 client.48829.objecter _session_command_op_assign 4 10 2024-04-14T15:16:16.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.817+0000 7f1c71be1700 20 client.48829.objecter put_session s=0x7f1c5800c480 osd=4 4 2024-04-14T15:16:16.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.817+0000 7f1c71be1700 10 client.48829.objecter _send_command 10 2024-04-14T15:16:16.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.817+0000 7f1c71be1700 1 -- 172.21.15.90:0/3322422450 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f1c5801a9b0 con 0x7f1c58011fa0 2024-04-14T15:16:16.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.818+0000 7f1c68ff9700 1 -- 172.21.15.90:0/3322422450 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1c5801a9b0 con 0x7f1c58011fa0 2024-04-14T15:16:16.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.818+0000 7f1c68ff9700 10 client.48829.objecter ms_dispatch 0x7f1c6c001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:16:16.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.818+0000 7f1c68ff9700 10 client.48829.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:16:16.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.818+0000 7f1c68ff9700 20 client.48829.objecter put_session s=0x7f1c5800c480 osd=4 4 2024-04-14T15:16:16.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.818+0000 7f1c68ff9700 15 client.48829.objecter _session_command_op_remove 4 10 2024-04-14T15:16:16.820 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:16:16.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.819+0000 7f1c71be1700 10 client.48829.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:16:16.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.819+0000 7f1c71be1700 15 client.48829.objecter _session_command_op_assign -1 11 2024-04-14T15:16:16.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.819+0000 7f1c71be1700 20 client.48829.objecter _get_session s=0x7f1c5801a9b0 osd=5 3 2024-04-14T15:16:16.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.819+0000 7f1c71be1700 20 client.48829.objecter put_session s=0x7f1c5801a9b0 osd=5 3 2024-04-14T15:16:16.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.819+0000 7f1c71be1700 20 client.48829.objecter _get_session s=0x7f1c5801a9b0 osd=5 3 2024-04-14T15:16:16.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.819+0000 7f1c71be1700 15 client.48829.objecter _session_command_op_remove -1 11 2024-04-14T15:16:16.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.819+0000 7f1c71be1700 20 client.48829.objecter get_session s=0x7f1c5801a9b0 osd=5 3 2024-04-14T15:16:16.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.819+0000 7f1c71be1700 15 client.48829.objecter _session_command_op_assign 5 11 2024-04-14T15:16:16.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.819+0000 7f1c71be1700 20 client.48829.objecter put_session s=0x7f1c5801a9b0 osd=5 4 2024-04-14T15:16:16.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.819+0000 7f1c71be1700 10 client.48829.objecter _send_command 11 2024-04-14T15:16:16.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.819+0000 7f1c71be1700 1 -- 172.21.15.90:0/3322422450 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f1c5801ecd0 con 0x7f1c5801b030 2024-04-14T15:16:16.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.819+0000 7f1c68ff9700 10 client.48829.objecter ms_handle_connect 0x7f1c5801b030 2024-04-14T15:16:16.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.820+0000 7f1c68ff9700 1 -- 172.21.15.90:0/3322422450 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1c5801ecd0 con 0x7f1c5801b030 2024-04-14T15:16:16.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.820+0000 7f1c68ff9700 10 client.48829.objecter ms_dispatch 0x7f1c6c001550 command_reply(tid 11: 0 ) 2024-04-14T15:16:16.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.820+0000 7f1c68ff9700 10 client.48829.objecter _finish_command 11 = system:0 2024-04-14T15:16:16.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.820+0000 7f1c68ff9700 20 client.48829.objecter put_session s=0x7f1c5801a9b0 osd=5 4 2024-04-14T15:16:16.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.820+0000 7f1c68ff9700 15 client.48829.objecter _session_command_op_remove 5 11 2024-04-14T15:16:16.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.836+0000 7f1c71be1700 10 client.48829.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:16:16.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.836+0000 7f1c71be1700 15 client.48829.objecter _session_command_op_assign -1 12 2024-04-14T15:16:16.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.836+0000 7f1c71be1700 20 client.48829.objecter _get_session s=0x7f1c5801a9b0 osd=5 3 2024-04-14T15:16:16.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.836+0000 7f1c71be1700 20 client.48829.objecter put_session s=0x7f1c5801a9b0 osd=5 3 2024-04-14T15:16:16.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.836+0000 7f1c71be1700 20 client.48829.objecter _get_session s=0x7f1c5801a9b0 osd=5 3 2024-04-14T15:16:16.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.836+0000 7f1c71be1700 15 client.48829.objecter _session_command_op_remove -1 12 2024-04-14T15:16:16.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.836+0000 7f1c71be1700 20 client.48829.objecter get_session s=0x7f1c5801a9b0 osd=5 3 2024-04-14T15:16:16.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.836+0000 7f1c71be1700 15 client.48829.objecter _session_command_op_assign 5 12 2024-04-14T15:16:16.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.836+0000 7f1c71be1700 20 client.48829.objecter put_session s=0x7f1c5801a9b0 osd=5 4 2024-04-14T15:16:16.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.836+0000 7f1c71be1700 10 client.48829.objecter _send_command 12 2024-04-14T15:16:16.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.836+0000 7f1c71be1700 1 -- 172.21.15.90:0/3322422450 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f1c5800c990 con 0x7f1c5801b030 2024-04-14T15:16:16.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.838+0000 7f1c68ff9700 1 -- 172.21.15.90:0/3322422450 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1c5800c990 con 0x7f1c5801b030 2024-04-14T15:16:16.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.838+0000 7f1c68ff9700 10 client.48829.objecter ms_dispatch 0x7f1c6c001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:16:16.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.838+0000 7f1c68ff9700 10 client.48829.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:16:16.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.838+0000 7f1c68ff9700 20 client.48829.objecter put_session s=0x7f1c5801a9b0 osd=5 4 2024-04-14T15:16:16.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.838+0000 7f1c68ff9700 15 client.48829.objecter _session_command_op_remove 5 12 2024-04-14T15:16:16.838 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:16:16.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.838+0000 7f1c71be1700 10 client.48829.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:16:16.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.838+0000 7f1c71be1700 15 client.48829.objecter _session_command_op_assign -1 13 2024-04-14T15:16:16.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.838+0000 7f1c71be1700 20 client.48829.objecter _get_session s=0x7f1c5800c990 osd=6 3 2024-04-14T15:16:16.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.838+0000 7f1c71be1700 20 client.48829.objecter put_session s=0x7f1c5800c990 osd=6 3 2024-04-14T15:16:16.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.838+0000 7f1c71be1700 20 client.48829.objecter _get_session s=0x7f1c5800c990 osd=6 3 2024-04-14T15:16:16.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.838+0000 7f1c71be1700 15 client.48829.objecter _session_command_op_remove -1 13 2024-04-14T15:16:16.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.838+0000 7f1c71be1700 20 client.48829.objecter get_session s=0x7f1c5800c990 osd=6 3 2024-04-14T15:16:16.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.838+0000 7f1c71be1700 15 client.48829.objecter _session_command_op_assign 6 13 2024-04-14T15:16:16.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.838+0000 7f1c71be1700 20 client.48829.objecter put_session s=0x7f1c5800c990 osd=6 4 2024-04-14T15:16:16.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.838+0000 7f1c71be1700 10 client.48829.objecter _send_command 13 2024-04-14T15:16:16.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.838+0000 7f1c71be1700 1 -- 172.21.15.90:0/3322422450 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f1c58016cf0 con 0x7f1c580168d0 2024-04-14T15:16:16.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.838+0000 7f1c68ff9700 10 client.48829.objecter ms_handle_connect 0x7f1c580168d0 2024-04-14T15:16:16.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.839+0000 7f1c68ff9700 1 -- 172.21.15.90:0/3322422450 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1c58016cf0 con 0x7f1c580168d0 2024-04-14T15:16:16.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.839+0000 7f1c68ff9700 10 client.48829.objecter ms_dispatch 0x7f1c6c001550 command_reply(tid 13: 0 ) 2024-04-14T15:16:16.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.839+0000 7f1c68ff9700 10 client.48829.objecter _finish_command 13 = system:0 2024-04-14T15:16:16.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.839+0000 7f1c68ff9700 20 client.48829.objecter put_session s=0x7f1c5800c990 osd=6 4 2024-04-14T15:16:16.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.839+0000 7f1c68ff9700 15 client.48829.objecter _session_command_op_remove 6 13 2024-04-14T15:16:16.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.855+0000 7f1c71be1700 10 client.48829.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:16:16.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.855+0000 7f1c71be1700 15 client.48829.objecter _session_command_op_assign -1 14 2024-04-14T15:16:16.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.855+0000 7f1c71be1700 20 client.48829.objecter _get_session s=0x7f1c5800c990 osd=6 3 2024-04-14T15:16:16.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.855+0000 7f1c71be1700 20 client.48829.objecter put_session s=0x7f1c5800c990 osd=6 3 2024-04-14T15:16:16.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.855+0000 7f1c71be1700 20 client.48829.objecter _get_session s=0x7f1c5800c990 osd=6 3 2024-04-14T15:16:16.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.855+0000 7f1c71be1700 15 client.48829.objecter _session_command_op_remove -1 14 2024-04-14T15:16:16.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.855+0000 7f1c71be1700 20 client.48829.objecter get_session s=0x7f1c5800c990 osd=6 3 2024-04-14T15:16:16.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.855+0000 7f1c71be1700 15 client.48829.objecter _session_command_op_assign 6 14 2024-04-14T15:16:16.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.855+0000 7f1c71be1700 20 client.48829.objecter put_session s=0x7f1c5800c990 osd=6 4 2024-04-14T15:16:16.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.855+0000 7f1c71be1700 10 client.48829.objecter _send_command 14 2024-04-14T15:16:16.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.855+0000 7f1c71be1700 1 -- 172.21.15.90:0/3322422450 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f1c58016cf0 con 0x7f1c580168d0 2024-04-14T15:16:16.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.857+0000 7f1c68ff9700 1 -- 172.21.15.90:0/3322422450 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1c58016cf0 con 0x7f1c580168d0 2024-04-14T15:16:16.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.857+0000 7f1c68ff9700 10 client.48829.objecter ms_dispatch 0x7f1c6c001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:16:16.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.857+0000 7f1c68ff9700 10 client.48829.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:16:16.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.857+0000 7f1c68ff9700 20 client.48829.objecter put_session s=0x7f1c5800c990 osd=6 4 2024-04-14T15:16:16.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.857+0000 7f1c68ff9700 15 client.48829.objecter _session_command_op_remove 6 14 2024-04-14T15:16:16.858 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:16:16.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.857+0000 7f1c71be1700 10 client.48829.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:16:16.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.857+0000 7f1c71be1700 15 client.48829.objecter _session_command_op_assign -1 15 2024-04-14T15:16:16.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.857+0000 7f1c71be1700 20 client.48829.objecter _get_session s=0x7f1c58016cf0 osd=7 3 2024-04-14T15:16:16.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.857+0000 7f1c71be1700 20 client.48829.objecter put_session s=0x7f1c58016cf0 osd=7 3 2024-04-14T15:16:16.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.857+0000 7f1c71be1700 20 client.48829.objecter _get_session s=0x7f1c58016cf0 osd=7 3 2024-04-14T15:16:16.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.857+0000 7f1c71be1700 15 client.48829.objecter _session_command_op_remove -1 15 2024-04-14T15:16:16.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.857+0000 7f1c71be1700 20 client.48829.objecter get_session s=0x7f1c58016cf0 osd=7 3 2024-04-14T15:16:16.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.857+0000 7f1c71be1700 15 client.48829.objecter _session_command_op_assign 7 15 2024-04-14T15:16:16.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.857+0000 7f1c71be1700 20 client.48829.objecter put_session s=0x7f1c58016cf0 osd=7 4 2024-04-14T15:16:16.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.857+0000 7f1c71be1700 10 client.48829.objecter _send_command 15 2024-04-14T15:16:16.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.857+0000 7f1c71be1700 1 -- 172.21.15.90:0/3322422450 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f1c58027630 con 0x7f1c58016e60 2024-04-14T15:16:16.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.858+0000 7f1c68ff9700 10 client.48829.objecter ms_handle_connect 0x7f1c58016e60 2024-04-14T15:16:16.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.858+0000 7f1c68ff9700 1 -- 172.21.15.90:0/3322422450 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1c58027630 con 0x7f1c58016e60 2024-04-14T15:16:16.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.858+0000 7f1c68ff9700 10 client.48829.objecter ms_dispatch 0x7f1c6c001550 command_reply(tid 15: 0 ) 2024-04-14T15:16:16.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.858+0000 7f1c68ff9700 10 client.48829.objecter _finish_command 15 = system:0 2024-04-14T15:16:16.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.858+0000 7f1c68ff9700 20 client.48829.objecter put_session s=0x7f1c58016cf0 osd=7 4 2024-04-14T15:16:16.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.858+0000 7f1c68ff9700 15 client.48829.objecter _session_command_op_remove 7 15 2024-04-14T15:16:16.888 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:16.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.889+0000 7f1c71be1700 10 client.48829.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:16:16.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.889+0000 7f1c71be1700 15 client.48829.objecter _session_command_op_assign -1 16 2024-04-14T15:16:16.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.889+0000 7f1c71be1700 20 client.48829.objecter _get_session s=0x7f1c58016cf0 osd=7 3 2024-04-14T15:16:16.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.889+0000 7f1c71be1700 20 client.48829.objecter put_session s=0x7f1c58016cf0 osd=7 3 2024-04-14T15:16:16.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.889+0000 7f1c71be1700 20 client.48829.objecter _get_session s=0x7f1c58016cf0 osd=7 3 2024-04-14T15:16:16.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.889+0000 7f1c71be1700 15 client.48829.objecter _session_command_op_remove -1 16 2024-04-14T15:16:16.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.889+0000 7f1c71be1700 20 client.48829.objecter get_session s=0x7f1c58016cf0 osd=7 3 2024-04-14T15:16:16.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.889+0000 7f1c71be1700 15 client.48829.objecter _session_command_op_assign 7 16 2024-04-14T15:16:16.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.889+0000 7f1c71be1700 20 client.48829.objecter put_session s=0x7f1c58016cf0 osd=7 4 2024-04-14T15:16:16.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.889+0000 7f1c71be1700 10 client.48829.objecter _send_command 16 2024-04-14T15:16:16.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.889+0000 7f1c71be1700 1 -- 172.21.15.90:0/3322422450 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f1c58028fe0 con 0x7f1c58016e60 2024-04-14T15:16:16.894 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:16:16.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.893+0000 7f1c68ff9700 1 -- 172.21.15.90:0/3322422450 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1c58028fe0 con 0x7f1c58016e60 2024-04-14T15:16:16.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.893+0000 7f1c68ff9700 10 client.48829.objecter ms_dispatch 0x7f1c6c001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:16:16.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.893+0000 7f1c68ff9700 10 client.48829.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:16:16.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.893+0000 7f1c68ff9700 20 client.48829.objecter put_session s=0x7f1c58016cf0 osd=7 4 2024-04-14T15:16:16.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.893+0000 7f1c68ff9700 15 client.48829.objecter _session_command_op_remove 7 16 2024-04-14T15:16:16.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.894+0000 7f1c71be1700 10 librados: watch_flush enter 2024-04-14T15:16:16.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.894+0000 7f1c71be1700 10 librados: watch_flush exit 2024-04-14T15:16:16.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.894+0000 7f1c71be1700 10 client.48829.objecter close_session for osd.0 2024-04-14T15:16:16.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.894+0000 7f1c71be1700 1 -- 172.21.15.90:0/3322422450 >> v1:172.21.15.90:6801/3761806885 conn(0x7f1c58004760 legacy=0x7f1c58002780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:16.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.894+0000 7f1c71be1700 20 client.48829.objecter put_session s=0x7f1c58005410 osd=0 1 2024-04-14T15:16:16.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.894+0000 7f1c71be1700 10 client.48829.objecter close_session for osd.1 2024-04-14T15:16:16.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.894+0000 7f1c71be1700 1 -- 172.21.15.90:0/3322422450 >> v1:172.21.15.90:6805/4170361699 conn(0x7f1c580020c0 legacy=0x7f1c5800ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:16.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.894+0000 7f1c71be1700 20 client.48829.objecter put_session s=0x7f1c58004120 osd=1 1 2024-04-14T15:16:16.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.894+0000 7f1c71be1700 10 client.48829.objecter close_session for osd.2 2024-04-14T15:16:16.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.894+0000 7f1c71be1700 1 -- 172.21.15.90:0/3322422450 >> v1:172.21.15.90:6809/1686184251 conn(0x7f1c5800d600 legacy=0x7f1c5800f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:16.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.894+0000 7f1c71be1700 20 client.48829.objecter put_session s=0x7f1c580083a0 osd=2 1 2024-04-14T15:16:16.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.894+0000 7f1c71be1700 10 client.48829.objecter close_session for osd.3 2024-04-14T15:16:16.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.894+0000 7f1c71be1700 1 -- 172.21.15.90:0/3322422450 >> v1:172.21.15.90:6813/2270166774 conn(0x7f1c58012440 legacy=0x7f1c58014830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:16.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.894+0000 7f1c71be1700 20 client.48829.objecter put_session s=0x7f1c5800cc50 osd=3 1 2024-04-14T15:16:16.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.894+0000 7f1c71be1700 10 client.48829.objecter close_session for osd.4 2024-04-14T15:16:16.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.894+0000 7f1c71be1700 1 -- 172.21.15.90:0/3322422450 >> v1:172.21.15.92:6800/3887549733 conn(0x7f1c58011fa0 legacy=0x7f1c58019290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:16.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.894+0000 7f1c71be1700 20 client.48829.objecter put_session s=0x7f1c5800c480 osd=4 1 2024-04-14T15:16:16.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.894+0000 7f1c71be1700 10 client.48829.objecter close_session for osd.5 2024-04-14T15:16:16.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.894+0000 7f1c71be1700 1 -- 172.21.15.90:0/3322422450 >> v1:172.21.15.92:6804/3244561440 conn(0x7f1c5801b030 legacy=0x7f1c5801d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:16.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.894+0000 7f1c71be1700 20 client.48829.objecter put_session s=0x7f1c5801a9b0 osd=5 1 2024-04-14T15:16:16.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.894+0000 7f1c71be1700 10 client.48829.objecter close_session for osd.6 2024-04-14T15:16:16.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.894+0000 7f1c71be1700 1 -- 172.21.15.90:0/3322422450 >> v1:172.21.15.92:6808/636125963 conn(0x7f1c580168d0 legacy=0x7f1c5801ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:16.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.894+0000 7f1c71be1700 20 client.48829.objecter put_session s=0x7f1c5800c990 osd=6 1 2024-04-14T15:16:16.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.894+0000 7f1c71be1700 10 client.48829.objecter close_session for osd.7 2024-04-14T15:16:16.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.894+0000 7f1c71be1700 1 -- 172.21.15.90:0/3322422450 >> v1:172.21.15.92:6812/1773706716 conn(0x7f1c58016e60 legacy=0x7f1c58025da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:16.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.894+0000 7f1c71be1700 20 client.48829.objecter put_session s=0x7f1c58016cf0 osd=7 1 2024-04-14T15:16:16.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.894+0000 7f1c71be1700 20 client.48829.objecter shutdown clearing up homeless session... 2024-04-14T15:16:16.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.894+0000 7f1c71be1700 10 client.48829.objecter successfully canceled tick 2024-04-14T15:16:16.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.894+0000 7f1c6bfff700 1 -- 172.21.15.90:0/3322422450 reap_dead start 2024-04-14T15:16:16.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.894+0000 7f1c6bfff700 1 -- 172.21.15.90:0/3322422450 reap_dead start 2024-04-14T15:16:16.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.894+0000 7f1c71be1700 1 -- 172.21.15.90:0/3322422450 >> v1:172.21.15.90:6800/420935427 conn(0x7f1c540713f0 legacy=0x7f1c540738b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:16.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.894+0000 7f1c6bfff700 1 -- 172.21.15.90:0/3322422450 reap_dead start 2024-04-14T15:16:16.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.894+0000 7f1c6bfff700 1 -- 172.21.15.90:0/3322422450 reap_dead start 2024-04-14T15:16:16.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.894+0000 7f1c6bfff700 1 -- 172.21.15.90:0/3322422450 reap_dead start 2024-04-14T15:16:16.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.894+0000 7f1c71be1700 1 -- 172.21.15.90:0/3322422450 >> v1:172.21.15.92:6789/0 conn(0x7f1c6c10e310 legacy=0x7f1c6c1a6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:16.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.894+0000 7f1c71be1700 1 -- 172.21.15.90:0/3322422450 shutdown_connections 2024-04-14T15:16:16.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.894+0000 7f1c71be1700 1 -- 172.21.15.90:0/3322422450 >> 172.21.15.90:0/3322422450 conn(0x7f1c6c0fd290 msgr2=0x7f1c6c06b110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:16:16.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.895+0000 7f1c71be1700 1 -- 172.21.15.90:0/3322422450 shutdown_connections 2024-04-14T15:16:16.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.895+0000 7f1c71be1700 1 -- 172.21.15.90:0/3322422450 wait complete. 2024-04-14T15:16:16.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:16.895+0000 7f1c71be1700 1 librados: shutdown 2024-04-14T15:16:16.899 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:16:16.899 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:16:16.899 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:16:16.900 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:16:16.900 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:16:16.900 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:16:16.900 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:16:16.900 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:16:16.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:16.915 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:16.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:16.981 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:16 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:16:16.981 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:16 smithi090 ceph-mon[41670]: pgmap v407: 206 pgs: 206 active+clean; 4.6 MiB data, 570 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-14T15:16:16.981 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:16 smithi090 ceph-mon[41670]: from='client.48811 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi090-93363-53"}]': finished 2024-04-14T15:16:16.981 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:16 smithi090 ceph-mon[41670]: osdmap e308: 8 total, 8 up, 8 in 2024-04-14T15:16:16.981 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:16 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/442343645' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi090-93363-53"}]: dispatch 2024-04-14T15:16:16.981 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:16 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:16:16.981 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:16 smithi090 ceph-mon[41670]: from='client.48811 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi090-93363-53"}]: dispatch 2024-04-14T15:16:16.981 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:16 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:16:16.981 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:16 smithi090 ceph-mon[41670]: 4.0 scrub starts 2024-04-14T15:16:16.981 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:16 smithi090 ceph-mon[41670]: 4.0 scrub ok 2024-04-14T15:16:16.982 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:16 smithi090 ceph-mon[41670]: 4.7 deep-scrub starts 2024-04-14T15:16:16.982 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:16 smithi090 ceph-mon[41670]: 4.7 deep-scrub ok 2024-04-14T15:16:16.982 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:16 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3322422450' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:16:16.982 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:16 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:16:16.982 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:16 smithi090 ceph-mon[30399]: pgmap v407: 206 pgs: 206 active+clean; 4.6 MiB data, 570 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-14T15:16:16.982 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:16 smithi090 ceph-mon[30399]: from='client.48811 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi090-93363-53"}]': finished 2024-04-14T15:16:16.982 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:16 smithi090 ceph-mon[30399]: osdmap e308: 8 total, 8 up, 8 in 2024-04-14T15:16:16.982 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:16 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/442343645' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi090-93363-53"}]: dispatch 2024-04-14T15:16:16.982 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:16 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:16:16.982 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:16 smithi090 ceph-mon[30399]: from='client.48811 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi090-93363-53"}]: dispatch 2024-04-14T15:16:16.982 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:16 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:16:16.983 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:16 smithi090 ceph-mon[30399]: 4.0 scrub starts 2024-04-14T15:16:16.983 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:16 smithi090 ceph-mon[30399]: 4.0 scrub ok 2024-04-14T15:16:16.983 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:16 smithi090 ceph-mon[30399]: 4.7 deep-scrub starts 2024-04-14T15:16:16.983 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:16 smithi090 ceph-mon[30399]: 4.7 deep-scrub ok 2024-04-14T15:16:16.985 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:16 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3322422450' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:16:17.016 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:17.040 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:17.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:17.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:17.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:16 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:16:17.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:16 smithi092 ceph-mon[33966]: pgmap v407: 206 pgs: 206 active+clean; 4.6 MiB data, 570 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-14T15:16:17.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:16 smithi092 ceph-mon[33966]: from='client.48811 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi090-93363-53"}]': finished 2024-04-14T15:16:17.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:16 smithi092 ceph-mon[33966]: osdmap e308: 8 total, 8 up, 8 in 2024-04-14T15:16:17.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:16 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/442343645' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi090-93363-53"}]: dispatch 2024-04-14T15:16:17.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:16 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:16:17.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:16 smithi092 ceph-mon[33966]: from='client.48811 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi090-93363-53"}]: dispatch 2024-04-14T15:16:17.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:16 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:16:17.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:16 smithi092 ceph-mon[33966]: 4.0 scrub starts 2024-04-14T15:16:17.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:16 smithi092 ceph-mon[33966]: 4.0 scrub ok 2024-04-14T15:16:17.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:16 smithi092 ceph-mon[33966]: 4.7 deep-scrub starts 2024-04-14T15:16:17.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:16 smithi092 ceph-mon[33966]: 4.7 deep-scrub ok 2024-04-14T15:16:17.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:16 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3322422450' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:16:17.141 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:17.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:17.169 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:17.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:17.270 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:17.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:17.294 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:17.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:17.395 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:16:17.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:17.422 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:17.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:16:17.523 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:17.548 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:17.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:17.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:17.579 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd set noscrub 2024-04-14T15:16:17.650 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:16:17.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:17.675 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:17.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:16:17.776 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:16:17.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:17.801 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:17.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:16:17.903 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:17.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:17.927 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:17.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:18.018 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:17 smithi092 ceph-mon[33966]: 5.a scrub starts 2024-04-14T15:16:18.018 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:17 smithi092 ceph-mon[33966]: 5.a scrub ok 2024-04-14T15:16:18.018 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:17 smithi092 ceph-mon[33966]: 2.b scrub starts 2024-04-14T15:16:18.018 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:17 smithi092 ceph-mon[33966]: 2.b scrub ok 2024-04-14T15:16:18.018 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:17 smithi092 ceph-mon[33966]: from='client.48811 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi090-93363-53"}]': finished 2024-04-14T15:16:18.018 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:17 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:16:18.018 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:17 smithi092 ceph-mon[33966]: osdmap e309: 8 total, 8 up, 8 in 2024-04-14T15:16:18.018 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:17 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:16:18.018 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:17 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:16:18.018 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:17 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/585933469' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi090-93363-54"}]: dispatch 2024-04-14T15:16:18.019 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:17 smithi092 ceph-mon[33966]: from='client.48835 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi090-93363-54"}]: dispatch 2024-04-14T15:16:18.019 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:17 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/585933469' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi090-93363-54"}]: dispatch 2024-04-14T15:16:18.019 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:17 smithi092 ceph-mon[33966]: from='client.48835 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi090-93363-54"}]: dispatch 2024-04-14T15:16:18.019 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:17 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/585933469' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi090-93363-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:16:18.019 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:17 smithi092 ceph-mon[33966]: from='client.48835 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi090-93363-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:16:18.019 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:17 smithi092 ceph-mon[33966]: 6.e scrub starts 2024-04-14T15:16:18.019 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:17 smithi092 ceph-mon[33966]: 6.e scrub ok 2024-04-14T15:16:18.019 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:17 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:16:18.019 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:17 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:16:18.019 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:17 smithi092 ceph-mon[33966]: 6.0 scrub starts 2024-04-14T15:16:18.019 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:17 smithi092 ceph-mon[33966]: 6.0 scrub ok 2024-04-14T15:16:18.029 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:16:18.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:18.054 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:18.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:16:18.155 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:18.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:18.180 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:18.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:18.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:17 smithi090 ceph-mon[30399]: 5.a scrub starts 2024-04-14T15:16:18.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:17 smithi090 ceph-mon[30399]: 5.a scrub ok 2024-04-14T15:16:18.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:17 smithi090 ceph-mon[30399]: 2.b scrub starts 2024-04-14T15:16:18.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:17 smithi090 ceph-mon[30399]: 2.b scrub ok 2024-04-14T15:16:18.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:17 smithi090 ceph-mon[30399]: from='client.48811 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi090-93363-53"}]': finished 2024-04-14T15:16:18.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:17 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:16:18.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:17 smithi090 ceph-mon[30399]: osdmap e309: 8 total, 8 up, 8 in 2024-04-14T15:16:18.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:17 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:16:18.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:17 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:16:18.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:17 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/585933469' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi090-93363-54"}]: dispatch 2024-04-14T15:16:18.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:17 smithi090 ceph-mon[30399]: from='client.48835 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi090-93363-54"}]: dispatch 2024-04-14T15:16:18.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:17 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/585933469' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi090-93363-54"}]: dispatch 2024-04-14T15:16:18.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:17 smithi090 ceph-mon[30399]: from='client.48835 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi090-93363-54"}]: dispatch 2024-04-14T15:16:18.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:17 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/585933469' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi090-93363-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:16:18.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:17 smithi090 ceph-mon[30399]: from='client.48835 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi090-93363-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:16:18.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:17 smithi090 ceph-mon[30399]: 6.e scrub starts 2024-04-14T15:16:18.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:17 smithi090 ceph-mon[30399]: 6.e scrub ok 2024-04-14T15:16:18.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:17 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:16:18.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:17 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:16:18.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:17 smithi090 ceph-mon[30399]: 6.0 scrub starts 2024-04-14T15:16:18.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:17 smithi090 ceph-mon[30399]: 6.0 scrub ok 2024-04-14T15:16:18.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:17 smithi090 ceph-mon[41670]: 5.a scrub starts 2024-04-14T15:16:18.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:17 smithi090 ceph-mon[41670]: 5.a scrub ok 2024-04-14T15:16:18.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:17 smithi090 ceph-mon[41670]: 2.b scrub starts 2024-04-14T15:16:18.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:17 smithi090 ceph-mon[41670]: 2.b scrub ok 2024-04-14T15:16:18.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:17 smithi090 ceph-mon[41670]: from='client.48811 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi090-93363-53"}]': finished 2024-04-14T15:16:18.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:17 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:16:18.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:17 smithi090 ceph-mon[41670]: osdmap e309: 8 total, 8 up, 8 in 2024-04-14T15:16:18.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:17 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:16:18.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:17 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:16:18.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:17 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/585933469' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi090-93363-54"}]: dispatch 2024-04-14T15:16:18.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:17 smithi090 ceph-mon[41670]: from='client.48835 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi090-93363-54"}]: dispatch 2024-04-14T15:16:18.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:17 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/585933469' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi090-93363-54"}]: dispatch 2024-04-14T15:16:18.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:17 smithi090 ceph-mon[41670]: from='client.48835 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi090-93363-54"}]: dispatch 2024-04-14T15:16:18.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:17 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/585933469' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi090-93363-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:16:18.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:17 smithi090 ceph-mon[41670]: from='client.48835 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi090-93363-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:16:18.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:17 smithi090 ceph-mon[41670]: 6.e scrub starts 2024-04-14T15:16:18.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:17 smithi090 ceph-mon[41670]: 6.e scrub ok 2024-04-14T15:16:18.195 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:17 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:16:18.195 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:17 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:16:18.195 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:17 smithi090 ceph-mon[41670]: 6.0 scrub starts 2024-04-14T15:16:18.195 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:17 smithi090 ceph-mon[41670]: 6.0 scrub ok 2024-04-14T15:16:18.272 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-04-14T15:16:18.273 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-04-14T15:16:18.273 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-04-14T15:16:18.273 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-04-14T15:16:18.273 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-04-14T15:16:18.281 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:18.306 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:18.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:18.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:18.408 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:16:18.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:18.434 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:18.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:16:18.535 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:16:18.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:18.560 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:18.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:16:18.661 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:16:18.687 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:18.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:18.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:16:18.751 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:16:18.789 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:18.813 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:18.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:18.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:18.882 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-14T15:16:18.915 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:16:18.939 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:18.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:18.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:16:18.977 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:18 smithi090 ceph-mon[41670]: pgmap v410: 238 pgs: 32 unknown, 206 active+clean; 4.6 MiB data, 570 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-14T15:16:18.977 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:18 smithi090 ceph-mon[41670]: 5.1a scrub starts 2024-04-14T15:16:18.977 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:18 smithi090 ceph-mon[41670]: 5.1a scrub ok 2024-04-14T15:16:18.977 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:18 smithi090 ceph-mon[41670]: 5.0 scrub starts 2024-04-14T15:16:18.977 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:18 smithi090 ceph-mon[41670]: 5.0 scrub ok 2024-04-14T15:16:18.977 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:18 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-39", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:16:18.977 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:18 smithi090 ceph-mon[41670]: from='client.48835 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi090-93363-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:16:18.977 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:18 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-39"}]: dispatch 2024-04-14T15:16:18.978 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:18 smithi090 ceph-mon[41670]: osdmap e310: 8 total, 8 up, 8 in 2024-04-14T15:16:18.978 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:18 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/585933469' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi090-93363-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi090-93363-54"}]: dispatch 2024-04-14T15:16:18.978 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:18 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-39"}]: dispatch 2024-04-14T15:16:18.978 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:18 smithi090 ceph-mon[41670]: from='client.48835 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi090-93363-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi090-93363-54"}]: dispatch 2024-04-14T15:16:18.978 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:18 smithi090 ceph-mon[41670]: 4.9 scrub starts 2024-04-14T15:16:18.978 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:18 smithi090 ceph-mon[41670]: 4.9 scrub ok 2024-04-14T15:16:18.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:18 smithi090 ceph-mon[30399]: pgmap v410: 238 pgs: 32 unknown, 206 active+clean; 4.6 MiB data, 570 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-14T15:16:18.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:18 smithi090 ceph-mon[30399]: 5.1a scrub starts 2024-04-14T15:16:18.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:18 smithi090 ceph-mon[30399]: 5.1a scrub ok 2024-04-14T15:16:18.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:18 smithi090 ceph-mon[30399]: 5.0 scrub starts 2024-04-14T15:16:18.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:18 smithi090 ceph-mon[30399]: 5.0 scrub ok 2024-04-14T15:16:18.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:18 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-39", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:16:18.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:18 smithi090 ceph-mon[30399]: from='client.48835 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi090-93363-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:16:18.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:18 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-39"}]: dispatch 2024-04-14T15:16:18.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:18 smithi090 ceph-mon[30399]: osdmap e310: 8 total, 8 up, 8 in 2024-04-14T15:16:18.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:18 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/585933469' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi090-93363-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi090-93363-54"}]: dispatch 2024-04-14T15:16:18.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:18 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-39"}]: dispatch 2024-04-14T15:16:18.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:18 smithi090 ceph-mon[30399]: from='client.48835 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi090-93363-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi090-93363-54"}]: dispatch 2024-04-14T15:16:18.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:18 smithi090 ceph-mon[30399]: 4.9 scrub starts 2024-04-14T15:16:18.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:18 smithi090 ceph-mon[30399]: 4.9 scrub ok 2024-04-14T15:16:19.041 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:19.066 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:19.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:19.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:19.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:18 smithi092 ceph-mon[33966]: pgmap v410: 238 pgs: 32 unknown, 206 active+clean; 4.6 MiB data, 570 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-14T15:16:19.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:18 smithi092 ceph-mon[33966]: 5.1a scrub starts 2024-04-14T15:16:19.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:18 smithi092 ceph-mon[33966]: 5.1a scrub ok 2024-04-14T15:16:19.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:18 smithi092 ceph-mon[33966]: 5.0 scrub starts 2024-04-14T15:16:19.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:18 smithi092 ceph-mon[33966]: 5.0 scrub ok 2024-04-14T15:16:19.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:18 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-39", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:16:19.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:18 smithi092 ceph-mon[33966]: from='client.48835 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi090-93363-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:16:19.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:18 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-39"}]: dispatch 2024-04-14T15:16:19.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:18 smithi092 ceph-mon[33966]: osdmap e310: 8 total, 8 up, 8 in 2024-04-14T15:16:19.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:18 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/585933469' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi090-93363-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi090-93363-54"}]: dispatch 2024-04-14T15:16:19.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:18 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-39"}]: dispatch 2024-04-14T15:16:19.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:18 smithi092 ceph-mon[33966]: from='client.48835 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi090-93363-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi090-93363-54"}]: dispatch 2024-04-14T15:16:19.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:18 smithi092 ceph-mon[33966]: 4.9 scrub starts 2024-04-14T15:16:19.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:18 smithi092 ceph-mon[33966]: 4.9 scrub ok 2024-04-14T15:16:19.125 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:16:19.169 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:16:19.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:19.194 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:19.194 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:16:19.295 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:16:19.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:19.321 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:19.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:16:19.422 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:19.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:19.446 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:19.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:19.547 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:16:19.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:19.571 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:19.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:16:19.672 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:16:19.696 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:19.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:19.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:16:19.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:19.777+0000 7f63eff53700 1 -- 172.21.15.90:0/907414656 >> v1:172.21.15.90:6790/0 conn(0x7f63e8102460 legacy=0x7f63e8102840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:19.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:19.777+0000 7f63eff53700 1 -- 172.21.15.90:0/907414656 shutdown_connections 2024-04-14T15:16:19.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:19.777+0000 7f63eff53700 1 -- 172.21.15.90:0/907414656 >> 172.21.15.90:0/907414656 conn(0x7f63e8076ad0 msgr2=0x7f63e8076ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:16:19.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:19.777+0000 7f63eff53700 1 -- 172.21.15.90:0/907414656 shutdown_connections 2024-04-14T15:16:19.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:19.777+0000 7f63eff53700 1 -- 172.21.15.90:0/907414656 wait complete. 2024-04-14T15:16:19.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:19.778+0000 7f63eff53700 1 librados: starting msgr at 2024-04-14T15:16:19.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:19.778+0000 7f63eff53700 1 librados: starting objecter 2024-04-14T15:16:19.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:19.778+0000 7f63eff53700 1 Processor -- start 2024-04-14T15:16:19.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:19.778+0000 7f63eff53700 1 -- start start 2024-04-14T15:16:19.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:19.778+0000 7f63eff53700 1 librados: setting wanted keys 2024-04-14T15:16:19.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:19.778+0000 7f63eff53700 1 librados: calling monclient init 2024-04-14T15:16:19.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:19.778+0000 7f63eff53700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f63e8107100 con 0x7f63e810e230 2024-04-14T15:16:19.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:19.778+0000 7f63eff53700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f63e81a7e00 con 0x7f63e810a6c0 2024-04-14T15:16:19.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:19.778+0000 7f63eff53700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f63e81a8fe0 con 0x7f63e8102460 2024-04-14T15:16:19.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:19.778+0000 7f63edcef700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f63e8102460 0x7f63e8103f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:54728/0 (socket says 172.21.15.90:54728) 2024-04-14T15:16:19.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:19.778+0000 7f63edcef700 1 -- 172.21.15.90:0/1574854201 learned_addr learned my addr 172.21.15.90:0/1574854201 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:16:19.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:19.778+0000 7f63ee4f0700 1 --1- 172.21.15.90:0/1574854201 >> v1:172.21.15.90:6789/0 conn(0x7f63e810e230 0x7f63e81a56d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:52034/0 (socket says 172.21.15.90:52034) 2024-04-14T15:16:19.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:19.778+0000 7f63deffd700 10 client.?.objecter ms_handle_connect 0x7f63e810e230 2024-04-14T15:16:19.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:19.778+0000 7f63deffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:16:19.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:19.778+0000 7f63deffd700 10 client.?.objecter ms_handle_connect 0x7f63e8102460 2024-04-14T15:16:19.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:19.778+0000 7f63deffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:16:19.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:19.778+0000 7f63deffd700 10 client.?.objecter ms_handle_connect 0x7f63e810a6c0 2024-04-14T15:16:19.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:19.778+0000 7f63deffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:16:19.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:19.778+0000 7f63deffd700 1 -- 172.21.15.90:0/1574854201 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4105064785 0 0) 0x7f63e8107100 con 0x7f63e810e230 2024-04-14T15:16:19.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:19.778+0000 7f63deffd700 1 -- 172.21.15.90:0/1574854201 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f63d40031f0 con 0x7f63e810e230 2024-04-14T15:16:19.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:19.778+0000 7f63deffd700 1 -- 172.21.15.90:0/1574854201 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1015112681 0 0) 0x7f63e81a7e00 con 0x7f63e810a6c0 2024-04-14T15:16:19.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:19.778+0000 7f63deffd700 1 -- 172.21.15.90:0/1574854201 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f63e8107100 con 0x7f63e810a6c0 2024-04-14T15:16:19.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:19.778+0000 7f63deffd700 1 -- 172.21.15.90:0/1574854201 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 806841332 0 0) 0x7f63e81a8fe0 con 0x7f63e8102460 2024-04-14T15:16:19.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:19.778+0000 7f63deffd700 1 -- 172.21.15.90:0/1574854201 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f63e81a7e00 con 0x7f63e8102460 2024-04-14T15:16:19.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:19.779+0000 7f63deffd700 1 -- 172.21.15.90:0/1574854201 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3259199803 0 0) 0x7f63e8107100 con 0x7f63e810a6c0 2024-04-14T15:16:19.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:19.779+0000 7f63deffd700 1 -- 172.21.15.90:0/1574854201 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f63e81a8fe0 con 0x7f63e810a6c0 2024-04-14T15:16:19.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:19.779+0000 7f63deffd700 1 -- 172.21.15.90:0/1574854201 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f63d80017b0 con 0x7f63e810a6c0 2024-04-14T15:16:19.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:19.779+0000 7f63deffd700 1 -- 172.21.15.90:0/1574854201 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3349084474 0 0) 0x7f63d40031f0 con 0x7f63e810e230 2024-04-14T15:16:19.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:19.779+0000 7f63deffd700 1 -- 172.21.15.90:0/1574854201 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f63e8107100 con 0x7f63e810e230 2024-04-14T15:16:19.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:19.779+0000 7f63deffd700 1 -- 172.21.15.90:0/1574854201 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f63e0001c10 con 0x7f63e810e230 2024-04-14T15:16:19.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:19.779+0000 7f63deffd700 1 -- 172.21.15.90:0/1574854201 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 378384412 0 0) 0x7f63e81a7e00 con 0x7f63e8102460 2024-04-14T15:16:19.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:19.779+0000 7f63deffd700 1 -- 172.21.15.90:0/1574854201 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f63d40031f0 con 0x7f63e8102460 2024-04-14T15:16:19.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:19.779+0000 7f63deffd700 1 -- 172.21.15.90:0/1574854201 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f63e40015d0 con 0x7f63e8102460 2024-04-14T15:16:19.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:19.779+0000 7f63deffd700 1 -- 172.21.15.90:0/1574854201 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1258782608 0 0) 0x7f63e81a8fe0 con 0x7f63e810a6c0 2024-04-14T15:16:19.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:19.779+0000 7f63deffd700 1 -- 172.21.15.90:0/1574854201 >> v1:172.21.15.90:6790/0 conn(0x7f63e8102460 legacy=0x7f63e8103f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:19.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:19.779+0000 7f63deffd700 1 -- 172.21.15.90:0/1574854201 >> v1:172.21.15.90:6789/0 conn(0x7f63e810e230 legacy=0x7f63e81a56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:19.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:19.779+0000 7f63deffd700 1 -- 172.21.15.90:0/1574854201 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63e81aa1c0 con 0x7f63e810a6c0 2024-04-14T15:16:19.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:19.779+0000 7f63eff53700 1 -- 172.21.15.90:0/1574854201 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f63e81a9210 con 0x7f63e810a6c0 2024-04-14T15:16:19.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:19.779+0000 7f63eff53700 10 client.48841.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:16:19.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:19.779+0000 7f63eff53700 1 -- 172.21.15.90:0/1574854201 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f63e81a96d0 con 0x7f63e810a6c0 2024-04-14T15:16:19.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:19.779+0000 7f63eff53700 1 librados: init done 2024-04-14T15:16:19.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:19.779+0000 7f63deffd700 1 -- 172.21.15.90:0/1574854201 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f63d8003b20 con 0x7f63e810a6c0 2024-04-14T15:16:19.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:19.779+0000 7f63deffd700 1 -- 172.21.15.90:0/1574854201 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f63d8004a00 con 0x7f63e810a6c0 2024-04-14T15:16:19.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:19.780+0000 7f63eff53700 1 -- 172.21.15.90:0/1574854201 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63cc005120 con 0x7f63e810a6c0 2024-04-14T15:16:19.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:19.780+0000 7f63deffd700 1 -- 172.21.15.90:0/1574854201 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f63d8003680 con 0x7f63e810a6c0 2024-04-14T15:16:19.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:19.781+0000 7f63deffd700 1 -- 172.21.15.90:0/1574854201 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(311..311 src has 51..311) ==== 8528+0+0 (unknown 3869185159 0 0) 0x7f63d8089a00 con 0x7f63e810a6c0 2024-04-14T15:16:19.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:19.781+0000 7f63deffd700 10 client.48841.objecter ms_dispatch 0x7f63e8001550 osd_map(311..311 src has 51..311) 2024-04-14T15:16:19.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:19.781+0000 7f63deffd700 3 client.48841.objecter handle_osd_map got epochs [311,311] > 0 2024-04-14T15:16:19.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:19.781+0000 7f63deffd700 3 client.48841.objecter handle_osd_map decoding full epoch 311 2024-04-14T15:16:19.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:19.781+0000 7f63deffd700 20 client.48841.objecter dump_active .. 0 homeless 2024-04-14T15:16:19.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:19.781+0000 7f63deffd700 10 client.48841.objecter ms_handle_connect 0x7f63d4071390 2024-04-14T15:16:19.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:19.786+0000 7f63deffd700 1 -- 172.21.15.90:0/1574854201 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f63d80562a0 con 0x7f63e810a6c0 2024-04-14T15:16:19.798 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:16:19.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:19.822 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:19.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:16:19.867 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-14T15:16:19.923 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:16:19.941 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:19 smithi090 ceph-mon[30399]: 2.5 scrub starts 2024-04-14T15:16:19.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:19 smithi090 ceph-mon[30399]: 2.5 scrub ok 2024-04-14T15:16:19.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:19 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-39"}]': finished 2024-04-14T15:16:19.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:19 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-39", "mode": "writeback"}]: dispatch 2024-04-14T15:16:19.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:19 smithi090 ceph-mon[30399]: osdmap e311: 8 total, 8 up, 8 in 2024-04-14T15:16:19.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:19 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-39", "mode": "writeback"}]: dispatch 2024-04-14T15:16:19.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:19 smithi090 ceph-mon[30399]: 5.18 scrub starts 2024-04-14T15:16:19.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:19 smithi090 ceph-mon[30399]: 5.18 scrub ok 2024-04-14T15:16:19.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:19 smithi090 ceph-mon[30399]: 2.2 scrub starts 2024-04-14T15:16:19.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:19 smithi090 ceph-mon[30399]: 2.2 scrub ok 2024-04-14T15:16:19.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:19 smithi090 ceph-mon[30399]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:16:19.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:19 smithi090 ceph-mon[30399]: from='client.48835 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi090-93363-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi090-93363-54"}]': finished 2024-04-14T15:16:19.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:19 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-39", "mode": "writeback"}]': finished 2024-04-14T15:16:19.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:19 smithi090 ceph-mon[30399]: osdmap e312: 8 total, 8 up, 8 in 2024-04-14T15:16:19.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:19 smithi090 ceph-mon[41670]: 2.5 scrub starts 2024-04-14T15:16:19.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:19 smithi090 ceph-mon[41670]: 2.5 scrub ok 2024-04-14T15:16:19.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:19 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-39"}]': finished 2024-04-14T15:16:19.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:19 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-39", "mode": "writeback"}]: dispatch 2024-04-14T15:16:19.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:19 smithi090 ceph-mon[41670]: osdmap e311: 8 total, 8 up, 8 in 2024-04-14T15:16:19.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:19 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-39", "mode": "writeback"}]: dispatch 2024-04-14T15:16:19.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:19 smithi090 ceph-mon[41670]: 5.18 scrub starts 2024-04-14T15:16:19.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:19 smithi090 ceph-mon[41670]: 5.18 scrub ok 2024-04-14T15:16:19.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:19 smithi090 ceph-mon[41670]: 2.2 scrub starts 2024-04-14T15:16:19.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:19 smithi090 ceph-mon[41670]: 2.2 scrub ok 2024-04-14T15:16:19.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:19 smithi090 ceph-mon[41670]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:16:19.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:19 smithi090 ceph-mon[41670]: from='client.48835 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi090-93363-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi090-93363-54"}]': finished 2024-04-14T15:16:19.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:19 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-39", "mode": "writeback"}]': finished 2024-04-14T15:16:19.944 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:19 smithi090 ceph-mon[41670]: osdmap e312: 8 total, 8 up, 8 in 2024-04-14T15:16:19.953 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:19.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:19.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:16:20.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.033+0000 7f63eff53700 1 -- 172.21.15.90:0/1574854201 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd pg-upmap-items", "pgid": "110.b", "id": [7, 2, 6, 5, 3, 0]} v 0) -- 0x7f63cc005410 con 0x7f63e810a6c0 2024-04-14T15:16:20.055 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:20.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:20.079 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:20.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:20.180 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:20.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:20.205 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:20.205 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:20.280 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:19 smithi092 ceph-mon[33966]: 2.5 scrub starts 2024-04-14T15:16:20.281 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:19 smithi092 ceph-mon[33966]: 2.5 scrub ok 2024-04-14T15:16:20.281 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:19 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-39"}]': finished 2024-04-14T15:16:20.281 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:19 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-39", "mode": "writeback"}]: dispatch 2024-04-14T15:16:20.281 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:19 smithi092 ceph-mon[33966]: osdmap e311: 8 total, 8 up, 8 in 2024-04-14T15:16:20.281 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:19 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-39", "mode": "writeback"}]: dispatch 2024-04-14T15:16:20.282 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:19 smithi092 ceph-mon[33966]: 5.18 scrub starts 2024-04-14T15:16:20.282 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:19 smithi092 ceph-mon[33966]: 5.18 scrub ok 2024-04-14T15:16:20.282 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:19 smithi092 ceph-mon[33966]: 2.2 scrub starts 2024-04-14T15:16:20.282 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:19 smithi092 ceph-mon[33966]: 2.2 scrub ok 2024-04-14T15:16:20.282 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:19 smithi092 ceph-mon[33966]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:16:20.282 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:19 smithi092 ceph-mon[33966]: from='client.48835 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi090-93363-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi090-93363-54"}]': finished 2024-04-14T15:16:20.283 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:19 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-39", "mode": "writeback"}]': finished 2024-04-14T15:16:20.283 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:19 smithi092 ceph-mon[33966]: osdmap e312: 8 total, 8 up, 8 in 2024-04-14T15:16:20.305 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:16:20.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:20.331 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:20.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:16:20.432 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:16:20.457 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:20.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:20.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:16:20.559 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:16:20.583 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:20.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:20.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:16:20.685 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:20.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:20.709 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:20.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:20.796 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-04-14T15:16:20.796 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-04-14T15:16:20.796 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-04-14T15:16:20.797 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-04-14T15:16:20.797 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.797 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.24982.0:2 28.1a 28:5c6776db:::0.obj:head [write 0~103] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-04-14T15:16:20.797 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:55.059506+0000", 2024-04-14T15:16:20.797 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 205.73443301399999, 2024-04-14T15:16:20.797 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4703437080000001, 2024-04-14T15:16:20.797 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:16:20.797 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:16:20.797 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:16:20.797 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:16:20.797 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.24982", 2024-04-14T15:16:20.797 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/3634669511", 2024-04-14T15:16:20.797 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 2 2024-04-14T15:16:20.797 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.797 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:16:20.798 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.798 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:16:20.798 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.059506+0000", 2024-04-14T15:16:20.798 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:20.798 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.798 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.798 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:16:20.798 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.059506+0000", 2024-04-14T15:16:20.798 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:20.798 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.798 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.798 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:16:20.798 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.059507+0000", 2024-04-14T15:16:20.799 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.635e-06 2024-04-14T15:16:20.799 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.799 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.799 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:16:20.799 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.059514+0000", 2024-04-14T15:16:20.799 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1709999999999996e-06 2024-04-14T15:16:20.799 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.799 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.799 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:16:20.799 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.059524+0000", 2024-04-14T15:16:20.799 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.5799999999999998e-06 2024-04-14T15:16:20.799 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.799 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.799 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:16:20.799 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.059535+0000", 2024-04-14T15:16:20.800 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0815999999999999e-05 2024-04-14T15:16:20.800 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.800 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.800 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:20.800 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.096886+0000", 2024-04-14T15:16:20.800 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.037350718999999998 2024-04-14T15:16:20.800 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.800 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.800 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:16:20.800 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.096894+0000", 2024-04-14T15:16:20.800 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3769999999999996e-06 2024-04-14T15:16:20.800 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.800 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.800 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:20.800 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.402177+0000", 2024-04-14T15:16:20.801 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.305282888 2024-04-14T15:16:20.801 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.801 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.801 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:16:20.801 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.402442+0000", 2024-04-14T15:16:20.801 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0002654 2024-04-14T15:16:20.801 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.801 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.801 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1,2", 2024-04-14T15:16:20.801 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.402597+0000", 2024-04-14T15:16:20.801 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000154919 2024-04-14T15:16:20.801 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.801 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.801 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:16:20.801 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.410479+0000", 2024-04-14T15:16:20.802 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0078818589999999997 2024-04-14T15:16:20.802 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.802 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.802 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:16:20.802 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.482683+0000", 2024-04-14T15:16:20.802 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.072203807999999994 2024-04-14T15:16:20.802 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.802 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.802 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:16:20.802 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.529775+0000", 2024-04-14T15:16:20.802 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.047092569000000001 2024-04-14T15:16:20.802 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.802 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.802 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:16:20.803 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.529810+0000", 2024-04-14T15:16:20.803 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4867999999999999e-05 2024-04-14T15:16:20.803 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.803 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.803 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:16:20.803 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.529849+0000", 2024-04-14T15:16:20.803 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9099e-05 2024-04-14T15:16:20.803 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:20.803 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:16:20.803 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:20.803 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.803 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.803 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25030.0:2 35.6 35:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-04-14T15:16:20.803 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:55.063884+0000", 2024-04-14T15:16:20.803 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 205.730055002, 2024-04-14T15:16:20.804 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.452423899, 2024-04-14T15:16:20.804 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:16:20.804 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:16:20.804 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:16:20.804 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:16:20.804 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25030", 2024-04-14T15:16:20.804 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/2062815860", 2024-04-14T15:16:20.804 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 2 2024-04-14T15:16:20.804 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.804 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:16:20.804 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.804 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:16:20.804 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.063884+0000", 2024-04-14T15:16:20.804 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:20.804 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.805 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.805 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:16:20.805 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.063884+0000", 2024-04-14T15:16:20.805 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:20.805 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.805 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.805 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:16:20.805 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.063885+0000", 2024-04-14T15:16:20.805 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.686e-06 2024-04-14T15:16:20.805 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.805 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.805 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:16:20.805 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.063891+0000", 2024-04-14T15:16:20.805 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4940000000000003e-06 2024-04-14T15:16:20.805 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.806 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.806 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:16:20.806 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.063900+0000", 2024-04-14T15:16:20.806 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0799999999999995e-06 2024-04-14T15:16:20.806 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.806 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.806 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:16:20.806 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.063911+0000", 2024-04-14T15:16:20.806 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0674e-05 2024-04-14T15:16:20.806 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.806 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.806 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:20.806 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.117251+0000", 2024-04-14T15:16:20.806 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.053340769000000003 2024-04-14T15:16:20.807 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.807 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.807 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:16:20.807 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.117261+0000", 2024-04-14T15:16:20.807 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.6150000000000003e-06 2024-04-14T15:16:20.807 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.807 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.807 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:20.807 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.423618+0000", 2024-04-14T15:16:20.807 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3063569589999999 2024-04-14T15:16:20.807 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.807 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.807 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:16:20.807 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.423744+0000", 2024-04-14T15:16:20.807 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000126191 2024-04-14T15:16:20.808 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.808 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.808 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,1", 2024-04-14T15:16:20.808 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.424326+0000", 2024-04-14T15:16:20.808 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00058220199999999998 2024-04-14T15:16:20.808 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.808 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.808 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:16:20.808 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.426268+0000", 2024-04-14T15:16:20.808 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0019420240000000001 2024-04-14T15:16:20.808 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.808 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.808 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:16:20.808 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.513796+0000", 2024-04-14T15:16:20.808 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.087527758999999997 2024-04-14T15:16:20.808 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.809 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.809 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:16:20.809 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.516224+0000", 2024-04-14T15:16:20.809 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0024277690000000002 2024-04-14T15:16:20.809 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.809 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.809 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:16:20.809 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.516268+0000", 2024-04-14T15:16:20.809 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3918999999999997e-05 2024-04-14T15:16:20.809 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.809 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.809 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:16:20.809 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.516308+0000", 2024-04-14T15:16:20.809 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9758000000000001e-05 2024-04-14T15:16:20.810 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:20.810 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:16:20.810 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:20.810 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.810 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.810 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.24817.0:2 11.1 11:8049a5cb:::exist:head [writefull 0~5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-04-14T15:16:20.810 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:55.134042+0000", 2024-04-14T15:16:20.810 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 205.65989687000001, 2024-04-14T15:16:20.810 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3821967850000001, 2024-04-14T15:16:20.810 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:16:20.810 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:16:20.810 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:16:20.810 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:16:20.810 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.24817", 2024-04-14T15:16:20.811 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/4078202427", 2024-04-14T15:16:20.811 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 2 2024-04-14T15:16:20.811 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.811 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:16:20.811 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.811 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:16:20.811 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.134042+0000", 2024-04-14T15:16:20.811 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:20.811 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.811 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.811 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:16:20.811 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.134042+0000", 2024-04-14T15:16:20.811 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:20.811 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.811 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.811 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:16:20.812 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.134044+0000", 2024-04-14T15:16:20.812 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0200000000000001e-06 2024-04-14T15:16:20.812 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.812 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.812 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:16:20.812 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.134051+0000", 2024-04-14T15:16:20.812 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8170000000000001e-06 2024-04-14T15:16:20.812 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.812 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.812 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:16:20.812 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.134062+0000", 2024-04-14T15:16:20.812 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1314999999999999e-05 2024-04-14T15:16:20.812 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.812 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.812 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:16:20.813 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.134073+0000", 2024-04-14T15:16:20.813 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0934999999999999e-05 2024-04-14T15:16:20.813 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.813 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.813 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:20.813 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.134128+0000", 2024-04-14T15:16:20.813 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4901000000000002e-05 2024-04-14T15:16:20.813 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.813 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.813 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:16:20.813 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.134138+0000", 2024-04-14T15:16:20.813 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0579e-05 2024-04-14T15:16:20.813 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.813 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.813 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:20.814 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.418676+0000", 2024-04-14T15:16:20.814 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.284537196 2024-04-14T15:16:20.814 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.814 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.814 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:16:20.814 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.418803+0000", 2024-04-14T15:16:20.814 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000127188 2024-04-14T15:16:20.814 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.814 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.814 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,3", 2024-04-14T15:16:20.814 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.418935+0000", 2024-04-14T15:16:20.814 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000132366 2024-04-14T15:16:20.814 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.815 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.815 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:16:20.815 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.421117+0000", 2024-04-14T15:16:20.815 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0021817949999999998 2024-04-14T15:16:20.815 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.815 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.815 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:16:20.815 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.492642+0000", 2024-04-14T15:16:20.815 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.071524852 2024-04-14T15:16:20.815 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.815 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.815 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:16:20.815 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.516135+0000", 2024-04-14T15:16:20.815 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.023493518000000001 2024-04-14T15:16:20.815 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.816 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.816 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:16:20.816 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.516180+0000", 2024-04-14T15:16:20.816 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.4295000000000002e-05 2024-04-14T15:16:20.816 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.816 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.816 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:16:20.816 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.516239+0000", 2024-04-14T15:16:20.816 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9008000000000002e-05 2024-04-14T15:16:20.816 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:20.816 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:16:20.816 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:20.816 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.816 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.816 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.24823.0:2 12.16 12:6b1560c3:test-rados-api-smithi090-93436-2::foo:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-04-14T15:16:20.817 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:56.073935+0000", 2024-04-14T15:16:20.817 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 204.720003636, 2024-04-14T15:16:20.817 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.46111299700000002, 2024-04-14T15:16:20.817 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:16:20.817 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:16:20.817 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:16:20.817 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:16:20.817 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.24823", 2024-04-14T15:16:20.817 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/3828074346", 2024-04-14T15:16:20.817 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 2 2024-04-14T15:16:20.817 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.817 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:16:20.817 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.817 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:16:20.817 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.073935+0000", 2024-04-14T15:16:20.818 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:20.818 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.818 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.818 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:16:20.818 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.073935+0000", 2024-04-14T15:16:20.818 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:20.818 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.818 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.818 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:16:20.818 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.073937+0000", 2024-04-14T15:16:20.818 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.624e-06 2024-04-14T15:16:20.818 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.818 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.818 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:16:20.819 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.073943+0000", 2024-04-14T15:16:20.819 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0379999999999996e-06 2024-04-14T15:16:20.819 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.819 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.819 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:16:20.819 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.073953+0000", 2024-04-14T15:16:20.819 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0042e-05 2024-04-14T15:16:20.819 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.819 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.819 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:16:20.819 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.073962+0000", 2024-04-14T15:16:20.819 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.38e-06 2024-04-14T15:16:20.819 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.819 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.819 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:20.820 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.074024+0000", 2024-04-14T15:16:20.820 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1587999999999994e-05 2024-04-14T15:16:20.820 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.820 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.820 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op must wait for map", 2024-04-14T15:16:20.820 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.074030+0000", 2024-04-14T15:16:20.820 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0560000000000001e-06 2024-04-14T15:16:20.820 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.820 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.820 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:20.820 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.130111+0000", 2024-04-14T15:16:20.820 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.056080928000000002 2024-04-14T15:16:20.820 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.820 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.820 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:16:20.821 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.130119+0000", 2024-04-14T15:16:20.821 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.7250000000000003e-06 2024-04-14T15:16:20.821 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.821 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.821 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:20.821 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.406792+0000", 2024-04-14T15:16:20.821 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.27667239399999999 2024-04-14T15:16:20.821 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.821 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.821 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:16:20.821 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.406922+0000", 2024-04-14T15:16:20.821 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013003900000000001 2024-04-14T15:16:20.821 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.821 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.821 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 2,4", 2024-04-14T15:16:20.822 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.408321+0000", 2024-04-14T15:16:20.822 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0013989849999999999 2024-04-14T15:16:20.822 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.822 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.822 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:16:20.822 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.411890+0000", 2024-04-14T15:16:20.822 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0035687280000000002 2024-04-14T15:16:20.822 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.822 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.822 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:16:20.822 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.423562+0000", 2024-04-14T15:16:20.822 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.011672633999999999 2024-04-14T15:16:20.822 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.822 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.823 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:16:20.824 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:16:20.824 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.534960+0000", 2024-04-14T15:16:20.824 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.111397605 2024-04-14T15:16:20.824 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.824 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.824 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:16:20.825 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.535009+0000", 2024-04-14T15:16:20.825 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.9200999999999999e-05 2024-04-14T15:16:20.825 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.825 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.825 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:16:20.825 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.535048+0000", 2024-04-14T15:16:20.825 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9029999999999997e-05 2024-04-14T15:16:20.825 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:20.825 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:16:20.825 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:20.825 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.825 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.825 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.24802.0:3 8.9 8:9294e480:nspace::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-04-14T15:16:20.825 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:56.432330+0000", 2024-04-14T15:16:20.825 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 204.36160887599999, 2024-04-14T15:16:20.826 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.084297617000000005, 2024-04-14T15:16:20.826 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:16:20.826 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:16:20.826 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:16:20.826 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:16:20.826 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.24802", 2024-04-14T15:16:20.826 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/1106328377", 2024-04-14T15:16:20.826 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 3 2024-04-14T15:16:20.826 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.826 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:16:20.826 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.826 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:16:20.826 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.432330+0000", 2024-04-14T15:16:20.827 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:20.827 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.827 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.827 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:16:20.827 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.432330+0000", 2024-04-14T15:16:20.827 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:20.827 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.827 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.827 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:16:20.827 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.432332+0000", 2024-04-14T15:16:20.827 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.057e-06 2024-04-14T15:16:20.827 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.827 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.827 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:16:20.827 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.432339+0000", 2024-04-14T15:16:20.828 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9349999999999997e-06 2024-04-14T15:16:20.828 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.828 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.828 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:16:20.828 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.432353+0000", 2024-04-14T15:16:20.828 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4586999999999999e-05 2024-04-14T15:16:20.828 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.828 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.828 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:16:20.828 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.432363+0000", 2024-04-14T15:16:20.828 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.3710000000000006e-06 2024-04-14T15:16:20.828 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.828 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.828 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:20.828 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.432442+0000", 2024-04-14T15:16:20.829 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.9568999999999997e-05 2024-04-14T15:16:20.829 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.829 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.829 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:16:20.829 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.432579+0000", 2024-04-14T15:16:20.829 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013618699999999999 2024-04-14T15:16:20.829 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.829 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.829 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,5", 2024-04-14T15:16:20.829 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.432721+0000", 2024-04-14T15:16:20.829 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014257199999999999 2024-04-14T15:16:20.829 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.829 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.829 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:16:20.829 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.433101+0000", 2024-04-14T15:16:20.830 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00038030399999999998 2024-04-14T15:16:20.830 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.830 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.830 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:16:20.830 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.433736+0000", 2024-04-14T15:16:20.830 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00063467899999999997 2024-04-14T15:16:20.830 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.830 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.830 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:16:20.830 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.516517+0000", 2024-04-14T15:16:20.830 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.082780949000000006 2024-04-14T15:16:20.830 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.830 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.830 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:16:20.830 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.516588+0000", 2024-04-14T15:16:20.831 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1347000000000007e-05 2024-04-14T15:16:20.831 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.831 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.831 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:16:20.831 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.516627+0000", 2024-04-14T15:16:20.831 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9060000000000002e-05 2024-04-14T15:16:20.831 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:20.831 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:16:20.831 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:20.831 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.831 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.831 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.24817.0:9 11.1 11:8049a5cb:::exist:head [write 0~5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-04-14T15:16:20.831 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:56.521862+0000", 2024-04-14T15:16:20.831 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 204.27207676099999, 2024-04-14T15:16:20.831 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.061932165999999997, 2024-04-14T15:16:20.832 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:16:20.832 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:16:20.832 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:16:20.832 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:16:20.832 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.24817", 2024-04-14T15:16:20.832 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/4078202427", 2024-04-14T15:16:20.832 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 9 2024-04-14T15:16:20.832 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.832 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:16:20.832 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.832 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:16:20.832 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.521862+0000", 2024-04-14T15:16:20.832 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:20.832 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.833 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.833 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:16:20.833 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.521862+0000", 2024-04-14T15:16:20.833 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:20.833 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.833 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.833 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:16:20.833 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.521863+0000", 2024-04-14T15:16:20.833 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.7499999999999999e-07 2024-04-14T15:16:20.833 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.833 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.833 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:16:20.833 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.521866+0000", 2024-04-14T15:16:20.833 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6629999999999999e-06 2024-04-14T15:16:20.833 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.834 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.834 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:16:20.834 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.521873+0000", 2024-04-14T15:16:20.834 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9519999999999997e-06 2024-04-14T15:16:20.834 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.834 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.834 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:16:20.834 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.521878+0000", 2024-04-14T15:16:20.834 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5800000000000002e-06 2024-04-14T15:16:20.834 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.834 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.834 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:20.834 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.521908+0000", 2024-04-14T15:16:20.834 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0392999999999999e-05 2024-04-14T15:16:20.834 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.835 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.835 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:16:20.835 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.521951+0000", 2024-04-14T15:16:20.835 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2981000000000001e-05 2024-04-14T15:16:20.835 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.835 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.835 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,3", 2024-04-14T15:16:20.835 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.522033+0000", 2024-04-14T15:16:20.835 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1366999999999995e-05 2024-04-14T15:16:20.835 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.835 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.835 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:16:20.835 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.522297+0000", 2024-04-14T15:16:20.835 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00026413400000000002 2024-04-14T15:16:20.836 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.836 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.836 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:16:20.836 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.542902+0000", 2024-04-14T15:16:20.836 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.020604852999999999 2024-04-14T15:16:20.836 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.836 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.836 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:16:20.836 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.583730+0000", 2024-04-14T15:16:20.836 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.040828873000000002 2024-04-14T15:16:20.836 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.836 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.836 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:16:20.836 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.583770+0000", 2024-04-14T15:16:20.837 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9169999999999999e-05 2024-04-14T15:16:20.837 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.837 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.837 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:16:20.837 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.583794+0000", 2024-04-14T15:16:20.837 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.4425000000000001e-05 2024-04-14T15:16:20.837 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:20.837 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:16:20.837 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:20.837 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.837 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.837 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25264.0:2 61.1b 61:d96a01e0:::testobj:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-04-14T15:16:20.837 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:58.078847+0000", 2024-04-14T15:16:20.837 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 202.715092145, 2024-04-14T15:16:20.837 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1975038659999999, 2024-04-14T15:16:20.838 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:16:20.838 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:16:20.838 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:16:20.838 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:16:20.838 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25264", 2024-04-14T15:16:20.838 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/4109623542", 2024-04-14T15:16:20.838 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 2 2024-04-14T15:16:20.838 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.838 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:16:20.838 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.838 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:16:20.838 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.078847+0000", 2024-04-14T15:16:20.838 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:20.838 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.839 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.839 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:16:20.839 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.078847+0000", 2024-04-14T15:16:20.839 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:20.839 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.839 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.839 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:16:20.839 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.078848+0000", 2024-04-14T15:16:20.839 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6589999999999999e-06 2024-04-14T15:16:20.839 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.839 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.839 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:16:20.839 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.078855+0000", 2024-04-14T15:16:20.839 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4339999999999997e-06 2024-04-14T15:16:20.839 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.840 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.840 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:16:20.840 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.078864+0000", 2024-04-14T15:16:20.840 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0969999999999995e-06 2024-04-14T15:16:20.840 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.840 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.840 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:16:20.840 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.078871+0000", 2024-04-14T15:16:20.840 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.7740000000000004e-06 2024-04-14T15:16:20.840 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.840 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.840 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:20.840 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.091465+0000", 2024-04-14T15:16:20.840 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.012593582000000001 2024-04-14T15:16:20.840 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.841 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.841 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:16:20.841 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.091473+0000", 2024-04-14T15:16:20.841 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.9200000000000004e-06 2024-04-14T15:16:20.841 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.841 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.841 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:20.841 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.240640+0000", 2024-04-14T15:16:20.841 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1491673760000001 2024-04-14T15:16:20.841 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.841 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.841 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:16:20.841 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.242664+0000", 2024-04-14T15:16:20.841 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0020239630000000002 2024-04-14T15:16:20.841 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.842 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.842 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1,3", 2024-04-14T15:16:20.842 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.242802+0000", 2024-04-14T15:16:20.842 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000138114 2024-04-14T15:16:20.842 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.842 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.842 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:16:20.842 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.245938+0000", 2024-04-14T15:16:20.842 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0031353269999999998 2024-04-14T15:16:20.842 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.842 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.842 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:16:20.842 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.269871+0000", 2024-04-14T15:16:20.842 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.023933184999999999 2024-04-14T15:16:20.842 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.843 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.843 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:16:20.843 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.276247+0000", 2024-04-14T15:16:20.843 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0063756079999999996 2024-04-14T15:16:20.843 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.843 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.843 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:16:20.843 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.276301+0000", 2024-04-14T15:16:20.843 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4725999999999997e-05 2024-04-14T15:16:20.843 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.843 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.843 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:16:20.843 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.276350+0000", 2024-04-14T15:16:20.843 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.9101000000000003e-05 2024-04-14T15:16:20.843 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:20.844 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:16:20.844 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:20.844 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.844 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.844 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.24853.0:2059 16.1 e110/107 16:85b90e3d:test-rados-api-smithi090-93601-16::big:head v 110'7, mlcod=108'5)", 2024-04-14T15:16:20.844 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:58.846418+0000", 2024-04-14T15:16:20.844 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 201.94752029899999, 2024-04-14T15:16:20.844 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.057216272999999998, 2024-04-14T15:16:20.844 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:16:20.844 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:16:20.844 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:16:20.844 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:16:20.844 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.844 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:16:20.844 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.846418+0000", 2024-04-14T15:16:20.845 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:20.845 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.845 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.845 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:16:20.845 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.846418+0000", 2024-04-14T15:16:20.845 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:20.845 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.845 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.845 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:16:20.845 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.846420+0000", 2024-04-14T15:16:20.845 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.319e-06 2024-04-14T15:16:20.845 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.845 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.845 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:16:20.845 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.870729+0000", 2024-04-14T15:16:20.846 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.024309560000000001 2024-04-14T15:16:20.846 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.846 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.846 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:16:20.846 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.870757+0000", 2024-04-14T15:16:20.846 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.7537000000000001e-05 2024-04-14T15:16:20.846 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.846 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.846 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:16:20.846 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.870787+0000", 2024-04-14T15:16:20.846 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0369e-05 2024-04-14T15:16:20.846 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.846 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.846 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:20.846 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.870884+0000", 2024-04-14T15:16:20.847 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.6343999999999998e-05 2024-04-14T15:16:20.847 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.847 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.847 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:16:20.847 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.870904+0000", 2024-04-14T15:16:20.847 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0271999999999998e-05 2024-04-14T15:16:20.847 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.847 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.847 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:16:20.847 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.903586+0000", 2024-04-14T15:16:20.847 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.032682267000000001 2024-04-14T15:16:20.847 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.847 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.847 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:16:20.847 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.903635+0000", 2024-04-14T15:16:20.848 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8605e-05 2024-04-14T15:16:20.848 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:20.848 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:16:20.848 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:20.848 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.848 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.848 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(osd.0.0:177 16.5 e110/107 -18:a697c087:::temp_16.5_0_24134_1:head v 0'0, mlcod=0'0)", 2024-04-14T15:16:20.848 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:58.919798+0000", 2024-04-14T15:16:20.848 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 201.87414074899999, 2024-04-14T15:16:20.848 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.033355888, 2024-04-14T15:16:20.848 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:16:20.848 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:16:20.848 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:16:20.848 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:16:20.848 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.849 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:16:20.849 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.919798+0000", 2024-04-14T15:16:20.849 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:20.849 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.849 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.849 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:16:20.849 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.919798+0000", 2024-04-14T15:16:20.849 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:20.849 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.849 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.849 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:16:20.849 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.919799+0000", 2024-04-14T15:16:20.849 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5489999999999999e-06 2024-04-14T15:16:20.849 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.849 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.850 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:16:20.850 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.934576+0000", 2024-04-14T15:16:20.850 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.014776432000000001 2024-04-14T15:16:20.850 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.850 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.850 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:16:20.850 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.934602+0000", 2024-04-14T15:16:20.850 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6506e-05 2024-04-14T15:16:20.850 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.850 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.850 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:16:20.850 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.934631+0000", 2024-04-14T15:16:20.850 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8472000000000001e-05 2024-04-14T15:16:20.851 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.851 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.851 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:20.851 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.934708+0000", 2024-04-14T15:16:20.851 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.6637000000000005e-05 2024-04-14T15:16:20.851 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.851 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.851 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:16:20.851 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.934740+0000", 2024-04-14T15:16:20.851 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2301999999999997e-05 2024-04-14T15:16:20.851 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.851 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.851 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:16:20.851 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.953095+0000", 2024-04-14T15:16:20.851 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.018355493000000001 2024-04-14T15:16:20.852 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.852 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.852 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:16:20.852 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.953154+0000", 2024-04-14T15:16:20.852 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.8496999999999999e-05 2024-04-14T15:16:20.852 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:20.852 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:16:20.852 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:20.852 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.852 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.852 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.24853.0:2070 16.12 16:4b6e22cc:test-rados-api-smithi090-93601-17::big2:head [writefull 0~25165828] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-14T15:16:20.852 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:59.435084+0000", 2024-04-14T15:16:20.852 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 201.358854605, 2024-04-14T15:16:20.852 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.089936579000000003, 2024-04-14T15:16:20.852 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:16:20.853 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:16:20.853 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:16:20.853 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:16:20.853 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.24853", 2024-04-14T15:16:20.853 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/3851923713", 2024-04-14T15:16:20.853 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 2070 2024-04-14T15:16:20.853 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.853 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:16:20.853 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.853 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:16:20.853 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.435084+0000", 2024-04-14T15:16:20.853 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:20.853 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.853 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.853 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:16:20.854 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.435084+0000", 2024-04-14T15:16:20.854 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:20.854 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.854 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.854 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:16:20.854 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.435085+0000", 2024-04-14T15:16:20.854 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3710000000000001e-06 2024-04-14T15:16:20.854 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.854 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.854 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:16:20.854 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.458849+0000", 2024-04-14T15:16:20.854 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.023763604000000001 2024-04-14T15:16:20.854 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.854 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.855 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:16:20.855 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.458863+0000", 2024-04-14T15:16:20.855 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4268e-05 2024-04-14T15:16:20.855 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.855 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.855 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:16:20.855 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.458870+0000", 2024-04-14T15:16:20.855 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0820000000000001e-06 2024-04-14T15:16:20.855 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.855 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.855 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:20.855 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.458920+0000", 2024-04-14T15:16:20.855 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.9873000000000002e-05 2024-04-14T15:16:20.855 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.855 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.855 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:16:20.856 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.459019+0000", 2024-04-14T15:16:20.856 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.9025999999999997e-05 2024-04-14T15:16:20.856 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.856 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.856 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,5", 2024-04-14T15:16:20.856 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.459123+0000", 2024-04-14T15:16:20.856 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000103714 2024-04-14T15:16:20.856 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.856 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.856 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:16:20.856 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.520361+0000", 2024-04-14T15:16:20.856 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.061237738999999999 2024-04-14T15:16:20.856 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.856 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.856 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:16:20.857 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.520941+0000", 2024-04-14T15:16:20.857 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00058017500000000005 2024-04-14T15:16:20.857 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.857 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.857 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:16:20.857 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.524944+0000", 2024-04-14T15:16:20.857 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0040031090000000004 2024-04-14T15:16:20.857 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.857 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.857 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:16:20.857 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.524984+0000", 2024-04-14T15:16:20.857 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9792000000000001e-05 2024-04-14T15:16:20.857 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.857 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.857 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:16:20.858 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.525021+0000", 2024-04-14T15:16:20.858 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6826000000000003e-05 2024-04-14T15:16:20.858 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:20.858 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:16:20.858 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:20.858 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.858 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.858 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.36731.0:2 144.1b 144:d96a01e0:::testobj:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-04-14T15:16:20.858 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:13:21.848371+0000", 2024-04-14T15:16:20.858 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 178.94556751499999, 2024-04-14T15:16:20.858 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1470619790000001, 2024-04-14T15:16:20.858 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:16:20.858 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:16:20.858 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:16:20.859 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:16:20.859 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.36731", 2024-04-14T15:16:20.859 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/2940319356", 2024-04-14T15:16:20.859 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 2 2024-04-14T15:16:20.859 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.859 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:16:20.859 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.859 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:16:20.859 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:21.848371+0000", 2024-04-14T15:16:20.859 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:20.859 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.859 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.859 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:16:20.859 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:21.848371+0000", 2024-04-14T15:16:20.859 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:20.859 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.860 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.860 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:16:20.860 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:21.848373+0000", 2024-04-14T15:16:20.860 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0389999999999999e-06 2024-04-14T15:16:20.860 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.860 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.860 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:16:20.860 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:21.848379+0000", 2024-04-14T15:16:20.860 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2820000000000002e-06 2024-04-14T15:16:20.860 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.860 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.860 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:16:20.860 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:21.848389+0000", 2024-04-14T15:16:20.860 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.2080000000000006e-06 2024-04-14T15:16:20.861 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.861 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.861 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:16:20.861 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:21.848404+0000", 2024-04-14T15:16:20.861 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4918e-05 2024-04-14T15:16:20.861 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.861 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.861 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:20.861 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:21.848447+0000", 2024-04-14T15:16:20.861 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3773999999999998e-05 2024-04-14T15:16:20.861 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.861 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.861 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:16:20.861 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:21.848457+0000", 2024-04-14T15:16:20.861 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.1859999999999998e-06 2024-04-14T15:16:20.862 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.862 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.862 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:20.862 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:22.939465+0000", 2024-04-14T15:16:20.862 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.091008709 2024-04-14T15:16:20.862 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.862 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.862 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:16:20.862 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:22.939589+0000", 2024-04-14T15:16:20.862 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000123313 2024-04-14T15:16:20.862 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.862 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.862 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1,4", 2024-04-14T15:16:20.862 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:22.939699+0000", 2024-04-14T15:16:20.862 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011075 2024-04-14T15:16:20.863 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.863 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.863 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:16:20.863 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:22.940452+0000", 2024-04-14T15:16:20.863 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00075242500000000003 2024-04-14T15:16:20.863 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.863 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.863 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:16:20.863 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:22.940632+0000", 2024-04-14T15:16:20.863 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00018008499999999999 2024-04-14T15:16:20.863 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.863 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.863 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:16:20.863 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:22.995315+0000", 2024-04-14T15:16:20.864 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.05468315 2024-04-14T15:16:20.864 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.864 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.864 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:16:20.864 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:22.995378+0000", 2024-04-14T15:16:20.864 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2500000000000001e-05 2024-04-14T15:16:20.864 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.864 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.864 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:16:20.864 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:22.995433+0000", 2024-04-14T15:16:20.864 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5640000000000003e-05 2024-04-14T15:16:20.864 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:20.864 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:16:20.864 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:20.864 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.865 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.865 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.41446.0:2 152.0 152:0384ab8e:::oid:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-04-14T15:16:20.865 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:13:24.858105+0000", 2024-04-14T15:16:20.865 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 175.935833395, 2024-04-14T15:16:20.865 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0159813529999999, 2024-04-14T15:16:20.865 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:16:20.865 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:16:20.865 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:16:20.865 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:16:20.865 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.41446", 2024-04-14T15:16:20.865 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/3264560952", 2024-04-14T15:16:20.865 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 2 2024-04-14T15:16:20.865 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.865 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:16:20.865 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.866 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:16:20.866 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:24.858105+0000", 2024-04-14T15:16:20.866 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:20.866 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.866 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.866 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:16:20.866 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:24.858105+0000", 2024-04-14T15:16:20.866 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:20.866 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.866 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.866 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:16:20.866 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:24.858107+0000", 2024-04-14T15:16:20.866 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0679999999999999e-06 2024-04-14T15:16:20.866 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.867 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.867 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:16:20.867 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:24.858113+0000", 2024-04-14T15:16:20.867 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.327e-06 2024-04-14T15:16:20.867 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.867 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.867 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:16:20.867 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:24.858123+0000", 2024-04-14T15:16:20.867 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0560999999999999e-05 2024-04-14T15:16:20.867 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.867 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.867 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:16:20.867 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:24.858134+0000", 2024-04-14T15:16:20.867 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0522e-05 2024-04-14T15:16:20.867 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.868 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.868 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:20.868 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:24.861997+0000", 2024-04-14T15:16:20.868 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0038629200000000002 2024-04-14T15:16:20.868 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.868 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.868 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:16:20.868 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:24.862004+0000", 2024-04-14T15:16:20.868 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5240000000000003e-06 2024-04-14T15:16:20.868 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.868 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.868 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:20.868 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.853997+0000", 2024-04-14T15:16:20.868 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.99199313700000002 2024-04-14T15:16:20.868 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.869 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.869 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:16:20.869 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.855171+0000", 2024-04-14T15:16:20.869 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0011740120000000001 2024-04-14T15:16:20.869 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.869 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.869 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,4", 2024-04-14T15:16:20.869 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.855307+0000", 2024-04-14T15:16:20.869 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013530200000000001 2024-04-14T15:16:20.869 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.869 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.869 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:16:20.869 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.857608+0000", 2024-04-14T15:16:20.869 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0023010320000000002 2024-04-14T15:16:20.869 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.869 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.870 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:16:20.870 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.858583+0000", 2024-04-14T15:16:20.870 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00097578499999999996 2024-04-14T15:16:20.870 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.870 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.870 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:16:20.870 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.873837+0000", 2024-04-14T15:16:20.870 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.015253697 2024-04-14T15:16:20.870 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.870 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.870 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:16:20.870 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.874020+0000", 2024-04-14T15:16:20.870 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00018252399999999999 2024-04-14T15:16:20.870 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.871 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.871 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:16:20.871 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.874087+0000", 2024-04-14T15:16:20.871 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.6941999999999997e-05 2024-04-14T15:16:20.871 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:20.871 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:16:20.871 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:20.871 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.871 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.871 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.41446.0:4 152.0 152:0384ab8e:::oid:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-04-14T15:16:20.871 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:13:25.879604+0000", 2024-04-14T15:16:20.871 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 174.914334661, 2024-04-14T15:16:20.871 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.033643864000000002, 2024-04-14T15:16:20.871 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:16:20.872 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:16:20.872 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:16:20.872 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:16:20.872 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.41446", 2024-04-14T15:16:20.872 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/3264560952", 2024-04-14T15:16:20.872 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 4 2024-04-14T15:16:20.872 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.872 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:16:20.872 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.872 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:16:20.872 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.879604+0000", 2024-04-14T15:16:20.872 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:20.872 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.872 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.873 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:16:20.873 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.879604+0000", 2024-04-14T15:16:20.873 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:20.873 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.873 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.873 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:16:20.873 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.879605+0000", 2024-04-14T15:16:20.873 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.7300000000000005e-07 2024-04-14T15:16:20.873 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.873 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.873 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:16:20.873 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.879608+0000", 2024-04-14T15:16:20.873 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8909999999999999e-06 2024-04-14T15:16:20.873 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.873 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.874 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:16:20.874 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.879616+0000", 2024-04-14T15:16:20.874 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.4030000000000007e-06 2024-04-14T15:16:20.874 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.874 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.874 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:16:20.874 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.879621+0000", 2024-04-14T15:16:20.874 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.057e-06 2024-04-14T15:16:20.874 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.874 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.874 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:20.874 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.879661+0000", 2024-04-14T15:16:20.874 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0209000000000001e-05 2024-04-14T15:16:20.874 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.875 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.875 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:16:20.875 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.879714+0000", 2024-04-14T15:16:20.875 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2561e-05 2024-04-14T15:16:20.875 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.875 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.875 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,4", 2024-04-14T15:16:20.875 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.879820+0000", 2024-04-14T15:16:20.875 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000105525 2024-04-14T15:16:20.875 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.875 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.875 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:16:20.875 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.880121+0000", 2024-04-14T15:16:20.875 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00030120100000000002 2024-04-14T15:16:20.876 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.876 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.876 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:16:20.876 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.880415+0000", 2024-04-14T15:16:20.876 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00029442999999999998 2024-04-14T15:16:20.876 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.876 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.876 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:16:20.876 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.913166+0000", 2024-04-14T15:16:20.876 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.032750385 2024-04-14T15:16:20.876 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.876 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.876 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:16:20.876 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.913212+0000", 2024-04-14T15:16:20.877 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6768999999999999e-05 2024-04-14T15:16:20.877 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.877 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.877 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:16:20.877 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.913248+0000", 2024-04-14T15:16:20.877 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5559999999999998e-05 2024-04-14T15:16:20.877 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:20.877 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:16:20.877 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:20.877 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.877 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.877 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.38086.0:4 149.1s0 149:83e8722a:test-rados-api-smithi090-93709-12::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-04-14T15:16:20.877 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:13:26.870731+0000", 2024-04-14T15:16:20.877 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 173.92320726599999, 2024-04-14T15:16:20.877 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.089545462000000006, 2024-04-14T15:16:20.878 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:16:20.878 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:16:20.878 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:16:20.878 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:16:20.878 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.38086", 2024-04-14T15:16:20.878 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/2598250370", 2024-04-14T15:16:20.878 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 4 2024-04-14T15:16:20.878 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.878 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:16:20.878 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.878 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:16:20.878 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.870731+0000", 2024-04-14T15:16:20.878 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:20.878 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.878 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.879 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:16:20.879 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.870731+0000", 2024-04-14T15:16:20.879 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:20.879 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.879 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.879 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:16:20.879 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.870733+0000", 2024-04-14T15:16:20.879 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4869999999999999e-06 2024-04-14T15:16:20.879 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.879 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.879 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:16:20.879 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.870740+0000", 2024-04-14T15:16:20.879 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0609999999999998e-06 2024-04-14T15:16:20.879 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.879 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.880 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:16:20.880 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.870749+0000", 2024-04-14T15:16:20.880 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.9330000000000007e-06 2024-04-14T15:16:20.880 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.880 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.880 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:16:20.880 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.870759+0000", 2024-04-14T15:16:20.880 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0057999999999999e-05 2024-04-14T15:16:20.880 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.880 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.880 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:20.880 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.871129+0000", 2024-04-14T15:16:20.880 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00037049300000000001 2024-04-14T15:16:20.880 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.880 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.881 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:16:20.881 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.871258+0000", 2024-04-14T15:16:20.881 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012877 2024-04-14T15:16:20.881 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.881 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.881 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_started", 2024-04-14T15:16:20.881 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.873490+0000", 2024-04-14T15:16:20.881 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002231813 2024-04-14T15:16:20.881 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.881 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.881 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_committed", 2024-04-14T15:16:20.881 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.876587+0000", 2024-04-14T15:16:20.881 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.003097074 2024-04-14T15:16:20.881 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.882 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.882 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:16:20.882 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.960175+0000", 2024-04-14T15:16:20.882 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.083587794000000007 2024-04-14T15:16:20.882 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.882 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.882 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:16:20.882 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.960277+0000", 2024-04-14T15:16:20.882 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010197899999999999 2024-04-14T15:16:20.882 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:20.882 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:16:20.882 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:20.882 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.882 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.882 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.38086.0:7 149.6s0 149:7a93d93e:test-rados-api-smithi090-93709-13::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-14T15:16:20.883 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:13:28.848462+0000", 2024-04-14T15:16:20.883 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 171.94547639000001, 2024-04-14T15:16:20.883 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.051448817000000001, 2024-04-14T15:16:20.883 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:16:20.883 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:16:20.883 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:16:20.883 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:16:20.883 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.38086", 2024-04-14T15:16:20.883 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/2598250370", 2024-04-14T15:16:20.883 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 7 2024-04-14T15:16:20.883 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.883 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:16:20.883 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.883 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:16:20.883 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.848462+0000", 2024-04-14T15:16:20.884 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:20.884 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.884 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.884 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:16:20.884 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.848462+0000", 2024-04-14T15:16:20.884 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:20.884 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.884 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.884 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:16:20.884 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.848464+0000", 2024-04-14T15:16:20.884 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9420000000000002e-06 2024-04-14T15:16:20.884 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.884 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.884 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:16:20.884 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.848471+0000", 2024-04-14T15:16:20.885 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1099999999999997e-06 2024-04-14T15:16:20.885 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.885 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.885 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:16:20.885 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.850165+0000", 2024-04-14T15:16:20.885 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001693214 2024-04-14T15:16:20.885 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.885 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.885 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:16:20.885 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.854375+0000", 2024-04-14T15:16:20.885 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0042108400000000004 2024-04-14T15:16:20.885 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.885 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.885 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:20.885 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.855368+0000", 2024-04-14T15:16:20.886 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0009924650000000001 2024-04-14T15:16:20.886 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.886 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.886 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op must wait for map", 2024-04-14T15:16:20.886 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.855375+0000", 2024-04-14T15:16:20.886 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1389999999999998e-06 2024-04-14T15:16:20.886 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.886 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.886 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:20.886 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.864124+0000", 2024-04-14T15:16:20.886 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0087492400000000001 2024-04-14T15:16:20.886 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.886 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.886 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:16:20.887 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.865419+0000", 2024-04-14T15:16:20.887 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0012949000000000001 2024-04-14T15:16:20.887 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.887 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.887 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_started", 2024-04-14T15:16:20.887 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.866179+0000", 2024-04-14T15:16:20.887 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00076017100000000002 2024-04-14T15:16:20.887 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.887 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.887 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_committed", 2024-04-14T15:16:20.888 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.866887+0000", 2024-04-14T15:16:20.888 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00070775799999999995 2024-04-14T15:16:20.888 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.888 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.888 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:16:20.888 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.899834+0000", 2024-04-14T15:16:20.888 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.032947392999999998 2024-04-14T15:16:20.888 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.888 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.888 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:16:20.888 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.899911+0000", 2024-04-14T15:16:20.888 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.6644999999999995e-05 2024-04-14T15:16:20.888 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:20.888 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:16:20.889 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:20.889 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.889 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.889 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.47521.0:2 163.0 163:0384ab8e:::oid:head [append 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-04-14T15:16:20.889 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:13:30.817720+0000", 2024-04-14T15:16:20.889 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 169.97621873099999, 2024-04-14T15:16:20.889 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.074209873, 2024-04-14T15:16:20.889 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:16:20.889 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:16:20.889 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:16:20.889 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:16:20.889 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.47521", 2024-04-14T15:16:20.889 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/3376207688", 2024-04-14T15:16:20.889 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 2 2024-04-14T15:16:20.889 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.890 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:16:20.890 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.890 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:16:20.891 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:30.817720+0000", 2024-04-14T15:16:20.891 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:20.892 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.892 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.892 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:16:20.892 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:30.817720+0000", 2024-04-14T15:16:20.892 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:20.892 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.892 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.892 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:16:20.892 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:30.817722+0000", 2024-04-14T15:16:20.892 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2419999999999999e-06 2024-04-14T15:16:20.892 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.892 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.892 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:16:20.893 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:30.817730+0000", 2024-04-14T15:16:20.893 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5259999999999996e-06 2024-04-14T15:16:20.893 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.893 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.893 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:16:20.893 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:30.817739+0000", 2024-04-14T15:16:20.893 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.2029999999999998e-06 2024-04-14T15:16:20.893 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.893 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.893 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:16:20.893 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:30.817766+0000", 2024-04-14T15:16:20.893 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.72e-05 2024-04-14T15:16:20.893 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.893 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.893 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:20.894 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:30.820672+0000", 2024-04-14T15:16:20.894 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029057979999999998 2024-04-14T15:16:20.894 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.894 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.894 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:16:20.894 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:30.820681+0000", 2024-04-14T15:16:20.894 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0240000000000003e-06 2024-04-14T15:16:20.894 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.894 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.894 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:20.894 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:31.877336+0000", 2024-04-14T15:16:20.894 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0566553460000001 2024-04-14T15:16:20.894 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.894 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.895 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:16:20.895 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:31.877442+0000", 2024-04-14T15:16:20.895 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000105764 2024-04-14T15:16:20.895 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.895 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.895 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,4", 2024-04-14T15:16:20.895 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:31.877591+0000", 2024-04-14T15:16:20.895 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014861599999999999 2024-04-14T15:16:20.895 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.895 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.895 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:16:20.895 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:31.878073+0000", 2024-04-14T15:16:20.895 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00048190800000000002 2024-04-14T15:16:20.895 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.895 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.896 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:16:20.896 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:31.878967+0000", 2024-04-14T15:16:20.896 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00089447500000000004 2024-04-14T15:16:20.896 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.896 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.896 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:16:20.896 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:31.891870+0000", 2024-04-14T15:16:20.896 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.012903402 2024-04-14T15:16:20.896 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.896 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.896 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:16:20.896 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:31.891901+0000", 2024-04-14T15:16:20.896 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1012000000000001e-05 2024-04-14T15:16:20.896 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.896 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.897 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:16:20.897 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:31.891930+0000", 2024-04-14T15:16:20.897 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8357e-05 2024-04-14T15:16:20.897 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:20.897 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:16:20.897 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:20.897 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.897 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.897 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.47545.0:2 165.6 165:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-04-14T15:16:20.897 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:13:31.832019+0000", 2024-04-14T15:16:20.897 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 168.96191946900001, 2024-04-14T15:16:20.897 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.052401587999999999, 2024-04-14T15:16:20.897 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:16:20.897 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:16:20.897 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:16:20.898 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:16:20.898 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.47545", 2024-04-14T15:16:20.898 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/3859057618", 2024-04-14T15:16:20.898 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 2 2024-04-14T15:16:20.898 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.898 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:16:20.898 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.898 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:16:20.898 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:31.832019+0000", 2024-04-14T15:16:20.898 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:20.898 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.898 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.898 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:16:20.898 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:31.832019+0000", 2024-04-14T15:16:20.899 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:20.899 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.899 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.899 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:16:20.899 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:31.832021+0000", 2024-04-14T15:16:20.899 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.092e-06 2024-04-14T15:16:20.899 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.899 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.899 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:16:20.899 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:31.832029+0000", 2024-04-14T15:16:20.899 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.6090000000000003e-06 2024-04-14T15:16:20.899 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.899 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.899 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:16:20.899 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:31.834577+0000", 2024-04-14T15:16:20.900 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0025480989999999999 2024-04-14T15:16:20.900 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.900 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.900 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:16:20.900 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:31.834592+0000", 2024-04-14T15:16:20.900 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4891e-05 2024-04-14T15:16:20.900 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.900 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.900 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:20.900 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:31.837068+0000", 2024-04-14T15:16:20.900 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0024762460000000001 2024-04-14T15:16:20.900 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.900 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.900 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:16:20.900 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:31.837077+0000", 2024-04-14T15:16:20.901 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.7669999999999992e-06 2024-04-14T15:16:20.901 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.901 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.901 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:20.901 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:31.872224+0000", 2024-04-14T15:16:20.901 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.035147014999999997 2024-04-14T15:16:20.901 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.901 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.901 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:16:20.901 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:31.872317+0000", 2024-04-14T15:16:20.901 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.3546000000000002e-05 2024-04-14T15:16:20.901 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.901 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.901 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,4", 2024-04-14T15:16:20.901 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:31.872441+0000", 2024-04-14T15:16:20.902 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012318400000000001 2024-04-14T15:16:20.902 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.902 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.902 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:16:20.902 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:31.873715+0000", 2024-04-14T15:16:20.902 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0012740150000000001 2024-04-14T15:16:20.902 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.902 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.902 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:16:20.902 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:31.874641+0000", 2024-04-14T15:16:20.902 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00092663700000000001 2024-04-14T15:16:20.902 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.902 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.902 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:16:20.902 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:31.884289+0000", 2024-04-14T15:16:20.903 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0096479400000000007 2024-04-14T15:16:20.903 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.903 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.903 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:16:20.903 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:31.884375+0000", 2024-04-14T15:16:20.903 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5916999999999997e-05 2024-04-14T15:16:20.903 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.903 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.903 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:16:20.903 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:31.884421+0000", 2024-04-14T15:16:20.903 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5630000000000002e-05 2024-04-14T15:16:20.903 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:20.903 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:16:20.903 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:20.903 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.904 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.904 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.48388.0:2 214.c 214:311cbe94:::oid_ExecuteClassPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e195)", 2024-04-14T15:16:20.904 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:14:22.850377+0000", 2024-04-14T15:16:20.904 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 117.94356123999999, 2024-04-14T15:16:20.904 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.052484141999999998, 2024-04-14T15:16:20.904 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:16:20.904 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:16:20.904 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:16:20.904 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:16:20.904 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.48388", 2024-04-14T15:16:20.904 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/679904944", 2024-04-14T15:16:20.904 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 2 2024-04-14T15:16:20.904 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.904 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:16:20.905 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.905 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:16:20.905 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:22.850377+0000", 2024-04-14T15:16:20.905 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:20.905 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.905 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.905 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:16:20.905 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:22.850377+0000", 2024-04-14T15:16:20.905 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:20.906 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.906 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.906 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:16:20.906 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:22.850379+0000", 2024-04-14T15:16:20.906 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6780000000000001e-06 2024-04-14T15:16:20.906 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.906 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.906 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:16:20.906 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:22.850386+0000", 2024-04-14T15:16:20.906 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.55e-06 2024-04-14T15:16:20.906 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.906 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.906 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:16:20.906 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:22.850395+0000", 2024-04-14T15:16:20.906 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.1519999999999998e-06 2024-04-14T15:16:20.907 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.907 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.907 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:16:20.907 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:22.850405+0000", 2024-04-14T15:16:20.907 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0655999999999999e-05 2024-04-14T15:16:20.907 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.907 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.907 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:20.907 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:22.850469+0000", 2024-04-14T15:16:20.907 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3652000000000007e-05 2024-04-14T15:16:20.907 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.907 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.907 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op must wait for map", 2024-04-14T15:16:20.907 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:22.850477+0000", 2024-04-14T15:16:20.908 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0979999999999994e-06 2024-04-14T15:16:20.908 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.908 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.908 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:20.908 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:22.856127+0000", 2024-04-14T15:16:20.908 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0056495089999999996 2024-04-14T15:16:20.908 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.908 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.908 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:16:20.908 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:22.856136+0000", 2024-04-14T15:16:20.908 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.9849999999999995e-06 2024-04-14T15:16:20.908 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.908 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.908 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:20.908 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:22.895778+0000", 2024-04-14T15:16:20.909 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.039642150000000001 2024-04-14T15:16:20.909 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.909 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.909 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:16:20.909 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:22.895888+0000", 2024-04-14T15:16:20.909 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011000100000000001 2024-04-14T15:16:20.909 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.909 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.909 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 2,7", 2024-04-14T15:16:20.909 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:22.896012+0000", 2024-04-14T15:16:20.909 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012449999999999999 2024-04-14T15:16:20.909 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.909 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.909 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:16:20.909 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:22.896739+0000", 2024-04-14T15:16:20.910 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00072682200000000001 2024-04-14T15:16:20.910 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.910 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.910 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:16:20.910 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:22.898060+0000", 2024-04-14T15:16:20.910 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0013207939999999999 2024-04-14T15:16:20.910 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.910 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.910 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:16:20.910 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:22.902770+0000", 2024-04-14T15:16:20.910 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0047102969999999996 2024-04-14T15:16:20.910 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.910 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.910 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:16:20.910 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:22.902823+0000", 2024-04-14T15:16:20.911 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2902999999999998e-05 2024-04-14T15:16:20.911 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.911 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.911 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:16:20.911 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:22.902862+0000", 2024-04-14T15:16:20.911 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8395000000000001e-05 2024-04-14T15:16:20.911 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:20.911 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:16:20.911 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:20.911 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.911 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.911 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.48481.0:1 227.6s0 227:602f83fe:::foo:head [append 0~8192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-04-14T15:16:20.911 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:14:42.851421+0000", 2024-04-14T15:16:20.911 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 97.942517347000006, 2024-04-14T15:16:20.911 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0204730660000001, 2024-04-14T15:16:20.912 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:16:20.912 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:16:20.912 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:16:20.912 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:16:20.912 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.48481", 2024-04-14T15:16:20.912 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/3131004652", 2024-04-14T15:16:20.912 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 1 2024-04-14T15:16:20.912 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.912 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:16:20.912 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.912 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:16:20.912 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:42.851421+0000", 2024-04-14T15:16:20.912 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:20.912 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.913 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.913 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:16:20.913 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:42.851421+0000", 2024-04-14T15:16:20.913 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:20.913 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.913 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.913 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:16:20.913 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:42.851423+0000", 2024-04-14T15:16:20.913 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.84e-06 2024-04-14T15:16:20.913 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.913 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.913 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:16:20.913 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:42.851437+0000", 2024-04-14T15:16:20.913 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3652000000000001e-05 2024-04-14T15:16:20.913 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.914 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.914 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:16:20.914 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:42.851447+0000", 2024-04-14T15:16:20.914 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8889999999999997e-06 2024-04-14T15:16:20.914 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.914 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.914 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:16:20.914 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:42.851457+0000", 2024-04-14T15:16:20.914 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0175e-05 2024-04-14T15:16:20.914 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.914 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.914 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:20.914 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:42.851514+0000", 2024-04-14T15:16:20.914 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7463999999999997e-05 2024-04-14T15:16:20.915 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.915 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.915 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:16:20.915 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:42.851518+0000", 2024-04-14T15:16:20.915 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.7850000000000002e-06 2024-04-14T15:16:20.915 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.915 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.915 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:20.915 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:43.866076+0000", 2024-04-14T15:16:20.915 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.014558361 2024-04-14T15:16:20.915 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.915 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.915 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:16:20.915 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:43.866170+0000", 2024-04-14T15:16:20.915 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.3744000000000002e-05 2024-04-14T15:16:20.916 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.916 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.916 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_started", 2024-04-14T15:16:20.916 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:43.866491+0000", 2024-04-14T15:16:20.916 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00032043099999999998 2024-04-14T15:16:20.916 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.916 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.916 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_committed", 2024-04-14T15:16:20.916 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:43.866747+0000", 2024-04-14T15:16:20.916 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00025597499999999997 2024-04-14T15:16:20.916 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.916 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.916 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:16:20.916 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:43.871842+0000", 2024-04-14T15:16:20.916 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0050949180000000004 2024-04-14T15:16:20.917 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.917 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.917 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:16:20.917 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:43.871894+0000", 2024-04-14T15:16:20.917 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2831999999999998e-05 2024-04-14T15:16:20.917 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:20.917 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:16:20.917 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:20.917 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.917 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.917 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.36780.0:2 249.a 249:57d91c0c:::oid_RemoveTestPP_:head [append 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-04-14T15:16:20.917 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:15:20.877445+0000", 2024-04-14T15:16:20.917 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 59.916493338999999, 2024-04-14T15:16:20.917 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.033481961999999997, 2024-04-14T15:16:20.917 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:16:20.918 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:16:20.918 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:16:20.918 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:16:20.918 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.36780", 2024-04-14T15:16:20.918 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/289305254", 2024-04-14T15:16:20.918 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 2 2024-04-14T15:16:20.918 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.918 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:16:20.918 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.918 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:16:20.918 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:20.877445+0000", 2024-04-14T15:16:20.918 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:20.918 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.918 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.919 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:16:20.919 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:20.877445+0000", 2024-04-14T15:16:20.919 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:16:20.919 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.919 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.919 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:16:20.919 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:20.877448+0000", 2024-04-14T15:16:20.919 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3630000000000001e-06 2024-04-14T15:16:20.919 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.919 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.919 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:16:20.919 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:20.877456+0000", 2024-04-14T15:16:20.919 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.7579999999999998e-06 2024-04-14T15:16:20.919 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.919 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.920 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:16:20.920 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:20.877468+0000", 2024-04-14T15:16:20.920 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1815e-05 2024-04-14T15:16:20.920 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.920 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.920 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:16:20.920 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:20.877479+0000", 2024-04-14T15:16:20.920 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0895e-05 2024-04-14T15:16:20.920 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.920 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.920 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:20.920 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:20.877541+0000", 2024-04-14T15:16:20.920 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1975000000000002e-05 2024-04-14T15:16:20.920 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.920 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.921 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:16:20.921 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:20.877551+0000", 2024-04-14T15:16:20.921 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8039999999999996e-06 2024-04-14T15:16:20.921 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.921 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.921 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:16:20.921 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:20.886958+0000", 2024-04-14T15:16:20.921 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0094066889999999993 2024-04-14T15:16:20.921 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.921 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.921 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:16:20.921 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:20.887102+0000", 2024-04-14T15:16:20.921 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000144182 2024-04-14T15:16:20.921 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.921 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.922 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 2,4", 2024-04-14T15:16:20.922 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:20.887330+0000", 2024-04-14T15:16:20.922 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000228567 2024-04-14T15:16:20.922 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.922 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.922 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:16:20.922 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:20.893884+0000", 2024-04-14T15:16:20.922 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0065540110000000002 2024-04-14T15:16:20.922 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.922 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.922 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:16:20.922 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:20.894871+0000", 2024-04-14T15:16:20.922 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00098655899999999992 2024-04-14T15:16:20.922 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.922 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.923 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:16:20.923 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:20.910850+0000", 2024-04-14T15:16:20.923 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.015978946000000001 2024-04-14T15:16:20.923 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.923 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.923 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:16:20.923 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:20.910891+0000", 2024-04-14T15:16:20.923 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0988000000000001e-05 2024-04-14T15:16:20.923 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:16:20.923 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:16:20.923 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:16:20.923 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:20.910927+0000", 2024-04-14T15:16:20.923 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6409999999999999e-05 2024-04-14T15:16:20.923 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:20.923 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:16:20.924 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:20.924 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:16:20.924 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:16:20.924 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-04-14T15:16:20.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.854+0000 7fa892873700 1 -- 172.21.15.90:0/1316259666 >> v1:172.21.15.90:6789/0 conn(0x7fa88c10d940 legacy=0x7fa88c10fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:20.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.854+0000 7fa892873700 1 -- 172.21.15.90:0/1316259666 shutdown_connections 2024-04-14T15:16:20.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.854+0000 7fa892873700 1 -- 172.21.15.90:0/1316259666 >> 172.21.15.90:0/1316259666 conn(0x7fa88c076aa0 msgr2=0x7fa88c076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:16:20.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.854+0000 7fa892873700 1 -- 172.21.15.90:0/1316259666 shutdown_connections 2024-04-14T15:16:20.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.854+0000 7fa892873700 1 -- 172.21.15.90:0/1316259666 wait complete. 2024-04-14T15:16:20.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.854+0000 7fa892873700 1 librados: starting msgr at 2024-04-14T15:16:20.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.854+0000 7fa892873700 1 librados: starting objecter 2024-04-14T15:16:20.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.854+0000 7fa892873700 1 Processor -- start 2024-04-14T15:16:20.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.854+0000 7fa892873700 1 -- start start 2024-04-14T15:16:20.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.854+0000 7fa892873700 1 librados: setting wanted keys 2024-04-14T15:16:20.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.854+0000 7fa892873700 1 librados: calling monclient init 2024-04-14T15:16:20.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.855+0000 7fa892873700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa88c1a7a40 con 0x7fa88c109da0 2024-04-14T15:16:20.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.855+0000 7fa892873700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa88c1a8c60 con 0x7fa88c10d940 2024-04-14T15:16:20.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.855+0000 7fa892873700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa88c1a9ea0 con 0x7fa88c106f80 2024-04-14T15:16:20.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.855+0000 7fa88b7fe700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fa88c109da0 0x7fa88c100d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:39054/0 (socket says 172.21.15.90:39054) 2024-04-14T15:16:20.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.855+0000 7fa88b7fe700 1 -- 172.21.15.90:0/2668278298 learned_addr learned my addr 172.21.15.90:0/2668278298 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:16:20.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.855+0000 7fa8897fa700 10 client.?.objecter ms_handle_connect 0x7fa88c109da0 2024-04-14T15:16:20.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.855+0000 7fa8897fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:16:20.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.855+0000 7fa8897fa700 10 client.?.objecter ms_handle_connect 0x7fa88c10d940 2024-04-14T15:16:20.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.855+0000 7fa8897fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:16:20.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.855+0000 7fa8897fa700 10 client.?.objecter ms_handle_connect 0x7fa88c106f80 2024-04-14T15:16:20.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.855+0000 7fa8897fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:16:20.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.855+0000 7fa8897fa700 1 -- 172.21.15.90:0/2668278298 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2075506713 0 0) 0x7fa88c1a7a40 con 0x7fa88c109da0 2024-04-14T15:16:20.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.855+0000 7fa8897fa700 1 -- 172.21.15.90:0/2668278298 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa874004370 con 0x7fa88c109da0 2024-04-14T15:16:20.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.855+0000 7fa8897fa700 1 -- 172.21.15.90:0/2668278298 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 280437462 0 0) 0x7fa88c1a8c60 con 0x7fa88c10d940 2024-04-14T15:16:20.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.855+0000 7fa8897fa700 1 -- 172.21.15.90:0/2668278298 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa88c1a7a40 con 0x7fa88c10d940 2024-04-14T15:16:20.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.855+0000 7fa8897fa700 1 -- 172.21.15.90:0/2668278298 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3731356443 0 0) 0x7fa88c1a9ea0 con 0x7fa88c106f80 2024-04-14T15:16:20.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.855+0000 7fa8897fa700 1 -- 172.21.15.90:0/2668278298 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa88c1a8c60 con 0x7fa88c106f80 2024-04-14T15:16:20.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.856+0000 7fa8897fa700 1 -- 172.21.15.90:0/2668278298 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3270854312 0 0) 0x7fa88c1a7a40 con 0x7fa88c10d940 2024-04-14T15:16:20.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.856+0000 7fa8897fa700 1 -- 172.21.15.90:0/2668278298 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa88c1a9ea0 con 0x7fa88c10d940 2024-04-14T15:16:20.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.856+0000 7fa8897fa700 1 -- 172.21.15.90:0/2668278298 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fa8840016f0 con 0x7fa88c10d940 2024-04-14T15:16:20.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.856+0000 7fa8897fa700 1 -- 172.21.15.90:0/2668278298 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2928140888 0 0) 0x7fa874004370 con 0x7fa88c109da0 2024-04-14T15:16:20.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.856+0000 7fa8897fa700 1 -- 172.21.15.90:0/2668278298 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa88c1a7a40 con 0x7fa88c109da0 2024-04-14T15:16:20.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.856+0000 7fa8897fa700 1 -- 172.21.15.90:0/2668278298 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fa880001a10 con 0x7fa88c109da0 2024-04-14T15:16:20.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.856+0000 7fa8897fa700 1 -- 172.21.15.90:0/2668278298 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4270009389 0 0) 0x7fa88c1a8c60 con 0x7fa88c106f80 2024-04-14T15:16:20.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.856+0000 7fa8897fa700 1 -- 172.21.15.90:0/2668278298 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa874004370 con 0x7fa88c106f80 2024-04-14T15:16:20.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.856+0000 7fa8897fa700 1 -- 172.21.15.90:0/2668278298 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fa87c001870 con 0x7fa88c106f80 2024-04-14T15:16:20.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.856+0000 7fa8897fa700 1 -- 172.21.15.90:0/2668278298 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1710660228 0 0) 0x7fa88c1a9ea0 con 0x7fa88c10d940 2024-04-14T15:16:20.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.856+0000 7fa8897fa700 1 -- 172.21.15.90:0/2668278298 >> v1:172.21.15.90:6790/0 conn(0x7fa88c106f80 legacy=0x7fa88c100640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:20.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.856+0000 7fa8897fa700 1 -- 172.21.15.90:0/2668278298 >> v1:172.21.15.90:6789/0 conn(0x7fa88c109da0 legacy=0x7fa88c100d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:20.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.856+0000 7fa8897fa700 1 -- 172.21.15.90:0/2668278298 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa88c1ab0e0 con 0x7fa88c10d940 2024-04-14T15:16:20.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.856+0000 7fa892873700 1 -- 172.21.15.90:0/2668278298 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa88c1a8e90 con 0x7fa88c10d940 2024-04-14T15:16:20.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.856+0000 7fa892873700 10 client.48847.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:16:20.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.856+0000 7fa892873700 1 -- 172.21.15.90:0/2668278298 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa88c1a9350 con 0x7fa88c10d940 2024-04-14T15:16:20.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.856+0000 7fa892873700 1 librados: init done 2024-04-14T15:16:20.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.856+0000 7fa8897fa700 1 -- 172.21.15.90:0/2668278298 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fa884001990 con 0x7fa88c10d940 2024-04-14T15:16:20.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.856+0000 7fa8897fa700 1 -- 172.21.15.90:0/2668278298 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fa884004f10 con 0x7fa88c10d940 2024-04-14T15:16:20.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.857+0000 7fa892873700 1 -- 172.21.15.90:0/2668278298 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa878005120 con 0x7fa88c10d940 2024-04-14T15:16:20.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.858+0000 7fa8897fa700 1 -- 172.21.15.90:0/2668278298 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fa88401b920 con 0x7fa88c10d940 2024-04-14T15:16:20.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.858+0000 7fa8897fa700 1 -- 172.21.15.90:0/2668278298 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(312..312 src has 51..312) ==== 8941+0+0 (unknown 1687004107 0 0) 0x7fa884089c30 con 0x7fa88c10d940 2024-04-14T15:16:20.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.858+0000 7fa8897fa700 10 client.48847.objecter ms_dispatch 0x7fa88c001550 osd_map(312..312 src has 51..312) 2024-04-14T15:16:20.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.858+0000 7fa8897fa700 3 client.48847.objecter handle_osd_map got epochs [312,312] > 0 2024-04-14T15:16:20.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.858+0000 7fa8897fa700 3 client.48847.objecter handle_osd_map decoding full epoch 312 2024-04-14T15:16:20.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.858+0000 7fa8897fa700 20 client.48847.objecter dump_active .. 0 homeless 2024-04-14T15:16:20.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.858+0000 7fa8897fa700 10 client.48847.objecter ms_handle_connect 0x7fa874071330 2024-04-14T15:16:20.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.863+0000 7fa8897fa700 1 -- 172.21.15.90:0/2668278298 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fa884056330 con 0x7fa88c10d940 2024-04-14T15:16:20.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.884+0000 7f63deffd700 1 -- 172.21.15.90:0/1574854201 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd pg-upmap-items", "pgid": "110.b", "id": [7, 2, 6, 5, 3, 0]}]=0 set 110.b pg_upmap_items mapping to [7->2,6->5,3->0] v313) ==== 161+0+0 (unknown 3135682510 0 0) 0x7f63d8059f50 con 0x7f63e810a6c0 2024-04-14T15:16:20.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.888+0000 7f63eff53700 10 librados: watch_flush enter 2024-04-14T15:16:20.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.888+0000 7f63eff53700 10 librados: watch_flush exit 2024-04-14T15:16:20.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.888+0000 7f63eff53700 20 client.48841.objecter shutdown clearing up homeless session... 2024-04-14T15:16:20.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.888+0000 7f63eff53700 10 client.48841.objecter successfully canceled tick 2024-04-14T15:16:20.931 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:16:20.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.889+0000 7f63eff53700 1 -- 172.21.15.90:0/1574854201 >> v1:172.21.15.90:6800/420935427 conn(0x7f63d4071390 legacy=0x7f63d4073850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:20.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.889+0000 7f63eff53700 1 -- 172.21.15.90:0/1574854201 >> v1:172.21.15.92:6789/0 conn(0x7f63e810a6c0 legacy=0x7f63e8104660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:20.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.891+0000 7f63eff53700 1 -- 172.21.15.90:0/1574854201 shutdown_connections 2024-04-14T15:16:20.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.891+0000 7f63eff53700 1 -- 172.21.15.90:0/1574854201 >> 172.21.15.90:0/1574854201 conn(0x7f63e8076ad0 msgr2=0x7f63e8111a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:16:20.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.894+0000 7f63eff53700 1 -- 172.21.15.90:0/1574854201 shutdown_connections 2024-04-14T15:16:20.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.895+0000 7f63eff53700 1 -- 172.21.15.90:0/1574854201 wait complete. 2024-04-14T15:16:20.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:20.895+0000 7f63eff53700 1 librados: shutdown 2024-04-14T15:16:20.933 INFO:teuthology.orchestra.run.smithi090.stderr:set 110.b pg_upmap_items mapping to [7->2,6->5,3->0] 2024-04-14T15:16:20.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:20.933 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:20.934 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:16:21.034 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:21.063 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:21.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:21.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:21.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:21.103+0000 7fa892873700 1 -- 172.21.15.90:0/2668278298 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7fa878005410 con 0x7fa88c10d940 2024-04-14T15:16:21.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:20 smithi092 ceph-mon[33966]: pgmap v413: 238 pgs: 8 unknown, 230 active+clean; 4.6 MiB data, 575 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:16:21.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:20 smithi092 ceph-mon[33966]: 11.2 scrub starts 2024-04-14T15:16:21.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:20 smithi092 ceph-mon[33966]: 11.2 scrub ok 2024-04-14T15:16:21.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:20 smithi092 ceph-mon[33966]: 4.5 scrub starts 2024-04-14T15:16:21.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:20 smithi092 ceph-mon[33966]: 4.5 scrub ok 2024-04-14T15:16:21.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:20 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:16:21.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:20 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:16:21.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:20 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1574854201' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "110.b", "id": [7, 2, 6, 5, 3, 0]}]: dispatch 2024-04-14T15:16:21.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:20 smithi092 ceph-mon[33966]: from='client.48841 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "110.b", "id": [7, 2, 6, 5, 3, 0]}]: dispatch 2024-04-14T15:16:21.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:20 smithi092 ceph-mon[33966]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:16:21.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:20 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:16:21.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:20 smithi092 ceph-mon[33966]: from='client.48841 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "110.b", "id": [7, 2, 6, 5, 3, 0]}]': finished 2024-04-14T15:16:21.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:20 smithi092 ceph-mon[33966]: osdmap e313: 8 total, 8 up, 8 in 2024-04-14T15:16:21.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:20 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-39"}]: dispatch 2024-04-14T15:16:21.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:20 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-39"}]: dispatch 2024-04-14T15:16:21.165 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:21.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:21.189 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:21.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:21.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:20 smithi090 ceph-mon[41670]: pgmap v413: 238 pgs: 8 unknown, 230 active+clean; 4.6 MiB data, 575 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:16:21.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:20 smithi090 ceph-mon[41670]: 11.2 scrub starts 2024-04-14T15:16:21.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:20 smithi090 ceph-mon[41670]: 11.2 scrub ok 2024-04-14T15:16:21.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:20 smithi090 ceph-mon[41670]: 4.5 scrub starts 2024-04-14T15:16:21.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:20 smithi090 ceph-mon[41670]: 4.5 scrub ok 2024-04-14T15:16:21.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:20 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:16:21.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:20 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:16:21.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:20 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1574854201' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "110.b", "id": [7, 2, 6, 5, 3, 0]}]: dispatch 2024-04-14T15:16:21.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:20 smithi090 ceph-mon[41670]: from='client.48841 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "110.b", "id": [7, 2, 6, 5, 3, 0]}]: dispatch 2024-04-14T15:16:21.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:20 smithi090 ceph-mon[41670]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:16:21.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:20 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:16:21.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:20 smithi090 ceph-mon[41670]: from='client.48841 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "110.b", "id": [7, 2, 6, 5, 3, 0]}]': finished 2024-04-14T15:16:21.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:20 smithi090 ceph-mon[41670]: osdmap e313: 8 total, 8 up, 8 in 2024-04-14T15:16:21.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:20 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-39"}]: dispatch 2024-04-14T15:16:21.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:20 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-39"}]: dispatch 2024-04-14T15:16:21.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:20 smithi090 ceph-mon[30399]: pgmap v413: 238 pgs: 8 unknown, 230 active+clean; 4.6 MiB data, 575 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:16:21.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:20 smithi090 ceph-mon[30399]: 11.2 scrub starts 2024-04-14T15:16:21.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:20 smithi090 ceph-mon[30399]: 11.2 scrub ok 2024-04-14T15:16:21.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:20 smithi090 ceph-mon[30399]: 4.5 scrub starts 2024-04-14T15:16:21.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:20 smithi090 ceph-mon[30399]: 4.5 scrub ok 2024-04-14T15:16:21.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:20 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:16:21.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:20 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:16:21.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:20 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1574854201' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "110.b", "id": [7, 2, 6, 5, 3, 0]}]: dispatch 2024-04-14T15:16:21.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:20 smithi090 ceph-mon[30399]: from='client.48841 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "110.b", "id": [7, 2, 6, 5, 3, 0]}]: dispatch 2024-04-14T15:16:21.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:20 smithi090 ceph-mon[30399]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:16:21.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:20 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:16:21.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:20 smithi090 ceph-mon[30399]: from='client.48841 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "110.b", "id": [7, 2, 6, 5, 3, 0]}]': finished 2024-04-14T15:16:21.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:20 smithi090 ceph-mon[30399]: osdmap e313: 8 total, 8 up, 8 in 2024-04-14T15:16:21.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:20 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-39"}]: dispatch 2024-04-14T15:16:21.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:20 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-39"}]: dispatch 2024-04-14T15:16:21.290 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:16:21.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:21.314 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:21.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:16:21.415 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:16:21.439 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:21.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:21.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:16:21.541 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:21.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:21.565 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:21.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:21.666 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:21.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:21.691 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:21.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:21.792 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:16:21.816 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:21.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:21.817 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:16:21.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:21.884+0000 7fa8897fa700 1 -- 172.21.15.90:0/2668278298 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v314) ==== 87+0+0 (unknown 3646743201 0 0) 0x7fa884059fe0 con 0x7fa88c10d940 2024-04-14T15:16:21.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:21.887+0000 7fa892873700 10 librados: watch_flush enter 2024-04-14T15:16:21.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:21.887+0000 7fa892873700 10 librados: watch_flush exit 2024-04-14T15:16:21.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:21.887+0000 7fa892873700 20 client.48847.objecter shutdown clearing up homeless session... 2024-04-14T15:16:21.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:21.887+0000 7fa892873700 10 client.48847.objecter successfully canceled tick 2024-04-14T15:16:21.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:21.887+0000 7fa892873700 1 -- 172.21.15.90:0/2668278298 >> v1:172.21.15.90:6800/420935427 conn(0x7fa874071330 legacy=0x7fa8740737f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:21.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:21.887+0000 7fa892873700 1 -- 172.21.15.90:0/2668278298 >> v1:172.21.15.92:6789/0 conn(0x7fa88c10d940 legacy=0x7fa88c1a6100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:21.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:21.887+0000 7fa892873700 1 -- 172.21.15.90:0/2668278298 shutdown_connections 2024-04-14T15:16:21.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:21.887+0000 7fa892873700 1 -- 172.21.15.90:0/2668278298 >> 172.21.15.90:0/2668278298 conn(0x7fa88c076aa0 msgr2=0x7fa88c108590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:16:21.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:21.887+0000 7fa892873700 1 -- 172.21.15.90:0/2668278298 shutdown_connections 2024-04-14T15:16:21.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:21.887+0000 7fa892873700 1 -- 172.21.15.90:0/2668278298 wait complete. 2024-04-14T15:16:21.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:21.887+0000 7fa892873700 1 librados: shutdown 2024-04-14T15:16:21.888 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is set 2024-04-14T15:16:21.918 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:16:21.947 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:21.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:21.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:16:21.963 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-14T15:16:22.049 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:22.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:22.074 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:22.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:22.175 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:16:22.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:21 smithi090 ceph-mon[30399]: 6.5 deep-scrub starts 2024-04-14T15:16:22.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:21 smithi090 ceph-mon[30399]: 6.5 deep-scrub ok 2024-04-14T15:16:22.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:21 smithi090 ceph-mon[30399]: 2.1b scrub starts 2024-04-14T15:16:22.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:21 smithi090 ceph-mon[30399]: 2.1b scrub ok 2024-04-14T15:16:22.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:21 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2668278298' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:16:22.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:21 smithi090 ceph-mon[30399]: from='client.48847 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:16:22.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:21 smithi090 ceph-mon[30399]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:16:22.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:21 smithi090 ceph-mon[30399]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:16:22.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:21 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-39"}]': finished 2024-04-14T15:16:22.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:21 smithi090 ceph-mon[30399]: from='client.48847 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-14T15:16:22.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:21 smithi090 ceph-mon[30399]: osdmap e314: 8 total, 8 up, 8 in 2024-04-14T15:16:22.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:21 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/585933469' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi090-93363-54"}]: dispatch 2024-04-14T15:16:22.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:21 smithi090 ceph-mon[30399]: from='client.48835 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi090-93363-54"}]: dispatch 2024-04-14T15:16:22.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:21 smithi090 ceph-mon[41670]: 6.5 deep-scrub starts 2024-04-14T15:16:22.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:21 smithi090 ceph-mon[41670]: 6.5 deep-scrub ok 2024-04-14T15:16:22.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:21 smithi090 ceph-mon[41670]: 2.1b scrub starts 2024-04-14T15:16:22.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:21 smithi090 ceph-mon[41670]: 2.1b scrub ok 2024-04-14T15:16:22.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:21 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2668278298' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:16:22.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:21 smithi090 ceph-mon[41670]: from='client.48847 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:16:22.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:21 smithi090 ceph-mon[41670]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:16:22.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:21 smithi090 ceph-mon[41670]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:16:22.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:21 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-39"}]': finished 2024-04-14T15:16:22.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:21 smithi090 ceph-mon[41670]: from='client.48847 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-14T15:16:22.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:21 smithi090 ceph-mon[41670]: osdmap e314: 8 total, 8 up, 8 in 2024-04-14T15:16:22.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:21 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/585933469' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi090-93363-54"}]: dispatch 2024-04-14T15:16:22.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:21 smithi090 ceph-mon[41670]: from='client.48835 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi090-93363-54"}]: dispatch 2024-04-14T15:16:22.199 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:22.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:22.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:16:22.206 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:16:22.301 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:16:22.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:22.326 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:22.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:16:22.328 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:21 smithi092 ceph-mon[33966]: 6.5 deep-scrub starts 2024-04-14T15:16:22.328 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:21 smithi092 ceph-mon[33966]: 6.5 deep-scrub ok 2024-04-14T15:16:22.328 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:21 smithi092 ceph-mon[33966]: 2.1b scrub starts 2024-04-14T15:16:22.328 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:21 smithi092 ceph-mon[33966]: 2.1b scrub ok 2024-04-14T15:16:22.328 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:21 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2668278298' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:16:22.328 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:21 smithi092 ceph-mon[33966]: from='client.48847 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:16:22.328 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:21 smithi092 ceph-mon[33966]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:16:22.328 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:21 smithi092 ceph-mon[33966]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:16:22.328 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:21 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-39"}]': finished 2024-04-14T15:16:22.328 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:21 smithi092 ceph-mon[33966]: from='client.48847 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-14T15:16:22.328 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:21 smithi092 ceph-mon[33966]: osdmap e314: 8 total, 8 up, 8 in 2024-04-14T15:16:22.328 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:21 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/585933469' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi090-93363-54"}]: dispatch 2024-04-14T15:16:22.328 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:21 smithi092 ceph-mon[33966]: from='client.48835 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi090-93363-54"}]: dispatch 2024-04-14T15:16:22.427 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:22.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:22.454 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:22.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:22.555 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:16:22.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:22.579 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:22.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:16:22.681 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:16:22.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:22.705 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:22.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:16:22.806 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:22.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:22.833 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:22.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:22.892 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [==========] Running 77 tests from 4 test suites. 2024-04-14T15:16:22.893 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [----------] Global test environment set-up. 2024-04-14T15:16:22.893 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2024-04-14T15:16:22.893 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: seed 93652 2024-04-14T15:16:22.893 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2024-04-14T15:16:22.893 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (481 ms) 2024-04-14T15:16:22.893 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2024-04-14T15:16:22.893 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (10219 ms) 2024-04-14T15:16:22.893 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2024-04-14T15:16:22.893 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (67 ms) 2024-04-14T15:16:22.893 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (10767 ms total) 2024-04-14T15:16:22.894 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: 2024-04-14T15:16:22.894 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP 2024-04-14T15:16:22.894 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2024-04-14T15:16:22.894 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (6965 ms) 2024-04-14T15:16:22.894 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2024-04-14T15:16:22.894 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (8025 ms) 2024-04-14T15:16:22.894 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2024-04-14T15:16:22.894 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (9986 ms) 2024-04-14T15:16:22.894 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2024-04-14T15:16:22.894 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: my_snaps [3] 2024-04-14T15:16:22.895 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: my_snaps [4,3] 2024-04-14T15:16:22.895 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: my_snaps [5,4,3] 2024-04-14T15:16:22.895 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: my_snaps [6,5,4,3] 2024-04-14T15:16:22.895 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: promoting some heads 2024-04-14T15:16:22.895 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: promoting from clones for snap 6 2024-04-14T15:16:22.895 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: promoting from clones for snap 5 2024-04-14T15:16:22.895 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: promoting from clones for snap 4 2024-04-14T15:16:22.895 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: promoting from clones for snap 3 2024-04-14T15:16:22.895 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: waiting for scrubs... 2024-04-14T15:16:22.895 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: done waiting 2024-04-14T15:16:22.895 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (46025 ms) 2024-04-14T15:16:22.895 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2024-04-14T15:16:22.895 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (9987 ms) 2024-04-14T15:16:22.895 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2024-04-14T15:16:22.895 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (7989 ms) 2024-04-14T15:16:22.896 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2024-04-14T15:16:22.896 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (8005 ms) 2024-04-14T15:16:22.896 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2024-04-14T15:16:22.896 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (8009 ms) 2024-04-14T15:16:22.896 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2024-04-14T15:16:22.896 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (10023 ms) 2024-04-14T15:16:22.896 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2024-04-14T15:16:22.896 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (8975 ms) 2024-04-14T15:16:22.896 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2024-04-14T15:16:22.896 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (11026 ms) 2024-04-14T15:16:22.896 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace 2024-04-14T15:16:22.896 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace (12991 ms) 2024-04-14T15:16:22.896 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2024-04-14T15:16:22.896 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (7991 ms) 2024-04-14T15:16:22.897 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2024-04-14T15:16:22.897 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (8022 ms) 2024-04-14T15:16:22.897 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2024-04-14T15:16:22.897 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (12990 ms) 2024-04-14T15:16:22.897 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2024-04-14T15:16:22.897 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (8005 ms) 2024-04-14T15:16:22.897 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2024-04-14T15:16:22.897 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (8018 ms) 2024-04-14T15:16:22.933 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:22.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:22.957 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:22.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:22.992 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:22 smithi090 ceph-mon[41670]: pgmap v416: 246 pgs: 8 unknown, 238 active+clean; 4.6 MiB data, 579 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:16:22.992 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:22 smithi090 ceph-mon[41670]: from='client.48835 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi090-93363-54"}]': finished 2024-04-14T15:16:22.992 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:22 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/585933469' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi090-93363-54"}]: dispatch 2024-04-14T15:16:22.992 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:22 smithi090 ceph-mon[41670]: osdmap e315: 8 total, 8 up, 8 in 2024-04-14T15:16:22.992 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:22 smithi090 ceph-mon[41670]: from='client.48835 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi090-93363-54"}]: dispatch 2024-04-14T15:16:22.992 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:22 smithi090 ceph-mon[30399]: pgmap v416: 246 pgs: 8 unknown, 238 active+clean; 4.6 MiB data, 579 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:16:22.992 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:22 smithi090 ceph-mon[30399]: from='client.48835 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi090-93363-54"}]': finished 2024-04-14T15:16:22.992 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:22 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/585933469' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi090-93363-54"}]: dispatch 2024-04-14T15:16:22.992 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:22 smithi090 ceph-mon[30399]: osdmap e315: 8 total, 8 up, 8 in 2024-04-14T15:16:22.993 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:22 smithi090 ceph-mon[30399]: from='client.48835 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi090-93363-54"}]: dispatch 2024-04-14T15:16:23.058 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:16:23.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:23.083 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:23.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:16:23.184 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:16:23.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:23.208 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:23.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:16:23.310 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:16:23.333 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:23.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:23.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:16:23.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:22 smithi092 ceph-mon[33966]: pgmap v416: 246 pgs: 8 unknown, 238 active+clean; 4.6 MiB data, 579 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:16:23.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:22 smithi092 ceph-mon[33966]: from='client.48835 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi090-93363-54"}]': finished 2024-04-14T15:16:23.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:22 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/585933469' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi090-93363-54"}]: dispatch 2024-04-14T15:16:23.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:22 smithi092 ceph-mon[33966]: osdmap e315: 8 total, 8 up, 8 in 2024-04-14T15:16:23.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:22 smithi092 ceph-mon[33966]: from='client.48835 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi090-93363-54"}]: dispatch 2024-04-14T15:16:23.435 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:16:23.459 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:23.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:23.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:16:23.561 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:16:23.584 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:23.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:23.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:16:23.686 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:16:23.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:23.712 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:23.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:16:23.813 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:16:23.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:23.839 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:23.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:16:23.940 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:16:23.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:23.967 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:23.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:16:23.995 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-04-14T15:16:23.995 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-04-14T15:16:23.995 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 0 2024-04-14T15:16:23.996 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-04-14T15:16:24.067 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:24.091 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:24.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:24.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:24.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.170+0000 7f9c701c8700 1 -- 172.21.15.90:0/1046635975 >> v1:172.21.15.90:6789/0 conn(0x7f9c6810f360 legacy=0x7f9c68111800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:24.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.170+0000 7f9c701c8700 1 -- 172.21.15.90:0/1046635975 shutdown_connections 2024-04-14T15:16:24.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.170+0000 7f9c701c8700 1 -- 172.21.15.90:0/1046635975 >> 172.21.15.90:0/1046635975 conn(0x7f9c68076a70 msgr2=0x7f9c68076e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:16:24.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.170+0000 7f9c701c8700 1 -- 172.21.15.90:0/1046635975 shutdown_connections 2024-04-14T15:16:24.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.170+0000 7f9c701c8700 1 -- 172.21.15.90:0/1046635975 wait complete. 2024-04-14T15:16:24.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.170+0000 7f9c701c8700 1 librados: starting msgr at 2024-04-14T15:16:24.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.170+0000 7f9c701c8700 1 librados: starting objecter 2024-04-14T15:16:24.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.170+0000 7f9c701c8700 1 Processor -- start 2024-04-14T15:16:24.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.170+0000 7f9c701c8700 1 -- start start 2024-04-14T15:16:24.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.170+0000 7f9c701c8700 1 librados: setting wanted keys 2024-04-14T15:16:24.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.170+0000 7f9c701c8700 1 librados: calling monclient init 2024-04-14T15:16:24.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.171+0000 7f9c701c8700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f9c68103e70 con 0x7f9c6810f360 2024-04-14T15:16:24.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.171+0000 7f9c701c8700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f9c68104080 con 0x7f9c6810b7c0 2024-04-14T15:16:24.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.171+0000 7f9c701c8700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f9c681ac350 con 0x7f9c681012c0 2024-04-14T15:16:24.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.171+0000 7f9c6df64700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f9c681012c0 0x7f9c68102240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:33590/0 (socket says 172.21.15.90:33590) 2024-04-14T15:16:24.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.171+0000 7f9c6df64700 1 -- 172.21.15.90:0/2862082745 learned_addr learned my addr 172.21.15.90:0/2862082745 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:16:24.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.171+0000 7f9c5effd700 10 client.?.objecter ms_handle_connect 0x7f9c6810f360 2024-04-14T15:16:24.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.171+0000 7f9c5effd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:16:24.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.171+0000 7f9c5effd700 10 client.?.objecter ms_handle_connect 0x7f9c681012c0 2024-04-14T15:16:24.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.171+0000 7f9c5effd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:16:24.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.171+0000 7f9c5effd700 10 client.?.objecter ms_handle_connect 0x7f9c6810b7c0 2024-04-14T15:16:24.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.171+0000 7f9c5effd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:16:24.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.171+0000 7f9c5effd700 1 -- 172.21.15.90:0/2862082745 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1035372516 0 0) 0x7f9c68103e70 con 0x7f9c6810f360 2024-04-14T15:16:24.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.171+0000 7f9c5effd700 1 -- 172.21.15.90:0/2862082745 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9c540035b0 con 0x7f9c6810f360 2024-04-14T15:16:24.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.171+0000 7f9c5effd700 1 -- 172.21.15.90:0/2862082745 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 792036065 0 0) 0x7f9c681ac350 con 0x7f9c681012c0 2024-04-14T15:16:24.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.171+0000 7f9c5effd700 1 -- 172.21.15.90:0/2862082745 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9c68103e70 con 0x7f9c681012c0 2024-04-14T15:16:24.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.171+0000 7f9c5effd700 1 -- 172.21.15.90:0/2862082745 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3774756931 0 0) 0x7f9c68104080 con 0x7f9c6810b7c0 2024-04-14T15:16:24.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.171+0000 7f9c5effd700 1 -- 172.21.15.90:0/2862082745 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9c681ac350 con 0x7f9c6810b7c0 2024-04-14T15:16:24.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.171+0000 7f9c5effd700 1 -- 172.21.15.90:0/2862082745 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1293352140 0 0) 0x7f9c540035b0 con 0x7f9c6810f360 2024-04-14T15:16:24.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.171+0000 7f9c5effd700 1 -- 172.21.15.90:0/2862082745 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9c68104080 con 0x7f9c6810f360 2024-04-14T15:16:24.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.171+0000 7f9c5effd700 1 -- 172.21.15.90:0/2862082745 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3624389885 0 0) 0x7f9c681ac350 con 0x7f9c6810b7c0 2024-04-14T15:16:24.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.171+0000 7f9c5effd700 1 -- 172.21.15.90:0/2862082745 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9c540035b0 con 0x7f9c6810b7c0 2024-04-14T15:16:24.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.171+0000 7f9c5effd700 1 -- 172.21.15.90:0/2862082745 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f9c60001950 con 0x7f9c6810f360 2024-04-14T15:16:24.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.171+0000 7f9c5effd700 1 -- 172.21.15.90:0/2862082745 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3424377473 0 0) 0x7f9c68103e70 con 0x7f9c681012c0 2024-04-14T15:16:24.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.171+0000 7f9c5effd700 1 -- 172.21.15.90:0/2862082745 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9c681ac350 con 0x7f9c681012c0 2024-04-14T15:16:24.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.171+0000 7f9c5effd700 1 -- 172.21.15.90:0/2862082745 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f9c58001830 con 0x7f9c6810b7c0 2024-04-14T15:16:24.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.171+0000 7f9c5effd700 1 -- 172.21.15.90:0/2862082745 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f9c640017f0 con 0x7f9c681012c0 2024-04-14T15:16:24.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.171+0000 7f9c5effd700 1 -- 172.21.15.90:0/2862082745 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2013581398 0 0) 0x7f9c540035b0 con 0x7f9c6810b7c0 2024-04-14T15:16:24.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.172+0000 7f9c5effd700 1 -- 172.21.15.90:0/2862082745 >> v1:172.21.15.90:6790/0 conn(0x7f9c681012c0 legacy=0x7f9c68102240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:24.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.172+0000 7f9c5effd700 1 -- 172.21.15.90:0/2862082745 >> v1:172.21.15.90:6789/0 conn(0x7f9c6810f360 legacy=0x7f9c681a8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:24.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.172+0000 7f9c5effd700 1 -- 172.21.15.90:0/2862082745 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c681ad530 con 0x7f9c6810b7c0 2024-04-14T15:16:24.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.172+0000 7f9c701c8700 1 -- 172.21.15.90:0/2862082745 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9c681ac580 con 0x7f9c6810b7c0 2024-04-14T15:16:24.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.172+0000 7f9c701c8700 10 client.48859.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:16:24.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.172+0000 7f9c701c8700 1 -- 172.21.15.90:0/2862082745 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f9c681aca40 con 0x7f9c6810b7c0 2024-04-14T15:16:24.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.172+0000 7f9c701c8700 1 librados: init done 2024-04-14T15:16:24.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.172+0000 7f9c5effd700 1 -- 172.21.15.90:0/2862082745 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f9c58003b40 con 0x7f9c6810b7c0 2024-04-14T15:16:24.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.172+0000 7f9c5effd700 1 -- 172.21.15.90:0/2862082745 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f9c58004930 con 0x7f9c6810b7c0 2024-04-14T15:16:24.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.172+0000 7f9c701c8700 1 -- 172.21.15.90:0/2862082745 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f9c4c005100 con 0x7f9c6810b7c0 2024-04-14T15:16:24.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.173+0000 7f9c5effd700 1 -- 172.21.15.90:0/2862082745 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f9c580036a0 con 0x7f9c6810b7c0 2024-04-14T15:16:24.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.174+0000 7f9c5effd700 1 -- 172.21.15.90:0/2862082745 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(316..316 src has 51..316) ==== 8131+0+0 (unknown 1364441133 0 0) 0x7f9c580894b0 con 0x7f9c6810b7c0 2024-04-14T15:16:24.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.174+0000 7f9c5effd700 10 client.48859.objecter ms_dispatch 0x7f9c68001550 osd_map(316..316 src has 51..316) 2024-04-14T15:16:24.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.174+0000 7f9c5effd700 3 client.48859.objecter handle_osd_map got epochs [316,316] > 0 2024-04-14T15:16:24.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.174+0000 7f9c5effd700 3 client.48859.objecter handle_osd_map decoding full epoch 316 2024-04-14T15:16:24.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.174+0000 7f9c5effd700 20 client.48859.objecter dump_active .. 0 homeless 2024-04-14T15:16:24.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.174+0000 7f9c5effd700 1 -- 172.21.15.90:0/2862082745 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v316) ==== 54+0+16 (unknown 984223001 0 2420000192) 0x7f9c580898b0 con 0x7f9c6810b7c0 2024-04-14T15:16:24.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.174+0000 7f9c5effd700 10 client.48859.objecter ms_handle_connect 0x7f9c54071450 2024-04-14T15:16:24.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.174+0000 7f9c701c8700 10 client.48859.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:16:24.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.174+0000 7f9c701c8700 15 client.48859.objecter _session_command_op_assign -1 1 2024-04-14T15:16:24.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.174+0000 7f9c701c8700 20 client.48859.objecter _get_session s=0x7f9c4c0053e0 osd=0 3 2024-04-14T15:16:24.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.174+0000 7f9c701c8700 20 client.48859.objecter put_session s=0x7f9c4c0053e0 osd=0 3 2024-04-14T15:16:24.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.174+0000 7f9c701c8700 20 client.48859.objecter _get_session s=0x7f9c4c0053e0 osd=0 3 2024-04-14T15:16:24.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.174+0000 7f9c701c8700 15 client.48859.objecter _session_command_op_remove -1 1 2024-04-14T15:16:24.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.174+0000 7f9c701c8700 20 client.48859.objecter get_session s=0x7f9c4c0053e0 osd=0 3 2024-04-14T15:16:24.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.174+0000 7f9c701c8700 15 client.48859.objecter _session_command_op_assign 0 1 2024-04-14T15:16:24.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.174+0000 7f9c701c8700 20 client.48859.objecter put_session s=0x7f9c4c0053e0 osd=0 4 2024-04-14T15:16:24.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.174+0000 7f9c701c8700 10 client.48859.objecter _send_command 1 2024-04-14T15:16:24.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.174+0000 7f9c701c8700 1 -- 172.21.15.90:0/2862082745 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9c4c004b50 con 0x7f9c4c004730 2024-04-14T15:16:24.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.174+0000 7f9c5effd700 10 client.48859.objecter ms_handle_connect 0x7f9c4c004730 2024-04-14T15:16:24.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.175+0000 7f9c5effd700 1 -- 172.21.15.90:0/2862082745 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f9c4c004b50 con 0x7f9c4c004730 2024-04-14T15:16:24.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.175+0000 7f9c5effd700 10 client.48859.objecter ms_dispatch 0x7f9c68001550 command_reply(tid 1: 0 ) 2024-04-14T15:16:24.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.175+0000 7f9c5effd700 10 client.48859.objecter _finish_command 1 = system:0 2024-04-14T15:16:24.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.175+0000 7f9c5effd700 20 client.48859.objecter put_session s=0x7f9c4c0053e0 osd=0 4 2024-04-14T15:16:24.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.175+0000 7f9c5effd700 15 client.48859.objecter _session_command_op_remove 0 1 2024-04-14T15:16:24.191 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:16:23 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:16:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:16:24.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:23 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [7, 2]}]: dispatch 2024-04-14T15:16:24.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:23 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-04-14T15:16:24.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:23 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:16:24.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:23 smithi090 ceph-mon[41670]: from='client.48835 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi090-93363-54"}]': finished 2024-04-14T15:16:24.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:23 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [7, 2]}]': finished 2024-04-14T15:16:24.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:23 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]': finished 2024-04-14T15:16:24.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:23 smithi090 ceph-mon[41670]: osdmap e316: 8 total, 8 up, 8 in 2024-04-14T15:16:24.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:23 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:16:24.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:23 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:16:24.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:23 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/379496824' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi090-93363-55"}]: dispatch 2024-04-14T15:16:24.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:23 smithi090 ceph-mon[41670]: from='client.46967 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi090-93363-55"}]: dispatch 2024-04-14T15:16:24.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:23 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/379496824' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi090-93363-55"}]: dispatch 2024-04-14T15:16:24.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:23 smithi090 ceph-mon[41670]: from='client.46967 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi090-93363-55"}]: dispatch 2024-04-14T15:16:24.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:23 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/379496824' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi090-93363-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:16:24.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:23 smithi090 ceph-mon[41670]: from='client.46967 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi090-93363-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:16:24.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:23 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [7, 2]}]: dispatch 2024-04-14T15:16:24.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:23 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-04-14T15:16:24.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:23 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:16:24.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:23 smithi090 ceph-mon[30399]: from='client.48835 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi090-93363-54"}]': finished 2024-04-14T15:16:24.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:23 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [7, 2]}]': finished 2024-04-14T15:16:24.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:23 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]': finished 2024-04-14T15:16:24.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:23 smithi090 ceph-mon[30399]: osdmap e316: 8 total, 8 up, 8 in 2024-04-14T15:16:24.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:23 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:16:24.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:23 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:16:24.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:23 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/379496824' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi090-93363-55"}]: dispatch 2024-04-14T15:16:24.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:23 smithi090 ceph-mon[30399]: from='client.46967 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi090-93363-55"}]: dispatch 2024-04-14T15:16:24.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:23 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/379496824' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi090-93363-55"}]: dispatch 2024-04-14T15:16:24.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:23 smithi090 ceph-mon[30399]: from='client.46967 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi090-93363-55"}]: dispatch 2024-04-14T15:16:24.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:23 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/379496824' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi090-93363-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:16:24.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:23 smithi090 ceph-mon[30399]: from='client.46967 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi090-93363-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:16:24.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.192+0000 7f9c701c8700 10 client.48859.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:16:24.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.192+0000 7f9c701c8700 15 client.48859.objecter _session_command_op_assign -1 2 2024-04-14T15:16:24.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.192+0000 7f9c701c8700 20 client.48859.objecter _get_session s=0x7f9c4c0053e0 osd=0 3 2024-04-14T15:16:24.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.192+0000 7f9c701c8700 20 client.48859.objecter put_session s=0x7f9c4c0053e0 osd=0 3 2024-04-14T15:16:24.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.192+0000 7f9c701c8700 20 client.48859.objecter _get_session s=0x7f9c4c0053e0 osd=0 3 2024-04-14T15:16:24.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.192+0000 7f9c701c8700 15 client.48859.objecter _session_command_op_remove -1 2 2024-04-14T15:16:24.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.192+0000 7f9c701c8700 20 client.48859.objecter get_session s=0x7f9c4c0053e0 osd=0 3 2024-04-14T15:16:24.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.192+0000 7f9c701c8700 15 client.48859.objecter _session_command_op_assign 0 2 2024-04-14T15:16:24.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.192+0000 7f9c701c8700 20 client.48859.objecter put_session s=0x7f9c4c0053e0 osd=0 4 2024-04-14T15:16:24.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.192+0000 7f9c701c8700 10 client.48859.objecter _send_command 2 2024-04-14T15:16:24.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.192+0000 7f9c701c8700 1 -- 172.21.15.90:0/2862082745 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9c4c0079f0 con 0x7f9c4c004730 2024-04-14T15:16:24.196 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:16:24.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.194+0000 7f9c5effd700 1 -- 172.21.15.90:0/2862082745 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f9c4c0079f0 con 0x7f9c4c004730 2024-04-14T15:16:24.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.194+0000 7f9c5effd700 10 client.48859.objecter ms_dispatch 0x7f9c68001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:16:24.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.194+0000 7f9c5effd700 10 client.48859.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:16:24.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.194+0000 7f9c5effd700 20 client.48859.objecter put_session s=0x7f9c4c0053e0 osd=0 4 2024-04-14T15:16:24.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.194+0000 7f9c5effd700 15 client.48859.objecter _session_command_op_remove 0 2 2024-04-14T15:16:24.198 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:16:24.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.195+0000 7f9c701c8700 10 client.48859.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:16:24.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.195+0000 7f9c701c8700 15 client.48859.objecter _session_command_op_assign -1 3 2024-04-14T15:16:24.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.195+0000 7f9c701c8700 20 client.48859.objecter _get_session s=0x7f9c4c004110 osd=1 3 2024-04-14T15:16:24.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.195+0000 7f9c701c8700 20 client.48859.objecter put_session s=0x7f9c4c004110 osd=1 3 2024-04-14T15:16:24.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.195+0000 7f9c701c8700 20 client.48859.objecter _get_session s=0x7f9c4c004110 osd=1 3 2024-04-14T15:16:24.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.195+0000 7f9c701c8700 15 client.48859.objecter _session_command_op_remove -1 3 2024-04-14T15:16:24.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.195+0000 7f9c701c8700 20 client.48859.objecter get_session s=0x7f9c4c004110 osd=1 3 2024-04-14T15:16:24.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.195+0000 7f9c701c8700 15 client.48859.objecter _session_command_op_assign 1 3 2024-04-14T15:16:24.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.195+0000 7f9c701c8700 20 client.48859.objecter put_session s=0x7f9c4c004110 osd=1 4 2024-04-14T15:16:24.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.195+0000 7f9c701c8700 10 client.48859.objecter _send_command 3 2024-04-14T15:16:24.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.195+0000 7f9c701c8700 1 -- 172.21.15.90:0/2862082745 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f9c4c002130 con 0x7f9c4c004280 2024-04-14T15:16:24.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.195+0000 7f9c5effd700 10 client.48859.objecter ms_handle_connect 0x7f9c4c004280 2024-04-14T15:16:24.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.196+0000 7f9c5effd700 1 -- 172.21.15.90:0/2862082745 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f9c4c002130 con 0x7f9c4c004280 2024-04-14T15:16:24.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.196+0000 7f9c5effd700 10 client.48859.objecter ms_dispatch 0x7f9c68001550 command_reply(tid 3: 0 ) 2024-04-14T15:16:24.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.196+0000 7f9c5effd700 10 client.48859.objecter _finish_command 3 = system:0 2024-04-14T15:16:24.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.196+0000 7f9c5effd700 20 client.48859.objecter put_session s=0x7f9c4c004110 osd=1 4 2024-04-14T15:16:24.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.196+0000 7f9c5effd700 15 client.48859.objecter _session_command_op_remove 1 3 2024-04-14T15:16:24.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.212+0000 7f9c701c8700 10 client.48859.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:16:24.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.212+0000 7f9c701c8700 15 client.48859.objecter _session_command_op_assign -1 4 2024-04-14T15:16:24.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.212+0000 7f9c701c8700 20 client.48859.objecter _get_session s=0x7f9c4c004110 osd=1 3 2024-04-14T15:16:24.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.212+0000 7f9c701c8700 20 client.48859.objecter put_session s=0x7f9c4c004110 osd=1 3 2024-04-14T15:16:24.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.212+0000 7f9c701c8700 20 client.48859.objecter _get_session s=0x7f9c4c004110 osd=1 3 2024-04-14T15:16:24.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.212+0000 7f9c701c8700 15 client.48859.objecter _session_command_op_remove -1 4 2024-04-14T15:16:24.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.212+0000 7f9c701c8700 20 client.48859.objecter get_session s=0x7f9c4c004110 osd=1 3 2024-04-14T15:16:24.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.212+0000 7f9c701c8700 15 client.48859.objecter _session_command_op_assign 1 4 2024-04-14T15:16:24.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.212+0000 7f9c701c8700 20 client.48859.objecter put_session s=0x7f9c4c004110 osd=1 4 2024-04-14T15:16:24.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.213+0000 7f9c701c8700 10 client.48859.objecter _send_command 4 2024-04-14T15:16:24.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.213+0000 7f9c701c8700 1 -- 172.21.15.90:0/2862082745 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9c4c001600 con 0x7f9c4c004280 2024-04-14T15:16:24.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.214+0000 7f9c5effd700 1 -- 172.21.15.90:0/2862082745 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f9c4c001600 con 0x7f9c4c004280 2024-04-14T15:16:24.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.214+0000 7f9c5effd700 10 client.48859.objecter ms_dispatch 0x7f9c68001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:16:24.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.214+0000 7f9c5effd700 10 client.48859.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:16:24.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.214+0000 7f9c5effd700 20 client.48859.objecter put_session s=0x7f9c4c004110 osd=1 4 2024-04-14T15:16:24.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.214+0000 7f9c5effd700 15 client.48859.objecter _session_command_op_remove 1 4 2024-04-14T15:16:24.215 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:16:24.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.214+0000 7f9c701c8700 10 client.48859.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:16:24.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.214+0000 7f9c701c8700 15 client.48859.objecter _session_command_op_assign -1 5 2024-04-14T15:16:24.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.214+0000 7f9c701c8700 20 client.48859.objecter _get_session s=0x7f9c4c001a00 osd=2 3 2024-04-14T15:16:24.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.214+0000 7f9c701c8700 20 client.48859.objecter put_session s=0x7f9c4c001a00 osd=2 3 2024-04-14T15:16:24.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.214+0000 7f9c701c8700 20 client.48859.objecter _get_session s=0x7f9c4c001a00 osd=2 3 2024-04-14T15:16:24.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.214+0000 7f9c701c8700 15 client.48859.objecter _session_command_op_remove -1 5 2024-04-14T15:16:24.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.214+0000 7f9c701c8700 20 client.48859.objecter get_session s=0x7f9c4c001a00 osd=2 3 2024-04-14T15:16:24.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.214+0000 7f9c701c8700 15 client.48859.objecter _session_command_op_assign 2 5 2024-04-14T15:16:24.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.214+0000 7f9c701c8700 20 client.48859.objecter put_session s=0x7f9c4c001a00 osd=2 4 2024-04-14T15:16:24.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.214+0000 7f9c701c8700 10 client.48859.objecter _send_command 5 2024-04-14T15:16:24.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.214+0000 7f9c701c8700 1 -- 172.21.15.90:0/2862082745 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f9c4c010f60 con 0x7f9c4c0088c0 2024-04-14T15:16:24.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.215+0000 7f9c5effd700 10 client.48859.objecter ms_handle_connect 0x7f9c4c0088c0 2024-04-14T15:16:24.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.215+0000 7f9c5effd700 1 -- 172.21.15.90:0/2862082745 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f9c4c010f60 con 0x7f9c4c0088c0 2024-04-14T15:16:24.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.215+0000 7f9c5effd700 10 client.48859.objecter ms_dispatch 0x7f9c68001550 command_reply(tid 5: 0 ) 2024-04-14T15:16:24.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.215+0000 7f9c5effd700 10 client.48859.objecter _finish_command 5 = system:0 2024-04-14T15:16:24.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.215+0000 7f9c5effd700 20 client.48859.objecter put_session s=0x7f9c4c001a00 osd=2 4 2024-04-14T15:16:24.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.215+0000 7f9c5effd700 15 client.48859.objecter _session_command_op_remove 2 5 2024-04-14T15:16:24.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:24.222 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:24.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:16:24.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.232+0000 7f9c701c8700 10 client.48859.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:16:24.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.232+0000 7f9c701c8700 15 client.48859.objecter _session_command_op_assign -1 6 2024-04-14T15:16:24.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.232+0000 7f9c701c8700 20 client.48859.objecter _get_session s=0x7f9c4c001a00 osd=2 3 2024-04-14T15:16:24.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.232+0000 7f9c701c8700 20 client.48859.objecter put_session s=0x7f9c4c001a00 osd=2 3 2024-04-14T15:16:24.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.232+0000 7f9c701c8700 20 client.48859.objecter _get_session s=0x7f9c4c001a00 osd=2 3 2024-04-14T15:16:24.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.232+0000 7f9c701c8700 15 client.48859.objecter _session_command_op_remove -1 6 2024-04-14T15:16:24.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.232+0000 7f9c701c8700 20 client.48859.objecter get_session s=0x7f9c4c001a00 osd=2 3 2024-04-14T15:16:24.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.232+0000 7f9c701c8700 15 client.48859.objecter _session_command_op_assign 2 6 2024-04-14T15:16:24.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.232+0000 7f9c701c8700 20 client.48859.objecter put_session s=0x7f9c4c001a00 osd=2 4 2024-04-14T15:16:24.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.232+0000 7f9c701c8700 10 client.48859.objecter _send_command 6 2024-04-14T15:16:24.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.232+0000 7f9c701c8700 1 -- 172.21.15.90:0/2862082745 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9c4c007e50 con 0x7f9c4c0088c0 2024-04-14T15:16:24.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.234+0000 7f9c5effd700 1 -- 172.21.15.90:0/2862082745 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f9c4c007e50 con 0x7f9c4c0088c0 2024-04-14T15:16:24.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.234+0000 7f9c5effd700 10 client.48859.objecter ms_dispatch 0x7f9c68001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:16:24.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.234+0000 7f9c5effd700 10 client.48859.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:16:24.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.234+0000 7f9c5effd700 20 client.48859.objecter put_session s=0x7f9c4c001a00 osd=2 4 2024-04-14T15:16:24.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.234+0000 7f9c5effd700 15 client.48859.objecter _session_command_op_remove 2 6 2024-04-14T15:16:24.235 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:16:24.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.234+0000 7f9c701c8700 10 client.48859.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:16:24.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.234+0000 7f9c701c8700 15 client.48859.objecter _session_command_op_assign -1 7 2024-04-14T15:16:24.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.234+0000 7f9c701c8700 20 client.48859.objecter _get_session s=0x7f9c4c00cc20 osd=3 3 2024-04-14T15:16:24.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.234+0000 7f9c701c8700 20 client.48859.objecter put_session s=0x7f9c4c00cc20 osd=3 3 2024-04-14T15:16:24.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.234+0000 7f9c701c8700 20 client.48859.objecter _get_session s=0x7f9c4c00cc20 osd=3 3 2024-04-14T15:16:24.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.234+0000 7f9c701c8700 15 client.48859.objecter _session_command_op_remove -1 7 2024-04-14T15:16:24.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.234+0000 7f9c701c8700 20 client.48859.objecter get_session s=0x7f9c4c00cc20 osd=3 3 2024-04-14T15:16:24.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.234+0000 7f9c701c8700 15 client.48859.objecter _session_command_op_assign 3 7 2024-04-14T15:16:24.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.234+0000 7f9c701c8700 20 client.48859.objecter put_session s=0x7f9c4c00cc20 osd=3 4 2024-04-14T15:16:24.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.234+0000 7f9c701c8700 10 client.48859.objecter _send_command 7 2024-04-14T15:16:24.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.234+0000 7f9c701c8700 1 -- 172.21.15.90:0/2862082745 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f9c4c011c60 con 0x7f9c4c011110 2024-04-14T15:16:24.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.234+0000 7f9c5effd700 10 client.48859.objecter ms_handle_connect 0x7f9c4c011110 2024-04-14T15:16:24.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.235+0000 7f9c5effd700 1 -- 172.21.15.90:0/2862082745 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f9c4c011c60 con 0x7f9c4c011110 2024-04-14T15:16:24.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.235+0000 7f9c5effd700 10 client.48859.objecter ms_dispatch 0x7f9c68001550 command_reply(tid 7: 0 ) 2024-04-14T15:16:24.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.235+0000 7f9c5effd700 10 client.48859.objecter _finish_command 7 = system:0 2024-04-14T15:16:24.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.235+0000 7f9c5effd700 20 client.48859.objecter put_session s=0x7f9c4c00cc20 osd=3 4 2024-04-14T15:16:24.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.235+0000 7f9c5effd700 15 client.48859.objecter _session_command_op_remove 3 7 2024-04-14T15:16:24.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.251+0000 7f9c701c8700 10 client.48859.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:16:24.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.251+0000 7f9c701c8700 15 client.48859.objecter _session_command_op_assign -1 8 2024-04-14T15:16:24.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.251+0000 7f9c701c8700 20 client.48859.objecter _get_session s=0x7f9c4c00cc20 osd=3 3 2024-04-14T15:16:24.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.251+0000 7f9c701c8700 20 client.48859.objecter put_session s=0x7f9c4c00cc20 osd=3 3 2024-04-14T15:16:24.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.251+0000 7f9c701c8700 20 client.48859.objecter _get_session s=0x7f9c4c00cc20 osd=3 3 2024-04-14T15:16:24.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.251+0000 7f9c701c8700 15 client.48859.objecter _session_command_op_remove -1 8 2024-04-14T15:16:24.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.251+0000 7f9c701c8700 20 client.48859.objecter get_session s=0x7f9c4c00cc20 osd=3 3 2024-04-14T15:16:24.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.251+0000 7f9c701c8700 15 client.48859.objecter _session_command_op_assign 3 8 2024-04-14T15:16:24.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.251+0000 7f9c701c8700 20 client.48859.objecter put_session s=0x7f9c4c00cc20 osd=3 4 2024-04-14T15:16:24.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.251+0000 7f9c701c8700 10 client.48859.objecter _send_command 8 2024-04-14T15:16:24.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.251+0000 7f9c701c8700 1 -- 172.21.15.90:0/2862082745 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9c4c0125e0 con 0x7f9c4c011110 2024-04-14T15:16:24.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.253+0000 7f9c5effd700 1 -- 172.21.15.90:0/2862082745 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f9c4c0125e0 con 0x7f9c4c011110 2024-04-14T15:16:24.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.253+0000 7f9c5effd700 10 client.48859.objecter ms_dispatch 0x7f9c68001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:16:24.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.253+0000 7f9c5effd700 10 client.48859.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:16:24.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.253+0000 7f9c5effd700 20 client.48859.objecter put_session s=0x7f9c4c00cc20 osd=3 4 2024-04-14T15:16:24.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.253+0000 7f9c5effd700 15 client.48859.objecter _session_command_op_remove 3 8 2024-04-14T15:16:24.254 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:16:24.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.253+0000 7f9c701c8700 10 client.48859.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:16:24.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.253+0000 7f9c701c8700 15 client.48859.objecter _session_command_op_assign -1 9 2024-04-14T15:16:24.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.253+0000 7f9c701c8700 20 client.48859.objecter _get_session s=0x7f9c4c0125e0 osd=4 3 2024-04-14T15:16:24.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.253+0000 7f9c701c8700 20 client.48859.objecter put_session s=0x7f9c4c0125e0 osd=4 3 2024-04-14T15:16:24.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.253+0000 7f9c701c8700 20 client.48859.objecter _get_session s=0x7f9c4c0125e0 osd=4 3 2024-04-14T15:16:24.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.253+0000 7f9c701c8700 15 client.48859.objecter _session_command_op_remove -1 9 2024-04-14T15:16:24.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.253+0000 7f9c701c8700 20 client.48859.objecter get_session s=0x7f9c4c0125e0 osd=4 3 2024-04-14T15:16:24.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.253+0000 7f9c701c8700 15 client.48859.objecter _session_command_op_assign 4 9 2024-04-14T15:16:24.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.253+0000 7f9c701c8700 20 client.48859.objecter put_session s=0x7f9c4c0125e0 osd=4 4 2024-04-14T15:16:24.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.253+0000 7f9c701c8700 10 client.48859.objecter _send_command 9 2024-04-14T15:16:24.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.253+0000 7f9c701c8700 1 -- 172.21.15.90:0/2862082745 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f9c4c00ca60 con 0x7f9c4c017030 2024-04-14T15:16:24.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.254+0000 7f9c5effd700 10 client.48859.objecter ms_handle_connect 0x7f9c4c017030 2024-04-14T15:16:24.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.254+0000 7f9c5effd700 1 -- 172.21.15.90:0/2862082745 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f9c6400ca40 con 0x7f9c4c017030 2024-04-14T15:16:24.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.254+0000 7f9c5effd700 10 client.48859.objecter ms_dispatch 0x7f9c68001550 command_reply(tid 9: 0 ) 2024-04-14T15:16:24.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.254+0000 7f9c5effd700 10 client.48859.objecter _finish_command 9 = system:0 2024-04-14T15:16:24.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.254+0000 7f9c5effd700 20 client.48859.objecter put_session s=0x7f9c4c0125e0 osd=4 4 2024-04-14T15:16:24.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.254+0000 7f9c5effd700 15 client.48859.objecter _session_command_op_remove 4 9 2024-04-14T15:16:24.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.271+0000 7f9c701c8700 10 client.48859.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:16:24.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.271+0000 7f9c701c8700 15 client.48859.objecter _session_command_op_assign -1 10 2024-04-14T15:16:24.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.271+0000 7f9c701c8700 20 client.48859.objecter _get_session s=0x7f9c4c0125e0 osd=4 3 2024-04-14T15:16:24.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.271+0000 7f9c701c8700 20 client.48859.objecter put_session s=0x7f9c4c0125e0 osd=4 3 2024-04-14T15:16:24.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.271+0000 7f9c701c8700 20 client.48859.objecter _get_session s=0x7f9c4c0125e0 osd=4 3 2024-04-14T15:16:24.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.271+0000 7f9c701c8700 15 client.48859.objecter _session_command_op_remove -1 10 2024-04-14T15:16:24.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.271+0000 7f9c701c8700 20 client.48859.objecter get_session s=0x7f9c4c0125e0 osd=4 3 2024-04-14T15:16:24.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.271+0000 7f9c701c8700 15 client.48859.objecter _session_command_op_assign 4 10 2024-04-14T15:16:24.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.271+0000 7f9c701c8700 20 client.48859.objecter put_session s=0x7f9c4c0125e0 osd=4 4 2024-04-14T15:16:24.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.271+0000 7f9c701c8700 10 client.48859.objecter _send_command 10 2024-04-14T15:16:24.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.271+0000 7f9c701c8700 1 -- 172.21.15.90:0/2862082745 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9c4c00c700 con 0x7f9c4c017030 2024-04-14T15:16:24.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.272+0000 7f9c5effd700 1 -- 172.21.15.90:0/2862082745 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f9c4c00c700 con 0x7f9c4c017030 2024-04-14T15:16:24.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.273+0000 7f9c5effd700 10 client.48859.objecter ms_dispatch 0x7f9c68001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:16:24.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.273+0000 7f9c5effd700 10 client.48859.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:16:24.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.273+0000 7f9c5effd700 20 client.48859.objecter put_session s=0x7f9c4c0125e0 osd=4 4 2024-04-14T15:16:24.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.273+0000 7f9c5effd700 15 client.48859.objecter _session_command_op_remove 4 10 2024-04-14T15:16:24.273 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:16:24.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.273+0000 7f9c701c8700 10 client.48859.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:16:24.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.273+0000 7f9c701c8700 15 client.48859.objecter _session_command_op_assign -1 11 2024-04-14T15:16:24.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.273+0000 7f9c701c8700 20 client.48859.objecter _get_session s=0x7f9c4c00c700 osd=5 3 2024-04-14T15:16:24.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.273+0000 7f9c701c8700 20 client.48859.objecter put_session s=0x7f9c4c00c700 osd=5 3 2024-04-14T15:16:24.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.273+0000 7f9c701c8700 20 client.48859.objecter _get_session s=0x7f9c4c00c700 osd=5 3 2024-04-14T15:16:24.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.273+0000 7f9c701c8700 15 client.48859.objecter _session_command_op_remove -1 11 2024-04-14T15:16:24.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.273+0000 7f9c701c8700 20 client.48859.objecter get_session s=0x7f9c4c00c700 osd=5 3 2024-04-14T15:16:24.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.273+0000 7f9c701c8700 15 client.48859.objecter _session_command_op_assign 5 11 2024-04-14T15:16:24.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.273+0000 7f9c701c8700 20 client.48859.objecter put_session s=0x7f9c4c00c700 osd=5 4 2024-04-14T15:16:24.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.273+0000 7f9c701c8700 10 client.48859.objecter _send_command 11 2024-04-14T15:16:24.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.273+0000 7f9c701c8700 1 -- 172.21.15.90:0/2862082745 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f9c4c0136c0 con 0x7f9c4c01bcf0 2024-04-14T15:16:24.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.273+0000 7f9c5effd700 10 client.48859.objecter ms_handle_connect 0x7f9c4c01bcf0 2024-04-14T15:16:24.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.274+0000 7f9c5effd700 1 -- 172.21.15.90:0/2862082745 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f9c4c0136c0 con 0x7f9c4c01bcf0 2024-04-14T15:16:24.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.274+0000 7f9c5effd700 10 client.48859.objecter ms_dispatch 0x7f9c68001550 command_reply(tid 11: 0 ) 2024-04-14T15:16:24.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.274+0000 7f9c5effd700 10 client.48859.objecter _finish_command 11 = system:0 2024-04-14T15:16:24.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.274+0000 7f9c5effd700 20 client.48859.objecter put_session s=0x7f9c4c00c700 osd=5 4 2024-04-14T15:16:24.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.274+0000 7f9c5effd700 15 client.48859.objecter _session_command_op_remove 5 11 2024-04-14T15:16:24.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.290+0000 7f9c701c8700 10 client.48859.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:16:24.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.290+0000 7f9c701c8700 15 client.48859.objecter _session_command_op_assign -1 12 2024-04-14T15:16:24.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.290+0000 7f9c701c8700 20 client.48859.objecter _get_session s=0x7f9c4c00c700 osd=5 3 2024-04-14T15:16:24.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.290+0000 7f9c701c8700 20 client.48859.objecter put_session s=0x7f9c4c00c700 osd=5 3 2024-04-14T15:16:24.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.290+0000 7f9c701c8700 20 client.48859.objecter _get_session s=0x7f9c4c00c700 osd=5 3 2024-04-14T15:16:24.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.290+0000 7f9c701c8700 15 client.48859.objecter _session_command_op_remove -1 12 2024-04-14T15:16:24.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.290+0000 7f9c701c8700 20 client.48859.objecter get_session s=0x7f9c4c00c700 osd=5 3 2024-04-14T15:16:24.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.290+0000 7f9c701c8700 15 client.48859.objecter _session_command_op_assign 5 12 2024-04-14T15:16:24.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.290+0000 7f9c701c8700 20 client.48859.objecter put_session s=0x7f9c4c00c700 osd=5 4 2024-04-14T15:16:24.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.290+0000 7f9c701c8700 10 client.48859.objecter _send_command 12 2024-04-14T15:16:24.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.290+0000 7f9c701c8700 1 -- 172.21.15.90:0/2862082745 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9c4c0136c0 con 0x7f9c4c01bcf0 2024-04-14T15:16:24.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.292+0000 7f9c5effd700 1 -- 172.21.15.90:0/2862082745 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f9c4c0136c0 con 0x7f9c4c01bcf0 2024-04-14T15:16:24.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.292+0000 7f9c5effd700 10 client.48859.objecter ms_dispatch 0x7f9c68001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:16:24.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.292+0000 7f9c5effd700 10 client.48859.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:16:24.293 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:16:24.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.292+0000 7f9c5effd700 20 client.48859.objecter put_session s=0x7f9c4c00c700 osd=5 4 2024-04-14T15:16:24.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.292+0000 7f9c5effd700 15 client.48859.objecter _session_command_op_remove 5 12 2024-04-14T15:16:24.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.293+0000 7f9c701c8700 10 client.48859.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:16:24.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.293+0000 7f9c701c8700 15 client.48859.objecter _session_command_op_assign -1 13 2024-04-14T15:16:24.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.293+0000 7f9c701c8700 20 client.48859.objecter _get_session s=0x7f9c4c0136c0 osd=6 3 2024-04-14T15:16:24.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.293+0000 7f9c701c8700 20 client.48859.objecter put_session s=0x7f9c4c0136c0 osd=6 3 2024-04-14T15:16:24.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.293+0000 7f9c701c8700 20 client.48859.objecter _get_session s=0x7f9c4c0136c0 osd=6 3 2024-04-14T15:16:24.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.293+0000 7f9c701c8700 15 client.48859.objecter _session_command_op_remove -1 13 2024-04-14T15:16:24.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.293+0000 7f9c701c8700 20 client.48859.objecter get_session s=0x7f9c4c0136c0 osd=6 3 2024-04-14T15:16:24.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.293+0000 7f9c701c8700 15 client.48859.objecter _session_command_op_assign 6 13 2024-04-14T15:16:24.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.293+0000 7f9c701c8700 20 client.48859.objecter put_session s=0x7f9c4c0136c0 osd=6 4 2024-04-14T15:16:24.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.293+0000 7f9c701c8700 10 client.48859.objecter _send_command 13 2024-04-14T15:16:24.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.293+0000 7f9c701c8700 1 -- 172.21.15.90:0/2862082745 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f9c4c0234d0 con 0x7f9c4c01f870 2024-04-14T15:16:24.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.293+0000 7f9c5effd700 10 client.48859.objecter ms_handle_connect 0x7f9c4c01f870 2024-04-14T15:16:24.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.294+0000 7f9c5effd700 1 -- 172.21.15.90:0/2862082745 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f9c4c0234d0 con 0x7f9c4c01f870 2024-04-14T15:16:24.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.294+0000 7f9c5effd700 10 client.48859.objecter ms_dispatch 0x7f9c68001550 command_reply(tid 13: 0 ) 2024-04-14T15:16:24.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.294+0000 7f9c5effd700 10 client.48859.objecter _finish_command 13 = system:0 2024-04-14T15:16:24.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.294+0000 7f9c5effd700 20 client.48859.objecter put_session s=0x7f9c4c0136c0 osd=6 4 2024-04-14T15:16:24.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.294+0000 7f9c5effd700 15 client.48859.objecter _session_command_op_remove 6 13 2024-04-14T15:16:24.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.310+0000 7f9c701c8700 10 client.48859.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:16:24.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.310+0000 7f9c701c8700 15 client.48859.objecter _session_command_op_assign -1 14 2024-04-14T15:16:24.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.310+0000 7f9c701c8700 20 client.48859.objecter _get_session s=0x7f9c4c0136c0 osd=6 3 2024-04-14T15:16:24.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.310+0000 7f9c701c8700 20 client.48859.objecter put_session s=0x7f9c4c0136c0 osd=6 3 2024-04-14T15:16:24.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.310+0000 7f9c701c8700 20 client.48859.objecter _get_session s=0x7f9c4c0136c0 osd=6 3 2024-04-14T15:16:24.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.310+0000 7f9c701c8700 15 client.48859.objecter _session_command_op_remove -1 14 2024-04-14T15:16:24.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.310+0000 7f9c701c8700 20 client.48859.objecter get_session s=0x7f9c4c0136c0 osd=6 3 2024-04-14T15:16:24.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.310+0000 7f9c701c8700 15 client.48859.objecter _session_command_op_assign 6 14 2024-04-14T15:16:24.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.310+0000 7f9c701c8700 20 client.48859.objecter put_session s=0x7f9c4c0136c0 osd=6 4 2024-04-14T15:16:24.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.310+0000 7f9c701c8700 10 client.48859.objecter _send_command 14 2024-04-14T15:16:24.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.310+0000 7f9c701c8700 1 -- 172.21.15.90:0/2862082745 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9c4c011ed0 con 0x7f9c4c01f870 2024-04-14T15:16:24.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.312+0000 7f9c5effd700 1 -- 172.21.15.90:0/2862082745 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f9c4c011ed0 con 0x7f9c4c01f870 2024-04-14T15:16:24.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.312+0000 7f9c5effd700 10 client.48859.objecter ms_dispatch 0x7f9c68001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:16:24.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.312+0000 7f9c5effd700 10 client.48859.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:16:24.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.312+0000 7f9c5effd700 20 client.48859.objecter put_session s=0x7f9c4c0136c0 osd=6 4 2024-04-14T15:16:24.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.312+0000 7f9c5effd700 15 client.48859.objecter _session_command_op_remove 6 14 2024-04-14T15:16:24.313 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:16:24.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.312+0000 7f9c701c8700 10 client.48859.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:16:24.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.312+0000 7f9c701c8700 15 client.48859.objecter _session_command_op_assign -1 15 2024-04-14T15:16:24.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.312+0000 7f9c701c8700 20 client.48859.objecter _get_session s=0x7f9c4c011f10 osd=7 3 2024-04-14T15:16:24.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.312+0000 7f9c701c8700 20 client.48859.objecter put_session s=0x7f9c4c011f10 osd=7 3 2024-04-14T15:16:24.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.312+0000 7f9c701c8700 20 client.48859.objecter _get_session s=0x7f9c4c011f10 osd=7 3 2024-04-14T15:16:24.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.312+0000 7f9c701c8700 15 client.48859.objecter _session_command_op_remove -1 15 2024-04-14T15:16:24.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.312+0000 7f9c701c8700 20 client.48859.objecter get_session s=0x7f9c4c011f10 osd=7 3 2024-04-14T15:16:24.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.312+0000 7f9c701c8700 15 client.48859.objecter _session_command_op_assign 7 15 2024-04-14T15:16:24.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.312+0000 7f9c701c8700 20 client.48859.objecter put_session s=0x7f9c4c011f10 osd=7 4 2024-04-14T15:16:24.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.312+0000 7f9c701c8700 10 client.48859.objecter _send_command 15 2024-04-14T15:16:24.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.312+0000 7f9c701c8700 1 -- 172.21.15.90:0/2862082745 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f9c4c024c30 con 0x7f9c4c01b340 2024-04-14T15:16:24.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.313+0000 7f9c5effd700 10 client.48859.objecter ms_handle_connect 0x7f9c4c01b340 2024-04-14T15:16:24.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.313+0000 7f9c5effd700 1 -- 172.21.15.90:0/2862082745 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f9c4c024c30 con 0x7f9c4c01b340 2024-04-14T15:16:24.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.313+0000 7f9c5effd700 10 client.48859.objecter ms_dispatch 0x7f9c68001550 command_reply(tid 15: 0 ) 2024-04-14T15:16:24.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.313+0000 7f9c5effd700 10 client.48859.objecter _finish_command 15 = system:0 2024-04-14T15:16:24.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.313+0000 7f9c5effd700 20 client.48859.objecter put_session s=0x7f9c4c011f10 osd=7 4 2024-04-14T15:16:24.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.313+0000 7f9c5effd700 15 client.48859.objecter _session_command_op_remove 7 15 2024-04-14T15:16:24.322 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:23 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [7, 2]}]: dispatch 2024-04-14T15:16:24.322 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:23 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-04-14T15:16:24.322 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:23 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:16:24.322 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:23 smithi092 ceph-mon[33966]: from='client.48835 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi090-93363-54"}]': finished 2024-04-14T15:16:24.322 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:23 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [7, 2]}]': finished 2024-04-14T15:16:24.322 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:23 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]': finished 2024-04-14T15:16:24.322 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:23 smithi092 ceph-mon[33966]: osdmap e316: 8 total, 8 up, 8 in 2024-04-14T15:16:24.322 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:23 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:16:24.322 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:23 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:16:24.322 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:23 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/379496824' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi090-93363-55"}]: dispatch 2024-04-14T15:16:24.322 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:23 smithi092 ceph-mon[33966]: from='client.46967 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi090-93363-55"}]: dispatch 2024-04-14T15:16:24.322 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:23 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/379496824' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi090-93363-55"}]: dispatch 2024-04-14T15:16:24.323 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:23 smithi092 ceph-mon[33966]: from='client.46967 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi090-93363-55"}]: dispatch 2024-04-14T15:16:24.323 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:23 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/379496824' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi090-93363-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:16:24.323 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:23 smithi092 ceph-mon[33966]: from='client.46967 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi090-93363-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:16:24.323 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:16:24.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.330+0000 7f9c701c8700 10 client.48859.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:16:24.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.330+0000 7f9c701c8700 15 client.48859.objecter _session_command_op_assign -1 16 2024-04-14T15:16:24.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.330+0000 7f9c701c8700 20 client.48859.objecter _get_session s=0x7f9c4c011f10 osd=7 3 2024-04-14T15:16:24.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.330+0000 7f9c701c8700 20 client.48859.objecter put_session s=0x7f9c4c011f10 osd=7 3 2024-04-14T15:16:24.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.330+0000 7f9c701c8700 20 client.48859.objecter _get_session s=0x7f9c4c011f10 osd=7 3 2024-04-14T15:16:24.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.330+0000 7f9c701c8700 15 client.48859.objecter _session_command_op_remove -1 16 2024-04-14T15:16:24.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.330+0000 7f9c701c8700 20 client.48859.objecter get_session s=0x7f9c4c011f10 osd=7 3 2024-04-14T15:16:24.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.330+0000 7f9c701c8700 15 client.48859.objecter _session_command_op_assign 7 16 2024-04-14T15:16:24.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.330+0000 7f9c701c8700 20 client.48859.objecter put_session s=0x7f9c4c011f10 osd=7 4 2024-04-14T15:16:24.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.330+0000 7f9c701c8700 10 client.48859.objecter _send_command 16 2024-04-14T15:16:24.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.330+0000 7f9c701c8700 1 -- 172.21.15.90:0/2862082745 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9c4c024c30 con 0x7f9c4c01b340 2024-04-14T15:16:24.332 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:16:24.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.332+0000 7f9c5effd700 1 -- 172.21.15.90:0/2862082745 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f9c4c024c30 con 0x7f9c4c01b340 2024-04-14T15:16:24.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.332+0000 7f9c5effd700 10 client.48859.objecter ms_dispatch 0x7f9c68001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:16:24.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.332+0000 7f9c5effd700 10 client.48859.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:16:24.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.332+0000 7f9c5effd700 20 client.48859.objecter put_session s=0x7f9c4c011f10 osd=7 4 2024-04-14T15:16:24.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.332+0000 7f9c5effd700 15 client.48859.objecter _session_command_op_remove 7 16 2024-04-14T15:16:24.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.332+0000 7f9c701c8700 10 librados: watch_flush enter 2024-04-14T15:16:24.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.332+0000 7f9c701c8700 10 librados: watch_flush exit 2024-04-14T15:16:24.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.332+0000 7f9c701c8700 10 client.48859.objecter close_session for osd.0 2024-04-14T15:16:24.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.332+0000 7f9c701c8700 1 -- 172.21.15.90:0/2862082745 >> v1:172.21.15.90:6801/3761806885 conn(0x7f9c4c004730 legacy=0x7f9c4c002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:24.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.332+0000 7f9c701c8700 20 client.48859.objecter put_session s=0x7f9c4c0053e0 osd=0 1 2024-04-14T15:16:24.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.332+0000 7f9c701c8700 10 client.48859.objecter close_session for osd.1 2024-04-14T15:16:24.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.332+0000 7f9c701c8700 1 -- 172.21.15.90:0/2862082745 >> v1:172.21.15.90:6805/4170361699 conn(0x7f9c4c004280 legacy=0x7f9c4c00ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:24.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.332+0000 7f9c701c8700 20 client.48859.objecter put_session s=0x7f9c4c004110 osd=1 1 2024-04-14T15:16:24.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.332+0000 7f9c701c8700 10 client.48859.objecter close_session for osd.2 2024-04-14T15:16:24.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.332+0000 7f9c701c8700 1 -- 172.21.15.90:0/2862082745 >> v1:172.21.15.90:6809/1686184251 conn(0x7f9c4c0088c0 legacy=0x7f9c4c00f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:24.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.332+0000 7f9c701c8700 20 client.48859.objecter put_session s=0x7f9c4c001a00 osd=2 1 2024-04-14T15:16:24.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.332+0000 7f9c701c8700 10 client.48859.objecter close_session for osd.3 2024-04-14T15:16:24.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.332+0000 7f9c701c8700 1 -- 172.21.15.90:0/2862082745 >> v1:172.21.15.90:6813/2270166774 conn(0x7f9c4c011110 legacy=0x7f9c4c011550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:24.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.332+0000 7f9c701c8700 20 client.48859.objecter put_session s=0x7f9c4c00cc20 osd=3 1 2024-04-14T15:16:24.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.332+0000 7f9c701c8700 10 client.48859.objecter close_session for osd.4 2024-04-14T15:16:24.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.332+0000 7f9c701c8700 1 -- 172.21.15.90:0/2862082745 >> v1:172.21.15.92:6800/3887549733 conn(0x7f9c4c017030 legacy=0x7f9c4c019420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:24.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.332+0000 7f9c701c8700 20 client.48859.objecter put_session s=0x7f9c4c0125e0 osd=4 1 2024-04-14T15:16:24.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.332+0000 7f9c701c8700 10 client.48859.objecter close_session for osd.5 2024-04-14T15:16:24.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.332+0000 7f9c701c8700 1 -- 172.21.15.90:0/2862082745 >> v1:172.21.15.92:6804/3244561440 conn(0x7f9c4c01bcf0 legacy=0x7f9c4c012fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:24.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.332+0000 7f9c701c8700 20 client.48859.objecter put_session s=0x7f9c4c00c700 osd=5 1 2024-04-14T15:16:24.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.332+0000 7f9c701c8700 10 client.48859.objecter close_session for osd.6 2024-04-14T15:16:24.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.332+0000 7f9c701c8700 1 -- 172.21.15.90:0/2862082745 >> v1:172.21.15.92:6808/636125963 conn(0x7f9c4c01f870 legacy=0x7f9c4c021c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:24.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.332+0000 7f9c701c8700 20 client.48859.objecter put_session s=0x7f9c4c0136c0 osd=6 1 2024-04-14T15:16:24.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.332+0000 7f9c701c8700 10 client.48859.objecter close_session for osd.7 2024-04-14T15:16:24.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.332+0000 7f9c701c8700 1 -- 172.21.15.90:0/2862082745 >> v1:172.21.15.92:6812/1773706716 conn(0x7f9c4c01b340 legacy=0x7f9c4c0234d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:24.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.332+0000 7f9c6e765700 1 -- 172.21.15.90:0/2862082745 reap_dead start 2024-04-14T15:16:24.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.332+0000 7f9c701c8700 20 client.48859.objecter put_session s=0x7f9c4c011f10 osd=7 1 2024-04-14T15:16:24.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.332+0000 7f9c701c8700 20 client.48859.objecter shutdown clearing up homeless session... 2024-04-14T15:16:24.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.332+0000 7f9c701c8700 10 client.48859.objecter successfully canceled tick 2024-04-14T15:16:24.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.332+0000 7f9c6e765700 1 -- 172.21.15.90:0/2862082745 reap_dead start 2024-04-14T15:16:24.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.332+0000 7f9c6e765700 1 -- 172.21.15.90:0/2862082745 reap_dead start 2024-04-14T15:16:24.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.332+0000 7f9c6e765700 1 -- 172.21.15.90:0/2862082745 reap_dead start 2024-04-14T15:16:24.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.332+0000 7f9c6e765700 1 -- 172.21.15.90:0/2862082745 reap_dead start 2024-04-14T15:16:24.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.333+0000 7f9c701c8700 1 -- 172.21.15.90:0/2862082745 >> v1:172.21.15.90:6800/420935427 conn(0x7f9c54071450 legacy=0x7f9c54073910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:24.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.333+0000 7f9c701c8700 1 -- 172.21.15.90:0/2862082745 >> v1:172.21.15.92:6789/0 conn(0x7f9c6810b7c0 legacy=0x7f9c68102950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:24.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.333+0000 7f9c701c8700 1 -- 172.21.15.90:0/2862082745 shutdown_connections 2024-04-14T15:16:24.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.333+0000 7f9c701c8700 1 -- 172.21.15.90:0/2862082745 >> 172.21.15.90:0/2862082745 conn(0x7f9c68076a70 msgr2=0x7f9c681117c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:16:24.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.333+0000 7f9c701c8700 1 -- 172.21.15.90:0/2862082745 shutdown_connections 2024-04-14T15:16:24.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.333+0000 7f9c701c8700 1 -- 172.21.15.90:0/2862082745 wait complete. 2024-04-14T15:16:24.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:24.333+0000 7f9c701c8700 1 librados: shutdown 2024-04-14T15:16:24.337 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:16:24.337 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:16:24.337 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:16:24.337 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:16:24.337 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:16:24.337 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:16:24.337 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:16:24.337 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:16:24.350 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:24.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:24.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:16:24.451 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:16:24.475 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:24.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:24.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:16:24.577 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:16:24.600 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:24.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:24.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:16:24.656 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-14T15:16:24.703 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:16:24.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:24.727 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:24.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:16:24.733 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-14T15:16:24.828 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:24.852 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:24.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:24.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:24.905 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:16:24.954 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:16:24.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:24.979 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:24.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:16:25.080 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:25.103 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:25.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:25.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:25.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:24 smithi090 ceph-mon[41670]: pgmap v419: 206 pgs: 206 active+clean; 4.6 MiB data, 579 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:16:25.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:24 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2862082745' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:16:25.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:24 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:16:25.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:24 smithi090 ceph-mon[41670]: from='client.46967 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi090-93363-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:16:25.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:24 smithi090 ceph-mon[41670]: osdmap e317: 8 total, 8 up, 8 in 2024-04-14T15:16:25.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:24 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:16:25.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:24 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:16:25.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:24 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/379496824' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi090-93363-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi090-93363-55"}]: dispatch 2024-04-14T15:16:25.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:24 smithi090 ceph-mon[41670]: from='client.46967 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi090-93363-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi090-93363-55"}]: dispatch 2024-04-14T15:16:25.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:24 smithi090 ceph-mon[30399]: pgmap v419: 206 pgs: 206 active+clean; 4.6 MiB data, 579 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:16:25.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:24 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2862082745' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:16:25.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:24 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:16:25.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:24 smithi090 ceph-mon[30399]: from='client.46967 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi090-93363-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:16:25.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:24 smithi090 ceph-mon[30399]: osdmap e317: 8 total, 8 up, 8 in 2024-04-14T15:16:25.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:24 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:16:25.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:24 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:16:25.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:24 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/379496824' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi090-93363-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi090-93363-55"}]: dispatch 2024-04-14T15:16:25.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:24 smithi090 ceph-mon[30399]: from='client.46967 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi090-93363-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi090-93363-55"}]: dispatch 2024-04-14T15:16:25.203 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:16:24 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:16:25.203 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:24 smithi092 ceph-mon[33966]: pgmap v419: 206 pgs: 206 active+clean; 4.6 MiB data, 579 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:16:25.204 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:24 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2862082745' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:16:25.204 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:24 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:16:25.204 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:24 smithi092 ceph-mon[33966]: from='client.46967 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi090-93363-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:16:25.204 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:24 smithi092 ceph-mon[33966]: osdmap e317: 8 total, 8 up, 8 in 2024-04-14T15:16:25.204 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:24 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:16:25.204 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:24 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:16:25.204 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:24 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/379496824' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi090-93363-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi090-93363-55"}]: dispatch 2024-04-14T15:16:25.204 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:24 smithi092 ceph-mon[33966]: from='client.46967 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi090-93363-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi090-93363-55"}]: dispatch 2024-04-14T15:16:25.205 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:16:25.228 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:25.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:25.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:16:25.330 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:16:25.336 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:16:25.353 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:25.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:25.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:16:25.454 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:16:25.477 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:25.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:25.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:16:25.579 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:16:25.603 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:25.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:25.604 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:16:25.704 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:16:25.728 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:25.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:25.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:16:25.830 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:16:25.854 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:25.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:25.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:16:25.957 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:25.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:25.983 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:25.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:26.084 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:26.108 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:26.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:26.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:26.210 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:16:26.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:26.235 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:26.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:16:26.336 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:26.361 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:26.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:26.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:26.463 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:26.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:26.488 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:26.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:26.588 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:26.592 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-04-14T15:16:26.592 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:16:26.614 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:26.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:26.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:26.716 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:16:26.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:26.740 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:26.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:16:26.841 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:26.865 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:26.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:26.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:26.967 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:16:26.991 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:26.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:26.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:16:27.075 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:26 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:16:27.075 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:26 smithi090 ceph-mon[30399]: pgmap v422: 238 pgs: 32 creating+peering, 2 peering, 204 active+clean; 8.4 MiB data, 591 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 977 KiB/s wr, 1 op/s 2024-04-14T15:16:27.076 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:26 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-41", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:16:27.076 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:26 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-41","var": "hit_set_count","val": "2"}]: dispatch 2024-04-14T15:16:27.076 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:26 smithi090 ceph-mon[30399]: osdmap e318: 8 total, 8 up, 8 in 2024-04-14T15:16:27.076 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:26 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-41","var": "hit_set_count","val": "2"}]: dispatch 2024-04-14T15:16:27.076 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:26 smithi090 ceph-mon[30399]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-04-14T15:16:27.076 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:26 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:16:27.076 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:26 smithi090 ceph-mon[41670]: pgmap v422: 238 pgs: 32 creating+peering, 2 peering, 204 active+clean; 8.4 MiB data, 591 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 977 KiB/s wr, 1 op/s 2024-04-14T15:16:27.076 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:26 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-41", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:16:27.076 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:26 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-41","var": "hit_set_count","val": "2"}]: dispatch 2024-04-14T15:16:27.076 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:26 smithi090 ceph-mon[41670]: osdmap e318: 8 total, 8 up, 8 in 2024-04-14T15:16:27.076 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:26 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-41","var": "hit_set_count","val": "2"}]: dispatch 2024-04-14T15:16:27.076 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:26 smithi090 ceph-mon[41670]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-04-14T15:16:27.093 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:16:27.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.093+0000 7f5e1d326700 1 -- 172.21.15.90:0/3430079644 >> v1:172.21.15.90:6789/0 conn(0x7f5e1810f300 legacy=0x7f5e181117c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:27.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.093+0000 7f5e1d326700 1 -- 172.21.15.90:0/3430079644 shutdown_connections 2024-04-14T15:16:27.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.093+0000 7f5e1d326700 1 -- 172.21.15.90:0/3430079644 >> 172.21.15.90:0/3430079644 conn(0x7f5e180fe390 msgr2=0x7f5e181007b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:16:27.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.093+0000 7f5e1d326700 1 -- 172.21.15.90:0/3430079644 shutdown_connections 2024-04-14T15:16:27.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.093+0000 7f5e1d326700 1 -- 172.21.15.90:0/3430079644 wait complete. 2024-04-14T15:16:27.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.093+0000 7f5e1d326700 1 librados: starting msgr at 2024-04-14T15:16:27.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.093+0000 7f5e1d326700 1 librados: starting objecter 2024-04-14T15:16:27.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.093+0000 7f5e1d326700 1 Processor -- start 2024-04-14T15:16:27.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.094+0000 7f5e1d326700 1 -- start start 2024-04-14T15:16:27.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.094+0000 7f5e1d326700 1 librados: setting wanted keys 2024-04-14T15:16:27.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.094+0000 7f5e1d326700 1 librados: calling monclient init 2024-04-14T15:16:27.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.094+0000 7f5e1d326700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5e181a7a30 con 0x7f5e1810b730 2024-04-14T15:16:27.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.094+0000 7f5e1d326700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5e181a8c50 con 0x7f5e18108880 2024-04-14T15:16:27.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.094+0000 7f5e1d326700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5e181a9e90 con 0x7f5e1810f300 2024-04-14T15:16:27.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.094+0000 7f5e167fc700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f5e1810b730 0x7f5e181a27b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:39108/0 (socket says 172.21.15.90:39108) 2024-04-14T15:16:27.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.094+0000 7f5e177fe700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f5e1810f300 0x7f5e181a60f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:33610/0 (socket says 172.21.15.90:33610) 2024-04-14T15:16:27.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.094+0000 7f5e167fc700 1 -- 172.21.15.90:0/1104670527 learned_addr learned my addr 172.21.15.90:0/1104670527 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:16:27.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.094+0000 7f5e0ffff700 10 client.?.objecter ms_handle_connect 0x7f5e18108880 2024-04-14T15:16:27.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.094+0000 7f5e0ffff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:16:27.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.094+0000 7f5e0ffff700 10 client.?.objecter ms_handle_connect 0x7f5e1810b730 2024-04-14T15:16:27.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.094+0000 7f5e0ffff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:16:27.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.094+0000 7f5e0ffff700 10 client.?.objecter ms_handle_connect 0x7f5e1810f300 2024-04-14T15:16:27.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.094+0000 7f5e0ffff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:16:27.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.094+0000 7f5e0ffff700 1 -- 172.21.15.90:0/1104670527 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 866461080 0 0) 0x7f5e181a7a30 con 0x7f5e1810b730 2024-04-14T15:16:27.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.094+0000 7f5e0ffff700 1 -- 172.21.15.90:0/1104670527 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5e040035b0 con 0x7f5e1810b730 2024-04-14T15:16:27.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.094+0000 7f5e0ffff700 1 -- 172.21.15.90:0/1104670527 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4039670749 0 0) 0x7f5e181a8c50 con 0x7f5e18108880 2024-04-14T15:16:27.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.094+0000 7f5e0ffff700 1 -- 172.21.15.90:0/1104670527 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5e181a7a30 con 0x7f5e18108880 2024-04-14T15:16:27.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.094+0000 7f5e0ffff700 1 -- 172.21.15.90:0/1104670527 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1974868203 0 0) 0x7f5e181a9e90 con 0x7f5e1810f300 2024-04-14T15:16:27.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.094+0000 7f5e0ffff700 1 -- 172.21.15.90:0/1104670527 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5e181a8c50 con 0x7f5e1810f300 2024-04-14T15:16:27.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.095+0000 7f5e0ffff700 1 -- 172.21.15.90:0/1104670527 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1072483808 0 0) 0x7f5e040035b0 con 0x7f5e1810b730 2024-04-14T15:16:27.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.095+0000 7f5e0ffff700 1 -- 172.21.15.90:0/1104670527 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5e181a9e90 con 0x7f5e1810b730 2024-04-14T15:16:27.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.095+0000 7f5e0ffff700 1 -- 172.21.15.90:0/1104670527 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 144917037 0 0) 0x7f5e181a8c50 con 0x7f5e1810f300 2024-04-14T15:16:27.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.095+0000 7f5e0ffff700 1 -- 172.21.15.90:0/1104670527 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5e040070d0 con 0x7f5e1810f300 2024-04-14T15:16:27.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.095+0000 7f5e0ffff700 1 -- 172.21.15.90:0/1104670527 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f5e08001370 con 0x7f5e1810f300 2024-04-14T15:16:27.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.095+0000 7f5e0ffff700 1 -- 172.21.15.90:0/1104670527 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f5e00001970 con 0x7f5e1810b730 2024-04-14T15:16:27.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.095+0000 7f5e0ffff700 1 -- 172.21.15.90:0/1104670527 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1270310128 0 0) 0x7f5e181a7a30 con 0x7f5e18108880 2024-04-14T15:16:27.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.095+0000 7f5e0ffff700 1 -- 172.21.15.90:0/1104670527 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5e181a8c50 con 0x7f5e18108880 2024-04-14T15:16:27.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.095+0000 7f5e0ffff700 1 -- 172.21.15.90:0/1104670527 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f5e10001b30 con 0x7f5e18108880 2024-04-14T15:16:27.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.095+0000 7f5e0ffff700 1 -- 172.21.15.90:0/1104670527 <== mon.2 v1:172.21.15.90:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1958652929 0 0) 0x7f5e040070d0 con 0x7f5e1810f300 2024-04-14T15:16:27.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.095+0000 7f5e0ffff700 1 -- 172.21.15.90:0/1104670527 >> v1:172.21.15.92:6789/0 conn(0x7f5e18108880 legacy=0x7f5e1810eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:27.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.095+0000 7f5e0ffff700 1 -- 172.21.15.90:0/1104670527 >> v1:172.21.15.90:6789/0 conn(0x7f5e1810b730 legacy=0x7f5e181a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:27.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.095+0000 7f5e0ffff700 1 -- 172.21.15.90:0/1104670527 --> v1:172.21.15.90:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e181ab0d0 con 0x7f5e1810f300 2024-04-14T15:16:27.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.095+0000 7f5e1d326700 1 -- 172.21.15.90:0/1104670527 --> v1:172.21.15.90:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5e181a7c60 con 0x7f5e1810f300 2024-04-14T15:16:27.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.095+0000 7f5e1d326700 10 client.46979.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:16:27.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.095+0000 7f5e1d326700 1 -- 172.21.15.90:0/1104670527 --> v1:172.21.15.90:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f5e181a8120 con 0x7f5e1810f300 2024-04-14T15:16:27.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.095+0000 7f5e1d326700 1 librados: init done 2024-04-14T15:16:27.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.095+0000 7f5e0ffff700 1 -- 172.21.15.90:0/1104670527 <== mon.2 v1:172.21.15.90:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f5e080015d0 con 0x7f5e1810f300 2024-04-14T15:16:27.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.095+0000 7f5e0ffff700 1 -- 172.21.15.90:0/1104670527 <== mon.2 v1:172.21.15.90:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f5e08004c50 con 0x7f5e1810f300 2024-04-14T15:16:27.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.096+0000 7f5e1d326700 1 -- 172.21.15.90:0/1104670527 --> v1:172.21.15.90:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5df8005150 con 0x7f5e1810f300 2024-04-14T15:16:27.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.097+0000 7f5e0ffff700 1 -- 172.21.15.90:0/1104670527 <== mon.2 v1:172.21.15.90:6790/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f5e08001af0 con 0x7f5e1810f300 2024-04-14T15:16:27.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.097+0000 7f5e0ffff700 1 -- 172.21.15.90:0/1104670527 <== mon.2 v1:172.21.15.90:6790/0 8 ==== osd_map(319..319 src has 51..319) ==== 8983+0+0 (unknown 3637832992 0 0) 0x7f5e0808aa30 con 0x7f5e1810f300 2024-04-14T15:16:27.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.097+0000 7f5e0ffff700 10 client.46979.objecter ms_dispatch 0x7f5e18001550 osd_map(319..319 src has 51..319) 2024-04-14T15:16:27.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.097+0000 7f5e0ffff700 3 client.46979.objecter handle_osd_map got epochs [319,319] > 0 2024-04-14T15:16:27.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.097+0000 7f5e0ffff700 3 client.46979.objecter handle_osd_map decoding full epoch 319 2024-04-14T15:16:27.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.097+0000 7f5e0ffff700 20 client.46979.objecter dump_active .. 0 homeless 2024-04-14T15:16:27.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.097+0000 7f5e0ffff700 10 client.46979.objecter ms_handle_connect 0x7f5e04071670 2024-04-14T15:16:27.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.102+0000 7f5e0ffff700 1 -- 172.21.15.90:0/1104670527 <== mon.2 v1:172.21.15.90:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f5e08057130 con 0x7f5e1810f300 2024-04-14T15:16:27.120 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-14T15:16:27.120 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-14T15:16:27.121 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-04-14T15:16:27.121 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph pg dump --format=json 2024-04-14T15:16:27.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:26 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:16:27.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:26 smithi092 ceph-mon[33966]: pgmap v422: 238 pgs: 32 creating+peering, 2 peering, 204 active+clean; 8.4 MiB data, 591 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 977 KiB/s wr, 1 op/s 2024-04-14T15:16:27.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:26 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-41", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:16:27.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:26 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-41","var": "hit_set_count","val": "2"}]: dispatch 2024-04-14T15:16:27.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:26 smithi092 ceph-mon[33966]: osdmap e318: 8 total, 8 up, 8 in 2024-04-14T15:16:27.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:26 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-41","var": "hit_set_count","val": "2"}]: dispatch 2024-04-14T15:16:27.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:26 smithi092 ceph-mon[33966]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-04-14T15:16:27.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:27.123 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:27.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:16:27.224 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:27.248 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:27.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:27.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:27.318 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-14T15:16:27.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.319+0000 7f5e1d326700 1 -- 172.21.15.90:0/1104670527 --> v1:172.21.15.90:6790/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7f5df8005440 con 0x7f5e1810f300 2024-04-14T15:16:27.327 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-14T15:16:27.350 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:16:27.380 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:16:27.394 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:27.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:27.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:16:27.496 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:16:27.520 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:27.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:27.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:16:27.568 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:16:27.621 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:27.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:27.646 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:27.646 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:27.747 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:16:27.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:27.771 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:27.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:16:27.873 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:16:27.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.882+0000 7f5e0ffff700 1 -- 172.21.15.90:0/1104670527 <== mon.2 v1:172.21.15.90:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v320) ==== 97+0+0 (unknown 364856896 0 0) 0x7f5e0805ade0 con 0x7f5e1810f300 2024-04-14T15:16:27.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.886+0000 7f5e1d326700 10 librados: watch_flush enter 2024-04-14T15:16:27.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.887+0000 7f5e1d326700 10 librados: watch_flush exit 2024-04-14T15:16:27.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.887+0000 7f5e1d326700 20 client.46979.objecter shutdown clearing up homeless session... 2024-04-14T15:16:27.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.887+0000 7f5e1d326700 10 client.46979.objecter successfully canceled tick 2024-04-14T15:16:27.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.888+0000 7f5e1d326700 1 -- 172.21.15.90:0/1104670527 >> v1:172.21.15.90:6800/420935427 conn(0x7f5e04071670 legacy=0x7f5e04073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:27.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.888+0000 7f5e1d326700 1 -- 172.21.15.90:0/1104670527 >> v1:172.21.15.90:6790/0 conn(0x7f5e1810f300 legacy=0x7f5e181a60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:27.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.889+0000 7f5e1d326700 1 -- 172.21.15.90:0/1104670527 shutdown_connections 2024-04-14T15:16:27.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.889+0000 7f5e1d326700 1 -- 172.21.15.90:0/1104670527 >> 172.21.15.90:0/1104670527 conn(0x7f5e180fe390 msgr2=0x7f5e18102880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:16:27.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.890+0000 7f5e1d326700 1 -- 172.21.15.90:0/1104670527 shutdown_connections 2024-04-14T15:16:27.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.890+0000 7f5e1d326700 1 -- 172.21.15.90:0/1104670527 wait complete. 2024-04-14T15:16:27.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:27.890+0000 7f5e1d326700 1 librados: shutdown 2024-04-14T15:16:27.894 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is set 2024-04-14T15:16:27.907 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:27 smithi090 ceph-mon[41670]: from='client.46967 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi090-93363-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi090-93363-55"}]': finished 2024-04-14T15:16:27.907 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:27 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-41","var": "hit_set_count","val": "2"}]': finished 2024-04-14T15:16:27.907 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:27 smithi090 ceph-mon[41670]: osdmap e319: 8 total, 8 up, 8 in 2024-04-14T15:16:27.907 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:27 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-41","var": "hit_set_period","val": "600"}]: dispatch 2024-04-14T15:16:27.907 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:27 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-41","var": "hit_set_period","val": "600"}]: dispatch 2024-04-14T15:16:27.907 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:27 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1104670527' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:16:27.907 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:27 smithi090 ceph-mon[41670]: from='client.46979 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:16:27.908 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:27 smithi090 ceph-mon[30399]: from='client.46967 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi090-93363-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi090-93363-55"}]': finished 2024-04-14T15:16:27.908 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:27 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-41","var": "hit_set_count","val": "2"}]': finished 2024-04-14T15:16:27.908 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:27 smithi090 ceph-mon[30399]: osdmap e319: 8 total, 8 up, 8 in 2024-04-14T15:16:27.908 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:27 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-41","var": "hit_set_period","val": "600"}]: dispatch 2024-04-14T15:16:27.908 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:27 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-41","var": "hit_set_period","val": "600"}]: dispatch 2024-04-14T15:16:27.908 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:27 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1104670527' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:16:27.908 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:27 smithi090 ceph-mon[30399]: from='client.46979 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:16:27.910 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:27.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:27.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:16:27.996 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:27 smithi092 ceph-mon[33966]: from='client.46967 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi090-93363-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi090-93363-55"}]': finished 2024-04-14T15:16:27.997 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:27 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-41","var": "hit_set_count","val": "2"}]': finished 2024-04-14T15:16:27.997 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:27 smithi092 ceph-mon[33966]: osdmap e319: 8 total, 8 up, 8 in 2024-04-14T15:16:27.997 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:27 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-41","var": "hit_set_period","val": "600"}]: dispatch 2024-04-14T15:16:27.997 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:27 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-41","var": "hit_set_period","val": "600"}]: dispatch 2024-04-14T15:16:27.997 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:27 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1104670527' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:16:27.997 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:27 smithi092 ceph-mon[33966]: from='client.46979 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:16:28.011 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:16:28.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:28.036 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:28.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:16:28.138 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:28.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:28.162 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:28.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:28.263 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:16:28.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:28.288 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:28.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:16:28.388 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:28.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:28.413 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:28.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:28.514 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:16:28.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:28.539 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:28.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:16:28.640 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:16:28.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:28.665 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:28.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:16:28.766 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:28.791 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:28.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:28.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:28.892 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:16:28.912 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:16:28.935 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:28.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:28.936 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:16:29.037 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:16:29.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:29.061 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:29.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:16:29.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:28 smithi092 ceph-mon[33966]: pgmap v425: 246 pgs: 8 unknown, 32 creating+peering, 2 peering, 204 active+clean; 8.4 MiB data, 591 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 977 KiB/s wr, 1 op/s 2024-04-14T15:16:29.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:28 smithi092 ceph-mon[33966]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:16:29.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:28 smithi092 ceph-mon[33966]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:16:29.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:28 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-41","var": "hit_set_period","val": "600"}]': finished 2024-04-14T15:16:29.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:28 smithi092 ceph-mon[33966]: from='client.46979 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-14T15:16:29.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:28 smithi092 ceph-mon[33966]: osdmap e320: 8 total, 8 up, 8 in 2024-04-14T15:16:29.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:28 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-14T15:16:29.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:28 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-14T15:16:29.162 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:16:29.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:29.186 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:29.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:16:29.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:28 smithi090 ceph-mon[41670]: pgmap v425: 246 pgs: 8 unknown, 32 creating+peering, 2 peering, 204 active+clean; 8.4 MiB data, 591 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 977 KiB/s wr, 1 op/s 2024-04-14T15:16:29.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:28 smithi090 ceph-mon[41670]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:16:29.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:28 smithi090 ceph-mon[41670]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:16:29.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:28 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-41","var": "hit_set_period","val": "600"}]': finished 2024-04-14T15:16:29.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:28 smithi090 ceph-mon[41670]: from='client.46979 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-14T15:16:29.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:28 smithi090 ceph-mon[41670]: osdmap e320: 8 total, 8 up, 8 in 2024-04-14T15:16:29.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:28 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-14T15:16:29.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:28 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-14T15:16:29.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:28 smithi090 ceph-mon[30399]: pgmap v425: 246 pgs: 8 unknown, 32 creating+peering, 2 peering, 204 active+clean; 8.4 MiB data, 591 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 977 KiB/s wr, 1 op/s 2024-04-14T15:16:29.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:28 smithi090 ceph-mon[30399]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:16:29.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:28 smithi090 ceph-mon[30399]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:16:29.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:28 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-41","var": "hit_set_period","val": "600"}]': finished 2024-04-14T15:16:29.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:28 smithi090 ceph-mon[30399]: from='client.46979 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-14T15:16:29.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:28 smithi090 ceph-mon[30399]: osdmap e320: 8 total, 8 up, 8 in 2024-04-14T15:16:29.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:28 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-14T15:16:29.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:28 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-14T15:16:29.287 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:16:29.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:29.312 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:29.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:16:29.413 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:16:29.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:29.438 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:29.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:16:29.505 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-04-14T15:16:29.505 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:16:29.540 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:16:29.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:29.563 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:29.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:16:29.664 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:16:29.689 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:29.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:29.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:16:29.791 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:16:29.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:29.816 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:29.816 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:16:29.917 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:16:29.941 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:29.941 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:29 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-41","var": "hit_set_type","val": "explicit_object"}]': finished 2024-04-14T15:16:29.941 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:29 smithi090 ceph-mon[41670]: osdmap e321: 8 total, 8 up, 8 in 2024-04-14T15:16:29.941 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:29 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/379496824' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi090-93363-55"}]: dispatch 2024-04-14T15:16:29.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:29 smithi090 ceph-mon[41670]: from='client.46967 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi090-93363-55"}]: dispatch 2024-04-14T15:16:29.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:29 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-41","var": "hit_set_type","val": "explicit_object"}]': finished 2024-04-14T15:16:29.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:29 smithi090 ceph-mon[30399]: osdmap e321: 8 total, 8 up, 8 in 2024-04-14T15:16:29.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:29 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/379496824' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi090-93363-55"}]: dispatch 2024-04-14T15:16:29.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:29 smithi090 ceph-mon[30399]: from='client.46967 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi090-93363-55"}]: dispatch 2024-04-14T15:16:29.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:29.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:16:30.044 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:16:30.067 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:30.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:30.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:16:30.169 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:16:30.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:30.194 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:30.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:16:30.291 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:29 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-41","var": "hit_set_type","val": "explicit_object"}]': finished 2024-04-14T15:16:30.291 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:29 smithi092 ceph-mon[33966]: osdmap e321: 8 total, 8 up, 8 in 2024-04-14T15:16:30.291 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:29 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/379496824' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi090-93363-55"}]: dispatch 2024-04-14T15:16:30.291 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:29 smithi092 ceph-mon[33966]: from='client.46967 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi090-93363-55"}]: dispatch 2024-04-14T15:16:30.295 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:16:30.318 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:30.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:30.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:16:30.321 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-14T15:16:30.420 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:30.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:30.448 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:30.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:30.549 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:30.575 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:30.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:30.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:30.677 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:16:30.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:30.702 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:30.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:16:30.803 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:30.830 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:30.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:30.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:30.932 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:30.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:30.961 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:30.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:31.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:30.089+0000 7f1ba6aa8700 1 -- 172.21.15.90:0/385992061 >> v1:172.21.15.90:6789/0 conn(0x7f1ba010a630 legacy=0x7f1ba010ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:31.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:30.089+0000 7f1ba6aa8700 1 -- 172.21.15.90:0/385992061 shutdown_connections 2024-04-14T15:16:31.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:30.089+0000 7f1ba6aa8700 1 -- 172.21.15.90:0/385992061 >> 172.21.15.90:0/385992061 conn(0x7f1ba00fd290 msgr2=0x7f1ba00ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:16:31.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:30.089+0000 7f1ba6aa8700 1 -- 172.21.15.90:0/385992061 shutdown_connections 2024-04-14T15:16:31.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:30.089+0000 7f1ba6aa8700 1 -- 172.21.15.90:0/385992061 wait complete. 2024-04-14T15:16:31.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:30.089+0000 7f1ba6aa8700 1 librados: starting msgr at 2024-04-14T15:16:31.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:30.089+0000 7f1ba6aa8700 1 librados: starting objecter 2024-04-14T15:16:31.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:30.089+0000 7f1ba6aa8700 1 Processor -- start 2024-04-14T15:16:31.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:30.089+0000 7f1ba6aa8700 1 -- start start 2024-04-14T15:16:31.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:30.089+0000 7f1ba6aa8700 1 librados: setting wanted keys 2024-04-14T15:16:31.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:30.089+0000 7f1ba6aa8700 1 librados: calling monclient init 2024-04-14T15:16:31.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:30.090+0000 7f1ba6aa8700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1ba01a79d0 con 0x7f1ba010a630 2024-04-14T15:16:31.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:30.090+0000 7f1ba6aa8700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1ba01a8bf0 con 0x7f1ba010e200 2024-04-14T15:16:31.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:30.090+0000 7f1ba6aa8700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1ba01a9e30 con 0x7f1ba0107780 2024-04-14T15:16:31.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:30.090+0000 7f1ba4844700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f1ba0107780 0x7f1ba010daa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:33624/0 (socket says 172.21.15.90:33624) 2024-04-14T15:16:31.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:30.090+0000 7f1ba4844700 1 -- 172.21.15.90:0/2277571117 learned_addr learned my addr 172.21.15.90:0/2277571117 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:16:31.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:30.090+0000 7f1b9dffb700 10 client.?.objecter ms_handle_connect 0x7f1ba0107780 2024-04-14T15:16:31.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:30.090+0000 7f1b9dffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:16:31.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:30.090+0000 7f1b9dffb700 10 client.?.objecter ms_handle_connect 0x7f1ba010a630 2024-04-14T15:16:31.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:30.090+0000 7f1b9dffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:16:31.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:30.090+0000 7f1b9dffb700 10 client.?.objecter ms_handle_connect 0x7f1ba010e200 2024-04-14T15:16:31.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:30.090+0000 7f1b9dffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:16:31.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:30.090+0000 7f1b9dffb700 1 -- 172.21.15.90:0/2277571117 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 767450435 0 0) 0x7f1ba01a79d0 con 0x7f1ba010a630 2024-04-14T15:16:31.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:30.090+0000 7f1b9dffb700 1 -- 172.21.15.90:0/2277571117 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1b900031f0 con 0x7f1ba010a630 2024-04-14T15:16:31.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:30.090+0000 7f1b9dffb700 1 -- 172.21.15.90:0/2277571117 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 362064839 0 0) 0x7f1ba01a9e30 con 0x7f1ba0107780 2024-04-14T15:16:31.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:30.090+0000 7f1b9dffb700 1 -- 172.21.15.90:0/2277571117 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1ba01a79d0 con 0x7f1ba0107780 2024-04-14T15:16:31.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:30.090+0000 7f1b9dffb700 1 -- 172.21.15.90:0/2277571117 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1777784568 0 0) 0x7f1ba01a8bf0 con 0x7f1ba010e200 2024-04-14T15:16:31.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:30.090+0000 7f1b9dffb700 1 -- 172.21.15.90:0/2277571117 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1ba01a9e30 con 0x7f1ba010e200 2024-04-14T15:16:31.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:30.090+0000 7f1b9dffb700 1 -- 172.21.15.90:0/2277571117 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1287700521 0 0) 0x7f1b900031f0 con 0x7f1ba010a630 2024-04-14T15:16:31.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:30.090+0000 7f1b9dffb700 1 -- 172.21.15.90:0/2277571117 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1ba01a8bf0 con 0x7f1ba010a630 2024-04-14T15:16:31.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:30.090+0000 7f1b9dffb700 1 -- 172.21.15.90:0/2277571117 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f1b8c004590 con 0x7f1ba010a630 2024-04-14T15:16:31.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:30.090+0000 7f1b9dffb700 1 -- 172.21.15.90:0/2277571117 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1444317879 0 0) 0x7f1ba01a79d0 con 0x7f1ba0107780 2024-04-14T15:16:31.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:30.090+0000 7f1b9dffb700 1 -- 172.21.15.90:0/2277571117 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1b900031f0 con 0x7f1ba0107780 2024-04-14T15:16:31.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:30.090+0000 7f1b9dffb700 1 -- 172.21.15.90:0/2277571117 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1340511853 0 0) 0x7f1ba01a9e30 con 0x7f1ba010e200 2024-04-14T15:16:31.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:30.090+0000 7f1b9dffb700 1 -- 172.21.15.90:0/2277571117 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1ba01a79d0 con 0x7f1ba010e200 2024-04-14T15:16:31.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:30.090+0000 7f1b9dffb700 1 -- 172.21.15.90:0/2277571117 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f1b940012d0 con 0x7f1ba0107780 2024-04-14T15:16:31.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:30.090+0000 7f1b9dffb700 1 -- 172.21.15.90:0/2277571117 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f1b98001bb0 con 0x7f1ba010e200 2024-04-14T15:16:31.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:30.090+0000 7f1b9dffb700 1 -- 172.21.15.90:0/2277571117 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3150553356 0 0) 0x7f1ba01a8bf0 con 0x7f1ba010a630 2024-04-14T15:16:31.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:30.090+0000 7f1b9dffb700 1 -- 172.21.15.90:0/2277571117 >> v1:172.21.15.90:6790/0 conn(0x7f1ba0107780 legacy=0x7f1ba010daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:31.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:30.090+0000 7f1b9dffb700 1 -- 172.21.15.90:0/2277571117 >> v1:172.21.15.92:6789/0 conn(0x7f1ba010e200 legacy=0x7f1ba01a6090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:31.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:30.090+0000 7f1b9dffb700 1 -- 172.21.15.90:0/2277571117 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ba01ab070 con 0x7f1ba010a630 2024-04-14T15:16:31.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:30.090+0000 7f1ba6aa8700 1 -- 172.21.15.90:0/2277571117 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1ba01a7c00 con 0x7f1ba010a630 2024-04-14T15:16:31.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:30.091+0000 7f1ba6aa8700 10 client.37032.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:16:31.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:30.091+0000 7f1ba6aa8700 1 -- 172.21.15.90:0/2277571117 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f1ba01a80c0 con 0x7f1ba010a630 2024-04-14T15:16:31.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:30.091+0000 7f1ba6aa8700 1 librados: init done 2024-04-14T15:16:31.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:30.091+0000 7f1b9dffb700 1 -- 172.21.15.90:0/2277571117 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f1b8c001130 con 0x7f1ba010a630 2024-04-14T15:16:31.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:30.091+0000 7f1b9dffb700 1 -- 172.21.15.90:0/2277571117 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f1b8c004c70 con 0x7f1ba010a630 2024-04-14T15:16:31.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:30.092+0000 7f1ba6aa8700 1 -- 172.21.15.90:0/2277571117 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b84005120 con 0x7f1ba010a630 2024-04-14T15:16:31.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:30.092+0000 7f1b9dffb700 1 -- 172.21.15.90:0/2277571117 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f1b8c001b80 con 0x7f1ba010a630 2024-04-14T15:16:31.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:30.093+0000 7f1b9dffb700 1 -- 172.21.15.90:0/2277571117 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(322..322 src has 51..322) ==== 8255+0+0 (unknown 318660659 0 0) 0x7f1b8c089940 con 0x7f1ba010a630 2024-04-14T15:16:31.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:30.093+0000 7f1b9dffb700 10 client.37032.objecter ms_dispatch 0x7f1ba0001550 osd_map(322..322 src has 51..322) 2024-04-14T15:16:31.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:30.093+0000 7f1b9dffb700 3 client.37032.objecter handle_osd_map got epochs [322,322] > 0 2024-04-14T15:16:31.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:30.093+0000 7f1b9dffb700 3 client.37032.objecter handle_osd_map decoding full epoch 322 2024-04-14T15:16:31.021 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:16:31.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:30.093+0000 7f1b9dffb700 20 client.37032.objecter dump_active .. 0 homeless 2024-04-14T15:16:31.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:30.093+0000 7f1b9dffb700 10 client.37032.objecter ms_handle_connect 0x7f1b90071360 2024-04-14T15:16:31.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:30.097+0000 7f1b9dffb700 1 -- 172.21.15.90:0/2277571117 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f1b8c056310 con 0x7f1ba010a630 2024-04-14T15:16:31.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:30.299+0000 7f1ba6aa8700 1 -- 172.21.15.90:0/2277571117 --> v1:172.21.15.90:6800/420935427 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1b84000b90 con 0x7f1b90071360 2024-04-14T15:16:31.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:30.318+0000 7f1b9dffb700 1 -- 172.21.15.90:0/2277571117 <== mgr.14610 v1:172.21.15.90:6800/420935427 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+614121 (unknown 2965378022 0 219479849) 0x7f1b84000b90 con 0x7f1b90071360 2024-04-14T15:16:31.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:31.020+0000 7f1ba6aa8700 10 librados: watch_flush enter 2024-04-14T15:16:31.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:31.020+0000 7f1ba6aa8700 10 librados: watch_flush exit 2024-04-14T15:16:31.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:31.020+0000 7f1ba6aa8700 20 client.37032.objecter shutdown clearing up homeless session... 2024-04-14T15:16:31.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:31.020+0000 7f1ba6aa8700 10 client.37032.objecter successfully canceled tick 2024-04-14T15:16:31.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:31.020+0000 7f1ba6aa8700 1 -- 172.21.15.90:0/2277571117 >> v1:172.21.15.90:6800/420935427 conn(0x7f1b90071360 legacy=0x7f1b90073820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:31.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:31.020+0000 7f1ba6aa8700 1 -- 172.21.15.90:0/2277571117 >> v1:172.21.15.90:6789/0 conn(0x7f1ba010a630 legacy=0x7f1ba01a26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:31.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:31.020+0000 7f1ba6aa8700 1 -- 172.21.15.90:0/2277571117 shutdown_connections 2024-04-14T15:16:31.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:31.020+0000 7f1ba6aa8700 1 -- 172.21.15.90:0/2277571117 >> 172.21.15.90:0/2277571117 conn(0x7f1ba00fd290 msgr2=0x7f1ba006b350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:16:31.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:31.021+0000 7f1ba6aa8700 1 -- 172.21.15.90:0/2277571117 shutdown_connections 2024-04-14T15:16:31.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:31.021+0000 7f1ba6aa8700 1 -- 172.21.15.90:0/2277571117 wait complete. 2024-04-14T15:16:31.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:31.021+0000 7f1ba6aa8700 1 librados: shutdown 2024-04-14T15:16:31.035 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-14T15:16:31.062 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:31.088 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:31.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:31.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:31.133 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:30 smithi090 ceph-mon[41670]: pgmap v428: 238 pgs: 5 creating+peering, 1 peering, 232 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 0 objects/s recovering 2024-04-14T15:16:31.134 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:30 smithi090 ceph-mon[41670]: from='client.46967 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi090-93363-55"}]': finished 2024-04-14T15:16:31.134 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:30 smithi090 ceph-mon[41670]: osdmap e322: 8 total, 8 up, 8 in 2024-04-14T15:16:31.134 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:30 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/379496824' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi090-93363-55"}]: dispatch 2024-04-14T15:16:31.134 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:30 smithi090 ceph-mon[41670]: from='client.46967 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi090-93363-55"}]: dispatch 2024-04-14T15:16:31.134 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:30 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:16:31.134 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:30 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:16:31.134 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:30 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-41"}]: dispatch 2024-04-14T15:16:31.134 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:30 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-41"}]: dispatch 2024-04-14T15:16:31.135 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:30 smithi090 ceph-mon[30399]: pgmap v428: 238 pgs: 5 creating+peering, 1 peering, 232 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 0 objects/s recovering 2024-04-14T15:16:31.135 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:30 smithi090 ceph-mon[30399]: from='client.46967 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi090-93363-55"}]': finished 2024-04-14T15:16:31.135 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:30 smithi090 ceph-mon[30399]: osdmap e322: 8 total, 8 up, 8 in 2024-04-14T15:16:31.135 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:30 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/379496824' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi090-93363-55"}]: dispatch 2024-04-14T15:16:31.135 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:30 smithi090 ceph-mon[30399]: from='client.46967 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi090-93363-55"}]: dispatch 2024-04-14T15:16:31.135 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:30 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:16:31.135 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:30 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:16:31.135 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:30 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-41"}]: dispatch 2024-04-14T15:16:31.135 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:30 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-41"}]: dispatch 2024-04-14T15:16:31.190 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:16:31.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:31.215 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:31.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:16:31.316 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:16:31.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:31.342 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:31.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:16:31.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:30 smithi092 ceph-mon[33966]: pgmap v428: 238 pgs: 5 creating+peering, 1 peering, 232 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 0 objects/s recovering 2024-04-14T15:16:31.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:30 smithi092 ceph-mon[33966]: from='client.46967 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi090-93363-55"}]': finished 2024-04-14T15:16:31.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:30 smithi092 ceph-mon[33966]: osdmap e322: 8 total, 8 up, 8 in 2024-04-14T15:16:31.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:30 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/379496824' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi090-93363-55"}]: dispatch 2024-04-14T15:16:31.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:30 smithi092 ceph-mon[33966]: from='client.46967 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi090-93363-55"}]: dispatch 2024-04-14T15:16:31.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:30 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:16:31.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:30 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:16:31.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:30 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-41"}]: dispatch 2024-04-14T15:16:31.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:30 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-41"}]: dispatch 2024-04-14T15:16:31.442 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd unset noscrub 2024-04-14T15:16:31.443 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:16:31.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:31.467 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:31.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:16:31.569 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:16:31.596 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:31.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:31.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:16:31.698 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:31.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:31.727 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:31.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:31.829 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:16:31.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:31.854 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:31.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:16:31.955 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:31.983 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:31.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:31.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:32.085 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:32.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:32.113 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:32.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:32.138 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":429,"stamp":"2024-04-14T15:16:29.899599+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8791921,"num_objects":323,"num_object_clones":2,"num_object_copies":969,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":323,"num_whiteouts":0,"num_read":1222,"num_read_kb":880,"num_write":994,"num_write_kb":12694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3796,"ondisk_log_size":3796,"up":714,"acting":714,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":727,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":614160,"kb_used_data":70300,"kb_used_omap":12,"kb_used_meta":543603,"kb_avail":749347056,"statfs":{"total":767960285184,"available":767331385344,"internally_reserved":0,"allocated":71987200,"data_stored":51430528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":556650034},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.031575"},"pg_stats":[{"pgid":"274.1f","version":"0'0","reported_seq":22,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.930327+0000","last_change":"2024-04-14T15:16:24.907304+0000","last_active":"2024-04-14T15:16:28.930327+0000","last_peered":"2024-04-14T15:16:28.930327+0000","last_clean":"2024-04-14T15:16:28.930327+0000","last_became_active":"2024-04-14T15:16:24.906942+0000","last_became_peered":"2024-04-14T15:16:24.906942+0000","last_unstale":"2024-04-14T15:16:28.930327+0000","last_undegraded":"2024-04-14T15:16:28.930327+0000","last_fullsized":"2024-04-14T15:16:28.930327+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:23.882853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:23.882853+0000","last_clean_scrub_stamp":"2024-04-14T15:16:23.882853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:44.716268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"274.1e","version":"0'0","reported_seq":22,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.881761+0000","last_change":"2024-04-14T15:16:24.905223+0000","last_active":"2024-04-14T15:16:28.881761+0000","last_peered":"2024-04-14T15:16:28.881761+0000","last_clean":"2024-04-14T15:16:28.881761+0000","last_became_active":"2024-04-14T15:16:24.904920+0000","last_became_peered":"2024-04-14T15:16:24.904920+0000","last_unstale":"2024-04-14T15:16:28.881761+0000","last_undegraded":"2024-04-14T15:16:28.881761+0000","last_fullsized":"2024-04-14T15:16:28.881761+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:23.882853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:23.882853+0000","last_clean_scrub_stamp":"2024-04-14T15:16:23.882853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:52.577853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"274.1d","version":"0'0","reported_seq":22,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.929590+0000","last_change":"2024-04-14T15:16:24.990861+0000","last_active":"2024-04-14T15:16:28.929590+0000","last_peered":"2024-04-14T15:16:28.929590+0000","last_clean":"2024-04-14T15:16:28.929590+0000","last_became_active":"2024-04-14T15:16:24.990562+0000","last_became_peered":"2024-04-14T15:16:24.990562+0000","last_unstale":"2024-04-14T15:16:28.929590+0000","last_undegraded":"2024-04-14T15:16:28.929590+0000","last_fullsized":"2024-04-14T15:16:28.929590+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:23.882853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:23.882853+0000","last_clean_scrub_stamp":"2024-04-14T15:16:23.882853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:26.609234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"274.1c","version":"0'0","reported_seq":22,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.881342+0000","last_change":"2024-04-14T15:16:24.907900+0000","last_active":"2024-04-14T15:16:28.881342+0000","last_peered":"2024-04-14T15:16:28.881342+0000","last_clean":"2024-04-14T15:16:28.881342+0000","last_became_active":"2024-04-14T15:16:24.907591+0000","last_became_peered":"2024-04-14T15:16:24.907591+0000","last_unstale":"2024-04-14T15:16:28.881342+0000","last_undegraded":"2024-04-14T15:16:28.881342+0000","last_fullsized":"2024-04-14T15:16:28.881342+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:23.882853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:23.882853+0000","last_clean_scrub_stamp":"2024-04-14T15:16:23.882853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:34.118468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"274.1b","version":"0'0","reported_seq":22,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.926652+0000","last_change":"2024-04-14T15:16:24.990176+0000","last_active":"2024-04-14T15:16:28.926652+0000","last_peered":"2024-04-14T15:16:28.926652+0000","last_clean":"2024-04-14T15:16:28.926652+0000","last_became_active":"2024-04-14T15:16:24.989833+0000","last_became_peered":"2024-04-14T15:16:24.989833+0000","last_unstale":"2024-04-14T15:16:28.926652+0000","last_undegraded":"2024-04-14T15:16:28.926652+0000","last_fullsized":"2024-04-14T15:16:28.926652+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:23.882853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:23.882853+0000","last_clean_scrub_stamp":"2024-04-14T15:16:23.882853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:39.170274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"274.1a","version":"0'0","reported_seq":22,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.927992+0000","last_change":"2024-04-14T15:16:24.910173+0000","last_active":"2024-04-14T15:16:28.927992+0000","last_peered":"2024-04-14T15:16:28.927992+0000","last_clean":"2024-04-14T15:16:28.927992+0000","last_became_active":"2024-04-14T15:16:24.909859+0000","last_became_peered":"2024-04-14T15:16:24.909859+0000","last_unstale":"2024-04-14T15:16:28.927992+0000","last_undegraded":"2024-04-14T15:16:28.927992+0000","last_fullsized":"2024-04-14T15:16:28.927992+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:23.882853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:23.882853+0000","last_clean_scrub_stamp":"2024-04-14T15:16:23.882853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:40.673087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"274.19","version":"0'0","reported_seq":22,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.886811+0000","last_change":"2024-04-14T15:16:24.900628+0000","last_active":"2024-04-14T15:16:28.886811+0000","last_peered":"2024-04-14T15:16:28.886811+0000","last_clean":"2024-04-14T15:16:28.886811+0000","last_became_active":"2024-04-14T15:16:24.900284+0000","last_became_peered":"2024-04-14T15:16:24.900284+0000","last_unstale":"2024-04-14T15:16:28.886811+0000","last_undegraded":"2024-04-14T15:16:28.886811+0000","last_fullsized":"2024-04-14T15:16:28.886811+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:23.882853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:23.882853+0000","last_clean_scrub_stamp":"2024-04-14T15:16:23.882853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:27.075042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"274.18","version":"0'0","reported_seq":22,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.926184+0000","last_change":"2024-04-14T15:16:24.990941+0000","last_active":"2024-04-14T15:16:28.926184+0000","last_peered":"2024-04-14T15:16:28.926184+0000","last_clean":"2024-04-14T15:16:28.926184+0000","last_became_active":"2024-04-14T15:16:24.990675+0000","last_became_peered":"2024-04-14T15:16:24.990675+0000","last_unstale":"2024-04-14T15:16:28.926184+0000","last_undegraded":"2024-04-14T15:16:28.926184+0000","last_fullsized":"2024-04-14T15:16:28.926184+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:23.882853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:23.882853+0000","last_clean_scrub_stamp":"2024-04-14T15:16:23.882853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:35.934091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"274.17","version":"0'0","reported_seq":22,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.930473+0000","last_change":"2024-04-14T15:16:24.906300+0000","last_active":"2024-04-14T15:16:28.930473+0000","last_peered":"2024-04-14T15:16:28.930473+0000","last_clean":"2024-04-14T15:16:28.930473+0000","last_became_active":"2024-04-14T15:16:24.906005+0000","last_became_peered":"2024-04-14T15:16:24.906005+0000","last_unstale":"2024-04-14T15:16:28.930473+0000","last_undegraded":"2024-04-14T15:16:28.930473+0000","last_fullsized":"2024-04-14T15:16:28.930473+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:23.882853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:23.882853+0000","last_clean_scrub_stamp":"2024-04-14T15:16:23.882853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:24.089044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"274.16","version":"0'0","reported_seq":22,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.891321+0000","last_change":"2024-04-14T15:16:24.990315+0000","last_active":"2024-04-14T15:16:28.891321+0000","last_peered":"2024-04-14T15:16:28.891321+0000","last_clean":"2024-04-14T15:16:28.891321+0000","last_became_active":"2024-04-14T15:16:24.990015+0000","last_became_peered":"2024-04-14T15:16:24.990015+0000","last_unstale":"2024-04-14T15:16:28.891321+0000","last_undegraded":"2024-04-14T15:16:28.891321+0000","last_fullsized":"2024-04-14T15:16:28.891321+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:23.882853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:23.882853+0000","last_clean_scrub_stamp":"2024-04-14T15:16:23.882853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:33.940917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"274.15","version":"0'0","reported_seq":22,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.929922+0000","last_change":"2024-04-14T15:16:24.901790+0000","last_active":"2024-04-14T15:16:28.929922+0000","last_peered":"2024-04-14T15:16:28.929922+0000","last_clean":"2024-04-14T15:16:28.929922+0000","last_became_active":"2024-04-14T15:16:24.900665+0000","last_became_peered":"2024-04-14T15:16:24.900665+0000","last_unstale":"2024-04-14T15:16:28.929922+0000","last_undegraded":"2024-04-14T15:16:28.929922+0000","last_fullsized":"2024-04-14T15:16:28.929922+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:23.882853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:23.882853+0000","last_clean_scrub_stamp":"2024-04-14T15:16:23.882853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:47.016803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"274.14","version":"0'0","reported_seq":22,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.926506+0000","last_change":"2024-04-14T15:16:24.905529+0000","last_active":"2024-04-14T15:16:28.926506+0000","last_peered":"2024-04-14T15:16:28.926506+0000","last_clean":"2024-04-14T15:16:28.926506+0000","last_became_active":"2024-04-14T15:16:24.904887+0000","last_became_peered":"2024-04-14T15:16:24.904887+0000","last_unstale":"2024-04-14T15:16:28.926506+0000","last_undegraded":"2024-04-14T15:16:28.926506+0000","last_fullsized":"2024-04-14T15:16:28.926506+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:23.882853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:23.882853+0000","last_clean_scrub_stamp":"2024-04-14T15:16:23.882853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:27.976184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"274.13","version":"0'0","reported_seq":22,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.886762+0000","last_change":"2024-04-14T15:16:24.908282+0000","last_active":"2024-04-14T15:16:28.886762+0000","last_peered":"2024-04-14T15:16:28.886762+0000","last_clean":"2024-04-14T15:16:28.886762+0000","last_became_active":"2024-04-14T15:16:24.908007+0000","last_became_peered":"2024-04-14T15:16:24.908007+0000","last_unstale":"2024-04-14T15:16:28.886762+0000","last_undegraded":"2024-04-14T15:16:28.886762+0000","last_fullsized":"2024-04-14T15:16:28.886762+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:23.882853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:23.882853+0000","last_clean_scrub_stamp":"2024-04-14T15:16:23.882853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:31.844679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"274.12","version":"0'0","reported_seq":22,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.927221+0000","last_change":"2024-04-14T15:16:24.910018+0000","last_active":"2024-04-14T15:16:28.927221+0000","last_peered":"2024-04-14T15:16:28.927221+0000","last_clean":"2024-04-14T15:16:28.927221+0000","last_became_active":"2024-04-14T15:16:24.909730+0000","last_became_peered":"2024-04-14T15:16:24.909730+0000","last_unstale":"2024-04-14T15:16:28.927221+0000","last_undegraded":"2024-04-14T15:16:28.927221+0000","last_fullsized":"2024-04-14T15:16:28.927221+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:23.882853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:23.882853+0000","last_clean_scrub_stamp":"2024-04-14T15:16:23.882853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:30.724137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"274.11","version":"0'0","reported_seq":22,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.886961+0000","last_change":"2024-04-14T15:16:24.991549+0000","last_active":"2024-04-14T15:16:28.886961+0000","last_peered":"2024-04-14T15:16:28.886961+0000","last_clean":"2024-04-14T15:16:28.886961+0000","last_became_active":"2024-04-14T15:16:24.991168+0000","last_became_peered":"2024-04-14T15:16:24.991168+0000","last_unstale":"2024-04-14T15:16:28.886961+0000","last_undegraded":"2024-04-14T15:16:28.886961+0000","last_fullsized":"2024-04-14T15:16:28.886961+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:23.882853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:23.882853+0000","last_clean_scrub_stamp":"2024-04-14T15:16:23.882853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:32.216555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"274.10","version":"0'0","reported_seq":22,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.888180+0000","last_change":"2024-04-14T15:16:24.899724+0000","last_active":"2024-04-14T15:16:28.888180+0000","last_peered":"2024-04-14T15:16:28.888180+0000","last_clean":"2024-04-14T15:16:28.888180+0000","last_became_active":"2024-04-14T15:16:24.899453+0000","last_became_peered":"2024-04-14T15:16:24.899453+0000","last_unstale":"2024-04-14T15:16:28.888180+0000","last_undegraded":"2024-04-14T15:16:28.888180+0000","last_fullsized":"2024-04-14T15:16:28.888180+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:23.882853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:23.882853+0000","last_clean_scrub_stamp":"2024-04-14T15:16:23.882853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:35.593209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"274.f","version":"0'0","reported_seq":22,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.926827+0000","last_change":"2024-04-14T15:16:24.991481+0000","last_active":"2024-04-14T15:16:28.926827+0000","last_peered":"2024-04-14T15:16:28.926827+0000","last_clean":"2024-04-14T15:16:28.926827+0000","last_became_active":"2024-04-14T15:16:24.991237+0000","last_became_peered":"2024-04-14T15:16:24.991237+0000","last_unstale":"2024-04-14T15:16:28.926827+0000","last_undegraded":"2024-04-14T15:16:28.926827+0000","last_fullsized":"2024-04-14T15:16:28.926827+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:23.882853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:23.882853+0000","last_clean_scrub_stamp":"2024-04-14T15:16:23.882853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:25.724466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"274.e","version":"0'0","reported_seq":22,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.929687+0000","last_change":"2024-04-14T15:16:24.899551+0000","last_active":"2024-04-14T15:16:28.929687+0000","last_peered":"2024-04-14T15:16:28.929687+0000","last_clean":"2024-04-14T15:16:28.929687+0000","last_became_active":"2024-04-14T15:16:24.899258+0000","last_became_peered":"2024-04-14T15:16:24.899258+0000","last_unstale":"2024-04-14T15:16:28.929687+0000","last_undegraded":"2024-04-14T15:16:28.929687+0000","last_fullsized":"2024-04-14T15:16:28.929687+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:23.882853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:23.882853+0000","last_clean_scrub_stamp":"2024-04-14T15:16:23.882853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:27.365045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"274.d","version":"0'0","reported_seq":22,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.890681+0000","last_change":"2024-04-14T15:16:24.990232+0000","last_active":"2024-04-14T15:16:28.890681+0000","last_peered":"2024-04-14T15:16:28.890681+0000","last_clean":"2024-04-14T15:16:28.890681+0000","last_became_active":"2024-04-14T15:16:24.989829+0000","last_became_peered":"2024-04-14T15:16:24.989829+0000","last_unstale":"2024-04-14T15:16:28.890681+0000","last_undegraded":"2024-04-14T15:16:28.890681+0000","last_fullsized":"2024-04-14T15:16:28.890681+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:23.882853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:23.882853+0000","last_clean_scrub_stamp":"2024-04-14T15:16:23.882853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:29.324544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"274.c","version":"0'0","reported_seq":22,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.927542+0000","last_change":"2024-04-14T15:16:24.990935+0000","last_active":"2024-04-14T15:16:28.927542+0000","last_peered":"2024-04-14T15:16:28.927542+0000","last_clean":"2024-04-14T15:16:28.927542+0000","last_became_active":"2024-04-14T15:16:24.990587+0000","last_became_peered":"2024-04-14T15:16:24.990587+0000","last_unstale":"2024-04-14T15:16:28.927542+0000","last_undegraded":"2024-04-14T15:16:28.927542+0000","last_fullsized":"2024-04-14T15:16:28.927542+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:23.882853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:23.882853+0000","last_clean_scrub_stamp":"2024-04-14T15:16:23.882853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:44.969729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"274.b","version":"0'0","reported_seq":22,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.885975+0000","last_change":"2024-04-14T15:16:24.907640+0000","last_active":"2024-04-14T15:16:28.885975+0000","last_peered":"2024-04-14T15:16:28.885975+0000","last_clean":"2024-04-14T15:16:28.885975+0000","last_became_active":"2024-04-14T15:16:24.907268+0000","last_became_peered":"2024-04-14T15:16:24.907268+0000","last_unstale":"2024-04-14T15:16:28.885975+0000","last_undegraded":"2024-04-14T15:16:28.885975+0000","last_fullsized":"2024-04-14T15:16:28.885975+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:23.882853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:23.882853+0000","last_clean_scrub_stamp":"2024-04-14T15:16:23.882853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:52.094427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"274.a","version":"0'0","reported_seq":22,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.927790+0000","last_change":"2024-04-14T15:16:24.990495+0000","last_active":"2024-04-14T15:16:28.927790+0000","last_peered":"2024-04-14T15:16:28.927790+0000","last_clean":"2024-04-14T15:16:28.927790+0000","last_became_active":"2024-04-14T15:16:24.990080+0000","last_became_peered":"2024-04-14T15:16:24.990080+0000","last_unstale":"2024-04-14T15:16:28.927790+0000","last_undegraded":"2024-04-14T15:16:28.927790+0000","last_fullsized":"2024-04-14T15:16:28.927790+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:23.882853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:23.882853+0000","last_clean_scrub_stamp":"2024-04-14T15:16:23.882853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:46.380504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"274.9","version":"0'0","reported_seq":22,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.925433+0000","last_change":"2024-04-14T15:16:24.902317+0000","last_active":"2024-04-14T15:16:28.925433+0000","last_peered":"2024-04-14T15:16:28.925433+0000","last_clean":"2024-04-14T15:16:28.925433+0000","last_became_active":"2024-04-14T15:16:24.902032+0000","last_became_peered":"2024-04-14T15:16:24.902032+0000","last_unstale":"2024-04-14T15:16:28.925433+0000","last_undegraded":"2024-04-14T15:16:28.925433+0000","last_fullsized":"2024-04-14T15:16:28.925433+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:23.882853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:23.882853+0000","last_clean_scrub_stamp":"2024-04-14T15:16:23.882853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:38.974553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"274.8","version":"0'0","reported_seq":22,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.885505+0000","last_change":"2024-04-14T15:16:24.991799+0000","last_active":"2024-04-14T15:16:28.885505+0000","last_peered":"2024-04-14T15:16:28.885505+0000","last_clean":"2024-04-14T15:16:28.885505+0000","last_became_active":"2024-04-14T15:16:24.991305+0000","last_became_peered":"2024-04-14T15:16:24.991305+0000","last_unstale":"2024-04-14T15:16:28.885505+0000","last_undegraded":"2024-04-14T15:16:28.885505+0000","last_fullsized":"2024-04-14T15:16:28.885505+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:23.882853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:23.882853+0000","last_clean_scrub_stamp":"2024-04-14T15:16:23.882853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:49.590025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"274.7","version":"0'0","reported_seq":22,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.891180+0000","last_change":"2024-04-14T15:16:24.993280+0000","last_active":"2024-04-14T15:16:28.891180+0000","last_peered":"2024-04-14T15:16:28.891180+0000","last_clean":"2024-04-14T15:16:28.891180+0000","last_became_active":"2024-04-14T15:16:24.992873+0000","last_became_peered":"2024-04-14T15:16:24.992873+0000","last_unstale":"2024-04-14T15:16:28.891180+0000","last_undegraded":"2024-04-14T15:16:28.891180+0000","last_fullsized":"2024-04-14T15:16:28.891180+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:23.882853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:23.882853+0000","last_clean_scrub_stamp":"2024-04-14T15:16:23.882853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:48.794614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"274.6","version":"0'0","reported_seq":22,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.891468+0000","last_change":"2024-04-14T15:16:24.907363+0000","last_active":"2024-04-14T15:16:28.891468+0000","last_peered":"2024-04-14T15:16:28.891468+0000","last_clean":"2024-04-14T15:16:28.891468+0000","last_became_active":"2024-04-14T15:16:24.907098+0000","last_became_peered":"2024-04-14T15:16:24.907098+0000","last_unstale":"2024-04-14T15:16:28.891468+0000","last_undegraded":"2024-04-14T15:16:28.891468+0000","last_fullsized":"2024-04-14T15:16:28.891468+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:23.882853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:23.882853+0000","last_clean_scrub_stamp":"2024-04-14T15:16:23.882853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:36.734368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"274.5","version":"0'0","reported_seq":22,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.928460+0000","last_change":"2024-04-14T15:16:24.907076+0000","last_active":"2024-04-14T15:16:28.928460+0000","last_peered":"2024-04-14T15:16:28.928460+0000","last_clean":"2024-04-14T15:16:28.928460+0000","last_became_active":"2024-04-14T15:16:24.906855+0000","last_became_peered":"2024-04-14T15:16:24.906855+0000","last_unstale":"2024-04-14T15:16:28.928460+0000","last_undegraded":"2024-04-14T15:16:28.928460+0000","last_fullsized":"2024-04-14T15:16:28.928460+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:23.882853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:23.882853+0000","last_clean_scrub_stamp":"2024-04-14T15:16:23.882853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:31.476220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"274.4","version":"0'0","reported_seq":22,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.927696+0000","last_change":"2024-04-14T15:16:24.910498+0000","last_active":"2024-04-14T15:16:28.927696+0000","last_peered":"2024-04-14T15:16:28.927696+0000","last_clean":"2024-04-14T15:16:28.927696+0000","last_became_active":"2024-04-14T15:16:24.909808+0000","last_became_peered":"2024-04-14T15:16:24.909808+0000","last_unstale":"2024-04-14T15:16:28.927696+0000","last_undegraded":"2024-04-14T15:16:28.927696+0000","last_fullsized":"2024-04-14T15:16:28.927696+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:23.882853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:23.882853+0000","last_clean_scrub_stamp":"2024-04-14T15:16:23.882853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:27.767272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"274.3","version":"0'0","reported_seq":22,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.928025+0000","last_change":"2024-04-14T15:16:24.990213+0000","last_active":"2024-04-14T15:16:28.928025+0000","last_peered":"2024-04-14T15:16:28.928025+0000","last_clean":"2024-04-14T15:16:28.928025+0000","last_became_active":"2024-04-14T15:16:24.989852+0000","last_became_peered":"2024-04-14T15:16:24.989852+0000","last_unstale":"2024-04-14T15:16:28.928025+0000","last_undegraded":"2024-04-14T15:16:28.928025+0000","last_fullsized":"2024-04-14T15:16:28.928025+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:23.882853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:23.882853+0000","last_clean_scrub_stamp":"2024-04-14T15:16:23.882853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:43.381005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"274.2","version":"0'0","reported_seq":22,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.887474+0000","last_change":"2024-04-14T15:16:24.908709+0000","last_active":"2024-04-14T15:16:28.887474+0000","last_peered":"2024-04-14T15:16:28.887474+0000","last_clean":"2024-04-14T15:16:28.887474+0000","last_became_active":"2024-04-14T15:16:24.907814+0000","last_became_peered":"2024-04-14T15:16:24.907814+0000","last_unstale":"2024-04-14T15:16:28.887474+0000","last_undegraded":"2024-04-14T15:16:28.887474+0000","last_fullsized":"2024-04-14T15:16:28.887474+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:23.882853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:23.882853+0000","last_clean_scrub_stamp":"2024-04-14T15:16:23.882853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:24.955930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"274.1","version":"0'0","reported_seq":22,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.887277+0000","last_change":"2024-04-14T15:16:24.897150+0000","last_active":"2024-04-14T15:16:28.887277+0000","last_peered":"2024-04-14T15:16:28.887277+0000","last_clean":"2024-04-14T15:16:28.887277+0000","last_became_active":"2024-04-14T15:16:24.896867+0000","last_became_peered":"2024-04-14T15:16:24.896867+0000","last_unstale":"2024-04-14T15:16:28.887277+0000","last_undegraded":"2024-04-14T15:16:28.887277+0000","last_fullsized":"2024-04-14T15:16:28.887277+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:23.882853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:23.882853+0000","last_clean_scrub_stamp":"2024-04-14T15:16:23.882853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:35.483803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"274.0","version":"0'0","reported_seq":22,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.885682+0000","last_change":"2024-04-14T15:16:24.991582+0000","last_active":"2024-04-14T15:16:28.885682+0000","last_peered":"2024-04-14T15:16:28.885682+0000","last_clean":"2024-04-14T15:16:28.885682+0000","last_became_active":"2024-04-14T15:16:24.990869+0000","last_became_peered":"2024-04-14T15:16:24.990869+0000","last_unstale":"2024-04-14T15:16:28.885682+0000","last_undegraded":"2024-04-14T15:16:28.885682+0000","last_fullsized":"2024-04-14T15:16:28.885682+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:23.882853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:23.882853+0000","last_clean_scrub_stamp":"2024-04-14T15:16:23.882853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:49.907600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"110.11","version":"253'39","reported_seq":591,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.925626+0000","last_change":"2024-04-14T15:16:03.875072+0000","last_active":"2024-04-14T15:16:28.925626+0000","last_peered":"2024-04-14T15:16:28.925626+0000","last_clean":"2024-04-14T15:16:28.925626+0000","last_became_active":"2024-04-14T15:13:09.936619+0000","last_became_peered":"2024-04-14T15:13:09.936619+0000","last_unstale":"2024-04-14T15:16:28.925626+0000","last_undegraded":"2024-04-14T15:16:28.925626+0000","last_fullsized":"2024-04-14T15:16:28.925626+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"253'39","last_scrub_stamp":"2024-04-14T15:15:38.146569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:15:38.146569+0000","objects_scrubbed":2,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:39.694596+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0016290829999999999,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"110.10","version":"253'158","reported_seq":676,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.880848+0000","last_change":"2024-04-14T15:16:03.881102+0000","last_active":"2024-04-14T15:16:28.880848+0000","last_peered":"2024-04-14T15:16:28.880848+0000","last_clean":"2024-04-14T15:16:28.880848+0000","last_became_active":"2024-04-14T15:13:09.928915+0000","last_became_peered":"2024-04-14T15:13:09.928915+0000","last_unstale":"2024-04-14T15:16:28.880848+0000","last_undegraded":"2024-04-14T15:16:28.880848+0000","last_fullsized":"2024-04-14T15:16:28.880848+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"253'158","last_scrub_stamp":"2024-04-14T15:15:29.853735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:15:29.853735+0000","objects_scrubbed":10,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:33.146852+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0018074390000000001,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":47,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"110.13","version":"234'39","reported_seq":594,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.926896+0000","last_change":"2024-04-14T15:16:03.896377+0000","last_active":"2024-04-14T15:16:28.926896+0000","last_peered":"2024-04-14T15:16:28.926896+0000","last_clean":"2024-04-14T15:16:28.926896+0000","last_became_active":"2024-04-14T15:13:09.918152+0000","last_became_peered":"2024-04-14T15:13:09.918152+0000","last_unstale":"2024-04-14T15:16:28.926896+0000","last_undegraded":"2024-04-14T15:16:28.926896+0000","last_fullsized":"2024-04-14T15:16:28.926896+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'37","last_scrub_stamp":"2024-04-14T15:15:01.282258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:15:01.282258+0000","objects_scrubbed":4,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0047971690000000004,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"110.12","version":"253'66","reported_seq":618,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.926617+0000","last_change":"2024-04-14T15:16:03.894781+0000","last_active":"2024-04-14T15:16:28.926617+0000","last_peered":"2024-04-14T15:16:28.926617+0000","last_clean":"2024-04-14T15:16:28.926617+0000","last_became_active":"2024-04-14T15:13:09.939566+0000","last_became_peered":"2024-04-14T15:13:09.939566+0000","last_unstale":"2024-04-14T15:16:28.926617+0000","last_undegraded":"2024-04-14T15:16:28.926617+0000","last_fullsized":"2024-04-14T15:16:28.926617+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'56","last_scrub_stamp":"2024-04-14T15:14:53.271937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:14:53.271937+0000","objects_scrubbed":4,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.003844564,"stat_sum":{"num_bytes":48,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":21,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"110.15","version":"190'34","reported_seq":588,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.928068+0000","last_change":"2024-04-14T15:16:06.494156+0000","last_active":"2024-04-14T15:16:28.928068+0000","last_peered":"2024-04-14T15:16:28.928068+0000","last_clean":"2024-04-14T15:16:28.928068+0000","last_became_active":"2024-04-14T15:13:09.960121+0000","last_became_peered":"2024-04-14T15:13:09.960121+0000","last_unstale":"2024-04-14T15:16:28.928068+0000","last_undegraded":"2024-04-14T15:16:28.928068+0000","last_fullsized":"2024-04-14T15:16:28.928068+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'34","last_scrub_stamp":"2024-04-14T15:16:06.494065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:16:06.494065+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:22.075048+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0020885510000000001,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"110.14","version":"275'86","reported_seq":633,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.928108+0000","last_change":"2024-04-14T15:16:03.883165+0000","last_active":"2024-04-14T15:16:28.928108+0000","last_peered":"2024-04-14T15:16:28.928108+0000","last_clean":"2024-04-14T15:16:28.928108+0000","last_became_active":"2024-04-14T15:13:09.899561+0000","last_became_peered":"2024-04-14T15:13:09.899561+0000","last_unstale":"2024-04-14T15:16:28.928108+0000","last_undegraded":"2024-04-14T15:16:28.928108+0000","last_fullsized":"2024-04-14T15:16:28.928108+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'85","last_scrub_stamp":"2024-04-14T15:15:42.409029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:15:42.409029+0000","objects_scrubbed":5,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:56.900824+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0039978469999999997,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":26,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"110.17","version":"234'6","reported_seq":567,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.930584+0000","last_change":"2024-04-14T15:16:07.459851+0000","last_active":"2024-04-14T15:16:28.930584+0000","last_peered":"2024-04-14T15:16:28.930584+0000","last_clean":"2024-04-14T15:16:28.930584+0000","last_became_active":"2024-04-14T15:13:09.898521+0000","last_became_peered":"2024-04-14T15:13:09.898521+0000","last_unstale":"2024-04-14T15:16:28.930584+0000","last_undegraded":"2024-04-14T15:16:28.930584+0000","last_fullsized":"2024-04-14T15:16:28.930584+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'6","last_scrub_stamp":"2024-04-14T15:16:07.459718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:16:07.459718+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:32.566497+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0019949540000000002,"stat_sum":{"num_bytes":13,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"110.16","version":"190'34","reported_seq":423,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.890505+0000","last_change":"2024-04-14T15:16:03.887023+0000","last_active":"2024-04-14T15:16:28.890505+0000","last_peered":"2024-04-14T15:16:28.890505+0000","last_clean":"2024-04-14T15:16:28.890505+0000","last_became_active":"2024-04-14T15:15:21.889783+0000","last_became_peered":"2024-04-14T15:15:21.889783+0000","last_unstale":"2024-04-14T15:16:28.890505+0000","last_undegraded":"2024-04-14T15:16:28.890505+0000","last_fullsized":"2024-04-14T15:16:28.890505+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'34","last_scrub_stamp":"2024-04-14T15:15:24.848169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:15:24.848169+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:45.401248+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.002398131,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"110.19","version":"237'85","reported_seq":637,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.887375+0000","last_change":"2024-04-14T15:16:04.177194+0000","last_active":"2024-04-14T15:16:28.887375+0000","last_peered":"2024-04-14T15:16:28.887375+0000","last_clean":"2024-04-14T15:16:28.887375+0000","last_became_active":"2024-04-14T15:13:09.998139+0000","last_became_peered":"2024-04-14T15:13:09.998139+0000","last_unstale":"2024-04-14T15:16:28.887375+0000","last_undegraded":"2024-04-14T15:16:28.887375+0000","last_fullsized":"2024-04-14T15:16:28.887375+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'85","last_scrub_stamp":"2024-04-14T15:16:04.177097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:16:04.177097+0000","objects_scrubbed":5,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:11.344540+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0060025429999999999,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"110.18","version":"296'93","reported_seq":636,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.885791+0000","last_change":"2024-04-14T15:16:05.887645+0000","last_active":"2024-04-14T15:16:28.885791+0000","last_peered":"2024-04-14T15:16:28.885791+0000","last_clean":"2024-04-14T15:16:28.885791+0000","last_became_active":"2024-04-14T15:13:09.998803+0000","last_became_peered":"2024-04-14T15:13:09.998803+0000","last_unstale":"2024-04-14T15:16:28.885791+0000","last_undegraded":"2024-04-14T15:16:28.885791+0000","last_fullsized":"2024-04-14T15:16:28.885791+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'85","last_scrub_stamp":"2024-04-14T15:15:35.308199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:15:35.308199+0000","objects_scrubbed":5,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:52.772617+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.024348310000000001,"stat_sum":{"num_bytes":57,"num_objects":7,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":28,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"110.1b","version":"234'90","reported_seq":637,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.926777+0000","last_change":"2024-04-14T15:16:03.895253+0000","last_active":"2024-04-14T15:16:28.926777+0000","last_peered":"2024-04-14T15:16:28.926777+0000","last_clean":"2024-04-14T15:16:28.926777+0000","last_became_active":"2024-04-14T15:13:09.946919+0000","last_became_peered":"2024-04-14T15:13:09.946919+0000","last_unstale":"2024-04-14T15:16:28.926777+0000","last_undegraded":"2024-04-14T15:16:28.926777+0000","last_fullsized":"2024-04-14T15:16:28.926777+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'90","last_scrub_stamp":"2024-04-14T15:15:27.424479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:15:27.424479+0000","objects_scrubbed":5,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:38.879520+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0037919410000000001,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":27,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"110.1a","version":"203'34","reported_seq":586,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.928745+0000","last_change":"2024-04-14T15:16:03.875517+0000","last_active":"2024-04-14T15:16:28.928745+0000","last_peered":"2024-04-14T15:16:28.928745+0000","last_clean":"2024-04-14T15:16:28.928745+0000","last_became_active":"2024-04-14T15:13:09.960319+0000","last_became_peered":"2024-04-14T15:13:09.960319+0000","last_unstale":"2024-04-14T15:16:28.928745+0000","last_undegraded":"2024-04-14T15:16:28.928745+0000","last_fullsized":"2024-04-14T15:16:28.928745+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'34","last_scrub_stamp":"2024-04-14T15:15:34.212376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:15:34.212376+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:53.949593+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0017807330000000001,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"110.1d","version":"225'85","reported_seq":633,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.929246+0000","last_change":"2024-04-14T15:16:03.896420+0000","last_active":"2024-04-14T15:16:28.929246+0000","last_peered":"2024-04-14T15:16:28.929246+0000","last_clean":"2024-04-14T15:16:28.929246+0000","last_became_active":"2024-04-14T15:13:09.868082+0000","last_became_peered":"2024-04-14T15:13:09.868082+0000","last_unstale":"2024-04-14T15:16:28.929246+0000","last_undegraded":"2024-04-14T15:16:28.929246+0000","last_fullsized":"2024-04-14T15:16:28.929246+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"225'85","last_scrub_stamp":"2024-04-14T15:15:26.704550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:15:26.704550+0000","objects_scrubbed":5,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:48.589263+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00141866,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"110.1c","version":"190'73","reported_seq":623,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.929080+0000","last_change":"2024-04-14T15:16:04.615669+0000","last_active":"2024-04-14T15:16:28.929080+0000","last_peered":"2024-04-14T15:16:28.929080+0000","last_clean":"2024-04-14T15:16:28.929080+0000","last_became_active":"2024-04-14T15:13:09.954091+0000","last_became_peered":"2024-04-14T15:13:09.954091+0000","last_unstale":"2024-04-14T15:16:28.929080+0000","last_undegraded":"2024-04-14T15:16:28.929080+0000","last_fullsized":"2024-04-14T15:16:28.929080+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'73","last_scrub_stamp":"2024-04-14T15:16:04.615572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:16:04.615572+0000","objects_scrubbed":5,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:10.131872+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0056190420000000003,"stat_sum":{"num_bytes":49,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"110.1f","version":"268'25","reported_seq":633,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.889316+0000","last_change":"2024-04-14T15:16:08.237015+0000","last_active":"2024-04-14T15:16:28.889316+0000","last_peered":"2024-04-14T15:16:28.889316+0000","last_clean":"2024-04-14T15:16:28.889316+0000","last_became_active":"2024-04-14T15:13:09.999629+0000","last_became_peered":"2024-04-14T15:13:09.999629+0000","last_unstale":"2024-04-14T15:16:28.889316+0000","last_undegraded":"2024-04-14T15:16:28.889316+0000","last_fullsized":"2024-04-14T15:16:28.889316+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'25","last_scrub_stamp":"2024-04-14T15:16:08.236881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:16:08.236881+0000","objects_scrubbed":2,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:36.117768+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0061123380000000001,"stat_sum":{"num_bytes":4194315,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":8,"num_write_kb":8198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"110.1e","version":"190'22","reported_seq":580,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.931272+0000","last_change":"2024-04-14T15:16:10.428089+0000","last_active":"2024-04-14T15:16:28.931272+0000","last_peered":"2024-04-14T15:16:28.931272+0000","last_clean":"2024-04-14T15:16:28.931272+0000","last_became_active":"2024-04-14T15:13:09.955932+0000","last_became_peered":"2024-04-14T15:13:09.955932+0000","last_unstale":"2024-04-14T15:16:28.931272+0000","last_undegraded":"2024-04-14T15:16:28.931272+0000","last_fullsized":"2024-04-14T15:16:28.931272+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'22","last_scrub_stamp":"2024-04-14T15:16:10.427994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:16:10.427994+0000","objects_scrubbed":1,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:37.611855+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.003436284,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"110.1","version":"312'25","reported_seq":586,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.926602+0000","last_change":"2024-04-14T15:16:03.895140+0000","last_active":"2024-04-14T15:16:28.926602+0000","last_peered":"2024-04-14T15:16:28.926602+0000","last_clean":"2024-04-14T15:16:28.926602+0000","last_became_active":"2024-04-14T15:13:09.929562+0000","last_became_peered":"2024-04-14T15:13:09.929562+0000","last_unstale":"2024-04-14T15:16:28.926602+0000","last_undegraded":"2024-04-14T15:16:28.926602+0000","last_fullsized":"2024-04-14T15:16:28.926602+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"219'24","last_scrub_stamp":"2024-04-14T15:15:35.591006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:15:35.591006+0000","objects_scrubbed":3,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:37.596403+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00097444399999999998,"stat_sum":{"num_bytes":4000032,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":10,"num_write_kb":3916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"110.0","version":"219'85","reported_seq":627,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.880920+0000","last_change":"2024-04-14T15:16:03.881458+0000","last_active":"2024-04-14T15:16:28.880920+0000","last_peered":"2024-04-14T15:16:28.880920+0000","last_clean":"2024-04-14T15:16:28.880920+0000","last_became_active":"2024-04-14T15:13:09.856407+0000","last_became_peered":"2024-04-14T15:13:09.856407+0000","last_unstale":"2024-04-14T15:16:28.880920+0000","last_undegraded":"2024-04-14T15:16:28.880920+0000","last_fullsized":"2024-04-14T15:16:28.880920+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"219'85","last_scrub_stamp":"2024-04-14T15:15:25.872949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:15:25.872949+0000","objects_scrubbed":5,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:39.593819+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0.0020747539999999998,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"110.3","version":"223'51","reported_seq":596,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.886201+0000","last_change":"2024-04-14T15:16:06.149432+0000","last_active":"2024-04-14T15:16:28.886201+0000","last_peered":"2024-04-14T15:16:28.886201+0000","last_clean":"2024-04-14T15:16:28.886201+0000","last_became_active":"2024-04-14T15:13:09.997208+0000","last_became_peered":"2024-04-14T15:13:09.997208+0000","last_unstale":"2024-04-14T15:16:28.886201+0000","last_undegraded":"2024-04-14T15:16:28.886201+0000","last_fullsized":"2024-04-14T15:16:28.886201+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"223'51","last_scrub_stamp":"2024-04-14T15:16:06.149271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:16:06.149271+0000","objects_scrubbed":3,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:09.788395+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0068076480000000003,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"110.2","version":"240'55","reported_seq":607,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.887365+0000","last_change":"2024-04-14T15:16:03.880577+0000","last_active":"2024-04-14T15:16:28.887365+0000","last_peered":"2024-04-14T15:16:28.887365+0000","last_clean":"2024-04-14T15:16:28.887365+0000","last_became_active":"2024-04-14T15:13:09.957187+0000","last_became_peered":"2024-04-14T15:13:09.957187+0000","last_unstale":"2024-04-14T15:16:28.887365+0000","last_undegraded":"2024-04-14T15:16:28.887365+0000","last_fullsized":"2024-04-14T15:16:28.887365+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'55","last_scrub_stamp":"2024-04-14T15:15:40.376133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:15:40.376133+0000","objects_scrubbed":6,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:09.884940+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0066344619999999998,"stat_sum":{"num_bytes":49,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":18,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"110.5","version":"283'59","reported_seq":605,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.881973+0000","last_change":"2024-04-14T15:16:03.882342+0000","last_active":"2024-04-14T15:16:28.881973+0000","last_peered":"2024-04-14T15:16:28.881973+0000","last_clean":"2024-04-14T15:16:28.881973+0000","last_became_active":"2024-04-14T15:13:09.853689+0000","last_became_peered":"2024-04-14T15:13:09.853689+0000","last_unstale":"2024-04-14T15:16:28.881973+0000","last_undegraded":"2024-04-14T15:16:28.881973+0000","last_fullsized":"2024-04-14T15:16:28.881973+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'57","last_scrub_stamp":"2024-04-14T15:15:27.862702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:15:27.862702+0000","objects_scrubbed":5,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:56.923474+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.001407202,"stat_sum":{"num_bytes":42,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":20,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"110.4","version":"190'51","reported_seq":600,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.928140+0000","last_change":"2024-04-14T15:16:05.487931+0000","last_active":"2024-04-14T15:16:28.928140+0000","last_peered":"2024-04-14T15:16:28.928140+0000","last_clean":"2024-04-14T15:16:28.928140+0000","last_became_active":"2024-04-14T15:13:09.958847+0000","last_became_peered":"2024-04-14T15:13:09.958847+0000","last_unstale":"2024-04-14T15:16:28.928140+0000","last_undegraded":"2024-04-14T15:16:28.928140+0000","last_fullsized":"2024-04-14T15:16:28.928140+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'51","last_scrub_stamp":"2024-04-14T15:16:05.487850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:16:05.487850+0000","objects_scrubbed":3,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:28.383618+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0048323250000000002,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"110.7","version":"201'17","reported_seq":573,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.930034+0000","last_change":"2024-04-14T15:16:03.876877+0000","last_active":"2024-04-14T15:16:28.930034+0000","last_peered":"2024-04-14T15:16:28.930034+0000","last_clean":"2024-04-14T15:16:28.930034+0000","last_became_active":"2024-04-14T15:13:09.936631+0000","last_became_peered":"2024-04-14T15:13:09.936631+0000","last_unstale":"2024-04-14T15:16:28.930034+0000","last_undegraded":"2024-04-14T15:16:28.930034+0000","last_fullsized":"2024-04-14T15:16:28.930034+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"201'17","last_scrub_stamp":"2024-04-14T15:15:30.308163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:15:30.308163+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:43.060029+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001103808,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"110.6","version":"274'74","reported_seq":618,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.927928+0000","last_change":"2024-04-14T15:16:03.897478+0000","last_active":"2024-04-14T15:16:28.927928+0000","last_peered":"2024-04-14T15:16:28.927928+0000","last_clean":"2024-04-14T15:16:28.927928+0000","last_became_active":"2024-04-14T15:13:09.930125+0000","last_became_peered":"2024-04-14T15:13:09.930125+0000","last_unstale":"2024-04-14T15:16:28.927928+0000","last_undegraded":"2024-04-14T15:16:28.927928+0000","last_fullsized":"2024-04-14T15:16:28.927928+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'74","last_scrub_stamp":"2024-04-14T15:15:42.641684+0000","last_deep_scrub":"274'74","last_deep_scrub_stamp":"2024-04-14T15:15:42.641684+0000","last_clean_scrub_stamp":"2024-04-14T15:15:42.641684+0000","objects_scrubbed":5,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:58.757895+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001879475,"stat_sum":{"num_bytes":44,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":21,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"110.9","version":"225'17","reported_seq":573,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.887186+0000","last_change":"2024-04-14T15:16:09.202557+0000","last_active":"2024-04-14T15:16:28.887186+0000","last_peered":"2024-04-14T15:16:28.887186+0000","last_clean":"2024-04-14T15:16:28.887186+0000","last_became_active":"2024-04-14T15:13:09.957324+0000","last_became_peered":"2024-04-14T15:13:09.957324+0000","last_unstale":"2024-04-14T15:16:28.887186+0000","last_undegraded":"2024-04-14T15:16:28.887186+0000","last_fullsized":"2024-04-14T15:16:28.887186+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"225'17","last_scrub_stamp":"2024-04-14T15:16:09.202462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:16:09.202462+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:19.324590+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0055950490000000004,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"110.8","version":"190'17","reported_seq":328,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.888220+0000","last_change":"2024-04-14T15:16:03.889092+0000","last_active":"2024-04-14T15:16:28.888220+0000","last_peered":"2024-04-14T15:16:28.888220+0000","last_clean":"2024-04-14T15:16:28.888220+0000","last_became_active":"2024-04-14T15:15:44.890057+0000","last_became_peered":"2024-04-14T15:15:44.890057+0000","last_unstale":"2024-04-14T15:16:28.888220+0000","last_undegraded":"2024-04-14T15:16:28.888220+0000","last_fullsized":"2024-04-14T15:16:28.888220+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'17","last_scrub_stamp":"2024-04-14T15:15:25.478924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:15:25.478924+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:42.620542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00271433,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"110.b","version":"190'68","reported_seq":615,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.927087+0000","last_change":"2024-04-14T15:16:03.898209+0000","last_active":"2024-04-14T15:16:28.927087+0000","last_peered":"2024-04-14T15:16:28.927087+0000","last_clean":"2024-04-14T15:16:28.927087+0000","last_became_active":"2024-04-14T15:13:09.964629+0000","last_became_peered":"2024-04-14T15:13:09.964629+0000","last_unstale":"2024-04-14T15:16:28.927087+0000","last_undegraded":"2024-04-14T15:16:28.927087+0000","last_fullsized":"2024-04-14T15:16:28.927087+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'68","last_scrub_stamp":"2024-04-14T15:14:51.349107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:14:51.349107+0000","objects_scrubbed":4,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0049053020000000003,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"110.a","version":"224'70","reported_seq":544,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.886273+0000","last_change":"2024-04-14T15:16:03.888790+0000","last_active":"2024-04-14T15:16:28.886273+0000","last_peered":"2024-04-14T15:16:28.886273+0000","last_clean":"2024-04-14T15:16:28.886273+0000","last_became_active":"2024-04-14T15:15:21.885346+0000","last_became_peered":"2024-04-14T15:15:21.885346+0000","last_unstale":"2024-04-14T15:16:28.886273+0000","last_undegraded":"2024-04-14T15:16:28.886273+0000","last_fullsized":"2024-04-14T15:16:28.886273+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'70","last_scrub_stamp":"2024-04-14T15:15:34.881731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:15:34.881731+0000","objects_scrubbed":5,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:40.406926+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0.00502967,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"110.d","version":"304'90","reported_seq":471,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.890894+0000","last_change":"2024-04-14T15:16:03.886262+0000","last_active":"2024-04-14T15:16:28.890894+0000","last_peered":"2024-04-14T15:16:28.890894+0000","last_clean":"2024-04-14T15:16:28.890894+0000","last_became_active":"2024-04-14T15:15:21.877073+0000","last_became_peered":"2024-04-14T15:15:21.877073+0000","last_unstale":"2024-04-14T15:16:28.890894+0000","last_undegraded":"2024-04-14T15:16:28.890894+0000","last_fullsized":"2024-04-14T15:16:28.890894+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'85","last_scrub_stamp":"2024-04-14T15:15:38.773256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:15:38.773256+0000","objects_scrubbed":5,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:05.854514+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0018858709999999999,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"110.c","version":"190'17","reported_seq":573,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.929192+0000","last_change":"2024-04-14T15:16:03.899855+0000","last_active":"2024-04-14T15:16:28.929192+0000","last_peered":"2024-04-14T15:16:28.929192+0000","last_clean":"2024-04-14T15:16:28.929192+0000","last_became_active":"2024-04-14T15:13:09.984609+0000","last_became_peered":"2024-04-14T15:13:09.984609+0000","last_unstale":"2024-04-14T15:16:28.929192+0000","last_undegraded":"2024-04-14T15:16:28.929192+0000","last_fullsized":"2024-04-14T15:16:28.929192+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'17","last_scrub_stamp":"2024-04-14T15:15:24.368920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:15:24.368920+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:29.970768+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0057052739999999998,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"110.f","version":"190'51","reported_seq":603,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.925554+0000","last_change":"2024-04-14T15:16:03.898253+0000","last_active":"2024-04-14T15:16:28.925554+0000","last_peered":"2024-04-14T15:16:28.925554+0000","last_clean":"2024-04-14T15:16:28.925554+0000","last_became_active":"2024-04-14T15:13:09.964057+0000","last_became_peered":"2024-04-14T15:13:09.964057+0000","last_unstale":"2024-04-14T15:16:28.925554+0000","last_undegraded":"2024-04-14T15:16:28.925554+0000","last_fullsized":"2024-04-14T15:16:28.925554+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'51","last_scrub_stamp":"2024-04-14T15:15:25.399677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:15:25.399677+0000","objects_scrubbed":3,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:43.303787+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0052446230000000003,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"110.e","version":"234'90","reported_seq":636,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.927639+0000","last_change":"2024-04-14T15:16:03.897032+0000","last_active":"2024-04-14T15:16:28.927639+0000","last_peered":"2024-04-14T15:16:28.927639+0000","last_clean":"2024-04-14T15:16:28.927639+0000","last_became_active":"2024-04-14T15:13:09.984980+0000","last_became_peered":"2024-04-14T15:13:09.984980+0000","last_unstale":"2024-04-14T15:16:28.927639+0000","last_undegraded":"2024-04-14T15:16:28.927639+0000","last_fullsized":"2024-04-14T15:16:28.927639+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'90","last_scrub_stamp":"2024-04-14T15:15:24.699013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:15:24.699013+0000","objects_scrubbed":6,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:50.594595+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0016350589999999999,"stat_sum":{"num_bytes":60,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":100,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.889434+0000","last_change":"2024-04-14T15:15:44.883275+0000","last_active":"2024-04-14T15:16:28.889434+0000","last_peered":"2024-04-14T15:16:28.889434+0000","last_clean":"2024-04-14T15:16:28.889434+0000","last_became_active":"2024-04-14T15:15:44.882941+0000","last_became_peered":"2024-04-14T15:15:44.882941+0000","last_unstale":"2024-04-14T15:16:28.889434+0000","last_undegraded":"2024-04-14T15:16:28.889434+0000","last_fullsized":"2024-04-14T15:16:28.889434+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:32.465858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:32.465858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:53.128842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.9","version":"85'9","reported_seq":633,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.927077+0000","last_change":"2024-04-14T15:14:45.904003+0000","last_active":"2024-04-14T15:16:28.927077+0000","last_peered":"2024-04-14T15:16:28.927077+0000","last_clean":"2024-04-14T15:16:28.927077+0000","last_became_active":"2024-04-14T15:09:47.045441+0000","last_became_peered":"2024-04-14T15:09:47.045441+0000","last_unstale":"2024-04-14T15:16:28.927077+0000","last_undegraded":"2024-04-14T15:16:28.927077+0000","last_fullsized":"2024-04-14T15:16:28.927077+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9","last_scrub_stamp":"2024-04-14T15:14:45.903923+0000","last_deep_scrub":"85'9","last_deep_scrub_stamp":"2024-04-14T15:10:51.950580+0000","last_clean_scrub_stamp":"2024-04-14T15:14:45.903923+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:29.785862+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"72'15","reported_seq":144,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.925653+0000","last_change":"2024-04-14T15:15:44.885742+0000","last_active":"2024-04-14T15:16:28.925653+0000","last_peered":"2024-04-14T15:16:28.925653+0000","last_clean":"2024-04-14T15:16:28.925653+0000","last_became_active":"2024-04-14T15:15:44.885409+0000","last_became_peered":"2024-04-14T15:15:44.885409+0000","last_unstale":"2024-04-14T15:16:28.925653+0000","last_undegraded":"2024-04-14T15:16:28.925653+0000","last_fullsized":"2024-04-14T15:16:28.925653+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-04-14T15:15:39.325129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:15:39.325129+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:00.619852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":550,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.927170+0000","last_change":"2024-04-14T15:15:35.560140+0000","last_active":"2024-04-14T15:16:28.927170+0000","last_peered":"2024-04-14T15:16:28.927170+0000","last_clean":"2024-04-14T15:16:28.927170+0000","last_became_active":"2024-04-14T15:09:43.021460+0000","last_became_peered":"2024-04-14T15:09:43.021460+0000","last_unstale":"2024-04-14T15:16:28.927170+0000","last_undegraded":"2024-04-14T15:16:28.927170+0000","last_fullsized":"2024-04-14T15:16:28.927170+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:35.560063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:15:35.560063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:41.037947+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":101,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.928908+0000","last_change":"2024-04-14T15:15:44.888612+0000","last_active":"2024-04-14T15:16:28.928908+0000","last_peered":"2024-04-14T15:16:28.928908+0000","last_clean":"2024-04-14T15:16:28.928908+0000","last_became_active":"2024-04-14T15:15:44.888154+0000","last_became_peered":"2024-04-14T15:15:44.888154+0000","last_unstale":"2024-04-14T15:16:28.928908+0000","last_undegraded":"2024-04-14T15:16:28.928908+0000","last_fullsized":"2024-04-14T15:16:28.928908+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:40.368219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:15:40.368219+0000","last_clean_scrub_stamp":"2024-04-14T15:15:40.368219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:00.403221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":446,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.926707+0000","last_change":"2024-04-14T15:15:32.596167+0000","last_active":"2024-04-14T15:16:28.926707+0000","last_peered":"2024-04-14T15:16:28.926707+0000","last_clean":"2024-04-14T15:16:28.926707+0000","last_became_active":"2024-04-14T15:12:56.355348+0000","last_became_peered":"2024-04-14T15:12:56.355348+0000","last_unstale":"2024-04-14T15:16:28.926707+0000","last_undegraded":"2024-04-14T15:16:28.926707+0000","last_fullsized":"2024-04-14T15:16:28.926707+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:32.596096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:32.596096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:53.341960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"72'9","reported_seq":562,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.928368+0000","last_change":"2024-04-14T15:14:08.367575+0000","last_active":"2024-04-14T15:16:28.928368+0000","last_peered":"2024-04-14T15:16:28.928368+0000","last_clean":"2024-04-14T15:16:28.928368+0000","last_became_active":"2024-04-14T15:09:45.037179+0000","last_became_peered":"2024-04-14T15:09:45.037179+0000","last_unstale":"2024-04-14T15:16:28.928368+0000","last_undegraded":"2024-04-14T15:16:28.928368+0000","last_fullsized":"2024-04-14T15:16:28.928368+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-04-14T15:14:08.367484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:14:08.367484+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":550,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.890842+0000","last_change":"2024-04-14T15:15:25.311260+0000","last_active":"2024-04-14T15:16:28.890842+0000","last_peered":"2024-04-14T15:16:28.890842+0000","last_clean":"2024-04-14T15:16:28.890842+0000","last_became_active":"2024-04-14T15:09:43.020044+0000","last_became_peered":"2024-04-14T15:09:43.020044+0000","last_unstale":"2024-04-14T15:16:28.890842+0000","last_undegraded":"2024-04-14T15:16:28.890842+0000","last_fullsized":"2024-04-14T15:16:28.890842+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:25.311184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:15:25.311184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:51.092544+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":550,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.926850+0000","last_change":"2024-04-14T15:16:16.655593+0000","last_active":"2024-04-14T15:16:28.926850+0000","last_peered":"2024-04-14T15:16:28.926850+0000","last_clean":"2024-04-14T15:16:28.926850+0000","last_became_active":"2024-04-14T15:09:47.046153+0000","last_became_peered":"2024-04-14T15:09:47.046153+0000","last_unstale":"2024-04-14T15:16:28.926850+0000","last_undegraded":"2024-04-14T15:16:28.926850+0000","last_fullsized":"2024-04-14T15:16:28.926850+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:16.655504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:16.655504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:32.376012+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":538,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.926841+0000","last_change":"2024-04-14T15:14:13.382977+0000","last_active":"2024-04-14T15:16:28.926841+0000","last_peered":"2024-04-14T15:16:28.926841+0000","last_clean":"2024-04-14T15:16:28.926841+0000","last_became_active":"2024-04-14T15:09:49.061495+0000","last_became_peered":"2024-04-14T15:09:49.061495+0000","last_unstale":"2024-04-14T15:16:28.926841+0000","last_undegraded":"2024-04-14T15:16:28.926841+0000","last_fullsized":"2024-04-14T15:16:28.926841+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:13.382879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:14:13.382879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":446,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.928982+0000","last_change":"2024-04-14T15:14:55.989719+0000","last_active":"2024-04-14T15:16:28.928982+0000","last_peered":"2024-04-14T15:16:28.928982+0000","last_clean":"2024-04-14T15:16:28.928982+0000","last_became_active":"2024-04-14T15:12:56.246062+0000","last_became_peered":"2024-04-14T15:12:56.246062+0000","last_unstale":"2024-04-14T15:16:28.928982+0000","last_undegraded":"2024-04-14T15:16:28.928982+0000","last_fullsized":"2024-04-14T15:16:28.928982+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:55.989627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:14:55.989627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:29.785892+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"222'28","reported_seq":586,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.929289+0000","last_change":"2024-04-14T15:14:50.055093+0000","last_active":"2024-04-14T15:16:28.929289+0000","last_peered":"2024-04-14T15:16:28.929289+0000","last_clean":"2024-04-14T15:16:28.929289+0000","last_became_active":"2024-04-14T15:09:45.178782+0000","last_became_peered":"2024-04-14T15:09:45.178782+0000","last_unstale":"2024-04-14T15:16:28.929289+0000","last_undegraded":"2024-04-14T15:16:28.929289+0000","last_fullsized":"2024-04-14T15:16:28.929289+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'28","last_scrub_stamp":"2024-04-14T15:14:50.054931+0000","last_deep_scrub":"222'28","last_deep_scrub_stamp":"2024-04-14T15:14:50.054931+0000","last_clean_scrub_stamp":"2024-04-14T15:14:50.054931+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:29.834831+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":558,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.929821+0000","last_change":"2024-04-14T15:15:43.569812+0000","last_active":"2024-04-14T15:16:28.929821+0000","last_peered":"2024-04-14T15:16:28.929821+0000","last_clean":"2024-04-14T15:16:28.929821+0000","last_became_active":"2024-04-14T15:09:43.020268+0000","last_became_peered":"2024-04-14T15:09:43.020268+0000","last_unstale":"2024-04-14T15:16:28.929821+0000","last_undegraded":"2024-04-14T15:16:28.929821+0000","last_fullsized":"2024-04-14T15:16:28.929821+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:43.569735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:34.342793+0000","last_clean_scrub_stamp":"2024-04-14T15:15:43.569735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:04.129563+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":541,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.927226+0000","last_change":"2024-04-14T15:15:30.488857+0000","last_active":"2024-04-14T15:16:28.927226+0000","last_peered":"2024-04-14T15:16:28.927226+0000","last_clean":"2024-04-14T15:16:28.927226+0000","last_became_active":"2024-04-14T15:09:47.045542+0000","last_became_peered":"2024-04-14T15:09:47.045542+0000","last_unstale":"2024-04-14T15:16:28.927226+0000","last_undegraded":"2024-04-14T15:16:28.927226+0000","last_fullsized":"2024-04-14T15:16:28.927226+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:30.488781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:15:30.488781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:56.619373+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":538,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.926061+0000","last_change":"2024-04-14T15:16:08.857476+0000","last_active":"2024-04-14T15:16:28.926061+0000","last_peered":"2024-04-14T15:16:28.926061+0000","last_clean":"2024-04-14T15:16:28.926061+0000","last_became_active":"2024-04-14T15:09:49.065397+0000","last_became_peered":"2024-04-14T15:09:49.065397+0000","last_unstale":"2024-04-14T15:16:28.926061+0000","last_undegraded":"2024-04-14T15:16:28.926061+0000","last_fullsized":"2024-04-14T15:16:28.926061+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:08.857397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:08.857397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:26.230364+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"91'130","reported_seq":425,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.924980+0000","last_change":"2024-04-14T15:16:07.609240+0000","last_active":"2024-04-14T15:16:28.924980+0000","last_peered":"2024-04-14T15:16:28.924980+0000","last_clean":"2024-04-14T15:16:28.924980+0000","last_became_active":"2024-04-14T15:15:44.886240+0000","last_became_peered":"2024-04-14T15:15:44.886240+0000","last_unstale":"2024-04-14T15:16:28.924980+0000","last_undegraded":"2024-04-14T15:16:28.924980+0000","last_fullsized":"2024-04-14T15:16:28.924980+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:16:07.609119+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:16:07.609119+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:27.856801+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":454,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.926768+0000","last_change":"2024-04-14T15:16:19.749313+0000","last_active":"2024-04-14T15:16:28.926768+0000","last_peered":"2024-04-14T15:16:28.926768+0000","last_clean":"2024-04-14T15:16:28.926768+0000","last_became_active":"2024-04-14T15:12:56.474322+0000","last_became_peered":"2024-04-14T15:12:56.474322+0000","last_unstale":"2024-04-14T15:16:28.926768+0000","last_undegraded":"2024-04-14T15:16:28.926768+0000","last_fullsized":"2024-04-14T15:16:28.926768+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:19.749208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:16:19.749208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:41.767235+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"72'17","reported_seq":621,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.882442+0000","last_change":"2024-04-14T15:15:39.992484+0000","last_active":"2024-04-14T15:16:28.882442+0000","last_peered":"2024-04-14T15:16:28.882442+0000","last_clean":"2024-04-14T15:16:28.882442+0000","last_became_active":"2024-04-14T15:13:12.933907+0000","last_became_peered":"2024-04-14T15:13:12.933907+0000","last_unstale":"2024-04-14T15:16:28.882442+0000","last_undegraded":"2024-04-14T15:16:28.882442+0000","last_fullsized":"2024-04-14T15:16:28.882442+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'17","last_scrub_stamp":"2024-04-14T15:15:39.992385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:15:39.992385+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:48.777854+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":550,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.926018+0000","last_change":"2024-04-14T15:16:16.820151+0000","last_active":"2024-04-14T15:16:28.926018+0000","last_peered":"2024-04-14T15:16:28.926018+0000","last_clean":"2024-04-14T15:16:28.926018+0000","last_became_active":"2024-04-14T15:09:43.022351+0000","last_became_peered":"2024-04-14T15:09:43.022351+0000","last_unstale":"2024-04-14T15:16:28.926018+0000","last_undegraded":"2024-04-14T15:16:28.926018+0000","last_fullsized":"2024-04-14T15:16:28.926018+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:16.820070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:16.820070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:45.859854+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":533,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.886414+0000","last_change":"2024-04-14T15:14:46.208558+0000","last_active":"2024-04-14T15:16:28.886414+0000","last_peered":"2024-04-14T15:16:28.886414+0000","last_clean":"2024-04-14T15:16:28.886414+0000","last_became_active":"2024-04-14T15:09:47.042810+0000","last_became_peered":"2024-04-14T15:09:47.042810+0000","last_unstale":"2024-04-14T15:16:28.886414+0000","last_undegraded":"2024-04-14T15:16:28.886414+0000","last_fullsized":"2024-04-14T15:16:28.886414+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:46.208455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:14:46.208455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":537,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.926631+0000","last_change":"2024-04-14T15:14:03.309357+0000","last_active":"2024-04-14T15:16:28.926631+0000","last_peered":"2024-04-14T15:16:28.926631+0000","last_clean":"2024-04-14T15:16:28.926631+0000","last_became_active":"2024-04-14T15:09:49.069594+0000","last_became_peered":"2024-04-14T15:09:49.069594+0000","last_unstale":"2024-04-14T15:16:28.926631+0000","last_undegraded":"2024-04-14T15:16:28.926631+0000","last_fullsized":"2024-04-14T15:16:28.926631+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.309280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.309280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"91'130","reported_seq":819,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.926413+0000","last_change":"2024-04-14T15:14:46.903338+0000","last_active":"2024-04-14T15:16:28.926413+0000","last_peered":"2024-04-14T15:16:28.926413+0000","last_clean":"2024-04-14T15:16:28.926413+0000","last_became_active":"2024-04-14T15:14:17.873403+0000","last_became_peered":"2024-04-14T15:14:17.873403+0000","last_unstale":"2024-04-14T15:16:28.926413+0000","last_undegraded":"2024-04-14T15:16:28.926413+0000","last_fullsized":"2024-04-14T15:16:28.926413+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:14:46.903248+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:14:46.903248+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:29.785650+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":446,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.927240+0000","last_change":"2024-04-14T15:14:52.110153+0000","last_active":"2024-04-14T15:16:28.927240+0000","last_peered":"2024-04-14T15:16:28.927240+0000","last_clean":"2024-04-14T15:16:28.927240+0000","last_became_active":"2024-04-14T15:12:56.289745+0000","last_became_peered":"2024-04-14T15:12:56.289745+0000","last_unstale":"2024-04-14T15:16:28.927240+0000","last_undegraded":"2024-04-14T15:16:28.927240+0000","last_fullsized":"2024-04-14T15:16:28.927240+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:52.110075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:14:52.110075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:29.834847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"179'15","reported_seq":557,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.881696+0000","last_change":"2024-04-14T15:14:07.488544+0000","last_active":"2024-04-14T15:16:28.881696+0000","last_peered":"2024-04-14T15:16:28.881696+0000","last_clean":"2024-04-14T15:16:28.881696+0000","last_became_active":"2024-04-14T15:09:45.029170+0000","last_became_peered":"2024-04-14T15:09:45.029170+0000","last_unstale":"2024-04-14T15:16:28.881696+0000","last_undegraded":"2024-04-14T15:16:28.881696+0000","last_fullsized":"2024-04-14T15:16:28.881696+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'15","last_scrub_stamp":"2024-04-14T15:14:07.488422+0000","last_deep_scrub":"179'15","last_deep_scrub_stamp":"2024-04-14T15:14:07.488422+0000","last_clean_scrub_stamp":"2024-04-14T15:14:07.488422+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:29.106719+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":550,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.887659+0000","last_change":"2024-04-14T15:14:40.222949+0000","last_active":"2024-04-14T15:16:28.887659+0000","last_peered":"2024-04-14T15:16:28.887659+0000","last_clean":"2024-04-14T15:16:28.887659+0000","last_became_active":"2024-04-14T15:09:43.018871+0000","last_became_peered":"2024-04-14T15:09:43.018871+0000","last_unstale":"2024-04-14T15:16:28.887659+0000","last_undegraded":"2024-04-14T15:16:28.887659+0000","last_fullsized":"2024-04-14T15:16:28.887659+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:40.222873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:14:40.222873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"72'8","reported_seq":631,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.929832+0000","last_change":"2024-04-14T15:15:40.617542+0000","last_active":"2024-04-14T15:16:28.929832+0000","last_peered":"2024-04-14T15:16:28.929832+0000","last_clean":"2024-04-14T15:16:28.929832+0000","last_became_active":"2024-04-14T15:09:47.044085+0000","last_became_peered":"2024-04-14T15:09:47.044085+0000","last_unstale":"2024-04-14T15:16:28.929832+0000","last_undegraded":"2024-04-14T15:16:28.929832+0000","last_fullsized":"2024-04-14T15:16:28.929832+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:15:40.617433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:15:40.617433+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:51.220120+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":537,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.882366+0000","last_change":"2024-04-14T15:16:17.041377+0000","last_active":"2024-04-14T15:16:28.882366+0000","last_peered":"2024-04-14T15:16:28.882366+0000","last_clean":"2024-04-14T15:16:28.882366+0000","last_became_active":"2024-04-14T15:09:49.065274+0000","last_became_peered":"2024-04-14T15:09:49.065274+0000","last_unstale":"2024-04-14T15:16:28.882366+0000","last_undegraded":"2024-04-14T15:16:28.882366+0000","last_fullsized":"2024-04-14T15:16:28.882366+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:17.041301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:17.041301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:30.547783+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"91'130","reported_seq":425,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.925686+0000","last_change":"2024-04-14T15:15:45.589217+0000","last_active":"2024-04-14T15:16:28.925686+0000","last_peered":"2024-04-14T15:16:28.925686+0000","last_clean":"2024-04-14T15:16:28.925686+0000","last_became_active":"2024-04-14T15:15:44.887114+0000","last_became_peered":"2024-04-14T15:15:44.887114+0000","last_unstale":"2024-04-14T15:16:28.925686+0000","last_undegraded":"2024-04-14T15:16:28.925686+0000","last_fullsized":"2024-04-14T15:16:28.925686+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:15:45.589132+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:15:45.589132+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:13.065344+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":446,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.886680+0000","last_change":"2024-04-14T15:15:29.366895+0000","last_active":"2024-04-14T15:16:28.886680+0000","last_peered":"2024-04-14T15:16:28.886680+0000","last_clean":"2024-04-14T15:16:28.886680+0000","last_became_active":"2024-04-14T15:12:56.487615+0000","last_became_peered":"2024-04-14T15:12:56.487615+0000","last_unstale":"2024-04-14T15:16:28.886680+0000","last_undegraded":"2024-04-14T15:16:28.886680+0000","last_fullsized":"2024-04-14T15:16:28.886680+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:29.366820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:29.366820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:55.079805+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"72'15","reported_seq":570,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.889571+0000","last_change":"2024-04-14T15:14:58.216134+0000","last_active":"2024-04-14T15:16:28.889571+0000","last_peered":"2024-04-14T15:16:28.889571+0000","last_clean":"2024-04-14T15:16:28.889571+0000","last_became_active":"2024-04-14T15:09:45.033331+0000","last_became_peered":"2024-04-14T15:09:45.033331+0000","last_unstale":"2024-04-14T15:16:28.889571+0000","last_undegraded":"2024-04-14T15:16:28.889571+0000","last_fullsized":"2024-04-14T15:16:28.889571+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-04-14T15:14:58.215973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:14:58.215973+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":550,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.887711+0000","last_change":"2024-04-14T15:15:00.203399+0000","last_active":"2024-04-14T15:16:28.887711+0000","last_peered":"2024-04-14T15:16:28.887711+0000","last_clean":"2024-04-14T15:16:28.887711+0000","last_became_active":"2024-04-14T15:09:43.020878+0000","last_became_peered":"2024-04-14T15:09:43.020878+0000","last_unstale":"2024-04-14T15:16:28.887711+0000","last_undegraded":"2024-04-14T15:16:28.887711+0000","last_fullsized":"2024-04-14T15:16:28.887711+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:00.203310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:15:00.203310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:32.228150+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"72'8","reported_seq":631,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.882307+0000","last_change":"2024-04-14T15:16:06.933663+0000","last_active":"2024-04-14T15:16:28.882307+0000","last_peered":"2024-04-14T15:16:28.882307+0000","last_clean":"2024-04-14T15:16:28.882307+0000","last_became_active":"2024-04-14T15:09:47.050962+0000","last_became_peered":"2024-04-14T15:09:47.050962+0000","last_unstale":"2024-04-14T15:16:28.882307+0000","last_undegraded":"2024-04-14T15:16:28.882307+0000","last_fullsized":"2024-04-14T15:16:28.882307+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:16:06.933489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:06.933489+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:32.343513+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":101,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.890935+0000","last_change":"2024-04-14T15:15:44.880052+0000","last_active":"2024-04-14T15:16:28.890935+0000","last_peered":"2024-04-14T15:16:28.890935+0000","last_clean":"2024-04-14T15:16:28.890935+0000","last_became_active":"2024-04-14T15:15:44.879699+0000","last_became_peered":"2024-04-14T15:15:44.879699+0000","last_unstale":"2024-04-14T15:16:28.890935+0000","last_undegraded":"2024-04-14T15:16:28.890935+0000","last_fullsized":"2024-04-14T15:16:28.890935+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:00.789082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:14:00.789082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"233'8","reported_seq":460,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.926184+0000","last_change":"2024-04-14T15:15:01.295884+0000","last_active":"2024-04-14T15:16:28.926184+0000","last_peered":"2024-04-14T15:16:28.926184+0000","last_clean":"2024-04-14T15:16:28.926184+0000","last_became_active":"2024-04-14T15:12:56.381026+0000","last_became_peered":"2024-04-14T15:12:56.381026+0000","last_unstale":"2024-04-14T15:16:28.926184+0000","last_undegraded":"2024-04-14T15:16:28.926184+0000","last_fullsized":"2024-04-14T15:16:28.926184+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'8","last_scrub_stamp":"2024-04-14T15:15:01.295800+0000","last_deep_scrub":"233'8","last_deep_scrub_stamp":"2024-04-14T15:15:01.295800+0000","last_clean_scrub_stamp":"2024-04-14T15:15:01.295800+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:31.822069+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"85'16","reported_seq":569,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.882027+0000","last_change":"2024-04-14T15:14:41.852173+0000","last_active":"2024-04-14T15:16:28.882027+0000","last_peered":"2024-04-14T15:16:28.882027+0000","last_clean":"2024-04-14T15:16:28.882027+0000","last_became_active":"2024-04-14T15:09:45.028212+0000","last_became_peered":"2024-04-14T15:09:45.028212+0000","last_unstale":"2024-04-14T15:16:28.882027+0000","last_undegraded":"2024-04-14T15:16:28.882027+0000","last_fullsized":"2024-04-14T15:16:28.882027+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-14T15:14:41.851725+0000","last_deep_scrub":"85'16","last_deep_scrub_stamp":"2024-04-14T15:10:47.456169+0000","last_clean_scrub_stamp":"2024-04-14T15:14:41.851725+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:29.106738+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":550,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.925799+0000","last_change":"2024-04-14T15:14:36.968608+0000","last_active":"2024-04-14T15:16:28.925799+0000","last_peered":"2024-04-14T15:16:28.925799+0000","last_clean":"2024-04-14T15:16:28.925799+0000","last_became_active":"2024-04-14T15:09:43.018601+0000","last_became_peered":"2024-04-14T15:09:43.018601+0000","last_unstale":"2024-04-14T15:16:28.925799+0000","last_undegraded":"2024-04-14T15:16:28.925799+0000","last_fullsized":"2024-04-14T15:16:28.925799+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:36.968530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:14:36.968530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:29.785613+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":100,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.882045+0000","last_change":"2024-04-14T15:15:44.887157+0000","last_active":"2024-04-14T15:16:28.882045+0000","last_peered":"2024-04-14T15:16:28.882045+0000","last_clean":"2024-04-14T15:16:28.882045+0000","last_became_active":"2024-04-14T15:15:44.886834+0000","last_became_peered":"2024-04-14T15:15:44.886834+0000","last_unstale":"2024-04-14T15:16:28.882045+0000","last_undegraded":"2024-04-14T15:16:28.882045+0000","last_fullsized":"2024-04-14T15:16:28.882045+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:43.288325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:14:43.288325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:29.106754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":214,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.890195+0000","last_change":"2024-04-14T15:15:21.888865+0000","last_active":"2024-04-14T15:16:28.890195+0000","last_peered":"2024-04-14T15:16:28.890195+0000","last_clean":"2024-04-14T15:16:28.890195+0000","last_became_active":"2024-04-14T15:15:21.888587+0000","last_became_peered":"2024-04-14T15:15:21.888587+0000","last_unstale":"2024-04-14T15:16:28.890195+0000","last_undegraded":"2024-04-14T15:16:28.890195+0000","last_fullsized":"2024-04-14T15:16:28.890195+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:47.061794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:14:47.061794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:28.969046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"91'130","reported_seq":600,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.891315+0000","last_change":"2024-04-14T15:14:40.097282+0000","last_active":"2024-04-14T15:16:28.891315+0000","last_peered":"2024-04-14T15:16:28.891315+0000","last_clean":"2024-04-14T15:16:28.891315+0000","last_became_active":"2024-04-14T15:14:17.863503+0000","last_became_peered":"2024-04-14T15:14:17.863503+0000","last_unstale":"2024-04-14T15:16:28.891315+0000","last_undegraded":"2024-04-14T15:16:28.891315+0000","last_fullsized":"2024-04-14T15:16:28.891315+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:14:40.097172+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:14:40.097172+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:28.968941+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":446,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.881930+0000","last_change":"2024-04-14T15:14:59.876005+0000","last_active":"2024-04-14T15:16:28.881930+0000","last_peered":"2024-04-14T15:16:28.881930+0000","last_clean":"2024-04-14T15:16:28.881930+0000","last_became_active":"2024-04-14T15:12:56.427276+0000","last_became_peered":"2024-04-14T15:12:56.427276+0000","last_unstale":"2024-04-14T15:16:28.881930+0000","last_undegraded":"2024-04-14T15:16:28.881930+0000","last_fullsized":"2024-04-14T15:16:28.881930+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:59.875917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:14:59.875917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:29.106803+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"255'10","reported_seq":277,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.888595+0000","last_change":"2024-04-14T15:16:17.868542+0000","last_active":"2024-04-14T15:16:28.888595+0000","last_peered":"2024-04-14T15:16:28.888595+0000","last_clean":"2024-04-14T15:16:28.888595+0000","last_became_active":"2024-04-14T15:15:21.873747+0000","last_became_peered":"2024-04-14T15:15:21.873747+0000","last_unstale":"2024-04-14T15:16:28.888595+0000","last_undegraded":"2024-04-14T15:16:28.888595+0000","last_fullsized":"2024-04-14T15:16:28.888595+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'10","last_scrub_stamp":"2024-04-14T15:16:17.868428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:17.868428+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:37.989641+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"72'14","reported_seq":567,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.881045+0000","last_change":"2024-04-14T15:15:38.962791+0000","last_active":"2024-04-14T15:16:28.881045+0000","last_peered":"2024-04-14T15:16:28.881045+0000","last_clean":"2024-04-14T15:16:28.881045+0000","last_became_active":"2024-04-14T15:09:45.025673+0000","last_became_peered":"2024-04-14T15:09:45.025673+0000","last_unstale":"2024-04-14T15:16:28.881045+0000","last_undegraded":"2024-04-14T15:16:28.881045+0000","last_fullsized":"2024-04-14T15:16:28.881045+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'14","last_scrub_stamp":"2024-04-14T15:15:38.962680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:15:38.962680+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:54.305088+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":550,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.930013+0000","last_change":"2024-04-14T15:15:39.123305+0000","last_active":"2024-04-14T15:16:28.930013+0000","last_peered":"2024-04-14T15:16:28.930013+0000","last_clean":"2024-04-14T15:16:28.930013+0000","last_became_active":"2024-04-14T15:09:43.020503+0000","last_became_peered":"2024-04-14T15:09:43.020503+0000","last_unstale":"2024-04-14T15:16:28.930013+0000","last_undegraded":"2024-04-14T15:16:28.930013+0000","last_fullsized":"2024-04-14T15:16:28.930013+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:39.123227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:15:39.123227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:07.799320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":530,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.926332+0000","last_change":"2024-04-14T15:13:25.116624+0000","last_active":"2024-04-14T15:16:28.926332+0000","last_peered":"2024-04-14T15:16:28.926332+0000","last_clean":"2024-04-14T15:16:28.926332+0000","last_became_active":"2024-04-14T15:09:49.129610+0000","last_became_peered":"2024-04-14T15:09:49.129610+0000","last_unstale":"2024-04-14T15:16:28.926332+0000","last_undegraded":"2024-04-14T15:16:28.926332+0000","last_fullsized":"2024-04-14T15:16:28.926332+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:25.116541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:13:25.116541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:29.785360+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"91'130","reported_seq":600,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.881841+0000","last_change":"2024-04-14T15:14:38.832437+0000","last_active":"2024-04-14T15:16:28.881841+0000","last_peered":"2024-04-14T15:16:28.881841+0000","last_clean":"2024-04-14T15:16:28.881841+0000","last_became_active":"2024-04-14T15:14:17.871922+0000","last_became_peered":"2024-04-14T15:14:17.871922+0000","last_unstale":"2024-04-14T15:16:28.881841+0000","last_undegraded":"2024-04-14T15:16:28.881841+0000","last_fullsized":"2024-04-14T15:16:28.881841+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:14:38.832326+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:14:38.832326+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:29.106787+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":446,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.927668+0000","last_change":"2024-04-14T15:14:52.954282+0000","last_active":"2024-04-14T15:16:28.927668+0000","last_peered":"2024-04-14T15:16:28.927668+0000","last_clean":"2024-04-14T15:16:28.927668+0000","last_became_active":"2024-04-14T15:12:56.355254+0000","last_became_peered":"2024-04-14T15:12:56.355254+0000","last_unstale":"2024-04-14T15:16:28.927668+0000","last_undegraded":"2024-04-14T15:16:28.927668+0000","last_fullsized":"2024-04-14T15:16:28.927668+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:52.954163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:14:52.954163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:29.785753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":542,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.881281+0000","last_change":"2024-04-14T15:16:05.917909+0000","last_active":"2024-04-14T15:16:28.881281+0000","last_peered":"2024-04-14T15:16:28.881281+0000","last_clean":"2024-04-14T15:16:28.881281+0000","last_became_active":"2024-04-14T15:09:47.044610+0000","last_became_peered":"2024-04-14T15:09:47.044610+0000","last_unstale":"2024-04-14T15:16:28.881281+0000","last_undegraded":"2024-04-14T15:16:28.881281+0000","last_fullsized":"2024-04-14T15:16:28.881281+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:05.917804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:05.917804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:15.789677+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"72'18","reported_seq":263,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.888493+0000","last_change":"2024-04-14T15:16:15.913377+0000","last_active":"2024-04-14T15:16:28.888493+0000","last_peered":"2024-04-14T15:16:28.888493+0000","last_clean":"2024-04-14T15:16:28.888493+0000","last_became_active":"2024-04-14T15:15:21.880565+0000","last_became_peered":"2024-04-14T15:15:21.880565+0000","last_unstale":"2024-04-14T15:16:28.888493+0000","last_undegraded":"2024-04-14T15:16:28.888493+0000","last_fullsized":"2024-04-14T15:16:28.888493+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'18","last_scrub_stamp":"2024-04-14T15:16:15.913273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:16:15.913273+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:36.172835+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'2","reported_seq":544,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.891778+0000","last_change":"2024-04-14T15:14:38.291600+0000","last_active":"2024-04-14T15:16:28.891778+0000","last_peered":"2024-04-14T15:16:28.891778+0000","last_clean":"2024-04-14T15:16:28.891778+0000","last_became_active":"2024-04-14T15:09:43.018478+0000","last_became_peered":"2024-04-14T15:09:43.018478+0000","last_unstale":"2024-04-14T15:16:28.891778+0000","last_undegraded":"2024-04-14T15:16:28.891778+0000","last_fullsized":"2024-04-14T15:16:28.891778+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'2","last_scrub_stamp":"2024-04-14T15:14:38.291510+0000","last_deep_scrub":"91'2","last_deep_scrub_stamp":"2024-04-14T15:11:37.974770+0000","last_clean_scrub_stamp":"2024-04-14T15:14:38.291510+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":538,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.881625+0000","last_change":"2024-04-14T15:16:14.017818+0000","last_active":"2024-04-14T15:16:28.881625+0000","last_peered":"2024-04-14T15:16:28.881625+0000","last_clean":"2024-04-14T15:16:28.881625+0000","last_became_active":"2024-04-14T15:09:49.065684+0000","last_became_peered":"2024-04-14T15:09:49.065684+0000","last_unstale":"2024-04-14T15:16:28.881625+0000","last_undegraded":"2024-04-14T15:16:28.881625+0000","last_fullsized":"2024-04-14T15:16:28.881625+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:14.017729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:14.017729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:15.823157+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"91'130","reported_seq":607,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.930319+0000","last_change":"2024-04-14T15:14:39.146276+0000","last_active":"2024-04-14T15:16:28.930319+0000","last_peered":"2024-04-14T15:16:28.930319+0000","last_clean":"2024-04-14T15:16:28.930319+0000","last_became_active":"2024-04-14T15:14:17.853143+0000","last_became_peered":"2024-04-14T15:14:17.853143+0000","last_unstale":"2024-04-14T15:16:28.930319+0000","last_undegraded":"2024-04-14T15:16:28.930319+0000","last_fullsized":"2024-04-14T15:16:28.930319+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:14:39.146174+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:14:39.146174+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:29.834871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":446,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.886194+0000","last_change":"2024-04-14T15:15:31.403165+0000","last_active":"2024-04-14T15:16:28.886194+0000","last_peered":"2024-04-14T15:16:28.886194+0000","last_clean":"2024-04-14T15:16:28.886194+0000","last_became_active":"2024-04-14T15:12:56.402023+0000","last_became_peered":"2024-04-14T15:12:56.402023+0000","last_unstale":"2024-04-14T15:16:28.886194+0000","last_undegraded":"2024-04-14T15:16:28.886194+0000","last_fullsized":"2024-04-14T15:16:28.886194+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:31.403086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:31.403086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:40.387194+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":542,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.929481+0000","last_change":"2024-04-14T15:14:14.070145+0000","last_active":"2024-04-14T15:16:28.929481+0000","last_peered":"2024-04-14T15:16:28.929481+0000","last_clean":"2024-04-14T15:16:28.929481+0000","last_became_active":"2024-04-14T15:09:47.048427+0000","last_became_peered":"2024-04-14T15:09:47.048427+0000","last_unstale":"2024-04-14T15:16:28.929481+0000","last_undegraded":"2024-04-14T15:16:28.929481+0000","last_fullsized":"2024-04-14T15:16:28.929481+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:14.070046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:14:14.070046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:29.834813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"102'27","reported_seq":590,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.928438+0000","last_change":"2024-04-14T15:14:12.391631+0000","last_active":"2024-04-14T15:16:28.928438+0000","last_peered":"2024-04-14T15:16:28.928438+0000","last_clean":"2024-04-14T15:16:28.928438+0000","last_became_active":"2024-04-14T15:09:45.179182+0000","last_became_peered":"2024-04-14T15:09:45.179182+0000","last_unstale":"2024-04-14T15:16:28.928438+0000","last_undegraded":"2024-04-14T15:16:28.928438+0000","last_fullsized":"2024-04-14T15:16:28.928438+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'27","last_scrub_stamp":"2024-04-14T15:14:12.391537+0000","last_deep_scrub":"102'27","last_deep_scrub_stamp":"2024-04-14T15:12:30.928216+0000","last_clean_scrub_stamp":"2024-04-14T15:14:12.391537+0000","objects_scrubbed":8,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":550,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.928700+0000","last_change":"2024-04-14T15:16:18.774542+0000","last_active":"2024-04-14T15:16:28.928700+0000","last_peered":"2024-04-14T15:16:28.928700+0000","last_clean":"2024-04-14T15:16:28.928700+0000","last_became_active":"2024-04-14T15:09:43.021006+0000","last_became_peered":"2024-04-14T15:09:43.021006+0000","last_unstale":"2024-04-14T15:16:28.928700+0000","last_undegraded":"2024-04-14T15:16:28.928700+0000","last_fullsized":"2024-04-14T15:16:28.928700+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:18.774463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:18.774463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:22.694554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":530,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.887800+0000","last_change":"2024-04-14T15:14:06.454685+0000","last_active":"2024-04-14T15:16:28.887800+0000","last_peered":"2024-04-14T15:16:28.887800+0000","last_clean":"2024-04-14T15:16:28.887800+0000","last_became_active":"2024-04-14T15:09:49.128136+0000","last_became_peered":"2024-04-14T15:09:49.128136+0000","last_unstale":"2024-04-14T15:16:28.887800+0000","last_undegraded":"2024-04-14T15:16:28.887800+0000","last_fullsized":"2024-04-14T15:16:28.887800+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:06.454602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:14:06.454602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"91'130","reported_seq":600,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.886098+0000","last_change":"2024-04-14T15:14:42.213499+0000","last_active":"2024-04-14T15:16:28.886098+0000","last_peered":"2024-04-14T15:16:28.886098+0000","last_clean":"2024-04-14T15:16:28.886098+0000","last_became_active":"2024-04-14T15:14:17.856694+0000","last_became_peered":"2024-04-14T15:14:17.856694+0000","last_unstale":"2024-04-14T15:16:28.886098+0000","last_undegraded":"2024-04-14T15:16:28.886098+0000","last_fullsized":"2024-04-14T15:16:28.886098+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:14:42.213346+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:14:42.213346+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":154,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.886891+0000","last_change":"2024-04-14T15:15:31.812967+0000","last_active":"2024-04-14T15:16:28.886891+0000","last_peered":"2024-04-14T15:16:28.886891+0000","last_clean":"2024-04-14T15:16:28.886891+0000","last_became_active":"2024-04-14T15:15:21.887769+0000","last_became_peered":"2024-04-14T15:15:21.887769+0000","last_unstale":"2024-04-14T15:16:28.886891+0000","last_undegraded":"2024-04-14T15:16:28.886891+0000","last_fullsized":"2024-04-14T15:16:28.886891+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:31.812888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:31.812888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:40.709024+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":100,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.889889+0000","last_change":"2024-04-14T15:15:44.884167+0000","last_active":"2024-04-14T15:16:28.889889+0000","last_peered":"2024-04-14T15:16:28.889889+0000","last_clean":"2024-04-14T15:16:28.889889+0000","last_became_active":"2024-04-14T15:15:44.883694+0000","last_became_peered":"2024-04-14T15:15:44.883694+0000","last_unstale":"2024-04-14T15:16:28.889889+0000","last_undegraded":"2024-04-14T15:16:28.889889+0000","last_fullsized":"2024-04-14T15:16:28.889889+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:12.743355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:14:12.743355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"182'18","reported_seq":570,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.931540+0000","last_change":"2024-04-14T15:14:10.384002+0000","last_active":"2024-04-14T15:16:28.931540+0000","last_peered":"2024-04-14T15:16:28.931540+0000","last_clean":"2024-04-14T15:16:28.931540+0000","last_became_active":"2024-04-14T15:09:45.037231+0000","last_became_peered":"2024-04-14T15:09:45.037231+0000","last_unstale":"2024-04-14T15:16:28.931540+0000","last_undegraded":"2024-04-14T15:16:28.931540+0000","last_fullsized":"2024-04-14T15:16:28.931540+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'18","last_scrub_stamp":"2024-04-14T15:14:10.383906+0000","last_deep_scrub":"182'18","last_deep_scrub_stamp":"2024-04-14T15:14:10.383906+0000","last_clean_scrub_stamp":"2024-04-14T15:14:10.383906+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":550,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.925978+0000","last_change":"2024-04-14T15:14:44.864919+0000","last_active":"2024-04-14T15:16:28.925978+0000","last_peered":"2024-04-14T15:16:28.925978+0000","last_clean":"2024-04-14T15:16:28.925978+0000","last_became_active":"2024-04-14T15:09:43.025095+0000","last_became_peered":"2024-04-14T15:09:43.025095+0000","last_unstale":"2024-04-14T15:16:28.925978+0000","last_undegraded":"2024-04-14T15:16:28.925978+0000","last_fullsized":"2024-04-14T15:16:28.925978+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:44.864789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:14:44.864789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:29.785720+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":557,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.926431+0000","last_change":"2024-04-14T15:14:41.295132+0000","last_active":"2024-04-14T15:16:28.926431+0000","last_peered":"2024-04-14T15:16:28.926431+0000","last_clean":"2024-04-14T15:16:28.926431+0000","last_became_active":"2024-04-14T15:09:43.018509+0000","last_became_peered":"2024-04-14T15:09:43.018509+0000","last_unstale":"2024-04-14T15:16:28.926431+0000","last_undegraded":"2024-04-14T15:16:28.926431+0000","last_fullsized":"2024-04-14T15:16:28.926431+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:41.295049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:14:41.295049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0018592680000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"72'1","reported_seq":531,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.886510+0000","last_change":"2024-04-14T15:14:05.500796+0000","last_active":"2024-04-14T15:16:28.886510+0000","last_peered":"2024-04-14T15:16:28.886510+0000","last_clean":"2024-04-14T15:16:28.886510+0000","last_became_active":"2024-04-14T15:09:49.065243+0000","last_became_peered":"2024-04-14T15:09:49.065243+0000","last_unstale":"2024-04-14T15:16:28.886510+0000","last_undegraded":"2024-04-14T15:16:28.886510+0000","last_fullsized":"2024-04-14T15:16:28.886510+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-04-14T15:14:05.500685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:14:05.500685+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":146,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.888731+0000","last_change":"2024-04-14T15:15:21.890855+0000","last_active":"2024-04-14T15:16:28.888731+0000","last_peered":"2024-04-14T15:16:28.888731+0000","last_clean":"2024-04-14T15:16:28.888731+0000","last_became_active":"2024-04-14T15:15:21.890115+0000","last_became_peered":"2024-04-14T15:15:21.890115+0000","last_unstale":"2024-04-14T15:16:28.888731+0000","last_undegraded":"2024-04-14T15:16:28.888731+0000","last_fullsized":"2024-04-14T15:16:28.888731+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:45.148342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:14:45.148342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:28.969133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":534,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.926857+0000","last_change":"2024-04-14T15:13:31.226741+0000","last_active":"2024-04-14T15:16:28.926857+0000","last_peered":"2024-04-14T15:16:28.926857+0000","last_clean":"2024-04-14T15:16:28.926857+0000","last_became_active":"2024-04-14T15:09:47.047201+0000","last_became_peered":"2024-04-14T15:09:47.047201+0000","last_unstale":"2024-04-14T15:16:28.926857+0000","last_undegraded":"2024-04-14T15:16:28.926857+0000","last_fullsized":"2024-04-14T15:16:28.926857+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:31.226659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:13:31.226659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:29.785424+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"90'24","reported_seq":658,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.930994+0000","last_change":"2024-04-14T15:16:19.842719+0000","last_active":"2024-04-14T15:16:28.930994+0000","last_peered":"2024-04-14T15:16:28.930994+0000","last_clean":"2024-04-14T15:16:28.930994+0000","last_became_active":"2024-04-14T15:09:45.179978+0000","last_became_peered":"2024-04-14T15:09:45.179978+0000","last_unstale":"2024-04-14T15:16:28.930994+0000","last_undegraded":"2024-04-14T15:16:28.930994+0000","last_fullsized":"2024-04-14T15:16:28.930994+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'24","last_scrub_stamp":"2024-04-14T15:16:19.842614+0000","last_deep_scrub":"90'24","last_deep_scrub_stamp":"2024-04-14T15:11:25.120692+0000","last_clean_scrub_stamp":"2024-04-14T15:16:19.842614+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:40.225520+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"91'130","reported_seq":811,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.925885+0000","last_change":"2024-04-14T15:14:13.215896+0000","last_active":"2024-04-14T15:16:28.925885+0000","last_peered":"2024-04-14T15:16:28.925885+0000","last_clean":"2024-04-14T15:16:28.925885+0000","last_became_active":"2024-04-14T15:14:12.858953+0000","last_became_peered":"2024-04-14T15:14:12.858953+0000","last_unstale":"2024-04-14T15:16:28.925885+0000","last_undegraded":"2024-04-14T15:16:28.925885+0000","last_fullsized":"2024-04-14T15:16:28.925885+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:14:13.215812+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:14:13.215812+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:29.785545+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":108,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.927310+0000","last_change":"2024-04-14T15:16:10.648565+0000","last_active":"2024-04-14T15:16:28.927310+0000","last_peered":"2024-04-14T15:16:28.927310+0000","last_clean":"2024-04-14T15:16:28.927310+0000","last_became_active":"2024-04-14T15:15:44.885315+0000","last_became_peered":"2024-04-14T15:15:44.885315+0000","last_unstale":"2024-04-14T15:16:28.927310+0000","last_undegraded":"2024-04-14T15:16:28.927310+0000","last_fullsized":"2024-04-14T15:16:28.927310+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:10.648486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:47.627305+0000","last_clean_scrub_stamp":"2024-04-14T15:16:10.648486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:34.882894+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"69'2","reported_seq":236,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.888791+0000","last_change":"2024-04-14T15:15:21.890189+0000","last_active":"2024-04-14T15:16:28.888791+0000","last_peered":"2024-04-14T15:16:28.888791+0000","last_clean":"2024-04-14T15:16:28.888791+0000","last_became_active":"2024-04-14T15:15:21.889851+0000","last_became_peered":"2024-04-14T15:15:21.889851+0000","last_unstale":"2024-04-14T15:16:28.888791+0000","last_undegraded":"2024-04-14T15:16:28.888791+0000","last_fullsized":"2024-04-14T15:16:28.888791+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-04-14T15:13:32.804904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:13:32.804904+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:28.968620+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0058046169999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":101,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.891720+0000","last_change":"2024-04-14T15:15:44.888223+0000","last_active":"2024-04-14T15:16:28.891720+0000","last_peered":"2024-04-14T15:16:28.891720+0000","last_clean":"2024-04-14T15:16:28.891720+0000","last_became_active":"2024-04-14T15:15:44.887934+0000","last_became_peered":"2024-04-14T15:15:44.887934+0000","last_unstale":"2024-04-14T15:16:28.891720+0000","last_undegraded":"2024-04-14T15:16:28.891720+0000","last_fullsized":"2024-04-14T15:16:28.891720+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:57.756721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:13:57.756721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:28.968683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"91'130","reported_seq":819,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.927917+0000","last_change":"2024-04-14T15:14:40.906412+0000","last_active":"2024-04-14T15:16:28.927917+0000","last_peered":"2024-04-14T15:16:28.927917+0000","last_clean":"2024-04-14T15:16:28.927917+0000","last_became_active":"2024-04-14T15:14:17.858029+0000","last_became_peered":"2024-04-14T15:14:17.858029+0000","last_unstale":"2024-04-14T15:16:28.927917+0000","last_undegraded":"2024-04-14T15:16:28.927917+0000","last_fullsized":"2024-04-14T15:16:28.927917+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:14:40.906329+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:14:40.906329+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:29.785686+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":146,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.888604+0000","last_change":"2024-04-14T15:15:21.890274+0000","last_active":"2024-04-14T15:16:28.888604+0000","last_peered":"2024-04-14T15:16:28.888604+0000","last_clean":"2024-04-14T15:16:28.888604+0000","last_became_active":"2024-04-14T15:15:21.889970+0000","last_became_peered":"2024-04-14T15:15:21.889970+0000","last_unstale":"2024-04-14T15:16:28.888604+0000","last_undegraded":"2024-04-14T15:16:28.888604+0000","last_fullsized":"2024-04-14T15:16:28.888604+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:54.156421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:14:54.156421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:28.969074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":542,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.927332+0000","last_change":"2024-04-14T15:14:45.465021+0000","last_active":"2024-04-14T15:16:28.927332+0000","last_peered":"2024-04-14T15:16:28.927332+0000","last_clean":"2024-04-14T15:16:28.927332+0000","last_became_active":"2024-04-14T15:09:47.049457+0000","last_became_peered":"2024-04-14T15:09:47.049457+0000","last_unstale":"2024-04-14T15:16:28.927332+0000","last_undegraded":"2024-04-14T15:16:28.927332+0000","last_fullsized":"2024-04-14T15:16:28.927332+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:45.464940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:14:45.464940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"308'18","reported_seq":578,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.889735+0000","last_change":"2024-04-14T15:16:16.054128+0000","last_active":"2024-04-14T15:16:28.889735+0000","last_peered":"2024-04-14T15:16:28.889735+0000","last_clean":"2024-04-14T15:16:28.889735+0000","last_became_active":"2024-04-14T15:09:45.033883+0000","last_became_peered":"2024-04-14T15:09:45.033883+0000","last_unstale":"2024-04-14T15:16:28.889735+0000","last_undegraded":"2024-04-14T15:16:28.889735+0000","last_fullsized":"2024-04-14T15:16:28.889735+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"308'18","last_scrub_stamp":"2024-04-14T15:16:16.054032+0000","last_deep_scrub":"308'18","last_deep_scrub_stamp":"2024-04-14T15:16:16.054032+0000","last_clean_scrub_stamp":"2024-04-14T15:16:16.054032+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:42.330440+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":550,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.927026+0000","last_change":"2024-04-14T15:14:05.378718+0000","last_active":"2024-04-14T15:16:28.927026+0000","last_peered":"2024-04-14T15:16:28.927026+0000","last_clean":"2024-04-14T15:16:28.927026+0000","last_became_active":"2024-04-14T15:09:43.021188+0000","last_became_peered":"2024-04-14T15:09:43.021188+0000","last_unstale":"2024-04-14T15:16:28.927026+0000","last_undegraded":"2024-04-14T15:16:28.927026+0000","last_fullsized":"2024-04-14T15:16:28.927026+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:05.378634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:14:05.378634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"73'5","reported_seq":959,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.886977+0000","last_change":"2024-04-14T15:15:28.328150+0000","last_active":"2024-04-14T15:16:28.886977+0000","last_peered":"2024-04-14T15:16:28.886977+0000","last_clean":"2024-04-14T15:16:28.886977+0000","last_became_active":"2024-04-14T15:09:43.016544+0000","last_became_peered":"2024-04-14T15:09:43.016544+0000","last_unstale":"2024-04-14T15:16:28.886977+0000","last_undegraded":"2024-04-14T15:16:28.886977+0000","last_fullsized":"2024-04-14T15:16:28.886977+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-04-14T15:15:28.328050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:15:28.328050+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:35.661053+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00056542099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":399,"num_read_kb":394,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":538,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.928794+0000","last_change":"2024-04-14T15:16:20.827681+0000","last_active":"2024-04-14T15:16:28.928794+0000","last_peered":"2024-04-14T15:16:28.928794+0000","last_clean":"2024-04-14T15:16:28.928794+0000","last_became_active":"2024-04-14T15:09:49.130307+0000","last_became_peered":"2024-04-14T15:09:49.130307+0000","last_unstale":"2024-04-14T15:16:28.928794+0000","last_undegraded":"2024-04-14T15:16:28.928794+0000","last_fullsized":"2024-04-14T15:16:28.928794+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:20.827589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:20.827589+0000","last_clean_scrub_stamp":"2024-04-14T15:16:20.827589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:26.908535+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"91'130","reported_seq":431,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.926181+0000","last_change":"2024-04-14T15:15:45.362962+0000","last_active":"2024-04-14T15:16:28.926181+0000","last_peered":"2024-04-14T15:16:28.926181+0000","last_clean":"2024-04-14T15:16:28.926181+0000","last_became_active":"2024-04-14T15:15:44.884623+0000","last_became_peered":"2024-04-14T15:15:44.884623+0000","last_unstale":"2024-04-14T15:16:28.926181+0000","last_undegraded":"2024-04-14T15:16:28.926181+0000","last_fullsized":"2024-04-14T15:16:28.926181+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:15:45.362860+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:15:45.362860+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:54.355120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":154,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.889657+0000","last_change":"2024-04-14T15:15:32.839576+0000","last_active":"2024-04-14T15:16:28.889657+0000","last_peered":"2024-04-14T15:16:28.889657+0000","last_clean":"2024-04-14T15:16:28.889657+0000","last_became_active":"2024-04-14T15:15:21.883717+0000","last_became_peered":"2024-04-14T15:15:21.883717+0000","last_unstale":"2024-04-14T15:16:28.889657+0000","last_undegraded":"2024-04-14T15:16:28.889657+0000","last_fullsized":"2024-04-14T15:16:28.889657+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:32.839491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:32.839491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:59.110217+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":534,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.929214+0000","last_change":"2024-04-14T15:13:35.587741+0000","last_active":"2024-04-14T15:16:28.929214+0000","last_peered":"2024-04-14T15:16:28.929214+0000","last_clean":"2024-04-14T15:16:28.929214+0000","last_became_active":"2024-04-14T15:09:47.040185+0000","last_became_peered":"2024-04-14T15:09:47.040185+0000","last_unstale":"2024-04-14T15:16:28.929214+0000","last_undegraded":"2024-04-14T15:16:28.929214+0000","last_fullsized":"2024-04-14T15:16:28.929214+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:35.587624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:35.587624+0000","last_clean_scrub_stamp":"2024-04-14T15:13:35.587624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"72'28","reported_seq":668,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.886095+0000","last_change":"2024-04-14T15:13:35.350892+0000","last_active":"2024-04-14T15:16:28.886095+0000","last_peered":"2024-04-14T15:16:28.886095+0000","last_clean":"2024-04-14T15:16:28.886095+0000","last_became_active":"2024-04-14T15:09:45.034168+0000","last_became_peered":"2024-04-14T15:09:45.034168+0000","last_unstale":"2024-04-14T15:16:28.886095+0000","last_undegraded":"2024-04-14T15:16:28.886095+0000","last_fullsized":"2024-04-14T15:16:28.886095+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'28","last_scrub_stamp":"2024-04-14T15:13:35.350790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:13:35.350790+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":108,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.887529+0000","last_change":"2024-04-14T15:16:19.101437+0000","last_active":"2024-04-14T15:16:28.887529+0000","last_peered":"2024-04-14T15:16:28.887529+0000","last_clean":"2024-04-14T15:16:28.887529+0000","last_became_active":"2024-04-14T15:15:44.885315+0000","last_became_peered":"2024-04-14T15:15:44.885315+0000","last_unstale":"2024-04-14T15:16:28.887529+0000","last_undegraded":"2024-04-14T15:16:28.887529+0000","last_fullsized":"2024-04-14T15:16:28.887529+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:19.101368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:19.101368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:23.639538+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":206,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.890877+0000","last_change":"2024-04-14T15:15:21.875075+0000","last_active":"2024-04-14T15:16:28.890877+0000","last_peered":"2024-04-14T15:16:28.890877+0000","last_clean":"2024-04-14T15:16:28.890877+0000","last_became_active":"2024-04-14T15:15:21.874742+0000","last_became_peered":"2024-04-14T15:15:21.874742+0000","last_unstale":"2024-04-14T15:16:28.890877+0000","last_undegraded":"2024-04-14T15:16:28.890877+0000","last_fullsized":"2024-04-14T15:16:28.890877+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:14.594652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:14:14.594652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:28.968877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"91'130","reported_seq":607,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.929773+0000","last_change":"2024-04-14T15:14:40.134739+0000","last_active":"2024-04-14T15:16:28.929773+0000","last_peered":"2024-04-14T15:16:28.929773+0000","last_clean":"2024-04-14T15:16:28.929773+0000","last_became_active":"2024-04-14T15:14:17.859910+0000","last_became_peered":"2024-04-14T15:14:17.859910+0000","last_unstale":"2024-04-14T15:16:28.929773+0000","last_undegraded":"2024-04-14T15:16:28.929773+0000","last_fullsized":"2024-04-14T15:16:28.929773+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:14:40.134645+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:14:40.134645+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:29.834904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":446,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.930023+0000","last_change":"2024-04-14T15:14:49.421861+0000","last_active":"2024-04-14T15:16:28.930023+0000","last_peered":"2024-04-14T15:16:28.930023+0000","last_clean":"2024-04-14T15:16:28.930023+0000","last_became_active":"2024-04-14T15:12:56.350703+0000","last_became_peered":"2024-04-14T15:12:56.350703+0000","last_unstale":"2024-04-14T15:16:28.930023+0000","last_undegraded":"2024-04-14T15:16:28.930023+0000","last_fullsized":"2024-04-14T15:16:28.930023+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:49.421768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:14:49.421768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"72'8","reported_seq":632,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.928221+0000","last_change":"2024-04-14T15:14:03.352331+0000","last_active":"2024-04-14T15:16:28.928221+0000","last_peered":"2024-04-14T15:16:28.928221+0000","last_clean":"2024-04-14T15:16:28.928221+0000","last_became_active":"2024-04-14T15:09:47.047699+0000","last_became_peered":"2024-04-14T15:09:47.047699+0000","last_unstale":"2024-04-14T15:16:28.928221+0000","last_undegraded":"2024-04-14T15:16:28.928221+0000","last_fullsized":"2024-04-14T15:16:28.928221+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:14:03.352195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.352195+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"72'10","reported_seq":540,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.927134+0000","last_change":"2024-04-14T15:15:41.610385+0000","last_active":"2024-04-14T15:16:28.927134+0000","last_peered":"2024-04-14T15:16:28.927134+0000","last_clean":"2024-04-14T15:16:28.927134+0000","last_became_active":"2024-04-14T15:10:07.764042+0000","last_became_peered":"2024-04-14T15:10:07.764042+0000","last_unstale":"2024-04-14T15:16:28.927134+0000","last_undegraded":"2024-04-14T15:16:28.927134+0000","last_fullsized":"2024-04-14T15:16:28.927134+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-04-14T15:15:41.610279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:15:41.610279+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:08.216750+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":542,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.886003+0000","last_change":"2024-04-14T15:14:14.462940+0000","last_active":"2024-04-14T15:16:28.886003+0000","last_peered":"2024-04-14T15:16:28.886003+0000","last_clean":"2024-04-14T15:16:28.886003+0000","last_became_active":"2024-04-14T15:09:43.022597+0000","last_became_peered":"2024-04-14T15:09:43.022597+0000","last_unstale":"2024-04-14T15:16:28.886003+0000","last_undegraded":"2024-04-14T15:16:28.886003+0000","last_fullsized":"2024-04-14T15:16:28.886003+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:14.462847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:14:14.462847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":538,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.926326+0000","last_change":"2024-04-14T15:16:17.405931+0000","last_active":"2024-04-14T15:16:28.926326+0000","last_peered":"2024-04-14T15:16:28.926326+0000","last_clean":"2024-04-14T15:16:28.926326+0000","last_became_active":"2024-04-14T15:09:49.066648+0000","last_became_peered":"2024-04-14T15:09:49.066648+0000","last_unstale":"2024-04-14T15:16:28.926326+0000","last_undegraded":"2024-04-14T15:16:28.926326+0000","last_fullsized":"2024-04-14T15:16:28.926326+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:17.405850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:17.405850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:22.263168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":446,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.931366+0000","last_change":"2024-04-14T15:15:36.403919+0000","last_active":"2024-04-14T15:16:28.931366+0000","last_peered":"2024-04-14T15:16:28.931366+0000","last_clean":"2024-04-14T15:16:28.931366+0000","last_became_active":"2024-04-14T15:12:56.563074+0000","last_became_peered":"2024-04-14T15:12:56.563074+0000","last_unstale":"2024-04-14T15:16:28.931366+0000","last_undegraded":"2024-04-14T15:16:28.931366+0000","last_fullsized":"2024-04-14T15:16:28.931366+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:36.403843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:36.403843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:36.624784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":542,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.926978+0000","last_change":"2024-04-14T15:15:33.526229+0000","last_active":"2024-04-14T15:16:28.926978+0000","last_peered":"2024-04-14T15:16:28.926978+0000","last_clean":"2024-04-14T15:16:28.926978+0000","last_became_active":"2024-04-14T15:09:47.049005+0000","last_became_peered":"2024-04-14T15:09:47.049005+0000","last_unstale":"2024-04-14T15:16:28.926978+0000","last_undegraded":"2024-04-14T15:16:28.926978+0000","last_fullsized":"2024-04-14T15:16:28.926978+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:33.526146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:15:33.526146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:53.685917+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"183'17","reported_seq":582,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.881201+0000","last_change":"2024-04-14T15:16:17.997616+0000","last_active":"2024-04-14T15:16:28.881201+0000","last_peered":"2024-04-14T15:16:28.881201+0000","last_clean":"2024-04-14T15:16:28.881201+0000","last_became_active":"2024-04-14T15:09:45.032892+0000","last_became_peered":"2024-04-14T15:09:45.032892+0000","last_unstale":"2024-04-14T15:16:28.881201+0000","last_undegraded":"2024-04-14T15:16:28.881201+0000","last_fullsized":"2024-04-14T15:16:28.881201+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'17","last_scrub_stamp":"2024-04-14T15:16:17.997529+0000","last_deep_scrub":"183'17","last_deep_scrub_stamp":"2024-04-14T15:14:11.577574+0000","last_clean_scrub_stamp":"2024-04-14T15:16:17.997529+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:33.337431+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"65'3","reported_seq":568,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.882024+0000","last_change":"2024-04-14T15:16:12.030448+0000","last_active":"2024-04-14T15:16:28.882024+0000","last_peered":"2024-04-14T15:16:28.882024+0000","last_clean":"2024-04-14T15:16:28.882024+0000","last_became_active":"2024-04-14T15:09:43.020490+0000","last_became_peered":"2024-04-14T15:09:43.020490+0000","last_unstale":"2024-04-14T15:16:28.882024+0000","last_undegraded":"2024-04-14T15:16:28.882024+0000","last_fullsized":"2024-04-14T15:16:28.882024+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3","last_scrub_stamp":"2024-04-14T15:16:12.030363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:12.030363+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:32.451845+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":528,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":206,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.889479+0000","last_change":"2024-04-14T15:15:21.878709+0000","last_active":"2024-04-14T15:16:28.889479+0000","last_peered":"2024-04-14T15:16:28.889479+0000","last_clean":"2024-04-14T15:16:28.889479+0000","last_became_active":"2024-04-14T15:15:21.876410+0000","last_became_peered":"2024-04-14T15:15:21.876410+0000","last_unstale":"2024-04-14T15:16:28.889479+0000","last_undegraded":"2024-04-14T15:16:28.889479+0000","last_fullsized":"2024-04-14T15:16:28.889479+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:14.171627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:14:14.171627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:28.968713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":446,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.926320+0000","last_change":"2024-04-14T15:15:34.382014+0000","last_active":"2024-04-14T15:16:28.926320+0000","last_peered":"2024-04-14T15:16:28.926320+0000","last_clean":"2024-04-14T15:16:28.926320+0000","last_became_active":"2024-04-14T15:12:56.551177+0000","last_became_peered":"2024-04-14T15:12:56.551177+0000","last_unstale":"2024-04-14T15:16:28.926320+0000","last_undegraded":"2024-04-14T15:16:28.926320+0000","last_fullsized":"2024-04-14T15:16:28.926320+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:34.381865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:34.381865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:42.325831+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"72'9","reported_seq":553,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.931629+0000","last_change":"2024-04-14T15:13:37.516095+0000","last_active":"2024-04-14T15:16:28.931629+0000","last_peered":"2024-04-14T15:16:28.931629+0000","last_clean":"2024-04-14T15:16:28.931629+0000","last_became_active":"2024-04-14T15:09:45.180375+0000","last_became_peered":"2024-04-14T15:09:45.180375+0000","last_unstale":"2024-04-14T15:16:28.931629+0000","last_undegraded":"2024-04-14T15:16:28.931629+0000","last_fullsized":"2024-04-14T15:16:28.931629+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-04-14T15:13:37.515984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:13:37.515984+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":549,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.925350+0000","last_change":"2024-04-14T15:14:11.340243+0000","last_active":"2024-04-14T15:16:28.925350+0000","last_peered":"2024-04-14T15:16:28.925350+0000","last_clean":"2024-04-14T15:16:28.925350+0000","last_became_active":"2024-04-14T15:09:43.016788+0000","last_became_peered":"2024-04-14T15:09:43.016788+0000","last_unstale":"2024-04-14T15:16:28.925350+0000","last_undegraded":"2024-04-14T15:16:28.925350+0000","last_fullsized":"2024-04-14T15:16:28.925350+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:11.340164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:14:11.340164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":226,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.891636+0000","last_change":"2024-04-14T15:15:42.769452+0000","last_active":"2024-04-14T15:16:28.891636+0000","last_peered":"2024-04-14T15:16:28.891636+0000","last_clean":"2024-04-14T15:16:28.891636+0000","last_became_active":"2024-04-14T15:15:21.873580+0000","last_became_peered":"2024-04-14T15:15:21.873580+0000","last_unstale":"2024-04-14T15:16:28.891636+0000","last_undegraded":"2024-04-14T15:16:28.891636+0000","last_fullsized":"2024-04-14T15:16:28.891636+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:42.769375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:15:42.769375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:50.977618+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":537,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.929628+0000","last_change":"2024-04-14T15:14:46.461336+0000","last_active":"2024-04-14T15:16:28.929628+0000","last_peered":"2024-04-14T15:16:28.929628+0000","last_clean":"2024-04-14T15:16:28.929628+0000","last_became_active":"2024-04-14T15:09:49.064296+0000","last_became_peered":"2024-04-14T15:09:49.064296+0000","last_unstale":"2024-04-14T15:16:28.929628+0000","last_undegraded":"2024-04-14T15:16:28.929628+0000","last_fullsized":"2024-04-14T15:16:28.929628+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:46.461257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:14:46.461257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":446,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.925168+0000","last_change":"2024-04-14T15:14:43.531950+0000","last_active":"2024-04-14T15:16:28.925168+0000","last_peered":"2024-04-14T15:16:28.925168+0000","last_clean":"2024-04-14T15:16:28.925168+0000","last_became_active":"2024-04-14T15:12:56.340066+0000","last_became_peered":"2024-04-14T15:12:56.340066+0000","last_unstale":"2024-04-14T15:16:28.925168+0000","last_undegraded":"2024-04-14T15:16:28.925168+0000","last_fullsized":"2024-04-14T15:16:28.925168+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:43.531866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:14:43.531866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.17","version":"72'6","reported_seq":554,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.891541+0000","last_change":"2024-04-14T15:14:05.104367+0000","last_active":"2024-04-14T15:16:28.891541+0000","last_peered":"2024-04-14T15:16:28.891541+0000","last_clean":"2024-04-14T15:16:28.891541+0000","last_became_active":"2024-04-14T15:09:45.180037+0000","last_became_peered":"2024-04-14T15:09:45.180037+0000","last_unstale":"2024-04-14T15:16:28.891541+0000","last_undegraded":"2024-04-14T15:16:28.891541+0000","last_fullsized":"2024-04-14T15:16:28.891541+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'6","last_scrub_stamp":"2024-04-14T15:14:05.104204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:14:05.104204+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:28.968814+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":549,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.925899+0000","last_change":"2024-04-14T15:14:06.048273+0000","last_active":"2024-04-14T15:16:28.925899+0000","last_peered":"2024-04-14T15:16:28.925899+0000","last_clean":"2024-04-14T15:16:28.925899+0000","last_became_active":"2024-04-14T15:09:43.018987+0000","last_became_peered":"2024-04-14T15:09:43.018987+0000","last_unstale":"2024-04-14T15:16:28.925899+0000","last_undegraded":"2024-04-14T15:16:28.925899+0000","last_fullsized":"2024-04-14T15:16:28.925899+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:06.048196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:14:06.048196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:29.834766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":100,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.890733+0000","last_change":"2024-04-14T15:15:44.890140+0000","last_active":"2024-04-14T15:16:28.890733+0000","last_peered":"2024-04-14T15:16:28.890733+0000","last_clean":"2024-04-14T15:16:28.890733+0000","last_became_active":"2024-04-14T15:15:44.889786+0000","last_became_peered":"2024-04-14T15:15:44.889786+0000","last_unstale":"2024-04-14T15:16:28.890733+0000","last_undegraded":"2024-04-14T15:16:28.890733+0000","last_fullsized":"2024-04-14T15:16:28.890733+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:13.735024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:13.735024+0000","last_clean_scrub_stamp":"2024-04-14T15:14:13.735024+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:28.968906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":529,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.929068+0000","last_change":"2024-04-14T15:13:36.173906+0000","last_active":"2024-04-14T15:16:28.929068+0000","last_peered":"2024-04-14T15:16:28.929068+0000","last_clean":"2024-04-14T15:16:28.929068+0000","last_became_active":"2024-04-14T15:09:49.130438+0000","last_became_peered":"2024-04-14T15:09:49.130438+0000","last_unstale":"2024-04-14T15:16:28.929068+0000","last_undegraded":"2024-04-14T15:16:28.929068+0000","last_fullsized":"2024-04-14T15:16:28.929068+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:36.173813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:13:36.173813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:29.785455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":446,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.881326+0000","last_change":"2024-04-14T15:15:24.902253+0000","last_active":"2024-04-14T15:16:28.881326+0000","last_peered":"2024-04-14T15:16:28.881326+0000","last_clean":"2024-04-14T15:16:28.881326+0000","last_became_active":"2024-04-14T15:12:56.203577+0000","last_became_peered":"2024-04-14T15:12:56.203577+0000","last_unstale":"2024-04-14T15:16:28.881326+0000","last_undegraded":"2024-04-14T15:16:28.881326+0000","last_fullsized":"2024-04-14T15:16:28.881326+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:24.902134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:24.902134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:39.833144+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"72'10","reported_seq":245,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.890292+0000","last_change":"2024-04-14T15:15:30.806041+0000","last_active":"2024-04-14T15:16:28.890292+0000","last_peered":"2024-04-14T15:16:28.890292+0000","last_clean":"2024-04-14T15:16:28.890292+0000","last_became_active":"2024-04-14T15:15:21.877667+0000","last_became_peered":"2024-04-14T15:15:21.877667+0000","last_unstale":"2024-04-14T15:16:28.890292+0000","last_undegraded":"2024-04-14T15:16:28.890292+0000","last_fullsized":"2024-04-14T15:16:28.890292+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-04-14T15:15:30.805923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:15:30.805923+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:31.513697+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":100,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.888912+0000","last_change":"2024-04-14T15:15:44.888478+0000","last_active":"2024-04-14T15:16:28.888912+0000","last_peered":"2024-04-14T15:16:28.888912+0000","last_clean":"2024-04-14T15:16:28.888912+0000","last_became_active":"2024-04-14T15:15:44.887452+0000","last_became_peered":"2024-04-14T15:15:44.887452+0000","last_unstale":"2024-04-14T15:16:28.888912+0000","last_undegraded":"2024-04-14T15:16:28.888912+0000","last_fullsized":"2024-04-14T15:16:28.888912+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:33.770408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:13:33.770408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:28.968548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"280'9","reported_seq":134,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.890494+0000","last_change":"2024-04-14T15:15:44.883512+0000","last_active":"2024-04-14T15:16:28.890494+0000","last_peered":"2024-04-14T15:16:28.890494+0000","last_clean":"2024-04-14T15:16:28.890494+0000","last_became_active":"2024-04-14T15:15:44.882916+0000","last_became_peered":"2024-04-14T15:15:44.882916+0000","last_unstale":"2024-04-14T15:16:28.890494+0000","last_undegraded":"2024-04-14T15:16:28.890494+0000","last_fullsized":"2024-04-14T15:16:28.890494+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:14:37.162603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:14:37.162603+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":529,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.931448+0000","last_change":"2024-04-14T15:14:00.319415+0000","last_active":"2024-04-14T15:16:28.931448+0000","last_peered":"2024-04-14T15:16:28.931448+0000","last_clean":"2024-04-14T15:16:28.931448+0000","last_became_active":"2024-04-14T15:09:49.066068+0000","last_became_peered":"2024-04-14T15:09:49.066068+0000","last_unstale":"2024-04-14T15:16:28.931448+0000","last_undegraded":"2024-04-14T15:16:28.931448+0000","last_fullsized":"2024-04-14T15:16:28.931448+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:00.319334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:14:00.319334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":100,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.888633+0000","last_change":"2024-04-14T15:15:44.889470+0000","last_active":"2024-04-14T15:16:28.888633+0000","last_peered":"2024-04-14T15:16:28.888633+0000","last_clean":"2024-04-14T15:16:28.888633+0000","last_became_active":"2024-04-14T15:15:44.889106+0000","last_became_peered":"2024-04-14T15:15:44.889106+0000","last_unstale":"2024-04-14T15:16:28.888633+0000","last_undegraded":"2024-04-14T15:16:28.888633+0000","last_fullsized":"2024-04-14T15:16:28.888633+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:01.356001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:01.356001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:28.969103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"183'13","reported_seq":379,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.928889+0000","last_change":"2024-04-14T15:15:44.884579+0000","last_active":"2024-04-14T15:16:28.928889+0000","last_peered":"2024-04-14T15:16:28.928889+0000","last_clean":"2024-04-14T15:16:28.928889+0000","last_became_active":"2024-04-14T15:15:44.884211+0000","last_became_peered":"2024-04-14T15:15:44.884211+0000","last_unstale":"2024-04-14T15:16:28.928889+0000","last_undegraded":"2024-04-14T15:16:28.928889+0000","last_fullsized":"2024-04-14T15:16:28.928889+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'13","last_scrub_stamp":"2024-04-14T15:14:11.755252+0000","last_deep_scrub":"183'13","last_deep_scrub_stamp":"2024-04-14T15:14:11.755252+0000","last_clean_scrub_stamp":"2024-04-14T15:14:11.755252+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:29.785581+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":549,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.928492+0000","last_change":"2024-04-14T15:14:11.348546+0000","last_active":"2024-04-14T15:16:28.928492+0000","last_peered":"2024-04-14T15:16:28.928492+0000","last_clean":"2024-04-14T15:16:28.928492+0000","last_became_active":"2024-04-14T15:09:43.019301+0000","last_became_peered":"2024-04-14T15:09:43.019301+0000","last_unstale":"2024-04-14T15:16:28.928492+0000","last_undegraded":"2024-04-14T15:16:28.928492+0000","last_fullsized":"2024-04-14T15:16:28.928492+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:11.348478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:14:11.348478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"255'10","reported_seq":271,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.890831+0000","last_change":"2024-04-14T15:15:21.884815+0000","last_active":"2024-04-14T15:16:28.890831+0000","last_peered":"2024-04-14T15:16:28.890831+0000","last_clean":"2024-04-14T15:16:28.890831+0000","last_became_active":"2024-04-14T15:15:21.884468+0000","last_became_peered":"2024-04-14T15:15:21.884468+0000","last_unstale":"2024-04-14T15:16:28.890831+0000","last_undegraded":"2024-04-14T15:16:28.890831+0000","last_fullsized":"2024-04-14T15:16:28.890831+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-04-14T15:14:12.169628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:14:12.169628+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:28.968749+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":537,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.882076+0000","last_change":"2024-04-14T15:16:03.948252+0000","last_active":"2024-04-14T15:16:28.882076+0000","last_peered":"2024-04-14T15:16:28.882076+0000","last_clean":"2024-04-14T15:16:28.882076+0000","last_became_active":"2024-04-14T15:09:49.065242+0000","last_became_peered":"2024-04-14T15:09:49.065242+0000","last_unstale":"2024-04-14T15:16:28.882076+0000","last_undegraded":"2024-04-14T15:16:28.882076+0000","last_fullsized":"2024-04-14T15:16:28.882076+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:03.948140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:03.948140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:17.120471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":446,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.929158+0000","last_change":"2024-04-14T15:15:34.555810+0000","last_active":"2024-04-14T15:16:28.929158+0000","last_peered":"2024-04-14T15:16:28.929158+0000","last_clean":"2024-04-14T15:16:28.929158+0000","last_became_active":"2024-04-14T15:12:56.389204+0000","last_became_peered":"2024-04-14T15:12:56.389204+0000","last_unstale":"2024-04-14T15:16:28.929158+0000","last_undegraded":"2024-04-14T15:16:28.929158+0000","last_fullsized":"2024-04-14T15:16:28.929158+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:34.555728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:34.555728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:02.062574+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"72'9","reported_seq":65,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.929156+0000","last_change":"2024-04-14T15:16:24.998072+0000","last_active":"2024-04-14T15:16:28.929156+0000","last_peered":"2024-04-14T15:16:28.929156+0000","last_clean":"2024-04-14T15:16:28.929156+0000","last_became_active":"2024-04-14T15:16:24.993130+0000","last_became_peered":"2024-04-14T15:16:24.993130+0000","last_unstale":"2024-04-14T15:16:28.929156+0000","last_undegraded":"2024-04-14T15:16:28.929156+0000","last_fullsized":"2024-04-14T15:16:28.929156+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-04-14T15:13:24.378555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:13:24.378555+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:31.822031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":549,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.929281+0000","last_change":"2024-04-14T15:14:08.010682+0000","last_active":"2024-04-14T15:16:28.929281+0000","last_peered":"2024-04-14T15:16:28.929281+0000","last_clean":"2024-04-14T15:16:28.929281+0000","last_became_active":"2024-04-14T15:09:43.022480+0000","last_became_peered":"2024-04-14T15:09:43.022480+0000","last_unstale":"2024-04-14T15:16:28.929281+0000","last_undegraded":"2024-04-14T15:16:28.929281+0000","last_fullsized":"2024-04-14T15:16:28.929281+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:08.010603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:14:08.010603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:29.834798+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":218,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.889287+0000","last_change":"2024-04-14T15:15:21.885176+0000","last_active":"2024-04-14T15:16:28.889287+0000","last_peered":"2024-04-14T15:16:28.889287+0000","last_clean":"2024-04-14T15:16:28.889287+0000","last_became_active":"2024-04-14T15:15:21.882643+0000","last_became_peered":"2024-04-14T15:15:21.882643+0000","last_unstale":"2024-04-14T15:16:28.889287+0000","last_undegraded":"2024-04-14T15:16:28.889287+0000","last_fullsized":"2024-04-14T15:16:28.889287+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:54.252608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:14:54.252608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:28.969162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":537,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.926322+0000","last_change":"2024-04-14T15:14:53.219773+0000","last_active":"2024-04-14T15:16:28.926322+0000","last_peered":"2024-04-14T15:16:28.926322+0000","last_clean":"2024-04-14T15:16:28.926322+0000","last_became_active":"2024-04-14T15:09:49.059711+0000","last_became_peered":"2024-04-14T15:09:49.059711+0000","last_unstale":"2024-04-14T15:16:28.926322+0000","last_undegraded":"2024-04-14T15:16:28.926322+0000","last_fullsized":"2024-04-14T15:16:28.926322+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:53.219686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:14:53.219686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":446,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.929132+0000","last_change":"2024-04-14T15:15:27.330424+0000","last_active":"2024-04-14T15:16:28.929132+0000","last_peered":"2024-04-14T15:16:28.929132+0000","last_clean":"2024-04-14T15:16:28.929132+0000","last_became_active":"2024-04-14T15:12:56.396294+0000","last_became_peered":"2024-04-14T15:12:56.396294+0000","last_unstale":"2024-04-14T15:16:28.929132+0000","last_undegraded":"2024-04-14T15:16:28.929132+0000","last_fullsized":"2024-04-14T15:16:28.929132+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:27.330328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:27.330328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:40.634678+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"72'11","reported_seq":564,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.881456+0000","last_change":"2024-04-14T15:16:08.917185+0000","last_active":"2024-04-14T15:16:28.881456+0000","last_peered":"2024-04-14T15:16:28.881456+0000","last_clean":"2024-04-14T15:16:28.881456+0000","last_became_active":"2024-04-14T15:09:45.026065+0000","last_became_peered":"2024-04-14T15:09:45.026065+0000","last_unstale":"2024-04-14T15:16:28.881456+0000","last_undegraded":"2024-04-14T15:16:28.881456+0000","last_fullsized":"2024-04-14T15:16:28.881456+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-04-14T15:16:08.917092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:16:08.917092+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:29.947559+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":549,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.890720+0000","last_change":"2024-04-14T15:16:11.135234+0000","last_active":"2024-04-14T15:16:28.890720+0000","last_peered":"2024-04-14T15:16:28.890720+0000","last_clean":"2024-04-14T15:16:28.890720+0000","last_became_active":"2024-04-14T15:09:43.023583+0000","last_became_peered":"2024-04-14T15:09:43.023583+0000","last_unstale":"2024-04-14T15:16:28.890720+0000","last_undegraded":"2024-04-14T15:16:28.890720+0000","last_fullsized":"2024-04-14T15:16:28.890720+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:11.135164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:11.135164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:33.659405+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":533,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.887716+0000","last_change":"2024-04-14T15:14:08.486305+0000","last_active":"2024-04-14T15:16:28.887716+0000","last_peered":"2024-04-14T15:16:28.887716+0000","last_clean":"2024-04-14T15:16:28.887716+0000","last_became_active":"2024-04-14T15:09:47.041683+0000","last_became_peered":"2024-04-14T15:09:47.041683+0000","last_unstale":"2024-04-14T15:16:28.887716+0000","last_undegraded":"2024-04-14T15:16:28.887716+0000","last_fullsized":"2024-04-14T15:16:28.887716+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:08.486217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:08.486217+0000","last_clean_scrub_stamp":"2024-04-14T15:14:08.486217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":214,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.886582+0000","last_change":"2024-04-14T15:15:21.890353+0000","last_active":"2024-04-14T15:16:28.886582+0000","last_peered":"2024-04-14T15:16:28.886582+0000","last_clean":"2024-04-14T15:16:28.886582+0000","last_became_active":"2024-04-14T15:15:21.889567+0000","last_became_peered":"2024-04-14T15:15:21.889567+0000","last_unstale":"2024-04-14T15:16:28.886582+0000","last_undegraded":"2024-04-14T15:16:28.886582+0000","last_fullsized":"2024-04-14T15:16:28.886582+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:34.547654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:13:34.547654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:28.968584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":446,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.925707+0000","last_change":"2024-04-14T15:14:44.506694+0000","last_active":"2024-04-14T15:16:28.925707+0000","last_peered":"2024-04-14T15:16:28.925707+0000","last_clean":"2024-04-14T15:16:28.925707+0000","last_became_active":"2024-04-14T15:12:56.387723+0000","last_became_peered":"2024-04-14T15:12:56.387723+0000","last_unstale":"2024-04-14T15:16:28.925707+0000","last_undegraded":"2024-04-14T15:16:28.925707+0000","last_fullsized":"2024-04-14T15:16:28.925707+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:44.506614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:14:44.506614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.10","version":"72'4","reported_seq":551,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.888328+0000","last_change":"2024-04-14T15:14:45.052235+0000","last_active":"2024-04-14T15:16:28.888328+0000","last_peered":"2024-04-14T15:16:28.888328+0000","last_clean":"2024-04-14T15:16:28.888328+0000","last_became_active":"2024-04-14T15:09:45.038815+0000","last_became_peered":"2024-04-14T15:09:45.038815+0000","last_unstale":"2024-04-14T15:16:28.888328+0000","last_undegraded":"2024-04-14T15:16:28.888328+0000","last_fullsized":"2024-04-14T15:16:28.888328+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4","last_scrub_stamp":"2024-04-14T15:14:45.052074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:14:45.052074+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:28.968976+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":100,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.930488+0000","last_change":"2024-04-14T15:15:44.885024+0000","last_active":"2024-04-14T15:16:28.930488+0000","last_peered":"2024-04-14T15:16:28.930488+0000","last_clean":"2024-04-14T15:16:28.930488+0000","last_became_active":"2024-04-14T15:15:44.884730+0000","last_became_peered":"2024-04-14T15:15:44.884730+0000","last_unstale":"2024-04-14T15:16:28.930488+0000","last_undegraded":"2024-04-14T15:16:28.930488+0000","last_fullsized":"2024-04-14T15:16:28.930488+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:57.260939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:08.712289+0000","last_clean_scrub_stamp":"2024-04-14T15:14:57.260939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:29.834887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":549,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.926038+0000","last_change":"2024-04-14T15:16:13.909554+0000","last_active":"2024-04-14T15:16:28.926038+0000","last_peered":"2024-04-14T15:16:28.926038+0000","last_clean":"2024-04-14T15:16:28.926038+0000","last_became_active":"2024-04-14T15:09:47.041074+0000","last_became_peered":"2024-04-14T15:09:47.041074+0000","last_unstale":"2024-04-14T15:16:28.926038+0000","last_undegraded":"2024-04-14T15:16:28.926038+0000","last_fullsized":"2024-04-14T15:16:28.926038+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:13.909454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:13.909454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:23.644882+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":529,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.928320+0000","last_change":"2024-04-14T15:13:26.142140+0000","last_active":"2024-04-14T15:16:28.928320+0000","last_peered":"2024-04-14T15:16:28.928320+0000","last_clean":"2024-04-14T15:16:28.928320+0000","last_became_active":"2024-04-14T15:09:49.061359+0000","last_became_peered":"2024-04-14T15:09:49.061359+0000","last_unstale":"2024-04-14T15:16:28.928320+0000","last_undegraded":"2024-04-14T15:16:28.928320+0000","last_fullsized":"2024-04-14T15:16:28.928320+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:26.142060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:13:26.142060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:29.785324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":446,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.928363+0000","last_change":"2024-04-14T15:15:43.680936+0000","last_active":"2024-04-14T15:16:28.928363+0000","last_peered":"2024-04-14T15:16:28.928363+0000","last_clean":"2024-04-14T15:16:28.928363+0000","last_became_active":"2024-04-14T15:12:56.352528+0000","last_became_peered":"2024-04-14T15:12:56.352528+0000","last_unstale":"2024-04-14T15:16:28.928363+0000","last_undegraded":"2024-04-14T15:16:28.928363+0000","last_fullsized":"2024-04-14T15:16:28.928363+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:43.680852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:15:43.680852+0000","last_clean_scrub_stamp":"2024-04-14T15:15:43.680852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:51.024108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"102'16","reported_seq":207,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.887637+0000","last_change":"2024-04-14T15:15:21.886770+0000","last_active":"2024-04-14T15:16:28.887637+0000","last_peered":"2024-04-14T15:16:28.887637+0000","last_clean":"2024-04-14T15:16:28.887637+0000","last_became_active":"2024-04-14T15:15:21.884783+0000","last_became_peered":"2024-04-14T15:15:21.884783+0000","last_unstale":"2024-04-14T15:16:28.887637+0000","last_undegraded":"2024-04-14T15:16:28.887637+0000","last_fullsized":"2024-04-14T15:16:28.887637+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'16","last_scrub_stamp":"2024-04-14T15:14:10.211163+0000","last_deep_scrub":"102'16","last_deep_scrub_stamp":"2024-04-14T15:12:35.999326+0000","last_clean_scrub_stamp":"2024-04-14T15:14:10.211163+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:28.968848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":549,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.929938+0000","last_change":"2024-04-14T15:14:15.075691+0000","last_active":"2024-04-14T15:16:28.929938+0000","last_peered":"2024-04-14T15:16:28.929938+0000","last_clean":"2024-04-14T15:16:28.929938+0000","last_became_active":"2024-04-14T15:09:43.023220+0000","last_became_peered":"2024-04-14T15:09:43.023220+0000","last_unstale":"2024-04-14T15:16:28.929938+0000","last_undegraded":"2024-04-14T15:16:28.929938+0000","last_fullsized":"2024-04-14T15:16:28.929938+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:15.075612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:14:15.075612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:29.834782+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":533,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.925545+0000","last_change":"2024-04-14T15:13:29.219729+0000","last_active":"2024-04-14T15:16:28.925545+0000","last_peered":"2024-04-14T15:16:28.925545+0000","last_clean":"2024-04-14T15:16:28.925545+0000","last_became_active":"2024-04-14T15:09:47.044665+0000","last_became_peered":"2024-04-14T15:09:47.044665+0000","last_unstale":"2024-04-14T15:16:28.925545+0000","last_undegraded":"2024-04-14T15:16:28.925545+0000","last_fullsized":"2024-04-14T15:16:28.925545+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:29.219589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:13:29.219589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:29.785393+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":222,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.889570+0000","last_change":"2024-04-14T15:16:05.885527+0000","last_active":"2024-04-14T15:16:28.889570+0000","last_peered":"2024-04-14T15:16:28.889570+0000","last_clean":"2024-04-14T15:16:28.889570+0000","last_became_active":"2024-04-14T15:15:21.883940+0000","last_became_peered":"2024-04-14T15:15:21.883940+0000","last_unstale":"2024-04-14T15:16:28.889570+0000","last_undegraded":"2024-04-14T15:16:28.889570+0000","last_fullsized":"2024-04-14T15:16:28.889570+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:05.885438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:05.885438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:27.063364+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":446,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.925388+0000","last_change":"2024-04-14T15:15:28.474035+0000","last_active":"2024-04-14T15:16:28.925388+0000","last_peered":"2024-04-14T15:16:28.925388+0000","last_clean":"2024-04-14T15:16:28.925388+0000","last_became_active":"2024-04-14T15:12:56.335230+0000","last_became_peered":"2024-04-14T15:12:56.335230+0000","last_unstale":"2024-04-14T15:16:28.925388+0000","last_undegraded":"2024-04-14T15:16:28.925388+0000","last_fullsized":"2024-04-14T15:16:28.925388+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:28.473899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:28.473899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:55.001475+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"72'10","reported_seq":553,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.931847+0000","last_change":"2024-04-14T15:13:17.411319+0000","last_active":"2024-04-14T15:16:28.931847+0000","last_peered":"2024-04-14T15:16:28.931847+0000","last_clean":"2024-04-14T15:16:28.931847+0000","last_became_active":"2024-04-14T15:09:45.181456+0000","last_became_peered":"2024-04-14T15:09:45.181456+0000","last_unstale":"2024-04-14T15:16:28.931847+0000","last_undegraded":"2024-04-14T15:16:28.931847+0000","last_fullsized":"2024-04-14T15:16:28.931847+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-04-14T15:13:17.411175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:13:17.411175+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":100,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.881124+0000","last_change":"2024-04-14T15:15:44.878646+0000","last_active":"2024-04-14T15:16:28.881124+0000","last_peered":"2024-04-14T15:16:28.881124+0000","last_clean":"2024-04-14T15:16:28.881124+0000","last_became_active":"2024-04-14T15:15:44.878336+0000","last_became_peered":"2024-04-14T15:15:44.878336+0000","last_unstale":"2024-04-14T15:16:28.881124+0000","last_undegraded":"2024-04-14T15:16:28.881124+0000","last_fullsized":"2024-04-14T15:16:28.881124+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:07.680508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:14:07.680508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:29.106698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"72'8","reported_seq":643,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.930795+0000","last_change":"2024-04-14T15:16:12.917059+0000","last_active":"2024-04-14T15:16:28.930795+0000","last_peered":"2024-04-14T15:16:28.930795+0000","last_clean":"2024-04-14T15:16:28.930795+0000","last_became_active":"2024-04-14T15:09:47.048216+0000","last_became_peered":"2024-04-14T15:09:47.048216+0000","last_unstale":"2024-04-14T15:16:28.930795+0000","last_undegraded":"2024-04-14T15:16:28.930795+0000","last_fullsized":"2024-04-14T15:16:28.930795+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:16:12.916977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:12.916977+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:19.365595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"72'1","reported_seq":538,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.926989+0000","last_change":"2024-04-14T15:14:37.932503+0000","last_active":"2024-04-14T15:16:28.926989+0000","last_peered":"2024-04-14T15:16:28.926989+0000","last_clean":"2024-04-14T15:16:28.926989+0000","last_became_active":"2024-04-14T15:09:49.062051+0000","last_became_peered":"2024-04-14T15:09:49.062051+0000","last_unstale":"2024-04-14T15:16:28.926989+0000","last_undegraded":"2024-04-14T15:16:28.926989+0000","last_fullsized":"2024-04-14T15:16:28.926989+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-04-14T15:14:37.932420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:14:37.932420+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:29.785794+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":446,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.926426+0000","last_change":"2024-04-14T15:16:09.471961+0000","last_active":"2024-04-14T15:16:28.926426+0000","last_peered":"2024-04-14T15:16:28.926426+0000","last_clean":"2024-04-14T15:16:28.926426+0000","last_became_active":"2024-04-14T15:12:56.543793+0000","last_became_peered":"2024-04-14T15:12:56.543793+0000","last_unstale":"2024-04-14T15:16:28.926426+0000","last_undegraded":"2024-04-14T15:16:28.926426+0000","last_fullsized":"2024-04-14T15:16:28.926426+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:09.471852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:16:09.471852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:19.732931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"72'11","reported_seq":583,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.930303+0000","last_change":"2024-04-14T15:16:06.886451+0000","last_active":"2024-04-14T15:16:28.930303+0000","last_peered":"2024-04-14T15:16:28.930303+0000","last_clean":"2024-04-14T15:16:28.930303+0000","last_became_active":"2024-04-14T15:10:07.764183+0000","last_became_peered":"2024-04-14T15:10:07.764183+0000","last_unstale":"2024-04-14T15:16:28.930303+0000","last_undegraded":"2024-04-14T15:16:28.930303+0000","last_fullsized":"2024-04-14T15:16:28.930303+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-04-14T15:16:06.886325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:16:06.886325+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:30.942226+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"65'2","reported_seq":233,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.886792+0000","last_change":"2024-04-14T15:15:21.887031+0000","last_active":"2024-04-14T15:16:28.886792+0000","last_peered":"2024-04-14T15:16:28.886792+0000","last_clean":"2024-04-14T15:16:28.886792+0000","last_became_active":"2024-04-14T15:15:21.885343+0000","last_became_peered":"2024-04-14T15:15:21.885343+0000","last_unstale":"2024-04-14T15:16:28.886792+0000","last_undegraded":"2024-04-14T15:16:28.886792+0000","last_fullsized":"2024-04-14T15:16:28.886792+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-04-14T15:14:01.114004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:14:01.114004+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:28.968654+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":534,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.931914+0000","last_change":"2024-04-14T15:13:28.464940+0000","last_active":"2024-04-14T15:16:28.931914+0000","last_peered":"2024-04-14T15:16:28.931914+0000","last_clean":"2024-04-14T15:16:28.931914+0000","last_became_active":"2024-04-14T15:09:47.045730+0000","last_became_peered":"2024-04-14T15:09:47.045730+0000","last_unstale":"2024-04-14T15:16:28.931914+0000","last_undegraded":"2024-04-14T15:16:28.931914+0000","last_fullsized":"2024-04-14T15:16:28.931914+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:28.464811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:13:28.464811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":529,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.890320+0000","last_change":"2024-04-14T15:14:03.551214+0000","last_active":"2024-04-14T15:16:28.890320+0000","last_peered":"2024-04-14T15:16:28.890320+0000","last_clean":"2024-04-14T15:16:28.890320+0000","last_became_active":"2024-04-14T15:09:49.057545+0000","last_became_peered":"2024-04-14T15:09:49.057545+0000","last_unstale":"2024-04-14T15:16:28.890320+0000","last_undegraded":"2024-04-14T15:16:28.890320+0000","last_fullsized":"2024-04-14T15:16:28.890320+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:03.551112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:21.247695+0000","last_clean_scrub_stamp":"2024-04-14T15:14:03.551112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":100,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.881317+0000","last_change":"2024-04-14T15:15:44.882920+0000","last_active":"2024-04-14T15:16:28.881317+0000","last_peered":"2024-04-14T15:16:28.881317+0000","last_clean":"2024-04-14T15:16:28.881317+0000","last_became_active":"2024-04-14T15:15:44.882604+0000","last_became_peered":"2024-04-14T15:15:44.882604+0000","last_unstale":"2024-04-14T15:16:28.881317+0000","last_undegraded":"2024-04-14T15:16:28.881317+0000","last_fullsized":"2024-04-14T15:16:28.881317+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:28.483617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:28.483617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:29.345071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"72'15","reported_seq":570,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.929875+0000","last_change":"2024-04-14T15:15:44.609525+0000","last_active":"2024-04-14T15:16:28.929875+0000","last_peered":"2024-04-14T15:16:28.929875+0000","last_clean":"2024-04-14T15:16:28.929875+0000","last_became_active":"2024-04-14T15:09:45.032887+0000","last_became_peered":"2024-04-14T15:09:45.032887+0000","last_unstale":"2024-04-14T15:16:28.929875+0000","last_undegraded":"2024-04-14T15:16:28.929875+0000","last_fullsized":"2024-04-14T15:16:28.929875+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-04-14T15:15:44.609332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:15:44.609332+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:57.512618+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":552,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.929315+0000","last_change":"2024-04-14T15:16:09.825296+0000","last_active":"2024-04-14T15:16:28.929315+0000","last_peered":"2024-04-14T15:16:28.929315+0000","last_clean":"2024-04-14T15:16:28.929315+0000","last_became_active":"2024-04-14T15:09:43.021494+0000","last_became_peered":"2024-04-14T15:09:43.021494+0000","last_unstale":"2024-04-14T15:16:28.929315+0000","last_undegraded":"2024-04-14T15:16:28.929315+0000","last_fullsized":"2024-04-14T15:16:28.929315+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:09.825220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:09.825220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:14.729781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":541,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.890633+0000","last_change":"2024-04-14T15:14:56.199840+0000","last_active":"2024-04-14T15:16:28.890633+0000","last_peered":"2024-04-14T15:16:28.890633+0000","last_clean":"2024-04-14T15:16:28.890633+0000","last_became_active":"2024-04-14T15:09:47.042733+0000","last_became_peered":"2024-04-14T15:09:47.042733+0000","last_unstale":"2024-04-14T15:16:28.890633+0000","last_undegraded":"2024-04-14T15:16:28.890633+0000","last_fullsized":"2024-04-14T15:16:28.890633+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:56.199756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:14:56.199756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":538,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.882033+0000","last_change":"2024-04-14T15:15:31.874372+0000","last_active":"2024-04-14T15:16:28.882033+0000","last_peered":"2024-04-14T15:16:28.882033+0000","last_clean":"2024-04-14T15:16:28.882033+0000","last_became_active":"2024-04-14T15:09:49.127979+0000","last_became_peered":"2024-04-14T15:09:49.127979+0000","last_unstale":"2024-04-14T15:16:28.882033+0000","last_undegraded":"2024-04-14T15:16:28.882033+0000","last_fullsized":"2024-04-14T15:16:28.882033+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:31.874232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:15:31.874232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:53.658126+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":446,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.928106+0000","last_change":"2024-04-14T15:15:31.635164+0000","last_active":"2024-04-14T15:16:28.928106+0000","last_peered":"2024-04-14T15:16:28.928106+0000","last_clean":"2024-04-14T15:16:28.928106+0000","last_became_active":"2024-04-14T15:12:56.392352+0000","last_became_peered":"2024-04-14T15:12:56.392352+0000","last_unstale":"2024-04-14T15:16:28.928106+0000","last_undegraded":"2024-04-14T15:16:28.928106+0000","last_fullsized":"2024-04-14T15:16:28.928106+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:31.635060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:31.635060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:37.237891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"72'12","reported_seq":253,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.891043+0000","last_change":"2024-04-14T15:16:11.903062+0000","last_active":"2024-04-14T15:16:28.891043+0000","last_peered":"2024-04-14T15:16:28.891043+0000","last_clean":"2024-04-14T15:16:28.891043+0000","last_became_active":"2024-04-14T15:15:21.877838+0000","last_became_peered":"2024-04-14T15:15:21.877838+0000","last_unstale":"2024-04-14T15:16:28.891043+0000","last_undegraded":"2024-04-14T15:16:28.891043+0000","last_fullsized":"2024-04-14T15:16:28.891043+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'12","last_scrub_stamp":"2024-04-14T15:16:11.902934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:16:11.902934+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:18.486882+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":234,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.889394+0000","last_change":"2024-04-14T15:16:20.969027+0000","last_active":"2024-04-14T15:16:28.889394+0000","last_peered":"2024-04-14T15:16:28.889394+0000","last_clean":"2024-04-14T15:16:28.889394+0000","last_became_active":"2024-04-14T15:15:21.883544+0000","last_became_peered":"2024-04-14T15:15:21.883544+0000","last_unstale":"2024-04-14T15:16:28.889394+0000","last_undegraded":"2024-04-14T15:16:28.889394+0000","last_fullsized":"2024-04-14T15:16:28.889394+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:20.968897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:20.968897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:32.701185+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":541,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.881777+0000","last_change":"2024-04-14T15:14:56.868446+0000","last_active":"2024-04-14T15:16:28.881777+0000","last_peered":"2024-04-14T15:16:28.881777+0000","last_clean":"2024-04-14T15:16:28.881777+0000","last_became_active":"2024-04-14T15:09:47.048045+0000","last_became_peered":"2024-04-14T15:09:47.048045+0000","last_unstale":"2024-04-14T15:16:28.881777+0000","last_undegraded":"2024-04-14T15:16:28.881777+0000","last_fullsized":"2024-04-14T15:16:28.881777+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:56.868328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:14:56.868328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:29.106819+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":223,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.891443+0000","last_change":"2024-04-14T15:16:07.879394+0000","last_active":"2024-04-14T15:16:28.891443+0000","last_peered":"2024-04-14T15:16:28.891443+0000","last_clean":"2024-04-14T15:16:28.891443+0000","last_became_active":"2024-04-14T15:15:21.889691+0000","last_became_peered":"2024-04-14T15:15:21.889691+0000","last_unstale":"2024-04-14T15:16:28.891443+0000","last_undegraded":"2024-04-14T15:16:28.891443+0000","last_fullsized":"2024-04-14T15:16:28.891443+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:07.879280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:07.879280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:23.294021+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":446,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.930117+0000","last_change":"2024-04-14T15:14:42.527202+0000","last_active":"2024-04-14T15:16:28.930117+0000","last_peered":"2024-04-14T15:16:28.930117+0000","last_clean":"2024-04-14T15:16:28.930117+0000","last_became_active":"2024-04-14T15:12:56.375278+0000","last_became_peered":"2024-04-14T15:12:56.375278+0000","last_unstale":"2024-04-14T15:16:28.930117+0000","last_undegraded":"2024-04-14T15:16:28.930117+0000","last_fullsized":"2024-04-14T15:16:28.930117+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:42.527099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:14:42.527099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"276'13","reported_seq":565,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.881279+0000","last_change":"2024-04-14T15:15:43.969860+0000","last_active":"2024-04-14T15:16:28.881279+0000","last_peered":"2024-04-14T15:16:28.881279+0000","last_clean":"2024-04-14T15:16:28.881279+0000","last_became_active":"2024-04-14T15:09:45.030093+0000","last_became_peered":"2024-04-14T15:09:45.030093+0000","last_unstale":"2024-04-14T15:16:28.881279+0000","last_undegraded":"2024-04-14T15:16:28.881279+0000","last_fullsized":"2024-04-14T15:16:28.881279+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'13","last_scrub_stamp":"2024-04-14T15:15:43.969775+0000","last_deep_scrub":"276'13","last_deep_scrub_stamp":"2024-04-14T15:15:43.969775+0000","last_clean_scrub_stamp":"2024-04-14T15:15:43.969775+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:52.603875+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":247,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.926513+0000","last_change":"2024-04-14T15:14:30.858015+0000","last_active":"2024-04-14T15:16:28.926513+0000","last_peered":"2024-04-14T15:16:28.926513+0000","last_clean":"2024-04-14T15:16:28.926513+0000","last_became_active":"2024-04-14T15:14:30.857727+0000","last_became_peered":"2024-04-14T15:14:30.857727+0000","last_unstale":"2024-04-14T15:16:28.926513+0000","last_undegraded":"2024-04-14T15:16:28.926513+0000","last_fullsized":"2024-04-14T15:16:28.926513+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:05.518321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:14:05.518321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:29.785506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":100,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.928537+0000","last_change":"2024-04-14T15:15:44.883445+0000","last_active":"2024-04-14T15:16:28.928537+0000","last_peered":"2024-04-14T15:16:28.928537+0000","last_clean":"2024-04-14T15:16:28.928537+0000","last_became_active":"2024-04-14T15:15:44.883121+0000","last_became_peered":"2024-04-14T15:15:44.883121+0000","last_unstale":"2024-04-14T15:16:28.928537+0000","last_undegraded":"2024-04-14T15:16:28.928537+0000","last_fullsized":"2024-04-14T15:16:28.928537+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:10.728836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:14:10.728836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":537,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.926648+0000","last_change":"2024-04-14T15:14:04.346959+0000","last_active":"2024-04-14T15:16:28.926648+0000","last_peered":"2024-04-14T15:16:28.926648+0000","last_clean":"2024-04-14T15:16:28.926648+0000","last_became_active":"2024-04-14T15:09:49.061756+0000","last_became_peered":"2024-04-14T15:09:49.061756+0000","last_unstale":"2024-04-14T15:16:28.926648+0000","last_undegraded":"2024-04-14T15:16:28.926648+0000","last_fullsized":"2024-04-14T15:16:28.926648+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:04.346872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:14:04.346872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":446,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.929635+0000","last_change":"2024-04-14T15:14:00.109792+0000","last_active":"2024-04-14T15:16:28.929635+0000","last_peered":"2024-04-14T15:16:28.929635+0000","last_clean":"2024-04-14T15:16:28.929635+0000","last_became_active":"2024-04-14T15:12:56.232831+0000","last_became_peered":"2024-04-14T15:12:56.232831+0000","last_unstale":"2024-04-14T15:16:28.929635+0000","last_undegraded":"2024-04-14T15:16:28.929635+0000","last_fullsized":"2024-04-14T15:16:28.929635+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:00.109696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:14:00.109696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:29.834746+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1b","version":"87'7","reported_seq":557,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.881605+0000","last_change":"2024-04-14T15:15:44.927867+0000","last_active":"2024-04-14T15:16:28.881605+0000","last_peered":"2024-04-14T15:16:28.881605+0000","last_clean":"2024-04-14T15:16:28.881605+0000","last_became_active":"2024-04-14T15:09:45.028667+0000","last_became_peered":"2024-04-14T15:09:45.028667+0000","last_unstale":"2024-04-14T15:16:28.881605+0000","last_undegraded":"2024-04-14T15:16:28.881605+0000","last_fullsized":"2024-04-14T15:16:28.881605+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'7","last_scrub_stamp":"2024-04-14T15:15:44.927776+0000","last_deep_scrub":"87'7","last_deep_scrub_stamp":"2024-04-14T15:11:10.510785+0000","last_clean_scrub_stamp":"2024-04-14T15:15:44.927776+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:09.960055+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":19,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.929443+0000","last_change":"2024-04-14T15:16:24.910604+0000","last_active":"2024-04-14T15:16:28.929443+0000","last_peered":"2024-04-14T15:16:28.929443+0000","last_clean":"2024-04-14T15:16:28.929443+0000","last_became_active":"2024-04-14T15:16:24.910246+0000","last_became_peered":"2024-04-14T15:16:24.910246+0000","last_unstale":"2024-04-14T15:16:28.929443+0000","last_undegraded":"2024-04-14T15:16:28.929443+0000","last_fullsized":"2024-04-14T15:16:28.929443+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:08.254601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:08.254601+0000","last_clean_scrub_stamp":"2024-04-14T15:14:08.254601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":541,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.928230+0000","last_change":"2024-04-14T15:16:17.804733+0000","last_active":"2024-04-14T15:16:28.928230+0000","last_peered":"2024-04-14T15:16:28.928230+0000","last_clean":"2024-04-14T15:16:28.928230+0000","last_became_active":"2024-04-14T15:09:47.046517+0000","last_became_peered":"2024-04-14T15:09:47.046517+0000","last_unstale":"2024-04-14T15:16:28.928230+0000","last_undegraded":"2024-04-14T15:16:28.928230+0000","last_fullsized":"2024-04-14T15:16:28.928230+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:17.804649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:17.804649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:23.540926+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":101,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.887851+0000","last_change":"2024-04-14T15:15:44.887620+0000","last_active":"2024-04-14T15:16:28.887851+0000","last_peered":"2024-04-14T15:16:28.887851+0000","last_clean":"2024-04-14T15:16:28.887851+0000","last_became_active":"2024-04-14T15:15:44.887227+0000","last_became_peered":"2024-04-14T15:15:44.887227+0000","last_unstale":"2024-04-14T15:16:28.887851+0000","last_undegraded":"2024-04-14T15:16:28.887851+0000","last_fullsized":"2024-04-14T15:16:28.887851+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:56.268086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:56.268086+0000","last_clean_scrub_stamp":"2024-04-14T15:14:56.268086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":446,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.927467+0000","last_change":"2024-04-14T15:15:37.629813+0000","last_active":"2024-04-14T15:16:28.927467+0000","last_peered":"2024-04-14T15:16:28.927467+0000","last_clean":"2024-04-14T15:16:28.927467+0000","last_became_active":"2024-04-14T15:12:56.189117+0000","last_became_peered":"2024-04-14T15:12:56.189117+0000","last_unstale":"2024-04-14T15:16:28.927467+0000","last_undegraded":"2024-04-14T15:16:28.927467+0000","last_fullsized":"2024-04-14T15:16:28.927467+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:37.629718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:37.629718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:48.883517+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.18","version":"72'9","reported_seq":561,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.881024+0000","last_change":"2024-04-14T15:16:09.965230+0000","last_active":"2024-04-14T15:16:28.881024+0000","last_peered":"2024-04-14T15:16:28.881024+0000","last_clean":"2024-04-14T15:16:28.881024+0000","last_became_active":"2024-04-14T15:09:45.032079+0000","last_became_peered":"2024-04-14T15:09:45.032079+0000","last_unstale":"2024-04-14T15:16:28.881024+0000","last_undegraded":"2024-04-14T15:16:28.881024+0000","last_fullsized":"2024-04-14T15:16:28.881024+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-04-14T15:16:09.965113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:16:09.965113+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:11.168994+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":227,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.890779+0000","last_change":"2024-04-14T15:15:21.892804+0000","last_active":"2024-04-14T15:16:28.890779+0000","last_peered":"2024-04-14T15:16:28.890779+0000","last_clean":"2024-04-14T15:16:28.890779+0000","last_became_active":"2024-04-14T15:15:21.891926+0000","last_became_peered":"2024-04-14T15:15:21.891926+0000","last_unstale":"2024-04-14T15:16:28.890779+0000","last_undegraded":"2024-04-14T15:16:28.890779+0000","last_fullsized":"2024-04-14T15:16:28.890779+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:59.366623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:13:59.366623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:28.968780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":533,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.887615+0000","last_change":"2024-04-14T15:13:25.370811+0000","last_active":"2024-04-14T15:16:28.887615+0000","last_peered":"2024-04-14T15:16:28.887615+0000","last_clean":"2024-04-14T15:16:28.887615+0000","last_became_active":"2024-04-14T15:09:47.041486+0000","last_became_peered":"2024-04-14T15:09:47.041486+0000","last_unstale":"2024-04-14T15:16:28.887615+0000","last_undegraded":"2024-04-14T15:16:28.887615+0000","last_fullsized":"2024-04-14T15:16:28.887615+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:13:25.370679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:13:25.370679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":529,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.881536+0000","last_change":"2024-04-14T15:14:37.812472+0000","last_active":"2024-04-14T15:16:28.881536+0000","last_peered":"2024-04-14T15:16:28.881536+0000","last_clean":"2024-04-14T15:16:28.881536+0000","last_became_active":"2024-04-14T15:09:49.058904+0000","last_became_peered":"2024-04-14T15:09:49.058904+0000","last_unstale":"2024-04-14T15:16:28.881536+0000","last_undegraded":"2024-04-14T15:16:28.881536+0000","last_fullsized":"2024-04-14T15:16:28.881536+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:37.812396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:14:37.812396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:29.106769+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":100,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.928223+0000","last_change":"2024-04-14T15:15:44.889891+0000","last_active":"2024-04-14T15:16:28.928223+0000","last_peered":"2024-04-14T15:16:28.928223+0000","last_clean":"2024-04-14T15:16:28.928223+0000","last_became_active":"2024-04-14T15:15:44.889605+0000","last_became_peered":"2024-04-14T15:15:44.889605+0000","last_unstale":"2024-04-14T15:16:28.928223+0000","last_undegraded":"2024-04-14T15:16:28.928223+0000","last_fullsized":"2024-04-14T15:16:28.928223+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:00.363827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:00.363827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:29.785826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"306'22","reported_seq":262,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.887737+0000","last_change":"2024-04-14T15:16:13.898263+0000","last_active":"2024-04-14T15:16:28.887737+0000","last_peered":"2024-04-14T15:16:28.887737+0000","last_clean":"2024-04-14T15:16:28.887737+0000","last_became_active":"2024-04-14T15:15:21.884310+0000","last_became_peered":"2024-04-14T15:15:21.884310+0000","last_unstale":"2024-04-14T15:16:28.887737+0000","last_undegraded":"2024-04-14T15:16:28.887737+0000","last_fullsized":"2024-04-14T15:16:28.887737+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'22","last_scrub_stamp":"2024-04-14T15:16:13.898106+0000","last_deep_scrub":"306'22","last_deep_scrub_stamp":"2024-04-14T15:16:13.898106+0000","last_clean_scrub_stamp":"2024-04-14T15:16:13.898106+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:39.744958+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":550,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.930741+0000","last_change":"2024-04-14T15:16:14.427502+0000","last_active":"2024-04-14T15:16:28.930741+0000","last_peered":"2024-04-14T15:16:28.930741+0000","last_clean":"2024-04-14T15:16:28.930741+0000","last_became_active":"2024-04-14T15:09:43.017351+0000","last_became_peered":"2024-04-14T15:09:43.017351+0000","last_unstale":"2024-04-14T15:16:28.930741+0000","last_undegraded":"2024-04-14T15:16:28.930741+0000","last_fullsized":"2024-04-14T15:16:28.930741+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:14.427425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:14.427425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:38.900235+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":541,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.881356+0000","last_change":"2024-04-14T15:16:19.041461+0000","last_active":"2024-04-14T15:16:28.881356+0000","last_peered":"2024-04-14T15:16:28.881356+0000","last_clean":"2024-04-14T15:16:28.881356+0000","last_became_active":"2024-04-14T15:09:47.042538+0000","last_became_peered":"2024-04-14T15:09:47.042538+0000","last_unstale":"2024-04-14T15:16:28.881356+0000","last_undegraded":"2024-04-14T15:16:28.881356+0000","last_fullsized":"2024-04-14T15:16:28.881356+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:19.041346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:08.512130+0000","last_clean_scrub_stamp":"2024-04-14T15:16:19.041346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:17:42.007434+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":537,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-14T15:16:28.889048+0000","last_change":"2024-04-14T15:14:38.023392+0000","last_active":"2024-04-14T15:16:28.889048+0000","last_peered":"2024-04-14T15:16:28.889048+0000","last_clean":"2024-04-14T15:16:28.889048+0000","last_became_active":"2024-04-14T15:09:49.129636+0000","last_became_peered":"2024-04-14T15:09:49.129636+0000","last_unstale":"2024-04-14T15:16:28.889048+0000","last_undegraded":"2024-04-14T15:16:28.889048+0000","last_fullsized":"2024-04-14T15:16:28.889048+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:38.023315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:14:38.023315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:28.969016+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":274,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":110,"num_pg":32,"stat_sum":{"num_bytes":8195524,"num_objects":124,"num_object_clones":2,"num_object_copies":372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":124,"num_whiteouts":0,"num_read":98,"num_read_kb":4,"num_write":551,"num_write_kb":12642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":26075136,"data_stored":24586572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1806,"ondisk_log_size":1806,"up":96,"acting":96,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1430,"ondisk_log_size":1430,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":399,"num_read_kb":394,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":695,"num_read_kb":452,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":466,"ondisk_log_size":466,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70,"ondisk_log_size":70,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103255,"num_pgs":101,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":67288,"kb_used_data":13880,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93677864,"statfs":{"total":95995035648,"available":95926132736,"internally_reserved":0,"allocated":14213120,"data_stored":11656085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":54655434},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397594,"num_pgs":88,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":81160,"kb_used_data":5544,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93663992,"statfs":{"total":95995035648,"available":95911927808,"internally_reserved":0,"allocated":5677056,"data_stored":3138455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574964,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":81180,"kb_used_data":9736,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93663972,"statfs":{"total":95995035648,"available":95911907328,"internally_reserved":0,"allocated":9969664,"data_stored":7328953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738489,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":76808,"kb_used_data":9500,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93668344,"statfs":{"total":95995035648,"available":95916384256,"internally_reserved":0,"allocated":9728000,"data_stored":7134802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411440,"num_pgs":93,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":77080,"kb_used_data":9772,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93668072,"statfs":{"total":95995035648,"available":95916105728,"internally_reserved":0,"allocated":10006528,"data_stored":7400562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182506,"num_pgs":93,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":63668,"kb_used_data":6112,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93681484,"statfs":{"total":95995035648,"available":95929839616,"internally_reserved":0,"allocated":6258688,"data_stored":3725083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986276,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":81160,"kb_used_data":5544,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93663992,"statfs":{"total":95995035648,"available":95911927808,"internally_reserved":0,"allocated":5677056,"data_stored":3124418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659233,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":85816,"kb_used_data":10212,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93659336,"statfs":{"total":95995035648,"available":95907160064,"internally_reserved":0,"allocated":10457088,"data_stored":7922170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4206592,"data_stored":4000487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4427776,"data_stored":4194882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4243456,"data_stored":4000606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":176128,"data_stored":428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4358144,"data_stored":4194699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8376320,"data_stored":8194748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":274,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":274,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":274,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":274,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":274,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":274,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":274,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":274,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-14T15:16:32.148 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd pool set .mgr pgp_num 11 2024-04-14T15:16:32.173 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:31 smithi090 ceph-mon[41670]: from='client.37032 v1:172.21.15.90:0/2277571117' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:16:32.174 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:31 smithi090 ceph-mon[41670]: from='client.46967 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi090-93363-55"}]': finished 2024-04-14T15:16:32.174 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:31 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-41"}]': finished 2024-04-14T15:16:32.174 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:31 smithi090 ceph-mon[41670]: osdmap e323: 8 total, 8 up, 8 in 2024-04-14T15:16:32.174 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:31 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1251688097' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi090-93363-56"}]: dispatch 2024-04-14T15:16:32.174 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:31 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1251688097' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi090-93363-56"}]: dispatch 2024-04-14T15:16:32.174 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:31 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1251688097' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi090-93363-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:16:32.175 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:31 smithi090 ceph-mon[30399]: from='client.37032 v1:172.21.15.90:0/2277571117' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:16:32.175 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:31 smithi090 ceph-mon[30399]: from='client.46967 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi090-93363-55"}]': finished 2024-04-14T15:16:32.175 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:31 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-41"}]': finished 2024-04-14T15:16:32.175 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:31 smithi090 ceph-mon[30399]: osdmap e323: 8 total, 8 up, 8 in 2024-04-14T15:16:32.175 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:31 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1251688097' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi090-93363-56"}]: dispatch 2024-04-14T15:16:32.175 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:31 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1251688097' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi090-93363-56"}]: dispatch 2024-04-14T15:16:32.175 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:31 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1251688097' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi090-93363-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:16:32.214 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:16:32.238 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:32.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:32.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:16:32.340 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:32.368 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:32.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:32.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:32.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:31 smithi092 ceph-mon[33966]: from='client.37032 v1:172.21.15.90:0/2277571117' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:16:32.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:31 smithi092 ceph-mon[33966]: from='client.46967 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi090-93363-55"}]': finished 2024-04-14T15:16:32.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:31 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-41"}]': finished 2024-04-14T15:16:32.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:31 smithi092 ceph-mon[33966]: osdmap e323: 8 total, 8 up, 8 in 2024-04-14T15:16:32.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:31 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1251688097' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi090-93363-56"}]: dispatch 2024-04-14T15:16:32.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:31 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1251688097' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi090-93363-56"}]: dispatch 2024-04-14T15:16:32.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:31 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1251688097' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi090-93363-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:16:32.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.371+0000 7f89db339700 1 -- 172.21.15.90:0/2773267042 >> v1:172.21.15.90:6790/0 conn(0x7f89d410a690 legacy=0x7f89d410ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:32.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.371+0000 7f89db339700 1 -- 172.21.15.90:0/2773267042 shutdown_connections 2024-04-14T15:16:32.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.371+0000 7f89db339700 1 -- 172.21.15.90:0/2773267042 >> 172.21.15.90:0/2773267042 conn(0x7f89d4076aa0 msgr2=0x7f89d4076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:16:32.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.371+0000 7f89db339700 1 -- 172.21.15.90:0/2773267042 shutdown_connections 2024-04-14T15:16:32.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.371+0000 7f89db339700 1 -- 172.21.15.90:0/2773267042 wait complete. 2024-04-14T15:16:32.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.372+0000 7f89db339700 1 librados: starting msgr at 2024-04-14T15:16:32.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.372+0000 7f89db339700 1 librados: starting objecter 2024-04-14T15:16:32.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.372+0000 7f89db339700 1 Processor -- start 2024-04-14T15:16:32.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.372+0000 7f89db339700 1 -- start start 2024-04-14T15:16:32.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.372+0000 7f89db339700 1 librados: setting wanted keys 2024-04-14T15:16:32.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.372+0000 7f89db339700 1 librados: calling monclient init 2024-04-14T15:16:32.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.372+0000 7f89db339700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f89d41a79f0 con 0x7f89d410a690 2024-04-14T15:16:32.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.372+0000 7f89db339700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f89d41a8c10 con 0x7f89d41023f0 2024-04-14T15:16:32.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.372+0000 7f89db339700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f89d41a9e50 con 0x7f89d410e230 2024-04-14T15:16:32.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.372+0000 7f89d88d4700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f89d410a690 0x7f89d41a2710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:60854/0 (socket says 172.21.15.90:60854) 2024-04-14T15:16:32.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.372+0000 7f89d98d6700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f89d410e230 0x7f89d41a60f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:42024/0 (socket says 172.21.15.90:42024) 2024-04-14T15:16:32.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.372+0000 7f89d88d4700 1 -- 172.21.15.90:0/3251375049 learned_addr learned my addr 172.21.15.90:0/3251375049 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:16:32.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.372+0000 7f89ca7fc700 10 client.?.objecter ms_handle_connect 0x7f89d410a690 2024-04-14T15:16:32.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.372+0000 7f89ca7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:16:32.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.372+0000 7f89ca7fc700 10 client.?.objecter ms_handle_connect 0x7f89d410e230 2024-04-14T15:16:32.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.372+0000 7f89ca7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:16:32.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.373+0000 7f89ca7fc700 10 client.?.objecter ms_handle_connect 0x7f89d41023f0 2024-04-14T15:16:32.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.373+0000 7f89ca7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:16:32.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.373+0000 7f89ca7fc700 1 -- 172.21.15.90:0/3251375049 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4227832347 0 0) 0x7f89d41a9e50 con 0x7f89d410e230 2024-04-14T15:16:32.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.373+0000 7f89ca7fc700 1 -- 172.21.15.90:0/3251375049 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89c00039c0 con 0x7f89d410e230 2024-04-14T15:16:32.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.373+0000 7f89ca7fc700 1 -- 172.21.15.90:0/3251375049 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 895848204 0 0) 0x7f89d41a79f0 con 0x7f89d410a690 2024-04-14T15:16:32.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.373+0000 7f89ca7fc700 1 -- 172.21.15.90:0/3251375049 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89d41a9e50 con 0x7f89d410a690 2024-04-14T15:16:32.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.373+0000 7f89ca7fc700 1 -- 172.21.15.90:0/3251375049 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2383557030 0 0) 0x7f89d41a8c10 con 0x7f89d41023f0 2024-04-14T15:16:32.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.373+0000 7f89ca7fc700 1 -- 172.21.15.90:0/3251375049 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89d41a79f0 con 0x7f89d41023f0 2024-04-14T15:16:32.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.373+0000 7f89ca7fc700 1 -- 172.21.15.90:0/3251375049 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2533247086 0 0) 0x7f89c00039c0 con 0x7f89d410e230 2024-04-14T15:16:32.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.373+0000 7f89ca7fc700 1 -- 172.21.15.90:0/3251375049 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f89d41a8c10 con 0x7f89d410e230 2024-04-14T15:16:32.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.373+0000 7f89ca7fc700 1 -- 172.21.15.90:0/3251375049 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f89cc001750 con 0x7f89d410e230 2024-04-14T15:16:32.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.373+0000 7f89ca7fc700 1 -- 172.21.15.90:0/3251375049 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 949947635 0 0) 0x7f89d41a9e50 con 0x7f89d410a690 2024-04-14T15:16:32.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.373+0000 7f89ca7fc700 1 -- 172.21.15.90:0/3251375049 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f89c00039c0 con 0x7f89d410a690 2024-04-14T15:16:32.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.373+0000 7f89ca7fc700 1 -- 172.21.15.90:0/3251375049 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f89d0004380 con 0x7f89d410a690 2024-04-14T15:16:32.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.373+0000 7f89ca7fc700 1 -- 172.21.15.90:0/3251375049 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3753544892 0 0) 0x7f89d41a79f0 con 0x7f89d41023f0 2024-04-14T15:16:32.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.373+0000 7f89ca7fc700 1 -- 172.21.15.90:0/3251375049 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f89d41a9e50 con 0x7f89d41023f0 2024-04-14T15:16:32.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.373+0000 7f89ca7fc700 1 -- 172.21.15.90:0/3251375049 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f89c40017f0 con 0x7f89d41023f0 2024-04-14T15:16:32.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.373+0000 7f89ca7fc700 1 -- 172.21.15.90:0/3251375049 <== mon.2 v1:172.21.15.90:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3105021341 0 0) 0x7f89d41a8c10 con 0x7f89d410e230 2024-04-14T15:16:32.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.373+0000 7f89ca7fc700 1 -- 172.21.15.90:0/3251375049 >> v1:172.21.15.92:6789/0 conn(0x7f89d41023f0 legacy=0x7f89d4106d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:32.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.373+0000 7f89ca7fc700 1 -- 172.21.15.90:0/3251375049 >> v1:172.21.15.90:6789/0 conn(0x7f89d410a690 legacy=0x7f89d41a2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:32.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.373+0000 7f89ca7fc700 1 -- 172.21.15.90:0/3251375049 --> v1:172.21.15.90:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89d41ab090 con 0x7f89d410e230 2024-04-14T15:16:32.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.373+0000 7f89db339700 1 -- 172.21.15.90:0/3251375049 --> v1:172.21.15.90:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f89d41aa080 con 0x7f89d410e230 2024-04-14T15:16:32.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.373+0000 7f89db339700 10 client.46997.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:16:32.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.373+0000 7f89db339700 1 -- 172.21.15.90:0/3251375049 --> v1:172.21.15.90:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f89d41aa540 con 0x7f89d410e230 2024-04-14T15:16:32.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.373+0000 7f89db339700 1 librados: init done 2024-04-14T15:16:32.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.373+0000 7f89ca7fc700 1 -- 172.21.15.90:0/3251375049 <== mon.2 v1:172.21.15.90:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f89cc0020b0 con 0x7f89d410e230 2024-04-14T15:16:32.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.373+0000 7f89ca7fc700 1 -- 172.21.15.90:0/3251375049 <== mon.2 v1:172.21.15.90:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f89cc005f00 con 0x7f89d410e230 2024-04-14T15:16:32.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.374+0000 7f89db339700 1 -- 172.21.15.90:0/3251375049 --> v1:172.21.15.90:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f89b8005100 con 0x7f89d410e230 2024-04-14T15:16:32.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.375+0000 7f89ca7fc700 1 -- 172.21.15.90:0/3251375049 <== mon.2 v1:172.21.15.90:6790/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f89cc003cb0 con 0x7f89d410e230 2024-04-14T15:16:32.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.375+0000 7f89ca7fc700 1 -- 172.21.15.90:0/3251375049 <== mon.2 v1:172.21.15.90:6790/0 8 ==== osd_map(324..324 src has 51..324) ==== 8087+0+0 (unknown 3704797461 0 0) 0x7f89cc08aab0 con 0x7f89d410e230 2024-04-14T15:16:32.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.375+0000 7f89ca7fc700 10 client.46997.objecter ms_dispatch 0x7f89d4001550 osd_map(324..324 src has 51..324) 2024-04-14T15:16:32.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.375+0000 7f89ca7fc700 3 client.46997.objecter handle_osd_map got epochs [324,324] > 0 2024-04-14T15:16:32.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.375+0000 7f89ca7fc700 3 client.46997.objecter handle_osd_map decoding full epoch 324 2024-04-14T15:16:32.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.375+0000 7f89ca7fc700 20 client.46997.objecter dump_active .. 0 homeless 2024-04-14T15:16:32.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.375+0000 7f89ca7fc700 1 -- 172.21.15.90:0/3251375049 <== mon.2 v1:172.21.15.90:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v324) ==== 54+0+16 (unknown 737359538 0 2420000192) 0x7f89cc08b060 con 0x7f89d410e230 2024-04-14T15:16:32.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.375+0000 7f89ca7fc700 10 client.46997.objecter ms_handle_connect 0x7f89c00713e0 2024-04-14T15:16:32.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.375+0000 7f89db339700 10 client.46997.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:16:32.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.375+0000 7f89db339700 15 client.46997.objecter _session_command_op_assign -1 1 2024-04-14T15:16:32.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.375+0000 7f89db339700 20 client.46997.objecter _get_session s=0x7f89b80053e0 osd=0 3 2024-04-14T15:16:32.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.375+0000 7f89db339700 20 client.46997.objecter put_session s=0x7f89b80053e0 osd=0 3 2024-04-14T15:16:32.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.375+0000 7f89db339700 20 client.46997.objecter _get_session s=0x7f89b80053e0 osd=0 3 2024-04-14T15:16:32.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.375+0000 7f89db339700 15 client.46997.objecter _session_command_op_remove -1 1 2024-04-14T15:16:32.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.375+0000 7f89db339700 20 client.46997.objecter get_session s=0x7f89b80053e0 osd=0 3 2024-04-14T15:16:32.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.375+0000 7f89db339700 15 client.46997.objecter _session_command_op_assign 0 1 2024-04-14T15:16:32.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.375+0000 7f89db339700 20 client.46997.objecter put_session s=0x7f89b80053e0 osd=0 4 2024-04-14T15:16:32.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.375+0000 7f89db339700 10 client.46997.objecter _send_command 1 2024-04-14T15:16:32.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.375+0000 7f89db339700 1 -- 172.21.15.90:0/3251375049 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f89b8004b50 con 0x7f89b8004730 2024-04-14T15:16:32.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.376+0000 7f89ca7fc700 10 client.46997.objecter ms_handle_connect 0x7f89b8004730 2024-04-14T15:16:32.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.376+0000 7f89ca7fc700 1 -- 172.21.15.90:0/3251375049 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f89b8004b50 con 0x7f89b8004730 2024-04-14T15:16:32.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.376+0000 7f89ca7fc700 10 client.46997.objecter ms_dispatch 0x7f89d4001550 command_reply(tid 1: 0 ) 2024-04-14T15:16:32.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.376+0000 7f89ca7fc700 10 client.46997.objecter _finish_command 1 = system:0 2024-04-14T15:16:32.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.376+0000 7f89ca7fc700 20 client.46997.objecter put_session s=0x7f89b80053e0 osd=0 4 2024-04-14T15:16:32.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.376+0000 7f89ca7fc700 15 client.46997.objecter _session_command_op_remove 0 1 2024-04-14T15:16:32.385 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:16:32.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.397+0000 7f89db339700 10 client.46997.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:16:32.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.397+0000 7f89db339700 15 client.46997.objecter _session_command_op_assign -1 2 2024-04-14T15:16:32.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.397+0000 7f89db339700 20 client.46997.objecter _get_session s=0x7f89b80053e0 osd=0 3 2024-04-14T15:16:32.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.397+0000 7f89db339700 20 client.46997.objecter put_session s=0x7f89b80053e0 osd=0 3 2024-04-14T15:16:32.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.397+0000 7f89db339700 20 client.46997.objecter _get_session s=0x7f89b80053e0 osd=0 3 2024-04-14T15:16:32.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.397+0000 7f89db339700 15 client.46997.objecter _session_command_op_remove -1 2 2024-04-14T15:16:32.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.397+0000 7f89db339700 20 client.46997.objecter get_session s=0x7f89b80053e0 osd=0 3 2024-04-14T15:16:32.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.397+0000 7f89db339700 15 client.46997.objecter _session_command_op_assign 0 2 2024-04-14T15:16:32.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.397+0000 7f89db339700 20 client.46997.objecter put_session s=0x7f89b80053e0 osd=0 4 2024-04-14T15:16:32.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.397+0000 7f89db339700 10 client.46997.objecter _send_command 2 2024-04-14T15:16:32.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.397+0000 7f89db339700 1 -- 172.21.15.90:0/3251375049 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f89b80079f0 con 0x7f89b8004730 2024-04-14T15:16:32.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.400+0000 7f89ca7fc700 1 -- 172.21.15.90:0/3251375049 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f89b80079f0 con 0x7f89b8004730 2024-04-14T15:16:32.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.400+0000 7f89ca7fc700 10 client.46997.objecter ms_dispatch 0x7f89d4001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:16:32.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.400+0000 7f89ca7fc700 10 client.46997.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:16:32.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.400+0000 7f89ca7fc700 20 client.46997.objecter put_session s=0x7f89b80053e0 osd=0 4 2024-04-14T15:16:32.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.400+0000 7f89ca7fc700 15 client.46997.objecter _session_command_op_remove 0 2 2024-04-14T15:16:32.401 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:16:32.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.400+0000 7f89db339700 10 client.46997.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:16:32.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.400+0000 7f89db339700 15 client.46997.objecter _session_command_op_assign -1 3 2024-04-14T15:16:32.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.400+0000 7f89db339700 20 client.46997.objecter _get_session s=0x7f89b8004110 osd=1 3 2024-04-14T15:16:32.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.400+0000 7f89db339700 20 client.46997.objecter put_session s=0x7f89b8004110 osd=1 3 2024-04-14T15:16:32.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.400+0000 7f89db339700 20 client.46997.objecter _get_session s=0x7f89b8004110 osd=1 3 2024-04-14T15:16:32.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.400+0000 7f89db339700 15 client.46997.objecter _session_command_op_remove -1 3 2024-04-14T15:16:32.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.400+0000 7f89db339700 20 client.46997.objecter get_session s=0x7f89b8004110 osd=1 3 2024-04-14T15:16:32.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.400+0000 7f89db339700 15 client.46997.objecter _session_command_op_assign 1 3 2024-04-14T15:16:32.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.400+0000 7f89db339700 20 client.46997.objecter put_session s=0x7f89b8004110 osd=1 4 2024-04-14T15:16:32.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.400+0000 7f89db339700 10 client.46997.objecter _send_command 3 2024-04-14T15:16:32.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.400+0000 7f89db339700 1 -- 172.21.15.90:0/3251375049 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f89b8002180 con 0x7f89b8004280 2024-04-14T15:16:32.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.400+0000 7f89ca7fc700 10 client.46997.objecter ms_handle_connect 0x7f89b8004280 2024-04-14T15:16:32.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.401+0000 7f89ca7fc700 1 -- 172.21.15.90:0/3251375049 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f89b8002180 con 0x7f89b8004280 2024-04-14T15:16:32.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.401+0000 7f89ca7fc700 10 client.46997.objecter ms_dispatch 0x7f89d4001550 command_reply(tid 3: 0 ) 2024-04-14T15:16:32.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.401+0000 7f89ca7fc700 10 client.46997.objecter _finish_command 3 = system:0 2024-04-14T15:16:32.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.401+0000 7f89ca7fc700 20 client.46997.objecter put_session s=0x7f89b8004110 osd=1 4 2024-04-14T15:16:32.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.401+0000 7f89ca7fc700 15 client.46997.objecter _session_command_op_remove 1 3 2024-04-14T15:16:32.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.421+0000 7f89db339700 10 client.46997.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:16:32.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.421+0000 7f89db339700 15 client.46997.objecter _session_command_op_assign -1 4 2024-04-14T15:16:32.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.421+0000 7f89db339700 20 client.46997.objecter _get_session s=0x7f89b8004110 osd=1 3 2024-04-14T15:16:32.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.421+0000 7f89db339700 20 client.46997.objecter put_session s=0x7f89b8004110 osd=1 3 2024-04-14T15:16:32.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.421+0000 7f89db339700 20 client.46997.objecter _get_session s=0x7f89b8004110 osd=1 3 2024-04-14T15:16:32.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.421+0000 7f89db339700 15 client.46997.objecter _session_command_op_remove -1 4 2024-04-14T15:16:32.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.421+0000 7f89db339700 20 client.46997.objecter get_session s=0x7f89b8004110 osd=1 3 2024-04-14T15:16:32.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.421+0000 7f89db339700 15 client.46997.objecter _session_command_op_assign 1 4 2024-04-14T15:16:32.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.421+0000 7f89db339700 20 client.46997.objecter put_session s=0x7f89b8004110 osd=1 4 2024-04-14T15:16:32.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.421+0000 7f89db339700 10 client.46997.objecter _send_command 4 2024-04-14T15:16:32.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.421+0000 7f89db339700 1 -- 172.21.15.90:0/3251375049 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f89b8001600 con 0x7f89b8004280 2024-04-14T15:16:32.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.423+0000 7f89ca7fc700 1 -- 172.21.15.90:0/3251375049 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f89b8001600 con 0x7f89b8004280 2024-04-14T15:16:32.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.423+0000 7f89ca7fc700 10 client.46997.objecter ms_dispatch 0x7f89d4001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:16:32.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.423+0000 7f89ca7fc700 10 client.46997.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:16:32.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.423+0000 7f89ca7fc700 20 client.46997.objecter put_session s=0x7f89b8004110 osd=1 4 2024-04-14T15:16:32.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.423+0000 7f89ca7fc700 15 client.46997.objecter _session_command_op_remove 1 4 2024-04-14T15:16:32.424 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:16:32.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.424+0000 7f89db339700 10 client.46997.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:16:32.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.424+0000 7f89db339700 15 client.46997.objecter _session_command_op_assign -1 5 2024-04-14T15:16:32.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.424+0000 7f89db339700 20 client.46997.objecter _get_session s=0x7f89b8008370 osd=2 3 2024-04-14T15:16:32.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.424+0000 7f89db339700 20 client.46997.objecter put_session s=0x7f89b8008370 osd=2 3 2024-04-14T15:16:32.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.424+0000 7f89db339700 20 client.46997.objecter _get_session s=0x7f89b8008370 osd=2 3 2024-04-14T15:16:32.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.424+0000 7f89db339700 15 client.46997.objecter _session_command_op_remove -1 5 2024-04-14T15:16:32.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.424+0000 7f89db339700 20 client.46997.objecter get_session s=0x7f89b8008370 osd=2 3 2024-04-14T15:16:32.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.424+0000 7f89db339700 15 client.46997.objecter _session_command_op_assign 2 5 2024-04-14T15:16:32.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.424+0000 7f89db339700 20 client.46997.objecter put_session s=0x7f89b8008370 osd=2 4 2024-04-14T15:16:32.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.424+0000 7f89db339700 10 client.46997.objecter _send_command 5 2024-04-14T15:16:32.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.424+0000 7f89db339700 1 -- 172.21.15.90:0/3251375049 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f89b80154f0 con 0x7f89b80119e0 2024-04-14T15:16:32.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.424+0000 7f89ca7fc700 10 client.46997.objecter ms_handle_connect 0x7f89b80119e0 2024-04-14T15:16:32.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.425+0000 7f89ca7fc700 1 -- 172.21.15.90:0/3251375049 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f89b80154f0 con 0x7f89b80119e0 2024-04-14T15:16:32.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.425+0000 7f89ca7fc700 10 client.46997.objecter ms_dispatch 0x7f89d4001550 command_reply(tid 5: 0 ) 2024-04-14T15:16:32.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.425+0000 7f89ca7fc700 10 client.46997.objecter _finish_command 5 = system:0 2024-04-14T15:16:32.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.425+0000 7f89ca7fc700 20 client.46997.objecter put_session s=0x7f89b8008370 osd=2 4 2024-04-14T15:16:32.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.425+0000 7f89ca7fc700 15 client.46997.objecter _session_command_op_remove 2 5 2024-04-14T15:16:32.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.443+0000 7f89db339700 10 client.46997.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:16:32.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.443+0000 7f89db339700 15 client.46997.objecter _session_command_op_assign -1 6 2024-04-14T15:16:32.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.443+0000 7f89db339700 20 client.46997.objecter _get_session s=0x7f89b8008370 osd=2 3 2024-04-14T15:16:32.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.443+0000 7f89db339700 20 client.46997.objecter put_session s=0x7f89b8008370 osd=2 3 2024-04-14T15:16:32.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.443+0000 7f89db339700 20 client.46997.objecter _get_session s=0x7f89b8008370 osd=2 3 2024-04-14T15:16:32.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.443+0000 7f89db339700 15 client.46997.objecter _session_command_op_remove -1 6 2024-04-14T15:16:32.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.443+0000 7f89db339700 20 client.46997.objecter get_session s=0x7f89b8008370 osd=2 3 2024-04-14T15:16:32.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.443+0000 7f89db339700 15 client.46997.objecter _session_command_op_assign 2 6 2024-04-14T15:16:32.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.443+0000 7f89db339700 20 client.46997.objecter put_session s=0x7f89b8008370 osd=2 4 2024-04-14T15:16:32.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.443+0000 7f89db339700 10 client.46997.objecter _send_command 6 2024-04-14T15:16:32.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.443+0000 7f89db339700 1 -- 172.21.15.90:0/3251375049 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f89b8007df0 con 0x7f89b80119e0 2024-04-14T15:16:32.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.445+0000 7f89ca7fc700 1 -- 172.21.15.90:0/3251375049 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f89b8007df0 con 0x7f89b80119e0 2024-04-14T15:16:32.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.445+0000 7f89ca7fc700 10 client.46997.objecter ms_dispatch 0x7f89d4001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:16:32.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.445+0000 7f89ca7fc700 10 client.46997.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:16:32.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.445+0000 7f89ca7fc700 20 client.46997.objecter put_session s=0x7f89b8008370 osd=2 4 2024-04-14T15:16:32.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.445+0000 7f89ca7fc700 15 client.46997.objecter _session_command_op_remove 2 6 2024-04-14T15:16:32.445 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:16:32.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.445+0000 7f89db339700 10 client.46997.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:16:32.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.445+0000 7f89db339700 15 client.46997.objecter _session_command_op_assign -1 7 2024-04-14T15:16:32.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.445+0000 7f89db339700 20 client.46997.objecter _get_session s=0x7f89b80081f0 osd=3 3 2024-04-14T15:16:32.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.445+0000 7f89db339700 20 client.46997.objecter put_session s=0x7f89b80081f0 osd=3 3 2024-04-14T15:16:32.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.445+0000 7f89db339700 20 client.46997.objecter _get_session s=0x7f89b80081f0 osd=3 3 2024-04-14T15:16:32.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.445+0000 7f89db339700 15 client.46997.objecter _session_command_op_remove -1 7 2024-04-14T15:16:32.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.445+0000 7f89db339700 20 client.46997.objecter get_session s=0x7f89b80081f0 osd=3 3 2024-04-14T15:16:32.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.445+0000 7f89db339700 15 client.46997.objecter _session_command_op_assign 3 7 2024-04-14T15:16:32.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.445+0000 7f89db339700 20 client.46997.objecter put_session s=0x7f89b80081f0 osd=3 4 2024-04-14T15:16:32.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.445+0000 7f89db339700 10 client.46997.objecter _send_command 7 2024-04-14T15:16:32.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.445+0000 7f89db339700 1 -- 172.21.15.90:0/3251375049 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f89b801a0c0 con 0x7f89b800d160 2024-04-14T15:16:32.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.445+0000 7f89ca7fc700 10 client.46997.objecter ms_handle_connect 0x7f89b800d160 2024-04-14T15:16:32.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.446+0000 7f89ca7fc700 1 -- 172.21.15.90:0/3251375049 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f89b801a0c0 con 0x7f89b800d160 2024-04-14T15:16:32.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.446+0000 7f89ca7fc700 10 client.46997.objecter ms_dispatch 0x7f89d4001550 command_reply(tid 7: 0 ) 2024-04-14T15:16:32.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.446+0000 7f89ca7fc700 10 client.46997.objecter _finish_command 7 = system:0 2024-04-14T15:16:32.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.446+0000 7f89ca7fc700 20 client.46997.objecter put_session s=0x7f89b80081f0 osd=3 4 2024-04-14T15:16:32.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.446+0000 7f89ca7fc700 15 client.46997.objecter _session_command_op_remove 3 7 2024-04-14T15:16:32.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.462+0000 7f89db339700 10 client.46997.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:16:32.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.462+0000 7f89db339700 15 client.46997.objecter _session_command_op_assign -1 8 2024-04-14T15:16:32.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.462+0000 7f89db339700 20 client.46997.objecter _get_session s=0x7f89b80081f0 osd=3 3 2024-04-14T15:16:32.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.462+0000 7f89db339700 20 client.46997.objecter put_session s=0x7f89b80081f0 osd=3 3 2024-04-14T15:16:32.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.462+0000 7f89db339700 20 client.46997.objecter _get_session s=0x7f89b80081f0 osd=3 3 2024-04-14T15:16:32.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.462+0000 7f89db339700 15 client.46997.objecter _session_command_op_remove -1 8 2024-04-14T15:16:32.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.462+0000 7f89db339700 20 client.46997.objecter get_session s=0x7f89b80081f0 osd=3 3 2024-04-14T15:16:32.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.462+0000 7f89db339700 15 client.46997.objecter _session_command_op_assign 3 8 2024-04-14T15:16:32.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.462+0000 7f89db339700 20 client.46997.objecter put_session s=0x7f89b80081f0 osd=3 4 2024-04-14T15:16:32.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.462+0000 7f89db339700 10 client.46997.objecter _send_command 8 2024-04-14T15:16:32.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.462+0000 7f89db339700 1 -- 172.21.15.90:0/3251375049 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f89b800c7c0 con 0x7f89b800d160 2024-04-14T15:16:32.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.464+0000 7f89ca7fc700 1 -- 172.21.15.90:0/3251375049 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f89b800c7c0 con 0x7f89b800d160 2024-04-14T15:16:32.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.464+0000 7f89ca7fc700 10 client.46997.objecter ms_dispatch 0x7f89d4001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:16:32.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.464+0000 7f89ca7fc700 10 client.46997.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:16:32.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.464+0000 7f89ca7fc700 20 client.46997.objecter put_session s=0x7f89b80081f0 osd=3 4 2024-04-14T15:16:32.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.464+0000 7f89ca7fc700 15 client.46997.objecter _session_command_op_remove 3 8 2024-04-14T15:16:32.466 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:16:32.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.464+0000 7f89db339700 10 client.46997.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:16:32.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.464+0000 7f89db339700 15 client.46997.objecter _session_command_op_assign -1 9 2024-04-14T15:16:32.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.464+0000 7f89db339700 20 client.46997.objecter _get_session s=0x7f89b800c7c0 osd=4 3 2024-04-14T15:16:32.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.464+0000 7f89db339700 20 client.46997.objecter put_session s=0x7f89b800c7c0 osd=4 3 2024-04-14T15:16:32.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.464+0000 7f89db339700 20 client.46997.objecter _get_session s=0x7f89b800c7c0 osd=4 3 2024-04-14T15:16:32.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.464+0000 7f89db339700 15 client.46997.objecter _session_command_op_remove -1 9 2024-04-14T15:16:32.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.464+0000 7f89db339700 20 client.46997.objecter get_session s=0x7f89b800c7c0 osd=4 3 2024-04-14T15:16:32.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.464+0000 7f89db339700 15 client.46997.objecter _session_command_op_assign 4 9 2024-04-14T15:16:32.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.464+0000 7f89db339700 20 client.46997.objecter put_session s=0x7f89b800c7c0 osd=4 4 2024-04-14T15:16:32.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.464+0000 7f89db339700 10 client.46997.objecter _send_command 9 2024-04-14T15:16:32.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.464+0000 7f89db339700 1 -- 172.21.15.90:0/3251375049 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f89b801ec70 con 0x7f89b8016380 2024-04-14T15:16:32.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.465+0000 7f89ca7fc700 10 client.46997.objecter ms_handle_connect 0x7f89b8016380 2024-04-14T15:16:32.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.466+0000 7f89ca7fc700 1 -- 172.21.15.90:0/3251375049 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f89b801ec70 con 0x7f89b8016380 2024-04-14T15:16:32.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.466+0000 7f89ca7fc700 10 client.46997.objecter ms_dispatch 0x7f89d4001550 command_reply(tid 9: 0 ) 2024-04-14T15:16:32.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.466+0000 7f89ca7fc700 10 client.46997.objecter _finish_command 9 = system:0 2024-04-14T15:16:32.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.466+0000 7f89ca7fc700 20 client.46997.objecter put_session s=0x7f89b800c7c0 osd=4 4 2024-04-14T15:16:32.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.466+0000 7f89ca7fc700 15 client.46997.objecter _session_command_op_remove 4 9 2024-04-14T15:16:32.470 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:16:32.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.482+0000 7f89db339700 10 client.46997.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:16:32.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.482+0000 7f89db339700 15 client.46997.objecter _session_command_op_assign -1 10 2024-04-14T15:16:32.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.482+0000 7f89db339700 20 client.46997.objecter _get_session s=0x7f89b800c7c0 osd=4 3 2024-04-14T15:16:32.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.482+0000 7f89db339700 20 client.46997.objecter put_session s=0x7f89b800c7c0 osd=4 3 2024-04-14T15:16:32.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.482+0000 7f89db339700 20 client.46997.objecter _get_session s=0x7f89b800c7c0 osd=4 3 2024-04-14T15:16:32.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.482+0000 7f89db339700 15 client.46997.objecter _session_command_op_remove -1 10 2024-04-14T15:16:32.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.482+0000 7f89db339700 20 client.46997.objecter get_session s=0x7f89b800c7c0 osd=4 3 2024-04-14T15:16:32.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.482+0000 7f89db339700 15 client.46997.objecter _session_command_op_assign 4 10 2024-04-14T15:16:32.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.482+0000 7f89db339700 20 client.46997.objecter put_session s=0x7f89b800c7c0 osd=4 4 2024-04-14T15:16:32.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.482+0000 7f89db339700 10 client.46997.objecter _send_command 10 2024-04-14T15:16:32.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.482+0000 7f89db339700 1 -- 172.21.15.90:0/3251375049 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f89b801ec70 con 0x7f89b8016380 2024-04-14T15:16:32.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.484+0000 7f89ca7fc700 1 -- 172.21.15.90:0/3251375049 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f89b801ec70 con 0x7f89b8016380 2024-04-14T15:16:32.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.484+0000 7f89ca7fc700 10 client.46997.objecter ms_dispatch 0x7f89d4001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:16:32.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.484+0000 7f89ca7fc700 10 client.46997.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:16:32.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.484+0000 7f89ca7fc700 20 client.46997.objecter put_session s=0x7f89b800c7c0 osd=4 4 2024-04-14T15:16:32.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.484+0000 7f89ca7fc700 15 client.46997.objecter _session_command_op_remove 4 10 2024-04-14T15:16:32.485 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:16:32.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.484+0000 7f89db339700 10 client.46997.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:16:32.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.484+0000 7f89db339700 15 client.46997.objecter _session_command_op_assign -1 11 2024-04-14T15:16:32.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.484+0000 7f89db339700 20 client.46997.objecter _get_session s=0x7f89b801ec70 osd=5 3 2024-04-14T15:16:32.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.484+0000 7f89db339700 20 client.46997.objecter put_session s=0x7f89b801ec70 osd=5 3 2024-04-14T15:16:32.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.484+0000 7f89db339700 20 client.46997.objecter _get_session s=0x7f89b801ec70 osd=5 3 2024-04-14T15:16:32.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.484+0000 7f89db339700 15 client.46997.objecter _session_command_op_remove -1 11 2024-04-14T15:16:32.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.484+0000 7f89db339700 20 client.46997.objecter get_session s=0x7f89b801ec70 osd=5 3 2024-04-14T15:16:32.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.484+0000 7f89db339700 15 client.46997.objecter _session_command_op_assign 5 11 2024-04-14T15:16:32.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.484+0000 7f89db339700 20 client.46997.objecter put_session s=0x7f89b801ec70 osd=5 4 2024-04-14T15:16:32.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.484+0000 7f89db339700 10 client.46997.objecter _send_command 11 2024-04-14T15:16:32.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.484+0000 7f89db339700 1 -- 172.21.15.90:0/3251375049 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f89b8023010 con 0x7f89b801f2f0 2024-04-14T15:16:32.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.485+0000 7f89ca7fc700 10 client.46997.objecter ms_handle_connect 0x7f89b801f2f0 2024-04-14T15:16:32.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.486+0000 7f89ca7fc700 1 -- 172.21.15.90:0/3251375049 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f89b8023010 con 0x7f89b801f2f0 2024-04-14T15:16:32.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.486+0000 7f89ca7fc700 10 client.46997.objecter ms_dispatch 0x7f89d4001550 command_reply(tid 11: 0 ) 2024-04-14T15:16:32.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.486+0000 7f89ca7fc700 10 client.46997.objecter _finish_command 11 = system:0 2024-04-14T15:16:32.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.486+0000 7f89ca7fc700 20 client.46997.objecter put_session s=0x7f89b801ec70 osd=5 4 2024-04-14T15:16:32.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.486+0000 7f89ca7fc700 15 client.46997.objecter _session_command_op_remove 5 11 2024-04-14T15:16:32.493 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:32.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:32.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:16:32.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.504+0000 7f89db339700 10 client.46997.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:16:32.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.504+0000 7f89db339700 15 client.46997.objecter _session_command_op_assign -1 12 2024-04-14T15:16:32.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.504+0000 7f89db339700 20 client.46997.objecter _get_session s=0x7f89b801ec70 osd=5 3 2024-04-14T15:16:32.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.504+0000 7f89db339700 20 client.46997.objecter put_session s=0x7f89b801ec70 osd=5 3 2024-04-14T15:16:32.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.504+0000 7f89db339700 20 client.46997.objecter _get_session s=0x7f89b801ec70 osd=5 3 2024-04-14T15:16:32.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.504+0000 7f89db339700 15 client.46997.objecter _session_command_op_remove -1 12 2024-04-14T15:16:32.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.504+0000 7f89db339700 20 client.46997.objecter get_session s=0x7f89b801ec70 osd=5 3 2024-04-14T15:16:32.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.504+0000 7f89db339700 15 client.46997.objecter _session_command_op_assign 5 12 2024-04-14T15:16:32.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.504+0000 7f89db339700 20 client.46997.objecter put_session s=0x7f89b801ec70 osd=5 4 2024-04-14T15:16:32.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.504+0000 7f89db339700 10 client.46997.objecter _send_command 12 2024-04-14T15:16:32.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.504+0000 7f89db339700 1 -- 172.21.15.90:0/3251375049 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f89b80231f0 con 0x7f89b801f2f0 2024-04-14T15:16:32.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.506+0000 7f89ca7fc700 1 -- 172.21.15.90:0/3251375049 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f89b80231f0 con 0x7f89b801f2f0 2024-04-14T15:16:32.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.506+0000 7f89ca7fc700 10 client.46997.objecter ms_dispatch 0x7f89d4001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:16:32.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.506+0000 7f89ca7fc700 10 client.46997.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:16:32.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.506+0000 7f89ca7fc700 20 client.46997.objecter put_session s=0x7f89b801ec70 osd=5 4 2024-04-14T15:16:32.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.506+0000 7f89ca7fc700 15 client.46997.objecter _session_command_op_remove 5 12 2024-04-14T15:16:32.511 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:16:32.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.506+0000 7f89db339700 10 client.46997.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:16:32.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.506+0000 7f89db339700 15 client.46997.objecter _session_command_op_assign -1 13 2024-04-14T15:16:32.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.506+0000 7f89db339700 20 client.46997.objecter _get_session s=0x7f89b80231f0 osd=6 3 2024-04-14T15:16:32.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.506+0000 7f89db339700 20 client.46997.objecter put_session s=0x7f89b80231f0 osd=6 3 2024-04-14T15:16:32.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.506+0000 7f89db339700 20 client.46997.objecter _get_session s=0x7f89b80231f0 osd=6 3 2024-04-14T15:16:32.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.506+0000 7f89db339700 15 client.46997.objecter _session_command_op_remove -1 13 2024-04-14T15:16:32.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.506+0000 7f89db339700 20 client.46997.objecter get_session s=0x7f89b80231f0 osd=6 3 2024-04-14T15:16:32.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.506+0000 7f89db339700 15 client.46997.objecter _session_command_op_assign 6 13 2024-04-14T15:16:32.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.506+0000 7f89db339700 20 client.46997.objecter put_session s=0x7f89b80231f0 osd=6 4 2024-04-14T15:16:32.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.506+0000 7f89db339700 10 client.46997.objecter _send_command 13 2024-04-14T15:16:32.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.506+0000 7f89db339700 1 -- 172.21.15.90:0/3251375049 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f89b80244f0 con 0x7f89b8023910 2024-04-14T15:16:32.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.507+0000 7f89ca7fc700 10 client.46997.objecter ms_handle_connect 0x7f89b8023910 2024-04-14T15:16:32.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.508+0000 7f89ca7fc700 1 -- 172.21.15.90:0/3251375049 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f89b80244f0 con 0x7f89b8023910 2024-04-14T15:16:32.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.508+0000 7f89ca7fc700 10 client.46997.objecter ms_dispatch 0x7f89d4001550 command_reply(tid 13: 0 ) 2024-04-14T15:16:32.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.508+0000 7f89ca7fc700 10 client.46997.objecter _finish_command 13 = system:0 2024-04-14T15:16:32.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.508+0000 7f89ca7fc700 20 client.46997.objecter put_session s=0x7f89b80231f0 osd=6 4 2024-04-14T15:16:32.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.508+0000 7f89ca7fc700 15 client.46997.objecter _session_command_op_remove 6 13 2024-04-14T15:16:32.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.524+0000 7f89db339700 10 client.46997.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:16:32.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.524+0000 7f89db339700 15 client.46997.objecter _session_command_op_assign -1 14 2024-04-14T15:16:32.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.524+0000 7f89db339700 20 client.46997.objecter _get_session s=0x7f89b80231f0 osd=6 3 2024-04-14T15:16:32.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.524+0000 7f89db339700 20 client.46997.objecter put_session s=0x7f89b80231f0 osd=6 3 2024-04-14T15:16:32.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.524+0000 7f89db339700 20 client.46997.objecter _get_session s=0x7f89b80231f0 osd=6 3 2024-04-14T15:16:32.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.524+0000 7f89db339700 15 client.46997.objecter _session_command_op_remove -1 14 2024-04-14T15:16:32.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.524+0000 7f89db339700 20 client.46997.objecter get_session s=0x7f89b80231f0 osd=6 3 2024-04-14T15:16:32.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.524+0000 7f89db339700 15 client.46997.objecter _session_command_op_assign 6 14 2024-04-14T15:16:32.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.524+0000 7f89db339700 20 client.46997.objecter put_session s=0x7f89b80231f0 osd=6 4 2024-04-14T15:16:32.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.524+0000 7f89db339700 10 client.46997.objecter _send_command 14 2024-04-14T15:16:32.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.524+0000 7f89db339700 1 -- 172.21.15.90:0/3251375049 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f89b80279e0 con 0x7f89b8023910 2024-04-14T15:16:32.526 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:16:32.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.526+0000 7f89ca7fc700 1 -- 172.21.15.90:0/3251375049 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f89b80279e0 con 0x7f89b8023910 2024-04-14T15:16:32.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.526+0000 7f89ca7fc700 10 client.46997.objecter ms_dispatch 0x7f89d4001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:16:32.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.526+0000 7f89ca7fc700 10 client.46997.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:16:32.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.526+0000 7f89ca7fc700 20 client.46997.objecter put_session s=0x7f89b80231f0 osd=6 4 2024-04-14T15:16:32.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.526+0000 7f89ca7fc700 15 client.46997.objecter _session_command_op_remove 6 14 2024-04-14T15:16:32.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.526+0000 7f89db339700 10 client.46997.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:16:32.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.526+0000 7f89db339700 15 client.46997.objecter _session_command_op_assign -1 15 2024-04-14T15:16:32.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.526+0000 7f89db339700 20 client.46997.objecter _get_session s=0x7f89b80279e0 osd=7 3 2024-04-14T15:16:32.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.526+0000 7f89db339700 20 client.46997.objecter put_session s=0x7f89b80279e0 osd=7 3 2024-04-14T15:16:32.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.526+0000 7f89db339700 20 client.46997.objecter _get_session s=0x7f89b80279e0 osd=7 3 2024-04-14T15:16:32.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.526+0000 7f89db339700 15 client.46997.objecter _session_command_op_remove -1 15 2024-04-14T15:16:32.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.526+0000 7f89db339700 20 client.46997.objecter get_session s=0x7f89b80279e0 osd=7 3 2024-04-14T15:16:32.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.526+0000 7f89db339700 15 client.46997.objecter _session_command_op_assign 7 15 2024-04-14T15:16:32.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.526+0000 7f89db339700 20 client.46997.objecter put_session s=0x7f89b80279e0 osd=7 4 2024-04-14T15:16:32.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.526+0000 7f89db339700 10 client.46997.objecter _send_command 15 2024-04-14T15:16:32.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.526+0000 7f89db339700 1 -- 172.21.15.90:0/3251375049 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f89b802bc90 con 0x7f89b8028100 2024-04-14T15:16:32.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.527+0000 7f89ca7fc700 10 client.46997.objecter ms_handle_connect 0x7f89b8028100 2024-04-14T15:16:32.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.528+0000 7f89ca7fc700 1 -- 172.21.15.90:0/3251375049 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f89b802bc90 con 0x7f89b8028100 2024-04-14T15:16:32.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.528+0000 7f89ca7fc700 10 client.46997.objecter ms_dispatch 0x7f89d4001550 command_reply(tid 15: 0 ) 2024-04-14T15:16:32.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.528+0000 7f89ca7fc700 10 client.46997.objecter _finish_command 15 = system:0 2024-04-14T15:16:32.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.528+0000 7f89ca7fc700 20 client.46997.objecter put_session s=0x7f89b80279e0 osd=7 4 2024-04-14T15:16:32.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.528+0000 7f89ca7fc700 15 client.46997.objecter _session_command_op_remove 7 15 2024-04-14T15:16:32.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.546+0000 7f89db339700 10 client.46997.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:16:32.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.546+0000 7f89db339700 15 client.46997.objecter _session_command_op_assign -1 16 2024-04-14T15:16:32.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.546+0000 7f89db339700 20 client.46997.objecter _get_session s=0x7f89b80279e0 osd=7 3 2024-04-14T15:16:32.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.546+0000 7f89db339700 20 client.46997.objecter put_session s=0x7f89b80279e0 osd=7 3 2024-04-14T15:16:32.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.546+0000 7f89db339700 20 client.46997.objecter _get_session s=0x7f89b80279e0 osd=7 3 2024-04-14T15:16:32.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.546+0000 7f89db339700 15 client.46997.objecter _session_command_op_remove -1 16 2024-04-14T15:16:32.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.546+0000 7f89db339700 20 client.46997.objecter get_session s=0x7f89b80279e0 osd=7 3 2024-04-14T15:16:32.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.546+0000 7f89db339700 15 client.46997.objecter _session_command_op_assign 7 16 2024-04-14T15:16:32.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.546+0000 7f89db339700 20 client.46997.objecter put_session s=0x7f89b80279e0 osd=7 4 2024-04-14T15:16:32.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.546+0000 7f89db339700 10 client.46997.objecter _send_command 16 2024-04-14T15:16:32.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.546+0000 7f89db339700 1 -- 172.21.15.90:0/3251375049 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f89b802be70 con 0x7f89b8028100 2024-04-14T15:16:32.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.548+0000 7f89ca7fc700 1 -- 172.21.15.90:0/3251375049 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f89b802be70 con 0x7f89b8028100 2024-04-14T15:16:32.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.548+0000 7f89ca7fc700 10 client.46997.objecter ms_dispatch 0x7f89d4001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:16:32.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.548+0000 7f89ca7fc700 10 client.46997.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:16:32.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.548+0000 7f89ca7fc700 20 client.46997.objecter put_session s=0x7f89b80279e0 osd=7 4 2024-04-14T15:16:32.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.548+0000 7f89ca7fc700 15 client.46997.objecter _session_command_op_remove 7 16 2024-04-14T15:16:32.549 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:16:32.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.549+0000 7f89db339700 10 librados: watch_flush enter 2024-04-14T15:16:32.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.549+0000 7f89db339700 10 librados: watch_flush exit 2024-04-14T15:16:32.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.549+0000 7f89db339700 10 client.46997.objecter close_session for osd.0 2024-04-14T15:16:32.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.549+0000 7f89db339700 1 -- 172.21.15.90:0/3251375049 >> v1:172.21.15.90:6801/3761806885 conn(0x7f89b8004730 legacy=0x7f89b8002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:32.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.549+0000 7f89db339700 20 client.46997.objecter put_session s=0x7f89b80053e0 osd=0 1 2024-04-14T15:16:32.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.549+0000 7f89db339700 10 client.46997.objecter close_session for osd.1 2024-04-14T15:16:32.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.549+0000 7f89db339700 1 -- 172.21.15.90:0/3251375049 >> v1:172.21.15.90:6805/4170361699 conn(0x7f89b8004280 legacy=0x7f89b800ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:32.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.549+0000 7f89db339700 20 client.46997.objecter put_session s=0x7f89b8004110 osd=1 1 2024-04-14T15:16:32.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.549+0000 7f89db339700 10 client.46997.objecter close_session for osd.2 2024-04-14T15:16:32.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.549+0000 7f89db339700 1 -- 172.21.15.90:0/3251375049 >> v1:172.21.15.90:6809/1686184251 conn(0x7f89b80119e0 legacy=0x7f89b8013dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:32.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.549+0000 7f89db339700 20 client.46997.objecter put_session s=0x7f89b8008370 osd=2 1 2024-04-14T15:16:32.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.549+0000 7f89db339700 10 client.46997.objecter close_session for osd.3 2024-04-14T15:16:32.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.549+0000 7f89db339700 1 -- 172.21.15.90:0/3251375049 >> v1:172.21.15.90:6813/2270166774 conn(0x7f89b800d160 legacy=0x7f89b8018830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:32.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.549+0000 7f89db339700 20 client.46997.objecter put_session s=0x7f89b80081f0 osd=3 1 2024-04-14T15:16:32.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.549+0000 7f89db339700 10 client.46997.objecter close_session for osd.4 2024-04-14T15:16:32.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.549+0000 7f89db339700 1 -- 172.21.15.90:0/3251375049 >> v1:172.21.15.92:6800/3887549733 conn(0x7f89b8016380 legacy=0x7f89b801a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:32.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.549+0000 7f89d98d6700 1 -- 172.21.15.90:0/3251375049 reap_dead start 2024-04-14T15:16:32.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.549+0000 7f89db339700 20 client.46997.objecter put_session s=0x7f89b800c7c0 osd=4 1 2024-04-14T15:16:32.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.549+0000 7f89db339700 10 client.46997.objecter close_session for osd.5 2024-04-14T15:16:32.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.549+0000 7f89db339700 1 -- 172.21.15.90:0/3251375049 >> v1:172.21.15.92:6804/3244561440 conn(0x7f89b801f2f0 legacy=0x7f89b8021720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:32.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.549+0000 7f89d98d6700 1 -- 172.21.15.90:0/3251375049 reap_dead start 2024-04-14T15:16:32.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.549+0000 7f89db339700 20 client.46997.objecter put_session s=0x7f89b801ec70 osd=5 1 2024-04-14T15:16:32.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.549+0000 7f89db339700 10 client.46997.objecter close_session for osd.6 2024-04-14T15:16:32.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.549+0000 7f89db339700 1 -- 172.21.15.90:0/3251375049 >> v1:172.21.15.92:6808/636125963 conn(0x7f89b8023910 legacy=0x7f89b8023d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:32.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.549+0000 7f89db339700 20 client.46997.objecter put_session s=0x7f89b80231f0 osd=6 1 2024-04-14T15:16:32.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.549+0000 7f89db339700 10 client.46997.objecter close_session for osd.7 2024-04-14T15:16:32.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.549+0000 7f89d98d6700 1 -- 172.21.15.90:0/3251375049 reap_dead start 2024-04-14T15:16:32.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.549+0000 7f89db339700 1 -- 172.21.15.90:0/3251375049 >> v1:172.21.15.92:6812/1773706716 conn(0x7f89b8028100 legacy=0x7f89b802a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:32.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.549+0000 7f89db339700 20 client.46997.objecter put_session s=0x7f89b80279e0 osd=7 1 2024-04-14T15:16:32.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.549+0000 7f89db339700 20 client.46997.objecter shutdown clearing up homeless session... 2024-04-14T15:16:32.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.549+0000 7f89db339700 10 client.46997.objecter successfully canceled tick 2024-04-14T15:16:32.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.549+0000 7f89db339700 1 -- 172.21.15.90:0/3251375049 >> v1:172.21.15.90:6800/420935427 conn(0x7f89c00713e0 legacy=0x7f89c00738a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:32.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.549+0000 7f89db339700 1 -- 172.21.15.90:0/3251375049 >> v1:172.21.15.90:6790/0 conn(0x7f89d410e230 legacy=0x7f89d41a60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:32.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.549+0000 7f89db339700 1 -- 172.21.15.90:0/3251375049 shutdown_connections 2024-04-14T15:16:32.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.549+0000 7f89db339700 1 -- 172.21.15.90:0/3251375049 >> 172.21.15.90:0/3251375049 conn(0x7f89d4076aa0 msgr2=0x7f89d4111a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:16:32.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.549+0000 7f89db339700 1 -- 172.21.15.90:0/3251375049 shutdown_connections 2024-04-14T15:16:32.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.549+0000 7f89db339700 1 -- 172.21.15.90:0/3251375049 wait complete. 2024-04-14T15:16:32.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:32.549+0000 7f89db339700 1 librados: shutdown 2024-04-14T15:16:32.554 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:16:32.554 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:16:32.555 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:16:32.555 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:16:32.555 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:16:32.555 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:16:32.555 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:16:32.555 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:16:32.596 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:32.623 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:32.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:32.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:32.725 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:16:32.752 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:32.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:32.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:16:32.854 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:16:32.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:32.880 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:32.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:16:32.980 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:33.008 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:33.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:33.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:33.065 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:32 smithi090 ceph-mon[30399]: pgmap v431: 238 pgs: 238 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 0 objects/s recovering 2024-04-14T15:16:33.065 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:32 smithi090 ceph-mon[30399]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-04-14T15:16:33.065 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:32 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1251688097' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi090-93363-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:16:33.066 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:32 smithi090 ceph-mon[30399]: osdmap e324: 8 total, 8 up, 8 in 2024-04-14T15:16:33.066 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:32 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1251688097' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi090-93363-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi090-93363-56"}]: dispatch 2024-04-14T15:16:33.066 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:32 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3251375049' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:16:33.066 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:32 smithi090 ceph-mon[41670]: pgmap v431: 238 pgs: 238 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 0 objects/s recovering 2024-04-14T15:16:33.066 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:32 smithi090 ceph-mon[41670]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-04-14T15:16:33.066 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:32 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1251688097' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi090-93363-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:16:33.066 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:32 smithi090 ceph-mon[41670]: osdmap e324: 8 total, 8 up, 8 in 2024-04-14T15:16:33.066 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:32 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1251688097' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi090-93363-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi090-93363-56"}]: dispatch 2024-04-14T15:16:33.066 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:32 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3251375049' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:16:33.110 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:16:33.136 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:33.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:33.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:16:33.238 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:16:33.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:33.264 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:33.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:16:33.365 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:16:33.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:32 smithi092 ceph-mon[33966]: pgmap v431: 238 pgs: 238 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 0 objects/s recovering 2024-04-14T15:16:33.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:32 smithi092 ceph-mon[33966]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-04-14T15:16:33.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:32 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1251688097' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi090-93363-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:16:33.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:32 smithi092 ceph-mon[33966]: osdmap e324: 8 total, 8 up, 8 in 2024-04-14T15:16:33.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:32 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1251688097' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi090-93363-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi090-93363-56"}]: dispatch 2024-04-14T15:16:33.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:32 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3251375049' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:16:33.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:33.391 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:33.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:16:33.492 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:33.517 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:33.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:33.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:33.619 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:16:33.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:33.645 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:33.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:16:33.693 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:16:33.746 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:33.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:33.772 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:33.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:33.873 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:33.898 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:33.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:33.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:33.999 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:16:34.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:34.025 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:34.025 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:16:34.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:33 smithi092 ceph-mon[33966]: osdmap e325: 8 total, 8 up, 8 in 2024-04-14T15:16:34.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:33 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:16:34.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:33 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:16:34.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:33 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1251688097' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi090-93363-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi090-93363-56"}]': finished 2024-04-14T15:16:34.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:33 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:16:34.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:33 smithi092 ceph-mon[33966]: osdmap e326: 8 total, 8 up, 8 in 2024-04-14T15:16:34.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:33 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi090-93652-6","var": "pg_num","format": "json"}]: dispatch 2024-04-14T15:16:34.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:33 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:16:34.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:33 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:16:34.125 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:16:34.141 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:16:33 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:16:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:16:34.142 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:33 smithi090 ceph-mon[41670]: osdmap e325: 8 total, 8 up, 8 in 2024-04-14T15:16:34.142 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:33 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:16:34.142 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:33 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:16:34.142 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:33 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1251688097' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi090-93363-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi090-93363-56"}]': finished 2024-04-14T15:16:34.142 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:33 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:16:34.142 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:33 smithi090 ceph-mon[41670]: osdmap e326: 8 total, 8 up, 8 in 2024-04-14T15:16:34.142 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:33 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi090-93652-6","var": "pg_num","format": "json"}]: dispatch 2024-04-14T15:16:34.142 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:33 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:16:34.142 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:33 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:16:34.146 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:33 smithi090 ceph-mon[30399]: osdmap e325: 8 total, 8 up, 8 in 2024-04-14T15:16:34.146 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:33 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:16:34.146 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:33 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:16:34.147 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:33 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1251688097' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi090-93363-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi090-93363-56"}]': finished 2024-04-14T15:16:34.147 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:33 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:16:34.147 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:33 smithi090 ceph-mon[30399]: osdmap e326: 8 total, 8 up, 8 in 2024-04-14T15:16:34.147 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:33 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi090-93652-6","var": "pg_num","format": "json"}]: dispatch 2024-04-14T15:16:34.147 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:33 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:16:34.147 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:33 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:16:34.149 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:34.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:34.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:16:34.156 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:16:34.250 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:16:34.275 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:34.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:34.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:16:34.376 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:34.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:34.402 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:34.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:34.503 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:16:34.527 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:34.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:34.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:16:34.628 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:16:34.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:34.653 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:34.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:16:34.754 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:16:34.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:34.782 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:34.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:16:34.883 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:16:34.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:34.907 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:34.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:16:35.008 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:16:35.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:35.035 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:35.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:16:35.136 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:16:35.153 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:34 smithi090 ceph-mon[30399]: pgmap v434: 238 pgs: 32 unknown, 206 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:16:35.154 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:34 smithi090 ceph-mon[30399]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:16:35.154 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:34 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-43", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:16:35.154 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:34 smithi090 ceph-mon[30399]: osdmap e327: 8 total, 8 up, 8 in 2024-04-14T15:16:35.154 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:34 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-43","var": "hit_set_count","val": "8"}]: dispatch 2024-04-14T15:16:35.154 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:34 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-43","var": "hit_set_count","val": "8"}]: dispatch 2024-04-14T15:16:35.154 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:34 smithi090 ceph-mon[41670]: pgmap v434: 238 pgs: 32 unknown, 206 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:16:35.154 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:34 smithi090 ceph-mon[41670]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:16:35.154 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:34 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-43", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:16:35.154 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:34 smithi090 ceph-mon[41670]: osdmap e327: 8 total, 8 up, 8 in 2024-04-14T15:16:35.154 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:34 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-43","var": "hit_set_count","val": "8"}]: dispatch 2024-04-14T15:16:35.155 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:34 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-43","var": "hit_set_count","val": "8"}]: dispatch 2024-04-14T15:16:35.161 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:35.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:35.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:16:35.263 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:16:35.288 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:35.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:35.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:16:35.372 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:16:34 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:16:35.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:34 smithi092 ceph-mon[33966]: pgmap v434: 238 pgs: 32 unknown, 206 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:16:35.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:34 smithi092 ceph-mon[33966]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:16:35.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:34 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-43", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:16:35.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:34 smithi092 ceph-mon[33966]: osdmap e327: 8 total, 8 up, 8 in 2024-04-14T15:16:35.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:34 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-43","var": "hit_set_count","val": "8"}]: dispatch 2024-04-14T15:16:35.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:34 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-43","var": "hit_set_count","val": "8"}]: dispatch 2024-04-14T15:16:35.390 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:35.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:35.419 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:35.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:35.520 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:35.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:35.545 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:35.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:35.646 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:16:35.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:35.671 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:35.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:16:35.772 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:16:35.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:35.798 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:35.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:16:35.898 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-14T15:16:35.898 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-14T15:16:35.898 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-14T15:16:35.898 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-14T15:16:35.900 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:35.924 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:35.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:35.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:36.026 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:36.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:36.051 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:36.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:36.152 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:36.175 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-14T15:16:36.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:36.177 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:36.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:36.278 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:16:36.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:36.303 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:36.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:16:36.405 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:16:36.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:36.431 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:36.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:16:36.532 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:36.549 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-14T15:16:36.555 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:36.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:36.556 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:36.657 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:16:36.682 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:36.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:36.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:16:36.784 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:16:36.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:36.808 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:36.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:16:36.909 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:16:36.934 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:36.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:36.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:16:37.036 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:37.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:37.062 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:37.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:37.143 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:36 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:16:37.144 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:36 smithi090 ceph-mon[41670]: pgmap v437: 246 pgs: 8 creating+peering, 238 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:16:37.144 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:36 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-43","var": "hit_set_count","val": "8"}]': finished 2024-04-14T15:16:37.144 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:36 smithi090 ceph-mon[41670]: osdmap e328: 8 total, 8 up, 8 in 2024-04-14T15:16:37.144 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:36 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1251688097' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi090-93363-56"}]: dispatch 2024-04-14T15:16:37.144 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:36 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-43","var": "hit_set_period","val": "600"}]: dispatch 2024-04-14T15:16:37.144 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:36 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-43","var": "hit_set_period","val": "600"}]: dispatch 2024-04-14T15:16:37.144 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:36 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:16:37.144 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:36 smithi090 ceph-mon[30399]: pgmap v437: 246 pgs: 8 creating+peering, 238 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:16:37.144 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:36 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-43","var": "hit_set_count","val": "8"}]': finished 2024-04-14T15:16:37.144 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:36 smithi090 ceph-mon[30399]: osdmap e328: 8 total, 8 up, 8 in 2024-04-14T15:16:37.144 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:36 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1251688097' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi090-93363-56"}]: dispatch 2024-04-14T15:16:37.145 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:36 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-43","var": "hit_set_period","val": "600"}]: dispatch 2024-04-14T15:16:37.145 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:36 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-43","var": "hit_set_period","val": "600"}]: dispatch 2024-04-14T15:16:37.163 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:37.187 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:37.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:37.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:37.289 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:37.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:37.313 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:37.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:37.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:36 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:16:37.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:36 smithi092 ceph-mon[33966]: pgmap v437: 246 pgs: 8 creating+peering, 238 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:16:37.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:36 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-43","var": "hit_set_count","val": "8"}]': finished 2024-04-14T15:16:37.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:36 smithi092 ceph-mon[33966]: osdmap e328: 8 total, 8 up, 8 in 2024-04-14T15:16:37.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:36 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1251688097' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi090-93363-56"}]: dispatch 2024-04-14T15:16:37.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:36 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-43","var": "hit_set_period","val": "600"}]: dispatch 2024-04-14T15:16:37.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:36 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-43","var": "hit_set_period","val": "600"}]: dispatch 2024-04-14T15:16:37.414 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:16:37.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:37.441 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:37.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:16:37.542 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:16:37.566 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:37.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:37.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:16:37.668 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:37.691 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:37.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:37.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:37.793 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:16:37.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:37.818 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:37.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:16:37.920 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:16:37.939 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:37 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1251688097' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi090-93363-56"}]': finished 2024-04-14T15:16:37.939 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:37 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-43","var": "hit_set_period","val": "600"}]': finished 2024-04-14T15:16:37.939 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:37 smithi090 ceph-mon[41670]: osdmap e329: 8 total, 8 up, 8 in 2024-04-14T15:16:37.939 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:37 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1251688097' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi090-93363-56"}]: dispatch 2024-04-14T15:16:37.939 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:37 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-04-14T15:16:37.939 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:37 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-04-14T15:16:37.939 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:37 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1251688097' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi090-93363-56"}]': finished 2024-04-14T15:16:37.939 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:37 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-43","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-04-14T15:16:37.940 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:37 smithi090 ceph-mon[41670]: osdmap e330: 8 total, 8 up, 8 in 2024-04-14T15:16:37.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:37 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1251688097' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi090-93363-56"}]': finished 2024-04-14T15:16:37.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:37 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-43","var": "hit_set_period","val": "600"}]': finished 2024-04-14T15:16:37.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:37 smithi090 ceph-mon[30399]: osdmap e329: 8 total, 8 up, 8 in 2024-04-14T15:16:37.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:37 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1251688097' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi090-93363-56"}]: dispatch 2024-04-14T15:16:37.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:37 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-04-14T15:16:37.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:37 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-04-14T15:16:37.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:37 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1251688097' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi090-93363-56"}]': finished 2024-04-14T15:16:37.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:37 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-43","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-04-14T15:16:37.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:37 smithi090 ceph-mon[30399]: osdmap e330: 8 total, 8 up, 8 in 2024-04-14T15:16:37.947 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:37.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:37.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:16:38.051 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:16:38.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:38.076 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:38.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:16:38.106 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetRead 2024-04-14T15:16:38.106 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: hmm, no HitSet yet 2024-04-14T15:16:38.106 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: ok, hit_set contains 274:602f83fe:::foo:head 2024-04-14T15:16:38.106 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (9027 ms) 2024-04-14T15:16:38.106 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2024-04-14T15:16:38.106 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: pg_num = 32 2024-04-14T15:16:38.106 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: pg 0 ls 1713107798,0 2024-04-14T15:16:38.106 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: pg 1 ls 1713107798,0 2024-04-14T15:16:38.106 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: pg 2 ls 1713107798,0 2024-04-14T15:16:38.107 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: pg 3 ls 1713107798,0 2024-04-14T15:16:38.107 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: pg 4 ls 1713107798,0 2024-04-14T15:16:38.107 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: pg 5 ls 1713107798,0 2024-04-14T15:16:38.107 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: pg 6 ls 1713107798,0 2024-04-14T15:16:38.107 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: pg 7 ls 1713107798,0 2024-04-14T15:16:38.107 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: pg 8 ls 1713107798,0 2024-04-14T15:16:38.107 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: pg 9 ls 1713107798,0 2024-04-14T15:16:38.107 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: pg 10 ls 1713107798,0 2024-04-14T15:16:38.107 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: pg 11 ls 1713107798,0 2024-04-14T15:16:38.107 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: pg 12 ls 1713107798,0 2024-04-14T15:16:38.107 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: pg 13 ls 1713107798,0 2024-04-14T15:16:38.107 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: pg 14 ls 1713107798,0 2024-04-14T15:16:38.107 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: pg 15 ls 1713107798,0 2024-04-14T15:16:38.107 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: pg 16 ls 1713107798,0 2024-04-14T15:16:38.108 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: pg 17 ls 1713107798,0 2024-04-14T15:16:38.108 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: pg 18 ls 1713107798,0 2024-04-14T15:16:38.108 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: pg 19 ls 1713107798,0 2024-04-14T15:16:38.108 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: pg 20 ls 1713107798,0 2024-04-14T15:16:38.108 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: pg 21 ls 1713107798,0 2024-04-14T15:16:38.108 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: pg 22 ls 1713107798,0 2024-04-14T15:16:38.108 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: pg 23 ls 1713107798,0 2024-04-14T15:16:38.108 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: pg 24 ls 1713107798,0 2024-04-14T15:16:38.108 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: pg 25 ls 1713107798,0 2024-04-14T15:16:38.108 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: pg 26 ls 1713107798,0 2024-04-14T15:16:38.108 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: pg 27 ls 1713107798,0 2024-04-14T15:16:38.108 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: pg 28 ls 1713107798,0 2024-04-14T15:16:38.108 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: pg 29 ls 1713107798,0 2024-04-14T15:16:38.108 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: pg 30 ls 1713107798,0 2024-04-14T15:16:38.109 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: pg 31 ls 1713107798,0 2024-04-14T15:16:38.109 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: pg_num = 32 2024-04-14T15:16:38.109 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:6cac518f:::0:head 2024-04-14T15:16:38.109 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:02547ec2:::1:head 2024-04-14T15:16:38.109 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:f905c69b:::2:head 2024-04-14T15:16:38.109 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:cfc208b3:::3:head 2024-04-14T15:16:38.109 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:d83876eb:::4:head 2024-04-14T15:16:38.109 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:b29083e3:::5:head 2024-04-14T15:16:38.109 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:c4fdafeb:::6:head 2024-04-14T15:16:38.109 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:5c6b0b28:::7:head 2024-04-14T15:16:38.109 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:bd63b0f1:::8:head 2024-04-14T15:16:38.109 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:e960b815:::9:head 2024-04-14T15:16:38.109 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:52ea6a34:::10:head 2024-04-14T15:16:38.109 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:89d3ae78:::11:head 2024-04-14T15:16:38.110 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:de5d7c5f:::12:head 2024-04-14T15:16:38.110 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:566253c9:::13:head 2024-04-14T15:16:38.110 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:62a1935d:::14:head 2024-04-14T15:16:38.110 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:863748b0:::15:head 2024-04-14T15:16:38.110 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:3958e169:::16:head 2024-04-14T15:16:38.110 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:4d4dabf9:::17:head 2024-04-14T15:16:38.110 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:8391935d:::18:head 2024-04-14T15:16:38.110 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:28883081:::19:head 2024-04-14T15:16:38.110 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:69259c59:::20:head 2024-04-14T15:16:38.110 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:4bdb80b7:::21:head 2024-04-14T15:16:38.110 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:a11c5d71:::22:head 2024-04-14T15:16:38.110 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:271af37b:::23:head 2024-04-14T15:16:38.110 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:95b121be:::24:head 2024-04-14T15:16:38.111 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:58d1031b:::25:head 2024-04-14T15:16:38.111 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:0a050783:::26:head 2024-04-14T15:16:38.111 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:c709704c:::27:head 2024-04-14T15:16:38.111 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:cbe56eaf:::28:head 2024-04-14T15:16:38.111 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:86b4b162:::29:head 2024-04-14T15:16:38.111 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:70d89383:::30:head 2024-04-14T15:16:38.111 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:dd450c7c:::31:head 2024-04-14T15:16:38.112 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:6d5729b1:::32:head 2024-04-14T15:16:38.112 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:c388f3fb:::33:head 2024-04-14T15:16:38.112 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:56cfea31:::34:head 2024-04-14T15:16:38.112 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:9dbc1bf7:::35:head 2024-04-14T15:16:38.112 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:40b74ccd:::36:head 2024-04-14T15:16:38.112 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:4d5aaf42:::37:head 2024-04-14T15:16:38.112 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:920f362c:::38:head 2024-04-14T15:16:38.112 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:6cc53222:::39:head 2024-04-14T15:16:38.112 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:9cad833f:::40:head 2024-04-14T15:16:38.112 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:1ea84d41:::41:head 2024-04-14T15:16:38.112 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:c4480ef6:::42:head 2024-04-14T15:16:38.112 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:a694361e:::43:head 2024-04-14T15:16:38.112 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:d1bd33e9:::44:head 2024-04-14T15:16:38.113 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:ddc2cd5d:::45:head 2024-04-14T15:16:38.113 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:2b782207:::46:head 2024-04-14T15:16:38.113 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:7b187fca:::47:head 2024-04-14T15:16:38.113 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:90ecdf6f:::48:head 2024-04-14T15:16:38.113 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:a5ed95fe:::49:head 2024-04-14T15:16:38.113 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:ea0eaa55:::50:head 2024-04-14T15:16:38.113 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:f33ef17b:::51:head 2024-04-14T15:16:38.113 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:a0d1b2f6:::52:head 2024-04-14T15:16:38.113 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:60c5229e:::53:head 2024-04-14T15:16:38.113 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:edcbc575:::54:head 2024-04-14T15:16:38.113 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:102cf253:::55:head 2024-04-14T15:16:38.113 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:efb7fb0b:::56:head 2024-04-14T15:16:38.113 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:50d0a326:::57:head 2024-04-14T15:16:38.113 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:d4dc5daf:::58:head 2024-04-14T15:16:38.113 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:3a130462:::59:head 2024-04-14T15:16:38.114 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:ec87ed71:::60:head 2024-04-14T15:16:38.114 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:d5bc9454:::61:head 2024-04-14T15:16:38.114 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:3ddfe313:::62:head 2024-04-14T15:16:38.114 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:7c2816b9:::63:head 2024-04-14T15:16:38.114 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:47e00e4d:::64:head 2024-04-14T15:16:38.114 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:c6410c18:::65:head 2024-04-14T15:16:38.114 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:b48ed237:::66:head 2024-04-14T15:16:38.114 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:cd63ad31:::67:head 2024-04-14T15:16:38.114 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:b179e92b:::68:head 2024-04-14T15:16:38.114 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:0d9f741a:::69:head 2024-04-14T15:16:38.114 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:6d3352ae:::70:head 2024-04-14T15:16:38.114 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:c6d5c19e:::71:head 2024-04-14T15:16:38.114 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:bc4729c3:::72:head 2024-04-14T15:16:38.115 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:77e930b9:::73:head 2024-04-14T15:16:38.115 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:0abeecfd:::74:head 2024-04-14T15:16:38.115 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:b7c37e15:::75:head 2024-04-14T15:16:38.115 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:b6378398:::76:head 2024-04-14T15:16:38.115 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:02bd68de:::77:head 2024-04-14T15:16:38.115 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:cc795d2d:::78:head 2024-04-14T15:16:38.115 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:630d4fea:::79:head 2024-04-14T15:16:38.115 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:e0d29ef5:::80:head 2024-04-14T15:16:38.115 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:fd6f13d2:::81:head 2024-04-14T15:16:38.115 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:606461d5:::82:head 2024-04-14T15:16:38.115 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:eadbdc43:::83:head 2024-04-14T15:16:38.115 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:8761d0bb:::84:head 2024-04-14T15:16:38.115 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:9ef0186f:::85:head 2024-04-14T15:16:38.115 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:e0d41294:::86:head 2024-04-14T15:16:38.115 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:961de695:::87:head 2024-04-14T15:16:38.116 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:1423148f:::88:head 2024-04-14T15:16:38.116 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:633a8fa2:::89:head 2024-04-14T15:16:38.116 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:a8653809:::90:head 2024-04-14T15:16:38.116 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:3dac8b33:::91:head 2024-04-14T15:16:38.116 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:35aad435:::92:head 2024-04-14T15:16:38.116 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:f6dcc343:::93:head 2024-04-14T15:16:38.116 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:dbbdad87:::94:head 2024-04-14T15:16:38.116 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:1cb48ce0:::95:head 2024-04-14T15:16:38.116 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:03cd461c:::96:head 2024-04-14T15:16:38.116 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:17a4ea99:::97:head 2024-04-14T15:16:38.116 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:9993c9a7:::98:head 2024-04-14T15:16:38.116 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:6394211c:::99:head 2024-04-14T15:16:38.116 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:94c7ae57:::100:head 2024-04-14T15:16:38.116 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:6fdee5bb:::101:head 2024-04-14T15:16:38.117 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:9a477fd1:::102:head 2024-04-14T15:16:38.117 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:eb850916:::103:head 2024-04-14T15:16:38.117 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:affc56b9:::104:head 2024-04-14T15:16:38.117 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:b42dc814:::105:head 2024-04-14T15:16:38.117 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:f319f8f0:::106:head 2024-04-14T15:16:38.117 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:9a40b9de:::107:head 2024-04-14T15:16:38.117 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:8b524f28:::108:head 2024-04-14T15:16:38.117 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:e3de589f:::109:head 2024-04-14T15:16:38.117 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:90f90a5b:::110:head 2024-04-14T15:16:38.117 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:a7b4f1d7:::111:head 2024-04-14T15:16:38.117 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:af51766e:::112:head 2024-04-14T15:16:38.117 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:b6f90bd1:::113:head 2024-04-14T15:16:38.117 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:e0261208:::114:head 2024-04-14T15:16:38.117 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:c9569ef7:::115:head 2024-04-14T15:16:38.118 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:61bebe50:::116:head 2024-04-14T15:16:38.118 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:fe93412b:::117:head 2024-04-14T15:16:38.118 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:d3d38bee:::118:head 2024-04-14T15:16:38.118 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:3100ba0c:::119:head 2024-04-14T15:16:38.118 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:d0560ada:::120:head 2024-04-14T15:16:38.118 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:f0ea8b35:::121:head 2024-04-14T15:16:38.118 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:766f231a:::122:head 2024-04-14T15:16:38.118 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:a07a2582:::123:head 2024-04-14T15:16:38.118 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:bd7c6b3a:::124:head 2024-04-14T15:16:38.118 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:fb2ddaff:::125:head 2024-04-14T15:16:38.119 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:4408e1fe:::126:head 2024-04-14T15:16:38.119 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:ee1df7a7:::127:head 2024-04-14T15:16:38.119 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:c3002909:::128:head 2024-04-14T15:16:38.119 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:4f48ffa9:::129:head 2024-04-14T15:16:38.119 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:edf38733:::130:head 2024-04-14T15:16:38.119 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:c08425c0:::131:head 2024-04-14T15:16:38.119 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:5f902d98:::132:head 2024-04-14T15:16:38.119 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:41ea2c93:::133:head 2024-04-14T15:16:38.119 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:813cee13:::134:head 2024-04-14T15:16:38.119 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:0131818d:::135:head 2024-04-14T15:16:38.119 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:26ba5a85:::136:head 2024-04-14T15:16:38.119 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:381b8a5a:::137:head 2024-04-14T15:16:38.119 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:28797e47:::138:head 2024-04-14T15:16:38.119 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:bfca7f22:::139:head 2024-04-14T15:16:38.120 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:36807075:::140:head 2024-04-14T15:16:38.120 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:80b03975:::141:head 2024-04-14T15:16:38.120 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:5c15709b:::142:head 2024-04-14T15:16:38.120 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:f39ea15e:::143:head 2024-04-14T15:16:38.120 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:ea992956:::144:head 2024-04-14T15:16:38.120 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:48887b1c:::145:head 2024-04-14T15:16:38.120 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:9f24a9dd:::146:head 2024-04-14T15:16:38.120 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:987f100b:::147:head 2024-04-14T15:16:38.120 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:d2dd3581:::148:head 2024-04-14T15:16:38.120 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:7fed1808:::149:head 2024-04-14T15:16:38.120 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:c80b70e9:::150:head 2024-04-14T15:16:38.120 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:85ed90f9:::151:head 2024-04-14T15:16:38.120 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:36428b24:::152:head 2024-04-14T15:16:38.120 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:d044c34a:::153:head 2024-04-14T15:16:38.121 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:7c18bf58:::154:head 2024-04-14T15:16:38.121 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:d1c21232:::155:head 2024-04-14T15:16:38.121 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:a7a3c575:::156:head 2024-04-14T15:16:38.121 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:87da0633:::157:head 2024-04-14T15:16:38.121 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:d5ac3822:::158:head 2024-04-14T15:16:38.121 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:3f20522d:::159:head 2024-04-14T15:16:38.121 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:6ca26563:::160:head 2024-04-14T15:16:38.121 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:532ce135:::161:head 2024-04-14T15:16:38.121 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:c78863e6:::162:head 2024-04-14T15:16:38.177 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:16:38.207 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:38.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:38.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:16:38.308 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:38.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:38.333 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:38.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:38.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:37 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1251688097' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi090-93363-56"}]': finished 2024-04-14T15:16:38.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:37 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-43","var": "hit_set_period","val": "600"}]': finished 2024-04-14T15:16:38.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:37 smithi092 ceph-mon[33966]: osdmap e329: 8 total, 8 up, 8 in 2024-04-14T15:16:38.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:37 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1251688097' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi090-93363-56"}]: dispatch 2024-04-14T15:16:38.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:37 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-04-14T15:16:38.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:37 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-04-14T15:16:38.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:37 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1251688097' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi090-93363-56"}]': finished 2024-04-14T15:16:38.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:37 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-43","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-04-14T15:16:38.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:37 smithi092 ceph-mon[33966]: osdmap e330: 8 total, 8 up, 8 in 2024-04-14T15:16:38.434 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:16:38.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:38.459 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:38.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:16:38.561 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:16:38.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:38.585 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:38.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:16:38.686 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:16:38.710 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:38.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:38.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:16:38.812 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:16:38.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:38.836 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:38.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:16:38.937 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:38.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:38.963 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:38.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:39.064 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:16:39.093 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:39.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:39.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:16:39.167 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:38 smithi090 ceph-mon[30399]: pgmap v440: 238 pgs: 238 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:16:39.167 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:38 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2085825169' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi090-93363-57"}]: dispatch 2024-04-14T15:16:39.167 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:38 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2085825169' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi090-93363-57"}]: dispatch 2024-04-14T15:16:39.167 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:38 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2085825169' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi090-93363-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:16:39.167 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:38 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi090-93652-43","var": "pg_num","format": "json"}]: dispatch 2024-04-14T15:16:39.167 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:38 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:16:39.167 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:38 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:16:39.167 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:38 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-43"}]: dispatch 2024-04-14T15:16:39.167 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:38 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-43"}]: dispatch 2024-04-14T15:16:39.167 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:38 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:16:39.168 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:38 smithi090 ceph-mon[41670]: pgmap v440: 238 pgs: 238 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:16:39.168 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:38 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2085825169' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi090-93363-57"}]: dispatch 2024-04-14T15:16:39.168 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:38 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2085825169' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi090-93363-57"}]: dispatch 2024-04-14T15:16:39.168 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:38 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2085825169' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi090-93363-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:16:39.168 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:38 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi090-93652-43","var": "pg_num","format": "json"}]: dispatch 2024-04-14T15:16:39.168 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:38 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:16:39.168 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:38 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:16:39.168 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:38 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-43"}]: dispatch 2024-04-14T15:16:39.168 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:38 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-43"}]: dispatch 2024-04-14T15:16:39.168 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:38 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:16:39.173 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:16:39.174 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:16:39.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.192+0000 7f364cc95700 1 -- 172.21.15.90:0/2784711339 >> v1:172.21.15.92:6789/0 conn(0x7f364810e200 legacy=0x7f36481106c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:39.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.192+0000 7f364cc95700 1 -- 172.21.15.90:0/2784711339 shutdown_connections 2024-04-14T15:16:39.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.192+0000 7f364cc95700 1 -- 172.21.15.90:0/2784711339 >> 172.21.15.90:0/2784711339 conn(0x7f36480fd2b0 msgr2=0x7f36480ff6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:16:39.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.192+0000 7f364cc95700 1 -- 172.21.15.90:0/2784711339 shutdown_connections 2024-04-14T15:16:39.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.192+0000 7f364cc95700 1 -- 172.21.15.90:0/2784711339 wait complete. 2024-04-14T15:16:39.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.193+0000 7f364cc95700 1 librados: starting msgr at 2024-04-14T15:16:39.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.193+0000 7f364cc95700 1 librados: starting objecter 2024-04-14T15:16:39.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.193+0000 7f364cc95700 1 Processor -- start 2024-04-14T15:16:39.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.193+0000 7f364cc95700 1 -- start start 2024-04-14T15:16:39.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.193+0000 7f364cc95700 1 librados: setting wanted keys 2024-04-14T15:16:39.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.193+0000 7f364cc95700 1 librados: calling monclient init 2024-04-14T15:16:39.195 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:16:39.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.194+0000 7f364659c700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f3648107780 0x7f364810dae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:60906/0 (socket says 172.21.15.90:60906) 2024-04-14T15:16:39.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.194+0000 7f364cc95700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f36481a7a30 con 0x7f3648107780 2024-04-14T15:16:39.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.194+0000 7f364659c700 1 -- 172.21.15.90:0/410550695 learned_addr learned my addr 172.21.15.90:0/410550695 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:16:39.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.194+0000 7f36377fe700 10 client.?.objecter ms_handle_connect 0x7f3648107780 2024-04-14T15:16:39.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.194+0000 7f36377fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:16:39.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.194+0000 7f36377fe700 10 client.?.objecter ms_handle_connect 0x7f364810a630 2024-04-14T15:16:39.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.194+0000 7f36377fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:16:39.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.194+0000 7f364cc95700 1 -- 172.21.15.90:0/410550695 --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f36481a7c00 con 0x7f364810a630 2024-04-14T15:16:39.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.194+0000 7f36377fe700 10 client.?.objecter ms_handle_connect 0x7f364810e200 2024-04-14T15:16:39.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.194+0000 7f36377fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:16:39.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.194+0000 7f364cc95700 1 -- 172.21.15.90:0/410550695 --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f36481a7e10 con 0x7f364810e200 2024-04-14T15:16:39.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.194+0000 7f36377fe700 1 -- 172.21.15.90:0/410550695 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 939170077 0 0) 0x7f36481a7a30 con 0x7f3648107780 2024-04-14T15:16:39.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.194+0000 7f36377fe700 1 -- 172.21.15.90:0/410550695 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f36300031f0 con 0x7f3648107780 2024-04-14T15:16:39.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.194+0000 7f36377fe700 1 -- 172.21.15.90:0/410550695 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1491552865 0 0) 0x7f36481a7c00 con 0x7f364810a630 2024-04-14T15:16:39.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.194+0000 7f36377fe700 1 -- 172.21.15.90:0/410550695 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f36481a7a30 con 0x7f364810a630 2024-04-14T15:16:39.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.195+0000 7f36377fe700 1 -- 172.21.15.90:0/410550695 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1184242286 0 0) 0x7f36481a7a30 con 0x7f364810a630 2024-04-14T15:16:39.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.195+0000 7f36377fe700 1 -- 172.21.15.90:0/410550695 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f36481a7c00 con 0x7f364810a630 2024-04-14T15:16:39.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.195+0000 7f36377fe700 1 -- 172.21.15.90:0/410550695 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f363c0018d0 con 0x7f364810a630 2024-04-14T15:16:39.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.195+0000 7f36377fe700 1 -- 172.21.15.90:0/410550695 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3945892926 0 0) 0x7f36300031f0 con 0x7f3648107780 2024-04-14T15:16:39.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.195+0000 7f36377fe700 1 -- 172.21.15.90:0/410550695 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f36481a7a30 con 0x7f3648107780 2024-04-14T15:16:39.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.195+0000 7f36377fe700 1 -- 172.21.15.90:0/410550695 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2559945906 0 0) 0x7f36481a7e10 con 0x7f364810e200 2024-04-14T15:16:39.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.195+0000 7f36377fe700 1 -- 172.21.15.90:0/410550695 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f36300083e0 con 0x7f364810e200 2024-04-14T15:16:39.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.195+0000 7f36377fe700 1 -- 172.21.15.90:0/410550695 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f3638001910 con 0x7f3648107780 2024-04-14T15:16:39.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.195+0000 7f36377fe700 1 -- 172.21.15.90:0/410550695 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2785771775 0 0) 0x7f36481a7c00 con 0x7f364810a630 2024-04-14T15:16:39.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.195+0000 7f36377fe700 1 -- 172.21.15.90:0/410550695 >> v1:172.21.15.90:6790/0 conn(0x7f364810e200 legacy=0x7f36481a60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:39.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.195+0000 7f36377fe700 1 -- 172.21.15.90:0/410550695 >> v1:172.21.15.90:6789/0 conn(0x7f3648107780 legacy=0x7f364810dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:39.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.195+0000 7f36377fe700 1 -- 172.21.15.90:0/410550695 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36481a8020 con 0x7f364810a630 2024-04-14T15:16:39.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.195+0000 7f36377fe700 1 -- 172.21.15.90:0/410550695 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f363c003b40 con 0x7f364810a630 2024-04-14T15:16:39.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.195+0000 7f36377fe700 1 -- 172.21.15.90:0/410550695 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f363c004a60 con 0x7f364810a630 2024-04-14T15:16:39.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.195+0000 7f364cc95700 1 -- 172.21.15.90:0/410550695 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f36481a8310 con 0x7f364810a630 2024-04-14T15:16:39.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.195+0000 7f364cc95700 10 client.48895.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:16:39.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.195+0000 7f364cc95700 1 -- 172.21.15.90:0/410550695 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f36481a87d0 con 0x7f364810a630 2024-04-14T15:16:39.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.195+0000 7f364cc95700 1 librados: init done 2024-04-14T15:16:39.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.197+0000 7f364cc95700 1 -- 172.21.15.90:0/410550695 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3648102ec0 con 0x7f364810a630 2024-04-14T15:16:39.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.197+0000 7f36377fe700 1 -- 172.21.15.90:0/410550695 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f363c0036a0 con 0x7f364810a630 2024-04-14T15:16:39.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.198+0000 7f36377fe700 1 -- 172.21.15.90:0/410550695 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(331..331 src has 51..331) ==== 8466+0+0 (unknown 2800731227 0 0) 0x7f363c0899b0 con 0x7f364810a630 2024-04-14T15:16:39.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.198+0000 7f36377fe700 10 client.48895.objecter ms_dispatch 0x7f3648001550 osd_map(331..331 src has 51..331) 2024-04-14T15:16:39.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.198+0000 7f36377fe700 3 client.48895.objecter handle_osd_map got epochs [331,331] > 0 2024-04-14T15:16:39.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.198+0000 7f36377fe700 3 client.48895.objecter handle_osd_map decoding full epoch 331 2024-04-14T15:16:39.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.198+0000 7f36377fe700 20 client.48895.objecter dump_active .. 0 homeless 2024-04-14T15:16:39.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.198+0000 7f36377fe700 10 client.48895.objecter ms_handle_connect 0x7f3630075e80 2024-04-14T15:16:39.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.202+0000 7f36377fe700 1 -- 172.21.15.90:0/410550695 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f363c056290 con 0x7f364810a630 2024-04-14T15:16:39.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:39.221 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:39.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:16:39.322 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:39.346 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:39.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:39.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:39.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:38 smithi092 ceph-mon[33966]: pgmap v440: 238 pgs: 238 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:16:39.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:38 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2085825169' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi090-93363-57"}]: dispatch 2024-04-14T15:16:39.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:38 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2085825169' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi090-93363-57"}]: dispatch 2024-04-14T15:16:39.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:38 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2085825169' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi090-93363-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:16:39.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:38 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi090-93652-43","var": "pg_num","format": "json"}]: dispatch 2024-04-14T15:16:39.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:38 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:16:39.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:38 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:16:39.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:38 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-43"}]: dispatch 2024-04-14T15:16:39.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:38 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-43"}]: dispatch 2024-04-14T15:16:39.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:38 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:16:39.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.411+0000 7f364cc95700 1 -- 172.21.15.90:0/410550695 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} v 0) -- 0x7f36480693d0 con 0x7f364810a630 2024-04-14T15:16:39.448 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:16:39.471 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:39.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:39.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:16:39.573 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:16:39.597 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:39.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:39.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:16:39.699 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:16:39.723 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:39.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:39.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:16:39.825 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:16:39.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:39.851 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:39.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:16:39.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.893+0000 7f36377fe700 1 -- 172.21.15.90:0/410550695 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]=0 set pool 1 pgp_num to 11 v332) ==== 131+0+0 (unknown 687792566 0 0) 0x7f363c059f40 con 0x7f364810a630 2024-04-14T15:16:39.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.896+0000 7f364cc95700 10 librados: watch_flush enter 2024-04-14T15:16:39.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.896+0000 7f364cc95700 10 librados: watch_flush exit 2024-04-14T15:16:39.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.896+0000 7f364cc95700 20 client.48895.objecter shutdown clearing up homeless session... 2024-04-14T15:16:39.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.896+0000 7f364cc95700 10 client.48895.objecter successfully canceled tick 2024-04-14T15:16:39.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.898+0000 7f364cc95700 1 -- 172.21.15.90:0/410550695 >> v1:172.21.15.90:6800/420935427 conn(0x7f3630075e80 legacy=0x7f3630078340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:39.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.898+0000 7f364cc95700 1 -- 172.21.15.90:0/410550695 >> v1:172.21.15.92:6789/0 conn(0x7f364810a630 legacy=0x7f36481a2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:39.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.901+0000 7f364cc95700 1 -- 172.21.15.90:0/410550695 shutdown_connections 2024-04-14T15:16:39.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.901+0000 7f364cc95700 1 -- 172.21.15.90:0/410550695 >> 172.21.15.90:0/410550695 conn(0x7f36480fd2b0 msgr2=0x7f364806b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:16:39.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.904+0000 7f364cc95700 1 -- 172.21.15.90:0/410550695 shutdown_connections 2024-04-14T15:16:39.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.904+0000 7f364cc95700 1 -- 172.21.15.90:0/410550695 wait complete. 2024-04-14T15:16:39.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.905+0000 7f364cc95700 1 librados: shutdown 2024-04-14T15:16:39.906 INFO:teuthology.orchestra.run.smithi090.stderr:set pool 1 pgp_num to 11 2024-04-14T15:16:39.952 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:16:39.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.951+0000 7fb436aa1700 1 -- 172.21.15.90:0/3553997739 >> v1:172.21.15.90:6789/0 conn(0x7fb430108980 legacy=0x7fb430108d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:39.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.951+0000 7fb436aa1700 1 -- 172.21.15.90:0/3553997739 shutdown_connections 2024-04-14T15:16:39.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.951+0000 7fb436aa1700 1 -- 172.21.15.90:0/3553997739 >> 172.21.15.90:0/3553997739 conn(0x7fb430076aa0 msgr2=0x7fb430076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:16:39.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.952+0000 7fb436aa1700 1 -- 172.21.15.90:0/3553997739 shutdown_connections 2024-04-14T15:16:39.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.952+0000 7fb436aa1700 1 -- 172.21.15.90:0/3553997739 wait complete. 2024-04-14T15:16:39.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.952+0000 7fb436aa1700 1 librados: starting msgr at 2024-04-14T15:16:39.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.952+0000 7fb436aa1700 1 librados: starting objecter 2024-04-14T15:16:39.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.952+0000 7fb436aa1700 1 Processor -- start 2024-04-14T15:16:39.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.952+0000 7fb436aa1700 1 -- start start 2024-04-14T15:16:39.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.952+0000 7fb436aa1700 1 librados: setting wanted keys 2024-04-14T15:16:39.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.952+0000 7fb436aa1700 1 librados: calling monclient init 2024-04-14T15:16:39.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.952+0000 7fb436aa1700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb430108170 con 0x7fb43010b6f0 2024-04-14T15:16:39.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.952+0000 7fb436aa1700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb4301a8f00 con 0x7fb43010f2f0 2024-04-14T15:16:39.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.952+0000 7fb43483d700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7fb430108980 0x7fb430105060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:42064/0 (socket says 172.21.15.90:42064) 2024-04-14T15:16:39.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.952+0000 7fb43483d700 1 -- 172.21.15.90:0/1962773701 learned_addr learned my addr 172.21.15.90:0/1962773701 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:16:39.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.953+0000 7fb42dffb700 10 client.?.objecter ms_handle_connect 0x7fb43010b6f0 2024-04-14T15:16:39.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.953+0000 7fb436aa1700 1 -- 172.21.15.90:0/1962773701 --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb4301a7ef0 con 0x7fb430108980 2024-04-14T15:16:39.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.953+0000 7fb42dffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:16:39.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.953+0000 7fb42dffb700 10 client.?.objecter ms_handle_connect 0x7fb430108980 2024-04-14T15:16:39.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.953+0000 7fb42dffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:16:39.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.953+0000 7fb42dffb700 10 client.?.objecter ms_handle_connect 0x7fb43010f2f0 2024-04-14T15:16:39.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.953+0000 7fb42dffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:16:39.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.953+0000 7fb42dffb700 1 -- 172.21.15.90:0/1962773701 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 476480977 0 0) 0x7fb430108170 con 0x7fb43010b6f0 2024-04-14T15:16:39.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.953+0000 7fb42dffb700 1 -- 172.21.15.90:0/1962773701 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb4200035b0 con 0x7fb43010b6f0 2024-04-14T15:16:39.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.953+0000 7fb42dffb700 1 -- 172.21.15.90:0/1962773701 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1873909562 0 0) 0x7fb4301a8f00 con 0x7fb43010f2f0 2024-04-14T15:16:39.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.953+0000 7fb42dffb700 1 -- 172.21.15.90:0/1962773701 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb430108170 con 0x7fb43010f2f0 2024-04-14T15:16:39.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.953+0000 7fb42dffb700 1 -- 172.21.15.90:0/1962773701 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2228605694 0 0) 0x7fb4301a7ef0 con 0x7fb430108980 2024-04-14T15:16:39.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.953+0000 7fb42dffb700 1 -- 172.21.15.90:0/1962773701 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb4301a8f00 con 0x7fb430108980 2024-04-14T15:16:39.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.953+0000 7fb42dffb700 1 -- 172.21.15.90:0/1962773701 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1927151091 0 0) 0x7fb4200035b0 con 0x7fb43010b6f0 2024-04-14T15:16:39.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.953+0000 7fb42dffb700 1 -- 172.21.15.90:0/1962773701 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb4301a7ef0 con 0x7fb43010b6f0 2024-04-14T15:16:39.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.953+0000 7fb42dffb700 1 -- 172.21.15.90:0/1962773701 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fb41c0018b0 con 0x7fb43010b6f0 2024-04-14T15:16:39.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.953+0000 7fb42dffb700 1 -- 172.21.15.90:0/1962773701 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 837672635 0 0) 0x7fb430108170 con 0x7fb43010f2f0 2024-04-14T15:16:39.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.953+0000 7fb42dffb700 1 -- 172.21.15.90:0/1962773701 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb4200035b0 con 0x7fb43010f2f0 2024-04-14T15:16:39.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.953+0000 7fb42dffb700 1 -- 172.21.15.90:0/1962773701 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fb428001470 con 0x7fb43010f2f0 2024-04-14T15:16:39.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.953+0000 7fb42dffb700 1 -- 172.21.15.90:0/1962773701 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2730203002 0 0) 0x7fb4301a8f00 con 0x7fb430108980 2024-04-14T15:16:39.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.953+0000 7fb42dffb700 1 -- 172.21.15.90:0/1962773701 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb430108170 con 0x7fb430108980 2024-04-14T15:16:39.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.953+0000 7fb42dffb700 1 -- 172.21.15.90:0/1962773701 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fb4240015b0 con 0x7fb430108980 2024-04-14T15:16:39.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.953+0000 7fb42dffb700 1 -- 172.21.15.90:0/1962773701 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1849769337 0 0) 0x7fb4301a7ef0 con 0x7fb43010b6f0 2024-04-14T15:16:39.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.953+0000 7fb42dffb700 1 -- 172.21.15.90:0/1962773701 >> v1:172.21.15.90:6790/0 conn(0x7fb430108980 legacy=0x7fb430105060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:39.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.953+0000 7fb42dffb700 1 -- 172.21.15.90:0/1962773701 >> v1:172.21.15.92:6789/0 conn(0x7fb43010f2f0 legacy=0x7fb4301a67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:39.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.953+0000 7fb42dffb700 1 -- 172.21.15.90:0/1962773701 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4301a80c0 con 0x7fb43010b6f0 2024-04-14T15:16:39.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.953+0000 7fb436aa1700 1 -- 172.21.15.90:0/1962773701 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb4301a83b0 con 0x7fb43010b6f0 2024-04-14T15:16:39.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.954+0000 7fb436aa1700 10 client.37062.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:16:39.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.954+0000 7fb436aa1700 1 -- 172.21.15.90:0/1962773701 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb4301a8870 con 0x7fb43010b6f0 2024-04-14T15:16:39.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.954+0000 7fb436aa1700 1 librados: init done 2024-04-14T15:16:39.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.954+0000 7fb42dffb700 1 -- 172.21.15.90:0/1962773701 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fb41c003b40 con 0x7fb43010b6f0 2024-04-14T15:16:39.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.954+0000 7fb42dffb700 1 -- 172.21.15.90:0/1962773701 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fb41c004ad0 con 0x7fb43010b6f0 2024-04-14T15:16:39.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.955+0000 7fb436aa1700 1 -- 172.21.15.90:0/1962773701 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb414005120 con 0x7fb43010b6f0 2024-04-14T15:16:39.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.956+0000 7fb42dffb700 1 -- 172.21.15.90:0/1962773701 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fb41c0036a0 con 0x7fb43010b6f0 2024-04-14T15:16:39.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.956+0000 7fb42dffb700 1 -- 172.21.15.90:0/1962773701 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(332..332 src has 51..332) ==== 8183+0+0 (unknown 494222253 0 0) 0x7fb41c089630 con 0x7fb43010b6f0 2024-04-14T15:16:39.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.956+0000 7fb42dffb700 10 client.37062.objecter ms_dispatch 0x7fb430001550 osd_map(332..332 src has 51..332) 2024-04-14T15:16:39.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.956+0000 7fb42dffb700 3 client.37062.objecter handle_osd_map got epochs [332,332] > 0 2024-04-14T15:16:39.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.956+0000 7fb42dffb700 3 client.37062.objecter handle_osd_map decoding full epoch 332 2024-04-14T15:16:39.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.956+0000 7fb42dffb700 20 client.37062.objecter dump_active .. 0 homeless 2024-04-14T15:16:39.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.957+0000 7fb42dffb700 10 client.37062.objecter ms_handle_connect 0x7fb4200713f0 2024-04-14T15:16:39.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:39.961+0000 7fb42dffb700 1 -- 172.21.15.90:0/1962773701 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fb41c056030 con 0x7fb43010b6f0 2024-04-14T15:16:39.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:39.977 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:39.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:16:40.078 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:16:40.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:40.104 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:40.104 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:16:40.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:40.172+0000 7fb436aa1700 1 -- 172.21.15.90:0/1962773701 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7fb414005410 con 0x7fb43010b6f0 2024-04-14T15:16:40.173 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:39 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2085825169' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi090-93363-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:16:40.173 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:39 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-43"}]': finished 2024-04-14T15:16:40.174 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:39 smithi090 ceph-mon[30399]: osdmap e331: 8 total, 8 up, 8 in 2024-04-14T15:16:40.174 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:39 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2085825169' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi090-93363-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi090-93363-57"}]: dispatch 2024-04-14T15:16:40.174 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:39 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/410550695' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-04-14T15:16:40.174 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:39 smithi090 ceph-mon[30399]: from='client.48895 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-04-14T15:16:40.174 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:39 smithi090 ceph-mon[30399]: from='client.48895 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-04-14T15:16:40.174 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:39 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2085825169' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi090-93363-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:16:40.174 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:39 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-43"}]': finished 2024-04-14T15:16:40.174 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:39 smithi090 ceph-mon[41670]: osdmap e331: 8 total, 8 up, 8 in 2024-04-14T15:16:40.174 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:39 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2085825169' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi090-93363-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi090-93363-57"}]: dispatch 2024-04-14T15:16:40.174 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:39 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/410550695' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-04-14T15:16:40.175 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:39 smithi090 ceph-mon[41670]: from='client.48895 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-04-14T15:16:40.175 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:39 smithi090 ceph-mon[41670]: from='client.48895 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-04-14T15:16:40.205 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:16:40.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:40.229 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:40.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:16:40.330 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:16:40.354 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:40.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:40.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:16:40.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:39 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2085825169' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi090-93363-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:16:40.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:39 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-43"}]': finished 2024-04-14T15:16:40.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:39 smithi092 ceph-mon[33966]: osdmap e331: 8 total, 8 up, 8 in 2024-04-14T15:16:40.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:39 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2085825169' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi090-93363-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi090-93363-57"}]: dispatch 2024-04-14T15:16:40.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:39 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/410550695' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-04-14T15:16:40.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:39 smithi092 ceph-mon[33966]: from='client.48895 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-04-14T15:16:40.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:39 smithi092 ceph-mon[33966]: from='client.48895 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-04-14T15:16:40.456 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:16:40.480 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:40.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:40.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:16:40.582 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:16:40.609 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:40.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:40.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:16:40.711 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:40.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:40.736 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:40.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:40.837 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:16:40.862 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:40.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:40.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:16:40.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:40.891+0000 7fb42dffb700 1 -- 172.21.15.90:0/1962773701 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v333) ==== 91+0+0 (unknown 3613035174 0 0) 0x7fb41c059ce0 con 0x7fb43010b6f0 2024-04-14T15:16:40.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:40.894+0000 7fb436aa1700 10 librados: watch_flush enter 2024-04-14T15:16:40.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:40.895+0000 7fb436aa1700 10 librados: watch_flush exit 2024-04-14T15:16:40.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:40.895+0000 7fb436aa1700 20 client.37062.objecter shutdown clearing up homeless session... 2024-04-14T15:16:40.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:40.895+0000 7fb436aa1700 10 client.37062.objecter successfully canceled tick 2024-04-14T15:16:40.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:40.895+0000 7fb436aa1700 1 -- 172.21.15.90:0/1962773701 >> v1:172.21.15.90:6800/420935427 conn(0x7fb4200713f0 legacy=0x7fb4200738b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:40.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:40.895+0000 7fb436aa1700 1 -- 172.21.15.90:0/1962773701 >> v1:172.21.15.90:6789/0 conn(0x7fb43010b6f0 legacy=0x7fb430105770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:40.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:40.895+0000 7fb436aa1700 1 -- 172.21.15.90:0/1962773701 shutdown_connections 2024-04-14T15:16:40.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:40.895+0000 7fb436aa1700 1 -- 172.21.15.90:0/1962773701 >> 172.21.15.90:0/1962773701 conn(0x7fb430076aa0 msgr2=0x7fb430111770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:16:40.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:40.899+0000 7fb436aa1700 1 -- 172.21.15.90:0/1962773701 shutdown_connections 2024-04-14T15:16:40.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:40.903+0000 7fb436aa1700 1 -- 172.21.15.90:0/1962773701 wait complete. 2024-04-14T15:16:40.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:40.903+0000 7fb436aa1700 1 librados: shutdown 2024-04-14T15:16:40.905 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is unset 2024-04-14T15:16:40.963 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:16:40.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:40.988 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:40.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:16:41.089 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:16:41.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:41.113 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:41.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:16:41.149 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:40 smithi090 ceph-mon[41670]: pgmap v443: 238 pgs: 238 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 42 KiB/s wr, 67 op/s 2024-04-14T15:16:41.150 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:40 smithi090 ceph-mon[41670]: osdmap e332: 8 total, 8 up, 8 in 2024-04-14T15:16:41.150 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:40 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1962773701' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:16:41.150 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:40 smithi090 ceph-mon[41670]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:16:41.150 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:40 smithi090 ceph-mon[41670]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:16:41.150 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:40 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2085825169' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi090-93363-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi090-93363-57"}]': finished 2024-04-14T15:16:41.150 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:40 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1962773701' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-14T15:16:41.150 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:40 smithi090 ceph-mon[41670]: osdmap e333: 8 total, 8 up, 8 in 2024-04-14T15:16:41.150 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:40 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:16:41.150 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:40 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:16:41.151 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:40 smithi090 ceph-mon[30399]: pgmap v443: 238 pgs: 238 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 42 KiB/s wr, 67 op/s 2024-04-14T15:16:41.151 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:40 smithi090 ceph-mon[30399]: osdmap e332: 8 total, 8 up, 8 in 2024-04-14T15:16:41.151 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:40 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1962773701' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:16:41.151 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:40 smithi090 ceph-mon[30399]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:16:41.151 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:40 smithi090 ceph-mon[30399]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:16:41.151 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:40 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2085825169' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi090-93363-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi090-93363-57"}]': finished 2024-04-14T15:16:41.151 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:40 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1962773701' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-14T15:16:41.151 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:40 smithi090 ceph-mon[30399]: osdmap e333: 8 total, 8 up, 8 in 2024-04-14T15:16:41.151 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:40 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:16:41.151 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:40 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:16:41.216 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:41.239 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:41.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:41.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:41.341 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:41.364 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:41.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:41.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:41.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:40 smithi092 ceph-mon[33966]: pgmap v443: 238 pgs: 238 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 42 KiB/s wr, 67 op/s 2024-04-14T15:16:41.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:40 smithi092 ceph-mon[33966]: osdmap e332: 8 total, 8 up, 8 in 2024-04-14T15:16:41.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:40 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1962773701' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:16:41.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:40 smithi092 ceph-mon[33966]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:16:41.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:40 smithi092 ceph-mon[33966]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:16:41.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:40 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2085825169' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi090-93363-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi090-93363-57"}]': finished 2024-04-14T15:16:41.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:40 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1962773701' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-14T15:16:41.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:40 smithi092 ceph-mon[33966]: osdmap e333: 8 total, 8 up, 8 in 2024-04-14T15:16:41.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:40 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:16:41.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:40 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:16:41.466 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:16:41.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:41.492 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:41.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:16:41.593 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:16:41.617 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:41.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:41.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:16:41.722 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:41.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:41.746 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:41.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:41.847 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:16:41.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:41.873 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:41.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:16:41.974 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:16:41.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:41.999 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:42.000 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:16:42.100 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:16:42.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:42.125 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:42.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:16:42.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:41 smithi090 ceph-mon[41670]: 1.5 scrub starts 2024-04-14T15:16:42.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:41 smithi090 ceph-mon[41670]: 1.5 scrub ok 2024-04-14T15:16:42.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:41 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:16:42.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:41 smithi090 ceph-mon[41670]: osdmap e334: 8 total, 8 up, 8 in 2024-04-14T15:16:42.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:41 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:16:42.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:41 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:16:42.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:41 smithi090 ceph-mon[30399]: 1.5 scrub starts 2024-04-14T15:16:42.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:41 smithi090 ceph-mon[30399]: 1.5 scrub ok 2024-04-14T15:16:42.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:41 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:16:42.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:41 smithi090 ceph-mon[30399]: osdmap e334: 8 total, 8 up, 8 in 2024-04-14T15:16:42.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:41 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:16:42.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:41 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:16:42.226 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:16:42.250 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:42.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:42.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:16:42.352 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:42.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:41 smithi092 ceph-mon[33966]: 1.5 scrub starts 2024-04-14T15:16:42.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:41 smithi092 ceph-mon[33966]: 1.5 scrub ok 2024-04-14T15:16:42.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:41 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:16:42.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:41 smithi092 ceph-mon[33966]: osdmap e334: 8 total, 8 up, 8 in 2024-04-14T15:16:42.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:41 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:16:42.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:41 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:16:42.377 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:42.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:42.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:42.479 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:16:42.503 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:42.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:42.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:16:42.605 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:16:42.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:42.629 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:42.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:16:42.730 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:16:42.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:42.756 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:42.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:16:42.857 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:16:42.881 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:42.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:42.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:16:42.983 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:16:43.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:43.008 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:43.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:16:43.109 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:16:43.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:43.134 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:43.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:16:43.136 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:42 smithi090 ceph-mon[41670]: pgmap v446: 246 pgs: 40 unknown, 206 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:16:43.136 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:42 smithi090 ceph-mon[41670]: 2.10 scrub starts 2024-04-14T15:16:43.136 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:42 smithi090 ceph-mon[41670]: 2.10 scrub ok 2024-04-14T15:16:43.137 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:42 smithi090 ceph-mon[41670]: 1.4 scrub starts 2024-04-14T15:16:43.137 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:42 smithi090 ceph-mon[41670]: 1.4 scrub ok 2024-04-14T15:16:43.137 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:42 smithi090 ceph-mon[41670]: 2.18 scrub starts 2024-04-14T15:16:43.137 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:42 smithi090 ceph-mon[41670]: 2.18 scrub ok 2024-04-14T15:16:43.137 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:42 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-45", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:16:43.137 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:42 smithi090 ceph-mon[41670]: osdmap e335: 8 total, 8 up, 8 in 2024-04-14T15:16:43.137 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:42 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-45","var": "hit_set_count","val": "3"}]: dispatch 2024-04-14T15:16:43.137 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:42 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-45","var": "hit_set_count","val": "3"}]: dispatch 2024-04-14T15:16:43.137 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:42 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2085825169' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi090-93363-57"}]: dispatch 2024-04-14T15:16:43.137 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:42 smithi090 ceph-mon[30399]: pgmap v446: 246 pgs: 40 unknown, 206 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:16:43.138 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:42 smithi090 ceph-mon[30399]: 2.10 scrub starts 2024-04-14T15:16:43.138 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:42 smithi090 ceph-mon[30399]: 2.10 scrub ok 2024-04-14T15:16:43.138 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:42 smithi090 ceph-mon[30399]: 1.4 scrub starts 2024-04-14T15:16:43.138 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:42 smithi090 ceph-mon[30399]: 1.4 scrub ok 2024-04-14T15:16:43.138 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:42 smithi090 ceph-mon[30399]: 2.18 scrub starts 2024-04-14T15:16:43.138 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:42 smithi090 ceph-mon[30399]: 2.18 scrub ok 2024-04-14T15:16:43.138 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:42 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-45", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:16:43.138 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:42 smithi090 ceph-mon[30399]: osdmap e335: 8 total, 8 up, 8 in 2024-04-14T15:16:43.138 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:42 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-45","var": "hit_set_count","val": "3"}]: dispatch 2024-04-14T15:16:43.138 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:42 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-45","var": "hit_set_count","val": "3"}]: dispatch 2024-04-14T15:16:43.138 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:42 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2085825169' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi090-93363-57"}]: dispatch 2024-04-14T15:16:43.235 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:16:43.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:43.258 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:43.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:16:43.359 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:43.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:42 smithi092 ceph-mon[33966]: pgmap v446: 246 pgs: 40 unknown, 206 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:16:43.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:42 smithi092 ceph-mon[33966]: 2.10 scrub starts 2024-04-14T15:16:43.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:42 smithi092 ceph-mon[33966]: 2.10 scrub ok 2024-04-14T15:16:43.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:42 smithi092 ceph-mon[33966]: 1.4 scrub starts 2024-04-14T15:16:43.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:42 smithi092 ceph-mon[33966]: 1.4 scrub ok 2024-04-14T15:16:43.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:42 smithi092 ceph-mon[33966]: 2.18 scrub starts 2024-04-14T15:16:43.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:42 smithi092 ceph-mon[33966]: 2.18 scrub ok 2024-04-14T15:16:43.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:42 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-45", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:16:43.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:42 smithi092 ceph-mon[33966]: osdmap e335: 8 total, 8 up, 8 in 2024-04-14T15:16:43.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:42 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-45","var": "hit_set_count","val": "3"}]: dispatch 2024-04-14T15:16:43.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:42 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-45","var": "hit_set_count","val": "3"}]: dispatch 2024-04-14T15:16:43.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:42 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2085825169' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi090-93363-57"}]: dispatch 2024-04-14T15:16:43.385 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:43.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:43.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:43.487 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:16:43.511 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:43.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:43.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:16:43.612 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:16:43.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:43.640 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:43.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:16:43.741 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:43.767 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:43.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:43.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:43.869 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:16:43.872 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-14T15:16:43.895 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:43.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:43.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:16:43.976 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-14T15:16:43.976 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-14T15:16:43.976 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-14T15:16:43.976 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-14T15:16:43.976 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-14T15:16:43.997 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:44.021 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:44.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:44.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:44.123 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:16:44.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:44.148 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:44.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:16:44.194 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:16:43 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:16:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:16:44.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:43 smithi090 ceph-mon[30399]: 4.4 scrub starts 2024-04-14T15:16:44.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:43 smithi090 ceph-mon[30399]: 4.4 scrub ok 2024-04-14T15:16:44.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:43 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-45","var": "hit_set_count","val": "3"}]': finished 2024-04-14T15:16:44.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:43 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2085825169' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi090-93363-57"}]': finished 2024-04-14T15:16:44.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:43 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-45","var": "hit_set_period","val": "3"}]: dispatch 2024-04-14T15:16:44.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:43 smithi090 ceph-mon[30399]: osdmap e336: 8 total, 8 up, 8 in 2024-04-14T15:16:44.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:43 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2085825169' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi090-93363-57"}]: dispatch 2024-04-14T15:16:44.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:43 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-45","var": "hit_set_period","val": "3"}]: dispatch 2024-04-14T15:16:44.195 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:43 smithi090 ceph-mon[41670]: 4.4 scrub starts 2024-04-14T15:16:44.195 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:43 smithi090 ceph-mon[41670]: 4.4 scrub ok 2024-04-14T15:16:44.195 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:43 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-45","var": "hit_set_count","val": "3"}]': finished 2024-04-14T15:16:44.195 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:43 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2085825169' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi090-93363-57"}]': finished 2024-04-14T15:16:44.195 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:43 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-45","var": "hit_set_period","val": "3"}]: dispatch 2024-04-14T15:16:44.195 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:43 smithi090 ceph-mon[41670]: osdmap e336: 8 total, 8 up, 8 in 2024-04-14T15:16:44.195 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:43 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2085825169' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi090-93363-57"}]: dispatch 2024-04-14T15:16:44.195 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:43 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-45","var": "hit_set_period","val": "3"}]: dispatch 2024-04-14T15:16:44.249 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:16:44.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:44.274 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:44.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:16:44.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:43 smithi092 ceph-mon[33966]: 4.4 scrub starts 2024-04-14T15:16:44.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:43 smithi092 ceph-mon[33966]: 4.4 scrub ok 2024-04-14T15:16:44.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:43 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-45","var": "hit_set_count","val": "3"}]': finished 2024-04-14T15:16:44.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:43 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2085825169' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi090-93363-57"}]': finished 2024-04-14T15:16:44.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:43 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-45","var": "hit_set_period","val": "3"}]: dispatch 2024-04-14T15:16:44.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:43 smithi092 ceph-mon[33966]: osdmap e336: 8 total, 8 up, 8 in 2024-04-14T15:16:44.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:43 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2085825169' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi090-93363-57"}]: dispatch 2024-04-14T15:16:44.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:43 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-45","var": "hit_set_period","val": "3"}]: dispatch 2024-04-14T15:16:44.375 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:16:44.399 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:44.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:44.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:16:44.501 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:16:44.528 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:44.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:44.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:16:44.626 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-14T15:16:44.630 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:44.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:44.659 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:44.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:44.760 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:16:44.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:44.788 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:44.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:16:44.889 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:16:44.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:44.916 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:44.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:16:45.017 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:45.044 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:45.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:45.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:45.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.083+0000 7f386d1d4700 1 -- 172.21.15.90:0/1773733965 >> v1:172.21.15.90:6789/0 conn(0x7f3868107890 legacy=0x7f3868107c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:45.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.083+0000 7f386d1d4700 1 -- 172.21.15.90:0/1773733965 shutdown_connections 2024-04-14T15:16:45.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.083+0000 7f386d1d4700 1 -- 172.21.15.90:0/1773733965 >> 172.21.15.90:0/1773733965 conn(0x7f38680fd290 msgr2=0x7f38680ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:16:45.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.083+0000 7f386d1d4700 1 -- 172.21.15.90:0/1773733965 shutdown_connections 2024-04-14T15:16:45.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.083+0000 7f386d1d4700 1 -- 172.21.15.90:0/1773733965 wait complete. 2024-04-14T15:16:45.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.084+0000 7f386d1d4700 1 librados: starting msgr at 2024-04-14T15:16:45.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.084+0000 7f386d1d4700 1 librados: starting objecter 2024-04-14T15:16:45.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.084+0000 7f386d1d4700 1 Processor -- start 2024-04-14T15:16:45.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.084+0000 7f386d1d4700 1 -- start start 2024-04-14T15:16:45.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.084+0000 7f386d1d4700 1 librados: setting wanted keys 2024-04-14T15:16:45.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.084+0000 7f386d1d4700 1 librados: calling monclient init 2024-04-14T15:16:45.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.085+0000 7f386d1d4700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f38681a7f20 con 0x7f386810a740 2024-04-14T15:16:45.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.085+0000 7f386d1d4700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f38681a9100 con 0x7f386810e310 2024-04-14T15:16:45.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.085+0000 7f386d1d4700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f38681aa2e0 con 0x7f3868107890 2024-04-14T15:16:45.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.085+0000 7f3866d9d700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f3868107890 0x7f386810c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:55440/0 (socket says 172.21.15.90:55440) 2024-04-14T15:16:45.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.085+0000 7f3866d9d700 1 -- 172.21.15.90:0/1194098772 learned_addr learned my addr 172.21.15.90:0/1194098772 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:16:45.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.085+0000 7f386659c700 1 --1- 172.21.15.90:0/1194098772 >> v1:172.21.15.90:6789/0 conn(0x7f386810a740 0x7f386810c710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:47884/0 (socket says 172.21.15.90:47884) 2024-04-14T15:16:45.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.085+0000 7f384ffff700 10 client.?.objecter ms_handle_connect 0x7f386810a740 2024-04-14T15:16:45.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.085+0000 7f384ffff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:16:45.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.085+0000 7f384ffff700 10 client.?.objecter ms_handle_connect 0x7f386810e310 2024-04-14T15:16:45.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.085+0000 7f384ffff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:16:45.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.085+0000 7f384ffff700 10 client.?.objecter ms_handle_connect 0x7f3868107890 2024-04-14T15:16:45.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.085+0000 7f384ffff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:16:45.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.085+0000 7f384ffff700 1 -- 172.21.15.90:0/1194098772 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3722205270 0 0) 0x7f38681a7f20 con 0x7f386810a740 2024-04-14T15:16:45.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.085+0000 7f384ffff700 1 -- 172.21.15.90:0/1194098772 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f38500035b0 con 0x7f386810a740 2024-04-14T15:16:45.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.085+0000 7f384ffff700 1 -- 172.21.15.90:0/1194098772 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1219174913 0 0) 0x7f38681a9100 con 0x7f386810e310 2024-04-14T15:16:45.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.085+0000 7f384ffff700 1 -- 172.21.15.90:0/1194098772 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f38681a7f20 con 0x7f386810e310 2024-04-14T15:16:45.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.085+0000 7f384ffff700 1 -- 172.21.15.90:0/1194098772 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 755783921 0 0) 0x7f38681aa2e0 con 0x7f3868107890 2024-04-14T15:16:45.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.085+0000 7f384ffff700 1 -- 172.21.15.90:0/1194098772 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f38681a9100 con 0x7f3868107890 2024-04-14T15:16:45.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.085+0000 7f384ffff700 1 -- 172.21.15.90:0/1194098772 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3085421721 0 0) 0x7f38681a7f20 con 0x7f386810e310 2024-04-14T15:16:45.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.085+0000 7f384ffff700 1 -- 172.21.15.90:0/1194098772 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f38681aa2e0 con 0x7f386810e310 2024-04-14T15:16:45.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.085+0000 7f384ffff700 1 -- 172.21.15.90:0/1194098772 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f3860001c50 con 0x7f386810e310 2024-04-14T15:16:45.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.085+0000 7f384ffff700 1 -- 172.21.15.90:0/1194098772 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 97322623 0 0) 0x7f38500035b0 con 0x7f386810a740 2024-04-14T15:16:45.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.086+0000 7f384ffff700 1 -- 172.21.15.90:0/1194098772 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f38681a7f20 con 0x7f386810a740 2024-04-14T15:16:45.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.086+0000 7f384ffff700 1 -- 172.21.15.90:0/1194098772 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f385c0018b0 con 0x7f386810a740 2024-04-14T15:16:45.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.086+0000 7f384ffff700 1 -- 172.21.15.90:0/1194098772 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 828529172 0 0) 0x7f38681a7f20 con 0x7f386810a740 2024-04-14T15:16:45.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.086+0000 7f384ffff700 1 -- 172.21.15.90:0/1194098772 >> v1:172.21.15.90:6790/0 conn(0x7f3868107890 legacy=0x7f386810c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:45.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.086+0000 7f384ffff700 1 -- 172.21.15.90:0/1194098772 >> v1:172.21.15.92:6789/0 conn(0x7f386810e310 legacy=0x7f38681a6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:45.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.086+0000 7f384ffff700 1 -- 172.21.15.90:0/1194098772 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38681ab4c0 con 0x7f386810a740 2024-04-14T15:16:45.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.086+0000 7f386d1d4700 1 -- 172.21.15.90:0/1194098772 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f38681a9330 con 0x7f386810a740 2024-04-14T15:16:45.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.086+0000 7f384ffff700 1 -- 172.21.15.90:0/1194098772 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f385c003b40 con 0x7f386810a740 2024-04-14T15:16:45.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.086+0000 7f384ffff700 1 -- 172.21.15.90:0/1194098772 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f385c005c60 con 0x7f386810a740 2024-04-14T15:16:45.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.086+0000 7f386d1d4700 10 client.37074.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:16:45.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.086+0000 7f386d1d4700 1 -- 172.21.15.90:0/1194098772 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f38681a97f0 con 0x7f386810a740 2024-04-14T15:16:45.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.087+0000 7f386d1d4700 1 librados: init done 2024-04-14T15:16:45.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.087+0000 7f386d1d4700 1 -- 172.21.15.90:0/1194098772 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f38680693d0 con 0x7f386810a740 2024-04-14T15:16:45.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.088+0000 7f384ffff700 1 -- 172.21.15.90:0/1194098772 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f385c0036a0 con 0x7f386810a740 2024-04-14T15:16:45.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.089+0000 7f384ffff700 1 -- 172.21.15.90:0/1194098772 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(337..337 src has 51..337) ==== 8152+0+0 (unknown 1211307971 0 0) 0x7f385c08a8f0 con 0x7f386810a740 2024-04-14T15:16:45.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.089+0000 7f384ffff700 10 client.37074.objecter ms_dispatch 0x7f3868001550 osd_map(337..337 src has 51..337) 2024-04-14T15:16:45.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.089+0000 7f384ffff700 3 client.37074.objecter handle_osd_map got epochs [337,337] > 0 2024-04-14T15:16:45.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.089+0000 7f384ffff700 3 client.37074.objecter handle_osd_map decoding full epoch 337 2024-04-14T15:16:45.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.089+0000 7f384ffff700 20 client.37074.objecter dump_active .. 0 homeless 2024-04-14T15:16:45.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.089+0000 7f384ffff700 1 -- 172.21.15.90:0/1194098772 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v337) ==== 54+0+16 (unknown 2759012399 0 2420000192) 0x7f385c08ada0 con 0x7f386810a740 2024-04-14T15:16:45.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.089+0000 7f384ffff700 10 client.37074.objecter ms_handle_connect 0x7f3850071090 2024-04-14T15:16:45.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.089+0000 7f384dffb700 10 client.37074.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:16:45.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.089+0000 7f384dffb700 15 client.37074.objecter _session_command_op_assign -1 1 2024-04-14T15:16:45.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.089+0000 7f384dffb700 20 client.37074.objecter _get_session s=0x7f38681122e0 osd=0 3 2024-04-14T15:16:45.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.089+0000 7f384dffb700 20 client.37074.objecter put_session s=0x7f38681122e0 osd=0 3 2024-04-14T15:16:45.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.089+0000 7f384dffb700 20 client.37074.objecter _get_session s=0x7f38681122e0 osd=0 3 2024-04-14T15:16:45.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.089+0000 7f384dffb700 15 client.37074.objecter _session_command_op_remove -1 1 2024-04-14T15:16:45.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.089+0000 7f384dffb700 20 client.37074.objecter get_session s=0x7f38681122e0 osd=0 3 2024-04-14T15:16:45.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.089+0000 7f384dffb700 15 client.37074.objecter _session_command_op_assign 0 1 2024-04-14T15:16:45.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.089+0000 7f384dffb700 20 client.37074.objecter put_session s=0x7f38681122e0 osd=0 4 2024-04-14T15:16:45.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.089+0000 7f384dffb700 10 client.37074.objecter _send_command 1 2024-04-14T15:16:45.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.089+0000 7f384dffb700 1 -- 172.21.15.90:0/1194098772 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f386810d530 con 0x7f38681aa9c0 2024-04-14T15:16:45.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.090+0000 7f384ffff700 10 client.37074.objecter ms_handle_connect 0x7f38681aa9c0 2024-04-14T15:16:45.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.090+0000 7f384ffff700 1 -- 172.21.15.90:0/1194098772 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f386810d530 con 0x7f38681aa9c0 2024-04-14T15:16:45.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.090+0000 7f384ffff700 10 client.37074.objecter ms_dispatch 0x7f3868001550 command_reply(tid 1: 0 ) 2024-04-14T15:16:45.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.090+0000 7f384ffff700 10 client.37074.objecter _finish_command 1 = system:0 2024-04-14T15:16:45.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.090+0000 7f384ffff700 20 client.37074.objecter put_session s=0x7f38681122e0 osd=0 4 2024-04-14T15:16:45.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.090+0000 7f384ffff700 15 client.37074.objecter _session_command_op_remove 0 1 2024-04-14T15:16:45.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.109+0000 7f384dffb700 10 client.37074.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:16:45.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.109+0000 7f384dffb700 15 client.37074.objecter _session_command_op_assign -1 2 2024-04-14T15:16:45.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.109+0000 7f384dffb700 20 client.37074.objecter _get_session s=0x7f38681122e0 osd=0 3 2024-04-14T15:16:45.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.109+0000 7f384dffb700 20 client.37074.objecter put_session s=0x7f38681122e0 osd=0 3 2024-04-14T15:16:45.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.109+0000 7f384dffb700 20 client.37074.objecter _get_session s=0x7f38681122e0 osd=0 3 2024-04-14T15:16:45.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.109+0000 7f384dffb700 15 client.37074.objecter _session_command_op_remove -1 2 2024-04-14T15:16:45.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.109+0000 7f384dffb700 20 client.37074.objecter get_session s=0x7f38681122e0 osd=0 3 2024-04-14T15:16:45.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.109+0000 7f384dffb700 15 client.37074.objecter _session_command_op_assign 0 2 2024-04-14T15:16:45.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.109+0000 7f384dffb700 20 client.37074.objecter put_session s=0x7f38681122e0 osd=0 4 2024-04-14T15:16:45.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.109+0000 7f384dffb700 10 client.37074.objecter _send_command 2 2024-04-14T15:16:45.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.109+0000 7f384dffb700 1 -- 172.21.15.90:0/1194098772 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f3868108710 con 0x7f38681aa9c0 2024-04-14T15:16:45.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.111+0000 7f384ffff700 1 -- 172.21.15.90:0/1194098772 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f3868108710 con 0x7f38681aa9c0 2024-04-14T15:16:45.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.111+0000 7f384ffff700 10 client.37074.objecter ms_dispatch 0x7f3868001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:16:45.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.111+0000 7f384ffff700 10 client.37074.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:16:45.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.111+0000 7f384ffff700 20 client.37074.objecter put_session s=0x7f38681122e0 osd=0 4 2024-04-14T15:16:45.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.111+0000 7f384ffff700 15 client.37074.objecter _session_command_op_remove 0 2 2024-04-14T15:16:45.112 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:16:45.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.112+0000 7f384dffb700 10 client.37074.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:16:45.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.112+0000 7f384dffb700 15 client.37074.objecter _session_command_op_assign -1 3 2024-04-14T15:16:45.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.112+0000 7f384dffb700 20 client.37074.objecter _get_session s=0x7f38681a9d30 osd=1 3 2024-04-14T15:16:45.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.112+0000 7f384dffb700 20 client.37074.objecter put_session s=0x7f38681a9d30 osd=1 3 2024-04-14T15:16:45.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.112+0000 7f384dffb700 20 client.37074.objecter _get_session s=0x7f38681a9d30 osd=1 3 2024-04-14T15:16:45.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.112+0000 7f384dffb700 15 client.37074.objecter _session_command_op_remove -1 3 2024-04-14T15:16:45.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.112+0000 7f384dffb700 20 client.37074.objecter get_session s=0x7f38681a9d30 osd=1 3 2024-04-14T15:16:45.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.112+0000 7f384dffb700 15 client.37074.objecter _session_command_op_assign 1 3 2024-04-14T15:16:45.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.112+0000 7f384dffb700 20 client.37074.objecter put_session s=0x7f38681a9d30 osd=1 4 2024-04-14T15:16:45.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.112+0000 7f384dffb700 10 client.37074.objecter _send_command 3 2024-04-14T15:16:45.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.112+0000 7f384dffb700 1 -- 172.21.15.90:0/1194098772 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f38681107d0 con 0x7f38681a9ea0 2024-04-14T15:16:45.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.112+0000 7f384ffff700 10 client.37074.objecter ms_handle_connect 0x7f38681a9ea0 2024-04-14T15:16:45.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.113+0000 7f384ffff700 1 -- 172.21.15.90:0/1194098772 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f38681107d0 con 0x7f38681a9ea0 2024-04-14T15:16:45.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.113+0000 7f384ffff700 10 client.37074.objecter ms_dispatch 0x7f3868001550 command_reply(tid 3: 0 ) 2024-04-14T15:16:45.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.113+0000 7f384ffff700 10 client.37074.objecter _finish_command 3 = system:0 2024-04-14T15:16:45.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.113+0000 7f384ffff700 20 client.37074.objecter put_session s=0x7f38681a9d30 osd=1 4 2024-04-14T15:16:45.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.113+0000 7f384ffff700 15 client.37074.objecter _session_command_op_remove 1 3 2024-04-14T15:16:45.133 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:16:45.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.133+0000 7f384dffb700 10 client.37074.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:16:45.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.133+0000 7f384dffb700 15 client.37074.objecter _session_command_op_assign -1 4 2024-04-14T15:16:45.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.133+0000 7f384dffb700 20 client.37074.objecter _get_session s=0x7f38681a9d30 osd=1 3 2024-04-14T15:16:45.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.133+0000 7f384dffb700 20 client.37074.objecter put_session s=0x7f38681a9d30 osd=1 3 2024-04-14T15:16:45.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.133+0000 7f384dffb700 20 client.37074.objecter _get_session s=0x7f38681a9d30 osd=1 3 2024-04-14T15:16:45.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.133+0000 7f384dffb700 15 client.37074.objecter _session_command_op_remove -1 4 2024-04-14T15:16:45.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.133+0000 7f384dffb700 20 client.37074.objecter get_session s=0x7f38681a9d30 osd=1 3 2024-04-14T15:16:45.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.133+0000 7f384dffb700 15 client.37074.objecter _session_command_op_assign 1 4 2024-04-14T15:16:45.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.133+0000 7f384dffb700 20 client.37074.objecter put_session s=0x7f38681a9d30 osd=1 4 2024-04-14T15:16:45.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.133+0000 7f384dffb700 10 client.37074.objecter _send_command 4 2024-04-14T15:16:45.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.133+0000 7f384dffb700 1 -- 172.21.15.90:0/1194098772 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f38681b3230 con 0x7f38681a9ea0 2024-04-14T15:16:45.135 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:44 smithi090 ceph-mon[41670]: pgmap v449: 238 pgs: 32 unknown, 206 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:16:45.135 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:44 smithi090 ceph-mon[41670]: 2.1 scrub starts 2024-04-14T15:16:45.135 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:44 smithi090 ceph-mon[41670]: 2.1 scrub ok 2024-04-14T15:16:45.135 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:44 smithi090 ceph-mon[41670]: 11.19 scrub starts 2024-04-14T15:16:45.135 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:44 smithi090 ceph-mon[41670]: 11.19 scrub ok 2024-04-14T15:16:45.135 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:44 smithi090 ceph-mon[41670]: 1.0 scrub starts 2024-04-14T15:16:45.135 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:44 smithi090 ceph-mon[41670]: 1.0 scrub ok 2024-04-14T15:16:45.135 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:44 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2085825169' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi090-93363-57"}]': finished 2024-04-14T15:16:45.135 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:44 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-45","var": "hit_set_period","val": "3"}]': finished 2024-04-14T15:16:45.136 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:44 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-14T15:16:45.136 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:44 smithi090 ceph-mon[41670]: osdmap e337: 8 total, 8 up, 8 in 2024-04-14T15:16:45.136 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:44 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-14T15:16:45.136 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:44 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1101952089' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi090-93363-58"}]: dispatch 2024-04-14T15:16:45.136 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:44 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1101952089' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi090-93363-58"}]: dispatch 2024-04-14T15:16:45.136 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:44 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1101952089' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi090-93363-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:16:45.136 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:44 smithi090 ceph-mon[30399]: pgmap v449: 238 pgs: 32 unknown, 206 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:16:45.136 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:44 smithi090 ceph-mon[30399]: 2.1 scrub starts 2024-04-14T15:16:45.136 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:44 smithi090 ceph-mon[30399]: 2.1 scrub ok 2024-04-14T15:16:45.136 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:44 smithi090 ceph-mon[30399]: 11.19 scrub starts 2024-04-14T15:16:45.136 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:44 smithi090 ceph-mon[30399]: 11.19 scrub ok 2024-04-14T15:16:45.136 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:44 smithi090 ceph-mon[30399]: 1.0 scrub starts 2024-04-14T15:16:45.136 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:44 smithi090 ceph-mon[30399]: 1.0 scrub ok 2024-04-14T15:16:45.137 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:44 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2085825169' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi090-93363-57"}]': finished 2024-04-14T15:16:45.137 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:44 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-45","var": "hit_set_period","val": "3"}]': finished 2024-04-14T15:16:45.137 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:44 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-14T15:16:45.137 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:44 smithi090 ceph-mon[30399]: osdmap e337: 8 total, 8 up, 8 in 2024-04-14T15:16:45.137 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:44 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-14T15:16:45.137 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:44 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1101952089' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi090-93363-58"}]: dispatch 2024-04-14T15:16:45.137 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:44 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1101952089' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi090-93363-58"}]: dispatch 2024-04-14T15:16:45.137 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:44 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1101952089' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi090-93363-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:16:45.137 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:16:45.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.135+0000 7f384ffff700 1 -- 172.21.15.90:0/1194098772 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f38681b3230 con 0x7f38681a9ea0 2024-04-14T15:16:45.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.135+0000 7f384ffff700 10 client.37074.objecter ms_dispatch 0x7f3868001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:16:45.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.135+0000 7f384ffff700 10 client.37074.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:16:45.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.135+0000 7f384ffff700 20 client.37074.objecter put_session s=0x7f38681a9d30 osd=1 4 2024-04-14T15:16:45.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.135+0000 7f384ffff700 15 client.37074.objecter _session_command_op_remove 1 4 2024-04-14T15:16:45.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.136+0000 7f384dffb700 10 client.37074.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:16:45.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.136+0000 7f384dffb700 15 client.37074.objecter _session_command_op_assign -1 5 2024-04-14T15:16:45.138 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:16:45.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.136+0000 7f384dffb700 20 client.37074.objecter _get_session s=0x7f3868108b10 osd=2 3 2024-04-14T15:16:45.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.136+0000 7f384dffb700 20 client.37074.objecter put_session s=0x7f3868108b10 osd=2 3 2024-04-14T15:16:45.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.136+0000 7f384dffb700 20 client.37074.objecter _get_session s=0x7f3868108b10 osd=2 3 2024-04-14T15:16:45.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.136+0000 7f384dffb700 15 client.37074.objecter _session_command_op_remove -1 5 2024-04-14T15:16:45.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.136+0000 7f384dffb700 20 client.37074.objecter get_session s=0x7f3868108b10 osd=2 3 2024-04-14T15:16:45.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.136+0000 7f384dffb700 15 client.37074.objecter _session_command_op_assign 2 5 2024-04-14T15:16:45.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.136+0000 7f384dffb700 20 client.37074.objecter put_session s=0x7f3868108b10 osd=2 4 2024-04-14T15:16:45.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.136+0000 7f384dffb700 10 client.37074.objecter _send_command 5 2024-04-14T15:16:45.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.136+0000 7f384dffb700 1 -- 172.21.15.90:0/1194098772 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f3868063c90 con 0x7f3868108c80 2024-04-14T15:16:45.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.136+0000 7f384ffff700 10 client.37074.objecter ms_handle_connect 0x7f3868108c80 2024-04-14T15:16:45.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.137+0000 7f384ffff700 1 -- 172.21.15.90:0/1194098772 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f3868063c90 con 0x7f3868108c80 2024-04-14T15:16:45.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.137+0000 7f384ffff700 10 client.37074.objecter ms_dispatch 0x7f3868001550 command_reply(tid 5: 0 ) 2024-04-14T15:16:45.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.137+0000 7f384ffff700 10 client.37074.objecter _finish_command 5 = system:0 2024-04-14T15:16:45.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.137+0000 7f384ffff700 20 client.37074.objecter put_session s=0x7f3868108b10 osd=2 4 2024-04-14T15:16:45.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.137+0000 7f384ffff700 15 client.37074.objecter _session_command_op_remove 2 5 2024-04-14T15:16:45.146 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:16:45.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.162+0000 7f384dffb700 10 client.37074.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:16:45.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.162+0000 7f384dffb700 15 client.37074.objecter _session_command_op_assign -1 6 2024-04-14T15:16:45.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.162+0000 7f384dffb700 20 client.37074.objecter _get_session s=0x7f3868108b10 osd=2 3 2024-04-14T15:16:45.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.162+0000 7f384dffb700 20 client.37074.objecter put_session s=0x7f3868108b10 osd=2 3 2024-04-14T15:16:45.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.162+0000 7f384dffb700 20 client.37074.objecter _get_session s=0x7f3868108b10 osd=2 3 2024-04-14T15:16:45.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.162+0000 7f384dffb700 15 client.37074.objecter _session_command_op_remove -1 6 2024-04-14T15:16:45.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.162+0000 7f384dffb700 20 client.37074.objecter get_session s=0x7f3868108b10 osd=2 3 2024-04-14T15:16:45.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.162+0000 7f384dffb700 15 client.37074.objecter _session_command_op_assign 2 6 2024-04-14T15:16:45.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.162+0000 7f384dffb700 20 client.37074.objecter put_session s=0x7f3868108b10 osd=2 4 2024-04-14T15:16:45.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.162+0000 7f384dffb700 10 client.37074.objecter _send_command 6 2024-04-14T15:16:45.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.162+0000 7f384dffb700 1 -- 172.21.15.90:0/1194098772 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f3868063140 con 0x7f3868108c80 2024-04-14T15:16:45.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.164+0000 7f384ffff700 1 -- 172.21.15.90:0/1194098772 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f3868063140 con 0x7f3868108c80 2024-04-14T15:16:45.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.164+0000 7f384ffff700 10 client.37074.objecter ms_dispatch 0x7f3868001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:16:45.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.164+0000 7f384ffff700 10 client.37074.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:16:45.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.164+0000 7f384ffff700 20 client.37074.objecter put_session s=0x7f3868108b10 osd=2 4 2024-04-14T15:16:45.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.164+0000 7f384ffff700 15 client.37074.objecter _session_command_op_remove 2 6 2024-04-14T15:16:45.164 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:16:45.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.164+0000 7f384dffb700 10 client.37074.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:16:45.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.164+0000 7f384dffb700 15 client.37074.objecter _session_command_op_assign -1 7 2024-04-14T15:16:45.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.164+0000 7f384dffb700 20 client.37074.objecter _get_session s=0x7f3868063540 osd=3 3 2024-04-14T15:16:45.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.164+0000 7f384dffb700 20 client.37074.objecter put_session s=0x7f3868063540 osd=3 3 2024-04-14T15:16:45.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.164+0000 7f384dffb700 20 client.37074.objecter _get_session s=0x7f3868063540 osd=3 3 2024-04-14T15:16:45.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.164+0000 7f384dffb700 15 client.37074.objecter _session_command_op_remove -1 7 2024-04-14T15:16:45.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.164+0000 7f384dffb700 20 client.37074.objecter get_session s=0x7f3868063540 osd=3 3 2024-04-14T15:16:45.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.164+0000 7f384dffb700 15 client.37074.objecter _session_command_op_assign 3 7 2024-04-14T15:16:45.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.164+0000 7f384dffb700 20 client.37074.objecter put_session s=0x7f3868063540 osd=3 4 2024-04-14T15:16:45.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.164+0000 7f384dffb700 10 client.37074.objecter _send_command 7 2024-04-14T15:16:45.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.164+0000 7f384dffb700 1 -- 172.21.15.90:0/1194098772 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f38681bc680 con 0x7f38681b40c0 2024-04-14T15:16:45.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.165+0000 7f384ffff700 10 client.37074.objecter ms_handle_connect 0x7f38681b40c0 2024-04-14T15:16:45.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.165+0000 7f384ffff700 1 -- 172.21.15.90:0/1194098772 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f38681bc680 con 0x7f38681b40c0 2024-04-14T15:16:45.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.165+0000 7f384ffff700 10 client.37074.objecter ms_dispatch 0x7f3868001550 command_reply(tid 7: 0 ) 2024-04-14T15:16:45.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.165+0000 7f384ffff700 10 client.37074.objecter _finish_command 7 = system:0 2024-04-14T15:16:45.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.165+0000 7f384ffff700 20 client.37074.objecter put_session s=0x7f3868063540 osd=3 4 2024-04-14T15:16:45.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.165+0000 7f384ffff700 15 client.37074.objecter _session_command_op_remove 3 7 2024-04-14T15:16:45.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:45.174 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:45.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:16:45.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.183+0000 7f384dffb700 10 client.37074.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:16:45.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.183+0000 7f384dffb700 15 client.37074.objecter _session_command_op_assign -1 8 2024-04-14T15:16:45.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.183+0000 7f384dffb700 20 client.37074.objecter _get_session s=0x7f3868063540 osd=3 3 2024-04-14T15:16:45.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.183+0000 7f384dffb700 20 client.37074.objecter put_session s=0x7f3868063540 osd=3 3 2024-04-14T15:16:45.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.183+0000 7f384dffb700 20 client.37074.objecter _get_session s=0x7f3868063540 osd=3 3 2024-04-14T15:16:45.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.183+0000 7f384dffb700 15 client.37074.objecter _session_command_op_remove -1 8 2024-04-14T15:16:45.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.183+0000 7f384dffb700 20 client.37074.objecter get_session s=0x7f3868063540 osd=3 3 2024-04-14T15:16:45.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.183+0000 7f384dffb700 15 client.37074.objecter _session_command_op_assign 3 8 2024-04-14T15:16:45.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.183+0000 7f384dffb700 20 client.37074.objecter put_session s=0x7f3868063540 osd=3 4 2024-04-14T15:16:45.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.183+0000 7f384dffb700 10 client.37074.objecter _send_command 8 2024-04-14T15:16:45.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.183+0000 7f384dffb700 1 -- 172.21.15.90:0/1194098772 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f38681b3690 con 0x7f38681b40c0 2024-04-14T15:16:45.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.185+0000 7f384ffff700 1 -- 172.21.15.90:0/1194098772 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f38681b3690 con 0x7f38681b40c0 2024-04-14T15:16:45.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.185+0000 7f384ffff700 10 client.37074.objecter ms_dispatch 0x7f3868001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:16:45.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.185+0000 7f384ffff700 10 client.37074.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:16:45.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.185+0000 7f384ffff700 20 client.37074.objecter put_session s=0x7f3868063540 osd=3 4 2024-04-14T15:16:45.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.185+0000 7f384ffff700 15 client.37074.objecter _session_command_op_remove 3 8 2024-04-14T15:16:45.186 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:16:45.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.185+0000 7f384dffb700 10 client.37074.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:16:45.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.185+0000 7f384dffb700 15 client.37074.objecter _session_command_op_assign -1 9 2024-04-14T15:16:45.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.185+0000 7f384dffb700 20 client.37074.objecter _get_session s=0x7f38681b3440 osd=4 3 2024-04-14T15:16:45.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.185+0000 7f384dffb700 20 client.37074.objecter put_session s=0x7f38681b3440 osd=4 3 2024-04-14T15:16:45.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.185+0000 7f384dffb700 20 client.37074.objecter _get_session s=0x7f38681b3440 osd=4 3 2024-04-14T15:16:45.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.185+0000 7f384dffb700 15 client.37074.objecter _session_command_op_remove -1 9 2024-04-14T15:16:45.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.185+0000 7f384dffb700 20 client.37074.objecter get_session s=0x7f38681b3440 osd=4 3 2024-04-14T15:16:45.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.185+0000 7f384dffb700 15 client.37074.objecter _session_command_op_assign 4 9 2024-04-14T15:16:45.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.185+0000 7f384dffb700 20 client.37074.objecter put_session s=0x7f38681b3440 osd=4 4 2024-04-14T15:16:45.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.185+0000 7f384dffb700 10 client.37074.objecter _send_command 9 2024-04-14T15:16:45.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.185+0000 7f384dffb700 1 -- 172.21.15.90:0/1194098772 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f38681b3640 con 0x7f38681b8970 2024-04-14T15:16:45.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.186+0000 7f384ffff700 10 client.37074.objecter ms_handle_connect 0x7f38681b8970 2024-04-14T15:16:45.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.187+0000 7f384ffff700 1 -- 172.21.15.90:0/1194098772 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f38681b3640 con 0x7f38681b8970 2024-04-14T15:16:45.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.187+0000 7f384ffff700 10 client.37074.objecter ms_dispatch 0x7f3868001550 command_reply(tid 9: 0 ) 2024-04-14T15:16:45.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.187+0000 7f384ffff700 10 client.37074.objecter _finish_command 9 = system:0 2024-04-14T15:16:45.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.187+0000 7f384ffff700 20 client.37074.objecter put_session s=0x7f38681b3440 osd=4 4 2024-04-14T15:16:45.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.187+0000 7f384ffff700 15 client.37074.objecter _session_command_op_remove 4 9 2024-04-14T15:16:45.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.203+0000 7f384dffb700 10 client.37074.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:16:45.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.203+0000 7f384dffb700 15 client.37074.objecter _session_command_op_assign -1 10 2024-04-14T15:16:45.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.203+0000 7f384dffb700 20 client.37074.objecter _get_session s=0x7f38681b3440 osd=4 3 2024-04-14T15:16:45.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.203+0000 7f384dffb700 20 client.37074.objecter put_session s=0x7f38681b3440 osd=4 3 2024-04-14T15:16:45.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.203+0000 7f384dffb700 20 client.37074.objecter _get_session s=0x7f38681b3440 osd=4 3 2024-04-14T15:16:45.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.203+0000 7f384dffb700 15 client.37074.objecter _session_command_op_remove -1 10 2024-04-14T15:16:45.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.203+0000 7f384dffb700 20 client.37074.objecter get_session s=0x7f38681b3440 osd=4 3 2024-04-14T15:16:45.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.203+0000 7f384dffb700 15 client.37074.objecter _session_command_op_assign 4 10 2024-04-14T15:16:45.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.203+0000 7f384dffb700 20 client.37074.objecter put_session s=0x7f38681b3440 osd=4 4 2024-04-14T15:16:45.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.203+0000 7f384dffb700 10 client.37074.objecter _send_command 10 2024-04-14T15:16:45.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.203+0000 7f384dffb700 1 -- 172.21.15.90:0/1194098772 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f38681bdfe0 con 0x7f38681b8970 2024-04-14T15:16:45.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.205+0000 7f384ffff700 1 -- 172.21.15.90:0/1194098772 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f38681bdfe0 con 0x7f38681b8970 2024-04-14T15:16:45.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.205+0000 7f384ffff700 10 client.37074.objecter ms_dispatch 0x7f3868001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:16:45.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.205+0000 7f384ffff700 10 client.37074.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:16:45.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.205+0000 7f384ffff700 20 client.37074.objecter put_session s=0x7f38681b3440 osd=4 4 2024-04-14T15:16:45.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.205+0000 7f384ffff700 15 client.37074.objecter _session_command_op_remove 4 10 2024-04-14T15:16:45.206 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:16:45.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.205+0000 7f384dffb700 10 client.37074.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:16:45.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.205+0000 7f384dffb700 15 client.37074.objecter _session_command_op_assign -1 11 2024-04-14T15:16:45.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.205+0000 7f384dffb700 20 client.37074.objecter _get_session s=0x7f38681bdfe0 osd=5 3 2024-04-14T15:16:45.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.205+0000 7f384dffb700 20 client.37074.objecter put_session s=0x7f38681bdfe0 osd=5 3 2024-04-14T15:16:45.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.205+0000 7f384dffb700 20 client.37074.objecter _get_session s=0x7f38681bdfe0 osd=5 3 2024-04-14T15:16:45.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.205+0000 7f384dffb700 15 client.37074.objecter _session_command_op_remove -1 11 2024-04-14T15:16:45.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.205+0000 7f384dffb700 20 client.37074.objecter get_session s=0x7f38681bdfe0 osd=5 3 2024-04-14T15:16:45.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.205+0000 7f384dffb700 15 client.37074.objecter _session_command_op_assign 5 11 2024-04-14T15:16:45.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.205+0000 7f384dffb700 20 client.37074.objecter put_session s=0x7f38681bdfe0 osd=5 4 2024-04-14T15:16:45.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.205+0000 7f384dffb700 10 client.37074.objecter _send_command 11 2024-04-14T15:16:45.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.205+0000 7f384dffb700 1 -- 172.21.15.90:0/1194098772 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f38681beb70 con 0x7f38681be700 2024-04-14T15:16:45.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.205+0000 7f384ffff700 10 client.37074.objecter ms_handle_connect 0x7f38681be700 2024-04-14T15:16:45.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.206+0000 7f384ffff700 1 -- 172.21.15.90:0/1194098772 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f38681beb70 con 0x7f38681be700 2024-04-14T15:16:45.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.206+0000 7f384ffff700 10 client.37074.objecter ms_dispatch 0x7f3868001550 command_reply(tid 11: 0 ) 2024-04-14T15:16:45.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.206+0000 7f384ffff700 10 client.37074.objecter _finish_command 11 = system:0 2024-04-14T15:16:45.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.206+0000 7f384ffff700 20 client.37074.objecter put_session s=0x7f38681bdfe0 osd=5 4 2024-04-14T15:16:45.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.206+0000 7f384ffff700 15 client.37074.objecter _session_command_op_remove 5 11 2024-04-14T15:16:45.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.222+0000 7f384dffb700 10 client.37074.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:16:45.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.222+0000 7f384dffb700 15 client.37074.objecter _session_command_op_assign -1 12 2024-04-14T15:16:45.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.222+0000 7f384dffb700 20 client.37074.objecter _get_session s=0x7f38681bdfe0 osd=5 3 2024-04-14T15:16:45.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.222+0000 7f384dffb700 20 client.37074.objecter put_session s=0x7f38681bdfe0 osd=5 3 2024-04-14T15:16:45.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.222+0000 7f384dffb700 20 client.37074.objecter _get_session s=0x7f38681bdfe0 osd=5 3 2024-04-14T15:16:45.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.222+0000 7f384dffb700 15 client.37074.objecter _session_command_op_remove -1 12 2024-04-14T15:16:45.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.222+0000 7f384dffb700 20 client.37074.objecter get_session s=0x7f38681bdfe0 osd=5 3 2024-04-14T15:16:45.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.222+0000 7f384dffb700 15 client.37074.objecter _session_command_op_assign 5 12 2024-04-14T15:16:45.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.222+0000 7f384dffb700 20 client.37074.objecter put_session s=0x7f38681bdfe0 osd=5 4 2024-04-14T15:16:45.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.222+0000 7f384dffb700 10 client.37074.objecter _send_command 12 2024-04-14T15:16:45.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.222+0000 7f384dffb700 1 -- 172.21.15.90:0/1194098772 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f38681beb70 con 0x7f38681be700 2024-04-14T15:16:45.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.224+0000 7f384ffff700 1 -- 172.21.15.90:0/1194098772 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f38681beb70 con 0x7f38681be700 2024-04-14T15:16:45.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.224+0000 7f384ffff700 10 client.37074.objecter ms_dispatch 0x7f3868001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:16:45.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.225+0000 7f384ffff700 10 client.37074.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:16:45.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.225+0000 7f384ffff700 20 client.37074.objecter put_session s=0x7f38681bdfe0 osd=5 4 2024-04-14T15:16:45.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.225+0000 7f384ffff700 15 client.37074.objecter _session_command_op_remove 5 12 2024-04-14T15:16:45.225 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:16:45.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.225+0000 7f384dffb700 10 client.37074.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:16:45.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.225+0000 7f384dffb700 15 client.37074.objecter _session_command_op_assign -1 13 2024-04-14T15:16:45.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.225+0000 7f384dffb700 20 client.37074.objecter _get_session s=0x7f38681beb70 osd=6 3 2024-04-14T15:16:45.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.225+0000 7f384dffb700 20 client.37074.objecter put_session s=0x7f38681beb70 osd=6 3 2024-04-14T15:16:45.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.225+0000 7f384dffb700 20 client.37074.objecter _get_session s=0x7f38681beb70 osd=6 3 2024-04-14T15:16:45.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.225+0000 7f384dffb700 15 client.37074.objecter _session_command_op_remove -1 13 2024-04-14T15:16:45.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.225+0000 7f384dffb700 20 client.37074.objecter get_session s=0x7f38681beb70 osd=6 3 2024-04-14T15:16:45.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.225+0000 7f384dffb700 15 client.37074.objecter _session_command_op_assign 6 13 2024-04-14T15:16:45.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.225+0000 7f384dffb700 20 client.37074.objecter put_session s=0x7f38681beb70 osd=6 4 2024-04-14T15:16:45.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.225+0000 7f384dffb700 10 client.37074.objecter _send_command 13 2024-04-14T15:16:45.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.225+0000 7f384dffb700 1 -- 172.21.15.90:0/1194098772 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f38681bf000 con 0x7f38681c60d0 2024-04-14T15:16:45.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.225+0000 7f384ffff700 10 client.37074.objecter ms_handle_connect 0x7f38681c60d0 2024-04-14T15:16:45.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.226+0000 7f384ffff700 1 -- 172.21.15.90:0/1194098772 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f38681bf000 con 0x7f38681c60d0 2024-04-14T15:16:45.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.226+0000 7f384ffff700 10 client.37074.objecter ms_dispatch 0x7f3868001550 command_reply(tid 13: 0 ) 2024-04-14T15:16:45.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.226+0000 7f384ffff700 10 client.37074.objecter _finish_command 13 = system:0 2024-04-14T15:16:45.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.226+0000 7f384ffff700 20 client.37074.objecter put_session s=0x7f38681beb70 osd=6 4 2024-04-14T15:16:45.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.226+0000 7f384ffff700 15 client.37074.objecter _session_command_op_remove 6 13 2024-04-14T15:16:45.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.242+0000 7f384dffb700 10 client.37074.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:16:45.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.242+0000 7f384dffb700 15 client.37074.objecter _session_command_op_assign -1 14 2024-04-14T15:16:45.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.242+0000 7f384dffb700 20 client.37074.objecter _get_session s=0x7f38681beb70 osd=6 3 2024-04-14T15:16:45.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.242+0000 7f384dffb700 20 client.37074.objecter put_session s=0x7f38681beb70 osd=6 3 2024-04-14T15:16:45.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.242+0000 7f384dffb700 20 client.37074.objecter _get_session s=0x7f38681beb70 osd=6 3 2024-04-14T15:16:45.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.242+0000 7f384dffb700 15 client.37074.objecter _session_command_op_remove -1 14 2024-04-14T15:16:45.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.242+0000 7f384dffb700 20 client.37074.objecter get_session s=0x7f38681beb70 osd=6 3 2024-04-14T15:16:45.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.242+0000 7f384dffb700 15 client.37074.objecter _session_command_op_assign 6 14 2024-04-14T15:16:45.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.242+0000 7f384dffb700 20 client.37074.objecter put_session s=0x7f38681beb70 osd=6 4 2024-04-14T15:16:45.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.242+0000 7f384dffb700 10 client.37074.objecter _send_command 14 2024-04-14T15:16:45.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.242+0000 7f384dffb700 1 -- 172.21.15.90:0/1194098772 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f38681c9c10 con 0x7f38681c60d0 2024-04-14T15:16:45.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.244+0000 7f384ffff700 1 -- 172.21.15.90:0/1194098772 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f38681c9c10 con 0x7f38681c60d0 2024-04-14T15:16:45.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.244+0000 7f384ffff700 10 client.37074.objecter ms_dispatch 0x7f3868001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:16:45.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.244+0000 7f384ffff700 10 client.37074.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:16:45.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.244+0000 7f384ffff700 20 client.37074.objecter put_session s=0x7f38681beb70 osd=6 4 2024-04-14T15:16:45.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.244+0000 7f384ffff700 15 client.37074.objecter _session_command_op_remove 6 14 2024-04-14T15:16:45.245 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:16:45.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.245+0000 7f384dffb700 10 client.37074.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:16:45.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.245+0000 7f384dffb700 15 client.37074.objecter _session_command_op_assign -1 15 2024-04-14T15:16:45.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.245+0000 7f384dffb700 20 client.37074.objecter _get_session s=0x7f38681bf080 osd=7 3 2024-04-14T15:16:45.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.245+0000 7f384dffb700 20 client.37074.objecter put_session s=0x7f38681bf080 osd=7 3 2024-04-14T15:16:45.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.245+0000 7f384dffb700 20 client.37074.objecter _get_session s=0x7f38681bf080 osd=7 3 2024-04-14T15:16:45.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.245+0000 7f384dffb700 15 client.37074.objecter _session_command_op_remove -1 15 2024-04-14T15:16:45.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.245+0000 7f384dffb700 20 client.37074.objecter get_session s=0x7f38681bf080 osd=7 3 2024-04-14T15:16:45.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.245+0000 7f384dffb700 15 client.37074.objecter _session_command_op_assign 7 15 2024-04-14T15:16:45.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.245+0000 7f384dffb700 20 client.37074.objecter put_session s=0x7f38681bf080 osd=7 4 2024-04-14T15:16:45.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.245+0000 7f384dffb700 10 client.37074.objecter _send_command 15 2024-04-14T15:16:45.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.245+0000 7f384dffb700 1 -- 172.21.15.90:0/1194098772 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f38681caef0 con 0x7f38681ca330 2024-04-14T15:16:45.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.245+0000 7f384ffff700 10 client.37074.objecter ms_handle_connect 0x7f38681ca330 2024-04-14T15:16:45.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.246+0000 7f384ffff700 1 -- 172.21.15.90:0/1194098772 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f38681caef0 con 0x7f38681ca330 2024-04-14T15:16:45.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.246+0000 7f384ffff700 10 client.37074.objecter ms_dispatch 0x7f3868001550 command_reply(tid 15: 0 ) 2024-04-14T15:16:45.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.246+0000 7f384ffff700 10 client.37074.objecter _finish_command 15 = system:0 2024-04-14T15:16:45.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.246+0000 7f384ffff700 20 client.37074.objecter put_session s=0x7f38681bf080 osd=7 4 2024-04-14T15:16:45.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.246+0000 7f384ffff700 15 client.37074.objecter _session_command_op_remove 7 15 2024-04-14T15:16:45.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.262+0000 7f384dffb700 10 client.37074.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:16:45.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.262+0000 7f384dffb700 15 client.37074.objecter _session_command_op_assign -1 16 2024-04-14T15:16:45.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.262+0000 7f384dffb700 20 client.37074.objecter _get_session s=0x7f38681bf080 osd=7 3 2024-04-14T15:16:45.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.262+0000 7f384dffb700 20 client.37074.objecter put_session s=0x7f38681bf080 osd=7 3 2024-04-14T15:16:45.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.262+0000 7f384dffb700 20 client.37074.objecter _get_session s=0x7f38681bf080 osd=7 3 2024-04-14T15:16:45.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.262+0000 7f384dffb700 15 client.37074.objecter _session_command_op_remove -1 16 2024-04-14T15:16:45.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.262+0000 7f384dffb700 20 client.37074.objecter get_session s=0x7f38681bf080 osd=7 3 2024-04-14T15:16:45.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.262+0000 7f384dffb700 15 client.37074.objecter _session_command_op_assign 7 16 2024-04-14T15:16:45.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.262+0000 7f384dffb700 20 client.37074.objecter put_session s=0x7f38681bf080 osd=7 4 2024-04-14T15:16:45.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.262+0000 7f384dffb700 10 client.37074.objecter _send_command 16 2024-04-14T15:16:45.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.262+0000 7f384dffb700 1 -- 172.21.15.90:0/1194098772 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f38681caef0 con 0x7f38681ca330 2024-04-14T15:16:45.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.264+0000 7f384ffff700 1 -- 172.21.15.90:0/1194098772 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f38681caef0 con 0x7f38681ca330 2024-04-14T15:16:45.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.264+0000 7f384ffff700 10 client.37074.objecter ms_dispatch 0x7f3868001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:16:45.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.264+0000 7f384ffff700 10 client.37074.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:16:45.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.264+0000 7f384ffff700 20 client.37074.objecter put_session s=0x7f38681bf080 osd=7 4 2024-04-14T15:16:45.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.264+0000 7f384ffff700 15 client.37074.objecter _session_command_op_remove 7 16 2024-04-14T15:16:45.264 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:16:45.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.264+0000 7f384dffb700 10 librados: watch_flush enter 2024-04-14T15:16:45.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.264+0000 7f384dffb700 10 librados: watch_flush exit 2024-04-14T15:16:45.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.264+0000 7f384dffb700 10 client.37074.objecter close_session for osd.0 2024-04-14T15:16:45.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.264+0000 7f384dffb700 1 -- 172.21.15.90:0/1194098772 >> v1:172.21.15.90:6801/3761806885 conn(0x7f38681aa9c0 legacy=0x7f38681aada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:45.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.264+0000 7f384dffb700 20 client.37074.objecter put_session s=0x7f38681122e0 osd=0 1 2024-04-14T15:16:45.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.264+0000 7f384dffb700 10 client.37074.objecter close_session for osd.1 2024-04-14T15:16:45.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.264+0000 7f384dffb700 1 -- 172.21.15.90:0/1194098772 >> v1:172.21.15.90:6805/4170361699 conn(0x7f38681a9ea0 legacy=0x7f38681b1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:45.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.264+0000 7f384dffb700 20 client.37074.objecter put_session s=0x7f38681a9d30 osd=1 1 2024-04-14T15:16:45.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.264+0000 7f384dffb700 10 client.37074.objecter close_session for osd.2 2024-04-14T15:16:45.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.264+0000 7f384dffb700 1 -- 172.21.15.90:0/1194098772 >> v1:172.21.15.90:6809/1686184251 conn(0x7f3868108c80 legacy=0x7f38681b6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:45.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.264+0000 7f384dffb700 20 client.37074.objecter put_session s=0x7f3868108b10 osd=2 1 2024-04-14T15:16:45.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.264+0000 7f384dffb700 10 client.37074.objecter close_session for osd.3 2024-04-14T15:16:45.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.264+0000 7f384dffb700 1 -- 172.21.15.90:0/1194098772 >> v1:172.21.15.90:6813/2270166774 conn(0x7f38681b40c0 legacy=0x7f38681bae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:45.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.264+0000 7f384dffb700 20 client.37074.objecter put_session s=0x7f3868063540 osd=3 1 2024-04-14T15:16:45.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.264+0000 7f384dffb700 10 client.37074.objecter close_session for osd.4 2024-04-14T15:16:45.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.264+0000 7f384dffb700 1 -- 172.21.15.90:0/1194098772 >> v1:172.21.15.92:6800/3887549733 conn(0x7f38681b8970 legacy=0x7f38681bc860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:45.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.264+0000 7f384dffb700 20 client.37074.objecter put_session s=0x7f38681b3440 osd=4 1 2024-04-14T15:16:45.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.264+0000 7f384dffb700 10 client.37074.objecter close_session for osd.5 2024-04-14T15:16:45.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.264+0000 7f384dffb700 1 -- 172.21.15.90:0/1194098772 >> v1:172.21.15.92:6804/3244561440 conn(0x7f38681be700 legacy=0x7f38681c1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:45.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.264+0000 7f384dffb700 20 client.37074.objecter put_session s=0x7f38681bdfe0 osd=5 1 2024-04-14T15:16:45.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.264+0000 7f384dffb700 10 client.37074.objecter close_session for osd.6 2024-04-14T15:16:45.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.264+0000 7f384dffb700 1 -- 172.21.15.90:0/1194098772 >> v1:172.21.15.92:6808/636125963 conn(0x7f38681c60d0 legacy=0x7f38681c84c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:45.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.264+0000 7f384dffb700 20 client.37074.objecter put_session s=0x7f38681beb70 osd=6 1 2024-04-14T15:16:45.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.264+0000 7f384dffb700 10 client.37074.objecter close_session for osd.7 2024-04-14T15:16:45.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.264+0000 7f384dffb700 1 -- 172.21.15.90:0/1194098772 >> v1:172.21.15.92:6812/1773706716 conn(0x7f38681ca330 legacy=0x7f38681ca7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:45.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.264+0000 7f384dffb700 20 client.37074.objecter put_session s=0x7f38681bf080 osd=7 1 2024-04-14T15:16:45.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.264+0000 7f384dffb700 20 client.37074.objecter shutdown clearing up homeless session... 2024-04-14T15:16:45.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.264+0000 7f384dffb700 10 client.37074.objecter successfully canceled tick 2024-04-14T15:16:45.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.264+0000 7f386759e700 1 -- 172.21.15.90:0/1194098772 reap_dead start 2024-04-14T15:16:45.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.264+0000 7f384dffb700 1 -- 172.21.15.90:0/1194098772 >> v1:172.21.15.90:6800/420935427 conn(0x7f3850071090 legacy=0x7f3850073550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:45.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.264+0000 7f386759e700 1 -- 172.21.15.90:0/1194098772 reap_dead start 2024-04-14T15:16:45.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.264+0000 7f384dffb700 1 -- 172.21.15.90:0/1194098772 >> v1:172.21.15.90:6789/0 conn(0x7f386810a740 legacy=0x7f386810c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:45.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.264+0000 7f386759e700 1 -- 172.21.15.90:0/1194098772 reap_dead start 2024-04-14T15:16:45.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.264+0000 7f386759e700 1 -- 172.21.15.90:0/1194098772 reap_dead start 2024-04-14T15:16:45.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.264+0000 7f386759e700 1 -- 172.21.15.90:0/1194098772 reap_dead start 2024-04-14T15:16:45.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.264+0000 7f384dffb700 1 -- 172.21.15.90:0/1194098772 shutdown_connections 2024-04-14T15:16:45.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.264+0000 7f384dffb700 1 -- 172.21.15.90:0/1194098772 >> 172.21.15.90:0/1194098772 conn(0x7f38680fd290 msgr2=0x7f3868100040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:16:45.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.264+0000 7f384dffb700 1 -- 172.21.15.90:0/1194098772 shutdown_connections 2024-04-14T15:16:45.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.264+0000 7f384dffb700 1 -- 172.21.15.90:0/1194098772 wait complete. 2024-04-14T15:16:45.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:45.264+0000 7f384dffb700 1 librados: shutdown 2024-04-14T15:16:45.269 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:16:45.269 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:16:45.269 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:16:45.269 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:16:45.269 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:16:45.269 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:16:45.269 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:16:45.269 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:16:45.275 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:16:45.301 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:45.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:45.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:16:45.372 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:16:44 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:16:45.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:44 smithi092 ceph-mon[33966]: pgmap v449: 238 pgs: 32 unknown, 206 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:16:45.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:44 smithi092 ceph-mon[33966]: 2.1 scrub starts 2024-04-14T15:16:45.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:44 smithi092 ceph-mon[33966]: 2.1 scrub ok 2024-04-14T15:16:45.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:44 smithi092 ceph-mon[33966]: 11.19 scrub starts 2024-04-14T15:16:45.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:44 smithi092 ceph-mon[33966]: 11.19 scrub ok 2024-04-14T15:16:45.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:44 smithi092 ceph-mon[33966]: 1.0 scrub starts 2024-04-14T15:16:45.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:44 smithi092 ceph-mon[33966]: 1.0 scrub ok 2024-04-14T15:16:45.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:44 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2085825169' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi090-93363-57"}]': finished 2024-04-14T15:16:45.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:44 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-45","var": "hit_set_period","val": "3"}]': finished 2024-04-14T15:16:45.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:44 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-14T15:16:45.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:44 smithi092 ceph-mon[33966]: osdmap e337: 8 total, 8 up, 8 in 2024-04-14T15:16:45.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:44 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-14T15:16:45.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:44 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1101952089' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi090-93363-58"}]: dispatch 2024-04-14T15:16:45.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:44 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1101952089' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi090-93363-58"}]: dispatch 2024-04-14T15:16:45.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:44 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1101952089' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi090-93363-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:16:45.403 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:45.431 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:45.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:45.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:45.533 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:16:45.557 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:45.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:45.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:16:45.659 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:45.684 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:45.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:45.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:45.728 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-14T15:16:45.729 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd reweight 0 1 2024-04-14T15:16:45.786 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:45.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:45.810 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:45.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:45.911 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:45.935 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:45.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:45.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:46.001 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:45 smithi090 ceph-mon[41670]: 6.12 scrub starts 2024-04-14T15:16:46.002 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:45 smithi090 ceph-mon[41670]: 6.12 scrub ok 2024-04-14T15:16:46.002 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:45 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1194098772' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:16:46.002 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:45 smithi090 ceph-mon[41670]: 5.19 scrub starts 2024-04-14T15:16:46.002 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:45 smithi090 ceph-mon[41670]: 5.19 scrub ok 2024-04-14T15:16:46.002 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:45 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:16:46.002 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:45 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-45","var": "hit_set_type","val": "bloom"}]': finished 2024-04-14T15:16:46.002 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:45 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1101952089' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi090-93363-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:16:46.002 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:45 smithi090 ceph-mon[41670]: osdmap e338: 8 total, 8 up, 8 in 2024-04-14T15:16:46.002 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:45 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1101952089' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi090-93363-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi090-93363-58"}]: dispatch 2024-04-14T15:16:46.002 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:45 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-04-14T15:16:46.002 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:45 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-04-14T15:16:46.003 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:45 smithi090 ceph-mon[30399]: 6.12 scrub starts 2024-04-14T15:16:46.003 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:45 smithi090 ceph-mon[30399]: 6.12 scrub ok 2024-04-14T15:16:46.003 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:45 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1194098772' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:16:46.003 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:45 smithi090 ceph-mon[30399]: 5.19 scrub starts 2024-04-14T15:16:46.003 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:45 smithi090 ceph-mon[30399]: 5.19 scrub ok 2024-04-14T15:16:46.003 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:45 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:16:46.003 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:45 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-45","var": "hit_set_type","val": "bloom"}]': finished 2024-04-14T15:16:46.003 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:45 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1101952089' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi090-93363-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:16:46.003 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:45 smithi090 ceph-mon[30399]: osdmap e338: 8 total, 8 up, 8 in 2024-04-14T15:16:46.003 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:45 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1101952089' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi090-93363-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi090-93363-58"}]: dispatch 2024-04-14T15:16:46.003 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:45 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-04-14T15:16:46.004 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:45 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-04-14T15:16:46.037 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:16:46.061 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:46.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:46.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:16:46.163 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:16:46.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:46.187 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:46.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:16:46.288 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:16:46.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:46.313 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:46.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:16:46.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:45 smithi092 ceph-mon[33966]: 6.12 scrub starts 2024-04-14T15:16:46.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:45 smithi092 ceph-mon[33966]: 6.12 scrub ok 2024-04-14T15:16:46.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:45 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1194098772' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:16:46.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:45 smithi092 ceph-mon[33966]: 5.19 scrub starts 2024-04-14T15:16:46.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:45 smithi092 ceph-mon[33966]: 5.19 scrub ok 2024-04-14T15:16:46.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:45 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:16:46.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:45 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-45","var": "hit_set_type","val": "bloom"}]': finished 2024-04-14T15:16:46.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:45 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1101952089' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi090-93363-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:16:46.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:45 smithi092 ceph-mon[33966]: osdmap e338: 8 total, 8 up, 8 in 2024-04-14T15:16:46.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:45 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1101952089' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi090-93363-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi090-93363-58"}]: dispatch 2024-04-14T15:16:46.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:45 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-04-14T15:16:46.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:45 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-04-14T15:16:46.415 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:46.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:46.443 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:46.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:46.544 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:16:46.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:46.570 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:46.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:16:46.671 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:16:46.695 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:46.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:46.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:16:46.797 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:16:46.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:46.825 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:46.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:16:46.926 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:16:46.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:46.951 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:46.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:16:47.052 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:47.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:47.077 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:47.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:47.178 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:16:47.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:46 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:16:47.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:46 smithi090 ceph-mon[30399]: pgmap v452: 238 pgs: 238 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:16:47.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:46 smithi090 ceph-mon[30399]: 6.f scrub starts 2024-04-14T15:16:47.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:46 smithi090 ceph-mon[30399]: 6.f scrub ok 2024-04-14T15:16:47.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:46 smithi090 ceph-mon[30399]: 6.3 scrub starts 2024-04-14T15:16:47.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:46 smithi090 ceph-mon[30399]: 6.3 scrub ok 2024-04-14T15:16:47.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:46 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:16:47.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:46 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:16:47.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:46 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-45","var": "hit_set_fpp","val": ".01"}]': finished 2024-04-14T15:16:47.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:46 smithi090 ceph-mon[30399]: osdmap e339: 8 total, 8 up, 8 in 2024-04-14T15:16:47.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:46 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:16:47.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:46 smithi090 ceph-mon[41670]: pgmap v452: 238 pgs: 238 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:16:47.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:46 smithi090 ceph-mon[41670]: 6.f scrub starts 2024-04-14T15:16:47.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:46 smithi090 ceph-mon[41670]: 6.f scrub ok 2024-04-14T15:16:47.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:46 smithi090 ceph-mon[41670]: 6.3 scrub starts 2024-04-14T15:16:47.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:46 smithi090 ceph-mon[41670]: 6.3 scrub ok 2024-04-14T15:16:47.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:46 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:16:47.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:46 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:16:47.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:46 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-45","var": "hit_set_fpp","val": ".01"}]': finished 2024-04-14T15:16:47.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:46 smithi090 ceph-mon[41670]: osdmap e339: 8 total, 8 up, 8 in 2024-04-14T15:16:47.201 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:47.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:47.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:16:47.302 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:47.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:47.327 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:47.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:47.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:46 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:16:47.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:46 smithi092 ceph-mon[33966]: pgmap v452: 238 pgs: 238 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:16:47.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:46 smithi092 ceph-mon[33966]: 6.f scrub starts 2024-04-14T15:16:47.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:46 smithi092 ceph-mon[33966]: 6.f scrub ok 2024-04-14T15:16:47.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:46 smithi092 ceph-mon[33966]: 6.3 scrub starts 2024-04-14T15:16:47.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:46 smithi092 ceph-mon[33966]: 6.3 scrub ok 2024-04-14T15:16:47.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:46 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:16:47.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:46 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:16:47.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:46 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-45","var": "hit_set_fpp","val": ".01"}]': finished 2024-04-14T15:16:47.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:46 smithi092 ceph-mon[33966]: osdmap e339: 8 total, 8 up, 8 in 2024-04-14T15:16:47.428 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:47.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:47.452 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:47.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:47.553 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:16:47.582 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:47.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:47.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:16:47.687 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:16:47.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:47.713 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:47.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:16:47.814 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:47.839 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:47.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:47.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:47.941 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:47.966 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:47.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:47.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:48.068 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:48.092 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:48.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:48.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:48.194 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:48.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:48.218 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:48.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:48.319 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:16:48.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:48.344 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:48.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:16:48.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:47 smithi092 ceph-mon[33966]: 2.1d scrub starts 2024-04-14T15:16:48.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:47 smithi092 ceph-mon[33966]: 2.1d scrub ok 2024-04-14T15:16:48.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:47 smithi092 ceph-mon[33966]: 2.12 scrub starts 2024-04-14T15:16:48.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:47 smithi092 ceph-mon[33966]: 2.12 scrub ok 2024-04-14T15:16:48.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:47 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1101952089' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi090-93363-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi090-93363-58"}]': finished 2024-04-14T15:16:48.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:47 smithi092 ceph-mon[33966]: osdmap e340: 8 total, 8 up, 8 in 2024-04-14T15:16:48.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:47 smithi090 ceph-mon[41670]: 2.1d scrub starts 2024-04-14T15:16:48.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:47 smithi090 ceph-mon[41670]: 2.1d scrub ok 2024-04-14T15:16:48.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:47 smithi090 ceph-mon[41670]: 2.12 scrub starts 2024-04-14T15:16:48.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:47 smithi090 ceph-mon[41670]: 2.12 scrub ok 2024-04-14T15:16:48.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:47 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1101952089' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi090-93363-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi090-93363-58"}]': finished 2024-04-14T15:16:48.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:47 smithi090 ceph-mon[41670]: osdmap e340: 8 total, 8 up, 8 in 2024-04-14T15:16:48.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:47 smithi090 ceph-mon[30399]: 2.1d scrub starts 2024-04-14T15:16:48.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:47 smithi090 ceph-mon[30399]: 2.1d scrub ok 2024-04-14T15:16:48.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:47 smithi090 ceph-mon[30399]: 2.12 scrub starts 2024-04-14T15:16:48.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:47 smithi090 ceph-mon[30399]: 2.12 scrub ok 2024-04-14T15:16:48.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:47 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1101952089' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi090-93363-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi090-93363-58"}]': finished 2024-04-14T15:16:48.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:47 smithi090 ceph-mon[30399]: osdmap e340: 8 total, 8 up, 8 in 2024-04-14T15:16:48.445 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:16:48.469 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:48.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:48.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:16:48.571 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:16:48.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:48.595 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:48.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:16:48.696 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:16:48.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:48.722 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:48.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:16:48.823 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:16:48.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:48.848 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:48.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:16:48.949 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:16:48.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:48.948+0000 7fc32638c700 1 -- 172.21.15.90:0/4244235965 >> v1:172.21.15.90:6789/0 conn(0x7fc320107760 legacy=0x7fc320107b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:48.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:48.948+0000 7fc32638c700 1 -- 172.21.15.90:0/4244235965 shutdown_connections 2024-04-14T15:16:48.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:48.948+0000 7fc32638c700 1 -- 172.21.15.90:0/4244235965 >> 172.21.15.90:0/4244235965 conn(0x7fc3200fd290 msgr2=0x7fc3200ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:16:48.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:48.948+0000 7fc32638c700 1 -- 172.21.15.90:0/4244235965 shutdown_connections 2024-04-14T15:16:48.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:48.948+0000 7fc32638c700 1 -- 172.21.15.90:0/4244235965 wait complete. 2024-04-14T15:16:48.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:48.949+0000 7fc32638c700 1 librados: starting msgr at 2024-04-14T15:16:48.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:48.949+0000 7fc32638c700 1 librados: starting objecter 2024-04-14T15:16:48.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:48.949+0000 7fc32638c700 1 Processor -- start 2024-04-14T15:16:48.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:48.949+0000 7fc32638c700 1 -- start start 2024-04-14T15:16:48.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:48.949+0000 7fc32638c700 1 librados: setting wanted keys 2024-04-14T15:16:48.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:48.949+0000 7fc32638c700 1 librados: calling monclient init 2024-04-14T15:16:48.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:48.950+0000 7fc32638c700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc32010df10 con 0x7fc32010e1e0 2024-04-14T15:16:48.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:48.950+0000 7fc32638c700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc3201a7df0 con 0x7fc320107760 2024-04-14T15:16:48.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:48.950+0000 7fc32638c700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc3201a8fd0 con 0x7fc32010a610 2024-04-14T15:16:48.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:48.950+0000 7fc324929700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fc32010e1e0 0x7fc3201a56c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:47908/0 (socket says 172.21.15.90:47908) 2024-04-14T15:16:48.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:48.950+0000 7fc324929700 1 -- 172.21.15.90:0/3829288646 learned_addr learned my addr 172.21.15.90:0/3829288646 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:16:48.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:48.950+0000 7fc31d7fa700 10 client.?.objecter ms_handle_connect 0x7fc32010a610 2024-04-14T15:16:48.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:48.950+0000 7fc31d7fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:16:48.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:48.950+0000 7fc31d7fa700 10 client.?.objecter ms_handle_connect 0x7fc32010e1e0 2024-04-14T15:16:48.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:48.950+0000 7fc31d7fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:16:48.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:48.950+0000 7fc31d7fa700 10 client.?.objecter ms_handle_connect 0x7fc320107760 2024-04-14T15:16:48.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:48.950+0000 7fc31d7fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:16:48.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:48.950+0000 7fc31d7fa700 1 -- 172.21.15.90:0/3829288646 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 261502539 0 0) 0x7fc3201a7df0 con 0x7fc320107760 2024-04-14T15:16:48.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:48.950+0000 7fc31d7fa700 1 -- 172.21.15.90:0/3829288646 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3080035b0 con 0x7fc320107760 2024-04-14T15:16:48.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:48.950+0000 7fc31d7fa700 1 -- 172.21.15.90:0/3829288646 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 381316073 0 0) 0x7fc3201a8fd0 con 0x7fc32010a610 2024-04-14T15:16:48.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:48.950+0000 7fc31d7fa700 1 -- 172.21.15.90:0/3829288646 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3201a7df0 con 0x7fc32010a610 2024-04-14T15:16:48.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:48.950+0000 7fc31d7fa700 1 -- 172.21.15.90:0/3829288646 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1838979159 0 0) 0x7fc32010df10 con 0x7fc32010e1e0 2024-04-14T15:16:48.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:48.950+0000 7fc31d7fa700 1 -- 172.21.15.90:0/3829288646 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3201a8fd0 con 0x7fc32010e1e0 2024-04-14T15:16:48.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:48.950+0000 7fc31d7fa700 1 -- 172.21.15.90:0/3829288646 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4057561571 0 0) 0x7fc3080035b0 con 0x7fc320107760 2024-04-14T15:16:48.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:48.950+0000 7fc31d7fa700 1 -- 172.21.15.90:0/3829288646 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc32010df10 con 0x7fc320107760 2024-04-14T15:16:48.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:48.950+0000 7fc31d7fa700 1 -- 172.21.15.90:0/3829288646 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fc310001830 con 0x7fc320107760 2024-04-14T15:16:48.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:48.950+0000 7fc31d7fa700 1 -- 172.21.15.90:0/3829288646 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2936395840 0 0) 0x7fc3201a7df0 con 0x7fc32010a610 2024-04-14T15:16:48.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:48.950+0000 7fc31d7fa700 1 -- 172.21.15.90:0/3829288646 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc3080035b0 con 0x7fc32010a610 2024-04-14T15:16:48.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:48.950+0000 7fc31d7fa700 1 -- 172.21.15.90:0/3829288646 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 804781948 0 0) 0x7fc3201a8fd0 con 0x7fc32010e1e0 2024-04-14T15:16:48.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:48.950+0000 7fc31d7fa700 1 -- 172.21.15.90:0/3829288646 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc3201a7df0 con 0x7fc32010e1e0 2024-04-14T15:16:48.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:48.950+0000 7fc31d7fa700 1 -- 172.21.15.90:0/3829288646 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fc3140018b0 con 0x7fc32010a610 2024-04-14T15:16:48.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:48.950+0000 7fc31d7fa700 1 -- 172.21.15.90:0/3829288646 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fc318001bf0 con 0x7fc32010e1e0 2024-04-14T15:16:48.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:48.950+0000 7fc31d7fa700 1 -- 172.21.15.90:0/3829288646 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2416011492 0 0) 0x7fc32010df10 con 0x7fc320107760 2024-04-14T15:16:48.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:48.951+0000 7fc31d7fa700 1 -- 172.21.15.90:0/3829288646 >> v1:172.21.15.90:6790/0 conn(0x7fc32010a610 legacy=0x7fc32010b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:48.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:48.951+0000 7fc31d7fa700 1 -- 172.21.15.90:0/3829288646 >> v1:172.21.15.90:6789/0 conn(0x7fc32010e1e0 legacy=0x7fc3201a56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:48.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:48.951+0000 7fc31d7fa700 1 -- 172.21.15.90:0/3829288646 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3201aa1b0 con 0x7fc320107760 2024-04-14T15:16:48.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:48.951+0000 7fc32638c700 1 -- 172.21.15.90:0/3829288646 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc3201a8020 con 0x7fc320107760 2024-04-14T15:16:48.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:48.951+0000 7fc31d7fa700 1 -- 172.21.15.90:0/3829288646 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fc310004740 con 0x7fc320107760 2024-04-14T15:16:48.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:48.951+0000 7fc31d7fa700 1 -- 172.21.15.90:0/3829288646 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fc310004ea0 con 0x7fc320107760 2024-04-14T15:16:48.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:48.951+0000 7fc32638c700 10 client.48919.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:16:48.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:48.951+0000 7fc32638c700 1 -- 172.21.15.90:0/3829288646 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc3201a84e0 con 0x7fc320107760 2024-04-14T15:16:48.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:48.951+0000 7fc32638c700 1 librados: init done 2024-04-14T15:16:48.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:48.952+0000 7fc32638c700 1 -- 172.21.15.90:0/3829288646 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc320102ea0 con 0x7fc320107760 2024-04-14T15:16:48.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:48.952+0000 7fc31d7fa700 1 -- 172.21.15.90:0/3829288646 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fc310006230 con 0x7fc320107760 2024-04-14T15:16:48.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:48.953+0000 7fc31d7fa700 1 -- 172.21.15.90:0/3829288646 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(341..341 src has 51..341) ==== 9025+0+0 (unknown 1637225801 0 0) 0x7fc310089cb0 con 0x7fc320107760 2024-04-14T15:16:48.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:48.953+0000 7fc31d7fa700 10 client.48919.objecter ms_dispatch 0x7fc320001550 osd_map(341..341 src has 51..341) 2024-04-14T15:16:48.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:48.953+0000 7fc31d7fa700 3 client.48919.objecter handle_osd_map got epochs [341,341] > 0 2024-04-14T15:16:48.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:48.953+0000 7fc31d7fa700 3 client.48919.objecter handle_osd_map decoding full epoch 341 2024-04-14T15:16:48.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:48.953+0000 7fc31d7fa700 20 client.48919.objecter dump_active .. 0 homeless 2024-04-14T15:16:48.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:48.953+0000 7fc31d7fa700 10 client.48919.objecter ms_handle_connect 0x7fc3080737a0 2024-04-14T15:16:48.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:48.958+0000 7fc31d7fa700 1 -- 172.21.15.90:0/3829288646 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fc310004ea0 con 0x7fc320107760 2024-04-14T15:16:48.975 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:48.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:48.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:16:49.062 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:48 smithi090 ceph-mon[30399]: pgmap v455: 238 pgs: 238 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:16:49.062 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:48 smithi090 ceph-mon[30399]: 6.14 scrub starts 2024-04-14T15:16:49.062 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:48 smithi090 ceph-mon[30399]: 6.14 scrub ok 2024-04-14T15:16:49.062 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:48 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:16:49.062 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:48 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:16:49.062 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:48 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:16:49.062 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:48 smithi090 ceph-mon[30399]: osdmap e341: 8 total, 8 up, 8 in 2024-04-14T15:16:49.063 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:48 smithi090 ceph-mon[41670]: pgmap v455: 238 pgs: 238 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:16:49.063 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:48 smithi090 ceph-mon[41670]: 6.14 scrub starts 2024-04-14T15:16:49.063 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:48 smithi090 ceph-mon[41670]: 6.14 scrub ok 2024-04-14T15:16:49.063 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:48 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:16:49.063 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:48 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:16:49.063 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:48 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:16:49.063 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:48 smithi090 ceph-mon[41670]: osdmap e341: 8 total, 8 up, 8 in 2024-04-14T15:16:49.077 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:49.102 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:49.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:49.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:49.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:49.175+0000 7fc32638c700 1 -- 172.21.15.90:0/3829288646 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7fc3200693d0 con 0x7fc320107760 2024-04-14T15:16:49.205 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:49.210 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:16:49.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:49.231 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:49.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:49.246 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-14T15:16:49.332 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:16:49.355 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:49.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:49.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:16:49.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:48 smithi092 ceph-mon[33966]: pgmap v455: 238 pgs: 238 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:16:49.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:48 smithi092 ceph-mon[33966]: 6.14 scrub starts 2024-04-14T15:16:49.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:48 smithi092 ceph-mon[33966]: 6.14 scrub ok 2024-04-14T15:16:49.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:48 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:16:49.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:48 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:16:49.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:48 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:16:49.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:48 smithi092 ceph-mon[33966]: osdmap e341: 8 total, 8 up, 8 in 2024-04-14T15:16:49.457 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:49.483 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:49.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:49.484 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:49.496 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-14T15:16:49.496 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:16:49.585 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:49.611 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:49.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:49.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:49.713 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:16:49.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:49.743 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:49.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:16:49.844 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:16:49.868 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:49.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:49.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:16:49.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:49.880+0000 7fc31d7fa700 1 -- 172.21.15.90:0/3829288646 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v342) ==== 101+0+0 (unknown 1557533545 0 0) 0x7fc310056360 con 0x7fc320107760 2024-04-14T15:16:49.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:49.884+0000 7fc32638c700 10 librados: watch_flush enter 2024-04-14T15:16:49.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:49.884+0000 7fc32638c700 10 librados: watch_flush exit 2024-04-14T15:16:49.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:49.884+0000 7fc32638c700 20 client.48919.objecter shutdown clearing up homeless session... 2024-04-14T15:16:49.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:49.884+0000 7fc32638c700 10 client.48919.objecter successfully canceled tick 2024-04-14T15:16:49.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:49.885+0000 7fc32638c700 1 -- 172.21.15.90:0/3829288646 >> v1:172.21.15.90:6800/420935427 conn(0x7fc3080737a0 legacy=0x7fc308075c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:49.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:49.885+0000 7fc32638c700 1 -- 172.21.15.90:0/3829288646 >> v1:172.21.15.92:6789/0 conn(0x7fc320107760 legacy=0x7fc32010ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:49.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:49.886+0000 7fc32638c700 1 -- 172.21.15.90:0/3829288646 shutdown_connections 2024-04-14T15:16:49.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:49.886+0000 7fc32638c700 1 -- 172.21.15.90:0/3829288646 >> 172.21.15.90:0/3829288646 conn(0x7fc3200fd290 msgr2=0x7fc320111760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:16:49.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:49.887+0000 7fc32638c700 1 -- 172.21.15.90:0/3829288646 shutdown_connections 2024-04-14T15:16:49.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:49.887+0000 7fc32638c700 1 -- 172.21.15.90:0/3829288646 wait complete. 2024-04-14T15:16:49.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:49.887+0000 7fc32638c700 1 librados: shutdown 2024-04-14T15:16:49.888 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is unset 2024-04-14T15:16:49.971 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:16:49.998 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:49.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:49.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:16:50.100 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:16:50.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:50.125 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:50.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:16:50.189 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:49 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:16:50.189 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:49 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:16:50.189 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:49 smithi090 ceph-mon[41670]: pgmap v458: 246 pgs: 4 creating+activating, 2 unknown, 240 active+clean; 8.4 MiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 2 op/s 2024-04-14T15:16:50.189 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:49 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3829288646' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:16:50.189 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:49 smithi090 ceph-mon[41670]: from='client.48919 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:16:50.189 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:49 smithi090 ceph-mon[41670]: 4.c scrub starts 2024-04-14T15:16:50.190 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:49 smithi090 ceph-mon[41670]: 4.c scrub ok 2024-04-14T15:16:50.190 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:49 smithi090 ceph-mon[41670]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-14T15:16:50.190 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:49 smithi090 ceph-mon[41670]: from='client.48919 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-14T15:16:50.190 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:49 smithi090 ceph-mon[41670]: osdmap e342: 8 total, 8 up, 8 in 2024-04-14T15:16:50.190 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:49 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1101952089' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi090-93363-58"}]: dispatch 2024-04-14T15:16:50.190 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:49 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:16:50.190 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:49 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:16:50.190 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:49 smithi090 ceph-mon[30399]: pgmap v458: 246 pgs: 4 creating+activating, 2 unknown, 240 active+clean; 8.4 MiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 2 op/s 2024-04-14T15:16:50.190 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:49 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3829288646' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:16:50.190 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:49 smithi090 ceph-mon[30399]: from='client.48919 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:16:50.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:49 smithi090 ceph-mon[30399]: 4.c scrub starts 2024-04-14T15:16:50.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:49 smithi090 ceph-mon[30399]: 4.c scrub ok 2024-04-14T15:16:50.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:49 smithi090 ceph-mon[30399]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-14T15:16:50.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:49 smithi090 ceph-mon[30399]: from='client.48919 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-14T15:16:50.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:49 smithi090 ceph-mon[30399]: osdmap e342: 8 total, 8 up, 8 in 2024-04-14T15:16:50.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:49 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1101952089' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi090-93363-58"}]: dispatch 2024-04-14T15:16:50.227 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:16:50.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:50.252 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:50.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:16:50.353 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:16:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:49 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:16:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:49 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:16:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:49 smithi092 ceph-mon[33966]: pgmap v458: 246 pgs: 4 creating+activating, 2 unknown, 240 active+clean; 8.4 MiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 2 op/s 2024-04-14T15:16:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:49 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3829288646' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:16:50.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:49 smithi092 ceph-mon[33966]: from='client.48919 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:16:50.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:49 smithi092 ceph-mon[33966]: 4.c scrub starts 2024-04-14T15:16:50.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:49 smithi092 ceph-mon[33966]: 4.c scrub ok 2024-04-14T15:16:50.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:49 smithi092 ceph-mon[33966]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-14T15:16:50.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:49 smithi092 ceph-mon[33966]: from='client.48919 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-14T15:16:50.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:49 smithi092 ceph-mon[33966]: osdmap e342: 8 total, 8 up, 8 in 2024-04-14T15:16:50.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:49 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1101952089' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi090-93363-58"}]: dispatch 2024-04-14T15:16:50.381 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:50.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:50.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:16:50.483 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:16:50.510 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:50.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:50.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:16:50.612 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:16:50.636 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:50.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:50.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:16:50.738 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:16:50.761 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:50.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:50.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:16:50.846 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-14T15:16:50.864 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:16:50.894 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:50.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:50.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:16:50.996 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:16:51.021 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:51.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:51.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:16:51.122 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:16:51.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:51.147 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:51.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:16:51.158 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:50 smithi090 ceph-mon[41670]: 110.b scrub starts 2024-04-14T15:16:51.158 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:50 smithi090 ceph-mon[41670]: 110.b scrub ok 2024-04-14T15:16:51.158 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:50 smithi090 ceph-mon[41670]: 4.1e scrub starts 2024-04-14T15:16:51.158 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:50 smithi090 ceph-mon[41670]: 4.1e scrub ok 2024-04-14T15:16:51.158 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:50 smithi090 ceph-mon[41670]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:16:51.158 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:50 smithi090 ceph-mon[41670]: 6.11 scrub starts 2024-04-14T15:16:51.158 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:50 smithi090 ceph-mon[41670]: 6.11 scrub ok 2024-04-14T15:16:51.159 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:50 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:16:51.159 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:50 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:16:51.159 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:50 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1101952089' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi090-93363-58"}]': finished 2024-04-14T15:16:51.159 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:50 smithi090 ceph-mon[41670]: osdmap e343: 8 total, 8 up, 8 in 2024-04-14T15:16:51.159 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:50 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1101952089' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi090-93363-58"}]: dispatch 2024-04-14T15:16:51.159 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:50 smithi090 ceph-mon[30399]: 110.b scrub starts 2024-04-14T15:16:51.159 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:50 smithi090 ceph-mon[30399]: 110.b scrub ok 2024-04-14T15:16:51.159 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:50 smithi090 ceph-mon[30399]: 4.1e scrub starts 2024-04-14T15:16:51.159 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:50 smithi090 ceph-mon[30399]: 4.1e scrub ok 2024-04-14T15:16:51.159 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:50 smithi090 ceph-mon[30399]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:16:51.159 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:50 smithi090 ceph-mon[30399]: 6.11 scrub starts 2024-04-14T15:16:51.160 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:50 smithi090 ceph-mon[30399]: 6.11 scrub ok 2024-04-14T15:16:51.160 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:50 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:16:51.160 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:50 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:16:51.160 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:50 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1101952089' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi090-93363-58"}]': finished 2024-04-14T15:16:51.160 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:50 smithi090 ceph-mon[30399]: osdmap e343: 8 total, 8 up, 8 in 2024-04-14T15:16:51.160 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:50 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1101952089' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi090-93363-58"}]: dispatch 2024-04-14T15:16:51.249 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:51.277 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:51.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:51.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:51.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:50 smithi092 ceph-mon[33966]: 110.b scrub starts 2024-04-14T15:16:51.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:50 smithi092 ceph-mon[33966]: 110.b scrub ok 2024-04-14T15:16:51.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:50 smithi092 ceph-mon[33966]: 4.1e scrub starts 2024-04-14T15:16:51.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:50 smithi092 ceph-mon[33966]: 4.1e scrub ok 2024-04-14T15:16:51.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:50 smithi092 ceph-mon[33966]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:16:51.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:50 smithi092 ceph-mon[33966]: 6.11 scrub starts 2024-04-14T15:16:51.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:50 smithi092 ceph-mon[33966]: 6.11 scrub ok 2024-04-14T15:16:51.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:50 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:16:51.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:50 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:16:51.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:50 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1101952089' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi090-93363-58"}]': finished 2024-04-14T15:16:51.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:50 smithi092 ceph-mon[33966]: osdmap e343: 8 total, 8 up, 8 in 2024-04-14T15:16:51.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:50 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1101952089' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi090-93363-58"}]: dispatch 2024-04-14T15:16:51.379 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:16:51.407 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:51.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:51.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:16:51.509 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:16:51.518 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:16:51.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:51.540 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:51.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:16:51.641 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:16:51.668 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:51.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:51.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:16:51.770 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:51.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:51.797 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:51.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:51.898 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:16:51.923 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:51.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:51.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:16:52.024 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:16:52.050 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:52.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:52.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:16:52.151 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:16:52.165 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:51 smithi090 ceph-mon[41670]: 5.10 scrub starts 2024-04-14T15:16:52.165 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:51 smithi090 ceph-mon[41670]: 5.10 scrub ok 2024-04-14T15:16:52.165 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:51 smithi090 ceph-mon[41670]: pgmap v461: 238 pgs: 238 active+clean; 8.4 MiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-04-14T15:16:52.165 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:51 smithi090 ceph-mon[41670]: 6.d scrub starts 2024-04-14T15:16:52.165 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:51 smithi090 ceph-mon[41670]: 6.d scrub ok 2024-04-14T15:16:52.165 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:51 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1101952089' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi090-93363-58"}]': finished 2024-04-14T15:16:52.165 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:51 smithi090 ceph-mon[41670]: osdmap e344: 8 total, 8 up, 8 in 2024-04-14T15:16:52.165 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:51 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2371781823' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi090-93363-59"}]: dispatch 2024-04-14T15:16:52.165 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:51 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2371781823' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi090-93363-59"}]: dispatch 2024-04-14T15:16:52.165 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:51 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2371781823' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi090-93363-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:16:52.166 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:51 smithi090 ceph-mon[30399]: 5.10 scrub starts 2024-04-14T15:16:52.166 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:51 smithi090 ceph-mon[30399]: 5.10 scrub ok 2024-04-14T15:16:52.166 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:51 smithi090 ceph-mon[30399]: pgmap v461: 238 pgs: 238 active+clean; 8.4 MiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-04-14T15:16:52.166 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:51 smithi090 ceph-mon[30399]: 6.d scrub starts 2024-04-14T15:16:52.166 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:51 smithi090 ceph-mon[30399]: 6.d scrub ok 2024-04-14T15:16:52.166 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:51 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1101952089' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi090-93363-58"}]': finished 2024-04-14T15:16:52.166 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:51 smithi090 ceph-mon[30399]: osdmap e344: 8 total, 8 up, 8 in 2024-04-14T15:16:52.166 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:51 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2371781823' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi090-93363-59"}]: dispatch 2024-04-14T15:16:52.166 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:51 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2371781823' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi090-93363-59"}]: dispatch 2024-04-14T15:16:52.166 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:51 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2371781823' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi090-93363-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:16:52.176 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:52.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:52.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:16:52.273 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:51 smithi092 ceph-mon[33966]: 5.10 scrub starts 2024-04-14T15:16:52.274 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:51 smithi092 ceph-mon[33966]: 5.10 scrub ok 2024-04-14T15:16:52.274 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:51 smithi092 ceph-mon[33966]: pgmap v461: 238 pgs: 238 active+clean; 8.4 MiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-04-14T15:16:52.274 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:51 smithi092 ceph-mon[33966]: 6.d scrub starts 2024-04-14T15:16:52.274 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:51 smithi092 ceph-mon[33966]: 6.d scrub ok 2024-04-14T15:16:52.274 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:51 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1101952089' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi090-93363-58"}]': finished 2024-04-14T15:16:52.274 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:51 smithi092 ceph-mon[33966]: osdmap e344: 8 total, 8 up, 8 in 2024-04-14T15:16:52.274 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:51 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2371781823' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi090-93363-59"}]: dispatch 2024-04-14T15:16:52.274 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:51 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2371781823' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi090-93363-59"}]: dispatch 2024-04-14T15:16:52.274 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:51 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2371781823' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi090-93363-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:16:52.279 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:16:52.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:52.304 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:52.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:16:52.405 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:52.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:52.430 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:52.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:52.506 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:16:52.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.525+0000 7fd8e3a09700 1 -- 172.21.15.90:0/2406877338 >> v1:172.21.15.92:6789/0 conn(0x7fd8dc10a740 legacy=0x7fd8dc10cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:52.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.525+0000 7fd8e3a09700 1 -- 172.21.15.90:0/2406877338 shutdown_connections 2024-04-14T15:16:52.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.525+0000 7fd8e3a09700 1 -- 172.21.15.90:0/2406877338 >> 172.21.15.90:0/2406877338 conn(0x7fd8dc0fd290 msgr2=0x7fd8dc0ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:16:52.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.525+0000 7fd8e3a09700 1 -- 172.21.15.90:0/2406877338 shutdown_connections 2024-04-14T15:16:52.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.525+0000 7fd8e3a09700 1 -- 172.21.15.90:0/2406877338 wait complete. 2024-04-14T15:16:52.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.526+0000 7fd8e3a09700 1 librados: starting msgr at 2024-04-14T15:16:52.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.526+0000 7fd8e3a09700 1 librados: starting objecter 2024-04-14T15:16:52.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.526+0000 7fd8e3a09700 1 Processor -- start 2024-04-14T15:16:52.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.526+0000 7fd8e3a09700 1 -- start start 2024-04-14T15:16:52.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.526+0000 7fd8e3a09700 1 librados: setting wanted keys 2024-04-14T15:16:52.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.526+0000 7fd8e3a09700 1 librados: calling monclient init 2024-04-14T15:16:52.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.526+0000 7fd8e3a09700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd8dc1a9b90 con 0x7fd8dc107890 2024-04-14T15:16:52.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.526+0000 7fd8e3a09700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd8dc1aadb0 con 0x7fd8dc10a740 2024-04-14T15:16:52.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.526+0000 7fd8e3a09700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd8dc1abff0 con 0x7fd8dc10e310 2024-04-14T15:16:52.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.527+0000 7fd8e17a5700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fd8dc107890 0x7fd8dc10d920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:33700/0 (socket says 172.21.15.90:33700) 2024-04-14T15:16:52.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.527+0000 7fd8e17a5700 1 -- 172.21.15.90:0/4047876510 learned_addr learned my addr 172.21.15.90:0/4047876510 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:16:52.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.527+0000 7fd8d27fc700 10 client.?.objecter ms_handle_connect 0x7fd8dc107890 2024-04-14T15:16:52.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.527+0000 7fd8d27fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:16:52.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.527+0000 7fd8d27fc700 10 client.?.objecter ms_handle_connect 0x7fd8dc10a740 2024-04-14T15:16:52.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.527+0000 7fd8d27fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:16:52.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.527+0000 7fd8d27fc700 10 client.?.objecter ms_handle_connect 0x7fd8dc10e310 2024-04-14T15:16:52.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.527+0000 7fd8d27fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:16:52.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.527+0000 7fd8d27fc700 1 -- 172.21.15.90:0/4047876510 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1943776847 0 0) 0x7fd8dc1a9b90 con 0x7fd8dc107890 2024-04-14T15:16:52.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.527+0000 7fd8d27fc700 1 -- 172.21.15.90:0/4047876510 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd8c8003aa0 con 0x7fd8dc107890 2024-04-14T15:16:52.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.527+0000 7fd8d27fc700 1 -- 172.21.15.90:0/4047876510 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2330067859 0 0) 0x7fd8dc1aadb0 con 0x7fd8dc10a740 2024-04-14T15:16:52.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.527+0000 7fd8d27fc700 1 -- 172.21.15.90:0/4047876510 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd8dc1a9b90 con 0x7fd8dc10a740 2024-04-14T15:16:52.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.527+0000 7fd8d27fc700 1 -- 172.21.15.90:0/4047876510 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 332984476 0 0) 0x7fd8dc1abff0 con 0x7fd8dc10e310 2024-04-14T15:16:52.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.527+0000 7fd8d27fc700 1 -- 172.21.15.90:0/4047876510 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd8dc1aadb0 con 0x7fd8dc10e310 2024-04-14T15:16:52.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.527+0000 7fd8d27fc700 1 -- 172.21.15.90:0/4047876510 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3739558130 0 0) 0x7fd8c8003aa0 con 0x7fd8dc107890 2024-04-14T15:16:52.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.527+0000 7fd8d27fc700 1 -- 172.21.15.90:0/4047876510 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd8dc1abff0 con 0x7fd8dc107890 2024-04-14T15:16:52.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.527+0000 7fd8d27fc700 1 -- 172.21.15.90:0/4047876510 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 940544057 0 0) 0x7fd8dc1a9b90 con 0x7fd8dc10a740 2024-04-14T15:16:52.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.527+0000 7fd8d27fc700 1 -- 172.21.15.90:0/4047876510 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd8c8003aa0 con 0x7fd8dc10a740 2024-04-14T15:16:52.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.527+0000 7fd8d27fc700 1 -- 172.21.15.90:0/4047876510 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 156593049 0 0) 0x7fd8dc1aadb0 con 0x7fd8dc10e310 2024-04-14T15:16:52.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.527+0000 7fd8d27fc700 1 -- 172.21.15.90:0/4047876510 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd8dc1a9b90 con 0x7fd8dc10e310 2024-04-14T15:16:52.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.527+0000 7fd8d27fc700 1 -- 172.21.15.90:0/4047876510 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fd8d80019a0 con 0x7fd8dc107890 2024-04-14T15:16:52.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.527+0000 7fd8d27fc700 1 -- 172.21.15.90:0/4047876510 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fd8cc0042d0 con 0x7fd8dc10a740 2024-04-14T15:16:52.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.527+0000 7fd8d27fc700 1 -- 172.21.15.90:0/4047876510 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fd8d4001bf0 con 0x7fd8dc10e310 2024-04-14T15:16:52.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.527+0000 7fd8d27fc700 1 -- 172.21.15.90:0/4047876510 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1367107581 0 0) 0x7fd8c8003aa0 con 0x7fd8dc10a740 2024-04-14T15:16:52.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.527+0000 7fd8d27fc700 1 -- 172.21.15.90:0/4047876510 >> v1:172.21.15.90:6790/0 conn(0x7fd8dc10e310 legacy=0x7fd8dc1a82f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:52.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.527+0000 7fd8d27fc700 1 -- 172.21.15.90:0/4047876510 >> v1:172.21.15.90:6789/0 conn(0x7fd8dc107890 legacy=0x7fd8dc10d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:52.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.528+0000 7fd8d27fc700 1 -- 172.21.15.90:0/4047876510 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8dc1ad230 con 0x7fd8dc10a740 2024-04-14T15:16:52.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.528+0000 7fd8e3a09700 1 -- 172.21.15.90:0/4047876510 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd8dc1aaf80 con 0x7fd8dc10a740 2024-04-14T15:16:52.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.528+0000 7fd8e3a09700 10 client.48931.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:16:52.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.528+0000 7fd8e3a09700 1 -- 172.21.15.90:0/4047876510 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd8dc1ab440 con 0x7fd8dc10a740 2024-04-14T15:16:52.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.528+0000 7fd8e3a09700 1 librados: init done 2024-04-14T15:16:52.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.528+0000 7fd8d27fc700 1 -- 172.21.15.90:0/4047876510 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fd8cc001ab0 con 0x7fd8dc10a740 2024-04-14T15:16:52.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.528+0000 7fd8d27fc700 1 -- 172.21.15.90:0/4047876510 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fd8cc004a10 con 0x7fd8dc10a740 2024-04-14T15:16:52.533 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:16:52.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.529+0000 7fd8e3a09700 1 -- 172.21.15.90:0/4047876510 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8c0005150 con 0x7fd8dc10a740 2024-04-14T15:16:52.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.530+0000 7fd8d27fc700 1 -- 172.21.15.90:0/4047876510 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fd8cc01b6f0 con 0x7fd8dc10a740 2024-04-14T15:16:52.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.530+0000 7fd8d27fc700 1 -- 172.21.15.90:0/4047876510 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(344..344 src has 51..344) ==== 8178+0+0 (unknown 3748196444 0 0) 0x7fd8cc089740 con 0x7fd8dc10a740 2024-04-14T15:16:52.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.530+0000 7fd8d27fc700 10 client.48931.objecter ms_dispatch 0x7fd8dc001550 osd_map(344..344 src has 51..344) 2024-04-14T15:16:52.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.530+0000 7fd8d27fc700 3 client.48931.objecter handle_osd_map got epochs [344,344] > 0 2024-04-14T15:16:52.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.530+0000 7fd8d27fc700 3 client.48931.objecter handle_osd_map decoding full epoch 344 2024-04-14T15:16:52.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.530+0000 7fd8d27fc700 20 client.48931.objecter dump_active .. 0 homeless 2024-04-14T15:16:52.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.530+0000 7fd8d27fc700 10 client.48931.objecter ms_handle_connect 0x7fd8c8071420 2024-04-14T15:16:52.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.536+0000 7fd8d27fc700 1 -- 172.21.15.90:0/4047876510 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fd8cc056140 con 0x7fd8dc10a740 2024-04-14T15:16:52.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:52.558 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:52.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:16:52.659 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:16:52.684 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:52.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:52.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:16:52.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.756+0000 7fd8e3a09700 1 -- 172.21.15.90:0/4047876510 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd reweight", "id": 0, "weight": 1.0} v 0) -- 0x7fd8c0005440 con 0x7fd8dc10a740 2024-04-14T15:16:52.785 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:52.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:52.811 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:52.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:52.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.882+0000 7fd8d27fc700 1 -- 172.21.15.90:0/4047876510 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 0, "weight": 1.0}]=0 reweighted osd.0 to 1 (10000) v345) ==== 113+0+0 (unknown 67727966 0 0) 0x7fd8cc059df0 con 0x7fd8dc10a740 2024-04-14T15:16:52.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.887+0000 7fd8e3a09700 10 librados: watch_flush enter 2024-04-14T15:16:52.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.887+0000 7fd8e3a09700 10 librados: watch_flush exit 2024-04-14T15:16:52.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.887+0000 7fd8e3a09700 20 client.48931.objecter shutdown clearing up homeless session... 2024-04-14T15:16:52.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.887+0000 7fd8e3a09700 10 client.48931.objecter successfully canceled tick 2024-04-14T15:16:52.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.887+0000 7fd8e3a09700 1 -- 172.21.15.90:0/4047876510 >> v1:172.21.15.90:6800/420935427 conn(0x7fd8c8071420 legacy=0x7fd8c80738e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:52.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.887+0000 7fd8e3a09700 1 -- 172.21.15.90:0/4047876510 >> v1:172.21.15.92:6789/0 conn(0x7fd8dc10a740 legacy=0x7fd8dc1a4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:52.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.887+0000 7fd8e3a09700 1 -- 172.21.15.90:0/4047876510 shutdown_connections 2024-04-14T15:16:52.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.887+0000 7fd8e3a09700 1 -- 172.21.15.90:0/4047876510 >> 172.21.15.90:0/4047876510 conn(0x7fd8dc0fd290 msgr2=0x7fd8dc110fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:16:52.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.887+0000 7fd8e3a09700 1 -- 172.21.15.90:0/4047876510 shutdown_connections 2024-04-14T15:16:52.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.887+0000 7fd8e3a09700 1 -- 172.21.15.90:0/4047876510 wait complete. 2024-04-14T15:16:52.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:52.887+0000 7fd8e3a09700 1 librados: shutdown 2024-04-14T15:16:52.889 INFO:teuthology.orchestra.run.smithi090.stderr:reweighted osd.0 to 1 (10000) 2024-04-14T15:16:52.911 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:16:52.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:52.936 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:52.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:16:53.037 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:16:53.062 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:52 smithi090 ceph-mon[41670]: 5.4 scrub starts 2024-04-14T15:16:53.062 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:52 smithi090 ceph-mon[41670]: 5.4 scrub ok 2024-04-14T15:16:53.062 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:52 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/4047876510' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-14T15:16:53.062 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:52 smithi090 ceph-mon[41670]: from='client.48931 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-14T15:16:53.063 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:52 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2371781823' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi090-93363-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:16:53.063 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:52 smithi090 ceph-mon[41670]: from='client.48931 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-14T15:16:53.063 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:52 smithi090 ceph-mon[41670]: osdmap e345: 8 total, 8 up, 8 in 2024-04-14T15:16:53.063 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:52 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2371781823' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi090-93363-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi090-93363-59"}]: dispatch 2024-04-14T15:16:53.064 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:52 smithi090 ceph-mon[30399]: 5.4 scrub starts 2024-04-14T15:16:53.064 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:52 smithi090 ceph-mon[30399]: 5.4 scrub ok 2024-04-14T15:16:53.064 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:52 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/4047876510' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-14T15:16:53.064 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:52 smithi090 ceph-mon[30399]: from='client.48931 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-14T15:16:53.064 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:52 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2371781823' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi090-93363-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:16:53.065 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:52 smithi090 ceph-mon[30399]: from='client.48931 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-14T15:16:53.065 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:52 smithi090 ceph-mon[30399]: osdmap e345: 8 total, 8 up, 8 in 2024-04-14T15:16:53.065 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:52 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2371781823' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi090-93363-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi090-93363-59"}]: dispatch 2024-04-14T15:16:53.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:53.067 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:53.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:16:53.168 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:53.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:53.193 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:53.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:53.294 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:16:53.318 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:53.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:53.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:16:53.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:52 smithi092 ceph-mon[33966]: 5.4 scrub starts 2024-04-14T15:16:53.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:52 smithi092 ceph-mon[33966]: 5.4 scrub ok 2024-04-14T15:16:53.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:52 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/4047876510' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-14T15:16:53.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:52 smithi092 ceph-mon[33966]: from='client.48931 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-14T15:16:53.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:52 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2371781823' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi090-93363-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:16:53.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:52 smithi092 ceph-mon[33966]: from='client.48931 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-14T15:16:53.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:52 smithi092 ceph-mon[33966]: osdmap e345: 8 total, 8 up, 8 in 2024-04-14T15:16:53.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:52 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2371781823' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi090-93363-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi090-93363-59"}]: dispatch 2024-04-14T15:16:53.421 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:16:53.445 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:53.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:53.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:16:53.547 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:53.571 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:53.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:53.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:53.672 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:16:53.696 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:53.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:53.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:16:53.798 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:16:53.822 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:53.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:53.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:16:53.919 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd set noscrub 2024-04-14T15:16:53.924 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:53.949 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:53.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:53.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:54.051 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:16:54.074 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:54.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:54.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:16:54.176 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:16:54.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:54.201 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:54.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:16:54.302 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:16:54.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:54.326 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:54.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:16:54.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:53 smithi092 ceph-mon[33966]: 4.12 deep-scrub starts 2024-04-14T15:16:54.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:53 smithi092 ceph-mon[33966]: 4.12 deep-scrub ok 2024-04-14T15:16:54.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:53 smithi092 ceph-mon[33966]: 5.1e scrub starts 2024-04-14T15:16:54.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:53 smithi092 ceph-mon[33966]: 5.1e scrub ok 2024-04-14T15:16:54.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:53 smithi092 ceph-mon[33966]: pgmap v464: 238 pgs: 238 active+clean; 8.4 MiB data, 606 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:16:54.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:53 smithi092 ceph-mon[33966]: 4.e scrub starts 2024-04-14T15:16:54.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:53 smithi092 ceph-mon[33966]: 4.e scrub ok 2024-04-14T15:16:54.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:53 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:16:54.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:53 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:16:54.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:53 smithi092 ceph-mon[33966]: osdmap e346: 8 total, 8 up, 8 in 2024-04-14T15:16:54.427 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:16:54.441 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:16:53 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:16:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:16:54.520 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:53 smithi090 ceph-mon[41670]: 4.12 deep-scrub starts 2024-04-14T15:16:54.520 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:53 smithi090 ceph-mon[41670]: 4.12 deep-scrub ok 2024-04-14T15:16:54.520 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:53 smithi090 ceph-mon[41670]: 5.1e scrub starts 2024-04-14T15:16:54.520 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:53 smithi090 ceph-mon[41670]: 5.1e scrub ok 2024-04-14T15:16:54.520 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:53 smithi090 ceph-mon[41670]: pgmap v464: 238 pgs: 238 active+clean; 8.4 MiB data, 606 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:16:54.520 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:53 smithi090 ceph-mon[41670]: 4.e scrub starts 2024-04-14T15:16:54.520 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:53 smithi090 ceph-mon[41670]: 4.e scrub ok 2024-04-14T15:16:54.520 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:53 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:16:54.520 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:53 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:16:54.520 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:53 smithi090 ceph-mon[41670]: osdmap e346: 8 total, 8 up, 8 in 2024-04-14T15:16:54.521 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:53 smithi090 ceph-mon[30399]: 4.12 deep-scrub starts 2024-04-14T15:16:54.521 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:53 smithi090 ceph-mon[30399]: 4.12 deep-scrub ok 2024-04-14T15:16:54.521 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:53 smithi090 ceph-mon[30399]: 5.1e scrub starts 2024-04-14T15:16:54.521 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:53 smithi090 ceph-mon[30399]: 5.1e scrub ok 2024-04-14T15:16:54.521 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:53 smithi090 ceph-mon[30399]: pgmap v464: 238 pgs: 238 active+clean; 8.4 MiB data, 606 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:16:54.521 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:53 smithi090 ceph-mon[30399]: 4.e scrub starts 2024-04-14T15:16:54.521 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:53 smithi090 ceph-mon[30399]: 4.e scrub ok 2024-04-14T15:16:54.521 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:53 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:16:54.521 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:53 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:16:54.521 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:53 smithi090 ceph-mon[30399]: osdmap e346: 8 total, 8 up, 8 in 2024-04-14T15:16:54.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:54.522 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:54.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:16:54.623 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:16:54.649 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:54.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:54.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:16:54.666 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd reweight 1 1 2024-04-14T15:16:54.751 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:16:54.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:54.775 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:54.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:16:54.876 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:16:54.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:54.905 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:54.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:16:55.006 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:55.035 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:55.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:55.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:55.137 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:55.164 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:55.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:55.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:55.266 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:16:55.290 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:55.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:55.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:16:55.293 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:54 smithi090 ceph-mon[41670]: 110.12 scrub starts 2024-04-14T15:16:55.293 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:54 smithi090 ceph-mon[41670]: 110.12 scrub ok 2024-04-14T15:16:55.293 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:54 smithi090 ceph-mon[41670]: 5.5 deep-scrub starts 2024-04-14T15:16:55.293 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:54 smithi090 ceph-mon[41670]: 5.5 deep-scrub ok 2024-04-14T15:16:55.293 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:54 smithi090 ceph-mon[41670]: 3.2 scrub starts 2024-04-14T15:16:55.293 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:54 smithi090 ceph-mon[41670]: 3.2 scrub ok 2024-04-14T15:16:55.293 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:54 smithi090 ceph-mon[41670]: 5.f scrub starts 2024-04-14T15:16:55.293 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:54 smithi090 ceph-mon[41670]: 5.f scrub ok 2024-04-14T15:16:55.293 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:54 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2371781823' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi090-93363-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi090-93363-59"}]': finished 2024-04-14T15:16:55.293 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:54 smithi090 ceph-mon[41670]: osdmap e347: 8 total, 8 up, 8 in 2024-04-14T15:16:55.294 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:54 smithi090 ceph-mon[30399]: 110.12 scrub starts 2024-04-14T15:16:55.294 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:54 smithi090 ceph-mon[30399]: 110.12 scrub ok 2024-04-14T15:16:55.294 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:54 smithi090 ceph-mon[30399]: 5.5 deep-scrub starts 2024-04-14T15:16:55.294 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:54 smithi090 ceph-mon[30399]: 5.5 deep-scrub ok 2024-04-14T15:16:55.294 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:54 smithi090 ceph-mon[30399]: 3.2 scrub starts 2024-04-14T15:16:55.294 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:54 smithi090 ceph-mon[30399]: 3.2 scrub ok 2024-04-14T15:16:55.294 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:54 smithi090 ceph-mon[30399]: 5.f scrub starts 2024-04-14T15:16:55.294 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:54 smithi090 ceph-mon[30399]: 5.f scrub ok 2024-04-14T15:16:55.294 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:54 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2371781823' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi090-93363-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi090-93363-59"}]': finished 2024-04-14T15:16:55.294 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:54 smithi090 ceph-mon[30399]: osdmap e347: 8 total, 8 up, 8 in 2024-04-14T15:16:55.372 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:16:55 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:16:55.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:54 smithi092 ceph-mon[33966]: 110.12 scrub starts 2024-04-14T15:16:55.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:54 smithi092 ceph-mon[33966]: 110.12 scrub ok 2024-04-14T15:16:55.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:54 smithi092 ceph-mon[33966]: 5.5 deep-scrub starts 2024-04-14T15:16:55.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:54 smithi092 ceph-mon[33966]: 5.5 deep-scrub ok 2024-04-14T15:16:55.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:54 smithi092 ceph-mon[33966]: 3.2 scrub starts 2024-04-14T15:16:55.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:54 smithi092 ceph-mon[33966]: 3.2 scrub ok 2024-04-14T15:16:55.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:54 smithi092 ceph-mon[33966]: 5.f scrub starts 2024-04-14T15:16:55.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:54 smithi092 ceph-mon[33966]: 5.f scrub ok 2024-04-14T15:16:55.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:54 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2371781823' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi090-93363-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi090-93363-59"}]': finished 2024-04-14T15:16:55.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:54 smithi092 ceph-mon[33966]: osdmap e347: 8 total, 8 up, 8 in 2024-04-14T15:16:55.391 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:16:55.419 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:55.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:55.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:16:55.521 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:55.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:55.549 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:55.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:55.650 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:16:55.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:55.675 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:55.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:16:55.777 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:16:55.803 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:55.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:55.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:16:55.905 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:16:55.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:55.932 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:55.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:16:56.033 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:16:56.057 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:56.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:56.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:16:56.159 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:56.186 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:56.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:56.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:56.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:55 smithi090 ceph-mon[30399]: 6.15 scrub starts 2024-04-14T15:16:56.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:55 smithi090 ceph-mon[30399]: 6.15 scrub ok 2024-04-14T15:16:56.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:55 smithi090 ceph-mon[30399]: 6.16 deep-scrub starts 2024-04-14T15:16:56.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:55 smithi090 ceph-mon[30399]: 6.16 deep-scrub ok 2024-04-14T15:16:56.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:55 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:16:56.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:55 smithi090 ceph-mon[30399]: pgmap v467: 246 pgs: 8 unknown, 238 active+clean; 8.4 MiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-04-14T15:16:56.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:55 smithi090 ceph-mon[30399]: 6.1a scrub starts 2024-04-14T15:16:56.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:55 smithi090 ceph-mon[30399]: 6.1a scrub ok 2024-04-14T15:16:56.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:55 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:16:56.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:55 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:16:56.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:55 smithi090 ceph-mon[30399]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:16:56.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:55 smithi090 ceph-mon[30399]: osdmap e348: 8 total, 8 up, 8 in 2024-04-14T15:16:56.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:55 smithi090 ceph-mon[41670]: 6.15 scrub starts 2024-04-14T15:16:56.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:55 smithi090 ceph-mon[41670]: 6.15 scrub ok 2024-04-14T15:16:56.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:55 smithi090 ceph-mon[41670]: 6.16 deep-scrub starts 2024-04-14T15:16:56.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:55 smithi090 ceph-mon[41670]: 6.16 deep-scrub ok 2024-04-14T15:16:56.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:55 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:16:56.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:55 smithi090 ceph-mon[41670]: pgmap v467: 246 pgs: 8 unknown, 238 active+clean; 8.4 MiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-04-14T15:16:56.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:55 smithi090 ceph-mon[41670]: 6.1a scrub starts 2024-04-14T15:16:56.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:55 smithi090 ceph-mon[41670]: 6.1a scrub ok 2024-04-14T15:16:56.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:55 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:16:56.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:55 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:16:56.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:55 smithi090 ceph-mon[41670]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:16:56.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:55 smithi090 ceph-mon[41670]: osdmap e348: 8 total, 8 up, 8 in 2024-04-14T15:16:56.288 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:16:56.311 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:56.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:56.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:16:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:55 smithi092 ceph-mon[33966]: 6.15 scrub starts 2024-04-14T15:16:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:55 smithi092 ceph-mon[33966]: 6.15 scrub ok 2024-04-14T15:16:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:55 smithi092 ceph-mon[33966]: 6.16 deep-scrub starts 2024-04-14T15:16:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:55 smithi092 ceph-mon[33966]: 6.16 deep-scrub ok 2024-04-14T15:16:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:55 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:16:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:55 smithi092 ceph-mon[33966]: pgmap v467: 246 pgs: 8 unknown, 238 active+clean; 8.4 MiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-04-14T15:16:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:55 smithi092 ceph-mon[33966]: 6.1a scrub starts 2024-04-14T15:16:56.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:55 smithi092 ceph-mon[33966]: 6.1a scrub ok 2024-04-14T15:16:56.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:55 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:16:56.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:55 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:16:56.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:55 smithi092 ceph-mon[33966]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:16:56.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:55 smithi092 ceph-mon[33966]: osdmap e348: 8 total, 8 up, 8 in 2024-04-14T15:16:56.412 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:56.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:56.440 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:56.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:56.541 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:56.569 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:56.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:56.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:56.671 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:16:56.698 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:56.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:56.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:16:56.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.722+0000 7f0bbabfd700 1 -- 172.21.15.90:0/1152386287 >> v1:172.21.15.90:6790/0 conn(0x7f0bb410e1e0 legacy=0x7f0bb41106a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:56.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.722+0000 7f0bbabfd700 1 -- 172.21.15.90:0/1152386287 shutdown_connections 2024-04-14T15:16:56.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.722+0000 7f0bbabfd700 1 -- 172.21.15.90:0/1152386287 >> 172.21.15.90:0/1152386287 conn(0x7f0bb40fd290 msgr2=0x7f0bb40ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:16:56.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.722+0000 7f0bbabfd700 1 -- 172.21.15.90:0/1152386287 shutdown_connections 2024-04-14T15:16:56.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.722+0000 7f0bbabfd700 1 -- 172.21.15.90:0/1152386287 wait complete. 2024-04-14T15:16:56.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.723+0000 7f0bbabfd700 1 librados: starting msgr at 2024-04-14T15:16:56.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.723+0000 7f0bbabfd700 1 librados: starting objecter 2024-04-14T15:16:56.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.723+0000 7f0bbabfd700 1 Processor -- start 2024-04-14T15:16:56.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.723+0000 7f0bbabfd700 1 -- start start 2024-04-14T15:16:56.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.723+0000 7f0bbabfd700 1 librados: setting wanted keys 2024-04-14T15:16:56.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.723+0000 7f0bbabfd700 1 librados: calling monclient init 2024-04-14T15:16:56.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.723+0000 7f0bbabfd700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f0bb41a9c30 con 0x7f0bb410e1e0 2024-04-14T15:16:56.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.723+0000 7f0bbabfd700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f0bb41aae50 con 0x7f0bb410a610 2024-04-14T15:16:56.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.723+0000 7f0bbabfd700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f0bb41ac090 con 0x7f0bb4107760 2024-04-14T15:16:56.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.723+0000 7f0bb8999700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f0bb4107760 0x7f0bb410d960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:57762/0 (socket says 172.21.15.90:57762) 2024-04-14T15:16:56.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.723+0000 7f0bb8999700 1 -- 172.21.15.90:0/366279944 learned_addr learned my addr 172.21.15.90:0/366279944 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:16:56.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.723+0000 7f0bb1ffb700 10 client.?.objecter ms_handle_connect 0x7f0bb4107760 2024-04-14T15:16:56.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.723+0000 7f0bb1ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:16:56.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.723+0000 7f0bb1ffb700 10 client.?.objecter ms_handle_connect 0x7f0bb410e1e0 2024-04-14T15:16:56.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.723+0000 7f0bb1ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:16:56.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.724+0000 7f0bb1ffb700 1 -- 172.21.15.90:0/366279944 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1543294865 0 0) 0x7f0bb41a9c30 con 0x7f0bb410e1e0 2024-04-14T15:16:56.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.724+0000 7f0bb1ffb700 1 -- 172.21.15.90:0/366279944 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ba00039c0 con 0x7f0bb410e1e0 2024-04-14T15:16:56.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.724+0000 7f0bb1ffb700 1 -- 172.21.15.90:0/366279944 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2903559353 0 0) 0x7f0bb41ac090 con 0x7f0bb4107760 2024-04-14T15:16:56.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.724+0000 7f0bb1ffb700 1 -- 172.21.15.90:0/366279944 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0bb41a9c30 con 0x7f0bb4107760 2024-04-14T15:16:56.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.724+0000 7f0bb1ffb700 10 client.?.objecter ms_handle_connect 0x7f0bb410a610 2024-04-14T15:16:56.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.724+0000 7f0bb1ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:16:56.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.724+0000 7f0bb1ffb700 1 -- 172.21.15.90:0/366279944 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2871890169 0 0) 0x7f0bb41aae50 con 0x7f0bb410a610 2024-04-14T15:16:56.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.724+0000 7f0bb1ffb700 1 -- 172.21.15.90:0/366279944 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0bb41ac090 con 0x7f0bb410a610 2024-04-14T15:16:56.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.724+0000 7f0bb1ffb700 1 -- 172.21.15.90:0/366279944 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2882952493 0 0) 0x7f0bb41a9c30 con 0x7f0bb4107760 2024-04-14T15:16:56.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.724+0000 7f0bb1ffb700 1 -- 172.21.15.90:0/366279944 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0bb41aae50 con 0x7f0bb4107760 2024-04-14T15:16:56.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.724+0000 7f0bb1ffb700 1 -- 172.21.15.90:0/366279944 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f0bac001cb0 con 0x7f0bb4107760 2024-04-14T15:16:56.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.724+0000 7f0bb1ffb700 1 -- 172.21.15.90:0/366279944 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 767496920 0 0) 0x7f0ba00039c0 con 0x7f0bb410e1e0 2024-04-14T15:16:56.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.724+0000 7f0bb1ffb700 1 -- 172.21.15.90:0/366279944 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0bb41a9c30 con 0x7f0bb410e1e0 2024-04-14T15:16:56.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.724+0000 7f0bb1ffb700 1 -- 172.21.15.90:0/366279944 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f0b9c0015b0 con 0x7f0bb410e1e0 2024-04-14T15:16:56.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.724+0000 7f0bb1ffb700 1 -- 172.21.15.90:0/366279944 <== mon.2 v1:172.21.15.90:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3899910445 0 0) 0x7f0bb41aae50 con 0x7f0bb4107760 2024-04-14T15:16:56.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.724+0000 7f0bb1ffb700 1 -- 172.21.15.90:0/366279944 >> v1:172.21.15.92:6789/0 conn(0x7f0bb410a610 legacy=0x7f0bb41a4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:56.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.724+0000 7f0bb1ffb700 1 -- 172.21.15.90:0/366279944 >> v1:172.21.15.90:6789/0 conn(0x7f0bb410e1e0 legacy=0x7f0bb41a82f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:56.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.724+0000 7f0bb1ffb700 1 -- 172.21.15.90:0/366279944 --> v1:172.21.15.90:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bb41ad2d0 con 0x7f0bb4107760 2024-04-14T15:16:56.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.724+0000 7f0bbabfd700 1 -- 172.21.15.90:0/366279944 --> v1:172.21.15.90:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0bb41a9e60 con 0x7f0bb4107760 2024-04-14T15:16:56.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.724+0000 7f0bb1ffb700 1 -- 172.21.15.90:0/366279944 <== mon.2 v1:172.21.15.90:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f0bac002080 con 0x7f0bb4107760 2024-04-14T15:16:56.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.724+0000 7f0bb1ffb700 1 -- 172.21.15.90:0/366279944 <== mon.2 v1:172.21.15.90:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f0bac0061c0 con 0x7f0bb4107760 2024-04-14T15:16:56.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.724+0000 7f0bbabfd700 10 client.47051.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:16:56.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.724+0000 7f0bbabfd700 1 -- 172.21.15.90:0/366279944 --> v1:172.21.15.90:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f0bb41aa320 con 0x7f0bb4107760 2024-04-14T15:16:56.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.724+0000 7f0bbabfd700 1 librados: init done 2024-04-14T15:16:56.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.725+0000 7f0bbabfd700 1 -- 172.21.15.90:0/366279944 --> v1:172.21.15.90:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f0bb4102ea0 con 0x7f0bb4107760 2024-04-14T15:16:56.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.726+0000 7f0bb1ffb700 1 -- 172.21.15.90:0/366279944 <== mon.2 v1:172.21.15.90:6790/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f0bac003cd0 con 0x7f0bb4107760 2024-04-14T15:16:56.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.726+0000 7f0bb1ffb700 1 -- 172.21.15.90:0/366279944 <== mon.2 v1:172.21.15.90:6790/0 8 ==== osd_map(348..348 src has 51..348) ==== 9057+0+0 (unknown 2761960016 0 0) 0x7f0bac08b1f0 con 0x7f0bb4107760 2024-04-14T15:16:56.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.726+0000 7f0bb1ffb700 10 client.47051.objecter ms_dispatch 0x7f0bb4001550 osd_map(348..348 src has 51..348) 2024-04-14T15:16:56.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.726+0000 7f0bb1ffb700 3 client.47051.objecter handle_osd_map got epochs [348,348] > 0 2024-04-14T15:16:56.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.726+0000 7f0bb1ffb700 3 client.47051.objecter handle_osd_map decoding full epoch 348 2024-04-14T15:16:56.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.726+0000 7f0bb1ffb700 20 client.47051.objecter dump_active .. 0 homeless 2024-04-14T15:16:56.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.726+0000 7f0bb1ffb700 1 -- 172.21.15.90:0/366279944 <== mon.2 v1:172.21.15.90:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v348) ==== 54+0+16 (unknown 3527797098 0 2420000192) 0x7f0bac08d5d0 con 0x7f0bb4107760 2024-04-14T15:16:56.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.726+0000 7f0bb1ffb700 10 client.47051.objecter ms_handle_connect 0x7f0ba00710c0 2024-04-14T15:16:56.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.727+0000 7f0bbabfd700 10 client.47051.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:16:56.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.727+0000 7f0bbabfd700 15 client.47051.objecter _session_command_op_assign -1 1 2024-04-14T15:16:56.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.727+0000 7f0bbabfd700 20 client.47051.objecter _get_session s=0x7f0bb40693d0 osd=0 3 2024-04-14T15:16:56.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.727+0000 7f0bbabfd700 20 client.47051.objecter put_session s=0x7f0bb40693d0 osd=0 3 2024-04-14T15:16:56.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.727+0000 7f0bbabfd700 20 client.47051.objecter _get_session s=0x7f0bb40693d0 osd=0 3 2024-04-14T15:16:56.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.727+0000 7f0bbabfd700 15 client.47051.objecter _session_command_op_remove -1 1 2024-04-14T15:16:56.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.727+0000 7f0bbabfd700 20 client.47051.objecter get_session s=0x7f0bb40693d0 osd=0 3 2024-04-14T15:16:56.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.727+0000 7f0bbabfd700 15 client.47051.objecter _session_command_op_assign 0 1 2024-04-14T15:16:56.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.727+0000 7f0bbabfd700 20 client.47051.objecter put_session s=0x7f0bb40693d0 osd=0 4 2024-04-14T15:16:56.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.727+0000 7f0bbabfd700 10 client.47051.objecter _send_command 1 2024-04-14T15:16:56.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.727+0000 7f0bbabfd700 1 -- 172.21.15.90:0/366279944 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0bb410afc0 con 0x7f0bb41ab530 2024-04-14T15:16:56.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.727+0000 7f0bb1ffb700 10 client.47051.objecter ms_handle_connect 0x7f0bb41ab530 2024-04-14T15:16:56.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.728+0000 7f0bb1ffb700 1 -- 172.21.15.90:0/366279944 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f0bb410afc0 con 0x7f0bb41ab530 2024-04-14T15:16:56.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.728+0000 7f0bb1ffb700 10 client.47051.objecter ms_dispatch 0x7f0bb4001550 command_reply(tid 1: 0 ) 2024-04-14T15:16:56.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.728+0000 7f0bb1ffb700 10 client.47051.objecter _finish_command 1 = system:0 2024-04-14T15:16:56.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.728+0000 7f0bb1ffb700 20 client.47051.objecter put_session s=0x7f0bb40693d0 osd=0 4 2024-04-14T15:16:56.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.728+0000 7f0bb1ffb700 15 client.47051.objecter _session_command_op_remove 0 1 2024-04-14T15:16:56.736 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:16:56.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.745+0000 7f0bbabfd700 10 client.47051.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:16:56.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.745+0000 7f0bbabfd700 15 client.47051.objecter _session_command_op_assign -1 2 2024-04-14T15:16:56.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.745+0000 7f0bbabfd700 20 client.47051.objecter _get_session s=0x7f0bb40693d0 osd=0 3 2024-04-14T15:16:56.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.745+0000 7f0bbabfd700 20 client.47051.objecter put_session s=0x7f0bb40693d0 osd=0 3 2024-04-14T15:16:56.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.745+0000 7f0bbabfd700 20 client.47051.objecter _get_session s=0x7f0bb40693d0 osd=0 3 2024-04-14T15:16:56.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.745+0000 7f0bbabfd700 15 client.47051.objecter _session_command_op_remove -1 2 2024-04-14T15:16:56.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.745+0000 7f0bbabfd700 20 client.47051.objecter get_session s=0x7f0bb40693d0 osd=0 3 2024-04-14T15:16:56.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.745+0000 7f0bbabfd700 15 client.47051.objecter _session_command_op_assign 0 2 2024-04-14T15:16:56.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.745+0000 7f0bbabfd700 20 client.47051.objecter put_session s=0x7f0bb40693d0 osd=0 4 2024-04-14T15:16:56.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.745+0000 7f0bbabfd700 10 client.47051.objecter _send_command 2 2024-04-14T15:16:56.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.745+0000 7f0bbabfd700 1 -- 172.21.15.90:0/366279944 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f0bb41aa860 con 0x7f0bb41ab530 2024-04-14T15:16:56.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.747+0000 7f0bb1ffb700 1 -- 172.21.15.90:0/366279944 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f0bb41aa860 con 0x7f0bb41ab530 2024-04-14T15:16:56.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.747+0000 7f0bb1ffb700 10 client.47051.objecter ms_dispatch 0x7f0bb4001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:16:56.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.747+0000 7f0bb1ffb700 10 client.47051.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:16:56.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.747+0000 7f0bb1ffb700 20 client.47051.objecter put_session s=0x7f0bb40693d0 osd=0 4 2024-04-14T15:16:56.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.747+0000 7f0bb1ffb700 15 client.47051.objecter _session_command_op_remove 0 2 2024-04-14T15:16:56.748 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:16:56.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.747+0000 7f0bbabfd700 10 client.47051.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:16:56.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.747+0000 7f0bbabfd700 15 client.47051.objecter _session_command_op_assign -1 3 2024-04-14T15:16:56.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.747+0000 7f0bbabfd700 20 client.47051.objecter _get_session s=0x7f0bb41aac60 osd=1 3 2024-04-14T15:16:56.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.747+0000 7f0bbabfd700 20 client.47051.objecter put_session s=0x7f0bb41aac60 osd=1 3 2024-04-14T15:16:56.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.747+0000 7f0bbabfd700 20 client.47051.objecter _get_session s=0x7f0bb41aac60 osd=1 3 2024-04-14T15:16:56.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.747+0000 7f0bbabfd700 15 client.47051.objecter _session_command_op_remove -1 3 2024-04-14T15:16:56.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.747+0000 7f0bbabfd700 20 client.47051.objecter get_session s=0x7f0bb41aac60 osd=1 3 2024-04-14T15:16:56.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.747+0000 7f0bbabfd700 15 client.47051.objecter _session_command_op_assign 1 3 2024-04-14T15:16:56.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.747+0000 7f0bbabfd700 20 client.47051.objecter put_session s=0x7f0bb41aac60 osd=1 4 2024-04-14T15:16:56.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.747+0000 7f0bbabfd700 10 client.47051.objecter _send_command 3 2024-04-14T15:16:56.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.747+0000 7f0bbabfd700 1 -- 172.21.15.90:0/366279944 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f0bb4109d00 con 0x7f0bb40636c0 2024-04-14T15:16:56.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.747+0000 7f0bb1ffb700 10 client.47051.objecter ms_handle_connect 0x7f0bb40636c0 2024-04-14T15:16:56.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.748+0000 7f0bb1ffb700 1 -- 172.21.15.90:0/366279944 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f0bb4109d00 con 0x7f0bb40636c0 2024-04-14T15:16:56.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.748+0000 7f0bb1ffb700 10 client.47051.objecter ms_dispatch 0x7f0bb4001550 command_reply(tid 3: 0 ) 2024-04-14T15:16:56.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.748+0000 7f0bb1ffb700 10 client.47051.objecter _finish_command 3 = system:0 2024-04-14T15:16:56.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.748+0000 7f0bb1ffb700 20 client.47051.objecter put_session s=0x7f0bb41aac60 osd=1 4 2024-04-14T15:16:56.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.748+0000 7f0bb1ffb700 15 client.47051.objecter _session_command_op_remove 1 3 2024-04-14T15:16:56.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.769+0000 7f0bbabfd700 10 client.47051.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:16:56.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.769+0000 7f0bbabfd700 15 client.47051.objecter _session_command_op_assign -1 4 2024-04-14T15:16:56.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.769+0000 7f0bbabfd700 20 client.47051.objecter _get_session s=0x7f0bb41aac60 osd=1 3 2024-04-14T15:16:56.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.769+0000 7f0bbabfd700 20 client.47051.objecter put_session s=0x7f0bb41aac60 osd=1 3 2024-04-14T15:16:56.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.769+0000 7f0bbabfd700 20 client.47051.objecter _get_session s=0x7f0bb41aac60 osd=1 3 2024-04-14T15:16:56.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.769+0000 7f0bbabfd700 15 client.47051.objecter _session_command_op_remove -1 4 2024-04-14T15:16:56.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.769+0000 7f0bbabfd700 20 client.47051.objecter get_session s=0x7f0bb41aac60 osd=1 3 2024-04-14T15:16:56.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.769+0000 7f0bbabfd700 15 client.47051.objecter _session_command_op_assign 1 4 2024-04-14T15:16:56.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.769+0000 7f0bbabfd700 20 client.47051.objecter put_session s=0x7f0bb41aac60 osd=1 4 2024-04-14T15:16:56.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.769+0000 7f0bbabfd700 10 client.47051.objecter _send_command 4 2024-04-14T15:16:56.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.769+0000 7f0bbabfd700 1 -- 172.21.15.90:0/366279944 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f0bb4063d50 con 0x7f0bb40636c0 2024-04-14T15:16:56.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.771+0000 7f0bb1ffb700 1 -- 172.21.15.90:0/366279944 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f0bb4063d50 con 0x7f0bb40636c0 2024-04-14T15:16:56.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.771+0000 7f0bb1ffb700 10 client.47051.objecter ms_dispatch 0x7f0bb4001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:16:56.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.771+0000 7f0bb1ffb700 10 client.47051.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:16:56.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.771+0000 7f0bb1ffb700 20 client.47051.objecter put_session s=0x7f0bb41aac60 osd=1 4 2024-04-14T15:16:56.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.771+0000 7f0bb1ffb700 15 client.47051.objecter _session_command_op_remove 1 4 2024-04-14T15:16:56.772 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:16:56.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.771+0000 7f0bbabfd700 10 client.47051.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:16:56.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.771+0000 7f0bbabfd700 15 client.47051.objecter _session_command_op_assign -1 5 2024-04-14T15:16:56.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.771+0000 7f0bbabfd700 20 client.47051.objecter _get_session s=0x7f0bb4063f00 osd=2 3 2024-04-14T15:16:56.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.771+0000 7f0bbabfd700 20 client.47051.objecter put_session s=0x7f0bb4063f00 osd=2 3 2024-04-14T15:16:56.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.771+0000 7f0bbabfd700 20 client.47051.objecter _get_session s=0x7f0bb4063f00 osd=2 3 2024-04-14T15:16:56.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.771+0000 7f0bbabfd700 15 client.47051.objecter _session_command_op_remove -1 5 2024-04-14T15:16:56.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.771+0000 7f0bbabfd700 20 client.47051.objecter get_session s=0x7f0bb4063f00 osd=2 3 2024-04-14T15:16:56.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.771+0000 7f0bbabfd700 15 client.47051.objecter _session_command_op_assign 2 5 2024-04-14T15:16:56.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.771+0000 7f0bbabfd700 20 client.47051.objecter put_session s=0x7f0bb4063f00 osd=2 4 2024-04-14T15:16:56.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.771+0000 7f0bbabfd700 10 client.47051.objecter _send_command 5 2024-04-14T15:16:56.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.771+0000 7f0bbabfd700 1 -- 172.21.15.90:0/366279944 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f0bb402da60 con 0x7f0bb402d5f0 2024-04-14T15:16:56.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.771+0000 7f0bb1ffb700 10 client.47051.objecter ms_handle_connect 0x7f0bb402d5f0 2024-04-14T15:16:56.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.772+0000 7f0bb1ffb700 1 -- 172.21.15.90:0/366279944 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f0bb402da60 con 0x7f0bb402d5f0 2024-04-14T15:16:56.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.772+0000 7f0bb1ffb700 10 client.47051.objecter ms_dispatch 0x7f0bb4001550 command_reply(tid 5: 0 ) 2024-04-14T15:16:56.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.772+0000 7f0bb1ffb700 10 client.47051.objecter _finish_command 5 = system:0 2024-04-14T15:16:56.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.772+0000 7f0bb1ffb700 20 client.47051.objecter put_session s=0x7f0bb4063f00 osd=2 4 2024-04-14T15:16:56.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.772+0000 7f0bb1ffb700 15 client.47051.objecter _session_command_op_remove 2 5 2024-04-14T15:16:56.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.788+0000 7f0bbabfd700 10 client.47051.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:16:56.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.788+0000 7f0bbabfd700 15 client.47051.objecter _session_command_op_assign -1 6 2024-04-14T15:16:56.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.788+0000 7f0bbabfd700 20 client.47051.objecter _get_session s=0x7f0bb4063f00 osd=2 3 2024-04-14T15:16:56.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.788+0000 7f0bbabfd700 20 client.47051.objecter put_session s=0x7f0bb4063f00 osd=2 3 2024-04-14T15:16:56.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.788+0000 7f0bbabfd700 20 client.47051.objecter _get_session s=0x7f0bb4063f00 osd=2 3 2024-04-14T15:16:56.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.788+0000 7f0bbabfd700 15 client.47051.objecter _session_command_op_remove -1 6 2024-04-14T15:16:56.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.788+0000 7f0bbabfd700 20 client.47051.objecter get_session s=0x7f0bb4063f00 osd=2 3 2024-04-14T15:16:56.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.788+0000 7f0bbabfd700 15 client.47051.objecter _session_command_op_assign 2 6 2024-04-14T15:16:56.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.788+0000 7f0bbabfd700 20 client.47051.objecter put_session s=0x7f0bb4063f00 osd=2 4 2024-04-14T15:16:56.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.788+0000 7f0bbabfd700 10 client.47051.objecter _send_command 6 2024-04-14T15:16:56.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.788+0000 7f0bbabfd700 1 -- 172.21.15.90:0/366279944 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f0bb41b5e30 con 0x7f0bb402d5f0 2024-04-14T15:16:56.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.790+0000 7f0bb1ffb700 1 -- 172.21.15.90:0/366279944 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f0bb41b5e30 con 0x7f0bb402d5f0 2024-04-14T15:16:56.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.790+0000 7f0bb1ffb700 10 client.47051.objecter ms_dispatch 0x7f0bb4001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:16:56.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.790+0000 7f0bb1ffb700 10 client.47051.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:16:56.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.790+0000 7f0bb1ffb700 20 client.47051.objecter put_session s=0x7f0bb4063f00 osd=2 4 2024-04-14T15:16:56.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.790+0000 7f0bb1ffb700 15 client.47051.objecter _session_command_op_remove 2 6 2024-04-14T15:16:56.791 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:16:56.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.790+0000 7f0bbabfd700 10 client.47051.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:16:56.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.790+0000 7f0bbabfd700 15 client.47051.objecter _session_command_op_assign -1 7 2024-04-14T15:16:56.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.790+0000 7f0bbabfd700 20 client.47051.objecter _get_session s=0x7f0bb402d070 osd=3 3 2024-04-14T15:16:56.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.790+0000 7f0bbabfd700 20 client.47051.objecter put_session s=0x7f0bb402d070 osd=3 3 2024-04-14T15:16:56.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.790+0000 7f0bbabfd700 20 client.47051.objecter _get_session s=0x7f0bb402d070 osd=3 3 2024-04-14T15:16:56.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.790+0000 7f0bbabfd700 15 client.47051.objecter _session_command_op_remove -1 7 2024-04-14T15:16:56.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.790+0000 7f0bbabfd700 20 client.47051.objecter get_session s=0x7f0bb402d070 osd=3 3 2024-04-14T15:16:56.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.790+0000 7f0bbabfd700 15 client.47051.objecter _session_command_op_assign 3 7 2024-04-14T15:16:56.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.790+0000 7f0bbabfd700 20 client.47051.objecter put_session s=0x7f0bb402d070 osd=3 4 2024-04-14T15:16:56.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.790+0000 7f0bbabfd700 10 client.47051.objecter _send_command 7 2024-04-14T15:16:56.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.790+0000 7f0bbabfd700 1 -- 172.21.15.90:0/366279944 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f0bb41b22d0 con 0x7f0bb402d1e0 2024-04-14T15:16:56.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.790+0000 7f0bb1ffb700 10 client.47051.objecter ms_handle_connect 0x7f0bb402d1e0 2024-04-14T15:16:56.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.791+0000 7f0bb1ffb700 1 -- 172.21.15.90:0/366279944 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f0bb41b22d0 con 0x7f0bb402d1e0 2024-04-14T15:16:56.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.791+0000 7f0bb1ffb700 10 client.47051.objecter ms_dispatch 0x7f0bb4001550 command_reply(tid 7: 0 ) 2024-04-14T15:16:56.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.791+0000 7f0bb1ffb700 10 client.47051.objecter _finish_command 7 = system:0 2024-04-14T15:16:56.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.791+0000 7f0bb1ffb700 20 client.47051.objecter put_session s=0x7f0bb402d070 osd=3 4 2024-04-14T15:16:56.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.791+0000 7f0bb1ffb700 15 client.47051.objecter _session_command_op_remove 3 7 2024-04-14T15:16:56.800 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:16:56.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.807+0000 7f0bbabfd700 10 client.47051.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:16:56.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.807+0000 7f0bbabfd700 15 client.47051.objecter _session_command_op_assign -1 8 2024-04-14T15:16:56.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.807+0000 7f0bbabfd700 20 client.47051.objecter _get_session s=0x7f0bb402d070 osd=3 3 2024-04-14T15:16:56.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.807+0000 7f0bbabfd700 20 client.47051.objecter put_session s=0x7f0bb402d070 osd=3 3 2024-04-14T15:16:56.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.807+0000 7f0bbabfd700 20 client.47051.objecter _get_session s=0x7f0bb402d070 osd=3 3 2024-04-14T15:16:56.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.807+0000 7f0bbabfd700 15 client.47051.objecter _session_command_op_remove -1 8 2024-04-14T15:16:56.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.807+0000 7f0bbabfd700 20 client.47051.objecter get_session s=0x7f0bb402d070 osd=3 3 2024-04-14T15:16:56.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.807+0000 7f0bbabfd700 15 client.47051.objecter _session_command_op_assign 3 8 2024-04-14T15:16:56.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.807+0000 7f0bbabfd700 20 client.47051.objecter put_session s=0x7f0bb402d070 osd=3 4 2024-04-14T15:16:56.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.807+0000 7f0bbabfd700 10 client.47051.objecter _send_command 8 2024-04-14T15:16:56.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.807+0000 7f0bbabfd700 1 -- 172.21.15.90:0/366279944 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f0bb41b2530 con 0x7f0bb402d1e0 2024-04-14T15:16:56.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.809+0000 7f0bb1ffb700 1 -- 172.21.15.90:0/366279944 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f0bb41b2530 con 0x7f0bb402d1e0 2024-04-14T15:16:56.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.809+0000 7f0bb1ffb700 10 client.47051.objecter ms_dispatch 0x7f0bb4001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:16:56.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.809+0000 7f0bb1ffb700 10 client.47051.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:16:56.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.809+0000 7f0bb1ffb700 20 client.47051.objecter put_session s=0x7f0bb402d070 osd=3 4 2024-04-14T15:16:56.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.809+0000 7f0bb1ffb700 15 client.47051.objecter _session_command_op_remove 3 8 2024-04-14T15:16:56.810 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:16:56.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.809+0000 7f0bbabfd700 10 client.47051.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:16:56.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.809+0000 7f0bbabfd700 15 client.47051.objecter _session_command_op_assign -1 9 2024-04-14T15:16:56.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.809+0000 7f0bbabfd700 20 client.47051.objecter _get_session s=0x7f0bb41b2530 osd=4 3 2024-04-14T15:16:56.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.809+0000 7f0bbabfd700 20 client.47051.objecter put_session s=0x7f0bb41b2530 osd=4 3 2024-04-14T15:16:56.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.809+0000 7f0bbabfd700 20 client.47051.objecter _get_session s=0x7f0bb41b2530 osd=4 3 2024-04-14T15:16:56.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.809+0000 7f0bbabfd700 15 client.47051.objecter _session_command_op_remove -1 9 2024-04-14T15:16:56.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.809+0000 7f0bbabfd700 20 client.47051.objecter get_session s=0x7f0bb41b2530 osd=4 3 2024-04-14T15:16:56.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.809+0000 7f0bbabfd700 15 client.47051.objecter _session_command_op_assign 4 9 2024-04-14T15:16:56.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.809+0000 7f0bbabfd700 20 client.47051.objecter put_session s=0x7f0bb41b2530 osd=4 4 2024-04-14T15:16:56.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.809+0000 7f0bbabfd700 10 client.47051.objecter _send_command 9 2024-04-14T15:16:56.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.809+0000 7f0bbabfd700 1 -- 172.21.15.90:0/366279944 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f0bb41b6c60 con 0x7f0bb41b67b0 2024-04-14T15:16:56.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.810+0000 7f0bb1ffb700 10 client.47051.objecter ms_handle_connect 0x7f0bb41b67b0 2024-04-14T15:16:56.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.811+0000 7f0bb1ffb700 1 -- 172.21.15.90:0/366279944 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f0bb41b6c60 con 0x7f0bb41b67b0 2024-04-14T15:16:56.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.811+0000 7f0bb1ffb700 10 client.47051.objecter ms_dispatch 0x7f0bb4001550 command_reply(tid 9: 0 ) 2024-04-14T15:16:56.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.811+0000 7f0bb1ffb700 10 client.47051.objecter _finish_command 9 = system:0 2024-04-14T15:16:56.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.811+0000 7f0bb1ffb700 20 client.47051.objecter put_session s=0x7f0bb41b2530 osd=4 4 2024-04-14T15:16:56.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.811+0000 7f0bb1ffb700 15 client.47051.objecter _session_command_op_remove 4 9 2024-04-14T15:16:56.827 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:56.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:56.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:16:56.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.827+0000 7f0bbabfd700 10 client.47051.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:16:56.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.827+0000 7f0bbabfd700 15 client.47051.objecter _session_command_op_assign -1 10 2024-04-14T15:16:56.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.827+0000 7f0bbabfd700 20 client.47051.objecter _get_session s=0x7f0bb41b2530 osd=4 3 2024-04-14T15:16:56.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.827+0000 7f0bbabfd700 20 client.47051.objecter put_session s=0x7f0bb41b2530 osd=4 3 2024-04-14T15:16:56.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.827+0000 7f0bbabfd700 20 client.47051.objecter _get_session s=0x7f0bb41b2530 osd=4 3 2024-04-14T15:16:56.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.827+0000 7f0bbabfd700 15 client.47051.objecter _session_command_op_remove -1 10 2024-04-14T15:16:56.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.827+0000 7f0bbabfd700 20 client.47051.objecter get_session s=0x7f0bb41b2530 osd=4 3 2024-04-14T15:16:56.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.827+0000 7f0bbabfd700 15 client.47051.objecter _session_command_op_assign 4 10 2024-04-14T15:16:56.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.827+0000 7f0bbabfd700 20 client.47051.objecter put_session s=0x7f0bb41b2530 osd=4 4 2024-04-14T15:16:56.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.827+0000 7f0bbabfd700 10 client.47051.objecter _send_command 10 2024-04-14T15:16:56.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.827+0000 7f0bbabfd700 1 -- 172.21.15.90:0/366279944 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f0bb41b6ee0 con 0x7f0bb41b67b0 2024-04-14T15:16:56.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.829+0000 7f0bb1ffb700 1 -- 172.21.15.90:0/366279944 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f0bb41b6ee0 con 0x7f0bb41b67b0 2024-04-14T15:16:56.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.829+0000 7f0bb1ffb700 10 client.47051.objecter ms_dispatch 0x7f0bb4001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:16:56.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.829+0000 7f0bb1ffb700 10 client.47051.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:16:56.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.829+0000 7f0bb1ffb700 20 client.47051.objecter put_session s=0x7f0bb41b2530 osd=4 4 2024-04-14T15:16:56.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.829+0000 7f0bb1ffb700 15 client.47051.objecter _session_command_op_remove 4 10 2024-04-14T15:16:56.831 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:16:56.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.830+0000 7f0bbabfd700 10 client.47051.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:16:56.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.830+0000 7f0bbabfd700 15 client.47051.objecter _session_command_op_assign -1 11 2024-04-14T15:16:56.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.830+0000 7f0bbabfd700 20 client.47051.objecter _get_session s=0x7f0bb41b6ee0 osd=5 3 2024-04-14T15:16:56.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.830+0000 7f0bbabfd700 20 client.47051.objecter put_session s=0x7f0bb41b6ee0 osd=5 3 2024-04-14T15:16:56.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.830+0000 7f0bbabfd700 20 client.47051.objecter _get_session s=0x7f0bb41b6ee0 osd=5 3 2024-04-14T15:16:56.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.830+0000 7f0bbabfd700 15 client.47051.objecter _session_command_op_remove -1 11 2024-04-14T15:16:56.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.830+0000 7f0bbabfd700 20 client.47051.objecter get_session s=0x7f0bb41b6ee0 osd=5 3 2024-04-14T15:16:56.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.830+0000 7f0bbabfd700 15 client.47051.objecter _session_command_op_assign 5 11 2024-04-14T15:16:56.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.830+0000 7f0bbabfd700 20 client.47051.objecter put_session s=0x7f0bb41b6ee0 osd=5 4 2024-04-14T15:16:56.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.830+0000 7f0bbabfd700 10 client.47051.objecter _send_command 11 2024-04-14T15:16:56.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.830+0000 7f0bbabfd700 1 -- 172.21.15.90:0/366279944 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f0bb41c31c0 con 0x7f0bb41bf6b0 2024-04-14T15:16:56.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.830+0000 7f0bb1ffb700 10 client.47051.objecter ms_handle_connect 0x7f0bb41bf6b0 2024-04-14T15:16:56.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.831+0000 7f0bb1ffb700 1 -- 172.21.15.90:0/366279944 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f0bb41c31c0 con 0x7f0bb41bf6b0 2024-04-14T15:16:56.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.831+0000 7f0bb1ffb700 10 client.47051.objecter ms_dispatch 0x7f0bb4001550 command_reply(tid 11: 0 ) 2024-04-14T15:16:56.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.831+0000 7f0bb1ffb700 10 client.47051.objecter _finish_command 11 = system:0 2024-04-14T15:16:56.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.831+0000 7f0bb1ffb700 20 client.47051.objecter put_session s=0x7f0bb41b6ee0 osd=5 4 2024-04-14T15:16:56.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.831+0000 7f0bb1ffb700 15 client.47051.objecter _session_command_op_remove 5 11 2024-04-14T15:16:56.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.847+0000 7f0bbabfd700 10 client.47051.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:16:56.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.847+0000 7f0bbabfd700 15 client.47051.objecter _session_command_op_assign -1 12 2024-04-14T15:16:56.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.847+0000 7f0bbabfd700 20 client.47051.objecter _get_session s=0x7f0bb41b6ee0 osd=5 3 2024-04-14T15:16:56.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.847+0000 7f0bbabfd700 20 client.47051.objecter put_session s=0x7f0bb41b6ee0 osd=5 3 2024-04-14T15:16:56.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.847+0000 7f0bbabfd700 20 client.47051.objecter _get_session s=0x7f0bb41b6ee0 osd=5 3 2024-04-14T15:16:56.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.847+0000 7f0bbabfd700 15 client.47051.objecter _session_command_op_remove -1 12 2024-04-14T15:16:56.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.847+0000 7f0bbabfd700 20 client.47051.objecter get_session s=0x7f0bb41b6ee0 osd=5 3 2024-04-14T15:16:56.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.847+0000 7f0bbabfd700 15 client.47051.objecter _session_command_op_assign 5 12 2024-04-14T15:16:56.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.847+0000 7f0bbabfd700 20 client.47051.objecter put_session s=0x7f0bb41b6ee0 osd=5 4 2024-04-14T15:16:56.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.847+0000 7f0bbabfd700 10 client.47051.objecter _send_command 12 2024-04-14T15:16:56.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.847+0000 7f0bbabfd700 1 -- 172.21.15.90:0/366279944 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f0bb41c4b40 con 0x7f0bb41bf6b0 2024-04-14T15:16:56.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.849+0000 7f0bb1ffb700 1 -- 172.21.15.90:0/366279944 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f0bb41c4b40 con 0x7f0bb41bf6b0 2024-04-14T15:16:56.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.849+0000 7f0bb1ffb700 10 client.47051.objecter ms_dispatch 0x7f0bb4001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:16:56.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.849+0000 7f0bb1ffb700 10 client.47051.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:16:56.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.849+0000 7f0bb1ffb700 20 client.47051.objecter put_session s=0x7f0bb41b6ee0 osd=5 4 2024-04-14T15:16:56.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.849+0000 7f0bb1ffb700 15 client.47051.objecter _session_command_op_remove 5 12 2024-04-14T15:16:56.850 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:16:56.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.849+0000 7f0bbabfd700 10 client.47051.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:16:56.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.849+0000 7f0bbabfd700 15 client.47051.objecter _session_command_op_assign -1 13 2024-04-14T15:16:56.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.849+0000 7f0bbabfd700 20 client.47051.objecter _get_session s=0x7f0bb41c4b40 osd=6 3 2024-04-14T15:16:56.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.849+0000 7f0bbabfd700 20 client.47051.objecter put_session s=0x7f0bb41c4b40 osd=6 3 2024-04-14T15:16:56.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.849+0000 7f0bbabfd700 20 client.47051.objecter _get_session s=0x7f0bb41c4b40 osd=6 3 2024-04-14T15:16:56.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.849+0000 7f0bbabfd700 15 client.47051.objecter _session_command_op_remove -1 13 2024-04-14T15:16:56.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.849+0000 7f0bbabfd700 20 client.47051.objecter get_session s=0x7f0bb41c4b40 osd=6 3 2024-04-14T15:16:56.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.849+0000 7f0bbabfd700 15 client.47051.objecter _session_command_op_assign 6 13 2024-04-14T15:16:56.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.849+0000 7f0bbabfd700 20 client.47051.objecter put_session s=0x7f0bb41c4b40 osd=6 4 2024-04-14T15:16:56.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.849+0000 7f0bbabfd700 10 client.47051.objecter _send_command 13 2024-04-14T15:16:56.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.849+0000 7f0bbabfd700 1 -- 172.21.15.90:0/366279944 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f0bb41c4950 con 0x7f0bb41c5260 2024-04-14T15:16:56.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.850+0000 7f0bb1ffb700 10 client.47051.objecter ms_handle_connect 0x7f0bb41c5260 2024-04-14T15:16:56.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.851+0000 7f0bb1ffb700 1 -- 172.21.15.90:0/366279944 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f0b9c003d50 con 0x7f0bb41c5260 2024-04-14T15:16:56.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.851+0000 7f0bb1ffb700 10 client.47051.objecter ms_dispatch 0x7f0bb4001550 command_reply(tid 13: 0 ) 2024-04-14T15:16:56.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.851+0000 7f0bb1ffb700 10 client.47051.objecter _finish_command 13 = system:0 2024-04-14T15:16:56.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.851+0000 7f0bb1ffb700 20 client.47051.objecter put_session s=0x7f0bb41c4b40 osd=6 4 2024-04-14T15:16:56.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.851+0000 7f0bb1ffb700 15 client.47051.objecter _session_command_op_remove 6 13 2024-04-14T15:16:56.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.867+0000 7f0bbabfd700 10 client.47051.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:16:56.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.867+0000 7f0bbabfd700 15 client.47051.objecter _session_command_op_assign -1 14 2024-04-14T15:16:56.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.867+0000 7f0bbabfd700 20 client.47051.objecter _get_session s=0x7f0bb41c4b40 osd=6 3 2024-04-14T15:16:56.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.867+0000 7f0bbabfd700 20 client.47051.objecter put_session s=0x7f0bb41c4b40 osd=6 3 2024-04-14T15:16:56.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.867+0000 7f0bbabfd700 20 client.47051.objecter _get_session s=0x7f0bb41c4b40 osd=6 3 2024-04-14T15:16:56.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.867+0000 7f0bbabfd700 15 client.47051.objecter _session_command_op_remove -1 14 2024-04-14T15:16:56.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.867+0000 7f0bbabfd700 20 client.47051.objecter get_session s=0x7f0bb41c4b40 osd=6 3 2024-04-14T15:16:56.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.867+0000 7f0bbabfd700 15 client.47051.objecter _session_command_op_assign 6 14 2024-04-14T15:16:56.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.867+0000 7f0bbabfd700 20 client.47051.objecter put_session s=0x7f0bb41c4b40 osd=6 4 2024-04-14T15:16:56.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.867+0000 7f0bbabfd700 10 client.47051.objecter _send_command 14 2024-04-14T15:16:56.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.867+0000 7f0bbabfd700 1 -- 172.21.15.90:0/366279944 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f0bb41c5850 con 0x7f0bb41c5260 2024-04-14T15:16:56.869 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:16:56.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.869+0000 7f0bb1ffb700 1 -- 172.21.15.90:0/366279944 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f0bb41c5850 con 0x7f0bb41c5260 2024-04-14T15:16:56.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.869+0000 7f0bb1ffb700 10 client.47051.objecter ms_dispatch 0x7f0bb4001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:16:56.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.869+0000 7f0bb1ffb700 10 client.47051.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:16:56.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.869+0000 7f0bb1ffb700 20 client.47051.objecter put_session s=0x7f0bb41c4b40 osd=6 4 2024-04-14T15:16:56.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.869+0000 7f0bb1ffb700 15 client.47051.objecter _session_command_op_remove 6 14 2024-04-14T15:16:56.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.869+0000 7f0bbabfd700 10 client.47051.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:16:56.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.869+0000 7f0bbabfd700 15 client.47051.objecter _session_command_op_assign -1 15 2024-04-14T15:16:56.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.869+0000 7f0bbabfd700 20 client.47051.objecter _get_session s=0x7f0bb41c5850 osd=7 3 2024-04-14T15:16:56.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.869+0000 7f0bbabfd700 20 client.47051.objecter put_session s=0x7f0bb41c5850 osd=7 3 2024-04-14T15:16:56.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.869+0000 7f0bbabfd700 20 client.47051.objecter _get_session s=0x7f0bb41c5850 osd=7 3 2024-04-14T15:16:56.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.869+0000 7f0bbabfd700 15 client.47051.objecter _session_command_op_remove -1 15 2024-04-14T15:16:56.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.869+0000 7f0bbabfd700 20 client.47051.objecter get_session s=0x7f0bb41c5850 osd=7 3 2024-04-14T15:16:56.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.869+0000 7f0bbabfd700 15 client.47051.objecter _session_command_op_assign 7 15 2024-04-14T15:16:56.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.869+0000 7f0bbabfd700 20 client.47051.objecter put_session s=0x7f0bb41c5850 osd=7 4 2024-04-14T15:16:56.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.869+0000 7f0bbabfd700 10 client.47051.objecter _send_command 15 2024-04-14T15:16:56.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.869+0000 7f0bbabfd700 1 -- 172.21.15.90:0/366279944 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f0bb41cbd60 con 0x7f0bb41c8250 2024-04-14T15:16:56.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.870+0000 7f0bb1ffb700 10 client.47051.objecter ms_handle_connect 0x7f0bb41c8250 2024-04-14T15:16:56.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.870+0000 7f0bb1ffb700 1 -- 172.21.15.90:0/366279944 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f0bb41cbd60 con 0x7f0bb41c8250 2024-04-14T15:16:56.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.870+0000 7f0bb1ffb700 10 client.47051.objecter ms_dispatch 0x7f0bb4001550 command_reply(tid 15: 0 ) 2024-04-14T15:16:56.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.870+0000 7f0bb1ffb700 10 client.47051.objecter _finish_command 15 = system:0 2024-04-14T15:16:56.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.870+0000 7f0bb1ffb700 20 client.47051.objecter put_session s=0x7f0bb41c5850 osd=7 4 2024-04-14T15:16:56.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.870+0000 7f0bb1ffb700 15 client.47051.objecter _session_command_op_remove 7 15 2024-04-14T15:16:56.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.892+0000 7f0bbabfd700 10 client.47051.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:16:56.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.892+0000 7f0bbabfd700 15 client.47051.objecter _session_command_op_assign -1 16 2024-04-14T15:16:56.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.892+0000 7f0bbabfd700 20 client.47051.objecter _get_session s=0x7f0bb41c5850 osd=7 3 2024-04-14T15:16:56.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.892+0000 7f0bbabfd700 20 client.47051.objecter put_session s=0x7f0bb41c5850 osd=7 3 2024-04-14T15:16:56.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.892+0000 7f0bbabfd700 20 client.47051.objecter _get_session s=0x7f0bb41c5850 osd=7 3 2024-04-14T15:16:56.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.892+0000 7f0bbabfd700 15 client.47051.objecter _session_command_op_remove -1 16 2024-04-14T15:16:56.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.892+0000 7f0bbabfd700 20 client.47051.objecter get_session s=0x7f0bb41c5850 osd=7 3 2024-04-14T15:16:56.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.892+0000 7f0bbabfd700 15 client.47051.objecter _session_command_op_assign 7 16 2024-04-14T15:16:56.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.892+0000 7f0bbabfd700 20 client.47051.objecter put_session s=0x7f0bb41c5850 osd=7 4 2024-04-14T15:16:56.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.892+0000 7f0bbabfd700 10 client.47051.objecter _send_command 16 2024-04-14T15:16:56.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.892+0000 7f0bbabfd700 1 -- 172.21.15.90:0/366279944 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f0bb41cbd60 con 0x7f0bb41c8250 2024-04-14T15:16:56.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.894+0000 7f0bb1ffb700 1 -- 172.21.15.90:0/366279944 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f0bb41cbd60 con 0x7f0bb41c8250 2024-04-14T15:16:56.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.894+0000 7f0bb1ffb700 10 client.47051.objecter ms_dispatch 0x7f0bb4001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:16:56.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.894+0000 7f0bb1ffb700 10 client.47051.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:16:56.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.894+0000 7f0bb1ffb700 20 client.47051.objecter put_session s=0x7f0bb41c5850 osd=7 4 2024-04-14T15:16:56.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.894+0000 7f0bb1ffb700 15 client.47051.objecter _session_command_op_remove 7 16 2024-04-14T15:16:56.894 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:16:56.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.894+0000 7f0bbabfd700 10 librados: watch_flush enter 2024-04-14T15:16:56.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.894+0000 7f0bbabfd700 10 librados: watch_flush exit 2024-04-14T15:16:56.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.894+0000 7f0bbabfd700 10 client.47051.objecter close_session for osd.0 2024-04-14T15:16:56.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.894+0000 7f0bbabfd700 1 -- 172.21.15.90:0/366279944 >> v1:172.21.15.90:6801/3761806885 conn(0x7f0bb41ab530 legacy=0x7f0bb41ab910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:56.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.894+0000 7f0bbabfd700 20 client.47051.objecter put_session s=0x7f0bb40693d0 osd=0 1 2024-04-14T15:16:56.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.894+0000 7f0bbabfd700 10 client.47051.objecter close_session for osd.1 2024-04-14T15:16:56.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.894+0000 7f0bbabfd700 1 -- 172.21.15.90:0/366279944 >> v1:172.21.15.90:6805/4170361699 conn(0x7f0bb40636c0 legacy=0x7f0bb41085e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:56.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.894+0000 7f0bbabfd700 20 client.47051.objecter put_session s=0x7f0bb41aac60 osd=1 1 2024-04-14T15:16:56.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.894+0000 7f0bbabfd700 10 client.47051.objecter close_session for osd.2 2024-04-14T15:16:56.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.894+0000 7f0bbabfd700 1 -- 172.21.15.90:0/366279944 >> v1:172.21.15.90:6809/1686184251 conn(0x7f0bb402d5f0 legacy=0x7f0bb41b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:56.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.894+0000 7f0bbabfd700 20 client.47051.objecter put_session s=0x7f0bb4063f00 osd=2 1 2024-04-14T15:16:56.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.894+0000 7f0bbabfd700 10 client.47051.objecter close_session for osd.3 2024-04-14T15:16:56.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.894+0000 7f0bbabfd700 1 -- 172.21.15.90:0/366279944 >> v1:172.21.15.90:6813/2270166774 conn(0x7f0bb402d1e0 legacy=0x7f0bb41b91d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:56.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.894+0000 7f0bbabfd700 20 client.47051.objecter put_session s=0x7f0bb402d070 osd=3 1 2024-04-14T15:16:56.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.894+0000 7f0bbabfd700 10 client.47051.objecter close_session for osd.4 2024-04-14T15:16:56.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.894+0000 7f0bbabfd700 1 -- 172.21.15.90:0/366279944 >> v1:172.21.15.92:6800/3887549733 conn(0x7f0bb41b67b0 legacy=0x7f0bb41bda80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:56.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.894+0000 7f0bbabfd700 20 client.47051.objecter put_session s=0x7f0bb41b2530 osd=4 1 2024-04-14T15:16:56.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.894+0000 7f0bbabfd700 10 client.47051.objecter close_session for osd.5 2024-04-14T15:16:56.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.894+0000 7f0bbabfd700 1 -- 172.21.15.90:0/366279944 >> v1:172.21.15.92:6804/3244561440 conn(0x7f0bb41bf6b0 legacy=0x7f0bb41c1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:56.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.894+0000 7f0bb919a700 1 -- 172.21.15.90:0/366279944 reap_dead start 2024-04-14T15:16:56.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.894+0000 7f0bb919a700 1 -- 172.21.15.90:0/366279944 reap_dead start 2024-04-14T15:16:56.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.894+0000 7f0bbabfd700 20 client.47051.objecter put_session s=0x7f0bb41b6ee0 osd=5 1 2024-04-14T15:16:56.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.894+0000 7f0bbabfd700 10 client.47051.objecter close_session for osd.6 2024-04-14T15:16:56.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.894+0000 7f0bb919a700 1 -- 172.21.15.90:0/366279944 reap_dead start 2024-04-14T15:16:56.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.894+0000 7f0bbabfd700 1 -- 172.21.15.90:0/366279944 >> v1:172.21.15.92:6808/636125963 conn(0x7f0bb41c5260 legacy=0x7f0bb41c31c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:56.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.894+0000 7f0bbabfd700 20 client.47051.objecter put_session s=0x7f0bb41c4b40 osd=6 1 2024-04-14T15:16:56.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.894+0000 7f0bbabfd700 10 client.47051.objecter close_session for osd.7 2024-04-14T15:16:56.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.894+0000 7f0bbabfd700 1 -- 172.21.15.90:0/366279944 >> v1:172.21.15.92:6812/1773706716 conn(0x7f0bb41c8250 legacy=0x7f0bb41ca640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:56.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.894+0000 7f0bbabfd700 20 client.47051.objecter put_session s=0x7f0bb41c5850 osd=7 1 2024-04-14T15:16:56.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.894+0000 7f0bbabfd700 20 client.47051.objecter shutdown clearing up homeless session... 2024-04-14T15:16:56.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.894+0000 7f0bbabfd700 10 client.47051.objecter successfully canceled tick 2024-04-14T15:16:56.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.894+0000 7f0bbabfd700 1 -- 172.21.15.90:0/366279944 >> v1:172.21.15.90:6800/420935427 conn(0x7f0ba00710c0 legacy=0x7f0ba0073580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:56.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.894+0000 7f0bbabfd700 1 -- 172.21.15.90:0/366279944 >> v1:172.21.15.90:6790/0 conn(0x7f0bb4107760 legacy=0x7f0bb410d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:16:56.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.894+0000 7f0bbabfd700 1 -- 172.21.15.90:0/366279944 shutdown_connections 2024-04-14T15:16:56.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.894+0000 7f0bbabfd700 1 -- 172.21.15.90:0/366279944 >> 172.21.15.90:0/366279944 conn(0x7f0bb40fd290 msgr2=0x7f0bb4111750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:16:56.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.894+0000 7f0bbabfd700 1 -- 172.21.15.90:0/366279944 shutdown_connections 2024-04-14T15:16:56.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.894+0000 7f0bbabfd700 1 -- 172.21.15.90:0/366279944 wait complete. 2024-04-14T15:16:56.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:16:56.895+0000 7f0bbabfd700 1 librados: shutdown 2024-04-14T15:16:56.898 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:16:56.898 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:16:56.898 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:16:56.898 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:16:56.899 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:16:56.899 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:16:56.899 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:16:56.899 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:16:56.928 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:16:56.954 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:56.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:56.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:16:57.056 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:16:57.084 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:57.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:57.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:16:57.186 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:57.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:56 smithi090 ceph-mon[41670]: 4.16 deep-scrub starts 2024-04-14T15:16:57.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:56 smithi090 ceph-mon[41670]: 4.16 deep-scrub ok 2024-04-14T15:16:57.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:56 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:16:57.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:56 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:16:57.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:56 smithi090 ceph-mon[41670]: 6.1 deep-scrub starts 2024-04-14T15:16:57.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:56 smithi090 ceph-mon[41670]: 6.1 deep-scrub ok 2024-04-14T15:16:57.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:56 smithi090 ceph-mon[41670]: 11.e scrub starts 2024-04-14T15:16:57.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:56 smithi090 ceph-mon[41670]: 11.e scrub ok 2024-04-14T15:16:57.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:56 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/366279944' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:16:57.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:56 smithi090 ceph-mon[41670]: osdmap e349: 8 total, 8 up, 8 in 2024-04-14T15:16:57.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:56 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2371781823' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi090-93363-59"}]: dispatch 2024-04-14T15:16:57.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:56 smithi090 ceph-mon[30399]: 4.16 deep-scrub starts 2024-04-14T15:16:57.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:56 smithi090 ceph-mon[30399]: 4.16 deep-scrub ok 2024-04-14T15:16:57.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:56 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:16:57.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:56 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:16:57.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:56 smithi090 ceph-mon[30399]: 6.1 deep-scrub starts 2024-04-14T15:16:57.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:56 smithi090 ceph-mon[30399]: 6.1 deep-scrub ok 2024-04-14T15:16:57.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:56 smithi090 ceph-mon[30399]: 11.e scrub starts 2024-04-14T15:16:57.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:56 smithi090 ceph-mon[30399]: 11.e scrub ok 2024-04-14T15:16:57.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:56 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/366279944' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:16:57.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:56 smithi090 ceph-mon[30399]: osdmap e349: 8 total, 8 up, 8 in 2024-04-14T15:16:57.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:56 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2371781823' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi090-93363-59"}]: dispatch 2024-04-14T15:16:57.214 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:57.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:57.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:57.316 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:16:57.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:57.344 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:57.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:16:57.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:56 smithi092 ceph-mon[33966]: 4.16 deep-scrub starts 2024-04-14T15:16:57.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:56 smithi092 ceph-mon[33966]: 4.16 deep-scrub ok 2024-04-14T15:16:57.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:56 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:16:57.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:56 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:16:57.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:56 smithi092 ceph-mon[33966]: 6.1 deep-scrub starts 2024-04-14T15:16:57.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:56 smithi092 ceph-mon[33966]: 6.1 deep-scrub ok 2024-04-14T15:16:57.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:56 smithi092 ceph-mon[33966]: 11.e scrub starts 2024-04-14T15:16:57.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:56 smithi092 ceph-mon[33966]: 11.e scrub ok 2024-04-14T15:16:57.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:56 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/366279944' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:16:57.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:56 smithi092 ceph-mon[33966]: osdmap e349: 8 total, 8 up, 8 in 2024-04-14T15:16:57.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:56 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2371781823' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi090-93363-59"}]: dispatch 2024-04-14T15:16:57.446 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:16:57.469 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:57.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:57.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:16:57.571 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:16:57.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:57.596 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:57.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:16:57.697 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:16:57.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:57.726 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:57.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:16:57.827 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:16:57.853 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:57.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:57.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:16:57.955 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:16:57.981 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:57.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:57.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:16:58.083 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:16:58.108 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:58.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:58.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:16:58.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:57 smithi090 ceph-mon[41670]: 4.b scrub starts 2024-04-14T15:16:58.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:57 smithi090 ceph-mon[41670]: 4.b scrub ok 2024-04-14T15:16:58.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:57 smithi090 ceph-mon[41670]: 2.19 scrub starts 2024-04-14T15:16:58.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:57 smithi090 ceph-mon[41670]: 2.19 scrub ok 2024-04-14T15:16:58.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:57 smithi090 ceph-mon[41670]: pgmap v470: 238 pgs: 238 active+clean; 8.4 MiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-04-14T15:16:58.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:57 smithi090 ceph-mon[41670]: 6.1d scrub starts 2024-04-14T15:16:58.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:57 smithi090 ceph-mon[41670]: 6.1d scrub ok 2024-04-14T15:16:58.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:57 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2371781823' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi090-93363-59"}]': finished 2024-04-14T15:16:58.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:57 smithi090 ceph-mon[41670]: osdmap e350: 8 total, 8 up, 8 in 2024-04-14T15:16:58.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:57 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2371781823' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi090-93363-59"}]: dispatch 2024-04-14T15:16:58.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:57 smithi090 ceph-mon[30399]: 4.b scrub starts 2024-04-14T15:16:58.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:57 smithi090 ceph-mon[30399]: 4.b scrub ok 2024-04-14T15:16:58.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:57 smithi090 ceph-mon[30399]: 2.19 scrub starts 2024-04-14T15:16:58.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:57 smithi090 ceph-mon[30399]: 2.19 scrub ok 2024-04-14T15:16:58.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:57 smithi090 ceph-mon[30399]: pgmap v470: 238 pgs: 238 active+clean; 8.4 MiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-04-14T15:16:58.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:57 smithi090 ceph-mon[30399]: 6.1d scrub starts 2024-04-14T15:16:58.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:57 smithi090 ceph-mon[30399]: 6.1d scrub ok 2024-04-14T15:16:58.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:57 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2371781823' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi090-93363-59"}]': finished 2024-04-14T15:16:58.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:57 smithi090 ceph-mon[30399]: osdmap e350: 8 total, 8 up, 8 in 2024-04-14T15:16:58.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:57 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2371781823' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi090-93363-59"}]: dispatch 2024-04-14T15:16:58.209 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:58.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:58.235 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:58.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:58.336 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:16:58.361 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:58.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:58.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:16:58.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:57 smithi092 ceph-mon[33966]: 4.b scrub starts 2024-04-14T15:16:58.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:57 smithi092 ceph-mon[33966]: 4.b scrub ok 2024-04-14T15:16:58.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:57 smithi092 ceph-mon[33966]: 2.19 scrub starts 2024-04-14T15:16:58.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:57 smithi092 ceph-mon[33966]: 2.19 scrub ok 2024-04-14T15:16:58.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:57 smithi092 ceph-mon[33966]: pgmap v470: 238 pgs: 238 active+clean; 8.4 MiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-04-14T15:16:58.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:57 smithi092 ceph-mon[33966]: 6.1d scrub starts 2024-04-14T15:16:58.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:57 smithi092 ceph-mon[33966]: 6.1d scrub ok 2024-04-14T15:16:58.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:57 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2371781823' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi090-93363-59"}]': finished 2024-04-14T15:16:58.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:57 smithi092 ceph-mon[33966]: osdmap e350: 8 total, 8 up, 8 in 2024-04-14T15:16:58.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:57 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2371781823' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi090-93363-59"}]: dispatch 2024-04-14T15:16:58.463 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:58.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:58.489 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:58.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:58.590 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:16:58.616 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:58.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:58.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:16:58.718 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:16:58.745 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:58.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:58.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:16:58.851 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:16:58.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:58.879 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:58.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:16:58.980 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:59.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:59.010 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:59.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:59.111 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:16:59.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:59.137 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:59.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:16:59.238 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:16:59.252 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:58 smithi090 ceph-mon[30399]: 11.14 scrub starts 2024-04-14T15:16:59.252 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:58 smithi090 ceph-mon[30399]: 11.14 scrub ok 2024-04-14T15:16:59.252 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:58 smithi090 ceph-mon[30399]: 6.7 scrub starts 2024-04-14T15:16:59.252 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:58 smithi090 ceph-mon[30399]: 6.7 scrub ok 2024-04-14T15:16:59.252 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:58 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2371781823' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi090-93363-59"}]': finished 2024-04-14T15:16:59.252 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:58 smithi090 ceph-mon[30399]: osdmap e351: 8 total, 8 up, 8 in 2024-04-14T15:16:59.252 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:58 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:16:59.252 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:58 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:16:59.252 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:58 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-45"}]: dispatch 2024-04-14T15:16:59.253 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:58 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-45"}]: dispatch 2024-04-14T15:16:59.253 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:58 smithi090 ceph-mon[41670]: 11.14 scrub starts 2024-04-14T15:16:59.253 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:58 smithi090 ceph-mon[41670]: 11.14 scrub ok 2024-04-14T15:16:59.253 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:58 smithi090 ceph-mon[41670]: 6.7 scrub starts 2024-04-14T15:16:59.253 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:58 smithi090 ceph-mon[41670]: 6.7 scrub ok 2024-04-14T15:16:59.253 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:58 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2371781823' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi090-93363-59"}]': finished 2024-04-14T15:16:59.253 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:58 smithi090 ceph-mon[41670]: osdmap e351: 8 total, 8 up, 8 in 2024-04-14T15:16:59.253 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:58 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:16:59.253 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:58 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:16:59.253 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:58 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-45"}]: dispatch 2024-04-14T15:16:59.253 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:58 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-45"}]: dispatch 2024-04-14T15:16:59.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:59.267 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:59.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:16:59.368 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:16:59.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:58 smithi092 ceph-mon[33966]: 11.14 scrub starts 2024-04-14T15:16:59.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:58 smithi092 ceph-mon[33966]: 11.14 scrub ok 2024-04-14T15:16:59.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:58 smithi092 ceph-mon[33966]: 6.7 scrub starts 2024-04-14T15:16:59.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:58 smithi092 ceph-mon[33966]: 6.7 scrub ok 2024-04-14T15:16:59.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:58 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2371781823' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi090-93363-59"}]': finished 2024-04-14T15:16:59.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:58 smithi092 ceph-mon[33966]: osdmap e351: 8 total, 8 up, 8 in 2024-04-14T15:16:59.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:58 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:16:59.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:58 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:16:59.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:58 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-45"}]: dispatch 2024-04-14T15:16:59.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:58 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-45"}]: dispatch 2024-04-14T15:16:59.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:59.394 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:59.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:16:59.495 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:16:59.522 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:59.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:59.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:16:59.625 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:59.653 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:59.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:59.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:59.754 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:16:59.782 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:16:59.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:59.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:16:59.884 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:16:59.919 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:16:59.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:16:59.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:17:00.022 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:17:00.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:00.047 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:00.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:17:00.148 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:17:00.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:00.173 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:00.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:17:00.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:59 smithi090 ceph-mon[41670]: 6.b scrub starts 2024-04-14T15:17:00.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:59 smithi090 ceph-mon[41670]: 6.b scrub ok 2024-04-14T15:17:00.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:59 smithi090 ceph-mon[41670]: pgmap v473: 238 pgs: 238 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-04-14T15:17:00.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:59 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-45"}]': finished 2024-04-14T15:17:00.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:59 smithi090 ceph-mon[41670]: osdmap e352: 8 total, 8 up, 8 in 2024-04-14T15:17:00.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:16:59 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/854835047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi090-93363-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:17:00.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:59 smithi090 ceph-mon[30399]: 6.b scrub starts 2024-04-14T15:17:00.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:59 smithi090 ceph-mon[30399]: 6.b scrub ok 2024-04-14T15:17:00.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:59 smithi090 ceph-mon[30399]: pgmap v473: 238 pgs: 238 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-04-14T15:17:00.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:59 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-45"}]': finished 2024-04-14T15:17:00.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:59 smithi090 ceph-mon[30399]: osdmap e352: 8 total, 8 up, 8 in 2024-04-14T15:17:00.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:16:59 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/854835047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi090-93363-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:17:00.274 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:17:00.298 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:00.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:00.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:17:00.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:59 smithi092 ceph-mon[33966]: 6.b scrub starts 2024-04-14T15:17:00.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:59 smithi092 ceph-mon[33966]: 6.b scrub ok 2024-04-14T15:17:00.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:59 smithi092 ceph-mon[33966]: pgmap v473: 238 pgs: 238 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-04-14T15:17:00.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:59 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-45"}]': finished 2024-04-14T15:17:00.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:59 smithi092 ceph-mon[33966]: osdmap e352: 8 total, 8 up, 8 in 2024-04-14T15:17:00.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:16:59 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/854835047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi090-93363-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:17:00.400 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:17:00.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:00.424 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:00.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:17:00.508 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:17:00.526 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:17:00.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:00.551 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:00.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:17:00.652 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:17:00.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:00.677 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:00.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:17:00.778 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:17:00.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:00.807 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:00.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:17:00.908 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:17:00.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:00.939 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:00.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:17:01.040 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:17:01.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:01.066 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:01.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:17:01.104 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-14T15:17:01.104 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-14T15:17:01.104 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-14T15:17:01.104 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-14T15:17:01.168 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:17:01.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:01.193 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:01.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:17:01.282 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-14T15:17:01.294 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:17:01.306 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:01 smithi090 ceph-mon[30399]: 2.11 deep-scrub starts 2024-04-14T15:17:01.307 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:01 smithi090 ceph-mon[30399]: 2.11 deep-scrub ok 2024-04-14T15:17:01.307 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:01 smithi090 ceph-mon[30399]: 5.1b scrub starts 2024-04-14T15:17:01.307 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:01 smithi090 ceph-mon[30399]: 5.1b scrub ok 2024-04-14T15:17:01.307 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:01 smithi090 ceph-mon[30399]: 5.14 scrub starts 2024-04-14T15:17:01.307 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:01 smithi090 ceph-mon[30399]: 5.14 scrub ok 2024-04-14T15:17:01.307 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:01 smithi090 ceph-mon[30399]: 5.12 scrub starts 2024-04-14T15:17:01.307 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:01 smithi090 ceph-mon[30399]: 5.12 scrub ok 2024-04-14T15:17:01.307 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:01 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/854835047' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi090-93363-60","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:17:01.307 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:01 smithi090 ceph-mon[30399]: osdmap e353: 8 total, 8 up, 8 in 2024-04-14T15:17:01.308 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:01 smithi090 ceph-mon[41670]: 2.11 deep-scrub starts 2024-04-14T15:17:01.308 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:01 smithi090 ceph-mon[41670]: 2.11 deep-scrub ok 2024-04-14T15:17:01.308 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:01 smithi090 ceph-mon[41670]: 5.1b scrub starts 2024-04-14T15:17:01.308 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:01 smithi090 ceph-mon[41670]: 5.1b scrub ok 2024-04-14T15:17:01.308 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:01 smithi090 ceph-mon[41670]: 5.14 scrub starts 2024-04-14T15:17:01.308 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:01 smithi090 ceph-mon[41670]: 5.14 scrub ok 2024-04-14T15:17:01.308 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:01 smithi090 ceph-mon[41670]: 5.12 scrub starts 2024-04-14T15:17:01.309 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:01 smithi090 ceph-mon[41670]: 5.12 scrub ok 2024-04-14T15:17:01.309 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:01 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/854835047' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi090-93363-60","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:17:01.309 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:01 smithi090 ceph-mon[41670]: osdmap e353: 8 total, 8 up, 8 in 2024-04-14T15:17:01.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:01.332 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:01.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:17:01.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:01 smithi092 ceph-mon[33966]: 2.11 deep-scrub starts 2024-04-14T15:17:01.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:01 smithi092 ceph-mon[33966]: 2.11 deep-scrub ok 2024-04-14T15:17:01.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:01 smithi092 ceph-mon[33966]: 5.1b scrub starts 2024-04-14T15:17:01.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:01 smithi092 ceph-mon[33966]: 5.1b scrub ok 2024-04-14T15:17:01.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:01 smithi092 ceph-mon[33966]: 5.14 scrub starts 2024-04-14T15:17:01.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:01 smithi092 ceph-mon[33966]: 5.14 scrub ok 2024-04-14T15:17:01.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:01 smithi092 ceph-mon[33966]: 5.12 scrub starts 2024-04-14T15:17:01.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:01 smithi092 ceph-mon[33966]: 5.12 scrub ok 2024-04-14T15:17:01.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:01 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/854835047' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi090-93363-60","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:17:01.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:01 smithi092 ceph-mon[33966]: osdmap e353: 8 total, 8 up, 8 in 2024-04-14T15:17:01.433 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:17:01.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:01.459 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:01.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:17:01.560 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:17:01.587 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:01.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:01.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:17:01.689 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:17:01.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:01.717 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:01.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:17:01.818 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:17:01.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:01.843 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:01.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:17:01.900 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:7462ddf6:::1.RoundTripAppendPP (2989 ms) 2024-04-14T15:17:01.900 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2024-04-14T15:17:01.900 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (3000 ms) 2024-04-14T15:17:01.900 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2024-04-14T15:17:01.901 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (3015 ms) 2024-04-14T15:17:01.901 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2024-04-14T15:17:01.901 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (2996 ms) 2024-04-14T15:17:01.901 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolEIOFlag 2024-04-14T15:17:01.901 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: setting pool EIO 2024-04-14T15:17:01.901 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: max_success 97, min_failed 98 2024-04-14T15:17:01.901 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolEIOFlag (4002 ms) 2024-04-14T15:17:01.901 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2024-04-14T15:17:01.901 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (2999 ms) 2024-04-14T15:17:01.901 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio (117705 ms total) 2024-04-14T15:17:01.901 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: 2024-04-14T15:17:01.901 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2024-04-14T15:17:01.901 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2024-04-14T15:17:01.902 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (3003 ms) 2024-04-14T15:17:01.902 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2024-04-14T15:17:01.902 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (9012 ms) 2024-04-14T15:17:01.902 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2024-04-14T15:17:01.902 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (14995 ms) 2024-04-14T15:17:01.902 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2024-04-14T15:17:01.902 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (3004 ms) 2024-04-14T15:17:01.902 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (30014 ms total) 2024-04-14T15:17:01.902 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: 2024-04-14T15:17:01.902 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2024-04-14T15:17:01.902 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-04-14T15:17:01.902 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (3013 ms) 2024-04-14T15:17:01.902 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (3013 ms total) 2024-04-14T15:17:01.902 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: 2024-04-14T15:17:01.903 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2024-04-14T15:17:01.903 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2024-04-14T15:17:01.903 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (13991 ms) 2024-04-14T15:17:01.903 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2024-04-14T15:17:01.903 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (7003 ms) 2024-04-14T15:17:01.903 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2024-04-14T15:17:01.903 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (7007 ms) 2024-04-14T15:17:01.903 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2024-04-14T15:17:01.903 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (6990 ms) 2024-04-14T15:17:01.903 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2024-04-14T15:17:01.903 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (3010 ms) 2024-04-14T15:17:01.903 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2024-04-14T15:17:01.903 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (6993 ms) 2024-04-14T15:17:01.903 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2024-04-14T15:17:01.904 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (7009 ms) 2024-04-14T15:17:01.904 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2024-04-14T15:17:01.904 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (7017 ms) 2024-04-14T15:17:01.904 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2024-04-14T15:17:01.904 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (7009 ms) 2024-04-14T15:17:01.904 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2024-04-14T15:17:01.904 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (6968 ms) 2024-04-14T15:17:01.904 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2024-04-14T15:17:01.904 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (7033 ms) 2024-04-14T15:17:01.904 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2024-04-14T15:17:01.904 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (6978 ms) 2024-04-14T15:17:01.904 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2024-04-14T15:17:01.904 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (7005 ms) 2024-04-14T15:17:01.904 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP2 2024-04-14T15:17:01.944 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:17:01.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:01.970 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:01.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:17:02.071 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:17:02.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:02.097 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:02.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:17:02.198 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:17:02.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:02.222 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:02.222 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:17:02.323 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:17:02.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:02.347 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:02.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:17:02.350 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:02 smithi090 ceph-mon[41670]: 11.d scrub starts 2024-04-14T15:17:02.350 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:02 smithi090 ceph-mon[41670]: 11.d scrub ok 2024-04-14T15:17:02.350 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:02 smithi090 ceph-mon[41670]: 2.1e scrub starts 2024-04-14T15:17:02.350 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:02 smithi090 ceph-mon[41670]: 2.1e scrub ok 2024-04-14T15:17:02.350 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:02 smithi090 ceph-mon[41670]: pgmap v476: 238 pgs: 32 unknown, 206 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:17:02.351 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:02 smithi090 ceph-mon[41670]: 5.1c scrub starts 2024-04-14T15:17:02.351 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:02 smithi090 ceph-mon[41670]: 5.1c scrub ok 2024-04-14T15:17:02.351 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:02 smithi090 ceph-mon[41670]: osdmap e354: 8 total, 8 up, 8 in 2024-04-14T15:17:02.351 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:02 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:17:02.351 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:02 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:17:02.351 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:02 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/784308691' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi090-93363-61"}]: dispatch 2024-04-14T15:17:02.351 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:02 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/784308691' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi090-93363-61"}]: dispatch 2024-04-14T15:17:02.351 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:02 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/784308691' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi090-93363-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:17:02.351 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:02 smithi090 ceph-mon[30399]: 11.d scrub starts 2024-04-14T15:17:02.351 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:02 smithi090 ceph-mon[30399]: 11.d scrub ok 2024-04-14T15:17:02.351 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:02 smithi090 ceph-mon[30399]: 2.1e scrub starts 2024-04-14T15:17:02.352 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:02 smithi090 ceph-mon[30399]: 2.1e scrub ok 2024-04-14T15:17:02.352 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:02 smithi090 ceph-mon[30399]: pgmap v476: 238 pgs: 32 unknown, 206 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:17:02.352 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:02 smithi090 ceph-mon[30399]: 5.1c scrub starts 2024-04-14T15:17:02.352 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:02 smithi090 ceph-mon[30399]: 5.1c scrub ok 2024-04-14T15:17:02.352 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:02 smithi090 ceph-mon[30399]: osdmap e354: 8 total, 8 up, 8 in 2024-04-14T15:17:02.352 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:02 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:17:02.352 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:02 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:17:02.352 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:02 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/784308691' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi090-93363-61"}]: dispatch 2024-04-14T15:17:02.352 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:02 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/784308691' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi090-93363-61"}]: dispatch 2024-04-14T15:17:02.352 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:02 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/784308691' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi090-93363-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:17:02.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:02 smithi092 ceph-mon[33966]: 11.d scrub starts 2024-04-14T15:17:02.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:02 smithi092 ceph-mon[33966]: 11.d scrub ok 2024-04-14T15:17:02.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:02 smithi092 ceph-mon[33966]: 2.1e scrub starts 2024-04-14T15:17:02.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:02 smithi092 ceph-mon[33966]: 2.1e scrub ok 2024-04-14T15:17:02.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:02 smithi092 ceph-mon[33966]: pgmap v476: 238 pgs: 32 unknown, 206 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:17:02.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:02 smithi092 ceph-mon[33966]: 5.1c scrub starts 2024-04-14T15:17:02.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:02 smithi092 ceph-mon[33966]: 5.1c scrub ok 2024-04-14T15:17:02.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:02 smithi092 ceph-mon[33966]: osdmap e354: 8 total, 8 up, 8 in 2024-04-14T15:17:02.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:02 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:17:02.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:02 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:17:02.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:02 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/784308691' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi090-93363-61"}]: dispatch 2024-04-14T15:17:02.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:02 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/784308691' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi090-93363-61"}]: dispatch 2024-04-14T15:17:02.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:02 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/784308691' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi090-93363-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:17:02.449 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:17:02.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:02.473 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:02.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:17:02.574 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:17:02.602 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:02.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:02.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:17:02.704 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:17:02.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:02.732 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:02.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:17:02.833 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:17:02.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:02.857 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:02.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:17:02.958 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:17:02.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:02.983 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:02.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:17:03.084 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:17:03.109 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:03.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:03.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:17:03.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:03 smithi090 ceph-mon[41670]: 2.17 scrub starts 2024-04-14T15:17:03.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:03 smithi090 ceph-mon[41670]: 2.17 scrub ok 2024-04-14T15:17:03.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:03 smithi090 ceph-mon[41670]: 5.6 scrub starts 2024-04-14T15:17:03.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:03 smithi090 ceph-mon[41670]: 5.6 scrub ok 2024-04-14T15:17:03.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:03 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:17:03.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:03 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/784308691' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi090-93363-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:17:03.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:03 smithi090 ceph-mon[41670]: osdmap e355: 8 total, 8 up, 8 in 2024-04-14T15:17:03.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:03 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/784308691' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi090-93363-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi090-93363-61"}]: dispatch 2024-04-14T15:17:03.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:03 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:17:03.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:03 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:17:03.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:03 smithi090 ceph-mon[30399]: 2.17 scrub starts 2024-04-14T15:17:03.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:03 smithi090 ceph-mon[30399]: 2.17 scrub ok 2024-04-14T15:17:03.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:03 smithi090 ceph-mon[30399]: 5.6 scrub starts 2024-04-14T15:17:03.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:03 smithi090 ceph-mon[30399]: 5.6 scrub ok 2024-04-14T15:17:03.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:03 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:17:03.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:03 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/784308691' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi090-93363-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:17:03.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:03 smithi090 ceph-mon[30399]: osdmap e355: 8 total, 8 up, 8 in 2024-04-14T15:17:03.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:03 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/784308691' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi090-93363-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi090-93363-61"}]: dispatch 2024-04-14T15:17:03.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:03 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:17:03.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:03 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:17:03.211 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:17:03.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:03.235 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:03.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:17:03.336 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:17:03.361 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:03.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:03.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:17:03.363 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-14T15:17:03.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:03 smithi092 ceph-mon[33966]: 2.17 scrub starts 2024-04-14T15:17:03.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:03 smithi092 ceph-mon[33966]: 2.17 scrub ok 2024-04-14T15:17:03.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:03 smithi092 ceph-mon[33966]: 5.6 scrub starts 2024-04-14T15:17:03.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:03 smithi092 ceph-mon[33966]: 5.6 scrub ok 2024-04-14T15:17:03.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:03 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:17:03.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:03 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/784308691' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi090-93363-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:17:03.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:03 smithi092 ceph-mon[33966]: osdmap e355: 8 total, 8 up, 8 in 2024-04-14T15:17:03.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:03 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/784308691' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi090-93363-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi090-93363-61"}]: dispatch 2024-04-14T15:17:03.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:03 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:17:03.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:03 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:17:03.463 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:17:03.492 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:03.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:03.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:17:03.594 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:17:03.620 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:03.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:03.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:17:03.722 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:17:03.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:03.748 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:03.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:17:03.849 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:17:03.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:03.874 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:03.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:17:03.975 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:17:03.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:04.000 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:04.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:17:04.101 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:17:04.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:04.128 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:04.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:17:04.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.066+0000 7f52e7368700 1 -- 172.21.15.90:0/1830573085 >> v1:172.21.15.90:6789/0 conn(0x7f52e0103aa0 legacy=0x7f52e0103e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:04.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.066+0000 7f52e7368700 1 -- 172.21.15.90:0/1830573085 shutdown_connections 2024-04-14T15:17:04.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.066+0000 7f52e7368700 1 -- 172.21.15.90:0/1830573085 >> 172.21.15.90:0/1830573085 conn(0x7f52e00fd2b0 msgr2=0x7f52e00ff6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:17:04.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.066+0000 7f52e7368700 1 -- 172.21.15.90:0/1830573085 shutdown_connections 2024-04-14T15:17:04.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.066+0000 7f52e7368700 1 -- 172.21.15.90:0/1830573085 wait complete. 2024-04-14T15:17:04.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.067+0000 7f52e7368700 1 librados: starting msgr at 2024-04-14T15:17:04.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.067+0000 7f52e7368700 1 librados: starting objecter 2024-04-14T15:17:04.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.067+0000 7f52e7368700 1 Processor -- start 2024-04-14T15:17:04.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.067+0000 7f52e7368700 1 -- start start 2024-04-14T15:17:04.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.067+0000 7f52e7368700 1 librados: setting wanted keys 2024-04-14T15:17:04.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.067+0000 7f52e7368700 1 librados: calling monclient init 2024-04-14T15:17:04.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.067+0000 7f52e7368700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f52e01033d0 con 0x7f52e0103aa0 2024-04-14T15:17:04.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.067+0000 7f52e7368700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f52e01035e0 con 0x7f52e010b8b0 2024-04-14T15:17:04.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.067+0000 7f52e7368700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f52e01ac300 con 0x7f52e010f3c0 2024-04-14T15:17:04.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.067+0000 7f52e5104700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f52e0103aa0 0x7f52e0101700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:44864/0 (socket says 172.21.15.90:44864) 2024-04-14T15:17:04.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.067+0000 7f52e5104700 1 -- 172.21.15.90:0/294050058 learned_addr learned my addr 172.21.15.90:0/294050058 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:17:04.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.067+0000 7f52d67fc700 10 client.?.objecter ms_handle_connect 0x7f52e0103aa0 2024-04-14T15:17:04.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.067+0000 7f52d67fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:17:04.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.067+0000 7f52d67fc700 10 client.?.objecter ms_handle_connect 0x7f52e010f3c0 2024-04-14T15:17:04.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.067+0000 7f52d67fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:17:04.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.067+0000 7f52d67fc700 10 client.?.objecter ms_handle_connect 0x7f52e010b8b0 2024-04-14T15:17:04.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.067+0000 7f52d67fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:17:04.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.068+0000 7f52d67fc700 1 -- 172.21.15.90:0/294050058 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1420296326 0 0) 0x7f52e01033d0 con 0x7f52e0103aa0 2024-04-14T15:17:04.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.068+0000 7f52d67fc700 1 -- 172.21.15.90:0/294050058 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f52cc0031f0 con 0x7f52e0103aa0 2024-04-14T15:17:04.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.068+0000 7f52d67fc700 1 -- 172.21.15.90:0/294050058 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2840857100 0 0) 0x7f52e01ac300 con 0x7f52e010f3c0 2024-04-14T15:17:04.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.068+0000 7f52d67fc700 1 -- 172.21.15.90:0/294050058 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f52e01033d0 con 0x7f52e010f3c0 2024-04-14T15:17:04.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.068+0000 7f52d67fc700 1 -- 172.21.15.90:0/294050058 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3112367286 0 0) 0x7f52e01035e0 con 0x7f52e010b8b0 2024-04-14T15:17:04.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.068+0000 7f52d67fc700 1 -- 172.21.15.90:0/294050058 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f52e01ac300 con 0x7f52e010b8b0 2024-04-14T15:17:04.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.068+0000 7f52d67fc700 1 -- 172.21.15.90:0/294050058 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 665422347 0 0) 0x7f52e01033d0 con 0x7f52e010f3c0 2024-04-14T15:17:04.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.068+0000 7f52d67fc700 1 -- 172.21.15.90:0/294050058 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f52e01035e0 con 0x7f52e010f3c0 2024-04-14T15:17:04.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.068+0000 7f52d67fc700 1 -- 172.21.15.90:0/294050058 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2187989100 0 0) 0x7f52e01ac300 con 0x7f52e010b8b0 2024-04-14T15:17:04.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.068+0000 7f52d67fc700 1 -- 172.21.15.90:0/294050058 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f52e01033d0 con 0x7f52e010b8b0 2024-04-14T15:17:04.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.068+0000 7f52d67fc700 1 -- 172.21.15.90:0/294050058 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f52d8001c30 con 0x7f52e010f3c0 2024-04-14T15:17:04.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.068+0000 7f52d67fc700 1 -- 172.21.15.90:0/294050058 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 562742770 0 0) 0x7f52cc0031f0 con 0x7f52e0103aa0 2024-04-14T15:17:04.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.068+0000 7f52d67fc700 1 -- 172.21.15.90:0/294050058 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f52e01ac300 con 0x7f52e0103aa0 2024-04-14T15:17:04.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.068+0000 7f52d67fc700 1 -- 172.21.15.90:0/294050058 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f52d00018b0 con 0x7f52e010b8b0 2024-04-14T15:17:04.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.068+0000 7f52d67fc700 1 -- 172.21.15.90:0/294050058 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f52dc0015d0 con 0x7f52e0103aa0 2024-04-14T15:17:04.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.068+0000 7f52d67fc700 1 -- 172.21.15.90:0/294050058 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2962348140 0 0) 0x7f52e01033d0 con 0x7f52e010b8b0 2024-04-14T15:17:04.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.068+0000 7f52d67fc700 1 -- 172.21.15.90:0/294050058 >> v1:172.21.15.90:6790/0 conn(0x7f52e010f3c0 legacy=0x7f52e01a8bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:04.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.068+0000 7f52d67fc700 1 -- 172.21.15.90:0/294050058 >> v1:172.21.15.90:6789/0 conn(0x7f52e0103aa0 legacy=0x7f52e0101700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:04.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.068+0000 7f52d67fc700 1 -- 172.21.15.90:0/294050058 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52e01ad520 con 0x7f52e010b8b0 2024-04-14T15:17:04.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.068+0000 7f52e7368700 1 -- 172.21.15.90:0/294050058 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f52e01ac530 con 0x7f52e010b8b0 2024-04-14T15:17:04.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.068+0000 7f52e7368700 10 client.48955.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:17:04.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.068+0000 7f52e7368700 1 -- 172.21.15.90:0/294050058 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f52e01ac9f0 con 0x7f52e010b8b0 2024-04-14T15:17:04.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.068+0000 7f52e7368700 1 librados: init done 2024-04-14T15:17:04.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.068+0000 7f52d67fc700 1 -- 172.21.15.90:0/294050058 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f52d0003b40 con 0x7f52e010b8b0 2024-04-14T15:17:04.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.068+0000 7f52d67fc700 1 -- 172.21.15.90:0/294050058 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f52d0004930 con 0x7f52e010b8b0 2024-04-14T15:17:04.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.069+0000 7f52e7368700 1 -- 172.21.15.90:0/294050058 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52c4005120 con 0x7f52e010b8b0 2024-04-14T15:17:04.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.070+0000 7f52d67fc700 1 -- 172.21.15.90:0/294050058 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f52d001b320 con 0x7f52e010b8b0 2024-04-14T15:17:04.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.070+0000 7f52d67fc700 1 -- 172.21.15.90:0/294050058 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(356..356 src has 51..356) ==== 8586+0+0 (unknown 758428308 0 0) 0x7f52d0089800 con 0x7f52e010b8b0 2024-04-14T15:17:04.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.070+0000 7f52d67fc700 10 client.48955.objecter ms_dispatch 0x7f52e0001550 osd_map(356..356 src has 51..356) 2024-04-14T15:17:04.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.070+0000 7f52d67fc700 3 client.48955.objecter handle_osd_map got epochs [356,356] > 0 2024-04-14T15:17:04.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.070+0000 7f52d67fc700 3 client.48955.objecter handle_osd_map decoding full epoch 356 2024-04-14T15:17:04.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.071+0000 7f52d67fc700 20 client.48955.objecter dump_active .. 0 homeless 2024-04-14T15:17:04.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.071+0000 7f52d67fc700 10 client.48955.objecter ms_handle_connect 0x7f52cc0713d0 2024-04-14T15:17:04.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.075+0000 7f52d67fc700 1 -- 172.21.15.90:0/294050058 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f52d0056060 con 0x7f52e010b8b0 2024-04-14T15:17:04.173 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:17:03 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:17:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:17:04.173 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:04 smithi090 ceph-mon[41670]: 2.13 scrub starts 2024-04-14T15:17:04.173 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:04 smithi090 ceph-mon[41670]: 2.13 scrub ok 2024-04-14T15:17:04.173 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:04 smithi090 ceph-mon[41670]: 2.1c scrub starts 2024-04-14T15:17:04.173 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:04 smithi090 ceph-mon[41670]: 2.1c scrub ok 2024-04-14T15:17:04.173 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:04 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:17:04.173 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:04 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:17:04.173 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:04 smithi090 ceph-mon[41670]: pgmap v479: 238 pgs: 32 unknown, 206 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:17:04.173 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:04 smithi090 ceph-mon[41670]: 6.4 deep-scrub starts 2024-04-14T15:17:04.174 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:04 smithi090 ceph-mon[41670]: 6.4 deep-scrub ok 2024-04-14T15:17:04.174 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:04 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-47", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:17:04.174 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:04 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-47"}]: dispatch 2024-04-14T15:17:04.174 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:04 smithi090 ceph-mon[41670]: osdmap e356: 8 total, 8 up, 8 in 2024-04-14T15:17:04.174 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:04 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-47"}]: dispatch 2024-04-14T15:17:04.174 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:04 smithi090 ceph-mon[30399]: 2.13 scrub starts 2024-04-14T15:17:04.174 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:04 smithi090 ceph-mon[30399]: 2.13 scrub ok 2024-04-14T15:17:04.174 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:04 smithi090 ceph-mon[30399]: 2.1c scrub starts 2024-04-14T15:17:04.174 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:04 smithi090 ceph-mon[30399]: 2.1c scrub ok 2024-04-14T15:17:04.174 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:04 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:17:04.174 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:04 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:17:04.175 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:04 smithi090 ceph-mon[30399]: pgmap v479: 238 pgs: 32 unknown, 206 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:17:04.175 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:04 smithi090 ceph-mon[30399]: 6.4 deep-scrub starts 2024-04-14T15:17:04.175 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:04 smithi090 ceph-mon[30399]: 6.4 deep-scrub ok 2024-04-14T15:17:04.175 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:04 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-47", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:17:04.175 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:04 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-47"}]: dispatch 2024-04-14T15:17:04.175 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:04 smithi090 ceph-mon[30399]: osdmap e356: 8 total, 8 up, 8 in 2024-04-14T15:17:04.175 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:04 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-47"}]: dispatch 2024-04-14T15:17:04.229 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:17:04.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:04.256 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:04.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:17:04.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.175+0000 7fe4fbc78700 1 -- 172.21.15.90:0/1405556448 >> v1:172.21.15.90:6790/0 conn(0x7fe4f4107760 legacy=0x7fe4f4107b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:04.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.175+0000 7fe4fbc78700 1 -- 172.21.15.90:0/1405556448 shutdown_connections 2024-04-14T15:17:04.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.175+0000 7fe4fbc78700 1 -- 172.21.15.90:0/1405556448 >> 172.21.15.90:0/1405556448 conn(0x7fe4f40fd290 msgr2=0x7fe4f40ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:17:04.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.175+0000 7fe4fbc78700 1 -- 172.21.15.90:0/1405556448 shutdown_connections 2024-04-14T15:17:04.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.175+0000 7fe4fbc78700 1 -- 172.21.15.90:0/1405556448 wait complete. 2024-04-14T15:17:04.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.176+0000 7fe4fbc78700 1 librados: starting msgr at 2024-04-14T15:17:04.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.176+0000 7fe4fbc78700 1 librados: starting objecter 2024-04-14T15:17:04.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.176+0000 7fe4fbc78700 1 Processor -- start 2024-04-14T15:17:04.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.176+0000 7fe4fbc78700 1 -- start start 2024-04-14T15:17:04.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.176+0000 7fe4fbc78700 1 librados: setting wanted keys 2024-04-14T15:17:04.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.176+0000 7fe4fbc78700 1 librados: calling monclient init 2024-04-14T15:17:04.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.176+0000 7fe4fbc78700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe4f41a9c40 con 0x7fe4f410e1e0 2024-04-14T15:17:04.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.176+0000 7fe4fbc78700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe4f41aae60 con 0x7fe4f410a610 2024-04-14T15:17:04.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.176+0000 7fe4fbc78700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe4f41ac0a0 con 0x7fe4f4107760 2024-04-14T15:17:04.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.176+0000 7fe4fa215700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fe4f410e1e0 0x7fe4f41a8300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:44872/0 (socket says 172.21.15.90:44872) 2024-04-14T15:17:04.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.176+0000 7fe4fa215700 1 -- 172.21.15.90:0/2621464513 learned_addr learned my addr 172.21.15.90:0/2621464513 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:17:04.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.176+0000 7fe4eaffd700 10 client.?.objecter ms_handle_connect 0x7fe4f410e1e0 2024-04-14T15:17:04.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.176+0000 7fe4eaffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:17:04.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.176+0000 7fe4eaffd700 10 client.?.objecter ms_handle_connect 0x7fe4f4107760 2024-04-14T15:17:04.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.176+0000 7fe4eaffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:17:04.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.176+0000 7fe4eaffd700 10 client.?.objecter ms_handle_connect 0x7fe4f410a610 2024-04-14T15:17:04.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.176+0000 7fe4eaffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:17:04.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.176+0000 7fe4eaffd700 1 -- 172.21.15.90:0/2621464513 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3075927768 0 0) 0x7fe4f41a9c40 con 0x7fe4f410e1e0 2024-04-14T15:17:04.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.176+0000 7fe4eaffd700 1 -- 172.21.15.90:0/2621464513 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe4e00031f0 con 0x7fe4f410e1e0 2024-04-14T15:17:04.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.176+0000 7fe4eaffd700 1 -- 172.21.15.90:0/2621464513 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3032714395 0 0) 0x7fe4f41ac0a0 con 0x7fe4f4107760 2024-04-14T15:17:04.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.176+0000 7fe4eaffd700 1 -- 172.21.15.90:0/2621464513 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe4f41a9c40 con 0x7fe4f4107760 2024-04-14T15:17:04.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.176+0000 7fe4eaffd700 1 -- 172.21.15.90:0/2621464513 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2550073595 0 0) 0x7fe4f41aae60 con 0x7fe4f410a610 2024-04-14T15:17:04.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.176+0000 7fe4eaffd700 1 -- 172.21.15.90:0/2621464513 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe4f41ac0a0 con 0x7fe4f410a610 2024-04-14T15:17:04.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.177+0000 7fe4eaffd700 1 -- 172.21.15.90:0/2621464513 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3064756005 0 0) 0x7fe4e00031f0 con 0x7fe4f410e1e0 2024-04-14T15:17:04.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.177+0000 7fe4eaffd700 1 -- 172.21.15.90:0/2621464513 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe4f41aae60 con 0x7fe4f410e1e0 2024-04-14T15:17:04.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.177+0000 7fe4eaffd700 1 -- 172.21.15.90:0/2621464513 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fe4ec001c70 con 0x7fe4f410e1e0 2024-04-14T15:17:04.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.177+0000 7fe4eaffd700 1 -- 172.21.15.90:0/2621464513 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1761765532 0 0) 0x7fe4f41a9c40 con 0x7fe4f4107760 2024-04-14T15:17:04.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.177+0000 7fe4eaffd700 1 -- 172.21.15.90:0/2621464513 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe4e00031f0 con 0x7fe4f4107760 2024-04-14T15:17:04.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.177+0000 7fe4eaffd700 1 -- 172.21.15.90:0/2621464513 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fe4f0004440 con 0x7fe4f4107760 2024-04-14T15:17:04.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.177+0000 7fe4eaffd700 1 -- 172.21.15.90:0/2621464513 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3590076200 0 0) 0x7fe4f41ac0a0 con 0x7fe4f410a610 2024-04-14T15:17:04.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.177+0000 7fe4eaffd700 1 -- 172.21.15.90:0/2621464513 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe4f41a9c40 con 0x7fe4f410a610 2024-04-14T15:17:04.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.177+0000 7fe4eaffd700 1 -- 172.21.15.90:0/2621464513 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fe4e4001810 con 0x7fe4f410a610 2024-04-14T15:17:04.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.177+0000 7fe4eaffd700 1 -- 172.21.15.90:0/2621464513 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1336387721 0 0) 0x7fe4f41aae60 con 0x7fe4f410e1e0 2024-04-14T15:17:04.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.177+0000 7fe4eaffd700 1 -- 172.21.15.90:0/2621464513 >> v1:172.21.15.90:6790/0 conn(0x7fe4f4107760 legacy=0x7fe4f4110180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:04.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.177+0000 7fe4eaffd700 1 -- 172.21.15.90:0/2621464513 >> v1:172.21.15.92:6789/0 conn(0x7fe4f410a610 legacy=0x7fe4f41a4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:04.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.177+0000 7fe4eaffd700 1 -- 172.21.15.90:0/2621464513 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4f41ad2e0 con 0x7fe4f410e1e0 2024-04-14T15:17:04.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.177+0000 7fe4fbc78700 1 -- 172.21.15.90:0/2621464513 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe4f41ac2d0 con 0x7fe4f410e1e0 2024-04-14T15:17:04.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.177+0000 7fe4fbc78700 10 client.37122.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:17:04.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.177+0000 7fe4fbc78700 1 -- 172.21.15.90:0/2621464513 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe4f41ac790 con 0x7fe4f410e1e0 2024-04-14T15:17:04.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.177+0000 7fe4fbc78700 1 librados: init done 2024-04-14T15:17:04.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.177+0000 7fe4eaffd700 1 -- 172.21.15.90:0/2621464513 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fe4ec002040 con 0x7fe4f410e1e0 2024-04-14T15:17:04.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.177+0000 7fe4eaffd700 1 -- 172.21.15.90:0/2621464513 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fe4ec006120 con 0x7fe4f410e1e0 2024-04-14T15:17:04.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.178+0000 7fe4fbc78700 1 -- 172.21.15.90:0/2621464513 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4d8005120 con 0x7fe4f410e1e0 2024-04-14T15:17:04.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.179+0000 7fe4eaffd700 1 -- 172.21.15.90:0/2621464513 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fe4ec0044e0 con 0x7fe4f410e1e0 2024-04-14T15:17:04.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.180+0000 7fe4eaffd700 1 -- 172.21.15.90:0/2621464513 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(356..356 src has 51..356) ==== 8586+0+0 (unknown 758428308 0 0) 0x7fe4ec08af80 con 0x7fe4f410e1e0 2024-04-14T15:17:04.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.180+0000 7fe4eaffd700 10 client.37122.objecter ms_dispatch 0x7fe4f4001550 osd_map(356..356 src has 51..356) 2024-04-14T15:17:04.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.180+0000 7fe4eaffd700 3 client.37122.objecter handle_osd_map got epochs [356,356] > 0 2024-04-14T15:17:04.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.180+0000 7fe4eaffd700 3 client.37122.objecter handle_osd_map decoding full epoch 356 2024-04-14T15:17:04.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.180+0000 7fe4eaffd700 20 client.37122.objecter dump_active .. 0 homeless 2024-04-14T15:17:04.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.180+0000 7fe4eaffd700 10 client.37122.objecter ms_handle_connect 0x7fe4e0071340 2024-04-14T15:17:04.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.184+0000 7fe4eaffd700 1 -- 172.21.15.90:0/2621464513 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fe4ec004850 con 0x7fe4f410e1e0 2024-04-14T15:17:04.293 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:17:04.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.306+0000 7f52e7368700 1 -- 172.21.15.90:0/294050058 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd reweight", "id": 1, "weight": 1.0} v 0) -- 0x7f52c4005410 con 0x7f52e010b8b0 2024-04-14T15:17:04.332 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:17:04.357 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:17:04.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:04 smithi092 ceph-mon[33966]: 2.13 scrub starts 2024-04-14T15:17:04.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:04 smithi092 ceph-mon[33966]: 2.13 scrub ok 2024-04-14T15:17:04.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:04 smithi092 ceph-mon[33966]: 2.1c scrub starts 2024-04-14T15:17:04.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:04 smithi092 ceph-mon[33966]: 2.1c scrub ok 2024-04-14T15:17:04.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:04 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:17:04.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:04 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:17:04.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:04 smithi092 ceph-mon[33966]: pgmap v479: 238 pgs: 32 unknown, 206 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:17:04.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:04 smithi092 ceph-mon[33966]: 6.4 deep-scrub starts 2024-04-14T15:17:04.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:04 smithi092 ceph-mon[33966]: 6.4 deep-scrub ok 2024-04-14T15:17:04.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:04 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-47", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:17:04.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:04 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-47"}]: dispatch 2024-04-14T15:17:04.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:04 smithi092 ceph-mon[33966]: osdmap e356: 8 total, 8 up, 8 in 2024-04-14T15:17:04.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:04 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-47"}]: dispatch 2024-04-14T15:17:04.380 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:04.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:04.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:17:04.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.420+0000 7fe4fbc78700 1 -- 172.21.15.90:0/2621464513 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7fe4d8005410 con 0x7fe4f410e1e0 2024-04-14T15:17:04.482 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:17:04.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:04.508 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:04.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:17:04.609 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:17:04.635 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:04.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:04.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:17:04.737 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:17:04.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:04.761 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:04.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:17:04.863 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:17:04.886 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:04.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:04.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:17:04.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.894+0000 7fe4eaffd700 1 -- 172.21.15.90:0/2621464513 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v357) ==== 87+0+0 (unknown 1186223263 0 0) 0x7fe4ec057800 con 0x7fe4f410e1e0 2024-04-14T15:17:04.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.896+0000 7fe4fbc78700 10 librados: watch_flush enter 2024-04-14T15:17:04.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.896+0000 7fe4fbc78700 10 librados: watch_flush exit 2024-04-14T15:17:04.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.896+0000 7fe4fbc78700 20 client.37122.objecter shutdown clearing up homeless session... 2024-04-14T15:17:04.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.896+0000 7fe4fbc78700 10 client.37122.objecter successfully canceled tick 2024-04-14T15:17:04.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.896+0000 7fe4fbc78700 1 -- 172.21.15.90:0/2621464513 >> v1:172.21.15.90:6800/420935427 conn(0x7fe4e0071340 legacy=0x7fe4e0073800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:04.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.896+0000 7fe4fbc78700 1 -- 172.21.15.90:0/2621464513 >> v1:172.21.15.90:6789/0 conn(0x7fe4f410e1e0 legacy=0x7fe4f41a8300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:04.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.897+0000 7fe4fbc78700 1 -- 172.21.15.90:0/2621464513 shutdown_connections 2024-04-14T15:17:04.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.897+0000 7fe4fbc78700 1 -- 172.21.15.90:0/2621464513 >> 172.21.15.90:0/2621464513 conn(0x7fe4f40fd290 msgr2=0x7fe4f4100540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:17:04.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.898+0000 7fe4fbc78700 1 -- 172.21.15.90:0/2621464513 shutdown_connections 2024-04-14T15:17:04.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.898+0000 7fe4fbc78700 1 -- 172.21.15.90:0/2621464513 wait complete. 2024-04-14T15:17:04.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.899+0000 7fe4fbc78700 1 librados: shutdown 2024-04-14T15:17:04.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.902+0000 7f52d67fc700 1 -- 172.21.15.90:0/294050058 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 1, "weight": 1.0}]=0 reweighted osd.1 to 1 (10000) v357) ==== 113+0+0 (unknown 625876991 0 0) 0x7f52d0059d10 con 0x7f52e010b8b0 2024-04-14T15:17:04.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.905+0000 7f52e7368700 10 librados: watch_flush enter 2024-04-14T15:17:04.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.905+0000 7f52e7368700 10 librados: watch_flush exit 2024-04-14T15:17:04.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.905+0000 7f52e7368700 20 client.48955.objecter shutdown clearing up homeless session... 2024-04-14T15:17:04.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.905+0000 7f52e7368700 10 client.48955.objecter successfully canceled tick 2024-04-14T15:17:04.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.906+0000 7f52e7368700 1 -- 172.21.15.90:0/294050058 >> v1:172.21.15.90:6800/420935427 conn(0x7f52cc0713d0 legacy=0x7f52cc073890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:04.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.906+0000 7f52e7368700 1 -- 172.21.15.90:0/294050058 >> v1:172.21.15.92:6789/0 conn(0x7f52e010b8b0 legacy=0x7f52e0101e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:04.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.906+0000 7f52e7368700 1 -- 172.21.15.90:0/294050058 shutdown_connections 2024-04-14T15:17:04.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.906+0000 7f52e7368700 1 -- 172.21.15.90:0/294050058 >> 172.21.15.90:0/294050058 conn(0x7f52e00fd2b0 msgr2=0x7f52e00ff6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:17:04.907 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is set 2024-04-14T15:17:04.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.909+0000 7f52e7368700 1 -- 172.21.15.90:0/294050058 shutdown_connections 2024-04-14T15:17:04.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.909+0000 7f52e7368700 1 -- 172.21.15.90:0/294050058 wait complete. 2024-04-14T15:17:04.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:04.909+0000 7f52e7368700 1 librados: shutdown 2024-04-14T15:17:04.910 INFO:teuthology.orchestra.run.smithi090.stderr:reweighted osd.1 to 1 (10000) 2024-04-14T15:17:04.989 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:17:05.016 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:05.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:05.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:17:05.018 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:05 smithi090 ceph-mon[41670]: 11.1f deep-scrub starts 2024-04-14T15:17:05.018 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:05 smithi090 ceph-mon[41670]: 11.1f deep-scrub ok 2024-04-14T15:17:05.018 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:05 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/294050058' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-14T15:17:05.018 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:05 smithi090 ceph-mon[41670]: from='client.48955 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-14T15:17:05.018 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:05 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2621464513' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:17:05.018 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:05 smithi090 ceph-mon[30399]: 11.1f deep-scrub starts 2024-04-14T15:17:05.019 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:05 smithi090 ceph-mon[30399]: 11.1f deep-scrub ok 2024-04-14T15:17:05.019 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:05 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/294050058' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-14T15:17:05.019 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:05 smithi090 ceph-mon[30399]: from='client.48955 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-14T15:17:05.019 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:05 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2621464513' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:17:05.019 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:05 smithi090 ceph-mon[30399]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:17:05.019 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:05 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/784308691' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi090-93363-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi090-93363-61"}]': finished 2024-04-14T15:17:05.019 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:05 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-47"}]': finished 2024-04-14T15:17:05.019 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:05 smithi090 ceph-mon[30399]: from='client.48955 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-14T15:17:05.019 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:05 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2621464513' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-14T15:17:05.019 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:05 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-47", "mode": "writeback"}]: dispatch 2024-04-14T15:17:05.021 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:05 smithi090 ceph-mon[30399]: osdmap e357: 8 total, 8 up, 8 in 2024-04-14T15:17:05.021 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:05 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-47", "mode": "writeback"}]: dispatch 2024-04-14T15:17:05.021 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:05 smithi090 ceph-mon[41670]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:17:05.021 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:05 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/784308691' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi090-93363-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi090-93363-61"}]': finished 2024-04-14T15:17:05.021 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:05 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-47"}]': finished 2024-04-14T15:17:05.021 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:05 smithi090 ceph-mon[41670]: from='client.48955 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-14T15:17:05.021 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:05 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2621464513' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-14T15:17:05.021 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:05 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-47", "mode": "writeback"}]: dispatch 2024-04-14T15:17:05.021 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:05 smithi090 ceph-mon[41670]: osdmap e357: 8 total, 8 up, 8 in 2024-04-14T15:17:05.021 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:05 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-47", "mode": "writeback"}]: dispatch 2024-04-14T15:17:05.118 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:17:05.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:05.143 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:05.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:17:05.244 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:17:05.270 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:05.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:05.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:17:05.372 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:17:05.373 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:17:05 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:17:05.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:05 smithi092 ceph-mon[33966]: 11.1f deep-scrub starts 2024-04-14T15:17:05.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:05 smithi092 ceph-mon[33966]: 11.1f deep-scrub ok 2024-04-14T15:17:05.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:05 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/294050058' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-14T15:17:05.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:05 smithi092 ceph-mon[33966]: from='client.48955 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-14T15:17:05.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:05 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2621464513' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:17:05.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:05 smithi092 ceph-mon[33966]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:17:05.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:05 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/784308691' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi090-93363-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi090-93363-61"}]': finished 2024-04-14T15:17:05.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:05 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-47"}]': finished 2024-04-14T15:17:05.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:05 smithi092 ceph-mon[33966]: from='client.48955 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-14T15:17:05.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:05 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2621464513' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-14T15:17:05.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:05 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-47", "mode": "writeback"}]: dispatch 2024-04-14T15:17:05.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:05 smithi092 ceph-mon[33966]: osdmap e357: 8 total, 8 up, 8 in 2024-04-14T15:17:05.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:05 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-47", "mode": "writeback"}]: dispatch 2024-04-14T15:17:05.399 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:05.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:05.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:17:05.501 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:17:05.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:05.528 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:05.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:17:05.629 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:17:05.654 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:05.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:05.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:17:05.756 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:17:05.783 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:05.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:05.784 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:17:05.885 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:17:05.910 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:05.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:05.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:17:06.012 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:17:06.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:06.038 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:06.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:17:06.139 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:17:06.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:06.164 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:06.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:17:06.265 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:17:06.289 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:06.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:06.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:17:06.296 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:06 smithi090 ceph-mon[41670]: 2.14 scrub starts 2024-04-14T15:17:06.296 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:06 smithi090 ceph-mon[41670]: 2.14 scrub ok 2024-04-14T15:17:06.296 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:06 smithi090 ceph-mon[41670]: 5.3 scrub starts 2024-04-14T15:17:06.296 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:06 smithi090 ceph-mon[41670]: 5.3 scrub ok 2024-04-14T15:17:06.296 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:06 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:17:06.296 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:06 smithi090 ceph-mon[41670]: pgmap v482: 246 pgs: 8 unknown, 238 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-04-14T15:17:06.296 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:06 smithi090 ceph-mon[41670]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:17:06.296 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:06 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-47", "mode": "writeback"}]': finished 2024-04-14T15:17:06.296 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:06 smithi090 ceph-mon[41670]: osdmap e358: 8 total, 8 up, 8 in 2024-04-14T15:17:06.297 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:06 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-47","var": "hit_set_count","val": "2"}]: dispatch 2024-04-14T15:17:06.297 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:06 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-47","var": "hit_set_count","val": "2"}]: dispatch 2024-04-14T15:17:06.297 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:06 smithi090 ceph-mon[30399]: 2.14 scrub starts 2024-04-14T15:17:06.297 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:06 smithi090 ceph-mon[30399]: 2.14 scrub ok 2024-04-14T15:17:06.297 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:06 smithi090 ceph-mon[30399]: 5.3 scrub starts 2024-04-14T15:17:06.297 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:06 smithi090 ceph-mon[30399]: 5.3 scrub ok 2024-04-14T15:17:06.297 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:06 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:17:06.297 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:06 smithi090 ceph-mon[30399]: pgmap v482: 246 pgs: 8 unknown, 238 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-04-14T15:17:06.298 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:06 smithi090 ceph-mon[30399]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:17:06.298 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:06 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-47", "mode": "writeback"}]': finished 2024-04-14T15:17:06.298 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:06 smithi090 ceph-mon[30399]: osdmap e358: 8 total, 8 up, 8 in 2024-04-14T15:17:06.298 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:06 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-47","var": "hit_set_count","val": "2"}]: dispatch 2024-04-14T15:17:06.298 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:06 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-47","var": "hit_set_count","val": "2"}]: dispatch 2024-04-14T15:17:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:06 smithi092 ceph-mon[33966]: 2.14 scrub starts 2024-04-14T15:17:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:06 smithi092 ceph-mon[33966]: 2.14 scrub ok 2024-04-14T15:17:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:06 smithi092 ceph-mon[33966]: 5.3 scrub starts 2024-04-14T15:17:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:06 smithi092 ceph-mon[33966]: 5.3 scrub ok 2024-04-14T15:17:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:06 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:17:06.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:06 smithi092 ceph-mon[33966]: pgmap v482: 246 pgs: 8 unknown, 238 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-04-14T15:17:06.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:06 smithi092 ceph-mon[33966]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:17:06.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:06 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-47", "mode": "writeback"}]': finished 2024-04-14T15:17:06.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:06 smithi092 ceph-mon[33966]: osdmap e358: 8 total, 8 up, 8 in 2024-04-14T15:17:06.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:06 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-47","var": "hit_set_count","val": "2"}]: dispatch 2024-04-14T15:17:06.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:06 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-47","var": "hit_set_count","val": "2"}]: dispatch 2024-04-14T15:17:06.391 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:17:06.415 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:06.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:06.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:17:06.517 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:17:06.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:06.543 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:06.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:17:06.644 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:17:06.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:06.672 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:06.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:17:06.773 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:17:06.800 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:06.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:06.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:17:06.902 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:17:06.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:06.931 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:06.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:17:07.032 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:17:07.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:07.060 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:07.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:17:07.161 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:17:07.186 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:07.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:07.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:17:07.288 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:17:07.314 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:07.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:07.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:17:07.350 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:07 smithi090 ceph-mon[41670]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:17:07.351 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:07 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-47","var": "hit_set_count","val": "2"}]': finished 2024-04-14T15:17:07.351 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:07 smithi090 ceph-mon[41670]: osdmap e359: 8 total, 8 up, 8 in 2024-04-14T15:17:07.351 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:07 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-47","var": "hit_set_period","val": "600"}]: dispatch 2024-04-14T15:17:07.351 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:07 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-47","var": "hit_set_period","val": "600"}]: dispatch 2024-04-14T15:17:07.351 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:07 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/784308691' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi090-93363-61"}]: dispatch 2024-04-14T15:17:07.351 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:07 smithi090 ceph-mon[30399]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:17:07.351 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:07 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-47","var": "hit_set_count","val": "2"}]': finished 2024-04-14T15:17:07.351 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:07 smithi090 ceph-mon[30399]: osdmap e359: 8 total, 8 up, 8 in 2024-04-14T15:17:07.351 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:07 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-47","var": "hit_set_period","val": "600"}]: dispatch 2024-04-14T15:17:07.351 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:07 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-47","var": "hit_set_period","val": "600"}]: dispatch 2024-04-14T15:17:07.351 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:07 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/784308691' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi090-93363-61"}]: dispatch 2024-04-14T15:17:07.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:07 smithi092 ceph-mon[33966]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:17:07.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:07 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-47","var": "hit_set_count","val": "2"}]': finished 2024-04-14T15:17:07.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:07 smithi092 ceph-mon[33966]: osdmap e359: 8 total, 8 up, 8 in 2024-04-14T15:17:07.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:07 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-47","var": "hit_set_period","val": "600"}]: dispatch 2024-04-14T15:17:07.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:07 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-47","var": "hit_set_period","val": "600"}]: dispatch 2024-04-14T15:17:07.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:07 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/784308691' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi090-93363-61"}]: dispatch 2024-04-14T15:17:07.416 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:17:07.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:07.440 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:07.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:17:07.541 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:17:07.565 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:07.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:07.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:17:07.667 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:17:07.694 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:07.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:07.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:17:07.796 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:17:07.820 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:07.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:07.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:17:07.864 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd reweight 2 1 2024-04-14T15:17:07.921 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:17:07.951 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:07.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:07.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:17:08.052 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:17:08.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:08.077 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:08.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:17:08.178 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:17:08.205 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:08.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:08.206 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:17:08.307 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:17:08.333 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:08.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:08.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:17:08.434 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:17:08.459 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:08.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:08.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:17:08.562 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:17:08.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:08.590 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:08.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:17:08.691 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:17:08.716 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:08.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:08.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:17:08.818 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:17:08.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:08.846 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:08.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:17:08.947 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:17:08.973 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:08.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:08.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:17:09.075 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:17:09.099 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:09.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:09.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:17:09.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:08 smithi090 ceph-mon[41670]: pgmap v485: 238 pgs: 238 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-04-14T15:17:09.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:08 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-47","var": "hit_set_period","val": "600"}]': finished 2024-04-14T15:17:09.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:08 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/784308691' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi090-93363-61"}]': finished 2024-04-14T15:17:09.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:08 smithi090 ceph-mon[41670]: osdmap e360: 8 total, 8 up, 8 in 2024-04-14T15:17:09.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:08 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-14T15:17:09.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:08 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/784308691' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi090-93363-61"}]: dispatch 2024-04-14T15:17:09.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:08 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-14T15:17:09.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:08 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:17:09.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:08 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:17:09.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:08 smithi090 ceph-mon[30399]: pgmap v485: 238 pgs: 238 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-04-14T15:17:09.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:08 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-47","var": "hit_set_period","val": "600"}]': finished 2024-04-14T15:17:09.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:08 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/784308691' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi090-93363-61"}]': finished 2024-04-14T15:17:09.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:08 smithi090 ceph-mon[30399]: osdmap e360: 8 total, 8 up, 8 in 2024-04-14T15:17:09.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:08 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-14T15:17:09.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:08 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/784308691' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi090-93363-61"}]: dispatch 2024-04-14T15:17:09.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:08 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-14T15:17:09.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:08 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:17:09.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:08 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:17:09.201 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:17:09.226 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:09.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:09.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:17:09.328 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:17:09.353 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:09.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:09.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:17:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:08 smithi092 ceph-mon[33966]: pgmap v485: 238 pgs: 238 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-04-14T15:17:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:08 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-47","var": "hit_set_period","val": "600"}]': finished 2024-04-14T15:17:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:08 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/784308691' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi090-93363-61"}]': finished 2024-04-14T15:17:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:08 smithi092 ceph-mon[33966]: osdmap e360: 8 total, 8 up, 8 in 2024-04-14T15:17:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:08 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-14T15:17:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:08 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/784308691' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi090-93363-61"}]: dispatch 2024-04-14T15:17:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:08 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-14T15:17:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:08 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:17:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:08 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:17:09.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.405+0000 7fe93aca7700 1 -- 172.21.15.90:0/1951628037 >> v1:172.21.15.90:6789/0 conn(0x7fe934107760 legacy=0x7fe934107b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:09.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.405+0000 7fe93aca7700 1 -- 172.21.15.90:0/1951628037 shutdown_connections 2024-04-14T15:17:09.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.405+0000 7fe93aca7700 1 -- 172.21.15.90:0/1951628037 >> 172.21.15.90:0/1951628037 conn(0x7fe9340fd290 msgr2=0x7fe9340ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:17:09.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.405+0000 7fe93aca7700 1 -- 172.21.15.90:0/1951628037 shutdown_connections 2024-04-14T15:17:09.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.405+0000 7fe93aca7700 1 -- 172.21.15.90:0/1951628037 wait complete. 2024-04-14T15:17:09.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.406+0000 7fe93aca7700 1 librados: starting msgr at 2024-04-14T15:17:09.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.406+0000 7fe93aca7700 1 librados: starting objecter 2024-04-14T15:17:09.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.406+0000 7fe93aca7700 1 Processor -- start 2024-04-14T15:17:09.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.406+0000 7fe93aca7700 1 -- start start 2024-04-14T15:17:09.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.406+0000 7fe93aca7700 1 librados: setting wanted keys 2024-04-14T15:17:09.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.406+0000 7fe93aca7700 1 librados: calling monclient init 2024-04-14T15:17:09.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.406+0000 7fe93aca7700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe9341a7df0 con 0x7fe93410a610 2024-04-14T15:17:09.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.406+0000 7fe93aca7700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe9341a8fd0 con 0x7fe934107760 2024-04-14T15:17:09.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.406+0000 7fe93aca7700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe9341aa1b0 con 0x7fe93410e1e0 2024-04-14T15:17:09.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.406+0000 7fe938a43700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fe934107760 0x7fe93410be40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.90:41436/0 (socket says 172.21.15.90:41436) 2024-04-14T15:17:09.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.406+0000 7fe933fff700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fe93410a610 0x7fe93410c550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:44908/0 (socket says 172.21.15.90:44908) 2024-04-14T15:17:09.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.406+0000 7fe938a43700 1 -- 172.21.15.90:0/1142507158 learned_addr learned my addr 172.21.15.90:0/1142507158 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:17:09.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.406+0000 7fe931ffb700 10 client.?.objecter ms_handle_connect 0x7fe93410e1e0 2024-04-14T15:17:09.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.406+0000 7fe931ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:17:09.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.406+0000 7fe931ffb700 10 client.?.objecter ms_handle_connect 0x7fe93410a610 2024-04-14T15:17:09.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.406+0000 7fe931ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:17:09.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.406+0000 7fe931ffb700 10 client.?.objecter ms_handle_connect 0x7fe934107760 2024-04-14T15:17:09.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.406+0000 7fe931ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:17:09.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.406+0000 7fe931ffb700 1 -- 172.21.15.90:0/1142507158 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1307044630 0 0) 0x7fe9341a7df0 con 0x7fe93410a610 2024-04-14T15:17:09.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.406+0000 7fe931ffb700 1 -- 172.21.15.90:0/1142507158 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe9240035b0 con 0x7fe93410a610 2024-04-14T15:17:09.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.406+0000 7fe931ffb700 1 -- 172.21.15.90:0/1142507158 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2023124927 0 0) 0x7fe9341a8fd0 con 0x7fe934107760 2024-04-14T15:17:09.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.406+0000 7fe931ffb700 1 -- 172.21.15.90:0/1142507158 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe9341a7df0 con 0x7fe934107760 2024-04-14T15:17:09.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.406+0000 7fe931ffb700 1 -- 172.21.15.90:0/1142507158 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1548299636 0 0) 0x7fe9341aa1b0 con 0x7fe93410e1e0 2024-04-14T15:17:09.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.406+0000 7fe931ffb700 1 -- 172.21.15.90:0/1142507158 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe9341a8fd0 con 0x7fe93410e1e0 2024-04-14T15:17:09.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.407+0000 7fe931ffb700 1 -- 172.21.15.90:0/1142507158 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 128324333 0 0) 0x7fe9240035b0 con 0x7fe93410a610 2024-04-14T15:17:09.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.407+0000 7fe931ffb700 1 -- 172.21.15.90:0/1142507158 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe9341aa1b0 con 0x7fe93410a610 2024-04-14T15:17:09.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.407+0000 7fe931ffb700 1 -- 172.21.15.90:0/1142507158 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fe9200018b0 con 0x7fe93410a610 2024-04-14T15:17:09.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.407+0000 7fe931ffb700 1 -- 172.21.15.90:0/1142507158 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4082904833 0 0) 0x7fe9341a7df0 con 0x7fe934107760 2024-04-14T15:17:09.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.407+0000 7fe931ffb700 1 -- 172.21.15.90:0/1142507158 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe9240035b0 con 0x7fe934107760 2024-04-14T15:17:09.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.407+0000 7fe931ffb700 1 -- 172.21.15.90:0/1142507158 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fe928001430 con 0x7fe934107760 2024-04-14T15:17:09.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.407+0000 7fe931ffb700 1 -- 172.21.15.90:0/1142507158 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1794077627 0 0) 0x7fe9341aa1b0 con 0x7fe93410a610 2024-04-14T15:17:09.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.407+0000 7fe931ffb700 1 -- 172.21.15.90:0/1142507158 >> v1:172.21.15.90:6790/0 conn(0x7fe93410e1e0 legacy=0x7fe9341a66d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:09.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.407+0000 7fe931ffb700 1 -- 172.21.15.90:0/1142507158 >> v1:172.21.15.92:6789/0 conn(0x7fe934107760 legacy=0x7fe93410be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:09.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.407+0000 7fe931ffb700 1 -- 172.21.15.90:0/1142507158 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9341ab390 con 0x7fe93410a610 2024-04-14T15:17:09.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.407+0000 7fe93aca7700 1 -- 172.21.15.90:0/1142507158 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe9341a8020 con 0x7fe93410a610 2024-04-14T15:17:09.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.407+0000 7fe931ffb700 1 -- 172.21.15.90:0/1142507158 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fe920003b40 con 0x7fe93410a610 2024-04-14T15:17:09.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.407+0000 7fe931ffb700 1 -- 172.21.15.90:0/1142507158 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fe920005ca0 con 0x7fe93410a610 2024-04-14T15:17:09.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.407+0000 7fe93aca7700 10 client.37134.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:17:09.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.407+0000 7fe93aca7700 1 -- 172.21.15.90:0/1142507158 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe9341a84e0 con 0x7fe93410a610 2024-04-14T15:17:09.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.407+0000 7fe93aca7700 1 librados: init done 2024-04-14T15:17:09.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.408+0000 7fe93aca7700 1 -- 172.21.15.90:0/1142507158 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fe934102ea0 con 0x7fe93410a610 2024-04-14T15:17:09.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.409+0000 7fe931ffb700 1 -- 172.21.15.90:0/1142507158 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fe9200036a0 con 0x7fe93410a610 2024-04-14T15:17:09.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.410+0000 7fe931ffb700 1 -- 172.21.15.90:0/1142507158 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(361..361 src has 51..361) ==== 8178+0+0 (unknown 2291187325 0 0) 0x7fe92008a980 con 0x7fe93410a610 2024-04-14T15:17:09.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.410+0000 7fe931ffb700 10 client.37134.objecter ms_dispatch 0x7fe934001550 osd_map(361..361 src has 51..361) 2024-04-14T15:17:09.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.410+0000 7fe931ffb700 3 client.37134.objecter handle_osd_map got epochs [361,361] > 0 2024-04-14T15:17:09.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.410+0000 7fe931ffb700 3 client.37134.objecter handle_osd_map decoding full epoch 361 2024-04-14T15:17:09.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.410+0000 7fe931ffb700 20 client.37134.objecter dump_active .. 0 homeless 2024-04-14T15:17:09.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.410+0000 7fe931ffb700 10 client.37134.objecter ms_handle_connect 0x7fe924071120 2024-04-14T15:17:09.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.410+0000 7fe931ffb700 1 -- 172.21.15.90:0/1142507158 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v361) ==== 54+0+16 (unknown 94325798 0 2420000192) 0x7fe92008b060 con 0x7fe93410a610 2024-04-14T15:17:09.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.410+0000 7fe93aca7700 10 client.37134.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:17:09.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.410+0000 7fe93aca7700 15 client.37134.objecter _session_command_op_assign -1 1 2024-04-14T15:17:09.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.410+0000 7fe93aca7700 20 client.37134.objecter _get_session s=0x7fe9340693d0 osd=0 3 2024-04-14T15:17:09.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.410+0000 7fe93aca7700 20 client.37134.objecter put_session s=0x7fe9340693d0 osd=0 3 2024-04-14T15:17:09.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.410+0000 7fe93aca7700 20 client.37134.objecter _get_session s=0x7fe9340693d0 osd=0 3 2024-04-14T15:17:09.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.410+0000 7fe93aca7700 15 client.37134.objecter _session_command_op_remove -1 1 2024-04-14T15:17:09.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.410+0000 7fe93aca7700 20 client.37134.objecter get_session s=0x7fe9340693d0 osd=0 3 2024-04-14T15:17:09.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.410+0000 7fe93aca7700 15 client.37134.objecter _session_command_op_assign 0 1 2024-04-14T15:17:09.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.410+0000 7fe93aca7700 20 client.37134.objecter put_session s=0x7fe9340693d0 osd=0 4 2024-04-14T15:17:09.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.410+0000 7fe93aca7700 10 client.37134.objecter _send_command 1 2024-04-14T15:17:09.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.410+0000 7fe93aca7700 1 -- 172.21.15.90:0/1142507158 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe93410d370 con 0x7fe9341a91a0 2024-04-14T15:17:09.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.410+0000 7fe931ffb700 10 client.37134.objecter ms_handle_connect 0x7fe9341a91a0 2024-04-14T15:17:09.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.411+0000 7fe931ffb700 1 -- 172.21.15.90:0/1142507158 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fe93410d370 con 0x7fe9341a91a0 2024-04-14T15:17:09.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.411+0000 7fe931ffb700 10 client.37134.objecter ms_dispatch 0x7fe934001550 command_reply(tid 1: 0 ) 2024-04-14T15:17:09.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.411+0000 7fe931ffb700 10 client.37134.objecter _finish_command 1 = system:0 2024-04-14T15:17:09.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.411+0000 7fe931ffb700 20 client.37134.objecter put_session s=0x7fe9340693d0 osd=0 4 2024-04-14T15:17:09.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.411+0000 7fe931ffb700 15 client.37134.objecter _session_command_op_remove 0 1 2024-04-14T15:17:09.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.428+0000 7fe93aca7700 10 client.37134.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:17:09.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.428+0000 7fe93aca7700 15 client.37134.objecter _session_command_op_assign -1 2 2024-04-14T15:17:09.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.428+0000 7fe93aca7700 20 client.37134.objecter _get_session s=0x7fe9340693d0 osd=0 3 2024-04-14T15:17:09.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.428+0000 7fe93aca7700 20 client.37134.objecter put_session s=0x7fe9340693d0 osd=0 3 2024-04-14T15:17:09.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.428+0000 7fe93aca7700 20 client.37134.objecter _get_session s=0x7fe9340693d0 osd=0 3 2024-04-14T15:17:09.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.428+0000 7fe93aca7700 15 client.37134.objecter _session_command_op_remove -1 2 2024-04-14T15:17:09.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.428+0000 7fe93aca7700 20 client.37134.objecter get_session s=0x7fe9340693d0 osd=0 3 2024-04-14T15:17:09.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.428+0000 7fe93aca7700 15 client.37134.objecter _session_command_op_assign 0 2 2024-04-14T15:17:09.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.428+0000 7fe93aca7700 20 client.37134.objecter put_session s=0x7fe9340693d0 osd=0 4 2024-04-14T15:17:09.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.428+0000 7fe93aca7700 10 client.37134.objecter _send_command 2 2024-04-14T15:17:09.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.428+0000 7fe93aca7700 1 -- 172.21.15.90:0/1142507158 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe9341a9c90 con 0x7fe9341a91a0 2024-04-14T15:17:09.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.430+0000 7fe931ffb700 1 -- 172.21.15.90:0/1142507158 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fe9341a9c90 con 0x7fe9341a91a0 2024-04-14T15:17:09.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.430+0000 7fe931ffb700 10 client.37134.objecter ms_dispatch 0x7fe934001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:17:09.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.430+0000 7fe931ffb700 10 client.37134.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:17:09.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.430+0000 7fe931ffb700 20 client.37134.objecter put_session s=0x7fe9340693d0 osd=0 4 2024-04-14T15:17:09.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.430+0000 7fe931ffb700 15 client.37134.objecter _session_command_op_remove 0 2 2024-04-14T15:17:09.431 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:17:09.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.430+0000 7fe93aca7700 10 client.37134.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:17:09.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.430+0000 7fe93aca7700 15 client.37134.objecter _session_command_op_assign -1 3 2024-04-14T15:17:09.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.430+0000 7fe93aca7700 20 client.37134.objecter _get_session s=0x7fe9340fd670 osd=1 3 2024-04-14T15:17:09.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.430+0000 7fe93aca7700 20 client.37134.objecter put_session s=0x7fe9340fd670 osd=1 3 2024-04-14T15:17:09.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.430+0000 7fe93aca7700 20 client.37134.objecter _get_session s=0x7fe9340fd670 osd=1 3 2024-04-14T15:17:09.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.430+0000 7fe93aca7700 15 client.37134.objecter _session_command_op_remove -1 3 2024-04-14T15:17:09.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.430+0000 7fe93aca7700 20 client.37134.objecter get_session s=0x7fe9340fd670 osd=1 3 2024-04-14T15:17:09.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.430+0000 7fe93aca7700 15 client.37134.objecter _session_command_op_assign 1 3 2024-04-14T15:17:09.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.430+0000 7fe93aca7700 20 client.37134.objecter put_session s=0x7fe9340fd670 osd=1 4 2024-04-14T15:17:09.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.430+0000 7fe93aca7700 10 client.37134.objecter _send_command 3 2024-04-14T15:17:09.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.430+0000 7fe93aca7700 1 -- 172.21.15.90:0/1142507158 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fe9340fe130 con 0x7fe9340fdcf0 2024-04-14T15:17:09.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.431+0000 7fe931ffb700 10 client.37134.objecter ms_handle_connect 0x7fe9340fdcf0 2024-04-14T15:17:09.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.431+0000 7fe931ffb700 1 -- 172.21.15.90:0/1142507158 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fe9340fe130 con 0x7fe9340fdcf0 2024-04-14T15:17:09.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.431+0000 7fe931ffb700 10 client.37134.objecter ms_dispatch 0x7fe934001550 command_reply(tid 3: 0 ) 2024-04-14T15:17:09.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.431+0000 7fe931ffb700 10 client.37134.objecter _finish_command 3 = system:0 2024-04-14T15:17:09.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.431+0000 7fe931ffb700 20 client.37134.objecter put_session s=0x7fe9340fd670 osd=1 4 2024-04-14T15:17:09.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.431+0000 7fe931ffb700 15 client.37134.objecter _session_command_op_remove 1 3 2024-04-14T15:17:09.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.448+0000 7fe93aca7700 10 client.37134.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:17:09.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.448+0000 7fe93aca7700 15 client.37134.objecter _session_command_op_assign -1 4 2024-04-14T15:17:09.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.448+0000 7fe93aca7700 20 client.37134.objecter _get_session s=0x7fe9340fd670 osd=1 3 2024-04-14T15:17:09.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.448+0000 7fe93aca7700 20 client.37134.objecter put_session s=0x7fe9340fd670 osd=1 3 2024-04-14T15:17:09.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.448+0000 7fe93aca7700 20 client.37134.objecter _get_session s=0x7fe9340fd670 osd=1 3 2024-04-14T15:17:09.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.448+0000 7fe93aca7700 15 client.37134.objecter _session_command_op_remove -1 4 2024-04-14T15:17:09.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.448+0000 7fe93aca7700 20 client.37134.objecter get_session s=0x7fe9340fd670 osd=1 3 2024-04-14T15:17:09.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.448+0000 7fe93aca7700 15 client.37134.objecter _session_command_op_assign 1 4 2024-04-14T15:17:09.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.448+0000 7fe93aca7700 20 client.37134.objecter put_session s=0x7fe9340fd670 osd=1 4 2024-04-14T15:17:09.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.448+0000 7fe93aca7700 10 client.37134.objecter _send_command 4 2024-04-14T15:17:09.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.448+0000 7fe93aca7700 1 -- 172.21.15.90:0/1142507158 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe9340fe560 con 0x7fe9340fdcf0 2024-04-14T15:17:09.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.450+0000 7fe931ffb700 1 -- 172.21.15.90:0/1142507158 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fe9340fe560 con 0x7fe9340fdcf0 2024-04-14T15:17:09.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.450+0000 7fe931ffb700 10 client.37134.objecter ms_dispatch 0x7fe934001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:17:09.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.450+0000 7fe931ffb700 10 client.37134.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:17:09.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.450+0000 7fe931ffb700 20 client.37134.objecter put_session s=0x7fe9340fd670 osd=1 4 2024-04-14T15:17:09.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.450+0000 7fe931ffb700 15 client.37134.objecter _session_command_op_remove 1 4 2024-04-14T15:17:09.451 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:17:09.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.450+0000 7fe93aca7700 10 client.37134.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:17:09.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.450+0000 7fe93aca7700 15 client.37134.objecter _session_command_op_assign -1 5 2024-04-14T15:17:09.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.450+0000 7fe93aca7700 20 client.37134.objecter _get_session s=0x7fe9341110a0 osd=2 3 2024-04-14T15:17:09.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.450+0000 7fe93aca7700 20 client.37134.objecter put_session s=0x7fe9341110a0 osd=2 3 2024-04-14T15:17:09.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.450+0000 7fe93aca7700 20 client.37134.objecter _get_session s=0x7fe9341110a0 osd=2 3 2024-04-14T15:17:09.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.450+0000 7fe93aca7700 15 client.37134.objecter _session_command_op_remove -1 5 2024-04-14T15:17:09.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.450+0000 7fe93aca7700 20 client.37134.objecter get_session s=0x7fe9341110a0 osd=2 3 2024-04-14T15:17:09.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.450+0000 7fe93aca7700 15 client.37134.objecter _session_command_op_assign 2 5 2024-04-14T15:17:09.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.450+0000 7fe93aca7700 20 client.37134.objecter put_session s=0x7fe9341110a0 osd=2 4 2024-04-14T15:17:09.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.450+0000 7fe93aca7700 10 client.37134.objecter _send_command 5 2024-04-14T15:17:09.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.450+0000 7fe93aca7700 1 -- 172.21.15.90:0/1142507158 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fe934063bd0 con 0x7fe934111210 2024-04-14T15:17:09.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.450+0000 7fe931ffb700 10 client.37134.objecter ms_handle_connect 0x7fe934111210 2024-04-14T15:17:09.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.451+0000 7fe931ffb700 1 -- 172.21.15.90:0/1142507158 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fe934063bd0 con 0x7fe934111210 2024-04-14T15:17:09.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.451+0000 7fe931ffb700 10 client.37134.objecter ms_dispatch 0x7fe934001550 command_reply(tid 5: 0 ) 2024-04-14T15:17:09.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.451+0000 7fe931ffb700 10 client.37134.objecter _finish_command 5 = system:0 2024-04-14T15:17:09.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.451+0000 7fe931ffb700 20 client.37134.objecter put_session s=0x7fe9341110a0 osd=2 4 2024-04-14T15:17:09.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.451+0000 7fe931ffb700 15 client.37134.objecter _session_command_op_remove 2 5 2024-04-14T15:17:09.455 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:17:09.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.468+0000 7fe93aca7700 10 client.37134.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:17:09.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.468+0000 7fe93aca7700 15 client.37134.objecter _session_command_op_assign -1 6 2024-04-14T15:17:09.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.468+0000 7fe93aca7700 20 client.37134.objecter _get_session s=0x7fe9341110a0 osd=2 3 2024-04-14T15:17:09.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.468+0000 7fe93aca7700 20 client.37134.objecter put_session s=0x7fe9341110a0 osd=2 3 2024-04-14T15:17:09.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.468+0000 7fe93aca7700 20 client.37134.objecter _get_session s=0x7fe9341110a0 osd=2 3 2024-04-14T15:17:09.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.468+0000 7fe93aca7700 15 client.37134.objecter _session_command_op_remove -1 6 2024-04-14T15:17:09.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.468+0000 7fe93aca7700 20 client.37134.objecter get_session s=0x7fe9341110a0 osd=2 3 2024-04-14T15:17:09.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.468+0000 7fe93aca7700 15 client.37134.objecter _session_command_op_assign 2 6 2024-04-14T15:17:09.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.468+0000 7fe93aca7700 20 client.37134.objecter put_session s=0x7fe9341110a0 osd=2 4 2024-04-14T15:17:09.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.468+0000 7fe93aca7700 10 client.37134.objecter _send_command 6 2024-04-14T15:17:09.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.468+0000 7fe93aca7700 1 -- 172.21.15.90:0/1142507158 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe9341b3e90 con 0x7fe934111210 2024-04-14T15:17:09.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.469+0000 7fe931ffb700 1 -- 172.21.15.90:0/1142507158 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fe9341b3e90 con 0x7fe934111210 2024-04-14T15:17:09.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.469+0000 7fe931ffb700 10 client.37134.objecter ms_dispatch 0x7fe934001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:17:09.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.469+0000 7fe931ffb700 10 client.37134.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:17:09.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.469+0000 7fe931ffb700 20 client.37134.objecter put_session s=0x7fe9341110a0 osd=2 4 2024-04-14T15:17:09.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.469+0000 7fe931ffb700 15 client.37134.objecter _session_command_op_remove 2 6 2024-04-14T15:17:09.470 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:17:09.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.470+0000 7fe93aca7700 10 client.37134.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:17:09.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.470+0000 7fe93aca7700 15 client.37134.objecter _session_command_op_assign -1 7 2024-04-14T15:17:09.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.470+0000 7fe93aca7700 20 client.37134.objecter _get_session s=0x7fe934063140 osd=3 3 2024-04-14T15:17:09.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.470+0000 7fe93aca7700 20 client.37134.objecter put_session s=0x7fe934063140 osd=3 3 2024-04-14T15:17:09.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.470+0000 7fe93aca7700 20 client.37134.objecter _get_session s=0x7fe934063140 osd=3 3 2024-04-14T15:17:09.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.470+0000 7fe93aca7700 15 client.37134.objecter _session_command_op_remove -1 7 2024-04-14T15:17:09.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.470+0000 7fe93aca7700 20 client.37134.objecter get_session s=0x7fe934063140 osd=3 3 2024-04-14T15:17:09.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.470+0000 7fe93aca7700 15 client.37134.objecter _session_command_op_assign 3 7 2024-04-14T15:17:09.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.470+0000 7fe93aca7700 20 client.37134.objecter put_session s=0x7fe934063140 osd=3 4 2024-04-14T15:17:09.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.470+0000 7fe93aca7700 10 client.37134.objecter _send_command 7 2024-04-14T15:17:09.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.470+0000 7fe93aca7700 1 -- 172.21.15.90:0/1142507158 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fe9341b6760 con 0x7fe9340632b0 2024-04-14T15:17:09.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.470+0000 7fe931ffb700 10 client.37134.objecter ms_handle_connect 0x7fe9340632b0 2024-04-14T15:17:09.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.471+0000 7fe931ffb700 1 -- 172.21.15.90:0/1142507158 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fe9341b6760 con 0x7fe9340632b0 2024-04-14T15:17:09.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.471+0000 7fe931ffb700 10 client.37134.objecter ms_dispatch 0x7fe934001550 command_reply(tid 7: 0 ) 2024-04-14T15:17:09.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.471+0000 7fe931ffb700 10 client.37134.objecter _finish_command 7 = system:0 2024-04-14T15:17:09.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.471+0000 7fe931ffb700 20 client.37134.objecter put_session s=0x7fe934063140 osd=3 4 2024-04-14T15:17:09.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.471+0000 7fe931ffb700 15 client.37134.objecter _session_command_op_remove 3 7 2024-04-14T15:17:09.480 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:09.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:09.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:17:09.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.488+0000 7fe93aca7700 10 client.37134.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:17:09.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.488+0000 7fe93aca7700 15 client.37134.objecter _session_command_op_assign -1 8 2024-04-14T15:17:09.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.488+0000 7fe93aca7700 20 client.37134.objecter _get_session s=0x7fe934063140 osd=3 3 2024-04-14T15:17:09.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.488+0000 7fe93aca7700 20 client.37134.objecter put_session s=0x7fe934063140 osd=3 3 2024-04-14T15:17:09.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.488+0000 7fe93aca7700 20 client.37134.objecter _get_session s=0x7fe934063140 osd=3 3 2024-04-14T15:17:09.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.488+0000 7fe93aca7700 15 client.37134.objecter _session_command_op_remove -1 8 2024-04-14T15:17:09.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.488+0000 7fe93aca7700 20 client.37134.objecter get_session s=0x7fe934063140 osd=3 3 2024-04-14T15:17:09.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.488+0000 7fe93aca7700 15 client.37134.objecter _session_command_op_assign 3 8 2024-04-14T15:17:09.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.488+0000 7fe93aca7700 20 client.37134.objecter put_session s=0x7fe934063140 osd=3 4 2024-04-14T15:17:09.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.488+0000 7fe93aca7700 10 client.37134.objecter _send_command 8 2024-04-14T15:17:09.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.488+0000 7fe93aca7700 1 -- 172.21.15.90:0/1142507158 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe9341bad70 con 0x7fe9340632b0 2024-04-14T15:17:09.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.489+0000 7fe931ffb700 1 -- 172.21.15.90:0/1142507158 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fe9341bad70 con 0x7fe9340632b0 2024-04-14T15:17:09.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.489+0000 7fe931ffb700 10 client.37134.objecter ms_dispatch 0x7fe934001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:17:09.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.489+0000 7fe931ffb700 10 client.37134.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:17:09.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.489+0000 7fe931ffb700 20 client.37134.objecter put_session s=0x7fe934063140 osd=3 4 2024-04-14T15:17:09.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.489+0000 7fe931ffb700 15 client.37134.objecter _session_command_op_remove 3 8 2024-04-14T15:17:09.490 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:17:09.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.489+0000 7fe93aca7700 10 client.37134.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:17:09.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.490+0000 7fe93aca7700 15 client.37134.objecter _session_command_op_assign -1 9 2024-04-14T15:17:09.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.490+0000 7fe93aca7700 20 client.37134.objecter _get_session s=0x7fe9341b05d0 osd=4 3 2024-04-14T15:17:09.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.490+0000 7fe93aca7700 20 client.37134.objecter put_session s=0x7fe9341b05d0 osd=4 3 2024-04-14T15:17:09.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.490+0000 7fe93aca7700 20 client.37134.objecter _get_session s=0x7fe9341b05d0 osd=4 3 2024-04-14T15:17:09.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.490+0000 7fe93aca7700 15 client.37134.objecter _session_command_op_remove -1 9 2024-04-14T15:17:09.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.490+0000 7fe93aca7700 20 client.37134.objecter get_session s=0x7fe9341b05d0 osd=4 3 2024-04-14T15:17:09.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.490+0000 7fe93aca7700 15 client.37134.objecter _session_command_op_assign 4 9 2024-04-14T15:17:09.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.490+0000 7fe93aca7700 20 client.37134.objecter put_session s=0x7fe9341b05d0 osd=4 4 2024-04-14T15:17:09.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.490+0000 7fe93aca7700 10 client.37134.objecter _send_command 9 2024-04-14T15:17:09.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.490+0000 7fe93aca7700 1 -- 172.21.15.90:0/1142507158 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fe9341bf040 con 0x7fe9341bb2f0 2024-04-14T15:17:09.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.490+0000 7fe931ffb700 10 client.37134.objecter ms_handle_connect 0x7fe9341bb2f0 2024-04-14T15:17:09.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.491+0000 7fe931ffb700 1 -- 172.21.15.90:0/1142507158 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fe9341bf040 con 0x7fe9341bb2f0 2024-04-14T15:17:09.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.491+0000 7fe931ffb700 10 client.37134.objecter ms_dispatch 0x7fe934001550 command_reply(tid 9: 0 ) 2024-04-14T15:17:09.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.491+0000 7fe931ffb700 10 client.37134.objecter _finish_command 9 = system:0 2024-04-14T15:17:09.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.491+0000 7fe931ffb700 20 client.37134.objecter put_session s=0x7fe9341b05d0 osd=4 4 2024-04-14T15:17:09.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.491+0000 7fe931ffb700 15 client.37134.objecter _session_command_op_remove 4 9 2024-04-14T15:17:09.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.507+0000 7fe93aca7700 10 client.37134.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:17:09.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.507+0000 7fe93aca7700 15 client.37134.objecter _session_command_op_assign -1 10 2024-04-14T15:17:09.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.507+0000 7fe93aca7700 20 client.37134.objecter _get_session s=0x7fe9341b05d0 osd=4 3 2024-04-14T15:17:09.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.507+0000 7fe93aca7700 20 client.37134.objecter put_session s=0x7fe9341b05d0 osd=4 3 2024-04-14T15:17:09.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.507+0000 7fe93aca7700 20 client.37134.objecter _get_session s=0x7fe9341b05d0 osd=4 3 2024-04-14T15:17:09.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.507+0000 7fe93aca7700 15 client.37134.objecter _session_command_op_remove -1 10 2024-04-14T15:17:09.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.507+0000 7fe93aca7700 20 client.37134.objecter get_session s=0x7fe9341b05d0 osd=4 3 2024-04-14T15:17:09.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.507+0000 7fe93aca7700 15 client.37134.objecter _session_command_op_assign 4 10 2024-04-14T15:17:09.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.507+0000 7fe93aca7700 20 client.37134.objecter put_session s=0x7fe9341b05d0 osd=4 4 2024-04-14T15:17:09.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.507+0000 7fe93aca7700 10 client.37134.objecter _send_command 10 2024-04-14T15:17:09.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.507+0000 7fe93aca7700 1 -- 172.21.15.90:0/1142507158 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe9341bf220 con 0x7fe9341bb2f0 2024-04-14T15:17:09.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.509+0000 7fe931ffb700 1 -- 172.21.15.90:0/1142507158 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fe9341bf220 con 0x7fe9341bb2f0 2024-04-14T15:17:09.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.509+0000 7fe931ffb700 10 client.37134.objecter ms_dispatch 0x7fe934001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:17:09.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.509+0000 7fe931ffb700 10 client.37134.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:17:09.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.509+0000 7fe931ffb700 20 client.37134.objecter put_session s=0x7fe9341b05d0 osd=4 4 2024-04-14T15:17:09.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.509+0000 7fe931ffb700 15 client.37134.objecter _session_command_op_remove 4 10 2024-04-14T15:17:09.510 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:17:09.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.510+0000 7fe93aca7700 10 client.37134.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:17:09.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.510+0000 7fe93aca7700 15 client.37134.objecter _session_command_op_assign -1 11 2024-04-14T15:17:09.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.510+0000 7fe93aca7700 20 client.37134.objecter _get_session s=0x7fe9341bf220 osd=5 3 2024-04-14T15:17:09.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.510+0000 7fe93aca7700 20 client.37134.objecter put_session s=0x7fe9341bf220 osd=5 3 2024-04-14T15:17:09.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.510+0000 7fe93aca7700 20 client.37134.objecter _get_session s=0x7fe9341bf220 osd=5 3 2024-04-14T15:17:09.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.510+0000 7fe93aca7700 15 client.37134.objecter _session_command_op_remove -1 11 2024-04-14T15:17:09.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.510+0000 7fe93aca7700 20 client.37134.objecter get_session s=0x7fe9341bf220 osd=5 3 2024-04-14T15:17:09.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.510+0000 7fe93aca7700 15 client.37134.objecter _session_command_op_assign 5 11 2024-04-14T15:17:09.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.510+0000 7fe93aca7700 20 client.37134.objecter put_session s=0x7fe9341bf220 osd=5 4 2024-04-14T15:17:09.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.510+0000 7fe93aca7700 10 client.37134.objecter _send_command 11 2024-04-14T15:17:09.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.510+0000 7fe93aca7700 1 -- 172.21.15.90:0/1142507158 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fe9341c04e0 con 0x7fe9341bf940 2024-04-14T15:17:09.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.510+0000 7fe931ffb700 10 client.37134.objecter ms_handle_connect 0x7fe9341bf940 2024-04-14T15:17:09.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.511+0000 7fe931ffb700 1 -- 172.21.15.90:0/1142507158 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fe9341c04e0 con 0x7fe9341bf940 2024-04-14T15:17:09.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.511+0000 7fe931ffb700 10 client.37134.objecter ms_dispatch 0x7fe934001550 command_reply(tid 11: 0 ) 2024-04-14T15:17:09.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.511+0000 7fe931ffb700 10 client.37134.objecter _finish_command 11 = system:0 2024-04-14T15:17:09.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.511+0000 7fe931ffb700 20 client.37134.objecter put_session s=0x7fe9341bf220 osd=5 4 2024-04-14T15:17:09.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.511+0000 7fe931ffb700 15 client.37134.objecter _session_command_op_remove 5 11 2024-04-14T15:17:09.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.527+0000 7fe93aca7700 10 client.37134.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:17:09.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.527+0000 7fe93aca7700 15 client.37134.objecter _session_command_op_assign -1 12 2024-04-14T15:17:09.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.527+0000 7fe93aca7700 20 client.37134.objecter _get_session s=0x7fe9341bf220 osd=5 3 2024-04-14T15:17:09.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.527+0000 7fe93aca7700 20 client.37134.objecter put_session s=0x7fe9341bf220 osd=5 3 2024-04-14T15:17:09.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.527+0000 7fe93aca7700 20 client.37134.objecter _get_session s=0x7fe9341bf220 osd=5 3 2024-04-14T15:17:09.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.527+0000 7fe93aca7700 15 client.37134.objecter _session_command_op_remove -1 12 2024-04-14T15:17:09.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.527+0000 7fe93aca7700 20 client.37134.objecter get_session s=0x7fe9341bf220 osd=5 3 2024-04-14T15:17:09.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.527+0000 7fe93aca7700 15 client.37134.objecter _session_command_op_assign 5 12 2024-04-14T15:17:09.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.527+0000 7fe93aca7700 20 client.37134.objecter put_session s=0x7fe9341bf220 osd=5 4 2024-04-14T15:17:09.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.527+0000 7fe93aca7700 10 client.37134.objecter _send_command 12 2024-04-14T15:17:09.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.527+0000 7fe93aca7700 1 -- 172.21.15.90:0/1142507158 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe9341c04e0 con 0x7fe9341bf940 2024-04-14T15:17:09.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.529+0000 7fe931ffb700 1 -- 172.21.15.90:0/1142507158 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fe9341c04e0 con 0x7fe9341bf940 2024-04-14T15:17:09.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.529+0000 7fe931ffb700 10 client.37134.objecter ms_dispatch 0x7fe934001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:17:09.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.529+0000 7fe931ffb700 10 client.37134.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:17:09.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.529+0000 7fe931ffb700 20 client.37134.objecter put_session s=0x7fe9341bf220 osd=5 4 2024-04-14T15:17:09.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.529+0000 7fe931ffb700 15 client.37134.objecter _session_command_op_remove 5 12 2024-04-14T15:17:09.530 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:17:09.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.529+0000 7fe93aca7700 10 client.37134.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:17:09.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.529+0000 7fe93aca7700 15 client.37134.objecter _session_command_op_assign -1 13 2024-04-14T15:17:09.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.529+0000 7fe93aca7700 20 client.37134.objecter _get_session s=0x7fe9341c04e0 osd=6 3 2024-04-14T15:17:09.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.529+0000 7fe93aca7700 20 client.37134.objecter put_session s=0x7fe9341c04e0 osd=6 3 2024-04-14T15:17:09.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.529+0000 7fe93aca7700 20 client.37134.objecter _get_session s=0x7fe9341c04e0 osd=6 3 2024-04-14T15:17:09.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.529+0000 7fe93aca7700 15 client.37134.objecter _session_command_op_remove -1 13 2024-04-14T15:17:09.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.529+0000 7fe93aca7700 20 client.37134.objecter get_session s=0x7fe9341c04e0 osd=6 3 2024-04-14T15:17:09.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.529+0000 7fe93aca7700 15 client.37134.objecter _session_command_op_assign 6 13 2024-04-14T15:17:09.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.529+0000 7fe93aca7700 20 client.37134.objecter put_session s=0x7fe9341c04e0 osd=6 4 2024-04-14T15:17:09.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.529+0000 7fe93aca7700 10 client.37134.objecter _send_command 13 2024-04-14T15:17:09.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.529+0000 7fe93aca7700 1 -- 172.21.15.90:0/1142507158 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fe9341c7a30 con 0x7fe9341c3f20 2024-04-14T15:17:09.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.530+0000 7fe931ffb700 10 client.37134.objecter ms_handle_connect 0x7fe9341c3f20 2024-04-14T15:17:09.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.531+0000 7fe931ffb700 1 -- 172.21.15.90:0/1142507158 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fe9341c7a30 con 0x7fe9341c3f20 2024-04-14T15:17:09.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.531+0000 7fe931ffb700 10 client.37134.objecter ms_dispatch 0x7fe934001550 command_reply(tid 13: 0 ) 2024-04-14T15:17:09.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.531+0000 7fe931ffb700 10 client.37134.objecter _finish_command 13 = system:0 2024-04-14T15:17:09.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.531+0000 7fe931ffb700 20 client.37134.objecter put_session s=0x7fe9341c04e0 osd=6 4 2024-04-14T15:17:09.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.531+0000 7fe931ffb700 15 client.37134.objecter _session_command_op_remove 6 13 2024-04-14T15:17:09.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.547+0000 7fe93aca7700 10 client.37134.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:17:09.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.547+0000 7fe93aca7700 15 client.37134.objecter _session_command_op_assign -1 14 2024-04-14T15:17:09.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.547+0000 7fe93aca7700 20 client.37134.objecter _get_session s=0x7fe9341c04e0 osd=6 3 2024-04-14T15:17:09.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.547+0000 7fe93aca7700 20 client.37134.objecter put_session s=0x7fe9341c04e0 osd=6 3 2024-04-14T15:17:09.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.547+0000 7fe93aca7700 20 client.37134.objecter _get_session s=0x7fe9341c04e0 osd=6 3 2024-04-14T15:17:09.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.547+0000 7fe93aca7700 15 client.37134.objecter _session_command_op_remove -1 14 2024-04-14T15:17:09.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.547+0000 7fe93aca7700 20 client.37134.objecter get_session s=0x7fe9341c04e0 osd=6 3 2024-04-14T15:17:09.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.547+0000 7fe93aca7700 15 client.37134.objecter _session_command_op_assign 6 14 2024-04-14T15:17:09.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.547+0000 7fe93aca7700 20 client.37134.objecter put_session s=0x7fe9341c04e0 osd=6 4 2024-04-14T15:17:09.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.547+0000 7fe93aca7700 10 client.37134.objecter _send_command 14 2024-04-14T15:17:09.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.547+0000 7fe93aca7700 1 -- 172.21.15.90:0/1142507158 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe9341b03e0 con 0x7fe9341c3f20 2024-04-14T15:17:09.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.549+0000 7fe931ffb700 1 -- 172.21.15.90:0/1142507158 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fe9341b03e0 con 0x7fe9341c3f20 2024-04-14T15:17:09.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.549+0000 7fe931ffb700 10 client.37134.objecter ms_dispatch 0x7fe934001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:17:09.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.549+0000 7fe931ffb700 10 client.37134.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:17:09.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.549+0000 7fe931ffb700 20 client.37134.objecter put_session s=0x7fe9341c04e0 osd=6 4 2024-04-14T15:17:09.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.549+0000 7fe931ffb700 15 client.37134.objecter _session_command_op_remove 6 14 2024-04-14T15:17:09.550 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:17:09.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.549+0000 7fe93aca7700 10 client.37134.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:17:09.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.549+0000 7fe93aca7700 15 client.37134.objecter _session_command_op_assign -1 15 2024-04-14T15:17:09.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.549+0000 7fe93aca7700 20 client.37134.objecter _get_session s=0x7fe9341af5e0 osd=7 3 2024-04-14T15:17:09.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.549+0000 7fe93aca7700 20 client.37134.objecter put_session s=0x7fe9341af5e0 osd=7 3 2024-04-14T15:17:09.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.549+0000 7fe93aca7700 20 client.37134.objecter _get_session s=0x7fe9341af5e0 osd=7 3 2024-04-14T15:17:09.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.549+0000 7fe93aca7700 15 client.37134.objecter _session_command_op_remove -1 15 2024-04-14T15:17:09.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.549+0000 7fe93aca7700 20 client.37134.objecter get_session s=0x7fe9341af5e0 osd=7 3 2024-04-14T15:17:09.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.549+0000 7fe93aca7700 15 client.37134.objecter _session_command_op_assign 7 15 2024-04-14T15:17:09.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.549+0000 7fe93aca7700 20 client.37134.objecter put_session s=0x7fe9341af5e0 osd=7 4 2024-04-14T15:17:09.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.549+0000 7fe93aca7700 10 client.37134.objecter _send_command 15 2024-04-14T15:17:09.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.549+0000 7fe93aca7700 1 -- 172.21.15.90:0/1142507158 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fe9341af990 con 0x7fe93410f370 2024-04-14T15:17:09.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.549+0000 7fe931ffb700 10 client.37134.objecter ms_handle_connect 0x7fe93410f370 2024-04-14T15:17:09.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.550+0000 7fe931ffb700 1 -- 172.21.15.90:0/1142507158 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fe9341af990 con 0x7fe93410f370 2024-04-14T15:17:09.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.550+0000 7fe931ffb700 10 client.37134.objecter ms_dispatch 0x7fe934001550 command_reply(tid 15: 0 ) 2024-04-14T15:17:09.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.550+0000 7fe931ffb700 10 client.37134.objecter _finish_command 15 = system:0 2024-04-14T15:17:09.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.550+0000 7fe931ffb700 20 client.37134.objecter put_session s=0x7fe9341af5e0 osd=7 4 2024-04-14T15:17:09.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.550+0000 7fe931ffb700 15 client.37134.objecter _session_command_op_remove 7 15 2024-04-14T15:17:09.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.566+0000 7fe93aca7700 10 client.37134.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:17:09.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.566+0000 7fe93aca7700 15 client.37134.objecter _session_command_op_assign -1 16 2024-04-14T15:17:09.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.566+0000 7fe93aca7700 20 client.37134.objecter _get_session s=0x7fe9341af5e0 osd=7 3 2024-04-14T15:17:09.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.566+0000 7fe93aca7700 20 client.37134.objecter put_session s=0x7fe9341af5e0 osd=7 3 2024-04-14T15:17:09.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.566+0000 7fe93aca7700 20 client.37134.objecter _get_session s=0x7fe9341af5e0 osd=7 3 2024-04-14T15:17:09.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.566+0000 7fe93aca7700 15 client.37134.objecter _session_command_op_remove -1 16 2024-04-14T15:17:09.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.566+0000 7fe93aca7700 20 client.37134.objecter get_session s=0x7fe9341af5e0 osd=7 3 2024-04-14T15:17:09.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.566+0000 7fe93aca7700 15 client.37134.objecter _session_command_op_assign 7 16 2024-04-14T15:17:09.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.566+0000 7fe93aca7700 20 client.37134.objecter put_session s=0x7fe9341af5e0 osd=7 4 2024-04-14T15:17:09.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.566+0000 7fe93aca7700 10 client.37134.objecter _send_command 16 2024-04-14T15:17:09.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.566+0000 7fe93aca7700 1 -- 172.21.15.90:0/1142507158 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe9341afba0 con 0x7fe93410f370 2024-04-14T15:17:09.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.568+0000 7fe931ffb700 1 -- 172.21.15.90:0/1142507158 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fe9341afba0 con 0x7fe93410f370 2024-04-14T15:17:09.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.568+0000 7fe931ffb700 10 client.37134.objecter ms_dispatch 0x7fe934001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:17:09.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.568+0000 7fe931ffb700 10 client.37134.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:17:09.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.568+0000 7fe931ffb700 20 client.37134.objecter put_session s=0x7fe9341af5e0 osd=7 4 2024-04-14T15:17:09.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.568+0000 7fe931ffb700 15 client.37134.objecter _session_command_op_remove 7 16 2024-04-14T15:17:09.569 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:17:09.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.569+0000 7fe93aca7700 10 librados: watch_flush enter 2024-04-14T15:17:09.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.569+0000 7fe93aca7700 10 librados: watch_flush exit 2024-04-14T15:17:09.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.569+0000 7fe93aca7700 10 client.37134.objecter close_session for osd.0 2024-04-14T15:17:09.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.569+0000 7fe93aca7700 1 -- 172.21.15.90:0/1142507158 >> v1:172.21.15.90:6801/3761806885 conn(0x7fe9341a91a0 legacy=0x7fe9341a9580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:09.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.569+0000 7fe93aca7700 20 client.37134.objecter put_session s=0x7fe9340693d0 osd=0 1 2024-04-14T15:17:09.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.569+0000 7fe93aca7700 10 client.37134.objecter close_session for osd.1 2024-04-14T15:17:09.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.569+0000 7fe93aca7700 1 -- 172.21.15.90:0/1142507158 >> v1:172.21.15.90:6805/4170361699 conn(0x7fe9340fdcf0 legacy=0x7fe93402ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:09.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.569+0000 7fe93aca7700 20 client.37134.objecter put_session s=0x7fe9340fd670 osd=1 1 2024-04-14T15:17:09.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.569+0000 7fe93aca7700 10 client.37134.objecter close_session for osd.2 2024-04-14T15:17:09.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.569+0000 7fe93aca7700 1 -- 172.21.15.90:0/1142507158 >> v1:172.21.15.90:6809/1686184251 conn(0x7fe934111210 legacy=0x7fe9341b2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:09.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.569+0000 7fe93aca7700 20 client.37134.objecter put_session s=0x7fe9341110a0 osd=2 1 2024-04-14T15:17:09.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.569+0000 7fe93aca7700 10 client.37134.objecter close_session for osd.3 2024-04-14T15:17:09.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.569+0000 7fe93aca7700 1 -- 172.21.15.90:0/1142507158 >> v1:172.21.15.90:6813/2270166774 conn(0x7fe9340632b0 legacy=0x7fe9341b6050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:09.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.569+0000 7fe93aca7700 20 client.37134.objecter put_session s=0x7fe934063140 osd=3 1 2024-04-14T15:17:09.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.569+0000 7fe93aca7700 10 client.37134.objecter close_session for osd.4 2024-04-14T15:17:09.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.569+0000 7fe93aca7700 1 -- 172.21.15.90:0/1142507158 >> v1:172.21.15.92:6800/3887549733 conn(0x7fe9341bb2f0 legacy=0x7fe9341bd750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:09.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.569+0000 7fe93aca7700 20 client.37134.objecter put_session s=0x7fe9341b05d0 osd=4 1 2024-04-14T15:17:09.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.569+0000 7fe93aca7700 10 client.37134.objecter close_session for osd.5 2024-04-14T15:17:09.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.569+0000 7fe93aca7700 1 -- 172.21.15.90:0/1142507158 >> v1:172.21.15.92:6804/3244561440 conn(0x7fe9341bf940 legacy=0x7fe9341bfda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:09.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.569+0000 7fe93aca7700 20 client.37134.objecter put_session s=0x7fe9341bf220 osd=5 1 2024-04-14T15:17:09.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.569+0000 7fe93aca7700 10 client.37134.objecter close_session for osd.6 2024-04-14T15:17:09.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.569+0000 7fe93aca7700 1 -- 172.21.15.90:0/1142507158 >> v1:172.21.15.92:6808/636125963 conn(0x7fe9341c3f20 legacy=0x7fe9341c6310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:09.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.569+0000 7fe93aca7700 20 client.37134.objecter put_session s=0x7fe9341c04e0 osd=6 1 2024-04-14T15:17:09.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.569+0000 7fe93aca7700 10 client.37134.objecter close_session for osd.7 2024-04-14T15:17:09.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.569+0000 7fe93aca7700 1 -- 172.21.15.90:0/1142507158 >> v1:172.21.15.92:6812/1773706716 conn(0x7fe93410f370 legacy=0x7fe9341cabc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:09.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.569+0000 7fe93aca7700 20 client.37134.objecter put_session s=0x7fe9341af5e0 osd=7 1 2024-04-14T15:17:09.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.569+0000 7fe93aca7700 20 client.37134.objecter shutdown clearing up homeless session... 2024-04-14T15:17:09.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.569+0000 7fe93aca7700 10 client.37134.objecter successfully canceled tick 2024-04-14T15:17:09.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.569+0000 7fe939244700 1 -- 172.21.15.90:0/1142507158 reap_dead start 2024-04-14T15:17:09.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.569+0000 7fe939244700 1 -- 172.21.15.90:0/1142507158 reap_dead start 2024-04-14T15:17:09.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.569+0000 7fe939244700 1 -- 172.21.15.90:0/1142507158 reap_dead start 2024-04-14T15:17:09.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.569+0000 7fe939244700 1 -- 172.21.15.90:0/1142507158 reap_dead start 2024-04-14T15:17:09.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.569+0000 7fe939244700 1 -- 172.21.15.90:0/1142507158 reap_dead start 2024-04-14T15:17:09.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.569+0000 7fe93aca7700 1 -- 172.21.15.90:0/1142507158 >> v1:172.21.15.90:6800/420935427 conn(0x7fe924071120 legacy=0x7fe9240735e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:09.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.569+0000 7fe93aca7700 1 -- 172.21.15.90:0/1142507158 >> v1:172.21.15.90:6789/0 conn(0x7fe93410a610 legacy=0x7fe93410c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:09.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.569+0000 7fe93aca7700 1 -- 172.21.15.90:0/1142507158 shutdown_connections 2024-04-14T15:17:09.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.569+0000 7fe93aca7700 1 -- 172.21.15.90:0/1142507158 >> 172.21.15.90:0/1142507158 conn(0x7fe9340fd290 msgr2=0x7fe9341000b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:17:09.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.569+0000 7fe93aca7700 1 -- 172.21.15.90:0/1142507158 shutdown_connections 2024-04-14T15:17:09.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.569+0000 7fe93aca7700 1 -- 172.21.15.90:0/1142507158 wait complete. 2024-04-14T15:17:09.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:09.569+0000 7fe93aca7700 1 librados: shutdown 2024-04-14T15:17:09.573 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:17:09.573 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:17:09.573 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:17:09.573 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:17:09.573 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:17:09.573 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:17:09.574 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:17:09.574 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:17:09.582 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:17:09.608 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:09.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:09.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:17:09.710 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:17:09.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:09.736 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:09.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:17:09.837 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:17:09.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:09.861 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:09.861 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:17:09.962 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:17:09.990 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:09.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:09.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:17:10.092 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:17:10.120 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:10.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:10.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:17:10.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:09 smithi090 ceph-mon[41670]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:17:10.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:09 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/784308691' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi090-93363-61"}]': finished 2024-04-14T15:17:10.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:09 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-47","var": "hit_set_type","val": "bloom"}]': finished 2024-04-14T15:17:10.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:09 smithi090 ceph-mon[41670]: osdmap e361: 8 total, 8 up, 8 in 2024-04-14T15:17:10.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:09 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-14T15:17:10.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:09 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-14T15:17:10.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:09 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3041873456' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi090-93363-62"}]: dispatch 2024-04-14T15:17:10.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:09 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3041873456' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi090-93363-62"}]: dispatch 2024-04-14T15:17:10.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:09 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3041873456' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi090-93363-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:17:10.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:09 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:17:10.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:09 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:17:10.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:09 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1142507158' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:17:10.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:09 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-47","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-04-14T15:17:10.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:09 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3041873456' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi090-93363-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:17:10.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:09 smithi090 ceph-mon[41670]: osdmap e362: 8 total, 8 up, 8 in 2024-04-14T15:17:10.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:09 smithi090 ceph-mon[30399]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:17:10.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:09 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/784308691' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi090-93363-61"}]': finished 2024-04-14T15:17:10.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:09 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-47","var": "hit_set_type","val": "bloom"}]': finished 2024-04-14T15:17:10.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:09 smithi090 ceph-mon[30399]: osdmap e361: 8 total, 8 up, 8 in 2024-04-14T15:17:10.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:09 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-14T15:17:10.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:09 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-14T15:17:10.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:09 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3041873456' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi090-93363-62"}]: dispatch 2024-04-14T15:17:10.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:09 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3041873456' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi090-93363-62"}]: dispatch 2024-04-14T15:17:10.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:09 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3041873456' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi090-93363-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:17:10.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:09 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:17:10.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:09 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:17:10.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:09 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1142507158' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:17:10.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:09 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-47","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-04-14T15:17:10.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:09 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3041873456' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi090-93363-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:17:10.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:09 smithi090 ceph-mon[30399]: osdmap e362: 8 total, 8 up, 8 in 2024-04-14T15:17:10.221 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:17:10.250 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:10.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:10.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:17:10.347 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-14T15:17:10.348 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:17:10.352 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:17:10.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:09 smithi092 ceph-mon[33966]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:17:10.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:09 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/784308691' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi090-93363-61"}]': finished 2024-04-14T15:17:10.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:09 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-47","var": "hit_set_type","val": "bloom"}]': finished 2024-04-14T15:17:10.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:09 smithi092 ceph-mon[33966]: osdmap e361: 8 total, 8 up, 8 in 2024-04-14T15:17:10.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:09 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-14T15:17:10.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:09 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-14T15:17:10.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:09 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3041873456' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi090-93363-62"}]: dispatch 2024-04-14T15:17:10.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:09 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3041873456' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi090-93363-62"}]: dispatch 2024-04-14T15:17:10.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:09 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3041873456' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi090-93363-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:17:10.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:09 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:17:10.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:09 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:17:10.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:09 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1142507158' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:17:10.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:09 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-47","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-04-14T15:17:10.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:09 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3041873456' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi090-93363-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:17:10.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:09 smithi092 ceph-mon[33966]: osdmap e362: 8 total, 8 up, 8 in 2024-04-14T15:17:10.378 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:10.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:10.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:17:10.480 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:17:10.504 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:10.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:10.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:17:10.520 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-14T15:17:10.606 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:17:10.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:10.630 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:10.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:17:10.731 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:17:10.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:10.756 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:10.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:17:10.857 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:17:10.881 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:10.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:10.882 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:17:10.941 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:10 smithi090 ceph-mon[41670]: pgmap v488: 238 pgs: 238 active+clean; 8.4 MiB data, 597 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:17:10.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:10 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3041873456' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi090-93363-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi090-93363-62"}]: dispatch 2024-04-14T15:17:10.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:10 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-04-14T15:17:10.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:10 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-04-14T15:17:10.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:10 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-47","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-04-14T15:17:10.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:10 smithi090 ceph-mon[30399]: pgmap v488: 238 pgs: 238 active+clean; 8.4 MiB data, 597 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:17:10.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:10 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3041873456' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi090-93363-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi090-93363-62"}]: dispatch 2024-04-14T15:17:10.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:10 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-04-14T15:17:10.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:10 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-04-14T15:17:10.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:10 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-47","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-04-14T15:17:10.983 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:17:11.010 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:11.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:11.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:17:11.112 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:17:11.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:11.136 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:11.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:17:11.237 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:17:11.267 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:11.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:11.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:17:11.369 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:17:11.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:10 smithi092 ceph-mon[33966]: pgmap v488: 238 pgs: 238 active+clean; 8.4 MiB data, 597 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:17:11.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:10 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3041873456' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi090-93363-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi090-93363-62"}]: dispatch 2024-04-14T15:17:11.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:10 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-04-14T15:17:11.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:10 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-04-14T15:17:11.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:10 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-47","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-04-14T15:17:11.396 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:11.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:11.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:17:11.498 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:17:11.522 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:11.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:11.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:17:11.624 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:17:11.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:11.650 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:11.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:17:11.751 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:17:11.775 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:11.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:11.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:17:11.861 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-14T15:17:11.864 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:17:11.878 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:17:11.911 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:11.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:11.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:17:12.013 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:17:12.036 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:12.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:12.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:17:12.138 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:17:12.161 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:12.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:12.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:17:12.168 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:11 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-04-14T15:17:12.168 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:11 smithi090 ceph-mon[41670]: osdmap e363: 8 total, 8 up, 8 in 2024-04-14T15:17:12.168 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:11 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-04-14T15:17:12.168 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:11 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3041873456' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi090-93363-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi090-93363-62"}]': finished 2024-04-14T15:17:12.168 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:11 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-47","var": "hit_set_search_last_n","val": "1"}]': finished 2024-04-14T15:17:12.168 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:11 smithi090 ceph-mon[41670]: osdmap e364: 8 total, 8 up, 8 in 2024-04-14T15:17:12.169 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:11 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-04-14T15:17:12.169 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:11 smithi090 ceph-mon[30399]: osdmap e363: 8 total, 8 up, 8 in 2024-04-14T15:17:12.169 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:11 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-04-14T15:17:12.169 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:11 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3041873456' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi090-93363-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi090-93363-62"}]': finished 2024-04-14T15:17:12.169 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:11 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-47","var": "hit_set_search_last_n","val": "1"}]': finished 2024-04-14T15:17:12.169 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:11 smithi090 ceph-mon[30399]: osdmap e364: 8 total, 8 up, 8 in 2024-04-14T15:17:12.264 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:17:12.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:12.289 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:12.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:17:12.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:11 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-04-14T15:17:12.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:11 smithi092 ceph-mon[33966]: osdmap e363: 8 total, 8 up, 8 in 2024-04-14T15:17:12.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:11 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-04-14T15:17:12.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:11 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3041873456' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi090-93363-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi090-93363-62"}]': finished 2024-04-14T15:17:12.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:11 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-47","var": "hit_set_search_last_n","val": "1"}]': finished 2024-04-14T15:17:12.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:11 smithi092 ceph-mon[33966]: osdmap e364: 8 total, 8 up, 8 in 2024-04-14T15:17:12.390 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:17:12.417 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:12.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:12.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:17:12.520 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:17:12.543 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:12.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:12.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:17:12.645 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:17:12.669 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:12.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:12.670 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:17:12.771 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:17:12.798 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:12.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:12.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:17:12.879 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:17:12.900 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:17:12.925 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:12.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:12.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:17:13.027 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:17:13.051 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:13.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:13.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:17:13.152 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:17:13.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:13.178 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:13.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:17:13.190 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-14T15:17:13.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:12 smithi090 ceph-mon[41670]: pgmap v491: 238 pgs: 238 active+clean; 8.4 MiB data, 597 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:17:13.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:12 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:17:13.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:12 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:17:13.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:12 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:17:13.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:12 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-47"}]: dispatch 2024-04-14T15:17:13.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:12 smithi090 ceph-mon[41670]: osdmap e365: 8 total, 8 up, 8 in 2024-04-14T15:17:13.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:12 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-47"}]: dispatch 2024-04-14T15:17:13.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:12 smithi090 ceph-mon[30399]: pgmap v491: 238 pgs: 238 active+clean; 8.4 MiB data, 597 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:17:13.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:12 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:17:13.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:12 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:17:13.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:12 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:17:13.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:12 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-47"}]: dispatch 2024-04-14T15:17:13.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:12 smithi090 ceph-mon[30399]: osdmap e365: 8 total, 8 up, 8 in 2024-04-14T15:17:13.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:12 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-47"}]: dispatch 2024-04-14T15:17:13.280 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:17:13.304 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:13.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:13.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:17:13.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:12 smithi092 ceph-mon[33966]: pgmap v491: 238 pgs: 238 active+clean; 8.4 MiB data, 597 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:17:13.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:12 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:17:13.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:12 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:17:13.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:12 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:17:13.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:12 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-47"}]: dispatch 2024-04-14T15:17:13.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:12 smithi092 ceph-mon[33966]: osdmap e365: 8 total, 8 up, 8 in 2024-04-14T15:17:13.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:12 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-47"}]: dispatch 2024-04-14T15:17:13.406 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:17:13.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:13.429 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:13.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:17:13.530 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:17:13.552 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:17:13.558 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:13.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:13.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:17:13.660 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:17:13.687 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:13.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:13.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:17:13.789 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:17:13.790 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:17:13.815 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:13.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:13.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:17:13.917 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:17:13.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:13.942 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:13.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:17:14.043 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:17:14.049 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:17:13 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:17:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:17:14.071 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:14.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:14.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:17:14.173 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:17:14.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:14.198 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:14.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:17:14.299 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:17:14.325 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:14.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:14.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:17:14.427 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:17:14.451 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:14.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:14.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:17:14.553 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:17:14.579 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:14.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:14.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:17:14.681 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:17:14.708 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:14.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:14.709 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:17:14.810 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:17:14.837 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:14.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:14.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:17:14.939 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:17:14.965 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:14.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:14.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:17:15.067 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:17:15.094 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:15.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:15.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:17:15.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:14 smithi090 ceph-mon[30399]: pgmap v494: 246 pgs: 8 unknown, 238 active+clean; 8.4 MiB data, 597 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:17:15.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:14 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-47"}]': finished 2024-04-14T15:17:15.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:14 smithi090 ceph-mon[30399]: osdmap e366: 8 total, 8 up, 8 in 2024-04-14T15:17:15.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:14 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3041873456' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi090-93363-62"}]: dispatch 2024-04-14T15:17:15.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:14 smithi090 ceph-mon[30399]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:17:15.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:14 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:17:15.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:14 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:17:15.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:14 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-47"}]: dispatch 2024-04-14T15:17:15.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:14 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-47"}]: dispatch 2024-04-14T15:17:15.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:14 smithi090 ceph-mon[41670]: pgmap v494: 246 pgs: 8 unknown, 238 active+clean; 8.4 MiB data, 597 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:17:15.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:14 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-47"}]': finished 2024-04-14T15:17:15.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:14 smithi090 ceph-mon[41670]: osdmap e366: 8 total, 8 up, 8 in 2024-04-14T15:17:15.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:14 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3041873456' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi090-93363-62"}]: dispatch 2024-04-14T15:17:15.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:14 smithi090 ceph-mon[41670]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:17:15.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:14 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:17:15.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:14 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:17:15.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:14 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-47"}]: dispatch 2024-04-14T15:17:15.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:14 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-47"}]: dispatch 2024-04-14T15:17:15.196 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:17:15.223 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:15.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:15.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:17:15.325 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:17:15.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:15.352 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:15.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:17:15.372 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:17:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:17:15.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:14 smithi092 ceph-mon[33966]: pgmap v494: 246 pgs: 8 unknown, 238 active+clean; 8.4 MiB data, 597 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:17:15.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:14 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-47"}]': finished 2024-04-14T15:17:15.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:14 smithi092 ceph-mon[33966]: osdmap e366: 8 total, 8 up, 8 in 2024-04-14T15:17:15.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:14 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3041873456' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi090-93363-62"}]: dispatch 2024-04-14T15:17:15.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:14 smithi092 ceph-mon[33966]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:17:15.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:14 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:17:15.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:14 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:17:15.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:14 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-47"}]: dispatch 2024-04-14T15:17:15.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:14 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-47"}]: dispatch 2024-04-14T15:17:15.453 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:17:15.478 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:15.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:15.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:17:15.580 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:17:15.604 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:15.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:15.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:17:15.706 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:17:15.732 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:15.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:15.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:17:15.834 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:17:15.859 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:15.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:15.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:17:15.961 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:17:15.988 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:15.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:15.989 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:17:16.073 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:15 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3041873456' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi090-93363-62"}]': finished 2024-04-14T15:17:16.073 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:15 smithi090 ceph-mon[41670]: osdmap e367: 8 total, 8 up, 8 in 2024-04-14T15:17:16.073 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:15 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3041873456' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi090-93363-62"}]: dispatch 2024-04-14T15:17:16.074 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:15 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3041873456' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi090-93363-62"}]': finished 2024-04-14T15:17:16.074 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:15 smithi090 ceph-mon[41670]: osdmap e368: 8 total, 8 up, 8 in 2024-04-14T15:17:16.074 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:15 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:17:16.074 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:15 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:17:16.074 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:15 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3041873456' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi090-93363-62"}]': finished 2024-04-14T15:17:16.074 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:15 smithi090 ceph-mon[30399]: osdmap e367: 8 total, 8 up, 8 in 2024-04-14T15:17:16.074 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:15 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3041873456' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi090-93363-62"}]: dispatch 2024-04-14T15:17:16.074 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:15 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3041873456' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi090-93363-62"}]': finished 2024-04-14T15:17:16.074 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:15 smithi090 ceph-mon[30399]: osdmap e368: 8 total, 8 up, 8 in 2024-04-14T15:17:16.074 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:15 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:17:16.074 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:15 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:17:16.090 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:17:16.114 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:16.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:16.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:17:16.215 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:17:16.240 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:16.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:16.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:17:16.342 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:17:16.366 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:16.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:16.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:17:16.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:15 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3041873456' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi090-93363-62"}]': finished 2024-04-14T15:17:16.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:15 smithi092 ceph-mon[33966]: osdmap e367: 8 total, 8 up, 8 in 2024-04-14T15:17:16.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:15 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3041873456' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi090-93363-62"}]: dispatch 2024-04-14T15:17:16.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:15 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3041873456' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi090-93363-62"}]': finished 2024-04-14T15:17:16.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:15 smithi092 ceph-mon[33966]: osdmap e368: 8 total, 8 up, 8 in 2024-04-14T15:17:16.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:15 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:17:16.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:15 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:17:16.468 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:17:16.493 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:16.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:16.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:17:16.594 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:17:16.619 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:16.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:16.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:17:16.721 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:17:16.746 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:16.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:16.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:17:16.848 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:17:16.873 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:16.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:16.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:17:16.975 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:17:16.999 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:17.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:17.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:17:17.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.015+0000 7f7360d65700 1 -- 172.21.15.90:0/820093887 >> v1:172.21.15.90:6790/0 conn(0x7f735c108980 legacy=0x7f735c108d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:17.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.015+0000 7f7360d65700 1 -- 172.21.15.90:0/820093887 shutdown_connections 2024-04-14T15:17:17.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.015+0000 7f7360d65700 1 -- 172.21.15.90:0/820093887 >> 172.21.15.90:0/820093887 conn(0x7f735c076aa0 msgr2=0x7f735c076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:17:17.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.015+0000 7f7360d65700 1 -- 172.21.15.90:0/820093887 shutdown_connections 2024-04-14T15:17:17.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.015+0000 7f7360d65700 1 -- 172.21.15.90:0/820093887 wait complete. 2024-04-14T15:17:17.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.015+0000 7f7360d65700 1 librados: starting msgr at 2024-04-14T15:17:17.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.015+0000 7f7360d65700 1 librados: starting objecter 2024-04-14T15:17:17.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.016+0000 7f7360d65700 1 Processor -- start 2024-04-14T15:17:17.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.016+0000 7f7360d65700 1 -- start start 2024-04-14T15:17:17.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.016+0000 7f7360d65700 1 librados: setting wanted keys 2024-04-14T15:17:17.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.016+0000 7f7360d65700 1 librados: calling monclient init 2024-04-14T15:17:17.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.016+0000 7f7360d65700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f735c1a7a40 con 0x7f735c108980 2024-04-14T15:17:17.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.016+0000 7f7360d65700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f735c1a8c60 con 0x7f735c10b6f0 2024-04-14T15:17:17.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.016+0000 7f7360d65700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f735c1a9ea0 con 0x7f735c10f2f0 2024-04-14T15:17:17.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.016+0000 7f735a59c700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f735c108980 0x7f735c107de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:56716/0 (socket says 172.21.15.90:56716) 2024-04-14T15:17:17.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.016+0000 7f735a59c700 1 -- 172.21.15.90:0/345227228 learned_addr learned my addr 172.21.15.90:0/345227228 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:17:17.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.016+0000 7f734b7fe700 10 client.?.objecter ms_handle_connect 0x7f735c108980 2024-04-14T15:17:17.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.016+0000 7f734b7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:17:17.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.016+0000 7f734b7fe700 10 client.?.objecter ms_handle_connect 0x7f735c10f2f0 2024-04-14T15:17:17.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.016+0000 7f734b7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:17:17.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.016+0000 7f734b7fe700 1 -- 172.21.15.90:0/345227228 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3454708051 0 0) 0x7f735c1a7a40 con 0x7f735c108980 2024-04-14T15:17:17.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.016+0000 7f734b7fe700 1 -- 172.21.15.90:0/345227228 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f73440039c0 con 0x7f735c108980 2024-04-14T15:17:17.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.016+0000 7f734b7fe700 10 client.?.objecter ms_handle_connect 0x7f735c10b6f0 2024-04-14T15:17:17.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.016+0000 7f734b7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:17:17.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.016+0000 7f734b7fe700 1 -- 172.21.15.90:0/345227228 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1095529482 0 0) 0x7f735c1a9ea0 con 0x7f735c10f2f0 2024-04-14T15:17:17.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.016+0000 7f734b7fe700 1 -- 172.21.15.90:0/345227228 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f735c1a7a40 con 0x7f735c10f2f0 2024-04-14T15:17:17.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.016+0000 7f734b7fe700 1 -- 172.21.15.90:0/345227228 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 507183776 0 0) 0x7f735c1a8c60 con 0x7f735c10b6f0 2024-04-14T15:17:17.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.016+0000 7f734b7fe700 1 -- 172.21.15.90:0/345227228 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f735c1a9ea0 con 0x7f735c10b6f0 2024-04-14T15:17:17.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.017+0000 7f734b7fe700 1 -- 172.21.15.90:0/345227228 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4025307615 0 0) 0x7f73440039c0 con 0x7f735c108980 2024-04-14T15:17:17.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.017+0000 7f734b7fe700 1 -- 172.21.15.90:0/345227228 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f735c1a8c60 con 0x7f735c108980 2024-04-14T15:17:17.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.017+0000 7f734b7fe700 1 -- 172.21.15.90:0/345227228 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f734c004470 con 0x7f735c108980 2024-04-14T15:17:17.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.017+0000 7f734b7fe700 1 -- 172.21.15.90:0/345227228 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3333281582 0 0) 0x7f735c1a9ea0 con 0x7f735c10b6f0 2024-04-14T15:17:17.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.017+0000 7f734b7fe700 1 -- 172.21.15.90:0/345227228 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f73440039c0 con 0x7f735c10b6f0 2024-04-14T15:17:17.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.017+0000 7f734b7fe700 1 -- 172.21.15.90:0/345227228 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1814533819 0 0) 0x7f735c1a7a40 con 0x7f735c10f2f0 2024-04-14T15:17:17.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.017+0000 7f734b7fe700 1 -- 172.21.15.90:0/345227228 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f735c1a9ea0 con 0x7f735c10f2f0 2024-04-14T15:17:17.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.017+0000 7f734b7fe700 1 -- 172.21.15.90:0/345227228 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f7350001870 con 0x7f735c10b6f0 2024-04-14T15:17:17.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.017+0000 7f734b7fe700 1 -- 172.21.15.90:0/345227228 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f73540013f0 con 0x7f735c10f2f0 2024-04-14T15:17:17.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.017+0000 7f734b7fe700 1 -- 172.21.15.90:0/345227228 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2450603457 0 0) 0x7f73440039c0 con 0x7f735c10b6f0 2024-04-14T15:17:17.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.017+0000 7f734b7fe700 1 -- 172.21.15.90:0/345227228 >> v1:172.21.15.90:6790/0 conn(0x7f735c10f2f0 legacy=0x7f735c1a6100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:17.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.017+0000 7f734b7fe700 1 -- 172.21.15.90:0/345227228 >> v1:172.21.15.90:6789/0 conn(0x7f735c108980 legacy=0x7f735c107de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:17.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.017+0000 7f734b7fe700 1 -- 172.21.15.90:0/345227228 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f735c1ab0e0 con 0x7f735c10b6f0 2024-04-14T15:17:17.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.017+0000 7f7360d65700 1 -- 172.21.15.90:0/345227228 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f735c1aa0d0 con 0x7f735c10b6f0 2024-04-14T15:17:17.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.017+0000 7f7360d65700 10 client.48982.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:17:17.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.017+0000 7f7360d65700 1 -- 172.21.15.90:0/345227228 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f735c1aa590 con 0x7f735c10b6f0 2024-04-14T15:17:17.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.017+0000 7f7360d65700 1 librados: init done 2024-04-14T15:17:17.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.017+0000 7f734b7fe700 1 -- 172.21.15.90:0/345227228 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f7350003b40 con 0x7f735c10b6f0 2024-04-14T15:17:17.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.017+0000 7f734b7fe700 1 -- 172.21.15.90:0/345227228 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f7350004ab0 con 0x7f735c10b6f0 2024-04-14T15:17:17.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.018+0000 7f7360d65700 1 -- 172.21.15.90:0/345227228 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f733c005120 con 0x7f735c10b6f0 2024-04-14T15:17:17.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.018+0000 7f734b7fe700 1 -- 172.21.15.90:0/345227228 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f73500036a0 con 0x7f735c10b6f0 2024-04-14T15:17:17.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.019+0000 7f734b7fe700 1 -- 172.21.15.90:0/345227228 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(369..369 src has 51..369) ==== 8471+0+0 (unknown 3305534353 0 0) 0x7f7350089880 con 0x7f735c10b6f0 2024-04-14T15:17:17.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.019+0000 7f734b7fe700 10 client.48982.objecter ms_dispatch 0x7f735c001550 osd_map(369..369 src has 51..369) 2024-04-14T15:17:17.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.019+0000 7f734b7fe700 3 client.48982.objecter handle_osd_map got epochs [369,369] > 0 2024-04-14T15:17:17.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.019+0000 7f734b7fe700 3 client.48982.objecter handle_osd_map decoding full epoch 369 2024-04-14T15:17:17.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.019+0000 7f734b7fe700 20 client.48982.objecter dump_active .. 0 homeless 2024-04-14T15:17:17.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.019+0000 7f734b7fe700 10 client.48982.objecter ms_handle_connect 0x7f7344071410 2024-04-14T15:17:17.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.024+0000 7f734b7fe700 1 -- 172.21.15.90:0/345227228 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f7350056160 con 0x7f735c10b6f0 2024-04-14T15:17:17.101 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:17:17.126 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:17.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:17.127 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:17:17.157 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:16 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:17:17.157 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:16 smithi090 ceph-mon[41670]: pgmap v497: 206 pgs: 206 active+clean; 8.4 MiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-14T15:17:17.157 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:16 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3541596265' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi090-93363-63"}]: dispatch 2024-04-14T15:17:17.157 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:16 smithi090 ceph-mon[41670]: from='client.48976 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi090-93363-63"}]: dispatch 2024-04-14T15:17:17.157 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:16 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3541596265' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi090-93363-63"}]: dispatch 2024-04-14T15:17:17.158 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:16 smithi090 ceph-mon[41670]: from='client.48976 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi090-93363-63"}]: dispatch 2024-04-14T15:17:17.158 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:16 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3541596265' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi090-93363-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:17:17.158 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:16 smithi090 ceph-mon[41670]: from='client.48976 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi090-93363-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:17:17.158 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:16 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:17:17.158 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:16 smithi090 ceph-mon[41670]: from='client.48976 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi090-93363-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:17:17.158 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:16 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3541596265' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi090-93363-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi090-93363-63"}]: dispatch 2024-04-14T15:17:17.158 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:16 smithi090 ceph-mon[41670]: osdmap e369: 8 total, 8 up, 8 in 2024-04-14T15:17:17.158 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:16 smithi090 ceph-mon[41670]: from='client.48976 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi090-93363-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi090-93363-63"}]: dispatch 2024-04-14T15:17:17.158 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:16 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:17:17.158 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:16 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:17:17.159 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:16 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:17:17.159 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:16 smithi090 ceph-mon[30399]: pgmap v497: 206 pgs: 206 active+clean; 8.4 MiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-14T15:17:17.159 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:16 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3541596265' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi090-93363-63"}]: dispatch 2024-04-14T15:17:17.159 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:16 smithi090 ceph-mon[30399]: from='client.48976 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi090-93363-63"}]: dispatch 2024-04-14T15:17:17.159 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:16 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3541596265' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi090-93363-63"}]: dispatch 2024-04-14T15:17:17.159 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:16 smithi090 ceph-mon[30399]: from='client.48976 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi090-93363-63"}]: dispatch 2024-04-14T15:17:17.159 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:16 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3541596265' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi090-93363-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:17:17.159 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:16 smithi090 ceph-mon[30399]: from='client.48976 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi090-93363-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:17:17.159 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:16 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:17:17.159 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:16 smithi090 ceph-mon[30399]: from='client.48976 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi090-93363-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:17:17.159 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:16 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3541596265' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi090-93363-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi090-93363-63"}]: dispatch 2024-04-14T15:17:17.159 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:16 smithi090 ceph-mon[30399]: osdmap e369: 8 total, 8 up, 8 in 2024-04-14T15:17:17.159 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:16 smithi090 ceph-mon[30399]: from='client.48976 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi090-93363-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi090-93363-63"}]: dispatch 2024-04-14T15:17:17.160 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:16 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:17:17.160 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:16 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:17:17.227 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:17:17.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.247+0000 7f7360d65700 1 -- 172.21.15.90:0/345227228 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd reweight", "id": 2, "weight": 1.0} v 0) -- 0x7f733c005410 con 0x7f735c10b6f0 2024-04-14T15:17:17.251 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:17.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:17.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:17:17.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.254+0000 7f5efdf97700 1 -- 172.21.15.90:0/1770530957 >> v1:172.21.15.90:6789/0 conn(0x7f5ef8107880 legacy=0x7f5ef8107c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:17.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.254+0000 7f5efdf97700 1 -- 172.21.15.90:0/1770530957 shutdown_connections 2024-04-14T15:17:17.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.254+0000 7f5efdf97700 1 -- 172.21.15.90:0/1770530957 >> 172.21.15.90:0/1770530957 conn(0x7f5ef8076aa0 msgr2=0x7f5ef8076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:17:17.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.254+0000 7f5efdf97700 1 -- 172.21.15.90:0/1770530957 shutdown_connections 2024-04-14T15:17:17.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.254+0000 7f5efdf97700 1 -- 172.21.15.90:0/1770530957 wait complete. 2024-04-14T15:17:17.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.254+0000 7f5efdf97700 1 librados: starting msgr at 2024-04-14T15:17:17.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.254+0000 7f5efdf97700 1 librados: starting objecter 2024-04-14T15:17:17.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.255+0000 7f5efdf97700 1 Processor -- start 2024-04-14T15:17:17.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.255+0000 7f5efdf97700 1 -- start start 2024-04-14T15:17:17.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.255+0000 7f5efdf97700 1 librados: setting wanted keys 2024-04-14T15:17:17.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.255+0000 7f5efdf97700 1 librados: calling monclient init 2024-04-14T15:17:17.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.255+0000 7f5efdf97700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5ef8101c90 con 0x7f5ef810b750 2024-04-14T15:17:17.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.255+0000 7f5efdf97700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5ef8101ea0 con 0x7f5ef810f320 2024-04-14T15:17:17.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.255+0000 7f5efdf97700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5ef81a9f50 con 0x7f5ef8107880 2024-04-14T15:17:17.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.255+0000 7f5eeeffd700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f5ef810b750 0x7f5ef80ff1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:56740/0 (socket says 172.21.15.90:56740) 2024-04-14T15:17:17.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.255+0000 7f5eeeffd700 1 -- 172.21.15.90:0/3622018062 learned_addr learned my addr 172.21.15.90:0/3622018062 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:17:17.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.255+0000 7f5ef77fe700 1 --1- 172.21.15.90:0/3622018062 >> v1:172.21.15.90:6790/0 conn(0x7f5ef8107880 0x7f5ef80feae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:42856/0 (socket says 172.21.15.90:42856) 2024-04-14T15:17:17.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.255+0000 7f5ef57fa700 10 client.?.objecter ms_handle_connect 0x7f5ef810b750 2024-04-14T15:17:17.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.255+0000 7f5ef57fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:17:17.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.255+0000 7f5ef57fa700 10 client.?.objecter ms_handle_connect 0x7f5ef8107880 2024-04-14T15:17:17.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.255+0000 7f5ef57fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:17:17.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.255+0000 7f5ef57fa700 10 client.?.objecter ms_handle_connect 0x7f5ef810f320 2024-04-14T15:17:17.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.255+0000 7f5ef57fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:17:17.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.255+0000 7f5ef57fa700 1 -- 172.21.15.90:0/3622018062 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2883609683 0 0) 0x7f5ef8101c90 con 0x7f5ef810b750 2024-04-14T15:17:17.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.255+0000 7f5ef57fa700 1 -- 172.21.15.90:0/3622018062 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ee40035b0 con 0x7f5ef810b750 2024-04-14T15:17:17.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.255+0000 7f5ef57fa700 1 -- 172.21.15.90:0/3622018062 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3555495851 0 0) 0x7f5ef81a9f50 con 0x7f5ef8107880 2024-04-14T15:17:17.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.255+0000 7f5ef57fa700 1 -- 172.21.15.90:0/3622018062 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ef8101c90 con 0x7f5ef8107880 2024-04-14T15:17:17.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.255+0000 7f5ef57fa700 1 -- 172.21.15.90:0/3622018062 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4061511301 0 0) 0x7f5ef8101ea0 con 0x7f5ef810f320 2024-04-14T15:17:17.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.255+0000 7f5ef57fa700 1 -- 172.21.15.90:0/3622018062 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ef81a9f50 con 0x7f5ef810f320 2024-04-14T15:17:17.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.256+0000 7f5ef57fa700 1 -- 172.21.15.90:0/3622018062 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3614323734 0 0) 0x7f5ee40035b0 con 0x7f5ef810b750 2024-04-14T15:17:17.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.256+0000 7f5ef57fa700 1 -- 172.21.15.90:0/3622018062 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5ef8101ea0 con 0x7f5ef810b750 2024-04-14T15:17:17.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.256+0000 7f5ef57fa700 1 -- 172.21.15.90:0/3622018062 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 949573153 0 0) 0x7f5ef8101c90 con 0x7f5ef8107880 2024-04-14T15:17:17.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.256+0000 7f5ef57fa700 1 -- 172.21.15.90:0/3622018062 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5ee40035b0 con 0x7f5ef8107880 2024-04-14T15:17:17.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.256+0000 7f5ef57fa700 1 -- 172.21.15.90:0/3622018062 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f5ee00018b0 con 0x7f5ef810b750 2024-04-14T15:17:17.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.256+0000 7f5ef57fa700 1 -- 172.21.15.90:0/3622018062 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 412846129 0 0) 0x7f5ef81a9f50 con 0x7f5ef810f320 2024-04-14T15:17:17.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.256+0000 7f5ef57fa700 1 -- 172.21.15.90:0/3622018062 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5ef8101c90 con 0x7f5ef810f320 2024-04-14T15:17:17.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.256+0000 7f5ef57fa700 1 -- 172.21.15.90:0/3622018062 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f5ee8001430 con 0x7f5ef8107880 2024-04-14T15:17:17.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.256+0000 7f5ef57fa700 1 -- 172.21.15.90:0/3622018062 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f5ef0001bb0 con 0x7f5ef810f320 2024-04-14T15:17:17.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.256+0000 7f5ef57fa700 1 -- 172.21.15.90:0/3622018062 <== mon.2 v1:172.21.15.90:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2893401951 0 0) 0x7f5ee40035b0 con 0x7f5ef8107880 2024-04-14T15:17:17.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.256+0000 7f5ef57fa700 1 -- 172.21.15.90:0/3622018062 >> v1:172.21.15.92:6789/0 conn(0x7f5ef810f320 legacy=0x7f5ef81a67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:17.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.256+0000 7f5ef57fa700 1 -- 172.21.15.90:0/3622018062 >> v1:172.21.15.90:6789/0 conn(0x7f5ef810b750 legacy=0x7f5ef80ff1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:17.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.256+0000 7f5ef57fa700 1 -- 172.21.15.90:0/3622018062 --> v1:172.21.15.90:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ef81ab190 con 0x7f5ef8107880 2024-04-14T15:17:17.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.256+0000 7f5efdf97700 1 -- 172.21.15.90:0/3622018062 --> v1:172.21.15.90:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5ef81aa180 con 0x7f5ef8107880 2024-04-14T15:17:17.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.256+0000 7f5efdf97700 10 client.47102.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:17:17.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.256+0000 7f5efdf97700 1 -- 172.21.15.90:0/3622018062 --> v1:172.21.15.90:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f5ef81aa6c0 con 0x7f5ef8107880 2024-04-14T15:17:17.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.256+0000 7f5efdf97700 1 librados: init done 2024-04-14T15:17:17.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.256+0000 7f5ef57fa700 1 -- 172.21.15.90:0/3622018062 <== mon.2 v1:172.21.15.90:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f5ee8003760 con 0x7f5ef8107880 2024-04-14T15:17:17.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.256+0000 7f5ef57fa700 1 -- 172.21.15.90:0/3622018062 <== mon.2 v1:172.21.15.90:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f5ee8004d60 con 0x7f5ef8107880 2024-04-14T15:17:17.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.257+0000 7f5efdf97700 1 -- 172.21.15.90:0/3622018062 --> v1:172.21.15.90:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ed8005120 con 0x7f5ef8107880 2024-04-14T15:17:17.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.258+0000 7f5ef57fa700 1 -- 172.21.15.90:0/3622018062 <== mon.2 v1:172.21.15.90:6790/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f5ee8001b70 con 0x7f5ef8107880 2024-04-14T15:17:17.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.258+0000 7f5ef57fa700 1 -- 172.21.15.90:0/3622018062 <== mon.2 v1:172.21.15.90:6790/0 8 ==== osd_map(369..369 src has 51..369) ==== 8471+0+0 (unknown 3305534353 0 0) 0x7f5ee808a8e0 con 0x7f5ef8107880 2024-04-14T15:17:17.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.258+0000 7f5ef57fa700 10 client.47102.objecter ms_dispatch 0x7f5ef8001550 osd_map(369..369 src has 51..369) 2024-04-14T15:17:17.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.258+0000 7f5ef57fa700 3 client.47102.objecter handle_osd_map got epochs [369,369] > 0 2024-04-14T15:17:17.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.258+0000 7f5ef57fa700 3 client.47102.objecter handle_osd_map decoding full epoch 369 2024-04-14T15:17:17.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.258+0000 7f5ef57fa700 20 client.47102.objecter dump_active .. 0 homeless 2024-04-14T15:17:17.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.258+0000 7f5ef57fa700 10 client.47102.objecter ms_handle_connect 0x7f5ee4071390 2024-04-14T15:17:17.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.262+0000 7f5ef57fa700 1 -- 172.21.15.90:0/3622018062 <== mon.2 v1:172.21.15.90:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f5ee8001690 con 0x7f5ef8107880 2024-04-14T15:17:17.353 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:17:17.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:16 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:17:17.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:16 smithi092 ceph-mon[33966]: pgmap v497: 206 pgs: 206 active+clean; 8.4 MiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-14T15:17:17.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:16 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3541596265' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi090-93363-63"}]: dispatch 2024-04-14T15:17:17.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:16 smithi092 ceph-mon[33966]: from='client.48976 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi090-93363-63"}]: dispatch 2024-04-14T15:17:17.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:16 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3541596265' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi090-93363-63"}]: dispatch 2024-04-14T15:17:17.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:16 smithi092 ceph-mon[33966]: from='client.48976 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi090-93363-63"}]: dispatch 2024-04-14T15:17:17.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:16 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3541596265' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi090-93363-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:17:17.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:16 smithi092 ceph-mon[33966]: from='client.48976 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi090-93363-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:17:17.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:16 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:17:17.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:16 smithi092 ceph-mon[33966]: from='client.48976 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi090-93363-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:17:17.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:16 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3541596265' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi090-93363-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi090-93363-63"}]: dispatch 2024-04-14T15:17:17.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:16 smithi092 ceph-mon[33966]: osdmap e369: 8 total, 8 up, 8 in 2024-04-14T15:17:17.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:16 smithi092 ceph-mon[33966]: from='client.48976 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi090-93363-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi090-93363-63"}]: dispatch 2024-04-14T15:17:17.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:16 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:17:17.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:16 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:17:17.377 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:17.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:17.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:17:17.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.477+0000 7f5efdf97700 1 -- 172.21.15.90:0/3622018062 --> v1:172.21.15.90:6790/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7f5ed8005410 con 0x7f5ef8107880 2024-04-14T15:17:17.480 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:17:17.504 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:17.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:17.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:17:17.606 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:17:17.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:17.631 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:17.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:17:17.731 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:17:17.755 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:17.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:17.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:17:17.857 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:17:17.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.882+0000 7f5ef57fa700 1 -- 172.21.15.90:0/3622018062 <== mon.2 v1:172.21.15.90:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v370) ==== 97+0+0 (unknown 4256235568 0 0) 0x7f5ee80571e0 con 0x7f5ef8107880 2024-04-14T15:17:17.883 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:17.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.883+0000 7f734b7fe700 1 -- 172.21.15.90:0/345227228 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 2, "weight": 1.0}]=0 reweighted osd.2 to 1 (10000) v370) ==== 113+0+0 (unknown 533122596 0 0) 0x7f7350059e10 con 0x7f735c10b6f0 2024-04-14T15:17:17.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:17.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:17:17.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.886+0000 7f5efdf97700 10 librados: watch_flush enter 2024-04-14T15:17:17.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.886+0000 7f5efdf97700 10 librados: watch_flush exit 2024-04-14T15:17:17.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.886+0000 7f5efdf97700 20 client.47102.objecter shutdown clearing up homeless session... 2024-04-14T15:17:17.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.886+0000 7f5efdf97700 10 client.47102.objecter successfully canceled tick 2024-04-14T15:17:17.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.886+0000 7f5efdf97700 1 -- 172.21.15.90:0/3622018062 >> v1:172.21.15.90:6800/420935427 conn(0x7f5ee4071390 legacy=0x7f5ee4073850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:17.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.886+0000 7f5efdf97700 1 -- 172.21.15.90:0/3622018062 >> v1:172.21.15.90:6790/0 conn(0x7f5ef8107880 legacy=0x7f5ef80feae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:17.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.886+0000 7f5efdf97700 1 -- 172.21.15.90:0/3622018062 shutdown_connections 2024-04-14T15:17:17.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.886+0000 7f5efdf97700 1 -- 172.21.15.90:0/3622018062 >> 172.21.15.90:0/3622018062 conn(0x7f5ef8076aa0 msgr2=0x7f5ef81117e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:17:17.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.886+0000 7f7360d65700 10 librados: watch_flush enter 2024-04-14T15:17:17.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.886+0000 7f7360d65700 10 librados: watch_flush exit 2024-04-14T15:17:17.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.886+0000 7f7360d65700 20 client.48982.objecter shutdown clearing up homeless session... 2024-04-14T15:17:17.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.886+0000 7f7360d65700 10 client.48982.objecter successfully canceled tick 2024-04-14T15:17:17.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.887+0000 7f5efdf97700 1 -- 172.21.15.90:0/3622018062 shutdown_connections 2024-04-14T15:17:17.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.887+0000 7f7360d65700 1 -- 172.21.15.90:0/345227228 >> v1:172.21.15.90:6800/420935427 conn(0x7f7344071410 legacy=0x7f73440738d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:17.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.887+0000 7f7360d65700 1 -- 172.21.15.90:0/345227228 >> v1:172.21.15.92:6789/0 conn(0x7f735c10b6f0 legacy=0x7f735c1a27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:17.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.887+0000 7f7360d65700 1 -- 172.21.15.90:0/345227228 shutdown_connections 2024-04-14T15:17:17.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.887+0000 7f7360d65700 1 -- 172.21.15.90:0/345227228 >> 172.21.15.90:0/345227228 conn(0x7f735c076aa0 msgr2=0x7f735c109620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:17:17.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.887+0000 7f7360d65700 1 -- 172.21.15.90:0/345227228 shutdown_connections 2024-04-14T15:17:17.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.887+0000 7f5efdf97700 1 -- 172.21.15.90:0/3622018062 wait complete. 2024-04-14T15:17:17.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.887+0000 7f7360d65700 1 -- 172.21.15.90:0/345227228 wait complete. 2024-04-14T15:17:17.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.887+0000 7f7360d65700 1 librados: shutdown 2024-04-14T15:17:17.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:17.887+0000 7f5efdf97700 1 librados: shutdown 2024-04-14T15:17:17.890 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is set 2024-04-14T15:17:17.890 INFO:teuthology.orchestra.run.smithi090.stderr:reweighted osd.2 to 1 (10000) 2024-04-14T15:17:17.985 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:17:18.009 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:18.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:18.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:17:18.110 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:17:18.136 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:18.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:18.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:17:18.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:17 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/345227228' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-14T15:17:18.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:17 smithi090 ceph-mon[41670]: from='client.48982 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-14T15:17:18.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:17 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3622018062' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:17:18.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:17 smithi090 ceph-mon[41670]: from='client.47102 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:17:18.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:17 smithi090 ceph-mon[41670]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:17:18.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:17 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-49", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:17:18.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:17 smithi090 ceph-mon[41670]: from='client.48982 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-14T15:17:18.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:17 smithi090 ceph-mon[41670]: from='client.47102 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-14T15:17:18.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:17 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-49"}]: dispatch 2024-04-14T15:17:18.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:17 smithi090 ceph-mon[41670]: osdmap e370: 8 total, 8 up, 8 in 2024-04-14T15:17:18.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:17 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-49"}]: dispatch 2024-04-14T15:17:18.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:17 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/345227228' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-14T15:17:18.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:17 smithi090 ceph-mon[30399]: from='client.48982 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-14T15:17:18.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:17 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3622018062' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:17:18.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:17 smithi090 ceph-mon[30399]: from='client.47102 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:17:18.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:17 smithi090 ceph-mon[30399]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:17:18.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:17 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-49", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:17:18.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:17 smithi090 ceph-mon[30399]: from='client.48982 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-14T15:17:18.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:17 smithi090 ceph-mon[30399]: from='client.47102 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-14T15:17:18.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:17 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-49"}]: dispatch 2024-04-14T15:17:18.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:17 smithi090 ceph-mon[30399]: osdmap e370: 8 total, 8 up, 8 in 2024-04-14T15:17:18.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:17 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-49"}]: dispatch 2024-04-14T15:17:18.238 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:17:18.263 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:18.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:18.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:17:18.365 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:17:18.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:17 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/345227228' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-14T15:17:18.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:17 smithi092 ceph-mon[33966]: from='client.48982 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-14T15:17:18.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:17 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3622018062' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:17:18.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:17 smithi092 ceph-mon[33966]: from='client.47102 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:17:18.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:17 smithi092 ceph-mon[33966]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:17:18.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:17 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-49", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:17:18.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:17 smithi092 ceph-mon[33966]: from='client.48982 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-14T15:17:18.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:17 smithi092 ceph-mon[33966]: from='client.47102 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-14T15:17:18.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:17 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-49"}]: dispatch 2024-04-14T15:17:18.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:17 smithi092 ceph-mon[33966]: osdmap e370: 8 total, 8 up, 8 in 2024-04-14T15:17:18.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:17 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-49"}]: dispatch 2024-04-14T15:17:18.391 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:18.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:18.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:17:18.493 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:17:18.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:18.518 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:18.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:17:18.619 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:17:18.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:18.645 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:18.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:17:18.746 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:17:18.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:18.772 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:18.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:17:18.873 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:17:18.906 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:18.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:18.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:17:19.008 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:17:19.034 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:19.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:19.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:17:19.135 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:17:19.159 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:19.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:19.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:17:19.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:18 smithi090 ceph-mon[41670]: pgmap v500: 238 pgs: 32 unknown, 206 active+clean; 8.4 MiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-14T15:17:19.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:18 smithi090 ceph-mon[41670]: from='client.48976 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi090-93363-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi090-93363-63"}]': finished 2024-04-14T15:17:19.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:18 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-49"}]': finished 2024-04-14T15:17:19.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:18 smithi090 ceph-mon[41670]: osdmap e371: 8 total, 8 up, 8 in 2024-04-14T15:17:19.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:18 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-49", "mode": "readproxy"}]: dispatch 2024-04-14T15:17:19.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:18 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-49", "mode": "readproxy"}]: dispatch 2024-04-14T15:17:19.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:18 smithi090 ceph-mon[30399]: pgmap v500: 238 pgs: 32 unknown, 206 active+clean; 8.4 MiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-14T15:17:19.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:18 smithi090 ceph-mon[30399]: from='client.48976 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi090-93363-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi090-93363-63"}]': finished 2024-04-14T15:17:19.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:18 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-49"}]': finished 2024-04-14T15:17:19.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:18 smithi090 ceph-mon[30399]: osdmap e371: 8 total, 8 up, 8 in 2024-04-14T15:17:19.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:18 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-49", "mode": "readproxy"}]: dispatch 2024-04-14T15:17:19.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:18 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-49", "mode": "readproxy"}]: dispatch 2024-04-14T15:17:19.261 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:17:19.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:19.287 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:19.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:17:19.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:18 smithi092 ceph-mon[33966]: pgmap v500: 238 pgs: 32 unknown, 206 active+clean; 8.4 MiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-14T15:17:19.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:18 smithi092 ceph-mon[33966]: from='client.48976 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi090-93363-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi090-93363-63"}]': finished 2024-04-14T15:17:19.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:18 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-49"}]': finished 2024-04-14T15:17:19.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:18 smithi092 ceph-mon[33966]: osdmap e371: 8 total, 8 up, 8 in 2024-04-14T15:17:19.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:18 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-49", "mode": "readproxy"}]: dispatch 2024-04-14T15:17:19.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:18 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-49", "mode": "readproxy"}]: dispatch 2024-04-14T15:17:19.388 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:17:19.411 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:19.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:19.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:17:19.513 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:17:19.540 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:19.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:19.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:17:19.642 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:17:19.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:19.670 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:19.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:17:19.771 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:17:19.798 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:19.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:19.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:17:19.900 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:17:19.926 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:19.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:19.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:17:20.029 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:17:20.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:19 smithi090 ceph-mon[41670]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:17:20.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:19 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-49", "mode": "readproxy"}]': finished 2024-04-14T15:17:20.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:19 smithi090 ceph-mon[41670]: osdmap e372: 8 total, 8 up, 8 in 2024-04-14T15:17:20.056 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:19 smithi090 ceph-mon[30399]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:17:20.056 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:19 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-49", "mode": "readproxy"}]': finished 2024-04-14T15:17:20.056 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:19 smithi090 ceph-mon[30399]: osdmap e372: 8 total, 8 up, 8 in 2024-04-14T15:17:20.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:20.057 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:20.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:17:20.081 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd reweight 3 1 2024-04-14T15:17:20.158 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:17:20.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:20.182 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:20.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:17:20.283 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:17:20.310 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:20.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:20.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:17:20.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:19 smithi092 ceph-mon[33966]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:17:20.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:19 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-49", "mode": "readproxy"}]': finished 2024-04-14T15:17:20.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:19 smithi092 ceph-mon[33966]: osdmap e372: 8 total, 8 up, 8 in 2024-04-14T15:17:20.412 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:17:20.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:20.439 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:20.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:17:20.540 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:17:20.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:20.565 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:20.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:17:20.666 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:17:20.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:20.691 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:20.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:17:20.793 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:17:20.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:20.818 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:20.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:17:20.919 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:17:20.946 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:20.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:20.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:17:21.048 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:17:21.071 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:20 smithi090 ceph-mon[41670]: pgmap v503: 246 pgs: 8 creating+peering, 5 unknown, 233 active+clean; 8.4 MiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-14T15:17:21.072 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:20 smithi090 ceph-mon[41670]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:17:21.072 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:20 smithi090 ceph-mon[41670]: osdmap e373: 8 total, 8 up, 8 in 2024-04-14T15:17:21.072 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:20 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3541596265' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi090-93363-63"}]: dispatch 2024-04-14T15:17:21.072 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:20 smithi090 ceph-mon[41670]: from='client.48976 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi090-93363-63"}]: dispatch 2024-04-14T15:17:21.072 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:20 smithi090 ceph-mon[30399]: pgmap v503: 246 pgs: 8 creating+peering, 5 unknown, 233 active+clean; 8.4 MiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-14T15:17:21.072 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:20 smithi090 ceph-mon[30399]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:17:21.072 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:20 smithi090 ceph-mon[30399]: osdmap e373: 8 total, 8 up, 8 in 2024-04-14T15:17:21.072 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:20 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3541596265' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi090-93363-63"}]: dispatch 2024-04-14T15:17:21.072 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:20 smithi090 ceph-mon[30399]: from='client.48976 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi090-93363-63"}]: dispatch 2024-04-14T15:17:21.073 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-14T15:17:21.073 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:17:21.073 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:21.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:21.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:17:21.175 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:17:21.200 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:21.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:21.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:17:21.302 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:17:21.325 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:21.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:21.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:17:21.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:20 smithi092 ceph-mon[33966]: pgmap v503: 246 pgs: 8 creating+peering, 5 unknown, 233 active+clean; 8.4 MiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-14T15:17:21.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:20 smithi092 ceph-mon[33966]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:17:21.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:20 smithi092 ceph-mon[33966]: osdmap e373: 8 total, 8 up, 8 in 2024-04-14T15:17:21.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:20 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3541596265' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi090-93363-63"}]: dispatch 2024-04-14T15:17:21.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:20 smithi092 ceph-mon[33966]: from='client.48976 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi090-93363-63"}]: dispatch 2024-04-14T15:17:21.427 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:17:21.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:21.455 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:21.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:17:21.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.490+0000 7f6a37d79700 1 -- 172.21.15.90:0/1374234916 >> v1:172.21.15.90:6789/0 conn(0x7f6a3010b830 legacy=0x7f6a3010dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:21.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.491+0000 7f6a37d79700 1 -- 172.21.15.90:0/1374234916 shutdown_connections 2024-04-14T15:17:21.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.491+0000 7f6a37d79700 1 -- 172.21.15.90:0/1374234916 >> 172.21.15.90:0/1374234916 conn(0x7f6a300fd290 msgr2=0x7f6a300ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:17:21.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.491+0000 7f6a37d79700 1 -- 172.21.15.90:0/1374234916 shutdown_connections 2024-04-14T15:17:21.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.491+0000 7f6a37d79700 1 -- 172.21.15.90:0/1374234916 wait complete. 2024-04-14T15:17:21.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.491+0000 7f6a37d79700 1 librados: starting msgr at 2024-04-14T15:17:21.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.491+0000 7f6a37d79700 1 librados: starting objecter 2024-04-14T15:17:21.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.491+0000 7f6a37d79700 1 Processor -- start 2024-04-14T15:17:21.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.491+0000 7f6a37d79700 1 -- start start 2024-04-14T15:17:21.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.491+0000 7f6a37d79700 1 librados: setting wanted keys 2024-04-14T15:17:21.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.491+0000 7f6a37d79700 1 librados: calling monclient init 2024-04-14T15:17:21.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.492+0000 7f6a37d79700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6a3010b450 con 0x7f6a30103a20 2024-04-14T15:17:21.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.492+0000 7f6a37d79700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6a3010b660 con 0x7f6a3010b830 2024-04-14T15:17:21.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.492+0000 7f6a37d79700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6a301a9f30 con 0x7f6a3010f340 2024-04-14T15:17:21.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.492+0000 7f6a35b15700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f6a30103a20 0x7f6a301082a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:56886/0 (socket says 172.21.15.90:56886) 2024-04-14T15:17:21.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.492+0000 7f6a35b15700 1 -- 172.21.15.90:0/4170869568 learned_addr learned my addr 172.21.15.90:0/4170869568 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:17:21.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.492+0000 7f6a26ffd700 10 client.?.objecter ms_handle_connect 0x7f6a30103a20 2024-04-14T15:17:21.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.492+0000 7f6a26ffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:17:21.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.492+0000 7f6a26ffd700 10 client.?.objecter ms_handle_connect 0x7f6a3010f340 2024-04-14T15:17:21.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.492+0000 7f6a26ffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:17:21.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.492+0000 7f6a26ffd700 10 client.?.objecter ms_handle_connect 0x7f6a3010b830 2024-04-14T15:17:21.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.492+0000 7f6a26ffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:17:21.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.492+0000 7f6a26ffd700 1 -- 172.21.15.90:0/4170869568 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 536988989 0 0) 0x7f6a301a9f30 con 0x7f6a3010f340 2024-04-14T15:17:21.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.492+0000 7f6a26ffd700 1 -- 172.21.15.90:0/4170869568 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a1c0035b0 con 0x7f6a3010f340 2024-04-14T15:17:21.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.492+0000 7f6a26ffd700 1 -- 172.21.15.90:0/4170869568 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1466908737 0 0) 0x7f6a3010b450 con 0x7f6a30103a20 2024-04-14T15:17:21.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.492+0000 7f6a26ffd700 1 -- 172.21.15.90:0/4170869568 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a301a9f30 con 0x7f6a30103a20 2024-04-14T15:17:21.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.492+0000 7f6a26ffd700 1 -- 172.21.15.90:0/4170869568 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 624855805 0 0) 0x7f6a3010b660 con 0x7f6a3010b830 2024-04-14T15:17:21.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.492+0000 7f6a26ffd700 1 -- 172.21.15.90:0/4170869568 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a3010b450 con 0x7f6a3010b830 2024-04-14T15:17:21.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.492+0000 7f6a26ffd700 1 -- 172.21.15.90:0/4170869568 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3331786212 0 0) 0x7f6a3010b450 con 0x7f6a3010b830 2024-04-14T15:17:21.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.492+0000 7f6a26ffd700 1 -- 172.21.15.90:0/4170869568 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6a3010b660 con 0x7f6a3010b830 2024-04-14T15:17:21.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.492+0000 7f6a26ffd700 1 -- 172.21.15.90:0/4170869568 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2193943411 0 0) 0x7f6a1c0035b0 con 0x7f6a3010f340 2024-04-14T15:17:21.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.492+0000 7f6a26ffd700 1 -- 172.21.15.90:0/4170869568 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6a3010b450 con 0x7f6a3010f340 2024-04-14T15:17:21.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.492+0000 7f6a26ffd700 1 -- 172.21.15.90:0/4170869568 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4198334511 0 0) 0x7f6a301a9f30 con 0x7f6a30103a20 2024-04-14T15:17:21.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.492+0000 7f6a26ffd700 1 -- 172.21.15.90:0/4170869568 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6a1c0035b0 con 0x7f6a30103a20 2024-04-14T15:17:21.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.492+0000 7f6a26ffd700 1 -- 172.21.15.90:0/4170869568 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f6a20001370 con 0x7f6a3010b830 2024-04-14T15:17:21.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.492+0000 7f6a26ffd700 1 -- 172.21.15.90:0/4170869568 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f6a28001bb0 con 0x7f6a3010f340 2024-04-14T15:17:21.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.493+0000 7f6a26ffd700 1 -- 172.21.15.90:0/4170869568 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f6a2c001870 con 0x7f6a30103a20 2024-04-14T15:17:21.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.493+0000 7f6a26ffd700 1 -- 172.21.15.90:0/4170869568 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3427785784 0 0) 0x7f6a3010b660 con 0x7f6a3010b830 2024-04-14T15:17:21.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.493+0000 7f6a26ffd700 1 -- 172.21.15.90:0/4170869568 >> v1:172.21.15.90:6790/0 conn(0x7f6a3010f340 legacy=0x7f6a301a6790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:21.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.493+0000 7f6a26ffd700 1 -- 172.21.15.90:0/4170869568 >> v1:172.21.15.90:6789/0 conn(0x7f6a30103a20 legacy=0x7f6a301082a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:21.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.493+0000 7f6a26ffd700 1 -- 172.21.15.90:0/4170869568 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a301ab170 con 0x7f6a3010b830 2024-04-14T15:17:21.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.493+0000 7f6a37d79700 1 -- 172.21.15.90:0/4170869568 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6a301aa160 con 0x7f6a3010b830 2024-04-14T15:17:21.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.493+0000 7f6a37d79700 10 client.48994.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:17:21.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.493+0000 7f6a37d79700 1 -- 172.21.15.90:0/4170869568 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f6a301aa620 con 0x7f6a3010b830 2024-04-14T15:17:21.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.493+0000 7f6a37d79700 1 librados: init done 2024-04-14T15:17:21.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.493+0000 7f6a26ffd700 1 -- 172.21.15.90:0/4170869568 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f6a20001ae0 con 0x7f6a3010b830 2024-04-14T15:17:21.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.493+0000 7f6a26ffd700 1 -- 172.21.15.90:0/4170869568 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f6a20004c40 con 0x7f6a3010b830 2024-04-14T15:17:21.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.494+0000 7f6a37d79700 1 -- 172.21.15.90:0/4170869568 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f6a14005100 con 0x7f6a3010b830 2024-04-14T15:17:21.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.494+0000 7f6a26ffd700 1 -- 172.21.15.90:0/4170869568 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f6a2001b870 con 0x7f6a3010b830 2024-04-14T15:17:21.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.495+0000 7f6a26ffd700 1 -- 172.21.15.90:0/4170869568 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(373..373 src has 51..373) ==== 8586+0+0 (unknown 721905348 0 0) 0x7f6a20089a10 con 0x7f6a3010b830 2024-04-14T15:17:21.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.495+0000 7f6a26ffd700 10 client.48994.objecter ms_dispatch 0x7f6a30001550 osd_map(373..373 src has 51..373) 2024-04-14T15:17:21.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.495+0000 7f6a26ffd700 3 client.48994.objecter handle_osd_map got epochs [373,373] > 0 2024-04-14T15:17:21.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.495+0000 7f6a26ffd700 3 client.48994.objecter handle_osd_map decoding full epoch 373 2024-04-14T15:17:21.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.495+0000 7f6a26ffd700 20 client.48994.objecter dump_active .. 0 homeless 2024-04-14T15:17:21.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.495+0000 7f6a26ffd700 1 -- 172.21.15.90:0/4170869568 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v373) ==== 54+0+16 (unknown 3617540442 0 2420000192) 0x7f6a20089e00 con 0x7f6a3010b830 2024-04-14T15:17:21.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.495+0000 7f6a26ffd700 10 client.48994.objecter ms_handle_connect 0x7f6a1c071490 2024-04-14T15:17:21.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.495+0000 7f6a37d79700 10 client.48994.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:17:21.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.495+0000 7f6a37d79700 15 client.48994.objecter _session_command_op_assign -1 1 2024-04-14T15:17:21.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.495+0000 7f6a37d79700 20 client.48994.objecter _get_session s=0x7f6a140053e0 osd=0 3 2024-04-14T15:17:21.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.495+0000 7f6a37d79700 20 client.48994.objecter put_session s=0x7f6a140053e0 osd=0 3 2024-04-14T15:17:21.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.495+0000 7f6a37d79700 20 client.48994.objecter _get_session s=0x7f6a140053e0 osd=0 3 2024-04-14T15:17:21.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.495+0000 7f6a37d79700 15 client.48994.objecter _session_command_op_remove -1 1 2024-04-14T15:17:21.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.495+0000 7f6a37d79700 20 client.48994.objecter get_session s=0x7f6a140053e0 osd=0 3 2024-04-14T15:17:21.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.495+0000 7f6a37d79700 15 client.48994.objecter _session_command_op_assign 0 1 2024-04-14T15:17:21.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.495+0000 7f6a37d79700 20 client.48994.objecter put_session s=0x7f6a140053e0 osd=0 4 2024-04-14T15:17:21.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.495+0000 7f6a37d79700 10 client.48994.objecter _send_command 1 2024-04-14T15:17:21.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.495+0000 7f6a37d79700 1 -- 172.21.15.90:0/4170869568 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6a14004b70 con 0x7f6a14004730 2024-04-14T15:17:21.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.496+0000 7f6a26ffd700 10 client.48994.objecter ms_handle_connect 0x7f6a14004730 2024-04-14T15:17:21.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.497+0000 7f6a26ffd700 1 -- 172.21.15.90:0/4170869568 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6a14004b70 con 0x7f6a14004730 2024-04-14T15:17:21.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.497+0000 7f6a26ffd700 10 client.48994.objecter ms_dispatch 0x7f6a30001550 command_reply(tid 1: 0 ) 2024-04-14T15:17:21.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.497+0000 7f6a26ffd700 10 client.48994.objecter _finish_command 1 = system:0 2024-04-14T15:17:21.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.497+0000 7f6a26ffd700 20 client.48994.objecter put_session s=0x7f6a140053e0 osd=0 4 2024-04-14T15:17:21.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.497+0000 7f6a26ffd700 15 client.48994.objecter _session_command_op_remove 0 1 2024-04-14T15:17:21.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.517+0000 7f6a37d79700 10 client.48994.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:17:21.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.517+0000 7f6a37d79700 15 client.48994.objecter _session_command_op_assign -1 2 2024-04-14T15:17:21.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.517+0000 7f6a37d79700 20 client.48994.objecter _get_session s=0x7f6a140053e0 osd=0 3 2024-04-14T15:17:21.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.517+0000 7f6a37d79700 20 client.48994.objecter put_session s=0x7f6a140053e0 osd=0 3 2024-04-14T15:17:21.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.517+0000 7f6a37d79700 20 client.48994.objecter _get_session s=0x7f6a140053e0 osd=0 3 2024-04-14T15:17:21.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.517+0000 7f6a37d79700 15 client.48994.objecter _session_command_op_remove -1 2 2024-04-14T15:17:21.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.517+0000 7f6a37d79700 20 client.48994.objecter get_session s=0x7f6a140053e0 osd=0 3 2024-04-14T15:17:21.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.517+0000 7f6a37d79700 15 client.48994.objecter _session_command_op_assign 0 2 2024-04-14T15:17:21.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.517+0000 7f6a37d79700 20 client.48994.objecter put_session s=0x7f6a140053e0 osd=0 4 2024-04-14T15:17:21.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.517+0000 7f6a37d79700 10 client.48994.objecter _send_command 2 2024-04-14T15:17:21.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.517+0000 7f6a37d79700 1 -- 172.21.15.90:0/4170869568 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6a1400be00 con 0x7f6a14004730 2024-04-14T15:17:21.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.519+0000 7f6a26ffd700 1 -- 172.21.15.90:0/4170869568 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6a1400be00 con 0x7f6a14004730 2024-04-14T15:17:21.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.519+0000 7f6a26ffd700 10 client.48994.objecter ms_dispatch 0x7f6a30001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:17:21.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.519+0000 7f6a26ffd700 10 client.48994.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:17:21.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.519+0000 7f6a26ffd700 20 client.48994.objecter put_session s=0x7f6a140053e0 osd=0 4 2024-04-14T15:17:21.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.519+0000 7f6a26ffd700 15 client.48994.objecter _session_command_op_remove 0 2 2024-04-14T15:17:21.520 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:17:21.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.519+0000 7f6a37d79700 10 client.48994.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:17:21.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.519+0000 7f6a37d79700 15 client.48994.objecter _session_command_op_assign -1 3 2024-04-14T15:17:21.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.519+0000 7f6a37d79700 20 client.48994.objecter _get_session s=0x7f6a14004130 osd=1 3 2024-04-14T15:17:21.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.519+0000 7f6a37d79700 20 client.48994.objecter put_session s=0x7f6a14004130 osd=1 3 2024-04-14T15:17:21.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.519+0000 7f6a37d79700 20 client.48994.objecter _get_session s=0x7f6a14004130 osd=1 3 2024-04-14T15:17:21.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.519+0000 7f6a37d79700 15 client.48994.objecter _session_command_op_remove -1 3 2024-04-14T15:17:21.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.519+0000 7f6a37d79700 20 client.48994.objecter get_session s=0x7f6a14004130 osd=1 3 2024-04-14T15:17:21.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.519+0000 7f6a37d79700 15 client.48994.objecter _session_command_op_assign 1 3 2024-04-14T15:17:21.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.519+0000 7f6a37d79700 20 client.48994.objecter put_session s=0x7f6a14004130 osd=1 4 2024-04-14T15:17:21.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.519+0000 7f6a37d79700 10 client.48994.objecter _send_command 3 2024-04-14T15:17:21.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.519+0000 7f6a37d79700 1 -- 172.21.15.90:0/4170869568 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f6a140022a0 con 0x7f6a14004310 2024-04-14T15:17:21.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.520+0000 7f6a26ffd700 10 client.48994.objecter ms_handle_connect 0x7f6a14004310 2024-04-14T15:17:21.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.520+0000 7f6a26ffd700 1 -- 172.21.15.90:0/4170869568 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6a140022a0 con 0x7f6a14004310 2024-04-14T15:17:21.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.520+0000 7f6a26ffd700 10 client.48994.objecter ms_dispatch 0x7f6a30001550 command_reply(tid 3: 0 ) 2024-04-14T15:17:21.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.520+0000 7f6a26ffd700 10 client.48994.objecter _finish_command 3 = system:0 2024-04-14T15:17:21.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.520+0000 7f6a26ffd700 20 client.48994.objecter put_session s=0x7f6a14004130 osd=1 4 2024-04-14T15:17:21.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.520+0000 7f6a26ffd700 15 client.48994.objecter _session_command_op_remove 1 3 2024-04-14T15:17:21.526 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:17:21.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.541+0000 7f6a37d79700 10 client.48994.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:17:21.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.541+0000 7f6a37d79700 15 client.48994.objecter _session_command_op_assign -1 4 2024-04-14T15:17:21.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.541+0000 7f6a37d79700 20 client.48994.objecter _get_session s=0x7f6a14004130 osd=1 3 2024-04-14T15:17:21.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.541+0000 7f6a37d79700 20 client.48994.objecter put_session s=0x7f6a14004130 osd=1 3 2024-04-14T15:17:21.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.541+0000 7f6a37d79700 20 client.48994.objecter _get_session s=0x7f6a14004130 osd=1 3 2024-04-14T15:17:21.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.541+0000 7f6a37d79700 15 client.48994.objecter _session_command_op_remove -1 4 2024-04-14T15:17:21.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.541+0000 7f6a37d79700 20 client.48994.objecter get_session s=0x7f6a14004130 osd=1 3 2024-04-14T15:17:21.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.541+0000 7f6a37d79700 15 client.48994.objecter _session_command_op_assign 1 4 2024-04-14T15:17:21.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.541+0000 7f6a37d79700 20 client.48994.objecter put_session s=0x7f6a14004130 osd=1 4 2024-04-14T15:17:21.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.541+0000 7f6a37d79700 10 client.48994.objecter _send_command 4 2024-04-14T15:17:21.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.541+0000 7f6a37d79700 1 -- 172.21.15.90:0/4170869568 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6a14001720 con 0x7f6a14004310 2024-04-14T15:17:21.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.543+0000 7f6a26ffd700 1 -- 172.21.15.90:0/4170869568 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6a14001720 con 0x7f6a14004310 2024-04-14T15:17:21.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.543+0000 7f6a26ffd700 10 client.48994.objecter ms_dispatch 0x7f6a30001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:17:21.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.543+0000 7f6a26ffd700 10 client.48994.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:17:21.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.544+0000 7f6a26ffd700 20 client.48994.objecter put_session s=0x7f6a14004130 osd=1 4 2024-04-14T15:17:21.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.544+0000 7f6a26ffd700 15 client.48994.objecter _session_command_op_remove 1 4 2024-04-14T15:17:21.545 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:17:21.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.544+0000 7f6a37d79700 10 client.48994.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:17:21.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.544+0000 7f6a37d79700 15 client.48994.objecter _session_command_op_assign -1 5 2024-04-14T15:17:21.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.544+0000 7f6a37d79700 20 client.48994.objecter _get_session s=0x7f6a14001980 osd=2 3 2024-04-14T15:17:21.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.544+0000 7f6a37d79700 20 client.48994.objecter put_session s=0x7f6a14001980 osd=2 3 2024-04-14T15:17:21.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.544+0000 7f6a37d79700 20 client.48994.objecter _get_session s=0x7f6a14001980 osd=2 3 2024-04-14T15:17:21.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.544+0000 7f6a37d79700 15 client.48994.objecter _session_command_op_remove -1 5 2024-04-14T15:17:21.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.544+0000 7f6a37d79700 20 client.48994.objecter get_session s=0x7f6a14001980 osd=2 3 2024-04-14T15:17:21.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.544+0000 7f6a37d79700 15 client.48994.objecter _session_command_op_assign 2 5 2024-04-14T15:17:21.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.544+0000 7f6a37d79700 20 client.48994.objecter put_session s=0x7f6a14001980 osd=2 4 2024-04-14T15:17:21.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.544+0000 7f6a37d79700 10 client.48994.objecter _send_command 5 2024-04-14T15:17:21.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.544+0000 7f6a37d79700 1 -- 172.21.15.90:0/4170869568 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f6a14015580 con 0x7f6a140119e0 2024-04-14T15:17:21.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.544+0000 7f6a26ffd700 10 client.48994.objecter ms_handle_connect 0x7f6a140119e0 2024-04-14T15:17:21.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.545+0000 7f6a26ffd700 1 -- 172.21.15.90:0/4170869568 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6a14015580 con 0x7f6a140119e0 2024-04-14T15:17:21.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.545+0000 7f6a26ffd700 10 client.48994.objecter ms_dispatch 0x7f6a30001550 command_reply(tid 5: 0 ) 2024-04-14T15:17:21.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.545+0000 7f6a26ffd700 10 client.48994.objecter _finish_command 5 = system:0 2024-04-14T15:17:21.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.545+0000 7f6a26ffd700 20 client.48994.objecter put_session s=0x7f6a14001980 osd=2 4 2024-04-14T15:17:21.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.545+0000 7f6a26ffd700 15 client.48994.objecter _session_command_op_remove 2 5 2024-04-14T15:17:21.555 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:17:21.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.563+0000 7f6a37d79700 10 client.48994.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:17:21.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.563+0000 7f6a37d79700 15 client.48994.objecter _session_command_op_assign -1 6 2024-04-14T15:17:21.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.563+0000 7f6a37d79700 20 client.48994.objecter _get_session s=0x7f6a14001980 osd=2 3 2024-04-14T15:17:21.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.563+0000 7f6a37d79700 20 client.48994.objecter put_session s=0x7f6a14001980 osd=2 3 2024-04-14T15:17:21.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.563+0000 7f6a37d79700 20 client.48994.objecter _get_session s=0x7f6a14001980 osd=2 3 2024-04-14T15:17:21.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.563+0000 7f6a37d79700 15 client.48994.objecter _session_command_op_remove -1 6 2024-04-14T15:17:21.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.563+0000 7f6a37d79700 20 client.48994.objecter get_session s=0x7f6a14001980 osd=2 3 2024-04-14T15:17:21.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.563+0000 7f6a37d79700 15 client.48994.objecter _session_command_op_assign 2 6 2024-04-14T15:17:21.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.563+0000 7f6a37d79700 20 client.48994.objecter put_session s=0x7f6a14001980 osd=2 4 2024-04-14T15:17:21.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.563+0000 7f6a37d79700 10 client.48994.objecter _send_command 6 2024-04-14T15:17:21.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.563+0000 7f6a37d79700 1 -- 172.21.15.90:0/4170869568 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6a1400c510 con 0x7f6a140119e0 2024-04-14T15:17:21.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.565+0000 7f6a26ffd700 1 -- 172.21.15.90:0/4170869568 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6a1400c510 con 0x7f6a140119e0 2024-04-14T15:17:21.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.565+0000 7f6a26ffd700 10 client.48994.objecter ms_dispatch 0x7f6a30001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:17:21.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.565+0000 7f6a26ffd700 10 client.48994.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:17:21.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.565+0000 7f6a26ffd700 20 client.48994.objecter put_session s=0x7f6a14001980 osd=2 4 2024-04-14T15:17:21.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.565+0000 7f6a26ffd700 15 client.48994.objecter _session_command_op_remove 2 6 2024-04-14T15:17:21.566 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:17:21.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.565+0000 7f6a37d79700 10 client.48994.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:17:21.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.565+0000 7f6a37d79700 15 client.48994.objecter _session_command_op_assign -1 7 2024-04-14T15:17:21.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.565+0000 7f6a37d79700 20 client.48994.objecter _get_session s=0x7f6a1400c7a0 osd=3 3 2024-04-14T15:17:21.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.565+0000 7f6a37d79700 20 client.48994.objecter put_session s=0x7f6a1400c7a0 osd=3 3 2024-04-14T15:17:21.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.565+0000 7f6a37d79700 20 client.48994.objecter _get_session s=0x7f6a1400c7a0 osd=3 3 2024-04-14T15:17:21.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.565+0000 7f6a37d79700 15 client.48994.objecter _session_command_op_remove -1 7 2024-04-14T15:17:21.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.565+0000 7f6a37d79700 20 client.48994.objecter get_session s=0x7f6a1400c7a0 osd=3 3 2024-04-14T15:17:21.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.565+0000 7f6a37d79700 15 client.48994.objecter _session_command_op_assign 3 7 2024-04-14T15:17:21.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.565+0000 7f6a37d79700 20 client.48994.objecter put_session s=0x7f6a1400c7a0 osd=3 4 2024-04-14T15:17:21.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.565+0000 7f6a37d79700 10 client.48994.objecter _send_command 7 2024-04-14T15:17:21.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.565+0000 7f6a37d79700 1 -- 172.21.15.90:0/4170869568 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f6a1401a3c0 con 0x7f6a140168b0 2024-04-14T15:17:21.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.565+0000 7f6a26ffd700 10 client.48994.objecter ms_handle_connect 0x7f6a140168b0 2024-04-14T15:17:21.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.566+0000 7f6a26ffd700 1 -- 172.21.15.90:0/4170869568 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6a1401a3c0 con 0x7f6a140168b0 2024-04-14T15:17:21.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.566+0000 7f6a26ffd700 10 client.48994.objecter ms_dispatch 0x7f6a30001550 command_reply(tid 7: 0 ) 2024-04-14T15:17:21.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.566+0000 7f6a26ffd700 10 client.48994.objecter _finish_command 7 = system:0 2024-04-14T15:17:21.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.566+0000 7f6a26ffd700 20 client.48994.objecter put_session s=0x7f6a1400c7a0 osd=3 4 2024-04-14T15:17:21.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.566+0000 7f6a26ffd700 15 client.48994.objecter _session_command_op_remove 3 7 2024-04-14T15:17:21.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:21.583 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:21.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:17:21.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.583+0000 7f6a37d79700 10 client.48994.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:17:21.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.583+0000 7f6a37d79700 15 client.48994.objecter _session_command_op_assign -1 8 2024-04-14T15:17:21.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.583+0000 7f6a37d79700 20 client.48994.objecter _get_session s=0x7f6a1400c7a0 osd=3 3 2024-04-14T15:17:21.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.583+0000 7f6a37d79700 20 client.48994.objecter put_session s=0x7f6a1400c7a0 osd=3 3 2024-04-14T15:17:21.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.583+0000 7f6a37d79700 20 client.48994.objecter _get_session s=0x7f6a1400c7a0 osd=3 3 2024-04-14T15:17:21.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.583+0000 7f6a37d79700 15 client.48994.objecter _session_command_op_remove -1 8 2024-04-14T15:17:21.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.583+0000 7f6a37d79700 20 client.48994.objecter get_session s=0x7f6a1400c7a0 osd=3 3 2024-04-14T15:17:21.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.583+0000 7f6a37d79700 15 client.48994.objecter _session_command_op_assign 3 8 2024-04-14T15:17:21.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.583+0000 7f6a37d79700 20 client.48994.objecter put_session s=0x7f6a1400c7a0 osd=3 4 2024-04-14T15:17:21.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.583+0000 7f6a37d79700 10 client.48994.objecter _send_command 8 2024-04-14T15:17:21.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.583+0000 7f6a37d79700 1 -- 172.21.15.90:0/4170869568 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6a14010dc0 con 0x7f6a140168b0 2024-04-14T15:17:21.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.584+0000 7f6a26ffd700 1 -- 172.21.15.90:0/4170869568 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6a14010dc0 con 0x7f6a140168b0 2024-04-14T15:17:21.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.584+0000 7f6a26ffd700 10 client.48994.objecter ms_dispatch 0x7f6a30001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:17:21.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.584+0000 7f6a26ffd700 10 client.48994.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:17:21.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.584+0000 7f6a26ffd700 20 client.48994.objecter put_session s=0x7f6a1400c7a0 osd=3 4 2024-04-14T15:17:21.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.584+0000 7f6a26ffd700 15 client.48994.objecter _session_command_op_remove 3 8 2024-04-14T15:17:21.586 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:17:21.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.584+0000 7f6a37d79700 10 client.48994.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:17:21.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.584+0000 7f6a37d79700 15 client.48994.objecter _session_command_op_assign -1 9 2024-04-14T15:17:21.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.584+0000 7f6a37d79700 20 client.48994.objecter _get_session s=0x7f6a14010dc0 osd=4 3 2024-04-14T15:17:21.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.584+0000 7f6a37d79700 20 client.48994.objecter put_session s=0x7f6a14010dc0 osd=4 3 2024-04-14T15:17:21.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.584+0000 7f6a37d79700 20 client.48994.objecter _get_session s=0x7f6a14010dc0 osd=4 3 2024-04-14T15:17:21.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.584+0000 7f6a37d79700 15 client.48994.objecter _session_command_op_remove -1 9 2024-04-14T15:17:21.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.584+0000 7f6a37d79700 20 client.48994.objecter get_session s=0x7f6a14010dc0 osd=4 3 2024-04-14T15:17:21.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.584+0000 7f6a37d79700 15 client.48994.objecter _session_command_op_assign 4 9 2024-04-14T15:17:21.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.584+0000 7f6a37d79700 20 client.48994.objecter put_session s=0x7f6a14010dc0 osd=4 4 2024-04-14T15:17:21.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.584+0000 7f6a37d79700 10 client.48994.objecter _send_command 9 2024-04-14T15:17:21.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.584+0000 7f6a37d79700 1 -- 172.21.15.90:0/4170869568 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f6a14011090 con 0x7f6a14016410 2024-04-14T15:17:21.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.585+0000 7f6a26ffd700 10 client.48994.objecter ms_handle_connect 0x7f6a14016410 2024-04-14T15:17:21.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.586+0000 7f6a26ffd700 1 -- 172.21.15.90:0/4170869568 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6a14011090 con 0x7f6a14016410 2024-04-14T15:17:21.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.586+0000 7f6a26ffd700 10 client.48994.objecter ms_dispatch 0x7f6a30001550 command_reply(tid 9: 0 ) 2024-04-14T15:17:21.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.586+0000 7f6a26ffd700 10 client.48994.objecter _finish_command 9 = system:0 2024-04-14T15:17:21.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.586+0000 7f6a26ffd700 20 client.48994.objecter put_session s=0x7f6a14010dc0 osd=4 4 2024-04-14T15:17:21.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.586+0000 7f6a26ffd700 15 client.48994.objecter _session_command_op_remove 4 9 2024-04-14T15:17:21.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.601+0000 7f6a37d79700 10 client.48994.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:17:21.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.602+0000 7f6a37d79700 15 client.48994.objecter _session_command_op_assign -1 10 2024-04-14T15:17:21.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.602+0000 7f6a37d79700 20 client.48994.objecter _get_session s=0x7f6a14010dc0 osd=4 3 2024-04-14T15:17:21.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.602+0000 7f6a37d79700 20 client.48994.objecter put_session s=0x7f6a14010dc0 osd=4 3 2024-04-14T15:17:21.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.602+0000 7f6a37d79700 20 client.48994.objecter _get_session s=0x7f6a14010dc0 osd=4 3 2024-04-14T15:17:21.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.602+0000 7f6a37d79700 15 client.48994.objecter _session_command_op_remove -1 10 2024-04-14T15:17:21.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.602+0000 7f6a37d79700 20 client.48994.objecter get_session s=0x7f6a14010dc0 osd=4 3 2024-04-14T15:17:21.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.602+0000 7f6a37d79700 15 client.48994.objecter _session_command_op_assign 4 10 2024-04-14T15:17:21.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.602+0000 7f6a37d79700 20 client.48994.objecter put_session s=0x7f6a14010dc0 osd=4 4 2024-04-14T15:17:21.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.602+0000 7f6a37d79700 10 client.48994.objecter _send_command 10 2024-04-14T15:17:21.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.602+0000 7f6a37d79700 1 -- 172.21.15.90:0/4170869568 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6a140205f0 con 0x7f6a14016410 2024-04-14T15:17:21.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.603+0000 7f6a26ffd700 1 -- 172.21.15.90:0/4170869568 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6a140205f0 con 0x7f6a14016410 2024-04-14T15:17:21.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.603+0000 7f6a26ffd700 10 client.48994.objecter ms_dispatch 0x7f6a30001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:17:21.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.603+0000 7f6a26ffd700 10 client.48994.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:17:21.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.603+0000 7f6a26ffd700 20 client.48994.objecter put_session s=0x7f6a14010dc0 osd=4 4 2024-04-14T15:17:21.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.603+0000 7f6a26ffd700 15 client.48994.objecter _session_command_op_remove 4 10 2024-04-14T15:17:21.604 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:17:21.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.604+0000 7f6a37d79700 10 client.48994.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:17:21.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.604+0000 7f6a37d79700 15 client.48994.objecter _session_command_op_assign -1 11 2024-04-14T15:17:21.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.604+0000 7f6a37d79700 20 client.48994.objecter _get_session s=0x7f6a140205f0 osd=5 3 2024-04-14T15:17:21.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.604+0000 7f6a37d79700 20 client.48994.objecter put_session s=0x7f6a140205f0 osd=5 3 2024-04-14T15:17:21.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.604+0000 7f6a37d79700 20 client.48994.objecter _get_session s=0x7f6a140205f0 osd=5 3 2024-04-14T15:17:21.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.604+0000 7f6a37d79700 15 client.48994.objecter _session_command_op_remove -1 11 2024-04-14T15:17:21.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.604+0000 7f6a37d79700 20 client.48994.objecter get_session s=0x7f6a140205f0 osd=5 3 2024-04-14T15:17:21.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.604+0000 7f6a37d79700 15 client.48994.objecter _session_command_op_assign 5 11 2024-04-14T15:17:21.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.604+0000 7f6a37d79700 20 client.48994.objecter put_session s=0x7f6a140205f0 osd=5 4 2024-04-14T15:17:21.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.604+0000 7f6a37d79700 10 client.48994.objecter _send_command 11 2024-04-14T15:17:21.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.604+0000 7f6a37d79700 1 -- 172.21.15.90:0/4170869568 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f6a14021150 con 0x7f6a14020d10 2024-04-14T15:17:21.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.604+0000 7f6a26ffd700 10 client.48994.objecter ms_handle_connect 0x7f6a14020d10 2024-04-14T15:17:21.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.605+0000 7f6a26ffd700 1 -- 172.21.15.90:0/4170869568 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6a14021150 con 0x7f6a14020d10 2024-04-14T15:17:21.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.605+0000 7f6a26ffd700 10 client.48994.objecter ms_dispatch 0x7f6a30001550 command_reply(tid 11: 0 ) 2024-04-14T15:17:21.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.605+0000 7f6a26ffd700 10 client.48994.objecter _finish_command 11 = system:0 2024-04-14T15:17:21.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.605+0000 7f6a26ffd700 20 client.48994.objecter put_session s=0x7f6a140205f0 osd=5 4 2024-04-14T15:17:21.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.605+0000 7f6a26ffd700 15 client.48994.objecter _session_command_op_remove 5 11 2024-04-14T15:17:21.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.621+0000 7f6a37d79700 10 client.48994.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:17:21.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.621+0000 7f6a37d79700 15 client.48994.objecter _session_command_op_assign -1 12 2024-04-14T15:17:21.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.621+0000 7f6a37d79700 20 client.48994.objecter _get_session s=0x7f6a140205f0 osd=5 3 2024-04-14T15:17:21.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.621+0000 7f6a37d79700 20 client.48994.objecter put_session s=0x7f6a140205f0 osd=5 3 2024-04-14T15:17:21.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.621+0000 7f6a37d79700 20 client.48994.objecter _get_session s=0x7f6a140205f0 osd=5 3 2024-04-14T15:17:21.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.621+0000 7f6a37d79700 15 client.48994.objecter _session_command_op_remove -1 12 2024-04-14T15:17:21.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.621+0000 7f6a37d79700 20 client.48994.objecter get_session s=0x7f6a140205f0 osd=5 3 2024-04-14T15:17:21.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.621+0000 7f6a37d79700 15 client.48994.objecter _session_command_op_assign 5 12 2024-04-14T15:17:21.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.621+0000 7f6a37d79700 20 client.48994.objecter put_session s=0x7f6a140205f0 osd=5 4 2024-04-14T15:17:21.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.621+0000 7f6a37d79700 10 client.48994.objecter _send_command 12 2024-04-14T15:17:21.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.621+0000 7f6a37d79700 1 -- 172.21.15.90:0/4170869568 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6a14021150 con 0x7f6a14020d10 2024-04-14T15:17:21.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.623+0000 7f6a26ffd700 1 -- 172.21.15.90:0/4170869568 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6a14021150 con 0x7f6a14020d10 2024-04-14T15:17:21.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.623+0000 7f6a26ffd700 10 client.48994.objecter ms_dispatch 0x7f6a30001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:17:21.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.623+0000 7f6a26ffd700 10 client.48994.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:17:21.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.623+0000 7f6a26ffd700 20 client.48994.objecter put_session s=0x7f6a140205f0 osd=5 4 2024-04-14T15:17:21.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.623+0000 7f6a26ffd700 15 client.48994.objecter _session_command_op_remove 5 12 2024-04-14T15:17:21.624 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:17:21.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.623+0000 7f6a37d79700 10 client.48994.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:17:21.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.623+0000 7f6a37d79700 15 client.48994.objecter _session_command_op_assign -1 13 2024-04-14T15:17:21.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.623+0000 7f6a37d79700 20 client.48994.objecter _get_session s=0x7f6a14021150 osd=6 3 2024-04-14T15:17:21.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.623+0000 7f6a37d79700 20 client.48994.objecter put_session s=0x7f6a14021150 osd=6 3 2024-04-14T15:17:21.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.623+0000 7f6a37d79700 20 client.48994.objecter _get_session s=0x7f6a14021150 osd=6 3 2024-04-14T15:17:21.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.623+0000 7f6a37d79700 15 client.48994.objecter _session_command_op_remove -1 13 2024-04-14T15:17:21.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.623+0000 7f6a37d79700 20 client.48994.objecter get_session s=0x7f6a14021150 osd=6 3 2024-04-14T15:17:21.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.623+0000 7f6a37d79700 15 client.48994.objecter _session_command_op_assign 6 13 2024-04-14T15:17:21.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.623+0000 7f6a37d79700 20 client.48994.objecter put_session s=0x7f6a14021150 osd=6 4 2024-04-14T15:17:21.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.623+0000 7f6a37d79700 10 client.48994.objecter _send_command 13 2024-04-14T15:17:21.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.623+0000 7f6a37d79700 1 -- 172.21.15.90:0/4170869568 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f6a14027640 con 0x7f6a14021380 2024-04-14T15:17:21.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.624+0000 7f6a26ffd700 10 client.48994.objecter ms_handle_connect 0x7f6a14021380 2024-04-14T15:17:21.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.624+0000 7f6a26ffd700 1 -- 172.21.15.90:0/4170869568 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6a14027640 con 0x7f6a14021380 2024-04-14T15:17:21.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.624+0000 7f6a26ffd700 10 client.48994.objecter ms_dispatch 0x7f6a30001550 command_reply(tid 13: 0 ) 2024-04-14T15:17:21.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.624+0000 7f6a26ffd700 10 client.48994.objecter _finish_command 13 = system:0 2024-04-14T15:17:21.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.624+0000 7f6a26ffd700 20 client.48994.objecter put_session s=0x7f6a14021150 osd=6 4 2024-04-14T15:17:21.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.624+0000 7f6a26ffd700 15 client.48994.objecter _session_command_op_remove 6 13 2024-04-14T15:17:21.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.641+0000 7f6a37d79700 10 client.48994.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:17:21.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.641+0000 7f6a37d79700 15 client.48994.objecter _session_command_op_assign -1 14 2024-04-14T15:17:21.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.641+0000 7f6a37d79700 20 client.48994.objecter _get_session s=0x7f6a14021150 osd=6 3 2024-04-14T15:17:21.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.641+0000 7f6a37d79700 20 client.48994.objecter put_session s=0x7f6a14021150 osd=6 3 2024-04-14T15:17:21.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.641+0000 7f6a37d79700 20 client.48994.objecter _get_session s=0x7f6a14021150 osd=6 3 2024-04-14T15:17:21.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.641+0000 7f6a37d79700 15 client.48994.objecter _session_command_op_remove -1 14 2024-04-14T15:17:21.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.641+0000 7f6a37d79700 20 client.48994.objecter get_session s=0x7f6a14021150 osd=6 3 2024-04-14T15:17:21.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.641+0000 7f6a37d79700 15 client.48994.objecter _session_command_op_assign 6 14 2024-04-14T15:17:21.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.641+0000 7f6a37d79700 20 client.48994.objecter put_session s=0x7f6a14021150 osd=6 4 2024-04-14T15:17:21.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.641+0000 7f6a37d79700 10 client.48994.objecter _send_command 14 2024-04-14T15:17:21.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.641+0000 7f6a37d79700 1 -- 172.21.15.90:0/4170869568 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6a14027820 con 0x7f6a14021380 2024-04-14T15:17:21.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.643+0000 7f6a26ffd700 1 -- 172.21.15.90:0/4170869568 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6a14027820 con 0x7f6a14021380 2024-04-14T15:17:21.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.643+0000 7f6a26ffd700 10 client.48994.objecter ms_dispatch 0x7f6a30001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:17:21.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.643+0000 7f6a26ffd700 10 client.48994.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:17:21.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.643+0000 7f6a26ffd700 20 client.48994.objecter put_session s=0x7f6a14021150 osd=6 4 2024-04-14T15:17:21.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.643+0000 7f6a26ffd700 15 client.48994.objecter _session_command_op_remove 6 14 2024-04-14T15:17:21.643 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:17:21.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.643+0000 7f6a37d79700 10 client.48994.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:17:21.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.643+0000 7f6a37d79700 15 client.48994.objecter _session_command_op_assign -1 15 2024-04-14T15:17:21.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.643+0000 7f6a37d79700 20 client.48994.objecter _get_session s=0x7f6a14027660 osd=7 3 2024-04-14T15:17:21.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.643+0000 7f6a37d79700 20 client.48994.objecter put_session s=0x7f6a14027660 osd=7 3 2024-04-14T15:17:21.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.643+0000 7f6a37d79700 20 client.48994.objecter _get_session s=0x7f6a14027660 osd=7 3 2024-04-14T15:17:21.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.643+0000 7f6a37d79700 15 client.48994.objecter _session_command_op_remove -1 15 2024-04-14T15:17:21.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.643+0000 7f6a37d79700 20 client.48994.objecter get_session s=0x7f6a14027660 osd=7 3 2024-04-14T15:17:21.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.643+0000 7f6a37d79700 15 client.48994.objecter _session_command_op_assign 7 15 2024-04-14T15:17:21.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.643+0000 7f6a37d79700 20 client.48994.objecter put_session s=0x7f6a14027660 osd=7 4 2024-04-14T15:17:21.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.643+0000 7f6a37d79700 10 client.48994.objecter _send_command 15 2024-04-14T15:17:21.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.643+0000 7f6a37d79700 1 -- 172.21.15.90:0/4170869568 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f6a14028ae0 con 0x7f6a14027f40 2024-04-14T15:17:21.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.643+0000 7f6a26ffd700 10 client.48994.objecter ms_handle_connect 0x7f6a14027f40 2024-04-14T15:17:21.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.644+0000 7f6a26ffd700 1 -- 172.21.15.90:0/4170869568 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6a14028ae0 con 0x7f6a14027f40 2024-04-14T15:17:21.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.644+0000 7f6a26ffd700 10 client.48994.objecter ms_dispatch 0x7f6a30001550 command_reply(tid 15: 0 ) 2024-04-14T15:17:21.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.644+0000 7f6a26ffd700 10 client.48994.objecter _finish_command 15 = system:0 2024-04-14T15:17:21.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.644+0000 7f6a26ffd700 20 client.48994.objecter put_session s=0x7f6a14027660 osd=7 4 2024-04-14T15:17:21.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.644+0000 7f6a26ffd700 15 client.48994.objecter _session_command_op_remove 7 15 2024-04-14T15:17:21.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.661+0000 7f6a37d79700 10 client.48994.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:17:21.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.661+0000 7f6a37d79700 15 client.48994.objecter _session_command_op_assign -1 16 2024-04-14T15:17:21.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.661+0000 7f6a37d79700 20 client.48994.objecter _get_session s=0x7f6a14027660 osd=7 3 2024-04-14T15:17:21.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.661+0000 7f6a37d79700 20 client.48994.objecter put_session s=0x7f6a14027660 osd=7 3 2024-04-14T15:17:21.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.661+0000 7f6a37d79700 20 client.48994.objecter _get_session s=0x7f6a14027660 osd=7 3 2024-04-14T15:17:21.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.661+0000 7f6a37d79700 15 client.48994.objecter _session_command_op_remove -1 16 2024-04-14T15:17:21.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.661+0000 7f6a37d79700 20 client.48994.objecter get_session s=0x7f6a14027660 osd=7 3 2024-04-14T15:17:21.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.661+0000 7f6a37d79700 15 client.48994.objecter _session_command_op_assign 7 16 2024-04-14T15:17:21.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.661+0000 7f6a37d79700 20 client.48994.objecter put_session s=0x7f6a14027660 osd=7 4 2024-04-14T15:17:21.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.661+0000 7f6a37d79700 10 client.48994.objecter _send_command 16 2024-04-14T15:17:21.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.661+0000 7f6a37d79700 1 -- 172.21.15.90:0/4170869568 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6a14028ae0 con 0x7f6a14027f40 2024-04-14T15:17:21.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.663+0000 7f6a26ffd700 1 -- 172.21.15.90:0/4170869568 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6a14028ae0 con 0x7f6a14027f40 2024-04-14T15:17:21.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.663+0000 7f6a26ffd700 10 client.48994.objecter ms_dispatch 0x7f6a30001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:17:21.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.663+0000 7f6a26ffd700 10 client.48994.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:17:21.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.663+0000 7f6a26ffd700 20 client.48994.objecter put_session s=0x7f6a14027660 osd=7 4 2024-04-14T15:17:21.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.663+0000 7f6a26ffd700 15 client.48994.objecter _session_command_op_remove 7 16 2024-04-14T15:17:21.664 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:17:21.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.663+0000 7f6a37d79700 10 librados: watch_flush enter 2024-04-14T15:17:21.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.663+0000 7f6a37d79700 10 librados: watch_flush exit 2024-04-14T15:17:21.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.663+0000 7f6a37d79700 10 client.48994.objecter close_session for osd.0 2024-04-14T15:17:21.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.663+0000 7f6a37d79700 1 -- 172.21.15.90:0/4170869568 >> v1:172.21.15.90:6801/3761806885 conn(0x7f6a14004730 legacy=0x7f6a14002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:21.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.663+0000 7f6a37d79700 20 client.48994.objecter put_session s=0x7f6a140053e0 osd=0 1 2024-04-14T15:17:21.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.663+0000 7f6a37d79700 10 client.48994.objecter close_session for osd.1 2024-04-14T15:17:21.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.663+0000 7f6a37d79700 1 -- 172.21.15.90:0/4170869568 >> v1:172.21.15.90:6805/4170361699 conn(0x7f6a14004310 legacy=0x7f6a1400f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:21.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.663+0000 7f6a37d79700 20 client.48994.objecter put_session s=0x7f6a14004130 osd=1 1 2024-04-14T15:17:21.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.663+0000 7f6a37d79700 10 client.48994.objecter close_session for osd.2 2024-04-14T15:17:21.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.663+0000 7f6a37d79700 1 -- 172.21.15.90:0/4170869568 >> v1:172.21.15.90:6809/1686184251 conn(0x7f6a140119e0 legacy=0x7f6a14013dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:21.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.663+0000 7f6a36316700 1 -- 172.21.15.90:0/4170869568 reap_dead start 2024-04-14T15:17:21.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.663+0000 7f6a37d79700 20 client.48994.objecter put_session s=0x7f6a14001980 osd=2 1 2024-04-14T15:17:21.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.663+0000 7f6a37d79700 10 client.48994.objecter close_session for osd.3 2024-04-14T15:17:21.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.663+0000 7f6a37d79700 1 -- 172.21.15.90:0/4170869568 >> v1:172.21.15.90:6813/2270166774 conn(0x7f6a140168b0 legacy=0x7f6a14018ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:21.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.663+0000 7f6a37d79700 20 client.48994.objecter put_session s=0x7f6a1400c7a0 osd=3 1 2024-04-14T15:17:21.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.663+0000 7f6a37d79700 10 client.48994.objecter close_session for osd.4 2024-04-14T15:17:21.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.663+0000 7f6a37d79700 1 -- 172.21.15.90:0/4170869568 >> v1:172.21.15.92:6800/3887549733 conn(0x7f6a14016410 legacy=0x7f6a1401d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:21.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.663+0000 7f6a37d79700 20 client.48994.objecter put_session s=0x7f6a14010dc0 osd=4 1 2024-04-14T15:17:21.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.663+0000 7f6a37d79700 10 client.48994.objecter close_session for osd.5 2024-04-14T15:17:21.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.663+0000 7f6a37d79700 1 -- 172.21.15.90:0/4170869568 >> v1:172.21.15.92:6804/3244561440 conn(0x7f6a14020d10 legacy=0x7f6a1401ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:21.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.663+0000 7f6a37d79700 20 client.48994.objecter put_session s=0x7f6a140205f0 osd=5 1 2024-04-14T15:17:21.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.663+0000 7f6a37d79700 10 client.48994.objecter close_session for osd.6 2024-04-14T15:17:21.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.663+0000 7f6a37d79700 1 -- 172.21.15.90:0/4170869568 >> v1:172.21.15.92:6808/636125963 conn(0x7f6a14021380 legacy=0x7f6a14025d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:21.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.663+0000 7f6a37d79700 20 client.48994.objecter put_session s=0x7f6a14021150 osd=6 1 2024-04-14T15:17:21.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.663+0000 7f6a37d79700 10 client.48994.objecter close_session for osd.7 2024-04-14T15:17:21.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.663+0000 7f6a37d79700 1 -- 172.21.15.90:0/4170869568 >> v1:172.21.15.92:6812/1773706716 conn(0x7f6a14027f40 legacy=0x7f6a140283a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:21.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.663+0000 7f6a37d79700 20 client.48994.objecter put_session s=0x7f6a14027660 osd=7 1 2024-04-14T15:17:21.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.663+0000 7f6a37d79700 20 client.48994.objecter shutdown clearing up homeless session... 2024-04-14T15:17:21.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.663+0000 7f6a37d79700 10 client.48994.objecter successfully canceled tick 2024-04-14T15:17:21.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.663+0000 7f6a36316700 1 -- 172.21.15.90:0/4170869568 reap_dead start 2024-04-14T15:17:21.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.664+0000 7f6a37d79700 1 -- 172.21.15.90:0/4170869568 >> v1:172.21.15.90:6800/420935427 conn(0x7f6a1c071490 legacy=0x7f6a1c073950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:21.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.664+0000 7f6a37d79700 1 -- 172.21.15.90:0/4170869568 >> v1:172.21.15.92:6789/0 conn(0x7f6a3010b830 legacy=0x7f6a301089b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:21.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.664+0000 7f6a37d79700 1 -- 172.21.15.90:0/4170869568 shutdown_connections 2024-04-14T15:17:21.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.664+0000 7f6a37d79700 1 -- 172.21.15.90:0/4170869568 >> 172.21.15.90:0/4170869568 conn(0x7f6a300fd290 msgr2=0x7f6a30111770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:17:21.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.664+0000 7f6a37d79700 1 -- 172.21.15.90:0/4170869568 shutdown_connections 2024-04-14T15:17:21.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.664+0000 7f6a37d79700 1 -- 172.21.15.90:0/4170869568 wait complete. 2024-04-14T15:17:21.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:21.664+0000 7f6a37d79700 1 librados: shutdown 2024-04-14T15:17:21.668 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:17:21.668 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:17:21.668 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:17:21.668 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:17:21.668 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:17:21.668 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:17:21.668 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:17:21.668 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:17:21.685 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:17:21.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:21.712 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:21.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:17:21.813 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:17:21.841 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:21.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:21.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:17:21.943 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:17:21.967 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:21.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:21.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:17:22.069 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:17:22.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:22.093 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:22.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:17:22.194 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:17:22.216 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:21 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/4170869568' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:17:22.217 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:21 smithi090 ceph-mon[41670]: from='client.48976 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi090-93363-63"}]': finished 2024-04-14T15:17:22.217 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:21 smithi090 ceph-mon[41670]: osdmap e374: 8 total, 8 up, 8 in 2024-04-14T15:17:22.217 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:21 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3541596265' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi090-93363-63"}]: dispatch 2024-04-14T15:17:22.217 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:21 smithi090 ceph-mon[41670]: from='client.48976 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi090-93363-63"}]: dispatch 2024-04-14T15:17:22.217 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:21 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/4170869568' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:17:22.217 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:21 smithi090 ceph-mon[30399]: from='client.48976 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi090-93363-63"}]': finished 2024-04-14T15:17:22.217 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:21 smithi090 ceph-mon[30399]: osdmap e374: 8 total, 8 up, 8 in 2024-04-14T15:17:22.217 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:21 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3541596265' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi090-93363-63"}]: dispatch 2024-04-14T15:17:22.217 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:21 smithi090 ceph-mon[30399]: from='client.48976 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi090-93363-63"}]: dispatch 2024-04-14T15:17:22.222 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:22.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:22.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:17:22.246 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-14T15:17:22.273 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:21 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/4170869568' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:17:22.273 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:21 smithi092 ceph-mon[33966]: from='client.48976 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi090-93363-63"}]': finished 2024-04-14T15:17:22.273 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:21 smithi092 ceph-mon[33966]: osdmap e374: 8 total, 8 up, 8 in 2024-04-14T15:17:22.274 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:21 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3541596265' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi090-93363-63"}]: dispatch 2024-04-14T15:17:22.274 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:21 smithi092 ceph-mon[33966]: from='client.48976 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi090-93363-63"}]: dispatch 2024-04-14T15:17:22.324 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:17:22.348 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:22.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:22.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:17:22.450 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:17:22.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:22.475 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:22.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:17:22.576 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:17:22.604 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:22.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:22.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:17:22.706 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:17:22.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:22.732 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:22.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:17:22.833 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:17:22.839 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd unset noscrub 2024-04-14T15:17:22.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:22.859 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:22.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:17:22.960 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:17:22.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:22.984 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:22.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:17:23.085 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:17:23.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:23.114 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:23.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:17:23.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:22 smithi090 ceph-mon[30399]: pgmap v506: 238 pgs: 238 active+clean; 8.4 MiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-14T15:17:23.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:22 smithi090 ceph-mon[30399]: from='client.48976 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi090-93363-63"}]': finished 2024-04-14T15:17:23.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:22 smithi090 ceph-mon[30399]: osdmap e375: 8 total, 8 up, 8 in 2024-04-14T15:17:23.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:22 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1312471356' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi090-93363-64"}]: dispatch 2024-04-14T15:17:23.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:22 smithi090 ceph-mon[30399]: from='client.47114 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi090-93363-64"}]: dispatch 2024-04-14T15:17:23.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:22 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1312471356' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi090-93363-64"}]: dispatch 2024-04-14T15:17:23.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:22 smithi090 ceph-mon[30399]: from='client.47114 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi090-93363-64"}]: dispatch 2024-04-14T15:17:23.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:22 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1312471356' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi090-93363-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:17:23.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:22 smithi090 ceph-mon[30399]: from='client.47114 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi090-93363-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:17:23.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:22 smithi090 ceph-mon[41670]: pgmap v506: 238 pgs: 238 active+clean; 8.4 MiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-14T15:17:23.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:22 smithi090 ceph-mon[41670]: from='client.48976 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi090-93363-63"}]': finished 2024-04-14T15:17:23.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:22 smithi090 ceph-mon[41670]: osdmap e375: 8 total, 8 up, 8 in 2024-04-14T15:17:23.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:22 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1312471356' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi090-93363-64"}]: dispatch 2024-04-14T15:17:23.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:22 smithi090 ceph-mon[41670]: from='client.47114 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi090-93363-64"}]: dispatch 2024-04-14T15:17:23.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:22 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1312471356' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi090-93363-64"}]: dispatch 2024-04-14T15:17:23.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:22 smithi090 ceph-mon[41670]: from='client.47114 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi090-93363-64"}]: dispatch 2024-04-14T15:17:23.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:22 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1312471356' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi090-93363-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:17:23.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:22 smithi090 ceph-mon[41670]: from='client.47114 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi090-93363-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:17:23.215 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:17:23.242 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:23.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:23.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:17:23.344 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:17:23.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:23.368 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:23.368 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:17:23.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:22 smithi092 ceph-mon[33966]: pgmap v506: 238 pgs: 238 active+clean; 8.4 MiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-14T15:17:23.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:22 smithi092 ceph-mon[33966]: from='client.48976 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi090-93363-63"}]': finished 2024-04-14T15:17:23.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:22 smithi092 ceph-mon[33966]: osdmap e375: 8 total, 8 up, 8 in 2024-04-14T15:17:23.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:22 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1312471356' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi090-93363-64"}]: dispatch 2024-04-14T15:17:23.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:22 smithi092 ceph-mon[33966]: from='client.47114 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi090-93363-64"}]: dispatch 2024-04-14T15:17:23.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:22 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1312471356' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi090-93363-64"}]: dispatch 2024-04-14T15:17:23.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:22 smithi092 ceph-mon[33966]: from='client.47114 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi090-93363-64"}]: dispatch 2024-04-14T15:17:23.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:22 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1312471356' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi090-93363-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:17:23.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:22 smithi092 ceph-mon[33966]: from='client.47114 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi090-93363-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:17:23.469 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:17:23.497 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:23.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:23.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:17:23.599 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:17:23.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:23.623 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:23.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:17:23.688 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:17:23.724 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:17:23.747 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:23.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:23.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:17:23.849 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:17:23.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:23.878 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:23.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:17:23.979 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:17:24.004 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:24.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:24.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:17:24.088 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:17:23 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:17:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:17:24.088 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:23 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:17:24.088 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:23 smithi090 ceph-mon[30399]: from='client.47114 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi090-93363-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:17:24.089 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:23 smithi090 ceph-mon[30399]: osdmap e376: 8 total, 8 up, 8 in 2024-04-14T15:17:24.089 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:23 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1312471356' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi090-93363-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi090-93363-64"}]: dispatch 2024-04-14T15:17:24.089 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:23 smithi090 ceph-mon[30399]: from='client.47114 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi090-93363-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi090-93363-64"}]: dispatch 2024-04-14T15:17:24.089 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:23 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:17:24.089 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:23 smithi090 ceph-mon[41670]: from='client.47114 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi090-93363-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:17:24.089 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:23 smithi090 ceph-mon[41670]: osdmap e376: 8 total, 8 up, 8 in 2024-04-14T15:17:24.089 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:23 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1312471356' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi090-93363-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi090-93363-64"}]: dispatch 2024-04-14T15:17:24.089 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:23 smithi090 ceph-mon[41670]: from='client.47114 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi090-93363-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi090-93363-64"}]: dispatch 2024-04-14T15:17:24.106 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:17:24.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:24.134 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:24.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:17:24.235 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:17:24.258 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:24.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:24.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:17:24.360 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:17:24.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:23 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:17:24.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:23 smithi092 ceph-mon[33966]: from='client.47114 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi090-93363-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:17:24.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:23 smithi092 ceph-mon[33966]: osdmap e376: 8 total, 8 up, 8 in 2024-04-14T15:17:24.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:23 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1312471356' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi090-93363-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi090-93363-64"}]: dispatch 2024-04-14T15:17:24.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:23 smithi092 ceph-mon[33966]: from='client.47114 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi090-93363-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi090-93363-64"}]: dispatch 2024-04-14T15:17:24.387 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:24.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:24.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:17:24.489 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:17:24.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:24.514 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:24.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:17:24.615 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:17:24.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:24.640 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:24.640 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:17:24.741 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:17:24.768 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:24.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:24.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:17:24.870 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:17:24.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:24.901 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:24.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:17:24.988 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:24 smithi090 ceph-mon[30399]: pgmap v509: 238 pgs: 238 active+clean; 8.4 MiB data, 598 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:17:24.988 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:24 smithi090 ceph-mon[30399]: osdmap e377: 8 total, 8 up, 8 in 2024-04-14T15:17:24.989 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:24 smithi090 ceph-mon[41670]: pgmap v509: 238 pgs: 238 active+clean; 8.4 MiB data, 598 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:17:24.989 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:24 smithi090 ceph-mon[41670]: osdmap e377: 8 total, 8 up, 8 in 2024-04-14T15:17:25.003 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:17:25.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:25.031 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:25.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:17:25.132 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:17:25.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:25.244 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:25.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:17:25.254 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-14T15:17:25.345 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:17:25.372 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:17:25 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:17:25.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:24 smithi092 ceph-mon[33966]: pgmap v509: 238 pgs: 238 active+clean; 8.4 MiB data, 598 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:17:25.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:24 smithi092 ceph-mon[33966]: osdmap e377: 8 total, 8 up, 8 in 2024-04-14T15:17:25.373 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:25.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:25.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:17:25.464 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:17:25.474 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:17:25.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:25.502 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:25.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:17:25.603 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:17:25.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:25.628 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:25.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:17:25.729 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:17:25.756 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:25.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:25.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:17:25.858 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:17:25.884 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:25.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:25.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:17:25.986 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:17:26.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:26.013 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:26.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:17:26.114 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:17:26.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.120+0000 7f2b6a31b700 1 -- 172.21.15.90:0/1407090583 >> v1:172.21.15.90:6789/0 conn(0x7f2b6410e200 legacy=0x7f2b641106c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:26.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.120+0000 7f2b6a31b700 1 -- 172.21.15.90:0/1407090583 shutdown_connections 2024-04-14T15:17:26.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.120+0000 7f2b6a31b700 1 -- 172.21.15.90:0/1407090583 >> 172.21.15.90:0/1407090583 conn(0x7f2b640fd2b0 msgr2=0x7f2b640ff6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:17:26.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.120+0000 7f2b6a31b700 1 -- 172.21.15.90:0/1407090583 shutdown_connections 2024-04-14T15:17:26.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.120+0000 7f2b6a31b700 1 -- 172.21.15.90:0/1407090583 wait complete. 2024-04-14T15:17:26.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.121+0000 7f2b6a31b700 1 librados: starting msgr at 2024-04-14T15:17:26.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.121+0000 7f2b6a31b700 1 librados: starting objecter 2024-04-14T15:17:26.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.121+0000 7f2b6a31b700 1 Processor -- start 2024-04-14T15:17:26.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.121+0000 7f2b6a31b700 1 -- start start 2024-04-14T15:17:26.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.121+0000 7f2b6a31b700 1 librados: setting wanted keys 2024-04-14T15:17:26.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.121+0000 7f2b6a31b700 1 librados: calling monclient init 2024-04-14T15:17:26.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.121+0000 7f2b6a31b700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f2b6410e000 con 0x7f2b6410e200 2024-04-14T15:17:26.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.121+0000 7f2b6a31b700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f2b641a8e20 con 0x7f2b64107780 2024-04-14T15:17:26.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.121+0000 7f2b6a31b700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f2b641aa080 con 0x7f2b6410a630 2024-04-14T15:17:26.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.121+0000 7f2b688b8700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f2b6410e200 0x7f2b641a66f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:56912/0 (socket says 172.21.15.90:56912) 2024-04-14T15:17:26.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.121+0000 7f2b688b8700 1 -- 172.21.15.90:0/1127137383 learned_addr learned my addr 172.21.15.90:0/1127137383 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:17:26.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.121+0000 7f2b637fe700 1 --1- 172.21.15.90:0/1127137383 >> v1:172.21.15.90:6790/0 conn(0x7f2b6410a630 0x7f2b6406b5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:58320/0 (socket says 172.21.15.90:58320) 2024-04-14T15:17:26.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.121+0000 7f2b617fa700 10 client.?.objecter ms_handle_connect 0x7f2b6410e200 2024-04-14T15:17:26.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.122+0000 7f2b617fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:17:26.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.122+0000 7f2b617fa700 10 client.?.objecter ms_handle_connect 0x7f2b6410a630 2024-04-14T15:17:26.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.122+0000 7f2b617fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:17:26.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.122+0000 7f2b617fa700 10 client.?.objecter ms_handle_connect 0x7f2b64107780 2024-04-14T15:17:26.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.122+0000 7f2b617fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:17:26.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.122+0000 7f2b617fa700 1 -- 172.21.15.90:0/1127137383 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 258068587 0 0) 0x7f2b641a8e20 con 0x7f2b64107780 2024-04-14T15:17:26.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.122+0000 7f2b617fa700 1 -- 172.21.15.90:0/1127137383 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b4c0031f0 con 0x7f2b64107780 2024-04-14T15:17:26.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.122+0000 7f2b617fa700 1 -- 172.21.15.90:0/1127137383 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1853729450 0 0) 0x7f2b6410e000 con 0x7f2b6410e200 2024-04-14T15:17:26.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.122+0000 7f2b617fa700 1 -- 172.21.15.90:0/1127137383 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b641a8e20 con 0x7f2b6410e200 2024-04-14T15:17:26.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.122+0000 7f2b617fa700 1 -- 172.21.15.90:0/1127137383 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2813347277 0 0) 0x7f2b641aa080 con 0x7f2b6410a630 2024-04-14T15:17:26.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.122+0000 7f2b617fa700 1 -- 172.21.15.90:0/1127137383 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b6410e000 con 0x7f2b6410a630 2024-04-14T15:17:26.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.122+0000 7f2b617fa700 1 -- 172.21.15.90:0/1127137383 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 164508473 0 0) 0x7f2b4c0031f0 con 0x7f2b64107780 2024-04-14T15:17:26.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.122+0000 7f2b617fa700 1 -- 172.21.15.90:0/1127137383 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2b641aa080 con 0x7f2b64107780 2024-04-14T15:17:26.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.122+0000 7f2b617fa700 1 -- 172.21.15.90:0/1127137383 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f2b58001830 con 0x7f2b64107780 2024-04-14T15:17:26.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.122+0000 7f2b617fa700 1 -- 172.21.15.90:0/1127137383 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2363034644 0 0) 0x7f2b641a8e20 con 0x7f2b6410e200 2024-04-14T15:17:26.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.122+0000 7f2b617fa700 1 -- 172.21.15.90:0/1127137383 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2b4c0031f0 con 0x7f2b6410e200 2024-04-14T15:17:26.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.122+0000 7f2b617fa700 1 -- 172.21.15.90:0/1127137383 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f2b5c0017b0 con 0x7f2b6410e200 2024-04-14T15:17:26.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.122+0000 7f2b617fa700 1 -- 172.21.15.90:0/1127137383 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3501692529 0 0) 0x7f2b6410e000 con 0x7f2b6410a630 2024-04-14T15:17:26.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.122+0000 7f2b617fa700 1 -- 172.21.15.90:0/1127137383 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2b641a8e20 con 0x7f2b6410a630 2024-04-14T15:17:26.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.122+0000 7f2b617fa700 1 -- 172.21.15.90:0/1127137383 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f2b54001830 con 0x7f2b6410a630 2024-04-14T15:17:26.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.122+0000 7f2b617fa700 1 -- 172.21.15.90:0/1127137383 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4082648661 0 0) 0x7f2b641aa080 con 0x7f2b64107780 2024-04-14T15:17:26.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.122+0000 7f2b617fa700 1 -- 172.21.15.90:0/1127137383 >> v1:172.21.15.90:6790/0 conn(0x7f2b6410a630 legacy=0x7f2b6406b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:26.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.122+0000 7f2b617fa700 1 -- 172.21.15.90:0/1127137383 >> v1:172.21.15.90:6789/0 conn(0x7f2b6410e200 legacy=0x7f2b641a66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:26.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.122+0000 7f2b617fa700 1 -- 172.21.15.90:0/1127137383 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b641ab2c0 con 0x7f2b64107780 2024-04-14T15:17:26.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.122+0000 7f2b6a31b700 1 -- 172.21.15.90:0/1127137383 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2b641aa2b0 con 0x7f2b64107780 2024-04-14T15:17:26.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.122+0000 7f2b6a31b700 10 client.49006.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:17:26.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.122+0000 7f2b6a31b700 1 -- 172.21.15.90:0/1127137383 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f2b641aa770 con 0x7f2b64107780 2024-04-14T15:17:26.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.122+0000 7f2b6a31b700 1 librados: init done 2024-04-14T15:17:26.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.122+0000 7f2b617fa700 1 -- 172.21.15.90:0/1127137383 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f2b58003b40 con 0x7f2b64107780 2024-04-14T15:17:26.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.122+0000 7f2b617fa700 1 -- 172.21.15.90:0/1127137383 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f2b580049d0 con 0x7f2b64107780 2024-04-14T15:17:26.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.124+0000 7f2b6a31b700 1 -- 172.21.15.90:0/1127137383 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b50005150 con 0x7f2b64107780 2024-04-14T15:17:26.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.124+0000 7f2b617fa700 1 -- 172.21.15.90:0/1127137383 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f2b5801b3c0 con 0x7f2b64107780 2024-04-14T15:17:26.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.124+0000 7f2b617fa700 1 -- 172.21.15.90:0/1127137383 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(378..378 src has 51..378) ==== 9007+0+0 (unknown 476709419 0 0) 0x7f2b58089a10 con 0x7f2b64107780 2024-04-14T15:17:26.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.124+0000 7f2b617fa700 10 client.49006.objecter ms_dispatch 0x7f2b64001550 osd_map(378..378 src has 51..378) 2024-04-14T15:17:26.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.124+0000 7f2b617fa700 3 client.49006.objecter handle_osd_map got epochs [378,378] > 0 2024-04-14T15:17:26.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.124+0000 7f2b617fa700 3 client.49006.objecter handle_osd_map decoding full epoch 378 2024-04-14T15:17:26.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.125+0000 7f2b617fa700 20 client.49006.objecter dump_active .. 0 homeless 2024-04-14T15:17:26.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.125+0000 7f2b617fa700 10 client.49006.objecter ms_handle_connect 0x7f2b4c071330 2024-04-14T15:17:26.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.129+0000 7f2b617fa700 1 -- 172.21.15.90:0/1127137383 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f2b580560d0 con 0x7f2b64107780 2024-04-14T15:17:26.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:26.140 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:26.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:17:26.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:25 smithi090 ceph-mon[30399]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:17:26.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:25 smithi090 ceph-mon[30399]: from='client.47114 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi090-93363-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi090-93363-64"}]': finished 2024-04-14T15:17:26.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:25 smithi090 ceph-mon[30399]: osdmap e378: 8 total, 8 up, 8 in 2024-04-14T15:17:26.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:25 smithi090 ceph-mon[41670]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:17:26.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:25 smithi090 ceph-mon[41670]: from='client.47114 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi090-93363-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi090-93363-64"}]': finished 2024-04-14T15:17:26.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:25 smithi090 ceph-mon[41670]: osdmap e378: 8 total, 8 up, 8 in 2024-04-14T15:17:26.242 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:17:26.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:26.268 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:26.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:17:26.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.355+0000 7f2b6a31b700 1 -- 172.21.15.90:0/1127137383 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd reweight", "id": 3, "weight": 1.0} v 0) -- 0x7f2b50005440 con 0x7f2b64107780 2024-04-14T15:17:26.369 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:17:26.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:25 smithi092 ceph-mon[33966]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:17:26.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:25 smithi092 ceph-mon[33966]: from='client.47114 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi090-93363-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi090-93363-64"}]': finished 2024-04-14T15:17:26.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:25 smithi092 ceph-mon[33966]: osdmap e378: 8 total, 8 up, 8 in 2024-04-14T15:17:26.395 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:26.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:26.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:17:26.498 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:17:26.524 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:26.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:26.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:17:26.626 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:17:26.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:26.651 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:26.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:17:26.752 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:17:26.780 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:26.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:26.781 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:17:26.809 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:17:26.881 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:17:26.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.883+0000 7f2b617fa700 1 -- 172.21.15.90:0/1127137383 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 3, "weight": 1.0}]=0 reweighted osd.3 to 1 (10000) v379) ==== 113+0+0 (unknown 3399009241 0 0) 0x7f2b58059d80 con 0x7f2b64107780 2024-04-14T15:17:26.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.885+0000 7f2b6a31b700 10 librados: watch_flush enter 2024-04-14T15:17:26.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.887+0000 7f2b6a31b700 10 librados: watch_flush exit 2024-04-14T15:17:26.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.887+0000 7f2b6a31b700 20 client.49006.objecter shutdown clearing up homeless session... 2024-04-14T15:17:26.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.887+0000 7f2b6a31b700 10 client.49006.objecter successfully canceled tick 2024-04-14T15:17:26.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.887+0000 7f2b6a31b700 1 -- 172.21.15.90:0/1127137383 >> v1:172.21.15.90:6800/420935427 conn(0x7f2b4c071330 legacy=0x7f2b4c0737f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:26.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.887+0000 7f2b6a31b700 1 -- 172.21.15.90:0/1127137383 >> v1:172.21.15.92:6789/0 conn(0x7f2b64107780 legacy=0x7f2b6406ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:26.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.888+0000 7f2b6a31b700 1 -- 172.21.15.90:0/1127137383 shutdown_connections 2024-04-14T15:17:26.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.888+0000 7f2b6a31b700 1 -- 172.21.15.90:0/1127137383 >> 172.21.15.90:0/1127137383 conn(0x7f2b640fd2b0 msgr2=0x7f2b64111770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:17:26.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.888+0000 7f2b6a31b700 1 -- 172.21.15.90:0/1127137383 shutdown_connections 2024-04-14T15:17:26.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.889+0000 7f2b6a31b700 1 -- 172.21.15.90:0/1127137383 wait complete. 2024-04-14T15:17:26.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:26.889+0000 7f2b6a31b700 1 librados: shutdown 2024-04-14T15:17:26.893 INFO:teuthology.orchestra.run.smithi090.stderr:reweighted osd.3 to 1 (10000) 2024-04-14T15:17:26.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:26.912 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:26.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:17:27.014 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:17:27.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:27.039 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:27.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:17:27.140 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:17:27.166 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:27.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:27.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:17:27.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:26 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:17:27.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:26 smithi090 ceph-mon[30399]: pgmap v512: 238 pgs: 238 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-14T15:17:27.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:26 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:17:27.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:26 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:17:27.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:26 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1127137383' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-14T15:17:27.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:26 smithi090 ceph-mon[30399]: from='client.49006 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-14T15:17:27.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:26 smithi090 ceph-mon[30399]: from='client.49006 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-14T15:17:27.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:26 smithi090 ceph-mon[30399]: osdmap e379: 8 total, 8 up, 8 in 2024-04-14T15:17:27.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:26 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:17:27.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:26 smithi090 ceph-mon[41670]: pgmap v512: 238 pgs: 238 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-14T15:17:27.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:26 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:17:27.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:26 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:17:27.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:26 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1127137383' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-14T15:17:27.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:26 smithi090 ceph-mon[41670]: from='client.49006 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-14T15:17:27.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:26 smithi090 ceph-mon[41670]: from='client.49006 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-14T15:17:27.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:26 smithi090 ceph-mon[41670]: osdmap e379: 8 total, 8 up, 8 in 2024-04-14T15:17:27.268 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:17:27.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:27.293 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:27.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:17:27.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:26 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:17:27.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:26 smithi092 ceph-mon[33966]: pgmap v512: 238 pgs: 238 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-14T15:17:27.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:26 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:17:27.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:26 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:17:27.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:26 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1127137383' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-14T15:17:27.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:26 smithi092 ceph-mon[33966]: from='client.49006 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-14T15:17:27.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:26 smithi092 ceph-mon[33966]: from='client.49006 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-14T15:17:27.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:26 smithi092 ceph-mon[33966]: osdmap e379: 8 total, 8 up, 8 in 2024-04-14T15:17:27.394 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:17:27.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:27.422 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:27.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:17:27.524 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:17:27.552 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:27.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:27.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:17:27.648 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd reweight 4 1 2024-04-14T15:17:27.654 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:17:27.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:27.717 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:27.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:17:27.818 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:17:27.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:27.844 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:27.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:17:27.945 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:17:27.973 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:27.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:27.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:17:28.075 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:17:28.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:28.100 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:28.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:17:28.201 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:17:28.225 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:28.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:28.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:17:28.245 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-14T15:17:28.245 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-14T15:17:28.246 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-14T15:17:28.246 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-14T15:17:28.327 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:17:28.353 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:28.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:28.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:17:28.455 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:17:28.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:28.483 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:28.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:17:28.584 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:17:28.610 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:28.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:28.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:17:28.712 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:17:28.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:28.737 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:28.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:17:28.838 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:17:28.865 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:28.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:28.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:17:28.941 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:28 smithi090 ceph-mon[41670]: pgmap v515: 246 pgs: 8 unknown, 238 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-14T15:17:28.941 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:28 smithi090 ceph-mon[41670]: osdmap e380: 8 total, 8 up, 8 in 2024-04-14T15:17:28.941 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:28 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1312471356' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi090-93363-64"}]: dispatch 2024-04-14T15:17:28.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:28 smithi090 ceph-mon[41670]: from='client.47114 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi090-93363-64"}]: dispatch 2024-04-14T15:17:28.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:28 smithi090 ceph-mon[30399]: pgmap v515: 246 pgs: 8 unknown, 238 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-14T15:17:28.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:28 smithi090 ceph-mon[30399]: osdmap e380: 8 total, 8 up, 8 in 2024-04-14T15:17:28.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:28 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1312471356' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi090-93363-64"}]: dispatch 2024-04-14T15:17:28.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:28 smithi090 ceph-mon[30399]: from='client.47114 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi090-93363-64"}]: dispatch 2024-04-14T15:17:28.968 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:17:28.992 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:28.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:28.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:17:29.094 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:17:29.120 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:29.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:29.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:17:29.222 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:17:29.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:29.247 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:29.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:17:29.348 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:17:29.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:28 smithi092 ceph-mon[33966]: pgmap v515: 246 pgs: 8 unknown, 238 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-14T15:17:29.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:28 smithi092 ceph-mon[33966]: osdmap e380: 8 total, 8 up, 8 in 2024-04-14T15:17:29.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:28 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1312471356' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi090-93363-64"}]: dispatch 2024-04-14T15:17:29.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:28 smithi092 ceph-mon[33966]: from='client.47114 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi090-93363-64"}]: dispatch 2024-04-14T15:17:29.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:29.373 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:29.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:17:29.474 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:17:29.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:29.502 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:29.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:17:29.603 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:17:29.630 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:29.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:29.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:17:29.731 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:17:29.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:29.758 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:29.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:17:29.823 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-14T15:17:29.859 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:17:29.890 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:29.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:29.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:17:29.992 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:17:30.018 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:30.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:30.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:17:30.120 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:17:30.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:30.147 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:30.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:17:30.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:29 smithi090 ceph-mon[41670]: from='client.47114 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi090-93363-64"}]': finished 2024-04-14T15:17:30.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:29 smithi090 ceph-mon[41670]: osdmap e381: 8 total, 8 up, 8 in 2024-04-14T15:17:30.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:29 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1312471356' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi090-93363-64"}]: dispatch 2024-04-14T15:17:30.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:29 smithi090 ceph-mon[41670]: from='client.47114 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi090-93363-64"}]: dispatch 2024-04-14T15:17:30.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:29 smithi090 ceph-mon[30399]: from='client.47114 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi090-93363-64"}]': finished 2024-04-14T15:17:30.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:29 smithi090 ceph-mon[30399]: osdmap e381: 8 total, 8 up, 8 in 2024-04-14T15:17:30.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:29 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1312471356' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi090-93363-64"}]: dispatch 2024-04-14T15:17:30.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:29 smithi090 ceph-mon[30399]: from='client.47114 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi090-93363-64"}]: dispatch 2024-04-14T15:17:30.249 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:17:30.276 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:30.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:30.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:17:30.285 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:17:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:29 smithi092 ceph-mon[33966]: from='client.47114 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi090-93363-64"}]': finished 2024-04-14T15:17:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:29 smithi092 ceph-mon[33966]: osdmap e381: 8 total, 8 up, 8 in 2024-04-14T15:17:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:29 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1312471356' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi090-93363-64"}]: dispatch 2024-04-14T15:17:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:29 smithi092 ceph-mon[33966]: from='client.47114 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi090-93363-64"}]: dispatch 2024-04-14T15:17:30.379 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:17:30.407 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:30.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:30.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:17:30.509 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:17:30.535 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:30.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:30.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:17:30.637 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:17:30.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:30.665 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:30.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:17:30.766 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:17:30.791 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:30.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:30.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:17:30.893 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:17:30.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:30.922 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:30.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:17:31.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.008+0000 7f7025164700 1 -- 172.21.15.90:0/1913742746 >> v1:172.21.15.90:6790/0 conn(0x7f7020108870 legacy=0x7f7020108c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:31.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.008+0000 7f7025164700 1 -- 172.21.15.90:0/1913742746 shutdown_connections 2024-04-14T15:17:31.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.008+0000 7f7025164700 1 -- 172.21.15.90:0/1913742746 >> 172.21.15.90:0/1913742746 conn(0x7f70200fe3c0 msgr2=0x7f70201007e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:17:31.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.008+0000 7f7025164700 1 -- 172.21.15.90:0/1913742746 shutdown_connections 2024-04-14T15:17:31.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.008+0000 7f7025164700 1 -- 172.21.15.90:0/1913742746 wait complete. 2024-04-14T15:17:31.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.009+0000 7f7025164700 1 librados: starting msgr at 2024-04-14T15:17:31.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.009+0000 7f7025164700 1 librados: starting objecter 2024-04-14T15:17:31.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.009+0000 7f7025164700 1 Processor -- start 2024-04-14T15:17:31.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.009+0000 7f7025164700 1 -- start start 2024-04-14T15:17:31.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.009+0000 7f7025164700 1 librados: setting wanted keys 2024-04-14T15:17:31.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.009+0000 7f7025164700 1 librados: calling monclient init 2024-04-14T15:17:31.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.009+0000 7f7025164700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f70201a79e0 con 0x7f7020108870 2024-04-14T15:17:31.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.009+0000 7f7025164700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f70201a8c00 con 0x7f702010b720 2024-04-14T15:17:31.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.009+0000 7f7025164700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f70201a9e40 con 0x7f702010f2f0 2024-04-14T15:17:31.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.010+0000 7f701ed9d700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f7020108870 0x7f702010eb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:54804/0 (socket says 172.21.15.90:54804) 2024-04-14T15:17:31.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.010+0000 7f701ed9d700 1 -- 172.21.15.90:0/3948071780 learned_addr learned my addr 172.21.15.90:0/3948071780 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:17:31.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.010+0000 7f7007fff700 10 client.?.objecter ms_handle_connect 0x7f7020108870 2024-04-14T15:17:31.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.010+0000 7f7007fff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:17:31.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.010+0000 7f7007fff700 10 client.?.objecter ms_handle_connect 0x7f702010f2f0 2024-04-14T15:17:31.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.010+0000 7f7007fff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:17:31.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.010+0000 7f7007fff700 10 client.?.objecter ms_handle_connect 0x7f702010b720 2024-04-14T15:17:31.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.010+0000 7f7007fff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:17:31.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.010+0000 7f7007fff700 1 -- 172.21.15.90:0/3948071780 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2621885576 0 0) 0x7f70201a79e0 con 0x7f7020108870 2024-04-14T15:17:31.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.010+0000 7f7007fff700 1 -- 172.21.15.90:0/3948071780 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f70080039c0 con 0x7f7020108870 2024-04-14T15:17:31.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.010+0000 7f7007fff700 1 -- 172.21.15.90:0/3948071780 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2098274688 0 0) 0x7f70201a8c00 con 0x7f702010b720 2024-04-14T15:17:31.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.010+0000 7f7007fff700 1 -- 172.21.15.90:0/3948071780 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f70201a79e0 con 0x7f702010b720 2024-04-14T15:17:31.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.010+0000 7f7007fff700 1 -- 172.21.15.90:0/3948071780 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1147751377 0 0) 0x7f70201a9e40 con 0x7f702010f2f0 2024-04-14T15:17:31.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.010+0000 7f7007fff700 1 -- 172.21.15.90:0/3948071780 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f70201a8c00 con 0x7f702010f2f0 2024-04-14T15:17:31.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.010+0000 7f7007fff700 1 -- 172.21.15.90:0/3948071780 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3799866581 0 0) 0x7f70201a79e0 con 0x7f702010b720 2024-04-14T15:17:31.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.010+0000 7f7007fff700 1 -- 172.21.15.90:0/3948071780 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f70201a9e40 con 0x7f702010b720 2024-04-14T15:17:31.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.010+0000 7f7007fff700 1 -- 172.21.15.90:0/3948071780 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f7014001a30 con 0x7f702010b720 2024-04-14T15:17:31.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.010+0000 7f7007fff700 1 -- 172.21.15.90:0/3948071780 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1802974876 0 0) 0x7f70201a8c00 con 0x7f702010f2f0 2024-04-14T15:17:31.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.010+0000 7f7007fff700 1 -- 172.21.15.90:0/3948071780 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f70201a79e0 con 0x7f702010f2f0 2024-04-14T15:17:31.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.010+0000 7f7007fff700 1 -- 172.21.15.90:0/3948071780 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f70100018b0 con 0x7f702010f2f0 2024-04-14T15:17:31.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.010+0000 7f7007fff700 1 -- 172.21.15.90:0/3948071780 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 256253070 0 0) 0x7f70080039c0 con 0x7f7020108870 2024-04-14T15:17:31.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.010+0000 7f7007fff700 1 -- 172.21.15.90:0/3948071780 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f70201a8c00 con 0x7f7020108870 2024-04-14T15:17:31.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.010+0000 7f7007fff700 1 -- 172.21.15.90:0/3948071780 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f70180047f0 con 0x7f7020108870 2024-04-14T15:17:31.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.010+0000 7f7007fff700 1 -- 172.21.15.90:0/3948071780 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 914156237 0 0) 0x7f70201a9e40 con 0x7f702010b720 2024-04-14T15:17:31.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.010+0000 7f7007fff700 1 -- 172.21.15.90:0/3948071780 >> v1:172.21.15.90:6790/0 conn(0x7f702010f2f0 legacy=0x7f70201a60a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:31.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.010+0000 7f7007fff700 1 -- 172.21.15.90:0/3948071780 >> v1:172.21.15.90:6789/0 conn(0x7f7020108870 legacy=0x7f702010eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:31.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.010+0000 7f7007fff700 1 -- 172.21.15.90:0/3948071780 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70201ab080 con 0x7f702010b720 2024-04-14T15:17:31.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.011+0000 7f7025164700 1 -- 172.21.15.90:0/3948071780 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f70201a8e30 con 0x7f702010b720 2024-04-14T15:17:31.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.011+0000 7f7025164700 10 client.49018.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:17:31.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.011+0000 7f7025164700 1 -- 172.21.15.90:0/3948071780 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f70201a92f0 con 0x7f702010b720 2024-04-14T15:17:31.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.011+0000 7f7025164700 1 librados: init done 2024-04-14T15:17:31.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.011+0000 7f7007fff700 1 -- 172.21.15.90:0/3948071780 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f7014001d20 con 0x7f702010b720 2024-04-14T15:17:31.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.011+0000 7f7007fff700 1 -- 172.21.15.90:0/3948071780 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f7014004a40 con 0x7f702010b720 2024-04-14T15:17:31.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.011+0000 7f7025164700 1 -- 172.21.15.90:0/3948071780 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f700c005130 con 0x7f702010b720 2024-04-14T15:17:31.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.012+0000 7f7007fff700 1 -- 172.21.15.90:0/3948071780 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f7014003930 con 0x7f702010b720 2024-04-14T15:17:31.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.013+0000 7f7007fff700 1 -- 172.21.15.90:0/3948071780 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(383..383 src has 51..383) ==== 8473+0+0 (unknown 1873486149 0 0) 0x7f7014089700 con 0x7f702010b720 2024-04-14T15:17:31.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.013+0000 7f7007fff700 10 client.49018.objecter ms_dispatch 0x7f7020001550 osd_map(383..383 src has 51..383) 2024-04-14T15:17:31.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.013+0000 7f7007fff700 3 client.49018.objecter handle_osd_map got epochs [383,383] > 0 2024-04-14T15:17:31.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.013+0000 7f7007fff700 3 client.49018.objecter handle_osd_map decoding full epoch 383 2024-04-14T15:17:31.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.013+0000 7f7007fff700 20 client.49018.objecter dump_active .. 0 homeless 2024-04-14T15:17:31.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.013+0000 7f7007fff700 1 -- 172.21.15.90:0/3948071780 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v383) ==== 54+0+16 (unknown 2502529344 0 2420000192) 0x7f7014089b00 con 0x7f702010b720 2024-04-14T15:17:31.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.013+0000 7f7007fff700 10 client.49018.objecter ms_handle_connect 0x7f7008079b40 2024-04-14T15:17:31.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.013+0000 7f7025164700 10 client.49018.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:17:31.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.013+0000 7f7025164700 15 client.49018.objecter _session_command_op_assign -1 1 2024-04-14T15:17:31.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.013+0000 7f7025164700 20 client.49018.objecter _get_session s=0x7f700c005410 osd=0 3 2024-04-14T15:17:31.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.013+0000 7f7025164700 20 client.49018.objecter put_session s=0x7f700c005410 osd=0 3 2024-04-14T15:17:31.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.013+0000 7f7025164700 20 client.49018.objecter _get_session s=0x7f700c005410 osd=0 3 2024-04-14T15:17:31.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.013+0000 7f7025164700 15 client.49018.objecter _session_command_op_remove -1 1 2024-04-14T15:17:31.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.013+0000 7f7025164700 20 client.49018.objecter get_session s=0x7f700c005410 osd=0 3 2024-04-14T15:17:31.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.013+0000 7f7025164700 15 client.49018.objecter _session_command_op_assign 0 1 2024-04-14T15:17:31.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.013+0000 7f7025164700 20 client.49018.objecter put_session s=0x7f700c005410 osd=0 4 2024-04-14T15:17:31.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.013+0000 7f7025164700 10 client.49018.objecter _send_command 1 2024-04-14T15:17:31.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.013+0000 7f7025164700 1 -- 172.21.15.90:0/3948071780 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f700c004b80 con 0x7f700c004760 2024-04-14T15:17:31.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.013+0000 7f7007fff700 10 client.49018.objecter ms_handle_connect 0x7f700c004760 2024-04-14T15:17:31.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.014+0000 7f7007fff700 1 -- 172.21.15.90:0/3948071780 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f700c004b80 con 0x7f700c004760 2024-04-14T15:17:31.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.014+0000 7f7007fff700 10 client.49018.objecter ms_dispatch 0x7f7020001550 command_reply(tid 1: 0 ) 2024-04-14T15:17:31.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.014+0000 7f7007fff700 10 client.49018.objecter _finish_command 1 = system:0 2024-04-14T15:17:31.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.014+0000 7f7007fff700 20 client.49018.objecter put_session s=0x7f700c005410 osd=0 4 2024-04-14T15:17:31.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.014+0000 7f7007fff700 15 client.49018.objecter _session_command_op_remove 0 1 2024-04-14T15:17:31.018 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:17:31.023 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:17:31.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.031+0000 7f7025164700 10 client.49018.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:17:31.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.031+0000 7f7025164700 15 client.49018.objecter _session_command_op_assign -1 2 2024-04-14T15:17:31.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.031+0000 7f7025164700 20 client.49018.objecter _get_session s=0x7f700c005410 osd=0 3 2024-04-14T15:17:31.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.031+0000 7f7025164700 20 client.49018.objecter put_session s=0x7f700c005410 osd=0 3 2024-04-14T15:17:31.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.031+0000 7f7025164700 20 client.49018.objecter _get_session s=0x7f700c005410 osd=0 3 2024-04-14T15:17:31.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.031+0000 7f7025164700 15 client.49018.objecter _session_command_op_remove -1 2 2024-04-14T15:17:31.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.031+0000 7f7025164700 20 client.49018.objecter get_session s=0x7f700c005410 osd=0 3 2024-04-14T15:17:31.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.031+0000 7f7025164700 15 client.49018.objecter _session_command_op_assign 0 2 2024-04-14T15:17:31.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.031+0000 7f7025164700 20 client.49018.objecter put_session s=0x7f700c005410 osd=0 4 2024-04-14T15:17:31.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.031+0000 7f7025164700 10 client.49018.objecter _send_command 2 2024-04-14T15:17:31.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.031+0000 7f7025164700 1 -- 172.21.15.90:0/3948071780 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f700c007a20 con 0x7f700c004760 2024-04-14T15:17:31.033 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:30 smithi090 ceph-mon[41670]: pgmap v518: 238 pgs: 238 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:17:31.033 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:30 smithi090 ceph-mon[41670]: from='client.47114 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi090-93363-64"}]': finished 2024-04-14T15:17:31.033 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:30 smithi090 ceph-mon[41670]: osdmap e382: 8 total, 8 up, 8 in 2024-04-14T15:17:31.033 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:30 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/468264671' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi090-93363-65"}]: dispatch 2024-04-14T15:17:31.033 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:30 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/468264671' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi090-93363-65"}]: dispatch 2024-04-14T15:17:31.033 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:30 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/468264671' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi090-93363-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:17:31.033 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:30 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:17:31.033 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:30 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:17:31.034 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:30 smithi090 ceph-mon[30399]: pgmap v518: 238 pgs: 238 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:17:31.034 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:30 smithi090 ceph-mon[30399]: from='client.47114 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi090-93363-64"}]': finished 2024-04-14T15:17:31.034 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:30 smithi090 ceph-mon[30399]: osdmap e382: 8 total, 8 up, 8 in 2024-04-14T15:17:31.034 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:30 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/468264671' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi090-93363-65"}]: dispatch 2024-04-14T15:17:31.034 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:30 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/468264671' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi090-93363-65"}]: dispatch 2024-04-14T15:17:31.034 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:30 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/468264671' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi090-93363-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:17:31.034 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:30 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:17:31.034 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:30 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:17:31.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.033+0000 7f7007fff700 1 -- 172.21.15.90:0/3948071780 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f700c007a20 con 0x7f700c004760 2024-04-14T15:17:31.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.033+0000 7f7007fff700 10 client.49018.objecter ms_dispatch 0x7f7020001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:17:31.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.033+0000 7f7007fff700 10 client.49018.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:17:31.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.033+0000 7f7007fff700 20 client.49018.objecter put_session s=0x7f700c005410 osd=0 4 2024-04-14T15:17:31.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.033+0000 7f7007fff700 15 client.49018.objecter _session_command_op_remove 0 2 2024-04-14T15:17:31.035 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:17:31.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.034+0000 7f7025164700 10 client.49018.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:17:31.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.034+0000 7f7025164700 15 client.49018.objecter _session_command_op_assign -1 3 2024-04-14T15:17:31.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.034+0000 7f7025164700 20 client.49018.objecter _get_session s=0x7f700c004140 osd=1 3 2024-04-14T15:17:31.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.034+0000 7f7025164700 20 client.49018.objecter put_session s=0x7f700c004140 osd=1 3 2024-04-14T15:17:31.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.034+0000 7f7025164700 20 client.49018.objecter _get_session s=0x7f700c004140 osd=1 3 2024-04-14T15:17:31.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.034+0000 7f7025164700 15 client.49018.objecter _session_command_op_remove -1 3 2024-04-14T15:17:31.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.034+0000 7f7025164700 20 client.49018.objecter get_session s=0x7f700c004140 osd=1 3 2024-04-14T15:17:31.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.034+0000 7f7025164700 15 client.49018.objecter _session_command_op_assign 1 3 2024-04-14T15:17:31.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.034+0000 7f7025164700 20 client.49018.objecter put_session s=0x7f700c004140 osd=1 4 2024-04-14T15:17:31.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.034+0000 7f7025164700 10 client.49018.objecter _send_command 3 2024-04-14T15:17:31.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.034+0000 7f7025164700 1 -- 172.21.15.90:0/3948071780 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f700c0021b0 con 0x7f700c0042b0 2024-04-14T15:17:31.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.034+0000 7f7007fff700 10 client.49018.objecter ms_handle_connect 0x7f700c0042b0 2024-04-14T15:17:31.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.035+0000 7f7007fff700 1 -- 172.21.15.90:0/3948071780 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f700c0021b0 con 0x7f700c0042b0 2024-04-14T15:17:31.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.035+0000 7f7007fff700 10 client.49018.objecter ms_dispatch 0x7f7020001550 command_reply(tid 3: 0 ) 2024-04-14T15:17:31.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.035+0000 7f7007fff700 10 client.49018.objecter _finish_command 3 = system:0 2024-04-14T15:17:31.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.035+0000 7f7007fff700 20 client.49018.objecter put_session s=0x7f700c004140 osd=1 4 2024-04-14T15:17:31.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.035+0000 7f7007fff700 15 client.49018.objecter _session_command_op_remove 1 3 2024-04-14T15:17:31.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:31.050 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:31.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:17:31.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.052+0000 7f7025164700 10 client.49018.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:17:31.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.052+0000 7f7025164700 15 client.49018.objecter _session_command_op_assign -1 4 2024-04-14T15:17:31.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.052+0000 7f7025164700 20 client.49018.objecter _get_session s=0x7f700c004140 osd=1 3 2024-04-14T15:17:31.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.052+0000 7f7025164700 20 client.49018.objecter put_session s=0x7f700c004140 osd=1 3 2024-04-14T15:17:31.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.052+0000 7f7025164700 20 client.49018.objecter _get_session s=0x7f700c004140 osd=1 3 2024-04-14T15:17:31.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.052+0000 7f7025164700 15 client.49018.objecter _session_command_op_remove -1 4 2024-04-14T15:17:31.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.052+0000 7f7025164700 20 client.49018.objecter get_session s=0x7f700c004140 osd=1 3 2024-04-14T15:17:31.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.052+0000 7f7025164700 15 client.49018.objecter _session_command_op_assign 1 4 2024-04-14T15:17:31.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.052+0000 7f7025164700 20 client.49018.objecter put_session s=0x7f700c004140 osd=1 4 2024-04-14T15:17:31.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.052+0000 7f7025164700 10 client.49018.objecter _send_command 4 2024-04-14T15:17:31.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.052+0000 7f7025164700 1 -- 172.21.15.90:0/3948071780 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f700c001630 con 0x7f700c0042b0 2024-04-14T15:17:31.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.054+0000 7f7007fff700 1 -- 172.21.15.90:0/3948071780 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f700c001630 con 0x7f700c0042b0 2024-04-14T15:17:31.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.054+0000 7f7007fff700 10 client.49018.objecter ms_dispatch 0x7f7020001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:17:31.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.054+0000 7f7007fff700 10 client.49018.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:17:31.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.054+0000 7f7007fff700 20 client.49018.objecter put_session s=0x7f700c004140 osd=1 4 2024-04-14T15:17:31.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.054+0000 7f7007fff700 15 client.49018.objecter _session_command_op_remove 1 4 2024-04-14T15:17:31.055 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:17:31.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.054+0000 7f7025164700 10 client.49018.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:17:31.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.054+0000 7f7025164700 15 client.49018.objecter _session_command_op_assign -1 5 2024-04-14T15:17:31.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.054+0000 7f7025164700 20 client.49018.objecter _get_session s=0x7f700c0083a0 osd=2 3 2024-04-14T15:17:31.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.054+0000 7f7025164700 20 client.49018.objecter put_session s=0x7f700c0083a0 osd=2 3 2024-04-14T15:17:31.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.054+0000 7f7025164700 20 client.49018.objecter _get_session s=0x7f700c0083a0 osd=2 3 2024-04-14T15:17:31.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.054+0000 7f7025164700 15 client.49018.objecter _session_command_op_remove -1 5 2024-04-14T15:17:31.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.054+0000 7f7025164700 20 client.49018.objecter get_session s=0x7f700c0083a0 osd=2 3 2024-04-14T15:17:31.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.054+0000 7f7025164700 15 client.49018.objecter _session_command_op_assign 2 5 2024-04-14T15:17:31.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.054+0000 7f7025164700 20 client.49018.objecter put_session s=0x7f700c0083a0 osd=2 4 2024-04-14T15:17:31.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.054+0000 7f7025164700 10 client.49018.objecter _send_command 5 2024-04-14T15:17:31.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.054+0000 7f7025164700 1 -- 172.21.15.90:0/3948071780 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f700c011110 con 0x7f700c00d600 2024-04-14T15:17:31.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.055+0000 7f7007fff700 10 client.49018.objecter ms_handle_connect 0x7f700c00d600 2024-04-14T15:17:31.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.055+0000 7f7007fff700 1 -- 172.21.15.90:0/3948071780 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f700c011110 con 0x7f700c00d600 2024-04-14T15:17:31.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.055+0000 7f7007fff700 10 client.49018.objecter ms_dispatch 0x7f7020001550 command_reply(tid 5: 0 ) 2024-04-14T15:17:31.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.055+0000 7f7007fff700 10 client.49018.objecter _finish_command 5 = system:0 2024-04-14T15:17:31.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.055+0000 7f7007fff700 20 client.49018.objecter put_session s=0x7f700c0083a0 osd=2 4 2024-04-14T15:17:31.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.055+0000 7f7007fff700 15 client.49018.objecter _session_command_op_remove 2 5 2024-04-14T15:17:31.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.074+0000 7f7025164700 10 client.49018.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:17:31.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.074+0000 7f7025164700 15 client.49018.objecter _session_command_op_assign -1 6 2024-04-14T15:17:31.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.074+0000 7f7025164700 20 client.49018.objecter _get_session s=0x7f700c0083a0 osd=2 3 2024-04-14T15:17:31.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.074+0000 7f7025164700 20 client.49018.objecter put_session s=0x7f700c0083a0 osd=2 3 2024-04-14T15:17:31.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.074+0000 7f7025164700 20 client.49018.objecter _get_session s=0x7f700c0083a0 osd=2 3 2024-04-14T15:17:31.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.074+0000 7f7025164700 15 client.49018.objecter _session_command_op_remove -1 6 2024-04-14T15:17:31.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.074+0000 7f7025164700 20 client.49018.objecter get_session s=0x7f700c0083a0 osd=2 3 2024-04-14T15:17:31.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.074+0000 7f7025164700 15 client.49018.objecter _session_command_op_assign 2 6 2024-04-14T15:17:31.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.074+0000 7f7025164700 20 client.49018.objecter put_session s=0x7f700c0083a0 osd=2 4 2024-04-14T15:17:31.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.074+0000 7f7025164700 10 client.49018.objecter _send_command 6 2024-04-14T15:17:31.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.074+0000 7f7025164700 1 -- 172.21.15.90:0/3948071780 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f700c007e20 con 0x7f700c00d600 2024-04-14T15:17:31.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.076+0000 7f7007fff700 1 -- 172.21.15.90:0/3948071780 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f700c007e20 con 0x7f700c00d600 2024-04-14T15:17:31.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.076+0000 7f7007fff700 10 client.49018.objecter ms_dispatch 0x7f7020001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:17:31.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.076+0000 7f7007fff700 10 client.49018.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:17:31.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.076+0000 7f7007fff700 20 client.49018.objecter put_session s=0x7f700c0083a0 osd=2 4 2024-04-14T15:17:31.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.076+0000 7f7007fff700 15 client.49018.objecter _session_command_op_remove 2 6 2024-04-14T15:17:31.076 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:17:31.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.076+0000 7f7025164700 10 client.49018.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:17:31.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.076+0000 7f7025164700 15 client.49018.objecter _session_command_op_assign -1 7 2024-04-14T15:17:31.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.076+0000 7f7025164700 20 client.49018.objecter _get_session s=0x7f700c00cc50 osd=3 3 2024-04-14T15:17:31.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.076+0000 7f7025164700 20 client.49018.objecter put_session s=0x7f700c00cc50 osd=3 3 2024-04-14T15:17:31.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.076+0000 7f7025164700 20 client.49018.objecter _get_session s=0x7f700c00cc50 osd=3 3 2024-04-14T15:17:31.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.076+0000 7f7025164700 15 client.49018.objecter _session_command_op_remove -1 7 2024-04-14T15:17:31.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.076+0000 7f7025164700 20 client.49018.objecter get_session s=0x7f700c00cc50 osd=3 3 2024-04-14T15:17:31.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.076+0000 7f7025164700 15 client.49018.objecter _session_command_op_assign 3 7 2024-04-14T15:17:31.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.076+0000 7f7025164700 20 client.49018.objecter put_session s=0x7f700c00cc50 osd=3 4 2024-04-14T15:17:31.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.076+0000 7f7025164700 10 client.49018.objecter _send_command 7 2024-04-14T15:17:31.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.076+0000 7f7025164700 1 -- 172.21.15.90:0/3948071780 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f700c00d400 con 0x7f700c012440 2024-04-14T15:17:31.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.076+0000 7f7007fff700 10 client.49018.objecter ms_handle_connect 0x7f700c012440 2024-04-14T15:17:31.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.077+0000 7f7007fff700 1 -- 172.21.15.90:0/3948071780 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f700c00d400 con 0x7f700c012440 2024-04-14T15:17:31.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.077+0000 7f7007fff700 10 client.49018.objecter ms_dispatch 0x7f7020001550 command_reply(tid 7: 0 ) 2024-04-14T15:17:31.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.077+0000 7f7007fff700 10 client.49018.objecter _finish_command 7 = system:0 2024-04-14T15:17:31.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.077+0000 7f7007fff700 20 client.49018.objecter put_session s=0x7f700c00cc50 osd=3 4 2024-04-14T15:17:31.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.077+0000 7f7007fff700 15 client.49018.objecter _session_command_op_remove 3 7 2024-04-14T15:17:31.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.094+0000 7f7025164700 10 client.49018.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:17:31.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.094+0000 7f7025164700 15 client.49018.objecter _session_command_op_assign -1 8 2024-04-14T15:17:31.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.094+0000 7f7025164700 20 client.49018.objecter _get_session s=0x7f700c00cc50 osd=3 3 2024-04-14T15:17:31.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.094+0000 7f7025164700 20 client.49018.objecter put_session s=0x7f700c00cc50 osd=3 3 2024-04-14T15:17:31.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.094+0000 7f7025164700 20 client.49018.objecter _get_session s=0x7f700c00cc50 osd=3 3 2024-04-14T15:17:31.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.094+0000 7f7025164700 15 client.49018.objecter _session_command_op_remove -1 8 2024-04-14T15:17:31.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.094+0000 7f7025164700 20 client.49018.objecter get_session s=0x7f700c00cc50 osd=3 3 2024-04-14T15:17:31.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.094+0000 7f7025164700 15 client.49018.objecter _session_command_op_assign 3 8 2024-04-14T15:17:31.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.094+0000 7f7025164700 20 client.49018.objecter put_session s=0x7f700c00cc50 osd=3 4 2024-04-14T15:17:31.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.094+0000 7f7025164700 10 client.49018.objecter _send_command 8 2024-04-14T15:17:31.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.094+0000 7f7025164700 1 -- 172.21.15.90:0/3948071780 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f700c00c7f0 con 0x7f700c012440 2024-04-14T15:17:31.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.095+0000 7f7007fff700 1 -- 172.21.15.90:0/3948071780 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f700c00c7f0 con 0x7f700c012440 2024-04-14T15:17:31.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.095+0000 7f7007fff700 10 client.49018.objecter ms_dispatch 0x7f7020001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:17:31.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.095+0000 7f7007fff700 10 client.49018.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:17:31.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.095+0000 7f7007fff700 20 client.49018.objecter put_session s=0x7f700c00cc50 osd=3 4 2024-04-14T15:17:31.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.095+0000 7f7007fff700 15 client.49018.objecter _session_command_op_remove 3 8 2024-04-14T15:17:31.096 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:17:31.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.096+0000 7f7025164700 10 client.49018.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:17:31.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.096+0000 7f7025164700 15 client.49018.objecter _session_command_op_assign -1 9 2024-04-14T15:17:31.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.096+0000 7f7025164700 20 client.49018.objecter _get_session s=0x7f700c00c5a0 osd=4 3 2024-04-14T15:17:31.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.096+0000 7f7025164700 20 client.49018.objecter put_session s=0x7f700c00c5a0 osd=4 3 2024-04-14T15:17:31.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.096+0000 7f7025164700 20 client.49018.objecter _get_session s=0x7f700c00c5a0 osd=4 3 2024-04-14T15:17:31.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.096+0000 7f7025164700 15 client.49018.objecter _session_command_op_remove -1 9 2024-04-14T15:17:31.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.096+0000 7f7025164700 20 client.49018.objecter get_session s=0x7f700c00c5a0 osd=4 3 2024-04-14T15:17:31.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.096+0000 7f7025164700 15 client.49018.objecter _session_command_op_assign 4 9 2024-04-14T15:17:31.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.096+0000 7f7025164700 20 client.49018.objecter put_session s=0x7f700c00c5a0 osd=4 4 2024-04-14T15:17:31.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.096+0000 7f7025164700 10 client.49018.objecter _send_command 9 2024-04-14T15:17:31.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.096+0000 7f7025164700 1 -- 172.21.15.90:0/3948071780 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f700c01a900 con 0x7f700c011fa0 2024-04-14T15:17:31.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.096+0000 7f7007fff700 10 client.49018.objecter ms_handle_connect 0x7f700c011fa0 2024-04-14T15:17:31.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.097+0000 7f7007fff700 1 -- 172.21.15.90:0/3948071780 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f700c01a900 con 0x7f700c011fa0 2024-04-14T15:17:31.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.097+0000 7f7007fff700 10 client.49018.objecter ms_dispatch 0x7f7020001550 command_reply(tid 9: 0 ) 2024-04-14T15:17:31.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.097+0000 7f7007fff700 10 client.49018.objecter _finish_command 9 = system:0 2024-04-14T15:17:31.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.097+0000 7f7007fff700 20 client.49018.objecter put_session s=0x7f700c00c5a0 osd=4 4 2024-04-14T15:17:31.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.097+0000 7f7007fff700 15 client.49018.objecter _session_command_op_remove 4 9 2024-04-14T15:17:31.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.113+0000 7f7025164700 10 client.49018.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:17:31.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.113+0000 7f7025164700 15 client.49018.objecter _session_command_op_assign -1 10 2024-04-14T15:17:31.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.113+0000 7f7025164700 20 client.49018.objecter _get_session s=0x7f700c00c5a0 osd=4 3 2024-04-14T15:17:31.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.113+0000 7f7025164700 20 client.49018.objecter put_session s=0x7f700c00c5a0 osd=4 3 2024-04-14T15:17:31.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.113+0000 7f7025164700 20 client.49018.objecter _get_session s=0x7f700c00c5a0 osd=4 3 2024-04-14T15:17:31.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.113+0000 7f7025164700 15 client.49018.objecter _session_command_op_remove -1 10 2024-04-14T15:17:31.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.113+0000 7f7025164700 20 client.49018.objecter get_session s=0x7f700c00c5a0 osd=4 3 2024-04-14T15:17:31.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.113+0000 7f7025164700 15 client.49018.objecter _session_command_op_assign 4 10 2024-04-14T15:17:31.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.113+0000 7f7025164700 20 client.49018.objecter put_session s=0x7f700c00c5a0 osd=4 4 2024-04-14T15:17:31.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.113+0000 7f7025164700 10 client.49018.objecter _send_command 10 2024-04-14T15:17:31.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.113+0000 7f7025164700 1 -- 172.21.15.90:0/3948071780 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f700c01a900 con 0x7f700c011fa0 2024-04-14T15:17:31.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.115+0000 7f7007fff700 1 -- 172.21.15.90:0/3948071780 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f700c01a900 con 0x7f700c011fa0 2024-04-14T15:17:31.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.115+0000 7f7007fff700 10 client.49018.objecter ms_dispatch 0x7f7020001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:17:31.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.115+0000 7f7007fff700 10 client.49018.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:17:31.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.115+0000 7f7007fff700 20 client.49018.objecter put_session s=0x7f700c00c5a0 osd=4 4 2024-04-14T15:17:31.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.115+0000 7f7007fff700 15 client.49018.objecter _session_command_op_remove 4 10 2024-04-14T15:17:31.116 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:17:31.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.115+0000 7f7025164700 10 client.49018.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:17:31.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.115+0000 7f7025164700 15 client.49018.objecter _session_command_op_assign -1 11 2024-04-14T15:17:31.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.115+0000 7f7025164700 20 client.49018.objecter _get_session s=0x7f700c01a900 osd=5 3 2024-04-14T15:17:31.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.115+0000 7f7025164700 20 client.49018.objecter put_session s=0x7f700c01a900 osd=5 3 2024-04-14T15:17:31.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.115+0000 7f7025164700 20 client.49018.objecter _get_session s=0x7f700c01a900 osd=5 3 2024-04-14T15:17:31.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.115+0000 7f7025164700 15 client.49018.objecter _session_command_op_remove -1 11 2024-04-14T15:17:31.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.115+0000 7f7025164700 20 client.49018.objecter get_session s=0x7f700c01a900 osd=5 3 2024-04-14T15:17:31.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.115+0000 7f7025164700 15 client.49018.objecter _session_command_op_assign 5 11 2024-04-14T15:17:31.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.115+0000 7f7025164700 20 client.49018.objecter put_session s=0x7f700c01a900 osd=5 4 2024-04-14T15:17:31.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.115+0000 7f7025164700 10 client.49018.objecter _send_command 11 2024-04-14T15:17:31.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.115+0000 7f7025164700 1 -- 172.21.15.90:0/3948071780 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f700c01ec40 con 0x7f700c01af80 2024-04-14T15:17:31.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.116+0000 7f7007fff700 10 client.49018.objecter ms_handle_connect 0x7f700c01af80 2024-04-14T15:17:31.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.116+0000 7f7007fff700 1 -- 172.21.15.90:0/3948071780 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f700c01ec40 con 0x7f700c01af80 2024-04-14T15:17:31.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.116+0000 7f7007fff700 10 client.49018.objecter ms_dispatch 0x7f7020001550 command_reply(tid 11: 0 ) 2024-04-14T15:17:31.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.116+0000 7f7007fff700 10 client.49018.objecter _finish_command 11 = system:0 2024-04-14T15:17:31.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.116+0000 7f7007fff700 20 client.49018.objecter put_session s=0x7f700c01a900 osd=5 4 2024-04-14T15:17:31.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.116+0000 7f7007fff700 15 client.49018.objecter _session_command_op_remove 5 11 2024-04-14T15:17:31.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.133+0000 7f7025164700 10 client.49018.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:17:31.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.133+0000 7f7025164700 15 client.49018.objecter _session_command_op_assign -1 12 2024-04-14T15:17:31.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.133+0000 7f7025164700 20 client.49018.objecter _get_session s=0x7f700c01a900 osd=5 3 2024-04-14T15:17:31.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.133+0000 7f7025164700 20 client.49018.objecter put_session s=0x7f700c01a900 osd=5 3 2024-04-14T15:17:31.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.133+0000 7f7025164700 20 client.49018.objecter _get_session s=0x7f700c01a900 osd=5 3 2024-04-14T15:17:31.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.133+0000 7f7025164700 15 client.49018.objecter _session_command_op_remove -1 12 2024-04-14T15:17:31.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.133+0000 7f7025164700 20 client.49018.objecter get_session s=0x7f700c01a900 osd=5 3 2024-04-14T15:17:31.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.133+0000 7f7025164700 15 client.49018.objecter _session_command_op_assign 5 12 2024-04-14T15:17:31.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.133+0000 7f7025164700 20 client.49018.objecter put_session s=0x7f700c01a900 osd=5 4 2024-04-14T15:17:31.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.133+0000 7f7025164700 10 client.49018.objecter _send_command 12 2024-04-14T15:17:31.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.133+0000 7f7025164700 1 -- 172.21.15.90:0/3948071780 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f700c011510 con 0x7f700c01af80 2024-04-14T15:17:31.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.135+0000 7f7007fff700 1 -- 172.21.15.90:0/3948071780 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f700c011510 con 0x7f700c01af80 2024-04-14T15:17:31.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.135+0000 7f7007fff700 10 client.49018.objecter ms_dispatch 0x7f7020001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:17:31.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.135+0000 7f7007fff700 10 client.49018.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:17:31.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.135+0000 7f7007fff700 20 client.49018.objecter put_session s=0x7f700c01a900 osd=5 4 2024-04-14T15:17:31.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.135+0000 7f7007fff700 15 client.49018.objecter _session_command_op_remove 5 12 2024-04-14T15:17:31.136 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:17:31.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.135+0000 7f7025164700 10 client.49018.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:17:31.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.135+0000 7f7025164700 15 client.49018.objecter _session_command_op_assign -1 13 2024-04-14T15:17:31.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.135+0000 7f7025164700 20 client.49018.objecter _get_session s=0x7f700c00c9f0 osd=6 3 2024-04-14T15:17:31.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.135+0000 7f7025164700 20 client.49018.objecter put_session s=0x7f700c00c9f0 osd=6 3 2024-04-14T15:17:31.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.135+0000 7f7025164700 20 client.49018.objecter _get_session s=0x7f700c00c9f0 osd=6 3 2024-04-14T15:17:31.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.135+0000 7f7025164700 15 client.49018.objecter _session_command_op_remove -1 13 2024-04-14T15:17:31.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.135+0000 7f7025164700 20 client.49018.objecter get_session s=0x7f700c00c9f0 osd=6 3 2024-04-14T15:17:31.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.135+0000 7f7025164700 15 client.49018.objecter _session_command_op_assign 6 13 2024-04-14T15:17:31.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.135+0000 7f7025164700 20 client.49018.objecter put_session s=0x7f700c00c9f0 osd=6 4 2024-04-14T15:17:31.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.135+0000 7f7025164700 10 client.49018.objecter _send_command 13 2024-04-14T15:17:31.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.135+0000 7f7025164700 1 -- 172.21.15.90:0/3948071780 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f700c020400 con 0x7f700c0112c0 2024-04-14T15:17:31.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.135+0000 7f7007fff700 10 client.49018.objecter ms_handle_connect 0x7f700c0112c0 2024-04-14T15:17:31.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.136+0000 7f7007fff700 1 -- 172.21.15.90:0/3948071780 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f701000c950 con 0x7f700c0112c0 2024-04-14T15:17:31.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.136+0000 7f7007fff700 10 client.49018.objecter ms_dispatch 0x7f7020001550 command_reply(tid 13: 0 ) 2024-04-14T15:17:31.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.136+0000 7f7007fff700 10 client.49018.objecter _finish_command 13 = system:0 2024-04-14T15:17:31.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.136+0000 7f7007fff700 20 client.49018.objecter put_session s=0x7f700c00c9f0 osd=6 4 2024-04-14T15:17:31.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.136+0000 7f7007fff700 15 client.49018.objecter _session_command_op_remove 6 13 2024-04-14T15:17:31.151 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:17:31.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.152+0000 7f7025164700 10 client.49018.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:17:31.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.152+0000 7f7025164700 15 client.49018.objecter _session_command_op_assign -1 14 2024-04-14T15:17:31.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.152+0000 7f7025164700 20 client.49018.objecter _get_session s=0x7f700c00c9f0 osd=6 3 2024-04-14T15:17:31.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.152+0000 7f7025164700 20 client.49018.objecter put_session s=0x7f700c00c9f0 osd=6 3 2024-04-14T15:17:31.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.152+0000 7f7025164700 20 client.49018.objecter _get_session s=0x7f700c00c9f0 osd=6 3 2024-04-14T15:17:31.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.152+0000 7f7025164700 15 client.49018.objecter _session_command_op_remove -1 14 2024-04-14T15:17:31.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.152+0000 7f7025164700 20 client.49018.objecter get_session s=0x7f700c00c9f0 osd=6 3 2024-04-14T15:17:31.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.152+0000 7f7025164700 15 client.49018.objecter _session_command_op_assign 6 14 2024-04-14T15:17:31.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.152+0000 7f7025164700 20 client.49018.objecter put_session s=0x7f700c00c9f0 osd=6 4 2024-04-14T15:17:31.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.152+0000 7f7025164700 10 client.49018.objecter _send_command 14 2024-04-14T15:17:31.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.152+0000 7f7025164700 1 -- 172.21.15.90:0/3948071780 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f700c0184a0 con 0x7f700c0112c0 2024-04-14T15:17:31.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.154+0000 7f7007fff700 1 -- 172.21.15.90:0/3948071780 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f700c0184a0 con 0x7f700c0112c0 2024-04-14T15:17:31.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.154+0000 7f7007fff700 10 client.49018.objecter ms_dispatch 0x7f7020001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:17:31.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.154+0000 7f7007fff700 10 client.49018.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:17:31.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.154+0000 7f7007fff700 20 client.49018.objecter put_session s=0x7f700c00c9f0 osd=6 4 2024-04-14T15:17:31.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.154+0000 7f7007fff700 15 client.49018.objecter _session_command_op_remove 6 14 2024-04-14T15:17:31.155 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:17:31.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.154+0000 7f7025164700 10 client.49018.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:17:31.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.154+0000 7f7025164700 15 client.49018.objecter _session_command_op_assign -1 15 2024-04-14T15:17:31.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.154+0000 7f7025164700 20 client.49018.objecter _get_session s=0x7f700c018700 osd=7 3 2024-04-14T15:17:31.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.154+0000 7f7025164700 20 client.49018.objecter put_session s=0x7f700c018700 osd=7 3 2024-04-14T15:17:31.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.154+0000 7f7025164700 20 client.49018.objecter _get_session s=0x7f700c018700 osd=7 3 2024-04-14T15:17:31.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.154+0000 7f7025164700 15 client.49018.objecter _session_command_op_remove -1 15 2024-04-14T15:17:31.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.154+0000 7f7025164700 20 client.49018.objecter get_session s=0x7f700c018700 osd=7 3 2024-04-14T15:17:31.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.154+0000 7f7025164700 15 client.49018.objecter _session_command_op_assign 7 15 2024-04-14T15:17:31.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.154+0000 7f7025164700 20 client.49018.objecter put_session s=0x7f700c018700 osd=7 4 2024-04-14T15:17:31.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.154+0000 7f7025164700 10 client.49018.objecter _send_command 15 2024-04-14T15:17:31.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.154+0000 7f7025164700 1 -- 172.21.15.90:0/3948071780 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f700c027810 con 0x7f700c023d00 2024-04-14T15:17:31.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.155+0000 7f7007fff700 10 client.49018.objecter ms_handle_connect 0x7f700c023d00 2024-04-14T15:17:31.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.156+0000 7f7007fff700 1 -- 172.21.15.90:0/3948071780 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f700c027810 con 0x7f700c023d00 2024-04-14T15:17:31.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.156+0000 7f7007fff700 10 client.49018.objecter ms_dispatch 0x7f7020001550 command_reply(tid 15: 0 ) 2024-04-14T15:17:31.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.156+0000 7f7007fff700 10 client.49018.objecter _finish_command 15 = system:0 2024-04-14T15:17:31.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.156+0000 7f7007fff700 20 client.49018.objecter put_session s=0x7f700c018700 osd=7 4 2024-04-14T15:17:31.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.156+0000 7f7007fff700 15 client.49018.objecter _session_command_op_remove 7 15 2024-04-14T15:17:31.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.172+0000 7f7025164700 10 client.49018.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:17:31.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.172+0000 7f7025164700 15 client.49018.objecter _session_command_op_assign -1 16 2024-04-14T15:17:31.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.172+0000 7f7025164700 20 client.49018.objecter _get_session s=0x7f700c018700 osd=7 3 2024-04-14T15:17:31.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.172+0000 7f7025164700 20 client.49018.objecter put_session s=0x7f700c018700 osd=7 3 2024-04-14T15:17:31.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.172+0000 7f7025164700 20 client.49018.objecter _get_session s=0x7f700c018700 osd=7 3 2024-04-14T15:17:31.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.172+0000 7f7025164700 15 client.49018.objecter _session_command_op_remove -1 16 2024-04-14T15:17:31.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.172+0000 7f7025164700 20 client.49018.objecter get_session s=0x7f700c018700 osd=7 3 2024-04-14T15:17:31.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.172+0000 7f7025164700 15 client.49018.objecter _session_command_op_assign 7 16 2024-04-14T15:17:31.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.172+0000 7f7025164700 20 client.49018.objecter put_session s=0x7f700c018700 osd=7 4 2024-04-14T15:17:31.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.172+0000 7f7025164700 10 client.49018.objecter _send_command 16 2024-04-14T15:17:31.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.172+0000 7f7025164700 1 -- 172.21.15.90:0/3948071780 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f700c011720 con 0x7f700c023d00 2024-04-14T15:17:31.174 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:31.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.174+0000 7f7007fff700 1 -- 172.21.15.90:0/3948071780 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f700c011720 con 0x7f700c023d00 2024-04-14T15:17:31.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.174+0000 7f7007fff700 10 client.49018.objecter ms_dispatch 0x7f7020001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:17:31.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.174+0000 7f7007fff700 10 client.49018.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:17:31.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.174+0000 7f7007fff700 20 client.49018.objecter put_session s=0x7f700c018700 osd=7 4 2024-04-14T15:17:31.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.174+0000 7f7007fff700 15 client.49018.objecter _session_command_op_remove 7 16 2024-04-14T15:17:31.175 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:17:31.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:31.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:17:31.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.175+0000 7f7025164700 10 librados: watch_flush enter 2024-04-14T15:17:31.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.175+0000 7f7025164700 10 librados: watch_flush exit 2024-04-14T15:17:31.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.175+0000 7f7025164700 10 client.49018.objecter close_session for osd.0 2024-04-14T15:17:31.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.175+0000 7f7025164700 1 -- 172.21.15.90:0/3948071780 >> v1:172.21.15.90:6801/3761806885 conn(0x7f700c004760 legacy=0x7f700c002780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:31.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.175+0000 7f7025164700 20 client.49018.objecter put_session s=0x7f700c005410 osd=0 1 2024-04-14T15:17:31.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.175+0000 7f7025164700 10 client.49018.objecter close_session for osd.1 2024-04-14T15:17:31.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.175+0000 7f7025164700 1 -- 172.21.15.90:0/3948071780 >> v1:172.21.15.90:6805/4170361699 conn(0x7f700c0042b0 legacy=0x7f700c00ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:31.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.175+0000 7f7025164700 20 client.49018.objecter put_session s=0x7f700c004140 osd=1 1 2024-04-14T15:17:31.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.175+0000 7f7025164700 10 client.49018.objecter close_session for osd.2 2024-04-14T15:17:31.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.175+0000 7f7025164700 1 -- 172.21.15.90:0/3948071780 >> v1:172.21.15.90:6809/1686184251 conn(0x7f700c00d600 legacy=0x7f700c00f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:31.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.175+0000 7f7025164700 20 client.49018.objecter put_session s=0x7f700c0083a0 osd=2 1 2024-04-14T15:17:31.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.175+0000 7f7025164700 10 client.49018.objecter close_session for osd.3 2024-04-14T15:17:31.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.175+0000 7f7025164700 1 -- 172.21.15.90:0/3948071780 >> v1:172.21.15.90:6813/2270166774 conn(0x7f700c012440 legacy=0x7f700c014830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:31.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.175+0000 7f7025164700 20 client.49018.objecter put_session s=0x7f700c00cc50 osd=3 1 2024-04-14T15:17:31.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.175+0000 7f7025164700 10 client.49018.objecter close_session for osd.4 2024-04-14T15:17:31.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.175+0000 7f7025164700 1 -- 172.21.15.90:0/3948071780 >> v1:172.21.15.92:6800/3887549733 conn(0x7f700c011fa0 legacy=0x7f700c015f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:31.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.175+0000 7f7025164700 20 client.49018.objecter put_session s=0x7f700c00c5a0 osd=4 1 2024-04-14T15:17:31.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.175+0000 7f7025164700 10 client.49018.objecter close_session for osd.5 2024-04-14T15:17:31.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.175+0000 7f7025164700 1 -- 172.21.15.90:0/3948071780 >> v1:172.21.15.92:6804/3244561440 conn(0x7f700c01af80 legacy=0x7f700c01d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:31.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.175+0000 7f7025164700 20 client.49018.objecter put_session s=0x7f700c01a900 osd=5 1 2024-04-14T15:17:31.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.175+0000 7f7025164700 10 client.49018.objecter close_session for osd.6 2024-04-14T15:17:31.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.175+0000 7f7025164700 1 -- 172.21.15.90:0/3948071780 >> v1:172.21.15.92:6808/636125963 conn(0x7f700c0112c0 legacy=0x7f700c01ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:31.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.175+0000 7f701f59e700 1 -- 172.21.15.90:0/3948071780 reap_dead start 2024-04-14T15:17:31.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.175+0000 7f7025164700 20 client.49018.objecter put_session s=0x7f700c00c9f0 osd=6 1 2024-04-14T15:17:31.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.175+0000 7f7025164700 10 client.49018.objecter close_session for osd.7 2024-04-14T15:17:31.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.175+0000 7f7025164700 1 -- 172.21.15.90:0/3948071780 >> v1:172.21.15.92:6812/1773706716 conn(0x7f700c023d00 legacy=0x7f700c0260f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:31.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.175+0000 7f701f59e700 1 -- 172.21.15.90:0/3948071780 reap_dead start 2024-04-14T15:17:31.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.175+0000 7f7025164700 20 client.49018.objecter put_session s=0x7f700c018700 osd=7 1 2024-04-14T15:17:31.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.175+0000 7f7025164700 20 client.49018.objecter shutdown clearing up homeless session... 2024-04-14T15:17:31.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.175+0000 7f7025164700 10 client.49018.objecter successfully canceled tick 2024-04-14T15:17:31.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.175+0000 7f701f59e700 1 -- 172.21.15.90:0/3948071780 reap_dead start 2024-04-14T15:17:31.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.175+0000 7f701f59e700 1 -- 172.21.15.90:0/3948071780 reap_dead start 2024-04-14T15:17:31.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.175+0000 7f7025164700 1 -- 172.21.15.90:0/3948071780 >> v1:172.21.15.90:6800/420935427 conn(0x7f7008079b40 legacy=0x7f700807c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:31.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.175+0000 7f7025164700 1 -- 172.21.15.90:0/3948071780 >> v1:172.21.15.92:6789/0 conn(0x7f702010b720 legacy=0x7f70201a27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:31.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.175+0000 7f7025164700 1 -- 172.21.15.90:0/3948071780 shutdown_connections 2024-04-14T15:17:31.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.175+0000 7f7025164700 1 -- 172.21.15.90:0/3948071780 >> 172.21.15.90:0/3948071780 conn(0x7f70200fe3c0 msgr2=0x7f7020102870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:17:31.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.175+0000 7f7025164700 1 -- 172.21.15.90:0/3948071780 shutdown_connections 2024-04-14T15:17:31.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.175+0000 7f7025164700 1 -- 172.21.15.90:0/3948071780 wait complete. 2024-04-14T15:17:31.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:31.175+0000 7f7025164700 1 librados: shutdown 2024-04-14T15:17:31.180 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:17:31.180 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:17:31.180 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:17:31.180 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:17:31.180 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:17:31.180 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:17:31.180 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:17:31.180 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:17:31.277 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:17:31.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:31.302 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:31.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:17:31.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:30 smithi092 ceph-mon[33966]: pgmap v518: 238 pgs: 238 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:17:31.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:30 smithi092 ceph-mon[33966]: from='client.47114 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi090-93363-64"}]': finished 2024-04-14T15:17:31.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:30 smithi092 ceph-mon[33966]: osdmap e382: 8 total, 8 up, 8 in 2024-04-14T15:17:31.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:30 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/468264671' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi090-93363-65"}]: dispatch 2024-04-14T15:17:31.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:30 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/468264671' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi090-93363-65"}]: dispatch 2024-04-14T15:17:31.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:30 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/468264671' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi090-93363-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:17:31.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:30 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:17:31.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:30 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:17:31.403 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:17:31.430 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:31.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:31.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:17:31.532 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:17:31.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:31.557 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:31.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:17:31.658 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:17:31.686 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:31.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:31.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:17:31.788 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:17:31.814 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:31.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:31.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:17:31.916 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:17:31.944 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:31.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:31.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:17:32.004 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:31 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/468264671' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi090-93363-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:17:32.004 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:31 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:17:32.004 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:31 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-49"}]: dispatch 2024-04-14T15:17:32.004 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:31 smithi090 ceph-mon[41670]: osdmap e383: 8 total, 8 up, 8 in 2024-04-14T15:17:32.004 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:31 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/468264671' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi090-93363-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi090-93363-65"}]: dispatch 2024-04-14T15:17:32.004 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:31 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-49"}]: dispatch 2024-04-14T15:17:32.004 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:31 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3948071780' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:17:32.005 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:31 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/468264671' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi090-93363-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:17:32.005 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:31 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:17:32.005 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:31 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-49"}]: dispatch 2024-04-14T15:17:32.005 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:31 smithi090 ceph-mon[30399]: osdmap e383: 8 total, 8 up, 8 in 2024-04-14T15:17:32.005 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:31 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/468264671' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi090-93363-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi090-93363-65"}]: dispatch 2024-04-14T15:17:32.005 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:31 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-49"}]: dispatch 2024-04-14T15:17:32.005 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:31 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3948071780' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:17:32.046 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:17:32.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:32.070 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:32.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:17:32.171 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:17:32.194 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:32.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:32.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:17:32.296 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:17:32.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:32.321 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:32.321 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:17:32.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:31 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/468264671' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi090-93363-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:17:32.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:31 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:17:32.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:31 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-49"}]: dispatch 2024-04-14T15:17:32.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:31 smithi092 ceph-mon[33966]: osdmap e383: 8 total, 8 up, 8 in 2024-04-14T15:17:32.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:31 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/468264671' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi090-93363-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi090-93363-65"}]: dispatch 2024-04-14T15:17:32.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:31 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-49"}]: dispatch 2024-04-14T15:17:32.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:31 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3948071780' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:17:32.422 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:17:32.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:32.447 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:32.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:17:32.548 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:17:32.576 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:32.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:32.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:17:32.677 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:17:32.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:32.702 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:32.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:17:32.803 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:17:32.831 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:32.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:32.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:17:32.924 INFO:tasks.workunit.client.0.smithi090.stdout: 63:head 2024-04-14T15:17:32.924 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:5d165639:::164:head 2024-04-14T15:17:32.924 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:f43765fc:::165:head 2024-04-14T15:17:32.924 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:b4c720e9:::166:head 2024-04-14T15:17:32.924 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:e694b040:::167:head 2024-04-14T15:17:32.924 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:afa38db2:::168:head 2024-04-14T15:17:32.925 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:77ba9f53:::169:head 2024-04-14T15:17:32.925 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:87495034:::170:head 2024-04-14T15:17:32.925 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:7c96bf0e:::171:head 2024-04-14T15:17:32.925 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:dbe346cc:::172:head 2024-04-14T15:17:32.925 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:e943ec24:::173:head 2024-04-14T15:17:32.925 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:f97a9c0c:::174:head 2024-04-14T15:17:32.925 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:6f26e74d:::175:head 2024-04-14T15:17:32.925 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:4f95e106:::176:head 2024-04-14T15:17:32.925 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:0e6f2f8f:::177:head 2024-04-14T15:17:32.925 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:05db05f1:::178:head 2024-04-14T15:17:32.925 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:38a78d66:::179:head 2024-04-14T15:17:32.925 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:d095610b:::180:head 2024-04-14T15:17:32.925 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:a1a9d709:::181:head 2024-04-14T15:17:32.925 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:1e5d39db:::182:head 2024-04-14T15:17:32.926 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:f7df4fb9:::183:head 2024-04-14T15:17:32.926 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:03a7f161:::184:head 2024-04-14T15:17:32.926 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:ba70721e:::185:head 2024-04-14T15:17:32.926 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:28e5662d:::186:head 2024-04-14T15:17:32.926 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:973d52de:::187:head 2024-04-14T15:17:32.926 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:4303eb1c:::188:head 2024-04-14T15:17:32.926 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:b990b48e:::189:head 2024-04-14T15:17:32.926 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:29b8165b:::190:head 2024-04-14T15:17:32.926 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:3547f197:::191:head 2024-04-14T15:17:32.926 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:7e260936:::192:head 2024-04-14T15:17:32.926 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:1abec7b1:::193:head 2024-04-14T15:17:32.926 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:10fdda93:::194:head 2024-04-14T15:17:32.926 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:15817eea:::195:head 2024-04-14T15:17:32.926 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:770bab57:::196:head 2024-04-14T15:17:32.927 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:ed9e13e7:::197:head 2024-04-14T15:17:32.927 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:71471a8f:::198:head 2024-04-14T15:17:32.927 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: checking for 276:10fb1d02:::199:head 2024-04-14T15:17:32.927 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (7988 ms) 2024-04-14T15:17:32.927 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2024-04-14T15:17:32.927 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: got ls 1713107807,0 2024-04-14T15:17:32.927 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: first is 1713107807 2024-04-14T15:17:32.927 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: got ls 1713107807,0 2024-04-14T15:17:32.927 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: got ls 1713107807,0 2024-04-14T15:17:32.927 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: got ls 1713107807,1713107809,1713107810,0 2024-04-14T15:17:32.927 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: got ls 1713107807,1713107809,1713107810,0 2024-04-14T15:17:32.927 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: got ls 1713107807,1713107809,1713107810,0 2024-04-14T15:17:32.927 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: got ls 1713107807,1713107809,1713107810,1713107812,1713107813,0 2024-04-14T15:17:32.927 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: got ls 1713107807,1713107809,1713107810,1713107812,1713107813,0 2024-04-14T15:17:32.927 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: got ls 1713107807,1713107809,1713107810,1713107812,1713107813,0 2024-04-14T15:17:32.928 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: got ls 1713107807,1713107809,1713107810,1713107812,1713107813,1713107815,1713107816,0 2024-04-14T15:17:32.928 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: got ls 1713107807,1713107809,1713107810,1713107812,1713107813,1713107815,1713107816,0 2024-04-14T15:17:32.928 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: got ls 1713107807,1713107809,1713107810,1713107812,1713107813,1713107815,1713107816,0 2024-04-14T15:17:32.928 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: got ls 1713107810,1713107812,1713107813,1713107815,1713107816,1713107818,1713107819,0 2024-04-14T15:17:32.928 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: first now 1713107810, trimmed 2024-04-14T15:17:32.928 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (20986 ms) 2024-04-14T15:17:32.928 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2024-04-14T15:17:32.928 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: foo0 2024-04-14T15:17:32.928 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-04-14T15:17:32.928 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (13998 ms) 2024-04-14T15:17:32.928 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2024-04-14T15:17:32.933 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:17:32.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:32.904+0000 7f701eddd700 1 -- 172.21.15.90:0/217404954 >> v1:172.21.15.90:6789/0 conn(0x7f7018103a20 legacy=0x7f7018103e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:32.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:32.904+0000 7f701eddd700 1 -- 172.21.15.90:0/217404954 shutdown_connections 2024-04-14T15:17:32.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:32.904+0000 7f701eddd700 1 -- 172.21.15.90:0/217404954 >> 172.21.15.90:0/217404954 conn(0x7f70180fd290 msgr2=0x7f70180ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:17:32.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:32.904+0000 7f701eddd700 1 -- 172.21.15.90:0/217404954 shutdown_connections 2024-04-14T15:17:32.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:32.904+0000 7f701eddd700 1 -- 172.21.15.90:0/217404954 wait complete. 2024-04-14T15:17:32.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:32.904+0000 7f701eddd700 1 librados: starting msgr at 2024-04-14T15:17:32.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:32.904+0000 7f701eddd700 1 librados: starting objecter 2024-04-14T15:17:32.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:32.905+0000 7f701eddd700 1 Processor -- start 2024-04-14T15:17:32.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:32.905+0000 7f701eddd700 1 -- start start 2024-04-14T15:17:32.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:32.905+0000 7f701eddd700 1 librados: setting wanted keys 2024-04-14T15:17:32.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:32.905+0000 7f701eddd700 1 librados: calling monclient init 2024-04-14T15:17:32.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:32.905+0000 7f701eddd700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f70181a9d30 con 0x7f701810b830 2024-04-14T15:17:32.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:32.905+0000 7f701eddd700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f70181aaf50 con 0x7f701810f340 2024-04-14T15:17:32.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:32.905+0000 7f701eddd700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f70181ac190 con 0x7f7018103a20 2024-04-14T15:17:32.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:32.905+0000 7f701cb79700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f7018103a20 0x7f7018103000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:48420/0 (socket says 172.21.15.90:48420) 2024-04-14T15:17:32.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:32.905+0000 7f701cb79700 1 -- 172.21.15.90:0/3775089771 learned_addr learned my addr 172.21.15.90:0/3775089771 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:17:32.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:32.905+0000 7f7015ffb700 10 client.?.objecter ms_handle_connect 0x7f7018103a20 2024-04-14T15:17:32.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:32.905+0000 7f7015ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:17:32.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:32.905+0000 7f7015ffb700 10 client.?.objecter ms_handle_connect 0x7f701810b830 2024-04-14T15:17:32.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:32.905+0000 7f7015ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:17:32.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:32.905+0000 7f7015ffb700 10 client.?.objecter ms_handle_connect 0x7f701810f340 2024-04-14T15:17:32.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:32.905+0000 7f7015ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:17:32.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:32.905+0000 7f7015ffb700 1 -- 172.21.15.90:0/3775089771 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1760113953 0 0) 0x7f70181ac190 con 0x7f7018103a20 2024-04-14T15:17:32.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:32.905+0000 7f7015ffb700 1 -- 172.21.15.90:0/3775089771 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f70080035b0 con 0x7f7018103a20 2024-04-14T15:17:32.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:32.905+0000 7f7015ffb700 1 -- 172.21.15.90:0/3775089771 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2677878149 0 0) 0x7f70181a9d30 con 0x7f701810b830 2024-04-14T15:17:32.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:32.905+0000 7f7015ffb700 1 -- 172.21.15.90:0/3775089771 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f70181ac190 con 0x7f701810b830 2024-04-14T15:17:32.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:32.905+0000 7f7015ffb700 1 -- 172.21.15.90:0/3775089771 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1780304720 0 0) 0x7f70181aaf50 con 0x7f701810f340 2024-04-14T15:17:32.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:32.905+0000 7f7015ffb700 1 -- 172.21.15.90:0/3775089771 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f70181a9d30 con 0x7f701810f340 2024-04-14T15:17:32.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:32.905+0000 7f7015ffb700 1 -- 172.21.15.90:0/3775089771 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3729101304 0 0) 0x7f70080035b0 con 0x7f7018103a20 2024-04-14T15:17:32.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:32.905+0000 7f7015ffb700 1 -- 172.21.15.90:0/3775089771 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f70181aaf50 con 0x7f7018103a20 2024-04-14T15:17:32.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:32.905+0000 7f7015ffb700 1 -- 172.21.15.90:0/3775089771 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f700c0015d0 con 0x7f7018103a20 2024-04-14T15:17:32.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:32.905+0000 7f7015ffb700 1 -- 172.21.15.90:0/3775089771 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3647909542 0 0) 0x7f70181ac190 con 0x7f701810b830 2024-04-14T15:17:32.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:32.905+0000 7f7015ffb700 1 -- 172.21.15.90:0/3775089771 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f70080035b0 con 0x7f701810b830 2024-04-14T15:17:32.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:32.905+0000 7f7015ffb700 1 -- 172.21.15.90:0/3775089771 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f7004001870 con 0x7f701810b830 2024-04-14T15:17:32.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:32.905+0000 7f7015ffb700 1 -- 172.21.15.90:0/3775089771 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3463145441 0 0) 0x7f70181a9d30 con 0x7f701810f340 2024-04-14T15:17:32.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:32.906+0000 7f7015ffb700 1 -- 172.21.15.90:0/3775089771 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f70181ac190 con 0x7f701810f340 2024-04-14T15:17:32.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:32.906+0000 7f7015ffb700 1 -- 172.21.15.90:0/3775089771 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f7010001bb0 con 0x7f701810f340 2024-04-14T15:17:32.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:32.906+0000 7f7015ffb700 1 -- 172.21.15.90:0/3775089771 <== mon.2 v1:172.21.15.90:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1371465608 0 0) 0x7f70181aaf50 con 0x7f7018103a20 2024-04-14T15:17:32.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:32.906+0000 7f7015ffb700 1 -- 172.21.15.90:0/3775089771 >> v1:172.21.15.92:6789/0 conn(0x7f701810f340 legacy=0x7f70181a83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:32.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:32.906+0000 7f7015ffb700 1 -- 172.21.15.90:0/3775089771 >> v1:172.21.15.90:6789/0 conn(0x7f701810b830 legacy=0x7f70181a4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:32.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:32.906+0000 7f7015ffb700 1 -- 172.21.15.90:0/3775089771 --> v1:172.21.15.90:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70181ad3d0 con 0x7f7018103a20 2024-04-14T15:17:32.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:32.906+0000 7f701eddd700 1 -- 172.21.15.90:0/3775089771 --> v1:172.21.15.90:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f70181a9f60 con 0x7f7018103a20 2024-04-14T15:17:32.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:32.906+0000 7f701eddd700 10 client.47138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:17:32.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:32.906+0000 7f701eddd700 1 -- 172.21.15.90:0/3775089771 --> v1:172.21.15.90:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f70181aa420 con 0x7f7018103a20 2024-04-14T15:17:32.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:32.906+0000 7f701eddd700 1 librados: init done 2024-04-14T15:17:32.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:32.906+0000 7f7015ffb700 1 -- 172.21.15.90:0/3775089771 <== mon.2 v1:172.21.15.90:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f700c003760 con 0x7f7018103a20 2024-04-14T15:17:32.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:32.906+0000 7f7015ffb700 1 -- 172.21.15.90:0/3775089771 <== mon.2 v1:172.21.15.90:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f700c004ea0 con 0x7f7018103a20 2024-04-14T15:17:32.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:32.907+0000 7f701eddd700 1 -- 172.21.15.90:0/3775089771 --> v1:172.21.15.90:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ffc005120 con 0x7f7018103a20 2024-04-14T15:17:32.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:32.908+0000 7f7015ffb700 1 -- 172.21.15.90:0/3775089771 <== mon.2 v1:172.21.15.90:6790/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f700c003ab0 con 0x7f7018103a20 2024-04-14T15:17:32.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:32.909+0000 7f7015ffb700 10 client.47138.objecter ms_handle_connect 0x7f7008071400 2024-04-14T15:17:32.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:32.912+0000 7f7015ffb700 1 -- 172.21.15.90:0/3775089771 <== mon.2 v1:172.21.15.90:6790/0 8 ==== osd_map(384..384 src has 51..384) ==== 8566+0+0 (unknown 3141613215 0 0) 0x7f700c088810 con 0x7f7018103a20 2024-04-14T15:17:32.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:32.912+0000 7f7015ffb700 10 client.47138.objecter ms_dispatch 0x7f7018001550 osd_map(384..384 src has 51..384) 2024-04-14T15:17:32.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:32.912+0000 7f7015ffb700 3 client.47138.objecter handle_osd_map got epochs [384,384] > 0 2024-04-14T15:17:32.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:32.912+0000 7f7015ffb700 3 client.47138.objecter handle_osd_map decoding full epoch 384 2024-04-14T15:17:32.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:32.912+0000 7f7015ffb700 20 client.47138.objecter dump_active .. 0 homeless 2024-04-14T15:17:32.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:32.917+0000 7f7015ffb700 1 -- 172.21.15.90:0/3775089771 <== mon.2 v1:172.21.15.90:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f700c004e80 con 0x7f7018103a20 2024-04-14T15:17:32.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:32.966 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:32.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:17:33.067 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:17:33.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:33.097 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:33.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:17:33.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:33.147+0000 7f701eddd700 1 -- 172.21.15.90:0/3775089771 --> v1:172.21.15.90:6790/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7f6ffc005410 con 0x7f7018103a20 2024-04-14T15:17:33.148 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:32 smithi090 ceph-mon[41670]: pgmap v521: 238 pgs: 238 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:17:33.148 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:32 smithi090 ceph-mon[41670]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:17:33.148 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:32 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-49"}]': finished 2024-04-14T15:17:33.148 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:32 smithi090 ceph-mon[41670]: osdmap e384: 8 total, 8 up, 8 in 2024-04-14T15:17:33.149 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:32 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:17:33.149 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:32 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:17:33.149 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:32 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-49"}]: dispatch 2024-04-14T15:17:33.149 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:32 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-49"}]: dispatch 2024-04-14T15:17:33.149 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:32 smithi090 ceph-mon[30399]: pgmap v521: 238 pgs: 238 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:17:33.149 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:32 smithi090 ceph-mon[30399]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:17:33.149 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:32 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-49"}]': finished 2024-04-14T15:17:33.149 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:32 smithi090 ceph-mon[30399]: osdmap e384: 8 total, 8 up, 8 in 2024-04-14T15:17:33.149 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:32 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:17:33.149 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:32 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:17:33.149 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:32 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-49"}]: dispatch 2024-04-14T15:17:33.150 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:32 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-49"}]: dispatch 2024-04-14T15:17:33.198 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:17:33.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:33.223 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:33.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:17:33.323 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:17:33.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:33.347 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:33.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:17:33.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:32 smithi092 ceph-mon[33966]: pgmap v521: 238 pgs: 238 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:17:33.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:32 smithi092 ceph-mon[33966]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:17:33.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:32 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-49"}]': finished 2024-04-14T15:17:33.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:32 smithi092 ceph-mon[33966]: osdmap e384: 8 total, 8 up, 8 in 2024-04-14T15:17:33.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:32 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:17:33.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:32 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:17:33.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:32 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-49"}]: dispatch 2024-04-14T15:17:33.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:32 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-49"}]: dispatch 2024-04-14T15:17:33.448 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:17:33.476 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:33.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:33.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:17:33.577 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:17:33.606 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:33.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:33.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:17:33.708 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:17:33.735 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:33.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:33.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:17:33.837 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:17:33.860 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:33.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:33.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:17:33.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:33.934+0000 7f7015ffb700 1 -- 172.21.15.90:0/3775089771 <== mon.2 v1:172.21.15.90:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v386) ==== 91+0+0 (unknown 3700240503 0 0) 0x7f700c0575e0 con 0x7f7018103a20 2024-04-14T15:17:33.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:33.937+0000 7f701eddd700 10 librados: watch_flush enter 2024-04-14T15:17:33.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:33.938+0000 7f701eddd700 10 librados: watch_flush exit 2024-04-14T15:17:33.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:33.938+0000 7f701eddd700 20 client.47138.objecter shutdown clearing up homeless session... 2024-04-14T15:17:33.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:33.938+0000 7f701eddd700 10 client.47138.objecter successfully canceled tick 2024-04-14T15:17:33.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:33.939+0000 7f701eddd700 1 -- 172.21.15.90:0/3775089771 >> v1:172.21.15.90:6800/420935427 conn(0x7f7008071400 legacy=0x7f70080738c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:33.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:33.939+0000 7f701eddd700 1 -- 172.21.15.90:0/3775089771 >> v1:172.21.15.90:6790/0 conn(0x7f7018103a20 legacy=0x7f7018103000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:33.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:33.940+0000 7f701eddd700 1 -- 172.21.15.90:0/3775089771 shutdown_connections 2024-04-14T15:17:33.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:33.940+0000 7f701eddd700 1 -- 172.21.15.90:0/3775089771 >> 172.21.15.90:0/3775089771 conn(0x7f70180fd290 msgr2=0x7f7018111770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:17:33.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:33.940+0000 7f701eddd700 1 -- 172.21.15.90:0/3775089771 shutdown_connections 2024-04-14T15:17:33.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:33.941+0000 7f701eddd700 1 -- 172.21.15.90:0/3775089771 wait complete. 2024-04-14T15:17:33.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:33.941+0000 7f701eddd700 1 librados: shutdown 2024-04-14T15:17:33.942 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is unset 2024-04-14T15:17:33.963 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:17:33.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:33.989 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:33.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:17:34.090 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:17:34.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:34.118 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:34.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:17:34.145 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:17:33 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:17:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:17:34.146 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:33 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/468264671' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi090-93363-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi090-93363-65"}]': finished 2024-04-14T15:17:34.146 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:33 smithi090 ceph-mon[30399]: osdmap e385: 8 total, 8 up, 8 in 2024-04-14T15:17:34.146 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:33 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3775089771' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:17:34.146 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:33 smithi090 ceph-mon[30399]: from='client.47138 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:17:34.146 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:33 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/468264671' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi090-93363-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi090-93363-65"}]': finished 2024-04-14T15:17:34.146 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:33 smithi090 ceph-mon[41670]: osdmap e385: 8 total, 8 up, 8 in 2024-04-14T15:17:34.147 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:33 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3775089771' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:17:34.147 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:33 smithi090 ceph-mon[41670]: from='client.47138 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:17:34.219 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:17:34.246 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:34.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:34.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:17:34.348 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:17:34.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:33 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/468264671' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi090-93363-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi090-93363-65"}]': finished 2024-04-14T15:17:34.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:33 smithi092 ceph-mon[33966]: osdmap e385: 8 total, 8 up, 8 in 2024-04-14T15:17:34.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:33 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3775089771' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:17:34.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:33 smithi092 ceph-mon[33966]: from='client.47138 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:17:34.374 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:34.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:34.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:17:34.475 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:17:34.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:34.501 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:34.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:17:34.602 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:17:34.628 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:34.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:34.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:17:34.730 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:17:34.756 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:34.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:34.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:17:34.858 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:17:34.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:34.883 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:34.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:17:34.984 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:17:34.988 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:34 smithi090 ceph-mon[30399]: pgmap v524: 214 pgs: 8 unknown, 206 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:17:34.988 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:34 smithi090 ceph-mon[30399]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:17:34.988 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:34 smithi090 ceph-mon[30399]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:17:34.988 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:34 smithi090 ceph-mon[30399]: from='client.47138 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-14T15:17:34.989 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:34 smithi090 ceph-mon[30399]: osdmap e386: 8 total, 8 up, 8 in 2024-04-14T15:17:34.989 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:34 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:17:34.989 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:34 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:17:34.989 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:34 smithi090 ceph-mon[30399]: 6.18 scrub starts 2024-04-14T15:17:34.989 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:34 smithi090 ceph-mon[30399]: 6.18 scrub ok 2024-04-14T15:17:34.989 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:34 smithi090 ceph-mon[30399]: 110.16 scrub starts 2024-04-14T15:17:34.989 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:34 smithi090 ceph-mon[30399]: 110.16 scrub ok 2024-04-14T15:17:34.989 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:34 smithi090 ceph-mon[30399]: 110.10 scrub starts 2024-04-14T15:17:34.989 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:34 smithi090 ceph-mon[30399]: 110.10 scrub ok 2024-04-14T15:17:34.989 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:34 smithi090 ceph-mon[41670]: pgmap v524: 214 pgs: 8 unknown, 206 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:17:34.989 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:34 smithi090 ceph-mon[41670]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:17:34.989 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:34 smithi090 ceph-mon[41670]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:17:34.989 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:34 smithi090 ceph-mon[41670]: from='client.47138 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-14T15:17:34.990 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:34 smithi090 ceph-mon[41670]: osdmap e386: 8 total, 8 up, 8 in 2024-04-14T15:17:34.990 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:34 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:17:34.990 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:34 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:17:34.990 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:34 smithi090 ceph-mon[41670]: 6.18 scrub starts 2024-04-14T15:17:34.990 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:34 smithi090 ceph-mon[41670]: 6.18 scrub ok 2024-04-14T15:17:34.990 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:34 smithi090 ceph-mon[41670]: 110.16 scrub starts 2024-04-14T15:17:34.990 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:34 smithi090 ceph-mon[41670]: 110.16 scrub ok 2024-04-14T15:17:34.990 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:34 smithi090 ceph-mon[41670]: 110.10 scrub starts 2024-04-14T15:17:34.990 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:34 smithi090 ceph-mon[41670]: 110.10 scrub ok 2024-04-14T15:17:35.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:35.010 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:35.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:17:35.111 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:17:35.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:35.136 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:35.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:17:35.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.226+0000 7f02a744e700 1 -- 172.21.15.90:0/25916669 >> v1:172.21.15.90:6790/0 conn(0x7f02a0107780 legacy=0x7f02a0107b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:35.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.226+0000 7f02a744e700 1 -- 172.21.15.90:0/25916669 shutdown_connections 2024-04-14T15:17:35.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.226+0000 7f02a744e700 1 -- 172.21.15.90:0/25916669 >> 172.21.15.90:0/25916669 conn(0x7f02a00fd2b0 msgr2=0x7f02a00ff6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:17:35.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.226+0000 7f02a744e700 1 -- 172.21.15.90:0/25916669 shutdown_connections 2024-04-14T15:17:35.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.226+0000 7f02a744e700 1 -- 172.21.15.90:0/25916669 wait complete. 2024-04-14T15:17:35.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.226+0000 7f02a744e700 1 librados: starting msgr at 2024-04-14T15:17:35.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.226+0000 7f02a744e700 1 librados: starting objecter 2024-04-14T15:17:35.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.226+0000 7f02a744e700 1 Processor -- start 2024-04-14T15:17:35.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.226+0000 7f02a744e700 1 -- start start 2024-04-14T15:17:35.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.226+0000 7f02a744e700 1 librados: setting wanted keys 2024-04-14T15:17:35.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.226+0000 7f02a744e700 1 librados: calling monclient init 2024-04-14T15:17:35.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.226+0000 7f02a744e700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f02a01a9c00 con 0x7f02a010e200 2024-04-14T15:17:35.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.226+0000 7f02a744e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f02a01aae20 con 0x7f02a0107780 2024-04-14T15:17:35.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.227+0000 7f02a744e700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f02a01ac060 con 0x7f02a010a630 2024-04-14T15:17:35.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.227+0000 7f02a49e9700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f02a010a630 0x7f02a01a4ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:48448/0 (socket says 172.21.15.90:48448) 2024-04-14T15:17:35.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.227+0000 7f02a49e9700 1 -- 172.21.15.90:0/2688521590 learned_addr learned my addr 172.21.15.90:0/2688521590 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:17:35.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.227+0000 7f02967fc700 10 client.?.objecter ms_handle_connect 0x7f02a0107780 2024-04-14T15:17:35.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.227+0000 7f02967fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:17:35.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.227+0000 7f02967fc700 10 client.?.objecter ms_handle_connect 0x7f02a010e200 2024-04-14T15:17:35.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.227+0000 7f02967fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:17:35.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.227+0000 7f02967fc700 10 client.?.objecter ms_handle_connect 0x7f02a010a630 2024-04-14T15:17:35.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.227+0000 7f02967fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:17:35.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.227+0000 7f02967fc700 1 -- 172.21.15.90:0/2688521590 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2971598921 0 0) 0x7f02a01a9c00 con 0x7f02a010e200 2024-04-14T15:17:35.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.227+0000 7f02967fc700 1 -- 172.21.15.90:0/2688521590 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f028c003aa0 con 0x7f02a010e200 2024-04-14T15:17:35.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.227+0000 7f02967fc700 1 -- 172.21.15.90:0/2688521590 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2774335621 0 0) 0x7f02a01aae20 con 0x7f02a0107780 2024-04-14T15:17:35.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.227+0000 7f02967fc700 1 -- 172.21.15.90:0/2688521590 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f02a01a9c00 con 0x7f02a0107780 2024-04-14T15:17:35.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.227+0000 7f02967fc700 1 -- 172.21.15.90:0/2688521590 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1783575866 0 0) 0x7f02a01ac060 con 0x7f02a010a630 2024-04-14T15:17:35.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.227+0000 7f02967fc700 1 -- 172.21.15.90:0/2688521590 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f02a01aae20 con 0x7f02a010a630 2024-04-14T15:17:35.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.227+0000 7f02967fc700 1 -- 172.21.15.90:0/2688521590 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1042425872 0 0) 0x7f028c003aa0 con 0x7f02a010e200 2024-04-14T15:17:35.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.227+0000 7f02967fc700 1 -- 172.21.15.90:0/2688521590 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f02a01ac060 con 0x7f02a010e200 2024-04-14T15:17:35.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.227+0000 7f02967fc700 1 -- 172.21.15.90:0/2688521590 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f0298001af0 con 0x7f02a010e200 2024-04-14T15:17:35.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.227+0000 7f02967fc700 1 -- 172.21.15.90:0/2688521590 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2635434419 0 0) 0x7f02a01a9c00 con 0x7f02a0107780 2024-04-14T15:17:35.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.227+0000 7f02967fc700 1 -- 172.21.15.90:0/2688521590 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f028c003aa0 con 0x7f02a0107780 2024-04-14T15:17:35.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.227+0000 7f02967fc700 1 -- 172.21.15.90:0/2688521590 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1658558025 0 0) 0x7f02a01aae20 con 0x7f02a010a630 2024-04-14T15:17:35.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.227+0000 7f02967fc700 1 -- 172.21.15.90:0/2688521590 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f02a01a9c00 con 0x7f02a010a630 2024-04-14T15:17:35.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.227+0000 7f02967fc700 1 -- 172.21.15.90:0/2688521590 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f029c001550 con 0x7f02a0107780 2024-04-14T15:17:35.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.227+0000 7f02967fc700 1 -- 172.21.15.90:0/2688521590 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f0290000fe0 con 0x7f02a010a630 2024-04-14T15:17:35.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.227+0000 7f02967fc700 1 -- 172.21.15.90:0/2688521590 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3466610363 0 0) 0x7f028c003aa0 con 0x7f02a0107780 2024-04-14T15:17:35.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.227+0000 7f02967fc700 1 -- 172.21.15.90:0/2688521590 >> v1:172.21.15.90:6790/0 conn(0x7f02a010a630 legacy=0x7f02a01a4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:35.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.227+0000 7f02967fc700 1 -- 172.21.15.90:0/2688521590 >> v1:172.21.15.90:6789/0 conn(0x7f02a010e200 legacy=0x7f02a01a82c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:35.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.227+0000 7f02967fc700 1 -- 172.21.15.90:0/2688521590 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02a01ad2a0 con 0x7f02a0107780 2024-04-14T15:17:35.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.228+0000 7f02a744e700 1 -- 172.21.15.90:0/2688521590 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f02a01a9e30 con 0x7f02a0107780 2024-04-14T15:17:35.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.228+0000 7f02a744e700 10 client.49030.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:17:35.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.228+0000 7f02a744e700 1 -- 172.21.15.90:0/2688521590 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f02a01aa2f0 con 0x7f02a0107780 2024-04-14T15:17:35.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.228+0000 7f02a744e700 1 librados: init done 2024-04-14T15:17:35.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.228+0000 7f02967fc700 1 -- 172.21.15.90:0/2688521590 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f029c001c50 con 0x7f02a0107780 2024-04-14T15:17:35.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.228+0000 7f02967fc700 1 -- 172.21.15.90:0/2688521590 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f029c004cd0 con 0x7f02a0107780 2024-04-14T15:17:35.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.228+0000 7f02a744e700 1 -- 172.21.15.90:0/2688521590 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0284005120 con 0x7f02a0107780 2024-04-14T15:17:35.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.229+0000 7f02967fc700 1 -- 172.21.15.90:0/2688521590 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f029c0043a0 con 0x7f02a0107780 2024-04-14T15:17:35.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.230+0000 7f02967fc700 1 -- 172.21.15.90:0/2688521590 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(387..387 src has 51..387) ==== 8566+0+0 (unknown 381362175 0 0) 0x7f029c0899e0 con 0x7f02a0107780 2024-04-14T15:17:35.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.230+0000 7f02967fc700 10 client.49030.objecter ms_dispatch 0x7f02a0001550 osd_map(387..387 src has 51..387) 2024-04-14T15:17:35.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.230+0000 7f02967fc700 3 client.49030.objecter handle_osd_map got epochs [387,387] > 0 2024-04-14T15:17:35.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.230+0000 7f02967fc700 3 client.49030.objecter handle_osd_map decoding full epoch 387 2024-04-14T15:17:35.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.230+0000 7f02967fc700 20 client.49030.objecter dump_active .. 0 homeless 2024-04-14T15:17:35.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.230+0000 7f02967fc700 10 client.49030.objecter ms_handle_connect 0x7f028c071430 2024-04-14T15:17:35.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.234+0000 7f02967fc700 1 -- 172.21.15.90:0/2688521590 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f029c056260 con 0x7f02a0107780 2024-04-14T15:17:35.237 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:17:35.263 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:35.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:35.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:17:35.299 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-14T15:17:35.365 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:17:35.372 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:17:35 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:17:35.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:34 smithi092 ceph-mon[33966]: pgmap v524: 214 pgs: 8 unknown, 206 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:17:35.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:34 smithi092 ceph-mon[33966]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:17:35.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:34 smithi092 ceph-mon[33966]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:17:35.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:34 smithi092 ceph-mon[33966]: from='client.47138 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-14T15:17:35.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:34 smithi092 ceph-mon[33966]: osdmap e386: 8 total, 8 up, 8 in 2024-04-14T15:17:35.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:34 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:17:35.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:34 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:17:35.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:34 smithi092 ceph-mon[33966]: 6.18 scrub starts 2024-04-14T15:17:35.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:34 smithi092 ceph-mon[33966]: 6.18 scrub ok 2024-04-14T15:17:35.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:34 smithi092 ceph-mon[33966]: 110.16 scrub starts 2024-04-14T15:17:35.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:34 smithi092 ceph-mon[33966]: 110.16 scrub ok 2024-04-14T15:17:35.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:34 smithi092 ceph-mon[33966]: 110.10 scrub starts 2024-04-14T15:17:35.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:34 smithi092 ceph-mon[33966]: 110.10 scrub ok 2024-04-14T15:17:35.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:35.391 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:35.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:17:35.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.455+0000 7f02a744e700 1 -- 172.21.15.90:0/2688521590 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd reweight", "id": 4, "weight": 1.0} v 0) -- 0x7f0284005410 con 0x7f02a0107780 2024-04-14T15:17:35.492 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:17:35.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:35.520 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:35.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:17:35.621 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:17:35.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:35.648 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:35.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:17:35.749 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:17:35.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:35.775 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:35.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:17:35.840 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:17:35.842 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-14T15:17:35.842 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:17:35.876 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:17:35.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.886+0000 7f02967fc700 1 -- 172.21.15.90:0/2688521590 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 4, "weight": 1.0}]=0 reweighted osd.4 to 1 (10000) v388) ==== 113+0+0 (unknown 4093691612 0 0) 0x7f029c059f10 con 0x7f02a0107780 2024-04-14T15:17:35.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.890+0000 7f02a744e700 10 librados: watch_flush enter 2024-04-14T15:17:35.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.890+0000 7f02a744e700 10 librados: watch_flush exit 2024-04-14T15:17:35.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.890+0000 7f02a744e700 20 client.49030.objecter shutdown clearing up homeless session... 2024-04-14T15:17:35.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.890+0000 7f02a744e700 10 client.49030.objecter successfully canceled tick 2024-04-14T15:17:35.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.890+0000 7f02a744e700 1 -- 172.21.15.90:0/2688521590 >> v1:172.21.15.90:6800/420935427 conn(0x7f028c071430 legacy=0x7f028c0738f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:35.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.890+0000 7f02a744e700 1 -- 172.21.15.90:0/2688521590 >> v1:172.21.15.92:6789/0 conn(0x7f02a0107780 legacy=0x7f02a010da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:35.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.891+0000 7f02a744e700 1 -- 172.21.15.90:0/2688521590 shutdown_connections 2024-04-14T15:17:35.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.891+0000 7f02a744e700 1 -- 172.21.15.90:0/2688521590 >> 172.21.15.90:0/2688521590 conn(0x7f02a00fd2b0 msgr2=0x7f02a0110ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:17:35.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.891+0000 7f02a744e700 1 -- 172.21.15.90:0/2688521590 shutdown_connections 2024-04-14T15:17:35.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.891+0000 7f02a744e700 1 -- 172.21.15.90:0/2688521590 wait complete. 2024-04-14T15:17:35.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:35.891+0000 7f02a744e700 1 librados: shutdown 2024-04-14T15:17:35.892 INFO:teuthology.orchestra.run.smithi090.stderr:reweighted osd.4 to 1 (10000) 2024-04-14T15:17:35.903 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:35.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:35.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:17:36.005 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:17:36.030 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:36.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:36.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:17:36.132 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:17:36.156 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:36.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:36.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:17:36.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:35 smithi090 ceph-mon[41670]: 110.e scrub starts 2024-04-14T15:17:36.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:35 smithi090 ceph-mon[41670]: 110.e scrub ok 2024-04-14T15:17:36.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:35 smithi090 ceph-mon[41670]: 110.7 scrub starts 2024-04-14T15:17:36.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:35 smithi090 ceph-mon[41670]: 110.7 scrub ok 2024-04-14T15:17:36.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:35 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:17:36.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:35 smithi090 ceph-mon[41670]: osdmap e387: 8 total, 8 up, 8 in 2024-04-14T15:17:36.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:35 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/468264671' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi090-93363-65"}]: dispatch 2024-04-14T15:17:36.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:35 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:17:36.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:35 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:17:36.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:35 smithi090 ceph-mon[41670]: 2.15 scrub starts 2024-04-14T15:17:36.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:35 smithi090 ceph-mon[41670]: 2.15 scrub ok 2024-04-14T15:17:36.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:35 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2688521590' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-14T15:17:36.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:35 smithi090 ceph-mon[41670]: from='client.49030 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-14T15:17:36.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:35 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/468264671' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi090-93363-65"}]': finished 2024-04-14T15:17:36.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:35 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-51", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:17:36.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:35 smithi090 ceph-mon[41670]: from='client.49030 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-14T15:17:36.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:35 smithi090 ceph-mon[41670]: osdmap e388: 8 total, 8 up, 8 in 2024-04-14T15:17:36.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:35 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/468264671' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi090-93363-65"}]: dispatch 2024-04-14T15:17:36.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:35 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-51"}]: dispatch 2024-04-14T15:17:36.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:35 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-51"}]: dispatch 2024-04-14T15:17:36.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:35 smithi090 ceph-mon[30399]: 110.e scrub starts 2024-04-14T15:17:36.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:35 smithi090 ceph-mon[30399]: 110.e scrub ok 2024-04-14T15:17:36.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:35 smithi090 ceph-mon[30399]: 110.7 scrub starts 2024-04-14T15:17:36.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:35 smithi090 ceph-mon[30399]: 110.7 scrub ok 2024-04-14T15:17:36.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:35 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:17:36.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:35 smithi090 ceph-mon[30399]: osdmap e387: 8 total, 8 up, 8 in 2024-04-14T15:17:36.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:35 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/468264671' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi090-93363-65"}]: dispatch 2024-04-14T15:17:36.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:35 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:17:36.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:35 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:17:36.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:35 smithi090 ceph-mon[30399]: 2.15 scrub starts 2024-04-14T15:17:36.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:35 smithi090 ceph-mon[30399]: 2.15 scrub ok 2024-04-14T15:17:36.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:35 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2688521590' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-14T15:17:36.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:35 smithi090 ceph-mon[30399]: from='client.49030 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-14T15:17:36.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:35 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/468264671' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi090-93363-65"}]': finished 2024-04-14T15:17:36.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:35 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-51", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:17:36.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:35 smithi090 ceph-mon[30399]: from='client.49030 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-14T15:17:36.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:35 smithi090 ceph-mon[30399]: osdmap e388: 8 total, 8 up, 8 in 2024-04-14T15:17:36.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:35 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/468264671' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi090-93363-65"}]: dispatch 2024-04-14T15:17:36.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:35 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-51"}]: dispatch 2024-04-14T15:17:36.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:35 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-51"}]: dispatch 2024-04-14T15:17:36.258 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:17:36.284 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:36.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:36.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:17:36.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:35 smithi092 ceph-mon[33966]: 110.e scrub starts 2024-04-14T15:17:36.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:35 smithi092 ceph-mon[33966]: 110.e scrub ok 2024-04-14T15:17:36.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:35 smithi092 ceph-mon[33966]: 110.7 scrub starts 2024-04-14T15:17:36.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:35 smithi092 ceph-mon[33966]: 110.7 scrub ok 2024-04-14T15:17:36.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:35 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:17:36.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:35 smithi092 ceph-mon[33966]: osdmap e387: 8 total, 8 up, 8 in 2024-04-14T15:17:36.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:35 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/468264671' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi090-93363-65"}]: dispatch 2024-04-14T15:17:36.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:35 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:17:36.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:35 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:17:36.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:35 smithi092 ceph-mon[33966]: 2.15 scrub starts 2024-04-14T15:17:36.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:35 smithi092 ceph-mon[33966]: 2.15 scrub ok 2024-04-14T15:17:36.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:35 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2688521590' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-14T15:17:36.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:35 smithi092 ceph-mon[33966]: from='client.49030 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-14T15:17:36.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:35 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/468264671' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi090-93363-65"}]': finished 2024-04-14T15:17:36.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:35 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-51", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:17:36.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:35 smithi092 ceph-mon[33966]: from='client.49030 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-14T15:17:36.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:35 smithi092 ceph-mon[33966]: osdmap e388: 8 total, 8 up, 8 in 2024-04-14T15:17:36.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:35 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/468264671' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi090-93363-65"}]: dispatch 2024-04-14T15:17:36.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:35 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-51"}]: dispatch 2024-04-14T15:17:36.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:35 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-51"}]: dispatch 2024-04-14T15:17:36.386 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:17:36.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:36.410 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:36.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:17:36.511 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:17:36.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:36.535 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:36.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:17:36.636 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:17:36.660 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:36.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:36.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:17:36.753 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-14T15:17:36.762 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:17:36.789 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:36.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:36.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:17:36.806 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-14T15:17:36.891 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:17:36.919 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:36.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:36.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:17:37.021 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:17:37.044 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:36 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:17:37.044 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:36 smithi090 ceph-mon[41670]: pgmap v527: 238 pgs: 32 creating+peering, 206 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:17:37.044 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:36 smithi090 ceph-mon[41670]: 2.3 scrub starts 2024-04-14T15:17:37.044 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:36 smithi090 ceph-mon[41670]: 2.3 scrub ok 2024-04-14T15:17:37.044 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:36 smithi090 ceph-mon[41670]: 110.18 scrub starts 2024-04-14T15:17:37.044 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:36 smithi090 ceph-mon[41670]: 110.18 scrub ok 2024-04-14T15:17:37.044 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:36 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/468264671' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi090-93363-65"}]': finished 2024-04-14T15:17:37.044 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:36 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-51"}]': finished 2024-04-14T15:17:37.045 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:36 smithi090 ceph-mon[41670]: osdmap e389: 8 total, 8 up, 8 in 2024-04-14T15:17:37.045 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:36 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-51", "mode": "writeback"}]: dispatch 2024-04-14T15:17:37.045 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:36 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-51", "mode": "writeback"}]: dispatch 2024-04-14T15:17:37.045 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:36 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1014922700' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi090-93363-66"}]: dispatch 2024-04-14T15:17:37.045 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:36 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1014922700' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi090-93363-66"}]: dispatch 2024-04-14T15:17:37.045 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:36 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1014922700' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi090-93363-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:17:37.045 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:36 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:17:37.045 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:36 smithi090 ceph-mon[30399]: pgmap v527: 238 pgs: 32 creating+peering, 206 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:17:37.045 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:36 smithi090 ceph-mon[30399]: 2.3 scrub starts 2024-04-14T15:17:37.045 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:36 smithi090 ceph-mon[30399]: 2.3 scrub ok 2024-04-14T15:17:37.046 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:36 smithi090 ceph-mon[30399]: 110.18 scrub starts 2024-04-14T15:17:37.046 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:36 smithi090 ceph-mon[30399]: 110.18 scrub ok 2024-04-14T15:17:37.046 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:36 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/468264671' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi090-93363-65"}]': finished 2024-04-14T15:17:37.046 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:36 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-51"}]': finished 2024-04-14T15:17:37.046 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:36 smithi090 ceph-mon[30399]: osdmap e389: 8 total, 8 up, 8 in 2024-04-14T15:17:37.046 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:36 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-51", "mode": "writeback"}]: dispatch 2024-04-14T15:17:37.046 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:36 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-51", "mode": "writeback"}]: dispatch 2024-04-14T15:17:37.046 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:36 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1014922700' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi090-93363-66"}]: dispatch 2024-04-14T15:17:37.046 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:36 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1014922700' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi090-93363-66"}]: dispatch 2024-04-14T15:17:37.046 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:36 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1014922700' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi090-93363-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:17:37.048 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:37.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:37.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:17:37.149 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:17:37.175 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:37.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:37.176 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:17:37.277 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:17:37.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:37.303 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:37.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:17:37.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:36 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:17:37.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:36 smithi092 ceph-mon[33966]: pgmap v527: 238 pgs: 32 creating+peering, 206 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:17:37.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:36 smithi092 ceph-mon[33966]: 2.3 scrub starts 2024-04-14T15:17:37.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:36 smithi092 ceph-mon[33966]: 2.3 scrub ok 2024-04-14T15:17:37.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:36 smithi092 ceph-mon[33966]: 110.18 scrub starts 2024-04-14T15:17:37.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:36 smithi092 ceph-mon[33966]: 110.18 scrub ok 2024-04-14T15:17:37.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:36 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/468264671' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi090-93363-65"}]': finished 2024-04-14T15:17:37.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:36 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-51"}]': finished 2024-04-14T15:17:37.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:36 smithi092 ceph-mon[33966]: osdmap e389: 8 total, 8 up, 8 in 2024-04-14T15:17:37.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:36 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-51", "mode": "writeback"}]: dispatch 2024-04-14T15:17:37.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:36 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-51", "mode": "writeback"}]: dispatch 2024-04-14T15:17:37.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:36 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1014922700' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi090-93363-66"}]: dispatch 2024-04-14T15:17:37.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:36 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1014922700' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi090-93363-66"}]: dispatch 2024-04-14T15:17:37.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:36 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1014922700' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi090-93363-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:17:37.404 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:17:37.428 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:37.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:37.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:17:37.530 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:17:37.555 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:37.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:37.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:17:37.569 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd reweight 5 1 2024-04-14T15:17:37.657 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:17:37.681 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:37.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:37.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:17:37.783 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:17:37.807 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:37.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:37.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:17:37.910 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:17:37.935 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:37.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:37.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:17:38.037 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:17:38.055 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:17:38.055 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:17:38.063 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:38.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:38.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:17:38.165 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:17:38.180 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:37 smithi090 ceph-mon[41670]: 110.1d scrub starts 2024-04-14T15:17:38.181 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:37 smithi090 ceph-mon[41670]: 110.1d scrub ok 2024-04-14T15:17:38.181 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:37 smithi090 ceph-mon[41670]: 11.18 scrub starts 2024-04-14T15:17:38.181 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:37 smithi090 ceph-mon[41670]: 11.18 scrub ok 2024-04-14T15:17:38.181 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:37 smithi090 ceph-mon[41670]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:17:38.181 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:37 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-51", "mode": "writeback"}]': finished 2024-04-14T15:17:38.181 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:37 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1014922700' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi090-93363-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:17:38.181 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:37 smithi090 ceph-mon[41670]: osdmap e390: 8 total, 8 up, 8 in 2024-04-14T15:17:38.181 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:37 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1014922700' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi090-93363-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi090-93363-66"}]: dispatch 2024-04-14T15:17:38.181 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:37 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-51","var": "hit_set_count","val": "2"}]: dispatch 2024-04-14T15:17:38.181 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:37 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-51","var": "hit_set_count","val": "2"}]: dispatch 2024-04-14T15:17:38.182 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:37 smithi090 ceph-mon[30399]: 110.1d scrub starts 2024-04-14T15:17:38.182 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:37 smithi090 ceph-mon[30399]: 110.1d scrub ok 2024-04-14T15:17:38.182 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:37 smithi090 ceph-mon[30399]: 11.18 scrub starts 2024-04-14T15:17:38.182 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:37 smithi090 ceph-mon[30399]: 11.18 scrub ok 2024-04-14T15:17:38.182 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:37 smithi090 ceph-mon[30399]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:17:38.182 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:37 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-51", "mode": "writeback"}]': finished 2024-04-14T15:17:38.182 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:37 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1014922700' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi090-93363-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:17:38.182 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:37 smithi090 ceph-mon[30399]: osdmap e390: 8 total, 8 up, 8 in 2024-04-14T15:17:38.182 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:37 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1014922700' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi090-93363-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi090-93363-66"}]: dispatch 2024-04-14T15:17:38.182 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:37 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-51","var": "hit_set_count","val": "2"}]: dispatch 2024-04-14T15:17:38.183 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:37 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-51","var": "hit_set_count","val": "2"}]: dispatch 2024-04-14T15:17:38.189 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:38.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:38.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:17:38.291 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:17:38.318 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:38.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:38.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:17:38.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:37 smithi092 ceph-mon[33966]: 110.1d scrub starts 2024-04-14T15:17:38.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:37 smithi092 ceph-mon[33966]: 110.1d scrub ok 2024-04-14T15:17:38.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:37 smithi092 ceph-mon[33966]: 11.18 scrub starts 2024-04-14T15:17:38.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:37 smithi092 ceph-mon[33966]: 11.18 scrub ok 2024-04-14T15:17:38.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:37 smithi092 ceph-mon[33966]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:17:38.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:37 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-51", "mode": "writeback"}]': finished 2024-04-14T15:17:38.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:37 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1014922700' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi090-93363-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:17:38.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:37 smithi092 ceph-mon[33966]: osdmap e390: 8 total, 8 up, 8 in 2024-04-14T15:17:38.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:37 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1014922700' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi090-93363-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi090-93363-66"}]: dispatch 2024-04-14T15:17:38.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:37 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-51","var": "hit_set_count","val": "2"}]: dispatch 2024-04-14T15:17:38.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:37 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-51","var": "hit_set_count","val": "2"}]: dispatch 2024-04-14T15:17:38.420 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:17:38.444 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:38.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:38.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:17:38.526 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:17:38.547 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:17:38.571 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:38.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:38.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:17:38.673 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:17:38.701 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:38.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:38.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:17:38.803 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:17:38.831 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:38.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:38.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:17:38.932 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:17:38.960 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:38.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:38.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:17:39.062 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:17:39.085 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:39.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:39.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:17:39.187 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:17:39.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:38 smithi090 ceph-mon[41670]: pgmap v530: 238 pgs: 32 creating+peering, 206 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:17:39.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:38 smithi090 ceph-mon[41670]: 4.15 scrub starts 2024-04-14T15:17:39.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:38 smithi090 ceph-mon[41670]: 4.15 scrub ok 2024-04-14T15:17:39.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:38 smithi090 ceph-mon[41670]: 11.15 scrub starts 2024-04-14T15:17:39.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:38 smithi090 ceph-mon[41670]: 11.15 scrub ok 2024-04-14T15:17:39.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:38 smithi090 ceph-mon[41670]: 6.9 scrub starts 2024-04-14T15:17:39.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:38 smithi090 ceph-mon[41670]: 6.9 scrub ok 2024-04-14T15:17:39.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:38 smithi090 ceph-mon[41670]: 110.2 scrub starts 2024-04-14T15:17:39.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:38 smithi090 ceph-mon[41670]: 110.2 scrub ok 2024-04-14T15:17:39.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:38 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:17:39.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:38 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-51","var": "hit_set_count","val": "2"}]': finished 2024-04-14T15:17:39.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:38 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-51","var": "hit_set_period","val": "600"}]: dispatch 2024-04-14T15:17:39.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:38 smithi090 ceph-mon[41670]: osdmap e391: 8 total, 8 up, 8 in 2024-04-14T15:17:39.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:38 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-51","var": "hit_set_period","val": "600"}]: dispatch 2024-04-14T15:17:39.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:38 smithi090 ceph-mon[30399]: pgmap v530: 238 pgs: 32 creating+peering, 206 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:17:39.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:38 smithi090 ceph-mon[30399]: 4.15 scrub starts 2024-04-14T15:17:39.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:38 smithi090 ceph-mon[30399]: 4.15 scrub ok 2024-04-14T15:17:39.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:38 smithi090 ceph-mon[30399]: 11.15 scrub starts 2024-04-14T15:17:39.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:38 smithi090 ceph-mon[30399]: 11.15 scrub ok 2024-04-14T15:17:39.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:38 smithi090 ceph-mon[30399]: 6.9 scrub starts 2024-04-14T15:17:39.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:38 smithi090 ceph-mon[30399]: 6.9 scrub ok 2024-04-14T15:17:39.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:38 smithi090 ceph-mon[30399]: 110.2 scrub starts 2024-04-14T15:17:39.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:38 smithi090 ceph-mon[30399]: 110.2 scrub ok 2024-04-14T15:17:39.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:38 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:17:39.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:38 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-51","var": "hit_set_count","val": "2"}]': finished 2024-04-14T15:17:39.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:38 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-51","var": "hit_set_period","val": "600"}]: dispatch 2024-04-14T15:17:39.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:38 smithi090 ceph-mon[30399]: osdmap e391: 8 total, 8 up, 8 in 2024-04-14T15:17:39.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:38 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-51","var": "hit_set_period","val": "600"}]: dispatch 2024-04-14T15:17:39.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:39.215 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:39.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:17:39.316 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:17:39.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:39.341 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:39.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:17:39.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:38 smithi092 ceph-mon[33966]: pgmap v530: 238 pgs: 32 creating+peering, 206 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:17:39.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:38 smithi092 ceph-mon[33966]: 4.15 scrub starts 2024-04-14T15:17:39.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:38 smithi092 ceph-mon[33966]: 4.15 scrub ok 2024-04-14T15:17:39.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:38 smithi092 ceph-mon[33966]: 11.15 scrub starts 2024-04-14T15:17:39.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:38 smithi092 ceph-mon[33966]: 11.15 scrub ok 2024-04-14T15:17:39.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:38 smithi092 ceph-mon[33966]: 6.9 scrub starts 2024-04-14T15:17:39.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:38 smithi092 ceph-mon[33966]: 6.9 scrub ok 2024-04-14T15:17:39.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:38 smithi092 ceph-mon[33966]: 110.2 scrub starts 2024-04-14T15:17:39.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:38 smithi092 ceph-mon[33966]: 110.2 scrub ok 2024-04-14T15:17:39.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:38 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:17:39.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:38 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-51","var": "hit_set_count","val": "2"}]': finished 2024-04-14T15:17:39.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:38 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-51","var": "hit_set_period","val": "600"}]: dispatch 2024-04-14T15:17:39.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:38 smithi092 ceph-mon[33966]: osdmap e391: 8 total, 8 up, 8 in 2024-04-14T15:17:39.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:38 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-51","var": "hit_set_period","val": "600"}]: dispatch 2024-04-14T15:17:39.443 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:17:39.473 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:39.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:39.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:17:39.575 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:17:39.602 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:39.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:39.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:17:39.704 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:17:39.731 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:39.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:39.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:17:39.833 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:17:39.861 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:39.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:39.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:17:39.963 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:17:39.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:39.990 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:39.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:17:40.090 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:17:40.115 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:40.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:40.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:17:40.216 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:17:40.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:40.241 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:40.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:17:40.342 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:17:40.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:40.367 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:40.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:17:40.468 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:17:40.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:40.493 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:40.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:17:40.594 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:17:40.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:40.620 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:40.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:17:40.721 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:17:40.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:40.745 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:40.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:17:40.846 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:17:40.875 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:40.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:40.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:17:40.977 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:17:41.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:41.002 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:41.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:17:41.033 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:40 smithi090 ceph-mon[41670]: pgmap v533: 238 pgs: 4 creating+peering, 234 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-04-14T15:17:41.033 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:40 smithi090 ceph-mon[41670]: 2.8 scrub starts 2024-04-14T15:17:41.033 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:40 smithi090 ceph-mon[41670]: 2.8 scrub ok 2024-04-14T15:17:41.033 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:40 smithi090 ceph-mon[41670]: 5.2 scrub starts 2024-04-14T15:17:41.033 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:40 smithi090 ceph-mon[41670]: 5.2 scrub ok 2024-04-14T15:17:41.033 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:40 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1014922700' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi090-93363-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi090-93363-66"}]': finished 2024-04-14T15:17:41.034 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:40 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-51","var": "hit_set_period","val": "600"}]': finished 2024-04-14T15:17:41.034 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:40 smithi090 ceph-mon[41670]: osdmap e392: 8 total, 8 up, 8 in 2024-04-14T15:17:41.034 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:40 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-14T15:17:41.034 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:40 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-14T15:17:41.034 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:40 smithi090 ceph-mon[41670]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:17:41.035 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:40 smithi090 ceph-mon[30399]: pgmap v533: 238 pgs: 4 creating+peering, 234 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-04-14T15:17:41.035 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:40 smithi090 ceph-mon[30399]: 2.8 scrub starts 2024-04-14T15:17:41.035 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:40 smithi090 ceph-mon[30399]: 2.8 scrub ok 2024-04-14T15:17:41.035 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:40 smithi090 ceph-mon[30399]: 5.2 scrub starts 2024-04-14T15:17:41.035 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:40 smithi090 ceph-mon[30399]: 5.2 scrub ok 2024-04-14T15:17:41.036 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:40 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1014922700' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi090-93363-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi090-93363-66"}]': finished 2024-04-14T15:17:41.036 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:40 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-51","var": "hit_set_period","val": "600"}]': finished 2024-04-14T15:17:41.036 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:40 smithi090 ceph-mon[30399]: osdmap e392: 8 total, 8 up, 8 in 2024-04-14T15:17:41.036 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:40 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-14T15:17:41.036 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:40 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-14T15:17:41.036 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:40 smithi090 ceph-mon[30399]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:17:41.104 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:17:41.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:41.130 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:41.130 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:17:41.231 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:17:41.255 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:41.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:41.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:17:41.357 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:17:41.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:40 smithi092 ceph-mon[33966]: pgmap v533: 238 pgs: 4 creating+peering, 234 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-04-14T15:17:41.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:40 smithi092 ceph-mon[33966]: 2.8 scrub starts 2024-04-14T15:17:41.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:40 smithi092 ceph-mon[33966]: 2.8 scrub ok 2024-04-14T15:17:41.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:40 smithi092 ceph-mon[33966]: 5.2 scrub starts 2024-04-14T15:17:41.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:40 smithi092 ceph-mon[33966]: 5.2 scrub ok 2024-04-14T15:17:41.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:40 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1014922700' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi090-93363-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi090-93363-66"}]': finished 2024-04-14T15:17:41.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:40 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-51","var": "hit_set_period","val": "600"}]': finished 2024-04-14T15:17:41.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:40 smithi092 ceph-mon[33966]: osdmap e392: 8 total, 8 up, 8 in 2024-04-14T15:17:41.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:40 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-14T15:17:41.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:40 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-14T15:17:41.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:40 smithi092 ceph-mon[33966]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:17:41.381 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:41.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:41.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:17:41.483 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:17:41.507 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:41.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:41.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:17:41.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.990+0000 7f9370209700 1 -- 172.21.15.90:0/1361945992 >> v1:172.21.15.90:6790/0 conn(0x7f936810f2f0 legacy=0x7f93681117b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:41.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.990+0000 7f9370209700 1 -- 172.21.15.90:0/1361945992 shutdown_connections 2024-04-14T15:17:41.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.990+0000 7f9370209700 1 -- 172.21.15.90:0/1361945992 >> 172.21.15.90:0/1361945992 conn(0x7f9368076aa0 msgr2=0x7f9368076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:17:41.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.990+0000 7f9370209700 1 -- 172.21.15.90:0/1361945992 shutdown_connections 2024-04-14T15:17:41.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.990+0000 7f9370209700 1 -- 172.21.15.90:0/1361945992 wait complete. 2024-04-14T15:17:41.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.990+0000 7f9370209700 1 librados: starting msgr at 2024-04-14T15:17:41.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.990+0000 7f9370209700 1 librados: starting objecter 2024-04-14T15:17:41.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.990+0000 7f9370209700 1 Processor -- start 2024-04-14T15:17:41.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.990+0000 7f9370209700 1 -- start start 2024-04-14T15:17:41.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.990+0000 7f9370209700 1 librados: setting wanted keys 2024-04-14T15:17:41.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.990+0000 7f9370209700 1 librados: calling monclient init 2024-04-14T15:17:41.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.991+0000 7f9370209700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f9368108170 con 0x7f9368108980 2024-04-14T15:17:41.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.991+0000 7f9370209700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f93681a8f00 con 0x7f936810f2f0 2024-04-14T15:17:41.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.991+0000 7f9370209700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f93681aa0e0 con 0x7f936810b6f0 2024-04-14T15:17:41.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.991+0000 7f936dfa5700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f9368108980 0x7f9368104fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:55320/0 (socket says 172.21.15.90:55320) 2024-04-14T15:17:41.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.991+0000 7f936dfa5700 1 -- 172.21.15.90:0/1162520788 learned_addr learned my addr 172.21.15.90:0/1162520788 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:17:41.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.991+0000 7f935effd700 10 client.?.objecter ms_handle_connect 0x7f9368108980 2024-04-14T15:17:41.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.991+0000 7f935effd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:17:41.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.991+0000 7f935effd700 1 -- 172.21.15.90:0/1162520788 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 703136200 0 0) 0x7f9368108170 con 0x7f9368108980 2024-04-14T15:17:41.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.991+0000 7f935effd700 1 -- 172.21.15.90:0/1162520788 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f93540031f0 con 0x7f9368108980 2024-04-14T15:17:41.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.991+0000 7f935effd700 10 client.?.objecter ms_handle_connect 0x7f936810b6f0 2024-04-14T15:17:41.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.991+0000 7f935effd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:17:41.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.991+0000 7f935effd700 10 client.?.objecter ms_handle_connect 0x7f936810f2f0 2024-04-14T15:17:41.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.991+0000 7f935effd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:17:41.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.991+0000 7f935effd700 1 -- 172.21.15.90:0/1162520788 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2967020917 0 0) 0x7f93681a8f00 con 0x7f936810f2f0 2024-04-14T15:17:41.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.991+0000 7f935effd700 1 -- 172.21.15.90:0/1162520788 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9368108170 con 0x7f936810f2f0 2024-04-14T15:17:41.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.991+0000 7f935effd700 1 -- 172.21.15.90:0/1162520788 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3691979410 0 0) 0x7f93681aa0e0 con 0x7f936810b6f0 2024-04-14T15:17:41.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.991+0000 7f935effd700 1 -- 172.21.15.90:0/1162520788 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f93681a8f00 con 0x7f936810b6f0 2024-04-14T15:17:41.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.992+0000 7f935effd700 1 -- 172.21.15.90:0/1162520788 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 270225426 0 0) 0x7f93540031f0 con 0x7f9368108980 2024-04-14T15:17:41.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.992+0000 7f935effd700 1 -- 172.21.15.90:0/1162520788 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f93681aa0e0 con 0x7f9368108980 2024-04-14T15:17:41.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.992+0000 7f935effd700 1 -- 172.21.15.90:0/1162520788 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f93640018b0 con 0x7f9368108980 2024-04-14T15:17:41.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.992+0000 7f935effd700 1 -- 172.21.15.90:0/1162520788 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3513930461 0 0) 0x7f9368108170 con 0x7f936810f2f0 2024-04-14T15:17:41.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.992+0000 7f935effd700 1 -- 172.21.15.90:0/1162520788 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f93540031f0 con 0x7f936810f2f0 2024-04-14T15:17:41.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.992+0000 7f935effd700 1 -- 172.21.15.90:0/1162520788 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f9360001950 con 0x7f936810f2f0 2024-04-14T15:17:41.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.992+0000 7f935effd700 1 -- 172.21.15.90:0/1162520788 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3784463335 0 0) 0x7f93681a8f00 con 0x7f936810b6f0 2024-04-14T15:17:41.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.992+0000 7f935effd700 1 -- 172.21.15.90:0/1162520788 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9368108170 con 0x7f936810b6f0 2024-04-14T15:17:41.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.992+0000 7f935effd700 1 -- 172.21.15.90:0/1162520788 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f9358001870 con 0x7f936810b6f0 2024-04-14T15:17:41.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.992+0000 7f935effd700 1 -- 172.21.15.90:0/1162520788 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3276086361 0 0) 0x7f93681aa0e0 con 0x7f9368108980 2024-04-14T15:17:41.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.992+0000 7f935effd700 1 -- 172.21.15.90:0/1162520788 >> v1:172.21.15.90:6790/0 conn(0x7f936810b6f0 legacy=0x7f93681056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:41.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.992+0000 7f935effd700 1 -- 172.21.15.90:0/1162520788 >> v1:172.21.15.92:6789/0 conn(0x7f936810f2f0 legacy=0x7f93681a67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:41.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.992+0000 7f935effd700 1 -- 172.21.15.90:0/1162520788 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93681ab2c0 con 0x7f9368108980 2024-04-14T15:17:41.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.992+0000 7f9370209700 1 -- 172.21.15.90:0/1162520788 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f93681aa310 con 0x7f9368108980 2024-04-14T15:17:41.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.992+0000 7f9370209700 10 client.37206.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:17:41.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.992+0000 7f9370209700 1 -- 172.21.15.90:0/1162520788 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f93681aa7d0 con 0x7f9368108980 2024-04-14T15:17:41.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.992+0000 7f9370209700 1 librados: init done 2024-04-14T15:17:41.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.992+0000 7f935effd700 1 -- 172.21.15.90:0/1162520788 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f9364003b40 con 0x7f9368108980 2024-04-14T15:17:41.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.992+0000 7f935effd700 1 -- 172.21.15.90:0/1162520788 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f9364005cf0 con 0x7f9368108980 2024-04-14T15:17:41.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.993+0000 7f9370209700 1 -- 172.21.15.90:0/1162520788 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f936806a060 con 0x7f9368108980 2024-04-14T15:17:41.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.994+0000 7f935effd700 1 -- 172.21.15.90:0/1162520788 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f936401c700 con 0x7f9368108980 2024-04-14T15:17:41.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.994+0000 7f935effd700 1 -- 172.21.15.90:0/1162520788 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(393..393 src has 51..393) ==== 9025+0+0 (unknown 641927582 0 0) 0x7f936408ad60 con 0x7f9368108980 2024-04-14T15:17:41.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.994+0000 7f935effd700 10 client.37206.objecter ms_dispatch 0x7f9368001550 osd_map(393..393 src has 51..393) 2024-04-14T15:17:41.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.994+0000 7f935effd700 3 client.37206.objecter handle_osd_map got epochs [393,393] > 0 2024-04-14T15:17:41.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.994+0000 7f935effd700 3 client.37206.objecter handle_osd_map decoding full epoch 393 2024-04-14T15:17:41.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.994+0000 7f935effd700 20 client.37206.objecter dump_active .. 0 homeless 2024-04-14T15:17:41.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.994+0000 7f935effd700 1 -- 172.21.15.90:0/1162520788 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v393) ==== 54+0+16 (unknown 2314621408 0 2420000192) 0x7f936408d0f0 con 0x7f9368108980 2024-04-14T15:17:41.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.994+0000 7f935effd700 10 client.37206.objecter ms_handle_connect 0x7f9354071200 2024-04-14T15:17:41.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.994+0000 7f9370209700 10 client.37206.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:17:41.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.994+0000 7f9370209700 15 client.37206.objecter _session_command_op_assign -1 1 2024-04-14T15:17:41.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.995+0000 7f9370209700 20 client.37206.objecter _get_session s=0x7f93681132c0 osd=0 3 2024-04-14T15:17:41.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.995+0000 7f9370209700 20 client.37206.objecter put_session s=0x7f93681132c0 osd=0 3 2024-04-14T15:17:41.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.995+0000 7f9370209700 20 client.37206.objecter _get_session s=0x7f93681132c0 osd=0 3 2024-04-14T15:17:41.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.995+0000 7f9370209700 15 client.37206.objecter _session_command_op_remove -1 1 2024-04-14T15:17:41.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.995+0000 7f9370209700 20 client.37206.objecter get_session s=0x7f93681132c0 osd=0 3 2024-04-14T15:17:41.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.995+0000 7f9370209700 15 client.37206.objecter _session_command_op_assign 0 1 2024-04-14T15:17:41.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.995+0000 7f9370209700 20 client.37206.objecter put_session s=0x7f93681132c0 osd=0 4 2024-04-14T15:17:41.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.995+0000 7f9370209700 10 client.37206.objecter _send_command 1 2024-04-14T15:17:41.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.995+0000 7f9370209700 1 -- 172.21.15.90:0/1162520788 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f93681064f0 con 0x7f93681a95e0 2024-04-14T15:17:41.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.995+0000 7f935effd700 10 client.37206.objecter ms_handle_connect 0x7f93681a95e0 2024-04-14T15:17:41.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.995+0000 7f935effd700 1 -- 172.21.15.90:0/1162520788 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f93681064f0 con 0x7f93681a95e0 2024-04-14T15:17:41.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.995+0000 7f935effd700 10 client.37206.objecter ms_dispatch 0x7f9368001550 command_reply(tid 1: 0 ) 2024-04-14T15:17:41.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.995+0000 7f935effd700 10 client.37206.objecter _finish_command 1 = system:0 2024-04-14T15:17:41.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.995+0000 7f935effd700 20 client.37206.objecter put_session s=0x7f93681132c0 osd=0 4 2024-04-14T15:17:41.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:40.995+0000 7f935effd700 15 client.37206.objecter _session_command_op_remove 0 1 2024-04-14T15:17:41.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.012+0000 7f9370209700 10 client.37206.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:17:41.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.012+0000 7f9370209700 15 client.37206.objecter _session_command_op_assign -1 2 2024-04-14T15:17:41.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.012+0000 7f9370209700 20 client.37206.objecter _get_session s=0x7f93681132c0 osd=0 3 2024-04-14T15:17:41.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.012+0000 7f9370209700 20 client.37206.objecter put_session s=0x7f93681132c0 osd=0 3 2024-04-14T15:17:41.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.012+0000 7f9370209700 20 client.37206.objecter _get_session s=0x7f93681132c0 osd=0 3 2024-04-14T15:17:41.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.012+0000 7f9370209700 15 client.37206.objecter _session_command_op_remove -1 2 2024-04-14T15:17:41.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.012+0000 7f9370209700 20 client.37206.objecter get_session s=0x7f93681132c0 osd=0 3 2024-04-14T15:17:41.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.012+0000 7f9370209700 15 client.37206.objecter _session_command_op_assign 0 2 2024-04-14T15:17:41.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.012+0000 7f9370209700 20 client.37206.objecter put_session s=0x7f93681132c0 osd=0 4 2024-04-14T15:17:41.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.012+0000 7f9370209700 10 client.37206.objecter _send_command 2 2024-04-14T15:17:41.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.012+0000 7f9370209700 1 -- 172.21.15.90:0/1162520788 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f93680693d0 con 0x7f93681a95e0 2024-04-14T15:17:41.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.014+0000 7f935effd700 1 -- 172.21.15.90:0/1162520788 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f93680693d0 con 0x7f93681a95e0 2024-04-14T15:17:41.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.014+0000 7f935effd700 10 client.37206.objecter ms_dispatch 0x7f9368001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:17:41.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.014+0000 7f935effd700 10 client.37206.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:17:41.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.014+0000 7f935effd700 20 client.37206.objecter put_session s=0x7f93681132c0 osd=0 4 2024-04-14T15:17:41.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.014+0000 7f935effd700 15 client.37206.objecter _session_command_op_remove 0 2 2024-04-14T15:17:41.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.014+0000 7f9370209700 10 client.37206.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:17:41.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.014+0000 7f9370209700 15 client.37206.objecter _session_command_op_assign -1 3 2024-04-14T15:17:41.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.014+0000 7f9370209700 20 client.37206.objecter _get_session s=0x7f93681aaf60 osd=1 3 2024-04-14T15:17:41.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.014+0000 7f9370209700 20 client.37206.objecter put_session s=0x7f93681aaf60 osd=1 3 2024-04-14T15:17:41.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.014+0000 7f9370209700 20 client.37206.objecter _get_session s=0x7f93681aaf60 osd=1 3 2024-04-14T15:17:41.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.014+0000 7f9370209700 15 client.37206.objecter _session_command_op_remove -1 3 2024-04-14T15:17:41.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.014+0000 7f9370209700 20 client.37206.objecter get_session s=0x7f93681aaf60 osd=1 3 2024-04-14T15:17:41.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.014+0000 7f9370209700 15 client.37206.objecter _session_command_op_assign 1 3 2024-04-14T15:17:41.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.014+0000 7f9370209700 20 client.37206.objecter put_session s=0x7f93681aaf60 osd=1 4 2024-04-14T15:17:41.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.014+0000 7f9370209700 10 client.37206.objecter _send_command 3 2024-04-14T15:17:41.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.014+0000 7f9370209700 1 -- 172.21.15.90:0/1162520788 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f93681ab0d0 con 0x7f93680636c0 2024-04-14T15:17:41.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.014+0000 7f935effd700 10 client.37206.objecter ms_handle_connect 0x7f93680636c0 2024-04-14T15:17:41.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.015+0000 7f935effd700 1 -- 172.21.15.90:0/1162520788 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f9364057430 con 0x7f93680636c0 2024-04-14T15:17:41.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.015+0000 7f935effd700 10 client.37206.objecter ms_dispatch 0x7f9368001550 command_reply(tid 3: 0 ) 2024-04-14T15:17:41.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.015+0000 7f935effd700 10 client.37206.objecter _finish_command 3 = system:0 2024-04-14T15:17:41.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.015+0000 7f935effd700 20 client.37206.objecter put_session s=0x7f93681aaf60 osd=1 4 2024-04-14T15:17:41.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.015+0000 7f935effd700 15 client.37206.objecter _session_command_op_remove 1 3 2024-04-14T15:17:41.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.032+0000 7f9370209700 10 client.37206.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:17:41.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.032+0000 7f9370209700 15 client.37206.objecter _session_command_op_assign -1 4 2024-04-14T15:17:41.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.032+0000 7f9370209700 20 client.37206.objecter _get_session s=0x7f93681aaf60 osd=1 3 2024-04-14T15:17:41.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.032+0000 7f9370209700 20 client.37206.objecter put_session s=0x7f93681aaf60 osd=1 3 2024-04-14T15:17:41.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.032+0000 7f9370209700 20 client.37206.objecter _get_session s=0x7f93681aaf60 osd=1 3 2024-04-14T15:17:41.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.032+0000 7f9370209700 15 client.37206.objecter _session_command_op_remove -1 4 2024-04-14T15:17:41.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.032+0000 7f9370209700 20 client.37206.objecter get_session s=0x7f93681aaf60 osd=1 3 2024-04-14T15:17:41.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.032+0000 7f9370209700 15 client.37206.objecter _session_command_op_assign 1 4 2024-04-14T15:17:41.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.032+0000 7f9370209700 20 client.37206.objecter put_session s=0x7f93681aaf60 osd=1 4 2024-04-14T15:17:41.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.032+0000 7f9370209700 10 client.37206.objecter _send_command 4 2024-04-14T15:17:41.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.032+0000 7f9370209700 1 -- 172.21.15.90:0/1162520788 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f936810e200 con 0x7f93680636c0 2024-04-14T15:17:41.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.033+0000 7f935effd700 1 -- 172.21.15.90:0/1162520788 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f936810e200 con 0x7f93680636c0 2024-04-14T15:17:41.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.033+0000 7f935effd700 10 client.37206.objecter ms_dispatch 0x7f9368001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:17:41.607 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:17:41.607 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:17:41.607 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:17:41.607 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:17:41.607 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:17:41.607 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:17:41.607 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:17:41.607 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:17:41.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.033+0000 7f935effd700 10 client.37206.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:17:41.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.033+0000 7f935effd700 20 client.37206.objecter put_session s=0x7f93681aaf60 osd=1 4 2024-04-14T15:17:41.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.033+0000 7f935effd700 15 client.37206.objecter _session_command_op_remove 1 4 2024-04-14T15:17:41.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.033+0000 7f9370209700 10 client.37206.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:17:41.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.033+0000 7f9370209700 15 client.37206.objecter _session_command_op_assign -1 5 2024-04-14T15:17:41.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.034+0000 7f9370209700 20 client.37206.objecter _get_session s=0x7f93681af510 osd=2 3 2024-04-14T15:17:41.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.034+0000 7f9370209700 20 client.37206.objecter put_session s=0x7f93681af510 osd=2 3 2024-04-14T15:17:41.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.034+0000 7f9370209700 20 client.37206.objecter _get_session s=0x7f93681af510 osd=2 3 2024-04-14T15:17:41.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.034+0000 7f9370209700 15 client.37206.objecter _session_command_op_remove -1 5 2024-04-14T15:17:41.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.034+0000 7f9370209700 20 client.37206.objecter get_session s=0x7f93681af510 osd=2 3 2024-04-14T15:17:41.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.034+0000 7f9370209700 15 client.37206.objecter _session_command_op_assign 2 5 2024-04-14T15:17:41.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.034+0000 7f9370209700 20 client.37206.objecter put_session s=0x7f93681af510 osd=2 4 2024-04-14T15:17:41.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.034+0000 7f9370209700 10 client.37206.objecter _send_command 5 2024-04-14T15:17:41.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.034+0000 7f9370209700 1 -- 172.21.15.90:0/1162520788 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f93681b3820 con 0x7f93681afb90 2024-04-14T15:17:41.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.034+0000 7f935effd700 10 client.37206.objecter ms_handle_connect 0x7f93681afb90 2024-04-14T15:17:41.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.034+0000 7f935effd700 1 -- 172.21.15.90:0/1162520788 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f93681b3820 con 0x7f93681afb90 2024-04-14T15:17:41.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.034+0000 7f935effd700 10 client.37206.objecter ms_dispatch 0x7f9368001550 command_reply(tid 5: 0 ) 2024-04-14T15:17:41.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.034+0000 7f935effd700 10 client.37206.objecter _finish_command 5 = system:0 2024-04-14T15:17:41.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.034+0000 7f935effd700 20 client.37206.objecter put_session s=0x7f93681af510 osd=2 4 2024-04-14T15:17:41.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.034+0000 7f935effd700 15 client.37206.objecter _session_command_op_remove 2 5 2024-04-14T15:17:41.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.051+0000 7f9370209700 10 client.37206.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:17:41.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.051+0000 7f9370209700 15 client.37206.objecter _session_command_op_assign -1 6 2024-04-14T15:17:41.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.051+0000 7f9370209700 20 client.37206.objecter _get_session s=0x7f93681af510 osd=2 3 2024-04-14T15:17:41.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.051+0000 7f9370209700 20 client.37206.objecter put_session s=0x7f93681af510 osd=2 3 2024-04-14T15:17:41.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.051+0000 7f9370209700 20 client.37206.objecter _get_session s=0x7f93681af510 osd=2 3 2024-04-14T15:17:41.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.051+0000 7f9370209700 15 client.37206.objecter _session_command_op_remove -1 6 2024-04-14T15:17:41.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.051+0000 7f9370209700 20 client.37206.objecter get_session s=0x7f93681af510 osd=2 3 2024-04-14T15:17:41.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.051+0000 7f9370209700 15 client.37206.objecter _session_command_op_assign 2 6 2024-04-14T15:17:41.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.051+0000 7f9370209700 20 client.37206.objecter put_session s=0x7f93681af510 osd=2 4 2024-04-14T15:17:41.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.051+0000 7f9370209700 10 client.37206.objecter _send_command 6 2024-04-14T15:17:41.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.051+0000 7f9370209700 1 -- 172.21.15.90:0/1162520788 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f93680631a0 con 0x7f93681afb90 2024-04-14T15:17:41.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.052+0000 7f935effd700 1 -- 172.21.15.90:0/1162520788 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f93680631a0 con 0x7f93681afb90 2024-04-14T15:17:41.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.052+0000 7f935effd700 10 client.37206.objecter ms_dispatch 0x7f9368001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:17:41.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.052+0000 7f935effd700 10 client.37206.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:17:41.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.052+0000 7f935effd700 20 client.37206.objecter put_session s=0x7f93681af510 osd=2 4 2024-04-14T15:17:41.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.052+0000 7f935effd700 15 client.37206.objecter _session_command_op_remove 2 6 2024-04-14T15:17:41.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.053+0000 7f9370209700 10 client.37206.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:17:41.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.053+0000 7f9370209700 15 client.37206.objecter _session_command_op_assign -1 7 2024-04-14T15:17:41.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.053+0000 7f9370209700 20 client.37206.objecter _get_session s=0x7f93681b4bb0 osd=3 3 2024-04-14T15:17:41.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.053+0000 7f9370209700 20 client.37206.objecter put_session s=0x7f93681b4bb0 osd=3 3 2024-04-14T15:17:41.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.053+0000 7f9370209700 20 client.37206.objecter _get_session s=0x7f93681b4bb0 osd=3 3 2024-04-14T15:17:41.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.053+0000 7f9370209700 15 client.37206.objecter _session_command_op_remove -1 7 2024-04-14T15:17:41.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.053+0000 7f9370209700 20 client.37206.objecter get_session s=0x7f93681b4bb0 osd=3 3 2024-04-14T15:17:41.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.053+0000 7f9370209700 15 client.37206.objecter _session_command_op_assign 3 7 2024-04-14T15:17:41.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.053+0000 7f9370209700 20 client.37206.objecter put_session s=0x7f93681b4bb0 osd=3 4 2024-04-14T15:17:41.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.053+0000 7f9370209700 10 client.37206.objecter _send_command 7 2024-04-14T15:17:41.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.053+0000 7f9370209700 1 -- 172.21.15.90:0/1162520788 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f93681b4e10 con 0x7f936802db00 2024-04-14T15:17:41.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.053+0000 7f935effd700 10 client.37206.objecter ms_handle_connect 0x7f936802db00 2024-04-14T15:17:41.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.053+0000 7f935effd700 1 -- 172.21.15.90:0/1162520788 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f93681b4e10 con 0x7f936802db00 2024-04-14T15:17:41.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.053+0000 7f935effd700 10 client.37206.objecter ms_dispatch 0x7f9368001550 command_reply(tid 7: 0 ) 2024-04-14T15:17:41.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.053+0000 7f935effd700 10 client.37206.objecter _finish_command 7 = system:0 2024-04-14T15:17:41.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.053+0000 7f935effd700 20 client.37206.objecter put_session s=0x7f93681b4bb0 osd=3 4 2024-04-14T15:17:41.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.053+0000 7f935effd700 15 client.37206.objecter _session_command_op_remove 3 7 2024-04-14T15:17:41.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.070+0000 7f9370209700 10 client.37206.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:17:41.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.070+0000 7f9370209700 15 client.37206.objecter _session_command_op_assign -1 8 2024-04-14T15:17:41.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.070+0000 7f9370209700 20 client.37206.objecter _get_session s=0x7f93681b4bb0 osd=3 3 2024-04-14T15:17:41.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.070+0000 7f9370209700 20 client.37206.objecter put_session s=0x7f93681b4bb0 osd=3 3 2024-04-14T15:17:41.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.070+0000 7f9370209700 20 client.37206.objecter _get_session s=0x7f93681b4bb0 osd=3 3 2024-04-14T15:17:41.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.070+0000 7f9370209700 15 client.37206.objecter _session_command_op_remove -1 8 2024-04-14T15:17:41.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.070+0000 7f9370209700 20 client.37206.objecter get_session s=0x7f93681b4bb0 osd=3 3 2024-04-14T15:17:41.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.070+0000 7f9370209700 15 client.37206.objecter _session_command_op_assign 3 8 2024-04-14T15:17:41.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.070+0000 7f9370209700 20 client.37206.objecter put_session s=0x7f93681b4bb0 osd=3 4 2024-04-14T15:17:41.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.070+0000 7f9370209700 10 client.37206.objecter _send_command 8 2024-04-14T15:17:41.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.070+0000 7f9370209700 1 -- 172.21.15.90:0/1162520788 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f936802d070 con 0x7f936802db00 2024-04-14T15:17:41.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.072+0000 7f935effd700 1 -- 172.21.15.90:0/1162520788 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f936802d070 con 0x7f936802db00 2024-04-14T15:17:41.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.072+0000 7f935effd700 10 client.37206.objecter ms_dispatch 0x7f9368001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:17:41.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.072+0000 7f935effd700 10 client.37206.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:17:41.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.072+0000 7f935effd700 20 client.37206.objecter put_session s=0x7f93681b4bb0 osd=3 4 2024-04-14T15:17:41.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.072+0000 7f935effd700 15 client.37206.objecter _session_command_op_remove 3 8 2024-04-14T15:17:41.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.072+0000 7f9370209700 10 client.37206.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:17:41.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.072+0000 7f9370209700 15 client.37206.objecter _session_command_op_assign -1 9 2024-04-14T15:17:41.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.072+0000 7f9370209700 20 client.37206.objecter _get_session s=0x7f936802d070 osd=4 3 2024-04-14T15:17:41.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.072+0000 7f9370209700 20 client.37206.objecter put_session s=0x7f936802d070 osd=4 3 2024-04-14T15:17:41.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.072+0000 7f9370209700 20 client.37206.objecter _get_session s=0x7f936802d070 osd=4 3 2024-04-14T15:17:41.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.072+0000 7f9370209700 15 client.37206.objecter _session_command_op_remove -1 9 2024-04-14T15:17:41.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.072+0000 7f9370209700 20 client.37206.objecter get_session s=0x7f936802d070 osd=4 3 2024-04-14T15:17:41.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.072+0000 7f9370209700 15 client.37206.objecter _session_command_op_assign 4 9 2024-04-14T15:17:41.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.072+0000 7f9370209700 20 client.37206.objecter put_session s=0x7f936802d070 osd=4 4 2024-04-14T15:17:41.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.072+0000 7f9370209700 10 client.37206.objecter _send_command 9 2024-04-14T15:17:41.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.072+0000 7f9370209700 1 -- 172.21.15.90:0/1162520788 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f93681b4870 con 0x7f936802d1e0 2024-04-14T15:17:41.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.073+0000 7f935effd700 10 client.37206.objecter ms_handle_connect 0x7f936802d1e0 2024-04-14T15:17:41.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.073+0000 7f935effd700 1 -- 172.21.15.90:0/1162520788 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f93681b4870 con 0x7f936802d1e0 2024-04-14T15:17:41.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.073+0000 7f935effd700 10 client.37206.objecter ms_dispatch 0x7f9368001550 command_reply(tid 9: 0 ) 2024-04-14T15:17:41.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.073+0000 7f935effd700 10 client.37206.objecter _finish_command 9 = system:0 2024-04-14T15:17:41.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.073+0000 7f935effd700 20 client.37206.objecter put_session s=0x7f936802d070 osd=4 4 2024-04-14T15:17:41.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.073+0000 7f935effd700 15 client.37206.objecter _session_command_op_remove 4 9 2024-04-14T15:17:41.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.090+0000 7f9370209700 10 client.37206.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:17:41.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.090+0000 7f9370209700 15 client.37206.objecter _session_command_op_assign -1 10 2024-04-14T15:17:41.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.090+0000 7f9370209700 20 client.37206.objecter _get_session s=0x7f936802d070 osd=4 3 2024-04-14T15:17:41.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.090+0000 7f9370209700 20 client.37206.objecter put_session s=0x7f936802d070 osd=4 3 2024-04-14T15:17:41.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.090+0000 7f9370209700 20 client.37206.objecter _get_session s=0x7f936802d070 osd=4 3 2024-04-14T15:17:41.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.090+0000 7f9370209700 15 client.37206.objecter _session_command_op_remove -1 10 2024-04-14T15:17:41.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.090+0000 7f9370209700 20 client.37206.objecter get_session s=0x7f936802d070 osd=4 3 2024-04-14T15:17:41.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.090+0000 7f9370209700 15 client.37206.objecter _session_command_op_assign 4 10 2024-04-14T15:17:41.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.090+0000 7f9370209700 20 client.37206.objecter put_session s=0x7f936802d070 osd=4 4 2024-04-14T15:17:41.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.090+0000 7f9370209700 10 client.37206.objecter _send_command 10 2024-04-14T15:17:41.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.090+0000 7f9370209700 1 -- 172.21.15.90:0/1162520788 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f93681b4870 con 0x7f936802d1e0 2024-04-14T15:17:41.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.092+0000 7f935effd700 1 -- 172.21.15.90:0/1162520788 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f93681b4870 con 0x7f936802d1e0 2024-04-14T15:17:41.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.092+0000 7f935effd700 10 client.37206.objecter ms_dispatch 0x7f9368001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:17:41.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.092+0000 7f935effd700 10 client.37206.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:17:41.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.092+0000 7f935effd700 20 client.37206.objecter put_session s=0x7f936802d070 osd=4 4 2024-04-14T15:17:41.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.092+0000 7f935effd700 15 client.37206.objecter _session_command_op_remove 4 10 2024-04-14T15:17:41.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.092+0000 7f9370209700 10 client.37206.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:17:41.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.092+0000 7f9370209700 15 client.37206.objecter _session_command_op_assign -1 11 2024-04-14T15:17:41.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.092+0000 7f9370209700 20 client.37206.objecter _get_session s=0x7f93681b4870 osd=5 3 2024-04-14T15:17:41.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.092+0000 7f9370209700 20 client.37206.objecter put_session s=0x7f93681b4870 osd=5 3 2024-04-14T15:17:41.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.092+0000 7f9370209700 20 client.37206.objecter _get_session s=0x7f93681b4870 osd=5 3 2024-04-14T15:17:41.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.092+0000 7f9370209700 15 client.37206.objecter _session_command_op_remove -1 11 2024-04-14T15:17:41.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.092+0000 7f9370209700 20 client.37206.objecter get_session s=0x7f93681b4870 osd=5 3 2024-04-14T15:17:41.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.092+0000 7f9370209700 15 client.37206.objecter _session_command_op_assign 5 11 2024-04-14T15:17:41.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.092+0000 7f9370209700 20 client.37206.objecter put_session s=0x7f93681b4870 osd=5 4 2024-04-14T15:17:41.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.092+0000 7f9370209700 10 client.37206.objecter _send_command 11 2024-04-14T15:17:41.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.092+0000 7f9370209700 1 -- 172.21.15.90:0/1162520788 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f93681c13a0 con 0x7f93681bd6f0 2024-04-14T15:17:41.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.093+0000 7f935effd700 10 client.37206.objecter ms_handle_connect 0x7f93681bd6f0 2024-04-14T15:17:41.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.094+0000 7f935effd700 1 -- 172.21.15.90:0/1162520788 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f93681c13a0 con 0x7f93681bd6f0 2024-04-14T15:17:41.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.094+0000 7f935effd700 10 client.37206.objecter ms_dispatch 0x7f9368001550 command_reply(tid 11: 0 ) 2024-04-14T15:17:41.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.094+0000 7f935effd700 10 client.37206.objecter _finish_command 11 = system:0 2024-04-14T15:17:41.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.094+0000 7f935effd700 20 client.37206.objecter put_session s=0x7f93681b4870 osd=5 4 2024-04-14T15:17:41.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.094+0000 7f935effd700 15 client.37206.objecter _session_command_op_remove 5 11 2024-04-14T15:17:41.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.110+0000 7f9370209700 10 client.37206.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:17:41.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.110+0000 7f9370209700 15 client.37206.objecter _session_command_op_assign -1 12 2024-04-14T15:17:41.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.110+0000 7f9370209700 20 client.37206.objecter _get_session s=0x7f93681b4870 osd=5 3 2024-04-14T15:17:41.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.110+0000 7f9370209700 20 client.37206.objecter put_session s=0x7f93681b4870 osd=5 3 2024-04-14T15:17:41.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.110+0000 7f9370209700 20 client.37206.objecter _get_session s=0x7f93681b4870 osd=5 3 2024-04-14T15:17:41.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.110+0000 7f9370209700 15 client.37206.objecter _session_command_op_remove -1 12 2024-04-14T15:17:41.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.110+0000 7f9370209700 20 client.37206.objecter get_session s=0x7f93681b4870 osd=5 3 2024-04-14T15:17:41.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.110+0000 7f9370209700 15 client.37206.objecter _session_command_op_assign 5 12 2024-04-14T15:17:41.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.110+0000 7f9370209700 20 client.37206.objecter put_session s=0x7f93681b4870 osd=5 4 2024-04-14T15:17:41.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.110+0000 7f9370209700 10 client.37206.objecter _send_command 12 2024-04-14T15:17:41.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.110+0000 7f9370209700 1 -- 172.21.15.90:0/1162520788 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f93681c13a0 con 0x7f93681bd6f0 2024-04-14T15:17:41.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.112+0000 7f935effd700 1 -- 172.21.15.90:0/1162520788 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f93681c13a0 con 0x7f93681bd6f0 2024-04-14T15:17:41.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.112+0000 7f935effd700 10 client.37206.objecter ms_dispatch 0x7f9368001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:17:41.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.112+0000 7f935effd700 10 client.37206.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:17:41.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.112+0000 7f935effd700 20 client.37206.objecter put_session s=0x7f93681b4870 osd=5 4 2024-04-14T15:17:41.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.112+0000 7f935effd700 15 client.37206.objecter _session_command_op_remove 5 12 2024-04-14T15:17:41.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.112+0000 7f9370209700 10 client.37206.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:17:41.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.112+0000 7f9370209700 15 client.37206.objecter _session_command_op_assign -1 13 2024-04-14T15:17:41.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.112+0000 7f9370209700 20 client.37206.objecter _get_session s=0x7f93681c13a0 osd=6 3 2024-04-14T15:17:41.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.112+0000 7f9370209700 20 client.37206.objecter put_session s=0x7f93681c13a0 osd=6 3 2024-04-14T15:17:41.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.112+0000 7f9370209700 20 client.37206.objecter _get_session s=0x7f93681c13a0 osd=6 3 2024-04-14T15:17:41.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.112+0000 7f9370209700 15 client.37206.objecter _session_command_op_remove -1 13 2024-04-14T15:17:41.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.112+0000 7f9370209700 20 client.37206.objecter get_session s=0x7f93681c13a0 osd=6 3 2024-04-14T15:17:41.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.112+0000 7f9370209700 15 client.37206.objecter _session_command_op_assign 6 13 2024-04-14T15:17:41.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.112+0000 7f9370209700 20 client.37206.objecter put_session s=0x7f93681c13a0 osd=6 4 2024-04-14T15:17:41.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.112+0000 7f9370209700 10 client.37206.objecter _send_command 13 2024-04-14T15:17:41.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.112+0000 7f9370209700 1 -- 172.21.15.90:0/1162520788 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f93681c2600 con 0x7f93681c3260 2024-04-14T15:17:41.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.112+0000 7f935effd700 10 client.37206.objecter ms_handle_connect 0x7f93681c3260 2024-04-14T15:17:41.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.113+0000 7f935effd700 1 -- 172.21.15.90:0/1162520788 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f93681c2600 con 0x7f93681c3260 2024-04-14T15:17:41.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.113+0000 7f935effd700 10 client.37206.objecter ms_dispatch 0x7f9368001550 command_reply(tid 13: 0 ) 2024-04-14T15:17:41.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.113+0000 7f935effd700 10 client.37206.objecter _finish_command 13 = system:0 2024-04-14T15:17:41.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.113+0000 7f935effd700 20 client.37206.objecter put_session s=0x7f93681c13a0 osd=6 4 2024-04-14T15:17:41.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.113+0000 7f935effd700 15 client.37206.objecter _session_command_op_remove 6 13 2024-04-14T15:17:41.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.130+0000 7f9370209700 10 client.37206.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:17:41.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.130+0000 7f9370209700 15 client.37206.objecter _session_command_op_assign -1 14 2024-04-14T15:17:41.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.130+0000 7f9370209700 20 client.37206.objecter _get_session s=0x7f93681c13a0 osd=6 3 2024-04-14T15:17:41.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.130+0000 7f9370209700 20 client.37206.objecter put_session s=0x7f93681c13a0 osd=6 3 2024-04-14T15:17:41.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.130+0000 7f9370209700 20 client.37206.objecter _get_session s=0x7f93681c13a0 osd=6 3 2024-04-14T15:17:41.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.130+0000 7f9370209700 15 client.37206.objecter _session_command_op_remove -1 14 2024-04-14T15:17:41.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.130+0000 7f9370209700 20 client.37206.objecter get_session s=0x7f93681c13a0 osd=6 3 2024-04-14T15:17:41.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.130+0000 7f9370209700 15 client.37206.objecter _session_command_op_assign 6 14 2024-04-14T15:17:41.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.130+0000 7f9370209700 20 client.37206.objecter put_session s=0x7f93681c13a0 osd=6 4 2024-04-14T15:17:41.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.130+0000 7f9370209700 10 client.37206.objecter _send_command 14 2024-04-14T15:17:41.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.130+0000 7f9370209700 1 -- 172.21.15.90:0/1162520788 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f93681c2600 con 0x7f93681c3260 2024-04-14T15:17:41.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.131+0000 7f935effd700 1 -- 172.21.15.90:0/1162520788 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f93681c2600 con 0x7f93681c3260 2024-04-14T15:17:41.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.131+0000 7f935effd700 10 client.37206.objecter ms_dispatch 0x7f9368001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:17:41.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.131+0000 7f935effd700 10 client.37206.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:17:41.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.131+0000 7f935effd700 20 client.37206.objecter put_session s=0x7f93681c13a0 osd=6 4 2024-04-14T15:17:41.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.131+0000 7f935effd700 15 client.37206.objecter _session_command_op_remove 6 14 2024-04-14T15:17:41.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.132+0000 7f9370209700 10 client.37206.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:17:41.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.132+0000 7f9370209700 15 client.37206.objecter _session_command_op_assign -1 15 2024-04-14T15:17:41.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.132+0000 7f9370209700 20 client.37206.objecter _get_session s=0x7f93681c2600 osd=7 3 2024-04-14T15:17:41.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.132+0000 7f9370209700 20 client.37206.objecter put_session s=0x7f93681c2600 osd=7 3 2024-04-14T15:17:41.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.132+0000 7f9370209700 20 client.37206.objecter _get_session s=0x7f93681c2600 osd=7 3 2024-04-14T15:17:41.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.132+0000 7f9370209700 15 client.37206.objecter _session_command_op_remove -1 15 2024-04-14T15:17:41.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.132+0000 7f9370209700 20 client.37206.objecter get_session s=0x7f93681c2600 osd=7 3 2024-04-14T15:17:41.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.132+0000 7f9370209700 15 client.37206.objecter _session_command_op_assign 7 15 2024-04-14T15:17:41.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.132+0000 7f9370209700 20 client.37206.objecter put_session s=0x7f93681c2600 osd=7 4 2024-04-14T15:17:41.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.132+0000 7f9370209700 10 client.37206.objecter _send_command 15 2024-04-14T15:17:41.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.132+0000 7f9370209700 1 -- 172.21.15.90:0/1162520788 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f93681c9ce0 con 0x7f93681c2880 2024-04-14T15:17:41.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.132+0000 7f935effd700 10 client.37206.objecter ms_handle_connect 0x7f93681c2880 2024-04-14T15:17:41.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.133+0000 7f935effd700 1 -- 172.21.15.90:0/1162520788 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f93681c9ce0 con 0x7f93681c2880 2024-04-14T15:17:41.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.133+0000 7f935effd700 10 client.37206.objecter ms_dispatch 0x7f9368001550 command_reply(tid 15: 0 ) 2024-04-14T15:17:41.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.133+0000 7f935effd700 10 client.37206.objecter _finish_command 15 = system:0 2024-04-14T15:17:41.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.133+0000 7f935effd700 20 client.37206.objecter put_session s=0x7f93681c2600 osd=7 4 2024-04-14T15:17:41.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.133+0000 7f935effd700 15 client.37206.objecter _session_command_op_remove 7 15 2024-04-14T15:17:41.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.150+0000 7f9370209700 10 client.37206.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:17:41.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.150+0000 7f9370209700 15 client.37206.objecter _session_command_op_assign -1 16 2024-04-14T15:17:41.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.150+0000 7f9370209700 20 client.37206.objecter _get_session s=0x7f93681c2600 osd=7 3 2024-04-14T15:17:41.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.150+0000 7f9370209700 20 client.37206.objecter put_session s=0x7f93681c2600 osd=7 3 2024-04-14T15:17:41.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.150+0000 7f9370209700 20 client.37206.objecter _get_session s=0x7f93681c2600 osd=7 3 2024-04-14T15:17:41.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.150+0000 7f9370209700 15 client.37206.objecter _session_command_op_remove -1 16 2024-04-14T15:17:41.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.150+0000 7f9370209700 20 client.37206.objecter get_session s=0x7f93681c2600 osd=7 3 2024-04-14T15:17:41.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.150+0000 7f9370209700 15 client.37206.objecter _session_command_op_assign 7 16 2024-04-14T15:17:41.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.150+0000 7f9370209700 20 client.37206.objecter put_session s=0x7f93681c2600 osd=7 4 2024-04-14T15:17:41.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.150+0000 7f9370209700 10 client.37206.objecter _send_command 16 2024-04-14T15:17:41.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.150+0000 7f9370209700 1 -- 172.21.15.90:0/1162520788 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f93681c9ca0 con 0x7f93681c2880 2024-04-14T15:17:41.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.151+0000 7f935effd700 1 -- 172.21.15.90:0/1162520788 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f93681c9ca0 con 0x7f93681c2880 2024-04-14T15:17:41.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.151+0000 7f935effd700 10 client.37206.objecter ms_dispatch 0x7f9368001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:17:41.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.151+0000 7f935effd700 10 client.37206.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:17:41.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.151+0000 7f935effd700 20 client.37206.objecter put_session s=0x7f93681c2600 osd=7 4 2024-04-14T15:17:41.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.151+0000 7f935effd700 15 client.37206.objecter _session_command_op_remove 7 16 2024-04-14T15:17:41.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.152+0000 7f9370209700 10 librados: watch_flush enter 2024-04-14T15:17:41.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.152+0000 7f9370209700 10 librados: watch_flush exit 2024-04-14T15:17:41.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.152+0000 7f9370209700 10 client.37206.objecter close_session for osd.0 2024-04-14T15:17:41.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.152+0000 7f9370209700 1 -- 172.21.15.90:0/1162520788 >> v1:172.21.15.90:6801/3761806885 conn(0x7f93681a95e0 legacy=0x7f93681a99c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:41.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.152+0000 7f9370209700 20 client.37206.objecter put_session s=0x7f93681132c0 osd=0 1 2024-04-14T15:17:41.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.152+0000 7f9370209700 10 client.37206.objecter close_session for osd.1 2024-04-14T15:17:41.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.152+0000 7f9370209700 1 -- 172.21.15.90:0/1162520788 >> v1:172.21.15.90:6805/4170361699 conn(0x7f93680636c0 legacy=0x7f936810cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:41.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.152+0000 7f9370209700 20 client.37206.objecter put_session s=0x7f93681aaf60 osd=1 1 2024-04-14T15:17:41.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.152+0000 7f9370209700 10 client.37206.objecter close_session for osd.2 2024-04-14T15:17:41.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.152+0000 7f9370209700 1 -- 172.21.15.90:0/1162520788 >> v1:172.21.15.90:6809/1686184251 conn(0x7f93681afb90 legacy=0x7f93681b1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:41.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.152+0000 7f9370209700 20 client.37206.objecter put_session s=0x7f93681af510 osd=2 1 2024-04-14T15:17:41.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.152+0000 7f9370209700 10 client.37206.objecter close_session for osd.3 2024-04-14T15:17:41.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.152+0000 7f9370209700 1 -- 172.21.15.90:0/1162520788 >> v1:172.21.15.90:6813/2270166774 conn(0x7f936802db00 legacy=0x7f93681b7210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:41.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.152+0000 7f9370209700 20 client.37206.objecter put_session s=0x7f93681b4bb0 osd=3 1 2024-04-14T15:17:41.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.152+0000 7f9370209700 10 client.37206.objecter close_session for osd.4 2024-04-14T15:17:41.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.152+0000 7f9370209700 1 -- 172.21.15.90:0/1162520788 >> v1:172.21.15.92:6800/3887549733 conn(0x7f936802d1e0 legacy=0x7f93681bbac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:41.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.152+0000 7f9370209700 20 client.37206.objecter put_session s=0x7f936802d070 osd=4 1 2024-04-14T15:17:41.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.152+0000 7f9370209700 10 client.37206.objecter close_session for osd.5 2024-04-14T15:17:41.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.152+0000 7f936e7a6700 1 -- 172.21.15.90:0/1162520788 reap_dead start 2024-04-14T15:17:41.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.152+0000 7f9370209700 1 -- 172.21.15.90:0/1162520788 >> v1:172.21.15.92:6804/3244561440 conn(0x7f93681bd6f0 legacy=0x7f93681bfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:41.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.152+0000 7f936e7a6700 1 -- 172.21.15.90:0/1162520788 reap_dead start 2024-04-14T15:17:41.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.152+0000 7f936e7a6700 1 -- 172.21.15.90:0/1162520788 reap_dead start 2024-04-14T15:17:41.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.152+0000 7f9370209700 20 client.37206.objecter put_session s=0x7f93681b4870 osd=5 1 2024-04-14T15:17:41.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.152+0000 7f9370209700 10 client.37206.objecter close_session for osd.6 2024-04-14T15:17:41.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.152+0000 7f9370209700 1 -- 172.21.15.90:0/1162520788 >> v1:172.21.15.92:6808/636125963 conn(0x7f93681c3260 legacy=0x7f93681c36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:41.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.152+0000 7f9370209700 20 client.37206.objecter put_session s=0x7f93681c13a0 osd=6 1 2024-04-14T15:17:41.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.152+0000 7f9370209700 10 client.37206.objecter close_session for osd.7 2024-04-14T15:17:41.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.152+0000 7f9370209700 1 -- 172.21.15.90:0/1162520788 >> v1:172.21.15.92:6812/1773706716 conn(0x7f93681c2880 legacy=0x7f93681c8470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:41.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.152+0000 7f9370209700 20 client.37206.objecter put_session s=0x7f93681c2600 osd=7 1 2024-04-14T15:17:41.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.152+0000 7f9370209700 20 client.37206.objecter shutdown clearing up homeless session... 2024-04-14T15:17:41.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.152+0000 7f9370209700 10 client.37206.objecter successfully canceled tick 2024-04-14T15:17:41.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.152+0000 7f9370209700 1 -- 172.21.15.90:0/1162520788 >> v1:172.21.15.90:6800/420935427 conn(0x7f9354071200 legacy=0x7f93540736c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:41.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.152+0000 7f9370209700 1 -- 172.21.15.90:0/1162520788 >> v1:172.21.15.90:6789/0 conn(0x7f9368108980 legacy=0x7f9368104fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:41.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.152+0000 7f936e7a6700 1 -- 172.21.15.90:0/1162520788 reap_dead start 2024-04-14T15:17:41.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.152+0000 7f9370209700 1 -- 172.21.15.90:0/1162520788 shutdown_connections 2024-04-14T15:17:41.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.152+0000 7f9370209700 1 -- 172.21.15.90:0/1162520788 >> 172.21.15.90:0/1162520788 conn(0x7f9368076aa0 msgr2=0x7f9368111770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:17:41.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.152+0000 7f9370209700 1 -- 172.21.15.90:0/1162520788 shutdown_connections 2024-04-14T15:17:41.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.152+0000 7f9370209700 1 -- 172.21.15.90:0/1162520788 wait complete. 2024-04-14T15:17:41.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:41.152+0000 7f9370209700 1 librados: shutdown 2024-04-14T15:17:41.625 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:17:41.625 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:17:41.625 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:17:41.625 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:17:41.625 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:17:41.625 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:17:41.626 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:17:41.626 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:17:41.626 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:17:41.651 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:41.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:41.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:17:41.753 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:17:41.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:41.780 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:41.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:17:41.881 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:17:41.907 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:41.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:41.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:17:42.009 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:17:42.035 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:42.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:42.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:17:42.137 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:17:42.161 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:42.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:42.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:17:42.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:41 smithi090 ceph-mon[41670]: 1.7 scrub starts 2024-04-14T15:17:42.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:41 smithi090 ceph-mon[41670]: 1.7 scrub ok 2024-04-14T15:17:42.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:41 smithi090 ceph-mon[41670]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:17:42.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:41 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-51","var": "hit_set_type","val": "bloom"}]': finished 2024-04-14T15:17:42.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:41 smithi090 ceph-mon[41670]: osdmap e393: 8 total, 8 up, 8 in 2024-04-14T15:17:42.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:41 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-14T15:17:42.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:41 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-14T15:17:42.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:41 smithi090 ceph-mon[41670]: 2.4 scrub starts 2024-04-14T15:17:42.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:41 smithi090 ceph-mon[41670]: 2.4 scrub ok 2024-04-14T15:17:42.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:41 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1162520788' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:17:42.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:41 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-51","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-04-14T15:17:42.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:41 smithi090 ceph-mon[41670]: osdmap e394: 8 total, 8 up, 8 in 2024-04-14T15:17:42.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:41 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-51","var": "target_max_objects","val": "1"}]: dispatch 2024-04-14T15:17:42.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:41 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1014922700' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi090-93363-66"}]: dispatch 2024-04-14T15:17:42.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:41 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-51","var": "target_max_objects","val": "1"}]: dispatch 2024-04-14T15:17:42.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:41 smithi090 ceph-mon[30399]: 1.7 scrub starts 2024-04-14T15:17:42.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:41 smithi090 ceph-mon[30399]: 1.7 scrub ok 2024-04-14T15:17:42.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:41 smithi090 ceph-mon[30399]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:17:42.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:41 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-51","var": "hit_set_type","val": "bloom"}]': finished 2024-04-14T15:17:42.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:41 smithi090 ceph-mon[30399]: osdmap e393: 8 total, 8 up, 8 in 2024-04-14T15:17:42.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:41 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-14T15:17:42.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:41 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-14T15:17:42.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:41 smithi090 ceph-mon[30399]: 2.4 scrub starts 2024-04-14T15:17:42.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:41 smithi090 ceph-mon[30399]: 2.4 scrub ok 2024-04-14T15:17:42.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:41 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1162520788' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:17:42.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:41 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-51","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-04-14T15:17:42.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:41 smithi090 ceph-mon[30399]: osdmap e394: 8 total, 8 up, 8 in 2024-04-14T15:17:42.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:41 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-51","var": "target_max_objects","val": "1"}]: dispatch 2024-04-14T15:17:42.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:41 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1014922700' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi090-93363-66"}]: dispatch 2024-04-14T15:17:42.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:41 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-51","var": "target_max_objects","val": "1"}]: dispatch 2024-04-14T15:17:42.263 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:17:42.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:42.290 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:42.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:17:42.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:41 smithi092 ceph-mon[33966]: 1.7 scrub starts 2024-04-14T15:17:42.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:41 smithi092 ceph-mon[33966]: 1.7 scrub ok 2024-04-14T15:17:42.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:41 smithi092 ceph-mon[33966]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:17:42.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:41 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-51","var": "hit_set_type","val": "bloom"}]': finished 2024-04-14T15:17:42.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:41 smithi092 ceph-mon[33966]: osdmap e393: 8 total, 8 up, 8 in 2024-04-14T15:17:42.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:41 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-14T15:17:42.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:41 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-14T15:17:42.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:41 smithi092 ceph-mon[33966]: 2.4 scrub starts 2024-04-14T15:17:42.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:41 smithi092 ceph-mon[33966]: 2.4 scrub ok 2024-04-14T15:17:42.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:41 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1162520788' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:17:42.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:41 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-51","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-04-14T15:17:42.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:41 smithi092 ceph-mon[33966]: osdmap e394: 8 total, 8 up, 8 in 2024-04-14T15:17:42.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:41 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-51","var": "target_max_objects","val": "1"}]: dispatch 2024-04-14T15:17:42.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:41 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1014922700' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi090-93363-66"}]: dispatch 2024-04-14T15:17:42.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:41 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-51","var": "target_max_objects","val": "1"}]: dispatch 2024-04-14T15:17:42.391 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:17:42.416 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:42.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:42.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:17:42.518 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:17:42.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:42.523+0000 7f6ec0440700 1 -- 172.21.15.90:0/3649222591 >> v1:172.21.15.90:6789/0 conn(0x7f6eb8108880 legacy=0x7f6eb8108c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:42.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:42.523+0000 7f6ec0440700 1 -- 172.21.15.90:0/3649222591 shutdown_connections 2024-04-14T15:17:42.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:42.523+0000 7f6ec0440700 1 -- 172.21.15.90:0/3649222591 >> 172.21.15.90:0/3649222591 conn(0x7f6eb80fe3d0 msgr2=0x7f6eb81007f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:17:42.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:42.523+0000 7f6ec0440700 1 -- 172.21.15.90:0/3649222591 shutdown_connections 2024-04-14T15:17:42.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:42.523+0000 7f6ec0440700 1 -- 172.21.15.90:0/3649222591 wait complete. 2024-04-14T15:17:42.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:42.523+0000 7f6ec0440700 1 librados: starting msgr at 2024-04-14T15:17:42.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:42.523+0000 7f6ec0440700 1 librados: starting objecter 2024-04-14T15:17:42.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:42.523+0000 7f6ec0440700 1 Processor -- start 2024-04-14T15:17:42.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:42.523+0000 7f6ec0440700 1 -- start start 2024-04-14T15:17:42.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:42.523+0000 7f6ec0440700 1 librados: setting wanted keys 2024-04-14T15:17:42.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:42.523+0000 7f6ec0440700 1 librados: calling monclient init 2024-04-14T15:17:42.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:42.524+0000 7f6ec0440700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6eb81a9d00 con 0x7f6eb810f300 2024-04-14T15:17:42.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:42.524+0000 7f6ec0440700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6eb81aaee0 con 0x7f6eb8108880 2024-04-14T15:17:42.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:42.524+0000 7f6ec0440700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6eb81ac120 con 0x7f6eb810b730 2024-04-14T15:17:42.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:42.524+0000 7f6ebe9dd700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f6eb810f300 0x7f6eb81a83d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:55328/0 (socket says 172.21.15.90:55328) 2024-04-14T15:17:42.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:42.524+0000 7f6ebe9dd700 1 -- 172.21.15.90:0/3412579979 learned_addr learned my addr 172.21.15.90:0/3412579979 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:17:42.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:42.524+0000 7f6eaf7fe700 10 client.?.objecter ms_handle_connect 0x7f6eb810b730 2024-04-14T15:17:42.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:42.524+0000 7f6eaf7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:17:42.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:42.524+0000 7f6eaf7fe700 10 client.?.objecter ms_handle_connect 0x7f6eb810f300 2024-04-14T15:17:42.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:42.524+0000 7f6eaf7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:17:42.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:42.524+0000 7f6eaf7fe700 10 client.?.objecter ms_handle_connect 0x7f6eb8108880 2024-04-14T15:17:42.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:42.524+0000 7f6eaf7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:17:42.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:42.524+0000 7f6eaf7fe700 1 -- 172.21.15.90:0/3412579979 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2268629142 0 0) 0x7f6eb81a9d00 con 0x7f6eb810f300 2024-04-14T15:17:42.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:42.524+0000 7f6eaf7fe700 1 -- 172.21.15.90:0/3412579979 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ea40035b0 con 0x7f6eb810f300 2024-04-14T15:17:42.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:42.524+0000 7f6eaf7fe700 1 -- 172.21.15.90:0/3412579979 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4121369212 0 0) 0x7f6eb81ac120 con 0x7f6eb810b730 2024-04-14T15:17:42.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:42.524+0000 7f6eaf7fe700 1 -- 172.21.15.90:0/3412579979 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6eb81a9d00 con 0x7f6eb810b730 2024-04-14T15:17:42.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:42.524+0000 7f6eaf7fe700 1 -- 172.21.15.90:0/3412579979 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1936539500 0 0) 0x7f6eb81aaee0 con 0x7f6eb8108880 2024-04-14T15:17:42.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:42.524+0000 7f6eaf7fe700 1 -- 172.21.15.90:0/3412579979 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6eb81ac120 con 0x7f6eb8108880 2024-04-14T15:17:42.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:42.524+0000 7f6eaf7fe700 1 -- 172.21.15.90:0/3412579979 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3115628681 0 0) 0x7f6ea40035b0 con 0x7f6eb810f300 2024-04-14T15:17:42.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:42.524+0000 7f6eaf7fe700 1 -- 172.21.15.90:0/3412579979 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6eb81aaee0 con 0x7f6eb810f300 2024-04-14T15:17:42.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:42.524+0000 7f6eaf7fe700 1 -- 172.21.15.90:0/3412579979 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f6eb0001c30 con 0x7f6eb810f300 2024-04-14T15:17:42.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:42.524+0000 7f6eaf7fe700 1 -- 172.21.15.90:0/3412579979 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3034070117 0 0) 0x7f6eb81ac120 con 0x7f6eb8108880 2024-04-14T15:17:42.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:42.524+0000 7f6eaf7fe700 1 -- 172.21.15.90:0/3412579979 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6ea40035b0 con 0x7f6eb8108880 2024-04-14T15:17:42.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:42.524+0000 7f6eaf7fe700 1 -- 172.21.15.90:0/3412579979 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f6eb4001430 con 0x7f6eb8108880 2024-04-14T15:17:42.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:42.524+0000 7f6eaf7fe700 1 -- 172.21.15.90:0/3412579979 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3662060064 0 0) 0x7f6eb81aaee0 con 0x7f6eb810f300 2024-04-14T15:17:42.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:42.524+0000 7f6eaf7fe700 1 -- 172.21.15.90:0/3412579979 >> v1:172.21.15.90:6790/0 conn(0x7f6eb810b730 legacy=0x7f6eb81a4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:42.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:42.524+0000 7f6eaf7fe700 1 -- 172.21.15.90:0/3412579979 >> v1:172.21.15.92:6789/0 conn(0x7f6eb8108880 legacy=0x7f6eb8076700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:42.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:42.524+0000 7f6eaf7fe700 1 -- 172.21.15.90:0/3412579979 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6eb81ad360 con 0x7f6eb810f300 2024-04-14T15:17:42.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:42.524+0000 7f6ec0440700 1 -- 172.21.15.90:0/3412579979 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6eb81a9f30 con 0x7f6eb810f300 2024-04-14T15:17:42.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:42.525+0000 7f6ec0440700 10 client.37212.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:17:42.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:42.525+0000 7f6ec0440700 1 -- 172.21.15.90:0/3412579979 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f6eb81aa3f0 con 0x7f6eb810f300 2024-04-14T15:17:42.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:42.525+0000 7f6ec0440700 1 librados: init done 2024-04-14T15:17:42.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:42.525+0000 7f6eaf7fe700 1 -- 172.21.15.90:0/3412579979 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f6eb0003ea0 con 0x7f6eb810f300 2024-04-14T15:17:42.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:42.525+0000 7f6eaf7fe700 1 -- 172.21.15.90:0/3412579979 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f6eb0005fb0 con 0x7f6eb810f300 2024-04-14T15:17:42.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:42.526+0000 7f6ec0440700 1 -- 172.21.15.90:0/3412579979 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e9c005150 con 0x7f6eb810f300 2024-04-14T15:17:42.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:42.526+0000 7f6eaf7fe700 1 -- 172.21.15.90:0/3412579979 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f6eb0003a00 con 0x7f6eb810f300 2024-04-14T15:17:42.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:42.527+0000 7f6eaf7fe700 1 -- 172.21.15.90:0/3412579979 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(394..394 src has 51..394) ==== 8612+0+0 (unknown 274710603 0 0) 0x7f6eb008af00 con 0x7f6eb810f300 2024-04-14T15:17:42.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:42.527+0000 7f6eaf7fe700 10 client.37212.objecter ms_dispatch 0x7f6eb8001550 osd_map(394..394 src has 51..394) 2024-04-14T15:17:42.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:42.527+0000 7f6eaf7fe700 3 client.37212.objecter handle_osd_map got epochs [394,394] > 0 2024-04-14T15:17:42.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:42.527+0000 7f6eaf7fe700 3 client.37212.objecter handle_osd_map decoding full epoch 394 2024-04-14T15:17:42.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:42.527+0000 7f6eaf7fe700 20 client.37212.objecter dump_active .. 0 homeless 2024-04-14T15:17:42.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:42.527+0000 7f6eaf7fe700 10 client.37212.objecter ms_handle_connect 0x7f6ea4071030 2024-04-14T15:17:42.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:42.531+0000 7f6eaf7fe700 1 -- 172.21.15.90:0/3412579979 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f6eb0057770 con 0x7f6eb810f300 2024-04-14T15:17:42.544 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:42.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:42.545 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:17:42.645 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:17:42.672 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:42.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:42.673 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:17:42.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:42.749+0000 7f6ec0440700 1 -- 172.21.15.90:0/3412579979 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7f6e9c005440 con 0x7f6eb810f300 2024-04-14T15:17:42.774 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:17:42.798 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:42.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:42.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:17:42.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:42.893+0000 7f6eaf7fe700 1 -- 172.21.15.90:0/3412579979 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v395) ==== 101+0+0 (unknown 853182624 0 0) 0x7f6eb005b420 con 0x7f6eb810f300 2024-04-14T15:17:42.900 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:17:42.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:42.896+0000 7f6ec0440700 10 librados: watch_flush enter 2024-04-14T15:17:42.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:42.896+0000 7f6ec0440700 10 librados: watch_flush exit 2024-04-14T15:17:42.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:42.896+0000 7f6ec0440700 20 client.37212.objecter shutdown clearing up homeless session... 2024-04-14T15:17:42.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:42.896+0000 7f6ec0440700 10 client.37212.objecter successfully canceled tick 2024-04-14T15:17:42.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:42.898+0000 7f6ec0440700 1 -- 172.21.15.90:0/3412579979 >> v1:172.21.15.90:6800/420935427 conn(0x7f6ea4071030 legacy=0x7f6ea40734f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:42.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:42.898+0000 7f6ec0440700 1 -- 172.21.15.90:0/3412579979 >> v1:172.21.15.90:6789/0 conn(0x7f6eb810f300 legacy=0x7f6eb81a83d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:42.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:42.899+0000 7f6ec0440700 1 -- 172.21.15.90:0/3412579979 shutdown_connections 2024-04-14T15:17:42.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:42.899+0000 7f6ec0440700 1 -- 172.21.15.90:0/3412579979 >> 172.21.15.90:0/3412579979 conn(0x7f6eb80fe3d0 msgr2=0x7f6eb8102880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:17:42.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:42.900+0000 7f6ec0440700 1 -- 172.21.15.90:0/3412579979 shutdown_connections 2024-04-14T15:17:42.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:42.900+0000 7f6ec0440700 1 -- 172.21.15.90:0/3412579979 wait complete. 2024-04-14T15:17:42.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:42.900+0000 7f6ec0440700 1 librados: shutdown 2024-04-14T15:17:42.903 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is unset 2024-04-14T15:17:42.925 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:42.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:42.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:17:43.027 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:17:43.056 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:43.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:43.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:17:43.158 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:17:43.182 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:43.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:43.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:17:43.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:42 smithi090 ceph-mon[41670]: pgmap v536: 246 pgs: 8 unknown, 238 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-04-14T15:17:43.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:42 smithi090 ceph-mon[41670]: 1.2 scrub starts 2024-04-14T15:17:43.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:42 smithi090 ceph-mon[41670]: 1.2 scrub ok 2024-04-14T15:17:43.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:42 smithi090 ceph-mon[41670]: 4.17 scrub starts 2024-04-14T15:17:43.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:42 smithi090 ceph-mon[41670]: 4.17 scrub ok 2024-04-14T15:17:43.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:42 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3412579979' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:17:43.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:42 smithi090 ceph-mon[41670]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-14T15:17:43.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:42 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1014922700' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi090-93363-66"}]': finished 2024-04-14T15:17:43.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:42 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-51","var": "target_max_objects","val": "1"}]': finished 2024-04-14T15:17:43.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:42 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3412579979' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-14T15:17:43.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:42 smithi090 ceph-mon[41670]: osdmap e395: 8 total, 8 up, 8 in 2024-04-14T15:17:43.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:42 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1014922700' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi090-93363-66"}]: dispatch 2024-04-14T15:17:43.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:42 smithi090 ceph-mon[30399]: pgmap v536: 246 pgs: 8 unknown, 238 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-04-14T15:17:43.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:42 smithi090 ceph-mon[30399]: 1.2 scrub starts 2024-04-14T15:17:43.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:42 smithi090 ceph-mon[30399]: 1.2 scrub ok 2024-04-14T15:17:43.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:42 smithi090 ceph-mon[30399]: 4.17 scrub starts 2024-04-14T15:17:43.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:42 smithi090 ceph-mon[30399]: 4.17 scrub ok 2024-04-14T15:17:43.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:42 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3412579979' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:17:43.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:42 smithi090 ceph-mon[30399]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-14T15:17:43.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:42 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1014922700' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi090-93363-66"}]': finished 2024-04-14T15:17:43.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:42 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-51","var": "target_max_objects","val": "1"}]': finished 2024-04-14T15:17:43.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:42 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3412579979' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-14T15:17:43.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:42 smithi090 ceph-mon[30399]: osdmap e395: 8 total, 8 up, 8 in 2024-04-14T15:17:43.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:42 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1014922700' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi090-93363-66"}]: dispatch 2024-04-14T15:17:43.284 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:17:43.308 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:43.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:43.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:17:43.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:42 smithi092 ceph-mon[33966]: pgmap v536: 246 pgs: 8 unknown, 238 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-04-14T15:17:43.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:42 smithi092 ceph-mon[33966]: 1.2 scrub starts 2024-04-14T15:17:43.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:42 smithi092 ceph-mon[33966]: 1.2 scrub ok 2024-04-14T15:17:43.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:42 smithi092 ceph-mon[33966]: 4.17 scrub starts 2024-04-14T15:17:43.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:42 smithi092 ceph-mon[33966]: 4.17 scrub ok 2024-04-14T15:17:43.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:42 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3412579979' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:17:43.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:42 smithi092 ceph-mon[33966]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-14T15:17:43.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:42 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1014922700' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi090-93363-66"}]': finished 2024-04-14T15:17:43.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:42 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-51","var": "target_max_objects","val": "1"}]': finished 2024-04-14T15:17:43.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:42 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3412579979' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-14T15:17:43.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:42 smithi092 ceph-mon[33966]: osdmap e395: 8 total, 8 up, 8 in 2024-04-14T15:17:43.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:42 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1014922700' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi090-93363-66"}]: dispatch 2024-04-14T15:17:43.409 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:17:43.436 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:43.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:43.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:17:43.538 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:17:43.562 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:43.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:43.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:17:43.664 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:17:43.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:43.693 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:43.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:17:43.794 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:17:43.822 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:43.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:43.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:17:43.904 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (3010 ms) 2024-04-14T15:17:43.905 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2024-04-14T15:17:43.905 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (6984 ms) 2024-04-14T15:17:43.905 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2024-04-14T15:17:43.905 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (7004 ms) 2024-04-14T15:17:43.905 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2024-04-14T15:17:43.905 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (7008 ms) 2024-04-14T15:17:43.905 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2024-04-14T15:17:43.905 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (7007 ms) 2024-04-14T15:17:43.905 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2024-04-14T15:17:43.905 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (6989 ms) 2024-04-14T15:17:43.905 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2024-04-14T15:17:43.905 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (7009 ms) 2024-04-14T15:17:43.906 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (139024 ms total) 2024-04-14T15:17:43.906 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: 2024-04-14T15:17:43.906 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [----------] Global test environment tear-down 2024-04-14T15:17:43.906 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [==========] 57 tests from 4 test suites ran. (289756 ms total) 2024-04-14T15:17:43.906 INFO:tasks.workunit.client.0.smithi090.stdout: api_aio_pp: [ PASSED ] 57 tests. 2024-04-14T15:17:43.924 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:17:43.948 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:43.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:43.949 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:17:44.050 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:17:44.077 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:44.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:44.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:17:44.179 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:17:44.191 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:17:43 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:17:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:17:44.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:43 smithi090 ceph-mon[30399]: 4.11 scrub starts 2024-04-14T15:17:44.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:43 smithi090 ceph-mon[30399]: 4.11 scrub ok 2024-04-14T15:17:44.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:43 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1014922700' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi090-93363-66"}]': finished 2024-04-14T15:17:44.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:43 smithi090 ceph-mon[30399]: osdmap e396: 8 total, 8 up, 8 in 2024-04-14T15:17:44.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:43 smithi090 ceph-mon[41670]: 4.11 scrub starts 2024-04-14T15:17:44.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:43 smithi090 ceph-mon[41670]: 4.11 scrub ok 2024-04-14T15:17:44.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:43 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1014922700' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi090-93363-66"}]': finished 2024-04-14T15:17:44.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:43 smithi090 ceph-mon[41670]: osdmap e396: 8 total, 8 up, 8 in 2024-04-14T15:17:44.206 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:44.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:44.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:17:44.308 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:17:44.329 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-14T15:17:44.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:44.333 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:44.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:17:44.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:43 smithi092 ceph-mon[33966]: 4.11 scrub starts 2024-04-14T15:17:44.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:43 smithi092 ceph-mon[33966]: 4.11 scrub ok 2024-04-14T15:17:44.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:43 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1014922700' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi090-93363-66"}]': finished 2024-04-14T15:17:44.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:43 smithi092 ceph-mon[33966]: osdmap e396: 8 total, 8 up, 8 in 2024-04-14T15:17:44.434 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:17:44.458 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:44.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:44.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:17:44.560 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:17:44.588 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:44.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:44.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:17:44.689 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:17:44.713 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:44.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:44.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:17:44.815 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:17:44.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:44.839 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:44.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:17:44.940 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:17:44.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:44.964 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:44.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:17:45.065 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:17:45.093 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:45.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:45.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:17:45.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:44 smithi090 ceph-mon[41670]: pgmap v539: 238 pgs: 238 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:17:45.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:44 smithi090 ceph-mon[41670]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-04-14T15:17:45.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:44 smithi090 ceph-mon[41670]: 6.6 deep-scrub starts 2024-04-14T15:17:45.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:44 smithi090 ceph-mon[41670]: 6.6 deep-scrub ok 2024-04-14T15:17:45.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:44 smithi090 ceph-mon[41670]: 11.13 scrub starts 2024-04-14T15:17:45.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:44 smithi090 ceph-mon[41670]: 11.13 scrub ok 2024-04-14T15:17:45.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:44 smithi090 ceph-mon[30399]: pgmap v539: 238 pgs: 238 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:17:45.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:44 smithi090 ceph-mon[30399]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-04-14T15:17:45.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:44 smithi090 ceph-mon[30399]: 6.6 deep-scrub starts 2024-04-14T15:17:45.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:44 smithi090 ceph-mon[30399]: 6.6 deep-scrub ok 2024-04-14T15:17:45.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:44 smithi090 ceph-mon[30399]: 11.13 scrub starts 2024-04-14T15:17:45.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:44 smithi090 ceph-mon[30399]: 11.13 scrub ok 2024-04-14T15:17:45.195 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:17:45.222 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:45.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:45.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:17:45.324 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:17:45.351 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:45.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:45.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:17:45.372 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:17:45 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:17:45.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:44 smithi092 ceph-mon[33966]: pgmap v539: 238 pgs: 238 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:17:45.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:44 smithi092 ceph-mon[33966]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-04-14T15:17:45.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:44 smithi092 ceph-mon[33966]: 6.6 deep-scrub starts 2024-04-14T15:17:45.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:44 smithi092 ceph-mon[33966]: 6.6 deep-scrub ok 2024-04-14T15:17:45.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:44 smithi092 ceph-mon[33966]: 11.13 scrub starts 2024-04-14T15:17:45.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:44 smithi092 ceph-mon[33966]: 11.13 scrub ok 2024-04-14T15:17:45.453 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:17:45.480 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:45.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:45.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:17:45.581 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:17:45.607 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:45.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:45.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:17:45.709 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:17:45.734 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:45.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:45.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:17:45.836 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:17:45.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:45.862 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:45.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:17:45.963 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:17:45.990 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:45.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:45.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:17:46.092 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:17:46.117 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:46.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:46.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:17:46.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:45 smithi090 ceph-mon[30399]: 4.a scrub starts 2024-04-14T15:17:46.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:45 smithi090 ceph-mon[30399]: 4.a scrub ok 2024-04-14T15:17:46.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:45 smithi090 ceph-mon[30399]: 110.13 scrub starts 2024-04-14T15:17:46.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:45 smithi090 ceph-mon[30399]: 110.13 scrub ok 2024-04-14T15:17:46.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:45 smithi090 ceph-mon[30399]: 4.6 scrub starts 2024-04-14T15:17:46.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:45 smithi090 ceph-mon[30399]: 4.6 scrub ok 2024-04-14T15:17:46.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:45 smithi090 ceph-mon[41670]: 4.a scrub starts 2024-04-14T15:17:46.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:45 smithi090 ceph-mon[41670]: 4.a scrub ok 2024-04-14T15:17:46.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:45 smithi090 ceph-mon[41670]: 110.13 scrub starts 2024-04-14T15:17:46.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:45 smithi090 ceph-mon[41670]: 110.13 scrub ok 2024-04-14T15:17:46.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:45 smithi090 ceph-mon[41670]: 4.6 scrub starts 2024-04-14T15:17:46.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:45 smithi090 ceph-mon[41670]: 4.6 scrub ok 2024-04-14T15:17:46.219 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:17:46.244 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:46.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:46.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:17:46.345 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:17:46.369 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:46.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:46.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:17:46.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:45 smithi092 ceph-mon[33966]: 4.a scrub starts 2024-04-14T15:17:46.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:45 smithi092 ceph-mon[33966]: 4.a scrub ok 2024-04-14T15:17:46.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:45 smithi092 ceph-mon[33966]: 110.13 scrub starts 2024-04-14T15:17:46.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:45 smithi092 ceph-mon[33966]: 110.13 scrub ok 2024-04-14T15:17:46.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:45 smithi092 ceph-mon[33966]: 4.6 scrub starts 2024-04-14T15:17:46.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:45 smithi092 ceph-mon[33966]: 4.6 scrub ok 2024-04-14T15:17:46.471 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:17:46.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:46.495 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:46.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:17:46.596 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:17:46.624 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:46.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:46.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:17:46.634 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-14T15:17:46.634 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-14T15:17:46.634 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-14T15:17:46.634 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-14T15:17:46.634 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.634 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24889.0:2 22.15 22:ac175842:test-rados-api-smithi090-93930-2::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-04-14T15:17:46.634 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:12:56.066705+0000", 2024-04-14T15:17:46.634 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 290.395313891, 2024-04-14T15:17:46.634 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.43969079999999999, 2024-04-14T15:17:46.634 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:17:46.635 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:17:46.635 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:17:46.635 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:17:46.635 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24889", 2024-04-14T15:17:46.635 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3182938272", 2024-04-14T15:17:46.635 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2 2024-04-14T15:17:46.635 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.635 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:17:46.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.635 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:17:46.635 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.066705+0000", 2024-04-14T15:17:46.635 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:17:46.635 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.636 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.636 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:17:46.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.066705+0000", 2024-04-14T15:17:46.636 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:17:46.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.636 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.636 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:17:46.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.066708+0000", 2024-04-14T15:17:46.636 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4320000000000002e-06 2024-04-14T15:17:46.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.636 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.636 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:17:46.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.066756+0000", 2024-04-14T15:17:46.637 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7888e-05 2024-04-14T15:17:46.637 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.637 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.637 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:17:46.637 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.066775+0000", 2024-04-14T15:17:46.637 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9391000000000001e-05 2024-04-14T15:17:46.637 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.637 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.637 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:17:46.637 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.066788+0000", 2024-04-14T15:17:46.637 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2519e-05 2024-04-14T15:17:46.637 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.637 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.638 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:17:46.638 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.066860+0000", 2024-04-14T15:17:46.638 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1919e-05 2024-04-14T15:17:46.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.638 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2024-04-14T15:17:46.638 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.066867+0000", 2024-04-14T15:17:46.638 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1130000000000003e-06 2024-04-14T15:17:46.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.638 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:17:46.638 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.145373+0000", 2024-04-14T15:17:46.638 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.078505825000000001 2024-04-14T15:17:46.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.639 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.639 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:17:46.639 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.145383+0000", 2024-04-14T15:17:46.639 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0196e-05 2024-04-14T15:17:46.639 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.639 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.639 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:17:46.639 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.288000+0000", 2024-04-14T15:17:46.639 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.14261763599999999 2024-04-14T15:17:46.639 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.639 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.639 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:17:46.639 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.288166+0000", 2024-04-14T15:17:46.639 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016534000000000001 2024-04-14T15:17:46.639 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.639 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.640 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4,7", 2024-04-14T15:17:46.640 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.288314+0000", 2024-04-14T15:17:46.640 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000148293 2024-04-14T15:17:46.640 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.640 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.640 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:17:46.640 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.337996+0000", 2024-04-14T15:17:46.640 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.049681625 2024-04-14T15:17:46.640 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.640 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.640 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:17:46.640 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.344006+0000", 2024-04-14T15:17:46.640 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0060103960000000003 2024-04-14T15:17:46.640 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.641 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.641 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:17:46.641 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.504829+0000", 2024-04-14T15:17:46.641 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.16082263299999999 2024-04-14T15:17:46.641 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.641 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.641 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:17:46.641 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.504878+0000", 2024-04-14T15:17:46.641 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9257e-05 2024-04-14T15:17:46.641 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.641 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.641 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:17:46.641 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.506396+0000", 2024-04-14T15:17:46.641 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001518337 2024-04-14T15:17:46.641 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:17:46.641 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:17:46.642 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:17:46.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.642 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.642 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24923.0:2 25.6 25:65dba387:test-rados-api-smithi090-94131-3::testobj:head [write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-04-14T15:17:46.642 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:12:56.071392+0000", 2024-04-14T15:17:46.642 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 290.390627623, 2024-04-14T15:17:46.642 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49329716000000001, 2024-04-14T15:17:46.642 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:17:46.642 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:17:46.642 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:17:46.642 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:17:46.642 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24923", 2024-04-14T15:17:46.642 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/4254723003", 2024-04-14T15:17:46.642 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2 2024-04-14T15:17:46.643 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.643 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:17:46.643 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.643 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:17:46.643 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.071392+0000", 2024-04-14T15:17:46.643 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:17:46.643 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.643 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.643 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:17:46.643 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.071392+0000", 2024-04-14T15:17:46.643 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:17:46.643 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.643 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.643 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:17:46.644 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.071394+0000", 2024-04-14T15:17:46.644 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.294e-06 2024-04-14T15:17:46.644 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.644 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.644 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:17:46.644 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.071403+0000", 2024-04-14T15:17:46.644 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8999999999999995e-06 2024-04-14T15:17:46.644 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.644 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.644 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:17:46.644 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.071416+0000", 2024-04-14T15:17:46.644 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3366999999999999e-05 2024-04-14T15:17:46.644 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.645 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.645 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:17:46.645 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.071426+0000", 2024-04-14T15:17:46.645 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5030000000000003e-06 2024-04-14T15:17:46.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.645 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.645 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:17:46.645 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.072369+0000", 2024-04-14T15:17:46.645 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00094297499999999998 2024-04-14T15:17:46.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.645 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.645 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2024-04-14T15:17:46.645 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.072375+0000", 2024-04-14T15:17:46.645 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5400000000000001e-06 2024-04-14T15:17:46.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.646 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.646 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:17:46.646 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.238121+0000", 2024-04-14T15:17:46.646 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.165745485 2024-04-14T15:17:46.646 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.646 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.646 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:17:46.646 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.238129+0000", 2024-04-14T15:17:46.646 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.49e-06 2024-04-14T15:17:46.646 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.647 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.647 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:17:46.647 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.380419+0000", 2024-04-14T15:17:46.647 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.142289742 2024-04-14T15:17:46.647 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.647 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.647 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:17:46.647 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.380539+0000", 2024-04-14T15:17:46.647 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000120221 2024-04-14T15:17:46.647 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.647 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.647 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,5", 2024-04-14T15:17:46.647 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.380682+0000", 2024-04-14T15:17:46.647 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000142924 2024-04-14T15:17:46.647 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.647 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.648 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:17:46.648 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.408898+0000", 2024-04-14T15:17:46.648 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.028216187 2024-04-14T15:17:46.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.648 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.648 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:17:46.648 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.564447+0000", 2024-04-14T15:17:46.648 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15554912700000001 2024-04-14T15:17:46.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.648 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.648 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:17:46.648 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.564550+0000", 2024-04-14T15:17:46.648 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000102725 2024-04-14T15:17:46.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.648 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.648 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:17:46.649 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.564602+0000", 2024-04-14T15:17:46.649 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1891999999999997e-05 2024-04-14T15:17:46.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.649 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:17:46.649 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.564689+0000", 2024-04-14T15:17:46.649 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6787999999999996e-05 2024-04-14T15:17:46.649 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:17:46.649 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:17:46.649 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:17:46.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.649 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24853.0:2 16.1d 16:b9b4aa4b:test-rados-api-smithi090-93601-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-04-14T15:17:46.649 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:12:56.373038+0000", 2024-04-14T15:17:46.649 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 290.08898157700003, 2024-04-14T15:17:46.649 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.193584592, 2024-04-14T15:17:46.650 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:17:46.650 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:17:46.650 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:17:46.650 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:17:46.650 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24853", 2024-04-14T15:17:46.650 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3851923713", 2024-04-14T15:17:46.650 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2 2024-04-14T15:17:46.650 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.650 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:17:46.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.650 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:17:46.650 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.373038+0000", 2024-04-14T15:17:46.650 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:17:46.650 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.651 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:17:46.651 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.373038+0000", 2024-04-14T15:17:46.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:17:46.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.651 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:17:46.651 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.373039+0000", 2024-04-14T15:17:46.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.711e-06 2024-04-14T15:17:46.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.651 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:17:46.651 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.373047+0000", 2024-04-14T15:17:46.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1300000000000003e-06 2024-04-14T15:17:46.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.651 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:17:46.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.373064+0000", 2024-04-14T15:17:46.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7603000000000001e-05 2024-04-14T15:17:46.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:17:46.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.373080+0000", 2024-04-14T15:17:46.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5344000000000001e-05 2024-04-14T15:17:46.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:17:46.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.404205+0000", 2024-04-14T15:17:46.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.03112556 2024-04-14T15:17:46.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:17:46.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.404429+0000", 2024-04-14T15:17:46.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022366199999999999 2024-04-14T15:17:46.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,7", 2024-04-14T15:17:46.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.404619+0000", 2024-04-14T15:17:46.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019027099999999999 2024-04-14T15:17:46.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:17:46.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.406595+0000", 2024-04-14T15:17:46.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0019755699999999998 2024-04-14T15:17:46.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:17:46.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.564357+0000", 2024-04-14T15:17:46.654 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15776261699999999 2024-04-14T15:17:46.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.654 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.654 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:17:46.654 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.566534+0000", 2024-04-14T15:17:46.654 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0021764670000000001 2024-04-14T15:17:46.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.654 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.654 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:17:46.654 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.566583+0000", 2024-04-14T15:17:46.654 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9251e-05 2024-04-14T15:17:46.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.654 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.654 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:17:46.654 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.566622+0000", 2024-04-14T15:17:46.654 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9406000000000002e-05 2024-04-14T15:17:46.655 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:17:46.655 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:17:46.655 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:17:46.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.655 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24860.0:10 21.15 21:abf5dd0a:test-rados-api-smithi090-93864-5::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-04-14T15:17:46.655 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:12:56.544723+0000", 2024-04-14T15:17:46.655 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 289.91729598900002, 2024-04-14T15:17:46.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.049845436, 2024-04-14T15:17:46.655 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:17:46.655 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:17:46.655 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:17:46.655 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:17:46.655 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24860", 2024-04-14T15:17:46.655 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/766142007", 2024-04-14T15:17:46.655 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 10 2024-04-14T15:17:46.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.656 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:17:46.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:17:46.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.544723+0000", 2024-04-14T15:17:46.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:17:46.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:17:46.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.544723+0000", 2024-04-14T15:17:46.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:17:46.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:17:46.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.544725+0000", 2024-04-14T15:17:46.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2359999999999999e-06 2024-04-14T15:17:46.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.657 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:17:46.657 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.544730+0000", 2024-04-14T15:17:46.657 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6439999999999997e-06 2024-04-14T15:17:46.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.657 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:17:46.657 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.544737+0000", 2024-04-14T15:17:46.657 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2509999999999997e-06 2024-04-14T15:17:46.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.657 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:17:46.657 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.544745+0000", 2024-04-14T15:17:46.657 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1029999999999996e-06 2024-04-14T15:17:46.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.658 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.658 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:17:46.658 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.544796+0000", 2024-04-14T15:17:46.658 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1461000000000001e-05 2024-04-14T15:17:46.658 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.658 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.658 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:17:46.658 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.544899+0000", 2024-04-14T15:17:46.658 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000102927 2024-04-14T15:17:46.658 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.658 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.658 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2024-04-14T15:17:46.658 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.545067+0000", 2024-04-14T15:17:46.658 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016825400000000001 2024-04-14T15:17:46.658 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.658 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.659 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:17:46.659 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.546327+0000", 2024-04-14T15:17:46.659 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001259713 2024-04-14T15:17:46.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.659 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:17:46.659 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.546461+0000", 2024-04-14T15:17:46.659 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013432799999999999 2024-04-14T15:17:46.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.659 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:17:46.659 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.594465+0000", 2024-04-14T15:17:46.659 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.048003571000000002 2024-04-14T15:17:46.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.659 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:17:46.660 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.594519+0000", 2024-04-14T15:17:46.660 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4620000000000002e-05 2024-04-14T15:17:46.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.660 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.660 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:17:46.660 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.594569+0000", 2024-04-14T15:17:46.660 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9327999999999999e-05 2024-04-14T15:17:46.660 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:17:46.660 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:17:46.660 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:17:46.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.660 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.660 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24853.0:2059 16.1 16:85b90e3d:test-rados-api-smithi090-93601-16::big:head [writefull 0~25165836] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-04-14T15:17:46.660 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:12:58.834624+0000", 2024-04-14T15:17:46.660 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 287.627395374, 2024-04-14T15:17:46.661 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.072638431000000003, 2024-04-14T15:17:46.661 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:17:46.661 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:17:46.661 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:17:46.661 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:17:46.661 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24853", 2024-04-14T15:17:46.661 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3851923713", 2024-04-14T15:17:46.661 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2059 2024-04-14T15:17:46.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.661 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:17:46.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:17:46.661 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.834624+0000", 2024-04-14T15:17:46.661 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:17:46.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.662 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:17:46.662 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.834624+0000", 2024-04-14T15:17:46.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:17:46.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.662 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:17:46.662 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.834628+0000", 2024-04-14T15:17:46.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0409999999999999e-06 2024-04-14T15:17:46.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.662 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:17:46.662 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.845804+0000", 2024-04-14T15:17:46.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011176303 2024-04-14T15:17:46.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.663 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.663 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:17:46.663 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.845830+0000", 2024-04-14T15:17:46.663 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5939999999999999e-05 2024-04-14T15:17:46.663 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.663 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.663 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:17:46.663 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.845841+0000", 2024-04-14T15:17:46.663 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0954999999999999e-05 2024-04-14T15:17:46.663 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.663 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.663 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:17:46.663 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.845938+0000", 2024-04-14T15:17:46.663 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7158999999999996e-05 2024-04-14T15:17:46.663 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.664 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:17:46.664 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.846060+0000", 2024-04-14T15:17:46.664 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012211700000000001 2024-04-14T15:17:46.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.664 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,6", 2024-04-14T15:17:46.664 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.846167+0000", 2024-04-14T15:17:46.664 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010677 2024-04-14T15:17:46.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.664 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:17:46.664 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.903809+0000", 2024-04-14T15:17:46.664 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.057642081999999997 2024-04-14T15:17:46.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:17:46.665 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.906802+0000", 2024-04-14T15:17:46.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002992279 2024-04-14T15:17:46.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:17:46.665 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.907187+0000", 2024-04-14T15:17:46.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038566799999999998 2024-04-14T15:17:46.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:17:46.665 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.907226+0000", 2024-04-14T15:17:46.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9014999999999998e-05 2024-04-14T15:17:46.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.666 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:17:46.666 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.907262+0000", 2024-04-14T15:17:46.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6102000000000001e-05 2024-04-14T15:17:46.666 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:17:46.666 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:17:46.669 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:17:46.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.669 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24853.0:2064 16.e 16:722084bd:test-rados-api-smithi090-93601-16::big.copy2:head [copy-from ver 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-04-14T15:17:46.669 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:12:59.026890+0000", 2024-04-14T15:17:46.670 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 287.43512923899999, 2024-04-14T15:17:46.670 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.178520341, 2024-04-14T15:17:46.670 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:17:46.670 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:17:46.670 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:17:46.670 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:17:46.670 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24853", 2024-04-14T15:17:46.670 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3851923713", 2024-04-14T15:17:46.670 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2064 2024-04-14T15:17:46.670 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.670 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:17:46.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.670 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:17:46.671 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.026890+0000", 2024-04-14T15:17:46.671 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:17:46.671 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.671 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:17:46.671 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.026890+0000", 2024-04-14T15:17:46.671 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:17:46.671 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.671 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:17:46.671 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.026891+0000", 2024-04-14T15:17:46.671 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6599999999999994e-07 2024-04-14T15:17:46.671 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.671 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:17:46.671 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.026897+0000", 2024-04-14T15:17:46.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6520000000000003e-06 2024-04-14T15:17:46.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.672 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:17:46.672 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.026904+0000", 2024-04-14T15:17:46.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1169999999999998e-06 2024-04-14T15:17:46.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.672 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:17:46.672 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.026909+0000", 2024-04-14T15:17:46.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0590000000000002e-06 2024-04-14T15:17:46.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.672 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:17:46.672 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.026951+0000", 2024-04-14T15:17:46.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1758999999999998e-05 2024-04-14T15:17:46.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.673 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:17:46.673 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.027040+0000", 2024-04-14T15:17:46.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9747999999999995e-05 2024-04-14T15:17:46.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.673 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2024-04-14T15:17:46.673 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.090575+0000", 2024-04-14T15:17:46.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.063534532000000005 2024-04-14T15:17:46.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.673 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:17:46.673 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.102440+0000", 2024-04-14T15:17:46.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0118651 2024-04-14T15:17:46.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:17:46.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.102518+0000", 2024-04-14T15:17:46.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7830000000000005e-05 2024-04-14T15:17:46.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:17:46.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.205327+0000", 2024-04-14T15:17:46.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10280921 2024-04-14T15:17:46.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:17:46.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.205373+0000", 2024-04-14T15:17:46.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6254999999999997e-05 2024-04-14T15:17:46.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.675 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:17:46.675 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.205410+0000", 2024-04-14T15:17:46.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7113000000000002e-05 2024-04-14T15:17:46.675 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:17:46.675 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:17:46.675 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:17:46.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.675 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25336.0:2 65.c 65:368a68c5:::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [writefull 0~7] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-14T15:17:46.675 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:12:59.135291+0000", 2024-04-14T15:17:46.675 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 287.326728694, 2024-04-14T15:17:46.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.183077902, 2024-04-14T15:17:46.675 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:17:46.676 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:17:46.676 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:17:46.676 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:17:46.676 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25336", 2024-04-14T15:17:46.676 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1239204013", 2024-04-14T15:17:46.676 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2 2024-04-14T15:17:46.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.676 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:17:46.676 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.676 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:17:46.676 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.135291+0000", 2024-04-14T15:17:46.676 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:17:46.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.676 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.676 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:17:46.676 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.135291+0000", 2024-04-14T15:17:46.676 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:17:46.677 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.677 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.677 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:17:46.677 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.135293+0000", 2024-04-14T15:17:46.677 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6390000000000001e-06 2024-04-14T15:17:46.677 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.677 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.677 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:17:46.677 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.135305+0000", 2024-04-14T15:17:46.677 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1826e-05 2024-04-14T15:17:46.677 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.677 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.677 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:17:46.677 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.159193+0000", 2024-04-14T15:17:46.677 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.023888105999999999 2024-04-14T15:17:46.677 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.678 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:17:46.678 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.159208+0000", 2024-04-14T15:17:46.678 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4513e-05 2024-04-14T15:17:46.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.678 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:17:46.678 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.241684+0000", 2024-04-14T15:17:46.678 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.082476771000000004 2024-04-14T15:17:46.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.678 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:17:46.678 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.241693+0000", 2024-04-14T15:17:46.678 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7229999999999993e-06 2024-04-14T15:17:46.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.679 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:17:46.679 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.273020+0000", 2024-04-14T15:17:46.679 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.031326838 2024-04-14T15:17:46.679 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.679 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:17:46.679 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.273177+0000", 2024-04-14T15:17:46.679 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015696899999999999 2024-04-14T15:17:46.679 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.679 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,7", 2024-04-14T15:17:46.679 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.273346+0000", 2024-04-14T15:17:46.679 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016890699999999999 2024-04-14T15:17:46.679 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.679 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:17:46.680 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.288442+0000", 2024-04-14T15:17:46.680 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015095844000000001 2024-04-14T15:17:46.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.680 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.680 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:17:46.680 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.289944+0000", 2024-04-14T15:17:46.680 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00150186 2024-04-14T15:17:46.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.680 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.680 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:17:46.680 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.318262+0000", 2024-04-14T15:17:46.680 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.028318073999999999 2024-04-14T15:17:46.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.680 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.680 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:17:46.680 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.318324+0000", 2024-04-14T15:17:46.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2242000000000005e-05 2024-04-14T15:17:46.681 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.681 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.681 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:17:46.681 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.318369+0000", 2024-04-14T15:17:46.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4589999999999998e-05 2024-04-14T15:17:46.681 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:17:46.681 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:17:46.681 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:17:46.681 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.681 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.681 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24853.0:2069 16.e 16:70021a00:test-rados-api-smithi090-93601-17::big:head [writefull 0~25165828] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-14T15:17:46.681 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:12:59.365160+0000", 2024-04-14T15:17:46.681 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 287.09685981799998, 2024-04-14T15:17:46.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.069670120000000002, 2024-04-14T15:17:46.681 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:17:46.682 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:17:46.682 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:17:46.682 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:17:46.682 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24853", 2024-04-14T15:17:46.682 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3851923713", 2024-04-14T15:17:46.682 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2069 2024-04-14T15:17:46.682 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.682 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:17:46.682 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.682 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:17:46.682 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.365160+0000", 2024-04-14T15:17:46.682 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:17:46.682 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.682 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.682 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:17:46.683 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.365160+0000", 2024-04-14T15:17:46.683 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:17:46.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.683 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.683 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:17:46.683 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.365162+0000", 2024-04-14T15:17:46.683 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1009999999999999e-06 2024-04-14T15:17:46.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.683 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.683 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:17:46.683 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.377376+0000", 2024-04-14T15:17:46.683 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012214849 2024-04-14T15:17:46.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.683 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.683 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:17:46.683 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.377400+0000", 2024-04-14T15:17:46.684 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3288000000000001e-05 2024-04-14T15:17:46.684 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.684 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.684 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:17:46.684 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.377410+0000", 2024-04-14T15:17:46.684 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0678e-05 2024-04-14T15:17:46.684 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.684 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.684 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:17:46.684 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.377478+0000", 2024-04-14T15:17:46.684 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7850999999999998e-05 2024-04-14T15:17:46.684 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.684 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.684 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:17:46.684 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.377615+0000", 2024-04-14T15:17:46.684 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000136907 2024-04-14T15:17:46.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.685 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2024-04-14T15:17:46.685 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.377710+0000", 2024-04-14T15:17:46.685 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5137999999999996e-05 2024-04-14T15:17:46.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.685 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:17:46.685 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.428359+0000", 2024-04-14T15:17:46.685 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.050648276999999998 2024-04-14T15:17:46.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.685 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:17:46.685 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.434675+0000", 2024-04-14T15:17:46.685 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0063159740000000002 2024-04-14T15:17:46.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.686 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.686 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:17:46.686 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.434736+0000", 2024-04-14T15:17:46.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.101e-05 2024-04-14T15:17:46.686 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.686 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.686 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:17:46.686 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.434787+0000", 2024-04-14T15:17:46.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1093e-05 2024-04-14T15:17:46.686 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.686 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.686 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:17:46.686 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.434830+0000", 2024-04-14T15:17:46.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2954000000000002e-05 2024-04-14T15:17:46.686 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:17:46.687 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:17:46.687 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:17:46.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.687 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25267.0:1 69.3s0 69:d43101ce:test-rados-api-smithi090-93864-8::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-04-14T15:17:46.687 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:13:00.272277+0000", 2024-04-14T15:17:46.687 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 286.18974225199997, 2024-04-14T15:17:46.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.68897849099999997, 2024-04-14T15:17:46.687 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:17:46.687 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:17:46.687 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:17:46.687 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:17:46.687 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25267", 2024-04-14T15:17:46.687 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1324117929", 2024-04-14T15:17:46.687 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1 2024-04-14T15:17:46.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.688 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:17:46.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.688 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:17:46.688 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.272277+0000", 2024-04-14T15:17:46.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:17:46.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.688 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:17:46.688 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.272277+0000", 2024-04-14T15:17:46.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:17:46.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.688 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:17:46.688 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.272280+0000", 2024-04-14T15:17:46.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6060000000000001e-06 2024-04-14T15:17:46.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:17:46.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.272289+0000", 2024-04-14T15:17:46.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3619999999999995e-06 2024-04-14T15:17:46.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:17:46.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.272299+0000", 2024-04-14T15:17:46.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6579999999999997e-06 2024-04-14T15:17:46.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:17:46.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.272309+0000", 2024-04-14T15:17:46.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0550999999999999e-05 2024-04-14T15:17:46.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.690 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:17:46.690 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.294275+0000", 2024-04-14T15:17:46.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.021965506999999999 2024-04-14T15:17:46.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.690 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:17:46.690 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.294279+0000", 2024-04-14T15:17:46.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8089999999999999e-06 2024-04-14T15:17:46.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.690 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:17:46.690 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.938589+0000", 2024-04-14T15:17:46.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.64431073299999997 2024-04-14T15:17:46.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.691 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:17:46.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.941384+0000", 2024-04-14T15:17:46.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002795021 2024-04-14T15:17:46.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.691 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2024-04-14T15:17:46.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.941783+0000", 2024-04-14T15:17:46.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039845599999999999 2024-04-14T15:17:46.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.691 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2024-04-14T15:17:46.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.955583+0000", 2024-04-14T15:17:46.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013800415 2024-04-14T15:17:46.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.692 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:17:46.692 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.961184+0000", 2024-04-14T15:17:46.692 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0056007729999999999 2024-04-14T15:17:46.692 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.692 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.692 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:17:46.692 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.961256+0000", 2024-04-14T15:17:46.692 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1600000000000006e-05 2024-04-14T15:17:46.692 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:17:46.692 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:17:46.692 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:17:46.692 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.692 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.692 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.41473.0:2 150.1b 150:d96a01e0:::testobj:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-04-14T15:17:46.692 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:13:24.876020+0000", 2024-04-14T15:17:46.692 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 261.58599958799999, 2024-04-14T15:17:46.693 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.107669923, 2024-04-14T15:17:46.693 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:17:46.693 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:17:46.693 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:17:46.693 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:17:46.693 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.41473", 2024-04-14T15:17:46.693 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1386986747", 2024-04-14T15:17:46.693 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2 2024-04-14T15:17:46.693 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.693 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:17:46.693 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.693 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:17:46.693 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:24.876020+0000", 2024-04-14T15:17:46.693 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:17:46.693 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.693 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.693 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:17:46.694 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:24.876020+0000", 2024-04-14T15:17:46.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:17:46.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.694 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.694 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:17:46.694 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:24.876022+0000", 2024-04-14T15:17:46.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8840000000000001e-06 2024-04-14T15:17:46.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.694 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.694 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:17:46.694 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:24.876030+0000", 2024-04-14T15:17:46.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9740000000000002e-06 2024-04-14T15:17:46.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.694 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.694 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:17:46.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:24.876041+0000", 2024-04-14T15:17:46.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.112e-05 2024-04-14T15:17:46.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:17:46.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:24.876052+0000", 2024-04-14T15:17:46.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0798999999999999e-05 2024-04-14T15:17:46.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:17:46.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:24.877471+0000", 2024-04-14T15:17:46.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0014197459999999999 2024-04-14T15:17:46.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:17:46.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:24.877480+0000", 2024-04-14T15:17:46.696 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.02e-06 2024-04-14T15:17:46.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.696 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.696 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:17:46.696 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:25.965298+0000", 2024-04-14T15:17:46.696 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0878175809999999 2024-04-14T15:17:46.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.696 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.696 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:17:46.696 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:25.965416+0000", 2024-04-14T15:17:46.696 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000118453 2024-04-14T15:17:46.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.696 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.696 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2024-04-14T15:17:46.696 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:25.965546+0000", 2024-04-14T15:17:46.696 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012987200000000001 2024-04-14T15:17:46.697 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.697 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.697 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:17:46.697 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:25.969566+0000", 2024-04-14T15:17:46.697 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040198910000000003 2024-04-14T15:17:46.697 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.697 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.697 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:17:46.697 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:25.980352+0000", 2024-04-14T15:17:46.697 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010785846 2024-04-14T15:17:46.697 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.697 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.697 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:17:46.697 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:25.983575+0000", 2024-04-14T15:17:46.697 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032233140000000001 2024-04-14T15:17:46.698 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.698 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.698 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:17:46.698 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:25.983636+0000", 2024-04-14T15:17:46.698 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0291000000000003e-05 2024-04-14T15:17:46.698 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.698 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.698 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:17:46.698 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:25.983690+0000", 2024-04-14T15:17:46.698 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4132000000000003e-05 2024-04-14T15:17:46.698 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:17:46.698 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:17:46.698 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:17:46.698 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.699 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25684.0:248 110.1c 110:3d310c86:test-rados-api-smithi090-93652-12::bar:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-04-14T15:17:46.699 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:13:26.901575+0000", 2024-04-14T15:17:46.699 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 259.56044470000001, 2024-04-14T15:17:46.699 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.096119394999999996, 2024-04-14T15:17:46.699 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:17:46.699 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:17:46.699 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:17:46.699 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:17:46.699 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25684", 2024-04-14T15:17:46.699 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1109245683", 2024-04-14T15:17:46.699 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 248 2024-04-14T15:17:46.699 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.699 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:17:46.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.700 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:17:46.700 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.901575+0000", 2024-04-14T15:17:46.700 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:17:46.700 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.700 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.700 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:17:46.700 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.901575+0000", 2024-04-14T15:17:46.700 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:17:46.700 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.700 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.700 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:17:46.700 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.901576+0000", 2024-04-14T15:17:46.700 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.542e-06 2024-04-14T15:17:46.700 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.700 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.701 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:17:46.701 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.901583+0000", 2024-04-14T15:17:46.701 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6869999999999997e-06 2024-04-14T15:17:46.701 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.701 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.701 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:17:46.701 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.901590+0000", 2024-04-14T15:17:46.701 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6199999999999999e-06 2024-04-14T15:17:46.701 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.701 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.701 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:17:46.701 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.901601+0000", 2024-04-14T15:17:46.701 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0037999999999999e-05 2024-04-14T15:17:46.701 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.701 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.702 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:17:46.702 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.903042+0000", 2024-04-14T15:17:46.702 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001440993 2024-04-14T15:17:46.702 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.702 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.702 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:17:46.702 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.903165+0000", 2024-04-14T15:17:46.702 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000123201 2024-04-14T15:17:46.702 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.702 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.702 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,3", 2024-04-14T15:17:46.702 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.903317+0000", 2024-04-14T15:17:46.702 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015206799999999999 2024-04-14T15:17:46.703 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.703 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.703 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:17:46.703 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.910951+0000", 2024-04-14T15:17:46.703 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0076343510000000002 2024-04-14T15:17:46.703 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.703 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.703 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:17:46.703 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.974802+0000", 2024-04-14T15:17:46.703 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.063850738000000004 2024-04-14T15:17:46.703 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.703 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.703 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:17:46.703 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.997592+0000", 2024-04-14T15:17:46.703 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022790299 2024-04-14T15:17:46.703 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.704 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.704 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:17:46.704 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.997648+0000", 2024-04-14T15:17:46.704 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5745999999999998e-05 2024-04-14T15:17:46.704 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.704 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.704 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:17:46.704 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.997694+0000", 2024-04-14T15:17:46.704 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6112000000000002e-05 2024-04-14T15:17:46.704 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:17:46.704 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:17:46.704 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:17:46.704 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.704 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.704 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24853.0:2078 16.1 16:81eb58c7:test-rados-api-smithi090-93601-17::big.copy:head [copy-from ver 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-14T15:17:46.705 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:13:29.666343+0000", 2024-04-14T15:17:46.705 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 256.795675948, 2024-04-14T15:17:46.705 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10434288999999999, 2024-04-14T15:17:46.705 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:17:46.705 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:17:46.705 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:17:46.705 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:17:46.705 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24853", 2024-04-14T15:17:46.705 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3851923713", 2024-04-14T15:17:46.705 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2078 2024-04-14T15:17:46.705 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.705 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:17:46.705 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.705 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:17:46.705 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.666343+0000", 2024-04-14T15:17:46.705 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:17:46.706 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.706 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.706 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:17:46.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.666343+0000", 2024-04-14T15:17:46.706 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:17:46.706 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.706 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.706 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:17:46.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.666347+0000", 2024-04-14T15:17:46.706 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0500000000000002e-06 2024-04-14T15:17:46.706 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.706 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.706 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:17:46.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.666363+0000", 2024-04-14T15:17:46.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5131e-05 2024-04-14T15:17:46.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:17:46.707 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.666388+0000", 2024-04-14T15:17:46.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5803e-05 2024-04-14T15:17:46.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:17:46.707 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.666403+0000", 2024-04-14T15:17:46.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5013e-05 2024-04-14T15:17:46.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:17:46.707 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.666497+0000", 2024-04-14T15:17:46.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3468999999999996e-05 2024-04-14T15:17:46.708 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.708 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.708 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:17:46.708 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.666664+0000", 2024-04-14T15:17:46.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016748699999999999 2024-04-14T15:17:46.708 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.708 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.708 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,6", 2024-04-14T15:17:46.708 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.755301+0000", 2024-04-14T15:17:46.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.088636912999999998 2024-04-14T15:17:46.708 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.708 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.708 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:17:46.708 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.770454+0000", 2024-04-14T15:17:46.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015152528 2024-04-14T15:17:46.708 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.709 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:17:46.709 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.770527+0000", 2024-04-14T15:17:46.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3238999999999995e-05 2024-04-14T15:17:46.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.709 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:17:46.709 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.770597+0000", 2024-04-14T15:17:46.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0469999999999994e-05 2024-04-14T15:17:46.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.709 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:17:46.709 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.770649+0000", 2024-04-14T15:17:46.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1309e-05 2024-04-14T15:17:46.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.710 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:17:46.710 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.770686+0000", 2024-04-14T15:17:46.710 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7478000000000003e-05 2024-04-14T15:17:46.710 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:17:46.710 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:17:46.710 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:17:46.710 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.710 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.26497.0:57 112.1a 112:5978e4a5:test-rados-api-smithi090-93724-12::foo:head [write 0~128] snapc b=[b] ondisk+write+ordersnap+known_if_redirected+supports_pool_eio e142)", 2024-04-14T15:17:46.710 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:13:29.880875+0000", 2024-04-14T15:17:46.710 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 256.58114435700003, 2024-04-14T15:17:46.710 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13447140699999999, 2024-04-14T15:17:46.710 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:17:46.711 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:17:46.711 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:17:46.711 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:17:46.711 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.26497", 2024-04-14T15:17:46.711 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3962832909", 2024-04-14T15:17:46.711 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 57 2024-04-14T15:17:46.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.711 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:17:46.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.711 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:17:46.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.880875+0000", 2024-04-14T15:17:46.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:17:46.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:17:46.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.880875+0000", 2024-04-14T15:17:46.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:17:46.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:17:46.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.880877+0000", 2024-04-14T15:17:46.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.776e-06 2024-04-14T15:17:46.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:17:46.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.880885+0000", 2024-04-14T15:17:46.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1130000000000001e-06 2024-04-14T15:17:46.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:17:46.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.880896+0000", 2024-04-14T15:17:46.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0868e-05 2024-04-14T15:17:46.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:17:46.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.880907+0000", 2024-04-14T15:17:46.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1119e-05 2024-04-14T15:17:46.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:17:46.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.880980+0000", 2024-04-14T15:17:46.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3484000000000003e-05 2024-04-14T15:17:46.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2024-04-14T15:17:46.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.880996+0000", 2024-04-14T15:17:46.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6120000000000002e-05 2024-04-14T15:17:46.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:17:46.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.900278+0000", 2024-04-14T15:17:46.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019281436999999998 2024-04-14T15:17:46.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:17:46.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.900405+0000", 2024-04-14T15:17:46.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000127463 2024-04-14T15:17:46.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,6", 2024-04-14T15:17:46.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.900555+0000", 2024-04-14T15:17:46.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014966799999999999 2024-04-14T15:17:46.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:17:46.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.928367+0000", 2024-04-14T15:17:46.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02781177 2024-04-14T15:17:46.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:17:46.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.930228+0000", 2024-04-14T15:17:46.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0018616939999999999 2024-04-14T15:17:46.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:17:46.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:30.015227+0000", 2024-04-14T15:17:46.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.084998517999999995 2024-04-14T15:17:46.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:17:46.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:30.015290+0000", 2024-04-14T15:17:46.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3338999999999998e-05 2024-04-14T15:17:46.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:17:46.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:30.015346+0000", 2024-04-14T15:17:46.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6038000000000001e-05 2024-04-14T15:17:46.717 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:17:46.717 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:17:46.717 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:17:46.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.717 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24853.0:2080 16.e 16:770790e7:test-rados-api-smithi090-93601-17::big3.copy:head [copy-from ver 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-14T15:17:46.717 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:13:29.886267+0000", 2024-04-14T15:17:46.717 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 256.575752228, 2024-04-14T15:17:46.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.25823401099999999, 2024-04-14T15:17:46.717 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:17:46.717 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:17:46.717 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:17:46.717 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:17:46.717 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24853", 2024-04-14T15:17:46.717 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3851923713", 2024-04-14T15:17:46.718 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2080 2024-04-14T15:17:46.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.718 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:17:46.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:17:46.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.886267+0000", 2024-04-14T15:17:46.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:17:46.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:17:46.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.886267+0000", 2024-04-14T15:17:46.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:17:46.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:17:46.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.886269+0000", 2024-04-14T15:17:46.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0820000000000001e-06 2024-04-14T15:17:46.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:17:46.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.886278+0000", 2024-04-14T15:17:46.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0550000000000005e-06 2024-04-14T15:17:46.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:17:46.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.890009+0000", 2024-04-14T15:17:46.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037304180000000001 2024-04-14T15:17:46.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.720 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:17:46.720 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.890026+0000", 2024-04-14T15:17:46.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7244999999999998e-05 2024-04-14T15:17:46.720 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.720 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:17:46.720 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.890095+0000", 2024-04-14T15:17:46.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8621000000000006e-05 2024-04-14T15:17:46.720 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.720 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:17:46.720 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.890254+0000", 2024-04-14T15:17:46.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015991900000000001 2024-04-14T15:17:46.720 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.721 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2024-04-14T15:17:46.721 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:30.073652+0000", 2024-04-14T15:17:46.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.183397687 2024-04-14T15:17:46.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.721 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:17:46.721 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:30.085273+0000", 2024-04-14T15:17:46.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011620966 2024-04-14T15:17:46.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.721 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:17:46.721 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:30.088107+0000", 2024-04-14T15:17:46.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028342459999999999 2024-04-14T15:17:46.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.721 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:17:46.722 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:30.144398+0000", 2024-04-14T15:17:46.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.056290426999999997 2024-04-14T15:17:46.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.722 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:17:46.722 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:30.144458+0000", 2024-04-14T15:17:46.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0164999999999997e-05 2024-04-14T15:17:46.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.722 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:17:46.722 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:30.144501+0000", 2024-04-14T15:17:46.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3180000000000003e-05 2024-04-14T15:17:46.722 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:17:46.722 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:17:46.723 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:17:46.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.723 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.46400.0:2 202.11 202:8de73fe4:test-rados-api-smithi090-93601-26::foo:head [writefull 0~8388608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e184)", 2024-04-14T15:17:46.723 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:14:11.840817+0000", 2024-04-14T15:17:46.723 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 214.621202393, 2024-04-14T15:17:46.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.052008092999999998, 2024-04-14T15:17:46.723 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:17:46.723 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:17:46.723 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:17:46.723 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:17:46.723 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.46400", 2024-04-14T15:17:46.723 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/844371530", 2024-04-14T15:17:46.723 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2 2024-04-14T15:17:46.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.724 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:17:46.724 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.724 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:17:46.724 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:11.840817+0000", 2024-04-14T15:17:46.724 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:17:46.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.724 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.724 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:17:46.724 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:11.840817+0000", 2024-04-14T15:17:46.724 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:17:46.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.724 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.724 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:17:46.724 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:11.840821+0000", 2024-04-14T15:17:46.724 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8920000000000002e-06 2024-04-14T15:17:46.725 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.725 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:17:46.725 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:11.847066+0000", 2024-04-14T15:17:46.725 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0062451310000000001 2024-04-14T15:17:46.725 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.725 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:17:46.725 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:11.847083+0000", 2024-04-14T15:17:46.725 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7419e-05 2024-04-14T15:17:46.725 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.725 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:17:46.726 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:11.847095+0000", 2024-04-14T15:17:46.726 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1341e-05 2024-04-14T15:17:46.726 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.726 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.726 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:17:46.726 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:11.848854+0000", 2024-04-14T15:17:46.726 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001759755 2024-04-14T15:17:46.726 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.726 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.726 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:17:46.726 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:11.848974+0000", 2024-04-14T15:17:46.726 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012000100000000001 2024-04-14T15:17:46.726 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.726 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.726 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 5,7", 2024-04-14T15:17:46.727 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:11.849110+0000", 2024-04-14T15:17:46.727 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013593399999999999 2024-04-14T15:17:46.727 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.727 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.727 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:17:46.727 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:11.866374+0000", 2024-04-14T15:17:46.727 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017263341000000001 2024-04-14T15:17:46.727 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.727 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.727 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:17:46.727 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:11.892695+0000", 2024-04-14T15:17:46.727 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.026321297 2024-04-14T15:17:46.727 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.727 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.727 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:17:46.728 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:11.892733+0000", 2024-04-14T15:17:46.728 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8220000000000003e-05 2024-04-14T15:17:46.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.728 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.728 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:17:46.728 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:11.892771+0000", 2024-04-14T15:17:46.728 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8169e-05 2024-04-14T15:17:46.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.728 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.728 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:17:46.728 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:11.892825+0000", 2024-04-14T15:17:46.728 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3593e-05 2024-04-14T15:17:46.728 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:17:46.728 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:17:46.728 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:17:46.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.729 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.729 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.36564.0:1 217.2s0 217:47f32b6d:test-rados-api-smithi090-93601-37::foo:head [write 0~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-04-14T15:17:46.729 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:14:27.854266+0000", 2024-04-14T15:17:46.729 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 198.60775372200001, 2024-04-14T15:17:46.729 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.007350245, 2024-04-14T15:17:46.729 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:17:46.729 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:17:46.729 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:17:46.729 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:17:46.729 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.36564", 2024-04-14T15:17:46.729 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3214396702", 2024-04-14T15:17:46.729 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1 2024-04-14T15:17:46.729 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.729 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:17:46.729 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.730 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:17:46.730 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:27.854266+0000", 2024-04-14T15:17:46.730 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:17:46.730 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.730 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.730 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:17:46.730 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:27.854266+0000", 2024-04-14T15:17:46.730 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:17:46.730 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.730 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.730 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:17:46.730 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:27.854268+0000", 2024-04-14T15:17:46.730 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1090000000000001e-06 2024-04-14T15:17:46.730 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.730 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.731 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:17:46.731 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:27.854277+0000", 2024-04-14T15:17:46.731 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9639999999999992e-06 2024-04-14T15:17:46.731 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.731 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.731 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:17:46.731 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:27.854288+0000", 2024-04-14T15:17:46.731 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1589e-05 2024-04-14T15:17:46.731 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.731 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.731 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:17:46.731 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:27.854297+0000", 2024-04-14T15:17:46.731 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1370000000000008e-06 2024-04-14T15:17:46.732 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.732 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.732 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:17:46.732 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:27.854384+0000", 2024-04-14T15:17:46.732 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6551999999999994e-05 2024-04-14T15:17:46.732 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.732 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.732 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:17:46.732 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:27.854388+0000", 2024-04-14T15:17:46.732 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6559999999999998e-06 2024-04-14T15:17:46.732 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.732 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.732 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:17:46.733 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:28.850918+0000", 2024-04-14T15:17:46.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99653077300000004 2024-04-14T15:17:46.733 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.733 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.733 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:17:46.733 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:28.851047+0000", 2024-04-14T15:17:46.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000128508 2024-04-14T15:17:46.733 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.733 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.733 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2024-04-14T15:17:46.733 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:28.851452+0000", 2024-04-14T15:17:46.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00040467600000000003 2024-04-14T15:17:46.733 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.733 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.733 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2024-04-14T15:17:46.733 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:28.852774+0000", 2024-04-14T15:17:46.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0013221179999999999 2024-04-14T15:17:46.734 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.734 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.734 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:17:46.734 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:28.861533+0000", 2024-04-14T15:17:46.734 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0087598079999999991 2024-04-14T15:17:46.734 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.734 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.734 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:17:46.734 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:28.861616+0000", 2024-04-14T15:17:46.734 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2354999999999993e-05 2024-04-14T15:17:46.734 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:17:46.734 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:17:46.734 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:17:46.734 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.734 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.735 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.48424.0:1 219.6s0 219:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-04-14T15:17:46.735 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:14:28.846506+0000", 2024-04-14T15:17:46.735 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 197.61551379100001, 2024-04-14T15:17:46.735 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0258293709999999, 2024-04-14T15:17:46.735 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:17:46.735 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:17:46.735 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:17:46.735 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:17:46.735 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.48424", 2024-04-14T15:17:46.735 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2359757960", 2024-04-14T15:17:46.735 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1 2024-04-14T15:17:46.735 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.735 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:17:46.735 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.736 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:17:46.736 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:28.846506+0000", 2024-04-14T15:17:46.736 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:17:46.736 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.736 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.736 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:17:46.736 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:28.846506+0000", 2024-04-14T15:17:46.736 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:17:46.736 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.736 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.736 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:17:46.736 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:28.846507+0000", 2024-04-14T15:17:46.736 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6160000000000001e-06 2024-04-14T15:17:46.736 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.736 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.737 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:17:46.737 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:28.846512+0000", 2024-04-14T15:17:46.737 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.22e-06 2024-04-14T15:17:46.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.737 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.737 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:17:46.737 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:28.846521+0000", 2024-04-14T15:17:46.737 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0089999999999996e-06 2024-04-14T15:17:46.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.737 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.737 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:17:46.737 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:28.846532+0000", 2024-04-14T15:17:46.737 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0229e-05 2024-04-14T15:17:46.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.737 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.737 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:17:46.738 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:28.846598+0000", 2024-04-14T15:17:46.738 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6029000000000001e-05 2024-04-14T15:17:46.738 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.738 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.738 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:17:46.738 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:28.846601+0000", 2024-04-14T15:17:46.738 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2349999999999999e-06 2024-04-14T15:17:46.738 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.738 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.738 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:17:46.738 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:29.869910+0000", 2024-04-14T15:17:46.738 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.023308635 2024-04-14T15:17:46.738 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.738 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.738 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:17:46.739 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:29.870033+0000", 2024-04-14T15:17:46.739 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000123795 2024-04-14T15:17:46.739 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.739 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.739 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2024-04-14T15:17:46.739 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:29.870480+0000", 2024-04-14T15:17:46.739 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00044620199999999998 2024-04-14T15:17:46.739 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.739 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.739 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2024-04-14T15:17:46.739 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:29.871280+0000", 2024-04-14T15:17:46.739 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00080055000000000002 2024-04-14T15:17:46.739 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.739 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.739 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:17:46.739 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:29.872259+0000", 2024-04-14T15:17:46.740 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00097881199999999991 2024-04-14T15:17:46.740 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.740 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.740 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:17:46.740 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:29.872335+0000", 2024-04-14T15:17:46.740 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6038999999999995e-05 2024-04-14T15:17:46.740 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:17:46.740 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:17:46.740 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:17:46.740 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.740 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.740 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.48793.0:1 269.0s0 269:1e0071ee:::oid_RoundTripSparseReadPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-04-14T15:17:46.740 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:16:05.868323+0000", 2024-04-14T15:17:46.740 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 100.59369673400001, 2024-04-14T15:17:46.740 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.007858315, 2024-04-14T15:17:46.740 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:17:46.741 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:17:46.741 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:17:46.741 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:17:46.741 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.48793", 2024-04-14T15:17:46.741 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3028951022", 2024-04-14T15:17:46.741 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1 2024-04-14T15:17:46.741 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.741 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:17:46.741 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.741 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:17:46.741 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:16:05.868323+0000", 2024-04-14T15:17:46.741 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:17:46.741 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.741 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.741 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:17:46.742 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:16:05.868323+0000", 2024-04-14T15:17:46.742 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:17:46.742 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.742 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.742 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:17:46.742 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:16:05.868324+0000", 2024-04-14T15:17:46.742 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6899999999999999e-06 2024-04-14T15:17:46.742 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.742 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.742 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:17:46.742 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:16:05.868332+0000", 2024-04-14T15:17:46.742 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5900000000000002e-06 2024-04-14T15:17:46.742 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.743 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.743 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:17:46.743 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:16:05.868341+0000", 2024-04-14T15:17:46.743 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5869999999999999e-06 2024-04-14T15:17:46.743 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.743 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.743 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:17:46.743 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:16:05.868352+0000", 2024-04-14T15:17:46.743 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0900000000000001e-05 2024-04-14T15:17:46.743 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.743 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.743 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:17:46.743 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:16:05.871436+0000", 2024-04-14T15:17:46.743 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00308348 2024-04-14T15:17:46.743 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.744 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.744 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:17:46.744 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:16:05.871440+0000", 2024-04-14T15:17:46.744 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3970000000000004e-06 2024-04-14T15:17:46.744 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.744 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.744 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:17:46.744 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:16:06.871794+0000", 2024-04-14T15:17:46.744 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.00035395 2024-04-14T15:17:46.744 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.744 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.744 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:17:46.744 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:16:06.871908+0000", 2024-04-14T15:17:46.744 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113795 2024-04-14T15:17:46.745 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.745 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.745 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2024-04-14T15:17:46.745 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:16:06.872654+0000", 2024-04-14T15:17:46.745 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00074600299999999997 2024-04-14T15:17:46.745 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.745 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.745 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2024-04-14T15:17:46.745 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:16:06.875274+0000", 2024-04-14T15:17:46.745 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026198860000000001 2024-04-14T15:17:46.745 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.745 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.745 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:17:46.745 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:16:06.876114+0000", 2024-04-14T15:17:46.745 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00084004299999999995 2024-04-14T15:17:46.746 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.746 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.746 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:17:46.746 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:16:06.876181+0000", 2024-04-14T15:17:46.746 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6994000000000003e-05 2024-04-14T15:17:46.746 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:17:46.746 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:17:46.746 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:17:46.746 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.746 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.746 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.37110.0:1 284.2s0 284:482bd08b:::oid_SimpleStatPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-04-14T15:17:46.746 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:17:04.907943+0000", 2024-04-14T15:17:46.746 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 41.554076080999998, 2024-04-14T15:17:46.746 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8548804969999999, 2024-04-14T15:17:46.746 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:17:46.747 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:17:46.747 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:17:46.747 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:17:46.747 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.37110", 2024-04-14T15:17:46.747 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/784308691", 2024-04-14T15:17:46.747 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1 2024-04-14T15:17:46.747 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.747 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:17:46.747 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.747 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:17:46.747 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:04.907943+0000", 2024-04-14T15:17:46.747 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:17:46.747 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.747 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.747 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:17:46.747 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:04.907943+0000", 2024-04-14T15:17:46.748 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:17:46.748 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.748 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.748 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:17:46.748 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:04.907945+0000", 2024-04-14T15:17:46.748 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.629e-06 2024-04-14T15:17:46.748 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.748 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.748 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:17:46.748 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:04.907952+0000", 2024-04-14T15:17:46.748 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1420000000000004e-06 2024-04-14T15:17:46.748 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.748 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.748 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:17:46.748 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:04.907962+0000", 2024-04-14T15:17:46.748 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.012e-05 2024-04-14T15:17:46.749 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.749 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.749 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:17:46.749 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:04.907973+0000", 2024-04-14T15:17:46.749 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0541e-05 2024-04-14T15:17:46.749 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.749 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.749 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:17:46.749 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:04.908034+0000", 2024-04-14T15:17:46.749 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1426000000000004e-05 2024-04-14T15:17:46.749 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.749 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.749 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:17:46.749 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:04.908038+0000", 2024-04-14T15:17:46.749 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5200000000000002e-06 2024-04-14T15:17:46.749 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.750 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.750 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:17:46.750 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:06.760349+0000", 2024-04-14T15:17:46.750 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8523112479999999 2024-04-14T15:17:46.750 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.750 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.750 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:17:46.750 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:06.760477+0000", 2024-04-14T15:17:46.750 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000128052 2024-04-14T15:17:46.750 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.750 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.750 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2024-04-14T15:17:46.750 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:06.760924+0000", 2024-04-14T15:17:46.750 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00044702499999999998 2024-04-14T15:17:46.750 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.751 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.751 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2024-04-14T15:17:46.751 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:06.761350+0000", 2024-04-14T15:17:46.751 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00042607799999999998 2024-04-14T15:17:46.751 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.751 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.751 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:17:46.751 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:06.762752+0000", 2024-04-14T15:17:46.751 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0014017599999999999 2024-04-14T15:17:46.751 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.751 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.751 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:17:46.751 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:06.762824+0000", 2024-04-14T15:17:46.751 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1956e-05 2024-04-14T15:17:46.751 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:17:46.751 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:17:46.752 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:17:46.752 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.752 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.752 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.47114.0:1 288.4s0 288:311cbe94:::oid_ExecuteClassPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-04-14T15:17:46.752 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:17:25.890728+0000", 2024-04-14T15:17:46.752 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 20.571291807000001, 2024-04-14T15:17:46.752 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0120479069999999, 2024-04-14T15:17:46.752 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:17:46.752 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:17:46.752 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:17:46.752 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:17:46.752 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.47114", 2024-04-14T15:17:46.752 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1312471356", 2024-04-14T15:17:46.752 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1 2024-04-14T15:17:46.753 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.753 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:17:46.753 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.753 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:17:46.753 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:25.890728+0000", 2024-04-14T15:17:46.753 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:17:46.753 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.753 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.753 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:17:46.753 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:25.890728+0000", 2024-04-14T15:17:46.753 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:17:46.753 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.753 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.753 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:17:46.753 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:25.890729+0000", 2024-04-14T15:17:46.754 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6029999999999999e-06 2024-04-14T15:17:46.754 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.754 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.754 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:17:46.754 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:25.890736+0000", 2024-04-14T15:17:46.754 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8360000000000003e-06 2024-04-14T15:17:46.754 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.754 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.754 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:17:46.754 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:25.890746+0000", 2024-04-14T15:17:46.754 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7729999999999994e-06 2024-04-14T15:17:46.754 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.754 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.755 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:17:46.755 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:25.890756+0000", 2024-04-14T15:17:46.755 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9410000000000002e-06 2024-04-14T15:17:46.755 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.755 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:17:46.755 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:25.890812+0000", 2024-04-14T15:17:46.755 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6164e-05 2024-04-14T15:17:46.755 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.755 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:17:46.755 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:25.890816+0000", 2024-04-14T15:17:46.755 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.856e-06 2024-04-14T15:17:46.755 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.755 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:17:46.756 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:26.901241+0000", 2024-04-14T15:17:46.756 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.010425718 2024-04-14T15:17:46.756 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.756 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.756 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:17:46.756 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:26.901363+0000", 2024-04-14T15:17:46.756 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000121956 2024-04-14T15:17:46.756 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.756 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.756 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2024-04-14T15:17:46.756 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:26.901787+0000", 2024-04-14T15:17:46.756 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00042398300000000001 2024-04-14T15:17:46.756 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.756 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.756 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2024-04-14T15:17:46.757 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:26.902091+0000", 2024-04-14T15:17:46.757 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000303689 2024-04-14T15:17:46.757 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.757 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.757 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:17:46.757 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:26.902721+0000", 2024-04-14T15:17:46.757 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00062978299999999997 2024-04-14T15:17:46.757 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:17:46.757 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:17:46.757 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:17:46.757 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:26.902775+0000", 2024-04-14T15:17:46.757 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4605000000000003e-05 2024-04-14T15:17:46.757 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:17:46.757 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:17:46.757 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:17:46.757 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:17:46.758 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:17:46.758 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-14T15:17:46.758 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:17:46.785 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:46.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:46.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:17:46.887 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:17:46.915 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:46.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:46.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:17:47.016 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:17:47.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:47.042 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:47.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:17:47.143 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:17:47.170 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:47.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:47.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:17:47.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:46 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:17:47.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:46 smithi090 ceph-mon[41670]: pgmap v541: 238 pgs: 238 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-14T15:17:47.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:46 smithi090 ceph-mon[41670]: 110.f scrub starts 2024-04-14T15:17:47.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:46 smithi090 ceph-mon[41670]: 110.f scrub ok 2024-04-14T15:17:47.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:46 smithi090 ceph-mon[41670]: 11.1b deep-scrub starts 2024-04-14T15:17:47.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:46 smithi090 ceph-mon[41670]: 11.1b deep-scrub ok 2024-04-14T15:17:47.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:46 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:17:47.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:46 smithi090 ceph-mon[30399]: pgmap v541: 238 pgs: 238 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-14T15:17:47.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:46 smithi090 ceph-mon[30399]: 110.f scrub starts 2024-04-14T15:17:47.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:46 smithi090 ceph-mon[30399]: 110.f scrub ok 2024-04-14T15:17:47.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:46 smithi090 ceph-mon[30399]: 11.1b deep-scrub starts 2024-04-14T15:17:47.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:46 smithi090 ceph-mon[30399]: 11.1b deep-scrub ok 2024-04-14T15:17:47.273 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:17:47.300 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:47.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:47.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:17:47.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:46 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:17:47.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:46 smithi092 ceph-mon[33966]: pgmap v541: 238 pgs: 238 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-14T15:17:47.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:46 smithi092 ceph-mon[33966]: 110.f scrub starts 2024-04-14T15:17:47.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:46 smithi092 ceph-mon[33966]: 110.f scrub ok 2024-04-14T15:17:47.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:46 smithi092 ceph-mon[33966]: 11.1b deep-scrub starts 2024-04-14T15:17:47.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:46 smithi092 ceph-mon[33966]: 11.1b deep-scrub ok 2024-04-14T15:17:47.402 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:17:47.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:47.427 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:47.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:17:47.528 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:17:47.554 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:47.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:47.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:17:47.655 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:17:47.679 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:47.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:47.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:17:47.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.537+0000 7f3cf908d700 1 -- 172.21.15.90:0/727293047 >> v1:172.21.15.90:6789/0 conn(0x7f3cf410f420 legacy=0x7f3cf4111810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:47.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.537+0000 7f3cf908d700 1 -- 172.21.15.90:0/727293047 shutdown_connections 2024-04-14T15:17:47.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.537+0000 7f3cf908d700 1 -- 172.21.15.90:0/727293047 >> 172.21.15.90:0/727293047 conn(0x7f3cf40fd2b0 msgr2=0x7f3cf40ff6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:17:47.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.537+0000 7f3cf908d700 1 -- 172.21.15.90:0/727293047 shutdown_connections 2024-04-14T15:17:47.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.537+0000 7f3cf908d700 1 -- 172.21.15.90:0/727293047 wait complete. 2024-04-14T15:17:47.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.537+0000 7f3cf908d700 1 librados: starting msgr at 2024-04-14T15:17:47.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.537+0000 7f3cf908d700 1 librados: starting objecter 2024-04-14T15:17:47.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.537+0000 7f3cf908d700 1 Processor -- start 2024-04-14T15:17:47.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.538+0000 7f3cf908d700 1 -- start start 2024-04-14T15:17:47.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.538+0000 7f3cf908d700 1 librados: setting wanted keys 2024-04-14T15:17:47.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.538+0000 7f3cf908d700 1 librados: calling monclient init 2024-04-14T15:17:47.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.538+0000 7f3cf908d700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3cf41033d0 con 0x7f3cf410b910 2024-04-14T15:17:47.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.538+0000 7f3cf908d700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3cf41035e0 con 0x7f3cf410f420 2024-04-14T15:17:47.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.538+0000 7f3cf908d700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3cf41ac300 con 0x7f3cf4103b00 2024-04-14T15:17:47.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.538+0000 7f3cf259c700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f3cf410b910 0x7f3cf4101e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:55352/0 (socket says 172.21.15.90:55352) 2024-04-14T15:17:47.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.538+0000 7f3cf259c700 1 -- 172.21.15.90:0/1186676320 learned_addr learned my addr 172.21.15.90:0/1186676320 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:17:47.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.538+0000 7f3cdbfff700 10 client.?.objecter ms_handle_connect 0x7f3cf410b910 2024-04-14T15:17:47.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.538+0000 7f3cdbfff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:17:47.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.538+0000 7f3cdbfff700 10 client.?.objecter ms_handle_connect 0x7f3cf4103b00 2024-04-14T15:17:47.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.538+0000 7f3cdbfff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:17:47.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.538+0000 7f3cdbfff700 10 client.?.objecter ms_handle_connect 0x7f3cf410f420 2024-04-14T15:17:47.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.538+0000 7f3cdbfff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:17:47.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.538+0000 7f3cdbfff700 1 -- 172.21.15.90:0/1186676320 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 56932961 0 0) 0x7f3cf41033d0 con 0x7f3cf410b910 2024-04-14T15:17:47.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.538+0000 7f3cdbfff700 1 -- 172.21.15.90:0/1186676320 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3cdc0035b0 con 0x7f3cf410b910 2024-04-14T15:17:47.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.538+0000 7f3cdbfff700 1 -- 172.21.15.90:0/1186676320 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 909460387 0 0) 0x7f3cf41ac300 con 0x7f3cf4103b00 2024-04-14T15:17:47.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.538+0000 7f3cdbfff700 1 -- 172.21.15.90:0/1186676320 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3cf41033d0 con 0x7f3cf4103b00 2024-04-14T15:17:47.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.539+0000 7f3cdbfff700 1 -- 172.21.15.90:0/1186676320 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1759540036 0 0) 0x7f3cf41035e0 con 0x7f3cf410f420 2024-04-14T15:17:47.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.539+0000 7f3cdbfff700 1 -- 172.21.15.90:0/1186676320 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3cf41ac300 con 0x7f3cf410f420 2024-04-14T15:17:47.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.539+0000 7f3cdbfff700 1 -- 172.21.15.90:0/1186676320 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4173544713 0 0) 0x7f3cdc0035b0 con 0x7f3cf410b910 2024-04-14T15:17:47.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.539+0000 7f3cdbfff700 1 -- 172.21.15.90:0/1186676320 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3cf41035e0 con 0x7f3cf410b910 2024-04-14T15:17:47.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.539+0000 7f3cdbfff700 1 -- 172.21.15.90:0/1186676320 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f3ce80018d0 con 0x7f3cf410b910 2024-04-14T15:17:47.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.539+0000 7f3cdbfff700 1 -- 172.21.15.90:0/1186676320 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1770851768 0 0) 0x7f3cf41033d0 con 0x7f3cf4103b00 2024-04-14T15:17:47.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.539+0000 7f3cdbfff700 1 -- 172.21.15.90:0/1186676320 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3cdc0035b0 con 0x7f3cf4103b00 2024-04-14T15:17:47.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.539+0000 7f3cdbfff700 1 -- 172.21.15.90:0/1186676320 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1945891897 0 0) 0x7f3cf41ac300 con 0x7f3cf410f420 2024-04-14T15:17:47.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.539+0000 7f3cdbfff700 1 -- 172.21.15.90:0/1186676320 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3cf41033d0 con 0x7f3cf410f420 2024-04-14T15:17:47.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.539+0000 7f3cdbfff700 1 -- 172.21.15.90:0/1186676320 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f3cec001c00 con 0x7f3cf410f420 2024-04-14T15:17:47.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.539+0000 7f3cdbfff700 1 -- 172.21.15.90:0/1186676320 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f3ce4001870 con 0x7f3cf4103b00 2024-04-14T15:17:47.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.539+0000 7f3cdbfff700 1 -- 172.21.15.90:0/1186676320 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 189693049 0 0) 0x7f3cf41035e0 con 0x7f3cf410b910 2024-04-14T15:17:47.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.539+0000 7f3cdbfff700 1 -- 172.21.15.90:0/1186676320 >> v1:172.21.15.90:6790/0 conn(0x7f3cf4103b00 legacy=0x7f3cf4101760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:47.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.539+0000 7f3cdbfff700 1 -- 172.21.15.90:0/1186676320 >> v1:172.21.15.92:6789/0 conn(0x7f3cf410f420 legacy=0x7f3cf41a8bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:47.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.539+0000 7f3cdbfff700 1 -- 172.21.15.90:0/1186676320 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cf41ad4e0 con 0x7f3cf410b910 2024-04-14T15:17:47.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.539+0000 7f3cf908d700 1 -- 172.21.15.90:0/1186676320 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3cf41ac530 con 0x7f3cf410b910 2024-04-14T15:17:47.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.539+0000 7f3cf908d700 10 client.37218.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:17:47.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.539+0000 7f3cf908d700 1 -- 172.21.15.90:0/1186676320 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f3cf41ac9f0 con 0x7f3cf410b910 2024-04-14T15:17:47.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.539+0000 7f3cf908d700 1 librados: init done 2024-04-14T15:17:47.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.539+0000 7f3cdbfff700 1 -- 172.21.15.90:0/1186676320 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f3ce8001c80 con 0x7f3cf410b910 2024-04-14T15:17:47.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.539+0000 7f3cdbfff700 1 -- 172.21.15.90:0/1186676320 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f3ce8005cd0 con 0x7f3cf410b910 2024-04-14T15:17:47.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.540+0000 7f3cf908d700 1 -- 172.21.15.90:0/1186676320 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ce0005120 con 0x7f3cf410b910 2024-04-14T15:17:47.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.541+0000 7f3cdbfff700 1 -- 172.21.15.90:0/1186676320 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f3ce801c6c0 con 0x7f3cf410b910 2024-04-14T15:17:47.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.541+0000 7f3cdbfff700 1 -- 172.21.15.90:0/1186676320 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(396..396 src has 51..396) ==== 8178+0+0 (unknown 4225396980 0 0) 0x7f3ce808a8b0 con 0x7f3cf410b910 2024-04-14T15:17:47.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.541+0000 7f3cdbfff700 10 client.37218.objecter ms_dispatch 0x7f3cf4001550 osd_map(396..396 src has 51..396) 2024-04-14T15:17:47.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.541+0000 7f3cdbfff700 3 client.37218.objecter handle_osd_map got epochs [396,396] > 0 2024-04-14T15:17:47.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.541+0000 7f3cdbfff700 3 client.37218.objecter handle_osd_map decoding full epoch 396 2024-04-14T15:17:47.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.541+0000 7f3cdbfff700 20 client.37218.objecter dump_active .. 0 homeless 2024-04-14T15:17:47.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.541+0000 7f3cdbfff700 10 client.37218.objecter ms_handle_connect 0x7f3cdc0713a0 2024-04-14T15:17:47.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.546+0000 7f3cdbfff700 1 -- 172.21.15.90:0/1186676320 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f3ce8057310 con 0x7f3cf410b910 2024-04-14T15:17:47.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.765+0000 7f3cf908d700 1 -- 172.21.15.90:0/1186676320 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd reweight", "id": 5, "weight": 1.0} v 0) -- 0x7f3ce0005410 con 0x7f3cf410b910 2024-04-14T15:17:47.781 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:17:47.809 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:47.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:47.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:17:47.845 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:17:47.911 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:17:47.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.930+0000 7f3cdbfff700 1 -- 172.21.15.90:0/1186676320 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 5, "weight": 1.0}]=0 reweighted osd.5 to 1 (10000) v397) ==== 113+0+0 (unknown 559860513 0 0) 0x7f3ce805afc0 con 0x7f3cf410b910 2024-04-14T15:17:47.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.933+0000 7f3cf908d700 10 librados: watch_flush enter 2024-04-14T15:17:47.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.934+0000 7f3cf908d700 10 librados: watch_flush exit 2024-04-14T15:17:47.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.934+0000 7f3cf908d700 20 client.37218.objecter shutdown clearing up homeless session... 2024-04-14T15:17:47.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.934+0000 7f3cf908d700 10 client.37218.objecter successfully canceled tick 2024-04-14T15:17:47.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.934+0000 7f3cf908d700 1 -- 172.21.15.90:0/1186676320 >> v1:172.21.15.90:6800/420935427 conn(0x7f3cdc0713a0 legacy=0x7f3cdc073860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:47.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.934+0000 7f3cf908d700 1 -- 172.21.15.90:0/1186676320 >> v1:172.21.15.90:6789/0 conn(0x7f3cf410b910 legacy=0x7f3cf4101e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:47.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.934+0000 7f3cf908d700 1 -- 172.21.15.90:0/1186676320 shutdown_connections 2024-04-14T15:17:47.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.934+0000 7f3cf908d700 1 -- 172.21.15.90:0/1186676320 >> 172.21.15.90:0/1186676320 conn(0x7f3cf40fd2b0 msgr2=0x7f3cf40ff6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:17:47.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.936+0000 7f3cf908d700 1 -- 172.21.15.90:0/1186676320 shutdown_connections 2024-04-14T15:17:47.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.936+0000 7f3cf908d700 1 -- 172.21.15.90:0/1186676320 wait complete. 2024-04-14T15:17:47.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:47.936+0000 7f3cf908d700 1 librados: shutdown 2024-04-14T15:17:47.939 INFO:teuthology.orchestra.run.smithi090.stderr:reweighted osd.5 to 1 (10000) 2024-04-14T15:17:47.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:47.940 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:47.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:17:48.041 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:17:48.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:48.071 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:48.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:17:48.071 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd set noscrub 2024-04-14T15:17:48.096 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:47 smithi090 ceph-mon[41670]: 2.0 scrub starts 2024-04-14T15:17:48.096 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:47 smithi090 ceph-mon[41670]: 2.0 scrub ok 2024-04-14T15:17:48.096 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:47 smithi090 ceph-mon[41670]: 11.3 scrub starts 2024-04-14T15:17:48.097 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:47 smithi090 ceph-mon[41670]: 11.3 scrub ok 2024-04-14T15:17:48.097 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:47 smithi090 ceph-mon[41670]: 4.3 deep-scrub starts 2024-04-14T15:17:48.097 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:47 smithi090 ceph-mon[41670]: 4.3 deep-scrub ok 2024-04-14T15:17:48.097 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:47 smithi090 ceph-mon[41670]: 5.16 scrub starts 2024-04-14T15:17:48.097 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:47 smithi090 ceph-mon[41670]: 5.16 scrub ok 2024-04-14T15:17:48.097 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:47 smithi090 ceph-mon[41670]: 110.5 deep-scrub starts 2024-04-14T15:17:48.097 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:47 smithi090 ceph-mon[41670]: 110.5 deep-scrub ok 2024-04-14T15:17:48.097 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:47 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1186676320' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-14T15:17:48.097 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:47 smithi090 ceph-mon[30399]: 2.0 scrub starts 2024-04-14T15:17:48.097 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:47 smithi090 ceph-mon[30399]: 2.0 scrub ok 2024-04-14T15:17:48.097 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:47 smithi090 ceph-mon[30399]: 11.3 scrub starts 2024-04-14T15:17:48.098 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:47 smithi090 ceph-mon[30399]: 11.3 scrub ok 2024-04-14T15:17:48.098 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:47 smithi090 ceph-mon[30399]: 4.3 deep-scrub starts 2024-04-14T15:17:48.098 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:47 smithi090 ceph-mon[30399]: 4.3 deep-scrub ok 2024-04-14T15:17:48.098 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:47 smithi090 ceph-mon[30399]: 5.16 scrub starts 2024-04-14T15:17:48.098 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:47 smithi090 ceph-mon[30399]: 5.16 scrub ok 2024-04-14T15:17:48.098 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:47 smithi090 ceph-mon[30399]: 110.5 deep-scrub starts 2024-04-14T15:17:48.098 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:47 smithi090 ceph-mon[30399]: 110.5 deep-scrub ok 2024-04-14T15:17:48.098 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:47 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1186676320' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-14T15:17:48.172 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:17:48.199 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:48.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:48.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:17:48.301 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:17:48.325 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:48.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:48.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:17:48.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:47 smithi092 ceph-mon[33966]: 2.0 scrub starts 2024-04-14T15:17:48.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:47 smithi092 ceph-mon[33966]: 2.0 scrub ok 2024-04-14T15:17:48.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:47 smithi092 ceph-mon[33966]: 11.3 scrub starts 2024-04-14T15:17:48.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:47 smithi092 ceph-mon[33966]: 11.3 scrub ok 2024-04-14T15:17:48.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:47 smithi092 ceph-mon[33966]: 4.3 deep-scrub starts 2024-04-14T15:17:48.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:47 smithi092 ceph-mon[33966]: 4.3 deep-scrub ok 2024-04-14T15:17:48.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:47 smithi092 ceph-mon[33966]: 5.16 scrub starts 2024-04-14T15:17:48.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:47 smithi092 ceph-mon[33966]: 5.16 scrub ok 2024-04-14T15:17:48.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:47 smithi092 ceph-mon[33966]: 110.5 deep-scrub starts 2024-04-14T15:17:48.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:47 smithi092 ceph-mon[33966]: 110.5 deep-scrub ok 2024-04-14T15:17:48.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:47 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1186676320' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-14T15:17:48.427 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:17:48.451 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:48.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:48.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:17:48.553 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:17:48.579 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:48.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:48.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:17:48.603 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-14T15:17:48.681 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:17:48.704 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:48.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:48.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:17:48.806 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:17:48.830 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:48.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:48.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:17:48.932 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:17:48.940 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:48 smithi090 ceph-mon[41670]: pgmap v542: 238 pgs: 238 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1 op/s 2024-04-14T15:17:48.941 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:48 smithi090 ceph-mon[41670]: 11.f scrub starts 2024-04-14T15:17:48.941 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:48 smithi090 ceph-mon[41670]: 11.f scrub ok 2024-04-14T15:17:48.941 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:48 smithi090 ceph-mon[41670]: 5.7 scrub starts 2024-04-14T15:17:48.941 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:48 smithi090 ceph-mon[41670]: 5.7 scrub ok 2024-04-14T15:17:48.941 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:48 smithi090 ceph-mon[41670]: 1.6 scrub starts 2024-04-14T15:17:48.941 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:48 smithi090 ceph-mon[30399]: pgmap v542: 238 pgs: 238 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1 op/s 2024-04-14T15:17:48.941 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:48 smithi090 ceph-mon[30399]: 11.f scrub starts 2024-04-14T15:17:48.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:48 smithi090 ceph-mon[30399]: 11.f scrub ok 2024-04-14T15:17:48.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:48 smithi090 ceph-mon[30399]: 5.7 scrub starts 2024-04-14T15:17:48.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:48 smithi090 ceph-mon[30399]: 5.7 scrub ok 2024-04-14T15:17:48.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:48 smithi090 ceph-mon[30399]: 1.6 scrub starts 2024-04-14T15:17:48.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:48 smithi090 ceph-mon[30399]: 1.6 scrub ok 2024-04-14T15:17:48.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:48 smithi090 ceph-mon[30399]: 110.1b scrub starts 2024-04-14T15:17:48.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:48 smithi090 ceph-mon[30399]: 110.1b scrub ok 2024-04-14T15:17:48.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:48 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1186676320' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-14T15:17:48.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:48 smithi090 ceph-mon[30399]: osdmap e397: 8 total, 8 up, 8 in 2024-04-14T15:17:48.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:48 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:17:48.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:48 smithi090 ceph-mon[30399]: 4.d scrub starts 2024-04-14T15:17:48.943 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:48 smithi090 ceph-mon[30399]: 4.d scrub ok 2024-04-14T15:17:48.955 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:48.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:48.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:17:48.989 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:17:48.998 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:17:49.057 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:17:49.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:49.082 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:49.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:17:49.183 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:17:49.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:48 smithi090 ceph-mon[41670]: 1.6 scrub ok 2024-04-14T15:17:49.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:48 smithi090 ceph-mon[41670]: 110.1b scrub starts 2024-04-14T15:17:49.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:48 smithi090 ceph-mon[41670]: 110.1b scrub ok 2024-04-14T15:17:49.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:48 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1186676320' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-14T15:17:49.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:48 smithi090 ceph-mon[41670]: osdmap e397: 8 total, 8 up, 8 in 2024-04-14T15:17:49.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:48 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:17:49.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:48 smithi090 ceph-mon[41670]: 4.d scrub starts 2024-04-14T15:17:49.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:48 smithi090 ceph-mon[41670]: 4.d scrub ok 2024-04-14T15:17:49.206 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:49.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:49.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:17:49.308 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:17:49.331 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:49.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:49.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:17:49.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:48 smithi092 ceph-mon[33966]: pgmap v542: 238 pgs: 238 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1 op/s 2024-04-14T15:17:49.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:48 smithi092 ceph-mon[33966]: 11.f scrub starts 2024-04-14T15:17:49.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:48 smithi092 ceph-mon[33966]: 11.f scrub ok 2024-04-14T15:17:49.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:48 smithi092 ceph-mon[33966]: 5.7 scrub starts 2024-04-14T15:17:49.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:48 smithi092 ceph-mon[33966]: 5.7 scrub ok 2024-04-14T15:17:49.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:48 smithi092 ceph-mon[33966]: 1.6 scrub starts 2024-04-14T15:17:49.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:48 smithi092 ceph-mon[33966]: 1.6 scrub ok 2024-04-14T15:17:49.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:48 smithi092 ceph-mon[33966]: 110.1b scrub starts 2024-04-14T15:17:49.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:48 smithi092 ceph-mon[33966]: 110.1b scrub ok 2024-04-14T15:17:49.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:48 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1186676320' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-14T15:17:49.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:48 smithi092 ceph-mon[33966]: osdmap e397: 8 total, 8 up, 8 in 2024-04-14T15:17:49.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:48 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:17:49.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:48 smithi092 ceph-mon[33966]: 4.d scrub starts 2024-04-14T15:17:49.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:48 smithi092 ceph-mon[33966]: 4.d scrub ok 2024-04-14T15:17:49.433 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:17:49.457 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:49.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:49.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:17:49.559 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:17:49.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:49.583 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:49.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:17:49.594 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd reweight 6 1 2024-04-14T15:17:49.684 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:17:49.707 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:49.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:49.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:17:49.809 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:17:49.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:49.833 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:49.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:17:49.934 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:17:49.959 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:49.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:49.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:17:50.061 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:17:50.085 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:50.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:50.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:17:50.177 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:49 smithi090 ceph-mon[41670]: 2.16 scrub starts 2024-04-14T15:17:50.177 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:49 smithi090 ceph-mon[41670]: 2.16 scrub ok 2024-04-14T15:17:50.177 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:49 smithi090 ceph-mon[41670]: 2.6 deep-scrub starts 2024-04-14T15:17:50.177 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:49 smithi090 ceph-mon[41670]: 2.6 deep-scrub ok 2024-04-14T15:17:50.177 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:49 smithi090 ceph-mon[41670]: 3.1 scrub starts 2024-04-14T15:17:50.177 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:49 smithi090 ceph-mon[41670]: 3.1 scrub ok 2024-04-14T15:17:50.177 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:49 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:17:50.177 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:49 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:17:50.177 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:49 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:17:50.177 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:49 smithi090 ceph-mon[30399]: 2.16 scrub starts 2024-04-14T15:17:50.178 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:49 smithi090 ceph-mon[30399]: 2.16 scrub ok 2024-04-14T15:17:50.178 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:49 smithi090 ceph-mon[30399]: 2.6 deep-scrub starts 2024-04-14T15:17:50.178 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:49 smithi090 ceph-mon[30399]: 2.6 deep-scrub ok 2024-04-14T15:17:50.178 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:49 smithi090 ceph-mon[30399]: 3.1 scrub starts 2024-04-14T15:17:50.178 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:49 smithi090 ceph-mon[30399]: 3.1 scrub ok 2024-04-14T15:17:50.178 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:49 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:17:50.178 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:49 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:17:50.178 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:49 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:17:50.186 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:17:50.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:50.211 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:50.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:17:50.312 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:17:50.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:50.336 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:50.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:17:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:49 smithi092 ceph-mon[33966]: 2.16 scrub starts 2024-04-14T15:17:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:49 smithi092 ceph-mon[33966]: 2.16 scrub ok 2024-04-14T15:17:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:49 smithi092 ceph-mon[33966]: 2.6 deep-scrub starts 2024-04-14T15:17:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:49 smithi092 ceph-mon[33966]: 2.6 deep-scrub ok 2024-04-14T15:17:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:49 smithi092 ceph-mon[33966]: 3.1 scrub starts 2024-04-14T15:17:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:49 smithi092 ceph-mon[33966]: 3.1 scrub ok 2024-04-14T15:17:50.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:49 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:17:50.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:49 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:17:50.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:49 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:17:50.437 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:17:50.461 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:50.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:50.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:17:50.563 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:17:50.586 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:50.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:50.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:17:50.688 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:17:50.711 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:50.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:50.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:17:50.813 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:17:50.836 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:50.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:50.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:17:50.938 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:17:50.962 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:50.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:50.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:17:51.064 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:17:51.087 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:51.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:51.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:17:51.189 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:17:51.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:50 smithi090 ceph-mon[30399]: pgmap v544: 238 pgs: 238 active+clean; 8.4 MiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-14T15:17:51.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:50 smithi090 ceph-mon[30399]: 1.3 scrub starts 2024-04-14T15:17:51.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:50 smithi090 ceph-mon[30399]: 1.3 scrub ok 2024-04-14T15:17:51.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:50 smithi090 ceph-mon[30399]: osdmap e398: 8 total, 8 up, 8 in 2024-04-14T15:17:51.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:50 smithi090 ceph-mon[41670]: pgmap v544: 238 pgs: 238 active+clean; 8.4 MiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-14T15:17:51.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:50 smithi090 ceph-mon[41670]: 1.3 scrub starts 2024-04-14T15:17:51.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:50 smithi090 ceph-mon[41670]: 1.3 scrub ok 2024-04-14T15:17:51.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:50 smithi090 ceph-mon[41670]: osdmap e398: 8 total, 8 up, 8 in 2024-04-14T15:17:51.214 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:51.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:51.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:17:51.316 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:17:51.339 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:51.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:51.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:17:51.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:50 smithi092 ceph-mon[33966]: pgmap v544: 238 pgs: 238 active+clean; 8.4 MiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-14T15:17:51.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:50 smithi092 ceph-mon[33966]: 1.3 scrub starts 2024-04-14T15:17:51.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:50 smithi092 ceph-mon[33966]: 1.3 scrub ok 2024-04-14T15:17:51.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:50 smithi092 ceph-mon[33966]: osdmap e398: 8 total, 8 up, 8 in 2024-04-14T15:17:51.441 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:17:51.466 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:51.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:51.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:17:51.568 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:17:51.591 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:51.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:51.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:17:51.693 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:17:51.717 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:51.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:51.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:17:51.819 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:17:51.843 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:51.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:51.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:17:51.945 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:17:51.969 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:51.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:51.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:17:51.976 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:51 smithi090 ceph-mon[41670]: 6.1c scrub starts 2024-04-14T15:17:51.976 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:51 smithi090 ceph-mon[41670]: 6.1c scrub ok 2024-04-14T15:17:51.977 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:51 smithi090 ceph-mon[41670]: 11.1 scrub starts 2024-04-14T15:17:51.977 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:51 smithi090 ceph-mon[41670]: 11.1 scrub ok 2024-04-14T15:17:51.977 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:51 smithi090 ceph-mon[41670]: 5.15 scrub starts 2024-04-14T15:17:51.977 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:51 smithi090 ceph-mon[41670]: 5.15 scrub ok 2024-04-14T15:17:51.977 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:51 smithi090 ceph-mon[41670]: 110.0 scrub starts 2024-04-14T15:17:51.977 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:51 smithi090 ceph-mon[41670]: 110.0 scrub ok 2024-04-14T15:17:51.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:51 smithi090 ceph-mon[30399]: 6.1c scrub starts 2024-04-14T15:17:51.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:51 smithi090 ceph-mon[30399]: 6.1c scrub ok 2024-04-14T15:17:51.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:51 smithi090 ceph-mon[30399]: 11.1 scrub starts 2024-04-14T15:17:51.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:51 smithi090 ceph-mon[30399]: 11.1 scrub ok 2024-04-14T15:17:51.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:51 smithi090 ceph-mon[30399]: 5.15 scrub starts 2024-04-14T15:17:51.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:51 smithi090 ceph-mon[30399]: 5.15 scrub ok 2024-04-14T15:17:51.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:51 smithi090 ceph-mon[30399]: 110.0 scrub starts 2024-04-14T15:17:51.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:51 smithi090 ceph-mon[30399]: 110.0 scrub ok 2024-04-14T15:17:52.012 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:17:52.072 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:17:52.096 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:52.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:52.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:17:52.198 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:17:52.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.203+0000 7fd1ba5ba700 1 -- 172.21.15.90:0/1672948607 >> v1:172.21.15.90:6789/0 conn(0x7fd1b410f360 legacy=0x7fd1b41117f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:52.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.203+0000 7fd1ba5ba700 1 -- 172.21.15.90:0/1672948607 shutdown_connections 2024-04-14T15:17:52.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.204+0000 7fd1ba5ba700 1 -- 172.21.15.90:0/1672948607 >> 172.21.15.90:0/1672948607 conn(0x7fd1b40fd290 msgr2=0x7fd1b40ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:17:52.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.204+0000 7fd1ba5ba700 1 -- 172.21.15.90:0/1672948607 shutdown_connections 2024-04-14T15:17:52.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.204+0000 7fd1ba5ba700 1 -- 172.21.15.90:0/1672948607 wait complete. 2024-04-14T15:17:52.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.204+0000 7fd1ba5ba700 1 librados: starting msgr at 2024-04-14T15:17:52.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.204+0000 7fd1ba5ba700 1 librados: starting objecter 2024-04-14T15:17:52.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.204+0000 7fd1ba5ba700 1 Processor -- start 2024-04-14T15:17:52.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.204+0000 7fd1ba5ba700 1 -- start start 2024-04-14T15:17:52.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.204+0000 7fd1ba5ba700 1 librados: setting wanted keys 2024-04-14T15:17:52.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.204+0000 7fd1ba5ba700 1 librados: calling monclient init 2024-04-14T15:17:52.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.205+0000 7fd1ba5ba700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd1b41a9c20 con 0x7fd1b4103a40 2024-04-14T15:17:52.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.205+0000 7fd1ba5ba700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd1b41aae40 con 0x7fd1b410f360 2024-04-14T15:17:52.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.205+0000 7fd1ba5ba700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd1b41ac080 con 0x7fd1b410b850 2024-04-14T15:17:52.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.205+0000 7fd1b3fff700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fd1b4103a40 0x7fd1b410b0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:36692/0 (socket says 172.21.15.90:36692) 2024-04-14T15:17:52.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.205+0000 7fd1b3fff700 1 -- 172.21.15.90:0/4024142102 learned_addr learned my addr 172.21.15.90:0/4024142102 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:17:52.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.205+0000 7fd1b17fa700 10 client.?.objecter ms_handle_connect 0x7fd1b4103a40 2024-04-14T15:17:52.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.205+0000 7fd1b17fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:17:52.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.205+0000 7fd1b17fa700 10 client.?.objecter ms_handle_connect 0x7fd1b410b850 2024-04-14T15:17:52.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.205+0000 7fd1b17fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:17:52.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.205+0000 7fd1b17fa700 10 client.?.objecter ms_handle_connect 0x7fd1b410f360 2024-04-14T15:17:52.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.205+0000 7fd1b17fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:17:52.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.205+0000 7fd1b17fa700 1 -- 172.21.15.90:0/4024142102 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3098510635 0 0) 0x7fd1b41a9c20 con 0x7fd1b4103a40 2024-04-14T15:17:52.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.205+0000 7fd1b17fa700 1 -- 172.21.15.90:0/4024142102 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd19c0031f0 con 0x7fd1b4103a40 2024-04-14T15:17:52.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.205+0000 7fd1b17fa700 1 -- 172.21.15.90:0/4024142102 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3459517296 0 0) 0x7fd1b41ac080 con 0x7fd1b410b850 2024-04-14T15:17:52.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.205+0000 7fd1b17fa700 1 -- 172.21.15.90:0/4024142102 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1b41a9c20 con 0x7fd1b410b850 2024-04-14T15:17:52.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.205+0000 7fd1b17fa700 1 -- 172.21.15.90:0/4024142102 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1339673491 0 0) 0x7fd1b41aae40 con 0x7fd1b410f360 2024-04-14T15:17:52.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.205+0000 7fd1b17fa700 1 -- 172.21.15.90:0/4024142102 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1b41ac080 con 0x7fd1b410f360 2024-04-14T15:17:52.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.205+0000 7fd1b17fa700 1 -- 172.21.15.90:0/4024142102 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1308146445 0 0) 0x7fd19c0031f0 con 0x7fd1b4103a40 2024-04-14T15:17:52.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.205+0000 7fd1b17fa700 1 -- 172.21.15.90:0/4024142102 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd1b41aae40 con 0x7fd1b4103a40 2024-04-14T15:17:52.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.205+0000 7fd1b17fa700 1 -- 172.21.15.90:0/4024142102 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fd1a4001930 con 0x7fd1b4103a40 2024-04-14T15:17:52.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.205+0000 7fd1b17fa700 1 -- 172.21.15.90:0/4024142102 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 560172221 0 0) 0x7fd1b41a9c20 con 0x7fd1b410b850 2024-04-14T15:17:52.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.205+0000 7fd1b17fa700 1 -- 172.21.15.90:0/4024142102 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd19c0031f0 con 0x7fd1b410b850 2024-04-14T15:17:52.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.205+0000 7fd1b17fa700 1 -- 172.21.15.90:0/4024142102 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3554308108 0 0) 0x7fd1b41ac080 con 0x7fd1b410f360 2024-04-14T15:17:52.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.205+0000 7fd1b17fa700 1 -- 172.21.15.90:0/4024142102 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd1b41a9c20 con 0x7fd1b410f360 2024-04-14T15:17:52.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.205+0000 7fd1b17fa700 1 -- 172.21.15.90:0/4024142102 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fd1a80019f0 con 0x7fd1b410b850 2024-04-14T15:17:52.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.205+0000 7fd1b17fa700 1 -- 172.21.15.90:0/4024142102 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fd1ac0048c0 con 0x7fd1b410f360 2024-04-14T15:17:52.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.205+0000 7fd1b17fa700 1 -- 172.21.15.90:0/4024142102 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2416440954 0 0) 0x7fd1b41aae40 con 0x7fd1b4103a40 2024-04-14T15:17:52.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.205+0000 7fd1b17fa700 1 -- 172.21.15.90:0/4024142102 >> v1:172.21.15.90:6790/0 conn(0x7fd1b410b850 legacy=0x7fd1b41a4b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:52.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.205+0000 7fd1b17fa700 1 -- 172.21.15.90:0/4024142102 >> v1:172.21.15.92:6789/0 conn(0x7fd1b410f360 legacy=0x7fd1b41a82e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:52.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.205+0000 7fd1b17fa700 1 -- 172.21.15.90:0/4024142102 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1b41ad2c0 con 0x7fd1b4103a40 2024-04-14T15:17:52.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.206+0000 7fd1ba5ba700 1 -- 172.21.15.90:0/4024142102 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd1b41ac2b0 con 0x7fd1b4103a40 2024-04-14T15:17:52.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.206+0000 7fd1ba5ba700 10 client.37224.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:17:52.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.206+0000 7fd1ba5ba700 1 -- 172.21.15.90:0/4024142102 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd1b41ac770 con 0x7fd1b4103a40 2024-04-14T15:17:52.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.206+0000 7fd1ba5ba700 1 librados: init done 2024-04-14T15:17:52.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.206+0000 7fd1b17fa700 1 -- 172.21.15.90:0/4024142102 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fd1a4001d00 con 0x7fd1b4103a40 2024-04-14T15:17:52.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.206+0000 7fd1b17fa700 1 -- 172.21.15.90:0/4024142102 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fd1a4005df0 con 0x7fd1b4103a40 2024-04-14T15:17:52.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.207+0000 7fd1ba5ba700 1 -- 172.21.15.90:0/4024142102 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fd1a0005100 con 0x7fd1b4103a40 2024-04-14T15:17:52.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.207+0000 7fd1b17fa700 1 -- 172.21.15.90:0/4024142102 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fd1a401ca80 con 0x7fd1b4103a40 2024-04-14T15:17:52.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.208+0000 7fd1b17fa700 1 -- 172.21.15.90:0/4024142102 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(398..398 src has 51..398) ==== 8178+0+0 (unknown 1447443593 0 0) 0x7fd1a408ac60 con 0x7fd1b4103a40 2024-04-14T15:17:52.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.208+0000 7fd1b17fa700 10 client.37224.objecter ms_dispatch 0x7fd1b4001550 osd_map(398..398 src has 51..398) 2024-04-14T15:17:52.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.208+0000 7fd1b17fa700 3 client.37224.objecter handle_osd_map got epochs [398,398] > 0 2024-04-14T15:17:52.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.208+0000 7fd1b17fa700 3 client.37224.objecter handle_osd_map decoding full epoch 398 2024-04-14T15:17:52.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.208+0000 7fd1b17fa700 20 client.37224.objecter dump_active .. 0 homeless 2024-04-14T15:17:52.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.208+0000 7fd1b17fa700 1 -- 172.21.15.90:0/4024142102 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v398) ==== 54+0+16 (unknown 3178647743 0 2420000192) 0x7fd1a408d0f0 con 0x7fd1b4103a40 2024-04-14T15:17:52.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.208+0000 7fd1ba5ba700 10 client.37224.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:17:52.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.208+0000 7fd1ba5ba700 15 client.37224.objecter _session_command_op_assign -1 1 2024-04-14T15:17:52.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.208+0000 7fd1ba5ba700 20 client.37224.objecter _get_session s=0x7fd1a00053e0 osd=0 3 2024-04-14T15:17:52.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.208+0000 7fd1ba5ba700 20 client.37224.objecter put_session s=0x7fd1a00053e0 osd=0 3 2024-04-14T15:17:52.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.208+0000 7fd1ba5ba700 20 client.37224.objecter _get_session s=0x7fd1a00053e0 osd=0 3 2024-04-14T15:17:52.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.208+0000 7fd1ba5ba700 15 client.37224.objecter _session_command_op_remove -1 1 2024-04-14T15:17:52.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.208+0000 7fd1ba5ba700 20 client.37224.objecter get_session s=0x7fd1a00053e0 osd=0 3 2024-04-14T15:17:52.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.208+0000 7fd1ba5ba700 15 client.37224.objecter _session_command_op_assign 0 1 2024-04-14T15:17:52.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.208+0000 7fd1ba5ba700 20 client.37224.objecter put_session s=0x7fd1a00053e0 osd=0 4 2024-04-14T15:17:52.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.208+0000 7fd1ba5ba700 10 client.37224.objecter _send_command 1 2024-04-14T15:17:52.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.208+0000 7fd1ba5ba700 1 -- 172.21.15.90:0/4024142102 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd1a0004b70 con 0x7fd1a0004730 2024-04-14T15:17:52.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.208+0000 7fd1b17fa700 10 client.37224.objecter ms_handle_connect 0x7fd19c071430 2024-04-14T15:17:52.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.208+0000 7fd1b17fa700 10 client.37224.objecter ms_handle_connect 0x7fd1a0004730 2024-04-14T15:17:52.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.209+0000 7fd1b17fa700 1 -- 172.21.15.90:0/4024142102 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd1a0004b70 con 0x7fd1a0004730 2024-04-14T15:17:52.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.209+0000 7fd1b17fa700 10 client.37224.objecter ms_dispatch 0x7fd1b4001550 command_reply(tid 1: 0 ) 2024-04-14T15:17:52.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.209+0000 7fd1b17fa700 10 client.37224.objecter _finish_command 1 = system:0 2024-04-14T15:17:52.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.209+0000 7fd1b17fa700 20 client.37224.objecter put_session s=0x7fd1a00053e0 osd=0 4 2024-04-14T15:17:52.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.209+0000 7fd1b17fa700 15 client.37224.objecter _session_command_op_remove 0 1 2024-04-14T15:17:52.226 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:52.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:52.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:17:52.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.226+0000 7fd1ba5ba700 10 client.37224.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:17:52.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.226+0000 7fd1ba5ba700 15 client.37224.objecter _session_command_op_assign -1 2 2024-04-14T15:17:52.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.226+0000 7fd1ba5ba700 20 client.37224.objecter _get_session s=0x7fd1a00053e0 osd=0 3 2024-04-14T15:17:52.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.226+0000 7fd1ba5ba700 20 client.37224.objecter put_session s=0x7fd1a00053e0 osd=0 3 2024-04-14T15:17:52.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.226+0000 7fd1ba5ba700 20 client.37224.objecter _get_session s=0x7fd1a00053e0 osd=0 3 2024-04-14T15:17:52.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.226+0000 7fd1ba5ba700 15 client.37224.objecter _session_command_op_remove -1 2 2024-04-14T15:17:52.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.226+0000 7fd1ba5ba700 20 client.37224.objecter get_session s=0x7fd1a00053e0 osd=0 3 2024-04-14T15:17:52.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.226+0000 7fd1ba5ba700 15 client.37224.objecter _session_command_op_assign 0 2 2024-04-14T15:17:52.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.226+0000 7fd1ba5ba700 20 client.37224.objecter put_session s=0x7fd1a00053e0 osd=0 4 2024-04-14T15:17:52.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.226+0000 7fd1ba5ba700 10 client.37224.objecter _send_command 2 2024-04-14T15:17:52.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.226+0000 7fd1ba5ba700 1 -- 172.21.15.90:0/4024142102 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd1a00079f0 con 0x7fd1a0004730 2024-04-14T15:17:52.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.228+0000 7fd1b17fa700 1 -- 172.21.15.90:0/4024142102 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd1a00079f0 con 0x7fd1a0004730 2024-04-14T15:17:52.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.228+0000 7fd1b17fa700 10 client.37224.objecter ms_dispatch 0x7fd1b4001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:17:52.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.228+0000 7fd1b17fa700 10 client.37224.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:17:52.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.228+0000 7fd1b17fa700 20 client.37224.objecter put_session s=0x7fd1a00053e0 osd=0 4 2024-04-14T15:17:52.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.228+0000 7fd1b17fa700 15 client.37224.objecter _session_command_op_remove 0 2 2024-04-14T15:17:52.229 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:17:52.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.228+0000 7fd1ba5ba700 10 client.37224.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:17:52.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.228+0000 7fd1ba5ba700 15 client.37224.objecter _session_command_op_assign -1 3 2024-04-14T15:17:52.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.228+0000 7fd1ba5ba700 20 client.37224.objecter _get_session s=0x7fd1a00040f0 osd=1 3 2024-04-14T15:17:52.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.228+0000 7fd1ba5ba700 20 client.37224.objecter put_session s=0x7fd1a00040f0 osd=1 3 2024-04-14T15:17:52.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.228+0000 7fd1ba5ba700 20 client.37224.objecter _get_session s=0x7fd1a00040f0 osd=1 3 2024-04-14T15:17:52.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.228+0000 7fd1ba5ba700 15 client.37224.objecter _session_command_op_remove -1 3 2024-04-14T15:17:52.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.228+0000 7fd1ba5ba700 20 client.37224.objecter get_session s=0x7fd1a00040f0 osd=1 3 2024-04-14T15:17:52.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.228+0000 7fd1ba5ba700 15 client.37224.objecter _session_command_op_assign 1 3 2024-04-14T15:17:52.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.228+0000 7fd1ba5ba700 20 client.37224.objecter put_session s=0x7fd1a00040f0 osd=1 4 2024-04-14T15:17:52.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.228+0000 7fd1ba5ba700 10 client.37224.objecter _send_command 3 2024-04-14T15:17:52.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.228+0000 7fd1ba5ba700 1 -- 172.21.15.90:0/4024142102 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fd1a0004430 con 0x7fd1a0002100 2024-04-14T15:17:52.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.228+0000 7fd1b17fa700 10 client.37224.objecter ms_handle_connect 0x7fd1a0002100 2024-04-14T15:17:52.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.229+0000 7fd1b17fa700 1 -- 172.21.15.90:0/4024142102 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd1a0004430 con 0x7fd1a0002100 2024-04-14T15:17:52.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.229+0000 7fd1b17fa700 10 client.37224.objecter ms_dispatch 0x7fd1b4001550 command_reply(tid 3: 0 ) 2024-04-14T15:17:52.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.229+0000 7fd1b17fa700 10 client.37224.objecter _finish_command 3 = system:0 2024-04-14T15:17:52.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.229+0000 7fd1b17fa700 20 client.37224.objecter put_session s=0x7fd1a00040f0 osd=1 4 2024-04-14T15:17:52.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.229+0000 7fd1b17fa700 15 client.37224.objecter _session_command_op_remove 1 3 2024-04-14T15:17:52.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.246+0000 7fd1ba5ba700 10 client.37224.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:17:52.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.246+0000 7fd1ba5ba700 15 client.37224.objecter _session_command_op_assign -1 4 2024-04-14T15:17:52.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.246+0000 7fd1ba5ba700 20 client.37224.objecter _get_session s=0x7fd1a00040f0 osd=1 3 2024-04-14T15:17:52.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.246+0000 7fd1ba5ba700 20 client.37224.objecter put_session s=0x7fd1a00040f0 osd=1 3 2024-04-14T15:17:52.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.246+0000 7fd1ba5ba700 20 client.37224.objecter _get_session s=0x7fd1a00040f0 osd=1 3 2024-04-14T15:17:52.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.246+0000 7fd1ba5ba700 15 client.37224.objecter _session_command_op_remove -1 4 2024-04-14T15:17:52.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.246+0000 7fd1ba5ba700 20 client.37224.objecter get_session s=0x7fd1a00040f0 osd=1 3 2024-04-14T15:17:52.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.246+0000 7fd1ba5ba700 15 client.37224.objecter _session_command_op_assign 1 4 2024-04-14T15:17:52.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.246+0000 7fd1ba5ba700 20 client.37224.objecter put_session s=0x7fd1a00040f0 osd=1 4 2024-04-14T15:17:52.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.246+0000 7fd1ba5ba700 10 client.37224.objecter _send_command 4 2024-04-14T15:17:52.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.246+0000 7fd1ba5ba700 1 -- 172.21.15.90:0/4024142102 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd1a00016c0 con 0x7fd1a0002100 2024-04-14T15:17:52.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.247+0000 7fd1b17fa700 1 -- 172.21.15.90:0/4024142102 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd1a00016c0 con 0x7fd1a0002100 2024-04-14T15:17:52.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.247+0000 7fd1b17fa700 10 client.37224.objecter ms_dispatch 0x7fd1b4001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:17:52.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.247+0000 7fd1b17fa700 10 client.37224.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:17:52.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.247+0000 7fd1b17fa700 20 client.37224.objecter put_session s=0x7fd1a00040f0 osd=1 4 2024-04-14T15:17:52.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.247+0000 7fd1b17fa700 15 client.37224.objecter _session_command_op_remove 1 4 2024-04-14T15:17:52.248 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:17:52.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.248+0000 7fd1ba5ba700 10 client.37224.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:17:52.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.248+0000 7fd1ba5ba700 15 client.37224.objecter _session_command_op_assign -1 5 2024-04-14T15:17:52.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.248+0000 7fd1ba5ba700 20 client.37224.objecter _get_session s=0x7fd1a0008370 osd=2 3 2024-04-14T15:17:52.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.248+0000 7fd1ba5ba700 20 client.37224.objecter put_session s=0x7fd1a0008370 osd=2 3 2024-04-14T15:17:52.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.248+0000 7fd1ba5ba700 20 client.37224.objecter _get_session s=0x7fd1a0008370 osd=2 3 2024-04-14T15:17:52.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.248+0000 7fd1ba5ba700 15 client.37224.objecter _session_command_op_remove -1 5 2024-04-14T15:17:52.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.248+0000 7fd1ba5ba700 20 client.37224.objecter get_session s=0x7fd1a0008370 osd=2 3 2024-04-14T15:17:52.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.248+0000 7fd1ba5ba700 15 client.37224.objecter _session_command_op_assign 2 5 2024-04-14T15:17:52.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.248+0000 7fd1ba5ba700 20 client.37224.objecter put_session s=0x7fd1a0008370 osd=2 4 2024-04-14T15:17:52.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.248+0000 7fd1ba5ba700 10 client.37224.objecter _send_command 5 2024-04-14T15:17:52.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.248+0000 7fd1ba5ba700 1 -- 172.21.15.90:0/4024142102 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fd1a00110e0 con 0x7fd1a000d5d0 2024-04-14T15:17:52.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.248+0000 7fd1b17fa700 10 client.37224.objecter ms_handle_connect 0x7fd1a000d5d0 2024-04-14T15:17:52.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.249+0000 7fd1b17fa700 1 -- 172.21.15.90:0/4024142102 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd1a00110e0 con 0x7fd1a000d5d0 2024-04-14T15:17:52.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.249+0000 7fd1b17fa700 10 client.37224.objecter ms_dispatch 0x7fd1b4001550 command_reply(tid 5: 0 ) 2024-04-14T15:17:52.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.249+0000 7fd1b17fa700 10 client.37224.objecter _finish_command 5 = system:0 2024-04-14T15:17:52.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.249+0000 7fd1b17fa700 20 client.37224.objecter put_session s=0x7fd1a0008370 osd=2 4 2024-04-14T15:17:52.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.249+0000 7fd1b17fa700 15 client.37224.objecter _session_command_op_remove 2 5 2024-04-14T15:17:52.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.265+0000 7fd1ba5ba700 10 client.37224.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:17:52.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.265+0000 7fd1ba5ba700 15 client.37224.objecter _session_command_op_assign -1 6 2024-04-14T15:17:52.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.265+0000 7fd1ba5ba700 20 client.37224.objecter _get_session s=0x7fd1a0008370 osd=2 3 2024-04-14T15:17:52.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.265+0000 7fd1ba5ba700 20 client.37224.objecter put_session s=0x7fd1a0008370 osd=2 3 2024-04-14T15:17:52.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.265+0000 7fd1ba5ba700 20 client.37224.objecter _get_session s=0x7fd1a0008370 osd=2 3 2024-04-14T15:17:52.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.265+0000 7fd1ba5ba700 15 client.37224.objecter _session_command_op_remove -1 6 2024-04-14T15:17:52.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.265+0000 7fd1ba5ba700 20 client.37224.objecter get_session s=0x7fd1a0008370 osd=2 3 2024-04-14T15:17:52.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.265+0000 7fd1ba5ba700 15 client.37224.objecter _session_command_op_assign 2 6 2024-04-14T15:17:52.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.265+0000 7fd1ba5ba700 20 client.37224.objecter put_session s=0x7fd1a0008370 osd=2 4 2024-04-14T15:17:52.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.265+0000 7fd1ba5ba700 10 client.37224.objecter _send_command 6 2024-04-14T15:17:52.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.265+0000 7fd1ba5ba700 1 -- 172.21.15.90:0/4024142102 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd1a0007f10 con 0x7fd1a000d5d0 2024-04-14T15:17:52.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.266+0000 7fd1b17fa700 1 -- 172.21.15.90:0/4024142102 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd1a0007f10 con 0x7fd1a000d5d0 2024-04-14T15:17:52.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.266+0000 7fd1b17fa700 10 client.37224.objecter ms_dispatch 0x7fd1b4001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:17:52.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.266+0000 7fd1b17fa700 10 client.37224.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:17:52.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.266+0000 7fd1b17fa700 20 client.37224.objecter put_session s=0x7fd1a0008370 osd=2 4 2024-04-14T15:17:52.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.266+0000 7fd1b17fa700 15 client.37224.objecter _session_command_op_remove 2 6 2024-04-14T15:17:52.268 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:17:52.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.266+0000 7fd1ba5ba700 10 client.37224.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:17:52.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.266+0000 7fd1ba5ba700 15 client.37224.objecter _session_command_op_assign -1 7 2024-04-14T15:17:52.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.266+0000 7fd1ba5ba700 20 client.37224.objecter _get_session s=0x7fd1a00081a0 osd=3 3 2024-04-14T15:17:52.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.266+0000 7fd1ba5ba700 20 client.37224.objecter put_session s=0x7fd1a00081a0 osd=3 3 2024-04-14T15:17:52.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.266+0000 7fd1ba5ba700 20 client.37224.objecter _get_session s=0x7fd1a00081a0 osd=3 3 2024-04-14T15:17:52.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.266+0000 7fd1ba5ba700 15 client.37224.objecter _session_command_op_remove -1 7 2024-04-14T15:17:52.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.266+0000 7fd1ba5ba700 20 client.37224.objecter get_session s=0x7fd1a00081a0 osd=3 3 2024-04-14T15:17:52.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.266+0000 7fd1ba5ba700 15 client.37224.objecter _session_command_op_assign 3 7 2024-04-14T15:17:52.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.266+0000 7fd1ba5ba700 20 client.37224.objecter put_session s=0x7fd1a00081a0 osd=3 4 2024-04-14T15:17:52.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.266+0000 7fd1ba5ba700 10 client.37224.objecter _send_command 7 2024-04-14T15:17:52.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.266+0000 7fd1ba5ba700 1 -- 172.21.15.90:0/4024142102 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fd1a0015f20 con 0x7fd1a0012410 2024-04-14T15:17:52.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.267+0000 7fd1b17fa700 10 client.37224.objecter ms_handle_connect 0x7fd1a0012410 2024-04-14T15:17:52.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.267+0000 7fd1b17fa700 1 -- 172.21.15.90:0/4024142102 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd1a0015f20 con 0x7fd1a0012410 2024-04-14T15:17:52.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.267+0000 7fd1b17fa700 10 client.37224.objecter ms_dispatch 0x7fd1b4001550 command_reply(tid 7: 0 ) 2024-04-14T15:17:52.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.267+0000 7fd1b17fa700 10 client.37224.objecter _finish_command 7 = system:0 2024-04-14T15:17:52.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.267+0000 7fd1b17fa700 20 client.37224.objecter put_session s=0x7fd1a00081a0 osd=3 4 2024-04-14T15:17:52.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.267+0000 7fd1b17fa700 15 client.37224.objecter _session_command_op_remove 3 7 2024-04-14T15:17:52.273 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:51 smithi092 ceph-mon[33966]: 6.1c scrub starts 2024-04-14T15:17:52.273 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:51 smithi092 ceph-mon[33966]: 6.1c scrub ok 2024-04-14T15:17:52.273 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:51 smithi092 ceph-mon[33966]: 11.1 scrub starts 2024-04-14T15:17:52.273 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:51 smithi092 ceph-mon[33966]: 11.1 scrub ok 2024-04-14T15:17:52.273 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:51 smithi092 ceph-mon[33966]: 5.15 scrub starts 2024-04-14T15:17:52.273 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:51 smithi092 ceph-mon[33966]: 5.15 scrub ok 2024-04-14T15:17:52.273 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:51 smithi092 ceph-mon[33966]: 110.0 scrub starts 2024-04-14T15:17:52.273 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:51 smithi092 ceph-mon[33966]: 110.0 scrub ok 2024-04-14T15:17:52.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.284+0000 7fd1ba5ba700 10 client.37224.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:17:52.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.284+0000 7fd1ba5ba700 15 client.37224.objecter _session_command_op_assign -1 8 2024-04-14T15:17:52.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.284+0000 7fd1ba5ba700 20 client.37224.objecter _get_session s=0x7fd1a00081a0 osd=3 3 2024-04-14T15:17:52.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.284+0000 7fd1ba5ba700 20 client.37224.objecter put_session s=0x7fd1a00081a0 osd=3 3 2024-04-14T15:17:52.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.284+0000 7fd1ba5ba700 20 client.37224.objecter _get_session s=0x7fd1a00081a0 osd=3 3 2024-04-14T15:17:52.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.284+0000 7fd1ba5ba700 15 client.37224.objecter _session_command_op_remove -1 8 2024-04-14T15:17:52.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.284+0000 7fd1ba5ba700 20 client.37224.objecter get_session s=0x7fd1a00081a0 osd=3 3 2024-04-14T15:17:52.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.284+0000 7fd1ba5ba700 15 client.37224.objecter _session_command_op_assign 3 8 2024-04-14T15:17:52.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.284+0000 7fd1ba5ba700 20 client.37224.objecter put_session s=0x7fd1a00081a0 osd=3 4 2024-04-14T15:17:52.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.284+0000 7fd1ba5ba700 10 client.37224.objecter _send_command 8 2024-04-14T15:17:52.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.284+0000 7fd1ba5ba700 1 -- 172.21.15.90:0/4024142102 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd1a000c9b0 con 0x7fd1a0012410 2024-04-14T15:17:52.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.286+0000 7fd1b17fa700 1 -- 172.21.15.90:0/4024142102 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd1a000c9b0 con 0x7fd1a0012410 2024-04-14T15:17:52.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.286+0000 7fd1b17fa700 10 client.37224.objecter ms_dispatch 0x7fd1b4001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:17:52.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.286+0000 7fd1b17fa700 10 client.37224.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:17:52.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.286+0000 7fd1b17fa700 20 client.37224.objecter put_session s=0x7fd1a00081a0 osd=3 4 2024-04-14T15:17:52.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.286+0000 7fd1b17fa700 15 client.37224.objecter _session_command_op_remove 3 8 2024-04-14T15:17:52.286 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:17:52.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.286+0000 7fd1ba5ba700 10 client.37224.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:17:52.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.286+0000 7fd1ba5ba700 15 client.37224.objecter _session_command_op_assign -1 9 2024-04-14T15:17:52.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.286+0000 7fd1ba5ba700 20 client.37224.objecter _get_session s=0x7fd1a000cc00 osd=4 3 2024-04-14T15:17:52.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.286+0000 7fd1ba5ba700 20 client.37224.objecter put_session s=0x7fd1a000cc00 osd=4 3 2024-04-14T15:17:52.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.286+0000 7fd1ba5ba700 20 client.37224.objecter _get_session s=0x7fd1a000cc00 osd=4 3 2024-04-14T15:17:52.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.286+0000 7fd1ba5ba700 15 client.37224.objecter _session_command_op_remove -1 9 2024-04-14T15:17:52.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.286+0000 7fd1ba5ba700 20 client.37224.objecter get_session s=0x7fd1a000cc00 osd=4 3 2024-04-14T15:17:52.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.286+0000 7fd1ba5ba700 15 client.37224.objecter _session_command_op_assign 4 9 2024-04-14T15:17:52.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.286+0000 7fd1ba5ba700 20 client.37224.objecter put_session s=0x7fd1a000cc00 osd=4 4 2024-04-14T15:17:52.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.286+0000 7fd1ba5ba700 10 client.37224.objecter _send_command 9 2024-04-14T15:17:52.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.286+0000 7fd1ba5ba700 1 -- 172.21.15.90:0/4024142102 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fd1a001a870 con 0x7fd1a0011f70 2024-04-14T15:17:52.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.286+0000 7fd1b17fa700 10 client.37224.objecter ms_handle_connect 0x7fd1a0011f70 2024-04-14T15:17:52.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.287+0000 7fd1b17fa700 1 -- 172.21.15.90:0/4024142102 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd1a001a870 con 0x7fd1a0011f70 2024-04-14T15:17:52.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.287+0000 7fd1b17fa700 10 client.37224.objecter ms_dispatch 0x7fd1b4001550 command_reply(tid 9: 0 ) 2024-04-14T15:17:52.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.287+0000 7fd1b17fa700 10 client.37224.objecter _finish_command 9 = system:0 2024-04-14T15:17:52.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.287+0000 7fd1b17fa700 20 client.37224.objecter put_session s=0x7fd1a000cc00 osd=4 4 2024-04-14T15:17:52.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.287+0000 7fd1b17fa700 15 client.37224.objecter _session_command_op_remove 4 9 2024-04-14T15:17:52.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.303+0000 7fd1ba5ba700 10 client.37224.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:17:52.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.303+0000 7fd1ba5ba700 15 client.37224.objecter _session_command_op_assign -1 10 2024-04-14T15:17:52.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.303+0000 7fd1ba5ba700 20 client.37224.objecter _get_session s=0x7fd1a000cc00 osd=4 3 2024-04-14T15:17:52.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.303+0000 7fd1ba5ba700 20 client.37224.objecter put_session s=0x7fd1a000cc00 osd=4 3 2024-04-14T15:17:52.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.303+0000 7fd1ba5ba700 20 client.37224.objecter _get_session s=0x7fd1a000cc00 osd=4 3 2024-04-14T15:17:52.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.303+0000 7fd1ba5ba700 15 client.37224.objecter _session_command_op_remove -1 10 2024-04-14T15:17:52.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.303+0000 7fd1ba5ba700 20 client.37224.objecter get_session s=0x7fd1a000cc00 osd=4 3 2024-04-14T15:17:52.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.303+0000 7fd1ba5ba700 15 client.37224.objecter _session_command_op_assign 4 10 2024-04-14T15:17:52.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.303+0000 7fd1ba5ba700 20 client.37224.objecter put_session s=0x7fd1a000cc00 osd=4 4 2024-04-14T15:17:52.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.303+0000 7fd1ba5ba700 10 client.37224.objecter _send_command 10 2024-04-14T15:17:52.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.303+0000 7fd1ba5ba700 1 -- 172.21.15.90:0/4024142102 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd1a00114e0 con 0x7fd1a0011f70 2024-04-14T15:17:52.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.305+0000 7fd1b17fa700 1 -- 172.21.15.90:0/4024142102 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd1a00114e0 con 0x7fd1a0011f70 2024-04-14T15:17:52.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.305+0000 7fd1b17fa700 10 client.37224.objecter ms_dispatch 0x7fd1b4001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:17:52.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.305+0000 7fd1b17fa700 10 client.37224.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:17:52.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.305+0000 7fd1b17fa700 20 client.37224.objecter put_session s=0x7fd1a000cc00 osd=4 4 2024-04-14T15:17:52.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.305+0000 7fd1b17fa700 15 client.37224.objecter _session_command_op_remove 4 10 2024-04-14T15:17:52.306 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:17:52.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.305+0000 7fd1ba5ba700 10 client.37224.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:17:52.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.305+0000 7fd1ba5ba700 15 client.37224.objecter _session_command_op_assign -1 11 2024-04-14T15:17:52.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.305+0000 7fd1ba5ba700 20 client.37224.objecter _get_session s=0x7fd1a000ca70 osd=5 3 2024-04-14T15:17:52.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.305+0000 7fd1ba5ba700 20 client.37224.objecter put_session s=0x7fd1a000ca70 osd=5 3 2024-04-14T15:17:52.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.305+0000 7fd1ba5ba700 20 client.37224.objecter _get_session s=0x7fd1a000ca70 osd=5 3 2024-04-14T15:17:52.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.305+0000 7fd1ba5ba700 15 client.37224.objecter _session_command_op_remove -1 11 2024-04-14T15:17:52.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.305+0000 7fd1ba5ba700 20 client.37224.objecter get_session s=0x7fd1a000ca70 osd=5 3 2024-04-14T15:17:52.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.305+0000 7fd1ba5ba700 15 client.37224.objecter _session_command_op_assign 5 11 2024-04-14T15:17:52.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.305+0000 7fd1ba5ba700 20 client.37224.objecter put_session s=0x7fd1a000ca70 osd=5 4 2024-04-14T15:17:52.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.305+0000 7fd1ba5ba700 10 client.37224.objecter _send_command 11 2024-04-14T15:17:52.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.305+0000 7fd1ba5ba700 1 -- 172.21.15.90:0/4024142102 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fd1a001c030 con 0x7fd1a0011290 2024-04-14T15:17:52.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.306+0000 7fd1b17fa700 10 client.37224.objecter ms_handle_connect 0x7fd1a0011290 2024-04-14T15:17:52.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.306+0000 7fd1b17fa700 1 -- 172.21.15.90:0/4024142102 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd1a800c920 con 0x7fd1a0011290 2024-04-14T15:17:52.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.306+0000 7fd1b17fa700 10 client.37224.objecter ms_dispatch 0x7fd1b4001550 command_reply(tid 11: 0 ) 2024-04-14T15:17:52.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.306+0000 7fd1b17fa700 10 client.37224.objecter _finish_command 11 = system:0 2024-04-14T15:17:52.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.306+0000 7fd1b17fa700 20 client.37224.objecter put_session s=0x7fd1a000ca70 osd=5 4 2024-04-14T15:17:52.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.306+0000 7fd1b17fa700 15 client.37224.objecter _session_command_op_remove 5 11 2024-04-14T15:17:52.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.322+0000 7fd1ba5ba700 10 client.37224.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:17:52.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.322+0000 7fd1ba5ba700 15 client.37224.objecter _session_command_op_assign -1 12 2024-04-14T15:17:52.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.322+0000 7fd1ba5ba700 20 client.37224.objecter _get_session s=0x7fd1a000ca70 osd=5 3 2024-04-14T15:17:52.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.322+0000 7fd1ba5ba700 20 client.37224.objecter put_session s=0x7fd1a000ca70 osd=5 3 2024-04-14T15:17:52.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.322+0000 7fd1ba5ba700 20 client.37224.objecter _get_session s=0x7fd1a000ca70 osd=5 3 2024-04-14T15:17:52.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.322+0000 7fd1ba5ba700 15 client.37224.objecter _session_command_op_remove -1 12 2024-04-14T15:17:52.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.322+0000 7fd1ba5ba700 20 client.37224.objecter get_session s=0x7fd1a000ca70 osd=5 3 2024-04-14T15:17:52.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.322+0000 7fd1ba5ba700 15 client.37224.objecter _session_command_op_assign 5 12 2024-04-14T15:17:52.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.322+0000 7fd1ba5ba700 20 client.37224.objecter put_session s=0x7fd1a000ca70 osd=5 4 2024-04-14T15:17:52.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.322+0000 7fd1ba5ba700 10 client.37224.objecter _send_command 12 2024-04-14T15:17:52.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.322+0000 7fd1ba5ba700 1 -- 172.21.15.90:0/4024142102 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd1a0011750 con 0x7fd1a0011290 2024-04-14T15:17:52.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.324+0000 7fd1b17fa700 1 -- 172.21.15.90:0/4024142102 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd1a0011750 con 0x7fd1a0011290 2024-04-14T15:17:52.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.324+0000 7fd1b17fa700 10 client.37224.objecter ms_dispatch 0x7fd1b4001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:17:52.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.324+0000 7fd1b17fa700 10 client.37224.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:17:52.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.324+0000 7fd1b17fa700 20 client.37224.objecter put_session s=0x7fd1a000ca70 osd=5 4 2024-04-14T15:17:52.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.324+0000 7fd1b17fa700 15 client.37224.objecter _session_command_op_remove 5 12 2024-04-14T15:17:52.325 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:17:52.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.325+0000 7fd1ba5ba700 10 client.37224.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:17:52.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.325+0000 7fd1ba5ba700 15 client.37224.objecter _session_command_op_assign -1 13 2024-04-14T15:17:52.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.325+0000 7fd1ba5ba700 20 client.37224.objecter _get_session s=0x7fd1a0016f20 osd=6 3 2024-04-14T15:17:52.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.325+0000 7fd1ba5ba700 20 client.37224.objecter put_session s=0x7fd1a0016f20 osd=6 3 2024-04-14T15:17:52.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.325+0000 7fd1ba5ba700 20 client.37224.objecter _get_session s=0x7fd1a0016f20 osd=6 3 2024-04-14T15:17:52.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.325+0000 7fd1ba5ba700 15 client.37224.objecter _session_command_op_remove -1 13 2024-04-14T15:17:52.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.325+0000 7fd1ba5ba700 20 client.37224.objecter get_session s=0x7fd1a0016f20 osd=6 3 2024-04-14T15:17:52.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.325+0000 7fd1ba5ba700 15 client.37224.objecter _session_command_op_assign 6 13 2024-04-14T15:17:52.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.325+0000 7fd1ba5ba700 20 client.37224.objecter put_session s=0x7fd1a0016f20 osd=6 4 2024-04-14T15:17:52.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.325+0000 7fd1ba5ba700 10 client.37224.objecter _send_command 13 2024-04-14T15:17:52.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.325+0000 7fd1ba5ba700 1 -- 172.21.15.90:0/4024142102 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fd1a0023440 con 0x7fd1a001f930 2024-04-14T15:17:52.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.325+0000 7fd1b17fa700 10 client.37224.objecter ms_handle_connect 0x7fd1a001f930 2024-04-14T15:17:52.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.326+0000 7fd1b17fa700 1 -- 172.21.15.90:0/4024142102 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd1a0023440 con 0x7fd1a001f930 2024-04-14T15:17:52.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.326+0000 7fd1b17fa700 10 client.37224.objecter ms_dispatch 0x7fd1b4001550 command_reply(tid 13: 0 ) 2024-04-14T15:17:52.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.326+0000 7fd1b17fa700 10 client.37224.objecter _finish_command 13 = system:0 2024-04-14T15:17:52.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.326+0000 7fd1b17fa700 20 client.37224.objecter put_session s=0x7fd1a0016f20 osd=6 4 2024-04-14T15:17:52.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.326+0000 7fd1b17fa700 15 client.37224.objecter _session_command_op_remove 6 13 2024-04-14T15:17:52.328 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:17:52.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.343+0000 7fd1ba5ba700 10 client.37224.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:17:52.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.343+0000 7fd1ba5ba700 15 client.37224.objecter _session_command_op_assign -1 14 2024-04-14T15:17:52.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.343+0000 7fd1ba5ba700 20 client.37224.objecter _get_session s=0x7fd1a0016f20 osd=6 3 2024-04-14T15:17:52.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.343+0000 7fd1ba5ba700 20 client.37224.objecter put_session s=0x7fd1a0016f20 osd=6 3 2024-04-14T15:17:52.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.343+0000 7fd1ba5ba700 20 client.37224.objecter _get_session s=0x7fd1a0016f20 osd=6 3 2024-04-14T15:17:52.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.343+0000 7fd1ba5ba700 15 client.37224.objecter _session_command_op_remove -1 14 2024-04-14T15:17:52.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.343+0000 7fd1ba5ba700 20 client.37224.objecter get_session s=0x7fd1a0016f20 osd=6 3 2024-04-14T15:17:52.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.343+0000 7fd1ba5ba700 15 client.37224.objecter _session_command_op_assign 6 14 2024-04-14T15:17:52.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.343+0000 7fd1ba5ba700 20 client.37224.objecter put_session s=0x7fd1a0016f20 osd=6 4 2024-04-14T15:17:52.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.343+0000 7fd1ba5ba700 10 client.37224.objecter _send_command 14 2024-04-14T15:17:52.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.343+0000 7fd1ba5ba700 1 -- 172.21.15.90:0/4024142102 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd1a0016170 con 0x7fd1a001f930 2024-04-14T15:17:52.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.345+0000 7fd1b17fa700 1 -- 172.21.15.90:0/4024142102 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd1a0016170 con 0x7fd1a001f930 2024-04-14T15:17:52.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.345+0000 7fd1b17fa700 10 client.37224.objecter ms_dispatch 0x7fd1b4001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:17:52.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.345+0000 7fd1b17fa700 10 client.37224.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:17:52.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.345+0000 7fd1b17fa700 20 client.37224.objecter put_session s=0x7fd1a0016f20 osd=6 4 2024-04-14T15:17:52.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.345+0000 7fd1b17fa700 15 client.37224.objecter _session_command_op_remove 6 14 2024-04-14T15:17:52.347 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:17:52.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.345+0000 7fd1ba5ba700 10 client.37224.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:17:52.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.345+0000 7fd1ba5ba700 15 client.37224.objecter _session_command_op_assign -1 15 2024-04-14T15:17:52.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.345+0000 7fd1ba5ba700 20 client.37224.objecter _get_session s=0x7fd1a0016d90 osd=7 3 2024-04-14T15:17:52.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.345+0000 7fd1ba5ba700 20 client.37224.objecter put_session s=0x7fd1a0016d90 osd=7 3 2024-04-14T15:17:52.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.345+0000 7fd1ba5ba700 20 client.37224.objecter _get_session s=0x7fd1a0016d90 osd=7 3 2024-04-14T15:17:52.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.345+0000 7fd1ba5ba700 15 client.37224.objecter _session_command_op_remove -1 15 2024-04-14T15:17:52.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.345+0000 7fd1ba5ba700 20 client.37224.objecter get_session s=0x7fd1a0016d90 osd=7 3 2024-04-14T15:17:52.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.345+0000 7fd1ba5ba700 15 client.37224.objecter _session_command_op_assign 7 15 2024-04-14T15:17:52.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.345+0000 7fd1ba5ba700 20 client.37224.objecter put_session s=0x7fd1a0016d90 osd=7 4 2024-04-14T15:17:52.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.345+0000 7fd1ba5ba700 10 client.37224.objecter _send_command 15 2024-04-14T15:17:52.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.345+0000 7fd1ba5ba700 1 -- 172.21.15.90:0/4024142102 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fd1a0025d80 con 0x7fd1a0015f20 2024-04-14T15:17:52.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.346+0000 7fd1b17fa700 10 client.37224.objecter ms_handle_connect 0x7fd1a0015f20 2024-04-14T15:17:52.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.347+0000 7fd1b17fa700 1 -- 172.21.15.90:0/4024142102 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd1a0025d80 con 0x7fd1a0015f20 2024-04-14T15:17:52.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.347+0000 7fd1b17fa700 10 client.37224.objecter ms_dispatch 0x7fd1b4001550 command_reply(tid 15: 0 ) 2024-04-14T15:17:52.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.347+0000 7fd1b17fa700 10 client.37224.objecter _finish_command 15 = system:0 2024-04-14T15:17:52.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.347+0000 7fd1b17fa700 20 client.37224.objecter put_session s=0x7fd1a0016d90 osd=7 4 2024-04-14T15:17:52.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.347+0000 7fd1b17fa700 15 client.37224.objecter _session_command_op_remove 7 15 2024-04-14T15:17:52.351 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:52.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:52.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:17:52.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.363+0000 7fd1ba5ba700 10 client.37224.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:17:52.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.363+0000 7fd1ba5ba700 15 client.37224.objecter _session_command_op_assign -1 16 2024-04-14T15:17:52.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.363+0000 7fd1ba5ba700 20 client.37224.objecter _get_session s=0x7fd1a0016d90 osd=7 3 2024-04-14T15:17:52.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.363+0000 7fd1ba5ba700 20 client.37224.objecter put_session s=0x7fd1a0016d90 osd=7 3 2024-04-14T15:17:52.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.363+0000 7fd1ba5ba700 20 client.37224.objecter _get_session s=0x7fd1a0016d90 osd=7 3 2024-04-14T15:17:52.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.363+0000 7fd1ba5ba700 15 client.37224.objecter _session_command_op_remove -1 16 2024-04-14T15:17:52.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.363+0000 7fd1ba5ba700 20 client.37224.objecter get_session s=0x7fd1a0016d90 osd=7 3 2024-04-14T15:17:52.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.363+0000 7fd1ba5ba700 15 client.37224.objecter _session_command_op_assign 7 16 2024-04-14T15:17:52.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.363+0000 7fd1ba5ba700 20 client.37224.objecter put_session s=0x7fd1a0016d90 osd=7 4 2024-04-14T15:17:52.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.363+0000 7fd1ba5ba700 10 client.37224.objecter _send_command 16 2024-04-14T15:17:52.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.363+0000 7fd1ba5ba700 1 -- 172.21.15.90:0/4024142102 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd1a0025d80 con 0x7fd1a0015f20 2024-04-14T15:17:52.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.365+0000 7fd1b17fa700 1 -- 172.21.15.90:0/4024142102 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd1a0025d80 con 0x7fd1a0015f20 2024-04-14T15:17:52.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.365+0000 7fd1b17fa700 10 client.37224.objecter ms_dispatch 0x7fd1b4001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:17:52.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.365+0000 7fd1b17fa700 10 client.37224.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:17:52.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.365+0000 7fd1b17fa700 20 client.37224.objecter put_session s=0x7fd1a0016d90 osd=7 4 2024-04-14T15:17:52.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.365+0000 7fd1b17fa700 15 client.37224.objecter _session_command_op_remove 7 16 2024-04-14T15:17:52.366 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:17:52.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.365+0000 7fd1ba5ba700 10 librados: watch_flush enter 2024-04-14T15:17:52.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.365+0000 7fd1ba5ba700 10 librados: watch_flush exit 2024-04-14T15:17:52.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.366+0000 7fd1ba5ba700 10 client.37224.objecter close_session for osd.0 2024-04-14T15:17:52.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.366+0000 7fd1ba5ba700 1 -- 172.21.15.90:0/4024142102 >> v1:172.21.15.90:6801/3761806885 conn(0x7fd1a0004730 legacy=0x7fd1a0002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:52.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.366+0000 7fd1ba5ba700 20 client.37224.objecter put_session s=0x7fd1a00053e0 osd=0 1 2024-04-14T15:17:52.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.366+0000 7fd1ba5ba700 10 client.37224.objecter close_session for osd.1 2024-04-14T15:17:52.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.366+0000 7fd1ba5ba700 1 -- 172.21.15.90:0/4024142102 >> v1:172.21.15.90:6805/4170361699 conn(0x7fd1a0002100 legacy=0x7fd1a000ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:52.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.366+0000 7fd1ba5ba700 20 client.37224.objecter put_session s=0x7fd1a00040f0 osd=1 1 2024-04-14T15:17:52.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.366+0000 7fd1ba5ba700 10 client.37224.objecter close_session for osd.2 2024-04-14T15:17:52.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.366+0000 7fd1ba5ba700 1 -- 172.21.15.90:0/4024142102 >> v1:172.21.15.90:6809/1686184251 conn(0x7fd1a000d5d0 legacy=0x7fd1a000f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:52.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.366+0000 7fd1b8b57700 1 -- 172.21.15.90:0/4024142102 reap_dead start 2024-04-14T15:17:52.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.366+0000 7fd1ba5ba700 20 client.37224.objecter put_session s=0x7fd1a0008370 osd=2 1 2024-04-14T15:17:52.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.366+0000 7fd1ba5ba700 10 client.37224.objecter close_session for osd.3 2024-04-14T15:17:52.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.366+0000 7fd1ba5ba700 1 -- 172.21.15.90:0/4024142102 >> v1:172.21.15.90:6813/2270166774 conn(0x7fd1a0012410 legacy=0x7fd1a0014800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:52.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.366+0000 7fd1ba5ba700 20 client.37224.objecter put_session s=0x7fd1a00081a0 osd=3 1 2024-04-14T15:17:52.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.366+0000 7fd1ba5ba700 10 client.37224.objecter close_session for osd.4 2024-04-14T15:17:52.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.366+0000 7fd1ba5ba700 1 -- 172.21.15.90:0/4024142102 >> v1:172.21.15.92:6800/3887549733 conn(0x7fd1a0011f70 legacy=0x7fd1a00190b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:52.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.366+0000 7fd1ba5ba700 20 client.37224.objecter put_session s=0x7fd1a000cc00 osd=4 1 2024-04-14T15:17:52.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.366+0000 7fd1ba5ba700 10 client.37224.objecter close_session for osd.5 2024-04-14T15:17:52.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.366+0000 7fd1ba5ba700 1 -- 172.21.15.90:0/4024142102 >> v1:172.21.15.92:6804/3244561440 conn(0x7fd1a0011290 legacy=0x7fd1a001a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:52.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.366+0000 7fd1ba5ba700 20 client.37224.objecter put_session s=0x7fd1a000ca70 osd=5 1 2024-04-14T15:17:52.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.366+0000 7fd1ba5ba700 10 client.37224.objecter close_session for osd.6 2024-04-14T15:17:52.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.366+0000 7fd1ba5ba700 1 -- 172.21.15.90:0/4024142102 >> v1:172.21.15.92:6808/636125963 conn(0x7fd1a001f930 legacy=0x7fd1a0021d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:52.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.366+0000 7fd1ba5ba700 20 client.37224.objecter put_session s=0x7fd1a0016f20 osd=6 1 2024-04-14T15:17:52.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.366+0000 7fd1ba5ba700 10 client.37224.objecter close_session for osd.7 2024-04-14T15:17:52.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.366+0000 7fd1ba5ba700 1 -- 172.21.15.90:0/4024142102 >> v1:172.21.15.92:6812/1773706716 conn(0x7fd1a0015f20 legacy=0x7fd1a00245c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:52.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.366+0000 7fd1ba5ba700 20 client.37224.objecter put_session s=0x7fd1a0016d90 osd=7 1 2024-04-14T15:17:52.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.366+0000 7fd1ba5ba700 20 client.37224.objecter shutdown clearing up homeless session... 2024-04-14T15:17:52.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.366+0000 7fd1ba5ba700 10 client.37224.objecter successfully canceled tick 2024-04-14T15:17:52.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.366+0000 7fd1b8b57700 1 -- 172.21.15.90:0/4024142102 reap_dead start 2024-04-14T15:17:52.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.366+0000 7fd1ba5ba700 1 -- 172.21.15.90:0/4024142102 >> v1:172.21.15.90:6800/420935427 conn(0x7fd19c071430 legacy=0x7fd19c0738f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:52.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.366+0000 7fd1ba5ba700 1 -- 172.21.15.90:0/4024142102 >> v1:172.21.15.90:6789/0 conn(0x7fd1b4103a40 legacy=0x7fd1b410b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:52.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.366+0000 7fd1ba5ba700 1 -- 172.21.15.90:0/4024142102 shutdown_connections 2024-04-14T15:17:52.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.366+0000 7fd1ba5ba700 1 -- 172.21.15.90:0/4024142102 >> 172.21.15.90:0/4024142102 conn(0x7fd1b40fd290 msgr2=0x7fd1b41117b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:17:52.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.366+0000 7fd1ba5ba700 1 -- 172.21.15.90:0/4024142102 shutdown_connections 2024-04-14T15:17:52.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.366+0000 7fd1ba5ba700 1 -- 172.21.15.90:0/4024142102 wait complete. 2024-04-14T15:17:52.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:52.366+0000 7fd1ba5ba700 1 librados: shutdown 2024-04-14T15:17:52.370 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:17:52.370 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:17:52.370 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:17:52.370 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:17:52.370 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:17:52.370 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:17:52.370 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:17:52.370 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:17:52.453 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:17:52.477 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:52.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:52.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:17:52.579 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:17:52.604 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:52.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:52.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:17:52.706 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:17:52.730 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:52.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:52.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:17:52.832 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:17:52.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:52.856 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:52.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:17:52.957 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:17:52.983 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:52.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:52.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:17:53.085 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:17:53.108 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:53.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:53.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:17:53.210 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:17:53.233 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:53.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:53.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:17:53.335 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:17:53.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:53.364 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:53.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:17:53.371 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:52 smithi090 ceph-mon[30399]: pgmap v546: 238 pgs: 238 active+clean; 8.4 MiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-14T15:17:53.372 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:52 smithi090 ceph-mon[30399]: 2.a scrub starts 2024-04-14T15:17:53.372 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:52 smithi090 ceph-mon[30399]: 2.a scrub ok 2024-04-14T15:17:53.372 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:52 smithi090 ceph-mon[30399]: 11.1c scrub starts 2024-04-14T15:17:53.372 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:52 smithi090 ceph-mon[30399]: 11.1c scrub ok 2024-04-14T15:17:53.372 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:52 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/4024142102' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:17:53.372 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:52 smithi090 ceph-mon[30399]: 4.1a scrub starts 2024-04-14T15:17:53.372 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:52 smithi090 ceph-mon[30399]: 4.1a scrub ok 2024-04-14T15:17:53.372 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:52 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:17:53.372 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:52 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:17:53.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:52 smithi092 ceph-mon[33966]: pgmap v546: 238 pgs: 238 active+clean; 8.4 MiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-14T15:17:53.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:52 smithi092 ceph-mon[33966]: 2.a scrub starts 2024-04-14T15:17:53.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:52 smithi092 ceph-mon[33966]: 2.a scrub ok 2024-04-14T15:17:53.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:52 smithi092 ceph-mon[33966]: 11.1c scrub starts 2024-04-14T15:17:53.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:52 smithi092 ceph-mon[33966]: 11.1c scrub ok 2024-04-14T15:17:53.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:52 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/4024142102' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:17:53.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:52 smithi092 ceph-mon[33966]: 4.1a scrub starts 2024-04-14T15:17:53.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:52 smithi092 ceph-mon[33966]: 4.1a scrub ok 2024-04-14T15:17:53.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:52 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:17:53.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:52 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:17:53.374 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:52 smithi090 ceph-mon[41670]: pgmap v546: 238 pgs: 238 active+clean; 8.4 MiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-14T15:17:53.374 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:52 smithi090 ceph-mon[41670]: 2.a scrub starts 2024-04-14T15:17:53.374 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:52 smithi090 ceph-mon[41670]: 2.a scrub ok 2024-04-14T15:17:53.374 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:52 smithi090 ceph-mon[41670]: 11.1c scrub starts 2024-04-14T15:17:53.374 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:52 smithi090 ceph-mon[41670]: 11.1c scrub ok 2024-04-14T15:17:53.374 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:52 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/4024142102' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:17:53.374 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:52 smithi090 ceph-mon[41670]: 4.1a scrub starts 2024-04-14T15:17:53.374 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:52 smithi090 ceph-mon[41670]: 4.1a scrub ok 2024-04-14T15:17:53.374 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:52 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:17:53.375 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:52 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:17:53.465 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:17:53.489 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:53.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:53.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:17:53.591 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:17:53.614 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:53.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:53.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:17:53.716 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:17:53.744 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:53.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:53.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:17:53.846 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:17:53.872 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:53.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:53.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:17:53.973 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:17:54.001 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:54.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:54.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:17:54.025 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:17:53 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:17:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:17:54.025 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:53 smithi090 ceph-mon[41670]: 11.16 scrub starts 2024-04-14T15:17:54.025 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:53 smithi090 ceph-mon[41670]: 11.16 scrub ok 2024-04-14T15:17:54.025 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:53 smithi090 ceph-mon[41670]: 4.f deep-scrub starts 2024-04-14T15:17:54.025 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:53 smithi090 ceph-mon[41670]: 110.c deep-scrub starts 2024-04-14T15:17:54.025 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:53 smithi090 ceph-mon[41670]: 110.c deep-scrub ok 2024-04-14T15:17:54.025 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:53 smithi090 ceph-mon[41670]: 4.f deep-scrub ok 2024-04-14T15:17:54.025 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:53 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:17:54.025 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:53 smithi090 ceph-mon[41670]: osdmap e399: 8 total, 8 up, 8 in 2024-04-14T15:17:54.026 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:53 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-51"}]: dispatch 2024-04-14T15:17:54.026 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:53 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-51"}]: dispatch 2024-04-14T15:17:54.026 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:53 smithi090 ceph-mon[41670]: 4.10 scrub starts 2024-04-14T15:17:54.026 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:53 smithi090 ceph-mon[41670]: 4.10 scrub ok 2024-04-14T15:17:54.026 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:53 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:17:54.026 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:53 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:17:54.026 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:53 smithi090 ceph-mon[30399]: 11.16 scrub starts 2024-04-14T15:17:54.026 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:53 smithi090 ceph-mon[30399]: 11.16 scrub ok 2024-04-14T15:17:54.026 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:53 smithi090 ceph-mon[30399]: 4.f deep-scrub starts 2024-04-14T15:17:54.026 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:53 smithi090 ceph-mon[30399]: 110.c deep-scrub starts 2024-04-14T15:17:54.026 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:53 smithi090 ceph-mon[30399]: 110.c deep-scrub ok 2024-04-14T15:17:54.026 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:53 smithi090 ceph-mon[30399]: 4.f deep-scrub ok 2024-04-14T15:17:54.027 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:53 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:17:54.027 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:53 smithi090 ceph-mon[30399]: osdmap e399: 8 total, 8 up, 8 in 2024-04-14T15:17:54.027 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:53 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-51"}]: dispatch 2024-04-14T15:17:54.027 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:53 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-51"}]: dispatch 2024-04-14T15:17:54.027 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:53 smithi090 ceph-mon[30399]: 4.10 scrub starts 2024-04-14T15:17:54.027 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:53 smithi090 ceph-mon[30399]: 4.10 scrub ok 2024-04-14T15:17:54.027 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:53 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:17:54.027 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:53 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:17:54.103 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:17:54.127 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:54.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:54.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:17:54.229 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:17:54.254 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:54.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:54.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:17:54.356 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:17:54.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:53 smithi092 ceph-mon[33966]: 11.16 scrub starts 2024-04-14T15:17:54.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:53 smithi092 ceph-mon[33966]: 11.16 scrub ok 2024-04-14T15:17:54.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:53 smithi092 ceph-mon[33966]: 4.f deep-scrub starts 2024-04-14T15:17:54.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:53 smithi092 ceph-mon[33966]: 110.c deep-scrub starts 2024-04-14T15:17:54.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:53 smithi092 ceph-mon[33966]: 110.c deep-scrub ok 2024-04-14T15:17:54.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:53 smithi092 ceph-mon[33966]: 4.f deep-scrub ok 2024-04-14T15:17:54.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:53 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:17:54.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:53 smithi092 ceph-mon[33966]: osdmap e399: 8 total, 8 up, 8 in 2024-04-14T15:17:54.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:53 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-51"}]: dispatch 2024-04-14T15:17:54.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:53 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-51"}]: dispatch 2024-04-14T15:17:54.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:53 smithi092 ceph-mon[33966]: 4.10 scrub starts 2024-04-14T15:17:54.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:53 smithi092 ceph-mon[33966]: 4.10 scrub ok 2024-04-14T15:17:54.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:53 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:17:54.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:53 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:17:54.380 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:54.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:54.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:17:54.481 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:17:54.510 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:54.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:54.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:17:54.612 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:17:54.635 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:54.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:54.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:17:54.737 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:17:54.765 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:54.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:54.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:17:54.867 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:17:54.891 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:54.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:54.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:17:54.992 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:17:54.996 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:54 smithi090 ceph-mon[41670]: pgmap v548: 238 pgs: 238 active+clean; 8.4 MiB data, 606 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:17:54.996 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:54 smithi090 ceph-mon[41670]: 2.e scrub starts 2024-04-14T15:17:54.997 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:54 smithi090 ceph-mon[41670]: 2.e scrub ok 2024-04-14T15:17:54.997 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:54 smithi090 ceph-mon[41670]: 110.1a scrub starts 2024-04-14T15:17:54.997 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:54 smithi090 ceph-mon[41670]: 110.1a scrub ok 2024-04-14T15:17:54.997 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:54 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-51"}]': finished 2024-04-14T15:17:54.997 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:54 smithi090 ceph-mon[41670]: osdmap e400: 8 total, 8 up, 8 in 2024-04-14T15:17:54.997 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:54 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:17:54.997 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:54 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:17:54.997 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:54 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-51"}]: dispatch 2024-04-14T15:17:54.997 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:54 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-51"}]: dispatch 2024-04-14T15:17:54.997 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:54 smithi090 ceph-mon[41670]: 6.1e deep-scrub starts 2024-04-14T15:17:54.997 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:54 smithi090 ceph-mon[41670]: 6.1e deep-scrub ok 2024-04-14T15:17:54.997 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:54 smithi090 ceph-mon[30399]: pgmap v548: 238 pgs: 238 active+clean; 8.4 MiB data, 606 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:17:54.998 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:54 smithi090 ceph-mon[30399]: 2.e scrub starts 2024-04-14T15:17:54.998 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:54 smithi090 ceph-mon[30399]: 2.e scrub ok 2024-04-14T15:17:54.998 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:54 smithi090 ceph-mon[30399]: 110.1a scrub starts 2024-04-14T15:17:54.998 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:54 smithi090 ceph-mon[30399]: 110.1a scrub ok 2024-04-14T15:17:54.998 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:54 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-51"}]': finished 2024-04-14T15:17:54.998 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:54 smithi090 ceph-mon[30399]: osdmap e400: 8 total, 8 up, 8 in 2024-04-14T15:17:54.998 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:54 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:17:54.998 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:54 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:17:54.998 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:54 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-51"}]: dispatch 2024-04-14T15:17:54.998 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:54 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-51"}]: dispatch 2024-04-14T15:17:54.998 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:54 smithi090 ceph-mon[30399]: 6.1e deep-scrub starts 2024-04-14T15:17:54.998 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:54 smithi090 ceph-mon[30399]: 6.1e deep-scrub ok 2024-04-14T15:17:55.018 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:55.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:55.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:17:55.119 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:17:55.142 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:55.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:55.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:17:55.244 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:17:55.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:55.274 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:55.274 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:17:55.372 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:17:55 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:17:55.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:54 smithi092 ceph-mon[33966]: pgmap v548: 238 pgs: 238 active+clean; 8.4 MiB data, 606 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:17:55.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:54 smithi092 ceph-mon[33966]: 2.e scrub starts 2024-04-14T15:17:55.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:54 smithi092 ceph-mon[33966]: 2.e scrub ok 2024-04-14T15:17:55.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:54 smithi092 ceph-mon[33966]: 110.1a scrub starts 2024-04-14T15:17:55.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:54 smithi092 ceph-mon[33966]: 110.1a scrub ok 2024-04-14T15:17:55.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:54 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-51"}]': finished 2024-04-14T15:17:55.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:54 smithi092 ceph-mon[33966]: osdmap e400: 8 total, 8 up, 8 in 2024-04-14T15:17:55.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:54 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:17:55.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:54 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:17:55.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:54 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-51"}]: dispatch 2024-04-14T15:17:55.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:54 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-51"}]: dispatch 2024-04-14T15:17:55.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:54 smithi092 ceph-mon[33966]: 6.1e deep-scrub starts 2024-04-14T15:17:55.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:54 smithi092 ceph-mon[33966]: 6.1e deep-scrub ok 2024-04-14T15:17:55.375 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:17:55.399 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-14T15:17:55.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:55.400 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:55.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:17:55.502 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:17:55.525 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:55.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:55.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:17:55.627 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:17:55.655 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:55.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:55.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:17:55.757 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:17:55.785 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:55.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:55.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:17:55.887 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:17:55.919 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:55.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:55.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:17:55.969 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-14T15:17:55.969 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:17:56.021 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:17:56.049 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:56.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:56.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:17:56.151 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:17:56.178 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:56.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:56.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:17:56.280 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:17:56.284 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:55 smithi090 ceph-mon[41670]: 5.9 scrub starts 2024-04-14T15:17:56.284 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:55 smithi090 ceph-mon[41670]: 5.9 scrub ok 2024-04-14T15:17:56.284 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:55 smithi090 ceph-mon[41670]: 5.d deep-scrub starts 2024-04-14T15:17:56.284 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:55 smithi090 ceph-mon[41670]: 5.d deep-scrub ok 2024-04-14T15:17:56.284 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:55 smithi090 ceph-mon[41670]: 5.c scrub starts 2024-04-14T15:17:56.284 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:55 smithi090 ceph-mon[41670]: 5.c scrub ok 2024-04-14T15:17:56.284 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:55 smithi090 ceph-mon[41670]: 6.a scrub starts 2024-04-14T15:17:56.284 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:55 smithi090 ceph-mon[41670]: 6.a scrub ok 2024-04-14T15:17:56.284 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:55 smithi090 ceph-mon[41670]: osdmap e401: 8 total, 8 up, 8 in 2024-04-14T15:17:56.284 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:55 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:17:56.284 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:55 smithi090 ceph-mon[41670]: pgmap v551: 206 pgs: 206 active+clean; 8.4 MiB data, 606 MiB used, 715 GiB / 715 GiB avail; 244 B/s rd, 0 op/s 2024-04-14T15:17:56.285 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:55 smithi090 ceph-mon[41670]: osdmap e402: 8 total, 8 up, 8 in 2024-04-14T15:17:56.285 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:55 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:17:56.285 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:55 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:17:56.285 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:55 smithi090 ceph-mon[30399]: 5.9 scrub starts 2024-04-14T15:17:56.285 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:55 smithi090 ceph-mon[30399]: 5.9 scrub ok 2024-04-14T15:17:56.285 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:55 smithi090 ceph-mon[30399]: 5.d deep-scrub starts 2024-04-14T15:17:56.285 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:55 smithi090 ceph-mon[30399]: 5.d deep-scrub ok 2024-04-14T15:17:56.285 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:55 smithi090 ceph-mon[30399]: 5.c scrub starts 2024-04-14T15:17:56.285 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:55 smithi090 ceph-mon[30399]: 5.c scrub ok 2024-04-14T15:17:56.285 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:55 smithi090 ceph-mon[30399]: 6.a scrub starts 2024-04-14T15:17:56.285 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:55 smithi090 ceph-mon[30399]: 6.a scrub ok 2024-04-14T15:17:56.285 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:55 smithi090 ceph-mon[30399]: osdmap e401: 8 total, 8 up, 8 in 2024-04-14T15:17:56.285 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:55 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:17:56.286 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:55 smithi090 ceph-mon[30399]: pgmap v551: 206 pgs: 206 active+clean; 8.4 MiB data, 606 MiB used, 715 GiB / 715 GiB avail; 244 B/s rd, 0 op/s 2024-04-14T15:17:56.286 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:55 smithi090 ceph-mon[30399]: osdmap e402: 8 total, 8 up, 8 in 2024-04-14T15:17:56.286 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:55 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:17:56.286 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:55 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:17:56.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:56.304 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:56.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:17:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:55 smithi092 ceph-mon[33966]: 5.9 scrub starts 2024-04-14T15:17:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:55 smithi092 ceph-mon[33966]: 5.9 scrub ok 2024-04-14T15:17:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:55 smithi092 ceph-mon[33966]: 5.d deep-scrub starts 2024-04-14T15:17:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:55 smithi092 ceph-mon[33966]: 5.d deep-scrub ok 2024-04-14T15:17:56.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:55 smithi092 ceph-mon[33966]: 5.c scrub starts 2024-04-14T15:17:56.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:55 smithi092 ceph-mon[33966]: 5.c scrub ok 2024-04-14T15:17:56.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:55 smithi092 ceph-mon[33966]: 6.a scrub starts 2024-04-14T15:17:56.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:55 smithi092 ceph-mon[33966]: 6.a scrub ok 2024-04-14T15:17:56.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:55 smithi092 ceph-mon[33966]: osdmap e401: 8 total, 8 up, 8 in 2024-04-14T15:17:56.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:55 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:17:56.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:55 smithi092 ceph-mon[33966]: pgmap v551: 206 pgs: 206 active+clean; 8.4 MiB data, 606 MiB used, 715 GiB / 715 GiB avail; 244 B/s rd, 0 op/s 2024-04-14T15:17:56.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:55 smithi092 ceph-mon[33966]: osdmap e402: 8 total, 8 up, 8 in 2024-04-14T15:17:56.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:55 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:17:56.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:55 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:17:56.406 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:17:56.433 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:56.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:56.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:17:56.535 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:17:56.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:56.560 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:56.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:17:56.643 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-14T15:17:56.662 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:17:56.689 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:56.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:56.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:17:56.791 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:17:56.816 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:56.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:56.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:17:56.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:56.837+0000 7f7afbddf700 1 -- 172.21.15.90:0/3926176218 >> v1:172.21.15.90:6789/0 conn(0x7f7af410b740 legacy=0x7f7af410db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:56.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:56.837+0000 7f7afbddf700 1 -- 172.21.15.90:0/3926176218 shutdown_connections 2024-04-14T15:17:56.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:56.837+0000 7f7afbddf700 1 -- 172.21.15.90:0/3926176218 >> 172.21.15.90:0/3926176218 conn(0x7f7af40fe380 msgr2=0x7f7af41007a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:17:56.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:56.837+0000 7f7afbddf700 1 -- 172.21.15.90:0/3926176218 shutdown_connections 2024-04-14T15:17:56.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:56.837+0000 7f7afbddf700 1 -- 172.21.15.90:0/3926176218 wait complete. 2024-04-14T15:17:56.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:56.838+0000 7f7afbddf700 1 librados: starting msgr at 2024-04-14T15:17:56.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:56.838+0000 7f7afbddf700 1 librados: starting objecter 2024-04-14T15:17:56.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:56.838+0000 7f7afbddf700 1 Processor -- start 2024-04-14T15:17:56.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:56.838+0000 7f7afbddf700 1 -- start start 2024-04-14T15:17:56.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:56.838+0000 7f7afbddf700 1 librados: setting wanted keys 2024-04-14T15:17:56.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:56.838+0000 7f7afbddf700 1 librados: calling monclient init 2024-04-14T15:17:56.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:56.838+0000 7f7afbddf700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7af41a9e80 con 0x7f7af410f310 2024-04-14T15:17:56.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:56.838+0000 7f7afbddf700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7af41ab0a0 con 0x7f7af410b740 2024-04-14T15:17:56.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:56.838+0000 7f7afbddf700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7af41ac2e0 con 0x7f7af4108890 2024-04-14T15:17:56.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:56.838+0000 7f7afa37c700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f7af410f310 0x7f7af41a8540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:36710/0 (socket says 172.21.15.90:36710) 2024-04-14T15:17:56.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:56.838+0000 7f7afa37c700 1 -- 172.21.15.90:0/2999560166 learned_addr learned my addr 172.21.15.90:0/2999560166 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:17:56.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:56.838+0000 7f7aeaffd700 10 client.?.objecter ms_handle_connect 0x7f7af410f310 2024-04-14T15:17:56.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:56.838+0000 7f7aeaffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:17:56.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:56.839+0000 7f7aeaffd700 10 client.?.objecter ms_handle_connect 0x7f7af4108890 2024-04-14T15:17:56.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:56.839+0000 7f7aeaffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:17:56.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:56.839+0000 7f7aeaffd700 1 -- 172.21.15.90:0/2999560166 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1404600573 0 0) 0x7f7af41a9e80 con 0x7f7af410f310 2024-04-14T15:17:56.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:56.839+0000 7f7aeaffd700 1 -- 172.21.15.90:0/2999560166 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7ae00035b0 con 0x7f7af410f310 2024-04-14T15:17:56.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:56.839+0000 7f7aeaffd700 10 client.?.objecter ms_handle_connect 0x7f7af410b740 2024-04-14T15:17:56.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:56.839+0000 7f7aeaffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:17:56.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:56.839+0000 7f7aeaffd700 1 -- 172.21.15.90:0/2999560166 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 768524367 0 0) 0x7f7af41ac2e0 con 0x7f7af4108890 2024-04-14T15:17:56.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:56.839+0000 7f7aeaffd700 1 -- 172.21.15.90:0/2999560166 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7af41a9e80 con 0x7f7af4108890 2024-04-14T15:17:56.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:56.839+0000 7f7aeaffd700 1 -- 172.21.15.90:0/2999560166 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 116073917 0 0) 0x7f7af41ab0a0 con 0x7f7af410b740 2024-04-14T15:17:56.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:56.839+0000 7f7aeaffd700 1 -- 172.21.15.90:0/2999560166 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7af41ac2e0 con 0x7f7af410b740 2024-04-14T15:17:56.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:56.839+0000 7f7aeaffd700 1 -- 172.21.15.90:0/2999560166 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1973022753 0 0) 0x7f7ae00035b0 con 0x7f7af410f310 2024-04-14T15:17:56.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:56.839+0000 7f7aeaffd700 1 -- 172.21.15.90:0/2999560166 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7af41ab0a0 con 0x7f7af410f310 2024-04-14T15:17:56.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:56.839+0000 7f7aeaffd700 1 -- 172.21.15.90:0/2999560166 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f7aec001c50 con 0x7f7af410f310 2024-04-14T15:17:56.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:56.839+0000 7f7aeaffd700 1 -- 172.21.15.90:0/2999560166 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3111444891 0 0) 0x7f7af41ac2e0 con 0x7f7af410b740 2024-04-14T15:17:56.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:56.839+0000 7f7aeaffd700 1 -- 172.21.15.90:0/2999560166 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7ae00035b0 con 0x7f7af410b740 2024-04-14T15:17:56.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:56.839+0000 7f7aeaffd700 1 -- 172.21.15.90:0/2999560166 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3916771550 0 0) 0x7f7af41a9e80 con 0x7f7af4108890 2024-04-14T15:17:56.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:56.839+0000 7f7aeaffd700 1 -- 172.21.15.90:0/2999560166 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7af41ac2e0 con 0x7f7af4108890 2024-04-14T15:17:56.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:56.839+0000 7f7aeaffd700 1 -- 172.21.15.90:0/2999560166 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f7af0001430 con 0x7f7af410b740 2024-04-14T15:17:56.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:56.839+0000 7f7aeaffd700 1 -- 172.21.15.90:0/2999560166 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2756803634 0 0) 0x7f7af41ab0a0 con 0x7f7af410f310 2024-04-14T15:17:56.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:56.839+0000 7f7aeaffd700 1 -- 172.21.15.90:0/2999560166 >> v1:172.21.15.90:6790/0 conn(0x7f7af4108890 legacy=0x7f7af410eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:56.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:56.839+0000 7f7aeaffd700 1 -- 172.21.15.90:0/2999560166 >> v1:172.21.15.92:6789/0 conn(0x7f7af410b740 legacy=0x7f7af41a4bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:56.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:56.839+0000 7f7aeaffd700 1 -- 172.21.15.90:0/2999560166 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7af41ad520 con 0x7f7af410f310 2024-04-14T15:17:56.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:56.839+0000 7f7afbddf700 1 -- 172.21.15.90:0/2999560166 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7af41ab2d0 con 0x7f7af410f310 2024-04-14T15:17:56.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:56.839+0000 7f7afbddf700 10 client.37230.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:17:56.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:56.839+0000 7f7afbddf700 1 -- 172.21.15.90:0/2999560166 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f7af41ab790 con 0x7f7af410f310 2024-04-14T15:17:56.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:56.839+0000 7f7afbddf700 1 librados: init done 2024-04-14T15:17:56.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:56.840+0000 7f7aeaffd700 1 -- 172.21.15.90:0/2999560166 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f7aec003ec0 con 0x7f7af410f310 2024-04-14T15:17:56.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:56.840+0000 7f7aeaffd700 1 -- 172.21.15.90:0/2999560166 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f7aec006010 con 0x7f7af410f310 2024-04-14T15:17:56.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:56.841+0000 7f7afbddf700 1 -- 172.21.15.90:0/2999560166 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ad8005120 con 0x7f7af410f310 2024-04-14T15:17:56.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:56.841+0000 7f7aeaffd700 1 -- 172.21.15.90:0/2999560166 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f7aec01ca20 con 0x7f7af410f310 2024-04-14T15:17:56.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:56.842+0000 7f7aeaffd700 1 -- 172.21.15.90:0/2999560166 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(402..402 src has 51..402) ==== 8131+0+0 (unknown 1031776454 0 0) 0x7f7aec08abd0 con 0x7f7af410f310 2024-04-14T15:17:56.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:56.842+0000 7f7aeaffd700 10 client.37230.objecter ms_dispatch 0x7f7af4001550 osd_map(402..402 src has 51..402) 2024-04-14T15:17:56.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:56.842+0000 7f7aeaffd700 3 client.37230.objecter handle_osd_map got epochs [402,402] > 0 2024-04-14T15:17:56.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:56.842+0000 7f7aeaffd700 3 client.37230.objecter handle_osd_map decoding full epoch 402 2024-04-14T15:17:56.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:56.842+0000 7f7aeaffd700 20 client.37230.objecter dump_active .. 0 homeless 2024-04-14T15:17:56.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:56.842+0000 7f7aeaffd700 10 client.37230.objecter ms_handle_connect 0x7f7ae0071440 2024-04-14T15:17:56.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:56.847+0000 7f7aeaffd700 1 -- 172.21.15.90:0/2999560166 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f7aec003a20 con 0x7f7af410f310 2024-04-14T15:17:56.919 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:17:56.942 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:56.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:56.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:17:57.044 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:17:57.066 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:57.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:57.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:17:57.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:57.081+0000 7f7afbddf700 1 -- 172.21.15.90:0/2999560166 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7f7ad8005410 con 0x7f7af410f310 2024-04-14T15:17:57.169 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:17:57.196 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:57.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:57.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:17:57.298 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:17:57.299 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:56 smithi090 ceph-mon[41670]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-04-14T15:17:57.299 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:56 smithi090 ceph-mon[41670]: 6.1b scrub starts 2024-04-14T15:17:57.299 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:56 smithi090 ceph-mon[41670]: 6.1b scrub ok 2024-04-14T15:17:57.299 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:56 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:17:57.300 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:56 smithi090 ceph-mon[41670]: osdmap e403: 8 total, 8 up, 8 in 2024-04-14T15:17:57.300 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:56 smithi090 ceph-mon[30399]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-04-14T15:17:57.300 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:56 smithi090 ceph-mon[30399]: 6.1b scrub starts 2024-04-14T15:17:57.300 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:56 smithi090 ceph-mon[30399]: 6.1b scrub ok 2024-04-14T15:17:57.300 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:56 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:17:57.300 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:56 smithi090 ceph-mon[30399]: osdmap e403: 8 total, 8 up, 8 in 2024-04-14T15:17:57.324 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:57.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:57.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:17:57.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:56 smithi092 ceph-mon[33966]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-04-14T15:17:57.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:56 smithi092 ceph-mon[33966]: 6.1b scrub starts 2024-04-14T15:17:57.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:56 smithi092 ceph-mon[33966]: 6.1b scrub ok 2024-04-14T15:17:57.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:56 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:17:57.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:56 smithi092 ceph-mon[33966]: osdmap e403: 8 total, 8 up, 8 in 2024-04-14T15:17:57.426 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:17:57.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:57.451 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:57.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:17:57.552 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:17:57.576 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:57.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:57.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:17:57.678 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:17:57.706 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:57.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:57.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:17:57.774 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:17:57.808 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:17:57.815 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:17:57.834 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:57.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:57.835 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:17:57.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:57.888+0000 7f7aeaffd700 1 -- 172.21.15.90:0/2999560166 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v404) ==== 87+0+0 (unknown 667993436 0 0) 0x7f7aec057640 con 0x7f7af410f310 2024-04-14T15:17:57.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:57.890+0000 7f7afbddf700 10 librados: watch_flush enter 2024-04-14T15:17:57.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:57.890+0000 7f7afbddf700 10 librados: watch_flush exit 2024-04-14T15:17:57.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:57.890+0000 7f7afbddf700 20 client.37230.objecter shutdown clearing up homeless session... 2024-04-14T15:17:57.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:57.890+0000 7f7afbddf700 10 client.37230.objecter successfully canceled tick 2024-04-14T15:17:57.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:57.891+0000 7f7afbddf700 1 -- 172.21.15.90:0/2999560166 >> v1:172.21.15.90:6800/420935427 conn(0x7f7ae0071440 legacy=0x7f7ae0073900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:57.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:57.891+0000 7f7afbddf700 1 -- 172.21.15.90:0/2999560166 >> v1:172.21.15.90:6789/0 conn(0x7f7af410f310 legacy=0x7f7af41a8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:17:57.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:57.894+0000 7f7afbddf700 1 -- 172.21.15.90:0/2999560166 shutdown_connections 2024-04-14T15:17:57.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:57.894+0000 7f7afbddf700 1 -- 172.21.15.90:0/2999560166 >> 172.21.15.90:0/2999560166 conn(0x7f7af40fe380 msgr2=0x7f7af4100f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:17:57.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:57.894+0000 7f7afbddf700 1 -- 172.21.15.90:0/2999560166 shutdown_connections 2024-04-14T15:17:57.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:57.894+0000 7f7afbddf700 1 -- 172.21.15.90:0/2999560166 wait complete. 2024-04-14T15:17:57.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:17:57.894+0000 7f7afbddf700 1 librados: shutdown 2024-04-14T15:17:57.896 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is set 2024-04-14T15:17:57.936 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:17:57.961 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:57.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:57.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:17:58.063 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:17:58.089 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:58.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:58.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:17:58.191 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:17:58.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:58.215 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:58.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:17:58.314 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:57 smithi090 ceph-mon[41670]: 110.11 deep-scrub starts 2024-04-14T15:17:58.314 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:57 smithi090 ceph-mon[41670]: 110.11 deep-scrub ok 2024-04-14T15:17:58.314 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:57 smithi090 ceph-mon[41670]: 5.8 deep-scrub starts 2024-04-14T15:17:58.314 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:57 smithi090 ceph-mon[41670]: 5.8 deep-scrub ok 2024-04-14T15:17:58.314 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:57 smithi090 ceph-mon[41670]: pgmap v554: 238 pgs: 32 unknown, 206 active+clean; 8.4 MiB data, 606 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-04-14T15:17:58.314 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:57 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2999560166' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:17:58.314 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:57 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:17:58.314 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:57 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:17:58.314 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:57 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-53"}]: dispatch 2024-04-14T15:17:58.314 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:57 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-53"}]: dispatch 2024-04-14T15:17:58.314 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:57 smithi090 ceph-mon[41670]: 6.c deep-scrub starts 2024-04-14T15:17:58.315 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:57 smithi090 ceph-mon[41670]: 6.c deep-scrub ok 2024-04-14T15:17:58.315 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:57 smithi090 ceph-mon[41670]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:17:58.315 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:57 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2999560166' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-14T15:17:58.315 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:57 smithi090 ceph-mon[41670]: osdmap e404: 8 total, 8 up, 8 in 2024-04-14T15:17:58.315 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:57 smithi090 ceph-mon[30399]: 110.11 deep-scrub starts 2024-04-14T15:17:58.315 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:57 smithi090 ceph-mon[30399]: 110.11 deep-scrub ok 2024-04-14T15:17:58.315 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:57 smithi090 ceph-mon[30399]: 5.8 deep-scrub starts 2024-04-14T15:17:58.315 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:57 smithi090 ceph-mon[30399]: 5.8 deep-scrub ok 2024-04-14T15:17:58.315 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:57 smithi090 ceph-mon[30399]: pgmap v554: 238 pgs: 32 unknown, 206 active+clean; 8.4 MiB data, 606 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-04-14T15:17:58.315 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:57 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2999560166' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:17:58.315 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:57 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:17:58.315 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:57 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:17:58.316 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:57 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-53"}]: dispatch 2024-04-14T15:17:58.316 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:57 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-53"}]: dispatch 2024-04-14T15:17:58.316 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:57 smithi090 ceph-mon[30399]: 6.c deep-scrub starts 2024-04-14T15:17:58.316 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:57 smithi090 ceph-mon[30399]: 6.c deep-scrub ok 2024-04-14T15:17:58.316 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:57 smithi090 ceph-mon[30399]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:17:58.316 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:57 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2999560166' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-14T15:17:58.316 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:57 smithi090 ceph-mon[30399]: osdmap e404: 8 total, 8 up, 8 in 2024-04-14T15:17:58.316 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:17:58.340 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:58.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:58.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:17:58.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:57 smithi092 ceph-mon[33966]: 110.11 deep-scrub starts 2024-04-14T15:17:58.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:57 smithi092 ceph-mon[33966]: 110.11 deep-scrub ok 2024-04-14T15:17:58.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:57 smithi092 ceph-mon[33966]: 5.8 deep-scrub starts 2024-04-14T15:17:58.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:57 smithi092 ceph-mon[33966]: 5.8 deep-scrub ok 2024-04-14T15:17:58.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:57 smithi092 ceph-mon[33966]: pgmap v554: 238 pgs: 32 unknown, 206 active+clean; 8.4 MiB data, 606 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-04-14T15:17:58.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:57 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2999560166' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:17:58.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:57 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:17:58.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:57 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:17:58.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:57 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-53"}]: dispatch 2024-04-14T15:17:58.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:57 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-53"}]: dispatch 2024-04-14T15:17:58.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:57 smithi092 ceph-mon[33966]: 6.c deep-scrub starts 2024-04-14T15:17:58.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:57 smithi092 ceph-mon[33966]: 6.c deep-scrub ok 2024-04-14T15:17:58.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:57 smithi092 ceph-mon[33966]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:17:58.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:57 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2999560166' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-14T15:17:58.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:57 smithi092 ceph-mon[33966]: osdmap e404: 8 total, 8 up, 8 in 2024-04-14T15:17:58.442 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:17:58.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:58.467 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:58.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:17:58.568 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:17:58.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:58.592 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:58.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:17:58.694 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:17:58.722 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:58.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:58.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:17:58.825 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:17:58.849 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:58.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:58.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:17:58.951 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:17:58.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:58.975 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:58.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:17:59.076 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:17:59.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:59.101 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:59.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:17:59.202 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:17:59.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:59.231 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:59.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:17:59.333 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:17:59.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:59.362 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:59.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:17:59.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:58 smithi092 ceph-mon[33966]: 6.10 scrub starts 2024-04-14T15:17:59.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:58 smithi092 ceph-mon[33966]: osdmap e405: 8 total, 8 up, 8 in 2024-04-14T15:17:59.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:58 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:17:59.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:17:58 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:17:59.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:58 smithi090 ceph-mon[30399]: 6.10 scrub starts 2024-04-14T15:17:59.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:58 smithi090 ceph-mon[30399]: osdmap e405: 8 total, 8 up, 8 in 2024-04-14T15:17:59.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:58 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:17:59.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:17:58 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:17:59.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:58 smithi090 ceph-mon[41670]: 6.10 scrub starts 2024-04-14T15:17:59.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:58 smithi090 ceph-mon[41670]: osdmap e405: 8 total, 8 up, 8 in 2024-04-14T15:17:59.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:58 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:17:59.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:17:58 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:17:59.463 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:17:59.491 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:59.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:59.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:17:59.593 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:17:59.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:59.621 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:59.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:17:59.722 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:17:59.749 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:17:59.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:59.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:17:59.851 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:17:59.879 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:17:59.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:17:59.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:17:59.981 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:18:00.007 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:00.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:00.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:18:00.109 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:18:00.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:00.136 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:00.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:18:00.237 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:18:00.260 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:00.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:00.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:18:00.362 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:18:00.387 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:00.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:00.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:18:00.489 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:18:00.516 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:00.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:00.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:18:00.617 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:18:00.644 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:00.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:00.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:18:00.746 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:18:00.769 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:00.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:00.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:18:00.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:00.812+0000 7f1934d02700 1 -- 172.21.15.90:0/4017887259 >> v1:172.21.15.90:6789/0 conn(0x7f1930106f80 legacy=0x7f1930107360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:00.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:00.812+0000 7f1934d02700 1 -- 172.21.15.90:0/4017887259 shutdown_connections 2024-04-14T15:18:00.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:00.812+0000 7f1934d02700 1 -- 172.21.15.90:0/4017887259 >> 172.21.15.90:0/4017887259 conn(0x7f1930076aa0 msgr2=0x7f1930076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:18:00.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:00.812+0000 7f1934d02700 1 -- 172.21.15.90:0/4017887259 shutdown_connections 2024-04-14T15:18:00.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:00.812+0000 7f1934d02700 1 -- 172.21.15.90:0/4017887259 wait complete. 2024-04-14T15:18:00.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:00.813+0000 7f1934d02700 1 librados: starting msgr at 2024-04-14T15:18:00.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:00.813+0000 7f1934d02700 1 librados: starting objecter 2024-04-14T15:18:00.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:00.813+0000 7f1934d02700 1 Processor -- start 2024-04-14T15:18:00.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:00.813+0000 7f1934d02700 1 -- start start 2024-04-14T15:18:00.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:00.813+0000 7f1934d02700 1 librados: setting wanted keys 2024-04-14T15:18:00.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:00.813+0000 7f1934d02700 1 librados: calling monclient init 2024-04-14T15:18:00.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:00.813+0000 7f1934d02700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f19301012f0 con 0x7f1930106f80 2024-04-14T15:18:00.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:00.813+0000 7f1934d02700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f19301a8da0 con 0x7f193010d940 2024-04-14T15:18:00.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:00.813+0000 7f1934d02700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f19301a9fc0 con 0x7f1930109da0 2024-04-14T15:18:00.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:00.813+0000 7f192dd9b700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f1930109da0 0x7f19301a2f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:45776/0 (socket says 172.21.15.90:45776) 2024-04-14T15:18:00.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:00.813+0000 7f192dd9b700 1 -- 172.21.15.90:0/471317433 learned_addr learned my addr 172.21.15.90:0/471317433 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:18:00.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:00.813+0000 7f191f7fe700 10 client.?.objecter ms_handle_connect 0x7f1930106f80 2024-04-14T15:18:00.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:00.813+0000 7f191f7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:18:00.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:00.813+0000 7f191f7fe700 10 client.?.objecter ms_handle_connect 0x7f1930109da0 2024-04-14T15:18:00.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:00.813+0000 7f191f7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:18:00.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:00.814+0000 7f191f7fe700 1 -- 172.21.15.90:0/471317433 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 353805751 0 0) 0x7f19301012f0 con 0x7f1930106f80 2024-04-14T15:18:00.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:00.814+0000 7f191f7fe700 1 -- 172.21.15.90:0/471317433 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f19180035b0 con 0x7f1930106f80 2024-04-14T15:18:00.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:00.814+0000 7f191f7fe700 10 client.?.objecter ms_handle_connect 0x7f193010d940 2024-04-14T15:18:00.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:00.814+0000 7f191f7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:18:00.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:00.814+0000 7f191f7fe700 1 -- 172.21.15.90:0/471317433 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1005649316 0 0) 0x7f19301a9fc0 con 0x7f1930109da0 2024-04-14T15:18:00.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:00.814+0000 7f191f7fe700 1 -- 172.21.15.90:0/471317433 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f19301012f0 con 0x7f1930109da0 2024-04-14T15:18:00.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:00.814+0000 7f191f7fe700 1 -- 172.21.15.90:0/471317433 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 814706453 0 0) 0x7f19301a8da0 con 0x7f193010d940 2024-04-14T15:18:00.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:00.814+0000 7f191f7fe700 1 -- 172.21.15.90:0/471317433 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f19301a9fc0 con 0x7f193010d940 2024-04-14T15:18:00.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:00.814+0000 7f191f7fe700 1 -- 172.21.15.90:0/471317433 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1320012399 0 0) 0x7f19180035b0 con 0x7f1930106f80 2024-04-14T15:18:00.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:00.814+0000 7f191f7fe700 1 -- 172.21.15.90:0/471317433 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f19301a8da0 con 0x7f1930106f80 2024-04-14T15:18:00.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:00.814+0000 7f191f7fe700 1 -- 172.21.15.90:0/471317433 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f19240015d0 con 0x7f1930106f80 2024-04-14T15:18:00.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:00.814+0000 7f191f7fe700 1 -- 172.21.15.90:0/471317433 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 635403873 0 0) 0x7f19301012f0 con 0x7f1930109da0 2024-04-14T15:18:00.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:00.814+0000 7f191f7fe700 1 -- 172.21.15.90:0/471317433 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f19180035b0 con 0x7f1930109da0 2024-04-14T15:18:00.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:00.814+0000 7f191f7fe700 1 -- 172.21.15.90:0/471317433 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f19200018b0 con 0x7f1930109da0 2024-04-14T15:18:00.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:00.814+0000 7f191f7fe700 1 -- 172.21.15.90:0/471317433 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2172899219 0 0) 0x7f19301a9fc0 con 0x7f193010d940 2024-04-14T15:18:00.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:00.814+0000 7f191f7fe700 1 -- 172.21.15.90:0/471317433 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f19301012f0 con 0x7f193010d940 2024-04-14T15:18:00.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:00.814+0000 7f191f7fe700 1 -- 172.21.15.90:0/471317433 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f1928001c30 con 0x7f193010d940 2024-04-14T15:18:00.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:00.814+0000 7f191f7fe700 1 -- 172.21.15.90:0/471317433 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1616347929 0 0) 0x7f19301a8da0 con 0x7f1930106f80 2024-04-14T15:18:00.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:00.814+0000 7f191f7fe700 1 -- 172.21.15.90:0/471317433 >> v1:172.21.15.90:6790/0 conn(0x7f1930109da0 legacy=0x7f19301a2f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:00.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:00.814+0000 7f191f7fe700 1 -- 172.21.15.90:0/471317433 >> v1:172.21.15.92:6789/0 conn(0x7f193010d940 legacy=0x7f19301a6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:00.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:00.814+0000 7f191f7fe700 1 -- 172.21.15.90:0/471317433 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19301ab200 con 0x7f1930106f80 2024-04-14T15:18:00.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:00.814+0000 7f1934d02700 1 -- 172.21.15.90:0/471317433 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f19301aa1f0 con 0x7f1930106f80 2024-04-14T15:18:00.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:00.814+0000 7f1934d02700 10 client.37236.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:18:00.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:00.814+0000 7f1934d02700 1 -- 172.21.15.90:0/471317433 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f19301aa6b0 con 0x7f1930106f80 2024-04-14T15:18:00.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:00.814+0000 7f1934d02700 1 librados: init done 2024-04-14T15:18:00.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:00.814+0000 7f191f7fe700 1 -- 172.21.15.90:0/471317433 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f1924003760 con 0x7f1930106f80 2024-04-14T15:18:00.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:00.814+0000 7f191f7fe700 1 -- 172.21.15.90:0/471317433 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f1924004de0 con 0x7f1930106f80 2024-04-14T15:18:00.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:00.815+0000 7f1934d02700 1 -- 172.21.15.90:0/471317433 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1910005120 con 0x7f1930106f80 2024-04-14T15:18:00.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:00.816+0000 7f191f7fe700 1 -- 172.21.15.90:0/471317433 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f1924001b70 con 0x7f1930106f80 2024-04-14T15:18:00.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:00.816+0000 7f191f7fe700 1 -- 172.21.15.90:0/471317433 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(406..406 src has 51..406) ==== 8144+0+0 (unknown 3236006354 0 0) 0x7f1924089730 con 0x7f1930106f80 2024-04-14T15:18:00.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:00.816+0000 7f191f7fe700 10 client.37236.objecter ms_dispatch 0x7f1930001550 osd_map(406..406 src has 51..406) 2024-04-14T15:18:00.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:00.816+0000 7f191f7fe700 3 client.37236.objecter handle_osd_map got epochs [406,406] > 0 2024-04-14T15:18:00.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:00.816+0000 7f191f7fe700 3 client.37236.objecter handle_osd_map decoding full epoch 406 2024-04-14T15:18:00.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:00.817+0000 7f191f7fe700 20 client.37236.objecter dump_active .. 0 homeless 2024-04-14T15:18:00.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:00.817+0000 7f191f7fe700 10 client.37236.objecter ms_handle_connect 0x7f19180757d0 2024-04-14T15:18:00.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:00.821+0000 7f191f7fe700 1 -- 172.21.15.90:0/471317433 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f19240562a0 con 0x7f1930106f80 2024-04-14T15:18:00.872 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:00.906 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:00.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:00.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:01.008 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:18:01.034 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:01.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:01.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:18:01.042 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-14T15:18:01.043 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:18:01.043 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:00 smithi090 ceph-mon[41670]: pgmap v557: 238 pgs: 30 creating+peering, 2 unknown, 206 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-04-14T15:18:01.043 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:00 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:18:01.043 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:00 smithi090 ceph-mon[41670]: osdmap e406: 8 total, 8 up, 8 in 2024-04-14T15:18:01.043 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:00 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-14T15:18:01.043 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:00 smithi090 ceph-mon[41670]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:18:01.043 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:00 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:18:01.043 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:00 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:18:01.044 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:00 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-55"}]: dispatch 2024-04-14T15:18:01.044 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:00 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-55"}]: dispatch 2024-04-14T15:18:01.044 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:00 smithi090 ceph-mon[30399]: pgmap v557: 238 pgs: 30 creating+peering, 2 unknown, 206 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-04-14T15:18:01.044 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:00 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:18:01.044 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:00 smithi090 ceph-mon[30399]: osdmap e406: 8 total, 8 up, 8 in 2024-04-14T15:18:01.044 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:00 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-14T15:18:01.044 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:00 smithi090 ceph-mon[30399]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:18:01.044 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:00 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:18:01.044 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:00 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:18:01.044 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:00 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-55"}]: dispatch 2024-04-14T15:18:01.044 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:00 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-55"}]: dispatch 2024-04-14T15:18:01.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:01.067+0000 7f1934d02700 1 -- 172.21.15.90:0/471317433 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd reweight", "id": 6, "weight": 1.0} v 0) -- 0x7f1910005410 con 0x7f1930106f80 2024-04-14T15:18:01.136 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:18:01.161 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:01.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:01.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:18:01.263 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:18:01.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:01.287 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:01.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:18:01.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:00 smithi092 ceph-mon[33966]: pgmap v557: 238 pgs: 30 creating+peering, 2 unknown, 206 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-04-14T15:18:01.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:00 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:18:01.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:00 smithi092 ceph-mon[33966]: osdmap e406: 8 total, 8 up, 8 in 2024-04-14T15:18:01.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:00 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-14T15:18:01.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:00 smithi092 ceph-mon[33966]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:18:01.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:00 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:18:01.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:00 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:18:01.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:00 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-55"}]: dispatch 2024-04-14T15:18:01.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:00 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-55"}]: dispatch 2024-04-14T15:18:01.388 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:18:01.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:01.412 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:01.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:18:01.513 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:18:01.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:01.537 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:01.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:18:01.638 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:18:01.661 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:01.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:01.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:18:01.753 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-14T15:18:01.764 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:18:01.792 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:01.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:01.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:18:01.894 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:01.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:01.910+0000 7f191f7fe700 1 -- 172.21.15.90:0/471317433 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 6, "weight": 1.0}]=0 reweighted osd.6 to 1 (10000) v408) ==== 113+0+0 (unknown 2051357600 0 0) 0x7f1924059f50 con 0x7f1930106f80 2024-04-14T15:18:01.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:01.913+0000 7f1934d02700 10 librados: watch_flush enter 2024-04-14T15:18:01.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:01.913+0000 7f1934d02700 10 librados: watch_flush exit 2024-04-14T15:18:01.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:01.913+0000 7f1934d02700 20 client.37236.objecter shutdown clearing up homeless session... 2024-04-14T15:18:01.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:01.913+0000 7f1934d02700 10 client.37236.objecter successfully canceled tick 2024-04-14T15:18:01.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:01.913+0000 7f1934d02700 1 -- 172.21.15.90:0/471317433 >> v1:172.21.15.90:6800/420935427 conn(0x7f19180757d0 legacy=0x7f1918077c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:01.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:01.913+0000 7f1934d02700 1 -- 172.21.15.90:0/471317433 >> v1:172.21.15.90:6789/0 conn(0x7f1930106f80 legacy=0x7f1930100700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:01.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:01.913+0000 7f1934d02700 1 -- 172.21.15.90:0/471317433 shutdown_connections 2024-04-14T15:18:01.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:01.914+0000 7f1934d02700 1 -- 172.21.15.90:0/471317433 >> 172.21.15.90:0/471317433 conn(0x7f1930076aa0 msgr2=0x7f1930107c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:18:01.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:01.915+0000 7f1934d02700 1 -- 172.21.15.90:0/471317433 shutdown_connections 2024-04-14T15:18:01.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:01.915+0000 7f1934d02700 1 -- 172.21.15.90:0/471317433 wait complete. 2024-04-14T15:18:01.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:01.915+0000 7f1934d02700 1 librados: shutdown 2024-04-14T15:18:01.923 INFO:teuthology.orchestra.run.smithi090.stderr:reweighted osd.6 to 1 (10000) 2024-04-14T15:18:01.927 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:01.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:01.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:02.028 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:18:02.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:02.053 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:02.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:18:02.154 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:18:02.182 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:02.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:02.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:18:02.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:01 smithi090 ceph-mon[41670]: osdmap e407: 8 total, 8 up, 8 in 2024-04-14T15:18:02.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:01 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/471317433' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-14T15:18:02.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:01 smithi090 ceph-mon[30399]: osdmap e407: 8 total, 8 up, 8 in 2024-04-14T15:18:02.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:01 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/471317433' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-14T15:18:02.285 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:18:02.313 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:02.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:02.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:18:02.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:01 smithi092 ceph-mon[33966]: osdmap e407: 8 total, 8 up, 8 in 2024-04-14T15:18:02.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:01 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/471317433' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-14T15:18:02.416 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:18:02.444 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:02.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:02.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:18:02.545 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:18:02.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:02.573 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:02.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:18:02.658 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd reweight 7 1 2024-04-14T15:18:02.674 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:18:02.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:02.725 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:02.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:18:02.826 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:18:02.854 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:02.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:02.855 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:18:02.956 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:18:02.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:02.980 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:02.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:18:03.080 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-14T15:18:03.082 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:18:03.105 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:03.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:03.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:18:03.176 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:02 smithi090 ceph-mon[41670]: pgmap v560: 206 pgs: 206 active+clean; 8.4 MiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-04-14T15:18:03.176 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:02 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/471317433' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-14T15:18:03.176 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:02 smithi090 ceph-mon[41670]: osdmap e408: 8 total, 8 up, 8 in 2024-04-14T15:18:03.176 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:02 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:18:03.176 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:02 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:18:03.177 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:02 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:18:03.177 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:02 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:18:03.177 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:02 smithi090 ceph-mon[30399]: pgmap v560: 206 pgs: 206 active+clean; 8.4 MiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-04-14T15:18:03.177 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:02 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/471317433' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-14T15:18:03.177 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:02 smithi090 ceph-mon[30399]: osdmap e408: 8 total, 8 up, 8 in 2024-04-14T15:18:03.177 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:02 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:18:03.177 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:02 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:18:03.177 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:02 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:18:03.177 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:02 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:18:03.207 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:03.232 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:03.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:03.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:03.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.273+0000 7f58bfe2e700 1 -- 172.21.15.90:0/1484771203 >> v1:172.21.15.90:6789/0 conn(0x7f58b810b720 legacy=0x7f58b810db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:03.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.273+0000 7f58bfe2e700 1 -- 172.21.15.90:0/1484771203 shutdown_connections 2024-04-14T15:18:03.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.273+0000 7f58bfe2e700 1 -- 172.21.15.90:0/1484771203 >> 172.21.15.90:0/1484771203 conn(0x7f58b80fe3a0 msgr2=0x7f58b81007c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:18:03.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.273+0000 7f58bfe2e700 1 -- 172.21.15.90:0/1484771203 shutdown_connections 2024-04-14T15:18:03.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.273+0000 7f58bfe2e700 1 -- 172.21.15.90:0/1484771203 wait complete. 2024-04-14T15:18:03.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.274+0000 7f58bfe2e700 1 librados: starting msgr at 2024-04-14T15:18:03.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.274+0000 7f58bfe2e700 1 librados: starting objecter 2024-04-14T15:18:03.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.274+0000 7f58bfe2e700 1 Processor -- start 2024-04-14T15:18:03.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.274+0000 7f58bfe2e700 1 -- start start 2024-04-14T15:18:03.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.274+0000 7f58bfe2e700 1 librados: setting wanted keys 2024-04-14T15:18:03.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.274+0000 7f58bfe2e700 1 librados: calling monclient init 2024-04-14T15:18:03.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.274+0000 7f58bfe2e700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f58b810eec0 con 0x7f58b810b720 2024-04-14T15:18:03.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.274+0000 7f58bfe2e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f58b810f0d0 con 0x7f58b810f2f0 2024-04-14T15:18:03.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.274+0000 7f58bfe2e700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f58b81ac300 con 0x7f58b8108870 2024-04-14T15:18:03.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.274+0000 7f58bd3c9700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f58b810b720 0x7f58b8105c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50624/0 (socket says 172.21.15.90:50624) 2024-04-14T15:18:03.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.274+0000 7f58bd3c9700 1 -- 172.21.15.90:0/2065677670 learned_addr learned my addr 172.21.15.90:0/2065677670 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:18:03.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.274+0000 7f58aeffd700 10 client.?.objecter ms_handle_connect 0x7f58b810b720 2024-04-14T15:18:03.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.274+0000 7f58aeffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:18:03.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.274+0000 7f58aeffd700 10 client.?.objecter ms_handle_connect 0x7f58b8108870 2024-04-14T15:18:03.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.274+0000 7f58aeffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:18:03.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.274+0000 7f58aeffd700 10 client.?.objecter ms_handle_connect 0x7f58b810f2f0 2024-04-14T15:18:03.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.274+0000 7f58aeffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:18:03.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.274+0000 7f58aeffd700 1 -- 172.21.15.90:0/2065677670 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2208773645 0 0) 0x7f58b810eec0 con 0x7f58b810b720 2024-04-14T15:18:03.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.274+0000 7f58aeffd700 1 -- 172.21.15.90:0/2065677670 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f58a40035b0 con 0x7f58b810b720 2024-04-14T15:18:03.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.274+0000 7f58aeffd700 1 -- 172.21.15.90:0/2065677670 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2667155023 0 0) 0x7f58b81ac300 con 0x7f58b8108870 2024-04-14T15:18:03.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.274+0000 7f58aeffd700 1 -- 172.21.15.90:0/2065677670 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f58b810eec0 con 0x7f58b8108870 2024-04-14T15:18:03.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.274+0000 7f58aeffd700 1 -- 172.21.15.90:0/2065677670 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1424005329 0 0) 0x7f58b810f0d0 con 0x7f58b810f2f0 2024-04-14T15:18:03.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.274+0000 7f58aeffd700 1 -- 172.21.15.90:0/2065677670 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f58b81ac300 con 0x7f58b810f2f0 2024-04-14T15:18:03.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.275+0000 7f58aeffd700 1 -- 172.21.15.90:0/2065677670 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3839417299 0 0) 0x7f58a40035b0 con 0x7f58b810b720 2024-04-14T15:18:03.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.275+0000 7f58aeffd700 1 -- 172.21.15.90:0/2065677670 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f58b810f0d0 con 0x7f58b810b720 2024-04-14T15:18:03.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.275+0000 7f58aeffd700 1 -- 172.21.15.90:0/2065677670 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f58b40015d0 con 0x7f58b810b720 2024-04-14T15:18:03.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.275+0000 7f58aeffd700 1 -- 172.21.15.90:0/2065677670 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 229460404 0 0) 0x7f58b810eec0 con 0x7f58b8108870 2024-04-14T15:18:03.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.275+0000 7f58aeffd700 1 -- 172.21.15.90:0/2065677670 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f58a40035b0 con 0x7f58b8108870 2024-04-14T15:18:03.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.275+0000 7f58aeffd700 1 -- 172.21.15.90:0/2065677670 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f58a8001760 con 0x7f58b8108870 2024-04-14T15:18:03.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.275+0000 7f58aeffd700 1 -- 172.21.15.90:0/2065677670 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1422665853 0 0) 0x7f58b81ac300 con 0x7f58b810f2f0 2024-04-14T15:18:03.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.275+0000 7f58aeffd700 1 -- 172.21.15.90:0/2065677670 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f58b810eec0 con 0x7f58b810f2f0 2024-04-14T15:18:03.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.275+0000 7f58aeffd700 1 -- 172.21.15.90:0/2065677670 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f58b0001b70 con 0x7f58b810f2f0 2024-04-14T15:18:03.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.275+0000 7f58aeffd700 1 -- 172.21.15.90:0/2065677670 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2152728559 0 0) 0x7f58b810f0d0 con 0x7f58b810b720 2024-04-14T15:18:03.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.275+0000 7f58aeffd700 1 -- 172.21.15.90:0/2065677670 >> v1:172.21.15.90:6790/0 conn(0x7f58b8108870 legacy=0x7f58b8105510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:03.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.275+0000 7f58aeffd700 1 -- 172.21.15.90:0/2065677670 >> v1:172.21.15.92:6789/0 conn(0x7f58b810f2f0 legacy=0x7f58b81a8bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:03.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.275+0000 7f58aeffd700 1 -- 172.21.15.90:0/2065677670 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58b81ad540 con 0x7f58b810b720 2024-04-14T15:18:03.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.275+0000 7f58bfe2e700 1 -- 172.21.15.90:0/2065677670 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f58b81ac530 con 0x7f58b810b720 2024-04-14T15:18:03.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.275+0000 7f58bfe2e700 10 client.37242.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:18:03.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.275+0000 7f58bfe2e700 1 -- 172.21.15.90:0/2065677670 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f58b81ac9f0 con 0x7f58b810b720 2024-04-14T15:18:03.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.275+0000 7f58bfe2e700 1 librados: init done 2024-04-14T15:18:03.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.275+0000 7f58aeffd700 1 -- 172.21.15.90:0/2065677670 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f58b4003760 con 0x7f58b810b720 2024-04-14T15:18:03.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.275+0000 7f58aeffd700 1 -- 172.21.15.90:0/2065677670 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f58b4004de0 con 0x7f58b810b720 2024-04-14T15:18:03.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.276+0000 7f58bfe2e700 1 -- 172.21.15.90:0/2065677670 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f589c005100 con 0x7f58b810b720 2024-04-14T15:18:03.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.276+0000 7f58aeffd700 1 -- 172.21.15.90:0/2065677670 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f58b4001b70 con 0x7f58b810b720 2024-04-14T15:18:03.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.277+0000 7f58aeffd700 1 -- 172.21.15.90:0/2065677670 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(409..409 src has 51..409) ==== 8144+0+0 (unknown 3735850983 0 0) 0x7f58b408aa20 con 0x7f58b810b720 2024-04-14T15:18:03.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.277+0000 7f58aeffd700 10 client.37242.objecter ms_dispatch 0x7f58b8001550 osd_map(409..409 src has 51..409) 2024-04-14T15:18:03.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.277+0000 7f58aeffd700 3 client.37242.objecter handle_osd_map got epochs [409,409] > 0 2024-04-14T15:18:03.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.277+0000 7f58aeffd700 3 client.37242.objecter handle_osd_map decoding full epoch 409 2024-04-14T15:18:03.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.277+0000 7f58aeffd700 20 client.37242.objecter dump_active .. 0 homeless 2024-04-14T15:18:03.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.277+0000 7f58aeffd700 10 client.37242.objecter ms_handle_connect 0x7f58a4071420 2024-04-14T15:18:03.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.277+0000 7f58aeffd700 1 -- 172.21.15.90:0/2065677670 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v409) ==== 54+0+16 (unknown 657275760 0 2420000192) 0x7f58b408b060 con 0x7f58b810b720 2024-04-14T15:18:03.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.277+0000 7f58bfe2e700 10 client.37242.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:18:03.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.277+0000 7f58bfe2e700 15 client.37242.objecter _session_command_op_assign -1 1 2024-04-14T15:18:03.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.277+0000 7f58bfe2e700 20 client.37242.objecter _get_session s=0x7f589c0053e0 osd=0 3 2024-04-14T15:18:03.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.277+0000 7f58bfe2e700 20 client.37242.objecter put_session s=0x7f589c0053e0 osd=0 3 2024-04-14T15:18:03.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.277+0000 7f58bfe2e700 20 client.37242.objecter _get_session s=0x7f589c0053e0 osd=0 3 2024-04-14T15:18:03.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.277+0000 7f58bfe2e700 15 client.37242.objecter _session_command_op_remove -1 1 2024-04-14T15:18:03.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.277+0000 7f58bfe2e700 20 client.37242.objecter get_session s=0x7f589c0053e0 osd=0 3 2024-04-14T15:18:03.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.277+0000 7f58bfe2e700 15 client.37242.objecter _session_command_op_assign 0 1 2024-04-14T15:18:03.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.277+0000 7f58bfe2e700 20 client.37242.objecter put_session s=0x7f589c0053e0 osd=0 4 2024-04-14T15:18:03.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.277+0000 7f58bfe2e700 10 client.37242.objecter _send_command 1 2024-04-14T15:18:03.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.277+0000 7f58bfe2e700 1 -- 172.21.15.90:0/2065677670 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f589c004b50 con 0x7f589c004730 2024-04-14T15:18:03.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.278+0000 7f58aeffd700 10 client.37242.objecter ms_handle_connect 0x7f589c004730 2024-04-14T15:18:03.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.278+0000 7f58aeffd700 1 -- 172.21.15.90:0/2065677670 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f589c004b50 con 0x7f589c004730 2024-04-14T15:18:03.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.278+0000 7f58aeffd700 10 client.37242.objecter ms_dispatch 0x7f58b8001550 command_reply(tid 1: 0 ) 2024-04-14T15:18:03.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.278+0000 7f58aeffd700 10 client.37242.objecter _finish_command 1 = system:0 2024-04-14T15:18:03.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.278+0000 7f58aeffd700 20 client.37242.objecter put_session s=0x7f589c0053e0 osd=0 4 2024-04-14T15:18:03.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.278+0000 7f58aeffd700 15 client.37242.objecter _session_command_op_remove 0 1 2024-04-14T15:18:03.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.300+0000 7f58bfe2e700 10 client.37242.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:18:03.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.300+0000 7f58bfe2e700 15 client.37242.objecter _session_command_op_assign -1 2 2024-04-14T15:18:03.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.300+0000 7f58bfe2e700 20 client.37242.objecter _get_session s=0x7f589c0053e0 osd=0 3 2024-04-14T15:18:03.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.300+0000 7f58bfe2e700 20 client.37242.objecter put_session s=0x7f589c0053e0 osd=0 3 2024-04-14T15:18:03.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.300+0000 7f58bfe2e700 20 client.37242.objecter _get_session s=0x7f589c0053e0 osd=0 3 2024-04-14T15:18:03.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.300+0000 7f58bfe2e700 15 client.37242.objecter _session_command_op_remove -1 2 2024-04-14T15:18:03.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.300+0000 7f58bfe2e700 20 client.37242.objecter get_session s=0x7f589c0053e0 osd=0 3 2024-04-14T15:18:03.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.300+0000 7f58bfe2e700 15 client.37242.objecter _session_command_op_assign 0 2 2024-04-14T15:18:03.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.300+0000 7f58bfe2e700 20 client.37242.objecter put_session s=0x7f589c0053e0 osd=0 4 2024-04-14T15:18:03.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.300+0000 7f58bfe2e700 10 client.37242.objecter _send_command 2 2024-04-14T15:18:03.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.300+0000 7f58bfe2e700 1 -- 172.21.15.90:0/2065677670 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f589c0079f0 con 0x7f589c004730 2024-04-14T15:18:03.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.303+0000 7f58aeffd700 1 -- 172.21.15.90:0/2065677670 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f589c0079f0 con 0x7f589c004730 2024-04-14T15:18:03.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.303+0000 7f58aeffd700 10 client.37242.objecter ms_dispatch 0x7f58b8001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:18:03.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.303+0000 7f58aeffd700 10 client.37242.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:18:03.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.303+0000 7f58aeffd700 20 client.37242.objecter put_session s=0x7f589c0053e0 osd=0 4 2024-04-14T15:18:03.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.303+0000 7f58aeffd700 15 client.37242.objecter _session_command_op_remove 0 2 2024-04-14T15:18:03.304 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:18:03.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.303+0000 7f58bfe2e700 10 client.37242.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:18:03.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.303+0000 7f58bfe2e700 15 client.37242.objecter _session_command_op_assign -1 3 2024-04-14T15:18:03.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.303+0000 7f58bfe2e700 20 client.37242.objecter _get_session s=0x7f589c004110 osd=1 3 2024-04-14T15:18:03.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.303+0000 7f58bfe2e700 20 client.37242.objecter put_session s=0x7f589c004110 osd=1 3 2024-04-14T15:18:03.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.303+0000 7f58bfe2e700 20 client.37242.objecter _get_session s=0x7f589c004110 osd=1 3 2024-04-14T15:18:03.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.303+0000 7f58bfe2e700 15 client.37242.objecter _session_command_op_remove -1 3 2024-04-14T15:18:03.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.303+0000 7f58bfe2e700 20 client.37242.objecter get_session s=0x7f589c004110 osd=1 3 2024-04-14T15:18:03.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.303+0000 7f58bfe2e700 15 client.37242.objecter _session_command_op_assign 1 3 2024-04-14T15:18:03.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.303+0000 7f58bfe2e700 20 client.37242.objecter put_session s=0x7f589c004110 osd=1 4 2024-04-14T15:18:03.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.303+0000 7f58bfe2e700 10 client.37242.objecter _send_command 3 2024-04-14T15:18:03.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.303+0000 7f58bfe2e700 1 -- 172.21.15.90:0/2065677670 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f589c002180 con 0x7f589c004280 2024-04-14T15:18:03.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.304+0000 7f58aeffd700 10 client.37242.objecter ms_handle_connect 0x7f589c004280 2024-04-14T15:18:03.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.304+0000 7f58aeffd700 1 -- 172.21.15.90:0/2065677670 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f589c002180 con 0x7f589c004280 2024-04-14T15:18:03.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.304+0000 7f58aeffd700 10 client.37242.objecter ms_dispatch 0x7f58b8001550 command_reply(tid 3: 0 ) 2024-04-14T15:18:03.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.304+0000 7f58aeffd700 10 client.37242.objecter _finish_command 3 = system:0 2024-04-14T15:18:03.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.304+0000 7f58aeffd700 20 client.37242.objecter put_session s=0x7f589c004110 osd=1 4 2024-04-14T15:18:03.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.304+0000 7f58aeffd700 15 client.37242.objecter _session_command_op_remove 1 3 2024-04-14T15:18:03.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.323+0000 7f58bfe2e700 10 client.37242.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:18:03.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.323+0000 7f58bfe2e700 15 client.37242.objecter _session_command_op_assign -1 4 2024-04-14T15:18:03.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.323+0000 7f58bfe2e700 20 client.37242.objecter _get_session s=0x7f589c004110 osd=1 3 2024-04-14T15:18:03.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.323+0000 7f58bfe2e700 20 client.37242.objecter put_session s=0x7f589c004110 osd=1 3 2024-04-14T15:18:03.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.323+0000 7f58bfe2e700 20 client.37242.objecter _get_session s=0x7f589c004110 osd=1 3 2024-04-14T15:18:03.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.323+0000 7f58bfe2e700 15 client.37242.objecter _session_command_op_remove -1 4 2024-04-14T15:18:03.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.323+0000 7f58bfe2e700 20 client.37242.objecter get_session s=0x7f589c004110 osd=1 3 2024-04-14T15:18:03.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.323+0000 7f58bfe2e700 15 client.37242.objecter _session_command_op_assign 1 4 2024-04-14T15:18:03.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.323+0000 7f58bfe2e700 20 client.37242.objecter put_session s=0x7f589c004110 osd=1 4 2024-04-14T15:18:03.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.323+0000 7f58bfe2e700 10 client.37242.objecter _send_command 4 2024-04-14T15:18:03.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.323+0000 7f58bfe2e700 1 -- 172.21.15.90:0/2065677670 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f589c001600 con 0x7f589c004280 2024-04-14T15:18:03.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.326+0000 7f58aeffd700 1 -- 172.21.15.90:0/2065677670 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f589c001600 con 0x7f589c004280 2024-04-14T15:18:03.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.326+0000 7f58aeffd700 10 client.37242.objecter ms_dispatch 0x7f58b8001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:18:03.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.326+0000 7f58aeffd700 10 client.37242.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:18:03.327 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:18:03.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.326+0000 7f58aeffd700 20 client.37242.objecter put_session s=0x7f589c004110 osd=1 4 2024-04-14T15:18:03.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.326+0000 7f58aeffd700 15 client.37242.objecter _session_command_op_remove 1 4 2024-04-14T15:18:03.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.326+0000 7f58bfe2e700 10 client.37242.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:18:03.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.326+0000 7f58bfe2e700 15 client.37242.objecter _session_command_op_assign -1 5 2024-04-14T15:18:03.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.326+0000 7f58bfe2e700 20 client.37242.objecter _get_session s=0x7f589c008370 osd=2 3 2024-04-14T15:18:03.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.326+0000 7f58bfe2e700 20 client.37242.objecter put_session s=0x7f589c008370 osd=2 3 2024-04-14T15:18:03.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.326+0000 7f58bfe2e700 20 client.37242.objecter _get_session s=0x7f589c008370 osd=2 3 2024-04-14T15:18:03.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.326+0000 7f58bfe2e700 15 client.37242.objecter _session_command_op_remove -1 5 2024-04-14T15:18:03.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.326+0000 7f58bfe2e700 20 client.37242.objecter get_session s=0x7f589c008370 osd=2 3 2024-04-14T15:18:03.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.326+0000 7f58bfe2e700 15 client.37242.objecter _session_command_op_assign 2 5 2024-04-14T15:18:03.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.326+0000 7f58bfe2e700 20 client.37242.objecter put_session s=0x7f589c008370 osd=2 4 2024-04-14T15:18:03.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.326+0000 7f58bfe2e700 10 client.37242.objecter _send_command 5 2024-04-14T15:18:03.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.326+0000 7f58bfe2e700 1 -- 172.21.15.90:0/2065677670 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f589c0110e0 con 0x7f589c00d5d0 2024-04-14T15:18:03.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.327+0000 7f58aeffd700 10 client.37242.objecter ms_handle_connect 0x7f589c00d5d0 2024-04-14T15:18:03.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.328+0000 7f58aeffd700 1 -- 172.21.15.90:0/2065677670 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f589c0110e0 con 0x7f589c00d5d0 2024-04-14T15:18:03.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.328+0000 7f58aeffd700 10 client.37242.objecter ms_dispatch 0x7f58b8001550 command_reply(tid 5: 0 ) 2024-04-14T15:18:03.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.328+0000 7f58aeffd700 10 client.37242.objecter _finish_command 5 = system:0 2024-04-14T15:18:03.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.328+0000 7f58aeffd700 20 client.37242.objecter put_session s=0x7f589c008370 osd=2 4 2024-04-14T15:18:03.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.328+0000 7f58aeffd700 15 client.37242.objecter _session_command_op_remove 2 5 2024-04-14T15:18:03.330 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:18:03.334 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:18:03.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.351+0000 7f58bfe2e700 10 client.37242.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:18:03.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.351+0000 7f58bfe2e700 15 client.37242.objecter _session_command_op_assign -1 6 2024-04-14T15:18:03.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.351+0000 7f58bfe2e700 20 client.37242.objecter _get_session s=0x7f589c008370 osd=2 3 2024-04-14T15:18:03.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.351+0000 7f58bfe2e700 20 client.37242.objecter put_session s=0x7f589c008370 osd=2 3 2024-04-14T15:18:03.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.351+0000 7f58bfe2e700 20 client.37242.objecter _get_session s=0x7f589c008370 osd=2 3 2024-04-14T15:18:03.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.351+0000 7f58bfe2e700 15 client.37242.objecter _session_command_op_remove -1 6 2024-04-14T15:18:03.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.351+0000 7f58bfe2e700 20 client.37242.objecter get_session s=0x7f589c008370 osd=2 3 2024-04-14T15:18:03.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.351+0000 7f58bfe2e700 15 client.37242.objecter _session_command_op_assign 2 6 2024-04-14T15:18:03.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.351+0000 7f58bfe2e700 20 client.37242.objecter put_session s=0x7f589c008370 osd=2 4 2024-04-14T15:18:03.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.351+0000 7f58bfe2e700 10 client.37242.objecter _send_command 6 2024-04-14T15:18:03.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.351+0000 7f58bfe2e700 1 -- 172.21.15.90:0/2065677670 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f589c007eb0 con 0x7f589c00d5d0 2024-04-14T15:18:03.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.353+0000 7f58aeffd700 1 -- 172.21.15.90:0/2065677670 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f589c007eb0 con 0x7f589c00d5d0 2024-04-14T15:18:03.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.353+0000 7f58aeffd700 10 client.37242.objecter ms_dispatch 0x7f58b8001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:18:03.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.353+0000 7f58aeffd700 10 client.37242.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:18:03.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.353+0000 7f58aeffd700 20 client.37242.objecter put_session s=0x7f589c008370 osd=2 4 2024-04-14T15:18:03.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.353+0000 7f58aeffd700 15 client.37242.objecter _session_command_op_remove 2 6 2024-04-14T15:18:03.354 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:18:03.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.353+0000 7f58bfe2e700 10 client.37242.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:18:03.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.353+0000 7f58bfe2e700 15 client.37242.objecter _session_command_op_assign -1 7 2024-04-14T15:18:03.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.353+0000 7f58bfe2e700 20 client.37242.objecter _get_session s=0x7f589c00cc20 osd=3 3 2024-04-14T15:18:03.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.353+0000 7f58bfe2e700 20 client.37242.objecter put_session s=0x7f589c00cc20 osd=3 3 2024-04-14T15:18:03.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.353+0000 7f58bfe2e700 20 client.37242.objecter _get_session s=0x7f589c00cc20 osd=3 3 2024-04-14T15:18:03.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.353+0000 7f58bfe2e700 15 client.37242.objecter _session_command_op_remove -1 7 2024-04-14T15:18:03.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.353+0000 7f58bfe2e700 20 client.37242.objecter get_session s=0x7f589c00cc20 osd=3 3 2024-04-14T15:18:03.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.353+0000 7f58bfe2e700 15 client.37242.objecter _session_command_op_assign 3 7 2024-04-14T15:18:03.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.353+0000 7f58bfe2e700 20 client.37242.objecter put_session s=0x7f589c00cc20 osd=3 4 2024-04-14T15:18:03.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.353+0000 7f58bfe2e700 10 client.37242.objecter _send_command 7 2024-04-14T15:18:03.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.353+0000 7f58bfe2e700 1 -- 172.21.15.90:0/2065677670 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f589c015f20 con 0x7f589c012410 2024-04-14T15:18:03.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.354+0000 7f58aeffd700 10 client.37242.objecter ms_handle_connect 0x7f589c012410 2024-04-14T15:18:03.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.355+0000 7f58aeffd700 1 -- 172.21.15.90:0/2065677670 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f589c015f20 con 0x7f589c012410 2024-04-14T15:18:03.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.355+0000 7f58aeffd700 10 client.37242.objecter ms_dispatch 0x7f58b8001550 command_reply(tid 7: 0 ) 2024-04-14T15:18:03.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.355+0000 7f58aeffd700 10 client.37242.objecter _finish_command 7 = system:0 2024-04-14T15:18:03.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.355+0000 7f58aeffd700 20 client.37242.objecter put_session s=0x7f589c00cc20 osd=3 4 2024-04-14T15:18:03.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.355+0000 7f58aeffd700 15 client.37242.objecter _session_command_op_remove 3 7 2024-04-14T15:18:03.360 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:03.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:03.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:18:03.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:02 smithi092 ceph-mon[33966]: pgmap v560: 206 pgs: 206 active+clean; 8.4 MiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-04-14T15:18:03.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:02 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/471317433' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-14T15:18:03.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:02 smithi092 ceph-mon[33966]: osdmap e408: 8 total, 8 up, 8 in 2024-04-14T15:18:03.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:02 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:18:03.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:02 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:18:03.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:02 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:18:03.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:02 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:18:03.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.376+0000 7f58bfe2e700 10 client.37242.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:18:03.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.376+0000 7f58bfe2e700 15 client.37242.objecter _session_command_op_assign -1 8 2024-04-14T15:18:03.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.376+0000 7f58bfe2e700 20 client.37242.objecter _get_session s=0x7f589c00cc20 osd=3 3 2024-04-14T15:18:03.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.376+0000 7f58bfe2e700 20 client.37242.objecter put_session s=0x7f589c00cc20 osd=3 3 2024-04-14T15:18:03.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.376+0000 7f58bfe2e700 20 client.37242.objecter _get_session s=0x7f589c00cc20 osd=3 3 2024-04-14T15:18:03.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.376+0000 7f58bfe2e700 15 client.37242.objecter _session_command_op_remove -1 8 2024-04-14T15:18:03.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.376+0000 7f58bfe2e700 20 client.37242.objecter get_session s=0x7f589c00cc20 osd=3 3 2024-04-14T15:18:03.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.376+0000 7f58bfe2e700 15 client.37242.objecter _session_command_op_assign 3 8 2024-04-14T15:18:03.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.376+0000 7f58bfe2e700 20 client.37242.objecter put_session s=0x7f589c00cc20 osd=3 4 2024-04-14T15:18:03.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.376+0000 7f58bfe2e700 10 client.37242.objecter _send_command 8 2024-04-14T15:18:03.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.376+0000 7f58bfe2e700 1 -- 172.21.15.90:0/2065677670 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f589c00c700 con 0x7f589c012410 2024-04-14T15:18:03.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.378+0000 7f58aeffd700 1 -- 172.21.15.90:0/2065677670 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f589c00c700 con 0x7f589c012410 2024-04-14T15:18:03.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.378+0000 7f58aeffd700 10 client.37242.objecter ms_dispatch 0x7f58b8001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:18:03.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.378+0000 7f58aeffd700 10 client.37242.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:18:03.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.378+0000 7f58aeffd700 20 client.37242.objecter put_session s=0x7f589c00cc20 osd=3 4 2024-04-14T15:18:03.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.378+0000 7f58aeffd700 15 client.37242.objecter _session_command_op_remove 3 8 2024-04-14T15:18:03.379 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:18:03.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.378+0000 7f58bfe2e700 10 client.37242.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:18:03.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.378+0000 7f58bfe2e700 15 client.37242.objecter _session_command_op_assign -1 9 2024-04-14T15:18:03.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.379+0000 7f58bfe2e700 20 client.37242.objecter _get_session s=0x7f589c00c700 osd=4 3 2024-04-14T15:18:03.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.379+0000 7f58bfe2e700 20 client.37242.objecter put_session s=0x7f589c00c700 osd=4 3 2024-04-14T15:18:03.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.379+0000 7f58bfe2e700 20 client.37242.objecter _get_session s=0x7f589c00c700 osd=4 3 2024-04-14T15:18:03.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.379+0000 7f58bfe2e700 15 client.37242.objecter _session_command_op_remove -1 9 2024-04-14T15:18:03.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.379+0000 7f58bfe2e700 20 client.37242.objecter get_session s=0x7f589c00c700 osd=4 3 2024-04-14T15:18:03.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.379+0000 7f58bfe2e700 15 client.37242.objecter _session_command_op_assign 4 9 2024-04-14T15:18:03.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.379+0000 7f58bfe2e700 20 client.37242.objecter put_session s=0x7f589c00c700 osd=4 4 2024-04-14T15:18:03.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.379+0000 7f58bfe2e700 10 client.37242.objecter _send_command 9 2024-04-14T15:18:03.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.379+0000 7f58bfe2e700 1 -- 172.21.15.90:0/2065677670 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f589c01a9f0 con 0x7f589c011f70 2024-04-14T15:18:03.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.379+0000 7f58aeffd700 10 client.37242.objecter ms_handle_connect 0x7f589c011f70 2024-04-14T15:18:03.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.380+0000 7f58aeffd700 1 -- 172.21.15.90:0/2065677670 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f589c01a9f0 con 0x7f589c011f70 2024-04-14T15:18:03.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.380+0000 7f58aeffd700 10 client.37242.objecter ms_dispatch 0x7f58b8001550 command_reply(tid 9: 0 ) 2024-04-14T15:18:03.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.380+0000 7f58aeffd700 10 client.37242.objecter _finish_command 9 = system:0 2024-04-14T15:18:03.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.380+0000 7f58aeffd700 20 client.37242.objecter put_session s=0x7f589c00c700 osd=4 4 2024-04-14T15:18:03.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.380+0000 7f58aeffd700 15 client.37242.objecter _session_command_op_remove 4 9 2024-04-14T15:18:03.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.396+0000 7f58bfe2e700 10 client.37242.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:18:03.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.396+0000 7f58bfe2e700 15 client.37242.objecter _session_command_op_assign -1 10 2024-04-14T15:18:03.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.396+0000 7f58bfe2e700 20 client.37242.objecter _get_session s=0x7f589c00c700 osd=4 3 2024-04-14T15:18:03.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.396+0000 7f58bfe2e700 20 client.37242.objecter put_session s=0x7f589c00c700 osd=4 3 2024-04-14T15:18:03.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.396+0000 7f58bfe2e700 20 client.37242.objecter _get_session s=0x7f589c00c700 osd=4 3 2024-04-14T15:18:03.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.396+0000 7f58bfe2e700 15 client.37242.objecter _session_command_op_remove -1 10 2024-04-14T15:18:03.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.396+0000 7f58bfe2e700 20 client.37242.objecter get_session s=0x7f589c00c700 osd=4 3 2024-04-14T15:18:03.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.396+0000 7f58bfe2e700 15 client.37242.objecter _session_command_op_assign 4 10 2024-04-14T15:18:03.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.396+0000 7f58bfe2e700 20 client.37242.objecter put_session s=0x7f589c00c700 osd=4 4 2024-04-14T15:18:03.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.396+0000 7f58bfe2e700 10 client.37242.objecter _send_command 10 2024-04-14T15:18:03.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.396+0000 7f58bfe2e700 1 -- 172.21.15.90:0/2065677670 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f589c01c3a0 con 0x7f589c011f70 2024-04-14T15:18:03.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.398+0000 7f58aeffd700 1 -- 172.21.15.90:0/2065677670 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f589c01c3a0 con 0x7f589c011f70 2024-04-14T15:18:03.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.398+0000 7f58aeffd700 10 client.37242.objecter ms_dispatch 0x7f58b8001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:18:03.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.398+0000 7f58aeffd700 10 client.37242.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:18:03.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.398+0000 7f58aeffd700 20 client.37242.objecter put_session s=0x7f589c00c700 osd=4 4 2024-04-14T15:18:03.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.398+0000 7f58aeffd700 15 client.37242.objecter _session_command_op_remove 4 10 2024-04-14T15:18:03.399 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:18:03.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.399+0000 7f58bfe2e700 10 client.37242.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:18:03.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.399+0000 7f58bfe2e700 15 client.37242.objecter _session_command_op_assign -1 11 2024-04-14T15:18:03.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.399+0000 7f58bfe2e700 20 client.37242.objecter _get_session s=0x7f589c01c3a0 osd=5 3 2024-04-14T15:18:03.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.399+0000 7f58bfe2e700 20 client.37242.objecter put_session s=0x7f589c01c3a0 osd=5 3 2024-04-14T15:18:03.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.399+0000 7f58bfe2e700 20 client.37242.objecter _get_session s=0x7f589c01c3a0 osd=5 3 2024-04-14T15:18:03.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.399+0000 7f58bfe2e700 15 client.37242.objecter _session_command_op_remove -1 11 2024-04-14T15:18:03.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.399+0000 7f58bfe2e700 20 client.37242.objecter get_session s=0x7f589c01c3a0 osd=5 3 2024-04-14T15:18:03.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.399+0000 7f58bfe2e700 15 client.37242.objecter _session_command_op_assign 5 11 2024-04-14T15:18:03.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.399+0000 7f58bfe2e700 20 client.37242.objecter put_session s=0x7f589c01c3a0 osd=5 4 2024-04-14T15:18:03.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.399+0000 7f58bfe2e700 10 client.37242.objecter _send_command 11 2024-04-14T15:18:03.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.399+0000 7f58bfe2e700 1 -- 172.21.15.90:0/2065677670 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f589c01cf40 con 0x7f589c01cac0 2024-04-14T15:18:03.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.399+0000 7f58aeffd700 10 client.37242.objecter ms_handle_connect 0x7f589c01cac0 2024-04-14T15:18:03.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.400+0000 7f58aeffd700 1 -- 172.21.15.90:0/2065677670 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f589c01cf40 con 0x7f589c01cac0 2024-04-14T15:18:03.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.400+0000 7f58aeffd700 10 client.37242.objecter ms_dispatch 0x7f58b8001550 command_reply(tid 11: 0 ) 2024-04-14T15:18:03.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.400+0000 7f58aeffd700 10 client.37242.objecter _finish_command 11 = system:0 2024-04-14T15:18:03.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.400+0000 7f58aeffd700 20 client.37242.objecter put_session s=0x7f589c01c3a0 osd=5 4 2024-04-14T15:18:03.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.400+0000 7f58aeffd700 15 client.37242.objecter _session_command_op_remove 5 11 2024-04-14T15:18:03.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.416+0000 7f58bfe2e700 10 client.37242.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:18:03.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.416+0000 7f58bfe2e700 15 client.37242.objecter _session_command_op_assign -1 12 2024-04-14T15:18:03.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.416+0000 7f58bfe2e700 20 client.37242.objecter _get_session s=0x7f589c01c3a0 osd=5 3 2024-04-14T15:18:03.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.416+0000 7f58bfe2e700 20 client.37242.objecter put_session s=0x7f589c01c3a0 osd=5 3 2024-04-14T15:18:03.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.416+0000 7f58bfe2e700 20 client.37242.objecter _get_session s=0x7f589c01c3a0 osd=5 3 2024-04-14T15:18:03.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.416+0000 7f58bfe2e700 15 client.37242.objecter _session_command_op_remove -1 12 2024-04-14T15:18:03.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.416+0000 7f58bfe2e700 20 client.37242.objecter get_session s=0x7f589c01c3a0 osd=5 3 2024-04-14T15:18:03.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.416+0000 7f58bfe2e700 15 client.37242.objecter _session_command_op_assign 5 12 2024-04-14T15:18:03.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.416+0000 7f58bfe2e700 20 client.37242.objecter put_session s=0x7f589c01c3a0 osd=5 4 2024-04-14T15:18:03.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.416+0000 7f58bfe2e700 10 client.37242.objecter _send_command 12 2024-04-14T15:18:03.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.416+0000 7f58bfe2e700 1 -- 172.21.15.90:0/2065677670 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f589c01d170 con 0x7f589c01cac0 2024-04-14T15:18:03.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.418+0000 7f58aeffd700 1 -- 172.21.15.90:0/2065677670 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f589c01d170 con 0x7f589c01cac0 2024-04-14T15:18:03.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.418+0000 7f58aeffd700 10 client.37242.objecter ms_dispatch 0x7f58b8001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:18:03.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.418+0000 7f58aeffd700 10 client.37242.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:18:03.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.418+0000 7f58aeffd700 20 client.37242.objecter put_session s=0x7f589c01c3a0 osd=5 4 2024-04-14T15:18:03.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.418+0000 7f58aeffd700 15 client.37242.objecter _session_command_op_remove 5 12 2024-04-14T15:18:03.419 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:18:03.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.419+0000 7f58bfe2e700 10 client.37242.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:18:03.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.419+0000 7f58bfe2e700 15 client.37242.objecter _session_command_op_assign -1 13 2024-04-14T15:18:03.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.419+0000 7f58bfe2e700 20 client.37242.objecter _get_session s=0x7f589c01d170 osd=6 3 2024-04-14T15:18:03.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.419+0000 7f58bfe2e700 20 client.37242.objecter put_session s=0x7f589c01d170 osd=6 3 2024-04-14T15:18:03.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.419+0000 7f58bfe2e700 20 client.37242.objecter _get_session s=0x7f589c01d170 osd=6 3 2024-04-14T15:18:03.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.419+0000 7f58bfe2e700 15 client.37242.objecter _session_command_op_remove -1 13 2024-04-14T15:18:03.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.419+0000 7f58bfe2e700 20 client.37242.objecter get_session s=0x7f589c01d170 osd=6 3 2024-04-14T15:18:03.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.419+0000 7f58bfe2e700 15 client.37242.objecter _session_command_op_assign 6 13 2024-04-14T15:18:03.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.419+0000 7f58bfe2e700 20 client.37242.objecter put_session s=0x7f589c01d170 osd=6 4 2024-04-14T15:18:03.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.419+0000 7f58bfe2e700 10 client.37242.objecter _send_command 13 2024-04-14T15:18:03.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.419+0000 7f58bfe2e700 1 -- 172.21.15.90:0/2065677670 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f589c0235c0 con 0x7f589c01fab0 2024-04-14T15:18:03.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.419+0000 7f58aeffd700 10 client.37242.objecter ms_handle_connect 0x7f589c01fab0 2024-04-14T15:18:03.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.420+0000 7f58aeffd700 1 -- 172.21.15.90:0/2065677670 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f589c0235c0 con 0x7f589c01fab0 2024-04-14T15:18:03.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.420+0000 7f58aeffd700 10 client.37242.objecter ms_dispatch 0x7f58b8001550 command_reply(tid 13: 0 ) 2024-04-14T15:18:03.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.420+0000 7f58aeffd700 10 client.37242.objecter _finish_command 13 = system:0 2024-04-14T15:18:03.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.420+0000 7f58aeffd700 20 client.37242.objecter put_session s=0x7f589c01d170 osd=6 4 2024-04-14T15:18:03.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.420+0000 7f58aeffd700 15 client.37242.objecter _session_command_op_remove 6 13 2024-04-14T15:18:03.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.436+0000 7f58bfe2e700 10 client.37242.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:18:03.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.436+0000 7f58bfe2e700 15 client.37242.objecter _session_command_op_assign -1 14 2024-04-14T15:18:03.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.436+0000 7f58bfe2e700 20 client.37242.objecter _get_session s=0x7f589c01d170 osd=6 3 2024-04-14T15:18:03.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.436+0000 7f58bfe2e700 20 client.37242.objecter put_session s=0x7f589c01d170 osd=6 3 2024-04-14T15:18:03.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.436+0000 7f58bfe2e700 20 client.37242.objecter _get_session s=0x7f589c01d170 osd=6 3 2024-04-14T15:18:03.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.436+0000 7f58bfe2e700 15 client.37242.objecter _session_command_op_remove -1 14 2024-04-14T15:18:03.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.436+0000 7f58bfe2e700 20 client.37242.objecter get_session s=0x7f589c01d170 osd=6 3 2024-04-14T15:18:03.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.436+0000 7f58bfe2e700 15 client.37242.objecter _session_command_op_assign 6 14 2024-04-14T15:18:03.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.436+0000 7f58bfe2e700 20 client.37242.objecter put_session s=0x7f589c01d170 osd=6 4 2024-04-14T15:18:03.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.436+0000 7f58bfe2e700 10 client.37242.objecter _send_command 14 2024-04-14T15:18:03.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.436+0000 7f58bfe2e700 1 -- 172.21.15.90:0/2065677670 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f589c0235c0 con 0x7f589c01fab0 2024-04-14T15:18:03.439 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:18:03.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.438+0000 7f58aeffd700 1 -- 172.21.15.90:0/2065677670 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f589c0235c0 con 0x7f589c01fab0 2024-04-14T15:18:03.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.438+0000 7f58aeffd700 10 client.37242.objecter ms_dispatch 0x7f58b8001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:18:03.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.438+0000 7f58aeffd700 10 client.37242.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:18:03.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.438+0000 7f58aeffd700 20 client.37242.objecter put_session s=0x7f589c01d170 osd=6 4 2024-04-14T15:18:03.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.438+0000 7f58aeffd700 15 client.37242.objecter _session_command_op_remove 6 14 2024-04-14T15:18:03.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.439+0000 7f58bfe2e700 10 client.37242.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:18:03.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.439+0000 7f58bfe2e700 15 client.37242.objecter _session_command_op_assign -1 15 2024-04-14T15:18:03.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.439+0000 7f58bfe2e700 20 client.37242.objecter _get_session s=0x7f589c0235c0 osd=7 3 2024-04-14T15:18:03.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.439+0000 7f58bfe2e700 20 client.37242.objecter put_session s=0x7f589c0235c0 osd=7 3 2024-04-14T15:18:03.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.439+0000 7f58bfe2e700 20 client.37242.objecter _get_session s=0x7f589c0235c0 osd=7 3 2024-04-14T15:18:03.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.439+0000 7f58bfe2e700 15 client.37242.objecter _session_command_op_remove -1 15 2024-04-14T15:18:03.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.439+0000 7f58bfe2e700 20 client.37242.objecter get_session s=0x7f589c0235c0 osd=7 3 2024-04-14T15:18:03.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.439+0000 7f58bfe2e700 15 client.37242.objecter _session_command_op_assign 7 15 2024-04-14T15:18:03.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.439+0000 7f58bfe2e700 20 client.37242.objecter put_session s=0x7f589c0235c0 osd=7 4 2024-04-14T15:18:03.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.439+0000 7f58bfe2e700 10 client.37242.objecter _send_command 15 2024-04-14T15:18:03.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.439+0000 7f58bfe2e700 1 -- 172.21.15.90:0/2065677670 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f589c027890 con 0x7f589c023c40 2024-04-14T15:18:03.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.439+0000 7f58aeffd700 10 client.37242.objecter ms_handle_connect 0x7f589c023c40 2024-04-14T15:18:03.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.440+0000 7f58aeffd700 1 -- 172.21.15.90:0/2065677670 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f589c027890 con 0x7f589c023c40 2024-04-14T15:18:03.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.440+0000 7f58aeffd700 10 client.37242.objecter ms_dispatch 0x7f58b8001550 command_reply(tid 15: 0 ) 2024-04-14T15:18:03.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.440+0000 7f58aeffd700 10 client.37242.objecter _finish_command 15 = system:0 2024-04-14T15:18:03.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.440+0000 7f58aeffd700 20 client.37242.objecter put_session s=0x7f589c0235c0 osd=7 4 2024-04-14T15:18:03.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.440+0000 7f58aeffd700 15 client.37242.objecter _session_command_op_remove 7 15 2024-04-14T15:18:03.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.456+0000 7f58bfe2e700 10 client.37242.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:18:03.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.456+0000 7f58bfe2e700 15 client.37242.objecter _session_command_op_assign -1 16 2024-04-14T15:18:03.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.456+0000 7f58bfe2e700 20 client.37242.objecter _get_session s=0x7f589c0235c0 osd=7 3 2024-04-14T15:18:03.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.456+0000 7f58bfe2e700 20 client.37242.objecter put_session s=0x7f589c0235c0 osd=7 3 2024-04-14T15:18:03.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.456+0000 7f58bfe2e700 20 client.37242.objecter _get_session s=0x7f589c0235c0 osd=7 3 2024-04-14T15:18:03.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.456+0000 7f58bfe2e700 15 client.37242.objecter _session_command_op_remove -1 16 2024-04-14T15:18:03.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.456+0000 7f58bfe2e700 20 client.37242.objecter get_session s=0x7f589c0235c0 osd=7 3 2024-04-14T15:18:03.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.456+0000 7f58bfe2e700 15 client.37242.objecter _session_command_op_assign 7 16 2024-04-14T15:18:03.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.456+0000 7f58bfe2e700 20 client.37242.objecter put_session s=0x7f589c0235c0 osd=7 4 2024-04-14T15:18:03.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.456+0000 7f58bfe2e700 10 client.37242.objecter _send_command 16 2024-04-14T15:18:03.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.456+0000 7f58bfe2e700 1 -- 172.21.15.90:0/2065677670 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f589c029240 con 0x7f589c023c40 2024-04-14T15:18:03.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.458+0000 7f58aeffd700 1 -- 172.21.15.90:0/2065677670 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f589c029240 con 0x7f589c023c40 2024-04-14T15:18:03.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.458+0000 7f58aeffd700 10 client.37242.objecter ms_dispatch 0x7f58b8001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:18:03.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.458+0000 7f58aeffd700 10 client.37242.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:18:03.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.458+0000 7f58aeffd700 20 client.37242.objecter put_session s=0x7f589c0235c0 osd=7 4 2024-04-14T15:18:03.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.458+0000 7f58aeffd700 15 client.37242.objecter _session_command_op_remove 7 16 2024-04-14T15:18:03.460 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:18:03.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.458+0000 7f58bfe2e700 10 librados: watch_flush enter 2024-04-14T15:18:03.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.458+0000 7f58bfe2e700 10 librados: watch_flush exit 2024-04-14T15:18:03.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.458+0000 7f58bfe2e700 10 client.37242.objecter close_session for osd.0 2024-04-14T15:18:03.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.458+0000 7f58bfe2e700 1 -- 172.21.15.90:0/2065677670 >> v1:172.21.15.90:6801/3761806885 conn(0x7f589c004730 legacy=0x7f589c002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:03.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.458+0000 7f58bfe2e700 20 client.37242.objecter put_session s=0x7f589c0053e0 osd=0 1 2024-04-14T15:18:03.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.458+0000 7f58bfe2e700 10 client.37242.objecter close_session for osd.1 2024-04-14T15:18:03.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.458+0000 7f58bfe2e700 1 -- 172.21.15.90:0/2065677670 >> v1:172.21.15.90:6805/4170361699 conn(0x7f589c004280 legacy=0x7f589c00ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:03.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.458+0000 7f58bfe2e700 20 client.37242.objecter put_session s=0x7f589c004110 osd=1 1 2024-04-14T15:18:03.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.458+0000 7f58bfe2e700 10 client.37242.objecter close_session for osd.2 2024-04-14T15:18:03.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.458+0000 7f58bfe2e700 1 -- 172.21.15.90:0/2065677670 >> v1:172.21.15.90:6809/1686184251 conn(0x7f589c00d5d0 legacy=0x7f589c00f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:03.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.458+0000 7f58bfe2e700 20 client.37242.objecter put_session s=0x7f589c008370 osd=2 1 2024-04-14T15:18:03.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.458+0000 7f58bfe2e700 10 client.37242.objecter close_session for osd.3 2024-04-14T15:18:03.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.458+0000 7f58bfe2e700 1 -- 172.21.15.90:0/2065677670 >> v1:172.21.15.90:6813/2270166774 conn(0x7f589c012410 legacy=0x7f589c014800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:03.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.458+0000 7f58bfe2e700 20 client.37242.objecter put_session s=0x7f589c00cc20 osd=3 1 2024-04-14T15:18:03.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.458+0000 7f58bfe2e700 10 client.37242.objecter close_session for osd.4 2024-04-14T15:18:03.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.458+0000 7f58bfe2e700 1 -- 172.21.15.90:0/2065677670 >> v1:172.21.15.92:6800/3887549733 conn(0x7f589c011f70 legacy=0x7f589c019260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:03.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.458+0000 7f58bfe2e700 20 client.37242.objecter put_session s=0x7f589c00c700 osd=4 1 2024-04-14T15:18:03.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.458+0000 7f58bfe2e700 10 client.37242.objecter close_session for osd.5 2024-04-14T15:18:03.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.458+0000 7f58bfe2e700 1 -- 172.21.15.90:0/2065677670 >> v1:172.21.15.92:6804/3244561440 conn(0x7f589c01cac0 legacy=0x7f589c01abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:03.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.458+0000 7f58bfe2e700 20 client.37242.objecter put_session s=0x7f589c01c3a0 osd=5 1 2024-04-14T15:18:03.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.458+0000 7f58bfe2e700 10 client.37242.objecter close_session for osd.6 2024-04-14T15:18:03.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.458+0000 7f58be3cb700 1 -- 172.21.15.90:0/2065677670 reap_dead start 2024-04-14T15:18:03.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.458+0000 7f58bfe2e700 1 -- 172.21.15.90:0/2065677670 >> v1:172.21.15.92:6808/636125963 conn(0x7f589c01fab0 legacy=0x7f589c021ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:03.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.458+0000 7f58be3cb700 1 -- 172.21.15.90:0/2065677670 reap_dead start 2024-04-14T15:18:03.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.458+0000 7f58bfe2e700 20 client.37242.objecter put_session s=0x7f589c01d170 osd=6 1 2024-04-14T15:18:03.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.458+0000 7f58bfe2e700 10 client.37242.objecter close_session for osd.7 2024-04-14T15:18:03.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.458+0000 7f58bfe2e700 1 -- 172.21.15.90:0/2065677670 >> v1:172.21.15.92:6812/1773706716 conn(0x7f589c023c40 legacy=0x7f589c026030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:03.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.458+0000 7f58bfe2e700 20 client.37242.objecter put_session s=0x7f589c0235c0 osd=7 1 2024-04-14T15:18:03.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.458+0000 7f58bfe2e700 20 client.37242.objecter shutdown clearing up homeless session... 2024-04-14T15:18:03.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.458+0000 7f58bfe2e700 10 client.37242.objecter successfully canceled tick 2024-04-14T15:18:03.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.458+0000 7f58bfe2e700 1 -- 172.21.15.90:0/2065677670 >> v1:172.21.15.90:6800/420935427 conn(0x7f58a4071420 legacy=0x7f58a40738e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:03.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.458+0000 7f58be3cb700 1 -- 172.21.15.90:0/2065677670 reap_dead start 2024-04-14T15:18:03.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.458+0000 7f58be3cb700 1 -- 172.21.15.90:0/2065677670 reap_dead start 2024-04-14T15:18:03.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.458+0000 7f58bfe2e700 1 -- 172.21.15.90:0/2065677670 >> v1:172.21.15.90:6789/0 conn(0x7f58b810b720 legacy=0x7f58b8105c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:03.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.458+0000 7f58bfe2e700 1 -- 172.21.15.90:0/2065677670 shutdown_connections 2024-04-14T15:18:03.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.458+0000 7f58bfe2e700 1 -- 172.21.15.90:0/2065677670 >> 172.21.15.90:0/2065677670 conn(0x7f58b80fe3a0 msgr2=0x7f58b8112040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:18:03.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.459+0000 7f58bfe2e700 1 -- 172.21.15.90:0/2065677670 shutdown_connections 2024-04-14T15:18:03.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.459+0000 7f58bfe2e700 1 -- 172.21.15.90:0/2065677670 wait complete. 2024-04-14T15:18:03.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:03.459+0000 7f58bfe2e700 1 librados: shutdown 2024-04-14T15:18:03.464 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:18:03.464 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:18:03.464 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:18:03.464 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:18:03.464 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:18:03.464 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:18:03.465 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:18:03.465 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:18:03.465 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:18:03.489 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:03.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:03.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:18:03.591 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:18:03.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:03.615 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:03.615 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:18:03.716 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:18:03.744 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:03.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:03.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:18:03.846 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:03.873 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:03.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:03.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:03.962 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:18:03.975 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:18:04.000 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:04.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:04.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:18:04.083 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:18:03 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:18:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:18:04.083 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:03 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:18:04.083 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:03 smithi090 ceph-mon[30399]: osdmap e409: 8 total, 8 up, 8 in 2024-04-14T15:18:04.083 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:03 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:18:04.083 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:03 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:18:04.084 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:03 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-57"}]: dispatch 2024-04-14T15:18:04.084 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:03 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-57"}]: dispatch 2024-04-14T15:18:04.084 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:03 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2065677670' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:18:04.084 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:03 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:18:04.084 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:03 smithi090 ceph-mon[41670]: osdmap e409: 8 total, 8 up, 8 in 2024-04-14T15:18:04.084 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:03 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:18:04.084 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:03 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:18:04.084 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:03 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-57"}]: dispatch 2024-04-14T15:18:04.084 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:03 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-57"}]: dispatch 2024-04-14T15:18:04.084 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:03 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2065677670' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:18:04.102 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:04.130 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:04.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:04.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:04.231 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:18:04.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:04.256 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:04.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:18:04.357 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:04.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:03 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:18:04.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:03 smithi092 ceph-mon[33966]: osdmap e409: 8 total, 8 up, 8 in 2024-04-14T15:18:04.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:03 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:18:04.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:03 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:18:04.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:03 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-57"}]: dispatch 2024-04-14T15:18:04.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:03 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-57"}]: dispatch 2024-04-14T15:18:04.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:03 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2065677670' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:18:04.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:04.387 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:04.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:04.488 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:18:04.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:04.517 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:04.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:18:04.618 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:18:04.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:04.643 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:04.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:18:04.744 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:18:04.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:04.769 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:04.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:18:04.870 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:18:04.898 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:04.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:04.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:18:05.001 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:18:05.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:05.026 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:05.026 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:18:05.128 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:18:05.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:05.157 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:05.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:18:05.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:04 smithi090 ceph-mon[30399]: pgmap v563: 238 pgs: 32 unknown, 206 active+clean; 8.4 MiB data, 615 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:18:05.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:04 smithi090 ceph-mon[30399]: osdmap e410: 8 total, 8 up, 8 in 2024-04-14T15:18:05.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:04 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:18:05.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:04 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:18:05.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:04 smithi090 ceph-mon[41670]: pgmap v563: 238 pgs: 32 unknown, 206 active+clean; 8.4 MiB data, 615 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:18:05.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:04 smithi090 ceph-mon[41670]: osdmap e410: 8 total, 8 up, 8 in 2024-04-14T15:18:05.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:04 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:18:05.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:04 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:18:05.258 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:18:05.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:05.283 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:05.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:18:05.372 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:18:05 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:18:05.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:04 smithi092 ceph-mon[33966]: pgmap v563: 238 pgs: 32 unknown, 206 active+clean; 8.4 MiB data, 615 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:18:05.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:04 smithi092 ceph-mon[33966]: osdmap e410: 8 total, 8 up, 8 in 2024-04-14T15:18:05.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:04 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:18:05.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:04 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:18:05.384 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:18:05.408 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:05.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:05.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:18:05.510 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:18:05.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:05.534 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:05.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:18:05.635 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:18:05.663 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:05.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:05.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:18:05.765 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:18:05.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:05.790 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:05.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:18:05.891 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:05.927 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:05.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:05.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:06.030 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:18:06.054 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:06.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:06.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:18:06.152 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:05 smithi090 ceph-mon[41670]: osdmap e411: 8 total, 8 up, 8 in 2024-04-14T15:18:06.152 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:05 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:18:06.152 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:05 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:18:06.152 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:05 smithi090 ceph-mon[41670]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:18:06.153 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:05 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:18:06.153 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:05 smithi090 ceph-mon[41670]: osdmap e412: 8 total, 8 up, 8 in 2024-04-14T15:18:06.153 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:05 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-14T15:18:06.153 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:05 smithi090 ceph-mon[30399]: osdmap e411: 8 total, 8 up, 8 in 2024-04-14T15:18:06.153 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:05 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:18:06.153 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:05 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:18:06.153 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:05 smithi090 ceph-mon[30399]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:18:06.153 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:05 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:18:06.153 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:05 smithi090 ceph-mon[30399]: osdmap e412: 8 total, 8 up, 8 in 2024-04-14T15:18:06.153 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:05 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-14T15:18:06.157 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:18:06.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:06.181 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:06.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:18:06.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.258+0000 7f16fce7b700 1 -- 172.21.15.90:0/2979699348 >> v1:172.21.15.90:6790/0 conn(0x7f16f8108880 legacy=0x7f16f8108c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:06.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.259+0000 7f16fce7b700 1 -- 172.21.15.90:0/2979699348 shutdown_connections 2024-04-14T15:18:06.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.259+0000 7f16fce7b700 1 -- 172.21.15.90:0/2979699348 >> 172.21.15.90:0/2979699348 conn(0x7f16f80fe3d0 msgr2=0x7f16f81007f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:18:06.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.259+0000 7f16fce7b700 1 -- 172.21.15.90:0/2979699348 shutdown_connections 2024-04-14T15:18:06.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.259+0000 7f16fce7b700 1 -- 172.21.15.90:0/2979699348 wait complete. 2024-04-14T15:18:06.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.259+0000 7f16fce7b700 1 librados: starting msgr at 2024-04-14T15:18:06.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.259+0000 7f16fce7b700 1 librados: starting objecter 2024-04-14T15:18:06.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.259+0000 7f16fce7b700 1 Processor -- start 2024-04-14T15:18:06.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.259+0000 7f16fce7b700 1 -- start start 2024-04-14T15:18:06.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.259+0000 7f16fce7b700 1 librados: setting wanted keys 2024-04-14T15:18:06.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.259+0000 7f16fce7b700 1 librados: calling monclient init 2024-04-14T15:18:06.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.260+0000 7f16fce7b700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f16f81a7840 con 0x7f16f8108880 2024-04-14T15:18:06.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.260+0000 7f16fce7b700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f16f81a8a80 con 0x7f16f810f300 2024-04-14T15:18:06.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.260+0000 7f16fce7b700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f16f81a9cc0 con 0x7f16f810b730 2024-04-14T15:18:06.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.260+0000 7f16f659c700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f16f8108880 0x7f16f810e910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50638/0 (socket says 172.21.15.90:50638) 2024-04-14T15:18:06.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.260+0000 7f16f659c700 1 -- 172.21.15.90:0/1241700589 learned_addr learned my addr 172.21.15.90:0/1241700589 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:18:06.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.260+0000 7f16f5d9b700 1 --1- 172.21.15.90:0/1241700589 >> v1:172.21.15.90:6790/0 conn(0x7f16f810b730 0x7f16f81a27b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:45816/0 (socket says 172.21.15.90:45816) 2024-04-14T15:18:06.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.260+0000 7f16e77fe700 10 client.?.objecter ms_handle_connect 0x7f16f810b730 2024-04-14T15:18:06.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.260+0000 7f16e77fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:18:06.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.260+0000 7f16e77fe700 10 client.?.objecter ms_handle_connect 0x7f16f8108880 2024-04-14T15:18:06.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.260+0000 7f16e77fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:18:06.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.260+0000 7f16e77fe700 10 client.?.objecter ms_handle_connect 0x7f16f810f300 2024-04-14T15:18:06.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.260+0000 7f16e77fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:18:06.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.260+0000 7f16e77fe700 1 -- 172.21.15.90:0/1241700589 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4167886815 0 0) 0x7f16f81a7840 con 0x7f16f8108880 2024-04-14T15:18:06.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.260+0000 7f16e77fe700 1 -- 172.21.15.90:0/1241700589 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f16e0003aa0 con 0x7f16f8108880 2024-04-14T15:18:06.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.260+0000 7f16e77fe700 1 -- 172.21.15.90:0/1241700589 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2598663474 0 0) 0x7f16f81a9cc0 con 0x7f16f810b730 2024-04-14T15:18:06.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.260+0000 7f16e77fe700 1 -- 172.21.15.90:0/1241700589 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f16f81a7840 con 0x7f16f810b730 2024-04-14T15:18:06.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.260+0000 7f16e77fe700 1 -- 172.21.15.90:0/1241700589 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2753112619 0 0) 0x7f16f81a8a80 con 0x7f16f810f300 2024-04-14T15:18:06.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.260+0000 7f16e77fe700 1 -- 172.21.15.90:0/1241700589 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f16f81a9cc0 con 0x7f16f810f300 2024-04-14T15:18:06.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.260+0000 7f16e77fe700 1 -- 172.21.15.90:0/1241700589 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 179420511 0 0) 0x7f16e0003aa0 con 0x7f16f8108880 2024-04-14T15:18:06.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.260+0000 7f16e77fe700 1 -- 172.21.15.90:0/1241700589 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f16f81a8a80 con 0x7f16f8108880 2024-04-14T15:18:06.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.260+0000 7f16e77fe700 1 -- 172.21.15.90:0/1241700589 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f16ec004590 con 0x7f16f8108880 2024-04-14T15:18:06.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.260+0000 7f16e77fe700 1 -- 172.21.15.90:0/1241700589 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3799142317 0 0) 0x7f16f81a7840 con 0x7f16f810b730 2024-04-14T15:18:06.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.260+0000 7f16e77fe700 1 -- 172.21.15.90:0/1241700589 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f16e0003aa0 con 0x7f16f810b730 2024-04-14T15:18:06.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.261+0000 7f16e77fe700 1 -- 172.21.15.90:0/1241700589 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1836821847 0 0) 0x7f16f81a9cc0 con 0x7f16f810f300 2024-04-14T15:18:06.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.261+0000 7f16e77fe700 1 -- 172.21.15.90:0/1241700589 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f16f81a7840 con 0x7f16f810f300 2024-04-14T15:18:06.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.261+0000 7f16e77fe700 1 -- 172.21.15.90:0/1241700589 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f16e8001850 con 0x7f16f810b730 2024-04-14T15:18:06.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.261+0000 7f16e77fe700 1 -- 172.21.15.90:0/1241700589 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f16f00013a0 con 0x7f16f810f300 2024-04-14T15:18:06.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.261+0000 7f16e77fe700 1 -- 172.21.15.90:0/1241700589 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2460845188 0 0) 0x7f16f81a8a80 con 0x7f16f8108880 2024-04-14T15:18:06.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.261+0000 7f16e77fe700 1 -- 172.21.15.90:0/1241700589 >> v1:172.21.15.90:6790/0 conn(0x7f16f810b730 legacy=0x7f16f81a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:06.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.261+0000 7f16e77fe700 1 -- 172.21.15.90:0/1241700589 >> v1:172.21.15.92:6789/0 conn(0x7f16f810f300 legacy=0x7f16f81a5f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:06.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.261+0000 7f16e77fe700 1 -- 172.21.15.90:0/1241700589 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16f81aaf00 con 0x7f16f8108880 2024-04-14T15:18:06.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.261+0000 7f16fce7b700 1 -- 172.21.15.90:0/1241700589 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f16f81a7a70 con 0x7f16f8108880 2024-04-14T15:18:06.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.261+0000 7f16fce7b700 10 client.37248.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:18:06.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.261+0000 7f16fce7b700 1 -- 172.21.15.90:0/1241700589 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f16f81a7f30 con 0x7f16f8108880 2024-04-14T15:18:06.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.261+0000 7f16fce7b700 1 librados: init done 2024-04-14T15:18:06.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.261+0000 7f16e77fe700 1 -- 172.21.15.90:0/1241700589 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f16ec001b10 con 0x7f16f8108880 2024-04-14T15:18:06.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.261+0000 7f16e77fe700 1 -- 172.21.15.90:0/1241700589 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f16ec004e70 con 0x7f16f8108880 2024-04-14T15:18:06.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.262+0000 7f16fce7b700 1 -- 172.21.15.90:0/1241700589 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16d8005150 con 0x7f16f8108880 2024-04-14T15:18:06.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.263+0000 7f16e77fe700 1 -- 172.21.15.90:0/1241700589 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f16ec004e50 con 0x7f16f8108880 2024-04-14T15:18:06.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.263+0000 7f16e77fe700 1 -- 172.21.15.90:0/1241700589 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(412..412 src has 51..412) ==== 8144+0+0 (unknown 674472027 0 0) 0x7f16ec08a920 con 0x7f16f8108880 2024-04-14T15:18:06.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.263+0000 7f16e77fe700 10 client.37248.objecter ms_dispatch 0x7f16f8001550 osd_map(412..412 src has 51..412) 2024-04-14T15:18:06.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.263+0000 7f16e77fe700 3 client.37248.objecter handle_osd_map got epochs [412,412] > 0 2024-04-14T15:18:06.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.263+0000 7f16e77fe700 3 client.37248.objecter handle_osd_map decoding full epoch 412 2024-04-14T15:18:06.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.263+0000 7f16e77fe700 20 client.37248.objecter dump_active .. 0 homeless 2024-04-14T15:18:06.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.263+0000 7f16e77fe700 10 client.37248.objecter ms_handle_connect 0x7f16e0071540 2024-04-14T15:18:06.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.268+0000 7f16e77fe700 1 -- 172.21.15.90:0/1241700589 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f16ec057390 con 0x7f16f8108880 2024-04-14T15:18:06.282 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:06.307 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:06.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:06.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:05 smithi092 ceph-mon[33966]: osdmap e411: 8 total, 8 up, 8 in 2024-04-14T15:18:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:05 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:18:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:05 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:18:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:05 smithi092 ceph-mon[33966]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:18:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:05 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:18:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:05 smithi092 ceph-mon[33966]: osdmap e412: 8 total, 8 up, 8 in 2024-04-14T15:18:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:05 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-14T15:18:06.409 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:18:06.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:06.434 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:06.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:18:06.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.480+0000 7f16fce7b700 1 -- 172.21.15.90:0/1241700589 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd reweight", "id": 7, "weight": 1.0} v 0) -- 0x7f16d8005440 con 0x7f16f8108880 2024-04-14T15:18:06.537 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:18:06.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:06.563 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:06.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:18:06.664 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:18:06.688 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:06.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:06.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:18:06.726 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-14T15:18:06.789 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:18:06.815 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:06.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:06.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:18:06.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.889+0000 7f16e77fe700 1 -- 172.21.15.90:0/1241700589 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 7, "weight": 1.0}]=0 reweighted osd.7 to 1 (10000) v413) ==== 113+0+0 (unknown 4020078448 0 0) 0x7f16ec05b040 con 0x7f16f8108880 2024-04-14T15:18:06.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.892+0000 7f16fce7b700 10 librados: watch_flush enter 2024-04-14T15:18:06.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.892+0000 7f16fce7b700 10 librados: watch_flush exit 2024-04-14T15:18:06.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.892+0000 7f16fce7b700 20 client.37248.objecter shutdown clearing up homeless session... 2024-04-14T15:18:06.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.892+0000 7f16fce7b700 10 client.37248.objecter successfully canceled tick 2024-04-14T15:18:06.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.892+0000 7f16fce7b700 1 -- 172.21.15.90:0/1241700589 >> v1:172.21.15.90:6800/420935427 conn(0x7f16e0071540 legacy=0x7f16e0073a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:06.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.892+0000 7f16fce7b700 1 -- 172.21.15.90:0/1241700589 >> v1:172.21.15.90:6789/0 conn(0x7f16f8108880 legacy=0x7f16f810e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:06.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.893+0000 7f16fce7b700 1 -- 172.21.15.90:0/1241700589 shutdown_connections 2024-04-14T15:18:06.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.893+0000 7f16fce7b700 1 -- 172.21.15.90:0/1241700589 >> 172.21.15.90:0/1241700589 conn(0x7f16f80fe3d0 msgr2=0x7f16f8111f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:18:06.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.893+0000 7f16fce7b700 1 -- 172.21.15.90:0/1241700589 shutdown_connections 2024-04-14T15:18:06.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.893+0000 7f16fce7b700 1 -- 172.21.15.90:0/1241700589 wait complete. 2024-04-14T15:18:06.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:06.894+0000 7f16fce7b700 1 librados: shutdown 2024-04-14T15:18:06.903 INFO:teuthology.orchestra.run.smithi090.stderr:reweighted osd.7 to 1 (10000) 2024-04-14T15:18:06.917 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:18:06.941 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:06.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:06.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:18:06.994 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:18:07.044 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:18:07.067 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:07.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:07.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:18:07.169 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:18:07.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:06 smithi090 ceph-mon[41670]: pgmap v566: 238 pgs: 32 unknown, 206 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-04-14T15:18:07.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:06 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:18:07.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:06 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:18:07.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:06 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:18:07.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:06 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-59"}]: dispatch 2024-04-14T15:18:07.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:06 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-59"}]: dispatch 2024-04-14T15:18:07.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:06 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1241700589' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-14T15:18:07.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:06 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1241700589' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-14T15:18:07.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:06 smithi090 ceph-mon[41670]: osdmap e413: 8 total, 8 up, 8 in 2024-04-14T15:18:07.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:06 smithi090 ceph-mon[30399]: pgmap v566: 238 pgs: 32 unknown, 206 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-04-14T15:18:07.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:06 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:18:07.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:06 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:18:07.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:06 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:18:07.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:06 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-59"}]: dispatch 2024-04-14T15:18:07.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:06 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-59"}]: dispatch 2024-04-14T15:18:07.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:06 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1241700589' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-14T15:18:07.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:06 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1241700589' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-14T15:18:07.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:06 smithi090 ceph-mon[30399]: osdmap e413: 8 total, 8 up, 8 in 2024-04-14T15:18:07.194 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:07.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:07.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:18:07.296 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:07.320 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:07.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:07.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:07.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:06 smithi092 ceph-mon[33966]: pgmap v566: 238 pgs: 32 unknown, 206 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-04-14T15:18:07.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:06 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:18:07.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:06 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:18:07.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:06 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:18:07.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:06 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-59"}]: dispatch 2024-04-14T15:18:07.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:06 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-59"}]: dispatch 2024-04-14T15:18:07.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:06 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1241700589' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-14T15:18:07.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:06 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1241700589' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-14T15:18:07.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:06 smithi092 ceph-mon[33966]: osdmap e413: 8 total, 8 up, 8 in 2024-04-14T15:18:07.422 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:18:07.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:07.448 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:07.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:18:07.549 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:07.574 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:07.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:07.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:07.676 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:07.700 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:07.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:07.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:07.802 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:07.827 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:07.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:07.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:07.929 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:18:07.953 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:07.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:07.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:18:08.054 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:18:08.080 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:08.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:08.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:18:08.181 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:18:08.208 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:08.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:08.209 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:18:08.310 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:18:08.335 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:08.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:08.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:18:08.437 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:18:08.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:08.462 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:08.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:18:08.563 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:18:08.587 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:08.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:08.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:18:08.689 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:08.710 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-14T15:18:08.710 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-14T15:18:08.710 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-14T15:18:08.710 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-14T15:18:08.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.710 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25009.0:2 32.0 32:0384ab8e:::oid:head [write 18446744073709551615~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-04-14T15:18:08.710 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:12:55.060999+0000", 2024-04-14T15:18:08.710 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 313.33280384099999, 2024-04-14T15:18:08.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.494352567, 2024-04-14T15:18:08.711 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:18:08.711 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:18:08.711 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-14T15:18:08.711 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:18:08.711 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25009", 2024-04-14T15:18:08.711 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2850614897", 2024-04-14T15:18:08.711 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2 2024-04-14T15:18:08.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.711 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:18:08.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.711 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:18:08.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.060999+0000", 2024-04-14T15:18:08.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:18:08.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:18:08.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.060999+0000", 2024-04-14T15:18:08.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:18:08.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:18:08.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.061001+0000", 2024-04-14T15:18:08.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0030000000000001e-06 2024-04-14T15:18:08.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:18:08.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.061006+0000", 2024-04-14T15:18:08.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5450000000000003e-06 2024-04-14T15:18:08.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:18:08.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.061017+0000", 2024-04-14T15:18:08.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1090999999999999e-05 2024-04-14T15:18:08.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:18:08.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.061028+0000", 2024-04-14T15:18:08.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0675e-05 2024-04-14T15:18:08.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:18:08.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.066663+0000", 2024-04-14T15:18:08.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0056352490000000002 2024-04-14T15:18:08.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:18:08.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.066673+0000", 2024-04-14T15:18:08.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3400000000000004e-06 2024-04-14T15:18:08.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:18:08.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.470857+0000", 2024-04-14T15:18:08.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.40418453 2024-04-14T15:18:08.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:18:08.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.471681+0000", 2024-04-14T15:18:08.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00082436499999999997 2024-04-14T15:18:08.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:18:08.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.555351+0000", 2024-04-14T15:18:08.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.083669769000000005 2024-04-14T15:18:08.715 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:18:08.715 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:18:08.715 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:18:08.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.715 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.24965.0:2 30.1a e108/107 30:5c6776db:::0.obj:head v 108'1, mlcod=0'0)", 2024-04-14T15:18:08.716 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:12:56.467908+0000", 2024-04-14T15:18:08.716 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 311.92589470500002, 2024-04-14T15:18:08.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.031681980999999998, 2024-04-14T15:18:08.716 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:18:08.716 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:18:08.716 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:18:08.716 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:18:08.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:18:08.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.467908+0000", 2024-04-14T15:18:08.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:18:08.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:18:08.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.467908+0000", 2024-04-14T15:18:08.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:18:08.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:18:08.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.467908+0000", 2024-04-14T15:18:08.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9700000000000003e-07 2024-04-14T15:18:08.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:18:08.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.467914+0000", 2024-04-14T15:18:08.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4700000000000001e-06 2024-04-14T15:18:08.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:18:08.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.467920+0000", 2024-04-14T15:18:08.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5509999999999996e-06 2024-04-14T15:18:08.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:18:08.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.467928+0000", 2024-04-14T15:18:08.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9729999999999997e-06 2024-04-14T15:18:08.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:18:08.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.469047+0000", 2024-04-14T15:18:08.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011187 2024-04-14T15:18:08.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:18:08.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.469065+0000", 2024-04-14T15:18:08.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8732e-05 2024-04-14T15:18:08.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:18:08.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.499557+0000", 2024-04-14T15:18:08.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030491082999999999 2024-04-14T15:18:08.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.720 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:18:08.720 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.499590+0000", 2024-04-14T15:18:08.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3175e-05 2024-04-14T15:18:08.720 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:18:08.720 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:18:08.720 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:18:08.720 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.720 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.25144.0:16 24.10 e108/107 24:09666ec2:::28.obj2:head v 108'13, mlcod=108'11)", 2024-04-14T15:18:08.720 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:12:57.294151+0000", 2024-04-14T15:18:08.720 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 311.09965178599998, 2024-04-14T15:18:08.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.042811227, 2024-04-14T15:18:08.720 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:18:08.720 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:18:08.721 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:18:08.721 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:18:08.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.721 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:18:08.721 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.294151+0000", 2024-04-14T15:18:08.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:18:08.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.721 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:18:08.721 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.294151+0000", 2024-04-14T15:18:08.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:18:08.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.721 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:18:08.722 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.294151+0000", 2024-04-14T15:18:08.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8200000000000001e-07 2024-04-14T15:18:08.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.722 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:18:08.722 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.294170+0000", 2024-04-14T15:18:08.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8800999999999998e-05 2024-04-14T15:18:08.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.722 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:18:08.722 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.294178+0000", 2024-04-14T15:18:08.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9710000000000004e-06 2024-04-14T15:18:08.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:18:08.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.294192+0000", 2024-04-14T15:18:08.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4623e-05 2024-04-14T15:18:08.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:18:08.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.294994+0000", 2024-04-14T15:18:08.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00080199199999999996 2024-04-14T15:18:08.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:18:08.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.295017+0000", 2024-04-14T15:18:08.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2606999999999999e-05 2024-04-14T15:18:08.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.724 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:18:08.724 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.336915+0000", 2024-04-14T15:18:08.724 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.041897917 2024-04-14T15:18:08.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.724 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.724 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:18:08.724 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.336962+0000", 2024-04-14T15:18:08.724 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7033999999999997e-05 2024-04-14T15:18:08.724 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:18:08.724 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:18:08.724 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:18:08.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.724 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.725 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.26497.0:35 112.1 e133/122 112:8133d882:test-rados-api-smithi090-93724-10::foo:head v 133'7, mlcod=132'4)", 2024-04-14T15:18:08.725 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:13:20.911793+0000", 2024-04-14T15:18:08.725 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 287.48200957400002, 2024-04-14T15:18:08.725 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.031446676, 2024-04-14T15:18:08.725 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:18:08.725 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:18:08.725 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:18:08.725 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:18:08.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.725 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:18:08.725 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.911793+0000", 2024-04-14T15:18:08.725 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:18:08.725 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.726 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:18:08.726 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.911793+0000", 2024-04-14T15:18:08.726 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:18:08.726 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.726 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.726 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:18:08.726 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.911793+0000", 2024-04-14T15:18:08.726 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1e-07 2024-04-14T15:18:08.726 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.726 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.726 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:18:08.726 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.911802+0000", 2024-04-14T15:18:08.727 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0100000000000001e-06 2024-04-14T15:18:08.727 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.727 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.727 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:18:08.727 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.911809+0000", 2024-04-14T15:18:08.727 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3319999999999997e-06 2024-04-14T15:18:08.727 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.727 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.727 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:18:08.728 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.911818+0000", 2024-04-14T15:18:08.728 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.166e-06 2024-04-14T15:18:08.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.728 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.728 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:18:08.728 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.912551+0000", 2024-04-14T15:18:08.728 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00073312300000000002 2024-04-14T15:18:08.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.728 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.728 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:18:08.728 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.912572+0000", 2024-04-14T15:18:08.728 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1274e-05 2024-04-14T15:18:08.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.728 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.729 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:18:08.729 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.943196+0000", 2024-04-14T15:18:08.729 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030623862000000002 2024-04-14T15:18:08.729 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.729 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.729 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:18:08.729 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.943240+0000", 2024-04-14T15:18:08.729 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3599e-05 2024-04-14T15:18:08.729 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:18:08.729 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:18:08.729 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:18:08.729 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.729 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.729 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.26497.0:36 112.1 e133/122 112:8133d882:test-rados-api-smithi090-93724-10::foo:head v 133'8, mlcod=132'5)", 2024-04-14T15:18:08.729 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:13:20.947031+0000", 2024-04-14T15:18:08.730 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 287.44677157299998, 2024-04-14T15:18:08.730 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.026026404999999999, 2024-04-14T15:18:08.730 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:18:08.730 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:18:08.730 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:18:08.730 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:18:08.730 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.730 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:18:08.730 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.947031+0000", 2024-04-14T15:18:08.730 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:18:08.730 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.730 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.731 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:18:08.731 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.947031+0000", 2024-04-14T15:18:08.731 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:18:08.731 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.731 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.731 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:18:08.731 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.947033+0000", 2024-04-14T15:18:08.731 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9139999999999998e-06 2024-04-14T15:18:08.731 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.731 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.731 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:18:08.731 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.947044+0000", 2024-04-14T15:18:08.731 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1134999999999999e-05 2024-04-14T15:18:08.731 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.731 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.732 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:18:08.732 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.947063+0000", 2024-04-14T15:18:08.732 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8853999999999999e-05 2024-04-14T15:18:08.732 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.732 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.732 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:18:08.732 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.947074+0000", 2024-04-14T15:18:08.732 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0815999999999999e-05 2024-04-14T15:18:08.732 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.732 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.732 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:18:08.732 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.947153+0000", 2024-04-14T15:18:08.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9678999999999994e-05 2024-04-14T15:18:08.733 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.733 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.733 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:18:08.733 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.947177+0000", 2024-04-14T15:18:08.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3332999999999998e-05 2024-04-14T15:18:08.733 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.733 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.733 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:18:08.733 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.972985+0000", 2024-04-14T15:18:08.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.025808122999999999 2024-04-14T15:18:08.733 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.733 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.733 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:18:08.733 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.973057+0000", 2024-04-14T15:18:08.734 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2551000000000004e-05 2024-04-14T15:18:08.734 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:18:08.734 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:18:08.734 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:18:08.734 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.734 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.734 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.26497.0:51 112.e 112:75b427dd:test-rados-api-smithi090-93724-11::foo:head [write 0~1048576] snapc 9=[9] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-04-14T15:18:08.734 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:13:25.825582+0000", 2024-04-14T15:18:08.734 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 282.56822021300002, 2024-04-14T15:18:08.734 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.141706371, 2024-04-14T15:18:08.734 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:18:08.734 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:18:08.734 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:18:08.735 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:18:08.735 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.26497", 2024-04-14T15:18:08.735 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3962832909", 2024-04-14T15:18:08.735 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 51 2024-04-14T15:18:08.735 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.735 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:18:08.735 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.735 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:18:08.735 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:25.825582+0000", 2024-04-14T15:18:08.735 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:18:08.735 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.735 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.735 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:18:08.735 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:25.825582+0000", 2024-04-14T15:18:08.736 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:18:08.736 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.736 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.736 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:18:08.736 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:25.825584+0000", 2024-04-14T15:18:08.736 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9069999999999999e-06 2024-04-14T15:18:08.736 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.736 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.736 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:18:08.736 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:25.826345+0000", 2024-04-14T15:18:08.736 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076040899999999998 2024-04-14T15:18:08.736 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.736 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.736 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:18:08.736 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:25.826358+0000", 2024-04-14T15:18:08.737 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2989999999999999e-05 2024-04-14T15:18:08.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.737 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.737 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:18:08.737 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:25.826369+0000", 2024-04-14T15:18:08.737 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.128e-05 2024-04-14T15:18:08.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.737 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.737 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:18:08.737 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:25.826434+0000", 2024-04-14T15:18:08.737 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4833999999999996e-05 2024-04-14T15:18:08.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.737 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.737 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2024-04-14T15:18:08.737 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:25.826441+0000", 2024-04-14T15:18:08.738 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9890000000000003e-06 2024-04-14T15:18:08.738 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.738 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.738 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:18:08.738 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:25.865309+0000", 2024-04-14T15:18:08.738 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.038867983000000002 2024-04-14T15:18:08.738 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.738 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.738 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:18:08.738 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:25.865442+0000", 2024-04-14T15:18:08.738 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133158 2024-04-14T15:18:08.738 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.739 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.739 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2024-04-14T15:18:08.739 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:25.865580+0000", 2024-04-14T15:18:08.739 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000138615 2024-04-14T15:18:08.739 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.739 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.739 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:18:08.739 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:25.872696+0000", 2024-04-14T15:18:08.739 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0071156220000000003 2024-04-14T15:18:08.739 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.739 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.739 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:18:08.739 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:25.921899+0000", 2024-04-14T15:18:08.739 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.049203021999999999 2024-04-14T15:18:08.740 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.740 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.740 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:18:08.740 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:25.967182+0000", 2024-04-14T15:18:08.740 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.045283367999999997 2024-04-14T15:18:08.740 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.740 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.740 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:18:08.740 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:25.967240+0000", 2024-04-14T15:18:08.740 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7874999999999997e-05 2024-04-14T15:18:08.740 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.740 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.741 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:18:08.741 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:25.967289+0000", 2024-04-14T15:18:08.741 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8319000000000003e-05 2024-04-14T15:18:08.741 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:18:08.741 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:18:08.741 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:18:08.741 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.741 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.741 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.26497.0:53 112.e 112:75b427dd:test-rados-api-smithi090-93724-11::foo:head [stat,delete] snapc a=[a,9] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-04-14T15:18:08.741 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:13:26.893558+0000", 2024-04-14T15:18:08.741 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 281.50024473299999, 2024-04-14T15:18:08.741 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.059330368000000001, 2024-04-14T15:18:08.741 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:18:08.742 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:18:08.742 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:18:08.742 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:18:08.742 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.26497", 2024-04-14T15:18:08.742 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3962832909", 2024-04-14T15:18:08.742 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 53 2024-04-14T15:18:08.742 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.742 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:18:08.742 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.742 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:18:08.743 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.893558+0000", 2024-04-14T15:18:08.743 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:18:08.743 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.743 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.743 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:18:08.743 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.893558+0000", 2024-04-14T15:18:08.743 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:18:08.743 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.743 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.743 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:18:08.744 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.893560+0000", 2024-04-14T15:18:08.744 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.863e-06 2024-04-14T15:18:08.744 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.744 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.744 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:18:08.744 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.893565+0000", 2024-04-14T15:18:08.744 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.344e-06 2024-04-14T15:18:08.744 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.744 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.744 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:18:08.744 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.893576+0000", 2024-04-14T15:18:08.744 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0743e-05 2024-04-14T15:18:08.744 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.745 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.745 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:18:08.745 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.893585+0000", 2024-04-14T15:18:08.745 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7990000000000005e-06 2024-04-14T15:18:08.745 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.745 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.745 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:18:08.745 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.894455+0000", 2024-04-14T15:18:08.745 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00086904300000000001 2024-04-14T15:18:08.746 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.746 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.746 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:18:08.746 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.897306+0000", 2024-04-14T15:18:08.746 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028518160000000001 2024-04-14T15:18:08.746 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.746 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.746 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2024-04-14T15:18:08.746 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.897553+0000", 2024-04-14T15:18:08.746 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024705700000000002 2024-04-14T15:18:08.746 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.746 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.746 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:18:08.746 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.913549+0000", 2024-04-14T15:18:08.747 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015995939000000001 2024-04-14T15:18:08.747 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.747 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.747 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:18:08.747 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.946641+0000", 2024-04-14T15:18:08.747 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.033091655999999997 2024-04-14T15:18:08.747 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.747 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.747 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:18:08.747 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.952773+0000", 2024-04-14T15:18:08.747 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0061320630000000001 2024-04-14T15:18:08.747 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.747 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.748 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:18:08.748 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.952835+0000", 2024-04-14T15:18:08.748 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2082999999999995e-05 2024-04-14T15:18:08.748 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.748 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.748 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:18:08.748 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.952888+0000", 2024-04-14T15:18:08.748 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2961999999999999e-05 2024-04-14T15:18:08.748 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:18:08.748 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:18:08.748 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:18:08.749 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.749 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.749 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.25684.0:248 110.1c e139/121 110:3d310c86:test-rados-api-smithi090-93652-12::bar:head v 139'1, mlcod=0'0)", 2024-04-14T15:18:08.749 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:13:26.946390+0000", 2024-04-14T15:18:08.749 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 281.44741222699997, 2024-04-14T15:18:08.749 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.026315974999999998, 2024-04-14T15:18:08.749 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:18:08.749 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:18:08.749 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:18:08.749 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:18:08.749 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.749 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:18:08.749 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.946390+0000", 2024-04-14T15:18:08.749 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:18:08.750 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.750 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.750 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:18:08.750 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.946390+0000", 2024-04-14T15:18:08.750 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:18:08.750 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.750 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.750 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:18:08.750 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.946391+0000", 2024-04-14T15:18:08.750 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9000000000000002e-07 2024-04-14T15:18:08.750 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.750 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.750 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:18:08.750 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.946397+0000", 2024-04-14T15:18:08.751 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7460000000000003e-06 2024-04-14T15:18:08.752 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.752 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.752 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:18:08.752 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.946403+0000", 2024-04-14T15:18:08.752 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4480000000000002e-06 2024-04-14T15:18:08.752 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.752 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.752 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:18:08.752 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.946411+0000", 2024-04-14T15:18:08.752 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2649999999999996e-06 2024-04-14T15:18:08.752 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.752 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.752 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:18:08.753 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.959679+0000", 2024-04-14T15:18:08.753 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013267582 2024-04-14T15:18:08.753 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.753 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.753 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:18:08.753 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.959698+0000", 2024-04-14T15:18:08.753 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9508e-05 2024-04-14T15:18:08.753 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.753 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.753 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:18:08.753 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.972667+0000", 2024-04-14T15:18:08.753 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012968841 2024-04-14T15:18:08.753 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.753 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.753 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:18:08.754 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.972706+0000", 2024-04-14T15:18:08.754 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9195e-05 2024-04-14T15:18:08.754 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:18:08.754 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:18:08.754 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:18:08.754 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.754 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.754 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(149.6s1 141/136 ECSubWrite(tid=801, reqid=client.38086.0:7, at_version=141'1, trim_to=0'0, roll_forward_to=0'0))", 2024-04-14T15:18:08.754 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:13:28.866071+0000", 2024-04-14T15:18:08.754 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 279.52773101100001, 2024-04-14T15:18:08.754 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.028275307, 2024-04-14T15:18:08.754 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:18:08.754 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:18:08.754 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-04-14T15:18:08.755 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:18:08.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.755 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:18:08.755 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.866071+0000", 2024-04-14T15:18:08.755 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:18:08.755 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.755 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:18:08.755 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.866071+0000", 2024-04-14T15:18:08.755 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:18:08.755 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.755 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:18:08.755 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.866072+0000", 2024-04-14T15:18:08.756 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3500000000000002e-07 2024-04-14T15:18:08.756 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.756 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.756 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:18:08.756 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.866102+0000", 2024-04-14T15:18:08.756 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9887e-05 2024-04-14T15:18:08.756 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.756 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.756 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:18:08.756 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.866110+0000", 2024-04-14T15:18:08.756 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0909999999999993e-06 2024-04-14T15:18:08.756 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.756 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.756 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:18:08.756 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.866120+0000", 2024-04-14T15:18:08.757 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0360000000000001e-05 2024-04-14T15:18:08.757 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.757 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.757 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:18:08.757 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.884475+0000", 2024-04-14T15:18:08.757 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.018355172999999999 2024-04-14T15:18:08.757 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.757 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.757 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2024-04-14T15:18:08.757 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.884486+0000", 2024-04-14T15:18:08.757 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0404999999999999e-05 2024-04-14T15:18:08.757 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.757 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.757 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2024-04-14T15:18:08.758 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.894318+0000", 2024-04-14T15:18:08.758 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0098320620000000008 2024-04-14T15:18:08.758 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.758 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.758 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:18:08.758 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.894347+0000", 2024-04-14T15:18:08.758 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8994000000000001e-05 2024-04-14T15:18:08.758 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:18:08.758 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:18:08.758 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:18:08.758 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.758 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.758 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(149.6s1 141/136 ECSubWrite(tid=802, reqid=client.38086.0:7, at_version=0'0, trim_to=0'0, roll_forward_to=141'1))", 2024-04-14T15:18:08.758 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:13:28.900478+0000", 2024-04-14T15:18:08.759 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 279.49332462500001, 2024-04-14T15:18:08.759 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.027266736999999999, 2024-04-14T15:18:08.759 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:18:08.759 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:18:08.759 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-04-14T15:18:08.759 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:18:08.759 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.759 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:18:08.759 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.900478+0000", 2024-04-14T15:18:08.759 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:18:08.759 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.759 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.759 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:18:08.759 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.900478+0000", 2024-04-14T15:18:08.759 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:18:08.760 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.760 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.760 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:18:08.760 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.900478+0000", 2024-04-14T15:18:08.760 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3300000000000002e-07 2024-04-14T15:18:08.760 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.760 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.760 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:18:08.760 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.900488+0000", 2024-04-14T15:18:08.760 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0078000000000001e-05 2024-04-14T15:18:08.760 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.760 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.760 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:18:08.760 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.900495+0000", 2024-04-14T15:18:08.761 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0369999999999997e-06 2024-04-14T15:18:08.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.761 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.761 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:18:08.761 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.900506+0000", 2024-04-14T15:18:08.761 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0472000000000001e-05 2024-04-14T15:18:08.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.761 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.761 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:18:08.761 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.901653+0000", 2024-04-14T15:18:08.761 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011474359999999999 2024-04-14T15:18:08.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.761 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.761 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2024-04-14T15:18:08.761 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.901664+0000", 2024-04-14T15:18:08.762 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0267e-05 2024-04-14T15:18:08.762 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.762 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.762 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2024-04-14T15:18:08.762 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.927711+0000", 2024-04-14T15:18:08.762 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.026047229000000002 2024-04-14T15:18:08.762 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.762 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.762 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:18:08.762 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.927745+0000", 2024-04-14T15:18:08.762 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3685e-05 2024-04-14T15:18:08.762 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:18:08.762 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:18:08.763 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:18:08.763 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.763 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.763 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.44341.0:2 161.3 e141/140 161:c656db43:::oid_RoundTripPP_:head v 141'1, mlcod=0'0)", 2024-04-14T15:18:08.763 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:13:28.923697+0000", 2024-04-14T15:18:08.763 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 279.47010522400001, 2024-04-14T15:18:08.763 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.023297067000000001, 2024-04-14T15:18:08.763 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:18:08.763 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:18:08.763 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:18:08.763 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:18:08.763 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.763 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:18:08.763 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.923697+0000", 2024-04-14T15:18:08.763 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:18:08.764 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.764 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.764 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:18:08.764 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.923697+0000", 2024-04-14T15:18:08.764 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:18:08.764 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.764 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.764 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:18:08.764 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.923698+0000", 2024-04-14T15:18:08.764 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2099999999999998e-07 2024-04-14T15:18:08.764 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.764 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.764 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:18:08.764 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.923703+0000", 2024-04-14T15:18:08.765 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4700000000000001e-06 2024-04-14T15:18:08.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.765 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.765 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:18:08.765 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.923709+0000", 2024-04-14T15:18:08.765 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5060000000000003e-06 2024-04-14T15:18:08.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.765 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.765 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:18:08.765 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.923718+0000", 2024-04-14T15:18:08.765 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2240000000000001e-06 2024-04-14T15:18:08.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.765 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.765 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:18:08.765 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.924458+0000", 2024-04-14T15:18:08.766 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00073990299999999998 2024-04-14T15:18:08.766 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.766 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.766 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:18:08.766 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.924477+0000", 2024-04-14T15:18:08.766 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9011000000000001e-05 2024-04-14T15:18:08.766 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.766 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.766 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:18:08.766 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.942083+0000", 2024-04-14T15:18:08.766 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017605873000000001 2024-04-14T15:18:08.766 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.766 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.766 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:18:08.766 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.946994+0000", 2024-04-14T15:18:08.767 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0049117589999999999 2024-04-14T15:18:08.767 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:18:08.767 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:18:08.767 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:18:08.767 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.767 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.767 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(149.6s1 142/136 ECSubWrite(tid=806, reqid=client.38086.0:10, at_version=0'0, trim_to=0'0, roll_forward_to=142'4))", 2024-04-14T15:18:08.767 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:13:29.900007+0000", 2024-04-14T15:18:08.767 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 278.493795664, 2024-04-14T15:18:08.767 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.024611383000000001, 2024-04-14T15:18:08.767 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:18:08.767 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:18:08.767 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-04-14T15:18:08.767 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:18:08.767 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.768 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:18:08.768 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.900007+0000", 2024-04-14T15:18:08.768 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:18:08.768 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.768 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.768 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:18:08.768 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.900007+0000", 2024-04-14T15:18:08.768 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:18:08.768 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.768 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.768 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:18:08.768 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.900009+0000", 2024-04-14T15:18:08.768 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.576e-06 2024-04-14T15:18:08.768 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.769 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.769 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:18:08.769 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.900031+0000", 2024-04-14T15:18:08.769 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1178e-05 2024-04-14T15:18:08.769 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.769 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.769 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:18:08.769 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.900042+0000", 2024-04-14T15:18:08.769 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1715999999999999e-05 2024-04-14T15:18:08.769 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.769 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.769 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:18:08.769 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.900054+0000", 2024-04-14T15:18:08.769 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1280999999999999e-05 2024-04-14T15:18:08.769 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.770 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.770 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:18:08.770 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.900127+0000", 2024-04-14T15:18:08.770 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3409000000000001e-05 2024-04-14T15:18:08.770 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.770 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.770 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2024-04-14T15:18:08.770 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.912284+0000", 2024-04-14T15:18:08.770 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012156945000000001 2024-04-14T15:18:08.770 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.770 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.770 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2024-04-14T15:18:08.770 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.924582+0000", 2024-04-14T15:18:08.771 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012298514999999999 2024-04-14T15:18:08.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.771 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:18:08.771 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.924618+0000", 2024-04-14T15:18:08.771 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5763000000000003e-05 2024-04-14T15:18:08.771 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:18:08.771 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:18:08.771 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:18:08.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.771 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.26497.0:57 112.1a e142/122 112:5978e4a5:test-rados-api-smithi090-93724-12::foo:head v 142'1, mlcod=0'0)", 2024-04-14T15:18:08.771 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:13:29.900967+0000", 2024-04-14T15:18:08.771 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 278.49283575200002, 2024-04-14T15:18:08.772 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.105006778, 2024-04-14T15:18:08.772 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:18:08.772 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:18:08.772 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:18:08.772 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:18:08.772 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.772 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:18:08.772 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.900967+0000", 2024-04-14T15:18:08.772 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:18:08.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.772 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.772 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:18:08.772 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.900967+0000", 2024-04-14T15:18:08.772 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:18:08.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.773 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:18:08.773 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.900967+0000", 2024-04-14T15:18:08.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0400000000000002e-07 2024-04-14T15:18:08.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.773 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:18:08.773 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.900976+0000", 2024-04-14T15:18:08.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9080000000000001e-06 2024-04-14T15:18:08.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.773 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:18:08.773 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.934576+0000", 2024-04-14T15:18:08.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.033600179000000001 2024-04-14T15:18:08.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.774 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:18:08.774 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.934588+0000", 2024-04-14T15:18:08.774 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1939e-05 2024-04-14T15:18:08.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.774 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:18:08.774 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.937063+0000", 2024-04-14T15:18:08.774 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0024749889999999999 2024-04-14T15:18:08.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.774 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:18:08.775 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.937084+0000", 2024-04-14T15:18:08.775 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1220999999999999e-05 2024-04-14T15:18:08.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.775 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.775 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:18:08.775 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:30.005783+0000", 2024-04-14T15:18:08.775 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.068699239999999995 2024-04-14T15:18:08.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.775 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.775 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:18:08.775 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:30.005973+0000", 2024-04-14T15:18:08.775 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018999800000000001 2024-04-14T15:18:08.775 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:18:08.775 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:18:08.775 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:18:08.776 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.776 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.776 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.48565.0:1 239.5s0 239:ab97ba35:::nonexistent:head [read 0~128] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e236)", 2024-04-14T15:18:08.776 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:15:03.863599+0000", 2024-04-14T15:18:08.776 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 184.53020377600001, 2024-04-14T15:18:08.776 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0325199839999999, 2024-04-14T15:18:08.776 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:18:08.776 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:18:08.776 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-04-14T15:18:08.776 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:18:08.776 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.48565", 2024-04-14T15:18:08.776 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3767512710", 2024-04-14T15:18:08.776 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1 2024-04-14T15:18:08.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.777 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:18:08.777 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.777 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:18:08.777 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:03.863599+0000", 2024-04-14T15:18:08.777 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:18:08.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.777 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.777 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:18:08.777 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:03.863599+0000", 2024-04-14T15:18:08.777 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:18:08.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.777 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.777 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:18:08.777 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:03.863601+0000", 2024-04-14T15:18:08.778 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.995e-06 2024-04-14T15:18:08.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.778 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.778 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:18:08.778 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:03.863607+0000", 2024-04-14T15:18:08.778 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2700000000000001e-06 2024-04-14T15:18:08.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.778 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.778 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:18:08.778 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:03.863618+0000", 2024-04-14T15:18:08.778 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1368e-05 2024-04-14T15:18:08.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.778 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.778 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:18:08.779 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:03.863630+0000", 2024-04-14T15:18:08.779 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1324e-05 2024-04-14T15:18:08.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.779 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:18:08.779 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:03.863700+0000", 2024-04-14T15:18:08.779 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0399000000000001e-05 2024-04-14T15:18:08.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.779 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:18:08.779 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:03.863703+0000", 2024-04-14T15:18:08.779 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2349999999999999e-06 2024-04-14T15:18:08.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.779 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:18:08.780 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:04.896017+0000", 2024-04-14T15:18:08.780 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0323134599999999 2024-04-14T15:18:08.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.780 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.780 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:18:08.780 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:04.896119+0000", 2024-04-14T15:18:08.780 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000101933 2024-04-14T15:18:08.780 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:18:08.780 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:18:08.780 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:18:08.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.780 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.780 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25684.0:2198 247.1f 247:f8000000:%01:::head [pgnls start_epoch 253] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e253)", 2024-04-14T15:18:08.780 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:15:20.904026+0000", 2024-04-14T15:18:08.780 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 167.489776503, 2024-04-14T15:18:08.781 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.98221185899999996, 2024-04-14T15:18:08.781 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:18:08.781 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:18:08.781 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-14T15:18:08.781 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:18:08.781 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25684", 2024-04-14T15:18:08.781 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1109245683", 2024-04-14T15:18:08.781 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2198 2024-04-14T15:18:08.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.781 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:18:08.781 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.781 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:18:08.781 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:20.904026+0000", 2024-04-14T15:18:08.782 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:18:08.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.782 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.782 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:18:08.782 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:20.904026+0000", 2024-04-14T15:18:08.782 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:18:08.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.782 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.782 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:18:08.782 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:20.904029+0000", 2024-04-14T15:18:08.782 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0900000000000001e-06 2024-04-14T15:18:08.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.782 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.782 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:18:08.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:20.904037+0000", 2024-04-14T15:18:08.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3319999999999992e-06 2024-04-14T15:18:08.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:18:08.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:20.904053+0000", 2024-04-14T15:18:08.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5311e-05 2024-04-14T15:18:08.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:18:08.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:20.904061+0000", 2024-04-14T15:18:08.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1659999999999994e-06 2024-04-14T15:18:08.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.784 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:18:08.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:20.904133+0000", 2024-04-14T15:18:08.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2409000000000004e-05 2024-04-14T15:18:08.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.784 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:18:08.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:20.904146+0000", 2024-04-14T15:18:08.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2639e-05 2024-04-14T15:18:08.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.784 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:18:08.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:21.885405+0000", 2024-04-14T15:18:08.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.981259192 2024-04-14T15:18:08.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:18:08.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:21.885442+0000", 2024-04-14T15:18:08.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7014000000000002e-05 2024-04-14T15:18:08.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:18:08.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:21.886238+0000", 2024-04-14T15:18:08.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00079570599999999997 2024-04-14T15:18:08.785 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:18:08.785 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:18:08.785 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:18:08.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.786 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.36840.0:1 256.6s0 256:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e270)", 2024-04-14T15:18:08.786 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:15:37.877392+0000", 2024-04-14T15:18:08.786 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 150.516410358, 2024-04-14T15:18:08.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0013250600000001, 2024-04-14T15:18:08.786 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:18:08.786 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:18:08.786 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:18:08.786 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:18:08.786 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.36840", 2024-04-14T15:18:08.786 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2015849731", 2024-04-14T15:18:08.786 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1 2024-04-14T15:18:08.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.786 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:18:08.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.787 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:18:08.787 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:37.877392+0000", 2024-04-14T15:18:08.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:18:08.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.787 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:18:08.787 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:37.877392+0000", 2024-04-14T15:18:08.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:18:08.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.787 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:18:08.787 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:37.877394+0000", 2024-04-14T15:18:08.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.702e-06 2024-04-14T15:18:08.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:18:08.788 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:37.877400+0000", 2024-04-14T15:18:08.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2689999999999996e-06 2024-04-14T15:18:08.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:18:08.788 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:37.877410+0000", 2024-04-14T15:18:08.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0365e-05 2024-04-14T15:18:08.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:18:08.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:37.877421+0000", 2024-04-14T15:18:08.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.082e-05 2024-04-14T15:18:08.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.789 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:18:08.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:37.877468+0000", 2024-04-14T15:18:08.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6479e-05 2024-04-14T15:18:08.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.789 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:18:08.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:37.877472+0000", 2024-04-14T15:18:08.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.907e-06 2024-04-14T15:18:08.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.789 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:18:08.790 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:38.875012+0000", 2024-04-14T15:18:08.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99754053399999998 2024-04-14T15:18:08.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:18:08.790 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:38.875130+0000", 2024-04-14T15:18:08.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011831499999999999 2024-04-14T15:18:08.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2024-04-14T15:18:08.790 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:38.876401+0000", 2024-04-14T15:18:08.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001270544 2024-04-14T15:18:08.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.791 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2024-04-14T15:18:08.791 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:38.876968+0000", 2024-04-14T15:18:08.791 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00056727100000000005 2024-04-14T15:18:08.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.791 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:18:08.791 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:38.878670+0000", 2024-04-14T15:18:08.791 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0017013379999999999 2024-04-14T15:18:08.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.791 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:18:08.791 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:38.878717+0000", 2024-04-14T15:18:08.791 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7515999999999997e-05 2024-04-14T15:18:08.791 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:18:08.791 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:18:08.792 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:18:08.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.792 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.792 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.48691.0:1 257.6s0 257:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-04-14T15:18:08.792 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:15:38.872646+0000", 2024-04-14T15:18:08.792 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 149.52115624699999, 2024-04-14T15:18:08.792 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.015757738, 2024-04-14T15:18:08.792 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:18:08.792 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:18:08.792 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:18:08.792 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:18:08.792 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.48691", 2024-04-14T15:18:08.792 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3211270109", 2024-04-14T15:18:08.792 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1 2024-04-14T15:18:08.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.793 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:18:08.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:18:08.793 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:38.872646+0000", 2024-04-14T15:18:08.793 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:18:08.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:18:08.793 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:38.872646+0000", 2024-04-14T15:18:08.793 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:18:08.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:18:08.793 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:38.872648+0000", 2024-04-14T15:18:08.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8899999999999999e-06 2024-04-14T15:18:08.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.794 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:18:08.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:38.872655+0000", 2024-04-14T15:18:08.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6599999999999998e-06 2024-04-14T15:18:08.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.794 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:18:08.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:38.872666+0000", 2024-04-14T15:18:08.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1158000000000001e-05 2024-04-14T15:18:08.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.794 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:18:08.795 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:38.872676+0000", 2024-04-14T15:18:08.795 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0295000000000001e-05 2024-04-14T15:18:08.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.795 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:18:08.795 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:38.873327+0000", 2024-04-14T15:18:08.795 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00065083600000000001 2024-04-14T15:18:08.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.795 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:18:08.795 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:38.873331+0000", 2024-04-14T15:18:08.795 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6030000000000001e-06 2024-04-14T15:18:08.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.796 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:18:08.796 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:39.881515+0000", 2024-04-14T15:18:08.796 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.00818429 2024-04-14T15:18:08.796 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.796 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.796 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:18:08.796 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:39.881635+0000", 2024-04-14T15:18:08.796 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000119824 2024-04-14T15:18:08.796 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.796 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.796 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2024-04-14T15:18:08.796 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:39.882074+0000", 2024-04-14T15:18:08.796 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00043970599999999999 2024-04-14T15:18:08.796 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.797 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.797 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2024-04-14T15:18:08.797 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:39.882644+0000", 2024-04-14T15:18:08.797 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000569251 2024-04-14T15:18:08.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.797 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.797 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:18:08.797 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:39.888332+0000", 2024-04-14T15:18:08.797 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0056887109999999999 2024-04-14T15:18:08.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.797 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.797 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:18:08.797 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:39.888404+0000", 2024-04-14T15:18:08.797 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1513999999999993e-05 2024-04-14T15:18:08.798 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:18:08.798 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:18:08.798 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:18:08.798 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.798 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.798 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.48715.0:1 261.6s0 261:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e278)", 2024-04-14T15:18:08.798 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:15:45.860394+0000", 2024-04-14T15:18:08.798 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 142.53340885599999, 2024-04-14T15:18:08.798 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0127147990000001, 2024-04-14T15:18:08.798 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:18:08.798 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:18:08.798 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:18:08.798 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:18:08.798 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.48715", 2024-04-14T15:18:08.799 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1788001083", 2024-04-14T15:18:08.799 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1 2024-04-14T15:18:08.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.799 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:18:08.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.799 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:18:08.799 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:45.860394+0000", 2024-04-14T15:18:08.799 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:18:08.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.799 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:18:08.799 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:45.860394+0000", 2024-04-14T15:18:08.799 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:18:08.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.800 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:18:08.800 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:45.860396+0000", 2024-04-14T15:18:08.800 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2979999999999999e-06 2024-04-14T15:18:08.800 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.800 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.800 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:18:08.800 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:45.860404+0000", 2024-04-14T15:18:08.800 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1429999999999998e-06 2024-04-14T15:18:08.800 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.800 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.800 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:18:08.800 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:45.860419+0000", 2024-04-14T15:18:08.800 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5013e-05 2024-04-14T15:18:08.800 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.801 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:18:08.801 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:45.860431+0000", 2024-04-14T15:18:08.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2327000000000001e-05 2024-04-14T15:18:08.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.801 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:18:08.801 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:45.899182+0000", 2024-04-14T15:18:08.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.038750740999999998 2024-04-14T15:18:08.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.801 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:18:08.801 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:45.899187+0000", 2024-04-14T15:18:08.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.515e-06 2024-04-14T15:18:08.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.802 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:18:08.802 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:46.869238+0000", 2024-04-14T15:18:08.802 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.970051628 2024-04-14T15:18:08.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.802 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:18:08.802 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:46.869425+0000", 2024-04-14T15:18:08.802 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018649899999999999 2024-04-14T15:18:08.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.802 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2024-04-14T15:18:08.803 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:46.870375+0000", 2024-04-14T15:18:08.803 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00095067800000000005 2024-04-14T15:18:08.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.803 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2024-04-14T15:18:08.803 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:46.871231+0000", 2024-04-14T15:18:08.803 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00085567499999999997 2024-04-14T15:18:08.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.803 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:18:08.803 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:46.873042+0000", 2024-04-14T15:18:08.803 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001810571 2024-04-14T15:18:08.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.804 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:18:08.804 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:46.873108+0000", 2024-04-14T15:18:08.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6711000000000006e-05 2024-04-14T15:18:08.804 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:18:08.804 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:18:08.804 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:18:08.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.804 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25684.0:3739 272.8 272:10000000::::head [pgnls start_epoch 314] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e314)", 2024-04-14T15:18:08.804 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:16:21.890450+0000", 2024-04-14T15:18:08.804 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 106.50335271100001, 2024-04-14T15:18:08.805 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.044063261999999999, 2024-04-14T15:18:08.805 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:18:08.805 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:18:08.805 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-14T15:18:08.805 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:18:08.805 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25684", 2024-04-14T15:18:08.805 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1109245683", 2024-04-14T15:18:08.805 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3739 2024-04-14T15:18:08.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.805 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:18:08.805 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.805 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:18:08.805 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:16:21.890450+0000", 2024-04-14T15:18:08.805 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:18:08.806 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.806 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.806 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:18:08.806 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:16:21.890450+0000", 2024-04-14T15:18:08.806 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:18:08.806 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.806 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.806 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:18:08.806 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:16:21.890454+0000", 2024-04-14T15:18:08.806 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1239999999999998e-06 2024-04-14T15:18:08.806 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.806 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.806 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:18:08.806 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:16:21.890468+0000", 2024-04-14T15:18:08.807 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4154999999999999e-05 2024-04-14T15:18:08.807 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.807 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.807 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:18:08.807 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:16:21.890486+0000", 2024-04-14T15:18:08.807 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7601e-05 2024-04-14T15:18:08.807 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.807 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.807 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:18:08.807 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:16:21.890496+0000", 2024-04-14T15:18:08.807 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0723999999999999e-05 2024-04-14T15:18:08.807 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.807 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.807 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:18:08.808 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:16:21.890565+0000", 2024-04-14T15:18:08.808 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8442000000000006e-05 2024-04-14T15:18:08.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.808 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.808 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2024-04-14T15:18:08.808 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:16:21.890570+0000", 2024-04-14T15:18:08.808 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.972e-06 2024-04-14T15:18:08.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.808 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.808 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:18:08.808 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:16:21.934313+0000", 2024-04-14T15:18:08.808 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.043743734999999999 2024-04-14T15:18:08.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.808 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.808 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:18:08.809 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:16:21.934353+0000", 2024-04-14T15:18:08.809 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9403999999999997e-05 2024-04-14T15:18:08.809 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.809 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.809 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:18:08.809 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:16:21.934513+0000", 2024-04-14T15:18:08.809 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016010500000000001 2024-04-14T15:18:08.809 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:18:08.809 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:18:08.809 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:18:08.809 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.809 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.809 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.37104.0:2 282.13 e353/352 282:cbb9903e:::test_obj:head v 353'1, mlcod=0'0)", 2024-04-14T15:18:08.809 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:17:00.914225+0000", 2024-04-14T15:18:08.810 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 67.479577734000003, 2024-04-14T15:18:08.810 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.025144527999999999, 2024-04-14T15:18:08.810 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:18:08.810 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:18:08.810 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:18:08.810 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:18:08.810 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.810 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:18:08.810 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:00.914225+0000", 2024-04-14T15:18:08.810 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:18:08.810 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.810 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.810 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:18:08.810 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:00.914225+0000", 2024-04-14T15:18:08.811 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:18:08.811 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.811 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.811 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:18:08.811 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:00.914225+0000", 2024-04-14T15:18:08.811 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9200000000000002e-07 2024-04-14T15:18:08.811 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.811 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.811 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:18:08.811 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:00.914235+0000", 2024-04-14T15:18:08.811 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0054999999999999e-05 2024-04-14T15:18:08.811 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.811 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.811 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:18:08.811 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:00.914241+0000", 2024-04-14T15:18:08.812 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5509999999999999e-06 2024-04-14T15:18:08.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.812 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:18:08.812 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:00.914249+0000", 2024-04-14T15:18:08.812 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4780000000000008e-06 2024-04-14T15:18:08.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.812 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:18:08.812 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:00.914393+0000", 2024-04-14T15:18:08.812 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014401999999999999 2024-04-14T15:18:08.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.812 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:18:08.812 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:00.914410+0000", 2024-04-14T15:18:08.813 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6631999999999999e-05 2024-04-14T15:18:08.813 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.813 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.813 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:18:08.813 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:00.938022+0000", 2024-04-14T15:18:08.813 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.023612016999999999 2024-04-14T15:18:08.813 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:18:08.813 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:18:08.813 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:18:08.813 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:00.939369+0000", 2024-04-14T15:18:08.813 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001347483 2024-04-14T15:18:08.813 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:18:08.813 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:18:08.813 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:18:08.813 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:18:08.814 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:18:08.814 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-14T15:18:08.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:08.815 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:08.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:08.916 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:18:08.943 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:08.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:08.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:18:09.045 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:18:09.070 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:09.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:09.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:18:09.172 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:18:09.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:08 smithi090 ceph-mon[41670]: pgmap v569: 206 pgs: 206 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-04-14T15:18:09.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:08 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:18:09.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:08 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:18:09.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:08 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:18:09.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:08 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:18:09.195 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:08 smithi090 ceph-mon[41670]: osdmap e414: 8 total, 8 up, 8 in 2024-04-14T15:18:09.195 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:08 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:18:09.195 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:08 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:18:09.195 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:08 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:18:09.195 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:08 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:18:09.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:08 smithi090 ceph-mon[30399]: pgmap v569: 206 pgs: 206 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-04-14T15:18:09.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:08 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:18:09.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:08 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:18:09.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:08 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:18:09.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:08 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:18:09.195 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:08 smithi090 ceph-mon[30399]: osdmap e414: 8 total, 8 up, 8 in 2024-04-14T15:18:09.196 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:08 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:18:09.196 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:08 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:18:09.196 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:08 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:18:09.196 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:08 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:18:09.197 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:09.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:09.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:18:09.299 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:18:09.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:09.326 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:09.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:18:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:08 smithi092 ceph-mon[33966]: pgmap v569: 206 pgs: 206 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-04-14T15:18:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:08 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:18:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:08 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:18:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:08 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:18:09.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:08 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:18:09.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:08 smithi092 ceph-mon[33966]: osdmap e414: 8 total, 8 up, 8 in 2024-04-14T15:18:09.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:08 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:18:09.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:08 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:18:09.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:08 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:18:09.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:08 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:18:09.428 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:18:09.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:09.457 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:09.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:18:09.558 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:18:09.586 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:09.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:09.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:18:09.688 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:18:09.713 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:09.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:09.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:18:09.815 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:18:09.839 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:09.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:09.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:18:09.941 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:18:09.966 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:09.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:09.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:18:10.068 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:10.096 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:10.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:10.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:10.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:09 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:18:10.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:09 smithi090 ceph-mon[41670]: osdmap e415: 8 total, 8 up, 8 in 2024-04-14T15:18:10.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:09 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-14T15:18:10.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:09 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-14T15:18:10.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:09 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-14T15:18:10.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:09 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:18:10.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:09 smithi090 ceph-mon[30399]: osdmap e415: 8 total, 8 up, 8 in 2024-04-14T15:18:10.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:09 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-14T15:18:10.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:09 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-14T15:18:10.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:09 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-14T15:18:10.198 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:18:10.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:10.223 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:10.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:18:10.325 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:10.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:10.353 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:10.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:10.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:09 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:18:10.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:09 smithi092 ceph-mon[33966]: osdmap e415: 8 total, 8 up, 8 in 2024-04-14T15:18:10.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:09 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-14T15:18:10.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:09 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-14T15:18:10.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:09 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-14T15:18:10.454 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:18:10.481 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:10.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:10.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:18:10.583 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:18:10.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:10.608 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:10.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:18:10.709 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:18:10.734 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:10.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:10.735 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:18:10.835 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:18:10.860 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:10.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:10.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:18:10.868 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-14T15:18:10.962 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:18:10.985 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:10.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:10.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:18:11.087 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:11.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:11.112 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:11.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:11.128 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:18:11.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:10 smithi090 ceph-mon[41670]: pgmap v572: 238 pgs: 6 unknown, 232 active+clean; 8.4 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-04-14T15:18:11.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:10 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-14T15:18:11.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:10 smithi090 ceph-mon[41670]: osdmap e416: 8 total, 8 up, 8 in 2024-04-14T15:18:11.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:10 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:18:11.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:10 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:18:11.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:10 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-61"}]: dispatch 2024-04-14T15:18:11.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:10 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-61"}]: dispatch 2024-04-14T15:18:11.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:10 smithi090 ceph-mon[30399]: pgmap v572: 238 pgs: 6 unknown, 232 active+clean; 8.4 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-04-14T15:18:11.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:10 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-14T15:18:11.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:10 smithi090 ceph-mon[30399]: osdmap e416: 8 total, 8 up, 8 in 2024-04-14T15:18:11.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:10 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:18:11.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:10 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:18:11.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:10 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-61"}]: dispatch 2024-04-14T15:18:11.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:10 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-61"}]: dispatch 2024-04-14T15:18:11.213 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:18:11.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:11.237 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:11.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:18:11.250 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:10 smithi092 ceph-mon[33966]: pgmap v572: 238 pgs: 6 unknown, 232 active+clean; 8.4 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-04-14T15:18:11.251 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:10 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-14T15:18:11.251 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:10 smithi092 ceph-mon[33966]: osdmap e416: 8 total, 8 up, 8 in 2024-04-14T15:18:11.251 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:10 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:18:11.251 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:10 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:18:11.251 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:10 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-61"}]: dispatch 2024-04-14T15:18:11.251 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:10 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-61"}]: dispatch 2024-04-14T15:18:11.338 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:18:11.363 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:11.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:11.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:18:11.465 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:18:11.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:11.490 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:11.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:18:11.591 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:11.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:11.617 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:11.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:11.718 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:18:11.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:11.745 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:11.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:18:11.846 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:11.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:11.872 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:11.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:11.927 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:18:11.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:11.949+0000 7fe0a3b8d700 1 -- 172.21.15.90:0/1594486177 >> v1:172.21.15.90:6790/0 conn(0x7fe09c115950 legacy=0x7fe09c117df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:11.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:11.950+0000 7fe0a3b8d700 1 -- 172.21.15.90:0/1594486177 shutdown_connections 2024-04-14T15:18:11.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:11.950+0000 7fe0a3b8d700 1 -- 172.21.15.90:0/1594486177 >> 172.21.15.90:0/1594486177 conn(0x7fe09c077b30 msgr2=0x7fe09c077f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:18:11.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:11.950+0000 7fe0a3b8d700 1 -- 172.21.15.90:0/1594486177 shutdown_connections 2024-04-14T15:18:11.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:11.950+0000 7fe0a3b8d700 1 -- 172.21.15.90:0/1594486177 wait complete. 2024-04-14T15:18:11.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:11.951+0000 7fe0a3b8d700 1 librados: starting msgr at 2024-04-14T15:18:11.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:11.951+0000 7fe0a3b8d700 1 librados: starting objecter 2024-04-14T15:18:11.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:11.951+0000 7fe0a3b8d700 1 Processor -- start 2024-04-14T15:18:11.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:11.951+0000 7fe0a3b8d700 1 -- start start 2024-04-14T15:18:11.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:11.951+0000 7fe0a3b8d700 1 librados: setting wanted keys 2024-04-14T15:18:11.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:11.951+0000 7fe0a3b8d700 1 librados: calling monclient init 2024-04-14T15:18:11.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:11.951+0000 7fe0a3b8d700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe09c115250 con 0x7fe09c115950 2024-04-14T15:18:11.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:11.951+0000 7fe0a3b8d700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe09c115420 con 0x7fe09c101500 2024-04-14T15:18:11.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:11.951+0000 7fe0a3b8d700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe09c115630 con 0x7fe09c100a10 2024-04-14T15:18:11.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:11.951+0000 7fe0a1128700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fe09c101500 0x7fe09c114430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.90:50486/0 (socket says 172.21.15.90:50486) 2024-04-14T15:18:11.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:11.951+0000 7fe0a1128700 1 -- 172.21.15.90:0/2568290024 learned_addr learned my addr 172.21.15.90:0/2568290024 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:18:11.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:11.951+0000 7fe0a212a700 1 --1- 172.21.15.90:0/2568290024 >> v1:172.21.15.90:6789/0 conn(0x7fe09c115950 0x7fe09c114b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:36228/0 (socket says 172.21.15.90:36228) 2024-04-14T15:18:11.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:11.952+0000 7fe092ffd700 10 client.?.objecter ms_handle_connect 0x7fe09c115950 2024-04-14T15:18:11.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:11.952+0000 7fe092ffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:18:11.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:11.952+0000 7fe092ffd700 10 client.?.objecter ms_handle_connect 0x7fe09c101500 2024-04-14T15:18:11.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:11.952+0000 7fe092ffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:18:11.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:11.952+0000 7fe092ffd700 10 client.?.objecter ms_handle_connect 0x7fe09c100a10 2024-04-14T15:18:11.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:11.952+0000 7fe092ffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:18:11.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:11.952+0000 7fe092ffd700 1 -- 172.21.15.90:0/2568290024 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2566086054 0 0) 0x7fe09c115420 con 0x7fe09c101500 2024-04-14T15:18:11.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:11.952+0000 7fe092ffd700 1 -- 172.21.15.90:0/2568290024 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe0880035b0 con 0x7fe09c101500 2024-04-14T15:18:11.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:11.952+0000 7fe092ffd700 1 -- 172.21.15.90:0/2568290024 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4017625830 0 0) 0x7fe09c115630 con 0x7fe09c100a10 2024-04-14T15:18:11.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:11.952+0000 7fe092ffd700 1 -- 172.21.15.90:0/2568290024 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe09c115420 con 0x7fe09c100a10 2024-04-14T15:18:11.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:11.952+0000 7fe092ffd700 1 -- 172.21.15.90:0/2568290024 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3973118397 0 0) 0x7fe09c115250 con 0x7fe09c115950 2024-04-14T15:18:11.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:11.952+0000 7fe092ffd700 1 -- 172.21.15.90:0/2568290024 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe09c115630 con 0x7fe09c115950 2024-04-14T15:18:11.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:11.952+0000 7fe092ffd700 1 -- 172.21.15.90:0/2568290024 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1074715836 0 0) 0x7fe09c115630 con 0x7fe09c115950 2024-04-14T15:18:11.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:11.952+0000 7fe092ffd700 1 -- 172.21.15.90:0/2568290024 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe09c115250 con 0x7fe09c115950 2024-04-14T15:18:11.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:11.952+0000 7fe092ffd700 1 -- 172.21.15.90:0/2568290024 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fe094001890 con 0x7fe09c115950 2024-04-14T15:18:11.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:11.953+0000 7fe092ffd700 1 -- 172.21.15.90:0/2568290024 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2290024998 0 0) 0x7fe09c115420 con 0x7fe09c100a10 2024-04-14T15:18:11.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:11.953+0000 7fe092ffd700 1 -- 172.21.15.90:0/2568290024 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe09c115630 con 0x7fe09c100a10 2024-04-14T15:18:11.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:11.953+0000 7fe092ffd700 1 -- 172.21.15.90:0/2568290024 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1928365963 0 0) 0x7fe0880035b0 con 0x7fe09c101500 2024-04-14T15:18:11.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:11.953+0000 7fe092ffd700 1 -- 172.21.15.90:0/2568290024 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe09c115420 con 0x7fe09c101500 2024-04-14T15:18:11.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:11.953+0000 7fe092ffd700 1 -- 172.21.15.90:0/2568290024 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fe098001680 con 0x7fe09c100a10 2024-04-14T15:18:11.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:11.953+0000 7fe092ffd700 1 -- 172.21.15.90:0/2568290024 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fe08c0018e0 con 0x7fe09c101500 2024-04-14T15:18:11.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:11.953+0000 7fe092ffd700 1 -- 172.21.15.90:0/2568290024 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3831077447 0 0) 0x7fe09c115420 con 0x7fe09c101500 2024-04-14T15:18:11.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:11.953+0000 7fe092ffd700 1 -- 172.21.15.90:0/2568290024 >> v1:172.21.15.90:6790/0 conn(0x7fe09c100a10 legacy=0x7fe09c10e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:11.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:11.953+0000 7fe092ffd700 1 -- 172.21.15.90:0/2568290024 >> v1:172.21.15.90:6789/0 conn(0x7fe09c115950 legacy=0x7fe09c114b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:11.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:11.953+0000 7fe092ffd700 1 -- 172.21.15.90:0/2568290024 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe09c1a8d90 con 0x7fe09c101500 2024-04-14T15:18:11.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:11.953+0000 7fe0a3b8d700 1 -- 172.21.15.90:0/2568290024 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe09c1a5dc0 con 0x7fe09c101500 2024-04-14T15:18:11.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:11.953+0000 7fe0a3b8d700 10 client.49090.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:18:11.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:11.953+0000 7fe0a3b8d700 1 -- 172.21.15.90:0/2568290024 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe09c1a6280 con 0x7fe09c101500 2024-04-14T15:18:11.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:11.953+0000 7fe0a3b8d700 1 librados: init done 2024-04-14T15:18:11.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:11.953+0000 7fe092ffd700 1 -- 172.21.15.90:0/2568290024 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fe08c003b20 con 0x7fe09c101500 2024-04-14T15:18:11.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:11.953+0000 7fe092ffd700 1 -- 172.21.15.90:0/2568290024 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fe08c005bf0 con 0x7fe09c101500 2024-04-14T15:18:11.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:11.954+0000 7fe0a3b8d700 1 -- 172.21.15.90:0/2568290024 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe09c101db0 con 0x7fe09c101500 2024-04-14T15:18:11.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:11.955+0000 7fe092ffd700 1 -- 172.21.15.90:0/2568290024 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fe08c006e80 con 0x7fe09c101500 2024-04-14T15:18:11.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:11.956+0000 7fe092ffd700 1 -- 172.21.15.90:0/2568290024 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(418..418 src has 51..418) ==== 8147+0+0 (unknown 3927190704 0 0) 0x7fe08c053200 con 0x7fe09c101500 2024-04-14T15:18:11.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:11.956+0000 7fe092ffd700 10 client.49090.objecter ms_dispatch 0x7fe09c001550 osd_map(418..418 src has 51..418) 2024-04-14T15:18:11.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:11.956+0000 7fe092ffd700 3 client.49090.objecter handle_osd_map got epochs [418,418] > 0 2024-04-14T15:18:11.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:11.956+0000 7fe092ffd700 3 client.49090.objecter handle_osd_map decoding full epoch 418 2024-04-14T15:18:11.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:11.956+0000 7fe092ffd700 20 client.49090.objecter dump_active .. 0 homeless 2024-04-14T15:18:11.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:11.956+0000 7fe092ffd700 10 client.49090.objecter ms_handle_connect 0x7fe088082520 2024-04-14T15:18:11.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:11.961+0000 7fe092ffd700 1 -- 172.21.15.90:0/2568290024 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fe08c057320 con 0x7fe09c101500 2024-04-14T15:18:11.969 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:11 smithi090 ceph-mon[41670]: osdmap e417: 8 total, 8 up, 8 in 2024-04-14T15:18:11.969 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:11 smithi090 ceph-mon[30399]: osdmap e417: 8 total, 8 up, 8 in 2024-04-14T15:18:11.970 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-04-14T15:18:11.971 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph pg dump --format=json 2024-04-14T15:18:11.972 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:18:11.996 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:11.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:11.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:18:12.098 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:18:12.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:12.122 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:12.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:18:12.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:12.180+0000 7fe0a3b8d700 1 -- 172.21.15.90:0/2568290024 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7fe09c0693d0 con 0x7fe09c101500 2024-04-14T15:18:12.224 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:18:12.230 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:11 smithi092 ceph-mon[33966]: osdmap e417: 8 total, 8 up, 8 in 2024-04-14T15:18:12.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:12.249 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:12.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:18:12.350 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:18:12.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:12.376 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:12.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:18:12.477 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:18:12.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:12.502 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:12.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:18:12.603 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:18:12.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:12.629 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:12.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:18:12.730 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:18:12.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:12.754 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:12.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:18:12.783 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-04-14T15:18:12.783 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-04-14T15:18:12.783 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 0 2024-04-14T15:18:12.784 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-04-14T15:18:12.855 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:18:12.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:12.879 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:12.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:18:12.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:12.937+0000 7fe092ffd700 1 -- 172.21.15.90:0/2568290024 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v419) ==== 97+0+0 (unknown 666608091 0 0) 0x7fe08c05afd0 con 0x7fe09c101500 2024-04-14T15:18:12.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:12.940+0000 7fe0a3b8d700 10 librados: watch_flush enter 2024-04-14T15:18:12.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:12.940+0000 7fe0a3b8d700 10 librados: watch_flush exit 2024-04-14T15:18:12.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:12.940+0000 7fe0a3b8d700 20 client.49090.objecter shutdown clearing up homeless session... 2024-04-14T15:18:12.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:12.940+0000 7fe0a3b8d700 10 client.49090.objecter successfully canceled tick 2024-04-14T15:18:12.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:12.941+0000 7fe0a3b8d700 1 -- 172.21.15.90:0/2568290024 >> v1:172.21.15.90:6800/420935427 conn(0x7fe088082520 legacy=0x7fe0880849e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:12.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:12.941+0000 7fe0a3b8d700 1 -- 172.21.15.90:0/2568290024 >> v1:172.21.15.92:6789/0 conn(0x7fe09c101500 legacy=0x7fe09c114430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:12.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:12.941+0000 7fe0a3b8d700 1 -- 172.21.15.90:0/2568290024 shutdown_connections 2024-04-14T15:18:12.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:12.941+0000 7fe0a3b8d700 1 -- 172.21.15.90:0/2568290024 >> 172.21.15.90:0/2568290024 conn(0x7fe09c077b30 msgr2=0x7fe09c1073e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:18:12.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:12.945+0000 7fe0a3b8d700 1 -- 172.21.15.90:0/2568290024 shutdown_connections 2024-04-14T15:18:12.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:12.945+0000 7fe0a3b8d700 1 -- 172.21.15.90:0/2568290024 wait complete. 2024-04-14T15:18:12.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:12.945+0000 7fe0a3b8d700 1 librados: shutdown 2024-04-14T15:18:12.953 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is set 2024-04-14T15:18:12.980 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:18:13.004 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:13.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:13.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:18:13.095 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:12 smithi092 ceph-mon[33966]: pgmap v575: 206 pgs: 206 active+clean; 8.4 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-04-14T15:18:13.095 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:12 smithi092 ceph-mon[33966]: osdmap e418: 8 total, 8 up, 8 in 2024-04-14T15:18:13.095 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:12 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:18:13.096 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:12 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:18:13.096 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:12 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2568290024' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:18:13.096 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:12 smithi092 ceph-mon[33966]: from='client.49090 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:18:13.106 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:18:13.130 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:13.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:13.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:18:13.183 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:12 smithi090 ceph-mon[30399]: pgmap v575: 206 pgs: 206 active+clean; 8.4 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-04-14T15:18:13.183 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:12 smithi090 ceph-mon[30399]: osdmap e418: 8 total, 8 up, 8 in 2024-04-14T15:18:13.183 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:12 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:18:13.183 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:12 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:18:13.183 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:12 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2568290024' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:18:13.183 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:12 smithi090 ceph-mon[30399]: from='client.49090 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:18:13.183 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:12 smithi090 ceph-mon[41670]: pgmap v575: 206 pgs: 206 active+clean; 8.4 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-04-14T15:18:13.183 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:12 smithi090 ceph-mon[41670]: osdmap e418: 8 total, 8 up, 8 in 2024-04-14T15:18:13.184 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:12 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:18:13.184 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:12 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:18:13.184 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:12 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2568290024' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:18:13.184 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:12 smithi090 ceph-mon[41670]: from='client.49090 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:18:13.232 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:18:13.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:13.257 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:13.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:18:13.358 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:18:13.381 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:13.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:13.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:18:13.412 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-14T15:18:13.484 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:18:13.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:13.508 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:13.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:18:13.609 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:18:13.633 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:13.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:13.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:18:13.692 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:18:13.734 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:18:13.759 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:13.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:13.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:18:13.785 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:18:13.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.791+0000 7f2d3f526700 1 -- 172.21.15.90:0/2513565788 >> v1:172.21.15.90:6790/0 conn(0x7f2d3810e310 legacy=0x7f2d381107d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:13.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.791+0000 7f2d3f526700 1 -- 172.21.15.90:0/2513565788 shutdown_connections 2024-04-14T15:18:13.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.791+0000 7f2d3f526700 1 -- 172.21.15.90:0/2513565788 >> 172.21.15.90:0/2513565788 conn(0x7f2d380fd290 msgr2=0x7f2d380ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:18:13.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.792+0000 7f2d3f526700 1 -- 172.21.15.90:0/2513565788 shutdown_connections 2024-04-14T15:18:13.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.792+0000 7f2d3f526700 1 -- 172.21.15.90:0/2513565788 wait complete. 2024-04-14T15:18:13.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.792+0000 7f2d3f526700 1 librados: starting msgr at 2024-04-14T15:18:13.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.792+0000 7f2d3f526700 1 librados: starting objecter 2024-04-14T15:18:13.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.792+0000 7f2d3f526700 1 Processor -- start 2024-04-14T15:18:13.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.792+0000 7f2d3f526700 1 -- start start 2024-04-14T15:18:13.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.792+0000 7f2d3f526700 1 librados: setting wanted keys 2024-04-14T15:18:13.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.792+0000 7f2d3f526700 1 librados: calling monclient init 2024-04-14T15:18:13.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.793+0000 7f2d3f526700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f2d3810dfb0 con 0x7f2d3810a740 2024-04-14T15:18:13.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.793+0000 7f2d3f526700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f2d381a8f30 con 0x7f2d38107890 2024-04-14T15:18:13.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.793+0000 7f2d3f526700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f2d381aa110 con 0x7f2d3810e310 2024-04-14T15:18:13.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.793+0000 7f2d3cac1700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f2d3810a740 0x7f2d3806b400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:36250/0 (socket says 172.21.15.90:36250) 2024-04-14T15:18:13.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.793+0000 7f2d3cac1700 1 -- 172.21.15.90:0/70627987 learned_addr learned my addr 172.21.15.90:0/70627987 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:18:13.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.793+0000 7f2d3d2c2700 1 --1- 172.21.15.90:0/70627987 >> v1:172.21.15.92:6789/0 conn(0x7f2d38107890 0x7f2d3806acf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.90:50512/0 (socket says 172.21.15.90:50512) 2024-04-14T15:18:13.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.793+0000 7f2d2e7fc700 10 client.?.objecter ms_handle_connect 0x7f2d38107890 2024-04-14T15:18:13.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.793+0000 7f2d2e7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:18:13.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.793+0000 7f2d2e7fc700 10 client.?.objecter ms_handle_connect 0x7f2d3810a740 2024-04-14T15:18:13.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.793+0000 7f2d2e7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:18:13.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.793+0000 7f2d2e7fc700 10 client.?.objecter ms_handle_connect 0x7f2d3810e310 2024-04-14T15:18:13.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.793+0000 7f2d2e7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:18:13.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.793+0000 7f2d2e7fc700 1 -- 172.21.15.90:0/70627987 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 700589208 0 0) 0x7f2d381a8f30 con 0x7f2d38107890 2024-04-14T15:18:13.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.793+0000 7f2d2e7fc700 1 -- 172.21.15.90:0/70627987 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d240033a0 con 0x7f2d38107890 2024-04-14T15:18:13.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.793+0000 7f2d2e7fc700 1 -- 172.21.15.90:0/70627987 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2433019380 0 0) 0x7f2d3810dfb0 con 0x7f2d3810a740 2024-04-14T15:18:13.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.793+0000 7f2d2e7fc700 1 -- 172.21.15.90:0/70627987 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d381a8f30 con 0x7f2d3810a740 2024-04-14T15:18:13.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.793+0000 7f2d2e7fc700 1 -- 172.21.15.90:0/70627987 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2935138797 0 0) 0x7f2d381aa110 con 0x7f2d3810e310 2024-04-14T15:18:13.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.793+0000 7f2d2e7fc700 1 -- 172.21.15.90:0/70627987 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d3810dfb0 con 0x7f2d3810e310 2024-04-14T15:18:13.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.793+0000 7f2d2e7fc700 1 -- 172.21.15.90:0/70627987 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 336808323 0 0) 0x7f2d240033a0 con 0x7f2d38107890 2024-04-14T15:18:13.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.793+0000 7f2d2e7fc700 1 -- 172.21.15.90:0/70627987 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2d381aa110 con 0x7f2d38107890 2024-04-14T15:18:13.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.793+0000 7f2d2e7fc700 1 -- 172.21.15.90:0/70627987 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f2d340018f0 con 0x7f2d38107890 2024-04-14T15:18:13.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.794+0000 7f2d2e7fc700 1 -- 172.21.15.90:0/70627987 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 153298433 0 0) 0x7f2d381a8f30 con 0x7f2d3810a740 2024-04-14T15:18:13.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.794+0000 7f2d2e7fc700 1 -- 172.21.15.90:0/70627987 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2d240033a0 con 0x7f2d3810a740 2024-04-14T15:18:13.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.794+0000 7f2d2e7fc700 1 -- 172.21.15.90:0/70627987 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f2d28001a30 con 0x7f2d3810a740 2024-04-14T15:18:13.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.794+0000 7f2d2e7fc700 1 -- 172.21.15.90:0/70627987 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1716731338 0 0) 0x7f2d381aa110 con 0x7f2d38107890 2024-04-14T15:18:13.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.794+0000 7f2d2e7fc700 1 -- 172.21.15.90:0/70627987 >> v1:172.21.15.90:6790/0 conn(0x7f2d3810e310 legacy=0x7f2d381a6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:13.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.794+0000 7f2d2e7fc700 1 -- 172.21.15.90:0/70627987 >> v1:172.21.15.90:6789/0 conn(0x7f2d3810a740 legacy=0x7f2d3806b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:13.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.794+0000 7f2d2e7fc700 1 -- 172.21.15.90:0/70627987 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d381ab2f0 con 0x7f2d38107890 2024-04-14T15:18:13.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.794+0000 7f2d3f526700 1 -- 172.21.15.90:0/70627987 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2d381a7f80 con 0x7f2d38107890 2024-04-14T15:18:13.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.794+0000 7f2d3f526700 10 client.49096.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:18:13.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.794+0000 7f2d3f526700 1 -- 172.21.15.90:0/70627987 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f2d381a8440 con 0x7f2d38107890 2024-04-14T15:18:13.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.794+0000 7f2d3f526700 1 librados: init done 2024-04-14T15:18:13.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.794+0000 7f2d2e7fc700 1 -- 172.21.15.90:0/70627987 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f2d34001cc0 con 0x7f2d38107890 2024-04-14T15:18:13.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.794+0000 7f2d2e7fc700 1 -- 172.21.15.90:0/70627987 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f2d34005d20 con 0x7f2d38107890 2024-04-14T15:18:13.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.795+0000 7f2d3f526700 1 -- 172.21.15.90:0/70627987 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f2d1c005100 con 0x7f2d38107890 2024-04-14T15:18:13.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.795+0000 7f2d2e7fc700 1 -- 172.21.15.90:0/70627987 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f2d3401c8d0 con 0x7f2d38107890 2024-04-14T15:18:13.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.796+0000 7f2d2e7fc700 1 -- 172.21.15.90:0/70627987 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(419..419 src has 51..419) ==== 8160+0+0 (unknown 3855095629 0 0) 0x7f2d3408ab50 con 0x7f2d38107890 2024-04-14T15:18:13.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.796+0000 7f2d2e7fc700 10 client.49096.objecter ms_dispatch 0x7f2d38001550 osd_map(419..419 src has 51..419) 2024-04-14T15:18:13.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.796+0000 7f2d2e7fc700 3 client.49096.objecter handle_osd_map got epochs [419,419] > 0 2024-04-14T15:18:13.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.796+0000 7f2d2e7fc700 3 client.49096.objecter handle_osd_map decoding full epoch 419 2024-04-14T15:18:13.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.796+0000 7f2d2e7fc700 20 client.49096.objecter dump_active .. 0 homeless 2024-04-14T15:18:13.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.796+0000 7f2d2e7fc700 1 -- 172.21.15.90:0/70627987 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v419) ==== 54+0+16 (unknown 2467965483 0 2420000192) 0x7f2d3408af40 con 0x7f2d38107890 2024-04-14T15:18:13.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.796+0000 7f2d2e7fc700 10 client.49096.objecter ms_handle_connect 0x7f2d24071140 2024-04-14T15:18:13.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.796+0000 7f2d3f526700 10 client.49096.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:18:13.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.796+0000 7f2d3f526700 15 client.49096.objecter _session_command_op_assign -1 1 2024-04-14T15:18:13.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.796+0000 7f2d3f526700 20 client.49096.objecter _get_session s=0x7f2d1c0053e0 osd=0 3 2024-04-14T15:18:13.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.796+0000 7f2d3f526700 20 client.49096.objecter put_session s=0x7f2d1c0053e0 osd=0 3 2024-04-14T15:18:13.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.796+0000 7f2d3f526700 20 client.49096.objecter _get_session s=0x7f2d1c0053e0 osd=0 3 2024-04-14T15:18:13.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.796+0000 7f2d3f526700 15 client.49096.objecter _session_command_op_remove -1 1 2024-04-14T15:18:13.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.796+0000 7f2d3f526700 20 client.49096.objecter get_session s=0x7f2d1c0053e0 osd=0 3 2024-04-14T15:18:13.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.796+0000 7f2d3f526700 15 client.49096.objecter _session_command_op_assign 0 1 2024-04-14T15:18:13.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.796+0000 7f2d3f526700 20 client.49096.objecter put_session s=0x7f2d1c0053e0 osd=0 4 2024-04-14T15:18:13.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.796+0000 7f2d3f526700 10 client.49096.objecter _send_command 1 2024-04-14T15:18:13.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.796+0000 7f2d3f526700 1 -- 172.21.15.90:0/70627987 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2d1c004b70 con 0x7f2d1c004730 2024-04-14T15:18:13.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.797+0000 7f2d2e7fc700 10 client.49096.objecter ms_handle_connect 0x7f2d1c004730 2024-04-14T15:18:13.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.798+0000 7f2d2e7fc700 1 -- 172.21.15.90:0/70627987 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2d1c004b70 con 0x7f2d1c004730 2024-04-14T15:18:13.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.798+0000 7f2d2e7fc700 10 client.49096.objecter ms_dispatch 0x7f2d38001550 command_reply(tid 1: 0 ) 2024-04-14T15:18:13.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.798+0000 7f2d2e7fc700 10 client.49096.objecter _finish_command 1 = system:0 2024-04-14T15:18:13.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.798+0000 7f2d2e7fc700 20 client.49096.objecter put_session s=0x7f2d1c0053e0 osd=0 4 2024-04-14T15:18:13.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.798+0000 7f2d2e7fc700 15 client.49096.objecter _session_command_op_remove 0 1 2024-04-14T15:18:13.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.822+0000 7f2d3f526700 10 client.49096.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:18:13.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.822+0000 7f2d3f526700 15 client.49096.objecter _session_command_op_assign -1 2 2024-04-14T15:18:13.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.822+0000 7f2d3f526700 20 client.49096.objecter _get_session s=0x7f2d1c0053e0 osd=0 3 2024-04-14T15:18:13.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.822+0000 7f2d3f526700 20 client.49096.objecter put_session s=0x7f2d1c0053e0 osd=0 3 2024-04-14T15:18:13.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.822+0000 7f2d3f526700 20 client.49096.objecter _get_session s=0x7f2d1c0053e0 osd=0 3 2024-04-14T15:18:13.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.822+0000 7f2d3f526700 15 client.49096.objecter _session_command_op_remove -1 2 2024-04-14T15:18:13.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.822+0000 7f2d3f526700 20 client.49096.objecter get_session s=0x7f2d1c0053e0 osd=0 3 2024-04-14T15:18:13.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.822+0000 7f2d3f526700 15 client.49096.objecter _session_command_op_assign 0 2 2024-04-14T15:18:13.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.822+0000 7f2d3f526700 20 client.49096.objecter put_session s=0x7f2d1c0053e0 osd=0 4 2024-04-14T15:18:13.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.822+0000 7f2d3f526700 10 client.49096.objecter _send_command 2 2024-04-14T15:18:13.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.822+0000 7f2d3f526700 1 -- 172.21.15.90:0/70627987 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f2d1c0079f0 con 0x7f2d1c004730 2024-04-14T15:18:13.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.824+0000 7f2d2e7fc700 1 -- 172.21.15.90:0/70627987 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2d1c0079f0 con 0x7f2d1c004730 2024-04-14T15:18:13.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.824+0000 7f2d2e7fc700 10 client.49096.objecter ms_dispatch 0x7f2d38001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:18:13.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.824+0000 7f2d2e7fc700 10 client.49096.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:18:13.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.824+0000 7f2d2e7fc700 20 client.49096.objecter put_session s=0x7f2d1c0053e0 osd=0 4 2024-04-14T15:18:13.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.825+0000 7f2d2e7fc700 15 client.49096.objecter _session_command_op_remove 0 2 2024-04-14T15:18:13.825 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:18:13.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.825+0000 7f2d3f526700 10 client.49096.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:18:13.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.825+0000 7f2d3f526700 15 client.49096.objecter _session_command_op_assign -1 3 2024-04-14T15:18:13.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.825+0000 7f2d3f526700 20 client.49096.objecter _get_session s=0x7f2d1c004130 osd=1 3 2024-04-14T15:18:13.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.825+0000 7f2d3f526700 20 client.49096.objecter put_session s=0x7f2d1c004130 osd=1 3 2024-04-14T15:18:13.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.825+0000 7f2d3f526700 20 client.49096.objecter _get_session s=0x7f2d1c004130 osd=1 3 2024-04-14T15:18:13.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.825+0000 7f2d3f526700 15 client.49096.objecter _session_command_op_remove -1 3 2024-04-14T15:18:13.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.825+0000 7f2d3f526700 20 client.49096.objecter get_session s=0x7f2d1c004130 osd=1 3 2024-04-14T15:18:13.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.825+0000 7f2d3f526700 15 client.49096.objecter _session_command_op_assign 1 3 2024-04-14T15:18:13.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.825+0000 7f2d3f526700 20 client.49096.objecter put_session s=0x7f2d1c004130 osd=1 4 2024-04-14T15:18:13.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.825+0000 7f2d3f526700 10 client.49096.objecter _send_command 3 2024-04-14T15:18:13.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.825+0000 7f2d3f526700 1 -- 172.21.15.90:0/70627987 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f2d1c002240 con 0x7f2d1c004310 2024-04-14T15:18:13.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.825+0000 7f2d2e7fc700 10 client.49096.objecter ms_handle_connect 0x7f2d1c004310 2024-04-14T15:18:13.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.826+0000 7f2d2e7fc700 1 -- 172.21.15.90:0/70627987 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2d1c002240 con 0x7f2d1c004310 2024-04-14T15:18:13.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.826+0000 7f2d2e7fc700 10 client.49096.objecter ms_dispatch 0x7f2d38001550 command_reply(tid 3: 0 ) 2024-04-14T15:18:13.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.826+0000 7f2d2e7fc700 10 client.49096.objecter _finish_command 3 = system:0 2024-04-14T15:18:13.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.826+0000 7f2d2e7fc700 20 client.49096.objecter put_session s=0x7f2d1c004130 osd=1 4 2024-04-14T15:18:13.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.826+0000 7f2d2e7fc700 15 client.49096.objecter _session_command_op_remove 1 3 2024-04-14T15:18:13.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.843+0000 7f2d3f526700 10 client.49096.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:18:13.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.843+0000 7f2d3f526700 15 client.49096.objecter _session_command_op_assign -1 4 2024-04-14T15:18:13.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.843+0000 7f2d3f526700 20 client.49096.objecter _get_session s=0x7f2d1c004130 osd=1 3 2024-04-14T15:18:13.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.843+0000 7f2d3f526700 20 client.49096.objecter put_session s=0x7f2d1c004130 osd=1 3 2024-04-14T15:18:13.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.843+0000 7f2d3f526700 20 client.49096.objecter _get_session s=0x7f2d1c004130 osd=1 3 2024-04-14T15:18:13.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.843+0000 7f2d3f526700 15 client.49096.objecter _session_command_op_remove -1 4 2024-04-14T15:18:13.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.843+0000 7f2d3f526700 20 client.49096.objecter get_session s=0x7f2d1c004130 osd=1 3 2024-04-14T15:18:13.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.843+0000 7f2d3f526700 15 client.49096.objecter _session_command_op_assign 1 4 2024-04-14T15:18:13.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.843+0000 7f2d3f526700 20 client.49096.objecter put_session s=0x7f2d1c004130 osd=1 4 2024-04-14T15:18:13.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.843+0000 7f2d3f526700 10 client.49096.objecter _send_command 4 2024-04-14T15:18:13.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.843+0000 7f2d3f526700 1 -- 172.21.15.90:0/70627987 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f2d1c001660 con 0x7f2d1c004310 2024-04-14T15:18:13.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.844+0000 7f2d2e7fc700 1 -- 172.21.15.90:0/70627987 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2d1c001660 con 0x7f2d1c004310 2024-04-14T15:18:13.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.844+0000 7f2d2e7fc700 10 client.49096.objecter ms_dispatch 0x7f2d38001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:18:13.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.844+0000 7f2d2e7fc700 10 client.49096.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:18:13.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.844+0000 7f2d2e7fc700 20 client.49096.objecter put_session s=0x7f2d1c004130 osd=1 4 2024-04-14T15:18:13.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.844+0000 7f2d2e7fc700 15 client.49096.objecter _session_command_op_remove 1 4 2024-04-14T15:18:13.845 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:18:13.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.844+0000 7f2d3f526700 10 client.49096.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:18:13.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.844+0000 7f2d3f526700 15 client.49096.objecter _session_command_op_assign -1 5 2024-04-14T15:18:13.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.844+0000 7f2d3f526700 20 client.49096.objecter _get_session s=0x7f2d1c008370 osd=2 3 2024-04-14T15:18:13.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.844+0000 7f2d3f526700 20 client.49096.objecter put_session s=0x7f2d1c008370 osd=2 3 2024-04-14T15:18:13.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.844+0000 7f2d3f526700 20 client.49096.objecter _get_session s=0x7f2d1c008370 osd=2 3 2024-04-14T15:18:13.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.844+0000 7f2d3f526700 15 client.49096.objecter _session_command_op_remove -1 5 2024-04-14T15:18:13.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.844+0000 7f2d3f526700 20 client.49096.objecter get_session s=0x7f2d1c008370 osd=2 3 2024-04-14T15:18:13.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.844+0000 7f2d3f526700 15 client.49096.objecter _session_command_op_assign 2 5 2024-04-14T15:18:13.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.844+0000 7f2d3f526700 20 client.49096.objecter put_session s=0x7f2d1c008370 osd=2 4 2024-04-14T15:18:13.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.844+0000 7f2d3f526700 10 client.49096.objecter _send_command 5 2024-04-14T15:18:13.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.844+0000 7f2d3f526700 1 -- 172.21.15.90:0/70627987 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f2d1c015570 con 0x7f2d1c011a60 2024-04-14T15:18:13.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.845+0000 7f2d2e7fc700 10 client.49096.objecter ms_handle_connect 0x7f2d1c011a60 2024-04-14T15:18:13.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.845+0000 7f2d2e7fc700 1 -- 172.21.15.90:0/70627987 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2d1c015570 con 0x7f2d1c011a60 2024-04-14T15:18:13.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.845+0000 7f2d2e7fc700 10 client.49096.objecter ms_dispatch 0x7f2d38001550 command_reply(tid 5: 0 ) 2024-04-14T15:18:13.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.845+0000 7f2d2e7fc700 10 client.49096.objecter _finish_command 5 = system:0 2024-04-14T15:18:13.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.845+0000 7f2d2e7fc700 20 client.49096.objecter put_session s=0x7f2d1c008370 osd=2 4 2024-04-14T15:18:13.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.845+0000 7f2d2e7fc700 15 client.49096.objecter _session_command_op_remove 2 5 2024-04-14T15:18:13.861 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:18:13.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.861+0000 7f2d3f526700 10 client.49096.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:18:13.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.861+0000 7f2d3f526700 15 client.49096.objecter _session_command_op_assign -1 6 2024-04-14T15:18:13.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.861+0000 7f2d3f526700 20 client.49096.objecter _get_session s=0x7f2d1c008370 osd=2 3 2024-04-14T15:18:13.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.861+0000 7f2d3f526700 20 client.49096.objecter put_session s=0x7f2d1c008370 osd=2 3 2024-04-14T15:18:13.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.861+0000 7f2d3f526700 20 client.49096.objecter _get_session s=0x7f2d1c008370 osd=2 3 2024-04-14T15:18:13.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.861+0000 7f2d3f526700 15 client.49096.objecter _session_command_op_remove -1 6 2024-04-14T15:18:13.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.861+0000 7f2d3f526700 20 client.49096.objecter get_session s=0x7f2d1c008370 osd=2 3 2024-04-14T15:18:13.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.861+0000 7f2d3f526700 15 client.49096.objecter _session_command_op_assign 2 6 2024-04-14T15:18:13.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.861+0000 7f2d3f526700 20 client.49096.objecter put_session s=0x7f2d1c008370 osd=2 4 2024-04-14T15:18:13.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.861+0000 7f2d3f526700 10 client.49096.objecter _send_command 6 2024-04-14T15:18:13.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.861+0000 7f2d3f526700 1 -- 172.21.15.90:0/70627987 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f2d1c007e50 con 0x7f2d1c011a60 2024-04-14T15:18:13.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.863+0000 7f2d2e7fc700 1 -- 172.21.15.90:0/70627987 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2d1c007e50 con 0x7f2d1c011a60 2024-04-14T15:18:13.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.863+0000 7f2d2e7fc700 10 client.49096.objecter ms_dispatch 0x7f2d38001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:18:13.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.863+0000 7f2d2e7fc700 10 client.49096.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:18:13.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.863+0000 7f2d2e7fc700 20 client.49096.objecter put_session s=0x7f2d1c008370 osd=2 4 2024-04-14T15:18:13.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.863+0000 7f2d2e7fc700 15 client.49096.objecter _session_command_op_remove 2 6 2024-04-14T15:18:13.865 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:18:13.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.863+0000 7f2d3f526700 10 client.49096.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:18:13.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.863+0000 7f2d3f526700 15 client.49096.objecter _session_command_op_assign -1 7 2024-04-14T15:18:13.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.863+0000 7f2d3f526700 20 client.49096.objecter _get_session s=0x7f2d1c0110b0 osd=3 3 2024-04-14T15:18:13.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.863+0000 7f2d3f526700 20 client.49096.objecter put_session s=0x7f2d1c0110b0 osd=3 3 2024-04-14T15:18:13.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.863+0000 7f2d3f526700 20 client.49096.objecter _get_session s=0x7f2d1c0110b0 osd=3 3 2024-04-14T15:18:13.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.863+0000 7f2d3f526700 15 client.49096.objecter _session_command_op_remove -1 7 2024-04-14T15:18:13.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.863+0000 7f2d3f526700 20 client.49096.objecter get_session s=0x7f2d1c0110b0 osd=3 3 2024-04-14T15:18:13.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.863+0000 7f2d3f526700 15 client.49096.objecter _session_command_op_assign 3 7 2024-04-14T15:18:13.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.863+0000 7f2d3f526700 20 client.49096.objecter put_session s=0x7f2d1c0110b0 osd=3 4 2024-04-14T15:18:13.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.863+0000 7f2d3f526700 10 client.49096.objecter _send_command 7 2024-04-14T15:18:13.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.863+0000 7f2d3f526700 1 -- 172.21.15.90:0/70627987 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f2d1c0173f0 con 0x7f2d1c0168a0 2024-04-14T15:18:13.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.864+0000 7f2d2e7fc700 10 client.49096.objecter ms_handle_connect 0x7f2d1c0168a0 2024-04-14T15:18:13.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.864+0000 7f2d2e7fc700 1 -- 172.21.15.90:0/70627987 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2d1c0173f0 con 0x7f2d1c0168a0 2024-04-14T15:18:13.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.864+0000 7f2d2e7fc700 10 client.49096.objecter ms_dispatch 0x7f2d38001550 command_reply(tid 7: 0 ) 2024-04-14T15:18:13.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.864+0000 7f2d2e7fc700 10 client.49096.objecter _finish_command 7 = system:0 2024-04-14T15:18:13.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.864+0000 7f2d2e7fc700 20 client.49096.objecter put_session s=0x7f2d1c0110b0 osd=3 4 2024-04-14T15:18:13.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.864+0000 7f2d2e7fc700 15 client.49096.objecter _session_command_op_remove 3 7 2024-04-14T15:18:13.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.883+0000 7f2d3f526700 10 client.49096.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:18:13.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.883+0000 7f2d3f526700 15 client.49096.objecter _session_command_op_assign -1 8 2024-04-14T15:18:13.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.883+0000 7f2d3f526700 20 client.49096.objecter _get_session s=0x7f2d1c0110b0 osd=3 3 2024-04-14T15:18:13.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.883+0000 7f2d3f526700 20 client.49096.objecter put_session s=0x7f2d1c0110b0 osd=3 3 2024-04-14T15:18:13.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.883+0000 7f2d3f526700 20 client.49096.objecter _get_session s=0x7f2d1c0110b0 osd=3 3 2024-04-14T15:18:13.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.883+0000 7f2d3f526700 15 client.49096.objecter _session_command_op_remove -1 8 2024-04-14T15:18:13.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.883+0000 7f2d3f526700 20 client.49096.objecter get_session s=0x7f2d1c0110b0 osd=3 3 2024-04-14T15:18:13.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.883+0000 7f2d3f526700 15 client.49096.objecter _session_command_op_assign 3 8 2024-04-14T15:18:13.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.883+0000 7f2d3f526700 20 client.49096.objecter put_session s=0x7f2d1c0110b0 osd=3 4 2024-04-14T15:18:13.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.883+0000 7f2d3f526700 10 client.49096.objecter _send_command 8 2024-04-14T15:18:13.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.883+0000 7f2d3f526700 1 -- 172.21.15.90:0/70627987 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f2d1c0173f0 con 0x7f2d1c0168a0 2024-04-14T15:18:13.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.885+0000 7f2d2e7fc700 1 -- 172.21.15.90:0/70627987 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2d1c0173f0 con 0x7f2d1c0168a0 2024-04-14T15:18:13.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.885+0000 7f2d2e7fc700 10 client.49096.objecter ms_dispatch 0x7f2d38001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:18:13.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.885+0000 7f2d2e7fc700 10 client.49096.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:18:13.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.885+0000 7f2d2e7fc700 20 client.49096.objecter put_session s=0x7f2d1c0110b0 osd=3 4 2024-04-14T15:18:13.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.885+0000 7f2d2e7fc700 15 client.49096.objecter _session_command_op_remove 3 8 2024-04-14T15:18:13.886 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:18:13.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.885+0000 7f2d3f526700 10 client.49096.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:18:13.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.885+0000 7f2d3f526700 15 client.49096.objecter _session_command_op_assign -1 9 2024-04-14T15:18:13.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.885+0000 7f2d3f526700 20 client.49096.objecter _get_session s=0x7f2d1c0173f0 osd=4 3 2024-04-14T15:18:13.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.885+0000 7f2d3f526700 20 client.49096.objecter put_session s=0x7f2d1c0173f0 osd=4 3 2024-04-14T15:18:13.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.885+0000 7f2d3f526700 20 client.49096.objecter _get_session s=0x7f2d1c0173f0 osd=4 3 2024-04-14T15:18:13.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.885+0000 7f2d3f526700 15 client.49096.objecter _session_command_op_remove -1 9 2024-04-14T15:18:13.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.885+0000 7f2d3f526700 20 client.49096.objecter get_session s=0x7f2d1c0173f0 osd=4 3 2024-04-14T15:18:13.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.885+0000 7f2d3f526700 15 client.49096.objecter _session_command_op_assign 4 9 2024-04-14T15:18:13.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.885+0000 7f2d3f526700 20 client.49096.objecter put_session s=0x7f2d1c0173f0 osd=4 4 2024-04-14T15:18:13.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.885+0000 7f2d3f526700 10 client.49096.objecter _send_command 9 2024-04-14T15:18:13.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.885+0000 7f2d3f526700 1 -- 172.21.15.90:0/70627987 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f2d1c01ebc0 con 0x7f2d1c01b0b0 2024-04-14T15:18:13.890 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:13.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.885+0000 7f2d2e7fc700 10 client.49096.objecter ms_handle_connect 0x7f2d1c01b0b0 2024-04-14T15:18:13.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:13.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:18:13.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.886+0000 7f2d2e7fc700 1 -- 172.21.15.90:0/70627987 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2d1c01ebc0 con 0x7f2d1c01b0b0 2024-04-14T15:18:13.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.886+0000 7f2d2e7fc700 10 client.49096.objecter ms_dispatch 0x7f2d38001550 command_reply(tid 9: 0 ) 2024-04-14T15:18:13.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.886+0000 7f2d2e7fc700 10 client.49096.objecter _finish_command 9 = system:0 2024-04-14T15:18:13.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.886+0000 7f2d2e7fc700 20 client.49096.objecter put_session s=0x7f2d1c0173f0 osd=4 4 2024-04-14T15:18:13.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.886+0000 7f2d2e7fc700 15 client.49096.objecter _session_command_op_remove 4 9 2024-04-14T15:18:13.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.903+0000 7f2d3f526700 10 client.49096.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:18:13.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.903+0000 7f2d3f526700 15 client.49096.objecter _session_command_op_assign -1 10 2024-04-14T15:18:13.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.903+0000 7f2d3f526700 20 client.49096.objecter _get_session s=0x7f2d1c0173f0 osd=4 3 2024-04-14T15:18:13.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.903+0000 7f2d3f526700 20 client.49096.objecter put_session s=0x7f2d1c0173f0 osd=4 3 2024-04-14T15:18:13.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.903+0000 7f2d3f526700 20 client.49096.objecter _get_session s=0x7f2d1c0173f0 osd=4 3 2024-04-14T15:18:13.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.903+0000 7f2d3f526700 15 client.49096.objecter _session_command_op_remove -1 10 2024-04-14T15:18:13.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.903+0000 7f2d3f526700 20 client.49096.objecter get_session s=0x7f2d1c0173f0 osd=4 3 2024-04-14T15:18:13.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.903+0000 7f2d3f526700 15 client.49096.objecter _session_command_op_assign 4 10 2024-04-14T15:18:13.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.903+0000 7f2d3f526700 20 client.49096.objecter put_session s=0x7f2d1c0173f0 osd=4 4 2024-04-14T15:18:13.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.903+0000 7f2d3f526700 10 client.49096.objecter _send_command 10 2024-04-14T15:18:13.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.903+0000 7f2d3f526700 1 -- 172.21.15.90:0/70627987 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f2d1c017600 con 0x7f2d1c01b0b0 2024-04-14T15:18:13.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.904+0000 7f2d2e7fc700 1 -- 172.21.15.90:0/70627987 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2d1c017600 con 0x7f2d1c01b0b0 2024-04-14T15:18:13.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.904+0000 7f2d2e7fc700 10 client.49096.objecter ms_dispatch 0x7f2d38001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:18:13.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.904+0000 7f2d2e7fc700 10 client.49096.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:18:13.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.904+0000 7f2d2e7fc700 20 client.49096.objecter put_session s=0x7f2d1c0173f0 osd=4 4 2024-04-14T15:18:13.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.904+0000 7f2d2e7fc700 15 client.49096.objecter _session_command_op_remove 4 10 2024-04-14T15:18:13.905 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:18:13.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.905+0000 7f2d3f526700 10 client.49096.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:18:13.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.905+0000 7f2d3f526700 15 client.49096.objecter _session_command_op_assign -1 11 2024-04-14T15:18:13.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.905+0000 7f2d3f526700 20 client.49096.objecter _get_session s=0x7f2d1c017600 osd=5 3 2024-04-14T15:18:13.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.905+0000 7f2d3f526700 20 client.49096.objecter put_session s=0x7f2d1c017600 osd=5 3 2024-04-14T15:18:13.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.905+0000 7f2d3f526700 20 client.49096.objecter _get_session s=0x7f2d1c017600 osd=5 3 2024-04-14T15:18:13.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.905+0000 7f2d3f526700 15 client.49096.objecter _session_command_op_remove -1 11 2024-04-14T15:18:13.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.905+0000 7f2d3f526700 20 client.49096.objecter get_session s=0x7f2d1c017600 osd=5 3 2024-04-14T15:18:13.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.905+0000 7f2d3f526700 15 client.49096.objecter _session_command_op_assign 5 11 2024-04-14T15:18:13.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.905+0000 7f2d3f526700 20 client.49096.objecter put_session s=0x7f2d1c017600 osd=5 4 2024-04-14T15:18:13.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.905+0000 7f2d3f526700 10 client.49096.objecter _send_command 11 2024-04-14T15:18:13.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.905+0000 7f2d3f526700 1 -- 172.21.15.90:0/70627987 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f2d1c016470 con 0x7f2d1c015ef0 2024-04-14T15:18:13.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.905+0000 7f2d2e7fc700 10 client.49096.objecter ms_handle_connect 0x7f2d1c015ef0 2024-04-14T15:18:13.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.906+0000 7f2d2e7fc700 1 -- 172.21.15.90:0/70627987 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2d1c016470 con 0x7f2d1c015ef0 2024-04-14T15:18:13.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.906+0000 7f2d2e7fc700 10 client.49096.objecter ms_dispatch 0x7f2d38001550 command_reply(tid 11: 0 ) 2024-04-14T15:18:13.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.906+0000 7f2d2e7fc700 10 client.49096.objecter _finish_command 11 = system:0 2024-04-14T15:18:13.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.906+0000 7f2d2e7fc700 20 client.49096.objecter put_session s=0x7f2d1c017600 osd=5 4 2024-04-14T15:18:13.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.906+0000 7f2d2e7fc700 15 client.49096.objecter _session_command_op_remove 5 11 2024-04-14T15:18:13.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.922+0000 7f2d3f526700 10 client.49096.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:18:13.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.922+0000 7f2d3f526700 15 client.49096.objecter _session_command_op_assign -1 12 2024-04-14T15:18:13.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.922+0000 7f2d3f526700 20 client.49096.objecter _get_session s=0x7f2d1c017600 osd=5 3 2024-04-14T15:18:13.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.922+0000 7f2d3f526700 20 client.49096.objecter put_session s=0x7f2d1c017600 osd=5 3 2024-04-14T15:18:13.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.922+0000 7f2d3f526700 20 client.49096.objecter _get_session s=0x7f2d1c017600 osd=5 3 2024-04-14T15:18:13.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.922+0000 7f2d3f526700 15 client.49096.objecter _session_command_op_remove -1 12 2024-04-14T15:18:13.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.922+0000 7f2d3f526700 20 client.49096.objecter get_session s=0x7f2d1c017600 osd=5 3 2024-04-14T15:18:13.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.922+0000 7f2d3f526700 15 client.49096.objecter _session_command_op_assign 5 12 2024-04-14T15:18:13.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.922+0000 7f2d3f526700 20 client.49096.objecter put_session s=0x7f2d1c017600 osd=5 4 2024-04-14T15:18:13.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.922+0000 7f2d3f526700 10 client.49096.objecter _send_command 12 2024-04-14T15:18:13.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.922+0000 7f2d3f526700 1 -- 172.21.15.90:0/70627987 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f2d1c016680 con 0x7f2d1c015ef0 2024-04-14T15:18:13.924 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:18:13.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.924+0000 7f2d2e7fc700 1 -- 172.21.15.90:0/70627987 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2d1c016680 con 0x7f2d1c015ef0 2024-04-14T15:18:13.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.924+0000 7f2d2e7fc700 10 client.49096.objecter ms_dispatch 0x7f2d38001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:18:13.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.924+0000 7f2d2e7fc700 10 client.49096.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:18:13.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.924+0000 7f2d2e7fc700 20 client.49096.objecter put_session s=0x7f2d1c017600 osd=5 4 2024-04-14T15:18:13.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.924+0000 7f2d2e7fc700 15 client.49096.objecter _session_command_op_remove 5 12 2024-04-14T15:18:13.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.924+0000 7f2d3f526700 10 client.49096.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:18:13.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.924+0000 7f2d3f526700 15 client.49096.objecter _session_command_op_assign -1 13 2024-04-14T15:18:13.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.924+0000 7f2d3f526700 20 client.49096.objecter _get_session s=0x7f2d1c016490 osd=6 3 2024-04-14T15:18:13.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.924+0000 7f2d3f526700 20 client.49096.objecter put_session s=0x7f2d1c016490 osd=6 3 2024-04-14T15:18:13.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.924+0000 7f2d3f526700 20 client.49096.objecter _get_session s=0x7f2d1c016490 osd=6 3 2024-04-14T15:18:13.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.924+0000 7f2d3f526700 15 client.49096.objecter _session_command_op_remove -1 13 2024-04-14T15:18:13.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.924+0000 7f2d3f526700 20 client.49096.objecter get_session s=0x7f2d1c016490 osd=6 3 2024-04-14T15:18:13.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.924+0000 7f2d3f526700 15 client.49096.objecter _session_command_op_assign 6 13 2024-04-14T15:18:13.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.924+0000 7f2d3f526700 20 client.49096.objecter put_session s=0x7f2d1c016490 osd=6 4 2024-04-14T15:18:13.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.924+0000 7f2d3f526700 10 client.49096.objecter _send_command 13 2024-04-14T15:18:13.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.924+0000 7f2d3f526700 1 -- 172.21.15.90:0/70627987 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f2d1c0275a0 con 0x7f2d1c023980 2024-04-14T15:18:13.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.924+0000 7f2d2e7fc700 10 client.49096.objecter ms_handle_connect 0x7f2d1c023980 2024-04-14T15:18:13.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.925+0000 7f2d2e7fc700 1 -- 172.21.15.90:0/70627987 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2d1c0275a0 con 0x7f2d1c023980 2024-04-14T15:18:13.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.925+0000 7f2d2e7fc700 10 client.49096.objecter ms_dispatch 0x7f2d38001550 command_reply(tid 13: 0 ) 2024-04-14T15:18:13.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.925+0000 7f2d2e7fc700 10 client.49096.objecter _finish_command 13 = system:0 2024-04-14T15:18:13.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.925+0000 7f2d2e7fc700 20 client.49096.objecter put_session s=0x7f2d1c016490 osd=6 4 2024-04-14T15:18:13.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.925+0000 7f2d2e7fc700 15 client.49096.objecter _session_command_op_remove 6 13 2024-04-14T15:18:13.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.944+0000 7f2d3f526700 10 client.49096.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:18:13.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.944+0000 7f2d3f526700 15 client.49096.objecter _session_command_op_assign -1 14 2024-04-14T15:18:13.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.944+0000 7f2d3f526700 20 client.49096.objecter _get_session s=0x7f2d1c016490 osd=6 3 2024-04-14T15:18:13.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.944+0000 7f2d3f526700 20 client.49096.objecter put_session s=0x7f2d1c016490 osd=6 3 2024-04-14T15:18:13.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.944+0000 7f2d3f526700 20 client.49096.objecter _get_session s=0x7f2d1c016490 osd=6 3 2024-04-14T15:18:13.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.944+0000 7f2d3f526700 15 client.49096.objecter _session_command_op_remove -1 14 2024-04-14T15:18:13.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.944+0000 7f2d3f526700 20 client.49096.objecter get_session s=0x7f2d1c016490 osd=6 3 2024-04-14T15:18:13.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.944+0000 7f2d3f526700 15 client.49096.objecter _session_command_op_assign 6 14 2024-04-14T15:18:13.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.944+0000 7f2d3f526700 20 client.49096.objecter put_session s=0x7f2d1c016490 osd=6 4 2024-04-14T15:18:13.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.944+0000 7f2d3f526700 10 client.49096.objecter _send_command 14 2024-04-14T15:18:13.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.944+0000 7f2d3f526700 1 -- 172.21.15.90:0/70627987 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f2d1c027780 con 0x7f2d1c023980 2024-04-14T15:18:13.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.947+0000 7f2d2e7fc700 1 -- 172.21.15.90:0/70627987 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2d1c027780 con 0x7f2d1c023980 2024-04-14T15:18:13.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.947+0000 7f2d2e7fc700 10 client.49096.objecter ms_dispatch 0x7f2d38001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:18:13.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.947+0000 7f2d2e7fc700 10 client.49096.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:18:13.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.947+0000 7f2d2e7fc700 20 client.49096.objecter put_session s=0x7f2d1c016490 osd=6 4 2024-04-14T15:18:13.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.947+0000 7f2d2e7fc700 15 client.49096.objecter _session_command_op_remove 6 14 2024-04-14T15:18:13.948 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:18:13.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.947+0000 7f2d3f526700 10 client.49096.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:18:13.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.947+0000 7f2d3f526700 15 client.49096.objecter _session_command_op_assign -1 15 2024-04-14T15:18:13.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.947+0000 7f2d3f526700 20 client.49096.objecter _get_session s=0x7f2d1c027780 osd=7 3 2024-04-14T15:18:13.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.947+0000 7f2d3f526700 20 client.49096.objecter put_session s=0x7f2d1c027780 osd=7 3 2024-04-14T15:18:13.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.947+0000 7f2d3f526700 20 client.49096.objecter _get_session s=0x7f2d1c027780 osd=7 3 2024-04-14T15:18:13.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.947+0000 7f2d3f526700 15 client.49096.objecter _session_command_op_remove -1 15 2024-04-14T15:18:13.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.947+0000 7f2d3f526700 20 client.49096.objecter get_session s=0x7f2d1c027780 osd=7 3 2024-04-14T15:18:13.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.947+0000 7f2d3f526700 15 client.49096.objecter _session_command_op_assign 7 15 2024-04-14T15:18:13.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.947+0000 7f2d3f526700 20 client.49096.objecter put_session s=0x7f2d1c027780 osd=7 4 2024-04-14T15:18:13.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.947+0000 7f2d3f526700 10 client.49096.objecter _send_command 15 2024-04-14T15:18:13.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.947+0000 7f2d3f526700 1 -- 172.21.15.90:0/70627987 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f2d1c028a80 con 0x7f2d1c027ea0 2024-04-14T15:18:13.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.948+0000 7f2d2e7fc700 10 client.49096.objecter ms_handle_connect 0x7f2d1c027ea0 2024-04-14T15:18:13.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.949+0000 7f2d2e7fc700 1 -- 172.21.15.90:0/70627987 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2d1c028a80 con 0x7f2d1c027ea0 2024-04-14T15:18:13.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.949+0000 7f2d2e7fc700 10 client.49096.objecter ms_dispatch 0x7f2d38001550 command_reply(tid 15: 0 ) 2024-04-14T15:18:13.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.949+0000 7f2d2e7fc700 10 client.49096.objecter _finish_command 15 = system:0 2024-04-14T15:18:13.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.949+0000 7f2d2e7fc700 20 client.49096.objecter put_session s=0x7f2d1c027780 osd=7 4 2024-04-14T15:18:13.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.949+0000 7f2d2e7fc700 15 client.49096.objecter _session_command_op_remove 7 15 2024-04-14T15:18:13.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.974+0000 7f2d3f526700 10 client.49096.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:18:13.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.974+0000 7f2d3f526700 15 client.49096.objecter _session_command_op_assign -1 16 2024-04-14T15:18:13.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.974+0000 7f2d3f526700 20 client.49096.objecter _get_session s=0x7f2d1c027780 osd=7 3 2024-04-14T15:18:13.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.974+0000 7f2d3f526700 20 client.49096.objecter put_session s=0x7f2d1c027780 osd=7 3 2024-04-14T15:18:13.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.974+0000 7f2d3f526700 20 client.49096.objecter _get_session s=0x7f2d1c027780 osd=7 3 2024-04-14T15:18:13.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.974+0000 7f2d3f526700 15 client.49096.objecter _session_command_op_remove -1 16 2024-04-14T15:18:13.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.974+0000 7f2d3f526700 20 client.49096.objecter get_session s=0x7f2d1c027780 osd=7 3 2024-04-14T15:18:13.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.974+0000 7f2d3f526700 15 client.49096.objecter _session_command_op_assign 7 16 2024-04-14T15:18:13.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.974+0000 7f2d3f526700 20 client.49096.objecter put_session s=0x7f2d1c027780 osd=7 4 2024-04-14T15:18:13.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.974+0000 7f2d3f526700 10 client.49096.objecter _send_command 16 2024-04-14T15:18:13.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.974+0000 7f2d3f526700 1 -- 172.21.15.90:0/70627987 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f2d1c028a80 con 0x7f2d1c027ea0 2024-04-14T15:18:13.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.975+0000 7f2d2e7fc700 1 -- 172.21.15.90:0/70627987 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2d1c028a80 con 0x7f2d1c027ea0 2024-04-14T15:18:13.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.976+0000 7f2d2e7fc700 10 client.49096.objecter ms_dispatch 0x7f2d38001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:18:13.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.976+0000 7f2d2e7fc700 10 client.49096.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:18:13.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.976+0000 7f2d2e7fc700 20 client.49096.objecter put_session s=0x7f2d1c027780 osd=7 4 2024-04-14T15:18:13.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.976+0000 7f2d2e7fc700 15 client.49096.objecter _session_command_op_remove 7 16 2024-04-14T15:18:13.977 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:18:13.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.976+0000 7f2d3f526700 10 librados: watch_flush enter 2024-04-14T15:18:13.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.976+0000 7f2d3f526700 10 librados: watch_flush exit 2024-04-14T15:18:13.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.976+0000 7f2d3f526700 10 client.49096.objecter close_session for osd.0 2024-04-14T15:18:13.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.976+0000 7f2d3f526700 1 -- 172.21.15.90:0/70627987 >> v1:172.21.15.90:6801/3761806885 conn(0x7f2d1c004730 legacy=0x7f2d1c002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:13.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.976+0000 7f2d3f526700 20 client.49096.objecter put_session s=0x7f2d1c0053e0 osd=0 1 2024-04-14T15:18:13.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.976+0000 7f2d3f526700 10 client.49096.objecter close_session for osd.1 2024-04-14T15:18:13.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.976+0000 7f2d3f526700 1 -- 172.21.15.90:0/70627987 >> v1:172.21.15.90:6805/4170361699 conn(0x7f2d1c004310 legacy=0x7f2d1c008d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:13.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.976+0000 7f2d3f526700 20 client.49096.objecter put_session s=0x7f2d1c004130 osd=1 1 2024-04-14T15:18:13.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.976+0000 7f2d3f526700 10 client.49096.objecter close_session for osd.2 2024-04-14T15:18:13.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.976+0000 7f2d3f526700 1 -- 172.21.15.90:0/70627987 >> v1:172.21.15.90:6809/1686184251 conn(0x7f2d1c011a60 legacy=0x7f2d1c013e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:13.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.976+0000 7f2d3f526700 20 client.49096.objecter put_session s=0x7f2d1c008370 osd=2 1 2024-04-14T15:18:13.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.976+0000 7f2d3f526700 10 client.49096.objecter close_session for osd.3 2024-04-14T15:18:13.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.976+0000 7f2d3f526700 1 -- 172.21.15.90:0/70627987 >> v1:172.21.15.90:6813/2270166774 conn(0x7f2d1c0168a0 legacy=0x7f2d1c016ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:13.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.976+0000 7f2d3f526700 20 client.49096.objecter put_session s=0x7f2d1c0110b0 osd=3 1 2024-04-14T15:18:13.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.976+0000 7f2d3f526700 10 client.49096.objecter close_session for osd.4 2024-04-14T15:18:13.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.976+0000 7f2d3f526700 1 -- 172.21.15.90:0/70627987 >> v1:172.21.15.92:6800/3887549733 conn(0x7f2d1c01b0b0 legacy=0x7f2d1c01d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:13.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.976+0000 7f2d3f526700 20 client.49096.objecter put_session s=0x7f2d1c0173f0 osd=4 1 2024-04-14T15:18:13.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.976+0000 7f2d3f526700 10 client.49096.objecter close_session for osd.5 2024-04-14T15:18:13.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.976+0000 7f2d3f526700 1 -- 172.21.15.90:0/70627987 >> v1:172.21.15.92:6804/3244561440 conn(0x7f2d1c015ef0 legacy=0x7f2d1c021d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:13.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.976+0000 7f2d3dac3700 1 -- 172.21.15.90:0/70627987 reap_dead start 2024-04-14T15:18:13.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.976+0000 7f2d3f526700 20 client.49096.objecter put_session s=0x7f2d1c017600 osd=5 1 2024-04-14T15:18:13.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.976+0000 7f2d3f526700 10 client.49096.objecter close_session for osd.6 2024-04-14T15:18:13.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.976+0000 7f2d3f526700 1 -- 172.21.15.90:0/70627987 >> v1:172.21.15.92:6808/636125963 conn(0x7f2d1c023980 legacy=0x7f2d1c025d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:13.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.976+0000 7f2d3dac3700 1 -- 172.21.15.90:0/70627987 reap_dead start 2024-04-14T15:18:13.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.976+0000 7f2d3dac3700 1 -- 172.21.15.90:0/70627987 reap_dead start 2024-04-14T15:18:13.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.976+0000 7f2d3f526700 20 client.49096.objecter put_session s=0x7f2d1c016490 osd=6 1 2024-04-14T15:18:13.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.976+0000 7f2d3f526700 10 client.49096.objecter close_session for osd.7 2024-04-14T15:18:13.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.976+0000 7f2d3dac3700 1 -- 172.21.15.90:0/70627987 reap_dead start 2024-04-14T15:18:13.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.976+0000 7f2d3f526700 1 -- 172.21.15.90:0/70627987 >> v1:172.21.15.92:6812/1773706716 conn(0x7f2d1c027ea0 legacy=0x7f2d1c028300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:13.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.976+0000 7f2d3f526700 20 client.49096.objecter put_session s=0x7f2d1c027780 osd=7 1 2024-04-14T15:18:13.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.976+0000 7f2d3f526700 20 client.49096.objecter shutdown clearing up homeless session... 2024-04-14T15:18:13.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.976+0000 7f2d3f526700 10 client.49096.objecter successfully canceled tick 2024-04-14T15:18:13.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.976+0000 7f2d3f526700 1 -- 172.21.15.90:0/70627987 >> v1:172.21.15.90:6800/420935427 conn(0x7f2d24071140 legacy=0x7f2d24073600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:13.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.976+0000 7f2d3f526700 1 -- 172.21.15.90:0/70627987 >> v1:172.21.15.92:6789/0 conn(0x7f2d38107890 legacy=0x7f2d3806acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:13.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.976+0000 7f2d3f526700 1 -- 172.21.15.90:0/70627987 shutdown_connections 2024-04-14T15:18:13.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.976+0000 7f2d3f526700 1 -- 172.21.15.90:0/70627987 >> 172.21.15.90:0/70627987 conn(0x7f2d380fd290 msgr2=0x7f2d38111550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:18:13.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.976+0000 7f2d3f526700 1 -- 172.21.15.90:0/70627987 shutdown_connections 2024-04-14T15:18:13.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.976+0000 7f2d3f526700 1 -- 172.21.15.90:0/70627987 wait complete. 2024-04-14T15:18:13.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:13.976+0000 7f2d3f526700 1 librados: shutdown 2024-04-14T15:18:13.981 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:18:13.981 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:18:13.981 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:18:13.981 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:18:13.981 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:18:13.981 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:18:13.982 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:18:13.982 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:18:13.992 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:18:14.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:14.018 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:14.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:18:14.099 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:13 smithi092 ceph-mon[33966]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:18:14.099 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:13 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:18:14.100 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:13 smithi092 ceph-mon[33966]: from='client.49090 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-14T15:18:14.100 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:13 smithi092 ceph-mon[33966]: osdmap e419: 8 total, 8 up, 8 in 2024-04-14T15:18:14.100 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:13 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-14T15:18:14.100 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:13 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-14T15:18:14.100 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:13 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-14T15:18:14.100 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:13 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/70627987' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:18:14.119 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:18:14.142 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:14.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:14.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:18:14.191 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:18:13 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:18:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:18:14.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:13 smithi090 ceph-mon[30399]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:18:14.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:13 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:18:14.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:13 smithi090 ceph-mon[30399]: from='client.49090 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-14T15:18:14.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:13 smithi090 ceph-mon[30399]: osdmap e419: 8 total, 8 up, 8 in 2024-04-14T15:18:14.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:13 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-14T15:18:14.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:13 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-14T15:18:14.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:13 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-14T15:18:14.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:13 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/70627987' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:18:14.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:13 smithi090 ceph-mon[41670]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:18:14.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:13 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:18:14.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:13 smithi090 ceph-mon[41670]: from='client.49090 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-14T15:18:14.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:13 smithi090 ceph-mon[41670]: osdmap e419: 8 total, 8 up, 8 in 2024-04-14T15:18:14.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:13 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-14T15:18:14.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:13 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-14T15:18:14.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:13 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-14T15:18:14.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:13 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/70627987' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:18:14.244 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:18:14.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:14.269 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:14.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:18:14.370 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:18:14.393 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:14.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:14.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:18:14.494 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:18:14.519 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:14.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:14.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:18:14.621 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:18:14.645 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:14.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:14.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:18:14.747 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:14.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:14.773 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:14.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:14.874 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:18:14.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:14.899 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:14.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:18:15.001 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:18:15.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:15.025 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:15.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:18:15.126 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:15.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:15.151 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:15.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:15.171 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:18:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:18:15.171 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:14 smithi092 ceph-mon[33966]: pgmap v578: 238 pgs: 32 unknown, 206 active+clean; 8.4 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-04-14T15:18:15.171 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:14 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:18:15.171 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:14 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:18:15.172 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:14 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-14T15:18:15.172 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:14 smithi092 ceph-mon[33966]: osdmap e420: 8 total, 8 up, 8 in 2024-04-14T15:18:15.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:14 smithi090 ceph-mon[41670]: pgmap v578: 238 pgs: 32 unknown, 206 active+clean; 8.4 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-04-14T15:18:15.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:14 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:18:15.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:14 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:18:15.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:14 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-14T15:18:15.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:14 smithi090 ceph-mon[41670]: osdmap e420: 8 total, 8 up, 8 in 2024-04-14T15:18:15.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:14 smithi090 ceph-mon[30399]: pgmap v578: 238 pgs: 32 unknown, 206 active+clean; 8.4 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-04-14T15:18:15.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:14 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:18:15.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:14 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:18:15.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:14 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-14T15:18:15.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:14 smithi090 ceph-mon[30399]: osdmap e420: 8 total, 8 up, 8 in 2024-04-14T15:18:15.252 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:18:15.277 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:15.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:15.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:18:15.378 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:18:15.404 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:15.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:15.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:18:15.506 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:18:15.515 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-04-14T15:18:15.515 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:18:15.530 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:15.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:15.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:18:15.632 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:18:15.657 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:15.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:15.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:18:15.759 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:18:15.784 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:15.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:15.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:18:15.886 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:18:15.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:15.916 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:15.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:18:16.017 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:18:16.040 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:16.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:16.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:18:16.090 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:15 smithi090 ceph-mon[30399]: osdmap e421: 8 total, 8 up, 8 in 2024-04-14T15:18:16.090 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:15 smithi090 ceph-mon[30399]: osdmap e422: 8 total, 8 up, 8 in 2024-04-14T15:18:16.090 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:15 smithi090 ceph-mon[41670]: osdmap e421: 8 total, 8 up, 8 in 2024-04-14T15:18:16.090 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:15 smithi090 ceph-mon[41670]: osdmap e422: 8 total, 8 up, 8 in 2024-04-14T15:18:16.142 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:18:16.167 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:16.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:16.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:18:16.268 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:18:16.270 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:15 smithi092 ceph-mon[33966]: osdmap e421: 8 total, 8 up, 8 in 2024-04-14T15:18:16.270 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:15 smithi092 ceph-mon[33966]: osdmap e422: 8 total, 8 up, 8 in 2024-04-14T15:18:16.292 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:16.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:16.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:18:16.295 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-14T15:18:16.315 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd unset noscrub 2024-04-14T15:18:16.394 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:18:16.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:16.420 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:16.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:18:16.521 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:16.535 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:18:16.545 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:16.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:16.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:16.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.617+0000 7f26cb9cc700 1 -- 172.21.15.90:0/683750987 >> v1:172.21.15.90:6790/0 conn(0x7f26c410b750 legacy=0x7f26c410db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:16.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.617+0000 7f26cb9cc700 1 -- 172.21.15.90:0/683750987 shutdown_connections 2024-04-14T15:18:16.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.617+0000 7f26cb9cc700 1 -- 172.21.15.90:0/683750987 >> 172.21.15.90:0/683750987 conn(0x7f26c4076aa0 msgr2=0x7f26c4076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:18:16.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.617+0000 7f26cb9cc700 1 -- 172.21.15.90:0/683750987 shutdown_connections 2024-04-14T15:18:16.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.617+0000 7f26cb9cc700 1 -- 172.21.15.90:0/683750987 wait complete. 2024-04-14T15:18:16.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.617+0000 7f26cb9cc700 1 librados: starting msgr at 2024-04-14T15:18:16.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.617+0000 7f26cb9cc700 1 librados: starting objecter 2024-04-14T15:18:16.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.617+0000 7f26cb9cc700 1 Processor -- start 2024-04-14T15:18:16.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.617+0000 7f26cb9cc700 1 -- start start 2024-04-14T15:18:16.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.617+0000 7f26cb9cc700 1 librados: setting wanted keys 2024-04-14T15:18:16.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.617+0000 7f26cb9cc700 1 librados: calling monclient init 2024-04-14T15:18:16.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.618+0000 7f26cb9cc700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f26c41a9b10 con 0x7f26c410f320 2024-04-14T15:18:16.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.618+0000 7f26cb9cc700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f26c41aad30 con 0x7f26c4107880 2024-04-14T15:18:16.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.618+0000 7f26cb9cc700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f26c41abf70 con 0x7f26c410b750 2024-04-14T15:18:16.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.618+0000 7f26c9f69700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f26c410f320 0x7f26c41a82c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:36262/0 (socket says 172.21.15.90:36262) 2024-04-14T15:18:16.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.618+0000 7f26c9f69700 1 -- 172.21.15.90:0/166863032 learned_addr learned my addr 172.21.15.90:0/166863032 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:18:16.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.618+0000 7f26ba7fc700 10 client.?.objecter ms_handle_connect 0x7f26c410f320 2024-04-14T15:18:16.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.618+0000 7f26ba7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:18:16.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.618+0000 7f26ba7fc700 10 client.?.objecter ms_handle_connect 0x7f26c4107880 2024-04-14T15:18:16.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.618+0000 7f26ba7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:18:16.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.618+0000 7f26ba7fc700 10 client.?.objecter ms_handle_connect 0x7f26c410b750 2024-04-14T15:18:16.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.618+0000 7f26ba7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:18:16.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.618+0000 7f26ba7fc700 1 -- 172.21.15.90:0/166863032 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2373305822 0 0) 0x7f26c41aad30 con 0x7f26c4107880 2024-04-14T15:18:16.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.618+0000 7f26ba7fc700 1 -- 172.21.15.90:0/166863032 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f26b00039c0 con 0x7f26c4107880 2024-04-14T15:18:16.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.618+0000 7f26ba7fc700 1 -- 172.21.15.90:0/166863032 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2948003967 0 0) 0x7f26c41a9b10 con 0x7f26c410f320 2024-04-14T15:18:16.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.618+0000 7f26ba7fc700 1 -- 172.21.15.90:0/166863032 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f26c41aad30 con 0x7f26c410f320 2024-04-14T15:18:16.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.618+0000 7f26ba7fc700 1 -- 172.21.15.90:0/166863032 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2114465346 0 0) 0x7f26c41abf70 con 0x7f26c410b750 2024-04-14T15:18:16.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.618+0000 7f26ba7fc700 1 -- 172.21.15.90:0/166863032 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f26c41a9b10 con 0x7f26c410b750 2024-04-14T15:18:16.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.618+0000 7f26ba7fc700 1 -- 172.21.15.90:0/166863032 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2450528139 0 0) 0x7f26b00039c0 con 0x7f26c4107880 2024-04-14T15:18:16.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.618+0000 7f26ba7fc700 1 -- 172.21.15.90:0/166863032 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f26c41abf70 con 0x7f26c4107880 2024-04-14T15:18:16.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.618+0000 7f26ba7fc700 1 -- 172.21.15.90:0/166863032 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f26c00018b0 con 0x7f26c4107880 2024-04-14T15:18:16.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.618+0000 7f26ba7fc700 1 -- 172.21.15.90:0/166863032 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1026297473 0 0) 0x7f26c41aad30 con 0x7f26c410f320 2024-04-14T15:18:16.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.618+0000 7f26ba7fc700 1 -- 172.21.15.90:0/166863032 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f26b00039c0 con 0x7f26c410f320 2024-04-14T15:18:16.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.618+0000 7f26ba7fc700 1 -- 172.21.15.90:0/166863032 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f26bc001750 con 0x7f26c410f320 2024-04-14T15:18:16.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.618+0000 7f26ba7fc700 1 -- 172.21.15.90:0/166863032 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2130492113 0 0) 0x7f26c41a9b10 con 0x7f26c410b750 2024-04-14T15:18:16.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.618+0000 7f26ba7fc700 1 -- 172.21.15.90:0/166863032 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f26c41aad30 con 0x7f26c410b750 2024-04-14T15:18:16.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.618+0000 7f26ba7fc700 1 -- 172.21.15.90:0/166863032 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f26b4004560 con 0x7f26c410b750 2024-04-14T15:18:16.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.618+0000 7f26ba7fc700 1 -- 172.21.15.90:0/166863032 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4152676332 0 0) 0x7f26c41abf70 con 0x7f26c4107880 2024-04-14T15:18:16.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.618+0000 7f26ba7fc700 1 -- 172.21.15.90:0/166863032 >> v1:172.21.15.90:6790/0 conn(0x7f26c410b750 legacy=0x7f26c41a4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:16.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.618+0000 7f26ba7fc700 1 -- 172.21.15.90:0/166863032 >> v1:172.21.15.90:6789/0 conn(0x7f26c410f320 legacy=0x7f26c41a82c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:16.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.618+0000 7f26ba7fc700 1 -- 172.21.15.90:0/166863032 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26c41ad1b0 con 0x7f26c4107880 2024-04-14T15:18:16.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.619+0000 7f26cb9cc700 1 -- 172.21.15.90:0/166863032 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f26c41aaf60 con 0x7f26c4107880 2024-04-14T15:18:16.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.619+0000 7f26cb9cc700 10 client.49102.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:18:16.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.619+0000 7f26cb9cc700 1 -- 172.21.15.90:0/166863032 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f26c41ab4a0 con 0x7f26c4107880 2024-04-14T15:18:16.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.619+0000 7f26cb9cc700 1 librados: init done 2024-04-14T15:18:16.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.619+0000 7f26ba7fc700 1 -- 172.21.15.90:0/166863032 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f26c0003b40 con 0x7f26c4107880 2024-04-14T15:18:16.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.619+0000 7f26ba7fc700 1 -- 172.21.15.90:0/166863032 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f26c0004aa0 con 0x7f26c4107880 2024-04-14T15:18:16.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.619+0000 7f26cb9cc700 1 -- 172.21.15.90:0/166863032 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26a8005120 con 0x7f26c4107880 2024-04-14T15:18:16.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.620+0000 7f26ba7fc700 1 -- 172.21.15.90:0/166863032 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f26c00036a0 con 0x7f26c4107880 2024-04-14T15:18:16.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.621+0000 7f26ba7fc700 1 -- 172.21.15.90:0/166863032 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(422..422 src has 51..422) ==== 8160+0+0 (unknown 2691507270 0 0) 0x7f26c00897a0 con 0x7f26c4107880 2024-04-14T15:18:16.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.621+0000 7f26ba7fc700 10 client.49102.objecter ms_dispatch 0x7f26c4001550 osd_map(422..422 src has 51..422) 2024-04-14T15:18:16.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.621+0000 7f26ba7fc700 3 client.49102.objecter handle_osd_map got epochs [422,422] > 0 2024-04-14T15:18:16.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.621+0000 7f26ba7fc700 3 client.49102.objecter handle_osd_map decoding full epoch 422 2024-04-14T15:18:16.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.621+0000 7f26ba7fc700 20 client.49102.objecter dump_active .. 0 homeless 2024-04-14T15:18:16.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.621+0000 7f26ba7fc700 10 client.49102.objecter ms_handle_connect 0x7f26b0071470 2024-04-14T15:18:16.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.625+0000 7f26ba7fc700 1 -- 172.21.15.90:0/166863032 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f26c00561c0 con 0x7f26c4107880 2024-04-14T15:18:16.647 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:16.671 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:16.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:16.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:16.699 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:18:16.773 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:18:16.798 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:16.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:16.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:18:16.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.832+0000 7f26cb9cc700 1 -- 172.21.15.90:0/166863032 --> v1:172.21.15.90:6800/420935427 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f26a8000b90 con 0x7f26b0071470 2024-04-14T15:18:16.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.848+0000 7f26ba7fc700 1 -- 172.21.15.90:0/166863032 <== mgr.14610 v1:172.21.15.90:6800/420935427 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+613522 (unknown 2965378022 0 878185532) 0x7f26a8000b90 con 0x7f26b0071470 2024-04-14T15:18:16.856 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:18:16.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.852+0000 7f26cb9cc700 10 librados: watch_flush enter 2024-04-14T15:18:16.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.852+0000 7f26cb9cc700 10 librados: watch_flush exit 2024-04-14T15:18:16.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.852+0000 7f26cb9cc700 20 client.49102.objecter shutdown clearing up homeless session... 2024-04-14T15:18:16.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.852+0000 7f26cb9cc700 10 client.49102.objecter successfully canceled tick 2024-04-14T15:18:16.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.852+0000 7f26cb9cc700 1 -- 172.21.15.90:0/166863032 >> v1:172.21.15.90:6800/420935427 conn(0x7f26b0071470 legacy=0x7f26b0073930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:16.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.852+0000 7f26cb9cc700 1 -- 172.21.15.90:0/166863032 >> v1:172.21.15.92:6789/0 conn(0x7f26c4107880 legacy=0x7f26c4103be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:16.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.852+0000 7f26cb9cc700 1 -- 172.21.15.90:0/166863032 shutdown_connections 2024-04-14T15:18:16.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.852+0000 7f26cb9cc700 1 -- 172.21.15.90:0/166863032 >> 172.21.15.90:0/166863032 conn(0x7f26c4076aa0 msgr2=0x7f26c4112b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:18:16.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.852+0000 7f26cb9cc700 1 -- 172.21.15.90:0/166863032 shutdown_connections 2024-04-14T15:18:16.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.852+0000 7f26cb9cc700 1 -- 172.21.15.90:0/166863032 wait complete. 2024-04-14T15:18:16.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:16.852+0000 7f26cb9cc700 1 librados: shutdown 2024-04-14T15:18:16.863 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-14T15:18:16.900 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:18:16.925 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:16.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:16.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:18:16.962 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-14T15:18:17.026 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:18:17.050 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:17.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:17.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:18:17.152 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:18:17.176 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:17.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:17.177 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:18:17.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:16 smithi090 ceph-mon[30399]: pgmap v581: 238 pgs: 238 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-04-14T15:18:17.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:16 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:18:17.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:16 smithi090 ceph-mon[30399]: osdmap e423: 8 total, 8 up, 8 in 2024-04-14T15:18:17.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:16 smithi090 ceph-mon[41670]: pgmap v581: 238 pgs: 238 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-04-14T15:18:17.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:16 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:18:17.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:16 smithi090 ceph-mon[41670]: osdmap e423: 8 total, 8 up, 8 in 2024-04-14T15:18:17.278 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:18:17.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:17.303 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:17.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:18:17.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:16 smithi092 ceph-mon[33966]: pgmap v581: 238 pgs: 238 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-04-14T15:18:17.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:16 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:18:17.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:16 smithi092 ceph-mon[33966]: osdmap e423: 8 total, 8 up, 8 in 2024-04-14T15:18:17.404 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:18:17.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:17.428 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:17.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:18:17.529 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:18:17.553 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:17.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:17.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:18:17.655 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:18:17.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:17.680 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:17.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:18:17.694 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":582,"stamp":"2024-04-14T15:18:15.896175+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8792194,"num_objects":362,"num_object_clones":2,"num_object_copies":1086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":362,"num_whiteouts":0,"num_read":1340,"num_read_kb":991,"num_write":1042,"num_write_kb":12738,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":7,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3880,"ondisk_log_size":3880,"up":714,"acting":714,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":692,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":610120,"kb_used_data":82864,"kb_used_omap":12,"kb_used_meta":526963,"kb_avail":749351096,"statfs":{"total":767960285184,"available":767335522304,"internally_reserved":0,"allocated":84852736,"data_stored":58864680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":539610674},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":38,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":8,"num_read_kb":6,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.962690"},"pg_stats":[{"pgid":"297.1f","version":"0'0","reported_seq":13,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.956023+0000","last_change":"2024-04-14T15:18:13.005585+0000","last_active":"2024-04-14T15:18:13.956023+0000","last_peered":"2024-04-14T15:18:13.956023+0000","last_clean":"2024-04-14T15:18:13.956023+0000","last_became_active":"2024-04-14T15:18:13.005337+0000","last_became_peered":"2024-04-14T15:18:13.005337+0000","last_unstale":"2024-04-14T15:18:13.956023+0000","last_undegraded":"2024-04-14T15:18:13.956023+0000","last_fullsized":"2024-04-14T15:18:13.956023+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:21.146893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"297.1e","version":"0'0","reported_seq":10,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.006451+0000","last_change":"2024-04-14T15:18:13.005874+0000","last_active":"2024-04-14T15:18:13.006451+0000","last_peered":"2024-04-14T15:18:13.006451+0000","last_clean":"2024-04-14T15:18:13.006451+0000","last_became_active":"2024-04-14T15:18:13.005571+0000","last_became_peered":"2024-04-14T15:18:13.005571+0000","last_unstale":"2024-04-14T15:18:13.006451+0000","last_undegraded":"2024-04-14T15:18:13.006451+0000","last_fullsized":"2024-04-14T15:18:13.006451+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:26.683879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"297.1d","version":"0'0","reported_seq":13,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.957188+0000","last_change":"2024-04-14T15:18:12.947350+0000","last_active":"2024-04-14T15:18:13.957188+0000","last_peered":"2024-04-14T15:18:13.957188+0000","last_clean":"2024-04-14T15:18:13.957188+0000","last_became_active":"2024-04-14T15:18:12.947006+0000","last_became_peered":"2024-04-14T15:18:12.947006+0000","last_unstale":"2024-04-14T15:18:13.957188+0000","last_undegraded":"2024-04-14T15:18:13.957188+0000","last_fullsized":"2024-04-14T15:18:13.957188+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:27.508637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"297.1c","version":"420'2","reported_seq":16,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.975636+0000","last_change":"2024-04-14T15:18:12.955879+0000","last_active":"2024-04-14T15:18:13.975636+0000","last_peered":"2024-04-14T15:18:13.975636+0000","last_clean":"2024-04-14T15:18:13.975636+0000","last_became_active":"2024-04-14T15:18:12.955448+0000","last_became_peered":"2024-04-14T15:18:12.955448+0000","last_unstale":"2024-04-14T15:18:13.975636+0000","last_undegraded":"2024-04-14T15:18:13.975636+0000","last_fullsized":"2024-04-14T15:18:13.975636+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:23.726057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"297.1b","version":"0'0","reported_seq":13,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.954030+0000","last_change":"2024-04-14T15:18:12.951404+0000","last_active":"2024-04-14T15:18:13.954030+0000","last_peered":"2024-04-14T15:18:13.954030+0000","last_clean":"2024-04-14T15:18:13.954030+0000","last_became_active":"2024-04-14T15:18:12.951112+0000","last_became_peered":"2024-04-14T15:18:12.951112+0000","last_unstale":"2024-04-14T15:18:13.954030+0000","last_undegraded":"2024-04-14T15:18:13.954030+0000","last_fullsized":"2024-04-14T15:18:13.954030+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:14.892479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"297.1a","version":"0'0","reported_seq":13,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.952502+0000","last_change":"2024-04-14T15:18:12.956068+0000","last_active":"2024-04-14T15:18:13.952502+0000","last_peered":"2024-04-14T15:18:13.952502+0000","last_clean":"2024-04-14T15:18:13.952502+0000","last_became_active":"2024-04-14T15:18:12.955801+0000","last_became_peered":"2024-04-14T15:18:12.955801+0000","last_unstale":"2024-04-14T15:18:13.952502+0000","last_undegraded":"2024-04-14T15:18:13.952502+0000","last_fullsized":"2024-04-14T15:18:13.952502+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:15.388092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"297.19","version":"0'0","reported_seq":10,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-14T15:18:12.956306+0000","last_change":"2024-04-14T15:18:12.955167+0000","last_active":"2024-04-14T15:18:12.956306+0000","last_peered":"2024-04-14T15:18:12.956306+0000","last_clean":"2024-04-14T15:18:12.956306+0000","last_became_active":"2024-04-14T15:18:12.954867+0000","last_became_peered":"2024-04-14T15:18:12.954867+0000","last_unstale":"2024-04-14T15:18:12.956306+0000","last_undegraded":"2024-04-14T15:18:12.956306+0000","last_fullsized":"2024-04-14T15:18:12.956306+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:24.120921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"297.18","version":"0'0","reported_seq":10,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-14T15:18:12.950808+0000","last_change":"2024-04-14T15:18:12.950510+0000","last_active":"2024-04-14T15:18:12.950808+0000","last_peered":"2024-04-14T15:18:12.950808+0000","last_clean":"2024-04-14T15:18:12.950808+0000","last_became_active":"2024-04-14T15:18:12.950312+0000","last_became_peered":"2024-04-14T15:18:12.950312+0000","last_unstale":"2024-04-14T15:18:12.950808+0000","last_undegraded":"2024-04-14T15:18:12.950808+0000","last_fullsized":"2024-04-14T15:18:12.950808+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:24.972443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"297.17","version":"0'0","reported_seq":10,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.006998+0000","last_change":"2024-04-14T15:18:13.006758+0000","last_active":"2024-04-14T15:18:13.006998+0000","last_peered":"2024-04-14T15:18:13.006998+0000","last_clean":"2024-04-14T15:18:13.006998+0000","last_became_active":"2024-04-14T15:18:13.006469+0000","last_became_peered":"2024-04-14T15:18:13.006469+0000","last_unstale":"2024-04-14T15:18:13.006998+0000","last_undegraded":"2024-04-14T15:18:13.006998+0000","last_fullsized":"2024-04-14T15:18:13.006998+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:12.963449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"297.16","version":"0'0","reported_seq":13,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.957902+0000","last_change":"2024-04-14T15:18:12.951467+0000","last_active":"2024-04-14T15:18:13.957902+0000","last_peered":"2024-04-14T15:18:13.957902+0000","last_clean":"2024-04-14T15:18:13.957902+0000","last_became_active":"2024-04-14T15:18:12.950999+0000","last_became_peered":"2024-04-14T15:18:12.950999+0000","last_unstale":"2024-04-14T15:18:13.957902+0000","last_undegraded":"2024-04-14T15:18:13.957902+0000","last_fullsized":"2024-04-14T15:18:13.957902+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:15.500639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"297.15","version":"0'0","reported_seq":13,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.963955+0000","last_change":"2024-04-14T15:18:13.004242+0000","last_active":"2024-04-14T15:18:13.963955+0000","last_peered":"2024-04-14T15:18:13.963955+0000","last_clean":"2024-04-14T15:18:13.963955+0000","last_became_active":"2024-04-14T15:18:13.003888+0000","last_became_peered":"2024-04-14T15:18:13.003888+0000","last_unstale":"2024-04-14T15:18:13.963955+0000","last_undegraded":"2024-04-14T15:18:13.963955+0000","last_fullsized":"2024-04-14T15:18:13.963955+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:29.490276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"297.14","version":"0'0","reported_seq":13,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.952525+0000","last_change":"2024-04-14T15:18:13.005434+0000","last_active":"2024-04-14T15:18:13.952525+0000","last_peered":"2024-04-14T15:18:13.952525+0000","last_clean":"2024-04-14T15:18:13.952525+0000","last_became_active":"2024-04-14T15:18:13.005051+0000","last_became_peered":"2024-04-14T15:18:13.005051+0000","last_unstale":"2024-04-14T15:18:13.952525+0000","last_undegraded":"2024-04-14T15:18:13.952525+0000","last_fullsized":"2024-04-14T15:18:13.952525+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:14.539808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"297.13","version":"0'0","reported_seq":13,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.959750+0000","last_change":"2024-04-14T15:18:13.004437+0000","last_active":"2024-04-14T15:18:13.959750+0000","last_peered":"2024-04-14T15:18:13.959750+0000","last_clean":"2024-04-14T15:18:13.959750+0000","last_became_active":"2024-04-14T15:18:13.004133+0000","last_became_peered":"2024-04-14T15:18:13.004133+0000","last_unstale":"2024-04-14T15:18:13.959750+0000","last_undegraded":"2024-04-14T15:18:13.959750+0000","last_fullsized":"2024-04-14T15:18:13.959750+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:32.102952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"297.12","version":"0'0","reported_seq":13,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.963162+0000","last_change":"2024-04-14T15:18:12.956427+0000","last_active":"2024-04-14T15:18:13.963162+0000","last_peered":"2024-04-14T15:18:13.963162+0000","last_clean":"2024-04-14T15:18:13.963162+0000","last_became_active":"2024-04-14T15:18:12.955274+0000","last_became_peered":"2024-04-14T15:18:12.955274+0000","last_unstale":"2024-04-14T15:18:13.963162+0000","last_undegraded":"2024-04-14T15:18:13.963162+0000","last_fullsized":"2024-04-14T15:18:13.963162+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:20.629003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"297.11","version":"0'0","reported_seq":13,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.954668+0000","last_change":"2024-04-14T15:18:12.956974+0000","last_active":"2024-04-14T15:18:13.954668+0000","last_peered":"2024-04-14T15:18:13.954668+0000","last_clean":"2024-04-14T15:18:13.954668+0000","last_became_active":"2024-04-14T15:18:12.955819+0000","last_became_peered":"2024-04-14T15:18:12.955819+0000","last_unstale":"2024-04-14T15:18:13.954668+0000","last_undegraded":"2024-04-14T15:18:13.954668+0000","last_fullsized":"2024-04-14T15:18:13.954668+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:23.104281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"297.10","version":"0'0","reported_seq":13,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.953408+0000","last_change":"2024-04-14T15:18:13.006273+0000","last_active":"2024-04-14T15:18:13.953408+0000","last_peered":"2024-04-14T15:18:13.953408+0000","last_clean":"2024-04-14T15:18:13.953408+0000","last_became_active":"2024-04-14T15:18:13.005968+0000","last_became_peered":"2024-04-14T15:18:13.005968+0000","last_unstale":"2024-04-14T15:18:13.953408+0000","last_undegraded":"2024-04-14T15:18:13.953408+0000","last_fullsized":"2024-04-14T15:18:13.953408+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:31.460743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"297.f","version":"0'0","reported_seq":13,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.956975+0000","last_change":"2024-04-14T15:18:13.005801+0000","last_active":"2024-04-14T15:18:13.956975+0000","last_peered":"2024-04-14T15:18:13.956975+0000","last_clean":"2024-04-14T15:18:13.956975+0000","last_became_active":"2024-04-14T15:18:13.005146+0000","last_became_peered":"2024-04-14T15:18:13.005146+0000","last_unstale":"2024-04-14T15:18:13.956975+0000","last_undegraded":"2024-04-14T15:18:13.956975+0000","last_fullsized":"2024-04-14T15:18:13.956975+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:34.836627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"297.e","version":"420'1","reported_seq":14,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.965568+0000","last_change":"2024-04-14T15:18:12.948109+0000","last_active":"2024-04-14T15:18:13.965568+0000","last_peered":"2024-04-14T15:18:13.965568+0000","last_clean":"2024-04-14T15:18:13.965568+0000","last_became_active":"2024-04-14T15:18:12.947811+0000","last_became_peered":"2024-04-14T15:18:12.947811+0000","last_unstale":"2024-04-14T15:18:13.965568+0000","last_undegraded":"2024-04-14T15:18:13.965568+0000","last_fullsized":"2024-04-14T15:18:13.965568+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:14.922501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"297.d","version":"0'0","reported_seq":10,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-14T15:18:12.949564+0000","last_change":"2024-04-14T15:18:12.949318+0000","last_active":"2024-04-14T15:18:12.949564+0000","last_peered":"2024-04-14T15:18:12.949564+0000","last_clean":"2024-04-14T15:18:12.949564+0000","last_became_active":"2024-04-14T15:18:12.949034+0000","last_became_peered":"2024-04-14T15:18:12.949034+0000","last_unstale":"2024-04-14T15:18:12.949564+0000","last_undegraded":"2024-04-14T15:18:12.949564+0000","last_fullsized":"2024-04-14T15:18:12.949564+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:34.173254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"297.c","version":"0'0","reported_seq":13,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.963965+0000","last_change":"2024-04-14T15:18:13.008426+0000","last_active":"2024-04-14T15:18:13.963965+0000","last_peered":"2024-04-14T15:18:13.963965+0000","last_clean":"2024-04-14T15:18:13.963965+0000","last_became_active":"2024-04-14T15:18:13.007723+0000","last_became_peered":"2024-04-14T15:18:13.007723+0000","last_unstale":"2024-04-14T15:18:13.963965+0000","last_undegraded":"2024-04-14T15:18:13.963965+0000","last_fullsized":"2024-04-14T15:18:13.963965+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:36.383377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"297.b","version":"0'0","reported_seq":10,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-14T15:18:12.951322+0000","last_change":"2024-04-14T15:18:12.951076+0000","last_active":"2024-04-14T15:18:12.951322+0000","last_peered":"2024-04-14T15:18:12.951322+0000","last_clean":"2024-04-14T15:18:12.951322+0000","last_became_active":"2024-04-14T15:18:12.950806+0000","last_became_peered":"2024-04-14T15:18:12.950806+0000","last_unstale":"2024-04-14T15:18:12.951322+0000","last_undegraded":"2024-04-14T15:18:12.951322+0000","last_fullsized":"2024-04-14T15:18:12.951322+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:13.683257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"297.a","version":"0'0","reported_seq":13,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.952533+0000","last_change":"2024-04-14T15:18:12.955873+0000","last_active":"2024-04-14T15:18:13.952533+0000","last_peered":"2024-04-14T15:18:13.952533+0000","last_clean":"2024-04-14T15:18:13.952533+0000","last_became_active":"2024-04-14T15:18:12.955571+0000","last_became_peered":"2024-04-14T15:18:12.955571+0000","last_unstale":"2024-04-14T15:18:13.952533+0000","last_undegraded":"2024-04-14T15:18:13.952533+0000","last_fullsized":"2024-04-14T15:18:13.952533+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:12.173921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"297.9","version":"0'0","reported_seq":13,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.958763+0000","last_change":"2024-04-14T15:18:13.004346+0000","last_active":"2024-04-14T15:18:13.958763+0000","last_peered":"2024-04-14T15:18:13.958763+0000","last_clean":"2024-04-14T15:18:13.958763+0000","last_became_active":"2024-04-14T15:18:13.004023+0000","last_became_peered":"2024-04-14T15:18:13.004023+0000","last_unstale":"2024-04-14T15:18:13.958763+0000","last_undegraded":"2024-04-14T15:18:13.958763+0000","last_fullsized":"2024-04-14T15:18:13.958763+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:14.413671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"297.8","version":"0'0","reported_seq":13,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.953627+0000","last_change":"2024-04-14T15:18:13.005840+0000","last_active":"2024-04-14T15:18:13.953627+0000","last_peered":"2024-04-14T15:18:13.953627+0000","last_clean":"2024-04-14T15:18:13.953627+0000","last_became_active":"2024-04-14T15:18:13.005478+0000","last_became_peered":"2024-04-14T15:18:13.005478+0000","last_unstale":"2024-04-14T15:18:13.953627+0000","last_undegraded":"2024-04-14T15:18:13.953627+0000","last_fullsized":"2024-04-14T15:18:13.953627+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:29.674300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"297.7","version":"420'2","reported_seq":15,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.974307+0000","last_change":"2024-04-14T15:18:12.955307+0000","last_active":"2024-04-14T15:18:13.974307+0000","last_peered":"2024-04-14T15:18:13.974307+0000","last_clean":"2024-04-14T15:18:13.974307+0000","last_became_active":"2024-04-14T15:18:12.955029+0000","last_became_peered":"2024-04-14T15:18:12.955029+0000","last_unstale":"2024-04-14T15:18:13.974307+0000","last_undegraded":"2024-04-14T15:18:13.974307+0000","last_fullsized":"2024-04-14T15:18:13.974307+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:18.760118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"297.6","version":"0'0","reported_seq":13,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.956066+0000","last_change":"2024-04-14T15:18:13.004180+0000","last_active":"2024-04-14T15:18:13.956066+0000","last_peered":"2024-04-14T15:18:13.956066+0000","last_clean":"2024-04-14T15:18:13.956066+0000","last_became_active":"2024-04-14T15:18:13.003841+0000","last_became_peered":"2024-04-14T15:18:13.003841+0000","last_unstale":"2024-04-14T15:18:13.956066+0000","last_undegraded":"2024-04-14T15:18:13.956066+0000","last_fullsized":"2024-04-14T15:18:13.956066+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:28.810731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"297.5","version":"0'0","reported_seq":10,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-14T15:18:12.956340+0000","last_change":"2024-04-14T15:18:12.955042+0000","last_active":"2024-04-14T15:18:12.956340+0000","last_peered":"2024-04-14T15:18:12.956340+0000","last_clean":"2024-04-14T15:18:12.956340+0000","last_became_active":"2024-04-14T15:18:12.954742+0000","last_became_peered":"2024-04-14T15:18:12.954742+0000","last_unstale":"2024-04-14T15:18:12.956340+0000","last_undegraded":"2024-04-14T15:18:12.956340+0000","last_fullsized":"2024-04-14T15:18:12.956340+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:30.673521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"297.4","version":"0'0","reported_seq":13,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.952674+0000","last_change":"2024-04-14T15:18:12.955231+0000","last_active":"2024-04-14T15:18:13.952674+0000","last_peered":"2024-04-14T15:18:13.952674+0000","last_clean":"2024-04-14T15:18:13.952674+0000","last_became_active":"2024-04-14T15:18:12.954904+0000","last_became_peered":"2024-04-14T15:18:12.954904+0000","last_unstale":"2024-04-14T15:18:13.952674+0000","last_undegraded":"2024-04-14T15:18:13.952674+0000","last_fullsized":"2024-04-14T15:18:13.952674+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:38.070409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"297.3","version":"0'0","reported_seq":13,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.955605+0000","last_change":"2024-04-14T15:18:13.009275+0000","last_active":"2024-04-14T15:18:13.955605+0000","last_peered":"2024-04-14T15:18:13.955605+0000","last_clean":"2024-04-14T15:18:13.955605+0000","last_became_active":"2024-04-14T15:18:13.008980+0000","last_became_peered":"2024-04-14T15:18:13.008980+0000","last_unstale":"2024-04-14T15:18:13.955605+0000","last_undegraded":"2024-04-14T15:18:13.955605+0000","last_fullsized":"2024-04-14T15:18:13.955605+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:15.759498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"297.2","version":"0'0","reported_seq":10,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-14T15:18:12.948682+0000","last_change":"2024-04-14T15:18:12.948399+0000","last_active":"2024-04-14T15:18:12.948682+0000","last_peered":"2024-04-14T15:18:12.948682+0000","last_clean":"2024-04-14T15:18:12.948682+0000","last_became_active":"2024-04-14T15:18:12.948080+0000","last_became_peered":"2024-04-14T15:18:12.948080+0000","last_unstale":"2024-04-14T15:18:12.948682+0000","last_undegraded":"2024-04-14T15:18:12.948682+0000","last_fullsized":"2024-04-14T15:18:12.948682+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:32.057790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"297.1","version":"0'0","reported_seq":13,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.952137+0000","last_change":"2024-04-14T15:18:13.006295+0000","last_active":"2024-04-14T15:18:13.952137+0000","last_peered":"2024-04-14T15:18:13.952137+0000","last_clean":"2024-04-14T15:18:13.952137+0000","last_became_active":"2024-04-14T15:18:13.006000+0000","last_became_peered":"2024-04-14T15:18:13.006000+0000","last_unstale":"2024-04-14T15:18:13.952137+0000","last_undegraded":"2024-04-14T15:18:13.952137+0000","last_fullsized":"2024-04-14T15:18:13.952137+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:17.146699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"297.0","version":"420'1","reported_seq":14,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.969375+0000","last_change":"2024-04-14T15:18:13.005460+0000","last_active":"2024-04-14T15:18:13.969375+0000","last_peered":"2024-04-14T15:18:13.969375+0000","last_clean":"2024-04-14T15:18:13.969375+0000","last_became_active":"2024-04-14T15:18:13.005082+0000","last_became_peered":"2024-04-14T15:18:13.005082+0000","last_unstale":"2024-04-14T15:18:13.969375+0000","last_undegraded":"2024-04-14T15:18:13.969375+0000","last_fullsized":"2024-04-14T15:18:13.969375+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:32.045130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"110.11","version":"403'41","reported_seq":818,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.963440+0000","last_change":"2024-04-14T15:17:56.911132+0000","last_active":"2024-04-14T15:18:13.963440+0000","last_peered":"2024-04-14T15:18:13.963440+0000","last_clean":"2024-04-14T15:18:13.963440+0000","last_became_active":"2024-04-14T15:13:09.936619+0000","last_became_peered":"2024-04-14T15:13:09.936619+0000","last_unstale":"2024-04-14T15:18:13.963440+0000","last_undegraded":"2024-04-14T15:18:13.963440+0000","last_fullsized":"2024-04-14T15:18:13.963440+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"403'41","last_scrub_stamp":"2024-04-14T15:17:56.911004+0000","last_deep_scrub":"403'41","last_deep_scrub_stamp":"2024-04-14T15:17:56.911004+0000","last_clean_scrub_stamp":"2024-04-14T15:17:56.911004+0000","objects_scrubbed":2,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:10.638978+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0.0016290829999999999,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"110.10","version":"253'158","reported_seq":900,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.952033+0000","last_change":"2024-04-14T15:17:34.291974+0000","last_active":"2024-04-14T15:18:13.952033+0000","last_peered":"2024-04-14T15:18:13.952033+0000","last_clean":"2024-04-14T15:18:13.952033+0000","last_became_active":"2024-04-14T15:13:09.928915+0000","last_became_peered":"2024-04-14T15:13:09.928915+0000","last_unstale":"2024-04-14T15:18:13.952033+0000","last_undegraded":"2024-04-14T15:18:13.952033+0000","last_fullsized":"2024-04-14T15:18:13.952033+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"253'158","last_scrub_stamp":"2024-04-14T15:17:34.291854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:34.291854+0000","objects_scrubbed":10,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:40.197864+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0018074390000000001,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":47,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"110.13","version":"409'41","reported_seq":819,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.959637+0000","last_change":"2024-04-14T15:17:44.873857+0000","last_active":"2024-04-14T15:18:13.959637+0000","last_peered":"2024-04-14T15:18:13.959637+0000","last_clean":"2024-04-14T15:18:13.959637+0000","last_became_active":"2024-04-14T15:13:09.918152+0000","last_became_peered":"2024-04-14T15:13:09.918152+0000","last_unstale":"2024-04-14T15:18:13.959637+0000","last_undegraded":"2024-04-14T15:18:13.959637+0000","last_fullsized":"2024-04-14T15:18:13.959637+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'39","last_scrub_stamp":"2024-04-14T15:17:44.873766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:44.873766+0000","objects_scrubbed":3,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:56.207896+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0047971690000000004,"stat_sum":{"num_bytes":27,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":14,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"110.12","version":"409'70","reported_seq":845,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.959150+0000","last_change":"2024-04-14T15:16:53.674009+0000","last_active":"2024-04-14T15:18:13.959150+0000","last_peered":"2024-04-14T15:18:13.959150+0000","last_clean":"2024-04-14T15:18:13.959150+0000","last_became_active":"2024-04-14T15:13:09.939566+0000","last_became_peered":"2024-04-14T15:13:09.939566+0000","last_unstale":"2024-04-14T15:18:13.959150+0000","last_undegraded":"2024-04-14T15:18:13.959150+0000","last_fullsized":"2024-04-14T15:18:13.959150+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"253'66","last_scrub_stamp":"2024-04-14T15:16:53.673906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:16:53.673906+0000","objects_scrubbed":6,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.003844564,"stat_sum":{"num_bytes":74,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":24,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"110.15","version":"190'34","reported_seq":806,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.957311+0000","last_change":"2024-04-14T15:16:06.494156+0000","last_active":"2024-04-14T15:18:13.957311+0000","last_peered":"2024-04-14T15:18:13.957311+0000","last_clean":"2024-04-14T15:18:13.957311+0000","last_became_active":"2024-04-14T15:13:09.960121+0000","last_became_peered":"2024-04-14T15:13:09.960121+0000","last_unstale":"2024-04-14T15:18:13.957311+0000","last_undegraded":"2024-04-14T15:18:13.957311+0000","last_fullsized":"2024-04-14T15:18:13.957311+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'34","last_scrub_stamp":"2024-04-14T15:16:06.494065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:16:06.494065+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0020885510000000001,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"110.14","version":"355'87","reported_seq":852,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.955393+0000","last_change":"2024-04-14T15:16:03.883165+0000","last_active":"2024-04-14T15:18:13.955393+0000","last_peered":"2024-04-14T15:18:13.955393+0000","last_clean":"2024-04-14T15:18:13.955393+0000","last_became_active":"2024-04-14T15:13:09.899561+0000","last_became_peered":"2024-04-14T15:13:09.899561+0000","last_unstale":"2024-04-14T15:18:13.955393+0000","last_undegraded":"2024-04-14T15:18:13.955393+0000","last_fullsized":"2024-04-14T15:18:13.955393+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'85","last_scrub_stamp":"2024-04-14T15:15:42.409029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:15:42.409029+0000","objects_scrubbed":5,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0039978469999999997,"stat_sum":{"num_bytes":71,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"110.17","version":"234'6","reported_seq":785,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.956149+0000","last_change":"2024-04-14T15:16:07.459851+0000","last_active":"2024-04-14T15:18:13.956149+0000","last_peered":"2024-04-14T15:18:13.956149+0000","last_clean":"2024-04-14T15:18:13.956149+0000","last_became_active":"2024-04-14T15:13:09.898521+0000","last_became_peered":"2024-04-14T15:13:09.898521+0000","last_unstale":"2024-04-14T15:18:13.956149+0000","last_undegraded":"2024-04-14T15:18:13.956149+0000","last_fullsized":"2024-04-14T15:18:13.956149+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'6","last_scrub_stamp":"2024-04-14T15:16:07.459718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:16:07.459718+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0019949540000000002,"stat_sum":{"num_bytes":13,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"110.16","version":"355'35","reported_seq":646,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.963504+0000","last_change":"2024-04-14T15:17:34.097658+0000","last_active":"2024-04-14T15:18:13.963504+0000","last_peered":"2024-04-14T15:18:13.963504+0000","last_clean":"2024-04-14T15:18:13.963504+0000","last_became_active":"2024-04-14T15:15:21.889783+0000","last_became_peered":"2024-04-14T15:15:21.889783+0000","last_unstale":"2024-04-14T15:18:13.963504+0000","last_undegraded":"2024-04-14T15:18:13.963504+0000","last_fullsized":"2024-04-14T15:18:13.963504+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'35","last_scrub_stamp":"2024-04-14T15:17:34.097563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:34.097563+0000","objects_scrubbed":3,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:46.257693+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.002398131,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"110.19","version":"237'85","reported_seq":855,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.953646+0000","last_change":"2024-04-14T15:16:04.177194+0000","last_active":"2024-04-14T15:18:13.953646+0000","last_peered":"2024-04-14T15:18:13.953646+0000","last_clean":"2024-04-14T15:18:13.953646+0000","last_became_active":"2024-04-14T15:13:09.998139+0000","last_became_peered":"2024-04-14T15:13:09.998139+0000","last_unstale":"2024-04-14T15:18:13.953646+0000","last_undegraded":"2024-04-14T15:18:13.953646+0000","last_fullsized":"2024-04-14T15:18:13.953646+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'85","last_scrub_stamp":"2024-04-14T15:16:04.177097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:16:04.177097+0000","objects_scrubbed":5,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0060025429999999999,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"110.18","version":"387'94","reported_seq":865,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.953515+0000","last_change":"2024-04-14T15:17:36.005686+0000","last_active":"2024-04-14T15:18:13.953515+0000","last_peered":"2024-04-14T15:18:13.953515+0000","last_clean":"2024-04-14T15:18:13.953515+0000","last_became_active":"2024-04-14T15:13:09.998803+0000","last_became_peered":"2024-04-14T15:13:09.998803+0000","last_unstale":"2024-04-14T15:18:13.953515+0000","last_undegraded":"2024-04-14T15:18:13.953515+0000","last_fullsized":"2024-04-14T15:18:13.953515+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'94","last_scrub_stamp":"2024-04-14T15:17:36.005598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:36.005598+0000","objects_scrubbed":8,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:37.953367+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.024348310000000001,"stat_sum":{"num_bytes":65,"num_objects":8,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":29,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"110.1b","version":"355'91","reported_seq":861,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.959364+0000","last_change":"2024-04-14T15:17:47.889492+0000","last_active":"2024-04-14T15:18:13.959364+0000","last_peered":"2024-04-14T15:18:13.959364+0000","last_clean":"2024-04-14T15:18:13.959364+0000","last_became_active":"2024-04-14T15:13:09.946919+0000","last_became_peered":"2024-04-14T15:13:09.946919+0000","last_unstale":"2024-04-14T15:18:13.959364+0000","last_undegraded":"2024-04-14T15:18:13.959364+0000","last_fullsized":"2024-04-14T15:18:13.959364+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'91","last_scrub_stamp":"2024-04-14T15:17:47.889364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:47.889364+0000","objects_scrubbed":6,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:01.463050+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0037919410000000001,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":28,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"110.1a","version":"387'39","reported_seq":822,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.963742+0000","last_change":"2024-04-14T15:17:53.871688+0000","last_active":"2024-04-14T15:18:13.963742+0000","last_peered":"2024-04-14T15:18:13.963742+0000","last_clean":"2024-04-14T15:18:13.963742+0000","last_became_active":"2024-04-14T15:13:09.960319+0000","last_became_peered":"2024-04-14T15:13:09.960319+0000","last_unstale":"2024-04-14T15:18:13.963742+0000","last_undegraded":"2024-04-14T15:18:13.963742+0000","last_fullsized":"2024-04-14T15:18:13.963742+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'39","last_scrub_stamp":"2024-04-14T15:17:53.871590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:53.871590+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:12.316799+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0017807330000000001,"stat_sum":{"num_bytes":62,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"110.1d","version":"225'85","reported_seq":854,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-14T15:18:12.946812+0000","last_change":"2024-04-14T15:17:36.419193+0000","last_active":"2024-04-14T15:18:12.946812+0000","last_peered":"2024-04-14T15:18:12.946812+0000","last_clean":"2024-04-14T15:18:12.946812+0000","last_became_active":"2024-04-14T15:13:09.868082+0000","last_became_peered":"2024-04-14T15:13:09.868082+0000","last_unstale":"2024-04-14T15:18:12.946812+0000","last_undegraded":"2024-04-14T15:18:12.946812+0000","last_fullsized":"2024-04-14T15:18:12.946812+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"225'85","last_scrub_stamp":"2024-04-14T15:17:36.419088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:36.419088+0000","objects_scrubbed":5,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:52.327307+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00141866,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"110.1c","version":"420'77","reported_seq":842,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.972849+0000","last_change":"2024-04-14T15:16:04.615669+0000","last_active":"2024-04-14T15:18:13.972849+0000","last_peered":"2024-04-14T15:18:13.972849+0000","last_clean":"2024-04-14T15:18:13.972849+0000","last_became_active":"2024-04-14T15:13:09.954091+0000","last_became_peered":"2024-04-14T15:13:09.954091+0000","last_unstale":"2024-04-14T15:18:13.972849+0000","last_undegraded":"2024-04-14T15:18:13.972849+0000","last_fullsized":"2024-04-14T15:18:13.972849+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'73","last_scrub_stamp":"2024-04-14T15:16:04.615572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:16:04.615572+0000","objects_scrubbed":5,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0056190420000000003,"stat_sum":{"num_bytes":65,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":24,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"110.1f","version":"268'25","reported_seq":851,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.957010+0000","last_change":"2024-04-14T15:16:08.237015+0000","last_active":"2024-04-14T15:18:13.957010+0000","last_peered":"2024-04-14T15:18:13.957010+0000","last_clean":"2024-04-14T15:18:13.957010+0000","last_became_active":"2024-04-14T15:13:09.999629+0000","last_became_peered":"2024-04-14T15:13:09.999629+0000","last_unstale":"2024-04-14T15:18:13.957010+0000","last_undegraded":"2024-04-14T15:18:13.957010+0000","last_fullsized":"2024-04-14T15:18:13.957010+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'25","last_scrub_stamp":"2024-04-14T15:16:08.236881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:16:08.236881+0000","objects_scrubbed":2,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0061123380000000001,"stat_sum":{"num_bytes":4194315,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":8,"num_write_kb":8198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"110.1e","version":"355'23","reported_seq":799,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.958018+0000","last_change":"2024-04-14T15:16:10.428089+0000","last_active":"2024-04-14T15:18:13.958018+0000","last_peered":"2024-04-14T15:18:13.958018+0000","last_clean":"2024-04-14T15:18:13.958018+0000","last_became_active":"2024-04-14T15:13:09.955932+0000","last_became_peered":"2024-04-14T15:13:09.955932+0000","last_unstale":"2024-04-14T15:18:13.958018+0000","last_undegraded":"2024-04-14T15:18:13.958018+0000","last_fullsized":"2024-04-14T15:18:13.958018+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'22","last_scrub_stamp":"2024-04-14T15:16:10.427994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:16:10.427994+0000","objects_scrubbed":1,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.003436284,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":8,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"110.1","version":"406'32","reported_seq":807,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-14T15:18:12.942948+0000","last_change":"2024-04-14T15:16:03.895140+0000","last_active":"2024-04-14T15:18:12.942948+0000","last_peered":"2024-04-14T15:18:12.942948+0000","last_clean":"2024-04-14T15:18:12.942948+0000","last_became_active":"2024-04-14T15:13:09.929562+0000","last_became_peered":"2024-04-14T15:13:09.929562+0000","last_unstale":"2024-04-14T15:18:12.942948+0000","last_undegraded":"2024-04-14T15:18:12.942948+0000","last_fullsized":"2024-04-14T15:18:12.942948+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"219'24","last_scrub_stamp":"2024-04-14T15:15:35.591006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:15:35.591006+0000","objects_scrubbed":3,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:15.661426+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00097444399999999998,"stat_sum":{"num_bytes":4000050,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":15,"num_write_kb":3921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"110.0","version":"219'85","reported_seq":851,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.952216+0000","last_change":"2024-04-14T15:17:51.438249+0000","last_active":"2024-04-14T15:18:13.952216+0000","last_peered":"2024-04-14T15:18:13.952216+0000","last_clean":"2024-04-14T15:18:13.952216+0000","last_became_active":"2024-04-14T15:13:09.856407+0000","last_became_peered":"2024-04-14T15:13:09.856407+0000","last_unstale":"2024-04-14T15:18:13.952216+0000","last_undegraded":"2024-04-14T15:18:13.952216+0000","last_fullsized":"2024-04-14T15:18:13.952216+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"219'85","last_scrub_stamp":"2024-04-14T15:17:51.438149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:51.438149+0000","objects_scrubbed":5,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:05.373103+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0020747539999999998,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"110.3","version":"355'52","reported_seq":815,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.954251+0000","last_change":"2024-04-14T15:16:06.149432+0000","last_active":"2024-04-14T15:18:13.954251+0000","last_peered":"2024-04-14T15:18:13.954251+0000","last_clean":"2024-04-14T15:18:13.954251+0000","last_became_active":"2024-04-14T15:13:09.997208+0000","last_became_peered":"2024-04-14T15:13:09.997208+0000","last_unstale":"2024-04-14T15:18:13.954251+0000","last_undegraded":"2024-04-14T15:18:13.954251+0000","last_fullsized":"2024-04-14T15:18:13.954251+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"223'51","last_scrub_stamp":"2024-04-14T15:16:06.149271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:16:06.149271+0000","objects_scrubbed":3,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0068076480000000003,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"110.2","version":"240'55","reported_seq":833,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.955419+0000","last_change":"2024-04-14T15:17:37.941679+0000","last_active":"2024-04-14T15:18:13.955419+0000","last_peered":"2024-04-14T15:18:13.955419+0000","last_clean":"2024-04-14T15:18:13.955419+0000","last_became_active":"2024-04-14T15:13:09.957187+0000","last_became_peered":"2024-04-14T15:13:09.957187+0000","last_unstale":"2024-04-14T15:18:13.955419+0000","last_undegraded":"2024-04-14T15:18:13.955419+0000","last_fullsized":"2024-04-14T15:18:13.955419+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'55","last_scrub_stamp":"2024-04-14T15:17:37.941589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:37.941589+0000","objects_scrubbed":6,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:44.165249+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0066344619999999998,"stat_sum":{"num_bytes":49,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":18,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"110.5","version":"396'64","reported_seq":834,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.953315+0000","last_change":"2024-04-14T15:17:47.418800+0000","last_active":"2024-04-14T15:18:13.953315+0000","last_peered":"2024-04-14T15:18:13.953315+0000","last_clean":"2024-04-14T15:18:13.953315+0000","last_became_active":"2024-04-14T15:13:09.853689+0000","last_became_peered":"2024-04-14T15:13:09.853689+0000","last_unstale":"2024-04-14T15:18:13.953315+0000","last_undegraded":"2024-04-14T15:18:13.953315+0000","last_fullsized":"2024-04-14T15:18:13.953315+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"396'64","last_scrub_stamp":"2024-04-14T15:17:47.418687+0000","last_deep_scrub":"396'64","last_deep_scrub_stamp":"2024-04-14T15:17:47.418687+0000","last_clean_scrub_stamp":"2024-04-14T15:17:47.418687+0000","objects_scrubbed":5,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:13.553428+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.001407202,"stat_sum":{"num_bytes":42,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":20,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"110.4","version":"406'55","reported_seq":823,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.955598+0000","last_change":"2024-04-14T15:16:05.487931+0000","last_active":"2024-04-14T15:18:13.955598+0000","last_peered":"2024-04-14T15:18:13.955598+0000","last_clean":"2024-04-14T15:18:13.955598+0000","last_became_active":"2024-04-14T15:13:09.958847+0000","last_became_peered":"2024-04-14T15:13:09.958847+0000","last_unstale":"2024-04-14T15:18:13.955598+0000","last_undegraded":"2024-04-14T15:18:13.955598+0000","last_fullsized":"2024-04-14T15:18:13.955598+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'51","last_scrub_stamp":"2024-04-14T15:16:05.487850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:16:05.487850+0000","objects_scrubbed":3,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0048323250000000002,"stat_sum":{"num_bytes":38,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"110.7","version":"355'18","reported_seq":799,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.964809+0000","last_change":"2024-04-14T15:17:34.552884+0000","last_active":"2024-04-14T15:18:13.964809+0000","last_peered":"2024-04-14T15:18:13.964809+0000","last_clean":"2024-04-14T15:18:13.964809+0000","last_became_active":"2024-04-14T15:13:09.936631+0000","last_became_peered":"2024-04-14T15:13:09.936631+0000","last_unstale":"2024-04-14T15:18:13.964809+0000","last_undegraded":"2024-04-14T15:18:13.964809+0000","last_fullsized":"2024-04-14T15:18:13.964809+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'18","last_scrub_stamp":"2024-04-14T15:17:34.552788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:34.552788+0000","objects_scrubbed":2,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:48.730286+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001103808,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"110.6","version":"274'74","reported_seq":831,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-14T15:18:12.947337+0000","last_change":"2024-04-14T15:16:03.897478+0000","last_active":"2024-04-14T15:18:12.947337+0000","last_peered":"2024-04-14T15:18:12.947337+0000","last_clean":"2024-04-14T15:18:12.947337+0000","last_became_active":"2024-04-14T15:13:09.930125+0000","last_became_peered":"2024-04-14T15:13:09.930125+0000","last_unstale":"2024-04-14T15:18:12.947337+0000","last_undegraded":"2024-04-14T15:18:12.947337+0000","last_fullsized":"2024-04-14T15:18:12.947337+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'74","last_scrub_stamp":"2024-04-14T15:15:42.641684+0000","last_deep_scrub":"274'74","last_deep_scrub_stamp":"2024-04-14T15:15:42.641684+0000","last_clean_scrub_stamp":"2024-04-14T15:15:42.641684+0000","objects_scrubbed":5,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:15.661452+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001879475,"stat_sum":{"num_bytes":44,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":21,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"110.9","version":"369'20","reported_seq":795,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.953631+0000","last_change":"2024-04-14T15:16:09.202557+0000","last_active":"2024-04-14T15:18:13.953631+0000","last_peered":"2024-04-14T15:18:13.953631+0000","last_clean":"2024-04-14T15:18:13.953631+0000","last_became_active":"2024-04-14T15:13:09.957324+0000","last_became_peered":"2024-04-14T15:13:09.957324+0000","last_unstale":"2024-04-14T15:18:13.953631+0000","last_undegraded":"2024-04-14T15:18:13.953631+0000","last_fullsized":"2024-04-14T15:18:13.953631+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"225'17","last_scrub_stamp":"2024-04-14T15:16:09.202462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:16:09.202462+0000","objects_scrubbed":1,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0055950490000000004,"stat_sum":{"num_bytes":35,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"110.8","version":"190'17","reported_seq":542,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.962297+0000","last_change":"2024-04-14T15:16:03.889092+0000","last_active":"2024-04-14T15:18:13.962297+0000","last_peered":"2024-04-14T15:18:13.962297+0000","last_clean":"2024-04-14T15:18:13.962297+0000","last_became_active":"2024-04-14T15:15:44.890057+0000","last_became_peered":"2024-04-14T15:15:44.890057+0000","last_unstale":"2024-04-14T15:18:13.962297+0000","last_undegraded":"2024-04-14T15:18:13.962297+0000","last_fullsized":"2024-04-14T15:18:13.962297+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'17","last_scrub_stamp":"2024-04-14T15:15:25.478924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:15:25.478924+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00271433,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"110.b","version":"412'71","reported_seq":841,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.959860+0000","last_change":"2024-04-14T15:16:49.611672+0000","last_active":"2024-04-14T15:18:13.959860+0000","last_peered":"2024-04-14T15:18:13.959860+0000","last_clean":"2024-04-14T15:18:13.959860+0000","last_became_active":"2024-04-14T15:13:09.964629+0000","last_became_peered":"2024-04-14T15:13:09.964629+0000","last_unstale":"2024-04-14T15:18:13.959860+0000","last_undegraded":"2024-04-14T15:18:13.959860+0000","last_fullsized":"2024-04-14T15:18:13.959860+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'68","last_scrub_stamp":"2024-04-14T15:16:49.611563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:16:49.611563+0000","objects_scrubbed":4,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.0049053020000000003,"stat_sum":{"num_bytes":54,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":22,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"110.a","version":"387'74","reported_seq":766,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.963165+0000","last_change":"2024-04-14T15:16:03.888790+0000","last_active":"2024-04-14T15:18:13.963165+0000","last_peered":"2024-04-14T15:18:13.963165+0000","last_clean":"2024-04-14T15:18:13.963165+0000","last_became_active":"2024-04-14T15:15:21.885346+0000","last_became_peered":"2024-04-14T15:15:21.885346+0000","last_unstale":"2024-04-14T15:18:13.963165+0000","last_undegraded":"2024-04-14T15:18:13.963165+0000","last_fullsized":"2024-04-14T15:18:13.963165+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'70","last_scrub_stamp":"2024-04-14T15:15:34.881731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:15:34.881731+0000","objects_scrubbed":5,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0.00502967,"stat_sum":{"num_bytes":84,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":26,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"110.d","version":"416'95","reported_seq":690,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.962895+0000","last_change":"2024-04-14T15:16:03.886262+0000","last_active":"2024-04-14T15:18:13.962895+0000","last_peered":"2024-04-14T15:18:13.962895+0000","last_clean":"2024-04-14T15:18:13.962895+0000","last_became_active":"2024-04-14T15:15:21.877073+0000","last_became_peered":"2024-04-14T15:15:21.877073+0000","last_unstale":"2024-04-14T15:18:13.962895+0000","last_undegraded":"2024-04-14T15:18:13.962895+0000","last_fullsized":"2024-04-14T15:18:13.962895+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'85","last_scrub_stamp":"2024-04-14T15:15:38.773256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:15:38.773256+0000","objects_scrubbed":5,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0018858709999999999,"stat_sum":{"num_bytes":71,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":34,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"110.c","version":"398'20","reported_seq":799,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.959741+0000","last_change":"2024-04-14T15:17:52.882647+0000","last_active":"2024-04-14T15:18:13.959741+0000","last_peered":"2024-04-14T15:18:13.959741+0000","last_clean":"2024-04-14T15:18:13.959741+0000","last_became_active":"2024-04-14T15:13:09.984609+0000","last_became_peered":"2024-04-14T15:13:09.984609+0000","last_unstale":"2024-04-14T15:18:13.959741+0000","last_undegraded":"2024-04-14T15:18:13.959741+0000","last_fullsized":"2024-04-14T15:18:13.959741+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'20","last_scrub_stamp":"2024-04-14T15:17:52.882509+0000","last_deep_scrub":"398'20","last_deep_scrub_stamp":"2024-04-14T15:17:52.882509+0000","last_clean_scrub_stamp":"2024-04-14T15:17:52.882509+0000","objects_scrubbed":2,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:16.770995+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0057052739999999998,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"110.f","version":"355'52","reported_seq":827,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.959403+0000","last_change":"2024-04-14T15:17:45.848294+0000","last_active":"2024-04-14T15:18:13.959403+0000","last_peered":"2024-04-14T15:18:13.959403+0000","last_clean":"2024-04-14T15:18:13.959403+0000","last_became_active":"2024-04-14T15:13:09.964057+0000","last_became_peered":"2024-04-14T15:13:09.964057+0000","last_unstale":"2024-04-14T15:18:13.959403+0000","last_undegraded":"2024-04-14T15:18:13.959403+0000","last_fullsized":"2024-04-14T15:18:13.959403+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'52","last_scrub_stamp":"2024-04-14T15:17:45.848166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:45.848166+0000","objects_scrubbed":4,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:13.513496+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0052446230000000003,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"110.e","version":"416'92","reported_seq":859,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-14T15:18:12.947224+0000","last_change":"2024-04-14T15:17:34.409475+0000","last_active":"2024-04-14T15:18:12.947224+0000","last_peered":"2024-04-14T15:18:12.947224+0000","last_clean":"2024-04-14T15:18:12.947224+0000","last_became_active":"2024-04-14T15:13:09.984980+0000","last_became_peered":"2024-04-14T15:13:09.984980+0000","last_unstale":"2024-04-14T15:18:12.947224+0000","last_undegraded":"2024-04-14T15:18:12.947224+0000","last_fullsized":"2024-04-14T15:18:12.947224+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'90","last_scrub_stamp":"2024-04-14T15:17:34.409367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:34.409367+0000","objects_scrubbed":6,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:58.827384+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0016350589999999999,"stat_sum":{"num_bytes":68,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":28,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":299,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.957052+0000","last_change":"2024-04-14T15:15:44.883275+0000","last_active":"2024-04-14T15:18:13.957052+0000","last_peered":"2024-04-14T15:18:13.957052+0000","last_clean":"2024-04-14T15:18:13.957052+0000","last_became_active":"2024-04-14T15:15:44.882941+0000","last_became_peered":"2024-04-14T15:15:44.882941+0000","last_unstale":"2024-04-14T15:18:13.957052+0000","last_undegraded":"2024-04-14T15:18:13.957052+0000","last_fullsized":"2024-04-14T15:18:13.957052+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:32.465858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:32.465858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.9","version":"85'9","reported_seq":856,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-14T15:18:12.943316+0000","last_change":"2024-04-14T15:17:54.606158+0000","last_active":"2024-04-14T15:18:12.943316+0000","last_peered":"2024-04-14T15:18:12.943316+0000","last_clean":"2024-04-14T15:18:12.943316+0000","last_became_active":"2024-04-14T15:09:47.045441+0000","last_became_peered":"2024-04-14T15:09:47.045441+0000","last_unstale":"2024-04-14T15:18:12.943316+0000","last_undegraded":"2024-04-14T15:18:12.943316+0000","last_fullsized":"2024-04-14T15:18:12.943316+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9","last_scrub_stamp":"2024-04-14T15:17:54.606051+0000","last_deep_scrub":"85'9","last_deep_scrub_stamp":"2024-04-14T15:10:51.950580+0000","last_clean_scrub_stamp":"2024-04-14T15:17:54.606051+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:07.946751+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"72'15","reported_seq":339,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-14T15:18:12.941703+0000","last_change":"2024-04-14T15:15:44.885742+0000","last_active":"2024-04-14T15:18:12.941703+0000","last_peered":"2024-04-14T15:18:12.941703+0000","last_clean":"2024-04-14T15:18:12.941703+0000","last_became_active":"2024-04-14T15:15:44.885409+0000","last_became_peered":"2024-04-14T15:15:44.885409+0000","last_unstale":"2024-04-14T15:18:12.941703+0000","last_undegraded":"2024-04-14T15:18:12.941703+0000","last_fullsized":"2024-04-14T15:18:12.941703+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-04-14T15:15:39.325129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:15:39.325129+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":753,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.960057+0000","last_change":"2024-04-14T15:17:53.867764+0000","last_active":"2024-04-14T15:18:13.960057+0000","last_peered":"2024-04-14T15:18:13.960057+0000","last_clean":"2024-04-14T15:18:13.960057+0000","last_became_active":"2024-04-14T15:09:43.021460+0000","last_became_peered":"2024-04-14T15:09:43.021460+0000","last_unstale":"2024-04-14T15:18:13.960057+0000","last_undegraded":"2024-04-14T15:18:13.960057+0000","last_fullsized":"2024-04-14T15:18:13.960057+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:53.867696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:53.867696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:56.203183+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":307,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.964139+0000","last_change":"2024-04-14T15:17:54.889170+0000","last_active":"2024-04-14T15:18:13.964139+0000","last_peered":"2024-04-14T15:18:13.964139+0000","last_clean":"2024-04-14T15:18:13.964139+0000","last_became_active":"2024-04-14T15:15:44.888154+0000","last_became_peered":"2024-04-14T15:15:44.888154+0000","last_unstale":"2024-04-14T15:18:13.964139+0000","last_undegraded":"2024-04-14T15:18:13.964139+0000","last_fullsized":"2024-04-14T15:18:13.964139+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:54.889066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:15:40.368219+0000","last_clean_scrub_stamp":"2024-04-14T15:17:54.889066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:11.492168+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":641,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-14T15:18:12.945578+0000","last_change":"2024-04-14T15:15:32.596167+0000","last_active":"2024-04-14T15:18:12.945578+0000","last_peered":"2024-04-14T15:18:12.945578+0000","last_clean":"2024-04-14T15:18:12.945578+0000","last_became_active":"2024-04-14T15:12:56.355348+0000","last_became_peered":"2024-04-14T15:12:56.355348+0000","last_unstale":"2024-04-14T15:18:12.945578+0000","last_undegraded":"2024-04-14T15:18:12.945578+0000","last_fullsized":"2024-04-14T15:18:12.945578+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:32.596096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:32.596096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"72'9","reported_seq":768,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.955807+0000","last_change":"2024-04-14T15:16:56.663996+0000","last_active":"2024-04-14T15:18:13.955807+0000","last_peered":"2024-04-14T15:18:13.955807+0000","last_clean":"2024-04-14T15:18:13.955807+0000","last_became_active":"2024-04-14T15:09:45.037179+0000","last_became_peered":"2024-04-14T15:09:45.037179+0000","last_unstale":"2024-04-14T15:18:13.955807+0000","last_undegraded":"2024-04-14T15:18:13.955807+0000","last_fullsized":"2024-04-14T15:18:13.955807+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-04-14T15:16:56.663900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:16:56.663900+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:22.985760+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":748,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.957127+0000","last_change":"2024-04-14T15:15:25.311260+0000","last_active":"2024-04-14T15:18:13.957127+0000","last_peered":"2024-04-14T15:18:13.957127+0000","last_clean":"2024-04-14T15:18:13.957127+0000","last_became_active":"2024-04-14T15:09:43.020044+0000","last_became_peered":"2024-04-14T15:09:43.020044+0000","last_unstale":"2024-04-14T15:18:13.957127+0000","last_undegraded":"2024-04-14T15:18:13.957127+0000","last_fullsized":"2024-04-14T15:18:13.957127+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:25.311184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:15:25.311184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":745,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.959300+0000","last_change":"2024-04-14T15:16:16.655593+0000","last_active":"2024-04-14T15:18:13.959300+0000","last_peered":"2024-04-14T15:18:13.959300+0000","last_clean":"2024-04-14T15:18:13.959300+0000","last_became_active":"2024-04-14T15:09:47.046153+0000","last_became_peered":"2024-04-14T15:09:47.046153+0000","last_unstale":"2024-04-14T15:18:13.959300+0000","last_undegraded":"2024-04-14T15:18:13.959300+0000","last_fullsized":"2024-04-14T15:18:13.959300+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:16.655504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:16.655504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:13.986058+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":744,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.955932+0000","last_change":"2024-04-14T15:17:37.919665+0000","last_active":"2024-04-14T15:18:13.955932+0000","last_peered":"2024-04-14T15:18:13.955932+0000","last_clean":"2024-04-14T15:18:13.955932+0000","last_became_active":"2024-04-14T15:09:49.061495+0000","last_became_peered":"2024-04-14T15:09:49.061495+0000","last_unstale":"2024-04-14T15:18:13.955932+0000","last_undegraded":"2024-04-14T15:18:13.955932+0000","last_fullsized":"2024-04-14T15:18:13.955932+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:37.919595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:17:37.919595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:42.403547+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":641,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-14T15:18:12.947094+0000","last_change":"2024-04-14T15:14:55.989719+0000","last_active":"2024-04-14T15:18:12.947094+0000","last_peered":"2024-04-14T15:18:12.947094+0000","last_clean":"2024-04-14T15:18:12.947094+0000","last_became_active":"2024-04-14T15:12:56.246062+0000","last_became_peered":"2024-04-14T15:12:56.246062+0000","last_unstale":"2024-04-14T15:18:12.947094+0000","last_undegraded":"2024-04-14T15:18:12.947094+0000","last_fullsized":"2024-04-14T15:18:12.947094+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:55.989627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:14:55.989627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"222'28","reported_seq":791,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.964598+0000","last_change":"2024-04-14T15:17:44.832902+0000","last_active":"2024-04-14T15:18:13.964598+0000","last_peered":"2024-04-14T15:18:13.964598+0000","last_clean":"2024-04-14T15:18:13.964598+0000","last_became_active":"2024-04-14T15:09:45.178782+0000","last_became_peered":"2024-04-14T15:09:45.178782+0000","last_unstale":"2024-04-14T15:18:13.964598+0000","last_undegraded":"2024-04-14T15:18:13.964598+0000","last_fullsized":"2024-04-14T15:18:13.964598+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'28","last_scrub_stamp":"2024-04-14T15:17:44.832769+0000","last_deep_scrub":"222'28","last_deep_scrub_stamp":"2024-04-14T15:14:50.054931+0000","last_clean_scrub_stamp":"2024-04-14T15:17:44.832769+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:54.927275+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":753,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.960196+0000","last_change":"2024-04-14T15:15:43.569812+0000","last_active":"2024-04-14T15:18:13.960196+0000","last_peered":"2024-04-14T15:18:13.960196+0000","last_clean":"2024-04-14T15:18:13.960196+0000","last_became_active":"2024-04-14T15:09:43.020268+0000","last_became_peered":"2024-04-14T15:09:43.020268+0000","last_unstale":"2024-04-14T15:18:13.960196+0000","last_undegraded":"2024-04-14T15:18:13.960196+0000","last_fullsized":"2024-04-14T15:18:13.960196+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:43.569735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:34.342793+0000","last_clean_scrub_stamp":"2024-04-14T15:15:43.569735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:13.985975+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":737,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.959851+0000","last_change":"2024-04-14T15:15:30.488857+0000","last_active":"2024-04-14T15:18:13.959851+0000","last_peered":"2024-04-14T15:18:13.959851+0000","last_clean":"2024-04-14T15:18:13.959851+0000","last_became_active":"2024-04-14T15:09:47.045542+0000","last_became_peered":"2024-04-14T15:09:47.045542+0000","last_unstale":"2024-04-14T15:18:13.959851+0000","last_undegraded":"2024-04-14T15:18:13.959851+0000","last_fullsized":"2024-04-14T15:18:13.959851+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:30.488781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:15:30.488781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:13.985922+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":732,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-14T15:18:12.942156+0000","last_change":"2024-04-14T15:16:08.857476+0000","last_active":"2024-04-14T15:18:12.942156+0000","last_peered":"2024-04-14T15:18:12.942156+0000","last_clean":"2024-04-14T15:18:12.942156+0000","last_became_active":"2024-04-14T15:09:49.065397+0000","last_became_peered":"2024-04-14T15:09:49.065397+0000","last_unstale":"2024-04-14T15:18:12.942156+0000","last_undegraded":"2024-04-14T15:18:12.942156+0000","last_fullsized":"2024-04-14T15:18:12.942156+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:08.857397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:08.857397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"91'130","reported_seq":622,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.958587+0000","last_change":"2024-04-14T15:16:07.609240+0000","last_active":"2024-04-14T15:18:13.958587+0000","last_peered":"2024-04-14T15:18:13.958587+0000","last_clean":"2024-04-14T15:18:13.958587+0000","last_became_active":"2024-04-14T15:15:44.886240+0000","last_became_peered":"2024-04-14T15:15:44.886240+0000","last_unstale":"2024-04-14T15:18:13.958587+0000","last_undegraded":"2024-04-14T15:18:13.958587+0000","last_fullsized":"2024-04-14T15:18:13.958587+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:16:07.609119+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:16:07.609119+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:13.986022+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":650,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.959275+0000","last_change":"2024-04-14T15:16:19.749313+0000","last_active":"2024-04-14T15:18:13.959275+0000","last_peered":"2024-04-14T15:18:13.959275+0000","last_clean":"2024-04-14T15:18:13.959275+0000","last_became_active":"2024-04-14T15:12:56.474322+0000","last_became_peered":"2024-04-14T15:12:56.474322+0000","last_unstale":"2024-04-14T15:18:13.959275+0000","last_undegraded":"2024-04-14T15:18:13.959275+0000","last_fullsized":"2024-04-14T15:18:13.959275+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:19.749208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:16:19.749208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:13.986074+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"72'17","reported_seq":826,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.953756+0000","last_change":"2024-04-14T15:17:48.374589+0000","last_active":"2024-04-14T15:18:13.953756+0000","last_peered":"2024-04-14T15:18:13.953756+0000","last_clean":"2024-04-14T15:18:13.953756+0000","last_became_active":"2024-04-14T15:13:12.933907+0000","last_became_peered":"2024-04-14T15:13:12.933907+0000","last_unstale":"2024-04-14T15:18:13.953756+0000","last_undegraded":"2024-04-14T15:18:13.953756+0000","last_fullsized":"2024-04-14T15:18:13.953756+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'17","last_scrub_stamp":"2024-04-14T15:17:48.374469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:17:48.374469+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:04.297073+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":744,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-14T15:18:12.945099+0000","last_change":"2024-04-14T15:16:16.820151+0000","last_active":"2024-04-14T15:18:12.945099+0000","last_peered":"2024-04-14T15:18:12.945099+0000","last_clean":"2024-04-14T15:18:12.945099+0000","last_became_active":"2024-04-14T15:09:43.022351+0000","last_became_peered":"2024-04-14T15:09:43.022351+0000","last_unstale":"2024-04-14T15:18:12.945099+0000","last_undegraded":"2024-04-14T15:18:12.945099+0000","last_fullsized":"2024-04-14T15:18:12.945099+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:16.820070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:16.820070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":740,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.955610+0000","last_change":"2024-04-14T15:17:54.871693+0000","last_active":"2024-04-14T15:18:13.955610+0000","last_peered":"2024-04-14T15:18:13.955610+0000","last_clean":"2024-04-14T15:18:13.955610+0000","last_became_active":"2024-04-14T15:09:47.042810+0000","last_became_peered":"2024-04-14T15:09:47.042810+0000","last_unstale":"2024-04-14T15:18:13.955610+0000","last_undegraded":"2024-04-14T15:18:13.955610+0000","last_fullsized":"2024-04-14T15:18:13.955610+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:54.871593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:17:54.871593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:21.351148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":741,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.959698+0000","last_change":"2024-04-14T15:16:45.669141+0000","last_active":"2024-04-14T15:18:13.959698+0000","last_peered":"2024-04-14T15:18:13.959698+0000","last_clean":"2024-04-14T15:18:13.959698+0000","last_became_active":"2024-04-14T15:09:49.069594+0000","last_became_peered":"2024-04-14T15:09:49.069594+0000","last_unstale":"2024-04-14T15:18:13.959698+0000","last_undegraded":"2024-04-14T15:18:13.959698+0000","last_fullsized":"2024-04-14T15:18:13.959698+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:45.669063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:45.669063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:16.883168+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"91'130","reported_seq":1014,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-14T15:18:12.942822+0000","last_change":"2024-04-14T15:14:46.903338+0000","last_active":"2024-04-14T15:18:12.942822+0000","last_peered":"2024-04-14T15:18:12.942822+0000","last_clean":"2024-04-14T15:18:12.942822+0000","last_became_active":"2024-04-14T15:14:17.873403+0000","last_became_peered":"2024-04-14T15:14:17.873403+0000","last_unstale":"2024-04-14T15:18:12.942822+0000","last_undegraded":"2024-04-14T15:18:12.942822+0000","last_fullsized":"2024-04-14T15:18:12.942822+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:14:46.903248+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:14:46.903248+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":652,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.963756+0000","last_change":"2024-04-14T15:17:46.871883+0000","last_active":"2024-04-14T15:18:13.963756+0000","last_peered":"2024-04-14T15:18:13.963756+0000","last_clean":"2024-04-14T15:18:13.963756+0000","last_became_active":"2024-04-14T15:12:56.289745+0000","last_became_peered":"2024-04-14T15:12:56.289745+0000","last_unstale":"2024-04-14T15:18:13.963756+0000","last_undegraded":"2024-04-14T15:18:13.963756+0000","last_fullsized":"2024-04-14T15:18:13.963756+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:46.871773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:17:46.871773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:58.233534+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"179'15","reported_seq":762,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.952740+0000","last_change":"2024-04-14T15:16:49.243096+0000","last_active":"2024-04-14T15:18:13.952740+0000","last_peered":"2024-04-14T15:18:13.952740+0000","last_clean":"2024-04-14T15:18:13.952740+0000","last_became_active":"2024-04-14T15:09:45.029170+0000","last_became_peered":"2024-04-14T15:09:45.029170+0000","last_unstale":"2024-04-14T15:18:13.952740+0000","last_undegraded":"2024-04-14T15:18:13.952740+0000","last_fullsized":"2024-04-14T15:18:13.952740+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'15","last_scrub_stamp":"2024-04-14T15:16:49.243002+0000","last_deep_scrub":"179'15","last_deep_scrub_stamp":"2024-04-14T15:14:07.488422+0000","last_clean_scrub_stamp":"2024-04-14T15:16:49.243002+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:16.454013+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":756,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.954103+0000","last_change":"2024-04-14T15:17:51.854742+0000","last_active":"2024-04-14T15:18:13.954103+0000","last_peered":"2024-04-14T15:18:13.954103+0000","last_clean":"2024-04-14T15:18:13.954103+0000","last_became_active":"2024-04-14T15:09:43.018871+0000","last_became_peered":"2024-04-14T15:09:43.018871+0000","last_unstale":"2024-04-14T15:18:13.954103+0000","last_undegraded":"2024-04-14T15:18:13.954103+0000","last_fullsized":"2024-04-14T15:18:13.954103+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:51.854659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:51.854659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:08.464204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"400'9","reported_seq":857,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.960076+0000","last_change":"2024-04-14T15:17:54.845967+0000","last_active":"2024-04-14T15:18:13.960076+0000","last_peered":"2024-04-14T15:18:13.960076+0000","last_clean":"2024-04-14T15:18:13.960076+0000","last_became_active":"2024-04-14T15:09:47.044085+0000","last_became_peered":"2024-04-14T15:09:47.044085+0000","last_unstale":"2024-04-14T15:18:13.960076+0000","last_undegraded":"2024-04-14T15:18:13.960076+0000","last_fullsized":"2024-04-14T15:18:13.960076+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'9","last_scrub_stamp":"2024-04-14T15:17:54.845881+0000","last_deep_scrub":"400'9","last_deep_scrub_stamp":"2024-04-14T15:17:54.845881+0000","last_clean_scrub_stamp":"2024-04-14T15:17:54.845881+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:58.974945+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":734,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.953425+0000","last_change":"2024-04-14T15:16:17.041377+0000","last_active":"2024-04-14T15:18:13.953425+0000","last_peered":"2024-04-14T15:18:13.953425+0000","last_clean":"2024-04-14T15:18:13.953425+0000","last_became_active":"2024-04-14T15:09:49.065274+0000","last_became_peered":"2024-04-14T15:09:49.065274+0000","last_unstale":"2024-04-14T15:18:13.953425+0000","last_undegraded":"2024-04-14T15:18:13.953425+0000","last_fullsized":"2024-04-14T15:18:13.953425+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:17.041301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:17.041301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:16.453761+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"91'130","reported_seq":622,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.958878+0000","last_change":"2024-04-14T15:15:45.589217+0000","last_active":"2024-04-14T15:18:13.958878+0000","last_peered":"2024-04-14T15:18:13.958878+0000","last_clean":"2024-04-14T15:18:13.958878+0000","last_became_active":"2024-04-14T15:15:44.887114+0000","last_became_peered":"2024-04-14T15:15:44.887114+0000","last_unstale":"2024-04-14T15:18:13.958878+0000","last_undegraded":"2024-04-14T15:18:13.958878+0000","last_fullsized":"2024-04-14T15:18:13.958878+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:15:45.589132+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:15:45.589132+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:13.985959+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":645,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.956811+0000","last_change":"2024-04-14T15:15:29.366895+0000","last_active":"2024-04-14T15:18:13.956811+0000","last_peered":"2024-04-14T15:18:13.956811+0000","last_clean":"2024-04-14T15:18:13.956811+0000","last_became_active":"2024-04-14T15:12:56.487615+0000","last_became_peered":"2024-04-14T15:12:56.487615+0000","last_unstale":"2024-04-14T15:18:13.956811+0000","last_undegraded":"2024-04-14T15:18:13.956811+0000","last_fullsized":"2024-04-14T15:18:13.956811+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:29.366820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:29.366820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"398'22","reported_seq":784,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.957157+0000","last_change":"2024-04-14T15:17:52.883984+0000","last_active":"2024-04-14T15:18:13.957157+0000","last_peered":"2024-04-14T15:18:13.957157+0000","last_clean":"2024-04-14T15:18:13.957157+0000","last_became_active":"2024-04-14T15:09:45.033331+0000","last_became_peered":"2024-04-14T15:09:45.033331+0000","last_unstale":"2024-04-14T15:18:13.957157+0000","last_undegraded":"2024-04-14T15:18:13.957157+0000","last_fullsized":"2024-04-14T15:18:13.957157+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'22","last_scrub_stamp":"2024-04-14T15:17:52.883850+0000","last_deep_scrub":"398'22","last_deep_scrub_stamp":"2024-04-14T15:17:52.883850+0000","last_clean_scrub_stamp":"2024-04-14T15:17:52.883850+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:58.280825+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":748,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.953941+0000","last_change":"2024-04-14T15:15:00.203399+0000","last_active":"2024-04-14T15:18:13.953941+0000","last_peered":"2024-04-14T15:18:13.953941+0000","last_clean":"2024-04-14T15:18:13.953941+0000","last_became_active":"2024-04-14T15:09:43.020878+0000","last_became_peered":"2024-04-14T15:09:43.020878+0000","last_unstale":"2024-04-14T15:18:13.953941+0000","last_undegraded":"2024-04-14T15:18:13.953941+0000","last_fullsized":"2024-04-14T15:18:13.953941+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:00.203310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:15:00.203310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"72'8","reported_seq":849,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.953421+0000","last_change":"2024-04-14T15:16:06.933663+0000","last_active":"2024-04-14T15:18:13.953421+0000","last_peered":"2024-04-14T15:18:13.953421+0000","last_clean":"2024-04-14T15:18:13.953421+0000","last_became_active":"2024-04-14T15:09:47.050962+0000","last_became_peered":"2024-04-14T15:09:47.050962+0000","last_unstale":"2024-04-14T15:18:13.953421+0000","last_undegraded":"2024-04-14T15:18:13.953421+0000","last_fullsized":"2024-04-14T15:18:13.953421+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:16:06.933489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:06.933489+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:16.453795+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":308,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.956685+0000","last_change":"2024-04-14T15:16:51.199025+0000","last_active":"2024-04-14T15:18:13.956685+0000","last_peered":"2024-04-14T15:18:13.956685+0000","last_clean":"2024-04-14T15:18:13.956685+0000","last_became_active":"2024-04-14T15:15:44.879699+0000","last_became_peered":"2024-04-14T15:15:44.879699+0000","last_unstale":"2024-04-14T15:18:13.956685+0000","last_undegraded":"2024-04-14T15:18:13.956685+0000","last_fullsized":"2024-04-14T15:18:13.956685+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:51.198907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:51.198907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:19.763218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"233'8","reported_seq":666,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.963777+0000","last_change":"2024-04-14T15:17:50.905698+0000","last_active":"2024-04-14T15:18:13.963777+0000","last_peered":"2024-04-14T15:18:13.963777+0000","last_clean":"2024-04-14T15:18:13.963777+0000","last_became_active":"2024-04-14T15:12:56.381026+0000","last_became_peered":"2024-04-14T15:12:56.381026+0000","last_unstale":"2024-04-14T15:18:13.963777+0000","last_undegraded":"2024-04-14T15:18:13.963777+0000","last_fullsized":"2024-04-14T15:18:13.963777+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'8","last_scrub_stamp":"2024-04-14T15:17:50.905608+0000","last_deep_scrub":"233'8","last_deep_scrub_stamp":"2024-04-14T15:15:01.295800+0000","last_clean_scrub_stamp":"2024-04-14T15:17:50.905608+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:14.140648+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"85'16","reported_seq":774,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.952804+0000","last_change":"2024-04-14T15:16:53.143329+0000","last_active":"2024-04-14T15:18:13.952804+0000","last_peered":"2024-04-14T15:18:13.952804+0000","last_clean":"2024-04-14T15:18:13.952804+0000","last_became_active":"2024-04-14T15:09:45.028212+0000","last_became_peered":"2024-04-14T15:09:45.028212+0000","last_unstale":"2024-04-14T15:18:13.952804+0000","last_undegraded":"2024-04-14T15:18:13.952804+0000","last_fullsized":"2024-04-14T15:18:13.952804+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-14T15:16:53.143239+0000","last_deep_scrub":"85'16","last_deep_scrub_stamp":"2024-04-14T15:10:47.456169+0000","last_clean_scrub_stamp":"2024-04-14T15:16:53.143239+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:21.167906+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":752,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-14T15:18:12.941918+0000","last_change":"2024-04-14T15:17:39.465617+0000","last_active":"2024-04-14T15:18:12.941918+0000","last_peered":"2024-04-14T15:18:12.941918+0000","last_clean":"2024-04-14T15:18:12.941918+0000","last_became_active":"2024-04-14T15:09:43.018601+0000","last_became_peered":"2024-04-14T15:09:43.018601+0000","last_unstale":"2024-04-14T15:18:12.941918+0000","last_undegraded":"2024-04-14T15:18:12.941918+0000","last_fullsized":"2024-04-14T15:18:12.941918+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:39.465475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:39.465475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:44.272999+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":305,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.953675+0000","last_change":"2024-04-14T15:16:54.187971+0000","last_active":"2024-04-14T15:18:13.953675+0000","last_peered":"2024-04-14T15:18:13.953675+0000","last_clean":"2024-04-14T15:18:13.953675+0000","last_became_active":"2024-04-14T15:15:44.886834+0000","last_became_peered":"2024-04-14T15:15:44.886834+0000","last_unstale":"2024-04-14T15:18:13.953675+0000","last_undegraded":"2024-04-14T15:18:13.953675+0000","last_fullsized":"2024-04-14T15:18:13.953675+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:54.187888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:54.187888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:16.453980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":417,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.963175+0000","last_change":"2024-04-14T15:17:57.214134+0000","last_active":"2024-04-14T15:18:13.963175+0000","last_peered":"2024-04-14T15:18:13.963175+0000","last_clean":"2024-04-14T15:18:13.963175+0000","last_became_active":"2024-04-14T15:15:21.888587+0000","last_became_peered":"2024-04-14T15:15:21.888587+0000","last_unstale":"2024-04-14T15:18:13.963175+0000","last_undegraded":"2024-04-14T15:18:13.963175+0000","last_fullsized":"2024-04-14T15:18:13.963175+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:57.214046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:57.214046+0000","last_clean_scrub_stamp":"2024-04-14T15:17:57.214046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:03.985520+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"91'130","reported_seq":803,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.963021+0000","last_change":"2024-04-14T15:16:41.944414+0000","last_active":"2024-04-14T15:18:13.963021+0000","last_peered":"2024-04-14T15:18:13.963021+0000","last_clean":"2024-04-14T15:18:13.963021+0000","last_became_active":"2024-04-14T15:14:17.863503+0000","last_became_peered":"2024-04-14T15:14:17.863503+0000","last_unstale":"2024-04-14T15:18:13.963021+0000","last_undegraded":"2024-04-14T15:18:13.963021+0000","last_fullsized":"2024-04-14T15:18:13.963021+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:16:41.944267+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:16:41.944267+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.359204+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":651,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.952694+0000","last_change":"2024-04-14T15:16:56.214275+0000","last_active":"2024-04-14T15:18:13.952694+0000","last_peered":"2024-04-14T15:18:13.952694+0000","last_clean":"2024-04-14T15:18:13.952694+0000","last_became_active":"2024-04-14T15:12:56.427276+0000","last_became_peered":"2024-04-14T15:12:56.427276+0000","last_unstale":"2024-04-14T15:18:13.952694+0000","last_undegraded":"2024-04-14T15:18:13.952694+0000","last_fullsized":"2024-04-14T15:18:13.952694+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:56.214178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:16:56.214178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:16.520034+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"255'10","reported_seq":493,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.962547+0000","last_change":"2024-04-14T15:16:17.868542+0000","last_active":"2024-04-14T15:18:13.962547+0000","last_peered":"2024-04-14T15:18:13.962547+0000","last_clean":"2024-04-14T15:18:13.962547+0000","last_became_active":"2024-04-14T15:15:21.873747+0000","last_became_peered":"2024-04-14T15:15:21.873747+0000","last_unstale":"2024-04-14T15:18:13.962547+0000","last_undegraded":"2024-04-14T15:18:13.962547+0000","last_fullsized":"2024-04-14T15:18:13.962547+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'10","last_scrub_stamp":"2024-04-14T15:16:17.868428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:17.868428+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.359102+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"72'14","reported_seq":764,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.952215+0000","last_change":"2024-04-14T15:15:38.962791+0000","last_active":"2024-04-14T15:18:13.952215+0000","last_peered":"2024-04-14T15:18:13.952215+0000","last_clean":"2024-04-14T15:18:13.952215+0000","last_became_active":"2024-04-14T15:09:45.025673+0000","last_became_peered":"2024-04-14T15:09:45.025673+0000","last_unstale":"2024-04-14T15:18:13.952215+0000","last_undegraded":"2024-04-14T15:18:13.952215+0000","last_fullsized":"2024-04-14T15:18:13.952215+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'14","last_scrub_stamp":"2024-04-14T15:15:38.962680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:15:38.962680+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:16.453543+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":747,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.964719+0000","last_change":"2024-04-14T15:15:39.123305+0000","last_active":"2024-04-14T15:18:13.964719+0000","last_peered":"2024-04-14T15:18:13.964719+0000","last_clean":"2024-04-14T15:18:13.964719+0000","last_became_active":"2024-04-14T15:09:43.020503+0000","last_became_peered":"2024-04-14T15:09:43.020503+0000","last_unstale":"2024-04-14T15:18:13.964719+0000","last_undegraded":"2024-04-14T15:18:13.964719+0000","last_fullsized":"2024-04-14T15:18:13.964719+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:39.123227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:15:39.123227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":733,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-14T15:18:12.945368+0000","last_change":"2024-04-14T15:16:45.705837+0000","last_active":"2024-04-14T15:18:12.945368+0000","last_peered":"2024-04-14T15:18:12.945368+0000","last_clean":"2024-04-14T15:18:12.945368+0000","last_became_active":"2024-04-14T15:09:49.129610+0000","last_became_peered":"2024-04-14T15:09:49.129610+0000","last_unstale":"2024-04-14T15:18:12.945368+0000","last_undegraded":"2024-04-14T15:18:12.945368+0000","last_fullsized":"2024-04-14T15:18:12.945368+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:45.705761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:45.705761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"91'130","reported_seq":805,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.952992+0000","last_change":"2024-04-14T15:16:41.193543+0000","last_active":"2024-04-14T15:18:13.952992+0000","last_peered":"2024-04-14T15:18:13.952992+0000","last_clean":"2024-04-14T15:18:13.952992+0000","last_became_active":"2024-04-14T15:14:17.871922+0000","last_became_peered":"2024-04-14T15:14:17.871922+0000","last_unstale":"2024-04-14T15:18:13.952992+0000","last_undegraded":"2024-04-14T15:18:13.952992+0000","last_fullsized":"2024-04-14T15:18:13.952992+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:16:41.193422+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:16:41.193422+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:16.453952+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":649,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-14T15:18:12.947743+0000","last_change":"2024-04-14T15:17:47.553179+0000","last_active":"2024-04-14T15:18:12.947743+0000","last_peered":"2024-04-14T15:18:12.947743+0000","last_clean":"2024-04-14T15:18:12.947743+0000","last_became_active":"2024-04-14T15:12:56.355254+0000","last_became_peered":"2024-04-14T15:12:56.355254+0000","last_unstale":"2024-04-14T15:18:12.947743+0000","last_undegraded":"2024-04-14T15:18:12.947743+0000","last_fullsized":"2024-04-14T15:18:12.947743+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:47.553055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:17:47.553055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:58.344778+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":739,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.952516+0000","last_change":"2024-04-14T15:16:05.917909+0000","last_active":"2024-04-14T15:18:13.952516+0000","last_peered":"2024-04-14T15:18:13.952516+0000","last_clean":"2024-04-14T15:18:13.952516+0000","last_became_active":"2024-04-14T15:09:47.044610+0000","last_became_peered":"2024-04-14T15:09:47.044610+0000","last_unstale":"2024-04-14T15:18:13.952516+0000","last_undegraded":"2024-04-14T15:18:13.952516+0000","last_fullsized":"2024-04-14T15:18:13.952516+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:05.917804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:05.917804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:16.453643+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"72'18","reported_seq":458,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.962542+0000","last_change":"2024-04-14T15:16:15.913377+0000","last_active":"2024-04-14T15:18:13.962542+0000","last_peered":"2024-04-14T15:18:13.962542+0000","last_clean":"2024-04-14T15:18:13.962542+0000","last_became_active":"2024-04-14T15:15:21.880565+0000","last_became_peered":"2024-04-14T15:15:21.880565+0000","last_unstale":"2024-04-14T15:18:13.962542+0000","last_undegraded":"2024-04-14T15:18:13.962542+0000","last_fullsized":"2024-04-14T15:18:13.962542+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'18","last_scrub_stamp":"2024-04-14T15:16:15.913273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:16:15.913273+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.359084+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'2","reported_seq":750,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.956316+0000","last_change":"2024-04-14T15:17:48.850178+0000","last_active":"2024-04-14T15:18:13.956316+0000","last_peered":"2024-04-14T15:18:13.956316+0000","last_clean":"2024-04-14T15:18:13.956316+0000","last_became_active":"2024-04-14T15:09:43.018478+0000","last_became_peered":"2024-04-14T15:09:43.018478+0000","last_unstale":"2024-04-14T15:18:13.956316+0000","last_undegraded":"2024-04-14T15:18:13.956316+0000","last_fullsized":"2024-04-14T15:18:13.956316+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'2","last_scrub_stamp":"2024-04-14T15:17:48.850069+0000","last_deep_scrub":"91'2","last_deep_scrub_stamp":"2024-04-14T15:17:48.850069+0000","last_clean_scrub_stamp":"2024-04-14T15:17:48.850069+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:12.966475+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":735,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.953229+0000","last_change":"2024-04-14T15:16:14.017818+0000","last_active":"2024-04-14T15:18:13.953229+0000","last_peered":"2024-04-14T15:18:13.953229+0000","last_clean":"2024-04-14T15:18:13.953229+0000","last_became_active":"2024-04-14T15:09:49.065684+0000","last_became_peered":"2024-04-14T15:09:49.065684+0000","last_unstale":"2024-04-14T15:18:13.953229+0000","last_undegraded":"2024-04-14T15:18:13.953229+0000","last_fullsized":"2024-04-14T15:18:13.953229+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:14.017729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:14.017729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:16.453672+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"91'130","reported_seq":813,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.964467+0000","last_change":"2024-04-14T15:17:47.885405+0000","last_active":"2024-04-14T15:18:13.964467+0000","last_peered":"2024-04-14T15:18:13.964467+0000","last_clean":"2024-04-14T15:18:13.964467+0000","last_became_active":"2024-04-14T15:14:17.853143+0000","last_became_peered":"2024-04-14T15:14:17.853143+0000","last_unstale":"2024-04-14T15:18:13.964467+0000","last_undegraded":"2024-04-14T15:18:13.964467+0000","last_fullsized":"2024-04-14T15:18:13.964467+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:17:47.885308+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:17:47.885308+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:05.853775+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":645,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.955386+0000","last_change":"2024-04-14T15:15:31.403165+0000","last_active":"2024-04-14T15:18:13.955386+0000","last_peered":"2024-04-14T15:18:13.955386+0000","last_clean":"2024-04-14T15:18:13.955386+0000","last_became_active":"2024-04-14T15:12:56.402023+0000","last_became_peered":"2024-04-14T15:12:56.402023+0000","last_unstale":"2024-04-14T15:18:13.955386+0000","last_undegraded":"2024-04-14T15:18:13.955386+0000","last_fullsized":"2024-04-14T15:18:13.955386+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:31.403086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:31.403086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":748,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.964776+0000","last_change":"2024-04-14T15:17:39.682323+0000","last_active":"2024-04-14T15:18:13.964776+0000","last_peered":"2024-04-14T15:18:13.964776+0000","last_clean":"2024-04-14T15:18:13.964776+0000","last_became_active":"2024-04-14T15:09:47.048427+0000","last_became_peered":"2024-04-14T15:09:47.048427+0000","last_unstale":"2024-04-14T15:18:13.964776+0000","last_undegraded":"2024-04-14T15:18:13.964776+0000","last_fullsized":"2024-04-14T15:18:13.964776+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:39.682203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:17:39.682203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:58.685184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"102'27","reported_seq":797,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.955995+0000","last_change":"2024-04-14T15:17:46.924432+0000","last_active":"2024-04-14T15:18:13.955995+0000","last_peered":"2024-04-14T15:18:13.955995+0000","last_clean":"2024-04-14T15:18:13.955995+0000","last_became_active":"2024-04-14T15:09:45.179182+0000","last_became_peered":"2024-04-14T15:09:45.179182+0000","last_unstale":"2024-04-14T15:18:13.955995+0000","last_undegraded":"2024-04-14T15:18:13.955995+0000","last_fullsized":"2024-04-14T15:18:13.955995+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'27","last_scrub_stamp":"2024-04-14T15:17:46.924330+0000","last_deep_scrub":"102'27","last_deep_scrub_stamp":"2024-04-14T15:17:46.924330+0000","last_clean_scrub_stamp":"2024-04-14T15:17:46.924330+0000","objects_scrubbed":8,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:54.826490+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":745,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-14T15:18:12.946940+0000","last_change":"2024-04-14T15:16:18.774542+0000","last_active":"2024-04-14T15:18:12.946940+0000","last_peered":"2024-04-14T15:18:12.946940+0000","last_clean":"2024-04-14T15:18:12.946940+0000","last_became_active":"2024-04-14T15:09:43.021006+0000","last_became_peered":"2024-04-14T15:09:43.021006+0000","last_unstale":"2024-04-14T15:18:12.946940+0000","last_undegraded":"2024-04-14T15:18:12.946940+0000","last_fullsized":"2024-04-14T15:18:12.946940+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:18.774463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:18.774463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":737,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.954725+0000","last_change":"2024-04-14T15:16:56.206766+0000","last_active":"2024-04-14T15:18:13.954725+0000","last_peered":"2024-04-14T15:18:13.954725+0000","last_clean":"2024-04-14T15:18:13.954725+0000","last_became_active":"2024-04-14T15:09:49.128136+0000","last_became_peered":"2024-04-14T15:09:49.128136+0000","last_unstale":"2024-04-14T15:18:13.954725+0000","last_undegraded":"2024-04-14T15:18:13.954725+0000","last_fullsized":"2024-04-14T15:18:13.954725+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:56.206649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:56.206649+0000","last_clean_scrub_stamp":"2024-04-14T15:16:56.206649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"91'130","reported_seq":807,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.954086+0000","last_change":"2024-04-14T15:17:49.890530+0000","last_active":"2024-04-14T15:18:13.954086+0000","last_peered":"2024-04-14T15:18:13.954086+0000","last_clean":"2024-04-14T15:18:13.954086+0000","last_became_active":"2024-04-14T15:14:17.856694+0000","last_became_peered":"2024-04-14T15:14:17.856694+0000","last_unstale":"2024-04-14T15:18:13.954086+0000","last_undegraded":"2024-04-14T15:18:13.954086+0000","last_fullsized":"2024-04-14T15:18:13.954086+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:17:49.890445+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:17:49.890445+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:16.061587+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":349,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.962647+0000","last_change":"2024-04-14T15:15:31.812967+0000","last_active":"2024-04-14T15:18:13.962647+0000","last_peered":"2024-04-14T15:18:13.962647+0000","last_clean":"2024-04-14T15:18:13.962647+0000","last_became_active":"2024-04-14T15:15:21.887769+0000","last_became_peered":"2024-04-14T15:15:21.887769+0000","last_unstale":"2024-04-14T15:18:13.962647+0000","last_undegraded":"2024-04-14T15:18:13.962647+0000","last_fullsized":"2024-04-14T15:18:13.962647+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:31.812888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:31.812888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.358900+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":307,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.957341+0000","last_change":"2024-04-14T15:17:47.871801+0000","last_active":"2024-04-14T15:18:13.957341+0000","last_peered":"2024-04-14T15:18:13.957341+0000","last_clean":"2024-04-14T15:18:13.957341+0000","last_became_active":"2024-04-14T15:15:44.883694+0000","last_became_peered":"2024-04-14T15:15:44.883694+0000","last_unstale":"2024-04-14T15:18:13.957341+0000","last_undegraded":"2024-04-14T15:18:13.957341+0000","last_fullsized":"2024-04-14T15:18:13.957341+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:47.871722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:17:47.871722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:15.357681+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"182'18","reported_seq":776,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.958135+0000","last_change":"2024-04-14T15:17:44.919445+0000","last_active":"2024-04-14T15:18:13.958135+0000","last_peered":"2024-04-14T15:18:13.958135+0000","last_clean":"2024-04-14T15:18:13.958135+0000","last_became_active":"2024-04-14T15:09:45.037231+0000","last_became_peered":"2024-04-14T15:09:45.037231+0000","last_unstale":"2024-04-14T15:18:13.958135+0000","last_undegraded":"2024-04-14T15:18:13.958135+0000","last_fullsized":"2024-04-14T15:18:13.958135+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'18","last_scrub_stamp":"2024-04-14T15:17:44.919358+0000","last_deep_scrub":"182'18","last_deep_scrub_stamp":"2024-04-14T15:14:10.383906+0000","last_clean_scrub_stamp":"2024-04-14T15:17:44.919358+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:48.791608+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":753,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-14T15:18:12.942058+0000","last_change":"2024-04-14T15:17:46.542125+0000","last_active":"2024-04-14T15:18:12.942058+0000","last_peered":"2024-04-14T15:18:12.942058+0000","last_clean":"2024-04-14T15:18:12.942058+0000","last_became_active":"2024-04-14T15:09:43.025095+0000","last_became_peered":"2024-04-14T15:09:43.025095+0000","last_unstale":"2024-04-14T15:18:12.942058+0000","last_undegraded":"2024-04-14T15:18:12.942058+0000","last_fullsized":"2024-04-14T15:18:12.942058+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:46.542030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:46.542030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:59.318501+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":764,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.955462+0000","last_change":"2024-04-14T15:17:48.866222+0000","last_active":"2024-04-14T15:18:13.955462+0000","last_peered":"2024-04-14T15:18:13.955462+0000","last_clean":"2024-04-14T15:18:13.955462+0000","last_became_active":"2024-04-14T15:09:43.018509+0000","last_became_peered":"2024-04-14T15:09:43.018509+0000","last_unstale":"2024-04-14T15:18:13.955462+0000","last_undegraded":"2024-04-14T15:18:13.955462+0000","last_fullsized":"2024-04-14T15:18:13.955462+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:48.866146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:48.866146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:57.582404+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0018592680000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"355'2","reported_seq":739,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.955747+0000","last_change":"2024-04-14T15:17:03.127190+0000","last_active":"2024-04-14T15:18:13.955747+0000","last_peered":"2024-04-14T15:18:13.955747+0000","last_clean":"2024-04-14T15:18:13.955747+0000","last_became_active":"2024-04-14T15:09:49.065243+0000","last_became_peered":"2024-04-14T15:09:49.065243+0000","last_unstale":"2024-04-14T15:18:13.955747+0000","last_undegraded":"2024-04-14T15:18:13.955747+0000","last_fullsized":"2024-04-14T15:18:13.955747+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'2","last_scrub_stamp":"2024-04-14T15:17:03.127112+0000","last_deep_scrub":"355'2","last_deep_scrub_stamp":"2024-04-14T15:17:03.127112+0000","last_clean_scrub_stamp":"2024-04-14T15:17:03.127112+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.009834+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":341,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.963408+0000","last_change":"2024-04-14T15:15:21.890855+0000","last_active":"2024-04-14T15:18:13.963408+0000","last_peered":"2024-04-14T15:18:13.963408+0000","last_clean":"2024-04-14T15:18:13.963408+0000","last_became_active":"2024-04-14T15:15:21.890115+0000","last_became_peered":"2024-04-14T15:15:21.890115+0000","last_unstale":"2024-04-14T15:18:13.963408+0000","last_undegraded":"2024-04-14T15:18:13.963408+0000","last_fullsized":"2024-04-14T15:18:13.963408+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:45.148342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:14:45.148342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.358831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":737,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-14T15:18:12.945824+0000","last_change":"2024-04-14T15:16:51.617717+0000","last_active":"2024-04-14T15:18:12.945824+0000","last_peered":"2024-04-14T15:18:12.945824+0000","last_clean":"2024-04-14T15:18:12.945824+0000","last_became_active":"2024-04-14T15:09:47.047201+0000","last_became_peered":"2024-04-14T15:09:47.047201+0000","last_unstale":"2024-04-14T15:18:12.945824+0000","last_undegraded":"2024-04-14T15:18:12.945824+0000","last_fullsized":"2024-04-14T15:18:12.945824+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:51.617615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:51.617615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.661149+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"90'24","reported_seq":877,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.964227+0000","last_change":"2024-04-14T15:16:19.842719+0000","last_active":"2024-04-14T15:18:13.964227+0000","last_peered":"2024-04-14T15:18:13.964227+0000","last_clean":"2024-04-14T15:18:13.964227+0000","last_became_active":"2024-04-14T15:09:45.179978+0000","last_became_peered":"2024-04-14T15:09:45.179978+0000","last_unstale":"2024-04-14T15:18:13.964227+0000","last_undegraded":"2024-04-14T15:18:13.964227+0000","last_fullsized":"2024-04-14T15:18:13.964227+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'24","last_scrub_stamp":"2024-04-14T15:16:19.842614+0000","last_deep_scrub":"90'24","last_deep_scrub_stamp":"2024-04-14T15:11:25.120692+0000","last_clean_scrub_stamp":"2024-04-14T15:16:19.842614+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"91'130","reported_seq":1015,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-14T15:18:12.942056+0000","last_change":"2024-04-14T15:16:43.730573+0000","last_active":"2024-04-14T15:18:12.942056+0000","last_peered":"2024-04-14T15:18:12.942056+0000","last_clean":"2024-04-14T15:18:12.942056+0000","last_became_active":"2024-04-14T15:14:12.858953+0000","last_became_peered":"2024-04-14T15:14:12.858953+0000","last_unstale":"2024-04-14T15:18:12.942056+0000","last_undegraded":"2024-04-14T15:18:12.942056+0000","last_fullsized":"2024-04-14T15:18:12.942056+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:16:43.730442+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:16:43.730442+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":312,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.959941+0000","last_change":"2024-04-14T15:17:35.705777+0000","last_active":"2024-04-14T15:18:13.959941+0000","last_peered":"2024-04-14T15:18:13.959941+0000","last_clean":"2024-04-14T15:18:13.959941+0000","last_became_active":"2024-04-14T15:15:44.885315+0000","last_became_peered":"2024-04-14T15:15:44.885315+0000","last_unstale":"2024-04-14T15:18:13.959941+0000","last_undegraded":"2024-04-14T15:18:13.959941+0000","last_fullsized":"2024-04-14T15:18:13.959941+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:35.705645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:47.627305+0000","last_clean_scrub_stamp":"2024-04-14T15:17:35.705645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:49.306972+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"69'2","reported_seq":439,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.963619+0000","last_change":"2024-04-14T15:16:53.972591+0000","last_active":"2024-04-14T15:18:13.963619+0000","last_peered":"2024-04-14T15:18:13.963619+0000","last_clean":"2024-04-14T15:18:13.963619+0000","last_became_active":"2024-04-14T15:15:21.889851+0000","last_became_peered":"2024-04-14T15:15:21.889851+0000","last_unstale":"2024-04-14T15:18:13.963619+0000","last_undegraded":"2024-04-14T15:18:13.963619+0000","last_fullsized":"2024-04-14T15:18:13.963619+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-04-14T15:16:53.972502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:53.972502+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:20.876593+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0058046169999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":304,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.964300+0000","last_change":"2024-04-14T15:16:57.935611+0000","last_active":"2024-04-14T15:18:13.964300+0000","last_peered":"2024-04-14T15:18:13.964300+0000","last_clean":"2024-04-14T15:18:13.964300+0000","last_became_active":"2024-04-14T15:15:44.887934+0000","last_became_peered":"2024-04-14T15:15:44.887934+0000","last_unstale":"2024-04-14T15:18:13.964300+0000","last_undegraded":"2024-04-14T15:18:13.964300+0000","last_fullsized":"2024-04-14T15:18:13.964300+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:57.935436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:57.935436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.359186+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"91'130","reported_seq":1022,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-14T15:18:12.944070+0000","last_change":"2024-04-14T15:17:41.509304+0000","last_active":"2024-04-14T15:18:12.944070+0000","last_peered":"2024-04-14T15:18:12.944070+0000","last_clean":"2024-04-14T15:18:12.944070+0000","last_became_active":"2024-04-14T15:14:17.858029+0000","last_became_peered":"2024-04-14T15:14:17.858029+0000","last_unstale":"2024-04-14T15:18:12.944070+0000","last_undegraded":"2024-04-14T15:18:12.944070+0000","last_fullsized":"2024-04-14T15:18:12.944070+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:17:41.509215+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:17:41.509215+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:05.623460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":341,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.962433+0000","last_change":"2024-04-14T15:15:21.890274+0000","last_active":"2024-04-14T15:18:13.962433+0000","last_peered":"2024-04-14T15:18:13.962433+0000","last_clean":"2024-04-14T15:18:13.962433+0000","last_became_active":"2024-04-14T15:15:21.889970+0000","last_became_peered":"2024-04-14T15:15:21.889970+0000","last_unstale":"2024-04-14T15:18:13.962433+0000","last_undegraded":"2024-04-14T15:18:13.962433+0000","last_fullsized":"2024-04-14T15:18:13.962433+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:54.156421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:14:54.156421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.358796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":745,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.960297+0000","last_change":"2024-04-14T15:17:01.653342+0000","last_active":"2024-04-14T15:18:13.960297+0000","last_peered":"2024-04-14T15:18:13.960297+0000","last_clean":"2024-04-14T15:18:13.960297+0000","last_became_active":"2024-04-14T15:09:47.049457+0000","last_became_peered":"2024-04-14T15:09:47.049457+0000","last_unstale":"2024-04-14T15:18:13.960297+0000","last_undegraded":"2024-04-14T15:18:13.960297+0000","last_fullsized":"2024-04-14T15:18:13.960297+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:01.653271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:17:01.653271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:29.819865+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"308'18","reported_seq":776,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.957283+0000","last_change":"2024-04-14T15:16:16.054128+0000","last_active":"2024-04-14T15:18:13.957283+0000","last_peered":"2024-04-14T15:18:13.957283+0000","last_clean":"2024-04-14T15:18:13.957283+0000","last_became_active":"2024-04-14T15:09:45.033883+0000","last_became_peered":"2024-04-14T15:09:45.033883+0000","last_unstale":"2024-04-14T15:18:13.957283+0000","last_undegraded":"2024-04-14T15:18:13.957283+0000","last_fullsized":"2024-04-14T15:18:13.957283+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"308'18","last_scrub_stamp":"2024-04-14T15:16:16.054032+0000","last_deep_scrub":"308'18","last_deep_scrub_stamp":"2024-04-14T15:16:16.054032+0000","last_clean_scrub_stamp":"2024-04-14T15:16:16.054032+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":754,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.959209+0000","last_change":"2024-04-14T15:16:43.686142+0000","last_active":"2024-04-14T15:18:13.959209+0000","last_peered":"2024-04-14T15:18:13.959209+0000","last_clean":"2024-04-14T15:18:13.959209+0000","last_became_active":"2024-04-14T15:09:43.021188+0000","last_became_peered":"2024-04-14T15:09:43.021188+0000","last_unstale":"2024-04-14T15:18:13.959209+0000","last_undegraded":"2024-04-14T15:18:13.959209+0000","last_fullsized":"2024-04-14T15:18:13.959209+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:43.686033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:43.686033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:13.986089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"73'5","reported_seq":1263,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.957372+0000","last_change":"2024-04-14T15:15:28.328150+0000","last_active":"2024-04-14T15:18:13.957372+0000","last_peered":"2024-04-14T15:18:13.957372+0000","last_clean":"2024-04-14T15:18:13.957372+0000","last_became_active":"2024-04-14T15:09:43.016544+0000","last_became_peered":"2024-04-14T15:09:43.016544+0000","last_unstale":"2024-04-14T15:18:13.957372+0000","last_undegraded":"2024-04-14T15:18:13.957372+0000","last_fullsized":"2024-04-14T15:18:13.957372+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-04-14T15:15:28.328050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:15:28.328050+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00056542099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":504,"num_read_kb":499,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":733,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-14T15:18:12.947045+0000","last_change":"2024-04-14T15:16:20.827681+0000","last_active":"2024-04-14T15:18:12.947045+0000","last_peered":"2024-04-14T15:18:12.947045+0000","last_clean":"2024-04-14T15:18:12.947045+0000","last_became_active":"2024-04-14T15:09:49.130307+0000","last_became_peered":"2024-04-14T15:09:49.130307+0000","last_unstale":"2024-04-14T15:18:12.947045+0000","last_undegraded":"2024-04-14T15:18:12.947045+0000","last_fullsized":"2024-04-14T15:18:12.947045+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:20.827589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:20.827589+0000","last_clean_scrub_stamp":"2024-04-14T15:16:20.827589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.661118+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"91'130","reported_seq":631,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.954754+0000","last_change":"2024-04-14T15:15:45.362962+0000","last_active":"2024-04-14T15:18:13.954754+0000","last_peered":"2024-04-14T15:18:13.954754+0000","last_clean":"2024-04-14T15:18:13.954754+0000","last_became_active":"2024-04-14T15:15:44.884623+0000","last_became_peered":"2024-04-14T15:15:44.884623+0000","last_unstale":"2024-04-14T15:18:13.954754+0000","last_undegraded":"2024-04-14T15:18:13.954754+0000","last_fullsized":"2024-04-14T15:18:13.954754+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:15:45.362860+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:15:45.362860+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":349,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.963522+0000","last_change":"2024-04-14T15:15:32.839576+0000","last_active":"2024-04-14T15:18:13.963522+0000","last_peered":"2024-04-14T15:18:13.963522+0000","last_clean":"2024-04-14T15:18:13.963522+0000","last_became_active":"2024-04-14T15:15:21.883717+0000","last_became_peered":"2024-04-14T15:15:21.883717+0000","last_unstale":"2024-04-14T15:18:13.963522+0000","last_undegraded":"2024-04-14T15:18:13.963522+0000","last_fullsized":"2024-04-14T15:18:13.963522+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:32.839491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:32.839491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.358965+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":741,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.957813+0000","last_change":"2024-04-14T15:16:53.731699+0000","last_active":"2024-04-14T15:18:13.957813+0000","last_peered":"2024-04-14T15:18:13.957813+0000","last_clean":"2024-04-14T15:18:13.957813+0000","last_became_active":"2024-04-14T15:09:47.040185+0000","last_became_peered":"2024-04-14T15:09:47.040185+0000","last_unstale":"2024-04-14T15:18:13.957813+0000","last_undegraded":"2024-04-14T15:18:13.957813+0000","last_fullsized":"2024-04-14T15:18:13.957813+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:53.731610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:53.731610+0000","last_clean_scrub_stamp":"2024-04-14T15:16:53.731610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"72'28","reported_seq":896,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.954367+0000","last_change":"2024-04-14T15:16:43.036263+0000","last_active":"2024-04-14T15:18:13.954367+0000","last_peered":"2024-04-14T15:18:13.954367+0000","last_clean":"2024-04-14T15:18:13.954367+0000","last_became_active":"2024-04-14T15:09:45.034168+0000","last_became_peered":"2024-04-14T15:09:45.034168+0000","last_unstale":"2024-04-14T15:18:13.954367+0000","last_undegraded":"2024-04-14T15:18:13.954367+0000","last_fullsized":"2024-04-14T15:18:13.954367+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'28","last_scrub_stamp":"2024-04-14T15:16:43.036159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:16:43.036159+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":307,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.953910+0000","last_change":"2024-04-14T15:16:19.101437+0000","last_active":"2024-04-14T15:18:13.953910+0000","last_peered":"2024-04-14T15:18:13.953910+0000","last_clean":"2024-04-14T15:18:13.953910+0000","last_became_active":"2024-04-14T15:15:44.885315+0000","last_became_peered":"2024-04-14T15:15:44.885315+0000","last_unstale":"2024-04-14T15:18:13.953910+0000","last_undegraded":"2024-04-14T15:18:13.953910+0000","last_fullsized":"2024-04-14T15:18:13.953910+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:19.101368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:19.101368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":409,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.963780+0000","last_change":"2024-04-14T15:17:44.084142+0000","last_active":"2024-04-14T15:18:13.963780+0000","last_peered":"2024-04-14T15:18:13.963780+0000","last_clean":"2024-04-14T15:18:13.963780+0000","last_became_active":"2024-04-14T15:15:21.874742+0000","last_became_peered":"2024-04-14T15:15:21.874742+0000","last_unstale":"2024-04-14T15:18:13.963780+0000","last_undegraded":"2024-04-14T15:18:13.963780+0000","last_fullsized":"2024-04-14T15:18:13.963780+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:44.084010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:44.084010+0000","last_clean_scrub_stamp":"2024-04-14T15:17:44.084010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:03.733939+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"91'130","reported_seq":813,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.964539+0000","last_change":"2024-04-14T15:17:40.725976+0000","last_active":"2024-04-14T15:18:13.964539+0000","last_peered":"2024-04-14T15:18:13.964539+0000","last_clean":"2024-04-14T15:18:13.964539+0000","last_became_active":"2024-04-14T15:14:17.859910+0000","last_became_peered":"2024-04-14T15:14:17.859910+0000","last_unstale":"2024-04-14T15:18:13.964539+0000","last_undegraded":"2024-04-14T15:18:13.964539+0000","last_fullsized":"2024-04-14T15:18:13.964539+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:17:40.725873+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:17:40.725873+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:54.360389+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":650,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.959929+0000","last_change":"2024-04-14T15:17:00.656174+0000","last_active":"2024-04-14T15:18:13.959929+0000","last_peered":"2024-04-14T15:18:13.959929+0000","last_clean":"2024-04-14T15:18:13.959929+0000","last_became_active":"2024-04-14T15:12:56.350703+0000","last_became_peered":"2024-04-14T15:12:56.350703+0000","last_unstale":"2024-04-14T15:18:13.959929+0000","last_undegraded":"2024-04-14T15:18:13.959929+0000","last_fullsized":"2024-04-14T15:18:13.959929+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:00.656085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:17:00.656085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"72'8","reported_seq":860,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.955709+0000","last_change":"2024-04-14T15:17:04.695161+0000","last_active":"2024-04-14T15:18:13.955709+0000","last_peered":"2024-04-14T15:18:13.955709+0000","last_clean":"2024-04-14T15:18:13.955709+0000","last_became_active":"2024-04-14T15:09:47.047699+0000","last_became_peered":"2024-04-14T15:09:47.047699+0000","last_unstale":"2024-04-14T15:18:13.955709+0000","last_undegraded":"2024-04-14T15:18:13.955709+0000","last_fullsized":"2024-04-14T15:18:13.955709+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:17:04.695079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:17:04.695079+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:15.967413+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"72'10","reported_seq":735,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.959557+0000","last_change":"2024-04-14T15:15:41.610385+0000","last_active":"2024-04-14T15:18:13.959557+0000","last_peered":"2024-04-14T15:18:13.959557+0000","last_clean":"2024-04-14T15:18:13.959557+0000","last_became_active":"2024-04-14T15:10:07.764042+0000","last_became_peered":"2024-04-14T15:10:07.764042+0000","last_unstale":"2024-04-14T15:18:13.959557+0000","last_undegraded":"2024-04-14T15:18:13.959557+0000","last_fullsized":"2024-04-14T15:18:13.959557+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-04-14T15:15:41.610279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:15:41.610279+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:13.986003+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":749,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.954280+0000","last_change":"2024-04-14T15:17:40.971492+0000","last_active":"2024-04-14T15:18:13.954280+0000","last_peered":"2024-04-14T15:18:13.954280+0000","last_clean":"2024-04-14T15:18:13.954280+0000","last_became_active":"2024-04-14T15:09:43.022597+0000","last_became_peered":"2024-04-14T15:09:43.022597+0000","last_unstale":"2024-04-14T15:18:13.954280+0000","last_undegraded":"2024-04-14T15:18:13.954280+0000","last_fullsized":"2024-04-14T15:18:13.954280+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:40.971412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:40.971412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:08.233588+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":737,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.953929+0000","last_change":"2024-04-14T15:16:17.405931+0000","last_active":"2024-04-14T15:18:13.953929+0000","last_peered":"2024-04-14T15:18:13.953929+0000","last_clean":"2024-04-14T15:18:13.953929+0000","last_became_active":"2024-04-14T15:09:49.066648+0000","last_became_peered":"2024-04-14T15:09:49.066648+0000","last_unstale":"2024-04-14T15:18:13.953929+0000","last_undegraded":"2024-04-14T15:18:13.953929+0000","last_fullsized":"2024-04-14T15:18:13.953929+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:17.405850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:17.405850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":645,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.957979+0000","last_change":"2024-04-14T15:15:36.403919+0000","last_active":"2024-04-14T15:18:13.957979+0000","last_peered":"2024-04-14T15:18:13.957979+0000","last_clean":"2024-04-14T15:18:13.957979+0000","last_became_active":"2024-04-14T15:12:56.563074+0000","last_became_peered":"2024-04-14T15:12:56.563074+0000","last_unstale":"2024-04-14T15:18:13.957979+0000","last_undegraded":"2024-04-14T15:18:13.957979+0000","last_fullsized":"2024-04-14T15:18:13.957979+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:36.403843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:36.403843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":745,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.958809+0000","last_change":"2024-04-14T15:17:56.915660+0000","last_active":"2024-04-14T15:18:13.958809+0000","last_peered":"2024-04-14T15:18:13.958809+0000","last_clean":"2024-04-14T15:18:13.958809+0000","last_became_active":"2024-04-14T15:09:47.049005+0000","last_became_peered":"2024-04-14T15:09:47.049005+0000","last_unstale":"2024-04-14T15:18:13.958809+0000","last_undegraded":"2024-04-14T15:18:13.958809+0000","last_fullsized":"2024-04-14T15:18:13.958809+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:56.915544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:56.915544+0000","last_clean_scrub_stamp":"2024-04-14T15:17:56.915544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:24.542571+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"183'17","reported_seq":778,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.952452+0000","last_change":"2024-04-14T15:16:17.997616+0000","last_active":"2024-04-14T15:18:13.952452+0000","last_peered":"2024-04-14T15:18:13.952452+0000","last_clean":"2024-04-14T15:18:13.952452+0000","last_became_active":"2024-04-14T15:09:45.032892+0000","last_became_peered":"2024-04-14T15:09:45.032892+0000","last_unstale":"2024-04-14T15:18:13.952452+0000","last_undegraded":"2024-04-14T15:18:13.952452+0000","last_fullsized":"2024-04-14T15:18:13.952452+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'17","last_scrub_stamp":"2024-04-14T15:16:17.997529+0000","last_deep_scrub":"183'17","last_deep_scrub_stamp":"2024-04-14T15:14:11.577574+0000","last_clean_scrub_stamp":"2024-04-14T15:16:17.997529+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:16.453860+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"65'3","reported_seq":764,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.953609+0000","last_change":"2024-04-14T15:16:12.030448+0000","last_active":"2024-04-14T15:18:13.953609+0000","last_peered":"2024-04-14T15:18:13.953609+0000","last_clean":"2024-04-14T15:18:13.953609+0000","last_became_active":"2024-04-14T15:09:43.020490+0000","last_became_peered":"2024-04-14T15:09:43.020490+0000","last_unstale":"2024-04-14T15:18:13.953609+0000","last_undegraded":"2024-04-14T15:18:13.953609+0000","last_fullsized":"2024-04-14T15:18:13.953609+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3","last_scrub_stamp":"2024-04-14T15:16:12.030363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:12.030363+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:16.453828+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":528,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":409,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.963884+0000","last_change":"2024-04-14T15:16:58.942591+0000","last_active":"2024-04-14T15:18:13.963884+0000","last_peered":"2024-04-14T15:18:13.963884+0000","last_clean":"2024-04-14T15:18:13.963884+0000","last_became_active":"2024-04-14T15:15:21.876410+0000","last_became_peered":"2024-04-14T15:15:21.876410+0000","last_unstale":"2024-04-14T15:18:13.963884+0000","last_undegraded":"2024-04-14T15:18:13.963884+0000","last_fullsized":"2024-04-14T15:18:13.963884+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:58.942499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:58.942499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.359220+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":653,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.955309+0000","last_change":"2024-04-14T15:16:43.696614+0000","last_active":"2024-04-14T15:18:13.955309+0000","last_peered":"2024-04-14T15:18:13.955309+0000","last_clean":"2024-04-14T15:18:13.955309+0000","last_became_active":"2024-04-14T15:12:56.551177+0000","last_became_peered":"2024-04-14T15:12:56.551177+0000","last_unstale":"2024-04-14T15:18:13.955309+0000","last_undegraded":"2024-04-14T15:18:13.955309+0000","last_fullsized":"2024-04-14T15:18:13.955309+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:43.696536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:16:43.696536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"347'13","reported_seq":764,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.958223+0000","last_change":"2024-04-14T15:16:55.694703+0000","last_active":"2024-04-14T15:18:13.958223+0000","last_peered":"2024-04-14T15:18:13.958223+0000","last_clean":"2024-04-14T15:18:13.958223+0000","last_became_active":"2024-04-14T15:09:45.180375+0000","last_became_peered":"2024-04-14T15:09:45.180375+0000","last_unstale":"2024-04-14T15:18:13.958223+0000","last_undegraded":"2024-04-14T15:18:13.958223+0000","last_fullsized":"2024-04-14T15:18:13.958223+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"347'13","last_scrub_stamp":"2024-04-14T15:16:55.694599+0000","last_deep_scrub":"347'13","last_deep_scrub_stamp":"2024-04-14T15:16:55.694599+0000","last_clean_scrub_stamp":"2024-04-14T15:16:55.694599+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:15.967396+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":753,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.958776+0000","last_change":"2024-04-14T15:16:41.677902+0000","last_active":"2024-04-14T15:18:13.958776+0000","last_peered":"2024-04-14T15:18:13.958776+0000","last_clean":"2024-04-14T15:18:13.958776+0000","last_became_active":"2024-04-14T15:09:43.016788+0000","last_became_peered":"2024-04-14T15:09:43.016788+0000","last_unstale":"2024-04-14T15:18:13.958776+0000","last_undegraded":"2024-04-14T15:18:13.958776+0000","last_fullsized":"2024-04-14T15:18:13.958776+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:41.677826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:41.677826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:13.986106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":421,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.963934+0000","last_change":"2024-04-14T15:15:42.769452+0000","last_active":"2024-04-14T15:18:13.963934+0000","last_peered":"2024-04-14T15:18:13.963934+0000","last_clean":"2024-04-14T15:18:13.963934+0000","last_became_active":"2024-04-14T15:15:21.873580+0000","last_became_peered":"2024-04-14T15:15:21.873580+0000","last_unstale":"2024-04-14T15:18:13.963934+0000","last_undegraded":"2024-04-14T15:18:13.963934+0000","last_fullsized":"2024-04-14T15:18:13.963934+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:42.769375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:15:42.769375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.358933+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":741,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.960205+0000","last_change":"2024-04-14T15:16:47.671808+0000","last_active":"2024-04-14T15:18:13.960205+0000","last_peered":"2024-04-14T15:18:13.960205+0000","last_clean":"2024-04-14T15:18:13.960205+0000","last_became_active":"2024-04-14T15:09:49.064296+0000","last_became_peered":"2024-04-14T15:09:49.064296+0000","last_unstale":"2024-04-14T15:18:13.960205+0000","last_undegraded":"2024-04-14T15:18:13.960205+0000","last_fullsized":"2024-04-14T15:18:13.960205+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:47.671717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:47.671717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:13.986123+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":650,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.958689+0000","last_change":"2024-04-14T15:17:36.728543+0000","last_active":"2024-04-14T15:18:13.958689+0000","last_peered":"2024-04-14T15:18:13.958689+0000","last_clean":"2024-04-14T15:18:13.958689+0000","last_became_active":"2024-04-14T15:12:56.340066+0000","last_became_peered":"2024-04-14T15:12:56.340066+0000","last_unstale":"2024-04-14T15:18:13.958689+0000","last_undegraded":"2024-04-14T15:18:13.958689+0000","last_fullsized":"2024-04-14T15:18:13.958689+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:36.728465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:17:36.728465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:54.134352+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.17","version":"72'6","reported_seq":757,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.963838+0000","last_change":"2024-04-14T15:17:42.019838+0000","last_active":"2024-04-14T15:18:13.963838+0000","last_peered":"2024-04-14T15:18:13.963838+0000","last_clean":"2024-04-14T15:18:13.963838+0000","last_became_active":"2024-04-14T15:09:45.180037+0000","last_became_peered":"2024-04-14T15:09:45.180037+0000","last_unstale":"2024-04-14T15:18:13.963838+0000","last_undegraded":"2024-04-14T15:18:13.963838+0000","last_fullsized":"2024-04-14T15:18:13.963838+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'6","last_scrub_stamp":"2024-04-14T15:17:42.019717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:17:42.019717+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:52.276684+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":755,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.963605+0000","last_change":"2024-04-14T15:16:59.640700+0000","last_active":"2024-04-14T15:18:13.963605+0000","last_peered":"2024-04-14T15:18:13.963605+0000","last_clean":"2024-04-14T15:18:13.963605+0000","last_became_active":"2024-04-14T15:09:43.018987+0000","last_became_peered":"2024-04-14T15:09:43.018987+0000","last_unstale":"2024-04-14T15:18:13.963605+0000","last_undegraded":"2024-04-14T15:18:13.963605+0000","last_fullsized":"2024-04-14T15:18:13.963605+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:59.640565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:59.640565+0000","last_clean_scrub_stamp":"2024-04-14T15:16:59.640565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:28.007249+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":303,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.963659+0000","last_change":"2024-04-14T15:17:47.072247+0000","last_active":"2024-04-14T15:18:13.963659+0000","last_peered":"2024-04-14T15:18:13.963659+0000","last_clean":"2024-04-14T15:18:13.963659+0000","last_became_active":"2024-04-14T15:15:44.889786+0000","last_became_peered":"2024-04-14T15:15:44.889786+0000","last_unstale":"2024-04-14T15:18:13.963659+0000","last_undegraded":"2024-04-14T15:18:13.963659+0000","last_fullsized":"2024-04-14T15:18:13.963659+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:47.072170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:13.735024+0000","last_clean_scrub_stamp":"2024-04-14T15:17:47.072170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:03.533500+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":732,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-14T15:18:12.947095+0000","last_change":"2024-04-14T15:16:54.720823+0000","last_active":"2024-04-14T15:18:12.947095+0000","last_peered":"2024-04-14T15:18:12.947095+0000","last_clean":"2024-04-14T15:18:12.947095+0000","last_became_active":"2024-04-14T15:09:49.130438+0000","last_became_peered":"2024-04-14T15:09:49.130438+0000","last_unstale":"2024-04-14T15:18:12.947095+0000","last_undegraded":"2024-04-14T15:18:12.947095+0000","last_fullsized":"2024-04-14T15:18:12.947095+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:54.720722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:54.720722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":651,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.952566+0000","last_change":"2024-04-14T15:17:46.369320+0000","last_active":"2024-04-14T15:18:13.952566+0000","last_peered":"2024-04-14T15:18:13.952566+0000","last_clean":"2024-04-14T15:18:13.952566+0000","last_became_active":"2024-04-14T15:12:56.203577+0000","last_became_peered":"2024-04-14T15:12:56.203577+0000","last_unstale":"2024-04-14T15:18:13.952566+0000","last_undegraded":"2024-04-14T15:18:13.952566+0000","last_fullsized":"2024-04-14T15:18:13.952566+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:46.369231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:46.369231+0000","last_clean_scrub_stamp":"2024-04-14T15:17:46.369231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:57.370694+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"72'10","reported_seq":440,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.963210+0000","last_change":"2024-04-14T15:15:30.806041+0000","last_active":"2024-04-14T15:18:13.963210+0000","last_peered":"2024-04-14T15:18:13.963210+0000","last_clean":"2024-04-14T15:18:13.963210+0000","last_became_active":"2024-04-14T15:15:21.877667+0000","last_became_peered":"2024-04-14T15:15:21.877667+0000","last_unstale":"2024-04-14T15:18:13.963210+0000","last_undegraded":"2024-04-14T15:18:13.963210+0000","last_fullsized":"2024-04-14T15:18:13.963210+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-04-14T15:15:30.805923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:15:30.805923+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.358866+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":303,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.963900+0000","last_change":"2024-04-14T15:16:47.075044+0000","last_active":"2024-04-14T15:18:13.963900+0000","last_peered":"2024-04-14T15:18:13.963900+0000","last_clean":"2024-04-14T15:18:13.963900+0000","last_became_active":"2024-04-14T15:15:44.887452+0000","last_became_peered":"2024-04-14T15:15:44.887452+0000","last_unstale":"2024-04-14T15:18:13.963900+0000","last_undegraded":"2024-04-14T15:18:13.963900+0000","last_fullsized":"2024-04-14T15:18:13.963900+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:47.074950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:47.074950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.359135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"280'9","reported_seq":362,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.956900+0000","last_change":"2024-04-14T15:17:50.908130+0000","last_active":"2024-04-14T15:18:13.956900+0000","last_peered":"2024-04-14T15:18:13.956900+0000","last_clean":"2024-04-14T15:18:13.956900+0000","last_became_active":"2024-04-14T15:15:44.882916+0000","last_became_peered":"2024-04-14T15:15:44.882916+0000","last_unstale":"2024-04-14T15:18:13.956900+0000","last_undegraded":"2024-04-14T15:18:13.956900+0000","last_fullsized":"2024-04-14T15:18:13.956900+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"280'9","last_scrub_stamp":"2024-04-14T15:17:50.908054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:17:50.908054+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:20.243489+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":736,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.958126+0000","last_change":"2024-04-14T15:16:54.724807+0000","last_active":"2024-04-14T15:18:13.958126+0000","last_peered":"2024-04-14T15:18:13.958126+0000","last_clean":"2024-04-14T15:18:13.958126+0000","last_became_active":"2024-04-14T15:09:49.066068+0000","last_became_peered":"2024-04-14T15:09:49.066068+0000","last_unstale":"2024-04-14T15:18:13.958126+0000","last_undegraded":"2024-04-14T15:18:13.958126+0000","last_fullsized":"2024-04-14T15:18:13.958126+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:54.724681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:54.724681+0000","last_clean_scrub_stamp":"2024-04-14T15:16:54.724681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:17.046734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":295,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.963269+0000","last_change":"2024-04-14T15:15:44.889470+0000","last_active":"2024-04-14T15:18:13.963269+0000","last_peered":"2024-04-14T15:18:13.963269+0000","last_clean":"2024-04-14T15:18:13.963269+0000","last_became_active":"2024-04-14T15:15:44.889106+0000","last_became_peered":"2024-04-14T15:15:44.889106+0000","last_unstale":"2024-04-14T15:18:13.963269+0000","last_undegraded":"2024-04-14T15:18:13.963269+0000","last_fullsized":"2024-04-14T15:18:13.963269+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:01.356001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:01.356001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.358815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"183'13","reported_seq":582,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-14T15:18:12.947036+0000","last_change":"2024-04-14T15:17:37.421888+0000","last_active":"2024-04-14T15:18:12.947036+0000","last_peered":"2024-04-14T15:18:12.947036+0000","last_clean":"2024-04-14T15:18:12.947036+0000","last_became_active":"2024-04-14T15:15:44.884211+0000","last_became_peered":"2024-04-14T15:15:44.884211+0000","last_unstale":"2024-04-14T15:18:12.947036+0000","last_undegraded":"2024-04-14T15:18:12.947036+0000","last_fullsized":"2024-04-14T15:18:12.947036+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'13","last_scrub_stamp":"2024-04-14T15:17:37.421739+0000","last_deep_scrub":"183'13","last_deep_scrub_stamp":"2024-04-14T15:14:11.755252+0000","last_clean_scrub_stamp":"2024-04-14T15:17:37.421739+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:55.179831+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":756,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.955902+0000","last_change":"2024-04-14T15:17:02.670294+0000","last_active":"2024-04-14T15:18:13.955902+0000","last_peered":"2024-04-14T15:18:13.955902+0000","last_clean":"2024-04-14T15:18:13.955902+0000","last_became_active":"2024-04-14T15:09:43.019301+0000","last_became_peered":"2024-04-14T15:09:43.019301+0000","last_unstale":"2024-04-14T15:18:13.955902+0000","last_undegraded":"2024-04-14T15:18:13.955902+0000","last_fullsized":"2024-04-14T15:18:13.955902+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:02.670213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:02.670213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:17.714709+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"255'10","reported_seq":495,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.963651+0000","last_change":"2024-04-14T15:16:59.946889+0000","last_active":"2024-04-14T15:18:13.963651+0000","last_peered":"2024-04-14T15:18:13.963651+0000","last_clean":"2024-04-14T15:18:13.963651+0000","last_became_active":"2024-04-14T15:15:21.884468+0000","last_became_peered":"2024-04-14T15:15:21.884468+0000","last_unstale":"2024-04-14T15:18:13.963651+0000","last_undegraded":"2024-04-14T15:18:13.963651+0000","last_fullsized":"2024-04-14T15:18:13.963651+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'10","last_scrub_stamp":"2024-04-14T15:16:59.946804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:59.946804+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.359169+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":734,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.953682+0000","last_change":"2024-04-14T15:16:03.948252+0000","last_active":"2024-04-14T15:18:13.953682+0000","last_peered":"2024-04-14T15:18:13.953682+0000","last_clean":"2024-04-14T15:18:13.953682+0000","last_became_active":"2024-04-14T15:09:49.065242+0000","last_became_peered":"2024-04-14T15:09:49.065242+0000","last_unstale":"2024-04-14T15:18:13.953682+0000","last_undegraded":"2024-04-14T15:18:13.953682+0000","last_fullsized":"2024-04-14T15:18:13.953682+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:03.948140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:03.948140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:16.453699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":641,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-14T15:18:12.947150+0000","last_change":"2024-04-14T15:15:34.555810+0000","last_active":"2024-04-14T15:18:12.947150+0000","last_peered":"2024-04-14T15:18:12.947150+0000","last_clean":"2024-04-14T15:18:12.947150+0000","last_became_active":"2024-04-14T15:12:56.389204+0000","last_became_peered":"2024-04-14T15:12:56.389204+0000","last_unstale":"2024-04-14T15:18:12.947150+0000","last_undegraded":"2024-04-14T15:18:12.947150+0000","last_fullsized":"2024-04-14T15:18:12.947150+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:34.555728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:34.555728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"344'13","reported_seq":274,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.964426+0000","last_change":"2024-04-14T15:16:52.738298+0000","last_active":"2024-04-14T15:18:13.964426+0000","last_peered":"2024-04-14T15:18:13.964426+0000","last_clean":"2024-04-14T15:18:13.964426+0000","last_became_active":"2024-04-14T15:16:24.993130+0000","last_became_peered":"2024-04-14T15:16:24.993130+0000","last_unstale":"2024-04-14T15:18:13.964426+0000","last_undegraded":"2024-04-14T15:18:13.964426+0000","last_fullsized":"2024-04-14T15:18:13.964426+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"344'13","last_scrub_stamp":"2024-04-14T15:16:52.738163+0000","last_deep_scrub":"344'13","last_deep_scrub_stamp":"2024-04-14T15:16:52.738163+0000","last_clean_scrub_stamp":"2024-04-14T15:16:52.738163+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":755,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.964136+0000","last_change":"2024-04-14T15:17:04.625913+0000","last_active":"2024-04-14T15:18:13.964136+0000","last_peered":"2024-04-14T15:18:13.964136+0000","last_clean":"2024-04-14T15:18:13.964136+0000","last_became_active":"2024-04-14T15:09:43.022480+0000","last_became_peered":"2024-04-14T15:09:43.022480+0000","last_unstale":"2024-04-14T15:18:13.964136+0000","last_undegraded":"2024-04-14T15:18:13.964136+0000","last_fullsized":"2024-04-14T15:18:13.964136+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:04.625824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:04.625824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:28.834815+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":413,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.963689+0000","last_change":"2024-04-14T15:15:21.885176+0000","last_active":"2024-04-14T15:18:13.963689+0000","last_peered":"2024-04-14T15:18:13.963689+0000","last_clean":"2024-04-14T15:18:13.963689+0000","last_became_active":"2024-04-14T15:15:21.882643+0000","last_became_peered":"2024-04-14T15:15:21.882643+0000","last_unstale":"2024-04-14T15:18:13.963689+0000","last_undegraded":"2024-04-14T15:18:13.963689+0000","last_fullsized":"2024-04-14T15:18:13.963689+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:54.252608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:14:54.252608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.358846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":739,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.953817+0000","last_change":"2024-04-14T15:17:58.899572+0000","last_active":"2024-04-14T15:18:13.953817+0000","last_peered":"2024-04-14T15:18:13.953817+0000","last_clean":"2024-04-14T15:18:13.953817+0000","last_became_active":"2024-04-14T15:09:49.059711+0000","last_became_peered":"2024-04-14T15:09:49.059711+0000","last_unstale":"2024-04-14T15:18:13.953817+0000","last_undegraded":"2024-04-14T15:18:13.953817+0000","last_fullsized":"2024-04-14T15:18:13.953817+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:53.219686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:14:53.219686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":652,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.964038+0000","last_change":"2024-04-14T15:17:51.921346+0000","last_active":"2024-04-14T15:18:13.964038+0000","last_peered":"2024-04-14T15:18:13.964038+0000","last_clean":"2024-04-14T15:18:13.964038+0000","last_became_active":"2024-04-14T15:12:56.396294+0000","last_became_peered":"2024-04-14T15:12:56.396294+0000","last_unstale":"2024-04-14T15:18:13.964038+0000","last_undegraded":"2024-04-14T15:18:13.964038+0000","last_fullsized":"2024-04-14T15:18:13.964038+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:51.921269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:17:51.921269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:54.899694+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"72'11","reported_seq":761,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.952737+0000","last_change":"2024-04-14T15:16:08.917185+0000","last_active":"2024-04-14T15:18:13.952737+0000","last_peered":"2024-04-14T15:18:13.952737+0000","last_clean":"2024-04-14T15:18:13.952737+0000","last_became_active":"2024-04-14T15:09:45.026065+0000","last_became_peered":"2024-04-14T15:09:45.026065+0000","last_unstale":"2024-04-14T15:18:13.952737+0000","last_undegraded":"2024-04-14T15:18:13.952737+0000","last_fullsized":"2024-04-14T15:18:13.952737+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-04-14T15:16:08.917092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:16:08.917092+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:16.453732+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":756,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.957042+0000","last_change":"2024-04-14T15:17:34.986735+0000","last_active":"2024-04-14T15:18:13.957042+0000","last_peered":"2024-04-14T15:18:13.957042+0000","last_clean":"2024-04-14T15:18:13.957042+0000","last_became_active":"2024-04-14T15:09:43.023583+0000","last_became_peered":"2024-04-14T15:09:43.023583+0000","last_unstale":"2024-04-14T15:18:13.957042+0000","last_undegraded":"2024-04-14T15:18:13.957042+0000","last_fullsized":"2024-04-14T15:18:13.957042+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:34.986655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:34.986655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:41.802704+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":740,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.954185+0000","last_change":"2024-04-14T15:17:00.205888+0000","last_active":"2024-04-14T15:18:13.954185+0000","last_peered":"2024-04-14T15:18:13.954185+0000","last_clean":"2024-04-14T15:18:13.954185+0000","last_became_active":"2024-04-14T15:09:47.041683+0000","last_became_peered":"2024-04-14T15:09:47.041683+0000","last_unstale":"2024-04-14T15:18:13.954185+0000","last_undegraded":"2024-04-14T15:18:13.954185+0000","last_fullsized":"2024-04-14T15:18:13.954185+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:00.205808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:08.486217+0000","last_clean_scrub_stamp":"2024-04-14T15:17:00.205808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.009817+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":417,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.962507+0000","last_change":"2024-04-14T15:16:50.021545+0000","last_active":"2024-04-14T15:18:13.962507+0000","last_peered":"2024-04-14T15:18:13.962507+0000","last_clean":"2024-04-14T15:18:13.962507+0000","last_became_active":"2024-04-14T15:15:21.889567+0000","last_became_peered":"2024-04-14T15:15:21.889567+0000","last_unstale":"2024-04-14T15:18:13.962507+0000","last_undegraded":"2024-04-14T15:18:13.962507+0000","last_fullsized":"2024-04-14T15:18:13.962507+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:50.021470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:50.021470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.359151+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":650,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.959426+0000","last_change":"2024-04-14T15:17:03.685135+0000","last_active":"2024-04-14T15:18:13.959426+0000","last_peered":"2024-04-14T15:18:13.959426+0000","last_clean":"2024-04-14T15:18:13.959426+0000","last_became_active":"2024-04-14T15:12:56.387723+0000","last_became_peered":"2024-04-14T15:12:56.387723+0000","last_unstale":"2024-04-14T15:18:13.959426+0000","last_undegraded":"2024-04-14T15:18:13.959426+0000","last_fullsized":"2024-04-14T15:18:13.959426+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:03.685045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:03.685045+0000","last_clean_scrub_stamp":"2024-04-14T15:17:03.685045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:16.883184+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.10","version":"72'4","reported_seq":754,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.962345+0000","last_change":"2024-04-14T15:17:53.181132+0000","last_active":"2024-04-14T15:18:13.962345+0000","last_peered":"2024-04-14T15:18:13.962345+0000","last_clean":"2024-04-14T15:18:13.962345+0000","last_became_active":"2024-04-14T15:09:45.038815+0000","last_became_peered":"2024-04-14T15:09:45.038815+0000","last_unstale":"2024-04-14T15:18:13.962345+0000","last_undegraded":"2024-04-14T15:18:13.962345+0000","last_fullsized":"2024-04-14T15:18:13.962345+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4","last_scrub_stamp":"2024-04-14T15:17:53.181053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:17:53.181053+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:15.210428+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":306,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.964799+0000","last_change":"2024-04-14T15:17:48.845163+0000","last_active":"2024-04-14T15:18:13.964799+0000","last_peered":"2024-04-14T15:18:13.964799+0000","last_clean":"2024-04-14T15:18:13.964799+0000","last_became_active":"2024-04-14T15:15:44.884730+0000","last_became_peered":"2024-04-14T15:15:44.884730+0000","last_unstale":"2024-04-14T15:18:13.964799+0000","last_undegraded":"2024-04-14T15:18:13.964799+0000","last_fullsized":"2024-04-14T15:18:13.964799+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:48.845081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:08.712289+0000","last_clean_scrub_stamp":"2024-04-14T15:17:48.845081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:15.427392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":747,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.963645+0000","last_change":"2024-04-14T15:16:13.909554+0000","last_active":"2024-04-14T15:18:13.963645+0000","last_peered":"2024-04-14T15:18:13.963645+0000","last_clean":"2024-04-14T15:18:13.963645+0000","last_became_active":"2024-04-14T15:09:47.041074+0000","last_became_peered":"2024-04-14T15:09:47.041074+0000","last_unstale":"2024-04-14T15:18:13.963645+0000","last_undegraded":"2024-04-14T15:18:13.963645+0000","last_fullsized":"2024-04-14T15:18:13.963645+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:13.909454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:13.909454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":732,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-14T15:18:12.944693+0000","last_change":"2024-04-14T15:16:44.728025+0000","last_active":"2024-04-14T15:18:12.944693+0000","last_peered":"2024-04-14T15:18:12.944693+0000","last_clean":"2024-04-14T15:18:12.944693+0000","last_became_active":"2024-04-14T15:09:49.061359+0000","last_became_peered":"2024-04-14T15:09:49.061359+0000","last_unstale":"2024-04-14T15:18:12.944693+0000","last_undegraded":"2024-04-14T15:18:12.944693+0000","last_fullsized":"2024-04-14T15:18:12.944693+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:44.727907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:44.727907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:13.246634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":641,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-14T15:18:12.947533+0000","last_change":"2024-04-14T15:15:43.680936+0000","last_active":"2024-04-14T15:18:12.947533+0000","last_peered":"2024-04-14T15:18:12.947533+0000","last_clean":"2024-04-14T15:18:12.947533+0000","last_became_active":"2024-04-14T15:12:56.352528+0000","last_became_peered":"2024-04-14T15:12:56.352528+0000","last_unstale":"2024-04-14T15:18:12.947533+0000","last_undegraded":"2024-04-14T15:18:12.947533+0000","last_fullsized":"2024-04-14T15:18:12.947533+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:43.680852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:15:43.680852+0000","last_clean_scrub_stamp":"2024-04-14T15:15:43.680852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"102'16","reported_seq":410,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.962875+0000","last_change":"2024-04-14T15:17:43.061551+0000","last_active":"2024-04-14T15:18:13.962875+0000","last_peered":"2024-04-14T15:18:13.962875+0000","last_clean":"2024-04-14T15:18:13.962875+0000","last_became_active":"2024-04-14T15:15:21.884783+0000","last_became_peered":"2024-04-14T15:15:21.884783+0000","last_unstale":"2024-04-14T15:18:13.962875+0000","last_undegraded":"2024-04-14T15:18:13.962875+0000","last_fullsized":"2024-04-14T15:18:13.962875+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'16","last_scrub_stamp":"2024-04-14T15:17:43.061458+0000","last_deep_scrub":"102'16","last_deep_scrub_stamp":"2024-04-14T15:12:35.999326+0000","last_clean_scrub_stamp":"2024-04-14T15:17:43.061458+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:06.716506+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":755,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.964662+0000","last_change":"2024-04-14T15:17:01.643838+0000","last_active":"2024-04-14T15:18:13.964662+0000","last_peered":"2024-04-14T15:18:13.964662+0000","last_clean":"2024-04-14T15:18:13.964662+0000","last_became_active":"2024-04-14T15:09:43.023220+0000","last_became_peered":"2024-04-14T15:09:43.023220+0000","last_unstale":"2024-04-14T15:18:13.964662+0000","last_undegraded":"2024-04-14T15:18:13.964662+0000","last_fullsized":"2024-04-14T15:18:13.964662+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:01.643745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:01.643745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:26.033027+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":736,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-14T15:18:12.941539+0000","last_change":"2024-04-14T15:16:50.661634+0000","last_active":"2024-04-14T15:18:12.941539+0000","last_peered":"2024-04-14T15:18:12.941539+0000","last_clean":"2024-04-14T15:18:12.941539+0000","last_became_active":"2024-04-14T15:09:47.044665+0000","last_became_peered":"2024-04-14T15:09:47.044665+0000","last_unstale":"2024-04-14T15:18:12.941539+0000","last_undegraded":"2024-04-14T15:18:12.941539+0000","last_fullsized":"2024-04-14T15:18:12.941539+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:50.661520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:50.661520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":417,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.963930+0000","last_change":"2024-04-14T15:16:05.885527+0000","last_active":"2024-04-14T15:18:13.963930+0000","last_peered":"2024-04-14T15:18:13.963930+0000","last_clean":"2024-04-14T15:18:13.963930+0000","last_became_active":"2024-04-14T15:15:21.883940+0000","last_became_peered":"2024-04-14T15:15:21.883940+0000","last_unstale":"2024-04-14T15:18:13.963930+0000","last_undegraded":"2024-04-14T15:18:13.963930+0000","last_fullsized":"2024-04-14T15:18:13.963930+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:05.885438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:05.885438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.359049+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":642,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.959079+0000","last_change":"2024-04-14T15:15:28.474035+0000","last_active":"2024-04-14T15:18:13.959079+0000","last_peered":"2024-04-14T15:18:13.959079+0000","last_clean":"2024-04-14T15:18:13.959079+0000","last_became_active":"2024-04-14T15:12:56.335230+0000","last_became_peered":"2024-04-14T15:12:56.335230+0000","last_unstale":"2024-04-14T15:18:13.959079+0000","last_undegraded":"2024-04-14T15:18:13.959079+0000","last_fullsized":"2024-04-14T15:18:13.959079+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:28.473899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:28.473899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:13.985904+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"72'10","reported_seq":760,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.958293+0000","last_change":"2024-04-14T15:16:49.779653+0000","last_active":"2024-04-14T15:18:13.958293+0000","last_peered":"2024-04-14T15:18:13.958293+0000","last_clean":"2024-04-14T15:18:13.958293+0000","last_became_active":"2024-04-14T15:09:45.181456+0000","last_became_peered":"2024-04-14T15:09:45.181456+0000","last_unstale":"2024-04-14T15:18:13.958293+0000","last_undegraded":"2024-04-14T15:18:13.958293+0000","last_fullsized":"2024-04-14T15:18:13.958293+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-04-14T15:16:49.779557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:16:49.779557+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":305,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.952465+0000","last_change":"2024-04-14T15:16:42.205142+0000","last_active":"2024-04-14T15:18:13.952465+0000","last_peered":"2024-04-14T15:18:13.952465+0000","last_clean":"2024-04-14T15:18:13.952465+0000","last_became_active":"2024-04-14T15:15:44.878336+0000","last_became_peered":"2024-04-14T15:15:44.878336+0000","last_unstale":"2024-04-14T15:18:13.952465+0000","last_undegraded":"2024-04-14T15:18:13.952465+0000","last_fullsized":"2024-04-14T15:18:13.952465+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:42.205063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:42.205063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:16.453918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"72'8","reported_seq":862,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.964840+0000","last_change":"2024-04-14T15:16:12.917059+0000","last_active":"2024-04-14T15:18:13.964840+0000","last_peered":"2024-04-14T15:18:13.964840+0000","last_clean":"2024-04-14T15:18:13.964840+0000","last_became_active":"2024-04-14T15:09:47.048216+0000","last_became_peered":"2024-04-14T15:09:47.048216+0000","last_unstale":"2024-04-14T15:18:13.964840+0000","last_undegraded":"2024-04-14T15:18:13.964840+0000","last_fullsized":"2024-04-14T15:18:13.964840+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:16:12.916977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:12.916977+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"72'1","reported_seq":741,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-14T15:18:12.945995+0000","last_change":"2024-04-14T15:17:50.529174+0000","last_active":"2024-04-14T15:18:12.945995+0000","last_peered":"2024-04-14T15:18:12.945995+0000","last_clean":"2024-04-14T15:18:12.945995+0000","last_became_active":"2024-04-14T15:09:49.062051+0000","last_became_peered":"2024-04-14T15:09:49.062051+0000","last_unstale":"2024-04-14T15:18:12.945995+0000","last_undegraded":"2024-04-14T15:18:12.945995+0000","last_fullsized":"2024-04-14T15:18:12.945995+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-04-14T15:17:50.529089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:17:50.529089+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:53.773686+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":645,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.954033+0000","last_change":"2024-04-14T15:16:09.471961+0000","last_active":"2024-04-14T15:18:13.954033+0000","last_peered":"2024-04-14T15:18:13.954033+0000","last_clean":"2024-04-14T15:18:13.954033+0000","last_became_active":"2024-04-14T15:12:56.543793+0000","last_became_peered":"2024-04-14T15:12:56.543793+0000","last_unstale":"2024-04-14T15:18:13.954033+0000","last_undegraded":"2024-04-14T15:18:13.954033+0000","last_fullsized":"2024-04-14T15:18:13.954033+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:09.471852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:16:09.471852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"72'11","reported_seq":781,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.964916+0000","last_change":"2024-04-14T15:16:06.886451+0000","last_active":"2024-04-14T15:18:13.964916+0000","last_peered":"2024-04-14T15:18:13.964916+0000","last_clean":"2024-04-14T15:18:13.964916+0000","last_became_active":"2024-04-14T15:10:07.764183+0000","last_became_peered":"2024-04-14T15:10:07.764183+0000","last_unstale":"2024-04-14T15:18:13.964916+0000","last_undegraded":"2024-04-14T15:18:13.964916+0000","last_fullsized":"2024-04-14T15:18:13.964916+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-04-14T15:16:06.886325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:16:06.886325+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"65'2","reported_seq":436,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.962707+0000","last_change":"2024-04-14T15:16:56.943047+0000","last_active":"2024-04-14T15:18:13.962707+0000","last_peered":"2024-04-14T15:18:13.962707+0000","last_clean":"2024-04-14T15:18:13.962707+0000","last_became_active":"2024-04-14T15:15:21.885343+0000","last_became_peered":"2024-04-14T15:15:21.885343+0000","last_unstale":"2024-04-14T15:18:13.962707+0000","last_undegraded":"2024-04-14T15:18:13.962707+0000","last_fullsized":"2024-04-14T15:18:13.962707+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-04-14T15:16:56.942840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:56.942840+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:20.441230+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":741,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.958337+0000","last_change":"2024-04-14T15:16:52.747151+0000","last_active":"2024-04-14T15:18:13.958337+0000","last_peered":"2024-04-14T15:18:13.958337+0000","last_clean":"2024-04-14T15:18:13.958337+0000","last_became_active":"2024-04-14T15:09:47.045730+0000","last_became_peered":"2024-04-14T15:09:47.045730+0000","last_unstale":"2024-04-14T15:18:13.958337+0000","last_undegraded":"2024-04-14T15:18:13.958337+0000","last_fullsized":"2024-04-14T15:18:13.958337+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:52.747070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:52.747070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":736,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.956773+0000","last_change":"2024-04-14T15:16:57.235339+0000","last_active":"2024-04-14T15:18:13.956773+0000","last_peered":"2024-04-14T15:18:13.956773+0000","last_clean":"2024-04-14T15:18:13.956773+0000","last_became_active":"2024-04-14T15:09:49.057545+0000","last_became_peered":"2024-04-14T15:09:49.057545+0000","last_unstale":"2024-04-14T15:18:13.956773+0000","last_undegraded":"2024-04-14T15:18:13.956773+0000","last_fullsized":"2024-04-14T15:18:13.956773+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:57.235261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:21.247695+0000","last_clean_scrub_stamp":"2024-04-14T15:16:57.235261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.621753+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":305,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.952426+0000","last_change":"2024-04-14T15:17:44.364114+0000","last_active":"2024-04-14T15:18:13.952426+0000","last_peered":"2024-04-14T15:18:13.952426+0000","last_clean":"2024-04-14T15:18:13.952426+0000","last_became_active":"2024-04-14T15:15:44.882604+0000","last_became_peered":"2024-04-14T15:15:44.882604+0000","last_unstale":"2024-04-14T15:18:13.952426+0000","last_undegraded":"2024-04-14T15:18:13.952426+0000","last_fullsized":"2024-04-14T15:18:13.952426+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:44.364013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:17:44.364013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:01.492147+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"72'15","reported_seq":766,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.960283+0000","last_change":"2024-04-14T15:15:44.609525+0000","last_active":"2024-04-14T15:18:13.960283+0000","last_peered":"2024-04-14T15:18:13.960283+0000","last_clean":"2024-04-14T15:18:13.960283+0000","last_became_active":"2024-04-14T15:09:45.032887+0000","last_became_peered":"2024-04-14T15:09:45.032887+0000","last_unstale":"2024-04-14T15:18:13.960283+0000","last_undegraded":"2024-04-14T15:18:13.960283+0000","last_fullsized":"2024-04-14T15:18:13.960283+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-04-14T15:15:44.609332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:15:44.609332+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:13.985940+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":750,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.964696+0000","last_change":"2024-04-14T15:16:09.825296+0000","last_active":"2024-04-14T15:18:13.964696+0000","last_peered":"2024-04-14T15:18:13.964696+0000","last_clean":"2024-04-14T15:18:13.964696+0000","last_became_active":"2024-04-14T15:09:43.021494+0000","last_became_peered":"2024-04-14T15:09:43.021494+0000","last_unstale":"2024-04-14T15:18:13.964696+0000","last_undegraded":"2024-04-14T15:18:13.964696+0000","last_fullsized":"2024-04-14T15:18:13.964696+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:09.825220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:09.825220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":740,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.956934+0000","last_change":"2024-04-14T15:14:56.199840+0000","last_active":"2024-04-14T15:18:13.956934+0000","last_peered":"2024-04-14T15:18:13.956934+0000","last_clean":"2024-04-14T15:18:13.956934+0000","last_became_active":"2024-04-14T15:09:47.042733+0000","last_became_peered":"2024-04-14T15:09:47.042733+0000","last_unstale":"2024-04-14T15:18:13.956934+0000","last_undegraded":"2024-04-14T15:18:13.956934+0000","last_fullsized":"2024-04-14T15:18:13.956934+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:56.199756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:14:56.199756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":743,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.953322+0000","last_change":"2024-04-14T15:17:54.453289+0000","last_active":"2024-04-14T15:18:13.953322+0000","last_peered":"2024-04-14T15:18:13.953322+0000","last_clean":"2024-04-14T15:18:13.953322+0000","last_became_active":"2024-04-14T15:09:49.127979+0000","last_became_peered":"2024-04-14T15:09:49.127979+0000","last_unstale":"2024-04-14T15:18:13.953322+0000","last_undegraded":"2024-04-14T15:18:13.953322+0000","last_fullsized":"2024-04-14T15:18:13.953322+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:54.453174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:54.453174+0000","last_clean_scrub_stamp":"2024-04-14T15:17:54.453174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:08.115614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":641,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-14T15:18:12.944324+0000","last_change":"2024-04-14T15:15:31.635164+0000","last_active":"2024-04-14T15:18:12.944324+0000","last_peered":"2024-04-14T15:18:12.944324+0000","last_clean":"2024-04-14T15:18:12.944324+0000","last_became_active":"2024-04-14T15:12:56.392352+0000","last_became_peered":"2024-04-14T15:12:56.392352+0000","last_unstale":"2024-04-14T15:18:12.944324+0000","last_undegraded":"2024-04-14T15:18:12.944324+0000","last_fullsized":"2024-04-14T15:18:12.944324+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:31.635060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:31.635060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"72'12","reported_seq":448,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.962944+0000","last_change":"2024-04-14T15:16:11.903062+0000","last_active":"2024-04-14T15:18:13.962944+0000","last_peered":"2024-04-14T15:18:13.962944+0000","last_clean":"2024-04-14T15:18:13.962944+0000","last_became_active":"2024-04-14T15:15:21.877838+0000","last_became_peered":"2024-04-14T15:15:21.877838+0000","last_unstale":"2024-04-14T15:18:13.962944+0000","last_undegraded":"2024-04-14T15:18:13.962944+0000","last_fullsized":"2024-04-14T15:18:13.962944+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'12","last_scrub_stamp":"2024-04-14T15:16:11.902934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:16:11.902934+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.359013+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":429,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.963796+0000","last_change":"2024-04-14T15:16:20.969027+0000","last_active":"2024-04-14T15:18:13.963796+0000","last_peered":"2024-04-14T15:18:13.963796+0000","last_clean":"2024-04-14T15:18:13.963796+0000","last_became_active":"2024-04-14T15:15:21.883544+0000","last_became_peered":"2024-04-14T15:15:21.883544+0000","last_unstale":"2024-04-14T15:18:13.963796+0000","last_undegraded":"2024-04-14T15:18:13.963796+0000","last_fullsized":"2024-04-14T15:18:13.963796+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:20.968897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:20.968897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.359066+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":746,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.952799+0000","last_change":"2024-04-14T15:17:01.098681+0000","last_active":"2024-04-14T15:18:13.952799+0000","last_peered":"2024-04-14T15:18:13.952799+0000","last_clean":"2024-04-14T15:18:13.952799+0000","last_became_active":"2024-04-14T15:09:47.048045+0000","last_became_peered":"2024-04-14T15:09:47.048045+0000","last_unstale":"2024-04-14T15:18:13.952799+0000","last_undegraded":"2024-04-14T15:18:13.952799+0000","last_fullsized":"2024-04-14T15:18:13.952799+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:01.098599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:17:01.098599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:20.592323+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":418,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.963793+0000","last_change":"2024-04-14T15:16:07.879394+0000","last_active":"2024-04-14T15:18:13.963793+0000","last_peered":"2024-04-14T15:18:13.963793+0000","last_clean":"2024-04-14T15:18:13.963793+0000","last_became_active":"2024-04-14T15:15:21.889691+0000","last_became_peered":"2024-04-14T15:15:21.889691+0000","last_unstale":"2024-04-14T15:18:13.963793+0000","last_undegraded":"2024-04-14T15:18:13.963793+0000","last_fullsized":"2024-04-14T15:18:13.963793+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:07.879280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:07.879280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.359032+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":650,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.960026+0000","last_change":"2024-04-14T15:17:37.766225+0000","last_active":"2024-04-14T15:18:13.960026+0000","last_peered":"2024-04-14T15:18:13.960026+0000","last_clean":"2024-04-14T15:18:13.960026+0000","last_became_active":"2024-04-14T15:12:56.375278+0000","last_became_peered":"2024-04-14T15:12:56.375278+0000","last_unstale":"2024-04-14T15:18:13.960026+0000","last_undegraded":"2024-04-14T15:18:13.960026+0000","last_fullsized":"2024-04-14T15:18:13.960026+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:37.766116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:17:37.766116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:42.299956+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"276'13","reported_seq":770,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.952901+0000","last_change":"2024-04-14T15:17:52.445394+0000","last_active":"2024-04-14T15:18:13.952901+0000","last_peered":"2024-04-14T15:18:13.952901+0000","last_clean":"2024-04-14T15:18:13.952901+0000","last_became_active":"2024-04-14T15:09:45.030093+0000","last_became_peered":"2024-04-14T15:09:45.030093+0000","last_unstale":"2024-04-14T15:18:13.952901+0000","last_undegraded":"2024-04-14T15:18:13.952901+0000","last_fullsized":"2024-04-14T15:18:13.952901+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'13","last_scrub_stamp":"2024-04-14T15:17:52.445294+0000","last_deep_scrub":"276'13","last_deep_scrub_stamp":"2024-04-14T15:15:43.969775+0000","last_clean_scrub_stamp":"2024-04-14T15:17:52.445294+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:59.668815+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":450,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-14T15:18:12.945220+0000","last_change":"2024-04-14T15:17:02.800274+0000","last_active":"2024-04-14T15:18:12.945220+0000","last_peered":"2024-04-14T15:18:12.945220+0000","last_clean":"2024-04-14T15:18:12.945220+0000","last_became_active":"2024-04-14T15:14:30.857727+0000","last_became_peered":"2024-04-14T15:14:30.857727+0000","last_unstale":"2024-04-14T15:18:12.945220+0000","last_undegraded":"2024-04-14T15:18:12.945220+0000","last_fullsized":"2024-04-14T15:18:12.945220+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:02.800166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:02.800166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:15.406779+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":307,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.956122+0000","last_change":"2024-04-14T15:16:59.695423+0000","last_active":"2024-04-14T15:18:13.956122+0000","last_peered":"2024-04-14T15:18:13.956122+0000","last_clean":"2024-04-14T15:18:13.956122+0000","last_became_active":"2024-04-14T15:15:44.883121+0000","last_became_peered":"2024-04-14T15:15:44.883121+0000","last_unstale":"2024-04-14T15:18:13.956122+0000","last_undegraded":"2024-04-14T15:18:13.956122+0000","last_fullsized":"2024-04-14T15:18:13.956122+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:59.695337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:59.695337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:23.332757+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":744,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.956896+0000","last_change":"2024-04-14T15:17:33.952503+0000","last_active":"2024-04-14T15:18:13.956896+0000","last_peered":"2024-04-14T15:18:13.956896+0000","last_clean":"2024-04-14T15:18:13.956896+0000","last_became_active":"2024-04-14T15:09:49.061756+0000","last_became_peered":"2024-04-14T15:09:49.061756+0000","last_unstale":"2024-04-14T15:18:13.956896+0000","last_undegraded":"2024-04-14T15:18:13.956896+0000","last_fullsized":"2024-04-14T15:18:13.956896+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:33.952312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:17:33.952312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:36.590330+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":652,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.964216+0000","last_change":"2024-04-14T15:16:57.667484+0000","last_active":"2024-04-14T15:18:13.964216+0000","last_peered":"2024-04-14T15:18:13.964216+0000","last_clean":"2024-04-14T15:18:13.964216+0000","last_became_active":"2024-04-14T15:12:56.232831+0000","last_became_peered":"2024-04-14T15:12:56.232831+0000","last_unstale":"2024-04-14T15:18:13.964216+0000","last_undegraded":"2024-04-14T15:18:13.964216+0000","last_fullsized":"2024-04-14T15:18:13.964216+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:57.667382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:16:57.667382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:26.482036+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1b","version":"87'7","reported_seq":754,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.952831+0000","last_change":"2024-04-14T15:15:44.927867+0000","last_active":"2024-04-14T15:18:13.952831+0000","last_peered":"2024-04-14T15:18:13.952831+0000","last_clean":"2024-04-14T15:18:13.952831+0000","last_became_active":"2024-04-14T15:09:45.028667+0000","last_became_peered":"2024-04-14T15:09:45.028667+0000","last_unstale":"2024-04-14T15:18:13.952831+0000","last_undegraded":"2024-04-14T15:18:13.952831+0000","last_fullsized":"2024-04-14T15:18:13.952831+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'7","last_scrub_stamp":"2024-04-14T15:15:44.927776+0000","last_deep_scrub":"87'7","last_deep_scrub_stamp":"2024-04-14T15:11:10.510785+0000","last_clean_scrub_stamp":"2024-04-14T15:15:44.927776+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:16.453581+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":222,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.959179+0000","last_change":"2024-04-14T15:16:46.686879+0000","last_active":"2024-04-14T15:18:13.959179+0000","last_peered":"2024-04-14T15:18:13.959179+0000","last_clean":"2024-04-14T15:18:13.959179+0000","last_became_active":"2024-04-14T15:16:24.910246+0000","last_became_peered":"2024-04-14T15:16:24.910246+0000","last_unstale":"2024-04-14T15:18:13.959179+0000","last_undegraded":"2024-04-14T15:18:13.959179+0000","last_fullsized":"2024-04-14T15:18:13.959179+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:46.686789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:08.254601+0000","last_clean_scrub_stamp":"2024-04-14T15:16:46.686789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:13.986157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":736,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-14T15:18:12.944578+0000","last_change":"2024-04-14T15:16:17.804733+0000","last_active":"2024-04-14T15:18:12.944578+0000","last_peered":"2024-04-14T15:18:12.944578+0000","last_clean":"2024-04-14T15:18:12.944578+0000","last_became_active":"2024-04-14T15:09:47.046517+0000","last_became_peered":"2024-04-14T15:09:47.046517+0000","last_unstale":"2024-04-14T15:18:12.944578+0000","last_undegraded":"2024-04-14T15:18:12.944578+0000","last_fullsized":"2024-04-14T15:18:12.944578+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:17.804649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:17.804649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":300,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.954817+0000","last_change":"2024-04-14T15:15:44.887620+0000","last_active":"2024-04-14T15:18:13.954817+0000","last_peered":"2024-04-14T15:18:13.954817+0000","last_clean":"2024-04-14T15:18:13.954817+0000","last_became_active":"2024-04-14T15:15:44.887227+0000","last_became_peered":"2024-04-14T15:15:44.887227+0000","last_unstale":"2024-04-14T15:18:13.954817+0000","last_undegraded":"2024-04-14T15:18:13.954817+0000","last_fullsized":"2024-04-14T15:18:13.954817+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:56.268086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:56.268086+0000","last_clean_scrub_stamp":"2024-04-14T15:14:56.268086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":641,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-14T15:18:12.947914+0000","last_change":"2024-04-14T15:15:37.629813+0000","last_active":"2024-04-14T15:18:12.947914+0000","last_peered":"2024-04-14T15:18:12.947914+0000","last_clean":"2024-04-14T15:18:12.947914+0000","last_became_active":"2024-04-14T15:12:56.189117+0000","last_became_peered":"2024-04-14T15:12:56.189117+0000","last_unstale":"2024-04-14T15:18:12.947914+0000","last_undegraded":"2024-04-14T15:18:12.947914+0000","last_fullsized":"2024-04-14T15:18:12.947914+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:37.629718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:37.629718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.18","version":"72'9","reported_seq":758,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.952304+0000","last_change":"2024-04-14T15:16:09.965230+0000","last_active":"2024-04-14T15:18:13.952304+0000","last_peered":"2024-04-14T15:18:13.952304+0000","last_clean":"2024-04-14T15:18:13.952304+0000","last_became_active":"2024-04-14T15:09:45.032079+0000","last_became_peered":"2024-04-14T15:09:45.032079+0000","last_unstale":"2024-04-14T15:18:13.952304+0000","last_undegraded":"2024-04-14T15:18:13.952304+0000","last_fullsized":"2024-04-14T15:18:13.952304+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-04-14T15:16:09.965113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:16:09.965113+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:16.453615+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":430,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.963734+0000","last_change":"2024-04-14T15:17:00.948362+0000","last_active":"2024-04-14T15:18:13.963734+0000","last_peered":"2024-04-14T15:18:13.963734+0000","last_clean":"2024-04-14T15:18:13.963734+0000","last_became_active":"2024-04-14T15:15:21.891926+0000","last_became_peered":"2024-04-14T15:15:21.891926+0000","last_unstale":"2024-04-14T15:18:13.963734+0000","last_undegraded":"2024-04-14T15:18:13.963734+0000","last_fullsized":"2024-04-14T15:18:13.963734+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:00.948283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:00.948283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:15.418233+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":740,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.953858+0000","last_change":"2024-04-14T15:16:45.114648+0000","last_active":"2024-04-14T15:18:13.953858+0000","last_peered":"2024-04-14T15:18:13.953858+0000","last_clean":"2024-04-14T15:18:13.953858+0000","last_became_active":"2024-04-14T15:09:47.041486+0000","last_became_peered":"2024-04-14T15:09:47.041486+0000","last_unstale":"2024-04-14T15:18:13.953858+0000","last_undegraded":"2024-04-14T15:18:13.953858+0000","last_fullsized":"2024-04-14T15:18:13.953858+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:45.114534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:45.114534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.009802+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":734,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.952802+0000","last_change":"2024-04-14T15:16:55.184211+0000","last_active":"2024-04-14T15:18:13.952802+0000","last_peered":"2024-04-14T15:18:13.952802+0000","last_clean":"2024-04-14T15:18:13.952802+0000","last_became_active":"2024-04-14T15:09:49.058904+0000","last_became_peered":"2024-04-14T15:09:49.058904+0000","last_unstale":"2024-04-14T15:18:13.952802+0000","last_undegraded":"2024-04-14T15:18:13.952802+0000","last_fullsized":"2024-04-14T15:18:13.952802+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:55.184094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:55.184094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:15.235195+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":303,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-14T15:18:12.947440+0000","last_change":"2024-04-14T15:17:52.593706+0000","last_active":"2024-04-14T15:18:12.947440+0000","last_peered":"2024-04-14T15:18:12.947440+0000","last_clean":"2024-04-14T15:18:12.947440+0000","last_became_active":"2024-04-14T15:15:44.889605+0000","last_became_peered":"2024-04-14T15:15:44.889605+0000","last_unstale":"2024-04-14T15:18:12.947440+0000","last_undegraded":"2024-04-14T15:18:12.947440+0000","last_fullsized":"2024-04-14T15:18:12.947440+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:52.593618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:17:52.593618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:09.797845+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"306'22","reported_seq":457,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.962910+0000","last_change":"2024-04-14T15:16:13.898263+0000","last_active":"2024-04-14T15:18:13.962910+0000","last_peered":"2024-04-14T15:18:13.962910+0000","last_clean":"2024-04-14T15:18:13.962910+0000","last_became_active":"2024-04-14T15:15:21.884310+0000","last_became_peered":"2024-04-14T15:15:21.884310+0000","last_unstale":"2024-04-14T15:18:13.962910+0000","last_undegraded":"2024-04-14T15:18:13.962910+0000","last_fullsized":"2024-04-14T15:18:13.962910+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'22","last_scrub_stamp":"2024-04-14T15:16:13.898106+0000","last_deep_scrub":"306'22","last_deep_scrub_stamp":"2024-04-14T15:16:13.898106+0000","last_clean_scrub_stamp":"2024-04-14T15:16:13.898106+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.359119+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":749,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.956323+0000","last_change":"2024-04-14T15:16:14.427502+0000","last_active":"2024-04-14T15:18:13.956323+0000","last_peered":"2024-04-14T15:18:13.956323+0000","last_clean":"2024-04-14T15:18:13.956323+0000","last_became_active":"2024-04-14T15:09:43.017351+0000","last_became_peered":"2024-04-14T15:09:43.017351+0000","last_unstale":"2024-04-14T15:18:13.956323+0000","last_undegraded":"2024-04-14T15:18:13.956323+0000","last_fullsized":"2024-04-14T15:18:13.956323+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:14.427425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:14.427425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":738,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.952601+0000","last_change":"2024-04-14T15:16:19.041461+0000","last_active":"2024-04-14T15:18:13.952601+0000","last_peered":"2024-04-14T15:18:13.952601+0000","last_clean":"2024-04-14T15:18:13.952601+0000","last_became_active":"2024-04-14T15:09:47.042538+0000","last_became_peered":"2024-04-14T15:09:47.042538+0000","last_unstale":"2024-04-14T15:18:13.952601+0000","last_undegraded":"2024-04-14T15:18:13.952601+0000","last_fullsized":"2024-04-14T15:18:13.952601+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:19.041346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:08.512130+0000","last_clean_scrub_stamp":"2024-04-14T15:16:19.041346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:16.453889+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":740,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-14T15:18:13.963488+0000","last_change":"2024-04-14T15:17:56.237366+0000","last_active":"2024-04-14T15:18:13.963488+0000","last_peered":"2024-04-14T15:18:13.963488+0000","last_clean":"2024-04-14T15:18:13.963488+0000","last_became_active":"2024-04-14T15:09:49.129636+0000","last_became_peered":"2024-04-14T15:09:49.129636+0000","last_unstale":"2024-04-14T15:18:13.963488+0000","last_undegraded":"2024-04-14T15:18:13.963488+0000","last_fullsized":"2024-04-14T15:18:13.963488+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:56.237273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:17:56.237273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:03.893408+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":297,"num_pg":32,"stat_sum":{"num_bytes":14,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":6,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":42,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":110,"num_pg":32,"stat_sum":{"num_bytes":8195783,"num_objects":159,"num_object_clones":2,"num_object_copies":477,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":159,"num_whiteouts":0,"num_read":108,"num_read_kb":9,"num_write":593,"num_write_kb":12682,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":7,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":26468352,"data_stored":24587349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1867,"ondisk_log_size":1867,"up":96,"acting":96,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1430,"ondisk_log_size":1430,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":504,"num_read_kb":499,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":695,"num_read_kb":452,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":481,"ondisk_log_size":481,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71,"ondisk_log_size":71,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103276,"num_pgs":99,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":73008,"kb_used_data":15440,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93672144,"statfs":{"total":95995035648,"available":95920275456,"internally_reserved":0,"allocated":15810560,"data_stored":12583404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":58915274},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397615,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":86908,"kb_used_data":7120,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":93658244,"statfs":{"total":95995035648,"available":95906041856,"internally_reserved":0,"allocated":7290880,"data_stored":4061717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":81656262},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574985,"num_pgs":93,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":86948,"kb_used_data":11332,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93658204,"statfs":{"total":95995035648,"available":95906000896,"internally_reserved":0,"allocated":11603968,"data_stored":8265424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738510,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":82532,"kb_used_data":11088,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93662620,"statfs":{"total":95995035648,"available":95910522880,"internally_reserved":0,"allocated":11354112,"data_stored":8071259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411461,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":82824,"kb_used_data":11356,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93662328,"statfs":{"total":95995035648,"available":95910223872,"internally_reserved":0,"allocated":11628544,"data_stored":8323846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182527,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":52744,"kb_used_data":7668,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93692408,"statfs":{"total":95995035648,"available":95941025792,"internally_reserved":0,"allocated":7852032,"data_stored":4652393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986297,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":70224,"kb_used_data":7088,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93674928,"statfs":{"total":95995035648,"available":95923126272,"internally_reserved":0,"allocated":7258112,"data_stored":4061235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659254,"num_pgs":84,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":74932,"kb_used_data":11772,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93670220,"statfs":{"total":95995035648,"available":95918305280,"internally_reserved":0,"allocated":12054528,"data_stored":8845402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4263936,"data_stored":4000600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4493312,"data_stored":4195009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4317184,"data_stored":4000756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":184320,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4403200,"data_stored":4194789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":225280,"data_stored":524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8417280,"data_stored":8194829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":297,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":297,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":297,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":2,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":297,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":12,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":297,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":14,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":297,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":2,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":297,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":12,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":297,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-14T15:18:17.704 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph pg dump --format=json 2024-04-14T15:18:17.780 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:18:17.804 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:17.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:17.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:18:17.905 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:18:17.930 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:17.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:17.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:18:17.992 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:17 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:18:17.992 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:17 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:18:17.992 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:17 smithi092 ceph-mon[33966]: from='client.49102 v1:172.21.15.90:0/166863032' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:18:17.992 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:17 smithi092 ceph-mon[33966]: osdmap e424: 8 total, 8 up, 8 in 2024-04-14T15:18:18.032 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:18.056 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:18.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:18.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:18.157 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:18:18.159 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:18:18.182 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:18.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:18.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:18:18.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:17 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:18:18.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:17 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:18:18.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:17 smithi090 ceph-mon[30399]: from='client.49102 v1:172.21.15.90:0/166863032' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:18:18.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:17 smithi090 ceph-mon[30399]: osdmap e424: 8 total, 8 up, 8 in 2024-04-14T15:18:18.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:17 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:18:18.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:17 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:18:18.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:17 smithi090 ceph-mon[41670]: from='client.49102 v1:172.21.15.90:0/166863032' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:18:18.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:17 smithi090 ceph-mon[41670]: osdmap e424: 8 total, 8 up, 8 in 2024-04-14T15:18:18.271 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-04-14T15:18:18.271 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:18:18.285 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:18:18.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:18.309 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:18.310 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:18:18.410 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:18:18.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:18.435 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:18.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:18:18.536 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:18:18.559 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:18.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:18.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:18:18.661 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:18.686 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:18.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:18.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:18.787 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:18:18.811 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:18.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:18.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:18:18.913 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:18:18.939 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:18.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:18.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:18:18.947 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-14T15:18:19.041 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:18:19.065 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:19.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:19.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:18:19.167 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:18:19.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:18 smithi090 ceph-mon[41670]: pgmap v584: 238 pgs: 238 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-04-14T15:18:19.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:18 smithi090 ceph-mon[30399]: pgmap v584: 238 pgs: 238 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-04-14T15:18:19.192 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:19.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:19.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:18:19.230 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:18:19.293 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:18:19.318 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:19.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:19.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:18:19.351 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:18 smithi092 ceph-mon[33966]: pgmap v584: 238 pgs: 238 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-04-14T15:18:19.420 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:18:19.444 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:19.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:19.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:18:19.545 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:18:19.569 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:19.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:19.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:18:19.671 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:18:19.694 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:19.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:19.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:18:19.796 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:18:19.819 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:19.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:19.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:18:19.920 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:18:19.944 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:19.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:19.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:18:20.046 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:18:20.070 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:20.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:20.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:18:20.171 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:18:20.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:20.196 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:20.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:18:20.297 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:18:20.322 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:20.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:20.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:18:20.424 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:18:20.447 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:20.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:20.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:18:20.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.533+0000 7fbc2027f700 1 -- 172.21.15.90:0/2186175545 >> v1:172.21.15.92:6789/0 conn(0x7fbc1810f3b0 legacy=0x7fbc181117a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:20.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.533+0000 7fbc2027f700 1 -- 172.21.15.90:0/2186175545 shutdown_connections 2024-04-14T15:18:20.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.533+0000 7fbc2027f700 1 -- 172.21.15.90:0/2186175545 >> 172.21.15.90:0/2186175545 conn(0x7fbc180fd280 msgr2=0x7fbc180ff6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:18:20.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.533+0000 7fbc2027f700 1 -- 172.21.15.90:0/2186175545 shutdown_connections 2024-04-14T15:18:20.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.533+0000 7fbc2027f700 1 -- 172.21.15.90:0/2186175545 wait complete. 2024-04-14T15:18:20.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.534+0000 7fbc2027f700 1 librados: starting msgr at 2024-04-14T15:18:20.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.534+0000 7fbc2027f700 1 librados: starting objecter 2024-04-14T15:18:20.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.534+0000 7fbc2027f700 1 Processor -- start 2024-04-14T15:18:20.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.534+0000 7fbc2027f700 1 -- start start 2024-04-14T15:18:20.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.534+0000 7fbc2027f700 1 librados: setting wanted keys 2024-04-14T15:18:20.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.534+0000 7fbc2027f700 1 librados: calling monclient init 2024-04-14T15:18:20.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.534+0000 7fbc2027f700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbc1810b580 con 0x7fbc18103a90 2024-04-14T15:18:20.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.534+0000 7fbc2027f700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbc181a8ed0 con 0x7fbc1810f3b0 2024-04-14T15:18:20.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.534+0000 7fbc2027f700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbc181aa0b0 con 0x7fbc1810b8a0 2024-04-14T15:18:20.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.534+0000 7fbc1d81a700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7fbc1810b8a0 0x7fbc18108ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:37104/0 (socket says 172.21.15.90:37104) 2024-04-14T15:18:20.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.534+0000 7fbc1d81a700 1 -- 172.21.15.90:0/3266312825 learned_addr learned my addr 172.21.15.90:0/3266312825 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:18:20.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.534+0000 7fbc0f7fe700 10 client.?.objecter ms_handle_connect 0x7fbc18103a90 2024-04-14T15:18:20.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.534+0000 7fbc0f7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:18:20.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.534+0000 7fbc0f7fe700 10 client.?.objecter ms_handle_connect 0x7fbc1810b8a0 2024-04-14T15:18:20.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.534+0000 7fbc0f7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:18:20.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.535+0000 7fbc0f7fe700 10 client.?.objecter ms_handle_connect 0x7fbc1810f3b0 2024-04-14T15:18:20.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.535+0000 7fbc0f7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:18:20.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.535+0000 7fbc0f7fe700 1 -- 172.21.15.90:0/3266312825 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2363144314 0 0) 0x7fbc1810b580 con 0x7fbc18103a90 2024-04-14T15:18:20.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.535+0000 7fbc0f7fe700 1 -- 172.21.15.90:0/3266312825 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc040035b0 con 0x7fbc18103a90 2024-04-14T15:18:20.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.535+0000 7fbc0f7fe700 1 -- 172.21.15.90:0/3266312825 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2457279632 0 0) 0x7fbc181a8ed0 con 0x7fbc1810f3b0 2024-04-14T15:18:20.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.535+0000 7fbc0f7fe700 1 -- 172.21.15.90:0/3266312825 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc1810b580 con 0x7fbc1810f3b0 2024-04-14T15:18:20.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.535+0000 7fbc0f7fe700 1 -- 172.21.15.90:0/3266312825 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 496530937 0 0) 0x7fbc181aa0b0 con 0x7fbc1810b8a0 2024-04-14T15:18:20.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.535+0000 7fbc0f7fe700 1 -- 172.21.15.90:0/3266312825 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc181a8ed0 con 0x7fbc1810b8a0 2024-04-14T15:18:20.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.535+0000 7fbc0f7fe700 1 -- 172.21.15.90:0/3266312825 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3401649219 0 0) 0x7fbc1810b580 con 0x7fbc1810f3b0 2024-04-14T15:18:20.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.535+0000 7fbc0f7fe700 1 -- 172.21.15.90:0/3266312825 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbc181aa0b0 con 0x7fbc1810f3b0 2024-04-14T15:18:20.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.535+0000 7fbc0f7fe700 1 -- 172.21.15.90:0/3266312825 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fbc100017b0 con 0x7fbc1810f3b0 2024-04-14T15:18:20.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.535+0000 7fbc0f7fe700 1 -- 172.21.15.90:0/3266312825 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1727634767 0 0) 0x7fbc040035b0 con 0x7fbc18103a90 2024-04-14T15:18:20.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.535+0000 7fbc0f7fe700 1 -- 172.21.15.90:0/3266312825 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbc1810b580 con 0x7fbc18103a90 2024-04-14T15:18:20.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.535+0000 7fbc0f7fe700 1 -- 172.21.15.90:0/3266312825 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3121628495 0 0) 0x7fbc181a8ed0 con 0x7fbc1810b8a0 2024-04-14T15:18:20.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.535+0000 7fbc0f7fe700 1 -- 172.21.15.90:0/3266312825 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbc040035b0 con 0x7fbc1810b8a0 2024-04-14T15:18:20.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.535+0000 7fbc0f7fe700 1 -- 172.21.15.90:0/3266312825 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fbc14001830 con 0x7fbc18103a90 2024-04-14T15:18:20.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.535+0000 7fbc0f7fe700 1 -- 172.21.15.90:0/3266312825 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fbc080018f0 con 0x7fbc1810b8a0 2024-04-14T15:18:20.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.535+0000 7fbc0f7fe700 1 -- 172.21.15.90:0/3266312825 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 887847804 0 0) 0x7fbc181aa0b0 con 0x7fbc1810f3b0 2024-04-14T15:18:20.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.535+0000 7fbc0f7fe700 1 -- 172.21.15.90:0/3266312825 >> v1:172.21.15.90:6790/0 conn(0x7fbc1810b8a0 legacy=0x7fbc18108ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:20.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.535+0000 7fbc0f7fe700 1 -- 172.21.15.90:0/3266312825 >> v1:172.21.15.90:6789/0 conn(0x7fbc18103a90 legacy=0x7fbc181083d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:20.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.535+0000 7fbc0f7fe700 1 -- 172.21.15.90:0/3266312825 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc181ab290 con 0x7fbc1810f3b0 2024-04-14T15:18:20.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.535+0000 7fbc2027f700 1 -- 172.21.15.90:0/3266312825 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbc181a9100 con 0x7fbc1810f3b0 2024-04-14T15:18:20.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.535+0000 7fbc2027f700 10 client.49108.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:18:20.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.535+0000 7fbc2027f700 1 -- 172.21.15.90:0/3266312825 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fbc181a95c0 con 0x7fbc1810f3b0 2024-04-14T15:18:20.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.535+0000 7fbc2027f700 1 librados: init done 2024-04-14T15:18:20.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.536+0000 7fbc0f7fe700 1 -- 172.21.15.90:0/3266312825 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fbc10003a80 con 0x7fbc1810f3b0 2024-04-14T15:18:20.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.536+0000 7fbc0f7fe700 1 -- 172.21.15.90:0/3266312825 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fbc100013a0 con 0x7fbc1810f3b0 2024-04-14T15:18:20.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.536+0000 7fbc2027f700 1 -- 172.21.15.90:0/3266312825 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbfc005120 con 0x7fbc1810f3b0 2024-04-14T15:18:20.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.537+0000 7fbc0f7fe700 1 -- 172.21.15.90:0/3266312825 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fbc1001ba70 con 0x7fbc1810f3b0 2024-04-14T15:18:20.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.538+0000 7fbc0f7fe700 1 -- 172.21.15.90:0/3266312825 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(424..424 src has 51..424) ==== 8216+0+0 (unknown 3277651793 0 0) 0x7fbc10089a80 con 0x7fbc1810f3b0 2024-04-14T15:18:20.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.538+0000 7fbc0f7fe700 10 client.49108.objecter ms_dispatch 0x7fbc18001550 osd_map(424..424 src has 51..424) 2024-04-14T15:18:20.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.538+0000 7fbc0f7fe700 3 client.49108.objecter handle_osd_map got epochs [424,424] > 0 2024-04-14T15:18:20.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.538+0000 7fbc0f7fe700 3 client.49108.objecter handle_osd_map decoding full epoch 424 2024-04-14T15:18:20.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.538+0000 7fbc0f7fe700 20 client.49108.objecter dump_active .. 0 homeless 2024-04-14T15:18:20.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.538+0000 7fbc0f7fe700 10 client.49108.objecter ms_handle_connect 0x7fbc040715e0 2024-04-14T15:18:20.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.542+0000 7fbc0f7fe700 1 -- 172.21.15.90:0/3266312825 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fbc10056460 con 0x7fbc1810f3b0 2024-04-14T15:18:20.549 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:18:20.573 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:20.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:20.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:18:20.675 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:20.699 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:20.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:20.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:20.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.761+0000 7fbc2027f700 1 -- 172.21.15.90:0/3266312825 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7fbbfc005410 con 0x7fbc1810f3b0 2024-04-14T15:18:20.789 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:18:20.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.790+0000 7f838950b700 1 -- 172.21.15.90:0/3050393410 >> v1:172.21.15.90:6789/0 conn(0x7f838410e1e0 legacy=0x7f83841106a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:20.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.791+0000 7f838950b700 1 -- 172.21.15.90:0/3050393410 shutdown_connections 2024-04-14T15:18:20.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.791+0000 7f838950b700 1 -- 172.21.15.90:0/3050393410 >> 172.21.15.90:0/3050393410 conn(0x7f83840fd290 msgr2=0x7f83840ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:18:20.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.791+0000 7f838950b700 1 -- 172.21.15.90:0/3050393410 shutdown_connections 2024-04-14T15:18:20.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.791+0000 7f838950b700 1 -- 172.21.15.90:0/3050393410 wait complete. 2024-04-14T15:18:20.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.791+0000 7f838950b700 1 librados: starting msgr at 2024-04-14T15:18:20.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.791+0000 7f838950b700 1 librados: starting objecter 2024-04-14T15:18:20.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.791+0000 7f838950b700 1 Processor -- start 2024-04-14T15:18:20.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.791+0000 7f838950b700 1 -- start start 2024-04-14T15:18:20.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.791+0000 7f838950b700 1 librados: setting wanted keys 2024-04-14T15:18:20.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.791+0000 7f838950b700 1 librados: calling monclient init 2024-04-14T15:18:20.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.792+0000 7f838950b700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f838410ded0 con 0x7f8384107760 2024-04-14T15:18:20.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.792+0000 7f838950b700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f83841a7df0 con 0x7f838410a610 2024-04-14T15:18:20.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.792+0000 7f838950b700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f83841a8fd0 con 0x7f838410e1e0 2024-04-14T15:18:20.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.792+0000 7f8382ffd700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f8384107760 0x7f838410c790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:48954/0 (socket says 172.21.15.90:48954) 2024-04-14T15:18:20.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.792+0000 7f8382ffd700 1 -- 172.21.15.90:0/2925376967 learned_addr learned my addr 172.21.15.90:0/2925376967 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:18:20.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.792+0000 7f836bfff700 10 client.?.objecter ms_handle_connect 0x7f838410a610 2024-04-14T15:18:20.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.792+0000 7f836bfff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:18:20.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.792+0000 7f836bfff700 10 client.?.objecter ms_handle_connect 0x7f8384107760 2024-04-14T15:18:20.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.792+0000 7f836bfff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:18:20.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.792+0000 7f836bfff700 10 client.?.objecter ms_handle_connect 0x7f838410e1e0 2024-04-14T15:18:20.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.792+0000 7f836bfff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:18:20.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.792+0000 7f836bfff700 1 -- 172.21.15.90:0/2925376967 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 252088519 0 0) 0x7f83841a7df0 con 0x7f838410a610 2024-04-14T15:18:20.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.792+0000 7f836bfff700 1 -- 172.21.15.90:0/2925376967 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f836c0035b0 con 0x7f838410a610 2024-04-14T15:18:20.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.792+0000 7f836bfff700 1 -- 172.21.15.90:0/2925376967 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 296399044 0 0) 0x7f838410ded0 con 0x7f8384107760 2024-04-14T15:18:20.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.792+0000 7f836bfff700 1 -- 172.21.15.90:0/2925376967 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f83841a7df0 con 0x7f8384107760 2024-04-14T15:18:20.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.792+0000 7f836bfff700 1 -- 172.21.15.90:0/2925376967 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2988247244 0 0) 0x7f83841a8fd0 con 0x7f838410e1e0 2024-04-14T15:18:20.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.792+0000 7f836bfff700 1 -- 172.21.15.90:0/2925376967 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f838410ded0 con 0x7f838410e1e0 2024-04-14T15:18:20.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.792+0000 7f836bfff700 1 -- 172.21.15.90:0/2925376967 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2689413100 0 0) 0x7f836c0035b0 con 0x7f838410a610 2024-04-14T15:18:20.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.792+0000 7f836bfff700 1 -- 172.21.15.90:0/2925376967 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f83841a8fd0 con 0x7f838410a610 2024-04-14T15:18:20.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.792+0000 7f836bfff700 1 -- 172.21.15.90:0/2925376967 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f83780018b0 con 0x7f838410a610 2024-04-14T15:18:20.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.792+0000 7f836bfff700 1 -- 172.21.15.90:0/2925376967 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1190590054 0 0) 0x7f83841a7df0 con 0x7f8384107760 2024-04-14T15:18:20.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.792+0000 7f836bfff700 1 -- 172.21.15.90:0/2925376967 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f836c0035b0 con 0x7f8384107760 2024-04-14T15:18:20.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.792+0000 7f836bfff700 1 -- 172.21.15.90:0/2925376967 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1842645417 0 0) 0x7f838410ded0 con 0x7f838410e1e0 2024-04-14T15:18:20.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.792+0000 7f836bfff700 1 -- 172.21.15.90:0/2925376967 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f83841a7df0 con 0x7f838410e1e0 2024-04-14T15:18:20.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.792+0000 7f836bfff700 1 -- 172.21.15.90:0/2925376967 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f83740018b0 con 0x7f8384107760 2024-04-14T15:18:20.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.792+0000 7f836bfff700 1 -- 172.21.15.90:0/2925376967 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f837c001950 con 0x7f838410e1e0 2024-04-14T15:18:20.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.793+0000 7f836bfff700 1 -- 172.21.15.90:0/2925376967 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1297792613 0 0) 0x7f83841a8fd0 con 0x7f838410a610 2024-04-14T15:18:20.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.793+0000 7f836bfff700 1 -- 172.21.15.90:0/2925376967 >> v1:172.21.15.90:6790/0 conn(0x7f838410e1e0 legacy=0x7f83841a56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:20.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.793+0000 7f836bfff700 1 -- 172.21.15.90:0/2925376967 >> v1:172.21.15.90:6789/0 conn(0x7f8384107760 legacy=0x7f838410c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:20.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.793+0000 7f836bfff700 1 -- 172.21.15.90:0/2925376967 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83841aa1f0 con 0x7f838410a610 2024-04-14T15:18:20.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.793+0000 7f836bfff700 1 -- 172.21.15.90:0/2925376967 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f8378003b40 con 0x7f838410a610 2024-04-14T15:18:20.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.793+0000 7f838950b700 1 -- 172.21.15.90:0/2925376967 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f83841a9200 con 0x7f838410a610 2024-04-14T15:18:20.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.793+0000 7f836bfff700 1 -- 172.21.15.90:0/2925376967 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f8378004ad0 con 0x7f838410a610 2024-04-14T15:18:20.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.793+0000 7f838950b700 10 client.49114.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:18:20.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.793+0000 7f838950b700 1 -- 172.21.15.90:0/2925376967 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f83841a96c0 con 0x7f838410a610 2024-04-14T15:18:20.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.793+0000 7f838950b700 1 librados: init done 2024-04-14T15:18:20.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.794+0000 7f838950b700 1 -- 172.21.15.90:0/2925376967 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f8384102ea0 con 0x7f838410a610 2024-04-14T15:18:20.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.794+0000 7f836bfff700 1 -- 172.21.15.90:0/2925376967 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f83780036a0 con 0x7f838410a610 2024-04-14T15:18:20.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.795+0000 7f836bfff700 1 -- 172.21.15.90:0/2925376967 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(424..424 src has 51..424) ==== 8216+0+0 (unknown 3277651793 0 0) 0x7f8378089a00 con 0x7f838410a610 2024-04-14T15:18:20.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.795+0000 7f836bfff700 10 client.49114.objecter ms_dispatch 0x7f8384001550 osd_map(424..424 src has 51..424) 2024-04-14T15:18:20.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.795+0000 7f836bfff700 3 client.49114.objecter handle_osd_map got epochs [424,424] > 0 2024-04-14T15:18:20.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.795+0000 7f836bfff700 3 client.49114.objecter handle_osd_map decoding full epoch 424 2024-04-14T15:18:20.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.795+0000 7f836bfff700 20 client.49114.objecter dump_active .. 0 homeless 2024-04-14T15:18:20.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.795+0000 7f836bfff700 1 -- 172.21.15.90:0/2925376967 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v424) ==== 54+0+16 (unknown 3678487704 0 2420000192) 0x7f8378089d70 con 0x7f838410a610 2024-04-14T15:18:20.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.795+0000 7f836bfff700 10 client.49114.objecter ms_handle_connect 0x7f836c0735d0 2024-04-14T15:18:20.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.795+0000 7f838950b700 10 client.49114.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:18:20.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.795+0000 7f838950b700 15 client.49114.objecter _session_command_op_assign -1 1 2024-04-14T15:18:20.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.795+0000 7f838950b700 20 client.49114.objecter _get_session s=0x7f83840693d0 osd=0 3 2024-04-14T15:18:20.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.795+0000 7f838950b700 20 client.49114.objecter put_session s=0x7f83840693d0 osd=0 3 2024-04-14T15:18:20.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.795+0000 7f838950b700 20 client.49114.objecter _get_session s=0x7f83840693d0 osd=0 3 2024-04-14T15:18:20.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.795+0000 7f838950b700 15 client.49114.objecter _session_command_op_remove -1 1 2024-04-14T15:18:20.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.795+0000 7f838950b700 20 client.49114.objecter get_session s=0x7f83840693d0 osd=0 3 2024-04-14T15:18:20.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.795+0000 7f838950b700 15 client.49114.objecter _session_command_op_assign 0 1 2024-04-14T15:18:20.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.795+0000 7f838950b700 20 client.49114.objecter put_session s=0x7f83840693d0 osd=0 4 2024-04-14T15:18:20.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.795+0000 7f838950b700 10 client.49114.objecter _send_command 1 2024-04-14T15:18:20.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.795+0000 7f838950b700 1 -- 172.21.15.90:0/2925376967 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f838410c250 con 0x7f83841a72f0 2024-04-14T15:18:20.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.796+0000 7f836bfff700 10 client.49114.objecter ms_handle_connect 0x7f83841a72f0 2024-04-14T15:18:20.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.796+0000 7f836bfff700 1 -- 172.21.15.90:0/2925376967 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f838410c250 con 0x7f83841a72f0 2024-04-14T15:18:20.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.796+0000 7f836bfff700 10 client.49114.objecter ms_dispatch 0x7f8384001550 command_reply(tid 1: 0 ) 2024-04-14T15:18:20.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.796+0000 7f836bfff700 10 client.49114.objecter _finish_command 1 = system:0 2024-04-14T15:18:20.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.796+0000 7f836bfff700 20 client.49114.objecter put_session s=0x7f83840693d0 osd=0 4 2024-04-14T15:18:20.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.796+0000 7f836bfff700 15 client.49114.objecter _session_command_op_remove 0 1 2024-04-14T15:18:20.800 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:18:20.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.814+0000 7f838950b700 10 client.49114.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:18:20.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.814+0000 7f838950b700 15 client.49114.objecter _session_command_op_assign -1 2 2024-04-14T15:18:20.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.814+0000 7f838950b700 20 client.49114.objecter _get_session s=0x7f83840693d0 osd=0 3 2024-04-14T15:18:20.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.814+0000 7f838950b700 20 client.49114.objecter put_session s=0x7f83840693d0 osd=0 3 2024-04-14T15:18:20.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.814+0000 7f838950b700 20 client.49114.objecter _get_session s=0x7f83840693d0 osd=0 3 2024-04-14T15:18:20.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.814+0000 7f838950b700 15 client.49114.objecter _session_command_op_remove -1 2 2024-04-14T15:18:20.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.814+0000 7f838950b700 20 client.49114.objecter get_session s=0x7f83840693d0 osd=0 3 2024-04-14T15:18:20.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.814+0000 7f838950b700 15 client.49114.objecter _session_command_op_assign 0 2 2024-04-14T15:18:20.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.814+0000 7f838950b700 20 client.49114.objecter put_session s=0x7f83840693d0 osd=0 4 2024-04-14T15:18:20.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.814+0000 7f838950b700 10 client.49114.objecter _send_command 2 2024-04-14T15:18:20.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.814+0000 7f838950b700 1 -- 172.21.15.90:0/2925376967 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f83841a9c00 con 0x7f83841a72f0 2024-04-14T15:18:20.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.816+0000 7f836bfff700 1 -- 172.21.15.90:0/2925376967 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f83841a9c00 con 0x7f83841a72f0 2024-04-14T15:18:20.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.816+0000 7f836bfff700 10 client.49114.objecter ms_dispatch 0x7f8384001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:18:20.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.816+0000 7f836bfff700 10 client.49114.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:18:20.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.816+0000 7f836bfff700 20 client.49114.objecter put_session s=0x7f83840693d0 osd=0 4 2024-04-14T15:18:20.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.816+0000 7f836bfff700 15 client.49114.objecter _session_command_op_remove 0 2 2024-04-14T15:18:20.817 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:18:20.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.817+0000 7f838950b700 10 client.49114.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:18:20.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.817+0000 7f838950b700 15 client.49114.objecter _session_command_op_assign -1 3 2024-04-14T15:18:20.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.817+0000 7f838950b700 20 client.49114.objecter _get_session s=0x7f83841aa000 osd=1 3 2024-04-14T15:18:20.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.817+0000 7f838950b700 20 client.49114.objecter put_session s=0x7f83841aa000 osd=1 3 2024-04-14T15:18:20.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.817+0000 7f838950b700 20 client.49114.objecter _get_session s=0x7f83841aa000 osd=1 3 2024-04-14T15:18:20.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.817+0000 7f838950b700 15 client.49114.objecter _session_command_op_remove -1 3 2024-04-14T15:18:20.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.817+0000 7f838950b700 20 client.49114.objecter get_session s=0x7f83841aa000 osd=1 3 2024-04-14T15:18:20.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.817+0000 7f838950b700 15 client.49114.objecter _session_command_op_assign 1 3 2024-04-14T15:18:20.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.817+0000 7f838950b700 20 client.49114.objecter put_session s=0x7f83841aa000 osd=1 4 2024-04-14T15:18:20.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.817+0000 7f838950b700 10 client.49114.objecter _send_command 3 2024-04-14T15:18:20.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.817+0000 7f838950b700 1 -- 172.21.15.90:0/2925376967 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f83840fd670 con 0x7f838406b140 2024-04-14T15:18:20.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.817+0000 7f836bfff700 10 client.49114.objecter ms_handle_connect 0x7f838406b140 2024-04-14T15:18:20.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.818+0000 7f836bfff700 1 -- 172.21.15.90:0/2925376967 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f83840fd670 con 0x7f838406b140 2024-04-14T15:18:20.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.818+0000 7f836bfff700 10 client.49114.objecter ms_dispatch 0x7f8384001550 command_reply(tid 3: 0 ) 2024-04-14T15:18:20.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.818+0000 7f836bfff700 10 client.49114.objecter _finish_command 3 = system:0 2024-04-14T15:18:20.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.818+0000 7f836bfff700 20 client.49114.objecter put_session s=0x7f83841aa000 osd=1 4 2024-04-14T15:18:20.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.818+0000 7f836bfff700 15 client.49114.objecter _session_command_op_remove 1 3 2024-04-14T15:18:20.823 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:20.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:20.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:18:20.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.835+0000 7f838950b700 10 client.49114.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:18:20.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.835+0000 7f838950b700 15 client.49114.objecter _session_command_op_assign -1 4 2024-04-14T15:18:20.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.835+0000 7f838950b700 20 client.49114.objecter _get_session s=0x7f83841aa000 osd=1 3 2024-04-14T15:18:20.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.835+0000 7f838950b700 20 client.49114.objecter put_session s=0x7f83841aa000 osd=1 3 2024-04-14T15:18:20.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.835+0000 7f838950b700 20 client.49114.objecter _get_session s=0x7f83841aa000 osd=1 3 2024-04-14T15:18:20.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.835+0000 7f838950b700 15 client.49114.objecter _session_command_op_remove -1 4 2024-04-14T15:18:20.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.835+0000 7f838950b700 20 client.49114.objecter get_session s=0x7f83841aa000 osd=1 3 2024-04-14T15:18:20.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.835+0000 7f838950b700 15 client.49114.objecter _session_command_op_assign 1 4 2024-04-14T15:18:20.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.835+0000 7f838950b700 20 client.49114.objecter put_session s=0x7f83841aa000 osd=1 4 2024-04-14T15:18:20.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.835+0000 7f838950b700 10 client.49114.objecter _send_command 4 2024-04-14T15:18:20.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.835+0000 7f838950b700 1 -- 172.21.15.90:0/2925376967 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f838406a6b0 con 0x7f838406b140 2024-04-14T15:18:20.837 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:18:20.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.836+0000 7f836bfff700 1 -- 172.21.15.90:0/2925376967 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f838406a6b0 con 0x7f838406b140 2024-04-14T15:18:20.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.836+0000 7f836bfff700 10 client.49114.objecter ms_dispatch 0x7f8384001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:18:20.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.836+0000 7f836bfff700 10 client.49114.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:18:20.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.836+0000 7f836bfff700 20 client.49114.objecter put_session s=0x7f83841aa000 osd=1 4 2024-04-14T15:18:20.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.836+0000 7f836bfff700 15 client.49114.objecter _session_command_op_remove 1 4 2024-04-14T15:18:20.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.837+0000 7f838950b700 10 client.49114.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:18:20.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.837+0000 7f838950b700 15 client.49114.objecter _session_command_op_assign -1 5 2024-04-14T15:18:20.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.837+0000 7f838950b700 20 client.49114.objecter _get_session s=0x7f83840fe9d0 osd=2 3 2024-04-14T15:18:20.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.837+0000 7f838950b700 20 client.49114.objecter put_session s=0x7f83840fe9d0 osd=2 3 2024-04-14T15:18:20.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.837+0000 7f838950b700 20 client.49114.objecter _get_session s=0x7f83840fe9d0 osd=2 3 2024-04-14T15:18:20.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.837+0000 7f838950b700 15 client.49114.objecter _session_command_op_remove -1 5 2024-04-14T15:18:20.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.837+0000 7f838950b700 20 client.49114.objecter get_session s=0x7f83840fe9d0 osd=2 3 2024-04-14T15:18:20.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.837+0000 7f838950b700 15 client.49114.objecter _session_command_op_assign 2 5 2024-04-14T15:18:20.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.837+0000 7f838950b700 20 client.49114.objecter put_session s=0x7f83840fe9d0 osd=2 4 2024-04-14T15:18:20.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.837+0000 7f838950b700 10 client.49114.objecter _send_command 5 2024-04-14T15:18:20.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.837+0000 7f838950b700 1 -- 172.21.15.90:0/2925376967 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f8384063bd0 con 0x7f83840feb40 2024-04-14T15:18:20.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.837+0000 7f836bfff700 10 client.49114.objecter ms_handle_connect 0x7f83840feb40 2024-04-14T15:18:20.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.838+0000 7f836bfff700 1 -- 172.21.15.90:0/2925376967 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f8384063bd0 con 0x7f83840feb40 2024-04-14T15:18:20.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.838+0000 7f836bfff700 10 client.49114.objecter ms_dispatch 0x7f8384001550 command_reply(tid 5: 0 ) 2024-04-14T15:18:20.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.838+0000 7f836bfff700 10 client.49114.objecter _finish_command 5 = system:0 2024-04-14T15:18:20.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.838+0000 7f836bfff700 20 client.49114.objecter put_session s=0x7f83840fe9d0 osd=2 4 2024-04-14T15:18:20.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.838+0000 7f836bfff700 15 client.49114.objecter _session_command_op_remove 2 5 2024-04-14T15:18:20.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.854+0000 7f838950b700 10 client.49114.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:18:20.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.854+0000 7f838950b700 15 client.49114.objecter _session_command_op_assign -1 6 2024-04-14T15:18:20.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.854+0000 7f838950b700 20 client.49114.objecter _get_session s=0x7f83840fe9d0 osd=2 3 2024-04-14T15:18:20.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.854+0000 7f838950b700 20 client.49114.objecter put_session s=0x7f83840fe9d0 osd=2 3 2024-04-14T15:18:20.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.854+0000 7f838950b700 20 client.49114.objecter _get_session s=0x7f83840fe9d0 osd=2 3 2024-04-14T15:18:20.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.854+0000 7f838950b700 15 client.49114.objecter _session_command_op_remove -1 6 2024-04-14T15:18:20.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.854+0000 7f838950b700 20 client.49114.objecter get_session s=0x7f83840fe9d0 osd=2 3 2024-04-14T15:18:20.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.854+0000 7f838950b700 15 client.49114.objecter _session_command_op_assign 2 6 2024-04-14T15:18:20.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.854+0000 7f838950b700 20 client.49114.objecter put_session s=0x7f83840fe9d0 osd=2 4 2024-04-14T15:18:20.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.854+0000 7f838950b700 10 client.49114.objecter _send_command 6 2024-04-14T15:18:20.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.854+0000 7f838950b700 1 -- 172.21.15.90:0/2925376967 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f83841b5fe0 con 0x7f83840feb40 2024-04-14T15:18:20.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.855+0000 7f836bfff700 1 -- 172.21.15.90:0/2925376967 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f83841b5fe0 con 0x7f83840feb40 2024-04-14T15:18:20.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.855+0000 7f836bfff700 10 client.49114.objecter ms_dispatch 0x7f8384001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:18:20.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.855+0000 7f836bfff700 10 client.49114.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:18:20.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.855+0000 7f836bfff700 20 client.49114.objecter put_session s=0x7f83840fe9d0 osd=2 4 2024-04-14T15:18:20.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.855+0000 7f836bfff700 15 client.49114.objecter _session_command_op_remove 2 6 2024-04-14T15:18:20.856 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:18:20.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.855+0000 7f838950b700 10 client.49114.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:18:20.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.855+0000 7f838950b700 15 client.49114.objecter _session_command_op_assign -1 7 2024-04-14T15:18:20.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.855+0000 7f838950b700 20 client.49114.objecter _get_session s=0x7f8384063140 osd=3 3 2024-04-14T15:18:20.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.855+0000 7f838950b700 20 client.49114.objecter put_session s=0x7f8384063140 osd=3 3 2024-04-14T15:18:20.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.855+0000 7f838950b700 20 client.49114.objecter _get_session s=0x7f8384063140 osd=3 3 2024-04-14T15:18:20.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.855+0000 7f838950b700 15 client.49114.objecter _session_command_op_remove -1 7 2024-04-14T15:18:20.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.855+0000 7f838950b700 20 client.49114.objecter get_session s=0x7f8384063140 osd=3 3 2024-04-14T15:18:20.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.855+0000 7f838950b700 15 client.49114.objecter _session_command_op_assign 3 7 2024-04-14T15:18:20.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.855+0000 7f838950b700 20 client.49114.objecter put_session s=0x7f8384063140 osd=3 4 2024-04-14T15:18:20.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.855+0000 7f838950b700 10 client.49114.objecter _send_command 7 2024-04-14T15:18:20.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.855+0000 7f838950b700 1 -- 172.21.15.90:0/2925376967 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f83840fe530 con 0x7f83840632b0 2024-04-14T15:18:20.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.856+0000 7f836bfff700 10 client.49114.objecter ms_handle_connect 0x7f83840632b0 2024-04-14T15:18:20.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.856+0000 7f836bfff700 1 -- 172.21.15.90:0/2925376967 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f83840fe530 con 0x7f83840632b0 2024-04-14T15:18:20.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.856+0000 7f836bfff700 10 client.49114.objecter ms_dispatch 0x7f8384001550 command_reply(tid 7: 0 ) 2024-04-14T15:18:20.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.856+0000 7f836bfff700 10 client.49114.objecter _finish_command 7 = system:0 2024-04-14T15:18:20.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.856+0000 7f836bfff700 20 client.49114.objecter put_session s=0x7f8384063140 osd=3 4 2024-04-14T15:18:20.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.856+0000 7f836bfff700 15 client.49114.objecter _session_command_op_remove 3 7 2024-04-14T15:18:20.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.873+0000 7f838950b700 10 client.49114.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:18:20.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.873+0000 7f838950b700 15 client.49114.objecter _session_command_op_assign -1 8 2024-04-14T15:18:20.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.873+0000 7f838950b700 20 client.49114.objecter _get_session s=0x7f8384063140 osd=3 3 2024-04-14T15:18:20.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.873+0000 7f838950b700 20 client.49114.objecter put_session s=0x7f8384063140 osd=3 3 2024-04-14T15:18:20.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.873+0000 7f838950b700 20 client.49114.objecter _get_session s=0x7f8384063140 osd=3 3 2024-04-14T15:18:20.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.873+0000 7f838950b700 15 client.49114.objecter _session_command_op_remove -1 8 2024-04-14T15:18:20.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.873+0000 7f838950b700 20 client.49114.objecter get_session s=0x7f8384063140 osd=3 3 2024-04-14T15:18:20.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.873+0000 7f838950b700 15 client.49114.objecter _session_command_op_assign 3 8 2024-04-14T15:18:20.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.873+0000 7f838950b700 20 client.49114.objecter put_session s=0x7f8384063140 osd=3 4 2024-04-14T15:18:20.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.873+0000 7f838950b700 10 client.49114.objecter _send_command 8 2024-04-14T15:18:20.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.873+0000 7f838950b700 1 -- 172.21.15.90:0/2925376967 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f83841baaa0 con 0x7f83840632b0 2024-04-14T15:18:20.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.875+0000 7f836bfff700 1 -- 172.21.15.90:0/2925376967 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f83841baaa0 con 0x7f83840632b0 2024-04-14T15:18:20.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.875+0000 7f836bfff700 10 client.49114.objecter ms_dispatch 0x7f8384001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:18:20.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.875+0000 7f836bfff700 10 client.49114.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:18:20.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.875+0000 7f836bfff700 20 client.49114.objecter put_session s=0x7f8384063140 osd=3 4 2024-04-14T15:18:20.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.875+0000 7f836bfff700 15 client.49114.objecter _session_command_op_remove 3 8 2024-04-14T15:18:20.876 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:18:20.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.875+0000 7f838950b700 10 client.49114.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:18:20.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.875+0000 7f838950b700 15 client.49114.objecter _session_command_op_assign -1 9 2024-04-14T15:18:20.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.875+0000 7f838950b700 20 client.49114.objecter _get_session s=0x7f83841baaa0 osd=4 3 2024-04-14T15:18:20.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.875+0000 7f838950b700 20 client.49114.objecter put_session s=0x7f83841baaa0 osd=4 3 2024-04-14T15:18:20.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.875+0000 7f838950b700 20 client.49114.objecter _get_session s=0x7f83841baaa0 osd=4 3 2024-04-14T15:18:20.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.875+0000 7f838950b700 15 client.49114.objecter _session_command_op_remove -1 9 2024-04-14T15:18:20.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.875+0000 7f838950b700 20 client.49114.objecter get_session s=0x7f83841baaa0 osd=4 3 2024-04-14T15:18:20.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.875+0000 7f838950b700 15 client.49114.objecter _session_command_op_assign 4 9 2024-04-14T15:18:20.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.875+0000 7f838950b700 20 client.49114.objecter put_session s=0x7f83841baaa0 osd=4 4 2024-04-14T15:18:20.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.875+0000 7f838950b700 10 client.49114.objecter _send_command 9 2024-04-14T15:18:20.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.875+0000 7f838950b700 1 -- 172.21.15.90:0/2925376967 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f83841bed70 con 0x7f83841bb120 2024-04-14T15:18:20.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.875+0000 7f836bfff700 10 client.49114.objecter ms_handle_connect 0x7f83841bb120 2024-04-14T15:18:20.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.876+0000 7f836bfff700 1 -- 172.21.15.90:0/2925376967 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f83841bed70 con 0x7f83841bb120 2024-04-14T15:18:20.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.876+0000 7f836bfff700 10 client.49114.objecter ms_dispatch 0x7f8384001550 command_reply(tid 9: 0 ) 2024-04-14T15:18:20.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.876+0000 7f836bfff700 10 client.49114.objecter _finish_command 9 = system:0 2024-04-14T15:18:20.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.876+0000 7f836bfff700 20 client.49114.objecter put_session s=0x7f83841baaa0 osd=4 4 2024-04-14T15:18:20.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.876+0000 7f836bfff700 15 client.49114.objecter _session_command_op_remove 4 9 2024-04-14T15:18:20.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.892+0000 7f838950b700 10 client.49114.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:18:20.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.892+0000 7f838950b700 15 client.49114.objecter _session_command_op_assign -1 10 2024-04-14T15:18:20.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.892+0000 7f838950b700 20 client.49114.objecter _get_session s=0x7f83841baaa0 osd=4 3 2024-04-14T15:18:20.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.892+0000 7f838950b700 20 client.49114.objecter put_session s=0x7f83841baaa0 osd=4 3 2024-04-14T15:18:20.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.892+0000 7f838950b700 20 client.49114.objecter _get_session s=0x7f83841baaa0 osd=4 3 2024-04-14T15:18:20.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.892+0000 7f838950b700 15 client.49114.objecter _session_command_op_remove -1 10 2024-04-14T15:18:20.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.892+0000 7f838950b700 20 client.49114.objecter get_session s=0x7f83841baaa0 osd=4 3 2024-04-14T15:18:20.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.892+0000 7f838950b700 15 client.49114.objecter _session_command_op_assign 4 10 2024-04-14T15:18:20.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.892+0000 7f838950b700 20 client.49114.objecter put_session s=0x7f83841baaa0 osd=4 4 2024-04-14T15:18:20.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.892+0000 7f838950b700 10 client.49114.objecter _send_command 10 2024-04-14T15:18:20.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.892+0000 7f838950b700 1 -- 172.21.15.90:0/2925376967 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f83841c0720 con 0x7f83841bb120 2024-04-14T15:18:20.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.894+0000 7f836bfff700 1 -- 172.21.15.90:0/2925376967 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f83841c0720 con 0x7f83841bb120 2024-04-14T15:18:20.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.894+0000 7f836bfff700 10 client.49114.objecter ms_dispatch 0x7f8384001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:18:20.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.894+0000 7f836bfff700 10 client.49114.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:18:20.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.894+0000 7f836bfff700 20 client.49114.objecter put_session s=0x7f83841baaa0 osd=4 4 2024-04-14T15:18:20.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.894+0000 7f836bfff700 15 client.49114.objecter _session_command_op_remove 4 10 2024-04-14T15:18:20.895 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:18:20.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.894+0000 7f838950b700 10 client.49114.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:18:20.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.894+0000 7f838950b700 15 client.49114.objecter _session_command_op_assign -1 11 2024-04-14T15:18:20.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.894+0000 7f838950b700 20 client.49114.objecter _get_session s=0x7f83841c0720 osd=5 3 2024-04-14T15:18:20.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.894+0000 7f838950b700 20 client.49114.objecter put_session s=0x7f83841c0720 osd=5 3 2024-04-14T15:18:20.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.894+0000 7f838950b700 20 client.49114.objecter _get_session s=0x7f83841c0720 osd=5 3 2024-04-14T15:18:20.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.894+0000 7f838950b700 15 client.49114.objecter _session_command_op_remove -1 11 2024-04-14T15:18:20.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.895+0000 7f838950b700 20 client.49114.objecter get_session s=0x7f83841c0720 osd=5 3 2024-04-14T15:18:20.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.895+0000 7f838950b700 15 client.49114.objecter _session_command_op_assign 5 11 2024-04-14T15:18:20.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.895+0000 7f838950b700 20 client.49114.objecter put_session s=0x7f83841c0720 osd=5 4 2024-04-14T15:18:20.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.895+0000 7f838950b700 10 client.49114.objecter _send_command 11 2024-04-14T15:18:20.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.895+0000 7f838950b700 1 -- 172.21.15.90:0/2925376967 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f83841c12a0 con 0x7f83841c0e40 2024-04-14T15:18:20.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.895+0000 7f836bfff700 10 client.49114.objecter ms_handle_connect 0x7f83841c0e40 2024-04-14T15:18:20.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.896+0000 7f836bfff700 1 -- 172.21.15.90:0/2925376967 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f83841c12a0 con 0x7f83841c0e40 2024-04-14T15:18:20.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.896+0000 7f836bfff700 10 client.49114.objecter ms_dispatch 0x7f8384001550 command_reply(tid 11: 0 ) 2024-04-14T15:18:20.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.896+0000 7f836bfff700 10 client.49114.objecter _finish_command 11 = system:0 2024-04-14T15:18:20.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.896+0000 7f836bfff700 20 client.49114.objecter put_session s=0x7f83841c0720 osd=5 4 2024-04-14T15:18:20.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.896+0000 7f836bfff700 15 client.49114.objecter _session_command_op_remove 5 11 2024-04-14T15:18:20.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.914+0000 7f838950b700 10 client.49114.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:18:20.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.914+0000 7f838950b700 15 client.49114.objecter _session_command_op_assign -1 12 2024-04-14T15:18:20.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.914+0000 7f838950b700 20 client.49114.objecter _get_session s=0x7f83841c0720 osd=5 3 2024-04-14T15:18:20.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.914+0000 7f838950b700 20 client.49114.objecter put_session s=0x7f83841c0720 osd=5 3 2024-04-14T15:18:20.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.914+0000 7f838950b700 20 client.49114.objecter _get_session s=0x7f83841c0720 osd=5 3 2024-04-14T15:18:20.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.914+0000 7f838950b700 15 client.49114.objecter _session_command_op_remove -1 12 2024-04-14T15:18:20.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.914+0000 7f838950b700 20 client.49114.objecter get_session s=0x7f83841c0720 osd=5 3 2024-04-14T15:18:20.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.914+0000 7f838950b700 15 client.49114.objecter _session_command_op_assign 5 12 2024-04-14T15:18:20.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.914+0000 7f838950b700 20 client.49114.objecter put_session s=0x7f83841c0720 osd=5 4 2024-04-14T15:18:20.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.914+0000 7f838950b700 10 client.49114.objecter _send_command 12 2024-04-14T15:18:20.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.914+0000 7f838950b700 1 -- 172.21.15.90:0/2925376967 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f83841c14d0 con 0x7f83841c0e40 2024-04-14T15:18:20.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.916+0000 7f836bfff700 1 -- 172.21.15.90:0/2925376967 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f83841c14d0 con 0x7f83841c0e40 2024-04-14T15:18:20.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.916+0000 7f836bfff700 10 client.49114.objecter ms_dispatch 0x7f8384001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:18:20.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.916+0000 7f836bfff700 10 client.49114.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:18:20.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.916+0000 7f836bfff700 20 client.49114.objecter put_session s=0x7f83841c0720 osd=5 4 2024-04-14T15:18:20.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.916+0000 7f836bfff700 15 client.49114.objecter _session_command_op_remove 5 12 2024-04-14T15:18:20.917 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:18:20.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.916+0000 7f838950b700 10 client.49114.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:18:20.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.916+0000 7f838950b700 15 client.49114.objecter _session_command_op_assign -1 13 2024-04-14T15:18:20.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.916+0000 7f838950b700 20 client.49114.objecter _get_session s=0x7f83841c1300 osd=6 3 2024-04-14T15:18:20.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.916+0000 7f838950b700 20 client.49114.objecter put_session s=0x7f83841c1300 osd=6 3 2024-04-14T15:18:20.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.916+0000 7f838950b700 20 client.49114.objecter _get_session s=0x7f83841c1300 osd=6 3 2024-04-14T15:18:20.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.916+0000 7f838950b700 15 client.49114.objecter _session_command_op_remove -1 13 2024-04-14T15:18:20.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.916+0000 7f838950b700 20 client.49114.objecter get_session s=0x7f83841c1300 osd=6 3 2024-04-14T15:18:20.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.916+0000 7f838950b700 15 client.49114.objecter _session_command_op_assign 6 13 2024-04-14T15:18:20.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.916+0000 7f838950b700 20 client.49114.objecter put_session s=0x7f83841c1300 osd=6 4 2024-04-14T15:18:20.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.916+0000 7f838950b700 10 client.49114.objecter _send_command 13 2024-04-14T15:18:20.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.916+0000 7f838950b700 1 -- 172.21.15.90:0/2925376967 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f83841c7790 con 0x7f83841c14d0 2024-04-14T15:18:20.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.917+0000 7f836bfff700 10 client.49114.objecter ms_handle_connect 0x7f83841c14d0 2024-04-14T15:18:20.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.917+0000 7f836bfff700 1 -- 172.21.15.90:0/2925376967 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f83841c7790 con 0x7f83841c14d0 2024-04-14T15:18:20.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.917+0000 7f836bfff700 10 client.49114.objecter ms_dispatch 0x7f8384001550 command_reply(tid 13: 0 ) 2024-04-14T15:18:20.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.917+0000 7f836bfff700 10 client.49114.objecter _finish_command 13 = system:0 2024-04-14T15:18:20.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.917+0000 7f836bfff700 20 client.49114.objecter put_session s=0x7f83841c1300 osd=6 4 2024-04-14T15:18:20.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.917+0000 7f836bfff700 15 client.49114.objecter _session_command_op_remove 6 13 2024-04-14T15:18:20.925 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:18:20.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.934+0000 7f838950b700 10 client.49114.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:18:20.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.934+0000 7f838950b700 15 client.49114.objecter _session_command_op_assign -1 14 2024-04-14T15:18:20.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.934+0000 7f838950b700 20 client.49114.objecter _get_session s=0x7f83841c1300 osd=6 3 2024-04-14T15:18:20.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.934+0000 7f838950b700 20 client.49114.objecter put_session s=0x7f83841c1300 osd=6 3 2024-04-14T15:18:20.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.934+0000 7f838950b700 20 client.49114.objecter _get_session s=0x7f83841c1300 osd=6 3 2024-04-14T15:18:20.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.934+0000 7f838950b700 15 client.49114.objecter _session_command_op_remove -1 14 2024-04-14T15:18:20.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.934+0000 7f838950b700 20 client.49114.objecter get_session s=0x7f83841c1300 osd=6 3 2024-04-14T15:18:20.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.934+0000 7f838950b700 15 client.49114.objecter _session_command_op_assign 6 14 2024-04-14T15:18:20.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.934+0000 7f838950b700 20 client.49114.objecter put_session s=0x7f83841c1300 osd=6 4 2024-04-14T15:18:20.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.934+0000 7f838950b700 10 client.49114.objecter _send_command 14 2024-04-14T15:18:20.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.934+0000 7f838950b700 1 -- 172.21.15.90:0/2925376967 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f83840fe740 con 0x7f83841c14d0 2024-04-14T15:18:20.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.935+0000 7f836bfff700 1 -- 172.21.15.90:0/2925376967 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f83840fe740 con 0x7f83841c14d0 2024-04-14T15:18:20.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.935+0000 7f836bfff700 10 client.49114.objecter ms_dispatch 0x7f8384001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:18:20.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.935+0000 7f836bfff700 10 client.49114.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:18:20.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.935+0000 7f836bfff700 20 client.49114.objecter put_session s=0x7f83841c1300 osd=6 4 2024-04-14T15:18:20.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.935+0000 7f836bfff700 15 client.49114.objecter _session_command_op_remove 6 14 2024-04-14T15:18:20.936 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:18:20.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.936+0000 7f838950b700 10 client.49114.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:18:20.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.936+0000 7f838950b700 15 client.49114.objecter _session_command_op_assign -1 15 2024-04-14T15:18:20.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.936+0000 7f838950b700 20 client.49114.objecter _get_session s=0x7f83840fe740 osd=7 3 2024-04-14T15:18:20.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.936+0000 7f838950b700 20 client.49114.objecter put_session s=0x7f83840fe740 osd=7 3 2024-04-14T15:18:20.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.936+0000 7f838950b700 20 client.49114.objecter _get_session s=0x7f83840fe740 osd=7 3 2024-04-14T15:18:20.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.936+0000 7f838950b700 15 client.49114.objecter _session_command_op_remove -1 15 2024-04-14T15:18:20.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.936+0000 7f838950b700 20 client.49114.objecter get_session s=0x7f83840fe740 osd=7 3 2024-04-14T15:18:20.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.936+0000 7f838950b700 15 client.49114.objecter _session_command_op_assign 7 15 2024-04-14T15:18:20.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.936+0000 7f838950b700 20 client.49114.objecter put_session s=0x7f83840fe740 osd=7 4 2024-04-14T15:18:20.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.936+0000 7f838950b700 10 client.49114.objecter _send_command 15 2024-04-14T15:18:20.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.936+0000 7f838950b700 1 -- 172.21.15.90:0/2925376967 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f83841c8ee0 con 0x7f83841b6960 2024-04-14T15:18:20.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.936+0000 7f836bfff700 10 client.49114.objecter ms_handle_connect 0x7f83841b6960 2024-04-14T15:18:20.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.937+0000 7f836bfff700 1 -- 172.21.15.90:0/2925376967 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f83841c8ee0 con 0x7f83841b6960 2024-04-14T15:18:20.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.937+0000 7f836bfff700 10 client.49114.objecter ms_dispatch 0x7f8384001550 command_reply(tid 15: 0 ) 2024-04-14T15:18:20.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.937+0000 7f836bfff700 10 client.49114.objecter _finish_command 15 = system:0 2024-04-14T15:18:20.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.937+0000 7f836bfff700 20 client.49114.objecter put_session s=0x7f83840fe740 osd=7 4 2024-04-14T15:18:20.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.937+0000 7f836bfff700 15 client.49114.objecter _session_command_op_remove 7 15 2024-04-14T15:18:20.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:20.950 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:20.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:18:20.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.955+0000 7f838950b700 10 client.49114.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:18:20.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.955+0000 7f838950b700 15 client.49114.objecter _session_command_op_assign -1 16 2024-04-14T15:18:20.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.955+0000 7f838950b700 20 client.49114.objecter _get_session s=0x7f83840fe740 osd=7 3 2024-04-14T15:18:20.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.955+0000 7f838950b700 20 client.49114.objecter put_session s=0x7f83840fe740 osd=7 3 2024-04-14T15:18:20.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.955+0000 7f838950b700 20 client.49114.objecter _get_session s=0x7f83840fe740 osd=7 3 2024-04-14T15:18:20.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.955+0000 7f838950b700 15 client.49114.objecter _session_command_op_remove -1 16 2024-04-14T15:18:20.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.955+0000 7f838950b700 20 client.49114.objecter get_session s=0x7f83840fe740 osd=7 3 2024-04-14T15:18:20.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.955+0000 7f838950b700 15 client.49114.objecter _session_command_op_assign 7 16 2024-04-14T15:18:20.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.955+0000 7f838950b700 20 client.49114.objecter put_session s=0x7f83840fe740 osd=7 4 2024-04-14T15:18:20.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.955+0000 7f838950b700 10 client.49114.objecter _send_command 16 2024-04-14T15:18:20.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.955+0000 7f838950b700 1 -- 172.21.15.90:0/2925376967 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f83841c8ee0 con 0x7f83841b6960 2024-04-14T15:18:20.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.957+0000 7f836bfff700 1 -- 172.21.15.90:0/2925376967 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f83841c8ee0 con 0x7f83841b6960 2024-04-14T15:18:20.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.957+0000 7f836bfff700 10 client.49114.objecter ms_dispatch 0x7f8384001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:18:20.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.957+0000 7f836bfff700 10 client.49114.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:18:20.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.957+0000 7f836bfff700 20 client.49114.objecter put_session s=0x7f83840fe740 osd=7 4 2024-04-14T15:18:20.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.957+0000 7f836bfff700 15 client.49114.objecter _session_command_op_remove 7 16 2024-04-14T15:18:20.958 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:18:20.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.958+0000 7f838950b700 10 librados: watch_flush enter 2024-04-14T15:18:20.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.958+0000 7f838950b700 10 librados: watch_flush exit 2024-04-14T15:18:20.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.958+0000 7f838950b700 10 client.49114.objecter close_session for osd.0 2024-04-14T15:18:20.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.958+0000 7f838950b700 1 -- 172.21.15.90:0/2925376967 >> v1:172.21.15.90:6801/3761806885 conn(0x7f83841a72f0 legacy=0x7f83841a76d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:20.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.958+0000 7f838950b700 20 client.49114.objecter put_session s=0x7f83840693d0 osd=0 1 2024-04-14T15:18:20.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.958+0000 7f838950b700 10 client.49114.objecter close_session for osd.1 2024-04-14T15:18:20.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.958+0000 7f838950b700 1 -- 172.21.15.90:0/2925376967 >> v1:172.21.15.90:6805/4170361699 conn(0x7f838406b140 legacy=0x7f838402ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:20.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.958+0000 7f838950b700 20 client.49114.objecter put_session s=0x7f83841aa000 osd=1 1 2024-04-14T15:18:20.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.958+0000 7f838950b700 10 client.49114.objecter close_session for osd.2 2024-04-14T15:18:20.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.958+0000 7f838950b700 1 -- 172.21.15.90:0/2925376967 >> v1:172.21.15.90:6809/1686184251 conn(0x7f83840feb40 legacy=0x7f83841b48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:20.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.958+0000 7f838950b700 20 client.49114.objecter put_session s=0x7f83840fe9d0 osd=2 1 2024-04-14T15:18:20.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.958+0000 7f838950b700 10 client.49114.objecter close_session for osd.3 2024-04-14T15:18:20.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.958+0000 7f838950b700 1 -- 172.21.15.90:0/2925376967 >> v1:172.21.15.90:6813/2270166774 conn(0x7f83840632b0 legacy=0x7f83841b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:20.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.958+0000 7f838950b700 20 client.49114.objecter put_session s=0x7f8384063140 osd=3 1 2024-04-14T15:18:20.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.958+0000 7f838950b700 10 client.49114.objecter close_session for osd.4 2024-04-14T15:18:20.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.958+0000 7f838950b700 1 -- 172.21.15.90:0/2925376967 >> v1:172.21.15.92:6800/3887549733 conn(0x7f83841bb120 legacy=0x7f83841bd510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:20.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.958+0000 7f838950b700 20 client.49114.objecter put_session s=0x7f83841baaa0 osd=4 1 2024-04-14T15:18:20.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.958+0000 7f838950b700 10 client.49114.objecter close_session for osd.5 2024-04-14T15:18:20.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.958+0000 7f838950b700 1 -- 172.21.15.90:0/2925376967 >> v1:172.21.15.92:6804/3244561440 conn(0x7f83841c0e40 legacy=0x7f83841bef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:20.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.958+0000 7f838950b700 20 client.49114.objecter put_session s=0x7f83841c0720 osd=5 1 2024-04-14T15:18:20.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.958+0000 7f838950b700 10 client.49114.objecter close_session for osd.6 2024-04-14T15:18:20.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.958+0000 7f838950b700 1 -- 172.21.15.90:0/2925376967 >> v1:172.21.15.92:6808/636125963 conn(0x7f83841c14d0 legacy=0x7f83841c5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:20.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.958+0000 7f838950b700 20 client.49114.objecter put_session s=0x7f83841c1300 osd=6 1 2024-04-14T15:18:20.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.958+0000 7f83837fe700 1 -- 172.21.15.90:0/2925376967 reap_dead start 2024-04-14T15:18:20.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.958+0000 7f838950b700 10 client.49114.objecter close_session for osd.7 2024-04-14T15:18:20.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.958+0000 7f838950b700 1 -- 172.21.15.90:0/2925376967 >> v1:172.21.15.92:6812/1773706716 conn(0x7f83841b6960 legacy=0x7f83841c7790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:20.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.958+0000 7f83837fe700 1 -- 172.21.15.90:0/2925376967 reap_dead start 2024-04-14T15:18:20.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.958+0000 7f838950b700 20 client.49114.objecter put_session s=0x7f83840fe740 osd=7 1 2024-04-14T15:18:20.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.958+0000 7f838950b700 20 client.49114.objecter shutdown clearing up homeless session... 2024-04-14T15:18:20.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.958+0000 7f838950b700 10 client.49114.objecter successfully canceled tick 2024-04-14T15:18:20.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.958+0000 7f83837fe700 1 -- 172.21.15.90:0/2925376967 reap_dead start 2024-04-14T15:18:20.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.958+0000 7f83837fe700 1 -- 172.21.15.90:0/2925376967 reap_dead start 2024-04-14T15:18:20.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.958+0000 7f838950b700 1 -- 172.21.15.90:0/2925376967 >> v1:172.21.15.90:6800/420935427 conn(0x7f836c0735d0 legacy=0x7f836c075a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:20.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.958+0000 7f838950b700 1 -- 172.21.15.90:0/2925376967 >> v1:172.21.15.92:6789/0 conn(0x7f838410a610 legacy=0x7f838410cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:20.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.958+0000 7f838950b700 1 -- 172.21.15.90:0/2925376967 shutdown_connections 2024-04-14T15:18:20.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.958+0000 7f838950b700 1 -- 172.21.15.90:0/2925376967 >> 172.21.15.90:0/2925376967 conn(0x7f83840fd290 msgr2=0x7f8384111780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:18:20.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.958+0000 7f838950b700 1 -- 172.21.15.90:0/2925376967 shutdown_connections 2024-04-14T15:18:20.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.958+0000 7f838950b700 1 -- 172.21.15.90:0/2925376967 wait complete. 2024-04-14T15:18:20.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.958+0000 7f838950b700 1 librados: shutdown 2024-04-14T15:18:20.962 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:18:20.962 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:18:20.962 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:18:20.962 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:18:20.962 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:18:20.963 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:18:20.963 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:18:20.963 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:18:20.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.989+0000 7fbc0f7fe700 1 -- 172.21.15.90:0/3266312825 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v425) ==== 91+0+0 (unknown 1178747326 0 0) 0x7fbc1005a110 con 0x7fbc1810f3b0 2024-04-14T15:18:20.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.992+0000 7fbc2027f700 10 librados: watch_flush enter 2024-04-14T15:18:20.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.993+0000 7fbc2027f700 10 librados: watch_flush exit 2024-04-14T15:18:20.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.993+0000 7fbc2027f700 20 client.49108.objecter shutdown clearing up homeless session... 2024-04-14T15:18:20.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.993+0000 7fbc2027f700 10 client.49108.objecter successfully canceled tick 2024-04-14T15:18:20.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.993+0000 7fbc2027f700 1 -- 172.21.15.90:0/3266312825 >> v1:172.21.15.90:6800/420935427 conn(0x7fbc040715e0 legacy=0x7fbc04073aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:20.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.993+0000 7fbc2027f700 1 -- 172.21.15.90:0/3266312825 >> v1:172.21.15.92:6789/0 conn(0x7fbc1810f3b0 legacy=0x7fbc181a67a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:20.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.993+0000 7fbc2027f700 1 -- 172.21.15.90:0/3266312825 shutdown_connections 2024-04-14T15:18:20.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.993+0000 7fbc2027f700 1 -- 172.21.15.90:0/3266312825 >> 172.21.15.90:0/3266312825 conn(0x7fbc180fd280 msgr2=0x7fbc180ff670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:18:20.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.994+0000 7fbc2027f700 1 -- 172.21.15.90:0/3266312825 shutdown_connections 2024-04-14T15:18:20.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.994+0000 7fbc2027f700 1 -- 172.21.15.90:0/3266312825 wait complete. 2024-04-14T15:18:20.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:20.994+0000 7fbc2027f700 1 librados: shutdown 2024-04-14T15:18:20.995 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is unset 2024-04-14T15:18:21.050 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:18:21.073 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:21.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:21.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:18:21.098 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-04-14T15:18:21.098 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-04-14T15:18:21.098 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 0 2024-04-14T15:18:21.098 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-04-14T15:18:21.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:20 smithi092 ceph-mon[33966]: pgmap v586: 238 pgs: 238 active+clean; 8.4 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 0 B/s wr, 3 op/s 2024-04-14T15:18:21.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:20 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3266312825' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:18:21.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:20 smithi092 ceph-mon[33966]: from='client.49108 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:18:21.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:20 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2925376967' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:18:21.176 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:18:21.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:20 smithi090 ceph-mon[30399]: pgmap v586: 238 pgs: 238 active+clean; 8.4 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 0 B/s wr, 3 op/s 2024-04-14T15:18:21.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:20 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3266312825' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:18:21.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:20 smithi090 ceph-mon[30399]: from='client.49108 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:18:21.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:20 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2925376967' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:18:21.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:20 smithi090 ceph-mon[41670]: pgmap v586: 238 pgs: 238 active+clean; 8.4 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 0 B/s wr, 3 op/s 2024-04-14T15:18:21.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:20 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3266312825' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:18:21.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:20 smithi090 ceph-mon[41670]: from='client.49108 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:18:21.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:20 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2925376967' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:18:21.200 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:21.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:21.201 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:18:21.301 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:21.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:21.327 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:21.328 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:21.428 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:18:21.452 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:21.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:21.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:18:21.554 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:21.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:21.578 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:21.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:21.679 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:21.703 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:21.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:21.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:21.773 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-14T15:18:21.805 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:21.840 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:21.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:21.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:21.942 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:21.966 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:21.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:21.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:22.021 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:18:22.068 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:18:22.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:22.093 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:22.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:18:22.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:21 smithi092 ceph-mon[33966]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:18:22.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:21 smithi092 ceph-mon[33966]: from='client.49108 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-14T15:18:22.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:21 smithi092 ceph-mon[33966]: osdmap e425: 8 total, 8 up, 8 in 2024-04-14T15:18:22.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:21 smithi092 ceph-mon[33966]: 6.19 scrub starts 2024-04-14T15:18:22.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:21 smithi092 ceph-mon[33966]: 6.19 scrub ok 2024-04-14T15:18:22.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:21 smithi092 ceph-mon[33966]: pgmap v588: 238 pgs: 238 active+clean; 8.4 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-04-14T15:18:22.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:21 smithi090 ceph-mon[41670]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:18:22.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:21 smithi090 ceph-mon[41670]: from='client.49108 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-14T15:18:22.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:21 smithi090 ceph-mon[41670]: osdmap e425: 8 total, 8 up, 8 in 2024-04-14T15:18:22.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:21 smithi090 ceph-mon[41670]: 6.19 scrub starts 2024-04-14T15:18:22.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:21 smithi090 ceph-mon[41670]: 6.19 scrub ok 2024-04-14T15:18:22.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:21 smithi090 ceph-mon[41670]: pgmap v588: 238 pgs: 238 active+clean; 8.4 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-04-14T15:18:22.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:21 smithi090 ceph-mon[30399]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:18:22.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:21 smithi090 ceph-mon[30399]: from='client.49108 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-14T15:18:22.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:21 smithi090 ceph-mon[30399]: osdmap e425: 8 total, 8 up, 8 in 2024-04-14T15:18:22.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:21 smithi090 ceph-mon[30399]: 6.19 scrub starts 2024-04-14T15:18:22.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:21 smithi090 ceph-mon[30399]: 6.19 scrub ok 2024-04-14T15:18:22.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:21 smithi090 ceph-mon[30399]: pgmap v588: 238 pgs: 238 active+clean; 8.4 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-04-14T15:18:22.194 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:18:22.217 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:22.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:22.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:18:22.319 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:18:22.344 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:22.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:22.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:18:22.445 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:18:22.469 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:22.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:22.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:18:22.571 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:18:22.594 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:22.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:22.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:18:22.696 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:18:22.719 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:22.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:22.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:18:22.821 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:18:22.844 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:22.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:22.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:18:22.946 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:18:22.971 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:22.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:22.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:18:23.073 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:18:23.096 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:23.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:23.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:18:23.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:22 smithi092 ceph-mon[33966]: 4.1 scrub starts 2024-04-14T15:18:23.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:22 smithi092 ceph-mon[33966]: 4.1 scrub ok 2024-04-14T15:18:23.198 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:23.224 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:23.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:23.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:23.262 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:22 smithi090 ceph-mon[41670]: 4.1 scrub starts 2024-04-14T15:18:23.262 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:22 smithi090 ceph-mon[41670]: 4.1 scrub ok 2024-04-14T15:18:23.262 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:22 smithi090 ceph-mon[30399]: 4.1 scrub starts 2024-04-14T15:18:23.262 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:22 smithi090 ceph-mon[30399]: 4.1 scrub ok 2024-04-14T15:18:23.326 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:18:23.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:23.351 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:23.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:18:23.452 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:18:23.475 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:23.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:23.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:18:23.577 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:23.601 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:23.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:23.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:23.705 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:18:23.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:23.730 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:23.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:18:23.731 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-04-14T15:18:23.731 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-04-14T15:18:23.731 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-04-14T15:18:23.731 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-04-14T15:18:23.731 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-04-14T15:18:23.831 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:18:23.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:23.855 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:23.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:18:23.956 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:18:23.980 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:23.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:23.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:18:24.081 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:18:24.108 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:24.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:24.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:18:24.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:24.193+0000 7f0ff7a0b700 1 -- 172.21.15.90:0/1247607057 >> v1:172.21.15.90:6789/0 conn(0x7f0ff010f310 legacy=0x7f0ff01117d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:24.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:24.193+0000 7f0ff7a0b700 1 -- 172.21.15.90:0/1247607057 shutdown_connections 2024-04-14T15:18:24.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:24.193+0000 7f0ff7a0b700 1 -- 172.21.15.90:0/1247607057 >> 172.21.15.90:0/1247607057 conn(0x7f0ff00fe380 msgr2=0x7f0ff01007a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:18:24.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:24.193+0000 7f0ff7a0b700 1 -- 172.21.15.90:0/1247607057 shutdown_connections 2024-04-14T15:18:24.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:24.193+0000 7f0ff7a0b700 1 -- 172.21.15.90:0/1247607057 wait complete. 2024-04-14T15:18:24.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:24.194+0000 7f0ff7a0b700 1 librados: starting msgr at 2024-04-14T15:18:24.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:24.194+0000 7f0ff7a0b700 1 librados: starting objecter 2024-04-14T15:18:24.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:24.194+0000 7f0ff7a0b700 1 Processor -- start 2024-04-14T15:18:24.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:24.194+0000 7f0ff7a0b700 1 -- start start 2024-04-14T15:18:24.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:24.194+0000 7f0ff7a0b700 1 librados: setting wanted keys 2024-04-14T15:18:24.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:24.194+0000 7f0ff7a0b700 1 librados: calling monclient init 2024-04-14T15:18:24.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:24.194+0000 7f0ff7a0b700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f0ff010f060 con 0x7f0ff010b740 2024-04-14T15:18:24.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:24.194+0000 7f0ff7a0b700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f0ff01a8f20 con 0x7f0ff0108890 2024-04-14T15:18:24.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:24.194+0000 7f0ff7a0b700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f0ff01aa100 con 0x7f0ff010f310 2024-04-14T15:18:24.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:24.194+0000 7f0ff4fa6700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f0ff010b740 0x7f0ff010df30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:48974/0 (socket says 172.21.15.90:48974) 2024-04-14T15:18:24.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:24.194+0000 7f0ff4fa6700 1 -- 172.21.15.90:0/22476038 learned_addr learned my addr 172.21.15.90:0/22476038 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:18:24.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:24.194+0000 7f0fe67fc700 10 client.?.objecter ms_handle_connect 0x7f0ff010f310 2024-04-14T15:18:24.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:24.194+0000 7f0fe67fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:18:24.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:24.194+0000 7f0fe67fc700 10 client.?.objecter ms_handle_connect 0x7f0ff0108890 2024-04-14T15:18:24.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:24.194+0000 7f0fe67fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:18:24.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:24.194+0000 7f0fe67fc700 10 client.?.objecter ms_handle_connect 0x7f0ff010b740 2024-04-14T15:18:24.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:24.194+0000 7f0fe67fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:18:24.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:24.194+0000 7f0fe67fc700 1 -- 172.21.15.90:0/22476038 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 259425103 0 0) 0x7f0ff01aa100 con 0x7f0ff010f310 2024-04-14T15:18:24.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:24.194+0000 7f0fe67fc700 1 -- 172.21.15.90:0/22476038 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0fdc0035b0 con 0x7f0ff010f310 2024-04-14T15:18:24.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:24.194+0000 7f0fe67fc700 1 -- 172.21.15.90:0/22476038 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2473107131 0 0) 0x7f0ff01a8f20 con 0x7f0ff0108890 2024-04-14T15:18:24.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:24.194+0000 7f0fe67fc700 1 -- 172.21.15.90:0/22476038 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ff01aa100 con 0x7f0ff0108890 2024-04-14T15:18:24.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:24.195+0000 7f0fe67fc700 1 -- 172.21.15.90:0/22476038 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 120557421 0 0) 0x7f0ff010f060 con 0x7f0ff010b740 2024-04-14T15:18:24.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:24.195+0000 7f0fe67fc700 1 -- 172.21.15.90:0/22476038 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ff01a8f20 con 0x7f0ff010b740 2024-04-14T15:18:24.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:24.195+0000 7f0fe67fc700 1 -- 172.21.15.90:0/22476038 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3116176673 0 0) 0x7f0ff01aa100 con 0x7f0ff0108890 2024-04-14T15:18:24.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:24.195+0000 7f0fe67fc700 1 -- 172.21.15.90:0/22476038 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0ff010f060 con 0x7f0ff0108890 2024-04-14T15:18:24.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:24.195+0000 7f0fe67fc700 1 -- 172.21.15.90:0/22476038 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f0fe00018b0 con 0x7f0ff0108890 2024-04-14T15:18:24.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:24.195+0000 7f0fe67fc700 1 -- 172.21.15.90:0/22476038 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3467754741 0 0) 0x7f0fdc0035b0 con 0x7f0ff010f310 2024-04-14T15:18:24.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:24.195+0000 7f0fe67fc700 1 -- 172.21.15.90:0/22476038 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0ff01aa100 con 0x7f0ff010f310 2024-04-14T15:18:24.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:24.195+0000 7f0fe67fc700 1 -- 172.21.15.90:0/22476038 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f0fe8001bb0 con 0x7f0ff010f310 2024-04-14T15:18:24.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:24.195+0000 7f0fe67fc700 1 -- 172.21.15.90:0/22476038 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 202300959 0 0) 0x7f0ff010f060 con 0x7f0ff0108890 2024-04-14T15:18:24.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:24.195+0000 7f0fe67fc700 1 -- 172.21.15.90:0/22476038 >> v1:172.21.15.90:6790/0 conn(0x7f0ff010f310 legacy=0x7f0ff01a67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:24.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:24.195+0000 7f0fe67fc700 1 -- 172.21.15.90:0/22476038 >> v1:172.21.15.90:6789/0 conn(0x7f0ff010b740 legacy=0x7f0ff010df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:24.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:24.195+0000 7f0fe67fc700 1 -- 172.21.15.90:0/22476038 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ff01ab340 con 0x7f0ff0108890 2024-04-14T15:18:24.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:24.195+0000 7f0ff7a0b700 1 -- 172.21.15.90:0/22476038 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0ff01a9150 con 0x7f0ff0108890 2024-04-14T15:18:24.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:24.195+0000 7f0ff7a0b700 10 client.49120.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:18:24.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:24.195+0000 7f0ff7a0b700 1 -- 172.21.15.90:0/22476038 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f0ff01a9610 con 0x7f0ff0108890 2024-04-14T15:18:24.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:24.195+0000 7f0ff7a0b700 1 librados: init done 2024-04-14T15:18:24.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:24.195+0000 7f0fe67fc700 1 -- 172.21.15.90:0/22476038 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f0fe0003b40 con 0x7f0ff0108890 2024-04-14T15:18:24.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:24.195+0000 7f0fe67fc700 1 -- 172.21.15.90:0/22476038 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f0fe0004ad0 con 0x7f0ff0108890 2024-04-14T15:18:24.200 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:18:23 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:18:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:18:24.200 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:24 smithi090 ceph-mon[41670]: 1.9 scrub starts 2024-04-14T15:18:24.200 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:24 smithi090 ceph-mon[41670]: 1.9 scrub ok 2024-04-14T15:18:24.200 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:24 smithi090 ceph-mon[41670]: 5.1d scrub starts 2024-04-14T15:18:24.200 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:24 smithi090 ceph-mon[41670]: 5.1d scrub ok 2024-04-14T15:18:24.200 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:24 smithi090 ceph-mon[41670]: pgmap v589: 238 pgs: 238 active+clean; 8.4 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 993 B/s wr, 3 op/s 2024-04-14T15:18:24.200 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:24 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:18:24.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:24.196+0000 7f0ff7a0b700 1 -- 172.21.15.90:0/22476038 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fd4005150 con 0x7f0ff0108890 2024-04-14T15:18:24.201 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:24 smithi090 ceph-mon[30399]: 1.9 scrub starts 2024-04-14T15:18:24.201 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:24 smithi090 ceph-mon[30399]: 1.9 scrub ok 2024-04-14T15:18:24.201 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:24 smithi090 ceph-mon[30399]: 5.1d scrub starts 2024-04-14T15:18:24.201 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:24 smithi090 ceph-mon[30399]: 5.1d scrub ok 2024-04-14T15:18:24.201 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:24 smithi090 ceph-mon[30399]: pgmap v589: 238 pgs: 238 active+clean; 8.4 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 993 B/s wr, 3 op/s 2024-04-14T15:18:24.201 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:24 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:18:24.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:24.197+0000 7f0fe67fc700 1 -- 172.21.15.90:0/22476038 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f0fe00036a0 con 0x7f0ff0108890 2024-04-14T15:18:24.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:24.197+0000 7f0fe67fc700 1 -- 172.21.15.90:0/22476038 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(425..425 src has 51..425) ==== 8188+0+0 (unknown 1279725432 0 0) 0x7f0fe00897b0 con 0x7f0ff0108890 2024-04-14T15:18:24.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:24.197+0000 7f0fe67fc700 10 client.49120.objecter ms_dispatch 0x7f0ff0001550 osd_map(425..425 src has 51..425) 2024-04-14T15:18:24.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:24.197+0000 7f0fe67fc700 3 client.49120.objecter handle_osd_map got epochs [425,425] > 0 2024-04-14T15:18:24.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:24.197+0000 7f0fe67fc700 3 client.49120.objecter handle_osd_map decoding full epoch 425 2024-04-14T15:18:24.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:24.197+0000 7f0fe67fc700 20 client.49120.objecter dump_active .. 0 homeless 2024-04-14T15:18:24.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:24.197+0000 7f0fe67fc700 10 client.49120.objecter ms_handle_connect 0x7f0fdc0711a0 2024-04-14T15:18:24.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:24.202+0000 7f0fe67fc700 1 -- 172.21.15.90:0/22476038 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f0fe00561b0 con 0x7f0ff0108890 2024-04-14T15:18:24.210 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:18:24.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:24.234 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:24.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:18:24.335 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:18:24.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:24.358 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:24.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:18:24.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:24 smithi092 ceph-mon[33966]: 1.9 scrub starts 2024-04-14T15:18:24.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:24 smithi092 ceph-mon[33966]: 1.9 scrub ok 2024-04-14T15:18:24.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:24 smithi092 ceph-mon[33966]: 5.1d scrub starts 2024-04-14T15:18:24.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:24 smithi092 ceph-mon[33966]: 5.1d scrub ok 2024-04-14T15:18:24.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:24 smithi092 ceph-mon[33966]: pgmap v589: 238 pgs: 238 active+clean; 8.4 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 993 B/s wr, 3 op/s 2024-04-14T15:18:24.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:24 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:18:24.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:24.397+0000 7f0ff7a0b700 1 -- 172.21.15.90:0/22476038 --> v1:172.21.15.90:6800/420935427 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0fd4000bc0 con 0x7f0fdc0711a0 2024-04-14T15:18:24.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:24.413+0000 7f0fe67fc700 1 -- 172.21.15.90:0/22476038 <== mgr.14610 v1:172.21.15.90:6800/420935427 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+616025 (unknown 2965378022 0 3600758002) 0x7f0fd4000bc0 con 0x7f0fdc0711a0 2024-04-14T15:18:24.420 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:18:24.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:24.417+0000 7f0ff7a0b700 10 librados: watch_flush enter 2024-04-14T15:18:24.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:24.417+0000 7f0ff7a0b700 10 librados: watch_flush exit 2024-04-14T15:18:24.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:24.417+0000 7f0ff7a0b700 20 client.49120.objecter shutdown clearing up homeless session... 2024-04-14T15:18:24.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:24.417+0000 7f0ff7a0b700 10 client.49120.objecter successfully canceled tick 2024-04-14T15:18:24.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:24.417+0000 7f0ff7a0b700 1 -- 172.21.15.90:0/22476038 >> v1:172.21.15.90:6800/420935427 conn(0x7f0fdc0711a0 legacy=0x7f0fdc073660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:24.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:24.417+0000 7f0ff7a0b700 1 -- 172.21.15.90:0/22476038 >> v1:172.21.15.92:6789/0 conn(0x7f0ff0108890 legacy=0x7f0ff010d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:24.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:24.417+0000 7f0ff7a0b700 1 -- 172.21.15.90:0/22476038 shutdown_connections 2024-04-14T15:18:24.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:24.417+0000 7f0ff7a0b700 1 -- 172.21.15.90:0/22476038 >> 172.21.15.90:0/22476038 conn(0x7f0ff00fe380 msgr2=0x7f0ff0102890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:18:24.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:24.417+0000 7f0ff7a0b700 1 -- 172.21.15.90:0/22476038 shutdown_connections 2024-04-14T15:18:24.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:24.417+0000 7f0ff7a0b700 1 -- 172.21.15.90:0/22476038 wait complete. 2024-04-14T15:18:24.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:24.418+0000 7f0ff7a0b700 1 librados: shutdown 2024-04-14T15:18:24.426 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-14T15:18:24.460 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:18:24.485 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:24.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:24.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:18:24.539 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-14T15:18:24.587 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:18:24.610 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:24.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:24.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:18:24.647 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-14T15:18:24.712 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:18:24.735 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:24.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:24.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:18:24.819 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:18:24.837 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:18:24.860 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:24.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:24.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:18:24.962 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:18:24.988 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:24.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:24.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:18:25.060 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-14T15:18:25.089 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:18:25.114 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:25.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:25.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:18:25.168 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":589,"stamp":"2024-04-14T15:18:23.076213+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8792204,"num_objects":364,"num_object_clones":4,"num_object_copies":1092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":364,"num_whiteouts":0,"num_read":1352,"num_read_kb":1002,"num_write":1052,"num_write_kb":12744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":9,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3898,"ondisk_log_size":3898,"up":714,"acting":714,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":714,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":598116,"kb_used_data":83388,"kb_used_omap":12,"kb_used_meta":514483,"kb_avail":749363100,"statfs":{"total":767960285184,"available":767347814400,"internally_reserved":0,"allocated":85389312,"data_stored":59181806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":526831154},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":10,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":11,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.181356"},"pg_stats":[{"pgid":"297.1f","version":"0'0","reported_seq":21,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.905805+0000","last_change":"2024-04-14T15:18:13.005585+0000","last_active":"2024-04-14T15:18:17.905805+0000","last_peered":"2024-04-14T15:18:17.905805+0000","last_clean":"2024-04-14T15:18:17.905805+0000","last_became_active":"2024-04-14T15:18:13.005337+0000","last_became_peered":"2024-04-14T15:18:13.005337+0000","last_unstale":"2024-04-14T15:18:17.905805+0000","last_undegraded":"2024-04-14T15:18:17.905805+0000","last_fullsized":"2024-04-14T15:18:17.905805+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:21.146893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"297.1e","version":"0'0","reported_seq":21,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:18.543424+0000","last_change":"2024-04-14T15:18:13.005874+0000","last_active":"2024-04-14T15:18:18.543424+0000","last_peered":"2024-04-14T15:18:18.543424+0000","last_clean":"2024-04-14T15:18:18.543424+0000","last_became_active":"2024-04-14T15:18:13.005571+0000","last_became_peered":"2024-04-14T15:18:13.005571+0000","last_unstale":"2024-04-14T15:18:18.543424+0000","last_undegraded":"2024-04-14T15:18:18.543424+0000","last_fullsized":"2024-04-14T15:18:18.543424+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:23.548046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"297.1d","version":"0'0","reported_seq":21,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.905843+0000","last_change":"2024-04-14T15:18:12.947350+0000","last_active":"2024-04-14T15:18:17.905843+0000","last_peered":"2024-04-14T15:18:17.905843+0000","last_clean":"2024-04-14T15:18:17.905843+0000","last_became_active":"2024-04-14T15:18:12.947006+0000","last_became_peered":"2024-04-14T15:18:12.947006+0000","last_unstale":"2024-04-14T15:18:17.905843+0000","last_undegraded":"2024-04-14T15:18:17.905843+0000","last_fullsized":"2024-04-14T15:18:17.905843+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:27.508637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"297.1c","version":"424'4","reported_seq":28,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.898213+0000","last_change":"2024-04-14T15:18:12.955879+0000","last_active":"2024-04-14T15:18:17.898213+0000","last_peered":"2024-04-14T15:18:17.898213+0000","last_clean":"2024-04-14T15:18:17.898213+0000","last_became_active":"2024-04-14T15:18:12.955448+0000","last_became_peered":"2024-04-14T15:18:12.955448+0000","last_unstale":"2024-04-14T15:18:17.898213+0000","last_undegraded":"2024-04-14T15:18:17.898213+0000","last_fullsized":"2024-04-14T15:18:17.898213+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:23.726057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"297.1b","version":"0'0","reported_seq":21,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.898684+0000","last_change":"2024-04-14T15:18:12.951404+0000","last_active":"2024-04-14T15:18:17.898684+0000","last_peered":"2024-04-14T15:18:17.898684+0000","last_clean":"2024-04-14T15:18:17.898684+0000","last_became_active":"2024-04-14T15:18:12.951112+0000","last_became_peered":"2024-04-14T15:18:12.951112+0000","last_unstale":"2024-04-14T15:18:17.898684+0000","last_undegraded":"2024-04-14T15:18:17.898684+0000","last_fullsized":"2024-04-14T15:18:17.898684+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:14.892479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"297.1a","version":"0'0","reported_seq":21,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.897847+0000","last_change":"2024-04-14T15:18:12.956068+0000","last_active":"2024-04-14T15:18:17.897847+0000","last_peered":"2024-04-14T15:18:17.897847+0000","last_clean":"2024-04-14T15:18:17.897847+0000","last_became_active":"2024-04-14T15:18:12.955801+0000","last_became_peered":"2024-04-14T15:18:12.955801+0000","last_unstale":"2024-04-14T15:18:17.897847+0000","last_undegraded":"2024-04-14T15:18:17.897847+0000","last_fullsized":"2024-04-14T15:18:17.897847+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:15.388092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"297.19","version":"0'0","reported_seq":21,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:18.542110+0000","last_change":"2024-04-14T15:18:12.955167+0000","last_active":"2024-04-14T15:18:18.542110+0000","last_peered":"2024-04-14T15:18:18.542110+0000","last_clean":"2024-04-14T15:18:18.542110+0000","last_became_active":"2024-04-14T15:18:12.954867+0000","last_became_peered":"2024-04-14T15:18:12.954867+0000","last_unstale":"2024-04-14T15:18:18.542110+0000","last_undegraded":"2024-04-14T15:18:18.542110+0000","last_fullsized":"2024-04-14T15:18:18.542110+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:32.831836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"297.18","version":"0'0","reported_seq":21,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:18.541921+0000","last_change":"2024-04-14T15:18:12.950510+0000","last_active":"2024-04-14T15:18:18.541921+0000","last_peered":"2024-04-14T15:18:18.541921+0000","last_clean":"2024-04-14T15:18:18.541921+0000","last_became_active":"2024-04-14T15:18:12.950312+0000","last_became_peered":"2024-04-14T15:18:12.950312+0000","last_unstale":"2024-04-14T15:18:18.541921+0000","last_undegraded":"2024-04-14T15:18:18.541921+0000","last_fullsized":"2024-04-14T15:18:18.541921+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:14.281458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"297.17","version":"0'0","reported_seq":21,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:18.543265+0000","last_change":"2024-04-14T15:18:13.006758+0000","last_active":"2024-04-14T15:18:18.543265+0000","last_peered":"2024-04-14T15:18:18.543265+0000","last_clean":"2024-04-14T15:18:18.543265+0000","last_became_active":"2024-04-14T15:18:13.006469+0000","last_became_peered":"2024-04-14T15:18:13.006469+0000","last_unstale":"2024-04-14T15:18:18.543265+0000","last_undegraded":"2024-04-14T15:18:18.543265+0000","last_fullsized":"2024-04-14T15:18:18.543265+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:41.665082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"297.16","version":"0'0","reported_seq":21,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.908285+0000","last_change":"2024-04-14T15:18:12.951467+0000","last_active":"2024-04-14T15:18:17.908285+0000","last_peered":"2024-04-14T15:18:17.908285+0000","last_clean":"2024-04-14T15:18:17.908285+0000","last_became_active":"2024-04-14T15:18:12.950999+0000","last_became_peered":"2024-04-14T15:18:12.950999+0000","last_unstale":"2024-04-14T15:18:17.908285+0000","last_undegraded":"2024-04-14T15:18:17.908285+0000","last_fullsized":"2024-04-14T15:18:17.908285+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:15.500639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"297.15","version":"0'0","reported_seq":21,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.909520+0000","last_change":"2024-04-14T15:18:13.004242+0000","last_active":"2024-04-14T15:18:17.909520+0000","last_peered":"2024-04-14T15:18:17.909520+0000","last_clean":"2024-04-14T15:18:17.909520+0000","last_became_active":"2024-04-14T15:18:13.003888+0000","last_became_peered":"2024-04-14T15:18:13.003888+0000","last_unstale":"2024-04-14T15:18:17.909520+0000","last_undegraded":"2024-04-14T15:18:17.909520+0000","last_fullsized":"2024-04-14T15:18:17.909520+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:29.490276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"297.14","version":"0'0","reported_seq":21,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.898183+0000","last_change":"2024-04-14T15:18:13.005434+0000","last_active":"2024-04-14T15:18:17.898183+0000","last_peered":"2024-04-14T15:18:17.898183+0000","last_clean":"2024-04-14T15:18:17.898183+0000","last_became_active":"2024-04-14T15:18:13.005051+0000","last_became_peered":"2024-04-14T15:18:13.005051+0000","last_unstale":"2024-04-14T15:18:17.898183+0000","last_undegraded":"2024-04-14T15:18:17.898183+0000","last_fullsized":"2024-04-14T15:18:17.898183+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:14.539808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"297.13","version":"0'0","reported_seq":21,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.899964+0000","last_change":"2024-04-14T15:18:13.004437+0000","last_active":"2024-04-14T15:18:17.899964+0000","last_peered":"2024-04-14T15:18:17.899964+0000","last_clean":"2024-04-14T15:18:17.899964+0000","last_became_active":"2024-04-14T15:18:13.004133+0000","last_became_peered":"2024-04-14T15:18:13.004133+0000","last_unstale":"2024-04-14T15:18:17.899964+0000","last_undegraded":"2024-04-14T15:18:17.899964+0000","last_fullsized":"2024-04-14T15:18:17.899964+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:32.102952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"297.12","version":"0'0","reported_seq":21,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.907755+0000","last_change":"2024-04-14T15:18:12.956427+0000","last_active":"2024-04-14T15:18:17.907755+0000","last_peered":"2024-04-14T15:18:17.907755+0000","last_clean":"2024-04-14T15:18:17.907755+0000","last_became_active":"2024-04-14T15:18:12.955274+0000","last_became_peered":"2024-04-14T15:18:12.955274+0000","last_unstale":"2024-04-14T15:18:17.907755+0000","last_undegraded":"2024-04-14T15:18:17.907755+0000","last_fullsized":"2024-04-14T15:18:17.907755+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:20.629003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"297.11","version":"0'0","reported_seq":21,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.905531+0000","last_change":"2024-04-14T15:18:12.956974+0000","last_active":"2024-04-14T15:18:17.905531+0000","last_peered":"2024-04-14T15:18:17.905531+0000","last_clean":"2024-04-14T15:18:17.905531+0000","last_became_active":"2024-04-14T15:18:12.955819+0000","last_became_peered":"2024-04-14T15:18:12.955819+0000","last_unstale":"2024-04-14T15:18:17.905531+0000","last_undegraded":"2024-04-14T15:18:17.905531+0000","last_fullsized":"2024-04-14T15:18:17.905531+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:23.104281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"297.10","version":"0'0","reported_seq":21,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.897651+0000","last_change":"2024-04-14T15:18:13.006273+0000","last_active":"2024-04-14T15:18:17.897651+0000","last_peered":"2024-04-14T15:18:17.897651+0000","last_clean":"2024-04-14T15:18:17.897651+0000","last_became_active":"2024-04-14T15:18:13.005968+0000","last_became_peered":"2024-04-14T15:18:13.005968+0000","last_unstale":"2024-04-14T15:18:17.897651+0000","last_undegraded":"2024-04-14T15:18:17.897651+0000","last_fullsized":"2024-04-14T15:18:17.897651+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:31.460743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"297.f","version":"0'0","reported_seq":21,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.897972+0000","last_change":"2024-04-14T15:18:13.005801+0000","last_active":"2024-04-14T15:18:17.897972+0000","last_peered":"2024-04-14T15:18:17.897972+0000","last_clean":"2024-04-14T15:18:17.897972+0000","last_became_active":"2024-04-14T15:18:13.005146+0000","last_became_peered":"2024-04-14T15:18:13.005146+0000","last_unstale":"2024-04-14T15:18:17.897972+0000","last_undegraded":"2024-04-14T15:18:17.897972+0000","last_fullsized":"2024-04-14T15:18:17.897972+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:34.836627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"297.e","version":"420'1","reported_seq":22,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.909624+0000","last_change":"2024-04-14T15:18:12.948109+0000","last_active":"2024-04-14T15:18:17.909624+0000","last_peered":"2024-04-14T15:18:17.909624+0000","last_clean":"2024-04-14T15:18:17.909624+0000","last_became_active":"2024-04-14T15:18:12.947811+0000","last_became_peered":"2024-04-14T15:18:12.947811+0000","last_unstale":"2024-04-14T15:18:17.909624+0000","last_undegraded":"2024-04-14T15:18:17.909624+0000","last_fullsized":"2024-04-14T15:18:17.909624+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:14.922501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"297.d","version":"0'0","reported_seq":21,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:18.543059+0000","last_change":"2024-04-14T15:18:12.949318+0000","last_active":"2024-04-14T15:18:18.543059+0000","last_peered":"2024-04-14T15:18:18.543059+0000","last_clean":"2024-04-14T15:18:18.543059+0000","last_became_active":"2024-04-14T15:18:12.949034+0000","last_became_peered":"2024-04-14T15:18:12.949034+0000","last_unstale":"2024-04-14T15:18:18.543059+0000","last_undegraded":"2024-04-14T15:18:18.543059+0000","last_fullsized":"2024-04-14T15:18:18.543059+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:22.209796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"297.c","version":"0'0","reported_seq":21,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.909225+0000","last_change":"2024-04-14T15:18:13.008426+0000","last_active":"2024-04-14T15:18:17.909225+0000","last_peered":"2024-04-14T15:18:17.909225+0000","last_clean":"2024-04-14T15:18:17.909225+0000","last_became_active":"2024-04-14T15:18:13.007723+0000","last_became_peered":"2024-04-14T15:18:13.007723+0000","last_unstale":"2024-04-14T15:18:17.909225+0000","last_undegraded":"2024-04-14T15:18:17.909225+0000","last_fullsized":"2024-04-14T15:18:17.909225+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:36.383377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"297.b","version":"0'0","reported_seq":21,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:18.543547+0000","last_change":"2024-04-14T15:18:12.951076+0000","last_active":"2024-04-14T15:18:18.543547+0000","last_peered":"2024-04-14T15:18:18.543547+0000","last_clean":"2024-04-14T15:18:18.543547+0000","last_became_active":"2024-04-14T15:18:12.950806+0000","last_became_peered":"2024-04-14T15:18:12.950806+0000","last_unstale":"2024-04-14T15:18:18.543547+0000","last_undegraded":"2024-04-14T15:18:18.543547+0000","last_fullsized":"2024-04-14T15:18:18.543547+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:22.237973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"297.a","version":"0'0","reported_seq":21,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.897847+0000","last_change":"2024-04-14T15:18:12.955873+0000","last_active":"2024-04-14T15:18:17.897847+0000","last_peered":"2024-04-14T15:18:17.897847+0000","last_clean":"2024-04-14T15:18:17.897847+0000","last_became_active":"2024-04-14T15:18:12.955571+0000","last_became_peered":"2024-04-14T15:18:12.955571+0000","last_unstale":"2024-04-14T15:18:17.897847+0000","last_undegraded":"2024-04-14T15:18:17.897847+0000","last_fullsized":"2024-04-14T15:18:17.897847+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:12.173921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"297.9","version":"0'0","reported_seq":21,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.897797+0000","last_change":"2024-04-14T15:18:13.004346+0000","last_active":"2024-04-14T15:18:17.897797+0000","last_peered":"2024-04-14T15:18:17.897797+0000","last_clean":"2024-04-14T15:18:17.897797+0000","last_became_active":"2024-04-14T15:18:13.004023+0000","last_became_peered":"2024-04-14T15:18:13.004023+0000","last_unstale":"2024-04-14T15:18:17.897797+0000","last_undegraded":"2024-04-14T15:18:17.897797+0000","last_fullsized":"2024-04-14T15:18:17.897797+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:14.413671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"297.8","version":"0'0","reported_seq":21,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.897751+0000","last_change":"2024-04-14T15:18:13.005840+0000","last_active":"2024-04-14T15:18:17.897751+0000","last_peered":"2024-04-14T15:18:17.897751+0000","last_clean":"2024-04-14T15:18:17.897751+0000","last_became_active":"2024-04-14T15:18:13.005478+0000","last_became_peered":"2024-04-14T15:18:13.005478+0000","last_unstale":"2024-04-14T15:18:17.897751+0000","last_undegraded":"2024-04-14T15:18:17.897751+0000","last_fullsized":"2024-04-14T15:18:17.897751+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:29.674300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"297.7","version":"420'2","reported_seq":24,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.910141+0000","last_change":"2024-04-14T15:18:12.955307+0000","last_active":"2024-04-14T15:18:17.910141+0000","last_peered":"2024-04-14T15:18:17.910141+0000","last_clean":"2024-04-14T15:18:17.910141+0000","last_became_active":"2024-04-14T15:18:12.955029+0000","last_became_peered":"2024-04-14T15:18:12.955029+0000","last_unstale":"2024-04-14T15:18:17.910141+0000","last_undegraded":"2024-04-14T15:18:17.910141+0000","last_fullsized":"2024-04-14T15:18:17.910141+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:18.760118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"297.6","version":"0'0","reported_seq":21,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.898049+0000","last_change":"2024-04-14T15:18:13.004180+0000","last_active":"2024-04-14T15:18:17.898049+0000","last_peered":"2024-04-14T15:18:17.898049+0000","last_clean":"2024-04-14T15:18:17.898049+0000","last_became_active":"2024-04-14T15:18:13.003841+0000","last_became_peered":"2024-04-14T15:18:13.003841+0000","last_unstale":"2024-04-14T15:18:17.898049+0000","last_undegraded":"2024-04-14T15:18:17.898049+0000","last_fullsized":"2024-04-14T15:18:17.898049+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:28.810731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"297.5","version":"0'0","reported_seq":21,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:18.543009+0000","last_change":"2024-04-14T15:18:12.955042+0000","last_active":"2024-04-14T15:18:18.543009+0000","last_peered":"2024-04-14T15:18:18.543009+0000","last_clean":"2024-04-14T15:18:18.543009+0000","last_became_active":"2024-04-14T15:18:12.954742+0000","last_became_peered":"2024-04-14T15:18:12.954742+0000","last_unstale":"2024-04-14T15:18:18.543009+0000","last_undegraded":"2024-04-14T15:18:18.543009+0000","last_fullsized":"2024-04-14T15:18:18.543009+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:12.119643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"297.4","version":"0'0","reported_seq":21,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.898322+0000","last_change":"2024-04-14T15:18:12.955231+0000","last_active":"2024-04-14T15:18:17.898322+0000","last_peered":"2024-04-14T15:18:17.898322+0000","last_clean":"2024-04-14T15:18:17.898322+0000","last_became_active":"2024-04-14T15:18:12.954904+0000","last_became_peered":"2024-04-14T15:18:12.954904+0000","last_unstale":"2024-04-14T15:18:17.898322+0000","last_undegraded":"2024-04-14T15:18:17.898322+0000","last_fullsized":"2024-04-14T15:18:17.898322+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:38.070409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"297.3","version":"0'0","reported_seq":21,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.905826+0000","last_change":"2024-04-14T15:18:13.009275+0000","last_active":"2024-04-14T15:18:17.905826+0000","last_peered":"2024-04-14T15:18:17.905826+0000","last_clean":"2024-04-14T15:18:17.905826+0000","last_became_active":"2024-04-14T15:18:13.008980+0000","last_became_peered":"2024-04-14T15:18:13.008980+0000","last_unstale":"2024-04-14T15:18:17.905826+0000","last_undegraded":"2024-04-14T15:18:17.905826+0000","last_fullsized":"2024-04-14T15:18:17.905826+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:15.759498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"297.2","version":"0'0","reported_seq":21,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:18.542480+0000","last_change":"2024-04-14T15:18:12.948399+0000","last_active":"2024-04-14T15:18:18.542480+0000","last_peered":"2024-04-14T15:18:18.542480+0000","last_clean":"2024-04-14T15:18:18.542480+0000","last_became_active":"2024-04-14T15:18:12.948080+0000","last_became_peered":"2024-04-14T15:18:12.948080+0000","last_unstale":"2024-04-14T15:18:18.542480+0000","last_undegraded":"2024-04-14T15:18:18.542480+0000","last_fullsized":"2024-04-14T15:18:18.542480+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:20.931990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"297.1","version":"0'0","reported_seq":21,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.897706+0000","last_change":"2024-04-14T15:18:13.006295+0000","last_active":"2024-04-14T15:18:17.897706+0000","last_peered":"2024-04-14T15:18:17.897706+0000","last_clean":"2024-04-14T15:18:17.897706+0000","last_became_active":"2024-04-14T15:18:13.006000+0000","last_became_peered":"2024-04-14T15:18:13.006000+0000","last_unstale":"2024-04-14T15:18:17.897706+0000","last_undegraded":"2024-04-14T15:18:17.897706+0000","last_fullsized":"2024-04-14T15:18:17.897706+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:17.146699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"297.0","version":"423'3","reported_seq":24,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.897646+0000","last_change":"2024-04-14T15:18:13.005460+0000","last_active":"2024-04-14T15:18:17.897646+0000","last_peered":"2024-04-14T15:18:17.897646+0000","last_clean":"2024-04-14T15:18:17.897646+0000","last_became_active":"2024-04-14T15:18:13.005082+0000","last_became_peered":"2024-04-14T15:18:13.005082+0000","last_unstale":"2024-04-14T15:18:17.897646+0000","last_undegraded":"2024-04-14T15:18:17.897646+0000","last_fullsized":"2024-04-14T15:18:17.897646+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:32.045130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"110.11","version":"403'41","reported_seq":834,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.909123+0000","last_change":"2024-04-14T15:18:17.909900+0000","last_active":"2024-04-14T15:18:17.909123+0000","last_peered":"2024-04-14T15:18:17.909123+0000","last_clean":"2024-04-14T15:18:17.909123+0000","last_became_active":"2024-04-14T15:13:09.936619+0000","last_became_peered":"2024-04-14T15:13:09.936619+0000","last_unstale":"2024-04-14T15:18:17.909123+0000","last_undegraded":"2024-04-14T15:18:17.909123+0000","last_fullsized":"2024-04-14T15:18:17.909123+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"403'41","last_scrub_stamp":"2024-04-14T15:17:56.911004+0000","last_deep_scrub":"403'41","last_deep_scrub_stamp":"2024-04-14T15:17:56.911004+0000","last_clean_scrub_stamp":"2024-04-14T15:17:56.911004+0000","objects_scrubbed":2,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:02.610139+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0.0010226409999999999,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"110.10","version":"253'158","reported_seq":916,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.897957+0000","last_change":"2024-04-14T15:18:17.898575+0000","last_active":"2024-04-14T15:18:17.897957+0000","last_peered":"2024-04-14T15:18:17.897957+0000","last_clean":"2024-04-14T15:18:17.897957+0000","last_became_active":"2024-04-14T15:13:09.928915+0000","last_became_peered":"2024-04-14T15:13:09.928915+0000","last_unstale":"2024-04-14T15:18:17.897957+0000","last_undegraded":"2024-04-14T15:18:17.897957+0000","last_fullsized":"2024-04-14T15:18:17.897957+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"253'158","last_scrub_stamp":"2024-04-14T15:17:34.291854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:34.291854+0000","objects_scrubbed":10,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:03.889264+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00081855199999999995,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":47,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"110.13","version":"409'41","reported_seq":835,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.902708+0000","last_change":"2024-04-14T15:18:17.903387+0000","last_active":"2024-04-14T15:18:17.902708+0000","last_peered":"2024-04-14T15:18:17.902708+0000","last_clean":"2024-04-14T15:18:17.902708+0000","last_became_active":"2024-04-14T15:13:09.918152+0000","last_became_peered":"2024-04-14T15:13:09.918152+0000","last_unstale":"2024-04-14T15:18:17.902708+0000","last_undegraded":"2024-04-14T15:18:17.902708+0000","last_fullsized":"2024-04-14T15:18:17.902708+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'39","last_scrub_stamp":"2024-04-14T15:17:44.873766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:44.873766+0000","objects_scrubbed":3,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:09.896100+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0031426990000000001,"stat_sum":{"num_bytes":27,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":14,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"110.12","version":"409'70","reported_seq":861,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.901549+0000","last_change":"2024-04-14T15:18:17.902993+0000","last_active":"2024-04-14T15:18:17.901549+0000","last_peered":"2024-04-14T15:18:17.901549+0000","last_clean":"2024-04-14T15:18:17.901549+0000","last_became_active":"2024-04-14T15:13:09.939566+0000","last_became_peered":"2024-04-14T15:13:09.939566+0000","last_unstale":"2024-04-14T15:18:17.901549+0000","last_undegraded":"2024-04-14T15:18:17.901549+0000","last_fullsized":"2024-04-14T15:18:17.901549+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"253'66","last_scrub_stamp":"2024-04-14T15:16:53.673906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:16:53.673906+0000","objects_scrubbed":6,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0026233390000000001,"stat_sum":{"num_bytes":74,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":24,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"110.15","version":"190'34","reported_seq":822,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.906452+0000","last_change":"2024-04-14T15:18:17.909308+0000","last_active":"2024-04-14T15:18:17.906452+0000","last_peered":"2024-04-14T15:18:17.906452+0000","last_clean":"2024-04-14T15:18:17.906452+0000","last_became_active":"2024-04-14T15:13:09.960121+0000","last_became_peered":"2024-04-14T15:13:09.960121+0000","last_unstale":"2024-04-14T15:18:17.906452+0000","last_undegraded":"2024-04-14T15:18:17.906452+0000","last_fullsized":"2024-04-14T15:18:17.906452+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'34","last_scrub_stamp":"2024-04-14T15:16:06.494065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:16:06.494065+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.003256952,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"110.14","version":"355'87","reported_seq":868,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.906508+0000","last_change":"2024-04-14T15:18:17.907509+0000","last_active":"2024-04-14T15:18:17.906508+0000","last_peered":"2024-04-14T15:18:17.906508+0000","last_clean":"2024-04-14T15:18:17.906508+0000","last_became_active":"2024-04-14T15:13:09.899561+0000","last_became_peered":"2024-04-14T15:13:09.899561+0000","last_unstale":"2024-04-14T15:18:17.906508+0000","last_undegraded":"2024-04-14T15:18:17.906508+0000","last_fullsized":"2024-04-14T15:18:17.906508+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'85","last_scrub_stamp":"2024-04-14T15:15:42.409029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:15:42.409029+0000","objects_scrubbed":5,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0011446900000000001,"stat_sum":{"num_bytes":71,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"110.17","version":"234'6","reported_seq":801,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.909360+0000","last_change":"2024-04-14T15:18:17.910186+0000","last_active":"2024-04-14T15:18:17.909360+0000","last_peered":"2024-04-14T15:18:17.909360+0000","last_clean":"2024-04-14T15:18:17.909360+0000","last_became_active":"2024-04-14T15:13:09.898521+0000","last_became_peered":"2024-04-14T15:13:09.898521+0000","last_unstale":"2024-04-14T15:18:17.909360+0000","last_undegraded":"2024-04-14T15:18:17.909360+0000","last_fullsized":"2024-04-14T15:18:17.909360+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'6","last_scrub_stamp":"2024-04-14T15:16:07.459718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:16:07.459718+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.004197788,"stat_sum":{"num_bytes":13,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"110.16","version":"355'35","reported_seq":662,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.909430+0000","last_change":"2024-04-14T15:18:17.910869+0000","last_active":"2024-04-14T15:18:17.909430+0000","last_peered":"2024-04-14T15:18:17.909430+0000","last_clean":"2024-04-14T15:18:17.909430+0000","last_became_active":"2024-04-14T15:15:21.889783+0000","last_became_peered":"2024-04-14T15:15:21.889783+0000","last_unstale":"2024-04-14T15:18:17.909430+0000","last_undegraded":"2024-04-14T15:18:17.909430+0000","last_fullsized":"2024-04-14T15:18:17.909430+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'35","last_scrub_stamp":"2024-04-14T15:17:34.097563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:34.097563+0000","objects_scrubbed":3,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:40.532428+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0028396139999999999,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"110.19","version":"237'85","reported_seq":871,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.901590+0000","last_change":"2024-04-14T15:18:17.901823+0000","last_active":"2024-04-14T15:18:17.901590+0000","last_peered":"2024-04-14T15:18:17.901590+0000","last_clean":"2024-04-14T15:18:17.901590+0000","last_became_active":"2024-04-14T15:13:09.998139+0000","last_became_peered":"2024-04-14T15:13:09.998139+0000","last_unstale":"2024-04-14T15:18:17.901590+0000","last_undegraded":"2024-04-14T15:18:17.901590+0000","last_fullsized":"2024-04-14T15:18:17.901590+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'85","last_scrub_stamp":"2024-04-14T15:16:04.177097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:16:04.177097+0000","objects_scrubbed":5,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0016340009999999999,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"110.18","version":"387'94","reported_seq":881,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.898315+0000","last_change":"2024-04-14T15:18:17.902408+0000","last_active":"2024-04-14T15:18:17.898315+0000","last_peered":"2024-04-14T15:18:17.898315+0000","last_clean":"2024-04-14T15:18:17.898315+0000","last_became_active":"2024-04-14T15:13:09.998803+0000","last_became_peered":"2024-04-14T15:13:09.998803+0000","last_unstale":"2024-04-14T15:18:17.898315+0000","last_undegraded":"2024-04-14T15:18:17.898315+0000","last_fullsized":"2024-04-14T15:18:17.898315+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'94","last_scrub_stamp":"2024-04-14T15:17:36.005598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:36.005598+0000","objects_scrubbed":8,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:03.292215+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0044780449999999999,"stat_sum":{"num_bytes":65,"num_objects":8,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":29,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"110.1b","version":"355'91","reported_seq":877,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.902525+0000","last_change":"2024-04-14T15:18:17.903662+0000","last_active":"2024-04-14T15:18:17.902525+0000","last_peered":"2024-04-14T15:18:17.902525+0000","last_clean":"2024-04-14T15:18:17.902525+0000","last_became_active":"2024-04-14T15:13:09.946919+0000","last_became_peered":"2024-04-14T15:13:09.946919+0000","last_unstale":"2024-04-14T15:18:17.902525+0000","last_undegraded":"2024-04-14T15:18:17.902525+0000","last_fullsized":"2024-04-14T15:18:17.902525+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'91","last_scrub_stamp":"2024-04-14T15:17:47.889364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:47.889364+0000","objects_scrubbed":6,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:57.325889+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.003513979,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":28,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"110.1a","version":"387'39","reported_seq":838,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.909664+0000","last_change":"2024-04-14T15:18:17.910183+0000","last_active":"2024-04-14T15:18:17.909664+0000","last_peered":"2024-04-14T15:18:17.909664+0000","last_clean":"2024-04-14T15:18:17.909664+0000","last_became_active":"2024-04-14T15:13:09.960319+0000","last_became_peered":"2024-04-14T15:13:09.960319+0000","last_unstale":"2024-04-14T15:18:17.909664+0000","last_undegraded":"2024-04-14T15:18:17.909664+0000","last_fullsized":"2024-04-14T15:18:17.909664+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'39","last_scrub_stamp":"2024-04-14T15:17:53.871590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:53.871590+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:14.813991+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00075180500000000003,"stat_sum":{"num_bytes":62,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"110.1d","version":"225'85","reported_seq":873,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:18.543639+0000","last_change":"2024-04-14T15:18:18.544412+0000","last_active":"2024-04-14T15:18:18.543639+0000","last_peered":"2024-04-14T15:18:18.543639+0000","last_clean":"2024-04-14T15:18:18.543639+0000","last_became_active":"2024-04-14T15:13:09.868082+0000","last_became_peered":"2024-04-14T15:13:09.868082+0000","last_unstale":"2024-04-14T15:18:18.543639+0000","last_undegraded":"2024-04-14T15:18:18.543639+0000","last_fullsized":"2024-04-14T15:18:18.543639+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"225'85","last_scrub_stamp":"2024-04-14T15:17:36.419088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:36.419088+0000","objects_scrubbed":5,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:38.903199+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0011714480000000001,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"110.1c","version":"424'91","reported_seq":870,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.906175+0000","last_change":"2024-04-14T15:18:17.906688+0000","last_active":"2024-04-14T15:18:17.906175+0000","last_peered":"2024-04-14T15:18:17.906175+0000","last_clean":"2024-04-14T15:18:17.906175+0000","last_became_active":"2024-04-14T15:13:09.954091+0000","last_became_peered":"2024-04-14T15:13:09.954091+0000","last_unstale":"2024-04-14T15:18:17.906175+0000","last_undegraded":"2024-04-14T15:18:17.906175+0000","last_fullsized":"2024-04-14T15:18:17.906175+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'73","last_scrub_stamp":"2024-04-14T15:16:04.615572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:16:04.615572+0000","objects_scrubbed":5,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":1,"snaptrim_duration":0.0061922130000000002,"stat_sum":{"num_bytes":75,"num_objects":9,"num_object_clones":2,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"110.1f","version":"268'25","reported_seq":867,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.899404+0000","last_change":"2024-04-14T15:18:17.899742+0000","last_active":"2024-04-14T15:18:17.899404+0000","last_peered":"2024-04-14T15:18:17.899404+0000","last_clean":"2024-04-14T15:18:17.899404+0000","last_became_active":"2024-04-14T15:13:09.999629+0000","last_became_peered":"2024-04-14T15:13:09.999629+0000","last_unstale":"2024-04-14T15:18:17.899404+0000","last_undegraded":"2024-04-14T15:18:17.899404+0000","last_fullsized":"2024-04-14T15:18:17.899404+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'25","last_scrub_stamp":"2024-04-14T15:16:08.236881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:16:08.236881+0000","objects_scrubbed":2,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0011967119999999999,"stat_sum":{"num_bytes":4194315,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":8,"num_write_kb":8198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"110.1e","version":"355'23","reported_seq":815,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.909778+0000","last_change":"2024-04-14T15:18:17.910523+0000","last_active":"2024-04-14T15:18:17.909778+0000","last_peered":"2024-04-14T15:18:17.909778+0000","last_clean":"2024-04-14T15:18:17.909778+0000","last_became_active":"2024-04-14T15:13:09.955932+0000","last_became_peered":"2024-04-14T15:13:09.955932+0000","last_unstale":"2024-04-14T15:18:17.909778+0000","last_undegraded":"2024-04-14T15:18:17.909778+0000","last_fullsized":"2024-04-14T15:18:17.909778+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'22","last_scrub_stamp":"2024-04-14T15:16:10.427994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:16:10.427994+0000","objects_scrubbed":1,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001894679,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":8,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"110.1","version":"406'32","reported_seq":826,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:18.542806+0000","last_change":"2024-04-14T15:18:18.543502+0000","last_active":"2024-04-14T15:18:18.542806+0000","last_peered":"2024-04-14T15:18:18.542806+0000","last_clean":"2024-04-14T15:18:18.542806+0000","last_became_active":"2024-04-14T15:13:09.929562+0000","last_became_peered":"2024-04-14T15:13:09.929562+0000","last_unstale":"2024-04-14T15:18:18.542806+0000","last_undegraded":"2024-04-14T15:18:18.542806+0000","last_fullsized":"2024-04-14T15:18:18.542806+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"219'24","last_scrub_stamp":"2024-04-14T15:15:35.591006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:15:35.591006+0000","objects_scrubbed":3,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00076140799999999998,"stat_sum":{"num_bytes":4000050,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":15,"num_write_kb":3921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"110.0","version":"219'85","reported_seq":867,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.898018+0000","last_change":"2024-04-14T15:18:17.898778+0000","last_active":"2024-04-14T15:18:17.898018+0000","last_peered":"2024-04-14T15:18:17.898018+0000","last_clean":"2024-04-14T15:18:17.898018+0000","last_became_active":"2024-04-14T15:13:09.856407+0000","last_became_peered":"2024-04-14T15:13:09.856407+0000","last_unstale":"2024-04-14T15:18:17.898018+0000","last_undegraded":"2024-04-14T15:18:17.898018+0000","last_fullsized":"2024-04-14T15:18:17.898018+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"219'85","last_scrub_stamp":"2024-04-14T15:17:51.438149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:51.438149+0000","objects_scrubbed":5,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:09.807854+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00080519800000000002,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"110.3","version":"355'52","reported_seq":831,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.900808+0000","last_change":"2024-04-14T15:18:17.901073+0000","last_active":"2024-04-14T15:18:17.900808+0000","last_peered":"2024-04-14T15:18:17.900808+0000","last_clean":"2024-04-14T15:18:17.900808+0000","last_became_active":"2024-04-14T15:13:09.997208+0000","last_became_peered":"2024-04-14T15:13:09.997208+0000","last_unstale":"2024-04-14T15:18:17.900808+0000","last_undegraded":"2024-04-14T15:18:17.900808+0000","last_fullsized":"2024-04-14T15:18:17.900808+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"223'51","last_scrub_stamp":"2024-04-14T15:16:06.149271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:16:06.149271+0000","objects_scrubbed":3,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0020983970000000001,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"110.2","version":"240'55","reported_seq":849,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.899830+0000","last_change":"2024-04-14T15:18:17.900110+0000","last_active":"2024-04-14T15:18:17.899830+0000","last_peered":"2024-04-14T15:18:17.899830+0000","last_clean":"2024-04-14T15:18:17.899830+0000","last_became_active":"2024-04-14T15:13:09.957187+0000","last_became_peered":"2024-04-14T15:13:09.957187+0000","last_unstale":"2024-04-14T15:18:17.899830+0000","last_undegraded":"2024-04-14T15:18:17.899830+0000","last_fullsized":"2024-04-14T15:18:17.899830+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'55","last_scrub_stamp":"2024-04-14T15:17:37.941589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:37.941589+0000","objects_scrubbed":6,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:59.133865+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001542893,"stat_sum":{"num_bytes":49,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":18,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"110.5","version":"396'64","reported_seq":850,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.899444+0000","last_change":"2024-04-14T15:18:17.899751+0000","last_active":"2024-04-14T15:18:17.899444+0000","last_peered":"2024-04-14T15:18:17.899444+0000","last_clean":"2024-04-14T15:18:17.899444+0000","last_became_active":"2024-04-14T15:13:09.853689+0000","last_became_peered":"2024-04-14T15:13:09.853689+0000","last_unstale":"2024-04-14T15:18:17.899444+0000","last_undegraded":"2024-04-14T15:18:17.899444+0000","last_fullsized":"2024-04-14T15:18:17.899444+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"396'64","last_scrub_stamp":"2024-04-14T15:17:47.418687+0000","last_deep_scrub":"396'64","last_deep_scrub_stamp":"2024-04-14T15:17:47.418687+0000","last_clean_scrub_stamp":"2024-04-14T15:17:47.418687+0000","objects_scrubbed":5,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:51.671405+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00048516899999999998,"stat_sum":{"num_bytes":42,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":20,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"110.4","version":"406'55","reported_seq":839,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.908880+0000","last_change":"2024-04-14T15:18:17.909717+0000","last_active":"2024-04-14T15:18:17.908880+0000","last_peered":"2024-04-14T15:18:17.908880+0000","last_clean":"2024-04-14T15:18:17.908880+0000","last_became_active":"2024-04-14T15:13:09.958847+0000","last_became_peered":"2024-04-14T15:13:09.958847+0000","last_unstale":"2024-04-14T15:18:17.908880+0000","last_undegraded":"2024-04-14T15:18:17.908880+0000","last_fullsized":"2024-04-14T15:18:17.908880+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'51","last_scrub_stamp":"2024-04-14T15:16:05.487850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:16:05.487850+0000","objects_scrubbed":3,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.003145634,"stat_sum":{"num_bytes":38,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"110.7","version":"355'18","reported_seq":815,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.910627+0000","last_change":"2024-04-14T15:18:17.910853+0000","last_active":"2024-04-14T15:18:17.910627+0000","last_peered":"2024-04-14T15:18:17.910627+0000","last_clean":"2024-04-14T15:18:17.910627+0000","last_became_active":"2024-04-14T15:13:09.936631+0000","last_became_peered":"2024-04-14T15:13:09.936631+0000","last_unstale":"2024-04-14T15:18:17.910627+0000","last_undegraded":"2024-04-14T15:18:17.910627+0000","last_fullsized":"2024-04-14T15:18:17.910627+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'18","last_scrub_stamp":"2024-04-14T15:17:34.552788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:34.552788+0000","objects_scrubbed":2,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:51.391931+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00038771399999999999,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"110.6","version":"274'74","reported_seq":850,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:18.543885+0000","last_change":"2024-04-14T15:18:18.544408+0000","last_active":"2024-04-14T15:18:18.543885+0000","last_peered":"2024-04-14T15:18:18.543885+0000","last_clean":"2024-04-14T15:18:18.543885+0000","last_became_active":"2024-04-14T15:13:09.930125+0000","last_became_peered":"2024-04-14T15:13:09.930125+0000","last_unstale":"2024-04-14T15:18:18.543885+0000","last_undegraded":"2024-04-14T15:18:18.543885+0000","last_fullsized":"2024-04-14T15:18:18.543885+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'74","last_scrub_stamp":"2024-04-14T15:15:42.641684+0000","last_deep_scrub":"274'74","last_deep_scrub_stamp":"2024-04-14T15:15:42.641684+0000","last_clean_scrub_stamp":"2024-04-14T15:15:42.641684+0000","objects_scrubbed":5,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00068844700000000002,"stat_sum":{"num_bytes":44,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":21,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"110.9","version":"369'20","reported_seq":811,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.901182+0000","last_change":"2024-04-14T15:18:17.901435+0000","last_active":"2024-04-14T15:18:17.901182+0000","last_peered":"2024-04-14T15:18:17.901182+0000","last_clean":"2024-04-14T15:18:17.901182+0000","last_became_active":"2024-04-14T15:13:09.957324+0000","last_became_peered":"2024-04-14T15:13:09.957324+0000","last_unstale":"2024-04-14T15:18:17.901182+0000","last_undegraded":"2024-04-14T15:18:17.901182+0000","last_fullsized":"2024-04-14T15:18:17.901182+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"225'17","last_scrub_stamp":"2024-04-14T15:16:09.202462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:16:09.202462+0000","objects_scrubbed":1,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0014809179999999999,"stat_sum":{"num_bytes":35,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"110.8","version":"190'17","reported_seq":558,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.906599+0000","last_change":"2024-04-14T15:18:17.908478+0000","last_active":"2024-04-14T15:18:17.906599+0000","last_peered":"2024-04-14T15:18:17.906599+0000","last_clean":"2024-04-14T15:18:17.906599+0000","last_became_active":"2024-04-14T15:15:44.890057+0000","last_became_peered":"2024-04-14T15:15:44.890057+0000","last_unstale":"2024-04-14T15:18:17.906599+0000","last_undegraded":"2024-04-14T15:18:17.906599+0000","last_fullsized":"2024-04-14T15:18:17.906599+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'17","last_scrub_stamp":"2024-04-14T15:15:25.478924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:15:25.478924+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020620840000000001,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"110.b","version":"412'71","reported_seq":857,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.903706+0000","last_change":"2024-04-14T15:18:17.904729+0000","last_active":"2024-04-14T15:18:17.903706+0000","last_peered":"2024-04-14T15:18:17.903706+0000","last_clean":"2024-04-14T15:18:17.903706+0000","last_became_active":"2024-04-14T15:13:09.964629+0000","last_became_peered":"2024-04-14T15:13:09.964629+0000","last_unstale":"2024-04-14T15:18:17.903706+0000","last_undegraded":"2024-04-14T15:18:17.903706+0000","last_fullsized":"2024-04-14T15:18:17.903706+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'68","last_scrub_stamp":"2024-04-14T15:16:49.611563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:16:49.611563+0000","objects_scrubbed":4,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:19.225294+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.0041064150000000004,"stat_sum":{"num_bytes":54,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":22,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"110.a","version":"387'74","reported_seq":782,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.909293+0000","last_change":"2024-04-14T15:18:17.910000+0000","last_active":"2024-04-14T15:18:17.909293+0000","last_peered":"2024-04-14T15:18:17.909293+0000","last_clean":"2024-04-14T15:18:17.909293+0000","last_became_active":"2024-04-14T15:15:21.885346+0000","last_became_peered":"2024-04-14T15:15:21.885346+0000","last_unstale":"2024-04-14T15:18:17.909293+0000","last_undegraded":"2024-04-14T15:18:17.909293+0000","last_fullsized":"2024-04-14T15:18:17.909293+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'70","last_scrub_stamp":"2024-04-14T15:15:34.881731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:15:34.881731+0000","objects_scrubbed":5,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0.0020873749999999998,"stat_sum":{"num_bytes":84,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":26,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"110.d","version":"416'95","reported_seq":706,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.908015+0000","last_change":"2024-04-14T15:18:17.908720+0000","last_active":"2024-04-14T15:18:17.908015+0000","last_peered":"2024-04-14T15:18:17.908015+0000","last_clean":"2024-04-14T15:18:17.908015+0000","last_became_active":"2024-04-14T15:15:21.877073+0000","last_became_peered":"2024-04-14T15:15:21.877073+0000","last_unstale":"2024-04-14T15:18:17.908015+0000","last_undegraded":"2024-04-14T15:18:17.908015+0000","last_fullsized":"2024-04-14T15:18:17.908015+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'85","last_scrub_stamp":"2024-04-14T15:15:38.773256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:15:38.773256+0000","objects_scrubbed":5,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00089788300000000004,"stat_sum":{"num_bytes":71,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":34,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"110.c","version":"398'20","reported_seq":815,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.904713+0000","last_change":"2024-04-14T15:18:17.905931+0000","last_active":"2024-04-14T15:18:17.904713+0000","last_peered":"2024-04-14T15:18:17.904713+0000","last_clean":"2024-04-14T15:18:17.904713+0000","last_became_active":"2024-04-14T15:13:09.984609+0000","last_became_peered":"2024-04-14T15:13:09.984609+0000","last_unstale":"2024-04-14T15:18:17.904713+0000","last_undegraded":"2024-04-14T15:18:17.904713+0000","last_fullsized":"2024-04-14T15:18:17.904713+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'20","last_scrub_stamp":"2024-04-14T15:17:52.882509+0000","last_deep_scrub":"398'20","last_deep_scrub_stamp":"2024-04-14T15:17:52.882509+0000","last_clean_scrub_stamp":"2024-04-14T15:17:52.882509+0000","objects_scrubbed":2,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:11.793329+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.005213278,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"110.f","version":"355'52","reported_seq":843,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.899875+0000","last_change":"2024-04-14T15:18:17.902434+0000","last_active":"2024-04-14T15:18:17.899875+0000","last_peered":"2024-04-14T15:18:17.899875+0000","last_clean":"2024-04-14T15:18:17.899875+0000","last_became_active":"2024-04-14T15:13:09.964057+0000","last_became_peered":"2024-04-14T15:13:09.964057+0000","last_unstale":"2024-04-14T15:18:17.899875+0000","last_undegraded":"2024-04-14T15:18:17.899875+0000","last_fullsized":"2024-04-14T15:18:17.899875+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'52","last_scrub_stamp":"2024-04-14T15:17:45.848166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:45.848166+0000","objects_scrubbed":4,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:59.536371+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0038738169999999999,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"110.e","version":"416'92","reported_seq":878,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:18.544455+0000","last_change":"2024-04-14T15:18:18.544790+0000","last_active":"2024-04-14T15:18:18.544455+0000","last_peered":"2024-04-14T15:18:18.544455+0000","last_clean":"2024-04-14T15:18:18.544455+0000","last_became_active":"2024-04-14T15:13:09.984980+0000","last_became_peered":"2024-04-14T15:13:09.984980+0000","last_unstale":"2024-04-14T15:18:18.544455+0000","last_undegraded":"2024-04-14T15:18:18.544455+0000","last_fullsized":"2024-04-14T15:18:18.544455+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'90","last_scrub_stamp":"2024-04-14T15:17:34.409367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:34.409367+0000","objects_scrubbed":6,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:50.340753+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00090280799999999995,"stat_sum":{"num_bytes":68,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":28,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"11.7","version":"0'0","reported_seq":307,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.898665+0000","last_change":"2024-04-14T15:15:44.883275+0000","last_active":"2024-04-14T15:18:17.898665+0000","last_peered":"2024-04-14T15:18:17.898665+0000","last_clean":"2024-04-14T15:18:17.898665+0000","last_became_active":"2024-04-14T15:15:44.882941+0000","last_became_peered":"2024-04-14T15:15:44.882941+0000","last_unstale":"2024-04-14T15:18:17.898665+0000","last_undegraded":"2024-04-14T15:18:17.898665+0000","last_fullsized":"2024-04-14T15:18:17.898665+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:32.465858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:32.465858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:20.951165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.9","version":"85'9","reported_seq":867,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:18.543563+0000","last_change":"2024-04-14T15:17:54.606158+0000","last_active":"2024-04-14T15:18:18.543563+0000","last_peered":"2024-04-14T15:18:18.543563+0000","last_clean":"2024-04-14T15:18:18.543563+0000","last_became_active":"2024-04-14T15:09:47.045441+0000","last_became_peered":"2024-04-14T15:09:47.045441+0000","last_unstale":"2024-04-14T15:18:18.543563+0000","last_undegraded":"2024-04-14T15:18:18.543563+0000","last_fullsized":"2024-04-14T15:18:18.543563+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9","last_scrub_stamp":"2024-04-14T15:17:54.606051+0000","last_deep_scrub":"85'9","last_deep_scrub_stamp":"2024-04-14T15:10:51.950580+0000","last_clean_scrub_stamp":"2024-04-14T15:17:54.606051+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:07.946751+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"72'15","reported_seq":349,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:18.542068+0000","last_change":"2024-04-14T15:15:44.885742+0000","last_active":"2024-04-14T15:18:18.542068+0000","last_peered":"2024-04-14T15:18:18.542068+0000","last_clean":"2024-04-14T15:18:18.542068+0000","last_became_active":"2024-04-14T15:15:44.885409+0000","last_became_peered":"2024-04-14T15:15:44.885409+0000","last_unstale":"2024-04-14T15:18:18.542068+0000","last_undegraded":"2024-04-14T15:18:18.542068+0000","last_fullsized":"2024-04-14T15:18:18.542068+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-04-14T15:15:39.325129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:15:39.325129+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.661039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":761,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.902383+0000","last_change":"2024-04-14T15:17:53.867764+0000","last_active":"2024-04-14T15:18:17.902383+0000","last_peered":"2024-04-14T15:18:17.902383+0000","last_clean":"2024-04-14T15:18:17.902383+0000","last_became_active":"2024-04-14T15:09:43.021460+0000","last_became_peered":"2024-04-14T15:09:43.021460+0000","last_unstale":"2024-04-14T15:18:17.902383+0000","last_undegraded":"2024-04-14T15:18:17.902383+0000","last_fullsized":"2024-04-14T15:18:17.902383+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:53.867696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:53.867696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:56.203183+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":315,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.909298+0000","last_change":"2024-04-14T15:17:54.889170+0000","last_active":"2024-04-14T15:18:17.909298+0000","last_peered":"2024-04-14T15:18:17.909298+0000","last_clean":"2024-04-14T15:18:17.909298+0000","last_became_active":"2024-04-14T15:15:44.888154+0000","last_became_peered":"2024-04-14T15:15:44.888154+0000","last_unstale":"2024-04-14T15:18:17.909298+0000","last_undegraded":"2024-04-14T15:18:17.909298+0000","last_fullsized":"2024-04-14T15:18:17.909298+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:54.889066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:15:40.368219+0000","last_clean_scrub_stamp":"2024-04-14T15:17:54.889066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:11.492168+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":651,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:18.543112+0000","last_change":"2024-04-14T15:15:32.596167+0000","last_active":"2024-04-14T15:18:18.543112+0000","last_peered":"2024-04-14T15:18:18.543112+0000","last_clean":"2024-04-14T15:18:18.543112+0000","last_became_active":"2024-04-14T15:12:56.355348+0000","last_became_peered":"2024-04-14T15:12:56.355348+0000","last_unstale":"2024-04-14T15:18:18.543112+0000","last_undegraded":"2024-04-14T15:18:18.543112+0000","last_fullsized":"2024-04-14T15:18:18.543112+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:32.596096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:32.596096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.661020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"72'9","reported_seq":776,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.906091+0000","last_change":"2024-04-14T15:16:56.663996+0000","last_active":"2024-04-14T15:18:17.906091+0000","last_peered":"2024-04-14T15:18:17.906091+0000","last_clean":"2024-04-14T15:18:17.906091+0000","last_became_active":"2024-04-14T15:09:45.037179+0000","last_became_peered":"2024-04-14T15:09:45.037179+0000","last_unstale":"2024-04-14T15:18:17.906091+0000","last_undegraded":"2024-04-14T15:18:17.906091+0000","last_fullsized":"2024-04-14T15:18:17.906091+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-04-14T15:16:56.663900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:16:56.663900+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:22.985760+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":756,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.899644+0000","last_change":"2024-04-14T15:15:25.311260+0000","last_active":"2024-04-14T15:18:17.899644+0000","last_peered":"2024-04-14T15:18:17.899644+0000","last_clean":"2024-04-14T15:18:17.899644+0000","last_became_active":"2024-04-14T15:09:43.020044+0000","last_became_peered":"2024-04-14T15:09:43.020044+0000","last_unstale":"2024-04-14T15:18:17.899644+0000","last_undegraded":"2024-04-14T15:18:17.899644+0000","last_fullsized":"2024-04-14T15:18:17.899644+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:25.311184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:15:25.311184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:20.951148+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":753,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.900321+0000","last_change":"2024-04-14T15:16:16.655593+0000","last_active":"2024-04-14T15:18:17.900321+0000","last_peered":"2024-04-14T15:18:17.900321+0000","last_clean":"2024-04-14T15:18:17.900321+0000","last_became_active":"2024-04-14T15:09:47.046153+0000","last_became_peered":"2024-04-14T15:09:47.046153+0000","last_unstale":"2024-04-14T15:18:17.900321+0000","last_undegraded":"2024-04-14T15:18:17.900321+0000","last_fullsized":"2024-04-14T15:18:17.900321+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:16.655504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:16.655504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:20.851947+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":752,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.907074+0000","last_change":"2024-04-14T15:17:37.919665+0000","last_active":"2024-04-14T15:18:17.907074+0000","last_peered":"2024-04-14T15:18:17.907074+0000","last_clean":"2024-04-14T15:18:17.907074+0000","last_became_active":"2024-04-14T15:09:49.061495+0000","last_became_peered":"2024-04-14T15:09:49.061495+0000","last_unstale":"2024-04-14T15:18:17.907074+0000","last_undegraded":"2024-04-14T15:18:17.907074+0000","last_fullsized":"2024-04-14T15:18:17.907074+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:37.919595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:17:37.919595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:42.403547+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":651,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:18.543910+0000","last_change":"2024-04-14T15:14:55.989719+0000","last_active":"2024-04-14T15:18:18.543910+0000","last_peered":"2024-04-14T15:18:18.543910+0000","last_clean":"2024-04-14T15:18:18.543910+0000","last_became_active":"2024-04-14T15:12:56.246062+0000","last_became_peered":"2024-04-14T15:12:56.246062+0000","last_unstale":"2024-04-14T15:18:18.543910+0000","last_undegraded":"2024-04-14T15:18:18.543910+0000","last_fullsized":"2024-04-14T15:18:18.543910+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:55.989627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:14:55.989627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.660951+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"222'28","reported_seq":799,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.909792+0000","last_change":"2024-04-14T15:17:44.832902+0000","last_active":"2024-04-14T15:18:17.909792+0000","last_peered":"2024-04-14T15:18:17.909792+0000","last_clean":"2024-04-14T15:18:17.909792+0000","last_became_active":"2024-04-14T15:09:45.178782+0000","last_became_peered":"2024-04-14T15:09:45.178782+0000","last_unstale":"2024-04-14T15:18:17.909792+0000","last_undegraded":"2024-04-14T15:18:17.909792+0000","last_fullsized":"2024-04-14T15:18:17.909792+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'28","last_scrub_stamp":"2024-04-14T15:17:44.832769+0000","last_deep_scrub":"222'28","last_deep_scrub_stamp":"2024-04-14T15:14:50.054931+0000","last_clean_scrub_stamp":"2024-04-14T15:17:44.832769+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:54.927275+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":761,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.901590+0000","last_change":"2024-04-14T15:15:43.569812+0000","last_active":"2024-04-14T15:18:17.901590+0000","last_peered":"2024-04-14T15:18:17.901590+0000","last_clean":"2024-04-14T15:18:17.901590+0000","last_became_active":"2024-04-14T15:09:43.020268+0000","last_became_peered":"2024-04-14T15:09:43.020268+0000","last_unstale":"2024-04-14T15:18:17.901590+0000","last_undegraded":"2024-04-14T15:18:17.901590+0000","last_fullsized":"2024-04-14T15:18:17.901590+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:43.569735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:34.342793+0000","last_clean_scrub_stamp":"2024-04-14T15:15:43.569735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:20.851900+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":745,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.900488+0000","last_change":"2024-04-14T15:15:30.488857+0000","last_active":"2024-04-14T15:18:17.900488+0000","last_peered":"2024-04-14T15:18:17.900488+0000","last_clean":"2024-04-14T15:18:17.900488+0000","last_became_active":"2024-04-14T15:09:47.045542+0000","last_became_peered":"2024-04-14T15:09:47.045542+0000","last_unstale":"2024-04-14T15:18:17.900488+0000","last_undegraded":"2024-04-14T15:18:17.900488+0000","last_fullsized":"2024-04-14T15:18:17.900488+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:30.488781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:15:30.488781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:20.851850+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":742,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:18.542475+0000","last_change":"2024-04-14T15:16:08.857476+0000","last_active":"2024-04-14T15:18:18.542475+0000","last_peered":"2024-04-14T15:18:18.542475+0000","last_clean":"2024-04-14T15:18:18.542475+0000","last_became_active":"2024-04-14T15:09:49.065397+0000","last_became_peered":"2024-04-14T15:09:49.065397+0000","last_unstale":"2024-04-14T15:18:18.542475+0000","last_undegraded":"2024-04-14T15:18:18.542475+0000","last_fullsized":"2024-04-14T15:18:18.542475+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:08.857397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:08.857397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.661103+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"91'130","reported_seq":630,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.897730+0000","last_change":"2024-04-14T15:16:07.609240+0000","last_active":"2024-04-14T15:18:17.897730+0000","last_peered":"2024-04-14T15:18:17.897730+0000","last_clean":"2024-04-14T15:18:17.897730+0000","last_became_active":"2024-04-14T15:15:44.886240+0000","last_became_peered":"2024-04-14T15:15:44.886240+0000","last_unstale":"2024-04-14T15:18:17.897730+0000","last_undegraded":"2024-04-14T15:18:17.897730+0000","last_fullsized":"2024-04-14T15:18:17.897730+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:16:07.609119+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:16:07.609119+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:20.851933+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":658,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.900221+0000","last_change":"2024-04-14T15:16:19.749313+0000","last_active":"2024-04-14T15:18:17.900221+0000","last_peered":"2024-04-14T15:18:17.900221+0000","last_clean":"2024-04-14T15:18:17.900221+0000","last_became_active":"2024-04-14T15:12:56.474322+0000","last_became_peered":"2024-04-14T15:12:56.474322+0000","last_unstale":"2024-04-14T15:18:17.900221+0000","last_undegraded":"2024-04-14T15:18:17.900221+0000","last_fullsized":"2024-04-14T15:18:17.900221+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:19.749208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:16:19.749208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:20.851982+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"72'17","reported_seq":834,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.899547+0000","last_change":"2024-04-14T15:17:48.374589+0000","last_active":"2024-04-14T15:18:17.899547+0000","last_peered":"2024-04-14T15:18:17.899547+0000","last_clean":"2024-04-14T15:18:17.899547+0000","last_became_active":"2024-04-14T15:13:12.933907+0000","last_became_peered":"2024-04-14T15:13:12.933907+0000","last_unstale":"2024-04-14T15:18:17.899547+0000","last_undegraded":"2024-04-14T15:18:17.899547+0000","last_fullsized":"2024-04-14T15:18:17.899547+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'17","last_scrub_stamp":"2024-04-14T15:17:48.374469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:17:48.374469+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:04.297073+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":754,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:18.543700+0000","last_change":"2024-04-14T15:16:16.820151+0000","last_active":"2024-04-14T15:18:18.543700+0000","last_peered":"2024-04-14T15:18:18.543700+0000","last_clean":"2024-04-14T15:18:18.543700+0000","last_became_active":"2024-04-14T15:09:43.022351+0000","last_became_peered":"2024-04-14T15:09:43.022351+0000","last_unstale":"2024-04-14T15:18:18.543700+0000","last_undegraded":"2024-04-14T15:18:18.543700+0000","last_fullsized":"2024-04-14T15:18:18.543700+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:16.820070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:16.820070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.661135+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":748,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.898732+0000","last_change":"2024-04-14T15:17:54.871693+0000","last_active":"2024-04-14T15:18:17.898732+0000","last_peered":"2024-04-14T15:18:17.898732+0000","last_clean":"2024-04-14T15:18:17.898732+0000","last_became_active":"2024-04-14T15:09:47.042810+0000","last_became_peered":"2024-04-14T15:09:47.042810+0000","last_unstale":"2024-04-14T15:18:17.898732+0000","last_undegraded":"2024-04-14T15:18:17.898732+0000","last_fullsized":"2024-04-14T15:18:17.898732+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:54.871593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:17:54.871593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:21.351148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":749,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.898937+0000","last_change":"2024-04-14T15:16:45.669141+0000","last_active":"2024-04-14T15:18:17.898937+0000","last_peered":"2024-04-14T15:18:17.898937+0000","last_clean":"2024-04-14T15:18:17.898937+0000","last_became_active":"2024-04-14T15:09:49.069594+0000","last_became_peered":"2024-04-14T15:09:49.069594+0000","last_unstale":"2024-04-14T15:18:17.898937+0000","last_undegraded":"2024-04-14T15:18:17.898937+0000","last_fullsized":"2024-04-14T15:18:17.898937+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:45.669063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:45.669063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"91'130","reported_seq":1024,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:18.542352+0000","last_change":"2024-04-14T15:14:46.903338+0000","last_active":"2024-04-14T15:18:18.542352+0000","last_peered":"2024-04-14T15:18:18.542352+0000","last_clean":"2024-04-14T15:18:18.542352+0000","last_became_active":"2024-04-14T15:14:17.873403+0000","last_became_peered":"2024-04-14T15:14:17.873403+0000","last_unstale":"2024-04-14T15:18:18.542352+0000","last_undegraded":"2024-04-14T15:18:18.542352+0000","last_fullsized":"2024-04-14T15:18:18.542352+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:14:46.903248+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:14:46.903248+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.660932+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":660,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.909167+0000","last_change":"2024-04-14T15:17:46.871883+0000","last_active":"2024-04-14T15:18:17.909167+0000","last_peered":"2024-04-14T15:18:17.909167+0000","last_clean":"2024-04-14T15:18:17.909167+0000","last_became_active":"2024-04-14T15:12:56.289745+0000","last_became_peered":"2024-04-14T15:12:56.289745+0000","last_unstale":"2024-04-14T15:18:17.909167+0000","last_undegraded":"2024-04-14T15:18:17.909167+0000","last_fullsized":"2024-04-14T15:18:17.909167+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:46.871773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:17:46.871773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:58.233534+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"179'15","reported_seq":770,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.898449+0000","last_change":"2024-04-14T15:16:49.243096+0000","last_active":"2024-04-14T15:18:17.898449+0000","last_peered":"2024-04-14T15:18:17.898449+0000","last_clean":"2024-04-14T15:18:17.898449+0000","last_became_active":"2024-04-14T15:09:45.029170+0000","last_became_peered":"2024-04-14T15:09:45.029170+0000","last_unstale":"2024-04-14T15:18:17.898449+0000","last_undegraded":"2024-04-14T15:18:17.898449+0000","last_fullsized":"2024-04-14T15:18:17.898449+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'15","last_scrub_stamp":"2024-04-14T15:16:49.243002+0000","last_deep_scrub":"179'15","last_deep_scrub_stamp":"2024-04-14T15:14:07.488422+0000","last_clean_scrub_stamp":"2024-04-14T15:16:49.243002+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:22.487978+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":764,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.898936+0000","last_change":"2024-04-14T15:17:51.854742+0000","last_active":"2024-04-14T15:18:17.898936+0000","last_peered":"2024-04-14T15:18:17.898936+0000","last_clean":"2024-04-14T15:18:17.898936+0000","last_became_active":"2024-04-14T15:09:43.018871+0000","last_became_peered":"2024-04-14T15:09:43.018871+0000","last_unstale":"2024-04-14T15:18:17.898936+0000","last_undegraded":"2024-04-14T15:18:17.898936+0000","last_fullsized":"2024-04-14T15:18:17.898936+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:51.854659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:51.854659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:08.464204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"400'9","reported_seq":866,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.900320+0000","last_change":"2024-04-14T15:17:54.845967+0000","last_active":"2024-04-14T15:18:17.900320+0000","last_peered":"2024-04-14T15:18:17.900320+0000","last_clean":"2024-04-14T15:18:17.900320+0000","last_became_active":"2024-04-14T15:09:47.044085+0000","last_became_peered":"2024-04-14T15:09:47.044085+0000","last_unstale":"2024-04-14T15:18:17.900320+0000","last_undegraded":"2024-04-14T15:18:17.900320+0000","last_fullsized":"2024-04-14T15:18:17.900320+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'9","last_scrub_stamp":"2024-04-14T15:17:54.845881+0000","last_deep_scrub":"400'9","last_deep_scrub_stamp":"2024-04-14T15:17:54.845881+0000","last_clean_scrub_stamp":"2024-04-14T15:17:54.845881+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:58.974945+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":742,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.899247+0000","last_change":"2024-04-14T15:16:17.041377+0000","last_active":"2024-04-14T15:18:17.899247+0000","last_peered":"2024-04-14T15:18:17.899247+0000","last_clean":"2024-04-14T15:18:17.899247+0000","last_became_active":"2024-04-14T15:09:49.065274+0000","last_became_peered":"2024-04-14T15:09:49.065274+0000","last_unstale":"2024-04-14T15:18:17.899247+0000","last_undegraded":"2024-04-14T15:18:17.899247+0000","last_fullsized":"2024-04-14T15:18:17.899247+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:17.041301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:17.041301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:22.487841+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"91'130","reported_seq":630,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.899846+0000","last_change":"2024-04-14T15:15:45.589217+0000","last_active":"2024-04-14T15:18:17.899846+0000","last_peered":"2024-04-14T15:18:17.899846+0000","last_clean":"2024-04-14T15:18:17.899846+0000","last_became_active":"2024-04-14T15:15:44.887114+0000","last_became_peered":"2024-04-14T15:15:44.887114+0000","last_unstale":"2024-04-14T15:18:17.899846+0000","last_undegraded":"2024-04-14T15:18:17.899846+0000","last_fullsized":"2024-04-14T15:18:17.899846+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:15:45.589132+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:15:45.589132+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:20.851885+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":653,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.899937+0000","last_change":"2024-04-14T15:15:29.366895+0000","last_active":"2024-04-14T15:18:17.899937+0000","last_peered":"2024-04-14T15:18:17.899937+0000","last_clean":"2024-04-14T15:18:17.899937+0000","last_became_active":"2024-04-14T15:12:56.487615+0000","last_became_peered":"2024-04-14T15:12:56.487615+0000","last_unstale":"2024-04-14T15:18:17.899937+0000","last_undegraded":"2024-04-14T15:18:17.899937+0000","last_fullsized":"2024-04-14T15:18:17.899937+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:29.366820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:29.366820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:20.951180+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"398'22","reported_seq":792,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.898826+0000","last_change":"2024-04-14T15:17:52.883984+0000","last_active":"2024-04-14T15:18:17.898826+0000","last_peered":"2024-04-14T15:18:17.898826+0000","last_clean":"2024-04-14T15:18:17.898826+0000","last_became_active":"2024-04-14T15:09:45.033331+0000","last_became_peered":"2024-04-14T15:09:45.033331+0000","last_unstale":"2024-04-14T15:18:17.898826+0000","last_undegraded":"2024-04-14T15:18:17.898826+0000","last_fullsized":"2024-04-14T15:18:17.898826+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'22","last_scrub_stamp":"2024-04-14T15:17:52.883850+0000","last_deep_scrub":"398'22","last_deep_scrub_stamp":"2024-04-14T15:17:52.883850+0000","last_clean_scrub_stamp":"2024-04-14T15:17:52.883850+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:58.280825+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":756,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.900969+0000","last_change":"2024-04-14T15:15:00.203399+0000","last_active":"2024-04-14T15:18:17.900969+0000","last_peered":"2024-04-14T15:18:17.900969+0000","last_clean":"2024-04-14T15:18:17.900969+0000","last_became_active":"2024-04-14T15:09:43.020878+0000","last_became_peered":"2024-04-14T15:09:43.020878+0000","last_unstale":"2024-04-14T15:18:17.900969+0000","last_undegraded":"2024-04-14T15:18:17.900969+0000","last_fullsized":"2024-04-14T15:18:17.900969+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:00.203310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:15:00.203310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:20.951097+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"72'8","reported_seq":858,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.899290+0000","last_change":"2024-04-14T15:16:06.933663+0000","last_active":"2024-04-14T15:18:17.899290+0000","last_peered":"2024-04-14T15:18:17.899290+0000","last_clean":"2024-04-14T15:18:17.899290+0000","last_became_active":"2024-04-14T15:09:47.050962+0000","last_became_peered":"2024-04-14T15:09:47.050962+0000","last_unstale":"2024-04-14T15:18:17.899290+0000","last_undegraded":"2024-04-14T15:18:17.899290+0000","last_fullsized":"2024-04-14T15:18:17.899290+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:16:06.933489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:06.933489+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:22.487858+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":316,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.899226+0000","last_change":"2024-04-14T15:16:51.199025+0000","last_active":"2024-04-14T15:18:17.899226+0000","last_peered":"2024-04-14T15:18:17.899226+0000","last_clean":"2024-04-14T15:18:17.899226+0000","last_became_active":"2024-04-14T15:15:44.879699+0000","last_became_peered":"2024-04-14T15:15:44.879699+0000","last_unstale":"2024-04-14T15:18:17.899226+0000","last_undegraded":"2024-04-14T15:18:17.899226+0000","last_fullsized":"2024-04-14T15:18:17.899226+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:51.198907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:51.198907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:19.763218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"233'8","reported_seq":674,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.909553+0000","last_change":"2024-04-14T15:17:50.905698+0000","last_active":"2024-04-14T15:18:17.909553+0000","last_peered":"2024-04-14T15:18:17.909553+0000","last_clean":"2024-04-14T15:18:17.909553+0000","last_became_active":"2024-04-14T15:12:56.381026+0000","last_became_peered":"2024-04-14T15:12:56.381026+0000","last_unstale":"2024-04-14T15:18:17.909553+0000","last_undegraded":"2024-04-14T15:18:17.909553+0000","last_fullsized":"2024-04-14T15:18:17.909553+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'8","last_scrub_stamp":"2024-04-14T15:17:50.905608+0000","last_deep_scrub":"233'8","last_deep_scrub_stamp":"2024-04-14T15:15:01.295800+0000","last_clean_scrub_stamp":"2024-04-14T15:17:50.905608+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:14.140648+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"85'16","reported_seq":782,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.899060+0000","last_change":"2024-04-14T15:16:53.143329+0000","last_active":"2024-04-14T15:18:17.899060+0000","last_peered":"2024-04-14T15:18:17.899060+0000","last_clean":"2024-04-14T15:18:17.899060+0000","last_became_active":"2024-04-14T15:09:45.028212+0000","last_became_peered":"2024-04-14T15:09:45.028212+0000","last_unstale":"2024-04-14T15:18:17.899060+0000","last_undegraded":"2024-04-14T15:18:17.899060+0000","last_fullsized":"2024-04-14T15:18:17.899060+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-14T15:16:53.143239+0000","last_deep_scrub":"85'16","last_deep_scrub_stamp":"2024-04-14T15:10:47.456169+0000","last_clean_scrub_stamp":"2024-04-14T15:16:53.143239+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:21.167906+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":762,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:18.542355+0000","last_change":"2024-04-14T15:17:39.465617+0000","last_active":"2024-04-14T15:18:18.542355+0000","last_peered":"2024-04-14T15:18:18.542355+0000","last_clean":"2024-04-14T15:18:18.542355+0000","last_became_active":"2024-04-14T15:09:43.018601+0000","last_became_peered":"2024-04-14T15:09:43.018601+0000","last_unstale":"2024-04-14T15:18:18.542355+0000","last_undegraded":"2024-04-14T15:18:18.542355+0000","last_fullsized":"2024-04-14T15:18:18.542355+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:39.465475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:39.465475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:44.272999+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":313,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.899320+0000","last_change":"2024-04-14T15:16:54.187971+0000","last_active":"2024-04-14T15:18:17.899320+0000","last_peered":"2024-04-14T15:18:17.899320+0000","last_clean":"2024-04-14T15:18:17.899320+0000","last_became_active":"2024-04-14T15:15:44.886834+0000","last_became_peered":"2024-04-14T15:15:44.886834+0000","last_unstale":"2024-04-14T15:18:17.899320+0000","last_undegraded":"2024-04-14T15:18:17.899320+0000","last_fullsized":"2024-04-14T15:18:17.899320+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:54.187888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:54.187888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:22.487960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":425,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.908603+0000","last_change":"2024-04-14T15:17:57.214134+0000","last_active":"2024-04-14T15:18:17.908603+0000","last_peered":"2024-04-14T15:18:17.908603+0000","last_clean":"2024-04-14T15:18:17.908603+0000","last_became_active":"2024-04-14T15:15:21.888587+0000","last_became_peered":"2024-04-14T15:15:21.888587+0000","last_unstale":"2024-04-14T15:18:17.908603+0000","last_undegraded":"2024-04-14T15:18:17.908603+0000","last_fullsized":"2024-04-14T15:18:17.908603+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:57.214046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:57.214046+0000","last_clean_scrub_stamp":"2024-04-14T15:17:57.214046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:03.985520+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"91'130","reported_seq":811,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.908442+0000","last_change":"2024-04-14T15:16:41.944414+0000","last_active":"2024-04-14T15:18:17.908442+0000","last_peered":"2024-04-14T15:18:17.908442+0000","last_clean":"2024-04-14T15:18:17.908442+0000","last_became_active":"2024-04-14T15:14:17.863503+0000","last_became_peered":"2024-04-14T15:14:17.863503+0000","last_unstale":"2024-04-14T15:18:17.908442+0000","last_undegraded":"2024-04-14T15:18:17.908442+0000","last_fullsized":"2024-04-14T15:18:17.908442+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:16:41.944267+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:16:41.944267+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.359204+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":659,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.899028+0000","last_change":"2024-04-14T15:16:56.214275+0000","last_active":"2024-04-14T15:18:17.899028+0000","last_peered":"2024-04-14T15:18:17.899028+0000","last_clean":"2024-04-14T15:18:17.899028+0000","last_became_active":"2024-04-14T15:12:56.427276+0000","last_became_peered":"2024-04-14T15:12:56.427276+0000","last_unstale":"2024-04-14T15:18:17.899028+0000","last_undegraded":"2024-04-14T15:18:17.899028+0000","last_fullsized":"2024-04-14T15:18:17.899028+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:56.214178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:16:56.214178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:22.487995+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"255'10","reported_seq":502,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.907081+0000","last_change":"2024-04-14T15:16:17.868542+0000","last_active":"2024-04-14T15:18:17.907081+0000","last_peered":"2024-04-14T15:18:17.907081+0000","last_clean":"2024-04-14T15:18:17.907081+0000","last_became_active":"2024-04-14T15:15:21.873747+0000","last_became_peered":"2024-04-14T15:15:21.873747+0000","last_unstale":"2024-04-14T15:18:17.907081+0000","last_undegraded":"2024-04-14T15:18:17.907081+0000","last_fullsized":"2024-04-14T15:18:17.907081+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'10","last_scrub_stamp":"2024-04-14T15:16:17.868428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:17.868428+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.359102+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"72'14","reported_seq":772,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.897912+0000","last_change":"2024-04-14T15:15:38.962791+0000","last_active":"2024-04-14T15:18:17.897912+0000","last_peered":"2024-04-14T15:18:17.897912+0000","last_clean":"2024-04-14T15:18:17.897912+0000","last_became_active":"2024-04-14T15:09:45.025673+0000","last_became_peered":"2024-04-14T15:09:45.025673+0000","last_unstale":"2024-04-14T15:18:17.897912+0000","last_undegraded":"2024-04-14T15:18:17.897912+0000","last_fullsized":"2024-04-14T15:18:17.897912+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'14","last_scrub_stamp":"2024-04-14T15:15:38.962680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:15:38.962680+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:22.487725+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":755,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.910323+0000","last_change":"2024-04-14T15:15:39.123305+0000","last_active":"2024-04-14T15:18:17.910323+0000","last_peered":"2024-04-14T15:18:17.910323+0000","last_clean":"2024-04-14T15:18:17.910323+0000","last_became_active":"2024-04-14T15:09:43.020503+0000","last_became_peered":"2024-04-14T15:09:43.020503+0000","last_unstale":"2024-04-14T15:18:17.910323+0000","last_undegraded":"2024-04-14T15:18:17.910323+0000","last_fullsized":"2024-04-14T15:18:17.910323+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:39.123227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:15:39.123227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:19.859165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":743,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:18.543919+0000","last_change":"2024-04-14T15:16:45.705837+0000","last_active":"2024-04-14T15:18:18.543919+0000","last_peered":"2024-04-14T15:18:18.543919+0000","last_clean":"2024-04-14T15:18:18.543919+0000","last_became_active":"2024-04-14T15:09:49.129610+0000","last_became_peered":"2024-04-14T15:09:49.129610+0000","last_unstale":"2024-04-14T15:18:18.543919+0000","last_undegraded":"2024-04-14T15:18:18.543919+0000","last_fullsized":"2024-04-14T15:18:18.543919+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:45.705761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:45.705761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.661209+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"91'130","reported_seq":813,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.898737+0000","last_change":"2024-04-14T15:16:41.193543+0000","last_active":"2024-04-14T15:18:17.898737+0000","last_peered":"2024-04-14T15:18:17.898737+0000","last_clean":"2024-04-14T15:18:17.898737+0000","last_became_active":"2024-04-14T15:14:17.871922+0000","last_became_peered":"2024-04-14T15:14:17.871922+0000","last_unstale":"2024-04-14T15:18:17.898737+0000","last_undegraded":"2024-04-14T15:18:17.898737+0000","last_fullsized":"2024-04-14T15:18:17.898737+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:16:41.193422+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:16:41.193422+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:22.487944+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":659,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:18.543480+0000","last_change":"2024-04-14T15:17:47.553179+0000","last_active":"2024-04-14T15:18:18.543480+0000","last_peered":"2024-04-14T15:18:18.543480+0000","last_clean":"2024-04-14T15:18:18.543480+0000","last_became_active":"2024-04-14T15:12:56.355254+0000","last_became_peered":"2024-04-14T15:12:56.355254+0000","last_unstale":"2024-04-14T15:18:18.543480+0000","last_undegraded":"2024-04-14T15:18:18.543480+0000","last_fullsized":"2024-04-14T15:18:18.543480+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:47.553055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:17:47.553055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:58.344778+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":747,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.898117+0000","last_change":"2024-04-14T15:16:05.917909+0000","last_active":"2024-04-14T15:18:17.898117+0000","last_peered":"2024-04-14T15:18:17.898117+0000","last_clean":"2024-04-14T15:18:17.898117+0000","last_became_active":"2024-04-14T15:09:47.044610+0000","last_became_peered":"2024-04-14T15:09:47.044610+0000","last_unstale":"2024-04-14T15:18:17.898117+0000","last_undegraded":"2024-04-14T15:18:17.898117+0000","last_fullsized":"2024-04-14T15:18:17.898117+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:05.917804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:05.917804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:22.487779+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"72'18","reported_seq":466,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.906931+0000","last_change":"2024-04-14T15:16:15.913377+0000","last_active":"2024-04-14T15:18:17.906931+0000","last_peered":"2024-04-14T15:18:17.906931+0000","last_clean":"2024-04-14T15:18:17.906931+0000","last_became_active":"2024-04-14T15:15:21.880565+0000","last_became_peered":"2024-04-14T15:15:21.880565+0000","last_unstale":"2024-04-14T15:18:17.906931+0000","last_undegraded":"2024-04-14T15:18:17.906931+0000","last_fullsized":"2024-04-14T15:18:17.906931+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'18","last_scrub_stamp":"2024-04-14T15:16:15.913273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:16:15.913273+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.359084+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'2","reported_seq":759,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.898370+0000","last_change":"2024-04-14T15:17:48.850178+0000","last_active":"2024-04-14T15:18:17.898370+0000","last_peered":"2024-04-14T15:18:17.898370+0000","last_clean":"2024-04-14T15:18:17.898370+0000","last_became_active":"2024-04-14T15:09:43.018478+0000","last_became_peered":"2024-04-14T15:09:43.018478+0000","last_unstale":"2024-04-14T15:18:17.898370+0000","last_undegraded":"2024-04-14T15:18:17.898370+0000","last_fullsized":"2024-04-14T15:18:17.898370+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'2","last_scrub_stamp":"2024-04-14T15:17:48.850069+0000","last_deep_scrub":"91'2","last_deep_scrub_stamp":"2024-04-14T15:17:48.850069+0000","last_clean_scrub_stamp":"2024-04-14T15:17:48.850069+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:12.966475+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":743,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.898808+0000","last_change":"2024-04-14T15:16:14.017818+0000","last_active":"2024-04-14T15:18:17.898808+0000","last_peered":"2024-04-14T15:18:17.898808+0000","last_clean":"2024-04-14T15:18:17.898808+0000","last_became_active":"2024-04-14T15:09:49.065684+0000","last_became_peered":"2024-04-14T15:09:49.065684+0000","last_unstale":"2024-04-14T15:18:17.898808+0000","last_undegraded":"2024-04-14T15:18:17.898808+0000","last_fullsized":"2024-04-14T15:18:17.898808+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:14.017729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:14.017729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:22.487794+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"91'130","reported_seq":821,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.909767+0000","last_change":"2024-04-14T15:17:47.885405+0000","last_active":"2024-04-14T15:18:17.909767+0000","last_peered":"2024-04-14T15:18:17.909767+0000","last_clean":"2024-04-14T15:18:17.909767+0000","last_became_active":"2024-04-14T15:14:17.853143+0000","last_became_peered":"2024-04-14T15:14:17.853143+0000","last_unstale":"2024-04-14T15:18:17.909767+0000","last_undegraded":"2024-04-14T15:18:17.909767+0000","last_fullsized":"2024-04-14T15:18:17.909767+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:17:47.885308+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:17:47.885308+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:05.853775+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":653,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.898522+0000","last_change":"2024-04-14T15:15:31.403165+0000","last_active":"2024-04-14T15:18:17.898522+0000","last_peered":"2024-04-14T15:18:17.898522+0000","last_clean":"2024-04-14T15:18:17.898522+0000","last_became_active":"2024-04-14T15:12:56.402023+0000","last_became_peered":"2024-04-14T15:12:56.402023+0000","last_unstale":"2024-04-14T15:18:17.898522+0000","last_undegraded":"2024-04-14T15:18:17.898522+0000","last_fullsized":"2024-04-14T15:18:17.898522+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:31.403086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:31.403086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:20.951131+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":756,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.909900+0000","last_change":"2024-04-14T15:17:39.682323+0000","last_active":"2024-04-14T15:18:17.909900+0000","last_peered":"2024-04-14T15:18:17.909900+0000","last_clean":"2024-04-14T15:18:17.909900+0000","last_became_active":"2024-04-14T15:09:47.048427+0000","last_became_peered":"2024-04-14T15:09:47.048427+0000","last_unstale":"2024-04-14T15:18:17.909900+0000","last_undegraded":"2024-04-14T15:18:17.909900+0000","last_fullsized":"2024-04-14T15:18:17.909900+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:39.682203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:17:39.682203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:58.685184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"102'27","reported_seq":805,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.906286+0000","last_change":"2024-04-14T15:17:46.924432+0000","last_active":"2024-04-14T15:18:17.906286+0000","last_peered":"2024-04-14T15:18:17.906286+0000","last_clean":"2024-04-14T15:18:17.906286+0000","last_became_active":"2024-04-14T15:09:45.179182+0000","last_became_peered":"2024-04-14T15:09:45.179182+0000","last_unstale":"2024-04-14T15:18:17.906286+0000","last_undegraded":"2024-04-14T15:18:17.906286+0000","last_fullsized":"2024-04-14T15:18:17.906286+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'27","last_scrub_stamp":"2024-04-14T15:17:46.924330+0000","last_deep_scrub":"102'27","last_deep_scrub_stamp":"2024-04-14T15:17:46.924330+0000","last_clean_scrub_stamp":"2024-04-14T15:17:46.924330+0000","objects_scrubbed":8,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:54.826490+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":755,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:18.543627+0000","last_change":"2024-04-14T15:16:18.774542+0000","last_active":"2024-04-14T15:18:18.543627+0000","last_peered":"2024-04-14T15:18:18.543627+0000","last_clean":"2024-04-14T15:18:18.543627+0000","last_became_active":"2024-04-14T15:09:43.021006+0000","last_became_peered":"2024-04-14T15:09:43.021006+0000","last_unstale":"2024-04-14T15:18:18.543627+0000","last_undegraded":"2024-04-14T15:18:18.543627+0000","last_fullsized":"2024-04-14T15:18:18.543627+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:18.774463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:18.774463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.661071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":745,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.901026+0000","last_change":"2024-04-14T15:16:56.206766+0000","last_active":"2024-04-14T15:18:17.901026+0000","last_peered":"2024-04-14T15:18:17.901026+0000","last_clean":"2024-04-14T15:18:17.901026+0000","last_became_active":"2024-04-14T15:09:49.128136+0000","last_became_peered":"2024-04-14T15:09:49.128136+0000","last_unstale":"2024-04-14T15:18:17.901026+0000","last_undegraded":"2024-04-14T15:18:17.901026+0000","last_fullsized":"2024-04-14T15:18:17.901026+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:56.206649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:56.206649+0000","last_clean_scrub_stamp":"2024-04-14T15:16:56.206649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:20.951333+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"91'130","reported_seq":815,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.898786+0000","last_change":"2024-04-14T15:17:49.890530+0000","last_active":"2024-04-14T15:18:17.898786+0000","last_peered":"2024-04-14T15:18:17.898786+0000","last_clean":"2024-04-14T15:18:17.898786+0000","last_became_active":"2024-04-14T15:14:17.856694+0000","last_became_peered":"2024-04-14T15:14:17.856694+0000","last_unstale":"2024-04-14T15:18:17.898786+0000","last_undegraded":"2024-04-14T15:18:17.898786+0000","last_fullsized":"2024-04-14T15:18:17.898786+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:17:49.890445+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:17:49.890445+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:16.061587+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":357,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.907493+0000","last_change":"2024-04-14T15:15:31.812967+0000","last_active":"2024-04-14T15:18:17.907493+0000","last_peered":"2024-04-14T15:18:17.907493+0000","last_clean":"2024-04-14T15:18:17.907493+0000","last_became_active":"2024-04-14T15:15:21.887769+0000","last_became_peered":"2024-04-14T15:15:21.887769+0000","last_unstale":"2024-04-14T15:18:17.907493+0000","last_undegraded":"2024-04-14T15:18:17.907493+0000","last_fullsized":"2024-04-14T15:18:17.907493+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:31.812888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:31.812888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.358900+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":315,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.899249+0000","last_change":"2024-04-14T15:17:47.871801+0000","last_active":"2024-04-14T15:18:17.899249+0000","last_peered":"2024-04-14T15:18:17.899249+0000","last_clean":"2024-04-14T15:18:17.899249+0000","last_became_active":"2024-04-14T15:15:44.883694+0000","last_became_peered":"2024-04-14T15:15:44.883694+0000","last_unstale":"2024-04-14T15:18:17.899249+0000","last_undegraded":"2024-04-14T15:18:17.899249+0000","last_fullsized":"2024-04-14T15:18:17.899249+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:47.871722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:17:47.871722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:15.357681+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"182'18","reported_seq":784,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.908948+0000","last_change":"2024-04-14T15:17:44.919445+0000","last_active":"2024-04-14T15:18:17.908948+0000","last_peered":"2024-04-14T15:18:17.908948+0000","last_clean":"2024-04-14T15:18:17.908948+0000","last_became_active":"2024-04-14T15:09:45.037231+0000","last_became_peered":"2024-04-14T15:09:45.037231+0000","last_unstale":"2024-04-14T15:18:17.908948+0000","last_undegraded":"2024-04-14T15:18:17.908948+0000","last_fullsized":"2024-04-14T15:18:17.908948+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'18","last_scrub_stamp":"2024-04-14T15:17:44.919358+0000","last_deep_scrub":"182'18","last_deep_scrub_stamp":"2024-04-14T15:14:10.383906+0000","last_clean_scrub_stamp":"2024-04-14T15:17:44.919358+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:48.791608+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":763,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:18.542470+0000","last_change":"2024-04-14T15:17:46.542125+0000","last_active":"2024-04-14T15:18:18.542470+0000","last_peered":"2024-04-14T15:18:18.542470+0000","last_clean":"2024-04-14T15:18:18.542470+0000","last_became_active":"2024-04-14T15:09:43.025095+0000","last_became_peered":"2024-04-14T15:09:43.025095+0000","last_unstale":"2024-04-14T15:18:18.542470+0000","last_undegraded":"2024-04-14T15:18:18.542470+0000","last_fullsized":"2024-04-14T15:18:18.542470+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:46.542030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:46.542030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:59.318501+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":772,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.906666+0000","last_change":"2024-04-14T15:17:48.866222+0000","last_active":"2024-04-14T15:18:17.906666+0000","last_peered":"2024-04-14T15:18:17.906666+0000","last_clean":"2024-04-14T15:18:17.906666+0000","last_became_active":"2024-04-14T15:09:43.018509+0000","last_became_peered":"2024-04-14T15:09:43.018509+0000","last_unstale":"2024-04-14T15:18:17.906666+0000","last_undegraded":"2024-04-14T15:18:17.906666+0000","last_fullsized":"2024-04-14T15:18:17.906666+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:48.866146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:48.866146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:57.582404+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0018592680000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"355'2","reported_seq":747,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.898829+0000","last_change":"2024-04-14T15:17:03.127190+0000","last_active":"2024-04-14T15:18:17.898829+0000","last_peered":"2024-04-14T15:18:17.898829+0000","last_clean":"2024-04-14T15:18:17.898829+0000","last_became_active":"2024-04-14T15:09:49.065243+0000","last_became_peered":"2024-04-14T15:09:49.065243+0000","last_unstale":"2024-04-14T15:18:17.898829+0000","last_undegraded":"2024-04-14T15:18:17.898829+0000","last_fullsized":"2024-04-14T15:18:17.898829+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'2","last_scrub_stamp":"2024-04-14T15:17:03.127112+0000","last_deep_scrub":"355'2","last_deep_scrub_stamp":"2024-04-14T15:17:03.127112+0000","last_clean_scrub_stamp":"2024-04-14T15:17:03.127112+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.009834+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":349,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.908112+0000","last_change":"2024-04-14T15:15:21.890855+0000","last_active":"2024-04-14T15:18:17.908112+0000","last_peered":"2024-04-14T15:18:17.908112+0000","last_clean":"2024-04-14T15:18:17.908112+0000","last_became_active":"2024-04-14T15:15:21.890115+0000","last_became_peered":"2024-04-14T15:15:21.890115+0000","last_unstale":"2024-04-14T15:18:17.908112+0000","last_undegraded":"2024-04-14T15:18:17.908112+0000","last_fullsized":"2024-04-14T15:18:17.908112+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:45.148342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:14:45.148342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.358831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":747,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:18.543293+0000","last_change":"2024-04-14T15:16:51.617717+0000","last_active":"2024-04-14T15:18:18.543293+0000","last_peered":"2024-04-14T15:18:18.543293+0000","last_clean":"2024-04-14T15:18:18.543293+0000","last_became_active":"2024-04-14T15:09:47.047201+0000","last_became_peered":"2024-04-14T15:09:47.047201+0000","last_unstale":"2024-04-14T15:18:18.543293+0000","last_undegraded":"2024-04-14T15:18:18.543293+0000","last_fullsized":"2024-04-14T15:18:18.543293+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:51.617615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:51.617615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.661149+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"90'24","reported_seq":886,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.909866+0000","last_change":"2024-04-14T15:16:19.842719+0000","last_active":"2024-04-14T15:18:17.909866+0000","last_peered":"2024-04-14T15:18:17.909866+0000","last_clean":"2024-04-14T15:18:17.909866+0000","last_became_active":"2024-04-14T15:09:45.179978+0000","last_became_peered":"2024-04-14T15:09:45.179978+0000","last_unstale":"2024-04-14T15:18:17.909866+0000","last_undegraded":"2024-04-14T15:18:17.909866+0000","last_fullsized":"2024-04-14T15:18:17.909866+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'24","last_scrub_stamp":"2024-04-14T15:16:19.842614+0000","last_deep_scrub":"90'24","last_deep_scrub_stamp":"2024-04-14T15:11:25.120692+0000","last_clean_scrub_stamp":"2024-04-14T15:16:19.842614+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:19.859263+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"91'130","reported_seq":1025,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:18.542373+0000","last_change":"2024-04-14T15:16:43.730573+0000","last_active":"2024-04-14T15:18:18.542373+0000","last_peered":"2024-04-14T15:18:18.542373+0000","last_clean":"2024-04-14T15:18:18.542373+0000","last_became_active":"2024-04-14T15:14:12.858953+0000","last_became_peered":"2024-04-14T15:14:12.858953+0000","last_unstale":"2024-04-14T15:18:18.542373+0000","last_undegraded":"2024-04-14T15:18:18.542373+0000","last_fullsized":"2024-04-14T15:18:18.542373+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:16:43.730442+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:16:43.730442+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.661183+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":320,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.900586+0000","last_change":"2024-04-14T15:17:35.705777+0000","last_active":"2024-04-14T15:18:17.900586+0000","last_peered":"2024-04-14T15:18:17.900586+0000","last_clean":"2024-04-14T15:18:17.900586+0000","last_became_active":"2024-04-14T15:15:44.885315+0000","last_became_peered":"2024-04-14T15:15:44.885315+0000","last_unstale":"2024-04-14T15:18:17.900586+0000","last_undegraded":"2024-04-14T15:18:17.900586+0000","last_fullsized":"2024-04-14T15:18:17.900586+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:35.705645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:47.627305+0000","last_clean_scrub_stamp":"2024-04-14T15:17:35.705645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:49.306972+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"69'2","reported_seq":447,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.908171+0000","last_change":"2024-04-14T15:16:53.972591+0000","last_active":"2024-04-14T15:18:17.908171+0000","last_peered":"2024-04-14T15:18:17.908171+0000","last_clean":"2024-04-14T15:18:17.908171+0000","last_became_active":"2024-04-14T15:15:21.889851+0000","last_became_peered":"2024-04-14T15:15:21.889851+0000","last_unstale":"2024-04-14T15:18:17.908171+0000","last_undegraded":"2024-04-14T15:18:17.908171+0000","last_fullsized":"2024-04-14T15:18:17.908171+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-04-14T15:16:53.972502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:53.972502+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:20.876593+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0058046169999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":312,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.910570+0000","last_change":"2024-04-14T15:16:57.935611+0000","last_active":"2024-04-14T15:18:17.910570+0000","last_peered":"2024-04-14T15:18:17.910570+0000","last_clean":"2024-04-14T15:18:17.910570+0000","last_became_active":"2024-04-14T15:15:44.887934+0000","last_became_peered":"2024-04-14T15:15:44.887934+0000","last_unstale":"2024-04-14T15:18:17.910570+0000","last_undegraded":"2024-04-14T15:18:17.910570+0000","last_fullsized":"2024-04-14T15:18:17.910570+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:57.935436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:57.935436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.359186+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"91'130","reported_seq":1032,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:18.542673+0000","last_change":"2024-04-14T15:17:41.509304+0000","last_active":"2024-04-14T15:18:18.542673+0000","last_peered":"2024-04-14T15:18:18.542673+0000","last_clean":"2024-04-14T15:18:18.542673+0000","last_became_active":"2024-04-14T15:14:17.858029+0000","last_became_peered":"2024-04-14T15:14:17.858029+0000","last_unstale":"2024-04-14T15:18:18.542673+0000","last_undegraded":"2024-04-14T15:18:18.542673+0000","last_fullsized":"2024-04-14T15:18:18.542673+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:17:41.509215+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:17:41.509215+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:05.623460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":349,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.906790+0000","last_change":"2024-04-14T15:15:21.890274+0000","last_active":"2024-04-14T15:18:17.906790+0000","last_peered":"2024-04-14T15:18:17.906790+0000","last_clean":"2024-04-14T15:18:17.906790+0000","last_became_active":"2024-04-14T15:15:21.889970+0000","last_became_peered":"2024-04-14T15:15:21.889970+0000","last_unstale":"2024-04-14T15:18:17.906790+0000","last_undegraded":"2024-04-14T15:18:17.906790+0000","last_fullsized":"2024-04-14T15:18:17.906790+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:54.156421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:14:54.156421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.358796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":753,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.902628+0000","last_change":"2024-04-14T15:17:01.653342+0000","last_active":"2024-04-14T15:18:17.902628+0000","last_peered":"2024-04-14T15:18:17.902628+0000","last_clean":"2024-04-14T15:18:17.902628+0000","last_became_active":"2024-04-14T15:09:47.049457+0000","last_became_peered":"2024-04-14T15:09:47.049457+0000","last_unstale":"2024-04-14T15:18:17.902628+0000","last_undegraded":"2024-04-14T15:18:17.902628+0000","last_fullsized":"2024-04-14T15:18:17.902628+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:01.653271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:17:01.653271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:29.819865+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"308'18","reported_seq":784,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.899090+0000","last_change":"2024-04-14T15:16:16.054128+0000","last_active":"2024-04-14T15:18:17.899090+0000","last_peered":"2024-04-14T15:18:17.899090+0000","last_clean":"2024-04-14T15:18:17.899090+0000","last_became_active":"2024-04-14T15:09:45.033883+0000","last_became_peered":"2024-04-14T15:09:45.033883+0000","last_unstale":"2024-04-14T15:18:17.899090+0000","last_undegraded":"2024-04-14T15:18:17.899090+0000","last_fullsized":"2024-04-14T15:18:17.899090+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"308'18","last_scrub_stamp":"2024-04-14T15:16:16.054032+0000","last_deep_scrub":"308'18","last_deep_scrub_stamp":"2024-04-14T15:16:16.054032+0000","last_clean_scrub_stamp":"2024-04-14T15:16:16.054032+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:20.951301+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":762,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.899701+0000","last_change":"2024-04-14T15:16:43.686142+0000","last_active":"2024-04-14T15:18:17.899701+0000","last_peered":"2024-04-14T15:18:17.899701+0000","last_clean":"2024-04-14T15:18:17.899701+0000","last_became_active":"2024-04-14T15:09:43.021188+0000","last_became_peered":"2024-04-14T15:09:43.021188+0000","last_unstale":"2024-04-14T15:18:17.899701+0000","last_undegraded":"2024-04-14T15:18:17.899701+0000","last_fullsized":"2024-04-14T15:18:17.899701+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:43.686033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:43.686033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:20.852024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"73'5","reported_seq":1276,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.901385+0000","last_change":"2024-04-14T15:15:28.328150+0000","last_active":"2024-04-14T15:18:17.901385+0000","last_peered":"2024-04-14T15:18:17.901385+0000","last_clean":"2024-04-14T15:18:17.901385+0000","last_became_active":"2024-04-14T15:09:43.016544+0000","last_became_peered":"2024-04-14T15:09:43.016544+0000","last_unstale":"2024-04-14T15:18:17.901385+0000","last_undegraded":"2024-04-14T15:18:17.901385+0000","last_fullsized":"2024-04-14T15:18:17.901385+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-04-14T15:15:28.328050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:15:28.328050+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:20.951116+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00056542099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":509,"num_read_kb":504,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":743,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:18.543750+0000","last_change":"2024-04-14T15:16:20.827681+0000","last_active":"2024-04-14T15:18:18.543750+0000","last_peered":"2024-04-14T15:18:18.543750+0000","last_clean":"2024-04-14T15:18:18.543750+0000","last_became_active":"2024-04-14T15:09:49.130307+0000","last_became_peered":"2024-04-14T15:09:49.130307+0000","last_unstale":"2024-04-14T15:18:18.543750+0000","last_undegraded":"2024-04-14T15:18:18.543750+0000","last_fullsized":"2024-04-14T15:18:18.543750+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:20.827589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:20.827589+0000","last_clean_scrub_stamp":"2024-04-14T15:16:20.827589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.661118+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"91'130","reported_seq":639,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.905632+0000","last_change":"2024-04-14T15:15:45.362962+0000","last_active":"2024-04-14T15:18:17.905632+0000","last_peered":"2024-04-14T15:18:17.905632+0000","last_clean":"2024-04-14T15:18:17.905632+0000","last_became_active":"2024-04-14T15:15:44.884623+0000","last_became_peered":"2024-04-14T15:15:44.884623+0000","last_unstale":"2024-04-14T15:18:17.905632+0000","last_undegraded":"2024-04-14T15:18:17.905632+0000","last_fullsized":"2024-04-14T15:18:17.905632+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:15:45.362860+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:15:45.362860+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:21.922463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":357,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.907219+0000","last_change":"2024-04-14T15:15:32.839576+0000","last_active":"2024-04-14T15:18:17.907219+0000","last_peered":"2024-04-14T15:18:17.907219+0000","last_clean":"2024-04-14T15:18:17.907219+0000","last_became_active":"2024-04-14T15:15:21.883717+0000","last_became_peered":"2024-04-14T15:15:21.883717+0000","last_unstale":"2024-04-14T15:18:17.907219+0000","last_undegraded":"2024-04-14T15:18:17.907219+0000","last_fullsized":"2024-04-14T15:18:17.907219+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:32.839491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:32.839491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.358965+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":749,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.906331+0000","last_change":"2024-04-14T15:16:53.731699+0000","last_active":"2024-04-14T15:18:17.906331+0000","last_peered":"2024-04-14T15:18:17.906331+0000","last_clean":"2024-04-14T15:18:17.906331+0000","last_became_active":"2024-04-14T15:09:47.040185+0000","last_became_peered":"2024-04-14T15:09:47.040185+0000","last_unstale":"2024-04-14T15:18:17.906331+0000","last_undegraded":"2024-04-14T15:18:17.906331+0000","last_fullsized":"2024-04-14T15:18:17.906331+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:53.731610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:53.731610+0000","last_clean_scrub_stamp":"2024-04-14T15:16:53.731610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:21.922645+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"72'28","reported_seq":905,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.898472+0000","last_change":"2024-04-14T15:16:43.036263+0000","last_active":"2024-04-14T15:18:17.898472+0000","last_peered":"2024-04-14T15:18:17.898472+0000","last_clean":"2024-04-14T15:18:17.898472+0000","last_became_active":"2024-04-14T15:09:45.034168+0000","last_became_peered":"2024-04-14T15:09:45.034168+0000","last_unstale":"2024-04-14T15:18:17.898472+0000","last_undegraded":"2024-04-14T15:18:17.898472+0000","last_fullsized":"2024-04-14T15:18:17.898472+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'28","last_scrub_stamp":"2024-04-14T15:16:43.036159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:16:43.036159+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:20.951318+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":315,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.898749+0000","last_change":"2024-04-14T15:16:19.101437+0000","last_active":"2024-04-14T15:18:17.898749+0000","last_peered":"2024-04-14T15:18:17.898749+0000","last_clean":"2024-04-14T15:18:17.898749+0000","last_became_active":"2024-04-14T15:15:44.885315+0000","last_became_peered":"2024-04-14T15:15:44.885315+0000","last_unstale":"2024-04-14T15:18:17.898749+0000","last_undegraded":"2024-04-14T15:18:17.898749+0000","last_fullsized":"2024-04-14T15:18:17.898749+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:19.101368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:19.101368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:20.951234+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":417,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.909720+0000","last_change":"2024-04-14T15:17:44.084142+0000","last_active":"2024-04-14T15:18:17.909720+0000","last_peered":"2024-04-14T15:18:17.909720+0000","last_clean":"2024-04-14T15:18:17.909720+0000","last_became_active":"2024-04-14T15:15:21.874742+0000","last_became_peered":"2024-04-14T15:15:21.874742+0000","last_unstale":"2024-04-14T15:18:17.909720+0000","last_undegraded":"2024-04-14T15:18:17.909720+0000","last_fullsized":"2024-04-14T15:18:17.909720+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:44.084010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:44.084010+0000","last_clean_scrub_stamp":"2024-04-14T15:17:44.084010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:03.733939+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"91'130","reported_seq":821,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.910143+0000","last_change":"2024-04-14T15:17:40.725976+0000","last_active":"2024-04-14T15:18:17.910143+0000","last_peered":"2024-04-14T15:18:17.910143+0000","last_clean":"2024-04-14T15:18:17.910143+0000","last_became_active":"2024-04-14T15:14:17.859910+0000","last_became_peered":"2024-04-14T15:14:17.859910+0000","last_unstale":"2024-04-14T15:18:17.910143+0000","last_undegraded":"2024-04-14T15:18:17.910143+0000","last_fullsized":"2024-04-14T15:18:17.910143+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:17:40.725873+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:17:40.725873+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:54.360389+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":658,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.900908+0000","last_change":"2024-04-14T15:17:00.656174+0000","last_active":"2024-04-14T15:18:17.900908+0000","last_peered":"2024-04-14T15:18:17.900908+0000","last_clean":"2024-04-14T15:18:17.900908+0000","last_became_active":"2024-04-14T15:12:56.350703+0000","last_became_peered":"2024-04-14T15:12:56.350703+0000","last_unstale":"2024-04-14T15:18:17.900908+0000","last_undegraded":"2024-04-14T15:18:17.900908+0000","last_fullsized":"2024-04-14T15:18:17.900908+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:00.656085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:17:00.656085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:20.852125+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"72'8","reported_seq":869,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.906033+0000","last_change":"2024-04-14T15:17:04.695161+0000","last_active":"2024-04-14T15:18:17.906033+0000","last_peered":"2024-04-14T15:18:17.906033+0000","last_clean":"2024-04-14T15:18:17.906033+0000","last_became_active":"2024-04-14T15:09:47.047699+0000","last_became_peered":"2024-04-14T15:09:47.047699+0000","last_unstale":"2024-04-14T15:18:17.906033+0000","last_undegraded":"2024-04-14T15:18:17.906033+0000","last_fullsized":"2024-04-14T15:18:17.906033+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:17:04.695079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:17:04.695079+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:21.922693+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"72'10","reported_seq":743,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.900551+0000","last_change":"2024-04-14T15:15:41.610385+0000","last_active":"2024-04-14T15:18:17.900551+0000","last_peered":"2024-04-14T15:18:17.900551+0000","last_clean":"2024-04-14T15:18:17.900551+0000","last_became_active":"2024-04-14T15:10:07.764042+0000","last_became_peered":"2024-04-14T15:10:07.764042+0000","last_unstale":"2024-04-14T15:18:17.900551+0000","last_undegraded":"2024-04-14T15:18:17.900551+0000","last_fullsized":"2024-04-14T15:18:17.900551+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-04-14T15:15:41.610279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:15:41.610279+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:20.851916+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":757,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.898344+0000","last_change":"2024-04-14T15:17:40.971492+0000","last_active":"2024-04-14T15:18:17.898344+0000","last_peered":"2024-04-14T15:18:17.898344+0000","last_clean":"2024-04-14T15:18:17.898344+0000","last_became_active":"2024-04-14T15:09:43.022597+0000","last_became_peered":"2024-04-14T15:09:43.022597+0000","last_unstale":"2024-04-14T15:18:17.898344+0000","last_undegraded":"2024-04-14T15:18:17.898344+0000","last_fullsized":"2024-04-14T15:18:17.898344+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:40.971412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:40.971412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:08.233588+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":745,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.905506+0000","last_change":"2024-04-14T15:16:17.405931+0000","last_active":"2024-04-14T15:18:17.905506+0000","last_peered":"2024-04-14T15:18:17.905506+0000","last_clean":"2024-04-14T15:18:17.905506+0000","last_became_active":"2024-04-14T15:09:49.066648+0000","last_became_peered":"2024-04-14T15:09:49.066648+0000","last_unstale":"2024-04-14T15:18:17.905506+0000","last_undegraded":"2024-04-14T15:18:17.905506+0000","last_fullsized":"2024-04-14T15:18:17.905506+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:17.405850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:17.405850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:21.922562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":653,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.908766+0000","last_change":"2024-04-14T15:15:36.403919+0000","last_active":"2024-04-14T15:18:17.908766+0000","last_peered":"2024-04-14T15:18:17.908766+0000","last_clean":"2024-04-14T15:18:17.908766+0000","last_became_active":"2024-04-14T15:12:56.563074+0000","last_became_peered":"2024-04-14T15:12:56.563074+0000","last_unstale":"2024-04-14T15:18:17.908766+0000","last_undegraded":"2024-04-14T15:18:17.908766+0000","last_fullsized":"2024-04-14T15:18:17.908766+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:36.403843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:36.403843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:21.922444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":753,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.897977+0000","last_change":"2024-04-14T15:17:56.915660+0000","last_active":"2024-04-14T15:18:17.897977+0000","last_peered":"2024-04-14T15:18:17.897977+0000","last_clean":"2024-04-14T15:18:17.897977+0000","last_became_active":"2024-04-14T15:09:47.049005+0000","last_became_peered":"2024-04-14T15:09:47.049005+0000","last_unstale":"2024-04-14T15:18:17.897977+0000","last_undegraded":"2024-04-14T15:18:17.897977+0000","last_fullsized":"2024-04-14T15:18:17.897977+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:56.915544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:56.915544+0000","last_clean_scrub_stamp":"2024-04-14T15:17:56.915544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:24.542571+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"183'17","reported_seq":786,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.898113+0000","last_change":"2024-04-14T15:16:17.997616+0000","last_active":"2024-04-14T15:18:17.898113+0000","last_peered":"2024-04-14T15:18:17.898113+0000","last_clean":"2024-04-14T15:18:17.898113+0000","last_became_active":"2024-04-14T15:09:45.032892+0000","last_became_peered":"2024-04-14T15:09:45.032892+0000","last_unstale":"2024-04-14T15:18:17.898113+0000","last_undegraded":"2024-04-14T15:18:17.898113+0000","last_fullsized":"2024-04-14T15:18:17.898113+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'17","last_scrub_stamp":"2024-04-14T15:16:17.997529+0000","last_deep_scrub":"183'17","last_deep_scrub_stamp":"2024-04-14T15:14:11.577574+0000","last_clean_scrub_stamp":"2024-04-14T15:16:17.997529+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:22.487892+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"65'3","reported_seq":772,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.899237+0000","last_change":"2024-04-14T15:16:12.030448+0000","last_active":"2024-04-14T15:18:17.899237+0000","last_peered":"2024-04-14T15:18:17.899237+0000","last_clean":"2024-04-14T15:18:17.899237+0000","last_became_active":"2024-04-14T15:09:43.020490+0000","last_became_peered":"2024-04-14T15:09:43.020490+0000","last_unstale":"2024-04-14T15:18:17.899237+0000","last_undegraded":"2024-04-14T15:18:17.899237+0000","last_fullsized":"2024-04-14T15:18:17.899237+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3","last_scrub_stamp":"2024-04-14T15:16:12.030363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:12.030363+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:22.487875+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":528,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":417,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.907509+0000","last_change":"2024-04-14T15:16:58.942591+0000","last_active":"2024-04-14T15:18:17.907509+0000","last_peered":"2024-04-14T15:18:17.907509+0000","last_clean":"2024-04-14T15:18:17.907509+0000","last_became_active":"2024-04-14T15:15:21.876410+0000","last_became_peered":"2024-04-14T15:15:21.876410+0000","last_unstale":"2024-04-14T15:18:17.907509+0000","last_undegraded":"2024-04-14T15:18:17.907509+0000","last_fullsized":"2024-04-14T15:18:17.907509+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:58.942499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:58.942499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.359220+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":661,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.906543+0000","last_change":"2024-04-14T15:16:43.696614+0000","last_active":"2024-04-14T15:18:17.906543+0000","last_peered":"2024-04-14T15:18:17.906543+0000","last_clean":"2024-04-14T15:18:17.906543+0000","last_became_active":"2024-04-14T15:12:56.551177+0000","last_became_peered":"2024-04-14T15:12:56.551177+0000","last_unstale":"2024-04-14T15:18:17.906543+0000","last_undegraded":"2024-04-14T15:18:17.906543+0000","last_fullsized":"2024-04-14T15:18:17.906543+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:43.696536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:16:43.696536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:21.922630+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"347'13","reported_seq":772,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.909051+0000","last_change":"2024-04-14T15:16:55.694703+0000","last_active":"2024-04-14T15:18:17.909051+0000","last_peered":"2024-04-14T15:18:17.909051+0000","last_clean":"2024-04-14T15:18:17.909051+0000","last_became_active":"2024-04-14T15:09:45.180375+0000","last_became_peered":"2024-04-14T15:09:45.180375+0000","last_unstale":"2024-04-14T15:18:17.909051+0000","last_undegraded":"2024-04-14T15:18:17.909051+0000","last_fullsized":"2024-04-14T15:18:17.909051+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"347'13","last_scrub_stamp":"2024-04-14T15:16:55.694599+0000","last_deep_scrub":"347'13","last_deep_scrub_stamp":"2024-04-14T15:16:55.694599+0000","last_clean_scrub_stamp":"2024-04-14T15:16:55.694599+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:21.922662+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":761,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.897940+0000","last_change":"2024-04-14T15:16:41.677902+0000","last_active":"2024-04-14T15:18:17.897940+0000","last_peered":"2024-04-14T15:18:17.897940+0000","last_clean":"2024-04-14T15:18:17.897940+0000","last_became_active":"2024-04-14T15:09:43.016788+0000","last_became_peered":"2024-04-14T15:09:43.016788+0000","last_unstale":"2024-04-14T15:18:17.897940+0000","last_undegraded":"2024-04-14T15:18:17.897940+0000","last_fullsized":"2024-04-14T15:18:17.897940+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:41.677826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:41.677826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:20.852039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":429,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.910484+0000","last_change":"2024-04-14T15:15:42.769452+0000","last_active":"2024-04-14T15:18:17.910484+0000","last_peered":"2024-04-14T15:18:17.910484+0000","last_clean":"2024-04-14T15:18:17.910484+0000","last_became_active":"2024-04-14T15:15:21.873580+0000","last_became_peered":"2024-04-14T15:15:21.873580+0000","last_unstale":"2024-04-14T15:18:17.910484+0000","last_undegraded":"2024-04-14T15:18:17.910484+0000","last_fullsized":"2024-04-14T15:18:17.910484+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:42.769375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:15:42.769375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.358933+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":749,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.900975+0000","last_change":"2024-04-14T15:16:47.671808+0000","last_active":"2024-04-14T15:18:17.900975+0000","last_peered":"2024-04-14T15:18:17.900975+0000","last_clean":"2024-04-14T15:18:17.900975+0000","last_became_active":"2024-04-14T15:09:49.064296+0000","last_became_peered":"2024-04-14T15:09:49.064296+0000","last_unstale":"2024-04-14T15:18:17.900975+0000","last_undegraded":"2024-04-14T15:18:17.900975+0000","last_fullsized":"2024-04-14T15:18:17.900975+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:47.671717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:47.671717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:20.852055+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":658,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.897850+0000","last_change":"2024-04-14T15:17:36.728543+0000","last_active":"2024-04-14T15:18:17.897850+0000","last_peered":"2024-04-14T15:18:17.897850+0000","last_clean":"2024-04-14T15:18:17.897850+0000","last_became_active":"2024-04-14T15:12:56.340066+0000","last_became_peered":"2024-04-14T15:12:56.340066+0000","last_unstale":"2024-04-14T15:18:17.897850+0000","last_undegraded":"2024-04-14T15:18:17.897850+0000","last_fullsized":"2024-04-14T15:18:17.897850+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:36.728465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:17:36.728465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:54.134352+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.17","version":"72'6","reported_seq":765,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.910391+0000","last_change":"2024-04-14T15:17:42.019838+0000","last_active":"2024-04-14T15:18:17.910391+0000","last_peered":"2024-04-14T15:18:17.910391+0000","last_clean":"2024-04-14T15:18:17.910391+0000","last_became_active":"2024-04-14T15:09:45.180037+0000","last_became_peered":"2024-04-14T15:09:45.180037+0000","last_unstale":"2024-04-14T15:18:17.910391+0000","last_undegraded":"2024-04-14T15:18:17.910391+0000","last_fullsized":"2024-04-14T15:18:17.910391+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'6","last_scrub_stamp":"2024-04-14T15:17:42.019717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:17:42.019717+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:52.276684+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":763,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.909048+0000","last_change":"2024-04-14T15:16:59.640700+0000","last_active":"2024-04-14T15:18:17.909048+0000","last_peered":"2024-04-14T15:18:17.909048+0000","last_clean":"2024-04-14T15:18:17.909048+0000","last_became_active":"2024-04-14T15:09:43.018987+0000","last_became_peered":"2024-04-14T15:09:43.018987+0000","last_unstale":"2024-04-14T15:18:17.909048+0000","last_undegraded":"2024-04-14T15:18:17.909048+0000","last_fullsized":"2024-04-14T15:18:17.909048+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:59.640565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:59.640565+0000","last_clean_scrub_stamp":"2024-04-14T15:16:59.640565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:28.007249+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":311,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.908540+0000","last_change":"2024-04-14T15:17:47.072247+0000","last_active":"2024-04-14T15:18:17.908540+0000","last_peered":"2024-04-14T15:18:17.908540+0000","last_clean":"2024-04-14T15:18:17.908540+0000","last_became_active":"2024-04-14T15:15:44.889786+0000","last_became_peered":"2024-04-14T15:15:44.889786+0000","last_unstale":"2024-04-14T15:18:17.908540+0000","last_undegraded":"2024-04-14T15:18:17.908540+0000","last_fullsized":"2024-04-14T15:18:17.908540+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:47.072170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:13.735024+0000","last_clean_scrub_stamp":"2024-04-14T15:17:47.072170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:03.533500+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":742,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:18.543961+0000","last_change":"2024-04-14T15:16:54.720823+0000","last_active":"2024-04-14T15:18:18.543961+0000","last_peered":"2024-04-14T15:18:18.543961+0000","last_clean":"2024-04-14T15:18:18.543961+0000","last_became_active":"2024-04-14T15:09:49.130438+0000","last_became_peered":"2024-04-14T15:09:49.130438+0000","last_unstale":"2024-04-14T15:18:18.543961+0000","last_undegraded":"2024-04-14T15:18:18.543961+0000","last_fullsized":"2024-04-14T15:18:18.543961+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:54.720722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:54.720722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.661165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":659,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.898233+0000","last_change":"2024-04-14T15:17:46.369320+0000","last_active":"2024-04-14T15:18:17.898233+0000","last_peered":"2024-04-14T15:18:17.898233+0000","last_clean":"2024-04-14T15:18:17.898233+0000","last_became_active":"2024-04-14T15:12:56.203577+0000","last_became_peered":"2024-04-14T15:12:56.203577+0000","last_unstale":"2024-04-14T15:18:17.898233+0000","last_undegraded":"2024-04-14T15:18:17.898233+0000","last_fullsized":"2024-04-14T15:18:17.898233+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:46.369231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:46.369231+0000","last_clean_scrub_stamp":"2024-04-14T15:17:46.369231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:57.370694+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"72'10","reported_seq":448,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.908639+0000","last_change":"2024-04-14T15:15:30.806041+0000","last_active":"2024-04-14T15:18:17.908639+0000","last_peered":"2024-04-14T15:18:17.908639+0000","last_clean":"2024-04-14T15:18:17.908639+0000","last_became_active":"2024-04-14T15:15:21.877667+0000","last_became_peered":"2024-04-14T15:15:21.877667+0000","last_unstale":"2024-04-14T15:18:17.908639+0000","last_undegraded":"2024-04-14T15:18:17.908639+0000","last_fullsized":"2024-04-14T15:18:17.908639+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-04-14T15:15:30.805923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:15:30.805923+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.358866+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":311,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.908275+0000","last_change":"2024-04-14T15:16:47.075044+0000","last_active":"2024-04-14T15:18:17.908275+0000","last_peered":"2024-04-14T15:18:17.908275+0000","last_clean":"2024-04-14T15:18:17.908275+0000","last_became_active":"2024-04-14T15:15:44.887452+0000","last_became_peered":"2024-04-14T15:15:44.887452+0000","last_unstale":"2024-04-14T15:18:17.908275+0000","last_undegraded":"2024-04-14T15:18:17.908275+0000","last_fullsized":"2024-04-14T15:18:17.908275+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:47.074950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:47.074950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.359135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"280'9","reported_seq":371,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.899527+0000","last_change":"2024-04-14T15:17:50.908130+0000","last_active":"2024-04-14T15:18:17.899527+0000","last_peered":"2024-04-14T15:18:17.899527+0000","last_clean":"2024-04-14T15:18:17.899527+0000","last_became_active":"2024-04-14T15:15:44.882916+0000","last_became_peered":"2024-04-14T15:15:44.882916+0000","last_unstale":"2024-04-14T15:18:17.899527+0000","last_undegraded":"2024-04-14T15:18:17.899527+0000","last_fullsized":"2024-04-14T15:18:17.899527+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"280'9","last_scrub_stamp":"2024-04-14T15:17:50.908054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:17:50.908054+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:20.243489+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":744,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.908846+0000","last_change":"2024-04-14T15:16:54.724807+0000","last_active":"2024-04-14T15:18:17.908846+0000","last_peered":"2024-04-14T15:18:17.908846+0000","last_clean":"2024-04-14T15:18:17.908846+0000","last_became_active":"2024-04-14T15:09:49.066068+0000","last_became_peered":"2024-04-14T15:09:49.066068+0000","last_unstale":"2024-04-14T15:18:17.908846+0000","last_undegraded":"2024-04-14T15:18:17.908846+0000","last_fullsized":"2024-04-14T15:18:17.908846+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:54.724681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:54.724681+0000","last_clean_scrub_stamp":"2024-04-14T15:16:54.724681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":303,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.908031+0000","last_change":"2024-04-14T15:15:44.889470+0000","last_active":"2024-04-14T15:18:17.908031+0000","last_peered":"2024-04-14T15:18:17.908031+0000","last_clean":"2024-04-14T15:18:17.908031+0000","last_became_active":"2024-04-14T15:15:44.889106+0000","last_became_peered":"2024-04-14T15:15:44.889106+0000","last_unstale":"2024-04-14T15:18:17.908031+0000","last_undegraded":"2024-04-14T15:18:17.908031+0000","last_fullsized":"2024-04-14T15:18:17.908031+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:01.356001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:01.356001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.358815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"183'13","reported_seq":592,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:18.543818+0000","last_change":"2024-04-14T15:17:37.421888+0000","last_active":"2024-04-14T15:18:18.543818+0000","last_peered":"2024-04-14T15:18:18.543818+0000","last_clean":"2024-04-14T15:18:18.543818+0000","last_became_active":"2024-04-14T15:15:44.884211+0000","last_became_peered":"2024-04-14T15:15:44.884211+0000","last_unstale":"2024-04-14T15:18:18.543818+0000","last_undegraded":"2024-04-14T15:18:18.543818+0000","last_fullsized":"2024-04-14T15:18:18.543818+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'13","last_scrub_stamp":"2024-04-14T15:17:37.421739+0000","last_deep_scrub":"183'13","last_deep_scrub_stamp":"2024-04-14T15:14:11.755252+0000","last_clean_scrub_stamp":"2024-04-14T15:17:37.421739+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:55.179831+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":764,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.906198+0000","last_change":"2024-04-14T15:17:02.670294+0000","last_active":"2024-04-14T15:18:17.906198+0000","last_peered":"2024-04-14T15:18:17.906198+0000","last_clean":"2024-04-14T15:18:17.906198+0000","last_became_active":"2024-04-14T15:09:43.019301+0000","last_became_peered":"2024-04-14T15:09:43.019301+0000","last_unstale":"2024-04-14T15:18:17.906198+0000","last_undegraded":"2024-04-14T15:18:17.906198+0000","last_fullsized":"2024-04-14T15:18:17.906198+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:02.670213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:02.670213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"255'10","reported_seq":504,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.909801+0000","last_change":"2024-04-14T15:16:59.946889+0000","last_active":"2024-04-14T15:18:17.909801+0000","last_peered":"2024-04-14T15:18:17.909801+0000","last_clean":"2024-04-14T15:18:17.909801+0000","last_became_active":"2024-04-14T15:15:21.884468+0000","last_became_peered":"2024-04-14T15:15:21.884468+0000","last_unstale":"2024-04-14T15:18:17.909801+0000","last_undegraded":"2024-04-14T15:18:17.909801+0000","last_fullsized":"2024-04-14T15:18:17.909801+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'10","last_scrub_stamp":"2024-04-14T15:16:59.946804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:59.946804+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.359169+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":742,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.899365+0000","last_change":"2024-04-14T15:16:03.948252+0000","last_active":"2024-04-14T15:18:17.899365+0000","last_peered":"2024-04-14T15:18:17.899365+0000","last_clean":"2024-04-14T15:18:17.899365+0000","last_became_active":"2024-04-14T15:09:49.065242+0000","last_became_peered":"2024-04-14T15:09:49.065242+0000","last_unstale":"2024-04-14T15:18:17.899365+0000","last_undegraded":"2024-04-14T15:18:17.899365+0000","last_fullsized":"2024-04-14T15:18:17.899365+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:03.948140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:03.948140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:22.487810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":651,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:18.544004+0000","last_change":"2024-04-14T15:15:34.555810+0000","last_active":"2024-04-14T15:18:18.544004+0000","last_peered":"2024-04-14T15:18:18.544004+0000","last_clean":"2024-04-14T15:18:18.544004+0000","last_became_active":"2024-04-14T15:12:56.389204+0000","last_became_peered":"2024-04-14T15:12:56.389204+0000","last_unstale":"2024-04-14T15:18:18.544004+0000","last_undegraded":"2024-04-14T15:18:18.544004+0000","last_fullsized":"2024-04-14T15:18:18.544004+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:34.555728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:34.555728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.661056+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"344'13","reported_seq":282,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.909664+0000","last_change":"2024-04-14T15:16:52.738298+0000","last_active":"2024-04-14T15:18:17.909664+0000","last_peered":"2024-04-14T15:18:17.909664+0000","last_clean":"2024-04-14T15:18:17.909664+0000","last_became_active":"2024-04-14T15:16:24.993130+0000","last_became_peered":"2024-04-14T15:16:24.993130+0000","last_unstale":"2024-04-14T15:18:17.909664+0000","last_undegraded":"2024-04-14T15:18:17.909664+0000","last_fullsized":"2024-04-14T15:18:17.909664+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"344'13","last_scrub_stamp":"2024-04-14T15:16:52.738163+0000","last_deep_scrub":"344'13","last_deep_scrub_stamp":"2024-04-14T15:16:52.738163+0000","last_clean_scrub_stamp":"2024-04-14T15:16:52.738163+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:19.859282+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":763,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.909477+0000","last_change":"2024-04-14T15:17:04.625913+0000","last_active":"2024-04-14T15:18:17.909477+0000","last_peered":"2024-04-14T15:18:17.909477+0000","last_clean":"2024-04-14T15:18:17.909477+0000","last_became_active":"2024-04-14T15:09:43.022480+0000","last_became_peered":"2024-04-14T15:09:43.022480+0000","last_unstale":"2024-04-14T15:18:17.909477+0000","last_undegraded":"2024-04-14T15:18:17.909477+0000","last_fullsized":"2024-04-14T15:18:17.909477+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:04.625824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:04.625824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:28.834815+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":421,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.907383+0000","last_change":"2024-04-14T15:15:21.885176+0000","last_active":"2024-04-14T15:18:17.907383+0000","last_peered":"2024-04-14T15:18:17.907383+0000","last_clean":"2024-04-14T15:18:17.907383+0000","last_became_active":"2024-04-14T15:15:21.882643+0000","last_became_peered":"2024-04-14T15:15:21.882643+0000","last_unstale":"2024-04-14T15:18:17.907383+0000","last_undegraded":"2024-04-14T15:18:17.907383+0000","last_fullsized":"2024-04-14T15:18:17.907383+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:54.252608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:14:54.252608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.358846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":747,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.905455+0000","last_change":"2024-04-14T15:17:58.899572+0000","last_active":"2024-04-14T15:18:17.905455+0000","last_peered":"2024-04-14T15:18:17.905455+0000","last_clean":"2024-04-14T15:18:17.905455+0000","last_became_active":"2024-04-14T15:09:49.059711+0000","last_became_peered":"2024-04-14T15:09:49.059711+0000","last_unstale":"2024-04-14T15:18:17.905455+0000","last_undegraded":"2024-04-14T15:18:17.905455+0000","last_fullsized":"2024-04-14T15:18:17.905455+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:53.219686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:14:53.219686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:21.922427+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":660,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.909344+0000","last_change":"2024-04-14T15:17:51.921346+0000","last_active":"2024-04-14T15:18:17.909344+0000","last_peered":"2024-04-14T15:18:17.909344+0000","last_clean":"2024-04-14T15:18:17.909344+0000","last_became_active":"2024-04-14T15:12:56.396294+0000","last_became_peered":"2024-04-14T15:12:56.396294+0000","last_unstale":"2024-04-14T15:18:17.909344+0000","last_undegraded":"2024-04-14T15:18:17.909344+0000","last_fullsized":"2024-04-14T15:18:17.909344+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:51.921269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:17:51.921269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:54.899694+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"72'11","reported_seq":769,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.898379+0000","last_change":"2024-04-14T15:16:08.917185+0000","last_active":"2024-04-14T15:18:17.898379+0000","last_peered":"2024-04-14T15:18:17.898379+0000","last_clean":"2024-04-14T15:18:17.898379+0000","last_became_active":"2024-04-14T15:09:45.026065+0000","last_became_peered":"2024-04-14T15:09:45.026065+0000","last_unstale":"2024-04-14T15:18:17.898379+0000","last_undegraded":"2024-04-14T15:18:17.898379+0000","last_fullsized":"2024-04-14T15:18:17.898379+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-04-14T15:16:08.917092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:16:08.917092+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:22.487827+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":764,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.899580+0000","last_change":"2024-04-14T15:17:34.986735+0000","last_active":"2024-04-14T15:18:17.899580+0000","last_peered":"2024-04-14T15:18:17.899580+0000","last_clean":"2024-04-14T15:18:17.899580+0000","last_became_active":"2024-04-14T15:09:43.023583+0000","last_became_peered":"2024-04-14T15:09:43.023583+0000","last_unstale":"2024-04-14T15:18:17.899580+0000","last_undegraded":"2024-04-14T15:18:17.899580+0000","last_fullsized":"2024-04-14T15:18:17.899580+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:34.986655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:34.986655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:41.802704+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":748,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.899030+0000","last_change":"2024-04-14T15:17:00.205888+0000","last_active":"2024-04-14T15:18:17.899030+0000","last_peered":"2024-04-14T15:18:17.899030+0000","last_clean":"2024-04-14T15:18:17.899030+0000","last_became_active":"2024-04-14T15:09:47.041683+0000","last_became_peered":"2024-04-14T15:09:47.041683+0000","last_unstale":"2024-04-14T15:18:17.899030+0000","last_undegraded":"2024-04-14T15:18:17.899030+0000","last_fullsized":"2024-04-14T15:18:17.899030+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:00.205808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:08.486217+0000","last_clean_scrub_stamp":"2024-04-14T15:17:00.205808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.009817+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":425,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.907143+0000","last_change":"2024-04-14T15:16:50.021545+0000","last_active":"2024-04-14T15:18:17.907143+0000","last_peered":"2024-04-14T15:18:17.907143+0000","last_clean":"2024-04-14T15:18:17.907143+0000","last_became_active":"2024-04-14T15:15:21.889567+0000","last_became_peered":"2024-04-14T15:15:21.889567+0000","last_unstale":"2024-04-14T15:18:17.907143+0000","last_undegraded":"2024-04-14T15:18:17.907143+0000","last_fullsized":"2024-04-14T15:18:17.907143+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:50.021470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:50.021470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.359151+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":658,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.898745+0000","last_change":"2024-04-14T15:17:03.685135+0000","last_active":"2024-04-14T15:18:17.898745+0000","last_peered":"2024-04-14T15:18:17.898745+0000","last_clean":"2024-04-14T15:18:17.898745+0000","last_became_active":"2024-04-14T15:12:56.387723+0000","last_became_peered":"2024-04-14T15:12:56.387723+0000","last_unstale":"2024-04-14T15:18:17.898745+0000","last_undegraded":"2024-04-14T15:18:17.898745+0000","last_fullsized":"2024-04-14T15:18:17.898745+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:03.685045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:03.685045+0000","last_clean_scrub_stamp":"2024-04-14T15:17:03.685045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.10","version":"72'4","reported_seq":762,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.906699+0000","last_change":"2024-04-14T15:17:53.181132+0000","last_active":"2024-04-14T15:18:17.906699+0000","last_peered":"2024-04-14T15:18:17.906699+0000","last_clean":"2024-04-14T15:18:17.906699+0000","last_became_active":"2024-04-14T15:09:45.038815+0000","last_became_peered":"2024-04-14T15:09:45.038815+0000","last_unstale":"2024-04-14T15:18:17.906699+0000","last_undegraded":"2024-04-14T15:18:17.906699+0000","last_fullsized":"2024-04-14T15:18:17.906699+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4","last_scrub_stamp":"2024-04-14T15:17:53.181053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:17:53.181053+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:15.210428+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":314,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.910155+0000","last_change":"2024-04-14T15:17:48.845163+0000","last_active":"2024-04-14T15:18:17.910155+0000","last_peered":"2024-04-14T15:18:17.910155+0000","last_clean":"2024-04-14T15:18:17.910155+0000","last_became_active":"2024-04-14T15:15:44.884730+0000","last_became_peered":"2024-04-14T15:15:44.884730+0000","last_unstale":"2024-04-14T15:18:17.910155+0000","last_undegraded":"2024-04-14T15:18:17.910155+0000","last_fullsized":"2024-04-14T15:18:17.910155+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:48.845081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:08.712289+0000","last_clean_scrub_stamp":"2024-04-14T15:17:48.845081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:15.427392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":755,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.909561+0000","last_change":"2024-04-14T15:16:13.909554+0000","last_active":"2024-04-14T15:18:17.909561+0000","last_peered":"2024-04-14T15:18:17.909561+0000","last_clean":"2024-04-14T15:18:17.909561+0000","last_became_active":"2024-04-14T15:09:47.041074+0000","last_became_peered":"2024-04-14T15:09:47.041074+0000","last_unstale":"2024-04-14T15:18:17.909561+0000","last_undegraded":"2024-04-14T15:18:17.909561+0000","last_fullsized":"2024-04-14T15:18:17.909561+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:13.909454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:13.909454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:19.859226+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":742,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:18.542987+0000","last_change":"2024-04-14T15:16:44.728025+0000","last_active":"2024-04-14T15:18:18.542987+0000","last_peered":"2024-04-14T15:18:18.542987+0000","last_clean":"2024-04-14T15:18:18.542987+0000","last_became_active":"2024-04-14T15:09:49.061359+0000","last_became_peered":"2024-04-14T15:09:49.061359+0000","last_unstale":"2024-04-14T15:18:18.542987+0000","last_undegraded":"2024-04-14T15:18:18.542987+0000","last_fullsized":"2024-04-14T15:18:18.542987+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:44.727907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:44.727907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.661241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":651,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:18.544030+0000","last_change":"2024-04-14T15:15:43.680936+0000","last_active":"2024-04-14T15:18:18.544030+0000","last_peered":"2024-04-14T15:18:18.544030+0000","last_clean":"2024-04-14T15:18:18.544030+0000","last_became_active":"2024-04-14T15:12:56.352528+0000","last_became_peered":"2024-04-14T15:12:56.352528+0000","last_unstale":"2024-04-14T15:18:18.544030+0000","last_undegraded":"2024-04-14T15:18:18.544030+0000","last_fullsized":"2024-04-14T15:18:18.544030+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:43.680852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:15:43.680852+0000","last_clean_scrub_stamp":"2024-04-14T15:15:43.680852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.661005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"102'16","reported_seq":418,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.907805+0000","last_change":"2024-04-14T15:17:43.061551+0000","last_active":"2024-04-14T15:18:17.907805+0000","last_peered":"2024-04-14T15:18:17.907805+0000","last_clean":"2024-04-14T15:18:17.907805+0000","last_became_active":"2024-04-14T15:15:21.884783+0000","last_became_peered":"2024-04-14T15:15:21.884783+0000","last_unstale":"2024-04-14T15:18:17.907805+0000","last_undegraded":"2024-04-14T15:18:17.907805+0000","last_fullsized":"2024-04-14T15:18:17.907805+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'16","last_scrub_stamp":"2024-04-14T15:17:43.061458+0000","last_deep_scrub":"102'16","last_deep_scrub_stamp":"2024-04-14T15:12:35.999326+0000","last_clean_scrub_stamp":"2024-04-14T15:17:43.061458+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:06.716506+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":763,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.910291+0000","last_change":"2024-04-14T15:17:01.643838+0000","last_active":"2024-04-14T15:18:17.910291+0000","last_peered":"2024-04-14T15:18:17.910291+0000","last_clean":"2024-04-14T15:18:17.910291+0000","last_became_active":"2024-04-14T15:09:43.023220+0000","last_became_peered":"2024-04-14T15:09:43.023220+0000","last_unstale":"2024-04-14T15:18:17.910291+0000","last_undegraded":"2024-04-14T15:18:17.910291+0000","last_fullsized":"2024-04-14T15:18:17.910291+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:01.643745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:01.643745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:26.033027+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":746,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:18.542029+0000","last_change":"2024-04-14T15:16:50.661634+0000","last_active":"2024-04-14T15:18:18.542029+0000","last_peered":"2024-04-14T15:18:18.542029+0000","last_clean":"2024-04-14T15:18:18.542029+0000","last_became_active":"2024-04-14T15:09:47.044665+0000","last_became_peered":"2024-04-14T15:09:47.044665+0000","last_unstale":"2024-04-14T15:18:18.542029+0000","last_undegraded":"2024-04-14T15:18:18.542029+0000","last_fullsized":"2024-04-14T15:18:18.542029+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:50.661520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:50.661520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.661226+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":425,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.907515+0000","last_change":"2024-04-14T15:16:05.885527+0000","last_active":"2024-04-14T15:18:17.907515+0000","last_peered":"2024-04-14T15:18:17.907515+0000","last_clean":"2024-04-14T15:18:17.907515+0000","last_became_active":"2024-04-14T15:15:21.883940+0000","last_became_peered":"2024-04-14T15:15:21.883940+0000","last_unstale":"2024-04-14T15:18:17.907515+0000","last_undegraded":"2024-04-14T15:18:17.907515+0000","last_fullsized":"2024-04-14T15:18:17.907515+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:05.885438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:05.885438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.359049+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":650,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.899431+0000","last_change":"2024-04-14T15:15:28.474035+0000","last_active":"2024-04-14T15:18:17.899431+0000","last_peered":"2024-04-14T15:18:17.899431+0000","last_clean":"2024-04-14T15:18:17.899431+0000","last_became_active":"2024-04-14T15:12:56.335230+0000","last_became_peered":"2024-04-14T15:12:56.335230+0000","last_unstale":"2024-04-14T15:18:17.899431+0000","last_undegraded":"2024-04-14T15:18:17.899431+0000","last_fullsized":"2024-04-14T15:18:17.899431+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:28.473899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:28.473899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:20.851831+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"72'10","reported_seq":768,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.909238+0000","last_change":"2024-04-14T15:16:49.779653+0000","last_active":"2024-04-14T15:18:17.909238+0000","last_peered":"2024-04-14T15:18:17.909238+0000","last_clean":"2024-04-14T15:18:17.909238+0000","last_became_active":"2024-04-14T15:09:45.181456+0000","last_became_peered":"2024-04-14T15:09:45.181456+0000","last_unstale":"2024-04-14T15:18:17.909238+0000","last_undegraded":"2024-04-14T15:18:17.909238+0000","last_fullsized":"2024-04-14T15:18:17.909238+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-04-14T15:16:49.779557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:16:49.779557+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:21.922710+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":313,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.898114+0000","last_change":"2024-04-14T15:16:42.205142+0000","last_active":"2024-04-14T15:18:17.898114+0000","last_peered":"2024-04-14T15:18:17.898114+0000","last_clean":"2024-04-14T15:18:17.898114+0000","last_became_active":"2024-04-14T15:15:44.878336+0000","last_became_peered":"2024-04-14T15:15:44.878336+0000","last_unstale":"2024-04-14T15:18:17.898114+0000","last_undegraded":"2024-04-14T15:18:17.898114+0000","last_fullsized":"2024-04-14T15:18:17.898114+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:42.205063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:42.205063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:22.487927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"72'8","reported_seq":871,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.910460+0000","last_change":"2024-04-14T15:16:12.917059+0000","last_active":"2024-04-14T15:18:17.910460+0000","last_peered":"2024-04-14T15:18:17.910460+0000","last_clean":"2024-04-14T15:18:17.910460+0000","last_became_active":"2024-04-14T15:09:47.048216+0000","last_became_peered":"2024-04-14T15:09:47.048216+0000","last_unstale":"2024-04-14T15:18:17.910460+0000","last_undegraded":"2024-04-14T15:18:17.910460+0000","last_fullsized":"2024-04-14T15:18:17.910460+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:16:12.916977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:12.916977+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:19.859209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"72'1","reported_seq":751,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:18.543406+0000","last_change":"2024-04-14T15:17:50.529174+0000","last_active":"2024-04-14T15:18:18.543406+0000","last_peered":"2024-04-14T15:18:18.543406+0000","last_clean":"2024-04-14T15:18:18.543406+0000","last_became_active":"2024-04-14T15:09:49.062051+0000","last_became_peered":"2024-04-14T15:09:49.062051+0000","last_unstale":"2024-04-14T15:18:18.543406+0000","last_undegraded":"2024-04-14T15:18:18.543406+0000","last_fullsized":"2024-04-14T15:18:18.543406+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-04-14T15:17:50.529089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:17:50.529089+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:53.773686+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":653,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.905571+0000","last_change":"2024-04-14T15:16:09.471961+0000","last_active":"2024-04-14T15:18:17.905571+0000","last_peered":"2024-04-14T15:18:17.905571+0000","last_clean":"2024-04-14T15:18:17.905571+0000","last_became_active":"2024-04-14T15:12:56.543793+0000","last_became_peered":"2024-04-14T15:12:56.543793+0000","last_unstale":"2024-04-14T15:18:17.905571+0000","last_undegraded":"2024-04-14T15:18:17.905571+0000","last_fullsized":"2024-04-14T15:18:17.905571+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:09.471852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:16:09.471852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:21.922547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"72'11","reported_seq":789,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.910519+0000","last_change":"2024-04-14T15:16:06.886451+0000","last_active":"2024-04-14T15:18:17.910519+0000","last_peered":"2024-04-14T15:18:17.910519+0000","last_clean":"2024-04-14T15:18:17.910519+0000","last_became_active":"2024-04-14T15:10:07.764183+0000","last_became_peered":"2024-04-14T15:10:07.764183+0000","last_unstale":"2024-04-14T15:18:17.910519+0000","last_undegraded":"2024-04-14T15:18:17.910519+0000","last_fullsized":"2024-04-14T15:18:17.910519+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-04-14T15:16:06.886325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:16:06.886325+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:19.859244+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"65'2","reported_seq":444,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.907387+0000","last_change":"2024-04-14T15:16:56.943047+0000","last_active":"2024-04-14T15:18:17.907387+0000","last_peered":"2024-04-14T15:18:17.907387+0000","last_clean":"2024-04-14T15:18:17.907387+0000","last_became_active":"2024-04-14T15:15:21.885343+0000","last_became_peered":"2024-04-14T15:15:21.885343+0000","last_unstale":"2024-04-14T15:18:17.907387+0000","last_undegraded":"2024-04-14T15:18:17.907387+0000","last_fullsized":"2024-04-14T15:18:17.907387+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-04-14T15:16:56.942840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:56.942840+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:20.441230+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":749,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.909506+0000","last_change":"2024-04-14T15:16:52.747151+0000","last_active":"2024-04-14T15:18:17.909506+0000","last_peered":"2024-04-14T15:18:17.909506+0000","last_clean":"2024-04-14T15:18:17.909506+0000","last_became_active":"2024-04-14T15:09:47.045730+0000","last_became_peered":"2024-04-14T15:09:47.045730+0000","last_unstale":"2024-04-14T15:18:17.909506+0000","last_undegraded":"2024-04-14T15:18:17.909506+0000","last_fullsized":"2024-04-14T15:18:17.909506+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:52.747070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:52.747070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:21.922615+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":744,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.899335+0000","last_change":"2024-04-14T15:16:57.235339+0000","last_active":"2024-04-14T15:18:17.899335+0000","last_peered":"2024-04-14T15:18:17.899335+0000","last_clean":"2024-04-14T15:18:17.899335+0000","last_became_active":"2024-04-14T15:09:49.057545+0000","last_became_peered":"2024-04-14T15:09:49.057545+0000","last_unstale":"2024-04-14T15:18:17.899335+0000","last_undegraded":"2024-04-14T15:18:17.899335+0000","last_fullsized":"2024-04-14T15:18:17.899335+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:57.235261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:21.247695+0000","last_clean_scrub_stamp":"2024-04-14T15:16:57.235261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.621753+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":313,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.898776+0000","last_change":"2024-04-14T15:17:44.364114+0000","last_active":"2024-04-14T15:18:17.898776+0000","last_peered":"2024-04-14T15:18:17.898776+0000","last_clean":"2024-04-14T15:18:17.898776+0000","last_became_active":"2024-04-14T15:15:44.882604+0000","last_became_peered":"2024-04-14T15:15:44.882604+0000","last_unstale":"2024-04-14T15:18:17.898776+0000","last_undegraded":"2024-04-14T15:18:17.898776+0000","last_fullsized":"2024-04-14T15:18:17.898776+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:44.364013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:17:44.364013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:01.492147+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"72'15","reported_seq":774,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.901694+0000","last_change":"2024-04-14T15:15:44.609525+0000","last_active":"2024-04-14T15:18:17.901694+0000","last_peered":"2024-04-14T15:18:17.901694+0000","last_clean":"2024-04-14T15:18:17.901694+0000","last_became_active":"2024-04-14T15:09:45.032887+0000","last_became_peered":"2024-04-14T15:09:45.032887+0000","last_unstale":"2024-04-14T15:18:17.901694+0000","last_undegraded":"2024-04-14T15:18:17.901694+0000","last_fullsized":"2024-04-14T15:18:17.901694+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-04-14T15:15:44.609332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:15:44.609332+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:20.851868+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":758,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.909794+0000","last_change":"2024-04-14T15:16:09.825296+0000","last_active":"2024-04-14T15:18:17.909794+0000","last_peered":"2024-04-14T15:18:17.909794+0000","last_clean":"2024-04-14T15:18:17.909794+0000","last_became_active":"2024-04-14T15:09:43.021494+0000","last_became_peered":"2024-04-14T15:09:43.021494+0000","last_unstale":"2024-04-14T15:18:17.909794+0000","last_undegraded":"2024-04-14T15:18:17.909794+0000","last_fullsized":"2024-04-14T15:18:17.909794+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:09.825220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:09.825220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:19.859185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":748,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.899554+0000","last_change":"2024-04-14T15:14:56.199840+0000","last_active":"2024-04-14T15:18:17.899554+0000","last_peered":"2024-04-14T15:18:17.899554+0000","last_clean":"2024-04-14T15:18:17.899554+0000","last_became_active":"2024-04-14T15:09:47.042733+0000","last_became_peered":"2024-04-14T15:09:47.042733+0000","last_unstale":"2024-04-14T15:18:17.899554+0000","last_undegraded":"2024-04-14T15:18:17.899554+0000","last_fullsized":"2024-04-14T15:18:17.899554+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:56.199756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:14:56.199756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:20.951080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":751,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.899125+0000","last_change":"2024-04-14T15:17:54.453289+0000","last_active":"2024-04-14T15:18:17.899125+0000","last_peered":"2024-04-14T15:18:17.899125+0000","last_clean":"2024-04-14T15:18:17.899125+0000","last_became_active":"2024-04-14T15:09:49.127979+0000","last_became_peered":"2024-04-14T15:09:49.127979+0000","last_unstale":"2024-04-14T15:18:17.899125+0000","last_undegraded":"2024-04-14T15:18:17.899125+0000","last_fullsized":"2024-04-14T15:18:17.899125+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:54.453174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:54.453174+0000","last_clean_scrub_stamp":"2024-04-14T15:17:54.453174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:08.115614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":651,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:18.542730+0000","last_change":"2024-04-14T15:15:31.635164+0000","last_active":"2024-04-14T15:18:18.542730+0000","last_peered":"2024-04-14T15:18:18.542730+0000","last_clean":"2024-04-14T15:18:18.542730+0000","last_became_active":"2024-04-14T15:12:56.392352+0000","last_became_peered":"2024-04-14T15:12:56.392352+0000","last_unstale":"2024-04-14T15:18:18.542730+0000","last_undegraded":"2024-04-14T15:18:18.542730+0000","last_fullsized":"2024-04-14T15:18:18.542730+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:31.635060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:31.635060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.660973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"72'12","reported_seq":456,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.907463+0000","last_change":"2024-04-14T15:16:11.903062+0000","last_active":"2024-04-14T15:18:17.907463+0000","last_peered":"2024-04-14T15:18:17.907463+0000","last_clean":"2024-04-14T15:18:17.907463+0000","last_became_active":"2024-04-14T15:15:21.877838+0000","last_became_peered":"2024-04-14T15:15:21.877838+0000","last_unstale":"2024-04-14T15:18:17.907463+0000","last_undegraded":"2024-04-14T15:18:17.907463+0000","last_fullsized":"2024-04-14T15:18:17.907463+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'12","last_scrub_stamp":"2024-04-14T15:16:11.902934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:16:11.902934+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.359013+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":437,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.907392+0000","last_change":"2024-04-14T15:16:20.969027+0000","last_active":"2024-04-14T15:18:17.907392+0000","last_peered":"2024-04-14T15:18:17.907392+0000","last_clean":"2024-04-14T15:18:17.907392+0000","last_became_active":"2024-04-14T15:15:21.883544+0000","last_became_peered":"2024-04-14T15:15:21.883544+0000","last_unstale":"2024-04-14T15:18:17.907392+0000","last_undegraded":"2024-04-14T15:18:17.907392+0000","last_fullsized":"2024-04-14T15:18:17.907392+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:20.968897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:20.968897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.359066+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":754,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.898487+0000","last_change":"2024-04-14T15:17:01.098681+0000","last_active":"2024-04-14T15:18:17.898487+0000","last_peered":"2024-04-14T15:18:17.898487+0000","last_clean":"2024-04-14T15:18:17.898487+0000","last_became_active":"2024-04-14T15:09:47.048045+0000","last_became_peered":"2024-04-14T15:09:47.048045+0000","last_unstale":"2024-04-14T15:18:17.898487+0000","last_undegraded":"2024-04-14T15:18:17.898487+0000","last_fullsized":"2024-04-14T15:18:17.898487+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:01.098599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:17:01.098599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:20.592323+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":426,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.910386+0000","last_change":"2024-04-14T15:16:07.879394+0000","last_active":"2024-04-14T15:18:17.910386+0000","last_peered":"2024-04-14T15:18:17.910386+0000","last_clean":"2024-04-14T15:18:17.910386+0000","last_became_active":"2024-04-14T15:15:21.889691+0000","last_became_peered":"2024-04-14T15:15:21.889691+0000","last_unstale":"2024-04-14T15:18:17.910386+0000","last_undegraded":"2024-04-14T15:18:17.910386+0000","last_fullsized":"2024-04-14T15:18:17.910386+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:07.879280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:07.879280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.359032+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":658,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.901002+0000","last_change":"2024-04-14T15:17:37.766225+0000","last_active":"2024-04-14T15:18:17.901002+0000","last_peered":"2024-04-14T15:18:17.901002+0000","last_clean":"2024-04-14T15:18:17.901002+0000","last_became_active":"2024-04-14T15:12:56.375278+0000","last_became_peered":"2024-04-14T15:12:56.375278+0000","last_unstale":"2024-04-14T15:18:17.901002+0000","last_undegraded":"2024-04-14T15:18:17.901002+0000","last_fullsized":"2024-04-14T15:18:17.901002+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:37.766116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:17:37.766116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:42.299956+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"276'13","reported_seq":778,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.898424+0000","last_change":"2024-04-14T15:17:52.445394+0000","last_active":"2024-04-14T15:18:17.898424+0000","last_peered":"2024-04-14T15:18:17.898424+0000","last_clean":"2024-04-14T15:18:17.898424+0000","last_became_active":"2024-04-14T15:09:45.030093+0000","last_became_peered":"2024-04-14T15:09:45.030093+0000","last_unstale":"2024-04-14T15:18:17.898424+0000","last_undegraded":"2024-04-14T15:18:17.898424+0000","last_fullsized":"2024-04-14T15:18:17.898424+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'13","last_scrub_stamp":"2024-04-14T15:17:52.445294+0000","last_deep_scrub":"276'13","last_deep_scrub_stamp":"2024-04-14T15:15:43.969775+0000","last_clean_scrub_stamp":"2024-04-14T15:17:52.445294+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:59.668815+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":460,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:18.542985+0000","last_change":"2024-04-14T15:17:02.800274+0000","last_active":"2024-04-14T15:18:18.542985+0000","last_peered":"2024-04-14T15:18:18.542985+0000","last_clean":"2024-04-14T15:18:18.542985+0000","last_became_active":"2024-04-14T15:14:30.857727+0000","last_became_peered":"2024-04-14T15:14:30.857727+0000","last_unstale":"2024-04-14T15:18:18.542985+0000","last_undegraded":"2024-04-14T15:18:18.542985+0000","last_fullsized":"2024-04-14T15:18:18.542985+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:02.800166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:02.800166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:20.592445+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":315,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.906388+0000","last_change":"2024-04-14T15:16:59.695423+0000","last_active":"2024-04-14T15:18:17.906388+0000","last_peered":"2024-04-14T15:18:17.906388+0000","last_clean":"2024-04-14T15:18:17.906388+0000","last_became_active":"2024-04-14T15:15:44.883121+0000","last_became_peered":"2024-04-14T15:15:44.883121+0000","last_unstale":"2024-04-14T15:18:17.906388+0000","last_undegraded":"2024-04-14T15:18:17.906388+0000","last_fullsized":"2024-04-14T15:18:17.906388+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:59.695337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:59.695337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:23.332757+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":752,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.907180+0000","last_change":"2024-04-14T15:17:33.952503+0000","last_active":"2024-04-14T15:18:17.907180+0000","last_peered":"2024-04-14T15:18:17.907180+0000","last_clean":"2024-04-14T15:18:17.907180+0000","last_became_active":"2024-04-14T15:09:49.061756+0000","last_became_peered":"2024-04-14T15:09:49.061756+0000","last_unstale":"2024-04-14T15:18:17.907180+0000","last_undegraded":"2024-04-14T15:18:17.907180+0000","last_fullsized":"2024-04-14T15:18:17.907180+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:33.952312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:17:33.952312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:36.590330+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":660,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.909532+0000","last_change":"2024-04-14T15:16:57.667484+0000","last_active":"2024-04-14T15:18:17.909532+0000","last_peered":"2024-04-14T15:18:17.909532+0000","last_clean":"2024-04-14T15:18:17.909532+0000","last_became_active":"2024-04-14T15:12:56.232831+0000","last_became_peered":"2024-04-14T15:12:56.232831+0000","last_unstale":"2024-04-14T15:18:17.909532+0000","last_undegraded":"2024-04-14T15:18:17.909532+0000","last_fullsized":"2024-04-14T15:18:17.909532+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:57.667382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:16:57.667382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:26.482036+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1b","version":"87'7","reported_seq":762,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.898560+0000","last_change":"2024-04-14T15:15:44.927867+0000","last_active":"2024-04-14T15:18:17.898560+0000","last_peered":"2024-04-14T15:18:17.898560+0000","last_clean":"2024-04-14T15:18:17.898560+0000","last_became_active":"2024-04-14T15:09:45.028667+0000","last_became_peered":"2024-04-14T15:09:45.028667+0000","last_unstale":"2024-04-14T15:18:17.898560+0000","last_undegraded":"2024-04-14T15:18:17.898560+0000","last_fullsized":"2024-04-14T15:18:17.898560+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'7","last_scrub_stamp":"2024-04-14T15:15:44.927776+0000","last_deep_scrub":"87'7","last_deep_scrub_stamp":"2024-04-14T15:11:10.510785+0000","last_clean_scrub_stamp":"2024-04-14T15:15:44.927776+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:22.487746+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":230,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.900046+0000","last_change":"2024-04-14T15:16:46.686879+0000","last_active":"2024-04-14T15:18:17.900046+0000","last_peered":"2024-04-14T15:18:17.900046+0000","last_clean":"2024-04-14T15:18:17.900046+0000","last_became_active":"2024-04-14T15:16:24.910246+0000","last_became_peered":"2024-04-14T15:16:24.910246+0000","last_unstale":"2024-04-14T15:18:17.900046+0000","last_undegraded":"2024-04-14T15:18:17.900046+0000","last_fullsized":"2024-04-14T15:18:17.900046+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:46.686789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:08.254601+0000","last_clean_scrub_stamp":"2024-04-14T15:16:46.686789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:20.852071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":746,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:18.542907+0000","last_change":"2024-04-14T15:16:17.804733+0000","last_active":"2024-04-14T15:18:18.542907+0000","last_peered":"2024-04-14T15:18:18.542907+0000","last_clean":"2024-04-14T15:18:18.542907+0000","last_became_active":"2024-04-14T15:09:47.046517+0000","last_became_peered":"2024-04-14T15:09:47.046517+0000","last_unstale":"2024-04-14T15:18:18.542907+0000","last_undegraded":"2024-04-14T15:18:18.542907+0000","last_fullsized":"2024-04-14T15:18:18.542907+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:17.804649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:17.804649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.661087+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":308,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.901075+0000","last_change":"2024-04-14T15:15:44.887620+0000","last_active":"2024-04-14T15:18:17.901075+0000","last_peered":"2024-04-14T15:18:17.901075+0000","last_clean":"2024-04-14T15:18:17.901075+0000","last_became_active":"2024-04-14T15:15:44.887227+0000","last_became_peered":"2024-04-14T15:15:44.887227+0000","last_unstale":"2024-04-14T15:18:17.901075+0000","last_undegraded":"2024-04-14T15:18:17.901075+0000","last_fullsized":"2024-04-14T15:18:17.901075+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:56.268086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:56.268086+0000","last_clean_scrub_stamp":"2024-04-14T15:14:56.268086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:20.951062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":651,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:18.543406+0000","last_change":"2024-04-14T15:15:37.629813+0000","last_active":"2024-04-14T15:18:18.543406+0000","last_peered":"2024-04-14T15:18:18.543406+0000","last_clean":"2024-04-14T15:18:18.543406+0000","last_became_active":"2024-04-14T15:12:56.189117+0000","last_became_peered":"2024-04-14T15:12:56.189117+0000","last_unstale":"2024-04-14T15:18:18.543406+0000","last_undegraded":"2024-04-14T15:18:18.543406+0000","last_fullsized":"2024-04-14T15:18:18.543406+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:37.629718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:37.629718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.660989+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.18","version":"72'9","reported_seq":766,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.898021+0000","last_change":"2024-04-14T15:16:09.965230+0000","last_active":"2024-04-14T15:18:17.898021+0000","last_peered":"2024-04-14T15:18:17.898021+0000","last_clean":"2024-04-14T15:18:17.898021+0000","last_became_active":"2024-04-14T15:09:45.032079+0000","last_became_peered":"2024-04-14T15:09:45.032079+0000","last_unstale":"2024-04-14T15:18:17.898021+0000","last_undegraded":"2024-04-14T15:18:17.898021+0000","last_fullsized":"2024-04-14T15:18:17.898021+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-04-14T15:16:09.965113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:16:09.965113+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:22.487763+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":438,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.908679+0000","last_change":"2024-04-14T15:17:00.948362+0000","last_active":"2024-04-14T15:18:17.908679+0000","last_peered":"2024-04-14T15:18:17.908679+0000","last_clean":"2024-04-14T15:18:17.908679+0000","last_became_active":"2024-04-14T15:15:21.891926+0000","last_became_peered":"2024-04-14T15:15:21.891926+0000","last_unstale":"2024-04-14T15:18:17.908679+0000","last_undegraded":"2024-04-14T15:18:17.908679+0000","last_fullsized":"2024-04-14T15:18:17.908679+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:00.948283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:00.948283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":748,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.900925+0000","last_change":"2024-04-14T15:16:45.114648+0000","last_active":"2024-04-14T15:18:17.900925+0000","last_peered":"2024-04-14T15:18:17.900925+0000","last_clean":"2024-04-14T15:18:17.900925+0000","last_became_active":"2024-04-14T15:09:47.041486+0000","last_became_peered":"2024-04-14T15:09:47.041486+0000","last_unstale":"2024-04-14T15:18:17.900925+0000","last_undegraded":"2024-04-14T15:18:17.900925+0000","last_fullsized":"2024-04-14T15:18:17.900925+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:45.114534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:45.114534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.009802+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":742,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.898320+0000","last_change":"2024-04-14T15:16:55.184211+0000","last_active":"2024-04-14T15:18:17.898320+0000","last_peered":"2024-04-14T15:18:17.898320+0000","last_clean":"2024-04-14T15:18:17.898320+0000","last_became_active":"2024-04-14T15:09:49.058904+0000","last_became_peered":"2024-04-14T15:09:49.058904+0000","last_unstale":"2024-04-14T15:18:17.898320+0000","last_undegraded":"2024-04-14T15:18:17.898320+0000","last_fullsized":"2024-04-14T15:18:17.898320+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:55.184094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:55.184094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:20.454966+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":313,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:18.543878+0000","last_change":"2024-04-14T15:17:52.593706+0000","last_active":"2024-04-14T15:18:18.543878+0000","last_peered":"2024-04-14T15:18:18.543878+0000","last_clean":"2024-04-14T15:18:18.543878+0000","last_became_active":"2024-04-14T15:15:44.889605+0000","last_became_peered":"2024-04-14T15:15:44.889605+0000","last_unstale":"2024-04-14T15:18:18.543878+0000","last_undegraded":"2024-04-14T15:18:18.543878+0000","last_fullsized":"2024-04-14T15:18:18.543878+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:52.593618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:17:52.593618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:09.797845+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"306'22","reported_seq":465,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.907904+0000","last_change":"2024-04-14T15:16:13.898263+0000","last_active":"2024-04-14T15:18:17.907904+0000","last_peered":"2024-04-14T15:18:17.907904+0000","last_clean":"2024-04-14T15:18:17.907904+0000","last_became_active":"2024-04-14T15:15:21.884310+0000","last_became_peered":"2024-04-14T15:15:21.884310+0000","last_unstale":"2024-04-14T15:18:17.907904+0000","last_undegraded":"2024-04-14T15:18:17.907904+0000","last_fullsized":"2024-04-14T15:18:17.907904+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'22","last_scrub_stamp":"2024-04-14T15:16:13.898106+0000","last_deep_scrub":"306'22","last_deep_scrub_stamp":"2024-04-14T15:16:13.898106+0000","last_clean_scrub_stamp":"2024-04-14T15:16:13.898106+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:18.359119+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":757,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.906348+0000","last_change":"2024-04-14T15:16:14.427502+0000","last_active":"2024-04-14T15:18:17.906348+0000","last_peered":"2024-04-14T15:18:17.906348+0000","last_clean":"2024-04-14T15:18:17.906348+0000","last_became_active":"2024-04-14T15:09:43.017351+0000","last_became_peered":"2024-04-14T15:09:43.017351+0000","last_unstale":"2024-04-14T15:18:17.906348+0000","last_undegraded":"2024-04-14T15:18:17.906348+0000","last_fullsized":"2024-04-14T15:18:17.906348+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:14.427425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:14.427425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:21.922576+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":746,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.898232+0000","last_change":"2024-04-14T15:16:19.041461+0000","last_active":"2024-04-14T15:18:17.898232+0000","last_peered":"2024-04-14T15:18:17.898232+0000","last_clean":"2024-04-14T15:18:17.898232+0000","last_became_active":"2024-04-14T15:09:47.042538+0000","last_became_peered":"2024-04-14T15:09:47.042538+0000","last_unstale":"2024-04-14T15:18:17.898232+0000","last_undegraded":"2024-04-14T15:18:17.898232+0000","last_fullsized":"2024-04-14T15:18:17.898232+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:19.041346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:08.512130+0000","last_clean_scrub_stamp":"2024-04-14T15:16:19.041346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:22.487909+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":748,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-14T15:18:17.907160+0000","last_change":"2024-04-14T15:17:56.237366+0000","last_active":"2024-04-14T15:18:17.907160+0000","last_peered":"2024-04-14T15:18:17.907160+0000","last_clean":"2024-04-14T15:18:17.907160+0000","last_became_active":"2024-04-14T15:09:49.129636+0000","last_became_peered":"2024-04-14T15:09:49.129636+0000","last_unstale":"2024-04-14T15:18:17.907160+0000","last_undegraded":"2024-04-14T15:18:17.907160+0000","last_fullsized":"2024-04-14T15:18:17.907160+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:56.237273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:17:56.237273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:03.893408+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":297,"num_pg":32,"stat_sum":{"num_bytes":14,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":10,"num_read_kb":7,"num_write":10,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":42,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8},{"poolid":110,"num_pg":32,"stat_sum":{"num_bytes":8195793,"num_objects":161,"num_object_clones":4,"num_object_copies":483,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":161,"num_whiteouts":0,"num_read":108,"num_read_kb":9,"num_write":599,"num_write_kb":12688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":9,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":26505216,"data_stored":24587397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1881,"ondisk_log_size":1881,"up":96,"acting":96,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1430,"ondisk_log_size":1430,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":509,"num_read_kb":504,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":695,"num_read_kb":452,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":481,"ondisk_log_size":481,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71,"ondisk_log_size":71,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103277,"num_pgs":99,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":73068,"kb_used_data":15488,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93672084,"statfs":{"total":95995035648,"available":95920214016,"internally_reserved":0,"allocated":15859712,"data_stored":12618708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":58915274},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397616,"num_pgs":84,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":70316,"kb_used_data":7192,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93674836,"statfs":{"total":95995035648,"available":95923032064,"internally_reserved":0,"allocated":7364608,"data_stored":4105833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574986,"num_pgs":92,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":87008,"kb_used_data":11380,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93658144,"statfs":{"total":95995035648,"available":95905939456,"internally_reserved":0,"allocated":11653120,"data_stored":8300422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738511,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":82604,"kb_used_data":11148,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93662548,"statfs":{"total":95995035648,"available":95910449152,"internally_reserved":0,"allocated":11415552,"data_stored":8106273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411462,"num_pgs":87,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":82880,"kb_used_data":11436,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93662272,"statfs":{"total":95995035648,"available":95910166528,"internally_reserved":0,"allocated":11710464,"data_stored":8367976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182528,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":57004,"kb_used_data":7756,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93688148,"statfs":{"total":95995035648,"available":95936663552,"internally_reserved":0,"allocated":7942144,"data_stored":4696533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986298,"num_pgs":87,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":70288,"kb_used_data":7164,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93674864,"statfs":{"total":95995035648,"available":95923060736,"internally_reserved":0,"allocated":7335936,"data_stored":4105353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659255,"num_pgs":88,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":74948,"kb_used_data":11824,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93670204,"statfs":{"total":95995035648,"available":95918288896,"internally_reserved":0,"allocated":12107776,"data_stored":8880708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4276224,"data_stored":4000616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4493312,"data_stored":4195009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4329472,"data_stored":4000772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":176128,"data_stored":414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":184320,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4403200,"data_stored":4194789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":225280,"data_stored":524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8417280,"data_stored":8194829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":297,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":297,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":297,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":2,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":297,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":12,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":297,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":14,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":297,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":2,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":297,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":12,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":297,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-14T15:18:25.177 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-04-14T15:18:25.178 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-14T15:18:25.178 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-04-14T15:18:25.178 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph pg dump --format=json 2024-04-14T15:18:25.216 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:25.226 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:25 smithi092 ceph-mon[33966]: 6.10 scrub starts 2024-04-14T15:18:25.227 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:25 smithi092 ceph-mon[33966]: 6.10 scrub ok 2024-04-14T15:18:25.227 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:25 smithi092 ceph-mon[33966]: from='client.49120 v1:172.21.15.90:0/22476038' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:18:25.227 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:18:25 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:18:25.240 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:25.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:25.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:25.252 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:18:25.342 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:18:25.365 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:25.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:25.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:18:25.377 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:25 smithi090 ceph-mon[41670]: 6.10 scrub starts 2024-04-14T15:18:25.377 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:25 smithi090 ceph-mon[41670]: 6.10 scrub ok 2024-04-14T15:18:25.377 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:25 smithi090 ceph-mon[41670]: from='client.49120 v1:172.21.15.90:0/22476038' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:18:25.377 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:25 smithi090 ceph-mon[30399]: 6.10 scrub starts 2024-04-14T15:18:25.377 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:25 smithi090 ceph-mon[30399]: 6.10 scrub ok 2024-04-14T15:18:25.377 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:25 smithi090 ceph-mon[30399]: from='client.49120 v1:172.21.15.90:0/22476038' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:18:25.467 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:18:25.491 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:25.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:25.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:18:25.592 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:18:25.616 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:25.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:25.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:18:25.718 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:18:25.742 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:25.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:25.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:18:25.844 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:18:25.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:25.867 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:25.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:18:25.968 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:18:25.994 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:25.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:25.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:18:26.096 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:18:26.119 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:26.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:26.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:18:26.194 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:18:26.195 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:26 smithi092 ceph-mon[33966]: 2.9 scrub starts 2024-04-14T15:18:26.195 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:26 smithi092 ceph-mon[33966]: 2.9 scrub ok 2024-04-14T15:18:26.195 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:26 smithi092 ceph-mon[33966]: pgmap v590: 238 pgs: 238 active+clean; 8.4 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 767 B/s wr, 3 op/s 2024-04-14T15:18:26.195 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:26 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:18:26.195 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:26 smithi092 ceph-mon[33966]: osdmap e426: 8 total, 8 up, 8 in 2024-04-14T15:18:26.221 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:26.245 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:26.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:26.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:26.315 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:26 smithi090 ceph-mon[41670]: 2.9 scrub starts 2024-04-14T15:18:26.315 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:26 smithi090 ceph-mon[41670]: 2.9 scrub ok 2024-04-14T15:18:26.315 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:26 smithi090 ceph-mon[41670]: pgmap v590: 238 pgs: 238 active+clean; 8.4 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 767 B/s wr, 3 op/s 2024-04-14T15:18:26.316 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:26 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:18:26.316 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:26 smithi090 ceph-mon[41670]: osdmap e426: 8 total, 8 up, 8 in 2024-04-14T15:18:26.316 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:26 smithi090 ceph-mon[30399]: 2.9 scrub starts 2024-04-14T15:18:26.316 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:26 smithi090 ceph-mon[30399]: 2.9 scrub ok 2024-04-14T15:18:26.316 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:26 smithi090 ceph-mon[30399]: pgmap v590: 238 pgs: 238 active+clean; 8.4 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 767 B/s wr, 3 op/s 2024-04-14T15:18:26.316 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:26 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:18:26.316 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:26 smithi090 ceph-mon[30399]: osdmap e426: 8 total, 8 up, 8 in 2024-04-14T15:18:26.346 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:26.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:26.371 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:26.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:26.472 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:18:26.495 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:26.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:26.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:18:26.597 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:18:26.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:26.660 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:26.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:18:26.752 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-04-14T15:18:26.753 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-04-14T15:18:26.753 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-04-14T15:18:26.753 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-04-14T15:18:26.753 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.753 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25063.0:2 39.0 39:0384ab8e:::oid:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-04-14T15:18:26.753 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:55.066308+0000", 2024-04-14T15:18:26.753 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 331.68412368600002, 2024-04-14T15:18:26.753 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.468718499, 2024-04-14T15:18:26.753 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:18:26.753 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:18:26.753 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:18:26.754 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:18:26.754 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25063", 2024-04-14T15:18:26.754 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/3585118987", 2024-04-14T15:18:26.754 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 2 2024-04-14T15:18:26.754 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.754 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:18:26.754 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.754 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:18:26.754 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.066308+0000", 2024-04-14T15:18:26.754 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:18:26.754 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.754 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.755 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:18:26.755 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.066308+0000", 2024-04-14T15:18:26.755 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:18:26.755 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.755 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.755 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:18:26.755 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.066310+0000", 2024-04-14T15:18:26.755 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.787e-06 2024-04-14T15:18:26.755 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.755 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.755 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:18:26.755 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.066316+0000", 2024-04-14T15:18:26.755 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5149999999999997e-06 2024-04-14T15:18:26.755 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.756 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.756 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:18:26.756 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.066326+0000", 2024-04-14T15:18:26.756 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0193999999999999e-05 2024-04-14T15:18:26.756 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.756 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.756 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:18:26.756 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.066336+0000", 2024-04-14T15:18:26.756 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0071000000000001e-05 2024-04-14T15:18:26.756 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.756 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.756 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:18:26.756 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.096000+0000", 2024-04-14T15:18:26.756 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.029663666000000002 2024-04-14T15:18:26.756 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.757 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.757 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:18:26.757 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.096009+0000", 2024-04-14T15:18:26.757 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.2199999999999998e-06 2024-04-14T15:18:26.757 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.757 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.757 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:18:26.757 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.391307+0000", 2024-04-14T15:18:26.757 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2952977379999999 2024-04-14T15:18:26.757 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.757 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.757 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:18:26.757 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.391405+0000", 2024-04-14T15:18:26.757 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8735999999999998e-05 2024-04-14T15:18:26.758 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.758 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.758 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,4", 2024-04-14T15:18:26.758 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.391512+0000", 2024-04-14T15:18:26.758 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000106436 2024-04-14T15:18:26.758 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.758 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.758 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:18:26.758 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.397856+0000", 2024-04-14T15:18:26.758 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0063441529999999999 2024-04-14T15:18:26.758 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.758 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.758 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:18:26.758 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.411535+0000", 2024-04-14T15:18:26.759 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.013678968 2024-04-14T15:18:26.759 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.759 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.759 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:18:26.759 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.534946+0000", 2024-04-14T15:18:26.759 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.123411443 2024-04-14T15:18:26.759 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.759 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.759 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:18:26.759 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.534986+0000", 2024-04-14T15:18:26.759 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9972999999999998e-05 2024-04-14T15:18:26.759 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.759 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.759 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:18:26.759 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.535027+0000", 2024-04-14T15:18:26.760 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0599000000000002e-05 2024-04-14T15:18:26.760 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:18:26.760 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:18:26.760 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:18:26.760 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.760 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.760 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.24938.0:2 27.1 27:847a3d55:::obj:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-04-14T15:18:26.760 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:55.066927+0000", 2024-04-14T15:18:26.760 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 331.68350485100001, 2024-04-14T15:18:26.760 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4681723280000001, 2024-04-14T15:18:26.760 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:18:26.760 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:18:26.760 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:18:26.760 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:18:26.760 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.24938", 2024-04-14T15:18:26.761 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/3748273494", 2024-04-14T15:18:26.761 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 2 2024-04-14T15:18:26.761 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.761 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:18:26.761 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.761 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:18:26.761 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.066927+0000", 2024-04-14T15:18:26.761 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:18:26.761 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.761 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.761 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:18:26.761 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.066927+0000", 2024-04-14T15:18:26.761 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:18:26.761 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.761 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.762 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:18:26.762 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.066929+0000", 2024-04-14T15:18:26.762 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.528e-06 2024-04-14T15:18:26.762 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.762 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.762 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:18:26.762 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.066959+0000", 2024-04-14T15:18:26.762 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0093000000000001e-05 2024-04-14T15:18:26.762 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.762 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.762 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:18:26.762 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.066979+0000", 2024-04-14T15:18:26.762 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0183e-05 2024-04-14T15:18:26.762 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.763 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.763 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:18:26.763 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.066994+0000", 2024-04-14T15:18:26.763 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5218000000000001e-05 2024-04-14T15:18:26.763 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.763 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.763 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:18:26.763 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.110325+0000", 2024-04-14T15:18:26.763 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.043331135 2024-04-14T15:18:26.763 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.763 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.763 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:18:26.763 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.110335+0000", 2024-04-14T15:18:26.763 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.5540000000000003e-06 2024-04-14T15:18:26.764 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.764 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.764 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:18:26.764 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.388426+0000", 2024-04-14T15:18:26.764 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2780914050000001 2024-04-14T15:18:26.764 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.764 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.764 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:18:26.764 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.388545+0000", 2024-04-14T15:18:26.764 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011851800000000001 2024-04-14T15:18:26.764 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.764 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.764 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,6", 2024-04-14T15:18:26.764 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.388667+0000", 2024-04-14T15:18:26.764 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000121766 2024-04-14T15:18:26.765 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.765 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.765 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:18:26.765 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.391689+0000", 2024-04-14T15:18:26.765 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0030224330000000002 2024-04-14T15:18:26.765 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.765 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.765 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:18:26.765 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.432617+0000", 2024-04-14T15:18:26.765 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.040928216000000003 2024-04-14T15:18:26.765 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.765 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.765 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:18:26.765 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.535021+0000", 2024-04-14T15:18:26.766 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.102403801 2024-04-14T15:18:26.766 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.766 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.766 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:18:26.766 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.535062+0000", 2024-04-14T15:18:26.766 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0646999999999999e-05 2024-04-14T15:18:26.766 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.766 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.766 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:18:26.766 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.535100+0000", 2024-04-14T15:18:26.766 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.7830999999999997e-05 2024-04-14T15:18:26.766 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:18:26.766 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:18:26.766 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:18:26.767 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.767 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.767 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.24991.0:3 29.14 29:2a647cd4:::obj0:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e108)", 2024-04-14T15:18:26.767 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:56.074507+0000", 2024-04-14T15:18:26.767 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 330.675924995, 2024-04-14T15:18:26.767 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.17118332999999999, 2024-04-14T15:18:26.767 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:18:26.767 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:18:26.767 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-04-14T15:18:26.767 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:18:26.767 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.24991", 2024-04-14T15:18:26.768 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/3757631150", 2024-04-14T15:18:26.768 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 3 2024-04-14T15:18:26.768 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.768 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:18:26.768 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.768 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:18:26.768 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.074507+0000", 2024-04-14T15:18:26.768 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:18:26.768 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.768 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.768 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:18:26.768 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.074507+0000", 2024-04-14T15:18:26.768 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:18:26.768 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.769 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.769 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:18:26.769 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.074509+0000", 2024-04-14T15:18:26.769 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.984e-06 2024-04-14T15:18:26.769 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.769 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.769 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:18:26.769 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.074516+0000", 2024-04-14T15:18:26.769 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8580000000000002e-06 2024-04-14T15:18:26.769 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.770 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.770 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:18:26.770 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.074530+0000", 2024-04-14T15:18:26.770 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3966e-05 2024-04-14T15:18:26.770 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.770 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.770 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:18:26.770 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.074539+0000", 2024-04-14T15:18:26.770 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.5079999999999994e-06 2024-04-14T15:18:26.770 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.770 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.770 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:18:26.770 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.074616+0000", 2024-04-14T15:18:26.770 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.6228000000000003e-05 2024-04-14T15:18:26.771 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.771 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.771 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op must wait for map", 2024-04-14T15:18:26.771 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.074621+0000", 2024-04-14T15:18:26.771 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6219999999999997e-06 2024-04-14T15:18:26.771 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.771 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.771 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:18:26.771 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.215446+0000", 2024-04-14T15:18:26.771 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.140824962 2024-04-14T15:18:26.771 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.771 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.771 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:18:26.773 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.215454+0000", 2024-04-14T15:18:26.773 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1009999999999992e-06 2024-04-14T15:18:26.773 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.773 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.773 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:18:26.773 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.245596+0000", 2024-04-14T15:18:26.773 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.030142163999999999 2024-04-14T15:18:26.773 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.774 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.774 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:18:26.774 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.245690+0000", 2024-04-14T15:18:26.774 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.3937000000000005e-05 2024-04-14T15:18:26.774 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:18:26.774 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:18:26.774 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:18:26.774 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.774 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.774 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.24955.0:2 26.1e 26:7a7d4169:::src_object.1:head [writefull 0~4096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-04-14T15:18:26.774 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:56.075137+0000", 2024-04-14T15:18:26.774 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 330.67529499900002, 2024-04-14T15:18:26.774 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.18840111900000001, 2024-04-14T15:18:26.774 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:18:26.775 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:18:26.775 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:18:26.775 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:18:26.775 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.24955", 2024-04-14T15:18:26.775 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/117387996", 2024-04-14T15:18:26.775 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 2 2024-04-14T15:18:26.775 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.775 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:18:26.775 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.775 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:18:26.775 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.075137+0000", 2024-04-14T15:18:26.775 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:18:26.775 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.775 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.775 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:18:26.776 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.075137+0000", 2024-04-14T15:18:26.776 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:18:26.776 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.776 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.776 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:18:26.776 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.075138+0000", 2024-04-14T15:18:26.776 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.398e-06 2024-04-14T15:18:26.776 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.776 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.776 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:18:26.776 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.075149+0000", 2024-04-14T15:18:26.776 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.022e-05 2024-04-14T15:18:26.776 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.776 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.776 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:18:26.777 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.075161+0000", 2024-04-14T15:18:26.777 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2126999999999999e-05 2024-04-14T15:18:26.777 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.777 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.777 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:18:26.777 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.075169+0000", 2024-04-14T15:18:26.777 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3799999999999994e-06 2024-04-14T15:18:26.777 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.777 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.777 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:18:26.777 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.075243+0000", 2024-04-14T15:18:26.777 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3870999999999998e-05 2024-04-14T15:18:26.777 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.777 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.778 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op must wait for map", 2024-04-14T15:18:26.778 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.075249+0000", 2024-04-14T15:18:26.778 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9449999999999998e-06 2024-04-14T15:18:26.778 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.778 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.778 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:18:26.778 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.184989+0000", 2024-04-14T15:18:26.778 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.109740052 2024-04-14T15:18:26.778 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.778 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.778 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:18:26.778 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.184999+0000", 2024-04-14T15:18:26.778 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.5449999999999992e-06 2024-04-14T15:18:26.778 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.779 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.779 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:18:26.779 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.238236+0000", 2024-04-14T15:18:26.779 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.053237643000000001 2024-04-14T15:18:26.779 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.779 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.779 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:18:26.779 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.238399+0000", 2024-04-14T15:18:26.779 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016226899999999999 2024-04-14T15:18:26.779 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.779 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.779 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4,7", 2024-04-14T15:18:26.779 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.238556+0000", 2024-04-14T15:18:26.779 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015753499999999999 2024-04-14T15:18:26.779 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.780 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.780 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:18:26.780 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.241212+0000", 2024-04-14T15:18:26.780 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0026559270000000002 2024-04-14T15:18:26.780 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.780 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.780 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:18:26.780 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.252026+0000", 2024-04-14T15:18:26.780 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.010814414 2024-04-14T15:18:26.780 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.780 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.781 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:18:26.781 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.263440+0000", 2024-04-14T15:18:26.781 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.011413637000000001 2024-04-14T15:18:26.781 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.781 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.781 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:18:26.782 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.263493+0000", 2024-04-14T15:18:26.782 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3086e-05 2024-04-14T15:18:26.782 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.782 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.782 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:18:26.783 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.263538+0000", 2024-04-14T15:18:26.783 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.507e-05 2024-04-14T15:18:26.783 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:18:26.783 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:18:26.783 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:18:26.783 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.784 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.784 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.24811.0:3 10.9 10:9294e480:nspace::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-04-14T15:18:26.784 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:56.449492+0000", 2024-04-14T15:18:26.784 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 330.300940293, 2024-04-14T15:18:26.784 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:18:26.785 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:18:26.786 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.095302850999999994, 2024-04-14T15:18:26.786 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:18:26.786 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:18:26.786 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:18:26.786 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:18:26.786 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.24811", 2024-04-14T15:18:26.786 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/473680876", 2024-04-14T15:18:26.787 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 3 2024-04-14T15:18:26.787 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.787 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:18:26.787 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.787 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:18:26.787 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.449492+0000", 2024-04-14T15:18:26.787 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:18:26.787 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.787 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.787 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:18:26.787 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.449492+0000", 2024-04-14T15:18:26.787 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:18:26.787 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.787 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.788 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:18:26.788 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.449495+0000", 2024-04-14T15:18:26.788 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2669999999999998e-06 2024-04-14T15:18:26.788 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.788 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.788 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:18:26.788 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.449507+0000", 2024-04-14T15:18:26.788 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1997000000000001e-05 2024-04-14T15:18:26.788 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.788 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.788 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:18:26.788 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.449533+0000", 2024-04-14T15:18:26.788 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6392000000000001e-05 2024-04-14T15:18:26.788 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.789 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.789 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:18:26.789 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.449552+0000", 2024-04-14T15:18:26.789 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8938999999999999e-05 2024-04-14T15:18:26.789 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.789 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.789 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:18:26.789 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.449687+0000", 2024-04-14T15:18:26.789 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000134807 2024-04-14T15:18:26.789 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.789 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.789 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:18:26.790 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.449874+0000", 2024-04-14T15:18:26.790 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000187164 2024-04-14T15:18:26.790 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.790 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.790 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 2,7", 2024-04-14T15:18:26.790 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.449967+0000", 2024-04-14T15:18:26.790 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.3089000000000003e-05 2024-04-14T15:18:26.790 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.790 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.790 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:18:26.790 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.450284+0000", 2024-04-14T15:18:26.791 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00031640299999999999 2024-04-14T15:18:26.791 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.791 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.791 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:18:26.791 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.450580+0000", 2024-04-14T15:18:26.791 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000296011 2024-04-14T15:18:26.791 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.791 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.791 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:18:26.792 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.544700+0000", 2024-04-14T15:18:26.792 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.094120081999999994 2024-04-14T15:18:26.792 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.792 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.792 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:18:26.792 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.544752+0000", 2024-04-14T15:18:26.792 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.1919999999999998e-05 2024-04-14T15:18:26.792 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.792 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.792 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:18:26.792 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.544795+0000", 2024-04-14T15:18:26.792 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.278e-05 2024-04-14T15:18:26.792 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:18:26.792 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:18:26.793 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:18:26.793 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.793 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.793 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.24811.0:17 10.6 10:62743f70:test-rados-api-smithi090-93398-5::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-04-14T15:18:26.793 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:56.552466+0000", 2024-04-14T15:18:26.793 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 330.19796565399997, 2024-04-14T15:18:26.793 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.036935908000000003, 2024-04-14T15:18:26.793 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:18:26.793 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:18:26.793 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:18:26.793 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:18:26.793 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.24811", 2024-04-14T15:18:26.793 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/473680876", 2024-04-14T15:18:26.793 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 17 2024-04-14T15:18:26.794 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.794 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:18:26.794 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.794 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:18:26.794 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.552466+0000", 2024-04-14T15:18:26.794 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:18:26.794 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.794 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.794 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:18:26.794 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.552466+0000", 2024-04-14T15:18:26.794 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:18:26.794 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.794 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.794 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:18:26.795 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.552468+0000", 2024-04-14T15:18:26.795 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.384e-06 2024-04-14T15:18:26.795 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.795 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.795 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:18:26.795 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.552475+0000", 2024-04-14T15:18:26.795 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.6229999999999997e-06 2024-04-14T15:18:26.795 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.795 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.795 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:18:26.795 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.552482+0000", 2024-04-14T15:18:26.795 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9609999999999999e-06 2024-04-14T15:18:26.795 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.796 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.796 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:18:26.796 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.552489+0000", 2024-04-14T15:18:26.796 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1219999999999997e-06 2024-04-14T15:18:26.796 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.796 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.796 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:18:26.796 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.552544+0000", 2024-04-14T15:18:26.796 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4731999999999997e-05 2024-04-14T15:18:26.796 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.796 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.796 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:18:26.796 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.552657+0000", 2024-04-14T15:18:26.797 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011296199999999999 2024-04-14T15:18:26.797 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.797 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.797 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,4", 2024-04-14T15:18:26.797 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.552783+0000", 2024-04-14T15:18:26.797 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012542200000000001 2024-04-14T15:18:26.797 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.797 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.797 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:18:26.797 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.553193+0000", 2024-04-14T15:18:26.797 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000410734 2024-04-14T15:18:26.797 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.797 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.797 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:18:26.797 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.553471+0000", 2024-04-14T15:18:26.798 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00027812099999999999 2024-04-14T15:18:26.798 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.798 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.798 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:18:26.798 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.589310+0000", 2024-04-14T15:18:26.798 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.035838137999999999 2024-04-14T15:18:26.798 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.798 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.798 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:18:26.798 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.589371+0000", 2024-04-14T15:18:26.798 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1273999999999996e-05 2024-04-14T15:18:26.798 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.798 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.798 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:18:26.799 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.589402+0000", 2024-04-14T15:18:26.799 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1435e-05 2024-04-14T15:18:26.799 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:18:26.799 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:18:26.799 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:18:26.799 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.799 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.799 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.24853.0:433 16.1b 16:dbd443e1:test-rados-api-smithi090-93601-15::foo:head [setxattr a192 in=4b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-04-14T15:18:26.799 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:57.070215+0000", 2024-04-14T15:18:26.799 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 329.680217072, 2024-04-14T15:18:26.799 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.17665454899999999, 2024-04-14T15:18:26.799 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:18:26.799 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:18:26.799 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:18:26.799 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:18:26.800 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.24853", 2024-04-14T15:18:26.800 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/3851923713", 2024-04-14T15:18:26.800 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 433 2024-04-14T15:18:26.800 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.800 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:18:26.800 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.800 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:18:26.800 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.070215+0000", 2024-04-14T15:18:26.800 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:18:26.800 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.800 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.800 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:18:26.800 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.070215+0000", 2024-04-14T15:18:26.800 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:18:26.800 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.801 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.801 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:18:26.801 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.070217+0000", 2024-04-14T15:18:26.801 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.801e-06 2024-04-14T15:18:26.801 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.801 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.801 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:18:26.801 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.070224+0000", 2024-04-14T15:18:26.801 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8210000000000004e-06 2024-04-14T15:18:26.801 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.802 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.802 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:18:26.802 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.070249+0000", 2024-04-14T15:18:26.802 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.4995999999999999e-05 2024-04-14T15:18:26.802 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.802 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.802 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:18:26.803 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.070260+0000", 2024-04-14T15:18:26.803 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1574e-05 2024-04-14T15:18:26.803 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.803 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.803 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:18:26.803 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.079670+0000", 2024-04-14T15:18:26.804 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0094096590000000008 2024-04-14T15:18:26.804 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.804 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.804 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:18:26.804 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.079767+0000", 2024-04-14T15:18:26.804 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.7183999999999997e-05 2024-04-14T15:18:26.804 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.805 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.805 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1,6", 2024-04-14T15:18:26.805 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.079914+0000", 2024-04-14T15:18:26.805 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014677599999999999 2024-04-14T15:18:26.805 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.805 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.805 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:18:26.805 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.083998+0000", 2024-04-14T15:18:26.805 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0040846600000000004 2024-04-14T15:18:26.805 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.805 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.805 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:18:26.806 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.157521+0000", 2024-04-14T15:18:26.806 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.073522865000000007 2024-04-14T15:18:26.806 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.806 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.806 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:18:26.806 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.246599+0000", 2024-04-14T15:18:26.806 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.089077323 2024-04-14T15:18:26.806 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.806 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.806 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:18:26.806 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.246815+0000", 2024-04-14T15:18:26.807 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000216527 2024-04-14T15:18:26.807 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.807 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.807 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:18:26.807 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.246870+0000", 2024-04-14T15:18:26.807 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4363000000000001e-05 2024-04-14T15:18:26.807 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:18:26.807 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:18:26.807 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:18:26.807 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.807 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.807 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25001.0:1 44.6s0 44:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-04-14T15:18:26.807 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:57.082702+0000", 2024-04-14T15:18:26.807 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 329.66773045799999, 2024-04-14T15:18:26.808 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.118265955, 2024-04-14T15:18:26.808 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:18:26.808 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:18:26.808 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:18:26.808 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:18:26.808 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25001", 2024-04-14T15:18:26.808 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/1465058495", 2024-04-14T15:18:26.808 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 1 2024-04-14T15:18:26.808 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.808 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:18:26.808 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.808 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:18:26.808 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.082702+0000", 2024-04-14T15:18:26.808 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:18:26.809 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.809 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.809 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:18:26.809 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.082702+0000", 2024-04-14T15:18:26.809 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:18:26.809 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.809 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.809 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:18:26.809 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.082703+0000", 2024-04-14T15:18:26.809 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8789999999999999e-06 2024-04-14T15:18:26.809 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.809 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.810 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:18:26.810 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.082709+0000", 2024-04-14T15:18:26.810 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7139999999999998e-06 2024-04-14T15:18:26.810 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.810 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.810 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:18:26.810 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.082718+0000", 2024-04-14T15:18:26.810 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5490000000000006e-06 2024-04-14T15:18:26.810 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.810 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.810 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:18:26.810 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.082727+0000", 2024-04-14T15:18:26.810 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.2709999999999998e-06 2024-04-14T15:18:26.810 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.811 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.811 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:18:26.811 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.083560+0000", 2024-04-14T15:18:26.811 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00083254599999999998 2024-04-14T15:18:26.811 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.811 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.811 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:18:26.811 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.083562+0000", 2024-04-14T15:18:26.811 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6089999999999999e-06 2024-04-14T15:18:26.811 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.811 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.811 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:18:26.811 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.198059+0000", 2024-04-14T15:18:26.811 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.114496822 2024-04-14T15:18:26.811 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.812 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.812 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:18:26.812 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.198174+0000", 2024-04-14T15:18:26.812 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000115389 2024-04-14T15:18:26.812 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.812 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.812 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_started", 2024-04-14T15:18:26.812 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.198510+0000", 2024-04-14T15:18:26.812 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00033594199999999999 2024-04-14T15:18:26.812 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.812 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.812 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_committed", 2024-04-14T15:18:26.812 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.198835+0000", 2024-04-14T15:18:26.812 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000324743 2024-04-14T15:18:26.812 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.812 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.812 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:18:26.813 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.200916+0000", 2024-04-14T15:18:26.813 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0020809399999999999 2024-04-14T15:18:26.813 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.813 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.813 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:18:26.813 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.200968+0000", 2024-04-14T15:18:26.813 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.1551000000000002e-05 2024-04-14T15:18:26.813 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:18:26.813 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:18:26.813 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:18:26.813 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.813 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.813 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.24853.0:435 16.1b 16:dbd443e1:test-rados-api-smithi090-93601-15::foo:head [setxattr a193 in=4b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-04-14T15:18:26.813 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:57.248940+0000", 2024-04-14T15:18:26.813 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 329.50149200200002, 2024-04-14T15:18:26.813 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.094945007999999997, 2024-04-14T15:18:26.814 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:18:26.814 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:18:26.814 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:18:26.814 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:18:26.814 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.24853", 2024-04-14T15:18:26.814 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/3851923713", 2024-04-14T15:18:26.814 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 435 2024-04-14T15:18:26.814 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.814 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:18:26.814 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.814 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:18:26.814 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.248940+0000", 2024-04-14T15:18:26.814 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:18:26.815 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.815 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.815 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:18:26.815 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.248940+0000", 2024-04-14T15:18:26.815 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:18:26.815 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.815 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.815 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:18:26.815 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.248942+0000", 2024-04-14T15:18:26.815 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.646e-06 2024-04-14T15:18:26.815 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.815 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.815 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:18:26.815 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.248949+0000", 2024-04-14T15:18:26.815 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.4889999999999998e-06 2024-04-14T15:18:26.816 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.816 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.816 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:18:26.816 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.248965+0000", 2024-04-14T15:18:26.816 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5620999999999998e-05 2024-04-14T15:18:26.816 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.816 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.816 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:18:26.816 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.248974+0000", 2024-04-14T15:18:26.816 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.1570000000000006e-06 2024-04-14T15:18:26.816 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.816 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.816 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:18:26.817 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.249027+0000", 2024-04-14T15:18:26.817 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3229999999999997e-05 2024-04-14T15:18:26.817 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.817 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.817 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:18:26.817 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.249109+0000", 2024-04-14T15:18:26.817 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1914000000000002e-05 2024-04-14T15:18:26.817 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.817 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.818 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1,6", 2024-04-14T15:18:26.818 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.249252+0000", 2024-04-14T15:18:26.818 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000143356 2024-04-14T15:18:26.818 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.818 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.818 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:18:26.818 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.249903+0000", 2024-04-14T15:18:26.819 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000651007 2024-04-14T15:18:26.819 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.819 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.819 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:18:26.819 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.253012+0000", 2024-04-14T15:18:26.819 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0031087699999999998 2024-04-14T15:18:26.819 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.819 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.820 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:18:26.820 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.343821+0000", 2024-04-14T15:18:26.820 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.090809216999999998 2024-04-14T15:18:26.820 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.820 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.820 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:18:26.820 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.343857+0000", 2024-04-14T15:18:26.821 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5116000000000001e-05 2024-04-14T15:18:26.821 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.821 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.821 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:18:26.821 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.343885+0000", 2024-04-14T15:18:26.821 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8484999999999999e-05 2024-04-14T15:18:26.821 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:18:26.821 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:18:26.822 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:18:26.822 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.822 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.822 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25156.0:2 51.0 51:0384ab8e:::oid:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-04-14T15:18:26.822 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:58.094739+0000", 2024-04-14T15:18:26.823 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 328.65569287099999, 2024-04-14T15:18:26.823 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2282847750000001, 2024-04-14T15:18:26.823 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:18:26.823 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:18:26.823 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:18:26.823 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:18:26.823 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25156", 2024-04-14T15:18:26.824 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/2499391840", 2024-04-14T15:18:26.824 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 2 2024-04-14T15:18:26.824 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.824 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:18:26.824 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.824 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:18:26.825 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.094739+0000", 2024-04-14T15:18:26.825 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:18:26.825 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.825 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.825 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:18:26.825 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.094739+0000", 2024-04-14T15:18:26.825 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:18:26.826 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.826 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.826 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:18:26.826 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.094756+0000", 2024-04-14T15:18:26.826 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7054e-05 2024-04-14T15:18:26.826 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.827 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.827 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:18:26.827 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.094764+0000", 2024-04-14T15:18:26.827 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0260000000000007e-06 2024-04-14T15:18:26.827 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.827 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.827 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:18:26.827 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.094774+0000", 2024-04-14T15:18:26.828 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.9080000000000007e-06 2024-04-14T15:18:26.828 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.828 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.828 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:18:26.828 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.094784+0000", 2024-04-14T15:18:26.828 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0139999999999999e-05 2024-04-14T15:18:26.829 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.829 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.829 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:18:26.829 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.095687+0000", 2024-04-14T15:18:26.829 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00090298600000000004 2024-04-14T15:18:26.829 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.829 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.830 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:18:26.830 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.095695+0000", 2024-04-14T15:18:26.830 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1200000000000002e-06 2024-04-14T15:18:26.830 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.830 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.830 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:18:26.830 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.287077+0000", 2024-04-14T15:18:26.830 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.191381926 2024-04-14T15:18:26.830 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.831 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.831 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:18:26.831 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.287203+0000", 2024-04-14T15:18:26.831 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000125635 2024-04-14T15:18:26.831 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.831 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.831 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,2", 2024-04-14T15:18:26.831 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.287329+0000", 2024-04-14T15:18:26.831 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000126105 2024-04-14T15:18:26.831 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.831 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.831 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:18:26.832 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.287810+0000", 2024-04-14T15:18:26.832 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000481387 2024-04-14T15:18:26.832 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.832 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.832 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:18:26.832 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.292936+0000", 2024-04-14T15:18:26.832 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0051251480000000004 2024-04-14T15:18:26.832 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.832 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.832 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:18:26.832 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.322940+0000", 2024-04-14T15:18:26.832 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.030003906 2024-04-14T15:18:26.832 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.832 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.833 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:18:26.833 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.322984+0000", 2024-04-14T15:18:26.833 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.4869e-05 2024-04-14T15:18:26.833 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.833 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.833 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:18:26.833 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.323024+0000", 2024-04-14T15:18:26.833 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9564999999999998e-05 2024-04-14T15:18:26.833 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:18:26.833 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:18:26.833 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:18:26.833 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.833 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.833 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.24853.0:1223 16.1b 16:dbd443e1:test-rados-api-smithi090-93601-15::foo:head [setxattr a587 in=4b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-04-14T15:18:26.833 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:58.099323+0000", 2024-04-14T15:18:26.833 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 328.65110919300002, 2024-04-14T15:18:26.834 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.097390662000000003, 2024-04-14T15:18:26.834 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:18:26.834 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:18:26.834 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:18:26.834 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:18:26.834 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.24853", 2024-04-14T15:18:26.834 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/3851923713", 2024-04-14T15:18:26.834 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 1223 2024-04-14T15:18:26.834 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.835 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:18:26.835 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.835 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:18:26.835 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.099323+0000", 2024-04-14T15:18:26.835 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:18:26.835 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.835 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.835 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:18:26.835 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.099323+0000", 2024-04-14T15:18:26.835 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:18:26.835 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.835 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.835 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:18:26.835 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.099325+0000", 2024-04-14T15:18:26.836 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.945e-06 2024-04-14T15:18:26.836 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.836 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.836 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:18:26.836 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.099333+0000", 2024-04-14T15:18:26.836 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.8369999999999997e-06 2024-04-14T15:18:26.836 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.836 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.836 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:18:26.836 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.099344+0000", 2024-04-14T15:18:26.836 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1307e-05 2024-04-14T15:18:26.836 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.836 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.836 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:18:26.836 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.099354+0000", 2024-04-14T15:18:26.836 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.9199999999999999e-06 2024-04-14T15:18:26.837 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.837 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.837 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:18:26.837 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.099507+0000", 2024-04-14T15:18:26.837 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000153156 2024-04-14T15:18:26.837 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.837 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.837 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:18:26.837 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.099597+0000", 2024-04-14T15:18:26.837 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0169999999999999e-05 2024-04-14T15:18:26.837 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.837 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.837 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1,6", 2024-04-14T15:18:26.838 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.099730+0000", 2024-04-14T15:18:26.838 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000133144 2024-04-14T15:18:26.838 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.838 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.838 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:18:26.838 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.102087+0000", 2024-04-14T15:18:26.838 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002356814 2024-04-14T15:18:26.838 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.838 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.838 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:18:26.838 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.164910+0000", 2024-04-14T15:18:26.838 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.062822391000000005 2024-04-14T15:18:26.838 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.839 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.839 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:18:26.839 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.196609+0000", 2024-04-14T15:18:26.839 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.031699790999999998 2024-04-14T15:18:26.839 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.839 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.839 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:18:26.839 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.196671+0000", 2024-04-14T15:18:26.839 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2024000000000001e-05 2024-04-14T15:18:26.839 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.839 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.839 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:18:26.840 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.196714+0000", 2024-04-14T15:18:26.840 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2163000000000003e-05 2024-04-14T15:18:26.840 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:18:26.840 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:18:26.840 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:18:26.840 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.840 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.840 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.24853.0:1853 16.1b 16:dbd443e1:test-rados-api-smithi090-93601-15::foo:head [setxattr a902 in=4b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-04-14T15:18:26.840 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:58.610106+0000", 2024-04-14T15:18:26.840 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 328.14032566999998, 2024-04-14T15:18:26.840 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.068056710000000006, 2024-04-14T15:18:26.840 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:18:26.840 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:18:26.840 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:18:26.840 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:18:26.841 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.24853", 2024-04-14T15:18:26.841 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/3851923713", 2024-04-14T15:18:26.841 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 1853 2024-04-14T15:18:26.841 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.841 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:18:26.841 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.841 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:18:26.841 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.610106+0000", 2024-04-14T15:18:26.842 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:18:26.842 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.842 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.842 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:18:26.842 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.610106+0000", 2024-04-14T15:18:26.842 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:18:26.842 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.842 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.842 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:18:26.843 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.610107+0000", 2024-04-14T15:18:26.843 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.3500000000000005e-07 2024-04-14T15:18:26.843 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.843 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.843 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:18:26.843 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.610112+0000", 2024-04-14T15:18:26.843 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.87e-06 2024-04-14T15:18:26.843 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.843 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.843 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:18:26.843 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.610122+0000", 2024-04-14T15:18:26.843 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0161e-05 2024-04-14T15:18:26.844 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.844 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.844 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:18:26.844 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.610129+0000", 2024-04-14T15:18:26.844 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8029999999999999e-06 2024-04-14T15:18:26.844 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.844 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.844 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:18:26.844 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.610174+0000", 2024-04-14T15:18:26.844 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.4799999999999998e-05 2024-04-14T15:18:26.844 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.844 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.845 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:18:26.845 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.610234+0000", 2024-04-14T15:18:26.845 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0414999999999997e-05 2024-04-14T15:18:26.845 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.845 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.845 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1,6", 2024-04-14T15:18:26.845 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.610309+0000", 2024-04-14T15:18:26.845 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.4377999999999998e-05 2024-04-14T15:18:26.845 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.845 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.845 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:18:26.845 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.611179+0000", 2024-04-14T15:18:26.845 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00087030300000000001 2024-04-14T15:18:26.845 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.846 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.846 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:18:26.846 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.611328+0000", 2024-04-14T15:18:26.846 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000148999 2024-04-14T15:18:26.846 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.846 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.846 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:18:26.846 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.678095+0000", 2024-04-14T15:18:26.846 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.066766958000000001 2024-04-14T15:18:26.846 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.846 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.846 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:18:26.846 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.678130+0000", 2024-04-14T15:18:26.847 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5423999999999998e-05 2024-04-14T15:18:26.847 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.847 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.847 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:18:26.847 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.678163+0000", 2024-04-14T15:18:26.847 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2663999999999998e-05 2024-04-14T15:18:26.847 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:18:26.847 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:18:26.847 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:18:26.847 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.847 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.847 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.24835.0:15 14.a 14:57934e76:test-rados-api-smithi090-93476-7::foo:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-04-14T15:18:26.847 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:58.632278+0000", 2024-04-14T15:18:26.847 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 328.118154543, 2024-04-14T15:18:26.848 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.046282352999999998, 2024-04-14T15:18:26.848 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:18:26.848 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:18:26.848 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:18:26.848 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:18:26.848 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.24835", 2024-04-14T15:18:26.848 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/1412026529", 2024-04-14T15:18:26.848 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 15 2024-04-14T15:18:26.848 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.848 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:18:26.848 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.848 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:18:26.848 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.632278+0000", 2024-04-14T15:18:26.848 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:18:26.848 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.849 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.849 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:18:26.849 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.632278+0000", 2024-04-14T15:18:26.849 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:18:26.849 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.849 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.849 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:18:26.849 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.632279+0000", 2024-04-14T15:18:26.849 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4670000000000001e-06 2024-04-14T15:18:26.849 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.849 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.849 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:18:26.849 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.632284+0000", 2024-04-14T15:18:26.849 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7079999999999996e-06 2024-04-14T15:18:26.849 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.849 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.850 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:18:26.850 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.632292+0000", 2024-04-14T15:18:26.850 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.9019999999999999e-06 2024-04-14T15:18:26.850 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.850 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.850 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:18:26.850 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.632297+0000", 2024-04-14T15:18:26.850 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6230000000000002e-06 2024-04-14T15:18:26.850 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.850 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.850 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:18:26.851 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.632345+0000", 2024-04-14T15:18:26.851 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7737000000000001e-05 2024-04-14T15:18:26.851 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.851 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.851 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:18:26.851 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.632423+0000", 2024-04-14T15:18:26.851 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.8055999999999999e-05 2024-04-14T15:18:26.851 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.851 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.851 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1,3", 2024-04-14T15:18:26.851 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.632537+0000", 2024-04-14T15:18:26.851 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000113689 2024-04-14T15:18:26.851 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.851 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.851 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:18:26.851 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.632805+0000", 2024-04-14T15:18:26.851 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00026789099999999997 2024-04-14T15:18:26.852 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.852 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.852 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:18:26.852 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.633077+0000", 2024-04-14T15:18:26.852 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000272279 2024-04-14T15:18:26.852 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.852 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.852 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:18:26.852 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.678497+0000", 2024-04-14T15:18:26.852 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.045419757999999998 2024-04-14T15:18:26.852 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.852 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.852 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:18:26.852 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.678526+0000", 2024-04-14T15:18:26.852 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9434e-05 2024-04-14T15:18:26.853 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.853 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.853 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:18:26.853 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.678560+0000", 2024-04-14T15:18:26.853 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.3809000000000001e-05 2024-04-14T15:18:26.853 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:18:26.853 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:18:26.853 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:18:26.853 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.853 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.853 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.24853.0:2070 16.12 e111/107 16:4b6e22cc:test-rados-api-smithi090-93601-17::big2:head v 111'7, mlcod=108'5)", 2024-04-14T15:18:26.853 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:59.459220+0000", 2024-04-14T15:18:26.853 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 327.291211589, 2024-04-14T15:18:26.853 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.061580874000000001, 2024-04-14T15:18:26.853 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:18:26.853 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:18:26.854 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:18:26.854 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:18:26.854 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.854 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:18:26.854 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.459220+0000", 2024-04-14T15:18:26.854 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:18:26.854 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.854 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.854 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:18:26.854 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.459220+0000", 2024-04-14T15:18:26.854 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:18:26.854 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.855 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.855 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:18:26.855 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.459222+0000", 2024-04-14T15:18:26.855 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4300000000000001e-06 2024-04-14T15:18:26.855 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.855 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.855 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:18:26.855 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.471887+0000", 2024-04-14T15:18:26.855 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.012665239 2024-04-14T15:18:26.855 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.855 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.855 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:18:26.856 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.471912+0000", 2024-04-14T15:18:26.856 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.4428000000000001e-05 2024-04-14T15:18:26.856 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.856 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.856 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:18:26.856 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.471923+0000", 2024-04-14T15:18:26.856 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1052e-05 2024-04-14T15:18:26.856 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.856 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.856 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:18:26.856 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.471994+0000", 2024-04-14T15:18:26.856 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1526000000000006e-05 2024-04-14T15:18:26.856 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.856 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.857 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:18:26.857 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.472022+0000", 2024-04-14T15:18:26.857 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.7789000000000001e-05 2024-04-14T15:18:26.857 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.857 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.857 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:18:26.857 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.520754+0000", 2024-04-14T15:18:26.857 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.048732005000000002 2024-04-14T15:18:26.857 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.857 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.857 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:18:26.857 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.520801+0000", 2024-04-14T15:18:26.858 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7404999999999998e-05 2024-04-14T15:18:26.858 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:18:26.858 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:18:26.858 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:18:26.858 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.858 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.858 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.24853.0:2074 16.c 16:31d79ce4:test-rados-api-smithi090-93601-17::big3:head [writefull 0~25165828] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-14T15:18:26.858 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:59.550683+0000", 2024-04-14T15:18:26.858 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 327.199748564, 2024-04-14T15:18:26.858 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.087008673999999994, 2024-04-14T15:18:26.858 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:18:26.858 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:18:26.858 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:18:26.859 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:18:26.859 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.24853", 2024-04-14T15:18:26.859 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/3851923713", 2024-04-14T15:18:26.859 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 2074 2024-04-14T15:18:26.859 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.859 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:18:26.859 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.859 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:18:26.859 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.550683+0000", 2024-04-14T15:18:26.859 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:18:26.859 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.859 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.859 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:18:26.859 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.550683+0000", 2024-04-14T15:18:26.860 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:18:26.860 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.860 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.860 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:18:26.860 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.550685+0000", 2024-04-14T15:18:26.860 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9e-06 2024-04-14T15:18:26.860 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.860 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.860 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:18:26.860 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.574320+0000", 2024-04-14T15:18:26.860 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.023634131999999999 2024-04-14T15:18:26.860 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.860 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.860 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:18:26.861 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.574340+0000", 2024-04-14T15:18:26.861 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0344999999999999e-05 2024-04-14T15:18:26.861 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.861 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.861 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:18:26.861 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.574352+0000", 2024-04-14T15:18:26.861 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2408999999999999e-05 2024-04-14T15:18:26.861 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.861 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.861 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:18:26.861 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.574415+0000", 2024-04-14T15:18:26.861 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2867000000000006e-05 2024-04-14T15:18:26.861 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.861 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.862 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:18:26.862 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.574521+0000", 2024-04-14T15:18:26.862 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010585 2024-04-14T15:18:26.862 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.862 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.862 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,4", 2024-04-14T15:18:26.862 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.574635+0000", 2024-04-14T15:18:26.862 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000114028 2024-04-14T15:18:26.862 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.862 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.862 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:18:26.862 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.628088+0000", 2024-04-14T15:18:26.863 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.053452491999999997 2024-04-14T15:18:26.863 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.863 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.863 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:18:26.863 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.637609+0000", 2024-04-14T15:18:26.863 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0095212950000000008 2024-04-14T15:18:26.863 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.863 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.863 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:18:26.863 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.637620+0000", 2024-04-14T15:18:26.863 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0696000000000001e-05 2024-04-14T15:18:26.863 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.863 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.863 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:18:26.864 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.637656+0000", 2024-04-14T15:18:26.864 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6294000000000002e-05 2024-04-14T15:18:26.864 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.864 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.864 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:18:26.864 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.637692+0000", 2024-04-14T15:18:26.864 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6365999999999997e-05 2024-04-14T15:18:26.864 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:18:26.864 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:18:26.864 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:18:26.864 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.864 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.864 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25240.0:1 67.0s0 67:005d9b81:test-rados-api-smithi090-93379-17::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-04-14T15:18:26.864 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:13:00.177410+0000", 2024-04-14T15:18:26.864 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 326.57302225299998, 2024-04-14T15:18:26.865 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.77316005899999996, 2024-04-14T15:18:26.865 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:18:26.865 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:18:26.865 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:18:26.865 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:18:26.865 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25240", 2024-04-14T15:18:26.865 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/3178004355", 2024-04-14T15:18:26.865 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 1 2024-04-14T15:18:26.865 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.865 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:18:26.865 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.865 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:18:26.865 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.177410+0000", 2024-04-14T15:18:26.865 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:18:26.865 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.866 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.866 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:18:26.867 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.177410+0000", 2024-04-14T15:18:26.867 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:18:26.868 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.868 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.868 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:18:26.868 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.177412+0000", 2024-04-14T15:18:26.868 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3939999999999999e-06 2024-04-14T15:18:26.868 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.868 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.868 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:18:26.868 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.177421+0000", 2024-04-14T15:18:26.868 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.1949999999999992e-06 2024-04-14T15:18:26.868 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.868 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.868 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:18:26.868 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.178275+0000", 2024-04-14T15:18:26.869 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00085384199999999995 2024-04-14T15:18:26.869 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.869 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.869 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:18:26.869 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.178289+0000", 2024-04-14T15:18:26.869 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3935e-05 2024-04-14T15:18:26.869 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.869 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.869 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:18:26.869 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.178811+0000", 2024-04-14T15:18:26.869 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00052192000000000004 2024-04-14T15:18:26.869 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.869 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.869 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:18:26.870 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.178814+0000", 2024-04-14T15:18:26.870 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.65e-06 2024-04-14T15:18:26.870 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.870 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.870 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:18:26.870 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.936581+0000", 2024-04-14T15:18:26.870 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.75776740499999995 2024-04-14T15:18:26.870 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.870 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.870 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:18:26.870 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.936707+0000", 2024-04-14T15:18:26.870 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000126119 2024-04-14T15:18:26.870 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.870 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.871 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_started", 2024-04-14T15:18:26.871 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.937166+0000", 2024-04-14T15:18:26.871 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00045856599999999999 2024-04-14T15:18:26.871 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.871 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.871 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_committed", 2024-04-14T15:18:26.871 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.937681+0000", 2024-04-14T15:18:26.871 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00051527300000000001 2024-04-14T15:18:26.871 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.871 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.871 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:18:26.871 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.950484+0000", 2024-04-14T15:18:26.871 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.012803162 2024-04-14T15:18:26.871 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.872 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.872 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:18:26.872 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.950570+0000", 2024-04-14T15:18:26.872 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5598000000000003e-05 2024-04-14T15:18:26.872 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:18:26.872 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:18:26.872 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:18:26.872 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.872 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.872 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.36752.0:2 145.0 145:0384ab8e:::oid:head [append 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-04-14T15:18:26.872 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:13:21.821544+0000", 2024-04-14T15:18:26.872 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 304.92888828899999, 2024-04-14T15:18:26.872 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1791939170000001, 2024-04-14T15:18:26.872 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:18:26.872 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:18:26.872 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:18:26.873 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:18:26.873 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.36752", 2024-04-14T15:18:26.873 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/3990091175", 2024-04-14T15:18:26.873 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 2 2024-04-14T15:18:26.873 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.873 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:18:26.873 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.873 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:18:26.873 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:21.821544+0000", 2024-04-14T15:18:26.873 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:18:26.873 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.873 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.873 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:18:26.873 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:21.821544+0000", 2024-04-14T15:18:26.873 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:18:26.873 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.874 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.874 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:18:26.874 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:21.821546+0000", 2024-04-14T15:18:26.874 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.889e-06 2024-04-14T15:18:26.874 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.874 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.874 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:18:26.874 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:21.821554+0000", 2024-04-14T15:18:26.874 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1019999999999997e-06 2024-04-14T15:18:26.874 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.874 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.874 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:18:26.874 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:21.821564+0000", 2024-04-14T15:18:26.874 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0441999999999999e-05 2024-04-14T15:18:26.875 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.875 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.875 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:18:26.875 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:21.822878+0000", 2024-04-14T15:18:26.875 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001313495 2024-04-14T15:18:26.875 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.875 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.875 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:18:26.875 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:21.831489+0000", 2024-04-14T15:18:26.875 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0086117720000000002 2024-04-14T15:18:26.875 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.875 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.875 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:18:26.875 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:21.831499+0000", 2024-04-14T15:18:26.875 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.2949999999999999e-06 2024-04-14T15:18:26.876 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.876 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.876 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:18:26.876 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:22.959687+0000", 2024-04-14T15:18:26.876 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1281882809999999 2024-04-14T15:18:26.876 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.876 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.876 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:18:26.876 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:22.959817+0000", 2024-04-14T15:18:26.876 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012978999999999999 2024-04-14T15:18:26.876 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.876 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.876 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 2,6", 2024-04-14T15:18:26.876 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:22.959961+0000", 2024-04-14T15:18:26.876 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014379200000000001 2024-04-14T15:18:26.876 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.877 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.877 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:18:26.877 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:22.961030+0000", 2024-04-14T15:18:26.877 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0010694400000000001 2024-04-14T15:18:26.877 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.877 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.877 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:18:26.877 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:22.961681+0000", 2024-04-14T15:18:26.877 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00065044799999999998 2024-04-14T15:18:26.877 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.877 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.877 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:18:26.877 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:23.000643+0000", 2024-04-14T15:18:26.877 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.038962055000000002 2024-04-14T15:18:26.877 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.878 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.878 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:18:26.878 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:23.000696+0000", 2024-04-14T15:18:26.878 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3875999999999997e-05 2024-04-14T15:18:26.878 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.878 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.878 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:18:26.878 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:23.000738+0000", 2024-04-14T15:18:26.878 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1239999999999998e-05 2024-04-14T15:18:26.878 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:18:26.878 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:18:26.879 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:18:26.879 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.879 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.879 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.44341.0:2 161.3 161:c656db43:::oid_RoundTripPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-14T15:18:26.879 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:13:28.864198+0000", 2024-04-14T15:18:26.879 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 297.88623441800002, 2024-04-14T15:18:26.879 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.083493583999999996, 2024-04-14T15:18:26.879 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:18:26.879 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:18:26.880 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:18:26.880 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:18:26.880 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.44341", 2024-04-14T15:18:26.880 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/4039543108", 2024-04-14T15:18:26.880 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 2 2024-04-14T15:18:26.880 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.880 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:18:26.880 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.880 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:18:26.880 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.864198+0000", 2024-04-14T15:18:26.880 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:18:26.880 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.880 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.880 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:18:26.881 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.864198+0000", 2024-04-14T15:18:26.881 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:18:26.881 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.881 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.881 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:18:26.881 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.864200+0000", 2024-04-14T15:18:26.881 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.018e-06 2024-04-14T15:18:26.881 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.881 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.881 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:18:26.881 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.864207+0000", 2024-04-14T15:18:26.881 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9280000000000004e-06 2024-04-14T15:18:26.881 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.881 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.882 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:18:26.882 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.864217+0000", 2024-04-14T15:18:26.882 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0249e-05 2024-04-14T15:18:26.882 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.882 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.882 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:18:26.882 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.864226+0000", 2024-04-14T15:18:26.882 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.2240000000000001e-06 2024-04-14T15:18:26.882 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.882 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.882 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:18:26.882 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.864291+0000", 2024-04-14T15:18:26.883 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5009e-05 2024-04-14T15:18:26.883 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.883 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.883 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:18:26.883 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.864299+0000", 2024-04-14T15:18:26.883 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.9570000000000002e-06 2024-04-14T15:18:26.883 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.883 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.883 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:18:26.883 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.906801+0000", 2024-04-14T15:18:26.883 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.042502054999999997 2024-04-14T15:18:26.883 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.883 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.883 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:18:26.883 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.906887+0000", 2024-04-14T15:18:26.884 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6168000000000006e-05 2024-04-14T15:18:26.884 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.884 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.884 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,3", 2024-04-14T15:18:26.884 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.907005+0000", 2024-04-14T15:18:26.884 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000117281 2024-04-14T15:18:26.884 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.884 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.884 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:18:26.884 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.907458+0000", 2024-04-14T15:18:26.884 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00045332400000000001 2024-04-14T15:18:26.885 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.885 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.885 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:18:26.885 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.914412+0000", 2024-04-14T15:18:26.885 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0069537720000000004 2024-04-14T15:18:26.885 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.885 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.885 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:18:26.885 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.947618+0000", 2024-04-14T15:18:26.885 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.033206688999999998 2024-04-14T15:18:26.885 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.885 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.885 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:18:26.885 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.947658+0000", 2024-04-14T15:18:26.886 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9946e-05 2024-04-14T15:18:26.886 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.886 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.886 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:18:26.886 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.947691+0000", 2024-04-14T15:18:26.886 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2963999999999998e-05 2024-04-14T15:18:26.886 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:18:26.886 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:18:26.886 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:18:26.886 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.886 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.886 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.24853.0:2079 16.11 16:8af1a83e:test-rados-api-smithi090-93601-17::big2.copy:head [copy-from ver 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-14T15:18:26.886 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:13:29.771058+0000", 2024-04-14T15:18:26.887 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 296.97937439999998, 2024-04-14T15:18:26.887 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.113103565, 2024-04-14T15:18:26.887 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:18:26.887 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:18:26.887 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:18:26.887 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:18:26.887 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.24853", 2024-04-14T15:18:26.887 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/3851923713", 2024-04-14T15:18:26.887 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 2079 2024-04-14T15:18:26.887 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.887 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:18:26.887 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.887 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:18:26.887 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:29.771058+0000", 2024-04-14T15:18:26.888 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:18:26.888 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.888 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.888 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:18:26.888 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:29.771058+0000", 2024-04-14T15:18:26.888 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:18:26.888 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.888 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.888 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:18:26.888 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:29.771061+0000", 2024-04-14T15:18:26.888 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.292e-06 2024-04-14T15:18:26.888 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.888 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.888 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:18:26.888 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:29.771071+0000", 2024-04-14T15:18:26.889 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0317e-05 2024-04-14T15:18:26.889 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.889 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.889 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:18:26.889 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:29.771093+0000", 2024-04-14T15:18:26.889 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1461999999999999e-05 2024-04-14T15:18:26.889 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.889 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.889 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:18:26.889 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:29.771105+0000", 2024-04-14T15:18:26.889 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2128e-05 2024-04-14T15:18:26.889 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.889 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.889 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:18:26.889 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:29.771208+0000", 2024-04-14T15:18:26.890 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010289300000000001 2024-04-14T15:18:26.890 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.890 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.890 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:18:26.890 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:29.771354+0000", 2024-04-14T15:18:26.890 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014649899999999999 2024-04-14T15:18:26.890 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.890 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.890 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,1", 2024-04-14T15:18:26.890 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:29.832534+0000", 2024-04-14T15:18:26.890 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.061179807000000003 2024-04-14T15:18:26.891 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.891 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.891 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:18:26.891 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:29.839859+0000", 2024-04-14T15:18:26.891 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0073249099999999996 2024-04-14T15:18:26.891 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.891 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.891 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:18:26.891 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:29.867794+0000", 2024-04-14T15:18:26.892 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.027935206000000001 2024-04-14T15:18:26.892 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.892 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.892 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:18:26.892 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:29.884083+0000", 2024-04-14T15:18:26.892 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.016289051999999998 2024-04-14T15:18:26.892 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.892 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.892 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:18:26.892 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:29.884125+0000", 2024-04-14T15:18:26.892 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1943000000000001e-05 2024-04-14T15:18:26.892 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.892 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.893 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:18:26.893 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:29.884161+0000", 2024-04-14T15:18:26.893 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6056000000000002e-05 2024-04-14T15:18:26.893 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:18:26.893 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:18:26.893 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:18:26.893 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.893 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.893 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.46400.0:2 202.11 e184/182 202:8de73fe4:test-rados-api-smithi090-93601-26::foo:head v 184'1, mlcod=0'0)", 2024-04-14T15:18:26.893 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:14:11.852282+0000", 2024-04-14T15:18:26.893 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 254.898149609, 2024-04-14T15:18:26.893 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.040243552000000002, 2024-04-14T15:18:26.893 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:18:26.894 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:18:26.894 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:18:26.894 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:18:26.894 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.894 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:18:26.894 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:11.852282+0000", 2024-04-14T15:18:26.894 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:18:26.894 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.894 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.894 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:18:26.894 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:11.852282+0000", 2024-04-14T15:18:26.894 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:18:26.894 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.895 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.895 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:18:26.895 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:11.852283+0000", 2024-04-14T15:18:26.895 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1300000000000001e-07 2024-04-14T15:18:26.895 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.895 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.895 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:18:26.895 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:11.876535+0000", 2024-04-14T15:18:26.895 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.024252234000000001 2024-04-14T15:18:26.895 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.895 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.895 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:18:26.895 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:11.876562+0000", 2024-04-14T15:18:26.896 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.654e-05 2024-04-14T15:18:26.896 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.896 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.896 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:18:26.896 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:11.876577+0000", 2024-04-14T15:18:26.896 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5503999999999999e-05 2024-04-14T15:18:26.896 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.896 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.896 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:18:26.896 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:11.876660+0000", 2024-04-14T15:18:26.896 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2367999999999995e-05 2024-04-14T15:18:26.896 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.897 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.897 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:18:26.897 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:11.876681+0000", 2024-04-14T15:18:26.897 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1022999999999998e-05 2024-04-14T15:18:26.897 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.897 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.897 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:18:26.897 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:11.892462+0000", 2024-04-14T15:18:26.897 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.015781309 2024-04-14T15:18:26.897 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:18:26.897 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:18:26.897 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:18:26.897 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:11.892526+0000", 2024-04-14T15:18:26.898 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4160999999999998e-05 2024-04-14T15:18:26.898 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:18:26.898 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:18:26.898 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:18:26.898 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:18:26.898 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:18:26.898 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-04-14T15:18:26.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:26.898 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:26.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:18:26.999 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:18:27.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:27.025 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:27.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:18:27.126 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:18:27.151 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:27.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:27.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:18:27.221 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:27 smithi090 ceph-mon[41670]: 11.5 scrub starts 2024-04-14T15:18:27.221 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:27 smithi090 ceph-mon[41670]: 11.5 scrub ok 2024-04-14T15:18:27.221 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:27 smithi090 ceph-mon[41670]: 11.11 scrub starts 2024-04-14T15:18:27.222 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:27 smithi090 ceph-mon[41670]: 11.11 scrub ok 2024-04-14T15:18:27.222 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:27 smithi090 ceph-mon[30399]: 11.5 scrub starts 2024-04-14T15:18:27.222 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:27 smithi090 ceph-mon[30399]: 11.5 scrub ok 2024-04-14T15:18:27.222 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:27 smithi090 ceph-mon[30399]: 11.11 scrub starts 2024-04-14T15:18:27.222 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:27 smithi090 ceph-mon[30399]: 11.11 scrub ok 2024-04-14T15:18:27.254 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:18:27.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:27.279 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:27.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:18:27.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:27 smithi092 ceph-mon[33966]: 11.5 scrub starts 2024-04-14T15:18:27.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:27 smithi092 ceph-mon[33966]: 11.5 scrub ok 2024-04-14T15:18:27.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:27 smithi092 ceph-mon[33966]: 11.11 scrub starts 2024-04-14T15:18:27.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:27 smithi092 ceph-mon[33966]: 11.11 scrub ok 2024-04-14T15:18:27.379 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:27.403 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:27.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:27.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:27.505 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:18:27.529 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:27.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:27.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:18:27.631 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:27.655 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:27.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:27.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:27.757 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:18:27.781 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:27.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:27.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:18:27.875 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-14T15:18:27.882 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:18:27.942 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:27.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:27.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:18:28.044 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:18:28.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:28.069 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:28.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:18:28.119 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:28 smithi090 ceph-mon[41670]: 11.8 scrub starts 2024-04-14T15:18:28.119 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:28 smithi090 ceph-mon[41670]: 11.8 scrub ok 2024-04-14T15:18:28.119 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:28 smithi090 ceph-mon[41670]: 11.12 scrub starts 2024-04-14T15:18:28.119 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:28 smithi090 ceph-mon[41670]: 11.12 scrub ok 2024-04-14T15:18:28.119 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:28 smithi090 ceph-mon[41670]: 2.7 scrub starts 2024-04-14T15:18:28.120 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:28 smithi090 ceph-mon[41670]: 2.7 scrub ok 2024-04-14T15:18:28.120 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:28 smithi090 ceph-mon[41670]: 3.0 scrub starts 2024-04-14T15:18:28.120 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:28 smithi090 ceph-mon[41670]: 3.0 scrub ok 2024-04-14T15:18:28.120 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:28 smithi090 ceph-mon[41670]: pgmap v592: 238 pgs: 238 active+clean; 8.4 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 767 B/s wr, 1 op/s 2024-04-14T15:18:28.120 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:28 smithi090 ceph-mon[41670]: 11.6 scrub starts 2024-04-14T15:18:28.120 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:28 smithi090 ceph-mon[41670]: 11.6 scrub ok 2024-04-14T15:18:28.121 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:28 smithi090 ceph-mon[30399]: 11.8 scrub starts 2024-04-14T15:18:28.121 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:28 smithi090 ceph-mon[30399]: 11.8 scrub ok 2024-04-14T15:18:28.121 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:28 smithi090 ceph-mon[30399]: 11.12 scrub starts 2024-04-14T15:18:28.121 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:28 smithi090 ceph-mon[30399]: 11.12 scrub ok 2024-04-14T15:18:28.121 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:28 smithi090 ceph-mon[30399]: 2.7 scrub starts 2024-04-14T15:18:28.121 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:28 smithi090 ceph-mon[30399]: 2.7 scrub ok 2024-04-14T15:18:28.122 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:28 smithi090 ceph-mon[30399]: 3.0 scrub starts 2024-04-14T15:18:28.122 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:28 smithi090 ceph-mon[30399]: 3.0 scrub ok 2024-04-14T15:18:28.122 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:28 smithi090 ceph-mon[30399]: pgmap v592: 238 pgs: 238 active+clean; 8.4 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 767 B/s wr, 1 op/s 2024-04-14T15:18:28.122 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:28 smithi090 ceph-mon[30399]: 11.6 scrub starts 2024-04-14T15:18:28.122 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:28 smithi090 ceph-mon[30399]: 11.6 scrub ok 2024-04-14T15:18:28.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:28 smithi092 ceph-mon[33966]: 11.8 scrub starts 2024-04-14T15:18:28.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:28 smithi092 ceph-mon[33966]: 11.8 scrub ok 2024-04-14T15:18:28.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:28 smithi092 ceph-mon[33966]: 11.12 scrub starts 2024-04-14T15:18:28.124 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:28 smithi092 ceph-mon[33966]: 11.12 scrub ok 2024-04-14T15:18:28.124 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:28 smithi092 ceph-mon[33966]: 2.7 scrub starts 2024-04-14T15:18:28.124 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:28 smithi092 ceph-mon[33966]: 2.7 scrub ok 2024-04-14T15:18:28.124 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:28 smithi092 ceph-mon[33966]: 3.0 scrub starts 2024-04-14T15:18:28.124 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:28 smithi092 ceph-mon[33966]: 3.0 scrub ok 2024-04-14T15:18:28.124 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:28 smithi092 ceph-mon[33966]: pgmap v592: 238 pgs: 238 active+clean; 8.4 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 767 B/s wr, 1 op/s 2024-04-14T15:18:28.125 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:28 smithi092 ceph-mon[33966]: 11.6 scrub starts 2024-04-14T15:18:28.125 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:28 smithi092 ceph-mon[33966]: 11.6 scrub ok 2024-04-14T15:18:28.125 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:18:28.170 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:18:28.193 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:28.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:28.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:18:28.294 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:18:28.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:28.319 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:28.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:18:28.420 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:18:28.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:28.445 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:28.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:18:28.546 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:18:28.571 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:28.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:28.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:18:28.673 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:18:28.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:28.697 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:28.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:18:28.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.769+0000 7f35321d0700 1 -- 172.21.15.90:0/1188408685 >> v1:172.21.15.90:6789/0 conn(0x7f352c107780 legacy=0x7f352c107b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:28.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.769+0000 7f35321d0700 1 -- 172.21.15.90:0/1188408685 shutdown_connections 2024-04-14T15:18:28.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.769+0000 7f35321d0700 1 -- 172.21.15.90:0/1188408685 >> 172.21.15.90:0/1188408685 conn(0x7f352c0fd2b0 msgr2=0x7f352c0ff6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:18:28.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.769+0000 7f35321d0700 1 -- 172.21.15.90:0/1188408685 shutdown_connections 2024-04-14T15:18:28.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.769+0000 7f35321d0700 1 -- 172.21.15.90:0/1188408685 wait complete. 2024-04-14T15:18:28.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.769+0000 7f35321d0700 1 librados: starting msgr at 2024-04-14T15:18:28.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.769+0000 7f35321d0700 1 librados: starting objecter 2024-04-14T15:18:28.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.770+0000 7f35321d0700 1 Processor -- start 2024-04-14T15:18:28.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.770+0000 7f35321d0700 1 -- start start 2024-04-14T15:18:28.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.770+0000 7f35321d0700 1 librados: setting wanted keys 2024-04-14T15:18:28.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.770+0000 7f35321d0700 1 librados: calling monclient init 2024-04-14T15:18:28.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.770+0000 7f35321d0700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f352c10dda0 con 0x7f352c10a630 2024-04-14T15:18:28.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.770+0000 7f35321d0700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f352c10dfb0 con 0x7f352c10e200 2024-04-14T15:18:28.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.770+0000 7f35321d0700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f352c1ac220 con 0x7f352c107780 2024-04-14T15:18:28.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.770+0000 7f352b7fe700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f352c107780 0x7f352c10aa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:37166/0 (socket says 172.21.15.90:37166) 2024-04-14T15:18:28.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.770+0000 7f3522ffd700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f352c10a630 0x7f352c10b120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:49002/0 (socket says 172.21.15.90:49002) 2024-04-14T15:18:28.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.770+0000 7f352b7fe700 1 -- 172.21.15.90:0/3282112766 learned_addr learned my addr 172.21.15.90:0/3282112766 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:18:28.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.770+0000 7f35297fa700 10 client.?.objecter ms_handle_connect 0x7f352c10a630 2024-04-14T15:18:28.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.770+0000 7f35297fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:18:28.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.770+0000 7f35297fa700 10 client.?.objecter ms_handle_connect 0x7f352c107780 2024-04-14T15:18:28.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.770+0000 7f35297fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:18:28.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.770+0000 7f35297fa700 10 client.?.objecter ms_handle_connect 0x7f352c10e200 2024-04-14T15:18:28.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.770+0000 7f35297fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:18:28.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.770+0000 7f35297fa700 1 -- 172.21.15.90:0/3282112766 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4235650988 0 0) 0x7f352c10dda0 con 0x7f352c10a630 2024-04-14T15:18:28.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.770+0000 7f35297fa700 1 -- 172.21.15.90:0/3282112766 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f35180035b0 con 0x7f352c10a630 2024-04-14T15:18:28.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.770+0000 7f35297fa700 1 -- 172.21.15.90:0/3282112766 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1074446085 0 0) 0x7f352c1ac220 con 0x7f352c107780 2024-04-14T15:18:28.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.770+0000 7f35297fa700 1 -- 172.21.15.90:0/3282112766 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f352c10dda0 con 0x7f352c107780 2024-04-14T15:18:28.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.770+0000 7f35297fa700 1 -- 172.21.15.90:0/3282112766 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1356903500 0 0) 0x7f352c10dfb0 con 0x7f352c10e200 2024-04-14T15:18:28.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.770+0000 7f35297fa700 1 -- 172.21.15.90:0/3282112766 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f352c1ac220 con 0x7f352c10e200 2024-04-14T15:18:28.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.771+0000 7f35297fa700 1 -- 172.21.15.90:0/3282112766 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2196237315 0 0) 0x7f352c1ac220 con 0x7f352c10e200 2024-04-14T15:18:28.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.771+0000 7f35297fa700 1 -- 172.21.15.90:0/3282112766 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f352c10dfb0 con 0x7f352c10e200 2024-04-14T15:18:28.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.771+0000 7f35297fa700 1 -- 172.21.15.90:0/3282112766 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1752722528 0 0) 0x7f35180035b0 con 0x7f352c10a630 2024-04-14T15:18:28.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.771+0000 7f35297fa700 1 -- 172.21.15.90:0/3282112766 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f352c1ac220 con 0x7f352c10a630 2024-04-14T15:18:28.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.771+0000 7f35297fa700 1 -- 172.21.15.90:0/3282112766 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f3524001bd0 con 0x7f352c10e200 2024-04-14T15:18:28.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.771+0000 7f35297fa700 1 -- 172.21.15.90:0/3282112766 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2677002052 0 0) 0x7f352c10dda0 con 0x7f352c107780 2024-04-14T15:18:28.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.771+0000 7f35297fa700 1 -- 172.21.15.90:0/3282112766 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f35180035b0 con 0x7f352c107780 2024-04-14T15:18:28.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.771+0000 7f35297fa700 1 -- 172.21.15.90:0/3282112766 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f3514001830 con 0x7f352c10a630 2024-04-14T15:18:28.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.771+0000 7f35297fa700 1 -- 172.21.15.90:0/3282112766 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f351c001430 con 0x7f352c107780 2024-04-14T15:18:28.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.771+0000 7f35297fa700 1 -- 172.21.15.90:0/3282112766 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2540441721 0 0) 0x7f352c10dfb0 con 0x7f352c10e200 2024-04-14T15:18:28.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.771+0000 7f35297fa700 1 -- 172.21.15.90:0/3282112766 >> v1:172.21.15.90:6790/0 conn(0x7f352c107780 legacy=0x7f352c10aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:28.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.771+0000 7f35297fa700 1 -- 172.21.15.90:0/3282112766 >> v1:172.21.15.90:6789/0 conn(0x7f352c10a630 legacy=0x7f352c10b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:28.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.771+0000 7f35297fa700 1 -- 172.21.15.90:0/3282112766 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f352c1ad440 con 0x7f352c10e200 2024-04-14T15:18:28.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.771+0000 7f35321d0700 1 -- 172.21.15.90:0/3282112766 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f352c1ac450 con 0x7f352c10e200 2024-04-14T15:18:28.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.771+0000 7f35321d0700 10 client.49126.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:18:28.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.771+0000 7f35321d0700 1 -- 172.21.15.90:0/3282112766 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f352c1ac910 con 0x7f352c10e200 2024-04-14T15:18:28.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.771+0000 7f35321d0700 1 librados: init done 2024-04-14T15:18:28.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.771+0000 7f35297fa700 1 -- 172.21.15.90:0/3282112766 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f3524003ec0 con 0x7f352c10e200 2024-04-14T15:18:28.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.771+0000 7f35297fa700 1 -- 172.21.15.90:0/3282112766 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f3524004cd0 con 0x7f352c10e200 2024-04-14T15:18:28.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.772+0000 7f35321d0700 1 -- 172.21.15.90:0/3282112766 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f350c005130 con 0x7f352c10e200 2024-04-14T15:18:28.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.772+0000 7f35297fa700 1 -- 172.21.15.90:0/3282112766 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f3524003a00 con 0x7f352c10e200 2024-04-14T15:18:28.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.773+0000 7f35297fa700 1 -- 172.21.15.90:0/3282112766 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(427..427 src has 51..427) ==== 8216+0+0 (unknown 1026895677 0 0) 0x7f35240898e0 con 0x7f352c10e200 2024-04-14T15:18:28.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.773+0000 7f35297fa700 10 client.49126.objecter ms_dispatch 0x7f352c001550 osd_map(427..427 src has 51..427) 2024-04-14T15:18:28.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.773+0000 7f35297fa700 3 client.49126.objecter handle_osd_map got epochs [427,427] > 0 2024-04-14T15:18:28.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.773+0000 7f35297fa700 3 client.49126.objecter handle_osd_map decoding full epoch 427 2024-04-14T15:18:28.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.773+0000 7f35297fa700 20 client.49126.objecter dump_active .. 0 homeless 2024-04-14T15:18:28.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.773+0000 7f35297fa700 1 -- 172.21.15.90:0/3282112766 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v427) ==== 54+0+16 (unknown 3296173923 0 2420000192) 0x7f352408a060 con 0x7f352c10e200 2024-04-14T15:18:28.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.773+0000 7f35297fa700 10 client.49126.objecter ms_handle_connect 0x7f35180714d0 2024-04-14T15:18:28.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.773+0000 7f35321d0700 10 client.49126.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:18:28.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.773+0000 7f35321d0700 15 client.49126.objecter _session_command_op_assign -1 1 2024-04-14T15:18:28.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.773+0000 7f35321d0700 20 client.49126.objecter _get_session s=0x7f350c005410 osd=0 3 2024-04-14T15:18:28.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.773+0000 7f35321d0700 20 client.49126.objecter put_session s=0x7f350c005410 osd=0 3 2024-04-14T15:18:28.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.773+0000 7f35321d0700 20 client.49126.objecter _get_session s=0x7f350c005410 osd=0 3 2024-04-14T15:18:28.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.773+0000 7f35321d0700 15 client.49126.objecter _session_command_op_remove -1 1 2024-04-14T15:18:28.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.773+0000 7f35321d0700 20 client.49126.objecter get_session s=0x7f350c005410 osd=0 3 2024-04-14T15:18:28.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.773+0000 7f35321d0700 15 client.49126.objecter _session_command_op_assign 0 1 2024-04-14T15:18:28.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.773+0000 7f35321d0700 20 client.49126.objecter put_session s=0x7f350c005410 osd=0 4 2024-04-14T15:18:28.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.773+0000 7f35321d0700 10 client.49126.objecter _send_command 1 2024-04-14T15:18:28.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.773+0000 7f35321d0700 1 -- 172.21.15.90:0/3282112766 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f350c004b80 con 0x7f350c004760 2024-04-14T15:18:28.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.774+0000 7f35297fa700 10 client.49126.objecter ms_handle_connect 0x7f350c004760 2024-04-14T15:18:28.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.774+0000 7f35297fa700 1 -- 172.21.15.90:0/3282112766 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f350c004b80 con 0x7f350c004760 2024-04-14T15:18:28.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.774+0000 7f35297fa700 10 client.49126.objecter ms_dispatch 0x7f352c001550 command_reply(tid 1: 0 ) 2024-04-14T15:18:28.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.774+0000 7f35297fa700 10 client.49126.objecter _finish_command 1 = system:0 2024-04-14T15:18:28.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.774+0000 7f35297fa700 20 client.49126.objecter put_session s=0x7f350c005410 osd=0 4 2024-04-14T15:18:28.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.774+0000 7f35297fa700 15 client.49126.objecter _session_command_op_remove 0 1 2024-04-14T15:18:28.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.791+0000 7f35321d0700 10 client.49126.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:18:28.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.791+0000 7f35321d0700 15 client.49126.objecter _session_command_op_assign -1 2 2024-04-14T15:18:28.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.791+0000 7f35321d0700 20 client.49126.objecter _get_session s=0x7f350c005410 osd=0 3 2024-04-14T15:18:28.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.791+0000 7f35321d0700 20 client.49126.objecter put_session s=0x7f350c005410 osd=0 3 2024-04-14T15:18:28.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.791+0000 7f35321d0700 20 client.49126.objecter _get_session s=0x7f350c005410 osd=0 3 2024-04-14T15:18:28.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.791+0000 7f35321d0700 15 client.49126.objecter _session_command_op_remove -1 2 2024-04-14T15:18:28.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.791+0000 7f35321d0700 20 client.49126.objecter get_session s=0x7f350c005410 osd=0 3 2024-04-14T15:18:28.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.791+0000 7f35321d0700 15 client.49126.objecter _session_command_op_assign 0 2 2024-04-14T15:18:28.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.791+0000 7f35321d0700 20 client.49126.objecter put_session s=0x7f350c005410 osd=0 4 2024-04-14T15:18:28.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.791+0000 7f35321d0700 10 client.49126.objecter _send_command 2 2024-04-14T15:18:28.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.791+0000 7f35321d0700 1 -- 172.21.15.90:0/3282112766 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f350c007a20 con 0x7f350c004760 2024-04-14T15:18:28.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.793+0000 7f35297fa700 1 -- 172.21.15.90:0/3282112766 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f350c007a20 con 0x7f350c004760 2024-04-14T15:18:28.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.793+0000 7f35297fa700 10 client.49126.objecter ms_dispatch 0x7f352c001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:18:28.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.793+0000 7f35297fa700 10 client.49126.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:18:28.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.793+0000 7f35297fa700 20 client.49126.objecter put_session s=0x7f350c005410 osd=0 4 2024-04-14T15:18:28.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.793+0000 7f35297fa700 15 client.49126.objecter _session_command_op_remove 0 2 2024-04-14T15:18:28.794 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:18:28.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.793+0000 7f35321d0700 10 client.49126.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:18:28.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.793+0000 7f35321d0700 15 client.49126.objecter _session_command_op_assign -1 3 2024-04-14T15:18:28.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.793+0000 7f35321d0700 20 client.49126.objecter _get_session s=0x7f350c004140 osd=1 3 2024-04-14T15:18:28.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.793+0000 7f35321d0700 20 client.49126.objecter put_session s=0x7f350c004140 osd=1 3 2024-04-14T15:18:28.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.793+0000 7f35321d0700 20 client.49126.objecter _get_session s=0x7f350c004140 osd=1 3 2024-04-14T15:18:28.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.793+0000 7f35321d0700 15 client.49126.objecter _session_command_op_remove -1 3 2024-04-14T15:18:28.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.793+0000 7f35321d0700 20 client.49126.objecter get_session s=0x7f350c004140 osd=1 3 2024-04-14T15:18:28.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.793+0000 7f35321d0700 15 client.49126.objecter _session_command_op_assign 1 3 2024-04-14T15:18:28.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.793+0000 7f35321d0700 20 client.49126.objecter put_session s=0x7f350c004140 osd=1 4 2024-04-14T15:18:28.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.793+0000 7f35321d0700 10 client.49126.objecter _send_command 3 2024-04-14T15:18:28.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.793+0000 7f35321d0700 1 -- 172.21.15.90:0/3282112766 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f350c0021b0 con 0x7f350c0042b0 2024-04-14T15:18:28.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.793+0000 7f35297fa700 10 client.49126.objecter ms_handle_connect 0x7f350c0042b0 2024-04-14T15:18:28.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.794+0000 7f35297fa700 1 -- 172.21.15.90:0/3282112766 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f350c0021b0 con 0x7f350c0042b0 2024-04-14T15:18:28.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.794+0000 7f35297fa700 10 client.49126.objecter ms_dispatch 0x7f352c001550 command_reply(tid 3: 0 ) 2024-04-14T15:18:28.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.794+0000 7f35297fa700 10 client.49126.objecter _finish_command 3 = system:0 2024-04-14T15:18:28.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.794+0000 7f35297fa700 20 client.49126.objecter put_session s=0x7f350c004140 osd=1 4 2024-04-14T15:18:28.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.794+0000 7f35297fa700 15 client.49126.objecter _session_command_op_remove 1 3 2024-04-14T15:18:28.798 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:28.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.810+0000 7f35321d0700 10 client.49126.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:18:28.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.810+0000 7f35321d0700 15 client.49126.objecter _session_command_op_assign -1 4 2024-04-14T15:18:28.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.810+0000 7f35321d0700 20 client.49126.objecter _get_session s=0x7f350c004140 osd=1 3 2024-04-14T15:18:28.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.810+0000 7f35321d0700 20 client.49126.objecter put_session s=0x7f350c004140 osd=1 3 2024-04-14T15:18:28.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.810+0000 7f35321d0700 20 client.49126.objecter _get_session s=0x7f350c004140 osd=1 3 2024-04-14T15:18:28.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.810+0000 7f35321d0700 15 client.49126.objecter _session_command_op_remove -1 4 2024-04-14T15:18:28.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.810+0000 7f35321d0700 20 client.49126.objecter get_session s=0x7f350c004140 osd=1 3 2024-04-14T15:18:28.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.810+0000 7f35321d0700 15 client.49126.objecter _session_command_op_assign 1 4 2024-04-14T15:18:28.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.810+0000 7f35321d0700 20 client.49126.objecter put_session s=0x7f350c004140 osd=1 4 2024-04-14T15:18:28.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.810+0000 7f35321d0700 10 client.49126.objecter _send_command 4 2024-04-14T15:18:28.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.810+0000 7f35321d0700 1 -- 172.21.15.90:0/3282112766 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f350c001630 con 0x7f350c0042b0 2024-04-14T15:18:28.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.812+0000 7f35297fa700 1 -- 172.21.15.90:0/3282112766 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f350c001630 con 0x7f350c0042b0 2024-04-14T15:18:28.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.812+0000 7f35297fa700 10 client.49126.objecter ms_dispatch 0x7f352c001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:18:28.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.812+0000 7f35297fa700 10 client.49126.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:18:28.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.812+0000 7f35297fa700 20 client.49126.objecter put_session s=0x7f350c004140 osd=1 4 2024-04-14T15:18:28.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.812+0000 7f35297fa700 15 client.49126.objecter _session_command_op_remove 1 4 2024-04-14T15:18:28.813 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:18:28.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.812+0000 7f35321d0700 10 client.49126.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:18:28.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.812+0000 7f35321d0700 15 client.49126.objecter _session_command_op_assign -1 5 2024-04-14T15:18:28.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.812+0000 7f35321d0700 20 client.49126.objecter _get_session s=0x7f350c0083a0 osd=2 3 2024-04-14T15:18:28.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.812+0000 7f35321d0700 20 client.49126.objecter put_session s=0x7f350c0083a0 osd=2 3 2024-04-14T15:18:28.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.812+0000 7f35321d0700 20 client.49126.objecter _get_session s=0x7f350c0083a0 osd=2 3 2024-04-14T15:18:28.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.812+0000 7f35321d0700 15 client.49126.objecter _session_command_op_remove -1 5 2024-04-14T15:18:28.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.812+0000 7f35321d0700 20 client.49126.objecter get_session s=0x7f350c0083a0 osd=2 3 2024-04-14T15:18:28.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.812+0000 7f35321d0700 15 client.49126.objecter _session_command_op_assign 2 5 2024-04-14T15:18:28.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.812+0000 7f35321d0700 20 client.49126.objecter put_session s=0x7f350c0083a0 osd=2 4 2024-04-14T15:18:28.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.812+0000 7f35321d0700 10 client.49126.objecter _send_command 5 2024-04-14T15:18:28.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.812+0000 7f35321d0700 1 -- 172.21.15.90:0/3282112766 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f350c011110 con 0x7f350c00d600 2024-04-14T15:18:28.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.812+0000 7f35297fa700 10 client.49126.objecter ms_handle_connect 0x7f350c00d600 2024-04-14T15:18:28.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.813+0000 7f35297fa700 1 -- 172.21.15.90:0/3282112766 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f350c011110 con 0x7f350c00d600 2024-04-14T15:18:28.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.813+0000 7f35297fa700 10 client.49126.objecter ms_dispatch 0x7f352c001550 command_reply(tid 5: 0 ) 2024-04-14T15:18:28.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.813+0000 7f35297fa700 10 client.49126.objecter _finish_command 5 = system:0 2024-04-14T15:18:28.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.813+0000 7f35297fa700 20 client.49126.objecter put_session s=0x7f350c0083a0 osd=2 4 2024-04-14T15:18:28.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.813+0000 7f35297fa700 15 client.49126.objecter _session_command_op_remove 2 5 2024-04-14T15:18:28.821 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:28.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:28.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:28.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.829+0000 7f35321d0700 10 client.49126.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:18:28.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.829+0000 7f35321d0700 15 client.49126.objecter _session_command_op_assign -1 6 2024-04-14T15:18:28.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.829+0000 7f35321d0700 20 client.49126.objecter _get_session s=0x7f350c0083a0 osd=2 3 2024-04-14T15:18:28.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.829+0000 7f35321d0700 20 client.49126.objecter put_session s=0x7f350c0083a0 osd=2 3 2024-04-14T15:18:28.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.829+0000 7f35321d0700 20 client.49126.objecter _get_session s=0x7f350c0083a0 osd=2 3 2024-04-14T15:18:28.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.829+0000 7f35321d0700 15 client.49126.objecter _session_command_op_remove -1 6 2024-04-14T15:18:28.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.829+0000 7f35321d0700 20 client.49126.objecter get_session s=0x7f350c0083a0 osd=2 3 2024-04-14T15:18:28.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.829+0000 7f35321d0700 15 client.49126.objecter _session_command_op_assign 2 6 2024-04-14T15:18:28.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.829+0000 7f35321d0700 20 client.49126.objecter put_session s=0x7f350c0083a0 osd=2 4 2024-04-14T15:18:28.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.829+0000 7f35321d0700 10 client.49126.objecter _send_command 6 2024-04-14T15:18:28.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.829+0000 7f35321d0700 1 -- 172.21.15.90:0/3282112766 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f350c007e80 con 0x7f350c00d600 2024-04-14T15:18:28.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.831+0000 7f35297fa700 1 -- 172.21.15.90:0/3282112766 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f350c007e80 con 0x7f350c00d600 2024-04-14T15:18:28.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.831+0000 7f35297fa700 10 client.49126.objecter ms_dispatch 0x7f352c001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:18:28.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.831+0000 7f35297fa700 10 client.49126.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:18:28.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.831+0000 7f35297fa700 20 client.49126.objecter put_session s=0x7f350c0083a0 osd=2 4 2024-04-14T15:18:28.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.831+0000 7f35297fa700 15 client.49126.objecter _session_command_op_remove 2 6 2024-04-14T15:18:28.832 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:18:28.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.831+0000 7f35321d0700 10 client.49126.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:18:28.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.831+0000 7f35321d0700 15 client.49126.objecter _session_command_op_assign -1 7 2024-04-14T15:18:28.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.831+0000 7f35321d0700 20 client.49126.objecter _get_session s=0x7f350c00cc50 osd=3 3 2024-04-14T15:18:28.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.831+0000 7f35321d0700 20 client.49126.objecter put_session s=0x7f350c00cc50 osd=3 3 2024-04-14T15:18:28.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.831+0000 7f35321d0700 20 client.49126.objecter _get_session s=0x7f350c00cc50 osd=3 3 2024-04-14T15:18:28.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.831+0000 7f35321d0700 15 client.49126.objecter _session_command_op_remove -1 7 2024-04-14T15:18:28.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.831+0000 7f35321d0700 20 client.49126.objecter get_session s=0x7f350c00cc50 osd=3 3 2024-04-14T15:18:28.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.831+0000 7f35321d0700 15 client.49126.objecter _session_command_op_assign 3 7 2024-04-14T15:18:28.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.831+0000 7f35321d0700 20 client.49126.objecter put_session s=0x7f350c00cc50 osd=3 4 2024-04-14T15:18:28.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.831+0000 7f35321d0700 10 client.49126.objecter _send_command 7 2024-04-14T15:18:28.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.831+0000 7f35321d0700 1 -- 172.21.15.90:0/3282112766 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f350c015f50 con 0x7f350c012440 2024-04-14T15:18:28.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.831+0000 7f35297fa700 10 client.49126.objecter ms_handle_connect 0x7f350c012440 2024-04-14T15:18:28.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.832+0000 7f35297fa700 1 -- 172.21.15.90:0/3282112766 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f350c015f50 con 0x7f350c012440 2024-04-14T15:18:28.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.832+0000 7f35297fa700 10 client.49126.objecter ms_dispatch 0x7f352c001550 command_reply(tid 7: 0 ) 2024-04-14T15:18:28.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.832+0000 7f35297fa700 10 client.49126.objecter _finish_command 7 = system:0 2024-04-14T15:18:28.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.832+0000 7f35297fa700 20 client.49126.objecter put_session s=0x7f350c00cc50 osd=3 4 2024-04-14T15:18:28.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.832+0000 7f35297fa700 15 client.49126.objecter _session_command_op_remove 3 7 2024-04-14T15:18:28.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.848+0000 7f35321d0700 10 client.49126.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:18:28.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.848+0000 7f35321d0700 15 client.49126.objecter _session_command_op_assign -1 8 2024-04-14T15:18:28.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.848+0000 7f35321d0700 20 client.49126.objecter _get_session s=0x7f350c00cc50 osd=3 3 2024-04-14T15:18:28.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.848+0000 7f35321d0700 20 client.49126.objecter put_session s=0x7f350c00cc50 osd=3 3 2024-04-14T15:18:28.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.848+0000 7f35321d0700 20 client.49126.objecter _get_session s=0x7f350c00cc50 osd=3 3 2024-04-14T15:18:28.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.848+0000 7f35321d0700 15 client.49126.objecter _session_command_op_remove -1 8 2024-04-14T15:18:28.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.848+0000 7f35321d0700 20 client.49126.objecter get_session s=0x7f350c00cc50 osd=3 3 2024-04-14T15:18:28.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.848+0000 7f35321d0700 15 client.49126.objecter _session_command_op_assign 3 8 2024-04-14T15:18:28.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.848+0000 7f35321d0700 20 client.49126.objecter put_session s=0x7f350c00cc50 osd=3 4 2024-04-14T15:18:28.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.848+0000 7f35321d0700 10 client.49126.objecter _send_command 8 2024-04-14T15:18:28.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.848+0000 7f35321d0700 1 -- 172.21.15.90:0/3282112766 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f350c00c730 con 0x7f350c012440 2024-04-14T15:18:28.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.850+0000 7f35297fa700 1 -- 172.21.15.90:0/3282112766 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f350c00c730 con 0x7f350c012440 2024-04-14T15:18:28.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.850+0000 7f35297fa700 10 client.49126.objecter ms_dispatch 0x7f352c001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:18:28.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.850+0000 7f35297fa700 10 client.49126.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:18:28.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.850+0000 7f35297fa700 20 client.49126.objecter put_session s=0x7f350c00cc50 osd=3 4 2024-04-14T15:18:28.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.850+0000 7f35297fa700 15 client.49126.objecter _session_command_op_remove 3 8 2024-04-14T15:18:28.851 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:18:28.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.850+0000 7f35321d0700 10 client.49126.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:18:28.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.850+0000 7f35321d0700 15 client.49126.objecter _session_command_op_assign -1 9 2024-04-14T15:18:28.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.850+0000 7f35321d0700 20 client.49126.objecter _get_session s=0x7f350c00c4e0 osd=4 3 2024-04-14T15:18:28.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.850+0000 7f35321d0700 20 client.49126.objecter put_session s=0x7f350c00c4e0 osd=4 3 2024-04-14T15:18:28.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.850+0000 7f35321d0700 20 client.49126.objecter _get_session s=0x7f350c00c4e0 osd=4 3 2024-04-14T15:18:28.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.850+0000 7f35321d0700 15 client.49126.objecter _session_command_op_remove -1 9 2024-04-14T15:18:28.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.850+0000 7f35321d0700 20 client.49126.objecter get_session s=0x7f350c00c4e0 osd=4 3 2024-04-14T15:18:28.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.850+0000 7f35321d0700 15 client.49126.objecter _session_command_op_assign 4 9 2024-04-14T15:18:28.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.850+0000 7f35321d0700 20 client.49126.objecter put_session s=0x7f350c00c4e0 osd=4 4 2024-04-14T15:18:28.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.850+0000 7f35321d0700 10 client.49126.objecter _send_command 9 2024-04-14T15:18:28.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.850+0000 7f35321d0700 1 -- 172.21.15.90:0/3282112766 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f350c00c710 con 0x7f350c011fa0 2024-04-14T15:18:28.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.851+0000 7f35297fa700 10 client.49126.objecter ms_handle_connect 0x7f350c011fa0 2024-04-14T15:18:28.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.852+0000 7f35297fa700 1 -- 172.21.15.90:0/3282112766 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f350c00c710 con 0x7f350c011fa0 2024-04-14T15:18:28.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.852+0000 7f35297fa700 10 client.49126.objecter ms_dispatch 0x7f352c001550 command_reply(tid 9: 0 ) 2024-04-14T15:18:28.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.852+0000 7f35297fa700 10 client.49126.objecter _finish_command 9 = system:0 2024-04-14T15:18:28.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.852+0000 7f35297fa700 20 client.49126.objecter put_session s=0x7f350c00c4e0 osd=4 4 2024-04-14T15:18:28.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.852+0000 7f35297fa700 15 client.49126.objecter _session_command_op_remove 4 9 2024-04-14T15:18:28.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.867+0000 7f35321d0700 10 client.49126.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:18:28.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.867+0000 7f35321d0700 15 client.49126.objecter _session_command_op_assign -1 10 2024-04-14T15:18:28.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.867+0000 7f35321d0700 20 client.49126.objecter _get_session s=0x7f350c00c4e0 osd=4 3 2024-04-14T15:18:28.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.867+0000 7f35321d0700 20 client.49126.objecter put_session s=0x7f350c00c4e0 osd=4 3 2024-04-14T15:18:28.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.867+0000 7f35321d0700 20 client.49126.objecter _get_session s=0x7f350c00c4e0 osd=4 3 2024-04-14T15:18:28.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.867+0000 7f35321d0700 15 client.49126.objecter _session_command_op_remove -1 10 2024-04-14T15:18:28.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.867+0000 7f35321d0700 20 client.49126.objecter get_session s=0x7f350c00c4e0 osd=4 3 2024-04-14T15:18:28.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.867+0000 7f35321d0700 15 client.49126.objecter _session_command_op_assign 4 10 2024-04-14T15:18:28.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.867+0000 7f35321d0700 20 client.49126.objecter put_session s=0x7f350c00c4e0 osd=4 4 2024-04-14T15:18:28.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.867+0000 7f35321d0700 10 client.49126.objecter _send_command 10 2024-04-14T15:18:28.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.867+0000 7f35321d0700 1 -- 172.21.15.90:0/3282112766 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f350c01c180 con 0x7f350c011fa0 2024-04-14T15:18:28.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.869+0000 7f35297fa700 1 -- 172.21.15.90:0/3282112766 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f350c01c180 con 0x7f350c011fa0 2024-04-14T15:18:28.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.869+0000 7f35297fa700 10 client.49126.objecter ms_dispatch 0x7f352c001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:18:28.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.869+0000 7f35297fa700 10 client.49126.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:18:28.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.869+0000 7f35297fa700 20 client.49126.objecter put_session s=0x7f350c00c4e0 osd=4 4 2024-04-14T15:18:28.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.869+0000 7f35297fa700 15 client.49126.objecter _session_command_op_remove 4 10 2024-04-14T15:18:28.870 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:18:28.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.870+0000 7f35321d0700 10 client.49126.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:18:28.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.870+0000 7f35321d0700 15 client.49126.objecter _session_command_op_assign -1 11 2024-04-14T15:18:28.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.870+0000 7f35321d0700 20 client.49126.objecter _get_session s=0x7f350c00c770 osd=5 3 2024-04-14T15:18:28.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.870+0000 7f35321d0700 20 client.49126.objecter put_session s=0x7f350c00c770 osd=5 3 2024-04-14T15:18:28.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.870+0000 7f35321d0700 20 client.49126.objecter _get_session s=0x7f350c00c770 osd=5 3 2024-04-14T15:18:28.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.870+0000 7f35321d0700 15 client.49126.objecter _session_command_op_remove -1 11 2024-04-14T15:18:28.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.870+0000 7f35321d0700 20 client.49126.objecter get_session s=0x7f350c00c770 osd=5 3 2024-04-14T15:18:28.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.870+0000 7f35321d0700 15 client.49126.objecter _session_command_op_assign 5 11 2024-04-14T15:18:28.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.870+0000 7f35321d0700 20 client.49126.objecter put_session s=0x7f350c00c770 osd=5 4 2024-04-14T15:18:28.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.870+0000 7f35321d0700 10 client.49126.objecter _send_command 11 2024-04-14T15:18:28.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.870+0000 7f35321d0700 1 -- 172.21.15.90:0/3282112766 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f350c01cd20 con 0x7f350c01c8a0 2024-04-14T15:18:28.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.870+0000 7f35297fa700 10 client.49126.objecter ms_handle_connect 0x7f350c01c8a0 2024-04-14T15:18:28.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.871+0000 7f35297fa700 1 -- 172.21.15.90:0/3282112766 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f350c01cd20 con 0x7f350c01c8a0 2024-04-14T15:18:28.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.871+0000 7f35297fa700 10 client.49126.objecter ms_dispatch 0x7f352c001550 command_reply(tid 11: 0 ) 2024-04-14T15:18:28.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.871+0000 7f35297fa700 10 client.49126.objecter _finish_command 11 = system:0 2024-04-14T15:18:28.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.871+0000 7f35297fa700 20 client.49126.objecter put_session s=0x7f350c00c770 osd=5 4 2024-04-14T15:18:28.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.871+0000 7f35297fa700 15 client.49126.objecter _session_command_op_remove 5 11 2024-04-14T15:18:28.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.889+0000 7f35321d0700 10 client.49126.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:18:28.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.889+0000 7f35321d0700 15 client.49126.objecter _session_command_op_assign -1 12 2024-04-14T15:18:28.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.889+0000 7f35321d0700 20 client.49126.objecter _get_session s=0x7f350c00c770 osd=5 3 2024-04-14T15:18:28.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.889+0000 7f35321d0700 20 client.49126.objecter put_session s=0x7f350c00c770 osd=5 3 2024-04-14T15:18:28.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.889+0000 7f35321d0700 20 client.49126.objecter _get_session s=0x7f350c00c770 osd=5 3 2024-04-14T15:18:28.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.889+0000 7f35321d0700 15 client.49126.objecter _session_command_op_remove -1 12 2024-04-14T15:18:28.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.889+0000 7f35321d0700 20 client.49126.objecter get_session s=0x7f350c00c770 osd=5 3 2024-04-14T15:18:28.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.889+0000 7f35321d0700 15 client.49126.objecter _session_command_op_assign 5 12 2024-04-14T15:18:28.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.889+0000 7f35321d0700 20 client.49126.objecter put_session s=0x7f350c00c770 osd=5 4 2024-04-14T15:18:28.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.889+0000 7f35321d0700 10 client.49126.objecter _send_command 12 2024-04-14T15:18:28.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.889+0000 7f35321d0700 1 -- 172.21.15.90:0/3282112766 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f350c01cf00 con 0x7f350c01c8a0 2024-04-14T15:18:28.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.891+0000 7f35297fa700 1 -- 172.21.15.90:0/3282112766 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f350c01cf00 con 0x7f350c01c8a0 2024-04-14T15:18:28.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.891+0000 7f35297fa700 10 client.49126.objecter ms_dispatch 0x7f352c001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:18:28.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.891+0000 7f35297fa700 10 client.49126.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:18:28.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.891+0000 7f35297fa700 20 client.49126.objecter put_session s=0x7f350c00c770 osd=5 4 2024-04-14T15:18:28.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.891+0000 7f35297fa700 15 client.49126.objecter _session_command_op_remove 5 12 2024-04-14T15:18:28.892 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:18:28.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.891+0000 7f35321d0700 10 client.49126.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:18:28.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.891+0000 7f35321d0700 15 client.49126.objecter _session_command_op_assign -1 13 2024-04-14T15:18:28.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.891+0000 7f35321d0700 20 client.49126.objecter _get_session s=0x7f350c01cd20 osd=6 3 2024-04-14T15:18:28.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.891+0000 7f35321d0700 20 client.49126.objecter put_session s=0x7f350c01cd20 osd=6 3 2024-04-14T15:18:28.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.891+0000 7f35321d0700 20 client.49126.objecter _get_session s=0x7f350c01cd20 osd=6 3 2024-04-14T15:18:28.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.891+0000 7f35321d0700 15 client.49126.objecter _session_command_op_remove -1 13 2024-04-14T15:18:28.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.891+0000 7f35321d0700 20 client.49126.objecter get_session s=0x7f350c01cd20 osd=6 3 2024-04-14T15:18:28.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.891+0000 7f35321d0700 15 client.49126.objecter _session_command_op_assign 6 13 2024-04-14T15:18:28.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.891+0000 7f35321d0700 20 client.49126.objecter put_session s=0x7f350c01cd20 osd=6 4 2024-04-14T15:18:28.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.891+0000 7f35321d0700 10 client.49126.objecter _send_command 13 2024-04-14T15:18:28.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.891+0000 7f35321d0700 1 -- 172.21.15.90:0/3282112766 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f350c023170 con 0x7f350c01ce90 2024-04-14T15:18:28.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.891+0000 7f35297fa700 10 client.49126.objecter ms_handle_connect 0x7f350c01ce90 2024-04-14T15:18:28.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.892+0000 7f35297fa700 1 -- 172.21.15.90:0/3282112766 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f350c023170 con 0x7f350c01ce90 2024-04-14T15:18:28.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.892+0000 7f35297fa700 10 client.49126.objecter ms_dispatch 0x7f352c001550 command_reply(tid 13: 0 ) 2024-04-14T15:18:28.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.892+0000 7f35297fa700 10 client.49126.objecter _finish_command 13 = system:0 2024-04-14T15:18:28.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.892+0000 7f35297fa700 20 client.49126.objecter put_session s=0x7f350c01cd20 osd=6 4 2024-04-14T15:18:28.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.892+0000 7f35297fa700 15 client.49126.objecter _session_command_op_remove 6 13 2024-04-14T15:18:28.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.908+0000 7f35321d0700 10 client.49126.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:18:28.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.908+0000 7f35321d0700 15 client.49126.objecter _session_command_op_assign -1 14 2024-04-14T15:18:28.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.908+0000 7f35321d0700 20 client.49126.objecter _get_session s=0x7f350c01cd20 osd=6 3 2024-04-14T15:18:28.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.908+0000 7f35321d0700 20 client.49126.objecter put_session s=0x7f350c01cd20 osd=6 3 2024-04-14T15:18:28.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.908+0000 7f35321d0700 20 client.49126.objecter _get_session s=0x7f350c01cd20 osd=6 3 2024-04-14T15:18:28.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.908+0000 7f35321d0700 15 client.49126.objecter _session_command_op_remove -1 14 2024-04-14T15:18:28.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.908+0000 7f35321d0700 20 client.49126.objecter get_session s=0x7f350c01cd20 osd=6 3 2024-04-14T15:18:28.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.908+0000 7f35321d0700 15 client.49126.objecter _session_command_op_assign 6 14 2024-04-14T15:18:28.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.908+0000 7f35321d0700 20 client.49126.objecter put_session s=0x7f350c01cd20 osd=6 4 2024-04-14T15:18:28.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.908+0000 7f35321d0700 10 client.49126.objecter _send_command 14 2024-04-14T15:18:28.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.908+0000 7f35321d0700 1 -- 172.21.15.90:0/3282112766 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f350c00ca00 con 0x7f350c01ce90 2024-04-14T15:18:28.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.910+0000 7f35297fa700 1 -- 172.21.15.90:0/3282112766 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f350c00ca00 con 0x7f350c01ce90 2024-04-14T15:18:28.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.910+0000 7f35297fa700 10 client.49126.objecter ms_dispatch 0x7f352c001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:18:28.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.910+0000 7f35297fa700 10 client.49126.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:18:28.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.910+0000 7f35297fa700 20 client.49126.objecter put_session s=0x7f350c01cd20 osd=6 4 2024-04-14T15:18:28.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.910+0000 7f35297fa700 15 client.49126.objecter _session_command_op_remove 6 14 2024-04-14T15:18:28.911 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:18:28.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.910+0000 7f35321d0700 10 client.49126.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:18:28.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.910+0000 7f35321d0700 15 client.49126.objecter _session_command_op_assign -1 15 2024-04-14T15:18:28.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.910+0000 7f35321d0700 20 client.49126.objecter _get_session s=0x7f350c00ca00 osd=7 3 2024-04-14T15:18:28.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.910+0000 7f35321d0700 20 client.49126.objecter put_session s=0x7f350c00ca00 osd=7 3 2024-04-14T15:18:28.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.910+0000 7f35321d0700 20 client.49126.objecter _get_session s=0x7f350c00ca00 osd=7 3 2024-04-14T15:18:28.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.910+0000 7f35321d0700 15 client.49126.objecter _session_command_op_remove -1 15 2024-04-14T15:18:28.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.910+0000 7f35321d0700 20 client.49126.objecter get_session s=0x7f350c00ca00 osd=7 3 2024-04-14T15:18:28.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.910+0000 7f35321d0700 15 client.49126.objecter _session_command_op_assign 7 15 2024-04-14T15:18:28.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.910+0000 7f35321d0700 20 client.49126.objecter put_session s=0x7f350c00ca00 osd=7 4 2024-04-14T15:18:28.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.910+0000 7f35321d0700 10 client.49126.objecter _send_command 15 2024-04-14T15:18:28.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.910+0000 7f35321d0700 1 -- 172.21.15.90:0/3282112766 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f350c0170d0 con 0x7f350c0115c0 2024-04-14T15:18:28.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.911+0000 7f35297fa700 10 client.49126.objecter ms_handle_connect 0x7f350c0115c0 2024-04-14T15:18:28.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.912+0000 7f35297fa700 1 -- 172.21.15.90:0/3282112766 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f350c0170d0 con 0x7f350c0115c0 2024-04-14T15:18:28.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.912+0000 7f35297fa700 10 client.49126.objecter ms_dispatch 0x7f352c001550 command_reply(tid 15: 0 ) 2024-04-14T15:18:28.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.912+0000 7f35297fa700 10 client.49126.objecter _finish_command 15 = system:0 2024-04-14T15:18:28.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.912+0000 7f35297fa700 20 client.49126.objecter put_session s=0x7f350c00ca00 osd=7 4 2024-04-14T15:18:28.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.912+0000 7f35297fa700 15 client.49126.objecter _session_command_op_remove 7 15 2024-04-14T15:18:28.923 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:18:28.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.928+0000 7f35321d0700 10 client.49126.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:18:28.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.928+0000 7f35321d0700 15 client.49126.objecter _session_command_op_assign -1 16 2024-04-14T15:18:28.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.928+0000 7f35321d0700 20 client.49126.objecter _get_session s=0x7f350c00ca00 osd=7 3 2024-04-14T15:18:28.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.928+0000 7f35321d0700 20 client.49126.objecter put_session s=0x7f350c00ca00 osd=7 3 2024-04-14T15:18:28.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.928+0000 7f35321d0700 20 client.49126.objecter _get_session s=0x7f350c00ca00 osd=7 3 2024-04-14T15:18:28.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.928+0000 7f35321d0700 15 client.49126.objecter _session_command_op_remove -1 16 2024-04-14T15:18:28.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.928+0000 7f35321d0700 20 client.49126.objecter get_session s=0x7f350c00ca00 osd=7 3 2024-04-14T15:18:28.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.928+0000 7f35321d0700 15 client.49126.objecter _session_command_op_assign 7 16 2024-04-14T15:18:28.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.928+0000 7f35321d0700 20 client.49126.objecter put_session s=0x7f350c00ca00 osd=7 4 2024-04-14T15:18:28.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.928+0000 7f35321d0700 10 client.49126.objecter _send_command 16 2024-04-14T15:18:28.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.928+0000 7f35321d0700 1 -- 172.21.15.90:0/3282112766 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f350c0170d0 con 0x7f350c0115c0 2024-04-14T15:18:28.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.930+0000 7f35297fa700 1 -- 172.21.15.90:0/3282112766 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f350c0170d0 con 0x7f350c0115c0 2024-04-14T15:18:28.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.930+0000 7f35297fa700 10 client.49126.objecter ms_dispatch 0x7f352c001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:18:28.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.930+0000 7f35297fa700 10 client.49126.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:18:28.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.930+0000 7f35297fa700 20 client.49126.objecter put_session s=0x7f350c00ca00 osd=7 4 2024-04-14T15:18:28.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.930+0000 7f35297fa700 15 client.49126.objecter _session_command_op_remove 7 16 2024-04-14T15:18:28.931 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:18:28.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.930+0000 7f35321d0700 10 librados: watch_flush enter 2024-04-14T15:18:28.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.930+0000 7f35321d0700 10 librados: watch_flush exit 2024-04-14T15:18:28.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.930+0000 7f35321d0700 10 client.49126.objecter close_session for osd.0 2024-04-14T15:18:28.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.930+0000 7f35321d0700 1 -- 172.21.15.90:0/3282112766 >> v1:172.21.15.90:6801/3761806885 conn(0x7f350c004760 legacy=0x7f350c002780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:28.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.930+0000 7f35321d0700 20 client.49126.objecter put_session s=0x7f350c005410 osd=0 1 2024-04-14T15:18:28.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.930+0000 7f35321d0700 10 client.49126.objecter close_session for osd.1 2024-04-14T15:18:28.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.930+0000 7f35321d0700 1 -- 172.21.15.90:0/3282112766 >> v1:172.21.15.90:6805/4170361699 conn(0x7f350c0042b0 legacy=0x7f350c00ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:28.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.930+0000 7f35321d0700 20 client.49126.objecter put_session s=0x7f350c004140 osd=1 1 2024-04-14T15:18:28.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.930+0000 7f35321d0700 10 client.49126.objecter close_session for osd.2 2024-04-14T15:18:28.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.930+0000 7f35321d0700 1 -- 172.21.15.90:0/3282112766 >> v1:172.21.15.90:6809/1686184251 conn(0x7f350c00d600 legacy=0x7f350c00f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:28.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.930+0000 7f35321d0700 20 client.49126.objecter put_session s=0x7f350c0083a0 osd=2 1 2024-04-14T15:18:28.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.930+0000 7f35321d0700 10 client.49126.objecter close_session for osd.3 2024-04-14T15:18:28.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.930+0000 7f35321d0700 1 -- 172.21.15.90:0/3282112766 >> v1:172.21.15.90:6813/2270166774 conn(0x7f350c012440 legacy=0x7f350c014830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:28.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.930+0000 7f35321d0700 20 client.49126.objecter put_session s=0x7f350c00cc50 osd=3 1 2024-04-14T15:18:28.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.930+0000 7f35321d0700 10 client.49126.objecter close_session for osd.4 2024-04-14T15:18:28.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.930+0000 7f35321d0700 1 -- 172.21.15.90:0/3282112766 >> v1:172.21.15.92:6800/3887549733 conn(0x7f350c011fa0 legacy=0x7f350c019290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:28.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.930+0000 7f35321d0700 20 client.49126.objecter put_session s=0x7f350c00c4e0 osd=4 1 2024-04-14T15:18:28.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.930+0000 7f35321d0700 10 client.49126.objecter close_session for osd.5 2024-04-14T15:18:28.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.930+0000 7f35321d0700 1 -- 172.21.15.90:0/3282112766 >> v1:172.21.15.92:6804/3244561440 conn(0x7f350c01c8a0 legacy=0x7f350c01a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:28.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.930+0000 7f35321d0700 20 client.49126.objecter put_session s=0x7f350c00c770 osd=5 1 2024-04-14T15:18:28.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.930+0000 7f35321d0700 10 client.49126.objecter close_session for osd.6 2024-04-14T15:18:28.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.930+0000 7f35321d0700 1 -- 172.21.15.90:0/3282112766 >> v1:172.21.15.92:6808/636125963 conn(0x7f350c01ce90 legacy=0x7f350c021880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:28.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.930+0000 7f35321d0700 20 client.49126.objecter put_session s=0x7f350c01cd20 osd=6 1 2024-04-14T15:18:28.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.930+0000 7f35321d0700 10 client.49126.objecter close_session for osd.7 2024-04-14T15:18:28.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.930+0000 7f35321d0700 1 -- 172.21.15.90:0/3282112766 >> v1:172.21.15.92:6812/1773706716 conn(0x7f350c0115c0 legacy=0x7f350c0168d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:28.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.930+0000 7f35321d0700 20 client.49126.objecter put_session s=0x7f350c00ca00 osd=7 1 2024-04-14T15:18:28.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.930+0000 7f35321d0700 20 client.49126.objecter shutdown clearing up homeless session... 2024-04-14T15:18:28.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.930+0000 7f35321d0700 10 client.49126.objecter successfully canceled tick 2024-04-14T15:18:28.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.930+0000 7f352bfff700 1 -- 172.21.15.90:0/3282112766 reap_dead start 2024-04-14T15:18:28.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.930+0000 7f352bfff700 1 -- 172.21.15.90:0/3282112766 reap_dead start 2024-04-14T15:18:28.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.930+0000 7f352bfff700 1 -- 172.21.15.90:0/3282112766 reap_dead start 2024-04-14T15:18:28.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.931+0000 7f352bfff700 1 -- 172.21.15.90:0/3282112766 reap_dead start 2024-04-14T15:18:28.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.931+0000 7f35321d0700 1 -- 172.21.15.90:0/3282112766 >> v1:172.21.15.90:6800/420935427 conn(0x7f35180714d0 legacy=0x7f3518073990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:28.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.931+0000 7f352bfff700 1 -- 172.21.15.90:0/3282112766 reap_dead start 2024-04-14T15:18:28.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.931+0000 7f35321d0700 1 -- 172.21.15.90:0/3282112766 >> v1:172.21.15.92:6789/0 conn(0x7f352c10e200 legacy=0x7f352c1a8ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:28.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.931+0000 7f35321d0700 1 -- 172.21.15.90:0/3282112766 shutdown_connections 2024-04-14T15:18:28.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.931+0000 7f35321d0700 1 -- 172.21.15.90:0/3282112766 >> 172.21.15.90:0/3282112766 conn(0x7f352c0fd2b0 msgr2=0x7f352c111670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:18:28.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.931+0000 7f35321d0700 1 -- 172.21.15.90:0/3282112766 shutdown_connections 2024-04-14T15:18:28.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.931+0000 7f35321d0700 1 -- 172.21.15.90:0/3282112766 wait complete. 2024-04-14T15:18:28.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:28.931+0000 7f35321d0700 1 librados: shutdown 2024-04-14T15:18:28.935 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:18:28.935 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:18:28.935 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:18:28.935 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:18:28.935 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:18:28.935 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:18:28.935 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:18:28.936 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:18:28.948 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:28.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:28.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:18:29.049 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:29.074 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:29.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:29.075 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:29.176 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:18:29.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:29 smithi090 ceph-mon[30399]: 2.1a scrub starts 2024-04-14T15:18:29.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:29 smithi090 ceph-mon[30399]: 2.1a scrub ok 2024-04-14T15:18:29.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:29 smithi090 ceph-mon[30399]: osdmap e427: 8 total, 8 up, 8 in 2024-04-14T15:18:29.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:29 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3282112766' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:18:29.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:29 smithi090 ceph-mon[41670]: 2.1a scrub starts 2024-04-14T15:18:29.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:29 smithi090 ceph-mon[41670]: 2.1a scrub ok 2024-04-14T15:18:29.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:29 smithi090 ceph-mon[41670]: osdmap e427: 8 total, 8 up, 8 in 2024-04-14T15:18:29.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:29 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3282112766' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:18:29.199 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:29.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:29.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:18:29.222 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:29 smithi092 ceph-mon[33966]: 2.1a scrub starts 2024-04-14T15:18:29.222 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:29 smithi092 ceph-mon[33966]: 2.1a scrub ok 2024-04-14T15:18:29.222 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:29 smithi092 ceph-mon[33966]: osdmap e427: 8 total, 8 up, 8 in 2024-04-14T15:18:29.222 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:29 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3282112766' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:18:29.301 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:18:29.324 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:29.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:29.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:18:29.426 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:18:29.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:29.451 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:29.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:18:29.552 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:18:29.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:29.576 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:29.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:18:29.677 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:18:29.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:29.702 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:29.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:18:29.759 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-04-14T15:18:29.759 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:18:29.803 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:18:29.828 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:29.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:29.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:18:29.930 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:18:29.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:29.954 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:29.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:18:29.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:29.962+0000 7f44cdcbf700 1 -- 172.21.15.90:0/1035471041 >> v1:172.21.15.90:6789/0 conn(0x7f44c810b720 legacy=0x7f44c810db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:29.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:29.962+0000 7f44cdcbf700 1 -- 172.21.15.90:0/1035471041 shutdown_connections 2024-04-14T15:18:29.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:29.962+0000 7f44cdcbf700 1 -- 172.21.15.90:0/1035471041 >> 172.21.15.90:0/1035471041 conn(0x7f44c80fe3a0 msgr2=0x7f44c81007c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:18:29.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:29.962+0000 7f44cdcbf700 1 -- 172.21.15.90:0/1035471041 shutdown_connections 2024-04-14T15:18:29.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:29.962+0000 7f44cdcbf700 1 -- 172.21.15.90:0/1035471041 wait complete. 2024-04-14T15:18:29.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:29.962+0000 7f44cdcbf700 1 librados: starting msgr at 2024-04-14T15:18:29.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:29.962+0000 7f44cdcbf700 1 librados: starting objecter 2024-04-14T15:18:29.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:29.963+0000 7f44cdcbf700 1 Processor -- start 2024-04-14T15:18:29.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:29.963+0000 7f44cdcbf700 1 -- start start 2024-04-14T15:18:29.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:29.963+0000 7f44cdcbf700 1 librados: setting wanted keys 2024-04-14T15:18:29.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:29.963+0000 7f44cdcbf700 1 librados: calling monclient init 2024-04-14T15:18:29.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:29.963+0000 7f44cdcbf700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f44c810f060 con 0x7f44c810f2f0 2024-04-14T15:18:29.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:29.963+0000 7f44cdcbf700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f44c81a8f00 con 0x7f44c8108870 2024-04-14T15:18:29.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:29.963+0000 7f44cdcbf700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f44c81aa0e0 con 0x7f44c810b720 2024-04-14T15:18:29.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:29.963+0000 7f44c77fe700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f44c8108870 0x7f44c810d8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.90:60884/0 (socket says 172.21.15.90:60884) 2024-04-14T15:18:29.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:29.963+0000 7f44c77fe700 1 -- 172.21.15.90:0/578090164 learned_addr learned my addr 172.21.15.90:0/578090164 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:18:29.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:29.963+0000 7f44c7fff700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f44c810f2f0 0x7f44c81a67d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:49018/0 (socket says 172.21.15.90:49018) 2024-04-14T15:18:29.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:29.963+0000 7f44c4ff9700 10 client.?.objecter ms_handle_connect 0x7f44c810f2f0 2024-04-14T15:18:29.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:29.963+0000 7f44c4ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:18:29.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:29.963+0000 7f44c4ff9700 10 client.?.objecter ms_handle_connect 0x7f44c8108870 2024-04-14T15:18:29.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:29.963+0000 7f44c4ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:18:29.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:29.963+0000 7f44c4ff9700 10 client.?.objecter ms_handle_connect 0x7f44c810b720 2024-04-14T15:18:29.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:29.963+0000 7f44c4ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:18:29.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:29.963+0000 7f44c4ff9700 1 -- 172.21.15.90:0/578090164 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3166411915 0 0) 0x7f44c81a8f00 con 0x7f44c8108870 2024-04-14T15:18:29.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:29.963+0000 7f44c4ff9700 1 -- 172.21.15.90:0/578090164 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f44b00035b0 con 0x7f44c8108870 2024-04-14T15:18:29.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:29.963+0000 7f44c4ff9700 1 -- 172.21.15.90:0/578090164 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1992073137 0 0) 0x7f44c810f060 con 0x7f44c810f2f0 2024-04-14T15:18:29.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:29.963+0000 7f44c4ff9700 1 -- 172.21.15.90:0/578090164 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f44c81a8f00 con 0x7f44c810f2f0 2024-04-14T15:18:29.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:29.963+0000 7f44c4ff9700 1 -- 172.21.15.90:0/578090164 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3725724592 0 0) 0x7f44c81aa0e0 con 0x7f44c810b720 2024-04-14T15:18:29.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:29.963+0000 7f44c4ff9700 1 -- 172.21.15.90:0/578090164 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f44c810f060 con 0x7f44c810b720 2024-04-14T15:18:29.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:29.964+0000 7f44c4ff9700 1 -- 172.21.15.90:0/578090164 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3734874581 0 0) 0x7f44b00035b0 con 0x7f44c8108870 2024-04-14T15:18:29.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:29.964+0000 7f44c4ff9700 1 -- 172.21.15.90:0/578090164 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f44c81aa0e0 con 0x7f44c8108870 2024-04-14T15:18:29.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:29.964+0000 7f44c4ff9700 1 -- 172.21.15.90:0/578090164 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f44b80018d0 con 0x7f44c8108870 2024-04-14T15:18:29.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:29.964+0000 7f44c4ff9700 1 -- 172.21.15.90:0/578090164 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1732333288 0 0) 0x7f44c81a8f00 con 0x7f44c810f2f0 2024-04-14T15:18:29.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:29.964+0000 7f44c4ff9700 1 -- 172.21.15.90:0/578090164 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f44b00035b0 con 0x7f44c810f2f0 2024-04-14T15:18:29.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:29.964+0000 7f44c4ff9700 1 -- 172.21.15.90:0/578090164 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f44c0001af0 con 0x7f44c810f2f0 2024-04-14T15:18:29.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:29.964+0000 7f44c4ff9700 1 -- 172.21.15.90:0/578090164 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1564649674 0 0) 0x7f44c810f060 con 0x7f44c810b720 2024-04-14T15:18:29.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:29.964+0000 7f44c4ff9700 1 -- 172.21.15.90:0/578090164 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f44c81a8f00 con 0x7f44c810b720 2024-04-14T15:18:29.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:29.964+0000 7f44c4ff9700 1 -- 172.21.15.90:0/578090164 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f44bc001020 con 0x7f44c810b720 2024-04-14T15:18:29.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:29.964+0000 7f44c4ff9700 1 -- 172.21.15.90:0/578090164 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2620177744 0 0) 0x7f44c81aa0e0 con 0x7f44c8108870 2024-04-14T15:18:29.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:29.964+0000 7f44c4ff9700 1 -- 172.21.15.90:0/578090164 >> v1:172.21.15.90:6790/0 conn(0x7f44c810b720 legacy=0x7f44c810e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:29.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:29.964+0000 7f44c4ff9700 1 -- 172.21.15.90:0/578090164 >> v1:172.21.15.90:6789/0 conn(0x7f44c810f2f0 legacy=0x7f44c81a67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:29.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:29.964+0000 7f44c4ff9700 1 -- 172.21.15.90:0/578090164 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44c81ab300 con 0x7f44c8108870 2024-04-14T15:18:29.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:29.964+0000 7f44cdcbf700 1 -- 172.21.15.90:0/578090164 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f44c81a9130 con 0x7f44c8108870 2024-04-14T15:18:29.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:29.964+0000 7f44cdcbf700 10 client.49132.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:18:29.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:29.964+0000 7f44cdcbf700 1 -- 172.21.15.90:0/578090164 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f44c81a95f0 con 0x7f44c8108870 2024-04-14T15:18:29.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:29.964+0000 7f44cdcbf700 1 librados: init done 2024-04-14T15:18:29.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:29.964+0000 7f44c4ff9700 1 -- 172.21.15.90:0/578090164 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f44b8003b40 con 0x7f44c8108870 2024-04-14T15:18:29.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:29.964+0000 7f44c4ff9700 1 -- 172.21.15.90:0/578090164 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f44b8005ce0 con 0x7f44c8108870 2024-04-14T15:18:29.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:29.965+0000 7f44cdcbf700 1 -- 172.21.15.90:0/578090164 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44b4005150 con 0x7f44c8108870 2024-04-14T15:18:29.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:29.965+0000 7f44c4ff9700 1 -- 172.21.15.90:0/578090164 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f44b801c950 con 0x7f44c8108870 2024-04-14T15:18:29.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:29.966+0000 7f44c4ff9700 1 -- 172.21.15.90:0/578090164 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(427..427 src has 51..427) ==== 8216+0+0 (unknown 1026895677 0 0) 0x7f44b808aa70 con 0x7f44c8108870 2024-04-14T15:18:29.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:29.966+0000 7f44c4ff9700 10 client.49132.objecter ms_dispatch 0x7f44c8001550 osd_map(427..427 src has 51..427) 2024-04-14T15:18:29.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:29.966+0000 7f44c4ff9700 3 client.49132.objecter handle_osd_map got epochs [427,427] > 0 2024-04-14T15:18:29.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:29.966+0000 7f44c4ff9700 3 client.49132.objecter handle_osd_map decoding full epoch 427 2024-04-14T15:18:29.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:29.966+0000 7f44c4ff9700 20 client.49132.objecter dump_active .. 0 homeless 2024-04-14T15:18:29.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:29.966+0000 7f44c4ff9700 10 client.49132.objecter ms_handle_connect 0x7f44b0071420 2024-04-14T15:18:29.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:29.971+0000 7f44c4ff9700 1 -- 172.21.15.90:0/578090164 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f44b8057450 con 0x7f44c8108870 2024-04-14T15:18:30.056 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:30.079 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:30.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:30.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:30.146 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:30 smithi090 ceph-mon[41670]: 5.1f scrub starts 2024-04-14T15:18:30.147 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:30 smithi090 ceph-mon[41670]: 5.1f scrub ok 2024-04-14T15:18:30.147 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:30 smithi090 ceph-mon[41670]: pgmap v594: 238 pgs: 238 active+clean; 8.4 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-14T15:18:30.147 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:30 smithi090 ceph-mon[30399]: 5.1f scrub starts 2024-04-14T15:18:30.147 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:30 smithi090 ceph-mon[30399]: 5.1f scrub ok 2024-04-14T15:18:30.147 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:30 smithi090 ceph-mon[30399]: pgmap v594: 238 pgs: 238 active+clean; 8.4 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-14T15:18:30.180 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:18:30.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:30.191+0000 7f44cdcbf700 1 -- 172.21.15.90:0/578090164 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7f44b4005440 con 0x7f44c8108870 2024-04-14T15:18:30.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:30.205 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:30.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:18:30.306 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:18:30.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:30.330 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:30.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:18:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:30 smithi092 ceph-mon[33966]: 5.1f scrub starts 2024-04-14T15:18:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:30 smithi092 ceph-mon[33966]: 5.1f scrub ok 2024-04-14T15:18:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:30 smithi092 ceph-mon[33966]: pgmap v594: 238 pgs: 238 active+clean; 8.4 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-14T15:18:30.431 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-14T15:18:30.431 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:18:30.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:30.455 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:30.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:18:30.556 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:18:30.580 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:30.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:30.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:18:30.670 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:18:30.681 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:18:30.705 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:30.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:30.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:18:30.807 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:18:30.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:30.831 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:30.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:18:30.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:30.897+0000 7f44c4ff9700 1 -- 172.21.15.90:0/578090164 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v428) ==== 101+0+0 (unknown 986728105 0 0) 0x7f44b805b100 con 0x7f44c8108870 2024-04-14T15:18:30.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:30.900+0000 7f44cdcbf700 10 librados: watch_flush enter 2024-04-14T15:18:30.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:30.900+0000 7f44cdcbf700 10 librados: watch_flush exit 2024-04-14T15:18:30.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:30.900+0000 7f44cdcbf700 20 client.49132.objecter shutdown clearing up homeless session... 2024-04-14T15:18:30.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:30.900+0000 7f44cdcbf700 10 client.49132.objecter successfully canceled tick 2024-04-14T15:18:30.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:30.900+0000 7f44cdcbf700 1 -- 172.21.15.90:0/578090164 >> v1:172.21.15.90:6800/420935427 conn(0x7f44b0071420 legacy=0x7f44b00738e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:30.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:30.901+0000 7f44cdcbf700 1 -- 172.21.15.90:0/578090164 >> v1:172.21.15.92:6789/0 conn(0x7f44c8108870 legacy=0x7f44c810d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:30.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:30.901+0000 7f44cdcbf700 1 -- 172.21.15.90:0/578090164 shutdown_connections 2024-04-14T15:18:30.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:30.901+0000 7f44cdcbf700 1 -- 172.21.15.90:0/578090164 >> 172.21.15.90:0/578090164 conn(0x7f44c80fe3a0 msgr2=0x7f44c8112870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:18:30.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:30.901+0000 7f44cdcbf700 1 -- 172.21.15.90:0/578090164 shutdown_connections 2024-04-14T15:18:30.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:30.901+0000 7f44cdcbf700 1 -- 172.21.15.90:0/578090164 wait complete. 2024-04-14T15:18:30.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:30.901+0000 7f44cdcbf700 1 librados: shutdown 2024-04-14T15:18:30.903 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is unset 2024-04-14T15:18:30.933 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:18:30.956 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:30.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:30.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:18:31.049 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:31 smithi092 ceph-mon[33966]: 5.11 scrub starts 2024-04-14T15:18:31.049 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:31 smithi092 ceph-mon[33966]: 5.11 scrub ok 2024-04-14T15:18:31.049 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:31 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:18:31.050 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:31 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:18:31.050 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:31 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/578090164' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:18:31.050 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:31 smithi092 ceph-mon[33966]: from='client.49132 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:18:31.050 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:31 smithi092 ceph-mon[33966]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-14T15:18:31.050 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:31 smithi092 ceph-mon[33966]: from='client.49132 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-14T15:18:31.050 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:31 smithi092 ceph-mon[33966]: osdmap e428: 8 total, 8 up, 8 in 2024-04-14T15:18:31.058 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:18:31.082 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:31.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:31.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:18:31.183 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:18:31.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:31 smithi090 ceph-mon[41670]: 5.11 scrub starts 2024-04-14T15:18:31.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:31 smithi090 ceph-mon[41670]: 5.11 scrub ok 2024-04-14T15:18:31.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:31 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:18:31.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:31 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:18:31.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:31 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/578090164' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:18:31.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:31 smithi090 ceph-mon[41670]: from='client.49132 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:18:31.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:31 smithi090 ceph-mon[41670]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-14T15:18:31.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:31 smithi090 ceph-mon[41670]: from='client.49132 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-14T15:18:31.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:31 smithi090 ceph-mon[41670]: osdmap e428: 8 total, 8 up, 8 in 2024-04-14T15:18:31.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:31 smithi090 ceph-mon[30399]: 5.11 scrub starts 2024-04-14T15:18:31.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:31 smithi090 ceph-mon[30399]: 5.11 scrub ok 2024-04-14T15:18:31.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:31 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:18:31.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:31 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:18:31.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:31 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/578090164' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:18:31.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:31 smithi090 ceph-mon[30399]: from='client.49132 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:18:31.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:31 smithi090 ceph-mon[30399]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-14T15:18:31.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:31 smithi090 ceph-mon[30399]: from='client.49132 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-14T15:18:31.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:31 smithi090 ceph-mon[30399]: osdmap e428: 8 total, 8 up, 8 in 2024-04-14T15:18:31.207 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:31.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:31.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:18:31.309 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:18:31.332 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:31.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:31.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:18:31.434 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:18:31.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:31.459 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:31.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:18:31.560 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:18:31.583 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:31.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:31.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:18:31.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.627+0000 7ff9f0e6c700 1 -- 172.21.15.90:0/53829452 >> v1:172.21.15.90:6789/0 conn(0x7ff9ec10a610 legacy=0x7ff9ec10ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:31.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.627+0000 7ff9f0e6c700 1 -- 172.21.15.90:0/53829452 shutdown_connections 2024-04-14T15:18:31.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.627+0000 7ff9f0e6c700 1 -- 172.21.15.90:0/53829452 >> 172.21.15.90:0/53829452 conn(0x7ff9ec0fd290 msgr2=0x7ff9ec0ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:18:31.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.627+0000 7ff9f0e6c700 1 -- 172.21.15.90:0/53829452 shutdown_connections 2024-04-14T15:18:31.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.627+0000 7ff9f0e6c700 1 -- 172.21.15.90:0/53829452 wait complete. 2024-04-14T15:18:31.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.627+0000 7ff9f0e6c700 1 librados: starting msgr at 2024-04-14T15:18:31.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.628+0000 7ff9f0e6c700 1 librados: starting objecter 2024-04-14T15:18:31.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.628+0000 7ff9f0e6c700 1 Processor -- start 2024-04-14T15:18:31.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.628+0000 7ff9f0e6c700 1 -- start start 2024-04-14T15:18:31.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.628+0000 7ff9f0e6c700 1 librados: setting wanted keys 2024-04-14T15:18:31.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.628+0000 7ff9f0e6c700 1 librados: calling monclient init 2024-04-14T15:18:31.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.628+0000 7ff9f0e6c700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff9ec10de50 con 0x7ff9ec10e1e0 2024-04-14T15:18:31.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.628+0000 7ff9f0e6c700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff9ec1a7df0 con 0x7ff9ec107760 2024-04-14T15:18:31.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.628+0000 7ff9f0e6c700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff9ec1a8fd0 con 0x7ff9ec10a610 2024-04-14T15:18:31.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.628+0000 7ff9e9d9b700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7ff9ec10a610 0x7ff9ec10b3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:42776/0 (socket says 172.21.15.90:42776) 2024-04-14T15:18:31.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.628+0000 7ff9e9d9b700 1 -- 172.21.15.90:0/1811547561 learned_addr learned my addr 172.21.15.90:0/1811547561 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:18:31.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.628+0000 7ff9db7fe700 10 client.?.objecter ms_handle_connect 0x7ff9ec10e1e0 2024-04-14T15:18:31.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.628+0000 7ff9db7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:18:31.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.628+0000 7ff9db7fe700 10 client.?.objecter ms_handle_connect 0x7ff9ec107760 2024-04-14T15:18:31.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.628+0000 7ff9db7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:18:31.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.628+0000 7ff9db7fe700 10 client.?.objecter ms_handle_connect 0x7ff9ec10a610 2024-04-14T15:18:31.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.628+0000 7ff9db7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:18:31.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.628+0000 7ff9db7fe700 1 -- 172.21.15.90:0/1811547561 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1194878581 0 0) 0x7ff9ec10de50 con 0x7ff9ec10e1e0 2024-04-14T15:18:31.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.628+0000 7ff9db7fe700 1 -- 172.21.15.90:0/1811547561 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff9d40035b0 con 0x7ff9ec10e1e0 2024-04-14T15:18:31.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.628+0000 7ff9db7fe700 1 -- 172.21.15.90:0/1811547561 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1587117635 0 0) 0x7ff9ec1a7df0 con 0x7ff9ec107760 2024-04-14T15:18:31.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.629+0000 7ff9db7fe700 1 -- 172.21.15.90:0/1811547561 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff9ec10de50 con 0x7ff9ec107760 2024-04-14T15:18:31.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.629+0000 7ff9db7fe700 1 -- 172.21.15.90:0/1811547561 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 704483757 0 0) 0x7ff9ec1a8fd0 con 0x7ff9ec10a610 2024-04-14T15:18:31.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.629+0000 7ff9db7fe700 1 -- 172.21.15.90:0/1811547561 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff9ec1a7df0 con 0x7ff9ec10a610 2024-04-14T15:18:31.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.629+0000 7ff9db7fe700 1 -- 172.21.15.90:0/1811547561 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2689227935 0 0) 0x7ff9d40035b0 con 0x7ff9ec10e1e0 2024-04-14T15:18:31.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.629+0000 7ff9db7fe700 1 -- 172.21.15.90:0/1811547561 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff9ec1a8fd0 con 0x7ff9ec10e1e0 2024-04-14T15:18:31.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.629+0000 7ff9db7fe700 1 -- 172.21.15.90:0/1811547561 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7ff9e4001b70 con 0x7ff9ec10e1e0 2024-04-14T15:18:31.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.629+0000 7ff9db7fe700 1 -- 172.21.15.90:0/1811547561 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2580964014 0 0) 0x7ff9ec10de50 con 0x7ff9ec107760 2024-04-14T15:18:31.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.629+0000 7ff9db7fe700 1 -- 172.21.15.90:0/1811547561 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff9d40035b0 con 0x7ff9ec107760 2024-04-14T15:18:31.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.629+0000 7ff9db7fe700 1 -- 172.21.15.90:0/1811547561 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1723514887 0 0) 0x7ff9ec1a7df0 con 0x7ff9ec10a610 2024-04-14T15:18:31.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.629+0000 7ff9db7fe700 1 -- 172.21.15.90:0/1811547561 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff9ec10de50 con 0x7ff9ec10a610 2024-04-14T15:18:31.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.629+0000 7ff9db7fe700 1 -- 172.21.15.90:0/1811547561 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7ff9dc001850 con 0x7ff9ec107760 2024-04-14T15:18:31.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.629+0000 7ff9db7fe700 1 -- 172.21.15.90:0/1811547561 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7ff9e0001430 con 0x7ff9ec10a610 2024-04-14T15:18:31.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.629+0000 7ff9db7fe700 1 -- 172.21.15.90:0/1811547561 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 922149031 0 0) 0x7ff9ec1a8fd0 con 0x7ff9ec10e1e0 2024-04-14T15:18:31.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.629+0000 7ff9db7fe700 1 -- 172.21.15.90:0/1811547561 >> v1:172.21.15.90:6790/0 conn(0x7ff9ec10a610 legacy=0x7ff9ec10b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:31.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.629+0000 7ff9db7fe700 1 -- 172.21.15.90:0/1811547561 >> v1:172.21.15.92:6789/0 conn(0x7ff9ec107760 legacy=0x7ff9ec10aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:31.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.629+0000 7ff9db7fe700 1 -- 172.21.15.90:0/1811547561 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9ec1aa1b0 con 0x7ff9ec10e1e0 2024-04-14T15:18:31.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.629+0000 7ff9f0e6c700 1 -- 172.21.15.90:0/1811547561 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff9ec1a8020 con 0x7ff9ec10e1e0 2024-04-14T15:18:31.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.629+0000 7ff9f0e6c700 10 client.37299.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:18:31.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.629+0000 7ff9f0e6c700 1 -- 172.21.15.90:0/1811547561 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff9ec1a84e0 con 0x7ff9ec10e1e0 2024-04-14T15:18:31.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.629+0000 7ff9f0e6c700 1 librados: init done 2024-04-14T15:18:31.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.630+0000 7ff9db7fe700 1 -- 172.21.15.90:0/1811547561 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7ff9e4003ea0 con 0x7ff9ec10e1e0 2024-04-14T15:18:31.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.630+0000 7ff9db7fe700 1 -- 172.21.15.90:0/1811547561 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7ff9e4004dc0 con 0x7ff9ec10e1e0 2024-04-14T15:18:31.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.630+0000 7ff9f0e6c700 1 -- 172.21.15.90:0/1811547561 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9cc005120 con 0x7ff9ec10e1e0 2024-04-14T15:18:31.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.631+0000 7ff9db7fe700 1 -- 172.21.15.90:0/1811547561 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7ff9e401b7d0 con 0x7ff9ec10e1e0 2024-04-14T15:18:31.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.631+0000 7ff9db7fe700 1 -- 172.21.15.90:0/1811547561 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(428..428 src has 51..428) ==== 8188+0+0 (unknown 1597913706 0 0) 0x7ff9e40899d0 con 0x7ff9ec10e1e0 2024-04-14T15:18:31.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.631+0000 7ff9db7fe700 10 client.37299.objecter ms_dispatch 0x7ff9ec001550 osd_map(428..428 src has 51..428) 2024-04-14T15:18:31.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.631+0000 7ff9db7fe700 3 client.37299.objecter handle_osd_map got epochs [428,428] > 0 2024-04-14T15:18:31.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.631+0000 7ff9db7fe700 3 client.37299.objecter handle_osd_map decoding full epoch 428 2024-04-14T15:18:31.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.632+0000 7ff9db7fe700 20 client.37299.objecter dump_active .. 0 homeless 2024-04-14T15:18:31.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.632+0000 7ff9db7fe700 10 client.37299.objecter ms_handle_connect 0x7ff9d40735c0 2024-04-14T15:18:31.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.636+0000 7ff9db7fe700 1 -- 172.21.15.90:0/1811547561 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7ff9e4056430 con 0x7ff9ec10e1e0 2024-04-14T15:18:31.685 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:31.708 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:31.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:31.709 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:31.810 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:18:31.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:31.836 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:31.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:18:31.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.837+0000 7ff9f0e6c700 1 -- 172.21.15.90:0/1811547561 --> v1:172.21.15.90:6800/420935427 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff9cc000b90 con 0x7ff9d40735c0 2024-04-14T15:18:31.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.853+0000 7ff9db7fe700 1 -- 172.21.15.90:0/1811547561 <== mgr.14610 v1:172.21.15.90:6800/420935427 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+611710 (unknown 2965378022 0 1918123848) 0x7ff9cc000b90 con 0x7ff9d40735c0 2024-04-14T15:18:31.867 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:18:31.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.857+0000 7ff9f0e6c700 10 librados: watch_flush enter 2024-04-14T15:18:31.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.857+0000 7ff9f0e6c700 10 librados: watch_flush exit 2024-04-14T15:18:31.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.857+0000 7ff9f0e6c700 20 client.37299.objecter shutdown clearing up homeless session... 2024-04-14T15:18:31.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.857+0000 7ff9f0e6c700 10 client.37299.objecter successfully canceled tick 2024-04-14T15:18:31.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.857+0000 7ff9f0e6c700 1 -- 172.21.15.90:0/1811547561 >> v1:172.21.15.90:6800/420935427 conn(0x7ff9d40735c0 legacy=0x7ff9d4075a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:31.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.857+0000 7ff9f0e6c700 1 -- 172.21.15.90:0/1811547561 >> v1:172.21.15.90:6789/0 conn(0x7ff9ec10e1e0 legacy=0x7ff9ec1a56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:31.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.857+0000 7ff9f0e6c700 1 -- 172.21.15.90:0/1811547561 shutdown_connections 2024-04-14T15:18:31.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.857+0000 7ff9f0e6c700 1 -- 172.21.15.90:0/1811547561 >> 172.21.15.90:0/1811547561 conn(0x7ff9ec0fd290 msgr2=0x7ff9ec111700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:18:31.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.858+0000 7ff9f0e6c700 1 -- 172.21.15.90:0/1811547561 shutdown_connections 2024-04-14T15:18:31.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.858+0000 7ff9f0e6c700 1 -- 172.21.15.90:0/1811547561 wait complete. 2024-04-14T15:18:31.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:31.858+0000 7ff9f0e6c700 1 librados: shutdown 2024-04-14T15:18:31.876 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-14T15:18:31.937 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:31.961 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:31.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:31.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:32.063 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:32.088 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:32.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:32.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:32.190 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:18:32.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:32 smithi090 ceph-mon[30399]: 11.1a scrub starts 2024-04-14T15:18:32.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:32 smithi090 ceph-mon[30399]: 11.1a scrub ok 2024-04-14T15:18:32.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:32 smithi090 ceph-mon[30399]: pgmap v596: 238 pgs: 238 active+clean; 8.4 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-14T15:18:32.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:32 smithi090 ceph-mon[30399]: 110.14 scrub starts 2024-04-14T15:18:32.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:32 smithi090 ceph-mon[30399]: 110.14 scrub ok 2024-04-14T15:18:32.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:32 smithi090 ceph-mon[30399]: from='client.37299 v1:172.21.15.90:0/1811547561' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:18:32.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:32 smithi090 ceph-mon[41670]: 11.1a scrub starts 2024-04-14T15:18:32.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:32 smithi090 ceph-mon[41670]: 11.1a scrub ok 2024-04-14T15:18:32.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:32 smithi090 ceph-mon[41670]: pgmap v596: 238 pgs: 238 active+clean; 8.4 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-14T15:18:32.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:32 smithi090 ceph-mon[41670]: 110.14 scrub starts 2024-04-14T15:18:32.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:32 smithi090 ceph-mon[41670]: 110.14 scrub ok 2024-04-14T15:18:32.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:32 smithi090 ceph-mon[41670]: from='client.37299 v1:172.21.15.90:0/1811547561' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:18:32.213 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:32.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:32.214 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:18:32.315 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:18:32.332 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:32 smithi092 ceph-mon[33966]: 11.1a scrub starts 2024-04-14T15:18:32.332 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:32 smithi092 ceph-mon[33966]: 11.1a scrub ok 2024-04-14T15:18:32.332 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:32 smithi092 ceph-mon[33966]: pgmap v596: 238 pgs: 238 active+clean; 8.4 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-14T15:18:32.332 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:32 smithi092 ceph-mon[33966]: 110.14 scrub starts 2024-04-14T15:18:32.332 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:32 smithi092 ceph-mon[33966]: 110.14 scrub ok 2024-04-14T15:18:32.332 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:32 smithi092 ceph-mon[33966]: from='client.37299 v1:172.21.15.90:0/1811547561' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:18:32.338 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:32.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:32.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:18:32.367 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-04-14T15:18:32.367 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:18:32.441 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:18:32.464 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:32.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:32.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:18:32.566 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:18:32.590 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:32.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:32.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:18:32.692 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:18:32.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:32.718 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:32.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:18:32.819 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:18:32.842 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:32.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:32.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:18:32.925 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-14T15:18:32.944 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:18:32.962 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-14T15:18:32.991 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:32.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:32.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:18:33.060 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":596,"stamp":"2024-04-14T15:18:31.078863+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8792197,"num_objects":362,"num_object_clones":3,"num_object_copies":1086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":362,"num_whiteouts":0,"num_read":1365,"num_read_kb":1015,"num_write":1054,"num_write_kb":12744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":8,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3902,"ondisk_log_size":3902,"up":714,"acting":714,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":714,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":598356,"kb_used_data":83676,"kb_used_omap":12,"kb_used_meta":514483,"kb_avail":749362860,"statfs":{"total":767960285184,"available":767347568640,"internally_reserved":0,"allocated":85684224,"data_stored":59404208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":526831154},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-7,"num_objects":-2,"num_object_clones":-1,"num_object_copies":-6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002279"},"pg_stats":[{"pgid":"297.1f","version":"0'0","reported_seq":27,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.024246+0000","last_change":"2024-04-14T15:18:13.005585+0000","last_active":"2024-04-14T15:18:28.024246+0000","last_peered":"2024-04-14T15:18:28.024246+0000","last_clean":"2024-04-14T15:18:28.024246+0000","last_became_active":"2024-04-14T15:18:13.005337+0000","last_became_peered":"2024-04-14T15:18:13.005337+0000","last_unstale":"2024-04-14T15:18:28.024246+0000","last_undegraded":"2024-04-14T15:18:28.024246+0000","last_fullsized":"2024-04-14T15:18:28.024246+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:21.146893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"297.1e","version":"0'0","reported_seq":27,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.205590+0000","last_change":"2024-04-14T15:18:13.005874+0000","last_active":"2024-04-14T15:18:28.205590+0000","last_peered":"2024-04-14T15:18:28.205590+0000","last_clean":"2024-04-14T15:18:28.205590+0000","last_became_active":"2024-04-14T15:18:13.005571+0000","last_became_peered":"2024-04-14T15:18:13.005571+0000","last_unstale":"2024-04-14T15:18:28.205590+0000","last_undegraded":"2024-04-14T15:18:28.205590+0000","last_fullsized":"2024-04-14T15:18:28.205590+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:23.548046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"297.1d","version":"0'0","reported_seq":27,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.020850+0000","last_change":"2024-04-14T15:18:12.947350+0000","last_active":"2024-04-14T15:18:28.020850+0000","last_peered":"2024-04-14T15:18:28.020850+0000","last_clean":"2024-04-14T15:18:28.020850+0000","last_became_active":"2024-04-14T15:18:12.947006+0000","last_became_peered":"2024-04-14T15:18:12.947006+0000","last_unstale":"2024-04-14T15:18:28.020850+0000","last_undegraded":"2024-04-14T15:18:28.020850+0000","last_fullsized":"2024-04-14T15:18:28.020850+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:27.508637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"297.1c","version":"427'5","reported_seq":35,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.020635+0000","last_change":"2024-04-14T15:18:12.955879+0000","last_active":"2024-04-14T15:18:28.020635+0000","last_peered":"2024-04-14T15:18:28.020635+0000","last_clean":"2024-04-14T15:18:28.020635+0000","last_became_active":"2024-04-14T15:18:12.955448+0000","last_became_peered":"2024-04-14T15:18:12.955448+0000","last_unstale":"2024-04-14T15:18:28.020635+0000","last_undegraded":"2024-04-14T15:18:28.020635+0000","last_fullsized":"2024-04-14T15:18:28.020635+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:23.726057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"297.1b","version":"0'0","reported_seq":27,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.021012+0000","last_change":"2024-04-14T15:18:12.951404+0000","last_active":"2024-04-14T15:18:28.021012+0000","last_peered":"2024-04-14T15:18:28.021012+0000","last_clean":"2024-04-14T15:18:28.021012+0000","last_became_active":"2024-04-14T15:18:12.951112+0000","last_became_peered":"2024-04-14T15:18:12.951112+0000","last_unstale":"2024-04-14T15:18:28.021012+0000","last_undegraded":"2024-04-14T15:18:28.021012+0000","last_fullsized":"2024-04-14T15:18:28.021012+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:14.892479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"297.1a","version":"0'0","reported_seq":27,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.020406+0000","last_change":"2024-04-14T15:18:12.956068+0000","last_active":"2024-04-14T15:18:28.020406+0000","last_peered":"2024-04-14T15:18:28.020406+0000","last_clean":"2024-04-14T15:18:28.020406+0000","last_became_active":"2024-04-14T15:18:12.955801+0000","last_became_peered":"2024-04-14T15:18:12.955801+0000","last_unstale":"2024-04-14T15:18:28.020406+0000","last_undegraded":"2024-04-14T15:18:28.020406+0000","last_fullsized":"2024-04-14T15:18:28.020406+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:15.388092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"297.19","version":"0'0","reported_seq":27,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.204576+0000","last_change":"2024-04-14T15:18:12.955167+0000","last_active":"2024-04-14T15:18:28.204576+0000","last_peered":"2024-04-14T15:18:28.204576+0000","last_clean":"2024-04-14T15:18:28.204576+0000","last_became_active":"2024-04-14T15:18:12.954867+0000","last_became_peered":"2024-04-14T15:18:12.954867+0000","last_unstale":"2024-04-14T15:18:28.204576+0000","last_undegraded":"2024-04-14T15:18:28.204576+0000","last_fullsized":"2024-04-14T15:18:28.204576+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:32.831836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"297.18","version":"0'0","reported_seq":27,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.204514+0000","last_change":"2024-04-14T15:18:12.950510+0000","last_active":"2024-04-14T15:18:28.204514+0000","last_peered":"2024-04-14T15:18:28.204514+0000","last_clean":"2024-04-14T15:18:28.204514+0000","last_became_active":"2024-04-14T15:18:12.950312+0000","last_became_peered":"2024-04-14T15:18:12.950312+0000","last_unstale":"2024-04-14T15:18:28.204514+0000","last_undegraded":"2024-04-14T15:18:28.204514+0000","last_fullsized":"2024-04-14T15:18:28.204514+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:14.281458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"297.17","version":"0'0","reported_seq":27,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.205721+0000","last_change":"2024-04-14T15:18:13.006758+0000","last_active":"2024-04-14T15:18:28.205721+0000","last_peered":"2024-04-14T15:18:28.205721+0000","last_clean":"2024-04-14T15:18:28.205721+0000","last_became_active":"2024-04-14T15:18:13.006469+0000","last_became_peered":"2024-04-14T15:18:13.006469+0000","last_unstale":"2024-04-14T15:18:28.205721+0000","last_undegraded":"2024-04-14T15:18:28.205721+0000","last_fullsized":"2024-04-14T15:18:28.205721+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:41.665082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"297.16","version":"0'0","reported_seq":27,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.023255+0000","last_change":"2024-04-14T15:18:12.951467+0000","last_active":"2024-04-14T15:18:28.023255+0000","last_peered":"2024-04-14T15:18:28.023255+0000","last_clean":"2024-04-14T15:18:28.023255+0000","last_became_active":"2024-04-14T15:18:12.950999+0000","last_became_peered":"2024-04-14T15:18:12.950999+0000","last_unstale":"2024-04-14T15:18:28.023255+0000","last_undegraded":"2024-04-14T15:18:28.023255+0000","last_fullsized":"2024-04-14T15:18:28.023255+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:15.500639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"297.15","version":"0'0","reported_seq":27,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.034589+0000","last_change":"2024-04-14T15:18:13.004242+0000","last_active":"2024-04-14T15:18:28.034589+0000","last_peered":"2024-04-14T15:18:28.034589+0000","last_clean":"2024-04-14T15:18:28.034589+0000","last_became_active":"2024-04-14T15:18:13.003888+0000","last_became_peered":"2024-04-14T15:18:13.003888+0000","last_unstale":"2024-04-14T15:18:28.034589+0000","last_undegraded":"2024-04-14T15:18:28.034589+0000","last_fullsized":"2024-04-14T15:18:28.034589+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:29.490276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"297.14","version":"0'0","reported_seq":27,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.020695+0000","last_change":"2024-04-14T15:18:13.005434+0000","last_active":"2024-04-14T15:18:28.020695+0000","last_peered":"2024-04-14T15:18:28.020695+0000","last_clean":"2024-04-14T15:18:28.020695+0000","last_became_active":"2024-04-14T15:18:13.005051+0000","last_became_peered":"2024-04-14T15:18:13.005051+0000","last_unstale":"2024-04-14T15:18:28.020695+0000","last_undegraded":"2024-04-14T15:18:28.020695+0000","last_fullsized":"2024-04-14T15:18:28.020695+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:14.539808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"297.13","version":"0'0","reported_seq":27,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.027504+0000","last_change":"2024-04-14T15:18:13.004437+0000","last_active":"2024-04-14T15:18:28.027504+0000","last_peered":"2024-04-14T15:18:28.027504+0000","last_clean":"2024-04-14T15:18:28.027504+0000","last_became_active":"2024-04-14T15:18:13.004133+0000","last_became_peered":"2024-04-14T15:18:13.004133+0000","last_unstale":"2024-04-14T15:18:28.027504+0000","last_undegraded":"2024-04-14T15:18:28.027504+0000","last_fullsized":"2024-04-14T15:18:28.027504+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:32.102952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"297.12","version":"0'0","reported_seq":27,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.032217+0000","last_change":"2024-04-14T15:18:12.956427+0000","last_active":"2024-04-14T15:18:28.032217+0000","last_peered":"2024-04-14T15:18:28.032217+0000","last_clean":"2024-04-14T15:18:28.032217+0000","last_became_active":"2024-04-14T15:18:12.955274+0000","last_became_peered":"2024-04-14T15:18:12.955274+0000","last_unstale":"2024-04-14T15:18:28.032217+0000","last_undegraded":"2024-04-14T15:18:28.032217+0000","last_fullsized":"2024-04-14T15:18:28.032217+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:20.629003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"297.11","version":"0'0","reported_seq":27,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.021056+0000","last_change":"2024-04-14T15:18:12.956974+0000","last_active":"2024-04-14T15:18:28.021056+0000","last_peered":"2024-04-14T15:18:28.021056+0000","last_clean":"2024-04-14T15:18:28.021056+0000","last_became_active":"2024-04-14T15:18:12.955819+0000","last_became_peered":"2024-04-14T15:18:12.955819+0000","last_unstale":"2024-04-14T15:18:28.021056+0000","last_undegraded":"2024-04-14T15:18:28.021056+0000","last_fullsized":"2024-04-14T15:18:28.021056+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:23.104281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"297.10","version":"0'0","reported_seq":27,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.020206+0000","last_change":"2024-04-14T15:18:13.006273+0000","last_active":"2024-04-14T15:18:28.020206+0000","last_peered":"2024-04-14T15:18:28.020206+0000","last_clean":"2024-04-14T15:18:28.020206+0000","last_became_active":"2024-04-14T15:18:13.005968+0000","last_became_peered":"2024-04-14T15:18:13.005968+0000","last_unstale":"2024-04-14T15:18:28.020206+0000","last_undegraded":"2024-04-14T15:18:28.020206+0000","last_fullsized":"2024-04-14T15:18:28.020206+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:31.460743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"297.f","version":"0'0","reported_seq":27,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.024725+0000","last_change":"2024-04-14T15:18:13.005801+0000","last_active":"2024-04-14T15:18:28.024725+0000","last_peered":"2024-04-14T15:18:28.024725+0000","last_clean":"2024-04-14T15:18:28.024725+0000","last_became_active":"2024-04-14T15:18:13.005146+0000","last_became_peered":"2024-04-14T15:18:13.005146+0000","last_unstale":"2024-04-14T15:18:28.024725+0000","last_undegraded":"2024-04-14T15:18:28.024725+0000","last_fullsized":"2024-04-14T15:18:28.024725+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:34.836627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"297.e","version":"420'1","reported_seq":28,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.035864+0000","last_change":"2024-04-14T15:18:12.948109+0000","last_active":"2024-04-14T15:18:28.035864+0000","last_peered":"2024-04-14T15:18:28.035864+0000","last_clean":"2024-04-14T15:18:28.035864+0000","last_became_active":"2024-04-14T15:18:12.947811+0000","last_became_peered":"2024-04-14T15:18:12.947811+0000","last_unstale":"2024-04-14T15:18:28.035864+0000","last_undegraded":"2024-04-14T15:18:28.035864+0000","last_fullsized":"2024-04-14T15:18:28.035864+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:14.922501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"297.d","version":"0'0","reported_seq":27,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.205513+0000","last_change":"2024-04-14T15:18:12.949318+0000","last_active":"2024-04-14T15:18:28.205513+0000","last_peered":"2024-04-14T15:18:28.205513+0000","last_clean":"2024-04-14T15:18:28.205513+0000","last_became_active":"2024-04-14T15:18:12.949034+0000","last_became_peered":"2024-04-14T15:18:12.949034+0000","last_unstale":"2024-04-14T15:18:28.205513+0000","last_undegraded":"2024-04-14T15:18:28.205513+0000","last_fullsized":"2024-04-14T15:18:28.205513+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:22.209796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"297.c","version":"0'0","reported_seq":27,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.034571+0000","last_change":"2024-04-14T15:18:13.008426+0000","last_active":"2024-04-14T15:18:28.034571+0000","last_peered":"2024-04-14T15:18:28.034571+0000","last_clean":"2024-04-14T15:18:28.034571+0000","last_became_active":"2024-04-14T15:18:13.007723+0000","last_became_peered":"2024-04-14T15:18:13.007723+0000","last_unstale":"2024-04-14T15:18:28.034571+0000","last_undegraded":"2024-04-14T15:18:28.034571+0000","last_fullsized":"2024-04-14T15:18:28.034571+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:36.383377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"297.b","version":"0'0","reported_seq":27,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.204775+0000","last_change":"2024-04-14T15:18:12.951076+0000","last_active":"2024-04-14T15:18:28.204775+0000","last_peered":"2024-04-14T15:18:28.204775+0000","last_clean":"2024-04-14T15:18:28.204775+0000","last_became_active":"2024-04-14T15:18:12.950806+0000","last_became_peered":"2024-04-14T15:18:12.950806+0000","last_unstale":"2024-04-14T15:18:28.204775+0000","last_undegraded":"2024-04-14T15:18:28.204775+0000","last_fullsized":"2024-04-14T15:18:28.204775+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:22.237973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"297.a","version":"0'0","reported_seq":27,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.020499+0000","last_change":"2024-04-14T15:18:12.955873+0000","last_active":"2024-04-14T15:18:28.020499+0000","last_peered":"2024-04-14T15:18:28.020499+0000","last_clean":"2024-04-14T15:18:28.020499+0000","last_became_active":"2024-04-14T15:18:12.955571+0000","last_became_peered":"2024-04-14T15:18:12.955571+0000","last_unstale":"2024-04-14T15:18:28.020499+0000","last_undegraded":"2024-04-14T15:18:28.020499+0000","last_fullsized":"2024-04-14T15:18:28.020499+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:12.173921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"297.9","version":"0'0","reported_seq":27,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.027031+0000","last_change":"2024-04-14T15:18:13.004346+0000","last_active":"2024-04-14T15:18:28.027031+0000","last_peered":"2024-04-14T15:18:28.027031+0000","last_clean":"2024-04-14T15:18:28.027031+0000","last_became_active":"2024-04-14T15:18:13.004023+0000","last_became_peered":"2024-04-14T15:18:13.004023+0000","last_unstale":"2024-04-14T15:18:28.027031+0000","last_undegraded":"2024-04-14T15:18:28.027031+0000","last_fullsized":"2024-04-14T15:18:28.027031+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:14.413671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"297.8","version":"0'0","reported_seq":27,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.020326+0000","last_change":"2024-04-14T15:18:13.005840+0000","last_active":"2024-04-14T15:18:28.020326+0000","last_peered":"2024-04-14T15:18:28.020326+0000","last_clean":"2024-04-14T15:18:28.020326+0000","last_became_active":"2024-04-14T15:18:13.005478+0000","last_became_peered":"2024-04-14T15:18:13.005478+0000","last_unstale":"2024-04-14T15:18:28.020326+0000","last_undegraded":"2024-04-14T15:18:28.020326+0000","last_fullsized":"2024-04-14T15:18:28.020326+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:29.674300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"297.7","version":"420'2","reported_seq":31,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.032951+0000","last_change":"2024-04-14T15:18:12.955307+0000","last_active":"2024-04-14T15:18:28.032951+0000","last_peered":"2024-04-14T15:18:28.032951+0000","last_clean":"2024-04-14T15:18:28.032951+0000","last_became_active":"2024-04-14T15:18:12.955029+0000","last_became_peered":"2024-04-14T15:18:12.955029+0000","last_unstale":"2024-04-14T15:18:28.032951+0000","last_undegraded":"2024-04-14T15:18:28.032951+0000","last_fullsized":"2024-04-14T15:18:28.032951+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:18.760118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"297.6","version":"0'0","reported_seq":27,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.023974+0000","last_change":"2024-04-14T15:18:13.004180+0000","last_active":"2024-04-14T15:18:28.023974+0000","last_peered":"2024-04-14T15:18:28.023974+0000","last_clean":"2024-04-14T15:18:28.023974+0000","last_became_active":"2024-04-14T15:18:13.003841+0000","last_became_peered":"2024-04-14T15:18:13.003841+0000","last_unstale":"2024-04-14T15:18:28.023974+0000","last_undegraded":"2024-04-14T15:18:28.023974+0000","last_fullsized":"2024-04-14T15:18:28.023974+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:28.810731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"297.5","version":"0'0","reported_seq":27,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.205363+0000","last_change":"2024-04-14T15:18:12.955042+0000","last_active":"2024-04-14T15:18:28.205363+0000","last_peered":"2024-04-14T15:18:28.205363+0000","last_clean":"2024-04-14T15:18:28.205363+0000","last_became_active":"2024-04-14T15:18:12.954742+0000","last_became_peered":"2024-04-14T15:18:12.954742+0000","last_unstale":"2024-04-14T15:18:28.205363+0000","last_undegraded":"2024-04-14T15:18:28.205363+0000","last_fullsized":"2024-04-14T15:18:28.205363+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:12.119643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"297.4","version":"0'0","reported_seq":27,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.020752+0000","last_change":"2024-04-14T15:18:12.955231+0000","last_active":"2024-04-14T15:18:28.020752+0000","last_peered":"2024-04-14T15:18:28.020752+0000","last_clean":"2024-04-14T15:18:28.020752+0000","last_became_active":"2024-04-14T15:18:12.954904+0000","last_became_peered":"2024-04-14T15:18:12.954904+0000","last_unstale":"2024-04-14T15:18:28.020752+0000","last_undegraded":"2024-04-14T15:18:28.020752+0000","last_fullsized":"2024-04-14T15:18:28.020752+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:38.070409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"297.3","version":"0'0","reported_seq":27,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.022044+0000","last_change":"2024-04-14T15:18:13.009275+0000","last_active":"2024-04-14T15:18:28.022044+0000","last_peered":"2024-04-14T15:18:28.022044+0000","last_clean":"2024-04-14T15:18:28.022044+0000","last_became_active":"2024-04-14T15:18:13.008980+0000","last_became_peered":"2024-04-14T15:18:13.008980+0000","last_unstale":"2024-04-14T15:18:28.022044+0000","last_undegraded":"2024-04-14T15:18:28.022044+0000","last_fullsized":"2024-04-14T15:18:28.022044+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:15.759498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"297.2","version":"0'0","reported_seq":27,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.204868+0000","last_change":"2024-04-14T15:18:12.948399+0000","last_active":"2024-04-14T15:18:28.204868+0000","last_peered":"2024-04-14T15:18:28.204868+0000","last_clean":"2024-04-14T15:18:28.204868+0000","last_became_active":"2024-04-14T15:18:12.948080+0000","last_became_peered":"2024-04-14T15:18:12.948080+0000","last_unstale":"2024-04-14T15:18:28.204868+0000","last_undegraded":"2024-04-14T15:18:28.204868+0000","last_fullsized":"2024-04-14T15:18:28.204868+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:20.931990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"297.1","version":"0'0","reported_seq":27,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.020111+0000","last_change":"2024-04-14T15:18:13.006295+0000","last_active":"2024-04-14T15:18:28.020111+0000","last_peered":"2024-04-14T15:18:28.020111+0000","last_clean":"2024-04-14T15:18:28.020111+0000","last_became_active":"2024-04-14T15:18:13.006000+0000","last_became_peered":"2024-04-14T15:18:13.006000+0000","last_unstale":"2024-04-14T15:18:28.020111+0000","last_undegraded":"2024-04-14T15:18:28.020111+0000","last_fullsized":"2024-04-14T15:18:28.020111+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:17.146699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"297.0","version":"427'4","reported_seq":31,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.020080+0000","last_change":"2024-04-14T15:18:13.005460+0000","last_active":"2024-04-14T15:18:28.020080+0000","last_peered":"2024-04-14T15:18:28.020080+0000","last_clean":"2024-04-14T15:18:28.020080+0000","last_became_active":"2024-04-14T15:18:13.005082+0000","last_became_peered":"2024-04-14T15:18:13.005082+0000","last_unstale":"2024-04-14T15:18:28.020080+0000","last_undegraded":"2024-04-14T15:18:28.020080+0000","last_fullsized":"2024-04-14T15:18:28.020080+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:11.925747+0000","last_clean_scrub_stamp":"2024-04-14T15:18:11.925747+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:32.045130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"110.11","version":"403'41","reported_seq":845,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.035011+0000","last_change":"2024-04-14T15:18:28.035460+0000","last_active":"2024-04-14T15:18:28.035011+0000","last_peered":"2024-04-14T15:18:28.035011+0000","last_clean":"2024-04-14T15:18:28.035011+0000","last_became_active":"2024-04-14T15:13:09.936619+0000","last_became_peered":"2024-04-14T15:13:09.936619+0000","last_unstale":"2024-04-14T15:18:28.035011+0000","last_undegraded":"2024-04-14T15:18:28.035011+0000","last_fullsized":"2024-04-14T15:18:28.035011+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"403'41","last_scrub_stamp":"2024-04-14T15:17:56.911004+0000","last_deep_scrub":"403'41","last_deep_scrub_stamp":"2024-04-14T15:17:56.911004+0000","last_clean_scrub_stamp":"2024-04-14T15:17:56.911004+0000","objects_scrubbed":2,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:23.044070+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0.001196981,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"11","length":"1"}]},{"pgid":"110.10","version":"253'158","reported_seq":927,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.020335+0000","last_change":"2024-04-14T15:18:28.021230+0000","last_active":"2024-04-14T15:18:28.020335+0000","last_peered":"2024-04-14T15:18:28.020335+0000","last_clean":"2024-04-14T15:18:28.020335+0000","last_became_active":"2024-04-14T15:13:09.928915+0000","last_became_peered":"2024-04-14T15:13:09.928915+0000","last_unstale":"2024-04-14T15:18:28.020335+0000","last_undegraded":"2024-04-14T15:18:28.020335+0000","last_fullsized":"2024-04-14T15:18:28.020335+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"253'158","last_scrub_stamp":"2024-04-14T15:17:34.291854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:34.291854+0000","objects_scrubbed":10,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:42.366665+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0010918989999999999,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":47,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"11","length":"1"}]},{"pgid":"110.13","version":"409'41","reported_seq":846,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.030186+0000","last_change":"2024-04-14T15:18:28.030462+0000","last_active":"2024-04-14T15:18:28.030186+0000","last_peered":"2024-04-14T15:18:28.030186+0000","last_clean":"2024-04-14T15:18:28.030186+0000","last_became_active":"2024-04-14T15:13:09.918152+0000","last_became_peered":"2024-04-14T15:13:09.918152+0000","last_unstale":"2024-04-14T15:18:28.030186+0000","last_undegraded":"2024-04-14T15:18:28.030186+0000","last_fullsized":"2024-04-14T15:18:28.030186+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'39","last_scrub_stamp":"2024-04-14T15:17:44.873766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:44.873766+0000","objects_scrubbed":3,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:55.075093+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0026908790000000002,"stat_sum":{"num_bytes":27,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":14,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"11","length":"1"}]},{"pgid":"110.12","version":"409'70","reported_seq":872,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.027708+0000","last_change":"2024-04-14T15:18:28.029301+0000","last_active":"2024-04-14T15:18:28.027708+0000","last_peered":"2024-04-14T15:18:28.027708+0000","last_clean":"2024-04-14T15:18:28.027708+0000","last_became_active":"2024-04-14T15:13:09.939566+0000","last_became_peered":"2024-04-14T15:13:09.939566+0000","last_unstale":"2024-04-14T15:18:28.027708+0000","last_undegraded":"2024-04-14T15:18:28.027708+0000","last_fullsized":"2024-04-14T15:18:28.027708+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"253'66","last_scrub_stamp":"2024-04-14T15:16:53.673906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:16:53.673906+0000","objects_scrubbed":6,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001870528,"stat_sum":{"num_bytes":74,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":24,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"11","length":"1"}]},{"pgid":"110.15","version":"190'34","reported_seq":833,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.021638+0000","last_change":"2024-04-14T15:18:28.022532+0000","last_active":"2024-04-14T15:18:28.021638+0000","last_peered":"2024-04-14T15:18:28.021638+0000","last_clean":"2024-04-14T15:18:28.021638+0000","last_became_active":"2024-04-14T15:13:09.960121+0000","last_became_peered":"2024-04-14T15:13:09.960121+0000","last_unstale":"2024-04-14T15:18:28.021638+0000","last_undegraded":"2024-04-14T15:18:28.021638+0000","last_fullsized":"2024-04-14T15:18:28.021638+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'34","last_scrub_stamp":"2024-04-14T15:16:06.494065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:16:06.494065+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001501018,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"11","length":"1"}]},{"pgid":"110.14","version":"355'87","reported_seq":879,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.023317+0000","last_change":"2024-04-14T15:18:28.025053+0000","last_active":"2024-04-14T15:18:28.023317+0000","last_peered":"2024-04-14T15:18:28.023317+0000","last_clean":"2024-04-14T15:18:28.023317+0000","last_became_active":"2024-04-14T15:13:09.899561+0000","last_became_peered":"2024-04-14T15:13:09.899561+0000","last_unstale":"2024-04-14T15:18:28.023317+0000","last_undegraded":"2024-04-14T15:18:28.023317+0000","last_fullsized":"2024-04-14T15:18:28.023317+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'85","last_scrub_stamp":"2024-04-14T15:15:42.409029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:15:42.409029+0000","objects_scrubbed":5,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0022610759999999999,"stat_sum":{"num_bytes":71,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"11","length":"1"}]},{"pgid":"110.17","version":"234'6","reported_seq":812,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.025203+0000","last_change":"2024-04-14T15:18:28.026711+0000","last_active":"2024-04-14T15:18:28.025203+0000","last_peered":"2024-04-14T15:18:28.025203+0000","last_clean":"2024-04-14T15:18:28.025203+0000","last_became_active":"2024-04-14T15:13:09.898521+0000","last_became_peered":"2024-04-14T15:13:09.898521+0000","last_unstale":"2024-04-14T15:18:28.025203+0000","last_undegraded":"2024-04-14T15:18:28.025203+0000","last_fullsized":"2024-04-14T15:18:28.025203+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'6","last_scrub_stamp":"2024-04-14T15:16:07.459718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:16:07.459718+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0022999589999999999,"stat_sum":{"num_bytes":13,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"11","length":"1"}]},{"pgid":"110.16","version":"355'35","reported_seq":673,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.033482+0000","last_change":"2024-04-14T15:18:28.033679+0000","last_active":"2024-04-14T15:18:28.033482+0000","last_peered":"2024-04-14T15:18:28.033482+0000","last_clean":"2024-04-14T15:18:28.033482+0000","last_became_active":"2024-04-14T15:15:21.889783+0000","last_became_peered":"2024-04-14T15:15:21.889783+0000","last_unstale":"2024-04-14T15:18:28.033482+0000","last_undegraded":"2024-04-14T15:18:28.033482+0000","last_fullsized":"2024-04-14T15:18:28.033482+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'35","last_scrub_stamp":"2024-04-14T15:17:34.097563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:34.097563+0000","objects_scrubbed":3,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:45.111414+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00065311999999999998,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"11","length":"1"}]},{"pgid":"110.19","version":"237'85","reported_seq":882,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.024134+0000","last_change":"2024-04-14T15:18:28.024871+0000","last_active":"2024-04-14T15:18:28.024134+0000","last_peered":"2024-04-14T15:18:28.024134+0000","last_clean":"2024-04-14T15:18:28.024134+0000","last_became_active":"2024-04-14T15:13:09.998139+0000","last_became_peered":"2024-04-14T15:13:09.998139+0000","last_unstale":"2024-04-14T15:18:28.024134+0000","last_undegraded":"2024-04-14T15:18:28.024134+0000","last_fullsized":"2024-04-14T15:18:28.024134+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'85","last_scrub_stamp":"2024-04-14T15:16:04.177097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:16:04.177097+0000","objects_scrubbed":5,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0014844470000000001,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"11","length":"1"}]},{"pgid":"110.18","version":"387'94","reported_seq":892,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.021621+0000","last_change":"2024-04-14T15:18:28.022209+0000","last_active":"2024-04-14T15:18:28.021621+0000","last_peered":"2024-04-14T15:18:28.021621+0000","last_clean":"2024-04-14T15:18:28.021621+0000","last_became_active":"2024-04-14T15:13:09.998803+0000","last_became_peered":"2024-04-14T15:13:09.998803+0000","last_unstale":"2024-04-14T15:18:28.021621+0000","last_undegraded":"2024-04-14T15:18:28.021621+0000","last_fullsized":"2024-04-14T15:18:28.021621+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'94","last_scrub_stamp":"2024-04-14T15:17:36.005598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:36.005598+0000","objects_scrubbed":8,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:46.902297+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.001133011,"stat_sum":{"num_bytes":65,"num_objects":8,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":29,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"11","length":"1"}]},{"pgid":"110.1b","version":"355'91","reported_seq":888,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.029353+0000","last_change":"2024-04-14T15:18:28.030134+0000","last_active":"2024-04-14T15:18:28.029353+0000","last_peered":"2024-04-14T15:18:28.029353+0000","last_clean":"2024-04-14T15:18:28.029353+0000","last_became_active":"2024-04-14T15:13:09.946919+0000","last_became_peered":"2024-04-14T15:13:09.946919+0000","last_unstale":"2024-04-14T15:18:28.029353+0000","last_undegraded":"2024-04-14T15:18:28.029353+0000","last_fullsized":"2024-04-14T15:18:28.029353+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'91","last_scrub_stamp":"2024-04-14T15:17:47.889364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:47.889364+0000","objects_scrubbed":6,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:02.988168+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0025446539999999999,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":28,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"11","length":"1"}]},{"pgid":"110.1a","version":"387'39","reported_seq":849,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.035067+0000","last_change":"2024-04-14T15:18:28.035373+0000","last_active":"2024-04-14T15:18:28.035067+0000","last_peered":"2024-04-14T15:18:28.035067+0000","last_clean":"2024-04-14T15:18:28.035067+0000","last_became_active":"2024-04-14T15:13:09.960319+0000","last_became_peered":"2024-04-14T15:13:09.960319+0000","last_unstale":"2024-04-14T15:18:28.035067+0000","last_undegraded":"2024-04-14T15:18:28.035067+0000","last_fullsized":"2024-04-14T15:18:28.035067+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'39","last_scrub_stamp":"2024-04-14T15:17:53.871590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:53.871590+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:06.667531+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00094488899999999997,"stat_sum":{"num_bytes":62,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"11","length":"1"}]},{"pgid":"110.1d","version":"225'85","reported_seq":884,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.205731+0000","last_change":"2024-04-14T15:18:28.206623+0000","last_active":"2024-04-14T15:18:28.205731+0000","last_peered":"2024-04-14T15:18:28.205731+0000","last_clean":"2024-04-14T15:18:28.205731+0000","last_became_active":"2024-04-14T15:13:09.868082+0000","last_became_peered":"2024-04-14T15:13:09.868082+0000","last_unstale":"2024-04-14T15:18:28.205731+0000","last_undegraded":"2024-04-14T15:18:28.205731+0000","last_fullsized":"2024-04-14T15:18:28.205731+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"225'85","last_scrub_stamp":"2024-04-14T15:17:36.419088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:36.419088+0000","objects_scrubbed":5,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:47.706360+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0010902010000000001,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"11","length":"1"}]},{"pgid":"110.1c","version":"427'93","reported_seq":884,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.030652+0000","last_change":"2024-04-14T15:18:28.031264+0000","last_active":"2024-04-14T15:18:28.030652+0000","last_peered":"2024-04-14T15:18:28.030652+0000","last_clean":"2024-04-14T15:18:28.030652+0000","last_became_active":"2024-04-14T15:13:09.954091+0000","last_became_peered":"2024-04-14T15:13:09.954091+0000","last_unstale":"2024-04-14T15:18:28.030652+0000","last_undegraded":"2024-04-14T15:18:28.030652+0000","last_fullsized":"2024-04-14T15:18:28.030652+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'73","last_scrub_stamp":"2024-04-14T15:16:04.615572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:16:04.615572+0000","objects_scrubbed":5,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":1,"snaptrim_duration":0.0034226899999999999,"stat_sum":{"num_bytes":70,"num_objects":8,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"11","length":"1"}]},{"pgid":"110.1f","version":"268'25","reported_seq":878,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.025239+0000","last_change":"2024-04-14T15:18:28.025889+0000","last_active":"2024-04-14T15:18:28.025239+0000","last_peered":"2024-04-14T15:18:28.025239+0000","last_clean":"2024-04-14T15:18:28.025239+0000","last_became_active":"2024-04-14T15:13:09.999629+0000","last_became_peered":"2024-04-14T15:13:09.999629+0000","last_unstale":"2024-04-14T15:18:28.025239+0000","last_undegraded":"2024-04-14T15:18:28.025239+0000","last_fullsized":"2024-04-14T15:18:28.025239+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'25","last_scrub_stamp":"2024-04-14T15:16:08.236881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:16:08.236881+0000","objects_scrubbed":2,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00090980200000000003,"stat_sum":{"num_bytes":4194315,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":8,"num_write_kb":8198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"11","length":"1"}]},{"pgid":"110.1e","version":"355'23","reported_seq":826,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.025149+0000","last_change":"2024-04-14T15:18:28.026042+0000","last_active":"2024-04-14T15:18:28.025149+0000","last_peered":"2024-04-14T15:18:28.025149+0000","last_clean":"2024-04-14T15:18:28.025149+0000","last_became_active":"2024-04-14T15:13:09.955932+0000","last_became_peered":"2024-04-14T15:13:09.955932+0000","last_unstale":"2024-04-14T15:18:28.025149+0000","last_undegraded":"2024-04-14T15:18:28.025149+0000","last_fullsized":"2024-04-14T15:18:28.025149+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'22","last_scrub_stamp":"2024-04-14T15:16:10.427994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:16:10.427994+0000","objects_scrubbed":1,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0024828229999999999,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":8,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"11","length":"1"}]},{"pgid":"110.1","version":"406'32","reported_seq":837,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.205009+0000","last_change":"2024-04-14T15:18:28.205665+0000","last_active":"2024-04-14T15:18:28.205009+0000","last_peered":"2024-04-14T15:18:28.205009+0000","last_clean":"2024-04-14T15:18:28.205009+0000","last_became_active":"2024-04-14T15:13:09.929562+0000","last_became_peered":"2024-04-14T15:13:09.929562+0000","last_unstale":"2024-04-14T15:18:28.205009+0000","last_undegraded":"2024-04-14T15:18:28.205009+0000","last_fullsized":"2024-04-14T15:18:28.205009+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"219'24","last_scrub_stamp":"2024-04-14T15:15:35.591006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:15:35.591006+0000","objects_scrubbed":3,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00085587599999999999,"stat_sum":{"num_bytes":4000050,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":15,"num_write_kb":3921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"11","length":"1"}]},{"pgid":"110.0","version":"219'85","reported_seq":878,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.020388+0000","last_change":"2024-04-14T15:18:28.021136+0000","last_active":"2024-04-14T15:18:28.020388+0000","last_peered":"2024-04-14T15:18:28.020388+0000","last_clean":"2024-04-14T15:18:28.020388+0000","last_became_active":"2024-04-14T15:13:09.856407+0000","last_became_peered":"2024-04-14T15:13:09.856407+0000","last_unstale":"2024-04-14T15:18:28.020388+0000","last_undegraded":"2024-04-14T15:18:28.020388+0000","last_fullsized":"2024-04-14T15:18:28.020388+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"219'85","last_scrub_stamp":"2024-04-14T15:17:51.438149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:51.438149+0000","objects_scrubbed":5,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:52.814648+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00089462899999999998,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"11","length":"1"}]},{"pgid":"110.3","version":"355'52","reported_seq":842,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.021683+0000","last_change":"2024-04-14T15:18:28.022449+0000","last_active":"2024-04-14T15:18:28.021683+0000","last_peered":"2024-04-14T15:18:28.021683+0000","last_clean":"2024-04-14T15:18:28.021683+0000","last_became_active":"2024-04-14T15:13:09.997208+0000","last_became_peered":"2024-04-14T15:13:09.997208+0000","last_unstale":"2024-04-14T15:18:28.021683+0000","last_undegraded":"2024-04-14T15:18:28.021683+0000","last_fullsized":"2024-04-14T15:18:28.021683+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"223'51","last_scrub_stamp":"2024-04-14T15:16:06.149271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:16:06.149271+0000","objects_scrubbed":3,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0010212439999999999,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"11","length":"1"}]},{"pgid":"110.2","version":"240'55","reported_seq":860,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.023573+0000","last_change":"2024-04-14T15:18:28.023900+0000","last_active":"2024-04-14T15:18:28.023573+0000","last_peered":"2024-04-14T15:18:28.023573+0000","last_clean":"2024-04-14T15:18:28.023573+0000","last_became_active":"2024-04-14T15:13:09.957187+0000","last_became_peered":"2024-04-14T15:13:09.957187+0000","last_unstale":"2024-04-14T15:18:28.023573+0000","last_undegraded":"2024-04-14T15:18:28.023573+0000","last_fullsized":"2024-04-14T15:18:28.023573+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'55","last_scrub_stamp":"2024-04-14T15:17:37.941589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:37.941589+0000","objects_scrubbed":6,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:49.248028+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0016898729999999999,"stat_sum":{"num_bytes":49,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":18,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"11","length":"1"}]},{"pgid":"110.5","version":"396'64","reported_seq":861,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.022021+0000","last_change":"2024-04-14T15:18:28.022530+0000","last_active":"2024-04-14T15:18:28.022021+0000","last_peered":"2024-04-14T15:18:28.022021+0000","last_clean":"2024-04-14T15:18:28.022021+0000","last_became_active":"2024-04-14T15:13:09.853689+0000","last_became_peered":"2024-04-14T15:13:09.853689+0000","last_unstale":"2024-04-14T15:18:28.022021+0000","last_undegraded":"2024-04-14T15:18:28.022021+0000","last_fullsized":"2024-04-14T15:18:28.022021+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"396'64","last_scrub_stamp":"2024-04-14T15:17:47.418687+0000","last_deep_scrub":"396'64","last_deep_scrub_stamp":"2024-04-14T15:17:47.418687+0000","last_clean_scrub_stamp":"2024-04-14T15:17:47.418687+0000","objects_scrubbed":5,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:50.612809+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00066524800000000001,"stat_sum":{"num_bytes":42,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":20,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"11","length":"1"}]},{"pgid":"110.4","version":"406'55","reported_seq":850,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.023365+0000","last_change":"2024-04-14T15:18:28.025093+0000","last_active":"2024-04-14T15:18:28.023365+0000","last_peered":"2024-04-14T15:18:28.023365+0000","last_clean":"2024-04-14T15:18:28.023365+0000","last_became_active":"2024-04-14T15:13:09.958847+0000","last_became_peered":"2024-04-14T15:13:09.958847+0000","last_unstale":"2024-04-14T15:18:28.023365+0000","last_undegraded":"2024-04-14T15:18:28.023365+0000","last_fullsized":"2024-04-14T15:18:28.023365+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'51","last_scrub_stamp":"2024-04-14T15:16:05.487850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:16:05.487850+0000","objects_scrubbed":3,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0020787599999999998,"stat_sum":{"num_bytes":38,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"11","length":"1"}]},{"pgid":"110.7","version":"355'18","reported_seq":826,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.036585+0000","last_change":"2024-04-14T15:18:28.036723+0000","last_active":"2024-04-14T15:18:28.036585+0000","last_peered":"2024-04-14T15:18:28.036585+0000","last_clean":"2024-04-14T15:18:28.036585+0000","last_became_active":"2024-04-14T15:13:09.936631+0000","last_became_peered":"2024-04-14T15:13:09.936631+0000","last_unstale":"2024-04-14T15:18:28.036585+0000","last_undegraded":"2024-04-14T15:18:28.036585+0000","last_fullsized":"2024-04-14T15:18:28.036585+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'18","last_scrub_stamp":"2024-04-14T15:17:34.552788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:34.552788+0000","objects_scrubbed":2,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:03.566506+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00029698600000000001,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"11","length":"1"}]},{"pgid":"110.6","version":"274'74","reported_seq":861,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.206672+0000","last_change":"2024-04-14T15:18:28.207048+0000","last_active":"2024-04-14T15:18:28.206672+0000","last_peered":"2024-04-14T15:18:28.206672+0000","last_clean":"2024-04-14T15:18:28.206672+0000","last_became_active":"2024-04-14T15:13:09.930125+0000","last_became_peered":"2024-04-14T15:13:09.930125+0000","last_unstale":"2024-04-14T15:18:28.206672+0000","last_undegraded":"2024-04-14T15:18:28.206672+0000","last_fullsized":"2024-04-14T15:18:28.206672+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'74","last_scrub_stamp":"2024-04-14T15:15:42.641684+0000","last_deep_scrub":"274'74","last_deep_scrub_stamp":"2024-04-14T15:15:42.641684+0000","last_clean_scrub_stamp":"2024-04-14T15:15:42.641684+0000","objects_scrubbed":5,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0012711949999999999,"stat_sum":{"num_bytes":44,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":21,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"11","length":"1"}]},{"pgid":"110.9","version":"369'20","reported_seq":822,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.024077+0000","last_change":"2024-04-14T15:18:28.024966+0000","last_active":"2024-04-14T15:18:28.024077+0000","last_peered":"2024-04-14T15:18:28.024077+0000","last_clean":"2024-04-14T15:18:28.024077+0000","last_became_active":"2024-04-14T15:13:09.957324+0000","last_became_peered":"2024-04-14T15:13:09.957324+0000","last_unstale":"2024-04-14T15:18:28.024077+0000","last_undegraded":"2024-04-14T15:18:28.024077+0000","last_fullsized":"2024-04-14T15:18:28.024077+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"225'17","last_scrub_stamp":"2024-04-14T15:16:09.202462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:16:09.202462+0000","objects_scrubbed":1,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0017906140000000001,"stat_sum":{"num_bytes":35,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"11","length":"1"}]},{"pgid":"110.8","version":"190'17","reported_seq":569,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.031686+0000","last_change":"2024-04-14T15:18:28.032025+0000","last_active":"2024-04-14T15:18:28.031686+0000","last_peered":"2024-04-14T15:18:28.031686+0000","last_clean":"2024-04-14T15:18:28.031686+0000","last_became_active":"2024-04-14T15:15:44.890057+0000","last_became_peered":"2024-04-14T15:15:44.890057+0000","last_unstale":"2024-04-14T15:18:28.031686+0000","last_undegraded":"2024-04-14T15:18:28.031686+0000","last_fullsized":"2024-04-14T15:18:28.031686+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'17","last_scrub_stamp":"2024-04-14T15:15:25.478924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:15:25.478924+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052530399999999998,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"11","length":"1"}]},{"pgid":"110.b","version":"412'71","reported_seq":868,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.030827+0000","last_change":"2024-04-14T15:18:28.031339+0000","last_active":"2024-04-14T15:18:28.030827+0000","last_peered":"2024-04-14T15:18:28.030827+0000","last_clean":"2024-04-14T15:18:28.030827+0000","last_became_active":"2024-04-14T15:13:09.964629+0000","last_became_peered":"2024-04-14T15:13:09.964629+0000","last_unstale":"2024-04-14T15:18:28.030827+0000","last_undegraded":"2024-04-14T15:18:28.030827+0000","last_fullsized":"2024-04-14T15:18:28.030827+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'68","last_scrub_stamp":"2024-04-14T15:16:49.611563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:16:49.611563+0000","objects_scrubbed":4,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.0032333919999999999,"stat_sum":{"num_bytes":54,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":22,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"11","length":"1"}]},{"pgid":"110.a","version":"387'74","reported_seq":793,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.032468+0000","last_change":"2024-04-14T15:18:28.032996+0000","last_active":"2024-04-14T15:18:28.032468+0000","last_peered":"2024-04-14T15:18:28.032468+0000","last_clean":"2024-04-14T15:18:28.032468+0000","last_became_active":"2024-04-14T15:15:21.885346+0000","last_became_peered":"2024-04-14T15:15:21.885346+0000","last_unstale":"2024-04-14T15:18:28.032468+0000","last_undegraded":"2024-04-14T15:18:28.032468+0000","last_fullsized":"2024-04-14T15:18:28.032468+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'70","last_scrub_stamp":"2024-04-14T15:15:34.881731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:15:34.881731+0000","objects_scrubbed":5,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0.00069983800000000004,"stat_sum":{"num_bytes":84,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":26,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"11","length":"1"}]},{"pgid":"110.d","version":"416'95","reported_seq":717,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.033427+0000","last_change":"2024-04-14T15:18:28.033667+0000","last_active":"2024-04-14T15:18:28.033427+0000","last_peered":"2024-04-14T15:18:28.033427+0000","last_clean":"2024-04-14T15:18:28.033427+0000","last_became_active":"2024-04-14T15:15:21.877073+0000","last_became_peered":"2024-04-14T15:15:21.877073+0000","last_unstale":"2024-04-14T15:18:28.033427+0000","last_undegraded":"2024-04-14T15:18:28.033427+0000","last_fullsized":"2024-04-14T15:18:28.033427+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'85","last_scrub_stamp":"2024-04-14T15:15:38.773256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:15:38.773256+0000","objects_scrubbed":5,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00063482399999999998,"stat_sum":{"num_bytes":71,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":34,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"11","length":"1"}]},{"pgid":"110.c","version":"398'20","reported_seq":826,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.030495+0000","last_change":"2024-04-14T15:18:28.030766+0000","last_active":"2024-04-14T15:18:28.030495+0000","last_peered":"2024-04-14T15:18:28.030495+0000","last_clean":"2024-04-14T15:18:28.030495+0000","last_became_active":"2024-04-14T15:13:09.984609+0000","last_became_peered":"2024-04-14T15:13:09.984609+0000","last_unstale":"2024-04-14T15:18:28.030495+0000","last_undegraded":"2024-04-14T15:18:28.030495+0000","last_fullsized":"2024-04-14T15:18:28.030495+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'20","last_scrub_stamp":"2024-04-14T15:17:52.882509+0000","last_deep_scrub":"398'20","last_deep_scrub_stamp":"2024-04-14T15:17:52.882509+0000","last_clean_scrub_stamp":"2024-04-14T15:17:52.882509+0000","objects_scrubbed":2,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:11.508519+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0029492400000000001,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"11","length":"1"}]},{"pgid":"110.f","version":"355'52","reported_seq":854,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.030981+0000","last_change":"2024-04-14T15:18:28.031306+0000","last_active":"2024-04-14T15:18:28.030981+0000","last_peered":"2024-04-14T15:18:28.030981+0000","last_clean":"2024-04-14T15:18:28.030981+0000","last_became_active":"2024-04-14T15:13:09.964057+0000","last_became_peered":"2024-04-14T15:13:09.964057+0000","last_unstale":"2024-04-14T15:18:28.030981+0000","last_undegraded":"2024-04-14T15:18:28.030981+0000","last_fullsized":"2024-04-14T15:18:28.030981+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'52","last_scrub_stamp":"2024-04-14T15:17:45.848166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:45.848166+0000","objects_scrubbed":4,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:08.165216+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0031262970000000001,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"11","length":"1"}]},{"pgid":"110.e","version":"416'92","reported_seq":889,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.205862+0000","last_change":"2024-04-14T15:18:28.206924+0000","last_active":"2024-04-14T15:18:28.205862+0000","last_peered":"2024-04-14T15:18:28.205862+0000","last_clean":"2024-04-14T15:18:28.205862+0000","last_became_active":"2024-04-14T15:13:09.984980+0000","last_became_peered":"2024-04-14T15:13:09.984980+0000","last_unstale":"2024-04-14T15:18:28.205862+0000","last_undegraded":"2024-04-14T15:18:28.205862+0000","last_fullsized":"2024-04-14T15:18:28.205862+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'90","last_scrub_stamp":"2024-04-14T15:17:34.409367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:34.409367+0000","objects_scrubbed":6,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:03.181367+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001242957,"stat_sum":{"num_bytes":68,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":28,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"11","length":"1"}]},{"pgid":"11.7","version":"0'0","reported_seq":313,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.025032+0000","last_change":"2024-04-14T15:15:44.883275+0000","last_active":"2024-04-14T15:18:28.025032+0000","last_peered":"2024-04-14T15:18:28.025032+0000","last_clean":"2024-04-14T15:18:28.025032+0000","last_became_active":"2024-04-14T15:15:44.882941+0000","last_became_peered":"2024-04-14T15:15:44.882941+0000","last_unstale":"2024-04-14T15:18:28.025032+0000","last_undegraded":"2024-04-14T15:18:28.025032+0000","last_fullsized":"2024-04-14T15:18:28.025032+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:32.465858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:32.465858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.9","version":"85'9","reported_seq":875,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.205214+0000","last_change":"2024-04-14T15:17:54.606158+0000","last_active":"2024-04-14T15:18:28.205214+0000","last_peered":"2024-04-14T15:18:28.205214+0000","last_clean":"2024-04-14T15:18:28.205214+0000","last_became_active":"2024-04-14T15:09:47.045441+0000","last_became_peered":"2024-04-14T15:09:47.045441+0000","last_unstale":"2024-04-14T15:18:28.205214+0000","last_undegraded":"2024-04-14T15:18:28.205214+0000","last_fullsized":"2024-04-14T15:18:28.205214+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9","last_scrub_stamp":"2024-04-14T15:17:54.606051+0000","last_deep_scrub":"85'9","last_deep_scrub_stamp":"2024-04-14T15:10:51.950580+0000","last_clean_scrub_stamp":"2024-04-14T15:17:54.606051+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:07.946751+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"72'15","reported_seq":355,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.204611+0000","last_change":"2024-04-14T15:15:44.885742+0000","last_active":"2024-04-14T15:18:28.204611+0000","last_peered":"2024-04-14T15:18:28.204611+0000","last_clean":"2024-04-14T15:18:28.204611+0000","last_became_active":"2024-04-14T15:15:44.885409+0000","last_became_peered":"2024-04-14T15:15:44.885409+0000","last_unstale":"2024-04-14T15:18:28.204611+0000","last_undegraded":"2024-04-14T15:18:28.204611+0000","last_fullsized":"2024-04-14T15:18:28.204611+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-04-14T15:15:39.325129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:15:39.325129+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":767,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.028266+0000","last_change":"2024-04-14T15:17:53.867764+0000","last_active":"2024-04-14T15:18:28.028266+0000","last_peered":"2024-04-14T15:18:28.028266+0000","last_clean":"2024-04-14T15:18:28.028266+0000","last_became_active":"2024-04-14T15:09:43.021460+0000","last_became_peered":"2024-04-14T15:09:43.021460+0000","last_unstale":"2024-04-14T15:18:28.028266+0000","last_undegraded":"2024-04-14T15:18:28.028266+0000","last_fullsized":"2024-04-14T15:18:28.028266+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:53.867696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:53.867696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:56.203183+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":321,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.034577+0000","last_change":"2024-04-14T15:17:54.889170+0000","last_active":"2024-04-14T15:18:28.034577+0000","last_peered":"2024-04-14T15:18:28.034577+0000","last_clean":"2024-04-14T15:18:28.034577+0000","last_became_active":"2024-04-14T15:15:44.888154+0000","last_became_peered":"2024-04-14T15:15:44.888154+0000","last_unstale":"2024-04-14T15:18:28.034577+0000","last_undegraded":"2024-04-14T15:18:28.034577+0000","last_fullsized":"2024-04-14T15:18:28.034577+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:54.889066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:15:40.368219+0000","last_clean_scrub_stamp":"2024-04-14T15:17:54.889066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:11.492168+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":657,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.205171+0000","last_change":"2024-04-14T15:15:32.596167+0000","last_active":"2024-04-14T15:18:28.205171+0000","last_peered":"2024-04-14T15:18:28.205171+0000","last_clean":"2024-04-14T15:18:28.205171+0000","last_became_active":"2024-04-14T15:12:56.355348+0000","last_became_peered":"2024-04-14T15:12:56.355348+0000","last_unstale":"2024-04-14T15:18:28.205171+0000","last_undegraded":"2024-04-14T15:18:28.205171+0000","last_fullsized":"2024-04-14T15:18:28.205171+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:32.596096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:32.596096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"72'9","reported_seq":782,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.022306+0000","last_change":"2024-04-14T15:16:56.663996+0000","last_active":"2024-04-14T15:18:28.022306+0000","last_peered":"2024-04-14T15:18:28.022306+0000","last_clean":"2024-04-14T15:18:28.022306+0000","last_became_active":"2024-04-14T15:09:45.037179+0000","last_became_peered":"2024-04-14T15:09:45.037179+0000","last_unstale":"2024-04-14T15:18:28.022306+0000","last_undegraded":"2024-04-14T15:18:28.022306+0000","last_fullsized":"2024-04-14T15:18:28.022306+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-04-14T15:16:56.663900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:16:56.663900+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":762,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.024610+0000","last_change":"2024-04-14T15:15:25.311260+0000","last_active":"2024-04-14T15:18:28.024610+0000","last_peered":"2024-04-14T15:18:28.024610+0000","last_clean":"2024-04-14T15:18:28.024610+0000","last_became_active":"2024-04-14T15:09:43.020044+0000","last_became_peered":"2024-04-14T15:09:43.020044+0000","last_unstale":"2024-04-14T15:18:28.024610+0000","last_undegraded":"2024-04-14T15:18:28.024610+0000","last_fullsized":"2024-04-14T15:18:28.024610+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:25.311184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:15:25.311184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":759,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.027497+0000","last_change":"2024-04-14T15:16:16.655593+0000","last_active":"2024-04-14T15:18:28.027497+0000","last_peered":"2024-04-14T15:18:28.027497+0000","last_clean":"2024-04-14T15:18:28.027497+0000","last_became_active":"2024-04-14T15:09:47.046153+0000","last_became_peered":"2024-04-14T15:09:47.046153+0000","last_unstale":"2024-04-14T15:18:28.027497+0000","last_undegraded":"2024-04-14T15:18:28.027497+0000","last_fullsized":"2024-04-14T15:18:28.027497+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:16.655504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:16.655504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":758,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.021764+0000","last_change":"2024-04-14T15:17:37.919665+0000","last_active":"2024-04-14T15:18:28.021764+0000","last_peered":"2024-04-14T15:18:28.021764+0000","last_clean":"2024-04-14T15:18:28.021764+0000","last_became_active":"2024-04-14T15:09:49.061495+0000","last_became_peered":"2024-04-14T15:09:49.061495+0000","last_unstale":"2024-04-14T15:18:28.021764+0000","last_undegraded":"2024-04-14T15:18:28.021764+0000","last_fullsized":"2024-04-14T15:18:28.021764+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:37.919595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:17:37.919595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:42.403547+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":665,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.205852+0000","last_change":"2024-04-14T15:18:25.523604+0000","last_active":"2024-04-14T15:18:28.205852+0000","last_peered":"2024-04-14T15:18:28.205852+0000","last_clean":"2024-04-14T15:18:28.205852+0000","last_became_active":"2024-04-14T15:12:56.246062+0000","last_became_peered":"2024-04-14T15:12:56.246062+0000","last_unstale":"2024-04-14T15:18:28.205852+0000","last_undegraded":"2024-04-14T15:18:28.205852+0000","last_fullsized":"2024-04-14T15:18:28.205852+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:25.523520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:18:25.523520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:36.616896+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"222'28","reported_seq":806,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.034959+0000","last_change":"2024-04-14T15:17:44.832902+0000","last_active":"2024-04-14T15:18:28.034959+0000","last_peered":"2024-04-14T15:18:28.034959+0000","last_clean":"2024-04-14T15:18:28.034959+0000","last_became_active":"2024-04-14T15:09:45.178782+0000","last_became_peered":"2024-04-14T15:09:45.178782+0000","last_unstale":"2024-04-14T15:18:28.034959+0000","last_undegraded":"2024-04-14T15:18:28.034959+0000","last_fullsized":"2024-04-14T15:18:28.034959+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'28","last_scrub_stamp":"2024-04-14T15:17:44.832769+0000","last_deep_scrub":"222'28","last_deep_scrub_stamp":"2024-04-14T15:14:50.054931+0000","last_clean_scrub_stamp":"2024-04-14T15:17:44.832769+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:54.927275+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":767,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.028159+0000","last_change":"2024-04-14T15:15:43.569812+0000","last_active":"2024-04-14T15:18:28.028159+0000","last_peered":"2024-04-14T15:18:28.028159+0000","last_clean":"2024-04-14T15:18:28.028159+0000","last_became_active":"2024-04-14T15:09:43.020268+0000","last_became_peered":"2024-04-14T15:09:43.020268+0000","last_unstale":"2024-04-14T15:18:28.028159+0000","last_undegraded":"2024-04-14T15:18:28.028159+0000","last_fullsized":"2024-04-14T15:18:28.028159+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:43.569735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:34.342793+0000","last_clean_scrub_stamp":"2024-04-14T15:15:43.569735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":751,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.027897+0000","last_change":"2024-04-14T15:15:30.488857+0000","last_active":"2024-04-14T15:18:28.027897+0000","last_peered":"2024-04-14T15:18:28.027897+0000","last_clean":"2024-04-14T15:18:28.027897+0000","last_became_active":"2024-04-14T15:09:47.045542+0000","last_became_peered":"2024-04-14T15:09:47.045542+0000","last_unstale":"2024-04-14T15:18:28.027897+0000","last_undegraded":"2024-04-14T15:18:28.027897+0000","last_fullsized":"2024-04-14T15:18:28.027897+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:30.488781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:15:30.488781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":749,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.204903+0000","last_change":"2024-04-14T15:16:08.857476+0000","last_active":"2024-04-14T15:18:28.204903+0000","last_peered":"2024-04-14T15:18:28.204903+0000","last_clean":"2024-04-14T15:18:28.204903+0000","last_became_active":"2024-04-14T15:09:49.065397+0000","last_became_peered":"2024-04-14T15:09:49.065397+0000","last_unstale":"2024-04-14T15:18:28.204903+0000","last_undegraded":"2024-04-14T15:18:28.204903+0000","last_fullsized":"2024-04-14T15:18:28.204903+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:08.857397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:08.857397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"91'130","reported_seq":636,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.026981+0000","last_change":"2024-04-14T15:16:07.609240+0000","last_active":"2024-04-14T15:18:28.026981+0000","last_peered":"2024-04-14T15:18:28.026981+0000","last_clean":"2024-04-14T15:18:28.026981+0000","last_became_active":"2024-04-14T15:15:44.886240+0000","last_became_peered":"2024-04-14T15:15:44.886240+0000","last_unstale":"2024-04-14T15:18:28.026981+0000","last_undegraded":"2024-04-14T15:18:28.026981+0000","last_fullsized":"2024-04-14T15:18:28.026981+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:16:07.609119+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:16:07.609119+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":664,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.027404+0000","last_change":"2024-04-14T15:16:19.749313+0000","last_active":"2024-04-14T15:18:28.027404+0000","last_peered":"2024-04-14T15:18:28.027404+0000","last_clean":"2024-04-14T15:18:28.027404+0000","last_became_active":"2024-04-14T15:12:56.474322+0000","last_became_peered":"2024-04-14T15:12:56.474322+0000","last_unstale":"2024-04-14T15:18:28.027404+0000","last_undegraded":"2024-04-14T15:18:28.027404+0000","last_fullsized":"2024-04-14T15:18:28.027404+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:19.749208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:16:19.749208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"72'17","reported_seq":840,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.022209+0000","last_change":"2024-04-14T15:17:48.374589+0000","last_active":"2024-04-14T15:18:28.022209+0000","last_peered":"2024-04-14T15:18:28.022209+0000","last_clean":"2024-04-14T15:18:28.022209+0000","last_became_active":"2024-04-14T15:13:12.933907+0000","last_became_peered":"2024-04-14T15:13:12.933907+0000","last_unstale":"2024-04-14T15:18:28.022209+0000","last_undegraded":"2024-04-14T15:18:28.022209+0000","last_fullsized":"2024-04-14T15:18:28.022209+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'17","last_scrub_stamp":"2024-04-14T15:17:48.374469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:17:48.374469+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:04.297073+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":760,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.205146+0000","last_change":"2024-04-14T15:16:16.820151+0000","last_active":"2024-04-14T15:18:28.205146+0000","last_peered":"2024-04-14T15:18:28.205146+0000","last_clean":"2024-04-14T15:18:28.205146+0000","last_became_active":"2024-04-14T15:09:43.022351+0000","last_became_peered":"2024-04-14T15:09:43.022351+0000","last_unstale":"2024-04-14T15:18:28.205146+0000","last_undegraded":"2024-04-14T15:18:28.205146+0000","last_fullsized":"2024-04-14T15:18:28.205146+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:16.820070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:16.820070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":754,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.024307+0000","last_change":"2024-04-14T15:17:54.871693+0000","last_active":"2024-04-14T15:18:28.024307+0000","last_peered":"2024-04-14T15:18:28.024307+0000","last_clean":"2024-04-14T15:18:28.024307+0000","last_became_active":"2024-04-14T15:09:47.042810+0000","last_became_peered":"2024-04-14T15:09:47.042810+0000","last_unstale":"2024-04-14T15:18:28.024307+0000","last_undegraded":"2024-04-14T15:18:28.024307+0000","last_fullsized":"2024-04-14T15:18:28.024307+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:54.871593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:17:54.871593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:21.351148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":755,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.028354+0000","last_change":"2024-04-14T15:16:45.669141+0000","last_active":"2024-04-14T15:18:28.028354+0000","last_peered":"2024-04-14T15:18:28.028354+0000","last_clean":"2024-04-14T15:18:28.028354+0000","last_became_active":"2024-04-14T15:09:49.069594+0000","last_became_peered":"2024-04-14T15:09:49.069594+0000","last_unstale":"2024-04-14T15:18:28.028354+0000","last_undegraded":"2024-04-14T15:18:28.028354+0000","last_fullsized":"2024-04-14T15:18:28.028354+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:45.669063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:45.669063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"91'130","reported_seq":1038,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.204590+0000","last_change":"2024-04-14T15:18:22.518430+0000","last_active":"2024-04-14T15:18:28.204590+0000","last_peered":"2024-04-14T15:18:28.204590+0000","last_clean":"2024-04-14T15:18:28.204590+0000","last_became_active":"2024-04-14T15:14:17.873403+0000","last_became_peered":"2024-04-14T15:14:17.873403+0000","last_unstale":"2024-04-14T15:18:28.204590+0000","last_undegraded":"2024-04-14T15:18:28.204590+0000","last_fullsized":"2024-04-14T15:18:28.204590+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:18:22.518348+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:18:22.518348+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:50.282840+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":666,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.034565+0000","last_change":"2024-04-14T15:17:46.871883+0000","last_active":"2024-04-14T15:18:28.034565+0000","last_peered":"2024-04-14T15:18:28.034565+0000","last_clean":"2024-04-14T15:18:28.034565+0000","last_became_active":"2024-04-14T15:12:56.289745+0000","last_became_peered":"2024-04-14T15:12:56.289745+0000","last_unstale":"2024-04-14T15:18:28.034565+0000","last_undegraded":"2024-04-14T15:18:28.034565+0000","last_fullsized":"2024-04-14T15:18:28.034565+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:46.871773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:17:46.871773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:58.233534+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"179'15","reported_seq":776,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.021125+0000","last_change":"2024-04-14T15:16:49.243096+0000","last_active":"2024-04-14T15:18:28.021125+0000","last_peered":"2024-04-14T15:18:28.021125+0000","last_clean":"2024-04-14T15:18:28.021125+0000","last_became_active":"2024-04-14T15:09:45.029170+0000","last_became_peered":"2024-04-14T15:09:45.029170+0000","last_unstale":"2024-04-14T15:18:28.021125+0000","last_undegraded":"2024-04-14T15:18:28.021125+0000","last_fullsized":"2024-04-14T15:18:28.021125+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'15","last_scrub_stamp":"2024-04-14T15:16:49.243002+0000","last_deep_scrub":"179'15","last_deep_scrub_stamp":"2024-04-14T15:14:07.488422+0000","last_clean_scrub_stamp":"2024-04-14T15:16:49.243002+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":771,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.023423+0000","last_change":"2024-04-14T15:17:51.854742+0000","last_active":"2024-04-14T15:18:28.023423+0000","last_peered":"2024-04-14T15:18:28.023423+0000","last_clean":"2024-04-14T15:18:28.023423+0000","last_became_active":"2024-04-14T15:09:43.018871+0000","last_became_peered":"2024-04-14T15:09:43.018871+0000","last_unstale":"2024-04-14T15:18:28.023423+0000","last_undegraded":"2024-04-14T15:18:28.023423+0000","last_fullsized":"2024-04-14T15:18:28.023423+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:51.854659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:51.854659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:08.464204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"400'9","reported_seq":874,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.027964+0000","last_change":"2024-04-14T15:17:54.845967+0000","last_active":"2024-04-14T15:18:28.027964+0000","last_peered":"2024-04-14T15:18:28.027964+0000","last_clean":"2024-04-14T15:18:28.027964+0000","last_became_active":"2024-04-14T15:09:47.044085+0000","last_became_peered":"2024-04-14T15:09:47.044085+0000","last_unstale":"2024-04-14T15:18:28.027964+0000","last_undegraded":"2024-04-14T15:18:28.027964+0000","last_fullsized":"2024-04-14T15:18:28.027964+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'9","last_scrub_stamp":"2024-04-14T15:17:54.845881+0000","last_deep_scrub":"400'9","last_deep_scrub_stamp":"2024-04-14T15:17:54.845881+0000","last_clean_scrub_stamp":"2024-04-14T15:17:54.845881+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:58.974945+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":748,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.022084+0000","last_change":"2024-04-14T15:16:17.041377+0000","last_active":"2024-04-14T15:18:28.022084+0000","last_peered":"2024-04-14T15:18:28.022084+0000","last_clean":"2024-04-14T15:18:28.022084+0000","last_became_active":"2024-04-14T15:09:49.065274+0000","last_became_peered":"2024-04-14T15:09:49.065274+0000","last_unstale":"2024-04-14T15:18:28.022084+0000","last_undegraded":"2024-04-14T15:18:28.022084+0000","last_fullsized":"2024-04-14T15:18:28.022084+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:17.041301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:17.041301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"91'130","reported_seq":636,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.027221+0000","last_change":"2024-04-14T15:15:45.589217+0000","last_active":"2024-04-14T15:18:28.027221+0000","last_peered":"2024-04-14T15:18:28.027221+0000","last_clean":"2024-04-14T15:18:28.027221+0000","last_became_active":"2024-04-14T15:15:44.887114+0000","last_became_peered":"2024-04-14T15:15:44.887114+0000","last_unstale":"2024-04-14T15:18:28.027221+0000","last_undegraded":"2024-04-14T15:18:28.027221+0000","last_fullsized":"2024-04-14T15:18:28.027221+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:15:45.589132+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:15:45.589132+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":659,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.021761+0000","last_change":"2024-04-14T15:15:29.366895+0000","last_active":"2024-04-14T15:18:28.021761+0000","last_peered":"2024-04-14T15:18:28.021761+0000","last_clean":"2024-04-14T15:18:28.021761+0000","last_became_active":"2024-04-14T15:12:56.487615+0000","last_became_peered":"2024-04-14T15:12:56.487615+0000","last_unstale":"2024-04-14T15:18:28.021761+0000","last_undegraded":"2024-04-14T15:18:28.021761+0000","last_fullsized":"2024-04-14T15:18:28.021761+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:29.366820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:29.366820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"398'22","reported_seq":798,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.025431+0000","last_change":"2024-04-14T15:17:52.883984+0000","last_active":"2024-04-14T15:18:28.025431+0000","last_peered":"2024-04-14T15:18:28.025431+0000","last_clean":"2024-04-14T15:18:28.025431+0000","last_became_active":"2024-04-14T15:09:45.033331+0000","last_became_peered":"2024-04-14T15:09:45.033331+0000","last_unstale":"2024-04-14T15:18:28.025431+0000","last_undegraded":"2024-04-14T15:18:28.025431+0000","last_fullsized":"2024-04-14T15:18:28.025431+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'22","last_scrub_stamp":"2024-04-14T15:17:52.883850+0000","last_deep_scrub":"398'22","last_deep_scrub_stamp":"2024-04-14T15:17:52.883850+0000","last_clean_scrub_stamp":"2024-04-14T15:17:52.883850+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:58.280825+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":771,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.023752+0000","last_change":"2024-04-14T15:18:25.009179+0000","last_active":"2024-04-14T15:18:28.023752+0000","last_peered":"2024-04-14T15:18:28.023752+0000","last_clean":"2024-04-14T15:18:28.023752+0000","last_became_active":"2024-04-14T15:09:43.020878+0000","last_became_peered":"2024-04-14T15:09:43.020878+0000","last_unstale":"2024-04-14T15:18:28.023752+0000","last_undegraded":"2024-04-14T15:18:28.023752+0000","last_fullsized":"2024-04-14T15:18:28.023752+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:25.009100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:18:25.009100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:54.811984+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"72'8","reported_seq":866,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.022028+0000","last_change":"2024-04-14T15:16:06.933663+0000","last_active":"2024-04-14T15:18:28.022028+0000","last_peered":"2024-04-14T15:18:28.022028+0000","last_clean":"2024-04-14T15:18:28.022028+0000","last_became_active":"2024-04-14T15:09:47.050962+0000","last_became_peered":"2024-04-14T15:09:47.050962+0000","last_unstale":"2024-04-14T15:18:28.022028+0000","last_undegraded":"2024-04-14T15:18:28.022028+0000","last_fullsized":"2024-04-14T15:18:28.022028+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:16:06.933489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:06.933489+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":322,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.024174+0000","last_change":"2024-04-14T15:16:51.199025+0000","last_active":"2024-04-14T15:18:28.024174+0000","last_peered":"2024-04-14T15:18:28.024174+0000","last_clean":"2024-04-14T15:18:28.024174+0000","last_became_active":"2024-04-14T15:15:44.879699+0000","last_became_peered":"2024-04-14T15:15:44.879699+0000","last_unstale":"2024-04-14T15:18:28.024174+0000","last_undegraded":"2024-04-14T15:18:28.024174+0000","last_fullsized":"2024-04-14T15:18:28.024174+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:51.198907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:51.198907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"233'8","reported_seq":680,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.034972+0000","last_change":"2024-04-14T15:17:50.905698+0000","last_active":"2024-04-14T15:18:28.034972+0000","last_peered":"2024-04-14T15:18:28.034972+0000","last_clean":"2024-04-14T15:18:28.034972+0000","last_became_active":"2024-04-14T15:12:56.381026+0000","last_became_peered":"2024-04-14T15:12:56.381026+0000","last_unstale":"2024-04-14T15:18:28.034972+0000","last_undegraded":"2024-04-14T15:18:28.034972+0000","last_fullsized":"2024-04-14T15:18:28.034972+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'8","last_scrub_stamp":"2024-04-14T15:17:50.905608+0000","last_deep_scrub":"233'8","last_deep_scrub_stamp":"2024-04-14T15:15:01.295800+0000","last_clean_scrub_stamp":"2024-04-14T15:17:50.905608+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:14.140648+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"85'16","reported_seq":788,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.022237+0000","last_change":"2024-04-14T15:16:53.143329+0000","last_active":"2024-04-14T15:18:28.022237+0000","last_peered":"2024-04-14T15:18:28.022237+0000","last_clean":"2024-04-14T15:18:28.022237+0000","last_became_active":"2024-04-14T15:09:45.028212+0000","last_became_peered":"2024-04-14T15:09:45.028212+0000","last_unstale":"2024-04-14T15:18:28.022237+0000","last_undegraded":"2024-04-14T15:18:28.022237+0000","last_fullsized":"2024-04-14T15:18:28.022237+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-14T15:16:53.143239+0000","last_deep_scrub":"85'16","last_deep_scrub_stamp":"2024-04-14T15:10:47.456169+0000","last_clean_scrub_stamp":"2024-04-14T15:16:53.143239+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":769,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.204732+0000","last_change":"2024-04-14T15:17:39.465617+0000","last_active":"2024-04-14T15:18:28.204732+0000","last_peered":"2024-04-14T15:18:28.204732+0000","last_clean":"2024-04-14T15:18:28.204732+0000","last_became_active":"2024-04-14T15:09:43.018601+0000","last_became_peered":"2024-04-14T15:09:43.018601+0000","last_unstale":"2024-04-14T15:18:28.204732+0000","last_undegraded":"2024-04-14T15:18:28.204732+0000","last_fullsized":"2024-04-14T15:18:28.204732+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:39.465475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:39.465475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:44.272999+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":319,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.021663+0000","last_change":"2024-04-14T15:16:54.187971+0000","last_active":"2024-04-14T15:18:28.021663+0000","last_peered":"2024-04-14T15:18:28.021663+0000","last_clean":"2024-04-14T15:18:28.021663+0000","last_became_active":"2024-04-14T15:15:44.886834+0000","last_became_peered":"2024-04-14T15:15:44.886834+0000","last_unstale":"2024-04-14T15:18:28.021663+0000","last_undegraded":"2024-04-14T15:18:28.021663+0000","last_fullsized":"2024-04-14T15:18:28.021663+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:54.187888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:54.187888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":431,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.032775+0000","last_change":"2024-04-14T15:17:57.214134+0000","last_active":"2024-04-14T15:18:28.032775+0000","last_peered":"2024-04-14T15:18:28.032775+0000","last_clean":"2024-04-14T15:18:28.032775+0000","last_became_active":"2024-04-14T15:15:21.888587+0000","last_became_peered":"2024-04-14T15:15:21.888587+0000","last_unstale":"2024-04-14T15:18:28.032775+0000","last_undegraded":"2024-04-14T15:18:28.032775+0000","last_fullsized":"2024-04-14T15:18:28.032775+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:57.214046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:57.214046+0000","last_clean_scrub_stamp":"2024-04-14T15:17:57.214046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:03.985520+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"91'130","reported_seq":818,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.032633+0000","last_change":"2024-04-14T15:16:41.944414+0000","last_active":"2024-04-14T15:18:28.032633+0000","last_peered":"2024-04-14T15:18:28.032633+0000","last_clean":"2024-04-14T15:18:28.032633+0000","last_became_active":"2024-04-14T15:14:17.863503+0000","last_became_peered":"2024-04-14T15:14:17.863503+0000","last_unstale":"2024-04-14T15:18:28.032633+0000","last_undegraded":"2024-04-14T15:18:28.032633+0000","last_fullsized":"2024-04-14T15:18:28.032633+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:16:41.944267+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:16:41.944267+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":665,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.021494+0000","last_change":"2024-04-14T15:16:56.214275+0000","last_active":"2024-04-14T15:18:28.021494+0000","last_peered":"2024-04-14T15:18:28.021494+0000","last_clean":"2024-04-14T15:18:28.021494+0000","last_became_active":"2024-04-14T15:12:56.427276+0000","last_became_peered":"2024-04-14T15:12:56.427276+0000","last_unstale":"2024-04-14T15:18:28.021494+0000","last_undegraded":"2024-04-14T15:18:28.021494+0000","last_fullsized":"2024-04-14T15:18:28.021494+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:56.214178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:16:56.214178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"255'10","reported_seq":510,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.032144+0000","last_change":"2024-04-14T15:16:17.868542+0000","last_active":"2024-04-14T15:18:28.032144+0000","last_peered":"2024-04-14T15:18:28.032144+0000","last_clean":"2024-04-14T15:18:28.032144+0000","last_became_active":"2024-04-14T15:15:21.873747+0000","last_became_peered":"2024-04-14T15:15:21.873747+0000","last_unstale":"2024-04-14T15:18:28.032144+0000","last_undegraded":"2024-04-14T15:18:28.032144+0000","last_fullsized":"2024-04-14T15:18:28.032144+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'10","last_scrub_stamp":"2024-04-14T15:16:17.868428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:17.868428+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"72'14","reported_seq":786,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.020306+0000","last_change":"2024-04-14T15:18:22.545734+0000","last_active":"2024-04-14T15:18:28.020306+0000","last_peered":"2024-04-14T15:18:28.020306+0000","last_clean":"2024-04-14T15:18:28.020306+0000","last_became_active":"2024-04-14T15:09:45.025673+0000","last_became_peered":"2024-04-14T15:09:45.025673+0000","last_unstale":"2024-04-14T15:18:28.020306+0000","last_undegraded":"2024-04-14T15:18:28.020306+0000","last_fullsized":"2024-04-14T15:18:28.020306+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'14","last_scrub_stamp":"2024-04-14T15:18:22.545646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:18:22.545646+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:33.421913+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":770,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.036307+0000","last_change":"2024-04-14T15:18:26.824852+0000","last_active":"2024-04-14T15:18:28.036307+0000","last_peered":"2024-04-14T15:18:28.036307+0000","last_clean":"2024-04-14T15:18:28.036307+0000","last_became_active":"2024-04-14T15:09:43.020503+0000","last_became_peered":"2024-04-14T15:09:43.020503+0000","last_unstale":"2024-04-14T15:18:28.036307+0000","last_undegraded":"2024-04-14T15:18:28.036307+0000","last_fullsized":"2024-04-14T15:18:28.036307+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:26.824773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:18:26.824773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:26.903997+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":749,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.205469+0000","last_change":"2024-04-14T15:16:45.705837+0000","last_active":"2024-04-14T15:18:28.205469+0000","last_peered":"2024-04-14T15:18:28.205469+0000","last_clean":"2024-04-14T15:18:28.205469+0000","last_became_active":"2024-04-14T15:09:49.129610+0000","last_became_peered":"2024-04-14T15:09:49.129610+0000","last_unstale":"2024-04-14T15:18:28.205469+0000","last_undegraded":"2024-04-14T15:18:28.205469+0000","last_fullsized":"2024-04-14T15:18:28.205469+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:45.705761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:45.705761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"91'130","reported_seq":820,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.021536+0000","last_change":"2024-04-14T15:16:41.193543+0000","last_active":"2024-04-14T15:18:28.021536+0000","last_peered":"2024-04-14T15:18:28.021536+0000","last_clean":"2024-04-14T15:18:28.021536+0000","last_became_active":"2024-04-14T15:14:17.871922+0000","last_became_peered":"2024-04-14T15:14:17.871922+0000","last_unstale":"2024-04-14T15:18:28.021536+0000","last_undegraded":"2024-04-14T15:18:28.021536+0000","last_fullsized":"2024-04-14T15:18:28.021536+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:16:41.193422+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:16:41.193422+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":665,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.206002+0000","last_change":"2024-04-14T15:17:47.553179+0000","last_active":"2024-04-14T15:18:28.206002+0000","last_peered":"2024-04-14T15:18:28.206002+0000","last_clean":"2024-04-14T15:18:28.206002+0000","last_became_active":"2024-04-14T15:12:56.355254+0000","last_became_peered":"2024-04-14T15:12:56.355254+0000","last_unstale":"2024-04-14T15:18:28.206002+0000","last_undegraded":"2024-04-14T15:18:28.206002+0000","last_fullsized":"2024-04-14T15:18:28.206002+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:47.553055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:17:47.553055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:58.344778+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":753,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.020465+0000","last_change":"2024-04-14T15:16:05.917909+0000","last_active":"2024-04-14T15:18:28.020465+0000","last_peered":"2024-04-14T15:18:28.020465+0000","last_clean":"2024-04-14T15:18:28.020465+0000","last_became_active":"2024-04-14T15:09:47.044610+0000","last_became_peered":"2024-04-14T15:09:47.044610+0000","last_unstale":"2024-04-14T15:18:28.020465+0000","last_undegraded":"2024-04-14T15:18:28.020465+0000","last_fullsized":"2024-04-14T15:18:28.020465+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:05.917804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:05.917804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"72'18","reported_seq":472,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.032021+0000","last_change":"2024-04-14T15:16:15.913377+0000","last_active":"2024-04-14T15:18:28.032021+0000","last_peered":"2024-04-14T15:18:28.032021+0000","last_clean":"2024-04-14T15:18:28.032021+0000","last_became_active":"2024-04-14T15:15:21.880565+0000","last_became_peered":"2024-04-14T15:15:21.880565+0000","last_unstale":"2024-04-14T15:18:28.032021+0000","last_undegraded":"2024-04-14T15:18:28.032021+0000","last_fullsized":"2024-04-14T15:18:28.032021+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'18","last_scrub_stamp":"2024-04-14T15:16:15.913273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:16:15.913273+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'2","reported_seq":765,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.024231+0000","last_change":"2024-04-14T15:17:48.850178+0000","last_active":"2024-04-14T15:18:28.024231+0000","last_peered":"2024-04-14T15:18:28.024231+0000","last_clean":"2024-04-14T15:18:28.024231+0000","last_became_active":"2024-04-14T15:09:43.018478+0000","last_became_peered":"2024-04-14T15:09:43.018478+0000","last_unstale":"2024-04-14T15:18:28.024231+0000","last_undegraded":"2024-04-14T15:18:28.024231+0000","last_fullsized":"2024-04-14T15:18:28.024231+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'2","last_scrub_stamp":"2024-04-14T15:17:48.850069+0000","last_deep_scrub":"91'2","last_deep_scrub_stamp":"2024-04-14T15:17:48.850069+0000","last_clean_scrub_stamp":"2024-04-14T15:17:48.850069+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:12.966475+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":749,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.021003+0000","last_change":"2024-04-14T15:16:14.017818+0000","last_active":"2024-04-14T15:18:28.021003+0000","last_peered":"2024-04-14T15:18:28.021003+0000","last_clean":"2024-04-14T15:18:28.021003+0000","last_became_active":"2024-04-14T15:09:49.065684+0000","last_became_peered":"2024-04-14T15:09:49.065684+0000","last_unstale":"2024-04-14T15:18:28.021003+0000","last_undegraded":"2024-04-14T15:18:28.021003+0000","last_fullsized":"2024-04-14T15:18:28.021003+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:14.017729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:14.017729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"91'130","reported_seq":827,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.035863+0000","last_change":"2024-04-14T15:17:47.885405+0000","last_active":"2024-04-14T15:18:28.035863+0000","last_peered":"2024-04-14T15:18:28.035863+0000","last_clean":"2024-04-14T15:18:28.035863+0000","last_became_active":"2024-04-14T15:14:17.853143+0000","last_became_peered":"2024-04-14T15:14:17.853143+0000","last_unstale":"2024-04-14T15:18:28.035863+0000","last_undegraded":"2024-04-14T15:18:28.035863+0000","last_fullsized":"2024-04-14T15:18:28.035863+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:17:47.885308+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:17:47.885308+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:05.853775+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":659,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.024030+0000","last_change":"2024-04-14T15:15:31.403165+0000","last_active":"2024-04-14T15:18:28.024030+0000","last_peered":"2024-04-14T15:18:28.024030+0000","last_clean":"2024-04-14T15:18:28.024030+0000","last_became_active":"2024-04-14T15:12:56.402023+0000","last_became_peered":"2024-04-14T15:12:56.402023+0000","last_unstale":"2024-04-14T15:18:28.024030+0000","last_undegraded":"2024-04-14T15:18:28.024030+0000","last_fullsized":"2024-04-14T15:18:28.024030+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:31.403086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:31.403086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":762,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.034996+0000","last_change":"2024-04-14T15:17:39.682323+0000","last_active":"2024-04-14T15:18:28.034996+0000","last_peered":"2024-04-14T15:18:28.034996+0000","last_clean":"2024-04-14T15:18:28.034996+0000","last_became_active":"2024-04-14T15:09:47.048427+0000","last_became_peered":"2024-04-14T15:09:47.048427+0000","last_unstale":"2024-04-14T15:18:28.034996+0000","last_undegraded":"2024-04-14T15:18:28.034996+0000","last_fullsized":"2024-04-14T15:18:28.034996+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:39.682203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:17:39.682203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:58.685184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"102'27","reported_seq":811,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.022684+0000","last_change":"2024-04-14T15:17:46.924432+0000","last_active":"2024-04-14T15:18:28.022684+0000","last_peered":"2024-04-14T15:18:28.022684+0000","last_clean":"2024-04-14T15:18:28.022684+0000","last_became_active":"2024-04-14T15:09:45.179182+0000","last_became_peered":"2024-04-14T15:09:45.179182+0000","last_unstale":"2024-04-14T15:18:28.022684+0000","last_undegraded":"2024-04-14T15:18:28.022684+0000","last_fullsized":"2024-04-14T15:18:28.022684+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'27","last_scrub_stamp":"2024-04-14T15:17:46.924330+0000","last_deep_scrub":"102'27","last_deep_scrub_stamp":"2024-04-14T15:17:46.924330+0000","last_clean_scrub_stamp":"2024-04-14T15:17:46.924330+0000","objects_scrubbed":8,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:54.826490+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":761,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.205732+0000","last_change":"2024-04-14T15:16:18.774542+0000","last_active":"2024-04-14T15:18:28.205732+0000","last_peered":"2024-04-14T15:18:28.205732+0000","last_clean":"2024-04-14T15:18:28.205732+0000","last_became_active":"2024-04-14T15:09:43.021006+0000","last_became_peered":"2024-04-14T15:09:43.021006+0000","last_unstale":"2024-04-14T15:18:28.205732+0000","last_undegraded":"2024-04-14T15:18:28.205732+0000","last_fullsized":"2024-04-14T15:18:28.205732+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:18.774463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:18.774463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":751,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.023850+0000","last_change":"2024-04-14T15:16:56.206766+0000","last_active":"2024-04-14T15:18:28.023850+0000","last_peered":"2024-04-14T15:18:28.023850+0000","last_clean":"2024-04-14T15:18:28.023850+0000","last_became_active":"2024-04-14T15:09:49.128136+0000","last_became_peered":"2024-04-14T15:09:49.128136+0000","last_unstale":"2024-04-14T15:18:28.023850+0000","last_undegraded":"2024-04-14T15:18:28.023850+0000","last_fullsized":"2024-04-14T15:18:28.023850+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:56.206649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:56.206649+0000","last_clean_scrub_stamp":"2024-04-14T15:16:56.206649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"91'130","reported_seq":822,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.021072+0000","last_change":"2024-04-14T15:17:49.890530+0000","last_active":"2024-04-14T15:18:28.021072+0000","last_peered":"2024-04-14T15:18:28.021072+0000","last_clean":"2024-04-14T15:18:28.021072+0000","last_became_active":"2024-04-14T15:14:17.856694+0000","last_became_peered":"2024-04-14T15:14:17.856694+0000","last_unstale":"2024-04-14T15:18:28.021072+0000","last_undegraded":"2024-04-14T15:18:28.021072+0000","last_fullsized":"2024-04-14T15:18:28.021072+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:17:49.890445+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:17:49.890445+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:16.061587+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":363,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.032289+0000","last_change":"2024-04-14T15:15:31.812967+0000","last_active":"2024-04-14T15:18:28.032289+0000","last_peered":"2024-04-14T15:18:28.032289+0000","last_clean":"2024-04-14T15:18:28.032289+0000","last_became_active":"2024-04-14T15:15:21.887769+0000","last_became_peered":"2024-04-14T15:15:21.887769+0000","last_unstale":"2024-04-14T15:18:28.032289+0000","last_undegraded":"2024-04-14T15:18:28.032289+0000","last_fullsized":"2024-04-14T15:18:28.032289+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:31.812888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:31.812888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":321,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.025709+0000","last_change":"2024-04-14T15:17:47.871801+0000","last_active":"2024-04-14T15:18:28.025709+0000","last_peered":"2024-04-14T15:18:28.025709+0000","last_clean":"2024-04-14T15:18:28.025709+0000","last_became_active":"2024-04-14T15:15:44.883694+0000","last_became_peered":"2024-04-14T15:15:44.883694+0000","last_unstale":"2024-04-14T15:18:28.025709+0000","last_undegraded":"2024-04-14T15:18:28.025709+0000","last_fullsized":"2024-04-14T15:18:28.025709+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:47.871722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:17:47.871722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:15.357681+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"182'18","reported_seq":791,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.023877+0000","last_change":"2024-04-14T15:17:44.919445+0000","last_active":"2024-04-14T15:18:28.023877+0000","last_peered":"2024-04-14T15:18:28.023877+0000","last_clean":"2024-04-14T15:18:28.023877+0000","last_became_active":"2024-04-14T15:09:45.037231+0000","last_became_peered":"2024-04-14T15:09:45.037231+0000","last_unstale":"2024-04-14T15:18:28.023877+0000","last_undegraded":"2024-04-14T15:18:28.023877+0000","last_fullsized":"2024-04-14T15:18:28.023877+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'18","last_scrub_stamp":"2024-04-14T15:17:44.919358+0000","last_deep_scrub":"182'18","last_deep_scrub_stamp":"2024-04-14T15:14:10.383906+0000","last_clean_scrub_stamp":"2024-04-14T15:17:44.919358+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:48.791608+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":769,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.204850+0000","last_change":"2024-04-14T15:17:46.542125+0000","last_active":"2024-04-14T15:18:28.204850+0000","last_peered":"2024-04-14T15:18:28.204850+0000","last_clean":"2024-04-14T15:18:28.204850+0000","last_became_active":"2024-04-14T15:09:43.025095+0000","last_became_peered":"2024-04-14T15:09:43.025095+0000","last_unstale":"2024-04-14T15:18:28.204850+0000","last_undegraded":"2024-04-14T15:18:28.204850+0000","last_fullsized":"2024-04-14T15:18:28.204850+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:46.542030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:46.542030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:59.318501+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":778,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.021143+0000","last_change":"2024-04-14T15:17:48.866222+0000","last_active":"2024-04-14T15:18:28.021143+0000","last_peered":"2024-04-14T15:18:28.021143+0000","last_clean":"2024-04-14T15:18:28.021143+0000","last_became_active":"2024-04-14T15:09:43.018509+0000","last_became_peered":"2024-04-14T15:09:43.018509+0000","last_unstale":"2024-04-14T15:18:28.021143+0000","last_undegraded":"2024-04-14T15:18:28.021143+0000","last_fullsized":"2024-04-14T15:18:28.021143+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:48.866146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:48.866146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:57.582404+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0018592680000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"355'2","reported_seq":753,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.024400+0000","last_change":"2024-04-14T15:17:03.127190+0000","last_active":"2024-04-14T15:18:28.024400+0000","last_peered":"2024-04-14T15:18:28.024400+0000","last_clean":"2024-04-14T15:18:28.024400+0000","last_became_active":"2024-04-14T15:09:49.065243+0000","last_became_peered":"2024-04-14T15:09:49.065243+0000","last_unstale":"2024-04-14T15:18:28.024400+0000","last_undegraded":"2024-04-14T15:18:28.024400+0000","last_fullsized":"2024-04-14T15:18:28.024400+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'2","last_scrub_stamp":"2024-04-14T15:17:03.127112+0000","last_deep_scrub":"355'2","last_deep_scrub_stamp":"2024-04-14T15:17:03.127112+0000","last_clean_scrub_stamp":"2024-04-14T15:17:03.127112+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":355,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.032451+0000","last_change":"2024-04-14T15:15:21.890855+0000","last_active":"2024-04-14T15:18:28.032451+0000","last_peered":"2024-04-14T15:18:28.032451+0000","last_clean":"2024-04-14T15:18:28.032451+0000","last_became_active":"2024-04-14T15:15:21.890115+0000","last_became_peered":"2024-04-14T15:15:21.890115+0000","last_unstale":"2024-04-14T15:18:28.032451+0000","last_undegraded":"2024-04-14T15:18:28.032451+0000","last_fullsized":"2024-04-14T15:18:28.032451+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:45.148342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:14:45.148342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":753,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.205356+0000","last_change":"2024-04-14T15:16:51.617717+0000","last_active":"2024-04-14T15:18:28.205356+0000","last_peered":"2024-04-14T15:18:28.205356+0000","last_clean":"2024-04-14T15:18:28.205356+0000","last_became_active":"2024-04-14T15:09:47.047201+0000","last_became_peered":"2024-04-14T15:09:47.047201+0000","last_unstale":"2024-04-14T15:18:28.205356+0000","last_undegraded":"2024-04-14T15:18:28.205356+0000","last_fullsized":"2024-04-14T15:18:28.205356+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:51.617615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:51.617615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"90'24","reported_seq":894,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.035571+0000","last_change":"2024-04-14T15:16:19.842719+0000","last_active":"2024-04-14T15:18:28.035571+0000","last_peered":"2024-04-14T15:18:28.035571+0000","last_clean":"2024-04-14T15:18:28.035571+0000","last_became_active":"2024-04-14T15:09:45.179978+0000","last_became_peered":"2024-04-14T15:09:45.179978+0000","last_unstale":"2024-04-14T15:18:28.035571+0000","last_undegraded":"2024-04-14T15:18:28.035571+0000","last_fullsized":"2024-04-14T15:18:28.035571+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'24","last_scrub_stamp":"2024-04-14T15:16:19.842614+0000","last_deep_scrub":"90'24","last_deep_scrub_stamp":"2024-04-14T15:11:25.120692+0000","last_clean_scrub_stamp":"2024-04-14T15:16:19.842614+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"91'130","reported_seq":1031,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.204738+0000","last_change":"2024-04-14T15:16:43.730573+0000","last_active":"2024-04-14T15:18:28.204738+0000","last_peered":"2024-04-14T15:18:28.204738+0000","last_clean":"2024-04-14T15:18:28.204738+0000","last_became_active":"2024-04-14T15:14:12.858953+0000","last_became_peered":"2024-04-14T15:14:12.858953+0000","last_unstale":"2024-04-14T15:18:28.204738+0000","last_undegraded":"2024-04-14T15:18:28.204738+0000","last_fullsized":"2024-04-14T15:18:28.204738+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:16:43.730442+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:16:43.730442+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":326,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.028007+0000","last_change":"2024-04-14T15:17:35.705777+0000","last_active":"2024-04-14T15:18:28.028007+0000","last_peered":"2024-04-14T15:18:28.028007+0000","last_clean":"2024-04-14T15:18:28.028007+0000","last_became_active":"2024-04-14T15:15:44.885315+0000","last_became_peered":"2024-04-14T15:15:44.885315+0000","last_unstale":"2024-04-14T15:18:28.028007+0000","last_undegraded":"2024-04-14T15:18:28.028007+0000","last_fullsized":"2024-04-14T15:18:28.028007+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:35.705645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:47.627305+0000","last_clean_scrub_stamp":"2024-04-14T15:17:35.705645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:49.306972+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"69'2","reported_seq":453,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.032472+0000","last_change":"2024-04-14T15:16:53.972591+0000","last_active":"2024-04-14T15:18:28.032472+0000","last_peered":"2024-04-14T15:18:28.032472+0000","last_clean":"2024-04-14T15:18:28.032472+0000","last_became_active":"2024-04-14T15:15:21.889851+0000","last_became_peered":"2024-04-14T15:15:21.889851+0000","last_unstale":"2024-04-14T15:18:28.032472+0000","last_undegraded":"2024-04-14T15:18:28.032472+0000","last_fullsized":"2024-04-14T15:18:28.032472+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-04-14T15:16:53.972502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:53.972502+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0058046169999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":318,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.033340+0000","last_change":"2024-04-14T15:16:57.935611+0000","last_active":"2024-04-14T15:18:28.033340+0000","last_peered":"2024-04-14T15:18:28.033340+0000","last_clean":"2024-04-14T15:18:28.033340+0000","last_became_active":"2024-04-14T15:15:44.887934+0000","last_became_peered":"2024-04-14T15:15:44.887934+0000","last_unstale":"2024-04-14T15:18:28.033340+0000","last_undegraded":"2024-04-14T15:18:28.033340+0000","last_fullsized":"2024-04-14T15:18:28.033340+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:57.935436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:57.935436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"91'130","reported_seq":1039,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.204767+0000","last_change":"2024-04-14T15:17:41.509304+0000","last_active":"2024-04-14T15:18:28.204767+0000","last_peered":"2024-04-14T15:18:28.204767+0000","last_clean":"2024-04-14T15:18:28.204767+0000","last_became_active":"2024-04-14T15:14:17.858029+0000","last_became_peered":"2024-04-14T15:14:17.858029+0000","last_unstale":"2024-04-14T15:18:28.204767+0000","last_undegraded":"2024-04-14T15:18:28.204767+0000","last_fullsized":"2024-04-14T15:18:28.204767+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:17:41.509215+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:17:41.509215+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:05.623460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":363,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.031848+0000","last_change":"2024-04-14T15:18:26.438896+0000","last_active":"2024-04-14T15:18:28.031848+0000","last_peered":"2024-04-14T15:18:28.031848+0000","last_clean":"2024-04-14T15:18:28.031848+0000","last_became_active":"2024-04-14T15:15:21.889970+0000","last_became_peered":"2024-04-14T15:15:21.889970+0000","last_unstale":"2024-04-14T15:18:28.031848+0000","last_undegraded":"2024-04-14T15:18:28.031848+0000","last_fullsized":"2024-04-14T15:18:28.031848+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:26.438820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:18:26.438820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:45.768277+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":760,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.029055+0000","last_change":"2024-04-14T15:17:01.653342+0000","last_active":"2024-04-14T15:18:28.029055+0000","last_peered":"2024-04-14T15:18:28.029055+0000","last_clean":"2024-04-14T15:18:28.029055+0000","last_became_active":"2024-04-14T15:09:47.049457+0000","last_became_peered":"2024-04-14T15:09:47.049457+0000","last_unstale":"2024-04-14T15:18:28.029055+0000","last_undegraded":"2024-04-14T15:18:28.029055+0000","last_fullsized":"2024-04-14T15:18:28.029055+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:01.653271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:17:01.653271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:29.819865+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"308'18","reported_seq":791,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.025551+0000","last_change":"2024-04-14T15:16:16.054128+0000","last_active":"2024-04-14T15:18:28.025551+0000","last_peered":"2024-04-14T15:18:28.025551+0000","last_clean":"2024-04-14T15:18:28.025551+0000","last_became_active":"2024-04-14T15:09:45.033883+0000","last_became_peered":"2024-04-14T15:09:45.033883+0000","last_unstale":"2024-04-14T15:18:28.025551+0000","last_undegraded":"2024-04-14T15:18:28.025551+0000","last_fullsized":"2024-04-14T15:18:28.025551+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"308'18","last_scrub_stamp":"2024-04-14T15:16:16.054032+0000","last_deep_scrub":"308'18","last_deep_scrub_stamp":"2024-04-14T15:16:16.054032+0000","last_clean_scrub_stamp":"2024-04-14T15:16:16.054032+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":768,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.027434+0000","last_change":"2024-04-14T15:16:43.686142+0000","last_active":"2024-04-14T15:18:28.027434+0000","last_peered":"2024-04-14T15:18:28.027434+0000","last_clean":"2024-04-14T15:18:28.027434+0000","last_became_active":"2024-04-14T15:09:43.021188+0000","last_became_peered":"2024-04-14T15:09:43.021188+0000","last_unstale":"2024-04-14T15:18:28.027434+0000","last_undegraded":"2024-04-14T15:18:28.027434+0000","last_fullsized":"2024-04-14T15:18:28.027434+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:43.686033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:43.686033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"73'5","reported_seq":1300,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.021940+0000","last_change":"2024-04-14T15:18:26.955802+0000","last_active":"2024-04-14T15:18:28.021940+0000","last_peered":"2024-04-14T15:18:28.021940+0000","last_clean":"2024-04-14T15:18:28.021940+0000","last_became_active":"2024-04-14T15:09:43.016544+0000","last_became_peered":"2024-04-14T15:09:43.016544+0000","last_unstale":"2024-04-14T15:18:28.021940+0000","last_undegraded":"2024-04-14T15:18:28.021940+0000","last_fullsized":"2024-04-14T15:18:28.021940+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-04-14T15:18:26.955711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:18:26.955711+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:54.355074+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00056542099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":519,"num_read_kb":514,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":749,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.205735+0000","last_change":"2024-04-14T15:16:20.827681+0000","last_active":"2024-04-14T15:18:28.205735+0000","last_peered":"2024-04-14T15:18:28.205735+0000","last_clean":"2024-04-14T15:18:28.205735+0000","last_became_active":"2024-04-14T15:09:49.130307+0000","last_became_peered":"2024-04-14T15:09:49.130307+0000","last_unstale":"2024-04-14T15:18:28.205735+0000","last_undegraded":"2024-04-14T15:18:28.205735+0000","last_fullsized":"2024-04-14T15:18:28.205735+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:20.827589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:20.827589+0000","last_clean_scrub_stamp":"2024-04-14T15:16:20.827589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"91'130","reported_seq":645,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.020604+0000","last_change":"2024-04-14T15:15:45.362962+0000","last_active":"2024-04-14T15:18:28.020604+0000","last_peered":"2024-04-14T15:18:28.020604+0000","last_clean":"2024-04-14T15:18:28.020604+0000","last_became_active":"2024-04-14T15:15:44.884623+0000","last_became_peered":"2024-04-14T15:15:44.884623+0000","last_unstale":"2024-04-14T15:18:28.020604+0000","last_undegraded":"2024-04-14T15:18:28.020604+0000","last_fullsized":"2024-04-14T15:18:28.020604+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:15:45.362860+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:15:45.362860+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":363,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.032376+0000","last_change":"2024-04-14T15:15:32.839576+0000","last_active":"2024-04-14T15:18:28.032376+0000","last_peered":"2024-04-14T15:18:28.032376+0000","last_clean":"2024-04-14T15:18:28.032376+0000","last_became_active":"2024-04-14T15:15:21.883717+0000","last_became_peered":"2024-04-14T15:15:21.883717+0000","last_unstale":"2024-04-14T15:18:28.032376+0000","last_undegraded":"2024-04-14T15:18:28.032376+0000","last_fullsized":"2024-04-14T15:18:28.032376+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:32.839491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:32.839491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":755,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.021490+0000","last_change":"2024-04-14T15:16:53.731699+0000","last_active":"2024-04-14T15:18:28.021490+0000","last_peered":"2024-04-14T15:18:28.021490+0000","last_clean":"2024-04-14T15:18:28.021490+0000","last_became_active":"2024-04-14T15:09:47.040185+0000","last_became_peered":"2024-04-14T15:09:47.040185+0000","last_unstale":"2024-04-14T15:18:28.021490+0000","last_undegraded":"2024-04-14T15:18:28.021490+0000","last_fullsized":"2024-04-14T15:18:28.021490+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:53.731610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:53.731610+0000","last_clean_scrub_stamp":"2024-04-14T15:16:53.731610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"72'28","reported_seq":913,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.023967+0000","last_change":"2024-04-14T15:16:43.036263+0000","last_active":"2024-04-14T15:18:28.023967+0000","last_peered":"2024-04-14T15:18:28.023967+0000","last_clean":"2024-04-14T15:18:28.023967+0000","last_became_active":"2024-04-14T15:09:45.034168+0000","last_became_peered":"2024-04-14T15:09:45.034168+0000","last_unstale":"2024-04-14T15:18:28.023967+0000","last_undegraded":"2024-04-14T15:18:28.023967+0000","last_fullsized":"2024-04-14T15:18:28.023967+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'28","last_scrub_stamp":"2024-04-14T15:16:43.036159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:16:43.036159+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":321,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.023232+0000","last_change":"2024-04-14T15:16:19.101437+0000","last_active":"2024-04-14T15:18:28.023232+0000","last_peered":"2024-04-14T15:18:28.023232+0000","last_clean":"2024-04-14T15:18:28.023232+0000","last_became_active":"2024-04-14T15:15:44.885315+0000","last_became_peered":"2024-04-14T15:15:44.885315+0000","last_unstale":"2024-04-14T15:18:28.023232+0000","last_undegraded":"2024-04-14T15:18:28.023232+0000","last_fullsized":"2024-04-14T15:18:28.023232+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:19.101368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:19.101368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":423,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.033266+0000","last_change":"2024-04-14T15:17:44.084142+0000","last_active":"2024-04-14T15:18:28.033266+0000","last_peered":"2024-04-14T15:18:28.033266+0000","last_clean":"2024-04-14T15:18:28.033266+0000","last_became_active":"2024-04-14T15:15:21.874742+0000","last_became_peered":"2024-04-14T15:15:21.874742+0000","last_unstale":"2024-04-14T15:18:28.033266+0000","last_undegraded":"2024-04-14T15:18:28.033266+0000","last_fullsized":"2024-04-14T15:18:28.033266+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:44.084010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:44.084010+0000","last_clean_scrub_stamp":"2024-04-14T15:17:44.084010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:03.733939+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"91'130","reported_seq":827,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.036110+0000","last_change":"2024-04-14T15:17:40.725976+0000","last_active":"2024-04-14T15:18:28.036110+0000","last_peered":"2024-04-14T15:18:28.036110+0000","last_clean":"2024-04-14T15:18:28.036110+0000","last_became_active":"2024-04-14T15:14:17.859910+0000","last_became_peered":"2024-04-14T15:14:17.859910+0000","last_unstale":"2024-04-14T15:18:28.036110+0000","last_undegraded":"2024-04-14T15:18:28.036110+0000","last_fullsized":"2024-04-14T15:18:28.036110+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:17:40.725873+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:17:40.725873+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:54.360389+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":664,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.027914+0000","last_change":"2024-04-14T15:17:00.656174+0000","last_active":"2024-04-14T15:18:28.027914+0000","last_peered":"2024-04-14T15:18:28.027914+0000","last_clean":"2024-04-14T15:18:28.027914+0000","last_became_active":"2024-04-14T15:12:56.350703+0000","last_became_peered":"2024-04-14T15:12:56.350703+0000","last_unstale":"2024-04-14T15:18:28.027914+0000","last_undegraded":"2024-04-14T15:18:28.027914+0000","last_fullsized":"2024-04-14T15:18:28.027914+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:00.656085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:17:00.656085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"72'8","reported_seq":877,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.022232+0000","last_change":"2024-04-14T15:17:04.695161+0000","last_active":"2024-04-14T15:18:28.022232+0000","last_peered":"2024-04-14T15:18:28.022232+0000","last_clean":"2024-04-14T15:18:28.022232+0000","last_became_active":"2024-04-14T15:09:47.047699+0000","last_became_peered":"2024-04-14T15:09:47.047699+0000","last_unstale":"2024-04-14T15:18:28.022232+0000","last_undegraded":"2024-04-14T15:18:28.022232+0000","last_fullsized":"2024-04-14T15:18:28.022232+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:17:04.695079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:17:04.695079+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"72'10","reported_seq":750,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.027780+0000","last_change":"2024-04-14T15:15:41.610385+0000","last_active":"2024-04-14T15:18:28.027780+0000","last_peered":"2024-04-14T15:18:28.027780+0000","last_clean":"2024-04-14T15:18:28.027780+0000","last_became_active":"2024-04-14T15:10:07.764042+0000","last_became_peered":"2024-04-14T15:10:07.764042+0000","last_unstale":"2024-04-14T15:18:28.027780+0000","last_undegraded":"2024-04-14T15:18:28.027780+0000","last_fullsized":"2024-04-14T15:18:28.027780+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-04-14T15:15:41.610279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:15:41.610279+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":763,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.023825+0000","last_change":"2024-04-14T15:17:40.971492+0000","last_active":"2024-04-14T15:18:28.023825+0000","last_peered":"2024-04-14T15:18:28.023825+0000","last_clean":"2024-04-14T15:18:28.023825+0000","last_became_active":"2024-04-14T15:09:43.022597+0000","last_became_peered":"2024-04-14T15:09:43.022597+0000","last_unstale":"2024-04-14T15:18:28.023825+0000","last_undegraded":"2024-04-14T15:18:28.023825+0000","last_fullsized":"2024-04-14T15:18:28.023825+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:40.971412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:40.971412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:08.233588+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":751,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.020661+0000","last_change":"2024-04-14T15:16:17.405931+0000","last_active":"2024-04-14T15:18:28.020661+0000","last_peered":"2024-04-14T15:18:28.020661+0000","last_clean":"2024-04-14T15:18:28.020661+0000","last_became_active":"2024-04-14T15:09:49.066648+0000","last_became_peered":"2024-04-14T15:09:49.066648+0000","last_unstale":"2024-04-14T15:18:28.020661+0000","last_undegraded":"2024-04-14T15:18:28.020661+0000","last_fullsized":"2024-04-14T15:18:28.020661+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:17.405850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:17.405850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":667,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.023703+0000","last_change":"2024-04-14T15:18:27.133391+0000","last_active":"2024-04-14T15:18:28.023703+0000","last_peered":"2024-04-14T15:18:28.023703+0000","last_clean":"2024-04-14T15:18:28.023703+0000","last_became_active":"2024-04-14T15:12:56.563074+0000","last_became_peered":"2024-04-14T15:12:56.563074+0000","last_unstale":"2024-04-14T15:18:28.023703+0000","last_undegraded":"2024-04-14T15:18:28.023703+0000","last_fullsized":"2024-04-14T15:18:28.023703+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:27.133310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:18:27.133310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:38.787205+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":760,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.027227+0000","last_change":"2024-04-14T15:17:56.915660+0000","last_active":"2024-04-14T15:18:28.027227+0000","last_peered":"2024-04-14T15:18:28.027227+0000","last_clean":"2024-04-14T15:18:28.027227+0000","last_became_active":"2024-04-14T15:09:47.049005+0000","last_became_peered":"2024-04-14T15:09:47.049005+0000","last_unstale":"2024-04-14T15:18:28.027227+0000","last_undegraded":"2024-04-14T15:18:28.027227+0000","last_fullsized":"2024-04-14T15:18:28.027227+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:56.915544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:56.915544+0000","last_clean_scrub_stamp":"2024-04-14T15:17:56.915544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:24.542571+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"183'17","reported_seq":793,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.020486+0000","last_change":"2024-04-14T15:16:17.997616+0000","last_active":"2024-04-14T15:18:28.020486+0000","last_peered":"2024-04-14T15:18:28.020486+0000","last_clean":"2024-04-14T15:18:28.020486+0000","last_became_active":"2024-04-14T15:09:45.032892+0000","last_became_peered":"2024-04-14T15:09:45.032892+0000","last_unstale":"2024-04-14T15:18:28.020486+0000","last_undegraded":"2024-04-14T15:18:28.020486+0000","last_fullsized":"2024-04-14T15:18:28.020486+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'17","last_scrub_stamp":"2024-04-14T15:16:17.997529+0000","last_deep_scrub":"183'17","last_deep_scrub_stamp":"2024-04-14T15:14:11.577574+0000","last_clean_scrub_stamp":"2024-04-14T15:16:17.997529+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"65'3","reported_seq":778,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.021505+0000","last_change":"2024-04-14T15:16:12.030448+0000","last_active":"2024-04-14T15:18:28.021505+0000","last_peered":"2024-04-14T15:18:28.021505+0000","last_clean":"2024-04-14T15:18:28.021505+0000","last_became_active":"2024-04-14T15:09:43.020490+0000","last_became_peered":"2024-04-14T15:09:43.020490+0000","last_unstale":"2024-04-14T15:18:28.021505+0000","last_undegraded":"2024-04-14T15:18:28.021505+0000","last_fullsized":"2024-04-14T15:18:28.021505+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3","last_scrub_stamp":"2024-04-14T15:16:12.030363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:12.030363+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":528,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":423,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.032621+0000","last_change":"2024-04-14T15:16:58.942591+0000","last_active":"2024-04-14T15:18:28.032621+0000","last_peered":"2024-04-14T15:18:28.032621+0000","last_clean":"2024-04-14T15:18:28.032621+0000","last_became_active":"2024-04-14T15:15:21.876410+0000","last_became_peered":"2024-04-14T15:15:21.876410+0000","last_unstale":"2024-04-14T15:18:28.032621+0000","last_undegraded":"2024-04-14T15:18:28.032621+0000","last_fullsized":"2024-04-14T15:18:28.032621+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:58.942499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:58.942499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":667,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.020861+0000","last_change":"2024-04-14T15:16:43.696614+0000","last_active":"2024-04-14T15:18:28.020861+0000","last_peered":"2024-04-14T15:18:28.020861+0000","last_clean":"2024-04-14T15:18:28.020861+0000","last_became_active":"2024-04-14T15:12:56.551177+0000","last_became_peered":"2024-04-14T15:12:56.551177+0000","last_unstale":"2024-04-14T15:18:28.020861+0000","last_undegraded":"2024-04-14T15:18:28.020861+0000","last_fullsized":"2024-04-14T15:18:28.020861+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:43.696536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:16:43.696536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"347'13","reported_seq":778,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.023973+0000","last_change":"2024-04-14T15:16:55.694703+0000","last_active":"2024-04-14T15:18:28.023973+0000","last_peered":"2024-04-14T15:18:28.023973+0000","last_clean":"2024-04-14T15:18:28.023973+0000","last_became_active":"2024-04-14T15:09:45.180375+0000","last_became_peered":"2024-04-14T15:09:45.180375+0000","last_unstale":"2024-04-14T15:18:28.023973+0000","last_undegraded":"2024-04-14T15:18:28.023973+0000","last_fullsized":"2024-04-14T15:18:28.023973+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"347'13","last_scrub_stamp":"2024-04-14T15:16:55.694599+0000","last_deep_scrub":"347'13","last_deep_scrub_stamp":"2024-04-14T15:16:55.694599+0000","last_clean_scrub_stamp":"2024-04-14T15:16:55.694599+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":767,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.027165+0000","last_change":"2024-04-14T15:16:41.677902+0000","last_active":"2024-04-14T15:18:28.027165+0000","last_peered":"2024-04-14T15:18:28.027165+0000","last_clean":"2024-04-14T15:18:28.027165+0000","last_became_active":"2024-04-14T15:09:43.016788+0000","last_became_peered":"2024-04-14T15:09:43.016788+0000","last_unstale":"2024-04-14T15:18:28.027165+0000","last_undegraded":"2024-04-14T15:18:28.027165+0000","last_fullsized":"2024-04-14T15:18:28.027165+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:41.677826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:41.677826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":435,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.033268+0000","last_change":"2024-04-14T15:15:42.769452+0000","last_active":"2024-04-14T15:18:28.033268+0000","last_peered":"2024-04-14T15:18:28.033268+0000","last_clean":"2024-04-14T15:18:28.033268+0000","last_became_active":"2024-04-14T15:15:21.873580+0000","last_became_peered":"2024-04-14T15:15:21.873580+0000","last_unstale":"2024-04-14T15:18:28.033268+0000","last_undegraded":"2024-04-14T15:18:28.033268+0000","last_fullsized":"2024-04-14T15:18:28.033268+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:42.769375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:15:42.769375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":755,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.027973+0000","last_change":"2024-04-14T15:16:47.671808+0000","last_active":"2024-04-14T15:18:28.027973+0000","last_peered":"2024-04-14T15:18:28.027973+0000","last_clean":"2024-04-14T15:18:28.027973+0000","last_became_active":"2024-04-14T15:09:49.064296+0000","last_became_peered":"2024-04-14T15:09:49.064296+0000","last_unstale":"2024-04-14T15:18:28.027973+0000","last_undegraded":"2024-04-14T15:18:28.027973+0000","last_fullsized":"2024-04-14T15:18:28.027973+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:47.671717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:47.671717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":664,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.027040+0000","last_change":"2024-04-14T15:17:36.728543+0000","last_active":"2024-04-14T15:18:28.027040+0000","last_peered":"2024-04-14T15:18:28.027040+0000","last_clean":"2024-04-14T15:18:28.027040+0000","last_became_active":"2024-04-14T15:12:56.340066+0000","last_became_peered":"2024-04-14T15:12:56.340066+0000","last_unstale":"2024-04-14T15:18:28.027040+0000","last_undegraded":"2024-04-14T15:18:28.027040+0000","last_fullsized":"2024-04-14T15:18:28.027040+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:36.728465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:17:36.728465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:54.134352+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.17","version":"72'6","reported_seq":771,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.033245+0000","last_change":"2024-04-14T15:17:42.019838+0000","last_active":"2024-04-14T15:18:28.033245+0000","last_peered":"2024-04-14T15:18:28.033245+0000","last_clean":"2024-04-14T15:18:28.033245+0000","last_became_active":"2024-04-14T15:09:45.180037+0000","last_became_peered":"2024-04-14T15:09:45.180037+0000","last_unstale":"2024-04-14T15:18:28.033245+0000","last_undegraded":"2024-04-14T15:18:28.033245+0000","last_fullsized":"2024-04-14T15:18:28.033245+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'6","last_scrub_stamp":"2024-04-14T15:17:42.019717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:17:42.019717+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:52.276684+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":769,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.034472+0000","last_change":"2024-04-14T15:16:59.640700+0000","last_active":"2024-04-14T15:18:28.034472+0000","last_peered":"2024-04-14T15:18:28.034472+0000","last_clean":"2024-04-14T15:18:28.034472+0000","last_became_active":"2024-04-14T15:09:43.018987+0000","last_became_peered":"2024-04-14T15:09:43.018987+0000","last_unstale":"2024-04-14T15:18:28.034472+0000","last_undegraded":"2024-04-14T15:18:28.034472+0000","last_fullsized":"2024-04-14T15:18:28.034472+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:59.640565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:59.640565+0000","last_clean_scrub_stamp":"2024-04-14T15:16:59.640565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":317,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.033176+0000","last_change":"2024-04-14T15:17:47.072247+0000","last_active":"2024-04-14T15:18:28.033176+0000","last_peered":"2024-04-14T15:18:28.033176+0000","last_clean":"2024-04-14T15:18:28.033176+0000","last_became_active":"2024-04-14T15:15:44.889786+0000","last_became_peered":"2024-04-14T15:15:44.889786+0000","last_unstale":"2024-04-14T15:18:28.033176+0000","last_undegraded":"2024-04-14T15:18:28.033176+0000","last_fullsized":"2024-04-14T15:18:28.033176+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:47.072170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:13.735024+0000","last_clean_scrub_stamp":"2024-04-14T15:17:47.072170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:03.533500+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":748,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.205944+0000","last_change":"2024-04-14T15:16:54.720823+0000","last_active":"2024-04-14T15:18:28.205944+0000","last_peered":"2024-04-14T15:18:28.205944+0000","last_clean":"2024-04-14T15:18:28.205944+0000","last_became_active":"2024-04-14T15:09:49.130438+0000","last_became_peered":"2024-04-14T15:09:49.130438+0000","last_unstale":"2024-04-14T15:18:28.205944+0000","last_undegraded":"2024-04-14T15:18:28.205944+0000","last_fullsized":"2024-04-14T15:18:28.205944+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:54.720722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:54.720722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":665,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.020793+0000","last_change":"2024-04-14T15:17:46.369320+0000","last_active":"2024-04-14T15:18:28.020793+0000","last_peered":"2024-04-14T15:18:28.020793+0000","last_clean":"2024-04-14T15:18:28.020793+0000","last_became_active":"2024-04-14T15:12:56.203577+0000","last_became_peered":"2024-04-14T15:12:56.203577+0000","last_unstale":"2024-04-14T15:18:28.020793+0000","last_undegraded":"2024-04-14T15:18:28.020793+0000","last_fullsized":"2024-04-14T15:18:28.020793+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:46.369231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:46.369231+0000","last_clean_scrub_stamp":"2024-04-14T15:17:46.369231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:57.370694+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"72'10","reported_seq":454,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.032836+0000","last_change":"2024-04-14T15:15:30.806041+0000","last_active":"2024-04-14T15:18:28.032836+0000","last_peered":"2024-04-14T15:18:28.032836+0000","last_clean":"2024-04-14T15:18:28.032836+0000","last_became_active":"2024-04-14T15:15:21.877667+0000","last_became_peered":"2024-04-14T15:15:21.877667+0000","last_unstale":"2024-04-14T15:18:28.032836+0000","last_undegraded":"2024-04-14T15:18:28.032836+0000","last_fullsized":"2024-04-14T15:18:28.032836+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-04-14T15:15:30.805923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:15:30.805923+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":317,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.032600+0000","last_change":"2024-04-14T15:16:47.075044+0000","last_active":"2024-04-14T15:18:28.032600+0000","last_peered":"2024-04-14T15:18:28.032600+0000","last_clean":"2024-04-14T15:18:28.032600+0000","last_became_active":"2024-04-14T15:15:44.887452+0000","last_became_peered":"2024-04-14T15:15:44.887452+0000","last_unstale":"2024-04-14T15:18:28.032600+0000","last_undegraded":"2024-04-14T15:18:28.032600+0000","last_fullsized":"2024-04-14T15:18:28.032600+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:47.074950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:47.074950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"280'9","reported_seq":379,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.024470+0000","last_change":"2024-04-14T15:17:50.908130+0000","last_active":"2024-04-14T15:18:28.024470+0000","last_peered":"2024-04-14T15:18:28.024470+0000","last_clean":"2024-04-14T15:18:28.024470+0000","last_became_active":"2024-04-14T15:15:44.882916+0000","last_became_peered":"2024-04-14T15:15:44.882916+0000","last_unstale":"2024-04-14T15:18:28.024470+0000","last_undegraded":"2024-04-14T15:18:28.024470+0000","last_fullsized":"2024-04-14T15:18:28.024470+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"280'9","last_scrub_stamp":"2024-04-14T15:17:50.908054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:17:50.908054+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:20.243489+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":750,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.023787+0000","last_change":"2024-04-14T15:16:54.724807+0000","last_active":"2024-04-14T15:18:28.023787+0000","last_peered":"2024-04-14T15:18:28.023787+0000","last_clean":"2024-04-14T15:18:28.023787+0000","last_became_active":"2024-04-14T15:09:49.066068+0000","last_became_peered":"2024-04-14T15:09:49.066068+0000","last_unstale":"2024-04-14T15:18:28.023787+0000","last_undegraded":"2024-04-14T15:18:28.023787+0000","last_fullsized":"2024-04-14T15:18:28.023787+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:54.724681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:54.724681+0000","last_clean_scrub_stamp":"2024-04-14T15:16:54.724681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":309,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.032297+0000","last_change":"2024-04-14T15:15:44.889470+0000","last_active":"2024-04-14T15:18:28.032297+0000","last_peered":"2024-04-14T15:18:28.032297+0000","last_clean":"2024-04-14T15:18:28.032297+0000","last_became_active":"2024-04-14T15:15:44.889106+0000","last_became_peered":"2024-04-14T15:15:44.889106+0000","last_unstale":"2024-04-14T15:18:28.032297+0000","last_undegraded":"2024-04-14T15:18:28.032297+0000","last_fullsized":"2024-04-14T15:18:28.032297+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:01.356001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:01.356001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"183'13","reported_seq":598,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.205860+0000","last_change":"2024-04-14T15:17:37.421888+0000","last_active":"2024-04-14T15:18:28.205860+0000","last_peered":"2024-04-14T15:18:28.205860+0000","last_clean":"2024-04-14T15:18:28.205860+0000","last_became_active":"2024-04-14T15:15:44.884211+0000","last_became_peered":"2024-04-14T15:15:44.884211+0000","last_unstale":"2024-04-14T15:18:28.205860+0000","last_undegraded":"2024-04-14T15:18:28.205860+0000","last_fullsized":"2024-04-14T15:18:28.205860+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'13","last_scrub_stamp":"2024-04-14T15:17:37.421739+0000","last_deep_scrub":"183'13","last_deep_scrub_stamp":"2024-04-14T15:14:11.755252+0000","last_clean_scrub_stamp":"2024-04-14T15:17:37.421739+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:55.179831+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":770,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.022419+0000","last_change":"2024-04-14T15:17:02.670294+0000","last_active":"2024-04-14T15:18:28.022419+0000","last_peered":"2024-04-14T15:18:28.022419+0000","last_clean":"2024-04-14T15:18:28.022419+0000","last_became_active":"2024-04-14T15:09:43.019301+0000","last_became_peered":"2024-04-14T15:09:43.019301+0000","last_unstale":"2024-04-14T15:18:28.022419+0000","last_undegraded":"2024-04-14T15:18:28.022419+0000","last_fullsized":"2024-04-14T15:18:28.022419+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:02.670213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:02.670213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"255'10","reported_seq":512,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.033116+0000","last_change":"2024-04-14T15:16:59.946889+0000","last_active":"2024-04-14T15:18:28.033116+0000","last_peered":"2024-04-14T15:18:28.033116+0000","last_clean":"2024-04-14T15:18:28.033116+0000","last_became_active":"2024-04-14T15:15:21.884468+0000","last_became_peered":"2024-04-14T15:15:21.884468+0000","last_unstale":"2024-04-14T15:18:28.033116+0000","last_undegraded":"2024-04-14T15:18:28.033116+0000","last_fullsized":"2024-04-14T15:18:28.033116+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'10","last_scrub_stamp":"2024-04-14T15:16:59.946804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:59.946804+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":748,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.021709+0000","last_change":"2024-04-14T15:16:03.948252+0000","last_active":"2024-04-14T15:18:28.021709+0000","last_peered":"2024-04-14T15:18:28.021709+0000","last_clean":"2024-04-14T15:18:28.021709+0000","last_became_active":"2024-04-14T15:09:49.065242+0000","last_became_peered":"2024-04-14T15:09:49.065242+0000","last_unstale":"2024-04-14T15:18:28.021709+0000","last_undegraded":"2024-04-14T15:18:28.021709+0000","last_fullsized":"2024-04-14T15:18:28.021709+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:03.948140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:03.948140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":657,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.205990+0000","last_change":"2024-04-14T15:15:34.555810+0000","last_active":"2024-04-14T15:18:28.205990+0000","last_peered":"2024-04-14T15:18:28.205990+0000","last_clean":"2024-04-14T15:18:28.205990+0000","last_became_active":"2024-04-14T15:12:56.389204+0000","last_became_peered":"2024-04-14T15:12:56.389204+0000","last_unstale":"2024-04-14T15:18:28.205990+0000","last_undegraded":"2024-04-14T15:18:28.205990+0000","last_fullsized":"2024-04-14T15:18:28.205990+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:34.555728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:34.555728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"344'13","reported_seq":288,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.034767+0000","last_change":"2024-04-14T15:16:52.738298+0000","last_active":"2024-04-14T15:18:28.034767+0000","last_peered":"2024-04-14T15:18:28.034767+0000","last_clean":"2024-04-14T15:18:28.034767+0000","last_became_active":"2024-04-14T15:16:24.993130+0000","last_became_peered":"2024-04-14T15:16:24.993130+0000","last_unstale":"2024-04-14T15:18:28.034767+0000","last_undegraded":"2024-04-14T15:18:28.034767+0000","last_fullsized":"2024-04-14T15:18:28.034767+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"344'13","last_scrub_stamp":"2024-04-14T15:16:52.738163+0000","last_deep_scrub":"344'13","last_deep_scrub_stamp":"2024-04-14T15:16:52.738163+0000","last_clean_scrub_stamp":"2024-04-14T15:16:52.738163+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":769,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.034931+0000","last_change":"2024-04-14T15:17:04.625913+0000","last_active":"2024-04-14T15:18:28.034931+0000","last_peered":"2024-04-14T15:18:28.034931+0000","last_clean":"2024-04-14T15:18:28.034931+0000","last_became_active":"2024-04-14T15:09:43.022480+0000","last_became_peered":"2024-04-14T15:09:43.022480+0000","last_unstale":"2024-04-14T15:18:28.034931+0000","last_undegraded":"2024-04-14T15:18:28.034931+0000","last_fullsized":"2024-04-14T15:18:28.034931+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:04.625824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:04.625824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:28.834815+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":427,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.032514+0000","last_change":"2024-04-14T15:15:21.885176+0000","last_active":"2024-04-14T15:18:28.032514+0000","last_peered":"2024-04-14T15:18:28.032514+0000","last_clean":"2024-04-14T15:18:28.032514+0000","last_became_active":"2024-04-14T15:15:21.882643+0000","last_became_peered":"2024-04-14T15:15:21.882643+0000","last_unstale":"2024-04-14T15:18:28.032514+0000","last_undegraded":"2024-04-14T15:18:28.032514+0000","last_fullsized":"2024-04-14T15:18:28.032514+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:14:54.252608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:14:54.252608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":761,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.020545+0000","last_change":"2024-04-14T15:18:24.039316+0000","last_active":"2024-04-14T15:18:28.020545+0000","last_peered":"2024-04-14T15:18:28.020545+0000","last_clean":"2024-04-14T15:18:28.020545+0000","last_became_active":"2024-04-14T15:09:49.059711+0000","last_became_peered":"2024-04-14T15:09:49.059711+0000","last_unstale":"2024-04-14T15:18:28.020545+0000","last_undegraded":"2024-04-14T15:18:28.020545+0000","last_fullsized":"2024-04-14T15:18:28.020545+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:24.039237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:18:24.039237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:25.190541+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":666,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.034754+0000","last_change":"2024-04-14T15:17:51.921346+0000","last_active":"2024-04-14T15:18:28.034754+0000","last_peered":"2024-04-14T15:18:28.034754+0000","last_clean":"2024-04-14T15:18:28.034754+0000","last_became_active":"2024-04-14T15:12:56.396294+0000","last_became_peered":"2024-04-14T15:12:56.396294+0000","last_unstale":"2024-04-14T15:18:28.034754+0000","last_undegraded":"2024-04-14T15:18:28.034754+0000","last_fullsized":"2024-04-14T15:18:28.034754+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:51.921269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:17:51.921269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:54.899694+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"72'11","reported_seq":775,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.020940+0000","last_change":"2024-04-14T15:16:08.917185+0000","last_active":"2024-04-14T15:18:28.020940+0000","last_peered":"2024-04-14T15:18:28.020940+0000","last_clean":"2024-04-14T15:18:28.020940+0000","last_became_active":"2024-04-14T15:09:45.026065+0000","last_became_peered":"2024-04-14T15:09:45.026065+0000","last_unstale":"2024-04-14T15:18:28.020940+0000","last_undegraded":"2024-04-14T15:18:28.020940+0000","last_fullsized":"2024-04-14T15:18:28.020940+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-04-14T15:16:08.917092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:16:08.917092+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":770,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.024607+0000","last_change":"2024-04-14T15:17:34.986735+0000","last_active":"2024-04-14T15:18:28.024607+0000","last_peered":"2024-04-14T15:18:28.024607+0000","last_clean":"2024-04-14T15:18:28.024607+0000","last_became_active":"2024-04-14T15:09:43.023583+0000","last_became_peered":"2024-04-14T15:09:43.023583+0000","last_unstale":"2024-04-14T15:18:28.024607+0000","last_undegraded":"2024-04-14T15:18:28.024607+0000","last_fullsized":"2024-04-14T15:18:28.024607+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:34.986655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:34.986655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:41.802704+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":754,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.023509+0000","last_change":"2024-04-14T15:17:00.205888+0000","last_active":"2024-04-14T15:18:28.023509+0000","last_peered":"2024-04-14T15:18:28.023509+0000","last_clean":"2024-04-14T15:18:28.023509+0000","last_became_active":"2024-04-14T15:09:47.041683+0000","last_became_peered":"2024-04-14T15:09:47.041683+0000","last_unstale":"2024-04-14T15:18:28.023509+0000","last_undegraded":"2024-04-14T15:18:28.023509+0000","last_fullsized":"2024-04-14T15:18:28.023509+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:00.205808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:08.486217+0000","last_clean_scrub_stamp":"2024-04-14T15:17:00.205808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":431,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.032130+0000","last_change":"2024-04-14T15:16:50.021545+0000","last_active":"2024-04-14T15:18:28.032130+0000","last_peered":"2024-04-14T15:18:28.032130+0000","last_clean":"2024-04-14T15:18:28.032130+0000","last_became_active":"2024-04-14T15:15:21.889567+0000","last_became_peered":"2024-04-14T15:15:21.889567+0000","last_unstale":"2024-04-14T15:18:28.032130+0000","last_undegraded":"2024-04-14T15:18:28.032130+0000","last_fullsized":"2024-04-14T15:18:28.032130+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:50.021470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:50.021470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":664,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.028229+0000","last_change":"2024-04-14T15:17:03.685135+0000","last_active":"2024-04-14T15:18:28.028229+0000","last_peered":"2024-04-14T15:18:28.028229+0000","last_clean":"2024-04-14T15:18:28.028229+0000","last_became_active":"2024-04-14T15:12:56.387723+0000","last_became_peered":"2024-04-14T15:12:56.387723+0000","last_unstale":"2024-04-14T15:18:28.028229+0000","last_undegraded":"2024-04-14T15:18:28.028229+0000","last_fullsized":"2024-04-14T15:18:28.028229+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:03.685045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:03.685045+0000","last_clean_scrub_stamp":"2024-04-14T15:17:03.685045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.10","version":"72'4","reported_seq":768,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.031708+0000","last_change":"2024-04-14T15:17:53.181132+0000","last_active":"2024-04-14T15:18:28.031708+0000","last_peered":"2024-04-14T15:18:28.031708+0000","last_clean":"2024-04-14T15:18:28.031708+0000","last_became_active":"2024-04-14T15:09:45.038815+0000","last_became_peered":"2024-04-14T15:09:45.038815+0000","last_unstale":"2024-04-14T15:18:28.031708+0000","last_undegraded":"2024-04-14T15:18:28.031708+0000","last_fullsized":"2024-04-14T15:18:28.031708+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4","last_scrub_stamp":"2024-04-14T15:17:53.181053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:17:53.181053+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:15.210428+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":320,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.036107+0000","last_change":"2024-04-14T15:17:48.845163+0000","last_active":"2024-04-14T15:18:28.036107+0000","last_peered":"2024-04-14T15:18:28.036107+0000","last_clean":"2024-04-14T15:18:28.036107+0000","last_became_active":"2024-04-14T15:15:44.884730+0000","last_became_peered":"2024-04-14T15:15:44.884730+0000","last_unstale":"2024-04-14T15:18:28.036107+0000","last_undegraded":"2024-04-14T15:18:28.036107+0000","last_fullsized":"2024-04-14T15:18:28.036107+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:48.845081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:08.712289+0000","last_clean_scrub_stamp":"2024-04-14T15:17:48.845081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:15.427392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":761,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.034669+0000","last_change":"2024-04-14T15:16:13.909554+0000","last_active":"2024-04-14T15:18:28.034669+0000","last_peered":"2024-04-14T15:18:28.034669+0000","last_clean":"2024-04-14T15:18:28.034669+0000","last_became_active":"2024-04-14T15:09:47.041074+0000","last_became_peered":"2024-04-14T15:09:47.041074+0000","last_unstale":"2024-04-14T15:18:28.034669+0000","last_undegraded":"2024-04-14T15:18:28.034669+0000","last_fullsized":"2024-04-14T15:18:28.034669+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:13.909454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:13.909454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":748,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.205246+0000","last_change":"2024-04-14T15:16:44.728025+0000","last_active":"2024-04-14T15:18:28.205246+0000","last_peered":"2024-04-14T15:18:28.205246+0000","last_clean":"2024-04-14T15:18:28.205246+0000","last_became_active":"2024-04-14T15:09:49.061359+0000","last_became_peered":"2024-04-14T15:09:49.061359+0000","last_unstale":"2024-04-14T15:18:28.205246+0000","last_undegraded":"2024-04-14T15:18:28.205246+0000","last_fullsized":"2024-04-14T15:18:28.205246+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:44.727907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:44.727907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":657,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.205997+0000","last_change":"2024-04-14T15:15:43.680936+0000","last_active":"2024-04-14T15:18:28.205997+0000","last_peered":"2024-04-14T15:18:28.205997+0000","last_clean":"2024-04-14T15:18:28.205997+0000","last_became_active":"2024-04-14T15:12:56.352528+0000","last_became_peered":"2024-04-14T15:12:56.352528+0000","last_unstale":"2024-04-14T15:18:28.205997+0000","last_undegraded":"2024-04-14T15:18:28.205997+0000","last_fullsized":"2024-04-14T15:18:28.205997+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:43.680852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:15:43.680852+0000","last_clean_scrub_stamp":"2024-04-14T15:15:43.680852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"102'16","reported_seq":424,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.032566+0000","last_change":"2024-04-14T15:17:43.061551+0000","last_active":"2024-04-14T15:18:28.032566+0000","last_peered":"2024-04-14T15:18:28.032566+0000","last_clean":"2024-04-14T15:18:28.032566+0000","last_became_active":"2024-04-14T15:15:21.884783+0000","last_became_peered":"2024-04-14T15:15:21.884783+0000","last_unstale":"2024-04-14T15:18:28.032566+0000","last_undegraded":"2024-04-14T15:18:28.032566+0000","last_fullsized":"2024-04-14T15:18:28.032566+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'16","last_scrub_stamp":"2024-04-14T15:17:43.061458+0000","last_deep_scrub":"102'16","last_deep_scrub_stamp":"2024-04-14T15:12:35.999326+0000","last_clean_scrub_stamp":"2024-04-14T15:17:43.061458+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:06.716506+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":769,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.036249+0000","last_change":"2024-04-14T15:17:01.643838+0000","last_active":"2024-04-14T15:18:28.036249+0000","last_peered":"2024-04-14T15:18:28.036249+0000","last_clean":"2024-04-14T15:18:28.036249+0000","last_became_active":"2024-04-14T15:09:43.023220+0000","last_became_peered":"2024-04-14T15:09:43.023220+0000","last_unstale":"2024-04-14T15:18:28.036249+0000","last_undegraded":"2024-04-14T15:18:28.036249+0000","last_fullsized":"2024-04-14T15:18:28.036249+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:01.643745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:01.643745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":752,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.204583+0000","last_change":"2024-04-14T15:16:50.661634+0000","last_active":"2024-04-14T15:18:28.204583+0000","last_peered":"2024-04-14T15:18:28.204583+0000","last_clean":"2024-04-14T15:18:28.204583+0000","last_became_active":"2024-04-14T15:09:47.044665+0000","last_became_peered":"2024-04-14T15:09:47.044665+0000","last_unstale":"2024-04-14T15:18:28.204583+0000","last_undegraded":"2024-04-14T15:18:28.204583+0000","last_fullsized":"2024-04-14T15:18:28.204583+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:50.661520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:50.661520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":431,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.032624+0000","last_change":"2024-04-14T15:16:05.885527+0000","last_active":"2024-04-14T15:18:28.032624+0000","last_peered":"2024-04-14T15:18:28.032624+0000","last_clean":"2024-04-14T15:18:28.032624+0000","last_became_active":"2024-04-14T15:15:21.883940+0000","last_became_peered":"2024-04-14T15:15:21.883940+0000","last_unstale":"2024-04-14T15:18:28.032624+0000","last_undegraded":"2024-04-14T15:18:28.032624+0000","last_fullsized":"2024-04-14T15:18:28.032624+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:05.885438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:05.885438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":664,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.027230+0000","last_change":"2024-04-14T15:18:25.997409+0000","last_active":"2024-04-14T15:18:28.027230+0000","last_peered":"2024-04-14T15:18:28.027230+0000","last_clean":"2024-04-14T15:18:28.027230+0000","last_became_active":"2024-04-14T15:12:56.335230+0000","last_became_peered":"2024-04-14T15:12:56.335230+0000","last_unstale":"2024-04-14T15:18:28.027230+0000","last_undegraded":"2024-04-14T15:18:28.027230+0000","last_fullsized":"2024-04-14T15:18:28.027230+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:25.997333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:18:25.997333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:41.530252+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"72'10","reported_seq":774,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.024189+0000","last_change":"2024-04-14T15:16:49.779653+0000","last_active":"2024-04-14T15:18:28.024189+0000","last_peered":"2024-04-14T15:18:28.024189+0000","last_clean":"2024-04-14T15:18:28.024189+0000","last_became_active":"2024-04-14T15:09:45.181456+0000","last_became_peered":"2024-04-14T15:09:45.181456+0000","last_unstale":"2024-04-14T15:18:28.024189+0000","last_undegraded":"2024-04-14T15:18:28.024189+0000","last_fullsized":"2024-04-14T15:18:28.024189+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-04-14T15:16:49.779557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:16:49.779557+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":319,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.020434+0000","last_change":"2024-04-14T15:16:42.205142+0000","last_active":"2024-04-14T15:18:28.020434+0000","last_peered":"2024-04-14T15:18:28.020434+0000","last_clean":"2024-04-14T15:18:28.020434+0000","last_became_active":"2024-04-14T15:15:44.878336+0000","last_became_peered":"2024-04-14T15:15:44.878336+0000","last_unstale":"2024-04-14T15:18:28.020434+0000","last_undegraded":"2024-04-14T15:18:28.020434+0000","last_fullsized":"2024-04-14T15:18:28.020434+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:42.205063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:42.205063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"72'8","reported_seq":887,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.793035+0000","last_change":"2024-04-14T15:18:28.793061+0000","last_active":"2024-04-14T15:18:28.793035+0000","last_peered":"2024-04-14T15:18:28.793035+0000","last_clean":"2024-04-14T15:18:28.793035+0000","last_became_active":"2024-04-14T15:09:47.048216+0000","last_became_peered":"2024-04-14T15:09:47.048216+0000","last_unstale":"2024-04-14T15:18:28.793035+0000","last_undegraded":"2024-04-14T15:18:28.793035+0000","last_fullsized":"2024-04-14T15:18:28.793035+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:18:28.792975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:18:28.792975+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:50.662051+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"72'1","reported_seq":757,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.205456+0000","last_change":"2024-04-14T15:17:50.529174+0000","last_active":"2024-04-14T15:18:28.205456+0000","last_peered":"2024-04-14T15:18:28.205456+0000","last_clean":"2024-04-14T15:18:28.205456+0000","last_became_active":"2024-04-14T15:09:49.062051+0000","last_became_peered":"2024-04-14T15:09:49.062051+0000","last_unstale":"2024-04-14T15:18:28.205456+0000","last_undegraded":"2024-04-14T15:18:28.205456+0000","last_fullsized":"2024-04-14T15:18:28.205456+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-04-14T15:17:50.529089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:17:50.529089+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:53.773686+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":659,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.020651+0000","last_change":"2024-04-14T15:16:09.471961+0000","last_active":"2024-04-14T15:18:28.020651+0000","last_peered":"2024-04-14T15:18:28.020651+0000","last_clean":"2024-04-14T15:18:28.020651+0000","last_became_active":"2024-04-14T15:12:56.543793+0000","last_became_peered":"2024-04-14T15:12:56.543793+0000","last_unstale":"2024-04-14T15:18:28.020651+0000","last_undegraded":"2024-04-14T15:18:28.020651+0000","last_fullsized":"2024-04-14T15:18:28.020651+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:09.471852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:16:09.471852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"72'11","reported_seq":795,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.036461+0000","last_change":"2024-04-14T15:16:06.886451+0000","last_active":"2024-04-14T15:18:28.036461+0000","last_peered":"2024-04-14T15:18:28.036461+0000","last_clean":"2024-04-14T15:18:28.036461+0000","last_became_active":"2024-04-14T15:10:07.764183+0000","last_became_peered":"2024-04-14T15:10:07.764183+0000","last_unstale":"2024-04-14T15:18:28.036461+0000","last_undegraded":"2024-04-14T15:18:28.036461+0000","last_fullsized":"2024-04-14T15:18:28.036461+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-04-14T15:16:06.886325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:16:06.886325+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"65'2","reported_seq":450,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.032282+0000","last_change":"2024-04-14T15:16:56.943047+0000","last_active":"2024-04-14T15:18:28.032282+0000","last_peered":"2024-04-14T15:18:28.032282+0000","last_clean":"2024-04-14T15:18:28.032282+0000","last_became_active":"2024-04-14T15:15:21.885343+0000","last_became_peered":"2024-04-14T15:15:21.885343+0000","last_unstale":"2024-04-14T15:18:28.032282+0000","last_undegraded":"2024-04-14T15:18:28.032282+0000","last_fullsized":"2024-04-14T15:18:28.032282+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-04-14T15:16:56.942840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:56.942840+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":755,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.024279+0000","last_change":"2024-04-14T15:16:52.747151+0000","last_active":"2024-04-14T15:18:28.024279+0000","last_peered":"2024-04-14T15:18:28.024279+0000","last_clean":"2024-04-14T15:18:28.024279+0000","last_became_active":"2024-04-14T15:09:47.045730+0000","last_became_peered":"2024-04-14T15:09:47.045730+0000","last_unstale":"2024-04-14T15:18:28.024279+0000","last_undegraded":"2024-04-14T15:18:28.024279+0000","last_fullsized":"2024-04-14T15:18:28.024279+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:52.747070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:52.747070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":750,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.024260+0000","last_change":"2024-04-14T15:16:57.235339+0000","last_active":"2024-04-14T15:18:28.024260+0000","last_peered":"2024-04-14T15:18:28.024260+0000","last_clean":"2024-04-14T15:18:28.024260+0000","last_became_active":"2024-04-14T15:09:49.057545+0000","last_became_peered":"2024-04-14T15:09:49.057545+0000","last_unstale":"2024-04-14T15:18:28.024260+0000","last_undegraded":"2024-04-14T15:18:28.024260+0000","last_fullsized":"2024-04-14T15:18:28.024260+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:57.235261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:21.247695+0000","last_clean_scrub_stamp":"2024-04-14T15:16:57.235261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":319,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.020697+0000","last_change":"2024-04-14T15:17:44.364114+0000","last_active":"2024-04-14T15:18:28.020697+0000","last_peered":"2024-04-14T15:18:28.020697+0000","last_clean":"2024-04-14T15:18:28.020697+0000","last_became_active":"2024-04-14T15:15:44.882604+0000","last_became_peered":"2024-04-14T15:15:44.882604+0000","last_unstale":"2024-04-14T15:18:28.020697+0000","last_undegraded":"2024-04-14T15:18:28.020697+0000","last_fullsized":"2024-04-14T15:18:28.020697+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:44.364013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:17:44.364013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:01.492147+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"72'15","reported_seq":780,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.028162+0000","last_change":"2024-04-14T15:15:44.609525+0000","last_active":"2024-04-14T15:18:28.028162+0000","last_peered":"2024-04-14T15:18:28.028162+0000","last_clean":"2024-04-14T15:18:28.028162+0000","last_became_active":"2024-04-14T15:09:45.032887+0000","last_became_peered":"2024-04-14T15:09:45.032887+0000","last_unstale":"2024-04-14T15:18:28.028162+0000","last_undegraded":"2024-04-14T15:18:28.028162+0000","last_fullsized":"2024-04-14T15:18:28.028162+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-04-14T15:15:44.609332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:15:44.609332+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":772,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.034897+0000","last_change":"2024-04-14T15:18:27.799078+0000","last_active":"2024-04-14T15:18:28.034897+0000","last_peered":"2024-04-14T15:18:28.034897+0000","last_clean":"2024-04-14T15:18:28.034897+0000","last_became_active":"2024-04-14T15:09:43.021494+0000","last_became_peered":"2024-04-14T15:09:43.021494+0000","last_unstale":"2024-04-14T15:18:28.034897+0000","last_undegraded":"2024-04-14T15:18:28.034897+0000","last_fullsized":"2024-04-14T15:18:28.034897+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:27.798964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:18:27.798964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:33.052017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":762,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.024522+0000","last_change":"2024-04-14T15:18:22.953738+0000","last_active":"2024-04-14T15:18:28.024522+0000","last_peered":"2024-04-14T15:18:28.024522+0000","last_clean":"2024-04-14T15:18:28.024522+0000","last_became_active":"2024-04-14T15:09:47.042733+0000","last_became_peered":"2024-04-14T15:09:47.042733+0000","last_unstale":"2024-04-14T15:18:28.024522+0000","last_undegraded":"2024-04-14T15:18:28.024522+0000","last_fullsized":"2024-04-14T15:18:28.024522+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:22.953661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:18:22.953661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:49.908442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":757,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.021632+0000","last_change":"2024-04-14T15:17:54.453289+0000","last_active":"2024-04-14T15:18:28.021632+0000","last_peered":"2024-04-14T15:18:28.021632+0000","last_clean":"2024-04-14T15:18:28.021632+0000","last_became_active":"2024-04-14T15:09:49.127979+0000","last_became_peered":"2024-04-14T15:09:49.127979+0000","last_unstale":"2024-04-14T15:18:28.021632+0000","last_undegraded":"2024-04-14T15:18:28.021632+0000","last_fullsized":"2024-04-14T15:18:28.021632+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:54.453174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:54.453174+0000","last_clean_scrub_stamp":"2024-04-14T15:17:54.453174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:08.115614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":665,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.204987+0000","last_change":"2024-04-14T15:18:26.532035+0000","last_active":"2024-04-14T15:18:28.204987+0000","last_peered":"2024-04-14T15:18:28.204987+0000","last_clean":"2024-04-14T15:18:28.204987+0000","last_became_active":"2024-04-14T15:12:56.392352+0000","last_became_peered":"2024-04-14T15:12:56.392352+0000","last_unstale":"2024-04-14T15:18:28.204987+0000","last_undegraded":"2024-04-14T15:18:28.204987+0000","last_fullsized":"2024-04-14T15:18:28.204987+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:26.531955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:18:26.531955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:43.681656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"72'12","reported_seq":462,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.033137+0000","last_change":"2024-04-14T15:16:11.903062+0000","last_active":"2024-04-14T15:18:28.033137+0000","last_peered":"2024-04-14T15:18:28.033137+0000","last_clean":"2024-04-14T15:18:28.033137+0000","last_became_active":"2024-04-14T15:15:21.877838+0000","last_became_peered":"2024-04-14T15:15:21.877838+0000","last_unstale":"2024-04-14T15:18:28.033137+0000","last_undegraded":"2024-04-14T15:18:28.033137+0000","last_fullsized":"2024-04-14T15:18:28.033137+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'12","last_scrub_stamp":"2024-04-14T15:16:11.902934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:16:11.902934+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":443,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.032539+0000","last_change":"2024-04-14T15:16:20.969027+0000","last_active":"2024-04-14T15:18:28.032539+0000","last_peered":"2024-04-14T15:18:28.032539+0000","last_clean":"2024-04-14T15:18:28.032539+0000","last_became_active":"2024-04-14T15:15:21.883544+0000","last_became_peered":"2024-04-14T15:15:21.883544+0000","last_unstale":"2024-04-14T15:18:28.032539+0000","last_undegraded":"2024-04-14T15:18:28.032539+0000","last_fullsized":"2024-04-14T15:18:28.032539+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:20.968897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:20.968897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":760,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.021257+0000","last_change":"2024-04-14T15:17:01.098681+0000","last_active":"2024-04-14T15:18:28.021257+0000","last_peered":"2024-04-14T15:18:28.021257+0000","last_clean":"2024-04-14T15:18:28.021257+0000","last_became_active":"2024-04-14T15:09:47.048045+0000","last_became_peered":"2024-04-14T15:09:47.048045+0000","last_unstale":"2024-04-14T15:18:28.021257+0000","last_undegraded":"2024-04-14T15:18:28.021257+0000","last_fullsized":"2024-04-14T15:18:28.021257+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:01.098599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:17:01.098599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":432,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.033143+0000","last_change":"2024-04-14T15:16:07.879394+0000","last_active":"2024-04-14T15:18:28.033143+0000","last_peered":"2024-04-14T15:18:28.033143+0000","last_clean":"2024-04-14T15:18:28.033143+0000","last_became_active":"2024-04-14T15:15:21.889691+0000","last_became_peered":"2024-04-14T15:15:21.889691+0000","last_unstale":"2024-04-14T15:18:28.033143+0000","last_undegraded":"2024-04-14T15:18:28.033143+0000","last_fullsized":"2024-04-14T15:18:28.033143+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:07.879280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:07.879280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":664,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.028032+0000","last_change":"2024-04-14T15:17:37.766225+0000","last_active":"2024-04-14T15:18:28.028032+0000","last_peered":"2024-04-14T15:18:28.028032+0000","last_clean":"2024-04-14T15:18:28.028032+0000","last_became_active":"2024-04-14T15:12:56.375278+0000","last_became_peered":"2024-04-14T15:12:56.375278+0000","last_unstale":"2024-04-14T15:18:28.028032+0000","last_undegraded":"2024-04-14T15:18:28.028032+0000","last_fullsized":"2024-04-14T15:18:28.028032+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:37.766116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:17:37.766116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:42.299956+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"276'13","reported_seq":784,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.020706+0000","last_change":"2024-04-14T15:17:52.445394+0000","last_active":"2024-04-14T15:18:28.020706+0000","last_peered":"2024-04-14T15:18:28.020706+0000","last_clean":"2024-04-14T15:18:28.020706+0000","last_became_active":"2024-04-14T15:09:45.030093+0000","last_became_peered":"2024-04-14T15:09:45.030093+0000","last_unstale":"2024-04-14T15:18:28.020706+0000","last_undegraded":"2024-04-14T15:18:28.020706+0000","last_fullsized":"2024-04-14T15:18:28.020706+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'13","last_scrub_stamp":"2024-04-14T15:17:52.445294+0000","last_deep_scrub":"276'13","last_deep_scrub_stamp":"2024-04-14T15:15:43.969775+0000","last_clean_scrub_stamp":"2024-04-14T15:17:52.445294+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:59.668815+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":466,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.204991+0000","last_change":"2024-04-14T15:17:02.800274+0000","last_active":"2024-04-14T15:18:28.204991+0000","last_peered":"2024-04-14T15:18:28.204991+0000","last_clean":"2024-04-14T15:18:28.204991+0000","last_became_active":"2024-04-14T15:14:30.857727+0000","last_became_peered":"2024-04-14T15:14:30.857727+0000","last_unstale":"2024-04-14T15:18:28.204991+0000","last_undegraded":"2024-04-14T15:18:28.204991+0000","last_fullsized":"2024-04-14T15:18:28.204991+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:02.800166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:02.800166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":321,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.022975+0000","last_change":"2024-04-14T15:16:59.695423+0000","last_active":"2024-04-14T15:18:28.022975+0000","last_peered":"2024-04-14T15:18:28.022975+0000","last_clean":"2024-04-14T15:18:28.022975+0000","last_became_active":"2024-04-14T15:15:44.883121+0000","last_became_peered":"2024-04-14T15:15:44.883121+0000","last_unstale":"2024-04-14T15:18:28.022975+0000","last_undegraded":"2024-04-14T15:18:28.022975+0000","last_fullsized":"2024-04-14T15:18:28.022975+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:59.695337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:59.695337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":758,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.020906+0000","last_change":"2024-04-14T15:17:33.952503+0000","last_active":"2024-04-14T15:18:28.020906+0000","last_peered":"2024-04-14T15:18:28.020906+0000","last_clean":"2024-04-14T15:18:28.020906+0000","last_became_active":"2024-04-14T15:09:49.061756+0000","last_became_peered":"2024-04-14T15:09:49.061756+0000","last_unstale":"2024-04-14T15:18:28.020906+0000","last_undegraded":"2024-04-14T15:18:28.020906+0000","last_fullsized":"2024-04-14T15:18:28.020906+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:33.952312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:17:33.952312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:36.590330+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":666,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.035078+0000","last_change":"2024-04-14T15:16:57.667484+0000","last_active":"2024-04-14T15:18:28.035078+0000","last_peered":"2024-04-14T15:18:28.035078+0000","last_clean":"2024-04-14T15:18:28.035078+0000","last_became_active":"2024-04-14T15:12:56.232831+0000","last_became_peered":"2024-04-14T15:12:56.232831+0000","last_unstale":"2024-04-14T15:18:28.035078+0000","last_undegraded":"2024-04-14T15:18:28.035078+0000","last_fullsized":"2024-04-14T15:18:28.035078+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:57.667382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:16:57.667382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1b","version":"87'7","reported_seq":768,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.021022+0000","last_change":"2024-04-14T15:15:44.927867+0000","last_active":"2024-04-14T15:18:28.021022+0000","last_peered":"2024-04-14T15:18:28.021022+0000","last_clean":"2024-04-14T15:18:28.021022+0000","last_became_active":"2024-04-14T15:09:45.028667+0000","last_became_peered":"2024-04-14T15:09:45.028667+0000","last_unstale":"2024-04-14T15:18:28.021022+0000","last_undegraded":"2024-04-14T15:18:28.021022+0000","last_fullsized":"2024-04-14T15:18:28.021022+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'7","last_scrub_stamp":"2024-04-14T15:15:44.927776+0000","last_deep_scrub":"87'7","last_deep_scrub_stamp":"2024-04-14T15:11:10.510785+0000","last_clean_scrub_stamp":"2024-04-14T15:15:44.927776+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":236,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.027606+0000","last_change":"2024-04-14T15:16:46.686879+0000","last_active":"2024-04-14T15:18:28.027606+0000","last_peered":"2024-04-14T15:18:28.027606+0000","last_clean":"2024-04-14T15:18:28.027606+0000","last_became_active":"2024-04-14T15:16:24.910246+0000","last_became_peered":"2024-04-14T15:16:24.910246+0000","last_unstale":"2024-04-14T15:18:28.027606+0000","last_undegraded":"2024-04-14T15:18:28.027606+0000","last_fullsized":"2024-04-14T15:18:28.027606+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:46.686789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:08.254601+0000","last_clean_scrub_stamp":"2024-04-14T15:16:46.686789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":752,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.205141+0000","last_change":"2024-04-14T15:16:17.804733+0000","last_active":"2024-04-14T15:18:28.205141+0000","last_peered":"2024-04-14T15:18:28.205141+0000","last_clean":"2024-04-14T15:18:28.205141+0000","last_became_active":"2024-04-14T15:09:47.046517+0000","last_became_peered":"2024-04-14T15:09:47.046517+0000","last_unstale":"2024-04-14T15:18:28.205141+0000","last_undegraded":"2024-04-14T15:18:28.205141+0000","last_fullsized":"2024-04-14T15:18:28.205141+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:17.804649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:17.804649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":322,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.024049+0000","last_change":"2024-04-14T15:18:21.023508+0000","last_active":"2024-04-14T15:18:28.024049+0000","last_peered":"2024-04-14T15:18:28.024049+0000","last_clean":"2024-04-14T15:18:28.024049+0000","last_became_active":"2024-04-14T15:15:44.887227+0000","last_became_peered":"2024-04-14T15:15:44.887227+0000","last_unstale":"2024-04-14T15:18:28.024049+0000","last_undegraded":"2024-04-14T15:18:28.024049+0000","last_fullsized":"2024-04-14T15:18:28.024049+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:21.023426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:56.268086+0000","last_clean_scrub_stamp":"2024-04-14T15:18:21.023426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:27.500436+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":657,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.205976+0000","last_change":"2024-04-14T15:15:37.629813+0000","last_active":"2024-04-14T15:18:28.205976+0000","last_peered":"2024-04-14T15:18:28.205976+0000","last_clean":"2024-04-14T15:18:28.205976+0000","last_became_active":"2024-04-14T15:12:56.189117+0000","last_became_peered":"2024-04-14T15:12:56.189117+0000","last_unstale":"2024-04-14T15:18:28.205976+0000","last_undegraded":"2024-04-14T15:18:28.205976+0000","last_fullsized":"2024-04-14T15:18:28.205976+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:37.629718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:37.629718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.18","version":"72'9","reported_seq":772,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.020561+0000","last_change":"2024-04-14T15:16:09.965230+0000","last_active":"2024-04-14T15:18:28.020561+0000","last_peered":"2024-04-14T15:18:28.020561+0000","last_clean":"2024-04-14T15:18:28.020561+0000","last_became_active":"2024-04-14T15:09:45.032079+0000","last_became_peered":"2024-04-14T15:09:45.032079+0000","last_unstale":"2024-04-14T15:18:28.020561+0000","last_undegraded":"2024-04-14T15:18:28.020561+0000","last_fullsized":"2024-04-14T15:18:28.020561+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-04-14T15:16:09.965113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:16:09.965113+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":444,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.033208+0000","last_change":"2024-04-14T15:17:00.948362+0000","last_active":"2024-04-14T15:18:28.033208+0000","last_peered":"2024-04-14T15:18:28.033208+0000","last_clean":"2024-04-14T15:18:28.033208+0000","last_became_active":"2024-04-14T15:15:21.891926+0000","last_became_peered":"2024-04-14T15:15:21.891926+0000","last_unstale":"2024-04-14T15:18:28.033208+0000","last_undegraded":"2024-04-14T15:18:28.033208+0000","last_fullsized":"2024-04-14T15:18:28.033208+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:00.948283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:00.948283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":754,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.023947+0000","last_change":"2024-04-14T15:16:45.114648+0000","last_active":"2024-04-14T15:18:28.023947+0000","last_peered":"2024-04-14T15:18:28.023947+0000","last_clean":"2024-04-14T15:18:28.023947+0000","last_became_active":"2024-04-14T15:09:47.041486+0000","last_became_peered":"2024-04-14T15:09:47.041486+0000","last_unstale":"2024-04-14T15:18:28.023947+0000","last_undegraded":"2024-04-14T15:18:28.023947+0000","last_fullsized":"2024-04-14T15:18:28.023947+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:45.114534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:45.114534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":748,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.020668+0000","last_change":"2024-04-14T15:16:55.184211+0000","last_active":"2024-04-14T15:18:28.020668+0000","last_peered":"2024-04-14T15:18:28.020668+0000","last_clean":"2024-04-14T15:18:28.020668+0000","last_became_active":"2024-04-14T15:09:49.058904+0000","last_became_peered":"2024-04-14T15:09:49.058904+0000","last_unstale":"2024-04-14T15:18:28.020668+0000","last_undegraded":"2024-04-14T15:18:28.020668+0000","last_fullsized":"2024-04-14T15:18:28.020668+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:55.184094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:55.184094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":319,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.205843+0000","last_change":"2024-04-14T15:17:52.593706+0000","last_active":"2024-04-14T15:18:28.205843+0000","last_peered":"2024-04-14T15:18:28.205843+0000","last_clean":"2024-04-14T15:18:28.205843+0000","last_became_active":"2024-04-14T15:15:44.889605+0000","last_became_peered":"2024-04-14T15:15:44.889605+0000","last_unstale":"2024-04-14T15:18:28.205843+0000","last_undegraded":"2024-04-14T15:18:28.205843+0000","last_fullsized":"2024-04-14T15:18:28.205843+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:52.593618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:17:52.593618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:09.797845+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"306'22","reported_seq":471,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.032683+0000","last_change":"2024-04-14T15:16:13.898263+0000","last_active":"2024-04-14T15:18:28.032683+0000","last_peered":"2024-04-14T15:18:28.032683+0000","last_clean":"2024-04-14T15:18:28.032683+0000","last_became_active":"2024-04-14T15:15:21.884310+0000","last_became_peered":"2024-04-14T15:15:21.884310+0000","last_unstale":"2024-04-14T15:18:28.032683+0000","last_undegraded":"2024-04-14T15:18:28.032683+0000","last_fullsized":"2024-04-14T15:18:28.032683+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'22","last_scrub_stamp":"2024-04-14T15:16:13.898106+0000","last_deep_scrub":"306'22","last_deep_scrub_stamp":"2024-04-14T15:16:13.898106+0000","last_clean_scrub_stamp":"2024-04-14T15:16:13.898106+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":763,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.024628+0000","last_change":"2024-04-14T15:16:14.427502+0000","last_active":"2024-04-14T15:18:28.024628+0000","last_peered":"2024-04-14T15:18:28.024628+0000","last_clean":"2024-04-14T15:18:28.024628+0000","last_became_active":"2024-04-14T15:09:43.017351+0000","last_became_peered":"2024-04-14T15:09:43.017351+0000","last_unstale":"2024-04-14T15:18:28.024628+0000","last_undegraded":"2024-04-14T15:18:28.024628+0000","last_fullsized":"2024-04-14T15:18:28.024628+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:14.427425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:14.427425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":752,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.020657+0000","last_change":"2024-04-14T15:16:19.041461+0000","last_active":"2024-04-14T15:18:28.020657+0000","last_peered":"2024-04-14T15:18:28.020657+0000","last_clean":"2024-04-14T15:18:28.020657+0000","last_became_active":"2024-04-14T15:09:47.042538+0000","last_became_peered":"2024-04-14T15:09:47.042538+0000","last_unstale":"2024-04-14T15:18:28.020657+0000","last_undegraded":"2024-04-14T15:18:28.020657+0000","last_fullsized":"2024-04-14T15:18:28.020657+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:19.041346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:08.512130+0000","last_clean_scrub_stamp":"2024-04-14T15:16:19.041346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":754,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-04-14T15:18:28.032349+0000","last_change":"2024-04-14T15:17:56.237366+0000","last_active":"2024-04-14T15:18:28.032349+0000","last_peered":"2024-04-14T15:18:28.032349+0000","last_clean":"2024-04-14T15:18:28.032349+0000","last_became_active":"2024-04-14T15:09:49.129636+0000","last_became_peered":"2024-04-14T15:09:49.129636+0000","last_unstale":"2024-04-14T15:18:28.032349+0000","last_undegraded":"2024-04-14T15:18:28.032349+0000","last_fullsized":"2024-04-14T15:18:28.032349+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:56.237273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:17:56.237273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:03.893408+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":297,"num_pg":32,"stat_sum":{"num_bytes":12,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":13,"num_read_kb":10,"num_write":12,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":96,"acting":96,"num_store_stats":8},{"poolid":110,"num_pg":32,"stat_sum":{"num_bytes":8195788,"num_objects":160,"num_object_clones":3,"num_object_copies":480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":160,"num_whiteouts":0,"num_read":108,"num_read_kb":9,"num_write":599,"num_write_kb":12688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":8,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":26492928,"data_stored":24587373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1883,"ondisk_log_size":1883,"up":96,"acting":96,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1430,"ondisk_log_size":1430,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":519,"num_read_kb":514,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":695,"num_read_kb":452,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":481,"ondisk_log_size":481,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71,"ondisk_log_size":71,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103279,"num_pgs":99,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":73092,"kb_used_data":15536,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93672060,"statfs":{"total":95995035648,"available":95920189440,"internally_reserved":0,"allocated":15908864,"data_stored":12653112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":58915274},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397618,"num_pgs":84,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":70348,"kb_used_data":7224,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93674804,"statfs":{"total":95995035648,"available":95922999296,"internally_reserved":0,"allocated":7397376,"data_stored":4131435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574988,"num_pgs":92,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":87020,"kb_used_data":11416,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93658132,"statfs":{"total":95995035648,"available":95905927168,"internally_reserved":0,"allocated":11689984,"data_stored":8326026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738513,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":82636,"kb_used_data":11180,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93662516,"statfs":{"total":95995035648,"available":95910416384,"internally_reserved":0,"allocated":11448320,"data_stored":8131869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411464,"num_pgs":87,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":82912,"kb_used_data":11468,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93662240,"statfs":{"total":95995035648,"available":95910133760,"internally_reserved":0,"allocated":11743232,"data_stored":8393572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182530,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":57020,"kb_used_data":7784,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93688132,"statfs":{"total":95995035648,"available":95936647168,"internally_reserved":0,"allocated":7970816,"data_stored":4722127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986300,"num_pgs":87,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":70320,"kb_used_data":7196,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93674832,"statfs":{"total":95995035648,"available":95923027968,"internally_reserved":0,"allocated":7368704,"data_stored":4130955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659257,"num_pgs":88,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":75008,"kb_used_data":11872,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93670144,"statfs":{"total":95995035648,"available":95918227456,"internally_reserved":0,"allocated":12156928,"data_stored":8915112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4272128,"data_stored":4000608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4493312,"data_stored":4195009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4325376,"data_stored":4000764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":172032,"data_stored":406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":184320,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4403200,"data_stored":4194789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":225280,"data_stored":524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8417280,"data_stored":8194829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":297,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":297,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":297,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":2,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":297,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":297,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":12,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":297,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":2,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":297,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":297,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-14T15:18:33.101 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-04-14T15:18:33.101 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd pool set .mgr pg_num 21 2024-04-14T15:18:33.101 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:18:33.127 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:33.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:33.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:18:33.188 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:18:33.203 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:18:33.229 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:33.252 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:33.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:33.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:33.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:33 smithi090 ceph-mon[41670]: 4.1f scrub starts 2024-04-14T15:18:33.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:33 smithi090 ceph-mon[41670]: 4.1f scrub ok 2024-04-14T15:18:33.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:33 smithi090 ceph-mon[41670]: 5.b scrub starts 2024-04-14T15:18:33.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:33 smithi090 ceph-mon[41670]: 5.b scrub ok 2024-04-14T15:18:33.306 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:33 smithi090 ceph-mon[30399]: 4.1f scrub starts 2024-04-14T15:18:33.306 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:33 smithi090 ceph-mon[30399]: 4.1f scrub ok 2024-04-14T15:18:33.306 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:33 smithi090 ceph-mon[30399]: 5.b scrub starts 2024-04-14T15:18:33.306 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:33 smithi090 ceph-mon[30399]: 5.b scrub ok 2024-04-14T15:18:33.325 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:33 smithi092 ceph-mon[33966]: 4.1f scrub starts 2024-04-14T15:18:33.325 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:33 smithi092 ceph-mon[33966]: 4.1f scrub ok 2024-04-14T15:18:33.325 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:33 smithi092 ceph-mon[33966]: 5.b scrub starts 2024-04-14T15:18:33.325 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:33 smithi092 ceph-mon[33966]: 5.b scrub ok 2024-04-14T15:18:33.354 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:18:33.377 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:33.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:33.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:18:33.479 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:18:33.502 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:33.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:33.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:18:33.604 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:18:33.628 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:33.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:33.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:18:33.730 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:18:33.753 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:33.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:33.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:18:33.855 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:18:33.860 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:18:33.880 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:33.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:33.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:18:33.982 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:18:34.005 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:34.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:34.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:18:34.107 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:18:34.131 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:34.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:34.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:18:34.233 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:18:34.257 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:34.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:34.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:18:34.260 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd set noscrub 2024-04-14T15:18:34.285 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:18:33 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:18:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:18:34.285 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:34 smithi090 ceph-mon[41670]: 11.a scrub starts 2024-04-14T15:18:34.285 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:34 smithi090 ceph-mon[41670]: 11.a scrub ok 2024-04-14T15:18:34.285 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:34 smithi090 ceph-mon[41670]: 4.1c scrub starts 2024-04-14T15:18:34.285 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:34 smithi090 ceph-mon[41670]: 4.1c scrub ok 2024-04-14T15:18:34.285 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:34 smithi090 ceph-mon[41670]: 1.1 scrub starts 2024-04-14T15:18:34.286 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:34 smithi090 ceph-mon[41670]: 1.1 scrub ok 2024-04-14T15:18:34.286 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:34 smithi090 ceph-mon[41670]: pgmap v597: 238 pgs: 238 active+clean; 8.4 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-14T15:18:34.286 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:34 smithi090 ceph-mon[30399]: 11.a scrub starts 2024-04-14T15:18:34.286 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:34 smithi090 ceph-mon[30399]: 11.a scrub ok 2024-04-14T15:18:34.286 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:34 smithi090 ceph-mon[30399]: 4.1c scrub starts 2024-04-14T15:18:34.286 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:34 smithi090 ceph-mon[30399]: 4.1c scrub ok 2024-04-14T15:18:34.286 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:34 smithi090 ceph-mon[30399]: 1.1 scrub starts 2024-04-14T15:18:34.286 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:34 smithi090 ceph-mon[30399]: 1.1 scrub ok 2024-04-14T15:18:34.286 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:34 smithi090 ceph-mon[30399]: pgmap v597: 238 pgs: 238 active+clean; 8.4 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-14T15:18:34.359 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:18:34.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:34 smithi092 ceph-mon[33966]: 11.a scrub starts 2024-04-14T15:18:34.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:34 smithi092 ceph-mon[33966]: 11.a scrub ok 2024-04-14T15:18:34.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:34 smithi092 ceph-mon[33966]: 4.1c scrub starts 2024-04-14T15:18:34.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:34 smithi092 ceph-mon[33966]: 4.1c scrub ok 2024-04-14T15:18:34.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:34 smithi092 ceph-mon[33966]: 1.1 scrub starts 2024-04-14T15:18:34.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:34 smithi092 ceph-mon[33966]: 1.1 scrub ok 2024-04-14T15:18:34.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:34 smithi092 ceph-mon[33966]: pgmap v597: 238 pgs: 238 active+clean; 8.4 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-14T15:18:34.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:34.385 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:34.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:18:34.486 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:18:34.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:34.511 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:34.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:18:34.612 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:18:34.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:34.637 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:34.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:18:34.738 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:18:34.761 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:34.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:34.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:18:34.863 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:34.888 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:34.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:34.889 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:34.898 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-04-14T15:18:34.898 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:18:34.990 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:35.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:35.015 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:35.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:35.116 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:18:35.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:35.140 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:35.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:18:35.241 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:18:35.264 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:35.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:35.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:18:35.366 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:18:35.372 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:18:35 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:18:35.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:35 smithi092 ceph-mon[33966]: 5.13 deep-scrub starts 2024-04-14T15:18:35.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:35 smithi092 ceph-mon[33966]: 5.13 deep-scrub ok 2024-04-14T15:18:35.386 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:35 smithi090 ceph-mon[41670]: 5.13 deep-scrub starts 2024-04-14T15:18:35.387 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:35 smithi090 ceph-mon[41670]: 5.13 deep-scrub ok 2024-04-14T15:18:35.387 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:35 smithi090 ceph-mon[30399]: 5.13 deep-scrub starts 2024-04-14T15:18:35.387 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:35 smithi090 ceph-mon[30399]: 5.13 deep-scrub ok 2024-04-14T15:18:35.391 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:35.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:35.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:18:35.493 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:18:35.509 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-14T15:18:35.517 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:35.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:35.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:18:35.619 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:18:35.642 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:35.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:35.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:18:35.744 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:18:35.748 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:18:35.767 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:35.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:35.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:18:35.869 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:18:35.893 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:35.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:35.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:18:35.995 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:18:36.019 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:36.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:36.020 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:18:36.049 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:36 smithi090 ceph-mon[41670]: 110.8 scrub starts 2024-04-14T15:18:36.049 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:36 smithi090 ceph-mon[41670]: 110.8 scrub ok 2024-04-14T15:18:36.049 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:36 smithi090 ceph-mon[41670]: 11.17 deep-scrub starts 2024-04-14T15:18:36.049 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:36 smithi090 ceph-mon[41670]: 11.17 deep-scrub ok 2024-04-14T15:18:36.049 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:36 smithi090 ceph-mon[41670]: 11.c deep-scrub starts 2024-04-14T15:18:36.049 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:36 smithi090 ceph-mon[41670]: 11.c deep-scrub ok 2024-04-14T15:18:36.049 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:36 smithi090 ceph-mon[41670]: pgmap v598: 238 pgs: 238 active+clean; 8.4 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-14T15:18:36.049 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:36 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:18:36.049 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:36 smithi090 ceph-mon[41670]: 4.1b scrub starts 2024-04-14T15:18:36.050 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:36 smithi090 ceph-mon[41670]: 4.1b scrub ok 2024-04-14T15:18:36.050 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:36 smithi090 ceph-mon[41670]: osdmap e429: 8 total, 8 up, 8 in 2024-04-14T15:18:36.050 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:36 smithi090 ceph-mon[30399]: 110.8 scrub starts 2024-04-14T15:18:36.050 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:36 smithi090 ceph-mon[30399]: 110.8 scrub ok 2024-04-14T15:18:36.050 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:36 smithi090 ceph-mon[30399]: 11.17 deep-scrub starts 2024-04-14T15:18:36.050 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:36 smithi090 ceph-mon[30399]: 11.17 deep-scrub ok 2024-04-14T15:18:36.050 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:36 smithi090 ceph-mon[30399]: 11.c deep-scrub starts 2024-04-14T15:18:36.050 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:36 smithi090 ceph-mon[30399]: 11.c deep-scrub ok 2024-04-14T15:18:36.050 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:36 smithi090 ceph-mon[30399]: pgmap v598: 238 pgs: 238 active+clean; 8.4 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-14T15:18:36.051 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:36 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:18:36.051 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:36 smithi090 ceph-mon[30399]: 4.1b scrub starts 2024-04-14T15:18:36.051 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:36 smithi090 ceph-mon[30399]: 4.1b scrub ok 2024-04-14T15:18:36.051 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:36 smithi090 ceph-mon[30399]: osdmap e429: 8 total, 8 up, 8 in 2024-04-14T15:18:36.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.048+0000 7f6dde67c700 1 -- 172.21.15.90:0/1891179042 >> v1:172.21.15.90:6789/0 conn(0x7f6dd810e1e0 legacy=0x7f6dd81106a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:36.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.048+0000 7f6dde67c700 1 -- 172.21.15.90:0/1891179042 shutdown_connections 2024-04-14T15:18:36.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.048+0000 7f6dde67c700 1 -- 172.21.15.90:0/1891179042 >> 172.21.15.90:0/1891179042 conn(0x7f6dd80fd290 msgr2=0x7f6dd80ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:18:36.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.048+0000 7f6dde67c700 1 -- 172.21.15.90:0/1891179042 shutdown_connections 2024-04-14T15:18:36.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.048+0000 7f6dde67c700 1 -- 172.21.15.90:0/1891179042 wait complete. 2024-04-14T15:18:36.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.048+0000 7f6dde67c700 1 librados: starting msgr at 2024-04-14T15:18:36.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.049+0000 7f6dde67c700 1 librados: starting objecter 2024-04-14T15:18:36.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.049+0000 7f6dde67c700 1 Processor -- start 2024-04-14T15:18:36.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.049+0000 7f6dde67c700 1 -- start start 2024-04-14T15:18:36.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.049+0000 7f6dde67c700 1 librados: setting wanted keys 2024-04-14T15:18:36.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.049+0000 7f6dde67c700 1 librados: calling monclient init 2024-04-14T15:18:36.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.049+0000 7f6dde67c700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6dd80ff4e0 con 0x7f6dd810e1e0 2024-04-14T15:18:36.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.049+0000 7f6dde67c700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6dd81ab1f0 con 0x7f6dd810a610 2024-04-14T15:18:36.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.049+0000 7f6dde67c700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6dd81ac3d0 con 0x7f6dd8107760 2024-04-14T15:18:36.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.049+0000 7f6dd7fff700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f6dd8107760 0x7f6dd80fd740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:42792/0 (socket says 172.21.15.90:42792) 2024-04-14T15:18:36.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.049+0000 7f6dd7fff700 1 -- 172.21.15.90:0/101459450 learned_addr learned my addr 172.21.15.90:0/101459450 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:18:36.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.049+0000 7f6dd57fa700 10 client.?.objecter ms_handle_connect 0x7f6dd810e1e0 2024-04-14T15:18:36.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.049+0000 7f6dd57fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:18:36.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.049+0000 7f6dd57fa700 10 client.?.objecter ms_handle_connect 0x7f6dd8107760 2024-04-14T15:18:36.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.049+0000 7f6dd57fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:18:36.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.049+0000 7f6dd57fa700 10 client.?.objecter ms_handle_connect 0x7f6dd810a610 2024-04-14T15:18:36.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.049+0000 7f6dd57fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:18:36.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.049+0000 7f6dd57fa700 1 -- 172.21.15.90:0/101459450 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3452925144 0 0) 0x7f6dd80ff4e0 con 0x7f6dd810e1e0 2024-04-14T15:18:36.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.049+0000 7f6dd57fa700 1 -- 172.21.15.90:0/101459450 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6dc00031f0 con 0x7f6dd810e1e0 2024-04-14T15:18:36.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.049+0000 7f6dd57fa700 1 -- 172.21.15.90:0/101459450 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1106233946 0 0) 0x7f6dd81ac3d0 con 0x7f6dd8107760 2024-04-14T15:18:36.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.049+0000 7f6dd57fa700 1 -- 172.21.15.90:0/101459450 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6dd80ff4e0 con 0x7f6dd8107760 2024-04-14T15:18:36.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.049+0000 7f6dd57fa700 1 -- 172.21.15.90:0/101459450 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2366758754 0 0) 0x7f6dd81ab1f0 con 0x7f6dd810a610 2024-04-14T15:18:36.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.049+0000 7f6dd57fa700 1 -- 172.21.15.90:0/101459450 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6dd81ac3d0 con 0x7f6dd810a610 2024-04-14T15:18:36.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.050+0000 7f6dd57fa700 1 -- 172.21.15.90:0/101459450 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1249305658 0 0) 0x7f6dc00031f0 con 0x7f6dd810e1e0 2024-04-14T15:18:36.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.050+0000 7f6dd57fa700 1 -- 172.21.15.90:0/101459450 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6dd81ab1f0 con 0x7f6dd810e1e0 2024-04-14T15:18:36.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.050+0000 7f6dd57fa700 1 -- 172.21.15.90:0/101459450 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f6dd0001950 con 0x7f6dd810e1e0 2024-04-14T15:18:36.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.050+0000 7f6dd57fa700 1 -- 172.21.15.90:0/101459450 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2099902973 0 0) 0x7f6dd80ff4e0 con 0x7f6dd8107760 2024-04-14T15:18:36.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.050+0000 7f6dd57fa700 1 -- 172.21.15.90:0/101459450 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6dc00031f0 con 0x7f6dd8107760 2024-04-14T15:18:36.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.050+0000 7f6dd57fa700 1 -- 172.21.15.90:0/101459450 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f6dc80018b0 con 0x7f6dd8107760 2024-04-14T15:18:36.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.050+0000 7f6dd57fa700 1 -- 172.21.15.90:0/101459450 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2984093119 0 0) 0x7f6dd81ac3d0 con 0x7f6dd810a610 2024-04-14T15:18:36.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.050+0000 7f6dd57fa700 1 -- 172.21.15.90:0/101459450 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6dd80ff4e0 con 0x7f6dd810a610 2024-04-14T15:18:36.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.050+0000 7f6dd57fa700 1 -- 172.21.15.90:0/101459450 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f6dcc0017f0 con 0x7f6dd810a610 2024-04-14T15:18:36.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.050+0000 7f6dd57fa700 1 -- 172.21.15.90:0/101459450 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2004525660 0 0) 0x7f6dd81ab1f0 con 0x7f6dd810e1e0 2024-04-14T15:18:36.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.050+0000 7f6dd57fa700 1 -- 172.21.15.90:0/101459450 >> v1:172.21.15.90:6790/0 conn(0x7f6dd8107760 legacy=0x7f6dd80fd740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:36.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.050+0000 7f6dd57fa700 1 -- 172.21.15.90:0/101459450 >> v1:172.21.15.92:6789/0 conn(0x7f6dd810a610 legacy=0x7f6dd80fde50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:36.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.050+0000 7f6dd57fa700 1 -- 172.21.15.90:0/101459450 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dd81ad5b0 con 0x7f6dd810e1e0 2024-04-14T15:18:36.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.050+0000 7f6dde67c700 1 -- 172.21.15.90:0/101459450 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6dd81ac600 con 0x7f6dd810e1e0 2024-04-14T15:18:36.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.050+0000 7f6dde67c700 10 client.37305.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:18:36.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.050+0000 7f6dde67c700 1 -- 172.21.15.90:0/101459450 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f6dd81acac0 con 0x7f6dd810e1e0 2024-04-14T15:18:36.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.050+0000 7f6dde67c700 1 librados: init done 2024-04-14T15:18:36.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.050+0000 7f6dd57fa700 1 -- 172.21.15.90:0/101459450 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f6dd0003a00 con 0x7f6dd810e1e0 2024-04-14T15:18:36.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.050+0000 7f6dd57fa700 1 -- 172.21.15.90:0/101459450 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f6dd00050a0 con 0x7f6dd810e1e0 2024-04-14T15:18:36.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.051+0000 7f6dde67c700 1 -- 172.21.15.90:0/101459450 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f6dc4005100 con 0x7f6dd810e1e0 2024-04-14T15:18:36.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.052+0000 7f6dd57fa700 1 -- 172.21.15.90:0/101459450 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f6dd0003d50 con 0x7f6dd810e1e0 2024-04-14T15:18:36.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.052+0000 7f6dd57fa700 1 -- 172.21.15.90:0/101459450 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(429..429 src has 51..429) ==== 8188+0+0 (unknown 250556405 0 0) 0x7f6dd0089c10 con 0x7f6dd810e1e0 2024-04-14T15:18:36.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.052+0000 7f6dd57fa700 10 client.37305.objecter ms_dispatch 0x7f6dd8001550 osd_map(429..429 src has 51..429) 2024-04-14T15:18:36.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.052+0000 7f6dd57fa700 3 client.37305.objecter handle_osd_map got epochs [429,429] > 0 2024-04-14T15:18:36.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.052+0000 7f6dd57fa700 3 client.37305.objecter handle_osd_map decoding full epoch 429 2024-04-14T15:18:36.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.052+0000 7f6dd57fa700 20 client.37305.objecter dump_active .. 0 homeless 2024-04-14T15:18:36.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.052+0000 7f6dd57fa700 1 -- 172.21.15.90:0/101459450 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v429) ==== 54+0+16 (unknown 4196018325 0 2420000192) 0x7f6dd0089ff0 con 0x7f6dd810e1e0 2024-04-14T15:18:36.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.052+0000 7f6dd57fa700 10 client.37305.objecter ms_handle_connect 0x7f6dc0071380 2024-04-14T15:18:36.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.053+0000 7f6dde67c700 10 client.37305.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:18:36.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.053+0000 7f6dde67c700 15 client.37305.objecter _session_command_op_assign -1 1 2024-04-14T15:18:36.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.053+0000 7f6dde67c700 20 client.37305.objecter _get_session s=0x7f6dc40053e0 osd=0 3 2024-04-14T15:18:36.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.053+0000 7f6dde67c700 20 client.37305.objecter put_session s=0x7f6dc40053e0 osd=0 3 2024-04-14T15:18:36.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.053+0000 7f6dde67c700 20 client.37305.objecter _get_session s=0x7f6dc40053e0 osd=0 3 2024-04-14T15:18:36.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.053+0000 7f6dde67c700 15 client.37305.objecter _session_command_op_remove -1 1 2024-04-14T15:18:36.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.053+0000 7f6dde67c700 20 client.37305.objecter get_session s=0x7f6dc40053e0 osd=0 3 2024-04-14T15:18:36.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.053+0000 7f6dde67c700 15 client.37305.objecter _session_command_op_assign 0 1 2024-04-14T15:18:36.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.053+0000 7f6dde67c700 20 client.37305.objecter put_session s=0x7f6dc40053e0 osd=0 4 2024-04-14T15:18:36.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.053+0000 7f6dde67c700 10 client.37305.objecter _send_command 1 2024-04-14T15:18:36.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.053+0000 7f6dde67c700 1 -- 172.21.15.90:0/101459450 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6dc4004b70 con 0x7f6dc4004730 2024-04-14T15:18:36.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.053+0000 7f6dd57fa700 10 client.37305.objecter ms_handle_connect 0x7f6dc4004730 2024-04-14T15:18:36.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.054+0000 7f6dd57fa700 1 -- 172.21.15.90:0/101459450 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6dc4004b70 con 0x7f6dc4004730 2024-04-14T15:18:36.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.054+0000 7f6dd57fa700 10 client.37305.objecter ms_dispatch 0x7f6dd8001550 command_reply(tid 1: 0 ) 2024-04-14T15:18:36.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.054+0000 7f6dd57fa700 10 client.37305.objecter _finish_command 1 = system:0 2024-04-14T15:18:36.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.054+0000 7f6dd57fa700 20 client.37305.objecter put_session s=0x7f6dc40053e0 osd=0 4 2024-04-14T15:18:36.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.054+0000 7f6dd57fa700 15 client.37305.objecter _session_command_op_remove 0 1 2024-04-14T15:18:36.060 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:18:36.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.072+0000 7f6dde67c700 10 client.37305.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:18:36.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.072+0000 7f6dde67c700 15 client.37305.objecter _session_command_op_assign -1 2 2024-04-14T15:18:36.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.072+0000 7f6dde67c700 20 client.37305.objecter _get_session s=0x7f6dc40053e0 osd=0 3 2024-04-14T15:18:36.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.072+0000 7f6dde67c700 20 client.37305.objecter put_session s=0x7f6dc40053e0 osd=0 3 2024-04-14T15:18:36.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.072+0000 7f6dde67c700 20 client.37305.objecter _get_session s=0x7f6dc40053e0 osd=0 3 2024-04-14T15:18:36.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.072+0000 7f6dde67c700 15 client.37305.objecter _session_command_op_remove -1 2 2024-04-14T15:18:36.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.072+0000 7f6dde67c700 20 client.37305.objecter get_session s=0x7f6dc40053e0 osd=0 3 2024-04-14T15:18:36.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.072+0000 7f6dde67c700 15 client.37305.objecter _session_command_op_assign 0 2 2024-04-14T15:18:36.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.072+0000 7f6dde67c700 20 client.37305.objecter put_session s=0x7f6dc40053e0 osd=0 4 2024-04-14T15:18:36.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.072+0000 7f6dde67c700 10 client.37305.objecter _send_command 2 2024-04-14T15:18:36.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.072+0000 7f6dde67c700 1 -- 172.21.15.90:0/101459450 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6dc40079f0 con 0x7f6dc4004730 2024-04-14T15:18:36.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.075+0000 7f6dd57fa700 1 -- 172.21.15.90:0/101459450 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6dc40079f0 con 0x7f6dc4004730 2024-04-14T15:18:36.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.075+0000 7f6dd57fa700 10 client.37305.objecter ms_dispatch 0x7f6dd8001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:18:36.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.075+0000 7f6dd57fa700 10 client.37305.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:18:36.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.075+0000 7f6dd57fa700 20 client.37305.objecter put_session s=0x7f6dc40053e0 osd=0 4 2024-04-14T15:18:36.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.075+0000 7f6dd57fa700 15 client.37305.objecter _session_command_op_remove 0 2 2024-04-14T15:18:36.076 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:18:36.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.075+0000 7f6dde67c700 10 client.37305.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:18:36.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.075+0000 7f6dde67c700 15 client.37305.objecter _session_command_op_assign -1 3 2024-04-14T15:18:36.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.075+0000 7f6dde67c700 20 client.37305.objecter _get_session s=0x7f6dc4004130 osd=1 3 2024-04-14T15:18:36.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.075+0000 7f6dde67c700 20 client.37305.objecter put_session s=0x7f6dc4004130 osd=1 3 2024-04-14T15:18:36.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.075+0000 7f6dde67c700 20 client.37305.objecter _get_session s=0x7f6dc4004130 osd=1 3 2024-04-14T15:18:36.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.075+0000 7f6dde67c700 15 client.37305.objecter _session_command_op_remove -1 3 2024-04-14T15:18:36.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.075+0000 7f6dde67c700 20 client.37305.objecter get_session s=0x7f6dc4004130 osd=1 3 2024-04-14T15:18:36.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.075+0000 7f6dde67c700 15 client.37305.objecter _session_command_op_assign 1 3 2024-04-14T15:18:36.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.075+0000 7f6dde67c700 20 client.37305.objecter put_session s=0x7f6dc4004130 osd=1 4 2024-04-14T15:18:36.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.075+0000 7f6dde67c700 10 client.37305.objecter _send_command 3 2024-04-14T15:18:36.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.075+0000 7f6dde67c700 1 -- 172.21.15.90:0/101459450 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f6dc40022a0 con 0x7f6dc4004310 2024-04-14T15:18:36.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.076+0000 7f6dd57fa700 10 client.37305.objecter ms_handle_connect 0x7f6dc4004310 2024-04-14T15:18:36.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.077+0000 7f6dd57fa700 1 -- 172.21.15.90:0/101459450 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6dc40022a0 con 0x7f6dc4004310 2024-04-14T15:18:36.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.077+0000 7f6dd57fa700 10 client.37305.objecter ms_dispatch 0x7f6dd8001550 command_reply(tid 3: 0 ) 2024-04-14T15:18:36.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.077+0000 7f6dd57fa700 10 client.37305.objecter _finish_command 3 = system:0 2024-04-14T15:18:36.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.077+0000 7f6dd57fa700 20 client.37305.objecter put_session s=0x7f6dc4004130 osd=1 4 2024-04-14T15:18:36.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.077+0000 7f6dd57fa700 15 client.37305.objecter _session_command_op_remove 1 3 2024-04-14T15:18:36.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.096+0000 7f6dde67c700 10 client.37305.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:18:36.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.096+0000 7f6dde67c700 15 client.37305.objecter _session_command_op_assign -1 4 2024-04-14T15:18:36.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.096+0000 7f6dde67c700 20 client.37305.objecter _get_session s=0x7f6dc4004130 osd=1 3 2024-04-14T15:18:36.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.096+0000 7f6dde67c700 20 client.37305.objecter put_session s=0x7f6dc4004130 osd=1 3 2024-04-14T15:18:36.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.096+0000 7f6dde67c700 20 client.37305.objecter _get_session s=0x7f6dc4004130 osd=1 3 2024-04-14T15:18:36.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.096+0000 7f6dde67c700 15 client.37305.objecter _session_command_op_remove -1 4 2024-04-14T15:18:36.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.096+0000 7f6dde67c700 20 client.37305.objecter get_session s=0x7f6dc4004130 osd=1 3 2024-04-14T15:18:36.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.096+0000 7f6dde67c700 15 client.37305.objecter _session_command_op_assign 1 4 2024-04-14T15:18:36.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.096+0000 7f6dde67c700 20 client.37305.objecter put_session s=0x7f6dc4004130 osd=1 4 2024-04-14T15:18:36.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.096+0000 7f6dde67c700 10 client.37305.objecter _send_command 4 2024-04-14T15:18:36.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.096+0000 7f6dde67c700 1 -- 172.21.15.90:0/101459450 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6dc4001720 con 0x7f6dc4004310 2024-04-14T15:18:36.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.098+0000 7f6dd57fa700 1 -- 172.21.15.90:0/101459450 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6dc4001720 con 0x7f6dc4004310 2024-04-14T15:18:36.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.098+0000 7f6dd57fa700 10 client.37305.objecter ms_dispatch 0x7f6dd8001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:18:36.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.098+0000 7f6dd57fa700 10 client.37305.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:18:36.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.098+0000 7f6dd57fa700 20 client.37305.objecter put_session s=0x7f6dc4004130 osd=1 4 2024-04-14T15:18:36.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.098+0000 7f6dd57fa700 15 client.37305.objecter _session_command_op_remove 1 4 2024-04-14T15:18:36.098 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:18:36.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.098+0000 7f6dde67c700 10 client.37305.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:18:36.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.098+0000 7f6dde67c700 15 client.37305.objecter _session_command_op_assign -1 5 2024-04-14T15:18:36.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.098+0000 7f6dde67c700 20 client.37305.objecter _get_session s=0x7f6dc4001980 osd=2 3 2024-04-14T15:18:36.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.098+0000 7f6dde67c700 20 client.37305.objecter put_session s=0x7f6dc4001980 osd=2 3 2024-04-14T15:18:36.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.098+0000 7f6dde67c700 20 client.37305.objecter _get_session s=0x7f6dc4001980 osd=2 3 2024-04-14T15:18:36.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.098+0000 7f6dde67c700 15 client.37305.objecter _session_command_op_remove -1 5 2024-04-14T15:18:36.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.098+0000 7f6dde67c700 20 client.37305.objecter get_session s=0x7f6dc4001980 osd=2 3 2024-04-14T15:18:36.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.098+0000 7f6dde67c700 15 client.37305.objecter _session_command_op_assign 2 5 2024-04-14T15:18:36.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.098+0000 7f6dde67c700 20 client.37305.objecter put_session s=0x7f6dc4001980 osd=2 4 2024-04-14T15:18:36.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.098+0000 7f6dde67c700 10 client.37305.objecter _send_command 5 2024-04-14T15:18:36.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.098+0000 7f6dde67c700 1 -- 172.21.15.90:0/101459450 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f6dc4011170 con 0x7f6dc400d5d0 2024-04-14T15:18:36.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.098+0000 7f6dd57fa700 10 client.37305.objecter ms_handle_connect 0x7f6dc400d5d0 2024-04-14T15:18:36.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.099+0000 7f6dd57fa700 1 -- 172.21.15.90:0/101459450 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6dc4011170 con 0x7f6dc400d5d0 2024-04-14T15:18:36.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.099+0000 7f6dd57fa700 10 client.37305.objecter ms_dispatch 0x7f6dd8001550 command_reply(tid 5: 0 ) 2024-04-14T15:18:36.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.099+0000 7f6dd57fa700 10 client.37305.objecter _finish_command 5 = system:0 2024-04-14T15:18:36.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.099+0000 7f6dd57fa700 20 client.37305.objecter put_session s=0x7f6dc4001980 osd=2 4 2024-04-14T15:18:36.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.099+0000 7f6dd57fa700 15 client.37305.objecter _session_command_op_remove 2 5 2024-04-14T15:18:36.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.115+0000 7f6dde67c700 10 client.37305.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:18:36.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.115+0000 7f6dde67c700 15 client.37305.objecter _session_command_op_assign -1 6 2024-04-14T15:18:36.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.115+0000 7f6dde67c700 20 client.37305.objecter _get_session s=0x7f6dc4001980 osd=2 3 2024-04-14T15:18:36.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.115+0000 7f6dde67c700 20 client.37305.objecter put_session s=0x7f6dc4001980 osd=2 3 2024-04-14T15:18:36.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.115+0000 7f6dde67c700 20 client.37305.objecter _get_session s=0x7f6dc4001980 osd=2 3 2024-04-14T15:18:36.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.115+0000 7f6dde67c700 15 client.37305.objecter _session_command_op_remove -1 6 2024-04-14T15:18:36.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.115+0000 7f6dde67c700 20 client.37305.objecter get_session s=0x7f6dc4001980 osd=2 3 2024-04-14T15:18:36.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.115+0000 7f6dde67c700 15 client.37305.objecter _session_command_op_assign 2 6 2024-04-14T15:18:36.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.115+0000 7f6dde67c700 20 client.37305.objecter put_session s=0x7f6dc4001980 osd=2 4 2024-04-14T15:18:36.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.115+0000 7f6dde67c700 10 client.37305.objecter _send_command 6 2024-04-14T15:18:36.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.115+0000 7f6dde67c700 1 -- 172.21.15.90:0/101459450 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6dc4008100 con 0x7f6dc400d5d0 2024-04-14T15:18:36.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.117+0000 7f6dd57fa700 1 -- 172.21.15.90:0/101459450 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6dc4008100 con 0x7f6dc400d5d0 2024-04-14T15:18:36.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.117+0000 7f6dd57fa700 10 client.37305.objecter ms_dispatch 0x7f6dd8001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:18:36.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.117+0000 7f6dd57fa700 10 client.37305.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:18:36.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.117+0000 7f6dd57fa700 20 client.37305.objecter put_session s=0x7f6dc4001980 osd=2 4 2024-04-14T15:18:36.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.117+0000 7f6dd57fa700 15 client.37305.objecter _session_command_op_remove 2 6 2024-04-14T15:18:36.118 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:18:36.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.117+0000 7f6dde67c700 10 client.37305.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:18:36.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.117+0000 7f6dde67c700 15 client.37305.objecter _session_command_op_assign -1 7 2024-04-14T15:18:36.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.117+0000 7f6dde67c700 20 client.37305.objecter _get_session s=0x7f6dc400cc20 osd=3 3 2024-04-14T15:18:36.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.117+0000 7f6dde67c700 20 client.37305.objecter put_session s=0x7f6dc400cc20 osd=3 3 2024-04-14T15:18:36.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.117+0000 7f6dde67c700 20 client.37305.objecter _get_session s=0x7f6dc400cc20 osd=3 3 2024-04-14T15:18:36.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.117+0000 7f6dde67c700 15 client.37305.objecter _session_command_op_remove -1 7 2024-04-14T15:18:36.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.117+0000 7f6dde67c700 20 client.37305.objecter get_session s=0x7f6dc400cc20 osd=3 3 2024-04-14T15:18:36.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.117+0000 7f6dde67c700 15 client.37305.objecter _session_command_op_assign 3 7 2024-04-14T15:18:36.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.117+0000 7f6dde67c700 20 client.37305.objecter put_session s=0x7f6dc400cc20 osd=3 4 2024-04-14T15:18:36.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.117+0000 7f6dde67c700 10 client.37305.objecter _send_command 7 2024-04-14T15:18:36.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.117+0000 7f6dde67c700 1 -- 172.21.15.90:0/101459450 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f6dc4015fb0 con 0x7f6dc40124a0 2024-04-14T15:18:36.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.117+0000 7f6dd57fa700 10 client.37305.objecter ms_handle_connect 0x7f6dc40124a0 2024-04-14T15:18:36.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.118+0000 7f6dd57fa700 1 -- 172.21.15.90:0/101459450 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6dc4015fb0 con 0x7f6dc40124a0 2024-04-14T15:18:36.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.118+0000 7f6dd57fa700 10 client.37305.objecter ms_dispatch 0x7f6dd8001550 command_reply(tid 7: 0 ) 2024-04-14T15:18:36.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.118+0000 7f6dd57fa700 10 client.37305.objecter _finish_command 7 = system:0 2024-04-14T15:18:36.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.118+0000 7f6dd57fa700 20 client.37305.objecter put_session s=0x7f6dc400cc20 osd=3 4 2024-04-14T15:18:36.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.118+0000 7f6dd57fa700 15 client.37305.objecter _session_command_op_remove 3 7 2024-04-14T15:18:36.121 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:18:36.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:36 smithi092 ceph-mon[33966]: 110.8 scrub starts 2024-04-14T15:18:36.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:36 smithi092 ceph-mon[33966]: 110.8 scrub ok 2024-04-14T15:18:36.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:36 smithi092 ceph-mon[33966]: 11.17 deep-scrub starts 2024-04-14T15:18:36.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:36 smithi092 ceph-mon[33966]: 11.17 deep-scrub ok 2024-04-14T15:18:36.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:36 smithi092 ceph-mon[33966]: 11.c deep-scrub starts 2024-04-14T15:18:36.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:36 smithi092 ceph-mon[33966]: 11.c deep-scrub ok 2024-04-14T15:18:36.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:36 smithi092 ceph-mon[33966]: pgmap v598: 238 pgs: 238 active+clean; 8.4 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-14T15:18:36.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:36 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:18:36.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:36 smithi092 ceph-mon[33966]: 4.1b scrub starts 2024-04-14T15:18:36.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:36 smithi092 ceph-mon[33966]: 4.1b scrub ok 2024-04-14T15:18:36.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:36 smithi092 ceph-mon[33966]: osdmap e429: 8 total, 8 up, 8 in 2024-04-14T15:18:36.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.135+0000 7f6dde67c700 10 client.37305.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:18:36.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.135+0000 7f6dde67c700 15 client.37305.objecter _session_command_op_assign -1 8 2024-04-14T15:18:36.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.135+0000 7f6dde67c700 20 client.37305.objecter _get_session s=0x7f6dc400cc20 osd=3 3 2024-04-14T15:18:36.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.135+0000 7f6dde67c700 20 client.37305.objecter put_session s=0x7f6dc400cc20 osd=3 3 2024-04-14T15:18:36.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.135+0000 7f6dde67c700 20 client.37305.objecter _get_session s=0x7f6dc400cc20 osd=3 3 2024-04-14T15:18:36.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.135+0000 7f6dde67c700 15 client.37305.objecter _session_command_op_remove -1 8 2024-04-14T15:18:36.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.135+0000 7f6dde67c700 20 client.37305.objecter get_session s=0x7f6dc400cc20 osd=3 3 2024-04-14T15:18:36.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.135+0000 7f6dde67c700 15 client.37305.objecter _session_command_op_assign 3 8 2024-04-14T15:18:36.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.135+0000 7f6dde67c700 20 client.37305.objecter put_session s=0x7f6dc400cc20 osd=3 4 2024-04-14T15:18:36.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.135+0000 7f6dde67c700 10 client.37305.objecter _send_command 8 2024-04-14T15:18:36.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.135+0000 7f6dde67c700 1 -- 172.21.15.90:0/101459450 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6dc400c6a0 con 0x7f6dc40124a0 2024-04-14T15:18:36.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.136+0000 7f6dd57fa700 1 -- 172.21.15.90:0/101459450 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6dc400c6a0 con 0x7f6dc40124a0 2024-04-14T15:18:36.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.136+0000 7f6dd57fa700 10 client.37305.objecter ms_dispatch 0x7f6dd8001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:18:36.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.136+0000 7f6dd57fa700 10 client.37305.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:18:36.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.136+0000 7f6dd57fa700 20 client.37305.objecter put_session s=0x7f6dc400cc20 osd=3 4 2024-04-14T15:18:36.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.136+0000 7f6dd57fa700 15 client.37305.objecter _session_command_op_remove 3 8 2024-04-14T15:18:36.137 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:18:36.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.137+0000 7f6dde67c700 10 client.37305.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:18:36.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.137+0000 7f6dde67c700 15 client.37305.objecter _session_command_op_assign -1 9 2024-04-14T15:18:36.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.137+0000 7f6dde67c700 20 client.37305.objecter _get_session s=0x7f6dc400c450 osd=4 3 2024-04-14T15:18:36.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.137+0000 7f6dde67c700 20 client.37305.objecter put_session s=0x7f6dc400c450 osd=4 3 2024-04-14T15:18:36.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.137+0000 7f6dde67c700 20 client.37305.objecter _get_session s=0x7f6dc400c450 osd=4 3 2024-04-14T15:18:36.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.137+0000 7f6dde67c700 15 client.37305.objecter _session_command_op_remove -1 9 2024-04-14T15:18:36.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.137+0000 7f6dde67c700 20 client.37305.objecter get_session s=0x7f6dc400c450 osd=4 3 2024-04-14T15:18:36.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.137+0000 7f6dde67c700 15 client.37305.objecter _session_command_op_assign 4 9 2024-04-14T15:18:36.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.137+0000 7f6dde67c700 20 client.37305.objecter put_session s=0x7f6dc400c450 osd=4 4 2024-04-14T15:18:36.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.137+0000 7f6dde67c700 10 client.37305.objecter _send_command 9 2024-04-14T15:18:36.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.137+0000 7f6dde67c700 1 -- 172.21.15.90:0/101459450 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f6dc400c680 con 0x7f6dc4012000 2024-04-14T15:18:36.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.137+0000 7f6dd57fa700 10 client.37305.objecter ms_handle_connect 0x7f6dc4012000 2024-04-14T15:18:36.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.138+0000 7f6dd57fa700 1 -- 172.21.15.90:0/101459450 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6dc400c680 con 0x7f6dc4012000 2024-04-14T15:18:36.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.138+0000 7f6dd57fa700 10 client.37305.objecter ms_dispatch 0x7f6dd8001550 command_reply(tid 9: 0 ) 2024-04-14T15:18:36.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.138+0000 7f6dd57fa700 10 client.37305.objecter _finish_command 9 = system:0 2024-04-14T15:18:36.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.138+0000 7f6dd57fa700 20 client.37305.objecter put_session s=0x7f6dc400c450 osd=4 4 2024-04-14T15:18:36.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.138+0000 7f6dd57fa700 15 client.37305.objecter _session_command_op_remove 4 9 2024-04-14T15:18:36.145 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:36.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:36.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:18:36.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.154+0000 7f6dde67c700 10 client.37305.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:18:36.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.154+0000 7f6dde67c700 15 client.37305.objecter _session_command_op_assign -1 10 2024-04-14T15:18:36.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.154+0000 7f6dde67c700 20 client.37305.objecter _get_session s=0x7f6dc400c450 osd=4 3 2024-04-14T15:18:36.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.154+0000 7f6dde67c700 20 client.37305.objecter put_session s=0x7f6dc400c450 osd=4 3 2024-04-14T15:18:36.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.154+0000 7f6dde67c700 20 client.37305.objecter _get_session s=0x7f6dc400c450 osd=4 3 2024-04-14T15:18:36.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.154+0000 7f6dde67c700 15 client.37305.objecter _session_command_op_remove -1 10 2024-04-14T15:18:36.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.154+0000 7f6dde67c700 20 client.37305.objecter get_session s=0x7f6dc400c450 osd=4 3 2024-04-14T15:18:36.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.154+0000 7f6dde67c700 15 client.37305.objecter _session_command_op_assign 4 10 2024-04-14T15:18:36.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.154+0000 7f6dde67c700 20 client.37305.objecter put_session s=0x7f6dc400c450 osd=4 4 2024-04-14T15:18:36.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.154+0000 7f6dde67c700 10 client.37305.objecter _send_command 10 2024-04-14T15:18:36.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.154+0000 7f6dde67c700 1 -- 172.21.15.90:0/101459450 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6dc401c1e0 con 0x7f6dc4012000 2024-04-14T15:18:36.156 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:18:36.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.155+0000 7f6dd57fa700 1 -- 172.21.15.90:0/101459450 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6dc401c1e0 con 0x7f6dc4012000 2024-04-14T15:18:36.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.155+0000 7f6dd57fa700 10 client.37305.objecter ms_dispatch 0x7f6dd8001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:18:36.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.155+0000 7f6dd57fa700 10 client.37305.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:18:36.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.155+0000 7f6dd57fa700 20 client.37305.objecter put_session s=0x7f6dc400c450 osd=4 4 2024-04-14T15:18:36.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.155+0000 7f6dd57fa700 15 client.37305.objecter _session_command_op_remove 4 10 2024-04-14T15:18:36.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.156+0000 7f6dde67c700 10 client.37305.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:18:36.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.156+0000 7f6dde67c700 15 client.37305.objecter _session_command_op_assign -1 11 2024-04-14T15:18:36.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.156+0000 7f6dde67c700 20 client.37305.objecter _get_session s=0x7f6dc401c1e0 osd=5 3 2024-04-14T15:18:36.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.156+0000 7f6dde67c700 20 client.37305.objecter put_session s=0x7f6dc401c1e0 osd=5 3 2024-04-14T15:18:36.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.156+0000 7f6dde67c700 20 client.37305.objecter _get_session s=0x7f6dc401c1e0 osd=5 3 2024-04-14T15:18:36.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.156+0000 7f6dde67c700 15 client.37305.objecter _session_command_op_remove -1 11 2024-04-14T15:18:36.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.156+0000 7f6dde67c700 20 client.37305.objecter get_session s=0x7f6dc401c1e0 osd=5 3 2024-04-14T15:18:36.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.156+0000 7f6dde67c700 15 client.37305.objecter _session_command_op_assign 5 11 2024-04-14T15:18:36.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.156+0000 7f6dde67c700 20 client.37305.objecter put_session s=0x7f6dc401c1e0 osd=5 4 2024-04-14T15:18:36.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.156+0000 7f6dde67c700 10 client.37305.objecter _send_command 11 2024-04-14T15:18:36.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.156+0000 7f6dde67c700 1 -- 172.21.15.90:0/101459450 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f6dc401cd20 con 0x7f6dc401c860 2024-04-14T15:18:36.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.156+0000 7f6dd57fa700 10 client.37305.objecter ms_handle_connect 0x7f6dc401c860 2024-04-14T15:18:36.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.157+0000 7f6dd57fa700 1 -- 172.21.15.90:0/101459450 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6dc401cd20 con 0x7f6dc401c860 2024-04-14T15:18:36.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.157+0000 7f6dd57fa700 10 client.37305.objecter ms_dispatch 0x7f6dd8001550 command_reply(tid 11: 0 ) 2024-04-14T15:18:36.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.157+0000 7f6dd57fa700 10 client.37305.objecter _finish_command 11 = system:0 2024-04-14T15:18:36.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.157+0000 7f6dd57fa700 20 client.37305.objecter put_session s=0x7f6dc401c1e0 osd=5 4 2024-04-14T15:18:36.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.157+0000 7f6dd57fa700 15 client.37305.objecter _session_command_op_remove 5 11 2024-04-14T15:18:36.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.174+0000 7f6dde67c700 10 client.37305.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:18:36.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.174+0000 7f6dde67c700 15 client.37305.objecter _session_command_op_assign -1 12 2024-04-14T15:18:36.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.174+0000 7f6dde67c700 20 client.37305.objecter _get_session s=0x7f6dc401c1e0 osd=5 3 2024-04-14T15:18:36.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.174+0000 7f6dde67c700 20 client.37305.objecter put_session s=0x7f6dc401c1e0 osd=5 3 2024-04-14T15:18:36.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.174+0000 7f6dde67c700 20 client.37305.objecter _get_session s=0x7f6dc401c1e0 osd=5 3 2024-04-14T15:18:36.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.174+0000 7f6dde67c700 15 client.37305.objecter _session_command_op_remove -1 12 2024-04-14T15:18:36.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.174+0000 7f6dde67c700 20 client.37305.objecter get_session s=0x7f6dc401c1e0 osd=5 3 2024-04-14T15:18:36.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.174+0000 7f6dde67c700 15 client.37305.objecter _session_command_op_assign 5 12 2024-04-14T15:18:36.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.174+0000 7f6dde67c700 20 client.37305.objecter put_session s=0x7f6dc401c1e0 osd=5 4 2024-04-14T15:18:36.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.174+0000 7f6dde67c700 10 client.37305.objecter _send_command 12 2024-04-14T15:18:36.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.174+0000 7f6dde67c700 1 -- 172.21.15.90:0/101459450 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6dc401cfa0 con 0x7f6dc401c860 2024-04-14T15:18:36.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.175+0000 7f6dd57fa700 1 -- 172.21.15.90:0/101459450 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6dc401cfa0 con 0x7f6dc401c860 2024-04-14T15:18:36.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.175+0000 7f6dd57fa700 10 client.37305.objecter ms_dispatch 0x7f6dd8001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:18:36.176 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:18:36.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.175+0000 7f6dd57fa700 10 client.37305.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:18:36.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.175+0000 7f6dd57fa700 20 client.37305.objecter put_session s=0x7f6dc401c1e0 osd=5 4 2024-04-14T15:18:36.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.175+0000 7f6dd57fa700 15 client.37305.objecter _session_command_op_remove 5 12 2024-04-14T15:18:36.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.175+0000 7f6dde67c700 10 client.37305.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:18:36.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.175+0000 7f6dde67c700 15 client.37305.objecter _session_command_op_assign -1 13 2024-04-14T15:18:36.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.175+0000 7f6dde67c700 20 client.37305.objecter _get_session s=0x7f6dc401cd80 osd=6 3 2024-04-14T15:18:36.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.175+0000 7f6dde67c700 20 client.37305.objecter put_session s=0x7f6dc401cd80 osd=6 3 2024-04-14T15:18:36.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.175+0000 7f6dde67c700 20 client.37305.objecter _get_session s=0x7f6dc401cd80 osd=6 3 2024-04-14T15:18:36.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.175+0000 7f6dde67c700 15 client.37305.objecter _session_command_op_remove -1 13 2024-04-14T15:18:36.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.175+0000 7f6dde67c700 20 client.37305.objecter get_session s=0x7f6dc401cd80 osd=6 3 2024-04-14T15:18:36.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.175+0000 7f6dde67c700 15 client.37305.objecter _session_command_op_assign 6 13 2024-04-14T15:18:36.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.175+0000 7f6dde67c700 20 client.37305.objecter put_session s=0x7f6dc401cd80 osd=6 4 2024-04-14T15:18:36.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.175+0000 7f6dde67c700 10 client.37305.objecter _send_command 13 2024-04-14T15:18:36.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.175+0000 7f6dde67c700 1 -- 172.21.15.90:0/101459450 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f6dc401d190 con 0x7f6dc401f8d0 2024-04-14T15:18:36.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.176+0000 7f6dd57fa700 10 client.37305.objecter ms_handle_connect 0x7f6dc401f8d0 2024-04-14T15:18:36.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.176+0000 7f6dd57fa700 1 -- 172.21.15.90:0/101459450 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6dc401d190 con 0x7f6dc401f8d0 2024-04-14T15:18:36.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.176+0000 7f6dd57fa700 10 client.37305.objecter ms_dispatch 0x7f6dd8001550 command_reply(tid 13: 0 ) 2024-04-14T15:18:36.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.176+0000 7f6dd57fa700 10 client.37305.objecter _finish_command 13 = system:0 2024-04-14T15:18:36.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.176+0000 7f6dd57fa700 20 client.37305.objecter put_session s=0x7f6dc401cd80 osd=6 4 2024-04-14T15:18:36.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.176+0000 7f6dd57fa700 15 client.37305.objecter _session_command_op_remove 6 13 2024-04-14T15:18:36.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.192+0000 7f6dde67c700 10 client.37305.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:18:36.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.192+0000 7f6dde67c700 15 client.37305.objecter _session_command_op_assign -1 14 2024-04-14T15:18:36.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.192+0000 7f6dde67c700 20 client.37305.objecter _get_session s=0x7f6dc401cd80 osd=6 3 2024-04-14T15:18:36.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.192+0000 7f6dde67c700 20 client.37305.objecter put_session s=0x7f6dc401cd80 osd=6 3 2024-04-14T15:18:36.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.192+0000 7f6dde67c700 20 client.37305.objecter _get_session s=0x7f6dc401cd80 osd=6 3 2024-04-14T15:18:36.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.192+0000 7f6dde67c700 15 client.37305.objecter _session_command_op_remove -1 14 2024-04-14T15:18:36.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.192+0000 7f6dde67c700 20 client.37305.objecter get_session s=0x7f6dc401cd80 osd=6 3 2024-04-14T15:18:36.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.192+0000 7f6dde67c700 15 client.37305.objecter _session_command_op_assign 6 14 2024-04-14T15:18:36.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.192+0000 7f6dde67c700 20 client.37305.objecter put_session s=0x7f6dc401cd80 osd=6 4 2024-04-14T15:18:36.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.192+0000 7f6dde67c700 10 client.37305.objecter _send_command 14 2024-04-14T15:18:36.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.192+0000 7f6dde67c700 1 -- 172.21.15.90:0/101459450 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6dc40233e0 con 0x7f6dc401f8d0 2024-04-14T15:18:36.194 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:18:36.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.194+0000 7f6dd57fa700 1 -- 172.21.15.90:0/101459450 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6dc40233e0 con 0x7f6dc401f8d0 2024-04-14T15:18:36.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.194+0000 7f6dd57fa700 10 client.37305.objecter ms_dispatch 0x7f6dd8001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:18:36.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.194+0000 7f6dd57fa700 10 client.37305.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:18:36.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.194+0000 7f6dd57fa700 20 client.37305.objecter put_session s=0x7f6dc401cd80 osd=6 4 2024-04-14T15:18:36.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.194+0000 7f6dd57fa700 15 client.37305.objecter _session_command_op_remove 6 14 2024-04-14T15:18:36.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.194+0000 7f6dde67c700 10 client.37305.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:18:36.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.194+0000 7f6dde67c700 15 client.37305.objecter _session_command_op_assign -1 15 2024-04-14T15:18:36.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.194+0000 7f6dde67c700 20 client.37305.objecter _get_session s=0x7f6dc40233e0 osd=7 3 2024-04-14T15:18:36.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.194+0000 7f6dde67c700 20 client.37305.objecter put_session s=0x7f6dc40233e0 osd=7 3 2024-04-14T15:18:36.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.194+0000 7f6dde67c700 20 client.37305.objecter _get_session s=0x7f6dc40233e0 osd=7 3 2024-04-14T15:18:36.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.194+0000 7f6dde67c700 15 client.37305.objecter _session_command_op_remove -1 15 2024-04-14T15:18:36.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.194+0000 7f6dde67c700 20 client.37305.objecter get_session s=0x7f6dc40233e0 osd=7 3 2024-04-14T15:18:36.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.194+0000 7f6dde67c700 15 client.37305.objecter _session_command_op_assign 7 15 2024-04-14T15:18:36.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.194+0000 7f6dde67c700 20 client.37305.objecter put_session s=0x7f6dc40233e0 osd=7 4 2024-04-14T15:18:36.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.194+0000 7f6dde67c700 10 client.37305.objecter _send_command 15 2024-04-14T15:18:36.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.194+0000 7f6dde67c700 1 -- 172.21.15.90:0/101459450 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f6dc4024710 con 0x7f6dc4023b00 2024-04-14T15:18:36.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.195+0000 7f6dd57fa700 10 client.37305.objecter ms_handle_connect 0x7f6dc4023b00 2024-04-14T15:18:36.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.196+0000 7f6dd57fa700 1 -- 172.21.15.90:0/101459450 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6dc4024710 con 0x7f6dc4023b00 2024-04-14T15:18:36.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.196+0000 7f6dd57fa700 10 client.37305.objecter ms_dispatch 0x7f6dd8001550 command_reply(tid 15: 0 ) 2024-04-14T15:18:36.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.196+0000 7f6dd57fa700 10 client.37305.objecter _finish_command 15 = system:0 2024-04-14T15:18:36.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.196+0000 7f6dd57fa700 20 client.37305.objecter put_session s=0x7f6dc40233e0 osd=7 4 2024-04-14T15:18:36.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.196+0000 7f6dd57fa700 15 client.37305.objecter _session_command_op_remove 7 15 2024-04-14T15:18:36.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.212+0000 7f6dde67c700 10 client.37305.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:18:36.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.212+0000 7f6dde67c700 15 client.37305.objecter _session_command_op_assign -1 16 2024-04-14T15:18:36.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.212+0000 7f6dde67c700 20 client.37305.objecter _get_session s=0x7f6dc40233e0 osd=7 3 2024-04-14T15:18:36.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.212+0000 7f6dde67c700 20 client.37305.objecter put_session s=0x7f6dc40233e0 osd=7 3 2024-04-14T15:18:36.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.212+0000 7f6dde67c700 20 client.37305.objecter _get_session s=0x7f6dc40233e0 osd=7 3 2024-04-14T15:18:36.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.212+0000 7f6dde67c700 15 client.37305.objecter _session_command_op_remove -1 16 2024-04-14T15:18:36.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.212+0000 7f6dde67c700 20 client.37305.objecter get_session s=0x7f6dc40233e0 osd=7 3 2024-04-14T15:18:36.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.212+0000 7f6dde67c700 15 client.37305.objecter _session_command_op_assign 7 16 2024-04-14T15:18:36.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.212+0000 7f6dde67c700 20 client.37305.objecter put_session s=0x7f6dc40233e0 osd=7 4 2024-04-14T15:18:36.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.212+0000 7f6dde67c700 10 client.37305.objecter _send_command 16 2024-04-14T15:18:36.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.212+0000 7f6dde67c700 1 -- 172.21.15.90:0/101459450 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6dc4024710 con 0x7f6dc4023b00 2024-04-14T15:18:36.214 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:18:36.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.214+0000 7f6dd57fa700 1 -- 172.21.15.90:0/101459450 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6dc4024710 con 0x7f6dc4023b00 2024-04-14T15:18:36.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.214+0000 7f6dd57fa700 10 client.37305.objecter ms_dispatch 0x7f6dd8001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:18:36.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.214+0000 7f6dd57fa700 10 client.37305.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:18:36.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.214+0000 7f6dd57fa700 20 client.37305.objecter put_session s=0x7f6dc40233e0 osd=7 4 2024-04-14T15:18:36.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.214+0000 7f6dd57fa700 15 client.37305.objecter _session_command_op_remove 7 16 2024-04-14T15:18:36.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.214+0000 7f6dde67c700 10 librados: watch_flush enter 2024-04-14T15:18:36.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.214+0000 7f6dde67c700 10 librados: watch_flush exit 2024-04-14T15:18:36.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.214+0000 7f6dde67c700 10 client.37305.objecter close_session for osd.0 2024-04-14T15:18:36.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.214+0000 7f6dde67c700 1 -- 172.21.15.90:0/101459450 >> v1:172.21.15.90:6801/3761806885 conn(0x7f6dc4004730 legacy=0x7f6dc4002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:36.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.214+0000 7f6dde67c700 20 client.37305.objecter put_session s=0x7f6dc40053e0 osd=0 1 2024-04-14T15:18:36.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.214+0000 7f6dde67c700 10 client.37305.objecter close_session for osd.1 2024-04-14T15:18:36.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.214+0000 7f6dde67c700 1 -- 172.21.15.90:0/101459450 >> v1:172.21.15.90:6805/4170361699 conn(0x7f6dc4004310 legacy=0x7f6dc400ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:36.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.214+0000 7f6dde67c700 20 client.37305.objecter put_session s=0x7f6dc4004130 osd=1 1 2024-04-14T15:18:36.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.214+0000 7f6dde67c700 10 client.37305.objecter close_session for osd.2 2024-04-14T15:18:36.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.214+0000 7f6dde67c700 1 -- 172.21.15.90:0/101459450 >> v1:172.21.15.90:6809/1686184251 conn(0x7f6dc400d5d0 legacy=0x7f6dc400f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:36.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.214+0000 7f6dde67c700 20 client.37305.objecter put_session s=0x7f6dc4001980 osd=2 1 2024-04-14T15:18:36.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.214+0000 7f6dde67c700 10 client.37305.objecter close_session for osd.3 2024-04-14T15:18:36.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.214+0000 7f6dde67c700 1 -- 172.21.15.90:0/101459450 >> v1:172.21.15.90:6813/2270166774 conn(0x7f6dc40124a0 legacy=0x7f6dc4014890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:36.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.214+0000 7f6dde67c700 20 client.37305.objecter put_session s=0x7f6dc400cc20 osd=3 1 2024-04-14T15:18:36.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.214+0000 7f6dde67c700 10 client.37305.objecter close_session for osd.4 2024-04-14T15:18:36.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.214+0000 7f6dde67c700 1 -- 172.21.15.90:0/101459450 >> v1:172.21.15.92:6800/3887549733 conn(0x7f6dc4012000 legacy=0x7f6dc40192f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:36.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.214+0000 7f6dde67c700 20 client.37305.objecter put_session s=0x7f6dc400c450 osd=4 1 2024-04-14T15:18:36.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.214+0000 7f6dde67c700 10 client.37305.objecter close_session for osd.5 2024-04-14T15:18:36.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.214+0000 7f6dde67c700 1 -- 172.21.15.90:0/101459450 >> v1:172.21.15.92:6804/3244561440 conn(0x7f6dc401c860 legacy=0x7f6dc401aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:36.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.214+0000 7f6dde67c700 20 client.37305.objecter put_session s=0x7f6dc401c1e0 osd=5 1 2024-04-14T15:18:36.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.214+0000 7f6dde67c700 10 client.37305.objecter close_session for osd.6 2024-04-14T15:18:36.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.214+0000 7f6dde67c700 1 -- 172.21.15.90:0/101459450 >> v1:172.21.15.92:6808/636125963 conn(0x7f6dc401f8d0 legacy=0x7f6dc4021cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:36.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.214+0000 7f6dde67c700 20 client.37305.objecter put_session s=0x7f6dc401cd80 osd=6 1 2024-04-14T15:18:36.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.214+0000 7f6dde67c700 10 client.37305.objecter close_session for osd.7 2024-04-14T15:18:36.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.214+0000 7f6dde67c700 1 -- 172.21.15.90:0/101459450 >> v1:172.21.15.92:6812/1773706716 conn(0x7f6dc4023b00 legacy=0x7f6dc4023f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:36.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.214+0000 7f6dde67c700 20 client.37305.objecter put_session s=0x7f6dc40233e0 osd=7 1 2024-04-14T15:18:36.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.214+0000 7f6dde67c700 20 client.37305.objecter shutdown clearing up homeless session... 2024-04-14T15:18:36.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.214+0000 7f6dde67c700 10 client.37305.objecter successfully canceled tick 2024-04-14T15:18:36.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.215+0000 7f6ddcc19700 1 -- 172.21.15.90:0/101459450 reap_dead start 2024-04-14T15:18:36.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.215+0000 7f6ddcc19700 1 -- 172.21.15.90:0/101459450 reap_dead start 2024-04-14T15:18:36.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.215+0000 7f6ddcc19700 1 -- 172.21.15.90:0/101459450 reap_dead start 2024-04-14T15:18:36.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.215+0000 7f6ddcc19700 1 -- 172.21.15.90:0/101459450 reap_dead start 2024-04-14T15:18:36.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.215+0000 7f6ddcc19700 1 -- 172.21.15.90:0/101459450 reap_dead start 2024-04-14T15:18:36.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.215+0000 7f6dde67c700 1 -- 172.21.15.90:0/101459450 >> v1:172.21.15.90:6800/420935427 conn(0x7f6dc0071380 legacy=0x7f6dc0073840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:36.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.215+0000 7f6dde67c700 1 -- 172.21.15.90:0/101459450 >> v1:172.21.15.90:6789/0 conn(0x7f6dd810e1e0 legacy=0x7f6dd81a8ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:36.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.215+0000 7f6dde67c700 1 -- 172.21.15.90:0/101459450 shutdown_connections 2024-04-14T15:18:36.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.215+0000 7f6dde67c700 1 -- 172.21.15.90:0/101459450 >> 172.21.15.90:0/101459450 conn(0x7f6dd80fd290 msgr2=0x7f6dd806a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:18:36.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.215+0000 7f6dde67c700 1 -- 172.21.15.90:0/101459450 shutdown_connections 2024-04-14T15:18:36.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.215+0000 7f6dde67c700 1 -- 172.21.15.90:0/101459450 wait complete. 2024-04-14T15:18:36.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:36.215+0000 7f6dde67c700 1 librados: shutdown 2024-04-14T15:18:36.219 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:18:36.219 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:18:36.219 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:18:36.219 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:18:36.219 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:18:36.219 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:18:36.219 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:18:36.219 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:18:36.247 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:18:36.271 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:36.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:36.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:18:36.373 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:18:36.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:36.397 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:36.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:18:36.498 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:18:36.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:36.522 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:36.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:18:36.623 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:18:36.647 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:36.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:36.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:18:36.749 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:18:36.773 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:36.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:36.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:18:36.875 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:36.899 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:36.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:36.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:37.001 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:18:37.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:37.025 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:37.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:18:37.126 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:18:37.150 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:37.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:37.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:18:37.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:37 smithi090 ceph-mon[41670]: 11.1e scrub starts 2024-04-14T15:18:37.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:37 smithi090 ceph-mon[41670]: 11.1e scrub ok 2024-04-14T15:18:37.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:37 smithi090 ceph-mon[41670]: 1.a scrub starts 2024-04-14T15:18:37.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:37 smithi090 ceph-mon[41670]: 1.a scrub ok 2024-04-14T15:18:37.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:37 smithi090 ceph-mon[41670]: 110.15 scrub starts 2024-04-14T15:18:37.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:37 smithi090 ceph-mon[41670]: 110.15 scrub ok 2024-04-14T15:18:37.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:37 smithi090 ceph-mon[41670]: 2.d scrub starts 2024-04-14T15:18:37.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:37 smithi090 ceph-mon[41670]: 2.d scrub ok 2024-04-14T15:18:37.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:37 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/101459450' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:18:37.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:37 smithi090 ceph-mon[41670]: 4.18 scrub starts 2024-04-14T15:18:37.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:37 smithi090 ceph-mon[41670]: 4.18 scrub ok 2024-04-14T15:18:37.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:37 smithi090 ceph-mon[30399]: 11.1e scrub starts 2024-04-14T15:18:37.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:37 smithi090 ceph-mon[30399]: 11.1e scrub ok 2024-04-14T15:18:37.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:37 smithi090 ceph-mon[30399]: 1.a scrub starts 2024-04-14T15:18:37.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:37 smithi090 ceph-mon[30399]: 1.a scrub ok 2024-04-14T15:18:37.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:37 smithi090 ceph-mon[30399]: 110.15 scrub starts 2024-04-14T15:18:37.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:37 smithi090 ceph-mon[30399]: 110.15 scrub ok 2024-04-14T15:18:37.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:37 smithi090 ceph-mon[30399]: 2.d scrub starts 2024-04-14T15:18:37.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:37 smithi090 ceph-mon[30399]: 2.d scrub ok 2024-04-14T15:18:37.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:37 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/101459450' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:18:37.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:37 smithi090 ceph-mon[30399]: 4.18 scrub starts 2024-04-14T15:18:37.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:37 smithi090 ceph-mon[30399]: 4.18 scrub ok 2024-04-14T15:18:37.252 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:18:37.275 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:37.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:37.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:18:37.292 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:37 smithi092 ceph-mon[33966]: 11.1e scrub starts 2024-04-14T15:18:37.292 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:37 smithi092 ceph-mon[33966]: 11.1e scrub ok 2024-04-14T15:18:37.292 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:37 smithi092 ceph-mon[33966]: 1.a scrub starts 2024-04-14T15:18:37.292 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:37 smithi092 ceph-mon[33966]: 1.a scrub ok 2024-04-14T15:18:37.293 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:37 smithi092 ceph-mon[33966]: 110.15 scrub starts 2024-04-14T15:18:37.293 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:37 smithi092 ceph-mon[33966]: 110.15 scrub ok 2024-04-14T15:18:37.293 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:37 smithi092 ceph-mon[33966]: 2.d scrub starts 2024-04-14T15:18:37.293 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:37 smithi092 ceph-mon[33966]: 2.d scrub ok 2024-04-14T15:18:37.293 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:37 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/101459450' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:18:37.293 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:37 smithi092 ceph-mon[33966]: 4.18 scrub starts 2024-04-14T15:18:37.293 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:37 smithi092 ceph-mon[33966]: 4.18 scrub ok 2024-04-14T15:18:37.377 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:18:37.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:37.402 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:37.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:18:37.503 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:18:37.526 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:37.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:37.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:18:37.628 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:18:37.639 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-04-14T15:18:37.639 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-04-14T15:18:37.639 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 0 2024-04-14T15:18:37.640 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-04-14T15:18:37.652 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:37.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:37.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:18:37.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:37.683+0000 7f85f8d61700 1 -- 172.21.15.90:0/2342284572 >> v1:172.21.15.90:6789/0 conn(0x7f85f4111560 legacy=0x7f85f4113a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:37.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:37.683+0000 7f85f8d61700 1 -- 172.21.15.90:0/2342284572 shutdown_connections 2024-04-14T15:18:37.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:37.683+0000 7f85f8d61700 1 -- 172.21.15.90:0/2342284572 >> 172.21.15.90:0/2342284572 conn(0x7f85f4076ab0 msgr2=0x7f85f4076ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:18:37.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:37.683+0000 7f85f8d61700 1 -- 172.21.15.90:0/2342284572 shutdown_connections 2024-04-14T15:18:37.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:37.683+0000 7f85f8d61700 1 -- 172.21.15.90:0/2342284572 wait complete. 2024-04-14T15:18:37.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:37.683+0000 7f85f8d61700 1 librados: starting msgr at 2024-04-14T15:18:37.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:37.683+0000 7f85f8d61700 1 librados: starting objecter 2024-04-14T15:18:37.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:37.684+0000 7f85f8d61700 1 Processor -- start 2024-04-14T15:18:37.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:37.684+0000 7f85f8d61700 1 -- start start 2024-04-14T15:18:37.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:37.684+0000 7f85f8d61700 1 librados: setting wanted keys 2024-04-14T15:18:37.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:37.684+0000 7f85f8d61700 1 librados: calling monclient init 2024-04-14T15:18:37.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:37.684+0000 7f85f8d61700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f85f41a9bc0 con 0x7f85f4103680 2024-04-14T15:18:37.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:37.684+0000 7f85f8d61700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f85f41aade0 con 0x7f85f410d960 2024-04-14T15:18:37.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:37.684+0000 7f85f8d61700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f85f41ac020 con 0x7f85f4111560 2024-04-14T15:18:37.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:37.684+0000 7f85f2d9d700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f85f4111560 0x7f85f41a8280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:42812/0 (socket says 172.21.15.90:42812) 2024-04-14T15:18:37.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:37.684+0000 7f85f259c700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f85f4103680 0x7f85f4101960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:40412/0 (socket says 172.21.15.90:40412) 2024-04-14T15:18:37.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:37.684+0000 7f85f2d9d700 1 -- 172.21.15.90:0/2405269730 learned_addr learned my addr 172.21.15.90:0/2405269730 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:18:37.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:37.684+0000 7f85e37fe700 10 client.?.objecter ms_handle_connect 0x7f85f4103680 2024-04-14T15:18:37.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:37.684+0000 7f85e37fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:18:37.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:37.684+0000 7f85e37fe700 10 client.?.objecter ms_handle_connect 0x7f85f4111560 2024-04-14T15:18:37.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:37.684+0000 7f85e37fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:18:37.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:37.684+0000 7f85e37fe700 10 client.?.objecter ms_handle_connect 0x7f85f410d960 2024-04-14T15:18:37.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:37.684+0000 7f85e37fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:18:37.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:37.684+0000 7f85e37fe700 1 -- 172.21.15.90:0/2405269730 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 134638814 0 0) 0x7f85f41a9bc0 con 0x7f85f4103680 2024-04-14T15:18:37.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:37.684+0000 7f85e37fe700 1 -- 172.21.15.90:0/2405269730 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f85dc0031f0 con 0x7f85f4103680 2024-04-14T15:18:37.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:37.684+0000 7f85e37fe700 1 -- 172.21.15.90:0/2405269730 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3647480821 0 0) 0x7f85f41aade0 con 0x7f85f410d960 2024-04-14T15:18:37.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:37.684+0000 7f85e37fe700 1 -- 172.21.15.90:0/2405269730 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f85f41a9bc0 con 0x7f85f410d960 2024-04-14T15:18:37.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:37.684+0000 7f85e37fe700 1 -- 172.21.15.90:0/2405269730 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3919769137 0 0) 0x7f85f41ac020 con 0x7f85f4111560 2024-04-14T15:18:37.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:37.684+0000 7f85e37fe700 1 -- 172.21.15.90:0/2405269730 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f85f41aade0 con 0x7f85f4111560 2024-04-14T15:18:37.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:37.685+0000 7f85e37fe700 1 -- 172.21.15.90:0/2405269730 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3116160416 0 0) 0x7f85f41a9bc0 con 0x7f85f410d960 2024-04-14T15:18:37.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:37.685+0000 7f85e37fe700 1 -- 172.21.15.90:0/2405269730 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f85f41ac020 con 0x7f85f410d960 2024-04-14T15:18:37.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:37.685+0000 7f85e37fe700 1 -- 172.21.15.90:0/2405269730 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3903315989 0 0) 0x7f85f41aade0 con 0x7f85f4111560 2024-04-14T15:18:37.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:37.685+0000 7f85e37fe700 1 -- 172.21.15.90:0/2405269730 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f85f41a9bc0 con 0x7f85f4111560 2024-04-14T15:18:37.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:37.685+0000 7f85e37fe700 1 -- 172.21.15.90:0/2405269730 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3936143953 0 0) 0x7f85dc0031f0 con 0x7f85f4103680 2024-04-14T15:18:37.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:37.685+0000 7f85e37fe700 1 -- 172.21.15.90:0/2405269730 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f85f41aade0 con 0x7f85f4103680 2024-04-14T15:18:37.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:37.685+0000 7f85e37fe700 1 -- 172.21.15.90:0/2405269730 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f85e8001870 con 0x7f85f410d960 2024-04-14T15:18:37.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:37.685+0000 7f85e37fe700 1 -- 172.21.15.90:0/2405269730 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f85ec004780 con 0x7f85f4111560 2024-04-14T15:18:37.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:37.685+0000 7f85e37fe700 1 -- 172.21.15.90:0/2405269730 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f85e4001830 con 0x7f85f4103680 2024-04-14T15:18:37.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:37.685+0000 7f85e37fe700 1 -- 172.21.15.90:0/2405269730 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1399000925 0 0) 0x7f85f41ac020 con 0x7f85f410d960 2024-04-14T15:18:37.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:37.685+0000 7f85e37fe700 1 -- 172.21.15.90:0/2405269730 >> v1:172.21.15.90:6790/0 conn(0x7f85f4111560 legacy=0x7f85f41a8280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:37.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:37.685+0000 7f85e37fe700 1 -- 172.21.15.90:0/2405269730 >> v1:172.21.15.90:6789/0 conn(0x7f85f4103680 legacy=0x7f85f4101960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:37.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:37.685+0000 7f85e37fe700 1 -- 172.21.15.90:0/2405269730 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85f41ad260 con 0x7f85f410d960 2024-04-14T15:18:37.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:37.685+0000 7f85f8d61700 1 -- 172.21.15.90:0/2405269730 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f85f41ac250 con 0x7f85f410d960 2024-04-14T15:18:37.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:37.685+0000 7f85f8d61700 10 client.49150.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:18:37.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:37.685+0000 7f85f8d61700 1 -- 172.21.15.90:0/2405269730 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f85f41ac710 con 0x7f85f410d960 2024-04-14T15:18:37.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:37.685+0000 7f85f8d61700 1 librados: init done 2024-04-14T15:18:37.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:37.685+0000 7f85e37fe700 1 -- 172.21.15.90:0/2405269730 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f85e8003b20 con 0x7f85f410d960 2024-04-14T15:18:37.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:37.685+0000 7f85e37fe700 1 -- 172.21.15.90:0/2405269730 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f85e8004aa0 con 0x7f85f410d960 2024-04-14T15:18:37.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:37.686+0000 7f85f8d61700 1 -- 172.21.15.90:0/2405269730 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85d4005120 con 0x7f85f410d960 2024-04-14T15:18:37.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:37.687+0000 7f85e37fe700 1 -- 172.21.15.90:0/2405269730 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f85e801b780 con 0x7f85f410d960 2024-04-14T15:18:37.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:37.687+0000 7f85e37fe700 1 -- 172.21.15.90:0/2405269730 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(429..429 src has 51..429) ==== 8188+0+0 (unknown 250556405 0 0) 0x7f85e8089840 con 0x7f85f410d960 2024-04-14T15:18:37.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:37.687+0000 7f85e37fe700 10 client.49150.objecter ms_dispatch 0x7f85f4001550 osd_map(429..429 src has 51..429) 2024-04-14T15:18:37.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:37.687+0000 7f85e37fe700 3 client.49150.objecter handle_osd_map got epochs [429,429] > 0 2024-04-14T15:18:37.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:37.687+0000 7f85e37fe700 3 client.49150.objecter handle_osd_map decoding full epoch 429 2024-04-14T15:18:37.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:37.687+0000 7f85e37fe700 20 client.49150.objecter dump_active .. 0 homeless 2024-04-14T15:18:37.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:37.687+0000 7f85e37fe700 10 client.49150.objecter ms_handle_connect 0x7f85dc0713d0 2024-04-14T15:18:37.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:37.692+0000 7f85e37fe700 1 -- 172.21.15.90:0/2405269730 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f85e8056240 con 0x7f85f410d960 2024-04-14T15:18:37.754 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:18:37.778 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:37.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:37.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:18:37.879 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:37.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:37.895+0000 7f85f8d61700 1 -- 172.21.15.90:0/2405269730 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} v 0) -- 0x7f85d4005410 con 0x7f85f410d960 2024-04-14T15:18:37.903 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:37.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:37.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:38.004 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:18:38.027 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:38.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:38.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:18:38.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:38.050+0000 7f85e37fe700 1 -- 172.21.15.90:0/2405269730 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]=0 set pool 1 pg_num to 21 v430) ==== 129+0+0 (unknown 3854156453 0 0) 0x7f85e8059ef0 con 0x7f85f410d960 2024-04-14T15:18:38.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:38.053+0000 7f85f8d61700 10 librados: watch_flush enter 2024-04-14T15:18:38.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:38.053+0000 7f85f8d61700 10 librados: watch_flush exit 2024-04-14T15:18:38.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:38.053+0000 7f85f8d61700 20 client.49150.objecter shutdown clearing up homeless session... 2024-04-14T15:18:38.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:38.053+0000 7f85f8d61700 10 client.49150.objecter successfully canceled tick 2024-04-14T15:18:38.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:38.053+0000 7f85f8d61700 1 -- 172.21.15.90:0/2405269730 >> v1:172.21.15.90:6800/420935427 conn(0x7f85dc0713d0 legacy=0x7f85dc073890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:38.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:38.053+0000 7f85f8d61700 1 -- 172.21.15.90:0/2405269730 >> v1:172.21.15.92:6789/0 conn(0x7f85f410d960 legacy=0x7f85f41a4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:38.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:38.053+0000 7f85f8d61700 1 -- 172.21.15.90:0/2405269730 shutdown_connections 2024-04-14T15:18:38.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:38.053+0000 7f85f8d61700 1 -- 172.21.15.90:0/2405269730 >> 172.21.15.90:0/2405269730 conn(0x7f85f4076ab0 msgr2=0x7f85f4114d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:18:38.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:38.054+0000 7f85f8d61700 1 -- 172.21.15.90:0/2405269730 shutdown_connections 2024-04-14T15:18:38.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:38.054+0000 7f85f8d61700 1 -- 172.21.15.90:0/2405269730 wait complete. 2024-04-14T15:18:38.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:38.054+0000 7f85f8d61700 1 librados: shutdown 2024-04-14T15:18:38.055 INFO:teuthology.orchestra.run.smithi090.stderr:set pool 1 pg_num to 21 2024-04-14T15:18:38.130 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:38.153 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:38.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:38.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:38.249 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:38 smithi092 ceph-mon[33966]: 4.14 scrub starts 2024-04-14T15:18:38.249 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:38 smithi092 ceph-mon[33966]: 4.14 scrub ok 2024-04-14T15:18:38.249 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:38 smithi092 ceph-mon[33966]: 2.c scrub starts 2024-04-14T15:18:38.249 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:38 smithi092 ceph-mon[33966]: 2.c scrub ok 2024-04-14T15:18:38.249 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:38 smithi092 ceph-mon[33966]: pgmap v600: 238 pgs: 238 active+clean; 8.4 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-14T15:18:38.249 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:38 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2405269730' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-04-14T15:18:38.249 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:38 smithi092 ceph-mon[33966]: from='client.49150 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-04-14T15:18:38.256 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:18:38.276 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-14T15:18:38.280 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:38.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:38.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:18:38.382 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:18:38.399 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:38 smithi090 ceph-mon[41670]: 4.14 scrub starts 2024-04-14T15:18:38.399 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:38 smithi090 ceph-mon[41670]: 4.14 scrub ok 2024-04-14T15:18:38.400 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:38 smithi090 ceph-mon[41670]: 2.c scrub starts 2024-04-14T15:18:38.400 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:38 smithi090 ceph-mon[41670]: 2.c scrub ok 2024-04-14T15:18:38.400 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:38 smithi090 ceph-mon[41670]: pgmap v600: 238 pgs: 238 active+clean; 8.4 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-14T15:18:38.400 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:38 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2405269730' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-04-14T15:18:38.400 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:38 smithi090 ceph-mon[41670]: from='client.49150 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-04-14T15:18:38.400 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:38 smithi090 ceph-mon[30399]: 4.14 scrub starts 2024-04-14T15:18:38.400 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:38 smithi090 ceph-mon[30399]: 4.14 scrub ok 2024-04-14T15:18:38.400 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:38 smithi090 ceph-mon[30399]: 2.c scrub starts 2024-04-14T15:18:38.400 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:38 smithi090 ceph-mon[30399]: 2.c scrub ok 2024-04-14T15:18:38.400 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:38 smithi090 ceph-mon[30399]: pgmap v600: 238 pgs: 238 active+clean; 8.4 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-14T15:18:38.400 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:38 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2405269730' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-04-14T15:18:38.400 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:38 smithi090 ceph-mon[30399]: from='client.49150 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-04-14T15:18:38.407 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:38.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:38.408 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:18:38.509 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:18:38.517 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:18:38.533 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:38.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:38.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:18:38.635 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:18:38.659 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:38.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:38.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:18:38.761 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:18:38.787 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:38.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:38.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:18:38.889 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:18:38.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:38.915 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:38.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:18:39.016 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:39.040 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:39.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:39.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:39.142 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:39.165 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:39.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:39.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:39.222 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-14T15:18:39.267 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:18:39.290 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:39.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:39.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:18:39.365 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:39 smithi092 ceph-mon[33966]: 11.4 scrub starts 2024-04-14T15:18:39.365 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:39 smithi092 ceph-mon[33966]: 11.4 scrub ok 2024-04-14T15:18:39.365 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:39 smithi092 ceph-mon[33966]: 110.17 scrub starts 2024-04-14T15:18:39.365 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:39 smithi092 ceph-mon[33966]: 110.17 scrub ok 2024-04-14T15:18:39.365 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:39 smithi092 ceph-mon[33966]: from='client.49150 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-04-14T15:18:39.365 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:39 smithi092 ceph-mon[33966]: osdmap e430: 8 total, 8 up, 8 in 2024-04-14T15:18:39.365 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:39 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:18:39.365 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:39 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:18:39.381 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:39 smithi090 ceph-mon[41670]: 11.4 scrub starts 2024-04-14T15:18:39.381 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:39 smithi090 ceph-mon[41670]: 11.4 scrub ok 2024-04-14T15:18:39.381 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:39 smithi090 ceph-mon[41670]: 110.17 scrub starts 2024-04-14T15:18:39.381 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:39 smithi090 ceph-mon[41670]: 110.17 scrub ok 2024-04-14T15:18:39.381 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:39 smithi090 ceph-mon[41670]: from='client.49150 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-04-14T15:18:39.381 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:39 smithi090 ceph-mon[41670]: osdmap e430: 8 total, 8 up, 8 in 2024-04-14T15:18:39.381 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:39 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:18:39.381 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:39 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:18:39.382 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:39 smithi090 ceph-mon[30399]: 11.4 scrub starts 2024-04-14T15:18:39.382 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:39 smithi090 ceph-mon[30399]: 11.4 scrub ok 2024-04-14T15:18:39.382 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:39 smithi090 ceph-mon[30399]: 110.17 scrub starts 2024-04-14T15:18:39.382 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:39 smithi090 ceph-mon[30399]: 110.17 scrub ok 2024-04-14T15:18:39.382 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:39 smithi090 ceph-mon[30399]: from='client.49150 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-04-14T15:18:39.382 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:39 smithi090 ceph-mon[30399]: osdmap e430: 8 total, 8 up, 8 in 2024-04-14T15:18:39.382 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:39 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:18:39.382 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:39 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:18:39.393 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:18:39.418 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:39.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:39.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:18:39.519 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:18:39.543 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:39.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:39.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:18:39.645 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:18:39.646 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:18:39.670 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:39.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:39.671 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:18:39.772 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:18:39.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:39.796 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:39.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:18:39.897 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:18:39.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:39.876+0000 7fd38fd76700 1 -- 172.21.15.90:0/3556261004 >> v1:172.21.15.90:6790/0 conn(0x7fd38810b930 legacy=0x7fd38810dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:39.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:39.876+0000 7fd38fd76700 1 -- 172.21.15.90:0/3556261004 shutdown_connections 2024-04-14T15:18:39.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:39.876+0000 7fd38fd76700 1 -- 172.21.15.90:0/3556261004 >> 172.21.15.90:0/3556261004 conn(0x7fd3880fd290 msgr2=0x7fd3880ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:18:39.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:39.876+0000 7fd38fd76700 1 -- 172.21.15.90:0/3556261004 shutdown_connections 2024-04-14T15:18:39.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:39.876+0000 7fd38fd76700 1 -- 172.21.15.90:0/3556261004 wait complete. 2024-04-14T15:18:39.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:39.876+0000 7fd38fd76700 1 librados: starting msgr at 2024-04-14T15:18:39.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:39.876+0000 7fd38fd76700 1 librados: starting objecter 2024-04-14T15:18:39.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:39.876+0000 7fd38fd76700 1 Processor -- start 2024-04-14T15:18:39.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:39.877+0000 7fd38fd76700 1 -- start start 2024-04-14T15:18:39.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:39.877+0000 7fd38fd76700 1 librados: setting wanted keys 2024-04-14T15:18:39.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:39.877+0000 7fd38fd76700 1 librados: calling monclient init 2024-04-14T15:18:39.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:39.877+0000 7fd38fd76700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd3881a7a90 con 0x7fd38810b930 2024-04-14T15:18:39.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:39.877+0000 7fd38fd76700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd3881a8cb0 con 0x7fd38810f4d0 2024-04-14T15:18:39.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:39.877+0000 7fd38fd76700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd3881a9ef0 con 0x7fd388103a40 2024-04-14T15:18:39.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:39.877+0000 7fd38db12700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7fd388103a40 0x7fd38810b100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:42828/0 (socket says 172.21.15.90:42828) 2024-04-14T15:18:39.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:39.877+0000 7fd38db12700 1 -- 172.21.15.90:0/622845211 learned_addr learned my addr 172.21.15.90:0/622845211 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:18:39.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:39.877+0000 7fd37effd700 10 client.?.objecter ms_handle_connect 0x7fd38810b930 2024-04-14T15:18:39.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:39.877+0000 7fd37effd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:18:39.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:39.877+0000 7fd37effd700 10 client.?.objecter ms_handle_connect 0x7fd388103a40 2024-04-14T15:18:39.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:39.877+0000 7fd37effd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:18:39.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:39.877+0000 7fd37effd700 10 client.?.objecter ms_handle_connect 0x7fd38810f4d0 2024-04-14T15:18:39.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:39.877+0000 7fd37effd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:18:39.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:39.877+0000 7fd37effd700 1 -- 172.21.15.90:0/622845211 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1533528922 0 0) 0x7fd3881a7a90 con 0x7fd38810b930 2024-04-14T15:18:39.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:39.877+0000 7fd37effd700 1 -- 172.21.15.90:0/622845211 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd3740033a0 con 0x7fd38810b930 2024-04-14T15:18:39.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:39.877+0000 7fd37effd700 1 -- 172.21.15.90:0/622845211 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1618394302 0 0) 0x7fd3881a9ef0 con 0x7fd388103a40 2024-04-14T15:18:39.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:39.877+0000 7fd37effd700 1 -- 172.21.15.90:0/622845211 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd3881a7a90 con 0x7fd388103a40 2024-04-14T15:18:39.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:39.877+0000 7fd37effd700 1 -- 172.21.15.90:0/622845211 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4181630233 0 0) 0x7fd3881a8cb0 con 0x7fd38810f4d0 2024-04-14T15:18:39.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:39.877+0000 7fd37effd700 1 -- 172.21.15.90:0/622845211 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd3881a9ef0 con 0x7fd38810f4d0 2024-04-14T15:18:39.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:39.877+0000 7fd37effd700 1 -- 172.21.15.90:0/622845211 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4249206140 0 0) 0x7fd3881a9ef0 con 0x7fd38810f4d0 2024-04-14T15:18:39.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:39.877+0000 7fd37effd700 1 -- 172.21.15.90:0/622845211 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd3881a8cb0 con 0x7fd38810f4d0 2024-04-14T15:18:39.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:39.877+0000 7fd37effd700 1 -- 172.21.15.90:0/622845211 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3896321099 0 0) 0x7fd3740033a0 con 0x7fd38810b930 2024-04-14T15:18:39.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:39.877+0000 7fd37effd700 1 -- 172.21.15.90:0/622845211 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd3881a9ef0 con 0x7fd38810b930 2024-04-14T15:18:39.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:39.877+0000 7fd37effd700 1 -- 172.21.15.90:0/622845211 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fd3800018b0 con 0x7fd38810f4d0 2024-04-14T15:18:39.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:39.877+0000 7fd37effd700 1 -- 172.21.15.90:0/622845211 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3144417691 0 0) 0x7fd3881a7a90 con 0x7fd388103a40 2024-04-14T15:18:39.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:39.877+0000 7fd37effd700 1 -- 172.21.15.90:0/622845211 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd3740033a0 con 0x7fd388103a40 2024-04-14T15:18:39.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:39.877+0000 7fd37effd700 1 -- 172.21.15.90:0/622845211 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fd3780014f0 con 0x7fd38810b930 2024-04-14T15:18:39.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:39.877+0000 7fd37effd700 1 -- 172.21.15.90:0/622845211 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fd384001c10 con 0x7fd388103a40 2024-04-14T15:18:39.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:39.877+0000 7fd37effd700 1 -- 172.21.15.90:0/622845211 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3430556829 0 0) 0x7fd3881a8cb0 con 0x7fd38810f4d0 2024-04-14T15:18:39.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:39.878+0000 7fd37effd700 1 -- 172.21.15.90:0/622845211 >> v1:172.21.15.90:6790/0 conn(0x7fd388103a40 legacy=0x7fd38810b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:39.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:39.878+0000 7fd37effd700 1 -- 172.21.15.90:0/622845211 >> v1:172.21.15.90:6789/0 conn(0x7fd38810b930 legacy=0x7fd3881a2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:39.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:39.878+0000 7fd37effd700 1 -- 172.21.15.90:0/622845211 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3881ab130 con 0x7fd38810f4d0 2024-04-14T15:18:39.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:39.878+0000 7fd38fd76700 1 -- 172.21.15.90:0/622845211 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd3881aa120 con 0x7fd38810f4d0 2024-04-14T15:18:39.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:39.878+0000 7fd38fd76700 10 client.49156.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:18:39.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:39.878+0000 7fd38fd76700 1 -- 172.21.15.90:0/622845211 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd3881aa5e0 con 0x7fd38810f4d0 2024-04-14T15:18:39.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:39.878+0000 7fd38fd76700 1 librados: init done 2024-04-14T15:18:39.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:39.878+0000 7fd37effd700 1 -- 172.21.15.90:0/622845211 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fd380003b40 con 0x7fd38810f4d0 2024-04-14T15:18:39.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:39.878+0000 7fd37effd700 1 -- 172.21.15.90:0/622845211 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fd380004a70 con 0x7fd38810f4d0 2024-04-14T15:18:39.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:39.878+0000 7fd38fd76700 1 -- 172.21.15.90:0/622845211 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd36c005120 con 0x7fd38810f4d0 2024-04-14T15:18:39.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:39.879+0000 7fd37effd700 1 -- 172.21.15.90:0/622845211 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fd380003680 con 0x7fd38810f4d0 2024-04-14T15:18:39.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:39.880+0000 7fd37effd700 1 -- 172.21.15.90:0/622845211 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(430..430 src has 51..430) ==== 8216+0+0 (unknown 979577208 0 0) 0x7fd3800896b0 con 0x7fd38810f4d0 2024-04-14T15:18:39.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:39.880+0000 7fd37effd700 10 client.49156.objecter ms_dispatch 0x7fd388001550 osd_map(430..430 src has 51..430) 2024-04-14T15:18:39.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:39.880+0000 7fd37effd700 3 client.49156.objecter handle_osd_map got epochs [430,430] > 0 2024-04-14T15:18:39.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:39.880+0000 7fd37effd700 3 client.49156.objecter handle_osd_map decoding full epoch 430 2024-04-14T15:18:39.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:39.880+0000 7fd37effd700 20 client.49156.objecter dump_active .. 0 homeless 2024-04-14T15:18:39.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:39.880+0000 7fd37effd700 10 client.49156.objecter ms_handle_connect 0x7fd374071450 2024-04-14T15:18:39.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:39.884+0000 7fd37effd700 1 -- 172.21.15.90:0/622845211 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fd380056090 con 0x7fd38810f4d0 2024-04-14T15:18:39.920 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:39.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:39.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:18:40.022 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:18:40.046 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:40.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:40.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:18:40.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:40.111+0000 7fd38fd76700 1 -- 172.21.15.90:0/622845211 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7fd36c005410 con 0x7fd38810f4d0 2024-04-14T15:18:40.147 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:40.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:40.171 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:40.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:40.172 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:40 smithi090 ceph-mon[30399]: 11.9 scrub starts 2024-04-14T15:18:40.173 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:40 smithi090 ceph-mon[30399]: 11.9 scrub ok 2024-04-14T15:18:40.173 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:40 smithi090 ceph-mon[30399]: 110.1 scrub starts 2024-04-14T15:18:40.173 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:40 smithi090 ceph-mon[30399]: 110.1 scrub ok 2024-04-14T15:18:40.173 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:40 smithi090 ceph-mon[30399]: 4.2 scrub starts 2024-04-14T15:18:40.173 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:40 smithi090 ceph-mon[30399]: 4.2 scrub ok 2024-04-14T15:18:40.173 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:40 smithi090 ceph-mon[30399]: pgmap v602: 238 pgs: 238 active+clean; 8.4 MiB data, 585 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-14T15:18:40.173 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:40 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-04-14T15:18:40.173 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:40 smithi090 ceph-mon[30399]: 5.1 deep-scrub starts 2024-04-14T15:18:40.173 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:40 smithi090 ceph-mon[30399]: 5.1 deep-scrub ok 2024-04-14T15:18:40.173 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:40 smithi090 ceph-mon[41670]: 11.9 scrub starts 2024-04-14T15:18:40.173 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:40 smithi090 ceph-mon[41670]: 11.9 scrub ok 2024-04-14T15:18:40.174 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:40 smithi090 ceph-mon[41670]: 110.1 scrub starts 2024-04-14T15:18:40.174 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:40 smithi090 ceph-mon[41670]: 110.1 scrub ok 2024-04-14T15:18:40.174 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:40 smithi090 ceph-mon[41670]: 4.2 scrub starts 2024-04-14T15:18:40.174 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:40 smithi090 ceph-mon[41670]: 4.2 scrub ok 2024-04-14T15:18:40.174 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:40 smithi090 ceph-mon[41670]: pgmap v602: 238 pgs: 238 active+clean; 8.4 MiB data, 585 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-14T15:18:40.174 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:40 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-04-14T15:18:40.174 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:40 smithi090 ceph-mon[41670]: 5.1 deep-scrub starts 2024-04-14T15:18:40.174 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:40 smithi090 ceph-mon[41670]: 5.1 deep-scrub ok 2024-04-14T15:18:40.246 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-04-14T15:18:40.246 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-04-14T15:18:40.246 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-04-14T15:18:40.246 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-04-14T15:18:40.246 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-04-14T15:18:40.246 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:40 smithi092 ceph-mon[33966]: 11.9 scrub starts 2024-04-14T15:18:40.246 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:40 smithi092 ceph-mon[33966]: 11.9 scrub ok 2024-04-14T15:18:40.247 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:40 smithi092 ceph-mon[33966]: 110.1 scrub starts 2024-04-14T15:18:40.247 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:40 smithi092 ceph-mon[33966]: 110.1 scrub ok 2024-04-14T15:18:40.247 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:40 smithi092 ceph-mon[33966]: 4.2 scrub starts 2024-04-14T15:18:40.247 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:40 smithi092 ceph-mon[33966]: 4.2 scrub ok 2024-04-14T15:18:40.247 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:40 smithi092 ceph-mon[33966]: pgmap v602: 238 pgs: 238 active+clean; 8.4 MiB data, 585 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-14T15:18:40.247 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:40 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-04-14T15:18:40.247 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:40 smithi092 ceph-mon[33966]: 5.1 deep-scrub starts 2024-04-14T15:18:40.247 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:40 smithi092 ceph-mon[33966]: 5.1 deep-scrub ok 2024-04-14T15:18:40.272 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:18:40.296 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:40.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:40.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:18:40.398 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:18:40.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:40.422 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:40.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:18:40.522 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:18:40.546 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:40.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:40.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:18:40.647 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:18:40.671 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:40.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:40.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:18:40.772 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:18:40.796 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:40.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:40.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:18:40.885 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-14T15:18:40.898 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:18:40.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:40.901+0000 7fd37effd700 1 -- 172.21.15.90:0/622845211 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v432) ==== 87+0+0 (unknown 2591196689 0 0) 0x7fd380059d40 con 0x7fd38810f4d0 2024-04-14T15:18:40.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:40.904+0000 7fd38fd76700 10 librados: watch_flush enter 2024-04-14T15:18:40.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:40.904+0000 7fd38fd76700 10 librados: watch_flush exit 2024-04-14T15:18:40.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:40.904+0000 7fd38fd76700 20 client.49156.objecter shutdown clearing up homeless session... 2024-04-14T15:18:40.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:40.904+0000 7fd38fd76700 10 client.49156.objecter successfully canceled tick 2024-04-14T15:18:40.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:40.904+0000 7fd38fd76700 1 -- 172.21.15.90:0/622845211 >> v1:172.21.15.90:6800/420935427 conn(0x7fd374071450 legacy=0x7fd374073910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:40.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:40.904+0000 7fd38fd76700 1 -- 172.21.15.90:0/622845211 >> v1:172.21.15.92:6789/0 conn(0x7fd38810f4d0 legacy=0x7fd3881a6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:40.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:40.906+0000 7fd38fd76700 1 -- 172.21.15.90:0/622845211 shutdown_connections 2024-04-14T15:18:40.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:40.906+0000 7fd38fd76700 1 -- 172.21.15.90:0/622845211 >> 172.21.15.90:0/622845211 conn(0x7fd3880fd290 msgr2=0x7fd3880ff680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:18:40.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:40.906+0000 7fd38fd76700 1 -- 172.21.15.90:0/622845211 shutdown_connections 2024-04-14T15:18:40.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:40.906+0000 7fd38fd76700 1 -- 172.21.15.90:0/622845211 wait complete. 2024-04-14T15:18:40.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:40.906+0000 7fd38fd76700 1 librados: shutdown 2024-04-14T15:18:40.911 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is set 2024-04-14T15:18:40.952 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:40.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:40.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:18:41.054 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:18:41.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:41.079 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:41.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:18:41.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:41 smithi092 ceph-mon[33966]: 4.8 deep-scrub starts 2024-04-14T15:18:41.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:41 smithi092 ceph-mon[33966]: 4.8 deep-scrub ok 2024-04-14T15:18:41.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:41 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-04-14T15:18:41.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:41 smithi092 ceph-mon[33966]: osdmap e431: 8 total, 8 up, 8 in 2024-04-14T15:18:41.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:41 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/622845211' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:18:41.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:41 smithi092 ceph-mon[33966]: from='client.49156 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:18:41.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:41 smithi092 ceph-mon[33966]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:18:41.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:41 smithi092 ceph-mon[33966]: from='client.49156 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-14T15:18:41.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:41 smithi092 ceph-mon[33966]: osdmap e432: 8 total, 8 up, 8 in 2024-04-14T15:18:41.138 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:18:41.179 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:18:41.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:41.204 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:41.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:18:41.305 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:18:41.328 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:41.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:41.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:18:41.430 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:41.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:41 smithi090 ceph-mon[41670]: 4.8 deep-scrub starts 2024-04-14T15:18:41.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:41 smithi090 ceph-mon[41670]: 4.8 deep-scrub ok 2024-04-14T15:18:41.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:41 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-04-14T15:18:41.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:41 smithi090 ceph-mon[41670]: osdmap e431: 8 total, 8 up, 8 in 2024-04-14T15:18:41.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:41 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/622845211' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:18:41.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:41 smithi090 ceph-mon[41670]: from='client.49156 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:18:41.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:41 smithi090 ceph-mon[41670]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:18:41.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:41 smithi090 ceph-mon[41670]: from='client.49156 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-14T15:18:41.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:41 smithi090 ceph-mon[41670]: osdmap e432: 8 total, 8 up, 8 in 2024-04-14T15:18:41.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:41 smithi090 ceph-mon[30399]: 4.8 deep-scrub starts 2024-04-14T15:18:41.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:41 smithi090 ceph-mon[30399]: 4.8 deep-scrub ok 2024-04-14T15:18:41.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:41 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-04-14T15:18:41.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:41 smithi090 ceph-mon[30399]: osdmap e431: 8 total, 8 up, 8 in 2024-04-14T15:18:41.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:41 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/622845211' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:18:41.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:41 smithi090 ceph-mon[30399]: from='client.49156 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:18:41.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:41 smithi090 ceph-mon[30399]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:18:41.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:41 smithi090 ceph-mon[30399]: from='client.49156 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-14T15:18:41.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:41 smithi090 ceph-mon[30399]: osdmap e432: 8 total, 8 up, 8 in 2024-04-14T15:18:41.454 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:41.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:41.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:41.556 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:18:41.579 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:41.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:41.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:18:41.681 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:41.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:41.706 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:41.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:41.807 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:18:41.831 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:41.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:41.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:18:41.933 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:18:41.957 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:41.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:41.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:18:42.059 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:18:42.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:42.090 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:42.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:18:42.191 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:18:42.214 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:42.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:42.215 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:18:42.232 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:42 smithi090 ceph-mon[30399]: 5.17 scrub starts 2024-04-14T15:18:42.233 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:42 smithi090 ceph-mon[30399]: 5.17 scrub ok 2024-04-14T15:18:42.233 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:42 smithi090 ceph-mon[30399]: 11.1d scrub starts 2024-04-14T15:18:42.233 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:42 smithi090 ceph-mon[30399]: 11.1d scrub ok 2024-04-14T15:18:42.233 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:42 smithi090 ceph-mon[30399]: pgmap v605: 248 pgs: 10 unknown, 238 active+clean; 8.4 MiB data, 585 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:18:42.233 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:42 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-04-14T15:18:42.233 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:42 smithi090 ceph-mon[41670]: 5.17 scrub starts 2024-04-14T15:18:42.233 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:42 smithi090 ceph-mon[41670]: 5.17 scrub ok 2024-04-14T15:18:42.233 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:42 smithi090 ceph-mon[41670]: 11.1d scrub starts 2024-04-14T15:18:42.233 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:42 smithi090 ceph-mon[41670]: 11.1d scrub ok 2024-04-14T15:18:42.233 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:42 smithi090 ceph-mon[41670]: pgmap v605: 248 pgs: 10 unknown, 238 active+clean; 8.4 MiB data, 585 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:18:42.233 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:42 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-04-14T15:18:42.316 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:18:42.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:42.342 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:42.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:18:42.371 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:42 smithi092 ceph-mon[33966]: 5.17 scrub starts 2024-04-14T15:18:42.371 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:42 smithi092 ceph-mon[33966]: 5.17 scrub ok 2024-04-14T15:18:42.371 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:42 smithi092 ceph-mon[33966]: 11.1d scrub starts 2024-04-14T15:18:42.371 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:42 smithi092 ceph-mon[33966]: 11.1d scrub ok 2024-04-14T15:18:42.371 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:42 smithi092 ceph-mon[33966]: pgmap v605: 248 pgs: 10 unknown, 238 active+clean; 8.4 MiB data, 585 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:18:42.371 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:42 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-04-14T15:18:42.443 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:18:42.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.447+0000 7f2e59f61700 1 -- 172.21.15.90:0/745824963 >> v1:172.21.15.90:6789/0 conn(0x7f2e5410a790 legacy=0x7f2e5410cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:42.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.447+0000 7f2e59f61700 1 -- 172.21.15.90:0/745824963 shutdown_connections 2024-04-14T15:18:42.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.447+0000 7f2e59f61700 1 -- 172.21.15.90:0/745824963 >> 172.21.15.90:0/745824963 conn(0x7f2e540fd2c0 msgr2=0x7f2e540ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:18:42.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.448+0000 7f2e59f61700 1 -- 172.21.15.90:0/745824963 shutdown_connections 2024-04-14T15:18:42.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.448+0000 7f2e59f61700 1 -- 172.21.15.90:0/745824963 wait complete. 2024-04-14T15:18:42.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.448+0000 7f2e59f61700 1 librados: starting msgr at 2024-04-14T15:18:42.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.448+0000 7f2e59f61700 1 librados: starting objecter 2024-04-14T15:18:42.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.448+0000 7f2e59f61700 1 Processor -- start 2024-04-14T15:18:42.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.448+0000 7f2e59f61700 1 -- start start 2024-04-14T15:18:42.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.448+0000 7f2e59f61700 1 librados: setting wanted keys 2024-04-14T15:18:42.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.448+0000 7f2e59f61700 1 librados: calling monclient init 2024-04-14T15:18:42.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.449+0000 7f2e59f61700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f2e541a7f70 con 0x7f2e5410a790 2024-04-14T15:18:42.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.449+0000 7f2e59f61700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f2e541a9150 con 0x7f2e541078e0 2024-04-14T15:18:42.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.449+0000 7f2e59f61700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f2e541aa330 con 0x7f2e5410e360 2024-04-14T15:18:42.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.449+0000 7f2e53fff700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f2e5410e360 0x7f2e541a6850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:50146/0 (socket says 172.21.15.90:50146) 2024-04-14T15:18:42.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.449+0000 7f2e53fff700 1 -- 172.21.15.90:0/2287089093 learned_addr learned my addr 172.21.15.90:0/2287089093 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:18:42.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.449+0000 7f2e537fe700 1 --1- 172.21.15.90:0/2287089093 >> v1:172.21.15.92:6789/0 conn(0x7f2e541078e0 0x7f2e5410c040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.90:57016/0 (socket says 172.21.15.90:57016) 2024-04-14T15:18:42.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.449+0000 7f2e52ffd700 1 --1- 172.21.15.90:0/2287089093 >> v1:172.21.15.90:6789/0 conn(0x7f2e5410a790 0x7f2e5410c750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:40832/0 (socket says 172.21.15.90:40832) 2024-04-14T15:18:42.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.449+0000 7f2e50ff9700 10 client.?.objecter ms_handle_connect 0x7f2e5410a790 2024-04-14T15:18:42.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.449+0000 7f2e50ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:18:42.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.449+0000 7f2e50ff9700 10 client.?.objecter ms_handle_connect 0x7f2e541078e0 2024-04-14T15:18:42.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.449+0000 7f2e50ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:18:42.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.449+0000 7f2e50ff9700 10 client.?.objecter ms_handle_connect 0x7f2e5410e360 2024-04-14T15:18:42.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.449+0000 7f2e50ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:18:42.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.449+0000 7f2e50ff9700 1 -- 172.21.15.90:0/2287089093 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2440148381 0 0) 0x7f2e541a7f70 con 0x7f2e5410a790 2024-04-14T15:18:42.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.449+0000 7f2e50ff9700 1 -- 172.21.15.90:0/2287089093 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2e3c0035b0 con 0x7f2e5410a790 2024-04-14T15:18:42.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.449+0000 7f2e50ff9700 1 -- 172.21.15.90:0/2287089093 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2354651162 0 0) 0x7f2e541a9150 con 0x7f2e541078e0 2024-04-14T15:18:42.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.449+0000 7f2e50ff9700 1 -- 172.21.15.90:0/2287089093 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2e541a7f70 con 0x7f2e541078e0 2024-04-14T15:18:42.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.449+0000 7f2e50ff9700 1 -- 172.21.15.90:0/2287089093 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2694561763 0 0) 0x7f2e541aa330 con 0x7f2e5410e360 2024-04-14T15:18:42.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.449+0000 7f2e50ff9700 1 -- 172.21.15.90:0/2287089093 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2e541a9150 con 0x7f2e5410e360 2024-04-14T15:18:42.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.449+0000 7f2e50ff9700 1 -- 172.21.15.90:0/2287089093 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 284573362 0 0) 0x7f2e3c0035b0 con 0x7f2e5410a790 2024-04-14T15:18:42.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.449+0000 7f2e50ff9700 1 -- 172.21.15.90:0/2287089093 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2e541aa330 con 0x7f2e5410a790 2024-04-14T15:18:42.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.449+0000 7f2e50ff9700 1 -- 172.21.15.90:0/2287089093 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f2e480015d0 con 0x7f2e5410a790 2024-04-14T15:18:42.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.449+0000 7f2e50ff9700 1 -- 172.21.15.90:0/2287089093 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1928433685 0 0) 0x7f2e541a7f70 con 0x7f2e541078e0 2024-04-14T15:18:42.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.449+0000 7f2e50ff9700 1 -- 172.21.15.90:0/2287089093 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2e3c0035b0 con 0x7f2e541078e0 2024-04-14T15:18:42.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.449+0000 7f2e50ff9700 1 -- 172.21.15.90:0/2287089093 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 666407702 0 0) 0x7f2e541a9150 con 0x7f2e5410e360 2024-04-14T15:18:42.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.449+0000 7f2e50ff9700 1 -- 172.21.15.90:0/2287089093 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2e541a7f70 con 0x7f2e5410e360 2024-04-14T15:18:42.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.449+0000 7f2e50ff9700 1 -- 172.21.15.90:0/2287089093 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f2e44001720 con 0x7f2e541078e0 2024-04-14T15:18:42.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.449+0000 7f2e50ff9700 1 -- 172.21.15.90:0/2287089093 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f2e4c001bb0 con 0x7f2e5410e360 2024-04-14T15:18:42.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.450+0000 7f2e50ff9700 1 -- 172.21.15.90:0/2287089093 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3326149366 0 0) 0x7f2e541aa330 con 0x7f2e5410a790 2024-04-14T15:18:42.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.450+0000 7f2e50ff9700 1 -- 172.21.15.90:0/2287089093 >> v1:172.21.15.90:6790/0 conn(0x7f2e5410e360 legacy=0x7f2e541a6850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:42.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.450+0000 7f2e50ff9700 1 -- 172.21.15.90:0/2287089093 >> v1:172.21.15.92:6789/0 conn(0x7f2e541078e0 legacy=0x7f2e5410c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:42.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.450+0000 7f2e50ff9700 1 -- 172.21.15.90:0/2287089093 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e541ab510 con 0x7f2e5410a790 2024-04-14T15:18:42.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.450+0000 7f2e59f61700 1 -- 172.21.15.90:0/2287089093 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2e541a9380 con 0x7f2e5410a790 2024-04-14T15:18:42.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.450+0000 7f2e59f61700 10 client.37323.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:18:42.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.450+0000 7f2e59f61700 1 -- 172.21.15.90:0/2287089093 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f2e541a9840 con 0x7f2e5410a790 2024-04-14T15:18:42.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.450+0000 7f2e59f61700 1 librados: init done 2024-04-14T15:18:42.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.450+0000 7f2e50ff9700 1 -- 172.21.15.90:0/2287089093 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f2e480011a0 con 0x7f2e5410a790 2024-04-14T15:18:42.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.450+0000 7f2e50ff9700 1 -- 172.21.15.90:0/2287089093 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f2e48004e00 con 0x7f2e5410a790 2024-04-14T15:18:42.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.451+0000 7f2e59f61700 1 -- 172.21.15.90:0/2287089093 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f2e40005100 con 0x7f2e5410a790 2024-04-14T15:18:42.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.451+0000 7f2e50ff9700 1 -- 172.21.15.90:0/2287089093 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f2e48001b70 con 0x7f2e5410a790 2024-04-14T15:18:42.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.452+0000 7f2e50ff9700 1 -- 172.21.15.90:0/2287089093 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(433..433 src has 51..433) ==== 8188+0+0 (unknown 3327900327 0 0) 0x7f2e4808a940 con 0x7f2e5410a790 2024-04-14T15:18:42.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.452+0000 7f2e50ff9700 10 client.37323.objecter ms_dispatch 0x7f2e54001550 osd_map(433..433 src has 51..433) 2024-04-14T15:18:42.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.452+0000 7f2e50ff9700 3 client.37323.objecter handle_osd_map got epochs [433,433] > 0 2024-04-14T15:18:42.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.452+0000 7f2e50ff9700 3 client.37323.objecter handle_osd_map decoding full epoch 433 2024-04-14T15:18:42.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.452+0000 7f2e50ff9700 20 client.37323.objecter dump_active .. 0 homeless 2024-04-14T15:18:42.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.452+0000 7f2e50ff9700 1 -- 172.21.15.90:0/2287089093 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v433) ==== 54+0+16 (unknown 672804329 0 2420000192) 0x7f2e4808b060 con 0x7f2e5410a790 2024-04-14T15:18:42.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.452+0000 7f2e50ff9700 10 client.37323.objecter ms_handle_connect 0x7f2e3c0713f0 2024-04-14T15:18:42.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.452+0000 7f2e59f61700 10 client.37323.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:18:42.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.452+0000 7f2e59f61700 15 client.37323.objecter _session_command_op_assign -1 1 2024-04-14T15:18:42.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.452+0000 7f2e59f61700 20 client.37323.objecter _get_session s=0x7f2e400053e0 osd=0 3 2024-04-14T15:18:42.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.452+0000 7f2e59f61700 20 client.37323.objecter put_session s=0x7f2e400053e0 osd=0 3 2024-04-14T15:18:42.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.452+0000 7f2e59f61700 20 client.37323.objecter _get_session s=0x7f2e400053e0 osd=0 3 2024-04-14T15:18:42.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.452+0000 7f2e59f61700 15 client.37323.objecter _session_command_op_remove -1 1 2024-04-14T15:18:42.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.452+0000 7f2e59f61700 20 client.37323.objecter get_session s=0x7f2e400053e0 osd=0 3 2024-04-14T15:18:42.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.452+0000 7f2e59f61700 15 client.37323.objecter _session_command_op_assign 0 1 2024-04-14T15:18:42.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.452+0000 7f2e59f61700 20 client.37323.objecter put_session s=0x7f2e400053e0 osd=0 4 2024-04-14T15:18:42.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.452+0000 7f2e59f61700 10 client.37323.objecter _send_command 1 2024-04-14T15:18:42.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.452+0000 7f2e59f61700 1 -- 172.21.15.90:0/2287089093 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2e40004b50 con 0x7f2e40004730 2024-04-14T15:18:42.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.452+0000 7f2e50ff9700 10 client.37323.objecter ms_handle_connect 0x7f2e40004730 2024-04-14T15:18:42.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.453+0000 7f2e50ff9700 1 -- 172.21.15.90:0/2287089093 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2e40004b50 con 0x7f2e40004730 2024-04-14T15:18:42.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.453+0000 7f2e50ff9700 10 client.37323.objecter ms_dispatch 0x7f2e54001550 command_reply(tid 1: 0 ) 2024-04-14T15:18:42.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.453+0000 7f2e50ff9700 10 client.37323.objecter _finish_command 1 = system:0 2024-04-14T15:18:42.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.453+0000 7f2e50ff9700 20 client.37323.objecter put_session s=0x7f2e400053e0 osd=0 4 2024-04-14T15:18:42.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.453+0000 7f2e50ff9700 15 client.37323.objecter _session_command_op_remove 0 1 2024-04-14T15:18:42.466 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:42.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:42.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:18:42.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.470+0000 7f2e59f61700 10 client.37323.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:18:42.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.470+0000 7f2e59f61700 15 client.37323.objecter _session_command_op_assign -1 2 2024-04-14T15:18:42.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.470+0000 7f2e59f61700 20 client.37323.objecter _get_session s=0x7f2e400053e0 osd=0 3 2024-04-14T15:18:42.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.470+0000 7f2e59f61700 20 client.37323.objecter put_session s=0x7f2e400053e0 osd=0 3 2024-04-14T15:18:42.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.470+0000 7f2e59f61700 20 client.37323.objecter _get_session s=0x7f2e400053e0 osd=0 3 2024-04-14T15:18:42.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.470+0000 7f2e59f61700 15 client.37323.objecter _session_command_op_remove -1 2 2024-04-14T15:18:42.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.470+0000 7f2e59f61700 20 client.37323.objecter get_session s=0x7f2e400053e0 osd=0 3 2024-04-14T15:18:42.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.470+0000 7f2e59f61700 15 client.37323.objecter _session_command_op_assign 0 2 2024-04-14T15:18:42.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.470+0000 7f2e59f61700 20 client.37323.objecter put_session s=0x7f2e400053e0 osd=0 4 2024-04-14T15:18:42.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.470+0000 7f2e59f61700 10 client.37323.objecter _send_command 2 2024-04-14T15:18:42.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.470+0000 7f2e59f61700 1 -- 172.21.15.90:0/2287089093 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f2e400079f0 con 0x7f2e40004730 2024-04-14T15:18:42.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.471+0000 7f2e50ff9700 1 -- 172.21.15.90:0/2287089093 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2e400079f0 con 0x7f2e40004730 2024-04-14T15:18:42.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.471+0000 7f2e50ff9700 10 client.37323.objecter ms_dispatch 0x7f2e54001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:18:42.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.471+0000 7f2e50ff9700 10 client.37323.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:18:42.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.471+0000 7f2e50ff9700 20 client.37323.objecter put_session s=0x7f2e400053e0 osd=0 4 2024-04-14T15:18:42.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.471+0000 7f2e50ff9700 15 client.37323.objecter _session_command_op_remove 0 2 2024-04-14T15:18:42.472 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:18:42.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.472+0000 7f2e59f61700 10 client.37323.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:18:42.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.472+0000 7f2e59f61700 15 client.37323.objecter _session_command_op_assign -1 3 2024-04-14T15:18:42.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.472+0000 7f2e59f61700 20 client.37323.objecter _get_session s=0x7f2e40004110 osd=1 3 2024-04-14T15:18:42.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.472+0000 7f2e59f61700 20 client.37323.objecter put_session s=0x7f2e40004110 osd=1 3 2024-04-14T15:18:42.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.472+0000 7f2e59f61700 20 client.37323.objecter _get_session s=0x7f2e40004110 osd=1 3 2024-04-14T15:18:42.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.472+0000 7f2e59f61700 15 client.37323.objecter _session_command_op_remove -1 3 2024-04-14T15:18:42.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.472+0000 7f2e59f61700 20 client.37323.objecter get_session s=0x7f2e40004110 osd=1 3 2024-04-14T15:18:42.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.472+0000 7f2e59f61700 15 client.37323.objecter _session_command_op_assign 1 3 2024-04-14T15:18:42.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.472+0000 7f2e59f61700 20 client.37323.objecter put_session s=0x7f2e40004110 osd=1 4 2024-04-14T15:18:42.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.472+0000 7f2e59f61700 10 client.37323.objecter _send_command 3 2024-04-14T15:18:42.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.472+0000 7f2e59f61700 1 -- 172.21.15.90:0/2287089093 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f2e40002180 con 0x7f2e40004280 2024-04-14T15:18:42.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.472+0000 7f2e50ff9700 10 client.37323.objecter ms_handle_connect 0x7f2e40004280 2024-04-14T15:18:42.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.473+0000 7f2e50ff9700 1 -- 172.21.15.90:0/2287089093 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2e40002180 con 0x7f2e40004280 2024-04-14T15:18:42.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.473+0000 7f2e50ff9700 10 client.37323.objecter ms_dispatch 0x7f2e54001550 command_reply(tid 3: 0 ) 2024-04-14T15:18:42.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.473+0000 7f2e50ff9700 10 client.37323.objecter _finish_command 3 = system:0 2024-04-14T15:18:42.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.473+0000 7f2e50ff9700 20 client.37323.objecter put_session s=0x7f2e40004110 osd=1 4 2024-04-14T15:18:42.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.473+0000 7f2e50ff9700 15 client.37323.objecter _session_command_op_remove 1 3 2024-04-14T15:18:42.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.489+0000 7f2e59f61700 10 client.37323.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:18:42.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.489+0000 7f2e59f61700 15 client.37323.objecter _session_command_op_assign -1 4 2024-04-14T15:18:42.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.489+0000 7f2e59f61700 20 client.37323.objecter _get_session s=0x7f2e40004110 osd=1 3 2024-04-14T15:18:42.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.489+0000 7f2e59f61700 20 client.37323.objecter put_session s=0x7f2e40004110 osd=1 3 2024-04-14T15:18:42.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.489+0000 7f2e59f61700 20 client.37323.objecter _get_session s=0x7f2e40004110 osd=1 3 2024-04-14T15:18:42.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.489+0000 7f2e59f61700 15 client.37323.objecter _session_command_op_remove -1 4 2024-04-14T15:18:42.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.489+0000 7f2e59f61700 20 client.37323.objecter get_session s=0x7f2e40004110 osd=1 3 2024-04-14T15:18:42.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.489+0000 7f2e59f61700 15 client.37323.objecter _session_command_op_assign 1 4 2024-04-14T15:18:42.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.489+0000 7f2e59f61700 20 client.37323.objecter put_session s=0x7f2e40004110 osd=1 4 2024-04-14T15:18:42.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.489+0000 7f2e59f61700 10 client.37323.objecter _send_command 4 2024-04-14T15:18:42.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.489+0000 7f2e59f61700 1 -- 172.21.15.90:0/2287089093 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f2e40001600 con 0x7f2e40004280 2024-04-14T15:18:42.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.491+0000 7f2e50ff9700 1 -- 172.21.15.90:0/2287089093 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2e40001600 con 0x7f2e40004280 2024-04-14T15:18:42.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.491+0000 7f2e50ff9700 10 client.37323.objecter ms_dispatch 0x7f2e54001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:18:42.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.491+0000 7f2e50ff9700 10 client.37323.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:18:42.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.491+0000 7f2e50ff9700 20 client.37323.objecter put_session s=0x7f2e40004110 osd=1 4 2024-04-14T15:18:42.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.491+0000 7f2e50ff9700 15 client.37323.objecter _session_command_op_remove 1 4 2024-04-14T15:18:42.492 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:18:42.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.491+0000 7f2e59f61700 10 client.37323.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:18:42.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.491+0000 7f2e59f61700 15 client.37323.objecter _session_command_op_assign -1 5 2024-04-14T15:18:42.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.491+0000 7f2e59f61700 20 client.37323.objecter _get_session s=0x7f2e40008370 osd=2 3 2024-04-14T15:18:42.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.491+0000 7f2e59f61700 20 client.37323.objecter put_session s=0x7f2e40008370 osd=2 3 2024-04-14T15:18:42.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.491+0000 7f2e59f61700 20 client.37323.objecter _get_session s=0x7f2e40008370 osd=2 3 2024-04-14T15:18:42.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.491+0000 7f2e59f61700 15 client.37323.objecter _session_command_op_remove -1 5 2024-04-14T15:18:42.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.491+0000 7f2e59f61700 20 client.37323.objecter get_session s=0x7f2e40008370 osd=2 3 2024-04-14T15:18:42.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.491+0000 7f2e59f61700 15 client.37323.objecter _session_command_op_assign 2 5 2024-04-14T15:18:42.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.491+0000 7f2e59f61700 20 client.37323.objecter put_session s=0x7f2e40008370 osd=2 4 2024-04-14T15:18:42.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.491+0000 7f2e59f61700 10 client.37323.objecter _send_command 5 2024-04-14T15:18:42.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.491+0000 7f2e59f61700 1 -- 172.21.15.90:0/2287089093 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f2e400110e0 con 0x7f2e4000d5d0 2024-04-14T15:18:42.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.492+0000 7f2e50ff9700 10 client.37323.objecter ms_handle_connect 0x7f2e4000d5d0 2024-04-14T15:18:42.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.492+0000 7f2e50ff9700 1 -- 172.21.15.90:0/2287089093 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2e400110e0 con 0x7f2e4000d5d0 2024-04-14T15:18:42.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.492+0000 7f2e50ff9700 10 client.37323.objecter ms_dispatch 0x7f2e54001550 command_reply(tid 5: 0 ) 2024-04-14T15:18:42.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.492+0000 7f2e50ff9700 10 client.37323.objecter _finish_command 5 = system:0 2024-04-14T15:18:42.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.492+0000 7f2e50ff9700 20 client.37323.objecter put_session s=0x7f2e40008370 osd=2 4 2024-04-14T15:18:42.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.492+0000 7f2e50ff9700 15 client.37323.objecter _session_command_op_remove 2 5 2024-04-14T15:18:42.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.508+0000 7f2e59f61700 10 client.37323.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:18:42.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.508+0000 7f2e59f61700 15 client.37323.objecter _session_command_op_assign -1 6 2024-04-14T15:18:42.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.508+0000 7f2e59f61700 20 client.37323.objecter _get_session s=0x7f2e40008370 osd=2 3 2024-04-14T15:18:42.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.508+0000 7f2e59f61700 20 client.37323.objecter put_session s=0x7f2e40008370 osd=2 3 2024-04-14T15:18:42.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.508+0000 7f2e59f61700 20 client.37323.objecter _get_session s=0x7f2e40008370 osd=2 3 2024-04-14T15:18:42.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.508+0000 7f2e59f61700 15 client.37323.objecter _session_command_op_remove -1 6 2024-04-14T15:18:42.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.508+0000 7f2e59f61700 20 client.37323.objecter get_session s=0x7f2e40008370 osd=2 3 2024-04-14T15:18:42.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.508+0000 7f2e59f61700 15 client.37323.objecter _session_command_op_assign 2 6 2024-04-14T15:18:42.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.508+0000 7f2e59f61700 20 client.37323.objecter put_session s=0x7f2e40008370 osd=2 4 2024-04-14T15:18:42.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.508+0000 7f2e59f61700 10 client.37323.objecter _send_command 6 2024-04-14T15:18:42.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.508+0000 7f2e59f61700 1 -- 172.21.15.90:0/2287089093 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f2e40007eb0 con 0x7f2e4000d5d0 2024-04-14T15:18:42.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.510+0000 7f2e50ff9700 1 -- 172.21.15.90:0/2287089093 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2e40007eb0 con 0x7f2e4000d5d0 2024-04-14T15:18:42.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.510+0000 7f2e50ff9700 10 client.37323.objecter ms_dispatch 0x7f2e54001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:18:42.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.510+0000 7f2e50ff9700 10 client.37323.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:18:42.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.510+0000 7f2e50ff9700 20 client.37323.objecter put_session s=0x7f2e40008370 osd=2 4 2024-04-14T15:18:42.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.510+0000 7f2e50ff9700 15 client.37323.objecter _session_command_op_remove 2 6 2024-04-14T15:18:42.511 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:18:42.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.511+0000 7f2e59f61700 10 client.37323.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:18:42.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.511+0000 7f2e59f61700 15 client.37323.objecter _session_command_op_assign -1 7 2024-04-14T15:18:42.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.511+0000 7f2e59f61700 20 client.37323.objecter _get_session s=0x7f2e4000cc20 osd=3 3 2024-04-14T15:18:42.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.511+0000 7f2e59f61700 20 client.37323.objecter put_session s=0x7f2e4000cc20 osd=3 3 2024-04-14T15:18:42.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.511+0000 7f2e59f61700 20 client.37323.objecter _get_session s=0x7f2e4000cc20 osd=3 3 2024-04-14T15:18:42.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.511+0000 7f2e59f61700 15 client.37323.objecter _session_command_op_remove -1 7 2024-04-14T15:18:42.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.511+0000 7f2e59f61700 20 client.37323.objecter get_session s=0x7f2e4000cc20 osd=3 3 2024-04-14T15:18:42.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.511+0000 7f2e59f61700 15 client.37323.objecter _session_command_op_assign 3 7 2024-04-14T15:18:42.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.511+0000 7f2e59f61700 20 client.37323.objecter put_session s=0x7f2e4000cc20 osd=3 4 2024-04-14T15:18:42.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.511+0000 7f2e59f61700 10 client.37323.objecter _send_command 7 2024-04-14T15:18:42.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.511+0000 7f2e59f61700 1 -- 172.21.15.90:0/2287089093 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f2e40015f20 con 0x7f2e40012410 2024-04-14T15:18:42.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.511+0000 7f2e50ff9700 10 client.37323.objecter ms_handle_connect 0x7f2e40012410 2024-04-14T15:18:42.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.511+0000 7f2e50ff9700 1 -- 172.21.15.90:0/2287089093 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2e40015f20 con 0x7f2e40012410 2024-04-14T15:18:42.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.511+0000 7f2e50ff9700 10 client.37323.objecter ms_dispatch 0x7f2e54001550 command_reply(tid 7: 0 ) 2024-04-14T15:18:42.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.511+0000 7f2e50ff9700 10 client.37323.objecter _finish_command 7 = system:0 2024-04-14T15:18:42.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.511+0000 7f2e50ff9700 20 client.37323.objecter put_session s=0x7f2e4000cc20 osd=3 4 2024-04-14T15:18:42.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.511+0000 7f2e50ff9700 15 client.37323.objecter _session_command_op_remove 3 7 2024-04-14T15:18:42.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.528+0000 7f2e59f61700 10 client.37323.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:18:42.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.528+0000 7f2e59f61700 15 client.37323.objecter _session_command_op_assign -1 8 2024-04-14T15:18:42.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.528+0000 7f2e59f61700 20 client.37323.objecter _get_session s=0x7f2e4000cc20 osd=3 3 2024-04-14T15:18:42.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.528+0000 7f2e59f61700 20 client.37323.objecter put_session s=0x7f2e4000cc20 osd=3 3 2024-04-14T15:18:42.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.528+0000 7f2e59f61700 20 client.37323.objecter _get_session s=0x7f2e4000cc20 osd=3 3 2024-04-14T15:18:42.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.528+0000 7f2e59f61700 15 client.37323.objecter _session_command_op_remove -1 8 2024-04-14T15:18:42.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.528+0000 7f2e59f61700 20 client.37323.objecter get_session s=0x7f2e4000cc20 osd=3 3 2024-04-14T15:18:42.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.528+0000 7f2e59f61700 15 client.37323.objecter _session_command_op_assign 3 8 2024-04-14T15:18:42.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.528+0000 7f2e59f61700 20 client.37323.objecter put_session s=0x7f2e4000cc20 osd=3 4 2024-04-14T15:18:42.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.528+0000 7f2e59f61700 10 client.37323.objecter _send_command 8 2024-04-14T15:18:42.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.528+0000 7f2e59f61700 1 -- 172.21.15.90:0/2287089093 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f2e4000c700 con 0x7f2e40012410 2024-04-14T15:18:42.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.530+0000 7f2e50ff9700 1 -- 172.21.15.90:0/2287089093 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2e4000c700 con 0x7f2e40012410 2024-04-14T15:18:42.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.530+0000 7f2e50ff9700 10 client.37323.objecter ms_dispatch 0x7f2e54001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:18:42.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.530+0000 7f2e50ff9700 10 client.37323.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:18:42.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.530+0000 7f2e50ff9700 20 client.37323.objecter put_session s=0x7f2e4000cc20 osd=3 4 2024-04-14T15:18:42.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.530+0000 7f2e50ff9700 15 client.37323.objecter _session_command_op_remove 3 8 2024-04-14T15:18:42.531 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:18:42.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.530+0000 7f2e59f61700 10 client.37323.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:18:42.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.530+0000 7f2e59f61700 15 client.37323.objecter _session_command_op_assign -1 9 2024-04-14T15:18:42.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.530+0000 7f2e59f61700 20 client.37323.objecter _get_session s=0x7f2e4000c700 osd=4 3 2024-04-14T15:18:42.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.530+0000 7f2e59f61700 20 client.37323.objecter put_session s=0x7f2e4000c700 osd=4 3 2024-04-14T15:18:42.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.530+0000 7f2e59f61700 20 client.37323.objecter _get_session s=0x7f2e4000c700 osd=4 3 2024-04-14T15:18:42.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.530+0000 7f2e59f61700 15 client.37323.objecter _session_command_op_remove -1 9 2024-04-14T15:18:42.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.530+0000 7f2e59f61700 20 client.37323.objecter get_session s=0x7f2e4000c700 osd=4 3 2024-04-14T15:18:42.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.530+0000 7f2e59f61700 15 client.37323.objecter _session_command_op_assign 4 9 2024-04-14T15:18:42.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.530+0000 7f2e59f61700 20 client.37323.objecter put_session s=0x7f2e4000c700 osd=4 4 2024-04-14T15:18:42.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.530+0000 7f2e59f61700 10 client.37323.objecter _send_command 9 2024-04-14T15:18:42.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.530+0000 7f2e59f61700 1 -- 172.21.15.90:0/2287089093 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f2e4001a9f0 con 0x7f2e40011f70 2024-04-14T15:18:42.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.531+0000 7f2e50ff9700 10 client.37323.objecter ms_handle_connect 0x7f2e40011f70 2024-04-14T15:18:42.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.531+0000 7f2e50ff9700 1 -- 172.21.15.90:0/2287089093 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2e4001a9f0 con 0x7f2e40011f70 2024-04-14T15:18:42.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.531+0000 7f2e50ff9700 10 client.37323.objecter ms_dispatch 0x7f2e54001550 command_reply(tid 9: 0 ) 2024-04-14T15:18:42.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.531+0000 7f2e50ff9700 10 client.37323.objecter _finish_command 9 = system:0 2024-04-14T15:18:42.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.531+0000 7f2e50ff9700 20 client.37323.objecter put_session s=0x7f2e4000c700 osd=4 4 2024-04-14T15:18:42.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.531+0000 7f2e50ff9700 15 client.37323.objecter _session_command_op_remove 4 9 2024-04-14T15:18:42.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.548+0000 7f2e59f61700 10 client.37323.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:18:42.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.548+0000 7f2e59f61700 15 client.37323.objecter _session_command_op_assign -1 10 2024-04-14T15:18:42.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.548+0000 7f2e59f61700 20 client.37323.objecter _get_session s=0x7f2e4000c700 osd=4 3 2024-04-14T15:18:42.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.548+0000 7f2e59f61700 20 client.37323.objecter put_session s=0x7f2e4000c700 osd=4 3 2024-04-14T15:18:42.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.548+0000 7f2e59f61700 20 client.37323.objecter _get_session s=0x7f2e4000c700 osd=4 3 2024-04-14T15:18:42.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.548+0000 7f2e59f61700 15 client.37323.objecter _session_command_op_remove -1 10 2024-04-14T15:18:42.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.548+0000 7f2e59f61700 20 client.37323.objecter get_session s=0x7f2e4000c700 osd=4 3 2024-04-14T15:18:42.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.548+0000 7f2e59f61700 15 client.37323.objecter _session_command_op_assign 4 10 2024-04-14T15:18:42.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.548+0000 7f2e59f61700 20 client.37323.objecter put_session s=0x7f2e4000c700 osd=4 4 2024-04-14T15:18:42.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.548+0000 7f2e59f61700 10 client.37323.objecter _send_command 10 2024-04-14T15:18:42.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.548+0000 7f2e59f61700 1 -- 172.21.15.90:0/2287089093 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f2e4001c3a0 con 0x7f2e40011f70 2024-04-14T15:18:42.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.549+0000 7f2e50ff9700 1 -- 172.21.15.90:0/2287089093 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2e4001c3a0 con 0x7f2e40011f70 2024-04-14T15:18:42.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.549+0000 7f2e50ff9700 10 client.37323.objecter ms_dispatch 0x7f2e54001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:18:42.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.549+0000 7f2e50ff9700 10 client.37323.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:18:42.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.549+0000 7f2e50ff9700 20 client.37323.objecter put_session s=0x7f2e4000c700 osd=4 4 2024-04-14T15:18:42.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.549+0000 7f2e50ff9700 15 client.37323.objecter _session_command_op_remove 4 10 2024-04-14T15:18:42.550 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:18:42.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.550+0000 7f2e59f61700 10 client.37323.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:18:42.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.550+0000 7f2e59f61700 15 client.37323.objecter _session_command_op_assign -1 11 2024-04-14T15:18:42.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.550+0000 7f2e59f61700 20 client.37323.objecter _get_session s=0x7f2e4001c3a0 osd=5 3 2024-04-14T15:18:42.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.550+0000 7f2e59f61700 20 client.37323.objecter put_session s=0x7f2e4001c3a0 osd=5 3 2024-04-14T15:18:42.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.550+0000 7f2e59f61700 20 client.37323.objecter _get_session s=0x7f2e4001c3a0 osd=5 3 2024-04-14T15:18:42.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.550+0000 7f2e59f61700 15 client.37323.objecter _session_command_op_remove -1 11 2024-04-14T15:18:42.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.550+0000 7f2e59f61700 20 client.37323.objecter get_session s=0x7f2e4001c3a0 osd=5 3 2024-04-14T15:18:42.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.550+0000 7f2e59f61700 15 client.37323.objecter _session_command_op_assign 5 11 2024-04-14T15:18:42.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.550+0000 7f2e59f61700 20 client.37323.objecter put_session s=0x7f2e4001c3a0 osd=5 4 2024-04-14T15:18:42.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.550+0000 7f2e59f61700 10 client.37323.objecter _send_command 11 2024-04-14T15:18:42.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.550+0000 7f2e59f61700 1 -- 172.21.15.90:0/2287089093 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f2e4001cf70 con 0x7f2e4001cac0 2024-04-14T15:18:42.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.550+0000 7f2e50ff9700 10 client.37323.objecter ms_handle_connect 0x7f2e4001cac0 2024-04-14T15:18:42.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.551+0000 7f2e50ff9700 1 -- 172.21.15.90:0/2287089093 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2e4001cf70 con 0x7f2e4001cac0 2024-04-14T15:18:42.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.551+0000 7f2e50ff9700 10 client.37323.objecter ms_dispatch 0x7f2e54001550 command_reply(tid 11: 0 ) 2024-04-14T15:18:42.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.551+0000 7f2e50ff9700 10 client.37323.objecter _finish_command 11 = system:0 2024-04-14T15:18:42.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.551+0000 7f2e50ff9700 20 client.37323.objecter put_session s=0x7f2e4001c3a0 osd=5 4 2024-04-14T15:18:42.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.551+0000 7f2e50ff9700 15 client.37323.objecter _session_command_op_remove 5 11 2024-04-14T15:18:42.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.567+0000 7f2e59f61700 10 client.37323.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:18:42.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.567+0000 7f2e59f61700 15 client.37323.objecter _session_command_op_assign -1 12 2024-04-14T15:18:42.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.567+0000 7f2e59f61700 20 client.37323.objecter _get_session s=0x7f2e4001c3a0 osd=5 3 2024-04-14T15:18:42.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.567+0000 7f2e59f61700 20 client.37323.objecter put_session s=0x7f2e4001c3a0 osd=5 3 2024-04-14T15:18:42.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.567+0000 7f2e59f61700 20 client.37323.objecter _get_session s=0x7f2e4001c3a0 osd=5 3 2024-04-14T15:18:42.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.567+0000 7f2e59f61700 15 client.37323.objecter _session_command_op_remove -1 12 2024-04-14T15:18:42.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.567+0000 7f2e59f61700 20 client.37323.objecter get_session s=0x7f2e4001c3a0 osd=5 3 2024-04-14T15:18:42.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.567+0000 7f2e59f61700 15 client.37323.objecter _session_command_op_assign 5 12 2024-04-14T15:18:42.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.567+0000 7f2e59f61700 20 client.37323.objecter put_session s=0x7f2e4001c3a0 osd=5 4 2024-04-14T15:18:42.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.567+0000 7f2e59f61700 10 client.37323.objecter _send_command 12 2024-04-14T15:18:42.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.567+0000 7f2e59f61700 1 -- 172.21.15.90:0/2287089093 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f2e4001d1a0 con 0x7f2e4001cac0 2024-04-14T15:18:42.568 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:18:42.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.569+0000 7f2e50ff9700 1 -- 172.21.15.90:0/2287089093 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2e4001d1a0 con 0x7f2e4001cac0 2024-04-14T15:18:42.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.569+0000 7f2e50ff9700 10 client.37323.objecter ms_dispatch 0x7f2e54001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:18:42.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.569+0000 7f2e50ff9700 10 client.37323.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:18:42.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.569+0000 7f2e50ff9700 20 client.37323.objecter put_session s=0x7f2e4001c3a0 osd=5 4 2024-04-14T15:18:42.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.569+0000 7f2e50ff9700 15 client.37323.objecter _session_command_op_remove 5 12 2024-04-14T15:18:42.570 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:18:42.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.569+0000 7f2e59f61700 10 client.37323.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:18:42.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.569+0000 7f2e59f61700 15 client.37323.objecter _session_command_op_assign -1 13 2024-04-14T15:18:42.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.569+0000 7f2e59f61700 20 client.37323.objecter _get_session s=0x7f2e4001d1a0 osd=6 3 2024-04-14T15:18:42.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.569+0000 7f2e59f61700 20 client.37323.objecter put_session s=0x7f2e4001d1a0 osd=6 3 2024-04-14T15:18:42.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.569+0000 7f2e59f61700 20 client.37323.objecter _get_session s=0x7f2e4001d1a0 osd=6 3 2024-04-14T15:18:42.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.569+0000 7f2e59f61700 15 client.37323.objecter _session_command_op_remove -1 13 2024-04-14T15:18:42.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.569+0000 7f2e59f61700 20 client.37323.objecter get_session s=0x7f2e4001d1a0 osd=6 3 2024-04-14T15:18:42.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.569+0000 7f2e59f61700 15 client.37323.objecter _session_command_op_assign 6 13 2024-04-14T15:18:42.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.569+0000 7f2e59f61700 20 client.37323.objecter put_session s=0x7f2e4001d1a0 osd=6 4 2024-04-14T15:18:42.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.569+0000 7f2e59f61700 10 client.37323.objecter _send_command 13 2024-04-14T15:18:42.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.569+0000 7f2e59f61700 1 -- 172.21.15.90:0/2287089093 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f2e400235c0 con 0x7f2e4001fab0 2024-04-14T15:18:42.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.569+0000 7f2e50ff9700 10 client.37323.objecter ms_handle_connect 0x7f2e4001fab0 2024-04-14T15:18:42.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.570+0000 7f2e50ff9700 1 -- 172.21.15.90:0/2287089093 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2e400235c0 con 0x7f2e4001fab0 2024-04-14T15:18:42.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.570+0000 7f2e50ff9700 10 client.37323.objecter ms_dispatch 0x7f2e54001550 command_reply(tid 13: 0 ) 2024-04-14T15:18:42.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.570+0000 7f2e50ff9700 10 client.37323.objecter _finish_command 13 = system:0 2024-04-14T15:18:42.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.570+0000 7f2e50ff9700 20 client.37323.objecter put_session s=0x7f2e4001d1a0 osd=6 4 2024-04-14T15:18:42.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.570+0000 7f2e50ff9700 15 client.37323.objecter _session_command_op_remove 6 13 2024-04-14T15:18:42.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.586+0000 7f2e59f61700 10 client.37323.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:18:42.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.586+0000 7f2e59f61700 15 client.37323.objecter _session_command_op_assign -1 14 2024-04-14T15:18:42.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.586+0000 7f2e59f61700 20 client.37323.objecter _get_session s=0x7f2e4001d1a0 osd=6 3 2024-04-14T15:18:42.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.586+0000 7f2e59f61700 20 client.37323.objecter put_session s=0x7f2e4001d1a0 osd=6 3 2024-04-14T15:18:42.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.586+0000 7f2e59f61700 20 client.37323.objecter _get_session s=0x7f2e4001d1a0 osd=6 3 2024-04-14T15:18:42.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.586+0000 7f2e59f61700 15 client.37323.objecter _session_command_op_remove -1 14 2024-04-14T15:18:42.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.586+0000 7f2e59f61700 20 client.37323.objecter get_session s=0x7f2e4001d1a0 osd=6 3 2024-04-14T15:18:42.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.586+0000 7f2e59f61700 15 client.37323.objecter _session_command_op_assign 6 14 2024-04-14T15:18:42.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.586+0000 7f2e59f61700 20 client.37323.objecter put_session s=0x7f2e4001d1a0 osd=6 4 2024-04-14T15:18:42.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.586+0000 7f2e59f61700 10 client.37323.objecter _send_command 14 2024-04-14T15:18:42.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.586+0000 7f2e59f61700 1 -- 172.21.15.90:0/2287089093 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f2e400235c0 con 0x7f2e4001fab0 2024-04-14T15:18:42.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.588+0000 7f2e50ff9700 1 -- 172.21.15.90:0/2287089093 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2e400235c0 con 0x7f2e4001fab0 2024-04-14T15:18:42.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.588+0000 7f2e50ff9700 10 client.37323.objecter ms_dispatch 0x7f2e54001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:18:42.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.588+0000 7f2e50ff9700 10 client.37323.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:18:42.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.588+0000 7f2e50ff9700 20 client.37323.objecter put_session s=0x7f2e4001d1a0 osd=6 4 2024-04-14T15:18:42.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.588+0000 7f2e50ff9700 15 client.37323.objecter _session_command_op_remove 6 14 2024-04-14T15:18:42.589 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:18:42.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.588+0000 7f2e59f61700 10 client.37323.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:18:42.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.588+0000 7f2e59f61700 15 client.37323.objecter _session_command_op_assign -1 15 2024-04-14T15:18:42.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.588+0000 7f2e59f61700 20 client.37323.objecter _get_session s=0x7f2e400235c0 osd=7 3 2024-04-14T15:18:42.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.588+0000 7f2e59f61700 20 client.37323.objecter put_session s=0x7f2e400235c0 osd=7 3 2024-04-14T15:18:42.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.588+0000 7f2e59f61700 20 client.37323.objecter _get_session s=0x7f2e400235c0 osd=7 3 2024-04-14T15:18:42.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.588+0000 7f2e59f61700 15 client.37323.objecter _session_command_op_remove -1 15 2024-04-14T15:18:42.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.588+0000 7f2e59f61700 20 client.37323.objecter get_session s=0x7f2e400235c0 osd=7 3 2024-04-14T15:18:42.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.588+0000 7f2e59f61700 15 client.37323.objecter _session_command_op_assign 7 15 2024-04-14T15:18:42.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.588+0000 7f2e59f61700 20 client.37323.objecter put_session s=0x7f2e400235c0 osd=7 4 2024-04-14T15:18:42.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.588+0000 7f2e59f61700 10 client.37323.objecter _send_command 15 2024-04-14T15:18:42.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.588+0000 7f2e59f61700 1 -- 172.21.15.90:0/2287089093 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f2e400278c0 con 0x7f2e40023c40 2024-04-14T15:18:42.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.589+0000 7f2e50ff9700 10 client.37323.objecter ms_handle_connect 0x7f2e40023c40 2024-04-14T15:18:42.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.590+0000 7f2e50ff9700 1 -- 172.21.15.90:0/2287089093 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2e400278c0 con 0x7f2e40023c40 2024-04-14T15:18:42.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.590+0000 7f2e50ff9700 10 client.37323.objecter ms_dispatch 0x7f2e54001550 command_reply(tid 15: 0 ) 2024-04-14T15:18:42.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.590+0000 7f2e50ff9700 10 client.37323.objecter _finish_command 15 = system:0 2024-04-14T15:18:42.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.590+0000 7f2e50ff9700 20 client.37323.objecter put_session s=0x7f2e400235c0 osd=7 4 2024-04-14T15:18:42.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.590+0000 7f2e50ff9700 15 client.37323.objecter _session_command_op_remove 7 15 2024-04-14T15:18:42.594 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:42.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:42.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:18:42.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.606+0000 7f2e59f61700 10 client.37323.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:18:42.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.606+0000 7f2e59f61700 15 client.37323.objecter _session_command_op_assign -1 16 2024-04-14T15:18:42.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.606+0000 7f2e59f61700 20 client.37323.objecter _get_session s=0x7f2e400235c0 osd=7 3 2024-04-14T15:18:42.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.606+0000 7f2e59f61700 20 client.37323.objecter put_session s=0x7f2e400235c0 osd=7 3 2024-04-14T15:18:42.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.606+0000 7f2e59f61700 20 client.37323.objecter _get_session s=0x7f2e400235c0 osd=7 3 2024-04-14T15:18:42.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.606+0000 7f2e59f61700 15 client.37323.objecter _session_command_op_remove -1 16 2024-04-14T15:18:42.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.606+0000 7f2e59f61700 20 client.37323.objecter get_session s=0x7f2e400235c0 osd=7 3 2024-04-14T15:18:42.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.606+0000 7f2e59f61700 15 client.37323.objecter _session_command_op_assign 7 16 2024-04-14T15:18:42.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.606+0000 7f2e59f61700 20 client.37323.objecter put_session s=0x7f2e400235c0 osd=7 4 2024-04-14T15:18:42.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.606+0000 7f2e59f61700 10 client.37323.objecter _send_command 16 2024-04-14T15:18:42.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.606+0000 7f2e59f61700 1 -- 172.21.15.90:0/2287089093 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f2e40029270 con 0x7f2e40023c40 2024-04-14T15:18:42.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.608+0000 7f2e50ff9700 1 -- 172.21.15.90:0/2287089093 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2e40029270 con 0x7f2e40023c40 2024-04-14T15:18:42.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.608+0000 7f2e50ff9700 10 client.37323.objecter ms_dispatch 0x7f2e54001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:18:42.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.608+0000 7f2e50ff9700 10 client.37323.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:18:42.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.608+0000 7f2e50ff9700 20 client.37323.objecter put_session s=0x7f2e400235c0 osd=7 4 2024-04-14T15:18:42.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.608+0000 7f2e50ff9700 15 client.37323.objecter _session_command_op_remove 7 16 2024-04-14T15:18:42.609 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:18:42.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.608+0000 7f2e59f61700 10 librados: watch_flush enter 2024-04-14T15:18:42.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.608+0000 7f2e59f61700 10 librados: watch_flush exit 2024-04-14T15:18:42.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.608+0000 7f2e59f61700 10 client.37323.objecter close_session for osd.0 2024-04-14T15:18:42.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.608+0000 7f2e59f61700 1 -- 172.21.15.90:0/2287089093 >> v1:172.21.15.90:6801/3761806885 conn(0x7f2e40004730 legacy=0x7f2e40002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:42.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.608+0000 7f2e59f61700 20 client.37323.objecter put_session s=0x7f2e400053e0 osd=0 1 2024-04-14T15:18:42.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.608+0000 7f2e59f61700 10 client.37323.objecter close_session for osd.1 2024-04-14T15:18:42.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.608+0000 7f2e59f61700 1 -- 172.21.15.90:0/2287089093 >> v1:172.21.15.90:6805/4170361699 conn(0x7f2e40004280 legacy=0x7f2e4000ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:42.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.608+0000 7f2e59f61700 20 client.37323.objecter put_session s=0x7f2e40004110 osd=1 1 2024-04-14T15:18:42.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.608+0000 7f2e59f61700 10 client.37323.objecter close_session for osd.2 2024-04-14T15:18:42.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.608+0000 7f2e59f61700 1 -- 172.21.15.90:0/2287089093 >> v1:172.21.15.90:6809/1686184251 conn(0x7f2e4000d5d0 legacy=0x7f2e4000f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:42.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.608+0000 7f2e59f61700 20 client.37323.objecter put_session s=0x7f2e40008370 osd=2 1 2024-04-14T15:18:42.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.608+0000 7f2e59f61700 10 client.37323.objecter close_session for osd.3 2024-04-14T15:18:42.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.608+0000 7f2e59f61700 1 -- 172.21.15.90:0/2287089093 >> v1:172.21.15.90:6813/2270166774 conn(0x7f2e40012410 legacy=0x7f2e40014800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:42.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.608+0000 7f2e59f61700 20 client.37323.objecter put_session s=0x7f2e4000cc20 osd=3 1 2024-04-14T15:18:42.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.608+0000 7f2e59f61700 10 client.37323.objecter close_session for osd.4 2024-04-14T15:18:42.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.608+0000 7f2e59f61700 1 -- 172.21.15.90:0/2287089093 >> v1:172.21.15.92:6800/3887549733 conn(0x7f2e40011f70 legacy=0x7f2e40019260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:42.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.608+0000 7f2e59f61700 20 client.37323.objecter put_session s=0x7f2e4000c700 osd=4 1 2024-04-14T15:18:42.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.608+0000 7f2e59f61700 10 client.37323.objecter close_session for osd.5 2024-04-14T15:18:42.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.608+0000 7f2e59f61700 1 -- 172.21.15.90:0/2287089093 >> v1:172.21.15.92:6804/3244561440 conn(0x7f2e4001cac0 legacy=0x7f2e4001abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:42.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.608+0000 7f2e53fff700 1 -- 172.21.15.90:0/2287089093 reap_dead start 2024-04-14T15:18:42.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.608+0000 7f2e53fff700 1 -- 172.21.15.90:0/2287089093 reap_dead start 2024-04-14T15:18:42.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.608+0000 7f2e53fff700 1 -- 172.21.15.90:0/2287089093 reap_dead start 2024-04-14T15:18:42.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.608+0000 7f2e59f61700 20 client.37323.objecter put_session s=0x7f2e4001c3a0 osd=5 1 2024-04-14T15:18:42.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.608+0000 7f2e59f61700 10 client.37323.objecter close_session for osd.6 2024-04-14T15:18:42.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.608+0000 7f2e59f61700 1 -- 172.21.15.90:0/2287089093 >> v1:172.21.15.92:6808/636125963 conn(0x7f2e4001fab0 legacy=0x7f2e40021ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:42.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.608+0000 7f2e59f61700 20 client.37323.objecter put_session s=0x7f2e4001d1a0 osd=6 1 2024-04-14T15:18:42.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.608+0000 7f2e59f61700 10 client.37323.objecter close_session for osd.7 2024-04-14T15:18:42.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.608+0000 7f2e59f61700 1 -- 172.21.15.90:0/2287089093 >> v1:172.21.15.92:6812/1773706716 conn(0x7f2e40023c40 legacy=0x7f2e40026030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:42.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.608+0000 7f2e59f61700 20 client.37323.objecter put_session s=0x7f2e400235c0 osd=7 1 2024-04-14T15:18:42.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.608+0000 7f2e59f61700 20 client.37323.objecter shutdown clearing up homeless session... 2024-04-14T15:18:42.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.608+0000 7f2e59f61700 10 client.37323.objecter successfully canceled tick 2024-04-14T15:18:42.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.608+0000 7f2e59f61700 1 -- 172.21.15.90:0/2287089093 >> v1:172.21.15.90:6800/420935427 conn(0x7f2e3c0713f0 legacy=0x7f2e3c0738b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:42.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.608+0000 7f2e59f61700 1 -- 172.21.15.90:0/2287089093 >> v1:172.21.15.90:6789/0 conn(0x7f2e5410a790 legacy=0x7f2e5410c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:42.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.608+0000 7f2e53fff700 1 -- 172.21.15.90:0/2287089093 reap_dead start 2024-04-14T15:18:42.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.608+0000 7f2e59f61700 1 -- 172.21.15.90:0/2287089093 shutdown_connections 2024-04-14T15:18:42.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.608+0000 7f2e59f61700 1 -- 172.21.15.90:0/2287089093 >> 172.21.15.90:0/2287089093 conn(0x7f2e540fd2c0 msgr2=0x7f2e5406b020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:18:42.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.609+0000 7f2e59f61700 1 -- 172.21.15.90:0/2287089093 shutdown_connections 2024-04-14T15:18:42.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.609+0000 7f2e59f61700 1 -- 172.21.15.90:0/2287089093 wait complete. 2024-04-14T15:18:42.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:42.609+0000 7f2e59f61700 1 librados: shutdown 2024-04-14T15:18:42.612 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:18:42.612 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:18:42.613 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:18:42.613 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:18:42.613 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:18:42.613 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:18:42.613 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:18:42.613 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:18:42.696 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:18:42.721 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:42.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:42.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:18:42.822 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:18:42.847 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:42.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:42.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:18:42.948 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:18:42.971 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:42.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:42.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:18:43.000 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-04-14T15:18:43.000 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:18:43.074 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:43.100 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:43.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:43.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:43.201 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:18:43.225 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:43.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:43.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:18:43.312 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:43 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-04-14T15:18:43.312 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:43 smithi092 ceph-mon[33966]: osdmap e433: 8 total, 8 up, 8 in 2024-04-14T15:18:43.312 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:43 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2287089093' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:18:43.327 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:18:43.350 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:43.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:43.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:18:43.430 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:43 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-04-14T15:18:43.431 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:43 smithi090 ceph-mon[30399]: osdmap e433: 8 total, 8 up, 8 in 2024-04-14T15:18:43.431 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:43 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2287089093' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:18:43.431 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:43 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-04-14T15:18:43.431 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:43 smithi090 ceph-mon[41670]: osdmap e433: 8 total, 8 up, 8 in 2024-04-14T15:18:43.431 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:43 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2287089093' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:18:43.451 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:18:43.475 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:43.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:43.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:18:43.576 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:18:43.588 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-14T15:18:43.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:43.600 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:43.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:18:43.701 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:18:43.725 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:43.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:43.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:18:43.805 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-14T15:18:43.805 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-14T15:18:43.805 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-04-14T15:18:43.805 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph pg dump --format=json 2024-04-14T15:18:43.827 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:18:43.837 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:18:43.861 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:43.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:43.861 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:18:43.962 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:18:43.980 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-14T15:18:43.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:43.986 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:43.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:18:44.088 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:18:44.092 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:44 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:18:44.092 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:44 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:18:44.092 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:44 smithi090 ceph-mon[30399]: osdmap e434: 8 total, 8 up, 8 in 2024-04-14T15:18:44.093 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:44 smithi090 ceph-mon[30399]: pgmap v608: 248 pgs: 10 unknown, 238 active+clean; 8.4 MiB data, 585 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:18:44.093 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:18:43 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:18:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:18:44.093 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:44 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:18:44.120 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:44.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:44.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:18:44.221 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:18:44.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:44.245 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:44.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:18:44.346 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:44.361 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:44 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:18:44.361 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:44 smithi090 ceph-mon[41670]: osdmap e434: 8 total, 8 up, 8 in 2024-04-14T15:18:44.361 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:44 smithi090 ceph-mon[41670]: pgmap v608: 248 pgs: 10 unknown, 238 active+clean; 8.4 MiB data, 585 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:18:44.372 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:44.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:44 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:18:44.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:44 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:18:44.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:44 smithi092 ceph-mon[33966]: osdmap e434: 8 total, 8 up, 8 in 2024-04-14T15:18:44.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:44 smithi092 ceph-mon[33966]: pgmap v608: 248 pgs: 10 unknown, 238 active+clean; 8.4 MiB data, 585 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:18:44.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:44.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:44.474 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:18:44.497 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:44.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:44.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:18:44.575 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:18:44.599 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:18:44.609 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:18:44.625 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:44.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:44.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:18:44.727 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:18:44.750 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:44.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:44.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:18:44.852 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:18:44.879 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:44.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:44.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:18:44.981 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:18:45.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:45.006 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:45.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:18:45.107 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:18:45.130 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:45.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:45.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:18:45.232 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:18:45.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:45.256 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:45.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:18:45.357 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:18:45.361 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:45 smithi090 ceph-mon[30399]: osdmap e435: 8 total, 8 up, 8 in 2024-04-14T15:18:45.361 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:45 smithi090 ceph-mon[41670]: osdmap e435: 8 total, 8 up, 8 in 2024-04-14T15:18:45.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:45 smithi092 ceph-mon[33966]: osdmap e435: 8 total, 8 up, 8 in 2024-04-14T15:18:45.372 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:18:45 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:18:45.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:45.383 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:45.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:18:45.458 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-14T15:18:45.484 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:45.512 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:45.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:45.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:45.614 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:45.640 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:45.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:45.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:45.742 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:18:45.766 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:45.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:45.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:18:45.868 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:18:45.895 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:45.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:45.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:18:45.997 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:18:46.022 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:46.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:46.027 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:18:46.128 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:18:46.151 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:46.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:46.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:18:46.253 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:18:46.277 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:46.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:46.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:18:46.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:46 smithi092 ceph-mon[33966]: pgmap v611: 248 pgs: 1 active+undersized+remapped, 9 remapped+peering, 238 active+clean; 8.5 MiB data, 585 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 82 KiB/s wr, 14 op/s; 59 KiB/s, 0 objects/s recovering 2024-04-14T15:18:46.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:46 smithi092 ceph-mon[33966]: osdmap e436: 8 total, 8 up, 8 in 2024-04-14T15:18:46.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:46 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:18:46.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:46 smithi092 ceph-mon[33966]: osdmap e437: 8 total, 8 up, 8 in 2024-04-14T15:18:46.379 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:18:46.404 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:46.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:46.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:18:46.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:46 smithi090 ceph-mon[41670]: pgmap v611: 248 pgs: 1 active+undersized+remapped, 9 remapped+peering, 238 active+clean; 8.5 MiB data, 585 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 82 KiB/s wr, 14 op/s; 59 KiB/s, 0 objects/s recovering 2024-04-14T15:18:46.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:46 smithi090 ceph-mon[41670]: osdmap e436: 8 total, 8 up, 8 in 2024-04-14T15:18:46.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:46 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:18:46.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:46 smithi090 ceph-mon[41670]: osdmap e437: 8 total, 8 up, 8 in 2024-04-14T15:18:46.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:46 smithi090 ceph-mon[30399]: pgmap v611: 248 pgs: 1 active+undersized+remapped, 9 remapped+peering, 238 active+clean; 8.5 MiB data, 585 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 82 KiB/s wr, 14 op/s; 59 KiB/s, 0 objects/s recovering 2024-04-14T15:18:46.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:46 smithi090 ceph-mon[30399]: osdmap e436: 8 total, 8 up, 8 in 2024-04-14T15:18:46.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:46 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:18:46.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:46 smithi090 ceph-mon[30399]: osdmap e437: 8 total, 8 up, 8 in 2024-04-14T15:18:46.505 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:18:46.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:46.530 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:46.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:18:46.631 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:18:46.656 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:46.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:46.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:18:46.758 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:18:46.781 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:46.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:46.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:18:46.883 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:18:46.908 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:46.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:46.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:18:47.010 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:18:47.034 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:47.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:47.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:18:47.136 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:18:47.161 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:47.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:47.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:18:47.263 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:18:47.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:47.288 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:47.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:18:47.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:47 smithi092 ceph-mon[33966]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-04-14T15:18:47.389 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:18:47.413 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:47.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:47.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:18:47.427 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:47 smithi090 ceph-mon[41670]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-04-14T15:18:47.428 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:47 smithi090 ceph-mon[30399]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-04-14T15:18:47.515 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:18:47.539 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:47.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:47.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:18:47.641 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:18:47.664 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:47.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:47.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:18:47.766 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:18:47.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:47.792 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:47.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:18:47.893 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:18:47.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:47.919 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:47.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:18:48.020 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:18:48.043 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:48.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:48.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:18:48.145 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:18:48.169 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:48.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:48.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:18:48.267 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:48 smithi090 ceph-mon[41670]: pgmap v613: 248 pgs: 1 active+undersized+remapped, 9 remapped+peering, 238 active+clean; 8.5 MiB data, 585 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 82 KiB/s wr, 14 op/s; 59 KiB/s, 0 objects/s recovering 2024-04-14T15:18:48.267 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:48 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:18:48.267 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:48 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:18:48.267 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:48 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-63"}]: dispatch 2024-04-14T15:18:48.268 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:48 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-63"}]: dispatch 2024-04-14T15:18:48.268 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:48 smithi090 ceph-mon[30399]: pgmap v613: 248 pgs: 1 active+undersized+remapped, 9 remapped+peering, 238 active+clean; 8.5 MiB data, 585 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 82 KiB/s wr, 14 op/s; 59 KiB/s, 0 objects/s recovering 2024-04-14T15:18:48.268 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:48 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:18:48.268 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:48 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:18:48.268 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:48 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-63"}]: dispatch 2024-04-14T15:18:48.268 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:48 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-63"}]: dispatch 2024-04-14T15:18:48.271 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:48.295 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:48.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:48.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:48.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.266+0000 7f250fbf1700 1 -- 172.21.15.90:0/2352905403 >> v1:172.21.15.90:6789/0 conn(0x7f25081088c0 legacy=0x7f2508108ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:48.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.266+0000 7f250fbf1700 1 -- 172.21.15.90:0/2352905403 shutdown_connections 2024-04-14T15:18:48.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.266+0000 7f250fbf1700 1 -- 172.21.15.90:0/2352905403 >> 172.21.15.90:0/2352905403 conn(0x7f25080fe3d0 msgr2=0x7f25081007f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:18:48.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.266+0000 7f250fbf1700 1 -- 172.21.15.90:0/2352905403 shutdown_connections 2024-04-14T15:18:48.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.266+0000 7f250fbf1700 1 -- 172.21.15.90:0/2352905403 wait complete. 2024-04-14T15:18:48.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.267+0000 7f250fbf1700 1 librados: starting msgr at 2024-04-14T15:18:48.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.267+0000 7f250fbf1700 1 librados: starting objecter 2024-04-14T15:18:48.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.267+0000 7f250fbf1700 1 Processor -- start 2024-04-14T15:18:48.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.267+0000 7f250fbf1700 1 -- start start 2024-04-14T15:18:48.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.267+0000 7f250fbf1700 1 librados: setting wanted keys 2024-04-14T15:18:48.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.267+0000 7f250fbf1700 1 librados: calling monclient init 2024-04-14T15:18:48.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.267+0000 7f250fbf1700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f25080769e0 con 0x7f250810f340 2024-04-14T15:18:48.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.267+0000 7f250fbf1700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f2508076bf0 con 0x7f250810b770 2024-04-14T15:18:48.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.267+0000 7f250fbf1700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f25081ac350 con 0x7f25081088c0 2024-04-14T15:18:48.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.267+0000 7f250d98d700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f25081088c0 0x7f2508074db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:50170/0 (socket says 172.21.15.90:50170) 2024-04-14T15:18:48.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.267+0000 7f250d98d700 1 -- 172.21.15.90:0/1276931502 learned_addr learned my addr 172.21.15.90:0/1276931502 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:18:48.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.267+0000 7f250e18e700 1 --1- 172.21.15.90:0/1276931502 >> v1:172.21.15.90:6789/0 conn(0x7f250810f340 0x7f25081a8c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:40862/0 (socket says 172.21.15.90:40862) 2024-04-14T15:18:48.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.267+0000 7f24feffd700 10 client.?.objecter ms_handle_connect 0x7f25081088c0 2024-04-14T15:18:48.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.267+0000 7f24feffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:18:48.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.267+0000 7f24feffd700 10 client.?.objecter ms_handle_connect 0x7f250810f340 2024-04-14T15:18:48.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.267+0000 7f24feffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:18:48.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.267+0000 7f24feffd700 10 client.?.objecter ms_handle_connect 0x7f250810b770 2024-04-14T15:18:48.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.267+0000 7f24feffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:18:48.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.268+0000 7f24feffd700 1 -- 172.21.15.90:0/1276931502 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3597929808 0 0) 0x7f25081ac350 con 0x7f25081088c0 2024-04-14T15:18:48.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.268+0000 7f24feffd700 1 -- 172.21.15.90:0/1276931502 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f24f40035b0 con 0x7f25081088c0 2024-04-14T15:18:48.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.268+0000 7f24feffd700 1 -- 172.21.15.90:0/1276931502 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1731208159 0 0) 0x7f25080769e0 con 0x7f250810f340 2024-04-14T15:18:48.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.268+0000 7f24feffd700 1 -- 172.21.15.90:0/1276931502 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f25081ac350 con 0x7f250810f340 2024-04-14T15:18:48.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.268+0000 7f24feffd700 1 -- 172.21.15.90:0/1276931502 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 658570008 0 0) 0x7f2508076bf0 con 0x7f250810b770 2024-04-14T15:18:48.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.268+0000 7f24feffd700 1 -- 172.21.15.90:0/1276931502 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f25080769e0 con 0x7f250810b770 2024-04-14T15:18:48.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.268+0000 7f24feffd700 1 -- 172.21.15.90:0/1276931502 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2254006679 0 0) 0x7f25080769e0 con 0x7f250810b770 2024-04-14T15:18:48.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.268+0000 7f24feffd700 1 -- 172.21.15.90:0/1276931502 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2508076bf0 con 0x7f250810b770 2024-04-14T15:18:48.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.268+0000 7f24feffd700 1 -- 172.21.15.90:0/1276931502 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f25040018b0 con 0x7f250810b770 2024-04-14T15:18:48.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.268+0000 7f24feffd700 1 -- 172.21.15.90:0/1276931502 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1238195365 0 0) 0x7f25081ac350 con 0x7f250810f340 2024-04-14T15:18:48.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.268+0000 7f24feffd700 1 -- 172.21.15.90:0/1276931502 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f25080769e0 con 0x7f250810f340 2024-04-14T15:18:48.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.268+0000 7f24feffd700 1 -- 172.21.15.90:0/1276931502 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3131551113 0 0) 0x7f24f40035b0 con 0x7f25081088c0 2024-04-14T15:18:48.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.268+0000 7f24feffd700 1 -- 172.21.15.90:0/1276931502 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f25081ac350 con 0x7f25081088c0 2024-04-14T15:18:48.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.268+0000 7f24feffd700 1 -- 172.21.15.90:0/1276931502 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f2500001bb0 con 0x7f250810f340 2024-04-14T15:18:48.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.268+0000 7f24feffd700 1 -- 172.21.15.90:0/1276931502 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f24f80015d0 con 0x7f25081088c0 2024-04-14T15:18:48.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.268+0000 7f24feffd700 1 -- 172.21.15.90:0/1276931502 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1695536085 0 0) 0x7f2508076bf0 con 0x7f250810b770 2024-04-14T15:18:48.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.268+0000 7f24feffd700 1 -- 172.21.15.90:0/1276931502 >> v1:172.21.15.90:6790/0 conn(0x7f25081088c0 legacy=0x7f2508074db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:48.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.268+0000 7f24feffd700 1 -- 172.21.15.90:0/1276931502 >> v1:172.21.15.90:6789/0 conn(0x7f250810f340 legacy=0x7f25081a8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:48.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.268+0000 7f24feffd700 1 -- 172.21.15.90:0/1276931502 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25081ad530 con 0x7f250810b770 2024-04-14T15:18:48.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.268+0000 7f250fbf1700 1 -- 172.21.15.90:0/1276931502 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f25081ac580 con 0x7f250810b770 2024-04-14T15:18:48.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.268+0000 7f250fbf1700 10 client.49168.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:18:48.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.268+0000 7f250fbf1700 1 -- 172.21.15.90:0/1276931502 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f25081aca40 con 0x7f250810b770 2024-04-14T15:18:48.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.268+0000 7f250fbf1700 1 librados: init done 2024-04-14T15:18:48.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.268+0000 7f24feffd700 1 -- 172.21.15.90:0/1276931502 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f2504003b40 con 0x7f250810b770 2024-04-14T15:18:48.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.268+0000 7f24feffd700 1 -- 172.21.15.90:0/1276931502 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f2504004af0 con 0x7f250810b770 2024-04-14T15:18:48.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.269+0000 7f250fbf1700 1 -- 172.21.15.90:0/1276931502 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24ec005120 con 0x7f250810b770 2024-04-14T15:18:48.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.269+0000 7f24feffd700 1 -- 172.21.15.90:0/1276931502 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f2504003680 con 0x7f250810b770 2024-04-14T15:18:48.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.270+0000 7f24feffd700 1 -- 172.21.15.90:0/1276931502 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(438..438 src has 51..438) ==== 7775+0+0 (unknown 347659255 0 0) 0x7f25040895a0 con 0x7f250810b770 2024-04-14T15:18:48.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.270+0000 7f24feffd700 10 client.49168.objecter ms_dispatch 0x7f2508001550 osd_map(438..438 src has 51..438) 2024-04-14T15:18:48.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.270+0000 7f24feffd700 3 client.49168.objecter handle_osd_map got epochs [438,438] > 0 2024-04-14T15:18:48.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.270+0000 7f24feffd700 3 client.49168.objecter handle_osd_map decoding full epoch 438 2024-04-14T15:18:48.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.270+0000 7f24feffd700 20 client.49168.objecter dump_active .. 0 homeless 2024-04-14T15:18:48.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.270+0000 7f24feffd700 10 client.49168.objecter ms_handle_connect 0x7f24f4071430 2024-04-14T15:18:48.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.275+0000 7f24feffd700 1 -- 172.21.15.90:0/1276931502 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f2504056140 con 0x7f250810b770 2024-04-14T15:18:48.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:48 smithi092 ceph-mon[33966]: pgmap v613: 248 pgs: 1 active+undersized+remapped, 9 remapped+peering, 238 active+clean; 8.5 MiB data, 585 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 82 KiB/s wr, 14 op/s; 59 KiB/s, 0 objects/s recovering 2024-04-14T15:18:48.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:48 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:18:48.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:48 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:18:48.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:48 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-63"}]: dispatch 2024-04-14T15:18:48.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:48 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-63"}]: dispatch 2024-04-14T15:18:48.396 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:18:48.420 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:48.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:48.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:18:48.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.490+0000 7f250fbf1700 1 -- 172.21.15.90:0/1276931502 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7f24ec005410 con 0x7f250810b770 2024-04-14T15:18:48.522 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:18:48.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:48.550 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:48.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:18:48.651 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:18:48.674 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:48.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:48.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:18:48.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.586+0000 7f76a8f6c700 1 -- 172.21.15.90:0/20633131 >> v1:172.21.15.90:6790/0 conn(0x7f76a41077a0 legacy=0x7f76a4107b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:48.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.587+0000 7f76a8f6c700 1 -- 172.21.15.90:0/20633131 shutdown_connections 2024-04-14T15:18:48.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.587+0000 7f76a8f6c700 1 -- 172.21.15.90:0/20633131 >> 172.21.15.90:0/20633131 conn(0x7f76a40fd2b0 msgr2=0x7f76a40ff6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:18:48.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.587+0000 7f76a8f6c700 1 -- 172.21.15.90:0/20633131 shutdown_connections 2024-04-14T15:18:48.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.587+0000 7f76a8f6c700 1 -- 172.21.15.90:0/20633131 wait complete. 2024-04-14T15:18:48.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.587+0000 7f76a8f6c700 1 librados: starting msgr at 2024-04-14T15:18:48.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.587+0000 7f76a8f6c700 1 librados: starting objecter 2024-04-14T15:18:48.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.587+0000 7f76a8f6c700 1 Processor -- start 2024-04-14T15:18:48.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.587+0000 7f76a8f6c700 1 -- start start 2024-04-14T15:18:48.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.587+0000 7f76a8f6c700 1 librados: setting wanted keys 2024-04-14T15:18:48.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.587+0000 7f76a8f6c700 1 librados: calling monclient init 2024-04-14T15:18:48.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.588+0000 7f76a8f6c700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f76a41a9b40 con 0x7f76a410a650 2024-04-14T15:18:48.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.588+0000 7f76a8f6c700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f76a41aad60 con 0x7f76a41077a0 2024-04-14T15:18:48.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.588+0000 7f76a8f6c700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f76a41abfa0 con 0x7f76a410e220 2024-04-14T15:18:48.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.588+0000 7f76a1d9b700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f76a410a650 0x7f76a41a4ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:40892/0 (socket says 172.21.15.90:40892) 2024-04-14T15:18:48.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.588+0000 7f76a1d9b700 1 -- 172.21.15.90:0/184547637 learned_addr learned my addr 172.21.15.90:0/184547637 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:18:48.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.588+0000 7f76a259c700 1 --1- 172.21.15.90:0/184547637 >> v1:172.21.15.92:6789/0 conn(0x7f76a41077a0 0x7f76a410d8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.90:57044/0 (socket says 172.21.15.90:57044) 2024-04-14T15:18:48.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.588+0000 7f76937fe700 10 client.?.objecter ms_handle_connect 0x7f76a410a650 2024-04-14T15:18:48.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.588+0000 7f76937fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:18:48.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.588+0000 7f76937fe700 10 client.?.objecter ms_handle_connect 0x7f76a41077a0 2024-04-14T15:18:48.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.588+0000 7f76937fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:18:48.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.588+0000 7f76937fe700 10 client.?.objecter ms_handle_connect 0x7f76a410e220 2024-04-14T15:18:48.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.588+0000 7f76937fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:18:48.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.588+0000 7f76937fe700 1 -- 172.21.15.90:0/184547637 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1033233912 0 0) 0x7f76a41aad60 con 0x7f76a41077a0 2024-04-14T15:18:48.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.588+0000 7f76937fe700 1 -- 172.21.15.90:0/184547637 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f768c0039c0 con 0x7f76a41077a0 2024-04-14T15:18:48.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.588+0000 7f76937fe700 1 -- 172.21.15.90:0/184547637 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3132647718 0 0) 0x7f76a41a9b40 con 0x7f76a410a650 2024-04-14T15:18:48.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.588+0000 7f76937fe700 1 -- 172.21.15.90:0/184547637 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f76a41aad60 con 0x7f76a410a650 2024-04-14T15:18:48.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.588+0000 7f76937fe700 1 -- 172.21.15.90:0/184547637 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2153730707 0 0) 0x7f76a41abfa0 con 0x7f76a410e220 2024-04-14T15:18:48.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.588+0000 7f76937fe700 1 -- 172.21.15.90:0/184547637 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f76a41a9b40 con 0x7f76a410e220 2024-04-14T15:18:48.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.588+0000 7f76937fe700 1 -- 172.21.15.90:0/184547637 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 120040166 0 0) 0x7f768c0039c0 con 0x7f76a41077a0 2024-04-14T15:18:48.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.588+0000 7f76937fe700 1 -- 172.21.15.90:0/184547637 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f76a41abfa0 con 0x7f76a41077a0 2024-04-14T15:18:48.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.588+0000 7f76937fe700 1 -- 172.21.15.90:0/184547637 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f7694004410 con 0x7f76a41077a0 2024-04-14T15:18:48.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.588+0000 7f76937fe700 1 -- 172.21.15.90:0/184547637 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2149130869 0 0) 0x7f76a41aad60 con 0x7f76a410a650 2024-04-14T15:18:48.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.588+0000 7f76937fe700 1 -- 172.21.15.90:0/184547637 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f768c0039c0 con 0x7f76a410a650 2024-04-14T15:18:48.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.588+0000 7f76937fe700 1 -- 172.21.15.90:0/184547637 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1847477609 0 0) 0x7f76a41a9b40 con 0x7f76a410e220 2024-04-14T15:18:48.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.588+0000 7f76937fe700 1 -- 172.21.15.90:0/184547637 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f76a41aad60 con 0x7f76a410e220 2024-04-14T15:18:48.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.588+0000 7f76937fe700 1 -- 172.21.15.90:0/184547637 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3330288354 0 0) 0x7f76a41abfa0 con 0x7f76a41077a0 2024-04-14T15:18:48.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.589+0000 7f76937fe700 1 -- 172.21.15.90:0/184547637 >> v1:172.21.15.90:6790/0 conn(0x7f76a410e220 legacy=0x7f76a41a8200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:48.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.589+0000 7f76937fe700 1 -- 172.21.15.90:0/184547637 >> v1:172.21.15.90:6789/0 conn(0x7f76a410a650 legacy=0x7f76a41a4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:48.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.589+0000 7f76937fe700 1 -- 172.21.15.90:0/184547637 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76a41ad1e0 con 0x7f76a41077a0 2024-04-14T15:18:48.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.589+0000 7f76a8f6c700 1 -- 172.21.15.90:0/184547637 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f76a41a9d70 con 0x7f76a41077a0 2024-04-14T15:18:48.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.589+0000 7f76a8f6c700 10 client.49174.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:18:48.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.589+0000 7f76a8f6c700 1 -- 172.21.15.90:0/184547637 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f76a41aa230 con 0x7f76a41077a0 2024-04-14T15:18:48.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.589+0000 7f76a8f6c700 1 librados: init done 2024-04-14T15:18:48.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.590+0000 7f76937fe700 1 -- 172.21.15.90:0/184547637 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f7694001c80 con 0x7f76a41077a0 2024-04-14T15:18:48.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.590+0000 7f76a8f6c700 1 -- 172.21.15.90:0/184547637 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76a4102ee0 con 0x7f76a41077a0 2024-04-14T15:18:48.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.590+0000 7f76937fe700 1 -- 172.21.15.90:0/184547637 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f7694004d70 con 0x7f76a41077a0 2024-04-14T15:18:48.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.591+0000 7f76937fe700 1 -- 172.21.15.90:0/184547637 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f7694005050 con 0x7f76a41077a0 2024-04-14T15:18:48.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.592+0000 7f76937fe700 1 -- 172.21.15.90:0/184547637 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(438..438 src has 51..438) ==== 7775+0+0 (unknown 347659255 0 0) 0x7f769408a940 con 0x7f76a41077a0 2024-04-14T15:18:48.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.592+0000 7f76937fe700 10 client.49174.objecter ms_dispatch 0x7f76a4001550 osd_map(438..438 src has 51..438) 2024-04-14T15:18:48.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.592+0000 7f76937fe700 3 client.49174.objecter handle_osd_map got epochs [438,438] > 0 2024-04-14T15:18:48.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.592+0000 7f76937fe700 3 client.49174.objecter handle_osd_map decoding full epoch 438 2024-04-14T15:18:48.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.592+0000 7f76937fe700 20 client.49174.objecter dump_active .. 0 homeless 2024-04-14T15:18:48.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.592+0000 7f76937fe700 10 client.49174.objecter ms_handle_connect 0x7f768c0713f0 2024-04-14T15:18:48.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.595+0000 7f76937fe700 1 -- 172.21.15.90:0/184547637 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f76940574e0 con 0x7f76a41077a0 2024-04-14T15:18:48.778 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:18:48.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.797+0000 7f76a8f6c700 1 -- 172.21.15.90:0/184547637 --> v1:172.21.15.90:6800/420935427 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f76a410aeb0 con 0x7f768c0713f0 2024-04-14T15:18:48.802 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:48.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:48.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:18:48.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.813+0000 7f76937fe700 1 -- 172.21.15.90:0/184547637 <== mgr.14610 v1:172.21.15.90:6800/420935427 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+557850 (unknown 2965378022 0 3986264780) 0x7f76a410aeb0 con 0x7f768c0713f0 2024-04-14T15:18:48.820 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:18:48.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.817+0000 7f76a8f6c700 10 librados: watch_flush enter 2024-04-14T15:18:48.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.817+0000 7f76a8f6c700 10 librados: watch_flush exit 2024-04-14T15:18:48.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.817+0000 7f76a8f6c700 20 client.49174.objecter shutdown clearing up homeless session... 2024-04-14T15:18:48.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.817+0000 7f76a8f6c700 10 client.49174.objecter successfully canceled tick 2024-04-14T15:18:48.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.817+0000 7f76a8f6c700 1 -- 172.21.15.90:0/184547637 >> v1:172.21.15.90:6800/420935427 conn(0x7f768c0713f0 legacy=0x7f768c0738b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:48.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.817+0000 7f76a8f6c700 1 -- 172.21.15.90:0/184547637 >> v1:172.21.15.92:6789/0 conn(0x7f76a41077a0 legacy=0x7f76a410d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:48.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.818+0000 7f76a8f6c700 1 -- 172.21.15.90:0/184547637 shutdown_connections 2024-04-14T15:18:48.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.818+0000 7f76a8f6c700 1 -- 172.21.15.90:0/184547637 >> 172.21.15.90:0/184547637 conn(0x7f76a40fd2b0 msgr2=0x7f76a4111470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:18:48.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.818+0000 7f76a8f6c700 1 -- 172.21.15.90:0/184547637 shutdown_connections 2024-04-14T15:18:48.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.818+0000 7f76a8f6c700 1 -- 172.21.15.90:0/184547637 wait complete. 2024-04-14T15:18:48.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:48.818+0000 7f76a8f6c700 1 librados: shutdown 2024-04-14T15:18:48.826 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-14T15:18:48.905 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:48.929 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:48.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:48.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:49.031 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:18:49.054 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:49.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:49.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:18:49.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:49.114+0000 7f24feffd700 1 -- 172.21.15.90:0/1276931502 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v439) ==== 97+0+0 (unknown 2257801178 0 0) 0x7f2504059df0 con 0x7f250810b770 2024-04-14T15:18:49.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:49.120+0000 7f250fbf1700 10 librados: watch_flush enter 2024-04-14T15:18:49.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:49.121+0000 7f250fbf1700 10 librados: watch_flush exit 2024-04-14T15:18:49.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:49.121+0000 7f250fbf1700 20 client.49168.objecter shutdown clearing up homeless session... 2024-04-14T15:18:49.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:49.121+0000 7f250fbf1700 10 client.49168.objecter successfully canceled tick 2024-04-14T15:18:49.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:49.122+0000 7f250fbf1700 1 -- 172.21.15.90:0/1276931502 >> v1:172.21.15.90:6800/420935427 conn(0x7f24f4071430 legacy=0x7f24f40738f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:49.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:49.122+0000 7f250fbf1700 1 -- 172.21.15.90:0/1276931502 >> v1:172.21.15.92:6789/0 conn(0x7f250810b770 legacy=0x7f25080754c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:49.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:49.122+0000 7f250fbf1700 1 -- 172.21.15.90:0/1276931502 shutdown_connections 2024-04-14T15:18:49.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:49.122+0000 7f250fbf1700 1 -- 172.21.15.90:0/1276931502 >> 172.21.15.90:0/1276931502 conn(0x7f25080fe3d0 msgr2=0x7f25081028c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:18:49.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:49.122+0000 7f250fbf1700 1 -- 172.21.15.90:0/1276931502 shutdown_connections 2024-04-14T15:18:49.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:49.123+0000 7f250fbf1700 1 -- 172.21.15.90:0/1276931502 wait complete. 2024-04-14T15:18:49.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:49.123+0000 7f250fbf1700 1 librados: shutdown 2024-04-14T15:18:49.124 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is set 2024-04-14T15:18:49.156 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:18:49.179 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:49.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:49.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:18:49.281 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:18:49.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:49.305 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:49.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:18:49.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:49 smithi092 ceph-mon[33966]: osdmap e438: 8 total, 8 up, 8 in 2024-04-14T15:18:49.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:49 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1276931502' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:18:49.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:49 smithi092 ceph-mon[33966]: from='client.49168 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:18:49.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:49 smithi092 ceph-mon[33966]: from='client.49174 v1:172.21.15.90:0/184547637' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:18:49.406 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:18:49.430 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:49.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:49.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:18:49.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:49 smithi090 ceph-mon[30399]: osdmap e438: 8 total, 8 up, 8 in 2024-04-14T15:18:49.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:49 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1276931502' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:18:49.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:49 smithi090 ceph-mon[30399]: from='client.49168 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:18:49.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:49 smithi090 ceph-mon[30399]: from='client.49174 v1:172.21.15.90:0/184547637' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:18:49.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:49 smithi090 ceph-mon[41670]: osdmap e438: 8 total, 8 up, 8 in 2024-04-14T15:18:49.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:49 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1276931502' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:18:49.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:49 smithi090 ceph-mon[41670]: from='client.49168 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:18:49.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:49 smithi090 ceph-mon[41670]: from='client.49174 v1:172.21.15.90:0/184547637' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:18:49.531 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:18:49.555 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:49.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:49.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:18:49.593 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":614,"stamp":"2024-04-14T15:18:48.100413+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8872685,"num_objects":359,"num_object_clones":2,"num_object_copies":1077,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":359,"num_whiteouts":0,"num_read":1397,"num_read_kb":1062,"num_write":1067,"num_write_kb":13070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":241955,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":7,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5199,"ondisk_log_size":5199,"up":648,"acting":638,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":745,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":599120,"kb_used_data":84368,"kb_used_omap":12,"kb_used_meta":514483,"kb_avail":749362096,"statfs":{"total":767960285184,"available":767346786304,"internally_reserved":0,"allocated":86392832,"data_stored":59892648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":526831154},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":80493,"num_objects":-2,"num_object_clones":0,"num_object_copies":-6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":21,"num_read_kb":36,"num_write":13,"num_write_kb":326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":241515,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.018025"},"pg_stats":[{"pgid":"110.11","version":"403'41","reported_seq":864,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.077595+0000","last_change":"2024-04-14T15:18:38.445844+0000","last_active":"2024-04-14T15:18:43.077595+0000","last_peered":"2024-04-14T15:18:43.077595+0000","last_clean":"2024-04-14T15:18:43.077595+0000","last_became_active":"2024-04-14T15:13:09.936619+0000","last_became_peered":"2024-04-14T15:13:09.936619+0000","last_unstale":"2024-04-14T15:18:43.077595+0000","last_undegraded":"2024-04-14T15:18:43.077595+0000","last_fullsized":"2024-04-14T15:18:43.077595+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"403'41","last_scrub_stamp":"2024-04-14T15:17:56.911004+0000","last_deep_scrub":"403'41","last_deep_scrub_stamp":"2024-04-14T15:17:56.911004+0000","last_clean_scrub_stamp":"2024-04-14T15:17:56.911004+0000","objects_scrubbed":2,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:15.028059+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0.00067473100000000003,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"10","length":"1"}]},{"pgid":"110.10","version":"253'158","reported_seq":946,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.072012+0000","last_change":"2024-04-14T15:18:38.049876+0000","last_active":"2024-04-14T15:18:43.072012+0000","last_peered":"2024-04-14T15:18:43.072012+0000","last_clean":"2024-04-14T15:18:43.072012+0000","last_became_active":"2024-04-14T15:13:09.928915+0000","last_became_peered":"2024-04-14T15:13:09.928915+0000","last_unstale":"2024-04-14T15:18:43.072012+0000","last_undegraded":"2024-04-14T15:18:43.072012+0000","last_fullsized":"2024-04-14T15:18:43.072012+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"253'158","last_scrub_stamp":"2024-04-14T15:17:34.291854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:34.291854+0000","objects_scrubbed":10,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:54.588177+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00087379199999999997,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":47,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"10","length":"1"}]},{"pgid":"110.13","version":"409'41","reported_seq":867,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-04-14T15:18:44.088202+0000","last_change":"2024-04-14T15:18:38.057933+0000","last_active":"2024-04-14T15:18:44.088202+0000","last_peered":"2024-04-14T15:18:44.088202+0000","last_clean":"2024-04-14T15:18:44.088202+0000","last_became_active":"2024-04-14T15:13:09.918152+0000","last_became_peered":"2024-04-14T15:13:09.918152+0000","last_unstale":"2024-04-14T15:18:44.088202+0000","last_undegraded":"2024-04-14T15:18:44.088202+0000","last_fullsized":"2024-04-14T15:18:44.088202+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'39","last_scrub_stamp":"2024-04-14T15:17:44.873766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:44.873766+0000","objects_scrubbed":3,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:12.557633+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0041410489999999999,"stat_sum":{"num_bytes":27,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":14,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"10","length":"1"}]},{"pgid":"110.12","version":"409'70","reported_seq":893,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-04-14T15:18:44.088183+0000","last_change":"2024-04-14T15:18:38.053556+0000","last_active":"2024-04-14T15:18:44.088183+0000","last_peered":"2024-04-14T15:18:44.088183+0000","last_clean":"2024-04-14T15:18:44.088183+0000","last_became_active":"2024-04-14T15:13:09.939566+0000","last_became_peered":"2024-04-14T15:13:09.939566+0000","last_unstale":"2024-04-14T15:18:44.088183+0000","last_undegraded":"2024-04-14T15:18:44.088183+0000","last_fullsized":"2024-04-14T15:18:44.088183+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"253'66","last_scrub_stamp":"2024-04-14T15:16:53.673906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:16:53.673906+0000","objects_scrubbed":6,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001852088,"stat_sum":{"num_bytes":74,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":24,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"10","length":"1"}]},{"pgid":"110.15","version":"190'34","reported_seq":860,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.078823+0000","last_change":"2024-04-14T15:18:38.063881+0000","last_active":"2024-04-14T15:18:43.078823+0000","last_peered":"2024-04-14T15:18:43.078823+0000","last_clean":"2024-04-14T15:18:43.078823+0000","last_became_active":"2024-04-14T15:13:09.960121+0000","last_became_peered":"2024-04-14T15:13:09.960121+0000","last_unstale":"2024-04-14T15:18:43.078823+0000","last_undegraded":"2024-04-14T15:18:43.078823+0000","last_fullsized":"2024-04-14T15:18:43.078823+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'34","last_scrub_stamp":"2024-04-14T15:18:36.013135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:18:36.013135+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:01.409788+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00058071800000000003,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"10","length":"1"}]},{"pgid":"110.14","version":"355'87","reported_seq":906,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.077320+0000","last_change":"2024-04-14T15:18:38.062676+0000","last_active":"2024-04-14T15:18:43.077320+0000","last_peered":"2024-04-14T15:18:43.077320+0000","last_clean":"2024-04-14T15:18:43.077320+0000","last_became_active":"2024-04-14T15:13:09.899561+0000","last_became_peered":"2024-04-14T15:13:09.899561+0000","last_unstale":"2024-04-14T15:18:43.077320+0000","last_undegraded":"2024-04-14T15:18:43.077320+0000","last_fullsized":"2024-04-14T15:18:43.077320+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'87","last_scrub_stamp":"2024-04-14T15:18:31.117033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:18:31.117033+0000","objects_scrubbed":7,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:47.204334+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0029321619999999999,"stat_sum":{"num_bytes":71,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"10","length":"1"}]},{"pgid":"110.17","version":"234'6","reported_seq":839,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.077753+0000","last_change":"2024-04-14T15:18:38.064277+0000","last_active":"2024-04-14T15:18:43.077753+0000","last_peered":"2024-04-14T15:18:43.077753+0000","last_clean":"2024-04-14T15:18:43.077753+0000","last_became_active":"2024-04-14T15:13:09.898521+0000","last_became_peered":"2024-04-14T15:13:09.898521+0000","last_unstale":"2024-04-14T15:18:43.077753+0000","last_undegraded":"2024-04-14T15:18:43.077753+0000","last_fullsized":"2024-04-14T15:18:43.077753+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'6","last_scrub_stamp":"2024-04-14T15:18:38.009133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:18:38.009133+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:52.099524+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.000889301,"stat_sum":{"num_bytes":13,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"10","length":"1"}]},{"pgid":"110.16","version":"355'35","reported_seq":692,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.082089+0000","last_change":"2024-04-14T15:18:38.065100+0000","last_active":"2024-04-14T15:18:43.082089+0000","last_peered":"2024-04-14T15:18:43.082089+0000","last_clean":"2024-04-14T15:18:43.082089+0000","last_became_active":"2024-04-14T15:15:21.889783+0000","last_became_peered":"2024-04-14T15:15:21.889783+0000","last_unstale":"2024-04-14T15:18:43.082089+0000","last_undegraded":"2024-04-14T15:18:43.082089+0000","last_fullsized":"2024-04-14T15:18:43.082089+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'35","last_scrub_stamp":"2024-04-14T15:17:34.097563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:34.097563+0000","objects_scrubbed":3,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:03.216748+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0011231329999999999,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"10","length":"1"}]},{"pgid":"110.19","version":"237'85","reported_seq":901,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.072879+0000","last_change":"2024-04-14T15:18:38.057219+0000","last_active":"2024-04-14T15:18:43.072879+0000","last_peered":"2024-04-14T15:18:43.072879+0000","last_clean":"2024-04-14T15:18:43.072879+0000","last_became_active":"2024-04-14T15:13:09.998139+0000","last_became_peered":"2024-04-14T15:13:09.998139+0000","last_unstale":"2024-04-14T15:18:43.072879+0000","last_undegraded":"2024-04-14T15:18:43.072879+0000","last_fullsized":"2024-04-14T15:18:43.072879+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'85","last_scrub_stamp":"2024-04-14T15:16:04.177097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:16:04.177097+0000","objects_scrubbed":5,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0058291940000000002,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"10","length":"1"}]},{"pgid":"110.18","version":"387'94","reported_seq":911,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.072576+0000","last_change":"2024-04-14T15:18:38.056279+0000","last_active":"2024-04-14T15:18:43.072576+0000","last_peered":"2024-04-14T15:18:43.072576+0000","last_clean":"2024-04-14T15:18:43.072576+0000","last_became_active":"2024-04-14T15:13:09.998803+0000","last_became_peered":"2024-04-14T15:13:09.998803+0000","last_unstale":"2024-04-14T15:18:43.072576+0000","last_undegraded":"2024-04-14T15:18:43.072576+0000","last_fullsized":"2024-04-14T15:18:43.072576+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'94","last_scrub_stamp":"2024-04-14T15:17:36.005598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:36.005598+0000","objects_scrubbed":8,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:59.942257+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0056524660000000001,"stat_sum":{"num_bytes":65,"num_objects":8,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":29,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"10","length":"1"}]},{"pgid":"110.1b","version":"355'91","reported_seq":909,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-04-14T15:18:44.088105+0000","last_change":"2024-04-14T15:18:38.055913+0000","last_active":"2024-04-14T15:18:44.088105+0000","last_peered":"2024-04-14T15:18:44.088105+0000","last_clean":"2024-04-14T15:18:44.088105+0000","last_became_active":"2024-04-14T15:13:09.946919+0000","last_became_peered":"2024-04-14T15:13:09.946919+0000","last_unstale":"2024-04-14T15:18:44.088105+0000","last_undegraded":"2024-04-14T15:18:44.088105+0000","last_fullsized":"2024-04-14T15:18:44.088105+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'91","last_scrub_stamp":"2024-04-14T15:17:47.889364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:47.889364+0000","objects_scrubbed":6,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:55.150676+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0020748450000000001,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":28,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"10","length":"1"}]},{"pgid":"110.1a","version":"387'39","reported_seq":868,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.077466+0000","last_change":"2024-04-14T15:18:38.446238+0000","last_active":"2024-04-14T15:18:43.077466+0000","last_peered":"2024-04-14T15:18:43.077466+0000","last_clean":"2024-04-14T15:18:43.077466+0000","last_became_active":"2024-04-14T15:13:09.960319+0000","last_became_peered":"2024-04-14T15:13:09.960319+0000","last_unstale":"2024-04-14T15:18:43.077466+0000","last_undegraded":"2024-04-14T15:18:43.077466+0000","last_fullsized":"2024-04-14T15:18:43.077466+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'39","last_scrub_stamp":"2024-04-14T15:17:53.871590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:53.871590+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:58.820650+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00070185699999999996,"stat_sum":{"num_bytes":62,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"10","length":"1"}]},{"pgid":"110.1d","version":"225'85","reported_seq":903,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.076767+0000","last_change":"2024-04-14T15:18:38.684883+0000","last_active":"2024-04-14T15:18:43.076767+0000","last_peered":"2024-04-14T15:18:43.076767+0000","last_clean":"2024-04-14T15:18:43.076767+0000","last_became_active":"2024-04-14T15:13:09.868082+0000","last_became_peered":"2024-04-14T15:13:09.868082+0000","last_unstale":"2024-04-14T15:18:43.076767+0000","last_undegraded":"2024-04-14T15:18:43.076767+0000","last_fullsized":"2024-04-14T15:18:43.076767+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"225'85","last_scrub_stamp":"2024-04-14T15:17:36.419088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:36.419088+0000","objects_scrubbed":5,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:03.415095+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001737242,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"10","length":"1"}]},{"pgid":"110.1c","version":"430'95","reported_seq":908,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-04-14T15:18:44.086332+0000","last_change":"2024-04-14T15:18:38.061233+0000","last_active":"2024-04-14T15:18:44.086332+0000","last_peered":"2024-04-14T15:18:44.086332+0000","last_clean":"2024-04-14T15:18:44.086332+0000","last_became_active":"2024-04-14T15:13:09.954091+0000","last_became_peered":"2024-04-14T15:13:09.954091+0000","last_unstale":"2024-04-14T15:18:44.086332+0000","last_undegraded":"2024-04-14T15:18:44.086332+0000","last_fullsized":"2024-04-14T15:18:44.086332+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'73","last_scrub_stamp":"2024-04-14T15:16:04.615572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:16:04.615572+0000","objects_scrubbed":5,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":1,"snaptrim_duration":0.0090083779999999992,"stat_sum":{"num_bytes":65,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"10","length":"1"}]},{"pgid":"110.1f","version":"268'25","reported_seq":897,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.075476+0000","last_change":"2024-04-14T15:18:38.057178+0000","last_active":"2024-04-14T15:18:43.075476+0000","last_peered":"2024-04-14T15:18:43.075476+0000","last_clean":"2024-04-14T15:18:43.075476+0000","last_became_active":"2024-04-14T15:13:09.999629+0000","last_became_peered":"2024-04-14T15:13:09.999629+0000","last_unstale":"2024-04-14T15:18:43.075476+0000","last_undegraded":"2024-04-14T15:18:43.075476+0000","last_fullsized":"2024-04-14T15:18:43.075476+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'25","last_scrub_stamp":"2024-04-14T15:16:08.236881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:16:08.236881+0000","objects_scrubbed":2,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.002565992,"stat_sum":{"num_bytes":4194315,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":8,"num_write_kb":8198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"10","length":"1"}]},{"pgid":"110.1e","version":"355'23","reported_seq":845,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.080285+0000","last_change":"2024-04-14T15:18:38.064024+0000","last_active":"2024-04-14T15:18:43.080285+0000","last_peered":"2024-04-14T15:18:43.080285+0000","last_clean":"2024-04-14T15:18:43.080285+0000","last_became_active":"2024-04-14T15:13:09.955932+0000","last_became_peered":"2024-04-14T15:13:09.955932+0000","last_unstale":"2024-04-14T15:18:43.080285+0000","last_undegraded":"2024-04-14T15:18:43.080285+0000","last_fullsized":"2024-04-14T15:18:43.080285+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'22","last_scrub_stamp":"2024-04-14T15:16:10.427994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:16:10.427994+0000","objects_scrubbed":1,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:46.956783+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00075683200000000003,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":8,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"10","length":"1"}]},{"pgid":"110.1","version":"406'32","reported_seq":864,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.075034+0000","last_change":"2024-04-14T15:18:38.685918+0000","last_active":"2024-04-14T15:18:43.075034+0000","last_peered":"2024-04-14T15:18:43.075034+0000","last_clean":"2024-04-14T15:18:43.075034+0000","last_became_active":"2024-04-14T15:13:09.929562+0000","last_became_peered":"2024-04-14T15:13:09.929562+0000","last_unstale":"2024-04-14T15:18:43.075034+0000","last_undegraded":"2024-04-14T15:18:43.075034+0000","last_fullsized":"2024-04-14T15:18:43.075034+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"406'32","last_scrub_stamp":"2024-04-14T15:18:38.685469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:18:38.685469+0000","objects_scrubbed":7,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:46.332124+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0037741139999999999,"stat_sum":{"num_bytes":4000050,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":15,"num_write_kb":3921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"10","length":"1"}]},{"pgid":"110.0","version":"219'85","reported_seq":897,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.072115+0000","last_change":"2024-04-14T15:18:38.049941+0000","last_active":"2024-04-14T15:18:43.072115+0000","last_peered":"2024-04-14T15:18:43.072115+0000","last_clean":"2024-04-14T15:18:43.072115+0000","last_became_active":"2024-04-14T15:13:09.856407+0000","last_became_peered":"2024-04-14T15:13:09.856407+0000","last_unstale":"2024-04-14T15:18:43.072115+0000","last_undegraded":"2024-04-14T15:18:43.072115+0000","last_fullsized":"2024-04-14T15:18:43.072115+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"219'85","last_scrub_stamp":"2024-04-14T15:17:51.438149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:51.438149+0000","objects_scrubbed":5,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:00.014309+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00078646200000000003,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"10","length":"1"}]},{"pgid":"110.3","version":"355'52","reported_seq":861,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.076757+0000","last_change":"2024-04-14T15:18:38.056841+0000","last_active":"2024-04-14T15:18:43.076757+0000","last_peered":"2024-04-14T15:18:43.076757+0000","last_clean":"2024-04-14T15:18:43.076757+0000","last_became_active":"2024-04-14T15:13:09.997208+0000","last_became_peered":"2024-04-14T15:13:09.997208+0000","last_unstale":"2024-04-14T15:18:43.076757+0000","last_undegraded":"2024-04-14T15:18:43.076757+0000","last_fullsized":"2024-04-14T15:18:43.076757+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"223'51","last_scrub_stamp":"2024-04-14T15:16:06.149271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:16:06.149271+0000","objects_scrubbed":3,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0032602920000000001,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"10","length":"1"}]},{"pgid":"110.2","version":"240'55","reported_seq":879,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.074115+0000","last_change":"2024-04-14T15:18:38.056503+0000","last_active":"2024-04-14T15:18:43.074115+0000","last_peered":"2024-04-14T15:18:43.074115+0000","last_clean":"2024-04-14T15:18:43.074115+0000","last_became_active":"2024-04-14T15:13:09.957187+0000","last_became_peered":"2024-04-14T15:13:09.957187+0000","last_unstale":"2024-04-14T15:18:43.074115+0000","last_undegraded":"2024-04-14T15:18:43.074115+0000","last_fullsized":"2024-04-14T15:18:43.074115+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'55","last_scrub_stamp":"2024-04-14T15:17:37.941589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:37.941589+0000","objects_scrubbed":6,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:59.789448+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.005509814,"stat_sum":{"num_bytes":49,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":18,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"10","length":"1"}]},{"pgid":"110.5","version":"396'64","reported_seq":880,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.073716+0000","last_change":"2024-04-14T15:18:38.051427+0000","last_active":"2024-04-14T15:18:43.073716+0000","last_peered":"2024-04-14T15:18:43.073716+0000","last_clean":"2024-04-14T15:18:43.073716+0000","last_became_active":"2024-04-14T15:13:09.853689+0000","last_became_peered":"2024-04-14T15:13:09.853689+0000","last_unstale":"2024-04-14T15:18:43.073716+0000","last_undegraded":"2024-04-14T15:18:43.073716+0000","last_fullsized":"2024-04-14T15:18:43.073716+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"396'64","last_scrub_stamp":"2024-04-14T15:17:47.418687+0000","last_deep_scrub":"396'64","last_deep_scrub_stamp":"2024-04-14T15:17:47.418687+0000","last_clean_scrub_stamp":"2024-04-14T15:17:47.418687+0000","objects_scrubbed":5,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:03.859673+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00070666900000000005,"stat_sum":{"num_bytes":42,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":20,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"10","length":"1"}]},{"pgid":"110.4","version":"406'55","reported_seq":869,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.077429+0000","last_change":"2024-04-14T15:18:38.062608+0000","last_active":"2024-04-14T15:18:43.077429+0000","last_peered":"2024-04-14T15:18:43.077429+0000","last_clean":"2024-04-14T15:18:43.077429+0000","last_became_active":"2024-04-14T15:13:09.958847+0000","last_became_peered":"2024-04-14T15:13:09.958847+0000","last_unstale":"2024-04-14T15:18:43.077429+0000","last_undegraded":"2024-04-14T15:18:43.077429+0000","last_fullsized":"2024-04-14T15:18:43.077429+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'51","last_scrub_stamp":"2024-04-14T15:16:05.487850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:16:05.487850+0000","objects_scrubbed":3,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:46.956688+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.002580583,"stat_sum":{"num_bytes":38,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"10","length":"1"}]},{"pgid":"110.7","version":"355'18","reported_seq":845,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.080928+0000","last_change":"2024-04-14T15:18:38.447551+0000","last_active":"2024-04-14T15:18:43.080928+0000","last_peered":"2024-04-14T15:18:43.080928+0000","last_clean":"2024-04-14T15:18:43.080928+0000","last_became_active":"2024-04-14T15:13:09.936631+0000","last_became_peered":"2024-04-14T15:13:09.936631+0000","last_unstale":"2024-04-14T15:18:43.080928+0000","last_undegraded":"2024-04-14T15:18:43.080928+0000","last_fullsized":"2024-04-14T15:18:43.080928+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'18","last_scrub_stamp":"2024-04-14T15:17:34.552788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:34.552788+0000","objects_scrubbed":2,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:04.371512+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00048435200000000001,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"10","length":"1"}]},{"pgid":"110.6","version":"274'74","reported_seq":880,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.076709+0000","last_change":"2024-04-14T15:18:38.684782+0000","last_active":"2024-04-14T15:18:43.076709+0000","last_peered":"2024-04-14T15:18:43.076709+0000","last_clean":"2024-04-14T15:18:43.076709+0000","last_became_active":"2024-04-14T15:13:09.930125+0000","last_became_peered":"2024-04-14T15:13:09.930125+0000","last_unstale":"2024-04-14T15:18:43.076709+0000","last_undegraded":"2024-04-14T15:18:43.076709+0000","last_fullsized":"2024-04-14T15:18:43.076709+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'74","last_scrub_stamp":"2024-04-14T15:15:42.641684+0000","last_deep_scrub":"274'74","last_deep_scrub_stamp":"2024-04-14T15:15:42.641684+0000","last_clean_scrub_stamp":"2024-04-14T15:15:42.641684+0000","objects_scrubbed":5,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0014868520000000001,"stat_sum":{"num_bytes":44,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":21,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"10","length":"1"}]},{"pgid":"110.9","version":"369'20","reported_seq":841,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.072765+0000","last_change":"2024-04-14T15:18:38.055881+0000","last_active":"2024-04-14T15:18:43.072765+0000","last_peered":"2024-04-14T15:18:43.072765+0000","last_clean":"2024-04-14T15:18:43.072765+0000","last_became_active":"2024-04-14T15:13:09.957324+0000","last_became_peered":"2024-04-14T15:13:09.957324+0000","last_unstale":"2024-04-14T15:18:43.072765+0000","last_undegraded":"2024-04-14T15:18:43.072765+0000","last_fullsized":"2024-04-14T15:18:43.072765+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"225'17","last_scrub_stamp":"2024-04-14T15:16:09.202462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:16:09.202462+0000","objects_scrubbed":1,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0051830060000000004,"stat_sum":{"num_bytes":35,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"10","length":"1"}]},{"pgid":"110.8","version":"190'17","reported_seq":596,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.076285+0000","last_change":"2024-04-14T15:18:38.060563+0000","last_active":"2024-04-14T15:18:43.076285+0000","last_peered":"2024-04-14T15:18:43.076285+0000","last_clean":"2024-04-14T15:18:43.076285+0000","last_became_active":"2024-04-14T15:15:44.890057+0000","last_became_peered":"2024-04-14T15:15:44.890057+0000","last_unstale":"2024-04-14T15:18:43.076285+0000","last_undegraded":"2024-04-14T15:18:43.076285+0000","last_fullsized":"2024-04-14T15:18:43.076285+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'17","last_scrub_stamp":"2024-04-14T15:18:34.505521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:18:34.505521+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:46.888739+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001114353,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"10","length":"1"}]},{"pgid":"110.b","version":"412'71","reported_seq":889,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-04-14T15:18:44.088351+0000","last_change":"2024-04-14T15:18:38.059200+0000","last_active":"2024-04-14T15:18:44.088351+0000","last_peered":"2024-04-14T15:18:44.088351+0000","last_clean":"2024-04-14T15:18:44.088351+0000","last_became_active":"2024-04-14T15:13:09.964629+0000","last_became_peered":"2024-04-14T15:13:09.964629+0000","last_unstale":"2024-04-14T15:18:44.088351+0000","last_undegraded":"2024-04-14T15:18:44.088351+0000","last_fullsized":"2024-04-14T15:18:44.088351+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'68","last_scrub_stamp":"2024-04-14T15:16:49.611563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:16:49.611563+0000","objects_scrubbed":4,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.0051718889999999998,"stat_sum":{"num_bytes":54,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":22,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"10","length":"1"}]},{"pgid":"110.a","version":"387'74","reported_seq":812,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.079467+0000","last_change":"2024-04-14T15:18:38.062499+0000","last_active":"2024-04-14T15:18:43.079467+0000","last_peered":"2024-04-14T15:18:43.079467+0000","last_clean":"2024-04-14T15:18:43.079467+0000","last_became_active":"2024-04-14T15:15:21.885346+0000","last_became_peered":"2024-04-14T15:15:21.885346+0000","last_unstale":"2024-04-14T15:18:43.079467+0000","last_undegraded":"2024-04-14T15:18:43.079467+0000","last_fullsized":"2024-04-14T15:18:43.079467+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'70","last_scrub_stamp":"2024-04-14T15:15:34.881731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:15:34.881731+0000","objects_scrubbed":5,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0.00090285599999999995,"stat_sum":{"num_bytes":84,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":26,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"10","length":"1"}]},{"pgid":"110.d","version":"416'95","reported_seq":736,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.081160+0000","last_change":"2024-04-14T15:18:38.062235+0000","last_active":"2024-04-14T15:18:43.081160+0000","last_peered":"2024-04-14T15:18:43.081160+0000","last_clean":"2024-04-14T15:18:43.081160+0000","last_became_active":"2024-04-14T15:15:21.877073+0000","last_became_peered":"2024-04-14T15:15:21.877073+0000","last_unstale":"2024-04-14T15:18:43.081160+0000","last_undegraded":"2024-04-14T15:18:43.081160+0000","last_fullsized":"2024-04-14T15:18:43.081160+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'85","last_scrub_stamp":"2024-04-14T15:15:38.773256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:15:38.773256+0000","objects_scrubbed":5,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00060486299999999997,"stat_sum":{"num_bytes":71,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":34,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"10","length":"1"}]},{"pgid":"110.c","version":"398'20","reported_seq":847,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-04-14T15:18:44.086446+0000","last_change":"2024-04-14T15:18:38.058646+0000","last_active":"2024-04-14T15:18:44.086446+0000","last_peered":"2024-04-14T15:18:44.086446+0000","last_clean":"2024-04-14T15:18:44.086446+0000","last_became_active":"2024-04-14T15:13:09.984609+0000","last_became_peered":"2024-04-14T15:13:09.984609+0000","last_unstale":"2024-04-14T15:18:44.086446+0000","last_undegraded":"2024-04-14T15:18:44.086446+0000","last_fullsized":"2024-04-14T15:18:44.086446+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'20","last_scrub_stamp":"2024-04-14T15:17:52.882509+0000","last_deep_scrub":"398'20","last_deep_scrub_stamp":"2024-04-14T15:17:52.882509+0000","last_clean_scrub_stamp":"2024-04-14T15:17:52.882509+0000","objects_scrubbed":2,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:01.243821+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0047393610000000001,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"10","length":"1"}]},{"pgid":"110.f","version":"355'52","reported_seq":875,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-04-14T15:18:44.083817+0000","last_change":"2024-04-14T15:18:38.054056+0000","last_active":"2024-04-14T15:18:44.083817+0000","last_peered":"2024-04-14T15:18:44.083817+0000","last_clean":"2024-04-14T15:18:44.083817+0000","last_became_active":"2024-04-14T15:13:09.964057+0000","last_became_peered":"2024-04-14T15:13:09.964057+0000","last_unstale":"2024-04-14T15:18:44.083817+0000","last_undegraded":"2024-04-14T15:18:44.083817+0000","last_fullsized":"2024-04-14T15:18:44.083817+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'52","last_scrub_stamp":"2024-04-14T15:17:45.848166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:45.848166+0000","objects_scrubbed":4,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:15.552690+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0019022220000000001,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"10","length":"1"}]},{"pgid":"110.e","version":"416'92","reported_seq":908,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.075613+0000","last_change":"2024-04-14T15:18:38.685139+0000","last_active":"2024-04-14T15:18:43.075613+0000","last_peered":"2024-04-14T15:18:43.075613+0000","last_clean":"2024-04-14T15:18:43.075613+0000","last_became_active":"2024-04-14T15:13:09.984980+0000","last_became_peered":"2024-04-14T15:13:09.984980+0000","last_unstale":"2024-04-14T15:18:43.075613+0000","last_undegraded":"2024-04-14T15:18:43.075613+0000","last_fullsized":"2024-04-14T15:18:43.075613+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'90","last_scrub_stamp":"2024-04-14T15:17:34.409367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:34.409367+0000","objects_scrubbed":6,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:43.560642+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0017801080000000001,"stat_sum":{"num_bytes":68,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":28,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"10","length":"1"}]},{"pgid":"1.d","version":"91'130","reported_seq":328,"reported_epoch":435,"state":"active+undersized+remapped","last_fresh":"2024-04-14T15:18:44.091174+0000","last_change":"2024-04-14T15:18:44.090863+0000","last_active":"2024-04-14T15:18:44.091174+0000","last_peered":"2024-04-14T15:18:44.091174+0000","last_clean":"2024-04-14T15:11:28.874295+0000","last_became_active":"2024-04-14T15:18:44.090863+0000","last_became_peered":"2024-04-14T15:18:44.090863+0000","last_unstale":"2024-04-14T15:18:44.091174+0000","last_undegraded":"2024-04-14T15:18:44.091174+0000","last_fullsized":"2024-04-14T15:18:44.084186+0000","mapping_epoch":434,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:16:41.193422+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:16:41.193422+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":14,"num_write":8,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":80505,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[2,4],"avail_no_missing":["2","1","4","6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":327,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.075572+0000","last_change":"2024-04-14T15:15:44.883275+0000","last_active":"2024-04-14T15:18:43.075572+0000","last_peered":"2024-04-14T15:18:43.075572+0000","last_clean":"2024-04-14T15:18:43.075572+0000","last_became_active":"2024-04-14T15:15:44.882941+0000","last_became_peered":"2024-04-14T15:15:44.882941+0000","last_unstale":"2024-04-14T15:18:43.075572+0000","last_undegraded":"2024-04-14T15:18:43.075572+0000","last_fullsized":"2024-04-14T15:18:43.075572+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:32.465858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:32.465858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.9","version":"85'9","reported_seq":893,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.075837+0000","last_change":"2024-04-14T15:17:54.606158+0000","last_active":"2024-04-14T15:18:43.075837+0000","last_peered":"2024-04-14T15:18:43.075837+0000","last_clean":"2024-04-14T15:18:43.075837+0000","last_became_active":"2024-04-14T15:09:47.045441+0000","last_became_peered":"2024-04-14T15:09:47.045441+0000","last_unstale":"2024-04-14T15:18:43.075837+0000","last_undegraded":"2024-04-14T15:18:43.075837+0000","last_fullsized":"2024-04-14T15:18:43.075837+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9","last_scrub_stamp":"2024-04-14T15:17:54.606051+0000","last_deep_scrub":"85'9","last_deep_scrub_stamp":"2024-04-14T15:10:51.950580+0000","last_clean_scrub_stamp":"2024-04-14T15:17:54.606051+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:07.946751+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"430'22","reported_seq":384,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.075103+0000","last_change":"2024-04-14T15:18:39.704989+0000","last_active":"2024-04-14T15:18:43.075103+0000","last_peered":"2024-04-14T15:18:43.075103+0000","last_clean":"2024-04-14T15:18:43.075103+0000","last_became_active":"2024-04-14T15:15:44.885409+0000","last_became_peered":"2024-04-14T15:15:44.885409+0000","last_unstale":"2024-04-14T15:18:43.075103+0000","last_undegraded":"2024-04-14T15:18:43.075103+0000","last_fullsized":"2024-04-14T15:18:43.075103+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'22","last_scrub_stamp":"2024-04-14T15:18:39.704883+0000","last_deep_scrub":"430'22","last_deep_scrub_stamp":"2024-04-14T15:18:39.704883+0000","last_clean_scrub_stamp":"2024-04-14T15:18:39.704883+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:51.340270+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":784,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-04-14T15:18:44.087554+0000","last_change":"2024-04-14T15:17:53.867764+0000","last_active":"2024-04-14T15:18:44.087554+0000","last_peered":"2024-04-14T15:18:44.087554+0000","last_clean":"2024-04-14T15:18:44.087554+0000","last_became_active":"2024-04-14T15:09:43.021460+0000","last_became_peered":"2024-04-14T15:09:43.021460+0000","last_unstale":"2024-04-14T15:18:44.087554+0000","last_undegraded":"2024-04-14T15:18:44.087554+0000","last_fullsized":"2024-04-14T15:18:44.087554+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:53.867696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:53.867696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:56.203183+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":335,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.077700+0000","last_change":"2024-04-14T15:17:54.889170+0000","last_active":"2024-04-14T15:18:43.077700+0000","last_peered":"2024-04-14T15:18:43.077700+0000","last_clean":"2024-04-14T15:18:43.077700+0000","last_became_active":"2024-04-14T15:15:44.888154+0000","last_became_peered":"2024-04-14T15:15:44.888154+0000","last_unstale":"2024-04-14T15:18:43.077700+0000","last_undegraded":"2024-04-14T15:18:43.077700+0000","last_fullsized":"2024-04-14T15:18:43.077700+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:54.889066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:15:40.368219+0000","last_clean_scrub_stamp":"2024-04-14T15:17:54.889066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:11.492168+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"91'130","reported_seq":318,"reported_epoch":434,"state":"remapped+peering","last_fresh":"2024-04-14T15:18:43.079791+0000","last_change":"2024-04-14T15:18:43.079791+0000","last_active":"2024-04-14T15:18:43.079791+0000","last_peered":"2024-04-14T15:11:28.874295+0000","last_clean":"2024-04-14T15:11:28.874295+0000","last_became_active":"2024-04-14T15:09:33.753828+0000","last_became_peered":"2024-04-14T15:09:33.753828+0000","last_unstale":"2024-04-14T15:18:43.079791+0000","last_undegraded":"2024-04-14T15:18:43.079791+0000","last_fullsized":"2024-04-14T15:18:43.079791+0000","mapping_epoch":434,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:17:47.885308+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:17:47.885308+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":14,"num_write":8,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":80505,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":679,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.076250+0000","last_change":"2024-04-14T15:18:37.636498+0000","last_active":"2024-04-14T15:18:43.076250+0000","last_peered":"2024-04-14T15:18:43.076250+0000","last_clean":"2024-04-14T15:18:43.076250+0000","last_became_active":"2024-04-14T15:12:56.355348+0000","last_became_peered":"2024-04-14T15:12:56.355348+0000","last_unstale":"2024-04-14T15:18:43.076250+0000","last_undegraded":"2024-04-14T15:18:43.076250+0000","last_fullsized":"2024-04-14T15:18:43.076250+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:37.636419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:18:37.636419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:06.978194+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"72'9","reported_seq":797,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.080061+0000","last_change":"2024-04-14T15:16:56.663996+0000","last_active":"2024-04-14T15:18:43.080061+0000","last_peered":"2024-04-14T15:18:43.080061+0000","last_clean":"2024-04-14T15:18:43.080061+0000","last_became_active":"2024-04-14T15:09:45.037179+0000","last_became_peered":"2024-04-14T15:09:45.037179+0000","last_unstale":"2024-04-14T15:18:43.080061+0000","last_undegraded":"2024-04-14T15:18:43.080061+0000","last_fullsized":"2024-04-14T15:18:43.080061+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-04-14T15:16:56.663900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:16:56.663900+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:46.957094+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":785,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.075801+0000","last_change":"2024-04-14T15:18:36.029705+0000","last_active":"2024-04-14T15:18:43.075801+0000","last_peered":"2024-04-14T15:18:43.075801+0000","last_clean":"2024-04-14T15:18:43.075801+0000","last_became_active":"2024-04-14T15:09:43.020044+0000","last_became_peered":"2024-04-14T15:09:43.020044+0000","last_unstale":"2024-04-14T15:18:43.075801+0000","last_undegraded":"2024-04-14T15:18:43.075801+0000","last_fullsized":"2024-04-14T15:18:43.075801+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:36.029630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:18:36.029630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:49.562846+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":776,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-04-14T15:18:44.088416+0000","last_change":"2024-04-14T15:16:16.655593+0000","last_active":"2024-04-14T15:18:44.088416+0000","last_peered":"2024-04-14T15:18:44.088416+0000","last_clean":"2024-04-14T15:18:44.088416+0000","last_became_active":"2024-04-14T15:09:47.046153+0000","last_became_peered":"2024-04-14T15:09:47.046153+0000","last_unstale":"2024-04-14T15:18:44.088416+0000","last_undegraded":"2024-04-14T15:18:44.088416+0000","last_fullsized":"2024-04-14T15:18:44.088416+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:16.655504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:16.655504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":773,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.082397+0000","last_change":"2024-04-14T15:17:37.919665+0000","last_active":"2024-04-14T15:18:43.082397+0000","last_peered":"2024-04-14T15:18:43.082397+0000","last_clean":"2024-04-14T15:18:43.082397+0000","last_became_active":"2024-04-14T15:09:49.061495+0000","last_became_peered":"2024-04-14T15:09:49.061495+0000","last_unstale":"2024-04-14T15:18:43.082397+0000","last_undegraded":"2024-04-14T15:18:43.082397+0000","last_fullsized":"2024-04-14T15:18:43.082397+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:37.919595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:17:37.919595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:48.004516+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"91'130","reported_seq":847,"reported_epoch":434,"state":"remapped+peering","last_fresh":"2024-04-14T15:18:43.078581+0000","last_change":"2024-04-14T15:18:43.078581+0000","last_active":"2024-04-14T15:18:42.075741+0000","last_peered":"2024-04-14T15:18:40.920312+0000","last_clean":"2024-04-14T15:18:40.920312+0000","last_became_active":"2024-04-14T15:18:40.919515+0000","last_became_peered":"2024-04-14T15:18:40.919515+0000","last_unstale":"2024-04-14T15:18:43.078581+0000","last_undegraded":"2024-04-14T15:18:43.078581+0000","last_fullsized":"2024-04-14T15:18:43.078581+0000","mapping_epoch":434,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:17:40.725873+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:17:40.725873+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":679,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.077026+0000","last_change":"2024-04-14T15:18:25.523604+0000","last_active":"2024-04-14T15:18:43.077026+0000","last_peered":"2024-04-14T15:18:43.077026+0000","last_clean":"2024-04-14T15:18:43.077026+0000","last_became_active":"2024-04-14T15:12:56.246062+0000","last_became_peered":"2024-04-14T15:12:56.246062+0000","last_unstale":"2024-04-14T15:18:43.077026+0000","last_undegraded":"2024-04-14T15:18:43.077026+0000","last_fullsized":"2024-04-14T15:18:43.077026+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:25.523520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:18:25.523520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:36.616896+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"222'28","reported_seq":820,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.079353+0000","last_change":"2024-04-14T15:17:44.832902+0000","last_active":"2024-04-14T15:18:43.079353+0000","last_peered":"2024-04-14T15:18:43.079353+0000","last_clean":"2024-04-14T15:18:43.079353+0000","last_became_active":"2024-04-14T15:09:45.178782+0000","last_became_peered":"2024-04-14T15:09:45.178782+0000","last_unstale":"2024-04-14T15:18:43.079353+0000","last_undegraded":"2024-04-14T15:18:43.079353+0000","last_fullsized":"2024-04-14T15:18:43.079353+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'28","last_scrub_stamp":"2024-04-14T15:17:44.832769+0000","last_deep_scrub":"222'28","last_deep_scrub_stamp":"2024-04-14T15:14:50.054931+0000","last_clean_scrub_stamp":"2024-04-14T15:17:44.832769+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:54.927275+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":792,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-04-14T15:18:44.089175+0000","last_change":"2024-04-14T15:18:36.925573+0000","last_active":"2024-04-14T15:18:44.089175+0000","last_peered":"2024-04-14T15:18:44.089175+0000","last_clean":"2024-04-14T15:18:44.089175+0000","last_became_active":"2024-04-14T15:09:43.020268+0000","last_became_peered":"2024-04-14T15:09:43.020268+0000","last_unstale":"2024-04-14T15:18:44.089175+0000","last_undegraded":"2024-04-14T15:18:44.089175+0000","last_fullsized":"2024-04-14T15:18:44.089175+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:36.925496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:34.342793+0000","last_clean_scrub_stamp":"2024-04-14T15:18:36.925496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:51.029386+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":775,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-04-14T15:18:44.088441+0000","last_change":"2024-04-14T15:18:31.989052+0000","last_active":"2024-04-14T15:18:44.088441+0000","last_peered":"2024-04-14T15:18:44.088441+0000","last_clean":"2024-04-14T15:18:44.088441+0000","last_became_active":"2024-04-14T15:09:47.045542+0000","last_became_peered":"2024-04-14T15:09:47.045542+0000","last_unstale":"2024-04-14T15:18:44.088441+0000","last_undegraded":"2024-04-14T15:18:44.088441+0000","last_fullsized":"2024-04-14T15:18:44.088441+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:31.988957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:18:31.988957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:57.227960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":763,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.075576+0000","last_change":"2024-04-14T15:16:08.857476+0000","last_active":"2024-04-14T15:18:43.075576+0000","last_peered":"2024-04-14T15:18:43.075576+0000","last_clean":"2024-04-14T15:18:43.075576+0000","last_became_active":"2024-04-14T15:09:49.065397+0000","last_became_peered":"2024-04-14T15:09:49.065397+0000","last_unstale":"2024-04-14T15:18:43.075576+0000","last_undegraded":"2024-04-14T15:18:43.075576+0000","last_fullsized":"2024-04-14T15:18:43.075576+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:08.857397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:08.857397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"91'130","reported_seq":655,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-04-14T15:18:44.084240+0000","last_change":"2024-04-14T15:16:07.609240+0000","last_active":"2024-04-14T15:18:44.084240+0000","last_peered":"2024-04-14T15:18:44.084240+0000","last_clean":"2024-04-14T15:18:44.084240+0000","last_became_active":"2024-04-14T15:15:44.886240+0000","last_became_peered":"2024-04-14T15:15:44.886240+0000","last_unstale":"2024-04-14T15:18:44.084240+0000","last_undegraded":"2024-04-14T15:18:44.084240+0000","last_fullsized":"2024-04-14T15:18:44.084240+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:16:07.609119+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:16:07.609119+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":680,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-04-14T15:18:44.088330+0000","last_change":"2024-04-14T15:16:19.749313+0000","last_active":"2024-04-14T15:18:44.088330+0000","last_peered":"2024-04-14T15:18:44.088330+0000","last_clean":"2024-04-14T15:18:44.088330+0000","last_became_active":"2024-04-14T15:12:56.474322+0000","last_became_peered":"2024-04-14T15:12:56.474322+0000","last_unstale":"2024-04-14T15:18:44.088330+0000","last_undegraded":"2024-04-14T15:18:44.088330+0000","last_fullsized":"2024-04-14T15:18:44.088330+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:19.749208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:16:19.749208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"72'17","reported_seq":854,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.074123+0000","last_change":"2024-04-14T15:17:48.374589+0000","last_active":"2024-04-14T15:18:43.074123+0000","last_peered":"2024-04-14T15:18:43.074123+0000","last_clean":"2024-04-14T15:18:43.074123+0000","last_became_active":"2024-04-14T15:13:12.933907+0000","last_became_peered":"2024-04-14T15:13:12.933907+0000","last_unstale":"2024-04-14T15:18:43.074123+0000","last_undegraded":"2024-04-14T15:18:43.074123+0000","last_fullsized":"2024-04-14T15:18:43.074123+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'17","last_scrub_stamp":"2024-04-14T15:17:48.374469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:17:48.374469+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:04.297073+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":775,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.076368+0000","last_change":"2024-04-14T15:16:16.820151+0000","last_active":"2024-04-14T15:18:43.076368+0000","last_peered":"2024-04-14T15:18:43.076368+0000","last_clean":"2024-04-14T15:18:43.076368+0000","last_became_active":"2024-04-14T15:09:43.022351+0000","last_became_peered":"2024-04-14T15:09:43.022351+0000","last_unstale":"2024-04-14T15:18:43.076368+0000","last_undegraded":"2024-04-14T15:18:43.076368+0000","last_fullsized":"2024-04-14T15:18:43.076368+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:16.820070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:16.820070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":768,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.074634+0000","last_change":"2024-04-14T15:17:54.871693+0000","last_active":"2024-04-14T15:18:43.074634+0000","last_peered":"2024-04-14T15:18:43.074634+0000","last_clean":"2024-04-14T15:18:43.074634+0000","last_became_active":"2024-04-14T15:09:47.042810+0000","last_became_peered":"2024-04-14T15:09:47.042810+0000","last_unstale":"2024-04-14T15:18:43.074634+0000","last_undegraded":"2024-04-14T15:18:43.074634+0000","last_fullsized":"2024-04-14T15:18:43.074634+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:54.871593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:17:54.871593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:21.351148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":771,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-04-14T15:18:44.084224+0000","last_change":"2024-04-14T15:16:45.669141+0000","last_active":"2024-04-14T15:18:44.084224+0000","last_peered":"2024-04-14T15:18:44.084224+0000","last_clean":"2024-04-14T15:18:44.084224+0000","last_became_active":"2024-04-14T15:09:49.069594+0000","last_became_peered":"2024-04-14T15:09:49.069594+0000","last_unstale":"2024-04-14T15:18:44.084224+0000","last_undegraded":"2024-04-14T15:18:44.084224+0000","last_fullsized":"2024-04-14T15:18:44.084224+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:45.669063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:45.669063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"91'130","reported_seq":1056,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.074918+0000","last_change":"2024-04-14T15:18:22.518430+0000","last_active":"2024-04-14T15:18:43.074918+0000","last_peered":"2024-04-14T15:18:43.074918+0000","last_clean":"2024-04-14T15:18:43.074918+0000","last_became_active":"2024-04-14T15:14:17.873403+0000","last_became_peered":"2024-04-14T15:14:17.873403+0000","last_unstale":"2024-04-14T15:18:43.074918+0000","last_undegraded":"2024-04-14T15:18:43.074918+0000","last_fullsized":"2024-04-14T15:18:43.074918+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:18:22.518348+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:18:22.518348+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:50.784671+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":680,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.079853+0000","last_change":"2024-04-14T15:17:46.871883+0000","last_active":"2024-04-14T15:18:43.079853+0000","last_peered":"2024-04-14T15:18:43.079853+0000","last_clean":"2024-04-14T15:18:43.079853+0000","last_became_active":"2024-04-14T15:12:56.289745+0000","last_became_peered":"2024-04-14T15:12:56.289745+0000","last_unstale":"2024-04-14T15:18:43.079853+0000","last_undegraded":"2024-04-14T15:18:43.079853+0000","last_fullsized":"2024-04-14T15:18:43.079853+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:46.871773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:17:46.871773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:58.233534+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"179'15","reported_seq":790,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.073216+0000","last_change":"2024-04-14T15:16:49.243096+0000","last_active":"2024-04-14T15:18:43.073216+0000","last_peered":"2024-04-14T15:18:43.073216+0000","last_clean":"2024-04-14T15:18:43.073216+0000","last_became_active":"2024-04-14T15:09:45.029170+0000","last_became_peered":"2024-04-14T15:09:45.029170+0000","last_unstale":"2024-04-14T15:18:43.073216+0000","last_undegraded":"2024-04-14T15:18:43.073216+0000","last_fullsized":"2024-04-14T15:18:43.073216+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'15","last_scrub_stamp":"2024-04-14T15:16:49.243002+0000","last_deep_scrub":"179'15","last_deep_scrub_stamp":"2024-04-14T15:14:07.488422+0000","last_clean_scrub_stamp":"2024-04-14T15:16:49.243002+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:47.435911+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":785,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.075199+0000","last_change":"2024-04-14T15:17:51.854742+0000","last_active":"2024-04-14T15:18:43.075199+0000","last_peered":"2024-04-14T15:18:43.075199+0000","last_clean":"2024-04-14T15:18:43.075199+0000","last_became_active":"2024-04-14T15:09:43.018871+0000","last_became_peered":"2024-04-14T15:09:43.018871+0000","last_unstale":"2024-04-14T15:18:43.075199+0000","last_undegraded":"2024-04-14T15:18:43.075199+0000","last_fullsized":"2024-04-14T15:18:43.075199+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:51.854659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:51.854659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:08.464204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"400'9","reported_seq":893,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-04-14T15:18:44.086916+0000","last_change":"2024-04-14T15:17:54.845967+0000","last_active":"2024-04-14T15:18:44.086916+0000","last_peered":"2024-04-14T15:18:44.086916+0000","last_clean":"2024-04-14T15:18:44.086916+0000","last_became_active":"2024-04-14T15:09:47.044085+0000","last_became_peered":"2024-04-14T15:09:47.044085+0000","last_unstale":"2024-04-14T15:18:44.086916+0000","last_undegraded":"2024-04-14T15:18:44.086916+0000","last_fullsized":"2024-04-14T15:18:44.086916+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'9","last_scrub_stamp":"2024-04-14T15:17:54.845881+0000","last_deep_scrub":"400'9","last_deep_scrub_stamp":"2024-04-14T15:17:54.845881+0000","last_clean_scrub_stamp":"2024-04-14T15:17:54.845881+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:58.974945+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":762,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.073972+0000","last_change":"2024-04-14T15:16:17.041377+0000","last_active":"2024-04-14T15:18:43.073972+0000","last_peered":"2024-04-14T15:18:43.073972+0000","last_clean":"2024-04-14T15:18:43.073972+0000","last_became_active":"2024-04-14T15:09:49.065274+0000","last_became_peered":"2024-04-14T15:09:49.065274+0000","last_unstale":"2024-04-14T15:18:43.073972+0000","last_undegraded":"2024-04-14T15:18:43.073972+0000","last_fullsized":"2024-04-14T15:18:43.073972+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:17.041301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:17.041301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:47.435771+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"91'130","reported_seq":663,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-04-14T15:18:44.088030+0000","last_change":"2024-04-14T15:18:35.931022+0000","last_active":"2024-04-14T15:18:44.088030+0000","last_peered":"2024-04-14T15:18:44.088030+0000","last_clean":"2024-04-14T15:18:44.088030+0000","last_became_active":"2024-04-14T15:15:44.887114+0000","last_became_peered":"2024-04-14T15:15:44.887114+0000","last_unstale":"2024-04-14T15:18:44.088030+0000","last_undegraded":"2024-04-14T15:18:44.088030+0000","last_fullsized":"2024-04-14T15:18:44.088030+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:18:35.930937+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:18:35.930937+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:57.215057+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":673,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.073302+0000","last_change":"2024-04-14T15:15:29.366895+0000","last_active":"2024-04-14T15:18:43.073302+0000","last_peered":"2024-04-14T15:18:43.073302+0000","last_clean":"2024-04-14T15:18:43.073302+0000","last_became_active":"2024-04-14T15:12:56.487615+0000","last_became_peered":"2024-04-14T15:12:56.487615+0000","last_unstale":"2024-04-14T15:18:43.073302+0000","last_undegraded":"2024-04-14T15:18:43.073302+0000","last_fullsized":"2024-04-14T15:18:43.073302+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:29.366820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:29.366820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"398'22","reported_seq":812,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.075700+0000","last_change":"2024-04-14T15:17:52.883984+0000","last_active":"2024-04-14T15:18:43.075700+0000","last_peered":"2024-04-14T15:18:43.075700+0000","last_clean":"2024-04-14T15:18:43.075700+0000","last_became_active":"2024-04-14T15:09:45.033331+0000","last_became_peered":"2024-04-14T15:09:45.033331+0000","last_unstale":"2024-04-14T15:18:43.075700+0000","last_undegraded":"2024-04-14T15:18:43.075700+0000","last_fullsized":"2024-04-14T15:18:43.075700+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'22","last_scrub_stamp":"2024-04-14T15:17:52.883850+0000","last_deep_scrub":"398'22","last_deep_scrub_stamp":"2024-04-14T15:17:52.883850+0000","last_clean_scrub_stamp":"2024-04-14T15:17:52.883850+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:58.280825+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":785,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.073514+0000","last_change":"2024-04-14T15:18:25.009179+0000","last_active":"2024-04-14T15:18:43.073514+0000","last_peered":"2024-04-14T15:18:43.073514+0000","last_clean":"2024-04-14T15:18:43.073514+0000","last_became_active":"2024-04-14T15:09:43.020878+0000","last_became_peered":"2024-04-14T15:09:43.020878+0000","last_unstale":"2024-04-14T15:18:43.073514+0000","last_undegraded":"2024-04-14T15:18:43.073514+0000","last_fullsized":"2024-04-14T15:18:43.073514+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:25.009100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:18:25.009100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:54.811984+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"72'8","reported_seq":883,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.073922+0000","last_change":"2024-04-14T15:16:06.933663+0000","last_active":"2024-04-14T15:18:43.073922+0000","last_peered":"2024-04-14T15:18:43.073922+0000","last_clean":"2024-04-14T15:18:43.073922+0000","last_became_active":"2024-04-14T15:09:47.050962+0000","last_became_peered":"2024-04-14T15:09:47.050962+0000","last_unstale":"2024-04-14T15:18:43.073922+0000","last_undegraded":"2024-04-14T15:18:43.073922+0000","last_fullsized":"2024-04-14T15:18:43.073922+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:16:06.933489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:06.933489+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:47.435787+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":336,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.075296+0000","last_change":"2024-04-14T15:16:51.199025+0000","last_active":"2024-04-14T15:18:43.075296+0000","last_peered":"2024-04-14T15:18:43.075296+0000","last_clean":"2024-04-14T15:18:43.075296+0000","last_became_active":"2024-04-14T15:15:44.879699+0000","last_became_peered":"2024-04-14T15:15:44.879699+0000","last_unstale":"2024-04-14T15:18:43.075296+0000","last_undegraded":"2024-04-14T15:18:43.075296+0000","last_fullsized":"2024-04-14T15:18:43.075296+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:51.198907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:51.198907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"91'130","reported_seq":841,"reported_epoch":434,"state":"remapped+peering","last_fresh":"2024-04-14T15:18:43.076550+0000","last_change":"2024-04-14T15:18:43.076550+0000","last_active":"2024-04-14T15:18:42.067073+0000","last_peered":"2024-04-14T15:18:40.916954+0000","last_clean":"2024-04-14T15:18:40.916954+0000","last_became_active":"2024-04-14T15:18:40.914819+0000","last_became_peered":"2024-04-14T15:18:40.914819+0000","last_unstale":"2024-04-14T15:18:43.076550+0000","last_undegraded":"2024-04-14T15:18:43.076550+0000","last_fullsized":"2024-04-14T15:18:43.076550+0000","mapping_epoch":434,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:17:49.890445+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:17:49.890445+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"233'8","reported_seq":694,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.078170+0000","last_change":"2024-04-14T15:17:50.905698+0000","last_active":"2024-04-14T15:18:43.078170+0000","last_peered":"2024-04-14T15:18:43.078170+0000","last_clean":"2024-04-14T15:18:43.078170+0000","last_became_active":"2024-04-14T15:12:56.381026+0000","last_became_peered":"2024-04-14T15:12:56.381026+0000","last_unstale":"2024-04-14T15:18:43.078170+0000","last_undegraded":"2024-04-14T15:18:43.078170+0000","last_fullsized":"2024-04-14T15:18:43.078170+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'8","last_scrub_stamp":"2024-04-14T15:17:50.905608+0000","last_deep_scrub":"233'8","last_deep_scrub_stamp":"2024-04-14T15:15:01.295800+0000","last_clean_scrub_stamp":"2024-04-14T15:17:50.905608+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:14.140648+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"85'16","reported_seq":802,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.073629+0000","last_change":"2024-04-14T15:16:53.143329+0000","last_active":"2024-04-14T15:18:43.073629+0000","last_peered":"2024-04-14T15:18:43.073629+0000","last_clean":"2024-04-14T15:18:43.073629+0000","last_became_active":"2024-04-14T15:09:45.028212+0000","last_became_peered":"2024-04-14T15:09:45.028212+0000","last_unstale":"2024-04-14T15:18:43.073629+0000","last_undegraded":"2024-04-14T15:18:43.073629+0000","last_fullsized":"2024-04-14T15:18:43.073629+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-14T15:16:53.143239+0000","last_deep_scrub":"85'16","last_deep_scrub_stamp":"2024-04-14T15:10:47.456169+0000","last_clean_scrub_stamp":"2024-04-14T15:16:53.143239+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:47.435977+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":783,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.075345+0000","last_change":"2024-04-14T15:17:39.465617+0000","last_active":"2024-04-14T15:18:43.075345+0000","last_peered":"2024-04-14T15:18:43.075345+0000","last_clean":"2024-04-14T15:18:43.075345+0000","last_became_active":"2024-04-14T15:09:43.018601+0000","last_became_peered":"2024-04-14T15:09:43.018601+0000","last_unstale":"2024-04-14T15:18:43.075345+0000","last_undegraded":"2024-04-14T15:18:43.075345+0000","last_fullsized":"2024-04-14T15:18:43.075345+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:39.465475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:39.465475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:44.272999+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":333,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.074317+0000","last_change":"2024-04-14T15:16:54.187971+0000","last_active":"2024-04-14T15:18:43.074317+0000","last_peered":"2024-04-14T15:18:43.074317+0000","last_clean":"2024-04-14T15:18:43.074317+0000","last_became_active":"2024-04-14T15:15:44.886834+0000","last_became_peered":"2024-04-14T15:15:44.886834+0000","last_unstale":"2024-04-14T15:18:43.074317+0000","last_undegraded":"2024-04-14T15:18:43.074317+0000","last_fullsized":"2024-04-14T15:18:43.074317+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:54.187888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:54.187888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:47.435889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":445,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.082281+0000","last_change":"2024-04-14T15:17:57.214134+0000","last_active":"2024-04-14T15:18:43.082281+0000","last_peered":"2024-04-14T15:18:43.082281+0000","last_clean":"2024-04-14T15:18:43.082281+0000","last_became_active":"2024-04-14T15:15:21.888587+0000","last_became_peered":"2024-04-14T15:15:21.888587+0000","last_unstale":"2024-04-14T15:18:43.082281+0000","last_undegraded":"2024-04-14T15:18:43.082281+0000","last_fullsized":"2024-04-14T15:18:43.082281+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:57.214046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:57.214046+0000","last_clean_scrub_stamp":"2024-04-14T15:17:57.214046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:03.985520+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"91'130","reported_seq":840,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.082048+0000","last_change":"2024-04-14T15:18:40.918682+0000","last_active":"2024-04-14T15:18:43.082048+0000","last_peered":"2024-04-14T15:18:43.082048+0000","last_clean":"2024-04-14T15:18:43.082048+0000","last_became_active":"2024-04-14T15:18:40.917537+0000","last_became_peered":"2024-04-14T15:18:40.917537+0000","last_unstale":"2024-04-14T15:18:43.082048+0000","last_undegraded":"2024-04-14T15:18:43.082048+0000","last_fullsized":"2024-04-14T15:18:43.082048+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:16:41.944267+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:16:41.944267+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:47.362639+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109226,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":679,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.073542+0000","last_change":"2024-04-14T15:16:56.214275+0000","last_active":"2024-04-14T15:18:43.073542+0000","last_peered":"2024-04-14T15:18:43.073542+0000","last_clean":"2024-04-14T15:18:43.073542+0000","last_became_active":"2024-04-14T15:12:56.427276+0000","last_became_peered":"2024-04-14T15:12:56.427276+0000","last_unstale":"2024-04-14T15:18:43.073542+0000","last_undegraded":"2024-04-14T15:18:43.073542+0000","last_fullsized":"2024-04-14T15:18:43.073542+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:56.214178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:16:56.214178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:47.435943+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"255'10","reported_seq":527,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.077235+0000","last_change":"2024-04-14T15:16:17.868542+0000","last_active":"2024-04-14T15:18:43.077235+0000","last_peered":"2024-04-14T15:18:43.077235+0000","last_clean":"2024-04-14T15:18:43.077235+0000","last_became_active":"2024-04-14T15:15:21.873747+0000","last_became_peered":"2024-04-14T15:15:21.873747+0000","last_unstale":"2024-04-14T15:18:43.077235+0000","last_undegraded":"2024-04-14T15:18:43.077235+0000","last_fullsized":"2024-04-14T15:18:43.077235+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'10","last_scrub_stamp":"2024-04-14T15:16:17.868428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:17.868428+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"72'14","reported_seq":800,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.072226+0000","last_change":"2024-04-14T15:18:22.545734+0000","last_active":"2024-04-14T15:18:43.072226+0000","last_peered":"2024-04-14T15:18:43.072226+0000","last_clean":"2024-04-14T15:18:43.072226+0000","last_became_active":"2024-04-14T15:09:45.025673+0000","last_became_peered":"2024-04-14T15:09:45.025673+0000","last_unstale":"2024-04-14T15:18:43.072226+0000","last_undegraded":"2024-04-14T15:18:43.072226+0000","last_fullsized":"2024-04-14T15:18:43.072226+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'14","last_scrub_stamp":"2024-04-14T15:18:22.545646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:18:22.545646+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:33.421913+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":784,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.080895+0000","last_change":"2024-04-14T15:18:26.824852+0000","last_active":"2024-04-14T15:18:43.080895+0000","last_peered":"2024-04-14T15:18:43.080895+0000","last_clean":"2024-04-14T15:18:43.080895+0000","last_became_active":"2024-04-14T15:09:43.020503+0000","last_became_peered":"2024-04-14T15:09:43.020503+0000","last_unstale":"2024-04-14T15:18:43.080895+0000","last_undegraded":"2024-04-14T15:18:43.080895+0000","last_fullsized":"2024-04-14T15:18:43.080895+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:26.824773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:18:26.824773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:26.903997+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":763,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.076643+0000","last_change":"2024-04-14T15:16:45.705837+0000","last_active":"2024-04-14T15:18:43.076643+0000","last_peered":"2024-04-14T15:18:43.076643+0000","last_clean":"2024-04-14T15:18:43.076643+0000","last_became_active":"2024-04-14T15:09:49.129610+0000","last_became_peered":"2024-04-14T15:09:49.129610+0000","last_unstale":"2024-04-14T15:18:43.076643+0000","last_undegraded":"2024-04-14T15:18:43.076643+0000","last_fullsized":"2024-04-14T15:18:43.076643+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:45.705761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:45.705761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"91'130","reported_seq":842,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.073547+0000","last_change":"2024-04-14T15:18:40.911989+0000","last_active":"2024-04-14T15:18:43.073547+0000","last_peered":"2024-04-14T15:18:43.073547+0000","last_clean":"2024-04-14T15:18:43.073547+0000","last_became_active":"2024-04-14T15:18:40.911665+0000","last_became_peered":"2024-04-14T15:18:40.911665+0000","last_unstale":"2024-04-14T15:18:43.073547+0000","last_undegraded":"2024-04-14T15:18:43.073547+0000","last_fullsized":"2024-04-14T15:18:43.073547+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:16:41.193422+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:16:41.193422+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:47.435705+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":679,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.078067+0000","last_change":"2024-04-14T15:17:47.553179+0000","last_active":"2024-04-14T15:18:43.078067+0000","last_peered":"2024-04-14T15:18:43.078067+0000","last_clean":"2024-04-14T15:18:43.078067+0000","last_became_active":"2024-04-14T15:12:56.355254+0000","last_became_peered":"2024-04-14T15:12:56.355254+0000","last_unstale":"2024-04-14T15:18:43.078067+0000","last_undegraded":"2024-04-14T15:18:43.078067+0000","last_fullsized":"2024-04-14T15:18:43.078067+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:47.553055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:17:47.553055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:58.344778+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":775,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.072483+0000","last_change":"2024-04-14T15:18:39.394736+0000","last_active":"2024-04-14T15:18:43.072483+0000","last_peered":"2024-04-14T15:18:43.072483+0000","last_clean":"2024-04-14T15:18:43.072483+0000","last_became_active":"2024-04-14T15:09:47.044610+0000","last_became_peered":"2024-04-14T15:09:47.044610+0000","last_unstale":"2024-04-14T15:18:43.072483+0000","last_undegraded":"2024-04-14T15:18:43.072483+0000","last_fullsized":"2024-04-14T15:18:43.072483+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:39.394655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:39.394655+0000","last_clean_scrub_stamp":"2024-04-14T15:18:39.394655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:52.787637+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"72'18","reported_seq":486,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.077062+0000","last_change":"2024-04-14T15:16:15.913377+0000","last_active":"2024-04-14T15:18:43.077062+0000","last_peered":"2024-04-14T15:18:43.077062+0000","last_clean":"2024-04-14T15:18:43.077062+0000","last_became_active":"2024-04-14T15:15:21.880565+0000","last_became_peered":"2024-04-14T15:15:21.880565+0000","last_unstale":"2024-04-14T15:18:43.077062+0000","last_undegraded":"2024-04-14T15:18:43.077062+0000","last_fullsized":"2024-04-14T15:18:43.077062+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'18","last_scrub_stamp":"2024-04-14T15:16:15.913273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:16:15.913273+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'2","reported_seq":779,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.076172+0000","last_change":"2024-04-14T15:17:48.850178+0000","last_active":"2024-04-14T15:18:43.076172+0000","last_peered":"2024-04-14T15:18:43.076172+0000","last_clean":"2024-04-14T15:18:43.076172+0000","last_became_active":"2024-04-14T15:09:43.018478+0000","last_became_peered":"2024-04-14T15:09:43.018478+0000","last_unstale":"2024-04-14T15:18:43.076172+0000","last_undegraded":"2024-04-14T15:18:43.076172+0000","last_fullsized":"2024-04-14T15:18:43.076172+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'2","last_scrub_stamp":"2024-04-14T15:17:48.850069+0000","last_deep_scrub":"91'2","last_deep_scrub_stamp":"2024-04-14T15:17:48.850069+0000","last_clean_scrub_stamp":"2024-04-14T15:17:48.850069+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:12.966475+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":763,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.072854+0000","last_change":"2024-04-14T15:16:14.017818+0000","last_active":"2024-04-14T15:18:43.072854+0000","last_peered":"2024-04-14T15:18:43.072854+0000","last_clean":"2024-04-14T15:18:43.072854+0000","last_became_active":"2024-04-14T15:09:49.065684+0000","last_became_peered":"2024-04-14T15:09:49.065684+0000","last_unstale":"2024-04-14T15:18:43.072854+0000","last_undegraded":"2024-04-14T15:18:43.072854+0000","last_fullsized":"2024-04-14T15:18:43.072854+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:14.017729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:14.017729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:47.435724+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"91'130","reported_seq":850,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.080608+0000","last_change":"2024-04-14T15:18:40.915335+0000","last_active":"2024-04-14T15:18:43.080608+0000","last_peered":"2024-04-14T15:18:43.080608+0000","last_clean":"2024-04-14T15:18:43.080608+0000","last_became_active":"2024-04-14T15:18:40.914994+0000","last_became_peered":"2024-04-14T15:18:40.914994+0000","last_unstale":"2024-04-14T15:18:43.080608+0000","last_undegraded":"2024-04-14T15:18:43.080608+0000","last_fullsized":"2024-04-14T15:18:43.080608+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:17:47.885308+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:17:47.885308+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:47.797403+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":681,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.074431+0000","last_change":"2024-04-14T15:18:35.004058+0000","last_active":"2024-04-14T15:18:43.074431+0000","last_peered":"2024-04-14T15:18:43.074431+0000","last_clean":"2024-04-14T15:18:43.074431+0000","last_became_active":"2024-04-14T15:12:56.402023+0000","last_became_peered":"2024-04-14T15:12:56.402023+0000","last_unstale":"2024-04-14T15:18:43.074431+0000","last_undegraded":"2024-04-14T15:18:43.074431+0000","last_fullsized":"2024-04-14T15:18:43.074431+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:35.003968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:35.003968+0000","last_clean_scrub_stamp":"2024-04-14T15:18:35.003968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:53.392985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":776,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.079597+0000","last_change":"2024-04-14T15:17:39.682323+0000","last_active":"2024-04-14T15:18:43.079597+0000","last_peered":"2024-04-14T15:18:43.079597+0000","last_clean":"2024-04-14T15:18:43.079597+0000","last_became_active":"2024-04-14T15:09:47.048427+0000","last_became_peered":"2024-04-14T15:09:47.048427+0000","last_unstale":"2024-04-14T15:18:43.079597+0000","last_undegraded":"2024-04-14T15:18:43.079597+0000","last_fullsized":"2024-04-14T15:18:43.079597+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:39.682203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:17:39.682203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:58.685184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"102'27","reported_seq":825,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.080210+0000","last_change":"2024-04-14T15:17:46.924432+0000","last_active":"2024-04-14T15:18:43.080210+0000","last_peered":"2024-04-14T15:18:43.080210+0000","last_clean":"2024-04-14T15:18:43.080210+0000","last_became_active":"2024-04-14T15:09:45.179182+0000","last_became_peered":"2024-04-14T15:09:45.179182+0000","last_unstale":"2024-04-14T15:18:43.080210+0000","last_undegraded":"2024-04-14T15:18:43.080210+0000","last_fullsized":"2024-04-14T15:18:43.080210+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'27","last_scrub_stamp":"2024-04-14T15:17:46.924330+0000","last_deep_scrub":"102'27","last_deep_scrub_stamp":"2024-04-14T15:17:46.924330+0000","last_clean_scrub_stamp":"2024-04-14T15:17:46.924330+0000","objects_scrubbed":8,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:54.826490+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":775,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.076863+0000","last_change":"2024-04-14T15:16:18.774542+0000","last_active":"2024-04-14T15:18:43.076863+0000","last_peered":"2024-04-14T15:18:43.076863+0000","last_clean":"2024-04-14T15:18:43.076863+0000","last_became_active":"2024-04-14T15:09:43.021006+0000","last_became_peered":"2024-04-14T15:09:43.021006+0000","last_unstale":"2024-04-14T15:18:43.076863+0000","last_undegraded":"2024-04-14T15:18:43.076863+0000","last_fullsized":"2024-04-14T15:18:43.076863+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:18.774463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:18.774463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":765,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.073121+0000","last_change":"2024-04-14T15:16:56.206766+0000","last_active":"2024-04-14T15:18:43.073121+0000","last_peered":"2024-04-14T15:18:43.073121+0000","last_clean":"2024-04-14T15:18:43.073121+0000","last_became_active":"2024-04-14T15:09:49.128136+0000","last_became_peered":"2024-04-14T15:09:49.128136+0000","last_unstale":"2024-04-14T15:18:43.073121+0000","last_undegraded":"2024-04-14T15:18:43.073121+0000","last_fullsized":"2024-04-14T15:18:43.073121+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:56.206649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:56.206649+0000","last_clean_scrub_stamp":"2024-04-14T15:16:56.206649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"91'130","reported_seq":844,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.076756+0000","last_change":"2024-04-14T15:18:40.910893+0000","last_active":"2024-04-14T15:18:43.076756+0000","last_peered":"2024-04-14T15:18:43.076756+0000","last_clean":"2024-04-14T15:18:43.076756+0000","last_became_active":"2024-04-14T15:18:40.910133+0000","last_became_peered":"2024-04-14T15:18:40.910133+0000","last_unstale":"2024-04-14T15:18:43.076756+0000","last_undegraded":"2024-04-14T15:18:43.076756+0000","last_fullsized":"2024-04-14T15:18:43.076756+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:17:49.890445+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:17:49.890445+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:47.129016+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":385,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.079357+0000","last_change":"2024-04-14T15:18:38.447429+0000","last_active":"2024-04-14T15:18:43.079357+0000","last_peered":"2024-04-14T15:18:43.079357+0000","last_clean":"2024-04-14T15:18:43.079357+0000","last_became_active":"2024-04-14T15:15:21.887769+0000","last_became_peered":"2024-04-14T15:15:21.887769+0000","last_unstale":"2024-04-14T15:18:43.079357+0000","last_undegraded":"2024-04-14T15:18:43.079357+0000","last_fullsized":"2024-04-14T15:18:43.079357+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:38.447351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:18:38.447351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:45.938002+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":335,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.076055+0000","last_change":"2024-04-14T15:17:47.871801+0000","last_active":"2024-04-14T15:18:43.076055+0000","last_peered":"2024-04-14T15:18:43.076055+0000","last_clean":"2024-04-14T15:18:43.076055+0000","last_became_active":"2024-04-14T15:15:44.883694+0000","last_became_peered":"2024-04-14T15:15:44.883694+0000","last_unstale":"2024-04-14T15:18:43.076055+0000","last_undegraded":"2024-04-14T15:18:43.076055+0000","last_fullsized":"2024-04-14T15:18:43.076055+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:47.871722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:17:47.871722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:15.357681+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"182'18","reported_seq":805,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.080566+0000","last_change":"2024-04-14T15:17:44.919445+0000","last_active":"2024-04-14T15:18:43.080566+0000","last_peered":"2024-04-14T15:18:43.080566+0000","last_clean":"2024-04-14T15:18:43.080566+0000","last_became_active":"2024-04-14T15:09:45.037231+0000","last_became_peered":"2024-04-14T15:09:45.037231+0000","last_unstale":"2024-04-14T15:18:43.080566+0000","last_undegraded":"2024-04-14T15:18:43.080566+0000","last_fullsized":"2024-04-14T15:18:43.080566+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'18","last_scrub_stamp":"2024-04-14T15:17:44.919358+0000","last_deep_scrub":"182'18","last_deep_scrub_stamp":"2024-04-14T15:14:10.383906+0000","last_clean_scrub_stamp":"2024-04-14T15:17:44.919358+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:48.791608+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":783,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.075537+0000","last_change":"2024-04-14T15:17:46.542125+0000","last_active":"2024-04-14T15:18:43.075537+0000","last_peered":"2024-04-14T15:18:43.075537+0000","last_clean":"2024-04-14T15:18:43.075537+0000","last_became_active":"2024-04-14T15:09:43.025095+0000","last_became_peered":"2024-04-14T15:09:43.025095+0000","last_unstale":"2024-04-14T15:18:43.075537+0000","last_undegraded":"2024-04-14T15:18:43.075537+0000","last_fullsized":"2024-04-14T15:18:43.075537+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:46.542030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:46.542030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:59.318501+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":792,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.082541+0000","last_change":"2024-04-14T15:17:48.866222+0000","last_active":"2024-04-14T15:18:43.082541+0000","last_peered":"2024-04-14T15:18:43.082541+0000","last_clean":"2024-04-14T15:18:43.082541+0000","last_became_active":"2024-04-14T15:09:43.018509+0000","last_became_peered":"2024-04-14T15:09:43.018509+0000","last_unstale":"2024-04-14T15:18:43.082541+0000","last_undegraded":"2024-04-14T15:18:43.082541+0000","last_fullsized":"2024-04-14T15:18:43.082541+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:48.866146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:48.866146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:57.582404+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0018592680000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"355'2","reported_seq":767,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.074719+0000","last_change":"2024-04-14T15:17:03.127190+0000","last_active":"2024-04-14T15:18:43.074719+0000","last_peered":"2024-04-14T15:18:43.074719+0000","last_clean":"2024-04-14T15:18:43.074719+0000","last_became_active":"2024-04-14T15:09:49.065243+0000","last_became_peered":"2024-04-14T15:09:49.065243+0000","last_unstale":"2024-04-14T15:18:43.074719+0000","last_undegraded":"2024-04-14T15:18:43.074719+0000","last_fullsized":"2024-04-14T15:18:43.074719+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'2","last_scrub_stamp":"2024-04-14T15:17:03.127112+0000","last_deep_scrub":"355'2","last_deep_scrub_stamp":"2024-04-14T15:17:03.127112+0000","last_clean_scrub_stamp":"2024-04-14T15:17:03.127112+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":377,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.079706+0000","last_change":"2024-04-14T15:18:32.532077+0000","last_active":"2024-04-14T15:18:43.079706+0000","last_peered":"2024-04-14T15:18:43.079706+0000","last_clean":"2024-04-14T15:18:43.079706+0000","last_became_active":"2024-04-14T15:15:21.890115+0000","last_became_peered":"2024-04-14T15:15:21.890115+0000","last_unstale":"2024-04-14T15:18:43.079706+0000","last_undegraded":"2024-04-14T15:18:43.079706+0000","last_fullsized":"2024-04-14T15:18:43.079706+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:32.531993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:18:32.531993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:40.127160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":767,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.076702+0000","last_change":"2024-04-14T15:16:51.617717+0000","last_active":"2024-04-14T15:18:43.076702+0000","last_peered":"2024-04-14T15:18:43.076702+0000","last_clean":"2024-04-14T15:18:43.076702+0000","last_became_active":"2024-04-14T15:09:47.047201+0000","last_became_peered":"2024-04-14T15:09:47.047201+0000","last_unstale":"2024-04-14T15:18:43.076702+0000","last_undegraded":"2024-04-14T15:18:43.076702+0000","last_fullsized":"2024-04-14T15:18:43.076702+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:51.617615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:51.617615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"90'24","reported_seq":911,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.080445+0000","last_change":"2024-04-14T15:16:19.842719+0000","last_active":"2024-04-14T15:18:43.080445+0000","last_peered":"2024-04-14T15:18:43.080445+0000","last_clean":"2024-04-14T15:18:43.080445+0000","last_became_active":"2024-04-14T15:09:45.179978+0000","last_became_peered":"2024-04-14T15:09:45.179978+0000","last_unstale":"2024-04-14T15:18:43.080445+0000","last_undegraded":"2024-04-14T15:18:43.080445+0000","last_fullsized":"2024-04-14T15:18:43.080445+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'24","last_scrub_stamp":"2024-04-14T15:16:19.842614+0000","last_deep_scrub":"90'24","last_deep_scrub_stamp":"2024-04-14T15:11:25.120692+0000","last_clean_scrub_stamp":"2024-04-14T15:16:19.842614+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:47.797443+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"91'130","reported_seq":1053,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.075441+0000","last_change":"2024-04-14T15:18:40.911518+0000","last_active":"2024-04-14T15:18:43.075441+0000","last_peered":"2024-04-14T15:18:43.075441+0000","last_clean":"2024-04-14T15:18:43.075441+0000","last_became_active":"2024-04-14T15:18:40.911206+0000","last_became_peered":"2024-04-14T15:18:40.911206+0000","last_unstale":"2024-04-14T15:18:43.075441+0000","last_undegraded":"2024-04-14T15:18:43.075441+0000","last_fullsized":"2024-04-14T15:18:43.075441+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:16:43.730442+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:16:43.730442+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:47.727511+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":342,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-04-14T15:18:44.088518+0000","last_change":"2024-04-14T15:17:35.705777+0000","last_active":"2024-04-14T15:18:44.088518+0000","last_peered":"2024-04-14T15:18:44.088518+0000","last_clean":"2024-04-14T15:18:44.088518+0000","last_became_active":"2024-04-14T15:15:44.885315+0000","last_became_peered":"2024-04-14T15:15:44.885315+0000","last_unstale":"2024-04-14T15:18:44.088518+0000","last_undegraded":"2024-04-14T15:18:44.088518+0000","last_fullsized":"2024-04-14T15:18:44.088518+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:35.705645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:47.627305+0000","last_clean_scrub_stamp":"2024-04-14T15:17:35.705645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:49.306972+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"69'2","reported_seq":467,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.079791+0000","last_change":"2024-04-14T15:16:53.972591+0000","last_active":"2024-04-14T15:18:43.079791+0000","last_peered":"2024-04-14T15:18:43.079791+0000","last_clean":"2024-04-14T15:18:43.079791+0000","last_became_active":"2024-04-14T15:15:21.889851+0000","last_became_peered":"2024-04-14T15:15:21.889851+0000","last_unstale":"2024-04-14T15:18:43.079791+0000","last_undegraded":"2024-04-14T15:18:43.079791+0000","last_fullsized":"2024-04-14T15:18:43.079791+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-04-14T15:16:53.972502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:53.972502+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0058046169999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":332,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.083286+0000","last_change":"2024-04-14T15:16:57.935611+0000","last_active":"2024-04-14T15:18:43.083286+0000","last_peered":"2024-04-14T15:18:43.083286+0000","last_clean":"2024-04-14T15:18:43.083286+0000","last_became_active":"2024-04-14T15:15:44.887934+0000","last_became_peered":"2024-04-14T15:15:44.887934+0000","last_unstale":"2024-04-14T15:18:43.083286+0000","last_undegraded":"2024-04-14T15:18:43.083286+0000","last_fullsized":"2024-04-14T15:18:43.083286+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:57.935436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:57.935436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"91'130","reported_seq":1061,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.075808+0000","last_change":"2024-04-14T15:18:40.912551+0000","last_active":"2024-04-14T15:18:43.075808+0000","last_peered":"2024-04-14T15:18:43.075808+0000","last_clean":"2024-04-14T15:18:43.075808+0000","last_became_active":"2024-04-14T15:18:40.912277+0000","last_became_peered":"2024-04-14T15:18:40.912277+0000","last_unstale":"2024-04-14T15:18:43.075808+0000","last_undegraded":"2024-04-14T15:18:43.075808+0000","last_fullsized":"2024-04-14T15:18:43.075808+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:17:41.509215+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:17:41.509215+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:47.727551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":377,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.076853+0000","last_change":"2024-04-14T15:18:26.438896+0000","last_active":"2024-04-14T15:18:43.076853+0000","last_peered":"2024-04-14T15:18:43.076853+0000","last_clean":"2024-04-14T15:18:43.076853+0000","last_became_active":"2024-04-14T15:15:21.889970+0000","last_became_peered":"2024-04-14T15:15:21.889970+0000","last_unstale":"2024-04-14T15:18:43.076853+0000","last_undegraded":"2024-04-14T15:18:43.076853+0000","last_fullsized":"2024-04-14T15:18:43.076853+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:26.438820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:18:26.438820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:45.768277+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":776,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-04-14T15:18:44.087772+0000","last_change":"2024-04-14T15:17:01.653342+0000","last_active":"2024-04-14T15:18:44.087772+0000","last_peered":"2024-04-14T15:18:44.087772+0000","last_clean":"2024-04-14T15:18:44.087772+0000","last_became_active":"2024-04-14T15:09:47.049457+0000","last_became_peered":"2024-04-14T15:09:47.049457+0000","last_unstale":"2024-04-14T15:18:44.087772+0000","last_undegraded":"2024-04-14T15:18:44.087772+0000","last_fullsized":"2024-04-14T15:18:44.087772+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:01.653271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:17:01.653271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"308'18","reported_seq":805,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.075900+0000","last_change":"2024-04-14T15:16:16.054128+0000","last_active":"2024-04-14T15:18:43.075900+0000","last_peered":"2024-04-14T15:18:43.075900+0000","last_clean":"2024-04-14T15:18:43.075900+0000","last_became_active":"2024-04-14T15:09:45.033883+0000","last_became_peered":"2024-04-14T15:09:45.033883+0000","last_unstale":"2024-04-14T15:18:43.075900+0000","last_undegraded":"2024-04-14T15:18:43.075900+0000","last_fullsized":"2024-04-14T15:18:43.075900+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"308'18","last_scrub_stamp":"2024-04-14T15:16:16.054032+0000","last_deep_scrub":"308'18","last_deep_scrub_stamp":"2024-04-14T15:16:16.054032+0000","last_clean_scrub_stamp":"2024-04-14T15:16:16.054032+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":784,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-04-14T15:18:44.087014+0000","last_change":"2024-04-14T15:16:43.686142+0000","last_active":"2024-04-14T15:18:44.087014+0000","last_peered":"2024-04-14T15:18:44.087014+0000","last_clean":"2024-04-14T15:18:44.087014+0000","last_became_active":"2024-04-14T15:09:43.021188+0000","last_became_peered":"2024-04-14T15:09:43.021188+0000","last_unstale":"2024-04-14T15:18:44.087014+0000","last_undegraded":"2024-04-14T15:18:44.087014+0000","last_fullsized":"2024-04-14T15:18:44.087014+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:43.686033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:43.686033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"73'5","reported_seq":1329,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.074740+0000","last_change":"2024-04-14T15:18:26.955802+0000","last_active":"2024-04-14T15:18:43.074740+0000","last_peered":"2024-04-14T15:18:43.074740+0000","last_clean":"2024-04-14T15:18:43.074740+0000","last_became_active":"2024-04-14T15:09:43.016544+0000","last_became_peered":"2024-04-14T15:09:43.016544+0000","last_unstale":"2024-04-14T15:18:43.074740+0000","last_undegraded":"2024-04-14T15:18:43.074740+0000","last_fullsized":"2024-04-14T15:18:43.074740+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-04-14T15:18:26.955711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:18:26.955711+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:54.355074+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00056542099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":534,"num_read_kb":529,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":763,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.076927+0000","last_change":"2024-04-14T15:16:20.827681+0000","last_active":"2024-04-14T15:18:43.076927+0000","last_peered":"2024-04-14T15:18:43.076927+0000","last_clean":"2024-04-14T15:18:43.076927+0000","last_became_active":"2024-04-14T15:09:49.130307+0000","last_became_peered":"2024-04-14T15:09:49.130307+0000","last_unstale":"2024-04-14T15:18:43.076927+0000","last_undegraded":"2024-04-14T15:18:43.076927+0000","last_fullsized":"2024-04-14T15:18:43.076927+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:20.827589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:20.827589+0000","last_clean_scrub_stamp":"2024-04-14T15:16:20.827589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"91'130","reported_seq":675,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.081720+0000","last_change":"2024-04-14T15:18:40.916009+0000","last_active":"2024-04-14T15:18:43.081720+0000","last_peered":"2024-04-14T15:18:43.081720+0000","last_clean":"2024-04-14T15:18:43.081720+0000","last_became_active":"2024-04-14T15:18:40.913215+0000","last_became_peered":"2024-04-14T15:18:40.913215+0000","last_unstale":"2024-04-14T15:18:43.081720+0000","last_undegraded":"2024-04-14T15:18:43.081720+0000","last_fullsized":"2024-04-14T15:18:43.081720+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:18:33.054034+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:18:33.054034+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:48.004498+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":377,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.077923+0000","last_change":"2024-04-14T15:15:32.839576+0000","last_active":"2024-04-14T15:18:43.077923+0000","last_peered":"2024-04-14T15:18:43.077923+0000","last_clean":"2024-04-14T15:18:43.077923+0000","last_became_active":"2024-04-14T15:15:21.883717+0000","last_became_peered":"2024-04-14T15:15:21.883717+0000","last_unstale":"2024-04-14T15:18:43.077923+0000","last_undegraded":"2024-04-14T15:18:43.077923+0000","last_fullsized":"2024-04-14T15:18:43.077923+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:15:32.839491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:15:32.839491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":769,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.079333+0000","last_change":"2024-04-14T15:16:53.731699+0000","last_active":"2024-04-14T15:18:43.079333+0000","last_peered":"2024-04-14T15:18:43.079333+0000","last_clean":"2024-04-14T15:18:43.079333+0000","last_became_active":"2024-04-14T15:09:47.040185+0000","last_became_peered":"2024-04-14T15:09:47.040185+0000","last_unstale":"2024-04-14T15:18:43.079333+0000","last_undegraded":"2024-04-14T15:18:43.079333+0000","last_fullsized":"2024-04-14T15:18:43.079333+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:53.731610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:53.731610+0000","last_clean_scrub_stamp":"2024-04-14T15:16:53.731610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:46.956899+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"72'28","reported_seq":930,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.074378+0000","last_change":"2024-04-14T15:16:43.036263+0000","last_active":"2024-04-14T15:18:43.074378+0000","last_peered":"2024-04-14T15:18:43.074378+0000","last_clean":"2024-04-14T15:18:43.074378+0000","last_became_active":"2024-04-14T15:09:45.034168+0000","last_became_peered":"2024-04-14T15:09:45.034168+0000","last_unstale":"2024-04-14T15:18:43.074378+0000","last_undegraded":"2024-04-14T15:18:43.074378+0000","last_fullsized":"2024-04-14T15:18:43.074378+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'28","last_scrub_stamp":"2024-04-14T15:16:43.036159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:16:43.036159+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:48.097584+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":335,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.074284+0000","last_change":"2024-04-14T15:16:19.101437+0000","last_active":"2024-04-14T15:18:43.074284+0000","last_peered":"2024-04-14T15:18:43.074284+0000","last_clean":"2024-04-14T15:18:43.074284+0000","last_became_active":"2024-04-14T15:15:44.885315+0000","last_became_peered":"2024-04-14T15:15:44.885315+0000","last_unstale":"2024-04-14T15:18:43.074284+0000","last_undegraded":"2024-04-14T15:18:43.074284+0000","last_fullsized":"2024-04-14T15:18:43.074284+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:19.101368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:19.101368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":437,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.082669+0000","last_change":"2024-04-14T15:17:44.084142+0000","last_active":"2024-04-14T15:18:43.082669+0000","last_peered":"2024-04-14T15:18:43.082669+0000","last_clean":"2024-04-14T15:18:43.082669+0000","last_became_active":"2024-04-14T15:15:21.874742+0000","last_became_peered":"2024-04-14T15:15:21.874742+0000","last_unstale":"2024-04-14T15:18:43.082669+0000","last_undegraded":"2024-04-14T15:18:43.082669+0000","last_fullsized":"2024-04-14T15:18:43.082669+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:44.084010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:44.084010+0000","last_clean_scrub_stamp":"2024-04-14T15:17:44.084010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:03.733939+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"91'130","reported_seq":850,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.080499+0000","last_change":"2024-04-14T15:18:40.915290+0000","last_active":"2024-04-14T15:18:43.080499+0000","last_peered":"2024-04-14T15:18:43.080499+0000","last_clean":"2024-04-14T15:18:43.080499+0000","last_became_active":"2024-04-14T15:18:40.914762+0000","last_became_peered":"2024-04-14T15:18:40.914762+0000","last_unstale":"2024-04-14T15:18:43.080499+0000","last_undegraded":"2024-04-14T15:18:43.080499+0000","last_fullsized":"2024-04-14T15:18:43.080499+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:17:40.725873+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:17:40.725873+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:47.797424+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":680,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-04-14T15:18:44.087082+0000","last_change":"2024-04-14T15:17:00.656174+0000","last_active":"2024-04-14T15:18:44.087082+0000","last_peered":"2024-04-14T15:18:44.087082+0000","last_clean":"2024-04-14T15:18:44.087082+0000","last_became_active":"2024-04-14T15:12:56.350703+0000","last_became_peered":"2024-04-14T15:12:56.350703+0000","last_unstale":"2024-04-14T15:18:44.087082+0000","last_undegraded":"2024-04-14T15:18:44.087082+0000","last_fullsized":"2024-04-14T15:18:44.087082+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:00.656085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:17:00.656085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"72'8","reported_seq":894,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.079935+0000","last_change":"2024-04-14T15:17:04.695161+0000","last_active":"2024-04-14T15:18:43.079935+0000","last_peered":"2024-04-14T15:18:43.079935+0000","last_clean":"2024-04-14T15:18:43.079935+0000","last_became_active":"2024-04-14T15:09:47.047699+0000","last_became_peered":"2024-04-14T15:09:47.047699+0000","last_unstale":"2024-04-14T15:18:43.079935+0000","last_undegraded":"2024-04-14T15:18:43.079935+0000","last_fullsized":"2024-04-14T15:18:43.079935+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:17:04.695079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:17:04.695079+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:46.956963+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"72'10","reported_seq":774,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-04-14T15:18:44.088600+0000","last_change":"2024-04-14T15:18:38.870798+0000","last_active":"2024-04-14T15:18:44.088600+0000","last_peered":"2024-04-14T15:18:44.088600+0000","last_clean":"2024-04-14T15:18:44.088600+0000","last_became_active":"2024-04-14T15:10:07.764042+0000","last_became_peered":"2024-04-14T15:10:07.764042+0000","last_unstale":"2024-04-14T15:18:44.088600+0000","last_undegraded":"2024-04-14T15:18:44.088600+0000","last_fullsized":"2024-04-14T15:18:44.088600+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-04-14T15:18:38.870710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:18:38.870710+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:50.575761+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":777,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.074196+0000","last_change":"2024-04-14T15:17:40.971492+0000","last_active":"2024-04-14T15:18:43.074196+0000","last_peered":"2024-04-14T15:18:43.074196+0000","last_clean":"2024-04-14T15:18:43.074196+0000","last_became_active":"2024-04-14T15:09:43.022597+0000","last_became_peered":"2024-04-14T15:09:43.022597+0000","last_unstale":"2024-04-14T15:18:43.074196+0000","last_undegraded":"2024-04-14T15:18:43.074196+0000","last_fullsized":"2024-04-14T15:18:43.074196+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:40.971412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:40.971412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:08.233588+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":765,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.079255+0000","last_change":"2024-04-14T15:16:17.405931+0000","last_active":"2024-04-14T15:18:43.079255+0000","last_peered":"2024-04-14T15:18:43.079255+0000","last_clean":"2024-04-14T15:18:43.079255+0000","last_became_active":"2024-04-14T15:09:49.066648+0000","last_became_peered":"2024-04-14T15:09:49.066648+0000","last_unstale":"2024-04-14T15:18:43.079255+0000","last_undegraded":"2024-04-14T15:18:43.079255+0000","last_fullsized":"2024-04-14T15:18:43.079255+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:17.405850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:17.405850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:46.956748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.c","version":"91'130","reported_seq":838,"reported_epoch":434,"state":"remapped+peering","last_fresh":"2024-04-14T15:18:43.078105+0000","last_change":"2024-04-14T15:18:43.078105+0000","last_active":"2024-04-14T15:18:42.070721+0000","last_peered":"2024-04-14T15:18:40.920444+0000","last_clean":"2024-04-14T15:18:40.920444+0000","last_became_active":"2024-04-14T15:18:40.919654+0000","last_became_peered":"2024-04-14T15:18:40.919654+0000","last_unstale":"2024-04-14T15:18:43.078105+0000","last_undegraded":"2024-04-14T15:18:43.078105+0000","last_fullsized":"2024-04-14T15:18:43.078105+0000","mapping_epoch":434,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:16:41.944267+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:16:41.944267+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109227,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":681,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.080387+0000","last_change":"2024-04-14T15:18:27.133391+0000","last_active":"2024-04-14T15:18:43.080387+0000","last_peered":"2024-04-14T15:18:43.080387+0000","last_clean":"2024-04-14T15:18:43.080387+0000","last_became_active":"2024-04-14T15:12:56.563074+0000","last_became_peered":"2024-04-14T15:12:56.563074+0000","last_unstale":"2024-04-14T15:18:43.080387+0000","last_undegraded":"2024-04-14T15:18:43.080387+0000","last_fullsized":"2024-04-14T15:18:43.080387+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:27.133310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:18:27.133310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:38.787205+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":776,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-04-14T15:18:44.084518+0000","last_change":"2024-04-14T15:17:56.915660+0000","last_active":"2024-04-14T15:18:44.084518+0000","last_peered":"2024-04-14T15:18:44.084518+0000","last_clean":"2024-04-14T15:18:44.084518+0000","last_became_active":"2024-04-14T15:09:47.049005+0000","last_became_peered":"2024-04-14T15:09:47.049005+0000","last_unstale":"2024-04-14T15:18:44.084518+0000","last_undegraded":"2024-04-14T15:18:44.084518+0000","last_fullsized":"2024-04-14T15:18:44.084518+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:56.915544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:56.915544+0000","last_clean_scrub_stamp":"2024-04-14T15:17:56.915544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:24.542571+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"183'17","reported_seq":807,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.072491+0000","last_change":"2024-04-14T15:16:17.997616+0000","last_active":"2024-04-14T15:18:43.072491+0000","last_peered":"2024-04-14T15:18:43.072491+0000","last_clean":"2024-04-14T15:18:43.072491+0000","last_became_active":"2024-04-14T15:09:45.032892+0000","last_became_peered":"2024-04-14T15:09:45.032892+0000","last_unstale":"2024-04-14T15:18:43.072491+0000","last_undegraded":"2024-04-14T15:18:43.072491+0000","last_fullsized":"2024-04-14T15:18:43.072491+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'17","last_scrub_stamp":"2024-04-14T15:16:17.997529+0000","last_deep_scrub":"183'17","last_deep_scrub_stamp":"2024-04-14T15:14:11.577574+0000","last_clean_scrub_stamp":"2024-04-14T15:16:17.997529+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:47.435824+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"65'3","reported_seq":793,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.074262+0000","last_change":"2024-04-14T15:16:12.030448+0000","last_active":"2024-04-14T15:18:43.074262+0000","last_peered":"2024-04-14T15:18:43.074262+0000","last_clean":"2024-04-14T15:18:43.074262+0000","last_became_active":"2024-04-14T15:09:43.020490+0000","last_became_peered":"2024-04-14T15:09:43.020490+0000","last_unstale":"2024-04-14T15:18:43.074262+0000","last_undegraded":"2024-04-14T15:18:43.074262+0000","last_fullsized":"2024-04-14T15:18:43.074262+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3","last_scrub_stamp":"2024-04-14T15:16:12.030363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:12.030363+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:47.435804+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":528,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":437,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.078712+0000","last_change":"2024-04-14T15:16:58.942591+0000","last_active":"2024-04-14T15:18:43.078712+0000","last_peered":"2024-04-14T15:18:43.078712+0000","last_clean":"2024-04-14T15:18:43.078712+0000","last_became_active":"2024-04-14T15:15:21.876410+0000","last_became_peered":"2024-04-14T15:15:21.876410+0000","last_unstale":"2024-04-14T15:18:43.078712+0000","last_undegraded":"2024-04-14T15:18:43.078712+0000","last_fullsized":"2024-04-14T15:18:43.078712+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:58.942499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:58.942499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"91'130","reported_seq":841,"reported_epoch":434,"state":"remapped+peering","last_fresh":"2024-04-14T15:18:43.076457+0000","last_change":"2024-04-14T15:18:43.076457+0000","last_active":"2024-04-14T15:18:42.067066+0000","last_peered":"2024-04-14T15:18:40.916977+0000","last_clean":"2024-04-14T15:18:40.916977+0000","last_became_active":"2024-04-14T15:18:40.915299+0000","last_became_peered":"2024-04-14T15:18:40.915299+0000","last_unstale":"2024-04-14T15:18:43.076457+0000","last_undegraded":"2024-04-14T15:18:43.076457+0000","last_fullsized":"2024-04-14T15:18:43.076457+0000","mapping_epoch":434,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:17:49.890445+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:17:49.890445+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":681,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.081976+0000","last_change":"2024-04-14T15:16:43.696614+0000","last_active":"2024-04-14T15:18:43.081976+0000","last_peered":"2024-04-14T15:18:43.081976+0000","last_clean":"2024-04-14T15:18:43.081976+0000","last_became_active":"2024-04-14T15:12:56.551177+0000","last_became_peered":"2024-04-14T15:12:56.551177+0000","last_unstale":"2024-04-14T15:18:43.081976+0000","last_undegraded":"2024-04-14T15:18:43.081976+0000","last_fullsized":"2024-04-14T15:18:43.081976+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:43.696536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:16:43.696536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:46.956872+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"347'13","reported_seq":792,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.080658+0000","last_change":"2024-04-14T15:16:55.694703+0000","last_active":"2024-04-14T15:18:43.080658+0000","last_peered":"2024-04-14T15:18:43.080658+0000","last_clean":"2024-04-14T15:18:43.080658+0000","last_became_active":"2024-04-14T15:09:45.180375+0000","last_became_peered":"2024-04-14T15:09:45.180375+0000","last_unstale":"2024-04-14T15:18:43.080658+0000","last_undegraded":"2024-04-14T15:18:43.080658+0000","last_fullsized":"2024-04-14T15:18:43.080658+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"347'13","last_scrub_stamp":"2024-04-14T15:16:55.694599+0000","last_deep_scrub":"347'13","last_deep_scrub_stamp":"2024-04-14T15:16:55.694599+0000","last_clean_scrub_stamp":"2024-04-14T15:16:55.694599+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:46.956931+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":783,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-04-14T15:18:44.084511+0000","last_change":"2024-04-14T15:16:41.677902+0000","last_active":"2024-04-14T15:18:44.084511+0000","last_peered":"2024-04-14T15:18:44.084511+0000","last_clean":"2024-04-14T15:18:44.084511+0000","last_became_active":"2024-04-14T15:09:43.016788+0000","last_became_peered":"2024-04-14T15:09:43.016788+0000","last_unstale":"2024-04-14T15:18:44.084511+0000","last_undegraded":"2024-04-14T15:18:44.084511+0000","last_fullsized":"2024-04-14T15:18:44.084511+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:41.677826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:41.677826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":457,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.083198+0000","last_change":"2024-04-14T15:18:40.385814+0000","last_active":"2024-04-14T15:18:43.083198+0000","last_peered":"2024-04-14T15:18:43.083198+0000","last_clean":"2024-04-14T15:18:43.083198+0000","last_became_active":"2024-04-14T15:15:21.873580+0000","last_became_peered":"2024-04-14T15:15:21.873580+0000","last_unstale":"2024-04-14T15:18:43.083198+0000","last_undegraded":"2024-04-14T15:18:43.083198+0000","last_fullsized":"2024-04-14T15:18:43.083198+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:40.385736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:18:40.385736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:01.230073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":771,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-04-14T15:18:44.087906+0000","last_change":"2024-04-14T15:16:47.671808+0000","last_active":"2024-04-14T15:18:44.087906+0000","last_peered":"2024-04-14T15:18:44.087906+0000","last_clean":"2024-04-14T15:18:44.087906+0000","last_became_active":"2024-04-14T15:09:49.064296+0000","last_became_peered":"2024-04-14T15:09:49.064296+0000","last_unstale":"2024-04-14T15:18:44.087906+0000","last_undegraded":"2024-04-14T15:18:44.087906+0000","last_fullsized":"2024-04-14T15:18:44.087906+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:47.671717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:47.671717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.12","version":"91'130","reported_seq":318,"reported_epoch":434,"state":"remapped+peering","last_fresh":"2024-04-14T15:18:43.079747+0000","last_change":"2024-04-14T15:18:43.079747+0000","last_active":"2024-04-14T15:18:43.079747+0000","last_peered":"2024-04-14T15:11:28.874295+0000","last_clean":"2024-04-14T15:11:28.874295+0000","last_became_active":"2024-04-14T15:09:33.753828+0000","last_became_peered":"2024-04-14T15:09:33.753828+0000","last_unstale":"2024-04-14T15:18:43.079747+0000","last_undegraded":"2024-04-14T15:18:43.079747+0000","last_fullsized":"2024-04-14T15:18:43.079747+0000","mapping_epoch":434,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:17:41.509215+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:17:41.509215+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":14,"num_write":9,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":80505,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":680,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-04-14T15:18:44.084408+0000","last_change":"2024-04-14T15:17:36.728543+0000","last_active":"2024-04-14T15:18:44.084408+0000","last_peered":"2024-04-14T15:18:44.084408+0000","last_clean":"2024-04-14T15:18:44.084408+0000","last_became_active":"2024-04-14T15:12:56.340066+0000","last_became_peered":"2024-04-14T15:12:56.340066+0000","last_unstale":"2024-04-14T15:18:44.084408+0000","last_undegraded":"2024-04-14T15:18:44.084408+0000","last_fullsized":"2024-04-14T15:18:44.084408+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:36.728465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:17:36.728465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:54.134352+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.17","version":"72'6","reported_seq":785,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.083050+0000","last_change":"2024-04-14T15:17:42.019838+0000","last_active":"2024-04-14T15:18:43.083050+0000","last_peered":"2024-04-14T15:18:43.083050+0000","last_clean":"2024-04-14T15:18:43.083050+0000","last_became_active":"2024-04-14T15:09:45.180037+0000","last_became_peered":"2024-04-14T15:09:45.180037+0000","last_unstale":"2024-04-14T15:18:43.083050+0000","last_undegraded":"2024-04-14T15:18:43.083050+0000","last_fullsized":"2024-04-14T15:18:43.083050+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'6","last_scrub_stamp":"2024-04-14T15:17:42.019717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:17:42.019717+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:52.276684+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":783,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.077884+0000","last_change":"2024-04-14T15:16:59.640700+0000","last_active":"2024-04-14T15:18:43.077884+0000","last_peered":"2024-04-14T15:18:43.077884+0000","last_clean":"2024-04-14T15:18:43.077884+0000","last_became_active":"2024-04-14T15:09:43.018987+0000","last_became_peered":"2024-04-14T15:09:43.018987+0000","last_unstale":"2024-04-14T15:18:43.077884+0000","last_undegraded":"2024-04-14T15:18:43.077884+0000","last_fullsized":"2024-04-14T15:18:43.077884+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:59.640565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:59.640565+0000","last_clean_scrub_stamp":"2024-04-14T15:16:59.640565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:47.797529+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":331,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.082564+0000","last_change":"2024-04-14T15:17:47.072247+0000","last_active":"2024-04-14T15:18:43.082564+0000","last_peered":"2024-04-14T15:18:43.082564+0000","last_clean":"2024-04-14T15:18:43.082564+0000","last_became_active":"2024-04-14T15:15:44.889786+0000","last_became_peered":"2024-04-14T15:15:44.889786+0000","last_unstale":"2024-04-14T15:18:43.082564+0000","last_undegraded":"2024-04-14T15:18:43.082564+0000","last_fullsized":"2024-04-14T15:18:43.082564+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:47.072170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:13.735024+0000","last_clean_scrub_stamp":"2024-04-14T15:17:47.072170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:03.533500+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":762,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.077050+0000","last_change":"2024-04-14T15:16:54.720823+0000","last_active":"2024-04-14T15:18:43.077050+0000","last_peered":"2024-04-14T15:18:43.077050+0000","last_clean":"2024-04-14T15:18:43.077050+0000","last_became_active":"2024-04-14T15:09:49.130438+0000","last_became_peered":"2024-04-14T15:09:49.130438+0000","last_unstale":"2024-04-14T15:18:43.077050+0000","last_undegraded":"2024-04-14T15:18:43.077050+0000","last_fullsized":"2024-04-14T15:18:43.077050+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:54.720722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:54.720722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.11","version":"91'130","reported_seq":672,"reported_epoch":434,"state":"remapped+peering","last_fresh":"2024-04-14T15:18:43.081813+0000","last_change":"2024-04-14T15:18:43.081813+0000","last_active":"2024-04-14T15:18:42.076565+0000","last_peered":"2024-04-14T15:18:40.918393+0000","last_clean":"2024-04-14T15:18:40.918393+0000","last_became_active":"2024-04-14T15:18:40.917589+0000","last_became_peered":"2024-04-14T15:18:40.917589+0000","last_unstale":"2024-04-14T15:18:43.081813+0000","last_undegraded":"2024-04-14T15:18:43.081813+0000","last_fullsized":"2024-04-14T15:18:43.081813+0000","mapping_epoch":434,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:18:33.054034+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:18:33.054034+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":679,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.072980+0000","last_change":"2024-04-14T15:17:46.369320+0000","last_active":"2024-04-14T15:18:43.072980+0000","last_peered":"2024-04-14T15:18:43.072980+0000","last_clean":"2024-04-14T15:18:43.072980+0000","last_became_active":"2024-04-14T15:12:56.203577+0000","last_became_peered":"2024-04-14T15:12:56.203577+0000","last_unstale":"2024-04-14T15:18:43.072980+0000","last_undegraded":"2024-04-14T15:18:43.072980+0000","last_fullsized":"2024-04-14T15:18:43.072980+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:46.369231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:46.369231+0000","last_clean_scrub_stamp":"2024-04-14T15:17:46.369231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:57.370694+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"72'10","reported_seq":476,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.082372+0000","last_change":"2024-04-14T15:18:36.534967+0000","last_active":"2024-04-14T15:18:43.082372+0000","last_peered":"2024-04-14T15:18:43.082372+0000","last_clean":"2024-04-14T15:18:43.082372+0000","last_became_active":"2024-04-14T15:15:21.877667+0000","last_became_peered":"2024-04-14T15:15:21.877667+0000","last_unstale":"2024-04-14T15:18:43.082372+0000","last_undegraded":"2024-04-14T15:18:43.082372+0000","last_fullsized":"2024-04-14T15:18:43.082372+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-04-14T15:18:36.534874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:18:36.534874+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:02.823104+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":331,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.079983+0000","last_change":"2024-04-14T15:16:47.075044+0000","last_active":"2024-04-14T15:18:43.079983+0000","last_peered":"2024-04-14T15:18:43.079983+0000","last_clean":"2024-04-14T15:18:43.079983+0000","last_became_active":"2024-04-14T15:15:44.887452+0000","last_became_peered":"2024-04-14T15:15:44.887452+0000","last_unstale":"2024-04-14T15:18:43.079983+0000","last_undegraded":"2024-04-14T15:18:43.079983+0000","last_fullsized":"2024-04-14T15:18:43.079983+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:47.074950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:47.074950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"280'9","reported_seq":396,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.075562+0000","last_change":"2024-04-14T15:17:50.908130+0000","last_active":"2024-04-14T15:18:43.075562+0000","last_peered":"2024-04-14T15:18:43.075562+0000","last_clean":"2024-04-14T15:18:43.075562+0000","last_became_active":"2024-04-14T15:15:44.882916+0000","last_became_peered":"2024-04-14T15:15:44.882916+0000","last_unstale":"2024-04-14T15:18:43.075562+0000","last_undegraded":"2024-04-14T15:18:43.075562+0000","last_fullsized":"2024-04-14T15:18:43.075562+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"280'9","last_scrub_stamp":"2024-04-14T15:17:50.908054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:17:50.908054+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:20.243489+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":764,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.080477+0000","last_change":"2024-04-14T15:16:54.724807+0000","last_active":"2024-04-14T15:18:43.080477+0000","last_peered":"2024-04-14T15:18:43.080477+0000","last_clean":"2024-04-14T15:18:43.080477+0000","last_became_active":"2024-04-14T15:09:49.066068+0000","last_became_peered":"2024-04-14T15:09:49.066068+0000","last_unstale":"2024-04-14T15:18:43.080477+0000","last_undegraded":"2024-04-14T15:18:43.080477+0000","last_fullsized":"2024-04-14T15:18:43.080477+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:54.724681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:54.724681+0000","last_clean_scrub_stamp":"2024-04-14T15:16:54.724681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:46.957033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"91'130","reported_seq":1050,"reported_epoch":434,"state":"remapped+peering","last_fresh":"2024-04-14T15:18:43.075324+0000","last_change":"2024-04-14T15:18:43.075324+0000","last_active":"2024-04-14T15:18:42.068271+0000","last_peered":"2024-04-14T15:18:40.918151+0000","last_clean":"2024-04-14T15:18:40.918151+0000","last_became_active":"2024-04-14T15:18:40.916970+0000","last_became_peered":"2024-04-14T15:18:40.916970+0000","last_unstale":"2024-04-14T15:18:43.075324+0000","last_undegraded":"2024-04-14T15:18:43.075324+0000","last_fullsized":"2024-04-14T15:18:43.075324+0000","mapping_epoch":434,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:16:43.730442+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:16:43.730442+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":331,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.079563+0000","last_change":"2024-04-14T15:18:30.438135+0000","last_active":"2024-04-14T15:18:43.079563+0000","last_peered":"2024-04-14T15:18:43.079563+0000","last_clean":"2024-04-14T15:18:43.079563+0000","last_became_active":"2024-04-14T15:15:44.889106+0000","last_became_peered":"2024-04-14T15:15:44.889106+0000","last_unstale":"2024-04-14T15:18:43.079563+0000","last_undegraded":"2024-04-14T15:18:43.079563+0000","last_fullsized":"2024-04-14T15:18:43.079563+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:30.438001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:18:30.438001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:48.397871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"183'13","reported_seq":612,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.076949+0000","last_change":"2024-04-14T15:17:37.421888+0000","last_active":"2024-04-14T15:18:43.076949+0000","last_peered":"2024-04-14T15:18:43.076949+0000","last_clean":"2024-04-14T15:18:43.076949+0000","last_became_active":"2024-04-14T15:15:44.884211+0000","last_became_peered":"2024-04-14T15:15:44.884211+0000","last_unstale":"2024-04-14T15:18:43.076949+0000","last_undegraded":"2024-04-14T15:18:43.076949+0000","last_fullsized":"2024-04-14T15:18:43.076949+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'13","last_scrub_stamp":"2024-04-14T15:17:37.421739+0000","last_deep_scrub":"183'13","last_deep_scrub_stamp":"2024-04-14T15:14:11.755252+0000","last_clean_scrub_stamp":"2024-04-14T15:17:37.421739+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:55.179831+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":784,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.080114+0000","last_change":"2024-04-14T15:17:02.670294+0000","last_active":"2024-04-14T15:18:43.080114+0000","last_peered":"2024-04-14T15:18:43.080114+0000","last_clean":"2024-04-14T15:18:43.080114+0000","last_became_active":"2024-04-14T15:09:43.019301+0000","last_became_peered":"2024-04-14T15:09:43.019301+0000","last_unstale":"2024-04-14T15:18:43.080114+0000","last_undegraded":"2024-04-14T15:18:43.080114+0000","last_fullsized":"2024-04-14T15:18:43.080114+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:02.670213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:02.670213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:46.957061+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"255'10","reported_seq":529,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.082730+0000","last_change":"2024-04-14T15:16:59.946889+0000","last_active":"2024-04-14T15:18:43.082730+0000","last_peered":"2024-04-14T15:18:43.082730+0000","last_clean":"2024-04-14T15:18:43.082730+0000","last_became_active":"2024-04-14T15:15:21.884468+0000","last_became_peered":"2024-04-14T15:15:21.884468+0000","last_unstale":"2024-04-14T15:18:43.082730+0000","last_undegraded":"2024-04-14T15:18:43.082730+0000","last_fullsized":"2024-04-14T15:18:43.082730+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'10","last_scrub_stamp":"2024-04-14T15:16:59.946804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:59.946804+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":762,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.074378+0000","last_change":"2024-04-14T15:16:03.948252+0000","last_active":"2024-04-14T15:18:43.074378+0000","last_peered":"2024-04-14T15:18:43.074378+0000","last_clean":"2024-04-14T15:18:43.074378+0000","last_became_active":"2024-04-14T15:09:49.065242+0000","last_became_peered":"2024-04-14T15:09:49.065242+0000","last_unstale":"2024-04-14T15:18:43.074378+0000","last_undegraded":"2024-04-14T15:18:43.074378+0000","last_fullsized":"2024-04-14T15:18:43.074378+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:03.948140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:03.948140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:47.435739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":679,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.077110+0000","last_change":"2024-04-14T15:18:40.735708+0000","last_active":"2024-04-14T15:18:43.077110+0000","last_peered":"2024-04-14T15:18:43.077110+0000","last_clean":"2024-04-14T15:18:43.077110+0000","last_became_active":"2024-04-14T15:12:56.389204+0000","last_became_peered":"2024-04-14T15:12:56.389204+0000","last_unstale":"2024-04-14T15:18:43.077110+0000","last_undegraded":"2024-04-14T15:18:43.077110+0000","last_fullsized":"2024-04-14T15:18:43.077110+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:40.735623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:18:40.735623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:41.555549+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"344'13","reported_seq":303,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.078663+0000","last_change":"2024-04-14T15:16:52.738298+0000","last_active":"2024-04-14T15:18:43.078663+0000","last_peered":"2024-04-14T15:18:43.078663+0000","last_clean":"2024-04-14T15:18:43.078663+0000","last_became_active":"2024-04-14T15:16:24.993130+0000","last_became_peered":"2024-04-14T15:16:24.993130+0000","last_unstale":"2024-04-14T15:18:43.078663+0000","last_undegraded":"2024-04-14T15:18:43.078663+0000","last_fullsized":"2024-04-14T15:18:43.078663+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"344'13","last_scrub_stamp":"2024-04-14T15:16:52.738163+0000","last_deep_scrub":"344'13","last_deep_scrub_stamp":"2024-04-14T15:16:52.738163+0000","last_clean_scrub_stamp":"2024-04-14T15:16:52.738163+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:47.797462+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":783,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.080661+0000","last_change":"2024-04-14T15:17:04.625913+0000","last_active":"2024-04-14T15:18:43.080661+0000","last_peered":"2024-04-14T15:18:43.080661+0000","last_clean":"2024-04-14T15:18:43.080661+0000","last_became_active":"2024-04-14T15:09:43.022480+0000","last_became_peered":"2024-04-14T15:09:43.022480+0000","last_unstale":"2024-04-14T15:18:43.080661+0000","last_undegraded":"2024-04-14T15:18:43.080661+0000","last_fullsized":"2024-04-14T15:18:43.080661+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:04.625824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:04.625824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:47.797545+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":449,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.078234+0000","last_change":"2024-04-14T15:18:33.485536+0000","last_active":"2024-04-14T15:18:43.078234+0000","last_peered":"2024-04-14T15:18:43.078234+0000","last_clean":"2024-04-14T15:18:43.078234+0000","last_became_active":"2024-04-14T15:15:21.882643+0000","last_became_peered":"2024-04-14T15:15:21.882643+0000","last_unstale":"2024-04-14T15:18:43.078234+0000","last_undegraded":"2024-04-14T15:18:43.078234+0000","last_fullsized":"2024-04-14T15:18:43.078234+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:33.485411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:33.485411+0000","last_clean_scrub_stamp":"2024-04-14T15:18:33.485411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:34.513174+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":775,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.078544+0000","last_change":"2024-04-14T15:18:24.039316+0000","last_active":"2024-04-14T15:18:43.078544+0000","last_peered":"2024-04-14T15:18:43.078544+0000","last_clean":"2024-04-14T15:18:43.078544+0000","last_became_active":"2024-04-14T15:09:49.059711+0000","last_became_peered":"2024-04-14T15:09:49.059711+0000","last_unstale":"2024-04-14T15:18:43.078544+0000","last_undegraded":"2024-04-14T15:18:43.078544+0000","last_fullsized":"2024-04-14T15:18:43.078544+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:24.039237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:18:24.039237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:25.190541+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":680,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.080509+0000","last_change":"2024-04-14T15:17:51.921346+0000","last_active":"2024-04-14T15:18:43.080509+0000","last_peered":"2024-04-14T15:18:43.080509+0000","last_clean":"2024-04-14T15:18:43.080509+0000","last_became_active":"2024-04-14T15:12:56.396294+0000","last_became_peered":"2024-04-14T15:12:56.396294+0000","last_unstale":"2024-04-14T15:18:43.080509+0000","last_undegraded":"2024-04-14T15:18:43.080509+0000","last_fullsized":"2024-04-14T15:18:43.080509+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:51.921269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:17:51.921269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:54.899694+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"72'11","reported_seq":789,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.073092+0000","last_change":"2024-04-14T15:16:08.917185+0000","last_active":"2024-04-14T15:18:43.073092+0000","last_peered":"2024-04-14T15:18:43.073092+0000","last_clean":"2024-04-14T15:18:43.073092+0000","last_became_active":"2024-04-14T15:09:45.026065+0000","last_became_peered":"2024-04-14T15:09:45.026065+0000","last_unstale":"2024-04-14T15:18:43.073092+0000","last_undegraded":"2024-04-14T15:18:43.073092+0000","last_fullsized":"2024-04-14T15:18:43.073092+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-04-14T15:16:08.917092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:16:08.917092+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:47.435756+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":784,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.075687+0000","last_change":"2024-04-14T15:17:34.986735+0000","last_active":"2024-04-14T15:18:43.075687+0000","last_peered":"2024-04-14T15:18:43.075687+0000","last_clean":"2024-04-14T15:18:43.075687+0000","last_became_active":"2024-04-14T15:09:43.023583+0000","last_became_peered":"2024-04-14T15:09:43.023583+0000","last_unstale":"2024-04-14T15:18:43.075687+0000","last_undegraded":"2024-04-14T15:18:43.075687+0000","last_fullsized":"2024-04-14T15:18:43.075687+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:34.986655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:34.986655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":768,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.075295+0000","last_change":"2024-04-14T15:17:00.205888+0000","last_active":"2024-04-14T15:18:43.075295+0000","last_peered":"2024-04-14T15:18:43.075295+0000","last_clean":"2024-04-14T15:18:43.075295+0000","last_became_active":"2024-04-14T15:09:47.041683+0000","last_became_peered":"2024-04-14T15:09:47.041683+0000","last_unstale":"2024-04-14T15:18:43.075295+0000","last_undegraded":"2024-04-14T15:18:43.075295+0000","last_fullsized":"2024-04-14T15:18:43.075295+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:00.205808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:08.486217+0000","last_clean_scrub_stamp":"2024-04-14T15:17:00.205808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":445,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.079045+0000","last_change":"2024-04-14T15:16:50.021545+0000","last_active":"2024-04-14T15:18:43.079045+0000","last_peered":"2024-04-14T15:18:43.079045+0000","last_clean":"2024-04-14T15:18:43.079045+0000","last_became_active":"2024-04-14T15:15:21.889567+0000","last_became_peered":"2024-04-14T15:15:21.889567+0000","last_unstale":"2024-04-14T15:18:43.079045+0000","last_undegraded":"2024-04-14T15:18:43.079045+0000","last_fullsized":"2024-04-14T15:18:43.079045+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:50.021470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:50.021470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":680,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-04-14T15:18:44.083975+0000","last_change":"2024-04-14T15:17:03.685135+0000","last_active":"2024-04-14T15:18:44.083975+0000","last_peered":"2024-04-14T15:18:44.083975+0000","last_clean":"2024-04-14T15:18:44.083975+0000","last_became_active":"2024-04-14T15:12:56.387723+0000","last_became_peered":"2024-04-14T15:12:56.387723+0000","last_unstale":"2024-04-14T15:18:44.083975+0000","last_undegraded":"2024-04-14T15:18:44.083975+0000","last_fullsized":"2024-04-14T15:18:44.083975+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:03.685045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:03.685045+0000","last_clean_scrub_stamp":"2024-04-14T15:17:03.685045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.10","version":"72'4","reported_seq":782,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.076677+0000","last_change":"2024-04-14T15:17:53.181132+0000","last_active":"2024-04-14T15:18:43.076677+0000","last_peered":"2024-04-14T15:18:43.076677+0000","last_clean":"2024-04-14T15:18:43.076677+0000","last_became_active":"2024-04-14T15:09:45.038815+0000","last_became_peered":"2024-04-14T15:09:45.038815+0000","last_unstale":"2024-04-14T15:18:43.076677+0000","last_undegraded":"2024-04-14T15:18:43.076677+0000","last_fullsized":"2024-04-14T15:18:43.076677+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4","last_scrub_stamp":"2024-04-14T15:17:53.181053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:17:53.181053+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:15.210428+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":334,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.080804+0000","last_change":"2024-04-14T15:17:48.845163+0000","last_active":"2024-04-14T15:18:43.080804+0000","last_peered":"2024-04-14T15:18:43.080804+0000","last_clean":"2024-04-14T15:18:43.080804+0000","last_became_active":"2024-04-14T15:15:44.884730+0000","last_became_peered":"2024-04-14T15:15:44.884730+0000","last_unstale":"2024-04-14T15:18:43.080804+0000","last_undegraded":"2024-04-14T15:18:43.080804+0000","last_fullsized":"2024-04-14T15:18:43.080804+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:48.845081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:08.712289+0000","last_clean_scrub_stamp":"2024-04-14T15:17:48.845081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:15.427392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":783,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.078022+0000","last_change":"2024-04-14T15:18:29.830291+0000","last_active":"2024-04-14T15:18:43.078022+0000","last_peered":"2024-04-14T15:18:43.078022+0000","last_clean":"2024-04-14T15:18:43.078022+0000","last_became_active":"2024-04-14T15:09:47.041074+0000","last_became_peered":"2024-04-14T15:09:47.041074+0000","last_unstale":"2024-04-14T15:18:43.078022+0000","last_undegraded":"2024-04-14T15:18:43.078022+0000","last_fullsized":"2024-04-14T15:18:43.078022+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:29.830198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:18:29.830198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:48.054530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":762,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.076459+0000","last_change":"2024-04-14T15:16:44.728025+0000","last_active":"2024-04-14T15:18:43.076459+0000","last_peered":"2024-04-14T15:18:43.076459+0000","last_clean":"2024-04-14T15:18:43.076459+0000","last_became_active":"2024-04-14T15:09:49.061359+0000","last_became_peered":"2024-04-14T15:09:49.061359+0000","last_unstale":"2024-04-14T15:18:43.076459+0000","last_undegraded":"2024-04-14T15:18:43.076459+0000","last_fullsized":"2024-04-14T15:18:43.076459+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:44.727907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:44.727907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"91'130","reported_seq":836,"reported_epoch":434,"state":"remapped+peering","last_fresh":"2024-04-14T15:18:43.077361+0000","last_change":"2024-04-14T15:18:43.077361+0000","last_active":"2024-04-14T15:18:43.077361+0000","last_peered":"2024-04-14T15:18:40.919919+0000","last_clean":"2024-04-14T15:18:40.919919+0000","last_became_active":"2024-04-14T15:18:40.919127+0000","last_became_peered":"2024-04-14T15:18:40.919127+0000","last_unstale":"2024-04-14T15:18:43.077361+0000","last_undegraded":"2024-04-14T15:18:43.077361+0000","last_fullsized":"2024-04-14T15:18:43.077361+0000","mapping_epoch":434,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:16:41.944267+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:16:41.944267+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109227,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":679,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.077308+0000","last_change":"2024-04-14T15:18:35.596732+0000","last_active":"2024-04-14T15:18:43.077308+0000","last_peered":"2024-04-14T15:18:43.077308+0000","last_clean":"2024-04-14T15:18:43.077308+0000","last_became_active":"2024-04-14T15:12:56.352528+0000","last_became_peered":"2024-04-14T15:12:56.352528+0000","last_unstale":"2024-04-14T15:18:43.077308+0000","last_undegraded":"2024-04-14T15:18:43.077308+0000","last_fullsized":"2024-04-14T15:18:43.077308+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:35.596656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:15:43.680852+0000","last_clean_scrub_stamp":"2024-04-14T15:18:35.596656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:36.273828+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"102'16","reported_seq":438,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.079632+0000","last_change":"2024-04-14T15:17:43.061551+0000","last_active":"2024-04-14T15:18:43.079632+0000","last_peered":"2024-04-14T15:18:43.079632+0000","last_clean":"2024-04-14T15:18:43.079632+0000","last_became_active":"2024-04-14T15:15:21.884783+0000","last_became_peered":"2024-04-14T15:15:21.884783+0000","last_unstale":"2024-04-14T15:18:43.079632+0000","last_undegraded":"2024-04-14T15:18:43.079632+0000","last_fullsized":"2024-04-14T15:18:43.079632+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'16","last_scrub_stamp":"2024-04-14T15:17:43.061458+0000","last_deep_scrub":"102'16","last_deep_scrub_stamp":"2024-04-14T15:12:35.999326+0000","last_clean_scrub_stamp":"2024-04-14T15:17:43.061458+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:06.716506+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":783,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.080665+0000","last_change":"2024-04-14T15:17:01.643838+0000","last_active":"2024-04-14T15:18:43.080665+0000","last_peered":"2024-04-14T15:18:43.080665+0000","last_clean":"2024-04-14T15:18:43.080665+0000","last_became_active":"2024-04-14T15:09:43.023220+0000","last_became_peered":"2024-04-14T15:09:43.023220+0000","last_unstale":"2024-04-14T15:18:43.080665+0000","last_undegraded":"2024-04-14T15:18:43.080665+0000","last_fullsized":"2024-04-14T15:18:43.080665+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:01.643745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:01.643745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:47.797480+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":766,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.074994+0000","last_change":"2024-04-14T15:16:50.661634+0000","last_active":"2024-04-14T15:18:43.074994+0000","last_peered":"2024-04-14T15:18:43.074994+0000","last_clean":"2024-04-14T15:18:43.074994+0000","last_became_active":"2024-04-14T15:09:47.044665+0000","last_became_peered":"2024-04-14T15:09:47.044665+0000","last_unstale":"2024-04-14T15:18:43.074994+0000","last_undegraded":"2024-04-14T15:18:43.074994+0000","last_fullsized":"2024-04-14T15:18:43.074994+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:50.661520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:50.661520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":445,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.078846+0000","last_change":"2024-04-14T15:16:05.885527+0000","last_active":"2024-04-14T15:18:43.078846+0000","last_peered":"2024-04-14T15:18:43.078846+0000","last_clean":"2024-04-14T15:18:43.078846+0000","last_became_active":"2024-04-14T15:15:21.883940+0000","last_became_peered":"2024-04-14T15:15:21.883940+0000","last_unstale":"2024-04-14T15:18:43.078846+0000","last_undegraded":"2024-04-14T15:18:43.078846+0000","last_fullsized":"2024-04-14T15:18:43.078846+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:05.885438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:05.885438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":680,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-04-14T15:18:44.086801+0000","last_change":"2024-04-14T15:18:25.997409+0000","last_active":"2024-04-14T15:18:44.086801+0000","last_peered":"2024-04-14T15:18:44.086801+0000","last_clean":"2024-04-14T15:18:44.086801+0000","last_became_active":"2024-04-14T15:12:56.335230+0000","last_became_peered":"2024-04-14T15:12:56.335230+0000","last_unstale":"2024-04-14T15:18:44.086801+0000","last_undegraded":"2024-04-14T15:18:44.086801+0000","last_fullsized":"2024-04-14T15:18:44.086801+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:25.997333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:18:25.997333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:41.530252+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"72'10","reported_seq":788,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.080865+0000","last_change":"2024-04-14T15:16:49.779653+0000","last_active":"2024-04-14T15:18:43.080865+0000","last_peered":"2024-04-14T15:18:43.080865+0000","last_clean":"2024-04-14T15:18:43.080865+0000","last_became_active":"2024-04-14T15:09:45.181456+0000","last_became_peered":"2024-04-14T15:09:45.181456+0000","last_unstale":"2024-04-14T15:18:43.080865+0000","last_undegraded":"2024-04-14T15:18:43.080865+0000","last_fullsized":"2024-04-14T15:18:43.080865+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-04-14T15:16:49.779557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:16:49.779557+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:46.957002+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":333,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.072330+0000","last_change":"2024-04-14T15:16:42.205142+0000","last_active":"2024-04-14T15:18:43.072330+0000","last_peered":"2024-04-14T15:18:43.072330+0000","last_clean":"2024-04-14T15:18:43.072330+0000","last_became_active":"2024-04-14T15:15:44.878336+0000","last_became_peered":"2024-04-14T15:15:44.878336+0000","last_unstale":"2024-04-14T15:18:43.072330+0000","last_undegraded":"2024-04-14T15:18:43.072330+0000","last_fullsized":"2024-04-14T15:18:43.072330+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:42.205063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:42.205063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:47.435873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"72'8","reported_seq":904,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.081066+0000","last_change":"2024-04-14T15:18:28.793061+0000","last_active":"2024-04-14T15:18:43.081066+0000","last_peered":"2024-04-14T15:18:43.081066+0000","last_clean":"2024-04-14T15:18:43.081066+0000","last_became_active":"2024-04-14T15:09:47.048216+0000","last_became_peered":"2024-04-14T15:09:47.048216+0000","last_unstale":"2024-04-14T15:18:43.081066+0000","last_undegraded":"2024-04-14T15:18:43.081066+0000","last_fullsized":"2024-04-14T15:18:43.081066+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:18:28.792975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:18:28.792975+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:50.662051+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"72'1","reported_seq":771,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.076948+0000","last_change":"2024-04-14T15:17:50.529174+0000","last_active":"2024-04-14T15:18:43.076948+0000","last_peered":"2024-04-14T15:18:43.076948+0000","last_clean":"2024-04-14T15:18:43.076948+0000","last_became_active":"2024-04-14T15:09:49.062051+0000","last_became_peered":"2024-04-14T15:09:49.062051+0000","last_unstale":"2024-04-14T15:18:43.076948+0000","last_undegraded":"2024-04-14T15:18:43.076948+0000","last_fullsized":"2024-04-14T15:18:43.076948+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-04-14T15:17:50.529089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:17:50.529089+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:53.773686+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":673,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.079355+0000","last_change":"2024-04-14T15:16:09.471961+0000","last_active":"2024-04-14T15:18:43.079355+0000","last_peered":"2024-04-14T15:18:43.079355+0000","last_clean":"2024-04-14T15:18:43.079355+0000","last_became_active":"2024-04-14T15:12:56.543793+0000","last_became_peered":"2024-04-14T15:12:56.543793+0000","last_unstale":"2024-04-14T15:18:43.079355+0000","last_undegraded":"2024-04-14T15:18:43.079355+0000","last_fullsized":"2024-04-14T15:18:43.079355+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:09.471852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:16:09.471852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:46.956720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"72'11","reported_seq":817,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.081122+0000","last_change":"2024-04-14T15:18:31.818435+0000","last_active":"2024-04-14T15:18:43.081122+0000","last_peered":"2024-04-14T15:18:43.081122+0000","last_clean":"2024-04-14T15:18:43.081122+0000","last_became_active":"2024-04-14T15:10:07.764183+0000","last_became_peered":"2024-04-14T15:10:07.764183+0000","last_unstale":"2024-04-14T15:18:43.081122+0000","last_undegraded":"2024-04-14T15:18:43.081122+0000","last_fullsized":"2024-04-14T15:18:43.081122+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-04-14T15:18:31.818311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:18:31.818311+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:33.770617+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"65'2","reported_seq":464,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.079263+0000","last_change":"2024-04-14T15:16:56.943047+0000","last_active":"2024-04-14T15:18:43.079263+0000","last_peered":"2024-04-14T15:18:43.079263+0000","last_clean":"2024-04-14T15:18:43.079263+0000","last_became_active":"2024-04-14T15:15:21.885343+0000","last_became_peered":"2024-04-14T15:15:21.885343+0000","last_unstale":"2024-04-14T15:18:43.079263+0000","last_undegraded":"2024-04-14T15:18:43.079263+0000","last_fullsized":"2024-04-14T15:18:43.079263+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-04-14T15:16:56.942840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:56.942840+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":769,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.080959+0000","last_change":"2024-04-14T15:16:52.747151+0000","last_active":"2024-04-14T15:18:43.080959+0000","last_peered":"2024-04-14T15:18:43.080959+0000","last_clean":"2024-04-14T15:18:43.080959+0000","last_became_active":"2024-04-14T15:09:47.045730+0000","last_became_peered":"2024-04-14T15:09:47.045730+0000","last_unstale":"2024-04-14T15:18:43.080959+0000","last_undegraded":"2024-04-14T15:18:43.080959+0000","last_fullsized":"2024-04-14T15:18:43.080959+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:52.747070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:52.747070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:46.956841+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":764,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.075386+0000","last_change":"2024-04-14T15:16:57.235339+0000","last_active":"2024-04-14T15:18:43.075386+0000","last_peered":"2024-04-14T15:18:43.075386+0000","last_clean":"2024-04-14T15:18:43.075386+0000","last_became_active":"2024-04-14T15:09:49.057545+0000","last_became_peered":"2024-04-14T15:09:49.057545+0000","last_unstale":"2024-04-14T15:18:43.075386+0000","last_undegraded":"2024-04-14T15:18:43.075386+0000","last_fullsized":"2024-04-14T15:18:43.075386+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:57.235261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:21.247695+0000","last_clean_scrub_stamp":"2024-04-14T15:16:57.235261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":333,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.072924+0000","last_change":"2024-04-14T15:17:44.364114+0000","last_active":"2024-04-14T15:18:43.072924+0000","last_peered":"2024-04-14T15:18:43.072924+0000","last_clean":"2024-04-14T15:18:43.072924+0000","last_became_active":"2024-04-14T15:15:44.882604+0000","last_became_peered":"2024-04-14T15:15:44.882604+0000","last_unstale":"2024-04-14T15:18:43.072924+0000","last_undegraded":"2024-04-14T15:18:43.072924+0000","last_fullsized":"2024-04-14T15:18:43.072924+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:44.364013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:17:44.364013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:01.492147+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"72'15","reported_seq":804,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-04-14T15:18:44.089230+0000","last_change":"2024-04-14T15:18:32.950805+0000","last_active":"2024-04-14T15:18:44.089230+0000","last_peered":"2024-04-14T15:18:44.089230+0000","last_clean":"2024-04-14T15:18:44.089230+0000","last_became_active":"2024-04-14T15:09:45.032887+0000","last_became_peered":"2024-04-14T15:09:45.032887+0000","last_unstale":"2024-04-14T15:18:44.089230+0000","last_undegraded":"2024-04-14T15:18:44.089230+0000","last_fullsized":"2024-04-14T15:18:44.089230+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-04-14T15:18:32.950706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:18:32.950706+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:54.341517+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":786,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.079478+0000","last_change":"2024-04-14T15:18:27.799078+0000","last_active":"2024-04-14T15:18:43.079478+0000","last_peered":"2024-04-14T15:18:43.079478+0000","last_clean":"2024-04-14T15:18:43.079478+0000","last_became_active":"2024-04-14T15:09:43.021494+0000","last_became_peered":"2024-04-14T15:09:43.021494+0000","last_unstale":"2024-04-14T15:18:43.079478+0000","last_undegraded":"2024-04-14T15:18:43.079478+0000","last_fullsized":"2024-04-14T15:18:43.079478+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:27.798964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:18:27.798964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:33.052017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":776,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.075610+0000","last_change":"2024-04-14T15:18:22.953738+0000","last_active":"2024-04-14T15:18:43.075610+0000","last_peered":"2024-04-14T15:18:43.075610+0000","last_clean":"2024-04-14T15:18:43.075610+0000","last_became_active":"2024-04-14T15:09:47.042733+0000","last_became_peered":"2024-04-14T15:09:47.042733+0000","last_unstale":"2024-04-14T15:18:43.075610+0000","last_undegraded":"2024-04-14T15:18:43.075610+0000","last_fullsized":"2024-04-14T15:18:43.075610+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:22.953661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:18:22.953661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:49.908442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":771,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.073716+0000","last_change":"2024-04-14T15:17:54.453289+0000","last_active":"2024-04-14T15:18:43.073716+0000","last_peered":"2024-04-14T15:18:43.073716+0000","last_clean":"2024-04-14T15:18:43.073716+0000","last_became_active":"2024-04-14T15:09:49.127979+0000","last_became_peered":"2024-04-14T15:09:49.127979+0000","last_unstale":"2024-04-14T15:18:43.073716+0000","last_undegraded":"2024-04-14T15:18:43.073716+0000","last_fullsized":"2024-04-14T15:18:43.073716+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:54.453174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:54.453174+0000","last_clean_scrub_stamp":"2024-04-14T15:17:54.453174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:08.115614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":679,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.075999+0000","last_change":"2024-04-14T15:18:26.532035+0000","last_active":"2024-04-14T15:18:43.075999+0000","last_peered":"2024-04-14T15:18:43.075999+0000","last_clean":"2024-04-14T15:18:43.075999+0000","last_became_active":"2024-04-14T15:12:56.392352+0000","last_became_peered":"2024-04-14T15:12:56.392352+0000","last_unstale":"2024-04-14T15:18:43.075999+0000","last_undegraded":"2024-04-14T15:18:43.075999+0000","last_fullsized":"2024-04-14T15:18:43.075999+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:26.531955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:18:26.531955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:43.681656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"72'12","reported_seq":476,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.081186+0000","last_change":"2024-04-14T15:16:11.903062+0000","last_active":"2024-04-14T15:18:43.081186+0000","last_peered":"2024-04-14T15:18:43.081186+0000","last_clean":"2024-04-14T15:18:43.081186+0000","last_became_active":"2024-04-14T15:15:21.877838+0000","last_became_peered":"2024-04-14T15:15:21.877838+0000","last_unstale":"2024-04-14T15:18:43.081186+0000","last_undegraded":"2024-04-14T15:18:43.081186+0000","last_fullsized":"2024-04-14T15:18:43.081186+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'12","last_scrub_stamp":"2024-04-14T15:16:11.902934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:16:11.902934+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":457,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.078560+0000","last_change":"2024-04-14T15:16:20.969027+0000","last_active":"2024-04-14T15:18:43.078560+0000","last_peered":"2024-04-14T15:18:43.078560+0000","last_clean":"2024-04-14T15:18:43.078560+0000","last_became_active":"2024-04-14T15:15:21.883544+0000","last_became_peered":"2024-04-14T15:15:21.883544+0000","last_unstale":"2024-04-14T15:18:43.078560+0000","last_undegraded":"2024-04-14T15:18:43.078560+0000","last_fullsized":"2024-04-14T15:18:43.078560+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:20.968897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:20.968897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":774,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.073250+0000","last_change":"2024-04-14T15:17:01.098681+0000","last_active":"2024-04-14T15:18:43.073250+0000","last_peered":"2024-04-14T15:18:43.073250+0000","last_clean":"2024-04-14T15:18:43.073250+0000","last_became_active":"2024-04-14T15:09:47.048045+0000","last_became_peered":"2024-04-14T15:09:47.048045+0000","last_unstale":"2024-04-14T15:18:43.073250+0000","last_undegraded":"2024-04-14T15:18:43.073250+0000","last_fullsized":"2024-04-14T15:18:43.073250+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:01.098599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:17:01.098599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:47.435959+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":446,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.082941+0000","last_change":"2024-04-14T15:16:07.879394+0000","last_active":"2024-04-14T15:18:43.082941+0000","last_peered":"2024-04-14T15:18:43.082941+0000","last_clean":"2024-04-14T15:18:43.082941+0000","last_became_active":"2024-04-14T15:15:21.889691+0000","last_became_peered":"2024-04-14T15:15:21.889691+0000","last_unstale":"2024-04-14T15:18:43.082941+0000","last_undegraded":"2024-04-14T15:18:43.082941+0000","last_fullsized":"2024-04-14T15:18:43.082941+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:07.879280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:07.879280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":680,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-04-14T15:18:44.087165+0000","last_change":"2024-04-14T15:17:37.766225+0000","last_active":"2024-04-14T15:18:44.087165+0000","last_peered":"2024-04-14T15:18:44.087165+0000","last_clean":"2024-04-14T15:18:44.087165+0000","last_became_active":"2024-04-14T15:12:56.375278+0000","last_became_peered":"2024-04-14T15:12:56.375278+0000","last_unstale":"2024-04-14T15:18:44.087165+0000","last_undegraded":"2024-04-14T15:18:44.087165+0000","last_fullsized":"2024-04-14T15:18:44.087165+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:37.766116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:17:37.766116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"276'13","reported_seq":798,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.072661+0000","last_change":"2024-04-14T15:17:52.445394+0000","last_active":"2024-04-14T15:18:43.072661+0000","last_peered":"2024-04-14T15:18:43.072661+0000","last_clean":"2024-04-14T15:18:43.072661+0000","last_became_active":"2024-04-14T15:09:45.030093+0000","last_became_peered":"2024-04-14T15:09:45.030093+0000","last_unstale":"2024-04-14T15:18:43.072661+0000","last_undegraded":"2024-04-14T15:18:43.072661+0000","last_fullsized":"2024-04-14T15:18:43.072661+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'13","last_scrub_stamp":"2024-04-14T15:17:52.445294+0000","last_deep_scrub":"276'13","last_deep_scrub_stamp":"2024-04-14T15:15:43.969775+0000","last_clean_scrub_stamp":"2024-04-14T15:17:52.445294+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:59.668815+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":480,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.075931+0000","last_change":"2024-04-14T15:17:02.800274+0000","last_active":"2024-04-14T15:18:43.075931+0000","last_peered":"2024-04-14T15:18:43.075931+0000","last_clean":"2024-04-14T15:18:43.075931+0000","last_became_active":"2024-04-14T15:14:30.857727+0000","last_became_peered":"2024-04-14T15:14:30.857727+0000","last_unstale":"2024-04-14T15:18:43.075931+0000","last_undegraded":"2024-04-14T15:18:43.075931+0000","last_fullsized":"2024-04-14T15:18:43.075931+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:02.800166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:02.800166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":335,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.080322+0000","last_change":"2024-04-14T15:16:59.695423+0000","last_active":"2024-04-14T15:18:43.080322+0000","last_peered":"2024-04-14T15:18:43.080322+0000","last_clean":"2024-04-14T15:18:43.080322+0000","last_became_active":"2024-04-14T15:15:44.883121+0000","last_became_peered":"2024-04-14T15:15:44.883121+0000","last_unstale":"2024-04-14T15:18:43.080322+0000","last_undegraded":"2024-04-14T15:18:43.080322+0000","last_fullsized":"2024-04-14T15:18:43.080322+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:59.695337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:59.695337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:46.957119+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":772,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.079798+0000","last_change":"2024-04-14T15:17:33.952503+0000","last_active":"2024-04-14T15:18:43.079798+0000","last_peered":"2024-04-14T15:18:43.079798+0000","last_clean":"2024-04-14T15:18:43.079798+0000","last_became_active":"2024-04-14T15:09:49.061756+0000","last_became_peered":"2024-04-14T15:09:49.061756+0000","last_unstale":"2024-04-14T15:18:43.079798+0000","last_undegraded":"2024-04-14T15:18:43.079798+0000","last_fullsized":"2024-04-14T15:18:43.079798+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:33.952312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:17:33.952312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:46.957141+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":680,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.080814+0000","last_change":"2024-04-14T15:16:57.667484+0000","last_active":"2024-04-14T15:18:43.080814+0000","last_peered":"2024-04-14T15:18:43.080814+0000","last_clean":"2024-04-14T15:18:43.080814+0000","last_became_active":"2024-04-14T15:12:56.232831+0000","last_became_peered":"2024-04-14T15:12:56.232831+0000","last_unstale":"2024-04-14T15:18:43.080814+0000","last_undegraded":"2024-04-14T15:18:43.080814+0000","last_fullsized":"2024-04-14T15:18:43.080814+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:57.667382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:16:57.667382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:47.797496+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1b","version":"87'7","reported_seq":790,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.073180+0000","last_change":"2024-04-14T15:18:35.481608+0000","last_active":"2024-04-14T15:18:43.073180+0000","last_peered":"2024-04-14T15:18:43.073180+0000","last_clean":"2024-04-14T15:18:43.073180+0000","last_became_active":"2024-04-14T15:09:45.028667+0000","last_became_peered":"2024-04-14T15:09:45.028667+0000","last_unstale":"2024-04-14T15:18:43.073180+0000","last_undegraded":"2024-04-14T15:18:43.073180+0000","last_fullsized":"2024-04-14T15:18:43.073180+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'7","last_scrub_stamp":"2024-04-14T15:18:35.481504+0000","last_deep_scrub":"87'7","last_deep_scrub_stamp":"2024-04-14T15:11:10.510785+0000","last_clean_scrub_stamp":"2024-04-14T15:18:35.481504+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:51.921756+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":252,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-04-14T15:18:44.086674+0000","last_change":"2024-04-14T15:16:46.686879+0000","last_active":"2024-04-14T15:18:44.086674+0000","last_peered":"2024-04-14T15:18:44.086674+0000","last_clean":"2024-04-14T15:18:44.086674+0000","last_became_active":"2024-04-14T15:16:24.910246+0000","last_became_peered":"2024-04-14T15:16:24.910246+0000","last_unstale":"2024-04-14T15:18:44.086674+0000","last_undegraded":"2024-04-14T15:18:44.086674+0000","last_fullsized":"2024-04-14T15:18:44.086674+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:46.686789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:08.254601+0000","last_clean_scrub_stamp":"2024-04-14T15:16:46.686789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":766,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.076297+0000","last_change":"2024-04-14T15:16:17.804733+0000","last_active":"2024-04-14T15:18:43.076297+0000","last_peered":"2024-04-14T15:18:43.076297+0000","last_clean":"2024-04-14T15:18:43.076297+0000","last_became_active":"2024-04-14T15:09:47.046517+0000","last_became_peered":"2024-04-14T15:09:47.046517+0000","last_unstale":"2024-04-14T15:18:43.076297+0000","last_undegraded":"2024-04-14T15:18:43.076297+0000","last_fullsized":"2024-04-14T15:18:43.076297+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:17.804649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:17.804649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":336,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.073203+0000","last_change":"2024-04-14T15:18:21.023508+0000","last_active":"2024-04-14T15:18:43.073203+0000","last_peered":"2024-04-14T15:18:43.073203+0000","last_clean":"2024-04-14T15:18:43.073203+0000","last_became_active":"2024-04-14T15:15:44.887227+0000","last_became_peered":"2024-04-14T15:15:44.887227+0000","last_unstale":"2024-04-14T15:18:43.073203+0000","last_undegraded":"2024-04-14T15:18:43.073203+0000","last_fullsized":"2024-04-14T15:18:43.073203+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:21.023426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:56.268086+0000","last_clean_scrub_stamp":"2024-04-14T15:18:21.023426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:27.500436+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":679,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.077948+0000","last_change":"2024-04-14T15:18:34.558447+0000","last_active":"2024-04-14T15:18:43.077948+0000","last_peered":"2024-04-14T15:18:43.077948+0000","last_clean":"2024-04-14T15:18:43.077948+0000","last_became_active":"2024-04-14T15:12:56.189117+0000","last_became_peered":"2024-04-14T15:12:56.189117+0000","last_unstale":"2024-04-14T15:18:43.077948+0000","last_undegraded":"2024-04-14T15:18:43.077948+0000","last_fullsized":"2024-04-14T15:18:43.077948+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:34.558312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:34.558312+0000","last_clean_scrub_stamp":"2024-04-14T15:18:34.558312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:58.210395+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.18","version":"72'9","reported_seq":794,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.072211+0000","last_change":"2024-04-14T15:18:36.441934+0000","last_active":"2024-04-14T15:18:43.072211+0000","last_peered":"2024-04-14T15:18:43.072211+0000","last_clean":"2024-04-14T15:18:43.072211+0000","last_became_active":"2024-04-14T15:09:45.032079+0000","last_became_peered":"2024-04-14T15:09:45.032079+0000","last_unstale":"2024-04-14T15:18:43.072211+0000","last_undegraded":"2024-04-14T15:18:43.072211+0000","last_fullsized":"2024-04-14T15:18:43.072211+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-04-14T15:18:36.441813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:18:36.441813+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:03.217663+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":458,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.082812+0000","last_change":"2024-04-14T15:17:00.948362+0000","last_active":"2024-04-14T15:18:43.082812+0000","last_peered":"2024-04-14T15:18:43.082812+0000","last_clean":"2024-04-14T15:18:43.082812+0000","last_became_active":"2024-04-14T15:15:21.891926+0000","last_became_peered":"2024-04-14T15:15:21.891926+0000","last_unstale":"2024-04-14T15:18:43.082812+0000","last_undegraded":"2024-04-14T15:18:43.082812+0000","last_fullsized":"2024-04-14T15:18:43.082812+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:00.948283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:00.948283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":768,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.073043+0000","last_change":"2024-04-14T15:16:45.114648+0000","last_active":"2024-04-14T15:18:43.073043+0000","last_peered":"2024-04-14T15:18:43.073043+0000","last_clean":"2024-04-14T15:18:43.073043+0000","last_became_active":"2024-04-14T15:09:47.041486+0000","last_became_peered":"2024-04-14T15:09:47.041486+0000","last_unstale":"2024-04-14T15:18:43.073043+0000","last_undegraded":"2024-04-14T15:18:43.073043+0000","last_fullsized":"2024-04-14T15:18:43.073043+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:45.114534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:45.114534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":762,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.072634+0000","last_change":"2024-04-14T15:16:55.184211+0000","last_active":"2024-04-14T15:18:43.072634+0000","last_peered":"2024-04-14T15:18:43.072634+0000","last_clean":"2024-04-14T15:18:43.072634+0000","last_became_active":"2024-04-14T15:09:49.058904+0000","last_became_peered":"2024-04-14T15:09:49.058904+0000","last_unstale":"2024-04-14T15:18:43.072634+0000","last_undegraded":"2024-04-14T15:18:43.072634+0000","last_fullsized":"2024-04-14T15:18:43.072634+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:55.184094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:55.184094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:47.435927+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":333,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.076700+0000","last_change":"2024-04-14T15:17:52.593706+0000","last_active":"2024-04-14T15:18:43.076700+0000","last_peered":"2024-04-14T15:18:43.076700+0000","last_clean":"2024-04-14T15:18:43.076700+0000","last_became_active":"2024-04-14T15:15:44.889605+0000","last_became_peered":"2024-04-14T15:15:44.889605+0000","last_unstale":"2024-04-14T15:18:43.076700+0000","last_undegraded":"2024-04-14T15:18:43.076700+0000","last_fullsized":"2024-04-14T15:18:43.076700+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:52.593618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:17:52.593618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:09.797845+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"306'22","reported_seq":485,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.079722+0000","last_change":"2024-04-14T15:16:13.898263+0000","last_active":"2024-04-14T15:18:43.079722+0000","last_peered":"2024-04-14T15:18:43.079722+0000","last_clean":"2024-04-14T15:18:43.079722+0000","last_became_active":"2024-04-14T15:15:21.884310+0000","last_became_peered":"2024-04-14T15:15:21.884310+0000","last_unstale":"2024-04-14T15:18:43.079722+0000","last_undegraded":"2024-04-14T15:18:43.079722+0000","last_fullsized":"2024-04-14T15:18:43.079722+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'22","last_scrub_stamp":"2024-04-14T15:16:13.898106+0000","last_deep_scrub":"306'22","last_deep_scrub_stamp":"2024-04-14T15:16:13.898106+0000","last_clean_scrub_stamp":"2024-04-14T15:16:13.898106+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":777,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.077901+0000","last_change":"2024-04-14T15:16:14.427502+0000","last_active":"2024-04-14T15:18:43.077901+0000","last_peered":"2024-04-14T15:18:43.077901+0000","last_clean":"2024-04-14T15:18:43.077901+0000","last_became_active":"2024-04-14T15:09:43.017351+0000","last_became_peered":"2024-04-14T15:09:43.017351+0000","last_unstale":"2024-04-14T15:18:43.077901+0000","last_undegraded":"2024-04-14T15:18:43.077901+0000","last_fullsized":"2024-04-14T15:18:43.077901+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:14.427425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:14.427425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:46.956812+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":766,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.072445+0000","last_change":"2024-04-14T15:16:19.041461+0000","last_active":"2024-04-14T15:18:43.072445+0000","last_peered":"2024-04-14T15:18:43.072445+0000","last_clean":"2024-04-14T15:18:43.072445+0000","last_became_active":"2024-04-14T15:09:47.042538+0000","last_became_peered":"2024-04-14T15:09:47.042538+0000","last_unstale":"2024-04-14T15:18:43.072445+0000","last_undegraded":"2024-04-14T15:18:43.072445+0000","last_fullsized":"2024-04-14T15:18:43.072445+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:19.041346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:08.512130+0000","last_clean_scrub_stamp":"2024-04-14T15:16:19.041346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:18:47.435842+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":768,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-14T15:18:43.077860+0000","last_change":"2024-04-14T15:17:56.237366+0000","last_active":"2024-04-14T15:18:43.077860+0000","last_peered":"2024-04-14T15:18:43.077860+0000","last_clean":"2024-04-14T15:18:43.077860+0000","last_became_active":"2024-04-14T15:09:49.129636+0000","last_became_peered":"2024-04-14T15:09:49.129636+0000","last_unstale":"2024-04-14T15:18:43.077860+0000","last_undegraded":"2024-04-14T15:18:43.077860+0000","last_fullsized":"2024-04-14T15:18:43.077860+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:56.237273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:17:56.237273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:03.893408+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":110,"num_pg":32,"stat_sum":{"num_bytes":8195783,"num_objects":159,"num_object_clones":2,"num_object_copies":477,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":159,"num_whiteouts":0,"num_read":108,"num_read_kb":9,"num_write":599,"num_write_kb":12688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":7,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":26480640,"data_stored":24587349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1885,"ondisk_log_size":1885,"up":96,"acting":96,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":670873,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":30,"num_read_kb":42,"num_write":25,"num_write_kb":330,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":241515,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2730,"ondisk_log_size":2730,"up":63,"acting":53,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":534,"num_read_kb":529,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":695,"num_read_kb":452,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":488,"ondisk_log_size":488,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71,"ondisk_log_size":71,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103282,"num_pgs":105,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":73192,"kb_used_data":15624,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93671960,"statfs":{"total":95995035648,"available":95920087040,"internally_reserved":0,"allocated":15998976,"data_stored":12714170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":58915274},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397621,"num_pgs":90,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":70448,"kb_used_data":7312,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93674704,"statfs":{"total":95995035648,"available":95922896896,"internally_reserved":0,"allocated":7487488,"data_stored":4192493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574991,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":87108,"kb_used_data":11504,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93658044,"statfs":{"total":95995035648,"available":95905837056,"internally_reserved":0,"allocated":11780096,"data_stored":8387084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738516,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":82732,"kb_used_data":11264,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93662420,"statfs":{"total":95995035648,"available":95910318080,"internally_reserved":0,"allocated":11534336,"data_stored":8192919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411467,"num_pgs":89,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":83008,"kb_used_data":11552,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93662144,"statfs":{"total":95995035648,"available":95910035456,"internally_reserved":0,"allocated":11829248,"data_stored":8454622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182533,"num_pgs":99,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":57104,"kb_used_data":7868,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93688048,"statfs":{"total":95995035648,"available":95936561152,"internally_reserved":0,"allocated":8056832,"data_stored":4783177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986303,"num_pgs":91,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":70420,"kb_used_data":7284,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93674732,"statfs":{"total":95995035648,"available":95922925568,"internally_reserved":0,"allocated":7458816,"data_stored":4192013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659260,"num_pgs":91,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":75108,"kb_used_data":11960,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93670044,"statfs":{"total":95995035648,"available":95918125056,"internally_reserved":0,"allocated":12247040,"data_stored":8976170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4268032,"data_stored":4000600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4493312,"data_stored":4195009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4321280,"data_stored":4000756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":184320,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4403200,"data_stored":4194789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":225280,"data_stored":524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8417280,"data_stored":8194829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-14T15:18:49.602 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-04-14T15:18:49.602 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd pool set .mgr pg_num 31 2024-04-14T15:18:49.657 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:18:49.683 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:49.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:49.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:18:49.785 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:49.789 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-14T15:18:49.789 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:18:49.808 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:49.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:49.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:49.910 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:18:49.935 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:49.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:49.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:18:50.037 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:18:50.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:50.061 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:50.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:18:50.141 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:18:50.162 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:18:50.185 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:50.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:50.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:18:50.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:50 smithi090 ceph-mon[41670]: pgmap v615: 216 pgs: 1 remapped+peering, 215 active+clean; 8.5 MiB data, 585 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 op/s; 80 KiB/s, 0 objects/s recovering 2024-04-14T15:18:50.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:50 smithi090 ceph-mon[41670]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-04-14T15:18:50.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:50 smithi090 ceph-mon[41670]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:18:50.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:50 smithi090 ceph-mon[41670]: from='client.49168 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-14T15:18:50.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:50 smithi090 ceph-mon[41670]: osdmap e439: 8 total, 8 up, 8 in 2024-04-14T15:18:50.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:50 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:18:50.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:50 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:18:50.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:50 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:18:50.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:50 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:18:50.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:50 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:18:50.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:50 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:18:50.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:50 smithi090 ceph-mon[30399]: pgmap v615: 216 pgs: 1 remapped+peering, 215 active+clean; 8.5 MiB data, 585 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 op/s; 80 KiB/s, 0 objects/s recovering 2024-04-14T15:18:50.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:50 smithi090 ceph-mon[30399]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-04-14T15:18:50.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:50 smithi090 ceph-mon[30399]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:18:50.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:50 smithi090 ceph-mon[30399]: from='client.49168 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-14T15:18:50.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:50 smithi090 ceph-mon[30399]: osdmap e439: 8 total, 8 up, 8 in 2024-04-14T15:18:50.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:50 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:18:50.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:50 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:18:50.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:50 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:18:50.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:50 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:18:50.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:50 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:18:50.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:50 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:18:50.287 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:18:50.311 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:50.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:50.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:18:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:50 smithi092 ceph-mon[33966]: pgmap v615: 216 pgs: 1 remapped+peering, 215 active+clean; 8.5 MiB data, 585 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 op/s; 80 KiB/s, 0 objects/s recovering 2024-04-14T15:18:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:50 smithi092 ceph-mon[33966]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-04-14T15:18:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:50 smithi092 ceph-mon[33966]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:18:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:50 smithi092 ceph-mon[33966]: from='client.49168 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-14T15:18:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:50 smithi092 ceph-mon[33966]: osdmap e439: 8 total, 8 up, 8 in 2024-04-14T15:18:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:50 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:18:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:50 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:18:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:50 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:18:50.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:50 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:18:50.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:50 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:18:50.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:50 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:18:50.412 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:18:50.436 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:50.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:50.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:18:50.538 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:50.561 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:50.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:50.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:50.589 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-14T15:18:50.663 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:18:50.686 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:50.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:50.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:18:50.789 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:50.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:50.814 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:50.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:50.916 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:18:50.940 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:50.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:50.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:18:51.042 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:18:51.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:51.068 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:51.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:18:51.169 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:18:51.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:51.194 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:51.194 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:18:51.295 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:18:51.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:51.320 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:51.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:18:51.421 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:18:51.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:51 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:18:51.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:51 smithi090 ceph-mon[30399]: osdmap e440: 8 total, 8 up, 8 in 2024-04-14T15:18:51.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:51 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-14T15:18:51.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:51 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-14T15:18:51.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:51 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-14T15:18:51.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:51 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:18:51.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:51 smithi090 ceph-mon[41670]: osdmap e440: 8 total, 8 up, 8 in 2024-04-14T15:18:51.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:51 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-14T15:18:51.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:51 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-14T15:18:51.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:51 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-14T15:18:51.445 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:51.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:51.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:18:51.546 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:18:51.572 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:51.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:51.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:18:51.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:51 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:18:51.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:51 smithi092 ceph-mon[33966]: osdmap e440: 8 total, 8 up, 8 in 2024-04-14T15:18:51.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:51 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-14T15:18:51.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:51 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-14T15:18:51.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:51 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-14T15:18:51.674 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:51.699 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:51.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:51.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:51.801 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:18:51.817 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:18:51.818 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:18:51.828 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:51.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:51.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:18:51.930 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:18:51.954 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:51.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:51.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:18:52.056 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:18:52.083 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:52.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:52.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:18:52.185 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:18:52.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:52.211 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:52.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:18:52.292 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:52 smithi090 ceph-mon[41670]: pgmap v618: 248 pgs: 32 unknown, 216 active+clean; 8.5 MiB data, 585 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-04-14T15:18:52.292 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:52 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-14T15:18:52.292 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:52 smithi090 ceph-mon[41670]: osdmap e441: 8 total, 8 up, 8 in 2024-04-14T15:18:52.293 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:52 smithi090 ceph-mon[30399]: pgmap v618: 248 pgs: 32 unknown, 216 active+clean; 8.5 MiB data, 585 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-04-14T15:18:52.293 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:52 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-14T15:18:52.293 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:52 smithi090 ceph-mon[30399]: osdmap e441: 8 total, 8 up, 8 in 2024-04-14T15:18:52.312 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:18:52.336 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:52.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:52.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:18:52.438 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:52.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:52.465 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:52.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:52.523 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:52 smithi092 ceph-mon[33966]: pgmap v618: 248 pgs: 32 unknown, 216 active+clean; 8.5 MiB data, 585 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-04-14T15:18:52.523 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:52 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-14T15:18:52.523 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:52 smithi092 ceph-mon[33966]: osdmap e441: 8 total, 8 up, 8 in 2024-04-14T15:18:52.565 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:18:52.591 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:52.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:52.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:18:52.692 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:52.716 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:52.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:52.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:52.817 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:18:52.842 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:52.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:52.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:18:52.943 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:18:52.967 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:52.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:52.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:18:53.069 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:18:53.094 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:53.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:53.095 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:18:53.196 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:18:53.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:53.220 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:53.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:18:53.322 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:18:53.332 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd unset noscrub 2024-04-14T15:18:53.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:53.346 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:53.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:18:53.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:53 smithi090 ceph-mon[30399]: osdmap e442: 8 total, 8 up, 8 in 2024-04-14T15:18:53.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:53 smithi090 ceph-mon[41670]: osdmap e442: 8 total, 8 up, 8 in 2024-04-14T15:18:53.447 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:18:53.470 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:53.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:53.471 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:18:53.572 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:18:53.595 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:53.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:53.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:18:53.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:53 smithi092 ceph-mon[33966]: osdmap e442: 8 total, 8 up, 8 in 2024-04-14T15:18:53.696 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:18:53.720 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:53.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:53.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:18:53.822 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:18:53.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:53.847 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:53.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:18:53.947 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:18:53.972 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:53.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:53.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:18:54.074 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:54.098 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:54.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:54.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:54.200 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:54.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:54.224 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:54.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:54.325 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:18:54.348 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:54.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:54.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:18:54.397 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:54 smithi090 ceph-mon[30399]: pgmap v621: 248 pgs: 32 unknown, 216 active+clean; 8.5 MiB data, 585 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:18:54.397 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:54 smithi090 ceph-mon[30399]: osdmap e443: 8 total, 8 up, 8 in 2024-04-14T15:18:54.397 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:54 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:18:54.397 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:54 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:18:54.397 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:18:53 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:18:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:18:54.398 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:54 smithi090 ceph-mon[41670]: pgmap v621: 248 pgs: 32 unknown, 216 active+clean; 8.5 MiB data, 585 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:18:54.398 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:54 smithi090 ceph-mon[41670]: osdmap e443: 8 total, 8 up, 8 in 2024-04-14T15:18:54.398 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:54 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:18:54.398 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:54 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:18:54.449 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:18:54.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:54.473 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:54.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:18:54.574 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:18:54.598 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:54.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:54.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:18:54.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:54 smithi092 ceph-mon[33966]: pgmap v621: 248 pgs: 32 unknown, 216 active+clean; 8.5 MiB data, 585 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:18:54.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:54 smithi092 ceph-mon[33966]: osdmap e443: 8 total, 8 up, 8 in 2024-04-14T15:18:54.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:54 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:18:54.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:54 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:18:54.700 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:18:54.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:54.724 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:54.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:18:54.825 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:18:54.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:54.849 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:54.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:18:54.950 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:54.973 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:54.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:54.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:55.075 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:18:55.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:55.099 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:55.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:18:55.200 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:18:55.224 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:55.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:55.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:18:55.289 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:55 smithi090 ceph-mon[41670]: osdmap e444: 8 total, 8 up, 8 in 2024-04-14T15:18:55.289 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:55 smithi090 ceph-mon[30399]: osdmap e444: 8 total, 8 up, 8 in 2024-04-14T15:18:55.326 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:18:55.349 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:55.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:55.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:18:55.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:55 smithi092 ceph-mon[33966]: osdmap e444: 8 total, 8 up, 8 in 2024-04-14T15:18:55.372 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:18:55 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:18:55.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.335+0000 7f0b92480700 1 -- 172.21.15.90:0/2983446191 >> v1:172.21.15.90:6789/0 conn(0x7f0b8c10e270 legacy=0x7f0b8c110710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:55.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.335+0000 7f0b92480700 1 -- 172.21.15.90:0/2983446191 shutdown_connections 2024-04-14T15:18:55.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.335+0000 7f0b92480700 1 -- 172.21.15.90:0/2983446191 >> 172.21.15.90:0/2983446191 conn(0x7f0b8c076aa0 msgr2=0x7f0b8c076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:18:55.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.335+0000 7f0b92480700 1 -- 172.21.15.90:0/2983446191 shutdown_connections 2024-04-14T15:18:55.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.335+0000 7f0b92480700 1 -- 172.21.15.90:0/2983446191 wait complete. 2024-04-14T15:18:55.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.335+0000 7f0b92480700 1 librados: starting msgr at 2024-04-14T15:18:55.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.335+0000 7f0b92480700 1 librados: starting objecter 2024-04-14T15:18:55.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.335+0000 7f0b92480700 1 Processor -- start 2024-04-14T15:18:55.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.335+0000 7f0b92480700 1 -- start start 2024-04-14T15:18:55.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.335+0000 7f0b92480700 1 librados: setting wanted keys 2024-04-14T15:18:55.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.335+0000 7f0b92480700 1 librados: calling monclient init 2024-04-14T15:18:55.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.336+0000 7f0b92480700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f0b8c1a68c0 con 0x7f0b8c1023f0 2024-04-14T15:18:55.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.336+0000 7f0b92480700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f0b8c1a7ae0 con 0x7f0b8c10e270 2024-04-14T15:18:55.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.336+0000 7f0b92480700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f0b8c1a8d20 con 0x7f0b8c10a6d0 2024-04-14T15:18:55.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.336+0000 7f0b8bfff700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f0b8c1023f0 0x7f0b8c106c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:53884/0 (socket says 172.21.15.90:53884) 2024-04-14T15:18:55.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.336+0000 7f0b8bfff700 1 -- 172.21.15.90:0/4012503850 learned_addr learned my addr 172.21.15.90:0/4012503850 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:18:55.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.336+0000 7f0b897fa700 10 client.?.objecter ms_handle_connect 0x7f0b8c1023f0 2024-04-14T15:18:55.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.336+0000 7f0b897fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:18:55.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.336+0000 7f0b897fa700 10 client.?.objecter ms_handle_connect 0x7f0b8c10e270 2024-04-14T15:18:55.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.336+0000 7f0b897fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:18:55.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.336+0000 7f0b897fa700 10 client.?.objecter ms_handle_connect 0x7f0b8c10a6d0 2024-04-14T15:18:55.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.336+0000 7f0b897fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:18:55.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.336+0000 7f0b897fa700 1 -- 172.21.15.90:0/4012503850 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2396122726 0 0) 0x7f0b8c1a68c0 con 0x7f0b8c1023f0 2024-04-14T15:18:55.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.336+0000 7f0b897fa700 1 -- 172.21.15.90:0/4012503850 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0b740031f0 con 0x7f0b8c1023f0 2024-04-14T15:18:55.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.336+0000 7f0b897fa700 1 -- 172.21.15.90:0/4012503850 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1999644485 0 0) 0x7f0b8c1a7ae0 con 0x7f0b8c10e270 2024-04-14T15:18:55.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.336+0000 7f0b897fa700 1 -- 172.21.15.90:0/4012503850 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0b8c1a68c0 con 0x7f0b8c10e270 2024-04-14T15:18:55.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.336+0000 7f0b897fa700 1 -- 172.21.15.90:0/4012503850 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2656123513 0 0) 0x7f0b8c1a8d20 con 0x7f0b8c10a6d0 2024-04-14T15:18:55.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.336+0000 7f0b897fa700 1 -- 172.21.15.90:0/4012503850 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0b8c1a7ae0 con 0x7f0b8c10a6d0 2024-04-14T15:18:55.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.336+0000 7f0b897fa700 1 -- 172.21.15.90:0/4012503850 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1370781417 0 0) 0x7f0b8c1a68c0 con 0x7f0b8c10e270 2024-04-14T15:18:55.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.336+0000 7f0b897fa700 1 -- 172.21.15.90:0/4012503850 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0b8c1a8d20 con 0x7f0b8c10e270 2024-04-14T15:18:55.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.336+0000 7f0b897fa700 1 -- 172.21.15.90:0/4012503850 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f0b84001bc0 con 0x7f0b8c10e270 2024-04-14T15:18:55.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.336+0000 7f0b897fa700 1 -- 172.21.15.90:0/4012503850 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1074765716 0 0) 0x7f0b740031f0 con 0x7f0b8c1023f0 2024-04-14T15:18:55.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.336+0000 7f0b897fa700 1 -- 172.21.15.90:0/4012503850 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0b8c1a68c0 con 0x7f0b8c1023f0 2024-04-14T15:18:55.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.336+0000 7f0b897fa700 1 -- 172.21.15.90:0/4012503850 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f0b7c0018b0 con 0x7f0b8c1023f0 2024-04-14T15:18:55.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.336+0000 7f0b897fa700 1 -- 172.21.15.90:0/4012503850 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2767765810 0 0) 0x7f0b8c1a7ae0 con 0x7f0b8c10a6d0 2024-04-14T15:18:55.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.336+0000 7f0b897fa700 1 -- 172.21.15.90:0/4012503850 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0b740031f0 con 0x7f0b8c10a6d0 2024-04-14T15:18:55.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.336+0000 7f0b897fa700 1 -- 172.21.15.90:0/4012503850 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f0b800012b0 con 0x7f0b8c10a6d0 2024-04-14T15:18:55.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.336+0000 7f0b897fa700 1 -- 172.21.15.90:0/4012503850 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2778605445 0 0) 0x7f0b8c1a8d20 con 0x7f0b8c10e270 2024-04-14T15:18:55.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.337+0000 7f0b897fa700 1 -- 172.21.15.90:0/4012503850 >> v1:172.21.15.90:6790/0 conn(0x7f0b8c10a6d0 legacy=0x7f0b8c1a16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:55.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.337+0000 7f0b897fa700 1 -- 172.21.15.90:0/4012503850 >> v1:172.21.15.90:6789/0 conn(0x7f0b8c1023f0 legacy=0x7f0b8c106c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:55.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.337+0000 7f0b897fa700 1 -- 172.21.15.90:0/4012503850 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b8c1a9f60 con 0x7f0b8c10e270 2024-04-14T15:18:55.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.337+0000 7f0b92480700 1 -- 172.21.15.90:0/4012503850 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0b8c1a7d10 con 0x7f0b8c10e270 2024-04-14T15:18:55.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.337+0000 7f0b92480700 10 client.49186.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:18:55.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.337+0000 7f0b92480700 1 -- 172.21.15.90:0/4012503850 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f0b8c1a81d0 con 0x7f0b8c10e270 2024-04-14T15:18:55.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.337+0000 7f0b92480700 1 librados: init done 2024-04-14T15:18:55.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.337+0000 7f0b897fa700 1 -- 172.21.15.90:0/4012503850 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f0b84001570 con 0x7f0b8c10e270 2024-04-14T15:18:55.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.337+0000 7f0b897fa700 1 -- 172.21.15.90:0/4012503850 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f0b84005040 con 0x7f0b8c10e270 2024-04-14T15:18:55.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.338+0000 7f0b92480700 1 -- 172.21.15.90:0/4012503850 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b78005120 con 0x7f0b8c10e270 2024-04-14T15:18:55.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.339+0000 7f0b897fa700 1 -- 172.21.15.90:0/4012503850 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f0b8401bcc0 con 0x7f0b8c10e270 2024-04-14T15:18:55.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.339+0000 7f0b897fa700 1 -- 172.21.15.90:0/4012503850 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(444..444 src has 51..444) ==== 8216+0+0 (unknown 3724863045 0 0) 0x7f0b84089b70 con 0x7f0b8c10e270 2024-04-14T15:18:55.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.339+0000 7f0b897fa700 10 client.49186.objecter ms_dispatch 0x7f0b8c001550 osd_map(444..444 src has 51..444) 2024-04-14T15:18:55.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.339+0000 7f0b897fa700 3 client.49186.objecter handle_osd_map got epochs [444,444] > 0 2024-04-14T15:18:55.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.339+0000 7f0b897fa700 3 client.49186.objecter handle_osd_map decoding full epoch 444 2024-04-14T15:18:55.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.339+0000 7f0b897fa700 20 client.49186.objecter dump_active .. 0 homeless 2024-04-14T15:18:55.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.339+0000 7f0b897fa700 10 client.49186.objecter ms_handle_connect 0x7f0b74073440 2024-04-14T15:18:55.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.344+0000 7f0b897fa700 1 -- 172.21.15.90:0/4012503850 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f0b84056550 con 0x7f0b8c10e270 2024-04-14T15:18:55.452 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:18:55.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:55.476 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:55.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:18:55.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.560+0000 7f0b92480700 1 -- 172.21.15.90:0/4012503850 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"} v 0) -- 0x7f0b78005410 con 0x7f0b8c10e270 2024-04-14T15:18:55.577 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:18:55.600 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:55.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:55.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:18:55.702 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:55.727 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:55.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:55.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:55.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.226+0000 7f29a7c82700 1 -- 172.21.15.90:0/539037577 >> v1:172.21.15.92:6789/0 conn(0x7f29a010e310 legacy=0x7f29a01107d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:55.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.226+0000 7f29a7c82700 1 -- 172.21.15.90:0/539037577 shutdown_connections 2024-04-14T15:18:55.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.226+0000 7f29a7c82700 1 -- 172.21.15.90:0/539037577 >> 172.21.15.90:0/539037577 conn(0x7f29a00fd290 msgr2=0x7f29a00ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:18:55.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.226+0000 7f29a7c82700 1 -- 172.21.15.90:0/539037577 shutdown_connections 2024-04-14T15:18:55.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.226+0000 7f29a7c82700 1 -- 172.21.15.90:0/539037577 wait complete. 2024-04-14T15:18:55.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.227+0000 7f29a7c82700 1 librados: starting msgr at 2024-04-14T15:18:55.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.227+0000 7f29a7c82700 1 librados: starting objecter 2024-04-14T15:18:55.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.227+0000 7f29a7c82700 1 Processor -- start 2024-04-14T15:18:55.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.227+0000 7f29a7c82700 1 -- start start 2024-04-14T15:18:55.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.227+0000 7f29a7c82700 1 librados: setting wanted keys 2024-04-14T15:18:55.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.227+0000 7f29a7c82700 1 librados: calling monclient init 2024-04-14T15:18:55.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.227+0000 7f29a7c82700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f29a01a68e0 con 0x7f29a010a740 2024-04-14T15:18:55.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.227+0000 7f29a7c82700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f29a01a7b00 con 0x7f29a0107890 2024-04-14T15:18:55.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.227+0000 7f29a7c82700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f29a01a8d40 con 0x7f29a010e310 2024-04-14T15:18:55.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.227+0000 7f29a621f700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f29a010e310 0x7f29a01a4fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:45134/0 (socket says 172.21.15.90:45134) 2024-04-14T15:18:55.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.227+0000 7f29a621f700 1 -- 172.21.15.90:0/3147868565 learned_addr learned my addr 172.21.15.90:0/3147868565 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:18:55.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.227+0000 7f2996ffd700 10 client.?.objecter ms_handle_connect 0x7f29a010a740 2024-04-14T15:18:55.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.227+0000 7f2996ffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:18:55.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.227+0000 7f2996ffd700 10 client.?.objecter ms_handle_connect 0x7f29a010e310 2024-04-14T15:18:55.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.227+0000 7f2996ffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:18:55.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.228+0000 7f2996ffd700 1 -- 172.21.15.90:0/3147868565 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 340075384 0 0) 0x7f29a01a68e0 con 0x7f29a010a740 2024-04-14T15:18:55.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.228+0000 7f2996ffd700 1 -- 172.21.15.90:0/3147868565 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f298c003aa0 con 0x7f29a010a740 2024-04-14T15:18:55.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.228+0000 7f2996ffd700 10 client.?.objecter ms_handle_connect 0x7f29a0107890 2024-04-14T15:18:55.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.228+0000 7f2996ffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:18:55.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.228+0000 7f2996ffd700 1 -- 172.21.15.90:0/3147868565 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1133661527 0 0) 0x7f29a01a8d40 con 0x7f29a010e310 2024-04-14T15:18:55.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.228+0000 7f2996ffd700 1 -- 172.21.15.90:0/3147868565 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29a01a68e0 con 0x7f29a010e310 2024-04-14T15:18:55.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.228+0000 7f2996ffd700 1 -- 172.21.15.90:0/3147868565 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 397287564 0 0) 0x7f29a01a7b00 con 0x7f29a0107890 2024-04-14T15:18:55.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.228+0000 7f2996ffd700 1 -- 172.21.15.90:0/3147868565 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29a01a8d40 con 0x7f29a0107890 2024-04-14T15:18:55.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.228+0000 7f2996ffd700 1 -- 172.21.15.90:0/3147868565 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3325490827 0 0) 0x7f298c003aa0 con 0x7f29a010a740 2024-04-14T15:18:55.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.228+0000 7f2996ffd700 1 -- 172.21.15.90:0/3147868565 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f29a01a7b00 con 0x7f29a010a740 2024-04-14T15:18:55.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.228+0000 7f2996ffd700 1 -- 172.21.15.90:0/3147868565 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f2990001a30 con 0x7f29a010a740 2024-04-14T15:18:55.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.228+0000 7f2996ffd700 1 -- 172.21.15.90:0/3147868565 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3398429092 0 0) 0x7f29a01a68e0 con 0x7f29a010e310 2024-04-14T15:18:55.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.228+0000 7f2996ffd700 1 -- 172.21.15.90:0/3147868565 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f298c003aa0 con 0x7f29a010e310 2024-04-14T15:18:55.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.228+0000 7f2996ffd700 1 -- 172.21.15.90:0/3147868565 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f2998001b60 con 0x7f29a010e310 2024-04-14T15:18:55.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.228+0000 7f2996ffd700 1 -- 172.21.15.90:0/3147868565 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 780555125 0 0) 0x7f29a01a8d40 con 0x7f29a0107890 2024-04-14T15:18:55.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.228+0000 7f2996ffd700 1 -- 172.21.15.90:0/3147868565 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f29a01a68e0 con 0x7f29a0107890 2024-04-14T15:18:55.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.228+0000 7f2996ffd700 1 -- 172.21.15.90:0/3147868565 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f299c001870 con 0x7f29a0107890 2024-04-14T15:18:55.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.228+0000 7f2996ffd700 1 -- 172.21.15.90:0/3147868565 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1707881325 0 0) 0x7f29a01a7b00 con 0x7f29a010a740 2024-04-14T15:18:55.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.228+0000 7f2996ffd700 1 -- 172.21.15.90:0/3147868565 >> v1:172.21.15.90:6790/0 conn(0x7f29a010e310 legacy=0x7f29a01a4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:55.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.228+0000 7f2996ffd700 1 -- 172.21.15.90:0/3147868565 >> v1:172.21.15.92:6789/0 conn(0x7f29a0107890 legacy=0x7f29a010da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:55.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.228+0000 7f2996ffd700 1 -- 172.21.15.90:0/3147868565 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29a01a9f80 con 0x7f29a010a740 2024-04-14T15:18:55.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.228+0000 7f29a7c82700 1 -- 172.21.15.90:0/3147868565 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f29a01a8f70 con 0x7f29a010a740 2024-04-14T15:18:55.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.228+0000 7f29a7c82700 10 client.37341.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:18:55.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.228+0000 7f29a7c82700 1 -- 172.21.15.90:0/3147868565 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f29a01a9430 con 0x7f29a010a740 2024-04-14T15:18:55.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.228+0000 7f29a7c82700 1 librados: init done 2024-04-14T15:18:55.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.228+0000 7f2996ffd700 1 -- 172.21.15.90:0/3147868565 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f2990001ce0 con 0x7f29a010a740 2024-04-14T15:18:55.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.228+0000 7f2996ffd700 1 -- 172.21.15.90:0/3147868565 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f2990004a70 con 0x7f29a010a740 2024-04-14T15:18:55.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.229+0000 7f29a7c82700 1 -- 172.21.15.90:0/3147868565 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f2984005130 con 0x7f29a010a740 2024-04-14T15:18:55.783 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:18:55.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.230+0000 7f2996ffd700 1 -- 172.21.15.90:0/3147868565 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f299001b480 con 0x7f29a010a740 2024-04-14T15:18:55.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.231+0000 7f2996ffd700 1 -- 172.21.15.90:0/3147868565 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(444..444 src has 51..444) ==== 8216+0+0 (unknown 3724863045 0 0) 0x7f29900896e0 con 0x7f29a010a740 2024-04-14T15:18:55.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.231+0000 7f2996ffd700 10 client.37341.objecter ms_dispatch 0x7f29a0001550 osd_map(444..444 src has 51..444) 2024-04-14T15:18:55.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.231+0000 7f2996ffd700 3 client.37341.objecter handle_osd_map got epochs [444,444] > 0 2024-04-14T15:18:55.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.231+0000 7f2996ffd700 3 client.37341.objecter handle_osd_map decoding full epoch 444 2024-04-14T15:18:55.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.231+0000 7f2996ffd700 20 client.37341.objecter dump_active .. 0 homeless 2024-04-14T15:18:55.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.231+0000 7f2996ffd700 10 client.37341.objecter ms_handle_connect 0x7f298c0734c0 2024-04-14T15:18:55.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.231+0000 7f2996ffd700 1 -- 172.21.15.90:0/3147868565 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v444) ==== 54+0+16 (unknown 1580004524 0 2420000192) 0x7f299008ba40 con 0x7f29a010a740 2024-04-14T15:18:55.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.231+0000 7f29a7c82700 10 client.37341.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:18:55.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.231+0000 7f29a7c82700 15 client.37341.objecter _session_command_op_assign -1 1 2024-04-14T15:18:55.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.231+0000 7f29a7c82700 20 client.37341.objecter _get_session s=0x7f2984005410 osd=0 3 2024-04-14T15:18:55.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.231+0000 7f29a7c82700 20 client.37341.objecter put_session s=0x7f2984005410 osd=0 3 2024-04-14T15:18:55.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.231+0000 7f29a7c82700 20 client.37341.objecter _get_session s=0x7f2984005410 osd=0 3 2024-04-14T15:18:55.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.231+0000 7f29a7c82700 15 client.37341.objecter _session_command_op_remove -1 1 2024-04-14T15:18:55.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.231+0000 7f29a7c82700 20 client.37341.objecter get_session s=0x7f2984005410 osd=0 3 2024-04-14T15:18:55.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.231+0000 7f29a7c82700 15 client.37341.objecter _session_command_op_assign 0 1 2024-04-14T15:18:55.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.231+0000 7f29a7c82700 20 client.37341.objecter put_session s=0x7f2984005410 osd=0 4 2024-04-14T15:18:55.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.231+0000 7f29a7c82700 10 client.37341.objecter _send_command 1 2024-04-14T15:18:55.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.231+0000 7f29a7c82700 1 -- 172.21.15.90:0/3147868565 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2984004ba0 con 0x7f2984004760 2024-04-14T15:18:55.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.232+0000 7f2996ffd700 10 client.37341.objecter ms_handle_connect 0x7f2984004760 2024-04-14T15:18:55.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.232+0000 7f2996ffd700 1 -- 172.21.15.90:0/3147868565 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2984004ba0 con 0x7f2984004760 2024-04-14T15:18:55.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.232+0000 7f2996ffd700 10 client.37341.objecter ms_dispatch 0x7f29a0001550 command_reply(tid 1: 0 ) 2024-04-14T15:18:55.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.232+0000 7f2996ffd700 10 client.37341.objecter _finish_command 1 = system:0 2024-04-14T15:18:55.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.232+0000 7f2996ffd700 20 client.37341.objecter put_session s=0x7f2984005410 osd=0 4 2024-04-14T15:18:55.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.232+0000 7f2996ffd700 15 client.37341.objecter _session_command_op_remove 0 1 2024-04-14T15:18:55.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.249+0000 7f29a7c82700 10 client.37341.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:18:55.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.249+0000 7f29a7c82700 15 client.37341.objecter _session_command_op_assign -1 2 2024-04-14T15:18:55.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.249+0000 7f29a7c82700 20 client.37341.objecter _get_session s=0x7f2984005410 osd=0 3 2024-04-14T15:18:55.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.249+0000 7f29a7c82700 20 client.37341.objecter put_session s=0x7f2984005410 osd=0 3 2024-04-14T15:18:55.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.249+0000 7f29a7c82700 20 client.37341.objecter _get_session s=0x7f2984005410 osd=0 3 2024-04-14T15:18:55.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.249+0000 7f29a7c82700 15 client.37341.objecter _session_command_op_remove -1 2 2024-04-14T15:18:55.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.249+0000 7f29a7c82700 20 client.37341.objecter get_session s=0x7f2984005410 osd=0 3 2024-04-14T15:18:55.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.249+0000 7f29a7c82700 15 client.37341.objecter _session_command_op_assign 0 2 2024-04-14T15:18:55.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.249+0000 7f29a7c82700 20 client.37341.objecter put_session s=0x7f2984005410 osd=0 4 2024-04-14T15:18:55.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.249+0000 7f29a7c82700 10 client.37341.objecter _send_command 2 2024-04-14T15:18:55.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.249+0000 7f29a7c82700 1 -- 172.21.15.90:0/3147868565 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f2984007a20 con 0x7f2984004760 2024-04-14T15:18:55.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.251+0000 7f2996ffd700 1 -- 172.21.15.90:0/3147868565 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2984007a20 con 0x7f2984004760 2024-04-14T15:18:55.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.251+0000 7f2996ffd700 10 client.37341.objecter ms_dispatch 0x7f29a0001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:18:55.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.251+0000 7f2996ffd700 10 client.37341.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:18:55.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.251+0000 7f2996ffd700 20 client.37341.objecter put_session s=0x7f2984005410 osd=0 4 2024-04-14T15:18:55.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.251+0000 7f2996ffd700 15 client.37341.objecter _session_command_op_remove 0 2 2024-04-14T15:18:55.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.251+0000 7f29a7c82700 10 client.37341.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:18:55.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.251+0000 7f29a7c82700 15 client.37341.objecter _session_command_op_assign -1 3 2024-04-14T15:18:55.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.251+0000 7f29a7c82700 20 client.37341.objecter _get_session s=0x7f2984004160 osd=1 3 2024-04-14T15:18:55.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.251+0000 7f29a7c82700 20 client.37341.objecter put_session s=0x7f2984004160 osd=1 3 2024-04-14T15:18:55.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.251+0000 7f29a7c82700 20 client.37341.objecter _get_session s=0x7f2984004160 osd=1 3 2024-04-14T15:18:55.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.251+0000 7f29a7c82700 15 client.37341.objecter _session_command_op_remove -1 3 2024-04-14T15:18:55.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.251+0000 7f29a7c82700 20 client.37341.objecter get_session s=0x7f2984004160 osd=1 3 2024-04-14T15:18:55.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.251+0000 7f29a7c82700 15 client.37341.objecter _session_command_op_assign 1 3 2024-04-14T15:18:55.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.251+0000 7f29a7c82700 20 client.37341.objecter put_session s=0x7f2984004160 osd=1 4 2024-04-14T15:18:55.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.251+0000 7f29a7c82700 10 client.37341.objecter _send_command 3 2024-04-14T15:18:55.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.251+0000 7f29a7c82700 1 -- 172.21.15.90:0/3147868565 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f29840022d0 con 0x7f2984004340 2024-04-14T15:18:55.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.251+0000 7f2996ffd700 10 client.37341.objecter ms_handle_connect 0x7f2984004340 2024-04-14T15:18:55.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.252+0000 7f2996ffd700 1 -- 172.21.15.90:0/3147868565 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f29840022d0 con 0x7f2984004340 2024-04-14T15:18:55.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.252+0000 7f2996ffd700 10 client.37341.objecter ms_dispatch 0x7f29a0001550 command_reply(tid 3: 0 ) 2024-04-14T15:18:55.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.252+0000 7f2996ffd700 10 client.37341.objecter _finish_command 3 = system:0 2024-04-14T15:18:55.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.252+0000 7f2996ffd700 20 client.37341.objecter put_session s=0x7f2984004160 osd=1 4 2024-04-14T15:18:55.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.252+0000 7f2996ffd700 15 client.37341.objecter _session_command_op_remove 1 3 2024-04-14T15:18:55.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.269+0000 7f29a7c82700 10 client.37341.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:18:55.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.269+0000 7f29a7c82700 15 client.37341.objecter _session_command_op_assign -1 4 2024-04-14T15:18:55.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.269+0000 7f29a7c82700 20 client.37341.objecter _get_session s=0x7f2984004160 osd=1 3 2024-04-14T15:18:55.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.269+0000 7f29a7c82700 20 client.37341.objecter put_session s=0x7f2984004160 osd=1 3 2024-04-14T15:18:55.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.269+0000 7f29a7c82700 20 client.37341.objecter _get_session s=0x7f2984004160 osd=1 3 2024-04-14T15:18:55.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.269+0000 7f29a7c82700 15 client.37341.objecter _session_command_op_remove -1 4 2024-04-14T15:18:55.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.269+0000 7f29a7c82700 20 client.37341.objecter get_session s=0x7f2984004160 osd=1 3 2024-04-14T15:18:55.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.269+0000 7f29a7c82700 15 client.37341.objecter _session_command_op_assign 1 4 2024-04-14T15:18:55.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.269+0000 7f29a7c82700 20 client.37341.objecter put_session s=0x7f2984004160 osd=1 4 2024-04-14T15:18:55.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.269+0000 7f29a7c82700 10 client.37341.objecter _send_command 4 2024-04-14T15:18:55.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.269+0000 7f29a7c82700 1 -- 172.21.15.90:0/3147868565 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f2984001750 con 0x7f2984004340 2024-04-14T15:18:55.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.271+0000 7f2996ffd700 1 -- 172.21.15.90:0/3147868565 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2984001750 con 0x7f2984004340 2024-04-14T15:18:55.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.271+0000 7f2996ffd700 10 client.37341.objecter ms_dispatch 0x7f29a0001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:18:55.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.271+0000 7f2996ffd700 10 client.37341.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:18:55.790 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:18:55.790 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:18:55.790 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:18:55.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.271+0000 7f2996ffd700 20 client.37341.objecter put_session s=0x7f2984004160 osd=1 4 2024-04-14T15:18:55.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.271+0000 7f2996ffd700 15 client.37341.objecter _session_command_op_remove 1 4 2024-04-14T15:18:55.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.271+0000 7f29a7c82700 10 client.37341.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:18:55.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.271+0000 7f29a7c82700 15 client.37341.objecter _session_command_op_assign -1 5 2024-04-14T15:18:55.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.271+0000 7f29a7c82700 20 client.37341.objecter _get_session s=0x7f29840019b0 osd=2 3 2024-04-14T15:18:55.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.271+0000 7f29a7c82700 20 client.37341.objecter put_session s=0x7f29840019b0 osd=2 3 2024-04-14T15:18:55.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.271+0000 7f29a7c82700 20 client.37341.objecter _get_session s=0x7f29840019b0 osd=2 3 2024-04-14T15:18:55.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.271+0000 7f29a7c82700 15 client.37341.objecter _session_command_op_remove -1 5 2024-04-14T15:18:55.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.271+0000 7f29a7c82700 20 client.37341.objecter get_session s=0x7f29840019b0 osd=2 3 2024-04-14T15:18:55.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.271+0000 7f29a7c82700 15 client.37341.objecter _session_command_op_assign 2 5 2024-04-14T15:18:55.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.271+0000 7f29a7c82700 20 client.37341.objecter put_session s=0x7f29840019b0 osd=2 4 2024-04-14T15:18:55.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.271+0000 7f29a7c82700 10 client.37341.objecter _send_command 5 2024-04-14T15:18:55.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.271+0000 7f29a7c82700 1 -- 172.21.15.90:0/3147868565 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f298400faa0 con 0x7f298400d600 2024-04-14T15:18:55.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.271+0000 7f2996ffd700 10 client.37341.objecter ms_handle_connect 0x7f298400d600 2024-04-14T15:18:55.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.272+0000 7f2996ffd700 1 -- 172.21.15.90:0/3147868565 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f298400faa0 con 0x7f298400d600 2024-04-14T15:18:55.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.272+0000 7f2996ffd700 10 client.37341.objecter ms_dispatch 0x7f29a0001550 command_reply(tid 5: 0 ) 2024-04-14T15:18:55.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.272+0000 7f2996ffd700 10 client.37341.objecter _finish_command 5 = system:0 2024-04-14T15:18:55.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.272+0000 7f2996ffd700 20 client.37341.objecter put_session s=0x7f29840019b0 osd=2 4 2024-04-14T15:18:55.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.272+0000 7f2996ffd700 15 client.37341.objecter _session_command_op_remove 2 5 2024-04-14T15:18:55.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.288+0000 7f29a7c82700 10 client.37341.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:18:55.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.288+0000 7f29a7c82700 15 client.37341.objecter _session_command_op_assign -1 6 2024-04-14T15:18:55.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.288+0000 7f29a7c82700 20 client.37341.objecter _get_session s=0x7f29840019b0 osd=2 3 2024-04-14T15:18:55.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.288+0000 7f29a7c82700 20 client.37341.objecter put_session s=0x7f29840019b0 osd=2 3 2024-04-14T15:18:55.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.288+0000 7f29a7c82700 20 client.37341.objecter _get_session s=0x7f29840019b0 osd=2 3 2024-04-14T15:18:55.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.288+0000 7f29a7c82700 15 client.37341.objecter _session_command_op_remove -1 6 2024-04-14T15:18:55.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.288+0000 7f29a7c82700 20 client.37341.objecter get_session s=0x7f29840019b0 osd=2 3 2024-04-14T15:18:55.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.288+0000 7f29a7c82700 15 client.37341.objecter _session_command_op_assign 2 6 2024-04-14T15:18:55.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.288+0000 7f29a7c82700 20 client.37341.objecter put_session s=0x7f29840019b0 osd=2 4 2024-04-14T15:18:55.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.288+0000 7f29a7c82700 10 client.37341.objecter _send_command 6 2024-04-14T15:18:55.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.288+0000 7f29a7c82700 1 -- 172.21.15.90:0/3147868565 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f298400fd10 con 0x7f298400d600 2024-04-14T15:18:55.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.290+0000 7f2996ffd700 1 -- 172.21.15.90:0/3147868565 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f298400fd10 con 0x7f298400d600 2024-04-14T15:18:55.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.290+0000 7f2996ffd700 10 client.37341.objecter ms_dispatch 0x7f29a0001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:18:55.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.290+0000 7f2996ffd700 10 client.37341.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:18:55.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.290+0000 7f2996ffd700 20 client.37341.objecter put_session s=0x7f29840019b0 osd=2 4 2024-04-14T15:18:55.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.290+0000 7f2996ffd700 15 client.37341.objecter _session_command_op_remove 2 6 2024-04-14T15:18:55.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.290+0000 7f29a7c82700 10 client.37341.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:18:55.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.290+0000 7f29a7c82700 15 client.37341.objecter _session_command_op_assign -1 7 2024-04-14T15:18:55.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.290+0000 7f29a7c82700 20 client.37341.objecter _get_session s=0x7f2984008310 osd=3 3 2024-04-14T15:18:55.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.290+0000 7f29a7c82700 20 client.37341.objecter put_session s=0x7f2984008310 osd=3 3 2024-04-14T15:18:55.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.290+0000 7f29a7c82700 20 client.37341.objecter _get_session s=0x7f2984008310 osd=3 3 2024-04-14T15:18:55.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.290+0000 7f29a7c82700 15 client.37341.objecter _session_command_op_remove -1 7 2024-04-14T15:18:55.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.290+0000 7f29a7c82700 20 client.37341.objecter get_session s=0x7f2984008310 osd=3 3 2024-04-14T15:18:55.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.290+0000 7f29a7c82700 15 client.37341.objecter _session_command_op_assign 3 7 2024-04-14T15:18:55.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.290+0000 7f29a7c82700 20 client.37341.objecter put_session s=0x7f2984008310 osd=3 4 2024-04-14T15:18:55.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.290+0000 7f29a7c82700 10 client.37341.objecter _send_command 7 2024-04-14T15:18:55.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.290+0000 7f29a7c82700 1 -- 172.21.15.90:0/3147868565 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f298401a370 con 0x7f298400d160 2024-04-14T15:18:55.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.290+0000 7f2996ffd700 10 client.37341.objecter ms_handle_connect 0x7f298400d160 2024-04-14T15:18:55.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.291+0000 7f2996ffd700 1 -- 172.21.15.90:0/3147868565 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f298401a370 con 0x7f298400d160 2024-04-14T15:18:55.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.291+0000 7f2996ffd700 10 client.37341.objecter ms_dispatch 0x7f29a0001550 command_reply(tid 7: 0 ) 2024-04-14T15:18:55.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.291+0000 7f2996ffd700 10 client.37341.objecter _finish_command 7 = system:0 2024-04-14T15:18:55.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.291+0000 7f2996ffd700 20 client.37341.objecter put_session s=0x7f2984008310 osd=3 4 2024-04-14T15:18:55.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.291+0000 7f2996ffd700 15 client.37341.objecter _session_command_op_remove 3 7 2024-04-14T15:18:55.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.308+0000 7f29a7c82700 10 client.37341.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:18:55.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.308+0000 7f29a7c82700 15 client.37341.objecter _session_command_op_assign -1 8 2024-04-14T15:18:55.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.308+0000 7f29a7c82700 20 client.37341.objecter _get_session s=0x7f2984008310 osd=3 3 2024-04-14T15:18:55.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.308+0000 7f29a7c82700 20 client.37341.objecter put_session s=0x7f2984008310 osd=3 3 2024-04-14T15:18:55.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.308+0000 7f29a7c82700 20 client.37341.objecter _get_session s=0x7f2984008310 osd=3 3 2024-04-14T15:18:55.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.308+0000 7f29a7c82700 15 client.37341.objecter _session_command_op_remove -1 8 2024-04-14T15:18:55.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.308+0000 7f29a7c82700 20 client.37341.objecter get_session s=0x7f2984008310 osd=3 3 2024-04-14T15:18:55.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.308+0000 7f29a7c82700 15 client.37341.objecter _session_command_op_assign 3 8 2024-04-14T15:18:55.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.308+0000 7f29a7c82700 20 client.37341.objecter put_session s=0x7f2984008310 osd=3 4 2024-04-14T15:18:55.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.308+0000 7f29a7c82700 10 client.37341.objecter _send_command 8 2024-04-14T15:18:55.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.308+0000 7f29a7c82700 1 -- 172.21.15.90:0/3147868565 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f298400c790 con 0x7f298400d160 2024-04-14T15:18:55.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.310+0000 7f2996ffd700 1 -- 172.21.15.90:0/3147868565 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f298400c790 con 0x7f298400d160 2024-04-14T15:18:55.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.310+0000 7f2996ffd700 10 client.37341.objecter ms_dispatch 0x7f29a0001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:18:55.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.310+0000 7f2996ffd700 10 client.37341.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:18:55.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.310+0000 7f2996ffd700 20 client.37341.objecter put_session s=0x7f2984008310 osd=3 4 2024-04-14T15:18:55.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.310+0000 7f2996ffd700 15 client.37341.objecter _session_command_op_remove 3 8 2024-04-14T15:18:55.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.310+0000 7f29a7c82700 10 client.37341.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:18:55.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.310+0000 7f29a7c82700 15 client.37341.objecter _session_command_op_assign -1 9 2024-04-14T15:18:55.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.310+0000 7f29a7c82700 20 client.37341.objecter _get_session s=0x7f298400c790 osd=4 3 2024-04-14T15:18:55.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.310+0000 7f29a7c82700 20 client.37341.objecter put_session s=0x7f298400c790 osd=4 3 2024-04-14T15:18:55.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.310+0000 7f29a7c82700 20 client.37341.objecter _get_session s=0x7f298400c790 osd=4 3 2024-04-14T15:18:55.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.310+0000 7f29a7c82700 15 client.37341.objecter _session_command_op_remove -1 9 2024-04-14T15:18:55.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.310+0000 7f29a7c82700 20 client.37341.objecter get_session s=0x7f298400c790 osd=4 3 2024-04-14T15:18:55.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.310+0000 7f29a7c82700 15 client.37341.objecter _session_command_op_assign 4 9 2024-04-14T15:18:55.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.310+0000 7f29a7c82700 20 client.37341.objecter put_session s=0x7f298400c790 osd=4 4 2024-04-14T15:18:55.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.310+0000 7f29a7c82700 10 client.37341.objecter _send_command 9 2024-04-14T15:18:55.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.310+0000 7f29a7c82700 1 -- 172.21.15.90:0/3147868565 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f298400ca60 con 0x7f2984016220 2024-04-14T15:18:55.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.311+0000 7f2996ffd700 10 client.37341.objecter ms_handle_connect 0x7f2984016220 2024-04-14T15:18:55.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.311+0000 7f2996ffd700 1 -- 172.21.15.90:0/3147868565 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f298400ca60 con 0x7f2984016220 2024-04-14T15:18:55.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.311+0000 7f2996ffd700 10 client.37341.objecter ms_dispatch 0x7f29a0001550 command_reply(tid 9: 0 ) 2024-04-14T15:18:55.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.311+0000 7f2996ffd700 10 client.37341.objecter _finish_command 9 = system:0 2024-04-14T15:18:55.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.311+0000 7f2996ffd700 20 client.37341.objecter put_session s=0x7f298400c790 osd=4 4 2024-04-14T15:18:55.797 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:18:55.797 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:18:55.797 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:18:55.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.311+0000 7f2996ffd700 15 client.37341.objecter _session_command_op_remove 4 9 2024-04-14T15:18:55.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.328+0000 7f29a7c82700 10 client.37341.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:18:55.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.328+0000 7f29a7c82700 15 client.37341.objecter _session_command_op_assign -1 10 2024-04-14T15:18:55.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.328+0000 7f29a7c82700 20 client.37341.objecter _get_session s=0x7f298400c790 osd=4 3 2024-04-14T15:18:55.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.328+0000 7f29a7c82700 20 client.37341.objecter put_session s=0x7f298400c790 osd=4 3 2024-04-14T15:18:55.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.328+0000 7f29a7c82700 20 client.37341.objecter _get_session s=0x7f298400c790 osd=4 3 2024-04-14T15:18:55.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.328+0000 7f29a7c82700 15 client.37341.objecter _session_command_op_remove -1 10 2024-04-14T15:18:55.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.328+0000 7f29a7c82700 20 client.37341.objecter get_session s=0x7f298400c790 osd=4 3 2024-04-14T15:18:55.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.328+0000 7f29a7c82700 15 client.37341.objecter _session_command_op_assign 4 10 2024-04-14T15:18:55.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.328+0000 7f29a7c82700 20 client.37341.objecter put_session s=0x7f298400c790 osd=4 4 2024-04-14T15:18:55.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.328+0000 7f29a7c82700 10 client.37341.objecter _send_command 10 2024-04-14T15:18:55.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.328+0000 7f29a7c82700 1 -- 172.21.15.90:0/3147868565 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f2984016700 con 0x7f2984016220 2024-04-14T15:18:55.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.330+0000 7f2996ffd700 1 -- 172.21.15.90:0/3147868565 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2984016700 con 0x7f2984016220 2024-04-14T15:18:55.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.330+0000 7f2996ffd700 10 client.37341.objecter ms_dispatch 0x7f29a0001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:18:55.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.330+0000 7f2996ffd700 10 client.37341.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:18:55.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.330+0000 7f2996ffd700 20 client.37341.objecter put_session s=0x7f298400c790 osd=4 4 2024-04-14T15:18:55.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.330+0000 7f2996ffd700 15 client.37341.objecter _session_command_op_remove 4 10 2024-04-14T15:18:55.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.330+0000 7f29a7c82700 10 client.37341.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:18:55.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.330+0000 7f29a7c82700 15 client.37341.objecter _session_command_op_assign -1 11 2024-04-14T15:18:55.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.330+0000 7f29a7c82700 20 client.37341.objecter _get_session s=0x7f298400ca60 osd=5 3 2024-04-14T15:18:55.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.330+0000 7f29a7c82700 20 client.37341.objecter put_session s=0x7f298400ca60 osd=5 3 2024-04-14T15:18:55.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.330+0000 7f29a7c82700 20 client.37341.objecter _get_session s=0x7f298400ca60 osd=5 3 2024-04-14T15:18:55.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.330+0000 7f29a7c82700 15 client.37341.objecter _session_command_op_remove -1 11 2024-04-14T15:18:55.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.330+0000 7f29a7c82700 20 client.37341.objecter get_session s=0x7f298400ca60 osd=5 3 2024-04-14T15:18:55.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.330+0000 7f29a7c82700 15 client.37341.objecter _session_command_op_assign 5 11 2024-04-14T15:18:55.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.330+0000 7f29a7c82700 20 client.37341.objecter put_session s=0x7f298400ca60 osd=5 4 2024-04-14T15:18:55.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.330+0000 7f29a7c82700 10 client.37341.objecter _send_command 11 2024-04-14T15:18:55.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.330+0000 7f29a7c82700 1 -- 172.21.15.90:0/3147868565 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f29840231e0 con 0x7f2984016700 2024-04-14T15:18:55.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.330+0000 7f2996ffd700 10 client.37341.objecter ms_handle_connect 0x7f2984016700 2024-04-14T15:18:55.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.331+0000 7f2996ffd700 1 -- 172.21.15.90:0/3147868565 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f29840231e0 con 0x7f2984016700 2024-04-14T15:18:55.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.331+0000 7f2996ffd700 10 client.37341.objecter ms_dispatch 0x7f29a0001550 command_reply(tid 11: 0 ) 2024-04-14T15:18:55.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.331+0000 7f2996ffd700 10 client.37341.objecter _finish_command 11 = system:0 2024-04-14T15:18:55.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.331+0000 7f2996ffd700 20 client.37341.objecter put_session s=0x7f298400ca60 osd=5 4 2024-04-14T15:18:55.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.331+0000 7f2996ffd700 15 client.37341.objecter _session_command_op_remove 5 11 2024-04-14T15:18:55.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.350+0000 7f29a7c82700 10 client.37341.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:18:55.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.350+0000 7f29a7c82700 15 client.37341.objecter _session_command_op_assign -1 12 2024-04-14T15:18:55.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.350+0000 7f29a7c82700 20 client.37341.objecter _get_session s=0x7f298400ca60 osd=5 3 2024-04-14T15:18:55.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.350+0000 7f29a7c82700 20 client.37341.objecter put_session s=0x7f298400ca60 osd=5 3 2024-04-14T15:18:55.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.350+0000 7f29a7c82700 20 client.37341.objecter _get_session s=0x7f298400ca60 osd=5 3 2024-04-14T15:18:55.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.350+0000 7f29a7c82700 15 client.37341.objecter _session_command_op_remove -1 12 2024-04-14T15:18:55.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.350+0000 7f29a7c82700 20 client.37341.objecter get_session s=0x7f298400ca60 osd=5 3 2024-04-14T15:18:55.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.350+0000 7f29a7c82700 15 client.37341.objecter _session_command_op_assign 5 12 2024-04-14T15:18:55.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.350+0000 7f29a7c82700 20 client.37341.objecter put_session s=0x7f298400ca60 osd=5 4 2024-04-14T15:18:55.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.350+0000 7f29a7c82700 10 client.37341.objecter _send_command 12 2024-04-14T15:18:55.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.350+0000 7f29a7c82700 1 -- 172.21.15.90:0/3147868565 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f2984015ca0 con 0x7f2984016700 2024-04-14T15:18:55.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.351+0000 7f2996ffd700 1 -- 172.21.15.90:0/3147868565 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2984015ca0 con 0x7f2984016700 2024-04-14T15:18:55.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.351+0000 7f2996ffd700 10 client.37341.objecter ms_dispatch 0x7f29a0001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:18:55.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.351+0000 7f2996ffd700 10 client.37341.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:18:55.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.351+0000 7f2996ffd700 20 client.37341.objecter put_session s=0x7f298400ca60 osd=5 4 2024-04-14T15:18:55.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.351+0000 7f2996ffd700 15 client.37341.objecter _session_command_op_remove 5 12 2024-04-14T15:18:55.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.352+0000 7f29a7c82700 10 client.37341.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:18:55.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.352+0000 7f29a7c82700 15 client.37341.objecter _session_command_op_assign -1 13 2024-04-14T15:18:55.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.352+0000 7f29a7c82700 20 client.37341.objecter _get_session s=0x7f298400c600 osd=6 3 2024-04-14T15:18:55.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.352+0000 7f29a7c82700 20 client.37341.objecter put_session s=0x7f298400c600 osd=6 3 2024-04-14T15:18:55.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.352+0000 7f29a7c82700 20 client.37341.objecter _get_session s=0x7f298400c600 osd=6 3 2024-04-14T15:18:55.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.352+0000 7f29a7c82700 15 client.37341.objecter _session_command_op_remove -1 13 2024-04-14T15:18:55.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.352+0000 7f29a7c82700 20 client.37341.objecter get_session s=0x7f298400c600 osd=6 3 2024-04-14T15:18:55.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.352+0000 7f29a7c82700 15 client.37341.objecter _session_command_op_assign 6 13 2024-04-14T15:18:55.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.352+0000 7f29a7c82700 20 client.37341.objecter put_session s=0x7f298400c600 osd=6 4 2024-04-14T15:18:55.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.352+0000 7f29a7c82700 10 client.37341.objecter _send_command 13 2024-04-14T15:18:55.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.352+0000 7f29a7c82700 1 -- 172.21.15.90:0/3147868565 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f298401b520 con 0x7f2984015a50 2024-04-14T15:18:55.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.352+0000 7f2996ffd700 10 client.37341.objecter ms_handle_connect 0x7f2984015a50 2024-04-14T15:18:55.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.353+0000 7f2996ffd700 1 -- 172.21.15.90:0/3147868565 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f298401b520 con 0x7f2984015a50 2024-04-14T15:18:55.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.353+0000 7f2996ffd700 10 client.37341.objecter ms_dispatch 0x7f29a0001550 command_reply(tid 13: 0 ) 2024-04-14T15:18:55.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.353+0000 7f2996ffd700 10 client.37341.objecter _finish_command 13 = system:0 2024-04-14T15:18:55.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.353+0000 7f2996ffd700 20 client.37341.objecter put_session s=0x7f298400c600 osd=6 4 2024-04-14T15:18:55.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.353+0000 7f2996ffd700 15 client.37341.objecter _session_command_op_remove 6 13 2024-04-14T15:18:55.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.371+0000 7f29a7c82700 10 client.37341.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:18:55.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.371+0000 7f29a7c82700 15 client.37341.objecter _session_command_op_assign -1 14 2024-04-14T15:18:55.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.371+0000 7f29a7c82700 20 client.37341.objecter _get_session s=0x7f298400c600 osd=6 3 2024-04-14T15:18:55.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.371+0000 7f29a7c82700 20 client.37341.objecter put_session s=0x7f298400c600 osd=6 3 2024-04-14T15:18:55.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.371+0000 7f29a7c82700 20 client.37341.objecter _get_session s=0x7f298400c600 osd=6 3 2024-04-14T15:18:55.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.371+0000 7f29a7c82700 15 client.37341.objecter _session_command_op_remove -1 14 2024-04-14T15:18:55.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.371+0000 7f29a7c82700 20 client.37341.objecter get_session s=0x7f298400c600 osd=6 3 2024-04-14T15:18:55.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.371+0000 7f29a7c82700 15 client.37341.objecter _session_command_op_assign 6 14 2024-04-14T15:18:55.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.371+0000 7f29a7c82700 20 client.37341.objecter put_session s=0x7f298400c600 osd=6 4 2024-04-14T15:18:55.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.371+0000 7f29a7c82700 10 client.37341.objecter _send_command 14 2024-04-14T15:18:55.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.371+0000 7f29a7c82700 1 -- 172.21.15.90:0/3147868565 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f298401b520 con 0x7f2984015a50 2024-04-14T15:18:55.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.373+0000 7f2996ffd700 1 -- 172.21.15.90:0/3147868565 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f298401b520 con 0x7f2984015a50 2024-04-14T15:18:55.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.373+0000 7f2996ffd700 10 client.37341.objecter ms_dispatch 0x7f29a0001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:18:55.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.373+0000 7f2996ffd700 10 client.37341.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:18:55.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.373+0000 7f2996ffd700 20 client.37341.objecter put_session s=0x7f298400c600 osd=6 4 2024-04-14T15:18:55.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.373+0000 7f2996ffd700 15 client.37341.objecter _session_command_op_remove 6 14 2024-04-14T15:18:55.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.373+0000 7f29a7c82700 10 client.37341.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:18:55.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.373+0000 7f29a7c82700 15 client.37341.objecter _session_command_op_assign -1 15 2024-04-14T15:18:55.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.373+0000 7f29a7c82700 20 client.37341.objecter _get_session s=0x7f298401b520 osd=7 3 2024-04-14T15:18:55.804 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:18:55.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.373+0000 7f29a7c82700 20 client.37341.objecter put_session s=0x7f298401b520 osd=7 3 2024-04-14T15:18:55.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.373+0000 7f29a7c82700 20 client.37341.objecter _get_session s=0x7f298401b520 osd=7 3 2024-04-14T15:18:55.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.373+0000 7f29a7c82700 15 client.37341.objecter _session_command_op_remove -1 15 2024-04-14T15:18:55.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.373+0000 7f29a7c82700 20 client.37341.objecter get_session s=0x7f298401b520 osd=7 3 2024-04-14T15:18:55.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.373+0000 7f29a7c82700 15 client.37341.objecter _session_command_op_assign 7 15 2024-04-14T15:18:55.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.373+0000 7f29a7c82700 20 client.37341.objecter put_session s=0x7f298401b520 osd=7 4 2024-04-14T15:18:55.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.373+0000 7f29a7c82700 10 client.37341.objecter _send_command 15 2024-04-14T15:18:55.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.373+0000 7f29a7c82700 1 -- 172.21.15.90:0/3147868565 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f298402bbd0 con 0x7f29840280c0 2024-04-14T15:18:55.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.374+0000 7f2996ffd700 10 client.37341.objecter ms_handle_connect 0x7f29840280c0 2024-04-14T15:18:55.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.374+0000 7f2996ffd700 1 -- 172.21.15.90:0/3147868565 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f298402bbd0 con 0x7f29840280c0 2024-04-14T15:18:55.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.374+0000 7f2996ffd700 10 client.37341.objecter ms_dispatch 0x7f29a0001550 command_reply(tid 15: 0 ) 2024-04-14T15:18:55.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.374+0000 7f2996ffd700 10 client.37341.objecter _finish_command 15 = system:0 2024-04-14T15:18:55.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.374+0000 7f2996ffd700 20 client.37341.objecter put_session s=0x7f298401b520 osd=7 4 2024-04-14T15:18:55.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.374+0000 7f2996ffd700 15 client.37341.objecter _session_command_op_remove 7 15 2024-04-14T15:18:55.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.391+0000 7f29a7c82700 10 client.37341.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:18:55.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.391+0000 7f29a7c82700 15 client.37341.objecter _session_command_op_assign -1 16 2024-04-14T15:18:55.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.391+0000 7f29a7c82700 20 client.37341.objecter _get_session s=0x7f298401b520 osd=7 3 2024-04-14T15:18:55.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.391+0000 7f29a7c82700 20 client.37341.objecter put_session s=0x7f298401b520 osd=7 3 2024-04-14T15:18:55.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.391+0000 7f29a7c82700 20 client.37341.objecter _get_session s=0x7f298401b520 osd=7 3 2024-04-14T15:18:55.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.391+0000 7f29a7c82700 15 client.37341.objecter _session_command_op_remove -1 16 2024-04-14T15:18:55.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.391+0000 7f29a7c82700 20 client.37341.objecter get_session s=0x7f298401b520 osd=7 3 2024-04-14T15:18:55.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.391+0000 7f29a7c82700 15 client.37341.objecter _session_command_op_assign 7 16 2024-04-14T15:18:55.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.391+0000 7f29a7c82700 20 client.37341.objecter put_session s=0x7f298401b520 osd=7 4 2024-04-14T15:18:55.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.391+0000 7f29a7c82700 10 client.37341.objecter _send_command 16 2024-04-14T15:18:55.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.391+0000 7f29a7c82700 1 -- 172.21.15.90:0/3147868565 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f2984000990 con 0x7f29840280c0 2024-04-14T15:18:55.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.393+0000 7f2996ffd700 1 -- 172.21.15.90:0/3147868565 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2984000990 con 0x7f29840280c0 2024-04-14T15:18:55.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.393+0000 7f2996ffd700 10 client.37341.objecter ms_dispatch 0x7f29a0001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:18:55.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.393+0000 7f2996ffd700 10 client.37341.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:18:55.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.393+0000 7f2996ffd700 20 client.37341.objecter put_session s=0x7f298401b520 osd=7 4 2024-04-14T15:18:55.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.393+0000 7f2996ffd700 15 client.37341.objecter _session_command_op_remove 7 16 2024-04-14T15:18:55.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.393+0000 7f29a7c82700 10 librados: watch_flush enter 2024-04-14T15:18:55.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.393+0000 7f29a7c82700 10 librados: watch_flush exit 2024-04-14T15:18:55.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.393+0000 7f29a7c82700 10 client.37341.objecter close_session for osd.0 2024-04-14T15:18:55.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.393+0000 7f29a7c82700 1 -- 172.21.15.90:0/3147868565 >> v1:172.21.15.90:6801/3761806885 conn(0x7f2984004760 legacy=0x7f2984002780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:55.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.393+0000 7f29a7c82700 20 client.37341.objecter put_session s=0x7f2984005410 osd=0 1 2024-04-14T15:18:55.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.393+0000 7f29a7c82700 10 client.37341.objecter close_session for osd.1 2024-04-14T15:18:55.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.393+0000 7f29a7c82700 1 -- 172.21.15.90:0/3147868565 >> v1:172.21.15.90:6805/4170361699 conn(0x7f2984004340 legacy=0x7f298400ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:55.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.394+0000 7f29a7c82700 20 client.37341.objecter put_session s=0x7f2984004160 osd=1 1 2024-04-14T15:18:55.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.394+0000 7f29a7c82700 10 client.37341.objecter close_session for osd.2 2024-04-14T15:18:55.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.394+0000 7f29a7c82700 1 -- 172.21.15.90:0/3147868565 >> v1:172.21.15.90:6809/1686184251 conn(0x7f298400d600 legacy=0x7f2984014330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:55.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.394+0000 7f29a7c82700 20 client.37341.objecter put_session s=0x7f29840019b0 osd=2 1 2024-04-14T15:18:55.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.394+0000 7f29a7c82700 10 client.37341.objecter close_session for osd.3 2024-04-14T15:18:55.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.394+0000 7f29a7c82700 1 -- 172.21.15.90:0/3147868565 >> v1:172.21.15.90:6813/2270166774 conn(0x7f298400d160 legacy=0x7f2984018be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:55.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.394+0000 7f29a7c82700 20 client.37341.objecter put_session s=0x7f2984008310 osd=3 1 2024-04-14T15:18:55.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.394+0000 7f29a7c82700 10 client.37341.objecter close_session for osd.4 2024-04-14T15:18:55.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.394+0000 7f29a7c82700 1 -- 172.21.15.90:0/3147868565 >> v1:172.21.15.92:6800/3887549733 conn(0x7f2984016220 legacy=0x7f298401dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:55.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.394+0000 7f29a7c82700 20 client.37341.objecter put_session s=0x7f298400c790 osd=4 1 2024-04-14T15:18:55.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.394+0000 7f29a7c82700 10 client.37341.objecter close_session for osd.5 2024-04-14T15:18:55.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.394+0000 7f29a7c82700 1 -- 172.21.15.90:0/3147868565 >> v1:172.21.15.92:6804/3244561440 conn(0x7f2984016700 legacy=0x7f2984021970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:55.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.394+0000 7f29a7c82700 20 client.37341.objecter put_session s=0x7f298400ca60 osd=5 1 2024-04-14T15:18:55.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.394+0000 7f29a7c82700 10 client.37341.objecter close_session for osd.6 2024-04-14T15:18:55.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.394+0000 7f29a7c82700 1 -- 172.21.15.90:0/3147868565 >> v1:172.21.15.92:6808/636125963 conn(0x7f2984015a50 legacy=0x7f298401ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:55.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.394+0000 7f29a7c82700 20 client.37341.objecter put_session s=0x7f298400c600 osd=6 1 2024-04-14T15:18:55.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.394+0000 7f29a7c82700 10 client.37341.objecter close_session for osd.7 2024-04-14T15:18:55.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.394+0000 7f29a7c82700 1 -- 172.21.15.90:0/3147868565 >> v1:172.21.15.92:6812/1773706716 conn(0x7f29840280c0 legacy=0x7f298402a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:55.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.394+0000 7f29a7c82700 20 client.37341.objecter put_session s=0x7f298401b520 osd=7 1 2024-04-14T15:18:55.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.394+0000 7f29a7c82700 20 client.37341.objecter shutdown clearing up homeless session... 2024-04-14T15:18:55.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.394+0000 7f29a7c82700 10 client.37341.objecter successfully canceled tick 2024-04-14T15:18:55.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.394+0000 7f29a621f700 1 -- 172.21.15.90:0/3147868565 reap_dead start 2024-04-14T15:18:55.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.394+0000 7f29a621f700 1 -- 172.21.15.90:0/3147868565 reap_dead start 2024-04-14T15:18:55.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.394+0000 7f29a621f700 1 -- 172.21.15.90:0/3147868565 reap_dead start 2024-04-14T15:18:55.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.394+0000 7f29a621f700 1 -- 172.21.15.90:0/3147868565 reap_dead start 2024-04-14T15:18:55.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.394+0000 7f29a621f700 1 -- 172.21.15.90:0/3147868565 reap_dead start 2024-04-14T15:18:55.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.395+0000 7f29a7c82700 1 -- 172.21.15.90:0/3147868565 >> v1:172.21.15.90:6800/420935427 conn(0x7f298c0734c0 legacy=0x7f298c075980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:55.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.395+0000 7f29a7c82700 1 -- 172.21.15.90:0/3147868565 >> v1:172.21.15.90:6789/0 conn(0x7f29a010a740 legacy=0x7f29a01a17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:55.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.395+0000 7f29a7c82700 1 -- 172.21.15.90:0/3147868565 shutdown_connections 2024-04-14T15:18:55.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.395+0000 7f29a7c82700 1 -- 172.21.15.90:0/3147868565 >> 172.21.15.90:0/3147868565 conn(0x7f29a00fd290 msgr2=0x7f29a0110fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:18:55.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.395+0000 7f29a7c82700 1 -- 172.21.15.90:0/3147868565 shutdown_connections 2024-04-14T15:18:55.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.395+0000 7f29a7c82700 1 -- 172.21.15.90:0/3147868565 wait complete. 2024-04-14T15:18:55.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:55.395+0000 7f29a7c82700 1 librados: shutdown 2024-04-14T15:18:55.809 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:18:55.809 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:18:55.809 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:18:55.809 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:18:55.809 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:18:55.809 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:18:55.809 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:18:55.809 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:18:55.829 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:18:55.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:55.854 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:55.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:18:55.955 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:18:55.980 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:55.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:55.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:18:56.082 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:18:56.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:56.107 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:56.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:18:56.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:56.157+0000 7f0b897fa700 1 -- 172.21.15.90:0/4012503850 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]=0 set pool 1 pg_num to 31 v445) ==== 129+0+0 (unknown 3196873580 0 0) 0x7f0b8405a200 con 0x7f0b8c10e270 2024-04-14T15:18:56.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:56.161+0000 7f0b92480700 10 librados: watch_flush enter 2024-04-14T15:18:56.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:56.161+0000 7f0b92480700 10 librados: watch_flush exit 2024-04-14T15:18:56.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:56.161+0000 7f0b92480700 20 client.49186.objecter shutdown clearing up homeless session... 2024-04-14T15:18:56.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:56.161+0000 7f0b92480700 10 client.49186.objecter successfully canceled tick 2024-04-14T15:18:56.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:56.161+0000 7f0b92480700 1 -- 172.21.15.90:0/4012503850 >> v1:172.21.15.90:6800/420935427 conn(0x7f0b74073440 legacy=0x7f0b74075900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:56.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:56.161+0000 7f0b92480700 1 -- 172.21.15.90:0/4012503850 >> v1:172.21.15.92:6789/0 conn(0x7f0b8c10e270 legacy=0x7f0b8c1a4f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:18:56.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:56.161+0000 7f0b92480700 1 -- 172.21.15.90:0/4012503850 shutdown_connections 2024-04-14T15:18:56.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:56.161+0000 7f0b92480700 1 -- 172.21.15.90:0/4012503850 >> 172.21.15.90:0/4012503850 conn(0x7f0b8c076aa0 msgr2=0x7f0b8c111a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:18:56.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:56.161+0000 7f0b92480700 1 -- 172.21.15.90:0/4012503850 shutdown_connections 2024-04-14T15:18:56.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:56.161+0000 7f0b92480700 1 -- 172.21.15.90:0/4012503850 wait complete. 2024-04-14T15:18:56.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:18:56.161+0000 7f0b92480700 1 librados: shutdown 2024-04-14T15:18:56.163 INFO:teuthology.orchestra.run.smithi090.stderr:set pool 1 pg_num to 31 2024-04-14T15:18:56.208 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:18:56.222 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-14T15:18:56.223 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-14T15:18:56.223 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-14T15:18:56.223 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-14T15:18:56.223 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-14T15:18:56.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:56.235 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:56.235 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:18:56.336 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:18:56.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:56.363 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:56.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:18:56.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:56 smithi090 ceph-mon[41670]: pgmap v624: 248 pgs: 248 active+clean; 8.5 MiB data, 586 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2.5 KiB/s wr, 7 op/s 2024-04-14T15:18:56.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:56 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:18:56.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:56 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3147868565' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:18:56.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:56 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/4012503850' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]: dispatch 2024-04-14T15:18:56.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:56 smithi090 ceph-mon[41670]: from='client.49186 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]: dispatch 2024-04-14T15:18:56.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:56 smithi090 ceph-mon[30399]: pgmap v624: 248 pgs: 248 active+clean; 8.5 MiB data, 586 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2.5 KiB/s wr, 7 op/s 2024-04-14T15:18:56.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:56 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:18:56.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:56 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3147868565' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:18:56.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:56 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/4012503850' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]: dispatch 2024-04-14T15:18:56.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:56 smithi090 ceph-mon[30399]: from='client.49186 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]: dispatch 2024-04-14T15:18:56.464 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:18:56.488 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:56.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:56.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:18:56.590 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:18:56.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:56.614 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:56.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:18:56.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:56 smithi092 ceph-mon[33966]: pgmap v624: 248 pgs: 248 active+clean; 8.5 MiB data, 586 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2.5 KiB/s wr, 7 op/s 2024-04-14T15:18:56.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:56 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:18:56.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:56 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3147868565' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:18:56.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:56 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/4012503850' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]: dispatch 2024-04-14T15:18:56.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:56 smithi092 ceph-mon[33966]: from='client.49186 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]: dispatch 2024-04-14T15:18:56.716 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:18:56.739 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:56.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:56.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:18:56.841 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:18:56.864 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:56.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:56.865 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:18:56.915 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-14T15:18:56.966 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:56.993 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:56.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:56.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:57.096 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:18:57.120 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:57.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:57.121 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:18:57.222 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:18:57.245 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:57.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:57.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:18:57.347 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:18:57.376 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:57.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:57.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:18:57.417 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:57 smithi090 ceph-mon[30399]: from='client.49186 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-04-14T15:18:57.417 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:57 smithi090 ceph-mon[30399]: osdmap e445: 8 total, 8 up, 8 in 2024-04-14T15:18:57.417 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:57 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-04-14T15:18:57.417 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:57 smithi090 ceph-mon[41670]: from='client.49186 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-04-14T15:18:57.417 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:57 smithi090 ceph-mon[41670]: osdmap e445: 8 total, 8 up, 8 in 2024-04-14T15:18:57.417 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:57 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-04-14T15:18:57.478 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:18:57.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:57.502 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:57.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:18:57.603 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:18:57.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:57 smithi092 ceph-mon[33966]: from='client.49186 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-04-14T15:18:57.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:57 smithi092 ceph-mon[33966]: osdmap e445: 8 total, 8 up, 8 in 2024-04-14T15:18:57.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:57 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-04-14T15:18:57.629 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:57.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:57.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:18:57.731 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:18:57.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:57.756 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:57.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:18:57.858 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:18:57.882 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:57.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:57.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:18:57.984 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:18:58.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:58.008 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:58.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:18:58.109 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:18:58.133 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:58.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:58.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:18:58.235 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:18:58.258 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:58.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:58.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:18:58.360 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:18:58.387 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:58.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:58.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:18:58.420 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:18:58.420 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:18:58.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:58 smithi090 ceph-mon[41670]: pgmap v626: 248 pgs: 248 active+clean; 8.5 MiB data, 586 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-04-14T15:18:58.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:58 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-04-14T15:18:58.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:58 smithi090 ceph-mon[41670]: osdmap e446: 8 total, 8 up, 8 in 2024-04-14T15:18:58.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:58 smithi090 ceph-mon[30399]: pgmap v626: 248 pgs: 248 active+clean; 8.5 MiB data, 586 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-04-14T15:18:58.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:58 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-04-14T15:18:58.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:58 smithi090 ceph-mon[30399]: osdmap e446: 8 total, 8 up, 8 in 2024-04-14T15:18:58.489 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:18:58.516 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:58.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:58.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:18:58.617 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:18:58.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:58 smithi092 ceph-mon[33966]: pgmap v626: 248 pgs: 248 active+clean; 8.5 MiB data, 586 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-04-14T15:18:58.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:58 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-04-14T15:18:58.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:58 smithi092 ceph-mon[33966]: osdmap e446: 8 total, 8 up, 8 in 2024-04-14T15:18:58.642 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:58.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:58.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:18:58.744 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:18:58.767 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:58.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:58.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:18:58.869 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:18:58.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:58.898 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:58.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:18:58.999 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:18:59.026 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:59.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:59.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:18:59.128 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:18:59.156 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:59.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:59.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:18:59.257 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:18:59.283 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:59.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:59.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:18:59.385 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:18:59.408 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:59.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:59.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:18:59.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:59 smithi090 ceph-mon[41670]: osdmap e447: 8 total, 8 up, 8 in 2024-04-14T15:18:59.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:18:59 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-04-14T15:18:59.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:59 smithi090 ceph-mon[30399]: osdmap e447: 8 total, 8 up, 8 in 2024-04-14T15:18:59.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:18:59 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-04-14T15:18:59.510 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:18:59.534 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:59.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:59.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:18:59.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:59 smithi092 ceph-mon[33966]: osdmap e447: 8 total, 8 up, 8 in 2024-04-14T15:18:59.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:18:59 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-04-14T15:18:59.635 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:18:59.658 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:18:59.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:59.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:18:59.759 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:18:59.786 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:59.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:59.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:18:59.888 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:18:59.914 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:18:59.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:18:59.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:19:00.016 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:19:00.040 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:00.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:00.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:19:00.141 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:19:00.168 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:00.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:00.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:19:00.270 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:19:00.293 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:00.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:00.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:19:00.381 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-14T15:19:00.395 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:19:00.422 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:00.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:00.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:19:00.477 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:00 smithi090 ceph-mon[41670]: pgmap v629: 258 pgs: 2 unknown, 256 active+clean; 8.5 MiB data, 586 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-14T15:19:00.477 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:00 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-04-14T15:19:00.477 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:00 smithi090 ceph-mon[41670]: osdmap e448: 8 total, 8 up, 8 in 2024-04-14T15:19:00.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:00 smithi090 ceph-mon[30399]: pgmap v629: 258 pgs: 2 unknown, 256 active+clean; 8.5 MiB data, 586 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-14T15:19:00.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:00 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-04-14T15:19:00.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:00 smithi090 ceph-mon[30399]: osdmap e448: 8 total, 8 up, 8 in 2024-04-14T15:19:00.523 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:19:00.550 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:00.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:00.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:19:00.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:00 smithi092 ceph-mon[33966]: pgmap v629: 258 pgs: 2 unknown, 256 active+clean; 8.5 MiB data, 586 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-14T15:19:00.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:00 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-04-14T15:19:00.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:00 smithi092 ceph-mon[33966]: osdmap e448: 8 total, 8 up, 8 in 2024-04-14T15:19:00.652 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:19:00.678 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:00.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:00.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:19:00.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:00.773+0000 7fb261df5700 1 -- 172.21.15.90:0/4255738227 >> v1:172.21.15.90:6789/0 conn(0x7fb25c10f300 legacy=0x7fb25c1117c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:00.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:00.773+0000 7fb261df5700 1 -- 172.21.15.90:0/4255738227 shutdown_connections 2024-04-14T15:19:00.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:00.773+0000 7fb261df5700 1 -- 172.21.15.90:0/4255738227 >> 172.21.15.90:0/4255738227 conn(0x7fb25c0fe390 msgr2=0x7fb25c1007b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:19:00.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:00.773+0000 7fb261df5700 1 -- 172.21.15.90:0/4255738227 shutdown_connections 2024-04-14T15:19:00.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:00.773+0000 7fb261df5700 1 -- 172.21.15.90:0/4255738227 wait complete. 2024-04-14T15:19:00.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:00.774+0000 7fb261df5700 1 librados: starting msgr at 2024-04-14T15:19:00.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:00.774+0000 7fb261df5700 1 librados: starting objecter 2024-04-14T15:19:00.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:00.774+0000 7fb261df5700 1 Processor -- start 2024-04-14T15:19:00.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:00.774+0000 7fb261df5700 1 -- start start 2024-04-14T15:19:00.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:00.774+0000 7fb261df5700 1 librados: setting wanted keys 2024-04-14T15:19:00.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:00.774+0000 7fb261df5700 1 librados: calling monclient init 2024-04-14T15:19:00.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:00.774+0000 7fb261df5700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb25c1a9c60 con 0x7fb25c108880 2024-04-14T15:19:00.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:00.774+0000 7fb261df5700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb25c1aaea0 con 0x7fb25c10b730 2024-04-14T15:19:00.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:00.774+0000 7fb261df5700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb25c1ac0e0 con 0x7fb25c10f300 2024-04-14T15:19:00.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:00.774+0000 7fb25b7fe700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fb25c108880 0x7fb25c0766a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:42174/0 (socket says 172.21.15.90:42174) 2024-04-14T15:19:00.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:00.774+0000 7fb25b7fe700 1 -- 172.21.15.90:0/3963865653 learned_addr learned my addr 172.21.15.90:0/3963865653 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:19:00.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:00.775+0000 7fb258ff9700 10 client.?.objecter ms_handle_connect 0x7fb25c108880 2024-04-14T15:19:00.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:00.775+0000 7fb258ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:19:00.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:00.775+0000 7fb258ff9700 10 client.?.objecter ms_handle_connect 0x7fb25c10f300 2024-04-14T15:19:00.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:00.775+0000 7fb258ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:19:00.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:00.775+0000 7fb258ff9700 1 -- 172.21.15.90:0/3963865653 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4215960039 0 0) 0x7fb25c1a9c60 con 0x7fb25c108880 2024-04-14T15:19:00.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:00.775+0000 7fb258ff9700 1 -- 172.21.15.90:0/3963865653 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb2440031f0 con 0x7fb25c108880 2024-04-14T15:19:00.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:00.775+0000 7fb258ff9700 10 client.?.objecter ms_handle_connect 0x7fb25c10b730 2024-04-14T15:19:00.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:00.775+0000 7fb258ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:19:00.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:00.775+0000 7fb258ff9700 1 -- 172.21.15.90:0/3963865653 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4112069317 0 0) 0x7fb25c1ac0e0 con 0x7fb25c10f300 2024-04-14T15:19:00.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:00.775+0000 7fb258ff9700 1 -- 172.21.15.90:0/3963865653 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb25c1a9c60 con 0x7fb25c10f300 2024-04-14T15:19:00.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:00.775+0000 7fb258ff9700 1 -- 172.21.15.90:0/3963865653 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3097708862 0 0) 0x7fb25c1aaea0 con 0x7fb25c10b730 2024-04-14T15:19:00.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:00.775+0000 7fb258ff9700 1 -- 172.21.15.90:0/3963865653 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb25c1ac0e0 con 0x7fb25c10b730 2024-04-14T15:19:00.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:00.775+0000 7fb258ff9700 1 -- 172.21.15.90:0/3963865653 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4282250349 0 0) 0x7fb2440031f0 con 0x7fb25c108880 2024-04-14T15:19:00.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:00.775+0000 7fb258ff9700 1 -- 172.21.15.90:0/3963865653 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb25c1aaea0 con 0x7fb25c108880 2024-04-14T15:19:00.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:00.775+0000 7fb258ff9700 1 -- 172.21.15.90:0/3963865653 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fb24c0012d0 con 0x7fb25c108880 2024-04-14T15:19:00.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:00.775+0000 7fb258ff9700 1 -- 172.21.15.90:0/3963865653 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1028892037 0 0) 0x7fb25c1a9c60 con 0x7fb25c10f300 2024-04-14T15:19:00.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:00.775+0000 7fb258ff9700 1 -- 172.21.15.90:0/3963865653 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb2440031f0 con 0x7fb25c10f300 2024-04-14T15:19:00.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:00.775+0000 7fb258ff9700 1 -- 172.21.15.90:0/3963865653 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fb2540048c0 con 0x7fb25c10f300 2024-04-14T15:19:00.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:00.775+0000 7fb258ff9700 1 -- 172.21.15.90:0/3963865653 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2688942219 0 0) 0x7fb25c1ac0e0 con 0x7fb25c10b730 2024-04-14T15:19:00.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:00.775+0000 7fb258ff9700 1 -- 172.21.15.90:0/3963865653 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb25c1a9c60 con 0x7fb25c10b730 2024-04-14T15:19:00.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:00.775+0000 7fb258ff9700 1 -- 172.21.15.90:0/3963865653 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fb250001870 con 0x7fb25c10b730 2024-04-14T15:19:00.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:00.775+0000 7fb258ff9700 1 -- 172.21.15.90:0/3963865653 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3303825735 0 0) 0x7fb25c1aaea0 con 0x7fb25c108880 2024-04-14T15:19:00.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:00.775+0000 7fb258ff9700 1 -- 172.21.15.90:0/3963865653 >> v1:172.21.15.90:6790/0 conn(0x7fb25c10f300 legacy=0x7fb25c1a8360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:00.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:00.775+0000 7fb258ff9700 1 -- 172.21.15.90:0/3963865653 >> v1:172.21.15.92:6789/0 conn(0x7fb25c10b730 legacy=0x7fb25c1a4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:00.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:00.775+0000 7fb258ff9700 1 -- 172.21.15.90:0/3963865653 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb25c1ad320 con 0x7fb25c108880 2024-04-14T15:19:00.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:00.775+0000 7fb261df5700 1 -- 172.21.15.90:0/3963865653 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb25c1a9e90 con 0x7fb25c108880 2024-04-14T15:19:00.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:00.776+0000 7fb261df5700 10 client.37353.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:19:00.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:00.776+0000 7fb261df5700 1 -- 172.21.15.90:0/3963865653 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb25c1aa350 con 0x7fb25c108880 2024-04-14T15:19:00.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:00.776+0000 7fb261df5700 1 librados: init done 2024-04-14T15:19:00.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:00.776+0000 7fb258ff9700 1 -- 172.21.15.90:0/3963865653 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fb24c001600 con 0x7fb25c108880 2024-04-14T15:19:00.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:00.776+0000 7fb258ff9700 1 -- 172.21.15.90:0/3963865653 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fb24c005cd0 con 0x7fb25c108880 2024-04-14T15:19:00.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:00.777+0000 7fb261df5700 1 -- 172.21.15.90:0/3963865653 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb248005120 con 0x7fb25c108880 2024-04-14T15:19:00.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:00.777+0000 7fb258ff9700 1 -- 172.21.15.90:0/3963865653 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fb24c01c880 con 0x7fb25c108880 2024-04-14T15:19:00.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:00.778+0000 7fb258ff9700 1 -- 172.21.15.90:0/3963865653 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(449..449 src has 51..449) ==== 8478+0+0 (unknown 1552739608 0 0) 0x7fb24c08ab70 con 0x7fb25c108880 2024-04-14T15:19:00.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:00.778+0000 7fb258ff9700 10 client.37353.objecter ms_dispatch 0x7fb25c001550 osd_map(449..449 src has 51..449) 2024-04-14T15:19:00.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:00.778+0000 7fb258ff9700 3 client.37353.objecter handle_osd_map got epochs [449,449] > 0 2024-04-14T15:19:00.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:00.778+0000 7fb258ff9700 3 client.37353.objecter handle_osd_map decoding full epoch 449 2024-04-14T15:19:00.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:00.778+0000 7fb258ff9700 20 client.37353.objecter dump_active .. 0 homeless 2024-04-14T15:19:00.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:00.778+0000 7fb258ff9700 10 client.37353.objecter ms_handle_connect 0x7fb2440712e0 2024-04-14T15:19:00.780 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:19:00.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:00.784+0000 7fb258ff9700 1 -- 172.21.15.90:0/3963865653 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fb24c057460 con 0x7fb25c108880 2024-04-14T15:19:00.804 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:00.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:00.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:19:00.815 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:19:00.906 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:19:00.943 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:00.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:00.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:19:01.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:01.024+0000 7fb261df5700 1 -- 172.21.15.90:0/3963865653 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7fb248005410 con 0x7fb25c108880 2024-04-14T15:19:01.045 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:01.071 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:01.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:01.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:01.173 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:19:01.196 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:01.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:01.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:19:01.298 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:19:01.321 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:01.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:01.322 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:19:01.357 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:01 smithi090 ceph-mon[30399]: osdmap e449: 8 total, 8 up, 8 in 2024-04-14T15:19:01.357 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:01 smithi090 ceph-mon[30399]: osdmap e450: 8 total, 8 up, 8 in 2024-04-14T15:19:01.357 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:01 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3963865653' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:19:01.357 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:01 smithi090 ceph-mon[41670]: osdmap e449: 8 total, 8 up, 8 in 2024-04-14T15:19:01.357 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:01 smithi090 ceph-mon[41670]: osdmap e450: 8 total, 8 up, 8 in 2024-04-14T15:19:01.358 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:01 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3963865653' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:19:01.423 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:19:01.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:01.448 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:01.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:19:01.548 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:19:01.572 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:01.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:01.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:19:01.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:01 smithi092 ceph-mon[33966]: osdmap e449: 8 total, 8 up, 8 in 2024-04-14T15:19:01.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:01 smithi092 ceph-mon[33966]: osdmap e450: 8 total, 8 up, 8 in 2024-04-14T15:19:01.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:01 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3963865653' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:19:01.674 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:01.701 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:01.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:01.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:01.803 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:19:01.828 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:01.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:01.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:19:01.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:01.919+0000 7fb258ff9700 1 -- 172.21.15.90:0/3963865653 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v451) ==== 91+0+0 (unknown 4039265110 0 0) 0x7fb24c05b110 con 0x7fb25c108880 2024-04-14T15:19:01.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:01.924+0000 7fb261df5700 10 librados: watch_flush enter 2024-04-14T15:19:01.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:01.924+0000 7fb261df5700 10 librados: watch_flush exit 2024-04-14T15:19:01.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:01.924+0000 7fb261df5700 20 client.37353.objecter shutdown clearing up homeless session... 2024-04-14T15:19:01.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:01.924+0000 7fb261df5700 10 client.37353.objecter successfully canceled tick 2024-04-14T15:19:01.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:01.925+0000 7fb261df5700 1 -- 172.21.15.90:0/3963865653 >> v1:172.21.15.90:6800/420935427 conn(0x7fb2440712e0 legacy=0x7fb2440737a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:01.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:01.925+0000 7fb261df5700 1 -- 172.21.15.90:0/3963865653 >> v1:172.21.15.90:6789/0 conn(0x7fb25c108880 legacy=0x7fb25c0766a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:01.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:01.925+0000 7fb261df5700 1 -- 172.21.15.90:0/3963865653 shutdown_connections 2024-04-14T15:19:01.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:01.925+0000 7fb261df5700 1 -- 172.21.15.90:0/3963865653 >> 172.21.15.90:0/3963865653 conn(0x7fb25c0fe390 msgr2=0x7fb25c102880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:19:01.930 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:19:01.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:01.930+0000 7fb261df5700 1 -- 172.21.15.90:0/3963865653 shutdown_connections 2024-04-14T15:19:01.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:01.930+0000 7fb261df5700 1 -- 172.21.15.90:0/3963865653 wait complete. 2024-04-14T15:19:01.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:01.930+0000 7fb261df5700 1 librados: shutdown 2024-04-14T15:19:01.933 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is unset 2024-04-14T15:19:01.956 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:01.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:01.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:19:02.058 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:19:02.085 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:02.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:02.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:19:02.187 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:19:02.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:02.212 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:02.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:19:02.313 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:02.341 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:02.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:02.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:02.443 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:19:02.469 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:02.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:02.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:19:02.571 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:19:02.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:02.598 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:02.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:19:02.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:02 smithi092 ceph-mon[33966]: pgmap v633: 258 pgs: 258 active+clean; 8.5 MiB data, 586 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 0 B/s wr, 3 op/s 2024-04-14T15:19:02.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:02 smithi092 ceph-mon[33966]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:19:02.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:02 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3963865653' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-14T15:19:02.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:02 smithi092 ceph-mon[33966]: osdmap e451: 8 total, 8 up, 8 in 2024-04-14T15:19:02.691 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:02 smithi090 ceph-mon[41670]: pgmap v633: 258 pgs: 258 active+clean; 8.5 MiB data, 586 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 0 B/s wr, 3 op/s 2024-04-14T15:19:02.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:02 smithi090 ceph-mon[41670]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:19:02.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:02 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3963865653' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-14T15:19:02.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:02 smithi090 ceph-mon[41670]: osdmap e451: 8 total, 8 up, 8 in 2024-04-14T15:19:02.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:02 smithi090 ceph-mon[30399]: pgmap v633: 258 pgs: 258 active+clean; 8.5 MiB data, 586 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 0 B/s wr, 3 op/s 2024-04-14T15:19:02.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:02 smithi090 ceph-mon[30399]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:19:02.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:02 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3963865653' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-14T15:19:02.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:02 smithi090 ceph-mon[30399]: osdmap e451: 8 total, 8 up, 8 in 2024-04-14T15:19:02.699 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:19:02.723 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:02.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:02.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:19:02.818 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-14T15:19:02.818 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-14T15:19:02.819 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-04-14T15:19:02.819 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd dump -f json-pretty 2024-04-14T15:19:02.824 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:19:02.886 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:02.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:02.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:19:02.988 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:19:03.012 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-14T15:19:03.013 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-14T15:19:03.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-14T15:19:03.013 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-14T15:19:03.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.013 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25024.0:2 34.1 34:847a3d55:::obj:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-04-14T15:19:03.013 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:12:55.073948+0000", 2024-04-14T15:19:03.013 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 367.93686215700001, 2024-04-14T15:19:03.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5228286629999999, 2024-04-14T15:19:03.013 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:19:03.013 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:19:03.013 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:19:03.013 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:19:03.013 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25024", 2024-04-14T15:19:03.014 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3705024715", 2024-04-14T15:19:03.014 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2 2024-04-14T15:19:03.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.014 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:19:03.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:19:03.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.073948+0000", 2024-04-14T15:19:03.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:19:03.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:19:03.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.073948+0000", 2024-04-14T15:19:03.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:19:03.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:19:03.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.073950+0000", 2024-04-14T15:19:03.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.951e-06 2024-04-14T15:19:03.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:19:03.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.073956+0000", 2024-04-14T15:19:03.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1759999999999998e-06 2024-04-14T15:19:03.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:19:03.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.073967+0000", 2024-04-14T15:19:03.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1151e-05 2024-04-14T15:19:03.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.016 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:19:03.016 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.073978+0000", 2024-04-14T15:19:03.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.079e-05 2024-04-14T15:19:03.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.016 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:19:03.016 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.104761+0000", 2024-04-14T15:19:03.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030782917 2024-04-14T15:19:03.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.016 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:19:03.016 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.104772+0000", 2024-04-14T15:19:03.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0781000000000001e-05 2024-04-14T15:19:03.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:19:03.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.556565+0000", 2024-04-14T15:19:03.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4517939100000001 2024-04-14T15:19:03.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:19:03.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.556645+0000", 2024-04-14T15:19:03.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9066999999999994e-05 2024-04-14T15:19:03.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,3", 2024-04-14T15:19:03.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.556733+0000", 2024-04-14T15:19:03.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8127000000000005e-05 2024-04-14T15:19:03.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:19:03.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.563586+0000", 2024-04-14T15:19:03.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0068534479999999998 2024-04-14T15:19:03.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:19:03.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.594501+0000", 2024-04-14T15:19:03.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030914984999999999 2024-04-14T15:19:03.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:19:03.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.596663+0000", 2024-04-14T15:19:03.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0021616080000000002 2024-04-14T15:19:03.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.019 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:19:03.019 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.596713+0000", 2024-04-14T15:19:03.019 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0009999999999997e-05 2024-04-14T15:19:03.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.019 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:19:03.019 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.596776+0000", 2024-04-14T15:19:03.019 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3742000000000001e-05 2024-04-14T15:19:03.019 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:03.019 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:03.019 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:03.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.020 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25054.0:2 38.1b 38:d96a01e0:::testobj:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-04-14T15:19:03.020 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:12:55.074069+0000", 2024-04-14T15:19:03.020 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 367.93674076100001, 2024-04-14T15:19:03.020 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.434224188, 2024-04-14T15:19:03.020 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:19:03.020 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:19:03.020 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:19:03.020 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:19:03.020 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25054", 2024-04-14T15:19:03.020 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/275760482", 2024-04-14T15:19:03.020 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2 2024-04-14T15:19:03.020 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.020 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:19:03.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:19:03.021 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.074069+0000", 2024-04-14T15:19:03.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:19:03.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:19:03.021 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.074069+0000", 2024-04-14T15:19:03.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:19:03.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:19:03.021 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.074070+0000", 2024-04-14T15:19:03.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2389999999999999e-06 2024-04-14T15:19:03.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.022 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:19:03.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.074076+0000", 2024-04-14T15:19:03.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5280000000000003e-06 2024-04-14T15:19:03.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.022 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:19:03.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.074086+0000", 2024-04-14T15:19:03.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0111e-05 2024-04-14T15:19:03.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.022 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:19:03.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.074095+0000", 2024-04-14T15:19:03.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6149999999999997e-06 2024-04-14T15:19:03.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:19:03.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.115262+0000", 2024-04-14T15:19:03.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.041167570000000001 2024-04-14T15:19:03.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:19:03.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.115269+0000", 2024-04-14T15:19:03.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5869999999999998e-06 2024-04-14T15:19:03.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:19:03.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.350019+0000", 2024-04-14T15:19:03.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2347499879999999 2024-04-14T15:19:03.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.024 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:19:03.024 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.353847+0000", 2024-04-14T15:19:03.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0038277509999999999 2024-04-14T15:19:03.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.024 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 5,6", 2024-04-14T15:19:03.024 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.353981+0000", 2024-04-14T15:19:03.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013471100000000001 2024-04-14T15:19:03.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.024 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:19:03.024 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.381419+0000", 2024-04-14T15:19:03.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.027438058000000001 2024-04-14T15:19:03.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:19:03.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.442226+0000", 2024-04-14T15:19:03.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.060806678000000003 2024-04-14T15:19:03.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:19:03.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.508201+0000", 2024-04-14T15:19:03.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.065975386999999996 2024-04-14T15:19:03.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:19:03.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.508249+0000", 2024-04-14T15:19:03.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7417999999999999e-05 2024-04-14T15:19:03.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:19:03.026 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:03.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.508293+0000", 2024-04-14T15:19:03.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4546999999999999e-05 2024-04-14T15:19:03.026 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:03.027 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:03.027 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:03.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.027 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25034.0:2 41.15 41:ad551702:::test:head [stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-04-14T15:19:03.027 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:12:55.074285+0000", 2024-04-14T15:19:03.027 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 367.93652479299999, 2024-04-14T15:19:03.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.481085719, 2024-04-14T15:19:03.027 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:19:03.027 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:19:03.027 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-04-14T15:19:03.027 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:19:03.027 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25034", 2024-04-14T15:19:03.028 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2502474599", 2024-04-14T15:19:03.028 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2 2024-04-14T15:19:03.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.028 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:19:03.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:19:03.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.074285+0000", 2024-04-14T15:19:03.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:19:03.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:19:03.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.074285+0000", 2024-04-14T15:19:03.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:19:03.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:19:03.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.074286+0000", 2024-04-14T15:19:03.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1519999999999999e-06 2024-04-14T15:19:03.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:19:03.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.074291+0000", 2024-04-14T15:19:03.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2769999999999999e-06 2024-04-14T15:19:03.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:19:03.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.074304+0000", 2024-04-14T15:19:03.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3891e-05 2024-04-14T15:19:03.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:19:03.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.074312+0000", 2024-04-14T15:19:03.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7090000000000002e-06 2024-04-14T15:19:03.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:19:03.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.114258+0000", 2024-04-14T15:19:03.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.039946138999999999 2024-04-14T15:19:03.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:19:03.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.114269+0000", 2024-04-14T15:19:03.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1039000000000001e-05 2024-04-14T15:19:03.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:19:03.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.555297+0000", 2024-04-14T15:19:03.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.441027869 2024-04-14T15:19:03.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:19:03.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.555371+0000", 2024-04-14T15:19:03.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3642999999999999e-05 2024-04-14T15:19:03.031 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:03.031 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:03.031 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:03.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.032 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24868.0:2 17.1a 17:5a9dc01b:test-rados-api-smithi090-93724-2::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-04-14T15:19:03.032 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:12:56.068401+0000", 2024-04-14T15:19:03.032 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 366.94240850400001, 2024-04-14T15:19:03.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.518847858, 2024-04-14T15:19:03.032 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:19:03.032 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:19:03.032 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:19:03.032 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:19:03.032 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24868", 2024-04-14T15:19:03.032 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1684644655", 2024-04-14T15:19:03.032 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2 2024-04-14T15:19:03.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.032 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:19:03.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:19:03.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.068401+0000", 2024-04-14T15:19:03.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:19:03.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:19:03.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.068401+0000", 2024-04-14T15:19:03.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:19:03.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:19:03.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.068403+0000", 2024-04-14T15:19:03.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9460000000000001e-06 2024-04-14T15:19:03.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:19:03.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.068411+0000", 2024-04-14T15:19:03.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2989999999999999e-06 2024-04-14T15:19:03.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:19:03.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.068426+0000", 2024-04-14T15:19:03.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5105e-05 2024-04-14T15:19:03.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:19:03.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.068436+0000", 2024-04-14T15:19:03.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0664e-05 2024-04-14T15:19:03.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:19:03.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.068500+0000", 2024-04-14T15:19:03.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3751000000000007e-05 2024-04-14T15:19:03.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2024-04-14T15:19:03.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.068506+0000", 2024-04-14T15:19:03.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1319999999999999e-06 2024-04-14T15:19:03.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:19:03.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.209626+0000", 2024-04-14T15:19:03.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.141119836 2024-04-14T15:19:03.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:19:03.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.209636+0000", 2024-04-14T15:19:03.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8199999999999992e-06 2024-04-14T15:19:03.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:19:03.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.520177+0000", 2024-04-14T15:19:03.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31054110200000001 2024-04-14T15:19:03.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:19:03.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.520306+0000", 2024-04-14T15:19:03.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000128429 2024-04-14T15:19:03.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2024-04-14T15:19:03.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.520463+0000", 2024-04-14T15:19:03.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015796399999999999 2024-04-14T15:19:03.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:19:03.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.531888+0000", 2024-04-14T15:19:03.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011424679 2024-04-14T15:19:03.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:19:03.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.533590+0000", 2024-04-14T15:19:03.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0017019470000000001 2024-04-14T15:19:03.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:19:03.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.587169+0000", 2024-04-14T15:19:03.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.05357891 2024-04-14T15:19:03.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:19:03.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.587210+0000", 2024-04-14T15:19:03.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1029999999999998e-05 2024-04-14T15:19:03.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:19:03.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.587249+0000", 2024-04-14T15:19:03.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9243999999999999e-05 2024-04-14T15:19:03.039 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:03.039 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:03.039 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:03.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.039 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25000.0:3 31.14 31:2a647cd4:::obj0:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e108)", 2024-04-14T15:19:03.039 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:12:56.091044+0000", 2024-04-14T15:19:03.039 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 366.91976625199999, 2024-04-14T15:19:03.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39133589000000002, 2024-04-14T15:19:03.040 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:19:03.040 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:19:03.040 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-04-14T15:19:03.040 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:19:03.040 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25000", 2024-04-14T15:19:03.040 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/757959112", 2024-04-14T15:19:03.040 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3 2024-04-14T15:19:03.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.040 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:19:03.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:03.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:19:03.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:19:03.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.091044+0000", 2024-04-14T15:19:03.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:19:03.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:19:03.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.091044+0000", 2024-04-14T15:19:03.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:19:03.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:19:03.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.091051+0000", 2024-04-14T15:19:03.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9700000000000002e-06 2024-04-14T15:19:03.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:19:03.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.091056+0000", 2024-04-14T15:19:03.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2070000000000003e-06 2024-04-14T15:19:03.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:19:03.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.091075+0000", 2024-04-14T15:19:03.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9145999999999999e-05 2024-04-14T15:19:03.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:19:03.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.091096+0000", 2024-04-14T15:19:03.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1299000000000001e-05 2024-04-14T15:19:03.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:19:03.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.091161+0000", 2024-04-14T15:19:03.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4613e-05 2024-04-14T15:19:03.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2024-04-14T15:19:03.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.091167+0000", 2024-04-14T15:19:03.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5520000000000001e-06 2024-04-14T15:19:03.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:19:03.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.197807+0000", 2024-04-14T15:19:03.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.106639095 2024-04-14T15:19:03.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:19:03.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.197816+0000", 2024-04-14T15:19:03.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.465e-06 2024-04-14T15:19:03.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:19:03.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.482281+0000", 2024-04-14T15:19:03.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.28446453599999999 2024-04-14T15:19:03.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:19:03.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.482380+0000", 2024-04-14T15:19:03.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9006999999999996e-05 2024-04-14T15:19:03.046 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:03.046 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:03.046 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:03.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.047 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24871.0:2 18.1d 18:b912fbc6:test-rados-api-smithi090-93709-2::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-04-14T15:19:03.047 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:12:56.517027+0000", 2024-04-14T15:19:03.047 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 366.49378246200001, 2024-04-14T15:19:03.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.069746945000000005, 2024-04-14T15:19:03.047 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:19:03.047 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:19:03.047 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:19:03.047 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:19:03.047 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24871", 2024-04-14T15:19:03.047 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1680088077", 2024-04-14T15:19:03.047 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2 2024-04-14T15:19:03.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.047 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:19:03.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:19:03.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.517027+0000", 2024-04-14T15:19:03.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:19:03.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:19:03.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.517027+0000", 2024-04-14T15:19:03.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:19:03.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:19:03.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.517029+0000", 2024-04-14T15:19:03.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8870000000000001e-06 2024-04-14T15:19:03.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:19:03.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.517041+0000", 2024-04-14T15:19:03.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1246e-05 2024-04-14T15:19:03.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:19:03.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.527081+0000", 2024-04-14T15:19:03.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010040423999999999 2024-04-14T15:19:03.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:19:03.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.527097+0000", 2024-04-14T15:19:03.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5542000000000001e-05 2024-04-14T15:19:03.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:19:03.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.527402+0000", 2024-04-14T15:19:03.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030516400000000002 2024-04-14T15:19:03.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:19:03.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.527528+0000", 2024-04-14T15:19:03.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000125902 2024-04-14T15:19:03.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,7", 2024-04-14T15:19:03.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.527671+0000", 2024-04-14T15:19:03.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000143287 2024-04-14T15:19:03.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:19:03.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.533025+0000", 2024-04-14T15:19:03.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0053542290000000003 2024-04-14T15:19:03.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:19:03.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.568203+0000", 2024-04-14T15:19:03.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.035178009000000003 2024-04-14T15:19:03.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:19:03.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.586678+0000", 2024-04-14T15:19:03.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.018474724000000001 2024-04-14T15:19:03.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:19:03.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.586725+0000", 2024-04-14T15:19:03.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7200999999999998e-05 2024-04-14T15:19:03.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:19:03.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.586774+0000", 2024-04-14T15:19:03.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9329999999999997e-05 2024-04-14T15:19:03.053 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:03.053 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:03.053 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:03.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.054 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24829.0:831 13.3 13:c0000000::::head [pgnls start_epoch 109] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e109)", 2024-04-14T15:19:03.054 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:12:57.272101+0000", 2024-04-14T15:19:03.054 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 365.73870889599999, 2024-04-14T15:19:03.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.106301906, 2024-04-14T15:19:03.054 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:19:03.054 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:19:03.054 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-14T15:19:03.054 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:19:03.054 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24829", 2024-04-14T15:19:03.054 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/218717313", 2024-04-14T15:19:03.054 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 831 2024-04-14T15:19:03.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.055 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:19:03.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:19:03.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.272101+0000", 2024-04-14T15:19:03.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:19:03.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:19:03.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.272101+0000", 2024-04-14T15:19:03.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:19:03.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:19:03.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.272103+0000", 2024-04-14T15:19:03.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.739e-06 2024-04-14T15:19:03.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.056 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:19:03.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.272110+0000", 2024-04-14T15:19:03.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2529999999999998e-06 2024-04-14T15:19:03.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.056 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:19:03.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.272121+0000", 2024-04-14T15:19:03.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1409e-05 2024-04-14T15:19:03.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:19:03.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.272133+0000", 2024-04-14T15:19:03.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1379000000000001e-05 2024-04-14T15:19:03.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:19:03.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.378286+0000", 2024-04-14T15:19:03.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10615343200000001 2024-04-14T15:19:03.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:19:03.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.378322+0000", 2024-04-14T15:19:03.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5800000000000003e-05 2024-04-14T15:19:03.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:19:03.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.378403+0000", 2024-04-14T15:19:03.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0894000000000002e-05 2024-04-14T15:19:03.058 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:03.058 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:03.058 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:03.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.059 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25144.0:18 24.12 24:48b3d3f4:::45.obj2:head [write 0~183] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-04-14T15:19:03.059 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:12:57.348480+0000", 2024-04-14T15:19:03.059 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 365.66232995600001, 2024-04-14T15:19:03.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.087810382000000006, 2024-04-14T15:19:03.059 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:19:03.059 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:19:03.059 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:19:03.059 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:19:03.059 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25144", 2024-04-14T15:19:03.059 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/790563004", 2024-04-14T15:19:03.059 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 18 2024-04-14T15:19:03.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.060 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:19:03.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:19:03.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.348480+0000", 2024-04-14T15:19:03.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:19:03.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:19:03.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.348480+0000", 2024-04-14T15:19:03.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:19:03.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:19:03.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.348482+0000", 2024-04-14T15:19:03.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3089999999999998e-06 2024-04-14T15:19:03.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:19:03.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.348493+0000", 2024-04-14T15:19:03.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0961000000000001e-05 2024-04-14T15:19:03.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:19:03.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.348510+0000", 2024-04-14T15:19:03.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6617999999999999e-05 2024-04-14T15:19:03.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:19:03.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.348521+0000", 2024-04-14T15:19:03.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1032e-05 2024-04-14T15:19:03.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.062 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:19:03.062 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.385863+0000", 2024-04-14T15:19:03.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.037341824000000003 2024-04-14T15:19:03.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.062 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:19:03.062 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.386005+0000", 2024-04-14T15:19:03.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000141831 2024-04-14T15:19:03.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2024-04-14T15:19:03.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.386158+0000", 2024-04-14T15:19:03.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015328799999999999 2024-04-14T15:19:03.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:19:03.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.388982+0000", 2024-04-14T15:19:03.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028236789999999999 2024-04-14T15:19:03.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:19:03.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.407473+0000", 2024-04-14T15:19:03.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.018491143000000002 2024-04-14T15:19:03.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:19:03.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.436189+0000", 2024-04-14T15:19:03.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.028716406999999999 2024-04-14T15:19:03.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:19:03.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.436247+0000", 2024-04-14T15:19:03.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7479000000000003e-05 2024-04-14T15:19:03.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:19:03.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.436290+0000", 2024-04-14T15:19:03.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3810999999999998e-05 2024-04-14T15:19:03.065 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:03.065 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:03.065 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:03.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.065 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25195.0:2 53.1a 53:5c6776db:::0.obj:head [write 0~103] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-04-14T15:19:03.065 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:12:58.085883+0000", 2024-04-14T15:19:03.065 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 364.924927438, 2024-04-14T15:19:03.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.195428578, 2024-04-14T15:19:03.066 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:19:03.066 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:19:03.066 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:19:03.066 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:19:03.066 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25195", 2024-04-14T15:19:03.066 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/82173666", 2024-04-14T15:19:03.066 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2 2024-04-14T15:19:03.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.067 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:19:03.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:19:03.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.085883+0000", 2024-04-14T15:19:03.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:19:03.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:19:03.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.085883+0000", 2024-04-14T15:19:03.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:19:03.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:19:03.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.085885+0000", 2024-04-14T15:19:03.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7089999999999998e-06 2024-04-14T15:19:03.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:19:03.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.085897+0000", 2024-04-14T15:19:03.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1443999999999999e-05 2024-04-14T15:19:03.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:19:03.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.085919+0000", 2024-04-14T15:19:03.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2147e-05 2024-04-14T15:19:03.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:19:03.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.085932+0000", 2024-04-14T15:19:03.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.314e-05 2024-04-14T15:19:03.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:19:03.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.126431+0000", 2024-04-14T15:19:03.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.040498884999999998 2024-04-14T15:19:03.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:19:03.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.126439+0000", 2024-04-14T15:19:03.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7589999999999997e-06 2024-04-14T15:19:03.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:19:03.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.265804+0000", 2024-04-14T15:19:03.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1393657829999999 2024-04-14T15:19:03.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:19:03.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.265916+0000", 2024-04-14T15:19:03.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000111226 2024-04-14T15:19:03.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 5,6", 2024-04-14T15:19:03.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.266049+0000", 2024-04-14T15:19:03.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013338 2024-04-14T15:19:03.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:19:03.074 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.268853+0000", 2024-04-14T15:19:03.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028036269999999999 2024-04-14T15:19:03.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.074 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:19:03.074 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.271676+0000", 2024-04-14T15:19:03.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028235959999999998 2024-04-14T15:19:03.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:19:03.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.280727+0000", 2024-04-14T15:19:03.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0090505110000000007 2024-04-14T15:19:03.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:19:03.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.280761+0000", 2024-04-14T15:19:03.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4394000000000003e-05 2024-04-14T15:19:03.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:19:03.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.281311+0000", 2024-04-14T15:19:03.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000549977 2024-04-14T15:19:03.076 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:03.076 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:03.077 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:03.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.077 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25189.0:2 56.1e 56:7a7d4169:::src_object.1:head [writefull 0~4096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-04-14T15:19:03.077 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:12:58.088602+0000", 2024-04-14T15:19:03.077 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 364.92220766499997, 2024-04-14T15:19:03.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1892207210000001, 2024-04-14T15:19:03.077 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:19:03.077 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:19:03.077 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:19:03.077 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:19:03.077 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25189", 2024-04-14T15:19:03.078 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/208306880", 2024-04-14T15:19:03.078 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2 2024-04-14T15:19:03.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.078 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:19:03.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:19:03.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.088602+0000", 2024-04-14T15:19:03.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:19:03.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:19:03.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.088602+0000", 2024-04-14T15:19:03.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:19:03.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:19:03.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.088605+0000", 2024-04-14T15:19:03.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2960000000000001e-06 2024-04-14T15:19:03.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:19:03.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.088644+0000", 2024-04-14T15:19:03.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9099e-05 2024-04-14T15:19:03.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:19:03.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.105297+0000", 2024-04-14T15:19:03.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016652970999999999 2024-04-14T15:19:03.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:19:03.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.105312+0000", 2024-04-14T15:19:03.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5244e-05 2024-04-14T15:19:03.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:19:03.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.105475+0000", 2024-04-14T15:19:03.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016360100000000001 2024-04-14T15:19:03.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:19:03.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.105485+0000", 2024-04-14T15:19:03.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3389999999999999e-06 2024-04-14T15:19:03.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:19:03.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.231630+0000", 2024-04-14T15:19:03.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1261448490000001 2024-04-14T15:19:03.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:19:03.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.231752+0000", 2024-04-14T15:19:03.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012195499999999999 2024-04-14T15:19:03.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2024-04-14T15:19:03.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.231893+0000", 2024-04-14T15:19:03.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014090500000000001 2024-04-14T15:19:03.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:19:03.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.263834+0000", 2024-04-14T15:19:03.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.031941268000000002 2024-04-14T15:19:03.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:19:03.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.263903+0000", 2024-04-14T15:19:03.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9031999999999999e-05 2024-04-14T15:19:03.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:19:03.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.277732+0000", 2024-04-14T15:19:03.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013828936 2024-04-14T15:19:03.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:19:03.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.277782+0000", 2024-04-14T15:19:03.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0281000000000002e-05 2024-04-14T15:19:03.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:19:03.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.277823+0000", 2024-04-14T15:19:03.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0945000000000002e-05 2024-04-14T15:19:03.084 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:03.084 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:03.084 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:03.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.084 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24853.0:2061 16.5 16:a697c087:test-rados-api-smithi090-93601-16::big.copy:head [copy-from ver 8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-04-14T15:19:03.084 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:12:58.908513+0000", 2024-04-14T15:19:03.084 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 364.10229680499998, 2024-04-14T15:19:03.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.085372185000000003, 2024-04-14T15:19:03.084 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:19:03.084 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:19:03.084 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:19:03.085 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:19:03.085 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24853", 2024-04-14T15:19:03.085 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3851923713", 2024-04-14T15:19:03.085 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2061 2024-04-14T15:19:03.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.085 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:19:03.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.085 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:19:03.085 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.908513+0000", 2024-04-14T15:19:03.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:19:03.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.085 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:19:03.085 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.908513+0000", 2024-04-14T15:19:03.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:19:03.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:19:03.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.908515+0000", 2024-04-14T15:19:03.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.305e-06 2024-04-14T15:19:03.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:19:03.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.908524+0000", 2024-04-14T15:19:03.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3659999999999992e-06 2024-04-14T15:19:03.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:19:03.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.908538+0000", 2024-04-14T15:19:03.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3851e-05 2024-04-14T15:19:03.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:19:03.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.908545+0000", 2024-04-14T15:19:03.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4109999999999999e-06 2024-04-14T15:19:03.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:19:03.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.908597+0000", 2024-04-14T15:19:03.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1549999999999999e-05 2024-04-14T15:19:03.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:19:03.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.908684+0000", 2024-04-14T15:19:03.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7003999999999996e-05 2024-04-14T15:19:03.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 5,6", 2024-04-14T15:19:03.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.968729+0000", 2024-04-14T15:19:03.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.060045424 2024-04-14T15:19:03.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:19:03.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.971874+0000", 2024-04-14T15:19:03.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031445760000000001 2024-04-14T15:19:03.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:19:03.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.993748+0000", 2024-04-14T15:19:03.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.021874350000000001 2024-04-14T15:19:03.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:19:03.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.993785+0000", 2024-04-14T15:19:03.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6835999999999998e-05 2024-04-14T15:19:03.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:19:03.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.993846+0000", 2024-04-14T15:19:03.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1101999999999999e-05 2024-04-14T15:19:03.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:19:03.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.993885+0000", 2024-04-14T15:19:03.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9409999999999997e-05 2024-04-14T15:19:03.090 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:03.090 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:03.090 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:03.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.090 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.24853.0:2070 16.12 e111/107 16:4b6e22cc:test-rados-api-smithi090-93601-17::big2:head v 111'7, mlcod=108'5)", 2024-04-14T15:19:03.090 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:12:59.459229+0000", 2024-04-14T15:19:03.090 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 363.551580586, 2024-04-14T15:19:03.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.065556860999999994, 2024-04-14T15:19:03.090 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:19:03.090 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:19:03.090 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:19:03.091 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:19:03.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:19:03.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.459229+0000", 2024-04-14T15:19:03.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:19:03.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:19:03.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.459229+0000", 2024-04-14T15:19:03.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:19:03.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:19:03.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.459231+0000", 2024-04-14T15:19:03.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6390000000000001e-06 2024-04-14T15:19:03.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:19:03.092 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.496663+0000", 2024-04-14T15:19:03.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.037432009000000002 2024-04-14T15:19:03.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:19:03.092 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.496696+0000", 2024-04-14T15:19:03.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2756999999999998e-05 2024-04-14T15:19:03.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:19:03.092 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.496710+0000", 2024-04-14T15:19:03.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3856000000000001e-05 2024-04-14T15:19:03.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:19:03.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.496800+0000", 2024-04-14T15:19:03.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0736999999999995e-05 2024-04-14T15:19:03.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:19:03.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.496832+0000", 2024-04-14T15:19:03.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.163e-05 2024-04-14T15:19:03.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:19:03.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.524715+0000", 2024-04-14T15:19:03.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.027883312 2024-04-14T15:19:03.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:19:03.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.524786+0000", 2024-04-14T15:19:03.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0920999999999994e-05 2024-04-14T15:19:03.094 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:03.094 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:03.094 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:03.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.094 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25243.0:1 68.7s0 68:e2e733b5:test-rados-api-smithi090-93398-24::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-04-14T15:19:03.094 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:13:00.136062+0000", 2024-04-14T15:19:03.094 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 362.87474776900001, 2024-04-14T15:19:03.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.81287744299999998, 2024-04-14T15:19:03.095 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:19:03.095 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:19:03.095 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:19:03.095 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:19:03.095 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25243", 2024-04-14T15:19:03.095 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/4291843131", 2024-04-14T15:19:03.095 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1 2024-04-14T15:19:03.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.095 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:19:03.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:19:03.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.136062+0000", 2024-04-14T15:19:03.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:19:03.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:19:03.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.136062+0000", 2024-04-14T15:19:03.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:19:03.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:19:03.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.136065+0000", 2024-04-14T15:19:03.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.762e-06 2024-04-14T15:19:03.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:19:03.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.136077+0000", 2024-04-14T15:19:03.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2136999999999999e-05 2024-04-14T15:19:03.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:19:03.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.136092+0000", 2024-04-14T15:19:03.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5270000000000001e-05 2024-04-14T15:19:03.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:19:03.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.136103+0000", 2024-04-14T15:19:03.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1148999999999999e-05 2024-04-14T15:19:03.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:19:03.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.208233+0000", 2024-04-14T15:19:03.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.072129844999999998 2024-04-14T15:19:03.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.098 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:19:03.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.208237+0000", 2024-04-14T15:19:03.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3170000000000002e-06 2024-04-14T15:19:03.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.098 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:19:03.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.940264+0000", 2024-04-14T15:19:03.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.73202700499999995 2024-04-14T15:19:03.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.098 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:19:03.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.940391+0000", 2024-04-14T15:19:03.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012699799999999999 2024-04-14T15:19:03.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2024-04-14T15:19:03.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.940798+0000", 2024-04-14T15:19:03.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000407222 2024-04-14T15:19:03.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2024-04-14T15:19:03.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.946299+0000", 2024-04-14T15:19:03.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0055010679999999996 2024-04-14T15:19:03.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:19:03.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.948871+0000", 2024-04-14T15:19:03.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00257238 2024-04-14T15:19:03.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.100 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:19:03.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.948940+0000", 2024-04-14T15:19:03.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8289999999999998e-05 2024-04-14T15:19:03.100 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:03.100 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:03.100 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:03.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.100 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.37144.0:5 142.0 142:0384ab8e:::oid:head [write 0~128] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-04-14T15:19:03.100 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:13:22.845350+0000", 2024-04-14T15:19:03.100 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 340.16546015799997, 2024-04-14T15:19:03.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19117366599999999, 2024-04-14T15:19:03.100 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:19:03.101 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:19:03.101 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:19:03.101 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:19:03.101 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.37144", 2024-04-14T15:19:03.101 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1278107768", 2024-04-14T15:19:03.101 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 5 2024-04-14T15:19:03.101 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.101 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:19:03.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.101 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:19:03.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:22.845350+0000", 2024-04-14T15:19:03.102 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:19:03.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:19:03.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:22.845350+0000", 2024-04-14T15:19:03.102 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:19:03.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:19:03.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:22.845352+0000", 2024-04-14T15:19:03.102 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3159999999999999e-06 2024-04-14T15:19:03.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.103 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:19:03.103 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:22.845361+0000", 2024-04-14T15:19:03.103 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2189999999999993e-06 2024-04-14T15:19:03.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.103 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:19:03.103 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:22.865357+0000", 2024-04-14T15:19:03.103 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019995186000000002 2024-04-14T15:19:03.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.103 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:19:03.103 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:22.865372+0000", 2024-04-14T15:19:03.103 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5855e-05 2024-04-14T15:19:03.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.104 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.104 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:19:03.104 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:22.866864+0000", 2024-04-14T15:19:03.104 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0014916599999999999 2024-04-14T15:19:03.104 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.104 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.104 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2024-04-14T15:19:03.104 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:22.866871+0000", 2024-04-14T15:19:03.104 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8029999999999999e-06 2024-04-14T15:19:03.104 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.104 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.104 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:19:03.104 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:22.975768+0000", 2024-04-14T15:19:03.104 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10889684500000001 2024-04-14T15:19:03.104 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.105 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:19:03.105 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:22.975862+0000", 2024-04-14T15:19:03.105 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4598000000000005e-05 2024-04-14T15:19:03.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.105 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 5,7", 2024-04-14T15:19:03.105 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:22.988754+0000", 2024-04-14T15:19:03.105 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012891929999999999 2024-04-14T15:19:03.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.105 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:19:03.105 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:23.001456+0000", 2024-04-14T15:19:03.105 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012702058 2024-04-14T15:19:03.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.106 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:19:03.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:23.035531+0000", 2024-04-14T15:19:03.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.034074522000000003 2024-04-14T15:19:03.106 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.106 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:19:03.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:23.036427+0000", 2024-04-14T15:19:03.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00089658800000000003 2024-04-14T15:19:03.106 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.106 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:19:03.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:23.036479+0000", 2024-04-14T15:19:03.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1864999999999999e-05 2024-04-14T15:19:03.106 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.107 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:19:03.107 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:23.036523+0000", 2024-04-14T15:19:03.107 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4221000000000002e-05 2024-04-14T15:19:03.107 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:03.107 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:03.107 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:03.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.107 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.38089.0:2 146.1f 146:fd8eaea7:nspace::oid_SimpleWritePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-04-14T15:19:03.107 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:13:22.892007+0000", 2024-04-14T15:19:03.107 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 340.11880302899999, 2024-04-14T15:19:03.107 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12663960699999999, 2024-04-14T15:19:03.107 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:19:03.108 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:19:03.108 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:19:03.108 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:19:03.108 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.38089", 2024-04-14T15:19:03.108 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2599630366", 2024-04-14T15:19:03.108 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2 2024-04-14T15:19:03.108 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.108 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:19:03.108 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.108 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:19:03.108 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:22.892007+0000", 2024-04-14T15:19:03.108 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:19:03.108 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:19:03.109 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:22.892007+0000", 2024-04-14T15:19:03.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:19:03.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:19:03.109 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:22.892009+0000", 2024-04-14T15:19:03.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3690000000000001e-06 2024-04-14T15:19:03.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:19:03.109 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:22.892018+0000", 2024-04-14T15:19:03.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6009999999999995e-06 2024-04-14T15:19:03.110 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.110 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.110 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:19:03.110 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:22.892029+0000", 2024-04-14T15:19:03.110 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1541e-05 2024-04-14T15:19:03.110 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.110 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.110 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:19:03.110 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:22.892040+0000", 2024-04-14T15:19:03.110 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.011e-05 2024-04-14T15:19:03.110 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.110 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.110 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:19:03.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:22.896577+0000", 2024-04-14T15:19:03.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0045371780000000002 2024-04-14T15:19:03.111 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.111 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.111 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:19:03.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:22.896706+0000", 2024-04-14T15:19:03.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129703 2024-04-14T15:19:03.111 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.111 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.111 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4,6", 2024-04-14T15:19:03.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:22.896851+0000", 2024-04-14T15:19:03.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000144196 2024-04-14T15:19:03.111 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.111 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.111 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:19:03.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:22.933476+0000", 2024-04-14T15:19:03.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.036625657999999998 2024-04-14T15:19:03.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:19:03.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:23.004050+0000", 2024-04-14T15:19:03.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.070573644000000005 2024-04-14T15:19:03.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:19:03.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:23.018555+0000", 2024-04-14T15:19:03.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014504813 2024-04-14T15:19:03.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:19:03.113 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:23.018606+0000", 2024-04-14T15:19:03.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1462000000000003e-05 2024-04-14T15:19:03.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.113 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.113 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:19:03.113 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:23.018647+0000", 2024-04-14T15:19:03.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0331999999999999e-05 2024-04-14T15:19:03.113 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:03.113 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:03.113 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:03.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.113 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.113 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.37144.0:8 142.0 142:0384ab8e:::oid:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-04-14T15:19:03.113 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:13:23.830111+0000", 2024-04-14T15:19:03.113 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 339.18069913099998, 2024-04-14T15:19:03.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.17823744899999999, 2024-04-14T15:19:03.114 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:19:03.114 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:19:03.114 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:19:03.114 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:19:03.114 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.37144", 2024-04-14T15:19:03.114 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1278107768", 2024-04-14T15:19:03.114 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 8 2024-04-14T15:19:03.114 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.114 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:19:03.114 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.114 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:19:03.114 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:23.830111+0000", 2024-04-14T15:19:03.114 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:19:03.114 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.115 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:19:03.115 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:23.830111+0000", 2024-04-14T15:19:03.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:19:03.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.115 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:19:03.115 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:23.830113+0000", 2024-04-14T15:19:03.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.832e-06 2024-04-14T15:19:03.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.115 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:19:03.115 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:23.830118+0000", 2024-04-14T15:19:03.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5169999999999999e-06 2024-04-14T15:19:03.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.115 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:19:03.116 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:23.830131+0000", 2024-04-14T15:19:03.116 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2738999999999999e-05 2024-04-14T15:19:03.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.116 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:19:03.116 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:23.830143+0000", 2024-04-14T15:19:03.116 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1737e-05 2024-04-14T15:19:03.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.116 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:19:03.116 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:23.830845+0000", 2024-04-14T15:19:03.116 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00070280199999999998 2024-04-14T15:19:03.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.117 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2024-04-14T15:19:03.117 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:23.830852+0000", 2024-04-14T15:19:03.117 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5289999999999997e-06 2024-04-14T15:19:03.117 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.117 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.117 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:19:03.117 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:23.960489+0000", 2024-04-14T15:19:03.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12963743 2024-04-14T15:19:03.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.118 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:19:03.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:23.960587+0000", 2024-04-14T15:19:03.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7536999999999998e-05 2024-04-14T15:19:03.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.118 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 5,7", 2024-04-14T15:19:03.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:23.960749+0000", 2024-04-14T15:19:03.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016159900000000001 2024-04-14T15:19:03.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.118 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:19:03.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:24.008116+0000", 2024-04-14T15:19:03.119 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.047367289 2024-04-14T15:19:03.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.119 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:19:03.119 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:24.008165+0000", 2024-04-14T15:19:03.119 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8742999999999997e-05 2024-04-14T15:19:03.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.119 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:19:03.119 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:24.008240+0000", 2024-04-14T15:19:03.119 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5189000000000001e-05 2024-04-14T15:19:03.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.119 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:19:03.119 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:24.008301+0000", 2024-04-14T15:19:03.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1086000000000004e-05 2024-04-14T15:19:03.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.120 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:19:03.120 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:24.008348+0000", 2024-04-14T15:19:03.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7420000000000003e-05 2024-04-14T15:19:03.120 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:03.120 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:03.120 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:03.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.120 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.41506.0:2 153.7 153:e732be06:::oid_WaitForSafePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-04-14T15:19:03.120 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:13:25.862818+0000", 2024-04-14T15:19:03.120 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 337.14799200800002, 2024-04-14T15:19:03.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12608759999999999, 2024-04-14T15:19:03.120 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:19:03.121 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:19:03.121 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:19:03.121 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:19:03.121 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.41506", 2024-04-14T15:19:03.121 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/790649677", 2024-04-14T15:19:03.121 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2 2024-04-14T15:19:03.121 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.121 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:19:03.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.121 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:19:03.121 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:25.862818+0000", 2024-04-14T15:19:03.121 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:19:03.121 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.121 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:19:03.122 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:25.862818+0000", 2024-04-14T15:19:03.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:19:03.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.122 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:19:03.122 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:25.862820+0000", 2024-04-14T15:19:03.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1050000000000002e-06 2024-04-14T15:19:03.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.122 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:19:03.122 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:25.862829+0000", 2024-04-14T15:19:03.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4719999999999995e-06 2024-04-14T15:19:03.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:19:03.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:25.862839+0000", 2024-04-14T15:19:03.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.048e-05 2024-04-14T15:19:03.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:19:03.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:25.862850+0000", 2024-04-14T15:19:03.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0798999999999999e-05 2024-04-14T15:19:03.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:19:03.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:25.865958+0000", 2024-04-14T15:19:03.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031085420000000002 2024-04-14T15:19:03.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.124 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:19:03.124 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:25.865967+0000", 2024-04-14T15:19:03.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2339999999999994e-06 2024-04-14T15:19:03.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.124 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:19:03.124 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:25.932885+0000", 2024-04-14T15:19:03.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.066918238000000005 2024-04-14T15:19:03.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.124 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:19:03.124 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:25.941025+0000", 2024-04-14T15:19:03.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0081406789999999996 2024-04-14T15:19:03.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,7", 2024-04-14T15:19:03.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:25.941175+0000", 2024-04-14T15:19:03.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000149023 2024-04-14T15:19:03.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:19:03.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:25.950227+0000", 2024-04-14T15:19:03.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0090528280000000006 2024-04-14T15:19:03.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:19:03.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:25.975965+0000", 2024-04-14T15:19:03.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.025738163000000001 2024-04-14T15:19:03.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:19:03.126 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:25.988801+0000", 2024-04-14T15:19:03.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012835736 2024-04-14T15:19:03.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.126 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:19:03.126 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:25.988861+0000", 2024-04-14T15:19:03.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9330999999999998e-05 2024-04-14T15:19:03.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.126 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:19:03.126 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:25.988906+0000", 2024-04-14T15:19:03.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4969999999999998e-05 2024-04-14T15:19:03.126 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:03.126 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:03.127 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:03.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.127 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.25684.0:248 110.1c e139/121 110:3d310c86:test-rados-api-smithi090-93652-12::bar:head v 139'1, mlcod=0'0)", 2024-04-14T15:19:03.127 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:13:26.930014+0000", 2024-04-14T15:19:03.127 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 336.08079556799998, 2024-04-14T15:19:03.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.064154581000000002, 2024-04-14T15:19:03.127 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:19:03.127 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:19:03.127 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:19:03.127 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:19:03.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.127 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:19:03.127 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.930014+0000", 2024-04-14T15:19:03.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:19:03.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.128 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:19:03.128 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.930014+0000", 2024-04-14T15:19:03.128 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:19:03.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.128 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:19:03.128 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.930015+0000", 2024-04-14T15:19:03.128 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0049999999999999e-06 2024-04-14T15:19:03.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.128 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:19:03.128 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.930031+0000", 2024-04-14T15:19:03.128 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5563e-05 2024-04-14T15:19:03.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.129 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.129 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:19:03.129 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.930040+0000", 2024-04-14T15:19:03.129 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9420000000000001e-06 2024-04-14T15:19:03.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.129 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.129 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:19:03.129 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.930049+0000", 2024-04-14T15:19:03.129 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.448e-06 2024-04-14T15:19:03.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.129 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.129 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:19:03.129 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.973354+0000", 2024-04-14T15:19:03.129 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.043304724000000003 2024-04-14T15:19:03.130 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.130 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.130 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:19:03.130 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.973376+0000", 2024-04-14T15:19:03.130 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1858999999999999e-05 2024-04-14T15:19:03.130 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.130 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.130 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:19:03.130 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.994130+0000", 2024-04-14T15:19:03.130 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020753898999999999 2024-04-14T15:19:03.130 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.130 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.130 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:19:03.131 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.994169+0000", 2024-04-14T15:19:03.131 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9140999999999997e-05 2024-04-14T15:19:03.131 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:03.131 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:03.131 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:03.131 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.131 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.131 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(osd.2.0:705 16.e e142/107 -18:770790e7:::temp_16.e_0_24188_2:head v 0'0, mlcod=110'1)", 2024-04-14T15:19:03.131 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:13:30.007123+0000", 2024-04-14T15:19:03.131 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 333.00368698800003, 2024-04-14T15:19:03.131 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.119559796, 2024-04-14T15:19:03.131 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:19:03.131 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:19:03.131 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:19:03.131 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:19:03.132 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.132 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:19:03.132 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:30.007123+0000", 2024-04-14T15:19:03.132 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:19:03.132 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.132 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.132 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:19:03.132 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:30.007123+0000", 2024-04-14T15:19:03.132 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:19:03.132 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.132 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.132 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:19:03.132 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:30.007123+0000", 2024-04-14T15:19:03.132 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5999999999999999e-07 2024-04-14T15:19:03.133 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.133 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.133 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:19:03.133 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:30.015550+0000", 2024-04-14T15:19:03.133 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00842621 2024-04-14T15:19:03.133 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.133 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.133 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:19:03.133 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:30.067223+0000", 2024-04-14T15:19:03.133 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.051673337 2024-04-14T15:19:03.133 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.133 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.133 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:19:03.133 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:30.096016+0000", 2024-04-14T15:19:03.134 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.028793531000000001 2024-04-14T15:19:03.134 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.134 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.134 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:19:03.134 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:30.100376+0000", 2024-04-14T15:19:03.134 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0043596549999999996 2024-04-14T15:19:03.134 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.134 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.134 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:19:03.134 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:30.100399+0000", 2024-04-14T15:19:03.134 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2719000000000001e-05 2024-04-14T15:19:03.134 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.134 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.134 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:19:03.135 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:30.126628+0000", 2024-04-14T15:19:03.135 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.026228930000000001 2024-04-14T15:19:03.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.135 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.135 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:19:03.135 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:30.126683+0000", 2024-04-14T15:19:03.135 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5053999999999998e-05 2024-04-14T15:19:03.135 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:03.135 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:03.135 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:03.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.135 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.135 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.48511.0:1 231.6s0 231:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e222)", 2024-04-14T15:19:03.135 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:14:49.856632+0000", 2024-04-14T15:19:03.136 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 253.154177581, 2024-04-14T15:19:03.136 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.016754532, 2024-04-14T15:19:03.136 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:19:03.136 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:19:03.136 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:19:03.136 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:19:03.136 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.48511", 2024-04-14T15:19:03.136 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1251627847", 2024-04-14T15:19:03.136 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1 2024-04-14T15:19:03.136 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.136 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:19:03.136 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.136 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:19:03.136 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:49.856632+0000", 2024-04-14T15:19:03.136 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:19:03.137 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.137 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.137 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:19:03.137 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:49.856632+0000", 2024-04-14T15:19:03.137 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:19:03.137 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.137 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.137 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:19:03.137 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:49.856635+0000", 2024-04-14T15:19:03.137 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2249999999999999e-06 2024-04-14T15:19:03.137 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.137 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.137 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:19:03.137 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:49.856643+0000", 2024-04-14T15:19:03.137 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6789999999999994e-06 2024-04-14T15:19:03.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.138 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.138 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:19:03.138 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:49.856655+0000", 2024-04-14T15:19:03.138 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2109e-05 2024-04-14T15:19:03.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.138 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.138 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:19:03.138 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:49.856667+0000", 2024-04-14T15:19:03.138 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1644999999999999e-05 2024-04-14T15:19:03.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.139 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.139 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:19:03.139 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:49.856757+0000", 2024-04-14T15:19:03.139 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9755999999999999e-05 2024-04-14T15:19:03.139 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.139 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.139 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:19:03.139 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:49.856761+0000", 2024-04-14T15:19:03.139 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4839999999999998e-06 2024-04-14T15:19:03.139 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.139 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.139 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:19:03.139 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:50.861976+0000", 2024-04-14T15:19:03.139 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.005215129 2024-04-14T15:19:03.139 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.140 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.140 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:19:03.140 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:50.862200+0000", 2024-04-14T15:19:03.140 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000223375 2024-04-14T15:19:03.140 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.140 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.140 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2024-04-14T15:19:03.140 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:50.862806+0000", 2024-04-14T15:19:03.140 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060652700000000004 2024-04-14T15:19:03.140 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.140 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.140 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2024-04-14T15:19:03.140 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:50.863880+0000", 2024-04-14T15:19:03.140 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001073854 2024-04-14T15:19:03.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.141 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:19:03.141 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:50.873310+0000", 2024-04-14T15:19:03.141 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0094293629999999996 2024-04-14T15:19:03.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:03.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:03.141 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:19:03.141 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:50.873387+0000", 2024-04-14T15:19:03.141 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7385999999999994e-05 2024-04-14T15:19:03.141 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:03.141 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:03.141 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:03.141 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:03.141 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:03.142 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-14T15:19:03.142 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:19:03.169 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:03.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:03.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:19:03.270 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:19:03.297 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:03.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:03.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:19:03.398 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:19:03.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:03.424 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:03.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:19:03.525 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:19:03.550 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:03.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:03.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:19:03.652 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:19:03.676 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:03.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:03.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:19:03.778 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:19:03.802 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:03.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:03.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:19:03.903 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:19:03.927 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:03.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:03.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:19:04.029 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:19:04.053 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:04.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:04.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:19:04.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:03 smithi092 ceph-mon[33966]: 4.5 scrub starts 2024-04-14T15:19:04.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:03 smithi092 ceph-mon[33966]: 4.5 scrub ok 2024-04-14T15:19:04.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:03 smithi092 ceph-mon[33966]: osdmap e452: 8 total, 8 up, 8 in 2024-04-14T15:19:04.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:03 smithi092 ceph-mon[33966]: 1.18 scrub starts 2024-04-14T15:19:04.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:03 smithi092 ceph-mon[33966]: 1.18 scrub ok 2024-04-14T15:19:04.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:03 smithi092 ceph-mon[33966]: 1.1d scrub starts 2024-04-14T15:19:04.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:03 smithi092 ceph-mon[33966]: 1.1d scrub ok 2024-04-14T15:19:04.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:03 smithi092 ceph-mon[33966]: 1.17 scrub starts 2024-04-14T15:19:04.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:03 smithi092 ceph-mon[33966]: 1.17 scrub ok 2024-04-14T15:19:04.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:03 smithi092 ceph-mon[33966]: 1.1a scrub starts 2024-04-14T15:19:04.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:03 smithi092 ceph-mon[33966]: 1.1a scrub ok 2024-04-14T15:19:04.154 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:19:04.178 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:04.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:04.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:19:04.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:03 smithi090 ceph-mon[30399]: 4.5 scrub starts 2024-04-14T15:19:04.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:03 smithi090 ceph-mon[30399]: 4.5 scrub ok 2024-04-14T15:19:04.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:03 smithi090 ceph-mon[30399]: osdmap e452: 8 total, 8 up, 8 in 2024-04-14T15:19:04.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:03 smithi090 ceph-mon[30399]: 1.18 scrub starts 2024-04-14T15:19:04.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:03 smithi090 ceph-mon[30399]: 1.18 scrub ok 2024-04-14T15:19:04.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:03 smithi090 ceph-mon[30399]: 1.1d scrub starts 2024-04-14T15:19:04.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:03 smithi090 ceph-mon[30399]: 1.1d scrub ok 2024-04-14T15:19:04.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:03 smithi090 ceph-mon[30399]: 1.17 scrub starts 2024-04-14T15:19:04.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:03 smithi090 ceph-mon[30399]: 1.17 scrub ok 2024-04-14T15:19:04.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:03 smithi090 ceph-mon[30399]: 1.1a scrub starts 2024-04-14T15:19:04.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:03 smithi090 ceph-mon[30399]: 1.1a scrub ok 2024-04-14T15:19:04.192 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:19:03 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:19:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:19:04.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:03 smithi090 ceph-mon[41670]: 4.5 scrub starts 2024-04-14T15:19:04.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:03 smithi090 ceph-mon[41670]: 4.5 scrub ok 2024-04-14T15:19:04.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:03 smithi090 ceph-mon[41670]: osdmap e452: 8 total, 8 up, 8 in 2024-04-14T15:19:04.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:03 smithi090 ceph-mon[41670]: 1.18 scrub starts 2024-04-14T15:19:04.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:03 smithi090 ceph-mon[41670]: 1.18 scrub ok 2024-04-14T15:19:04.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:03 smithi090 ceph-mon[41670]: 1.1d scrub starts 2024-04-14T15:19:04.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:03 smithi090 ceph-mon[41670]: 1.1d scrub ok 2024-04-14T15:19:04.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:03 smithi090 ceph-mon[41670]: 1.17 scrub starts 2024-04-14T15:19:04.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:03 smithi090 ceph-mon[41670]: 1.17 scrub ok 2024-04-14T15:19:04.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:03 smithi090 ceph-mon[41670]: 1.1a scrub starts 2024-04-14T15:19:04.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:03 smithi090 ceph-mon[41670]: 1.1a scrub ok 2024-04-14T15:19:04.280 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:19:04.304 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:04.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:04.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:19:04.405 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:19:04.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:04.430 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:04.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:19:04.531 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:04.554 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:04.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:04.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:04.656 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:19:04.680 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:04.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:04.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:19:04.781 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:04.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:04.805 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:04.805 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:04.814 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-14T15:19:04.906 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:04.929 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:04.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:04.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:05.031 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:19:05.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:05.058 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:05.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:19:05.159 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:19:05.164 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:04 smithi090 ceph-mon[41670]: pgmap v636: 258 pgs: 258 active+clean; 8.5 MiB data, 586 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:19:05.164 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:04 smithi090 ceph-mon[41670]: 1.1e scrub starts 2024-04-14T15:19:05.164 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:04 smithi090 ceph-mon[41670]: 1.1e scrub ok 2024-04-14T15:19:05.164 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:04 smithi090 ceph-mon[41670]: 1.1c scrub starts 2024-04-14T15:19:05.164 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:04 smithi090 ceph-mon[41670]: 1.1c scrub ok 2024-04-14T15:19:05.164 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:04 smithi090 ceph-mon[41670]: 1.16 scrub starts 2024-04-14T15:19:05.164 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:04 smithi090 ceph-mon[41670]: 1.16 scrub ok 2024-04-14T15:19:05.164 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:04 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:19:05.164 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:04 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:19:05.164 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:04 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-65"}]: dispatch 2024-04-14T15:19:05.164 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:04 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-65"}]: dispatch 2024-04-14T15:19:05.164 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:04 smithi090 ceph-mon[41670]: 1.5 scrub starts 2024-04-14T15:19:05.164 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:04 smithi090 ceph-mon[41670]: 1.5 scrub ok 2024-04-14T15:19:05.165 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:04 smithi090 ceph-mon[30399]: pgmap v636: 258 pgs: 258 active+clean; 8.5 MiB data, 586 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:19:05.165 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:04 smithi090 ceph-mon[30399]: 1.1e scrub starts 2024-04-14T15:19:05.165 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:04 smithi090 ceph-mon[30399]: 1.1e scrub ok 2024-04-14T15:19:05.165 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:04 smithi090 ceph-mon[30399]: 1.1c scrub starts 2024-04-14T15:19:05.165 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:04 smithi090 ceph-mon[30399]: 1.1c scrub ok 2024-04-14T15:19:05.165 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:04 smithi090 ceph-mon[30399]: 1.16 scrub starts 2024-04-14T15:19:05.165 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:04 smithi090 ceph-mon[30399]: 1.16 scrub ok 2024-04-14T15:19:05.165 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:04 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:19:05.165 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:04 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:19:05.165 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:04 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-65"}]: dispatch 2024-04-14T15:19:05.165 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:04 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-65"}]: dispatch 2024-04-14T15:19:05.165 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:04 smithi090 ceph-mon[30399]: 1.5 scrub starts 2024-04-14T15:19:05.165 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:04 smithi090 ceph-mon[30399]: 1.5 scrub ok 2024-04-14T15:19:05.182 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:05.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:05.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:19:05.284 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:19:05.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:05.310 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:05.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:19:05.372 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:19:05 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:19:05.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:04 smithi092 ceph-mon[33966]: pgmap v636: 258 pgs: 258 active+clean; 8.5 MiB data, 586 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:19:05.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:04 smithi092 ceph-mon[33966]: 1.1e scrub starts 2024-04-14T15:19:05.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:04 smithi092 ceph-mon[33966]: 1.1e scrub ok 2024-04-14T15:19:05.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:04 smithi092 ceph-mon[33966]: 1.1c scrub starts 2024-04-14T15:19:05.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:04 smithi092 ceph-mon[33966]: 1.1c scrub ok 2024-04-14T15:19:05.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:04 smithi092 ceph-mon[33966]: 1.16 scrub starts 2024-04-14T15:19:05.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:04 smithi092 ceph-mon[33966]: 1.16 scrub ok 2024-04-14T15:19:05.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:04 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:19:05.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:04 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:19:05.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:04 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-65"}]: dispatch 2024-04-14T15:19:05.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:04 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-65"}]: dispatch 2024-04-14T15:19:05.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:04 smithi092 ceph-mon[33966]: 1.5 scrub starts 2024-04-14T15:19:05.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:04 smithi092 ceph-mon[33966]: 1.5 scrub ok 2024-04-14T15:19:05.411 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:19:05.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:05.435 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:05.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:19:05.536 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:19:05.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:05.561 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:05.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:19:05.662 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:19:05.687 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:05.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:05.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:19:05.789 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:19:05.813 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:05.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:05.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:19:05.915 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:05.939 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:05.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:05.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:06.041 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:19:06.064 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:06.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:06.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:19:06.166 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:19:06.190 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:06.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:06.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:19:06.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:05 smithi090 ceph-mon[30399]: 1.c scrub starts 2024-04-14T15:19:06.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:05 smithi090 ceph-mon[30399]: 1.c scrub ok 2024-04-14T15:19:06.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:05 smithi090 ceph-mon[30399]: 1.15 scrub starts 2024-04-14T15:19:06.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:05 smithi090 ceph-mon[30399]: 1.15 scrub ok 2024-04-14T15:19:06.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:05 smithi090 ceph-mon[30399]: osdmap e453: 8 total, 8 up, 8 in 2024-04-14T15:19:06.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:05 smithi090 ceph-mon[30399]: 1.8 scrub starts 2024-04-14T15:19:06.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:05 smithi090 ceph-mon[30399]: 1.8 scrub ok 2024-04-14T15:19:06.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:05 smithi090 ceph-mon[30399]: 1.11 scrub starts 2024-04-14T15:19:06.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:05 smithi090 ceph-mon[30399]: 1.11 scrub ok 2024-04-14T15:19:06.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:05 smithi090 ceph-mon[30399]: osdmap e454: 8 total, 8 up, 8 in 2024-04-14T15:19:06.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:05 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:19:06.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:05 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:19:06.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:05 smithi090 ceph-mon[41670]: 1.c scrub starts 2024-04-14T15:19:06.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:05 smithi090 ceph-mon[41670]: 1.c scrub ok 2024-04-14T15:19:06.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:05 smithi090 ceph-mon[41670]: 1.15 scrub starts 2024-04-14T15:19:06.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:05 smithi090 ceph-mon[41670]: 1.15 scrub ok 2024-04-14T15:19:06.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:05 smithi090 ceph-mon[41670]: osdmap e453: 8 total, 8 up, 8 in 2024-04-14T15:19:06.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:05 smithi090 ceph-mon[41670]: 1.8 scrub starts 2024-04-14T15:19:06.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:05 smithi090 ceph-mon[41670]: 1.8 scrub ok 2024-04-14T15:19:06.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:05 smithi090 ceph-mon[41670]: 1.11 scrub starts 2024-04-14T15:19:06.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:05 smithi090 ceph-mon[41670]: 1.11 scrub ok 2024-04-14T15:19:06.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:05 smithi090 ceph-mon[41670]: osdmap e454: 8 total, 8 up, 8 in 2024-04-14T15:19:06.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:05 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:19:06.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:05 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:19:06.292 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:19:06.315 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:06.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:06.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:19:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:05 smithi092 ceph-mon[33966]: 1.c scrub starts 2024-04-14T15:19:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:05 smithi092 ceph-mon[33966]: 1.c scrub ok 2024-04-14T15:19:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:05 smithi092 ceph-mon[33966]: 1.15 scrub starts 2024-04-14T15:19:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:05 smithi092 ceph-mon[33966]: 1.15 scrub ok 2024-04-14T15:19:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:05 smithi092 ceph-mon[33966]: osdmap e453: 8 total, 8 up, 8 in 2024-04-14T15:19:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:05 smithi092 ceph-mon[33966]: 1.8 scrub starts 2024-04-14T15:19:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:05 smithi092 ceph-mon[33966]: 1.8 scrub ok 2024-04-14T15:19:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:05 smithi092 ceph-mon[33966]: 1.11 scrub starts 2024-04-14T15:19:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:05 smithi092 ceph-mon[33966]: 1.11 scrub ok 2024-04-14T15:19:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:05 smithi092 ceph-mon[33966]: osdmap e454: 8 total, 8 up, 8 in 2024-04-14T15:19:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:05 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:19:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:05 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:19:06.417 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:19:06.440 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:06.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:06.440 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:19:06.447 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-14T15:19:06.541 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:19:06.565 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:06.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:06.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:19:06.667 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:19:06.690 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:06.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:06.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:19:06.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.694+0000 7f1b166bd700 1 -- 172.21.15.90:0/2792455129 >> v1:172.21.15.90:6789/0 conn(0x7f1b1010b720 legacy=0x7f1b1010db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:06.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.694+0000 7f1b166bd700 1 -- 172.21.15.90:0/2792455129 shutdown_connections 2024-04-14T15:19:06.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.694+0000 7f1b166bd700 1 -- 172.21.15.90:0/2792455129 >> 172.21.15.90:0/2792455129 conn(0x7f1b100fe3a0 msgr2=0x7f1b101007c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:19:06.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.694+0000 7f1b166bd700 1 -- 172.21.15.90:0/2792455129 shutdown_connections 2024-04-14T15:19:06.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.694+0000 7f1b166bd700 1 -- 172.21.15.90:0/2792455129 wait complete. 2024-04-14T15:19:06.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.694+0000 7f1b166bd700 1 librados: starting msgr at 2024-04-14T15:19:06.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.694+0000 7f1b166bd700 1 librados: starting objecter 2024-04-14T15:19:06.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.694+0000 7f1b166bd700 1 Processor -- start 2024-04-14T15:19:06.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.694+0000 7f1b166bd700 1 -- start start 2024-04-14T15:19:06.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.694+0000 7f1b166bd700 1 librados: setting wanted keys 2024-04-14T15:19:06.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.694+0000 7f1b166bd700 1 librados: calling monclient init 2024-04-14T15:19:06.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.695+0000 7f1b166bd700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1b101a9be0 con 0x7f1b1010b720 2024-04-14T15:19:06.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.695+0000 7f1b166bd700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1b101aae00 con 0x7f1b10108870 2024-04-14T15:19:06.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.695+0000 7f1b166bd700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1b101ac040 con 0x7f1b1010f2f0 2024-04-14T15:19:06.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.695+0000 7f1b0f7fe700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f1b1010b720 0x7f1b101a4b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:42182/0 (socket says 172.21.15.90:42182) 2024-04-14T15:19:06.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.695+0000 7f1b0f7fe700 1 -- 172.21.15.90:0/850305187 learned_addr learned my addr 172.21.15.90:0/850305187 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:19:06.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.695+0000 7f1b0ffff700 1 --1- 172.21.15.90:0/850305187 >> v1:172.21.15.92:6789/0 conn(0x7f1b10108870 0x7f1b10076640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.90:34902/0 (socket says 172.21.15.90:34902) 2024-04-14T15:19:06.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.695+0000 7f1b14c5a700 1 --1- 172.21.15.90:0/850305187 >> v1:172.21.15.90:6790/0 conn(0x7f1b1010f2f0 0x7f1b101a82c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:39960/0 (socket says 172.21.15.90:39960) 2024-04-14T15:19:06.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.695+0000 7f1b0d7fa700 10 client.?.objecter ms_handle_connect 0x7f1b1010b720 2024-04-14T15:19:06.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.695+0000 7f1b0d7fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:19:06.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.695+0000 7f1b0d7fa700 10 client.?.objecter ms_handle_connect 0x7f1b10108870 2024-04-14T15:19:06.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.695+0000 7f1b0d7fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:19:06.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.695+0000 7f1b0d7fa700 10 client.?.objecter ms_handle_connect 0x7f1b1010f2f0 2024-04-14T15:19:06.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.695+0000 7f1b0d7fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:19:06.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.695+0000 7f1b0d7fa700 1 -- 172.21.15.90:0/850305187 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1327230154 0 0) 0x7f1b101aae00 con 0x7f1b10108870 2024-04-14T15:19:06.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.695+0000 7f1b0d7fa700 1 -- 172.21.15.90:0/850305187 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1af80031f0 con 0x7f1b10108870 2024-04-14T15:19:06.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.695+0000 7f1b0d7fa700 1 -- 172.21.15.90:0/850305187 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4038350078 0 0) 0x7f1b101a9be0 con 0x7f1b1010b720 2024-04-14T15:19:06.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.695+0000 7f1b0d7fa700 1 -- 172.21.15.90:0/850305187 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1b101aae00 con 0x7f1b1010b720 2024-04-14T15:19:06.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.695+0000 7f1b0d7fa700 1 -- 172.21.15.90:0/850305187 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3952031368 0 0) 0x7f1b101ac040 con 0x7f1b1010f2f0 2024-04-14T15:19:06.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.695+0000 7f1b0d7fa700 1 -- 172.21.15.90:0/850305187 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1b101a9be0 con 0x7f1b1010f2f0 2024-04-14T15:19:06.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.695+0000 7f1b0d7fa700 1 -- 172.21.15.90:0/850305187 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 35230450 0 0) 0x7f1af80031f0 con 0x7f1b10108870 2024-04-14T15:19:06.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.695+0000 7f1b0d7fa700 1 -- 172.21.15.90:0/850305187 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1b101ac040 con 0x7f1b10108870 2024-04-14T15:19:06.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.695+0000 7f1b0d7fa700 1 -- 172.21.15.90:0/850305187 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f1b000019d0 con 0x7f1b10108870 2024-04-14T15:19:06.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.695+0000 7f1b0d7fa700 1 -- 172.21.15.90:0/850305187 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3668594127 0 0) 0x7f1b101aae00 con 0x7f1b1010b720 2024-04-14T15:19:06.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.695+0000 7f1b0d7fa700 1 -- 172.21.15.90:0/850305187 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1af80031f0 con 0x7f1b1010b720 2024-04-14T15:19:06.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.695+0000 7f1b0d7fa700 1 -- 172.21.15.90:0/850305187 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f1b040046e0 con 0x7f1b1010b720 2024-04-14T15:19:06.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.695+0000 7f1b0d7fa700 1 -- 172.21.15.90:0/850305187 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 297617871 0 0) 0x7f1b101a9be0 con 0x7f1b1010f2f0 2024-04-14T15:19:06.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.695+0000 7f1b0d7fa700 1 -- 172.21.15.90:0/850305187 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1b101aae00 con 0x7f1b1010f2f0 2024-04-14T15:19:06.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.695+0000 7f1b0d7fa700 1 -- 172.21.15.90:0/850305187 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f1b08001c30 con 0x7f1b1010f2f0 2024-04-14T15:19:06.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.696+0000 7f1b0d7fa700 1 -- 172.21.15.90:0/850305187 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3640835303 0 0) 0x7f1b101ac040 con 0x7f1b10108870 2024-04-14T15:19:06.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.696+0000 7f1b0d7fa700 1 -- 172.21.15.90:0/850305187 >> v1:172.21.15.90:6790/0 conn(0x7f1b1010f2f0 legacy=0x7f1b101a82c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:06.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.696+0000 7f1b0d7fa700 1 -- 172.21.15.90:0/850305187 >> v1:172.21.15.90:6789/0 conn(0x7f1b1010b720 legacy=0x7f1b101a4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:06.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.696+0000 7f1b0d7fa700 1 -- 172.21.15.90:0/850305187 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b101ad280 con 0x7f1b10108870 2024-04-14T15:19:06.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.696+0000 7f1b166bd700 1 -- 172.21.15.90:0/850305187 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1b101ab030 con 0x7f1b10108870 2024-04-14T15:19:06.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.696+0000 7f1b166bd700 10 client.49198.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:19:06.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.696+0000 7f1b166bd700 1 -- 172.21.15.90:0/850305187 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f1b101ab4f0 con 0x7f1b10108870 2024-04-14T15:19:06.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.696+0000 7f1b166bd700 1 librados: init done 2024-04-14T15:19:06.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.696+0000 7f1b0d7fa700 1 -- 172.21.15.90:0/850305187 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f1b00003b20 con 0x7f1b10108870 2024-04-14T15:19:06.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.696+0000 7f1b0d7fa700 1 -- 172.21.15.90:0/850305187 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f1b00005c60 con 0x7f1b10108870 2024-04-14T15:19:06.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.696+0000 7f1b166bd700 1 -- 172.21.15.90:0/850305187 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f1b10103fb0 con 0x7f1b10108870 2024-04-14T15:19:06.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.698+0000 7f1b0d7fa700 1 -- 172.21.15.90:0/850305187 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f1b00003680 con 0x7f1b10108870 2024-04-14T15:19:06.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.698+0000 7f1b0d7fa700 1 -- 172.21.15.90:0/850305187 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(454..454 src has 51..454) ==== 8175+0+0 (unknown 455453532 0 0) 0x7f1b0008a990 con 0x7f1b10108870 2024-04-14T15:19:06.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.698+0000 7f1b0d7fa700 10 client.49198.objecter ms_dispatch 0x7f1b10001550 osd_map(454..454 src has 51..454) 2024-04-14T15:19:06.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.698+0000 7f1b0d7fa700 3 client.49198.objecter handle_osd_map got epochs [454,454] > 0 2024-04-14T15:19:06.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.698+0000 7f1b0d7fa700 3 client.49198.objecter handle_osd_map decoding full epoch 454 2024-04-14T15:19:06.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.698+0000 7f1b0d7fa700 20 client.49198.objecter dump_active .. 0 homeless 2024-04-14T15:19:06.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.698+0000 7f1b0d7fa700 1 -- 172.21.15.90:0/850305187 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v454) ==== 54+0+16 (unknown 1521394261 0 2420000192) 0x7f1b0008ad80 con 0x7f1b10108870 2024-04-14T15:19:06.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.699+0000 7f1b0d7fa700 10 client.49198.objecter ms_handle_connect 0x7f1af8071460 2024-04-14T15:19:06.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.699+0000 7f1b166bd700 10 client.49198.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:19:06.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.699+0000 7f1b166bd700 15 client.49198.objecter _session_command_op_assign -1 1 2024-04-14T15:19:06.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.699+0000 7f1b166bd700 20 client.49198.objecter _get_session s=0x7f1b101132c0 osd=0 3 2024-04-14T15:19:06.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.699+0000 7f1b166bd700 20 client.49198.objecter put_session s=0x7f1b101132c0 osd=0 3 2024-04-14T15:19:06.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.699+0000 7f1b166bd700 20 client.49198.objecter _get_session s=0x7f1b101132c0 osd=0 3 2024-04-14T15:19:06.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.699+0000 7f1b166bd700 15 client.49198.objecter _session_command_op_remove -1 1 2024-04-14T15:19:06.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.699+0000 7f1b166bd700 20 client.49198.objecter get_session s=0x7f1b101132c0 osd=0 3 2024-04-14T15:19:06.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.699+0000 7f1b166bd700 15 client.49198.objecter _session_command_op_assign 0 1 2024-04-14T15:19:06.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.699+0000 7f1b166bd700 20 client.49198.objecter put_session s=0x7f1b101132c0 osd=0 4 2024-04-14T15:19:06.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.699+0000 7f1b166bd700 10 client.49198.objecter _send_command 1 2024-04-14T15:19:06.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.699+0000 7f1b166bd700 1 -- 172.21.15.90:0/850305187 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1b10076100 con 0x7f1b101ac210 2024-04-14T15:19:06.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.699+0000 7f1b0d7fa700 10 client.49198.objecter ms_handle_connect 0x7f1b101ac210 2024-04-14T15:19:06.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.700+0000 7f1b0d7fa700 1 -- 172.21.15.90:0/850305187 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1b10076100 con 0x7f1b101ac210 2024-04-14T15:19:06.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.700+0000 7f1b0d7fa700 10 client.49198.objecter ms_dispatch 0x7f1b10001550 command_reply(tid 1: 0 ) 2024-04-14T15:19:06.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.700+0000 7f1b0d7fa700 10 client.49198.objecter _finish_command 1 = system:0 2024-04-14T15:19:06.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.700+0000 7f1b0d7fa700 20 client.49198.objecter put_session s=0x7f1b101132c0 osd=0 4 2024-04-14T15:19:06.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.700+0000 7f1b0d7fa700 15 client.49198.objecter _session_command_op_remove 0 1 2024-04-14T15:19:06.703 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:19:06.703 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:19:06.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.722+0000 7f1b166bd700 10 client.49198.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:19:06.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.722+0000 7f1b166bd700 15 client.49198.objecter _session_command_op_assign -1 2 2024-04-14T15:19:06.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.722+0000 7f1b166bd700 20 client.49198.objecter _get_session s=0x7f1b101132c0 osd=0 3 2024-04-14T15:19:06.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.722+0000 7f1b166bd700 20 client.49198.objecter put_session s=0x7f1b101132c0 osd=0 3 2024-04-14T15:19:06.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.722+0000 7f1b166bd700 20 client.49198.objecter _get_session s=0x7f1b101132c0 osd=0 3 2024-04-14T15:19:06.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.722+0000 7f1b166bd700 15 client.49198.objecter _session_command_op_remove -1 2 2024-04-14T15:19:06.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.722+0000 7f1b166bd700 20 client.49198.objecter get_session s=0x7f1b101132c0 osd=0 3 2024-04-14T15:19:06.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.722+0000 7f1b166bd700 15 client.49198.objecter _session_command_op_assign 0 2 2024-04-14T15:19:06.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.722+0000 7f1b166bd700 20 client.49198.objecter put_session s=0x7f1b101132c0 osd=0 4 2024-04-14T15:19:06.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.722+0000 7f1b166bd700 10 client.49198.objecter _send_command 2 2024-04-14T15:19:06.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.722+0000 7f1b166bd700 1 -- 172.21.15.90:0/850305187 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1b100693d0 con 0x7f1b101ac210 2024-04-14T15:19:06.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.725+0000 7f1b0d7fa700 1 -- 172.21.15.90:0/850305187 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1b100693d0 con 0x7f1b101ac210 2024-04-14T15:19:06.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.725+0000 7f1b0d7fa700 10 client.49198.objecter ms_dispatch 0x7f1b10001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:19:06.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.725+0000 7f1b0d7fa700 10 client.49198.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:19:06.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.725+0000 7f1b0d7fa700 20 client.49198.objecter put_session s=0x7f1b101132c0 osd=0 4 2024-04-14T15:19:06.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.725+0000 7f1b0d7fa700 15 client.49198.objecter _session_command_op_remove 0 2 2024-04-14T15:19:06.726 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:19:06.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.725+0000 7f1b166bd700 10 client.49198.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:19:06.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.725+0000 7f1b166bd700 15 client.49198.objecter _session_command_op_assign -1 3 2024-04-14T15:19:06.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.726+0000 7f1b166bd700 20 client.49198.objecter _get_session s=0x7f1b101acf50 osd=1 3 2024-04-14T15:19:06.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.726+0000 7f1b166bd700 20 client.49198.objecter put_session s=0x7f1b101acf50 osd=1 3 2024-04-14T15:19:06.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.726+0000 7f1b166bd700 20 client.49198.objecter _get_session s=0x7f1b101acf50 osd=1 3 2024-04-14T15:19:06.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.726+0000 7f1b166bd700 15 client.49198.objecter _session_command_op_remove -1 3 2024-04-14T15:19:06.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.726+0000 7f1b166bd700 20 client.49198.objecter get_session s=0x7f1b101acf50 osd=1 3 2024-04-14T15:19:06.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.726+0000 7f1b166bd700 15 client.49198.objecter _session_command_op_assign 1 3 2024-04-14T15:19:06.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.726+0000 7f1b166bd700 20 client.49198.objecter put_session s=0x7f1b101acf50 osd=1 4 2024-04-14T15:19:06.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.726+0000 7f1b166bd700 10 client.49198.objecter _send_command 3 2024-04-14T15:19:06.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.726+0000 7f1b166bd700 1 -- 172.21.15.90:0/850305187 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f1b1002de30 con 0x7f1b10110cc0 2024-04-14T15:19:06.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.726+0000 7f1b0d7fa700 10 client.49198.objecter ms_handle_connect 0x7f1b10110cc0 2024-04-14T15:19:06.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.727+0000 7f1b0d7fa700 1 -- 172.21.15.90:0/850305187 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1b1002de30 con 0x7f1b10110cc0 2024-04-14T15:19:06.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.727+0000 7f1b0d7fa700 10 client.49198.objecter ms_dispatch 0x7f1b10001550 command_reply(tid 3: 0 ) 2024-04-14T15:19:06.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.727+0000 7f1b0d7fa700 10 client.49198.objecter _finish_command 3 = system:0 2024-04-14T15:19:06.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.727+0000 7f1b0d7fa700 20 client.49198.objecter put_session s=0x7f1b101acf50 osd=1 4 2024-04-14T15:19:06.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.727+0000 7f1b0d7fa700 15 client.49198.objecter _session_command_op_remove 1 3 2024-04-14T15:19:06.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.746+0000 7f1b166bd700 10 client.49198.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:19:06.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.746+0000 7f1b166bd700 15 client.49198.objecter _session_command_op_assign -1 4 2024-04-14T15:19:06.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.746+0000 7f1b166bd700 20 client.49198.objecter _get_session s=0x7f1b101acf50 osd=1 3 2024-04-14T15:19:06.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.746+0000 7f1b166bd700 20 client.49198.objecter put_session s=0x7f1b101acf50 osd=1 3 2024-04-14T15:19:06.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.746+0000 7f1b166bd700 20 client.49198.objecter _get_session s=0x7f1b101acf50 osd=1 3 2024-04-14T15:19:06.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.746+0000 7f1b166bd700 15 client.49198.objecter _session_command_op_remove -1 4 2024-04-14T15:19:06.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.746+0000 7f1b166bd700 20 client.49198.objecter get_session s=0x7f1b101acf50 osd=1 3 2024-04-14T15:19:06.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.746+0000 7f1b166bd700 15 client.49198.objecter _session_command_op_assign 1 4 2024-04-14T15:19:06.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.746+0000 7f1b166bd700 20 client.49198.objecter put_session s=0x7f1b101acf50 osd=1 4 2024-04-14T15:19:06.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.746+0000 7f1b166bd700 10 client.49198.objecter _send_command 4 2024-04-14T15:19:06.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.746+0000 7f1b166bd700 1 -- 172.21.15.90:0/850305187 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1b1002e230 con 0x7f1b10110cc0 2024-04-14T15:19:06.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.748+0000 7f1b0d7fa700 1 -- 172.21.15.90:0/850305187 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1b1002e230 con 0x7f1b10110cc0 2024-04-14T15:19:06.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.748+0000 7f1b0d7fa700 10 client.49198.objecter ms_dispatch 0x7f1b10001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:19:06.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.748+0000 7f1b0d7fa700 10 client.49198.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:19:06.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.748+0000 7f1b0d7fa700 20 client.49198.objecter put_session s=0x7f1b101acf50 osd=1 4 2024-04-14T15:19:06.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.748+0000 7f1b0d7fa700 15 client.49198.objecter _session_command_op_remove 1 4 2024-04-14T15:19:06.749 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:19:06.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.748+0000 7f1b166bd700 10 client.49198.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:19:06.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.748+0000 7f1b166bd700 15 client.49198.objecter _session_command_op_assign -1 5 2024-04-14T15:19:06.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.748+0000 7f1b166bd700 20 client.49198.objecter _get_session s=0x7f1b1002e3e0 osd=2 3 2024-04-14T15:19:06.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.748+0000 7f1b166bd700 20 client.49198.objecter put_session s=0x7f1b1002e3e0 osd=2 3 2024-04-14T15:19:06.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.748+0000 7f1b166bd700 20 client.49198.objecter _get_session s=0x7f1b1002e3e0 osd=2 3 2024-04-14T15:19:06.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.748+0000 7f1b166bd700 15 client.49198.objecter _session_command_op_remove -1 5 2024-04-14T15:19:06.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.748+0000 7f1b166bd700 20 client.49198.objecter get_session s=0x7f1b1002e3e0 osd=2 3 2024-04-14T15:19:06.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.748+0000 7f1b166bd700 15 client.49198.objecter _session_command_op_assign 2 5 2024-04-14T15:19:06.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.748+0000 7f1b166bd700 20 client.49198.objecter put_session s=0x7f1b1002e3e0 osd=2 4 2024-04-14T15:19:06.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.748+0000 7f1b166bd700 10 client.49198.objecter _send_command 5 2024-04-14T15:19:06.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.748+0000 7f1b166bd700 1 -- 172.21.15.90:0/850305187 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f1b10063b20 con 0x7f1b100636c0 2024-04-14T15:19:06.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.749+0000 7f1b0d7fa700 10 client.49198.objecter ms_handle_connect 0x7f1b100636c0 2024-04-14T15:19:06.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.749+0000 7f1b0d7fa700 1 -- 172.21.15.90:0/850305187 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1b10063b20 con 0x7f1b100636c0 2024-04-14T15:19:06.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.749+0000 7f1b0d7fa700 10 client.49198.objecter ms_dispatch 0x7f1b10001550 command_reply(tid 5: 0 ) 2024-04-14T15:19:06.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.749+0000 7f1b0d7fa700 10 client.49198.objecter _finish_command 5 = system:0 2024-04-14T15:19:06.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.749+0000 7f1b0d7fa700 20 client.49198.objecter put_session s=0x7f1b1002e3e0 osd=2 4 2024-04-14T15:19:06.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.749+0000 7f1b0d7fa700 15 client.49198.objecter _session_command_op_remove 2 5 2024-04-14T15:19:06.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.766+0000 7f1b166bd700 10 client.49198.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:19:06.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.766+0000 7f1b166bd700 15 client.49198.objecter _session_command_op_assign -1 6 2024-04-14T15:19:06.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.766+0000 7f1b166bd700 20 client.49198.objecter _get_session s=0x7f1b1002e3e0 osd=2 3 2024-04-14T15:19:06.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.766+0000 7f1b166bd700 20 client.49198.objecter put_session s=0x7f1b1002e3e0 osd=2 3 2024-04-14T15:19:06.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.766+0000 7f1b166bd700 20 client.49198.objecter _get_session s=0x7f1b1002e3e0 osd=2 3 2024-04-14T15:19:06.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.766+0000 7f1b166bd700 15 client.49198.objecter _session_command_op_remove -1 6 2024-04-14T15:19:06.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.766+0000 7f1b166bd700 20 client.49198.objecter get_session s=0x7f1b1002e3e0 osd=2 3 2024-04-14T15:19:06.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.766+0000 7f1b166bd700 15 client.49198.objecter _session_command_op_assign 2 6 2024-04-14T15:19:06.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.766+0000 7f1b166bd700 20 client.49198.objecter put_session s=0x7f1b1002e3e0 osd=2 4 2024-04-14T15:19:06.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.766+0000 7f1b166bd700 10 client.49198.objecter _send_command 6 2024-04-14T15:19:06.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.766+0000 7f1b166bd700 1 -- 172.21.15.90:0/850305187 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1b101b5de0 con 0x7f1b100636c0 2024-04-14T15:19:06.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.768+0000 7f1b0d7fa700 1 -- 172.21.15.90:0/850305187 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1b101b5de0 con 0x7f1b100636c0 2024-04-14T15:19:06.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.768+0000 7f1b0d7fa700 10 client.49198.objecter ms_dispatch 0x7f1b10001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:19:06.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.768+0000 7f1b0d7fa700 10 client.49198.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:19:06.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.768+0000 7f1b0d7fa700 20 client.49198.objecter put_session s=0x7f1b1002e3e0 osd=2 4 2024-04-14T15:19:06.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.768+0000 7f1b0d7fa700 15 client.49198.objecter _session_command_op_remove 2 6 2024-04-14T15:19:06.768 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:19:06.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.768+0000 7f1b166bd700 10 client.49198.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:19:06.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.768+0000 7f1b166bd700 15 client.49198.objecter _session_command_op_assign -1 7 2024-04-14T15:19:06.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.768+0000 7f1b166bd700 20 client.49198.objecter _get_session s=0x7f1b10063140 osd=3 3 2024-04-14T15:19:06.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.768+0000 7f1b166bd700 20 client.49198.objecter put_session s=0x7f1b10063140 osd=3 3 2024-04-14T15:19:06.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.768+0000 7f1b166bd700 20 client.49198.objecter _get_session s=0x7f1b10063140 osd=3 3 2024-04-14T15:19:06.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.768+0000 7f1b166bd700 15 client.49198.objecter _session_command_op_remove -1 7 2024-04-14T15:19:06.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.768+0000 7f1b166bd700 20 client.49198.objecter get_session s=0x7f1b10063140 osd=3 3 2024-04-14T15:19:06.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.768+0000 7f1b166bd700 15 client.49198.objecter _session_command_op_assign 3 7 2024-04-14T15:19:06.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.768+0000 7f1b166bd700 20 client.49198.objecter put_session s=0x7f1b10063140 osd=3 4 2024-04-14T15:19:06.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.768+0000 7f1b166bd700 10 client.49198.objecter _send_command 7 2024-04-14T15:19:06.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.768+0000 7f1b166bd700 1 -- 172.21.15.90:0/850305187 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f1b101b2280 con 0x7f1b100632b0 2024-04-14T15:19:06.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.768+0000 7f1b0d7fa700 10 client.49198.objecter ms_handle_connect 0x7f1b100632b0 2024-04-14T15:19:06.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.769+0000 7f1b0d7fa700 1 -- 172.21.15.90:0/850305187 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1b101b2280 con 0x7f1b100632b0 2024-04-14T15:19:06.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.769+0000 7f1b0d7fa700 10 client.49198.objecter ms_dispatch 0x7f1b10001550 command_reply(tid 7: 0 ) 2024-04-14T15:19:06.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.769+0000 7f1b0d7fa700 10 client.49198.objecter _finish_command 7 = system:0 2024-04-14T15:19:06.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.769+0000 7f1b0d7fa700 20 client.49198.objecter put_session s=0x7f1b10063140 osd=3 4 2024-04-14T15:19:06.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.769+0000 7f1b0d7fa700 15 client.49198.objecter _session_command_op_remove 3 7 2024-04-14T15:19:06.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.786+0000 7f1b166bd700 10 client.49198.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:19:06.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.786+0000 7f1b166bd700 15 client.49198.objecter _session_command_op_assign -1 8 2024-04-14T15:19:06.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.786+0000 7f1b166bd700 20 client.49198.objecter _get_session s=0x7f1b10063140 osd=3 3 2024-04-14T15:19:06.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.786+0000 7f1b166bd700 20 client.49198.objecter put_session s=0x7f1b10063140 osd=3 3 2024-04-14T15:19:06.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.786+0000 7f1b166bd700 20 client.49198.objecter _get_session s=0x7f1b10063140 osd=3 3 2024-04-14T15:19:06.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.786+0000 7f1b166bd700 15 client.49198.objecter _session_command_op_remove -1 8 2024-04-14T15:19:06.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.786+0000 7f1b166bd700 20 client.49198.objecter get_session s=0x7f1b10063140 osd=3 3 2024-04-14T15:19:06.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.786+0000 7f1b166bd700 15 client.49198.objecter _session_command_op_assign 3 8 2024-04-14T15:19:06.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.786+0000 7f1b166bd700 20 client.49198.objecter put_session s=0x7f1b10063140 osd=3 4 2024-04-14T15:19:06.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.786+0000 7f1b166bd700 10 client.49198.objecter _send_command 8 2024-04-14T15:19:06.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.786+0000 7f1b166bd700 1 -- 172.21.15.90:0/850305187 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1b101b1780 con 0x7f1b100632b0 2024-04-14T15:19:06.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.787+0000 7f1b0d7fa700 1 -- 172.21.15.90:0/850305187 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1b101b1780 con 0x7f1b100632b0 2024-04-14T15:19:06.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.787+0000 7f1b0d7fa700 10 client.49198.objecter ms_dispatch 0x7f1b10001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:19:06.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.787+0000 7f1b0d7fa700 10 client.49198.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:19:06.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.787+0000 7f1b0d7fa700 20 client.49198.objecter put_session s=0x7f1b10063140 osd=3 4 2024-04-14T15:19:06.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.787+0000 7f1b0d7fa700 15 client.49198.objecter _session_command_op_remove 3 8 2024-04-14T15:19:06.788 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:19:06.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.787+0000 7f1b166bd700 10 client.49198.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:19:06.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.787+0000 7f1b166bd700 15 client.49198.objecter _session_command_op_assign -1 9 2024-04-14T15:19:06.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.787+0000 7f1b166bd700 20 client.49198.objecter _get_session s=0x7f1b101b1780 osd=4 3 2024-04-14T15:19:06.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.787+0000 7f1b166bd700 20 client.49198.objecter put_session s=0x7f1b101b1780 osd=4 3 2024-04-14T15:19:06.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.787+0000 7f1b166bd700 20 client.49198.objecter _get_session s=0x7f1b101b1780 osd=4 3 2024-04-14T15:19:06.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.787+0000 7f1b166bd700 15 client.49198.objecter _session_command_op_remove -1 9 2024-04-14T15:19:06.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.787+0000 7f1b166bd700 20 client.49198.objecter get_session s=0x7f1b101b1780 osd=4 3 2024-04-14T15:19:06.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.787+0000 7f1b166bd700 15 client.49198.objecter _session_command_op_assign 4 9 2024-04-14T15:19:06.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.787+0000 7f1b166bd700 20 client.49198.objecter put_session s=0x7f1b101b1780 osd=4 4 2024-04-14T15:19:06.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.787+0000 7f1b166bd700 10 client.49198.objecter _send_command 9 2024-04-14T15:19:06.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.787+0000 7f1b166bd700 1 -- 172.21.15.90:0/850305187 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f1b101b6dd0 con 0x7f1b101b18f0 2024-04-14T15:19:06.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.788+0000 7f1b0d7fa700 10 client.49198.objecter ms_handle_connect 0x7f1b101b18f0 2024-04-14T15:19:06.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.788+0000 7f1b0d7fa700 1 -- 172.21.15.90:0/850305187 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1b101b6dd0 con 0x7f1b101b18f0 2024-04-14T15:19:06.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.788+0000 7f1b0d7fa700 10 client.49198.objecter ms_dispatch 0x7f1b10001550 command_reply(tid 9: 0 ) 2024-04-14T15:19:06.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.788+0000 7f1b0d7fa700 10 client.49198.objecter _finish_command 9 = system:0 2024-04-14T15:19:06.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.788+0000 7f1b0d7fa700 20 client.49198.objecter put_session s=0x7f1b101b1780 osd=4 4 2024-04-14T15:19:06.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.788+0000 7f1b0d7fa700 15 client.49198.objecter _session_command_op_remove 4 9 2024-04-14T15:19:06.792 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:19:06.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.805+0000 7f1b166bd700 10 client.49198.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:19:06.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.805+0000 7f1b166bd700 15 client.49198.objecter _session_command_op_assign -1 10 2024-04-14T15:19:06.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.805+0000 7f1b166bd700 20 client.49198.objecter _get_session s=0x7f1b101b1780 osd=4 3 2024-04-14T15:19:06.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.805+0000 7f1b166bd700 20 client.49198.objecter put_session s=0x7f1b101b1780 osd=4 3 2024-04-14T15:19:06.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.805+0000 7f1b166bd700 20 client.49198.objecter _get_session s=0x7f1b101b1780 osd=4 3 2024-04-14T15:19:06.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.805+0000 7f1b166bd700 15 client.49198.objecter _session_command_op_remove -1 10 2024-04-14T15:19:06.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.805+0000 7f1b166bd700 20 client.49198.objecter get_session s=0x7f1b101b1780 osd=4 3 2024-04-14T15:19:06.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.805+0000 7f1b166bd700 15 client.49198.objecter _session_command_op_assign 4 10 2024-04-14T15:19:06.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.805+0000 7f1b166bd700 20 client.49198.objecter put_session s=0x7f1b101b1780 osd=4 4 2024-04-14T15:19:06.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.805+0000 7f1b166bd700 10 client.49198.objecter _send_command 10 2024-04-14T15:19:06.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.805+0000 7f1b166bd700 1 -- 172.21.15.90:0/850305187 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1b101b6dd0 con 0x7f1b101b18f0 2024-04-14T15:19:06.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.807+0000 7f1b0d7fa700 1 -- 172.21.15.90:0/850305187 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1b101b6dd0 con 0x7f1b101b18f0 2024-04-14T15:19:06.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.807+0000 7f1b0d7fa700 10 client.49198.objecter ms_dispatch 0x7f1b10001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:19:06.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.807+0000 7f1b0d7fa700 10 client.49198.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:19:06.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.807+0000 7f1b0d7fa700 20 client.49198.objecter put_session s=0x7f1b101b1780 osd=4 4 2024-04-14T15:19:06.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.807+0000 7f1b0d7fa700 15 client.49198.objecter _session_command_op_remove 4 10 2024-04-14T15:19:06.808 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:19:06.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.807+0000 7f1b166bd700 10 client.49198.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:19:06.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.807+0000 7f1b166bd700 15 client.49198.objecter _session_command_op_assign -1 11 2024-04-14T15:19:06.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.807+0000 7f1b166bd700 20 client.49198.objecter _get_session s=0x7f1b101b6dd0 osd=5 3 2024-04-14T15:19:06.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.807+0000 7f1b166bd700 20 client.49198.objecter put_session s=0x7f1b101b6dd0 osd=5 3 2024-04-14T15:19:06.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.807+0000 7f1b166bd700 20 client.49198.objecter _get_session s=0x7f1b101b6dd0 osd=5 3 2024-04-14T15:19:06.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.807+0000 7f1b166bd700 15 client.49198.objecter _session_command_op_remove -1 11 2024-04-14T15:19:06.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.807+0000 7f1b166bd700 20 client.49198.objecter get_session s=0x7f1b101b6dd0 osd=5 3 2024-04-14T15:19:06.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.807+0000 7f1b166bd700 15 client.49198.objecter _session_command_op_assign 5 11 2024-04-14T15:19:06.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.807+0000 7f1b166bd700 20 client.49198.objecter put_session s=0x7f1b101b6dd0 osd=5 4 2024-04-14T15:19:06.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.807+0000 7f1b166bd700 10 client.49198.objecter _send_command 11 2024-04-14T15:19:06.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.807+0000 7f1b166bd700 1 -- 172.21.15.90:0/850305187 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f1b101c32b0 con 0x7f1b101bf600 2024-04-14T15:19:06.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.807+0000 7f1b0d7fa700 10 client.49198.objecter ms_handle_connect 0x7f1b101bf600 2024-04-14T15:19:06.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.808+0000 7f1b0d7fa700 1 -- 172.21.15.90:0/850305187 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1b101c32b0 con 0x7f1b101bf600 2024-04-14T15:19:06.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.808+0000 7f1b0d7fa700 10 client.49198.objecter ms_dispatch 0x7f1b10001550 command_reply(tid 11: 0 ) 2024-04-14T15:19:06.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.808+0000 7f1b0d7fa700 10 client.49198.objecter _finish_command 11 = system:0 2024-04-14T15:19:06.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.808+0000 7f1b0d7fa700 20 client.49198.objecter put_session s=0x7f1b101b6dd0 osd=5 4 2024-04-14T15:19:06.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.808+0000 7f1b0d7fa700 15 client.49198.objecter _session_command_op_remove 5 11 2024-04-14T15:19:06.815 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:06.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:06.816 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:19:06.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.825+0000 7f1b166bd700 10 client.49198.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:19:06.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.825+0000 7f1b166bd700 15 client.49198.objecter _session_command_op_assign -1 12 2024-04-14T15:19:06.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.825+0000 7f1b166bd700 20 client.49198.objecter _get_session s=0x7f1b101b6dd0 osd=5 3 2024-04-14T15:19:06.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.825+0000 7f1b166bd700 20 client.49198.objecter put_session s=0x7f1b101b6dd0 osd=5 3 2024-04-14T15:19:06.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.825+0000 7f1b166bd700 20 client.49198.objecter _get_session s=0x7f1b101b6dd0 osd=5 3 2024-04-14T15:19:06.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.825+0000 7f1b166bd700 15 client.49198.objecter _session_command_op_remove -1 12 2024-04-14T15:19:06.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.825+0000 7f1b166bd700 20 client.49198.objecter get_session s=0x7f1b101b6dd0 osd=5 3 2024-04-14T15:19:06.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.825+0000 7f1b166bd700 15 client.49198.objecter _session_command_op_assign 5 12 2024-04-14T15:19:06.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.825+0000 7f1b166bd700 20 client.49198.objecter put_session s=0x7f1b101b6dd0 osd=5 4 2024-04-14T15:19:06.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.825+0000 7f1b166bd700 10 client.49198.objecter _send_command 12 2024-04-14T15:19:06.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.825+0000 7f1b166bd700 1 -- 172.21.15.90:0/850305187 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1b101c4c60 con 0x7f1b101bf600 2024-04-14T15:19:06.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.826+0000 7f1b0d7fa700 1 -- 172.21.15.90:0/850305187 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1b101c4c60 con 0x7f1b101bf600 2024-04-14T15:19:06.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.827+0000 7f1b0d7fa700 10 client.49198.objecter ms_dispatch 0x7f1b10001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:19:06.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.827+0000 7f1b0d7fa700 10 client.49198.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:19:06.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.827+0000 7f1b0d7fa700 20 client.49198.objecter put_session s=0x7f1b101b6dd0 osd=5 4 2024-04-14T15:19:06.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.827+0000 7f1b0d7fa700 15 client.49198.objecter _session_command_op_remove 5 12 2024-04-14T15:19:06.827 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:19:06.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.827+0000 7f1b166bd700 10 client.49198.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:19:06.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.827+0000 7f1b166bd700 15 client.49198.objecter _session_command_op_assign -1 13 2024-04-14T15:19:06.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.827+0000 7f1b166bd700 20 client.49198.objecter _get_session s=0x7f1b101c4c60 osd=6 3 2024-04-14T15:19:06.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.827+0000 7f1b166bd700 20 client.49198.objecter put_session s=0x7f1b101c4c60 osd=6 3 2024-04-14T15:19:06.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.827+0000 7f1b166bd700 20 client.49198.objecter _get_session s=0x7f1b101c4c60 osd=6 3 2024-04-14T15:19:06.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.827+0000 7f1b166bd700 15 client.49198.objecter _session_command_op_remove -1 13 2024-04-14T15:19:06.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.827+0000 7f1b166bd700 20 client.49198.objecter get_session s=0x7f1b101c4c60 osd=6 3 2024-04-14T15:19:06.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.827+0000 7f1b166bd700 15 client.49198.objecter _session_command_op_assign 6 13 2024-04-14T15:19:06.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.827+0000 7f1b166bd700 20 client.49198.objecter put_session s=0x7f1b101c4c60 osd=6 4 2024-04-14T15:19:06.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.827+0000 7f1b166bd700 10 client.49198.objecter _send_command 13 2024-04-14T15:19:06.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.827+0000 7f1b166bd700 1 -- 172.21.15.90:0/850305187 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f1b101c5800 con 0x7f1b101c5380 2024-04-14T15:19:06.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.827+0000 7f1b0d7fa700 10 client.49198.objecter ms_handle_connect 0x7f1b101c5380 2024-04-14T15:19:06.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.828+0000 7f1b0d7fa700 1 -- 172.21.15.90:0/850305187 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1b101c5800 con 0x7f1b101c5380 2024-04-14T15:19:06.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.828+0000 7f1b0d7fa700 10 client.49198.objecter ms_dispatch 0x7f1b10001550 command_reply(tid 13: 0 ) 2024-04-14T15:19:06.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.828+0000 7f1b0d7fa700 10 client.49198.objecter _finish_command 13 = system:0 2024-04-14T15:19:06.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.828+0000 7f1b0d7fa700 20 client.49198.objecter put_session s=0x7f1b101c4c60 osd=6 4 2024-04-14T15:19:06.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.828+0000 7f1b0d7fa700 15 client.49198.objecter _session_command_op_remove 6 13 2024-04-14T15:19:06.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.844+0000 7f1b166bd700 10 client.49198.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:19:06.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.844+0000 7f1b166bd700 15 client.49198.objecter _session_command_op_assign -1 14 2024-04-14T15:19:06.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.844+0000 7f1b166bd700 20 client.49198.objecter _get_session s=0x7f1b101c4c60 osd=6 3 2024-04-14T15:19:06.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.844+0000 7f1b166bd700 20 client.49198.objecter put_session s=0x7f1b101c4c60 osd=6 3 2024-04-14T15:19:06.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.844+0000 7f1b166bd700 20 client.49198.objecter _get_session s=0x7f1b101c4c60 osd=6 3 2024-04-14T15:19:06.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.844+0000 7f1b166bd700 15 client.49198.objecter _session_command_op_remove -1 14 2024-04-14T15:19:06.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.844+0000 7f1b166bd700 20 client.49198.objecter get_session s=0x7f1b101c4c60 osd=6 3 2024-04-14T15:19:06.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.844+0000 7f1b166bd700 15 client.49198.objecter _session_command_op_assign 6 14 2024-04-14T15:19:06.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.844+0000 7f1b166bd700 20 client.49198.objecter put_session s=0x7f1b101c4c60 osd=6 4 2024-04-14T15:19:06.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.844+0000 7f1b166bd700 10 client.49198.objecter _send_command 14 2024-04-14T15:19:06.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.844+0000 7f1b166bd700 1 -- 172.21.15.90:0/850305187 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1b101c5800 con 0x7f1b101c5380 2024-04-14T15:19:06.846 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:19:06.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.846+0000 7f1b0d7fa700 1 -- 172.21.15.90:0/850305187 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1b101c5800 con 0x7f1b101c5380 2024-04-14T15:19:06.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.846+0000 7f1b0d7fa700 10 client.49198.objecter ms_dispatch 0x7f1b10001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:19:06.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.846+0000 7f1b0d7fa700 10 client.49198.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:19:06.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.846+0000 7f1b0d7fa700 20 client.49198.objecter put_session s=0x7f1b101c4c60 osd=6 4 2024-04-14T15:19:06.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.846+0000 7f1b0d7fa700 15 client.49198.objecter _session_command_op_remove 6 14 2024-04-14T15:19:06.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.846+0000 7f1b166bd700 10 client.49198.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:19:06.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.846+0000 7f1b166bd700 15 client.49198.objecter _session_command_op_assign -1 15 2024-04-14T15:19:06.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.846+0000 7f1b166bd700 20 client.49198.objecter _get_session s=0x7f1b101c5800 osd=7 3 2024-04-14T15:19:06.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.846+0000 7f1b166bd700 20 client.49198.objecter put_session s=0x7f1b101c5800 osd=7 3 2024-04-14T15:19:06.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.846+0000 7f1b166bd700 20 client.49198.objecter _get_session s=0x7f1b101c5800 osd=7 3 2024-04-14T15:19:06.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.846+0000 7f1b166bd700 15 client.49198.objecter _session_command_op_remove -1 15 2024-04-14T15:19:06.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.846+0000 7f1b166bd700 20 client.49198.objecter get_session s=0x7f1b101c5800 osd=7 3 2024-04-14T15:19:06.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.846+0000 7f1b166bd700 15 client.49198.objecter _session_command_op_assign 7 15 2024-04-14T15:19:06.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.846+0000 7f1b166bd700 20 client.49198.objecter put_session s=0x7f1b101c5800 osd=7 4 2024-04-14T15:19:06.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.846+0000 7f1b166bd700 10 client.49198.objecter _send_command 15 2024-04-14T15:19:06.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.846+0000 7f1b166bd700 1 -- 172.21.15.90:0/850305187 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f1b101c5be0 con 0x7f1b101c8370 2024-04-14T15:19:06.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.847+0000 7f1b0d7fa700 10 client.49198.objecter ms_handle_connect 0x7f1b101c8370 2024-04-14T15:19:06.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.847+0000 7f1b0d7fa700 1 -- 172.21.15.90:0/850305187 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1b101c5be0 con 0x7f1b101c8370 2024-04-14T15:19:06.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.847+0000 7f1b0d7fa700 10 client.49198.objecter ms_dispatch 0x7f1b10001550 command_reply(tid 15: 0 ) 2024-04-14T15:19:06.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.847+0000 7f1b0d7fa700 10 client.49198.objecter _finish_command 15 = system:0 2024-04-14T15:19:06.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.847+0000 7f1b0d7fa700 20 client.49198.objecter put_session s=0x7f1b101c5800 osd=7 4 2024-04-14T15:19:06.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.847+0000 7f1b0d7fa700 15 client.49198.objecter _session_command_op_remove 7 15 2024-04-14T15:19:06.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.864+0000 7f1b166bd700 10 client.49198.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:19:06.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.864+0000 7f1b166bd700 15 client.49198.objecter _session_command_op_assign -1 16 2024-04-14T15:19:06.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.864+0000 7f1b166bd700 20 client.49198.objecter _get_session s=0x7f1b101c5800 osd=7 3 2024-04-14T15:19:06.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.864+0000 7f1b166bd700 20 client.49198.objecter put_session s=0x7f1b101c5800 osd=7 3 2024-04-14T15:19:06.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.864+0000 7f1b166bd700 20 client.49198.objecter _get_session s=0x7f1b101c5800 osd=7 3 2024-04-14T15:19:06.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.864+0000 7f1b166bd700 15 client.49198.objecter _session_command_op_remove -1 16 2024-04-14T15:19:06.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.864+0000 7f1b166bd700 20 client.49198.objecter get_session s=0x7f1b101c5800 osd=7 3 2024-04-14T15:19:06.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.864+0000 7f1b166bd700 15 client.49198.objecter _session_command_op_assign 7 16 2024-04-14T15:19:06.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.864+0000 7f1b166bd700 20 client.49198.objecter put_session s=0x7f1b101c5800 osd=7 4 2024-04-14T15:19:06.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.864+0000 7f1b166bd700 10 client.49198.objecter _send_command 16 2024-04-14T15:19:06.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.864+0000 7f1b166bd700 1 -- 172.21.15.90:0/850305187 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1b101cbe80 con 0x7f1b101c8370 2024-04-14T15:19:06.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.866+0000 7f1b0d7fa700 1 -- 172.21.15.90:0/850305187 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1b101cbe80 con 0x7f1b101c8370 2024-04-14T15:19:06.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.866+0000 7f1b0d7fa700 10 client.49198.objecter ms_dispatch 0x7f1b10001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:19:06.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.866+0000 7f1b0d7fa700 10 client.49198.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:19:06.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.866+0000 7f1b0d7fa700 20 client.49198.objecter put_session s=0x7f1b101c5800 osd=7 4 2024-04-14T15:19:06.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.866+0000 7f1b0d7fa700 15 client.49198.objecter _session_command_op_remove 7 16 2024-04-14T15:19:06.867 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:19:06.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.866+0000 7f1b166bd700 10 librados: watch_flush enter 2024-04-14T15:19:06.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.866+0000 7f1b166bd700 10 librados: watch_flush exit 2024-04-14T15:19:06.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.866+0000 7f1b166bd700 10 client.49198.objecter close_session for osd.0 2024-04-14T15:19:06.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.866+0000 7f1b166bd700 1 -- 172.21.15.90:0/850305187 >> v1:172.21.15.90:6801/3761806885 conn(0x7f1b101ac210 legacy=0x7f1b101ac5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:06.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.866+0000 7f1b166bd700 20 client.49198.objecter put_session s=0x7f1b101132c0 osd=0 1 2024-04-14T15:19:06.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.866+0000 7f1b166bd700 10 client.49198.objecter close_session for osd.1 2024-04-14T15:19:06.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.866+0000 7f1b166bd700 1 -- 172.21.15.90:0/850305187 >> v1:172.21.15.90:6805/4170361699 conn(0x7f1b10110cc0 legacy=0x7f1b101110c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:06.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.866+0000 7f1b166bd700 20 client.49198.objecter put_session s=0x7f1b101acf50 osd=1 1 2024-04-14T15:19:06.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.866+0000 7f1b166bd700 10 client.49198.objecter close_session for osd.2 2024-04-14T15:19:06.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.866+0000 7f1b166bd700 1 -- 172.21.15.90:0/850305187 >> v1:172.21.15.90:6809/1686184251 conn(0x7f1b100636c0 legacy=0x7f1b101b46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:06.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.866+0000 7f1b166bd700 20 client.49198.objecter put_session s=0x7f1b1002e3e0 osd=2 1 2024-04-14T15:19:06.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.866+0000 7f1b166bd700 10 client.49198.objecter close_session for osd.3 2024-04-14T15:19:06.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.866+0000 7f1b166bd700 1 -- 172.21.15.90:0/850305187 >> v1:172.21.15.90:6813/2270166774 conn(0x7f1b100632b0 legacy=0x7f1b101b9120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:06.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.866+0000 7f1b166bd700 20 client.49198.objecter put_session s=0x7f1b10063140 osd=3 1 2024-04-14T15:19:06.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.866+0000 7f1b166bd700 10 client.49198.objecter close_session for osd.4 2024-04-14T15:19:06.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.866+0000 7f1b166bd700 1 -- 172.21.15.90:0/850305187 >> v1:172.21.15.92:6800/3887549733 conn(0x7f1b101b18f0 legacy=0x7f1b101bd9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:06.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.866+0000 7f1b166bd700 20 client.49198.objecter put_session s=0x7f1b101b1780 osd=4 1 2024-04-14T15:19:06.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.866+0000 7f1b166bd700 10 client.49198.objecter close_session for osd.5 2024-04-14T15:19:06.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.866+0000 7f1b166bd700 1 -- 172.21.15.90:0/850305187 >> v1:172.21.15.92:6804/3244561440 conn(0x7f1b101bf600 legacy=0x7f1b101c1a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:06.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.866+0000 7f1b166bd700 20 client.49198.objecter put_session s=0x7f1b101b6dd0 osd=5 1 2024-04-14T15:19:06.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.866+0000 7f1b166bd700 10 client.49198.objecter close_session for osd.6 2024-04-14T15:19:06.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.866+0000 7f1b14c5a700 1 -- 172.21.15.90:0/850305187 reap_dead start 2024-04-14T15:19:06.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.866+0000 7f1b166bd700 1 -- 172.21.15.90:0/850305187 >> v1:172.21.15.92:6808/636125963 conn(0x7f1b101c5380 legacy=0x7f1b101c3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:06.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.866+0000 7f1b14c5a700 1 -- 172.21.15.90:0/850305187 reap_dead start 2024-04-14T15:19:06.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.866+0000 7f1b14c5a700 1 -- 172.21.15.90:0/850305187 reap_dead start 2024-04-14T15:19:06.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.866+0000 7f1b14c5a700 1 -- 172.21.15.90:0/850305187 reap_dead start 2024-04-14T15:19:06.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.866+0000 7f1b166bd700 20 client.49198.objecter put_session s=0x7f1b101c4c60 osd=6 1 2024-04-14T15:19:06.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.866+0000 7f1b166bd700 10 client.49198.objecter close_session for osd.7 2024-04-14T15:19:06.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.866+0000 7f1b166bd700 1 -- 172.21.15.90:0/850305187 >> v1:172.21.15.92:6812/1773706716 conn(0x7f1b101c8370 legacy=0x7f1b101ca760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:06.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.866+0000 7f1b166bd700 20 client.49198.objecter put_session s=0x7f1b101c5800 osd=7 1 2024-04-14T15:19:06.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.866+0000 7f1b166bd700 20 client.49198.objecter shutdown clearing up homeless session... 2024-04-14T15:19:06.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.866+0000 7f1b166bd700 10 client.49198.objecter successfully canceled tick 2024-04-14T15:19:06.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.866+0000 7f1b166bd700 1 -- 172.21.15.90:0/850305187 >> v1:172.21.15.90:6800/420935427 conn(0x7f1af8071460 legacy=0x7f1af8073920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:06.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.866+0000 7f1b166bd700 1 -- 172.21.15.90:0/850305187 >> v1:172.21.15.92:6789/0 conn(0x7f1b10108870 legacy=0x7f1b10076640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:06.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.866+0000 7f1b166bd700 1 -- 172.21.15.90:0/850305187 shutdown_connections 2024-04-14T15:19:06.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.866+0000 7f1b166bd700 1 -- 172.21.15.90:0/850305187 >> 172.21.15.90:0/850305187 conn(0x7f1b100fe3a0 msgr2=0x7f1b10100f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:19:06.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.866+0000 7f1b166bd700 1 -- 172.21.15.90:0/850305187 shutdown_connections 2024-04-14T15:19:06.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.866+0000 7f1b166bd700 1 -- 172.21.15.90:0/850305187 wait complete. 2024-04-14T15:19:06.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:06.867+0000 7f1b166bd700 1 librados: shutdown 2024-04-14T15:19:06.871 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:19:06.871 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:19:06.871 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:19:06.871 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:19:06.871 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:19:06.871 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:19:06.871 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:19:06.871 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:19:06.917 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:19:06.946 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:06.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:06.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:19:06.981 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:06 smithi090 ceph-mon[30399]: pgmap v638: 226 pgs: 226 active+clean; 8.5 MiB data, 587 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:19:06.982 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:06 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:19:06.982 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:06 smithi090 ceph-mon[30399]: 1.4 scrub starts 2024-04-14T15:19:06.982 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:06 smithi090 ceph-mon[30399]: 1.4 scrub ok 2024-04-14T15:19:06.982 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:06 smithi090 ceph-mon[30399]: 1.10 scrub starts 2024-04-14T15:19:06.982 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:06 smithi090 ceph-mon[30399]: 1.10 scrub ok 2024-04-14T15:19:06.982 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:06 smithi090 ceph-mon[30399]: 1.e scrub starts 2024-04-14T15:19:06.982 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:06 smithi090 ceph-mon[30399]: 1.e scrub ok 2024-04-14T15:19:06.982 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:06 smithi090 ceph-mon[30399]: 1.6 scrub starts 2024-04-14T15:19:06.982 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:06 smithi090 ceph-mon[30399]: 1.6 scrub ok 2024-04-14T15:19:06.982 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:06 smithi090 ceph-mon[30399]: 1.a scrub starts 2024-04-14T15:19:06.982 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:06 smithi090 ceph-mon[30399]: 1.a scrub ok 2024-04-14T15:19:06.982 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:06 smithi090 ceph-mon[30399]: 1.12 scrub starts 2024-04-14T15:19:06.982 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:06 smithi090 ceph-mon[30399]: 1.12 scrub ok 2024-04-14T15:19:06.982 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:06 smithi090 ceph-mon[30399]: 1.1 scrub starts 2024-04-14T15:19:06.982 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:06 smithi090 ceph-mon[30399]: 1.1 scrub ok 2024-04-14T15:19:06.983 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:06 smithi090 ceph-mon[30399]: 1.1b scrub starts 2024-04-14T15:19:06.983 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:06 smithi090 ceph-mon[30399]: 1.1b scrub ok 2024-04-14T15:19:06.983 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:06 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/850305187' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:19:06.983 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:06 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:19:06.983 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:06 smithi090 ceph-mon[30399]: osdmap e455: 8 total, 8 up, 8 in 2024-04-14T15:19:06.983 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:06 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-14T15:19:06.983 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:06 smithi090 ceph-mon[41670]: pgmap v638: 226 pgs: 226 active+clean; 8.5 MiB data, 587 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:19:06.983 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:06 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:19:06.983 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:06 smithi090 ceph-mon[41670]: 1.4 scrub starts 2024-04-14T15:19:06.983 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:06 smithi090 ceph-mon[41670]: 1.4 scrub ok 2024-04-14T15:19:06.983 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:06 smithi090 ceph-mon[41670]: 1.10 scrub starts 2024-04-14T15:19:06.983 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:06 smithi090 ceph-mon[41670]: 1.10 scrub ok 2024-04-14T15:19:06.984 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:06 smithi090 ceph-mon[41670]: 1.e scrub starts 2024-04-14T15:19:06.984 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:06 smithi090 ceph-mon[41670]: 1.e scrub ok 2024-04-14T15:19:06.984 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:06 smithi090 ceph-mon[41670]: 1.6 scrub starts 2024-04-14T15:19:06.984 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:06 smithi090 ceph-mon[41670]: 1.6 scrub ok 2024-04-14T15:19:06.984 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:06 smithi090 ceph-mon[41670]: 1.a scrub starts 2024-04-14T15:19:06.984 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:06 smithi090 ceph-mon[41670]: 1.a scrub ok 2024-04-14T15:19:06.984 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:06 smithi090 ceph-mon[41670]: 1.12 scrub starts 2024-04-14T15:19:06.984 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:06 smithi090 ceph-mon[41670]: 1.12 scrub ok 2024-04-14T15:19:06.984 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:06 smithi090 ceph-mon[41670]: 1.1 scrub starts 2024-04-14T15:19:06.984 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:06 smithi090 ceph-mon[41670]: 1.1 scrub ok 2024-04-14T15:19:06.984 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:06 smithi090 ceph-mon[41670]: 1.1b scrub starts 2024-04-14T15:19:06.984 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:06 smithi090 ceph-mon[41670]: 1.1b scrub ok 2024-04-14T15:19:06.984 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:06 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/850305187' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:19:06.984 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:06 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:19:06.984 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:06 smithi090 ceph-mon[41670]: osdmap e455: 8 total, 8 up, 8 in 2024-04-14T15:19:06.984 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:06 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-14T15:19:07.048 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:19:07.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:07.072 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:07.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:19:07.173 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:19:07.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:07.197 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:07.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:19:07.298 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:19:07.322 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:07.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:07.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:19:07.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:06 smithi092 ceph-mon[33966]: pgmap v638: 226 pgs: 226 active+clean; 8.5 MiB data, 587 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:19:07.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:06 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:19:07.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:06 smithi092 ceph-mon[33966]: 1.4 scrub starts 2024-04-14T15:19:07.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:06 smithi092 ceph-mon[33966]: 1.4 scrub ok 2024-04-14T15:19:07.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:06 smithi092 ceph-mon[33966]: 1.10 scrub starts 2024-04-14T15:19:07.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:06 smithi092 ceph-mon[33966]: 1.10 scrub ok 2024-04-14T15:19:07.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:06 smithi092 ceph-mon[33966]: 1.e scrub starts 2024-04-14T15:19:07.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:06 smithi092 ceph-mon[33966]: 1.e scrub ok 2024-04-14T15:19:07.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:06 smithi092 ceph-mon[33966]: 1.6 scrub starts 2024-04-14T15:19:07.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:06 smithi092 ceph-mon[33966]: 1.6 scrub ok 2024-04-14T15:19:07.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:06 smithi092 ceph-mon[33966]: 1.a scrub starts 2024-04-14T15:19:07.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:06 smithi092 ceph-mon[33966]: 1.a scrub ok 2024-04-14T15:19:07.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:06 smithi092 ceph-mon[33966]: 1.12 scrub starts 2024-04-14T15:19:07.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:06 smithi092 ceph-mon[33966]: 1.12 scrub ok 2024-04-14T15:19:07.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:06 smithi092 ceph-mon[33966]: 1.1 scrub starts 2024-04-14T15:19:07.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:06 smithi092 ceph-mon[33966]: 1.1 scrub ok 2024-04-14T15:19:07.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:06 smithi092 ceph-mon[33966]: 1.1b scrub starts 2024-04-14T15:19:07.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:06 smithi092 ceph-mon[33966]: 1.1b scrub ok 2024-04-14T15:19:07.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:06 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/850305187' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:19:07.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:06 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:19:07.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:06 smithi092 ceph-mon[33966]: osdmap e455: 8 total, 8 up, 8 in 2024-04-14T15:19:07.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:06 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-14T15:19:07.424 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:19:07.448 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:07.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:07.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:19:07.549 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:19:07.573 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:07.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:07.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:19:07.675 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:19:07.699 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:07.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:07.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:19:07.800 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:19:07.824 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:07.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:07.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:19:07.926 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:19:07.953 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:07.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:07.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:19:08.055 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:19:08.079 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:08.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:08.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:19:08.181 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:19:08.203 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:08.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:08.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:19:08.266 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:07 smithi090 ceph-mon[30399]: 1.19 scrub starts 2024-04-14T15:19:08.266 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:07 smithi090 ceph-mon[30399]: 1.19 scrub ok 2024-04-14T15:19:08.266 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:07 smithi090 ceph-mon[30399]: 1.9 scrub starts 2024-04-14T15:19:08.266 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:07 smithi090 ceph-mon[30399]: 1.9 scrub ok 2024-04-14T15:19:08.266 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:07 smithi090 ceph-mon[30399]: 1.f scrub starts 2024-04-14T15:19:08.266 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:07 smithi090 ceph-mon[30399]: 1.f scrub ok 2024-04-14T15:19:08.266 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:07 smithi090 ceph-mon[30399]: 1.14 scrub starts 2024-04-14T15:19:08.266 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:07 smithi090 ceph-mon[30399]: 1.14 scrub ok 2024-04-14T15:19:08.266 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:07 smithi090 ceph-mon[30399]: 1.13 scrub starts 2024-04-14T15:19:08.266 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:07 smithi090 ceph-mon[30399]: 1.13 scrub ok 2024-04-14T15:19:08.266 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:07 smithi090 ceph-mon[30399]: osdmap e456: 8 total, 8 up, 8 in 2024-04-14T15:19:08.266 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:07 smithi090 ceph-mon[41670]: 1.19 scrub starts 2024-04-14T15:19:08.267 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:07 smithi090 ceph-mon[41670]: 1.19 scrub ok 2024-04-14T15:19:08.267 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:07 smithi090 ceph-mon[41670]: 1.9 scrub starts 2024-04-14T15:19:08.267 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:07 smithi090 ceph-mon[41670]: 1.9 scrub ok 2024-04-14T15:19:08.267 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:07 smithi090 ceph-mon[41670]: 1.f scrub starts 2024-04-14T15:19:08.267 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:07 smithi090 ceph-mon[41670]: 1.f scrub ok 2024-04-14T15:19:08.267 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:07 smithi090 ceph-mon[41670]: 1.14 scrub starts 2024-04-14T15:19:08.267 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:07 smithi090 ceph-mon[41670]: 1.14 scrub ok 2024-04-14T15:19:08.267 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:07 smithi090 ceph-mon[41670]: 1.13 scrub starts 2024-04-14T15:19:08.267 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:07 smithi090 ceph-mon[41670]: 1.13 scrub ok 2024-04-14T15:19:08.267 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:07 smithi090 ceph-mon[41670]: osdmap e456: 8 total, 8 up, 8 in 2024-04-14T15:19:08.306 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:19:08.329 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:08.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:08.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:19:08.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:07 smithi092 ceph-mon[33966]: 1.19 scrub starts 2024-04-14T15:19:08.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:07 smithi092 ceph-mon[33966]: 1.19 scrub ok 2024-04-14T15:19:08.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:07 smithi092 ceph-mon[33966]: 1.9 scrub starts 2024-04-14T15:19:08.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:07 smithi092 ceph-mon[33966]: 1.9 scrub ok 2024-04-14T15:19:08.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:07 smithi092 ceph-mon[33966]: 1.f scrub starts 2024-04-14T15:19:08.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:07 smithi092 ceph-mon[33966]: 1.f scrub ok 2024-04-14T15:19:08.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:07 smithi092 ceph-mon[33966]: 1.14 scrub starts 2024-04-14T15:19:08.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:07 smithi092 ceph-mon[33966]: 1.14 scrub ok 2024-04-14T15:19:08.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:07 smithi092 ceph-mon[33966]: 1.13 scrub starts 2024-04-14T15:19:08.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:07 smithi092 ceph-mon[33966]: 1.13 scrub ok 2024-04-14T15:19:08.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:07 smithi092 ceph-mon[33966]: osdmap e456: 8 total, 8 up, 8 in 2024-04-14T15:19:08.430 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:08.455 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:08.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:08.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:08.557 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:19:08.581 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:08.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:08.582 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:19:08.683 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:08.707 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:08.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:08.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:08.809 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:19:08.833 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:08.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:08.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:19:08.934 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:19:08.959 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:08.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:08.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:19:09.060 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:09.084 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:09.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:09.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:09.185 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:19:09.209 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:09.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:09.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:19:09.311 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:19:09.334 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:09.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:09.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:19:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:08 smithi092 ceph-mon[33966]: pgmap v641: 258 pgs: 32 unknown, 226 active+clean; 8.5 MiB data, 587 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:19:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:08 smithi092 ceph-mon[33966]: 1.2 scrub starts 2024-04-14T15:19:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:08 smithi092 ceph-mon[33966]: 1.2 scrub ok 2024-04-14T15:19:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:08 smithi092 ceph-mon[33966]: 1.7 scrub starts 2024-04-14T15:19:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:08 smithi092 ceph-mon[33966]: 1.7 scrub ok 2024-04-14T15:19:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:08 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:19:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:08 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:19:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:08 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-67"}]: dispatch 2024-04-14T15:19:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:08 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-67"}]: dispatch 2024-04-14T15:19:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:08 smithi092 ceph-mon[33966]: 1.b scrub starts 2024-04-14T15:19:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:08 smithi092 ceph-mon[33966]: 1.b scrub ok 2024-04-14T15:19:09.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:08 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:19:09.436 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:19:09.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:08 smithi090 ceph-mon[41670]: pgmap v641: 258 pgs: 32 unknown, 226 active+clean; 8.5 MiB data, 587 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:19:09.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:08 smithi090 ceph-mon[41670]: 1.2 scrub starts 2024-04-14T15:19:09.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:08 smithi090 ceph-mon[41670]: 1.2 scrub ok 2024-04-14T15:19:09.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:08 smithi090 ceph-mon[41670]: 1.7 scrub starts 2024-04-14T15:19:09.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:08 smithi090 ceph-mon[41670]: 1.7 scrub ok 2024-04-14T15:19:09.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:08 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:19:09.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:08 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:19:09.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:08 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-67"}]: dispatch 2024-04-14T15:19:09.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:08 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-67"}]: dispatch 2024-04-14T15:19:09.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:08 smithi090 ceph-mon[41670]: 1.b scrub starts 2024-04-14T15:19:09.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:08 smithi090 ceph-mon[41670]: 1.b scrub ok 2024-04-14T15:19:09.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:08 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:19:09.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:08 smithi090 ceph-mon[30399]: pgmap v641: 258 pgs: 32 unknown, 226 active+clean; 8.5 MiB data, 587 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:19:09.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:08 smithi090 ceph-mon[30399]: 1.2 scrub starts 2024-04-14T15:19:09.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:08 smithi090 ceph-mon[30399]: 1.2 scrub ok 2024-04-14T15:19:09.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:08 smithi090 ceph-mon[30399]: 1.7 scrub starts 2024-04-14T15:19:09.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:08 smithi090 ceph-mon[30399]: 1.7 scrub ok 2024-04-14T15:19:09.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:08 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:19:09.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:08 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:19:09.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:08 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-67"}]: dispatch 2024-04-14T15:19:09.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:08 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-67"}]: dispatch 2024-04-14T15:19:09.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:08 smithi090 ceph-mon[30399]: 1.b scrub starts 2024-04-14T15:19:09.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:08 smithi090 ceph-mon[30399]: 1.b scrub ok 2024-04-14T15:19:09.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:08 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:19:09.460 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:09.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:09.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:19:09.562 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:19:09.586 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:09.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:09.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:19:09.687 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:19:09.715 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:09.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:09.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:19:09.816 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:19:09.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:09.843 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:09.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:19:09.944 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:19:09.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:09.970 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:09.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:19:10.071 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:19:10.096 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:10.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:10.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:19:10.149 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:19:10.185 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-14T15:19:10.198 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:19:10.222 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:10.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:10.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:19:10.244 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:09 smithi090 ceph-mon[41670]: 1.0 scrub starts 2024-04-14T15:19:10.244 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:09 smithi090 ceph-mon[41670]: 1.0 scrub ok 2024-04-14T15:19:10.244 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:09 smithi090 ceph-mon[41670]: osdmap e457: 8 total, 8 up, 8 in 2024-04-14T15:19:10.244 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:09 smithi090 ceph-mon[41670]: 1.3 scrub starts 2024-04-14T15:19:10.244 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:09 smithi090 ceph-mon[41670]: 1.3 scrub ok 2024-04-14T15:19:10.244 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:09 smithi090 ceph-mon[30399]: 1.0 scrub starts 2024-04-14T15:19:10.244 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:09 smithi090 ceph-mon[30399]: 1.0 scrub ok 2024-04-14T15:19:10.244 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:09 smithi090 ceph-mon[30399]: osdmap e457: 8 total, 8 up, 8 in 2024-04-14T15:19:10.244 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:09 smithi090 ceph-mon[30399]: 1.3 scrub starts 2024-04-14T15:19:10.244 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:09 smithi090 ceph-mon[30399]: 1.3 scrub ok 2024-04-14T15:19:10.324 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:19:10.347 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:10.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:10.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:19:10.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:09 smithi092 ceph-mon[33966]: 1.0 scrub starts 2024-04-14T15:19:10.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:09 smithi092 ceph-mon[33966]: 1.0 scrub ok 2024-04-14T15:19:10.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:09 smithi092 ceph-mon[33966]: osdmap e457: 8 total, 8 up, 8 in 2024-04-14T15:19:10.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:09 smithi092 ceph-mon[33966]: 1.3 scrub starts 2024-04-14T15:19:10.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:09 smithi092 ceph-mon[33966]: 1.3 scrub ok 2024-04-14T15:19:10.449 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:19:10.473 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:10.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:10.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:19:10.574 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:19:10.599 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:10.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:10.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:19:10.701 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:19:10.725 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:10.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:10.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:19:10.826 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:19:10.850 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:10.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:10.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:19:10.951 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:19:10.977 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:10.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:10.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:19:11.078 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:19:11.102 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:11.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:11.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:19:11.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:10 smithi090 ceph-mon[41670]: pgmap v644: 226 pgs: 226 active+clean; 8.6 MiB data, 587 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:19:11.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:10 smithi090 ceph-mon[41670]: 1.d scrub starts 2024-04-14T15:19:11.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:10 smithi090 ceph-mon[41670]: 1.d scrub ok 2024-04-14T15:19:11.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:10 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:19:11.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:10 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:19:11.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:10 smithi090 ceph-mon[41670]: osdmap e458: 8 total, 8 up, 8 in 2024-04-14T15:19:11.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:10 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:19:11.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:10 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:19:11.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:10 smithi090 ceph-mon[41670]: 11.7 scrub starts 2024-04-14T15:19:11.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:10 smithi090 ceph-mon[41670]: 11.7 scrub ok 2024-04-14T15:19:11.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:10 smithi090 ceph-mon[30399]: pgmap v644: 226 pgs: 226 active+clean; 8.6 MiB data, 587 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:19:11.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:10 smithi090 ceph-mon[30399]: 1.d scrub starts 2024-04-14T15:19:11.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:10 smithi090 ceph-mon[30399]: 1.d scrub ok 2024-04-14T15:19:11.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:10 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:19:11.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:10 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:19:11.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:10 smithi090 ceph-mon[30399]: osdmap e458: 8 total, 8 up, 8 in 2024-04-14T15:19:11.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:10 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:19:11.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:10 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:19:11.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:10 smithi090 ceph-mon[30399]: 11.7 scrub starts 2024-04-14T15:19:11.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:10 smithi090 ceph-mon[30399]: 11.7 scrub ok 2024-04-14T15:19:11.204 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:11.227 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:11.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:11.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:11.329 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:19:11.352 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:11.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:11.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:19:11.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:10 smithi092 ceph-mon[33966]: pgmap v644: 226 pgs: 226 active+clean; 8.6 MiB data, 587 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:19:11.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:10 smithi092 ceph-mon[33966]: 1.d scrub starts 2024-04-14T15:19:11.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:10 smithi092 ceph-mon[33966]: 1.d scrub ok 2024-04-14T15:19:11.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:10 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:19:11.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:10 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:19:11.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:10 smithi092 ceph-mon[33966]: osdmap e458: 8 total, 8 up, 8 in 2024-04-14T15:19:11.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:10 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:19:11.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:10 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:19:11.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:10 smithi092 ceph-mon[33966]: 11.7 scrub starts 2024-04-14T15:19:11.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:10 smithi092 ceph-mon[33966]: 11.7 scrub ok 2024-04-14T15:19:11.454 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:19:11.478 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:11.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:11.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:19:11.580 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:19:11.604 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:11.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:11.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:19:11.706 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:19:11.729 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:11.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:11.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:19:11.831 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:19:11.855 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:11.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:11.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:19:11.957 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:11.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:11.983 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:11.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:12.084 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:19:12.108 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:12.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:12.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:19:12.188 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:11 smithi090 ceph-mon[41670]: 4.12 scrub starts 2024-04-14T15:19:12.188 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:11 smithi090 ceph-mon[41670]: 4.12 scrub ok 2024-04-14T15:19:12.188 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:11 smithi090 ceph-mon[41670]: 110.1c scrub starts 2024-04-14T15:19:12.188 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:11 smithi090 ceph-mon[41670]: 110.1c scrub ok 2024-04-14T15:19:12.188 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:11 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:19:12.188 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:11 smithi090 ceph-mon[41670]: osdmap e459: 8 total, 8 up, 8 in 2024-04-14T15:19:12.188 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:11 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-14T15:19:12.188 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:11 smithi090 ceph-mon[41670]: 11.0 scrub starts 2024-04-14T15:19:12.188 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:11 smithi090 ceph-mon[41670]: 11.0 scrub ok 2024-04-14T15:19:12.188 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:11 smithi090 ceph-mon[41670]: 110.4 scrub starts 2024-04-14T15:19:12.188 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:11 smithi090 ceph-mon[41670]: 110.4 scrub ok 2024-04-14T15:19:12.189 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:11 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:19:12.189 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:11 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:19:12.189 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:11 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-69"}]: dispatch 2024-04-14T15:19:12.189 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:11 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-69"}]: dispatch 2024-04-14T15:19:12.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:11 smithi090 ceph-mon[30399]: 4.12 scrub starts 2024-04-14T15:19:12.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:11 smithi090 ceph-mon[30399]: 4.12 scrub ok 2024-04-14T15:19:12.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:11 smithi090 ceph-mon[30399]: 110.1c scrub starts 2024-04-14T15:19:12.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:11 smithi090 ceph-mon[30399]: 110.1c scrub ok 2024-04-14T15:19:12.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:11 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:19:12.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:11 smithi090 ceph-mon[30399]: osdmap e459: 8 total, 8 up, 8 in 2024-04-14T15:19:12.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:11 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-14T15:19:12.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:11 smithi090 ceph-mon[30399]: 11.0 scrub starts 2024-04-14T15:19:12.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:11 smithi090 ceph-mon[30399]: 11.0 scrub ok 2024-04-14T15:19:12.190 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:11 smithi090 ceph-mon[30399]: 110.4 scrub starts 2024-04-14T15:19:12.190 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:11 smithi090 ceph-mon[30399]: 110.4 scrub ok 2024-04-14T15:19:12.190 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:11 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:19:12.190 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:11 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:19:12.190 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:11 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-69"}]: dispatch 2024-04-14T15:19:12.190 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:11 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-69"}]: dispatch 2024-04-14T15:19:12.209 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:19:12.233 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:12.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:12.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:19:12.335 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:19:12.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:12.360 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:12.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:19:12.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:11 smithi092 ceph-mon[33966]: 4.12 scrub starts 2024-04-14T15:19:12.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:11 smithi092 ceph-mon[33966]: 4.12 scrub ok 2024-04-14T15:19:12.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:11 smithi092 ceph-mon[33966]: 110.1c scrub starts 2024-04-14T15:19:12.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:11 smithi092 ceph-mon[33966]: 110.1c scrub ok 2024-04-14T15:19:12.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:11 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:19:12.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:11 smithi092 ceph-mon[33966]: osdmap e459: 8 total, 8 up, 8 in 2024-04-14T15:19:12.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:11 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-14T15:19:12.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:11 smithi092 ceph-mon[33966]: 11.0 scrub starts 2024-04-14T15:19:12.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:11 smithi092 ceph-mon[33966]: 11.0 scrub ok 2024-04-14T15:19:12.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:11 smithi092 ceph-mon[33966]: 110.4 scrub starts 2024-04-14T15:19:12.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:11 smithi092 ceph-mon[33966]: 110.4 scrub ok 2024-04-14T15:19:12.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:11 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:19:12.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:11 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:19:12.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:11 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-69"}]: dispatch 2024-04-14T15:19:12.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:11 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-69"}]: dispatch 2024-04-14T15:19:12.462 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:19:12.485 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:12.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:12.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:19:12.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.549+0000 7f7057298700 1 -- 172.21.15.90:0/3267751886 >> v1:172.21.15.90:6789/0 conn(0x7f7050107760 legacy=0x7f7050107b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:12.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.549+0000 7f7057298700 1 -- 172.21.15.90:0/3267751886 shutdown_connections 2024-04-14T15:19:12.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.549+0000 7f7057298700 1 -- 172.21.15.90:0/3267751886 >> 172.21.15.90:0/3267751886 conn(0x7f70500fd290 msgr2=0x7f70500ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:19:12.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.549+0000 7f7057298700 1 -- 172.21.15.90:0/3267751886 shutdown_connections 2024-04-14T15:19:12.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.549+0000 7f7057298700 1 -- 172.21.15.90:0/3267751886 wait complete. 2024-04-14T15:19:12.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.549+0000 7f7057298700 1 librados: starting msgr at 2024-04-14T15:19:12.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.549+0000 7f7057298700 1 librados: starting objecter 2024-04-14T15:19:12.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.550+0000 7f7057298700 1 Processor -- start 2024-04-14T15:19:12.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.550+0000 7f7057298700 1 -- start start 2024-04-14T15:19:12.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.550+0000 7f7057298700 1 librados: setting wanted keys 2024-04-14T15:19:12.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.550+0000 7f7057298700 1 librados: calling monclient init 2024-04-14T15:19:12.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.550+0000 7f7057298700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f705010ddb0 con 0x7f705010a610 2024-04-14T15:19:12.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.550+0000 7f7057298700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f705010dfc0 con 0x7f705010e1e0 2024-04-14T15:19:12.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.550+0000 7f7057298700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f70501ac240 con 0x7f7050107760 2024-04-14T15:19:12.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.550+0000 7f7055034700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f7050107760 0x7f705006a460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:51744/0 (socket says 172.21.15.90:51744) 2024-04-14T15:19:12.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.550+0000 7f7055034700 1 -- 172.21.15.90:0/3547720726 learned_addr learned my addr 172.21.15.90:0/3547720726 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:19:12.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.550+0000 7f7054833700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f705010a610 0x7f705006ab70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:44172/0 (socket says 172.21.15.90:44172) 2024-04-14T15:19:12.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.550+0000 7f70467fc700 10 client.?.objecter ms_handle_connect 0x7f7050107760 2024-04-14T15:19:12.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.550+0000 7f70467fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:19:12.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.550+0000 7f70467fc700 10 client.?.objecter ms_handle_connect 0x7f705010a610 2024-04-14T15:19:12.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.550+0000 7f70467fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:19:12.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.550+0000 7f70467fc700 10 client.?.objecter ms_handle_connect 0x7f705010e1e0 2024-04-14T15:19:12.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.550+0000 7f70467fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:19:12.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.550+0000 7f70467fc700 1 -- 172.21.15.90:0/3547720726 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4203609936 0 0) 0x7f705010dfc0 con 0x7f705010e1e0 2024-04-14T15:19:12.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.550+0000 7f70467fc700 1 -- 172.21.15.90:0/3547720726 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f703c0035b0 con 0x7f705010e1e0 2024-04-14T15:19:12.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.550+0000 7f70467fc700 1 -- 172.21.15.90:0/3547720726 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2082693335 0 0) 0x7f70501ac240 con 0x7f7050107760 2024-04-14T15:19:12.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.550+0000 7f70467fc700 1 -- 172.21.15.90:0/3547720726 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f705010dfc0 con 0x7f7050107760 2024-04-14T15:19:12.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.550+0000 7f70467fc700 1 -- 172.21.15.90:0/3547720726 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4041207717 0 0) 0x7f705010ddb0 con 0x7f705010a610 2024-04-14T15:19:12.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.550+0000 7f70467fc700 1 -- 172.21.15.90:0/3547720726 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f70501ac240 con 0x7f705010a610 2024-04-14T15:19:12.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.550+0000 7f70467fc700 1 -- 172.21.15.90:0/3547720726 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4258965472 0 0) 0x7f703c0035b0 con 0x7f705010e1e0 2024-04-14T15:19:12.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.550+0000 7f70467fc700 1 -- 172.21.15.90:0/3547720726 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f705010ddb0 con 0x7f705010e1e0 2024-04-14T15:19:12.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.550+0000 7f70467fc700 1 -- 172.21.15.90:0/3547720726 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f7048001bb0 con 0x7f705010e1e0 2024-04-14T15:19:12.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.551+0000 7f70467fc700 1 -- 172.21.15.90:0/3547720726 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1516895380 0 0) 0x7f705010dfc0 con 0x7f7050107760 2024-04-14T15:19:12.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.551+0000 7f70467fc700 1 -- 172.21.15.90:0/3547720726 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f703c0035b0 con 0x7f7050107760 2024-04-14T15:19:12.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.551+0000 7f70467fc700 1 -- 172.21.15.90:0/3547720726 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f70400015d0 con 0x7f7050107760 2024-04-14T15:19:12.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.551+0000 7f70467fc700 1 -- 172.21.15.90:0/3547720726 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3242623045 0 0) 0x7f70501ac240 con 0x7f705010a610 2024-04-14T15:19:12.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.551+0000 7f70467fc700 1 -- 172.21.15.90:0/3547720726 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f705010dfc0 con 0x7f705010a610 2024-04-14T15:19:12.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.551+0000 7f70467fc700 1 -- 172.21.15.90:0/3547720726 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 853808479 0 0) 0x7f705010ddb0 con 0x7f705010e1e0 2024-04-14T15:19:12.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.551+0000 7f70467fc700 1 -- 172.21.15.90:0/3547720726 >> v1:172.21.15.90:6790/0 conn(0x7f7050107760 legacy=0x7f705006a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:12.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.551+0000 7f70467fc700 1 -- 172.21.15.90:0/3547720726 >> v1:172.21.15.90:6789/0 conn(0x7f705010a610 legacy=0x7f705006ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:12.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.551+0000 7f70467fc700 1 -- 172.21.15.90:0/3547720726 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70501ad460 con 0x7f705010e1e0 2024-04-14T15:19:12.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.551+0000 7f7057298700 1 -- 172.21.15.90:0/3547720726 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f70501ac470 con 0x7f705010e1e0 2024-04-14T15:19:12.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.551+0000 7f7057298700 10 client.49204.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:19:12.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.551+0000 7f7057298700 1 -- 172.21.15.90:0/3547720726 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f70501ac930 con 0x7f705010e1e0 2024-04-14T15:19:12.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.551+0000 7f7057298700 1 librados: init done 2024-04-14T15:19:12.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.551+0000 7f70467fc700 1 -- 172.21.15.90:0/3547720726 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f7048003ec0 con 0x7f705010e1e0 2024-04-14T15:19:12.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.551+0000 7f70467fc700 1 -- 172.21.15.90:0/3547720726 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f7048004da0 con 0x7f705010e1e0 2024-04-14T15:19:12.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.552+0000 7f7057298700 1 -- 172.21.15.90:0/3547720726 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7034005150 con 0x7f705010e1e0 2024-04-14T15:19:12.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.552+0000 7f70467fc700 1 -- 172.21.15.90:0/3547720726 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f7048003a20 con 0x7f705010e1e0 2024-04-14T15:19:12.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.553+0000 7f70467fc700 1 -- 172.21.15.90:0/3547720726 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(460..460 src has 51..460) ==== 7775+0+0 (unknown 1077718892 0 0) 0x7f70480898b0 con 0x7f705010e1e0 2024-04-14T15:19:12.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.553+0000 7f70467fc700 10 client.49204.objecter ms_dispatch 0x7f7050001550 osd_map(460..460 src has 51..460) 2024-04-14T15:19:12.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.553+0000 7f70467fc700 3 client.49204.objecter handle_osd_map got epochs [460,460] > 0 2024-04-14T15:19:12.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.553+0000 7f70467fc700 3 client.49204.objecter handle_osd_map decoding full epoch 460 2024-04-14T15:19:12.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.553+0000 7f70467fc700 20 client.49204.objecter dump_active .. 0 homeless 2024-04-14T15:19:12.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.553+0000 7f70467fc700 10 client.49204.objecter ms_handle_connect 0x7f703c071450 2024-04-14T15:19:12.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.557+0000 7f70467fc700 1 -- 172.21.15.90:0/3547720726 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f7048056450 con 0x7f705010e1e0 2024-04-14T15:19:12.587 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:19:12.610 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:12.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:12.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:19:12.712 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:19:12.735 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:12.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:12.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:19:12.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.759+0000 7f7057298700 1 -- 172.21.15.90:0/3547720726 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) -- 0x7f7034005440 con 0x7f705010e1e0 2024-04-14T15:19:12.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.762+0000 7f70467fc700 1 -- 172.21.15.90:0/3547720726 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v460) ==== 81+0+49817 (unknown 587158694 0 4245348038) 0x7f704805a100 con 0x7f705010e1e0 2024-04-14T15:19:12.762 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:19:12.763 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-14T15:19:12.763 INFO:teuthology.orchestra.run.smithi090.stdout: "epoch": 460, 2024-04-14T15:19:12.763 INFO:teuthology.orchestra.run.smithi090.stdout: "fsid": "0e288098-fa70-11ee-bc8e-c7b262605968", 2024-04-14T15:19:12.763 INFO:teuthology.orchestra.run.smithi090.stdout: "created": "2024-04-14T15:04:05.808313+0000", 2024-04-14T15:19:12.763 INFO:teuthology.orchestra.run.smithi090.stdout: "modified": "2024-04-14T15:19:11.977606+0000", 2024-04-14T15:19:12.763 INFO:teuthology.orchestra.run.smithi090.stdout: "last_up_change": "2024-04-14T15:09:29.769352+0000", 2024-04-14T15:19:12.763 INFO:teuthology.orchestra.run.smithi090.stdout: "last_in_change": "2024-04-14T15:09:11.803164+0000", 2024-04-14T15:19:12.763 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-14T15:19:12.763 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_num": 5804032, 2024-04-14T15:19:12.763 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_set": [ 2024-04-14T15:19:12.763 INFO:teuthology.orchestra.run.smithi090.stdout: "nodeep-scrub", 2024-04-14T15:19:12.763 INFO:teuthology.orchestra.run.smithi090.stdout: "pglog_hardlimit", 2024-04-14T15:19:12.763 INFO:teuthology.orchestra.run.smithi090.stdout: "purged_snapdirs", 2024-04-14T15:19:12.763 INFO:teuthology.orchestra.run.smithi090.stdout: "recovery_deletes", 2024-04-14T15:19:12.764 INFO:teuthology.orchestra.run.smithi090.stdout: "sortbitwise" 2024-04-14T15:19:12.764 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-14T15:19:12.764 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_version": 113, 2024-04-14T15:19:12.764 INFO:teuthology.orchestra.run.smithi090.stdout: "full_ratio": 0.94999998807907104, 2024-04-14T15:19:12.764 INFO:teuthology.orchestra.run.smithi090.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-14T15:19:12.764 INFO:teuthology.orchestra.run.smithi090.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-14T15:19:12.764 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_snapshot": "", 2024-04-14T15:19:12.764 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_max": 300, 2024-04-14T15:19:12.764 INFO:teuthology.orchestra.run.smithi090.stdout: "max_osd": 8, 2024-04-14T15:19:12.764 INFO:teuthology.orchestra.run.smithi090.stdout: "require_min_compat_client": "luminous", 2024-04-14T15:19:12.764 INFO:teuthology.orchestra.run.smithi090.stdout: "min_compat_client": "luminous", 2024-04-14T15:19:12.764 INFO:teuthology.orchestra.run.smithi090.stdout: "require_osd_release": "squid", 2024-04-14T15:19:12.764 INFO:teuthology.orchestra.run.smithi090.stdout: "allow_crimson": false, 2024-04-14T15:19:12.765 INFO:teuthology.orchestra.run.smithi090.stdout: "pools": [ 2024-04-14T15:19:12.765 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.765 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 1, 2024-04-14T15:19:12.765 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": ".mgr", 2024-04-14T15:19:12.765 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:07:23.959628+0000", 2024-04-14T15:19:12.765 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-14T15:19:12.765 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool", 2024-04-14T15:19:12.765 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:19:12.765 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:19:12.765 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:19:12.765 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:19:12.765 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:19:12.765 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:19:12.765 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:19:12.766 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:19:12.766 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:19:12.766 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:19:12.766 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 31, 2024-04-14T15:19:12.766 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 31, 2024-04-14T15:19:12.766 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 31, 2024-04-14T15:19:12.766 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 31, 2024-04-14T15:19:12.766 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 31, 2024-04-14T15:19:12.766 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:19:12.766 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:19:12.766 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:19:12.766 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:19:12.766 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:19:12.766 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:19:12.767 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:19:12.767 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.767 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "448", 2024-04-14T15:19:12.767 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:19:12.767 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:19:12.767 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "446", 2024-04-14T15:19:12.767 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:19:12.767 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:19:12.767 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:19:12.767 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:19:12.767 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:19:12.767 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:19:12.768 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:19:12.768 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:19:12.768 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:19:12.768 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:19:12.768 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:19:12.768 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:19:12.768 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:19:12.768 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:19:12.768 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:19:12.768 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:19:12.768 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:19:12.768 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:19:12.768 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:19:12.768 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:19:12.769 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:19:12.769 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:19:12.769 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:19:12.769 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.769 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:19:12.769 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:19:12.769 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:19:12.769 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:19:12.769 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:19:12.769 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:19:12.769 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:19:12.769 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:19:12.769 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:19:12.769 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:19:12.769 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:19:12.770 INFO:teuthology.orchestra.run.smithi090.stdout: "options": { 2024-04-14T15:19:12.770 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_max": 32, 2024-04-14T15:19:12.770 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_min": 1 2024-04-14T15:19:12.770 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.770 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:19:12.770 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": {} 2024-04-14T15:19:12.770 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.770 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:19:12.770 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:19:12.770 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 1.5900000333786011, 2024-04-14T15:19:12.770 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 1.5900000333786011, 2024-04-14T15:19:12.770 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1.1399999856948853, 2024-04-14T15:19:12.770 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 1.809999942779541, 2024-04-14T15:19:12.770 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 1.809999942779541, 2024-04-14T15:19:12.771 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.875, 2024-04-14T15:19:12.771 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 1, 2024-04-14T15:19:12.771 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-04-14T15:19:12.771 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.771 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 2, 2024-04-14T15:19:12.771 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": ".rgw.root", 2024-04-14T15:19:12.771 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:09:41.608420+0000", 2024-04-14T15:19:12.771 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-14T15:19:12.771 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool", 2024-04-14T15:19:12.771 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:19:12.771 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:19:12.771 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:19:12.771 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:19:12.772 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:19:12.772 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:19:12.772 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:19:12.772 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:19:12.772 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:19:12.772 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:19:12.772 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:19:12.772 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:19:12.772 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:19:12.772 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:19:12.772 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:19:12.772 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:19:12.772 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:19:12.772 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:19:12.772 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:19:12.772 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:19:12.773 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:19:12.773 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:19:12.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.773 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "66", 2024-04-14T15:19:12.773 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:19:12.773 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:19:12.773 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:19:12.773 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:19:12.773 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:19:12.773 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:19:12.773 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:19:12.773 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:19:12.773 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:19:12.774 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:19:12.774 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:19:12.774 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:19:12.774 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:19:12.774 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:19:12.774 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:19:12.774 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:19:12.774 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:19:12.774 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:19:12.774 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:19:12.774 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:19:12.774 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:19:12.774 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:19:12.774 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:19:12.774 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:19:12.775 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:19:12.775 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:19:12.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.775 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:19:12.775 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:19:12.775 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:19:12.775 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:19:12.775 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:19:12.775 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:19:12.775 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:19:12.775 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:19:12.775 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:19:12.775 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:19:12.775 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:19:12.775 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:19:12.776 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:19:12.776 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw": {} 2024-04-14T15:19:12.776 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.776 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:19:12.776 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:19:12.776 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 1.5399999618530273, 2024-04-14T15:19:12.776 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 1.5399999618530273, 2024-04-14T15:19:12.776 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1.1399999856948853, 2024-04-14T15:19:12.776 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 1.75, 2024-04-14T15:19:12.776 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 1.75, 2024-04-14T15:19:12.776 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.875, 2024-04-14T15:19:12.776 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 1, 2024-04-14T15:19:12.776 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-04-14T15:19:12.776 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.776 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.776 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.777 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 3, 2024-04-14T15:19:12.777 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "datapool", 2024-04-14T15:19:12.777 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:09:41.744391+0000", 2024-04-14T15:19:12.777 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 8193, 2024-04-14T15:19:12.777 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-14T15:19:12.777 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:19:12.777 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:19:12.777 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:19:12.777 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:19:12.777 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:19:12.777 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:19:12.777 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:19:12.777 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:19:12.777 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:19:12.777 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:19:12.777 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 3, 2024-04-14T15:19:12.777 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 3, 2024-04-14T15:19:12.778 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 3, 2024-04-14T15:19:12.778 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 3, 2024-04-14T15:19:12.778 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 3, 2024-04-14T15:19:12.778 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:19:12.778 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:19:12.778 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:19:12.778 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:19:12.778 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:19:12.778 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:19:12.779 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:19:12.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.779 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "69", 2024-04-14T15:19:12.779 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:19:12.779 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:19:12.779 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:19:12.779 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:19:12.779 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:19:12.779 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 2, 2024-04-14T15:19:12.779 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 69, 2024-04-14T15:19:12.779 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:19:12.779 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:19:12.780 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:19:12.780 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:19:12.780 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:19:12.780 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:19:12.780 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:19:12.780 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:19:12.780 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:19:12.780 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:19:12.780 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:19:12.780 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:19:12.780 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:19:12.780 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:19:12.780 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:19:12.780 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:19:12.781 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:19:12.781 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:19:12.781 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:19:12.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.781 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:19:12.781 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:19:12.781 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:19:12.781 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:19:12.781 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:19:12.781 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:19:12.781 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:19:12.781 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:19:12.781 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:19:12.781 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:19:12.781 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:19:12.782 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:19:12.782 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:19:12.782 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd": {} 2024-04-14T15:19:12.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.782 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:19:12.782 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:19:12.782 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 2.3299999237060547, 2024-04-14T15:19:12.782 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 2.3299999237060547, 2024-04-14T15:19:12.782 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:19:12.782 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 2.3299999237060547, 2024-04-14T15:19:12.782 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 2.3299999237060547, 2024-04-14T15:19:12.782 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.875, 2024-04-14T15:19:12.782 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 1, 2024-04-14T15:19:12.782 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-04-14T15:19:12.782 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.783 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 4, 2024-04-14T15:19:12.783 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "default.rgw.log", 2024-04-14T15:19:12.783 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:09:43.088157+0000", 2024-04-14T15:19:12.783 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-14T15:19:12.783 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool", 2024-04-14T15:19:12.783 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:19:12.783 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:19:12.783 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:19:12.783 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:19:12.783 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:19:12.783 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:19:12.783 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:19:12.783 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:19:12.783 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:19:12.783 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:19:12.784 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:19:12.784 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:19:12.784 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:19:12.784 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:19:12.784 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:19:12.784 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:19:12.784 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:19:12.784 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:19:12.784 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:19:12.784 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:19:12.784 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:19:12.784 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:19:12.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.784 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "68", 2024-04-14T15:19:12.784 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:19:12.784 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:19:12.785 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:19:12.785 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:19:12.785 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:19:12.785 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:19:12.785 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:19:12.785 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:19:12.785 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:19:12.785 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:19:12.785 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:19:12.785 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:19:12.785 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:19:12.785 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:19:12.785 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:19:12.785 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:19:12.785 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:19:12.785 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:19:12.785 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:19:12.786 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:19:12.786 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:19:12.786 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:19:12.786 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:19:12.786 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:19:12.786 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:19:12.786 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:19:12.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.786 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:19:12.786 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:19:12.786 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:19:12.786 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:19:12.786 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:19:12.787 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:19:12.787 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:19:12.787 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:19:12.787 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:19:12.787 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:19:12.787 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:19:12.787 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:19:12.787 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:19:12.787 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw": {} 2024-04-14T15:19:12.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.787 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:19:12.787 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:19:12.787 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 1.9700000286102295, 2024-04-14T15:19:12.787 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 1.9700000286102295, 2024-04-14T15:19:12.787 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1.1399999856948853, 2024-04-14T15:19:12.788 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 2.25, 2024-04-14T15:19:12.788 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 2.25, 2024-04-14T15:19:12.788 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.875, 2024-04-14T15:19:12.788 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 1, 2024-04-14T15:19:12.788 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-04-14T15:19:12.788 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.788 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 5, 2024-04-14T15:19:12.788 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "default.rgw.control", 2024-04-14T15:19:12.788 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:09:45.042722+0000", 2024-04-14T15:19:12.788 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-14T15:19:12.788 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool", 2024-04-14T15:19:12.788 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:19:12.788 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:19:12.788 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:19:12.789 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:19:12.789 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:19:12.789 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:19:12.789 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:19:12.789 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:19:12.789 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:19:12.789 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:19:12.789 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:19:12.789 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:19:12.789 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:19:12.789 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:19:12.789 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:19:12.789 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:19:12.789 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:19:12.789 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:19:12.789 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:19:12.790 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:19:12.790 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:19:12.790 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:19:12.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.790 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "70", 2024-04-14T15:19:12.790 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:19:12.790 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:19:12.790 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:19:12.790 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:19:12.790 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:19:12.790 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:19:12.790 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:19:12.790 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:19:12.790 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:19:12.791 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:19:12.791 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:19:12.791 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:19:12.791 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:19:12.791 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:19:12.791 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:19:12.791 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:19:12.791 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:19:12.791 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:19:12.791 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:19:12.791 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:19:12.791 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:19:12.791 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:19:12.791 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:19:12.791 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:19:12.791 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:19:12.792 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:19:12.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.792 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:19:12.792 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:19:12.792 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:19:12.792 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:19:12.792 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:19:12.792 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:19:12.792 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:19:12.792 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:19:12.792 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:19:12.792 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:19:12.792 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:19:12.792 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:19:12.792 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:19:12.793 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw": {} 2024-04-14T15:19:12.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.793 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:19:12.793 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:19:12.793 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 1.3200000524520874, 2024-04-14T15:19:12.793 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 1.3200000524520874, 2024-04-14T15:19:12.793 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1.1399999856948853, 2024-04-14T15:19:12.793 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 1.5, 2024-04-14T15:19:12.793 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 1.5, 2024-04-14T15:19:12.793 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.875, 2024-04-14T15:19:12.793 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 1, 2024-04-14T15:19:12.793 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-04-14T15:19:12.793 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.793 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 6, 2024-04-14T15:19:12.794 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "default.rgw.meta", 2024-04-14T15:19:12.794 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:09:47.111336+0000", 2024-04-14T15:19:12.794 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-14T15:19:12.794 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool", 2024-04-14T15:19:12.794 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:19:12.794 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:19:12.794 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:19:12.794 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:19:12.794 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:19:12.794 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:19:12.794 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:19:12.794 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:19:12.794 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:19:12.794 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:19:12.794 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:19:12.795 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:19:12.795 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:19:12.795 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:19:12.795 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:19:12.795 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:19:12.795 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:19:12.795 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:19:12.795 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:19:12.795 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:19:12.795 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:19:12.795 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:19:12.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.795 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "72", 2024-04-14T15:19:12.795 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:19:12.795 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:19:12.795 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:19:12.796 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:19:12.796 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:19:12.796 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:19:12.796 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:19:12.796 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:19:12.796 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:19:12.796 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:19:12.796 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:19:12.796 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:19:12.796 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:19:12.796 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:19:12.796 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:19:12.796 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:19:12.796 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:19:12.796 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:19:12.796 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:19:12.796 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:19:12.797 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:19:12.797 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:19:12.797 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:19:12.797 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:19:12.797 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:19:12.797 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:19:12.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.797 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:19:12.797 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:19:12.797 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:19:12.797 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:19:12.797 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:19:12.797 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:19:12.797 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:19:12.797 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:19:12.797 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:19:12.798 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:19:12.798 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:19:12.798 INFO:teuthology.orchestra.run.smithi090.stdout: "options": { 2024-04-14T15:19:12.798 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_bias": 4 2024-04-14T15:19:12.798 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.798 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:19:12.798 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw": {} 2024-04-14T15:19:12.798 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.798 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:19:12.798 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:19:12.798 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 1.75, 2024-04-14T15:19:12.798 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 1.75, 2024-04-14T15:19:12.798 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1.1399999856948853, 2024-04-14T15:19:12.798 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 2, 2024-04-14T15:19:12.798 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 2, 2024-04-14T15:19:12.798 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.875, 2024-04-14T15:19:12.799 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 1, 2024-04-14T15:19:12.799 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-04-14T15:19:12.799 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.799 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 11, 2024-04-14T15:19:12.799 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "ceph_test_rados_api_asio", 2024-04-14T15:19:12.799 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:12:54.225671+0000", 2024-04-14T15:19:12.799 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-14T15:19:12.799 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool", 2024-04-14T15:19:12.799 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:19:12.799 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:19:12.799 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:19:12.799 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:19:12.799 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:19:12.799 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:19:12.800 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:19:12.800 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:19:12.800 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:19:12.800 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:19:12.800 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:19:12.800 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:19:12.800 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:19:12.800 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:19:12.800 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:19:12.800 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:19:12.800 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:19:12.800 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:19:12.800 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:19:12.800 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:19:12.800 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:19:12.800 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:19:12.800 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.801 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "109", 2024-04-14T15:19:12.801 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:19:12.801 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:19:12.801 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:19:12.801 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:19:12.801 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:19:12.801 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:19:12.801 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:19:12.801 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:19:12.801 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:19:12.801 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:19:12.801 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:19:12.801 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:19:12.801 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:19:12.801 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:19:12.801 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:19:12.802 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:19:12.802 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:19:12.802 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:19:12.802 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:19:12.802 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:19:12.802 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:19:12.802 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:19:12.802 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:19:12.802 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:19:12.802 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:19:12.802 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:19:12.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.802 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:19:12.802 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:19:12.802 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:19:12.802 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:19:12.803 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:19:12.803 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:19:12.803 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:19:12.803 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:19:12.803 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:19:12.803 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:19:12.803 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:19:12.803 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:19:12.803 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": {}, 2024-04-14T15:19:12.803 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:19:12.803 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:19:12.803 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 1.75, 2024-04-14T15:19:12.803 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 1.75, 2024-04-14T15:19:12.803 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1.1399999856948853, 2024-04-14T15:19:12.803 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 2, 2024-04-14T15:19:12.803 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 2, 2024-04-14T15:19:12.803 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.875, 2024-04-14T15:19:12.804 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 1, 2024-04-14T15:19:12.804 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-04-14T15:19:12.804 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.804 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 110, 2024-04-14T15:19:12.804 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "test-rados-api-smithi090-93652-6", 2024-04-14T15:19:12.804 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:13:07.960794+0000", 2024-04-14T15:19:12.804 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 8193, 2024-04-14T15:19:12.804 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-14T15:19:12.804 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:19:12.804 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:19:12.804 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:19:12.804 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:19:12.804 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:19:12.804 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:19:12.805 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:19:12.805 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:19:12.805 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:19:12.805 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:19:12.805 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:19:12.805 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:19:12.805 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:19:12.805 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:19:12.805 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:19:12.805 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:19:12.805 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:19:12.805 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:19:12.805 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:19:12.805 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:19:12.805 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:19:12.805 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:19:12.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.806 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "456", 2024-04-14T15:19:12.806 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "399", 2024-04-14T15:19:12.806 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "399", 2024-04-14T15:19:12.806 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "399", 2024-04-14T15:19:12.806 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:19:12.806 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:19:12.806 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 20, 2024-04-14T15:19:12.806 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 456, 2024-04-14T15:19:12.806 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:19:12.806 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:19:12.806 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:19:12.806 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:19:12.806 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:19:12.806 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:19:12.807 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:19:12.807 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:19:12.807 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:19:12.807 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:19:12.807 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:19:12.807 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:19:12.807 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:19:12.807 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:19:12.807 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:19:12.807 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:19:12.807 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:19:12.807 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:19:12.807 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:19:12.807 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.808 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:19:12.808 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:19:12.808 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:19:12.808 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:19:12.808 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:19:12.808 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:19:12.808 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:19:12.808 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:19:12.808 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:19:12.808 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:19:12.808 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:19:12.808 INFO:teuthology.orchestra.run.smithi090.stdout: "options": { 2024-04-14T15:19:12.808 INFO:teuthology.orchestra.run.smithi090.stdout: "fingerprint_algorithm": "sha1" 2024-04-14T15:19:12.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.808 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:19:12.808 INFO:teuthology.orchestra.run.smithi090.stdout: "rados": {} 2024-04-14T15:19:12.809 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.809 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:19:12.809 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:19:12.809 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 1.5399999618530273, 2024-04-14T15:19:12.809 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 1.5399999618530273, 2024-04-14T15:19:12.809 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1.1399999856948853, 2024-04-14T15:19:12.809 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 1.75, 2024-04-14T15:19:12.809 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 1.75, 2024-04-14T15:19:12.809 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.875, 2024-04-14T15:19:12.809 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 1, 2024-04-14T15:19:12.809 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-04-14T15:19:12.809 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.809 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.809 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-14T15:19:12.809 INFO:teuthology.orchestra.run.smithi090.stdout: "osds": [ 2024-04-14T15:19:12.809 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.810 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 0, 2024-04-14T15:19:12.810 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "76772cfc-89d1-4013-b617-32284ed8dfd9", 2024-04-14T15:19:12.810 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-14T15:19:12.810 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-14T15:19:12.810 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-14T15:19:12.810 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-14T15:19:12.810 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-14T15:19:12.810 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-14T15:19:12.810 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 8, 2024-04-14T15:19:12.810 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 458, 2024-04-14T15:19:12.810 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-14T15:19:12.810 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-14T15:19:12.810 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-14T15:19:12.810 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:19:12.810 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.811 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:19:12.811 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6801", 2024-04-14T15:19:12.811 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3761806885 2024-04-14T15:19:12.811 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.811 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:12.811 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.811 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-14T15:19:12.811 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:19:12.811 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.811 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:19:12.811 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6802", 2024-04-14T15:19:12.811 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3761806885 2024-04-14T15:19:12.811 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.811 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:12.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.812 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-14T15:19:12.812 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:19:12.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.812 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:19:12.812 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6804", 2024-04-14T15:19:12.812 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3761806885 2024-04-14T15:19:12.812 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.812 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:12.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.812 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-14T15:19:12.812 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:19:12.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.812 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:19:12.812 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6803", 2024-04-14T15:19:12.812 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3761806885 2024-04-14T15:19:12.813 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.813 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:12.813 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.813 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6801/3761806885", 2024-04-14T15:19:12.813 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6802/3761806885", 2024-04-14T15:19:12.813 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6804/3761806885", 2024-04-14T15:19:12.813 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6803/3761806885", 2024-04-14T15:19:12.813 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-14T15:19:12.813 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-14T15:19:12.813 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-14T15:19:12.813 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:12.813 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.813 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.813 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 1, 2024-04-14T15:19:12.813 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "de4a7fa2-7d57-4ff3-a4fa-28c0cde8ef6c", 2024-04-14T15:19:12.813 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-14T15:19:12.814 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-14T15:19:12.814 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-14T15:19:12.814 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-14T15:19:12.814 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-14T15:19:12.814 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-14T15:19:12.814 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2024-04-14T15:19:12.814 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 458, 2024-04-14T15:19:12.814 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-14T15:19:12.814 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-14T15:19:12.814 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-14T15:19:12.814 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:19:12.814 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.814 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:19:12.814 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6805", 2024-04-14T15:19:12.814 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 4170361699 2024-04-14T15:19:12.814 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.815 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:12.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.815 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-14T15:19:12.815 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:19:12.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.815 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:19:12.815 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6806", 2024-04-14T15:19:12.815 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 4170361699 2024-04-14T15:19:12.815 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.815 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:12.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.815 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-14T15:19:12.815 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:19:12.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.815 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:19:12.815 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6808", 2024-04-14T15:19:12.816 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 4170361699 2024-04-14T15:19:12.816 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.816 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:12.816 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.816 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-14T15:19:12.816 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:19:12.816 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.816 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:19:12.816 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6807", 2024-04-14T15:19:12.816 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 4170361699 2024-04-14T15:19:12.816 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.816 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:12.816 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.816 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6805/4170361699", 2024-04-14T15:19:12.816 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6806/4170361699", 2024-04-14T15:19:12.816 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6808/4170361699", 2024-04-14T15:19:12.817 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6807/4170361699", 2024-04-14T15:19:12.817 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-14T15:19:12.817 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-14T15:19:12.817 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-14T15:19:12.817 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:12.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.817 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 2, 2024-04-14T15:19:12.817 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "9c39b5ff-6175-46ba-9d71-370581c6ff3d", 2024-04-14T15:19:12.817 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-14T15:19:12.817 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-14T15:19:12.817 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-14T15:19:12.817 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-14T15:19:12.817 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-14T15:19:12.818 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-14T15:19:12.818 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 18, 2024-04-14T15:19:12.818 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 458, 2024-04-14T15:19:12.818 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-14T15:19:12.818 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-14T15:19:12.818 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-14T15:19:12.818 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:19:12.818 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.818 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:19:12.818 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6809", 2024-04-14T15:19:12.818 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 1686184251 2024-04-14T15:19:12.818 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.818 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:12.818 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.819 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-14T15:19:12.819 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:19:12.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.819 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:19:12.819 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6810", 2024-04-14T15:19:12.819 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 1686184251 2024-04-14T15:19:12.819 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.819 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:12.819 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.819 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-14T15:19:12.819 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:19:12.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.819 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:19:12.819 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6812", 2024-04-14T15:19:12.819 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 1686184251 2024-04-14T15:19:12.820 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.820 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:12.820 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.820 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-14T15:19:12.820 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:19:12.820 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.820 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:19:12.820 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6811", 2024-04-14T15:19:12.820 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 1686184251 2024-04-14T15:19:12.820 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.820 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:12.820 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.820 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6809/1686184251", 2024-04-14T15:19:12.820 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6810/1686184251", 2024-04-14T15:19:12.820 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6812/1686184251", 2024-04-14T15:19:12.820 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6811/1686184251", 2024-04-14T15:19:12.821 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-14T15:19:12.821 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-14T15:19:12.821 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-14T15:19:12.821 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:12.821 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.821 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.821 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 3, 2024-04-14T15:19:12.821 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "efd2ca45-5b80-4013-a571-499e2cb5b2e8", 2024-04-14T15:19:12.821 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-14T15:19:12.821 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-14T15:19:12.821 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-14T15:19:12.821 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-14T15:19:12.821 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-14T15:19:12.821 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-14T15:19:12.821 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 25, 2024-04-14T15:19:12.821 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 458, 2024-04-14T15:19:12.821 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-14T15:19:12.822 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-14T15:19:12.822 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-14T15:19:12.822 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:19:12.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.822 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:19:12.822 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6813", 2024-04-14T15:19:12.822 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 2270166774 2024-04-14T15:19:12.822 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.822 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:12.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.822 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-14T15:19:12.822 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:19:12.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.822 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:19:12.823 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6814", 2024-04-14T15:19:12.823 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 2270166774 2024-04-14T15:19:12.823 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.823 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:12.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.823 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-14T15:19:12.823 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:19:12.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.823 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:19:12.823 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6816", 2024-04-14T15:19:12.823 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 2270166774 2024-04-14T15:19:12.823 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.823 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:12.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.823 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-14T15:19:12.823 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:19:12.824 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.824 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:19:12.824 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6815", 2024-04-14T15:19:12.824 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 2270166774 2024-04-14T15:19:12.824 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.824 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:12.824 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.764+0000 7f7057298700 10 librados: watch_flush enter 2024-04-14T15:19:12.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.764+0000 7f7057298700 10 librados: watch_flush exit 2024-04-14T15:19:12.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.764+0000 7f7057298700 20 client.49204.objecter shutdown clearing up homeless session... 2024-04-14T15:19:12.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.764+0000 7f7057298700 10 client.49204.objecter successfully canceled tick 2024-04-14T15:19:12.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.764+0000 7f7057298700 1 -- 172.21.15.90:0/3547720726 >> v1:172.21.15.90:6800/420935427 conn(0x7f703c071450 legacy=0x7f703c073910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:12.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.764+0000 7f7057298700 1 -- 172.21.15.90:0/3547720726 >> v1:172.21.15.92:6789/0 conn(0x7f705010e1e0 legacy=0x7f70501a8ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:12.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.764+0000 7f7057298700 1 -- 172.21.15.90:0/3547720726 shutdown_connections 2024-04-14T15:19:12.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.764+0000 7f7057298700 1 -- 172.21.15.90:0/3547720726 >> 172.21.15.90:0/3547720726 conn(0x7f70500fd290 msgr2=0x7f7050111700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:19:12.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.764+0000 7f7057298700 1 -- 172.21.15.90:0/3547720726 shutdown_connections 2024-04-14T15:19:12.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.764+0000 7f7057298700 1 -- 172.21.15.90:0/3547720726 wait complete. 2024-04-14T15:19:12.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:12.764+0000 7f7057298700 1 librados: shutdown 2024-04-14T15:19:12.825 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6813/2270166774", 2024-04-14T15:19:12.825 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6814/2270166774", 2024-04-14T15:19:12.826 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6816/2270166774", 2024-04-14T15:19:12.826 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6815/2270166774", 2024-04-14T15:19:12.826 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-14T15:19:12.826 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-14T15:19:12.826 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-14T15:19:12.826 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:12.826 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.826 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.826 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 4, 2024-04-14T15:19:12.826 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "e3d9013d-5eb6-433c-96b3-3fc38ba7ac8d", 2024-04-14T15:19:12.826 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-14T15:19:12.826 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-14T15:19:12.826 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-14T15:19:12.826 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-14T15:19:12.827 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-14T15:19:12.827 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-14T15:19:12.827 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 31, 2024-04-14T15:19:12.827 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 458, 2024-04-14T15:19:12.827 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-14T15:19:12.827 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-14T15:19:12.827 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-14T15:19:12.827 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:19:12.827 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.827 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:19:12.827 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6800", 2024-04-14T15:19:12.827 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3887549733 2024-04-14T15:19:12.827 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.827 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:12.827 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.828 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-14T15:19:12.828 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:19:12.828 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.828 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:19:12.828 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6801", 2024-04-14T15:19:12.828 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3887549733 2024-04-14T15:19:12.828 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.828 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:12.828 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.828 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-14T15:19:12.828 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:19:12.828 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.828 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:19:12.828 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6803", 2024-04-14T15:19:12.828 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3887549733 2024-04-14T15:19:12.829 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.829 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:12.829 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.829 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-14T15:19:12.829 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:19:12.829 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.829 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:19:12.829 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6802", 2024-04-14T15:19:12.829 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3887549733 2024-04-14T15:19:12.829 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.829 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:12.829 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.829 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.92:6800/3887549733", 2024-04-14T15:19:12.830 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.92:6801/3887549733", 2024-04-14T15:19:12.830 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.92:6803/3887549733", 2024-04-14T15:19:12.830 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.92:6802/3887549733", 2024-04-14T15:19:12.830 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-14T15:19:12.830 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-14T15:19:12.830 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-14T15:19:12.830 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:12.830 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.830 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.830 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 5, 2024-04-14T15:19:12.830 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "1ec41805-6f82-4139-81bc-75305ba9e958", 2024-04-14T15:19:12.830 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-14T15:19:12.830 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-14T15:19:12.830 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-14T15:19:12.831 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 0, 2024-04-14T15:19:12.831 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-14T15:19:12.831 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-14T15:19:12.831 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 41, 2024-04-14T15:19:12.831 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 271, 2024-04-14T15:19:12.831 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-14T15:19:12.831 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-14T15:19:12.831 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-14T15:19:12.831 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:19:12.831 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.831 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:19:12.831 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6804", 2024-04-14T15:19:12.831 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3244561440 2024-04-14T15:19:12.831 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.831 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:12.831 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.831 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-14T15:19:12.832 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:19:12.832 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.832 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:19:12.832 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6805", 2024-04-14T15:19:12.832 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3244561440 2024-04-14T15:19:12.832 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.832 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:12.832 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.832 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-14T15:19:12.832 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:19:12.832 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.832 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:19:12.832 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6807", 2024-04-14T15:19:12.832 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3244561440 2024-04-14T15:19:12.832 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.832 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:12.833 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.833 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-14T15:19:12.833 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:19:12.833 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.833 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:19:12.833 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6806", 2024-04-14T15:19:12.833 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3244561440 2024-04-14T15:19:12.833 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.833 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:12.833 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.833 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.92:6804/3244561440", 2024-04-14T15:19:12.833 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.92:6805/3244561440", 2024-04-14T15:19:12.833 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.92:6807/3244561440", 2024-04-14T15:19:12.833 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.92:6806/3244561440", 2024-04-14T15:19:12.833 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-14T15:19:12.833 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-14T15:19:12.833 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-14T15:19:12.834 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:12.834 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.834 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.834 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 6, 2024-04-14T15:19:12.834 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "ca977ab2-bf8d-4b3a-ae5e-0efa5bc7a3ad", 2024-04-14T15:19:12.834 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-14T15:19:12.834 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-14T15:19:12.834 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-14T15:19:12.834 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-14T15:19:12.834 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-14T15:19:12.834 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-14T15:19:12.834 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 49, 2024-04-14T15:19:12.834 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 458, 2024-04-14T15:19:12.834 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-14T15:19:12.834 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-14T15:19:12.835 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-14T15:19:12.835 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:19:12.835 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.835 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:19:12.835 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6808", 2024-04-14T15:19:12.835 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 636125963 2024-04-14T15:19:12.835 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.835 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:12.835 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.835 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-14T15:19:12.835 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:19:12.835 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.835 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:19:12.835 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6809", 2024-04-14T15:19:12.835 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 636125963 2024-04-14T15:19:12.835 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.836 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:12.836 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.836 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-14T15:19:12.836 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:19:12.836 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.836 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:19:12.836 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6811", 2024-04-14T15:19:12.836 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 636125963 2024-04-14T15:19:12.836 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.836 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:12.836 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.836 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-14T15:19:12.836 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:19:12.836 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.837 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:19:12.837 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6810", 2024-04-14T15:19:12.837 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 636125963 2024-04-14T15:19:12.837 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.837 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:12.837 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.837 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.92:6808/636125963", 2024-04-14T15:19:12.837 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.92:6809/636125963", 2024-04-14T15:19:12.837 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.92:6811/636125963", 2024-04-14T15:19:12.837 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.92:6810/636125963", 2024-04-14T15:19:12.837 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-14T15:19:12.837 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-14T15:19:12.837 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-14T15:19:12.837 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:12.837 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.838 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.838 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 7, 2024-04-14T15:19:12.838 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "e5354645-79ec-4d4c-b6e5-f9b0653915ee", 2024-04-14T15:19:12.838 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-14T15:19:12.838 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-14T15:19:12.838 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-14T15:19:12.838 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-14T15:19:12.838 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-14T15:19:12.838 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-14T15:19:12.838 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 58, 2024-04-14T15:19:12.838 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 458, 2024-04-14T15:19:12.838 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-14T15:19:12.838 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-14T15:19:12.839 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-14T15:19:12.839 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:19:12.839 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.839 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:19:12.839 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6812", 2024-04-14T15:19:12.839 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 1773706716 2024-04-14T15:19:12.839 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.839 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:12.839 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.839 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-14T15:19:12.839 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:19:12.839 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.839 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:19:12.839 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6813", 2024-04-14T15:19:12.839 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 1773706716 2024-04-14T15:19:12.840 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.840 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:12.840 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.840 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-14T15:19:12.840 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:19:12.840 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.840 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:19:12.840 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6815", 2024-04-14T15:19:12.840 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 1773706716 2024-04-14T15:19:12.840 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.840 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:12.840 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.840 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-14T15:19:12.840 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:19:12.841 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.841 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:19:12.841 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6814", 2024-04-14T15:19:12.841 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 1773706716 2024-04-14T15:19:12.841 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.841 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:12.841 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.841 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.92:6812/1773706716", 2024-04-14T15:19:12.841 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.92:6813/1773706716", 2024-04-14T15:19:12.841 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.92:6815/1773706716", 2024-04-14T15:19:12.841 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.92:6814/1773706716", 2024-04-14T15:19:12.841 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-14T15:19:12.841 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-14T15:19:12.841 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-14T15:19:12.841 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:12.841 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.841 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-14T15:19:12.842 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_xinfo": [ 2024-04-14T15:19:12.842 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.842 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 0, 2024-04-14T15:19:12.842 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-14T15:19:12.842 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-14T15:19:12.842 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-14T15:19:12.842 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540701547738038271, 2024-04-14T15:19:12.842 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-14T15:19:12.842 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-14T15:18:04.359927+0000", 2024-04-14T15:19:12.842 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-14T15:19:12.842 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.842 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.842 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 1, 2024-04-14T15:19:12.842 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-14T15:19:12.842 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-14T15:19:12.842 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-14T15:19:12.843 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540701547738038271, 2024-04-14T15:19:12.843 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-14T15:19:12.843 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-14T15:18:13.747612+0000", 2024-04-14T15:19:12.843 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-14T15:19:12.843 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.843 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.843 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 2, 2024-04-14T15:19:12.843 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-14T15:19:12.843 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-14T15:19:12.843 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-14T15:19:12.843 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540701547738038271, 2024-04-14T15:19:12.843 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-14T15:19:12.843 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-14T15:18:16.562315+0000", 2024-04-14T15:19:12.843 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-14T15:19:12.843 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.843 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.844 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 3, 2024-04-14T15:19:12.844 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-14T15:19:12.844 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-14T15:19:12.844 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-14T15:19:12.844 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540701547738038271, 2024-04-14T15:19:12.844 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-14T15:19:12.844 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-14T15:18:02.163024+0000", 2024-04-14T15:19:12.844 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-14T15:19:12.844 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.844 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.844 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 4, 2024-04-14T15:19:12.844 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-14T15:19:12.844 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-14T15:19:12.844 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-14T15:19:12.845 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540701547738038271, 2024-04-14T15:19:12.845 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-14T15:19:12.845 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-14T15:18:07.656348+0000", 2024-04-14T15:19:12.845 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-14T15:19:12.845 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.845 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.845 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 5, 2024-04-14T15:19:12.845 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-14T15:19:12.845 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-14T15:19:12.845 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-14T15:19:12.845 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540701547738038271, 2024-04-14T15:19:12.845 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-14T15:19:12.845 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-14T15:18:42.906398+0000", 2024-04-14T15:19:12.845 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-14T15:19:12.845 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.845 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.846 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 6, 2024-04-14T15:19:12.846 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-14T15:19:12.846 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-14T15:19:12.846 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-14T15:19:12.846 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540701547738038271, 2024-04-14T15:19:12.846 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-14T15:19:12.846 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-14T15:19:09.569245+0000", 2024-04-14T15:19:12.846 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-14T15:19:12.846 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.846 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.846 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 7, 2024-04-14T15:19:12.846 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-14T15:19:12.846 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-14T15:19:12.846 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-14T15:19:12.846 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540701547738038271, 2024-04-14T15:19:12.846 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-14T15:19:12.847 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-14T15:18:30.160705+0000", 2024-04-14T15:19:12.847 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-14T15:19:12.847 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.847 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-14T15:19:12.847 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_upmap": [], 2024-04-14T15:19:12.847 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_upmap_items": [ 2024-04-14T15:19:12.847 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.847 INFO:teuthology.orchestra.run.smithi090.stdout: "pgid": "2.1d", 2024-04-14T15:19:12.847 INFO:teuthology.orchestra.run.smithi090.stdout: "mappings": [ 2024-04-14T15:19:12.847 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.847 INFO:teuthology.orchestra.run.smithi090.stdout: "from": 7, 2024-04-14T15:19:12.847 INFO:teuthology.orchestra.run.smithi090.stdout: "to": 2 2024-04-14T15:19:12.847 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.847 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:12.847 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.847 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.848 INFO:teuthology.orchestra.run.smithi090.stdout: "pgid": "4.2", 2024-04-14T15:19:12.848 INFO:teuthology.orchestra.run.smithi090.stdout: "mappings": [ 2024-04-14T15:19:12.848 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.848 INFO:teuthology.orchestra.run.smithi090.stdout: "from": 1, 2024-04-14T15:19:12.848 INFO:teuthology.orchestra.run.smithi090.stdout: "to": 2 2024-04-14T15:19:12.848 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.848 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:12.848 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.848 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.848 INFO:teuthology.orchestra.run.smithi090.stdout: "pgid": "4.12", 2024-04-14T15:19:12.848 INFO:teuthology.orchestra.run.smithi090.stdout: "mappings": [ 2024-04-14T15:19:12.848 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.848 INFO:teuthology.orchestra.run.smithi090.stdout: "from": 1, 2024-04-14T15:19:12.848 INFO:teuthology.orchestra.run.smithi090.stdout: "to": 5 2024-04-14T15:19:12.848 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.848 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:12.848 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.849 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.849 INFO:teuthology.orchestra.run.smithi090.stdout: "pgid": "4.1f", 2024-04-14T15:19:12.849 INFO:teuthology.orchestra.run.smithi090.stdout: "mappings": [ 2024-04-14T15:19:12.849 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.849 INFO:teuthology.orchestra.run.smithi090.stdout: "from": 1, 2024-04-14T15:19:12.849 INFO:teuthology.orchestra.run.smithi090.stdout: "to": 2 2024-04-14T15:19:12.849 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.849 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:12.849 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.849 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-14T15:19:12.849 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_upmap_primaries": [], 2024-04-14T15:19:12.849 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_temp": [], 2024-04-14T15:19:12.849 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_temp": [], 2024-04-14T15:19:12.849 INFO:teuthology.orchestra.run.smithi090.stdout: "blocklist": { 2024-04-14T15:19:12.849 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:0/508768479": "2024-04-15T15:10:22.918674+0000", 2024-04-14T15:19:12.849 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:0/1823072952": "2024-04-15T15:10:22.918674+0000", 2024-04-14T15:19:12.850 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:6800/491071118": "2024-04-15T15:10:22.918674+0000", 2024-04-14T15:19:12.850 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:0/1337064035": "2024-04-15T15:10:22.918674+0000", 2024-04-14T15:19:12.850 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:0/769462306": "2024-04-15T15:10:22.918674+0000", 2024-04-14T15:19:12.850 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:0/1139868610": "2024-04-15T15:05:05.882361+0000", 2024-04-14T15:19:12.850 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:0/93763408": "2024-04-15T15:10:22.918674+0000", 2024-04-14T15:19:12.850 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:0/977818204": "2024-04-15T15:04:39.251711+0000", 2024-04-14T15:19:12.850 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:6800/941091171": "2024-04-15T15:05:05.882361+0000", 2024-04-14T15:19:12.850 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:0/3671453561": "2024-04-15T15:05:05.882361+0000", 2024-04-14T15:19:12.850 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:0/2611856200": "2024-04-15T15:05:05.882361+0000", 2024-04-14T15:19:12.850 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:6800/2306258962": "2024-04-15T15:04:39.251711+0000", 2024-04-14T15:19:12.850 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:0/4051692380": "2024-04-15T15:04:39.251711+0000", 2024-04-14T15:19:12.850 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:0/2681502561": "2024-04-15T15:04:39.251711+0000" 2024-04-14T15:19:12.850 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.850 INFO:teuthology.orchestra.run.smithi090.stdout: "range_blocklist": {}, 2024-04-14T15:19:12.850 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profiles": { 2024-04-14T15:19:12.850 INFO:teuthology.orchestra.run.smithi090.stdout: "default": { 2024-04-14T15:19:12.851 INFO:teuthology.orchestra.run.smithi090.stdout: "crush-failure-domain": "osd", 2024-04-14T15:19:12.851 INFO:teuthology.orchestra.run.smithi090.stdout: "k": "2", 2024-04-14T15:19:12.851 INFO:teuthology.orchestra.run.smithi090.stdout: "m": "1", 2024-04-14T15:19:12.851 INFO:teuthology.orchestra.run.smithi090.stdout: "plugin": "jerasure", 2024-04-14T15:19:12.851 INFO:teuthology.orchestra.run.smithi090.stdout: "technique": "reed_sol_van" 2024-04-14T15:19:12.851 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.851 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.851 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps_queue": [ 2024-04-14T15:19:12.851 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.851 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 17, 2024-04-14T15:19:12.851 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:19:12.851 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.851 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 2, 2024-04-14T15:19:12.851 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 5 2024-04-14T15:19:12.851 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.851 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:12.852 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.852 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.852 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 18, 2024-04-14T15:19:12.852 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:19:12.852 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.852 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 2, 2024-04-14T15:19:12.852 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 3 2024-04-14T15:19:12.852 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.852 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:12.852 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.852 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.852 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 39, 2024-04-14T15:19:12.852 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:19:12.852 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.852 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 1, 2024-04-14T15:19:12.852 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2024-04-14T15:19:12.853 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.853 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:12.853 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.853 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.853 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 72, 2024-04-14T15:19:12.853 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:19:12.853 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.853 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 1, 2024-04-14T15:19:12.853 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2024-04-14T15:19:12.853 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.853 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:12.853 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.853 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.853 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 99, 2024-04-14T15:19:12.853 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:19:12.853 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.853 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 4, 2024-04-14T15:19:12.854 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 2 2024-04-14T15:19:12.854 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.854 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.854 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 7, 2024-04-14T15:19:12.854 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 2 2024-04-14T15:19:12.854 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.854 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:12.854 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.854 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.854 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 109, 2024-04-14T15:19:12.854 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:19:12.854 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.854 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 1, 2024-04-14T15:19:12.854 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2024-04-14T15:19:12.855 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.855 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:12.855 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.855 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.855 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 128, 2024-04-14T15:19:12.855 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:19:12.855 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.855 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 1, 2024-04-14T15:19:12.855 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 2 2024-04-14T15:19:12.855 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.855 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:12.855 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.855 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.855 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 142, 2024-04-14T15:19:12.855 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:19:12.855 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.856 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 3, 2024-04-14T15:19:12.856 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2024-04-14T15:19:12.856 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.856 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:12.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.856 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 149, 2024-04-14T15:19:12.856 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:19:12.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.856 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 1, 2024-04-14T15:19:12.856 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 4 2024-04-14T15:19:12.856 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.856 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:12.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.856 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 156, 2024-04-14T15:19:12.857 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:19:12.857 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.857 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 2, 2024-04-14T15:19:12.857 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2024-04-14T15:19:12.857 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.857 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:12.857 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.857 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.857 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 157, 2024-04-14T15:19:12.857 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:19:12.857 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.857 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 2, 2024-04-14T15:19:12.857 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 2 2024-04-14T15:19:12.857 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.857 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:12.858 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.858 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.858 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 167, 2024-04-14T15:19:12.858 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:19:12.858 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.858 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 2, 2024-04-14T15:19:12.858 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 3 2024-04-14T15:19:12.858 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.858 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:12.858 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.858 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.858 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 172, 2024-04-14T15:19:12.858 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:19:12.858 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.858 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 3, 2024-04-14T15:19:12.858 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 4 2024-04-14T15:19:12.859 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.859 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:12.859 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.859 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.859 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 175, 2024-04-14T15:19:12.859 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:19:12.859 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.859 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 2, 2024-04-14T15:19:12.859 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 4 2024-04-14T15:19:12.859 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.859 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:12.859 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.859 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.859 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 176, 2024-04-14T15:19:12.859 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:19:12.860 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.860 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 2, 2024-04-14T15:19:12.860 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 2 2024-04-14T15:19:12.860 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.860 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:12.860 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.860 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.860 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 187, 2024-04-14T15:19:12.860 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:19:12.860 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.860 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 2, 2024-04-14T15:19:12.860 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2024-04-14T15:19:12.860 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.860 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:12.861 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.861 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.861 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 192, 2024-04-14T15:19:12.861 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:19:12.861 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.861 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 1, 2024-04-14T15:19:12.861 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 4 2024-04-14T15:19:12.861 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.861 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:12.861 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.861 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.861 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 205, 2024-04-14T15:19:12.861 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:19:12.861 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.861 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 2, 2024-04-14T15:19:12.862 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 6 2024-04-14T15:19:12.862 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.862 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:12.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.862 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.862 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 213, 2024-04-14T15:19:12.862 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:19:12.862 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.862 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 7, 2024-04-14T15:19:12.862 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2024-04-14T15:19:12.862 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.862 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:12.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.862 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.862 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 234, 2024-04-14T15:19:12.862 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:19:12.863 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.863 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 8, 2024-04-14T15:19:12.863 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2024-04-14T15:19:12.863 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.863 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:12.863 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.863 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.863 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 247, 2024-04-14T15:19:12.863 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:19:12.863 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.863 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 10, 2024-04-14T15:19:12.863 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2024-04-14T15:19:12.863 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.863 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:12.863 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.863 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.864 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 252, 2024-04-14T15:19:12.864 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:19:12.864 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.864 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 11, 2024-04-14T15:19:12.864 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 2 2024-04-14T15:19:12.864 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.864 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:12.864 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:19:12.864 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.864 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 265, 2024-04-14T15:19:12.864 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:19:12.864 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:19:12.864 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 14, 2024-04-14T15:19:12.864 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2024-04-14T15:19:12.864 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.865 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:19:12.865 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.865 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-14T15:19:12.865 INFO:teuthology.orchestra.run.smithi090.stdout: "new_removed_snaps": [], 2024-04-14T15:19:12.865 INFO:teuthology.orchestra.run.smithi090.stdout: "new_purged_snaps": [], 2024-04-14T15:19:12.865 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_node_flags": {}, 2024-04-14T15:19:12.865 INFO:teuthology.orchestra.run.smithi090.stdout: "device_class_flags": {}, 2024-04-14T15:19:12.865 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_mode": { 2024-04-14T15:19:12.865 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_mode_enabled": false, 2024-04-14T15:19:12.865 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_bucket_count": 0, 2024-04-14T15:19:12.865 INFO:teuthology.orchestra.run.smithi090.stdout: "degraded_stretch_mode": 0, 2024-04-14T15:19:12.865 INFO:teuthology.orchestra.run.smithi090.stdout: "recovering_stretch_mode": 0, 2024-04-14T15:19:12.865 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_mode_bucket": 0 2024-04-14T15:19:12.865 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:19:12.865 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-14T15:19:12.866 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:19:12.889 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:12.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:12.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:19:12.990 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:19:13.015 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:13.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:13.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:19:13.117 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:13.140 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:13.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:13.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:13.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:12 smithi090 ceph-mon[30399]: pgmap v647: 258 pgs: 32 unknown, 226 active+clean; 8.4 MiB data, 587 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:19:13.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:12 smithi090 ceph-mon[30399]: 2.17 scrub starts 2024-04-14T15:19:13.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:12 smithi090 ceph-mon[30399]: 2.17 scrub ok 2024-04-14T15:19:13.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:12 smithi090 ceph-mon[30399]: 5.a scrub starts 2024-04-14T15:19:13.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:12 smithi090 ceph-mon[30399]: 5.a scrub ok 2024-04-14T15:19:13.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:12 smithi090 ceph-mon[30399]: osdmap e460: 8 total, 8 up, 8 in 2024-04-14T15:19:13.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:12 smithi090 ceph-mon[30399]: 110.19 scrub starts 2024-04-14T15:19:13.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:12 smithi090 ceph-mon[30399]: 110.19 scrub ok 2024-04-14T15:19:13.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:12 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3547720726' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-14T15:19:13.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:12 smithi090 ceph-mon[41670]: pgmap v647: 258 pgs: 32 unknown, 226 active+clean; 8.4 MiB data, 587 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:19:13.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:12 smithi090 ceph-mon[41670]: 2.17 scrub starts 2024-04-14T15:19:13.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:12 smithi090 ceph-mon[41670]: 2.17 scrub ok 2024-04-14T15:19:13.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:12 smithi090 ceph-mon[41670]: 5.a scrub starts 2024-04-14T15:19:13.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:12 smithi090 ceph-mon[41670]: 5.a scrub ok 2024-04-14T15:19:13.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:12 smithi090 ceph-mon[41670]: osdmap e460: 8 total, 8 up, 8 in 2024-04-14T15:19:13.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:12 smithi090 ceph-mon[41670]: 110.19 scrub starts 2024-04-14T15:19:13.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:12 smithi090 ceph-mon[41670]: 110.19 scrub ok 2024-04-14T15:19:13.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:12 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3547720726' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-14T15:19:13.242 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:19:13.266 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:13.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:13.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:19:13.367 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:19:13.371 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:12 smithi092 ceph-mon[33966]: pgmap v647: 258 pgs: 32 unknown, 226 active+clean; 8.4 MiB data, 587 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:19:13.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:12 smithi092 ceph-mon[33966]: 2.17 scrub starts 2024-04-14T15:19:13.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:12 smithi092 ceph-mon[33966]: 2.17 scrub ok 2024-04-14T15:19:13.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:12 smithi092 ceph-mon[33966]: 5.a scrub starts 2024-04-14T15:19:13.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:12 smithi092 ceph-mon[33966]: 5.a scrub ok 2024-04-14T15:19:13.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:12 smithi092 ceph-mon[33966]: osdmap e460: 8 total, 8 up, 8 in 2024-04-14T15:19:13.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:12 smithi092 ceph-mon[33966]: 110.19 scrub starts 2024-04-14T15:19:13.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:12 smithi092 ceph-mon[33966]: 110.19 scrub ok 2024-04-14T15:19:13.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:12 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3547720726' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-14T15:19:13.391 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:13.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:13.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:19:13.492 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:19:13.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:13.516 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:13.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:19:13.617 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:19:13.642 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:13.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:13.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:19:13.747 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:19:13.771 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:13.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:13.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:19:13.873 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:19:13.899 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:13.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:13.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:19:14.001 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:19:14.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:14.009+0000 7f7906685700 1 -- 172.21.15.90:0/3467926305 >> v1:172.21.15.90:6790/0 conn(0x7f790010efa0 legacy=0x7f790010f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:14.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:14.012+0000 7f7906685700 1 -- 172.21.15.90:0/3467926305 shutdown_connections 2024-04-14T15:19:14.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:14.012+0000 7f7906685700 1 -- 172.21.15.90:0/3467926305 >> 172.21.15.90:0/3467926305 conn(0x7f790006ddf0 msgr2=0x7f790006e200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:19:14.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:14.012+0000 7f7906685700 1 -- 172.21.15.90:0/3467926305 shutdown_connections 2024-04-14T15:19:14.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:14.014+0000 7f7906685700 1 -- 172.21.15.90:0/3467926305 wait complete. 2024-04-14T15:19:14.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:14.015+0000 7f7906685700 1 librados: starting msgr at 2024-04-14T15:19:14.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:14.015+0000 7f7906685700 1 librados: starting objecter 2024-04-14T15:19:14.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:14.015+0000 7f7906685700 1 Processor -- start 2024-04-14T15:19:14.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:14.016+0000 7f7906685700 1 -- start start 2024-04-14T15:19:14.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:14.016+0000 7f7906685700 1 librados: setting wanted keys 2024-04-14T15:19:14.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:14.016+0000 7f7906685700 1 librados: calling monclient init 2024-04-14T15:19:14.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:14.017+0000 7f7906685700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f79001243d0 con 0x7f7900112e20 2024-04-14T15:19:14.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:14.017+0000 7f7906685700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f79001b7af0 con 0x7f790010efa0 2024-04-14T15:19:14.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:14.017+0000 7f7906685700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f79001b8cd0 con 0x7f7900116a20 2024-04-14T15:19:14.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:14.018+0000 7f78fffff700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f790010efa0 0x7f790010e420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.90:48324/0 (socket says 172.21.15.90:48324) 2024-04-14T15:19:14.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:14.018+0000 7f78fffff700 1 -- 172.21.15.90:0/1511992209 learned_addr learned my addr 172.21.15.90:0/1511992209 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:19:14.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:14.018+0000 7f78fd7fa700 10 client.?.objecter ms_handle_connect 0x7f790010efa0 2024-04-14T15:19:14.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:14.018+0000 7f78fd7fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:19:14.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:14.019+0000 7f78fd7fa700 1 -- 172.21.15.90:0/1511992209 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 710599483 0 0) 0x7f79001b7af0 con 0x7f790010efa0 2024-04-14T15:19:14.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:14.019+0000 7f78fd7fa700 1 -- 172.21.15.90:0/1511992209 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78e80031f0 con 0x7f790010efa0 2024-04-14T15:19:14.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:14.020+0000 7f78fd7fa700 1 -- 172.21.15.90:0/1511992209 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 463810350 0 0) 0x7f78e80031f0 con 0x7f790010efa0 2024-04-14T15:19:14.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:14.020+0000 7f78fd7fa700 1 -- 172.21.15.90:0/1511992209 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f79001b7af0 con 0x7f790010efa0 2024-04-14T15:19:14.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:14.020+0000 7f78fd7fa700 10 client.?.objecter ms_handle_connect 0x7f7900112e20 2024-04-14T15:19:14.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:14.020+0000 7f78fd7fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:19:14.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:14.020+0000 7f78fd7fa700 10 client.?.objecter ms_handle_connect 0x7f7900116a20 2024-04-14T15:19:14.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:14.020+0000 7f78fd7fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:19:14.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:14.020+0000 7f78fd7fa700 1 -- 172.21.15.90:0/1511992209 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f78f00018d0 con 0x7f790010efa0 2024-04-14T15:19:14.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:14.020+0000 7f78fd7fa700 1 -- 172.21.15.90:0/1511992209 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3258271129 0 0) 0x7f79001b8cd0 con 0x7f7900116a20 2024-04-14T15:19:14.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:14.020+0000 7f78fd7fa700 1 -- 172.21.15.90:0/1511992209 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78e80031f0 con 0x7f7900116a20 2024-04-14T15:19:14.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:14.021+0000 7f78fd7fa700 1 -- 172.21.15.90:0/1511992209 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2957105534 0 0) 0x7f79001b7af0 con 0x7f790010efa0 2024-04-14T15:19:14.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:14.021+0000 7f78fd7fa700 1 -- 172.21.15.90:0/1511992209 >> v1:172.21.15.90:6790/0 conn(0x7f7900116a20 legacy=0x7f79001b53c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:14.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:14.022+0000 7f78fd7fa700 1 -- 172.21.15.90:0/1511992209 >> v1:172.21.15.90:6789/0 conn(0x7f7900112e20 legacy=0x7f79001b1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:14.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:14.022+0000 7f78fd7fa700 1 -- 172.21.15.90:0/1511992209 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79001b9eb0 con 0x7f790010efa0 2024-04-14T15:19:14.022 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:13 smithi090 ceph-mon[30399]: 110.6 scrub starts 2024-04-14T15:19:14.023 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:13 smithi090 ceph-mon[30399]: 110.6 scrub ok 2024-04-14T15:19:14.023 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:13 smithi090 ceph-mon[30399]: 11.14 scrub starts 2024-04-14T15:19:14.023 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:13 smithi090 ceph-mon[30399]: 11.14 scrub ok 2024-04-14T15:19:14.023 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:13 smithi090 ceph-mon[30399]: osdmap e461: 8 total, 8 up, 8 in 2024-04-14T15:19:14.023 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:13 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:19:14.023 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:13 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:19:14.023 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:13 smithi090 ceph-mon[30399]: pgmap v650: 258 pgs: 32 unknown, 226 active+clean; 8.4 MiB data, 587 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:19:14.023 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:13 smithi090 ceph-mon[30399]: 11.10 scrub starts 2024-04-14T15:19:14.023 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:13 smithi090 ceph-mon[30399]: 11.10 scrub ok 2024-04-14T15:19:14.023 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:19:13 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:19:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:19:14.024 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:13 smithi090 ceph-mon[41670]: 110.6 scrub starts 2024-04-14T15:19:14.024 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:13 smithi090 ceph-mon[41670]: 110.6 scrub ok 2024-04-14T15:19:14.024 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:13 smithi090 ceph-mon[41670]: 11.14 scrub starts 2024-04-14T15:19:14.024 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:13 smithi090 ceph-mon[41670]: 11.14 scrub ok 2024-04-14T15:19:14.024 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:13 smithi090 ceph-mon[41670]: osdmap e461: 8 total, 8 up, 8 in 2024-04-14T15:19:14.024 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:13 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:19:14.024 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:13 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:19:14.024 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:13 smithi090 ceph-mon[41670]: pgmap v650: 258 pgs: 32 unknown, 226 active+clean; 8.4 MiB data, 587 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:19:14.024 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:13 smithi090 ceph-mon[41670]: 11.10 scrub starts 2024-04-14T15:19:14.024 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:13 smithi090 ceph-mon[41670]: 11.10 scrub ok 2024-04-14T15:19:14.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:14.022+0000 7f7906685700 1 -- 172.21.15.90:0/1511992209 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f79001b6b40 con 0x7f790010efa0 2024-04-14T15:19:14.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:14.022+0000 7f7906685700 10 client.49210.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:19:14.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:14.022+0000 7f7906685700 1 -- 172.21.15.90:0/1511992209 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f79001b7080 con 0x7f790010efa0 2024-04-14T15:19:14.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:14.022+0000 7f7906685700 1 librados: init done 2024-04-14T15:19:14.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:14.024+0000 7f7906685700 1 -- 172.21.15.90:0/1511992209 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7900074690 con 0x7f790010efa0 2024-04-14T15:19:14.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:14.025+0000 7f78fd7fa700 1 -- 172.21.15.90:0/1511992209 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f78f0004800 con 0x7f790010efa0 2024-04-14T15:19:14.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:14.025+0000 7f78fd7fa700 1 -- 172.21.15.90:0/1511992209 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f78f0004ce0 con 0x7f790010efa0 2024-04-14T15:19:14.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:14.026+0000 7f78fd7fa700 1 -- 172.21.15.90:0/1511992209 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f78f0005010 con 0x7f790010efa0 2024-04-14T15:19:14.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:14.026+0000 7f78fd7fa700 1 -- 172.21.15.90:0/1511992209 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(462..462 src has 51..462) ==== 8160+0+0 (unknown 1970488486 0 0) 0x7f78f008baa0 con 0x7f790010efa0 2024-04-14T15:19:14.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:14.026+0000 7f78fd7fa700 10 client.49210.objecter ms_dispatch 0x7f7900001550 osd_map(462..462 src has 51..462) 2024-04-14T15:19:14.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:14.026+0000 7f78fd7fa700 3 client.49210.objecter handle_osd_map got epochs [462,462] > 0 2024-04-14T15:19:14.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:14.026+0000 7f78fd7fa700 3 client.49210.objecter handle_osd_map decoding full epoch 462 2024-04-14T15:19:14.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:14.027+0000 7f78fd7fa700 20 client.49210.objecter dump_active .. 0 homeless 2024-04-14T15:19:14.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:14.027+0000 7f78fd7fa700 10 client.49210.objecter ms_handle_connect 0x7f78e8071110 2024-04-14T15:19:14.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:14.030+0000 7f78fd7fa700 1 -- 172.21.15.90:0/1511992209 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f78f00584c0 con 0x7f790010efa0 2024-04-14T15:19:14.041 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:14.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:14.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:19:14.143 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:19:14.166 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:14.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:14.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:19:14.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:14.249+0000 7f7906685700 1 -- 172.21.15.90:0/1511992209 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7f790011c7a0 con 0x7f790010efa0 2024-04-14T15:19:14.268 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:19:14.293 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:14.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:14.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:19:14.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:13 smithi092 ceph-mon[33966]: 110.6 scrub starts 2024-04-14T15:19:14.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:13 smithi092 ceph-mon[33966]: 110.6 scrub ok 2024-04-14T15:19:14.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:13 smithi092 ceph-mon[33966]: 11.14 scrub starts 2024-04-14T15:19:14.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:13 smithi092 ceph-mon[33966]: 11.14 scrub ok 2024-04-14T15:19:14.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:13 smithi092 ceph-mon[33966]: osdmap e461: 8 total, 8 up, 8 in 2024-04-14T15:19:14.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:13 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:19:14.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:13 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:19:14.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:13 smithi092 ceph-mon[33966]: pgmap v650: 258 pgs: 32 unknown, 226 active+clean; 8.4 MiB data, 587 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:19:14.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:13 smithi092 ceph-mon[33966]: 11.10 scrub starts 2024-04-14T15:19:14.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:13 smithi092 ceph-mon[33966]: 11.10 scrub ok 2024-04-14T15:19:14.396 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:19:14.420 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:14.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:14.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:19:14.521 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:19:14.547 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:14.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:14.548 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:19:14.647 INFO:tasks.thrashosds.thrasher:j is {'epoch': 460, 'fsid': '0e288098-fa70-11ee-bc8e-c7b262605968', 'created': '2024-04-14T15:04:05.808313+0000', 'modified': '2024-04-14T15:19:11.977606+0000', 'last_up_change': '2024-04-14T15:09:29.769352+0000', 'last_in_change': '2024-04-14T15:09:11.803164+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 113, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 300, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-14T15:07:23.959628+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '448', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '446', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.590000033378601, 'score_stable': 1.590000033378601, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 1.809999942779541, 'raw_score_stable': 1.809999942779541, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-04-14T15:09:41.608420+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5399999618530273, 'score_stable': 1.5399999618530273, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 3, 'pool_name': 'datapool', 'create_time': '2024-04-14T15:09:41.744391+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 69, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.3299999237060547, 'score_stable': 2.3299999237060547, 'optimal_score': 1, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-04-14T15:09:43.088157+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9700000286102295, 'score_stable': 1.9700000286102295, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-04-14T15:09:45.042722+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3200000524520874, 'score_stable': 1.3200000524520874, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-04-14T15:09:47.111336+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 11, 'pool_name': 'ceph_test_rados_api_asio', 'create_time': '2024-04-14T15:12:54.225671+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 110, 'pool_name': 'test-rados-api-smithi090-93652-6', 'create_time': '2024-04-14T15:13:07.960794+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '456', 'last_force_op_resend': '399', 'last_force_op_resend_prenautilus': '399', 'last_force_op_resend_preluminous': '399', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 20, 'snap_epoch': 456, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'fingerprint_algorithm': 'sha1'}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5399999618530273, 'score_stable': 1.5399999618530273, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}], 'osds': [{'osd': 0, 'uuid': '76772cfc-89d1-4013-b617-32284ed8dfd9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 458, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6801', 'nonce': 3761806885}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6802', 'nonce': 3761806885}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6804', 'nonce': 3761806885}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6803', 'nonce': 3761806885}]}, 'public_addr': '172.21.15.90:6801/3761806885', 'cluster_addr': '172.21.15.90:6802/3761806885', 'heartbeat_back_addr': '172.21.15.90:6804/3761806885', 'heartbeat_front_addr': '172.21.15.90:6803/3761806885', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'de4a7fa2-7d57-4ff3-a4fa-28c0cde8ef6c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 458, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6805', 'nonce': 4170361699}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6806', 'nonce': 4170361699}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6808', 'nonce': 4170361699}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6807', 'nonce': 4170361699}]}, 'public_addr': '172.21.15.90:6805/4170361699', 'cluster_addr': '172.21.15.90:6806/4170361699', 'heartbeat_back_addr': '172.21.15.90:6808/4170361699', 'heartbeat_front_addr': '172.21.15.90:6807/4170361699', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '9c39b5ff-6175-46ba-9d71-370581c6ff3d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 458, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6809', 'nonce': 1686184251}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6810', 'nonce': 1686184251}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6812', 'nonce': 1686184251}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6811', 'nonce': 1686184251}]}, 'public_addr': '172.21.15.90:6809/1686184251', 'cluster_addr': '172.21.15.90:6810/1686184251', 'heartbeat_back_addr': '172.21.15.90:6812/1686184251', 'heartbeat_front_addr': '172.21.15.90:6811/1686184251', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'efd2ca45-5b80-4013-a571-499e2cb5b2e8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 458, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6813', 'nonce': 2270166774}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6814', 'nonce': 2270166774}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6816', 'nonce': 2270166774}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6815', 'nonce': 2270166774}]}, 'public_addr': '172.21.15.90:6813/2270166774', 'cluster_addr': '172.21.15.90:6814/2270166774', 'heartbeat_back_addr': '172.21.15.90:6816/2270166774', 'heartbeat_front_addr': '172.21.15.90:6815/2270166774', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'e3d9013d-5eb6-433c-96b3-3fc38ba7ac8d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 458, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6800', 'nonce': 3887549733}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6801', 'nonce': 3887549733}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6803', 'nonce': 3887549733}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6802', 'nonce': 3887549733}]}, 'public_addr': '172.21.15.92:6800/3887549733', 'cluster_addr': '172.21.15.92:6801/3887549733', 'heartbeat_back_addr': '172.21.15.92:6803/3887549733', 'heartbeat_front_addr': '172.21.15.92:6802/3887549733', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '1ec41805-6f82-4139-81bc-75305ba9e958', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 41, 'up_thru': 271, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6804', 'nonce': 3244561440}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6805', 'nonce': 3244561440}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6807', 'nonce': 3244561440}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6806', 'nonce': 3244561440}]}, 'public_addr': '172.21.15.92:6804/3244561440', 'cluster_addr': '172.21.15.92:6805/3244561440', 'heartbeat_back_addr': '172.21.15.92:6807/3244561440', 'heartbeat_front_addr': '172.21.15.92:6806/3244561440', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'ca977ab2-bf8d-4b3a-ae5e-0efa5bc7a3ad', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 49, 'up_thru': 458, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6808', 'nonce': 636125963}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6809', 'nonce': 636125963}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6811', 'nonce': 636125963}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6810', 'nonce': 636125963}]}, 'public_addr': '172.21.15.92:6808/636125963', 'cluster_addr': '172.21.15.92:6809/636125963', 'heartbeat_back_addr': '172.21.15.92:6811/636125963', 'heartbeat_front_addr': '172.21.15.92:6810/636125963', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'e5354645-79ec-4d4c-b6e5-f9b0653915ee', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 58, 'up_thru': 458, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6812', 'nonce': 1773706716}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6813', 'nonce': 1773706716}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6815', 'nonce': 1773706716}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6814', 'nonce': 1773706716}]}, 'public_addr': '172.21.15.92:6812/1773706716', 'cluster_addr': '172.21.15.92:6813/1773706716', 'heartbeat_back_addr': '172.21.15.92:6815/1773706716', 'heartbeat_front_addr': '172.21.15.92:6814/1773706716', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-14T15:18:04.359927+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-14T15:18:13.747612+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-14T15:18:16.562315+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-14T15:18:02.163024+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-14T15:18:07.656348+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-14T15:18:42.906398+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-14T15:19:09.569245+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-14T15:18:30.160705+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '2.1d', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '4.2', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.12', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.1f', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.90:0/508768479': '2024-04-15T15:10:22.918674+0000', '172.21.15.90:0/1823072952': '2024-04-15T15:10:22.918674+0000', '172.21.15.90:6800/491071118': '2024-04-15T15:10:22.918674+0000', '172.21.15.90:0/1337064035': '2024-04-15T15:10:22.918674+0000', '172.21.15.90:0/769462306': '2024-04-15T15:10:22.918674+0000', '172.21.15.90:0/1139868610': '2024-04-15T15:05:05.882361+0000', '172.21.15.90:0/93763408': '2024-04-15T15:10:22.918674+0000', '172.21.15.90:0/977818204': '2024-04-15T15:04:39.251711+0000', '172.21.15.90:6800/941091171': '2024-04-15T15:05:05.882361+0000', '172.21.15.90:0/3671453561': '2024-04-15T15:05:05.882361+0000', '172.21.15.90:0/2611856200': '2024-04-15T15:05:05.882361+0000', '172.21.15.90:6800/2306258962': '2024-04-15T15:04:39.251711+0000', '172.21.15.90:0/4051692380': '2024-04-15T15:04:39.251711+0000', '172.21.15.90:0/2681502561': '2024-04-15T15:04:39.251711+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 17, 'snaps': [{'begin': 2, 'length': 5}]}, {'pool': 18, 'snaps': [{'begin': 2, 'length': 3}]}, {'pool': 39, 'snaps': [{'begin': 1, 'length': 1}]}, {'pool': 72, 'snaps': [{'begin': 1, 'length': 1}]}, {'pool': 99, 'snaps': [{'begin': 4, 'length': 2}, {'begin': 7, 'length': 2}]}, {'pool': 109, 'snaps': [{'begin': 1, 'length': 1}]}, {'pool': 128, 'snaps': [{'begin': 1, 'length': 2}]}, {'pool': 142, 'snaps': [{'begin': 3, 'length': 1}]}, {'pool': 149, 'snaps': [{'begin': 1, 'length': 4}]}, {'pool': 156, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 157, 'snaps': [{'begin': 2, 'length': 2}]}, {'pool': 167, 'snaps': [{'begin': 2, 'length': 3}]}, {'pool': 172, 'snaps': [{'begin': 3, 'length': 4}]}, {'pool': 175, 'snaps': [{'begin': 2, 'length': 4}]}, {'pool': 176, 'snaps': [{'begin': 2, 'length': 2}]}, {'pool': 187, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 192, 'snaps': [{'begin': 1, 'length': 4}]}, {'pool': 205, 'snaps': [{'begin': 2, 'length': 6}]}, {'pool': 213, 'snaps': [{'begin': 7, 'length': 1}]}, {'pool': 234, 'snaps': [{'begin': 8, 'length': 1}]}, {'pool': 247, 'snaps': [{'begin': 10, 'length': 1}]}, {'pool': 252, 'snaps': [{'begin': 11, 'length': 2}]}, {'pool': 265, 'snaps': [{'begin': 14, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-14T15:19:14.648 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph pg dump --format=json 2024-04-14T15:19:14.648 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:19:14.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:14.674 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:14.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:19:14.775 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:19:14.797 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:14.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:14.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:19:14.899 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:19:14.924 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:14.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:14.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:19:15.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:15.007+0000 7f78fd7fa700 1 -- 172.21.15.90:0/1511992209 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v463) ==== 101+0+0 (unknown 3048674231 0 0) 0x7f78f005c170 con 0x7f790010efa0 2024-04-14T15:19:15.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:15.010+0000 7f7906685700 10 librados: watch_flush enter 2024-04-14T15:19:15.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:15.010+0000 7f7906685700 10 librados: watch_flush exit 2024-04-14T15:19:15.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:15.010+0000 7f7906685700 20 client.49210.objecter shutdown clearing up homeless session... 2024-04-14T15:19:15.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:15.010+0000 7f7906685700 10 client.49210.objecter successfully canceled tick 2024-04-14T15:19:15.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:15.010+0000 7f7906685700 1 -- 172.21.15.90:0/1511992209 >> v1:172.21.15.90:6800/420935427 conn(0x7f78e8071110 legacy=0x7f78e80735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:15.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:15.010+0000 7f7906685700 1 -- 172.21.15.90:0/1511992209 >> v1:172.21.15.92:6789/0 conn(0x7f790010efa0 legacy=0x7f790010e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:15.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:15.010+0000 7f7906685700 1 -- 172.21.15.90:0/1511992209 shutdown_connections 2024-04-14T15:19:15.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:15.010+0000 7f7906685700 1 -- 172.21.15.90:0/1511992209 >> 172.21.15.90:0/1511992209 conn(0x7f790006ddf0 msgr2=0x7f7900118ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:19:15.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:15.010+0000 7f7906685700 1 -- 172.21.15.90:0/1511992209 shutdown_connections 2024-04-14T15:19:15.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:15.010+0000 7f7906685700 1 -- 172.21.15.90:0/1511992209 wait complete. 2024-04-14T15:19:15.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:15.010+0000 7f7906685700 1 librados: shutdown 2024-04-14T15:19:15.012 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is unset 2024-04-14T15:19:15.026 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:19:15.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:15.054 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:15.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:19:15.155 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:15.157 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:14 smithi090 ceph-mon[41670]: 110.d scrub starts 2024-04-14T15:19:15.157 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:14 smithi090 ceph-mon[41670]: 110.d scrub ok 2024-04-14T15:19:15.157 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:14 smithi090 ceph-mon[41670]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:19:15.157 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:14 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:19:15.158 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:14 smithi090 ceph-mon[41670]: osdmap e462: 8 total, 8 up, 8 in 2024-04-14T15:19:15.158 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:14 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-14T15:19:15.158 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:14 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-14T15:19:15.158 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:14 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-14T15:19:15.158 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:14 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:19:15.158 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:14 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:19:15.158 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:14 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1511992209' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:19:15.158 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:14 smithi090 ceph-mon[41670]: from='client.49210 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:19:15.158 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:14 smithi090 ceph-mon[30399]: 110.d scrub starts 2024-04-14T15:19:15.158 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:14 smithi090 ceph-mon[30399]: 110.d scrub ok 2024-04-14T15:19:15.158 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:14 smithi090 ceph-mon[30399]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:19:15.158 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:14 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:19:15.159 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:14 smithi090 ceph-mon[30399]: osdmap e462: 8 total, 8 up, 8 in 2024-04-14T15:19:15.159 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:14 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-14T15:19:15.159 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:14 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-14T15:19:15.159 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:14 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-14T15:19:15.159 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:14 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:19:15.159 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:14 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:19:15.159 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:14 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1511992209' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:19:15.159 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:14 smithi090 ceph-mon[30399]: from='client.49210 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:19:15.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:15.183 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:15.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:15.284 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:15.311 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:15.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:15.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:15.359 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-14T15:19:15.360 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:19:15.372 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:19:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:19:15.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:14 smithi092 ceph-mon[33966]: 110.d scrub starts 2024-04-14T15:19:15.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:14 smithi092 ceph-mon[33966]: 110.d scrub ok 2024-04-14T15:19:15.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:14 smithi092 ceph-mon[33966]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:19:15.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:14 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:19:15.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:14 smithi092 ceph-mon[33966]: osdmap e462: 8 total, 8 up, 8 in 2024-04-14T15:19:15.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:14 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-14T15:19:15.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:14 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-14T15:19:15.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:14 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-14T15:19:15.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:14 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:19:15.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:14 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:19:15.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:14 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1511992209' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:19:15.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:14 smithi092 ceph-mon[33966]: from='client.49210 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:19:15.413 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:19:15.436 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:15.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:15.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:19:15.538 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:19:15.566 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:15.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:15.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:19:15.582 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:19:15.584 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:19:15.668 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:19:15.694 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:15.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:15.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:19:15.796 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:19:15.820 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:15.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:15.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:19:15.921 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:19:15.951 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:15.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:15.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:19:16.053 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:19:16.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:16.079 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:16.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:19:16.179 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:19:16.204 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:16.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:16.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:19:16.306 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:19:16.329 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:16.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:16.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:19:16.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:15 smithi092 ceph-mon[33966]: 110.a scrub starts 2024-04-14T15:19:16.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:15 smithi092 ceph-mon[33966]: 110.a scrub ok 2024-04-14T15:19:16.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:15 smithi092 ceph-mon[33966]: 2.5 scrub starts 2024-04-14T15:19:16.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:15 smithi092 ceph-mon[33966]: 2.5 scrub ok 2024-04-14T15:19:16.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:15 smithi092 ceph-mon[33966]: 6.2 scrub starts 2024-04-14T15:19:16.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:15 smithi092 ceph-mon[33966]: 6.2 scrub ok 2024-04-14T15:19:16.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:15 smithi092 ceph-mon[33966]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-14T15:19:16.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:15 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-14T15:19:16.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:15 smithi092 ceph-mon[33966]: from='client.49210 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-14T15:19:16.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:15 smithi092 ceph-mon[33966]: osdmap e463: 8 total, 8 up, 8 in 2024-04-14T15:19:16.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:15 smithi092 ceph-mon[33966]: pgmap v653: 258 pgs: 5 creating+peering, 17 creating+activating, 236 active+clean; 8.4 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-04-14T15:19:16.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:15 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:19:16.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:15 smithi092 ceph-mon[33966]: osdmap e464: 8 total, 8 up, 8 in 2024-04-14T15:19:16.431 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:19:16.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:15 smithi090 ceph-mon[41670]: 110.a scrub starts 2024-04-14T15:19:16.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:15 smithi090 ceph-mon[41670]: 110.a scrub ok 2024-04-14T15:19:16.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:15 smithi090 ceph-mon[41670]: 2.5 scrub starts 2024-04-14T15:19:16.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:15 smithi090 ceph-mon[41670]: 2.5 scrub ok 2024-04-14T15:19:16.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:15 smithi090 ceph-mon[41670]: 6.2 scrub starts 2024-04-14T15:19:16.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:15 smithi090 ceph-mon[41670]: 6.2 scrub ok 2024-04-14T15:19:16.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:15 smithi090 ceph-mon[41670]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-14T15:19:16.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:15 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-14T15:19:16.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:15 smithi090 ceph-mon[41670]: from='client.49210 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-14T15:19:16.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:15 smithi090 ceph-mon[41670]: osdmap e463: 8 total, 8 up, 8 in 2024-04-14T15:19:16.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:15 smithi090 ceph-mon[41670]: pgmap v653: 258 pgs: 5 creating+peering, 17 creating+activating, 236 active+clean; 8.4 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-04-14T15:19:16.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:15 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:19:16.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:15 smithi090 ceph-mon[41670]: osdmap e464: 8 total, 8 up, 8 in 2024-04-14T15:19:16.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:15 smithi090 ceph-mon[30399]: 110.a scrub starts 2024-04-14T15:19:16.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:15 smithi090 ceph-mon[30399]: 110.a scrub ok 2024-04-14T15:19:16.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:15 smithi090 ceph-mon[30399]: 2.5 scrub starts 2024-04-14T15:19:16.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:15 smithi090 ceph-mon[30399]: 2.5 scrub ok 2024-04-14T15:19:16.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:15 smithi090 ceph-mon[30399]: 6.2 scrub starts 2024-04-14T15:19:16.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:15 smithi090 ceph-mon[30399]: 6.2 scrub ok 2024-04-14T15:19:16.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:15 smithi090 ceph-mon[30399]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-14T15:19:16.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:15 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-14T15:19:16.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:15 smithi090 ceph-mon[30399]: from='client.49210 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-14T15:19:16.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:15 smithi090 ceph-mon[30399]: osdmap e463: 8 total, 8 up, 8 in 2024-04-14T15:19:16.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:15 smithi090 ceph-mon[30399]: pgmap v653: 258 pgs: 5 creating+peering, 17 creating+activating, 236 active+clean; 8.4 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-04-14T15:19:16.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:15 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:19:16.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:15 smithi090 ceph-mon[30399]: osdmap e464: 8 total, 8 up, 8 in 2024-04-14T15:19:16.454 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:16.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:16.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:19:16.556 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:19:16.560 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-14T15:19:16.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:16.581 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:16.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:19:16.682 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:16.709 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:16.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:16.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:16.811 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:19:16.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:16.835 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:16.835 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:19:16.936 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:19:16.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:16.960 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:16.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:19:17.061 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:19:17.089 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:17.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:17.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:19:17.132 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:17 smithi090 ceph-mon[41670]: 11.b scrub starts 2024-04-14T15:19:17.133 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:17 smithi090 ceph-mon[41670]: 11.b scrub ok 2024-04-14T15:19:17.133 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:17 smithi090 ceph-mon[41670]: 5.1a scrub starts 2024-04-14T15:19:17.133 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:17 smithi090 ceph-mon[41670]: 5.1a scrub ok 2024-04-14T15:19:17.133 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:17 smithi090 ceph-mon[41670]: 6.17 scrub starts 2024-04-14T15:19:17.133 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:17 smithi090 ceph-mon[41670]: 6.17 scrub ok 2024-04-14T15:19:17.133 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:17 smithi090 ceph-mon[41670]: 11.2 scrub starts 2024-04-14T15:19:17.133 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:17 smithi090 ceph-mon[41670]: 11.2 scrub ok 2024-04-14T15:19:17.133 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:17 smithi090 ceph-mon[41670]: 6.0 scrub starts 2024-04-14T15:19:17.133 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:17 smithi090 ceph-mon[41670]: 6.0 scrub ok 2024-04-14T15:19:17.133 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:17 smithi090 ceph-mon[30399]: 11.b scrub starts 2024-04-14T15:19:17.133 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:17 smithi090 ceph-mon[30399]: 11.b scrub ok 2024-04-14T15:19:17.134 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:17 smithi090 ceph-mon[30399]: 5.1a scrub starts 2024-04-14T15:19:17.134 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:17 smithi090 ceph-mon[30399]: 5.1a scrub ok 2024-04-14T15:19:17.134 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:17 smithi090 ceph-mon[30399]: 6.17 scrub starts 2024-04-14T15:19:17.134 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:17 smithi090 ceph-mon[30399]: 6.17 scrub ok 2024-04-14T15:19:17.134 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:17 smithi090 ceph-mon[30399]: 11.2 scrub starts 2024-04-14T15:19:17.134 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:17 smithi090 ceph-mon[30399]: 11.2 scrub ok 2024-04-14T15:19:17.134 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:17 smithi090 ceph-mon[30399]: 6.0 scrub starts 2024-04-14T15:19:17.134 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:17 smithi090 ceph-mon[30399]: 6.0 scrub ok 2024-04-14T15:19:17.190 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:19:17.214 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:17.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:17.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:19:17.316 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:19:17.339 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:17.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:17.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:19:17.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:17 smithi092 ceph-mon[33966]: 11.b scrub starts 2024-04-14T15:19:17.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:17 smithi092 ceph-mon[33966]: 11.b scrub ok 2024-04-14T15:19:17.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:17 smithi092 ceph-mon[33966]: 5.1a scrub starts 2024-04-14T15:19:17.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:17 smithi092 ceph-mon[33966]: 5.1a scrub ok 2024-04-14T15:19:17.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:17 smithi092 ceph-mon[33966]: 6.17 scrub starts 2024-04-14T15:19:17.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:17 smithi092 ceph-mon[33966]: 6.17 scrub ok 2024-04-14T15:19:17.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:17 smithi092 ceph-mon[33966]: 11.2 scrub starts 2024-04-14T15:19:17.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:17 smithi092 ceph-mon[33966]: 11.2 scrub ok 2024-04-14T15:19:17.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:17 smithi092 ceph-mon[33966]: 6.0 scrub starts 2024-04-14T15:19:17.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:17 smithi092 ceph-mon[33966]: 6.0 scrub ok 2024-04-14T15:19:17.441 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:19:17.465 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:17.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:17.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:19:17.566 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:19:17.595 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:17.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:17.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:19:17.697 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:19:17.721 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:17.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:17.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:19:17.823 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:19:17.846 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:17.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:17.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:19:17.948 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:19:17.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:17.972 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:17.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:19:18.026 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd set noscrub 2024-04-14T15:19:18.073 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:19:18.099 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:18.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:18.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:19:18.201 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:19:18.224 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:18.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:18.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:19:18.326 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:18.354 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:18.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:18.355 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:18.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:18 smithi092 ceph-mon[33966]: 6.8 scrub starts 2024-04-14T15:19:18.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:18 smithi092 ceph-mon[33966]: 6.8 scrub ok 2024-04-14T15:19:18.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:18 smithi092 ceph-mon[33966]: 2.1 scrub starts 2024-04-14T15:19:18.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:18 smithi092 ceph-mon[33966]: 2.1 scrub ok 2024-04-14T15:19:18.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:18 smithi092 ceph-mon[33966]: 110.3 deep-scrub starts 2024-04-14T15:19:18.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:18 smithi092 ceph-mon[33966]: 110.3 deep-scrub ok 2024-04-14T15:19:18.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:18 smithi092 ceph-mon[33966]: pgmap v655: 258 pgs: 5 creating+peering, 17 creating+activating, 236 active+clean; 8.4 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-04-14T15:19:18.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:18 smithi092 ceph-mon[33966]: 110.1e scrub starts 2024-04-14T15:19:18.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:18 smithi092 ceph-mon[33966]: 110.1e scrub ok 2024-04-14T15:19:18.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:18 smithi090 ceph-mon[41670]: 6.8 scrub starts 2024-04-14T15:19:18.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:18 smithi090 ceph-mon[41670]: 6.8 scrub ok 2024-04-14T15:19:18.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:18 smithi090 ceph-mon[41670]: 2.1 scrub starts 2024-04-14T15:19:18.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:18 smithi090 ceph-mon[41670]: 2.1 scrub ok 2024-04-14T15:19:18.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:18 smithi090 ceph-mon[41670]: 110.3 deep-scrub starts 2024-04-14T15:19:18.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:18 smithi090 ceph-mon[41670]: 110.3 deep-scrub ok 2024-04-14T15:19:18.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:18 smithi090 ceph-mon[41670]: pgmap v655: 258 pgs: 5 creating+peering, 17 creating+activating, 236 active+clean; 8.4 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-04-14T15:19:18.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:18 smithi090 ceph-mon[41670]: 110.1e scrub starts 2024-04-14T15:19:18.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:18 smithi090 ceph-mon[41670]: 110.1e scrub ok 2024-04-14T15:19:18.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:18 smithi090 ceph-mon[30399]: 6.8 scrub starts 2024-04-14T15:19:18.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:18 smithi090 ceph-mon[30399]: 6.8 scrub ok 2024-04-14T15:19:18.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:18 smithi090 ceph-mon[30399]: 2.1 scrub starts 2024-04-14T15:19:18.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:18 smithi090 ceph-mon[30399]: 2.1 scrub ok 2024-04-14T15:19:18.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:18 smithi090 ceph-mon[30399]: 110.3 deep-scrub starts 2024-04-14T15:19:18.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:18 smithi090 ceph-mon[30399]: 110.3 deep-scrub ok 2024-04-14T15:19:18.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:18 smithi090 ceph-mon[30399]: pgmap v655: 258 pgs: 5 creating+peering, 17 creating+activating, 236 active+clean; 8.4 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-04-14T15:19:18.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:18 smithi090 ceph-mon[30399]: 110.1e scrub starts 2024-04-14T15:19:18.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:18 smithi090 ceph-mon[30399]: 110.1e scrub ok 2024-04-14T15:19:18.455 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:18.483 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:18.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:18.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:18.584 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:19:18.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:18.611 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:18.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:19:18.713 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:18.740 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:18.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:18.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:18.842 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:19:18.869 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:18.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:18.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:19:18.971 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:19.001 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:19.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:19.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:19.103 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:19:19.129 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:19.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:19.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:19:19.231 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:19.241 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:19 smithi090 ceph-mon[30399]: 4.1d scrub starts 2024-04-14T15:19:19.241 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:19 smithi090 ceph-mon[30399]: 4.1d scrub ok 2024-04-14T15:19:19.241 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:19 smithi090 ceph-mon[30399]: 4.13 deep-scrub starts 2024-04-14T15:19:19.241 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:19 smithi090 ceph-mon[30399]: 4.13 deep-scrub ok 2024-04-14T15:19:19.241 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:19 smithi090 ceph-mon[30399]: 2.1f scrub starts 2024-04-14T15:19:19.241 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:19 smithi090 ceph-mon[30399]: 2.1f scrub ok 2024-04-14T15:19:19.241 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:19 smithi090 ceph-mon[41670]: 4.1d scrub starts 2024-04-14T15:19:19.241 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:19 smithi090 ceph-mon[41670]: 4.1d scrub ok 2024-04-14T15:19:19.241 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:19 smithi090 ceph-mon[41670]: 4.13 deep-scrub starts 2024-04-14T15:19:19.241 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:19 smithi090 ceph-mon[41670]: 4.13 deep-scrub ok 2024-04-14T15:19:19.241 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:19 smithi090 ceph-mon[41670]: 2.1f scrub starts 2024-04-14T15:19:19.242 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:19 smithi090 ceph-mon[41670]: 2.1f scrub ok 2024-04-14T15:19:19.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.251+0000 7fbcd435a700 1 -- 172.21.15.90:0/3925991633 >> v1:172.21.15.90:6789/0 conn(0x7fbccc10b720 legacy=0x7fbccc10db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:19.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.251+0000 7fbcd435a700 1 -- 172.21.15.90:0/3925991633 shutdown_connections 2024-04-14T15:19:19.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.251+0000 7fbcd435a700 1 -- 172.21.15.90:0/3925991633 >> 172.21.15.90:0/3925991633 conn(0x7fbccc0fe3a0 msgr2=0x7fbccc1007c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:19:19.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.251+0000 7fbcd435a700 1 -- 172.21.15.90:0/3925991633 shutdown_connections 2024-04-14T15:19:19.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.251+0000 7fbcd435a700 1 -- 172.21.15.90:0/3925991633 wait complete. 2024-04-14T15:19:19.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.251+0000 7fbcd435a700 1 librados: starting msgr at 2024-04-14T15:19:19.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.251+0000 7fbcd435a700 1 librados: starting objecter 2024-04-14T15:19:19.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.251+0000 7fbcd435a700 1 Processor -- start 2024-04-14T15:19:19.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.252+0000 7fbcd435a700 1 -- start start 2024-04-14T15:19:19.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.252+0000 7fbcd435a700 1 librados: setting wanted keys 2024-04-14T15:19:19.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.252+0000 7fbcd435a700 1 librados: calling monclient init 2024-04-14T15:19:19.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.252+0000 7fbcd435a700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbccc076a00 con 0x7fbccc10b720 2024-04-14T15:19:19.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.252+0000 7fbcd435a700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbccc076c10 con 0x7fbccc10f2f0 2024-04-14T15:19:19.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.252+0000 7fbcd435a700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbccc1ac300 con 0x7fbccc108870 2024-04-14T15:19:19.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.252+0000 7fbcd20f6700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7fbccc108870 0x7fbccc074ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:51784/0 (socket says 172.21.15.90:51784) 2024-04-14T15:19:19.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.252+0000 7fbcd18f5700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fbccc10b720 0x7fbccc0753f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:44216/0 (socket says 172.21.15.90:44216) 2024-04-14T15:19:19.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.252+0000 7fbcd20f6700 1 -- 172.21.15.90:0/3122204600 learned_addr learned my addr 172.21.15.90:0/3122204600 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:19:19.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.252+0000 7fbccb7fe700 10 client.?.objecter ms_handle_connect 0x7fbccc10b720 2024-04-14T15:19:19.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.252+0000 7fbccb7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:19:19.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.252+0000 7fbccb7fe700 10 client.?.objecter ms_handle_connect 0x7fbccc108870 2024-04-14T15:19:19.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.252+0000 7fbccb7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:19:19.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.252+0000 7fbccb7fe700 10 client.?.objecter ms_handle_connect 0x7fbccc10f2f0 2024-04-14T15:19:19.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.252+0000 7fbccb7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:19:19.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.252+0000 7fbccb7fe700 1 -- 172.21.15.90:0/3122204600 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3177188631 0 0) 0x7fbccc076a00 con 0x7fbccc10b720 2024-04-14T15:19:19.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.252+0000 7fbccb7fe700 1 -- 172.21.15.90:0/3122204600 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbcb40031f0 con 0x7fbccc10b720 2024-04-14T15:19:19.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.252+0000 7fbccb7fe700 1 -- 172.21.15.90:0/3122204600 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4201139501 0 0) 0x7fbccc076c10 con 0x7fbccc10f2f0 2024-04-14T15:19:19.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.252+0000 7fbccb7fe700 1 -- 172.21.15.90:0/3122204600 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbccc076a00 con 0x7fbccc10f2f0 2024-04-14T15:19:19.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.252+0000 7fbccb7fe700 1 -- 172.21.15.90:0/3122204600 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 538791379 0 0) 0x7fbccc1ac300 con 0x7fbccc108870 2024-04-14T15:19:19.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.252+0000 7fbccb7fe700 1 -- 172.21.15.90:0/3122204600 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbccc076c10 con 0x7fbccc108870 2024-04-14T15:19:19.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.252+0000 7fbccb7fe700 1 -- 172.21.15.90:0/3122204600 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3885149262 0 0) 0x7fbcb40031f0 con 0x7fbccc10b720 2024-04-14T15:19:19.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.252+0000 7fbccb7fe700 1 -- 172.21.15.90:0/3122204600 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbccc1ac300 con 0x7fbccc10b720 2024-04-14T15:19:19.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.252+0000 7fbccb7fe700 1 -- 172.21.15.90:0/3122204600 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fbcc00015d0 con 0x7fbccc10b720 2024-04-14T15:19:19.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.252+0000 7fbccb7fe700 1 -- 172.21.15.90:0/3122204600 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2092664953 0 0) 0x7fbccc076a00 con 0x7fbccc10f2f0 2024-04-14T15:19:19.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.252+0000 7fbccb7fe700 1 -- 172.21.15.90:0/3122204600 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbcb40031f0 con 0x7fbccc10f2f0 2024-04-14T15:19:19.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.252+0000 7fbccb7fe700 1 -- 172.21.15.90:0/3122204600 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fbcbc001830 con 0x7fbccc10f2f0 2024-04-14T15:19:19.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.252+0000 7fbccb7fe700 1 -- 172.21.15.90:0/3122204600 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1369021285 0 0) 0x7fbccc076c10 con 0x7fbccc108870 2024-04-14T15:19:19.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.252+0000 7fbccb7fe700 1 -- 172.21.15.90:0/3122204600 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbccc076a00 con 0x7fbccc108870 2024-04-14T15:19:19.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.252+0000 7fbccb7fe700 1 -- 172.21.15.90:0/3122204600 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fbcc4001c70 con 0x7fbccc108870 2024-04-14T15:19:19.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.253+0000 7fbccb7fe700 1 -- 172.21.15.90:0/3122204600 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4052135878 0 0) 0x7fbccc1ac300 con 0x7fbccc10b720 2024-04-14T15:19:19.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.253+0000 7fbccb7fe700 1 -- 172.21.15.90:0/3122204600 >> v1:172.21.15.90:6790/0 conn(0x7fbccc108870 legacy=0x7fbccc074ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:19.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.253+0000 7fbccb7fe700 1 -- 172.21.15.90:0/3122204600 >> v1:172.21.15.92:6789/0 conn(0x7fbccc10f2f0 legacy=0x7fbccc1a8bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:19.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.253+0000 7fbccb7fe700 1 -- 172.21.15.90:0/3122204600 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbccc1ad4e0 con 0x7fbccc10b720 2024-04-14T15:19:19.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.253+0000 7fbcd435a700 1 -- 172.21.15.90:0/3122204600 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbccc1ac530 con 0x7fbccc10b720 2024-04-14T15:19:19.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.253+0000 7fbcd435a700 10 client.37377.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:19:19.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.253+0000 7fbcd435a700 1 -- 172.21.15.90:0/3122204600 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fbccc1ac9f0 con 0x7fbccc10b720 2024-04-14T15:19:19.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.253+0000 7fbcd435a700 1 librados: init done 2024-04-14T15:19:19.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.253+0000 7fbccb7fe700 1 -- 172.21.15.90:0/3122204600 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fbcc0003760 con 0x7fbccc10b720 2024-04-14T15:19:19.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.253+0000 7fbccb7fe700 1 -- 172.21.15.90:0/3122204600 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fbcc0004d80 con 0x7fbccc10b720 2024-04-14T15:19:19.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:19.258 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:19.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:19.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.253+0000 7fbcd435a700 1 -- 172.21.15.90:0/3122204600 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fbcb8005100 con 0x7fbccc10b720 2024-04-14T15:19:19.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.254+0000 7fbccb7fe700 1 -- 172.21.15.90:0/3122204600 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fbcc0001060 con 0x7fbccc10b720 2024-04-14T15:19:19.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.255+0000 7fbccb7fe700 1 -- 172.21.15.90:0/3122204600 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(464..464 src has 51..464) ==== 8160+0+0 (unknown 3980499053 0 0) 0x7fbcc008a910 con 0x7fbccc10b720 2024-04-14T15:19:19.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.255+0000 7fbccb7fe700 10 client.37377.objecter ms_dispatch 0x7fbccc001550 osd_map(464..464 src has 51..464) 2024-04-14T15:19:19.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.255+0000 7fbccb7fe700 3 client.37377.objecter handle_osd_map got epochs [464,464] > 0 2024-04-14T15:19:19.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.255+0000 7fbccb7fe700 3 client.37377.objecter handle_osd_map decoding full epoch 464 2024-04-14T15:19:19.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.255+0000 7fbccb7fe700 20 client.37377.objecter dump_active .. 0 homeless 2024-04-14T15:19:19.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.255+0000 7fbccb7fe700 1 -- 172.21.15.90:0/3122204600 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v464) ==== 54+0+16 (unknown 3390603059 0 2420000192) 0x7fbcc008b060 con 0x7fbccc10b720 2024-04-14T15:19:19.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.255+0000 7fbccb7fe700 10 client.37377.objecter ms_handle_connect 0x7fbcb4071380 2024-04-14T15:19:19.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.255+0000 7fbcd435a700 10 client.37377.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:19:19.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.255+0000 7fbcd435a700 15 client.37377.objecter _session_command_op_assign -1 1 2024-04-14T15:19:19.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.255+0000 7fbcd435a700 20 client.37377.objecter _get_session s=0x7fbcb80053e0 osd=0 3 2024-04-14T15:19:19.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.255+0000 7fbcd435a700 20 client.37377.objecter put_session s=0x7fbcb80053e0 osd=0 3 2024-04-14T15:19:19.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.255+0000 7fbcd435a700 20 client.37377.objecter _get_session s=0x7fbcb80053e0 osd=0 3 2024-04-14T15:19:19.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.255+0000 7fbcd435a700 15 client.37377.objecter _session_command_op_remove -1 1 2024-04-14T15:19:19.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.255+0000 7fbcd435a700 20 client.37377.objecter get_session s=0x7fbcb80053e0 osd=0 3 2024-04-14T15:19:19.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.255+0000 7fbcd435a700 15 client.37377.objecter _session_command_op_assign 0 1 2024-04-14T15:19:19.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.255+0000 7fbcd435a700 20 client.37377.objecter put_session s=0x7fbcb80053e0 osd=0 4 2024-04-14T15:19:19.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.255+0000 7fbcd435a700 10 client.37377.objecter _send_command 1 2024-04-14T15:19:19.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.255+0000 7fbcd435a700 1 -- 172.21.15.90:0/3122204600 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbcb8004b50 con 0x7fbcb8004730 2024-04-14T15:19:19.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.255+0000 7fbccb7fe700 10 client.37377.objecter ms_handle_connect 0x7fbcb8004730 2024-04-14T15:19:19.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.256+0000 7fbccb7fe700 1 -- 172.21.15.90:0/3122204600 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fbcb8004b50 con 0x7fbcb8004730 2024-04-14T15:19:19.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.256+0000 7fbccb7fe700 10 client.37377.objecter ms_dispatch 0x7fbccc001550 command_reply(tid 1: 0 ) 2024-04-14T15:19:19.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.256+0000 7fbccb7fe700 10 client.37377.objecter _finish_command 1 = system:0 2024-04-14T15:19:19.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.256+0000 7fbccb7fe700 20 client.37377.objecter put_session s=0x7fbcb80053e0 osd=0 4 2024-04-14T15:19:19.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.256+0000 7fbccb7fe700 15 client.37377.objecter _session_command_op_remove 0 1 2024-04-14T15:19:19.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.273+0000 7fbcd435a700 10 client.37377.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:19:19.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.273+0000 7fbcd435a700 15 client.37377.objecter _session_command_op_assign -1 2 2024-04-14T15:19:19.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.273+0000 7fbcd435a700 20 client.37377.objecter _get_session s=0x7fbcb80053e0 osd=0 3 2024-04-14T15:19:19.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.273+0000 7fbcd435a700 20 client.37377.objecter put_session s=0x7fbcb80053e0 osd=0 3 2024-04-14T15:19:19.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.273+0000 7fbcd435a700 20 client.37377.objecter _get_session s=0x7fbcb80053e0 osd=0 3 2024-04-14T15:19:19.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.273+0000 7fbcd435a700 15 client.37377.objecter _session_command_op_remove -1 2 2024-04-14T15:19:19.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.273+0000 7fbcd435a700 20 client.37377.objecter get_session s=0x7fbcb80053e0 osd=0 3 2024-04-14T15:19:19.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.273+0000 7fbcd435a700 15 client.37377.objecter _session_command_op_assign 0 2 2024-04-14T15:19:19.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.273+0000 7fbcd435a700 20 client.37377.objecter put_session s=0x7fbcb80053e0 osd=0 4 2024-04-14T15:19:19.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.273+0000 7fbcd435a700 10 client.37377.objecter _send_command 2 2024-04-14T15:19:19.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.273+0000 7fbcd435a700 1 -- 172.21.15.90:0/3122204600 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fbcb80079f0 con 0x7fbcb8004730 2024-04-14T15:19:19.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.274+0000 7fbccb7fe700 1 -- 172.21.15.90:0/3122204600 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fbcb80079f0 con 0x7fbcb8004730 2024-04-14T15:19:19.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.274+0000 7fbccb7fe700 10 client.37377.objecter ms_dispatch 0x7fbccc001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:19:19.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.274+0000 7fbccb7fe700 10 client.37377.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:19:19.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.274+0000 7fbccb7fe700 20 client.37377.objecter put_session s=0x7fbcb80053e0 osd=0 4 2024-04-14T15:19:19.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.274+0000 7fbccb7fe700 15 client.37377.objecter _session_command_op_remove 0 2 2024-04-14T15:19:19.275 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:19:19.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.274+0000 7fbcd435a700 10 client.37377.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:19:19.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.274+0000 7fbcd435a700 15 client.37377.objecter _session_command_op_assign -1 3 2024-04-14T15:19:19.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.274+0000 7fbcd435a700 20 client.37377.objecter _get_session s=0x7fbcb8004110 osd=1 3 2024-04-14T15:19:19.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.274+0000 7fbcd435a700 20 client.37377.objecter put_session s=0x7fbcb8004110 osd=1 3 2024-04-14T15:19:19.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.274+0000 7fbcd435a700 20 client.37377.objecter _get_session s=0x7fbcb8004110 osd=1 3 2024-04-14T15:19:19.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.274+0000 7fbcd435a700 15 client.37377.objecter _session_command_op_remove -1 3 2024-04-14T15:19:19.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.274+0000 7fbcd435a700 20 client.37377.objecter get_session s=0x7fbcb8004110 osd=1 3 2024-04-14T15:19:19.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.274+0000 7fbcd435a700 15 client.37377.objecter _session_command_op_assign 1 3 2024-04-14T15:19:19.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.274+0000 7fbcd435a700 20 client.37377.objecter put_session s=0x7fbcb8004110 osd=1 4 2024-04-14T15:19:19.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.274+0000 7fbcd435a700 10 client.37377.objecter _send_command 3 2024-04-14T15:19:19.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.274+0000 7fbcd435a700 1 -- 172.21.15.90:0/3122204600 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fbcb8002180 con 0x7fbcb8004280 2024-04-14T15:19:19.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.275+0000 7fbccb7fe700 10 client.37377.objecter ms_handle_connect 0x7fbcb8004280 2024-04-14T15:19:19.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.275+0000 7fbccb7fe700 1 -- 172.21.15.90:0/3122204600 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fbcb8002180 con 0x7fbcb8004280 2024-04-14T15:19:19.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.275+0000 7fbccb7fe700 10 client.37377.objecter ms_dispatch 0x7fbccc001550 command_reply(tid 3: 0 ) 2024-04-14T15:19:19.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.275+0000 7fbccb7fe700 10 client.37377.objecter _finish_command 3 = system:0 2024-04-14T15:19:19.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.275+0000 7fbccb7fe700 20 client.37377.objecter put_session s=0x7fbcb8004110 osd=1 4 2024-04-14T15:19:19.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.275+0000 7fbccb7fe700 15 client.37377.objecter _session_command_op_remove 1 3 2024-04-14T15:19:19.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.292+0000 7fbcd435a700 10 client.37377.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:19:19.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.292+0000 7fbcd435a700 15 client.37377.objecter _session_command_op_assign -1 4 2024-04-14T15:19:19.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.292+0000 7fbcd435a700 20 client.37377.objecter _get_session s=0x7fbcb8004110 osd=1 3 2024-04-14T15:19:19.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.292+0000 7fbcd435a700 20 client.37377.objecter put_session s=0x7fbcb8004110 osd=1 3 2024-04-14T15:19:19.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.292+0000 7fbcd435a700 20 client.37377.objecter _get_session s=0x7fbcb8004110 osd=1 3 2024-04-14T15:19:19.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.292+0000 7fbcd435a700 15 client.37377.objecter _session_command_op_remove -1 4 2024-04-14T15:19:19.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.292+0000 7fbcd435a700 20 client.37377.objecter get_session s=0x7fbcb8004110 osd=1 3 2024-04-14T15:19:19.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.292+0000 7fbcd435a700 15 client.37377.objecter _session_command_op_assign 1 4 2024-04-14T15:19:19.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.292+0000 7fbcd435a700 20 client.37377.objecter put_session s=0x7fbcb8004110 osd=1 4 2024-04-14T15:19:19.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.292+0000 7fbcd435a700 10 client.37377.objecter _send_command 4 2024-04-14T15:19:19.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.292+0000 7fbcd435a700 1 -- 172.21.15.90:0/3122204600 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fbcb8001600 con 0x7fbcb8004280 2024-04-14T15:19:19.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.294+0000 7fbccb7fe700 1 -- 172.21.15.90:0/3122204600 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fbcb8001600 con 0x7fbcb8004280 2024-04-14T15:19:19.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.294+0000 7fbccb7fe700 10 client.37377.objecter ms_dispatch 0x7fbccc001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:19:19.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.294+0000 7fbccb7fe700 10 client.37377.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:19:19.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.294+0000 7fbccb7fe700 20 client.37377.objecter put_session s=0x7fbcb8004110 osd=1 4 2024-04-14T15:19:19.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.294+0000 7fbccb7fe700 15 client.37377.objecter _session_command_op_remove 1 4 2024-04-14T15:19:19.295 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:19:19.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.294+0000 7fbcd435a700 10 client.37377.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:19:19.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.294+0000 7fbcd435a700 15 client.37377.objecter _session_command_op_assign -1 5 2024-04-14T15:19:19.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.294+0000 7fbcd435a700 20 client.37377.objecter _get_session s=0x7fbcb8008370 osd=2 3 2024-04-14T15:19:19.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.294+0000 7fbcd435a700 20 client.37377.objecter put_session s=0x7fbcb8008370 osd=2 3 2024-04-14T15:19:19.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.294+0000 7fbcd435a700 20 client.37377.objecter _get_session s=0x7fbcb8008370 osd=2 3 2024-04-14T15:19:19.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.294+0000 7fbcd435a700 15 client.37377.objecter _session_command_op_remove -1 5 2024-04-14T15:19:19.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.294+0000 7fbcd435a700 20 client.37377.objecter get_session s=0x7fbcb8008370 osd=2 3 2024-04-14T15:19:19.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.294+0000 7fbcd435a700 15 client.37377.objecter _session_command_op_assign 2 5 2024-04-14T15:19:19.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.294+0000 7fbcd435a700 20 client.37377.objecter put_session s=0x7fbcb8008370 osd=2 4 2024-04-14T15:19:19.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.294+0000 7fbcd435a700 10 client.37377.objecter _send_command 5 2024-04-14T15:19:19.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.294+0000 7fbcd435a700 1 -- 172.21.15.90:0/3122204600 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fbcb80110e0 con 0x7fbcb800d5d0 2024-04-14T15:19:19.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.294+0000 7fbccb7fe700 10 client.37377.objecter ms_handle_connect 0x7fbcb800d5d0 2024-04-14T15:19:19.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.295+0000 7fbccb7fe700 1 -- 172.21.15.90:0/3122204600 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fbcb80110e0 con 0x7fbcb800d5d0 2024-04-14T15:19:19.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.295+0000 7fbccb7fe700 10 client.37377.objecter ms_dispatch 0x7fbccc001550 command_reply(tid 5: 0 ) 2024-04-14T15:19:19.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.295+0000 7fbccb7fe700 10 client.37377.objecter _finish_command 5 = system:0 2024-04-14T15:19:19.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.295+0000 7fbccb7fe700 20 client.37377.objecter put_session s=0x7fbcb8008370 osd=2 4 2024-04-14T15:19:19.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.295+0000 7fbccb7fe700 15 client.37377.objecter _session_command_op_remove 2 5 2024-04-14T15:19:19.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.311+0000 7fbcd435a700 10 client.37377.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:19:19.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.311+0000 7fbcd435a700 15 client.37377.objecter _session_command_op_assign -1 6 2024-04-14T15:19:19.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.311+0000 7fbcd435a700 20 client.37377.objecter _get_session s=0x7fbcb8008370 osd=2 3 2024-04-14T15:19:19.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.311+0000 7fbcd435a700 20 client.37377.objecter put_session s=0x7fbcb8008370 osd=2 3 2024-04-14T15:19:19.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.311+0000 7fbcd435a700 20 client.37377.objecter _get_session s=0x7fbcb8008370 osd=2 3 2024-04-14T15:19:19.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.311+0000 7fbcd435a700 15 client.37377.objecter _session_command_op_remove -1 6 2024-04-14T15:19:19.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.311+0000 7fbcd435a700 20 client.37377.objecter get_session s=0x7fbcb8008370 osd=2 3 2024-04-14T15:19:19.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.311+0000 7fbcd435a700 15 client.37377.objecter _session_command_op_assign 2 6 2024-04-14T15:19:19.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.311+0000 7fbcd435a700 20 client.37377.objecter put_session s=0x7fbcb8008370 osd=2 4 2024-04-14T15:19:19.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.311+0000 7fbcd435a700 10 client.37377.objecter _send_command 6 2024-04-14T15:19:19.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.311+0000 7fbcd435a700 1 -- 172.21.15.90:0/3122204600 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fbcb8007eb0 con 0x7fbcb800d5d0 2024-04-14T15:19:19.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.313+0000 7fbccb7fe700 1 -- 172.21.15.90:0/3122204600 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fbcb8007eb0 con 0x7fbcb800d5d0 2024-04-14T15:19:19.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.313+0000 7fbccb7fe700 10 client.37377.objecter ms_dispatch 0x7fbccc001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:19:19.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.313+0000 7fbccb7fe700 10 client.37377.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:19:19.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.313+0000 7fbccb7fe700 20 client.37377.objecter put_session s=0x7fbcb8008370 osd=2 4 2024-04-14T15:19:19.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.313+0000 7fbccb7fe700 15 client.37377.objecter _session_command_op_remove 2 6 2024-04-14T15:19:19.314 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:19:19.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.313+0000 7fbcd435a700 10 client.37377.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:19:19.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.313+0000 7fbcd435a700 15 client.37377.objecter _session_command_op_assign -1 7 2024-04-14T15:19:19.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.313+0000 7fbcd435a700 20 client.37377.objecter _get_session s=0x7fbcb800cc20 osd=3 3 2024-04-14T15:19:19.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.313+0000 7fbcd435a700 20 client.37377.objecter put_session s=0x7fbcb800cc20 osd=3 3 2024-04-14T15:19:19.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.313+0000 7fbcd435a700 20 client.37377.objecter _get_session s=0x7fbcb800cc20 osd=3 3 2024-04-14T15:19:19.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.313+0000 7fbcd435a700 15 client.37377.objecter _session_command_op_remove -1 7 2024-04-14T15:19:19.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.313+0000 7fbcd435a700 20 client.37377.objecter get_session s=0x7fbcb800cc20 osd=3 3 2024-04-14T15:19:19.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.313+0000 7fbcd435a700 15 client.37377.objecter _session_command_op_assign 3 7 2024-04-14T15:19:19.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.313+0000 7fbcd435a700 20 client.37377.objecter put_session s=0x7fbcb800cc20 osd=3 4 2024-04-14T15:19:19.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.313+0000 7fbcd435a700 10 client.37377.objecter _send_command 7 2024-04-14T15:19:19.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.313+0000 7fbcd435a700 1 -- 172.21.15.90:0/3122204600 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fbcb8015f20 con 0x7fbcb8012410 2024-04-14T15:19:19.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.313+0000 7fbccb7fe700 10 client.37377.objecter ms_handle_connect 0x7fbcb8012410 2024-04-14T15:19:19.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.314+0000 7fbccb7fe700 1 -- 172.21.15.90:0/3122204600 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fbcb8015f20 con 0x7fbcb8012410 2024-04-14T15:19:19.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.314+0000 7fbccb7fe700 10 client.37377.objecter ms_dispatch 0x7fbccc001550 command_reply(tid 7: 0 ) 2024-04-14T15:19:19.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.314+0000 7fbccb7fe700 10 client.37377.objecter _finish_command 7 = system:0 2024-04-14T15:19:19.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.314+0000 7fbccb7fe700 20 client.37377.objecter put_session s=0x7fbcb800cc20 osd=3 4 2024-04-14T15:19:19.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.314+0000 7fbccb7fe700 15 client.37377.objecter _session_command_op_remove 3 7 2024-04-14T15:19:19.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.330+0000 7fbcd435a700 10 client.37377.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:19:19.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.330+0000 7fbcd435a700 15 client.37377.objecter _session_command_op_assign -1 8 2024-04-14T15:19:19.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.330+0000 7fbcd435a700 20 client.37377.objecter _get_session s=0x7fbcb800cc20 osd=3 3 2024-04-14T15:19:19.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.330+0000 7fbcd435a700 20 client.37377.objecter put_session s=0x7fbcb800cc20 osd=3 3 2024-04-14T15:19:19.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.330+0000 7fbcd435a700 20 client.37377.objecter _get_session s=0x7fbcb800cc20 osd=3 3 2024-04-14T15:19:19.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.330+0000 7fbcd435a700 15 client.37377.objecter _session_command_op_remove -1 8 2024-04-14T15:19:19.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.330+0000 7fbcd435a700 20 client.37377.objecter get_session s=0x7fbcb800cc20 osd=3 3 2024-04-14T15:19:19.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.330+0000 7fbcd435a700 15 client.37377.objecter _session_command_op_assign 3 8 2024-04-14T15:19:19.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.330+0000 7fbcd435a700 20 client.37377.objecter put_session s=0x7fbcb800cc20 osd=3 4 2024-04-14T15:19:19.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.330+0000 7fbcd435a700 10 client.37377.objecter _send_command 8 2024-04-14T15:19:19.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.330+0000 7fbcd435a700 1 -- 172.21.15.90:0/3122204600 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fbcb800c700 con 0x7fbcb8012410 2024-04-14T15:19:19.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.332+0000 7fbccb7fe700 1 -- 172.21.15.90:0/3122204600 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fbcb800c700 con 0x7fbcb8012410 2024-04-14T15:19:19.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.332+0000 7fbccb7fe700 10 client.37377.objecter ms_dispatch 0x7fbccc001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:19:19.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.332+0000 7fbccb7fe700 10 client.37377.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:19:19.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.332+0000 7fbccb7fe700 20 client.37377.objecter put_session s=0x7fbcb800cc20 osd=3 4 2024-04-14T15:19:19.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.332+0000 7fbccb7fe700 15 client.37377.objecter _session_command_op_remove 3 8 2024-04-14T15:19:19.333 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:19:19.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.332+0000 7fbcd435a700 10 client.37377.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:19:19.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.332+0000 7fbcd435a700 15 client.37377.objecter _session_command_op_assign -1 9 2024-04-14T15:19:19.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.332+0000 7fbcd435a700 20 client.37377.objecter _get_session s=0x7fbcb800c700 osd=4 3 2024-04-14T15:19:19.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.332+0000 7fbcd435a700 20 client.37377.objecter put_session s=0x7fbcb800c700 osd=4 3 2024-04-14T15:19:19.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.332+0000 7fbcd435a700 20 client.37377.objecter _get_session s=0x7fbcb800c700 osd=4 3 2024-04-14T15:19:19.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.332+0000 7fbcd435a700 15 client.37377.objecter _session_command_op_remove -1 9 2024-04-14T15:19:19.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.332+0000 7fbcd435a700 20 client.37377.objecter get_session s=0x7fbcb800c700 osd=4 3 2024-04-14T15:19:19.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.332+0000 7fbcd435a700 15 client.37377.objecter _session_command_op_assign 4 9 2024-04-14T15:19:19.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.332+0000 7fbcd435a700 20 client.37377.objecter put_session s=0x7fbcb800c700 osd=4 4 2024-04-14T15:19:19.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.332+0000 7fbcd435a700 10 client.37377.objecter _send_command 9 2024-04-14T15:19:19.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.332+0000 7fbcd435a700 1 -- 172.21.15.90:0/3122204600 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fbcb801a9c0 con 0x7fbcb8011f70 2024-04-14T15:19:19.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.333+0000 7fbccb7fe700 10 client.37377.objecter ms_handle_connect 0x7fbcb8011f70 2024-04-14T15:19:19.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.334+0000 7fbccb7fe700 1 -- 172.21.15.90:0/3122204600 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fbcb801a9c0 con 0x7fbcb8011f70 2024-04-14T15:19:19.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.334+0000 7fbccb7fe700 10 client.37377.objecter ms_dispatch 0x7fbccc001550 command_reply(tid 9: 0 ) 2024-04-14T15:19:19.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.334+0000 7fbccb7fe700 10 client.37377.objecter _finish_command 9 = system:0 2024-04-14T15:19:19.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.334+0000 7fbccb7fe700 20 client.37377.objecter put_session s=0x7fbcb800c700 osd=4 4 2024-04-14T15:19:19.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.334+0000 7fbccb7fe700 15 client.37377.objecter _session_command_op_remove 4 9 2024-04-14T15:19:19.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.350+0000 7fbcd435a700 10 client.37377.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:19:19.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.350+0000 7fbcd435a700 15 client.37377.objecter _session_command_op_assign -1 10 2024-04-14T15:19:19.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.350+0000 7fbcd435a700 20 client.37377.objecter _get_session s=0x7fbcb800c700 osd=4 3 2024-04-14T15:19:19.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.350+0000 7fbcd435a700 20 client.37377.objecter put_session s=0x7fbcb800c700 osd=4 3 2024-04-14T15:19:19.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.350+0000 7fbcd435a700 20 client.37377.objecter _get_session s=0x7fbcb800c700 osd=4 3 2024-04-14T15:19:19.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.350+0000 7fbcd435a700 15 client.37377.objecter _session_command_op_remove -1 10 2024-04-14T15:19:19.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.350+0000 7fbcd435a700 20 client.37377.objecter get_session s=0x7fbcb800c700 osd=4 3 2024-04-14T15:19:19.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.350+0000 7fbcd435a700 15 client.37377.objecter _session_command_op_assign 4 10 2024-04-14T15:19:19.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.350+0000 7fbcd435a700 20 client.37377.objecter put_session s=0x7fbcb800c700 osd=4 4 2024-04-14T15:19:19.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.350+0000 7fbcd435a700 10 client.37377.objecter _send_command 10 2024-04-14T15:19:19.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.350+0000 7fbcd435a700 1 -- 172.21.15.90:0/3122204600 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fbcb801c370 con 0x7fbcb8011f70 2024-04-14T15:19:19.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.351+0000 7fbccb7fe700 1 -- 172.21.15.90:0/3122204600 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fbcb801c370 con 0x7fbcb8011f70 2024-04-14T15:19:19.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.351+0000 7fbccb7fe700 10 client.37377.objecter ms_dispatch 0x7fbccc001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:19:19.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.351+0000 7fbccb7fe700 10 client.37377.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:19:19.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.351+0000 7fbccb7fe700 20 client.37377.objecter put_session s=0x7fbcb800c700 osd=4 4 2024-04-14T15:19:19.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.351+0000 7fbccb7fe700 15 client.37377.objecter _session_command_op_remove 4 10 2024-04-14T15:19:19.352 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:19:19.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.352+0000 7fbcd435a700 10 client.37377.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:19:19.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.352+0000 7fbcd435a700 15 client.37377.objecter _session_command_op_assign -1 11 2024-04-14T15:19:19.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.352+0000 7fbcd435a700 20 client.37377.objecter _get_session s=0x7fbcb801c370 osd=5 3 2024-04-14T15:19:19.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.352+0000 7fbcd435a700 20 client.37377.objecter put_session s=0x7fbcb801c370 osd=5 3 2024-04-14T15:19:19.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.352+0000 7fbcd435a700 20 client.37377.objecter _get_session s=0x7fbcb801c370 osd=5 3 2024-04-14T15:19:19.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.352+0000 7fbcd435a700 15 client.37377.objecter _session_command_op_remove -1 11 2024-04-14T15:19:19.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.352+0000 7fbcd435a700 20 client.37377.objecter get_session s=0x7fbcb801c370 osd=5 3 2024-04-14T15:19:19.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.352+0000 7fbcd435a700 15 client.37377.objecter _session_command_op_assign 5 11 2024-04-14T15:19:19.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.352+0000 7fbcd435a700 20 client.37377.objecter put_session s=0x7fbcb801c370 osd=5 4 2024-04-14T15:19:19.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.352+0000 7fbcd435a700 10 client.37377.objecter _send_command 11 2024-04-14T15:19:19.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.352+0000 7fbcd435a700 1 -- 172.21.15.90:0/3122204600 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fbcb801cf40 con 0x7fbcb801ca90 2024-04-14T15:19:19.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.352+0000 7fbccb7fe700 10 client.37377.objecter ms_handle_connect 0x7fbcb801ca90 2024-04-14T15:19:19.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.353+0000 7fbccb7fe700 1 -- 172.21.15.90:0/3122204600 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fbcb801cf40 con 0x7fbcb801ca90 2024-04-14T15:19:19.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.353+0000 7fbccb7fe700 10 client.37377.objecter ms_dispatch 0x7fbccc001550 command_reply(tid 11: 0 ) 2024-04-14T15:19:19.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.353+0000 7fbccb7fe700 10 client.37377.objecter _finish_command 11 = system:0 2024-04-14T15:19:19.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.353+0000 7fbccb7fe700 20 client.37377.objecter put_session s=0x7fbcb801c370 osd=5 4 2024-04-14T15:19:19.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.353+0000 7fbccb7fe700 15 client.37377.objecter _session_command_op_remove 5 11 2024-04-14T15:19:19.360 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:19:19.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.369+0000 7fbcd435a700 10 client.37377.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:19:19.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.369+0000 7fbcd435a700 15 client.37377.objecter _session_command_op_assign -1 12 2024-04-14T15:19:19.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.369+0000 7fbcd435a700 20 client.37377.objecter _get_session s=0x7fbcb801c370 osd=5 3 2024-04-14T15:19:19.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.369+0000 7fbcd435a700 20 client.37377.objecter put_session s=0x7fbcb801c370 osd=5 3 2024-04-14T15:19:19.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.369+0000 7fbcd435a700 20 client.37377.objecter _get_session s=0x7fbcb801c370 osd=5 3 2024-04-14T15:19:19.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.369+0000 7fbcd435a700 15 client.37377.objecter _session_command_op_remove -1 12 2024-04-14T15:19:19.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.369+0000 7fbcd435a700 20 client.37377.objecter get_session s=0x7fbcb801c370 osd=5 3 2024-04-14T15:19:19.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.369+0000 7fbcd435a700 15 client.37377.objecter _session_command_op_assign 5 12 2024-04-14T15:19:19.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.369+0000 7fbcd435a700 20 client.37377.objecter put_session s=0x7fbcb801c370 osd=5 4 2024-04-14T15:19:19.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.369+0000 7fbcd435a700 10 client.37377.objecter _send_command 12 2024-04-14T15:19:19.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.369+0000 7fbcd435a700 1 -- 172.21.15.90:0/3122204600 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fbcb801d1a0 con 0x7fbcb801ca90 2024-04-14T15:19:19.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.371+0000 7fbccb7fe700 1 -- 172.21.15.90:0/3122204600 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fbcb801d1a0 con 0x7fbcb801ca90 2024-04-14T15:19:19.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.371+0000 7fbccb7fe700 10 client.37377.objecter ms_dispatch 0x7fbccc001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:19:19.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.371+0000 7fbccb7fe700 10 client.37377.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:19:19.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.371+0000 7fbccb7fe700 20 client.37377.objecter put_session s=0x7fbcb801c370 osd=5 4 2024-04-14T15:19:19.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.371+0000 7fbccb7fe700 15 client.37377.objecter _session_command_op_remove 5 12 2024-04-14T15:19:19.372 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:19:19.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.371+0000 7fbcd435a700 10 client.37377.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:19:19.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.371+0000 7fbcd435a700 15 client.37377.objecter _session_command_op_assign -1 13 2024-04-14T15:19:19.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.371+0000 7fbcd435a700 20 client.37377.objecter _get_session s=0x7fbcb801cfa0 osd=6 3 2024-04-14T15:19:19.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.371+0000 7fbcd435a700 20 client.37377.objecter put_session s=0x7fbcb801cfa0 osd=6 3 2024-04-14T15:19:19.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.371+0000 7fbcd435a700 20 client.37377.objecter _get_session s=0x7fbcb801cfa0 osd=6 3 2024-04-14T15:19:19.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.371+0000 7fbcd435a700 15 client.37377.objecter _session_command_op_remove -1 13 2024-04-14T15:19:19.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.371+0000 7fbcd435a700 20 client.37377.objecter get_session s=0x7fbcb801cfa0 osd=6 3 2024-04-14T15:19:19.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.371+0000 7fbcd435a700 15 client.37377.objecter _session_command_op_assign 6 13 2024-04-14T15:19:19.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.371+0000 7fbcd435a700 20 client.37377.objecter put_session s=0x7fbcb801cfa0 osd=6 4 2024-04-14T15:19:19.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.371+0000 7fbcd435a700 10 client.37377.objecter _send_command 13 2024-04-14T15:19:19.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.371+0000 7fbcd435a700 1 -- 172.21.15.90:0/3122204600 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fbcb8023590 con 0x7fbcb801fa80 2024-04-14T15:19:19.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.372+0000 7fbccb7fe700 10 client.37377.objecter ms_handle_connect 0x7fbcb801fa80 2024-04-14T15:19:19.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:19 smithi092 ceph-mon[33966]: 4.1d scrub starts 2024-04-14T15:19:19.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:19 smithi092 ceph-mon[33966]: 4.1d scrub ok 2024-04-14T15:19:19.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:19 smithi092 ceph-mon[33966]: 4.13 deep-scrub starts 2024-04-14T15:19:19.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:19 smithi092 ceph-mon[33966]: 4.13 deep-scrub ok 2024-04-14T15:19:19.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:19 smithi092 ceph-mon[33966]: 2.1f scrub starts 2024-04-14T15:19:19.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:19 smithi092 ceph-mon[33966]: 2.1f scrub ok 2024-04-14T15:19:19.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.373+0000 7fbccb7fe700 1 -- 172.21.15.90:0/3122204600 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fbcb8023590 con 0x7fbcb801fa80 2024-04-14T15:19:19.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.373+0000 7fbccb7fe700 10 client.37377.objecter ms_dispatch 0x7fbccc001550 command_reply(tid 13: 0 ) 2024-04-14T15:19:19.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.373+0000 7fbccb7fe700 10 client.37377.objecter _finish_command 13 = system:0 2024-04-14T15:19:19.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.373+0000 7fbccb7fe700 20 client.37377.objecter put_session s=0x7fbcb801cfa0 osd=6 4 2024-04-14T15:19:19.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.373+0000 7fbccb7fe700 15 client.37377.objecter _session_command_op_remove 6 13 2024-04-14T15:19:19.385 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:19.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:19.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:19:19.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.389+0000 7fbcd435a700 10 client.37377.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:19:19.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.389+0000 7fbcd435a700 15 client.37377.objecter _session_command_op_assign -1 14 2024-04-14T15:19:19.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.389+0000 7fbcd435a700 20 client.37377.objecter _get_session s=0x7fbcb801cfa0 osd=6 3 2024-04-14T15:19:19.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.389+0000 7fbcd435a700 20 client.37377.objecter put_session s=0x7fbcb801cfa0 osd=6 3 2024-04-14T15:19:19.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.389+0000 7fbcd435a700 20 client.37377.objecter _get_session s=0x7fbcb801cfa0 osd=6 3 2024-04-14T15:19:19.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.389+0000 7fbcd435a700 15 client.37377.objecter _session_command_op_remove -1 14 2024-04-14T15:19:19.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.389+0000 7fbcd435a700 20 client.37377.objecter get_session s=0x7fbcb801cfa0 osd=6 3 2024-04-14T15:19:19.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.389+0000 7fbcd435a700 15 client.37377.objecter _session_command_op_assign 6 14 2024-04-14T15:19:19.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.389+0000 7fbcd435a700 20 client.37377.objecter put_session s=0x7fbcb801cfa0 osd=6 4 2024-04-14T15:19:19.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.389+0000 7fbcd435a700 10 client.37377.objecter _send_command 14 2024-04-14T15:19:19.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.389+0000 7fbcd435a700 1 -- 172.21.15.90:0/3122204600 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fbcb8023590 con 0x7fbcb801fa80 2024-04-14T15:19:19.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.391+0000 7fbccb7fe700 1 -- 172.21.15.90:0/3122204600 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fbcb8023590 con 0x7fbcb801fa80 2024-04-14T15:19:19.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.391+0000 7fbccb7fe700 10 client.37377.objecter ms_dispatch 0x7fbccc001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:19:19.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.391+0000 7fbccb7fe700 10 client.37377.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:19:19.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.391+0000 7fbccb7fe700 20 client.37377.objecter put_session s=0x7fbcb801cfa0 osd=6 4 2024-04-14T15:19:19.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.391+0000 7fbccb7fe700 15 client.37377.objecter _session_command_op_remove 6 14 2024-04-14T15:19:19.392 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:19:19.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.391+0000 7fbcd435a700 10 client.37377.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:19:19.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.391+0000 7fbcd435a700 15 client.37377.objecter _session_command_op_assign -1 15 2024-04-14T15:19:19.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.391+0000 7fbcd435a700 20 client.37377.objecter _get_session s=0x7fbcb8023590 osd=7 3 2024-04-14T15:19:19.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.391+0000 7fbcd435a700 20 client.37377.objecter put_session s=0x7fbcb8023590 osd=7 3 2024-04-14T15:19:19.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.391+0000 7fbcd435a700 20 client.37377.objecter _get_session s=0x7fbcb8023590 osd=7 3 2024-04-14T15:19:19.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.391+0000 7fbcd435a700 15 client.37377.objecter _session_command_op_remove -1 15 2024-04-14T15:19:19.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.391+0000 7fbcd435a700 20 client.37377.objecter get_session s=0x7fbcb8023590 osd=7 3 2024-04-14T15:19:19.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.391+0000 7fbcd435a700 15 client.37377.objecter _session_command_op_assign 7 15 2024-04-14T15:19:19.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.391+0000 7fbcd435a700 20 client.37377.objecter put_session s=0x7fbcb8023590 osd=7 4 2024-04-14T15:19:19.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.391+0000 7fbcd435a700 10 client.37377.objecter _send_command 15 2024-04-14T15:19:19.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.391+0000 7fbcd435a700 1 -- 172.21.15.90:0/3122204600 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fbcb8027830 con 0x7fbcb8023c10 2024-04-14T15:19:19.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.391+0000 7fbccb7fe700 10 client.37377.objecter ms_handle_connect 0x7fbcb8023c10 2024-04-14T15:19:19.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.392+0000 7fbccb7fe700 1 -- 172.21.15.90:0/3122204600 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fbcb8027830 con 0x7fbcb8023c10 2024-04-14T15:19:19.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.392+0000 7fbccb7fe700 10 client.37377.objecter ms_dispatch 0x7fbccc001550 command_reply(tid 15: 0 ) 2024-04-14T15:19:19.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.392+0000 7fbccb7fe700 10 client.37377.objecter _finish_command 15 = system:0 2024-04-14T15:19:19.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.392+0000 7fbccb7fe700 20 client.37377.objecter put_session s=0x7fbcb8023590 osd=7 4 2024-04-14T15:19:19.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.392+0000 7fbccb7fe700 15 client.37377.objecter _session_command_op_remove 7 15 2024-04-14T15:19:19.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.409+0000 7fbcd435a700 10 client.37377.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:19:19.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.409+0000 7fbcd435a700 15 client.37377.objecter _session_command_op_assign -1 16 2024-04-14T15:19:19.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.409+0000 7fbcd435a700 20 client.37377.objecter _get_session s=0x7fbcb8023590 osd=7 3 2024-04-14T15:19:19.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.409+0000 7fbcd435a700 20 client.37377.objecter put_session s=0x7fbcb8023590 osd=7 3 2024-04-14T15:19:19.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.409+0000 7fbcd435a700 20 client.37377.objecter _get_session s=0x7fbcb8023590 osd=7 3 2024-04-14T15:19:19.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.409+0000 7fbcd435a700 15 client.37377.objecter _session_command_op_remove -1 16 2024-04-14T15:19:19.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.409+0000 7fbcd435a700 20 client.37377.objecter get_session s=0x7fbcb8023590 osd=7 3 2024-04-14T15:19:19.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.409+0000 7fbcd435a700 15 client.37377.objecter _session_command_op_assign 7 16 2024-04-14T15:19:19.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.409+0000 7fbcd435a700 20 client.37377.objecter put_session s=0x7fbcb8023590 osd=7 4 2024-04-14T15:19:19.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.409+0000 7fbcd435a700 10 client.37377.objecter _send_command 16 2024-04-14T15:19:19.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.409+0000 7fbcd435a700 1 -- 172.21.15.90:0/3122204600 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fbcb80291e0 con 0x7fbcb8023c10 2024-04-14T15:19:19.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.410+0000 7fbccb7fe700 1 -- 172.21.15.90:0/3122204600 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fbcb80291e0 con 0x7fbcb8023c10 2024-04-14T15:19:19.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.410+0000 7fbccb7fe700 10 client.37377.objecter ms_dispatch 0x7fbccc001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:19:19.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.410+0000 7fbccb7fe700 10 client.37377.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:19:19.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.410+0000 7fbccb7fe700 20 client.37377.objecter put_session s=0x7fbcb8023590 osd=7 4 2024-04-14T15:19:19.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.410+0000 7fbccb7fe700 15 client.37377.objecter _session_command_op_remove 7 16 2024-04-14T15:19:19.411 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:19:19.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.411+0000 7fbcd435a700 10 librados: watch_flush enter 2024-04-14T15:19:19.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.411+0000 7fbcd435a700 10 librados: watch_flush exit 2024-04-14T15:19:19.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.411+0000 7fbcd435a700 10 client.37377.objecter close_session for osd.0 2024-04-14T15:19:19.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.411+0000 7fbcd435a700 1 -- 172.21.15.90:0/3122204600 >> v1:172.21.15.90:6801/3761806885 conn(0x7fbcb8004730 legacy=0x7fbcb8002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:19.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.411+0000 7fbcd435a700 20 client.37377.objecter put_session s=0x7fbcb80053e0 osd=0 1 2024-04-14T15:19:19.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.411+0000 7fbcd435a700 10 client.37377.objecter close_session for osd.1 2024-04-14T15:19:19.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.411+0000 7fbcd435a700 1 -- 172.21.15.90:0/3122204600 >> v1:172.21.15.90:6805/4170361699 conn(0x7fbcb8004280 legacy=0x7fbcb800ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:19.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.411+0000 7fbcd435a700 20 client.37377.objecter put_session s=0x7fbcb8004110 osd=1 1 2024-04-14T15:19:19.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.411+0000 7fbcd435a700 10 client.37377.objecter close_session for osd.2 2024-04-14T15:19:19.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.411+0000 7fbcd435a700 1 -- 172.21.15.90:0/3122204600 >> v1:172.21.15.90:6809/1686184251 conn(0x7fbcb800d5d0 legacy=0x7fbcb800f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:19.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.411+0000 7fbcd435a700 20 client.37377.objecter put_session s=0x7fbcb8008370 osd=2 1 2024-04-14T15:19:19.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.411+0000 7fbcd435a700 10 client.37377.objecter close_session for osd.3 2024-04-14T15:19:19.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.411+0000 7fbcd435a700 1 -- 172.21.15.90:0/3122204600 >> v1:172.21.15.90:6813/2270166774 conn(0x7fbcb8012410 legacy=0x7fbcb8014800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:19.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.411+0000 7fbcd435a700 20 client.37377.objecter put_session s=0x7fbcb800cc20 osd=3 1 2024-04-14T15:19:19.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.411+0000 7fbcd435a700 10 client.37377.objecter close_session for osd.4 2024-04-14T15:19:19.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.411+0000 7fbcd435a700 1 -- 172.21.15.90:0/3122204600 >> v1:172.21.15.92:6800/3887549733 conn(0x7fbcb8011f70 legacy=0x7fbcb8019260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:19.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.411+0000 7fbcd435a700 20 client.37377.objecter put_session s=0x7fbcb800c700 osd=4 1 2024-04-14T15:19:19.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.411+0000 7fbcd435a700 10 client.37377.objecter close_session for osd.5 2024-04-14T15:19:19.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.411+0000 7fbcd435a700 1 -- 172.21.15.90:0/3122204600 >> v1:172.21.15.92:6804/3244561440 conn(0x7fbcb801ca90 legacy=0x7fbcb801aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:19.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.411+0000 7fbcd435a700 20 client.37377.objecter put_session s=0x7fbcb801c370 osd=5 1 2024-04-14T15:19:19.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.411+0000 7fbcd435a700 10 client.37377.objecter close_session for osd.6 2024-04-14T15:19:19.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.411+0000 7fbcd435a700 1 -- 172.21.15.90:0/3122204600 >> v1:172.21.15.92:6808/636125963 conn(0x7fbcb801fa80 legacy=0x7fbcb8021e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:19.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.411+0000 7fbcd435a700 20 client.37377.objecter put_session s=0x7fbcb801cfa0 osd=6 1 2024-04-14T15:19:19.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.411+0000 7fbcd435a700 10 client.37377.objecter close_session for osd.7 2024-04-14T15:19:19.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.411+0000 7fbcd435a700 1 -- 172.21.15.90:0/3122204600 >> v1:172.21.15.92:6812/1773706716 conn(0x7fbcb8023c10 legacy=0x7fbcb8026000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:19.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.411+0000 7fbcd435a700 20 client.37377.objecter put_session s=0x7fbcb8023590 osd=7 1 2024-04-14T15:19:19.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.411+0000 7fbcd435a700 20 client.37377.objecter shutdown clearing up homeless session... 2024-04-14T15:19:19.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.411+0000 7fbcd435a700 10 client.37377.objecter successfully canceled tick 2024-04-14T15:19:19.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.411+0000 7fbcd435a700 1 -- 172.21.15.90:0/3122204600 >> v1:172.21.15.90:6800/420935427 conn(0x7fbcb4071380 legacy=0x7fbcb4073840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:19.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.411+0000 7fbcd435a700 1 -- 172.21.15.90:0/3122204600 >> v1:172.21.15.90:6789/0 conn(0x7fbccc10b720 legacy=0x7fbccc0753f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:19.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.411+0000 7fbcd28f7700 1 -- 172.21.15.90:0/3122204600 reap_dead start 2024-04-14T15:19:19.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.411+0000 7fbcd28f7700 1 -- 172.21.15.90:0/3122204600 reap_dead start 2024-04-14T15:19:19.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.411+0000 7fbcd28f7700 1 -- 172.21.15.90:0/3122204600 reap_dead start 2024-04-14T15:19:19.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.411+0000 7fbcd28f7700 1 -- 172.21.15.90:0/3122204600 reap_dead start 2024-04-14T15:19:19.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.411+0000 7fbcd28f7700 1 -- 172.21.15.90:0/3122204600 reap_dead start 2024-04-14T15:19:19.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.411+0000 7fbcd28f7700 1 -- 172.21.15.90:0/3122204600 reap_dead start 2024-04-14T15:19:19.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.411+0000 7fbcd435a700 1 -- 172.21.15.90:0/3122204600 shutdown_connections 2024-04-14T15:19:19.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.411+0000 7fbcd435a700 1 -- 172.21.15.90:0/3122204600 >> 172.21.15.90:0/3122204600 conn(0x7fbccc0fe3a0 msgr2=0x7fbccc112870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:19:19.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.411+0000 7fbcd435a700 1 -- 172.21.15.90:0/3122204600 shutdown_connections 2024-04-14T15:19:19.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.411+0000 7fbcd435a700 1 -- 172.21.15.90:0/3122204600 wait complete. 2024-04-14T15:19:19.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:19.411+0000 7fbcd435a700 1 librados: shutdown 2024-04-14T15:19:19.416 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:19:19.416 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:19:19.416 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:19:19.416 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:19:19.416 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:19:19.416 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:19:19.416 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:19:19.416 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:19:19.487 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:19:19.511 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:19.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:19.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:19:19.613 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:19.637 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:19.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:19.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:19.738 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:19:19.763 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:19.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:19.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:19:19.865 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:19.889 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:19.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:19.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:19.991 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:19:20.014 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:20.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:20.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:19:20.116 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:19:20.141 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:20.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:20.142 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:19:20.243 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:19:20.267 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:20.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:20.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:19:20.368 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:19:20.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:20 smithi092 ceph-mon[33966]: 6.e deep-scrub starts 2024-04-14T15:19:20.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:20 smithi092 ceph-mon[33966]: 6.e deep-scrub ok 2024-04-14T15:19:20.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:20 smithi092 ceph-mon[33966]: pgmap v656: 258 pgs: 4 creating+activating, 254 active+clean; 8.4 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 B/s wr, 3 op/s 2024-04-14T15:19:20.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:20 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3122204600' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:19:20.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:20.395 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:20.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:19:20.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:20 smithi090 ceph-mon[41670]: 6.e deep-scrub starts 2024-04-14T15:19:20.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:20 smithi090 ceph-mon[41670]: 6.e deep-scrub ok 2024-04-14T15:19:20.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:20 smithi090 ceph-mon[41670]: pgmap v656: 258 pgs: 4 creating+activating, 254 active+clean; 8.4 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 B/s wr, 3 op/s 2024-04-14T15:19:20.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:20 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3122204600' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:19:20.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:20 smithi090 ceph-mon[30399]: 6.e deep-scrub starts 2024-04-14T15:19:20.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:20 smithi090 ceph-mon[30399]: 6.e deep-scrub ok 2024-04-14T15:19:20.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:20 smithi090 ceph-mon[30399]: pgmap v656: 258 pgs: 4 creating+activating, 254 active+clean; 8.4 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 B/s wr, 3 op/s 2024-04-14T15:19:20.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:20 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3122204600' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:19:20.496 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:20.521 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:20.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:20.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:20.623 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:19:20.647 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:20.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:20.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:19:20.749 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:19:20.772 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:20.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:20.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:19:20.874 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:20.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:20.904 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:20.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:21.005 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:19:21.029 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:21.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:21.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:19:21.131 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:19:21.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:21.155 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:21.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:19:21.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:21 smithi090 ceph-mon[41670]: 6.1f scrub starts 2024-04-14T15:19:21.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:21 smithi090 ceph-mon[41670]: 6.1f scrub ok 2024-04-14T15:19:21.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:21 smithi090 ceph-mon[41670]: 5.e scrub starts 2024-04-14T15:19:21.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:21 smithi090 ceph-mon[41670]: 5.e scrub ok 2024-04-14T15:19:21.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:21 smithi090 ceph-mon[41670]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:19:21.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:21 smithi090 ceph-mon[30399]: 6.1f scrub starts 2024-04-14T15:19:21.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:21 smithi090 ceph-mon[30399]: 6.1f scrub ok 2024-04-14T15:19:21.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:21 smithi090 ceph-mon[30399]: 5.e scrub starts 2024-04-14T15:19:21.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:21 smithi090 ceph-mon[30399]: 5.e scrub ok 2024-04-14T15:19:21.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:21 smithi090 ceph-mon[30399]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:19:21.257 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:19:21.285 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:21.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:21.286 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:19:21.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:21 smithi092 ceph-mon[33966]: 6.1f scrub starts 2024-04-14T15:19:21.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:21 smithi092 ceph-mon[33966]: 6.1f scrub ok 2024-04-14T15:19:21.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:21 smithi092 ceph-mon[33966]: 5.e scrub starts 2024-04-14T15:19:21.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:21 smithi092 ceph-mon[33966]: 5.e scrub ok 2024-04-14T15:19:21.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:21 smithi092 ceph-mon[33966]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:19:21.387 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:19:21.414 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:21.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:21.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:19:21.516 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:19:21.541 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:21.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:21.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:19:21.643 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:19:21.670 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:21.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:21.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:19:21.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:21.559+0000 7fc1c0e5b700 1 -- 172.21.15.90:0/1464548110 >> v1:172.21.15.90:6789/0 conn(0x7fc1bc10f310 legacy=0x7fc1bc1117d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:21.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:21.559+0000 7fc1c0e5b700 1 -- 172.21.15.90:0/1464548110 shutdown_connections 2024-04-14T15:19:21.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:21.559+0000 7fc1c0e5b700 1 -- 172.21.15.90:0/1464548110 >> 172.21.15.90:0/1464548110 conn(0x7fc1bc0fe380 msgr2=0x7fc1bc1007a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:19:21.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:21.559+0000 7fc1c0e5b700 1 -- 172.21.15.90:0/1464548110 shutdown_connections 2024-04-14T15:19:21.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:21.559+0000 7fc1c0e5b700 1 -- 172.21.15.90:0/1464548110 wait complete. 2024-04-14T15:19:21.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:21.560+0000 7fc1c0e5b700 1 librados: starting msgr at 2024-04-14T15:19:21.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:21.560+0000 7fc1c0e5b700 1 librados: starting objecter 2024-04-14T15:19:21.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:21.560+0000 7fc1c0e5b700 1 Processor -- start 2024-04-14T15:19:21.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:21.560+0000 7fc1c0e5b700 1 -- start start 2024-04-14T15:19:21.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:21.560+0000 7fc1c0e5b700 1 librados: setting wanted keys 2024-04-14T15:19:21.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:21.560+0000 7fc1c0e5b700 1 librados: calling monclient init 2024-04-14T15:19:21.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:21.560+0000 7fc1c0e5b700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc1bc076a90 con 0x7fc1bc10f310 2024-04-14T15:19:21.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:21.560+0000 7fc1c0e5b700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc1bc076ca0 con 0x7fc1bc108890 2024-04-14T15:19:21.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:21.560+0000 7fc1c0e5b700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc1bc1ac320 con 0x7fc1bc10b740 2024-04-14T15:19:21.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:21.560+0000 7fc1b9d9b700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7fc1bc10b740 0x7fc1bc0753f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:41478/0 (socket says 172.21.15.90:41478) 2024-04-14T15:19:21.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:21.560+0000 7fc1bad9d700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fc1bc10f310 0x7fc1bc1a8be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:37110/0 (socket says 172.21.15.90:37110) 2024-04-14T15:19:21.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:21.560+0000 7fc1b9d9b700 1 -- 172.21.15.90:0/2779362139 learned_addr learned my addr 172.21.15.90:0/2779362139 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:19:21.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:21.560+0000 7fc1ab7fe700 10 client.?.objecter ms_handle_connect 0x7fc1bc10f310 2024-04-14T15:19:21.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:21.560+0000 7fc1ab7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:19:21.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:21.560+0000 7fc1ab7fe700 10 client.?.objecter ms_handle_connect 0x7fc1bc10b740 2024-04-14T15:19:21.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:21.560+0000 7fc1ab7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:19:21.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:21.561+0000 7fc1ab7fe700 1 -- 172.21.15.90:0/2779362139 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2164232181 0 0) 0x7fc1bc076a90 con 0x7fc1bc10f310 2024-04-14T15:19:21.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:21.561+0000 7fc1ab7fe700 1 -- 172.21.15.90:0/2779362139 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc1a40031f0 con 0x7fc1bc10f310 2024-04-14T15:19:21.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:21.561+0000 7fc1ab7fe700 1 -- 172.21.15.90:0/2779362139 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 115706263 0 0) 0x7fc1bc1ac320 con 0x7fc1bc10b740 2024-04-14T15:19:21.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:21.561+0000 7fc1ab7fe700 1 -- 172.21.15.90:0/2779362139 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc1bc076a90 con 0x7fc1bc10b740 2024-04-14T15:19:21.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:21.561+0000 7fc1ab7fe700 10 client.?.objecter ms_handle_connect 0x7fc1bc108890 2024-04-14T15:19:21.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:21.561+0000 7fc1ab7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:19:21.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:21.561+0000 7fc1ab7fe700 1 -- 172.21.15.90:0/2779362139 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4009820062 0 0) 0x7fc1bc076ca0 con 0x7fc1bc108890 2024-04-14T15:19:21.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:21.561+0000 7fc1ab7fe700 1 -- 172.21.15.90:0/2779362139 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc1bc1ac320 con 0x7fc1bc108890 2024-04-14T15:19:21.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:21.561+0000 7fc1ab7fe700 1 -- 172.21.15.90:0/2779362139 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 447688062 0 0) 0x7fc1a40031f0 con 0x7fc1bc10f310 2024-04-14T15:19:21.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:21.561+0000 7fc1ab7fe700 1 -- 172.21.15.90:0/2779362139 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc1bc076ca0 con 0x7fc1bc10f310 2024-04-14T15:19:21.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:21.561+0000 7fc1ab7fe700 1 -- 172.21.15.90:0/2779362139 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fc1b40017b0 con 0x7fc1bc10f310 2024-04-14T15:19:21.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:21.561+0000 7fc1ab7fe700 1 -- 172.21.15.90:0/2779362139 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 251148896 0 0) 0x7fc1bc076a90 con 0x7fc1bc10b740 2024-04-14T15:19:21.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:21.561+0000 7fc1ab7fe700 1 -- 172.21.15.90:0/2779362139 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc1a40031f0 con 0x7fc1bc10b740 2024-04-14T15:19:21.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:21.561+0000 7fc1ab7fe700 1 -- 172.21.15.90:0/2779362139 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fc1ac001910 con 0x7fc1bc10b740 2024-04-14T15:19:21.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:21.561+0000 7fc1ab7fe700 1 -- 172.21.15.90:0/2779362139 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2238974541 0 0) 0x7fc1bc076ca0 con 0x7fc1bc10f310 2024-04-14T15:19:21.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:21.561+0000 7fc1ab7fe700 1 -- 172.21.15.90:0/2779362139 >> v1:172.21.15.90:6790/0 conn(0x7fc1bc10b740 legacy=0x7fc1bc0753f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:21.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:21.561+0000 7fc1ab7fe700 1 -- 172.21.15.90:0/2779362139 >> v1:172.21.15.92:6789/0 conn(0x7fc1bc108890 legacy=0x7fc1bc074ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:21.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:21.561+0000 7fc1ab7fe700 1 -- 172.21.15.90:0/2779362139 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1bc1ad540 con 0x7fc1bc10f310 2024-04-14T15:19:21.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:21.561+0000 7fc1c0e5b700 1 -- 172.21.15.90:0/2779362139 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc1bc1ac550 con 0x7fc1bc10f310 2024-04-14T15:19:21.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:21.561+0000 7fc1c0e5b700 10 client.37383.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:19:21.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:21.561+0000 7fc1c0e5b700 1 -- 172.21.15.90:0/2779362139 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc1bc1aca10 con 0x7fc1bc10f310 2024-04-14T15:19:21.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:21.561+0000 7fc1c0e5b700 1 librados: init done 2024-04-14T15:19:21.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:21.561+0000 7fc1ab7fe700 1 -- 172.21.15.90:0/2779362139 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fc1b4003ac0 con 0x7fc1bc10f310 2024-04-14T15:19:21.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:21.561+0000 7fc1ab7fe700 1 -- 172.21.15.90:0/2779362139 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fc1b4005130 con 0x7fc1bc10f310 2024-04-14T15:19:21.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:21.562+0000 7fc1c0e5b700 1 -- 172.21.15.90:0/2779362139 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc19c005150 con 0x7fc1bc10f310 2024-04-14T15:19:21.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:21.563+0000 7fc1ab7fe700 1 -- 172.21.15.90:0/2779362139 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fc1b4003dd0 con 0x7fc1bc10f310 2024-04-14T15:19:21.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:21.563+0000 7fc1ab7fe700 1 -- 172.21.15.90:0/2779362139 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(464..464 src has 51..464) ==== 8160+0+0 (unknown 3980499053 0 0) 0x7fc1b408ace0 con 0x7fc1bc10f310 2024-04-14T15:19:21.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:21.563+0000 7fc1ab7fe700 10 client.37383.objecter ms_dispatch 0x7fc1bc001550 osd_map(464..464 src has 51..464) 2024-04-14T15:19:21.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:21.563+0000 7fc1ab7fe700 3 client.37383.objecter handle_osd_map got epochs [464,464] > 0 2024-04-14T15:19:21.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:21.563+0000 7fc1ab7fe700 3 client.37383.objecter handle_osd_map decoding full epoch 464 2024-04-14T15:19:21.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:21.563+0000 7fc1ab7fe700 20 client.37383.objecter dump_active .. 0 homeless 2024-04-14T15:19:21.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:21.563+0000 7fc1ab7fe700 10 client.37383.objecter ms_handle_connect 0x7fc1a4071050 2024-04-14T15:19:21.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:21.568+0000 7fc1ab7fe700 1 -- 172.21.15.90:0/2779362139 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fc1b4057700 con 0x7fc1bc10f310 2024-04-14T15:19:21.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:21.762+0000 7fc1c0e5b700 1 -- 172.21.15.90:0/2779362139 --> v1:172.21.15.90:6800/420935427 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc19c000bc0 con 0x7fc1a4071050 2024-04-14T15:19:21.772 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:19:21.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:21.779+0000 7fc1ab7fe700 1 -- 172.21.15.90:0/2779362139 <== mgr.14610 v1:172.21.15.90:6800/420935427 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+659917 (unknown 2965378022 0 3487502934) 0x7fc19c000bc0 con 0x7fc1a4071050 2024-04-14T15:19:21.788 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:19:21.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:21.785+0000 7fc1c0e5b700 10 librados: watch_flush enter 2024-04-14T15:19:21.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:21.785+0000 7fc1c0e5b700 10 librados: watch_flush exit 2024-04-14T15:19:21.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:21.785+0000 7fc1c0e5b700 20 client.37383.objecter shutdown clearing up homeless session... 2024-04-14T15:19:21.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:21.785+0000 7fc1c0e5b700 10 client.37383.objecter successfully canceled tick 2024-04-14T15:19:21.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:21.785+0000 7fc1c0e5b700 1 -- 172.21.15.90:0/2779362139 >> v1:172.21.15.90:6800/420935427 conn(0x7fc1a4071050 legacy=0x7fc1a4073510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:21.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:21.785+0000 7fc1c0e5b700 1 -- 172.21.15.90:0/2779362139 >> v1:172.21.15.90:6789/0 conn(0x7fc1bc10f310 legacy=0x7fc1bc1a8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:21.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:21.785+0000 7fc1c0e5b700 1 -- 172.21.15.90:0/2779362139 shutdown_connections 2024-04-14T15:19:21.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:21.785+0000 7fc1c0e5b700 1 -- 172.21.15.90:0/2779362139 >> 172.21.15.90:0/2779362139 conn(0x7fc1bc0fe380 msgr2=0x7fc1bc1128b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:19:21.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:21.785+0000 7fc1c0e5b700 1 -- 172.21.15.90:0/2779362139 shutdown_connections 2024-04-14T15:19:21.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:21.785+0000 7fc1c0e5b700 1 -- 172.21.15.90:0/2779362139 wait complete. 2024-04-14T15:19:21.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:21.785+0000 7fc1c0e5b700 1 librados: shutdown 2024-04-14T15:19:21.796 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-14T15:19:21.800 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:21.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:21.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:19:21.902 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:19:21.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:21.926 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:21.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:19:22.027 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:19:22.053 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:22.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:22.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:19:22.154 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:19:22.178 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:22.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:22.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:19:22.273 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:22 smithi092 ceph-mon[33966]: pgmap v657: 258 pgs: 258 active+clean; 8.4 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 721 B/s rd, 0 B/s wr, 4 op/s 2024-04-14T15:19:22.273 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:22 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:19:22.274 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:22 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:19:22.274 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:22 smithi092 ceph-mon[33966]: 5.1e scrub starts 2024-04-14T15:19:22.274 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:22 smithi092 ceph-mon[33966]: 5.1e scrub ok 2024-04-14T15:19:22.274 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:22 smithi092 ceph-mon[33966]: from='client.37383 v1:172.21.15.90:0/2779362139' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:19:22.280 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:19:22.311 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:22.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:22.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:19:22.413 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:19:22.436 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:22.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:22.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:19:22.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:22 smithi090 ceph-mon[41670]: pgmap v657: 258 pgs: 258 active+clean; 8.4 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 721 B/s rd, 0 B/s wr, 4 op/s 2024-04-14T15:19:22.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:22 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:19:22.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:22 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:19:22.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:22 smithi090 ceph-mon[41670]: 5.1e scrub starts 2024-04-14T15:19:22.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:22 smithi090 ceph-mon[41670]: 5.1e scrub ok 2024-04-14T15:19:22.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:22 smithi090 ceph-mon[41670]: from='client.37383 v1:172.21.15.90:0/2779362139' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:19:22.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:22 smithi090 ceph-mon[30399]: pgmap v657: 258 pgs: 258 active+clean; 8.4 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 721 B/s rd, 0 B/s wr, 4 op/s 2024-04-14T15:19:22.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:22 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:19:22.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:22 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:19:22.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:22 smithi090 ceph-mon[30399]: 5.1e scrub starts 2024-04-14T15:19:22.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:22 smithi090 ceph-mon[30399]: 5.1e scrub ok 2024-04-14T15:19:22.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:22 smithi090 ceph-mon[30399]: from='client.37383 v1:172.21.15.90:0/2779362139' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:19:22.539 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:19:22.563 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:22.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:22.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:19:22.664 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:19:22.689 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:22.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:22.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:19:22.765 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":657,"stamp":"2024-04-14T15:19:21.092245+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8792244,"num_objects":372,"num_object_clones":5,"num_object_copies":1116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":372,"num_whiteouts":0,"num_read":1410,"num_read_kb":1057,"num_write":1071,"num_write_kb":12763,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":13,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6560,"ondisk_log_size":6560,"up":774,"acting":774,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":767,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":606408,"kb_used_data":87604,"kb_used_omap":12,"kb_used_meta":518643,"kb_avail":749354808,"statfs":{"total":767960285184,"available":767339323392,"internally_reserved":0,"allocated":89706496,"data_stored":61981712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":531090994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13382,"num_objects":10,"num_object_clones":1,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":12,"num_read_kb":5,"num_write":18,"num_write_kb":-39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":-40252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.099653"},"pg_stats":[{"pgid":"301.1f","version":"0'0","reported_seq":15,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.914856+0000","last_change":"2024-04-14T15:19:14.012160+0000","last_active":"2024-04-14T15:19:15.914856+0000","last_peered":"2024-04-14T15:19:15.914856+0000","last_clean":"2024-04-14T15:19:15.914856+0000","last_became_active":"2024-04-14T15:19:14.011848+0000","last_became_peered":"2024-04-14T15:19:14.011848+0000","last_unstale":"2024-04-14T15:19:15.914856+0000","last_undegraded":"2024-04-14T15:19:15.914856+0000","last_fullsized":"2024-04-14T15:19:15.914856+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:12.984089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:12.984089+0000","last_clean_scrub_stamp":"2024-04-14T15:19:12.984089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:32.024770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"301.1e","version":"0'0","reported_seq":15,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.917785+0000","last_change":"2024-04-14T15:19:14.021047+0000","last_active":"2024-04-14T15:19:15.917785+0000","last_peered":"2024-04-14T15:19:15.917785+0000","last_clean":"2024-04-14T15:19:15.917785+0000","last_became_active":"2024-04-14T15:19:14.020492+0000","last_became_peered":"2024-04-14T15:19:14.020492+0000","last_unstale":"2024-04-14T15:19:15.917785+0000","last_undegraded":"2024-04-14T15:19:15.917785+0000","last_fullsized":"2024-04-14T15:19:15.917785+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:12.984089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:12.984089+0000","last_clean_scrub_stamp":"2024-04-14T15:19:12.984089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:15.991129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"301.1d","version":"0'0","reported_seq":15,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.936522+0000","last_change":"2024-04-14T15:19:14.014824+0000","last_active":"2024-04-14T15:19:15.936522+0000","last_peered":"2024-04-14T15:19:15.936522+0000","last_clean":"2024-04-14T15:19:15.936522+0000","last_became_active":"2024-04-14T15:19:14.014517+0000","last_became_peered":"2024-04-14T15:19:14.014517+0000","last_unstale":"2024-04-14T15:19:15.936522+0000","last_undegraded":"2024-04-14T15:19:15.936522+0000","last_fullsized":"2024-04-14T15:19:15.936522+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:12.984089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:12.984089+0000","last_clean_scrub_stamp":"2024-04-14T15:19:12.984089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:22.261740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"301.1c","version":"0'0","reported_seq":15,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.936267+0000","last_change":"2024-04-14T15:19:14.015201+0000","last_active":"2024-04-14T15:19:15.936267+0000","last_peered":"2024-04-14T15:19:15.936267+0000","last_clean":"2024-04-14T15:19:15.936267+0000","last_became_active":"2024-04-14T15:19:14.014254+0000","last_became_peered":"2024-04-14T15:19:14.014254+0000","last_unstale":"2024-04-14T15:19:15.936267+0000","last_undegraded":"2024-04-14T15:19:15.936267+0000","last_fullsized":"2024-04-14T15:19:15.936267+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:12.984089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:12.984089+0000","last_clean_scrub_stamp":"2024-04-14T15:19:12.984089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:40.296502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"301.1b","version":"0'0","reported_seq":15,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.936161+0000","last_change":"2024-04-14T15:19:14.014651+0000","last_active":"2024-04-14T15:19:15.936161+0000","last_peered":"2024-04-14T15:19:15.936161+0000","last_clean":"2024-04-14T15:19:15.936161+0000","last_became_active":"2024-04-14T15:19:14.014348+0000","last_became_peered":"2024-04-14T15:19:14.014348+0000","last_unstale":"2024-04-14T15:19:15.936161+0000","last_undegraded":"2024-04-14T15:19:15.936161+0000","last_fullsized":"2024-04-14T15:19:15.936161+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:12.984089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:12.984089+0000","last_clean_scrub_stamp":"2024-04-14T15:19:12.984089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:29.472598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"301.1a","version":"0'0","reported_seq":15,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.916561+0000","last_change":"2024-04-14T15:19:14.298212+0000","last_active":"2024-04-14T15:19:15.916561+0000","last_peered":"2024-04-14T15:19:15.916561+0000","last_clean":"2024-04-14T15:19:15.916561+0000","last_became_active":"2024-04-14T15:19:14.297805+0000","last_became_peered":"2024-04-14T15:19:14.297805+0000","last_unstale":"2024-04-14T15:19:15.916561+0000","last_undegraded":"2024-04-14T15:19:15.916561+0000","last_fullsized":"2024-04-14T15:19:15.916561+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:12.984089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:12.984089+0000","last_clean_scrub_stamp":"2024-04-14T15:19:12.984089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:15.707986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"301.19","version":"0'0","reported_seq":15,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.964270+0000","last_change":"2024-04-14T15:19:14.011056+0000","last_active":"2024-04-14T15:19:15.964270+0000","last_peered":"2024-04-14T15:19:15.964270+0000","last_clean":"2024-04-14T15:19:15.964270+0000","last_became_active":"2024-04-14T15:19:14.010781+0000","last_became_peered":"2024-04-14T15:19:14.010781+0000","last_unstale":"2024-04-14T15:19:15.964270+0000","last_undegraded":"2024-04-14T15:19:15.964270+0000","last_fullsized":"2024-04-14T15:19:15.964270+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:12.984089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:12.984089+0000","last_clean_scrub_stamp":"2024-04-14T15:19:12.984089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:31.751339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"301.18","version":"0'0","reported_seq":15,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.964162+0000","last_change":"2024-04-14T15:19:14.012643+0000","last_active":"2024-04-14T15:19:15.964162+0000","last_peered":"2024-04-14T15:19:15.964162+0000","last_clean":"2024-04-14T15:19:15.964162+0000","last_became_active":"2024-04-14T15:19:14.012287+0000","last_became_peered":"2024-04-14T15:19:14.012287+0000","last_unstale":"2024-04-14T15:19:15.964162+0000","last_undegraded":"2024-04-14T15:19:15.964162+0000","last_fullsized":"2024-04-14T15:19:15.964162+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:12.984089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:12.984089+0000","last_clean_scrub_stamp":"2024-04-14T15:19:12.984089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:28.991985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"301.17","version":"0'0","reported_seq":15,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.914855+0000","last_change":"2024-04-14T15:19:14.297068+0000","last_active":"2024-04-14T15:19:15.914855+0000","last_peered":"2024-04-14T15:19:15.914855+0000","last_clean":"2024-04-14T15:19:15.914855+0000","last_became_active":"2024-04-14T15:19:14.296648+0000","last_became_peered":"2024-04-14T15:19:14.296648+0000","last_unstale":"2024-04-14T15:19:15.914855+0000","last_undegraded":"2024-04-14T15:19:15.914855+0000","last_fullsized":"2024-04-14T15:19:15.914855+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:12.984089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:12.984089+0000","last_clean_scrub_stamp":"2024-04-14T15:19:12.984089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:18.133293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"301.16","version":"0'0","reported_seq":15,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.914393+0000","last_change":"2024-04-14T15:19:14.298254+0000","last_active":"2024-04-14T15:19:15.914393+0000","last_peered":"2024-04-14T15:19:15.914393+0000","last_clean":"2024-04-14T15:19:15.914393+0000","last_became_active":"2024-04-14T15:19:14.296794+0000","last_became_peered":"2024-04-14T15:19:14.296794+0000","last_unstale":"2024-04-14T15:19:15.914393+0000","last_undegraded":"2024-04-14T15:19:15.914393+0000","last_fullsized":"2024-04-14T15:19:15.914393+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:12.984089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:12.984089+0000","last_clean_scrub_stamp":"2024-04-14T15:19:12.984089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:38.730577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"301.15","version":"0'0","reported_seq":15,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.918646+0000","last_change":"2024-04-14T15:19:14.297244+0000","last_active":"2024-04-14T15:19:15.918646+0000","last_peered":"2024-04-14T15:19:15.918646+0000","last_clean":"2024-04-14T15:19:15.918646+0000","last_became_active":"2024-04-14T15:19:14.296942+0000","last_became_peered":"2024-04-14T15:19:14.296942+0000","last_unstale":"2024-04-14T15:19:15.918646+0000","last_undegraded":"2024-04-14T15:19:15.918646+0000","last_fullsized":"2024-04-14T15:19:15.918646+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:12.984089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:12.984089+0000","last_clean_scrub_stamp":"2024-04-14T15:19:12.984089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:28.810597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"301.14","version":"0'0","reported_seq":15,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.913918+0000","last_change":"2024-04-14T15:19:14.015219+0000","last_active":"2024-04-14T15:19:15.913918+0000","last_peered":"2024-04-14T15:19:15.913918+0000","last_clean":"2024-04-14T15:19:15.913918+0000","last_became_active":"2024-04-14T15:19:14.014812+0000","last_became_peered":"2024-04-14T15:19:14.014812+0000","last_unstale":"2024-04-14T15:19:15.913918+0000","last_undegraded":"2024-04-14T15:19:15.913918+0000","last_fullsized":"2024-04-14T15:19:15.913918+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:12.984089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:12.984089+0000","last_clean_scrub_stamp":"2024-04-14T15:19:12.984089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:39.766323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"301.13","version":"0'0","reported_seq":15,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.917569+0000","last_change":"2024-04-14T15:19:14.022131+0000","last_active":"2024-04-14T15:19:15.917569+0000","last_peered":"2024-04-14T15:19:15.917569+0000","last_clean":"2024-04-14T15:19:15.917569+0000","last_became_active":"2024-04-14T15:19:14.021802+0000","last_became_peered":"2024-04-14T15:19:14.021802+0000","last_unstale":"2024-04-14T15:19:15.917569+0000","last_undegraded":"2024-04-14T15:19:15.917569+0000","last_fullsized":"2024-04-14T15:19:15.917569+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:12.984089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:12.984089+0000","last_clean_scrub_stamp":"2024-04-14T15:19:12.984089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:18.394083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"301.12","version":"0'0","reported_seq":15,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.924475+0000","last_change":"2024-04-14T15:19:14.016363+0000","last_active":"2024-04-14T15:19:15.924475+0000","last_peered":"2024-04-14T15:19:15.924475+0000","last_clean":"2024-04-14T15:19:15.924475+0000","last_became_active":"2024-04-14T15:19:14.016062+0000","last_became_peered":"2024-04-14T15:19:14.016062+0000","last_unstale":"2024-04-14T15:19:15.924475+0000","last_undegraded":"2024-04-14T15:19:15.924475+0000","last_fullsized":"2024-04-14T15:19:15.924475+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:12.984089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:12.984089+0000","last_clean_scrub_stamp":"2024-04-14T15:19:12.984089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:29.582547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"301.11","version":"463'2","reported_seq":17,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.964270+0000","last_change":"2024-04-14T15:19:14.019103+0000","last_active":"2024-04-14T15:19:15.964270+0000","last_peered":"2024-04-14T15:19:15.964270+0000","last_clean":"2024-04-14T15:19:15.964270+0000","last_became_active":"2024-04-14T15:19:14.018739+0000","last_became_peered":"2024-04-14T15:19:14.018739+0000","last_unstale":"2024-04-14T15:19:15.964270+0000","last_undegraded":"2024-04-14T15:19:15.964270+0000","last_fullsized":"2024-04-14T15:19:15.964270+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:12.984089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:12.984089+0000","last_clean_scrub_stamp":"2024-04-14T15:19:12.984089+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:37.351074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"301.10","version":"463'3","reported_seq":18,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.915498+0000","last_change":"2024-04-14T15:19:14.013261+0000","last_active":"2024-04-14T15:19:15.915498+0000","last_peered":"2024-04-14T15:19:15.915498+0000","last_clean":"2024-04-14T15:19:15.915498+0000","last_became_active":"2024-04-14T15:19:14.012978+0000","last_became_peered":"2024-04-14T15:19:14.012978+0000","last_unstale":"2024-04-14T15:19:15.915498+0000","last_undegraded":"2024-04-14T15:19:15.915498+0000","last_fullsized":"2024-04-14T15:19:15.915498+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:12.984089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:12.984089+0000","last_clean_scrub_stamp":"2024-04-14T15:19:12.984089+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:19.102642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"301.f","version":"0'0","reported_seq":15,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.965910+0000","last_change":"2024-04-14T15:19:14.019522+0000","last_active":"2024-04-14T15:19:15.965910+0000","last_peered":"2024-04-14T15:19:15.965910+0000","last_clean":"2024-04-14T15:19:15.965910+0000","last_became_active":"2024-04-14T15:19:14.019100+0000","last_became_peered":"2024-04-14T15:19:14.019100+0000","last_unstale":"2024-04-14T15:19:15.965910+0000","last_undegraded":"2024-04-14T15:19:15.965910+0000","last_fullsized":"2024-04-14T15:19:15.965910+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:12.984089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:12.984089+0000","last_clean_scrub_stamp":"2024-04-14T15:19:12.984089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:39.408714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"301.e","version":"0'0","reported_seq":15,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.917865+0000","last_change":"2024-04-14T15:19:14.296947+0000","last_active":"2024-04-14T15:19:15.917865+0000","last_peered":"2024-04-14T15:19:15.917865+0000","last_clean":"2024-04-14T15:19:15.917865+0000","last_became_active":"2024-04-14T15:19:14.296658+0000","last_became_peered":"2024-04-14T15:19:14.296658+0000","last_unstale":"2024-04-14T15:19:15.917865+0000","last_undegraded":"2024-04-14T15:19:15.917865+0000","last_fullsized":"2024-04-14T15:19:15.917865+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:12.984089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:12.984089+0000","last_clean_scrub_stamp":"2024-04-14T15:19:12.984089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:25.956486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"301.d","version":"0'0","reported_seq":15,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.930033+0000","last_change":"2024-04-14T15:19:14.019779+0000","last_active":"2024-04-14T15:19:15.930033+0000","last_peered":"2024-04-14T15:19:15.930033+0000","last_clean":"2024-04-14T15:19:15.930033+0000","last_became_active":"2024-04-14T15:19:14.019494+0000","last_became_peered":"2024-04-14T15:19:14.019494+0000","last_unstale":"2024-04-14T15:19:15.930033+0000","last_undegraded":"2024-04-14T15:19:15.930033+0000","last_fullsized":"2024-04-14T15:19:15.930033+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:12.984089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:12.984089+0000","last_clean_scrub_stamp":"2024-04-14T15:19:12.984089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:39.097072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"301.c","version":"0'0","reported_seq":15,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.964881+0000","last_change":"2024-04-14T15:19:14.019228+0000","last_active":"2024-04-14T15:19:15.964881+0000","last_peered":"2024-04-14T15:19:15.964881+0000","last_clean":"2024-04-14T15:19:15.964881+0000","last_became_active":"2024-04-14T15:19:14.018882+0000","last_became_peered":"2024-04-14T15:19:14.018882+0000","last_unstale":"2024-04-14T15:19:15.964881+0000","last_undegraded":"2024-04-14T15:19:15.964881+0000","last_fullsized":"2024-04-14T15:19:15.964881+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:12.984089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:12.984089+0000","last_clean_scrub_stamp":"2024-04-14T15:19:12.984089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:36.301113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"301.b","version":"463'1","reported_seq":16,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.917656+0000","last_change":"2024-04-14T15:19:14.016505+0000","last_active":"2024-04-14T15:19:15.917656+0000","last_peered":"2024-04-14T15:19:15.917656+0000","last_clean":"2024-04-14T15:19:15.917656+0000","last_became_active":"2024-04-14T15:19:14.016121+0000","last_became_peered":"2024-04-14T15:19:14.016121+0000","last_unstale":"2024-04-14T15:19:15.917656+0000","last_undegraded":"2024-04-14T15:19:15.917656+0000","last_fullsized":"2024-04-14T15:19:15.917656+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:12.984089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:12.984089+0000","last_clean_scrub_stamp":"2024-04-14T15:19:12.984089+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:28.340260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"301.a","version":"0'0","reported_seq":15,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.929330+0000","last_change":"2024-04-14T15:19:14.021249+0000","last_active":"2024-04-14T15:19:15.929330+0000","last_peered":"2024-04-14T15:19:15.929330+0000","last_clean":"2024-04-14T15:19:15.929330+0000","last_became_active":"2024-04-14T15:19:14.020965+0000","last_became_peered":"2024-04-14T15:19:14.020965+0000","last_unstale":"2024-04-14T15:19:15.929330+0000","last_undegraded":"2024-04-14T15:19:15.929330+0000","last_fullsized":"2024-04-14T15:19:15.929330+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:12.984089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:12.984089+0000","last_clean_scrub_stamp":"2024-04-14T15:19:12.984089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:18.251127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"301.9","version":"464'4","reported_seq":19,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.936020+0000","last_change":"2024-04-14T15:19:14.296032+0000","last_active":"2024-04-14T15:19:15.936020+0000","last_peered":"2024-04-14T15:19:15.936020+0000","last_clean":"2024-04-14T15:19:15.936020+0000","last_became_active":"2024-04-14T15:19:14.295707+0000","last_became_peered":"2024-04-14T15:19:14.295707+0000","last_unstale":"2024-04-14T15:19:15.936020+0000","last_undegraded":"2024-04-14T15:19:15.936020+0000","last_fullsized":"2024-04-14T15:19:15.936020+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:12.984089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:12.984089+0000","last_clean_scrub_stamp":"2024-04-14T15:19:12.984089+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:14.552455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"301.8","version":"0'0","reported_seq":15,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.928801+0000","last_change":"2024-04-14T15:19:14.296557+0000","last_active":"2024-04-14T15:19:15.928801+0000","last_peered":"2024-04-14T15:19:15.928801+0000","last_clean":"2024-04-14T15:19:15.928801+0000","last_became_active":"2024-04-14T15:19:14.296174+0000","last_became_peered":"2024-04-14T15:19:14.296174+0000","last_unstale":"2024-04-14T15:19:15.928801+0000","last_undegraded":"2024-04-14T15:19:15.928801+0000","last_fullsized":"2024-04-14T15:19:15.928801+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:12.984089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:12.984089+0000","last_clean_scrub_stamp":"2024-04-14T15:19:12.984089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:39.000829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"301.7","version":"0'0","reported_seq":15,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.930739+0000","last_change":"2024-04-14T15:19:14.020149+0000","last_active":"2024-04-14T15:19:15.930739+0000","last_peered":"2024-04-14T15:19:15.930739+0000","last_clean":"2024-04-14T15:19:15.930739+0000","last_became_active":"2024-04-14T15:19:14.018915+0000","last_became_peered":"2024-04-14T15:19:14.018915+0000","last_unstale":"2024-04-14T15:19:15.930739+0000","last_undegraded":"2024-04-14T15:19:15.930739+0000","last_fullsized":"2024-04-14T15:19:15.930739+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:12.984089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:12.984089+0000","last_clean_scrub_stamp":"2024-04-14T15:19:12.984089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:21.732909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"301.6","version":"0'0","reported_seq":15,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.965650+0000","last_change":"2024-04-14T15:19:14.017082+0000","last_active":"2024-04-14T15:19:15.965650+0000","last_peered":"2024-04-14T15:19:15.965650+0000","last_clean":"2024-04-14T15:19:15.965650+0000","last_became_active":"2024-04-14T15:19:14.016823+0000","last_became_peered":"2024-04-14T15:19:14.016823+0000","last_unstale":"2024-04-14T15:19:15.965650+0000","last_undegraded":"2024-04-14T15:19:15.965650+0000","last_fullsized":"2024-04-14T15:19:15.965650+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:12.984089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:12.984089+0000","last_clean_scrub_stamp":"2024-04-14T15:19:12.984089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:16.928278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"301.5","version":"0'0","reported_seq":15,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.917568+0000","last_change":"2024-04-14T15:19:14.297577+0000","last_active":"2024-04-14T15:19:15.917568+0000","last_peered":"2024-04-14T15:19:15.917568+0000","last_clean":"2024-04-14T15:19:15.917568+0000","last_became_active":"2024-04-14T15:19:14.296650+0000","last_became_peered":"2024-04-14T15:19:14.296650+0000","last_unstale":"2024-04-14T15:19:15.917568+0000","last_undegraded":"2024-04-14T15:19:15.917568+0000","last_fullsized":"2024-04-14T15:19:15.917568+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:12.984089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:12.984089+0000","last_clean_scrub_stamp":"2024-04-14T15:19:12.984089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:37.315758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"301.4","version":"0'0","reported_seq":15,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.936487+0000","last_change":"2024-04-14T15:19:14.006335+0000","last_active":"2024-04-14T15:19:15.936487+0000","last_peered":"2024-04-14T15:19:15.936487+0000","last_clean":"2024-04-14T15:19:15.936487+0000","last_became_active":"2024-04-14T15:19:14.006027+0000","last_became_peered":"2024-04-14T15:19:14.006027+0000","last_unstale":"2024-04-14T15:19:15.936487+0000","last_undegraded":"2024-04-14T15:19:15.936487+0000","last_fullsized":"2024-04-14T15:19:15.936487+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:12.984089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:12.984089+0000","last_clean_scrub_stamp":"2024-04-14T15:19:12.984089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:26.405202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"301.3","version":"0'0","reported_seq":15,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.917743+0000","last_change":"2024-04-14T15:19:14.012596+0000","last_active":"2024-04-14T15:19:15.917743+0000","last_peered":"2024-04-14T15:19:15.917743+0000","last_clean":"2024-04-14T15:19:15.917743+0000","last_became_active":"2024-04-14T15:19:14.011642+0000","last_became_peered":"2024-04-14T15:19:14.011642+0000","last_unstale":"2024-04-14T15:19:15.917743+0000","last_undegraded":"2024-04-14T15:19:15.917743+0000","last_fullsized":"2024-04-14T15:19:15.917743+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:12.984089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:12.984089+0000","last_clean_scrub_stamp":"2024-04-14T15:19:12.984089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:40.231775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"301.2","version":"0'0","reported_seq":15,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.913181+0000","last_change":"2024-04-14T15:19:14.021759+0000","last_active":"2024-04-14T15:19:15.913181+0000","last_peered":"2024-04-14T15:19:15.913181+0000","last_clean":"2024-04-14T15:19:15.913181+0000","last_became_active":"2024-04-14T15:19:14.021389+0000","last_became_peered":"2024-04-14T15:19:14.021389+0000","last_unstale":"2024-04-14T15:19:15.913181+0000","last_undegraded":"2024-04-14T15:19:15.913181+0000","last_fullsized":"2024-04-14T15:19:15.913181+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:12.984089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:12.984089+0000","last_clean_scrub_stamp":"2024-04-14T15:19:12.984089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:29.944286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"301.1","version":"0'0","reported_seq":15,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.915933+0000","last_change":"2024-04-14T15:19:14.296764+0000","last_active":"2024-04-14T15:19:15.915933+0000","last_peered":"2024-04-14T15:19:15.915933+0000","last_clean":"2024-04-14T15:19:15.915933+0000","last_became_active":"2024-04-14T15:19:14.296474+0000","last_became_peered":"2024-04-14T15:19:14.296474+0000","last_unstale":"2024-04-14T15:19:15.915933+0000","last_undegraded":"2024-04-14T15:19:15.915933+0000","last_fullsized":"2024-04-14T15:19:15.915933+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:12.984089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:12.984089+0000","last_clean_scrub_stamp":"2024-04-14T15:19:12.984089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:42.033673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"301.0","version":"464'4","reported_seq":19,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.928150+0000","last_change":"2024-04-14T15:19:14.022796+0000","last_active":"2024-04-14T15:19:15.928150+0000","last_peered":"2024-04-14T15:19:15.928150+0000","last_clean":"2024-04-14T15:19:15.928150+0000","last_became_active":"2024-04-14T15:19:14.022517+0000","last_became_peered":"2024-04-14T15:19:14.022517+0000","last_unstale":"2024-04-14T15:19:15.928150+0000","last_undegraded":"2024-04-14T15:19:15.928150+0000","last_fullsized":"2024-04-14T15:19:15.928150+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:12.984089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:12.984089+0000","last_clean_scrub_stamp":"2024-04-14T15:19:12.984089+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:15.469904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"110.11","version":"403'41","reported_seq":935,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.964727+0000","last_change":"2024-04-14T15:18:54.159422+0000","last_active":"2024-04-14T15:19:15.964727+0000","last_peered":"2024-04-14T15:19:15.964727+0000","last_clean":"2024-04-14T15:19:15.964727+0000","last_became_active":"2024-04-14T15:13:09.936619+0000","last_became_peered":"2024-04-14T15:13:09.936619+0000","last_unstale":"2024-04-14T15:19:15.964727+0000","last_undegraded":"2024-04-14T15:19:15.964727+0000","last_fullsized":"2024-04-14T15:19:15.964727+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"403'41","last_scrub_stamp":"2024-04-14T15:17:56.911004+0000","last_deep_scrub":"403'41","last_deep_scrub_stamp":"2024-04-14T15:17:56.911004+0000","last_clean_scrub_stamp":"2024-04-14T15:17:56.911004+0000","objects_scrubbed":2,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:18.175290+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0.00061958300000000005,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"110.10","version":"253'158","reported_seq":1017,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.935957+0000","last_change":"2024-04-14T15:18:54.158668+0000","last_active":"2024-04-14T15:19:15.935957+0000","last_peered":"2024-04-14T15:19:15.935957+0000","last_clean":"2024-04-14T15:19:15.935957+0000","last_became_active":"2024-04-14T15:13:09.928915+0000","last_became_peered":"2024-04-14T15:13:09.928915+0000","last_unstale":"2024-04-14T15:19:15.935957+0000","last_undegraded":"2024-04-14T15:19:15.935957+0000","last_fullsized":"2024-04-14T15:19:15.935957+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"253'158","last_scrub_stamp":"2024-04-14T15:17:34.291854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:34.291854+0000","objects_scrubbed":10,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00045995200000000002,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":47,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"110.13","version":"409'41","reported_seq":936,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.918344+0000","last_change":"2024-04-14T15:18:54.157558+0000","last_active":"2024-04-14T15:19:15.918344+0000","last_peered":"2024-04-14T15:19:15.918344+0000","last_clean":"2024-04-14T15:19:15.918344+0000","last_became_active":"2024-04-14T15:13:09.918152+0000","last_became_peered":"2024-04-14T15:13:09.918152+0000","last_unstale":"2024-04-14T15:19:15.918344+0000","last_undegraded":"2024-04-14T15:19:15.918344+0000","last_fullsized":"2024-04-14T15:19:15.918344+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'39","last_scrub_stamp":"2024-04-14T15:17:44.873766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:44.873766+0000","objects_scrubbed":3,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001942764,"stat_sum":{"num_bytes":27,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":14,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"110.12","version":"409'70","reported_seq":962,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.916863+0000","last_change":"2024-04-14T15:18:54.156143+0000","last_active":"2024-04-14T15:19:15.916863+0000","last_peered":"2024-04-14T15:19:15.916863+0000","last_clean":"2024-04-14T15:19:15.916863+0000","last_became_active":"2024-04-14T15:13:09.939566+0000","last_became_peered":"2024-04-14T15:13:09.939566+0000","last_unstale":"2024-04-14T15:19:15.916863+0000","last_undegraded":"2024-04-14T15:19:15.916863+0000","last_fullsized":"2024-04-14T15:19:15.916863+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"253'66","last_scrub_stamp":"2024-04-14T15:16:53.673906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:16:53.673906+0000","objects_scrubbed":6,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000970527,"stat_sum":{"num_bytes":74,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":24,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"110.15","version":"190'34","reported_seq":931,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.930330+0000","last_change":"2024-04-14T15:18:54.155028+0000","last_active":"2024-04-14T15:19:15.930330+0000","last_peered":"2024-04-14T15:19:15.930330+0000","last_clean":"2024-04-14T15:19:15.930330+0000","last_became_active":"2024-04-14T15:13:09.960121+0000","last_became_peered":"2024-04-14T15:13:09.960121+0000","last_unstale":"2024-04-14T15:19:15.930330+0000","last_undegraded":"2024-04-14T15:19:15.930330+0000","last_fullsized":"2024-04-14T15:19:15.930330+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'34","last_scrub_stamp":"2024-04-14T15:18:36.013135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:18:36.013135+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:54.434891+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0029213049999999999,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"110.14","version":"355'87","reported_seq":977,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.930058+0000","last_change":"2024-04-14T15:18:54.151534+0000","last_active":"2024-04-14T15:19:15.930058+0000","last_peered":"2024-04-14T15:19:15.930058+0000","last_clean":"2024-04-14T15:19:15.930058+0000","last_became_active":"2024-04-14T15:13:09.899561+0000","last_became_peered":"2024-04-14T15:13:09.899561+0000","last_unstale":"2024-04-14T15:19:15.930058+0000","last_undegraded":"2024-04-14T15:19:15.930058+0000","last_fullsized":"2024-04-14T15:19:15.930058+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'87","last_scrub_stamp":"2024-04-14T15:18:31.117033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:18:31.117033+0000","objects_scrubbed":7,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:38.957602+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0014558730000000001,"stat_sum":{"num_bytes":71,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"110.17","version":"456'12","reported_seq":916,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.931021+0000","last_change":"2024-04-14T15:18:54.154013+0000","last_active":"2024-04-14T15:19:15.931021+0000","last_peered":"2024-04-14T15:19:15.931021+0000","last_clean":"2024-04-14T15:19:15.931021+0000","last_became_active":"2024-04-14T15:13:09.898521+0000","last_became_peered":"2024-04-14T15:13:09.898521+0000","last_unstale":"2024-04-14T15:19:15.931021+0000","last_undegraded":"2024-04-14T15:19:15.931021+0000","last_fullsized":"2024-04-14T15:19:15.931021+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'6","last_scrub_stamp":"2024-04-14T15:18:38.009133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:18:38.009133+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:02.756745+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0018681640000000001,"stat_sum":{"num_bytes":24,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"110.16","version":"355'35","reported_seq":763,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.925751+0000","last_change":"2024-04-14T15:18:54.160925+0000","last_active":"2024-04-14T15:19:15.925751+0000","last_peered":"2024-04-14T15:19:15.925751+0000","last_clean":"2024-04-14T15:19:15.925751+0000","last_became_active":"2024-04-14T15:15:21.889783+0000","last_became_peered":"2024-04-14T15:15:21.889783+0000","last_unstale":"2024-04-14T15:19:15.925751+0000","last_undegraded":"2024-04-14T15:19:15.925751+0000","last_fullsized":"2024-04-14T15:19:15.925751+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'35","last_scrub_stamp":"2024-04-14T15:17:34.097563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:34.097563+0000","objects_scrubbed":3,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000499903,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"110.19","version":"464'96","reported_seq":990,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.933317+0000","last_change":"2024-04-14T15:19:12.058233+0000","last_active":"2024-04-14T15:19:15.933317+0000","last_peered":"2024-04-14T15:19:15.933317+0000","last_clean":"2024-04-14T15:19:15.933317+0000","last_became_active":"2024-04-14T15:13:09.998139+0000","last_became_peered":"2024-04-14T15:13:09.998139+0000","last_unstale":"2024-04-14T15:19:15.933317+0000","last_undegraded":"2024-04-14T15:19:15.933317+0000","last_fullsized":"2024-04-14T15:19:15.933317+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'85","last_scrub_stamp":"2024-04-14T15:19:12.058127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:19:12.058127+0000","objects_scrubbed":5,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:30.555029+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0.0012903700000000001,"stat_sum":{"num_bytes":66,"num_objects":7,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"110.18","version":"387'94","reported_seq":982,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.915594+0000","last_change":"2024-04-14T15:18:54.150377+0000","last_active":"2024-04-14T15:19:15.915594+0000","last_peered":"2024-04-14T15:19:15.915594+0000","last_clean":"2024-04-14T15:19:15.915594+0000","last_became_active":"2024-04-14T15:13:09.998803+0000","last_became_peered":"2024-04-14T15:13:09.998803+0000","last_unstale":"2024-04-14T15:19:15.915594+0000","last_undegraded":"2024-04-14T15:19:15.915594+0000","last_fullsized":"2024-04-14T15:19:15.915594+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'94","last_scrub_stamp":"2024-04-14T15:17:36.005598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:36.005598+0000","objects_scrubbed":8,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0013897930000000001,"stat_sum":{"num_bytes":65,"num_objects":8,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":29,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"110.1b","version":"355'91","reported_seq":978,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.917891+0000","last_change":"2024-04-14T15:18:54.156913+0000","last_active":"2024-04-14T15:19:15.917891+0000","last_peered":"2024-04-14T15:19:15.917891+0000","last_clean":"2024-04-14T15:19:15.917891+0000","last_became_active":"2024-04-14T15:13:09.946919+0000","last_became_peered":"2024-04-14T15:13:09.946919+0000","last_unstale":"2024-04-14T15:19:15.917891+0000","last_undegraded":"2024-04-14T15:19:15.917891+0000","last_fullsized":"2024-04-14T15:19:15.917891+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'91","last_scrub_stamp":"2024-04-14T15:17:47.889364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:47.889364+0000","objects_scrubbed":6,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001643863,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":28,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"110.1a","version":"387'39","reported_seq":939,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.965169+0000","last_change":"2024-04-14T15:18:54.160775+0000","last_active":"2024-04-14T15:19:15.965169+0000","last_peered":"2024-04-14T15:19:15.965169+0000","last_clean":"2024-04-14T15:19:15.965169+0000","last_became_active":"2024-04-14T15:13:09.960319+0000","last_became_peered":"2024-04-14T15:13:09.960319+0000","last_unstale":"2024-04-14T15:19:15.965169+0000","last_undegraded":"2024-04-14T15:19:15.965169+0000","last_fullsized":"2024-04-14T15:19:15.965169+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'39","last_scrub_stamp":"2024-04-14T15:17:53.871590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:53.871590+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0018412890000000001,"stat_sum":{"num_bytes":62,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"110.1d","version":"225'85","reported_seq":974,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.914421+0000","last_change":"2024-04-14T15:18:54.151417+0000","last_active":"2024-04-14T15:19:15.914421+0000","last_peered":"2024-04-14T15:19:15.914421+0000","last_clean":"2024-04-14T15:19:15.914421+0000","last_became_active":"2024-04-14T15:13:09.868082+0000","last_became_peered":"2024-04-14T15:13:09.868082+0000","last_unstale":"2024-04-14T15:19:15.914421+0000","last_undegraded":"2024-04-14T15:19:15.914421+0000","last_fullsized":"2024-04-14T15:19:15.914421+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"225'85","last_scrub_stamp":"2024-04-14T15:17:36.419088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:36.419088+0000","objects_scrubbed":5,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0010543219999999999,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"110.1c","version":"430'95","reported_seq":987,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.918657+0000","last_change":"2024-04-14T15:19:10.839468+0000","last_active":"2024-04-14T15:19:15.918657+0000","last_peered":"2024-04-14T15:19:15.918657+0000","last_clean":"2024-04-14T15:19:15.918657+0000","last_became_active":"2024-04-14T15:13:09.954091+0000","last_became_peered":"2024-04-14T15:13:09.954091+0000","last_unstale":"2024-04-14T15:19:15.918657+0000","last_undegraded":"2024-04-14T15:19:15.918657+0000","last_fullsized":"2024-04-14T15:19:15.918657+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'95","last_scrub_stamp":"2024-04-14T15:19:10.839339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:19:10.839339+0000","objects_scrubbed":7,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:37.884199+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0011789070000000001,"stat_sum":{"num_bytes":65,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"110.1f","version":"268'25","reported_seq":968,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.916361+0000","last_change":"2024-04-14T15:18:54.153191+0000","last_active":"2024-04-14T15:19:15.916361+0000","last_peered":"2024-04-14T15:19:15.916361+0000","last_clean":"2024-04-14T15:19:15.916361+0000","last_became_active":"2024-04-14T15:13:09.999629+0000","last_became_peered":"2024-04-14T15:13:09.999629+0000","last_unstale":"2024-04-14T15:19:15.916361+0000","last_undegraded":"2024-04-14T15:19:15.916361+0000","last_fullsized":"2024-04-14T15:19:15.916361+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'25","last_scrub_stamp":"2024-04-14T15:16:08.236881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:16:08.236881+0000","objects_scrubbed":2,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001392454,"stat_sum":{"num_bytes":4194315,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":8,"num_write_kb":8198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"110.1e","version":"355'23","reported_seq":925,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:17.364824+0000","last_change":"2024-04-14T15:19:17.364851+0000","last_active":"2024-04-14T15:19:17.364824+0000","last_peered":"2024-04-14T15:19:17.364824+0000","last_clean":"2024-04-14T15:19:17.364824+0000","last_became_active":"2024-04-14T15:13:09.955932+0000","last_became_peered":"2024-04-14T15:13:09.955932+0000","last_unstale":"2024-04-14T15:19:17.364824+0000","last_undegraded":"2024-04-14T15:19:17.364824+0000","last_fullsized":"2024-04-14T15:19:17.364824+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'23","last_scrub_stamp":"2024-04-14T15:19:17.364769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:19:17.364769+0000","objects_scrubbed":2,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:26.786872+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0019783700000000001,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":8,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"110.1","version":"406'32","reported_seq":935,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.913340+0000","last_change":"2024-04-14T15:18:54.150210+0000","last_active":"2024-04-14T15:19:15.913340+0000","last_peered":"2024-04-14T15:19:15.913340+0000","last_clean":"2024-04-14T15:19:15.913340+0000","last_became_active":"2024-04-14T15:13:09.929562+0000","last_became_peered":"2024-04-14T15:13:09.929562+0000","last_unstale":"2024-04-14T15:19:15.913340+0000","last_undegraded":"2024-04-14T15:19:15.913340+0000","last_fullsized":"2024-04-14T15:19:15.913340+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"406'32","last_scrub_stamp":"2024-04-14T15:18:38.685469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:18:38.685469+0000","objects_scrubbed":7,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:05.005124+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00089761599999999995,"stat_sum":{"num_bytes":4000050,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":15,"num_write_kb":3921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"110.0","version":"219'85","reported_seq":969,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.935994+0000","last_change":"2024-04-14T15:18:54.159255+0000","last_active":"2024-04-14T15:19:15.935994+0000","last_peered":"2024-04-14T15:19:15.935994+0000","last_clean":"2024-04-14T15:19:15.935994+0000","last_became_active":"2024-04-14T15:13:09.856407+0000","last_became_peered":"2024-04-14T15:13:09.856407+0000","last_unstale":"2024-04-14T15:19:15.935994+0000","last_undegraded":"2024-04-14T15:19:15.935994+0000","last_fullsized":"2024-04-14T15:19:15.935994+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"219'85","last_scrub_stamp":"2024-04-14T15:17:51.438149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:51.438149+0000","objects_scrubbed":5,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00052820199999999997,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"110.3","version":"464'56","reported_seq":944,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:17.063799+0000","last_change":"2024-04-14T15:19:17.063851+0000","last_active":"2024-04-14T15:19:17.063799+0000","last_peered":"2024-04-14T15:19:17.063799+0000","last_clean":"2024-04-14T15:19:17.063799+0000","last_became_active":"2024-04-14T15:13:09.997208+0000","last_became_peered":"2024-04-14T15:13:09.997208+0000","last_unstale":"2024-04-14T15:19:17.063799+0000","last_undegraded":"2024-04-14T15:19:17.063799+0000","last_fullsized":"2024-04-14T15:19:17.063799+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"464'56","last_scrub_stamp":"2024-04-14T15:19:17.063716+0000","last_deep_scrub":"464'56","last_deep_scrub_stamp":"2024-04-14T15:19:17.063716+0000","last_clean_scrub_stamp":"2024-04-14T15:19:17.063716+0000","objects_scrubbed":4,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:23.583426+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0019291829999999999,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"110.2","version":"240'55","reported_seq":950,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.916392+0000","last_change":"2024-04-14T15:18:54.152834+0000","last_active":"2024-04-14T15:19:15.916392+0000","last_peered":"2024-04-14T15:19:15.916392+0000","last_clean":"2024-04-14T15:19:15.916392+0000","last_became_active":"2024-04-14T15:13:09.957187+0000","last_became_peered":"2024-04-14T15:13:09.957187+0000","last_unstale":"2024-04-14T15:19:15.916392+0000","last_undegraded":"2024-04-14T15:19:15.916392+0000","last_fullsized":"2024-04-14T15:19:15.916392+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'55","last_scrub_stamp":"2024-04-14T15:17:37.941589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:37.941589+0000","objects_scrubbed":6,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0020075969999999999,"stat_sum":{"num_bytes":49,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":18,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"110.5","version":"396'64","reported_seq":951,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.937400+0000","last_change":"2024-04-14T15:18:54.162113+0000","last_active":"2024-04-14T15:19:15.937400+0000","last_peered":"2024-04-14T15:19:15.937400+0000","last_clean":"2024-04-14T15:19:15.937400+0000","last_became_active":"2024-04-14T15:13:09.853689+0000","last_became_peered":"2024-04-14T15:13:09.853689+0000","last_unstale":"2024-04-14T15:19:15.937400+0000","last_undegraded":"2024-04-14T15:19:15.937400+0000","last_fullsized":"2024-04-14T15:19:15.937400+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"396'64","last_scrub_stamp":"2024-04-14T15:17:47.418687+0000","last_deep_scrub":"396'64","last_deep_scrub_stamp":"2024-04-14T15:17:47.418687+0000","last_clean_scrub_stamp":"2024-04-14T15:17:47.418687+0000","objects_scrubbed":5,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00099928400000000002,"stat_sum":{"num_bytes":42,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":20,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"110.4","version":"406'55","reported_seq":948,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.930235+0000","last_change":"2024-04-14T15:19:11.276191+0000","last_active":"2024-04-14T15:19:15.930235+0000","last_peered":"2024-04-14T15:19:15.930235+0000","last_clean":"2024-04-14T15:19:15.930235+0000","last_became_active":"2024-04-14T15:13:09.958847+0000","last_became_peered":"2024-04-14T15:13:09.958847+0000","last_unstale":"2024-04-14T15:19:15.930235+0000","last_undegraded":"2024-04-14T15:19:15.930235+0000","last_fullsized":"2024-04-14T15:19:15.930235+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"406'55","last_scrub_stamp":"2024-04-14T15:19:11.276056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:19:11.276056+0000","objects_scrubbed":4,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:29.496851+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001395522,"stat_sum":{"num_bytes":38,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"110.7","version":"355'18","reported_seq":916,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.966279+0000","last_change":"2024-04-14T15:18:54.163286+0000","last_active":"2024-04-14T15:19:15.966279+0000","last_peered":"2024-04-14T15:19:15.966279+0000","last_clean":"2024-04-14T15:19:15.966279+0000","last_became_active":"2024-04-14T15:13:09.936631+0000","last_became_peered":"2024-04-14T15:13:09.936631+0000","last_unstale":"2024-04-14T15:19:15.966279+0000","last_undegraded":"2024-04-14T15:19:15.966279+0000","last_fullsized":"2024-04-14T15:19:15.966279+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'18","last_scrub_stamp":"2024-04-14T15:17:34.552788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:34.552788+0000","objects_scrubbed":2,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00050163699999999998,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"110.6","version":"444'90","reported_seq":973,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.914833+0000","last_change":"2024-04-14T15:19:12.507223+0000","last_active":"2024-04-14T15:19:15.914833+0000","last_peered":"2024-04-14T15:19:15.914833+0000","last_clean":"2024-04-14T15:19:15.914833+0000","last_became_active":"2024-04-14T15:13:09.930125+0000","last_became_peered":"2024-04-14T15:13:09.930125+0000","last_unstale":"2024-04-14T15:19:15.914833+0000","last_undegraded":"2024-04-14T15:19:15.914833+0000","last_fullsized":"2024-04-14T15:19:15.914833+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"444'90","last_scrub_stamp":"2024-04-14T15:19:12.507119+0000","last_deep_scrub":"274'74","last_deep_scrub_stamp":"2024-04-14T15:15:42.641684+0000","last_clean_scrub_stamp":"2024-04-14T15:19:12.507119+0000","objects_scrubbed":7,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:22.780372+0000","scrub_duration":1,"objects_trimmed":1,"snaptrim_duration":2.0059427689999998,"stat_sum":{"num_bytes":62,"num_objects":7,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":26,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"110.9","version":"369'20","reported_seq":912,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.918486+0000","last_change":"2024-04-14T15:18:54.150227+0000","last_active":"2024-04-14T15:19:15.918486+0000","last_peered":"2024-04-14T15:19:15.918486+0000","last_clean":"2024-04-14T15:19:15.918486+0000","last_became_active":"2024-04-14T15:13:09.957324+0000","last_became_peered":"2024-04-14T15:13:09.957324+0000","last_unstale":"2024-04-14T15:19:15.918486+0000","last_undegraded":"2024-04-14T15:19:15.918486+0000","last_fullsized":"2024-04-14T15:19:15.918486+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"225'17","last_scrub_stamp":"2024-04-14T15:16:09.202462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:16:09.202462+0000","objects_scrubbed":1,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00098494500000000009,"stat_sum":{"num_bytes":35,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"110.8","version":"190'17","reported_seq":667,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.924013+0000","last_change":"2024-04-14T15:18:54.159492+0000","last_active":"2024-04-14T15:19:15.924013+0000","last_peered":"2024-04-14T15:19:15.924013+0000","last_clean":"2024-04-14T15:19:15.924013+0000","last_became_active":"2024-04-14T15:15:44.890057+0000","last_became_peered":"2024-04-14T15:15:44.890057+0000","last_unstale":"2024-04-14T15:19:15.924013+0000","last_undegraded":"2024-04-14T15:19:15.924013+0000","last_fullsized":"2024-04-14T15:19:15.924013+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'17","last_scrub_stamp":"2024-04-14T15:18:34.505521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:18:34.505521+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:39.730078+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00097299799999999998,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"110.b","version":"412'71","reported_seq":958,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.918161+0000","last_change":"2024-04-14T15:18:54.157640+0000","last_active":"2024-04-14T15:19:15.918161+0000","last_peered":"2024-04-14T15:19:15.918161+0000","last_clean":"2024-04-14T15:19:15.918161+0000","last_became_active":"2024-04-14T15:13:09.964629+0000","last_became_peered":"2024-04-14T15:13:09.964629+0000","last_unstale":"2024-04-14T15:19:15.918161+0000","last_undegraded":"2024-04-14T15:19:15.918161+0000","last_fullsized":"2024-04-14T15:19:15.918161+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'68","last_scrub_stamp":"2024-04-14T15:16:49.611563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:16:49.611563+0000","objects_scrubbed":4,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.0017521979999999999,"stat_sum":{"num_bytes":54,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":22,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"110.a","version":"387'74","reported_seq":891,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.924684+0000","last_change":"2024-04-14T15:19:14.371878+0000","last_active":"2024-04-14T15:19:15.924684+0000","last_peered":"2024-04-14T15:19:15.924684+0000","last_clean":"2024-04-14T15:19:15.924684+0000","last_became_active":"2024-04-14T15:15:21.885346+0000","last_became_peered":"2024-04-14T15:15:21.885346+0000","last_unstale":"2024-04-14T15:19:15.924684+0000","last_undegraded":"2024-04-14T15:19:15.924684+0000","last_fullsized":"2024-04-14T15:19:15.924684+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'74","last_scrub_stamp":"2024-04-14T15:19:14.371777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:19:14.371777+0000","objects_scrubbed":9,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:15.555650+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00095557400000000005,"stat_sum":{"num_bytes":84,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":26,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"110.d","version":"416'95","reported_seq":815,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.925384+0000","last_change":"2024-04-14T15:19:13.327926+0000","last_active":"2024-04-14T15:19:15.925384+0000","last_peered":"2024-04-14T15:19:15.925384+0000","last_clean":"2024-04-14T15:19:15.925384+0000","last_became_active":"2024-04-14T15:15:21.877073+0000","last_became_peered":"2024-04-14T15:15:21.877073+0000","last_unstale":"2024-04-14T15:19:15.925384+0000","last_undegraded":"2024-04-14T15:19:15.925384+0000","last_fullsized":"2024-04-14T15:19:15.925384+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'95","last_scrub_stamp":"2024-04-14T15:19:13.327836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:19:13.327836+0000","objects_scrubbed":8,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:13.548953+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00067428099999999999,"stat_sum":{"num_bytes":71,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":34,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"110.c","version":"459'25","reported_seq":922,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.918658+0000","last_change":"2024-04-14T15:18:54.157224+0000","last_active":"2024-04-14T15:19:15.918658+0000","last_peered":"2024-04-14T15:19:15.918658+0000","last_clean":"2024-04-14T15:19:15.918658+0000","last_became_active":"2024-04-14T15:13:09.984609+0000","last_became_peered":"2024-04-14T15:13:09.984609+0000","last_unstale":"2024-04-14T15:19:15.918658+0000","last_undegraded":"2024-04-14T15:19:15.918658+0000","last_fullsized":"2024-04-14T15:19:15.918658+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'20","last_scrub_stamp":"2024-04-14T15:17:52.882509+0000","last_deep_scrub":"398'20","last_deep_scrub_stamp":"2024-04-14T15:17:52.882509+0000","last_clean_scrub_stamp":"2024-04-14T15:17:52.882509+0000","objects_scrubbed":2,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001761999,"stat_sum":{"num_bytes":29,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":9,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"110.f","version":"355'52","reported_seq":944,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.916741+0000","last_change":"2024-04-14T15:18:54.157838+0000","last_active":"2024-04-14T15:19:15.916741+0000","last_peered":"2024-04-14T15:19:15.916741+0000","last_clean":"2024-04-14T15:19:15.916741+0000","last_became_active":"2024-04-14T15:13:09.964057+0000","last_became_peered":"2024-04-14T15:13:09.964057+0000","last_unstale":"2024-04-14T15:19:15.916741+0000","last_undegraded":"2024-04-14T15:19:15.916741+0000","last_fullsized":"2024-04-14T15:19:15.916741+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'52","last_scrub_stamp":"2024-04-14T15:17:45.848166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:45.848166+0000","objects_scrubbed":4,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001409509,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"110.e","version":"416'92","reported_seq":979,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.914781+0000","last_change":"2024-04-14T15:18:54.151705+0000","last_active":"2024-04-14T15:19:15.914781+0000","last_peered":"2024-04-14T15:19:15.914781+0000","last_clean":"2024-04-14T15:19:15.914781+0000","last_became_active":"2024-04-14T15:13:09.984980+0000","last_became_peered":"2024-04-14T15:13:09.984980+0000","last_unstale":"2024-04-14T15:19:15.914781+0000","last_undegraded":"2024-04-14T15:19:15.914781+0000","last_fullsized":"2024-04-14T15:19:15.914781+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'90","last_scrub_stamp":"2024-04-14T15:17:34.409367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:34.409367+0000","objects_scrubbed":6,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0012202070000000001,"stat_sum":{"num_bytes":68,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":28,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"91'130","reported_seq":402,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.914312+0000","last_change":"2024-04-14T15:19:09.497449+0000","last_active":"2024-04-14T15:19:15.914312+0000","last_peered":"2024-04-14T15:19:15.914312+0000","last_clean":"2024-04-14T15:19:15.914312+0000","last_became_active":"2024-04-14T15:18:58.187712+0000","last_became_peered":"2024-04-14T15:18:58.187712+0000","last_unstale":"2024-04-14T15:19:15.914312+0000","last_undegraded":"2024-04-14T15:19:15.914312+0000","last_fullsized":"2024-04-14T15:19:15.914312+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:09.497308+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:09.497308+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:23.450906+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":395,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.916447+0000","last_change":"2024-04-14T15:19:10.053883+0000","last_active":"2024-04-14T15:19:15.916447+0000","last_peered":"2024-04-14T15:19:15.916447+0000","last_clean":"2024-04-14T15:19:15.916447+0000","last_became_active":"2024-04-14T15:15:44.882941+0000","last_became_peered":"2024-04-14T15:15:44.882941+0000","last_unstale":"2024-04-14T15:19:15.916447+0000","last_undegraded":"2024-04-14T15:19:15.916447+0000","last_fullsized":"2024-04-14T15:19:15.916447+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:10.053804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:19:10.053804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:28.076611+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.9","version":"85'9","reported_seq":960,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.913691+0000","last_change":"2024-04-14T15:17:54.606158+0000","last_active":"2024-04-14T15:19:15.913691+0000","last_peered":"2024-04-14T15:19:15.913691+0000","last_clean":"2024-04-14T15:19:15.913691+0000","last_became_active":"2024-04-14T15:09:47.045441+0000","last_became_peered":"2024-04-14T15:09:47.045441+0000","last_unstale":"2024-04-14T15:19:15.913691+0000","last_undegraded":"2024-04-14T15:19:15.913691+0000","last_fullsized":"2024-04-14T15:19:15.913691+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9","last_scrub_stamp":"2024-04-14T15:17:54.606051+0000","last_deep_scrub":"85'9","last_deep_scrub_stamp":"2024-04-14T15:10:51.950580+0000","last_clean_scrub_stamp":"2024-04-14T15:17:54.606051+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"430'22","reported_seq":444,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.913076+0000","last_change":"2024-04-14T15:18:39.704989+0000","last_active":"2024-04-14T15:19:15.913076+0000","last_peered":"2024-04-14T15:19:15.913076+0000","last_clean":"2024-04-14T15:19:15.913076+0000","last_became_active":"2024-04-14T15:15:44.885409+0000","last_became_peered":"2024-04-14T15:15:44.885409+0000","last_unstale":"2024-04-14T15:19:15.913076+0000","last_undegraded":"2024-04-14T15:19:15.913076+0000","last_fullsized":"2024-04-14T15:19:15.913076+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'22","last_scrub_stamp":"2024-04-14T15:18:39.704883+0000","last_deep_scrub":"430'22","last_deep_scrub_stamp":"2024-04-14T15:18:39.704883+0000","last_clean_scrub_stamp":"2024-04-14T15:18:39.704883+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:51.340270+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":842,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.918648+0000","last_change":"2024-04-14T15:17:53.867764+0000","last_active":"2024-04-14T15:19:15.918648+0000","last_peered":"2024-04-14T15:19:15.918648+0000","last_clean":"2024-04-14T15:19:15.918648+0000","last_became_active":"2024-04-14T15:09:43.021460+0000","last_became_peered":"2024-04-14T15:09:43.021460+0000","last_unstale":"2024-04-14T15:19:15.918648+0000","last_undegraded":"2024-04-14T15:19:15.918648+0000","last_fullsized":"2024-04-14T15:19:15.918648+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:53.867696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:53.867696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":395,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.964884+0000","last_change":"2024-04-14T15:17:54.889170+0000","last_active":"2024-04-14T15:19:15.964884+0000","last_peered":"2024-04-14T15:19:15.964884+0000","last_clean":"2024-04-14T15:19:15.964884+0000","last_became_active":"2024-04-14T15:15:44.888154+0000","last_became_peered":"2024-04-14T15:15:44.888154+0000","last_unstale":"2024-04-14T15:19:15.964884+0000","last_undegraded":"2024-04-14T15:19:15.964884+0000","last_fullsized":"2024-04-14T15:19:15.964884+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:54.889066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:15:40.368219+0000","last_clean_scrub_stamp":"2024-04-14T15:17:54.889066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"91'130","reported_seq":401,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.936987+0000","last_change":"2024-04-14T15:19:05.573366+0000","last_active":"2024-04-14T15:19:15.936987+0000","last_peered":"2024-04-14T15:19:15.936987+0000","last_clean":"2024-04-14T15:19:15.936987+0000","last_became_active":"2024-04-14T15:18:58.182420+0000","last_became_peered":"2024-04-14T15:18:58.182420+0000","last_unstale":"2024-04-14T15:19:15.936987+0000","last_undegraded":"2024-04-14T15:19:15.936987+0000","last_fullsized":"2024-04-14T15:19:15.936987+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:05.573281+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:05.573281+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:17.624412+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":740,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.914313+0000","last_change":"2024-04-14T15:18:37.636498+0000","last_active":"2024-04-14T15:19:15.914313+0000","last_peered":"2024-04-14T15:19:15.914313+0000","last_clean":"2024-04-14T15:19:15.914313+0000","last_became_active":"2024-04-14T15:12:56.355348+0000","last_became_peered":"2024-04-14T15:12:56.355348+0000","last_unstale":"2024-04-14T15:19:15.914313+0000","last_undegraded":"2024-04-14T15:19:15.914313+0000","last_fullsized":"2024-04-14T15:19:15.914313+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:37.636419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:18:37.636419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:06.978194+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"72'9","reported_seq":857,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.930231+0000","last_change":"2024-04-14T15:16:56.663996+0000","last_active":"2024-04-14T15:19:15.930231+0000","last_peered":"2024-04-14T15:19:15.930231+0000","last_clean":"2024-04-14T15:19:15.930231+0000","last_became_active":"2024-04-14T15:09:45.037179+0000","last_became_peered":"2024-04-14T15:09:45.037179+0000","last_unstale":"2024-04-14T15:19:15.930231+0000","last_undegraded":"2024-04-14T15:19:15.930231+0000","last_fullsized":"2024-04-14T15:19:15.930231+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-04-14T15:16:56.663900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:16:56.663900+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":845,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.919297+0000","last_change":"2024-04-14T15:18:36.029705+0000","last_active":"2024-04-14T15:19:15.919297+0000","last_peered":"2024-04-14T15:19:15.919297+0000","last_clean":"2024-04-14T15:19:15.919297+0000","last_became_active":"2024-04-14T15:09:43.020044+0000","last_became_peered":"2024-04-14T15:09:43.020044+0000","last_unstale":"2024-04-14T15:19:15.919297+0000","last_undegraded":"2024-04-14T15:19:15.919297+0000","last_fullsized":"2024-04-14T15:19:15.919297+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:36.029630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:18:36.029630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:49.562846+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":842,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.917115+0000","last_change":"2024-04-14T15:19:11.838298+0000","last_active":"2024-04-14T15:19:15.917115+0000","last_peered":"2024-04-14T15:19:15.917115+0000","last_clean":"2024-04-14T15:19:15.917115+0000","last_became_active":"2024-04-14T15:09:47.046153+0000","last_became_peered":"2024-04-14T15:09:47.046153+0000","last_unstale":"2024-04-14T15:19:15.917115+0000","last_undegraded":"2024-04-14T15:19:15.917115+0000","last_fullsized":"2024-04-14T15:19:15.917115+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:11.838212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:19:11.838212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:19.779219+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":833,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.929914+0000","last_change":"2024-04-14T15:17:37.919665+0000","last_active":"2024-04-14T15:19:15.929914+0000","last_peered":"2024-04-14T15:19:15.929914+0000","last_clean":"2024-04-14T15:19:15.929914+0000","last_became_active":"2024-04-14T15:09:49.061495+0000","last_became_peered":"2024-04-14T15:09:49.061495+0000","last_unstale":"2024-04-14T15:19:15.929914+0000","last_undegraded":"2024-04-14T15:19:15.929914+0000","last_fullsized":"2024-04-14T15:19:15.929914+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:37.919595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:17:37.919595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"91'130","reported_seq":929,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.966014+0000","last_change":"2024-04-14T15:19:06.754212+0000","last_active":"2024-04-14T15:19:15.966014+0000","last_peered":"2024-04-14T15:19:15.966014+0000","last_clean":"2024-04-14T15:19:15.966014+0000","last_became_active":"2024-04-14T15:18:45.919901+0000","last_became_peered":"2024-04-14T15:18:45.919901+0000","last_unstale":"2024-04-14T15:19:15.966014+0000","last_undegraded":"2024-04-14T15:19:15.966014+0000","last_fullsized":"2024-04-14T15:19:15.966014+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:06.754126+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:06.754126+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:30.044936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":740,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.914858+0000","last_change":"2024-04-14T15:18:25.523604+0000","last_active":"2024-04-14T15:19:15.914858+0000","last_peered":"2024-04-14T15:19:15.914858+0000","last_clean":"2024-04-14T15:19:15.914858+0000","last_became_active":"2024-04-14T15:12:56.246062+0000","last_became_peered":"2024-04-14T15:12:56.246062+0000","last_unstale":"2024-04-14T15:19:15.914858+0000","last_undegraded":"2024-04-14T15:19:15.914858+0000","last_fullsized":"2024-04-14T15:19:15.914858+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:25.523520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:18:25.523520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:36.616896+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"222'28","reported_seq":880,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.965408+0000","last_change":"2024-04-14T15:17:44.832902+0000","last_active":"2024-04-14T15:19:15.965408+0000","last_peered":"2024-04-14T15:19:15.965408+0000","last_clean":"2024-04-14T15:19:15.965408+0000","last_became_active":"2024-04-14T15:09:45.178782+0000","last_became_peered":"2024-04-14T15:09:45.178782+0000","last_unstale":"2024-04-14T15:19:15.965408+0000","last_undegraded":"2024-04-14T15:19:15.965408+0000","last_fullsized":"2024-04-14T15:19:15.965408+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'28","last_scrub_stamp":"2024-04-14T15:17:44.832769+0000","last_deep_scrub":"222'28","last_deep_scrub_stamp":"2024-04-14T15:14:50.054931+0000","last_clean_scrub_stamp":"2024-04-14T15:17:44.832769+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":850,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.919046+0000","last_change":"2024-04-14T15:18:36.925573+0000","last_active":"2024-04-14T15:19:15.919046+0000","last_peered":"2024-04-14T15:19:15.919046+0000","last_clean":"2024-04-14T15:19:15.919046+0000","last_became_active":"2024-04-14T15:09:43.020268+0000","last_became_peered":"2024-04-14T15:09:43.020268+0000","last_unstale":"2024-04-14T15:19:15.919046+0000","last_undegraded":"2024-04-14T15:19:15.919046+0000","last_fullsized":"2024-04-14T15:19:15.919046+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:36.925496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:34.342793+0000","last_clean_scrub_stamp":"2024-04-14T15:18:36.925496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:51.029386+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":834,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.918253+0000","last_change":"2024-04-14T15:18:31.989052+0000","last_active":"2024-04-14T15:19:15.918253+0000","last_peered":"2024-04-14T15:19:15.918253+0000","last_clean":"2024-04-14T15:19:15.918253+0000","last_became_active":"2024-04-14T15:09:47.045542+0000","last_became_peered":"2024-04-14T15:09:47.045542+0000","last_unstale":"2024-04-14T15:19:15.918253+0000","last_undegraded":"2024-04-14T15:19:15.918253+0000","last_fullsized":"2024-04-14T15:19:15.918253+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:31.988957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:18:31.988957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:57.227960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":831,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:16.566571+0000","last_change":"2024-04-14T15:19:16.566595+0000","last_active":"2024-04-14T15:19:16.566571+0000","last_peered":"2024-04-14T15:19:16.566571+0000","last_clean":"2024-04-14T15:19:16.566571+0000","last_became_active":"2024-04-14T15:09:49.065397+0000","last_became_peered":"2024-04-14T15:09:49.065397+0000","last_unstale":"2024-04-14T15:19:16.566571+0000","last_undegraded":"2024-04-14T15:19:16.566571+0000","last_fullsized":"2024-04-14T15:19:16.566571+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:16.566496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:19:16.566496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:28.994328+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"91'130","reported_seq":729,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.915796+0000","last_change":"2024-04-14T15:19:04.951104+0000","last_active":"2024-04-14T15:19:15.915796+0000","last_peered":"2024-04-14T15:19:15.915796+0000","last_clean":"2024-04-14T15:19:15.915796+0000","last_became_active":"2024-04-14T15:18:58.181194+0000","last_became_peered":"2024-04-14T15:18:58.181194+0000","last_unstale":"2024-04-14T15:19:15.915796+0000","last_undegraded":"2024-04-14T15:19:15.915796+0000","last_fullsized":"2024-04-14T15:19:15.915796+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:04.950991+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:04.950991+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:06.456834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":747,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.917028+0000","last_change":"2024-04-14T15:19:15.778552+0000","last_active":"2024-04-14T15:19:15.917028+0000","last_peered":"2024-04-14T15:19:15.917028+0000","last_clean":"2024-04-14T15:19:15.917028+0000","last_became_active":"2024-04-14T15:12:56.474322+0000","last_became_peered":"2024-04-14T15:12:56.474322+0000","last_unstale":"2024-04-14T15:19:15.917028+0000","last_undegraded":"2024-04-14T15:19:15.917028+0000","last_fullsized":"2024-04-14T15:19:15.917028+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:15.778469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:19:15.778469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:40.510707+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"72'17","reported_seq":915,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.937879+0000","last_change":"2024-04-14T15:17:48.374589+0000","last_active":"2024-04-14T15:19:15.937879+0000","last_peered":"2024-04-14T15:19:15.937879+0000","last_clean":"2024-04-14T15:19:15.937879+0000","last_became_active":"2024-04-14T15:13:12.933907+0000","last_became_peered":"2024-04-14T15:13:12.933907+0000","last_unstale":"2024-04-14T15:19:15.937879+0000","last_undegraded":"2024-04-14T15:19:15.937879+0000","last_fullsized":"2024-04-14T15:19:15.937879+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'17","last_scrub_stamp":"2024-04-14T15:17:48.374469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:17:48.374469+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":835,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.913890+0000","last_change":"2024-04-14T15:16:16.820151+0000","last_active":"2024-04-14T15:19:15.913890+0000","last_peered":"2024-04-14T15:19:15.913890+0000","last_clean":"2024-04-14T15:19:15.913890+0000","last_became_active":"2024-04-14T15:09:43.022351+0000","last_became_peered":"2024-04-14T15:09:43.022351+0000","last_unstale":"2024-04-14T15:19:15.913890+0000","last_undegraded":"2024-04-14T15:19:15.913890+0000","last_fullsized":"2024-04-14T15:19:15.913890+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:16.820070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:16.820070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":829,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.917726+0000","last_change":"2024-04-14T15:17:54.871693+0000","last_active":"2024-04-14T15:19:15.917726+0000","last_peered":"2024-04-14T15:19:15.917726+0000","last_clean":"2024-04-14T15:19:15.917726+0000","last_became_active":"2024-04-14T15:09:47.042810+0000","last_became_peered":"2024-04-14T15:09:47.042810+0000","last_unstale":"2024-04-14T15:19:15.917726+0000","last_undegraded":"2024-04-14T15:19:15.917726+0000","last_fullsized":"2024-04-14T15:19:15.917726+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:54.871593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:17:54.871593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:21.351148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":830,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.917573+0000","last_change":"2024-04-14T15:16:45.669141+0000","last_active":"2024-04-14T15:19:15.917573+0000","last_peered":"2024-04-14T15:19:15.917573+0000","last_clean":"2024-04-14T15:19:15.917573+0000","last_became_active":"2024-04-14T15:09:49.069594+0000","last_became_peered":"2024-04-14T15:09:49.069594+0000","last_unstale":"2024-04-14T15:19:15.917573+0000","last_undegraded":"2024-04-14T15:19:15.917573+0000","last_fullsized":"2024-04-14T15:19:15.917573+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:45.669063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:45.669063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"91'130","reported_seq":1132,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.912977+0000","last_change":"2024-04-14T15:19:06.490294+0000","last_active":"2024-04-14T15:19:15.912977+0000","last_peered":"2024-04-14T15:19:15.912977+0000","last_clean":"2024-04-14T15:19:15.912977+0000","last_became_active":"2024-04-14T15:18:58.189483+0000","last_became_peered":"2024-04-14T15:18:58.189483+0000","last_unstale":"2024-04-14T15:19:15.912977+0000","last_undegraded":"2024-04-14T15:19:15.912977+0000","last_fullsized":"2024-04-14T15:19:15.912977+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:06.490208+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:06.490208+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:16.848376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":741,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.964756+0000","last_change":"2024-04-14T15:17:46.871883+0000","last_active":"2024-04-14T15:19:15.964756+0000","last_peered":"2024-04-14T15:19:15.964756+0000","last_clean":"2024-04-14T15:19:15.964756+0000","last_became_active":"2024-04-14T15:12:56.289745+0000","last_became_peered":"2024-04-14T15:12:56.289745+0000","last_unstale":"2024-04-14T15:19:15.964756+0000","last_undegraded":"2024-04-14T15:19:15.964756+0000","last_fullsized":"2024-04-14T15:19:15.964756+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:46.871773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:17:46.871773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"179'15","reported_seq":851,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.936833+0000","last_change":"2024-04-14T15:16:49.243096+0000","last_active":"2024-04-14T15:19:15.936833+0000","last_peered":"2024-04-14T15:19:15.936833+0000","last_clean":"2024-04-14T15:19:15.936833+0000","last_became_active":"2024-04-14T15:09:45.029170+0000","last_became_peered":"2024-04-14T15:09:45.029170+0000","last_unstale":"2024-04-14T15:19:15.936833+0000","last_undegraded":"2024-04-14T15:19:15.936833+0000","last_fullsized":"2024-04-14T15:19:15.936833+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'15","last_scrub_stamp":"2024-04-14T15:16:49.243002+0000","last_deep_scrub":"179'15","last_deep_scrub_stamp":"2024-04-14T15:14:07.488422+0000","last_clean_scrub_stamp":"2024-04-14T15:16:49.243002+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":845,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.919307+0000","last_change":"2024-04-14T15:17:51.854742+0000","last_active":"2024-04-14T15:19:15.919307+0000","last_peered":"2024-04-14T15:19:15.919307+0000","last_clean":"2024-04-14T15:19:15.919307+0000","last_became_active":"2024-04-14T15:09:43.018871+0000","last_became_peered":"2024-04-14T15:09:43.018871+0000","last_unstale":"2024-04-14T15:19:15.919307+0000","last_undegraded":"2024-04-14T15:19:15.919307+0000","last_fullsized":"2024-04-14T15:19:15.919307+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:51.854659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:51.854659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"400'9","reported_seq":959,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.918939+0000","last_change":"2024-04-14T15:17:54.845967+0000","last_active":"2024-04-14T15:19:15.918939+0000","last_peered":"2024-04-14T15:19:15.918939+0000","last_clean":"2024-04-14T15:19:15.918939+0000","last_became_active":"2024-04-14T15:09:47.044085+0000","last_became_peered":"2024-04-14T15:09:47.044085+0000","last_unstale":"2024-04-14T15:19:15.918939+0000","last_undegraded":"2024-04-14T15:19:15.918939+0000","last_fullsized":"2024-04-14T15:19:15.918939+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'9","last_scrub_stamp":"2024-04-14T15:17:54.845881+0000","last_deep_scrub":"400'9","last_deep_scrub_stamp":"2024-04-14T15:17:54.845881+0000","last_clean_scrub_stamp":"2024-04-14T15:17:54.845881+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":831,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:18.588637+0000","last_change":"2024-04-14T15:19:18.588681+0000","last_active":"2024-04-14T15:19:18.588637+0000","last_peered":"2024-04-14T15:19:18.588637+0000","last_clean":"2024-04-14T15:19:18.588637+0000","last_became_active":"2024-04-14T15:09:49.065274+0000","last_became_peered":"2024-04-14T15:09:49.065274+0000","last_unstale":"2024-04-14T15:19:18.588637+0000","last_undegraded":"2024-04-14T15:19:18.588637+0000","last_fullsized":"2024-04-14T15:19:18.588637+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:18.588546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:18.588546+0000","last_clean_scrub_stamp":"2024-04-14T15:19:18.588546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:36.014977+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"91'130","reported_seq":737,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.916643+0000","last_change":"2024-04-14T15:19:05.972132+0000","last_active":"2024-04-14T15:19:15.916643+0000","last_peered":"2024-04-14T15:19:15.916643+0000","last_clean":"2024-04-14T15:19:15.916643+0000","last_became_active":"2024-04-14T15:18:58.187649+0000","last_became_peered":"2024-04-14T15:18:58.187649+0000","last_unstale":"2024-04-14T15:19:15.916643+0000","last_undegraded":"2024-04-14T15:19:15.916643+0000","last_fullsized":"2024-04-14T15:19:15.916643+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:05.972050+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:05.972050+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:15.882926+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":742,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.916067+0000","last_change":"2024-04-14T15:19:11.008294+0000","last_active":"2024-04-14T15:19:15.916067+0000","last_peered":"2024-04-14T15:19:15.916067+0000","last_clean":"2024-04-14T15:19:15.916067+0000","last_became_active":"2024-04-14T15:12:56.487615+0000","last_became_peered":"2024-04-14T15:12:56.487615+0000","last_unstale":"2024-04-14T15:19:15.916067+0000","last_undegraded":"2024-04-14T15:19:15.916067+0000","last_fullsized":"2024-04-14T15:19:15.916067+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:11.008225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:19:11.008225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:29.857396+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"398'22","reported_seq":873,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.916570+0000","last_change":"2024-04-14T15:17:52.883984+0000","last_active":"2024-04-14T15:19:15.916570+0000","last_peered":"2024-04-14T15:19:15.916570+0000","last_clean":"2024-04-14T15:19:15.916570+0000","last_became_active":"2024-04-14T15:09:45.033331+0000","last_became_peered":"2024-04-14T15:09:45.033331+0000","last_unstale":"2024-04-14T15:19:15.916570+0000","last_undegraded":"2024-04-14T15:19:15.916570+0000","last_fullsized":"2024-04-14T15:19:15.916570+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'22","last_scrub_stamp":"2024-04-14T15:17:52.883850+0000","last_deep_scrub":"398'22","last_deep_scrub_stamp":"2024-04-14T15:17:52.883850+0000","last_clean_scrub_stamp":"2024-04-14T15:17:52.883850+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":845,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.919487+0000","last_change":"2024-04-14T15:18:25.009179+0000","last_active":"2024-04-14T15:19:15.919487+0000","last_peered":"2024-04-14T15:19:15.919487+0000","last_clean":"2024-04-14T15:19:15.919487+0000","last_became_active":"2024-04-14T15:09:43.020878+0000","last_became_peered":"2024-04-14T15:09:43.020878+0000","last_unstale":"2024-04-14T15:19:15.919487+0000","last_undegraded":"2024-04-14T15:19:15.919487+0000","last_fullsized":"2024-04-14T15:19:15.919487+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:25.009100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:18:25.009100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:54.811984+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"72'8","reported_seq":951,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.937930+0000","last_change":"2024-04-14T15:16:06.933663+0000","last_active":"2024-04-14T15:19:15.937930+0000","last_peered":"2024-04-14T15:19:15.937930+0000","last_clean":"2024-04-14T15:19:15.937930+0000","last_became_active":"2024-04-14T15:09:47.050962+0000","last_became_peered":"2024-04-14T15:09:47.050962+0000","last_unstale":"2024-04-14T15:19:15.937930+0000","last_undegraded":"2024-04-14T15:19:15.937930+0000","last_fullsized":"2024-04-14T15:19:15.937930+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:16:06.933489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:06.933489+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":396,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.918265+0000","last_change":"2024-04-14T15:16:51.199025+0000","last_active":"2024-04-14T15:19:15.918265+0000","last_peered":"2024-04-14T15:19:15.918265+0000","last_clean":"2024-04-14T15:19:15.918265+0000","last_became_active":"2024-04-14T15:15:44.879699+0000","last_became_peered":"2024-04-14T15:15:44.879699+0000","last_unstale":"2024-04-14T15:19:15.918265+0000","last_undegraded":"2024-04-14T15:19:15.918265+0000","last_fullsized":"2024-04-14T15:19:15.918265+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:51.198907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:51.198907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"91'130","reported_seq":928,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.916019+0000","last_change":"2024-04-14T15:19:08.097458+0000","last_active":"2024-04-14T15:19:15.916019+0000","last_peered":"2024-04-14T15:19:15.916019+0000","last_clean":"2024-04-14T15:19:15.916019+0000","last_became_active":"2024-04-14T15:18:58.179785+0000","last_became_peered":"2024-04-14T15:18:58.179785+0000","last_unstale":"2024-04-14T15:19:15.916019+0000","last_undegraded":"2024-04-14T15:19:15.916019+0000","last_fullsized":"2024-04-14T15:19:15.916019+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:08.097372+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:08.097372+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:22.282987+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"233'8","reported_seq":755,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.965326+0000","last_change":"2024-04-14T15:17:50.905698+0000","last_active":"2024-04-14T15:19:15.965326+0000","last_peered":"2024-04-14T15:19:15.965326+0000","last_clean":"2024-04-14T15:19:15.965326+0000","last_became_active":"2024-04-14T15:12:56.381026+0000","last_became_peered":"2024-04-14T15:12:56.381026+0000","last_unstale":"2024-04-14T15:19:15.965326+0000","last_undegraded":"2024-04-14T15:19:15.965326+0000","last_fullsized":"2024-04-14T15:19:15.965326+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'8","last_scrub_stamp":"2024-04-14T15:17:50.905608+0000","last_deep_scrub":"233'8","last_deep_scrub_stamp":"2024-04-14T15:15:01.295800+0000","last_clean_scrub_stamp":"2024-04-14T15:17:50.905608+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"85'16","reported_seq":863,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.937584+0000","last_change":"2024-04-14T15:16:53.143329+0000","last_active":"2024-04-14T15:19:15.937584+0000","last_peered":"2024-04-14T15:19:15.937584+0000","last_clean":"2024-04-14T15:19:15.937584+0000","last_became_active":"2024-04-14T15:09:45.028212+0000","last_became_peered":"2024-04-14T15:09:45.028212+0000","last_unstale":"2024-04-14T15:19:15.937584+0000","last_undegraded":"2024-04-14T15:19:15.937584+0000","last_fullsized":"2024-04-14T15:19:15.937584+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-14T15:16:53.143239+0000","last_deep_scrub":"85'16","last_deep_scrub_stamp":"2024-04-14T15:10:47.456169+0000","last_clean_scrub_stamp":"2024-04-14T15:16:53.143239+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":843,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.913164+0000","last_change":"2024-04-14T15:17:39.465617+0000","last_active":"2024-04-14T15:19:15.913164+0000","last_peered":"2024-04-14T15:19:15.913164+0000","last_clean":"2024-04-14T15:19:15.913164+0000","last_became_active":"2024-04-14T15:09:43.018601+0000","last_became_peered":"2024-04-14T15:09:43.018601+0000","last_unstale":"2024-04-14T15:19:15.913164+0000","last_undegraded":"2024-04-14T15:19:15.913164+0000","last_fullsized":"2024-04-14T15:19:15.913164+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:39.465475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:39.465475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":393,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.938888+0000","last_change":"2024-04-14T15:16:54.187971+0000","last_active":"2024-04-14T15:19:15.938888+0000","last_peered":"2024-04-14T15:19:15.938888+0000","last_clean":"2024-04-14T15:19:15.938888+0000","last_became_active":"2024-04-14T15:15:44.886834+0000","last_became_peered":"2024-04-14T15:15:44.886834+0000","last_unstale":"2024-04-14T15:19:15.938888+0000","last_undegraded":"2024-04-14T15:19:15.938888+0000","last_fullsized":"2024-04-14T15:19:15.938888+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:54.187888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:54.187888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":505,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.925337+0000","last_change":"2024-04-14T15:17:57.214134+0000","last_active":"2024-04-14T15:19:15.925337+0000","last_peered":"2024-04-14T15:19:15.925337+0000","last_clean":"2024-04-14T15:19:15.925337+0000","last_became_active":"2024-04-14T15:15:21.888587+0000","last_became_peered":"2024-04-14T15:15:21.888587+0000","last_unstale":"2024-04-14T15:19:15.925337+0000","last_undegraded":"2024-04-14T15:19:15.925337+0000","last_fullsized":"2024-04-14T15:19:15.925337+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:57.214046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:57.214046+0000","last_clean_scrub_stamp":"2024-04-14T15:17:57.214046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"91'130","reported_seq":911,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.925233+0000","last_change":"2024-04-14T15:19:05.396196+0000","last_active":"2024-04-14T15:19:15.925233+0000","last_peered":"2024-04-14T15:19:15.925233+0000","last_clean":"2024-04-14T15:19:15.925233+0000","last_became_active":"2024-04-14T15:18:40.917537+0000","last_became_peered":"2024-04-14T15:18:40.917537+0000","last_unstale":"2024-04-14T15:19:15.925233+0000","last_undegraded":"2024-04-14T15:19:15.925233+0000","last_fullsized":"2024-04-14T15:19:15.925233+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:05.396090+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:05.396090+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:10.820552+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":739,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.937479+0000","last_change":"2024-04-14T15:16:56.214275+0000","last_active":"2024-04-14T15:19:15.937479+0000","last_peered":"2024-04-14T15:19:15.937479+0000","last_clean":"2024-04-14T15:19:15.937479+0000","last_became_active":"2024-04-14T15:12:56.427276+0000","last_became_peered":"2024-04-14T15:12:56.427276+0000","last_unstale":"2024-04-14T15:19:15.937479+0000","last_undegraded":"2024-04-14T15:19:15.937479+0000","last_fullsized":"2024-04-14T15:19:15.937479+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:56.214178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:16:56.214178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"255'10","reported_seq":594,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.924327+0000","last_change":"2024-04-14T15:16:17.868542+0000","last_active":"2024-04-14T15:19:15.924327+0000","last_peered":"2024-04-14T15:19:15.924327+0000","last_clean":"2024-04-14T15:19:15.924327+0000","last_became_active":"2024-04-14T15:15:21.873747+0000","last_became_peered":"2024-04-14T15:15:21.873747+0000","last_unstale":"2024-04-14T15:19:15.924327+0000","last_undegraded":"2024-04-14T15:19:15.924327+0000","last_fullsized":"2024-04-14T15:19:15.924327+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'10","last_scrub_stamp":"2024-04-14T15:16:17.868428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:17.868428+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"72'14","reported_seq":860,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.936234+0000","last_change":"2024-04-14T15:18:22.545734+0000","last_active":"2024-04-14T15:19:15.936234+0000","last_peered":"2024-04-14T15:19:15.936234+0000","last_clean":"2024-04-14T15:19:15.936234+0000","last_became_active":"2024-04-14T15:09:45.025673+0000","last_became_peered":"2024-04-14T15:09:45.025673+0000","last_unstale":"2024-04-14T15:19:15.936234+0000","last_undegraded":"2024-04-14T15:19:15.936234+0000","last_fullsized":"2024-04-14T15:19:15.936234+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'14","last_scrub_stamp":"2024-04-14T15:18:22.545646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:18:22.545646+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:33.421913+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":844,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.966170+0000","last_change":"2024-04-14T15:18:26.824852+0000","last_active":"2024-04-14T15:19:15.966170+0000","last_peered":"2024-04-14T15:19:15.966170+0000","last_clean":"2024-04-14T15:19:15.966170+0000","last_became_active":"2024-04-14T15:09:43.020503+0000","last_became_peered":"2024-04-14T15:09:43.020503+0000","last_unstale":"2024-04-14T15:19:15.966170+0000","last_undegraded":"2024-04-14T15:19:15.966170+0000","last_fullsized":"2024-04-14T15:19:15.966170+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:26.824773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:18:26.824773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:26.903997+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":823,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.914109+0000","last_change":"2024-04-14T15:16:45.705837+0000","last_active":"2024-04-14T15:19:15.914109+0000","last_peered":"2024-04-14T15:19:15.914109+0000","last_clean":"2024-04-14T15:19:15.914109+0000","last_became_active":"2024-04-14T15:09:49.129610+0000","last_became_peered":"2024-04-14T15:09:49.129610+0000","last_unstale":"2024-04-14T15:19:15.914109+0000","last_undegraded":"2024-04-14T15:19:15.914109+0000","last_fullsized":"2024-04-14T15:19:15.914109+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:45.705761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:45.705761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"91'130","reported_seq":918,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.936883+0000","last_change":"2024-04-14T15:19:04.526327+0000","last_active":"2024-04-14T15:19:15.936883+0000","last_peered":"2024-04-14T15:19:15.936883+0000","last_clean":"2024-04-14T15:19:15.936883+0000","last_became_active":"2024-04-14T15:18:58.179181+0000","last_became_peered":"2024-04-14T15:18:58.179181+0000","last_unstale":"2024-04-14T15:19:15.936883+0000","last_undegraded":"2024-04-14T15:19:15.936883+0000","last_fullsized":"2024-04-14T15:19:15.936883+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:04.526214+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:04.526214+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:30.900098+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":739,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.915202+0000","last_change":"2024-04-14T15:17:47.553179+0000","last_active":"2024-04-14T15:19:15.915202+0000","last_peered":"2024-04-14T15:19:15.915202+0000","last_clean":"2024-04-14T15:19:15.915202+0000","last_became_active":"2024-04-14T15:12:56.355254+0000","last_became_peered":"2024-04-14T15:12:56.355254+0000","last_unstale":"2024-04-14T15:19:15.915202+0000","last_undegraded":"2024-04-14T15:19:15.915202+0000","last_fullsized":"2024-04-14T15:19:15.915202+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:47.553055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:17:47.553055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":835,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.936604+0000","last_change":"2024-04-14T15:18:39.394736+0000","last_active":"2024-04-14T15:19:15.936604+0000","last_peered":"2024-04-14T15:19:15.936604+0000","last_clean":"2024-04-14T15:19:15.936604+0000","last_became_active":"2024-04-14T15:09:47.044610+0000","last_became_peered":"2024-04-14T15:09:47.044610+0000","last_unstale":"2024-04-14T15:19:15.936604+0000","last_undegraded":"2024-04-14T15:19:15.936604+0000","last_fullsized":"2024-04-14T15:19:15.936604+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:39.394655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:39.394655+0000","last_clean_scrub_stamp":"2024-04-14T15:18:39.394655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:52.787637+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"72'18","reported_seq":546,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.924249+0000","last_change":"2024-04-14T15:16:15.913377+0000","last_active":"2024-04-14T15:19:15.924249+0000","last_peered":"2024-04-14T15:19:15.924249+0000","last_clean":"2024-04-14T15:19:15.924249+0000","last_became_active":"2024-04-14T15:15:21.880565+0000","last_became_peered":"2024-04-14T15:15:21.880565+0000","last_unstale":"2024-04-14T15:19:15.924249+0000","last_undegraded":"2024-04-14T15:19:15.924249+0000","last_fullsized":"2024-04-14T15:19:15.924249+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'18","last_scrub_stamp":"2024-04-14T15:16:15.913273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:16:15.913273+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'2","reported_seq":839,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.917559+0000","last_change":"2024-04-14T15:17:48.850178+0000","last_active":"2024-04-14T15:19:15.917559+0000","last_peered":"2024-04-14T15:19:15.917559+0000","last_clean":"2024-04-14T15:19:15.917559+0000","last_became_active":"2024-04-14T15:09:43.018478+0000","last_became_peered":"2024-04-14T15:09:43.018478+0000","last_unstale":"2024-04-14T15:19:15.917559+0000","last_undegraded":"2024-04-14T15:19:15.917559+0000","last_fullsized":"2024-04-14T15:19:15.917559+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'2","last_scrub_stamp":"2024-04-14T15:17:48.850069+0000","last_deep_scrub":"91'2","last_deep_scrub_stamp":"2024-04-14T15:17:48.850069+0000","last_clean_scrub_stamp":"2024-04-14T15:17:48.850069+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":831,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.937895+0000","last_change":"2024-04-14T15:19:14.609181+0000","last_active":"2024-04-14T15:19:15.937895+0000","last_peered":"2024-04-14T15:19:15.937895+0000","last_clean":"2024-04-14T15:19:15.937895+0000","last_became_active":"2024-04-14T15:09:49.065684+0000","last_became_peered":"2024-04-14T15:09:49.065684+0000","last_unstale":"2024-04-14T15:19:15.937895+0000","last_undegraded":"2024-04-14T15:19:15.937895+0000","last_fullsized":"2024-04-14T15:19:15.937895+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:14.609079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:19:14.609079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:23.254598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"91'130","reported_seq":926,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.965794+0000","last_change":"2024-04-14T15:19:05.797940+0000","last_active":"2024-04-14T15:19:15.965794+0000","last_peered":"2024-04-14T15:19:15.965794+0000","last_clean":"2024-04-14T15:19:15.965794+0000","last_became_active":"2024-04-14T15:18:58.182649+0000","last_became_peered":"2024-04-14T15:18:58.182649+0000","last_unstale":"2024-04-14T15:19:15.965794+0000","last_undegraded":"2024-04-14T15:19:15.965794+0000","last_fullsized":"2024-04-14T15:19:15.965794+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:05.797859+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:05.797859+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:17.283641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":741,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.917531+0000","last_change":"2024-04-14T15:18:35.004058+0000","last_active":"2024-04-14T15:19:15.917531+0000","last_peered":"2024-04-14T15:19:15.917531+0000","last_clean":"2024-04-14T15:19:15.917531+0000","last_became_active":"2024-04-14T15:12:56.402023+0000","last_became_peered":"2024-04-14T15:12:56.402023+0000","last_unstale":"2024-04-14T15:19:15.917531+0000","last_undegraded":"2024-04-14T15:19:15.917531+0000","last_fullsized":"2024-04-14T15:19:15.917531+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:35.003968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:35.003968+0000","last_clean_scrub_stamp":"2024-04-14T15:18:35.003968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:53.392985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":836,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.965535+0000","last_change":"2024-04-14T15:17:39.682323+0000","last_active":"2024-04-14T15:19:15.965535+0000","last_peered":"2024-04-14T15:19:15.965535+0000","last_clean":"2024-04-14T15:19:15.965535+0000","last_became_active":"2024-04-14T15:09:47.048427+0000","last_became_peered":"2024-04-14T15:09:47.048427+0000","last_unstale":"2024-04-14T15:19:15.965535+0000","last_undegraded":"2024-04-14T15:19:15.965535+0000","last_fullsized":"2024-04-14T15:19:15.965535+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:39.682203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:17:39.682203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"102'27","reported_seq":885,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.930372+0000","last_change":"2024-04-14T15:17:46.924432+0000","last_active":"2024-04-14T15:19:15.930372+0000","last_peered":"2024-04-14T15:19:15.930372+0000","last_clean":"2024-04-14T15:19:15.930372+0000","last_became_active":"2024-04-14T15:09:45.179182+0000","last_became_peered":"2024-04-14T15:09:45.179182+0000","last_unstale":"2024-04-14T15:19:15.930372+0000","last_undegraded":"2024-04-14T15:19:15.930372+0000","last_fullsized":"2024-04-14T15:19:15.930372+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'27","last_scrub_stamp":"2024-04-14T15:17:46.924330+0000","last_deep_scrub":"102'27","last_deep_scrub_stamp":"2024-04-14T15:17:46.924330+0000","last_clean_scrub_stamp":"2024-04-14T15:17:46.924330+0000","objects_scrubbed":8,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":843,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.914691+0000","last_change":"2024-04-14T15:19:14.569862+0000","last_active":"2024-04-14T15:19:15.914691+0000","last_peered":"2024-04-14T15:19:15.914691+0000","last_clean":"2024-04-14T15:19:15.914691+0000","last_became_active":"2024-04-14T15:09:43.021006+0000","last_became_peered":"2024-04-14T15:09:43.021006+0000","last_unstale":"2024-04-14T15:19:15.914691+0000","last_undegraded":"2024-04-14T15:19:15.914691+0000","last_fullsized":"2024-04-14T15:19:15.914691+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:14.569778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:19:14.569778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:38.152634+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":825,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.919552+0000","last_change":"2024-04-14T15:16:56.206766+0000","last_active":"2024-04-14T15:19:15.919552+0000","last_peered":"2024-04-14T15:19:15.919552+0000","last_clean":"2024-04-14T15:19:15.919552+0000","last_became_active":"2024-04-14T15:09:49.128136+0000","last_became_peered":"2024-04-14T15:09:49.128136+0000","last_unstale":"2024-04-14T15:19:15.919552+0000","last_undegraded":"2024-04-14T15:19:15.919552+0000","last_fullsized":"2024-04-14T15:19:15.919552+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:56.206649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:56.206649+0000","last_clean_scrub_stamp":"2024-04-14T15:16:56.206649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"91'130","reported_seq":915,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.916169+0000","last_change":"2024-04-14T15:19:09.060997+0000","last_active":"2024-04-14T15:19:15.916169+0000","last_peered":"2024-04-14T15:19:15.916169+0000","last_clean":"2024-04-14T15:19:15.916169+0000","last_became_active":"2024-04-14T15:18:40.910133+0000","last_became_peered":"2024-04-14T15:18:40.910133+0000","last_unstale":"2024-04-14T15:19:15.916169+0000","last_undegraded":"2024-04-14T15:19:15.916169+0000","last_fullsized":"2024-04-14T15:19:15.916169+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:09.060906+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:09.060906+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:11.385135+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":445,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.924340+0000","last_change":"2024-04-14T15:18:38.447429+0000","last_active":"2024-04-14T15:19:15.924340+0000","last_peered":"2024-04-14T15:19:15.924340+0000","last_clean":"2024-04-14T15:19:15.924340+0000","last_became_active":"2024-04-14T15:15:21.887769+0000","last_became_peered":"2024-04-14T15:15:21.887769+0000","last_unstale":"2024-04-14T15:19:15.924340+0000","last_undegraded":"2024-04-14T15:19:15.924340+0000","last_fullsized":"2024-04-14T15:19:15.924340+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:38.447351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:18:38.447351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:45.938002+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":395,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.916995+0000","last_change":"2024-04-14T15:17:47.871801+0000","last_active":"2024-04-14T15:19:15.916995+0000","last_peered":"2024-04-14T15:19:15.916995+0000","last_clean":"2024-04-14T15:19:15.916995+0000","last_became_active":"2024-04-14T15:15:44.883694+0000","last_became_peered":"2024-04-14T15:15:44.883694+0000","last_unstale":"2024-04-14T15:19:15.916995+0000","last_undegraded":"2024-04-14T15:19:15.916995+0000","last_fullsized":"2024-04-14T15:19:15.916995+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:47.871722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:17:47.871722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"182'18","reported_seq":865,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.930953+0000","last_change":"2024-04-14T15:17:44.919445+0000","last_active":"2024-04-14T15:19:15.930953+0000","last_peered":"2024-04-14T15:19:15.930953+0000","last_clean":"2024-04-14T15:19:15.930953+0000","last_became_active":"2024-04-14T15:09:45.037231+0000","last_became_peered":"2024-04-14T15:09:45.037231+0000","last_unstale":"2024-04-14T15:19:15.930953+0000","last_undegraded":"2024-04-14T15:19:15.930953+0000","last_fullsized":"2024-04-14T15:19:15.930953+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'18","last_scrub_stamp":"2024-04-14T15:17:44.919358+0000","last_deep_scrub":"182'18","last_deep_scrub_stamp":"2024-04-14T15:14:10.383906+0000","last_clean_scrub_stamp":"2024-04-14T15:17:44.919358+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":843,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.913369+0000","last_change":"2024-04-14T15:17:46.542125+0000","last_active":"2024-04-14T15:19:15.913369+0000","last_peered":"2024-04-14T15:19:15.913369+0000","last_clean":"2024-04-14T15:19:15.913369+0000","last_became_active":"2024-04-14T15:09:43.025095+0000","last_became_peered":"2024-04-14T15:09:43.025095+0000","last_unstale":"2024-04-14T15:19:15.913369+0000","last_undegraded":"2024-04-14T15:19:15.913369+0000","last_fullsized":"2024-04-14T15:19:15.913369+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:46.542030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:46.542030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":852,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.929565+0000","last_change":"2024-04-14T15:17:48.866222+0000","last_active":"2024-04-14T15:19:15.929565+0000","last_peered":"2024-04-14T15:19:15.929565+0000","last_clean":"2024-04-14T15:19:15.929565+0000","last_became_active":"2024-04-14T15:09:43.018509+0000","last_became_peered":"2024-04-14T15:09:43.018509+0000","last_unstale":"2024-04-14T15:19:15.929565+0000","last_undegraded":"2024-04-14T15:19:15.929565+0000","last_fullsized":"2024-04-14T15:19:15.929565+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:48.866146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:48.866146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0018592680000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"355'2","reported_seq":827,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.917808+0000","last_change":"2024-04-14T15:17:03.127190+0000","last_active":"2024-04-14T15:19:15.917808+0000","last_peered":"2024-04-14T15:19:15.917808+0000","last_clean":"2024-04-14T15:19:15.917808+0000","last_became_active":"2024-04-14T15:09:49.065243+0000","last_became_peered":"2024-04-14T15:09:49.065243+0000","last_unstale":"2024-04-14T15:19:15.917808+0000","last_undegraded":"2024-04-14T15:19:15.917808+0000","last_fullsized":"2024-04-14T15:19:15.917808+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'2","last_scrub_stamp":"2024-04-14T15:17:03.127112+0000","last_deep_scrub":"355'2","last_deep_scrub_stamp":"2024-04-14T15:17:03.127112+0000","last_clean_scrub_stamp":"2024-04-14T15:17:03.127112+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":437,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.924760+0000","last_change":"2024-04-14T15:18:32.532077+0000","last_active":"2024-04-14T15:19:15.924760+0000","last_peered":"2024-04-14T15:19:15.924760+0000","last_clean":"2024-04-14T15:19:15.924760+0000","last_became_active":"2024-04-14T15:15:21.890115+0000","last_became_peered":"2024-04-14T15:15:21.890115+0000","last_unstale":"2024-04-14T15:19:15.924760+0000","last_undegraded":"2024-04-14T15:19:15.924760+0000","last_fullsized":"2024-04-14T15:19:15.924760+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:32.531993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:18:32.531993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:40.127160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":827,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.914432+0000","last_change":"2024-04-14T15:16:51.617717+0000","last_active":"2024-04-14T15:19:15.914432+0000","last_peered":"2024-04-14T15:19:15.914432+0000","last_clean":"2024-04-14T15:19:15.914432+0000","last_became_active":"2024-04-14T15:09:47.047201+0000","last_became_peered":"2024-04-14T15:09:47.047201+0000","last_unstale":"2024-04-14T15:19:15.914432+0000","last_undegraded":"2024-04-14T15:19:15.914432+0000","last_fullsized":"2024-04-14T15:19:15.914432+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:51.617615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:51.617615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"90'24","reported_seq":986,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.965699+0000","last_change":"2024-04-14T15:19:02.841789+0000","last_active":"2024-04-14T15:19:15.965699+0000","last_peered":"2024-04-14T15:19:15.965699+0000","last_clean":"2024-04-14T15:19:15.965699+0000","last_became_active":"2024-04-14T15:09:45.179978+0000","last_became_peered":"2024-04-14T15:09:45.179978+0000","last_unstale":"2024-04-14T15:19:15.965699+0000","last_undegraded":"2024-04-14T15:19:15.965699+0000","last_fullsized":"2024-04-14T15:19:15.965699+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'24","last_scrub_stamp":"2024-04-14T15:19:02.841668+0000","last_deep_scrub":"90'24","last_deep_scrub_stamp":"2024-04-14T15:11:25.120692+0000","last_clean_scrub_stamp":"2024-04-14T15:19:02.841668+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:20.808249+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"91'130","reported_seq":1124,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.913299+0000","last_change":"2024-04-14T15:19:08.494753+0000","last_active":"2024-04-14T15:19:15.913299+0000","last_peered":"2024-04-14T15:19:15.913299+0000","last_clean":"2024-04-14T15:19:15.913299+0000","last_became_active":"2024-04-14T15:18:40.911206+0000","last_became_peered":"2024-04-14T15:18:40.911206+0000","last_unstale":"2024-04-14T15:19:15.913299+0000","last_undegraded":"2024-04-14T15:19:15.913299+0000","last_fullsized":"2024-04-14T15:19:15.913299+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:08.494666+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:08.494666+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:25.546717+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":400,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.918343+0000","last_change":"2024-04-14T15:17:35.705777+0000","last_active":"2024-04-14T15:19:15.918343+0000","last_peered":"2024-04-14T15:19:15.918343+0000","last_clean":"2024-04-14T15:19:15.918343+0000","last_became_active":"2024-04-14T15:15:44.885315+0000","last_became_peered":"2024-04-14T15:15:44.885315+0000","last_unstale":"2024-04-14T15:19:15.918343+0000","last_undegraded":"2024-04-14T15:19:15.918343+0000","last_fullsized":"2024-04-14T15:19:15.918343+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:35.705645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:47.627305+0000","last_clean_scrub_stamp":"2024-04-14T15:17:35.705645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"69'2","reported_seq":527,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.924818+0000","last_change":"2024-04-14T15:16:53.972591+0000","last_active":"2024-04-14T15:19:15.924818+0000","last_peered":"2024-04-14T15:19:15.924818+0000","last_clean":"2024-04-14T15:19:15.924818+0000","last_became_active":"2024-04-14T15:15:21.889851+0000","last_became_peered":"2024-04-14T15:15:21.889851+0000","last_unstale":"2024-04-14T15:19:15.924818+0000","last_undegraded":"2024-04-14T15:19:15.924818+0000","last_fullsized":"2024-04-14T15:19:15.924818+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-04-14T15:16:53.972502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:53.972502+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0058046169999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":392,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.926033+0000","last_change":"2024-04-14T15:16:57.935611+0000","last_active":"2024-04-14T15:19:15.926033+0000","last_peered":"2024-04-14T15:19:15.926033+0000","last_clean":"2024-04-14T15:19:15.926033+0000","last_became_active":"2024-04-14T15:15:44.887934+0000","last_became_peered":"2024-04-14T15:15:44.887934+0000","last_unstale":"2024-04-14T15:19:15.926033+0000","last_undegraded":"2024-04-14T15:19:15.926033+0000","last_fullsized":"2024-04-14T15:19:15.926033+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:57.935436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:57.935436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"91'130","reported_seq":1132,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.913388+0000","last_change":"2024-04-14T15:19:07.501791+0000","last_active":"2024-04-14T15:19:15.913388+0000","last_peered":"2024-04-14T15:19:15.913388+0000","last_clean":"2024-04-14T15:19:15.913388+0000","last_became_active":"2024-04-14T15:18:40.912277+0000","last_became_peered":"2024-04-14T15:18:40.912277+0000","last_unstale":"2024-04-14T15:19:15.913388+0000","last_undegraded":"2024-04-14T15:19:15.913388+0000","last_fullsized":"2024-04-14T15:19:15.913388+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:07.501704+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:07.501704+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:10.701083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":437,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.924137+0000","last_change":"2024-04-14T15:18:26.438896+0000","last_active":"2024-04-14T15:19:15.924137+0000","last_peered":"2024-04-14T15:19:15.924137+0000","last_clean":"2024-04-14T15:19:15.924137+0000","last_became_active":"2024-04-14T15:15:21.889970+0000","last_became_peered":"2024-04-14T15:15:21.889970+0000","last_unstale":"2024-04-14T15:19:15.924137+0000","last_undegraded":"2024-04-14T15:19:15.924137+0000","last_fullsized":"2024-04-14T15:19:15.924137+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:26.438820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:18:26.438820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:45.768277+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":834,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.918850+0000","last_change":"2024-04-14T15:17:01.653342+0000","last_active":"2024-04-14T15:19:15.918850+0000","last_peered":"2024-04-14T15:19:15.918850+0000","last_clean":"2024-04-14T15:19:15.918850+0000","last_became_active":"2024-04-14T15:09:47.049457+0000","last_became_peered":"2024-04-14T15:09:47.049457+0000","last_unstale":"2024-04-14T15:19:15.918850+0000","last_undegraded":"2024-04-14T15:19:15.918850+0000","last_fullsized":"2024-04-14T15:19:15.918850+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:01.653271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:17:01.653271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"308'18","reported_seq":865,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.916888+0000","last_change":"2024-04-14T15:16:16.054128+0000","last_active":"2024-04-14T15:19:15.916888+0000","last_peered":"2024-04-14T15:19:15.916888+0000","last_clean":"2024-04-14T15:19:15.916888+0000","last_became_active":"2024-04-14T15:09:45.033883+0000","last_became_peered":"2024-04-14T15:09:45.033883+0000","last_unstale":"2024-04-14T15:19:15.916888+0000","last_undegraded":"2024-04-14T15:19:15.916888+0000","last_fullsized":"2024-04-14T15:19:15.916888+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"308'18","last_scrub_stamp":"2024-04-14T15:16:16.054032+0000","last_deep_scrub":"308'18","last_deep_scrub_stamp":"2024-04-14T15:16:16.054032+0000","last_clean_scrub_stamp":"2024-04-14T15:16:16.054032+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":850,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:16.788853+0000","last_change":"2024-04-14T15:19:16.788873+0000","last_active":"2024-04-14T15:19:16.788853+0000","last_peered":"2024-04-14T15:19:16.788853+0000","last_clean":"2024-04-14T15:19:16.788853+0000","last_became_active":"2024-04-14T15:09:43.021188+0000","last_became_peered":"2024-04-14T15:09:43.021188+0000","last_unstale":"2024-04-14T15:19:16.788853+0000","last_undegraded":"2024-04-14T15:19:16.788853+0000","last_fullsized":"2024-04-14T15:19:16.788853+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:16.788795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:19:16.788795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:35.846721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"73'5","reported_seq":1424,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:17.699220+0000","last_change":"2024-04-14T15:18:26.955802+0000","last_active":"2024-04-14T15:19:17.699220+0000","last_peered":"2024-04-14T15:19:17.699220+0000","last_clean":"2024-04-14T15:19:17.699220+0000","last_became_active":"2024-04-14T15:09:43.016544+0000","last_became_peered":"2024-04-14T15:09:43.016544+0000","last_unstale":"2024-04-14T15:19:17.699220+0000","last_undegraded":"2024-04-14T15:19:17.699220+0000","last_fullsized":"2024-04-14T15:19:17.699220+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-04-14T15:18:26.955711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:18:26.955711+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:54.355074+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00056542099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":569,"num_read_kb":564,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":823,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.914759+0000","last_change":"2024-04-14T15:16:20.827681+0000","last_active":"2024-04-14T15:19:15.914759+0000","last_peered":"2024-04-14T15:19:15.914759+0000","last_clean":"2024-04-14T15:19:15.914759+0000","last_became_active":"2024-04-14T15:09:49.130307+0000","last_became_peered":"2024-04-14T15:09:49.130307+0000","last_unstale":"2024-04-14T15:19:15.914759+0000","last_undegraded":"2024-04-14T15:19:15.914759+0000","last_fullsized":"2024-04-14T15:19:15.914759+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:20.827589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:20.827589+0000","last_clean_scrub_stamp":"2024-04-14T15:16:20.827589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"91'130","reported_seq":746,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.929126+0000","last_change":"2024-04-14T15:19:06.124178+0000","last_active":"2024-04-14T15:19:15.929126+0000","last_peered":"2024-04-14T15:19:15.929126+0000","last_clean":"2024-04-14T15:19:15.929126+0000","last_became_active":"2024-04-14T15:18:40.913215+0000","last_became_peered":"2024-04-14T15:18:40.913215+0000","last_unstale":"2024-04-14T15:19:15.929126+0000","last_undegraded":"2024-04-14T15:19:15.929126+0000","last_fullsized":"2024-04-14T15:19:15.929126+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:06.124071+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:06.124071+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:23.152389+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":445,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.924805+0000","last_change":"2024-04-14T15:19:15.372917+0000","last_active":"2024-04-14T15:19:15.924805+0000","last_peered":"2024-04-14T15:19:15.924805+0000","last_clean":"2024-04-14T15:19:15.924805+0000","last_became_active":"2024-04-14T15:15:21.883717+0000","last_became_peered":"2024-04-14T15:15:21.883717+0000","last_unstale":"2024-04-14T15:19:15.924805+0000","last_undegraded":"2024-04-14T15:19:15.924805+0000","last_fullsized":"2024-04-14T15:19:15.924805+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:15.372837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:19:15.372837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:42.207119+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":829,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.931413+0000","last_change":"2024-04-14T15:16:53.731699+0000","last_active":"2024-04-14T15:19:15.931413+0000","last_peered":"2024-04-14T15:19:15.931413+0000","last_clean":"2024-04-14T15:19:15.931413+0000","last_became_active":"2024-04-14T15:09:47.040185+0000","last_became_peered":"2024-04-14T15:09:47.040185+0000","last_unstale":"2024-04-14T15:19:15.931413+0000","last_undegraded":"2024-04-14T15:19:15.931413+0000","last_fullsized":"2024-04-14T15:19:15.931413+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:53.731610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:53.731610+0000","last_clean_scrub_stamp":"2024-04-14T15:16:53.731610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"72'28","reported_seq":997,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.917479+0000","last_change":"2024-04-14T15:16:43.036263+0000","last_active":"2024-04-14T15:19:15.917479+0000","last_peered":"2024-04-14T15:19:15.917479+0000","last_clean":"2024-04-14T15:19:15.917479+0000","last_became_active":"2024-04-14T15:09:45.034168+0000","last_became_peered":"2024-04-14T15:09:45.034168+0000","last_unstale":"2024-04-14T15:19:15.917479+0000","last_undegraded":"2024-04-14T15:19:15.917479+0000","last_fullsized":"2024-04-14T15:19:15.917479+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'28","last_scrub_stamp":"2024-04-14T15:16:43.036159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:16:43.036159+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":395,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.916569+0000","last_change":"2024-04-14T15:16:19.101437+0000","last_active":"2024-04-14T15:19:15.916569+0000","last_peered":"2024-04-14T15:19:15.916569+0000","last_clean":"2024-04-14T15:19:15.916569+0000","last_became_active":"2024-04-14T15:15:44.885315+0000","last_became_peered":"2024-04-14T15:15:44.885315+0000","last_unstale":"2024-04-14T15:19:15.916569+0000","last_undegraded":"2024-04-14T15:19:15.916569+0000","last_fullsized":"2024-04-14T15:19:15.916569+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:19.101368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:19.101368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":497,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.925938+0000","last_change":"2024-04-14T15:17:44.084142+0000","last_active":"2024-04-14T15:19:15.925938+0000","last_peered":"2024-04-14T15:19:15.925938+0000","last_clean":"2024-04-14T15:19:15.925938+0000","last_became_active":"2024-04-14T15:15:21.874742+0000","last_became_peered":"2024-04-14T15:15:21.874742+0000","last_unstale":"2024-04-14T15:19:15.925938+0000","last_undegraded":"2024-04-14T15:19:15.925938+0000","last_fullsized":"2024-04-14T15:19:15.925938+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:44.084010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:44.084010+0000","last_clean_scrub_stamp":"2024-04-14T15:17:44.084010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"91'130","reported_seq":926,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.966024+0000","last_change":"2024-04-14T15:19:07.770220+0000","last_active":"2024-04-14T15:19:15.966024+0000","last_peered":"2024-04-14T15:19:15.966024+0000","last_clean":"2024-04-14T15:19:15.966024+0000","last_became_active":"2024-04-14T15:18:58.189402+0000","last_became_peered":"2024-04-14T15:18:58.189402+0000","last_unstale":"2024-04-14T15:19:15.966024+0000","last_undegraded":"2024-04-14T15:19:15.966024+0000","last_fullsized":"2024-04-14T15:19:15.966024+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:07.770125+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:07.770125+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:09.567335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":738,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.918995+0000","last_change":"2024-04-14T15:17:00.656174+0000","last_active":"2024-04-14T15:19:15.918995+0000","last_peered":"2024-04-14T15:19:15.918995+0000","last_clean":"2024-04-14T15:19:15.918995+0000","last_became_active":"2024-04-14T15:12:56.350703+0000","last_became_peered":"2024-04-14T15:12:56.350703+0000","last_unstale":"2024-04-14T15:19:15.918995+0000","last_undegraded":"2024-04-14T15:19:15.918995+0000","last_fullsized":"2024-04-14T15:19:15.918995+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:00.656085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:17:00.656085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"72'8","reported_seq":961,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.930172+0000","last_change":"2024-04-14T15:17:04.695161+0000","last_active":"2024-04-14T15:19:15.930172+0000","last_peered":"2024-04-14T15:19:15.930172+0000","last_clean":"2024-04-14T15:19:15.930172+0000","last_became_active":"2024-04-14T15:09:47.047699+0000","last_became_peered":"2024-04-14T15:09:47.047699+0000","last_unstale":"2024-04-14T15:19:15.930172+0000","last_undegraded":"2024-04-14T15:19:15.930172+0000","last_fullsized":"2024-04-14T15:19:15.930172+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:17:04.695079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:17:04.695079+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"72'10","reported_seq":832,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.917330+0000","last_change":"2024-04-14T15:18:38.870798+0000","last_active":"2024-04-14T15:19:15.917330+0000","last_peered":"2024-04-14T15:19:15.917330+0000","last_clean":"2024-04-14T15:19:15.917330+0000","last_became_active":"2024-04-14T15:10:07.764042+0000","last_became_peered":"2024-04-14T15:10:07.764042+0000","last_unstale":"2024-04-14T15:19:15.917330+0000","last_undegraded":"2024-04-14T15:19:15.917330+0000","last_fullsized":"2024-04-14T15:19:15.917330+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-04-14T15:18:38.870710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:18:38.870710+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:50.575761+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":837,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.917345+0000","last_change":"2024-04-14T15:17:40.971492+0000","last_active":"2024-04-14T15:19:15.917345+0000","last_peered":"2024-04-14T15:19:15.917345+0000","last_clean":"2024-04-14T15:19:15.917345+0000","last_became_active":"2024-04-14T15:09:43.022597+0000","last_became_peered":"2024-04-14T15:09:43.022597+0000","last_unstale":"2024-04-14T15:19:15.917345+0000","last_undegraded":"2024-04-14T15:19:15.917345+0000","last_fullsized":"2024-04-14T15:19:15.917345+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:40.971412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:40.971412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":833,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:16.390110+0000","last_change":"2024-04-14T15:19:16.390131+0000","last_active":"2024-04-14T15:19:16.390110+0000","last_peered":"2024-04-14T15:19:16.390110+0000","last_clean":"2024-04-14T15:19:16.390110+0000","last_became_active":"2024-04-14T15:09:49.066648+0000","last_became_peered":"2024-04-14T15:09:49.066648+0000","last_unstale":"2024-04-14T15:19:16.390110+0000","last_undegraded":"2024-04-14T15:19:16.390110+0000","last_fullsized":"2024-04-14T15:19:16.390110+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:16.390049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:19:16.390049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:36.162522+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.c","version":"91'130","reported_seq":931,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.925097+0000","last_change":"2024-04-14T15:19:04.397426+0000","last_active":"2024-04-14T15:19:15.925097+0000","last_peered":"2024-04-14T15:19:15.925097+0000","last_clean":"2024-04-14T15:19:15.925097+0000","last_became_active":"2024-04-14T15:18:58.186036+0000","last_became_peered":"2024-04-14T15:18:58.186036+0000","last_unstale":"2024-04-14T15:19:15.925097+0000","last_undegraded":"2024-04-14T15:19:15.925097+0000","last_fullsized":"2024-04-14T15:19:15.925097+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:04.397343+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:04.397343+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:31.837520+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":741,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.930776+0000","last_change":"2024-04-14T15:18:27.133391+0000","last_active":"2024-04-14T15:19:15.930776+0000","last_peered":"2024-04-14T15:19:15.930776+0000","last_clean":"2024-04-14T15:19:15.930776+0000","last_became_active":"2024-04-14T15:12:56.563074+0000","last_became_peered":"2024-04-14T15:12:56.563074+0000","last_unstale":"2024-04-14T15:19:15.930776+0000","last_undegraded":"2024-04-14T15:19:15.930776+0000","last_fullsized":"2024-04-14T15:19:15.930776+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:27.133310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:18:27.133310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:38.787205+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":834,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.916193+0000","last_change":"2024-04-14T15:17:56.915660+0000","last_active":"2024-04-14T15:19:15.916193+0000","last_peered":"2024-04-14T15:19:15.916193+0000","last_clean":"2024-04-14T15:19:15.916193+0000","last_became_active":"2024-04-14T15:09:47.049005+0000","last_became_peered":"2024-04-14T15:09:47.049005+0000","last_unstale":"2024-04-14T15:19:15.916193+0000","last_undegraded":"2024-04-14T15:19:15.916193+0000","last_fullsized":"2024-04-14T15:19:15.916193+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:56.915544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:56.915544+0000","last_clean_scrub_stamp":"2024-04-14T15:17:56.915544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:24.542571+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"183'17","reported_seq":867,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.936478+0000","last_change":"2024-04-14T15:16:17.997616+0000","last_active":"2024-04-14T15:19:15.936478+0000","last_peered":"2024-04-14T15:19:15.936478+0000","last_clean":"2024-04-14T15:19:15.936478+0000","last_became_active":"2024-04-14T15:09:45.032892+0000","last_became_peered":"2024-04-14T15:09:45.032892+0000","last_unstale":"2024-04-14T15:19:15.936478+0000","last_undegraded":"2024-04-14T15:19:15.936478+0000","last_fullsized":"2024-04-14T15:19:15.936478+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'17","last_scrub_stamp":"2024-04-14T15:16:17.997529+0000","last_deep_scrub":"183'17","last_deep_scrub_stamp":"2024-04-14T15:14:11.577574+0000","last_clean_scrub_stamp":"2024-04-14T15:16:17.997529+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"65'3","reported_seq":853,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.938801+0000","last_change":"2024-04-14T15:16:12.030448+0000","last_active":"2024-04-14T15:19:15.938801+0000","last_peered":"2024-04-14T15:19:15.938801+0000","last_clean":"2024-04-14T15:19:15.938801+0000","last_became_active":"2024-04-14T15:09:43.020490+0000","last_became_peered":"2024-04-14T15:09:43.020490+0000","last_unstale":"2024-04-14T15:19:15.938801+0000","last_undegraded":"2024-04-14T15:19:15.938801+0000","last_fullsized":"2024-04-14T15:19:15.938801+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3","last_scrub_stamp":"2024-04-14T15:16:12.030363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:12.030363+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":528,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":497,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.925086+0000","last_change":"2024-04-14T15:16:58.942591+0000","last_active":"2024-04-14T15:19:15.925086+0000","last_peered":"2024-04-14T15:19:15.925086+0000","last_clean":"2024-04-14T15:19:15.925086+0000","last_became_active":"2024-04-14T15:15:21.876410+0000","last_became_peered":"2024-04-14T15:15:21.876410+0000","last_unstale":"2024-04-14T15:19:15.925086+0000","last_undegraded":"2024-04-14T15:19:15.925086+0000","last_fullsized":"2024-04-14T15:19:15.925086+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:58.942499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:58.942499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"91'130","reported_seq":923,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.915909+0000","last_change":"2024-04-14T15:19:07.091520+0000","last_active":"2024-04-14T15:19:15.915909+0000","last_peered":"2024-04-14T15:19:15.915909+0000","last_clean":"2024-04-14T15:19:15.915909+0000","last_became_active":"2024-04-14T15:18:45.912049+0000","last_became_peered":"2024-04-14T15:18:45.912049+0000","last_unstale":"2024-04-14T15:19:15.915909+0000","last_undegraded":"2024-04-14T15:19:15.915909+0000","last_fullsized":"2024-04-14T15:19:15.915909+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:07.091439+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:07.091439+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:25.022161+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":741,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.929437+0000","last_change":"2024-04-14T15:16:43.696614+0000","last_active":"2024-04-14T15:19:15.929437+0000","last_peered":"2024-04-14T15:19:15.929437+0000","last_clean":"2024-04-14T15:19:15.929437+0000","last_became_active":"2024-04-14T15:12:56.551177+0000","last_became_peered":"2024-04-14T15:12:56.551177+0000","last_unstale":"2024-04-14T15:19:15.929437+0000","last_undegraded":"2024-04-14T15:19:15.929437+0000","last_fullsized":"2024-04-14T15:19:15.929437+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:43.696536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:16:43.696536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"347'13","reported_seq":852,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.931255+0000","last_change":"2024-04-14T15:16:55.694703+0000","last_active":"2024-04-14T15:19:15.931255+0000","last_peered":"2024-04-14T15:19:15.931255+0000","last_clean":"2024-04-14T15:19:15.931255+0000","last_became_active":"2024-04-14T15:09:45.180375+0000","last_became_peered":"2024-04-14T15:09:45.180375+0000","last_unstale":"2024-04-14T15:19:15.931255+0000","last_undegraded":"2024-04-14T15:19:15.931255+0000","last_fullsized":"2024-04-14T15:19:15.931255+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"347'13","last_scrub_stamp":"2024-04-14T15:16:55.694599+0000","last_deep_scrub":"347'13","last_deep_scrub_stamp":"2024-04-14T15:16:55.694599+0000","last_clean_scrub_stamp":"2024-04-14T15:16:55.694599+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":842,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.916113+0000","last_change":"2024-04-14T15:16:41.677902+0000","last_active":"2024-04-14T15:19:15.916113+0000","last_peered":"2024-04-14T15:19:15.916113+0000","last_clean":"2024-04-14T15:19:15.916113+0000","last_became_active":"2024-04-14T15:09:43.016788+0000","last_became_peered":"2024-04-14T15:09:43.016788+0000","last_unstale":"2024-04-14T15:19:15.916113+0000","last_undegraded":"2024-04-14T15:19:15.916113+0000","last_fullsized":"2024-04-14T15:19:15.916113+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:41.677826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:41.677826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":517,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.926017+0000","last_change":"2024-04-14T15:18:40.385814+0000","last_active":"2024-04-14T15:19:15.926017+0000","last_peered":"2024-04-14T15:19:15.926017+0000","last_clean":"2024-04-14T15:19:15.926017+0000","last_became_active":"2024-04-14T15:15:21.873580+0000","last_became_peered":"2024-04-14T15:15:21.873580+0000","last_unstale":"2024-04-14T15:19:15.926017+0000","last_undegraded":"2024-04-14T15:19:15.926017+0000","last_fullsized":"2024-04-14T15:19:15.926017+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:40.385736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:18:40.385736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:01.230073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":830,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.918937+0000","last_change":"2024-04-14T15:16:47.671808+0000","last_active":"2024-04-14T15:19:15.918937+0000","last_peered":"2024-04-14T15:19:15.918937+0000","last_clean":"2024-04-14T15:19:15.918937+0000","last_became_active":"2024-04-14T15:09:49.064296+0000","last_became_peered":"2024-04-14T15:09:49.064296+0000","last_unstale":"2024-04-14T15:19:15.918937+0000","last_undegraded":"2024-04-14T15:19:15.918937+0000","last_fullsized":"2024-04-14T15:19:15.918937+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:47.671717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:47.671717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.12","version":"91'130","reported_seq":395,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.916121+0000","last_change":"2024-04-14T15:19:06.098758+0000","last_active":"2024-04-14T15:19:15.916121+0000","last_peered":"2024-04-14T15:19:15.916121+0000","last_clean":"2024-04-14T15:19:15.916121+0000","last_became_active":"2024-04-14T15:18:45.918413+0000","last_became_peered":"2024-04-14T15:18:45.918413+0000","last_unstale":"2024-04-14T15:19:15.916121+0000","last_undegraded":"2024-04-14T15:19:15.916121+0000","last_fullsized":"2024-04-14T15:19:15.916121+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:06.098672+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:06.098672+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:32.614691+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":738,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.915966+0000","last_change":"2024-04-14T15:17:36.728543+0000","last_active":"2024-04-14T15:19:15.915966+0000","last_peered":"2024-04-14T15:19:15.915966+0000","last_clean":"2024-04-14T15:19:15.915966+0000","last_became_active":"2024-04-14T15:12:56.340066+0000","last_became_peered":"2024-04-14T15:12:56.340066+0000","last_unstale":"2024-04-14T15:19:15.915966+0000","last_undegraded":"2024-04-14T15:19:15.915966+0000","last_fullsized":"2024-04-14T15:19:15.915966+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:36.728465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:17:36.728465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.17","version":"72'6","reported_seq":845,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.925915+0000","last_change":"2024-04-14T15:17:42.019838+0000","last_active":"2024-04-14T15:19:15.925915+0000","last_peered":"2024-04-14T15:19:15.925915+0000","last_clean":"2024-04-14T15:19:15.925915+0000","last_became_active":"2024-04-14T15:09:45.180037+0000","last_became_peered":"2024-04-14T15:09:45.180037+0000","last_unstale":"2024-04-14T15:19:15.925915+0000","last_undegraded":"2024-04-14T15:19:15.925915+0000","last_fullsized":"2024-04-14T15:19:15.925915+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'6","last_scrub_stamp":"2024-04-14T15:17:42.019717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:17:42.019717+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":844,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.964792+0000","last_change":"2024-04-14T15:16:59.640700+0000","last_active":"2024-04-14T15:19:15.964792+0000","last_peered":"2024-04-14T15:19:15.964792+0000","last_clean":"2024-04-14T15:19:15.964792+0000","last_became_active":"2024-04-14T15:09:43.018987+0000","last_became_peered":"2024-04-14T15:09:43.018987+0000","last_unstale":"2024-04-14T15:19:15.964792+0000","last_undegraded":"2024-04-14T15:19:15.964792+0000","last_fullsized":"2024-04-14T15:19:15.964792+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:59.640565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:59.640565+0000","last_clean_scrub_stamp":"2024-04-14T15:16:59.640565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":391,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.925862+0000","last_change":"2024-04-14T15:17:47.072247+0000","last_active":"2024-04-14T15:19:15.925862+0000","last_peered":"2024-04-14T15:19:15.925862+0000","last_clean":"2024-04-14T15:19:15.925862+0000","last_became_active":"2024-04-14T15:15:44.889786+0000","last_became_peered":"2024-04-14T15:15:44.889786+0000","last_unstale":"2024-04-14T15:19:15.925862+0000","last_undegraded":"2024-04-14T15:19:15.925862+0000","last_fullsized":"2024-04-14T15:19:15.925862+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:47.072170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:13.735024+0000","last_clean_scrub_stamp":"2024-04-14T15:17:47.072170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":822,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.914955+0000","last_change":"2024-04-14T15:16:54.720823+0000","last_active":"2024-04-14T15:19:15.914955+0000","last_peered":"2024-04-14T15:19:15.914955+0000","last_clean":"2024-04-14T15:19:15.914955+0000","last_became_active":"2024-04-14T15:09:49.130438+0000","last_became_peered":"2024-04-14T15:09:49.130438+0000","last_unstale":"2024-04-14T15:19:15.914955+0000","last_undegraded":"2024-04-14T15:19:15.914955+0000","last_fullsized":"2024-04-14T15:19:15.914955+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:54.720722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:54.720722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.11","version":"91'130","reported_seq":754,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.928801+0000","last_change":"2024-04-14T15:19:05.136092+0000","last_active":"2024-04-14T15:19:15.928801+0000","last_peered":"2024-04-14T15:19:15.928801+0000","last_clean":"2024-04-14T15:19:15.928801+0000","last_became_active":"2024-04-14T15:18:45.915385+0000","last_became_peered":"2024-04-14T15:18:45.915385+0000","last_unstale":"2024-04-14T15:19:15.928801+0000","last_undegraded":"2024-04-14T15:19:15.928801+0000","last_fullsized":"2024-04-14T15:19:15.928801+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:05.136001+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:05.136001+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:15.595631+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":739,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.937038+0000","last_change":"2024-04-14T15:17:46.369320+0000","last_active":"2024-04-14T15:19:15.937038+0000","last_peered":"2024-04-14T15:19:15.937038+0000","last_clean":"2024-04-14T15:19:15.937038+0000","last_became_active":"2024-04-14T15:12:56.203577+0000","last_became_peered":"2024-04-14T15:12:56.203577+0000","last_unstale":"2024-04-14T15:19:15.937038+0000","last_undegraded":"2024-04-14T15:19:15.937038+0000","last_fullsized":"2024-04-14T15:19:15.937038+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:46.369231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:46.369231+0000","last_clean_scrub_stamp":"2024-04-14T15:17:46.369231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"72'10","reported_seq":536,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.925357+0000","last_change":"2024-04-14T15:18:36.534967+0000","last_active":"2024-04-14T15:19:15.925357+0000","last_peered":"2024-04-14T15:19:15.925357+0000","last_clean":"2024-04-14T15:19:15.925357+0000","last_became_active":"2024-04-14T15:15:21.877667+0000","last_became_peered":"2024-04-14T15:15:21.877667+0000","last_unstale":"2024-04-14T15:19:15.925357+0000","last_undegraded":"2024-04-14T15:19:15.925357+0000","last_fullsized":"2024-04-14T15:19:15.925357+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-04-14T15:18:36.534874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:18:36.534874+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:02.823104+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":391,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.924945+0000","last_change":"2024-04-14T15:16:47.075044+0000","last_active":"2024-04-14T15:19:15.924945+0000","last_peered":"2024-04-14T15:19:15.924945+0000","last_clean":"2024-04-14T15:19:15.924945+0000","last_became_active":"2024-04-14T15:15:44.887452+0000","last_became_peered":"2024-04-14T15:15:44.887452+0000","last_unstale":"2024-04-14T15:19:15.924945+0000","last_undegraded":"2024-04-14T15:19:15.924945+0000","last_fullsized":"2024-04-14T15:19:15.924945+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:47.074950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:47.074950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"280'9","reported_seq":463,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.918452+0000","last_change":"2024-04-14T15:17:50.908130+0000","last_active":"2024-04-14T15:19:15.918452+0000","last_peered":"2024-04-14T15:19:15.918452+0000","last_clean":"2024-04-14T15:19:15.918452+0000","last_became_active":"2024-04-14T15:15:44.882916+0000","last_became_peered":"2024-04-14T15:15:44.882916+0000","last_unstale":"2024-04-14T15:19:15.918452+0000","last_undegraded":"2024-04-14T15:19:15.918452+0000","last_fullsized":"2024-04-14T15:19:15.918452+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"280'9","last_scrub_stamp":"2024-04-14T15:17:50.908054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:17:50.908054+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:20.243489+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":824,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.930856+0000","last_change":"2024-04-14T15:16:54.724807+0000","last_active":"2024-04-14T15:19:15.930856+0000","last_peered":"2024-04-14T15:19:15.930856+0000","last_clean":"2024-04-14T15:19:15.930856+0000","last_became_active":"2024-04-14T15:09:49.066068+0000","last_became_peered":"2024-04-14T15:09:49.066068+0000","last_unstale":"2024-04-14T15:19:15.930856+0000","last_undegraded":"2024-04-14T15:19:15.930856+0000","last_fullsized":"2024-04-14T15:19:15.930856+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:54.724681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:54.724681+0000","last_clean_scrub_stamp":"2024-04-14T15:16:54.724681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"91'130","reported_seq":1132,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.912899+0000","last_change":"2024-04-14T15:19:05.496449+0000","last_active":"2024-04-14T15:19:15.912899+0000","last_peered":"2024-04-14T15:19:15.912899+0000","last_clean":"2024-04-14T15:19:15.912899+0000","last_became_active":"2024-04-14T15:18:45.917220+0000","last_became_peered":"2024-04-14T15:18:45.917220+0000","last_unstale":"2024-04-14T15:19:15.912899+0000","last_undegraded":"2024-04-14T15:19:15.912899+0000","last_fullsized":"2024-04-14T15:19:15.912899+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:05.496303+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:05.496303+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:22.805010+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":391,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.924610+0000","last_change":"2024-04-14T15:18:30.438135+0000","last_active":"2024-04-14T15:19:15.924610+0000","last_peered":"2024-04-14T15:19:15.924610+0000","last_clean":"2024-04-14T15:19:15.924610+0000","last_became_active":"2024-04-14T15:15:44.889106+0000","last_became_peered":"2024-04-14T15:15:44.889106+0000","last_unstale":"2024-04-14T15:19:15.924610+0000","last_undegraded":"2024-04-14T15:19:15.924610+0000","last_fullsized":"2024-04-14T15:19:15.924610+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:30.438001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:18:30.438001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:48.397871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"183'13","reported_seq":672,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.914828+0000","last_change":"2024-04-14T15:17:37.421888+0000","last_active":"2024-04-14T15:19:15.914828+0000","last_peered":"2024-04-14T15:19:15.914828+0000","last_clean":"2024-04-14T15:19:15.914828+0000","last_became_active":"2024-04-14T15:15:44.884211+0000","last_became_peered":"2024-04-14T15:15:44.884211+0000","last_unstale":"2024-04-14T15:19:15.914828+0000","last_undegraded":"2024-04-14T15:19:15.914828+0000","last_fullsized":"2024-04-14T15:19:15.914828+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'13","last_scrub_stamp":"2024-04-14T15:17:37.421739+0000","last_deep_scrub":"183'13","last_deep_scrub_stamp":"2024-04-14T15:14:11.755252+0000","last_clean_scrub_stamp":"2024-04-14T15:17:37.421739+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":845,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.930303+0000","last_change":"2024-04-14T15:17:02.670294+0000","last_active":"2024-04-14T15:19:15.930303+0000","last_peered":"2024-04-14T15:19:15.930303+0000","last_clean":"2024-04-14T15:19:15.930303+0000","last_became_active":"2024-04-14T15:09:43.019301+0000","last_became_peered":"2024-04-14T15:09:43.019301+0000","last_unstale":"2024-04-14T15:19:15.930303+0000","last_undegraded":"2024-04-14T15:19:15.930303+0000","last_fullsized":"2024-04-14T15:19:15.930303+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:02.670213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:02.670213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"255'10","reported_seq":596,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.925637+0000","last_change":"2024-04-14T15:16:59.946889+0000","last_active":"2024-04-14T15:19:15.925637+0000","last_peered":"2024-04-14T15:19:15.925637+0000","last_clean":"2024-04-14T15:19:15.925637+0000","last_became_active":"2024-04-14T15:15:21.884468+0000","last_became_peered":"2024-04-14T15:15:21.884468+0000","last_unstale":"2024-04-14T15:19:15.925637+0000","last_undegraded":"2024-04-14T15:19:15.925637+0000","last_fullsized":"2024-04-14T15:19:15.925637+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'10","last_scrub_stamp":"2024-04-14T15:16:59.946804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:59.946804+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":830,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.938917+0000","last_change":"2024-04-14T15:19:15.642110+0000","last_active":"2024-04-14T15:19:15.938917+0000","last_peered":"2024-04-14T15:19:15.938917+0000","last_clean":"2024-04-14T15:19:15.938917+0000","last_became_active":"2024-04-14T15:09:49.065242+0000","last_became_peered":"2024-04-14T15:09:49.065242+0000","last_unstale":"2024-04-14T15:19:15.938917+0000","last_undegraded":"2024-04-14T15:19:15.938917+0000","last_fullsized":"2024-04-14T15:19:15.938917+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:15.642017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:19:15.642017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:37.747416+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.17","version":"91'130","reported_seq":366,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.930836+0000","last_change":"2024-04-14T15:19:03.143086+0000","last_active":"2024-04-14T15:19:15.930836+0000","last_peered":"2024-04-14T15:19:15.930836+0000","last_clean":"2024-04-14T15:19:15.930836+0000","last_became_active":"2024-04-14T15:19:02.936495+0000","last_became_peered":"2024-04-14T15:19:02.936495+0000","last_unstale":"2024-04-14T15:19:15.930836+0000","last_undegraded":"2024-04-14T15:19:15.930836+0000","last_fullsized":"2024-04-14T15:19:15.930836+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":446,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:03.143008+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:03.143008+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:11.581275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":739,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.914970+0000","last_change":"2024-04-14T15:18:40.735708+0000","last_active":"2024-04-14T15:19:15.914970+0000","last_peered":"2024-04-14T15:19:15.914970+0000","last_clean":"2024-04-14T15:19:15.914970+0000","last_became_active":"2024-04-14T15:12:56.389204+0000","last_became_peered":"2024-04-14T15:12:56.389204+0000","last_unstale":"2024-04-14T15:19:15.914970+0000","last_undegraded":"2024-04-14T15:19:15.914970+0000","last_fullsized":"2024-04-14T15:19:15.914970+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:40.735623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:18:40.735623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:41.555549+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"344'13","reported_seq":371,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.965174+0000","last_change":"2024-04-14T15:19:10.762280+0000","last_active":"2024-04-14T15:19:15.965174+0000","last_peered":"2024-04-14T15:19:15.965174+0000","last_clean":"2024-04-14T15:19:15.965174+0000","last_became_active":"2024-04-14T15:16:24.993130+0000","last_became_peered":"2024-04-14T15:16:24.993130+0000","last_unstale":"2024-04-14T15:19:15.965174+0000","last_undegraded":"2024-04-14T15:19:15.965174+0000","last_fullsized":"2024-04-14T15:19:15.965174+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"344'13","last_scrub_stamp":"2024-04-14T15:19:10.762147+0000","last_deep_scrub":"344'13","last_deep_scrub_stamp":"2024-04-14T15:16:52.738163+0000","last_clean_scrub_stamp":"2024-04-14T15:19:10.762147+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:28.923046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":844,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.965363+0000","last_change":"2024-04-14T15:17:04.625913+0000","last_active":"2024-04-14T15:19:15.965363+0000","last_peered":"2024-04-14T15:19:15.965363+0000","last_clean":"2024-04-14T15:19:15.965363+0000","last_became_active":"2024-04-14T15:09:43.022480+0000","last_became_peered":"2024-04-14T15:09:43.022480+0000","last_unstale":"2024-04-14T15:19:15.965363+0000","last_undegraded":"2024-04-14T15:19:15.965363+0000","last_fullsized":"2024-04-14T15:19:15.965363+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:04.625824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:04.625824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":509,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.924953+0000","last_change":"2024-04-14T15:18:33.485536+0000","last_active":"2024-04-14T15:19:15.924953+0000","last_peered":"2024-04-14T15:19:15.924953+0000","last_clean":"2024-04-14T15:19:15.924953+0000","last_became_active":"2024-04-14T15:15:21.882643+0000","last_became_peered":"2024-04-14T15:15:21.882643+0000","last_unstale":"2024-04-14T15:19:15.924953+0000","last_undegraded":"2024-04-14T15:19:15.924953+0000","last_fullsized":"2024-04-14T15:19:15.924953+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:33.485411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:33.485411+0000","last_clean_scrub_stamp":"2024-04-14T15:18:33.485411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:34.513174+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":836,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.928907+0000","last_change":"2024-04-14T15:18:24.039316+0000","last_active":"2024-04-14T15:19:15.928907+0000","last_peered":"2024-04-14T15:19:15.928907+0000","last_clean":"2024-04-14T15:19:15.928907+0000","last_became_active":"2024-04-14T15:09:49.059711+0000","last_became_peered":"2024-04-14T15:09:49.059711+0000","last_unstale":"2024-04-14T15:19:15.928907+0000","last_undegraded":"2024-04-14T15:19:15.928907+0000","last_fullsized":"2024-04-14T15:19:15.928907+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:24.039237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:18:24.039237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:25.190541+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"91'130","reported_seq":362,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.917270+0000","last_change":"2024-04-14T15:19:04.045598+0000","last_active":"2024-04-14T15:19:15.917270+0000","last_peered":"2024-04-14T15:19:15.917270+0000","last_clean":"2024-04-14T15:19:15.917270+0000","last_became_active":"2024-04-14T15:19:02.938648+0000","last_became_peered":"2024-04-14T15:19:02.938648+0000","last_unstale":"2024-04-14T15:19:15.917270+0000","last_undegraded":"2024-04-14T15:19:15.917270+0000","last_fullsized":"2024-04-14T15:19:15.917270+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":446,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:04.045513+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:04.045513+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:30.423540+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":740,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.965238+0000","last_change":"2024-04-14T15:17:51.921346+0000","last_active":"2024-04-14T15:19:15.965238+0000","last_peered":"2024-04-14T15:19:15.965238+0000","last_clean":"2024-04-14T15:19:15.965238+0000","last_became_active":"2024-04-14T15:12:56.396294+0000","last_became_peered":"2024-04-14T15:12:56.396294+0000","last_unstale":"2024-04-14T15:19:15.965238+0000","last_undegraded":"2024-04-14T15:19:15.965238+0000","last_fullsized":"2024-04-14T15:19:15.965238+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:51.921269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:17:51.921269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"464'16","reported_seq":863,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:17.590078+0000","last_change":"2024-04-14T15:19:17.590124+0000","last_active":"2024-04-14T15:19:17.590078+0000","last_peered":"2024-04-14T15:19:17.590078+0000","last_clean":"2024-04-14T15:19:17.590078+0000","last_became_active":"2024-04-14T15:09:45.026065+0000","last_became_peered":"2024-04-14T15:09:45.026065+0000","last_unstale":"2024-04-14T15:19:17.590078+0000","last_undegraded":"2024-04-14T15:19:17.590078+0000","last_fullsized":"2024-04-14T15:19:17.590078+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"464'16","last_scrub_stamp":"2024-04-14T15:19:17.589999+0000","last_deep_scrub":"464'16","last_deep_scrub_stamp":"2024-04-14T15:19:17.589999+0000","last_clean_scrub_stamp":"2024-04-14T15:19:17.589999+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:35.929485+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":845,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.919299+0000","last_change":"2024-04-14T15:17:34.986735+0000","last_active":"2024-04-14T15:19:15.919299+0000","last_peered":"2024-04-14T15:19:15.919299+0000","last_clean":"2024-04-14T15:19:15.919299+0000","last_became_active":"2024-04-14T15:09:43.023583+0000","last_became_peered":"2024-04-14T15:09:43.023583+0000","last_unstale":"2024-04-14T15:19:15.919299+0000","last_undegraded":"2024-04-14T15:19:15.919299+0000","last_fullsized":"2024-04-14T15:19:15.919299+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:34.986655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:34.986655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":829,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.919308+0000","last_change":"2024-04-14T15:17:00.205888+0000","last_active":"2024-04-14T15:19:15.919308+0000","last_peered":"2024-04-14T15:19:15.919308+0000","last_clean":"2024-04-14T15:19:15.919308+0000","last_became_active":"2024-04-14T15:09:47.041683+0000","last_became_peered":"2024-04-14T15:09:47.041683+0000","last_unstale":"2024-04-14T15:19:15.919308+0000","last_undegraded":"2024-04-14T15:19:15.919308+0000","last_fullsized":"2024-04-14T15:19:15.919308+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:00.205808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:08.486217+0000","last_clean_scrub_stamp":"2024-04-14T15:17:00.205808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":505,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.924199+0000","last_change":"2024-04-14T15:16:50.021545+0000","last_active":"2024-04-14T15:19:15.924199+0000","last_peered":"2024-04-14T15:19:15.924199+0000","last_clean":"2024-04-14T15:19:15.924199+0000","last_became_active":"2024-04-14T15:15:21.889567+0000","last_became_peered":"2024-04-14T15:15:21.889567+0000","last_unstale":"2024-04-14T15:19:15.924199+0000","last_undegraded":"2024-04-14T15:19:15.924199+0000","last_fullsized":"2024-04-14T15:19:15.924199+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:50.021470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:50.021470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.15","version":"91'130","reported_seq":364,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.914214+0000","last_change":"2024-04-14T15:19:04.485222+0000","last_active":"2024-04-14T15:19:15.914214+0000","last_peered":"2024-04-14T15:19:15.914214+0000","last_clean":"2024-04-14T15:19:15.914214+0000","last_became_active":"2024-04-14T15:19:02.934529+0000","last_became_peered":"2024-04-14T15:19:02.934529+0000","last_unstale":"2024-04-14T15:19:15.914214+0000","last_undegraded":"2024-04-14T15:19:15.914214+0000","last_fullsized":"2024-04-14T15:19:15.914214+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":446,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:04.485114+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:04.485114+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:29.092721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":739,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.917231+0000","last_change":"2024-04-14T15:17:03.685135+0000","last_active":"2024-04-14T15:19:15.917231+0000","last_peered":"2024-04-14T15:19:15.917231+0000","last_clean":"2024-04-14T15:19:15.917231+0000","last_became_active":"2024-04-14T15:12:56.387723+0000","last_became_peered":"2024-04-14T15:12:56.387723+0000","last_unstale":"2024-04-14T15:19:15.917231+0000","last_undegraded":"2024-04-14T15:19:15.917231+0000","last_fullsized":"2024-04-14T15:19:15.917231+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:03.685045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:03.685045+0000","last_clean_scrub_stamp":"2024-04-14T15:17:03.685045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.10","version":"72'4","reported_seq":843,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.924109+0000","last_change":"2024-04-14T15:17:53.181132+0000","last_active":"2024-04-14T15:19:15.924109+0000","last_peered":"2024-04-14T15:19:15.924109+0000","last_clean":"2024-04-14T15:19:15.924109+0000","last_became_active":"2024-04-14T15:09:45.038815+0000","last_became_peered":"2024-04-14T15:09:45.038815+0000","last_unstale":"2024-04-14T15:19:15.924109+0000","last_undegraded":"2024-04-14T15:19:15.924109+0000","last_fullsized":"2024-04-14T15:19:15.924109+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4","last_scrub_stamp":"2024-04-14T15:17:53.181053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:17:53.181053+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":394,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.965972+0000","last_change":"2024-04-14T15:17:48.845163+0000","last_active":"2024-04-14T15:19:15.965972+0000","last_peered":"2024-04-14T15:19:15.965972+0000","last_clean":"2024-04-14T15:19:15.965972+0000","last_became_active":"2024-04-14T15:15:44.884730+0000","last_became_peered":"2024-04-14T15:15:44.884730+0000","last_unstale":"2024-04-14T15:19:15.965972+0000","last_undegraded":"2024-04-14T15:19:15.965972+0000","last_fullsized":"2024-04-14T15:19:15.965972+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:48.845081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:08.712289+0000","last_clean_scrub_stamp":"2024-04-14T15:17:48.845081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":844,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.964916+0000","last_change":"2024-04-14T15:18:29.830291+0000","last_active":"2024-04-14T15:19:15.964916+0000","last_peered":"2024-04-14T15:19:15.964916+0000","last_clean":"2024-04-14T15:19:15.964916+0000","last_became_active":"2024-04-14T15:09:47.041074+0000","last_became_peered":"2024-04-14T15:09:47.041074+0000","last_unstale":"2024-04-14T15:19:15.964916+0000","last_undegraded":"2024-04-14T15:19:15.964916+0000","last_fullsized":"2024-04-14T15:19:15.964916+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:29.830198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:18:29.830198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:48.054530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":823,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.913862+0000","last_change":"2024-04-14T15:16:44.728025+0000","last_active":"2024-04-14T15:19:15.913862+0000","last_peered":"2024-04-14T15:19:15.913862+0000","last_clean":"2024-04-14T15:19:15.913862+0000","last_became_active":"2024-04-14T15:09:49.061359+0000","last_became_peered":"2024-04-14T15:09:49.061359+0000","last_unstale":"2024-04-14T15:19:15.913862+0000","last_undegraded":"2024-04-14T15:19:15.913862+0000","last_fullsized":"2024-04-14T15:19:15.913862+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:44.727907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:44.727907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"91'130","reported_seq":914,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.918460+0000","last_change":"2024-04-14T15:19:06.939565+0000","last_active":"2024-04-14T15:19:15.918460+0000","last_peered":"2024-04-14T15:19:15.918460+0000","last_clean":"2024-04-14T15:19:15.918460+0000","last_became_active":"2024-04-14T15:18:45.913983+0000","last_became_peered":"2024-04-14T15:18:45.913983+0000","last_unstale":"2024-04-14T15:19:15.918460+0000","last_undegraded":"2024-04-14T15:19:15.918460+0000","last_fullsized":"2024-04-14T15:19:15.918460+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:06.939444+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:06.939444+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:20.862256+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":740,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.915072+0000","last_change":"2024-04-14T15:18:35.596732+0000","last_active":"2024-04-14T15:19:15.915072+0000","last_peered":"2024-04-14T15:19:15.915072+0000","last_clean":"2024-04-14T15:19:15.915072+0000","last_became_active":"2024-04-14T15:12:56.352528+0000","last_became_peered":"2024-04-14T15:12:56.352528+0000","last_unstale":"2024-04-14T15:19:15.915072+0000","last_undegraded":"2024-04-14T15:19:15.915072+0000","last_fullsized":"2024-04-14T15:19:15.915072+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:35.596656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:15:43.680852+0000","last_clean_scrub_stamp":"2024-04-14T15:18:35.596656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:36.273828+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"102'16","reported_seq":498,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.924560+0000","last_change":"2024-04-14T15:17:43.061551+0000","last_active":"2024-04-14T15:19:15.924560+0000","last_peered":"2024-04-14T15:19:15.924560+0000","last_clean":"2024-04-14T15:19:15.924560+0000","last_became_active":"2024-04-14T15:15:21.884783+0000","last_became_peered":"2024-04-14T15:15:21.884783+0000","last_unstale":"2024-04-14T15:19:15.924560+0000","last_undegraded":"2024-04-14T15:19:15.924560+0000","last_fullsized":"2024-04-14T15:19:15.924560+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'16","last_scrub_stamp":"2024-04-14T15:17:43.061458+0000","last_deep_scrub":"102'16","last_deep_scrub_stamp":"2024-04-14T15:12:35.999326+0000","last_clean_scrub_stamp":"2024-04-14T15:17:43.061458+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":851,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.966123+0000","last_change":"2024-04-14T15:19:11.750498+0000","last_active":"2024-04-14T15:19:15.966123+0000","last_peered":"2024-04-14T15:19:15.966123+0000","last_clean":"2024-04-14T15:19:15.966123+0000","last_became_active":"2024-04-14T15:09:43.023220+0000","last_became_peered":"2024-04-14T15:09:43.023220+0000","last_unstale":"2024-04-14T15:19:15.966123+0000","last_undegraded":"2024-04-14T15:19:15.966123+0000","last_fullsized":"2024-04-14T15:19:15.966123+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:11.750419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:19:11.750419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:20.900268+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":827,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.912997+0000","last_change":"2024-04-14T15:16:50.661634+0000","last_active":"2024-04-14T15:19:15.912997+0000","last_peered":"2024-04-14T15:19:15.912997+0000","last_clean":"2024-04-14T15:19:15.912997+0000","last_became_active":"2024-04-14T15:09:47.044665+0000","last_became_peered":"2024-04-14T15:09:47.044665+0000","last_unstale":"2024-04-14T15:19:15.912997+0000","last_undegraded":"2024-04-14T15:19:15.912997+0000","last_fullsized":"2024-04-14T15:19:15.912997+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:50.661520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:50.661520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":505,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.925080+0000","last_change":"2024-04-14T15:16:05.885527+0000","last_active":"2024-04-14T15:19:15.925080+0000","last_peered":"2024-04-14T15:19:15.925080+0000","last_clean":"2024-04-14T15:19:15.925080+0000","last_became_active":"2024-04-14T15:15:21.883940+0000","last_became_peered":"2024-04-14T15:15:21.883940+0000","last_unstale":"2024-04-14T15:19:15.925080+0000","last_undegraded":"2024-04-14T15:19:15.925080+0000","last_fullsized":"2024-04-14T15:19:15.925080+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:05.885438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:05.885438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1b","version":"91'130","reported_seq":888,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.936901+0000","last_change":"2024-04-14T15:19:06.558576+0000","last_active":"2024-04-14T15:19:15.936901+0000","last_peered":"2024-04-14T15:19:15.936901+0000","last_clean":"2024-04-14T15:19:15.936901+0000","last_became_active":"2024-04-14T15:19:02.936749+0000","last_became_peered":"2024-04-14T15:19:02.936749+0000","last_unstale":"2024-04-14T15:19:15.936901+0000","last_undegraded":"2024-04-14T15:19:15.936901+0000","last_fullsized":"2024-04-14T15:19:15.936901+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":446,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:06.558465+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:06.558465+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:28.851475+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":738,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.916244+0000","last_change":"2024-04-14T15:18:25.997409+0000","last_active":"2024-04-14T15:19:15.916244+0000","last_peered":"2024-04-14T15:19:15.916244+0000","last_clean":"2024-04-14T15:19:15.916244+0000","last_became_active":"2024-04-14T15:12:56.335230+0000","last_became_peered":"2024-04-14T15:12:56.335230+0000","last_unstale":"2024-04-14T15:19:15.916244+0000","last_undegraded":"2024-04-14T15:19:15.916244+0000","last_fullsized":"2024-04-14T15:19:15.916244+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:25.997333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:18:25.997333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:41.530252+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"72'10","reported_seq":848,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.931209+0000","last_change":"2024-04-14T15:16:49.779653+0000","last_active":"2024-04-14T15:19:15.931209+0000","last_peered":"2024-04-14T15:19:15.931209+0000","last_clean":"2024-04-14T15:19:15.931209+0000","last_became_active":"2024-04-14T15:09:45.181456+0000","last_became_peered":"2024-04-14T15:09:45.181456+0000","last_unstale":"2024-04-14T15:19:15.931209+0000","last_undegraded":"2024-04-14T15:19:15.931209+0000","last_fullsized":"2024-04-14T15:19:15.931209+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-04-14T15:16:49.779557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:16:49.779557+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":393,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.936442+0000","last_change":"2024-04-14T15:16:42.205142+0000","last_active":"2024-04-14T15:19:15.936442+0000","last_peered":"2024-04-14T15:19:15.936442+0000","last_clean":"2024-04-14T15:19:15.936442+0000","last_became_active":"2024-04-14T15:15:44.878336+0000","last_became_peered":"2024-04-14T15:15:44.878336+0000","last_unstale":"2024-04-14T15:19:15.936442+0000","last_undegraded":"2024-04-14T15:19:15.936442+0000","last_fullsized":"2024-04-14T15:19:15.936442+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:42.205063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:42.205063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"72'8","reported_seq":971,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.966322+0000","last_change":"2024-04-14T15:18:28.793061+0000","last_active":"2024-04-14T15:19:15.966322+0000","last_peered":"2024-04-14T15:19:15.966322+0000","last_clean":"2024-04-14T15:19:15.966322+0000","last_became_active":"2024-04-14T15:09:47.048216+0000","last_became_peered":"2024-04-14T15:09:47.048216+0000","last_unstale":"2024-04-14T15:19:15.966322+0000","last_undegraded":"2024-04-14T15:19:15.966322+0000","last_fullsized":"2024-04-14T15:19:15.966322+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:18:28.792975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:18:28.792975+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:50.662051+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"72'1","reported_seq":831,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.914605+0000","last_change":"2024-04-14T15:17:50.529174+0000","last_active":"2024-04-14T15:19:15.914605+0000","last_peered":"2024-04-14T15:19:15.914605+0000","last_clean":"2024-04-14T15:19:15.914605+0000","last_became_active":"2024-04-14T15:09:49.062051+0000","last_became_peered":"2024-04-14T15:09:49.062051+0000","last_unstale":"2024-04-14T15:19:15.914605+0000","last_undegraded":"2024-04-14T15:19:15.914605+0000","last_fullsized":"2024-04-14T15:19:15.914605+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-04-14T15:17:50.529089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:17:50.529089+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"91'130","reported_seq":740,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.936403+0000","last_change":"2024-04-14T15:19:03.537885+0000","last_active":"2024-04-14T15:19:15.936403+0000","last_peered":"2024-04-14T15:19:15.936403+0000","last_clean":"2024-04-14T15:19:15.936403+0000","last_became_active":"2024-04-14T15:19:02.930025+0000","last_became_peered":"2024-04-14T15:19:02.930025+0000","last_unstale":"2024-04-14T15:19:15.936403+0000","last_undegraded":"2024-04-14T15:19:15.936403+0000","last_fullsized":"2024-04-14T15:19:15.936403+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":446,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:03.537803+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:03.537803+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:03.649948+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":741,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.929001+0000","last_change":"2024-04-14T15:19:13.345390+0000","last_active":"2024-04-14T15:19:15.929001+0000","last_peered":"2024-04-14T15:19:15.929001+0000","last_clean":"2024-04-14T15:19:15.929001+0000","last_became_active":"2024-04-14T15:12:56.543793+0000","last_became_peered":"2024-04-14T15:12:56.543793+0000","last_unstale":"2024-04-14T15:19:15.929001+0000","last_undegraded":"2024-04-14T15:19:15.929001+0000","last_fullsized":"2024-04-14T15:19:15.929001+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:13.345316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:19:13.345316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:38.673629+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"72'11","reported_seq":877,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.966383+0000","last_change":"2024-04-14T15:18:31.818435+0000","last_active":"2024-04-14T15:19:15.966383+0000","last_peered":"2024-04-14T15:19:15.966383+0000","last_clean":"2024-04-14T15:19:15.966383+0000","last_became_active":"2024-04-14T15:10:07.764183+0000","last_became_peered":"2024-04-14T15:10:07.764183+0000","last_unstale":"2024-04-14T15:19:15.966383+0000","last_undegraded":"2024-04-14T15:19:15.966383+0000","last_fullsized":"2024-04-14T15:19:15.966383+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-04-14T15:18:31.818311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:18:31.818311+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:33.770617+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"65'2","reported_seq":524,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.924339+0000","last_change":"2024-04-14T15:16:56.943047+0000","last_active":"2024-04-14T15:19:15.924339+0000","last_peered":"2024-04-14T15:19:15.924339+0000","last_clean":"2024-04-14T15:19:15.924339+0000","last_became_active":"2024-04-14T15:15:21.885343+0000","last_became_peered":"2024-04-14T15:15:21.885343+0000","last_unstale":"2024-04-14T15:19:15.924339+0000","last_undegraded":"2024-04-14T15:19:15.924339+0000","last_fullsized":"2024-04-14T15:19:15.924339+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-04-14T15:16:56.942840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:56.942840+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":829,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.931365+0000","last_change":"2024-04-14T15:16:52.747151+0000","last_active":"2024-04-14T15:19:15.931365+0000","last_peered":"2024-04-14T15:19:15.931365+0000","last_clean":"2024-04-14T15:19:15.931365+0000","last_became_active":"2024-04-14T15:09:47.045730+0000","last_became_peered":"2024-04-14T15:09:47.045730+0000","last_unstale":"2024-04-14T15:19:15.931365+0000","last_undegraded":"2024-04-14T15:19:15.931365+0000","last_fullsized":"2024-04-14T15:19:15.931365+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:52.747070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:52.747070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":824,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.918320+0000","last_change":"2024-04-14T15:16:57.235339+0000","last_active":"2024-04-14T15:19:15.918320+0000","last_peered":"2024-04-14T15:19:15.918320+0000","last_clean":"2024-04-14T15:19:15.918320+0000","last_became_active":"2024-04-14T15:09:49.057545+0000","last_became_peered":"2024-04-14T15:09:49.057545+0000","last_unstale":"2024-04-14T15:19:15.918320+0000","last_undegraded":"2024-04-14T15:19:15.918320+0000","last_fullsized":"2024-04-14T15:19:15.918320+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:57.235261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:21.247695+0000","last_clean_scrub_stamp":"2024-04-14T15:16:57.235261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.19","version":"91'130","reported_seq":584,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.924025+0000","last_change":"2024-04-14T15:19:06.383564+0000","last_active":"2024-04-14T15:19:15.924025+0000","last_peered":"2024-04-14T15:19:15.924025+0000","last_clean":"2024-04-14T15:19:15.924025+0000","last_became_active":"2024-04-14T15:19:02.941363+0000","last_became_peered":"2024-04-14T15:19:02.941363+0000","last_unstale":"2024-04-14T15:19:15.924025+0000","last_undegraded":"2024-04-14T15:19:15.924025+0000","last_fullsized":"2024-04-14T15:19:15.924025+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":446,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:06.383478+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:06.383478+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:24.667992+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":393,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.937014+0000","last_change":"2024-04-14T15:17:44.364114+0000","last_active":"2024-04-14T15:19:15.937014+0000","last_peered":"2024-04-14T15:19:15.937014+0000","last_clean":"2024-04-14T15:19:15.937014+0000","last_became_active":"2024-04-14T15:15:44.882604+0000","last_became_peered":"2024-04-14T15:15:44.882604+0000","last_unstale":"2024-04-14T15:19:15.937014+0000","last_undegraded":"2024-04-14T15:19:15.937014+0000","last_fullsized":"2024-04-14T15:19:15.937014+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:44.364013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:17:44.364013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"72'15","reported_seq":862,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.919059+0000","last_change":"2024-04-14T15:18:32.950805+0000","last_active":"2024-04-14T15:19:15.919059+0000","last_peered":"2024-04-14T15:19:15.919059+0000","last_clean":"2024-04-14T15:19:15.919059+0000","last_became_active":"2024-04-14T15:09:45.032887+0000","last_became_peered":"2024-04-14T15:09:45.032887+0000","last_unstale":"2024-04-14T15:19:15.919059+0000","last_undegraded":"2024-04-14T15:19:15.919059+0000","last_fullsized":"2024-04-14T15:19:15.919059+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-04-14T15:18:32.950706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:18:32.950706+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:54.341517+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":846,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.965412+0000","last_change":"2024-04-14T15:18:27.799078+0000","last_active":"2024-04-14T15:19:15.965412+0000","last_peered":"2024-04-14T15:19:15.965412+0000","last_clean":"2024-04-14T15:19:15.965412+0000","last_became_active":"2024-04-14T15:09:43.021494+0000","last_became_peered":"2024-04-14T15:09:43.021494+0000","last_unstale":"2024-04-14T15:19:15.965412+0000","last_undegraded":"2024-04-14T15:19:15.965412+0000","last_fullsized":"2024-04-14T15:19:15.965412+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:27.798964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:18:27.798964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:33.052017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":836,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.918434+0000","last_change":"2024-04-14T15:18:22.953738+0000","last_active":"2024-04-14T15:19:15.918434+0000","last_peered":"2024-04-14T15:19:15.918434+0000","last_clean":"2024-04-14T15:19:15.918434+0000","last_became_active":"2024-04-14T15:09:47.042733+0000","last_became_peered":"2024-04-14T15:09:47.042733+0000","last_unstale":"2024-04-14T15:19:15.918434+0000","last_undegraded":"2024-04-14T15:19:15.918434+0000","last_fullsized":"2024-04-14T15:19:15.918434+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:22.953661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:18:22.953661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:49.908442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":831,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.937696+0000","last_change":"2024-04-14T15:17:54.453289+0000","last_active":"2024-04-14T15:19:15.937696+0000","last_peered":"2024-04-14T15:19:15.937696+0000","last_clean":"2024-04-14T15:19:15.937696+0000","last_became_active":"2024-04-14T15:09:49.127979+0000","last_became_peered":"2024-04-14T15:09:49.127979+0000","last_unstale":"2024-04-14T15:19:15.937696+0000","last_undegraded":"2024-04-14T15:19:15.937696+0000","last_fullsized":"2024-04-14T15:19:15.937696+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:54.453174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:54.453174+0000","last_clean_scrub_stamp":"2024-04-14T15:17:54.453174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.18","version":"91'130","reported_seq":742,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.915699+0000","last_change":"2024-04-14T15:19:02.957443+0000","last_active":"2024-04-14T15:19:15.915699+0000","last_peered":"2024-04-14T15:19:15.915699+0000","last_clean":"2024-04-14T15:19:15.915699+0000","last_became_active":"2024-04-14T15:19:02.940290+0000","last_became_peered":"2024-04-14T15:19:02.940290+0000","last_unstale":"2024-04-14T15:19:15.915699+0000","last_undegraded":"2024-04-14T15:19:15.915699+0000","last_fullsized":"2024-04-14T15:19:15.915699+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":446,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:02.957360+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:02.957360+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:07.851285+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":739,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.913551+0000","last_change":"2024-04-14T15:18:26.532035+0000","last_active":"2024-04-14T15:19:15.913551+0000","last_peered":"2024-04-14T15:19:15.913551+0000","last_clean":"2024-04-14T15:19:15.913551+0000","last_became_active":"2024-04-14T15:12:56.392352+0000","last_became_peered":"2024-04-14T15:12:56.392352+0000","last_unstale":"2024-04-14T15:19:15.913551+0000","last_undegraded":"2024-04-14T15:19:15.913551+0000","last_fullsized":"2024-04-14T15:19:15.913551+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:26.531955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:18:26.531955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:43.681656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"72'12","reported_seq":544,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:17.364144+0000","last_change":"2024-04-14T15:19:17.364178+0000","last_active":"2024-04-14T15:19:17.364144+0000","last_peered":"2024-04-14T15:19:17.364144+0000","last_clean":"2024-04-14T15:19:17.364144+0000","last_became_active":"2024-04-14T15:15:21.877838+0000","last_became_peered":"2024-04-14T15:15:21.877838+0000","last_unstale":"2024-04-14T15:19:17.364144+0000","last_undegraded":"2024-04-14T15:19:17.364144+0000","last_fullsized":"2024-04-14T15:19:17.364144+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'12","last_scrub_stamp":"2024-04-14T15:19:17.364059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:19:17.364059+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:22.503752+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":517,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.924964+0000","last_change":"2024-04-14T15:16:20.969027+0000","last_active":"2024-04-14T15:19:15.924964+0000","last_peered":"2024-04-14T15:19:15.924964+0000","last_clean":"2024-04-14T15:19:15.924964+0000","last_became_active":"2024-04-14T15:15:21.883544+0000","last_became_peered":"2024-04-14T15:15:21.883544+0000","last_unstale":"2024-04-14T15:19:15.924964+0000","last_undegraded":"2024-04-14T15:19:15.924964+0000","last_fullsized":"2024-04-14T15:19:15.924964+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:20.968897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:20.968897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":834,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.936875+0000","last_change":"2024-04-14T15:17:01.098681+0000","last_active":"2024-04-14T15:19:15.936875+0000","last_peered":"2024-04-14T15:19:15.936875+0000","last_clean":"2024-04-14T15:19:15.936875+0000","last_became_active":"2024-04-14T15:09:47.048045+0000","last_became_peered":"2024-04-14T15:09:47.048045+0000","last_unstale":"2024-04-14T15:19:15.936875+0000","last_undegraded":"2024-04-14T15:19:15.936875+0000","last_fullsized":"2024-04-14T15:19:15.936875+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:01.098599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:17:01.098599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":506,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.925791+0000","last_change":"2024-04-14T15:16:07.879394+0000","last_active":"2024-04-14T15:19:15.925791+0000","last_peered":"2024-04-14T15:19:15.925791+0000","last_clean":"2024-04-14T15:19:15.925791+0000","last_became_active":"2024-04-14T15:15:21.889691+0000","last_became_peered":"2024-04-14T15:15:21.889691+0000","last_unstale":"2024-04-14T15:19:15.925791+0000","last_undegraded":"2024-04-14T15:19:15.925791+0000","last_fullsized":"2024-04-14T15:19:15.925791+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:07.879280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:07.879280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":738,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.919031+0000","last_change":"2024-04-14T15:17:37.766225+0000","last_active":"2024-04-14T15:19:15.919031+0000","last_peered":"2024-04-14T15:19:15.919031+0000","last_clean":"2024-04-14T15:19:15.919031+0000","last_became_active":"2024-04-14T15:12:56.375278+0000","last_became_peered":"2024-04-14T15:12:56.375278+0000","last_unstale":"2024-04-14T15:19:15.919031+0000","last_undegraded":"2024-04-14T15:19:15.919031+0000","last_fullsized":"2024-04-14T15:19:15.919031+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:37.766116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:17:37.766116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"276'13","reported_seq":858,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.937236+0000","last_change":"2024-04-14T15:17:52.445394+0000","last_active":"2024-04-14T15:19:15.937236+0000","last_peered":"2024-04-14T15:19:15.937236+0000","last_clean":"2024-04-14T15:19:15.937236+0000","last_became_active":"2024-04-14T15:09:45.030093+0000","last_became_peered":"2024-04-14T15:09:45.030093+0000","last_unstale":"2024-04-14T15:19:15.937236+0000","last_undegraded":"2024-04-14T15:19:15.937236+0000","last_fullsized":"2024-04-14T15:19:15.937236+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'13","last_scrub_stamp":"2024-04-14T15:17:52.445294+0000","last_deep_scrub":"276'13","last_deep_scrub_stamp":"2024-04-14T15:15:43.969775+0000","last_clean_scrub_stamp":"2024-04-14T15:17:52.445294+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":540,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.914133+0000","last_change":"2024-04-14T15:17:02.800274+0000","last_active":"2024-04-14T15:19:15.914133+0000","last_peered":"2024-04-14T15:19:15.914133+0000","last_clean":"2024-04-14T15:19:15.914133+0000","last_became_active":"2024-04-14T15:14:30.857727+0000","last_became_peered":"2024-04-14T15:14:30.857727+0000","last_unstale":"2024-04-14T15:19:15.914133+0000","last_undegraded":"2024-04-14T15:19:15.914133+0000","last_fullsized":"2024-04-14T15:19:15.914133+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:02.800166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:02.800166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":395,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.930670+0000","last_change":"2024-04-14T15:16:59.695423+0000","last_active":"2024-04-14T15:19:15.930670+0000","last_peered":"2024-04-14T15:19:15.930670+0000","last_clean":"2024-04-14T15:19:15.930670+0000","last_became_active":"2024-04-14T15:15:44.883121+0000","last_became_peered":"2024-04-14T15:15:44.883121+0000","last_unstale":"2024-04-14T15:19:15.930670+0000","last_undegraded":"2024-04-14T15:19:15.930670+0000","last_fullsized":"2024-04-14T15:19:15.930670+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:59.695337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:59.695337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":832,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.929253+0000","last_change":"2024-04-14T15:17:33.952503+0000","last_active":"2024-04-14T15:19:15.929253+0000","last_peered":"2024-04-14T15:19:15.929253+0000","last_clean":"2024-04-14T15:19:15.929253+0000","last_became_active":"2024-04-14T15:09:49.061756+0000","last_became_peered":"2024-04-14T15:09:49.061756+0000","last_unstale":"2024-04-14T15:19:15.929253+0000","last_undegraded":"2024-04-14T15:19:15.929253+0000","last_fullsized":"2024-04-14T15:19:15.929253+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:33.952312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:17:33.952312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"91'130","reported_seq":372,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.925416+0000","last_change":"2024-04-14T15:19:03.395696+0000","last_active":"2024-04-14T15:19:15.925416+0000","last_peered":"2024-04-14T15:19:15.925416+0000","last_clean":"2024-04-14T15:19:15.925416+0000","last_became_active":"2024-04-14T15:19:02.936733+0000","last_became_peered":"2024-04-14T15:19:02.936733+0000","last_unstale":"2024-04-14T15:19:15.925416+0000","last_undegraded":"2024-04-14T15:19:15.925416+0000","last_fullsized":"2024-04-14T15:19:15.925416+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":446,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:03.395612+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:03.395612+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:11.326775+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":748,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.965483+0000","last_change":"2024-04-14T15:19:12.723151+0000","last_active":"2024-04-14T15:19:15.965483+0000","last_peered":"2024-04-14T15:19:15.965483+0000","last_clean":"2024-04-14T15:19:15.965483+0000","last_became_active":"2024-04-14T15:12:56.232831+0000","last_became_peered":"2024-04-14T15:12:56.232831+0000","last_unstale":"2024-04-14T15:19:15.965483+0000","last_undegraded":"2024-04-14T15:19:15.965483+0000","last_fullsized":"2024-04-14T15:19:15.965483+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:12.723073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:19:12.723073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:34.125961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1b","version":"87'7","reported_seq":850,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.939049+0000","last_change":"2024-04-14T15:18:35.481608+0000","last_active":"2024-04-14T15:19:15.939049+0000","last_peered":"2024-04-14T15:19:15.939049+0000","last_clean":"2024-04-14T15:19:15.939049+0000","last_became_active":"2024-04-14T15:09:45.028667+0000","last_became_peered":"2024-04-14T15:09:45.028667+0000","last_unstale":"2024-04-14T15:19:15.939049+0000","last_undegraded":"2024-04-14T15:19:15.939049+0000","last_fullsized":"2024-04-14T15:19:15.939049+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'7","last_scrub_stamp":"2024-04-14T15:18:35.481504+0000","last_deep_scrub":"87'7","last_deep_scrub_stamp":"2024-04-14T15:11:10.510785+0000","last_clean_scrub_stamp":"2024-04-14T15:18:35.481504+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:51.921756+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":311,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.918741+0000","last_change":"2024-04-14T15:16:46.686879+0000","last_active":"2024-04-14T15:19:15.918741+0000","last_peered":"2024-04-14T15:19:15.918741+0000","last_clean":"2024-04-14T15:19:15.918741+0000","last_became_active":"2024-04-14T15:16:24.910246+0000","last_became_peered":"2024-04-14T15:16:24.910246+0000","last_unstale":"2024-04-14T15:19:15.918741+0000","last_undegraded":"2024-04-14T15:19:15.918741+0000","last_fullsized":"2024-04-14T15:19:15.918741+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:46.686789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:08.254601+0000","last_clean_scrub_stamp":"2024-04-14T15:16:46.686789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":834,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.913758+0000","last_change":"2024-04-14T15:19:15.567537+0000","last_active":"2024-04-14T15:19:15.913758+0000","last_peered":"2024-04-14T15:19:15.913758+0000","last_clean":"2024-04-14T15:19:15.913758+0000","last_became_active":"2024-04-14T15:09:47.046517+0000","last_became_peered":"2024-04-14T15:09:47.046517+0000","last_unstale":"2024-04-14T15:19:15.913758+0000","last_undegraded":"2024-04-14T15:19:15.913758+0000","last_fullsized":"2024-04-14T15:19:15.913758+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:15.567432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:19:15.567432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:24.223440+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":396,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.919554+0000","last_change":"2024-04-14T15:18:21.023508+0000","last_active":"2024-04-14T15:19:15.919554+0000","last_peered":"2024-04-14T15:19:15.919554+0000","last_clean":"2024-04-14T15:19:15.919554+0000","last_became_active":"2024-04-14T15:15:44.887227+0000","last_became_peered":"2024-04-14T15:15:44.887227+0000","last_unstale":"2024-04-14T15:19:15.919554+0000","last_undegraded":"2024-04-14T15:19:15.919554+0000","last_fullsized":"2024-04-14T15:19:15.919554+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:21.023426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:56.268086+0000","last_clean_scrub_stamp":"2024-04-14T15:18:21.023426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:27.500436+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1d","version":"91'130","reported_seq":372,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.917720+0000","last_change":"2024-04-14T15:19:03.081107+0000","last_active":"2024-04-14T15:19:15.917720+0000","last_peered":"2024-04-14T15:19:15.917720+0000","last_clean":"2024-04-14T15:19:15.917720+0000","last_became_active":"2024-04-14T15:19:02.934382+0000","last_became_peered":"2024-04-14T15:19:02.934382+0000","last_unstale":"2024-04-14T15:19:15.917720+0000","last_undegraded":"2024-04-14T15:19:15.917720+0000","last_fullsized":"2024-04-14T15:19:15.917720+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":446,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:03.081016+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:03.081016+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:09.658638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":739,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.915181+0000","last_change":"2024-04-14T15:18:34.558447+0000","last_active":"2024-04-14T15:19:15.915181+0000","last_peered":"2024-04-14T15:19:15.915181+0000","last_clean":"2024-04-14T15:19:15.915181+0000","last_became_active":"2024-04-14T15:12:56.189117+0000","last_became_peered":"2024-04-14T15:12:56.189117+0000","last_unstale":"2024-04-14T15:19:15.915181+0000","last_undegraded":"2024-04-14T15:19:15.915181+0000","last_fullsized":"2024-04-14T15:19:15.915181+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:34.558312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:34.558312+0000","last_clean_scrub_stamp":"2024-04-14T15:18:34.558312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:58.210395+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.18","version":"72'9","reported_seq":854,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.936227+0000","last_change":"2024-04-14T15:18:36.441934+0000","last_active":"2024-04-14T15:19:15.936227+0000","last_peered":"2024-04-14T15:19:15.936227+0000","last_clean":"2024-04-14T15:19:15.936227+0000","last_became_active":"2024-04-14T15:09:45.032079+0000","last_became_peered":"2024-04-14T15:09:45.032079+0000","last_unstale":"2024-04-14T15:19:15.936227+0000","last_undegraded":"2024-04-14T15:19:15.936227+0000","last_fullsized":"2024-04-14T15:19:15.936227+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-04-14T15:18:36.441813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:18:36.441813+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:03.217663+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":518,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.925860+0000","last_change":"2024-04-14T15:17:00.948362+0000","last_active":"2024-04-14T15:19:15.925860+0000","last_peered":"2024-04-14T15:19:15.925860+0000","last_clean":"2024-04-14T15:19:15.925860+0000","last_became_active":"2024-04-14T15:15:21.891926+0000","last_became_peered":"2024-04-14T15:15:21.891926+0000","last_unstale":"2024-04-14T15:19:15.925860+0000","last_undegraded":"2024-04-14T15:19:15.925860+0000","last_fullsized":"2024-04-14T15:19:15.925860+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:00.948283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:00.948283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":828,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.919460+0000","last_change":"2024-04-14T15:16:45.114648+0000","last_active":"2024-04-14T15:19:15.919460+0000","last_peered":"2024-04-14T15:19:15.919460+0000","last_clean":"2024-04-14T15:19:15.919460+0000","last_became_active":"2024-04-14T15:09:47.041486+0000","last_became_peered":"2024-04-14T15:09:47.041486+0000","last_unstale":"2024-04-14T15:19:15.919460+0000","last_undegraded":"2024-04-14T15:19:15.919460+0000","last_fullsized":"2024-04-14T15:19:15.919460+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:45.114534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:45.114534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":822,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.937120+0000","last_change":"2024-04-14T15:16:55.184211+0000","last_active":"2024-04-14T15:19:15.937120+0000","last_peered":"2024-04-14T15:19:15.937120+0000","last_clean":"2024-04-14T15:19:15.937120+0000","last_became_active":"2024-04-14T15:09:49.058904+0000","last_became_peered":"2024-04-14T15:09:49.058904+0000","last_unstale":"2024-04-14T15:19:15.937120+0000","last_undegraded":"2024-04-14T15:19:15.937120+0000","last_fullsized":"2024-04-14T15:19:15.937120+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:55.184094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:55.184094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1c","version":"91'130","reported_seq":885,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.913993+0000","last_change":"2024-04-14T15:19:03.486816+0000","last_active":"2024-04-14T15:19:15.913993+0000","last_peered":"2024-04-14T15:19:15.913993+0000","last_clean":"2024-04-14T15:19:15.913993+0000","last_became_active":"2024-04-14T15:19:02.934458+0000","last_became_peered":"2024-04-14T15:19:02.934458+0000","last_unstale":"2024-04-14T15:19:15.913993+0000","last_undegraded":"2024-04-14T15:19:15.913993+0000","last_fullsized":"2024-04-14T15:19:15.913993+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":446,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:03.486726+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:03.486726+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:08.034646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":393,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.914943+0000","last_change":"2024-04-14T15:17:52.593706+0000","last_active":"2024-04-14T15:19:15.914943+0000","last_peered":"2024-04-14T15:19:15.914943+0000","last_clean":"2024-04-14T15:19:15.914943+0000","last_became_active":"2024-04-14T15:15:44.889605+0000","last_became_peered":"2024-04-14T15:15:44.889605+0000","last_unstale":"2024-04-14T15:19:15.914943+0000","last_undegraded":"2024-04-14T15:19:15.914943+0000","last_fullsized":"2024-04-14T15:19:15.914943+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:52.593618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:17:52.593618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"306'22","reported_seq":545,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.924558+0000","last_change":"2024-04-14T15:16:13.898263+0000","last_active":"2024-04-14T15:19:15.924558+0000","last_peered":"2024-04-14T15:19:15.924558+0000","last_clean":"2024-04-14T15:19:15.924558+0000","last_became_active":"2024-04-14T15:15:21.884310+0000","last_became_peered":"2024-04-14T15:15:21.884310+0000","last_unstale":"2024-04-14T15:19:15.924558+0000","last_undegraded":"2024-04-14T15:19:15.924558+0000","last_fullsized":"2024-04-14T15:19:15.924558+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'22","last_scrub_stamp":"2024-04-14T15:16:13.898106+0000","last_deep_scrub":"306'22","last_deep_scrub_stamp":"2024-04-14T15:16:13.898106+0000","last_clean_scrub_stamp":"2024-04-14T15:16:13.898106+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":845,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:18.347611+0000","last_change":"2024-04-14T15:19:18.347632+0000","last_active":"2024-04-14T15:19:18.347611+0000","last_peered":"2024-04-14T15:19:18.347611+0000","last_clean":"2024-04-14T15:19:18.347611+0000","last_became_active":"2024-04-14T15:09:43.017351+0000","last_became_peered":"2024-04-14T15:09:43.017351+0000","last_unstale":"2024-04-14T15:19:18.347611+0000","last_undegraded":"2024-04-14T15:19:18.347611+0000","last_fullsized":"2024-04-14T15:19:18.347611+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:18.347533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:19:18.347533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:22.403586+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":826,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.936651+0000","last_change":"2024-04-14T15:16:19.041461+0000","last_active":"2024-04-14T15:19:15.936651+0000","last_peered":"2024-04-14T15:19:15.936651+0000","last_clean":"2024-04-14T15:19:15.936651+0000","last_became_active":"2024-04-14T15:09:47.042538+0000","last_became_peered":"2024-04-14T15:09:47.042538+0000","last_unstale":"2024-04-14T15:19:15.936651+0000","last_undegraded":"2024-04-14T15:19:15.936651+0000","last_fullsized":"2024-04-14T15:19:15.936651+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:19.041346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:08.512130+0000","last_clean_scrub_stamp":"2024-04-14T15:16:19.041346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":828,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-14T15:19:15.924784+0000","last_change":"2024-04-14T15:17:56.237366+0000","last_active":"2024-04-14T15:19:15.924784+0000","last_peered":"2024-04-14T15:19:15.924784+0000","last_clean":"2024-04-14T15:19:15.924784+0000","last_became_active":"2024-04-14T15:09:49.129636+0000","last_became_peered":"2024-04-14T15:09:49.129636+0000","last_unstale":"2024-04-14T15:19:15.924784+0000","last_undegraded":"2024-04-14T15:19:15.924784+0000","last_fullsized":"2024-04-14T15:19:15.924784+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:56.237273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:17:56.237273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":301,"num_pg":32,"stat_sum":{"num_bytes":14,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":14,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":42,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14,"ondisk_log_size":14,"up":96,"acting":96,"num_store_stats":8},{"poolid":110,"num_pg":32,"stat_sum":{"num_bytes":8195833,"num_objects":166,"num_object_clones":5,"num_object_copies":498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":0,"num_read":110,"num_read_kb":10,"num_write":614,"num_write_kb":12703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":13,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":26566656,"data_stored":24587559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1927,"ondisk_log_size":1927,"up":96,"acting":96,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4030,"ondisk_log_size":4030,"up":93,"acting":93,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":569,"num_read_kb":564,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":695,"num_read_kb":452,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":493,"ondisk_log_size":493,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71,"ondisk_log_size":71,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103289,"num_pgs":111,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":73544,"kb_used_data":15976,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93671608,"statfs":{"total":95995035648,"available":95919726592,"internally_reserved":0,"allocated":16359424,"data_stored":12910344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":58915274},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397628,"num_pgs":89,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":71160,"kb_used_data":8024,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93673992,"statfs":{"total":95995035648,"available":95922167808,"internally_reserved":0,"allocated":8216576,"data_stored":4781283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574998,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":91684,"kb_used_data":11920,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":93653468,"statfs":{"total":95995035648,"available":95901151232,"internally_reserved":0,"allocated":12206080,"data_stored":8648228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81656261},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738523,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":83436,"kb_used_data":12004,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93661716,"statfs":{"total":95995035648,"available":95909597184,"internally_reserved":0,"allocated":12292096,"data_stored":8781753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411474,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":83420,"kb_used_data":11964,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93661732,"statfs":{"total":95995035648,"available":95909613568,"internally_reserved":0,"allocated":12251136,"data_stored":8715764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182540,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":57232,"kb_used_data":8008,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93687920,"statfs":{"total":95995035648,"available":95936430080,"internally_reserved":0,"allocated":8200192,"data_stored":4781611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986310,"num_pgs":93,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":70780,"kb_used_data":7668,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93674372,"statfs":{"total":95995035648,"available":95922556928,"internally_reserved":0,"allocated":7852032,"data_stored":4453111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659267,"num_pgs":91,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":75152,"kb_used_data":12040,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93670000,"statfs":{"total":95995035648,"available":95918080000,"internally_reserved":0,"allocated":12328960,"data_stored":8909618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4284416,"data_stored":4000640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4505600,"data_stored":4195039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4333568,"data_stored":4000786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":176128,"data_stored":418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":184320,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4411392,"data_stored":4194809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":229376,"data_stored":534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8441856,"data_stored":8194889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":301,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":8,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":301,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":8,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":301,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":8,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":301,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":6,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":301,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":4,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":301,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":4,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":301,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":4,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":301,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-14T15:19:22.775 INFO:tasks.thrashosds.thrasher:Setting 110.3 to [5, 7, 2, 1, 4, 3] 2024-04-14T15:19:22.775 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '110.3', '5', '7', '2', '1', '4', '3'] 2024-04-14T15:19:22.775 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd pg-upmap-items 110.3 5 7 2 1 4 3 2024-04-14T15:19:22.791 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:22.814 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:19:22.817 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:19:22.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:22.818 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:22.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:22.904 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-14T15:19:22.919 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:19:22.942 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:22.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:22.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:19:23.044 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:19:23.068 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:23.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:23.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:19:23.170 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:19:23.195 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:23.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:23.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:19:23.297 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:19:23.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:23.321 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:23.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:19:23.350 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:23 smithi090 ceph-mon[41670]: 2.10 scrub starts 2024-04-14T15:19:23.351 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:23 smithi090 ceph-mon[41670]: 2.10 scrub ok 2024-04-14T15:19:23.351 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:23 smithi090 ceph-mon[41670]: 110.9 scrub starts 2024-04-14T15:19:23.351 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:23 smithi090 ceph-mon[41670]: 110.9 scrub ok 2024-04-14T15:19:23.351 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:23 smithi090 ceph-mon[41670]: osdmap e465: 8 total, 8 up, 8 in 2024-04-14T15:19:23.351 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:23 smithi090 ceph-mon[30399]: 2.10 scrub starts 2024-04-14T15:19:23.351 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:23 smithi090 ceph-mon[30399]: 2.10 scrub ok 2024-04-14T15:19:23.351 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:23 smithi090 ceph-mon[30399]: 110.9 scrub starts 2024-04-14T15:19:23.351 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:23 smithi090 ceph-mon[30399]: 110.9 scrub ok 2024-04-14T15:19:23.351 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:23 smithi090 ceph-mon[30399]: osdmap e465: 8 total, 8 up, 8 in 2024-04-14T15:19:23.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:23 smithi092 ceph-mon[33966]: 2.10 scrub starts 2024-04-14T15:19:23.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:23 smithi092 ceph-mon[33966]: 2.10 scrub ok 2024-04-14T15:19:23.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:23 smithi092 ceph-mon[33966]: 110.9 scrub starts 2024-04-14T15:19:23.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:23 smithi092 ceph-mon[33966]: 110.9 scrub ok 2024-04-14T15:19:23.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:23 smithi092 ceph-mon[33966]: osdmap e465: 8 total, 8 up, 8 in 2024-04-14T15:19:23.422 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:19:23.446 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:23.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:23.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:19:23.548 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:23.572 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:23.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:23.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:23.674 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:19:23.697 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:23.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:23.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:19:23.799 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:19:23.827 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:23.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:23.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:19:23.929 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:19:23.953 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:23.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:23.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:19:24.055 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:19:24.079 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:24.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:24.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:19:24.180 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:19:24.191 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:19:23 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:19:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:19:24.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:24 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:19:24.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:24 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:19:24.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:24 smithi090 ceph-mon[30399]: 6.5 scrub starts 2024-04-14T15:19:24.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:24 smithi090 ceph-mon[30399]: 6.5 scrub ok 2024-04-14T15:19:24.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:24 smithi090 ceph-mon[30399]: 2.f scrub starts 2024-04-14T15:19:24.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:24 smithi090 ceph-mon[30399]: 2.f scrub ok 2024-04-14T15:19:24.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:24 smithi090 ceph-mon[30399]: 110.b scrub starts 2024-04-14T15:19:24.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:24 smithi090 ceph-mon[30399]: 110.b scrub ok 2024-04-14T15:19:24.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:24 smithi090 ceph-mon[30399]: osdmap e466: 8 total, 8 up, 8 in 2024-04-14T15:19:24.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:24 smithi090 ceph-mon[30399]: pgmap v660: 258 pgs: 258 active+clean; 8.4 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 855 B/s rd, 1.8 KiB/s wr, 4 op/s 2024-04-14T15:19:24.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:24 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:19:24.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:24 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:19:24.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:24 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:19:24.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:24 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:19:24.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:24 smithi090 ceph-mon[41670]: 6.5 scrub starts 2024-04-14T15:19:24.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:24 smithi090 ceph-mon[41670]: 6.5 scrub ok 2024-04-14T15:19:24.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:24 smithi090 ceph-mon[41670]: 2.f scrub starts 2024-04-14T15:19:24.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:24 smithi090 ceph-mon[41670]: 2.f scrub ok 2024-04-14T15:19:24.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:24 smithi090 ceph-mon[41670]: 110.b scrub starts 2024-04-14T15:19:24.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:24 smithi090 ceph-mon[41670]: 110.b scrub ok 2024-04-14T15:19:24.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:24 smithi090 ceph-mon[41670]: osdmap e466: 8 total, 8 up, 8 in 2024-04-14T15:19:24.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:24 smithi090 ceph-mon[41670]: pgmap v660: 258 pgs: 258 active+clean; 8.4 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 855 B/s rd, 1.8 KiB/s wr, 4 op/s 2024-04-14T15:19:24.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:24 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:19:24.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:24 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:19:24.204 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:24.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:24.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:19:24.306 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:19:24.331 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:24.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:24.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:19:24.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:24 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:19:24.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:24 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:19:24.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:24 smithi092 ceph-mon[33966]: 6.5 scrub starts 2024-04-14T15:19:24.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:24 smithi092 ceph-mon[33966]: 6.5 scrub ok 2024-04-14T15:19:24.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:24 smithi092 ceph-mon[33966]: 2.f scrub starts 2024-04-14T15:19:24.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:24 smithi092 ceph-mon[33966]: 2.f scrub ok 2024-04-14T15:19:24.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:24 smithi092 ceph-mon[33966]: 110.b scrub starts 2024-04-14T15:19:24.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:24 smithi092 ceph-mon[33966]: 110.b scrub ok 2024-04-14T15:19:24.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:24 smithi092 ceph-mon[33966]: osdmap e466: 8 total, 8 up, 8 in 2024-04-14T15:19:24.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:24 smithi092 ceph-mon[33966]: pgmap v660: 258 pgs: 258 active+clean; 8.4 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 855 B/s rd, 1.8 KiB/s wr, 4 op/s 2024-04-14T15:19:24.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:24 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:19:24.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:24 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:19:24.433 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:19:24.457 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:24.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:24.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:19:24.559 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:19:24.583 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:24.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:24.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:19:24.685 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:24.709 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:24.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:24.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:24.811 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:24.835 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:24.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:24.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:24.937 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:19:24.961 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:24.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:24.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:19:25.063 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:19:25.087 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:25.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:25.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:19:25.189 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:19:25.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:25 smithi090 ceph-mon[41670]: 6.13 scrub starts 2024-04-14T15:19:25.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:25 smithi090 ceph-mon[41670]: 6.13 scrub ok 2024-04-14T15:19:25.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:25 smithi090 ceph-mon[41670]: 2.11 scrub starts 2024-04-14T15:19:25.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:25 smithi090 ceph-mon[41670]: 2.11 scrub ok 2024-04-14T15:19:25.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:25 smithi090 ceph-mon[41670]: 2.2 deep-scrub starts 2024-04-14T15:19:25.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:25 smithi090 ceph-mon[41670]: 2.2 deep-scrub ok 2024-04-14T15:19:25.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:25 smithi090 ceph-mon[41670]: 11.19 scrub starts 2024-04-14T15:19:25.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:25 smithi090 ceph-mon[41670]: 11.19 scrub ok 2024-04-14T15:19:25.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:25 smithi090 ceph-mon[30399]: 6.13 scrub starts 2024-04-14T15:19:25.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:25 smithi090 ceph-mon[30399]: 6.13 scrub ok 2024-04-14T15:19:25.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:25 smithi090 ceph-mon[30399]: 2.11 scrub starts 2024-04-14T15:19:25.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:25 smithi090 ceph-mon[30399]: 2.11 scrub ok 2024-04-14T15:19:25.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:25 smithi090 ceph-mon[30399]: 2.2 deep-scrub starts 2024-04-14T15:19:25.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:25 smithi090 ceph-mon[30399]: 2.2 deep-scrub ok 2024-04-14T15:19:25.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:25 smithi090 ceph-mon[30399]: 11.19 scrub starts 2024-04-14T15:19:25.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:25 smithi090 ceph-mon[30399]: 11.19 scrub ok 2024-04-14T15:19:25.213 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:25.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:25.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:19:25.315 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:19:25.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:25.339 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:25.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:19:25.372 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:19:25 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:19:25.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:25 smithi092 ceph-mon[33966]: 6.13 scrub starts 2024-04-14T15:19:25.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:25 smithi092 ceph-mon[33966]: 6.13 scrub ok 2024-04-14T15:19:25.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:25 smithi092 ceph-mon[33966]: 2.11 scrub starts 2024-04-14T15:19:25.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:25 smithi092 ceph-mon[33966]: 2.11 scrub ok 2024-04-14T15:19:25.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:25 smithi092 ceph-mon[33966]: 2.2 deep-scrub starts 2024-04-14T15:19:25.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:25 smithi092 ceph-mon[33966]: 2.2 deep-scrub ok 2024-04-14T15:19:25.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:25 smithi092 ceph-mon[33966]: 11.19 scrub starts 2024-04-14T15:19:25.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:25 smithi092 ceph-mon[33966]: 11.19 scrub ok 2024-04-14T15:19:25.440 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:19:25.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:25.465 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:25.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:19:25.566 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:19:25.590 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:25.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:25.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:19:25.691 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:19:25.715 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:25.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:25.716 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:19:25.817 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:19:25.841 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:25.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:25.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:19:25.942 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:19:25.968 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:25.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:25.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:19:26.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:26.064+0000 7f30ed8e1700 1 -- 172.21.15.90:0/1631870733 >> v1:172.21.15.90:6789/0 conn(0x7f30e810f2f0 legacy=0x7f30e81117b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:26.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:26.064+0000 7f30ed8e1700 1 -- 172.21.15.90:0/1631870733 shutdown_connections 2024-04-14T15:19:26.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:26.064+0000 7f30ed8e1700 1 -- 172.21.15.90:0/1631870733 >> 172.21.15.90:0/1631870733 conn(0x7f30e80fe3c0 msgr2=0x7f30e81007e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:19:26.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:26.064+0000 7f30ed8e1700 1 -- 172.21.15.90:0/1631870733 shutdown_connections 2024-04-14T15:19:26.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:26.064+0000 7f30ed8e1700 1 -- 172.21.15.90:0/1631870733 wait complete. 2024-04-14T15:19:26.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:26.065+0000 7f30ed8e1700 1 librados: starting msgr at 2024-04-14T15:19:26.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:26.065+0000 7f30ed8e1700 1 librados: starting objecter 2024-04-14T15:19:26.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:26.065+0000 7f30ed8e1700 1 Processor -- start 2024-04-14T15:19:26.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:26.065+0000 7f30ed8e1700 1 -- start start 2024-04-14T15:19:26.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:26.065+0000 7f30ed8e1700 1 librados: setting wanted keys 2024-04-14T15:19:26.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:26.065+0000 7f30ed8e1700 1 librados: calling monclient init 2024-04-14T15:19:26.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:26.065+0000 7f30ed8e1700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f30e810efa0 con 0x7f30e810b720 2024-04-14T15:19:26.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:26.065+0000 7f30ed8e1700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f30e81a8f00 con 0x7f30e810f2f0 2024-04-14T15:19:26.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:26.065+0000 7f30ed8e1700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f30e81aa0e0 con 0x7f30e8108870 2024-04-14T15:19:26.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:26.066+0000 7f30e67fc700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f30e810b720 0x7f30e810df70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:37126/0 (socket says 172.21.15.90:37126) 2024-04-14T15:19:26.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:26.066+0000 7f30e6ffd700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f30e8108870 0x7f30e810d860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:41494/0 (socket says 172.21.15.90:41494) 2024-04-14T15:19:26.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:26.066+0000 7f30e67fc700 1 -- 172.21.15.90:0/3602299662 learned_addr learned my addr 172.21.15.90:0/3602299662 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:19:26.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:26.066+0000 7f30ec8df700 10 client.?.objecter ms_handle_connect 0x7f30e8108870 2024-04-14T15:19:26.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:26.066+0000 7f30ec8df700 10 client.?.objecter resend_mon_ops 2024-04-14T15:19:26.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:26.066+0000 7f30ec8df700 10 client.?.objecter ms_handle_connect 0x7f30e810b720 2024-04-14T15:19:26.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:26.066+0000 7f30ec8df700 10 client.?.objecter resend_mon_ops 2024-04-14T15:19:26.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:26.066+0000 7f30ec8df700 10 client.?.objecter ms_handle_connect 0x7f30e810f2f0 2024-04-14T15:19:26.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:26.066+0000 7f30ec8df700 10 client.?.objecter resend_mon_ops 2024-04-14T15:19:26.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:26.066+0000 7f30ec8df700 1 -- 172.21.15.90:0/3602299662 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 820026384 0 0) 0x7f30e810efa0 con 0x7f30e810b720 2024-04-14T15:19:26.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:26.066+0000 7f30ec8df700 1 -- 172.21.15.90:0/3602299662 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f30d00035b0 con 0x7f30e810b720 2024-04-14T15:19:26.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:26.066+0000 7f30ec8df700 1 -- 172.21.15.90:0/3602299662 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3938628522 0 0) 0x7f30e81aa0e0 con 0x7f30e8108870 2024-04-14T15:19:26.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:26.066+0000 7f30ec8df700 1 -- 172.21.15.90:0/3602299662 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f30e810efa0 con 0x7f30e8108870 2024-04-14T15:19:26.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:26.066+0000 7f30ec8df700 1 -- 172.21.15.90:0/3602299662 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1184047979 0 0) 0x7f30e81a8f00 con 0x7f30e810f2f0 2024-04-14T15:19:26.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:26.066+0000 7f30ec8df700 1 -- 172.21.15.90:0/3602299662 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f30e81aa0e0 con 0x7f30e810f2f0 2024-04-14T15:19:26.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:26.066+0000 7f30ec8df700 1 -- 172.21.15.90:0/3602299662 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1557212625 0 0) 0x7f30d00035b0 con 0x7f30e810b720 2024-04-14T15:19:26.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:26.066+0000 7f30ec8df700 1 -- 172.21.15.90:0/3602299662 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f30e81a8f00 con 0x7f30e810b720 2024-04-14T15:19:26.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:26.066+0000 7f30ec8df700 1 -- 172.21.15.90:0/3602299662 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 754761998 0 0) 0x7f30e810efa0 con 0x7f30e8108870 2024-04-14T15:19:26.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:26.066+0000 7f30ec8df700 1 -- 172.21.15.90:0/3602299662 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f30d00035b0 con 0x7f30e8108870 2024-04-14T15:19:26.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:26.066+0000 7f30ec8df700 1 -- 172.21.15.90:0/3602299662 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f30dc0018d0 con 0x7f30e810b720 2024-04-14T15:19:26.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:26.066+0000 7f30ec8df700 1 -- 172.21.15.90:0/3602299662 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 192017485 0 0) 0x7f30e81aa0e0 con 0x7f30e810f2f0 2024-04-14T15:19:26.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:26.066+0000 7f30ec8df700 1 -- 172.21.15.90:0/3602299662 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f30e810efa0 con 0x7f30e810f2f0 2024-04-14T15:19:26.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:26.066+0000 7f30ec8df700 1 -- 172.21.15.90:0/3602299662 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f30d8001830 con 0x7f30e8108870 2024-04-14T15:19:26.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:26.066+0000 7f30ec8df700 1 -- 172.21.15.90:0/3602299662 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f30e00017b0 con 0x7f30e810f2f0 2024-04-14T15:19:26.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:26.066+0000 7f30ec8df700 1 -- 172.21.15.90:0/3602299662 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3221194478 0 0) 0x7f30e81a8f00 con 0x7f30e810b720 2024-04-14T15:19:26.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:26.066+0000 7f30ec8df700 1 -- 172.21.15.90:0/3602299662 >> v1:172.21.15.90:6790/0 conn(0x7f30e8108870 legacy=0x7f30e810d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:26.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:26.066+0000 7f30ec8df700 1 -- 172.21.15.90:0/3602299662 >> v1:172.21.15.92:6789/0 conn(0x7f30e810f2f0 legacy=0x7f30e81a67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:26.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:26.066+0000 7f30ec8df700 1 -- 172.21.15.90:0/3602299662 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30e81ab2c0 con 0x7f30e810b720 2024-04-14T15:19:26.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:26.066+0000 7f30ed8e1700 1 -- 172.21.15.90:0/3602299662 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f30e81a7f50 con 0x7f30e810b720 2024-04-14T15:19:26.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:26.066+0000 7f30ed8e1700 10 client.37389.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:19:26.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:26.066+0000 7f30ed8e1700 1 -- 172.21.15.90:0/3602299662 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f30e81a8410 con 0x7f30e810b720 2024-04-14T15:19:26.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:26.066+0000 7f30ed8e1700 1 librados: init done 2024-04-14T15:19:26.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:26.067+0000 7f30ec8df700 1 -- 172.21.15.90:0/3602299662 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f30dc001ca0 con 0x7f30e810b720 2024-04-14T15:19:26.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:26.067+0000 7f30ec8df700 1 -- 172.21.15.90:0/3602299662 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f30dc005d10 con 0x7f30e810b720 2024-04-14T15:19:26.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:26.067+0000 7f30ed8e1700 1 -- 172.21.15.90:0/3602299662 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30d4005120 con 0x7f30e810b720 2024-04-14T15:19:26.071 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:19:26.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:26.068+0000 7f30ec8df700 1 -- 172.21.15.90:0/3602299662 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f30dc003950 con 0x7f30e810b720 2024-04-14T15:19:26.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:26.069+0000 7f30ec8df700 1 -- 172.21.15.90:0/3602299662 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(466..466 src has 51..466) ==== 8160+0+0 (unknown 957153644 0 0) 0x7f30dc08aa10 con 0x7f30e810b720 2024-04-14T15:19:26.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:26.069+0000 7f30ec8df700 10 client.37389.objecter ms_dispatch 0x7f30e8001550 osd_map(466..466 src has 51..466) 2024-04-14T15:19:26.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:26.069+0000 7f30ec8df700 3 client.37389.objecter handle_osd_map got epochs [466,466] > 0 2024-04-14T15:19:26.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:26.069+0000 7f30ec8df700 3 client.37389.objecter handle_osd_map decoding full epoch 466 2024-04-14T15:19:26.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:26.069+0000 7f30ec8df700 20 client.37389.objecter dump_active .. 0 homeless 2024-04-14T15:19:26.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:26.069+0000 7f30ec8df700 10 client.37389.objecter ms_handle_connect 0x7f30d0071350 2024-04-14T15:19:26.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:26.073+0000 7f30ec8df700 1 -- 172.21.15.90:0/3602299662 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f30dc005d10 con 0x7f30e810b720 2024-04-14T15:19:26.095 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:26.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:26.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:19:26.197 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:19:26.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:26.221 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:26.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:19:26.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:26.292+0000 7f30ed8e1700 1 -- 172.21.15.90:0/3602299662 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7f30d4005410 con 0x7f30e810b720 2024-04-14T15:19:26.293 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:26 smithi090 ceph-mon[41670]: 6.14 scrub starts 2024-04-14T15:19:26.293 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:26 smithi090 ceph-mon[41670]: 6.14 scrub ok 2024-04-14T15:19:26.293 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:26 smithi090 ceph-mon[41670]: 2.14 scrub starts 2024-04-14T15:19:26.293 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:26 smithi090 ceph-mon[41670]: 2.14 scrub ok 2024-04-14T15:19:26.293 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:26 smithi090 ceph-mon[41670]: 110.1f deep-scrub starts 2024-04-14T15:19:26.293 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:26 smithi090 ceph-mon[41670]: 110.1f deep-scrub ok 2024-04-14T15:19:26.293 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:26 smithi090 ceph-mon[41670]: pgmap v661: 258 pgs: 258 active+clean; 8.4 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-04-14T15:19:26.293 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:26 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:19:26.294 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:26 smithi090 ceph-mon[41670]: 5.5 scrub starts 2024-04-14T15:19:26.294 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:26 smithi090 ceph-mon[41670]: 5.5 scrub ok 2024-04-14T15:19:26.294 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:26 smithi090 ceph-mon[30399]: 6.14 scrub starts 2024-04-14T15:19:26.294 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:26 smithi090 ceph-mon[30399]: 6.14 scrub ok 2024-04-14T15:19:26.294 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:26 smithi090 ceph-mon[30399]: 2.14 scrub starts 2024-04-14T15:19:26.294 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:26 smithi090 ceph-mon[30399]: 2.14 scrub ok 2024-04-14T15:19:26.294 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:26 smithi090 ceph-mon[30399]: 110.1f deep-scrub starts 2024-04-14T15:19:26.294 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:26 smithi090 ceph-mon[30399]: 110.1f deep-scrub ok 2024-04-14T15:19:26.294 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:26 smithi090 ceph-mon[30399]: pgmap v661: 258 pgs: 258 active+clean; 8.4 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-04-14T15:19:26.294 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:26 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:19:26.294 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:26 smithi090 ceph-mon[30399]: 5.5 scrub starts 2024-04-14T15:19:26.295 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:26 smithi090 ceph-mon[30399]: 5.5 scrub ok 2024-04-14T15:19:26.322 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:19:26.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:26.347 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:26.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:19:26.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:26 smithi092 ceph-mon[33966]: 6.14 scrub starts 2024-04-14T15:19:26.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:26 smithi092 ceph-mon[33966]: 6.14 scrub ok 2024-04-14T15:19:26.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:26 smithi092 ceph-mon[33966]: 2.14 scrub starts 2024-04-14T15:19:26.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:26 smithi092 ceph-mon[33966]: 2.14 scrub ok 2024-04-14T15:19:26.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:26 smithi092 ceph-mon[33966]: 110.1f deep-scrub starts 2024-04-14T15:19:26.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:26 smithi092 ceph-mon[33966]: 110.1f deep-scrub ok 2024-04-14T15:19:26.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:26 smithi092 ceph-mon[33966]: pgmap v661: 258 pgs: 258 active+clean; 8.4 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-04-14T15:19:26.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:26 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:19:26.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:26 smithi092 ceph-mon[33966]: 5.5 scrub starts 2024-04-14T15:19:26.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:26 smithi092 ceph-mon[33966]: 5.5 scrub ok 2024-04-14T15:19:26.448 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:19:26.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:26.472 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:26.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:19:26.573 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:26.598 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:26.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:26.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:26.700 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:26.724 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:26.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:26.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:26.826 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:19:26.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:26.850 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:26.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:19:26.927 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:19:26.928 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:19:26.951 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:19:26.976 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:26.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:26.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:19:27.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:27.046+0000 7f30ec8df700 1 -- 172.21.15.90:0/3602299662 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v467) ==== 87+0+0 (unknown 517066922 0 0) 0x7f30dc057430 con 0x7f30e810b720 2024-04-14T15:19:27.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:27.048+0000 7f30ed8e1700 10 librados: watch_flush enter 2024-04-14T15:19:27.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:27.048+0000 7f30ed8e1700 10 librados: watch_flush exit 2024-04-14T15:19:27.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:27.048+0000 7f30ed8e1700 20 client.37389.objecter shutdown clearing up homeless session... 2024-04-14T15:19:27.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:27.048+0000 7f30ed8e1700 10 client.37389.objecter successfully canceled tick 2024-04-14T15:19:27.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:27.048+0000 7f30ed8e1700 1 -- 172.21.15.90:0/3602299662 >> v1:172.21.15.90:6800/420935427 conn(0x7f30d0071350 legacy=0x7f30d0073810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:27.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:27.048+0000 7f30ed8e1700 1 -- 172.21.15.90:0/3602299662 >> v1:172.21.15.90:6789/0 conn(0x7f30e810b720 legacy=0x7f30e810df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:27.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:27.048+0000 7f30ed8e1700 1 -- 172.21.15.90:0/3602299662 shutdown_connections 2024-04-14T15:19:27.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:27.048+0000 7f30ed8e1700 1 -- 172.21.15.90:0/3602299662 >> 172.21.15.90:0/3602299662 conn(0x7f30e80fe3c0 msgr2=0x7f30e8102870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:19:27.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:27.049+0000 7f30ed8e1700 1 -- 172.21.15.90:0/3602299662 shutdown_connections 2024-04-14T15:19:27.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:27.049+0000 7f30ed8e1700 1 -- 172.21.15.90:0/3602299662 wait complete. 2024-04-14T15:19:27.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:27.049+0000 7f30ed8e1700 1 librados: shutdown 2024-04-14T15:19:27.050 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is set 2024-04-14T15:19:27.077 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:27.101 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:27.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:27.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:27.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:27 smithi090 ceph-mon[41670]: 2.1b scrub starts 2024-04-14T15:19:27.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:27 smithi090 ceph-mon[41670]: 2.1b scrub ok 2024-04-14T15:19:27.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:27 smithi090 ceph-mon[41670]: 4.7 scrub starts 2024-04-14T15:19:27.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:27 smithi090 ceph-mon[41670]: 4.7 scrub ok 2024-04-14T15:19:27.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:27 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3602299662' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:19:27.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:27 smithi090 ceph-mon[30399]: 2.1b scrub starts 2024-04-14T15:19:27.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:27 smithi090 ceph-mon[30399]: 2.1b scrub ok 2024-04-14T15:19:27.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:27 smithi090 ceph-mon[30399]: 4.7 scrub starts 2024-04-14T15:19:27.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:27 smithi090 ceph-mon[30399]: 4.7 scrub ok 2024-04-14T15:19:27.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:27 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3602299662' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:19:27.202 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:19:27.227 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:27.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:27.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:19:27.328 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:19:27.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:27.353 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:27.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:19:27.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:27 smithi092 ceph-mon[33966]: 2.1b scrub starts 2024-04-14T15:19:27.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:27 smithi092 ceph-mon[33966]: 2.1b scrub ok 2024-04-14T15:19:27.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:27 smithi092 ceph-mon[33966]: 4.7 scrub starts 2024-04-14T15:19:27.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:27 smithi092 ceph-mon[33966]: 4.7 scrub ok 2024-04-14T15:19:27.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:27 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3602299662' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:19:27.454 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:19:27.479 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:27.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:27.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:19:27.581 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:19:27.604 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:27.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:27.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:19:27.706 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:27.731 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:27.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:27.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:27.832 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:19:27.857 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:27.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:27.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:19:27.960 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:19:27.986 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:27.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:27.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:19:28.087 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:19:28.111 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:28.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:28.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:19:28.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:28 smithi090 ceph-mon[41670]: 4.0 scrub starts 2024-04-14T15:19:28.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:28 smithi090 ceph-mon[41670]: 4.0 scrub ok 2024-04-14T15:19:28.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:28 smithi090 ceph-mon[41670]: 2.b scrub starts 2024-04-14T15:19:28.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:28 smithi090 ceph-mon[41670]: 2.b scrub ok 2024-04-14T15:19:28.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:28 smithi090 ceph-mon[41670]: 2.1d scrub starts 2024-04-14T15:19:28.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:28 smithi090 ceph-mon[41670]: 2.1d scrub ok 2024-04-14T15:19:28.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:28 smithi090 ceph-mon[41670]: 11.1c scrub starts 2024-04-14T15:19:28.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:28 smithi090 ceph-mon[41670]: 11.1c scrub ok 2024-04-14T15:19:28.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:28 smithi090 ceph-mon[41670]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:19:28.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:28 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3602299662' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-14T15:19:28.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:28 smithi090 ceph-mon[41670]: osdmap e467: 8 total, 8 up, 8 in 2024-04-14T15:19:28.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:28 smithi090 ceph-mon[41670]: pgmap v663: 258 pgs: 258 active+clean; 8.4 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:19:28.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:28 smithi090 ceph-mon[30399]: 4.0 scrub starts 2024-04-14T15:19:28.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:28 smithi090 ceph-mon[30399]: 4.0 scrub ok 2024-04-14T15:19:28.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:28 smithi090 ceph-mon[30399]: 2.b scrub starts 2024-04-14T15:19:28.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:28 smithi090 ceph-mon[30399]: 2.b scrub ok 2024-04-14T15:19:28.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:28 smithi090 ceph-mon[30399]: 2.1d scrub starts 2024-04-14T15:19:28.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:28 smithi090 ceph-mon[30399]: 2.1d scrub ok 2024-04-14T15:19:28.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:28 smithi090 ceph-mon[30399]: 11.1c scrub starts 2024-04-14T15:19:28.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:28 smithi090 ceph-mon[30399]: 11.1c scrub ok 2024-04-14T15:19:28.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:28 smithi090 ceph-mon[30399]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:19:28.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:28 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3602299662' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-14T15:19:28.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:28 smithi090 ceph-mon[30399]: osdmap e467: 8 total, 8 up, 8 in 2024-04-14T15:19:28.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:28 smithi090 ceph-mon[30399]: pgmap v663: 258 pgs: 258 active+clean; 8.4 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:19:28.214 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:19:28.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:28.238 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:28.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:19:28.339 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:19:28.363 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:28.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:28.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:19:28.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:28 smithi092 ceph-mon[33966]: 4.0 scrub starts 2024-04-14T15:19:28.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:28 smithi092 ceph-mon[33966]: 4.0 scrub ok 2024-04-14T15:19:28.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:28 smithi092 ceph-mon[33966]: 2.b scrub starts 2024-04-14T15:19:28.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:28 smithi092 ceph-mon[33966]: 2.b scrub ok 2024-04-14T15:19:28.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:28 smithi092 ceph-mon[33966]: 2.1d scrub starts 2024-04-14T15:19:28.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:28 smithi092 ceph-mon[33966]: 2.1d scrub ok 2024-04-14T15:19:28.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:28 smithi092 ceph-mon[33966]: 11.1c scrub starts 2024-04-14T15:19:28.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:28 smithi092 ceph-mon[33966]: 11.1c scrub ok 2024-04-14T15:19:28.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:28 smithi092 ceph-mon[33966]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:19:28.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:28 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3602299662' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-14T15:19:28.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:28 smithi092 ceph-mon[33966]: osdmap e467: 8 total, 8 up, 8 in 2024-04-14T15:19:28.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:28 smithi092 ceph-mon[33966]: pgmap v663: 258 pgs: 258 active+clean; 8.4 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:19:28.465 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:19:28.489 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:28.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:28.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:19:28.591 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:19:28.615 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:28.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:28.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:19:28.716 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:19:28.739 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:28.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:28.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:19:28.841 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:19:28.864 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:28.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:28.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:19:28.966 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:19:28.989 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:28.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:28.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:19:29.091 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:19:29.119 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:29.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:29.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:19:29.221 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:29.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:29.249 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:29.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:29.350 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:19:29.374 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:29.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:29.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:19:29.476 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:29.503 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:29.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:29.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:29.605 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:19:29.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:29.633 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:29.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:19:29.734 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:19:29.761 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:29.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:29.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:19:29.863 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:19:29.890 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:29.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:29.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:19:29.991 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:19:30.016 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:30.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:30.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:19:30.118 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:19:30.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:30.142 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:30.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:19:30.243 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:19:30.267 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:30.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:30.268 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:19:30.272 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:30 smithi090 ceph-mon[41670]: pgmap v664: 258 pgs: 258 active+clean; 8.4 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 434 B/s wr, 2 op/s 2024-04-14T15:19:30.273 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:30 smithi090 ceph-mon[30399]: pgmap v664: 258 pgs: 258 active+clean; 8.4 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 434 B/s wr, 2 op/s 2024-04-14T15:19:30.369 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:30.397 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:30.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:30.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:30.499 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:19:30.523 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:30.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:30.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:19:30.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:30 smithi092 ceph-mon[33966]: pgmap v664: 258 pgs: 258 active+clean; 8.4 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 434 B/s wr, 2 op/s 2024-04-14T15:19:30.625 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:19:30.631 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-14T15:19:30.631 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-14T15:19:30.631 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-14T15:19:30.632 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-14T15:19:30.632 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-14T15:19:30.649 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:30.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:30.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:19:30.751 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:19:30.778 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:30.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:30.779 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:19:30.880 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:19:30.903 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:30.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:30.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:19:31.005 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:19:31.029 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:31.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:31.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:19:31.131 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:19:31.155 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:31.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:31.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:19:31.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.233+0000 7fe3a7da0700 1 -- 172.21.15.90:0/221992720 >> v1:172.21.15.90:6789/0 conn(0x7fe3a010b720 legacy=0x7fe3a010db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:31.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.233+0000 7fe3a7da0700 1 -- 172.21.15.90:0/221992720 shutdown_connections 2024-04-14T15:19:31.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.233+0000 7fe3a7da0700 1 -- 172.21.15.90:0/221992720 >> 172.21.15.90:0/221992720 conn(0x7fe3a00fe3a0 msgr2=0x7fe3a01007c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:19:31.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.233+0000 7fe3a7da0700 1 -- 172.21.15.90:0/221992720 shutdown_connections 2024-04-14T15:19:31.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.233+0000 7fe3a7da0700 1 -- 172.21.15.90:0/221992720 wait complete. 2024-04-14T15:19:31.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.233+0000 7fe3a7da0700 1 librados: starting msgr at 2024-04-14T15:19:31.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.233+0000 7fe3a7da0700 1 librados: starting objecter 2024-04-14T15:19:31.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.234+0000 7fe3a7da0700 1 Processor -- start 2024-04-14T15:19:31.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.234+0000 7fe3a7da0700 1 -- start start 2024-04-14T15:19:31.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.234+0000 7fe3a7da0700 1 librados: setting wanted keys 2024-04-14T15:19:31.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.234+0000 7fe3a7da0700 1 librados: calling monclient init 2024-04-14T15:19:31.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.234+0000 7fe3a7da0700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe3a01a7a20 con 0x7fe3a010b720 2024-04-14T15:19:31.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.234+0000 7fe3a7da0700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe3a01a8c40 con 0x7fe3a010f2f0 2024-04-14T15:19:31.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.234+0000 7fe3a7da0700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe3a01a9e80 con 0x7fe3a0108870 2024-04-14T15:19:31.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.234+0000 7fe3a533b700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fe3a010b720 0x7fe3a01a27a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:39030/0 (socket says 172.21.15.90:39030) 2024-04-14T15:19:31.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.234+0000 7fe3a533b700 1 -- 172.21.15.90:0/2072206776 learned_addr learned my addr 172.21.15.90:0/2072206776 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:19:31.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.234+0000 7fe3a5b3c700 1 --1- 172.21.15.90:0/2072206776 >> v1:172.21.15.90:6790/0 conn(0x7fe3a0108870 0x7fe3a010ebb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:53110/0 (socket says 172.21.15.90:53110) 2024-04-14T15:19:31.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.234+0000 7fe396ffd700 10 client.?.objecter ms_handle_connect 0x7fe3a010b720 2024-04-14T15:19:31.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.234+0000 7fe396ffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:19:31.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.234+0000 7fe396ffd700 10 client.?.objecter ms_handle_connect 0x7fe3a0108870 2024-04-14T15:19:31.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.234+0000 7fe396ffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:19:31.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.234+0000 7fe396ffd700 10 client.?.objecter ms_handle_connect 0x7fe3a010f2f0 2024-04-14T15:19:31.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.234+0000 7fe396ffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:19:31.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.234+0000 7fe396ffd700 1 -- 172.21.15.90:0/2072206776 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2388974768 0 0) 0x7fe3a01a7a20 con 0x7fe3a010b720 2024-04-14T15:19:31.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.234+0000 7fe396ffd700 1 -- 172.21.15.90:0/2072206776 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe38c0035b0 con 0x7fe3a010b720 2024-04-14T15:19:31.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.234+0000 7fe396ffd700 1 -- 172.21.15.90:0/2072206776 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2084074318 0 0) 0x7fe3a01a9e80 con 0x7fe3a0108870 2024-04-14T15:19:31.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.234+0000 7fe396ffd700 1 -- 172.21.15.90:0/2072206776 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe3a01a7a20 con 0x7fe3a0108870 2024-04-14T15:19:31.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.234+0000 7fe396ffd700 1 -- 172.21.15.90:0/2072206776 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 699572771 0 0) 0x7fe3a01a8c40 con 0x7fe3a010f2f0 2024-04-14T15:19:31.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.234+0000 7fe396ffd700 1 -- 172.21.15.90:0/2072206776 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe3a01a9e80 con 0x7fe3a010f2f0 2024-04-14T15:19:31.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.234+0000 7fe396ffd700 1 -- 172.21.15.90:0/2072206776 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 59253732 0 0) 0x7fe38c0035b0 con 0x7fe3a010b720 2024-04-14T15:19:31.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.234+0000 7fe396ffd700 1 -- 172.21.15.90:0/2072206776 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe3a01a8c40 con 0x7fe3a010b720 2024-04-14T15:19:31.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.234+0000 7fe396ffd700 1 -- 172.21.15.90:0/2072206776 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fe3900046a0 con 0x7fe3a010b720 2024-04-14T15:19:31.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.234+0000 7fe396ffd700 1 -- 172.21.15.90:0/2072206776 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4104255513 0 0) 0x7fe3a01a7a20 con 0x7fe3a0108870 2024-04-14T15:19:31.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.234+0000 7fe396ffd700 1 -- 172.21.15.90:0/2072206776 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe38c0035b0 con 0x7fe3a0108870 2024-04-14T15:19:31.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.235+0000 7fe396ffd700 1 -- 172.21.15.90:0/2072206776 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fe39c001a30 con 0x7fe3a0108870 2024-04-14T15:19:31.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.235+0000 7fe396ffd700 1 -- 172.21.15.90:0/2072206776 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3612403836 0 0) 0x7fe3a01a8c40 con 0x7fe3a010b720 2024-04-14T15:19:31.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.235+0000 7fe396ffd700 1 -- 172.21.15.90:0/2072206776 >> v1:172.21.15.90:6790/0 conn(0x7fe3a0108870 legacy=0x7fe3a010ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:31.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.235+0000 7fe396ffd700 1 -- 172.21.15.90:0/2072206776 >> v1:172.21.15.92:6789/0 conn(0x7fe3a010f2f0 legacy=0x7fe3a01a60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:31.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.235+0000 7fe396ffd700 1 -- 172.21.15.90:0/2072206776 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3a01ab0c0 con 0x7fe3a010b720 2024-04-14T15:19:31.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.235+0000 7fe3a7da0700 1 -- 172.21.15.90:0/2072206776 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe3a01aa0b0 con 0x7fe3a010b720 2024-04-14T15:19:31.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.235+0000 7fe3a7da0700 10 client.37395.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:19:31.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.235+0000 7fe3a7da0700 1 -- 172.21.15.90:0/2072206776 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe3a01aa570 con 0x7fe3a010b720 2024-04-14T15:19:31.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.235+0000 7fe3a7da0700 1 librados: init done 2024-04-14T15:19:31.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.235+0000 7fe396ffd700 1 -- 172.21.15.90:0/2072206776 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fe390001b50 con 0x7fe3a010b720 2024-04-14T15:19:31.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.235+0000 7fe396ffd700 1 -- 172.21.15.90:0/2072206776 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fe390004da0 con 0x7fe3a010b720 2024-04-14T15:19:31.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.236+0000 7fe3a7da0700 1 -- 172.21.15.90:0/2072206776 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fe384005100 con 0x7fe3a010b720 2024-04-14T15:19:31.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.236+0000 7fe396ffd700 1 -- 172.21.15.90:0/2072206776 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fe390004f40 con 0x7fe3a010b720 2024-04-14T15:19:31.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.237+0000 7fe396ffd700 1 -- 172.21.15.90:0/2072206776 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(468..468 src has 51..468) ==== 8160+0+0 (unknown 1077206174 0 0) 0x7fe39008aa40 con 0x7fe3a010b720 2024-04-14T15:19:31.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.237+0000 7fe396ffd700 10 client.37395.objecter ms_dispatch 0x7fe3a0001550 osd_map(468..468 src has 51..468) 2024-04-14T15:19:31.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.237+0000 7fe396ffd700 3 client.37395.objecter handle_osd_map got epochs [468,468] > 0 2024-04-14T15:19:31.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.237+0000 7fe396ffd700 3 client.37395.objecter handle_osd_map decoding full epoch 468 2024-04-14T15:19:31.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.237+0000 7fe396ffd700 20 client.37395.objecter dump_active .. 0 homeless 2024-04-14T15:19:31.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.237+0000 7fe396ffd700 1 -- 172.21.15.90:0/2072206776 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v468) ==== 54+0+16 (unknown 3786319255 0 2420000192) 0x7fe39008aef0 con 0x7fe3a010b720 2024-04-14T15:19:31.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.237+0000 7fe396ffd700 10 client.37395.objecter ms_handle_connect 0x7fe38c071230 2024-04-14T15:19:31.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.237+0000 7fe3a7da0700 10 client.37395.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:19:31.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.237+0000 7fe3a7da0700 15 client.37395.objecter _session_command_op_assign -1 1 2024-04-14T15:19:31.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.237+0000 7fe3a7da0700 20 client.37395.objecter _get_session s=0x7fe3840053e0 osd=0 3 2024-04-14T15:19:31.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.237+0000 7fe3a7da0700 20 client.37395.objecter put_session s=0x7fe3840053e0 osd=0 3 2024-04-14T15:19:31.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.237+0000 7fe3a7da0700 20 client.37395.objecter _get_session s=0x7fe3840053e0 osd=0 3 2024-04-14T15:19:31.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.237+0000 7fe3a7da0700 15 client.37395.objecter _session_command_op_remove -1 1 2024-04-14T15:19:31.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.237+0000 7fe3a7da0700 20 client.37395.objecter get_session s=0x7fe3840053e0 osd=0 3 2024-04-14T15:19:31.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.237+0000 7fe3a7da0700 15 client.37395.objecter _session_command_op_assign 0 1 2024-04-14T15:19:31.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.237+0000 7fe3a7da0700 20 client.37395.objecter put_session s=0x7fe3840053e0 osd=0 4 2024-04-14T15:19:31.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.237+0000 7fe3a7da0700 10 client.37395.objecter _send_command 1 2024-04-14T15:19:31.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.237+0000 7fe3a7da0700 1 -- 172.21.15.90:0/2072206776 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe384004b70 con 0x7fe384004730 2024-04-14T15:19:31.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.237+0000 7fe396ffd700 10 client.37395.objecter ms_handle_connect 0x7fe384004730 2024-04-14T15:19:31.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.238+0000 7fe396ffd700 1 -- 172.21.15.90:0/2072206776 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fe384004b70 con 0x7fe384004730 2024-04-14T15:19:31.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.238+0000 7fe396ffd700 10 client.37395.objecter ms_dispatch 0x7fe3a0001550 command_reply(tid 1: 0 ) 2024-04-14T15:19:31.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.238+0000 7fe396ffd700 10 client.37395.objecter _finish_command 1 = system:0 2024-04-14T15:19:31.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.238+0000 7fe396ffd700 20 client.37395.objecter put_session s=0x7fe3840053e0 osd=0 4 2024-04-14T15:19:31.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.238+0000 7fe396ffd700 15 client.37395.objecter _session_command_op_remove 0 1 2024-04-14T15:19:31.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.255+0000 7fe3a7da0700 10 client.37395.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:19:31.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.255+0000 7fe3a7da0700 15 client.37395.objecter _session_command_op_assign -1 2 2024-04-14T15:19:31.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.255+0000 7fe3a7da0700 20 client.37395.objecter _get_session s=0x7fe3840053e0 osd=0 3 2024-04-14T15:19:31.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.255+0000 7fe3a7da0700 20 client.37395.objecter put_session s=0x7fe3840053e0 osd=0 3 2024-04-14T15:19:31.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.255+0000 7fe3a7da0700 20 client.37395.objecter _get_session s=0x7fe3840053e0 osd=0 3 2024-04-14T15:19:31.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.255+0000 7fe3a7da0700 15 client.37395.objecter _session_command_op_remove -1 2 2024-04-14T15:19:31.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.255+0000 7fe3a7da0700 20 client.37395.objecter get_session s=0x7fe3840053e0 osd=0 3 2024-04-14T15:19:31.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.255+0000 7fe3a7da0700 15 client.37395.objecter _session_command_op_assign 0 2 2024-04-14T15:19:31.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.255+0000 7fe3a7da0700 20 client.37395.objecter put_session s=0x7fe3840053e0 osd=0 4 2024-04-14T15:19:31.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.255+0000 7fe3a7da0700 10 client.37395.objecter _send_command 2 2024-04-14T15:19:31.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.255+0000 7fe3a7da0700 1 -- 172.21.15.90:0/2072206776 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe3840079f0 con 0x7fe384004730 2024-04-14T15:19:31.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.256+0000 7fe396ffd700 1 -- 172.21.15.90:0/2072206776 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fe3840079f0 con 0x7fe384004730 2024-04-14T15:19:31.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.256+0000 7fe396ffd700 10 client.37395.objecter ms_dispatch 0x7fe3a0001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:19:31.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.256+0000 7fe396ffd700 10 client.37395.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:19:31.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.256+0000 7fe396ffd700 20 client.37395.objecter put_session s=0x7fe3840053e0 osd=0 4 2024-04-14T15:19:31.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.256+0000 7fe396ffd700 15 client.37395.objecter _session_command_op_remove 0 2 2024-04-14T15:19:31.257 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:19:31.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.257+0000 7fe3a7da0700 10 client.37395.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:19:31.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.257+0000 7fe3a7da0700 15 client.37395.objecter _session_command_op_assign -1 3 2024-04-14T15:19:31.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.257+0000 7fe3a7da0700 20 client.37395.objecter _get_session s=0x7fe384004130 osd=1 3 2024-04-14T15:19:31.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.257+0000 7fe3a7da0700 20 client.37395.objecter put_session s=0x7fe384004130 osd=1 3 2024-04-14T15:19:31.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.257+0000 7fe3a7da0700 20 client.37395.objecter _get_session s=0x7fe384004130 osd=1 3 2024-04-14T15:19:31.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.257+0000 7fe3a7da0700 15 client.37395.objecter _session_command_op_remove -1 3 2024-04-14T15:19:31.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.257+0000 7fe3a7da0700 20 client.37395.objecter get_session s=0x7fe384004130 osd=1 3 2024-04-14T15:19:31.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.257+0000 7fe3a7da0700 15 client.37395.objecter _session_command_op_assign 1 3 2024-04-14T15:19:31.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.257+0000 7fe3a7da0700 20 client.37395.objecter put_session s=0x7fe384004130 osd=1 4 2024-04-14T15:19:31.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.257+0000 7fe3a7da0700 10 client.37395.objecter _send_command 3 2024-04-14T15:19:31.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.257+0000 7fe3a7da0700 1 -- 172.21.15.90:0/2072206776 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fe3840022a0 con 0x7fe384004310 2024-04-14T15:19:31.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.257+0000 7fe396ffd700 10 client.37395.objecter ms_handle_connect 0x7fe384004310 2024-04-14T15:19:31.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.258+0000 7fe396ffd700 1 -- 172.21.15.90:0/2072206776 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fe3840022a0 con 0x7fe384004310 2024-04-14T15:19:31.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.258+0000 7fe396ffd700 10 client.37395.objecter ms_dispatch 0x7fe3a0001550 command_reply(tid 3: 0 ) 2024-04-14T15:19:31.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.258+0000 7fe396ffd700 10 client.37395.objecter _finish_command 3 = system:0 2024-04-14T15:19:31.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.258+0000 7fe396ffd700 20 client.37395.objecter put_session s=0x7fe384004130 osd=1 4 2024-04-14T15:19:31.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.258+0000 7fe396ffd700 15 client.37395.objecter _session_command_op_remove 1 3 2024-04-14T15:19:31.260 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:19:31.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.275+0000 7fe3a7da0700 10 client.37395.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:19:31.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.275+0000 7fe3a7da0700 15 client.37395.objecter _session_command_op_assign -1 4 2024-04-14T15:19:31.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.275+0000 7fe3a7da0700 20 client.37395.objecter _get_session s=0x7fe384004130 osd=1 3 2024-04-14T15:19:31.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.275+0000 7fe3a7da0700 20 client.37395.objecter put_session s=0x7fe384004130 osd=1 3 2024-04-14T15:19:31.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.275+0000 7fe3a7da0700 20 client.37395.objecter _get_session s=0x7fe384004130 osd=1 3 2024-04-14T15:19:31.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.275+0000 7fe3a7da0700 15 client.37395.objecter _session_command_op_remove -1 4 2024-04-14T15:19:31.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.275+0000 7fe3a7da0700 20 client.37395.objecter get_session s=0x7fe384004130 osd=1 3 2024-04-14T15:19:31.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.275+0000 7fe3a7da0700 15 client.37395.objecter _session_command_op_assign 1 4 2024-04-14T15:19:31.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.275+0000 7fe3a7da0700 20 client.37395.objecter put_session s=0x7fe384004130 osd=1 4 2024-04-14T15:19:31.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.275+0000 7fe3a7da0700 10 client.37395.objecter _send_command 4 2024-04-14T15:19:31.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.275+0000 7fe3a7da0700 1 -- 172.21.15.90:0/2072206776 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe384001720 con 0x7fe384004310 2024-04-14T15:19:31.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.277+0000 7fe396ffd700 1 -- 172.21.15.90:0/2072206776 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fe384001720 con 0x7fe384004310 2024-04-14T15:19:31.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.277+0000 7fe396ffd700 10 client.37395.objecter ms_dispatch 0x7fe3a0001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:19:31.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.277+0000 7fe396ffd700 10 client.37395.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:19:31.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.277+0000 7fe396ffd700 20 client.37395.objecter put_session s=0x7fe384004130 osd=1 4 2024-04-14T15:19:31.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.277+0000 7fe396ffd700 15 client.37395.objecter _session_command_op_remove 1 4 2024-04-14T15:19:31.278 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:19:31.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.278+0000 7fe3a7da0700 10 client.37395.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:19:31.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.278+0000 7fe3a7da0700 15 client.37395.objecter _session_command_op_assign -1 5 2024-04-14T15:19:31.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.278+0000 7fe3a7da0700 20 client.37395.objecter _get_session s=0x7fe384001980 osd=2 3 2024-04-14T15:19:31.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.278+0000 7fe3a7da0700 20 client.37395.objecter put_session s=0x7fe384001980 osd=2 3 2024-04-14T15:19:31.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.278+0000 7fe3a7da0700 20 client.37395.objecter _get_session s=0x7fe384001980 osd=2 3 2024-04-14T15:19:31.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.278+0000 7fe3a7da0700 15 client.37395.objecter _session_command_op_remove -1 5 2024-04-14T15:19:31.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.278+0000 7fe3a7da0700 20 client.37395.objecter get_session s=0x7fe384001980 osd=2 3 2024-04-14T15:19:31.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.278+0000 7fe3a7da0700 15 client.37395.objecter _session_command_op_assign 2 5 2024-04-14T15:19:31.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.278+0000 7fe3a7da0700 20 client.37395.objecter put_session s=0x7fe384001980 osd=2 4 2024-04-14T15:19:31.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.278+0000 7fe3a7da0700 10 client.37395.objecter _send_command 5 2024-04-14T15:19:31.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.278+0000 7fe3a7da0700 1 -- 172.21.15.90:0/2072206776 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fe384011170 con 0x7fe38400d5d0 2024-04-14T15:19:31.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.278+0000 7fe396ffd700 10 client.37395.objecter ms_handle_connect 0x7fe38400d5d0 2024-04-14T15:19:31.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.279+0000 7fe396ffd700 1 -- 172.21.15.90:0/2072206776 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fe384011170 con 0x7fe38400d5d0 2024-04-14T15:19:31.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.279+0000 7fe396ffd700 10 client.37395.objecter ms_dispatch 0x7fe3a0001550 command_reply(tid 5: 0 ) 2024-04-14T15:19:31.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.279+0000 7fe396ffd700 10 client.37395.objecter _finish_command 5 = system:0 2024-04-14T15:19:31.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.279+0000 7fe396ffd700 20 client.37395.objecter put_session s=0x7fe384001980 osd=2 4 2024-04-14T15:19:31.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.279+0000 7fe396ffd700 15 client.37395.objecter _session_command_op_remove 2 5 2024-04-14T15:19:31.284 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:31.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:31.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:19:31.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.297+0000 7fe3a7da0700 10 client.37395.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:19:31.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.297+0000 7fe3a7da0700 15 client.37395.objecter _session_command_op_assign -1 6 2024-04-14T15:19:31.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.297+0000 7fe3a7da0700 20 client.37395.objecter _get_session s=0x7fe384001980 osd=2 3 2024-04-14T15:19:31.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.297+0000 7fe3a7da0700 20 client.37395.objecter put_session s=0x7fe384001980 osd=2 3 2024-04-14T15:19:31.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.297+0000 7fe3a7da0700 20 client.37395.objecter _get_session s=0x7fe384001980 osd=2 3 2024-04-14T15:19:31.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.297+0000 7fe3a7da0700 15 client.37395.objecter _session_command_op_remove -1 6 2024-04-14T15:19:31.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.297+0000 7fe3a7da0700 20 client.37395.objecter get_session s=0x7fe384001980 osd=2 3 2024-04-14T15:19:31.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.297+0000 7fe3a7da0700 15 client.37395.objecter _session_command_op_assign 2 6 2024-04-14T15:19:31.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.297+0000 7fe3a7da0700 20 client.37395.objecter put_session s=0x7fe384001980 osd=2 4 2024-04-14T15:19:31.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.297+0000 7fe3a7da0700 10 client.37395.objecter _send_command 6 2024-04-14T15:19:31.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.297+0000 7fe3a7da0700 1 -- 172.21.15.90:0/2072206776 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe384008100 con 0x7fe38400d5d0 2024-04-14T15:19:31.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.299+0000 7fe396ffd700 1 -- 172.21.15.90:0/2072206776 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fe384008100 con 0x7fe38400d5d0 2024-04-14T15:19:31.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.299+0000 7fe396ffd700 10 client.37395.objecter ms_dispatch 0x7fe3a0001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:19:31.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.299+0000 7fe396ffd700 10 client.37395.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:19:31.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.299+0000 7fe396ffd700 20 client.37395.objecter put_session s=0x7fe384001980 osd=2 4 2024-04-14T15:19:31.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.299+0000 7fe396ffd700 15 client.37395.objecter _session_command_op_remove 2 6 2024-04-14T15:19:31.300 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:19:31.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.299+0000 7fe3a7da0700 10 client.37395.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:19:31.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.299+0000 7fe3a7da0700 15 client.37395.objecter _session_command_op_assign -1 7 2024-04-14T15:19:31.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.299+0000 7fe3a7da0700 20 client.37395.objecter _get_session s=0x7fe384008390 osd=3 3 2024-04-14T15:19:31.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.299+0000 7fe3a7da0700 20 client.37395.objecter put_session s=0x7fe384008390 osd=3 3 2024-04-14T15:19:31.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.299+0000 7fe3a7da0700 20 client.37395.objecter _get_session s=0x7fe384008390 osd=3 3 2024-04-14T15:19:31.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.299+0000 7fe3a7da0700 15 client.37395.objecter _session_command_op_remove -1 7 2024-04-14T15:19:31.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.299+0000 7fe3a7da0700 20 client.37395.objecter get_session s=0x7fe384008390 osd=3 3 2024-04-14T15:19:31.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.299+0000 7fe3a7da0700 15 client.37395.objecter _session_command_op_assign 3 7 2024-04-14T15:19:31.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.299+0000 7fe3a7da0700 20 client.37395.objecter put_session s=0x7fe384008390 osd=3 4 2024-04-14T15:19:31.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.299+0000 7fe3a7da0700 10 client.37395.objecter _send_command 7 2024-04-14T15:19:31.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.299+0000 7fe3a7da0700 1 -- 172.21.15.90:0/2072206776 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fe384015fb0 con 0x7fe3840124a0 2024-04-14T15:19:31.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.299+0000 7fe396ffd700 10 client.37395.objecter ms_handle_connect 0x7fe3840124a0 2024-04-14T15:19:31.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.300+0000 7fe396ffd700 1 -- 172.21.15.90:0/2072206776 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fe384015fb0 con 0x7fe3840124a0 2024-04-14T15:19:31.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.300+0000 7fe396ffd700 10 client.37395.objecter ms_dispatch 0x7fe3a0001550 command_reply(tid 7: 0 ) 2024-04-14T15:19:31.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.300+0000 7fe396ffd700 10 client.37395.objecter _finish_command 7 = system:0 2024-04-14T15:19:31.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.300+0000 7fe396ffd700 20 client.37395.objecter put_session s=0x7fe384008390 osd=3 4 2024-04-14T15:19:31.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.300+0000 7fe396ffd700 15 client.37395.objecter _session_command_op_remove 3 7 2024-04-14T15:19:31.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.317+0000 7fe3a7da0700 10 client.37395.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:19:31.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.317+0000 7fe3a7da0700 15 client.37395.objecter _session_command_op_assign -1 8 2024-04-14T15:19:31.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.317+0000 7fe3a7da0700 20 client.37395.objecter _get_session s=0x7fe384008390 osd=3 3 2024-04-14T15:19:31.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.317+0000 7fe3a7da0700 20 client.37395.objecter put_session s=0x7fe384008390 osd=3 3 2024-04-14T15:19:31.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.317+0000 7fe3a7da0700 20 client.37395.objecter _get_session s=0x7fe384008390 osd=3 3 2024-04-14T15:19:31.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.317+0000 7fe3a7da0700 15 client.37395.objecter _session_command_op_remove -1 8 2024-04-14T15:19:31.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.317+0000 7fe3a7da0700 20 client.37395.objecter get_session s=0x7fe384008390 osd=3 3 2024-04-14T15:19:31.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.317+0000 7fe3a7da0700 15 client.37395.objecter _session_command_op_assign 3 8 2024-04-14T15:19:31.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.317+0000 7fe3a7da0700 20 client.37395.objecter put_session s=0x7fe384008390 osd=3 4 2024-04-14T15:19:31.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.317+0000 7fe3a7da0700 10 client.37395.objecter _send_command 8 2024-04-14T15:19:31.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.317+0000 7fe3a7da0700 1 -- 172.21.15.90:0/2072206776 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe38400c9b0 con 0x7fe3840124a0 2024-04-14T15:19:31.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.319+0000 7fe396ffd700 1 -- 172.21.15.90:0/2072206776 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fe38400c9b0 con 0x7fe3840124a0 2024-04-14T15:19:31.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.319+0000 7fe396ffd700 10 client.37395.objecter ms_dispatch 0x7fe3a0001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:19:31.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.319+0000 7fe396ffd700 10 client.37395.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:19:31.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.319+0000 7fe396ffd700 20 client.37395.objecter put_session s=0x7fe384008390 osd=3 4 2024-04-14T15:19:31.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.319+0000 7fe396ffd700 15 client.37395.objecter _session_command_op_remove 3 8 2024-04-14T15:19:31.319 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:19:31.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.319+0000 7fe3a7da0700 10 client.37395.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:19:31.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.319+0000 7fe3a7da0700 15 client.37395.objecter _session_command_op_assign -1 9 2024-04-14T15:19:31.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.319+0000 7fe3a7da0700 20 client.37395.objecter _get_session s=0x7fe38400c760 osd=4 3 2024-04-14T15:19:31.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.319+0000 7fe3a7da0700 20 client.37395.objecter put_session s=0x7fe38400c760 osd=4 3 2024-04-14T15:19:31.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.319+0000 7fe3a7da0700 20 client.37395.objecter _get_session s=0x7fe38400c760 osd=4 3 2024-04-14T15:19:31.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.319+0000 7fe3a7da0700 15 client.37395.objecter _session_command_op_remove -1 9 2024-04-14T15:19:31.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.319+0000 7fe3a7da0700 20 client.37395.objecter get_session s=0x7fe38400c760 osd=4 3 2024-04-14T15:19:31.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.319+0000 7fe3a7da0700 15 client.37395.objecter _session_command_op_assign 4 9 2024-04-14T15:19:31.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.319+0000 7fe3a7da0700 20 client.37395.objecter put_session s=0x7fe38400c760 osd=4 4 2024-04-14T15:19:31.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.319+0000 7fe3a7da0700 10 client.37395.objecter _send_command 9 2024-04-14T15:19:31.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.319+0000 7fe3a7da0700 1 -- 172.21.15.90:0/2072206776 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fe38400c990 con 0x7fe384012000 2024-04-14T15:19:31.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.319+0000 7fe396ffd700 10 client.37395.objecter ms_handle_connect 0x7fe384012000 2024-04-14T15:19:31.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.320+0000 7fe396ffd700 1 -- 172.21.15.90:0/2072206776 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fe38400c990 con 0x7fe384012000 2024-04-14T15:19:31.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.320+0000 7fe396ffd700 10 client.37395.objecter ms_dispatch 0x7fe3a0001550 command_reply(tid 9: 0 ) 2024-04-14T15:19:31.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.320+0000 7fe396ffd700 10 client.37395.objecter _finish_command 9 = system:0 2024-04-14T15:19:31.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.320+0000 7fe396ffd700 20 client.37395.objecter put_session s=0x7fe38400c760 osd=4 4 2024-04-14T15:19:31.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.320+0000 7fe396ffd700 15 client.37395.objecter _session_command_op_remove 4 9 2024-04-14T15:19:31.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.336+0000 7fe3a7da0700 10 client.37395.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:19:31.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.336+0000 7fe3a7da0700 15 client.37395.objecter _session_command_op_assign -1 10 2024-04-14T15:19:31.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.336+0000 7fe3a7da0700 20 client.37395.objecter _get_session s=0x7fe38400c760 osd=4 3 2024-04-14T15:19:31.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.336+0000 7fe3a7da0700 20 client.37395.objecter put_session s=0x7fe38400c760 osd=4 3 2024-04-14T15:19:31.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.336+0000 7fe3a7da0700 20 client.37395.objecter _get_session s=0x7fe38400c760 osd=4 3 2024-04-14T15:19:31.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.337+0000 7fe3a7da0700 15 client.37395.objecter _session_command_op_remove -1 10 2024-04-14T15:19:31.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.337+0000 7fe3a7da0700 20 client.37395.objecter get_session s=0x7fe38400c760 osd=4 3 2024-04-14T15:19:31.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.337+0000 7fe3a7da0700 15 client.37395.objecter _session_command_op_assign 4 10 2024-04-14T15:19:31.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.337+0000 7fe3a7da0700 20 client.37395.objecter put_session s=0x7fe38400c760 osd=4 4 2024-04-14T15:19:31.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.337+0000 7fe3a7da0700 10 client.37395.objecter _send_command 10 2024-04-14T15:19:31.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.337+0000 7fe3a7da0700 1 -- 172.21.15.90:0/2072206776 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe38401c1e0 con 0x7fe384012000 2024-04-14T15:19:31.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.339+0000 7fe396ffd700 1 -- 172.21.15.90:0/2072206776 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fe38401c1e0 con 0x7fe384012000 2024-04-14T15:19:31.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.339+0000 7fe396ffd700 10 client.37395.objecter ms_dispatch 0x7fe3a0001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:19:31.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.339+0000 7fe396ffd700 10 client.37395.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:19:31.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.339+0000 7fe396ffd700 20 client.37395.objecter put_session s=0x7fe38400c760 osd=4 4 2024-04-14T15:19:31.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.339+0000 7fe396ffd700 15 client.37395.objecter _session_command_op_remove 4 10 2024-04-14T15:19:31.339 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:19:31.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.339+0000 7fe3a7da0700 10 client.37395.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:19:31.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.339+0000 7fe3a7da0700 15 client.37395.objecter _session_command_op_assign -1 11 2024-04-14T15:19:31.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.339+0000 7fe3a7da0700 20 client.37395.objecter _get_session s=0x7fe38401c1e0 osd=5 3 2024-04-14T15:19:31.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.339+0000 7fe3a7da0700 20 client.37395.objecter put_session s=0x7fe38401c1e0 osd=5 3 2024-04-14T15:19:31.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.339+0000 7fe3a7da0700 20 client.37395.objecter _get_session s=0x7fe38401c1e0 osd=5 3 2024-04-14T15:19:31.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.339+0000 7fe3a7da0700 15 client.37395.objecter _session_command_op_remove -1 11 2024-04-14T15:19:31.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.339+0000 7fe3a7da0700 20 client.37395.objecter get_session s=0x7fe38401c1e0 osd=5 3 2024-04-14T15:19:31.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.339+0000 7fe3a7da0700 15 client.37395.objecter _session_command_op_assign 5 11 2024-04-14T15:19:31.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.339+0000 7fe3a7da0700 20 client.37395.objecter put_session s=0x7fe38401c1e0 osd=5 4 2024-04-14T15:19:31.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.339+0000 7fe3a7da0700 10 client.37395.objecter _send_command 11 2024-04-14T15:19:31.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.339+0000 7fe3a7da0700 1 -- 172.21.15.90:0/2072206776 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fe38401cdd0 con 0x7fe38401c860 2024-04-14T15:19:31.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.340+0000 7fe396ffd700 10 client.37395.objecter ms_handle_connect 0x7fe38401c860 2024-04-14T15:19:31.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.340+0000 7fe396ffd700 1 -- 172.21.15.90:0/2072206776 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fe38401cdd0 con 0x7fe38401c860 2024-04-14T15:19:31.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.340+0000 7fe396ffd700 10 client.37395.objecter ms_dispatch 0x7fe3a0001550 command_reply(tid 11: 0 ) 2024-04-14T15:19:31.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.340+0000 7fe396ffd700 10 client.37395.objecter _finish_command 11 = system:0 2024-04-14T15:19:31.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.340+0000 7fe396ffd700 20 client.37395.objecter put_session s=0x7fe38401c1e0 osd=5 4 2024-04-14T15:19:31.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.340+0000 7fe396ffd700 15 client.37395.objecter _session_command_op_remove 5 11 2024-04-14T15:19:31.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.357+0000 7fe3a7da0700 10 client.37395.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:19:31.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.357+0000 7fe3a7da0700 15 client.37395.objecter _session_command_op_assign -1 12 2024-04-14T15:19:31.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.357+0000 7fe3a7da0700 20 client.37395.objecter _get_session s=0x7fe38401c1e0 osd=5 3 2024-04-14T15:19:31.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.357+0000 7fe3a7da0700 20 client.37395.objecter put_session s=0x7fe38401c1e0 osd=5 3 2024-04-14T15:19:31.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.357+0000 7fe3a7da0700 20 client.37395.objecter _get_session s=0x7fe38401c1e0 osd=5 3 2024-04-14T15:19:31.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.357+0000 7fe3a7da0700 15 client.37395.objecter _session_command_op_remove -1 12 2024-04-14T15:19:31.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.357+0000 7fe3a7da0700 20 client.37395.objecter get_session s=0x7fe38401c1e0 osd=5 3 2024-04-14T15:19:31.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.357+0000 7fe3a7da0700 15 client.37395.objecter _session_command_op_assign 5 12 2024-04-14T15:19:31.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.357+0000 7fe3a7da0700 20 client.37395.objecter put_session s=0x7fe38401c1e0 osd=5 4 2024-04-14T15:19:31.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.357+0000 7fe3a7da0700 10 client.37395.objecter _send_command 12 2024-04-14T15:19:31.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.357+0000 7fe3a7da0700 1 -- 172.21.15.90:0/2072206776 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe38401d000 con 0x7fe38401c860 2024-04-14T15:19:31.359 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:19:31.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.359+0000 7fe396ffd700 1 -- 172.21.15.90:0/2072206776 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fe38401d000 con 0x7fe38401c860 2024-04-14T15:19:31.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.359+0000 7fe396ffd700 10 client.37395.objecter ms_dispatch 0x7fe3a0001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:19:31.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.359+0000 7fe396ffd700 10 client.37395.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:19:31.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.359+0000 7fe396ffd700 20 client.37395.objecter put_session s=0x7fe38401c1e0 osd=5 4 2024-04-14T15:19:31.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.359+0000 7fe396ffd700 15 client.37395.objecter _session_command_op_remove 5 12 2024-04-14T15:19:31.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.359+0000 7fe3a7da0700 10 client.37395.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:19:31.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.359+0000 7fe3a7da0700 15 client.37395.objecter _session_command_op_assign -1 13 2024-04-14T15:19:31.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.359+0000 7fe3a7da0700 20 client.37395.objecter _get_session s=0x7fe38401cdd0 osd=6 3 2024-04-14T15:19:31.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.359+0000 7fe3a7da0700 20 client.37395.objecter put_session s=0x7fe38401cdd0 osd=6 3 2024-04-14T15:19:31.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.359+0000 7fe3a7da0700 20 client.37395.objecter _get_session s=0x7fe38401cdd0 osd=6 3 2024-04-14T15:19:31.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.359+0000 7fe3a7da0700 15 client.37395.objecter _session_command_op_remove -1 13 2024-04-14T15:19:31.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.359+0000 7fe3a7da0700 20 client.37395.objecter get_session s=0x7fe38401cdd0 osd=6 3 2024-04-14T15:19:31.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.359+0000 7fe3a7da0700 15 client.37395.objecter _session_command_op_assign 6 13 2024-04-14T15:19:31.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.359+0000 7fe3a7da0700 20 client.37395.objecter put_session s=0x7fe38401cdd0 osd=6 4 2024-04-14T15:19:31.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.359+0000 7fe3a7da0700 10 client.37395.objecter _send_command 13 2024-04-14T15:19:31.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.359+0000 7fe3a7da0700 1 -- 172.21.15.90:0/2072206776 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fe3840233e0 con 0x7fe38401f8d0 2024-04-14T15:19:31.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.360+0000 7fe396ffd700 10 client.37395.objecter ms_handle_connect 0x7fe38401f8d0 2024-04-14T15:19:31.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.360+0000 7fe396ffd700 1 -- 172.21.15.90:0/2072206776 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fe3840233e0 con 0x7fe38401f8d0 2024-04-14T15:19:31.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.360+0000 7fe396ffd700 10 client.37395.objecter ms_dispatch 0x7fe3a0001550 command_reply(tid 13: 0 ) 2024-04-14T15:19:31.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.360+0000 7fe396ffd700 10 client.37395.objecter _finish_command 13 = system:0 2024-04-14T15:19:31.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.360+0000 7fe396ffd700 20 client.37395.objecter put_session s=0x7fe38401cdd0 osd=6 4 2024-04-14T15:19:31.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.360+0000 7fe396ffd700 15 client.37395.objecter _session_command_op_remove 6 13 2024-04-14T15:19:31.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.377+0000 7fe3a7da0700 10 client.37395.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:19:31.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.377+0000 7fe3a7da0700 15 client.37395.objecter _session_command_op_assign -1 14 2024-04-14T15:19:31.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.377+0000 7fe3a7da0700 20 client.37395.objecter _get_session s=0x7fe38401cdd0 osd=6 3 2024-04-14T15:19:31.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.377+0000 7fe3a7da0700 20 client.37395.objecter put_session s=0x7fe38401cdd0 osd=6 3 2024-04-14T15:19:31.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.377+0000 7fe3a7da0700 20 client.37395.objecter _get_session s=0x7fe38401cdd0 osd=6 3 2024-04-14T15:19:31.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.377+0000 7fe3a7da0700 15 client.37395.objecter _session_command_op_remove -1 14 2024-04-14T15:19:31.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.377+0000 7fe3a7da0700 20 client.37395.objecter get_session s=0x7fe38401cdd0 osd=6 3 2024-04-14T15:19:31.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.377+0000 7fe3a7da0700 15 client.37395.objecter _session_command_op_assign 6 14 2024-04-14T15:19:31.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.377+0000 7fe3a7da0700 20 client.37395.objecter put_session s=0x7fe38401cdd0 osd=6 4 2024-04-14T15:19:31.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.377+0000 7fe3a7da0700 10 client.37395.objecter _send_command 14 2024-04-14T15:19:31.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.377+0000 7fe3a7da0700 1 -- 172.21.15.90:0/2072206776 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe3840233e0 con 0x7fe38401f8d0 2024-04-14T15:19:31.379 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:19:31.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.379+0000 7fe396ffd700 1 -- 172.21.15.90:0/2072206776 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fe3840233e0 con 0x7fe38401f8d0 2024-04-14T15:19:31.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.379+0000 7fe396ffd700 10 client.37395.objecter ms_dispatch 0x7fe3a0001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:19:31.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.379+0000 7fe396ffd700 10 client.37395.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:19:31.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.379+0000 7fe396ffd700 20 client.37395.objecter put_session s=0x7fe38401cdd0 osd=6 4 2024-04-14T15:19:31.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.379+0000 7fe396ffd700 15 client.37395.objecter _session_command_op_remove 6 14 2024-04-14T15:19:31.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.379+0000 7fe3a7da0700 10 client.37395.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:19:31.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.379+0000 7fe3a7da0700 15 client.37395.objecter _session_command_op_assign -1 15 2024-04-14T15:19:31.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.379+0000 7fe3a7da0700 20 client.37395.objecter _get_session s=0x7fe3840233e0 osd=7 3 2024-04-14T15:19:31.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.379+0000 7fe3a7da0700 20 client.37395.objecter put_session s=0x7fe3840233e0 osd=7 3 2024-04-14T15:19:31.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.379+0000 7fe3a7da0700 20 client.37395.objecter _get_session s=0x7fe3840233e0 osd=7 3 2024-04-14T15:19:31.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.379+0000 7fe3a7da0700 15 client.37395.objecter _session_command_op_remove -1 15 2024-04-14T15:19:31.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.379+0000 7fe3a7da0700 20 client.37395.objecter get_session s=0x7fe3840233e0 osd=7 3 2024-04-14T15:19:31.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.379+0000 7fe3a7da0700 15 client.37395.objecter _session_command_op_assign 7 15 2024-04-14T15:19:31.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.379+0000 7fe3a7da0700 20 client.37395.objecter put_session s=0x7fe3840233e0 osd=7 4 2024-04-14T15:19:31.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.379+0000 7fe3a7da0700 10 client.37395.objecter _send_command 15 2024-04-14T15:19:31.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.379+0000 7fe3a7da0700 1 -- 172.21.15.90:0/2072206776 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fe384027740 con 0x7fe384023a60 2024-04-14T15:19:31.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.380+0000 7fe396ffd700 10 client.37395.objecter ms_handle_connect 0x7fe384023a60 2024-04-14T15:19:31.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.381+0000 7fe396ffd700 1 -- 172.21.15.90:0/2072206776 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fe384027740 con 0x7fe384023a60 2024-04-14T15:19:31.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.381+0000 7fe396ffd700 10 client.37395.objecter ms_dispatch 0x7fe3a0001550 command_reply(tid 15: 0 ) 2024-04-14T15:19:31.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.381+0000 7fe396ffd700 10 client.37395.objecter _finish_command 15 = system:0 2024-04-14T15:19:31.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.381+0000 7fe396ffd700 20 client.37395.objecter put_session s=0x7fe3840233e0 osd=7 4 2024-04-14T15:19:31.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.381+0000 7fe396ffd700 15 client.37395.objecter _session_command_op_remove 7 15 2024-04-14T15:19:31.386 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:19:31.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.397+0000 7fe3a7da0700 10 client.37395.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:19:31.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.397+0000 7fe3a7da0700 15 client.37395.objecter _session_command_op_assign -1 16 2024-04-14T15:19:31.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.397+0000 7fe3a7da0700 20 client.37395.objecter _get_session s=0x7fe3840233e0 osd=7 3 2024-04-14T15:19:31.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.397+0000 7fe3a7da0700 20 client.37395.objecter put_session s=0x7fe3840233e0 osd=7 3 2024-04-14T15:19:31.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.397+0000 7fe3a7da0700 20 client.37395.objecter _get_session s=0x7fe3840233e0 osd=7 3 2024-04-14T15:19:31.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.397+0000 7fe3a7da0700 15 client.37395.objecter _session_command_op_remove -1 16 2024-04-14T15:19:31.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.397+0000 7fe3a7da0700 20 client.37395.objecter get_session s=0x7fe3840233e0 osd=7 3 2024-04-14T15:19:31.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.397+0000 7fe3a7da0700 15 client.37395.objecter _session_command_op_assign 7 16 2024-04-14T15:19:31.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.397+0000 7fe3a7da0700 20 client.37395.objecter put_session s=0x7fe3840233e0 osd=7 4 2024-04-14T15:19:31.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.397+0000 7fe3a7da0700 10 client.37395.objecter _send_command 16 2024-04-14T15:19:31.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.397+0000 7fe3a7da0700 1 -- 172.21.15.90:0/2072206776 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe384027950 con 0x7fe384023a60 2024-04-14T15:19:31.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.400+0000 7fe396ffd700 1 -- 172.21.15.90:0/2072206776 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fe384027950 con 0x7fe384023a60 2024-04-14T15:19:31.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.400+0000 7fe396ffd700 10 client.37395.objecter ms_dispatch 0x7fe3a0001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:19:31.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.400+0000 7fe396ffd700 10 client.37395.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:19:31.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.400+0000 7fe396ffd700 20 client.37395.objecter put_session s=0x7fe3840233e0 osd=7 4 2024-04-14T15:19:31.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.400+0000 7fe396ffd700 15 client.37395.objecter _session_command_op_remove 7 16 2024-04-14T15:19:31.400 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:19:31.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.400+0000 7fe3a7da0700 10 librados: watch_flush enter 2024-04-14T15:19:31.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.400+0000 7fe3a7da0700 10 librados: watch_flush exit 2024-04-14T15:19:31.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.400+0000 7fe3a7da0700 10 client.37395.objecter close_session for osd.0 2024-04-14T15:19:31.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.400+0000 7fe3a7da0700 1 -- 172.21.15.90:0/2072206776 >> v1:172.21.15.90:6801/3761806885 conn(0x7fe384004730 legacy=0x7fe384002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:31.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.400+0000 7fe3a7da0700 20 client.37395.objecter put_session s=0x7fe3840053e0 osd=0 1 2024-04-14T15:19:31.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.400+0000 7fe3a7da0700 10 client.37395.objecter close_session for osd.1 2024-04-14T15:19:31.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.400+0000 7fe3a7da0700 1 -- 172.21.15.90:0/2072206776 >> v1:172.21.15.90:6805/4170361699 conn(0x7fe384004310 legacy=0x7fe38400ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:31.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.400+0000 7fe3a7da0700 20 client.37395.objecter put_session s=0x7fe384004130 osd=1 1 2024-04-14T15:19:31.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.400+0000 7fe3a7da0700 10 client.37395.objecter close_session for osd.2 2024-04-14T15:19:31.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.400+0000 7fe3a7da0700 1 -- 172.21.15.90:0/2072206776 >> v1:172.21.15.90:6809/1686184251 conn(0x7fe38400d5d0 legacy=0x7fe38400f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:31.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.400+0000 7fe3a7da0700 20 client.37395.objecter put_session s=0x7fe384001980 osd=2 1 2024-04-14T15:19:31.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.400+0000 7fe3a7da0700 10 client.37395.objecter close_session for osd.3 2024-04-14T15:19:31.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.400+0000 7fe3a7da0700 1 -- 172.21.15.90:0/2072206776 >> v1:172.21.15.90:6813/2270166774 conn(0x7fe3840124a0 legacy=0x7fe384014890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:31.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.400+0000 7fe3a7da0700 20 client.37395.objecter put_session s=0x7fe384008390 osd=3 1 2024-04-14T15:19:31.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.400+0000 7fe3a7da0700 10 client.37395.objecter close_session for osd.4 2024-04-14T15:19:31.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.400+0000 7fe3a7da0700 1 -- 172.21.15.90:0/2072206776 >> v1:172.21.15.92:6800/3887549733 conn(0x7fe384012000 legacy=0x7fe3840192f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:31.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.400+0000 7fe3a7da0700 20 client.37395.objecter put_session s=0x7fe38400c760 osd=4 1 2024-04-14T15:19:31.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.400+0000 7fe3a633d700 1 -- 172.21.15.90:0/2072206776 reap_dead start 2024-04-14T15:19:31.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.400+0000 7fe3a7da0700 10 client.37395.objecter close_session for osd.5 2024-04-14T15:19:31.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.400+0000 7fe3a7da0700 1 -- 172.21.15.90:0/2072206776 >> v1:172.21.15.92:6804/3244561440 conn(0x7fe38401c860 legacy=0x7fe38401aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:31.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.400+0000 7fe3a633d700 1 -- 172.21.15.90:0/2072206776 reap_dead start 2024-04-14T15:19:31.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.400+0000 7fe3a7da0700 20 client.37395.objecter put_session s=0x7fe38401c1e0 osd=5 1 2024-04-14T15:19:31.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.400+0000 7fe3a7da0700 10 client.37395.objecter close_session for osd.6 2024-04-14T15:19:31.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.400+0000 7fe3a7da0700 1 -- 172.21.15.90:0/2072206776 >> v1:172.21.15.92:6808/636125963 conn(0x7fe38401f8d0 legacy=0x7fe384021cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:31.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.400+0000 7fe3a7da0700 20 client.37395.objecter put_session s=0x7fe38401cdd0 osd=6 1 2024-04-14T15:19:31.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.400+0000 7fe3a7da0700 10 client.37395.objecter close_session for osd.7 2024-04-14T15:19:31.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.400+0000 7fe3a7da0700 1 -- 172.21.15.90:0/2072206776 >> v1:172.21.15.92:6812/1773706716 conn(0x7fe384023a60 legacy=0x7fe384025e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:31.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.400+0000 7fe3a7da0700 20 client.37395.objecter put_session s=0x7fe3840233e0 osd=7 1 2024-04-14T15:19:31.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.400+0000 7fe3a7da0700 20 client.37395.objecter shutdown clearing up homeless session... 2024-04-14T15:19:31.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.400+0000 7fe3a7da0700 10 client.37395.objecter successfully canceled tick 2024-04-14T15:19:31.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.400+0000 7fe3a633d700 1 -- 172.21.15.90:0/2072206776 reap_dead start 2024-04-14T15:19:31.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.400+0000 7fe3a7da0700 1 -- 172.21.15.90:0/2072206776 >> v1:172.21.15.90:6800/420935427 conn(0x7fe38c071230 legacy=0x7fe38c0736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:31.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.400+0000 7fe3a7da0700 1 -- 172.21.15.90:0/2072206776 >> v1:172.21.15.90:6789/0 conn(0x7fe3a010b720 legacy=0x7fe3a01a27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:31.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.400+0000 7fe3a7da0700 1 -- 172.21.15.90:0/2072206776 shutdown_connections 2024-04-14T15:19:31.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.400+0000 7fe3a7da0700 1 -- 172.21.15.90:0/2072206776 >> 172.21.15.90:0/2072206776 conn(0x7fe3a00fe3a0 msgr2=0x7fe3a010ad00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:19:31.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.401+0000 7fe3a7da0700 1 -- 172.21.15.90:0/2072206776 shutdown_connections 2024-04-14T15:19:31.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.401+0000 7fe3a7da0700 1 -- 172.21.15.90:0/2072206776 wait complete. 2024-04-14T15:19:31.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:31.401+0000 7fe3a7da0700 1 librados: shutdown 2024-04-14T15:19:31.404 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:19:31.405 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:19:31.405 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:19:31.405 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:19:31.405 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:19:31.405 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:19:31.405 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:19:31.405 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:19:31.409 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:31.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:31.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:19:31.511 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:31.538 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:31.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:31.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:31.640 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:19:31.668 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:31.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:31.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:19:31.770 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:19:31.797 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:31.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:31.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:19:31.899 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:31.926 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:31.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:31.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:32.028 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:32.055 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:32.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:32.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:32.156 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:19:32.183 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:32.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:32.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:19:32.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:31 smithi090 ceph-mon[41670]: osdmap e468: 8 total, 8 up, 8 in 2024-04-14T15:19:32.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:31 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2072206776' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:19:32.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:31 smithi090 ceph-mon[30399]: osdmap e468: 8 total, 8 up, 8 in 2024-04-14T15:19:32.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:31 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2072206776' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:19:32.285 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:19:32.298 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-14T15:19:32.311 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:32.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:32.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:19:32.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:31 smithi092 ceph-mon[33966]: osdmap e468: 8 total, 8 up, 8 in 2024-04-14T15:19:32.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:31 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2072206776' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:19:32.388 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-14T15:19:32.413 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:19:32.453 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:32.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:32.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:19:32.555 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:19:32.578 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:32.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:32.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:19:32.680 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:19:32.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:32.706 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:32.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:19:32.807 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:19:32.832 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:32.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:32.832 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:19:32.933 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:19:32.957 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:32.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:32.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:19:33.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.033+0000 7f09f75f5700 1 -- 172.21.15.90:0/4127335035 >> v1:172.21.15.90:6789/0 conn(0x7f09f0108880 legacy=0x7f09f0108c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:33.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.033+0000 7f09f75f5700 1 -- 172.21.15.90:0/4127335035 shutdown_connections 2024-04-14T15:19:33.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.033+0000 7f09f75f5700 1 -- 172.21.15.90:0/4127335035 >> 172.21.15.90:0/4127335035 conn(0x7f09f00fe390 msgr2=0x7f09f01007b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:19:33.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.034+0000 7f09f75f5700 1 -- 172.21.15.90:0/4127335035 shutdown_connections 2024-04-14T15:19:33.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.034+0000 7f09f75f5700 1 -- 172.21.15.90:0/4127335035 wait complete. 2024-04-14T15:19:33.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.034+0000 7f09f75f5700 1 librados: starting msgr at 2024-04-14T15:19:33.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.034+0000 7f09f75f5700 1 librados: starting objecter 2024-04-14T15:19:33.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.034+0000 7f09f75f5700 1 Processor -- start 2024-04-14T15:19:33.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.034+0000 7f09f75f5700 1 -- start start 2024-04-14T15:19:33.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.034+0000 7f09f75f5700 1 librados: setting wanted keys 2024-04-14T15:19:33.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.034+0000 7f09f75f5700 1 librados: calling monclient init 2024-04-14T15:19:33.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.034+0000 7f09f75f5700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f09f01a7a20 con 0x7f09f0108880 2024-04-14T15:19:33.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.034+0000 7f09f75f5700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f09f01a8c40 con 0x7f09f010f300 2024-04-14T15:19:33.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.034+0000 7f09f75f5700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f09f01a9e80 con 0x7f09f010b730 2024-04-14T15:19:33.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.035+0000 7f09f5391700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f09f0108880 0x7f09f010eb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:39044/0 (socket says 172.21.15.90:39044) 2024-04-14T15:19:33.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.035+0000 7f09f5391700 1 -- 172.21.15.90:0/3459529904 learned_addr learned my addr 172.21.15.90:0/3459529904 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:19:33.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.035+0000 7f09e67fc700 10 client.?.objecter ms_handle_connect 0x7f09f0108880 2024-04-14T15:19:33.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.035+0000 7f09e67fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:19:33.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.035+0000 7f09e67fc700 10 client.?.objecter ms_handle_connect 0x7f09f010b730 2024-04-14T15:19:33.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.035+0000 7f09e67fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:19:33.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.035+0000 7f09e67fc700 10 client.?.objecter ms_handle_connect 0x7f09f010f300 2024-04-14T15:19:33.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.035+0000 7f09e67fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:19:33.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.035+0000 7f09e67fc700 1 -- 172.21.15.90:0/3459529904 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4293129173 0 0) 0x7f09f01a7a20 con 0x7f09f0108880 2024-04-14T15:19:33.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.035+0000 7f09e67fc700 1 -- 172.21.15.90:0/3459529904 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f09dc0031f0 con 0x7f09f0108880 2024-04-14T15:19:33.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.035+0000 7f09e67fc700 1 -- 172.21.15.90:0/3459529904 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 918985131 0 0) 0x7f09f01a9e80 con 0x7f09f010b730 2024-04-14T15:19:33.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.035+0000 7f09e67fc700 1 -- 172.21.15.90:0/3459529904 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f09f01a7a20 con 0x7f09f010b730 2024-04-14T15:19:33.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.035+0000 7f09e67fc700 1 -- 172.21.15.90:0/3459529904 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2900977874 0 0) 0x7f09f01a8c40 con 0x7f09f010f300 2024-04-14T15:19:33.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.035+0000 7f09e67fc700 1 -- 172.21.15.90:0/3459529904 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f09f01a9e80 con 0x7f09f010f300 2024-04-14T15:19:33.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.035+0000 7f09e67fc700 1 -- 172.21.15.90:0/3459529904 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2134660393 0 0) 0x7f09f01a7a20 con 0x7f09f010b730 2024-04-14T15:19:33.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.035+0000 7f09e67fc700 1 -- 172.21.15.90:0/3459529904 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f09f01a8c40 con 0x7f09f010b730 2024-04-14T15:19:33.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.035+0000 7f09e67fc700 1 -- 172.21.15.90:0/3459529904 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f09e0001890 con 0x7f09f010b730 2024-04-14T15:19:33.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.035+0000 7f09e67fc700 1 -- 172.21.15.90:0/3459529904 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3258518809 0 0) 0x7f09dc0031f0 con 0x7f09f0108880 2024-04-14T15:19:33.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.035+0000 7f09e67fc700 1 -- 172.21.15.90:0/3459529904 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f09f01a7a20 con 0x7f09f0108880 2024-04-14T15:19:33.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.035+0000 7f09e67fc700 1 -- 172.21.15.90:0/3459529904 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f09ec004470 con 0x7f09f0108880 2024-04-14T15:19:33.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.035+0000 7f09e67fc700 1 -- 172.21.15.90:0/3459529904 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4212968873 0 0) 0x7f09f01a9e80 con 0x7f09f010f300 2024-04-14T15:19:33.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.035+0000 7f09e67fc700 1 -- 172.21.15.90:0/3459529904 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f09dc0031f0 con 0x7f09f010f300 2024-04-14T15:19:33.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.035+0000 7f09e67fc700 1 -- 172.21.15.90:0/3459529904 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f09e8001af0 con 0x7f09f010f300 2024-04-14T15:19:33.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.035+0000 7f09e67fc700 1 -- 172.21.15.90:0/3459529904 <== mon.2 v1:172.21.15.90:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1462808678 0 0) 0x7f09f01a8c40 con 0x7f09f010b730 2024-04-14T15:19:33.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.035+0000 7f09e67fc700 1 -- 172.21.15.90:0/3459529904 >> v1:172.21.15.92:6789/0 conn(0x7f09f010f300 legacy=0x7f09f01a60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:33.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.035+0000 7f09e67fc700 1 -- 172.21.15.90:0/3459529904 >> v1:172.21.15.90:6789/0 conn(0x7f09f0108880 legacy=0x7f09f010eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:33.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.035+0000 7f09e67fc700 1 -- 172.21.15.90:0/3459529904 --> v1:172.21.15.90:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09f01ab0c0 con 0x7f09f010b730 2024-04-14T15:19:33.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.035+0000 7f09f75f5700 1 -- 172.21.15.90:0/3459529904 --> v1:172.21.15.90:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f09f01aa0b0 con 0x7f09f010b730 2024-04-14T15:19:33.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.036+0000 7f09f75f5700 10 client.47354.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:19:33.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.036+0000 7f09f75f5700 1 -- 172.21.15.90:0/3459529904 --> v1:172.21.15.90:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f09f01aa570 con 0x7f09f010b730 2024-04-14T15:19:33.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.036+0000 7f09f75f5700 1 librados: init done 2024-04-14T15:19:33.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.036+0000 7f09e67fc700 1 -- 172.21.15.90:0/3459529904 <== mon.2 v1:172.21.15.90:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f09e0001c60 con 0x7f09f010b730 2024-04-14T15:19:33.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.036+0000 7f09e67fc700 1 -- 172.21.15.90:0/3459529904 <== mon.2 v1:172.21.15.90:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f09e0004af0 con 0x7f09f010b730 2024-04-14T15:19:33.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.037+0000 7f09f75f5700 1 -- 172.21.15.90:0/3459529904 --> v1:172.21.15.90:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09d4005120 con 0x7f09f010b730 2024-04-14T15:19:33.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.037+0000 7f09e67fc700 1 -- 172.21.15.90:0/3459529904 <== mon.2 v1:172.21.15.90:6790/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f09e0003930 con 0x7f09f010b730 2024-04-14T15:19:33.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.038+0000 7f09e67fc700 1 -- 172.21.15.90:0/3459529904 <== mon.2 v1:172.21.15.90:6790/0 8 ==== osd_map(469..469 src has 51..469) ==== 8160+0+0 (unknown 2833558374 0 0) 0x7f09e0089890 con 0x7f09f010b730 2024-04-14T15:19:33.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.038+0000 7f09e67fc700 10 client.47354.objecter ms_dispatch 0x7f09f0001550 osd_map(469..469 src has 51..469) 2024-04-14T15:19:33.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.038+0000 7f09e67fc700 3 client.47354.objecter handle_osd_map got epochs [469,469] > 0 2024-04-14T15:19:33.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.038+0000 7f09e67fc700 3 client.47354.objecter handle_osd_map decoding full epoch 469 2024-04-14T15:19:33.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.038+0000 7f09e67fc700 20 client.47354.objecter dump_active .. 0 homeless 2024-04-14T15:19:33.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.038+0000 7f09e67fc700 10 client.47354.objecter ms_handle_connect 0x7f09dc071330 2024-04-14T15:19:33.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.042+0000 7f09e67fc700 1 -- 172.21.15.90:0/3459529904 <== mon.2 v1:172.21.15.90:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f09e00562f0 con 0x7f09f010b730 2024-04-14T15:19:33.060 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:19:33.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:33.087 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:33.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:19:33.188 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:19:33.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:32 smithi090 ceph-mon[41670]: pgmap v666: 258 pgs: 258 active+clean; 8.4 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 383 B/s wr, 3 op/s 2024-04-14T15:19:33.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:32 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:19:33.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:32 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:19:33.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:32 smithi090 ceph-mon[30399]: pgmap v666: 258 pgs: 258 active+clean; 8.4 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 383 B/s wr, 3 op/s 2024-04-14T15:19:33.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:32 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:19:33.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:32 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:19:33.211 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:33.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:33.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:19:33.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.271+0000 7f09f75f5700 1 -- 172.21.15.90:0/3459529904 --> v1:172.21.15.90:6790/0 -- mon_command({"prefix": "osd pg-upmap-items", "pgid": "110.3", "id": [5, 7, 2, 1, 4, 3]} v 0) -- 0x7f09d4005410 con 0x7f09f010b730 2024-04-14T15:19:33.313 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:19:33.336 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:33.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:33.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:19:33.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:32 smithi092 ceph-mon[33966]: pgmap v666: 258 pgs: 258 active+clean; 8.4 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 383 B/s wr, 3 op/s 2024-04-14T15:19:33.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:32 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:19:33.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:32 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:19:33.438 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:19:33.461 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:33.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:33.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:19:33.563 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:19:33.587 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:19:33.588 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:33.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:33.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:19:33.689 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:19:33.716 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:33.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:33.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:19:33.817 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:19:33.843 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:33.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:33.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:19:33.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.935+0000 7f09e67fc700 1 -- 172.21.15.90:0/3459529904 <== mon.2 v1:172.21.15.90:6790/0 10 ==== mon_command_ack([{"prefix": "osd pg-upmap-items", "pgid": "110.3", "id": [5, 7, 2, 1, 4, 3]}]=0 set 110.3 pg_upmap_items mapping to [5->7,2->1,4->3] v470) ==== 161+0+0 (unknown 549563617 0 0) 0x7f09e0059fa0 con 0x7f09f010b730 2024-04-14T15:19:33.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.939+0000 7f09f75f5700 10 librados: watch_flush enter 2024-04-14T15:19:33.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.939+0000 7f09f75f5700 10 librados: watch_flush exit 2024-04-14T15:19:33.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.939+0000 7f09f75f5700 20 client.47354.objecter shutdown clearing up homeless session... 2024-04-14T15:19:33.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.939+0000 7f09f75f5700 10 client.47354.objecter successfully canceled tick 2024-04-14T15:19:33.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.940+0000 7f09f75f5700 1 -- 172.21.15.90:0/3459529904 >> v1:172.21.15.90:6800/420935427 conn(0x7f09dc071330 legacy=0x7f09dc0737f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:33.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.940+0000 7f09f75f5700 1 -- 172.21.15.90:0/3459529904 >> v1:172.21.15.90:6790/0 conn(0x7f09f010b730 legacy=0x7f09f01a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:33.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.940+0000 7f09f75f5700 1 -- 172.21.15.90:0/3459529904 shutdown_connections 2024-04-14T15:19:33.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.940+0000 7f09f75f5700 1 -- 172.21.15.90:0/3459529904 >> 172.21.15.90:0/3459529904 conn(0x7f09f00fe390 msgr2=0x7f09f0112660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:19:33.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.940+0000 7f09f75f5700 1 -- 172.21.15.90:0/3459529904 shutdown_connections 2024-04-14T15:19:33.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.940+0000 7f09f75f5700 1 -- 172.21.15.90:0/3459529904 wait complete. 2024-04-14T15:19:33.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:33.940+0000 7f09f75f5700 1 librados: shutdown 2024-04-14T15:19:33.941 INFO:teuthology.orchestra.run.smithi090.stderr:set 110.3 pg_upmap_items mapping to [5->7,2->1,4->3] 2024-04-14T15:19:33.944 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:19:33.970 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:33.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:33.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:19:34.071 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:34.098 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:34.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:34.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:34.192 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:19:33 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:19:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:19:34.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:33 smithi090 ceph-mon[41670]: osdmap e469: 8 total, 8 up, 8 in 2024-04-14T15:19:34.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:33 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3459529904' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "110.3", "id": [5, 7, 2, 1, 4, 3]}]: dispatch 2024-04-14T15:19:34.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:33 smithi090 ceph-mon[41670]: from='client.47354 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "110.3", "id": [5, 7, 2, 1, 4, 3]}]: dispatch 2024-04-14T15:19:34.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:33 smithi090 ceph-mon[30399]: osdmap e469: 8 total, 8 up, 8 in 2024-04-14T15:19:34.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:33 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3459529904' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "110.3", "id": [5, 7, 2, 1, 4, 3]}]: dispatch 2024-04-14T15:19:34.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:33 smithi090 ceph-mon[30399]: from='client.47354 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "110.3", "id": [5, 7, 2, 1, 4, 3]}]: dispatch 2024-04-14T15:19:34.200 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:19:34.226 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:34.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:34.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:19:34.328 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:19:34.355 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:34.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:34.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:19:34.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:33 smithi092 ceph-mon[33966]: osdmap e469: 8 total, 8 up, 8 in 2024-04-14T15:19:34.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:33 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3459529904' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "110.3", "id": [5, 7, 2, 1, 4, 3]}]: dispatch 2024-04-14T15:19:34.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:33 smithi092 ceph-mon[33966]: from='client.47354 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "110.3", "id": [5, 7, 2, 1, 4, 3]}]: dispatch 2024-04-14T15:19:34.458 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:34.485 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:34.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:34.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:34.587 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:19:34.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:34.612 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:34.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:19:34.714 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:19:34.741 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:34.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:34.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:19:34.843 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:19:34.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:34.871 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:34.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:19:34.972 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:19:34.998 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:34.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:34.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:19:35.100 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:19:35.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:35.126 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:35.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:19:35.227 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:19:35.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:35.252 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:35.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:19:35.343 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:34 smithi090 ceph-mon[41670]: pgmap v668: 258 pgs: 258 active+clean; 8.4 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 508 B/s wr, 3 op/s 2024-04-14T15:19:35.343 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:34 smithi090 ceph-mon[41670]: from='client.47354 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "110.3", "id": [5, 7, 2, 1, 4, 3]}]': finished 2024-04-14T15:19:35.343 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:34 smithi090 ceph-mon[41670]: osdmap e470: 8 total, 8 up, 8 in 2024-04-14T15:19:35.344 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:34 smithi090 ceph-mon[30399]: pgmap v668: 258 pgs: 258 active+clean; 8.4 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 508 B/s wr, 3 op/s 2024-04-14T15:19:35.344 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:34 smithi090 ceph-mon[30399]: from='client.47354 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "110.3", "id": [5, 7, 2, 1, 4, 3]}]': finished 2024-04-14T15:19:35.344 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:34 smithi090 ceph-mon[30399]: osdmap e470: 8 total, 8 up, 8 in 2024-04-14T15:19:35.353 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:19:35.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:34 smithi092 ceph-mon[33966]: pgmap v668: 258 pgs: 258 active+clean; 8.4 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 508 B/s wr, 3 op/s 2024-04-14T15:19:35.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:34 smithi092 ceph-mon[33966]: from='client.47354 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "110.3", "id": [5, 7, 2, 1, 4, 3]}]': finished 2024-04-14T15:19:35.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:34 smithi092 ceph-mon[33966]: osdmap e470: 8 total, 8 up, 8 in 2024-04-14T15:19:35.372 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:19:35 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:19:35.380 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:35.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:35.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:19:35.416 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:19:35.481 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:19:35.509 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:35.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:35.511 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:19:35.581 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-14T15:19:35.611 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:19:35.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:35.653 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:35.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:19:35.753 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:19:35.781 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:35.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:35.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:19:35.883 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:19:35.906 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:35.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:35.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:19:36.008 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:19:36.031 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:36.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:36.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:19:36.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.075+0000 7f65a6f6d700 1 -- 172.21.15.90:0/649510669 >> v1:172.21.15.90:6789/0 conn(0x7f65a01077a0 legacy=0x7f65a0107b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:36.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.075+0000 7f65a6f6d700 1 -- 172.21.15.90:0/649510669 shutdown_connections 2024-04-14T15:19:36.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.075+0000 7f65a6f6d700 1 -- 172.21.15.90:0/649510669 >> 172.21.15.90:0/649510669 conn(0x7f65a00fd2b0 msgr2=0x7f65a00ff6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:19:36.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.075+0000 7f65a6f6d700 1 -- 172.21.15.90:0/649510669 shutdown_connections 2024-04-14T15:19:36.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.075+0000 7f65a6f6d700 1 -- 172.21.15.90:0/649510669 wait complete. 2024-04-14T15:19:36.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.075+0000 7f65a6f6d700 1 librados: starting msgr at 2024-04-14T15:19:36.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.075+0000 7f65a6f6d700 1 librados: starting objecter 2024-04-14T15:19:36.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.075+0000 7f65a6f6d700 1 Processor -- start 2024-04-14T15:19:36.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.075+0000 7f65a6f6d700 1 -- start start 2024-04-14T15:19:36.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.075+0000 7f65a6f6d700 1 librados: setting wanted keys 2024-04-14T15:19:36.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.075+0000 7f65a6f6d700 1 librados: calling monclient init 2024-04-14T15:19:36.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.075+0000 7f65a6f6d700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f65a01a9cc0 con 0x7f65a010e220 2024-04-14T15:19:36.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.075+0000 7f65a6f6d700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f65a01aaee0 con 0x7f65a010a650 2024-04-14T15:19:36.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.075+0000 7f65a6f6d700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f65a01ac120 con 0x7f65a01077a0 2024-04-14T15:19:36.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.076+0000 7f65a4d09700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f65a01077a0 0x7f65a010da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:53152/0 (socket says 172.21.15.90:53152) 2024-04-14T15:19:36.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.076+0000 7f65a4d09700 1 -- 172.21.15.90:0/1686257089 learned_addr learned my addr 172.21.15.90:0/1686257089 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:19:36.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.076+0000 7f659dffb700 10 client.?.objecter ms_handle_connect 0x7f65a01077a0 2024-04-14T15:19:36.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.076+0000 7f659dffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:19:36.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.076+0000 7f659dffb700 10 client.?.objecter ms_handle_connect 0x7f65a010e220 2024-04-14T15:19:36.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.076+0000 7f659dffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:19:36.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.076+0000 7f659dffb700 1 -- 172.21.15.90:0/1686257089 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3750185245 0 0) 0x7f65a01ac120 con 0x7f65a01077a0 2024-04-14T15:19:36.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.076+0000 7f659dffb700 1 -- 172.21.15.90:0/1686257089 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f65900031f0 con 0x7f65a01077a0 2024-04-14T15:19:36.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.076+0000 7f659dffb700 10 client.?.objecter ms_handle_connect 0x7f65a010a650 2024-04-14T15:19:36.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.076+0000 7f659dffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:19:36.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.076+0000 7f659dffb700 1 -- 172.21.15.90:0/1686257089 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2660994232 0 0) 0x7f65a01a9cc0 con 0x7f65a010e220 2024-04-14T15:19:36.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.076+0000 7f659dffb700 1 -- 172.21.15.90:0/1686257089 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f65a01ac120 con 0x7f65a010e220 2024-04-14T15:19:36.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.076+0000 7f659dffb700 1 -- 172.21.15.90:0/1686257089 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3318764130 0 0) 0x7f65a01aaee0 con 0x7f65a010a650 2024-04-14T15:19:36.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.076+0000 7f659dffb700 1 -- 172.21.15.90:0/1686257089 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f65a01a9cc0 con 0x7f65a010a650 2024-04-14T15:19:36.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.076+0000 7f659dffb700 1 -- 172.21.15.90:0/1686257089 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3667132646 0 0) 0x7f65a01ac120 con 0x7f65a010e220 2024-04-14T15:19:36.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.076+0000 7f659dffb700 1 -- 172.21.15.90:0/1686257089 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f65a01aaee0 con 0x7f65a010e220 2024-04-14T15:19:36.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.076+0000 7f659dffb700 1 -- 172.21.15.90:0/1686257089 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f6598001b30 con 0x7f65a010e220 2024-04-14T15:19:36.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.076+0000 7f659dffb700 1 -- 172.21.15.90:0/1686257089 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3665020806 0 0) 0x7f65900031f0 con 0x7f65a01077a0 2024-04-14T15:19:36.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.076+0000 7f659dffb700 1 -- 172.21.15.90:0/1686257089 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f65a01ac120 con 0x7f65a01077a0 2024-04-14T15:19:36.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.076+0000 7f659dffb700 1 -- 172.21.15.90:0/1686257089 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f65940015d0 con 0x7f65a01077a0 2024-04-14T15:19:36.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.076+0000 7f659dffb700 1 -- 172.21.15.90:0/1686257089 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 239332574 0 0) 0x7f65a01a9cc0 con 0x7f65a010a650 2024-04-14T15:19:36.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.076+0000 7f659dffb700 1 -- 172.21.15.90:0/1686257089 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f65900031f0 con 0x7f65a010a650 2024-04-14T15:19:36.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.076+0000 7f659dffb700 1 -- 172.21.15.90:0/1686257089 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f658c0018d0 con 0x7f65a010a650 2024-04-14T15:19:36.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.076+0000 7f659dffb700 1 -- 172.21.15.90:0/1686257089 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2372394396 0 0) 0x7f65a01aaee0 con 0x7f65a010e220 2024-04-14T15:19:36.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.076+0000 7f659dffb700 1 -- 172.21.15.90:0/1686257089 >> v1:172.21.15.90:6790/0 conn(0x7f65a01077a0 legacy=0x7f65a010da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:36.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.076+0000 7f659dffb700 1 -- 172.21.15.90:0/1686257089 >> v1:172.21.15.92:6789/0 conn(0x7f65a010a650 legacy=0x7f65a01a4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:36.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.076+0000 7f659dffb700 1 -- 172.21.15.90:0/1686257089 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65a01ad360 con 0x7f65a010e220 2024-04-14T15:19:36.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.076+0000 7f65a6f6d700 1 -- 172.21.15.90:0/1686257089 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f65a01a9ef0 con 0x7f65a010e220 2024-04-14T15:19:36.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.077+0000 7f659dffb700 1 -- 172.21.15.90:0/1686257089 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f65980020a0 con 0x7f65a010e220 2024-04-14T15:19:36.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.077+0000 7f659dffb700 1 -- 172.21.15.90:0/1686257089 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f6598004ce0 con 0x7f65a010e220 2024-04-14T15:19:36.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.077+0000 7f65a6f6d700 10 client.37407.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:19:36.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.077+0000 7f65a6f6d700 1 -- 172.21.15.90:0/1686257089 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f65a01aa3b0 con 0x7f65a010e220 2024-04-14T15:19:36.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.077+0000 7f65a6f6d700 1 librados: init done 2024-04-14T15:19:36.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.078+0000 7f65a6f6d700 1 -- 172.21.15.90:0/1686257089 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65a0102ee0 con 0x7f65a010e220 2024-04-14T15:19:36.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.078+0000 7f659dffb700 1 -- 172.21.15.90:0/1686257089 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f659801b970 con 0x7f65a010e220 2024-04-14T15:19:36.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.078+0000 7f659dffb700 1 -- 172.21.15.90:0/1686257089 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(472..472 src has 51..472) ==== 8197+0+0 (unknown 1958073560 0 0) 0x7f6598089bd0 con 0x7f65a010e220 2024-04-14T15:19:36.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.078+0000 7f659dffb700 10 client.37407.objecter ms_dispatch 0x7f65a0001550 osd_map(472..472 src has 51..472) 2024-04-14T15:19:36.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.078+0000 7f659dffb700 3 client.37407.objecter handle_osd_map got epochs [472,472] > 0 2024-04-14T15:19:36.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.078+0000 7f659dffb700 3 client.37407.objecter handle_osd_map decoding full epoch 472 2024-04-14T15:19:36.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.078+0000 7f659dffb700 20 client.37407.objecter dump_active .. 0 homeless 2024-04-14T15:19:36.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.079+0000 7f659dffb700 10 client.37407.objecter ms_handle_connect 0x7f65900713e0 2024-04-14T15:19:36.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.084+0000 7f659dffb700 1 -- 172.21.15.90:0/1686257089 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f65980565e0 con 0x7f65a010e220 2024-04-14T15:19:36.133 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:19:36.156 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:36.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:36.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:19:36.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:35 smithi090 ceph-mon[41670]: osdmap e471: 8 total, 8 up, 8 in 2024-04-14T15:19:36.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:35 smithi090 ceph-mon[41670]: osdmap e472: 8 total, 8 up, 8 in 2024-04-14T15:19:36.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:35 smithi090 ceph-mon[30399]: osdmap e471: 8 total, 8 up, 8 in 2024-04-14T15:19:36.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:35 smithi090 ceph-mon[30399]: osdmap e472: 8 total, 8 up, 8 in 2024-04-14T15:19:36.258 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:19:36.281 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:36.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:36.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:19:36.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.300+0000 7f65a6f6d700 1 -- 172.21.15.90:0/1686257089 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7f65a01058d0 con 0x7f65a010e220 2024-04-14T15:19:36.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:35 smithi092 ceph-mon[33966]: osdmap e471: 8 total, 8 up, 8 in 2024-04-14T15:19:36.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:35 smithi092 ceph-mon[33966]: osdmap e472: 8 total, 8 up, 8 in 2024-04-14T15:19:36.383 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:19:36.406 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:36.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:36.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:19:36.508 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:36.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:36.537 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:36.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:36.638 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:19:36.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:36.662 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:36.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:19:36.763 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:19:36.790 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:36.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:36.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:19:36.892 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:19:36.916 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:36.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:36.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:19:36.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.948+0000 7f659dffb700 1 -- 172.21.15.90:0/1686257089 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v473) ==== 97+0+0 (unknown 2164390323 0 0) 0x7f659805a290 con 0x7f65a010e220 2024-04-14T15:19:36.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.951+0000 7f65a6f6d700 10 librados: watch_flush enter 2024-04-14T15:19:36.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.951+0000 7f65a6f6d700 10 librados: watch_flush exit 2024-04-14T15:19:36.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.951+0000 7f65a6f6d700 20 client.37407.objecter shutdown clearing up homeless session... 2024-04-14T15:19:36.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.951+0000 7f65a6f6d700 10 client.37407.objecter successfully canceled tick 2024-04-14T15:19:36.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.951+0000 7f65a6f6d700 1 -- 172.21.15.90:0/1686257089 >> v1:172.21.15.90:6800/420935427 conn(0x7f65900713e0 legacy=0x7f65900738a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:36.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.951+0000 7f65a6f6d700 1 -- 172.21.15.90:0/1686257089 >> v1:172.21.15.90:6789/0 conn(0x7f65a010e220 legacy=0x7f65a01a83c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:36.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.951+0000 7f65a6f6d700 1 -- 172.21.15.90:0/1686257089 shutdown_connections 2024-04-14T15:19:36.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.951+0000 7f65a6f6d700 1 -- 172.21.15.90:0/1686257089 >> 172.21.15.90:0/1686257089 conn(0x7f65a00fd2b0 msgr2=0x7f65a00ffe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:19:36.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.953+0000 7f65a6f6d700 1 -- 172.21.15.90:0/1686257089 shutdown_connections 2024-04-14T15:19:36.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.953+0000 7f65a6f6d700 1 -- 172.21.15.90:0/1686257089 wait complete. 2024-04-14T15:19:36.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:36.953+0000 7f65a6f6d700 1 librados: shutdown 2024-04-14T15:19:36.959 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is set 2024-04-14T15:19:37.018 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:19:37.043 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:37.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:37.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:19:37.145 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:19:37.169 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:37.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:37.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:19:37.271 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:19:37.296 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:37.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:37.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:19:37.310 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:36 smithi090 ceph-mon[41670]: pgmap v671: 258 pgs: 1 peering, 257 active+clean; 8.4 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:19:37.310 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:36 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:19:37.310 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:36 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:19:37.311 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:36 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:19:37.311 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:36 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-71"}]: dispatch 2024-04-14T15:19:37.311 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:36 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-71"}]: dispatch 2024-04-14T15:19:37.311 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:36 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1686257089' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:19:37.311 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:36 smithi090 ceph-mon[30399]: pgmap v671: 258 pgs: 1 peering, 257 active+clean; 8.4 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:19:37.311 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:36 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:19:37.311 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:36 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:19:37.311 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:36 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:19:37.311 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:36 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-71"}]: dispatch 2024-04-14T15:19:37.311 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:36 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-71"}]: dispatch 2024-04-14T15:19:37.311 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:36 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1686257089' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:19:37.317 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:19:37.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:36 smithi092 ceph-mon[33966]: pgmap v671: 258 pgs: 1 peering, 257 active+clean; 8.4 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:19:37.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:36 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:19:37.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:36 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:19:37.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:36 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:19:37.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:36 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-71"}]: dispatch 2024-04-14T15:19:37.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:36 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-71"}]: dispatch 2024-04-14T15:19:37.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:36 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1686257089' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:19:37.397 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:19:37.421 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:37.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:37.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:19:37.523 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:19:37.548 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:37.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:37.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:19:37.649 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:19:37.673 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:37.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:37.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:19:37.774 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:19:37.798 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:37.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:37.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:19:37.899 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:19:37.926 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:37.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:37.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:19:38.027 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:19:38.052 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:38.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:38.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:19:38.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:37 smithi090 ceph-mon[41670]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:19:38.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:37 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1686257089' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-14T15:19:38.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:37 smithi090 ceph-mon[41670]: osdmap e473: 8 total, 8 up, 8 in 2024-04-14T15:19:38.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:37 smithi090 ceph-mon[41670]: osdmap e474: 8 total, 8 up, 8 in 2024-04-14T15:19:38.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:37 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:19:38.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:37 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:19:38.056 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:37 smithi090 ceph-mon[30399]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:19:38.056 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:37 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1686257089' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-14T15:19:38.056 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:37 smithi090 ceph-mon[30399]: osdmap e473: 8 total, 8 up, 8 in 2024-04-14T15:19:38.057 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:37 smithi090 ceph-mon[30399]: osdmap e474: 8 total, 8 up, 8 in 2024-04-14T15:19:38.057 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:37 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:19:38.057 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:37 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:19:38.154 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:19:38.180 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:38.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:38.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:19:38.282 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:38.290 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-14T15:19:38.290 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:19:38.307 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:38.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:38.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:38.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:37 smithi092 ceph-mon[33966]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:19:38.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:37 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1686257089' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-14T15:19:38.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:37 smithi092 ceph-mon[33966]: osdmap e473: 8 total, 8 up, 8 in 2024-04-14T15:19:38.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:37 smithi092 ceph-mon[33966]: osdmap e474: 8 total, 8 up, 8 in 2024-04-14T15:19:38.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:37 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:19:38.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:37 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:19:38.410 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:19:38.434 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:38.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:38.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:19:38.536 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:19:38.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:38.561 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:38.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:19:38.662 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:19:38.687 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:38.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:38.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:19:38.788 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:19:38.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:38.813 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:38.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:19:38.914 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:19:38.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:38.938 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:38.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:19:39.039 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:19:39.064 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:39.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:39.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:19:39.166 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:19:39.191 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:39.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:38 smithi090 ceph-mon[41670]: pgmap v674: 226 pgs: 1 peering, 225 active+clean; 8.4 MiB data, 576 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:19:39.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:38 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:19:39.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:38 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:19:39.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:38 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:19:39.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:38 smithi090 ceph-mon[41670]: osdmap e475: 8 total, 8 up, 8 in 2024-04-14T15:19:39.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:38 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-14T15:19:39.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:38 smithi090 ceph-mon[30399]: pgmap v674: 226 pgs: 1 peering, 225 active+clean; 8.4 MiB data, 576 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:19:39.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:38 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:19:39.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:38 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:19:39.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:38 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:19:39.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:38 smithi090 ceph-mon[30399]: osdmap e475: 8 total, 8 up, 8 in 2024-04-14T15:19:39.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:38 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-14T15:19:39.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:39.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:19:39.294 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:39.318 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:39.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:39.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:39.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:38 smithi092 ceph-mon[33966]: pgmap v674: 226 pgs: 1 peering, 225 active+clean; 8.4 MiB data, 576 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:19:39.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:38 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:19:39.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:38 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:19:39.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:38 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:19:39.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:38 smithi092 ceph-mon[33966]: osdmap e475: 8 total, 8 up, 8 in 2024-04-14T15:19:39.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:38 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-14T15:19:39.420 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:19:39.443 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:39.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:39.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:19:39.545 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:19:39.569 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:39.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:39.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:19:39.606 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-14T15:19:39.670 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:19:39.697 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:39.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:39.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:19:39.702 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-14T15:19:39.702 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-14T15:19:39.703 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-04-14T15:19:39.703 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-14T15:19:39.703 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-14T15:19:39.703 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-04-14T15:19:39.799 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:19:39.822 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:39.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:39.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:19:39.831 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd unset noscrub 2024-04-14T15:19:39.924 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:19:39.947 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:39.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:39.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:19:40.049 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:19:40.075 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:40.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:40.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:19:40.177 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:19:40.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:39 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:19:40.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:39 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:19:40.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:39 smithi090 ceph-mon[41670]: osdmap e476: 8 total, 8 up, 8 in 2024-04-14T15:19:40.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:39 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:19:40.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:39 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:19:40.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:39 smithi090 ceph-mon[30399]: osdmap e476: 8 total, 8 up, 8 in 2024-04-14T15:19:40.200 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:40.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:40.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:19:40.302 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:19:40.328 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:40.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:40.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:19:40.344 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:19:40.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:39 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:19:40.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:39 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:19:40.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:39 smithi092 ceph-mon[33966]: osdmap e476: 8 total, 8 up, 8 in 2024-04-14T15:19:40.429 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:19:40.453 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:40.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:40.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:19:40.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.532+0000 7fd1ff213700 1 -- 172.21.15.90:0/4021733811 >> v1:172.21.15.92:6789/0 conn(0x7fd1f8107890 legacy=0x7fd1f8107c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:40.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.532+0000 7fd1ff213700 1 -- 172.21.15.90:0/4021733811 shutdown_connections 2024-04-14T15:19:40.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.532+0000 7fd1ff213700 1 -- 172.21.15.90:0/4021733811 >> 172.21.15.90:0/4021733811 conn(0x7fd1f80fd290 msgr2=0x7fd1f80ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:19:40.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.532+0000 7fd1ff213700 1 -- 172.21.15.90:0/4021733811 shutdown_connections 2024-04-14T15:19:40.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.532+0000 7fd1ff213700 1 -- 172.21.15.90:0/4021733811 wait complete. 2024-04-14T15:19:40.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.532+0000 7fd1ff213700 1 librados: starting msgr at 2024-04-14T15:19:40.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.532+0000 7fd1ff213700 1 librados: starting objecter 2024-04-14T15:19:40.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.532+0000 7fd1ff213700 1 Processor -- start 2024-04-14T15:19:40.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.532+0000 7fd1ff213700 1 -- start start 2024-04-14T15:19:40.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.532+0000 7fd1ff213700 1 librados: setting wanted keys 2024-04-14T15:19:40.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.532+0000 7fd1ff213700 1 librados: calling monclient init 2024-04-14T15:19:40.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.533+0000 7fd1ff213700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd1f81a7900 con 0x7fd1f8107890 2024-04-14T15:19:40.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.533+0000 7fd1ff213700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd1f81a8b20 con 0x7fd1f810a740 2024-04-14T15:19:40.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.533+0000 7fd1ff213700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd1f81a9d60 con 0x7fd1f810e310 2024-04-14T15:19:40.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.533+0000 7fd1fcfaf700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fd1f8107890 0x7fd1f810d9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:53354/0 (socket says 172.21.15.90:53354) 2024-04-14T15:19:40.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.533+0000 7fd1fd7b0700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7fd1f810e310 0x7fd1f81a6000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:35586/0 (socket says 172.21.15.90:35586) 2024-04-14T15:19:40.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.533+0000 7fd1fcfaf700 1 -- 172.21.15.90:0/3815000204 learned_addr learned my addr 172.21.15.90:0/3815000204 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:19:40.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.533+0000 7fd1f5ffb700 10 client.?.objecter ms_handle_connect 0x7fd1f8107890 2024-04-14T15:19:40.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.533+0000 7fd1f5ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:19:40.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.533+0000 7fd1f5ffb700 10 client.?.objecter ms_handle_connect 0x7fd1f810e310 2024-04-14T15:19:40.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.533+0000 7fd1f5ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:19:40.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.533+0000 7fd1f5ffb700 10 client.?.objecter ms_handle_connect 0x7fd1f810a740 2024-04-14T15:19:40.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.533+0000 7fd1f5ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:19:40.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.533+0000 7fd1f5ffb700 1 -- 172.21.15.90:0/3815000204 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4265896820 0 0) 0x7fd1f81a7900 con 0x7fd1f8107890 2024-04-14T15:19:40.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.533+0000 7fd1f5ffb700 1 -- 172.21.15.90:0/3815000204 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1e4003aa0 con 0x7fd1f8107890 2024-04-14T15:19:40.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.533+0000 7fd1f5ffb700 1 -- 172.21.15.90:0/3815000204 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 565540813 0 0) 0x7fd1f81a9d60 con 0x7fd1f810e310 2024-04-14T15:19:40.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.533+0000 7fd1f5ffb700 1 -- 172.21.15.90:0/3815000204 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1f81a7900 con 0x7fd1f810e310 2024-04-14T15:19:40.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.533+0000 7fd1f5ffb700 1 -- 172.21.15.90:0/3815000204 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2808347526 0 0) 0x7fd1f81a8b20 con 0x7fd1f810a740 2024-04-14T15:19:40.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.533+0000 7fd1f5ffb700 1 -- 172.21.15.90:0/3815000204 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1f81a9d60 con 0x7fd1f810a740 2024-04-14T15:19:40.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.534+0000 7fd1f5ffb700 1 -- 172.21.15.90:0/3815000204 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2293363604 0 0) 0x7fd1e4003aa0 con 0x7fd1f8107890 2024-04-14T15:19:40.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.534+0000 7fd1f5ffb700 1 -- 172.21.15.90:0/3815000204 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd1f81a8b20 con 0x7fd1f8107890 2024-04-14T15:19:40.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.534+0000 7fd1f5ffb700 1 -- 172.21.15.90:0/3815000204 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3390608770 0 0) 0x7fd1f81a7900 con 0x7fd1f810e310 2024-04-14T15:19:40.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.534+0000 7fd1f5ffb700 1 -- 172.21.15.90:0/3815000204 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd1e4003aa0 con 0x7fd1f810e310 2024-04-14T15:19:40.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.534+0000 7fd1f5ffb700 1 -- 172.21.15.90:0/3815000204 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fd1f0001b50 con 0x7fd1f8107890 2024-04-14T15:19:40.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.534+0000 7fd1f5ffb700 1 -- 172.21.15.90:0/3815000204 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fd1e0001870 con 0x7fd1f810e310 2024-04-14T15:19:40.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.534+0000 7fd1f5ffb700 1 -- 172.21.15.90:0/3815000204 <== mon.2 v1:172.21.15.90:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2413859717 0 0) 0x7fd1e4003aa0 con 0x7fd1f810e310 2024-04-14T15:19:40.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.534+0000 7fd1f5ffb700 1 -- 172.21.15.90:0/3815000204 >> v1:172.21.15.92:6789/0 conn(0x7fd1f810a740 legacy=0x7fd1f81a27d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:40.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.534+0000 7fd1f5ffb700 1 -- 172.21.15.90:0/3815000204 >> v1:172.21.15.90:6789/0 conn(0x7fd1f8107890 legacy=0x7fd1f810d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:40.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.534+0000 7fd1f5ffb700 1 -- 172.21.15.90:0/3815000204 --> v1:172.21.15.90:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1f81aafa0 con 0x7fd1f810e310 2024-04-14T15:19:40.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.534+0000 7fd1ff213700 1 -- 172.21.15.90:0/3815000204 --> v1:172.21.15.90:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd1f81a9f90 con 0x7fd1f810e310 2024-04-14T15:19:40.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.534+0000 7fd1f5ffb700 1 -- 172.21.15.90:0/3815000204 <== mon.2 v1:172.21.15.90:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fd1e0003b20 con 0x7fd1f810e310 2024-04-14T15:19:40.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.534+0000 7fd1f5ffb700 1 -- 172.21.15.90:0/3815000204 <== mon.2 v1:172.21.15.90:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fd1e0005b40 con 0x7fd1f810e310 2024-04-14T15:19:40.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.534+0000 7fd1ff213700 10 client.47366.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:19:40.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.534+0000 7fd1ff213700 1 -- 172.21.15.90:0/3815000204 --> v1:172.21.15.90:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fd1f81aa450 con 0x7fd1f810e310 2024-04-14T15:19:40.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.534+0000 7fd1ff213700 1 librados: init done 2024-04-14T15:19:40.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.535+0000 7fd1ff213700 1 -- 172.21.15.90:0/3815000204 --> v1:172.21.15.90:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fd1f80693d0 con 0x7fd1f810e310 2024-04-14T15:19:40.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.536+0000 7fd1f5ffb700 1 -- 172.21.15.90:0/3815000204 <== mon.2 v1:172.21.15.90:6790/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fd1e0003680 con 0x7fd1f810e310 2024-04-14T15:19:40.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.536+0000 7fd1f5ffb700 1 -- 172.21.15.90:0/3815000204 <== mon.2 v1:172.21.15.90:6790/0 8 ==== osd_map(476..476 src has 51..476) ==== 8197+0+0 (unknown 1168989332 0 0) 0x7fd1e008aa60 con 0x7fd1f810e310 2024-04-14T15:19:40.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.536+0000 7fd1f5ffb700 10 client.47366.objecter ms_dispatch 0x7fd1f8001550 osd_map(476..476 src has 51..476) 2024-04-14T15:19:40.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.536+0000 7fd1f5ffb700 3 client.47366.objecter handle_osd_map got epochs [476,476] > 0 2024-04-14T15:19:40.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.536+0000 7fd1f5ffb700 3 client.47366.objecter handle_osd_map decoding full epoch 476 2024-04-14T15:19:40.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.536+0000 7fd1f5ffb700 20 client.47366.objecter dump_active .. 0 homeless 2024-04-14T15:19:40.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.536+0000 7fd1f5ffb700 1 -- 172.21.15.90:0/3815000204 <== mon.2 v1:172.21.15.90:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v476) ==== 54+0+16 (unknown 3066237151 0 2420000192) 0x7fd1e008ae50 con 0x7fd1f810e310 2024-04-14T15:19:40.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.537+0000 7fd1f5ffb700 10 client.47366.objecter ms_handle_connect 0x7fd1e4075540 2024-04-14T15:19:40.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.537+0000 7fd1ff213700 10 client.47366.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:19:40.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.537+0000 7fd1ff213700 15 client.47366.objecter _session_command_op_assign -1 1 2024-04-14T15:19:40.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.537+0000 7fd1ff213700 20 client.47366.objecter _get_session s=0x7fd1f81122e0 osd=0 3 2024-04-14T15:19:40.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.537+0000 7fd1ff213700 20 client.47366.objecter put_session s=0x7fd1f81122e0 osd=0 3 2024-04-14T15:19:40.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.537+0000 7fd1ff213700 20 client.47366.objecter _get_session s=0x7fd1f81122e0 osd=0 3 2024-04-14T15:19:40.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.537+0000 7fd1ff213700 15 client.47366.objecter _session_command_op_remove -1 1 2024-04-14T15:19:40.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.537+0000 7fd1ff213700 20 client.47366.objecter get_session s=0x7fd1f81122e0 osd=0 3 2024-04-14T15:19:40.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.537+0000 7fd1ff213700 15 client.47366.objecter _session_command_op_assign 0 1 2024-04-14T15:19:40.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.537+0000 7fd1ff213700 20 client.47366.objecter put_session s=0x7fd1f81122e0 osd=0 4 2024-04-14T15:19:40.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.537+0000 7fd1ff213700 10 client.47366.objecter _send_command 1 2024-04-14T15:19:40.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.537+0000 7fd1ff213700 1 -- 172.21.15.90:0/3815000204 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd1f80ff680 con 0x7fd1f81a8cf0 2024-04-14T15:19:40.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.537+0000 7fd1f5ffb700 10 client.47366.objecter ms_handle_connect 0x7fd1f81a8cf0 2024-04-14T15:19:40.540 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:19:40.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.538+0000 7fd1f5ffb700 1 -- 172.21.15.90:0/3815000204 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd1f80ff680 con 0x7fd1f81a8cf0 2024-04-14T15:19:40.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.538+0000 7fd1f5ffb700 10 client.47366.objecter ms_dispatch 0x7fd1f8001550 command_reply(tid 1: 0 ) 2024-04-14T15:19:40.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.538+0000 7fd1f5ffb700 10 client.47366.objecter _finish_command 1 = system:0 2024-04-14T15:19:40.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.538+0000 7fd1f5ffb700 20 client.47366.objecter put_session s=0x7fd1f81122e0 osd=0 4 2024-04-14T15:19:40.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.538+0000 7fd1f5ffb700 15 client.47366.objecter _session_command_op_remove 0 1 2024-04-14T15:19:40.555 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:19:40.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.559+0000 7fd1ff213700 10 client.47366.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:19:40.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.559+0000 7fd1ff213700 15 client.47366.objecter _session_command_op_assign -1 2 2024-04-14T15:19:40.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.559+0000 7fd1ff213700 20 client.47366.objecter _get_session s=0x7fd1f81122e0 osd=0 3 2024-04-14T15:19:40.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.559+0000 7fd1ff213700 20 client.47366.objecter put_session s=0x7fd1f81122e0 osd=0 3 2024-04-14T15:19:40.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.559+0000 7fd1ff213700 20 client.47366.objecter _get_session s=0x7fd1f81122e0 osd=0 3 2024-04-14T15:19:40.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.559+0000 7fd1ff213700 15 client.47366.objecter _session_command_op_remove -1 2 2024-04-14T15:19:40.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.559+0000 7fd1ff213700 20 client.47366.objecter get_session s=0x7fd1f81122e0 osd=0 3 2024-04-14T15:19:40.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.559+0000 7fd1ff213700 15 client.47366.objecter _session_command_op_assign 0 2 2024-04-14T15:19:40.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.559+0000 7fd1ff213700 20 client.47366.objecter put_session s=0x7fd1f81122e0 osd=0 4 2024-04-14T15:19:40.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.559+0000 7fd1ff213700 10 client.47366.objecter _send_command 2 2024-04-14T15:19:40.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.559+0000 7fd1ff213700 1 -- 172.21.15.90:0/3815000204 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fd1f810d050 con 0x7fd1f81a8cf0 2024-04-14T15:19:40.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.562+0000 7fd1f5ffb700 1 -- 172.21.15.90:0/3815000204 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd1f810d050 con 0x7fd1f81a8cf0 2024-04-14T15:19:40.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.562+0000 7fd1f5ffb700 10 client.47366.objecter ms_dispatch 0x7fd1f8001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:19:40.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.562+0000 7fd1f5ffb700 10 client.47366.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:19:40.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.562+0000 7fd1f5ffb700 20 client.47366.objecter put_session s=0x7fd1f81122e0 osd=0 4 2024-04-14T15:19:40.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.562+0000 7fd1f5ffb700 15 client.47366.objecter _session_command_op_remove 0 2 2024-04-14T15:19:40.563 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:19:40.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.562+0000 7fd1ff213700 10 client.47366.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:19:40.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.562+0000 7fd1ff213700 15 client.47366.objecter _session_command_op_assign -1 3 2024-04-14T15:19:40.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.562+0000 7fd1ff213700 20 client.47366.objecter _get_session s=0x7fd1f81a97e0 osd=1 3 2024-04-14T15:19:40.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.562+0000 7fd1ff213700 20 client.47366.objecter put_session s=0x7fd1f81a97e0 osd=1 3 2024-04-14T15:19:40.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.562+0000 7fd1ff213700 20 client.47366.objecter _get_session s=0x7fd1f81a97e0 osd=1 3 2024-04-14T15:19:40.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.562+0000 7fd1ff213700 15 client.47366.objecter _session_command_op_remove -1 3 2024-04-14T15:19:40.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.562+0000 7fd1ff213700 20 client.47366.objecter get_session s=0x7fd1f81a97e0 osd=1 3 2024-04-14T15:19:40.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.562+0000 7fd1ff213700 15 client.47366.objecter _session_command_op_assign 1 3 2024-04-14T15:19:40.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.562+0000 7fd1ff213700 20 client.47366.objecter put_session s=0x7fd1f81a97e0 osd=1 4 2024-04-14T15:19:40.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.562+0000 7fd1ff213700 10 client.47366.objecter _send_command 3 2024-04-14T15:19:40.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.562+0000 7fd1ff213700 1 -- 172.21.15.90:0/3815000204 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fd1f81a9950 con 0x7fd1f810ec90 2024-04-14T15:19:40.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.563+0000 7fd1f5ffb700 10 client.47366.objecter ms_handle_connect 0x7fd1f810ec90 2024-04-14T15:19:40.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.564+0000 7fd1f5ffb700 1 -- 172.21.15.90:0/3815000204 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd1f81a9950 con 0x7fd1f810ec90 2024-04-14T15:19:40.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.564+0000 7fd1f5ffb700 10 client.47366.objecter ms_dispatch 0x7fd1f8001550 command_reply(tid 3: 0 ) 2024-04-14T15:19:40.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.564+0000 7fd1f5ffb700 10 client.47366.objecter _finish_command 3 = system:0 2024-04-14T15:19:40.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.564+0000 7fd1f5ffb700 20 client.47366.objecter put_session s=0x7fd1f81a97e0 osd=1 4 2024-04-14T15:19:40.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.564+0000 7fd1f5ffb700 15 client.47366.objecter _session_command_op_remove 1 3 2024-04-14T15:19:40.581 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:40.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.581+0000 7fd1ff213700 10 client.47366.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:19:40.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.581+0000 7fd1ff213700 15 client.47366.objecter _session_command_op_assign -1 4 2024-04-14T15:19:40.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.581+0000 7fd1ff213700 20 client.47366.objecter _get_session s=0x7fd1f81a97e0 osd=1 3 2024-04-14T15:19:40.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.581+0000 7fd1ff213700 20 client.47366.objecter put_session s=0x7fd1f81a97e0 osd=1 3 2024-04-14T15:19:40.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.581+0000 7fd1ff213700 20 client.47366.objecter _get_session s=0x7fd1f81a97e0 osd=1 3 2024-04-14T15:19:40.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.581+0000 7fd1ff213700 15 client.47366.objecter _session_command_op_remove -1 4 2024-04-14T15:19:40.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.581+0000 7fd1ff213700 20 client.47366.objecter get_session s=0x7fd1f81a97e0 osd=1 3 2024-04-14T15:19:40.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.581+0000 7fd1ff213700 15 client.47366.objecter _session_command_op_assign 1 4 2024-04-14T15:19:40.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.581+0000 7fd1ff213700 20 client.47366.objecter put_session s=0x7fd1f81a97e0 osd=1 4 2024-04-14T15:19:40.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.581+0000 7fd1ff213700 10 client.47366.objecter _send_command 4 2024-04-14T15:19:40.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.581+0000 7fd1ff213700 1 -- 172.21.15.90:0/3815000204 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fd1f81a9b30 con 0x7fd1f810ec90 2024-04-14T15:19:40.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:40.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:19:40.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.582+0000 7fd1f5ffb700 1 -- 172.21.15.90:0/3815000204 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd1f81a9b30 con 0x7fd1f810ec90 2024-04-14T15:19:40.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.582+0000 7fd1f5ffb700 10 client.47366.objecter ms_dispatch 0x7fd1f8001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:19:40.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.582+0000 7fd1f5ffb700 10 client.47366.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:19:40.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.582+0000 7fd1f5ffb700 20 client.47366.objecter put_session s=0x7fd1f81a97e0 osd=1 4 2024-04-14T15:19:40.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.582+0000 7fd1f5ffb700 15 client.47366.objecter _session_command_op_remove 1 4 2024-04-14T15:19:40.584 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:19:40.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.583+0000 7fd1ff213700 10 client.47366.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:19:40.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.583+0000 7fd1ff213700 15 client.47366.objecter _session_command_op_assign -1 5 2024-04-14T15:19:40.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.583+0000 7fd1ff213700 20 client.47366.objecter _get_session s=0x7fd1f802e2d0 osd=2 3 2024-04-14T15:19:40.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.583+0000 7fd1ff213700 20 client.47366.objecter put_session s=0x7fd1f802e2d0 osd=2 3 2024-04-14T15:19:40.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.583+0000 7fd1ff213700 20 client.47366.objecter _get_session s=0x7fd1f802e2d0 osd=2 3 2024-04-14T15:19:40.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.583+0000 7fd1ff213700 15 client.47366.objecter _session_command_op_remove -1 5 2024-04-14T15:19:40.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.583+0000 7fd1ff213700 20 client.47366.objecter get_session s=0x7fd1f802e2d0 osd=2 3 2024-04-14T15:19:40.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.583+0000 7fd1ff213700 15 client.47366.objecter _session_command_op_assign 2 5 2024-04-14T15:19:40.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.583+0000 7fd1ff213700 20 client.47366.objecter put_session s=0x7fd1f802e2d0 osd=2 4 2024-04-14T15:19:40.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.583+0000 7fd1ff213700 10 client.47366.objecter _send_command 5 2024-04-14T15:19:40.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.583+0000 7fd1ff213700 1 -- 172.21.15.90:0/3815000204 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fd1f81b1260 con 0x7fd1f81ad750 2024-04-14T15:19:40.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.583+0000 7fd1f5ffb700 10 client.47366.objecter ms_handle_connect 0x7fd1f81ad750 2024-04-14T15:19:40.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.584+0000 7fd1f5ffb700 1 -- 172.21.15.90:0/3815000204 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd1f81b1260 con 0x7fd1f81ad750 2024-04-14T15:19:40.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.584+0000 7fd1f5ffb700 10 client.47366.objecter ms_dispatch 0x7fd1f8001550 command_reply(tid 5: 0 ) 2024-04-14T15:19:40.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.584+0000 7fd1f5ffb700 10 client.47366.objecter _finish_command 5 = system:0 2024-04-14T15:19:40.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.584+0000 7fd1f5ffb700 20 client.47366.objecter put_session s=0x7fd1f802e2d0 osd=2 4 2024-04-14T15:19:40.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.584+0000 7fd1f5ffb700 15 client.47366.objecter _session_command_op_remove 2 5 2024-04-14T15:19:40.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.600+0000 7fd1ff213700 10 client.47366.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:19:40.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.600+0000 7fd1ff213700 15 client.47366.objecter _session_command_op_assign -1 6 2024-04-14T15:19:40.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.600+0000 7fd1ff213700 20 client.47366.objecter _get_session s=0x7fd1f802e2d0 osd=2 3 2024-04-14T15:19:40.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.600+0000 7fd1ff213700 20 client.47366.objecter put_session s=0x7fd1f802e2d0 osd=2 3 2024-04-14T15:19:40.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.600+0000 7fd1ff213700 20 client.47366.objecter _get_session s=0x7fd1f802e2d0 osd=2 3 2024-04-14T15:19:40.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.600+0000 7fd1ff213700 15 client.47366.objecter _session_command_op_remove -1 6 2024-04-14T15:19:40.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.600+0000 7fd1ff213700 20 client.47366.objecter get_session s=0x7fd1f802e2d0 osd=2 3 2024-04-14T15:19:40.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.600+0000 7fd1ff213700 15 client.47366.objecter _session_command_op_assign 2 6 2024-04-14T15:19:40.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.600+0000 7fd1ff213700 20 client.47366.objecter put_session s=0x7fd1f802e2d0 osd=2 4 2024-04-14T15:19:40.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.600+0000 7fd1ff213700 10 client.47366.objecter _send_command 6 2024-04-14T15:19:40.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.600+0000 7fd1ff213700 1 -- 172.21.15.90:0/3815000204 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fd1f8111250 con 0x7fd1f81ad750 2024-04-14T15:19:40.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.602+0000 7fd1f5ffb700 1 -- 172.21.15.90:0/3815000204 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd1f8111250 con 0x7fd1f81ad750 2024-04-14T15:19:40.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.602+0000 7fd1f5ffb700 10 client.47366.objecter ms_dispatch 0x7fd1f8001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:19:40.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.602+0000 7fd1f5ffb700 10 client.47366.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:19:40.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.602+0000 7fd1f5ffb700 20 client.47366.objecter put_session s=0x7fd1f802e2d0 osd=2 4 2024-04-14T15:19:40.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.602+0000 7fd1f5ffb700 15 client.47366.objecter _session_command_op_remove 2 6 2024-04-14T15:19:40.603 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:19:40.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.602+0000 7fd1ff213700 10 client.47366.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:19:40.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.602+0000 7fd1ff213700 15 client.47366.objecter _session_command_op_assign -1 7 2024-04-14T15:19:40.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.602+0000 7fd1ff213700 20 client.47366.objecter _get_session s=0x7fd1f80636c0 osd=3 3 2024-04-14T15:19:40.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.602+0000 7fd1ff213700 20 client.47366.objecter put_session s=0x7fd1f80636c0 osd=3 3 2024-04-14T15:19:40.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.602+0000 7fd1ff213700 20 client.47366.objecter _get_session s=0x7fd1f80636c0 osd=3 3 2024-04-14T15:19:40.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.602+0000 7fd1ff213700 15 client.47366.objecter _session_command_op_remove -1 7 2024-04-14T15:19:40.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.602+0000 7fd1ff213700 20 client.47366.objecter get_session s=0x7fd1f80636c0 osd=3 3 2024-04-14T15:19:40.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.602+0000 7fd1ff213700 15 client.47366.objecter _session_command_op_assign 3 7 2024-04-14T15:19:40.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.602+0000 7fd1ff213700 20 client.47366.objecter put_session s=0x7fd1f80636c0 osd=3 4 2024-04-14T15:19:40.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.602+0000 7fd1ff213700 10 client.47366.objecter _send_command 7 2024-04-14T15:19:40.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.602+0000 7fd1ff213700 1 -- 172.21.15.90:0/3815000204 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fd1f8063ea0 con 0x7fd1f81b2590 2024-04-14T15:19:40.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.602+0000 7fd1f5ffb700 10 client.47366.objecter ms_handle_connect 0x7fd1f81b2590 2024-04-14T15:19:40.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.603+0000 7fd1f5ffb700 1 -- 172.21.15.90:0/3815000204 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd1f8063ea0 con 0x7fd1f81b2590 2024-04-14T15:19:40.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.603+0000 7fd1f5ffb700 10 client.47366.objecter ms_dispatch 0x7fd1f8001550 command_reply(tid 7: 0 ) 2024-04-14T15:19:40.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.603+0000 7fd1f5ffb700 10 client.47366.objecter _finish_command 7 = system:0 2024-04-14T15:19:40.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.603+0000 7fd1f5ffb700 20 client.47366.objecter put_session s=0x7fd1f80636c0 osd=3 4 2024-04-14T15:19:40.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.603+0000 7fd1f5ffb700 15 client.47366.objecter _session_command_op_remove 3 7 2024-04-14T15:19:40.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.620+0000 7fd1ff213700 10 client.47366.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:19:40.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.620+0000 7fd1ff213700 15 client.47366.objecter _session_command_op_assign -1 8 2024-04-14T15:19:40.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.620+0000 7fd1ff213700 20 client.47366.objecter _get_session s=0x7fd1f80636c0 osd=3 3 2024-04-14T15:19:40.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.620+0000 7fd1ff213700 20 client.47366.objecter put_session s=0x7fd1f80636c0 osd=3 3 2024-04-14T15:19:40.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.620+0000 7fd1ff213700 20 client.47366.objecter _get_session s=0x7fd1f80636c0 osd=3 3 2024-04-14T15:19:40.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.620+0000 7fd1ff213700 15 client.47366.objecter _session_command_op_remove -1 8 2024-04-14T15:19:40.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.620+0000 7fd1ff213700 20 client.47366.objecter get_session s=0x7fd1f80636c0 osd=3 3 2024-04-14T15:19:40.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.620+0000 7fd1ff213700 15 client.47366.objecter _session_command_op_assign 3 8 2024-04-14T15:19:40.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.620+0000 7fd1ff213700 20 client.47366.objecter put_session s=0x7fd1f80636c0 osd=3 4 2024-04-14T15:19:40.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.620+0000 7fd1ff213700 10 client.47366.objecter _send_command 8 2024-04-14T15:19:40.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.620+0000 7fd1ff213700 1 -- 172.21.15.90:0/3815000204 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fd1f8063200 con 0x7fd1f81b2590 2024-04-14T15:19:40.621 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:19:40.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.621+0000 7fd1f5ffb700 1 -- 172.21.15.90:0/3815000204 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd1f8063200 con 0x7fd1f81b2590 2024-04-14T15:19:40.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.621+0000 7fd1f5ffb700 10 client.47366.objecter ms_dispatch 0x7fd1f8001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:19:40.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.621+0000 7fd1f5ffb700 10 client.47366.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:19:40.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.621+0000 7fd1f5ffb700 20 client.47366.objecter put_session s=0x7fd1f80636c0 osd=3 4 2024-04-14T15:19:40.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.621+0000 7fd1f5ffb700 15 client.47366.objecter _session_command_op_remove 3 8 2024-04-14T15:19:40.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.621+0000 7fd1ff213700 10 client.47366.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:19:40.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.621+0000 7fd1ff213700 15 client.47366.objecter _session_command_op_assign -1 9 2024-04-14T15:19:40.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.621+0000 7fd1ff213700 20 client.47366.objecter _get_session s=0x7fd1f8063200 osd=4 3 2024-04-14T15:19:40.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.621+0000 7fd1ff213700 20 client.47366.objecter put_session s=0x7fd1f8063200 osd=4 3 2024-04-14T15:19:40.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.621+0000 7fd1ff213700 20 client.47366.objecter _get_session s=0x7fd1f8063200 osd=4 3 2024-04-14T15:19:40.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.621+0000 7fd1ff213700 15 client.47366.objecter _session_command_op_remove -1 9 2024-04-14T15:19:40.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.621+0000 7fd1ff213700 20 client.47366.objecter get_session s=0x7fd1f8063200 osd=4 3 2024-04-14T15:19:40.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.621+0000 7fd1ff213700 15 client.47366.objecter _session_command_op_assign 4 9 2024-04-14T15:19:40.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.621+0000 7fd1ff213700 20 client.47366.objecter put_session s=0x7fd1f8063200 osd=4 4 2024-04-14T15:19:40.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.621+0000 7fd1ff213700 10 client.47366.objecter _send_command 9 2024-04-14T15:19:40.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.621+0000 7fd1ff213700 1 -- 172.21.15.90:0/3815000204 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fd1f81baa80 con 0x7fd1f81b20f0 2024-04-14T15:19:40.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.622+0000 7fd1f5ffb700 10 client.47366.objecter ms_handle_connect 0x7fd1f81b20f0 2024-04-14T15:19:40.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.623+0000 7fd1f5ffb700 1 -- 172.21.15.90:0/3815000204 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd1f81baa80 con 0x7fd1f81b20f0 2024-04-14T15:19:40.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.623+0000 7fd1f5ffb700 10 client.47366.objecter ms_dispatch 0x7fd1f8001550 command_reply(tid 9: 0 ) 2024-04-14T15:19:40.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.623+0000 7fd1f5ffb700 10 client.47366.objecter _finish_command 9 = system:0 2024-04-14T15:19:40.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.623+0000 7fd1f5ffb700 20 client.47366.objecter put_session s=0x7fd1f8063200 osd=4 4 2024-04-14T15:19:40.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.623+0000 7fd1f5ffb700 15 client.47366.objecter _session_command_op_remove 4 9 2024-04-14T15:19:40.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.639+0000 7fd1ff213700 10 client.47366.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:19:40.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.639+0000 7fd1ff213700 15 client.47366.objecter _session_command_op_assign -1 10 2024-04-14T15:19:40.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.639+0000 7fd1ff213700 20 client.47366.objecter _get_session s=0x7fd1f8063200 osd=4 3 2024-04-14T15:19:40.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.639+0000 7fd1ff213700 20 client.47366.objecter put_session s=0x7fd1f8063200 osd=4 3 2024-04-14T15:19:40.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.639+0000 7fd1ff213700 20 client.47366.objecter _get_session s=0x7fd1f8063200 osd=4 3 2024-04-14T15:19:40.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.639+0000 7fd1ff213700 15 client.47366.objecter _session_command_op_remove -1 10 2024-04-14T15:19:40.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.639+0000 7fd1ff213700 20 client.47366.objecter get_session s=0x7fd1f8063200 osd=4 3 2024-04-14T15:19:40.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.639+0000 7fd1ff213700 15 client.47366.objecter _session_command_op_assign 4 10 2024-04-14T15:19:40.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.639+0000 7fd1ff213700 20 client.47366.objecter put_session s=0x7fd1f8063200 osd=4 4 2024-04-14T15:19:40.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.639+0000 7fd1ff213700 10 client.47366.objecter _send_command 10 2024-04-14T15:19:40.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.639+0000 7fd1ff213700 1 -- 172.21.15.90:0/3815000204 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fd1f81baa80 con 0x7fd1f81b20f0 2024-04-14T15:19:40.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.642+0000 7fd1f5ffb700 1 -- 172.21.15.90:0/3815000204 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd1f81baa80 con 0x7fd1f81b20f0 2024-04-14T15:19:40.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.642+0000 7fd1f5ffb700 10 client.47366.objecter ms_dispatch 0x7fd1f8001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:19:40.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.642+0000 7fd1f5ffb700 10 client.47366.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:19:40.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.642+0000 7fd1f5ffb700 20 client.47366.objecter put_session s=0x7fd1f8063200 osd=4 4 2024-04-14T15:19:40.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.642+0000 7fd1f5ffb700 15 client.47366.objecter _session_command_op_remove 4 10 2024-04-14T15:19:40.643 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:19:40.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.642+0000 7fd1ff213700 10 client.47366.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:19:40.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.642+0000 7fd1ff213700 15 client.47366.objecter _session_command_op_assign -1 11 2024-04-14T15:19:40.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.642+0000 7fd1ff213700 20 client.47366.objecter _get_session s=0x7fd1f81baa80 osd=5 3 2024-04-14T15:19:40.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.642+0000 7fd1ff213700 20 client.47366.objecter put_session s=0x7fd1f81baa80 osd=5 3 2024-04-14T15:19:40.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.642+0000 7fd1ff213700 20 client.47366.objecter _get_session s=0x7fd1f81baa80 osd=5 3 2024-04-14T15:19:40.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.642+0000 7fd1ff213700 15 client.47366.objecter _session_command_op_remove -1 11 2024-04-14T15:19:40.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.642+0000 7fd1ff213700 20 client.47366.objecter get_session s=0x7fd1f81baa80 osd=5 3 2024-04-14T15:19:40.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.642+0000 7fd1ff213700 15 client.47366.objecter _session_command_op_assign 5 11 2024-04-14T15:19:40.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.642+0000 7fd1ff213700 20 client.47366.objecter put_session s=0x7fd1f81baa80 osd=5 4 2024-04-14T15:19:40.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.642+0000 7fd1ff213700 10 client.47366.objecter _send_command 11 2024-04-14T15:19:40.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.642+0000 7fd1ff213700 1 -- 172.21.15.90:0/3815000204 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fd1f81bee30 con 0x7fd1f81bb100 2024-04-14T15:19:40.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.643+0000 7fd1f5ffb700 10 client.47366.objecter ms_handle_connect 0x7fd1f81bb100 2024-04-14T15:19:40.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.643+0000 7fd1f5ffb700 1 -- 172.21.15.90:0/3815000204 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd1f81bee30 con 0x7fd1f81bb100 2024-04-14T15:19:40.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.643+0000 7fd1f5ffb700 10 client.47366.objecter ms_dispatch 0x7fd1f8001550 command_reply(tid 11: 0 ) 2024-04-14T15:19:40.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.643+0000 7fd1f5ffb700 10 client.47366.objecter _finish_command 11 = system:0 2024-04-14T15:19:40.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.643+0000 7fd1f5ffb700 20 client.47366.objecter put_session s=0x7fd1f81baa80 osd=5 4 2024-04-14T15:19:40.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.643+0000 7fd1f5ffb700 15 client.47366.objecter _session_command_op_remove 5 11 2024-04-14T15:19:40.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.660+0000 7fd1ff213700 10 client.47366.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:19:40.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.660+0000 7fd1ff213700 15 client.47366.objecter _session_command_op_assign -1 12 2024-04-14T15:19:40.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.660+0000 7fd1ff213700 20 client.47366.objecter _get_session s=0x7fd1f81baa80 osd=5 3 2024-04-14T15:19:40.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.660+0000 7fd1ff213700 20 client.47366.objecter put_session s=0x7fd1f81baa80 osd=5 3 2024-04-14T15:19:40.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.660+0000 7fd1ff213700 20 client.47366.objecter _get_session s=0x7fd1f81baa80 osd=5 3 2024-04-14T15:19:40.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.660+0000 7fd1ff213700 15 client.47366.objecter _session_command_op_remove -1 12 2024-04-14T15:19:40.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.660+0000 7fd1ff213700 20 client.47366.objecter get_session s=0x7fd1f81baa80 osd=5 3 2024-04-14T15:19:40.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.660+0000 7fd1ff213700 15 client.47366.objecter _session_command_op_assign 5 12 2024-04-14T15:19:40.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.660+0000 7fd1ff213700 20 client.47366.objecter put_session s=0x7fd1f81baa80 osd=5 4 2024-04-14T15:19:40.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.660+0000 7fd1ff213700 10 client.47366.objecter _send_command 12 2024-04-14T15:19:40.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.660+0000 7fd1ff213700 1 -- 172.21.15.90:0/3815000204 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fd1f81c07e0 con 0x7fd1f81bb100 2024-04-14T15:19:40.662 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:19:40.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.662+0000 7fd1f5ffb700 1 -- 172.21.15.90:0/3815000204 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd1f81c07e0 con 0x7fd1f81bb100 2024-04-14T15:19:40.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.662+0000 7fd1f5ffb700 10 client.47366.objecter ms_dispatch 0x7fd1f8001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:19:40.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.662+0000 7fd1f5ffb700 10 client.47366.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:19:40.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.662+0000 7fd1f5ffb700 20 client.47366.objecter put_session s=0x7fd1f81baa80 osd=5 4 2024-04-14T15:19:40.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.662+0000 7fd1f5ffb700 15 client.47366.objecter _session_command_op_remove 5 12 2024-04-14T15:19:40.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.662+0000 7fd1ff213700 10 client.47366.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:19:40.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.662+0000 7fd1ff213700 15 client.47366.objecter _session_command_op_assign -1 13 2024-04-14T15:19:40.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.662+0000 7fd1ff213700 20 client.47366.objecter _get_session s=0x7fd1f81c07e0 osd=6 3 2024-04-14T15:19:40.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.662+0000 7fd1ff213700 20 client.47366.objecter put_session s=0x7fd1f81c07e0 osd=6 3 2024-04-14T15:19:40.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.662+0000 7fd1ff213700 20 client.47366.objecter _get_session s=0x7fd1f81c07e0 osd=6 3 2024-04-14T15:19:40.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.662+0000 7fd1ff213700 15 client.47366.objecter _session_command_op_remove -1 13 2024-04-14T15:19:40.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.662+0000 7fd1ff213700 20 client.47366.objecter get_session s=0x7fd1f81c07e0 osd=6 3 2024-04-14T15:19:40.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.662+0000 7fd1ff213700 15 client.47366.objecter _session_command_op_assign 6 13 2024-04-14T15:19:40.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.662+0000 7fd1ff213700 20 client.47366.objecter put_session s=0x7fd1f81c07e0 osd=6 4 2024-04-14T15:19:40.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.662+0000 7fd1ff213700 10 client.47366.objecter _send_command 13 2024-04-14T15:19:40.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.662+0000 7fd1ff213700 1 -- 172.21.15.90:0/3815000204 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fd1f81c1360 con 0x7fd1f81c0f00 2024-04-14T15:19:40.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.663+0000 7fd1f5ffb700 10 client.47366.objecter ms_handle_connect 0x7fd1f81c0f00 2024-04-14T15:19:40.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.663+0000 7fd1f5ffb700 1 -- 172.21.15.90:0/3815000204 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd1f81c1360 con 0x7fd1f81c0f00 2024-04-14T15:19:40.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.664+0000 7fd1f5ffb700 10 client.47366.objecter ms_dispatch 0x7fd1f8001550 command_reply(tid 13: 0 ) 2024-04-14T15:19:40.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.664+0000 7fd1f5ffb700 10 client.47366.objecter _finish_command 13 = system:0 2024-04-14T15:19:40.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.664+0000 7fd1f5ffb700 20 client.47366.objecter put_session s=0x7fd1f81c07e0 osd=6 4 2024-04-14T15:19:40.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.664+0000 7fd1f5ffb700 15 client.47366.objecter _session_command_op_remove 6 13 2024-04-14T15:19:40.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.682+0000 7fd1ff213700 10 client.47366.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:19:40.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.682+0000 7fd1ff213700 15 client.47366.objecter _session_command_op_assign -1 14 2024-04-14T15:19:40.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.682+0000 7fd1ff213700 20 client.47366.objecter _get_session s=0x7fd1f81c07e0 osd=6 3 2024-04-14T15:19:40.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.682+0000 7fd1ff213700 20 client.47366.objecter put_session s=0x7fd1f81c07e0 osd=6 3 2024-04-14T15:19:40.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.682+0000 7fd1ff213700 20 client.47366.objecter _get_session s=0x7fd1f81c07e0 osd=6 3 2024-04-14T15:19:40.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.682+0000 7fd1ff213700 15 client.47366.objecter _session_command_op_remove -1 14 2024-04-14T15:19:40.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.682+0000 7fd1ff213700 20 client.47366.objecter get_session s=0x7fd1f81c07e0 osd=6 3 2024-04-14T15:19:40.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.682+0000 7fd1ff213700 15 client.47366.objecter _session_command_op_assign 6 14 2024-04-14T15:19:40.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.682+0000 7fd1ff213700 20 client.47366.objecter put_session s=0x7fd1f81c07e0 osd=6 4 2024-04-14T15:19:40.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.682+0000 7fd1ff213700 10 client.47366.objecter _send_command 14 2024-04-14T15:19:40.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.682+0000 7fd1ff213700 1 -- 172.21.15.90:0/3815000204 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fd1f81c1360 con 0x7fd1f81c0f00 2024-04-14T15:19:40.683 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:40.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.684+0000 7fd1f5ffb700 1 -- 172.21.15.90:0/3815000204 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd1f81c1360 con 0x7fd1f81c0f00 2024-04-14T15:19:40.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.684+0000 7fd1f5ffb700 10 client.47366.objecter ms_dispatch 0x7fd1f8001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:19:40.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.684+0000 7fd1f5ffb700 10 client.47366.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:19:40.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.684+0000 7fd1f5ffb700 20 client.47366.objecter put_session s=0x7fd1f81c07e0 osd=6 4 2024-04-14T15:19:40.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.684+0000 7fd1f5ffb700 15 client.47366.objecter _session_command_op_remove 6 14 2024-04-14T15:19:40.685 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:19:40.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.685+0000 7fd1ff213700 10 client.47366.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:19:40.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.685+0000 7fd1ff213700 15 client.47366.objecter _session_command_op_assign -1 15 2024-04-14T15:19:40.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.685+0000 7fd1ff213700 20 client.47366.objecter _get_session s=0x7fd1f81c1360 osd=7 3 2024-04-14T15:19:40.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.685+0000 7fd1ff213700 20 client.47366.objecter put_session s=0x7fd1f81c1360 osd=7 3 2024-04-14T15:19:40.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.685+0000 7fd1ff213700 20 client.47366.objecter _get_session s=0x7fd1f81c1360 osd=7 3 2024-04-14T15:19:40.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.685+0000 7fd1ff213700 15 client.47366.objecter _session_command_op_remove -1 15 2024-04-14T15:19:40.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.685+0000 7fd1ff213700 20 client.47366.objecter get_session s=0x7fd1f81c1360 osd=7 3 2024-04-14T15:19:40.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.685+0000 7fd1ff213700 15 client.47366.objecter _session_command_op_assign 7 15 2024-04-14T15:19:40.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.685+0000 7fd1ff213700 20 client.47366.objecter put_session s=0x7fd1f81c1360 osd=7 4 2024-04-14T15:19:40.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.685+0000 7fd1ff213700 10 client.47366.objecter _send_command 15 2024-04-14T15:19:40.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.685+0000 7fd1ff213700 1 -- 172.21.15.90:0/3815000204 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fd1f81c77d0 con 0x7fd1f81c1590 2024-04-14T15:19:40.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.685+0000 7fd1f5ffb700 10 client.47366.objecter ms_handle_connect 0x7fd1f81c1590 2024-04-14T15:19:40.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.686+0000 7fd1f5ffb700 1 -- 172.21.15.90:0/3815000204 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd1f81c77d0 con 0x7fd1f81c1590 2024-04-14T15:19:40.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.686+0000 7fd1f5ffb700 10 client.47366.objecter ms_dispatch 0x7fd1f8001550 command_reply(tid 15: 0 ) 2024-04-14T15:19:40.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.686+0000 7fd1f5ffb700 10 client.47366.objecter _finish_command 15 = system:0 2024-04-14T15:19:40.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.686+0000 7fd1f5ffb700 20 client.47366.objecter put_session s=0x7fd1f81c1360 osd=7 4 2024-04-14T15:19:40.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.686+0000 7fd1f5ffb700 15 client.47366.objecter _session_command_op_remove 7 15 2024-04-14T15:19:40.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.702+0000 7fd1ff213700 10 client.47366.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:19:40.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.702+0000 7fd1ff213700 15 client.47366.objecter _session_command_op_assign -1 16 2024-04-14T15:19:40.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.702+0000 7fd1ff213700 20 client.47366.objecter _get_session s=0x7fd1f81c1360 osd=7 3 2024-04-14T15:19:40.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.702+0000 7fd1ff213700 20 client.47366.objecter put_session s=0x7fd1f81c1360 osd=7 3 2024-04-14T15:19:40.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.702+0000 7fd1ff213700 20 client.47366.objecter _get_session s=0x7fd1f81c1360 osd=7 3 2024-04-14T15:19:40.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.702+0000 7fd1ff213700 15 client.47366.objecter _session_command_op_remove -1 16 2024-04-14T15:19:40.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.702+0000 7fd1ff213700 20 client.47366.objecter get_session s=0x7fd1f81c1360 osd=7 3 2024-04-14T15:19:40.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.702+0000 7fd1ff213700 15 client.47366.objecter _session_command_op_assign 7 16 2024-04-14T15:19:40.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.702+0000 7fd1ff213700 20 client.47366.objecter put_session s=0x7fd1f81c1360 osd=7 4 2024-04-14T15:19:40.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.702+0000 7fd1ff213700 10 client.47366.objecter _send_command 16 2024-04-14T15:19:40.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.702+0000 7fd1ff213700 1 -- 172.21.15.90:0/3815000204 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fd1f8063010 con 0x7fd1f81c1590 2024-04-14T15:19:40.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.704+0000 7fd1f5ffb700 1 -- 172.21.15.90:0/3815000204 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd1f8063010 con 0x7fd1f81c1590 2024-04-14T15:19:40.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.704+0000 7fd1f5ffb700 10 client.47366.objecter ms_dispatch 0x7fd1f8001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:19:40.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.704+0000 7fd1f5ffb700 10 client.47366.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:19:40.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.704+0000 7fd1f5ffb700 20 client.47366.objecter put_session s=0x7fd1f81c1360 osd=7 4 2024-04-14T15:19:40.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.704+0000 7fd1f5ffb700 15 client.47366.objecter _session_command_op_remove 7 16 2024-04-14T15:19:40.705 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:19:40.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.704+0000 7fd1ff213700 10 librados: watch_flush enter 2024-04-14T15:19:40.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.704+0000 7fd1ff213700 10 librados: watch_flush exit 2024-04-14T15:19:40.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.704+0000 7fd1ff213700 10 client.47366.objecter close_session for osd.0 2024-04-14T15:19:40.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.704+0000 7fd1ff213700 1 -- 172.21.15.90:0/3815000204 >> v1:172.21.15.90:6801/3761806885 conn(0x7fd1f81a8cf0 legacy=0x7fd1f81a90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:40.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.704+0000 7fd1ff213700 20 client.47366.objecter put_session s=0x7fd1f81122e0 osd=0 1 2024-04-14T15:19:40.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.704+0000 7fd1ff213700 10 client.47366.objecter close_session for osd.1 2024-04-14T15:19:40.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.704+0000 7fd1ff213700 1 -- 172.21.15.90:0/3815000204 >> v1:172.21.15.90:6805/4170361699 conn(0x7fd1f810ec90 legacy=0x7fd1f810f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:40.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.704+0000 7fd1ff213700 20 client.47366.objecter put_session s=0x7fd1f81a97e0 osd=1 1 2024-04-14T15:19:40.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.704+0000 7fd1ff213700 10 client.47366.objecter close_session for osd.2 2024-04-14T15:19:40.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.704+0000 7fd1ff213700 1 -- 172.21.15.90:0/3815000204 >> v1:172.21.15.90:6809/1686184251 conn(0x7fd1f81ad750 legacy=0x7fd1f81afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:40.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.704+0000 7fd1ff213700 20 client.47366.objecter put_session s=0x7fd1f802e2d0 osd=2 1 2024-04-14T15:19:40.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.704+0000 7fd1ff213700 10 client.47366.objecter close_session for osd.3 2024-04-14T15:19:40.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.704+0000 7fd1ff213700 1 -- 172.21.15.90:0/3815000204 >> v1:172.21.15.90:6813/2270166774 conn(0x7fd1f81b2590 legacy=0x7fd1f81b4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:40.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.704+0000 7fd1ff213700 20 client.47366.objecter put_session s=0x7fd1f80636c0 osd=3 1 2024-04-14T15:19:40.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.704+0000 7fd1ff213700 10 client.47366.objecter close_session for osd.4 2024-04-14T15:19:40.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.704+0000 7fd1ff213700 1 -- 172.21.15.90:0/3815000204 >> v1:172.21.15.92:6800/3887549733 conn(0x7fd1f81b20f0 legacy=0x7fd1f81b60a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:40.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.704+0000 7fd1ff213700 20 client.47366.objecter put_session s=0x7fd1f8063200 osd=4 1 2024-04-14T15:19:40.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.704+0000 7fd1ff213700 10 client.47366.objecter close_session for osd.5 2024-04-14T15:19:40.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.704+0000 7fd1ff213700 1 -- 172.21.15.90:0/3815000204 >> v1:172.21.15.92:6804/3244561440 conn(0x7fd1f81bb100 legacy=0x7fd1f81bd580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:40.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.704+0000 7fd1fd7b0700 1 -- 172.21.15.90:0/3815000204 reap_dead start 2024-04-14T15:19:40.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.704+0000 7fd1ff213700 20 client.47366.objecter put_session s=0x7fd1f81baa80 osd=5 1 2024-04-14T15:19:40.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.704+0000 7fd1ff213700 10 client.47366.objecter close_session for osd.6 2024-04-14T15:19:40.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.704+0000 7fd1ff213700 1 -- 172.21.15.90:0/3815000204 >> v1:172.21.15.92:6808/636125963 conn(0x7fd1f81c0f00 legacy=0x7fd1f81bf010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:40.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.704+0000 7fd1fd7b0700 1 -- 172.21.15.90:0/3815000204 reap_dead start 2024-04-14T15:19:40.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.704+0000 7fd1ff213700 20 client.47366.objecter put_session s=0x7fd1f81c07e0 osd=6 1 2024-04-14T15:19:40.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.704+0000 7fd1ff213700 10 client.47366.objecter close_session for osd.7 2024-04-14T15:19:40.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.704+0000 7fd1ff213700 1 -- 172.21.15.90:0/3815000204 >> v1:172.21.15.92:6812/1773706716 conn(0x7fd1f81c1590 legacy=0x7fd1f81c5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:40.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.704+0000 7fd1ff213700 20 client.47366.objecter put_session s=0x7fd1f81c1360 osd=7 1 2024-04-14T15:19:40.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.704+0000 7fd1ff213700 20 client.47366.objecter shutdown clearing up homeless session... 2024-04-14T15:19:40.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.704+0000 7fd1ff213700 10 client.47366.objecter successfully canceled tick 2024-04-14T15:19:40.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.704+0000 7fd1fd7b0700 1 -- 172.21.15.90:0/3815000204 reap_dead start 2024-04-14T15:19:40.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.705+0000 7fd1fd7b0700 1 -- 172.21.15.90:0/3815000204 reap_dead start 2024-04-14T15:19:40.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.705+0000 7fd1ff213700 1 -- 172.21.15.90:0/3815000204 >> v1:172.21.15.90:6800/420935427 conn(0x7fd1e4075540 legacy=0x7fd1e4077a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:40.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.705+0000 7fd1ff213700 1 -- 172.21.15.90:0/3815000204 >> v1:172.21.15.90:6790/0 conn(0x7fd1f810e310 legacy=0x7fd1f81a6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:40.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.705+0000 7fd1ff213700 1 -- 172.21.15.90:0/3815000204 shutdown_connections 2024-04-14T15:19:40.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.705+0000 7fd1ff213700 1 -- 172.21.15.90:0/3815000204 >> 172.21.15.90:0/3815000204 conn(0x7fd1f80fd290 msgr2=0x7fd1f81006f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:19:40.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.705+0000 7fd1ff213700 1 -- 172.21.15.90:0/3815000204 shutdown_connections 2024-04-14T15:19:40.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.705+0000 7fd1ff213700 1 -- 172.21.15.90:0/3815000204 wait complete. 2024-04-14T15:19:40.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:40.705+0000 7fd1ff213700 1 librados: shutdown 2024-04-14T15:19:40.709 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:19:40.709 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:19:40.709 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:19:40.709 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:19:40.709 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:19:40.709 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:19:40.709 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:19:40.709 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:19:40.711 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:40.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:40.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:40.813 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:19:40.840 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:40.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:40.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:19:40.941 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:40.969 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:40.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:40.970 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:41.071 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:19:41.096 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:41.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:41.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:19:41.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:41 smithi090 ceph-mon[41670]: pgmap v677: 258 pgs: 10 creating+activating, 6 unknown, 242 active+clean; 8.4 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 10 B/s, 3 objects/s recovering 2024-04-14T15:19:41.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:41 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3815000204' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:19:41.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:41 smithi090 ceph-mon[41670]: osdmap e477: 8 total, 8 up, 8 in 2024-04-14T15:19:41.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:41 smithi090 ceph-mon[30399]: pgmap v677: 258 pgs: 10 creating+activating, 6 unknown, 242 active+clean; 8.4 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 10 B/s, 3 objects/s recovering 2024-04-14T15:19:41.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:41 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3815000204' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:19:41.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:41 smithi090 ceph-mon[30399]: osdmap e477: 8 total, 8 up, 8 in 2024-04-14T15:19:41.198 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:41.223 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:41.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:41.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:41.325 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:41.349 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:41.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:41.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:41.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:41 smithi092 ceph-mon[33966]: pgmap v677: 258 pgs: 10 creating+activating, 6 unknown, 242 active+clean; 8.4 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 10 B/s, 3 objects/s recovering 2024-04-14T15:19:41.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:41 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3815000204' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:19:41.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:41 smithi092 ceph-mon[33966]: osdmap e477: 8 total, 8 up, 8 in 2024-04-14T15:19:41.451 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:19:41.475 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:41.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:41.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:19:41.576 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:19:41.602 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:41.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:41.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:19:41.704 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:19:41.727 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:41.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:41.728 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:19:41.828 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:19:41.855 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:41.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:41.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:19:41.957 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:19:41.983 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:41.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:41.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:19:42.085 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:19:42.107 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:42.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:42.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:19:42.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:42 smithi090 ceph-mon[41670]: pgmap v680: 258 pgs: 12 creating+activating, 246 active+clean; 8.4 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 10 B/s, 3 objects/s recovering 2024-04-14T15:19:42.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:42 smithi090 ceph-mon[30399]: pgmap v680: 258 pgs: 12 creating+activating, 246 active+clean; 8.4 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 10 B/s, 3 objects/s recovering 2024-04-14T15:19:42.210 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:19:42.236 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:42.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:42.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:19:42.311 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:19:42.338 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:42.364 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:42.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:42.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:42.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:42 smithi092 ceph-mon[33966]: pgmap v680: 258 pgs: 12 creating+activating, 246 active+clean; 8.4 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 10 B/s, 3 objects/s recovering 2024-04-14T15:19:42.466 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:19:42.492 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:42.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:42.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:19:42.594 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:42.620 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:42.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:42.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:42.722 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:19:42.747 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:42.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:42.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:19:42.849 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:42.874 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:42.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:42.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:42.977 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:43.002 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:43.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:43.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:43.104 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:19:43.128 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:43.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:43.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:19:43.229 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:43.255 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:43.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:43.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:43.357 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:19:43.382 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:43.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:43.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:19:43.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:43.392+0000 7f864e2e5700 1 -- 172.21.15.90:0/1872894182 >> v1:172.21.15.90:6790/0 conn(0x7f8648107780 legacy=0x7f8648107b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:43.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:43.393+0000 7f864e2e5700 1 -- 172.21.15.90:0/1872894182 shutdown_connections 2024-04-14T15:19:43.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:43.393+0000 7f864e2e5700 1 -- 172.21.15.90:0/1872894182 >> 172.21.15.90:0/1872894182 conn(0x7f86480fd2b0 msgr2=0x7f86480ff6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:19:43.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:43.393+0000 7f864e2e5700 1 -- 172.21.15.90:0/1872894182 shutdown_connections 2024-04-14T15:19:43.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:43.393+0000 7f864e2e5700 1 -- 172.21.15.90:0/1872894182 wait complete. 2024-04-14T15:19:43.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:43.393+0000 7f864e2e5700 1 librados: starting msgr at 2024-04-14T15:19:43.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:43.393+0000 7f864e2e5700 1 librados: starting objecter 2024-04-14T15:19:43.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:43.393+0000 7f864e2e5700 1 Processor -- start 2024-04-14T15:19:43.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:43.393+0000 7f864e2e5700 1 -- start start 2024-04-14T15:19:43.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:43.393+0000 7f864e2e5700 1 librados: setting wanted keys 2024-04-14T15:19:43.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:43.393+0000 7f864e2e5700 1 librados: calling monclient init 2024-04-14T15:19:43.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:43.394+0000 7f864e2e5700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f86481a79b0 con 0x7f864810e200 2024-04-14T15:19:43.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:43.394+0000 7f864e2e5700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f86481a8bd0 con 0x7f8648107780 2024-04-14T15:19:43.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:43.394+0000 7f864e2e5700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f86481a9e10 con 0x7f864810a630 2024-04-14T15:19:43.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:43.394+0000 7f86477fe700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f864810a630 0x7f86481a26c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:35596/0 (socket says 172.21.15.90:35596) 2024-04-14T15:19:43.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:43.394+0000 7f86477fe700 1 -- 172.21.15.90:0/3290133441 learned_addr learned my addr 172.21.15.90:0/3290133441 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:19:43.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:43.394+0000 7f864c882700 1 --1- 172.21.15.90:0/3290133441 >> v1:172.21.15.90:6789/0 conn(0x7f864810e200 0x7f86481a6070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:53382/0 (socket says 172.21.15.90:53382) 2024-04-14T15:19:43.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:43.394+0000 7f86457fa700 10 client.?.objecter ms_handle_connect 0x7f864810e200 2024-04-14T15:19:43.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:43.394+0000 7f86457fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:19:43.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:43.394+0000 7f86457fa700 10 client.?.objecter ms_handle_connect 0x7f864810a630 2024-04-14T15:19:43.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:43.394+0000 7f86457fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:19:43.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:43.394+0000 7f86457fa700 10 client.?.objecter ms_handle_connect 0x7f8648107780 2024-04-14T15:19:43.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:43.394+0000 7f86457fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:19:43.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:43.394+0000 7f86457fa700 1 -- 172.21.15.90:0/3290133441 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4071603528 0 0) 0x7f86481a8bd0 con 0x7f8648107780 2024-04-14T15:19:43.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:43.394+0000 7f86457fa700 1 -- 172.21.15.90:0/3290133441 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8630003480 con 0x7f8648107780 2024-04-14T15:19:43.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:43.394+0000 7f86457fa700 1 -- 172.21.15.90:0/3290133441 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3204469215 0 0) 0x7f86481a79b0 con 0x7f864810e200 2024-04-14T15:19:43.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:43.394+0000 7f86457fa700 1 -- 172.21.15.90:0/3290133441 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f86481a8bd0 con 0x7f864810e200 2024-04-14T15:19:43.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:43.394+0000 7f86457fa700 1 -- 172.21.15.90:0/3290133441 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2087172958 0 0) 0x7f86481a9e10 con 0x7f864810a630 2024-04-14T15:19:43.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:43.394+0000 7f86457fa700 1 -- 172.21.15.90:0/3290133441 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f86481a79b0 con 0x7f864810a630 2024-04-14T15:19:43.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:43.394+0000 7f86457fa700 1 -- 172.21.15.90:0/3290133441 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1573904359 0 0) 0x7f8630003480 con 0x7f8648107780 2024-04-14T15:19:43.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:43.394+0000 7f86457fa700 1 -- 172.21.15.90:0/3290133441 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f86481a9e10 con 0x7f8648107780 2024-04-14T15:19:43.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:43.394+0000 7f86457fa700 1 -- 172.21.15.90:0/3290133441 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f8638003820 con 0x7f8648107780 2024-04-14T15:19:43.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:43.394+0000 7f86457fa700 1 -- 172.21.15.90:0/3290133441 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 203078629 0 0) 0x7f86481a79b0 con 0x7f864810a630 2024-04-14T15:19:43.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:43.394+0000 7f86457fa700 1 -- 172.21.15.90:0/3290133441 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8630003480 con 0x7f864810a630 2024-04-14T15:19:43.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:43.394+0000 7f86457fa700 1 -- 172.21.15.90:0/3290133441 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 483310027 0 0) 0x7f86481a8bd0 con 0x7f864810e200 2024-04-14T15:19:43.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:43.394+0000 7f86457fa700 1 -- 172.21.15.90:0/3290133441 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f86481a79b0 con 0x7f864810e200 2024-04-14T15:19:43.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:43.394+0000 7f86457fa700 1 -- 172.21.15.90:0/3290133441 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f863c001230 con 0x7f864810a630 2024-04-14T15:19:43.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:43.394+0000 7f86457fa700 1 -- 172.21.15.90:0/3290133441 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f8640001af0 con 0x7f864810e200 2024-04-14T15:19:43.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:43.395+0000 7f86457fa700 1 -- 172.21.15.90:0/3290133441 <== mon.2 v1:172.21.15.90:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 171633974 0 0) 0x7f8630003480 con 0x7f864810a630 2024-04-14T15:19:43.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:43.395+0000 7f86457fa700 1 -- 172.21.15.90:0/3290133441 >> v1:172.21.15.92:6789/0 conn(0x7f8648107780 legacy=0x7f864810da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:43.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:43.395+0000 7f86457fa700 1 -- 172.21.15.90:0/3290133441 >> v1:172.21.15.90:6789/0 conn(0x7f864810e200 legacy=0x7f86481a6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:43.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:43.395+0000 7f86457fa700 1 -- 172.21.15.90:0/3290133441 --> v1:172.21.15.90:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86481ab050 con 0x7f864810a630 2024-04-14T15:19:43.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:43.395+0000 7f864e2e5700 1 -- 172.21.15.90:0/3290133441 --> v1:172.21.15.90:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f86481a8e00 con 0x7f864810a630 2024-04-14T15:19:43.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:43.395+0000 7f864e2e5700 10 client.47372.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:19:43.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:43.395+0000 7f864e2e5700 1 -- 172.21.15.90:0/3290133441 --> v1:172.21.15.90:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f86481a92c0 con 0x7f864810a630 2024-04-14T15:19:43.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:43.395+0000 7f864e2e5700 1 librados: init done 2024-04-14T15:19:43.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:43.395+0000 7f86457fa700 1 -- 172.21.15.90:0/3290133441 <== mon.2 v1:172.21.15.90:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f863c003b20 con 0x7f864810a630 2024-04-14T15:19:43.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:43.395+0000 7f86457fa700 1 -- 172.21.15.90:0/3290133441 <== mon.2 v1:172.21.15.90:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f863c005ba0 con 0x7f864810a630 2024-04-14T15:19:43.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:43.396+0000 7f864e2e5700 1 -- 172.21.15.90:0/3290133441 --> v1:172.21.15.90:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8634005120 con 0x7f864810a630 2024-04-14T15:19:43.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:43.396+0000 7f86457fa700 1 -- 172.21.15.90:0/3290133441 <== mon.2 v1:172.21.15.90:6790/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f863c01c810 con 0x7f864810a630 2024-04-14T15:19:43.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:43.397+0000 7f86457fa700 1 -- 172.21.15.90:0/3290133441 <== mon.2 v1:172.21.15.90:6790/0 8 ==== osd_map(477..477 src has 51..477) ==== 8197+0+0 (unknown 1240170119 0 0) 0x7f863c08a8e0 con 0x7f864810a630 2024-04-14T15:19:43.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:43.397+0000 7f86457fa700 10 client.47372.objecter ms_dispatch 0x7f8648001550 osd_map(477..477 src has 51..477) 2024-04-14T15:19:43.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:43.397+0000 7f86457fa700 3 client.47372.objecter handle_osd_map got epochs [477,477] > 0 2024-04-14T15:19:43.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:43.397+0000 7f86457fa700 3 client.47372.objecter handle_osd_map decoding full epoch 477 2024-04-14T15:19:43.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:43.397+0000 7f86457fa700 20 client.47372.objecter dump_active .. 0 homeless 2024-04-14T15:19:43.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:43.397+0000 7f86457fa700 10 client.47372.objecter ms_handle_connect 0x7f8630071330 2024-04-14T15:19:43.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:43.401+0000 7f86457fa700 1 -- 172.21.15.90:0/3290133441 <== mon.2 v1:172.21.15.90:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f863c0572f0 con 0x7f864810a630 2024-04-14T15:19:43.484 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:19:43.506 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:43.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:43.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:19:43.608 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:19:43.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:43.615+0000 7f864e2e5700 1 -- 172.21.15.90:0/3290133441 --> v1:172.21.15.90:6790/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7f8634005410 con 0x7f864810a630 2024-04-14T15:19:43.631 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:43.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:43.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:19:43.732 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:19:43.755 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:43.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:43.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:19:43.857 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:43.884 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:43.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:43.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:43.955 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-14T15:19:43.986 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:44.014 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:44.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:44.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:44.115 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:44.142 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:44.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:44.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:44.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:44.164+0000 7f86457fa700 1 -- 172.21.15.90:0/3290133441 <== mon.2 v1:172.21.15.90:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v478) ==== 91+0+0 (unknown 2893711743 0 0) 0x7f863c05afa0 con 0x7f864810a630 2024-04-14T15:19:44.165 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:44 smithi090 ceph-mon[41670]: pgmap v681: 258 pgs: 12 creating+activating, 246 active+clean; 8.4 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 995 B/s rd, 199 B/s wr, 1 op/s; 7 B/s, 3 objects/s recovering 2024-04-14T15:19:44.165 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:44 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3290133441' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:19:44.165 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:44 smithi090 ceph-mon[41670]: from='client.47372 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:19:44.165 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:19:43 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:19:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:19:44.166 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:44 smithi090 ceph-mon[30399]: pgmap v681: 258 pgs: 12 creating+activating, 246 active+clean; 8.4 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 995 B/s rd, 199 B/s wr, 1 op/s; 7 B/s, 3 objects/s recovering 2024-04-14T15:19:44.166 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:44 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3290133441' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:19:44.166 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:44 smithi090 ceph-mon[30399]: from='client.47372 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:19:44.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:44.167+0000 7f864e2e5700 10 librados: watch_flush enter 2024-04-14T15:19:44.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:44.167+0000 7f864e2e5700 10 librados: watch_flush exit 2024-04-14T15:19:44.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:44.167+0000 7f864e2e5700 20 client.47372.objecter shutdown clearing up homeless session... 2024-04-14T15:19:44.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:44.167+0000 7f864e2e5700 10 client.47372.objecter successfully canceled tick 2024-04-14T15:19:44.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:44.167+0000 7f864e2e5700 1 -- 172.21.15.90:0/3290133441 >> v1:172.21.15.90:6800/420935427 conn(0x7f8630071330 legacy=0x7f86300737f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:44.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:44.167+0000 7f864e2e5700 1 -- 172.21.15.90:0/3290133441 >> v1:172.21.15.90:6790/0 conn(0x7f864810a630 legacy=0x7f86481a26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:44.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:44.167+0000 7f864e2e5700 1 -- 172.21.15.90:0/3290133441 shutdown_connections 2024-04-14T15:19:44.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:44.167+0000 7f864e2e5700 1 -- 172.21.15.90:0/3290133441 >> 172.21.15.90:0/3290133441 conn(0x7f86480fd2b0 msgr2=0x7f8648101780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:19:44.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:44.167+0000 7f864e2e5700 1 -- 172.21.15.90:0/3290133441 shutdown_connections 2024-04-14T15:19:44.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:44.167+0000 7f864e2e5700 1 -- 172.21.15.90:0/3290133441 wait complete. 2024-04-14T15:19:44.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:44.168+0000 7f864e2e5700 1 librados: shutdown 2024-04-14T15:19:44.169 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is unset 2024-04-14T15:19:44.244 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:44.274 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:44.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:44.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:44.376 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:19:44.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:44.401 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:44.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:19:44.502 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:19:44.530 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:44.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:44.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:19:44.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:44 smithi092 ceph-mon[33966]: pgmap v681: 258 pgs: 12 creating+activating, 246 active+clean; 8.4 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 995 B/s rd, 199 B/s wr, 1 op/s; 7 B/s, 3 objects/s recovering 2024-04-14T15:19:44.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:44 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3290133441' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:19:44.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:44 smithi092 ceph-mon[33966]: from='client.47372 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:19:44.632 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:19:44.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:44.657 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:44.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:19:44.758 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:19:44.784 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:44.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:44.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:19:44.886 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:19:44.912 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:44.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:44.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:19:45.014 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:19:45.040 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:45.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:45.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:19:45.141 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:19:45.166 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:45.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:45.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:19:45.267 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:19:45.291 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:45.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:45.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:19:45.392 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:19:45.416 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:45.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:45.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:19:45.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:45 smithi090 ceph-mon[41670]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:19:45.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:45 smithi090 ceph-mon[41670]: from='client.47372 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-14T15:19:45.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:45 smithi090 ceph-mon[41670]: osdmap e478: 8 total, 8 up, 8 in 2024-04-14T15:19:45.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:45 smithi090 ceph-mon[41670]: 4.9 scrub starts 2024-04-14T15:19:45.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:45 smithi090 ceph-mon[41670]: 4.9 scrub ok 2024-04-14T15:19:45.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:45 smithi090 ceph-mon[30399]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:19:45.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:45 smithi090 ceph-mon[30399]: from='client.47372 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-14T15:19:45.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:45 smithi090 ceph-mon[30399]: osdmap e478: 8 total, 8 up, 8 in 2024-04-14T15:19:45.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:45 smithi090 ceph-mon[30399]: 4.9 scrub starts 2024-04-14T15:19:45.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:45 smithi090 ceph-mon[30399]: 4.9 scrub ok 2024-04-14T15:19:45.519 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:19:45.542 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:45.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:45.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:19:45.622 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:19:45 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:19:45.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:45 smithi092 ceph-mon[33966]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:19:45.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:45 smithi092 ceph-mon[33966]: from='client.47372 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-14T15:19:45.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:45 smithi092 ceph-mon[33966]: osdmap e478: 8 total, 8 up, 8 in 2024-04-14T15:19:45.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:45 smithi092 ceph-mon[33966]: 4.9 scrub starts 2024-04-14T15:19:45.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:45 smithi092 ceph-mon[33966]: 4.9 scrub ok 2024-04-14T15:19:45.645 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:19:45.668 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:45.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:45.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:19:45.770 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:19:45.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:45.798 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:45.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:19:45.899 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:19:45.926 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:45.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:45.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:19:46.027 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:46.054 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:46.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:46.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:46.156 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:19:46.182 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:46.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:46.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:19:46.283 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:19:46.307 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:46.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:46.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:19:46.364 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:46 smithi090 ceph-mon[30399]: pgmap v683: 258 pgs: 258 active+clean; 8.4 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-04-14T15:19:46.364 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:46 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:19:46.365 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:46 smithi090 ceph-mon[41670]: pgmap v683: 258 pgs: 258 active+clean; 8.4 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-04-14T15:19:46.365 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:46 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:19:46.409 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:19:46.434 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:46.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:46.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:19:46.536 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:19:46.564 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:46.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:46.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:19:46.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:46 smithi092 ceph-mon[33966]: pgmap v683: 258 pgs: 258 active+clean; 8.4 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-04-14T15:19:46.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:46 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:19:46.665 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:19:46.689 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:46.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:46.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:19:46.791 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:19:46.819 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:46.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:46.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:19:46.860 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:19:46.902 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-14T15:19:46.902 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-14T15:19:46.902 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-14T15:19:46.902 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-14T15:19:46.920 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:19:46.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:46.951 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:46.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:19:47.052 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:19:47.080 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:47.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:47.081 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:19:47.121 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-14T15:19:47.121 INFO:teuthology.orchestra.run.smithi090.stdout: "success": "" 2024-04-14T15:19:47.122 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-14T15:19:47.181 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:19:47.209 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:47.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:47.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:19:47.310 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:47.338 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:47.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:47.339 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:47.439 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:19:47.466 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:47.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:47.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:19:47.568 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:47.596 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:47.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:47.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:47.698 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:19:47.722 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:47.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:47.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:19:47.726 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-14T15:19:47.823 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:47.851 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:47.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:47.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:47.953 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:19:47.980 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:47.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:47.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:19:48.082 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:19:48.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:48.107 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:48.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:19:48.208 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:19:48.237 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:48.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:48.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:19:48.338 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:19:48.366 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:48.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:48.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:19:48.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:48 smithi090 ceph-mon[30399]: 11.8 scrub starts 2024-04-14T15:19:48.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:48 smithi090 ceph-mon[30399]: 11.8 scrub ok 2024-04-14T15:19:48.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:48 smithi090 ceph-mon[30399]: 5.4 scrub starts 2024-04-14T15:19:48.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:48 smithi090 ceph-mon[30399]: 5.4 scrub ok 2024-04-14T15:19:48.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:48 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:19:48.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:48 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:19:48.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:48 smithi090 ceph-mon[30399]: pgmap v684: 258 pgs: 258 active+clean; 8.4 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 859 B/s wr, 3 op/s 2024-04-14T15:19:48.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:48 smithi090 ceph-mon[30399]: osdmap e479: 8 total, 8 up, 8 in 2024-04-14T15:19:48.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:48 smithi090 ceph-mon[41670]: 11.8 scrub starts 2024-04-14T15:19:48.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:48 smithi090 ceph-mon[41670]: 11.8 scrub ok 2024-04-14T15:19:48.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:48 smithi090 ceph-mon[41670]: 5.4 scrub starts 2024-04-14T15:19:48.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:48 smithi090 ceph-mon[41670]: 5.4 scrub ok 2024-04-14T15:19:48.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:48 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:19:48.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:48 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:19:48.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:48 smithi090 ceph-mon[41670]: pgmap v684: 258 pgs: 258 active+clean; 8.4 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 859 B/s wr, 3 op/s 2024-04-14T15:19:48.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:48 smithi090 ceph-mon[41670]: osdmap e479: 8 total, 8 up, 8 in 2024-04-14T15:19:48.468 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:19:48.494 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:48.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:48.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:19:48.596 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:19:48.621 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:48.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:48.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:19:48.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:48 smithi092 ceph-mon[33966]: 11.8 scrub starts 2024-04-14T15:19:48.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:48 smithi092 ceph-mon[33966]: 11.8 scrub ok 2024-04-14T15:19:48.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:48 smithi092 ceph-mon[33966]: 5.4 scrub starts 2024-04-14T15:19:48.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:48 smithi092 ceph-mon[33966]: 5.4 scrub ok 2024-04-14T15:19:48.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:48 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:19:48.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:48 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:19:48.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:48 smithi092 ceph-mon[33966]: pgmap v684: 258 pgs: 258 active+clean; 8.4 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 859 B/s wr, 3 op/s 2024-04-14T15:19:48.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:48 smithi092 ceph-mon[33966]: osdmap e479: 8 total, 8 up, 8 in 2024-04-14T15:19:48.701 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-14T15:19:48.723 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:19:48.751 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:48.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:48.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:19:48.853 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:19:48.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:48.877 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:48.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:19:48.978 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:19:49.002 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:49.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:49.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:19:49.104 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:19:49.132 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:49.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:49.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:19:49.145 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:19:49.233 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:19:49.260 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:49.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:49.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:19:49.363 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:19:49.387 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:49.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:49.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:19:49.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:49 smithi090 ceph-mon[30399]: 6.15 scrub starts 2024-04-14T15:19:49.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:49 smithi090 ceph-mon[30399]: 6.15 scrub ok 2024-04-14T15:19:49.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:49 smithi090 ceph-mon[30399]: 11.9 scrub starts 2024-04-14T15:19:49.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:49 smithi090 ceph-mon[30399]: 11.9 scrub ok 2024-04-14T15:19:49.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:49 smithi090 ceph-mon[30399]: 110.12 scrub starts 2024-04-14T15:19:49.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:49 smithi090 ceph-mon[30399]: 110.12 scrub ok 2024-04-14T15:19:49.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:49 smithi090 ceph-mon[30399]: 110.17 scrub starts 2024-04-14T15:19:49.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:49 smithi090 ceph-mon[30399]: 110.17 scrub ok 2024-04-14T15:19:49.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:49 smithi090 ceph-mon[41670]: 6.15 scrub starts 2024-04-14T15:19:49.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:49 smithi090 ceph-mon[41670]: 6.15 scrub ok 2024-04-14T15:19:49.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:49 smithi090 ceph-mon[41670]: 11.9 scrub starts 2024-04-14T15:19:49.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:49 smithi090 ceph-mon[41670]: 11.9 scrub ok 2024-04-14T15:19:49.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:49 smithi090 ceph-mon[41670]: 110.12 scrub starts 2024-04-14T15:19:49.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:49 smithi090 ceph-mon[41670]: 110.12 scrub ok 2024-04-14T15:19:49.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:49 smithi090 ceph-mon[41670]: 110.17 scrub starts 2024-04-14T15:19:49.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:49 smithi090 ceph-mon[41670]: 110.17 scrub ok 2024-04-14T15:19:49.488 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:19:49.515 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:49.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:49.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:19:49.617 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:19:49.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:49 smithi092 ceph-mon[33966]: 6.15 scrub starts 2024-04-14T15:19:49.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:49 smithi092 ceph-mon[33966]: 6.15 scrub ok 2024-04-14T15:19:49.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:49 smithi092 ceph-mon[33966]: 11.9 scrub starts 2024-04-14T15:19:49.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:49 smithi092 ceph-mon[33966]: 11.9 scrub ok 2024-04-14T15:19:49.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:49 smithi092 ceph-mon[33966]: 110.12 scrub starts 2024-04-14T15:19:49.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:49 smithi092 ceph-mon[33966]: 110.12 scrub ok 2024-04-14T15:19:49.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:49 smithi092 ceph-mon[33966]: 110.17 scrub starts 2024-04-14T15:19:49.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:49 smithi092 ceph-mon[33966]: 110.17 scrub ok 2024-04-14T15:19:49.640 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:49.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:49.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:19:49.742 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:19:49.766 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:49.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:49.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:19:49.828 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:19:49.867 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:19:49.890 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:49.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:49.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:19:49.992 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:19:50.019 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:50.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:50.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:19:50.121 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:19:50.145 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:50.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:50.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:19:50.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:50 smithi090 ceph-mon[41670]: 5.11 scrub starts 2024-04-14T15:19:50.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:50 smithi090 ceph-mon[41670]: 5.11 scrub ok 2024-04-14T15:19:50.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:50 smithi090 ceph-mon[41670]: pgmap v686: 258 pgs: 258 active+clean; 8.4 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 767 B/s wr, 3 op/s 2024-04-14T15:19:50.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:50 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:19:50.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:50 smithi090 ceph-mon[30399]: 5.11 scrub starts 2024-04-14T15:19:50.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:50 smithi090 ceph-mon[30399]: 5.11 scrub ok 2024-04-14T15:19:50.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:50 smithi090 ceph-mon[30399]: pgmap v686: 258 pgs: 258 active+clean; 8.4 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 767 B/s wr, 3 op/s 2024-04-14T15:19:50.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:50 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:19:50.247 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:50.270 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:50.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:50.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:50.372 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:19:50.395 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:50.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:50.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:19:50.497 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:19:50.525 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:50.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:50.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:19:50.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:50 smithi092 ceph-mon[33966]: 5.11 scrub starts 2024-04-14T15:19:50.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:50 smithi092 ceph-mon[33966]: 5.11 scrub ok 2024-04-14T15:19:50.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:50 smithi092 ceph-mon[33966]: pgmap v686: 258 pgs: 258 active+clean; 8.4 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 767 B/s wr, 3 op/s 2024-04-14T15:19:50.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:50 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:19:50.627 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:19:50.653 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:50.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:50.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:19:50.755 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:50.778 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:50.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:50.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:50.880 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:19:50.903 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:50.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:50.904 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:19:51.004 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:51.029 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:51.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:51.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:51.131 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:51.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:51.155 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:51.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:51.251 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:51 smithi090 ceph-mon[41670]: 5.18 scrub starts 2024-04-14T15:19:51.251 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:51 smithi090 ceph-mon[41670]: 5.18 scrub ok 2024-04-14T15:19:51.251 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:51 smithi090 ceph-mon[41670]: 4.4 scrub starts 2024-04-14T15:19:51.251 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:51 smithi090 ceph-mon[41670]: 4.4 scrub ok 2024-04-14T15:19:51.251 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:51 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:19:51.251 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:51 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:19:51.251 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:51 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:19:51.251 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:51 smithi090 ceph-mon[41670]: osdmap e480: 8 total, 8 up, 8 in 2024-04-14T15:19:51.251 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:51 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:19:51.251 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:51 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:19:51.251 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:51 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-73"}]: dispatch 2024-04-14T15:19:51.252 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:51 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-73"}]: dispatch 2024-04-14T15:19:51.252 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:51 smithi090 ceph-mon[30399]: 5.18 scrub starts 2024-04-14T15:19:51.252 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:51 smithi090 ceph-mon[30399]: 5.18 scrub ok 2024-04-14T15:19:51.252 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:51 smithi090 ceph-mon[30399]: 4.4 scrub starts 2024-04-14T15:19:51.252 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:51 smithi090 ceph-mon[30399]: 4.4 scrub ok 2024-04-14T15:19:51.252 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:51 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:19:51.252 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:51 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:19:51.252 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:51 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:19:51.252 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:51 smithi090 ceph-mon[30399]: osdmap e480: 8 total, 8 up, 8 in 2024-04-14T15:19:51.252 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:51 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:19:51.252 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:51 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:19:51.252 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:51 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-73"}]: dispatch 2024-04-14T15:19:51.253 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:51 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-73"}]: dispatch 2024-04-14T15:19:51.256 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:19:51.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:51.280 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:51.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:19:51.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.351+0000 7fbc19e35700 1 -- 172.21.15.90:0/4007019542 >> v1:172.21.15.90:6789/0 conn(0x7fbc14103630 legacy=0x7fbc14103a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:51.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.351+0000 7fbc19e35700 1 -- 172.21.15.90:0/4007019542 shutdown_connections 2024-04-14T15:19:51.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.351+0000 7fbc19e35700 1 -- 172.21.15.90:0/4007019542 >> 172.21.15.90:0/4007019542 conn(0x7fbc14076a70 msgr2=0x7fbc14076e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:19:51.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.351+0000 7fbc19e35700 1 -- 172.21.15.90:0/4007019542 shutdown_connections 2024-04-14T15:19:51.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.351+0000 7fbc19e35700 1 -- 172.21.15.90:0/4007019542 wait complete. 2024-04-14T15:19:51.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.351+0000 7fbc19e35700 1 librados: starting msgr at 2024-04-14T15:19:51.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.351+0000 7fbc19e35700 1 librados: starting objecter 2024-04-14T15:19:51.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.352+0000 7fbc19e35700 1 Processor -- start 2024-04-14T15:19:51.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.352+0000 7fbc19e35700 1 -- start start 2024-04-14T15:19:51.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.352+0000 7fbc19e35700 1 librados: setting wanted keys 2024-04-14T15:19:51.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.352+0000 7fbc19e35700 1 librados: calling monclient init 2024-04-14T15:19:51.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.352+0000 7fbc19e35700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbc141a9c60 con 0x7fbc14111630 2024-04-14T15:19:51.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.352+0000 7fbc19e35700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbc141aae80 con 0x7fbc14103630 2024-04-14T15:19:51.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.352+0000 7fbc19e35700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbc141ac0c0 con 0x7fbc1410da00 2024-04-14T15:19:51.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.352+0000 7fbc12ffd700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7fbc1410da00 0x7fbc141a4ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:53682/0 (socket says 172.21.15.90:53682) 2024-04-14T15:19:51.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.352+0000 7fbc12ffd700 1 -- 172.21.15.90:0/3838575409 learned_addr learned my addr 172.21.15.90:0/3838575409 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:19:51.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.352+0000 7fbc13fff700 1 --1- 172.21.15.90:0/3838575409 >> v1:172.21.15.90:6789/0 conn(0x7fbc14111630 0x7fbc141a8320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:54898/0 (socket says 172.21.15.90:54898) 2024-04-14T15:19:51.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.352+0000 7fbc10ff9700 10 client.?.objecter ms_handle_connect 0x7fbc14111630 2024-04-14T15:19:51.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.352+0000 7fbc10ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:19:51.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.352+0000 7fbc10ff9700 10 client.?.objecter ms_handle_connect 0x7fbc1410da00 2024-04-14T15:19:51.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.352+0000 7fbc10ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:19:51.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.352+0000 7fbc10ff9700 1 -- 172.21.15.90:0/3838575409 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3915999586 0 0) 0x7fbc141a9c60 con 0x7fbc14111630 2024-04-14T15:19:51.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.352+0000 7fbc10ff9700 1 -- 172.21.15.90:0/3838575409 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbbfc0031f0 con 0x7fbc14111630 2024-04-14T15:19:51.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.352+0000 7fbc10ff9700 10 client.?.objecter ms_handle_connect 0x7fbc14103630 2024-04-14T15:19:51.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.352+0000 7fbc10ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:19:51.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.352+0000 7fbc10ff9700 1 -- 172.21.15.90:0/3838575409 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3546870910 0 0) 0x7fbc141ac0c0 con 0x7fbc1410da00 2024-04-14T15:19:51.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.352+0000 7fbc10ff9700 1 -- 172.21.15.90:0/3838575409 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc141a9c60 con 0x7fbc1410da00 2024-04-14T15:19:51.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.353+0000 7fbc10ff9700 1 -- 172.21.15.90:0/3838575409 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1262077175 0 0) 0x7fbc141aae80 con 0x7fbc14103630 2024-04-14T15:19:51.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.353+0000 7fbc10ff9700 1 -- 172.21.15.90:0/3838575409 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc141ac0c0 con 0x7fbc14103630 2024-04-14T15:19:51.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.353+0000 7fbc10ff9700 1 -- 172.21.15.90:0/3838575409 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2379153798 0 0) 0x7fbbfc0031f0 con 0x7fbc14111630 2024-04-14T15:19:51.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.353+0000 7fbc10ff9700 1 -- 172.21.15.90:0/3838575409 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbc141aae80 con 0x7fbc14111630 2024-04-14T15:19:51.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.353+0000 7fbc10ff9700 1 -- 172.21.15.90:0/3838575409 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fbc04001870 con 0x7fbc14111630 2024-04-14T15:19:51.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.353+0000 7fbc10ff9700 1 -- 172.21.15.90:0/3838575409 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1604208163 0 0) 0x7fbc141a9c60 con 0x7fbc1410da00 2024-04-14T15:19:51.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.353+0000 7fbc10ff9700 1 -- 172.21.15.90:0/3838575409 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbbfc0031f0 con 0x7fbc1410da00 2024-04-14T15:19:51.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.353+0000 7fbc10ff9700 1 -- 172.21.15.90:0/3838575409 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2913534983 0 0) 0x7fbc141ac0c0 con 0x7fbc14103630 2024-04-14T15:19:51.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.353+0000 7fbc10ff9700 1 -- 172.21.15.90:0/3838575409 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbc141a9c60 con 0x7fbc14103630 2024-04-14T15:19:51.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.353+0000 7fbc10ff9700 1 -- 172.21.15.90:0/3838575409 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fbc08001db0 con 0x7fbc1410da00 2024-04-14T15:19:51.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.353+0000 7fbc10ff9700 1 -- 172.21.15.90:0/3838575409 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fbc0c004400 con 0x7fbc14103630 2024-04-14T15:19:51.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.353+0000 7fbc10ff9700 1 -- 172.21.15.90:0/3838575409 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 419228130 0 0) 0x7fbc141aae80 con 0x7fbc14111630 2024-04-14T15:19:51.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.353+0000 7fbc10ff9700 1 -- 172.21.15.90:0/3838575409 >> v1:172.21.15.90:6790/0 conn(0x7fbc1410da00 legacy=0x7fbc141a4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:51.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.353+0000 7fbc10ff9700 1 -- 172.21.15.90:0/3838575409 >> v1:172.21.15.92:6789/0 conn(0x7fbc14103630 legacy=0x7fbc14101990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:51.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.353+0000 7fbc10ff9700 1 -- 172.21.15.90:0/3838575409 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc141ad300 con 0x7fbc14111630 2024-04-14T15:19:51.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.353+0000 7fbc19e35700 1 -- 172.21.15.90:0/3838575409 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbc141ac2f0 con 0x7fbc14111630 2024-04-14T15:19:51.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.353+0000 7fbc19e35700 10 client.37425.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:19:51.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.353+0000 7fbc19e35700 1 -- 172.21.15.90:0/3838575409 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fbc141ac7b0 con 0x7fbc14111630 2024-04-14T15:19:51.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.353+0000 7fbc19e35700 1 librados: init done 2024-04-14T15:19:51.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.353+0000 7fbc10ff9700 1 -- 172.21.15.90:0/3838575409 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fbc04003b40 con 0x7fbc14111630 2024-04-14T15:19:51.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.353+0000 7fbc10ff9700 1 -- 172.21.15.90:0/3838575409 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fbc04005d10 con 0x7fbc14111630 2024-04-14T15:19:51.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.354+0000 7fbc19e35700 1 -- 172.21.15.90:0/3838575409 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fbc00005100 con 0x7fbc14111630 2024-04-14T15:19:51.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.354+0000 7fbc10ff9700 1 -- 172.21.15.90:0/3838575409 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fbc0401c720 con 0x7fbc14111630 2024-04-14T15:19:51.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.355+0000 7fbc10ff9700 1 -- 172.21.15.90:0/3838575409 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(480..480 src has 51..480) ==== 8197+0+0 (unknown 1607389665 0 0) 0x7fbc0408a930 con 0x7fbc14111630 2024-04-14T15:19:51.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.355+0000 7fbc10ff9700 10 client.37425.objecter ms_dispatch 0x7fbc14001550 osd_map(480..480 src has 51..480) 2024-04-14T15:19:51.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.355+0000 7fbc10ff9700 3 client.37425.objecter handle_osd_map got epochs [480,480] > 0 2024-04-14T15:19:51.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.355+0000 7fbc10ff9700 3 client.37425.objecter handle_osd_map decoding full epoch 480 2024-04-14T15:19:51.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.355+0000 7fbc10ff9700 20 client.37425.objecter dump_active .. 0 homeless 2024-04-14T15:19:51.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.355+0000 7fbc10ff9700 1 -- 172.21.15.90:0/3838575409 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v480) ==== 54+0+16 (unknown 1016706674 0 2420000192) 0x7fbc0408ad20 con 0x7fbc14111630 2024-04-14T15:19:51.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.355+0000 7fbc10ff9700 10 client.37425.objecter ms_handle_connect 0x7fbbfc071430 2024-04-14T15:19:51.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.355+0000 7fbc19e35700 10 client.37425.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:19:51.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.355+0000 7fbc19e35700 15 client.37425.objecter _session_command_op_assign -1 1 2024-04-14T15:19:51.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.355+0000 7fbc19e35700 20 client.37425.objecter _get_session s=0x7fbc000053e0 osd=0 3 2024-04-14T15:19:51.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.355+0000 7fbc19e35700 20 client.37425.objecter put_session s=0x7fbc000053e0 osd=0 3 2024-04-14T15:19:51.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.355+0000 7fbc19e35700 20 client.37425.objecter _get_session s=0x7fbc000053e0 osd=0 3 2024-04-14T15:19:51.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.355+0000 7fbc19e35700 15 client.37425.objecter _session_command_op_remove -1 1 2024-04-14T15:19:51.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.355+0000 7fbc19e35700 20 client.37425.objecter get_session s=0x7fbc000053e0 osd=0 3 2024-04-14T15:19:51.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.355+0000 7fbc19e35700 15 client.37425.objecter _session_command_op_assign 0 1 2024-04-14T15:19:51.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.355+0000 7fbc19e35700 20 client.37425.objecter put_session s=0x7fbc000053e0 osd=0 4 2024-04-14T15:19:51.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.355+0000 7fbc19e35700 10 client.37425.objecter _send_command 1 2024-04-14T15:19:51.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.355+0000 7fbc19e35700 1 -- 172.21.15.90:0/3838575409 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbc00004b70 con 0x7fbc00004730 2024-04-14T15:19:51.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.356+0000 7fbc10ff9700 10 client.37425.objecter ms_handle_connect 0x7fbc00004730 2024-04-14T15:19:51.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.356+0000 7fbc10ff9700 1 -- 172.21.15.90:0/3838575409 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fbc00004b70 con 0x7fbc00004730 2024-04-14T15:19:51.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.356+0000 7fbc10ff9700 10 client.37425.objecter ms_dispatch 0x7fbc14001550 command_reply(tid 1: 0 ) 2024-04-14T15:19:51.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.356+0000 7fbc10ff9700 10 client.37425.objecter _finish_command 1 = system:0 2024-04-14T15:19:51.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.356+0000 7fbc10ff9700 20 client.37425.objecter put_session s=0x7fbc000053e0 osd=0 4 2024-04-14T15:19:51.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.356+0000 7fbc10ff9700 15 client.37425.objecter _session_command_op_remove 0 1 2024-04-14T15:19:51.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.373+0000 7fbc19e35700 10 client.37425.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:19:51.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.373+0000 7fbc19e35700 15 client.37425.objecter _session_command_op_assign -1 2 2024-04-14T15:19:51.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.373+0000 7fbc19e35700 20 client.37425.objecter _get_session s=0x7fbc000053e0 osd=0 3 2024-04-14T15:19:51.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.373+0000 7fbc19e35700 20 client.37425.objecter put_session s=0x7fbc000053e0 osd=0 3 2024-04-14T15:19:51.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.373+0000 7fbc19e35700 20 client.37425.objecter _get_session s=0x7fbc000053e0 osd=0 3 2024-04-14T15:19:51.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.373+0000 7fbc19e35700 15 client.37425.objecter _session_command_op_remove -1 2 2024-04-14T15:19:51.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.373+0000 7fbc19e35700 20 client.37425.objecter get_session s=0x7fbc000053e0 osd=0 3 2024-04-14T15:19:51.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.373+0000 7fbc19e35700 15 client.37425.objecter _session_command_op_assign 0 2 2024-04-14T15:19:51.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.373+0000 7fbc19e35700 20 client.37425.objecter put_session s=0x7fbc000053e0 osd=0 4 2024-04-14T15:19:51.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.373+0000 7fbc19e35700 10 client.37425.objecter _send_command 2 2024-04-14T15:19:51.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.373+0000 7fbc19e35700 1 -- 172.21.15.90:0/3838575409 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fbc000079f0 con 0x7fbc00004730 2024-04-14T15:19:51.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.374+0000 7fbc10ff9700 1 -- 172.21.15.90:0/3838575409 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fbc000079f0 con 0x7fbc00004730 2024-04-14T15:19:51.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.374+0000 7fbc10ff9700 10 client.37425.objecter ms_dispatch 0x7fbc14001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:19:51.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.374+0000 7fbc10ff9700 10 client.37425.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:19:51.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.374+0000 7fbc10ff9700 20 client.37425.objecter put_session s=0x7fbc000053e0 osd=0 4 2024-04-14T15:19:51.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.374+0000 7fbc10ff9700 15 client.37425.objecter _session_command_op_remove 0 2 2024-04-14T15:19:51.375 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:19:51.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.375+0000 7fbc19e35700 10 client.37425.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:19:51.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.375+0000 7fbc19e35700 15 client.37425.objecter _session_command_op_assign -1 3 2024-04-14T15:19:51.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.375+0000 7fbc19e35700 20 client.37425.objecter _get_session s=0x7fbc00004130 osd=1 3 2024-04-14T15:19:51.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.375+0000 7fbc19e35700 20 client.37425.objecter put_session s=0x7fbc00004130 osd=1 3 2024-04-14T15:19:51.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.375+0000 7fbc19e35700 20 client.37425.objecter _get_session s=0x7fbc00004130 osd=1 3 2024-04-14T15:19:51.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.375+0000 7fbc19e35700 15 client.37425.objecter _session_command_op_remove -1 3 2024-04-14T15:19:51.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.375+0000 7fbc19e35700 20 client.37425.objecter get_session s=0x7fbc00004130 osd=1 3 2024-04-14T15:19:51.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.375+0000 7fbc19e35700 15 client.37425.objecter _session_command_op_assign 1 3 2024-04-14T15:19:51.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.375+0000 7fbc19e35700 20 client.37425.objecter put_session s=0x7fbc00004130 osd=1 4 2024-04-14T15:19:51.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.375+0000 7fbc19e35700 10 client.37425.objecter _send_command 3 2024-04-14T15:19:51.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.375+0000 7fbc19e35700 1 -- 172.21.15.90:0/3838575409 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fbc000022a0 con 0x7fbc00004310 2024-04-14T15:19:51.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.375+0000 7fbc10ff9700 10 client.37425.objecter ms_handle_connect 0x7fbc00004310 2024-04-14T15:19:51.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.376+0000 7fbc10ff9700 1 -- 172.21.15.90:0/3838575409 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fbc000022a0 con 0x7fbc00004310 2024-04-14T15:19:51.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.376+0000 7fbc10ff9700 10 client.37425.objecter ms_dispatch 0x7fbc14001550 command_reply(tid 3: 0 ) 2024-04-14T15:19:51.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.376+0000 7fbc10ff9700 10 client.37425.objecter _finish_command 3 = system:0 2024-04-14T15:19:51.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.376+0000 7fbc10ff9700 20 client.37425.objecter put_session s=0x7fbc00004130 osd=1 4 2024-04-14T15:19:51.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.376+0000 7fbc10ff9700 15 client.37425.objecter _session_command_op_remove 1 3 2024-04-14T15:19:51.382 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:19:51.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.393+0000 7fbc19e35700 10 client.37425.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:19:51.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.393+0000 7fbc19e35700 15 client.37425.objecter _session_command_op_assign -1 4 2024-04-14T15:19:51.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.393+0000 7fbc19e35700 20 client.37425.objecter _get_session s=0x7fbc00004130 osd=1 3 2024-04-14T15:19:51.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.393+0000 7fbc19e35700 20 client.37425.objecter put_session s=0x7fbc00004130 osd=1 3 2024-04-14T15:19:51.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.393+0000 7fbc19e35700 20 client.37425.objecter _get_session s=0x7fbc00004130 osd=1 3 2024-04-14T15:19:51.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.393+0000 7fbc19e35700 15 client.37425.objecter _session_command_op_remove -1 4 2024-04-14T15:19:51.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.393+0000 7fbc19e35700 20 client.37425.objecter get_session s=0x7fbc00004130 osd=1 3 2024-04-14T15:19:51.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.393+0000 7fbc19e35700 15 client.37425.objecter _session_command_op_assign 1 4 2024-04-14T15:19:51.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.393+0000 7fbc19e35700 20 client.37425.objecter put_session s=0x7fbc00004130 osd=1 4 2024-04-14T15:19:51.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.393+0000 7fbc19e35700 10 client.37425.objecter _send_command 4 2024-04-14T15:19:51.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.393+0000 7fbc19e35700 1 -- 172.21.15.90:0/3838575409 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fbc00001720 con 0x7fbc00004310 2024-04-14T15:19:51.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.394+0000 7fbc10ff9700 1 -- 172.21.15.90:0/3838575409 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fbc00001720 con 0x7fbc00004310 2024-04-14T15:19:51.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.394+0000 7fbc10ff9700 10 client.37425.objecter ms_dispatch 0x7fbc14001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:19:51.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.394+0000 7fbc10ff9700 10 client.37425.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:19:51.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.394+0000 7fbc10ff9700 20 client.37425.objecter put_session s=0x7fbc00004130 osd=1 4 2024-04-14T15:19:51.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.394+0000 7fbc10ff9700 15 client.37425.objecter _session_command_op_remove 1 4 2024-04-14T15:19:51.395 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:19:51.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.395+0000 7fbc19e35700 10 client.37425.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:19:51.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.395+0000 7fbc19e35700 15 client.37425.objecter _session_command_op_assign -1 5 2024-04-14T15:19:51.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.395+0000 7fbc19e35700 20 client.37425.objecter _get_session s=0x7fbc00001980 osd=2 3 2024-04-14T15:19:51.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.395+0000 7fbc19e35700 20 client.37425.objecter put_session s=0x7fbc00001980 osd=2 3 2024-04-14T15:19:51.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.395+0000 7fbc19e35700 20 client.37425.objecter _get_session s=0x7fbc00001980 osd=2 3 2024-04-14T15:19:51.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.395+0000 7fbc19e35700 15 client.37425.objecter _session_command_op_remove -1 5 2024-04-14T15:19:51.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.395+0000 7fbc19e35700 20 client.37425.objecter get_session s=0x7fbc00001980 osd=2 3 2024-04-14T15:19:51.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.395+0000 7fbc19e35700 15 client.37425.objecter _session_command_op_assign 2 5 2024-04-14T15:19:51.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.395+0000 7fbc19e35700 20 client.37425.objecter put_session s=0x7fbc00001980 osd=2 4 2024-04-14T15:19:51.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.395+0000 7fbc19e35700 10 client.37425.objecter _send_command 5 2024-04-14T15:19:51.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.395+0000 7fbc19e35700 1 -- 172.21.15.90:0/3838575409 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fbc00011170 con 0x7fbc0000d5d0 2024-04-14T15:19:51.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.395+0000 7fbc10ff9700 10 client.37425.objecter ms_handle_connect 0x7fbc0000d5d0 2024-04-14T15:19:51.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.395+0000 7fbc10ff9700 1 -- 172.21.15.90:0/3838575409 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fbc00011170 con 0x7fbc0000d5d0 2024-04-14T15:19:51.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.395+0000 7fbc10ff9700 10 client.37425.objecter ms_dispatch 0x7fbc14001550 command_reply(tid 5: 0 ) 2024-04-14T15:19:51.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.395+0000 7fbc10ff9700 10 client.37425.objecter _finish_command 5 = system:0 2024-04-14T15:19:51.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.395+0000 7fbc10ff9700 20 client.37425.objecter put_session s=0x7fbc00001980 osd=2 4 2024-04-14T15:19:51.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.395+0000 7fbc10ff9700 15 client.37425.objecter _session_command_op_remove 2 5 2024-04-14T15:19:51.405 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:51.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:51.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:19:51.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.412+0000 7fbc19e35700 10 client.37425.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:19:51.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.412+0000 7fbc19e35700 15 client.37425.objecter _session_command_op_assign -1 6 2024-04-14T15:19:51.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.412+0000 7fbc19e35700 20 client.37425.objecter _get_session s=0x7fbc00001980 osd=2 3 2024-04-14T15:19:51.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.412+0000 7fbc19e35700 20 client.37425.objecter put_session s=0x7fbc00001980 osd=2 3 2024-04-14T15:19:51.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.412+0000 7fbc19e35700 20 client.37425.objecter _get_session s=0x7fbc00001980 osd=2 3 2024-04-14T15:19:51.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.412+0000 7fbc19e35700 15 client.37425.objecter _session_command_op_remove -1 6 2024-04-14T15:19:51.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.412+0000 7fbc19e35700 20 client.37425.objecter get_session s=0x7fbc00001980 osd=2 3 2024-04-14T15:19:51.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.412+0000 7fbc19e35700 15 client.37425.objecter _session_command_op_assign 2 6 2024-04-14T15:19:51.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.412+0000 7fbc19e35700 20 client.37425.objecter put_session s=0x7fbc00001980 osd=2 4 2024-04-14T15:19:51.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.412+0000 7fbc19e35700 10 client.37425.objecter _send_command 6 2024-04-14T15:19:51.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.412+0000 7fbc19e35700 1 -- 172.21.15.90:0/3838575409 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fbc00008100 con 0x7fbc0000d5d0 2024-04-14T15:19:51.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.413+0000 7fbc10ff9700 1 -- 172.21.15.90:0/3838575409 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fbc00008100 con 0x7fbc0000d5d0 2024-04-14T15:19:51.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.413+0000 7fbc10ff9700 10 client.37425.objecter ms_dispatch 0x7fbc14001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:19:51.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.413+0000 7fbc10ff9700 10 client.37425.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:19:51.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.413+0000 7fbc10ff9700 20 client.37425.objecter put_session s=0x7fbc00001980 osd=2 4 2024-04-14T15:19:51.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.413+0000 7fbc10ff9700 15 client.37425.objecter _session_command_op_remove 2 6 2024-04-14T15:19:51.414 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:19:51.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.413+0000 7fbc19e35700 10 client.37425.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:19:51.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.413+0000 7fbc19e35700 15 client.37425.objecter _session_command_op_assign -1 7 2024-04-14T15:19:51.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.413+0000 7fbc19e35700 20 client.37425.objecter _get_session s=0x7fbc0000cc20 osd=3 3 2024-04-14T15:19:51.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.413+0000 7fbc19e35700 20 client.37425.objecter put_session s=0x7fbc0000cc20 osd=3 3 2024-04-14T15:19:51.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.413+0000 7fbc19e35700 20 client.37425.objecter _get_session s=0x7fbc0000cc20 osd=3 3 2024-04-14T15:19:51.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.413+0000 7fbc19e35700 15 client.37425.objecter _session_command_op_remove -1 7 2024-04-14T15:19:51.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.413+0000 7fbc19e35700 20 client.37425.objecter get_session s=0x7fbc0000cc20 osd=3 3 2024-04-14T15:19:51.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.413+0000 7fbc19e35700 15 client.37425.objecter _session_command_op_assign 3 7 2024-04-14T15:19:51.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.413+0000 7fbc19e35700 20 client.37425.objecter put_session s=0x7fbc0000cc20 osd=3 4 2024-04-14T15:19:51.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.413+0000 7fbc19e35700 10 client.37425.objecter _send_command 7 2024-04-14T15:19:51.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.413+0000 7fbc19e35700 1 -- 172.21.15.90:0/3838575409 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fbc00015fb0 con 0x7fbc000124a0 2024-04-14T15:19:51.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.414+0000 7fbc10ff9700 10 client.37425.objecter ms_handle_connect 0x7fbc000124a0 2024-04-14T15:19:51.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.414+0000 7fbc10ff9700 1 -- 172.21.15.90:0/3838575409 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fbc00015fb0 con 0x7fbc000124a0 2024-04-14T15:19:51.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.414+0000 7fbc10ff9700 10 client.37425.objecter ms_dispatch 0x7fbc14001550 command_reply(tid 7: 0 ) 2024-04-14T15:19:51.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.414+0000 7fbc10ff9700 10 client.37425.objecter _finish_command 7 = system:0 2024-04-14T15:19:51.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.414+0000 7fbc10ff9700 20 client.37425.objecter put_session s=0x7fbc0000cc20 osd=3 4 2024-04-14T15:19:51.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.414+0000 7fbc10ff9700 15 client.37425.objecter _session_command_op_remove 3 7 2024-04-14T15:19:51.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.431+0000 7fbc19e35700 10 client.37425.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:19:51.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.431+0000 7fbc19e35700 15 client.37425.objecter _session_command_op_assign -1 8 2024-04-14T15:19:51.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.431+0000 7fbc19e35700 20 client.37425.objecter _get_session s=0x7fbc0000cc20 osd=3 3 2024-04-14T15:19:51.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.431+0000 7fbc19e35700 20 client.37425.objecter put_session s=0x7fbc0000cc20 osd=3 3 2024-04-14T15:19:51.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.431+0000 7fbc19e35700 20 client.37425.objecter _get_session s=0x7fbc0000cc20 osd=3 3 2024-04-14T15:19:51.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.431+0000 7fbc19e35700 15 client.37425.objecter _session_command_op_remove -1 8 2024-04-14T15:19:51.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.431+0000 7fbc19e35700 20 client.37425.objecter get_session s=0x7fbc0000cc20 osd=3 3 2024-04-14T15:19:51.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.431+0000 7fbc19e35700 15 client.37425.objecter _session_command_op_assign 3 8 2024-04-14T15:19:51.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.431+0000 7fbc19e35700 20 client.37425.objecter put_session s=0x7fbc0000cc20 osd=3 4 2024-04-14T15:19:51.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.431+0000 7fbc19e35700 10 client.37425.objecter _send_command 8 2024-04-14T15:19:51.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.431+0000 7fbc19e35700 1 -- 172.21.15.90:0/3838575409 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fbc0000c700 con 0x7fbc000124a0 2024-04-14T15:19:51.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.432+0000 7fbc10ff9700 1 -- 172.21.15.90:0/3838575409 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fbc0000c700 con 0x7fbc000124a0 2024-04-14T15:19:51.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.432+0000 7fbc10ff9700 10 client.37425.objecter ms_dispatch 0x7fbc14001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:19:51.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.432+0000 7fbc10ff9700 10 client.37425.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:19:51.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.432+0000 7fbc10ff9700 20 client.37425.objecter put_session s=0x7fbc0000cc20 osd=3 4 2024-04-14T15:19:51.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.432+0000 7fbc10ff9700 15 client.37425.objecter _session_command_op_remove 3 8 2024-04-14T15:19:51.433 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:19:51.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.433+0000 7fbc19e35700 10 client.37425.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:19:51.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.433+0000 7fbc19e35700 15 client.37425.objecter _session_command_op_assign -1 9 2024-04-14T15:19:51.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.433+0000 7fbc19e35700 20 client.37425.objecter _get_session s=0x7fbc0000c4b0 osd=4 3 2024-04-14T15:19:51.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.433+0000 7fbc19e35700 20 client.37425.objecter put_session s=0x7fbc0000c4b0 osd=4 3 2024-04-14T15:19:51.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.433+0000 7fbc19e35700 20 client.37425.objecter _get_session s=0x7fbc0000c4b0 osd=4 3 2024-04-14T15:19:51.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.433+0000 7fbc19e35700 15 client.37425.objecter _session_command_op_remove -1 9 2024-04-14T15:19:51.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.433+0000 7fbc19e35700 20 client.37425.objecter get_session s=0x7fbc0000c4b0 osd=4 3 2024-04-14T15:19:51.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.433+0000 7fbc19e35700 15 client.37425.objecter _session_command_op_assign 4 9 2024-04-14T15:19:51.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.433+0000 7fbc19e35700 20 client.37425.objecter put_session s=0x7fbc0000c4b0 osd=4 4 2024-04-14T15:19:51.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.433+0000 7fbc19e35700 10 client.37425.objecter _send_command 9 2024-04-14T15:19:51.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.433+0000 7fbc19e35700 1 -- 172.21.15.90:0/3838575409 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fbc0000c6e0 con 0x7fbc00012000 2024-04-14T15:19:51.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.433+0000 7fbc10ff9700 10 client.37425.objecter ms_handle_connect 0x7fbc00012000 2024-04-14T15:19:51.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.434+0000 7fbc10ff9700 1 -- 172.21.15.90:0/3838575409 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fbc0000c6e0 con 0x7fbc00012000 2024-04-14T15:19:51.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.434+0000 7fbc10ff9700 10 client.37425.objecter ms_dispatch 0x7fbc14001550 command_reply(tid 9: 0 ) 2024-04-14T15:19:51.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.434+0000 7fbc10ff9700 10 client.37425.objecter _finish_command 9 = system:0 2024-04-14T15:19:51.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.434+0000 7fbc10ff9700 20 client.37425.objecter put_session s=0x7fbc0000c4b0 osd=4 4 2024-04-14T15:19:51.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.434+0000 7fbc10ff9700 15 client.37425.objecter _session_command_op_remove 4 9 2024-04-14T15:19:51.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.450+0000 7fbc19e35700 10 client.37425.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:19:51.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.450+0000 7fbc19e35700 15 client.37425.objecter _session_command_op_assign -1 10 2024-04-14T15:19:51.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.450+0000 7fbc19e35700 20 client.37425.objecter _get_session s=0x7fbc0000c4b0 osd=4 3 2024-04-14T15:19:51.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.450+0000 7fbc19e35700 20 client.37425.objecter put_session s=0x7fbc0000c4b0 osd=4 3 2024-04-14T15:19:51.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.450+0000 7fbc19e35700 20 client.37425.objecter _get_session s=0x7fbc0000c4b0 osd=4 3 2024-04-14T15:19:51.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.450+0000 7fbc19e35700 15 client.37425.objecter _session_command_op_remove -1 10 2024-04-14T15:19:51.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.450+0000 7fbc19e35700 20 client.37425.objecter get_session s=0x7fbc0000c4b0 osd=4 3 2024-04-14T15:19:51.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.450+0000 7fbc19e35700 15 client.37425.objecter _session_command_op_assign 4 10 2024-04-14T15:19:51.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.450+0000 7fbc19e35700 20 client.37425.objecter put_session s=0x7fbc0000c4b0 osd=4 4 2024-04-14T15:19:51.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.450+0000 7fbc19e35700 10 client.37425.objecter _send_command 10 2024-04-14T15:19:51.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.450+0000 7fbc19e35700 1 -- 172.21.15.90:0/3838575409 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fbc0001c210 con 0x7fbc00012000 2024-04-14T15:19:51.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.452+0000 7fbc10ff9700 1 -- 172.21.15.90:0/3838575409 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fbc0001c210 con 0x7fbc00012000 2024-04-14T15:19:51.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.452+0000 7fbc10ff9700 10 client.37425.objecter ms_dispatch 0x7fbc14001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:19:51.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.452+0000 7fbc10ff9700 10 client.37425.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:19:51.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.452+0000 7fbc10ff9700 20 client.37425.objecter put_session s=0x7fbc0000c4b0 osd=4 4 2024-04-14T15:19:51.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.452+0000 7fbc10ff9700 15 client.37425.objecter _session_command_op_remove 4 10 2024-04-14T15:19:51.452 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:19:51.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.452+0000 7fbc19e35700 10 client.37425.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:19:51.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.452+0000 7fbc19e35700 15 client.37425.objecter _session_command_op_assign -1 11 2024-04-14T15:19:51.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.452+0000 7fbc19e35700 20 client.37425.objecter _get_session s=0x7fbc0001c210 osd=5 3 2024-04-14T15:19:51.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.452+0000 7fbc19e35700 20 client.37425.objecter put_session s=0x7fbc0001c210 osd=5 3 2024-04-14T15:19:51.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.452+0000 7fbc19e35700 20 client.37425.objecter _get_session s=0x7fbc0001c210 osd=5 3 2024-04-14T15:19:51.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.452+0000 7fbc19e35700 15 client.37425.objecter _session_command_op_remove -1 11 2024-04-14T15:19:51.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.452+0000 7fbc19e35700 20 client.37425.objecter get_session s=0x7fbc0001c210 osd=5 3 2024-04-14T15:19:51.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.452+0000 7fbc19e35700 15 client.37425.objecter _session_command_op_assign 5 11 2024-04-14T15:19:51.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.452+0000 7fbc19e35700 20 client.37425.objecter put_session s=0x7fbc0001c210 osd=5 4 2024-04-14T15:19:51.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.452+0000 7fbc19e35700 10 client.37425.objecter _send_command 11 2024-04-14T15:19:51.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.452+0000 7fbc19e35700 1 -- 172.21.15.90:0/3838575409 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fbc0001cdc0 con 0x7fbc0001c930 2024-04-14T15:19:51.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.452+0000 7fbc10ff9700 10 client.37425.objecter ms_handle_connect 0x7fbc0001c930 2024-04-14T15:19:51.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.453+0000 7fbc10ff9700 1 -- 172.21.15.90:0/3838575409 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fbc0001cdc0 con 0x7fbc0001c930 2024-04-14T15:19:51.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.453+0000 7fbc10ff9700 10 client.37425.objecter ms_dispatch 0x7fbc14001550 command_reply(tid 11: 0 ) 2024-04-14T15:19:51.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.453+0000 7fbc10ff9700 10 client.37425.objecter _finish_command 11 = system:0 2024-04-14T15:19:51.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.453+0000 7fbc10ff9700 20 client.37425.objecter put_session s=0x7fbc0001c210 osd=5 4 2024-04-14T15:19:51.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.453+0000 7fbc10ff9700 15 client.37425.objecter _session_command_op_remove 5 11 2024-04-14T15:19:51.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.469+0000 7fbc19e35700 10 client.37425.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:19:51.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.469+0000 7fbc19e35700 15 client.37425.objecter _session_command_op_assign -1 12 2024-04-14T15:19:51.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.469+0000 7fbc19e35700 20 client.37425.objecter _get_session s=0x7fbc0001c210 osd=5 3 2024-04-14T15:19:51.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.469+0000 7fbc19e35700 20 client.37425.objecter put_session s=0x7fbc0001c210 osd=5 3 2024-04-14T15:19:51.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.469+0000 7fbc19e35700 20 client.37425.objecter _get_session s=0x7fbc0001c210 osd=5 3 2024-04-14T15:19:51.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.469+0000 7fbc19e35700 15 client.37425.objecter _session_command_op_remove -1 12 2024-04-14T15:19:51.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.469+0000 7fbc19e35700 20 client.37425.objecter get_session s=0x7fbc0001c210 osd=5 3 2024-04-14T15:19:51.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.469+0000 7fbc19e35700 15 client.37425.objecter _session_command_op_assign 5 12 2024-04-14T15:19:51.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.469+0000 7fbc19e35700 20 client.37425.objecter put_session s=0x7fbc0001c210 osd=5 4 2024-04-14T15:19:51.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.469+0000 7fbc19e35700 10 client.37425.objecter _send_command 12 2024-04-14T15:19:51.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.469+0000 7fbc19e35700 1 -- 172.21.15.90:0/3838575409 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fbc0001cdc0 con 0x7fbc0001c930 2024-04-14T15:19:51.471 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:19:51.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.471+0000 7fbc10ff9700 1 -- 172.21.15.90:0/3838575409 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fbc0001cdc0 con 0x7fbc0001c930 2024-04-14T15:19:51.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.471+0000 7fbc10ff9700 10 client.37425.objecter ms_dispatch 0x7fbc14001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:19:51.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.471+0000 7fbc10ff9700 10 client.37425.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:19:51.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.471+0000 7fbc10ff9700 20 client.37425.objecter put_session s=0x7fbc0001c210 osd=5 4 2024-04-14T15:19:51.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.471+0000 7fbc10ff9700 15 client.37425.objecter _session_command_op_remove 5 12 2024-04-14T15:19:51.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.471+0000 7fbc19e35700 10 client.37425.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:19:51.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.471+0000 7fbc19e35700 15 client.37425.objecter _session_command_op_assign -1 13 2024-04-14T15:19:51.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.471+0000 7fbc19e35700 20 client.37425.objecter _get_session s=0x7fbc0001cdc0 osd=6 3 2024-04-14T15:19:51.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.471+0000 7fbc19e35700 20 client.37425.objecter put_session s=0x7fbc0001cdc0 osd=6 3 2024-04-14T15:19:51.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.471+0000 7fbc19e35700 20 client.37425.objecter _get_session s=0x7fbc0001cdc0 osd=6 3 2024-04-14T15:19:51.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.471+0000 7fbc19e35700 15 client.37425.objecter _session_command_op_remove -1 13 2024-04-14T15:19:51.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.471+0000 7fbc19e35700 20 client.37425.objecter get_session s=0x7fbc0001cdc0 osd=6 3 2024-04-14T15:19:51.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.471+0000 7fbc19e35700 15 client.37425.objecter _session_command_op_assign 6 13 2024-04-14T15:19:51.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.471+0000 7fbc19e35700 20 client.37425.objecter put_session s=0x7fbc0001cdc0 osd=6 4 2024-04-14T15:19:51.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.471+0000 7fbc19e35700 10 client.37425.objecter _send_command 13 2024-04-14T15:19:51.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.471+0000 7fbc19e35700 1 -- 172.21.15.90:0/3838575409 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fbc00023410 con 0x7fbc0001f900 2024-04-14T15:19:51.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.472+0000 7fbc10ff9700 10 client.37425.objecter ms_handle_connect 0x7fbc0001f900 2024-04-14T15:19:51.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.472+0000 7fbc10ff9700 1 -- 172.21.15.90:0/3838575409 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fbc00023410 con 0x7fbc0001f900 2024-04-14T15:19:51.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.472+0000 7fbc10ff9700 10 client.37425.objecter ms_dispatch 0x7fbc14001550 command_reply(tid 13: 0 ) 2024-04-14T15:19:51.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.472+0000 7fbc10ff9700 10 client.37425.objecter _finish_command 13 = system:0 2024-04-14T15:19:51.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.472+0000 7fbc10ff9700 20 client.37425.objecter put_session s=0x7fbc0001cdc0 osd=6 4 2024-04-14T15:19:51.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.472+0000 7fbc10ff9700 15 client.37425.objecter _session_command_op_remove 6 13 2024-04-14T15:19:51.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.489+0000 7fbc19e35700 10 client.37425.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:19:51.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.489+0000 7fbc19e35700 15 client.37425.objecter _session_command_op_assign -1 14 2024-04-14T15:19:51.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.489+0000 7fbc19e35700 20 client.37425.objecter _get_session s=0x7fbc0001cdc0 osd=6 3 2024-04-14T15:19:51.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.489+0000 7fbc19e35700 20 client.37425.objecter put_session s=0x7fbc0001cdc0 osd=6 3 2024-04-14T15:19:51.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.489+0000 7fbc19e35700 20 client.37425.objecter _get_session s=0x7fbc0001cdc0 osd=6 3 2024-04-14T15:19:51.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.489+0000 7fbc19e35700 15 client.37425.objecter _session_command_op_remove -1 14 2024-04-14T15:19:51.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.489+0000 7fbc19e35700 20 client.37425.objecter get_session s=0x7fbc0001cdc0 osd=6 3 2024-04-14T15:19:51.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.489+0000 7fbc19e35700 15 client.37425.objecter _session_command_op_assign 6 14 2024-04-14T15:19:51.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.489+0000 7fbc19e35700 20 client.37425.objecter put_session s=0x7fbc0001cdc0 osd=6 4 2024-04-14T15:19:51.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.489+0000 7fbc19e35700 10 client.37425.objecter _send_command 14 2024-04-14T15:19:51.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.489+0000 7fbc19e35700 1 -- 172.21.15.90:0/3838575409 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fbc00023410 con 0x7fbc0001f900 2024-04-14T15:19:51.492 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:19:51.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.491+0000 7fbc10ff9700 1 -- 172.21.15.90:0/3838575409 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fbc00023410 con 0x7fbc0001f900 2024-04-14T15:19:51.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.491+0000 7fbc10ff9700 10 client.37425.objecter ms_dispatch 0x7fbc14001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:19:51.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.491+0000 7fbc10ff9700 10 client.37425.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:19:51.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.491+0000 7fbc10ff9700 20 client.37425.objecter put_session s=0x7fbc0001cdc0 osd=6 4 2024-04-14T15:19:51.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.491+0000 7fbc10ff9700 15 client.37425.objecter _session_command_op_remove 6 14 2024-04-14T15:19:51.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.492+0000 7fbc19e35700 10 client.37425.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:19:51.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.492+0000 7fbc19e35700 15 client.37425.objecter _session_command_op_assign -1 15 2024-04-14T15:19:51.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.492+0000 7fbc19e35700 20 client.37425.objecter _get_session s=0x7fbc00023410 osd=7 3 2024-04-14T15:19:51.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.492+0000 7fbc19e35700 20 client.37425.objecter put_session s=0x7fbc00023410 osd=7 3 2024-04-14T15:19:51.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.492+0000 7fbc19e35700 20 client.37425.objecter _get_session s=0x7fbc00023410 osd=7 3 2024-04-14T15:19:51.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.492+0000 7fbc19e35700 15 client.37425.objecter _session_command_op_remove -1 15 2024-04-14T15:19:51.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.492+0000 7fbc19e35700 20 client.37425.objecter get_session s=0x7fbc00023410 osd=7 3 2024-04-14T15:19:51.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.492+0000 7fbc19e35700 15 client.37425.objecter _session_command_op_assign 7 15 2024-04-14T15:19:51.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.492+0000 7fbc19e35700 20 client.37425.objecter put_session s=0x7fbc00023410 osd=7 4 2024-04-14T15:19:51.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.492+0000 7fbc19e35700 10 client.37425.objecter _send_command 15 2024-04-14T15:19:51.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.492+0000 7fbc19e35700 1 -- 172.21.15.90:0/3838575409 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fbc000277b0 con 0x7fbc00023a90 2024-04-14T15:19:51.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.492+0000 7fbc10ff9700 10 client.37425.objecter ms_handle_connect 0x7fbc00023a90 2024-04-14T15:19:51.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.493+0000 7fbc10ff9700 1 -- 172.21.15.90:0/3838575409 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fbc000277b0 con 0x7fbc00023a90 2024-04-14T15:19:51.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.493+0000 7fbc10ff9700 10 client.37425.objecter ms_dispatch 0x7fbc14001550 command_reply(tid 15: 0 ) 2024-04-14T15:19:51.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.493+0000 7fbc10ff9700 10 client.37425.objecter _finish_command 15 = system:0 2024-04-14T15:19:51.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.493+0000 7fbc10ff9700 20 client.37425.objecter put_session s=0x7fbc00023410 osd=7 4 2024-04-14T15:19:51.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.493+0000 7fbc10ff9700 15 client.37425.objecter _session_command_op_remove 7 15 2024-04-14T15:19:51.507 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:51.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.510+0000 7fbc19e35700 10 client.37425.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:19:51.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.510+0000 7fbc19e35700 15 client.37425.objecter _session_command_op_assign -1 16 2024-04-14T15:19:51.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.510+0000 7fbc19e35700 20 client.37425.objecter _get_session s=0x7fbc00023410 osd=7 3 2024-04-14T15:19:51.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.510+0000 7fbc19e35700 20 client.37425.objecter put_session s=0x7fbc00023410 osd=7 3 2024-04-14T15:19:51.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.510+0000 7fbc19e35700 20 client.37425.objecter _get_session s=0x7fbc00023410 osd=7 3 2024-04-14T15:19:51.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.510+0000 7fbc19e35700 15 client.37425.objecter _session_command_op_remove -1 16 2024-04-14T15:19:51.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.510+0000 7fbc19e35700 20 client.37425.objecter get_session s=0x7fbc00023410 osd=7 3 2024-04-14T15:19:51.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.510+0000 7fbc19e35700 15 client.37425.objecter _session_command_op_assign 7 16 2024-04-14T15:19:51.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.510+0000 7fbc19e35700 20 client.37425.objecter put_session s=0x7fbc00023410 osd=7 4 2024-04-14T15:19:51.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.510+0000 7fbc19e35700 10 client.37425.objecter _send_command 16 2024-04-14T15:19:51.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.510+0000 7fbc19e35700 1 -- 172.21.15.90:0/3838575409 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fbc00027990 con 0x7fbc00023a90 2024-04-14T15:19:51.511 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:19:51.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.511+0000 7fbc10ff9700 1 -- 172.21.15.90:0/3838575409 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fbc00027990 con 0x7fbc00023a90 2024-04-14T15:19:51.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.511+0000 7fbc10ff9700 10 client.37425.objecter ms_dispatch 0x7fbc14001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:19:51.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.511+0000 7fbc10ff9700 10 client.37425.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:19:51.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.511+0000 7fbc10ff9700 20 client.37425.objecter put_session s=0x7fbc00023410 osd=7 4 2024-04-14T15:19:51.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.511+0000 7fbc10ff9700 15 client.37425.objecter _session_command_op_remove 7 16 2024-04-14T15:19:51.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.511+0000 7fbc19e35700 10 librados: watch_flush enter 2024-04-14T15:19:51.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.511+0000 7fbc19e35700 10 librados: watch_flush exit 2024-04-14T15:19:51.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.511+0000 7fbc19e35700 10 client.37425.objecter close_session for osd.0 2024-04-14T15:19:51.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.512+0000 7fbc19e35700 1 -- 172.21.15.90:0/3838575409 >> v1:172.21.15.90:6801/3761806885 conn(0x7fbc00004730 legacy=0x7fbc00002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:51.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.512+0000 7fbc19e35700 20 client.37425.objecter put_session s=0x7fbc000053e0 osd=0 1 2024-04-14T15:19:51.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.512+0000 7fbc19e35700 10 client.37425.objecter close_session for osd.1 2024-04-14T15:19:51.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.512+0000 7fbc19e35700 1 -- 172.21.15.90:0/3838575409 >> v1:172.21.15.90:6805/4170361699 conn(0x7fbc00004310 legacy=0x7fbc0000ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:51.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.512+0000 7fbc19e35700 20 client.37425.objecter put_session s=0x7fbc00004130 osd=1 1 2024-04-14T15:19:51.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.512+0000 7fbc19e35700 10 client.37425.objecter close_session for osd.2 2024-04-14T15:19:51.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.512+0000 7fbc19e35700 1 -- 172.21.15.90:0/3838575409 >> v1:172.21.15.90:6809/1686184251 conn(0x7fbc0000d5d0 legacy=0x7fbc0000f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:51.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.512+0000 7fbc19e35700 20 client.37425.objecter put_session s=0x7fbc00001980 osd=2 1 2024-04-14T15:19:51.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.512+0000 7fbc19e35700 10 client.37425.objecter close_session for osd.3 2024-04-14T15:19:51.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.512+0000 7fbc19e35700 1 -- 172.21.15.90:0/3838575409 >> v1:172.21.15.90:6813/2270166774 conn(0x7fbc000124a0 legacy=0x7fbc00014890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:51.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.512+0000 7fbc19e35700 20 client.37425.objecter put_session s=0x7fbc0000cc20 osd=3 1 2024-04-14T15:19:51.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.512+0000 7fbc19e35700 10 client.37425.objecter close_session for osd.4 2024-04-14T15:19:51.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.512+0000 7fbc19e35700 1 -- 172.21.15.90:0/3838575409 >> v1:172.21.15.92:6800/3887549733 conn(0x7fbc00012000 legacy=0x7fbc000192f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:51.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.512+0000 7fbc19e35700 20 client.37425.objecter put_session s=0x7fbc0000c4b0 osd=4 1 2024-04-14T15:19:51.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.512+0000 7fbc19e35700 10 client.37425.objecter close_session for osd.5 2024-04-14T15:19:51.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.512+0000 7fbc19e35700 1 -- 172.21.15.90:0/3838575409 >> v1:172.21.15.92:6804/3244561440 conn(0x7fbc0001c930 legacy=0x7fbc0001aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:51.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.512+0000 7fbc19e35700 20 client.37425.objecter put_session s=0x7fbc0001c210 osd=5 1 2024-04-14T15:19:51.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.512+0000 7fbc19e35700 10 client.37425.objecter close_session for osd.6 2024-04-14T15:19:51.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.512+0000 7fbc19e35700 1 -- 172.21.15.90:0/3838575409 >> v1:172.21.15.92:6808/636125963 conn(0x7fbc0001f900 legacy=0x7fbc00021cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:51.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.512+0000 7fbc19e35700 20 client.37425.objecter put_session s=0x7fbc0001cdc0 osd=6 1 2024-04-14T15:19:51.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.512+0000 7fbc19e35700 10 client.37425.objecter close_session for osd.7 2024-04-14T15:19:51.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.512+0000 7fbc19e35700 1 -- 172.21.15.90:0/3838575409 >> v1:172.21.15.92:6812/1773706716 conn(0x7fbc00023a90 legacy=0x7fbc00025ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:51.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.512+0000 7fbc19e35700 20 client.37425.objecter put_session s=0x7fbc00023410 osd=7 1 2024-04-14T15:19:51.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.512+0000 7fbc19e35700 20 client.37425.objecter shutdown clearing up homeless session... 2024-04-14T15:19:51.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.512+0000 7fbc19e35700 10 client.37425.objecter successfully canceled tick 2024-04-14T15:19:51.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.512+0000 7fbc19e35700 1 -- 172.21.15.90:0/3838575409 >> v1:172.21.15.90:6800/420935427 conn(0x7fbbfc071430 legacy=0x7fbbfc0738f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:51.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.512+0000 7fbc19e35700 1 -- 172.21.15.90:0/3838575409 >> v1:172.21.15.90:6789/0 conn(0x7fbc14111630 legacy=0x7fbc141a8320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:51.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.512+0000 7fbc13fff700 1 -- 172.21.15.90:0/3838575409 reap_dead start 2024-04-14T15:19:51.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.512+0000 7fbc13fff700 1 -- 172.21.15.90:0/3838575409 reap_dead start 2024-04-14T15:19:51.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.512+0000 7fbc13fff700 1 -- 172.21.15.90:0/3838575409 reap_dead start 2024-04-14T15:19:51.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.512+0000 7fbc13fff700 1 -- 172.21.15.90:0/3838575409 reap_dead start 2024-04-14T15:19:51.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.512+0000 7fbc13fff700 1 -- 172.21.15.90:0/3838575409 reap_dead start 2024-04-14T15:19:51.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.512+0000 7fbc13fff700 1 -- 172.21.15.90:0/3838575409 reap_dead start 2024-04-14T15:19:51.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.512+0000 7fbc13fff700 1 -- 172.21.15.90:0/3838575409 reap_dead start 2024-04-14T15:19:51.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.512+0000 7fbc19e35700 1 -- 172.21.15.90:0/3838575409 shutdown_connections 2024-04-14T15:19:51.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.512+0000 7fbc19e35700 1 -- 172.21.15.90:0/3838575409 >> 172.21.15.90:0/3838575409 conn(0x7fbc14076a70 msgr2=0x7fbc14114d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:19:51.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.512+0000 7fbc19e35700 1 -- 172.21.15.90:0/3838575409 shutdown_connections 2024-04-14T15:19:51.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.512+0000 7fbc19e35700 1 -- 172.21.15.90:0/3838575409 wait complete. 2024-04-14T15:19:51.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:51.512+0000 7fbc19e35700 1 librados: shutdown 2024-04-14T15:19:51.516 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:19:51.516 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:19:51.516 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:19:51.516 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:19:51.516 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:19:51.517 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:19:51.517 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:19:51.517 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:19:51.531 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:51.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:51.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:51.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:51 smithi092 ceph-mon[33966]: 5.18 scrub starts 2024-04-14T15:19:51.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:51 smithi092 ceph-mon[33966]: 5.18 scrub ok 2024-04-14T15:19:51.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:51 smithi092 ceph-mon[33966]: 4.4 scrub starts 2024-04-14T15:19:51.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:51 smithi092 ceph-mon[33966]: 4.4 scrub ok 2024-04-14T15:19:51.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:51 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:19:51.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:51 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:19:51.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:51 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:19:51.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:51 smithi092 ceph-mon[33966]: osdmap e480: 8 total, 8 up, 8 in 2024-04-14T15:19:51.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:51 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:19:51.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:51 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:19:51.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:51 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-73"}]: dispatch 2024-04-14T15:19:51.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:51 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-73"}]: dispatch 2024-04-14T15:19:51.632 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:19:51.655 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:51.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:51.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:19:51.757 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:19:51.780 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:51.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:51.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:19:51.882 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:19:51.906 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:51.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:51.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:19:52.007 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:19:52.030 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:52.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:52.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:19:52.132 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:19:52.155 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:52.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:52.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:19:52.256 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:52.280 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:52.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:52.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:52.381 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:19:52.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:52.405 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:52.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:19:52.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:52 smithi090 ceph-mon[41670]: 6.3 scrub starts 2024-04-14T15:19:52.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:52 smithi090 ceph-mon[41670]: 6.3 scrub ok 2024-04-14T15:19:52.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:52 smithi090 ceph-mon[41670]: 5.0 scrub starts 2024-04-14T15:19:52.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:52 smithi090 ceph-mon[41670]: 5.0 scrub ok 2024-04-14T15:19:52.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:52 smithi090 ceph-mon[41670]: pgmap v688: 258 pgs: 258 active+clean; 8.4 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 737 B/s rd, 0 op/s 2024-04-14T15:19:52.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:52 smithi090 ceph-mon[41670]: 4.16 scrub starts 2024-04-14T15:19:52.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:52 smithi090 ceph-mon[41670]: 4.16 scrub ok 2024-04-14T15:19:52.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:52 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3838575409' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:19:52.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:52 smithi090 ceph-mon[41670]: osdmap e481: 8 total, 8 up, 8 in 2024-04-14T15:19:52.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:52 smithi090 ceph-mon[30399]: 6.3 scrub starts 2024-04-14T15:19:52.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:52 smithi090 ceph-mon[30399]: 6.3 scrub ok 2024-04-14T15:19:52.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:52 smithi090 ceph-mon[30399]: 5.0 scrub starts 2024-04-14T15:19:52.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:52 smithi090 ceph-mon[30399]: 5.0 scrub ok 2024-04-14T15:19:52.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:52 smithi090 ceph-mon[30399]: pgmap v688: 258 pgs: 258 active+clean; 8.4 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 737 B/s rd, 0 op/s 2024-04-14T15:19:52.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:52 smithi090 ceph-mon[30399]: 4.16 scrub starts 2024-04-14T15:19:52.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:52 smithi090 ceph-mon[30399]: 4.16 scrub ok 2024-04-14T15:19:52.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:52 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3838575409' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:19:52.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:52 smithi090 ceph-mon[30399]: osdmap e481: 8 total, 8 up, 8 in 2024-04-14T15:19:52.506 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:19:52.523 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:52 smithi092 ceph-mon[33966]: 6.3 scrub starts 2024-04-14T15:19:52.523 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:52 smithi092 ceph-mon[33966]: 6.3 scrub ok 2024-04-14T15:19:52.523 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:52 smithi092 ceph-mon[33966]: 5.0 scrub starts 2024-04-14T15:19:52.523 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:52 smithi092 ceph-mon[33966]: 5.0 scrub ok 2024-04-14T15:19:52.523 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:52 smithi092 ceph-mon[33966]: pgmap v688: 258 pgs: 258 active+clean; 8.4 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 737 B/s rd, 0 op/s 2024-04-14T15:19:52.523 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:52 smithi092 ceph-mon[33966]: 4.16 scrub starts 2024-04-14T15:19:52.524 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:52 smithi092 ceph-mon[33966]: 4.16 scrub ok 2024-04-14T15:19:52.524 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:52 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3838575409' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:19:52.524 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:52 smithi092 ceph-mon[33966]: osdmap e481: 8 total, 8 up, 8 in 2024-04-14T15:19:52.530 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:52.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:52.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:19:52.631 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:52.655 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:52.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:52.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:52.757 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:19:52.781 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:52.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:52.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:19:52.882 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:19:52.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:52.908 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:52.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:19:52.914 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-14T15:19:52.914 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-14T15:19:52.914 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph pg dump --format=json 2024-04-14T15:19:53.010 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:19:53.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:53.034 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:53.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:19:53.135 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:19:53.159 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:53.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:53.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:19:53.261 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:19:53.286 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:53.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:53.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:19:53.291 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:53 smithi090 ceph-mon[41670]: 6.f scrub starts 2024-04-14T15:19:53.291 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:53 smithi090 ceph-mon[41670]: 6.f scrub ok 2024-04-14T15:19:53.291 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:53 smithi090 ceph-mon[41670]: osdmap e482: 8 total, 8 up, 8 in 2024-04-14T15:19:53.292 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:53 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:19:53.292 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:53 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:19:53.292 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:53 smithi090 ceph-mon[30399]: 6.f scrub starts 2024-04-14T15:19:53.292 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:53 smithi090 ceph-mon[30399]: 6.f scrub ok 2024-04-14T15:19:53.292 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:53 smithi090 ceph-mon[30399]: osdmap e482: 8 total, 8 up, 8 in 2024-04-14T15:19:53.292 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:53 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:19:53.292 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:53 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:19:53.388 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:19:53.412 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:53.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:53.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:19:53.514 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:19:53.530 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-14T15:19:53.530 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:19:53.538 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:53.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:53.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:19:53.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:53 smithi092 ceph-mon[33966]: 6.f scrub starts 2024-04-14T15:19:53.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:53 smithi092 ceph-mon[33966]: 6.f scrub ok 2024-04-14T15:19:53.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:53 smithi092 ceph-mon[33966]: osdmap e482: 8 total, 8 up, 8 in 2024-04-14T15:19:53.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:53 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:19:53.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:53 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:19:53.640 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:53.665 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:53.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:53.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:53.767 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:19:53.792 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:53.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:53.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:19:53.893 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:19:53.917 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:53.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:53.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:19:53.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:53.952+0000 7f2f50839700 1 -- 172.21.15.90:0/4265765808 >> v1:172.21.15.92:6789/0 conn(0x7f2f4c10a1c0 legacy=0x7f2f4c1166d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:53.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:53.953+0000 7f2f50839700 1 -- 172.21.15.90:0/4265765808 shutdown_connections 2024-04-14T15:19:53.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:53.953+0000 7f2f50839700 1 -- 172.21.15.90:0/4265765808 >> 172.21.15.90:0/4265765808 conn(0x7f2f4c06d990 msgr2=0x7f2f4c06dda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:19:53.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:53.954+0000 7f2f50839700 1 -- 172.21.15.90:0/4265765808 shutdown_connections 2024-04-14T15:19:53.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:53.954+0000 7f2f50839700 1 -- 172.21.15.90:0/4265765808 wait complete. 2024-04-14T15:19:53.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:53.955+0000 7f2f50839700 1 librados: starting msgr at 2024-04-14T15:19:53.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:53.955+0000 7f2f50839700 1 librados: starting objecter 2024-04-14T15:19:53.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:53.955+0000 7f2f50839700 1 Processor -- start 2024-04-14T15:19:53.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:53.956+0000 7f2f50839700 1 -- start start 2024-04-14T15:19:53.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:53.956+0000 7f2f50839700 1 librados: setting wanted keys 2024-04-14T15:19:53.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:53.956+0000 7f2f50839700 1 librados: calling monclient init 2024-04-14T15:19:53.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:53.957+0000 7f2f50839700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f2f4c1bee20 con 0x7f2f4c10a1c0 2024-04-14T15:19:53.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:53.957+0000 7f2f50839700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f2f4c1c0000 con 0x7f2f4c06ebd0 2024-04-14T15:19:53.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:53.957+0000 7f2f50839700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f2f4c1c11e0 con 0x7f2f4c117df0 2024-04-14T15:19:53.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:53.957+0000 7f2f49d9b700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f2f4c10a1c0 0x7f2f4c1b9fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:54918/0 (socket says 172.21.15.90:54918) 2024-04-14T15:19:53.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:53.957+0000 7f2f49d9b700 1 -- 172.21.15.90:0/3592119721 learned_addr learned my addr 172.21.15.90:0/3592119721 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:19:53.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:53.958+0000 7f2f3b7fe700 10 client.?.objecter ms_handle_connect 0x7f2f4c10a1c0 2024-04-14T15:19:53.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:53.958+0000 7f2f3b7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:19:53.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:53.958+0000 7f2f3b7fe700 10 client.?.objecter ms_handle_connect 0x7f2f4c06ebd0 2024-04-14T15:19:53.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:53.958+0000 7f2f3b7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:19:53.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:53.958+0000 7f2f3b7fe700 10 client.?.objecter ms_handle_connect 0x7f2f4c117df0 2024-04-14T15:19:53.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:53.958+0000 7f2f3b7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:19:53.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:53.958+0000 7f2f3b7fe700 1 -- 172.21.15.90:0/3592119721 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 26687458 0 0) 0x7f2f4c1c0000 con 0x7f2f4c06ebd0 2024-04-14T15:19:53.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:53.959+0000 7f2f3b7fe700 1 -- 172.21.15.90:0/3592119721 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f340031f0 con 0x7f2f4c06ebd0 2024-04-14T15:19:53.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:53.959+0000 7f2f3b7fe700 1 -- 172.21.15.90:0/3592119721 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3006509081 0 0) 0x7f2f4c1bee20 con 0x7f2f4c10a1c0 2024-04-14T15:19:53.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:53.959+0000 7f2f3b7fe700 1 -- 172.21.15.90:0/3592119721 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f4c1c0000 con 0x7f2f4c10a1c0 2024-04-14T15:19:53.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:53.959+0000 7f2f3b7fe700 1 -- 172.21.15.90:0/3592119721 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3357009537 0 0) 0x7f2f4c1c11e0 con 0x7f2f4c117df0 2024-04-14T15:19:53.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:53.959+0000 7f2f3b7fe700 1 -- 172.21.15.90:0/3592119721 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f4c1bee20 con 0x7f2f4c117df0 2024-04-14T15:19:53.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:53.959+0000 7f2f3b7fe700 1 -- 172.21.15.90:0/3592119721 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2316542396 0 0) 0x7f2f4c1c0000 con 0x7f2f4c10a1c0 2024-04-14T15:19:53.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:53.959+0000 7f2f3b7fe700 1 -- 172.21.15.90:0/3592119721 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2f4c1c11e0 con 0x7f2f4c10a1c0 2024-04-14T15:19:53.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:53.959+0000 7f2f3b7fe700 1 -- 172.21.15.90:0/3592119721 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f2f3c0043f0 con 0x7f2f4c10a1c0 2024-04-14T15:19:53.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:53.959+0000 7f2f3b7fe700 1 -- 172.21.15.90:0/3592119721 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 357781987 0 0) 0x7f2f4c1bee20 con 0x7f2f4c117df0 2024-04-14T15:19:53.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:53.959+0000 7f2f3b7fe700 1 -- 172.21.15.90:0/3592119721 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2f4c1c0000 con 0x7f2f4c117df0 2024-04-14T15:19:53.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:53.959+0000 7f2f3b7fe700 1 -- 172.21.15.90:0/3592119721 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1172102277 0 0) 0x7f2f340031f0 con 0x7f2f4c06ebd0 2024-04-14T15:19:53.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:53.959+0000 7f2f3b7fe700 1 -- 172.21.15.90:0/3592119721 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2f4c1bee20 con 0x7f2f4c06ebd0 2024-04-14T15:19:53.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:53.959+0000 7f2f3b7fe700 1 -- 172.21.15.90:0/3592119721 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f2f400017b0 con 0x7f2f4c117df0 2024-04-14T15:19:53.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:53.959+0000 7f2f3b7fe700 1 -- 172.21.15.90:0/3592119721 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f2f44001bf0 con 0x7f2f4c06ebd0 2024-04-14T15:19:53.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:53.959+0000 7f2f3b7fe700 1 -- 172.21.15.90:0/3592119721 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2011705295 0 0) 0x7f2f4c1c11e0 con 0x7f2f4c10a1c0 2024-04-14T15:19:53.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:53.959+0000 7f2f3b7fe700 1 -- 172.21.15.90:0/3592119721 >> v1:172.21.15.90:6790/0 conn(0x7f2f4c117df0 legacy=0x7f2f4c1bd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:53.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:53.959+0000 7f2f3b7fe700 1 -- 172.21.15.90:0/3592119721 >> v1:172.21.15.92:6789/0 conn(0x7f2f4c06ebd0 legacy=0x7f2f4c117140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:53.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:53.959+0000 7f2f3b7fe700 1 -- 172.21.15.90:0/3592119721 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f4c1c2400 con 0x7f2f4c10a1c0 2024-04-14T15:19:53.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:53.959+0000 7f2f50839700 1 -- 172.21.15.90:0/3592119721 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2f4c1c0230 con 0x7f2f4c10a1c0 2024-04-14T15:19:53.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:53.959+0000 7f2f50839700 10 client.37431.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:19:53.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:53.959+0000 7f2f50839700 1 -- 172.21.15.90:0/3592119721 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f2f4c1c06f0 con 0x7f2f4c10a1c0 2024-04-14T15:19:53.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:53.960+0000 7f2f50839700 1 librados: init done 2024-04-14T15:19:53.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:53.960+0000 7f2f3b7fe700 1 -- 172.21.15.90:0/3592119721 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f2f3c001110 con 0x7f2f4c10a1c0 2024-04-14T15:19:53.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:53.960+0000 7f2f3b7fe700 1 -- 172.21.15.90:0/3592119721 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f2f3c004d40 con 0x7f2f4c10a1c0 2024-04-14T15:19:53.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:53.961+0000 7f2f50839700 1 -- 172.21.15.90:0/3592119721 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f2c005120 con 0x7f2f4c10a1c0 2024-04-14T15:19:53.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:53.961+0000 7f2f3b7fe700 1 -- 172.21.15.90:0/3592119721 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f2f3c004fc0 con 0x7f2f4c10a1c0 2024-04-14T15:19:53.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:53.962+0000 7f2f3b7fe700 1 -- 172.21.15.90:0/3592119721 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(483..483 src has 51..483) ==== 8197+0+0 (unknown 1258600404 0 0) 0x7f2f3c08aa90 con 0x7f2f4c10a1c0 2024-04-14T15:19:53.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:53.962+0000 7f2f3b7fe700 10 client.37431.objecter ms_dispatch 0x7f2f4c001550 osd_map(483..483 src has 51..483) 2024-04-14T15:19:53.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:53.962+0000 7f2f3b7fe700 3 client.37431.objecter handle_osd_map got epochs [483,483] > 0 2024-04-14T15:19:53.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:53.962+0000 7f2f3b7fe700 3 client.37431.objecter handle_osd_map decoding full epoch 483 2024-04-14T15:19:53.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:53.962+0000 7f2f3b7fe700 20 client.37431.objecter dump_active .. 0 homeless 2024-04-14T15:19:53.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:53.962+0000 7f2f3b7fe700 10 client.37431.objecter ms_handle_connect 0x7f2f340715b0 2024-04-14T15:19:53.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:53.967+0000 7f2f3b7fe700 1 -- 172.21.15.90:0/3592119721 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f2f3c0574a0 con 0x7f2f4c10a1c0 2024-04-14T15:19:54.019 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:19:54.042 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:54.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:54.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:19:54.144 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:19:54.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:54.167 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:54.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:19:54.182 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:19:53 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:19:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:19:54.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:54.187+0000 7f2f50839700 1 -- 172.21.15.90:0/3592119721 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7f2f2c005410 con 0x7f2f4c10a1c0 2024-04-14T15:19:54.269 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:19:54.294 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:54.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:54.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:19:54.395 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:19:54.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:54.420 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:54.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:19:54.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:54 smithi090 ceph-mon[41670]: pgmap v691: 258 pgs: 32 unknown, 226 active+clean; 8.4 MiB data, 577 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:19:54.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:54 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:19:54.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:54 smithi090 ceph-mon[41670]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:19:54.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:54 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:19:54.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:54 smithi090 ceph-mon[41670]: osdmap e483: 8 total, 8 up, 8 in 2024-04-14T15:19:54.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:54 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-14T15:19:54.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:54 smithi090 ceph-mon[30399]: pgmap v691: 258 pgs: 32 unknown, 226 active+clean; 8.4 MiB data, 577 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:19:54.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:54 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:19:54.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:54 smithi090 ceph-mon[30399]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:19:54.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:54 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:19:54.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:54 smithi090 ceph-mon[30399]: osdmap e483: 8 total, 8 up, 8 in 2024-04-14T15:19:54.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:54 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-14T15:19:54.455 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-14T15:19:54.521 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:19:54.545 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:54.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:54.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:19:54.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:54 smithi092 ceph-mon[33966]: pgmap v691: 258 pgs: 32 unknown, 226 active+clean; 8.4 MiB data, 577 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:19:54.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:54 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:19:54.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:54 smithi092 ceph-mon[33966]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:19:54.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:54 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:19:54.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:54 smithi092 ceph-mon[33966]: osdmap e483: 8 total, 8 up, 8 in 2024-04-14T15:19:54.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:54 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-14T15:19:54.647 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:19:54.672 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:54.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:54.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:19:54.774 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:19:54.797 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:54.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:54.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:19:54.899 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:19:54.924 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:54.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:54.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:19:54.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:54.933+0000 7f2f3b7fe700 1 -- 172.21.15.90:0/3592119721 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v484) ==== 101+0+0 (unknown 2763393559 0 0) 0x7f2f3c05b150 con 0x7f2f4c10a1c0 2024-04-14T15:19:54.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:54.935+0000 7f2f50839700 10 librados: watch_flush enter 2024-04-14T15:19:54.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:54.935+0000 7f2f50839700 10 librados: watch_flush exit 2024-04-14T15:19:54.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:54.935+0000 7f2f50839700 20 client.37431.objecter shutdown clearing up homeless session... 2024-04-14T15:19:54.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:54.935+0000 7f2f50839700 10 client.37431.objecter successfully canceled tick 2024-04-14T15:19:54.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:54.935+0000 7f2f50839700 1 -- 172.21.15.90:0/3592119721 >> v1:172.21.15.90:6800/420935427 conn(0x7f2f340715b0 legacy=0x7f2f34073a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:54.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:54.935+0000 7f2f50839700 1 -- 172.21.15.90:0/3592119721 >> v1:172.21.15.90:6789/0 conn(0x7f2f4c10a1c0 legacy=0x7f2f4c1b9fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:19:54.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:54.936+0000 7f2f50839700 1 -- 172.21.15.90:0/3592119721 shutdown_connections 2024-04-14T15:19:54.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:54.936+0000 7f2f50839700 1 -- 172.21.15.90:0/3592119721 >> 172.21.15.90:0/3592119721 conn(0x7f2f4c06d990 msgr2=0x7f2f4c072800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:19:54.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:54.936+0000 7f2f50839700 1 -- 172.21.15.90:0/3592119721 shutdown_connections 2024-04-14T15:19:54.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:54.936+0000 7f2f50839700 1 -- 172.21.15.90:0/3592119721 wait complete. 2024-04-14T15:19:54.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:19:54.936+0000 7f2f50839700 1 librados: shutdown 2024-04-14T15:19:54.937 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is unset 2024-04-14T15:19:55.026 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:19:55.050 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:55.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:55.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:19:55.152 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:19:55.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:55.178 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:55.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:19:55.279 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:19:55.303 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:55.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:55.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:19:55.405 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:19:55.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:55.431 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:55.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:19:55.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:55 smithi090 ceph-mon[41670]: 4.19 scrub starts 2024-04-14T15:19:55.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:55 smithi090 ceph-mon[41670]: 4.19 scrub ok 2024-04-14T15:19:55.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:55 smithi090 ceph-mon[41670]: 11.1f scrub starts 2024-04-14T15:19:55.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:55 smithi090 ceph-mon[41670]: 11.1f scrub ok 2024-04-14T15:19:55.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:55 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3592119721' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:19:55.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:55 smithi090 ceph-mon[41670]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-14T15:19:55.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:55 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3592119721' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-14T15:19:55.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:55 smithi090 ceph-mon[41670]: osdmap e484: 8 total, 8 up, 8 in 2024-04-14T15:19:55.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:55 smithi090 ceph-mon[30399]: 4.19 scrub starts 2024-04-14T15:19:55.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:55 smithi090 ceph-mon[30399]: 4.19 scrub ok 2024-04-14T15:19:55.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:55 smithi090 ceph-mon[30399]: 11.1f scrub starts 2024-04-14T15:19:55.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:55 smithi090 ceph-mon[30399]: 11.1f scrub ok 2024-04-14T15:19:55.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:55 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3592119721' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:19:55.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:55 smithi090 ceph-mon[30399]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-14T15:19:55.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:55 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3592119721' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-14T15:19:55.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:55 smithi090 ceph-mon[30399]: osdmap e484: 8 total, 8 up, 8 in 2024-04-14T15:19:55.531 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:19:55.555 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:55.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:55.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:19:55.622 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:19:55 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:19:55.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:55 smithi092 ceph-mon[33966]: 4.19 scrub starts 2024-04-14T15:19:55.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:55 smithi092 ceph-mon[33966]: 4.19 scrub ok 2024-04-14T15:19:55.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:55 smithi092 ceph-mon[33966]: 11.1f scrub starts 2024-04-14T15:19:55.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:55 smithi092 ceph-mon[33966]: 11.1f scrub ok 2024-04-14T15:19:55.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:55 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3592119721' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:19:55.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:55 smithi092 ceph-mon[33966]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-14T15:19:55.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:55 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3592119721' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-14T15:19:55.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:55 smithi092 ceph-mon[33966]: osdmap e484: 8 total, 8 up, 8 in 2024-04-14T15:19:55.656 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:19:55.680 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:55.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:55.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:19:55.782 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:55.807 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:55.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:55.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:55.908 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:19:55.933 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:55.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:55.934 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:19:55.989 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:19:56.035 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:19:56.059 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:56.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:56.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:19:56.160 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:19:56.184 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:56.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:56.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:19:56.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:56 smithi090 ceph-mon[41670]: 5.10 scrub starts 2024-04-14T15:19:56.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:56 smithi090 ceph-mon[41670]: 5.10 scrub ok 2024-04-14T15:19:56.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:56 smithi090 ceph-mon[41670]: pgmap v694: 258 pgs: 258 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 107 KiB/s rd, 255 B/s wr, 176 op/s 2024-04-14T15:19:56.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:56 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:19:56.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:56 smithi090 ceph-mon[41670]: osdmap e485: 8 total, 8 up, 8 in 2024-04-14T15:19:56.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:56 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:19:56.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:56 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:19:56.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:56 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-75"}]: dispatch 2024-04-14T15:19:56.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:56 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-75"}]: dispatch 2024-04-14T15:19:56.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:56 smithi090 ceph-mon[30399]: 5.10 scrub starts 2024-04-14T15:19:56.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:56 smithi090 ceph-mon[30399]: 5.10 scrub ok 2024-04-14T15:19:56.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:56 smithi090 ceph-mon[30399]: pgmap v694: 258 pgs: 258 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 107 KiB/s rd, 255 B/s wr, 176 op/s 2024-04-14T15:19:56.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:56 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:19:56.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:56 smithi090 ceph-mon[30399]: osdmap e485: 8 total, 8 up, 8 in 2024-04-14T15:19:56.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:56 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:19:56.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:56 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:19:56.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:56 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-75"}]: dispatch 2024-04-14T15:19:56.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:56 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-75"}]: dispatch 2024-04-14T15:19:56.285 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:19:56.310 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:56.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:56.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:19:56.412 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:19:56.436 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:56.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:56.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:19:56.537 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:19:56.562 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:56.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:56.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:19:56.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:56 smithi092 ceph-mon[33966]: 5.10 scrub starts 2024-04-14T15:19:56.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:56 smithi092 ceph-mon[33966]: 5.10 scrub ok 2024-04-14T15:19:56.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:56 smithi092 ceph-mon[33966]: pgmap v694: 258 pgs: 258 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 107 KiB/s rd, 255 B/s wr, 176 op/s 2024-04-14T15:19:56.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:56 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:19:56.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:56 smithi092 ceph-mon[33966]: osdmap e485: 8 total, 8 up, 8 in 2024-04-14T15:19:56.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:56 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:19:56.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:56 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:19:56.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:56 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-75"}]: dispatch 2024-04-14T15:19:56.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:56 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-75"}]: dispatch 2024-04-14T15:19:56.664 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:19:56.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:56.690 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:56.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:19:56.725 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:19:56.791 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:19:56.816 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:56.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:56.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:19:56.917 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:19:56.942 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:56.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:56.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:19:57.043 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:19:57.075 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:57.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:57.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:19:57.091 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-14T15:19:57.177 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:19:57.201 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:57.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:57.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:19:57.303 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:57.328 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:57.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:57.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:57.430 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:19:57.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:57 smithi090 ceph-mon[41670]: 11.d scrub starts 2024-04-14T15:19:57.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:57 smithi090 ceph-mon[41670]: 11.d scrub ok 2024-04-14T15:19:57.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:57 smithi090 ceph-mon[41670]: osdmap e486: 8 total, 8 up, 8 in 2024-04-14T15:19:57.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:57 smithi090 ceph-mon[30399]: 11.d scrub starts 2024-04-14T15:19:57.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:57 smithi090 ceph-mon[30399]: 11.d scrub ok 2024-04-14T15:19:57.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:57 smithi090 ceph-mon[30399]: osdmap e486: 8 total, 8 up, 8 in 2024-04-14T15:19:57.456 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:57.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:57.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:19:57.558 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:19:57.585 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:57.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:57.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:19:57.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:57 smithi092 ceph-mon[33966]: 11.d scrub starts 2024-04-14T15:19:57.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:57 smithi092 ceph-mon[33966]: 11.d scrub ok 2024-04-14T15:19:57.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:57 smithi092 ceph-mon[33966]: osdmap e486: 8 total, 8 up, 8 in 2024-04-14T15:19:57.686 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:19:57.713 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:57.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:57.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:19:57.814 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:57.840 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:57.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:57.841 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:57.942 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:19:57.966 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:57.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:57.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:19:57.971 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd set noscrub 2024-04-14T15:19:58.068 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:19:58.091 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:58.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:58.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:19:58.192 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:19:58.216 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:58.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:58.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:19:58.286 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:58 smithi090 ceph-mon[41670]: 2.12 scrub starts 2024-04-14T15:19:58.287 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:58 smithi090 ceph-mon[41670]: 2.12 scrub ok 2024-04-14T15:19:58.287 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:58 smithi090 ceph-mon[41670]: 5.19 scrub starts 2024-04-14T15:19:58.287 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:58 smithi090 ceph-mon[41670]: 5.19 scrub ok 2024-04-14T15:19:58.287 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:58 smithi090 ceph-mon[41670]: 110.7 scrub starts 2024-04-14T15:19:58.287 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:58 smithi090 ceph-mon[41670]: 110.7 scrub ok 2024-04-14T15:19:58.287 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:58 smithi090 ceph-mon[41670]: pgmap v697: 226 pgs: 226 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s rd, 255 B/s wr, 180 op/s 2024-04-14T15:19:58.287 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:58 smithi090 ceph-mon[41670]: 2.18 scrub starts 2024-04-14T15:19:58.287 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:58 smithi090 ceph-mon[41670]: 2.18 scrub ok 2024-04-14T15:19:58.287 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:58 smithi090 ceph-mon[41670]: osdmap e487: 8 total, 8 up, 8 in 2024-04-14T15:19:58.287 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:58 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:19:58.287 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:58 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:19:58.287 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:58 smithi090 ceph-mon[30399]: 2.12 scrub starts 2024-04-14T15:19:58.288 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:58 smithi090 ceph-mon[30399]: 2.12 scrub ok 2024-04-14T15:19:58.288 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:58 smithi090 ceph-mon[30399]: 5.19 scrub starts 2024-04-14T15:19:58.288 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:58 smithi090 ceph-mon[30399]: 5.19 scrub ok 2024-04-14T15:19:58.288 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:58 smithi090 ceph-mon[30399]: 110.7 scrub starts 2024-04-14T15:19:58.288 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:58 smithi090 ceph-mon[30399]: 110.7 scrub ok 2024-04-14T15:19:58.288 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:58 smithi090 ceph-mon[30399]: pgmap v697: 226 pgs: 226 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s rd, 255 B/s wr, 180 op/s 2024-04-14T15:19:58.288 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:58 smithi090 ceph-mon[30399]: 2.18 scrub starts 2024-04-14T15:19:58.288 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:58 smithi090 ceph-mon[30399]: 2.18 scrub ok 2024-04-14T15:19:58.288 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:58 smithi090 ceph-mon[30399]: osdmap e487: 8 total, 8 up, 8 in 2024-04-14T15:19:58.288 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:58 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:19:58.288 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:58 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:19:58.318 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:19:58.341 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:58.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:58.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:19:58.442 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:58.470 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:58.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:58.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:58.571 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:19:58.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:58.599 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:58.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:19:58.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:58 smithi092 ceph-mon[33966]: 2.12 scrub starts 2024-04-14T15:19:58.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:58 smithi092 ceph-mon[33966]: 2.12 scrub ok 2024-04-14T15:19:58.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:58 smithi092 ceph-mon[33966]: 5.19 scrub starts 2024-04-14T15:19:58.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:58 smithi092 ceph-mon[33966]: 5.19 scrub ok 2024-04-14T15:19:58.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:58 smithi092 ceph-mon[33966]: 110.7 scrub starts 2024-04-14T15:19:58.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:58 smithi092 ceph-mon[33966]: 110.7 scrub ok 2024-04-14T15:19:58.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:58 smithi092 ceph-mon[33966]: pgmap v697: 226 pgs: 226 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s rd, 255 B/s wr, 180 op/s 2024-04-14T15:19:58.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:58 smithi092 ceph-mon[33966]: 2.18 scrub starts 2024-04-14T15:19:58.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:58 smithi092 ceph-mon[33966]: 2.18 scrub ok 2024-04-14T15:19:58.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:58 smithi092 ceph-mon[33966]: osdmap e487: 8 total, 8 up, 8 in 2024-04-14T15:19:58.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:58 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:19:58.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:58 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:19:58.700 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:19:58.723 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:58.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:58.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:19:58.825 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:19:58.853 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:58.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:58.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:19:58.908 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-14T15:19:58.908 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:19:58.955 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:19:58.987 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:58.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:58.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:19:59.089 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:19:59.112 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:59.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:59.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:19:59.214 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:19:59.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:59.240 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:59.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:19:59.340 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:19:59.360 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:19:59.365 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:59.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:59.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:19:59.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:59 smithi090 ceph-mon[30399]: 6.11 scrub starts 2024-04-14T15:19:59.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:59 smithi090 ceph-mon[30399]: 6.11 scrub ok 2024-04-14T15:19:59.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:59 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:19:59.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:59 smithi090 ceph-mon[30399]: osdmap e488: 8 total, 8 up, 8 in 2024-04-14T15:19:59.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:19:59 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-14T15:19:59.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:59 smithi090 ceph-mon[41670]: 6.11 scrub starts 2024-04-14T15:19:59.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:59 smithi090 ceph-mon[41670]: 6.11 scrub ok 2024-04-14T15:19:59.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:59 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:19:59.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:59 smithi090 ceph-mon[41670]: osdmap e488: 8 total, 8 up, 8 in 2024-04-14T15:19:59.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:19:59 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-14T15:19:59.467 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:19:59.494 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:59.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:59.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:19:59.596 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:19:59.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:59 smithi092 ceph-mon[33966]: 6.11 scrub starts 2024-04-14T15:19:59.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:59 smithi092 ceph-mon[33966]: 6.11 scrub ok 2024-04-14T15:19:59.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:59 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:19:59.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:59 smithi092 ceph-mon[33966]: osdmap e488: 8 total, 8 up, 8 in 2024-04-14T15:19:59.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:19:59 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-14T15:19:59.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:59.623 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:19:59.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:19:59.724 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:19:59.748 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:59.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:59.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:19:59.849 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:19:59.873 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:19:59.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:19:59.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:19:59.974 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:20:00.003 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:00.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:00.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:20:00.069 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-14T15:20:00.105 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:00.137 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:00.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:00.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:00.238 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:20:00.266 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:00.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:00.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:20:00.296 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:00 smithi090 ceph-mon[41670]: 4.a deep-scrub starts 2024-04-14T15:20:00.296 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:00 smithi090 ceph-mon[41670]: 4.a deep-scrub ok 2024-04-14T15:20:00.297 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:00 smithi090 ceph-mon[41670]: pgmap v700: 258 pgs: 14 creating+activating, 18 unknown, 226 active+clean; 8.4 MiB data, 598 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:20:00.297 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:00 smithi090 ceph-mon[41670]: osdmap e489: 8 total, 8 up, 8 in 2024-04-14T15:20:00.297 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:00 smithi090 ceph-mon[41670]: Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-04-14T15:20:00.297 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:00 smithi090 ceph-mon[41670]: [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-04-14T15:20:00.297 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:00 smithi090 ceph-mon[41670]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-04-14T15:20:00.297 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:00 smithi090 ceph-mon[41670]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-04-14T15:20:00.297 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:00 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:20:00.297 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:00 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:20:00.297 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:00 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-77"}]: dispatch 2024-04-14T15:20:00.297 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:00 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-77"}]: dispatch 2024-04-14T15:20:00.297 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:00 smithi090 ceph-mon[30399]: 4.a deep-scrub starts 2024-04-14T15:20:00.297 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:00 smithi090 ceph-mon[30399]: 4.a deep-scrub ok 2024-04-14T15:20:00.297 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:00 smithi090 ceph-mon[30399]: pgmap v700: 258 pgs: 14 creating+activating, 18 unknown, 226 active+clean; 8.4 MiB data, 598 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:20:00.298 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:00 smithi090 ceph-mon[30399]: osdmap e489: 8 total, 8 up, 8 in 2024-04-14T15:20:00.298 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:00 smithi090 ceph-mon[30399]: Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-04-14T15:20:00.298 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:00 smithi090 ceph-mon[30399]: [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-04-14T15:20:00.298 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:00 smithi090 ceph-mon[30399]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-04-14T15:20:00.298 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:00 smithi090 ceph-mon[30399]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-04-14T15:20:00.298 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:00 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:20:00.298 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:00 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:20:00.298 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:00 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-77"}]: dispatch 2024-04-14T15:20:00.298 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:00 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-77"}]: dispatch 2024-04-14T15:20:00.369 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:20:00.396 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:00.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:00.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:20:00.497 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:20:00.524 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:00.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:00.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:20:00.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:00.482+0000 7fb8f1729700 1 -- 172.21.15.90:0/1398256893 >> v1:172.21.15.90:6789/0 conn(0x7fb8ec10f300 legacy=0x7fb8ec1117c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:00.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:00.482+0000 7fb8f1729700 1 -- 172.21.15.90:0/1398256893 shutdown_connections 2024-04-14T15:20:00.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:00.482+0000 7fb8f1729700 1 -- 172.21.15.90:0/1398256893 >> 172.21.15.90:0/1398256893 conn(0x7fb8ec0fe3d0 msgr2=0x7fb8ec1007f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:20:00.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:00.482+0000 7fb8f1729700 1 -- 172.21.15.90:0/1398256893 shutdown_connections 2024-04-14T15:20:00.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:00.482+0000 7fb8f1729700 1 -- 172.21.15.90:0/1398256893 wait complete. 2024-04-14T15:20:00.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:00.482+0000 7fb8f1729700 1 librados: starting msgr at 2024-04-14T15:20:00.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:00.482+0000 7fb8f1729700 1 librados: starting objecter 2024-04-14T15:20:00.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:00.482+0000 7fb8f1729700 1 Processor -- start 2024-04-14T15:20:00.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:00.483+0000 7fb8f1729700 1 -- start start 2024-04-14T15:20:00.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:00.483+0000 7fb8f1729700 1 librados: setting wanted keys 2024-04-14T15:20:00.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:00.483+0000 7fb8f1729700 1 librados: calling monclient init 2024-04-14T15:20:00.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:00.483+0000 7fb8f1729700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb8ec10ef70 con 0x7fb8ec10f300 2024-04-14T15:20:00.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:00.483+0000 7fb8f1729700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb8ec1a8f10 con 0x7fb8ec10b730 2024-04-14T15:20:00.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:00.483+0000 7fb8f1729700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb8ec1aa0f0 con 0x7fb8ec108880 2024-04-14T15:20:00.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:00.483+0000 7fb8eaffd700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7fb8ec108880 0x7fb8ec10bdc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:53710/0 (socket says 172.21.15.90:53710) 2024-04-14T15:20:00.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:00.483+0000 7fb8eaffd700 1 -- 172.21.15.90:0/3367719493 learned_addr learned my addr 172.21.15.90:0/3367719493 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:20:00.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:00.483+0000 7fb8eb7fe700 1 --1- 172.21.15.90:0/3367719493 >> v1:172.21.15.90:6789/0 conn(0x7fb8ec10f300 0x7fb8ec1a67e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:54932/0 (socket says 172.21.15.90:54932) 2024-04-14T15:20:00.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:00.483+0000 7fb8d3fff700 10 client.?.objecter ms_handle_connect 0x7fb8ec10f300 2024-04-14T15:20:00.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:00.483+0000 7fb8d3fff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:20:00.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:00.483+0000 7fb8d3fff700 10 client.?.objecter ms_handle_connect 0x7fb8ec108880 2024-04-14T15:20:00.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:00.483+0000 7fb8d3fff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:20:00.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:00.483+0000 7fb8d3fff700 10 client.?.objecter ms_handle_connect 0x7fb8ec10b730 2024-04-14T15:20:00.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:00.483+0000 7fb8d3fff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:20:00.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:00.483+0000 7fb8d3fff700 1 -- 172.21.15.90:0/3367719493 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2763516956 0 0) 0x7fb8ec10ef70 con 0x7fb8ec10f300 2024-04-14T15:20:00.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:00.483+0000 7fb8d3fff700 1 -- 172.21.15.90:0/3367719493 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb8d40031f0 con 0x7fb8ec10f300 2024-04-14T15:20:00.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:00.483+0000 7fb8d3fff700 1 -- 172.21.15.90:0/3367719493 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1051452369 0 0) 0x7fb8ec1aa0f0 con 0x7fb8ec108880 2024-04-14T15:20:00.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:00.483+0000 7fb8d3fff700 1 -- 172.21.15.90:0/3367719493 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb8ec10ef70 con 0x7fb8ec108880 2024-04-14T15:20:00.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:00.483+0000 7fb8d3fff700 1 -- 172.21.15.90:0/3367719493 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2136619812 0 0) 0x7fb8ec1a8f10 con 0x7fb8ec10b730 2024-04-14T15:20:00.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:00.483+0000 7fb8d3fff700 1 -- 172.21.15.90:0/3367719493 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb8ec1aa0f0 con 0x7fb8ec10b730 2024-04-14T15:20:00.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:00.483+0000 7fb8d3fff700 1 -- 172.21.15.90:0/3367719493 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 232555881 0 0) 0x7fb8d40031f0 con 0x7fb8ec10f300 2024-04-14T15:20:00.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:00.483+0000 7fb8d3fff700 1 -- 172.21.15.90:0/3367719493 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb8ec1a8f10 con 0x7fb8ec10f300 2024-04-14T15:20:00.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:00.483+0000 7fb8d3fff700 1 -- 172.21.15.90:0/3367719493 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fb8e4001950 con 0x7fb8ec10f300 2024-04-14T15:20:00.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:00.483+0000 7fb8d3fff700 1 -- 172.21.15.90:0/3367719493 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3070288664 0 0) 0x7fb8ec10ef70 con 0x7fb8ec108880 2024-04-14T15:20:00.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:00.483+0000 7fb8d3fff700 1 -- 172.21.15.90:0/3367719493 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb8d40031f0 con 0x7fb8ec108880 2024-04-14T15:20:00.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:00.483+0000 7fb8d3fff700 1 -- 172.21.15.90:0/3367719493 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fb8dc001830 con 0x7fb8ec108880 2024-04-14T15:20:00.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:00.484+0000 7fb8d3fff700 1 -- 172.21.15.90:0/3367719493 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 854643118 0 0) 0x7fb8ec1a8f10 con 0x7fb8ec10f300 2024-04-14T15:20:00.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:00.484+0000 7fb8d3fff700 1 -- 172.21.15.90:0/3367719493 >> v1:172.21.15.90:6790/0 conn(0x7fb8ec108880 legacy=0x7fb8ec10bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:00.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:00.484+0000 7fb8d3fff700 1 -- 172.21.15.90:0/3367719493 >> v1:172.21.15.92:6789/0 conn(0x7fb8ec10b730 legacy=0x7fb8ec10c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:00.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:00.484+0000 7fb8d3fff700 1 -- 172.21.15.90:0/3367719493 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8ec1ab2d0 con 0x7fb8ec10f300 2024-04-14T15:20:00.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:00.484+0000 7fb8f1729700 1 -- 172.21.15.90:0/3367719493 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb8ec1aa320 con 0x7fb8ec10f300 2024-04-14T15:20:00.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:00.484+0000 7fb8f1729700 10 client.37437.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:20:00.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:00.484+0000 7fb8f1729700 1 -- 172.21.15.90:0/3367719493 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb8ec1aa7e0 con 0x7fb8ec10f300 2024-04-14T15:20:00.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:00.484+0000 7fb8f1729700 1 librados: init done 2024-04-14T15:20:00.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:00.484+0000 7fb8d3fff700 1 -- 172.21.15.90:0/3367719493 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fb8e4003a00 con 0x7fb8ec10f300 2024-04-14T15:20:00.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:00.484+0000 7fb8d3fff700 1 -- 172.21.15.90:0/3367719493 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fb8e40050c0 con 0x7fb8ec10f300 2024-04-14T15:20:00.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:00.485+0000 7fb8f1729700 1 -- 172.21.15.90:0/3367719493 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8d8005150 con 0x7fb8ec10f300 2024-04-14T15:20:00.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:00.485+0000 7fb8d3fff700 1 -- 172.21.15.90:0/3367719493 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fb8e4005300 con 0x7fb8ec10f300 2024-04-14T15:20:00.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:00.486+0000 7fb8d3fff700 1 -- 172.21.15.90:0/3367719493 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(489..489 src has 51..489) ==== 8197+0+0 (unknown 834265962 0 0) 0x7fb8e408aeb0 con 0x7fb8ec10f300 2024-04-14T15:20:00.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:00.486+0000 7fb8d3fff700 10 client.37437.objecter ms_dispatch 0x7fb8ec001550 osd_map(489..489 src has 51..489) 2024-04-14T15:20:00.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:00.486+0000 7fb8d3fff700 3 client.37437.objecter handle_osd_map got epochs [489,489] > 0 2024-04-14T15:20:00.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:00.486+0000 7fb8d3fff700 3 client.37437.objecter handle_osd_map decoding full epoch 489 2024-04-14T15:20:00.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:00.486+0000 7fb8d3fff700 20 client.37437.objecter dump_active .. 0 homeless 2024-04-14T15:20:00.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:00.486+0000 7fb8d3fff700 10 client.37437.objecter ms_handle_connect 0x7fb8d40712e0 2024-04-14T15:20:00.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:00.491+0000 7fb8d3fff700 1 -- 172.21.15.90:0/3367719493 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fb8e40578c0 con 0x7fb8ec10f300 2024-04-14T15:20:00.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:00 smithi092 ceph-mon[33966]: 4.a deep-scrub starts 2024-04-14T15:20:00.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:00 smithi092 ceph-mon[33966]: 4.a deep-scrub ok 2024-04-14T15:20:00.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:00 smithi092 ceph-mon[33966]: pgmap v700: 258 pgs: 14 creating+activating, 18 unknown, 226 active+clean; 8.4 MiB data, 598 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:20:00.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:00 smithi092 ceph-mon[33966]: osdmap e489: 8 total, 8 up, 8 in 2024-04-14T15:20:00.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:00 smithi092 ceph-mon[33966]: Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-04-14T15:20:00.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:00 smithi092 ceph-mon[33966]: [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-04-14T15:20:00.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:00 smithi092 ceph-mon[33966]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-04-14T15:20:00.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:00 smithi092 ceph-mon[33966]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-04-14T15:20:00.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:00 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:20:00.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:00 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:20:00.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:00 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-77"}]: dispatch 2024-04-14T15:20:00.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:00 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-77"}]: dispatch 2024-04-14T15:20:00.626 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:20:00.650 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:00.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:00.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:20:00.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:00.692+0000 7fb8f1729700 1 -- 172.21.15.90:0/3367719493 --> v1:172.21.15.90:6800/420935427 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb8d8000bc0 con 0x7fb8d40712e0 2024-04-14T15:20:00.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:00.709+0000 7fb8d3fff700 1 -- 172.21.15.90:0/3367719493 <== mgr.14610 v1:172.21.15.90:6800/420935427 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+658064 (unknown 2965378022 0 1435376620) 0x7fb8d8000bc0 con 0x7fb8d40712e0 2024-04-14T15:20:00.717 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:20:00.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:00.714+0000 7fb8f1729700 10 librados: watch_flush enter 2024-04-14T15:20:00.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:00.714+0000 7fb8f1729700 10 librados: watch_flush exit 2024-04-14T15:20:00.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:00.714+0000 7fb8f1729700 20 client.37437.objecter shutdown clearing up homeless session... 2024-04-14T15:20:00.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:00.714+0000 7fb8f1729700 10 client.37437.objecter successfully canceled tick 2024-04-14T15:20:00.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:00.714+0000 7fb8f1729700 1 -- 172.21.15.90:0/3367719493 >> v1:172.21.15.90:6800/420935427 conn(0x7fb8d40712e0 legacy=0x7fb8d40737a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:00.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:00.714+0000 7fb8f1729700 1 -- 172.21.15.90:0/3367719493 >> v1:172.21.15.90:6789/0 conn(0x7fb8ec10f300 legacy=0x7fb8ec1a67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:00.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:00.714+0000 7fb8f1729700 1 -- 172.21.15.90:0/3367719493 shutdown_connections 2024-04-14T15:20:00.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:00.714+0000 7fb8f1729700 1 -- 172.21.15.90:0/3367719493 >> 172.21.15.90:0/3367719493 conn(0x7fb8ec0fe3d0 msgr2=0x7fb8ec112830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:20:00.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:00.714+0000 7fb8f1729700 1 -- 172.21.15.90:0/3367719493 shutdown_connections 2024-04-14T15:20:00.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:00.714+0000 7fb8f1729700 1 -- 172.21.15.90:0/3367719493 wait complete. 2024-04-14T15:20:00.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:00.714+0000 7fb8f1729700 1 librados: shutdown 2024-04-14T15:20:00.724 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-14T15:20:00.752 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:20:00.774 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:00.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:00.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:20:00.876 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:20:00.899 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:00.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:00.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:20:01.001 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:20:01.024 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:01.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:01.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:20:01.125 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:20:01.153 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:01.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:01.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:20:01.255 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:01.278 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:01.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:01.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:01.380 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:20:01.408 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:01.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:01.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:20:01.428 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:01 smithi090 ceph-mon[30399]: 5.f scrub starts 2024-04-14T15:20:01.428 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:01 smithi090 ceph-mon[30399]: 5.f scrub ok 2024-04-14T15:20:01.428 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:01 smithi090 ceph-mon[30399]: 6.12 scrub starts 2024-04-14T15:20:01.428 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:01 smithi090 ceph-mon[30399]: 6.12 scrub ok 2024-04-14T15:20:01.428 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:01 smithi090 ceph-mon[30399]: 5.14 scrub starts 2024-04-14T15:20:01.428 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:01 smithi090 ceph-mon[30399]: 5.14 scrub ok 2024-04-14T15:20:01.428 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:01 smithi090 ceph-mon[30399]: from='client.37437 v1:172.21.15.90:0/3367719493' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:20:01.428 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:01 smithi090 ceph-mon[30399]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:20:01.429 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:01 smithi090 ceph-mon[30399]: osdmap e490: 8 total, 8 up, 8 in 2024-04-14T15:20:01.429 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:01 smithi090 ceph-mon[41670]: 5.f scrub starts 2024-04-14T15:20:01.429 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:01 smithi090 ceph-mon[41670]: 5.f scrub ok 2024-04-14T15:20:01.429 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:01 smithi090 ceph-mon[41670]: 6.12 scrub starts 2024-04-14T15:20:01.429 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:01 smithi090 ceph-mon[41670]: 6.12 scrub ok 2024-04-14T15:20:01.429 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:01 smithi090 ceph-mon[41670]: 5.14 scrub starts 2024-04-14T15:20:01.429 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:01 smithi090 ceph-mon[41670]: 5.14 scrub ok 2024-04-14T15:20:01.429 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:01 smithi090 ceph-mon[41670]: from='client.37437 v1:172.21.15.90:0/3367719493' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:20:01.429 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:01 smithi090 ceph-mon[41670]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:20:01.429 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:01 smithi090 ceph-mon[41670]: osdmap e490: 8 total, 8 up, 8 in 2024-04-14T15:20:01.510 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:20:01.533 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:01.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:01.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:20:01.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:01 smithi092 ceph-mon[33966]: 5.f scrub starts 2024-04-14T15:20:01.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:01 smithi092 ceph-mon[33966]: 5.f scrub ok 2024-04-14T15:20:01.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:01 smithi092 ceph-mon[33966]: 6.12 scrub starts 2024-04-14T15:20:01.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:01 smithi092 ceph-mon[33966]: 6.12 scrub ok 2024-04-14T15:20:01.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:01 smithi092 ceph-mon[33966]: 5.14 scrub starts 2024-04-14T15:20:01.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:01 smithi092 ceph-mon[33966]: 5.14 scrub ok 2024-04-14T15:20:01.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:01 smithi092 ceph-mon[33966]: from='client.37437 v1:172.21.15.90:0/3367719493' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:20:01.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:01 smithi092 ceph-mon[33966]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:20:01.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:01 smithi092 ceph-mon[33966]: osdmap e490: 8 total, 8 up, 8 in 2024-04-14T15:20:01.634 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:01.658 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:01.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:01.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:01.759 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:20:01.787 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:01.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:01.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:20:01.889 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:20:01.915 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:01.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:01.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:20:02.017 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:20:02.044 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:02.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:02.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:20:02.145 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:02.169 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:02.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:02.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:02.270 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:20:02.297 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:02.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:02.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:20:02.398 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:02.422 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:02.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:02.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:02.429 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:02 smithi090 ceph-mon[41670]: 2.1c deep-scrub starts 2024-04-14T15:20:02.429 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:02 smithi090 ceph-mon[41670]: 2.1c deep-scrub ok 2024-04-14T15:20:02.429 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:02 smithi090 ceph-mon[41670]: 110.11 scrub starts 2024-04-14T15:20:02.429 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:02 smithi090 ceph-mon[41670]: 110.11 scrub ok 2024-04-14T15:20:02.429 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:02 smithi090 ceph-mon[41670]: 5.6 scrub starts 2024-04-14T15:20:02.429 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:02 smithi090 ceph-mon[41670]: 5.6 scrub ok 2024-04-14T15:20:02.429 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:02 smithi090 ceph-mon[41670]: 6.1 scrub starts 2024-04-14T15:20:02.429 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:02 smithi090 ceph-mon[41670]: 6.1 scrub ok 2024-04-14T15:20:02.429 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:02 smithi090 ceph-mon[41670]: pgmap v703: 226 pgs: 226 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-14T15:20:02.429 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:02 smithi090 ceph-mon[41670]: osdmap e491: 8 total, 8 up, 8 in 2024-04-14T15:20:02.429 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:02 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:20:02.429 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:02 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:20:02.430 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:02 smithi090 ceph-mon[30399]: 2.1c deep-scrub starts 2024-04-14T15:20:02.430 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:02 smithi090 ceph-mon[30399]: 2.1c deep-scrub ok 2024-04-14T15:20:02.430 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:02 smithi090 ceph-mon[30399]: 110.11 scrub starts 2024-04-14T15:20:02.430 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:02 smithi090 ceph-mon[30399]: 110.11 scrub ok 2024-04-14T15:20:02.430 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:02 smithi090 ceph-mon[30399]: 5.6 scrub starts 2024-04-14T15:20:02.430 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:02 smithi090 ceph-mon[30399]: 5.6 scrub ok 2024-04-14T15:20:02.430 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:02 smithi090 ceph-mon[30399]: 6.1 scrub starts 2024-04-14T15:20:02.430 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:02 smithi090 ceph-mon[30399]: 6.1 scrub ok 2024-04-14T15:20:02.430 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:02 smithi090 ceph-mon[30399]: pgmap v703: 226 pgs: 226 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-14T15:20:02.430 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:02 smithi090 ceph-mon[30399]: osdmap e491: 8 total, 8 up, 8 in 2024-04-14T15:20:02.430 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:02 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:20:02.430 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:02 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:20:02.454 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":701,"stamp":"2024-04-14T15:19:59.979177+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8792299,"num_objects":371,"num_object_clones":9,"num_object_copies":1113,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":370,"num_whiteouts":0,"num_read":1875,"num_read_kb":1525,"num_write":1354,"num_write_kb":12764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":481,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":18,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6867,"ondisk_log_size":6867,"up":774,"acting":774,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":690,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":617140,"kb_used_data":89884,"kb_used_omap":12,"kb_used_meta":526963,"kb_avail":749344076,"statfs":{"total":767960285184,"available":767328333824,"internally_reserved":0,"allocated":92041216,"data_stored":63560192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":539610674},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20,"num_objects":-2,"num_object_clones":2,"num_object_copies":-6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3,"num_whiteouts":0,"num_read":7,"num_read_kb":5,"num_write":0,"num_write_kb":-2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.017912"},"pg_stats":[{"pgid":"304.1f","version":"0'0","reported_seq":4,"reported_epoch":488,"state":"creating+activating","last_fresh":"2024-04-14T15:19:58.983930+0000","last_change":"2024-04-14T15:19:58.978310+0000","last_active":"2024-04-14T15:19:57.962510+0000","last_peered":"2024-04-14T15:19:57.962510+0000","last_clean":"2024-04-14T15:19:57.962510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:19:58.983930+0000","last_undegraded":"2024-04-14T15:19:58.983930+0000","last_fullsized":"2024-04-14T15:19:58.983930+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":487,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:57.962510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:57.962510+0000","last_clean_scrub_stamp":"2024-04-14T15:19:57.962510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":["6","3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"304.1e","version":"0'0","reported_seq":3,"reported_epoch":488,"state":"creating+activating","last_fresh":"2024-04-14T15:19:58.976933+0000","last_change":"2024-04-14T15:19:58.976933+0000","last_active":"2024-04-14T15:19:57.962510+0000","last_peered":"2024-04-14T15:19:57.962510+0000","last_clean":"2024-04-14T15:19:57.962510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:19:58.976933+0000","last_undegraded":"2024-04-14T15:19:58.976933+0000","last_fullsized":"2024-04-14T15:19:58.976933+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":487,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:57.962510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:57.962510+0000","last_clean_scrub_stamp":"2024-04-14T15:19:57.962510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":["6","3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"304.1d","version":"0'0","reported_seq":4,"reported_epoch":488,"state":"creating+activating","last_fresh":"2024-04-14T15:19:59.130262+0000","last_change":"2024-04-14T15:19:59.126421+0000","last_active":"2024-04-14T15:19:57.962510+0000","last_peered":"2024-04-14T15:19:57.962510+0000","last_clean":"2024-04-14T15:19:57.962510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:19:59.130262+0000","last_undegraded":"2024-04-14T15:19:59.130262+0000","last_fullsized":"2024-04-14T15:19:59.130262+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":487,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:57.962510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:57.962510+0000","last_clean_scrub_stamp":"2024-04-14T15:19:57.962510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":["2","0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"304.1c","version":"0'0","reported_seq":4,"reported_epoch":488,"state":"creating+activating","last_fresh":"2024-04-14T15:19:59.128704+0000","last_change":"2024-04-14T15:19:59.125795+0000","last_active":"2024-04-14T15:19:57.962510+0000","last_peered":"2024-04-14T15:19:57.962510+0000","last_clean":"2024-04-14T15:19:57.962510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:19:59.128704+0000","last_undegraded":"2024-04-14T15:19:59.128704+0000","last_fullsized":"2024-04-14T15:19:59.128704+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":487,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:57.962510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:57.962510+0000","last_clean_scrub_stamp":"2024-04-14T15:19:57.962510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":["2","1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"304.1b","version":"0'0","reported_seq":3,"reported_epoch":488,"state":"creating+activating","last_fresh":"2024-04-14T15:19:58.979662+0000","last_change":"2024-04-14T15:19:58.979662+0000","last_active":"2024-04-14T15:19:57.962510+0000","last_peered":"2024-04-14T15:19:57.962510+0000","last_clean":"2024-04-14T15:19:57.962510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:19:58.979662+0000","last_undegraded":"2024-04-14T15:19:58.979662+0000","last_fullsized":"2024-04-14T15:19:58.979662+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":487,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:57.962510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:57.962510+0000","last_clean_scrub_stamp":"2024-04-14T15:19:57.962510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":["6","3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"304.1a","version":"0'0","reported_seq":3,"reported_epoch":488,"state":"creating+activating","last_fresh":"2024-04-14T15:19:59.123440+0000","last_change":"2024-04-14T15:19:59.123440+0000","last_active":"2024-04-14T15:19:57.962510+0000","last_peered":"2024-04-14T15:19:57.962510+0000","last_clean":"2024-04-14T15:19:57.962510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:19:59.123440+0000","last_undegraded":"2024-04-14T15:19:59.123440+0000","last_fullsized":"2024-04-14T15:19:59.123440+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":487,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:57.962510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:57.962510+0000","last_clean_scrub_stamp":"2024-04-14T15:19:57.962510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":["1","2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"304.19","version":"0'0","reported_seq":3,"reported_epoch":488,"state":"creating+activating","last_fresh":"2024-04-14T15:19:58.971277+0000","last_change":"2024-04-14T15:19:58.971277+0000","last_active":"2024-04-14T15:19:57.962510+0000","last_peered":"2024-04-14T15:19:57.962510+0000","last_clean":"2024-04-14T15:19:57.962510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:19:58.971277+0000","last_undegraded":"2024-04-14T15:19:58.971277+0000","last_fullsized":"2024-04-14T15:19:58.971277+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":487,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:57.962510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:57.962510+0000","last_clean_scrub_stamp":"2024-04-14T15:19:57.962510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":["1","3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"304.18","version":"0'0","reported_seq":3,"reported_epoch":488,"state":"creating+activating","last_fresh":"2024-04-14T15:19:58.977230+0000","last_change":"2024-04-14T15:19:58.977230+0000","last_active":"2024-04-14T15:19:57.962510+0000","last_peered":"2024-04-14T15:19:57.962510+0000","last_clean":"2024-04-14T15:19:57.962510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:19:58.977230+0000","last_undegraded":"2024-04-14T15:19:58.977230+0000","last_fullsized":"2024-04-14T15:19:58.977230+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":487,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:57.962510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:57.962510+0000","last_clean_scrub_stamp":"2024-04-14T15:19:57.962510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":["7","2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"304.17","version":"0'0","reported_seq":3,"reported_epoch":488,"state":"creating+activating","last_fresh":"2024-04-14T15:19:58.971431+0000","last_change":"2024-04-14T15:19:58.971431+0000","last_active":"2024-04-14T15:19:57.962510+0000","last_peered":"2024-04-14T15:19:57.962510+0000","last_clean":"2024-04-14T15:19:57.962510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:19:58.971431+0000","last_undegraded":"2024-04-14T15:19:58.971431+0000","last_fullsized":"2024-04-14T15:19:58.971431+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":487,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:57.962510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:57.962510+0000","last_clean_scrub_stamp":"2024-04-14T15:19:57.962510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":["4","2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"304.16","version":"0'0","reported_seq":3,"reported_epoch":488,"state":"creating+activating","last_fresh":"2024-04-14T15:19:58.971433+0000","last_change":"2024-04-14T15:19:58.971433+0000","last_active":"2024-04-14T15:19:57.962510+0000","last_peered":"2024-04-14T15:19:57.962510+0000","last_clean":"2024-04-14T15:19:57.962510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:19:58.971433+0000","last_undegraded":"2024-04-14T15:19:58.971433+0000","last_fullsized":"2024-04-14T15:19:58.971433+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":487,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:57.962510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:57.962510+0000","last_clean_scrub_stamp":"2024-04-14T15:19:57.962510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":["4","0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"304.15","version":"0'0","reported_seq":3,"reported_epoch":488,"state":"creating+activating","last_fresh":"2024-04-14T15:19:58.983115+0000","last_change":"2024-04-14T15:19:58.983115+0000","last_active":"2024-04-14T15:19:57.962510+0000","last_peered":"2024-04-14T15:19:57.962510+0000","last_clean":"2024-04-14T15:19:57.962510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:19:58.983115+0000","last_undegraded":"2024-04-14T15:19:58.983115+0000","last_fullsized":"2024-04-14T15:19:58.983115+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":487,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:57.962510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:57.962510+0000","last_clean_scrub_stamp":"2024-04-14T15:19:57.962510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":["7","0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"304.14","version":"0'0","reported_seq":4,"reported_epoch":488,"state":"creating+activating","last_fresh":"2024-04-14T15:19:59.129212+0000","last_change":"2024-04-14T15:19:59.125452+0000","last_active":"2024-04-14T15:19:57.962510+0000","last_peered":"2024-04-14T15:19:57.962510+0000","last_clean":"2024-04-14T15:19:57.962510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:19:59.129212+0000","last_undegraded":"2024-04-14T15:19:59.129212+0000","last_fullsized":"2024-04-14T15:19:59.129212+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":487,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:57.962510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:57.962510+0000","last_clean_scrub_stamp":"2024-04-14T15:19:57.962510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":["2","1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"304.13","version":"0'0","reported_seq":3,"reported_epoch":488,"state":"creating+activating","last_fresh":"2024-04-14T15:19:58.973818+0000","last_change":"2024-04-14T15:19:58.973818+0000","last_active":"2024-04-14T15:19:57.962510+0000","last_peered":"2024-04-14T15:19:57.962510+0000","last_clean":"2024-04-14T15:19:57.962510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:19:58.973818+0000","last_undegraded":"2024-04-14T15:19:58.973818+0000","last_fullsized":"2024-04-14T15:19:58.973818+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":487,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:57.962510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:57.962510+0000","last_clean_scrub_stamp":"2024-04-14T15:19:57.962510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":["2","0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"304.12","version":"0'0","reported_seq":4,"reported_epoch":488,"state":"creating+activating","last_fresh":"2024-04-14T15:19:58.982889+0000","last_change":"2024-04-14T15:19:58.978699+0000","last_active":"2024-04-14T15:19:57.962510+0000","last_peered":"2024-04-14T15:19:57.962510+0000","last_clean":"2024-04-14T15:19:57.962510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:19:58.982889+0000","last_undegraded":"2024-04-14T15:19:58.982889+0000","last_fullsized":"2024-04-14T15:19:58.982889+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":487,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:57.962510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:57.962510+0000","last_clean_scrub_stamp":"2024-04-14T15:19:57.962510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":["6","2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"304.11","version":"0'0","reported_seq":3,"reported_epoch":488,"state":"creating+activating","last_fresh":"2024-04-14T15:19:58.977741+0000","last_change":"2024-04-14T15:19:58.977741+0000","last_active":"2024-04-14T15:19:57.962510+0000","last_peered":"2024-04-14T15:19:57.962510+0000","last_clean":"2024-04-14T15:19:57.962510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:19:58.977741+0000","last_undegraded":"2024-04-14T15:19:58.977741+0000","last_fullsized":"2024-04-14T15:19:58.977741+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":487,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:57.962510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:57.962510+0000","last_clean_scrub_stamp":"2024-04-14T15:19:57.962510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":["7","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"304.10","version":"0'0","reported_seq":9,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.134955+0000","last_change":"2024-04-14T15:19:59.134955+0000","last_active":"2024-04-14T15:19:59.134955+0000","last_peered":"2024-04-14T15:19:59.134955+0000","last_clean":"2024-04-14T15:19:59.134955+0000","last_became_active":"2024-04-14T15:19:59.132908+0000","last_became_peered":"2024-04-14T15:19:59.132908+0000","last_unstale":"2024-04-14T15:19:59.134955+0000","last_undegraded":"2024-04-14T15:19:59.134955+0000","last_fullsized":"2024-04-14T15:19:59.134955+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":487,"last_epoch_clean":488,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:57.962510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:57.962510+0000","last_clean_scrub_stamp":"2024-04-14T15:19:57.962510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:14.499183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"304.f","version":"0'0","reported_seq":3,"reported_epoch":488,"state":"creating+activating","last_fresh":"2024-04-14T15:19:58.982394+0000","last_change":"2024-04-14T15:19:58.982394+0000","last_active":"2024-04-14T15:19:57.962510+0000","last_peered":"2024-04-14T15:19:57.962510+0000","last_clean":"2024-04-14T15:19:57.962510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:19:58.982394+0000","last_undegraded":"2024-04-14T15:19:58.982394+0000","last_fullsized":"2024-04-14T15:19:58.982394+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":487,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:57.962510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:57.962510+0000","last_clean_scrub_stamp":"2024-04-14T15:19:57.962510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":["7","1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"304.e","version":"0'0","reported_seq":4,"reported_epoch":488,"state":"creating+activating","last_fresh":"2024-04-14T15:19:59.127831+0000","last_change":"2024-04-14T15:19:59.124204+0000","last_active":"2024-04-14T15:19:57.962510+0000","last_peered":"2024-04-14T15:19:57.962510+0000","last_clean":"2024-04-14T15:19:57.962510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:19:59.127831+0000","last_undegraded":"2024-04-14T15:19:59.127831+0000","last_fullsized":"2024-04-14T15:19:59.127831+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":487,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:57.962510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:57.962510+0000","last_clean_scrub_stamp":"2024-04-14T15:19:57.962510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":["0","4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"304.d","version":"0'0","reported_seq":4,"reported_epoch":488,"state":"creating+activating","last_fresh":"2024-04-14T15:19:58.984022+0000","last_change":"2024-04-14T15:19:58.981847+0000","last_active":"2024-04-14T15:19:57.962510+0000","last_peered":"2024-04-14T15:19:57.962510+0000","last_clean":"2024-04-14T15:19:57.962510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:19:58.984022+0000","last_undegraded":"2024-04-14T15:19:58.984022+0000","last_fullsized":"2024-04-14T15:19:58.984022+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":487,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:57.962510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:57.962510+0000","last_clean_scrub_stamp":"2024-04-14T15:19:57.962510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":["7","1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"304.c","version":"0'0","reported_seq":10,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.132885+0000","last_change":"2024-04-14T15:19:59.132250+0000","last_active":"2024-04-14T15:19:59.132885+0000","last_peered":"2024-04-14T15:19:59.132885+0000","last_clean":"2024-04-14T15:19:59.132885+0000","last_became_active":"2024-04-14T15:19:59.131673+0000","last_became_peered":"2024-04-14T15:19:59.131673+0000","last_unstale":"2024-04-14T15:19:59.132885+0000","last_undegraded":"2024-04-14T15:19:59.132885+0000","last_fullsized":"2024-04-14T15:19:59.132885+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":487,"last_epoch_clean":488,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:57.962510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:57.962510+0000","last_clean_scrub_stamp":"2024-04-14T15:19:57.962510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:12.982186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"304.b","version":"0'0","reported_seq":4,"reported_epoch":488,"state":"creating+activating","last_fresh":"2024-04-14T15:19:59.131646+0000","last_change":"2024-04-14T15:19:58.973758+0000","last_active":"2024-04-14T15:19:57.962510+0000","last_peered":"2024-04-14T15:19:57.962510+0000","last_clean":"2024-04-14T15:19:57.962510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:19:59.131646+0000","last_undegraded":"2024-04-14T15:19:59.131646+0000","last_fullsized":"2024-04-14T15:19:59.131646+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":487,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:57.962510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:57.962510+0000","last_clean_scrub_stamp":"2024-04-14T15:19:57.962510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":["3","1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"304.a","version":"0'0","reported_seq":10,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.128172+0000","last_change":"2024-04-14T15:19:59.126365+0000","last_active":"2024-04-14T15:19:59.128172+0000","last_peered":"2024-04-14T15:19:59.128172+0000","last_clean":"2024-04-14T15:19:59.128172+0000","last_became_active":"2024-04-14T15:19:59.126072+0000","last_became_peered":"2024-04-14T15:19:59.126072+0000","last_unstale":"2024-04-14T15:19:59.128172+0000","last_undegraded":"2024-04-14T15:19:59.128172+0000","last_fullsized":"2024-04-14T15:19:59.128172+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":487,"last_epoch_clean":488,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:57.962510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:57.962510+0000","last_clean_scrub_stamp":"2024-04-14T15:19:57.962510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:12.744007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"304.9","version":"0'0","reported_seq":3,"reported_epoch":488,"state":"creating+activating","last_fresh":"2024-04-14T15:19:58.973816+0000","last_change":"2024-04-14T15:19:58.973816+0000","last_active":"2024-04-14T15:19:57.962510+0000","last_peered":"2024-04-14T15:19:57.962510+0000","last_clean":"2024-04-14T15:19:57.962510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:19:58.973816+0000","last_undegraded":"2024-04-14T15:19:58.973816+0000","last_fullsized":"2024-04-14T15:19:58.973816+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":487,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:57.962510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:57.962510+0000","last_clean_scrub_stamp":"2024-04-14T15:19:57.962510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":["2","3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"304.8","version":"0'0","reported_seq":3,"reported_epoch":488,"state":"creating+activating","last_fresh":"2024-04-14T15:19:59.123160+0000","last_change":"2024-04-14T15:19:59.123160+0000","last_active":"2024-04-14T15:19:57.962510+0000","last_peered":"2024-04-14T15:19:57.962510+0000","last_clean":"2024-04-14T15:19:57.962510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:19:59.123160+0000","last_undegraded":"2024-04-14T15:19:59.123160+0000","last_fullsized":"2024-04-14T15:19:59.123160+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":487,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:57.962510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:57.962510+0000","last_clean_scrub_stamp":"2024-04-14T15:19:57.962510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":["2","1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"304.7","version":"0'0","reported_seq":4,"reported_epoch":488,"state":"creating+activating","last_fresh":"2024-04-14T15:19:59.130292+0000","last_change":"2024-04-14T15:19:59.128683+0000","last_active":"2024-04-14T15:19:57.962510+0000","last_peered":"2024-04-14T15:19:57.962510+0000","last_clean":"2024-04-14T15:19:57.962510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:19:59.130292+0000","last_undegraded":"2024-04-14T15:19:59.130292+0000","last_fullsized":"2024-04-14T15:19:59.130292+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":487,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:57.962510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:57.962510+0000","last_clean_scrub_stamp":"2024-04-14T15:19:57.962510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":["2","0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"304.6","version":"0'0","reported_seq":4,"reported_epoch":488,"state":"creating+activating","last_fresh":"2024-04-14T15:19:58.981008+0000","last_change":"2024-04-14T15:19:58.972000+0000","last_active":"2024-04-14T15:19:57.962510+0000","last_peered":"2024-04-14T15:19:57.962510+0000","last_clean":"2024-04-14T15:19:57.962510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:19:58.981008+0000","last_undegraded":"2024-04-14T15:19:58.981008+0000","last_fullsized":"2024-04-14T15:19:58.981008+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":487,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:57.962510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:57.962510+0000","last_clean_scrub_stamp":"2024-04-14T15:19:57.962510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":["4","1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"304.5","version":"0'0","reported_seq":4,"reported_epoch":488,"state":"creating+activating","last_fresh":"2024-04-14T15:19:58.980863+0000","last_change":"2024-04-14T15:19:58.971579+0000","last_active":"2024-04-14T15:19:57.962510+0000","last_peered":"2024-04-14T15:19:57.962510+0000","last_clean":"2024-04-14T15:19:57.962510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:19:58.980863+0000","last_undegraded":"2024-04-14T15:19:58.980863+0000","last_fullsized":"2024-04-14T15:19:58.980863+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":487,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:57.962510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:57.962510+0000","last_clean_scrub_stamp":"2024-04-14T15:19:57.962510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":["4","5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"304.4","version":"0'0","reported_seq":5,"reported_epoch":488,"state":"creating+activating","last_fresh":"2024-04-14T15:19:59.130213+0000","last_change":"2024-04-14T15:19:59.124872+0000","last_active":"2024-04-14T15:19:57.962510+0000","last_peered":"2024-04-14T15:19:57.962510+0000","last_clean":"2024-04-14T15:19:57.962510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:19:59.130213+0000","last_undegraded":"2024-04-14T15:19:59.130213+0000","last_fullsized":"2024-04-14T15:19:59.130213+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":487,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:57.962510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:57.962510+0000","last_clean_scrub_stamp":"2024-04-14T15:19:57.962510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":["2","3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"304.3","version":"0'0","reported_seq":3,"reported_epoch":488,"state":"creating+activating","last_fresh":"2024-04-14T15:19:59.126556+0000","last_change":"2024-04-14T15:19:59.126556+0000","last_active":"2024-04-14T15:19:57.962510+0000","last_peered":"2024-04-14T15:19:57.962510+0000","last_clean":"2024-04-14T15:19:57.962510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:19:59.126556+0000","last_undegraded":"2024-04-14T15:19:59.126556+0000","last_fullsized":"2024-04-14T15:19:59.126556+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":487,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:57.962510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:57.962510+0000","last_clean_scrub_stamp":"2024-04-14T15:19:57.962510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":["0","1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"304.2","version":"0'0","reported_seq":4,"reported_epoch":488,"state":"creating+activating","last_fresh":"2024-04-14T15:19:58.981607+0000","last_change":"2024-04-14T15:19:58.978007+0000","last_active":"2024-04-14T15:19:57.962510+0000","last_peered":"2024-04-14T15:19:57.962510+0000","last_clean":"2024-04-14T15:19:57.962510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:19:58.981607+0000","last_undegraded":"2024-04-14T15:19:58.981607+0000","last_fullsized":"2024-04-14T15:19:58.981607+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":487,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:57.962510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:57.962510+0000","last_clean_scrub_stamp":"2024-04-14T15:19:57.962510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":["6","2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"304.1","version":"0'0","reported_seq":4,"reported_epoch":488,"state":"creating+activating","last_fresh":"2024-04-14T15:19:59.125805+0000","last_change":"2024-04-14T15:19:58.971198+0000","last_active":"2024-04-14T15:19:57.962510+0000","last_peered":"2024-04-14T15:19:57.962510+0000","last_clean":"2024-04-14T15:19:57.962510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:19:59.125805+0000","last_undegraded":"2024-04-14T15:19:59.125805+0000","last_fullsized":"2024-04-14T15:19:59.125805+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":487,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:57.962510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:57.962510+0000","last_clean_scrub_stamp":"2024-04-14T15:19:57.962510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":["0","5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"304.0","version":"0'0","reported_seq":4,"reported_epoch":488,"state":"creating+activating","last_fresh":"2024-04-14T15:19:59.126553+0000","last_change":"2024-04-14T15:19:58.973791+0000","last_active":"2024-04-14T15:19:57.962510+0000","last_peered":"2024-04-14T15:19:57.962510+0000","last_clean":"2024-04-14T15:19:57.962510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:19:59.126553+0000","last_undegraded":"2024-04-14T15:19:59.126553+0000","last_fullsized":"2024-04-14T15:19:59.126553+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":487,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:57.962510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:57.962510+0000","last_clean_scrub_stamp":"2024-04-14T15:19:57.962510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":["2","3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"110.11","version":"403'41","reported_seq":987,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.981968+0000","last_change":"2024-04-14T15:18:54.159422+0000","last_active":"2024-04-14T15:19:58.981968+0000","last_peered":"2024-04-14T15:19:58.981968+0000","last_clean":"2024-04-14T15:19:58.981968+0000","last_became_active":"2024-04-14T15:13:09.936619+0000","last_became_peered":"2024-04-14T15:13:09.936619+0000","last_unstale":"2024-04-14T15:19:58.981968+0000","last_undegraded":"2024-04-14T15:19:58.981968+0000","last_fullsized":"2024-04-14T15:19:58.981968+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"403'41","last_scrub_stamp":"2024-04-14T15:17:56.911004+0000","last_deep_scrub":"403'41","last_deep_scrub_stamp":"2024-04-14T15:17:56.911004+0000","last_clean_scrub_stamp":"2024-04-14T15:17:56.911004+0000","objects_scrubbed":2,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0.00061958300000000005,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"110.10","version":"253'158","reported_seq":1069,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.971291+0000","last_change":"2024-04-14T15:18:54.158668+0000","last_active":"2024-04-14T15:19:58.971291+0000","last_peered":"2024-04-14T15:19:58.971291+0000","last_clean":"2024-04-14T15:19:58.971291+0000","last_became_active":"2024-04-14T15:13:09.928915+0000","last_became_peered":"2024-04-14T15:13:09.928915+0000","last_unstale":"2024-04-14T15:19:58.971291+0000","last_undegraded":"2024-04-14T15:19:58.971291+0000","last_fullsized":"2024-04-14T15:19:58.971291+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"253'158","last_scrub_stamp":"2024-04-14T15:17:34.291854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:34.291854+0000","objects_scrubbed":10,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00045995200000000002,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":47,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"110.13","version":"477'53","reported_seq":997,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-04-14T15:19:57.976413+0000","last_change":"2024-04-14T15:18:54.157558+0000","last_active":"2024-04-14T15:19:57.976413+0000","last_peered":"2024-04-14T15:19:57.976413+0000","last_clean":"2024-04-14T15:19:57.976413+0000","last_became_active":"2024-04-14T15:13:09.918152+0000","last_became_peered":"2024-04-14T15:13:09.918152+0000","last_unstale":"2024-04-14T15:19:57.976413+0000","last_undegraded":"2024-04-14T15:19:57.976413+0000","last_fullsized":"2024-04-14T15:19:57.976413+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'39","last_scrub_stamp":"2024-04-14T15:17:44.873766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:44.873766+0000","objects_scrubbed":3,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001942764,"stat_sum":{"num_bytes":57,"num_objects":7,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"110.12","version":"409'70","reported_seq":1020,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-04-14T15:19:57.975640+0000","last_change":"2024-04-14T15:19:47.921780+0000","last_active":"2024-04-14T15:19:57.975640+0000","last_peered":"2024-04-14T15:19:57.975640+0000","last_clean":"2024-04-14T15:19:57.975640+0000","last_became_active":"2024-04-14T15:13:09.939566+0000","last_became_peered":"2024-04-14T15:13:09.939566+0000","last_unstale":"2024-04-14T15:19:57.975640+0000","last_undegraded":"2024-04-14T15:19:57.975640+0000","last_fullsized":"2024-04-14T15:19:57.975640+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"409'70","last_scrub_stamp":"2024-04-14T15:19:47.921684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:19:47.921684+0000","objects_scrubbed":9,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:16.596255+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.000970527,"stat_sum":{"num_bytes":74,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":24,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"110.15","version":"190'34","reported_seq":982,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.122141+0000","last_change":"2024-04-14T15:18:54.155028+0000","last_active":"2024-04-14T15:19:59.122141+0000","last_peered":"2024-04-14T15:19:59.122141+0000","last_clean":"2024-04-14T15:19:59.122141+0000","last_became_active":"2024-04-14T15:13:09.960121+0000","last_became_peered":"2024-04-14T15:13:09.960121+0000","last_unstale":"2024-04-14T15:19:59.122141+0000","last_undegraded":"2024-04-14T15:19:59.122141+0000","last_fullsized":"2024-04-14T15:19:59.122141+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'34","last_scrub_stamp":"2024-04-14T15:18:36.013135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:18:36.013135+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0029213049999999999,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"110.14","version":"355'87","reported_seq":1028,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.122669+0000","last_change":"2024-04-14T15:18:54.151534+0000","last_active":"2024-04-14T15:19:59.122669+0000","last_peered":"2024-04-14T15:19:59.122669+0000","last_clean":"2024-04-14T15:19:59.122669+0000","last_became_active":"2024-04-14T15:13:09.899561+0000","last_became_peered":"2024-04-14T15:13:09.899561+0000","last_unstale":"2024-04-14T15:19:59.122669+0000","last_undegraded":"2024-04-14T15:19:59.122669+0000","last_fullsized":"2024-04-14T15:19:59.122669+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'87","last_scrub_stamp":"2024-04-14T15:18:31.117033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:18:31.117033+0000","objects_scrubbed":7,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0014558730000000001,"stat_sum":{"num_bytes":71,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"110.17","version":"456'12","reported_seq":975,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.971547+0000","last_change":"2024-04-14T15:19:48.225230+0000","last_active":"2024-04-14T15:19:58.971547+0000","last_peered":"2024-04-14T15:19:58.971547+0000","last_clean":"2024-04-14T15:19:58.971547+0000","last_became_active":"2024-04-14T15:13:09.898521+0000","last_became_peered":"2024-04-14T15:13:09.898521+0000","last_unstale":"2024-04-14T15:19:58.971547+0000","last_undegraded":"2024-04-14T15:19:58.971547+0000","last_fullsized":"2024-04-14T15:19:58.971547+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'12","last_scrub_stamp":"2024-04-14T15:19:48.225130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:19:48.225130+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:50.368701+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0018681640000000001,"stat_sum":{"num_bytes":24,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"110.16","version":"355'35","reported_seq":815,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.132494+0000","last_change":"2024-04-14T15:18:54.160925+0000","last_active":"2024-04-14T15:19:59.132494+0000","last_peered":"2024-04-14T15:19:59.132494+0000","last_clean":"2024-04-14T15:19:59.132494+0000","last_became_active":"2024-04-14T15:15:21.889783+0000","last_became_peered":"2024-04-14T15:15:21.889783+0000","last_unstale":"2024-04-14T15:19:59.132494+0000","last_undegraded":"2024-04-14T15:19:59.132494+0000","last_fullsized":"2024-04-14T15:19:59.132494+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'35","last_scrub_stamp":"2024-04-14T15:17:34.097563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:34.097563+0000","objects_scrubbed":3,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000499903,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"110.19","version":"466'102","reported_seq":1048,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.128910+0000","last_change":"2024-04-14T15:19:12.058233+0000","last_active":"2024-04-14T15:19:59.128910+0000","last_peered":"2024-04-14T15:19:59.128910+0000","last_clean":"2024-04-14T15:19:59.128910+0000","last_became_active":"2024-04-14T15:13:09.998139+0000","last_became_peered":"2024-04-14T15:13:09.998139+0000","last_unstale":"2024-04-14T15:19:59.128910+0000","last_undegraded":"2024-04-14T15:19:59.128910+0000","last_fullsized":"2024-04-14T15:19:59.128910+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'85","last_scrub_stamp":"2024-04-14T15:19:12.058127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:19:12.058127+0000","objects_scrubbed":5,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:33.609699+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0.0012903700000000001,"stat_sum":{"num_bytes":75,"num_objects":7,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":33,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"110.18","version":"387'94","reported_seq":1034,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.970956+0000","last_change":"2024-04-14T15:18:54.150377+0000","last_active":"2024-04-14T15:19:58.970956+0000","last_peered":"2024-04-14T15:19:58.970956+0000","last_clean":"2024-04-14T15:19:58.970956+0000","last_became_active":"2024-04-14T15:13:09.998803+0000","last_became_peered":"2024-04-14T15:13:09.998803+0000","last_unstale":"2024-04-14T15:19:58.970956+0000","last_undegraded":"2024-04-14T15:19:58.970956+0000","last_fullsized":"2024-04-14T15:19:58.970956+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'94","last_scrub_stamp":"2024-04-14T15:17:36.005598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:36.005598+0000","objects_scrubbed":8,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0013897930000000001,"stat_sum":{"num_bytes":65,"num_objects":8,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":29,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"110.1b","version":"485'106","reported_seq":1043,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-04-14T15:19:57.974392+0000","last_change":"2024-04-14T15:18:54.156913+0000","last_active":"2024-04-14T15:19:57.974392+0000","last_peered":"2024-04-14T15:19:57.974392+0000","last_clean":"2024-04-14T15:19:57.974392+0000","last_became_active":"2024-04-14T15:13:09.946919+0000","last_became_peered":"2024-04-14T15:13:09.946919+0000","last_unstale":"2024-04-14T15:19:57.974392+0000","last_undegraded":"2024-04-14T15:19:57.974392+0000","last_fullsized":"2024-04-14T15:19:57.974392+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'91","last_scrub_stamp":"2024-04-14T15:17:47.889364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:47.889364+0000","objects_scrubbed":6,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001643863,"stat_sum":{"num_bytes":93,"num_objects":9,"num_object_clones":2,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":33,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"110.1a","version":"387'39","reported_seq":991,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.979634+0000","last_change":"2024-04-14T15:18:54.160775+0000","last_active":"2024-04-14T15:19:58.979634+0000","last_peered":"2024-04-14T15:19:58.979634+0000","last_clean":"2024-04-14T15:19:58.979634+0000","last_became_active":"2024-04-14T15:13:09.960319+0000","last_became_peered":"2024-04-14T15:13:09.960319+0000","last_unstale":"2024-04-14T15:19:58.979634+0000","last_undegraded":"2024-04-14T15:19:58.979634+0000","last_fullsized":"2024-04-14T15:19:58.979634+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'39","last_scrub_stamp":"2024-04-14T15:17:53.871590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:53.871590+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0018412890000000001,"stat_sum":{"num_bytes":62,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"110.1d","version":"225'85","reported_seq":1026,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.982914+0000","last_change":"2024-04-14T15:18:54.151417+0000","last_active":"2024-04-14T15:19:58.982914+0000","last_peered":"2024-04-14T15:19:58.982914+0000","last_clean":"2024-04-14T15:19:58.982914+0000","last_became_active":"2024-04-14T15:13:09.868082+0000","last_became_peered":"2024-04-14T15:13:09.868082+0000","last_unstale":"2024-04-14T15:19:58.982914+0000","last_undegraded":"2024-04-14T15:19:58.982914+0000","last_fullsized":"2024-04-14T15:19:58.982914+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"225'85","last_scrub_stamp":"2024-04-14T15:17:36.419088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:36.419088+0000","objects_scrubbed":5,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0010543219999999999,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"110.1c","version":"430'95","reported_seq":1037,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-04-14T15:19:57.976381+0000","last_change":"2024-04-14T15:19:10.839468+0000","last_active":"2024-04-14T15:19:57.976381+0000","last_peered":"2024-04-14T15:19:57.976381+0000","last_clean":"2024-04-14T15:19:57.976381+0000","last_became_active":"2024-04-14T15:13:09.954091+0000","last_became_peered":"2024-04-14T15:13:09.954091+0000","last_unstale":"2024-04-14T15:19:57.976381+0000","last_undegraded":"2024-04-14T15:19:57.976381+0000","last_fullsized":"2024-04-14T15:19:57.976381+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'95","last_scrub_stamp":"2024-04-14T15:19:10.839339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:19:10.839339+0000","objects_scrubbed":7,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:31.174397+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0011789070000000001,"stat_sum":{"num_bytes":65,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"110.1f","version":"466'27","reported_seq":1031,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.123657+0000","last_change":"2024-04-14T15:19:25.078771+0000","last_active":"2024-04-14T15:19:59.123657+0000","last_peered":"2024-04-14T15:19:59.123657+0000","last_clean":"2024-04-14T15:19:59.123657+0000","last_became_active":"2024-04-14T15:13:09.999629+0000","last_became_peered":"2024-04-14T15:13:09.999629+0000","last_unstale":"2024-04-14T15:19:59.123657+0000","last_undegraded":"2024-04-14T15:19:59.123657+0000","last_fullsized":"2024-04-14T15:19:59.123657+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"466'27","last_scrub_stamp":"2024-04-14T15:19:25.078678+0000","last_deep_scrub":"466'27","last_deep_scrub_stamp":"2024-04-14T15:19:25.078678+0000","last_clean_scrub_stamp":"2024-04-14T15:19:25.078678+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:48.690407+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.001392454,"stat_sum":{"num_bytes":4194315,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":8,"num_write_kb":8198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"110.1e","version":"355'23","reported_seq":976,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.125635+0000","last_change":"2024-04-14T15:19:17.364851+0000","last_active":"2024-04-14T15:19:59.125635+0000","last_peered":"2024-04-14T15:19:59.125635+0000","last_clean":"2024-04-14T15:19:59.125635+0000","last_became_active":"2024-04-14T15:13:09.955932+0000","last_became_peered":"2024-04-14T15:13:09.955932+0000","last_unstale":"2024-04-14T15:19:59.125635+0000","last_undegraded":"2024-04-14T15:19:59.125635+0000","last_fullsized":"2024-04-14T15:19:59.125635+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'23","last_scrub_stamp":"2024-04-14T15:19:17.364769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:19:17.364769+0000","objects_scrubbed":2,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:33.534158+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0019783700000000001,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":8,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"110.1","version":"406'32","reported_seq":988,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.978329+0000","last_change":"2024-04-14T15:18:54.150210+0000","last_active":"2024-04-14T15:19:58.978329+0000","last_peered":"2024-04-14T15:19:58.978329+0000","last_clean":"2024-04-14T15:19:58.978329+0000","last_became_active":"2024-04-14T15:13:09.929562+0000","last_became_peered":"2024-04-14T15:13:09.929562+0000","last_unstale":"2024-04-14T15:19:58.978329+0000","last_undegraded":"2024-04-14T15:19:58.978329+0000","last_fullsized":"2024-04-14T15:19:58.978329+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"406'32","last_scrub_stamp":"2024-04-14T15:18:38.685469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:18:38.685469+0000","objects_scrubbed":7,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:19:59.768177+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00089761599999999995,"stat_sum":{"num_bytes":4000050,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":15,"num_write_kb":3921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"110.0","version":"219'85","reported_seq":1021,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.971315+0000","last_change":"2024-04-14T15:18:54.159255+0000","last_active":"2024-04-14T15:19:58.971315+0000","last_peered":"2024-04-14T15:19:58.971315+0000","last_clean":"2024-04-14T15:19:58.971315+0000","last_became_active":"2024-04-14T15:13:09.856407+0000","last_became_peered":"2024-04-14T15:13:09.856407+0000","last_unstale":"2024-04-14T15:19:58.971315+0000","last_undegraded":"2024-04-14T15:19:58.971315+0000","last_fullsized":"2024-04-14T15:19:58.971315+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"219'85","last_scrub_stamp":"2024-04-14T15:17:51.438149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:51.438149+0000","objects_scrubbed":5,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00052820199999999997,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"110.3","version":"464'56","reported_seq":1037,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.122437+0000","last_change":"2024-04-14T15:19:35.504089+0000","last_active":"2024-04-14T15:19:59.122437+0000","last_peered":"2024-04-14T15:19:59.122437+0000","last_clean":"2024-04-14T15:19:59.122437+0000","last_became_active":"2024-04-14T15:19:35.486976+0000","last_became_peered":"2024-04-14T15:19:35.486976+0000","last_unstale":"2024-04-14T15:19:59.122437+0000","last_undegraded":"2024-04-14T15:19:59.122437+0000","last_fullsized":"2024-04-14T15:19:59.122437+0000","mapping_epoch":470,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":471,"parent":"0.0","parent_split_bits":0,"last_scrub":"464'56","last_scrub_stamp":"2024-04-14T15:19:17.063716+0000","last_deep_scrub":"464'56","last_deep_scrub_stamp":"2024-04-14T15:19:17.063716+0000","last_clean_scrub_stamp":"2024-04-14T15:19:17.063716+0000","objects_scrubbed":4,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:46.465261+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0019291829999999999,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":41,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"110.2","version":"240'55","reported_seq":1003,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.123776+0000","last_change":"2024-04-14T15:18:54.152834+0000","last_active":"2024-04-14T15:19:59.123776+0000","last_peered":"2024-04-14T15:19:59.123776+0000","last_clean":"2024-04-14T15:19:59.123776+0000","last_became_active":"2024-04-14T15:13:09.957187+0000","last_became_peered":"2024-04-14T15:13:09.957187+0000","last_unstale":"2024-04-14T15:19:59.123776+0000","last_undegraded":"2024-04-14T15:19:59.123776+0000","last_fullsized":"2024-04-14T15:19:59.123776+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'55","last_scrub_stamp":"2024-04-14T15:17:37.941589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:37.941589+0000","objects_scrubbed":6,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0020075969999999999,"stat_sum":{"num_bytes":49,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":18,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"110.5","version":"396'64","reported_seq":1004,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.973820+0000","last_change":"2024-04-14T15:18:54.162113+0000","last_active":"2024-04-14T15:19:58.973820+0000","last_peered":"2024-04-14T15:19:58.973820+0000","last_clean":"2024-04-14T15:19:58.973820+0000","last_became_active":"2024-04-14T15:13:09.853689+0000","last_became_peered":"2024-04-14T15:13:09.853689+0000","last_unstale":"2024-04-14T15:19:58.973820+0000","last_undegraded":"2024-04-14T15:19:58.973820+0000","last_fullsized":"2024-04-14T15:19:58.973820+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"396'64","last_scrub_stamp":"2024-04-14T15:17:47.418687+0000","last_deep_scrub":"396'64","last_deep_scrub_stamp":"2024-04-14T15:17:47.418687+0000","last_clean_scrub_stamp":"2024-04-14T15:17:47.418687+0000","objects_scrubbed":5,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00099928400000000002,"stat_sum":{"num_bytes":42,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":20,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"110.4","version":"406'55","reported_seq":1000,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.122780+0000","last_change":"2024-04-14T15:19:11.276191+0000","last_active":"2024-04-14T15:19:59.122780+0000","last_peered":"2024-04-14T15:19:59.122780+0000","last_clean":"2024-04-14T15:19:59.122780+0000","last_became_active":"2024-04-14T15:13:09.958847+0000","last_became_peered":"2024-04-14T15:13:09.958847+0000","last_unstale":"2024-04-14T15:19:59.122780+0000","last_undegraded":"2024-04-14T15:19:59.122780+0000","last_fullsized":"2024-04-14T15:19:59.122780+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"406'55","last_scrub_stamp":"2024-04-14T15:19:11.276056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:19:11.276056+0000","objects_scrubbed":4,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:40.515075+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001395522,"stat_sum":{"num_bytes":38,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"110.7","version":"355'18","reported_seq":977,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.982064+0000","last_change":"2024-04-14T15:19:56.930358+0000","last_active":"2024-04-14T15:19:58.982064+0000","last_peered":"2024-04-14T15:19:58.982064+0000","last_clean":"2024-04-14T15:19:58.982064+0000","last_became_active":"2024-04-14T15:13:09.936631+0000","last_became_peered":"2024-04-14T15:13:09.936631+0000","last_unstale":"2024-04-14T15:19:58.982064+0000","last_undegraded":"2024-04-14T15:19:58.982064+0000","last_fullsized":"2024-04-14T15:19:58.982064+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'18","last_scrub_stamp":"2024-04-14T15:19:56.930258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:19:56.930258+0000","objects_scrubbed":2,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:03.142228+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00050163699999999998,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"110.6","version":"444'90","reported_seq":1026,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.982540+0000","last_change":"2024-04-14T15:19:12.507223+0000","last_active":"2024-04-14T15:19:58.982540+0000","last_peered":"2024-04-14T15:19:58.982540+0000","last_clean":"2024-04-14T15:19:58.982540+0000","last_became_active":"2024-04-14T15:13:09.930125+0000","last_became_peered":"2024-04-14T15:13:09.930125+0000","last_unstale":"2024-04-14T15:19:58.982540+0000","last_undegraded":"2024-04-14T15:19:58.982540+0000","last_fullsized":"2024-04-14T15:19:58.982540+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"444'90","last_scrub_stamp":"2024-04-14T15:19:12.507119+0000","last_deep_scrub":"274'74","last_deep_scrub_stamp":"2024-04-14T15:15:42.641684+0000","last_clean_scrub_stamp":"2024-04-14T15:19:12.507119+0000","objects_scrubbed":7,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:14.287806+0000","scrub_duration":1,"objects_trimmed":1,"snaptrim_duration":2.0059427689999998,"stat_sum":{"num_bytes":62,"num_objects":7,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":26,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"110.9","version":"369'20","reported_seq":973,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.971641+0000","last_change":"2024-04-14T15:19:22.004327+0000","last_active":"2024-04-14T15:19:58.971641+0000","last_peered":"2024-04-14T15:19:58.971641+0000","last_clean":"2024-04-14T15:19:58.971641+0000","last_became_active":"2024-04-14T15:13:09.957324+0000","last_became_peered":"2024-04-14T15:13:09.957324+0000","last_unstale":"2024-04-14T15:19:58.971641+0000","last_undegraded":"2024-04-14T15:19:58.971641+0000","last_fullsized":"2024-04-14T15:19:58.971641+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"369'20","last_scrub_stamp":"2024-04-14T15:19:22.004242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:19:22.004242+0000","objects_scrubbed":4,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:27.025034+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00098494500000000009,"stat_sum":{"num_bytes":35,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"110.8","version":"190'17","reported_seq":719,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.128255+0000","last_change":"2024-04-14T15:18:54.159492+0000","last_active":"2024-04-14T15:19:59.128255+0000","last_peered":"2024-04-14T15:19:59.128255+0000","last_clean":"2024-04-14T15:19:59.128255+0000","last_became_active":"2024-04-14T15:15:44.890057+0000","last_became_peered":"2024-04-14T15:15:44.890057+0000","last_unstale":"2024-04-14T15:19:59.128255+0000","last_undegraded":"2024-04-14T15:19:59.128255+0000","last_fullsized":"2024-04-14T15:19:59.128255+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'17","last_scrub_stamp":"2024-04-14T15:18:34.505521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:18:34.505521+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00097299799999999998,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"110.b","version":"412'71","reported_seq":1017,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-04-14T15:19:57.976188+0000","last_change":"2024-04-14T15:19:22.798118+0000","last_active":"2024-04-14T15:19:57.976188+0000","last_peered":"2024-04-14T15:19:57.976188+0000","last_clean":"2024-04-14T15:19:57.976188+0000","last_became_active":"2024-04-14T15:13:09.964629+0000","last_became_peered":"2024-04-14T15:13:09.964629+0000","last_unstale":"2024-04-14T15:19:57.976188+0000","last_undegraded":"2024-04-14T15:19:57.976188+0000","last_fullsized":"2024-04-14T15:19:57.976188+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'71","last_scrub_stamp":"2024-04-14T15:19:22.798030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:19:22.798030+0000","objects_scrubbed":5,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:45.936171+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0017521979999999999,"stat_sum":{"num_bytes":54,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":22,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"110.a","version":"387'74","reported_seq":944,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.131307+0000","last_change":"2024-04-14T15:19:14.371878+0000","last_active":"2024-04-14T15:19:59.131307+0000","last_peered":"2024-04-14T15:19:59.131307+0000","last_clean":"2024-04-14T15:19:59.131307+0000","last_became_active":"2024-04-14T15:15:21.885346+0000","last_became_peered":"2024-04-14T15:15:21.885346+0000","last_unstale":"2024-04-14T15:19:59.131307+0000","last_undegraded":"2024-04-14T15:19:59.131307+0000","last_fullsized":"2024-04-14T15:19:59.131307+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'74","last_scrub_stamp":"2024-04-14T15:19:14.371777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:19:14.371777+0000","objects_scrubbed":9,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:17.703016+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00095557400000000005,"stat_sum":{"num_bytes":84,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":26,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"110.d","version":"416'95","reported_seq":868,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.128604+0000","last_change":"2024-04-14T15:19:13.327926+0000","last_active":"2024-04-14T15:19:59.128604+0000","last_peered":"2024-04-14T15:19:59.128604+0000","last_clean":"2024-04-14T15:19:59.128604+0000","last_became_active":"2024-04-14T15:15:21.877073+0000","last_became_peered":"2024-04-14T15:15:21.877073+0000","last_unstale":"2024-04-14T15:19:59.128604+0000","last_undegraded":"2024-04-14T15:19:59.128604+0000","last_fullsized":"2024-04-14T15:19:59.128604+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'95","last_scrub_stamp":"2024-04-14T15:19:13.327836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:19:13.327836+0000","objects_scrubbed":8,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:40.750563+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00067428099999999999,"stat_sum":{"num_bytes":71,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":34,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"110.c","version":"459'25","reported_seq":973,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-04-14T15:19:57.976611+0000","last_change":"2024-04-14T15:18:54.157224+0000","last_active":"2024-04-14T15:19:57.976611+0000","last_peered":"2024-04-14T15:19:57.976611+0000","last_clean":"2024-04-14T15:19:57.976611+0000","last_became_active":"2024-04-14T15:13:09.984609+0000","last_became_peered":"2024-04-14T15:13:09.984609+0000","last_unstale":"2024-04-14T15:19:57.976611+0000","last_undegraded":"2024-04-14T15:19:57.976611+0000","last_fullsized":"2024-04-14T15:19:57.976611+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'20","last_scrub_stamp":"2024-04-14T15:17:52.882509+0000","last_deep_scrub":"398'20","last_deep_scrub_stamp":"2024-04-14T15:17:52.882509+0000","last_clean_scrub_stamp":"2024-04-14T15:17:52.882509+0000","objects_scrubbed":2,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001761999,"stat_sum":{"num_bytes":29,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":9,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"110.f","version":"355'52","reported_seq":995,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-04-14T15:19:57.977359+0000","last_change":"2024-04-14T15:18:54.157838+0000","last_active":"2024-04-14T15:19:57.977359+0000","last_peered":"2024-04-14T15:19:57.977359+0000","last_clean":"2024-04-14T15:19:57.977359+0000","last_became_active":"2024-04-14T15:13:09.964057+0000","last_became_peered":"2024-04-14T15:13:09.964057+0000","last_unstale":"2024-04-14T15:19:57.977359+0000","last_undegraded":"2024-04-14T15:19:57.977359+0000","last_fullsized":"2024-04-14T15:19:57.977359+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'52","last_scrub_stamp":"2024-04-14T15:17:45.848166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:45.848166+0000","objects_scrubbed":4,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001409509,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"110.e","version":"416'92","reported_seq":1032,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.981896+0000","last_change":"2024-04-14T15:18:54.151705+0000","last_active":"2024-04-14T15:19:58.981896+0000","last_peered":"2024-04-14T15:19:58.981896+0000","last_clean":"2024-04-14T15:19:58.981896+0000","last_became_active":"2024-04-14T15:13:09.984980+0000","last_became_peered":"2024-04-14T15:13:09.984980+0000","last_unstale":"2024-04-14T15:19:58.981896+0000","last_undegraded":"2024-04-14T15:19:58.981896+0000","last_fullsized":"2024-04-14T15:19:58.981896+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'90","last_scrub_stamp":"2024-04-14T15:17:34.409367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:34.409367+0000","objects_scrubbed":6,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0012202070000000001,"stat_sum":{"num_bytes":68,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":28,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"91'130","reported_seq":450,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.982812+0000","last_change":"2024-04-14T15:19:09.497449+0000","last_active":"2024-04-14T15:19:58.982812+0000","last_peered":"2024-04-14T15:19:58.982812+0000","last_clean":"2024-04-14T15:19:58.982812+0000","last_became_active":"2024-04-14T15:18:58.187712+0000","last_became_peered":"2024-04-14T15:18:58.187712+0000","last_unstale":"2024-04-14T15:19:58.982812+0000","last_undegraded":"2024-04-14T15:19:58.982812+0000","last_fullsized":"2024-04-14T15:19:58.982812+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:09.497308+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:09.497308+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:23.450906+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":443,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.124227+0000","last_change":"2024-04-14T15:19:10.053883+0000","last_active":"2024-04-14T15:19:59.124227+0000","last_peered":"2024-04-14T15:19:59.124227+0000","last_clean":"2024-04-14T15:19:59.124227+0000","last_became_active":"2024-04-14T15:15:44.882941+0000","last_became_peered":"2024-04-14T15:15:44.882941+0000","last_unstale":"2024-04-14T15:19:59.124227+0000","last_undegraded":"2024-04-14T15:19:59.124227+0000","last_fullsized":"2024-04-14T15:19:59.124227+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:10.053804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:19:10.053804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:28.076611+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.9","version":"85'9","reported_seq":1016,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.979165+0000","last_change":"2024-04-14T15:17:54.606158+0000","last_active":"2024-04-14T15:19:58.979165+0000","last_peered":"2024-04-14T15:19:58.979165+0000","last_clean":"2024-04-14T15:19:58.979165+0000","last_became_active":"2024-04-14T15:09:47.045441+0000","last_became_peered":"2024-04-14T15:09:47.045441+0000","last_unstale":"2024-04-14T15:19:58.979165+0000","last_undegraded":"2024-04-14T15:19:58.979165+0000","last_fullsized":"2024-04-14T15:19:58.979165+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9","last_scrub_stamp":"2024-04-14T15:17:54.606051+0000","last_deep_scrub":"85'9","last_deep_scrub_stamp":"2024-04-14T15:10:51.950580+0000","last_clean_scrub_stamp":"2024-04-14T15:17:54.606051+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"479'34","reported_seq":510,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.977389+0000","last_change":"2024-04-14T15:18:39.704989+0000","last_active":"2024-04-14T15:19:58.977389+0000","last_peered":"2024-04-14T15:19:58.977389+0000","last_clean":"2024-04-14T15:19:58.977389+0000","last_became_active":"2024-04-14T15:15:44.885409+0000","last_became_peered":"2024-04-14T15:15:44.885409+0000","last_unstale":"2024-04-14T15:19:58.977389+0000","last_undegraded":"2024-04-14T15:19:58.977389+0000","last_fullsized":"2024-04-14T15:19:58.977389+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'22","last_scrub_stamp":"2024-04-14T15:18:39.704883+0000","last_deep_scrub":"430'22","last_deep_scrub_stamp":"2024-04-14T15:18:39.704883+0000","last_clean_scrub_stamp":"2024-04-14T15:18:39.704883+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":890,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.130793+0000","last_change":"2024-04-14T15:17:53.867764+0000","last_active":"2024-04-14T15:19:59.130793+0000","last_peered":"2024-04-14T15:19:59.130793+0000","last_clean":"2024-04-14T15:19:59.130793+0000","last_became_active":"2024-04-14T15:09:43.021460+0000","last_became_peered":"2024-04-14T15:09:43.021460+0000","last_unstale":"2024-04-14T15:19:59.130793+0000","last_undegraded":"2024-04-14T15:19:59.130793+0000","last_fullsized":"2024-04-14T15:19:59.130793+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:53.867696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:53.867696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":443,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.979823+0000","last_change":"2024-04-14T15:17:54.889170+0000","last_active":"2024-04-14T15:19:58.979823+0000","last_peered":"2024-04-14T15:19:58.979823+0000","last_clean":"2024-04-14T15:19:58.979823+0000","last_became_active":"2024-04-14T15:15:44.888154+0000","last_became_peered":"2024-04-14T15:15:44.888154+0000","last_unstale":"2024-04-14T15:19:58.979823+0000","last_undegraded":"2024-04-14T15:19:58.979823+0000","last_fullsized":"2024-04-14T15:19:58.979823+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:54.889066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:15:40.368219+0000","last_clean_scrub_stamp":"2024-04-14T15:17:54.889066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"91'130","reported_seq":449,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.974730+0000","last_change":"2024-04-14T15:19:05.573366+0000","last_active":"2024-04-14T15:19:58.974730+0000","last_peered":"2024-04-14T15:19:58.974730+0000","last_clean":"2024-04-14T15:19:58.974730+0000","last_became_active":"2024-04-14T15:18:58.182420+0000","last_became_peered":"2024-04-14T15:18:58.182420+0000","last_unstale":"2024-04-14T15:19:58.974730+0000","last_undegraded":"2024-04-14T15:19:58.974730+0000","last_fullsized":"2024-04-14T15:19:58.974730+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:05.573281+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:05.573281+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:17.624412+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":788,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.981367+0000","last_change":"2024-04-14T15:18:37.636498+0000","last_active":"2024-04-14T15:19:58.981367+0000","last_peered":"2024-04-14T15:19:58.981367+0000","last_clean":"2024-04-14T15:19:58.981367+0000","last_became_active":"2024-04-14T15:12:56.355348+0000","last_became_peered":"2024-04-14T15:12:56.355348+0000","last_unstale":"2024-04-14T15:19:58.981367+0000","last_undegraded":"2024-04-14T15:19:58.981367+0000","last_fullsized":"2024-04-14T15:19:58.981367+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:37.636419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:18:37.636419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:06.978194+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"479'15","reported_seq":913,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.127036+0000","last_change":"2024-04-14T15:16:56.663996+0000","last_active":"2024-04-14T15:19:59.127036+0000","last_peered":"2024-04-14T15:19:59.127036+0000","last_clean":"2024-04-14T15:19:59.127036+0000","last_became_active":"2024-04-14T15:09:45.037179+0000","last_became_peered":"2024-04-14T15:09:45.037179+0000","last_unstale":"2024-04-14T15:19:59.127036+0000","last_undegraded":"2024-04-14T15:19:59.127036+0000","last_fullsized":"2024-04-14T15:19:59.127036+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-04-14T15:16:56.663900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:16:56.663900+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":893,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.128638+0000","last_change":"2024-04-14T15:18:36.029705+0000","last_active":"2024-04-14T15:19:59.128638+0000","last_peered":"2024-04-14T15:19:59.128638+0000","last_clean":"2024-04-14T15:19:59.128638+0000","last_became_active":"2024-04-14T15:09:43.020044+0000","last_became_peered":"2024-04-14T15:09:43.020044+0000","last_unstale":"2024-04-14T15:19:59.128638+0000","last_undegraded":"2024-04-14T15:19:59.128638+0000","last_fullsized":"2024-04-14T15:19:59.128638+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:36.029630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:18:36.029630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":888,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-04-14T15:19:57.975749+0000","last_change":"2024-04-14T15:19:11.838298+0000","last_active":"2024-04-14T15:19:57.975749+0000","last_peered":"2024-04-14T15:19:57.975749+0000","last_clean":"2024-04-14T15:19:57.975749+0000","last_became_active":"2024-04-14T15:09:47.046153+0000","last_became_peered":"2024-04-14T15:09:47.046153+0000","last_unstale":"2024-04-14T15:19:57.975749+0000","last_undegraded":"2024-04-14T15:19:57.975749+0000","last_fullsized":"2024-04-14T15:19:57.975749+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:11.838212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:19:11.838212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:19.779219+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":880,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.125613+0000","last_change":"2024-04-14T15:17:37.919665+0000","last_active":"2024-04-14T15:19:59.125613+0000","last_peered":"2024-04-14T15:19:59.125613+0000","last_clean":"2024-04-14T15:19:59.125613+0000","last_became_active":"2024-04-14T15:09:49.061495+0000","last_became_peered":"2024-04-14T15:09:49.061495+0000","last_unstale":"2024-04-14T15:19:59.125613+0000","last_undegraded":"2024-04-14T15:19:59.125613+0000","last_fullsized":"2024-04-14T15:19:59.125613+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:37.919595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:17:37.919595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"91'130","reported_seq":977,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.981570+0000","last_change":"2024-04-14T15:19:06.754212+0000","last_active":"2024-04-14T15:19:58.981570+0000","last_peered":"2024-04-14T15:19:58.981570+0000","last_clean":"2024-04-14T15:19:58.981570+0000","last_became_active":"2024-04-14T15:18:45.919901+0000","last_became_peered":"2024-04-14T15:18:45.919901+0000","last_unstale":"2024-04-14T15:19:58.981570+0000","last_undegraded":"2024-04-14T15:19:58.981570+0000","last_fullsized":"2024-04-14T15:19:58.981570+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:06.754126+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:06.754126+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:30.044936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":788,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.983587+0000","last_change":"2024-04-14T15:18:25.523604+0000","last_active":"2024-04-14T15:19:58.983587+0000","last_peered":"2024-04-14T15:19:58.983587+0000","last_clean":"2024-04-14T15:19:58.983587+0000","last_became_active":"2024-04-14T15:12:56.246062+0000","last_became_peered":"2024-04-14T15:12:56.246062+0000","last_unstale":"2024-04-14T15:19:58.983587+0000","last_undegraded":"2024-04-14T15:19:58.983587+0000","last_fullsized":"2024-04-14T15:19:58.983587+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:25.523520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:18:25.523520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"479'44","reported_seq":960,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.980256+0000","last_change":"2024-04-14T15:19:58.856267+0000","last_active":"2024-04-14T15:19:58.980256+0000","last_peered":"2024-04-14T15:19:58.980256+0000","last_clean":"2024-04-14T15:19:58.980256+0000","last_became_active":"2024-04-14T15:09:45.178782+0000","last_became_peered":"2024-04-14T15:09:45.178782+0000","last_unstale":"2024-04-14T15:19:58.980256+0000","last_undegraded":"2024-04-14T15:19:58.980256+0000","last_fullsized":"2024-04-14T15:19:58.980256+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'44","last_scrub_stamp":"2024-04-14T15:19:58.856115+0000","last_deep_scrub":"479'44","last_deep_scrub_stamp":"2024-04-14T15:19:58.856115+0000","last_clean_scrub_stamp":"2024-04-14T15:19:58.856115+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:10.582292+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":898,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.130970+0000","last_change":"2024-04-14T15:18:36.925573+0000","last_active":"2024-04-14T15:19:59.130970+0000","last_peered":"2024-04-14T15:19:59.130970+0000","last_clean":"2024-04-14T15:19:59.130970+0000","last_became_active":"2024-04-14T15:09:43.020268+0000","last_became_peered":"2024-04-14T15:09:43.020268+0000","last_unstale":"2024-04-14T15:19:59.130970+0000","last_undegraded":"2024-04-14T15:19:59.130970+0000","last_fullsized":"2024-04-14T15:19:59.130970+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:36.925496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:34.342793+0000","last_clean_scrub_stamp":"2024-04-14T15:18:36.925496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":880,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-04-14T15:19:57.976283+0000","last_change":"2024-04-14T15:18:31.989052+0000","last_active":"2024-04-14T15:19:57.976283+0000","last_peered":"2024-04-14T15:19:57.976283+0000","last_clean":"2024-04-14T15:19:57.976283+0000","last_became_active":"2024-04-14T15:09:47.045542+0000","last_became_peered":"2024-04-14T15:09:47.045542+0000","last_unstale":"2024-04-14T15:19:57.976283+0000","last_undegraded":"2024-04-14T15:19:57.976283+0000","last_fullsized":"2024-04-14T15:19:57.976283+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:31.988957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:18:31.988957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":879,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.977718+0000","last_change":"2024-04-14T15:19:16.566595+0000","last_active":"2024-04-14T15:19:58.977718+0000","last_peered":"2024-04-14T15:19:58.977718+0000","last_clean":"2024-04-14T15:19:58.977718+0000","last_became_active":"2024-04-14T15:09:49.065397+0000","last_became_peered":"2024-04-14T15:09:49.065397+0000","last_unstale":"2024-04-14T15:19:58.977718+0000","last_undegraded":"2024-04-14T15:19:58.977718+0000","last_fullsized":"2024-04-14T15:19:58.977718+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:16.566496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:19:16.566496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:28.994328+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"91'130","reported_seq":777,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.123472+0000","last_change":"2024-04-14T15:19:04.951104+0000","last_active":"2024-04-14T15:19:59.123472+0000","last_peered":"2024-04-14T15:19:59.123472+0000","last_clean":"2024-04-14T15:19:59.123472+0000","last_became_active":"2024-04-14T15:18:58.181194+0000","last_became_peered":"2024-04-14T15:18:58.181194+0000","last_unstale":"2024-04-14T15:19:59.123472+0000","last_undegraded":"2024-04-14T15:19:59.123472+0000","last_fullsized":"2024-04-14T15:19:59.123472+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:04.950991+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:04.950991+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:06.456834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":793,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-04-14T15:19:57.975719+0000","last_change":"2024-04-14T15:19:15.778552+0000","last_active":"2024-04-14T15:19:57.975719+0000","last_peered":"2024-04-14T15:19:57.975719+0000","last_clean":"2024-04-14T15:19:57.975719+0000","last_became_active":"2024-04-14T15:12:56.474322+0000","last_became_peered":"2024-04-14T15:12:56.474322+0000","last_unstale":"2024-04-14T15:19:57.975719+0000","last_undegraded":"2024-04-14T15:19:57.975719+0000","last_fullsized":"2024-04-14T15:19:57.975719+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:15.778469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:19:15.778469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:40.510707+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"479'31","reported_seq":984,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.974343+0000","last_change":"2024-04-14T15:17:48.374589+0000","last_active":"2024-04-14T15:19:58.974343+0000","last_peered":"2024-04-14T15:19:58.974343+0000","last_clean":"2024-04-14T15:19:58.974343+0000","last_became_active":"2024-04-14T15:13:12.933907+0000","last_became_peered":"2024-04-14T15:13:12.933907+0000","last_unstale":"2024-04-14T15:19:58.974343+0000","last_undegraded":"2024-04-14T15:19:58.974343+0000","last_fullsized":"2024-04-14T15:19:58.974343+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'17","last_scrub_stamp":"2024-04-14T15:17:48.374469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:17:48.374469+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":891,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.979926+0000","last_change":"2024-04-14T15:19:26.515133+0000","last_active":"2024-04-14T15:19:58.979926+0000","last_peered":"2024-04-14T15:19:58.979926+0000","last_clean":"2024-04-14T15:19:58.979926+0000","last_became_active":"2024-04-14T15:09:43.022351+0000","last_became_peered":"2024-04-14T15:09:43.022351+0000","last_unstale":"2024-04-14T15:19:58.979926+0000","last_undegraded":"2024-04-14T15:19:58.979926+0000","last_fullsized":"2024-04-14T15:19:58.979926+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:26.515055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:19:26.515055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:51.899280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":877,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.127130+0000","last_change":"2024-04-14T15:17:54.871693+0000","last_active":"2024-04-14T15:19:59.127130+0000","last_peered":"2024-04-14T15:19:59.127130+0000","last_clean":"2024-04-14T15:19:59.127130+0000","last_became_active":"2024-04-14T15:09:47.042810+0000","last_became_peered":"2024-04-14T15:09:47.042810+0000","last_unstale":"2024-04-14T15:19:59.127130+0000","last_undegraded":"2024-04-14T15:19:59.127130+0000","last_fullsized":"2024-04-14T15:19:59.127130+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:54.871593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:17:54.871593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":884,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-04-14T15:19:57.977765+0000","last_change":"2024-04-14T15:19:51.902814+0000","last_active":"2024-04-14T15:19:57.977765+0000","last_peered":"2024-04-14T15:19:57.977765+0000","last_clean":"2024-04-14T15:19:57.977765+0000","last_became_active":"2024-04-14T15:09:49.069594+0000","last_became_peered":"2024-04-14T15:09:49.069594+0000","last_unstale":"2024-04-14T15:19:57.977765+0000","last_undegraded":"2024-04-14T15:19:57.977765+0000","last_fullsized":"2024-04-14T15:19:57.977765+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:51.902730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:19:51.902730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:19.807697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"91'130","reported_seq":1180,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.977938+0000","last_change":"2024-04-14T15:19:06.490294+0000","last_active":"2024-04-14T15:19:58.977938+0000","last_peered":"2024-04-14T15:19:58.977938+0000","last_clean":"2024-04-14T15:19:58.977938+0000","last_became_active":"2024-04-14T15:18:58.189483+0000","last_became_peered":"2024-04-14T15:18:58.189483+0000","last_unstale":"2024-04-14T15:19:58.977938+0000","last_undegraded":"2024-04-14T15:19:58.977938+0000","last_fullsized":"2024-04-14T15:19:58.977938+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:06.490208+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:06.490208+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:16.848376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":789,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.979932+0000","last_change":"2024-04-14T15:17:46.871883+0000","last_active":"2024-04-14T15:19:58.979932+0000","last_peered":"2024-04-14T15:19:58.979932+0000","last_clean":"2024-04-14T15:19:58.979932+0000","last_became_active":"2024-04-14T15:12:56.289745+0000","last_became_peered":"2024-04-14T15:12:56.289745+0000","last_unstale":"2024-04-14T15:19:58.979932+0000","last_undegraded":"2024-04-14T15:19:58.979932+0000","last_fullsized":"2024-04-14T15:19:58.979932+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:46.871773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:17:46.871773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"479'25","reported_seq":914,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.972327+0000","last_change":"2024-04-14T15:16:49.243096+0000","last_active":"2024-04-14T15:19:58.972327+0000","last_peered":"2024-04-14T15:19:58.972327+0000","last_clean":"2024-04-14T15:19:58.972327+0000","last_became_active":"2024-04-14T15:09:45.029170+0000","last_became_peered":"2024-04-14T15:09:45.029170+0000","last_unstale":"2024-04-14T15:19:58.972327+0000","last_undegraded":"2024-04-14T15:19:58.972327+0000","last_fullsized":"2024-04-14T15:19:58.972327+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'15","last_scrub_stamp":"2024-04-14T15:16:49.243002+0000","last_deep_scrub":"179'15","last_deep_scrub_stamp":"2024-04-14T15:14:07.488422+0000","last_clean_scrub_stamp":"2024-04-14T15:16:49.243002+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":893,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.124008+0000","last_change":"2024-04-14T15:17:51.854742+0000","last_active":"2024-04-14T15:19:59.124008+0000","last_peered":"2024-04-14T15:19:59.124008+0000","last_clean":"2024-04-14T15:19:59.124008+0000","last_became_active":"2024-04-14T15:09:43.018871+0000","last_became_peered":"2024-04-14T15:09:43.018871+0000","last_unstale":"2024-04-14T15:19:59.124008+0000","last_undegraded":"2024-04-14T15:19:59.124008+0000","last_fullsized":"2024-04-14T15:19:59.124008+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:51.854659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:51.854659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"400'9","reported_seq":1013,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-04-14T15:19:57.977502+0000","last_change":"2024-04-14T15:17:54.845967+0000","last_active":"2024-04-14T15:19:57.977502+0000","last_peered":"2024-04-14T15:19:57.977502+0000","last_clean":"2024-04-14T15:19:57.977502+0000","last_became_active":"2024-04-14T15:09:47.044085+0000","last_became_peered":"2024-04-14T15:09:47.044085+0000","last_unstale":"2024-04-14T15:19:57.977502+0000","last_undegraded":"2024-04-14T15:19:57.977502+0000","last_fullsized":"2024-04-14T15:19:57.977502+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'9","last_scrub_stamp":"2024-04-14T15:17:54.845881+0000","last_deep_scrub":"400'9","last_deep_scrub_stamp":"2024-04-14T15:17:54.845881+0000","last_clean_scrub_stamp":"2024-04-14T15:17:54.845881+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":879,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.975235+0000","last_change":"2024-04-14T15:19:18.588681+0000","last_active":"2024-04-14T15:19:58.975235+0000","last_peered":"2024-04-14T15:19:58.975235+0000","last_clean":"2024-04-14T15:19:58.975235+0000","last_became_active":"2024-04-14T15:09:49.065274+0000","last_became_peered":"2024-04-14T15:09:49.065274+0000","last_unstale":"2024-04-14T15:19:58.975235+0000","last_undegraded":"2024-04-14T15:19:58.975235+0000","last_fullsized":"2024-04-14T15:19:58.975235+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:18.588546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:18.588546+0000","last_clean_scrub_stamp":"2024-04-14T15:19:18.588546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:36.014977+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"91'130","reported_seq":783,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-04-14T15:19:57.975463+0000","last_change":"2024-04-14T15:19:05.972132+0000","last_active":"2024-04-14T15:19:57.975463+0000","last_peered":"2024-04-14T15:19:57.975463+0000","last_clean":"2024-04-14T15:19:57.975463+0000","last_became_active":"2024-04-14T15:18:58.187649+0000","last_became_peered":"2024-04-14T15:18:58.187649+0000","last_unstale":"2024-04-14T15:19:57.975463+0000","last_undegraded":"2024-04-14T15:19:57.975463+0000","last_fullsized":"2024-04-14T15:19:57.975463+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:05.972050+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:05.972050+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:15.882926+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":790,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.971318+0000","last_change":"2024-04-14T15:19:11.008294+0000","last_active":"2024-04-14T15:19:58.971318+0000","last_peered":"2024-04-14T15:19:58.971318+0000","last_clean":"2024-04-14T15:19:58.971318+0000","last_became_active":"2024-04-14T15:12:56.487615+0000","last_became_peered":"2024-04-14T15:12:56.487615+0000","last_unstale":"2024-04-14T15:19:58.971318+0000","last_undegraded":"2024-04-14T15:19:58.971318+0000","last_fullsized":"2024-04-14T15:19:58.971318+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:11.008225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:19:11.008225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:29.857396+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"479'34","reported_seq":939,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.124463+0000","last_change":"2024-04-14T15:17:52.883984+0000","last_active":"2024-04-14T15:19:59.124463+0000","last_peered":"2024-04-14T15:19:59.124463+0000","last_clean":"2024-04-14T15:19:59.124463+0000","last_became_active":"2024-04-14T15:09:45.033331+0000","last_became_peered":"2024-04-14T15:09:45.033331+0000","last_unstale":"2024-04-14T15:19:59.124463+0000","last_undegraded":"2024-04-14T15:19:59.124463+0000","last_fullsized":"2024-04-14T15:19:59.124463+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'22","last_scrub_stamp":"2024-04-14T15:17:52.883850+0000","last_deep_scrub":"398'22","last_deep_scrub_stamp":"2024-04-14T15:17:52.883850+0000","last_clean_scrub_stamp":"2024-04-14T15:17:52.883850+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":893,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.129231+0000","last_change":"2024-04-14T15:18:25.009179+0000","last_active":"2024-04-14T15:19:59.129231+0000","last_peered":"2024-04-14T15:19:59.129231+0000","last_clean":"2024-04-14T15:19:59.129231+0000","last_became_active":"2024-04-14T15:09:43.020878+0000","last_became_peered":"2024-04-14T15:09:43.020878+0000","last_unstale":"2024-04-14T15:19:59.129231+0000","last_undegraded":"2024-04-14T15:19:59.129231+0000","last_fullsized":"2024-04-14T15:19:59.129231+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:25.009100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:18:25.009100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"72'8","reported_seq":1015,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.975178+0000","last_change":"2024-04-14T15:19:19.596882+0000","last_active":"2024-04-14T15:19:58.975178+0000","last_peered":"2024-04-14T15:19:58.975178+0000","last_clean":"2024-04-14T15:19:58.975178+0000","last_became_active":"2024-04-14T15:09:47.050962+0000","last_became_peered":"2024-04-14T15:09:47.050962+0000","last_unstale":"2024-04-14T15:19:58.975178+0000","last_undegraded":"2024-04-14T15:19:58.975178+0000","last_fullsized":"2024-04-14T15:19:58.975178+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:19:19.596773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:19:19.596773+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:44.419959+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":444,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.128145+0000","last_change":"2024-04-14T15:16:51.199025+0000","last_active":"2024-04-14T15:19:59.128145+0000","last_peered":"2024-04-14T15:19:59.128145+0000","last_clean":"2024-04-14T15:19:59.128145+0000","last_became_active":"2024-04-14T15:15:44.879699+0000","last_became_peered":"2024-04-14T15:15:44.879699+0000","last_unstale":"2024-04-14T15:19:59.128145+0000","last_undegraded":"2024-04-14T15:19:59.128145+0000","last_fullsized":"2024-04-14T15:19:59.128145+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:51.198907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:51.198907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"91'130","reported_seq":976,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.122260+0000","last_change":"2024-04-14T15:19:08.097458+0000","last_active":"2024-04-14T15:19:59.122260+0000","last_peered":"2024-04-14T15:19:59.122260+0000","last_clean":"2024-04-14T15:19:59.122260+0000","last_became_active":"2024-04-14T15:18:58.179785+0000","last_became_peered":"2024-04-14T15:18:58.179785+0000","last_unstale":"2024-04-14T15:19:59.122260+0000","last_undegraded":"2024-04-14T15:19:59.122260+0000","last_fullsized":"2024-04-14T15:19:59.122260+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:08.097372+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:08.097372+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:22.282987+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"233'8","reported_seq":803,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.982627+0000","last_change":"2024-04-14T15:17:50.905698+0000","last_active":"2024-04-14T15:19:58.982627+0000","last_peered":"2024-04-14T15:19:58.982627+0000","last_clean":"2024-04-14T15:19:58.982627+0000","last_became_active":"2024-04-14T15:12:56.381026+0000","last_became_peered":"2024-04-14T15:12:56.381026+0000","last_unstale":"2024-04-14T15:19:58.982627+0000","last_undegraded":"2024-04-14T15:19:58.982627+0000","last_fullsized":"2024-04-14T15:19:58.982627+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'8","last_scrub_stamp":"2024-04-14T15:17:50.905608+0000","last_deep_scrub":"233'8","last_deep_scrub_stamp":"2024-04-14T15:15:01.295800+0000","last_clean_scrub_stamp":"2024-04-14T15:17:50.905608+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"479'24","reported_seq":923,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.974989+0000","last_change":"2024-04-14T15:16:53.143329+0000","last_active":"2024-04-14T15:19:58.974989+0000","last_peered":"2024-04-14T15:19:58.974989+0000","last_clean":"2024-04-14T15:19:58.974989+0000","last_became_active":"2024-04-14T15:09:45.028212+0000","last_became_peered":"2024-04-14T15:09:45.028212+0000","last_unstale":"2024-04-14T15:19:58.974989+0000","last_undegraded":"2024-04-14T15:19:58.974989+0000","last_fullsized":"2024-04-14T15:19:58.974989+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-14T15:16:53.143239+0000","last_deep_scrub":"85'16","last_deep_scrub_stamp":"2024-04-14T15:10:47.456169+0000","last_clean_scrub_stamp":"2024-04-14T15:16:53.143239+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":891,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.978170+0000","last_change":"2024-04-14T15:17:39.465617+0000","last_active":"2024-04-14T15:19:58.978170+0000","last_peered":"2024-04-14T15:19:58.978170+0000","last_clean":"2024-04-14T15:19:58.978170+0000","last_became_active":"2024-04-14T15:09:43.018601+0000","last_became_peered":"2024-04-14T15:09:43.018601+0000","last_unstale":"2024-04-14T15:19:58.978170+0000","last_undegraded":"2024-04-14T15:19:58.978170+0000","last_fullsized":"2024-04-14T15:19:58.978170+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:39.465475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:39.465475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":441,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.975491+0000","last_change":"2024-04-14T15:16:54.187971+0000","last_active":"2024-04-14T15:19:58.975491+0000","last_peered":"2024-04-14T15:19:58.975491+0000","last_clean":"2024-04-14T15:19:58.975491+0000","last_became_active":"2024-04-14T15:15:44.886834+0000","last_became_peered":"2024-04-14T15:15:44.886834+0000","last_unstale":"2024-04-14T15:19:58.975491+0000","last_undegraded":"2024-04-14T15:19:58.975491+0000","last_fullsized":"2024-04-14T15:19:58.975491+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:54.187888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:54.187888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":554,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.129339+0000","last_change":"2024-04-14T15:17:57.214134+0000","last_active":"2024-04-14T15:19:59.129339+0000","last_peered":"2024-04-14T15:19:59.129339+0000","last_clean":"2024-04-14T15:19:59.129339+0000","last_became_active":"2024-04-14T15:15:21.888587+0000","last_became_peered":"2024-04-14T15:15:21.888587+0000","last_unstale":"2024-04-14T15:19:59.129339+0000","last_undegraded":"2024-04-14T15:19:59.129339+0000","last_fullsized":"2024-04-14T15:19:59.129339+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:57.214046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:57.214046+0000","last_clean_scrub_stamp":"2024-04-14T15:17:57.214046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"91'130","reported_seq":959,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.129111+0000","last_change":"2024-04-14T15:19:05.396196+0000","last_active":"2024-04-14T15:19:59.129111+0000","last_peered":"2024-04-14T15:19:59.129111+0000","last_clean":"2024-04-14T15:19:59.129111+0000","last_became_active":"2024-04-14T15:18:40.917537+0000","last_became_peered":"2024-04-14T15:18:40.917537+0000","last_unstale":"2024-04-14T15:19:59.129111+0000","last_undegraded":"2024-04-14T15:19:59.129111+0000","last_fullsized":"2024-04-14T15:19:59.129111+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:05.396090+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:05.396090+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:10.820552+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":788,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.974855+0000","last_change":"2024-04-14T15:16:56.214275+0000","last_active":"2024-04-14T15:19:58.974855+0000","last_peered":"2024-04-14T15:19:58.974855+0000","last_clean":"2024-04-14T15:19:58.974855+0000","last_became_active":"2024-04-14T15:12:56.427276+0000","last_became_peered":"2024-04-14T15:12:56.427276+0000","last_unstale":"2024-04-14T15:19:58.974855+0000","last_undegraded":"2024-04-14T15:19:58.974855+0000","last_fullsized":"2024-04-14T15:19:58.974855+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:56.214178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:16:56.214178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"255'10","reported_seq":659,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.128804+0000","last_change":"2024-04-14T15:19:50.669678+0000","last_active":"2024-04-14T15:19:59.128804+0000","last_peered":"2024-04-14T15:19:59.128804+0000","last_clean":"2024-04-14T15:19:59.128804+0000","last_became_active":"2024-04-14T15:15:21.873747+0000","last_became_peered":"2024-04-14T15:15:21.873747+0000","last_unstale":"2024-04-14T15:19:59.128804+0000","last_undegraded":"2024-04-14T15:19:59.128804+0000","last_fullsized":"2024-04-14T15:19:59.128804+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'10","last_scrub_stamp":"2024-04-14T15:19:50.669574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:19:50.669574+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:12.441595+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"479'28","reported_seq":929,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.972847+0000","last_change":"2024-04-14T15:18:22.545734+0000","last_active":"2024-04-14T15:19:58.972847+0000","last_peered":"2024-04-14T15:19:58.972847+0000","last_clean":"2024-04-14T15:19:58.972847+0000","last_became_active":"2024-04-14T15:09:45.025673+0000","last_became_peered":"2024-04-14T15:09:45.025673+0000","last_unstale":"2024-04-14T15:19:58.972847+0000","last_undegraded":"2024-04-14T15:19:58.972847+0000","last_fullsized":"2024-04-14T15:19:58.972847+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'14","last_scrub_stamp":"2024-04-14T15:18:22.545646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:18:22.545646+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":892,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.981962+0000","last_change":"2024-04-14T15:18:26.824852+0000","last_active":"2024-04-14T15:19:58.981962+0000","last_peered":"2024-04-14T15:19:58.981962+0000","last_clean":"2024-04-14T15:19:58.981962+0000","last_became_active":"2024-04-14T15:09:43.020503+0000","last_became_peered":"2024-04-14T15:09:43.020503+0000","last_unstale":"2024-04-14T15:19:58.981962+0000","last_undegraded":"2024-04-14T15:19:58.981962+0000","last_fullsized":"2024-04-14T15:19:58.981962+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:26.824773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:18:26.824773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":879,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.980728+0000","last_change":"2024-04-14T15:19:50.609813+0000","last_active":"2024-04-14T15:19:58.980728+0000","last_peered":"2024-04-14T15:19:58.980728+0000","last_clean":"2024-04-14T15:19:58.980728+0000","last_became_active":"2024-04-14T15:09:49.129610+0000","last_became_peered":"2024-04-14T15:09:49.129610+0000","last_unstale":"2024-04-14T15:19:58.980728+0000","last_undegraded":"2024-04-14T15:19:58.980728+0000","last_fullsized":"2024-04-14T15:19:58.980728+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:50.609691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:19:50.609691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:58.284777+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"91'130","reported_seq":966,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.972823+0000","last_change":"2024-04-14T15:19:04.526327+0000","last_active":"2024-04-14T15:19:58.972823+0000","last_peered":"2024-04-14T15:19:58.972823+0000","last_clean":"2024-04-14T15:19:58.972823+0000","last_became_active":"2024-04-14T15:18:58.179181+0000","last_became_peered":"2024-04-14T15:18:58.179181+0000","last_unstale":"2024-04-14T15:19:58.972823+0000","last_undegraded":"2024-04-14T15:19:58.972823+0000","last_fullsized":"2024-04-14T15:19:58.972823+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:04.526214+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:04.526214+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:30.900098+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":788,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.982992+0000","last_change":"2024-04-14T15:17:47.553179+0000","last_active":"2024-04-14T15:19:58.982992+0000","last_peered":"2024-04-14T15:19:58.982992+0000","last_clean":"2024-04-14T15:19:58.982992+0000","last_became_active":"2024-04-14T15:12:56.355254+0000","last_became_peered":"2024-04-14T15:12:56.355254+0000","last_unstale":"2024-04-14T15:19:58.982992+0000","last_undegraded":"2024-04-14T15:19:58.982992+0000","last_fullsized":"2024-04-14T15:19:58.982992+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:47.553055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:17:47.553055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":883,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.972980+0000","last_change":"2024-04-14T15:18:39.394736+0000","last_active":"2024-04-14T15:19:58.972980+0000","last_peered":"2024-04-14T15:19:58.972980+0000","last_clean":"2024-04-14T15:19:58.972980+0000","last_became_active":"2024-04-14T15:09:47.044610+0000","last_became_peered":"2024-04-14T15:09:47.044610+0000","last_unstale":"2024-04-14T15:19:58.972980+0000","last_undegraded":"2024-04-14T15:19:58.972980+0000","last_fullsized":"2024-04-14T15:19:58.972980+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:39.394655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:39.394655+0000","last_clean_scrub_stamp":"2024-04-14T15:18:39.394655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"479'30","reported_seq":621,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.128703+0000","last_change":"2024-04-14T15:19:26.509637+0000","last_active":"2024-04-14T15:19:59.128703+0000","last_peered":"2024-04-14T15:19:59.128703+0000","last_clean":"2024-04-14T15:19:59.128703+0000","last_became_active":"2024-04-14T15:15:21.880565+0000","last_became_peered":"2024-04-14T15:15:21.880565+0000","last_unstale":"2024-04-14T15:19:59.128703+0000","last_undegraded":"2024-04-14T15:19:59.128703+0000","last_fullsized":"2024-04-14T15:19:59.128703+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'18","last_scrub_stamp":"2024-04-14T15:19:26.509541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:19:26.509541+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:53.956765+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'2","reported_seq":887,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.123269+0000","last_change":"2024-04-14T15:17:48.850178+0000","last_active":"2024-04-14T15:19:59.123269+0000","last_peered":"2024-04-14T15:19:59.123269+0000","last_clean":"2024-04-14T15:19:59.123269+0000","last_became_active":"2024-04-14T15:09:43.018478+0000","last_became_peered":"2024-04-14T15:09:43.018478+0000","last_unstale":"2024-04-14T15:19:59.123269+0000","last_undegraded":"2024-04-14T15:19:59.123269+0000","last_fullsized":"2024-04-14T15:19:59.123269+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'2","last_scrub_stamp":"2024-04-14T15:17:48.850069+0000","last_deep_scrub":"91'2","last_deep_scrub_stamp":"2024-04-14T15:17:48.850069+0000","last_clean_scrub_stamp":"2024-04-14T15:17:48.850069+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":879,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.972738+0000","last_change":"2024-04-14T15:19:14.609181+0000","last_active":"2024-04-14T15:19:58.972738+0000","last_peered":"2024-04-14T15:19:58.972738+0000","last_clean":"2024-04-14T15:19:58.972738+0000","last_became_active":"2024-04-14T15:09:49.065684+0000","last_became_peered":"2024-04-14T15:09:49.065684+0000","last_unstale":"2024-04-14T15:19:58.972738+0000","last_undegraded":"2024-04-14T15:19:58.972738+0000","last_fullsized":"2024-04-14T15:19:58.972738+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:14.609079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:19:14.609079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:23.254598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"91'130","reported_seq":974,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.979217+0000","last_change":"2024-04-14T15:19:05.797940+0000","last_active":"2024-04-14T15:19:58.979217+0000","last_peered":"2024-04-14T15:19:58.979217+0000","last_clean":"2024-04-14T15:19:58.979217+0000","last_became_active":"2024-04-14T15:18:58.182649+0000","last_became_peered":"2024-04-14T15:18:58.182649+0000","last_unstale":"2024-04-14T15:19:58.979217+0000","last_undegraded":"2024-04-14T15:19:58.979217+0000","last_fullsized":"2024-04-14T15:19:58.979217+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:05.797859+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:05.797859+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:17.283641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":790,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.124371+0000","last_change":"2024-04-14T15:18:35.004058+0000","last_active":"2024-04-14T15:19:59.124371+0000","last_peered":"2024-04-14T15:19:59.124371+0000","last_clean":"2024-04-14T15:19:59.124371+0000","last_became_active":"2024-04-14T15:12:56.402023+0000","last_became_peered":"2024-04-14T15:12:56.402023+0000","last_unstale":"2024-04-14T15:19:59.124371+0000","last_undegraded":"2024-04-14T15:19:59.124371+0000","last_fullsized":"2024-04-14T15:19:59.124371+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:35.003968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:35.003968+0000","last_clean_scrub_stamp":"2024-04-14T15:18:35.003968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":884,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.980460+0000","last_change":"2024-04-14T15:17:39.682323+0000","last_active":"2024-04-14T15:19:58.980460+0000","last_peered":"2024-04-14T15:19:58.980460+0000","last_clean":"2024-04-14T15:19:58.980460+0000","last_became_active":"2024-04-14T15:09:47.048427+0000","last_became_peered":"2024-04-14T15:09:47.048427+0000","last_unstale":"2024-04-14T15:19:58.980460+0000","last_undegraded":"2024-04-14T15:19:58.980460+0000","last_fullsized":"2024-04-14T15:19:58.980460+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:39.682203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:17:39.682203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"479'37","reported_seq":947,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.127232+0000","last_change":"2024-04-14T15:17:46.924432+0000","last_active":"2024-04-14T15:19:59.127232+0000","last_peered":"2024-04-14T15:19:59.127232+0000","last_clean":"2024-04-14T15:19:59.127232+0000","last_became_active":"2024-04-14T15:09:45.179182+0000","last_became_peered":"2024-04-14T15:09:45.179182+0000","last_unstale":"2024-04-14T15:19:59.127232+0000","last_undegraded":"2024-04-14T15:19:59.127232+0000","last_fullsized":"2024-04-14T15:19:59.127232+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'27","last_scrub_stamp":"2024-04-14T15:17:46.924330+0000","last_deep_scrub":"102'27","last_deep_scrub_stamp":"2024-04-14T15:17:46.924330+0000","last_clean_scrub_stamp":"2024-04-14T15:17:46.924330+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":891,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.983141+0000","last_change":"2024-04-14T15:19:14.569862+0000","last_active":"2024-04-14T15:19:58.983141+0000","last_peered":"2024-04-14T15:19:58.983141+0000","last_clean":"2024-04-14T15:19:58.983141+0000","last_became_active":"2024-04-14T15:09:43.021006+0000","last_became_peered":"2024-04-14T15:09:43.021006+0000","last_unstale":"2024-04-14T15:19:58.983141+0000","last_undegraded":"2024-04-14T15:19:58.983141+0000","last_fullsized":"2024-04-14T15:19:58.983141+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:14.569778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:19:14.569778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:38.152634+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":873,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.129318+0000","last_change":"2024-04-14T15:16:56.206766+0000","last_active":"2024-04-14T15:19:59.129318+0000","last_peered":"2024-04-14T15:19:59.129318+0000","last_clean":"2024-04-14T15:19:59.129318+0000","last_became_active":"2024-04-14T15:09:49.128136+0000","last_became_peered":"2024-04-14T15:09:49.128136+0000","last_unstale":"2024-04-14T15:19:59.129318+0000","last_undegraded":"2024-04-14T15:19:59.129318+0000","last_fullsized":"2024-04-14T15:19:59.129318+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:56.206649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:56.206649+0000","last_clean_scrub_stamp":"2024-04-14T15:16:56.206649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"91'130","reported_seq":963,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.122349+0000","last_change":"2024-04-14T15:19:09.060997+0000","last_active":"2024-04-14T15:19:59.122349+0000","last_peered":"2024-04-14T15:19:59.122349+0000","last_clean":"2024-04-14T15:19:59.122349+0000","last_became_active":"2024-04-14T15:18:40.910133+0000","last_became_peered":"2024-04-14T15:18:40.910133+0000","last_unstale":"2024-04-14T15:19:59.122349+0000","last_undegraded":"2024-04-14T15:19:59.122349+0000","last_fullsized":"2024-04-14T15:19:59.122349+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:09.060906+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:09.060906+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:11.385135+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":502,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.129633+0000","last_change":"2024-04-14T15:19:47.610791+0000","last_active":"2024-04-14T15:19:59.129633+0000","last_peered":"2024-04-14T15:19:59.129633+0000","last_clean":"2024-04-14T15:19:59.129633+0000","last_became_active":"2024-04-14T15:15:21.887769+0000","last_became_peered":"2024-04-14T15:15:21.887769+0000","last_unstale":"2024-04-14T15:19:59.129633+0000","last_undegraded":"2024-04-14T15:19:59.129633+0000","last_fullsized":"2024-04-14T15:19:59.129633+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:47.610667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:19:47.610667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:55.096189+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":443,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.127614+0000","last_change":"2024-04-14T15:17:47.871801+0000","last_active":"2024-04-14T15:19:59.127614+0000","last_peered":"2024-04-14T15:19:59.127614+0000","last_clean":"2024-04-14T15:19:59.127614+0000","last_became_active":"2024-04-14T15:15:44.883694+0000","last_became_peered":"2024-04-14T15:15:44.883694+0000","last_unstale":"2024-04-14T15:19:59.127614+0000","last_undegraded":"2024-04-14T15:19:59.127614+0000","last_fullsized":"2024-04-14T15:19:59.127614+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:47.871722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:17:47.871722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"479'30","reported_seq":930,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.126009+0000","last_change":"2024-04-14T15:17:44.919445+0000","last_active":"2024-04-14T15:19:59.126009+0000","last_peered":"2024-04-14T15:19:59.126009+0000","last_clean":"2024-04-14T15:19:59.126009+0000","last_became_active":"2024-04-14T15:09:45.037231+0000","last_became_peered":"2024-04-14T15:09:45.037231+0000","last_unstale":"2024-04-14T15:19:59.126009+0000","last_undegraded":"2024-04-14T15:19:59.126009+0000","last_fullsized":"2024-04-14T15:19:59.126009+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'18","last_scrub_stamp":"2024-04-14T15:17:44.919358+0000","last_deep_scrub":"182'18","last_deep_scrub_stamp":"2024-04-14T15:14:10.383906+0000","last_clean_scrub_stamp":"2024-04-14T15:17:44.919358+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":892,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.977622+0000","last_change":"2024-04-14T15:17:46.542125+0000","last_active":"2024-04-14T15:19:58.977622+0000","last_peered":"2024-04-14T15:19:58.977622+0000","last_clean":"2024-04-14T15:19:58.977622+0000","last_became_active":"2024-04-14T15:09:43.025095+0000","last_became_peered":"2024-04-14T15:09:43.025095+0000","last_unstale":"2024-04-14T15:19:58.977622+0000","last_undegraded":"2024-04-14T15:19:58.977622+0000","last_fullsized":"2024-04-14T15:19:58.977622+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:46.542030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:46.542030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":899,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.125285+0000","last_change":"2024-04-14T15:17:48.866222+0000","last_active":"2024-04-14T15:19:59.125285+0000","last_peered":"2024-04-14T15:19:59.125285+0000","last_clean":"2024-04-14T15:19:59.125285+0000","last_became_active":"2024-04-14T15:09:43.018509+0000","last_became_peered":"2024-04-14T15:09:43.018509+0000","last_unstale":"2024-04-14T15:19:59.125285+0000","last_undegraded":"2024-04-14T15:19:59.125285+0000","last_fullsized":"2024-04-14T15:19:59.125285+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:48.866146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:48.866146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0018592680000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"355'2","reported_seq":875,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.127221+0000","last_change":"2024-04-14T15:17:03.127190+0000","last_active":"2024-04-14T15:19:59.127221+0000","last_peered":"2024-04-14T15:19:59.127221+0000","last_clean":"2024-04-14T15:19:59.127221+0000","last_became_active":"2024-04-14T15:09:49.065243+0000","last_became_peered":"2024-04-14T15:09:49.065243+0000","last_unstale":"2024-04-14T15:19:59.127221+0000","last_undegraded":"2024-04-14T15:19:59.127221+0000","last_fullsized":"2024-04-14T15:19:59.127221+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'2","last_scrub_stamp":"2024-04-14T15:17:03.127112+0000","last_deep_scrub":"355'2","last_deep_scrub_stamp":"2024-04-14T15:17:03.127112+0000","last_clean_scrub_stamp":"2024-04-14T15:17:03.127112+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":486,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.131547+0000","last_change":"2024-04-14T15:18:32.532077+0000","last_active":"2024-04-14T15:19:59.131547+0000","last_peered":"2024-04-14T15:19:59.131547+0000","last_clean":"2024-04-14T15:19:59.131547+0000","last_became_active":"2024-04-14T15:15:21.890115+0000","last_became_peered":"2024-04-14T15:15:21.890115+0000","last_unstale":"2024-04-14T15:19:59.131547+0000","last_undegraded":"2024-04-14T15:19:59.131547+0000","last_fullsized":"2024-04-14T15:19:59.131547+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:32.531993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:18:32.531993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":883,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.981639+0000","last_change":"2024-04-14T15:19:46.631011+0000","last_active":"2024-04-14T15:19:58.981639+0000","last_peered":"2024-04-14T15:19:58.981639+0000","last_clean":"2024-04-14T15:19:58.981639+0000","last_became_active":"2024-04-14T15:09:47.047201+0000","last_became_peered":"2024-04-14T15:09:47.047201+0000","last_unstale":"2024-04-14T15:19:58.981639+0000","last_undegraded":"2024-04-14T15:19:58.981639+0000","last_fullsized":"2024-04-14T15:19:58.981639+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:46.630886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:19:46.630886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:47.188067+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"479'36","reported_seq":1060,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.980924+0000","last_change":"2024-04-14T15:19:02.841789+0000","last_active":"2024-04-14T15:19:58.980924+0000","last_peered":"2024-04-14T15:19:58.980924+0000","last_clean":"2024-04-14T15:19:58.980924+0000","last_became_active":"2024-04-14T15:09:45.179978+0000","last_became_peered":"2024-04-14T15:09:45.179978+0000","last_unstale":"2024-04-14T15:19:58.980924+0000","last_undegraded":"2024-04-14T15:19:58.980924+0000","last_fullsized":"2024-04-14T15:19:58.980924+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'24","last_scrub_stamp":"2024-04-14T15:19:02.841668+0000","last_deep_scrub":"90'24","last_deep_scrub_stamp":"2024-04-14T15:11:25.120692+0000","last_clean_scrub_stamp":"2024-04-14T15:19:02.841668+0000","objects_scrubbed":8,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:20.808249+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"91'130","reported_seq":1173,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.977521+0000","last_change":"2024-04-14T15:19:08.494753+0000","last_active":"2024-04-14T15:19:58.977521+0000","last_peered":"2024-04-14T15:19:58.977521+0000","last_clean":"2024-04-14T15:19:58.977521+0000","last_became_active":"2024-04-14T15:18:40.911206+0000","last_became_peered":"2024-04-14T15:18:40.911206+0000","last_unstale":"2024-04-14T15:19:58.977521+0000","last_undegraded":"2024-04-14T15:19:58.977521+0000","last_fullsized":"2024-04-14T15:19:58.977521+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:08.494666+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:08.494666+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:25.546717+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":447,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-04-14T15:19:57.976382+0000","last_change":"2024-04-14T15:17:35.705777+0000","last_active":"2024-04-14T15:19:57.976382+0000","last_peered":"2024-04-14T15:19:57.976382+0000","last_clean":"2024-04-14T15:19:57.976382+0000","last_became_active":"2024-04-14T15:15:44.885315+0000","last_became_peered":"2024-04-14T15:15:44.885315+0000","last_unstale":"2024-04-14T15:19:57.976382+0000","last_undegraded":"2024-04-14T15:19:57.976382+0000","last_fullsized":"2024-04-14T15:19:57.976382+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:35.705645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:47.627305+0000","last_clean_scrub_stamp":"2024-04-14T15:17:35.705645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"69'2","reported_seq":576,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.131427+0000","last_change":"2024-04-14T15:16:53.972591+0000","last_active":"2024-04-14T15:19:59.131427+0000","last_peered":"2024-04-14T15:19:59.131427+0000","last_clean":"2024-04-14T15:19:59.131427+0000","last_became_active":"2024-04-14T15:15:21.889851+0000","last_became_peered":"2024-04-14T15:15:21.889851+0000","last_unstale":"2024-04-14T15:19:59.131427+0000","last_undegraded":"2024-04-14T15:19:59.131427+0000","last_fullsized":"2024-04-14T15:19:59.131427+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-04-14T15:16:53.972502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:53.972502+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0058046169999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":440,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.129579+0000","last_change":"2024-04-14T15:16:57.935611+0000","last_active":"2024-04-14T15:19:59.129579+0000","last_peered":"2024-04-14T15:19:59.129579+0000","last_clean":"2024-04-14T15:19:59.129579+0000","last_became_active":"2024-04-14T15:15:44.887934+0000","last_became_peered":"2024-04-14T15:15:44.887934+0000","last_unstale":"2024-04-14T15:19:59.129579+0000","last_undegraded":"2024-04-14T15:19:59.129579+0000","last_fullsized":"2024-04-14T15:19:59.129579+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:57.935436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:57.935436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"91'130","reported_seq":1180,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.979637+0000","last_change":"2024-04-14T15:19:07.501791+0000","last_active":"2024-04-14T15:19:58.979637+0000","last_peered":"2024-04-14T15:19:58.979637+0000","last_clean":"2024-04-14T15:19:58.979637+0000","last_became_active":"2024-04-14T15:18:40.912277+0000","last_became_peered":"2024-04-14T15:18:40.912277+0000","last_unstale":"2024-04-14T15:19:58.979637+0000","last_undegraded":"2024-04-14T15:19:58.979637+0000","last_fullsized":"2024-04-14T15:19:58.979637+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:07.501704+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:07.501704+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:10.701083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":494,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.128547+0000","last_change":"2024-04-14T15:19:46.589280+0000","last_active":"2024-04-14T15:19:59.128547+0000","last_peered":"2024-04-14T15:19:59.128547+0000","last_clean":"2024-04-14T15:19:59.128547+0000","last_became_active":"2024-04-14T15:15:21.889970+0000","last_became_peered":"2024-04-14T15:15:21.889970+0000","last_unstale":"2024-04-14T15:19:59.128547+0000","last_undegraded":"2024-04-14T15:19:59.128547+0000","last_fullsized":"2024-04-14T15:19:59.128547+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:46.589189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:19:46.589189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:15.849303+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":880,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-04-14T15:19:57.976512+0000","last_change":"2024-04-14T15:17:01.653342+0000","last_active":"2024-04-14T15:19:57.976512+0000","last_peered":"2024-04-14T15:19:57.976512+0000","last_clean":"2024-04-14T15:19:57.976512+0000","last_became_active":"2024-04-14T15:09:47.049457+0000","last_became_peered":"2024-04-14T15:09:47.049457+0000","last_unstale":"2024-04-14T15:19:57.976512+0000","last_undegraded":"2024-04-14T15:19:57.976512+0000","last_fullsized":"2024-04-14T15:19:57.976512+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:01.653271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:17:01.653271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"479'22","reported_seq":927,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.127453+0000","last_change":"2024-04-14T15:19:26.078482+0000","last_active":"2024-04-14T15:19:59.127453+0000","last_peered":"2024-04-14T15:19:59.127453+0000","last_clean":"2024-04-14T15:19:59.127453+0000","last_became_active":"2024-04-14T15:09:45.033883+0000","last_became_peered":"2024-04-14T15:09:45.033883+0000","last_unstale":"2024-04-14T15:19:59.127453+0000","last_undegraded":"2024-04-14T15:19:59.127453+0000","last_fullsized":"2024-04-14T15:19:59.127453+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"308'18","last_scrub_stamp":"2024-04-14T15:19:26.078396+0000","last_deep_scrub":"308'18","last_deep_scrub_stamp":"2024-04-14T15:16:16.054032+0000","last_clean_scrub_stamp":"2024-04-14T15:19:26.078396+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:27.837657+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":898,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.128149+0000","last_change":"2024-04-14T15:19:16.788873+0000","last_active":"2024-04-14T15:19:59.128149+0000","last_peered":"2024-04-14T15:19:59.128149+0000","last_clean":"2024-04-14T15:19:59.128149+0000","last_became_active":"2024-04-14T15:09:43.021188+0000","last_became_peered":"2024-04-14T15:09:43.021188+0000","last_unstale":"2024-04-14T15:19:59.128149+0000","last_undegraded":"2024-04-14T15:19:59.128149+0000","last_fullsized":"2024-04-14T15:19:59.128149+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:16.788795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:19:16.788795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:35.846721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"73'5","reported_seq":1513,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.971479+0000","last_change":"2024-04-14T15:18:26.955802+0000","last_active":"2024-04-14T15:19:58.971479+0000","last_peered":"2024-04-14T15:19:58.971479+0000","last_clean":"2024-04-14T15:19:58.971479+0000","last_became_active":"2024-04-14T15:09:43.016544+0000","last_became_peered":"2024-04-14T15:09:43.016544+0000","last_unstale":"2024-04-14T15:19:58.971479+0000","last_undegraded":"2024-04-14T15:19:58.971479+0000","last_fullsized":"2024-04-14T15:19:58.971479+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-04-14T15:18:26.955711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:18:26.955711+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00056542099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":609,"num_read_kb":604,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":879,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.983263+0000","last_change":"2024-04-14T15:19:22.535232+0000","last_active":"2024-04-14T15:19:58.983263+0000","last_peered":"2024-04-14T15:19:58.983263+0000","last_clean":"2024-04-14T15:19:58.983263+0000","last_became_active":"2024-04-14T15:09:49.130307+0000","last_became_peered":"2024-04-14T15:09:49.130307+0000","last_unstale":"2024-04-14T15:19:58.983263+0000","last_undegraded":"2024-04-14T15:19:58.983263+0000","last_fullsized":"2024-04-14T15:19:58.983263+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:22.535145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:20.827589+0000","last_clean_scrub_stamp":"2024-04-14T15:19:22.535145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:35.538494+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"91'130","reported_seq":794,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.122724+0000","last_change":"2024-04-14T15:19:06.124178+0000","last_active":"2024-04-14T15:19:59.122724+0000","last_peered":"2024-04-14T15:19:59.122724+0000","last_clean":"2024-04-14T15:19:59.122724+0000","last_became_active":"2024-04-14T15:18:40.913215+0000","last_became_peered":"2024-04-14T15:18:40.913215+0000","last_unstale":"2024-04-14T15:19:59.122724+0000","last_undegraded":"2024-04-14T15:19:59.122724+0000","last_fullsized":"2024-04-14T15:19:59.122724+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:06.124071+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:06.124071+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:23.152389+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":494,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.132052+0000","last_change":"2024-04-14T15:19:15.372917+0000","last_active":"2024-04-14T15:19:59.132052+0000","last_peered":"2024-04-14T15:19:59.132052+0000","last_clean":"2024-04-14T15:19:59.132052+0000","last_became_active":"2024-04-14T15:15:21.883717+0000","last_became_peered":"2024-04-14T15:15:21.883717+0000","last_unstale":"2024-04-14T15:19:59.132052+0000","last_undegraded":"2024-04-14T15:19:59.132052+0000","last_fullsized":"2024-04-14T15:19:59.132052+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:15.372837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:19:15.372837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:42.207119+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":884,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.122664+0000","last_change":"2024-04-14T15:19:25.297173+0000","last_active":"2024-04-14T15:19:59.122664+0000","last_peered":"2024-04-14T15:19:59.122664+0000","last_clean":"2024-04-14T15:19:59.122664+0000","last_became_active":"2024-04-14T15:09:47.040185+0000","last_became_peered":"2024-04-14T15:09:47.040185+0000","last_unstale":"2024-04-14T15:19:59.122664+0000","last_undegraded":"2024-04-14T15:19:59.122664+0000","last_fullsized":"2024-04-14T15:19:59.122664+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:25.297044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:53.731610+0000","last_clean_scrub_stamp":"2024-04-14T15:19:25.297044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:25.349894+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"479'40","reported_seq":1079,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.124275+0000","last_change":"2024-04-14T15:19:49.915826+0000","last_active":"2024-04-14T15:19:59.124275+0000","last_peered":"2024-04-14T15:19:59.124275+0000","last_clean":"2024-04-14T15:19:59.124275+0000","last_became_active":"2024-04-14T15:09:45.034168+0000","last_became_peered":"2024-04-14T15:09:45.034168+0000","last_unstale":"2024-04-14T15:19:59.124275+0000","last_undegraded":"2024-04-14T15:19:59.124275+0000","last_fullsized":"2024-04-14T15:19:59.124275+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'28","last_scrub_stamp":"2024-04-14T15:19:49.915672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:19:49.915672+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:54.452998+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":452,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.123834+0000","last_change":"2024-04-14T15:19:24.066365+0000","last_active":"2024-04-14T15:19:59.123834+0000","last_peered":"2024-04-14T15:19:59.123834+0000","last_clean":"2024-04-14T15:19:59.123834+0000","last_became_active":"2024-04-14T15:15:44.885315+0000","last_became_peered":"2024-04-14T15:15:44.885315+0000","last_unstale":"2024-04-14T15:19:59.123834+0000","last_undegraded":"2024-04-14T15:19:59.123834+0000","last_fullsized":"2024-04-14T15:19:59.123834+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:24.066275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:24.066275+0000","last_clean_scrub_stamp":"2024-04-14T15:19:24.066275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:43.166123+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":546,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.134794+0000","last_change":"2024-04-14T15:17:44.084142+0000","last_active":"2024-04-14T15:19:59.134794+0000","last_peered":"2024-04-14T15:19:59.134794+0000","last_clean":"2024-04-14T15:19:59.134794+0000","last_became_active":"2024-04-14T15:15:21.874742+0000","last_became_peered":"2024-04-14T15:15:21.874742+0000","last_unstale":"2024-04-14T15:19:59.134794+0000","last_undegraded":"2024-04-14T15:19:59.134794+0000","last_fullsized":"2024-04-14T15:19:59.134794+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:44.084010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:44.084010+0000","last_clean_scrub_stamp":"2024-04-14T15:17:44.084010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"91'130","reported_seq":974,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.981691+0000","last_change":"2024-04-14T15:19:07.770220+0000","last_active":"2024-04-14T15:19:58.981691+0000","last_peered":"2024-04-14T15:19:58.981691+0000","last_clean":"2024-04-14T15:19:58.981691+0000","last_became_active":"2024-04-14T15:18:58.189402+0000","last_became_peered":"2024-04-14T15:18:58.189402+0000","last_unstale":"2024-04-14T15:19:58.981691+0000","last_undegraded":"2024-04-14T15:19:58.981691+0000","last_fullsized":"2024-04-14T15:19:58.981691+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:07.770125+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:07.770125+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:09.567335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":793,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-04-14T15:19:57.977762+0000","last_change":"2024-04-14T15:19:55.858944+0000","last_active":"2024-04-14T15:19:57.977762+0000","last_peered":"2024-04-14T15:19:57.977762+0000","last_clean":"2024-04-14T15:19:57.977762+0000","last_became_active":"2024-04-14T15:12:56.350703+0000","last_became_peered":"2024-04-14T15:12:56.350703+0000","last_unstale":"2024-04-14T15:19:57.977762+0000","last_undegraded":"2024-04-14T15:19:57.977762+0000","last_fullsized":"2024-04-14T15:19:57.977762+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:55.858835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:19:55.858835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:09.317978+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"72'8","reported_seq":1016,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.126918+0000","last_change":"2024-04-14T15:17:04.695161+0000","last_active":"2024-04-14T15:19:59.126918+0000","last_peered":"2024-04-14T15:19:59.126918+0000","last_clean":"2024-04-14T15:19:59.126918+0000","last_became_active":"2024-04-14T15:09:47.047699+0000","last_became_peered":"2024-04-14T15:09:47.047699+0000","last_unstale":"2024-04-14T15:19:59.126918+0000","last_undegraded":"2024-04-14T15:19:59.126918+0000","last_fullsized":"2024-04-14T15:19:59.126918+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:17:04.695079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:17:04.695079+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"479'20","reported_seq":893,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-04-14T15:19:57.975898+0000","last_change":"2024-04-14T15:18:38.870798+0000","last_active":"2024-04-14T15:19:57.975898+0000","last_peered":"2024-04-14T15:19:57.975898+0000","last_clean":"2024-04-14T15:19:57.975898+0000","last_became_active":"2024-04-14T15:10:07.764042+0000","last_became_peered":"2024-04-14T15:10:07.764042+0000","last_unstale":"2024-04-14T15:19:57.975898+0000","last_undegraded":"2024-04-14T15:19:57.975898+0000","last_fullsized":"2024-04-14T15:19:57.975898+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-04-14T15:18:38.870710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:18:38.870710+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":885,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.124186+0000","last_change":"2024-04-14T15:17:40.971492+0000","last_active":"2024-04-14T15:19:59.124186+0000","last_peered":"2024-04-14T15:19:59.124186+0000","last_clean":"2024-04-14T15:19:59.124186+0000","last_became_active":"2024-04-14T15:09:43.022597+0000","last_became_peered":"2024-04-14T15:09:43.022597+0000","last_unstale":"2024-04-14T15:19:59.124186+0000","last_undegraded":"2024-04-14T15:19:59.124186+0000","last_fullsized":"2024-04-14T15:19:59.124186+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:40.971412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:40.971412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":881,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.123323+0000","last_change":"2024-04-14T15:19:16.390131+0000","last_active":"2024-04-14T15:19:59.123323+0000","last_peered":"2024-04-14T15:19:59.123323+0000","last_clean":"2024-04-14T15:19:59.123323+0000","last_became_active":"2024-04-14T15:09:49.066648+0000","last_became_peered":"2024-04-14T15:09:49.066648+0000","last_unstale":"2024-04-14T15:19:59.123323+0000","last_undegraded":"2024-04-14T15:19:59.123323+0000","last_fullsized":"2024-04-14T15:19:59.123323+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:16.390049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:19:16.390049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:36.162522+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.c","version":"91'130","reported_seq":979,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.129009+0000","last_change":"2024-04-14T15:19:04.397426+0000","last_active":"2024-04-14T15:19:59.129009+0000","last_peered":"2024-04-14T15:19:59.129009+0000","last_clean":"2024-04-14T15:19:59.129009+0000","last_became_active":"2024-04-14T15:18:58.186036+0000","last_became_peered":"2024-04-14T15:18:58.186036+0000","last_unstale":"2024-04-14T15:19:59.129009+0000","last_undegraded":"2024-04-14T15:19:59.129009+0000","last_fullsized":"2024-04-14T15:19:59.129009+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:04.397343+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:04.397343+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:31.837520+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":789,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.125881+0000","last_change":"2024-04-14T15:18:27.133391+0000","last_active":"2024-04-14T15:19:59.125881+0000","last_peered":"2024-04-14T15:19:59.125881+0000","last_clean":"2024-04-14T15:19:59.125881+0000","last_became_active":"2024-04-14T15:12:56.563074+0000","last_became_peered":"2024-04-14T15:12:56.563074+0000","last_unstale":"2024-04-14T15:19:59.125881+0000","last_undegraded":"2024-04-14T15:19:59.125881+0000","last_fullsized":"2024-04-14T15:19:59.125881+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:27.133310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:18:27.133310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":882,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.123864+0000","last_change":"2024-04-14T15:17:56.915660+0000","last_active":"2024-04-14T15:19:59.123864+0000","last_peered":"2024-04-14T15:19:59.123864+0000","last_clean":"2024-04-14T15:19:59.123864+0000","last_became_active":"2024-04-14T15:09:47.049005+0000","last_became_peered":"2024-04-14T15:09:47.049005+0000","last_unstale":"2024-04-14T15:19:59.123864+0000","last_undegraded":"2024-04-14T15:19:59.123864+0000","last_fullsized":"2024-04-14T15:19:59.123864+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:56.915544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:56.915544+0000","last_clean_scrub_stamp":"2024-04-14T15:17:56.915544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"479'23","reported_seq":932,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.972844+0000","last_change":"2024-04-14T15:19:44.525830+0000","last_active":"2024-04-14T15:19:58.972844+0000","last_peered":"2024-04-14T15:19:58.972844+0000","last_clean":"2024-04-14T15:19:58.972844+0000","last_became_active":"2024-04-14T15:09:45.032892+0000","last_became_peered":"2024-04-14T15:09:45.032892+0000","last_unstale":"2024-04-14T15:19:58.972844+0000","last_undegraded":"2024-04-14T15:19:58.972844+0000","last_fullsized":"2024-04-14T15:19:58.972844+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'17","last_scrub_stamp":"2024-04-14T15:19:44.525712+0000","last_deep_scrub":"183'17","last_deep_scrub_stamp":"2024-04-14T15:14:11.577574+0000","last_clean_scrub_stamp":"2024-04-14T15:19:44.525712+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:47.139793+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"65'3","reported_seq":909,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.975307+0000","last_change":"2024-04-14T15:19:22.603785+0000","last_active":"2024-04-14T15:19:58.975307+0000","last_peered":"2024-04-14T15:19:58.975307+0000","last_clean":"2024-04-14T15:19:58.975307+0000","last_became_active":"2024-04-14T15:09:43.020490+0000","last_became_peered":"2024-04-14T15:09:43.020490+0000","last_unstale":"2024-04-14T15:19:58.975307+0000","last_undegraded":"2024-04-14T15:19:58.975307+0000","last_fullsized":"2024-04-14T15:19:58.975307+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3","last_scrub_stamp":"2024-04-14T15:19:22.603651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:19:22.603651+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:27.010861+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":528,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":546,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.132374+0000","last_change":"2024-04-14T15:16:58.942591+0000","last_active":"2024-04-14T15:19:59.132374+0000","last_peered":"2024-04-14T15:19:59.132374+0000","last_clean":"2024-04-14T15:19:59.132374+0000","last_became_active":"2024-04-14T15:15:21.876410+0000","last_became_peered":"2024-04-14T15:15:21.876410+0000","last_unstale":"2024-04-14T15:19:59.132374+0000","last_undegraded":"2024-04-14T15:19:59.132374+0000","last_fullsized":"2024-04-14T15:19:59.132374+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:58.942499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:58.942499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"91'130","reported_seq":971,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.122143+0000","last_change":"2024-04-14T15:19:07.091520+0000","last_active":"2024-04-14T15:19:59.122143+0000","last_peered":"2024-04-14T15:19:59.122143+0000","last_clean":"2024-04-14T15:19:59.122143+0000","last_became_active":"2024-04-14T15:18:45.912049+0000","last_became_peered":"2024-04-14T15:18:45.912049+0000","last_unstale":"2024-04-14T15:19:59.122143+0000","last_undegraded":"2024-04-14T15:19:59.122143+0000","last_fullsized":"2024-04-14T15:19:59.122143+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:07.091439+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:07.091439+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:25.022161+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":796,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.125175+0000","last_change":"2024-04-14T15:19:24.289915+0000","last_active":"2024-04-14T15:19:59.125175+0000","last_peered":"2024-04-14T15:19:59.125175+0000","last_clean":"2024-04-14T15:19:59.125175+0000","last_became_active":"2024-04-14T15:12:56.551177+0000","last_became_peered":"2024-04-14T15:12:56.551177+0000","last_unstale":"2024-04-14T15:19:59.125175+0000","last_undegraded":"2024-04-14T15:19:59.125175+0000","last_fullsized":"2024-04-14T15:19:59.125175+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:24.289806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:19:24.289806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:44.967529+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"479'19","reported_seq":917,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.126113+0000","last_change":"2024-04-14T15:19:51.290310+0000","last_active":"2024-04-14T15:19:59.126113+0000","last_peered":"2024-04-14T15:19:59.126113+0000","last_clean":"2024-04-14T15:19:59.126113+0000","last_became_active":"2024-04-14T15:09:45.180375+0000","last_became_peered":"2024-04-14T15:09:45.180375+0000","last_unstale":"2024-04-14T15:19:59.126113+0000","last_undegraded":"2024-04-14T15:19:59.126113+0000","last_fullsized":"2024-04-14T15:19:59.126113+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'19","last_scrub_stamp":"2024-04-14T15:19:51.290211+0000","last_deep_scrub":"347'13","last_deep_scrub_stamp":"2024-04-14T15:16:55.694599+0000","last_clean_scrub_stamp":"2024-04-14T15:19:51.290211+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:18.067434+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":898,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.123774+0000","last_change":"2024-04-14T15:19:21.774315+0000","last_active":"2024-04-14T15:19:59.123774+0000","last_peered":"2024-04-14T15:19:59.123774+0000","last_clean":"2024-04-14T15:19:59.123774+0000","last_became_active":"2024-04-14T15:09:43.016788+0000","last_became_peered":"2024-04-14T15:09:43.016788+0000","last_unstale":"2024-04-14T15:19:59.123774+0000","last_undegraded":"2024-04-14T15:19:59.123774+0000","last_fullsized":"2024-04-14T15:19:59.123774+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:21.774184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:19:21.774184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:48.086427+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":565,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.129487+0000","last_change":"2024-04-14T15:18:40.385814+0000","last_active":"2024-04-14T15:19:59.129487+0000","last_peered":"2024-04-14T15:19:59.129487+0000","last_clean":"2024-04-14T15:19:59.129487+0000","last_became_active":"2024-04-14T15:15:21.873580+0000","last_became_peered":"2024-04-14T15:15:21.873580+0000","last_unstale":"2024-04-14T15:19:59.129487+0000","last_undegraded":"2024-04-14T15:19:59.129487+0000","last_fullsized":"2024-04-14T15:19:59.129487+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:40.385736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:18:40.385736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:01.230073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":884,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-04-14T15:19:57.976948+0000","last_change":"2024-04-14T15:19:24.821597+0000","last_active":"2024-04-14T15:19:57.976948+0000","last_peered":"2024-04-14T15:19:57.976948+0000","last_clean":"2024-04-14T15:19:57.976948+0000","last_became_active":"2024-04-14T15:09:49.064296+0000","last_became_peered":"2024-04-14T15:09:49.064296+0000","last_unstale":"2024-04-14T15:19:57.976948+0000","last_undegraded":"2024-04-14T15:19:57.976948+0000","last_fullsized":"2024-04-14T15:19:57.976948+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:24.821514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:19:24.821514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:39.516148+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.12","version":"91'130","reported_seq":443,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.123670+0000","last_change":"2024-04-14T15:19:06.098758+0000","last_active":"2024-04-14T15:19:59.123670+0000","last_peered":"2024-04-14T15:19:59.123670+0000","last_clean":"2024-04-14T15:19:59.123670+0000","last_became_active":"2024-04-14T15:18:45.918413+0000","last_became_peered":"2024-04-14T15:18:45.918413+0000","last_unstale":"2024-04-14T15:19:59.123670+0000","last_undegraded":"2024-04-14T15:19:59.123670+0000","last_fullsized":"2024-04-14T15:19:59.123670+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:06.098672+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:06.098672+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:32.614691+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":786,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.123630+0000","last_change":"2024-04-14T15:17:36.728543+0000","last_active":"2024-04-14T15:19:59.123630+0000","last_peered":"2024-04-14T15:19:59.123630+0000","last_clean":"2024-04-14T15:19:59.123630+0000","last_became_active":"2024-04-14T15:12:56.340066+0000","last_became_peered":"2024-04-14T15:12:56.340066+0000","last_unstale":"2024-04-14T15:19:59.123630+0000","last_undegraded":"2024-04-14T15:19:59.123630+0000","last_fullsized":"2024-04-14T15:19:59.123630+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:36.728465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:17:36.728465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.17","version":"479'12","reported_seq":903,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.129386+0000","last_change":"2024-04-14T15:17:42.019838+0000","last_active":"2024-04-14T15:19:59.129386+0000","last_peered":"2024-04-14T15:19:59.129386+0000","last_clean":"2024-04-14T15:19:59.129386+0000","last_became_active":"2024-04-14T15:09:45.180037+0000","last_became_peered":"2024-04-14T15:09:45.180037+0000","last_unstale":"2024-04-14T15:19:59.129386+0000","last_undegraded":"2024-04-14T15:19:59.129386+0000","last_fullsized":"2024-04-14T15:19:59.129386+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'6","last_scrub_stamp":"2024-04-14T15:17:42.019717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:17:42.019717+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":900,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.982309+0000","last_change":"2024-04-14T15:19:23.915584+0000","last_active":"2024-04-14T15:19:58.982309+0000","last_peered":"2024-04-14T15:19:58.982309+0000","last_clean":"2024-04-14T15:19:58.982309+0000","last_became_active":"2024-04-14T15:09:43.018987+0000","last_became_peered":"2024-04-14T15:09:43.018987+0000","last_unstale":"2024-04-14T15:19:58.982309+0000","last_undegraded":"2024-04-14T15:19:58.982309+0000","last_fullsized":"2024-04-14T15:19:58.982309+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:23.915503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:59.640565+0000","last_clean_scrub_stamp":"2024-04-14T15:19:23.915503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:36.396647+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":439,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.132808+0000","last_change":"2024-04-14T15:17:47.072247+0000","last_active":"2024-04-14T15:19:59.132808+0000","last_peered":"2024-04-14T15:19:59.132808+0000","last_clean":"2024-04-14T15:19:59.132808+0000","last_became_active":"2024-04-14T15:15:44.889786+0000","last_became_peered":"2024-04-14T15:15:44.889786+0000","last_unstale":"2024-04-14T15:19:59.132808+0000","last_undegraded":"2024-04-14T15:19:59.132808+0000","last_fullsized":"2024-04-14T15:19:59.132808+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:47.072170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:13.735024+0000","last_clean_scrub_stamp":"2024-04-14T15:17:47.072170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":879,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.983798+0000","last_change":"2024-04-14T15:19:47.597543+0000","last_active":"2024-04-14T15:19:58.983798+0000","last_peered":"2024-04-14T15:19:58.983798+0000","last_clean":"2024-04-14T15:19:58.983798+0000","last_became_active":"2024-04-14T15:09:49.130438+0000","last_became_peered":"2024-04-14T15:09:49.130438+0000","last_unstale":"2024-04-14T15:19:58.983798+0000","last_undegraded":"2024-04-14T15:19:58.983798+0000","last_fullsized":"2024-04-14T15:19:58.983798+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:47.597268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:19:47.597268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:59.841711+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.11","version":"91'130","reported_seq":801,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.125114+0000","last_change":"2024-04-14T15:19:05.136092+0000","last_active":"2024-04-14T15:19:59.125114+0000","last_peered":"2024-04-14T15:19:59.125114+0000","last_clean":"2024-04-14T15:19:59.125114+0000","last_became_active":"2024-04-14T15:18:45.915385+0000","last_became_peered":"2024-04-14T15:18:45.915385+0000","last_unstale":"2024-04-14T15:19:59.125114+0000","last_undegraded":"2024-04-14T15:19:59.125114+0000","last_fullsized":"2024-04-14T15:19:59.125114+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:05.136001+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:05.136001+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:15.595631+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":787,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.972538+0000","last_change":"2024-04-14T15:17:46.369320+0000","last_active":"2024-04-14T15:19:58.972538+0000","last_peered":"2024-04-14T15:19:58.972538+0000","last_clean":"2024-04-14T15:19:58.972538+0000","last_became_active":"2024-04-14T15:12:56.203577+0000","last_became_peered":"2024-04-14T15:12:56.203577+0000","last_unstale":"2024-04-14T15:19:58.972538+0000","last_undegraded":"2024-04-14T15:19:58.972538+0000","last_fullsized":"2024-04-14T15:19:58.972538+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:46.369231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:46.369231+0000","last_clean_scrub_stamp":"2024-04-14T15:17:46.369231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"479'20","reported_seq":599,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.129433+0000","last_change":"2024-04-14T15:18:36.534967+0000","last_active":"2024-04-14T15:19:59.129433+0000","last_peered":"2024-04-14T15:19:59.129433+0000","last_clean":"2024-04-14T15:19:59.129433+0000","last_became_active":"2024-04-14T15:15:21.877667+0000","last_became_peered":"2024-04-14T15:15:21.877667+0000","last_unstale":"2024-04-14T15:19:59.129433+0000","last_undegraded":"2024-04-14T15:19:59.129433+0000","last_fullsized":"2024-04-14T15:19:59.129433+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-04-14T15:18:36.534874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:18:36.534874+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:02.823104+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":447,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.132104+0000","last_change":"2024-04-14T15:19:56.775138+0000","last_active":"2024-04-14T15:19:59.132104+0000","last_peered":"2024-04-14T15:19:59.132104+0000","last_clean":"2024-04-14T15:19:59.132104+0000","last_became_active":"2024-04-14T15:15:44.887452+0000","last_became_peered":"2024-04-14T15:15:44.887452+0000","last_unstale":"2024-04-14T15:19:59.132104+0000","last_undegraded":"2024-04-14T15:19:59.132104+0000","last_fullsized":"2024-04-14T15:19:59.132104+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:56.775036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:19:56.775036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:23.210279+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"280'9","reported_seq":519,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.128377+0000","last_change":"2024-04-14T15:17:50.908130+0000","last_active":"2024-04-14T15:19:59.128377+0000","last_peered":"2024-04-14T15:19:59.128377+0000","last_clean":"2024-04-14T15:19:59.128377+0000","last_became_active":"2024-04-14T15:15:44.882916+0000","last_became_peered":"2024-04-14T15:15:44.882916+0000","last_unstale":"2024-04-14T15:19:59.128377+0000","last_undegraded":"2024-04-14T15:19:59.128377+0000","last_fullsized":"2024-04-14T15:19:59.128377+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"280'9","last_scrub_stamp":"2024-04-14T15:17:50.908054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:17:50.908054+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":872,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.125828+0000","last_change":"2024-04-14T15:16:54.724807+0000","last_active":"2024-04-14T15:19:59.125828+0000","last_peered":"2024-04-14T15:19:59.125828+0000","last_clean":"2024-04-14T15:19:59.125828+0000","last_became_active":"2024-04-14T15:09:49.066068+0000","last_became_peered":"2024-04-14T15:09:49.066068+0000","last_unstale":"2024-04-14T15:19:59.125828+0000","last_undegraded":"2024-04-14T15:19:59.125828+0000","last_fullsized":"2024-04-14T15:19:59.125828+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:54.724681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:54.724681+0000","last_clean_scrub_stamp":"2024-04-14T15:16:54.724681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"91'130","reported_seq":1180,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.977118+0000","last_change":"2024-04-14T15:19:05.496449+0000","last_active":"2024-04-14T15:19:58.977118+0000","last_peered":"2024-04-14T15:19:58.977118+0000","last_clean":"2024-04-14T15:19:58.977118+0000","last_became_active":"2024-04-14T15:18:45.917220+0000","last_became_peered":"2024-04-14T15:18:45.917220+0000","last_unstale":"2024-04-14T15:19:58.977118+0000","last_undegraded":"2024-04-14T15:19:58.977118+0000","last_fullsized":"2024-04-14T15:19:58.977118+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:05.496303+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:05.496303+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:22.805010+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":439,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.126202+0000","last_change":"2024-04-14T15:18:30.438135+0000","last_active":"2024-04-14T15:19:59.126202+0000","last_peered":"2024-04-14T15:19:59.126202+0000","last_clean":"2024-04-14T15:19:59.126202+0000","last_became_active":"2024-04-14T15:15:44.889106+0000","last_became_peered":"2024-04-14T15:15:44.889106+0000","last_unstale":"2024-04-14T15:19:59.126202+0000","last_undegraded":"2024-04-14T15:19:59.126202+0000","last_fullsized":"2024-04-14T15:19:59.126202+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:30.438001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:18:30.438001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"479'19","reported_seq":729,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.983377+0000","last_change":"2024-04-14T15:17:37.421888+0000","last_active":"2024-04-14T15:19:58.983377+0000","last_peered":"2024-04-14T15:19:58.983377+0000","last_clean":"2024-04-14T15:19:58.983377+0000","last_became_active":"2024-04-14T15:15:44.884211+0000","last_became_peered":"2024-04-14T15:15:44.884211+0000","last_unstale":"2024-04-14T15:19:58.983377+0000","last_undegraded":"2024-04-14T15:19:58.983377+0000","last_fullsized":"2024-04-14T15:19:58.983377+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'13","last_scrub_stamp":"2024-04-14T15:17:37.421739+0000","last_deep_scrub":"183'13","last_deep_scrub_stamp":"2024-04-14T15:14:11.755252+0000","last_clean_scrub_stamp":"2024-04-14T15:17:37.421739+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":892,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.127139+0000","last_change":"2024-04-14T15:17:02.670294+0000","last_active":"2024-04-14T15:19:59.127139+0000","last_peered":"2024-04-14T15:19:59.127139+0000","last_clean":"2024-04-14T15:19:59.127139+0000","last_became_active":"2024-04-14T15:09:43.019301+0000","last_became_peered":"2024-04-14T15:09:43.019301+0000","last_unstale":"2024-04-14T15:19:59.127139+0000","last_undegraded":"2024-04-14T15:19:59.127139+0000","last_fullsized":"2024-04-14T15:19:59.127139+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:02.670213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:02.670213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"255'10","reported_seq":652,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.129759+0000","last_change":"2024-04-14T15:16:59.946889+0000","last_active":"2024-04-14T15:19:59.129759+0000","last_peered":"2024-04-14T15:19:59.129759+0000","last_clean":"2024-04-14T15:19:59.129759+0000","last_became_active":"2024-04-14T15:15:21.884468+0000","last_became_peered":"2024-04-14T15:15:21.884468+0000","last_unstale":"2024-04-14T15:19:59.129759+0000","last_undegraded":"2024-04-14T15:19:59.129759+0000","last_fullsized":"2024-04-14T15:19:59.129759+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'10","last_scrub_stamp":"2024-04-14T15:16:59.946804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:59.946804+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":879,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.975526+0000","last_change":"2024-04-14T15:19:15.642110+0000","last_active":"2024-04-14T15:19:58.975526+0000","last_peered":"2024-04-14T15:19:58.975526+0000","last_clean":"2024-04-14T15:19:58.975526+0000","last_became_active":"2024-04-14T15:09:49.065242+0000","last_became_peered":"2024-04-14T15:09:49.065242+0000","last_unstale":"2024-04-14T15:19:58.975526+0000","last_undegraded":"2024-04-14T15:19:58.975526+0000","last_fullsized":"2024-04-14T15:19:58.975526+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:15.642017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:19:15.642017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:37.747416+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.17","version":"91'130","reported_seq":413,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.971434+0000","last_change":"2024-04-14T15:19:03.143086+0000","last_active":"2024-04-14T15:19:58.971434+0000","last_peered":"2024-04-14T15:19:58.971434+0000","last_clean":"2024-04-14T15:19:58.971434+0000","last_became_active":"2024-04-14T15:19:02.936495+0000","last_became_peered":"2024-04-14T15:19:02.936495+0000","last_unstale":"2024-04-14T15:19:58.971434+0000","last_undegraded":"2024-04-14T15:19:58.971434+0000","last_fullsized":"2024-04-14T15:19:58.971434+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":446,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:03.143008+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:03.143008+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:11.581275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":787,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.983830+0000","last_change":"2024-04-14T15:18:40.735708+0000","last_active":"2024-04-14T15:19:58.983830+0000","last_peered":"2024-04-14T15:19:58.983830+0000","last_clean":"2024-04-14T15:19:58.983830+0000","last_became_active":"2024-04-14T15:12:56.389204+0000","last_became_peered":"2024-04-14T15:12:56.389204+0000","last_unstale":"2024-04-14T15:19:58.983830+0000","last_undegraded":"2024-04-14T15:19:58.983830+0000","last_fullsized":"2024-04-14T15:19:58.983830+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:40.735623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:18:40.735623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"479'19","reported_seq":428,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.980063+0000","last_change":"2024-04-14T15:19:10.762280+0000","last_active":"2024-04-14T15:19:58.980063+0000","last_peered":"2024-04-14T15:19:58.980063+0000","last_clean":"2024-04-14T15:19:58.980063+0000","last_became_active":"2024-04-14T15:16:24.993130+0000","last_became_peered":"2024-04-14T15:16:24.993130+0000","last_unstale":"2024-04-14T15:19:58.980063+0000","last_undegraded":"2024-04-14T15:19:58.980063+0000","last_fullsized":"2024-04-14T15:19:58.980063+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"344'13","last_scrub_stamp":"2024-04-14T15:19:10.762147+0000","last_deep_scrub":"344'13","last_deep_scrub_stamp":"2024-04-14T15:16:52.738163+0000","last_clean_scrub_stamp":"2024-04-14T15:19:10.762147+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:28.923046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":900,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.980348+0000","last_change":"2024-04-14T15:19:24.898168+0000","last_active":"2024-04-14T15:19:58.980348+0000","last_peered":"2024-04-14T15:19:58.980348+0000","last_clean":"2024-04-14T15:19:58.980348+0000","last_became_active":"2024-04-14T15:09:43.022480+0000","last_became_peered":"2024-04-14T15:09:43.022480+0000","last_unstale":"2024-04-14T15:19:58.980348+0000","last_undegraded":"2024-04-14T15:19:58.980348+0000","last_fullsized":"2024-04-14T15:19:58.980348+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:24.898086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:19:24.898086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:33.891662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":557,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.132213+0000","last_change":"2024-04-14T15:18:33.485536+0000","last_active":"2024-04-14T15:19:59.132213+0000","last_peered":"2024-04-14T15:19:59.132213+0000","last_clean":"2024-04-14T15:19:59.132213+0000","last_became_active":"2024-04-14T15:15:21.882643+0000","last_became_peered":"2024-04-14T15:15:21.882643+0000","last_unstale":"2024-04-14T15:19:59.132213+0000","last_undegraded":"2024-04-14T15:19:59.132213+0000","last_fullsized":"2024-04-14T15:19:59.132213+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:33.485411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:33.485411+0000","last_clean_scrub_stamp":"2024-04-14T15:18:33.485411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":883,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.123238+0000","last_change":"2024-04-14T15:18:24.039316+0000","last_active":"2024-04-14T15:19:59.123238+0000","last_peered":"2024-04-14T15:19:59.123238+0000","last_clean":"2024-04-14T15:19:59.123238+0000","last_became_active":"2024-04-14T15:09:49.059711+0000","last_became_peered":"2024-04-14T15:09:49.059711+0000","last_unstale":"2024-04-14T15:19:59.123238+0000","last_undegraded":"2024-04-14T15:19:59.123238+0000","last_fullsized":"2024-04-14T15:19:59.123238+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:24.039237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:18:24.039237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"91'130","reported_seq":410,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.122957+0000","last_change":"2024-04-14T15:19:04.045598+0000","last_active":"2024-04-14T15:19:59.122957+0000","last_peered":"2024-04-14T15:19:59.122957+0000","last_clean":"2024-04-14T15:19:59.122957+0000","last_became_active":"2024-04-14T15:19:02.938648+0000","last_became_peered":"2024-04-14T15:19:02.938648+0000","last_unstale":"2024-04-14T15:19:59.122957+0000","last_undegraded":"2024-04-14T15:19:59.122957+0000","last_fullsized":"2024-04-14T15:19:59.122957+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":446,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:04.045513+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:04.045513+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:30.423540+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":796,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.980158+0000","last_change":"2024-04-14T15:19:26.891939+0000","last_active":"2024-04-14T15:19:58.980158+0000","last_peered":"2024-04-14T15:19:58.980158+0000","last_clean":"2024-04-14T15:19:58.980158+0000","last_became_active":"2024-04-14T15:12:56.396294+0000","last_became_peered":"2024-04-14T15:12:56.396294+0000","last_unstale":"2024-04-14T15:19:58.980158+0000","last_undegraded":"2024-04-14T15:19:58.980158+0000","last_fullsized":"2024-04-14T15:19:58.980158+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:26.891862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:19:26.891862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:48.097098+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"479'24","reported_seq":923,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.972980+0000","last_change":"2024-04-14T15:19:17.590124+0000","last_active":"2024-04-14T15:19:58.972980+0000","last_peered":"2024-04-14T15:19:58.972980+0000","last_clean":"2024-04-14T15:19:58.972980+0000","last_became_active":"2024-04-14T15:09:45.026065+0000","last_became_peered":"2024-04-14T15:09:45.026065+0000","last_unstale":"2024-04-14T15:19:58.972980+0000","last_undegraded":"2024-04-14T15:19:58.972980+0000","last_fullsized":"2024-04-14T15:19:58.972980+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"464'16","last_scrub_stamp":"2024-04-14T15:19:17.589999+0000","last_deep_scrub":"464'16","last_deep_scrub_stamp":"2024-04-14T15:19:17.589999+0000","last_clean_scrub_stamp":"2024-04-14T15:19:17.589999+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:35.929485+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":893,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.128554+0000","last_change":"2024-04-14T15:17:34.986735+0000","last_active":"2024-04-14T15:19:59.128554+0000","last_peered":"2024-04-14T15:19:59.128554+0000","last_clean":"2024-04-14T15:19:59.128554+0000","last_became_active":"2024-04-14T15:09:43.023583+0000","last_became_peered":"2024-04-14T15:09:43.023583+0000","last_unstale":"2024-04-14T15:19:59.128554+0000","last_undegraded":"2024-04-14T15:19:59.128554+0000","last_fullsized":"2024-04-14T15:19:59.128554+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:34.986655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:34.986655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":877,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.124094+0000","last_change":"2024-04-14T15:17:00.205888+0000","last_active":"2024-04-14T15:19:59.124094+0000","last_peered":"2024-04-14T15:19:59.124094+0000","last_clean":"2024-04-14T15:19:59.124094+0000","last_became_active":"2024-04-14T15:09:47.041683+0000","last_became_peered":"2024-04-14T15:09:47.041683+0000","last_unstale":"2024-04-14T15:19:59.124094+0000","last_undegraded":"2024-04-14T15:19:59.124094+0000","last_fullsized":"2024-04-14T15:19:59.124094+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:00.205808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:08.486217+0000","last_clean_scrub_stamp":"2024-04-14T15:17:00.205808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":561,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.128040+0000","last_change":"2024-04-14T15:19:57.757627+0000","last_active":"2024-04-14T15:19:59.128040+0000","last_peered":"2024-04-14T15:19:59.128040+0000","last_clean":"2024-04-14T15:19:59.128040+0000","last_became_active":"2024-04-14T15:15:21.889567+0000","last_became_peered":"2024-04-14T15:15:21.889567+0000","last_unstale":"2024-04-14T15:19:59.128040+0000","last_undegraded":"2024-04-14T15:19:59.128040+0000","last_fullsized":"2024-04-14T15:19:59.128040+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:57.757547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:19:57.757547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:15.034106+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.15","version":"91'130","reported_seq":412,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.982697+0000","last_change":"2024-04-14T15:19:04.485222+0000","last_active":"2024-04-14T15:19:58.982697+0000","last_peered":"2024-04-14T15:19:58.982697+0000","last_clean":"2024-04-14T15:19:58.982697+0000","last_became_active":"2024-04-14T15:19:02.934529+0000","last_became_peered":"2024-04-14T15:19:02.934529+0000","last_unstale":"2024-04-14T15:19:58.982697+0000","last_undegraded":"2024-04-14T15:19:58.982697+0000","last_fullsized":"2024-04-14T15:19:58.982697+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":446,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:04.485114+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:04.485114+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:29.092721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":793,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-04-14T15:19:57.977569+0000","last_change":"2024-04-14T15:19:53.922021+0000","last_active":"2024-04-14T15:19:57.977569+0000","last_peered":"2024-04-14T15:19:57.977569+0000","last_clean":"2024-04-14T15:19:57.977569+0000","last_became_active":"2024-04-14T15:12:56.387723+0000","last_became_peered":"2024-04-14T15:12:56.387723+0000","last_unstale":"2024-04-14T15:19:57.977569+0000","last_undegraded":"2024-04-14T15:19:57.977569+0000","last_fullsized":"2024-04-14T15:19:57.977569+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:53.921934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:03.685045+0000","last_clean_scrub_stamp":"2024-04-14T15:19:53.921934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:08.254525+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.10","version":"479'8","reported_seq":897,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.128434+0000","last_change":"2024-04-14T15:17:53.181132+0000","last_active":"2024-04-14T15:19:59.128434+0000","last_peered":"2024-04-14T15:19:59.128434+0000","last_clean":"2024-04-14T15:19:59.128434+0000","last_became_active":"2024-04-14T15:09:45.038815+0000","last_became_peered":"2024-04-14T15:09:45.038815+0000","last_unstale":"2024-04-14T15:19:59.128434+0000","last_undegraded":"2024-04-14T15:19:59.128434+0000","last_fullsized":"2024-04-14T15:19:59.128434+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4","last_scrub_stamp":"2024-04-14T15:17:53.181053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:17:53.181053+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":442,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.979528+0000","last_change":"2024-04-14T15:17:48.845163+0000","last_active":"2024-04-14T15:19:58.979528+0000","last_peered":"2024-04-14T15:19:58.979528+0000","last_clean":"2024-04-14T15:19:58.979528+0000","last_became_active":"2024-04-14T15:15:44.884730+0000","last_became_peered":"2024-04-14T15:15:44.884730+0000","last_unstale":"2024-04-14T15:19:58.979528+0000","last_undegraded":"2024-04-14T15:19:58.979528+0000","last_fullsized":"2024-04-14T15:19:58.979528+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:48.845081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:08.712289+0000","last_clean_scrub_stamp":"2024-04-14T15:17:48.845081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":900,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.982468+0000","last_change":"2024-04-14T15:19:48.867805+0000","last_active":"2024-04-14T15:19:58.982468+0000","last_peered":"2024-04-14T15:19:58.982468+0000","last_clean":"2024-04-14T15:19:58.982468+0000","last_became_active":"2024-04-14T15:09:47.041074+0000","last_became_peered":"2024-04-14T15:09:47.041074+0000","last_unstale":"2024-04-14T15:19:58.982468+0000","last_undegraded":"2024-04-14T15:19:58.982468+0000","last_fullsized":"2024-04-14T15:19:58.982468+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:48.867704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:19:48.867704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:59.267600+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":871,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.980267+0000","last_change":"2024-04-14T15:16:44.728025+0000","last_active":"2024-04-14T15:19:58.980267+0000","last_peered":"2024-04-14T15:19:58.980267+0000","last_clean":"2024-04-14T15:19:58.980267+0000","last_became_active":"2024-04-14T15:09:49.061359+0000","last_became_peered":"2024-04-14T15:09:49.061359+0000","last_unstale":"2024-04-14T15:19:58.980267+0000","last_undegraded":"2024-04-14T15:19:58.980267+0000","last_fullsized":"2024-04-14T15:19:58.980267+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:44.727907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:44.727907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"91'130","reported_seq":960,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-04-14T15:19:57.976254+0000","last_change":"2024-04-14T15:19:06.939565+0000","last_active":"2024-04-14T15:19:57.976254+0000","last_peered":"2024-04-14T15:19:57.976254+0000","last_clean":"2024-04-14T15:19:57.976254+0000","last_became_active":"2024-04-14T15:18:45.913983+0000","last_became_peered":"2024-04-14T15:18:45.913983+0000","last_unstale":"2024-04-14T15:19:57.976254+0000","last_undegraded":"2024-04-14T15:19:57.976254+0000","last_fullsized":"2024-04-14T15:19:57.976254+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:06.939444+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:06.939444+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:20.862256+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":788,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.982368+0000","last_change":"2024-04-14T15:18:35.596732+0000","last_active":"2024-04-14T15:19:58.982368+0000","last_peered":"2024-04-14T15:19:58.982368+0000","last_clean":"2024-04-14T15:19:58.982368+0000","last_became_active":"2024-04-14T15:12:56.352528+0000","last_became_peered":"2024-04-14T15:12:56.352528+0000","last_unstale":"2024-04-14T15:19:58.982368+0000","last_undegraded":"2024-04-14T15:19:58.982368+0000","last_fullsized":"2024-04-14T15:19:58.982368+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:35.596656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:15:43.680852+0000","last_clean_scrub_stamp":"2024-04-14T15:18:35.596656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"479'24","reported_seq":558,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.129939+0000","last_change":"2024-04-14T15:17:43.061551+0000","last_active":"2024-04-14T15:19:59.129939+0000","last_peered":"2024-04-14T15:19:59.129939+0000","last_clean":"2024-04-14T15:19:59.129939+0000","last_became_active":"2024-04-14T15:15:21.884783+0000","last_became_peered":"2024-04-14T15:15:21.884783+0000","last_unstale":"2024-04-14T15:19:59.129939+0000","last_undegraded":"2024-04-14T15:19:59.129939+0000","last_fullsized":"2024-04-14T15:19:59.129939+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'16","last_scrub_stamp":"2024-04-14T15:17:43.061458+0000","last_deep_scrub":"102'16","last_deep_scrub_stamp":"2024-04-14T15:12:35.999326+0000","last_clean_scrub_stamp":"2024-04-14T15:17:43.061458+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":900,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.981859+0000","last_change":"2024-04-14T15:19:11.750498+0000","last_active":"2024-04-14T15:19:58.981859+0000","last_peered":"2024-04-14T15:19:58.981859+0000","last_clean":"2024-04-14T15:19:58.981859+0000","last_became_active":"2024-04-14T15:09:43.023220+0000","last_became_peered":"2024-04-14T15:09:43.023220+0000","last_unstale":"2024-04-14T15:19:58.981859+0000","last_undegraded":"2024-04-14T15:19:58.981859+0000","last_fullsized":"2024-04-14T15:19:58.981859+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:11.750419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:19:11.750419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:20.900268+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":883,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.977360+0000","last_change":"2024-04-14T15:19:54.634365+0000","last_active":"2024-04-14T15:19:58.977360+0000","last_peered":"2024-04-14T15:19:58.977360+0000","last_clean":"2024-04-14T15:19:58.977360+0000","last_became_active":"2024-04-14T15:09:47.044665+0000","last_became_peered":"2024-04-14T15:09:47.044665+0000","last_unstale":"2024-04-14T15:19:58.977360+0000","last_undegraded":"2024-04-14T15:19:58.977360+0000","last_fullsized":"2024-04-14T15:19:58.977360+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:54.634279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:19:54.634279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:23.855974+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":561,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.132464+0000","last_change":"2024-04-14T15:19:23.462816+0000","last_active":"2024-04-14T15:19:59.132464+0000","last_peered":"2024-04-14T15:19:59.132464+0000","last_clean":"2024-04-14T15:19:59.132464+0000","last_became_active":"2024-04-14T15:15:21.883940+0000","last_became_peered":"2024-04-14T15:15:21.883940+0000","last_unstale":"2024-04-14T15:19:59.132464+0000","last_undegraded":"2024-04-14T15:19:59.132464+0000","last_fullsized":"2024-04-14T15:19:59.132464+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:23.462727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:19:23.462727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:46.221952+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1b","version":"91'130","reported_seq":936,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.971626+0000","last_change":"2024-04-14T15:19:06.558576+0000","last_active":"2024-04-14T15:19:58.971626+0000","last_peered":"2024-04-14T15:19:58.971626+0000","last_clean":"2024-04-14T15:19:58.971626+0000","last_became_active":"2024-04-14T15:19:02.936749+0000","last_became_peered":"2024-04-14T15:19:02.936749+0000","last_unstale":"2024-04-14T15:19:58.971626+0000","last_undegraded":"2024-04-14T15:19:58.971626+0000","last_fullsized":"2024-04-14T15:19:58.971626+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":446,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:06.558465+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:06.558465+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:28.851475+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":787,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.127964+0000","last_change":"2024-04-14T15:18:25.997409+0000","last_active":"2024-04-14T15:19:59.127964+0000","last_peered":"2024-04-14T15:19:59.127964+0000","last_clean":"2024-04-14T15:19:59.127964+0000","last_became_active":"2024-04-14T15:12:56.335230+0000","last_became_peered":"2024-04-14T15:12:56.335230+0000","last_unstale":"2024-04-14T15:19:59.127964+0000","last_undegraded":"2024-04-14T15:19:59.127964+0000","last_fullsized":"2024-04-14T15:19:59.127964+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:25.997333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:18:25.997333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"479'20","reported_seq":911,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.126487+0000","last_change":"2024-04-14T15:16:49.779653+0000","last_active":"2024-04-14T15:19:59.126487+0000","last_peered":"2024-04-14T15:19:59.126487+0000","last_clean":"2024-04-14T15:19:59.126487+0000","last_became_active":"2024-04-14T15:09:45.181456+0000","last_became_peered":"2024-04-14T15:09:45.181456+0000","last_unstale":"2024-04-14T15:19:59.126487+0000","last_undegraded":"2024-04-14T15:19:59.126487+0000","last_fullsized":"2024-04-14T15:19:59.126487+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-04-14T15:16:49.779557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:16:49.779557+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":449,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.971726+0000","last_change":"2024-04-14T15:19:57.575006+0000","last_active":"2024-04-14T15:19:58.971726+0000","last_peered":"2024-04-14T15:19:58.971726+0000","last_clean":"2024-04-14T15:19:58.971726+0000","last_became_active":"2024-04-14T15:15:44.878336+0000","last_became_peered":"2024-04-14T15:15:44.878336+0000","last_unstale":"2024-04-14T15:19:58.971726+0000","last_undegraded":"2024-04-14T15:19:58.971726+0000","last_fullsized":"2024-04-14T15:19:58.971726+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:57.574879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:19:57.574879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:02.545636+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"72'8","reported_seq":1027,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.982181+0000","last_change":"2024-04-14T15:18:28.793061+0000","last_active":"2024-04-14T15:19:58.982181+0000","last_peered":"2024-04-14T15:19:58.982181+0000","last_clean":"2024-04-14T15:19:58.982181+0000","last_became_active":"2024-04-14T15:09:47.048216+0000","last_became_peered":"2024-04-14T15:09:47.048216+0000","last_unstale":"2024-04-14T15:19:58.982181+0000","last_undegraded":"2024-04-14T15:19:58.982181+0000","last_fullsized":"2024-04-14T15:19:58.982181+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:18:28.792975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:18:28.792975+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"72'1","reported_seq":880,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.982060+0000","last_change":"2024-04-14T15:17:50.529174+0000","last_active":"2024-04-14T15:19:58.982060+0000","last_peered":"2024-04-14T15:19:58.982060+0000","last_clean":"2024-04-14T15:19:58.982060+0000","last_became_active":"2024-04-14T15:09:49.062051+0000","last_became_peered":"2024-04-14T15:09:49.062051+0000","last_unstale":"2024-04-14T15:19:58.982060+0000","last_undegraded":"2024-04-14T15:19:58.982060+0000","last_fullsized":"2024-04-14T15:19:58.982060+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-04-14T15:17:50.529089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:17:50.529089+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"91'130","reported_seq":788,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.971822+0000","last_change":"2024-04-14T15:19:03.537885+0000","last_active":"2024-04-14T15:19:58.971822+0000","last_peered":"2024-04-14T15:19:58.971822+0000","last_clean":"2024-04-14T15:19:58.971822+0000","last_became_active":"2024-04-14T15:19:02.930025+0000","last_became_peered":"2024-04-14T15:19:02.930025+0000","last_unstale":"2024-04-14T15:19:58.971822+0000","last_undegraded":"2024-04-14T15:19:58.971822+0000","last_fullsized":"2024-04-14T15:19:58.971822+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":446,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:03.537803+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:03.537803+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:03.649948+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":789,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.123455+0000","last_change":"2024-04-14T15:19:13.345390+0000","last_active":"2024-04-14T15:19:59.123455+0000","last_peered":"2024-04-14T15:19:59.123455+0000","last_clean":"2024-04-14T15:19:59.123455+0000","last_became_active":"2024-04-14T15:12:56.543793+0000","last_became_peered":"2024-04-14T15:12:56.543793+0000","last_unstale":"2024-04-14T15:19:59.123455+0000","last_undegraded":"2024-04-14T15:19:59.123455+0000","last_fullsized":"2024-04-14T15:19:59.123455+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:13.345316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:19:13.345316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:38.673629+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"479'19","reported_seq":937,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.982307+0000","last_change":"2024-04-14T15:18:31.818435+0000","last_active":"2024-04-14T15:19:58.982307+0000","last_peered":"2024-04-14T15:19:58.982307+0000","last_clean":"2024-04-14T15:19:58.982307+0000","last_became_active":"2024-04-14T15:10:07.764183+0000","last_became_peered":"2024-04-14T15:10:07.764183+0000","last_unstale":"2024-04-14T15:19:58.982307+0000","last_undegraded":"2024-04-14T15:19:58.982307+0000","last_fullsized":"2024-04-14T15:19:58.982307+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-04-14T15:18:31.818311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:18:31.818311+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"65'2","reported_seq":572,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.128445+0000","last_change":"2024-04-14T15:16:56.943047+0000","last_active":"2024-04-14T15:19:59.128445+0000","last_peered":"2024-04-14T15:19:59.128445+0000","last_clean":"2024-04-14T15:19:59.128445+0000","last_became_active":"2024-04-14T15:15:21.885343+0000","last_became_peered":"2024-04-14T15:15:21.885343+0000","last_unstale":"2024-04-14T15:19:59.128445+0000","last_undegraded":"2024-04-14T15:19:59.128445+0000","last_fullsized":"2024-04-14T15:19:59.128445+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-04-14T15:16:56.942840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:56.942840+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":885,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.126608+0000","last_change":"2024-04-14T15:19:21.349524+0000","last_active":"2024-04-14T15:19:59.126608+0000","last_peered":"2024-04-14T15:19:59.126608+0000","last_clean":"2024-04-14T15:19:59.126608+0000","last_became_active":"2024-04-14T15:09:47.045730+0000","last_became_peered":"2024-04-14T15:09:47.045730+0000","last_unstale":"2024-04-14T15:19:59.126608+0000","last_undegraded":"2024-04-14T15:19:59.126608+0000","last_fullsized":"2024-04-14T15:19:59.126608+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:21.349441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:19:21.349441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:47.517518+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":873,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.128238+0000","last_change":"2024-04-14T15:16:57.235339+0000","last_active":"2024-04-14T15:19:59.128238+0000","last_peered":"2024-04-14T15:19:59.128238+0000","last_clean":"2024-04-14T15:19:59.128238+0000","last_became_active":"2024-04-14T15:09:49.057545+0000","last_became_peered":"2024-04-14T15:09:49.057545+0000","last_unstale":"2024-04-14T15:19:59.128238+0000","last_undegraded":"2024-04-14T15:19:59.128238+0000","last_fullsized":"2024-04-14T15:19:59.128238+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:57.235261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:21.247695+0000","last_clean_scrub_stamp":"2024-04-14T15:16:57.235261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.19","version":"91'130","reported_seq":632,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.127673+0000","last_change":"2024-04-14T15:19:06.383564+0000","last_active":"2024-04-14T15:19:59.127673+0000","last_peered":"2024-04-14T15:19:59.127673+0000","last_clean":"2024-04-14T15:19:59.127673+0000","last_became_active":"2024-04-14T15:19:02.941363+0000","last_became_peered":"2024-04-14T15:19:02.941363+0000","last_unstale":"2024-04-14T15:19:59.127673+0000","last_undegraded":"2024-04-14T15:19:59.127673+0000","last_fullsized":"2024-04-14T15:19:59.127673+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":446,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:06.383478+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:06.383478+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:24.667992+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":441,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.972427+0000","last_change":"2024-04-14T15:17:44.364114+0000","last_active":"2024-04-14T15:19:58.972427+0000","last_peered":"2024-04-14T15:19:58.972427+0000","last_clean":"2024-04-14T15:19:58.972427+0000","last_became_active":"2024-04-14T15:15:44.882604+0000","last_became_peered":"2024-04-14T15:15:44.882604+0000","last_unstale":"2024-04-14T15:19:58.972427+0000","last_undegraded":"2024-04-14T15:19:58.972427+0000","last_fullsized":"2024-04-14T15:19:58.972427+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:44.364013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:17:44.364013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"479'27","reported_seq":927,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-04-14T15:19:57.977082+0000","last_change":"2024-04-14T15:18:32.950805+0000","last_active":"2024-04-14T15:19:57.977082+0000","last_peered":"2024-04-14T15:19:57.977082+0000","last_clean":"2024-04-14T15:19:57.977082+0000","last_became_active":"2024-04-14T15:09:45.032887+0000","last_became_peered":"2024-04-14T15:09:45.032887+0000","last_unstale":"2024-04-14T15:19:57.977082+0000","last_undegraded":"2024-04-14T15:19:57.977082+0000","last_fullsized":"2024-04-14T15:19:57.977082+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-04-14T15:18:32.950706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:18:32.950706+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":895,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.980361+0000","last_change":"2024-04-14T15:18:27.799078+0000","last_active":"2024-04-14T15:19:58.980361+0000","last_peered":"2024-04-14T15:19:58.980361+0000","last_clean":"2024-04-14T15:19:58.980361+0000","last_became_active":"2024-04-14T15:09:43.021494+0000","last_became_peered":"2024-04-14T15:09:43.021494+0000","last_unstale":"2024-04-14T15:19:58.980361+0000","last_undegraded":"2024-04-14T15:19:58.980361+0000","last_fullsized":"2024-04-14T15:19:58.980361+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:27.798964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:18:27.798964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":885,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.128469+0000","last_change":"2024-04-14T15:18:22.953738+0000","last_active":"2024-04-14T15:19:59.128469+0000","last_peered":"2024-04-14T15:19:59.128469+0000","last_clean":"2024-04-14T15:19:59.128469+0000","last_became_active":"2024-04-14T15:09:47.042733+0000","last_became_peered":"2024-04-14T15:09:47.042733+0000","last_unstale":"2024-04-14T15:19:59.128469+0000","last_undegraded":"2024-04-14T15:19:59.128469+0000","last_fullsized":"2024-04-14T15:19:59.128469+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:22.953661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:18:22.953661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":880,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.975102+0000","last_change":"2024-04-14T15:17:54.453289+0000","last_active":"2024-04-14T15:19:58.975102+0000","last_peered":"2024-04-14T15:19:58.975102+0000","last_clean":"2024-04-14T15:19:58.975102+0000","last_became_active":"2024-04-14T15:09:49.127979+0000","last_became_peered":"2024-04-14T15:09:49.127979+0000","last_unstale":"2024-04-14T15:19:58.975102+0000","last_undegraded":"2024-04-14T15:19:58.975102+0000","last_fullsized":"2024-04-14T15:19:58.975102+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:54.453174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:54.453174+0000","last_clean_scrub_stamp":"2024-04-14T15:17:54.453174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.18","version":"91'130","reported_seq":790,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.123367+0000","last_change":"2024-04-14T15:19:02.957443+0000","last_active":"2024-04-14T15:19:59.123367+0000","last_peered":"2024-04-14T15:19:59.123367+0000","last_clean":"2024-04-14T15:19:59.123367+0000","last_became_active":"2024-04-14T15:19:02.940290+0000","last_became_peered":"2024-04-14T15:19:02.940290+0000","last_unstale":"2024-04-14T15:19:59.123367+0000","last_undegraded":"2024-04-14T15:19:59.123367+0000","last_fullsized":"2024-04-14T15:19:59.123367+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":446,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:02.957360+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:02.957360+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:07.851285+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":788,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.979864+0000","last_change":"2024-04-14T15:18:26.532035+0000","last_active":"2024-04-14T15:19:58.979864+0000","last_peered":"2024-04-14T15:19:58.979864+0000","last_clean":"2024-04-14T15:19:58.979864+0000","last_became_active":"2024-04-14T15:12:56.392352+0000","last_became_peered":"2024-04-14T15:12:56.392352+0000","last_unstale":"2024-04-14T15:19:58.979864+0000","last_undegraded":"2024-04-14T15:19:58.979864+0000","last_fullsized":"2024-04-14T15:19:58.979864+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:26.531955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:18:26.531955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"479'18","reported_seq":601,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.128706+0000","last_change":"2024-04-14T15:19:17.364178+0000","last_active":"2024-04-14T15:19:59.128706+0000","last_peered":"2024-04-14T15:19:59.128706+0000","last_clean":"2024-04-14T15:19:59.128706+0000","last_became_active":"2024-04-14T15:15:21.877838+0000","last_became_peered":"2024-04-14T15:15:21.877838+0000","last_unstale":"2024-04-14T15:19:59.128706+0000","last_undegraded":"2024-04-14T15:19:59.128706+0000","last_fullsized":"2024-04-14T15:19:59.128706+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'12","last_scrub_stamp":"2024-04-14T15:19:17.364059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:19:17.364059+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:22.503752+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":573,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.132296+0000","last_change":"2024-04-14T15:19:25.470235+0000","last_active":"2024-04-14T15:19:59.132296+0000","last_peered":"2024-04-14T15:19:59.132296+0000","last_clean":"2024-04-14T15:19:59.132296+0000","last_became_active":"2024-04-14T15:15:21.883544+0000","last_became_peered":"2024-04-14T15:15:21.883544+0000","last_unstale":"2024-04-14T15:19:59.132296+0000","last_undegraded":"2024-04-14T15:19:59.132296+0000","last_fullsized":"2024-04-14T15:19:59.132296+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:25.470160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:19:25.470160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:47.982067+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":883,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.972449+0000","last_change":"2024-04-14T15:17:01.098681+0000","last_active":"2024-04-14T15:19:58.972449+0000","last_peered":"2024-04-14T15:19:58.972449+0000","last_clean":"2024-04-14T15:19:58.972449+0000","last_became_active":"2024-04-14T15:09:47.048045+0000","last_became_peered":"2024-04-14T15:09:47.048045+0000","last_unstale":"2024-04-14T15:19:58.972449+0000","last_undegraded":"2024-04-14T15:19:58.972449+0000","last_fullsized":"2024-04-14T15:19:58.972449+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:01.098599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:17:01.098599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":563,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.129286+0000","last_change":"2024-04-14T15:19:19.371279+0000","last_active":"2024-04-14T15:19:59.129286+0000","last_peered":"2024-04-14T15:19:59.129286+0000","last_clean":"2024-04-14T15:19:59.129286+0000","last_became_active":"2024-04-14T15:15:21.889691+0000","last_became_peered":"2024-04-14T15:15:21.889691+0000","last_unstale":"2024-04-14T15:19:59.129286+0000","last_undegraded":"2024-04-14T15:19:59.129286+0000","last_fullsized":"2024-04-14T15:19:59.129286+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:19.371182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:19:19.371182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:48.666091+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":784,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-04-14T15:19:57.977864+0000","last_change":"2024-04-14T15:17:37.766225+0000","last_active":"2024-04-14T15:19:57.977864+0000","last_peered":"2024-04-14T15:19:57.977864+0000","last_clean":"2024-04-14T15:19:57.977864+0000","last_became_active":"2024-04-14T15:12:56.375278+0000","last_became_peered":"2024-04-14T15:12:56.375278+0000","last_unstale":"2024-04-14T15:19:57.977864+0000","last_undegraded":"2024-04-14T15:19:57.977864+0000","last_fullsized":"2024-04-14T15:19:57.977864+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:37.766116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:17:37.766116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"479'19","reported_seq":916,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.972325+0000","last_change":"2024-04-14T15:17:52.445394+0000","last_active":"2024-04-14T15:19:58.972325+0000","last_peered":"2024-04-14T15:19:58.972325+0000","last_clean":"2024-04-14T15:19:58.972325+0000","last_became_active":"2024-04-14T15:09:45.030093+0000","last_became_peered":"2024-04-14T15:09:45.030093+0000","last_unstale":"2024-04-14T15:19:58.972325+0000","last_undegraded":"2024-04-14T15:19:58.972325+0000","last_fullsized":"2024-04-14T15:19:58.972325+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'13","last_scrub_stamp":"2024-04-14T15:17:52.445294+0000","last_deep_scrub":"276'13","last_deep_scrub_stamp":"2024-04-14T15:15:43.969775+0000","last_clean_scrub_stamp":"2024-04-14T15:17:52.445294+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":589,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.981215+0000","last_change":"2024-04-14T15:17:02.800274+0000","last_active":"2024-04-14T15:19:58.981215+0000","last_peered":"2024-04-14T15:19:58.981215+0000","last_clean":"2024-04-14T15:19:58.981215+0000","last_became_active":"2024-04-14T15:14:30.857727+0000","last_became_peered":"2024-04-14T15:14:30.857727+0000","last_unstale":"2024-04-14T15:19:58.981215+0000","last_undegraded":"2024-04-14T15:19:58.981215+0000","last_fullsized":"2024-04-14T15:19:58.981215+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:02.800166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:02.800166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":442,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.127343+0000","last_change":"2024-04-14T15:16:59.695423+0000","last_active":"2024-04-14T15:19:59.127343+0000","last_peered":"2024-04-14T15:19:59.127343+0000","last_clean":"2024-04-14T15:19:59.127343+0000","last_became_active":"2024-04-14T15:15:44.883121+0000","last_became_peered":"2024-04-14T15:15:44.883121+0000","last_unstale":"2024-04-14T15:19:59.127343+0000","last_undegraded":"2024-04-14T15:19:59.127343+0000","last_fullsized":"2024-04-14T15:19:59.127343+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:59.695337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:59.695337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":880,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.123762+0000","last_change":"2024-04-14T15:17:33.952503+0000","last_active":"2024-04-14T15:19:59.123762+0000","last_peered":"2024-04-14T15:19:59.123762+0000","last_clean":"2024-04-14T15:19:59.123762+0000","last_became_active":"2024-04-14T15:09:49.061756+0000","last_became_peered":"2024-04-14T15:09:49.061756+0000","last_unstale":"2024-04-14T15:19:59.123762+0000","last_undegraded":"2024-04-14T15:19:59.123762+0000","last_fullsized":"2024-04-14T15:19:59.123762+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:33.952312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:17:33.952312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"91'130","reported_seq":420,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.129964+0000","last_change":"2024-04-14T15:19:03.395696+0000","last_active":"2024-04-14T15:19:59.129964+0000","last_peered":"2024-04-14T15:19:59.129964+0000","last_clean":"2024-04-14T15:19:59.129964+0000","last_became_active":"2024-04-14T15:19:02.936733+0000","last_became_peered":"2024-04-14T15:19:02.936733+0000","last_unstale":"2024-04-14T15:19:59.129964+0000","last_undegraded":"2024-04-14T15:19:59.129964+0000","last_fullsized":"2024-04-14T15:19:59.129964+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":446,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:03.395612+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:03.395612+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:11.326775+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":797,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.980636+0000","last_change":"2024-04-14T15:19:12.723151+0000","last_active":"2024-04-14T15:19:58.980636+0000","last_peered":"2024-04-14T15:19:58.980636+0000","last_clean":"2024-04-14T15:19:58.980636+0000","last_became_active":"2024-04-14T15:12:56.232831+0000","last_became_peered":"2024-04-14T15:12:56.232831+0000","last_unstale":"2024-04-14T15:19:58.980636+0000","last_undegraded":"2024-04-14T15:19:58.980636+0000","last_fullsized":"2024-04-14T15:19:58.980636+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:12.723073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:19:12.723073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:34.125961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1b","version":"479'9","reported_seq":902,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.973172+0000","last_change":"2024-04-14T15:18:35.481608+0000","last_active":"2024-04-14T15:19:58.973172+0000","last_peered":"2024-04-14T15:19:58.973172+0000","last_clean":"2024-04-14T15:19:58.973172+0000","last_became_active":"2024-04-14T15:09:45.028667+0000","last_became_peered":"2024-04-14T15:09:45.028667+0000","last_unstale":"2024-04-14T15:19:58.973172+0000","last_undegraded":"2024-04-14T15:19:58.973172+0000","last_fullsized":"2024-04-14T15:19:58.973172+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'7","last_scrub_stamp":"2024-04-14T15:18:35.481504+0000","last_deep_scrub":"87'7","last_deep_scrub_stamp":"2024-04-14T15:11:10.510785+0000","last_clean_scrub_stamp":"2024-04-14T15:18:35.481504+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":365,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-04-14T15:19:57.977234+0000","last_change":"2024-04-14T15:19:26.879798+0000","last_active":"2024-04-14T15:19:57.977234+0000","last_peered":"2024-04-14T15:19:57.977234+0000","last_clean":"2024-04-14T15:19:57.977234+0000","last_became_active":"2024-04-14T15:16:24.910246+0000","last_became_peered":"2024-04-14T15:16:24.910246+0000","last_unstale":"2024-04-14T15:19:57.977234+0000","last_undegraded":"2024-04-14T15:19:57.977234+0000","last_fullsized":"2024-04-14T15:19:57.977234+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:26.879685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:08.254601+0000","last_clean_scrub_stamp":"2024-04-14T15:19:26.879685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:37.880480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":883,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.980089+0000","last_change":"2024-04-14T15:19:15.567537+0000","last_active":"2024-04-14T15:19:58.980089+0000","last_peered":"2024-04-14T15:19:58.980089+0000","last_clean":"2024-04-14T15:19:58.980089+0000","last_became_active":"2024-04-14T15:09:47.046517+0000","last_became_peered":"2024-04-14T15:09:47.046517+0000","last_unstale":"2024-04-14T15:19:58.980089+0000","last_undegraded":"2024-04-14T15:19:58.980089+0000","last_fullsized":"2024-04-14T15:19:58.980089+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:15.567432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:19:15.567432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:24.223440+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":444,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.129547+0000","last_change":"2024-04-14T15:18:21.023508+0000","last_active":"2024-04-14T15:19:59.129547+0000","last_peered":"2024-04-14T15:19:59.129547+0000","last_clean":"2024-04-14T15:19:59.129547+0000","last_became_active":"2024-04-14T15:15:44.887227+0000","last_became_peered":"2024-04-14T15:15:44.887227+0000","last_unstale":"2024-04-14T15:19:59.129547+0000","last_undegraded":"2024-04-14T15:19:59.129547+0000","last_fullsized":"2024-04-14T15:19:59.129547+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:21.023426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:56.268086+0000","last_clean_scrub_stamp":"2024-04-14T15:18:21.023426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1d","version":"91'130","reported_seq":420,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.127758+0000","last_change":"2024-04-14T15:19:03.081107+0000","last_active":"2024-04-14T15:19:59.127758+0000","last_peered":"2024-04-14T15:19:59.127758+0000","last_clean":"2024-04-14T15:19:59.127758+0000","last_became_active":"2024-04-14T15:19:02.934382+0000","last_became_peered":"2024-04-14T15:19:02.934382+0000","last_unstale":"2024-04-14T15:19:59.127758+0000","last_undegraded":"2024-04-14T15:19:59.127758+0000","last_fullsized":"2024-04-14T15:19:59.127758+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":446,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:03.081016+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:03.081016+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:09.658638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":787,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.982890+0000","last_change":"2024-04-14T15:18:34.558447+0000","last_active":"2024-04-14T15:19:58.982890+0000","last_peered":"2024-04-14T15:19:58.982890+0000","last_clean":"2024-04-14T15:19:58.982890+0000","last_became_active":"2024-04-14T15:12:56.189117+0000","last_became_peered":"2024-04-14T15:12:56.189117+0000","last_unstale":"2024-04-14T15:19:58.982890+0000","last_undegraded":"2024-04-14T15:19:58.982890+0000","last_fullsized":"2024-04-14T15:19:58.982890+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:34.558312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:34.558312+0000","last_clean_scrub_stamp":"2024-04-14T15:18:34.558312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.18","version":"479'15","reported_seq":912,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.971456+0000","last_change":"2024-04-14T15:18:36.441934+0000","last_active":"2024-04-14T15:19:58.971456+0000","last_peered":"2024-04-14T15:19:58.971456+0000","last_clean":"2024-04-14T15:19:58.971456+0000","last_became_active":"2024-04-14T15:09:45.032079+0000","last_became_peered":"2024-04-14T15:09:45.032079+0000","last_unstale":"2024-04-14T15:19:58.971456+0000","last_undegraded":"2024-04-14T15:19:58.971456+0000","last_fullsized":"2024-04-14T15:19:58.971456+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-04-14T15:18:36.441813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:18:36.441813+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:03.217663+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":567,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.132894+0000","last_change":"2024-04-14T15:17:00.948362+0000","last_active":"2024-04-14T15:19:59.132894+0000","last_peered":"2024-04-14T15:19:59.132894+0000","last_clean":"2024-04-14T15:19:59.132894+0000","last_became_active":"2024-04-14T15:15:21.891926+0000","last_became_peered":"2024-04-14T15:15:21.891926+0000","last_unstale":"2024-04-14T15:19:59.132894+0000","last_undegraded":"2024-04-14T15:19:59.132894+0000","last_fullsized":"2024-04-14T15:19:59.132894+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:00.948283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:00.948283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":885,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.129143+0000","last_change":"2024-04-14T15:19:56.903138+0000","last_active":"2024-04-14T15:19:59.129143+0000","last_peered":"2024-04-14T15:19:59.129143+0000","last_clean":"2024-04-14T15:19:59.129143+0000","last_became_active":"2024-04-14T15:09:47.041486+0000","last_became_peered":"2024-04-14T15:09:47.041486+0000","last_unstale":"2024-04-14T15:19:59.129143+0000","last_undegraded":"2024-04-14T15:19:59.129143+0000","last_fullsized":"2024-04-14T15:19:59.129143+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:56.903059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:19:56.903059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:25.052114+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":871,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.972151+0000","last_change":"2024-04-14T15:16:55.184211+0000","last_active":"2024-04-14T15:19:58.972151+0000","last_peered":"2024-04-14T15:19:58.972151+0000","last_clean":"2024-04-14T15:19:58.972151+0000","last_became_active":"2024-04-14T15:09:49.058904+0000","last_became_peered":"2024-04-14T15:09:49.058904+0000","last_unstale":"2024-04-14T15:19:58.972151+0000","last_undegraded":"2024-04-14T15:19:58.972151+0000","last_fullsized":"2024-04-14T15:19:58.972151+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:55.184094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:55.184094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1c","version":"91'130","reported_seq":933,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.980893+0000","last_change":"2024-04-14T15:19:03.486816+0000","last_active":"2024-04-14T15:19:58.980893+0000","last_peered":"2024-04-14T15:19:58.980893+0000","last_clean":"2024-04-14T15:19:58.980893+0000","last_became_active":"2024-04-14T15:19:02.934458+0000","last_became_peered":"2024-04-14T15:19:02.934458+0000","last_unstale":"2024-04-14T15:19:58.980893+0000","last_undegraded":"2024-04-14T15:19:58.980893+0000","last_fullsized":"2024-04-14T15:19:58.980893+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":446,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:03.486726+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:03.486726+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:08.034646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":441,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.982207+0000","last_change":"2024-04-14T15:17:52.593706+0000","last_active":"2024-04-14T15:19:58.982207+0000","last_peered":"2024-04-14T15:19:58.982207+0000","last_clean":"2024-04-14T15:19:58.982207+0000","last_became_active":"2024-04-14T15:15:44.889605+0000","last_became_peered":"2024-04-14T15:15:44.889605+0000","last_unstale":"2024-04-14T15:19:58.982207+0000","last_undegraded":"2024-04-14T15:19:58.982207+0000","last_fullsized":"2024-04-14T15:19:58.982207+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:52.593618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:17:52.593618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"479'34","reported_seq":619,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.130057+0000","last_change":"2024-04-14T15:19:53.781367+0000","last_active":"2024-04-14T15:19:59.130057+0000","last_peered":"2024-04-14T15:19:59.130057+0000","last_clean":"2024-04-14T15:19:59.130057+0000","last_became_active":"2024-04-14T15:15:21.884310+0000","last_became_peered":"2024-04-14T15:15:21.884310+0000","last_unstale":"2024-04-14T15:19:59.130057+0000","last_undegraded":"2024-04-14T15:19:59.130057+0000","last_fullsized":"2024-04-14T15:19:59.130057+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'34","last_scrub_stamp":"2024-04-14T15:19:53.781241+0000","last_deep_scrub":"306'22","last_deep_scrub_stamp":"2024-04-14T15:16:13.898106+0000","last_clean_scrub_stamp":"2024-04-14T15:19:53.781241+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:54.804612+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":892,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.122872+0000","last_change":"2024-04-14T15:19:18.347632+0000","last_active":"2024-04-14T15:19:59.122872+0000","last_peered":"2024-04-14T15:19:59.122872+0000","last_clean":"2024-04-14T15:19:59.122872+0000","last_became_active":"2024-04-14T15:09:43.017351+0000","last_became_peered":"2024-04-14T15:09:43.017351+0000","last_unstale":"2024-04-14T15:19:59.122872+0000","last_undegraded":"2024-04-14T15:19:59.122872+0000","last_fullsized":"2024-04-14T15:19:59.122872+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:18.347533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:19:18.347533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:22.403586+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":883,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:58.971707+0000","last_change":"2024-04-14T15:19:49.631364+0000","last_active":"2024-04-14T15:19:58.971707+0000","last_peered":"2024-04-14T15:19:58.971707+0000","last_clean":"2024-04-14T15:19:58.971707+0000","last_became_active":"2024-04-14T15:09:47.042538+0000","last_became_peered":"2024-04-14T15:09:47.042538+0000","last_unstale":"2024-04-14T15:19:58.971707+0000","last_undegraded":"2024-04-14T15:19:58.971707+0000","last_fullsized":"2024-04-14T15:19:58.971707+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:49.631273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:08.512130+0000","last_clean_scrub_stamp":"2024-04-14T15:19:49.631273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:09.671784+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":877,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-14T15:19:59.131944+0000","last_change":"2024-04-14T15:17:56.237366+0000","last_active":"2024-04-14T15:19:59.131944+0000","last_peered":"2024-04-14T15:19:59.131944+0000","last_clean":"2024-04-14T15:19:59.131944+0000","last_became_active":"2024-04-14T15:09:49.129636+0000","last_became_peered":"2024-04-14T15:09:49.129636+0000","last_unstale":"2024-04-14T15:19:59.131944+0000","last_undegraded":"2024-04-14T15:19:59.131944+0000","last_fullsized":"2024-04-14T15:19:59.131944+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:56.237273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:17:56.237273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":304,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":7},{"poolid":110,"num_pg":32,"stat_sum":{"num_bytes":8195902,"num_objects":172,"num_object_clones":9,"num_object_copies":516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":171,"num_whiteouts":0,"num_read":112,"num_read_kb":10,"num_write":625,"num_write_kb":12712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":41,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":18,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":26628096,"data_stored":24587697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1962,"ondisk_log_size":1962,"up":96,"acting":96,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4030,"ondisk_log_size":4030,"up":93,"acting":93,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":609,"num_read_kb":604,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1124,"num_read_kb":881,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":779,"ondisk_log_size":779,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71,"ondisk_log_size":71,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103297,"num_pgs":99,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":78020,"kb_used_data":16292,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":93667132,"statfs":{"total":95995035648,"available":95915143168,"internally_reserved":0,"allocated":16683008,"data_stored":13107713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":63175114},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397636,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":75600,"kb_used_data":8304,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93669552,"statfs":{"total":95995035648,"available":95917621248,"internally_reserved":0,"allocated":8503296,"data_stored":4978577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575006,"num_pgs":90,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":75316,"kb_used_data":12192,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93669836,"statfs":{"total":95995035648,"available":95917912064,"internally_reserved":0,"allocated":12484608,"data_stored":8845518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738531,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":87904,"kb_used_data":12276,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93657248,"statfs":{"total":95995035648,"available":95905021952,"internally_reserved":0,"allocated":12570624,"data_stored":8979043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77396423},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411482,"num_pgs":87,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":87872,"kb_used_data":12256,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93657280,"statfs":{"total":95995035648,"available":95905054720,"internally_reserved":0,"allocated":12550144,"data_stored":8913100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182548,"num_pgs":90,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":57552,"kb_used_data":8292,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93687600,"statfs":{"total":95995035648,"available":95936102400,"internally_reserved":0,"allocated":8491008,"data_stored":4978919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986318,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":75256,"kb_used_data":7960,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93669896,"statfs":{"total":95995035648,"available":95917973504,"internally_reserved":0,"allocated":8151040,"data_stored":4650435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659275,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":79620,"kb_used_data":12312,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93665532,"statfs":{"total":95995035648,"available":95913504768,"internally_reserved":0,"allocated":12607488,"data_stored":9106887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4284416,"data_stored":4000640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4505600,"data_stored":4195039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4354048,"data_stored":4000832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":184320,"data_stored":434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4403200,"data_stored":4194784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":229376,"data_stored":534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8470528,"data_stored":8194960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":304,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":304,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":304,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":304,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":304,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":304,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":304,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-14T15:20:02.524 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:20:02.547 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:02.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:02.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:20:02.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:02 smithi092 ceph-mon[33966]: 2.1c deep-scrub starts 2024-04-14T15:20:02.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:02 smithi092 ceph-mon[33966]: 2.1c deep-scrub ok 2024-04-14T15:20:02.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:02 smithi092 ceph-mon[33966]: 110.11 scrub starts 2024-04-14T15:20:02.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:02 smithi092 ceph-mon[33966]: 110.11 scrub ok 2024-04-14T15:20:02.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:02 smithi092 ceph-mon[33966]: 5.6 scrub starts 2024-04-14T15:20:02.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:02 smithi092 ceph-mon[33966]: 5.6 scrub ok 2024-04-14T15:20:02.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:02 smithi092 ceph-mon[33966]: 6.1 scrub starts 2024-04-14T15:20:02.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:02 smithi092 ceph-mon[33966]: 6.1 scrub ok 2024-04-14T15:20:02.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:02 smithi092 ceph-mon[33966]: pgmap v703: 226 pgs: 226 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-14T15:20:02.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:02 smithi092 ceph-mon[33966]: osdmap e491: 8 total, 8 up, 8 in 2024-04-14T15:20:02.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:02 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:20:02.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:02 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:20:02.649 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:20:02.673 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:02.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:02.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:20:02.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.698+0000 7fb6e430f700 1 -- 172.21.15.90:0/1813649877 >> v1:172.21.15.90:6789/0 conn(0x7fb6dc10a610 legacy=0x7fb6dc10ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:02.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.698+0000 7fb6e430f700 1 -- 172.21.15.90:0/1813649877 shutdown_connections 2024-04-14T15:20:02.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.698+0000 7fb6e430f700 1 -- 172.21.15.90:0/1813649877 >> 172.21.15.90:0/1813649877 conn(0x7fb6dc0fd290 msgr2=0x7fb6dc0ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:20:02.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.698+0000 7fb6e430f700 1 -- 172.21.15.90:0/1813649877 shutdown_connections 2024-04-14T15:20:02.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.698+0000 7fb6e430f700 1 -- 172.21.15.90:0/1813649877 wait complete. 2024-04-14T15:20:02.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.698+0000 7fb6e430f700 1 librados: starting msgr at 2024-04-14T15:20:02.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.698+0000 7fb6e430f700 1 librados: starting objecter 2024-04-14T15:20:02.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.699+0000 7fb6e430f700 1 Processor -- start 2024-04-14T15:20:02.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.699+0000 7fb6e430f700 1 -- start start 2024-04-14T15:20:02.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.699+0000 7fb6e430f700 1 librados: setting wanted keys 2024-04-14T15:20:02.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.699+0000 7fb6e430f700 1 librados: calling monclient init 2024-04-14T15:20:02.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.699+0000 7fb6e430f700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb6dc1a7df0 con 0x7fb6dc107760 2024-04-14T15:20:02.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.699+0000 7fb6e430f700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb6dc1a8fd0 con 0x7fb6dc10e1e0 2024-04-14T15:20:02.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.699+0000 7fb6e430f700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb6dc1aa1b0 con 0x7fb6dc10a610 2024-04-14T15:20:02.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.699+0000 7fb6e18aa700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7fb6dc10a610 0x7fb6dc10c510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:58808/0 (socket says 172.21.15.90:58808) 2024-04-14T15:20:02.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.699+0000 7fb6e18aa700 1 -- 172.21.15.90:0/318593675 learned_addr learned my addr 172.21.15.90:0/318593675 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:20:02.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.699+0000 7fb6db7fe700 10 client.?.objecter ms_handle_connect 0x7fb6dc10a610 2024-04-14T15:20:02.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.699+0000 7fb6db7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:20:02.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.699+0000 7fb6db7fe700 1 -- 172.21.15.90:0/318593675 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 64860928 0 0) 0x7fb6dc1aa1b0 con 0x7fb6dc10a610 2024-04-14T15:20:02.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.699+0000 7fb6db7fe700 1 -- 172.21.15.90:0/318593675 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6c40035b0 con 0x7fb6dc10a610 2024-04-14T15:20:02.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.699+0000 7fb6db7fe700 10 client.?.objecter ms_handle_connect 0x7fb6dc107760 2024-04-14T15:20:02.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.699+0000 7fb6db7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:20:02.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.699+0000 7fb6db7fe700 10 client.?.objecter ms_handle_connect 0x7fb6dc10e1e0 2024-04-14T15:20:02.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.699+0000 7fb6db7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:20:02.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.699+0000 7fb6db7fe700 1 -- 172.21.15.90:0/318593675 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3957734895 0 0) 0x7fb6dc1a7df0 con 0x7fb6dc107760 2024-04-14T15:20:02.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.699+0000 7fb6db7fe700 1 -- 172.21.15.90:0/318593675 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6dc1aa1b0 con 0x7fb6dc107760 2024-04-14T15:20:02.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.699+0000 7fb6db7fe700 1 -- 172.21.15.90:0/318593675 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 131361911 0 0) 0x7fb6dc1a8fd0 con 0x7fb6dc10e1e0 2024-04-14T15:20:02.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.699+0000 7fb6db7fe700 1 -- 172.21.15.90:0/318593675 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6dc1a7df0 con 0x7fb6dc10e1e0 2024-04-14T15:20:02.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.699+0000 7fb6db7fe700 1 -- 172.21.15.90:0/318593675 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 114738379 0 0) 0x7fb6c40035b0 con 0x7fb6dc10a610 2024-04-14T15:20:02.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.699+0000 7fb6db7fe700 1 -- 172.21.15.90:0/318593675 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb6dc1a8fd0 con 0x7fb6dc10a610 2024-04-14T15:20:02.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.699+0000 7fb6db7fe700 1 -- 172.21.15.90:0/318593675 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fb6d00015d0 con 0x7fb6dc10a610 2024-04-14T15:20:02.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.700+0000 7fb6db7fe700 1 -- 172.21.15.90:0/318593675 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3271324133 0 0) 0x7fb6dc1aa1b0 con 0x7fb6dc107760 2024-04-14T15:20:02.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.700+0000 7fb6db7fe700 1 -- 172.21.15.90:0/318593675 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb6c40035b0 con 0x7fb6dc107760 2024-04-14T15:20:02.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.700+0000 7fb6db7fe700 1 -- 172.21.15.90:0/318593675 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fb6d40018b0 con 0x7fb6dc107760 2024-04-14T15:20:02.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.700+0000 7fb6db7fe700 1 -- 172.21.15.90:0/318593675 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1228037780 0 0) 0x7fb6dc1a7df0 con 0x7fb6dc10e1e0 2024-04-14T15:20:02.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.700+0000 7fb6db7fe700 1 -- 172.21.15.90:0/318593675 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb6dc1aa1b0 con 0x7fb6dc10e1e0 2024-04-14T15:20:02.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.700+0000 7fb6db7fe700 1 -- 172.21.15.90:0/318593675 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fb6cc001bb0 con 0x7fb6dc10e1e0 2024-04-14T15:20:02.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.700+0000 7fb6db7fe700 1 -- 172.21.15.90:0/318593675 <== mon.2 v1:172.21.15.90:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2285003875 0 0) 0x7fb6dc1a8fd0 con 0x7fb6dc10a610 2024-04-14T15:20:02.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.700+0000 7fb6db7fe700 1 -- 172.21.15.90:0/318593675 >> v1:172.21.15.92:6789/0 conn(0x7fb6dc10e1e0 legacy=0x7fb6dc1a66d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:02.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.700+0000 7fb6db7fe700 1 -- 172.21.15.90:0/318593675 >> v1:172.21.15.90:6789/0 conn(0x7fb6dc107760 legacy=0x7fb6dc10be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:02.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.700+0000 7fb6db7fe700 1 -- 172.21.15.90:0/318593675 --> v1:172.21.15.90:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6dc1ab390 con 0x7fb6dc10a610 2024-04-14T15:20:02.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.700+0000 7fb6e430f700 1 -- 172.21.15.90:0/318593675 --> v1:172.21.15.90:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb6dc1a8020 con 0x7fb6dc10a610 2024-04-14T15:20:02.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.700+0000 7fb6e430f700 10 client.47396.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:20:02.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.700+0000 7fb6e430f700 1 -- 172.21.15.90:0/318593675 --> v1:172.21.15.90:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fb6dc1a84e0 con 0x7fb6dc10a610 2024-04-14T15:20:02.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.700+0000 7fb6e430f700 1 librados: init done 2024-04-14T15:20:02.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.700+0000 7fb6db7fe700 1 -- 172.21.15.90:0/318593675 <== mon.2 v1:172.21.15.90:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fb6d0003680 con 0x7fb6dc10a610 2024-04-14T15:20:02.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.700+0000 7fb6db7fe700 1 -- 172.21.15.90:0/318593675 <== mon.2 v1:172.21.15.90:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fb6d0004e60 con 0x7fb6dc10a610 2024-04-14T15:20:02.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.701+0000 7fb6e430f700 1 -- 172.21.15.90:0/318593675 --> v1:172.21.15.90:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fb6c8005100 con 0x7fb6dc10a610 2024-04-14T15:20:02.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.701+0000 7fb6db7fe700 1 -- 172.21.15.90:0/318593675 <== mon.2 v1:172.21.15.90:6790/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fb6d0004e60 con 0x7fb6dc10a610 2024-04-14T15:20:02.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.702+0000 7fb6db7fe700 1 -- 172.21.15.90:0/318593675 <== mon.2 v1:172.21.15.90:6790/0 8 ==== osd_map(491..491 src has 51..491) ==== 8184+0+0 (unknown 3601408288 0 0) 0x7fb6d008a9e0 con 0x7fb6dc10a610 2024-04-14T15:20:02.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.702+0000 7fb6db7fe700 10 client.47396.objecter ms_dispatch 0x7fb6dc001550 osd_map(491..491 src has 51..491) 2024-04-14T15:20:02.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.702+0000 7fb6db7fe700 3 client.47396.objecter handle_osd_map got epochs [491,491] > 0 2024-04-14T15:20:02.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.702+0000 7fb6db7fe700 3 client.47396.objecter handle_osd_map decoding full epoch 491 2024-04-14T15:20:02.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.702+0000 7fb6db7fe700 20 client.47396.objecter dump_active .. 0 homeless 2024-04-14T15:20:02.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.702+0000 7fb6db7fe700 1 -- 172.21.15.90:0/318593675 <== mon.2 v1:172.21.15.90:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v491) ==== 54+0+16 (unknown 1958926529 0 2420000192) 0x7fb6d008ae90 con 0x7fb6dc10a610 2024-04-14T15:20:02.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.702+0000 7fb6db7fe700 10 client.47396.objecter ms_handle_connect 0x7fb6c4071300 2024-04-14T15:20:02.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.702+0000 7fb6e430f700 10 client.47396.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:20:02.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.702+0000 7fb6e430f700 15 client.47396.objecter _session_command_op_assign -1 1 2024-04-14T15:20:02.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.702+0000 7fb6e430f700 20 client.47396.objecter _get_session s=0x7fb6c80053e0 osd=0 3 2024-04-14T15:20:02.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.702+0000 7fb6e430f700 20 client.47396.objecter put_session s=0x7fb6c80053e0 osd=0 3 2024-04-14T15:20:02.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.702+0000 7fb6e430f700 20 client.47396.objecter _get_session s=0x7fb6c80053e0 osd=0 3 2024-04-14T15:20:02.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.702+0000 7fb6e430f700 15 client.47396.objecter _session_command_op_remove -1 1 2024-04-14T15:20:02.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.702+0000 7fb6e430f700 20 client.47396.objecter get_session s=0x7fb6c80053e0 osd=0 3 2024-04-14T15:20:02.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.702+0000 7fb6e430f700 15 client.47396.objecter _session_command_op_assign 0 1 2024-04-14T15:20:02.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.702+0000 7fb6e430f700 20 client.47396.objecter put_session s=0x7fb6c80053e0 osd=0 4 2024-04-14T15:20:02.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.702+0000 7fb6e430f700 10 client.47396.objecter _send_command 1 2024-04-14T15:20:02.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.702+0000 7fb6e430f700 1 -- 172.21.15.90:0/318593675 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb6c8004b70 con 0x7fb6c8004730 2024-04-14T15:20:02.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.702+0000 7fb6db7fe700 10 client.47396.objecter ms_handle_connect 0x7fb6c8004730 2024-04-14T15:20:02.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.703+0000 7fb6db7fe700 1 -- 172.21.15.90:0/318593675 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fb6c8004b70 con 0x7fb6c8004730 2024-04-14T15:20:02.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.703+0000 7fb6db7fe700 10 client.47396.objecter ms_dispatch 0x7fb6dc001550 command_reply(tid 1: 0 ) 2024-04-14T15:20:02.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.703+0000 7fb6db7fe700 10 client.47396.objecter _finish_command 1 = system:0 2024-04-14T15:20:02.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.703+0000 7fb6db7fe700 20 client.47396.objecter put_session s=0x7fb6c80053e0 osd=0 4 2024-04-14T15:20:02.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.703+0000 7fb6db7fe700 15 client.47396.objecter _session_command_op_remove 0 1 2024-04-14T15:20:02.720 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:20:02.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.720+0000 7fb6e430f700 10 client.47396.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:20:02.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.720+0000 7fb6e430f700 15 client.47396.objecter _session_command_op_assign -1 2 2024-04-14T15:20:02.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.720+0000 7fb6e430f700 20 client.47396.objecter _get_session s=0x7fb6c80053e0 osd=0 3 2024-04-14T15:20:02.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.720+0000 7fb6e430f700 20 client.47396.objecter put_session s=0x7fb6c80053e0 osd=0 3 2024-04-14T15:20:02.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.720+0000 7fb6e430f700 20 client.47396.objecter _get_session s=0x7fb6c80053e0 osd=0 3 2024-04-14T15:20:02.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.720+0000 7fb6e430f700 15 client.47396.objecter _session_command_op_remove -1 2 2024-04-14T15:20:02.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.720+0000 7fb6e430f700 20 client.47396.objecter get_session s=0x7fb6c80053e0 osd=0 3 2024-04-14T15:20:02.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.720+0000 7fb6e430f700 15 client.47396.objecter _session_command_op_assign 0 2 2024-04-14T15:20:02.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.720+0000 7fb6e430f700 20 client.47396.objecter put_session s=0x7fb6c80053e0 osd=0 4 2024-04-14T15:20:02.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.720+0000 7fb6e430f700 10 client.47396.objecter _send_command 2 2024-04-14T15:20:02.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.720+0000 7fb6e430f700 1 -- 172.21.15.90:0/318593675 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fb6c800be00 con 0x7fb6c8004730 2024-04-14T15:20:02.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.722+0000 7fb6db7fe700 1 -- 172.21.15.90:0/318593675 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fb6c800be00 con 0x7fb6c8004730 2024-04-14T15:20:02.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.722+0000 7fb6db7fe700 10 client.47396.objecter ms_dispatch 0x7fb6dc001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:20:02.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.722+0000 7fb6db7fe700 10 client.47396.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:20:02.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.722+0000 7fb6db7fe700 20 client.47396.objecter put_session s=0x7fb6c80053e0 osd=0 4 2024-04-14T15:20:02.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.722+0000 7fb6db7fe700 15 client.47396.objecter _session_command_op_remove 0 2 2024-04-14T15:20:02.723 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:20:02.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.722+0000 7fb6e430f700 10 client.47396.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:20:02.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.722+0000 7fb6e430f700 15 client.47396.objecter _session_command_op_assign -1 3 2024-04-14T15:20:02.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.722+0000 7fb6e430f700 20 client.47396.objecter _get_session s=0x7fb6c8004130 osd=1 3 2024-04-14T15:20:02.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.722+0000 7fb6e430f700 20 client.47396.objecter put_session s=0x7fb6c8004130 osd=1 3 2024-04-14T15:20:02.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.722+0000 7fb6e430f700 20 client.47396.objecter _get_session s=0x7fb6c8004130 osd=1 3 2024-04-14T15:20:02.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.722+0000 7fb6e430f700 15 client.47396.objecter _session_command_op_remove -1 3 2024-04-14T15:20:02.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.722+0000 7fb6e430f700 20 client.47396.objecter get_session s=0x7fb6c8004130 osd=1 3 2024-04-14T15:20:02.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.722+0000 7fb6e430f700 15 client.47396.objecter _session_command_op_assign 1 3 2024-04-14T15:20:02.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.722+0000 7fb6e430f700 20 client.47396.objecter put_session s=0x7fb6c8004130 osd=1 4 2024-04-14T15:20:02.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.722+0000 7fb6e430f700 10 client.47396.objecter _send_command 3 2024-04-14T15:20:02.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.722+0000 7fb6e430f700 1 -- 172.21.15.90:0/318593675 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fb6c80022a0 con 0x7fb6c8004310 2024-04-14T15:20:02.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.722+0000 7fb6db7fe700 10 client.47396.objecter ms_handle_connect 0x7fb6c8004310 2024-04-14T15:20:02.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.723+0000 7fb6db7fe700 1 -- 172.21.15.90:0/318593675 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fb6c80022a0 con 0x7fb6c8004310 2024-04-14T15:20:02.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.723+0000 7fb6db7fe700 10 client.47396.objecter ms_dispatch 0x7fb6dc001550 command_reply(tid 3: 0 ) 2024-04-14T15:20:02.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.723+0000 7fb6db7fe700 10 client.47396.objecter _finish_command 3 = system:0 2024-04-14T15:20:02.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.723+0000 7fb6db7fe700 20 client.47396.objecter put_session s=0x7fb6c8004130 osd=1 4 2024-04-14T15:20:02.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.723+0000 7fb6db7fe700 15 client.47396.objecter _session_command_op_remove 1 3 2024-04-14T15:20:02.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.743+0000 7fb6e430f700 10 client.47396.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:20:02.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.743+0000 7fb6e430f700 15 client.47396.objecter _session_command_op_assign -1 4 2024-04-14T15:20:02.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.743+0000 7fb6e430f700 20 client.47396.objecter _get_session s=0x7fb6c8004130 osd=1 3 2024-04-14T15:20:02.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.743+0000 7fb6e430f700 20 client.47396.objecter put_session s=0x7fb6c8004130 osd=1 3 2024-04-14T15:20:02.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.743+0000 7fb6e430f700 20 client.47396.objecter _get_session s=0x7fb6c8004130 osd=1 3 2024-04-14T15:20:02.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.743+0000 7fb6e430f700 15 client.47396.objecter _session_command_op_remove -1 4 2024-04-14T15:20:02.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.743+0000 7fb6e430f700 20 client.47396.objecter get_session s=0x7fb6c8004130 osd=1 3 2024-04-14T15:20:02.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.743+0000 7fb6e430f700 15 client.47396.objecter _session_command_op_assign 1 4 2024-04-14T15:20:02.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.743+0000 7fb6e430f700 20 client.47396.objecter put_session s=0x7fb6c8004130 osd=1 4 2024-04-14T15:20:02.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.743+0000 7fb6e430f700 10 client.47396.objecter _send_command 4 2024-04-14T15:20:02.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.743+0000 7fb6e430f700 1 -- 172.21.15.90:0/318593675 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fb6c8001720 con 0x7fb6c8004310 2024-04-14T15:20:02.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.745+0000 7fb6db7fe700 1 -- 172.21.15.90:0/318593675 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fb6c8001720 con 0x7fb6c8004310 2024-04-14T15:20:02.745 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:20:02.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.745+0000 7fb6db7fe700 10 client.47396.objecter ms_dispatch 0x7fb6dc001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:20:02.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.745+0000 7fb6db7fe700 10 client.47396.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:20:02.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.745+0000 7fb6db7fe700 20 client.47396.objecter put_session s=0x7fb6c8004130 osd=1 4 2024-04-14T15:20:02.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.745+0000 7fb6db7fe700 15 client.47396.objecter _session_command_op_remove 1 4 2024-04-14T15:20:02.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.745+0000 7fb6e430f700 10 client.47396.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:20:02.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.745+0000 7fb6e430f700 15 client.47396.objecter _session_command_op_assign -1 5 2024-04-14T15:20:02.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.745+0000 7fb6e430f700 20 client.47396.objecter _get_session s=0x7fb6c8001980 osd=2 3 2024-04-14T15:20:02.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.745+0000 7fb6e430f700 20 client.47396.objecter put_session s=0x7fb6c8001980 osd=2 3 2024-04-14T15:20:02.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.745+0000 7fb6e430f700 20 client.47396.objecter _get_session s=0x7fb6c8001980 osd=2 3 2024-04-14T15:20:02.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.745+0000 7fb6e430f700 15 client.47396.objecter _session_command_op_remove -1 5 2024-04-14T15:20:02.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.745+0000 7fb6e430f700 20 client.47396.objecter get_session s=0x7fb6c8001980 osd=2 3 2024-04-14T15:20:02.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.745+0000 7fb6e430f700 15 client.47396.objecter _session_command_op_assign 2 5 2024-04-14T15:20:02.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.745+0000 7fb6e430f700 20 client.47396.objecter put_session s=0x7fb6c8001980 osd=2 4 2024-04-14T15:20:02.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.745+0000 7fb6e430f700 10 client.47396.objecter _send_command 5 2024-04-14T15:20:02.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.745+0000 7fb6e430f700 1 -- 172.21.15.90:0/318593675 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fb6c8015580 con 0x7fb6c80119e0 2024-04-14T15:20:02.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.746+0000 7fb6db7fe700 10 client.47396.objecter ms_handle_connect 0x7fb6c80119e0 2024-04-14T15:20:02.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.747+0000 7fb6db7fe700 1 -- 172.21.15.90:0/318593675 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fb6c8015580 con 0x7fb6c80119e0 2024-04-14T15:20:02.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.747+0000 7fb6db7fe700 10 client.47396.objecter ms_dispatch 0x7fb6dc001550 command_reply(tid 5: 0 ) 2024-04-14T15:20:02.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.747+0000 7fb6db7fe700 10 client.47396.objecter _finish_command 5 = system:0 2024-04-14T15:20:02.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.747+0000 7fb6db7fe700 20 client.47396.objecter put_session s=0x7fb6c8001980 osd=2 4 2024-04-14T15:20:02.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.747+0000 7fb6db7fe700 15 client.47396.objecter _session_command_op_remove 2 5 2024-04-14T15:20:02.764 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:20:02.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.764+0000 7fb6e430f700 10 client.47396.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:20:02.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.764+0000 7fb6e430f700 15 client.47396.objecter _session_command_op_assign -1 6 2024-04-14T15:20:02.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.764+0000 7fb6e430f700 20 client.47396.objecter _get_session s=0x7fb6c8001980 osd=2 3 2024-04-14T15:20:02.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.764+0000 7fb6e430f700 20 client.47396.objecter put_session s=0x7fb6c8001980 osd=2 3 2024-04-14T15:20:02.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.764+0000 7fb6e430f700 20 client.47396.objecter _get_session s=0x7fb6c8001980 osd=2 3 2024-04-14T15:20:02.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.764+0000 7fb6e430f700 15 client.47396.objecter _session_command_op_remove -1 6 2024-04-14T15:20:02.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.764+0000 7fb6e430f700 20 client.47396.objecter get_session s=0x7fb6c8001980 osd=2 3 2024-04-14T15:20:02.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.764+0000 7fb6e430f700 15 client.47396.objecter _session_command_op_assign 2 6 2024-04-14T15:20:02.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.764+0000 7fb6e430f700 20 client.47396.objecter put_session s=0x7fb6c8001980 osd=2 4 2024-04-14T15:20:02.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.764+0000 7fb6e430f700 10 client.47396.objecter _send_command 6 2024-04-14T15:20:02.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.764+0000 7fb6e430f700 1 -- 172.21.15.90:0/318593675 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fb6c800c510 con 0x7fb6c80119e0 2024-04-14T15:20:02.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.766+0000 7fb6db7fe700 1 -- 172.21.15.90:0/318593675 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fb6c800c510 con 0x7fb6c80119e0 2024-04-14T15:20:02.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.766+0000 7fb6db7fe700 10 client.47396.objecter ms_dispatch 0x7fb6dc001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:20:02.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.766+0000 7fb6db7fe700 10 client.47396.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:20:02.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.766+0000 7fb6db7fe700 20 client.47396.objecter put_session s=0x7fb6c8001980 osd=2 4 2024-04-14T15:20:02.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.766+0000 7fb6db7fe700 15 client.47396.objecter _session_command_op_remove 2 6 2024-04-14T15:20:02.767 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:20:02.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.767+0000 7fb6e430f700 10 client.47396.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:20:02.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.767+0000 7fb6e430f700 15 client.47396.objecter _session_command_op_assign -1 7 2024-04-14T15:20:02.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.767+0000 7fb6e430f700 20 client.47396.objecter _get_session s=0x7fb6c800c7a0 osd=3 3 2024-04-14T15:20:02.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.767+0000 7fb6e430f700 20 client.47396.objecter put_session s=0x7fb6c800c7a0 osd=3 3 2024-04-14T15:20:02.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.767+0000 7fb6e430f700 20 client.47396.objecter _get_session s=0x7fb6c800c7a0 osd=3 3 2024-04-14T15:20:02.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.767+0000 7fb6e430f700 15 client.47396.objecter _session_command_op_remove -1 7 2024-04-14T15:20:02.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.767+0000 7fb6e430f700 20 client.47396.objecter get_session s=0x7fb6c800c7a0 osd=3 3 2024-04-14T15:20:02.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.767+0000 7fb6e430f700 15 client.47396.objecter _session_command_op_assign 3 7 2024-04-14T15:20:02.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.767+0000 7fb6e430f700 20 client.47396.objecter put_session s=0x7fb6c800c7a0 osd=3 4 2024-04-14T15:20:02.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.767+0000 7fb6e430f700 10 client.47396.objecter _send_command 7 2024-04-14T15:20:02.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.767+0000 7fb6e430f700 1 -- 172.21.15.90:0/318593675 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fb6c801a3c0 con 0x7fb6c80168b0 2024-04-14T15:20:02.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.767+0000 7fb6db7fe700 10 client.47396.objecter ms_handle_connect 0x7fb6c80168b0 2024-04-14T15:20:02.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.768+0000 7fb6db7fe700 1 -- 172.21.15.90:0/318593675 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fb6c801a3c0 con 0x7fb6c80168b0 2024-04-14T15:20:02.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.768+0000 7fb6db7fe700 10 client.47396.objecter ms_dispatch 0x7fb6dc001550 command_reply(tid 7: 0 ) 2024-04-14T15:20:02.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.768+0000 7fb6db7fe700 10 client.47396.objecter _finish_command 7 = system:0 2024-04-14T15:20:02.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.768+0000 7fb6db7fe700 20 client.47396.objecter put_session s=0x7fb6c800c7a0 osd=3 4 2024-04-14T15:20:02.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.768+0000 7fb6db7fe700 15 client.47396.objecter _session_command_op_remove 3 7 2024-04-14T15:20:02.775 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:20:02.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.786+0000 7fb6e430f700 10 client.47396.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:20:02.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.786+0000 7fb6e430f700 15 client.47396.objecter _session_command_op_assign -1 8 2024-04-14T15:20:02.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.786+0000 7fb6e430f700 20 client.47396.objecter _get_session s=0x7fb6c800c7a0 osd=3 3 2024-04-14T15:20:02.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.786+0000 7fb6e430f700 20 client.47396.objecter put_session s=0x7fb6c800c7a0 osd=3 3 2024-04-14T15:20:02.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.786+0000 7fb6e430f700 20 client.47396.objecter _get_session s=0x7fb6c800c7a0 osd=3 3 2024-04-14T15:20:02.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.786+0000 7fb6e430f700 15 client.47396.objecter _session_command_op_remove -1 8 2024-04-14T15:20:02.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.786+0000 7fb6e430f700 20 client.47396.objecter get_session s=0x7fb6c800c7a0 osd=3 3 2024-04-14T15:20:02.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.786+0000 7fb6e430f700 15 client.47396.objecter _session_command_op_assign 3 8 2024-04-14T15:20:02.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.786+0000 7fb6e430f700 20 client.47396.objecter put_session s=0x7fb6c800c7a0 osd=3 4 2024-04-14T15:20:02.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.786+0000 7fb6e430f700 10 client.47396.objecter _send_command 8 2024-04-14T15:20:02.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.786+0000 7fb6e430f700 1 -- 172.21.15.90:0/318593675 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fb6c8010dc0 con 0x7fb6c80168b0 2024-04-14T15:20:02.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.788+0000 7fb6db7fe700 1 -- 172.21.15.90:0/318593675 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fb6c8010dc0 con 0x7fb6c80168b0 2024-04-14T15:20:02.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.788+0000 7fb6db7fe700 10 client.47396.objecter ms_dispatch 0x7fb6dc001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:20:02.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.788+0000 7fb6db7fe700 10 client.47396.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:20:02.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.788+0000 7fb6db7fe700 20 client.47396.objecter put_session s=0x7fb6c800c7a0 osd=3 4 2024-04-14T15:20:02.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.788+0000 7fb6db7fe700 15 client.47396.objecter _session_command_op_remove 3 8 2024-04-14T15:20:02.789 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:20:02.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.788+0000 7fb6e430f700 10 client.47396.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:20:02.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.788+0000 7fb6e430f700 15 client.47396.objecter _session_command_op_assign -1 9 2024-04-14T15:20:02.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.788+0000 7fb6e430f700 20 client.47396.objecter _get_session s=0x7fb6c8010dc0 osd=4 3 2024-04-14T15:20:02.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.788+0000 7fb6e430f700 20 client.47396.objecter put_session s=0x7fb6c8010dc0 osd=4 3 2024-04-14T15:20:02.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.788+0000 7fb6e430f700 20 client.47396.objecter _get_session s=0x7fb6c8010dc0 osd=4 3 2024-04-14T15:20:02.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.788+0000 7fb6e430f700 15 client.47396.objecter _session_command_op_remove -1 9 2024-04-14T15:20:02.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.788+0000 7fb6e430f700 20 client.47396.objecter get_session s=0x7fb6c8010dc0 osd=4 3 2024-04-14T15:20:02.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.788+0000 7fb6e430f700 15 client.47396.objecter _session_command_op_assign 4 9 2024-04-14T15:20:02.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.788+0000 7fb6e430f700 20 client.47396.objecter put_session s=0x7fb6c8010dc0 osd=4 4 2024-04-14T15:20:02.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.788+0000 7fb6e430f700 10 client.47396.objecter _send_command 9 2024-04-14T15:20:02.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.788+0000 7fb6e430f700 1 -- 172.21.15.90:0/318593675 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fb6c8011090 con 0x7fb6c8016410 2024-04-14T15:20:02.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.789+0000 7fb6db7fe700 10 client.47396.objecter ms_handle_connect 0x7fb6c8016410 2024-04-14T15:20:02.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.790+0000 7fb6db7fe700 1 -- 172.21.15.90:0/318593675 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fb6c8011090 con 0x7fb6c8016410 2024-04-14T15:20:02.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.790+0000 7fb6db7fe700 10 client.47396.objecter ms_dispatch 0x7fb6dc001550 command_reply(tid 9: 0 ) 2024-04-14T15:20:02.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.790+0000 7fb6db7fe700 10 client.47396.objecter _finish_command 9 = system:0 2024-04-14T15:20:02.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.790+0000 7fb6db7fe700 20 client.47396.objecter put_session s=0x7fb6c8010dc0 osd=4 4 2024-04-14T15:20:02.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.790+0000 7fb6db7fe700 15 client.47396.objecter _session_command_op_remove 4 9 2024-04-14T15:20:02.801 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:02.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:02.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:20:02.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.810+0000 7fb6e430f700 10 client.47396.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:20:02.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.810+0000 7fb6e430f700 15 client.47396.objecter _session_command_op_assign -1 10 2024-04-14T15:20:02.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.810+0000 7fb6e430f700 20 client.47396.objecter _get_session s=0x7fb6c8010dc0 osd=4 3 2024-04-14T15:20:02.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.810+0000 7fb6e430f700 20 client.47396.objecter put_session s=0x7fb6c8010dc0 osd=4 3 2024-04-14T15:20:02.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.810+0000 7fb6e430f700 20 client.47396.objecter _get_session s=0x7fb6c8010dc0 osd=4 3 2024-04-14T15:20:02.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.810+0000 7fb6e430f700 15 client.47396.objecter _session_command_op_remove -1 10 2024-04-14T15:20:02.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.810+0000 7fb6e430f700 20 client.47396.objecter get_session s=0x7fb6c8010dc0 osd=4 3 2024-04-14T15:20:02.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.810+0000 7fb6e430f700 15 client.47396.objecter _session_command_op_assign 4 10 2024-04-14T15:20:02.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.810+0000 7fb6e430f700 20 client.47396.objecter put_session s=0x7fb6c8010dc0 osd=4 4 2024-04-14T15:20:02.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.810+0000 7fb6e430f700 10 client.47396.objecter _send_command 10 2024-04-14T15:20:02.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.810+0000 7fb6e430f700 1 -- 172.21.15.90:0/318593675 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fb6c80205f0 con 0x7fb6c8016410 2024-04-14T15:20:02.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.812+0000 7fb6db7fe700 1 -- 172.21.15.90:0/318593675 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fb6c80205f0 con 0x7fb6c8016410 2024-04-14T15:20:02.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.812+0000 7fb6db7fe700 10 client.47396.objecter ms_dispatch 0x7fb6dc001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:20:02.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.812+0000 7fb6db7fe700 10 client.47396.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:20:02.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.812+0000 7fb6db7fe700 20 client.47396.objecter put_session s=0x7fb6c8010dc0 osd=4 4 2024-04-14T15:20:02.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.812+0000 7fb6db7fe700 15 client.47396.objecter _session_command_op_remove 4 10 2024-04-14T15:20:02.813 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:20:02.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.812+0000 7fb6e430f700 10 client.47396.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:20:02.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.812+0000 7fb6e430f700 15 client.47396.objecter _session_command_op_assign -1 11 2024-04-14T15:20:02.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.812+0000 7fb6e430f700 20 client.47396.objecter _get_session s=0x7fb6c80205f0 osd=5 3 2024-04-14T15:20:02.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.812+0000 7fb6e430f700 20 client.47396.objecter put_session s=0x7fb6c80205f0 osd=5 3 2024-04-14T15:20:02.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.812+0000 7fb6e430f700 20 client.47396.objecter _get_session s=0x7fb6c80205f0 osd=5 3 2024-04-14T15:20:02.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.812+0000 7fb6e430f700 15 client.47396.objecter _session_command_op_remove -1 11 2024-04-14T15:20:02.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.812+0000 7fb6e430f700 20 client.47396.objecter get_session s=0x7fb6c80205f0 osd=5 3 2024-04-14T15:20:02.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.812+0000 7fb6e430f700 15 client.47396.objecter _session_command_op_assign 5 11 2024-04-14T15:20:02.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.812+0000 7fb6e430f700 20 client.47396.objecter put_session s=0x7fb6c80205f0 osd=5 4 2024-04-14T15:20:02.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.812+0000 7fb6e430f700 10 client.47396.objecter _send_command 11 2024-04-14T15:20:02.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.812+0000 7fb6e430f700 1 -- 172.21.15.90:0/318593675 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fb6c8021170 con 0x7fb6c8020d10 2024-04-14T15:20:02.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.813+0000 7fb6db7fe700 10 client.47396.objecter ms_handle_connect 0x7fb6c8020d10 2024-04-14T15:20:02.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.813+0000 7fb6db7fe700 1 -- 172.21.15.90:0/318593675 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fb6c8021170 con 0x7fb6c8020d10 2024-04-14T15:20:02.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.813+0000 7fb6db7fe700 10 client.47396.objecter ms_dispatch 0x7fb6dc001550 command_reply(tid 11: 0 ) 2024-04-14T15:20:02.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.813+0000 7fb6db7fe700 10 client.47396.objecter _finish_command 11 = system:0 2024-04-14T15:20:02.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.813+0000 7fb6db7fe700 20 client.47396.objecter put_session s=0x7fb6c80205f0 osd=5 4 2024-04-14T15:20:02.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.813+0000 7fb6db7fe700 15 client.47396.objecter _session_command_op_remove 5 11 2024-04-14T15:20:02.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.829+0000 7fb6e430f700 10 client.47396.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:20:02.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.829+0000 7fb6e430f700 15 client.47396.objecter _session_command_op_assign -1 12 2024-04-14T15:20:02.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.829+0000 7fb6e430f700 20 client.47396.objecter _get_session s=0x7fb6c80205f0 osd=5 3 2024-04-14T15:20:02.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.829+0000 7fb6e430f700 20 client.47396.objecter put_session s=0x7fb6c80205f0 osd=5 3 2024-04-14T15:20:02.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.829+0000 7fb6e430f700 20 client.47396.objecter _get_session s=0x7fb6c80205f0 osd=5 3 2024-04-14T15:20:02.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.829+0000 7fb6e430f700 15 client.47396.objecter _session_command_op_remove -1 12 2024-04-14T15:20:02.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.829+0000 7fb6e430f700 20 client.47396.objecter get_session s=0x7fb6c80205f0 osd=5 3 2024-04-14T15:20:02.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.829+0000 7fb6e430f700 15 client.47396.objecter _session_command_op_assign 5 12 2024-04-14T15:20:02.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.829+0000 7fb6e430f700 20 client.47396.objecter put_session s=0x7fb6c80205f0 osd=5 4 2024-04-14T15:20:02.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.829+0000 7fb6e430f700 10 client.47396.objecter _send_command 12 2024-04-14T15:20:02.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.829+0000 7fb6e430f700 1 -- 172.21.15.90:0/318593675 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fb6c80213a0 con 0x7fb6c8020d10 2024-04-14T15:20:02.832 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:20:02.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.831+0000 7fb6db7fe700 1 -- 172.21.15.90:0/318593675 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fb6c80213a0 con 0x7fb6c8020d10 2024-04-14T15:20:02.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.831+0000 7fb6db7fe700 10 client.47396.objecter ms_dispatch 0x7fb6dc001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:20:02.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.831+0000 7fb6db7fe700 10 client.47396.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:20:02.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.831+0000 7fb6db7fe700 20 client.47396.objecter put_session s=0x7fb6c80205f0 osd=5 4 2024-04-14T15:20:02.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.831+0000 7fb6db7fe700 15 client.47396.objecter _session_command_op_remove 5 12 2024-04-14T15:20:02.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.832+0000 7fb6e430f700 10 client.47396.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:20:02.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.832+0000 7fb6e430f700 15 client.47396.objecter _session_command_op_assign -1 13 2024-04-14T15:20:02.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.832+0000 7fb6e430f700 20 client.47396.objecter _get_session s=0x7fb6c8021170 osd=6 3 2024-04-14T15:20:02.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.832+0000 7fb6e430f700 20 client.47396.objecter put_session s=0x7fb6c8021170 osd=6 3 2024-04-14T15:20:02.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.832+0000 7fb6e430f700 20 client.47396.objecter _get_session s=0x7fb6c8021170 osd=6 3 2024-04-14T15:20:02.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.832+0000 7fb6e430f700 15 client.47396.objecter _session_command_op_remove -1 13 2024-04-14T15:20:02.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.832+0000 7fb6e430f700 20 client.47396.objecter get_session s=0x7fb6c8021170 osd=6 3 2024-04-14T15:20:02.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.832+0000 7fb6e430f700 15 client.47396.objecter _session_command_op_assign 6 13 2024-04-14T15:20:02.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.832+0000 7fb6e430f700 20 client.47396.objecter put_session s=0x7fb6c8021170 osd=6 4 2024-04-14T15:20:02.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.832+0000 7fb6e430f700 10 client.47396.objecter _send_command 13 2024-04-14T15:20:02.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.832+0000 7fb6e430f700 1 -- 172.21.15.90:0/318593675 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fb6c8027660 con 0x7fb6c80213a0 2024-04-14T15:20:02.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.832+0000 7fb6db7fe700 10 client.47396.objecter ms_handle_connect 0x7fb6c80213a0 2024-04-14T15:20:02.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.833+0000 7fb6db7fe700 1 -- 172.21.15.90:0/318593675 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fb6c8027660 con 0x7fb6c80213a0 2024-04-14T15:20:02.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.833+0000 7fb6db7fe700 10 client.47396.objecter ms_dispatch 0x7fb6dc001550 command_reply(tid 13: 0 ) 2024-04-14T15:20:02.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.833+0000 7fb6db7fe700 10 client.47396.objecter _finish_command 13 = system:0 2024-04-14T15:20:02.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.833+0000 7fb6db7fe700 20 client.47396.objecter put_session s=0x7fb6c8021170 osd=6 4 2024-04-14T15:20:02.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.833+0000 7fb6db7fe700 15 client.47396.objecter _session_command_op_remove 6 13 2024-04-14T15:20:02.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.849+0000 7fb6e430f700 10 client.47396.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:20:02.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.849+0000 7fb6e430f700 15 client.47396.objecter _session_command_op_assign -1 14 2024-04-14T15:20:02.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.849+0000 7fb6e430f700 20 client.47396.objecter _get_session s=0x7fb6c8021170 osd=6 3 2024-04-14T15:20:02.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.849+0000 7fb6e430f700 20 client.47396.objecter put_session s=0x7fb6c8021170 osd=6 3 2024-04-14T15:20:02.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.849+0000 7fb6e430f700 20 client.47396.objecter _get_session s=0x7fb6c8021170 osd=6 3 2024-04-14T15:20:02.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.849+0000 7fb6e430f700 15 client.47396.objecter _session_command_op_remove -1 14 2024-04-14T15:20:02.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.849+0000 7fb6e430f700 20 client.47396.objecter get_session s=0x7fb6c8021170 osd=6 3 2024-04-14T15:20:02.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.849+0000 7fb6e430f700 15 client.47396.objecter _session_command_op_assign 6 14 2024-04-14T15:20:02.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.849+0000 7fb6e430f700 20 client.47396.objecter put_session s=0x7fb6c8021170 osd=6 4 2024-04-14T15:20:02.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.849+0000 7fb6e430f700 10 client.47396.objecter _send_command 14 2024-04-14T15:20:02.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.849+0000 7fb6e430f700 1 -- 172.21.15.90:0/318593675 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fb6c8027840 con 0x7fb6c80213a0 2024-04-14T15:20:02.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.851+0000 7fb6db7fe700 1 -- 172.21.15.90:0/318593675 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fb6c8027840 con 0x7fb6c80213a0 2024-04-14T15:20:02.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.852+0000 7fb6db7fe700 10 client.47396.objecter ms_dispatch 0x7fb6dc001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:20:02.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.852+0000 7fb6db7fe700 10 client.47396.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:20:02.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.852+0000 7fb6db7fe700 20 client.47396.objecter put_session s=0x7fb6c8021170 osd=6 4 2024-04-14T15:20:02.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.852+0000 7fb6db7fe700 15 client.47396.objecter _session_command_op_remove 6 14 2024-04-14T15:20:02.853 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:20:02.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.852+0000 7fb6e430f700 10 client.47396.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:20:02.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.852+0000 7fb6e430f700 15 client.47396.objecter _session_command_op_assign -1 15 2024-04-14T15:20:02.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.852+0000 7fb6e430f700 20 client.47396.objecter _get_session s=0x7fb6c8027840 osd=7 3 2024-04-14T15:20:02.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.852+0000 7fb6e430f700 20 client.47396.objecter put_session s=0x7fb6c8027840 osd=7 3 2024-04-14T15:20:02.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.852+0000 7fb6e430f700 20 client.47396.objecter _get_session s=0x7fb6c8027840 osd=7 3 2024-04-14T15:20:02.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.852+0000 7fb6e430f700 15 client.47396.objecter _session_command_op_remove -1 15 2024-04-14T15:20:02.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.852+0000 7fb6e430f700 20 client.47396.objecter get_session s=0x7fb6c8027840 osd=7 3 2024-04-14T15:20:02.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.852+0000 7fb6e430f700 15 client.47396.objecter _session_command_op_assign 7 15 2024-04-14T15:20:02.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.852+0000 7fb6e430f700 20 client.47396.objecter put_session s=0x7fb6c8027840 osd=7 4 2024-04-14T15:20:02.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.852+0000 7fb6e430f700 10 client.47396.objecter _send_command 15 2024-04-14T15:20:02.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.852+0000 7fb6e430f700 1 -- 172.21.15.90:0/318593675 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fb6c8028b70 con 0x7fb6c8027f60 2024-04-14T15:20:02.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.852+0000 7fb6db7fe700 10 client.47396.objecter ms_handle_connect 0x7fb6c8027f60 2024-04-14T15:20:02.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.853+0000 7fb6db7fe700 1 -- 172.21.15.90:0/318593675 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fb6c8028b70 con 0x7fb6c8027f60 2024-04-14T15:20:02.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.853+0000 7fb6db7fe700 10 client.47396.objecter ms_dispatch 0x7fb6dc001550 command_reply(tid 15: 0 ) 2024-04-14T15:20:02.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.853+0000 7fb6db7fe700 10 client.47396.objecter _finish_command 15 = system:0 2024-04-14T15:20:02.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.853+0000 7fb6db7fe700 20 client.47396.objecter put_session s=0x7fb6c8027840 osd=7 4 2024-04-14T15:20:02.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.853+0000 7fb6db7fe700 15 client.47396.objecter _session_command_op_remove 7 15 2024-04-14T15:20:02.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.870+0000 7fb6e430f700 10 client.47396.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:20:02.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.870+0000 7fb6e430f700 15 client.47396.objecter _session_command_op_assign -1 16 2024-04-14T15:20:02.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.870+0000 7fb6e430f700 20 client.47396.objecter _get_session s=0x7fb6c8027840 osd=7 3 2024-04-14T15:20:02.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.870+0000 7fb6e430f700 20 client.47396.objecter put_session s=0x7fb6c8027840 osd=7 3 2024-04-14T15:20:02.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.870+0000 7fb6e430f700 20 client.47396.objecter _get_session s=0x7fb6c8027840 osd=7 3 2024-04-14T15:20:02.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.870+0000 7fb6e430f700 15 client.47396.objecter _session_command_op_remove -1 16 2024-04-14T15:20:02.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.870+0000 7fb6e430f700 20 client.47396.objecter get_session s=0x7fb6c8027840 osd=7 3 2024-04-14T15:20:02.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.870+0000 7fb6e430f700 15 client.47396.objecter _session_command_op_assign 7 16 2024-04-14T15:20:02.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.870+0000 7fb6e430f700 20 client.47396.objecter put_session s=0x7fb6c8027840 osd=7 4 2024-04-14T15:20:02.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.870+0000 7fb6e430f700 10 client.47396.objecter _send_command 16 2024-04-14T15:20:02.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.870+0000 7fb6e430f700 1 -- 172.21.15.90:0/318593675 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fb6c8028b70 con 0x7fb6c8027f60 2024-04-14T15:20:02.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.871+0000 7fb6db7fe700 1 -- 172.21.15.90:0/318593675 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fb6c8028b70 con 0x7fb6c8027f60 2024-04-14T15:20:02.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.871+0000 7fb6db7fe700 10 client.47396.objecter ms_dispatch 0x7fb6dc001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:20:02.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.871+0000 7fb6db7fe700 10 client.47396.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:20:02.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.871+0000 7fb6db7fe700 20 client.47396.objecter put_session s=0x7fb6c8027840 osd=7 4 2024-04-14T15:20:02.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.871+0000 7fb6db7fe700 15 client.47396.objecter _session_command_op_remove 7 16 2024-04-14T15:20:02.872 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:20:02.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.872+0000 7fb6e430f700 10 librados: watch_flush enter 2024-04-14T15:20:02.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.872+0000 7fb6e430f700 10 librados: watch_flush exit 2024-04-14T15:20:02.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.872+0000 7fb6e430f700 10 client.47396.objecter close_session for osd.0 2024-04-14T15:20:02.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.872+0000 7fb6e430f700 1 -- 172.21.15.90:0/318593675 >> v1:172.21.15.90:6801/3761806885 conn(0x7fb6c8004730 legacy=0x7fb6c8002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:02.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.872+0000 7fb6e430f700 20 client.47396.objecter put_session s=0x7fb6c80053e0 osd=0 1 2024-04-14T15:20:02.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.872+0000 7fb6e430f700 10 client.47396.objecter close_session for osd.1 2024-04-14T15:20:02.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.872+0000 7fb6e430f700 1 -- 172.21.15.90:0/318593675 >> v1:172.21.15.90:6805/4170361699 conn(0x7fb6c8004310 legacy=0x7fb6c800f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:02.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.872+0000 7fb6e430f700 20 client.47396.objecter put_session s=0x7fb6c8004130 osd=1 1 2024-04-14T15:20:02.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.872+0000 7fb6e430f700 10 client.47396.objecter close_session for osd.2 2024-04-14T15:20:02.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.872+0000 7fb6e430f700 1 -- 172.21.15.90:0/318593675 >> v1:172.21.15.90:6809/1686184251 conn(0x7fb6c80119e0 legacy=0x7fb6c8013dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:02.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.872+0000 7fb6e430f700 20 client.47396.objecter put_session s=0x7fb6c8001980 osd=2 1 2024-04-14T15:20:02.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.872+0000 7fb6e430f700 10 client.47396.objecter close_session for osd.3 2024-04-14T15:20:02.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.872+0000 7fb6e430f700 1 -- 172.21.15.90:0/318593675 >> v1:172.21.15.90:6813/2270166774 conn(0x7fb6c80168b0 legacy=0x7fb6c8018ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:02.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.872+0000 7fb6e28ac700 1 -- 172.21.15.90:0/318593675 reap_dead start 2024-04-14T15:20:02.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.872+0000 7fb6e430f700 20 client.47396.objecter put_session s=0x7fb6c800c7a0 osd=3 1 2024-04-14T15:20:02.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.872+0000 7fb6e430f700 10 client.47396.objecter close_session for osd.4 2024-04-14T15:20:02.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.872+0000 7fb6e430f700 1 -- 172.21.15.90:0/318593675 >> v1:172.21.15.92:6800/3887549733 conn(0x7fb6c8016410 legacy=0x7fb6c801d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:02.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.872+0000 7fb6e430f700 20 client.47396.objecter put_session s=0x7fb6c8010dc0 osd=4 1 2024-04-14T15:20:02.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.872+0000 7fb6e430f700 10 client.47396.objecter close_session for osd.5 2024-04-14T15:20:02.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.872+0000 7fb6e430f700 1 -- 172.21.15.90:0/318593675 >> v1:172.21.15.92:6804/3244561440 conn(0x7fb6c8020d10 legacy=0x7fb6c801ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:02.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.872+0000 7fb6e430f700 20 client.47396.objecter put_session s=0x7fb6c80205f0 osd=5 1 2024-04-14T15:20:02.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.872+0000 7fb6e430f700 10 client.47396.objecter close_session for osd.6 2024-04-14T15:20:02.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.872+0000 7fb6e430f700 1 -- 172.21.15.90:0/318593675 >> v1:172.21.15.92:6808/636125963 conn(0x7fb6c80213a0 legacy=0x7fb6c8025d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:02.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.872+0000 7fb6e430f700 20 client.47396.objecter put_session s=0x7fb6c8021170 osd=6 1 2024-04-14T15:20:02.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.872+0000 7fb6e430f700 10 client.47396.objecter close_session for osd.7 2024-04-14T15:20:02.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.872+0000 7fb6e430f700 1 -- 172.21.15.90:0/318593675 >> v1:172.21.15.92:6812/1773706716 conn(0x7fb6c8027f60 legacy=0x7fb6c80283f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:02.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.872+0000 7fb6e430f700 20 client.47396.objecter put_session s=0x7fb6c8027840 osd=7 1 2024-04-14T15:20:02.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.872+0000 7fb6e430f700 20 client.47396.objecter shutdown clearing up homeless session... 2024-04-14T15:20:02.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.872+0000 7fb6e430f700 10 client.47396.objecter successfully canceled tick 2024-04-14T15:20:02.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.872+0000 7fb6e28ac700 1 -- 172.21.15.90:0/318593675 reap_dead start 2024-04-14T15:20:02.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.872+0000 7fb6e430f700 1 -- 172.21.15.90:0/318593675 >> v1:172.21.15.90:6800/420935427 conn(0x7fb6c4071300 legacy=0x7fb6c40737c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:02.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.872+0000 7fb6e430f700 1 -- 172.21.15.90:0/318593675 >> v1:172.21.15.90:6790/0 conn(0x7fb6dc10a610 legacy=0x7fb6dc10c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:02.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.872+0000 7fb6e430f700 1 -- 172.21.15.90:0/318593675 shutdown_connections 2024-04-14T15:20:02.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.872+0000 7fb6e430f700 1 -- 172.21.15.90:0/318593675 >> 172.21.15.90:0/318593675 conn(0x7fb6dc0fd290 msgr2=0x7fb6dc06a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:20:02.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.872+0000 7fb6e430f700 1 -- 172.21.15.90:0/318593675 shutdown_connections 2024-04-14T15:20:02.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.872+0000 7fb6e430f700 1 -- 172.21.15.90:0/318593675 wait complete. 2024-04-14T15:20:02.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:02.872+0000 7fb6e430f700 1 librados: shutdown 2024-04-14T15:20:02.876 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:20:02.876 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:20:02.876 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:20:02.876 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:20:02.877 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:20:02.877 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:20:02.877 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:20:02.877 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:20:02.903 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:20:02.931 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:02.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:02.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:20:03.033 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:20:03.059 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:03.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:03.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:20:03.161 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:20:03.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:03.188 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:03.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:20:03.289 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:20:03.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:03.315 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:03.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:20:03.416 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:20:03.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:03 smithi090 ceph-mon[41670]: 11.15 deep-scrub starts 2024-04-14T15:20:03.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:03 smithi090 ceph-mon[41670]: 11.15 deep-scrub ok 2024-04-14T15:20:03.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:03 smithi090 ceph-mon[41670]: 5.12 scrub starts 2024-04-14T15:20:03.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:03 smithi090 ceph-mon[41670]: 5.12 scrub ok 2024-04-14T15:20:03.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:03 smithi090 ceph-mon[41670]: 5.3 scrub starts 2024-04-14T15:20:03.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:03 smithi090 ceph-mon[41670]: 5.3 scrub ok 2024-04-14T15:20:03.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:03 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/318593675' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:20:03.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:03 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:20:03.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:03 smithi090 ceph-mon[41670]: osdmap e492: 8 total, 8 up, 8 in 2024-04-14T15:20:03.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:03 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-14T15:20:03.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:03 smithi090 ceph-mon[30399]: 11.15 deep-scrub starts 2024-04-14T15:20:03.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:03 smithi090 ceph-mon[30399]: 11.15 deep-scrub ok 2024-04-14T15:20:03.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:03 smithi090 ceph-mon[30399]: 5.12 scrub starts 2024-04-14T15:20:03.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:03 smithi090 ceph-mon[30399]: 5.12 scrub ok 2024-04-14T15:20:03.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:03 smithi090 ceph-mon[30399]: 5.3 scrub starts 2024-04-14T15:20:03.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:03 smithi090 ceph-mon[30399]: 5.3 scrub ok 2024-04-14T15:20:03.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:03 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/318593675' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:20:03.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:03 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:20:03.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:03 smithi090 ceph-mon[30399]: osdmap e492: 8 total, 8 up, 8 in 2024-04-14T15:20:03.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:03 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-14T15:20:03.443 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:03.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:03.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:20:03.545 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:20:03.569 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:03.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:03.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:20:03.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:03 smithi092 ceph-mon[33966]: 11.15 deep-scrub starts 2024-04-14T15:20:03.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:03 smithi092 ceph-mon[33966]: 11.15 deep-scrub ok 2024-04-14T15:20:03.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:03 smithi092 ceph-mon[33966]: 5.12 scrub starts 2024-04-14T15:20:03.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:03 smithi092 ceph-mon[33966]: 5.12 scrub ok 2024-04-14T15:20:03.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:03 smithi092 ceph-mon[33966]: 5.3 scrub starts 2024-04-14T15:20:03.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:03 smithi092 ceph-mon[33966]: 5.3 scrub ok 2024-04-14T15:20:03.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:03 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/318593675' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:20:03.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:03 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:20:03.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:03 smithi092 ceph-mon[33966]: osdmap e492: 8 total, 8 up, 8 in 2024-04-14T15:20:03.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:03 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-14T15:20:03.670 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:20:03.693 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:03.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:03.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:20:03.795 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:03.818 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:03.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:03.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:03.919 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:20:03.946 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:03.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:03.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:20:04.048 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:20:04.074 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:04.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:04.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:20:04.175 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:04.199 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:04.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:04.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:04.262 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:20:03 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:20:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:20:04.262 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:04 smithi090 ceph-mon[41670]: 110.e scrub starts 2024-04-14T15:20:04.262 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:04 smithi090 ceph-mon[41670]: 110.e scrub ok 2024-04-14T15:20:04.262 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:04 smithi090 ceph-mon[41670]: 11.3 scrub starts 2024-04-14T15:20:04.263 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:04 smithi090 ceph-mon[41670]: 11.3 scrub ok 2024-04-14T15:20:04.263 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:04 smithi090 ceph-mon[41670]: pgmap v706: 258 pgs: 32 unknown, 226 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-14T15:20:04.263 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:04 smithi090 ceph-mon[41670]: 4.c deep-scrub starts 2024-04-14T15:20:04.263 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:04 smithi090 ceph-mon[41670]: 4.c deep-scrub ok 2024-04-14T15:20:04.263 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:04 smithi090 ceph-mon[41670]: osdmap e493: 8 total, 8 up, 8 in 2024-04-14T15:20:04.263 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:04 smithi090 ceph-mon[30399]: 110.e scrub starts 2024-04-14T15:20:04.263 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:04 smithi090 ceph-mon[30399]: 110.e scrub ok 2024-04-14T15:20:04.263 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:04 smithi090 ceph-mon[30399]: 11.3 scrub starts 2024-04-14T15:20:04.263 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:04 smithi090 ceph-mon[30399]: 11.3 scrub ok 2024-04-14T15:20:04.263 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:04 smithi090 ceph-mon[30399]: pgmap v706: 258 pgs: 32 unknown, 226 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-14T15:20:04.263 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:04 smithi090 ceph-mon[30399]: 4.c deep-scrub starts 2024-04-14T15:20:04.263 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:04 smithi090 ceph-mon[30399]: 4.c deep-scrub ok 2024-04-14T15:20:04.263 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:04 smithi090 ceph-mon[30399]: osdmap e493: 8 total, 8 up, 8 in 2024-04-14T15:20:04.301 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:04.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:04.326 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:04.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:04.427 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:20:04.455 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:04.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:04.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:20:04.557 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:20:04.584 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:04.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:04.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:20:04.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:04 smithi092 ceph-mon[33966]: 110.e scrub starts 2024-04-14T15:20:04.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:04 smithi092 ceph-mon[33966]: 110.e scrub ok 2024-04-14T15:20:04.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:04 smithi092 ceph-mon[33966]: 11.3 scrub starts 2024-04-14T15:20:04.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:04 smithi092 ceph-mon[33966]: 11.3 scrub ok 2024-04-14T15:20:04.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:04 smithi092 ceph-mon[33966]: pgmap v706: 258 pgs: 32 unknown, 226 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-14T15:20:04.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:04 smithi092 ceph-mon[33966]: 4.c deep-scrub starts 2024-04-14T15:20:04.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:04 smithi092 ceph-mon[33966]: 4.c deep-scrub ok 2024-04-14T15:20:04.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:04 smithi092 ceph-mon[33966]: osdmap e493: 8 total, 8 up, 8 in 2024-04-14T15:20:04.686 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:20:04.714 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:04.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:04.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:20:04.815 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:20:04.843 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:04.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:04.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:20:04.944 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:20:04.971 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:04.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:04.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:20:05.072 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:20:05.098 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:05.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:05.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:20:05.199 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:20:05.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:05.225 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:05.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:20:05.326 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:20:05.351 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:05.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:05.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:20:05.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:05 smithi090 ceph-mon[41670]: 2.8 scrub starts 2024-04-14T15:20:05.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:05 smithi090 ceph-mon[41670]: 2.8 scrub ok 2024-04-14T15:20:05.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:05 smithi090 ceph-mon[41670]: 6.7 scrub starts 2024-04-14T15:20:05.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:05 smithi090 ceph-mon[41670]: 6.7 scrub ok 2024-04-14T15:20:05.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:05 smithi090 ceph-mon[41670]: 110.f scrub starts 2024-04-14T15:20:05.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:05 smithi090 ceph-mon[41670]: 110.f scrub ok 2024-04-14T15:20:05.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:05 smithi090 ceph-mon[41670]: osdmap e494: 8 total, 8 up, 8 in 2024-04-14T15:20:05.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:05 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "305.1b"}]: dispatch 2024-04-14T15:20:05.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:05 smithi090 ceph-mon[41670]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "305.1b"}]: dispatch 2024-04-14T15:20:05.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:05 smithi090 ceph-mon[30399]: 2.8 scrub starts 2024-04-14T15:20:05.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:05 smithi090 ceph-mon[30399]: 2.8 scrub ok 2024-04-14T15:20:05.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:05 smithi090 ceph-mon[30399]: 6.7 scrub starts 2024-04-14T15:20:05.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:05 smithi090 ceph-mon[30399]: 6.7 scrub ok 2024-04-14T15:20:05.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:05 smithi090 ceph-mon[30399]: 110.f scrub starts 2024-04-14T15:20:05.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:05 smithi090 ceph-mon[30399]: 110.f scrub ok 2024-04-14T15:20:05.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:05 smithi090 ceph-mon[30399]: osdmap e494: 8 total, 8 up, 8 in 2024-04-14T15:20:05.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:05 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "305.1b"}]: dispatch 2024-04-14T15:20:05.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:05 smithi090 ceph-mon[30399]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "305.1b"}]: dispatch 2024-04-14T15:20:05.452 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:20:05.477 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:05.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:05.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:20:05.579 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:20:05.604 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:05.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:05.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:20:05.622 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:20:05 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:20:05.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:05 smithi092 ceph-mon[33966]: 2.8 scrub starts 2024-04-14T15:20:05.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:05 smithi092 ceph-mon[33966]: 2.8 scrub ok 2024-04-14T15:20:05.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:05 smithi092 ceph-mon[33966]: 6.7 scrub starts 2024-04-14T15:20:05.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:05 smithi092 ceph-mon[33966]: 6.7 scrub ok 2024-04-14T15:20:05.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:05 smithi092 ceph-mon[33966]: 110.f scrub starts 2024-04-14T15:20:05.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:05 smithi092 ceph-mon[33966]: 110.f scrub ok 2024-04-14T15:20:05.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:05 smithi092 ceph-mon[33966]: osdmap e494: 8 total, 8 up, 8 in 2024-04-14T15:20:05.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:05 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "305.1b"}]: dispatch 2024-04-14T15:20:05.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:05 smithi092 ceph-mon[33966]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "305.1b"}]: dispatch 2024-04-14T15:20:05.706 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:20:05.732 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:05.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:05.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:20:05.834 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:05.858 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:05.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:05.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:05.960 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:05.983 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:05.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:05.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:06.085 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:20:06.110 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:06.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:06.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:20:06.121 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-14T15:20:06.212 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:20:06.238 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:06.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:06.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:20:06.340 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:20:06.363 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:06.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:06.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:20:06.402 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:06 smithi090 ceph-mon[41670]: 6.1c scrub starts 2024-04-14T15:20:06.402 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:06 smithi090 ceph-mon[41670]: 6.1c scrub ok 2024-04-14T15:20:06.402 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:06 smithi090 ceph-mon[41670]: 6.4 deep-scrub starts 2024-04-14T15:20:06.402 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:06 smithi090 ceph-mon[41670]: 6.4 deep-scrub ok 2024-04-14T15:20:06.402 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:06 smithi090 ceph-mon[41670]: pgmap v709: 258 pgs: 258 active+clean; 8.4 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-04-14T15:20:06.402 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:06 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:20:06.403 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:06 smithi090 ceph-mon[30399]: 6.1c scrub starts 2024-04-14T15:20:06.403 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:06 smithi090 ceph-mon[30399]: 6.1c scrub ok 2024-04-14T15:20:06.403 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:06 smithi090 ceph-mon[30399]: 6.4 deep-scrub starts 2024-04-14T15:20:06.403 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:06 smithi090 ceph-mon[30399]: 6.4 deep-scrub ok 2024-04-14T15:20:06.403 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:06 smithi090 ceph-mon[30399]: pgmap v709: 258 pgs: 258 active+clean; 8.4 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-04-14T15:20:06.403 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:06 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:20:06.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:06.401+0000 7fe3c571d700 1 -- 172.21.15.90:0/3520746222 >> v1:172.21.15.90:6789/0 conn(0x7fe3c010f300 legacy=0x7fe3c01117c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:06.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:06.402+0000 7fe3c571d700 1 -- 172.21.15.90:0/3520746222 shutdown_connections 2024-04-14T15:20:06.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:06.402+0000 7fe3c571d700 1 -- 172.21.15.90:0/3520746222 >> 172.21.15.90:0/3520746222 conn(0x7fe3c00fe3d0 msgr2=0x7fe3c01007f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:20:06.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:06.402+0000 7fe3c571d700 1 -- 172.21.15.90:0/3520746222 shutdown_connections 2024-04-14T15:20:06.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:06.402+0000 7fe3c571d700 1 -- 172.21.15.90:0/3520746222 wait complete. 2024-04-14T15:20:06.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:06.402+0000 7fe3c571d700 1 librados: starting msgr at 2024-04-14T15:20:06.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:06.402+0000 7fe3c571d700 1 librados: starting objecter 2024-04-14T15:20:06.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:06.402+0000 7fe3c571d700 1 Processor -- start 2024-04-14T15:20:06.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:06.402+0000 7fe3c571d700 1 -- start start 2024-04-14T15:20:06.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:06.402+0000 7fe3c571d700 1 librados: setting wanted keys 2024-04-14T15:20:06.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:06.402+0000 7fe3c571d700 1 librados: calling monclient init 2024-04-14T15:20:06.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:06.403+0000 7fe3c571d700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe3c010efb0 con 0x7fe3c0108880 2024-04-14T15:20:06.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:06.403+0000 7fe3c571d700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe3c01a8f10 con 0x7fe3c010f300 2024-04-14T15:20:06.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:06.403+0000 7fe3c571d700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe3c01aa0f0 con 0x7fe3c010b730 2024-04-14T15:20:06.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:06.403+0000 7fe3be7fc700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7fe3c010b730 0x7fe3c010c510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:58828/0 (socket says 172.21.15.90:58828) 2024-04-14T15:20:06.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:06.403+0000 7fe3be7fc700 1 -- 172.21.15.90:0/4096591461 learned_addr learned my addr 172.21.15.90:0/4096591461 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:20:06.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:06.403+0000 7fe3beffd700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fe3c0108880 0x7fe3c010be00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:41188/0 (socket says 172.21.15.90:41188) 2024-04-14T15:20:06.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:06.403+0000 7fe3a7fff700 10 client.?.objecter ms_handle_connect 0x7fe3c0108880 2024-04-14T15:20:06.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:06.403+0000 7fe3a7fff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:20:06.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:06.403+0000 7fe3a7fff700 10 client.?.objecter ms_handle_connect 0x7fe3c010b730 2024-04-14T15:20:06.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:06.403+0000 7fe3a7fff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:20:06.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:06.403+0000 7fe3a7fff700 10 client.?.objecter ms_handle_connect 0x7fe3c010f300 2024-04-14T15:20:06.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:06.403+0000 7fe3a7fff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:20:06.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:06.403+0000 7fe3a7fff700 1 -- 172.21.15.90:0/4096591461 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 374432498 0 0) 0x7fe3c010efb0 con 0x7fe3c0108880 2024-04-14T15:20:06.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:06.403+0000 7fe3a7fff700 1 -- 172.21.15.90:0/4096591461 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe3a80035b0 con 0x7fe3c0108880 2024-04-14T15:20:06.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:06.403+0000 7fe3a7fff700 1 -- 172.21.15.90:0/4096591461 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3244663754 0 0) 0x7fe3c01aa0f0 con 0x7fe3c010b730 2024-04-14T15:20:06.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:06.403+0000 7fe3a7fff700 1 -- 172.21.15.90:0/4096591461 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe3c010efb0 con 0x7fe3c010b730 2024-04-14T15:20:06.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:06.403+0000 7fe3a7fff700 1 -- 172.21.15.90:0/4096591461 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1813987858 0 0) 0x7fe3c01a8f10 con 0x7fe3c010f300 2024-04-14T15:20:06.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:06.403+0000 7fe3a7fff700 1 -- 172.21.15.90:0/4096591461 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe3c01aa0f0 con 0x7fe3c010f300 2024-04-14T15:20:06.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:06.403+0000 7fe3a7fff700 1 -- 172.21.15.90:0/4096591461 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 422674660 0 0) 0x7fe3a80035b0 con 0x7fe3c0108880 2024-04-14T15:20:06.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:06.403+0000 7fe3a7fff700 1 -- 172.21.15.90:0/4096591461 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe3c01a8f10 con 0x7fe3c0108880 2024-04-14T15:20:06.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:06.403+0000 7fe3a7fff700 1 -- 172.21.15.90:0/4096591461 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fe3b00018d0 con 0x7fe3c0108880 2024-04-14T15:20:06.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:06.403+0000 7fe3a7fff700 1 -- 172.21.15.90:0/4096591461 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2420274011 0 0) 0x7fe3c010efb0 con 0x7fe3c010b730 2024-04-14T15:20:06.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:06.403+0000 7fe3a7fff700 1 -- 172.21.15.90:0/4096591461 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe3a80035b0 con 0x7fe3c010b730 2024-04-14T15:20:06.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:06.403+0000 7fe3a7fff700 1 -- 172.21.15.90:0/4096591461 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fe3b40018b0 con 0x7fe3c010b730 2024-04-14T15:20:06.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:06.403+0000 7fe3a7fff700 1 -- 172.21.15.90:0/4096591461 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1514101994 0 0) 0x7fe3c01aa0f0 con 0x7fe3c010f300 2024-04-14T15:20:06.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:06.404+0000 7fe3a7fff700 1 -- 172.21.15.90:0/4096591461 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe3c010efb0 con 0x7fe3c010f300 2024-04-14T15:20:06.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:06.404+0000 7fe3a7fff700 1 -- 172.21.15.90:0/4096591461 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fe3b80017b0 con 0x7fe3c010f300 2024-04-14T15:20:06.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:06.404+0000 7fe3a7fff700 1 -- 172.21.15.90:0/4096591461 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 131375374 0 0) 0x7fe3c01a8f10 con 0x7fe3c0108880 2024-04-14T15:20:06.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:06.404+0000 7fe3a7fff700 1 -- 172.21.15.90:0/4096591461 >> v1:172.21.15.90:6790/0 conn(0x7fe3c010b730 legacy=0x7fe3c010c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:06.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:06.404+0000 7fe3a7fff700 1 -- 172.21.15.90:0/4096591461 >> v1:172.21.15.92:6789/0 conn(0x7fe3c010f300 legacy=0x7fe3c01a67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:06.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:06.404+0000 7fe3a7fff700 1 -- 172.21.15.90:0/4096591461 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3c01ab2d0 con 0x7fe3c0108880 2024-04-14T15:20:06.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:06.404+0000 7fe3c571d700 1 -- 172.21.15.90:0/4096591461 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe3c01aa2c0 con 0x7fe3c0108880 2024-04-14T15:20:06.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:06.404+0000 7fe3c571d700 10 client.37449.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:20:06.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:06.404+0000 7fe3c571d700 1 -- 172.21.15.90:0/4096591461 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe3c01aa780 con 0x7fe3c0108880 2024-04-14T15:20:06.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:06.404+0000 7fe3c571d700 1 librados: init done 2024-04-14T15:20:06.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:06.404+0000 7fe3a7fff700 1 -- 172.21.15.90:0/4096591461 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fe3b0003b40 con 0x7fe3c0108880 2024-04-14T15:20:06.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:06.404+0000 7fe3a7fff700 1 -- 172.21.15.90:0/4096591461 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fe3b0005c50 con 0x7fe3c0108880 2024-04-14T15:20:06.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:06.405+0000 7fe3c571d700 1 -- 172.21.15.90:0/4096591461 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3ac005150 con 0x7fe3c0108880 2024-04-14T15:20:06.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:06.405+0000 7fe3a7fff700 1 -- 172.21.15.90:0/4096591461 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fe3b001c920 con 0x7fe3c0108880 2024-04-14T15:20:06.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:06.406+0000 7fe3a7fff700 1 -- 172.21.15.90:0/4096591461 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(494..494 src has 51..494) ==== 8197+0+0 (unknown 342560019 0 0) 0x7fe3b008aa00 con 0x7fe3c0108880 2024-04-14T15:20:06.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:06.406+0000 7fe3a7fff700 10 client.37449.objecter ms_dispatch 0x7fe3c0001550 osd_map(494..494 src has 51..494) 2024-04-14T15:20:06.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:06.406+0000 7fe3a7fff700 3 client.37449.objecter handle_osd_map got epochs [494,494] > 0 2024-04-14T15:20:06.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:06.406+0000 7fe3a7fff700 3 client.37449.objecter handle_osd_map decoding full epoch 494 2024-04-14T15:20:06.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:06.406+0000 7fe3a7fff700 20 client.37449.objecter dump_active .. 0 homeless 2024-04-14T15:20:06.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:06.406+0000 7fe3a7fff700 10 client.37449.objecter ms_handle_connect 0x7fe3a8071380 2024-04-14T15:20:06.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:06.411+0000 7fe3a7fff700 1 -- 172.21.15.90:0/4096591461 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fe3b0057410 con 0x7fe3c0108880 2024-04-14T15:20:06.465 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:20:06.493 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:06.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:06.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:20:06.595 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:20:06.618 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:06.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:06.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:20:06.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:06 smithi092 ceph-mon[33966]: 6.1c scrub starts 2024-04-14T15:20:06.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:06 smithi092 ceph-mon[33966]: 6.1c scrub ok 2024-04-14T15:20:06.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:06 smithi092 ceph-mon[33966]: 6.4 deep-scrub starts 2024-04-14T15:20:06.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:06 smithi092 ceph-mon[33966]: 6.4 deep-scrub ok 2024-04-14T15:20:06.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:06 smithi092 ceph-mon[33966]: pgmap v709: 258 pgs: 258 active+clean; 8.4 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-04-14T15:20:06.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:06 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:20:06.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:06.638+0000 7fe3c571d700 1 -- 172.21.15.90:0/4096591461 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7fe3ac005440 con 0x7fe3c0108880 2024-04-14T15:20:06.719 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:20:06.747 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:06.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:06.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:20:06.849 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:20:06.876 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:06.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:06.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:20:06.897 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:20:06.930 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-14T15:20:06.930 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-14T15:20:06.930 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-14T15:20:06.930 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-14T15:20:06.978 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:20:07.004 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:07.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:07.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:20:07.105 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:07.130 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:07.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:07.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:07.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:07.201+0000 7fe3a7fff700 1 -- 172.21.15.90:0/4096591461 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v495) ==== 87+0+0 (unknown 93990959 0 0) 0x7fe3b005b0c0 con 0x7fe3c0108880 2024-04-14T15:20:07.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:07.204+0000 7fe3c571d700 10 librados: watch_flush enter 2024-04-14T15:20:07.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:07.204+0000 7fe3c571d700 10 librados: watch_flush exit 2024-04-14T15:20:07.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:07.204+0000 7fe3c571d700 20 client.37449.objecter shutdown clearing up homeless session... 2024-04-14T15:20:07.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:07.204+0000 7fe3c571d700 10 client.37449.objecter successfully canceled tick 2024-04-14T15:20:07.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:07.204+0000 7fe3c571d700 1 -- 172.21.15.90:0/4096591461 >> v1:172.21.15.90:6800/420935427 conn(0x7fe3a8071380 legacy=0x7fe3a8073840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:07.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:07.204+0000 7fe3c571d700 1 -- 172.21.15.90:0/4096591461 >> v1:172.21.15.90:6789/0 conn(0x7fe3c0108880 legacy=0x7fe3c010be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:07.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:07.204+0000 7fe3c571d700 1 -- 172.21.15.90:0/4096591461 shutdown_connections 2024-04-14T15:20:07.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:07.204+0000 7fe3c571d700 1 -- 172.21.15.90:0/4096591461 >> 172.21.15.90:0/4096591461 conn(0x7fe3c00fe3d0 msgr2=0x7fe3c0112880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:20:07.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:07.204+0000 7fe3c571d700 1 -- 172.21.15.90:0/4096591461 shutdown_connections 2024-04-14T15:20:07.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:07.204+0000 7fe3c571d700 1 -- 172.21.15.90:0/4096591461 wait complete. 2024-04-14T15:20:07.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:07.204+0000 7fe3c571d700 1 librados: shutdown 2024-04-14T15:20:07.207 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is set 2024-04-14T15:20:07.232 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:20:07.255 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:07.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:07.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:20:07.357 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:07.380 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:07.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:07.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:07.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:07 smithi090 ceph-mon[41670]: 305.1b deep-scrub starts 2024-04-14T15:20:07.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:07 smithi090 ceph-mon[41670]: 305.1b deep-scrub ok 2024-04-14T15:20:07.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:07 smithi090 ceph-mon[41670]: 4.1e scrub starts 2024-04-14T15:20:07.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:07 smithi090 ceph-mon[41670]: 4.1e scrub ok 2024-04-14T15:20:07.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:07 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/4096591461' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:20:07.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:07 smithi090 ceph-mon[30399]: 305.1b deep-scrub starts 2024-04-14T15:20:07.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:07 smithi090 ceph-mon[30399]: 305.1b deep-scrub ok 2024-04-14T15:20:07.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:07 smithi090 ceph-mon[30399]: 4.1e scrub starts 2024-04-14T15:20:07.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:07 smithi090 ceph-mon[30399]: 4.1e scrub ok 2024-04-14T15:20:07.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:07 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/4096591461' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:20:07.464 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-14T15:20:07.464 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-14T15:20:07.465 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-04-14T15:20:07.465 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-04-14T15:20:07.465 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph pg dump --format=json 2024-04-14T15:20:07.482 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:07.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:07.532 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:07.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:07.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:07 smithi092 ceph-mon[33966]: 305.1b deep-scrub starts 2024-04-14T15:20:07.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:07 smithi092 ceph-mon[33966]: 305.1b deep-scrub ok 2024-04-14T15:20:07.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:07 smithi092 ceph-mon[33966]: 4.1e scrub starts 2024-04-14T15:20:07.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:07 smithi092 ceph-mon[33966]: 4.1e scrub ok 2024-04-14T15:20:07.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:07 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/4096591461' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:20:07.633 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:20:07.656 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:07.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:07.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:20:07.758 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:07.782 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:07.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:07.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:07.883 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:20:07.911 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:07.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:07.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:20:08.013 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:08.037 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:08.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:08.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:08.138 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:20:08.166 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:08.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:08.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:20:08.267 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:20:08.295 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:08.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:08.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:20:08.397 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:20:08.420 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:08.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:08.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:20:08.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:08 smithi090 ceph-mon[41670]: 5.2 scrub starts 2024-04-14T15:20:08.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:08 smithi090 ceph-mon[41670]: 5.2 scrub ok 2024-04-14T15:20:08.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:08 smithi090 ceph-mon[41670]: pgmap v710: 258 pgs: 258 active+clean; 8.4 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-14T15:20:08.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:08 smithi090 ceph-mon[41670]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:20:08.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:08 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/4096591461' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-14T15:20:08.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:08 smithi090 ceph-mon[41670]: osdmap e495: 8 total, 8 up, 8 in 2024-04-14T15:20:08.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:08 smithi090 ceph-mon[30399]: 5.2 scrub starts 2024-04-14T15:20:08.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:08 smithi090 ceph-mon[30399]: 5.2 scrub ok 2024-04-14T15:20:08.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:08 smithi090 ceph-mon[30399]: pgmap v710: 258 pgs: 258 active+clean; 8.4 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-14T15:20:08.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:08 smithi090 ceph-mon[30399]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:20:08.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:08 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/4096591461' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-14T15:20:08.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:08 smithi090 ceph-mon[30399]: osdmap e495: 8 total, 8 up, 8 in 2024-04-14T15:20:08.522 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:20:08.546 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:08.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:08.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:20:08.573 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-14T15:20:08.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:08 smithi092 ceph-mon[33966]: 5.2 scrub starts 2024-04-14T15:20:08.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:08 smithi092 ceph-mon[33966]: 5.2 scrub ok 2024-04-14T15:20:08.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:08 smithi092 ceph-mon[33966]: pgmap v710: 258 pgs: 258 active+clean; 8.4 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-14T15:20:08.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:08 smithi092 ceph-mon[33966]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:20:08.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:08 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/4096591461' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-14T15:20:08.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:08 smithi092 ceph-mon[33966]: osdmap e495: 8 total, 8 up, 8 in 2024-04-14T15:20:08.647 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:08.680 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:08.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:08.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:08.782 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:20:08.810 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:08.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:08.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:20:08.911 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:20:08.939 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:08.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:08.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:20:09.040 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:20:09.064 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:09.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:09.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:20:09.166 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:20:09.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:09.194 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:09.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:20:09.295 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:20:09.323 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:09.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:09.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:20:09.425 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:20:09.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:09 smithi090 ceph-mon[41670]: 4.15 scrub starts 2024-04-14T15:20:09.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:09 smithi090 ceph-mon[41670]: 4.15 scrub ok 2024-04-14T15:20:09.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:09 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:20:09.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:09 smithi090 ceph-mon[30399]: 4.15 scrub starts 2024-04-14T15:20:09.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:09 smithi090 ceph-mon[30399]: 4.15 scrub ok 2024-04-14T15:20:09.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:09 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:20:09.448 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:09.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:09.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:20:09.550 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:20:09.577 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:09.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:09.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:20:09.580 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:20:09.584 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:20:09.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:09 smithi092 ceph-mon[33966]: 4.15 scrub starts 2024-04-14T15:20:09.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:09 smithi092 ceph-mon[33966]: 4.15 scrub ok 2024-04-14T15:20:09.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:09 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:20:09.679 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:20:09.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:09.704 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:09.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:20:09.805 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:20:09.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:09.829 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:09.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:20:09.930 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:20:09.957 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:09.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:09.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:20:10.059 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:20:10.084 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:10.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:10.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:20:10.145 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-14T15:20:10.186 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:20:10.213 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:10.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:10.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:20:10.315 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:20:10.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:10.342 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:10.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:20:10.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:10 smithi090 ceph-mon[30399]: pgmap v712: 258 pgs: 258 active+clean; 8.4 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1023 B/s wr, 2 op/s 2024-04-14T15:20:10.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:10 smithi090 ceph-mon[41670]: pgmap v712: 258 pgs: 258 active+clean; 8.4 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1023 B/s wr, 2 op/s 2024-04-14T15:20:10.443 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:20:10.466 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:10.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:10.467 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:20:10.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.497+0000 7ff2862db700 1 -- 172.21.15.90:0/3790502559 >> v1:172.21.15.90:6789/0 conn(0x7ff28010b730 legacy=0x7ff28010db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:10.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.497+0000 7ff2862db700 1 -- 172.21.15.90:0/3790502559 shutdown_connections 2024-04-14T15:20:10.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.497+0000 7ff2862db700 1 -- 172.21.15.90:0/3790502559 >> 172.21.15.90:0/3790502559 conn(0x7ff2800fe3d0 msgr2=0x7ff2801007f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:20:10.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.497+0000 7ff2862db700 1 -- 172.21.15.90:0/3790502559 shutdown_connections 2024-04-14T15:20:10.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.497+0000 7ff2862db700 1 -- 172.21.15.90:0/3790502559 wait complete. 2024-04-14T15:20:10.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.497+0000 7ff2862db700 1 librados: starting msgr at 2024-04-14T15:20:10.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.497+0000 7ff2862db700 1 librados: starting objecter 2024-04-14T15:20:10.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.498+0000 7ff2862db700 1 Processor -- start 2024-04-14T15:20:10.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.498+0000 7ff2862db700 1 -- start start 2024-04-14T15:20:10.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.498+0000 7ff2862db700 1 librados: setting wanted keys 2024-04-14T15:20:10.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.498+0000 7ff2862db700 1 librados: calling monclient init 2024-04-14T15:20:10.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.498+0000 7ff2862db700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff2801a79f0 con 0x7ff280108880 2024-04-14T15:20:10.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.498+0000 7ff2862db700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff2801a8c30 con 0x7ff28010f300 2024-04-14T15:20:10.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.498+0000 7ff2862db700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff2801a9e70 con 0x7ff28010b730 2024-04-14T15:20:10.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.498+0000 7ff27ffff700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7ff280108880 0x7ff28010eae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:49322/0 (socket says 172.21.15.90:49322) 2024-04-14T15:20:10.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.498+0000 7ff27ffff700 1 -- 172.21.15.90:0/2378288715 learned_addr learned my addr 172.21.15.90:0/2378288715 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:20:10.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.498+0000 7ff27d7fa700 10 client.?.objecter ms_handle_connect 0x7ff280108880 2024-04-14T15:20:10.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.498+0000 7ff27d7fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:20:10.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.498+0000 7ff27d7fa700 10 client.?.objecter ms_handle_connect 0x7ff28010b730 2024-04-14T15:20:10.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.498+0000 7ff27d7fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:20:10.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.498+0000 7ff27d7fa700 1 -- 172.21.15.90:0/2378288715 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1312500671 0 0) 0x7ff2801a9e70 con 0x7ff28010b730 2024-04-14T15:20:10.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.498+0000 7ff27d7fa700 1 -- 172.21.15.90:0/2378288715 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff2680031f0 con 0x7ff28010b730 2024-04-14T15:20:10.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.498+0000 7ff27d7fa700 10 client.?.objecter ms_handle_connect 0x7ff28010f300 2024-04-14T15:20:10.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.498+0000 7ff27d7fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:20:10.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.498+0000 7ff27d7fa700 1 -- 172.21.15.90:0/2378288715 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2433157748 0 0) 0x7ff2801a79f0 con 0x7ff280108880 2024-04-14T15:20:10.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.498+0000 7ff27d7fa700 1 -- 172.21.15.90:0/2378288715 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff2801a9e70 con 0x7ff280108880 2024-04-14T15:20:10.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.499+0000 7ff27d7fa700 1 -- 172.21.15.90:0/2378288715 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 867655159 0 0) 0x7ff2801a8c30 con 0x7ff28010f300 2024-04-14T15:20:10.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.499+0000 7ff27d7fa700 1 -- 172.21.15.90:0/2378288715 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff2801a79f0 con 0x7ff28010f300 2024-04-14T15:20:10.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.499+0000 7ff27d7fa700 1 -- 172.21.15.90:0/2378288715 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3024688757 0 0) 0x7ff2801a9e70 con 0x7ff280108880 2024-04-14T15:20:10.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.499+0000 7ff27d7fa700 1 -- 172.21.15.90:0/2378288715 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff2801a8c30 con 0x7ff280108880 2024-04-14T15:20:10.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.499+0000 7ff27d7fa700 1 -- 172.21.15.90:0/2378288715 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7ff270001930 con 0x7ff280108880 2024-04-14T15:20:10.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.499+0000 7ff27d7fa700 1 -- 172.21.15.90:0/2378288715 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3983844351 0 0) 0x7ff2680031f0 con 0x7ff28010b730 2024-04-14T15:20:10.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.499+0000 7ff27d7fa700 1 -- 172.21.15.90:0/2378288715 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff2801a9e70 con 0x7ff28010b730 2024-04-14T15:20:10.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.499+0000 7ff27d7fa700 1 -- 172.21.15.90:0/2378288715 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7ff274004470 con 0x7ff28010b730 2024-04-14T15:20:10.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.499+0000 7ff27d7fa700 1 -- 172.21.15.90:0/2378288715 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3026622993 0 0) 0x7ff2801a8c30 con 0x7ff280108880 2024-04-14T15:20:10.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.499+0000 7ff27d7fa700 1 -- 172.21.15.90:0/2378288715 >> v1:172.21.15.90:6790/0 conn(0x7ff28010b730 legacy=0x7ff2801a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:10.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.499+0000 7ff27d7fa700 1 -- 172.21.15.90:0/2378288715 >> v1:172.21.15.92:6789/0 conn(0x7ff28010f300 legacy=0x7ff2801a60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:10.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.499+0000 7ff27d7fa700 1 -- 172.21.15.90:0/2378288715 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2801ab0b0 con 0x7ff280108880 2024-04-14T15:20:10.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.499+0000 7ff2862db700 1 -- 172.21.15.90:0/2378288715 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff2801aa0a0 con 0x7ff280108880 2024-04-14T15:20:10.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.499+0000 7ff2862db700 10 client.37455.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:20:10.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.499+0000 7ff2862db700 1 -- 172.21.15.90:0/2378288715 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff2801aa560 con 0x7ff280108880 2024-04-14T15:20:10.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.499+0000 7ff2862db700 1 librados: init done 2024-04-14T15:20:10.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.499+0000 7ff27d7fa700 1 -- 172.21.15.90:0/2378288715 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7ff270001d00 con 0x7ff280108880 2024-04-14T15:20:10.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.499+0000 7ff27d7fa700 1 -- 172.21.15.90:0/2378288715 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7ff270005df0 con 0x7ff280108880 2024-04-14T15:20:10.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.500+0000 7ff2862db700 1 -- 172.21.15.90:0/2378288715 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7ff26c005100 con 0x7ff280108880 2024-04-14T15:20:10.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.501+0000 7ff27d7fa700 1 -- 172.21.15.90:0/2378288715 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7ff270003950 con 0x7ff280108880 2024-04-14T15:20:10.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.501+0000 7ff27d7fa700 1 -- 172.21.15.90:0/2378288715 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(496..496 src has 51..496) ==== 8197+0+0 (unknown 2611802321 0 0) 0x7ff27008aa10 con 0x7ff280108880 2024-04-14T15:20:10.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.501+0000 7ff27d7fa700 10 client.37455.objecter ms_dispatch 0x7ff280001550 osd_map(496..496 src has 51..496) 2024-04-14T15:20:10.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.501+0000 7ff27d7fa700 3 client.37455.objecter handle_osd_map got epochs [496,496] > 0 2024-04-14T15:20:10.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.501+0000 7ff27d7fa700 3 client.37455.objecter handle_osd_map decoding full epoch 496 2024-04-14T15:20:10.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.501+0000 7ff27d7fa700 20 client.37455.objecter dump_active .. 0 homeless 2024-04-14T15:20:10.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.501+0000 7ff27d7fa700 1 -- 172.21.15.90:0/2378288715 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v496) ==== 54+0+16 (unknown 2453855458 0 2420000192) 0x7ff27008add0 con 0x7ff280108880 2024-04-14T15:20:10.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.501+0000 7ff27d7fa700 10 client.37455.objecter ms_handle_connect 0x7ff268071050 2024-04-14T15:20:10.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.501+0000 7ff2862db700 10 client.37455.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:20:10.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.501+0000 7ff2862db700 15 client.37455.objecter _session_command_op_assign -1 1 2024-04-14T15:20:10.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.501+0000 7ff2862db700 20 client.37455.objecter _get_session s=0x7ff26c0053e0 osd=0 3 2024-04-14T15:20:10.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.501+0000 7ff2862db700 20 client.37455.objecter put_session s=0x7ff26c0053e0 osd=0 3 2024-04-14T15:20:10.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.501+0000 7ff2862db700 20 client.37455.objecter _get_session s=0x7ff26c0053e0 osd=0 3 2024-04-14T15:20:10.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.501+0000 7ff2862db700 15 client.37455.objecter _session_command_op_remove -1 1 2024-04-14T15:20:10.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.501+0000 7ff2862db700 20 client.37455.objecter get_session s=0x7ff26c0053e0 osd=0 3 2024-04-14T15:20:10.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.501+0000 7ff2862db700 15 client.37455.objecter _session_command_op_assign 0 1 2024-04-14T15:20:10.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.501+0000 7ff2862db700 20 client.37455.objecter put_session s=0x7ff26c0053e0 osd=0 4 2024-04-14T15:20:10.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.501+0000 7ff2862db700 10 client.37455.objecter _send_command 1 2024-04-14T15:20:10.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.501+0000 7ff2862db700 1 -- 172.21.15.90:0/2378288715 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff26c004b70 con 0x7ff26c004730 2024-04-14T15:20:10.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.502+0000 7ff27d7fa700 10 client.37455.objecter ms_handle_connect 0x7ff26c004730 2024-04-14T15:20:10.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.502+0000 7ff27d7fa700 1 -- 172.21.15.90:0/2378288715 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7ff26c004b70 con 0x7ff26c004730 2024-04-14T15:20:10.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.502+0000 7ff27d7fa700 10 client.37455.objecter ms_dispatch 0x7ff280001550 command_reply(tid 1: 0 ) 2024-04-14T15:20:10.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.502+0000 7ff27d7fa700 10 client.37455.objecter _finish_command 1 = system:0 2024-04-14T15:20:10.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.502+0000 7ff27d7fa700 20 client.37455.objecter put_session s=0x7ff26c0053e0 osd=0 4 2024-04-14T15:20:10.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.502+0000 7ff27d7fa700 15 client.37455.objecter _session_command_op_remove 0 1 2024-04-14T15:20:10.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.522+0000 7ff2862db700 10 client.37455.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:20:10.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.522+0000 7ff2862db700 15 client.37455.objecter _session_command_op_assign -1 2 2024-04-14T15:20:10.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.522+0000 7ff2862db700 20 client.37455.objecter _get_session s=0x7ff26c0053e0 osd=0 3 2024-04-14T15:20:10.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.522+0000 7ff2862db700 20 client.37455.objecter put_session s=0x7ff26c0053e0 osd=0 3 2024-04-14T15:20:10.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.522+0000 7ff2862db700 20 client.37455.objecter _get_session s=0x7ff26c0053e0 osd=0 3 2024-04-14T15:20:10.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.522+0000 7ff2862db700 15 client.37455.objecter _session_command_op_remove -1 2 2024-04-14T15:20:10.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.522+0000 7ff2862db700 20 client.37455.objecter get_session s=0x7ff26c0053e0 osd=0 3 2024-04-14T15:20:10.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.522+0000 7ff2862db700 15 client.37455.objecter _session_command_op_assign 0 2 2024-04-14T15:20:10.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.522+0000 7ff2862db700 20 client.37455.objecter put_session s=0x7ff26c0053e0 osd=0 4 2024-04-14T15:20:10.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.522+0000 7ff2862db700 10 client.37455.objecter _send_command 2 2024-04-14T15:20:10.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.522+0000 7ff2862db700 1 -- 172.21.15.90:0/2378288715 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff26c0079f0 con 0x7ff26c004730 2024-04-14T15:20:10.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.524+0000 7ff27d7fa700 1 -- 172.21.15.90:0/2378288715 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff26c0079f0 con 0x7ff26c004730 2024-04-14T15:20:10.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.524+0000 7ff27d7fa700 10 client.37455.objecter ms_dispatch 0x7ff280001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:20:10.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.524+0000 7ff27d7fa700 10 client.37455.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:20:10.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.524+0000 7ff27d7fa700 20 client.37455.objecter put_session s=0x7ff26c0053e0 osd=0 4 2024-04-14T15:20:10.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.524+0000 7ff27d7fa700 15 client.37455.objecter _session_command_op_remove 0 2 2024-04-14T15:20:10.524 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:20:10.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.524+0000 7ff2862db700 10 client.37455.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:20:10.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.524+0000 7ff2862db700 15 client.37455.objecter _session_command_op_assign -1 3 2024-04-14T15:20:10.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.524+0000 7ff2862db700 20 client.37455.objecter _get_session s=0x7ff26c004130 osd=1 3 2024-04-14T15:20:10.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.524+0000 7ff2862db700 20 client.37455.objecter put_session s=0x7ff26c004130 osd=1 3 2024-04-14T15:20:10.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.524+0000 7ff2862db700 20 client.37455.objecter _get_session s=0x7ff26c004130 osd=1 3 2024-04-14T15:20:10.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.524+0000 7ff2862db700 15 client.37455.objecter _session_command_op_remove -1 3 2024-04-14T15:20:10.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.524+0000 7ff2862db700 20 client.37455.objecter get_session s=0x7ff26c004130 osd=1 3 2024-04-14T15:20:10.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.524+0000 7ff2862db700 15 client.37455.objecter _session_command_op_assign 1 3 2024-04-14T15:20:10.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.524+0000 7ff2862db700 20 client.37455.objecter put_session s=0x7ff26c004130 osd=1 4 2024-04-14T15:20:10.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.524+0000 7ff2862db700 10 client.37455.objecter _send_command 3 2024-04-14T15:20:10.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.524+0000 7ff2862db700 1 -- 172.21.15.90:0/2378288715 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7ff26c0022a0 con 0x7ff26c004310 2024-04-14T15:20:10.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.524+0000 7ff27d7fa700 10 client.37455.objecter ms_handle_connect 0x7ff26c004310 2024-04-14T15:20:10.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.525+0000 7ff27d7fa700 1 -- 172.21.15.90:0/2378288715 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7ff26c0022a0 con 0x7ff26c004310 2024-04-14T15:20:10.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.525+0000 7ff27d7fa700 10 client.37455.objecter ms_dispatch 0x7ff280001550 command_reply(tid 3: 0 ) 2024-04-14T15:20:10.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.525+0000 7ff27d7fa700 10 client.37455.objecter _finish_command 3 = system:0 2024-04-14T15:20:10.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.525+0000 7ff27d7fa700 20 client.37455.objecter put_session s=0x7ff26c004130 osd=1 4 2024-04-14T15:20:10.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.525+0000 7ff27d7fa700 15 client.37455.objecter _session_command_op_remove 1 3 2024-04-14T15:20:10.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.548+0000 7ff2862db700 10 client.37455.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:20:10.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.548+0000 7ff2862db700 15 client.37455.objecter _session_command_op_assign -1 4 2024-04-14T15:20:10.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.548+0000 7ff2862db700 20 client.37455.objecter _get_session s=0x7ff26c004130 osd=1 3 2024-04-14T15:20:10.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.548+0000 7ff2862db700 20 client.37455.objecter put_session s=0x7ff26c004130 osd=1 3 2024-04-14T15:20:10.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.548+0000 7ff2862db700 20 client.37455.objecter _get_session s=0x7ff26c004130 osd=1 3 2024-04-14T15:20:10.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.548+0000 7ff2862db700 15 client.37455.objecter _session_command_op_remove -1 4 2024-04-14T15:20:10.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.548+0000 7ff2862db700 20 client.37455.objecter get_session s=0x7ff26c004130 osd=1 3 2024-04-14T15:20:10.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.548+0000 7ff2862db700 15 client.37455.objecter _session_command_op_assign 1 4 2024-04-14T15:20:10.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.548+0000 7ff2862db700 20 client.37455.objecter put_session s=0x7ff26c004130 osd=1 4 2024-04-14T15:20:10.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.548+0000 7ff2862db700 10 client.37455.objecter _send_command 4 2024-04-14T15:20:10.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.548+0000 7ff2862db700 1 -- 172.21.15.90:0/2378288715 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff26c001720 con 0x7ff26c004310 2024-04-14T15:20:10.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.549+0000 7ff27d7fa700 1 -- 172.21.15.90:0/2378288715 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff26c001720 con 0x7ff26c004310 2024-04-14T15:20:10.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.549+0000 7ff27d7fa700 10 client.37455.objecter ms_dispatch 0x7ff280001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:20:10.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.549+0000 7ff27d7fa700 10 client.37455.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:20:10.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.549+0000 7ff27d7fa700 20 client.37455.objecter put_session s=0x7ff26c004130 osd=1 4 2024-04-14T15:20:10.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.549+0000 7ff27d7fa700 15 client.37455.objecter _session_command_op_remove 1 4 2024-04-14T15:20:10.550 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:20:10.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.550+0000 7ff2862db700 10 client.37455.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:20:10.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.550+0000 7ff2862db700 15 client.37455.objecter _session_command_op_assign -1 5 2024-04-14T15:20:10.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.550+0000 7ff2862db700 20 client.37455.objecter _get_session s=0x7ff26c001980 osd=2 3 2024-04-14T15:20:10.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.550+0000 7ff2862db700 20 client.37455.objecter put_session s=0x7ff26c001980 osd=2 3 2024-04-14T15:20:10.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.550+0000 7ff2862db700 20 client.37455.objecter _get_session s=0x7ff26c001980 osd=2 3 2024-04-14T15:20:10.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.550+0000 7ff2862db700 15 client.37455.objecter _session_command_op_remove -1 5 2024-04-14T15:20:10.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.550+0000 7ff2862db700 20 client.37455.objecter get_session s=0x7ff26c001980 osd=2 3 2024-04-14T15:20:10.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.550+0000 7ff2862db700 15 client.37455.objecter _session_command_op_assign 2 5 2024-04-14T15:20:10.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.550+0000 7ff2862db700 20 client.37455.objecter put_session s=0x7ff26c001980 osd=2 4 2024-04-14T15:20:10.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.550+0000 7ff2862db700 10 client.37455.objecter _send_command 5 2024-04-14T15:20:10.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.550+0000 7ff2862db700 1 -- 172.21.15.90:0/2378288715 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7ff26c0110e0 con 0x7ff26c00d5d0 2024-04-14T15:20:10.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.550+0000 7ff27d7fa700 10 client.37455.objecter ms_handle_connect 0x7ff26c00d5d0 2024-04-14T15:20:10.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.551+0000 7ff27d7fa700 1 -- 172.21.15.90:0/2378288715 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7ff26c0110e0 con 0x7ff26c00d5d0 2024-04-14T15:20:10.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.551+0000 7ff27d7fa700 10 client.37455.objecter ms_dispatch 0x7ff280001550 command_reply(tid 5: 0 ) 2024-04-14T15:20:10.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.551+0000 7ff27d7fa700 10 client.37455.objecter _finish_command 5 = system:0 2024-04-14T15:20:10.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.551+0000 7ff27d7fa700 20 client.37455.objecter put_session s=0x7ff26c001980 osd=2 4 2024-04-14T15:20:10.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.551+0000 7ff27d7fa700 15 client.37455.objecter _session_command_op_remove 2 5 2024-04-14T15:20:10.568 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:20:10.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.569+0000 7ff2862db700 10 client.37455.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:20:10.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.569+0000 7ff2862db700 15 client.37455.objecter _session_command_op_assign -1 6 2024-04-14T15:20:10.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.569+0000 7ff2862db700 20 client.37455.objecter _get_session s=0x7ff26c001980 osd=2 3 2024-04-14T15:20:10.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.569+0000 7ff2862db700 20 client.37455.objecter put_session s=0x7ff26c001980 osd=2 3 2024-04-14T15:20:10.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.569+0000 7ff2862db700 20 client.37455.objecter _get_session s=0x7ff26c001980 osd=2 3 2024-04-14T15:20:10.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.569+0000 7ff2862db700 15 client.37455.objecter _session_command_op_remove -1 6 2024-04-14T15:20:10.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.569+0000 7ff2862db700 20 client.37455.objecter get_session s=0x7ff26c001980 osd=2 3 2024-04-14T15:20:10.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.569+0000 7ff2862db700 15 client.37455.objecter _session_command_op_assign 2 6 2024-04-14T15:20:10.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.569+0000 7ff2862db700 20 client.37455.objecter put_session s=0x7ff26c001980 osd=2 4 2024-04-14T15:20:10.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.569+0000 7ff2862db700 10 client.37455.objecter _send_command 6 2024-04-14T15:20:10.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.569+0000 7ff2862db700 1 -- 172.21.15.90:0/2378288715 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff26c008100 con 0x7ff26c00d5d0 2024-04-14T15:20:10.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.571+0000 7ff27d7fa700 1 -- 172.21.15.90:0/2378288715 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff26c008100 con 0x7ff26c00d5d0 2024-04-14T15:20:10.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.571+0000 7ff27d7fa700 10 client.37455.objecter ms_dispatch 0x7ff280001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:20:10.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.571+0000 7ff27d7fa700 10 client.37455.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:20:10.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.571+0000 7ff27d7fa700 20 client.37455.objecter put_session s=0x7ff26c001980 osd=2 4 2024-04-14T15:20:10.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.571+0000 7ff27d7fa700 15 client.37455.objecter _session_command_op_remove 2 6 2024-04-14T15:20:10.572 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:20:10.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.571+0000 7ff2862db700 10 client.37455.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:20:10.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.571+0000 7ff2862db700 15 client.37455.objecter _session_command_op_assign -1 7 2024-04-14T15:20:10.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.571+0000 7ff2862db700 20 client.37455.objecter _get_session s=0x7ff26c008390 osd=3 3 2024-04-14T15:20:10.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.571+0000 7ff2862db700 20 client.37455.objecter put_session s=0x7ff26c008390 osd=3 3 2024-04-14T15:20:10.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.571+0000 7ff2862db700 20 client.37455.objecter _get_session s=0x7ff26c008390 osd=3 3 2024-04-14T15:20:10.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.571+0000 7ff2862db700 15 client.37455.objecter _session_command_op_remove -1 7 2024-04-14T15:20:10.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.571+0000 7ff2862db700 20 client.37455.objecter get_session s=0x7ff26c008390 osd=3 3 2024-04-14T15:20:10.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.571+0000 7ff2862db700 15 client.37455.objecter _session_command_op_assign 3 7 2024-04-14T15:20:10.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.571+0000 7ff2862db700 20 client.37455.objecter put_session s=0x7ff26c008390 osd=3 4 2024-04-14T15:20:10.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.571+0000 7ff2862db700 10 client.37455.objecter _send_command 7 2024-04-14T15:20:10.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.571+0000 7ff2862db700 1 -- 172.21.15.90:0/2378288715 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7ff26c012fd0 con 0x7ff26c012410 2024-04-14T15:20:10.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.572+0000 7ff27d7fa700 10 client.37455.objecter ms_handle_connect 0x7ff26c012410 2024-04-14T15:20:10.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.572+0000 7ff27d7fa700 1 -- 172.21.15.90:0/2378288715 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7ff26c012fd0 con 0x7ff26c012410 2024-04-14T15:20:10.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.572+0000 7ff27d7fa700 10 client.37455.objecter ms_dispatch 0x7ff280001550 command_reply(tid 7: 0 ) 2024-04-14T15:20:10.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.572+0000 7ff27d7fa700 10 client.37455.objecter _finish_command 7 = system:0 2024-04-14T15:20:10.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.572+0000 7ff27d7fa700 20 client.37455.objecter put_session s=0x7ff26c008390 osd=3 4 2024-04-14T15:20:10.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.572+0000 7ff27d7fa700 15 client.37455.objecter _session_command_op_remove 3 7 2024-04-14T15:20:10.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.589+0000 7ff2862db700 10 client.37455.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:20:10.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.589+0000 7ff2862db700 15 client.37455.objecter _session_command_op_assign -1 8 2024-04-14T15:20:10.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.589+0000 7ff2862db700 20 client.37455.objecter _get_session s=0x7ff26c008390 osd=3 3 2024-04-14T15:20:10.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.589+0000 7ff2862db700 20 client.37455.objecter put_session s=0x7ff26c008390 osd=3 3 2024-04-14T15:20:10.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.589+0000 7ff2862db700 20 client.37455.objecter _get_session s=0x7ff26c008390 osd=3 3 2024-04-14T15:20:10.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.589+0000 7ff2862db700 15 client.37455.objecter _session_command_op_remove -1 8 2024-04-14T15:20:10.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.589+0000 7ff2862db700 20 client.37455.objecter get_session s=0x7ff26c008390 osd=3 3 2024-04-14T15:20:10.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.589+0000 7ff2862db700 15 client.37455.objecter _session_command_op_assign 3 8 2024-04-14T15:20:10.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.589+0000 7ff2862db700 20 client.37455.objecter put_session s=0x7ff26c008390 osd=3 4 2024-04-14T15:20:10.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.589+0000 7ff2862db700 10 client.37455.objecter _send_command 8 2024-04-14T15:20:10.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.589+0000 7ff2862db700 1 -- 172.21.15.90:0/2378288715 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff26c012fd0 con 0x7ff26c012410 2024-04-14T15:20:10.591 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:10.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:10.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:20:10.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.591+0000 7ff27d7fa700 1 -- 172.21.15.90:0/2378288715 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff26c012fd0 con 0x7ff26c012410 2024-04-14T15:20:10.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.591+0000 7ff27d7fa700 10 client.37455.objecter ms_dispatch 0x7ff280001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:20:10.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.591+0000 7ff27d7fa700 10 client.37455.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:20:10.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.591+0000 7ff27d7fa700 20 client.37455.objecter put_session s=0x7ff26c008390 osd=3 4 2024-04-14T15:20:10.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.591+0000 7ff27d7fa700 15 client.37455.objecter _session_command_op_remove 3 8 2024-04-14T15:20:10.592 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:20:10.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.591+0000 7ff2862db700 10 client.37455.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:20:10.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.591+0000 7ff2862db700 15 client.37455.objecter _session_command_op_assign -1 9 2024-04-14T15:20:10.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.591+0000 7ff2862db700 20 client.37455.objecter _get_session s=0x7ff26c012fd0 osd=4 3 2024-04-14T15:20:10.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.591+0000 7ff2862db700 20 client.37455.objecter put_session s=0x7ff26c012fd0 osd=4 3 2024-04-14T15:20:10.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.591+0000 7ff2862db700 20 client.37455.objecter _get_session s=0x7ff26c012fd0 osd=4 3 2024-04-14T15:20:10.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.591+0000 7ff2862db700 15 client.37455.objecter _session_command_op_remove -1 9 2024-04-14T15:20:10.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.591+0000 7ff2862db700 20 client.37455.objecter get_session s=0x7ff26c012fd0 osd=4 3 2024-04-14T15:20:10.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.591+0000 7ff2862db700 15 client.37455.objecter _session_command_op_assign 4 9 2024-04-14T15:20:10.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.591+0000 7ff2862db700 20 client.37455.objecter put_session s=0x7ff26c012fd0 osd=4 4 2024-04-14T15:20:10.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.591+0000 7ff2862db700 10 client.37455.objecter _send_command 9 2024-04-14T15:20:10.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.591+0000 7ff2862db700 1 -- 172.21.15.90:0/2378288715 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7ff26c01a750 con 0x7ff26c016c40 2024-04-14T15:20:10.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.592+0000 7ff27d7fa700 10 client.37455.objecter ms_handle_connect 0x7ff26c016c40 2024-04-14T15:20:10.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.593+0000 7ff27d7fa700 1 -- 172.21.15.90:0/2378288715 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7ff26c01a750 con 0x7ff26c016c40 2024-04-14T15:20:10.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.593+0000 7ff27d7fa700 10 client.37455.objecter ms_dispatch 0x7ff280001550 command_reply(tid 9: 0 ) 2024-04-14T15:20:10.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.593+0000 7ff27d7fa700 10 client.37455.objecter _finish_command 9 = system:0 2024-04-14T15:20:10.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.593+0000 7ff27d7fa700 20 client.37455.objecter put_session s=0x7ff26c012fd0 osd=4 4 2024-04-14T15:20:10.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.593+0000 7ff27d7fa700 15 client.37455.objecter _session_command_op_remove 4 9 2024-04-14T15:20:10.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.609+0000 7ff2862db700 10 client.37455.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:20:10.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.609+0000 7ff2862db700 15 client.37455.objecter _session_command_op_assign -1 10 2024-04-14T15:20:10.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.609+0000 7ff2862db700 20 client.37455.objecter _get_session s=0x7ff26c012fd0 osd=4 3 2024-04-14T15:20:10.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.609+0000 7ff2862db700 20 client.37455.objecter put_session s=0x7ff26c012fd0 osd=4 3 2024-04-14T15:20:10.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.609+0000 7ff2862db700 20 client.37455.objecter _get_session s=0x7ff26c012fd0 osd=4 3 2024-04-14T15:20:10.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.609+0000 7ff2862db700 15 client.37455.objecter _session_command_op_remove -1 10 2024-04-14T15:20:10.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.609+0000 7ff2862db700 20 client.37455.objecter get_session s=0x7ff26c012fd0 osd=4 3 2024-04-14T15:20:10.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.609+0000 7ff2862db700 15 client.37455.objecter _session_command_op_assign 4 10 2024-04-14T15:20:10.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.609+0000 7ff2862db700 20 client.37455.objecter put_session s=0x7ff26c012fd0 osd=4 4 2024-04-14T15:20:10.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.609+0000 7ff2862db700 10 client.37455.objecter _send_command 10 2024-04-14T15:20:10.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.609+0000 7ff2862db700 1 -- 172.21.15.90:0/2378288715 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff26c013400 con 0x7ff26c016c40 2024-04-14T15:20:10.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.611+0000 7ff27d7fa700 1 -- 172.21.15.90:0/2378288715 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff26c013400 con 0x7ff26c016c40 2024-04-14T15:20:10.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.611+0000 7ff27d7fa700 10 client.37455.objecter ms_dispatch 0x7ff280001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:20:10.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.611+0000 7ff27d7fa700 10 client.37455.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:20:10.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.611+0000 7ff27d7fa700 20 client.37455.objecter put_session s=0x7ff26c012fd0 osd=4 4 2024-04-14T15:20:10.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.611+0000 7ff27d7fa700 15 client.37455.objecter _session_command_op_remove 4 10 2024-04-14T15:20:10.612 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:20:10.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.612+0000 7ff2862db700 10 client.37455.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:20:10.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.612+0000 7ff2862db700 15 client.37455.objecter _session_command_op_assign -1 11 2024-04-14T15:20:10.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.612+0000 7ff2862db700 20 client.37455.objecter _get_session s=0x7ff26c00c700 osd=5 3 2024-04-14T15:20:10.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.612+0000 7ff2862db700 20 client.37455.objecter put_session s=0x7ff26c00c700 osd=5 3 2024-04-14T15:20:10.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.612+0000 7ff2862db700 20 client.37455.objecter _get_session s=0x7ff26c00c700 osd=5 3 2024-04-14T15:20:10.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.612+0000 7ff2862db700 15 client.37455.objecter _session_command_op_remove -1 11 2024-04-14T15:20:10.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.612+0000 7ff2862db700 20 client.37455.objecter get_session s=0x7ff26c00c700 osd=5 3 2024-04-14T15:20:10.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.612+0000 7ff2862db700 15 client.37455.objecter _session_command_op_assign 5 11 2024-04-14T15:20:10.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.612+0000 7ff2862db700 20 client.37455.objecter put_session s=0x7ff26c00c700 osd=5 4 2024-04-14T15:20:10.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.612+0000 7ff2862db700 10 client.37455.objecter _send_command 11 2024-04-14T15:20:10.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.612+0000 7ff2862db700 1 -- 172.21.15.90:0/2378288715 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7ff26c011f90 con 0x7ff26c011a60 2024-04-14T15:20:10.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.612+0000 7ff27d7fa700 10 client.37455.objecter ms_handle_connect 0x7ff26c011a60 2024-04-14T15:20:10.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.613+0000 7ff27d7fa700 1 -- 172.21.15.90:0/2378288715 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7ff26c011f90 con 0x7ff26c011a60 2024-04-14T15:20:10.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.613+0000 7ff27d7fa700 10 client.37455.objecter ms_dispatch 0x7ff280001550 command_reply(tid 11: 0 ) 2024-04-14T15:20:10.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.613+0000 7ff27d7fa700 10 client.37455.objecter _finish_command 11 = system:0 2024-04-14T15:20:10.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.613+0000 7ff27d7fa700 20 client.37455.objecter put_session s=0x7ff26c00c700 osd=5 4 2024-04-14T15:20:10.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.613+0000 7ff27d7fa700 15 client.37455.objecter _session_command_op_remove 5 11 2024-04-14T15:20:10.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:10 smithi092 ceph-mon[33966]: pgmap v712: 258 pgs: 258 active+clean; 8.4 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1023 B/s wr, 2 op/s 2024-04-14T15:20:10.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.631+0000 7ff2862db700 10 client.37455.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:20:10.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.631+0000 7ff2862db700 15 client.37455.objecter _session_command_op_assign -1 12 2024-04-14T15:20:10.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.631+0000 7ff2862db700 20 client.37455.objecter _get_session s=0x7ff26c00c700 osd=5 3 2024-04-14T15:20:10.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.631+0000 7ff2862db700 20 client.37455.objecter put_session s=0x7ff26c00c700 osd=5 3 2024-04-14T15:20:10.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.631+0000 7ff2862db700 20 client.37455.objecter _get_session s=0x7ff26c00c700 osd=5 3 2024-04-14T15:20:10.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.631+0000 7ff2862db700 15 client.37455.objecter _session_command_op_remove -1 12 2024-04-14T15:20:10.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.631+0000 7ff2862db700 20 client.37455.objecter get_session s=0x7ff26c00c700 osd=5 3 2024-04-14T15:20:10.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.631+0000 7ff2862db700 15 client.37455.objecter _session_command_op_assign 5 12 2024-04-14T15:20:10.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.631+0000 7ff2862db700 20 client.37455.objecter put_session s=0x7ff26c00c700 osd=5 4 2024-04-14T15:20:10.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.631+0000 7ff2862db700 10 client.37455.objecter _send_command 12 2024-04-14T15:20:10.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.631+0000 7ff2862db700 1 -- 172.21.15.90:0/2378288715 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff26c01f000 con 0x7ff26c011a60 2024-04-14T15:20:10.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.632+0000 7ff27d7fa700 1 -- 172.21.15.90:0/2378288715 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff26c01f000 con 0x7ff26c011a60 2024-04-14T15:20:10.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.632+0000 7ff27d7fa700 10 client.37455.objecter ms_dispatch 0x7ff280001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:20:10.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.632+0000 7ff27d7fa700 10 client.37455.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:20:10.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.632+0000 7ff27d7fa700 20 client.37455.objecter put_session s=0x7ff26c00c700 osd=5 4 2024-04-14T15:20:10.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.632+0000 7ff27d7fa700 15 client.37455.objecter _session_command_op_remove 5 12 2024-04-14T15:20:10.633 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:20:10.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.633+0000 7ff2862db700 10 client.37455.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:20:10.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.633+0000 7ff2862db700 15 client.37455.objecter _session_command_op_assign -1 13 2024-04-14T15:20:10.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.633+0000 7ff2862db700 20 client.37455.objecter _get_session s=0x7ff26c01f000 osd=6 3 2024-04-14T15:20:10.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.633+0000 7ff2862db700 20 client.37455.objecter put_session s=0x7ff26c01f000 osd=6 3 2024-04-14T15:20:10.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.633+0000 7ff2862db700 20 client.37455.objecter _get_session s=0x7ff26c01f000 osd=6 3 2024-04-14T15:20:10.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.633+0000 7ff2862db700 15 client.37455.objecter _session_command_op_remove -1 13 2024-04-14T15:20:10.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.633+0000 7ff2862db700 20 client.37455.objecter get_session s=0x7ff26c01f000 osd=6 3 2024-04-14T15:20:10.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.633+0000 7ff2862db700 15 client.37455.objecter _session_command_op_assign 6 13 2024-04-14T15:20:10.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.633+0000 7ff2862db700 20 client.37455.objecter put_session s=0x7ff26c01f000 osd=6 4 2024-04-14T15:20:10.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.633+0000 7ff2862db700 10 client.37455.objecter _send_command 13 2024-04-14T15:20:10.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.633+0000 7ff2862db700 1 -- 172.21.15.90:0/2378288715 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7ff26c023300 con 0x7ff26c01f720 2024-04-14T15:20:10.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.633+0000 7ff27d7fa700 10 client.37455.objecter ms_handle_connect 0x7ff26c01f720 2024-04-14T15:20:10.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.634+0000 7ff27d7fa700 1 -- 172.21.15.90:0/2378288715 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7ff26c023300 con 0x7ff26c01f720 2024-04-14T15:20:10.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.634+0000 7ff27d7fa700 10 client.37455.objecter ms_dispatch 0x7ff280001550 command_reply(tid 13: 0 ) 2024-04-14T15:20:10.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.634+0000 7ff27d7fa700 10 client.37455.objecter _finish_command 13 = system:0 2024-04-14T15:20:10.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.634+0000 7ff27d7fa700 20 client.37455.objecter put_session s=0x7ff26c01f000 osd=6 4 2024-04-14T15:20:10.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.634+0000 7ff27d7fa700 15 client.37455.objecter _session_command_op_remove 6 13 2024-04-14T15:20:10.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.651+0000 7ff2862db700 10 client.37455.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:20:10.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.651+0000 7ff2862db700 15 client.37455.objecter _session_command_op_assign -1 14 2024-04-14T15:20:10.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.651+0000 7ff2862db700 20 client.37455.objecter _get_session s=0x7ff26c01f000 osd=6 3 2024-04-14T15:20:10.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.651+0000 7ff2862db700 20 client.37455.objecter put_session s=0x7ff26c01f000 osd=6 3 2024-04-14T15:20:10.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.651+0000 7ff2862db700 20 client.37455.objecter _get_session s=0x7ff26c01f000 osd=6 3 2024-04-14T15:20:10.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.651+0000 7ff2862db700 15 client.37455.objecter _session_command_op_remove -1 14 2024-04-14T15:20:10.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.651+0000 7ff2862db700 20 client.37455.objecter get_session s=0x7ff26c01f000 osd=6 3 2024-04-14T15:20:10.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.651+0000 7ff2862db700 15 client.37455.objecter _session_command_op_assign 6 14 2024-04-14T15:20:10.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.651+0000 7ff2862db700 20 client.37455.objecter put_session s=0x7ff26c01f000 osd=6 4 2024-04-14T15:20:10.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.651+0000 7ff2862db700 10 client.37455.objecter _send_command 14 2024-04-14T15:20:10.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.651+0000 7ff2862db700 1 -- 172.21.15.90:0/2378288715 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff26c023300 con 0x7ff26c01f720 2024-04-14T15:20:10.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.653+0000 7ff27d7fa700 1 -- 172.21.15.90:0/2378288715 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff26c023300 con 0x7ff26c01f720 2024-04-14T15:20:10.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.653+0000 7ff27d7fa700 10 client.37455.objecter ms_dispatch 0x7ff280001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:20:10.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.653+0000 7ff27d7fa700 10 client.37455.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:20:10.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.653+0000 7ff27d7fa700 20 client.37455.objecter put_session s=0x7ff26c01f000 osd=6 4 2024-04-14T15:20:10.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.653+0000 7ff27d7fa700 15 client.37455.objecter _session_command_op_remove 6 14 2024-04-14T15:20:10.654 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:20:10.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.653+0000 7ff2862db700 10 client.37455.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:20:10.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.653+0000 7ff2862db700 15 client.37455.objecter _session_command_op_assign -1 15 2024-04-14T15:20:10.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.653+0000 7ff2862db700 20 client.37455.objecter _get_session s=0x7ff26c023340 osd=7 3 2024-04-14T15:20:10.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.653+0000 7ff2862db700 20 client.37455.objecter put_session s=0x7ff26c023340 osd=7 3 2024-04-14T15:20:10.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.653+0000 7ff2862db700 20 client.37455.objecter _get_session s=0x7ff26c023340 osd=7 3 2024-04-14T15:20:10.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.653+0000 7ff2862db700 15 client.37455.objecter _session_command_op_remove -1 15 2024-04-14T15:20:10.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.653+0000 7ff2862db700 20 client.37455.objecter get_session s=0x7ff26c023340 osd=7 3 2024-04-14T15:20:10.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.653+0000 7ff2862db700 15 client.37455.objecter _session_command_op_assign 7 15 2024-04-14T15:20:10.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.653+0000 7ff2862db700 20 client.37455.objecter put_session s=0x7ff26c023340 osd=7 4 2024-04-14T15:20:10.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.653+0000 7ff2862db700 10 client.37455.objecter _send_command 15 2024-04-14T15:20:10.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.653+0000 7ff2862db700 1 -- 172.21.15.90:0/2378288715 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7ff26c024620 con 0x7ff26c0252b0 2024-04-14T15:20:10.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.654+0000 7ff27d7fa700 10 client.37455.objecter ms_handle_connect 0x7ff26c0252b0 2024-04-14T15:20:10.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.655+0000 7ff27d7fa700 1 -- 172.21.15.90:0/2378288715 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7ff26c024620 con 0x7ff26c0252b0 2024-04-14T15:20:10.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.655+0000 7ff27d7fa700 10 client.37455.objecter ms_dispatch 0x7ff280001550 command_reply(tid 15: 0 ) 2024-04-14T15:20:10.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.655+0000 7ff27d7fa700 10 client.37455.objecter _finish_command 15 = system:0 2024-04-14T15:20:10.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.655+0000 7ff27d7fa700 20 client.37455.objecter put_session s=0x7ff26c023340 osd=7 4 2024-04-14T15:20:10.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.655+0000 7ff27d7fa700 15 client.37455.objecter _session_command_op_remove 7 15 2024-04-14T15:20:10.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.687+0000 7ff2862db700 10 client.37455.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:20:10.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.687+0000 7ff2862db700 15 client.37455.objecter _session_command_op_assign -1 16 2024-04-14T15:20:10.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.687+0000 7ff2862db700 20 client.37455.objecter _get_session s=0x7ff26c023340 osd=7 3 2024-04-14T15:20:10.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.687+0000 7ff2862db700 20 client.37455.objecter put_session s=0x7ff26c023340 osd=7 3 2024-04-14T15:20:10.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.687+0000 7ff2862db700 20 client.37455.objecter _get_session s=0x7ff26c023340 osd=7 3 2024-04-14T15:20:10.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.687+0000 7ff2862db700 15 client.37455.objecter _session_command_op_remove -1 16 2024-04-14T15:20:10.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.687+0000 7ff2862db700 20 client.37455.objecter get_session s=0x7ff26c023340 osd=7 3 2024-04-14T15:20:10.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.687+0000 7ff2862db700 15 client.37455.objecter _session_command_op_assign 7 16 2024-04-14T15:20:10.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.687+0000 7ff2862db700 20 client.37455.objecter put_session s=0x7ff26c023340 osd=7 4 2024-04-14T15:20:10.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.687+0000 7ff2862db700 10 client.37455.objecter _send_command 16 2024-04-14T15:20:10.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.687+0000 7ff2862db700 1 -- 172.21.15.90:0/2378288715 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff26c024620 con 0x7ff26c0252b0 2024-04-14T15:20:10.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.689+0000 7ff27d7fa700 1 -- 172.21.15.90:0/2378288715 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff26c024620 con 0x7ff26c0252b0 2024-04-14T15:20:10.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.689+0000 7ff27d7fa700 10 client.37455.objecter ms_dispatch 0x7ff280001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:20:10.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.689+0000 7ff27d7fa700 10 client.37455.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:20:10.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.689+0000 7ff27d7fa700 20 client.37455.objecter put_session s=0x7ff26c023340 osd=7 4 2024-04-14T15:20:10.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.689+0000 7ff27d7fa700 15 client.37455.objecter _session_command_op_remove 7 16 2024-04-14T15:20:10.690 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:20:10.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.689+0000 7ff2862db700 10 librados: watch_flush enter 2024-04-14T15:20:10.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.689+0000 7ff2862db700 10 librados: watch_flush exit 2024-04-14T15:20:10.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.689+0000 7ff2862db700 10 client.37455.objecter close_session for osd.0 2024-04-14T15:20:10.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.689+0000 7ff2862db700 1 -- 172.21.15.90:0/2378288715 >> v1:172.21.15.90:6801/3761806885 conn(0x7ff26c004730 legacy=0x7ff26c002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:10.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.689+0000 7ff2862db700 20 client.37455.objecter put_session s=0x7ff26c0053e0 osd=0 1 2024-04-14T15:20:10.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.689+0000 7ff2862db700 10 client.37455.objecter close_session for osd.1 2024-04-14T15:20:10.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.689+0000 7ff2862db700 1 -- 172.21.15.90:0/2378288715 >> v1:172.21.15.90:6805/4170361699 conn(0x7ff26c004310 legacy=0x7ff26c00ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:10.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.689+0000 7ff2862db700 20 client.37455.objecter put_session s=0x7ff26c004130 osd=1 1 2024-04-14T15:20:10.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.689+0000 7ff2862db700 10 client.37455.objecter close_session for osd.2 2024-04-14T15:20:10.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.689+0000 7ff2862db700 1 -- 172.21.15.90:0/2378288715 >> v1:172.21.15.90:6809/1686184251 conn(0x7ff26c00d5d0 legacy=0x7ff26c00f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:10.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.689+0000 7ff2862db700 20 client.37455.objecter put_session s=0x7ff26c001980 osd=2 1 2024-04-14T15:20:10.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.689+0000 7ff2862db700 10 client.37455.objecter close_session for osd.3 2024-04-14T15:20:10.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.689+0000 7ff2862db700 1 -- 172.21.15.90:0/2378288715 >> v1:172.21.15.90:6813/2270166774 conn(0x7ff26c012410 legacy=0x7ff26c0128c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:10.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.689+0000 7ff2862db700 20 client.37455.objecter put_session s=0x7ff26c008390 osd=3 1 2024-04-14T15:20:10.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.689+0000 7ff2862db700 10 client.37455.objecter close_session for osd.4 2024-04-14T15:20:10.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.689+0000 7ff2862db700 1 -- 172.21.15.90:0/2378288715 >> v1:172.21.15.92:6800/3887549733 conn(0x7ff26c016c40 legacy=0x7ff26c019030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:10.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.689+0000 7ff2862db700 20 client.37455.objecter put_session s=0x7ff26c012fd0 osd=4 1 2024-04-14T15:20:10.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.689+0000 7ff2862db700 10 client.37455.objecter close_session for osd.5 2024-04-14T15:20:10.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.689+0000 7ff2862db700 1 -- 172.21.15.90:0/2378288715 >> v1:172.21.15.92:6804/3244561440 conn(0x7ff26c011a60 legacy=0x7ff26c01d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:10.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.689+0000 7ff2862db700 20 client.37455.objecter put_session s=0x7ff26c00c700 osd=5 1 2024-04-14T15:20:10.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.689+0000 7ff284878700 1 -- 172.21.15.90:0/2378288715 reap_dead start 2024-04-14T15:20:10.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.689+0000 7ff2862db700 10 client.37455.objecter close_session for osd.6 2024-04-14T15:20:10.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.689+0000 7ff2862db700 1 -- 172.21.15.90:0/2378288715 >> v1:172.21.15.92:6808/636125963 conn(0x7ff26c01f720 legacy=0x7ff26c021b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:10.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.689+0000 7ff284878700 1 -- 172.21.15.90:0/2378288715 reap_dead start 2024-04-14T15:20:10.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.689+0000 7ff2862db700 20 client.37455.objecter put_session s=0x7ff26c01f000 osd=6 1 2024-04-14T15:20:10.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.689+0000 7ff2862db700 10 client.37455.objecter close_session for osd.7 2024-04-14T15:20:10.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.689+0000 7ff2862db700 1 -- 172.21.15.90:0/2378288715 >> v1:172.21.15.92:6812/1773706716 conn(0x7ff26c0252b0 legacy=0x7ff26c025710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:10.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.689+0000 7ff2862db700 20 client.37455.objecter put_session s=0x7ff26c023340 osd=7 1 2024-04-14T15:20:10.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.689+0000 7ff2862db700 20 client.37455.objecter shutdown clearing up homeless session... 2024-04-14T15:20:10.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.689+0000 7ff2862db700 10 client.37455.objecter successfully canceled tick 2024-04-14T15:20:10.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.689+0000 7ff2862db700 1 -- 172.21.15.90:0/2378288715 >> v1:172.21.15.90:6800/420935427 conn(0x7ff268071050 legacy=0x7ff268073510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:10.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.689+0000 7ff2862db700 1 -- 172.21.15.90:0/2378288715 >> v1:172.21.15.90:6789/0 conn(0x7ff280108880 legacy=0x7ff28010eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:10.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.689+0000 7ff284878700 1 -- 172.21.15.90:0/2378288715 reap_dead start 2024-04-14T15:20:10.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.689+0000 7ff284878700 1 -- 172.21.15.90:0/2378288715 reap_dead start 2024-04-14T15:20:10.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.689+0000 7ff2862db700 1 -- 172.21.15.90:0/2378288715 shutdown_connections 2024-04-14T15:20:10.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.689+0000 7ff2862db700 1 -- 172.21.15.90:0/2378288715 >> 172.21.15.90:0/2378288715 conn(0x7ff2800fe3d0 msgr2=0x7ff280112000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:20:10.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.689+0000 7ff2862db700 1 -- 172.21.15.90:0/2378288715 shutdown_connections 2024-04-14T15:20:10.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.689+0000 7ff2862db700 1 -- 172.21.15.90:0/2378288715 wait complete. 2024-04-14T15:20:10.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:10.689+0000 7ff2862db700 1 librados: shutdown 2024-04-14T15:20:10.694 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:20:10.694 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:20:10.694 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:20:10.694 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:20:10.694 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:20:10.694 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:20:10.694 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:20:10.694 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:20:10.694 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:20:10.718 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:10.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:10.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:20:10.820 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:20:10.848 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:10.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:10.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:20:10.950 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:20:10.977 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:10.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:10.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:20:11.078 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:20:11.105 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:11.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:11.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:20:11.207 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:20:11.234 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:11.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:11.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:20:11.336 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:11.360 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:11.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:11.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:11.430 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:11 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:20:11.430 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:11 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:20:11.430 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:11 smithi090 ceph-mon[41670]: osdmap e496: 8 total, 8 up, 8 in 2024-04-14T15:20:11.430 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:11 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2378288715' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:20:11.431 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:11 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:20:11.431 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:11 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:20:11.431 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:11 smithi090 ceph-mon[30399]: osdmap e496: 8 total, 8 up, 8 in 2024-04-14T15:20:11.431 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:11 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2378288715' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:20:11.461 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:20:11.489 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:11.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:11.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:20:11.591 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:20:11.617 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:11.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:11.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:20:11.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:11 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:20:11.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:11 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:20:11.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:11 smithi092 ceph-mon[33966]: osdmap e496: 8 total, 8 up, 8 in 2024-04-14T15:20:11.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:11 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2378288715' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:20:11.719 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:20:11.747 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:11.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:11.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:20:11.849 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:20:11.873 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:11.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:11.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:20:11.974 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:20:12.002 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:12.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:12.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:20:12.104 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:12.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:12.129 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:12.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:12.230 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:12.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:12.255 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:12.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:12.355 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:20:12.380 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:12.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:12.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:20:12.481 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:20:12.509 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:12.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:12.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:20:12.610 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:20:12.634 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:12.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:12.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:20:12.736 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:20:12.763 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:12.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:12.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:20:12.865 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:20:12.866 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:20:12.890 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:12.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:12.891 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:20:12.992 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:20:13.020 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:13.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:13.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:20:13.086 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:12 smithi090 ceph-mon[41670]: pgmap v714: 258 pgs: 258 active+clean; 8.4 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 830 B/s rd, 166 B/s wr, 1 op/s 2024-04-14T15:20:13.086 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:12 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:20:13.086 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:12 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:20:13.086 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:12 smithi090 ceph-mon[30399]: pgmap v714: 258 pgs: 258 active+clean; 8.4 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 830 B/s rd, 166 B/s wr, 1 op/s 2024-04-14T15:20:13.087 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:12 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:20:13.087 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:12 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:20:13.121 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:20:13.149 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:13.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:13.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:20:13.250 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:20:13.275 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:13.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:13.275 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:20:13.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:12 smithi092 ceph-mon[33966]: pgmap v714: 258 pgs: 258 active+clean; 8.4 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 830 B/s rd, 166 B/s wr, 1 op/s 2024-04-14T15:20:13.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:12 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:20:13.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:12 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:20:13.376 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:20:13.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:13.404 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:13.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:20:13.505 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:20:13.533 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:13.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:13.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:20:13.634 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:20:13.663 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:13.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:13.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:20:13.765 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:20:13.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:13.792 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:13.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:20:13.892 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:20:13.916 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:13.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:13.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:20:14.017 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:14.040 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:14.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:14.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:14.064 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-14T15:20:14.064 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:20:14.126 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:13 smithi090 ceph-mon[41670]: osdmap e497: 8 total, 8 up, 8 in 2024-04-14T15:20:14.126 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:20:13 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:20:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:20:14.126 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:13 smithi090 ceph-mon[30399]: osdmap e497: 8 total, 8 up, 8 in 2024-04-14T15:20:14.142 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:20:14.169 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:14.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:14.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:20:14.271 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:20:14.298 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:14.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:14.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:20:14.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:13 smithi092 ceph-mon[33966]: osdmap e497: 8 total, 8 up, 8 in 2024-04-14T15:20:14.400 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:20:14.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:14.425 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:14.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:20:14.526 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:20:14.554 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:14.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:14.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:20:14.655 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:20:14.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:14.680 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:14.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:20:14.780 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:20:14.808 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:14.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:14.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:20:14.854 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-14T15:20:14.909 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:20:14.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:14.934 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:14.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:20:14.977 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-14T15:20:15.035 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:15.059 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:15.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:15.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:15.160 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:20:15.184 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:15.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:15.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:20:15.188 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:14 smithi092 ceph-mon[33966]: pgmap v716: 258 pgs: 258 active+clean; 8.4 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 170 B/s wr, 1 op/s 2024-04-14T15:20:15.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:14 smithi090 ceph-mon[41670]: pgmap v716: 258 pgs: 258 active+clean; 8.4 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 170 B/s wr, 1 op/s 2024-04-14T15:20:15.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:14 smithi090 ceph-mon[30399]: pgmap v716: 258 pgs: 258 active+clean; 8.4 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 170 B/s wr, 1 op/s 2024-04-14T15:20:15.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:15.216+0000 7f1b096e4700 1 -- 172.21.15.90:0/949537563 >> v1:172.21.15.90:6789/0 conn(0x7f1b0410b6f0 legacy=0x7f1b0410dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:15.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:15.216+0000 7f1b096e4700 1 -- 172.21.15.90:0/949537563 shutdown_connections 2024-04-14T15:20:15.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:15.216+0000 7f1b096e4700 1 -- 172.21.15.90:0/949537563 >> 172.21.15.90:0/949537563 conn(0x7f1b04076aa0 msgr2=0x7f1b04076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:20:15.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:15.216+0000 7f1b096e4700 1 -- 172.21.15.90:0/949537563 shutdown_connections 2024-04-14T15:20:15.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:15.216+0000 7f1b096e4700 1 -- 172.21.15.90:0/949537563 wait complete. 2024-04-14T15:20:15.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:15.216+0000 7f1b096e4700 1 librados: starting msgr at 2024-04-14T15:20:15.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:15.216+0000 7f1b096e4700 1 librados: starting objecter 2024-04-14T15:20:15.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:15.216+0000 7f1b096e4700 1 Processor -- start 2024-04-14T15:20:15.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:15.217+0000 7f1b096e4700 1 -- start start 2024-04-14T15:20:15.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:15.217+0000 7f1b096e4700 1 librados: setting wanted keys 2024-04-14T15:20:15.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:15.217+0000 7f1b096e4700 1 librados: calling monclient init 2024-04-14T15:20:15.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:15.217+0000 7f1b096e4700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1b04108110 con 0x7f1b04108980 2024-04-14T15:20:15.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:15.217+0000 7f1b096e4700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1b04108320 con 0x7f1b0410b6f0 2024-04-14T15:20:15.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:15.217+0000 7f1b096e4700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1b041a9f70 con 0x7f1b0410f2f0 2024-04-14T15:20:15.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:15.217+0000 7f1b02ffd700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f1b04108980 0x7f1b04104f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:49348/0 (socket says 172.21.15.90:49348) 2024-04-14T15:20:15.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:15.217+0000 7f1b02ffd700 1 -- 172.21.15.90:0/1440636058 learned_addr learned my addr 172.21.15.90:0/1440636058 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:20:15.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:15.217+0000 7f1b00ff9700 10 client.?.objecter ms_handle_connect 0x7f1b0410f2f0 2024-04-14T15:20:15.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:15.217+0000 7f1b00ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:20:15.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:15.217+0000 7f1b00ff9700 10 client.?.objecter ms_handle_connect 0x7f1b04108980 2024-04-14T15:20:15.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:15.217+0000 7f1b00ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:20:15.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:15.217+0000 7f1b00ff9700 1 -- 172.21.15.90:0/1440636058 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1238754612 0 0) 0x7f1b041a9f70 con 0x7f1b0410f2f0 2024-04-14T15:20:15.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:15.217+0000 7f1b00ff9700 1 -- 172.21.15.90:0/1440636058 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1af00035b0 con 0x7f1b0410f2f0 2024-04-14T15:20:15.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:15.217+0000 7f1b00ff9700 10 client.?.objecter ms_handle_connect 0x7f1b0410b6f0 2024-04-14T15:20:15.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:15.217+0000 7f1b00ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:20:15.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:15.217+0000 7f1b00ff9700 1 -- 172.21.15.90:0/1440636058 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4038517444 0 0) 0x7f1b04108110 con 0x7f1b04108980 2024-04-14T15:20:15.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:15.217+0000 7f1b00ff9700 1 -- 172.21.15.90:0/1440636058 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1b041a9f70 con 0x7f1b04108980 2024-04-14T15:20:15.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:15.217+0000 7f1b00ff9700 1 -- 172.21.15.90:0/1440636058 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4123446912 0 0) 0x7f1b04108320 con 0x7f1b0410b6f0 2024-04-14T15:20:15.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:15.217+0000 7f1b00ff9700 1 -- 172.21.15.90:0/1440636058 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1b04108110 con 0x7f1b0410b6f0 2024-04-14T15:20:15.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:15.217+0000 7f1b00ff9700 1 -- 172.21.15.90:0/1440636058 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2603390073 0 0) 0x7f1af00035b0 con 0x7f1b0410f2f0 2024-04-14T15:20:15.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:15.217+0000 7f1b00ff9700 1 -- 172.21.15.90:0/1440636058 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1b04108320 con 0x7f1b0410f2f0 2024-04-14T15:20:15.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:15.217+0000 7f1b00ff9700 1 -- 172.21.15.90:0/1440636058 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f1afc001b70 con 0x7f1b0410f2f0 2024-04-14T15:20:15.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:15.217+0000 7f1b00ff9700 1 -- 172.21.15.90:0/1440636058 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4007821780 0 0) 0x7f1b041a9f70 con 0x7f1b04108980 2024-04-14T15:20:15.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:15.217+0000 7f1b00ff9700 1 -- 172.21.15.90:0/1440636058 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1af00035b0 con 0x7f1b04108980 2024-04-14T15:20:15.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:15.217+0000 7f1b00ff9700 1 -- 172.21.15.90:0/1440636058 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f1af4001850 con 0x7f1b04108980 2024-04-14T15:20:15.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:15.218+0000 7f1b00ff9700 1 -- 172.21.15.90:0/1440636058 <== mon.2 v1:172.21.15.90:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2929445750 0 0) 0x7f1b04108320 con 0x7f1b0410f2f0 2024-04-14T15:20:15.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:15.218+0000 7f1b00ff9700 1 -- 172.21.15.90:0/1440636058 >> v1:172.21.15.92:6789/0 conn(0x7f1b0410b6f0 legacy=0x7f1b04105670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:15.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:15.218+0000 7f1b00ff9700 1 -- 172.21.15.90:0/1440636058 >> v1:172.21.15.90:6789/0 conn(0x7f1b04108980 legacy=0x7f1b04104f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:15.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:15.218+0000 7f1b00ff9700 1 -- 172.21.15.90:0/1440636058 --> v1:172.21.15.90:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b041ab1b0 con 0x7f1b0410f2f0 2024-04-14T15:20:15.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:15.218+0000 7f1b096e4700 1 -- 172.21.15.90:0/1440636058 --> v1:172.21.15.90:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1b041aa1a0 con 0x7f1b0410f2f0 2024-04-14T15:20:15.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:15.218+0000 7f1b096e4700 10 client.47414.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:20:15.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:15.218+0000 7f1b096e4700 1 -- 172.21.15.90:0/1440636058 --> v1:172.21.15.90:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f1b041aa660 con 0x7f1b0410f2f0 2024-04-14T15:20:15.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:15.218+0000 7f1b096e4700 1 librados: init done 2024-04-14T15:20:15.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:15.218+0000 7f1b00ff9700 1 -- 172.21.15.90:0/1440636058 <== mon.2 v1:172.21.15.90:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f1afc003ea0 con 0x7f1b0410f2f0 2024-04-14T15:20:15.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:15.218+0000 7f1b00ff9700 1 -- 172.21.15.90:0/1440636058 <== mon.2 v1:172.21.15.90:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f1afc004df0 con 0x7f1b0410f2f0 2024-04-14T15:20:15.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:15.219+0000 7f1b096e4700 1 -- 172.21.15.90:0/1440636058 --> v1:172.21.15.90:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ae4005120 con 0x7f1b0410f2f0 2024-04-14T15:20:15.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:15.219+0000 7f1b00ff9700 1 -- 172.21.15.90:0/1440636058 <== mon.2 v1:172.21.15.90:6790/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f1afc003a00 con 0x7f1b0410f2f0 2024-04-14T15:20:15.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:15.220+0000 7f1b00ff9700 1 -- 172.21.15.90:0/1440636058 <== mon.2 v1:172.21.15.90:6790/0 8 ==== osd_map(497..497 src has 51..497) ==== 8197+0+0 (unknown 2548748994 0 0) 0x7f1afc089c30 con 0x7f1b0410f2f0 2024-04-14T15:20:15.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:15.220+0000 7f1b00ff9700 10 client.47414.objecter ms_dispatch 0x7f1b04001550 osd_map(497..497 src has 51..497) 2024-04-14T15:20:15.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:15.220+0000 7f1b00ff9700 3 client.47414.objecter handle_osd_map got epochs [497,497] > 0 2024-04-14T15:20:15.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:15.220+0000 7f1b00ff9700 3 client.47414.objecter handle_osd_map decoding full epoch 497 2024-04-14T15:20:15.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:15.220+0000 7f1b00ff9700 20 client.47414.objecter dump_active .. 0 homeless 2024-04-14T15:20:15.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:15.220+0000 7f1b00ff9700 10 client.47414.objecter ms_handle_connect 0x7f1af00710f0 2024-04-14T15:20:15.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:15.227+0000 7f1b00ff9700 1 -- 172.21.15.90:0/1440636058 <== mon.2 v1:172.21.15.90:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f1afc056640 con 0x7f1b0410f2f0 2024-04-14T15:20:15.285 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:20:15.314 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:15.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:15.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:20:15.415 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:20:15.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:15.430+0000 7f1b096e4700 1 -- 172.21.15.90:0/1440636058 --> v1:172.21.15.90:6800/420935427 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1ae4000b90 con 0x7f1af00710f0 2024-04-14T15:20:15.439 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:15.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:15.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:20:15.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:15.448+0000 7f1b00ff9700 1 -- 172.21.15.90:0/1440636058 <== mgr.14610 v1:172.21.15.90:6800/420935427 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+664486 (unknown 2965378022 0 307965836) 0x7f1ae4000b90 con 0x7f1af00710f0 2024-04-14T15:20:15.455 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:20:15.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:15.452+0000 7f1b096e4700 10 librados: watch_flush enter 2024-04-14T15:20:15.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:15.452+0000 7f1b096e4700 10 librados: watch_flush exit 2024-04-14T15:20:15.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:15.452+0000 7f1b096e4700 20 client.47414.objecter shutdown clearing up homeless session... 2024-04-14T15:20:15.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:15.452+0000 7f1b096e4700 10 client.47414.objecter successfully canceled tick 2024-04-14T15:20:15.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:15.452+0000 7f1b096e4700 1 -- 172.21.15.90:0/1440636058 >> v1:172.21.15.90:6800/420935427 conn(0x7f1af00710f0 legacy=0x7f1af00735b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:15.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:15.452+0000 7f1b096e4700 1 -- 172.21.15.90:0/1440636058 >> v1:172.21.15.90:6790/0 conn(0x7f1b0410f2f0 legacy=0x7f1b041a67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:15.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:15.452+0000 7f1b096e4700 1 -- 172.21.15.90:0/1440636058 shutdown_connections 2024-04-14T15:20:15.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:15.452+0000 7f1b096e4700 1 -- 172.21.15.90:0/1440636058 >> 172.21.15.90:0/1440636058 conn(0x7f1b04076aa0 msgr2=0x7f1b041117b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:20:15.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:15.452+0000 7f1b096e4700 1 -- 172.21.15.90:0/1440636058 shutdown_connections 2024-04-14T15:20:15.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:15.452+0000 7f1b096e4700 1 -- 172.21.15.90:0/1440636058 wait complete. 2024-04-14T15:20:15.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:15.452+0000 7f1b096e4700 1 librados: shutdown 2024-04-14T15:20:15.462 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-14T15:20:15.541 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:20:15.568 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:15.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:15.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:20:15.622 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:20:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:20:15.669 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:20:15.697 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:15.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:15.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:20:15.799 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:20:15.827 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:15.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:15.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:20:15.928 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:20:15.956 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:15.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:15.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:20:16.058 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:20:16.085 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:16.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:16.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:20:16.186 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:20:16.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:16.213 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:16.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:20:16.314 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:20:16.340 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:16.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:16.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:20:16.441 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:20:16.455 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":717,"stamp":"2024-04-14T15:20:15.106984+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8792364,"num_objects":378,"num_object_clones":12,"num_object_copies":1134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":377,"num_whiteouts":0,"num_read":1894,"num_read_kb":1541,"num_write":1365,"num_write_kb":12771,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":481,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":22,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6886,"ondisk_log_size":6886,"up":774,"acting":774,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":774,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":601408,"kb_used_data":90900,"kb_used_omap":12,"kb_used_meta":510323,"kb_avail":749359808,"statfs":{"total":767960285184,"available":767344443392,"internally_reserved":0,"allocated":93081600,"data_stored":64219554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":522571314},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":10,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.902801"},"pg_stats":[{"pgid":"305.1f","version":"0'0","reported_seq":22,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.934437+0000","last_change":"2024-04-14T15:20:02.947594+0000","last_active":"2024-04-14T15:20:12.934437+0000","last_peered":"2024-04-14T15:20:12.934437+0000","last_clean":"2024-04-14T15:20:12.934437+0000","last_became_active":"2024-04-14T15:20:02.947275+0000","last_became_peered":"2024-04-14T15:20:02.947275+0000","last_unstale":"2024-04-14T15:20:12.934437+0000","last_undegraded":"2024-04-14T15:20:12.934437+0000","last_fullsized":"2024-04-14T15:20:12.934437+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":491,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:01.924260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:01.924260+0000","last_clean_scrub_stamp":"2024-04-14T15:20:01.924260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:04.245930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"305.1e","version":"0'0","reported_seq":22,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.386006+0000","last_change":"2024-04-14T15:20:03.135132+0000","last_active":"2024-04-14T15:20:13.386006+0000","last_peered":"2024-04-14T15:20:13.386006+0000","last_clean":"2024-04-14T15:20:13.386006+0000","last_became_active":"2024-04-14T15:20:03.134675+0000","last_became_peered":"2024-04-14T15:20:03.134675+0000","last_unstale":"2024-04-14T15:20:13.386006+0000","last_undegraded":"2024-04-14T15:20:13.386006+0000","last_fullsized":"2024-04-14T15:20:13.386006+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":491,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:01.924260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:01.924260+0000","last_clean_scrub_stamp":"2024-04-14T15:20:01.924260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:31.748811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"305.1d","version":"0'0","reported_seq":22,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.385721+0000","last_change":"2024-04-14T15:20:02.947547+0000","last_active":"2024-04-14T15:20:13.385721+0000","last_peered":"2024-04-14T15:20:13.385721+0000","last_clean":"2024-04-14T15:20:13.385721+0000","last_became_active":"2024-04-14T15:20:02.947239+0000","last_became_peered":"2024-04-14T15:20:02.947239+0000","last_unstale":"2024-04-14T15:20:13.385721+0000","last_undegraded":"2024-04-14T15:20:13.385721+0000","last_fullsized":"2024-04-14T15:20:13.385721+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":491,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:01.924260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:01.924260+0000","last_clean_scrub_stamp":"2024-04-14T15:20:01.924260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:27.776702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"305.1c","version":"0'0","reported_seq":22,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.937113+0000","last_change":"2024-04-14T15:20:03.133975+0000","last_active":"2024-04-14T15:20:12.937113+0000","last_peered":"2024-04-14T15:20:12.937113+0000","last_clean":"2024-04-14T15:20:12.937113+0000","last_became_active":"2024-04-14T15:20:03.133289+0000","last_became_peered":"2024-04-14T15:20:03.133289+0000","last_unstale":"2024-04-14T15:20:12.937113+0000","last_undegraded":"2024-04-14T15:20:12.937113+0000","last_fullsized":"2024-04-14T15:20:12.937113+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":491,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:01.924260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:01.924260+0000","last_clean_scrub_stamp":"2024-04-14T15:20:01.924260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:22.530235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"305.1b","version":"494'8","reported_seq":37,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.936941+0000","last_change":"2024-04-14T15:20:05.749307+0000","last_active":"2024-04-14T15:20:12.936941+0000","last_peered":"2024-04-14T15:20:12.936941+0000","last_clean":"2024-04-14T15:20:12.936941+0000","last_became_active":"2024-04-14T15:20:03.133608+0000","last_became_peered":"2024-04-14T15:20:03.133608+0000","last_unstale":"2024-04-14T15:20:12.936941+0000","last_undegraded":"2024-04-14T15:20:12.936941+0000","last_fullsized":"2024-04-14T15:20:12.936941+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":491,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"494'8","last_scrub_stamp":"2024-04-14T15:20:05.749212+0000","last_deep_scrub":"494'8","last_deep_scrub_stamp":"2024-04-14T15:20:05.749212+0000","last_clean_scrub_stamp":"2024-04-14T15:20:05.749212+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:11.925396+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"305.1a","version":"0'0","reported_seq":22,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.934407+0000","last_change":"2024-04-14T15:20:02.947923+0000","last_active":"2024-04-14T15:20:12.934407+0000","last_peered":"2024-04-14T15:20:12.934407+0000","last_clean":"2024-04-14T15:20:12.934407+0000","last_became_active":"2024-04-14T15:20:02.947538+0000","last_became_peered":"2024-04-14T15:20:02.947538+0000","last_unstale":"2024-04-14T15:20:12.934407+0000","last_undegraded":"2024-04-14T15:20:12.934407+0000","last_fullsized":"2024-04-14T15:20:12.934407+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":491,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:01.924260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:01.924260+0000","last_clean_scrub_stamp":"2024-04-14T15:20:01.924260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:15.216175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"305.19","version":"0'0","reported_seq":22,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.337291+0000","last_change":"2024-04-14T15:20:02.952176+0000","last_active":"2024-04-14T15:20:13.337291+0000","last_peered":"2024-04-14T15:20:13.337291+0000","last_clean":"2024-04-14T15:20:13.337291+0000","last_became_active":"2024-04-14T15:20:02.951708+0000","last_became_peered":"2024-04-14T15:20:02.951708+0000","last_unstale":"2024-04-14T15:20:13.337291+0000","last_undegraded":"2024-04-14T15:20:13.337291+0000","last_fullsized":"2024-04-14T15:20:13.337291+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":491,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:01.924260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:01.924260+0000","last_clean_scrub_stamp":"2024-04-14T15:20:01.924260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:21.238359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"305.18","version":"0'0","reported_seq":22,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.933053+0000","last_change":"2024-04-14T15:20:02.950063+0000","last_active":"2024-04-14T15:20:12.933053+0000","last_peered":"2024-04-14T15:20:12.933053+0000","last_clean":"2024-04-14T15:20:12.933053+0000","last_became_active":"2024-04-14T15:20:02.949718+0000","last_became_peered":"2024-04-14T15:20:02.949718+0000","last_unstale":"2024-04-14T15:20:12.933053+0000","last_undegraded":"2024-04-14T15:20:12.933053+0000","last_fullsized":"2024-04-14T15:20:12.933053+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":491,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:01.924260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:01.924260+0000","last_clean_scrub_stamp":"2024-04-14T15:20:01.924260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:21.185151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"305.17","version":"0'0","reported_seq":22,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.936429+0000","last_change":"2024-04-14T15:20:02.942721+0000","last_active":"2024-04-14T15:20:12.936429+0000","last_peered":"2024-04-14T15:20:12.936429+0000","last_clean":"2024-04-14T15:20:12.936429+0000","last_became_active":"2024-04-14T15:20:02.942445+0000","last_became_peered":"2024-04-14T15:20:02.942445+0000","last_unstale":"2024-04-14T15:20:12.936429+0000","last_undegraded":"2024-04-14T15:20:12.936429+0000","last_fullsized":"2024-04-14T15:20:12.936429+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":491,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:01.924260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:01.924260+0000","last_clean_scrub_stamp":"2024-04-14T15:20:01.924260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:13.411034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"305.16","version":"0'0","reported_seq":22,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.937196+0000","last_change":"2024-04-14T15:20:02.947268+0000","last_active":"2024-04-14T15:20:12.937196+0000","last_peered":"2024-04-14T15:20:12.937196+0000","last_clean":"2024-04-14T15:20:12.937196+0000","last_became_active":"2024-04-14T15:20:02.946294+0000","last_became_peered":"2024-04-14T15:20:02.946294+0000","last_unstale":"2024-04-14T15:20:12.937196+0000","last_undegraded":"2024-04-14T15:20:12.937196+0000","last_fullsized":"2024-04-14T15:20:12.937196+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":491,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:01.924260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:01.924260+0000","last_clean_scrub_stamp":"2024-04-14T15:20:01.924260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:23.080767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"305.15","version":"0'0","reported_seq":22,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.337891+0000","last_change":"2024-04-14T15:20:02.950010+0000","last_active":"2024-04-14T15:20:13.337891+0000","last_peered":"2024-04-14T15:20:13.337891+0000","last_clean":"2024-04-14T15:20:13.337891+0000","last_became_active":"2024-04-14T15:20:02.949493+0000","last_became_peered":"2024-04-14T15:20:02.949493+0000","last_unstale":"2024-04-14T15:20:13.337891+0000","last_undegraded":"2024-04-14T15:20:13.337891+0000","last_fullsized":"2024-04-14T15:20:13.337891+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":491,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:01.924260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:01.924260+0000","last_clean_scrub_stamp":"2024-04-14T15:20:01.924260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:13.283118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"305.14","version":"0'0","reported_seq":22,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.933570+0000","last_change":"2024-04-14T15:20:02.950305+0000","last_active":"2024-04-14T15:20:12.933570+0000","last_peered":"2024-04-14T15:20:12.933570+0000","last_clean":"2024-04-14T15:20:12.933570+0000","last_became_active":"2024-04-14T15:20:02.949941+0000","last_became_peered":"2024-04-14T15:20:02.949941+0000","last_unstale":"2024-04-14T15:20:12.933570+0000","last_undegraded":"2024-04-14T15:20:12.933570+0000","last_fullsized":"2024-04-14T15:20:12.933570+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":491,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:01.924260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:01.924260+0000","last_clean_scrub_stamp":"2024-04-14T15:20:01.924260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:27.594649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"305.13","version":"0'0","reported_seq":22,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.934531+0000","last_change":"2024-04-14T15:20:02.949896+0000","last_active":"2024-04-14T15:20:12.934531+0000","last_peered":"2024-04-14T15:20:12.934531+0000","last_clean":"2024-04-14T15:20:12.934531+0000","last_became_active":"2024-04-14T15:20:02.948872+0000","last_became_peered":"2024-04-14T15:20:02.948872+0000","last_unstale":"2024-04-14T15:20:12.934531+0000","last_undegraded":"2024-04-14T15:20:12.934531+0000","last_fullsized":"2024-04-14T15:20:12.934531+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":491,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:01.924260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:01.924260+0000","last_clean_scrub_stamp":"2024-04-14T15:20:01.924260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:20.818480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"305.12","version":"0'0","reported_seq":22,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.934450+0000","last_change":"2024-04-14T15:20:02.948366+0000","last_active":"2024-04-14T15:20:12.934450+0000","last_peered":"2024-04-14T15:20:12.934450+0000","last_clean":"2024-04-14T15:20:12.934450+0000","last_became_active":"2024-04-14T15:20:02.947018+0000","last_became_peered":"2024-04-14T15:20:02.947018+0000","last_unstale":"2024-04-14T15:20:12.934450+0000","last_undegraded":"2024-04-14T15:20:12.934450+0000","last_fullsized":"2024-04-14T15:20:12.934450+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":491,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:01.924260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:01.924260+0000","last_clean_scrub_stamp":"2024-04-14T15:20:01.924260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:30.374671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"305.11","version":"0'0","reported_seq":22,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.337353+0000","last_change":"2024-04-14T15:20:02.951689+0000","last_active":"2024-04-14T15:20:13.337353+0000","last_peered":"2024-04-14T15:20:13.337353+0000","last_clean":"2024-04-14T15:20:13.337353+0000","last_became_active":"2024-04-14T15:20:02.951414+0000","last_became_peered":"2024-04-14T15:20:02.951414+0000","last_unstale":"2024-04-14T15:20:13.337353+0000","last_undegraded":"2024-04-14T15:20:13.337353+0000","last_fullsized":"2024-04-14T15:20:13.337353+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":491,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:01.924260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:01.924260+0000","last_clean_scrub_stamp":"2024-04-14T15:20:01.924260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:20.576017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"305.10","version":"0'0","reported_seq":22,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.385126+0000","last_change":"2024-04-14T15:20:02.942638+0000","last_active":"2024-04-14T15:20:13.385126+0000","last_peered":"2024-04-14T15:20:13.385126+0000","last_clean":"2024-04-14T15:20:13.385126+0000","last_became_active":"2024-04-14T15:20:02.942266+0000","last_became_peered":"2024-04-14T15:20:02.942266+0000","last_unstale":"2024-04-14T15:20:13.385126+0000","last_undegraded":"2024-04-14T15:20:13.385126+0000","last_fullsized":"2024-04-14T15:20:13.385126+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":491,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:01.924260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:01.924260+0000","last_clean_scrub_stamp":"2024-04-14T15:20:01.924260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:30.194242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"305.f","version":"0'0","reported_seq":22,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.748217+0000","last_change":"2024-04-14T15:20:02.942184+0000","last_active":"2024-04-14T15:20:13.748217+0000","last_peered":"2024-04-14T15:20:13.748217+0000","last_clean":"2024-04-14T15:20:13.748217+0000","last_became_active":"2024-04-14T15:20:02.941829+0000","last_became_peered":"2024-04-14T15:20:02.941829+0000","last_unstale":"2024-04-14T15:20:13.748217+0000","last_undegraded":"2024-04-14T15:20:13.748217+0000","last_fullsized":"2024-04-14T15:20:13.748217+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":491,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:01.924260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:01.924260+0000","last_clean_scrub_stamp":"2024-04-14T15:20:01.924260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:05.544206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"305.e","version":"0'0","reported_seq":22,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.661437+0000","last_change":"2024-04-14T15:20:02.941298+0000","last_active":"2024-04-14T15:20:13.661437+0000","last_peered":"2024-04-14T15:20:13.661437+0000","last_clean":"2024-04-14T15:20:13.661437+0000","last_became_active":"2024-04-14T15:20:02.940646+0000","last_became_peered":"2024-04-14T15:20:02.940646+0000","last_unstale":"2024-04-14T15:20:13.661437+0000","last_undegraded":"2024-04-14T15:20:13.661437+0000","last_fullsized":"2024-04-14T15:20:13.661437+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":491,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:01.924260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:01.924260+0000","last_clean_scrub_stamp":"2024-04-14T15:20:01.924260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:11.436940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"305.d","version":"0'0","reported_seq":22,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.937001+0000","last_change":"2024-04-14T15:20:02.948370+0000","last_active":"2024-04-14T15:20:12.937001+0000","last_peered":"2024-04-14T15:20:12.937001+0000","last_clean":"2024-04-14T15:20:12.937001+0000","last_became_active":"2024-04-14T15:20:02.948079+0000","last_became_peered":"2024-04-14T15:20:02.948079+0000","last_unstale":"2024-04-14T15:20:12.937001+0000","last_undegraded":"2024-04-14T15:20:12.937001+0000","last_fullsized":"2024-04-14T15:20:12.937001+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":491,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:01.924260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:01.924260+0000","last_clean_scrub_stamp":"2024-04-14T15:20:01.924260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:16.452691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"305.c","version":"0'0","reported_seq":22,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.747674+0000","last_change":"2024-04-14T15:20:02.949545+0000","last_active":"2024-04-14T15:20:13.747674+0000","last_peered":"2024-04-14T15:20:13.747674+0000","last_clean":"2024-04-14T15:20:13.747674+0000","last_became_active":"2024-04-14T15:20:02.949211+0000","last_became_peered":"2024-04-14T15:20:02.949211+0000","last_unstale":"2024-04-14T15:20:13.747674+0000","last_undegraded":"2024-04-14T15:20:13.747674+0000","last_fullsized":"2024-04-14T15:20:13.747674+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":491,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:01.924260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:01.924260+0000","last_clean_scrub_stamp":"2024-04-14T15:20:01.924260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:04.301927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"305.b","version":"0'0","reported_seq":22,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.660137+0000","last_change":"2024-04-14T15:20:02.946225+0000","last_active":"2024-04-14T15:20:13.660137+0000","last_peered":"2024-04-14T15:20:13.660137+0000","last_clean":"2024-04-14T15:20:13.660137+0000","last_became_active":"2024-04-14T15:20:02.945897+0000","last_became_peered":"2024-04-14T15:20:02.945897+0000","last_unstale":"2024-04-14T15:20:13.660137+0000","last_undegraded":"2024-04-14T15:20:13.660137+0000","last_fullsized":"2024-04-14T15:20:13.660137+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":491,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:01.924260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:01.924260+0000","last_clean_scrub_stamp":"2024-04-14T15:20:01.924260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:21.484637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"305.a","version":"0'0","reported_seq":22,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.385405+0000","last_change":"2024-04-14T15:20:02.949734+0000","last_active":"2024-04-14T15:20:13.385405+0000","last_peered":"2024-04-14T15:20:13.385405+0000","last_clean":"2024-04-14T15:20:13.385405+0000","last_became_active":"2024-04-14T15:20:02.949054+0000","last_became_peered":"2024-04-14T15:20:02.949054+0000","last_unstale":"2024-04-14T15:20:13.385405+0000","last_undegraded":"2024-04-14T15:20:13.385405+0000","last_fullsized":"2024-04-14T15:20:13.385405+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":491,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:01.924260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:01.924260+0000","last_clean_scrub_stamp":"2024-04-14T15:20:01.924260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:21.940835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"305.9","version":"0'0","reported_seq":22,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.337387+0000","last_change":"2024-04-14T15:20:02.951922+0000","last_active":"2024-04-14T15:20:13.337387+0000","last_peered":"2024-04-14T15:20:13.337387+0000","last_clean":"2024-04-14T15:20:13.337387+0000","last_became_active":"2024-04-14T15:20:02.951432+0000","last_became_peered":"2024-04-14T15:20:02.951432+0000","last_unstale":"2024-04-14T15:20:13.337387+0000","last_undegraded":"2024-04-14T15:20:13.337387+0000","last_fullsized":"2024-04-14T15:20:13.337387+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":491,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:01.924260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:01.924260+0000","last_clean_scrub_stamp":"2024-04-14T15:20:01.924260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:23.689539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"305.8","version":"0'0","reported_seq":22,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.337309+0000","last_change":"2024-04-14T15:20:03.135145+0000","last_active":"2024-04-14T15:20:13.337309+0000","last_peered":"2024-04-14T15:20:13.337309+0000","last_clean":"2024-04-14T15:20:13.337309+0000","last_became_active":"2024-04-14T15:20:03.134803+0000","last_became_peered":"2024-04-14T15:20:03.134803+0000","last_unstale":"2024-04-14T15:20:13.337309+0000","last_undegraded":"2024-04-14T15:20:13.337309+0000","last_fullsized":"2024-04-14T15:20:13.337309+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":491,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:01.924260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:01.924260+0000","last_clean_scrub_stamp":"2024-04-14T15:20:01.924260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:08.908042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"305.7","version":"0'0","reported_seq":22,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.661089+0000","last_change":"2024-04-14T15:20:03.133481+0000","last_active":"2024-04-14T15:20:13.661089+0000","last_peered":"2024-04-14T15:20:13.661089+0000","last_clean":"2024-04-14T15:20:13.661089+0000","last_became_active":"2024-04-14T15:20:03.133161+0000","last_became_peered":"2024-04-14T15:20:03.133161+0000","last_unstale":"2024-04-14T15:20:13.661089+0000","last_undegraded":"2024-04-14T15:20:13.661089+0000","last_fullsized":"2024-04-14T15:20:13.661089+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":491,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:01.924260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:01.924260+0000","last_clean_scrub_stamp":"2024-04-14T15:20:01.924260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:17.719536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"305.6","version":"0'0","reported_seq":22,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.935756+0000","last_change":"2024-04-14T15:20:03.134006+0000","last_active":"2024-04-14T15:20:12.935756+0000","last_peered":"2024-04-14T15:20:12.935756+0000","last_clean":"2024-04-14T15:20:12.935756+0000","last_became_active":"2024-04-14T15:20:03.133464+0000","last_became_peered":"2024-04-14T15:20:03.133464+0000","last_unstale":"2024-04-14T15:20:12.935756+0000","last_undegraded":"2024-04-14T15:20:12.935756+0000","last_fullsized":"2024-04-14T15:20:12.935756+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":491,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:01.924260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:01.924260+0000","last_clean_scrub_stamp":"2024-04-14T15:20:01.924260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:24.691218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"305.5","version":"0'0","reported_seq":22,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.385758+0000","last_change":"2024-04-14T15:20:03.133538+0000","last_active":"2024-04-14T15:20:13.385758+0000","last_peered":"2024-04-14T15:20:13.385758+0000","last_clean":"2024-04-14T15:20:13.385758+0000","last_became_active":"2024-04-14T15:20:03.133205+0000","last_became_peered":"2024-04-14T15:20:03.133205+0000","last_unstale":"2024-04-14T15:20:13.385758+0000","last_undegraded":"2024-04-14T15:20:13.385758+0000","last_fullsized":"2024-04-14T15:20:13.385758+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":491,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:01.924260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:01.924260+0000","last_clean_scrub_stamp":"2024-04-14T15:20:01.924260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:09.430072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"305.4","version":"0'0","reported_seq":22,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.935406+0000","last_change":"2024-04-14T15:20:03.133642+0000","last_active":"2024-04-14T15:20:12.935406+0000","last_peered":"2024-04-14T15:20:12.935406+0000","last_clean":"2024-04-14T15:20:12.935406+0000","last_became_active":"2024-04-14T15:20:03.133227+0000","last_became_peered":"2024-04-14T15:20:03.133227+0000","last_unstale":"2024-04-14T15:20:12.935406+0000","last_undegraded":"2024-04-14T15:20:12.935406+0000","last_fullsized":"2024-04-14T15:20:12.935406+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":491,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:01.924260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:01.924260+0000","last_clean_scrub_stamp":"2024-04-14T15:20:01.924260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:25.264944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"305.3","version":"0'0","reported_seq":22,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.934339+0000","last_change":"2024-04-14T15:20:02.943933+0000","last_active":"2024-04-14T15:20:12.934339+0000","last_peered":"2024-04-14T15:20:12.934339+0000","last_clean":"2024-04-14T15:20:12.934339+0000","last_became_active":"2024-04-14T15:20:02.942285+0000","last_became_peered":"2024-04-14T15:20:02.942285+0000","last_unstale":"2024-04-14T15:20:12.934339+0000","last_undegraded":"2024-04-14T15:20:12.934339+0000","last_fullsized":"2024-04-14T15:20:12.934339+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":491,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:01.924260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:01.924260+0000","last_clean_scrub_stamp":"2024-04-14T15:20:01.924260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:28.174936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"305.2","version":"0'0","reported_seq":22,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.936123+0000","last_change":"2024-04-14T15:20:03.133673+0000","last_active":"2024-04-14T15:20:12.936123+0000","last_peered":"2024-04-14T15:20:12.936123+0000","last_clean":"2024-04-14T15:20:12.936123+0000","last_became_active":"2024-04-14T15:20:03.133369+0000","last_became_peered":"2024-04-14T15:20:03.133369+0000","last_unstale":"2024-04-14T15:20:12.936123+0000","last_undegraded":"2024-04-14T15:20:12.936123+0000","last_fullsized":"2024-04-14T15:20:12.936123+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":491,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:01.924260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:01.924260+0000","last_clean_scrub_stamp":"2024-04-14T15:20:01.924260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:26.895022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"305.1","version":"0'0","reported_seq":22,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.933238+0000","last_change":"2024-04-14T15:20:03.133617+0000","last_active":"2024-04-14T15:20:12.933238+0000","last_peered":"2024-04-14T15:20:12.933238+0000","last_clean":"2024-04-14T15:20:12.933238+0000","last_became_active":"2024-04-14T15:20:03.133138+0000","last_became_peered":"2024-04-14T15:20:03.133138+0000","last_unstale":"2024-04-14T15:20:12.933238+0000","last_undegraded":"2024-04-14T15:20:12.933238+0000","last_fullsized":"2024-04-14T15:20:12.933238+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":491,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:01.924260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:01.924260+0000","last_clean_scrub_stamp":"2024-04-14T15:20:01.924260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:08.537559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"305.0","version":"0'0","reported_seq":22,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.933053+0000","last_change":"2024-04-14T15:20:03.133479+0000","last_active":"2024-04-14T15:20:12.933053+0000","last_peered":"2024-04-14T15:20:12.933053+0000","last_clean":"2024-04-14T15:20:12.933053+0000","last_became_active":"2024-04-14T15:20:03.133219+0000","last_became_peered":"2024-04-14T15:20:03.133219+0000","last_unstale":"2024-04-14T15:20:12.933053+0000","last_undegraded":"2024-04-14T15:20:12.933053+0000","last_fullsized":"2024-04-14T15:20:12.933053+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":491,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:01.924260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:01.924260+0000","last_clean_scrub_stamp":"2024-04-14T15:20:01.924260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:17.569224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"110.11","version":"403'41","reported_seq":1015,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.747133+0000","last_change":"2024-04-14T15:20:00.830028+0000","last_active":"2024-04-14T15:20:13.747133+0000","last_peered":"2024-04-14T15:20:13.747133+0000","last_clean":"2024-04-14T15:20:13.747133+0000","last_became_active":"2024-04-14T15:13:09.936619+0000","last_became_peered":"2024-04-14T15:13:09.936619+0000","last_unstale":"2024-04-14T15:20:13.747133+0000","last_undegraded":"2024-04-14T15:20:13.747133+0000","last_fullsized":"2024-04-14T15:20:13.747133+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"403'41","last_scrub_stamp":"2024-04-14T15:20:00.829932+0000","last_deep_scrub":"403'41","last_deep_scrub_stamp":"2024-04-14T15:17:56.911004+0000","last_clean_scrub_stamp":"2024-04-14T15:20:00.829932+0000","objects_scrubbed":2,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:03.495441+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00061958300000000005,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"110.10","version":"253'158","reported_seq":1089,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.933300+0000","last_change":"2024-04-14T15:18:54.158668+0000","last_active":"2024-04-14T15:20:12.933300+0000","last_peered":"2024-04-14T15:20:12.933300+0000","last_clean":"2024-04-14T15:20:12.933300+0000","last_became_active":"2024-04-14T15:13:09.928915+0000","last_became_peered":"2024-04-14T15:13:09.928915+0000","last_unstale":"2024-04-14T15:20:12.933300+0000","last_undegraded":"2024-04-14T15:20:12.933300+0000","last_fullsized":"2024-04-14T15:20:12.933300+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"253'158","last_scrub_stamp":"2024-04-14T15:17:34.291854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:34.291854+0000","objects_scrubbed":10,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:14.561216+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00045995200000000002,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":47,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"110.13","version":"477'53","reported_seq":1019,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.385821+0000","last_change":"2024-04-14T15:18:54.157558+0000","last_active":"2024-04-14T15:20:13.385821+0000","last_peered":"2024-04-14T15:20:13.385821+0000","last_clean":"2024-04-14T15:20:13.385821+0000","last_became_active":"2024-04-14T15:13:09.918152+0000","last_became_peered":"2024-04-14T15:13:09.918152+0000","last_unstale":"2024-04-14T15:20:13.385821+0000","last_undegraded":"2024-04-14T15:20:13.385821+0000","last_fullsized":"2024-04-14T15:20:13.385821+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'39","last_scrub_stamp":"2024-04-14T15:17:44.873766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:44.873766+0000","objects_scrubbed":3,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:15.826615+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001942764,"stat_sum":{"num_bytes":57,"num_objects":7,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"110.12","version":"489'77","reported_seq":1050,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.385586+0000","last_change":"2024-04-14T15:19:47.921780+0000","last_active":"2024-04-14T15:20:13.385586+0000","last_peered":"2024-04-14T15:20:13.385586+0000","last_clean":"2024-04-14T15:20:13.385586+0000","last_became_active":"2024-04-14T15:13:09.939566+0000","last_became_peered":"2024-04-14T15:13:09.939566+0000","last_unstale":"2024-04-14T15:20:13.385586+0000","last_undegraded":"2024-04-14T15:20:13.385586+0000","last_fullsized":"2024-04-14T15:20:13.385586+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"409'70","last_scrub_stamp":"2024-04-14T15:19:47.921684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:19:47.921684+0000","objects_scrubbed":9,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:12.085817+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.000970527,"stat_sum":{"num_bytes":89,"num_objects":11,"num_object_clones":1,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":27,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"110.15","version":"190'34","reported_seq":1001,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.337827+0000","last_change":"2024-04-14T15:18:54.155028+0000","last_active":"2024-04-14T15:20:13.337827+0000","last_peered":"2024-04-14T15:20:13.337827+0000","last_clean":"2024-04-14T15:20:13.337827+0000","last_became_active":"2024-04-14T15:13:09.960121+0000","last_became_peered":"2024-04-14T15:13:09.960121+0000","last_unstale":"2024-04-14T15:20:13.337827+0000","last_undegraded":"2024-04-14T15:20:13.337827+0000","last_fullsized":"2024-04-14T15:20:13.337827+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'34","last_scrub_stamp":"2024-04-14T15:18:36.013135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:18:36.013135+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:17.378280+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0029213049999999999,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"110.14","version":"355'87","reported_seq":1048,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.337849+0000","last_change":"2024-04-14T15:18:54.151534+0000","last_active":"2024-04-14T15:20:13.337849+0000","last_peered":"2024-04-14T15:20:13.337849+0000","last_clean":"2024-04-14T15:20:13.337849+0000","last_became_active":"2024-04-14T15:13:09.899561+0000","last_became_peered":"2024-04-14T15:13:09.899561+0000","last_unstale":"2024-04-14T15:20:13.337849+0000","last_undegraded":"2024-04-14T15:20:13.337849+0000","last_fullsized":"2024-04-14T15:20:13.337849+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'87","last_scrub_stamp":"2024-04-14T15:18:31.117033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:18:31.117033+0000","objects_scrubbed":7,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:17.378299+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0014558730000000001,"stat_sum":{"num_bytes":71,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"110.17","version":"456'12","reported_seq":994,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.338174+0000","last_change":"2024-04-14T15:19:48.225230+0000","last_active":"2024-04-14T15:20:13.338174+0000","last_peered":"2024-04-14T15:20:13.338174+0000","last_clean":"2024-04-14T15:20:13.338174+0000","last_became_active":"2024-04-14T15:13:09.898521+0000","last_became_peered":"2024-04-14T15:13:09.898521+0000","last_unstale":"2024-04-14T15:20:13.338174+0000","last_undegraded":"2024-04-14T15:20:13.338174+0000","last_fullsized":"2024-04-14T15:20:13.338174+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'12","last_scrub_stamp":"2024-04-14T15:19:48.225130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:19:48.225130+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:56.459153+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0018681640000000001,"stat_sum":{"num_bytes":24,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"110.16","version":"355'35","reported_seq":835,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.937507+0000","last_change":"2024-04-14T15:18:54.160925+0000","last_active":"2024-04-14T15:20:12.937507+0000","last_peered":"2024-04-14T15:20:12.937507+0000","last_clean":"2024-04-14T15:20:12.937507+0000","last_became_active":"2024-04-14T15:15:21.889783+0000","last_became_peered":"2024-04-14T15:15:21.889783+0000","last_unstale":"2024-04-14T15:20:12.937507+0000","last_undegraded":"2024-04-14T15:20:12.937507+0000","last_fullsized":"2024-04-14T15:20:12.937507+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'35","last_scrub_stamp":"2024-04-14T15:17:34.097563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:34.097563+0000","objects_scrubbed":3,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000499903,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"110.19","version":"494'104","reported_seq":1069,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.934266+0000","last_change":"2024-04-14T15:19:12.058233+0000","last_active":"2024-04-14T15:20:12.934266+0000","last_peered":"2024-04-14T15:20:12.934266+0000","last_clean":"2024-04-14T15:20:12.934266+0000","last_became_active":"2024-04-14T15:13:09.998139+0000","last_became_peered":"2024-04-14T15:13:09.998139+0000","last_unstale":"2024-04-14T15:20:12.934266+0000","last_undegraded":"2024-04-14T15:20:12.934266+0000","last_fullsized":"2024-04-14T15:20:12.934266+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'85","last_scrub_stamp":"2024-04-14T15:19:12.058127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:19:12.058127+0000","objects_scrubbed":5,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:16.119461+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0.0012903700000000001,"stat_sum":{"num_bytes":85,"num_objects":8,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":35,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"110.18","version":"387'94","reported_seq":1053,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.934173+0000","last_change":"2024-04-14T15:18:54.150377+0000","last_active":"2024-04-14T15:20:12.934173+0000","last_peered":"2024-04-14T15:20:12.934173+0000","last_clean":"2024-04-14T15:20:12.934173+0000","last_became_active":"2024-04-14T15:13:09.998803+0000","last_became_peered":"2024-04-14T15:13:09.998803+0000","last_unstale":"2024-04-14T15:20:12.934173+0000","last_undegraded":"2024-04-14T15:20:12.934173+0000","last_fullsized":"2024-04-14T15:20:12.934173+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'94","last_scrub_stamp":"2024-04-14T15:17:36.005598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:36.005598+0000","objects_scrubbed":8,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:15.771434+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0013897930000000001,"stat_sum":{"num_bytes":65,"num_objects":8,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":29,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"110.1b","version":"494'108","reported_seq":1066,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.385758+0000","last_change":"2024-04-14T15:18:54.156913+0000","last_active":"2024-04-14T15:20:13.385758+0000","last_peered":"2024-04-14T15:20:13.385758+0000","last_clean":"2024-04-14T15:20:13.385758+0000","last_became_active":"2024-04-14T15:13:09.946919+0000","last_became_peered":"2024-04-14T15:13:09.946919+0000","last_unstale":"2024-04-14T15:20:13.385758+0000","last_undegraded":"2024-04-14T15:20:13.385758+0000","last_fullsized":"2024-04-14T15:20:13.385758+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'91","last_scrub_stamp":"2024-04-14T15:17:47.889364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:47.889364+0000","objects_scrubbed":6,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:15.826598+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001643863,"stat_sum":{"num_bytes":103,"num_objects":10,"num_object_clones":2,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":8,"num_read_kb":0,"num_write":35,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"110.1a","version":"387'39","reported_seq":1010,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.747280+0000","last_change":"2024-04-14T15:18:54.160775+0000","last_active":"2024-04-14T15:20:13.747280+0000","last_peered":"2024-04-14T15:20:13.747280+0000","last_clean":"2024-04-14T15:20:13.747280+0000","last_became_active":"2024-04-14T15:13:09.960319+0000","last_became_peered":"2024-04-14T15:13:09.960319+0000","last_unstale":"2024-04-14T15:20:13.747280+0000","last_undegraded":"2024-04-14T15:20:13.747280+0000","last_fullsized":"2024-04-14T15:20:13.747280+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'39","last_scrub_stamp":"2024-04-14T15:17:53.871590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:53.871590+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0018412890000000001,"stat_sum":{"num_bytes":62,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"110.1d","version":"225'85","reported_seq":1045,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.660567+0000","last_change":"2024-04-14T15:18:54.151417+0000","last_active":"2024-04-14T15:20:13.660567+0000","last_peered":"2024-04-14T15:20:13.660567+0000","last_clean":"2024-04-14T15:20:13.660567+0000","last_became_active":"2024-04-14T15:13:09.868082+0000","last_became_peered":"2024-04-14T15:13:09.868082+0000","last_unstale":"2024-04-14T15:20:13.660567+0000","last_undegraded":"2024-04-14T15:20:13.660567+0000","last_fullsized":"2024-04-14T15:20:13.660567+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"225'85","last_scrub_stamp":"2024-04-14T15:17:36.419088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:36.419088+0000","objects_scrubbed":5,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:18.657771+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0010543219999999999,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"110.1c","version":"430'95","reported_seq":1058,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.385720+0000","last_change":"2024-04-14T15:19:10.839468+0000","last_active":"2024-04-14T15:20:13.385720+0000","last_peered":"2024-04-14T15:20:13.385720+0000","last_clean":"2024-04-14T15:20:13.385720+0000","last_became_active":"2024-04-14T15:13:09.954091+0000","last_became_peered":"2024-04-14T15:13:09.954091+0000","last_unstale":"2024-04-14T15:20:13.385720+0000","last_undegraded":"2024-04-14T15:20:13.385720+0000","last_fullsized":"2024-04-14T15:20:13.385720+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'95","last_scrub_stamp":"2024-04-14T15:19:10.839339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:19:10.839339+0000","objects_scrubbed":7,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:25.832932+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0011789070000000001,"stat_sum":{"num_bytes":65,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"110.1f","version":"466'27","reported_seq":1050,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.936568+0000","last_change":"2024-04-14T15:19:25.078771+0000","last_active":"2024-04-14T15:20:12.936568+0000","last_peered":"2024-04-14T15:20:12.936568+0000","last_clean":"2024-04-14T15:20:12.936568+0000","last_became_active":"2024-04-14T15:13:09.999629+0000","last_became_peered":"2024-04-14T15:13:09.999629+0000","last_unstale":"2024-04-14T15:20:12.936568+0000","last_undegraded":"2024-04-14T15:20:12.936568+0000","last_fullsized":"2024-04-14T15:20:12.936568+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"466'27","last_scrub_stamp":"2024-04-14T15:19:25.078678+0000","last_deep_scrub":"466'27","last_deep_scrub_stamp":"2024-04-14T15:19:25.078678+0000","last_clean_scrub_stamp":"2024-04-14T15:19:25.078678+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:50.898358+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.001392454,"stat_sum":{"num_bytes":4194315,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":8,"num_write_kb":8198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"110.1e","version":"355'23","reported_seq":995,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.338107+0000","last_change":"2024-04-14T15:19:17.364851+0000","last_active":"2024-04-14T15:20:13.338107+0000","last_peered":"2024-04-14T15:20:13.338107+0000","last_clean":"2024-04-14T15:20:13.338107+0000","last_became_active":"2024-04-14T15:13:09.955932+0000","last_became_peered":"2024-04-14T15:13:09.955932+0000","last_unstale":"2024-04-14T15:20:13.338107+0000","last_undegraded":"2024-04-14T15:20:13.338107+0000","last_fullsized":"2024-04-14T15:20:13.338107+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'23","last_scrub_stamp":"2024-04-14T15:19:17.364769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:19:17.364769+0000","objects_scrubbed":2,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:23.437044+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0019783700000000001,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":8,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"110.1","version":"406'32","reported_seq":1007,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.659626+0000","last_change":"2024-04-14T15:18:54.150210+0000","last_active":"2024-04-14T15:20:13.659626+0000","last_peered":"2024-04-14T15:20:13.659626+0000","last_clean":"2024-04-14T15:20:13.659626+0000","last_became_active":"2024-04-14T15:13:09.929562+0000","last_became_peered":"2024-04-14T15:13:09.929562+0000","last_unstale":"2024-04-14T15:20:13.659626+0000","last_undegraded":"2024-04-14T15:20:13.659626+0000","last_fullsized":"2024-04-14T15:20:13.659626+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"406'32","last_scrub_stamp":"2024-04-14T15:18:38.685469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:18:38.685469+0000","objects_scrubbed":7,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:18.657914+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00089761599999999995,"stat_sum":{"num_bytes":4000050,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":15,"num_write_kb":3921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"110.0","version":"219'85","reported_seq":1040,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.933389+0000","last_change":"2024-04-14T15:18:54.159255+0000","last_active":"2024-04-14T15:20:12.933389+0000","last_peered":"2024-04-14T15:20:12.933389+0000","last_clean":"2024-04-14T15:20:12.933389+0000","last_became_active":"2024-04-14T15:13:09.856407+0000","last_became_peered":"2024-04-14T15:13:09.856407+0000","last_unstale":"2024-04-14T15:20:12.933389+0000","last_undegraded":"2024-04-14T15:20:12.933389+0000","last_fullsized":"2024-04-14T15:20:12.933389+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"219'85","last_scrub_stamp":"2024-04-14T15:17:51.438149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:51.438149+0000","objects_scrubbed":5,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:14.561354+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00052820199999999997,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"110.3","version":"464'56","reported_seq":1056,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.934977+0000","last_change":"2024-04-14T15:19:35.504089+0000","last_active":"2024-04-14T15:20:12.934977+0000","last_peered":"2024-04-14T15:20:12.934977+0000","last_clean":"2024-04-14T15:20:12.934977+0000","last_became_active":"2024-04-14T15:19:35.486976+0000","last_became_peered":"2024-04-14T15:19:35.486976+0000","last_unstale":"2024-04-14T15:20:12.934977+0000","last_undegraded":"2024-04-14T15:20:12.934977+0000","last_fullsized":"2024-04-14T15:20:12.934977+0000","mapping_epoch":470,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":471,"parent":"0.0","parent_split_bits":0,"last_scrub":"464'56","last_scrub_stamp":"2024-04-14T15:19:17.063716+0000","last_deep_scrub":"464'56","last_deep_scrub_stamp":"2024-04-14T15:19:17.063716+0000","last_clean_scrub_stamp":"2024-04-14T15:19:17.063716+0000","objects_scrubbed":4,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:20.334627+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0019291829999999999,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":41,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"110.2","version":"240'55","reported_seq":1022,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.934646+0000","last_change":"2024-04-14T15:18:54.152834+0000","last_active":"2024-04-14T15:20:12.934646+0000","last_peered":"2024-04-14T15:20:12.934646+0000","last_clean":"2024-04-14T15:20:12.934646+0000","last_became_active":"2024-04-14T15:13:09.957187+0000","last_became_peered":"2024-04-14T15:13:09.957187+0000","last_unstale":"2024-04-14T15:20:12.934646+0000","last_undegraded":"2024-04-14T15:20:12.934646+0000","last_fullsized":"2024-04-14T15:20:12.934646+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'55","last_scrub_stamp":"2024-04-14T15:17:37.941589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:37.941589+0000","objects_scrubbed":6,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:15.771415+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0020075969999999999,"stat_sum":{"num_bytes":49,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":18,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"110.5","version":"396'64","reported_seq":1023,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.935157+0000","last_change":"2024-04-14T15:18:54.162113+0000","last_active":"2024-04-14T15:20:12.935157+0000","last_peered":"2024-04-14T15:20:12.935157+0000","last_clean":"2024-04-14T15:20:12.935157+0000","last_became_active":"2024-04-14T15:13:09.853689+0000","last_became_peered":"2024-04-14T15:13:09.853689+0000","last_unstale":"2024-04-14T15:20:12.935157+0000","last_undegraded":"2024-04-14T15:20:12.935157+0000","last_fullsized":"2024-04-14T15:20:12.935157+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"396'64","last_scrub_stamp":"2024-04-14T15:17:47.418687+0000","last_deep_scrub":"396'64","last_deep_scrub_stamp":"2024-04-14T15:17:47.418687+0000","last_clean_scrub_stamp":"2024-04-14T15:17:47.418687+0000","objects_scrubbed":5,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:14.561239+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00099928400000000002,"stat_sum":{"num_bytes":42,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":20,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"110.4","version":"406'55","reported_seq":1019,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.337892+0000","last_change":"2024-04-14T15:19:11.276191+0000","last_active":"2024-04-14T15:20:13.337892+0000","last_peered":"2024-04-14T15:20:13.337892+0000","last_clean":"2024-04-14T15:20:13.337892+0000","last_became_active":"2024-04-14T15:13:09.958847+0000","last_became_peered":"2024-04-14T15:13:09.958847+0000","last_unstale":"2024-04-14T15:20:13.337892+0000","last_undegraded":"2024-04-14T15:20:13.337892+0000","last_fullsized":"2024-04-14T15:20:13.337892+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"406'55","last_scrub_stamp":"2024-04-14T15:19:11.276056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:19:11.276056+0000","objects_scrubbed":4,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001395522,"stat_sum":{"num_bytes":38,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"110.7","version":"355'18","reported_seq":996,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.748543+0000","last_change":"2024-04-14T15:19:56.930358+0000","last_active":"2024-04-14T15:20:13.748543+0000","last_peered":"2024-04-14T15:20:13.748543+0000","last_clean":"2024-04-14T15:20:13.748543+0000","last_became_active":"2024-04-14T15:13:09.936631+0000","last_became_peered":"2024-04-14T15:13:09.936631+0000","last_unstale":"2024-04-14T15:20:13.748543+0000","last_undegraded":"2024-04-14T15:20:13.748543+0000","last_fullsized":"2024-04-14T15:20:13.748543+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'18","last_scrub_stamp":"2024-04-14T15:19:56.930258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:19:56.930258+0000","objects_scrubbed":2,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:09.254833+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00050163699999999998,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"110.6","version":"444'90","reported_seq":1045,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.661100+0000","last_change":"2024-04-14T15:19:12.507223+0000","last_active":"2024-04-14T15:20:13.661100+0000","last_peered":"2024-04-14T15:20:13.661100+0000","last_clean":"2024-04-14T15:20:13.661100+0000","last_became_active":"2024-04-14T15:13:09.930125+0000","last_became_peered":"2024-04-14T15:13:09.930125+0000","last_unstale":"2024-04-14T15:20:13.661100+0000","last_undegraded":"2024-04-14T15:20:13.661100+0000","last_fullsized":"2024-04-14T15:20:13.661100+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"444'90","last_scrub_stamp":"2024-04-14T15:19:12.507119+0000","last_deep_scrub":"274'74","last_deep_scrub_stamp":"2024-04-14T15:15:42.641684+0000","last_clean_scrub_stamp":"2024-04-14T15:19:12.507119+0000","objects_scrubbed":7,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:17.335479+0000","scrub_duration":1,"objects_trimmed":1,"snaptrim_duration":2.0059427689999998,"stat_sum":{"num_bytes":62,"num_objects":7,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":26,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"110.9","version":"369'20","reported_seq":992,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.934223+0000","last_change":"2024-04-14T15:19:22.004327+0000","last_active":"2024-04-14T15:20:12.934223+0000","last_peered":"2024-04-14T15:20:12.934223+0000","last_clean":"2024-04-14T15:20:12.934223+0000","last_became_active":"2024-04-14T15:13:09.957324+0000","last_became_peered":"2024-04-14T15:13:09.957324+0000","last_unstale":"2024-04-14T15:20:12.934223+0000","last_undegraded":"2024-04-14T15:20:12.934223+0000","last_fullsized":"2024-04-14T15:20:12.934223+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"369'20","last_scrub_stamp":"2024-04-14T15:19:22.004242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:19:22.004242+0000","objects_scrubbed":4,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:42.606912+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00098494500000000009,"stat_sum":{"num_bytes":35,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"110.8","version":"190'17","reported_seq":739,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.935425+0000","last_change":"2024-04-14T15:18:54.159492+0000","last_active":"2024-04-14T15:20:12.935425+0000","last_peered":"2024-04-14T15:20:12.935425+0000","last_clean":"2024-04-14T15:20:12.935425+0000","last_became_active":"2024-04-14T15:15:44.890057+0000","last_became_peered":"2024-04-14T15:15:44.890057+0000","last_unstale":"2024-04-14T15:20:12.935425+0000","last_undegraded":"2024-04-14T15:20:12.935425+0000","last_fullsized":"2024-04-14T15:20:12.935425+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'17","last_scrub_stamp":"2024-04-14T15:18:34.505521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:18:34.505521+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:13.731256+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00097299799999999998,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"110.b","version":"412'71","reported_seq":1038,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.385932+0000","last_change":"2024-04-14T15:19:22.798118+0000","last_active":"2024-04-14T15:20:13.385932+0000","last_peered":"2024-04-14T15:20:13.385932+0000","last_clean":"2024-04-14T15:20:13.385932+0000","last_became_active":"2024-04-14T15:13:09.964629+0000","last_became_peered":"2024-04-14T15:13:09.964629+0000","last_unstale":"2024-04-14T15:20:13.385932+0000","last_undegraded":"2024-04-14T15:20:13.385932+0000","last_fullsized":"2024-04-14T15:20:13.385932+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'71","last_scrub_stamp":"2024-04-14T15:19:22.798030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:19:22.798030+0000","objects_scrubbed":5,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:40.892661+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0017521979999999999,"stat_sum":{"num_bytes":54,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":22,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"110.a","version":"387'74","reported_seq":963,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.936259+0000","last_change":"2024-04-14T15:19:14.371878+0000","last_active":"2024-04-14T15:20:12.936259+0000","last_peered":"2024-04-14T15:20:12.936259+0000","last_clean":"2024-04-14T15:20:12.936259+0000","last_became_active":"2024-04-14T15:15:21.885346+0000","last_became_peered":"2024-04-14T15:15:21.885346+0000","last_unstale":"2024-04-14T15:20:12.936259+0000","last_undegraded":"2024-04-14T15:20:12.936259+0000","last_fullsized":"2024-04-14T15:20:12.936259+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'74","last_scrub_stamp":"2024-04-14T15:19:14.371777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:19:14.371777+0000","objects_scrubbed":9,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:23.703842+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00095557400000000005,"stat_sum":{"num_bytes":84,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":26,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"110.d","version":"416'95","reported_seq":887,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.937120+0000","last_change":"2024-04-14T15:19:13.327926+0000","last_active":"2024-04-14T15:20:12.937120+0000","last_peered":"2024-04-14T15:20:12.937120+0000","last_clean":"2024-04-14T15:20:12.937120+0000","last_became_active":"2024-04-14T15:15:21.877073+0000","last_became_peered":"2024-04-14T15:15:21.877073+0000","last_unstale":"2024-04-14T15:20:12.937120+0000","last_undegraded":"2024-04-14T15:20:12.937120+0000","last_fullsized":"2024-04-14T15:20:12.937120+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'95","last_scrub_stamp":"2024-04-14T15:19:13.327836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:19:13.327836+0000","objects_scrubbed":8,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:32.729185+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00067428099999999999,"stat_sum":{"num_bytes":71,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":34,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"110.c","version":"459'25","reported_seq":994,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.385797+0000","last_change":"2024-04-14T15:18:54.157224+0000","last_active":"2024-04-14T15:20:13.385797+0000","last_peered":"2024-04-14T15:20:13.385797+0000","last_clean":"2024-04-14T15:20:13.385797+0000","last_became_active":"2024-04-14T15:13:09.984609+0000","last_became_peered":"2024-04-14T15:13:09.984609+0000","last_unstale":"2024-04-14T15:20:13.385797+0000","last_undegraded":"2024-04-14T15:20:13.385797+0000","last_fullsized":"2024-04-14T15:20:13.385797+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'20","last_scrub_stamp":"2024-04-14T15:17:52.882509+0000","last_deep_scrub":"398'20","last_deep_scrub_stamp":"2024-04-14T15:17:52.882509+0000","last_clean_scrub_stamp":"2024-04-14T15:17:52.882509+0000","objects_scrubbed":2,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:15.826632+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001761999,"stat_sum":{"num_bytes":29,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":9,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"110.f","version":"355'52","reported_seq":1024,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.386268+0000","last_change":"2024-04-14T15:20:03.860729+0000","last_active":"2024-04-14T15:20:13.386268+0000","last_peered":"2024-04-14T15:20:13.386268+0000","last_clean":"2024-04-14T15:20:13.386268+0000","last_became_active":"2024-04-14T15:13:09.964057+0000","last_became_peered":"2024-04-14T15:13:09.964057+0000","last_unstale":"2024-04-14T15:20:13.386268+0000","last_undegraded":"2024-04-14T15:20:13.386268+0000","last_fullsized":"2024-04-14T15:20:13.386268+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'52","last_scrub_stamp":"2024-04-14T15:20:03.860616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:20:03.860616+0000","objects_scrubbed":4,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:21.988635+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001409509,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"110.e","version":"416'92","reported_seq":1059,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.661098+0000","last_change":"2024-04-14T15:20:02.654434+0000","last_active":"2024-04-14T15:20:13.661098+0000","last_peered":"2024-04-14T15:20:13.661098+0000","last_clean":"2024-04-14T15:20:13.661098+0000","last_became_active":"2024-04-14T15:13:09.984980+0000","last_became_peered":"2024-04-14T15:13:09.984980+0000","last_unstale":"2024-04-14T15:20:13.661098+0000","last_undegraded":"2024-04-14T15:20:13.661098+0000","last_fullsized":"2024-04-14T15:20:13.661098+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'92","last_scrub_stamp":"2024-04-14T15:20:02.654315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:20:02.654315+0000","objects_scrubbed":7,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:18.006902+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0012202070000000001,"stat_sum":{"num_bytes":68,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":28,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"91'130","reported_seq":468,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.660462+0000","last_change":"2024-04-14T15:19:09.497449+0000","last_active":"2024-04-14T15:20:13.660462+0000","last_peered":"2024-04-14T15:20:13.660462+0000","last_clean":"2024-04-14T15:20:13.660462+0000","last_became_active":"2024-04-14T15:18:58.187712+0000","last_became_peered":"2024-04-14T15:18:58.187712+0000","last_unstale":"2024-04-14T15:20:13.660462+0000","last_undegraded":"2024-04-14T15:20:13.660462+0000","last_fullsized":"2024-04-14T15:20:13.660462+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:09.497308+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:09.497308+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:23.450906+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":462,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.936659+0000","last_change":"2024-04-14T15:19:10.053883+0000","last_active":"2024-04-14T15:20:12.936659+0000","last_peered":"2024-04-14T15:20:12.936659+0000","last_clean":"2024-04-14T15:20:12.936659+0000","last_became_active":"2024-04-14T15:15:44.882941+0000","last_became_peered":"2024-04-14T15:15:44.882941+0000","last_unstale":"2024-04-14T15:20:12.936659+0000","last_undegraded":"2024-04-14T15:20:12.936659+0000","last_fullsized":"2024-04-14T15:20:12.936659+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:10.053804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:19:10.053804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:28.076611+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.9","version":"85'9","reported_seq":1037,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.659966+0000","last_change":"2024-04-14T15:17:54.606158+0000","last_active":"2024-04-14T15:20:13.659966+0000","last_peered":"2024-04-14T15:20:13.659966+0000","last_clean":"2024-04-14T15:20:13.659966+0000","last_became_active":"2024-04-14T15:09:47.045441+0000","last_became_peered":"2024-04-14T15:09:47.045441+0000","last_unstale":"2024-04-14T15:20:13.659966+0000","last_undegraded":"2024-04-14T15:20:13.659966+0000","last_fullsized":"2024-04-14T15:20:13.659966+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9","last_scrub_stamp":"2024-04-14T15:17:54.606051+0000","last_deep_scrub":"85'9","last_deep_scrub_stamp":"2024-04-14T15:10:51.950580+0000","last_clean_scrub_stamp":"2024-04-14T15:17:54.606051+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:18.657835+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"479'34","reported_seq":529,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.659550+0000","last_change":"2024-04-14T15:18:39.704989+0000","last_active":"2024-04-14T15:20:13.659550+0000","last_peered":"2024-04-14T15:20:13.659550+0000","last_clean":"2024-04-14T15:20:13.659550+0000","last_became_active":"2024-04-14T15:15:44.885409+0000","last_became_peered":"2024-04-14T15:15:44.885409+0000","last_unstale":"2024-04-14T15:20:13.659550+0000","last_undegraded":"2024-04-14T15:20:13.659550+0000","last_fullsized":"2024-04-14T15:20:13.659550+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'22","last_scrub_stamp":"2024-04-14T15:18:39.704883+0000","last_deep_scrub":"430'22","last_deep_scrub_stamp":"2024-04-14T15:18:39.704883+0000","last_clean_scrub_stamp":"2024-04-14T15:18:39.704883+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:18.657973+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":908,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.386372+0000","last_change":"2024-04-14T15:17:53.867764+0000","last_active":"2024-04-14T15:20:13.386372+0000","last_peered":"2024-04-14T15:20:13.386372+0000","last_clean":"2024-04-14T15:20:13.386372+0000","last_became_active":"2024-04-14T15:09:43.021460+0000","last_became_peered":"2024-04-14T15:09:43.021460+0000","last_unstale":"2024-04-14T15:20:13.386372+0000","last_undegraded":"2024-04-14T15:20:13.386372+0000","last_fullsized":"2024-04-14T15:20:13.386372+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:53.867696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:53.867696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:15.826563+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":462,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.747441+0000","last_change":"2024-04-14T15:17:54.889170+0000","last_active":"2024-04-14T15:20:13.747441+0000","last_peered":"2024-04-14T15:20:13.747441+0000","last_clean":"2024-04-14T15:20:13.747441+0000","last_became_active":"2024-04-14T15:15:44.888154+0000","last_became_peered":"2024-04-14T15:15:44.888154+0000","last_unstale":"2024-04-14T15:20:13.747441+0000","last_undegraded":"2024-04-14T15:20:13.747441+0000","last_fullsized":"2024-04-14T15:20:13.747441+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:54.889066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:15:40.368219+0000","last_clean_scrub_stamp":"2024-04-14T15:17:54.889066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"91'130","reported_seq":467,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.934371+0000","last_change":"2024-04-14T15:19:05.573366+0000","last_active":"2024-04-14T15:20:12.934371+0000","last_peered":"2024-04-14T15:20:12.934371+0000","last_clean":"2024-04-14T15:20:12.934371+0000","last_became_active":"2024-04-14T15:18:58.182420+0000","last_became_peered":"2024-04-14T15:18:58.182420+0000","last_unstale":"2024-04-14T15:20:12.934371+0000","last_undegraded":"2024-04-14T15:20:12.934371+0000","last_fullsized":"2024-04-14T15:20:12.934371+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:05.573281+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:05.573281+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:17.624412+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":806,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.660595+0000","last_change":"2024-04-14T15:18:37.636498+0000","last_active":"2024-04-14T15:20:13.660595+0000","last_peered":"2024-04-14T15:20:13.660595+0000","last_clean":"2024-04-14T15:20:13.660595+0000","last_became_active":"2024-04-14T15:12:56.355348+0000","last_became_peered":"2024-04-14T15:12:56.355348+0000","last_unstale":"2024-04-14T15:20:13.660595+0000","last_undegraded":"2024-04-14T15:20:13.660595+0000","last_fullsized":"2024-04-14T15:20:13.660595+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:37.636419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:18:37.636419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:18.658010+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"479'15","reported_seq":931,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.337932+0000","last_change":"2024-04-14T15:16:56.663996+0000","last_active":"2024-04-14T15:20:13.337932+0000","last_peered":"2024-04-14T15:20:13.337932+0000","last_clean":"2024-04-14T15:20:13.337932+0000","last_became_active":"2024-04-14T15:09:45.037179+0000","last_became_peered":"2024-04-14T15:09:45.037179+0000","last_unstale":"2024-04-14T15:20:13.337932+0000","last_undegraded":"2024-04-14T15:20:13.337932+0000","last_fullsized":"2024-04-14T15:20:13.337932+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-04-14T15:16:56.663900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:16:56.663900+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:17.378125+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":911,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.937061+0000","last_change":"2024-04-14T15:18:36.029705+0000","last_active":"2024-04-14T15:20:12.937061+0000","last_peered":"2024-04-14T15:20:12.937061+0000","last_clean":"2024-04-14T15:20:12.937061+0000","last_became_active":"2024-04-14T15:09:43.020044+0000","last_became_peered":"2024-04-14T15:09:43.020044+0000","last_unstale":"2024-04-14T15:20:12.937061+0000","last_undegraded":"2024-04-14T15:20:12.937061+0000","last_fullsized":"2024-04-14T15:20:12.937061+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:36.029630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:18:36.029630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:15.771564+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":908,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.385697+0000","last_change":"2024-04-14T15:19:11.838298+0000","last_active":"2024-04-14T15:20:13.385697+0000","last_peered":"2024-04-14T15:20:13.385697+0000","last_clean":"2024-04-14T15:20:13.385697+0000","last_became_active":"2024-04-14T15:09:47.046153+0000","last_became_peered":"2024-04-14T15:09:47.046153+0000","last_unstale":"2024-04-14T15:20:13.385697+0000","last_undegraded":"2024-04-14T15:20:13.385697+0000","last_fullsized":"2024-04-14T15:20:13.385697+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:11.838212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:19:11.838212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:19.779219+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":898,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.338629+0000","last_change":"2024-04-14T15:17:37.919665+0000","last_active":"2024-04-14T15:20:13.338629+0000","last_peered":"2024-04-14T15:20:13.338629+0000","last_clean":"2024-04-14T15:20:13.338629+0000","last_became_active":"2024-04-14T15:09:49.061495+0000","last_became_peered":"2024-04-14T15:09:49.061495+0000","last_unstale":"2024-04-14T15:20:13.338629+0000","last_undegraded":"2024-04-14T15:20:13.338629+0000","last_fullsized":"2024-04-14T15:20:13.338629+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:37.919595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:17:37.919595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:17.378174+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"91'130","reported_seq":995,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.748349+0000","last_change":"2024-04-14T15:19:06.754212+0000","last_active":"2024-04-14T15:20:13.748349+0000","last_peered":"2024-04-14T15:20:13.748349+0000","last_clean":"2024-04-14T15:20:13.748349+0000","last_became_active":"2024-04-14T15:18:45.919901+0000","last_became_peered":"2024-04-14T15:18:45.919901+0000","last_unstale":"2024-04-14T15:20:13.748349+0000","last_undegraded":"2024-04-14T15:20:13.748349+0000","last_fullsized":"2024-04-14T15:20:13.748349+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:06.754126+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:06.754126+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:30.044936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":806,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.660794+0000","last_change":"2024-04-14T15:18:25.523604+0000","last_active":"2024-04-14T15:20:13.660794+0000","last_peered":"2024-04-14T15:20:13.660794+0000","last_clean":"2024-04-14T15:20:13.660794+0000","last_became_active":"2024-04-14T15:12:56.246062+0000","last_became_peered":"2024-04-14T15:12:56.246062+0000","last_unstale":"2024-04-14T15:20:13.660794+0000","last_undegraded":"2024-04-14T15:20:13.660794+0000","last_fullsized":"2024-04-14T15:20:13.660794+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:25.523520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:18:25.523520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:18.657893+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"479'44","reported_seq":978,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.747666+0000","last_change":"2024-04-14T15:19:58.856267+0000","last_active":"2024-04-14T15:20:13.747666+0000","last_peered":"2024-04-14T15:20:13.747666+0000","last_clean":"2024-04-14T15:20:13.747666+0000","last_became_active":"2024-04-14T15:09:45.178782+0000","last_became_peered":"2024-04-14T15:09:45.178782+0000","last_unstale":"2024-04-14T15:20:13.747666+0000","last_undegraded":"2024-04-14T15:20:13.747666+0000","last_fullsized":"2024-04-14T15:20:13.747666+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'44","last_scrub_stamp":"2024-04-14T15:19:58.856115+0000","last_deep_scrub":"479'44","last_deep_scrub_stamp":"2024-04-14T15:19:58.856115+0000","last_clean_scrub_stamp":"2024-04-14T15:19:58.856115+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:10.582292+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":916,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.386448+0000","last_change":"2024-04-14T15:18:36.925573+0000","last_active":"2024-04-14T15:20:13.386448+0000","last_peered":"2024-04-14T15:20:13.386448+0000","last_clean":"2024-04-14T15:20:13.386448+0000","last_became_active":"2024-04-14T15:09:43.020268+0000","last_became_peered":"2024-04-14T15:09:43.020268+0000","last_unstale":"2024-04-14T15:20:13.386448+0000","last_undegraded":"2024-04-14T15:20:13.386448+0000","last_fullsized":"2024-04-14T15:20:13.386448+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:36.925496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:34.342793+0000","last_clean_scrub_stamp":"2024-04-14T15:18:36.925496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:15.826708+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":900,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.385991+0000","last_change":"2024-04-14T15:18:31.989052+0000","last_active":"2024-04-14T15:20:13.385991+0000","last_peered":"2024-04-14T15:20:13.385991+0000","last_clean":"2024-04-14T15:20:13.385991+0000","last_became_active":"2024-04-14T15:09:47.045542+0000","last_became_peered":"2024-04-14T15:09:47.045542+0000","last_unstale":"2024-04-14T15:20:13.385991+0000","last_undegraded":"2024-04-14T15:20:13.385991+0000","last_fullsized":"2024-04-14T15:20:13.385991+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:31.988957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:18:31.988957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:15.826738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":897,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.659086+0000","last_change":"2024-04-14T15:19:16.566595+0000","last_active":"2024-04-14T15:20:13.659086+0000","last_peered":"2024-04-14T15:20:13.659086+0000","last_clean":"2024-04-14T15:20:13.659086+0000","last_became_active":"2024-04-14T15:09:49.065397+0000","last_became_peered":"2024-04-14T15:09:49.065397+0000","last_unstale":"2024-04-14T15:20:13.659086+0000","last_undegraded":"2024-04-14T15:20:13.659086+0000","last_fullsized":"2024-04-14T15:20:13.659086+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:16.566496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:19:16.566496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:28.994328+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"91'130","reported_seq":796,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.385197+0000","last_change":"2024-04-14T15:19:04.951104+0000","last_active":"2024-04-14T15:20:13.385197+0000","last_peered":"2024-04-14T15:20:13.385197+0000","last_clean":"2024-04-14T15:20:13.385197+0000","last_became_active":"2024-04-14T15:18:58.181194+0000","last_became_peered":"2024-04-14T15:18:58.181194+0000","last_unstale":"2024-04-14T15:20:13.385197+0000","last_undegraded":"2024-04-14T15:20:13.385197+0000","last_fullsized":"2024-04-14T15:20:13.385197+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:04.950991+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:04.950991+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:15.826754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":813,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.385652+0000","last_change":"2024-04-14T15:19:15.778552+0000","last_active":"2024-04-14T15:20:13.385652+0000","last_peered":"2024-04-14T15:20:13.385652+0000","last_clean":"2024-04-14T15:20:13.385652+0000","last_became_active":"2024-04-14T15:12:56.474322+0000","last_became_peered":"2024-04-14T15:12:56.474322+0000","last_unstale":"2024-04-14T15:20:13.385652+0000","last_undegraded":"2024-04-14T15:20:13.385652+0000","last_fullsized":"2024-04-14T15:20:13.385652+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:15.778469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:19:15.778469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:40.510707+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"479'31","reported_seq":1002,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.935653+0000","last_change":"2024-04-14T15:17:48.374589+0000","last_active":"2024-04-14T15:20:12.935653+0000","last_peered":"2024-04-14T15:20:12.935653+0000","last_clean":"2024-04-14T15:20:12.935653+0000","last_became_active":"2024-04-14T15:13:12.933907+0000","last_became_peered":"2024-04-14T15:13:12.933907+0000","last_unstale":"2024-04-14T15:20:12.935653+0000","last_undegraded":"2024-04-14T15:20:12.935653+0000","last_fullsized":"2024-04-14T15:20:12.935653+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'17","last_scrub_stamp":"2024-04-14T15:17:48.374469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:17:48.374469+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:14.561315+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":909,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.660354+0000","last_change":"2024-04-14T15:19:26.515133+0000","last_active":"2024-04-14T15:20:13.660354+0000","last_peered":"2024-04-14T15:20:13.660354+0000","last_clean":"2024-04-14T15:20:13.660354+0000","last_became_active":"2024-04-14T15:09:43.022351+0000","last_became_peered":"2024-04-14T15:09:43.022351+0000","last_unstale":"2024-04-14T15:20:13.660354+0000","last_undegraded":"2024-04-14T15:20:13.660354+0000","last_fullsized":"2024-04-14T15:20:13.660354+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:26.515055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:19:26.515055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:51.899280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":895,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.935860+0000","last_change":"2024-04-14T15:17:54.871693+0000","last_active":"2024-04-14T15:20:12.935860+0000","last_peered":"2024-04-14T15:20:12.935860+0000","last_clean":"2024-04-14T15:20:12.935860+0000","last_became_active":"2024-04-14T15:09:47.042810+0000","last_became_peered":"2024-04-14T15:09:47.042810+0000","last_unstale":"2024-04-14T15:20:12.935860+0000","last_undegraded":"2024-04-14T15:20:12.935860+0000","last_fullsized":"2024-04-14T15:20:12.935860+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:54.871593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:17:54.871593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:15.771533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":904,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.385002+0000","last_change":"2024-04-14T15:19:51.902814+0000","last_active":"2024-04-14T15:20:13.385002+0000","last_peered":"2024-04-14T15:20:13.385002+0000","last_clean":"2024-04-14T15:20:13.385002+0000","last_became_active":"2024-04-14T15:09:49.069594+0000","last_became_peered":"2024-04-14T15:09:49.069594+0000","last_unstale":"2024-04-14T15:20:13.385002+0000","last_undegraded":"2024-04-14T15:20:13.385002+0000","last_fullsized":"2024-04-14T15:20:13.385002+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:51.902730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:19:51.902730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:19.807697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"91'130","reported_seq":1198,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.659478+0000","last_change":"2024-04-14T15:19:06.490294+0000","last_active":"2024-04-14T15:20:13.659478+0000","last_peered":"2024-04-14T15:20:13.659478+0000","last_clean":"2024-04-14T15:20:13.659478+0000","last_became_active":"2024-04-14T15:18:58.189483+0000","last_became_peered":"2024-04-14T15:18:58.189483+0000","last_unstale":"2024-04-14T15:20:13.659478+0000","last_undegraded":"2024-04-14T15:20:13.659478+0000","last_fullsized":"2024-04-14T15:20:13.659478+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:06.490208+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:06.490208+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:16.848376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":815,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.747610+0000","last_change":"2024-04-14T15:20:02.774050+0000","last_active":"2024-04-14T15:20:13.747610+0000","last_peered":"2024-04-14T15:20:13.747610+0000","last_clean":"2024-04-14T15:20:13.747610+0000","last_became_active":"2024-04-14T15:12:56.289745+0000","last_became_peered":"2024-04-14T15:12:56.289745+0000","last_unstale":"2024-04-14T15:20:13.747610+0000","last_undegraded":"2024-04-14T15:20:13.747610+0000","last_fullsized":"2024-04-14T15:20:13.747610+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:02.773961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:20:02.773961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:08.770183+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"479'25","reported_seq":940,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.933939+0000","last_change":"2024-04-14T15:20:03.571298+0000","last_active":"2024-04-14T15:20:12.933939+0000","last_peered":"2024-04-14T15:20:12.933939+0000","last_clean":"2024-04-14T15:20:12.933939+0000","last_became_active":"2024-04-14T15:09:45.029170+0000","last_became_peered":"2024-04-14T15:09:45.029170+0000","last_unstale":"2024-04-14T15:20:12.933939+0000","last_undegraded":"2024-04-14T15:20:12.933939+0000","last_fullsized":"2024-04-14T15:20:12.933939+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'25","last_scrub_stamp":"2024-04-14T15:20:03.571160+0000","last_deep_scrub":"479'25","last_deep_scrub_stamp":"2024-04-14T15:20:03.571160+0000","last_clean_scrub_stamp":"2024-04-14T15:20:03.571160+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:14.606809+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":911,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.934814+0000","last_change":"2024-04-14T15:17:51.854742+0000","last_active":"2024-04-14T15:20:12.934814+0000","last_peered":"2024-04-14T15:20:12.934814+0000","last_clean":"2024-04-14T15:20:12.934814+0000","last_became_active":"2024-04-14T15:09:43.018871+0000","last_became_peered":"2024-04-14T15:09:43.018871+0000","last_unstale":"2024-04-14T15:20:12.934814+0000","last_undegraded":"2024-04-14T15:20:12.934814+0000","last_fullsized":"2024-04-14T15:20:12.934814+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:51.854659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:51.854659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:15.771467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"400'9","reported_seq":1036,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.386017+0000","last_change":"2024-04-14T15:17:54.845967+0000","last_active":"2024-04-14T15:20:13.386017+0000","last_peered":"2024-04-14T15:20:13.386017+0000","last_clean":"2024-04-14T15:20:13.386017+0000","last_became_active":"2024-04-14T15:09:47.044085+0000","last_became_peered":"2024-04-14T15:09:47.044085+0000","last_unstale":"2024-04-14T15:20:13.386017+0000","last_undegraded":"2024-04-14T15:20:13.386017+0000","last_fullsized":"2024-04-14T15:20:13.386017+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'9","last_scrub_stamp":"2024-04-14T15:17:54.845881+0000","last_deep_scrub":"400'9","last_deep_scrub_stamp":"2024-04-14T15:17:54.845881+0000","last_clean_scrub_stamp":"2024-04-14T15:17:54.845881+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:15.826580+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":897,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.935323+0000","last_change":"2024-04-14T15:19:18.588681+0000","last_active":"2024-04-14T15:20:12.935323+0000","last_peered":"2024-04-14T15:20:12.935323+0000","last_clean":"2024-04-14T15:20:12.935323+0000","last_became_active":"2024-04-14T15:09:49.065274+0000","last_became_peered":"2024-04-14T15:09:49.065274+0000","last_unstale":"2024-04-14T15:20:12.935323+0000","last_undegraded":"2024-04-14T15:20:12.935323+0000","last_fullsized":"2024-04-14T15:20:12.935323+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:18.588546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:18.588546+0000","last_clean_scrub_stamp":"2024-04-14T15:19:18.588546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:36.014977+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"91'130","reported_seq":804,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.385481+0000","last_change":"2024-04-14T15:19:05.972132+0000","last_active":"2024-04-14T15:20:13.385481+0000","last_peered":"2024-04-14T15:20:13.385481+0000","last_clean":"2024-04-14T15:20:13.385481+0000","last_became_active":"2024-04-14T15:18:58.187649+0000","last_became_peered":"2024-04-14T15:18:58.187649+0000","last_unstale":"2024-04-14T15:20:13.385481+0000","last_undegraded":"2024-04-14T15:20:13.385481+0000","last_fullsized":"2024-04-14T15:20:13.385481+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:05.972050+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:05.972050+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:15.882926+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":808,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.934733+0000","last_change":"2024-04-14T15:19:11.008294+0000","last_active":"2024-04-14T15:20:12.934733+0000","last_peered":"2024-04-14T15:20:12.934733+0000","last_clean":"2024-04-14T15:20:12.934733+0000","last_became_active":"2024-04-14T15:12:56.487615+0000","last_became_peered":"2024-04-14T15:12:56.487615+0000","last_unstale":"2024-04-14T15:20:12.934733+0000","last_undegraded":"2024-04-14T15:20:12.934733+0000","last_fullsized":"2024-04-14T15:20:12.934733+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:11.008225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:19:11.008225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:29.857396+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"479'34","reported_seq":957,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.936741+0000","last_change":"2024-04-14T15:17:52.883984+0000","last_active":"2024-04-14T15:20:12.936741+0000","last_peered":"2024-04-14T15:20:12.936741+0000","last_clean":"2024-04-14T15:20:12.936741+0000","last_became_active":"2024-04-14T15:09:45.033331+0000","last_became_peered":"2024-04-14T15:09:45.033331+0000","last_unstale":"2024-04-14T15:20:12.936741+0000","last_undegraded":"2024-04-14T15:20:12.936741+0000","last_fullsized":"2024-04-14T15:20:12.936741+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'22","last_scrub_stamp":"2024-04-14T15:17:52.883850+0000","last_deep_scrub":"398'22","last_deep_scrub_stamp":"2024-04-14T15:17:52.883850+0000","last_clean_scrub_stamp":"2024-04-14T15:17:52.883850+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:15.771396+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":911,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.934439+0000","last_change":"2024-04-14T15:18:25.009179+0000","last_active":"2024-04-14T15:20:12.934439+0000","last_peered":"2024-04-14T15:20:12.934439+0000","last_clean":"2024-04-14T15:20:12.934439+0000","last_became_active":"2024-04-14T15:09:43.020878+0000","last_became_peered":"2024-04-14T15:09:43.020878+0000","last_unstale":"2024-04-14T15:20:12.934439+0000","last_undegraded":"2024-04-14T15:20:12.934439+0000","last_fullsized":"2024-04-14T15:20:12.934439+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:25.009100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:18:25.009100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:15.771634+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"72'8","reported_seq":1036,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.935210+0000","last_change":"2024-04-14T15:19:19.596882+0000","last_active":"2024-04-14T15:20:12.935210+0000","last_peered":"2024-04-14T15:20:12.935210+0000","last_clean":"2024-04-14T15:20:12.935210+0000","last_became_active":"2024-04-14T15:09:47.050962+0000","last_became_peered":"2024-04-14T15:09:47.050962+0000","last_unstale":"2024-04-14T15:20:12.935210+0000","last_undegraded":"2024-04-14T15:20:12.935210+0000","last_fullsized":"2024-04-14T15:20:12.935210+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:19:19.596773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:19:19.596773+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:44.419959+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":463,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.936660+0000","last_change":"2024-04-14T15:16:51.199025+0000","last_active":"2024-04-14T15:20:12.936660+0000","last_peered":"2024-04-14T15:20:12.936660+0000","last_clean":"2024-04-14T15:20:12.936660+0000","last_became_active":"2024-04-14T15:15:44.879699+0000","last_became_peered":"2024-04-14T15:15:44.879699+0000","last_unstale":"2024-04-14T15:20:12.936660+0000","last_undegraded":"2024-04-14T15:20:12.936660+0000","last_fullsized":"2024-04-14T15:20:12.936660+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:51.198907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:51.198907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:15.771358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"91'130","reported_seq":994,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.934615+0000","last_change":"2024-04-14T15:19:08.097458+0000","last_active":"2024-04-14T15:20:12.934615+0000","last_peered":"2024-04-14T15:20:12.934615+0000","last_clean":"2024-04-14T15:20:12.934615+0000","last_became_active":"2024-04-14T15:18:58.179785+0000","last_became_peered":"2024-04-14T15:18:58.179785+0000","last_unstale":"2024-04-14T15:20:12.934615+0000","last_undegraded":"2024-04-14T15:20:12.934615+0000","last_fullsized":"2024-04-14T15:20:12.934615+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:08.097372+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:08.097372+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:22.282987+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"233'8","reported_seq":821,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.747669+0000","last_change":"2024-04-14T15:17:50.905698+0000","last_active":"2024-04-14T15:20:13.747669+0000","last_peered":"2024-04-14T15:20:13.747669+0000","last_clean":"2024-04-14T15:20:13.747669+0000","last_became_active":"2024-04-14T15:12:56.381026+0000","last_became_peered":"2024-04-14T15:12:56.381026+0000","last_unstale":"2024-04-14T15:20:13.747669+0000","last_undegraded":"2024-04-14T15:20:13.747669+0000","last_fullsized":"2024-04-14T15:20:13.747669+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'8","last_scrub_stamp":"2024-04-14T15:17:50.905608+0000","last_deep_scrub":"233'8","last_deep_scrub_stamp":"2024-04-14T15:15:01.295800+0000","last_clean_scrub_stamp":"2024-04-14T15:17:50.905608+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"479'24","reported_seq":941,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.934714+0000","last_change":"2024-04-14T15:16:53.143329+0000","last_active":"2024-04-14T15:20:12.934714+0000","last_peered":"2024-04-14T15:20:12.934714+0000","last_clean":"2024-04-14T15:20:12.934714+0000","last_became_active":"2024-04-14T15:09:45.028212+0000","last_became_peered":"2024-04-14T15:09:45.028212+0000","last_unstale":"2024-04-14T15:20:12.934714+0000","last_undegraded":"2024-04-14T15:20:12.934714+0000","last_fullsized":"2024-04-14T15:20:12.934714+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-14T15:16:53.143239+0000","last_deep_scrub":"85'16","last_deep_scrub_stamp":"2024-04-14T15:10:47.456169+0000","last_clean_scrub_stamp":"2024-04-14T15:16:53.143239+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:14.561183+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":917,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.659661+0000","last_change":"2024-04-14T15:20:03.642215+0000","last_active":"2024-04-14T15:20:13.659661+0000","last_peered":"2024-04-14T15:20:13.659661+0000","last_clean":"2024-04-14T15:20:13.659661+0000","last_became_active":"2024-04-14T15:09:43.018601+0000","last_became_peered":"2024-04-14T15:09:43.018601+0000","last_unstale":"2024-04-14T15:20:13.659661+0000","last_undegraded":"2024-04-14T15:20:13.659661+0000","last_fullsized":"2024-04-14T15:20:13.659661+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:03.642112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:20:03.642112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:09.466843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":468,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.935110+0000","last_change":"2024-04-14T15:19:59.597112+0000","last_active":"2024-04-14T15:20:12.935110+0000","last_peered":"2024-04-14T15:20:12.935110+0000","last_clean":"2024-04-14T15:20:12.935110+0000","last_became_active":"2024-04-14T15:15:44.886834+0000","last_became_peered":"2024-04-14T15:15:44.886834+0000","last_unstale":"2024-04-14T15:20:12.935110+0000","last_undegraded":"2024-04-14T15:20:12.935110+0000","last_fullsized":"2024-04-14T15:20:12.935110+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:59.596964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:19:59.596964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:07.158046+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":572,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.937573+0000","last_change":"2024-04-14T15:17:57.214134+0000","last_active":"2024-04-14T15:20:12.937573+0000","last_peered":"2024-04-14T15:20:12.937573+0000","last_clean":"2024-04-14T15:20:12.937573+0000","last_became_active":"2024-04-14T15:15:21.888587+0000","last_became_peered":"2024-04-14T15:15:21.888587+0000","last_unstale":"2024-04-14T15:20:12.937573+0000","last_undegraded":"2024-04-14T15:20:12.937573+0000","last_fullsized":"2024-04-14T15:20:12.937573+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:57.214046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:57.214046+0000","last_clean_scrub_stamp":"2024-04-14T15:17:57.214046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:13.731146+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"91'130","reported_seq":977,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.937316+0000","last_change":"2024-04-14T15:19:05.396196+0000","last_active":"2024-04-14T15:20:12.937316+0000","last_peered":"2024-04-14T15:20:12.937316+0000","last_clean":"2024-04-14T15:20:12.937316+0000","last_became_active":"2024-04-14T15:18:40.917537+0000","last_became_peered":"2024-04-14T15:18:40.917537+0000","last_unstale":"2024-04-14T15:20:12.937316+0000","last_undegraded":"2024-04-14T15:20:12.937316+0000","last_fullsized":"2024-04-14T15:20:12.937316+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:05.396090+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:05.396090+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":806,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.934616+0000","last_change":"2024-04-14T15:16:56.214275+0000","last_active":"2024-04-14T15:20:12.934616+0000","last_peered":"2024-04-14T15:20:12.934616+0000","last_clean":"2024-04-14T15:20:12.934616+0000","last_became_active":"2024-04-14T15:12:56.427276+0000","last_became_peered":"2024-04-14T15:12:56.427276+0000","last_unstale":"2024-04-14T15:20:12.934616+0000","last_undegraded":"2024-04-14T15:20:12.934616+0000","last_fullsized":"2024-04-14T15:20:12.934616+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:56.214178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:16:56.214178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:14.561142+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"255'10","reported_seq":680,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.935727+0000","last_change":"2024-04-14T15:19:50.669678+0000","last_active":"2024-04-14T15:20:12.935727+0000","last_peered":"2024-04-14T15:20:12.935727+0000","last_clean":"2024-04-14T15:20:12.935727+0000","last_became_active":"2024-04-14T15:15:21.873747+0000","last_became_peered":"2024-04-14T15:15:21.873747+0000","last_unstale":"2024-04-14T15:20:12.935727+0000","last_undegraded":"2024-04-14T15:20:12.935727+0000","last_fullsized":"2024-04-14T15:20:12.935727+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'10","last_scrub_stamp":"2024-04-14T15:19:50.669574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:19:50.669574+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:12.441595+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"479'28","reported_seq":948,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.934440+0000","last_change":"2024-04-14T15:18:22.545734+0000","last_active":"2024-04-14T15:20:12.934440+0000","last_peered":"2024-04-14T15:20:12.934440+0000","last_clean":"2024-04-14T15:20:12.934440+0000","last_became_active":"2024-04-14T15:09:45.025673+0000","last_became_peered":"2024-04-14T15:09:45.025673+0000","last_unstale":"2024-04-14T15:20:12.934440+0000","last_undegraded":"2024-04-14T15:20:12.934440+0000","last_fullsized":"2024-04-14T15:20:12.934440+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'14","last_scrub_stamp":"2024-04-14T15:18:22.545646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:18:22.545646+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:14.561375+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":910,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.748500+0000","last_change":"2024-04-14T15:18:26.824852+0000","last_active":"2024-04-14T15:20:13.748500+0000","last_peered":"2024-04-14T15:20:13.748500+0000","last_clean":"2024-04-14T15:20:13.748500+0000","last_became_active":"2024-04-14T15:09:43.020503+0000","last_became_peered":"2024-04-14T15:09:43.020503+0000","last_unstale":"2024-04-14T15:20:13.748500+0000","last_undegraded":"2024-04-14T15:20:13.748500+0000","last_fullsized":"2024-04-14T15:20:13.748500+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:26.824773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:18:26.824773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":898,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.660690+0000","last_change":"2024-04-14T15:19:50.609813+0000","last_active":"2024-04-14T15:20:13.660690+0000","last_peered":"2024-04-14T15:20:13.660690+0000","last_clean":"2024-04-14T15:20:13.660690+0000","last_became_active":"2024-04-14T15:09:49.129610+0000","last_became_peered":"2024-04-14T15:09:49.129610+0000","last_unstale":"2024-04-14T15:20:13.660690+0000","last_undegraded":"2024-04-14T15:20:13.660690+0000","last_fullsized":"2024-04-14T15:20:13.660690+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:50.609691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:19:50.609691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:58.284777+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"91'130","reported_seq":984,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.934797+0000","last_change":"2024-04-14T15:19:04.526327+0000","last_active":"2024-04-14T15:20:12.934797+0000","last_peered":"2024-04-14T15:20:12.934797+0000","last_clean":"2024-04-14T15:20:12.934797+0000","last_became_active":"2024-04-14T15:18:58.179181+0000","last_became_peered":"2024-04-14T15:18:58.179181+0000","last_unstale":"2024-04-14T15:20:12.934797+0000","last_undegraded":"2024-04-14T15:20:12.934797+0000","last_fullsized":"2024-04-14T15:20:12.934797+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:04.526214+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:04.526214+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:30.900098+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":806,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.661350+0000","last_change":"2024-04-14T15:17:47.553179+0000","last_active":"2024-04-14T15:20:13.661350+0000","last_peered":"2024-04-14T15:20:13.661350+0000","last_clean":"2024-04-14T15:20:13.661350+0000","last_became_active":"2024-04-14T15:12:56.355254+0000","last_became_peered":"2024-04-14T15:12:56.355254+0000","last_unstale":"2024-04-14T15:20:13.661350+0000","last_undegraded":"2024-04-14T15:20:13.661350+0000","last_fullsized":"2024-04-14T15:20:13.661350+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:47.553055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:17:47.553055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:18.657795+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":902,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.934795+0000","last_change":"2024-04-14T15:18:39.394736+0000","last_active":"2024-04-14T15:20:12.934795+0000","last_peered":"2024-04-14T15:20:12.934795+0000","last_clean":"2024-04-14T15:20:12.934795+0000","last_became_active":"2024-04-14T15:09:47.044610+0000","last_became_peered":"2024-04-14T15:09:47.044610+0000","last_unstale":"2024-04-14T15:20:12.934795+0000","last_undegraded":"2024-04-14T15:20:12.934795+0000","last_fullsized":"2024-04-14T15:20:12.934795+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:39.394655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:39.394655+0000","last_clean_scrub_stamp":"2024-04-14T15:18:39.394655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:14.561432+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"479'30","reported_seq":639,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.935635+0000","last_change":"2024-04-14T15:19:26.509637+0000","last_active":"2024-04-14T15:20:12.935635+0000","last_peered":"2024-04-14T15:20:12.935635+0000","last_clean":"2024-04-14T15:20:12.935635+0000","last_became_active":"2024-04-14T15:15:21.880565+0000","last_became_peered":"2024-04-14T15:15:21.880565+0000","last_unstale":"2024-04-14T15:20:12.935635+0000","last_undegraded":"2024-04-14T15:20:12.935635+0000","last_fullsized":"2024-04-14T15:20:12.935635+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'18","last_scrub_stamp":"2024-04-14T15:19:26.509541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:19:26.509541+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:53.956765+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'2","reported_seq":905,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.936105+0000","last_change":"2024-04-14T15:17:48.850178+0000","last_active":"2024-04-14T15:20:12.936105+0000","last_peered":"2024-04-14T15:20:12.936105+0000","last_clean":"2024-04-14T15:20:12.936105+0000","last_became_active":"2024-04-14T15:09:43.018478+0000","last_became_peered":"2024-04-14T15:09:43.018478+0000","last_unstale":"2024-04-14T15:20:12.936105+0000","last_undegraded":"2024-04-14T15:20:12.936105+0000","last_fullsized":"2024-04-14T15:20:12.936105+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'2","last_scrub_stamp":"2024-04-14T15:17:48.850069+0000","last_deep_scrub":"91'2","last_deep_scrub_stamp":"2024-04-14T15:17:48.850069+0000","last_clean_scrub_stamp":"2024-04-14T15:17:48.850069+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:15.771486+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":897,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.934334+0000","last_change":"2024-04-14T15:19:14.609181+0000","last_active":"2024-04-14T15:20:12.934334+0000","last_peered":"2024-04-14T15:20:12.934334+0000","last_clean":"2024-04-14T15:20:12.934334+0000","last_became_active":"2024-04-14T15:09:49.065684+0000","last_became_peered":"2024-04-14T15:09:49.065684+0000","last_unstale":"2024-04-14T15:20:12.934334+0000","last_undegraded":"2024-04-14T15:20:12.934334+0000","last_fullsized":"2024-04-14T15:20:12.934334+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:14.609079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:19:14.609079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:23.254598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"91'130","reported_seq":992,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.748091+0000","last_change":"2024-04-14T15:19:05.797940+0000","last_active":"2024-04-14T15:20:13.748091+0000","last_peered":"2024-04-14T15:20:13.748091+0000","last_clean":"2024-04-14T15:20:13.748091+0000","last_became_active":"2024-04-14T15:18:58.182649+0000","last_became_peered":"2024-04-14T15:18:58.182649+0000","last_unstale":"2024-04-14T15:20:13.748091+0000","last_undegraded":"2024-04-14T15:20:13.748091+0000","last_fullsized":"2024-04-14T15:20:13.748091+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:05.797859+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:05.797859+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:17.283641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":808,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.935660+0000","last_change":"2024-04-14T15:18:35.004058+0000","last_active":"2024-04-14T15:20:12.935660+0000","last_peered":"2024-04-14T15:20:12.935660+0000","last_clean":"2024-04-14T15:20:12.935660+0000","last_became_active":"2024-04-14T15:12:56.402023+0000","last_became_peered":"2024-04-14T15:12:56.402023+0000","last_unstale":"2024-04-14T15:20:12.935660+0000","last_undegraded":"2024-04-14T15:20:12.935660+0000","last_fullsized":"2024-04-14T15:20:12.935660+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:35.003968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:35.003968+0000","last_clean_scrub_stamp":"2024-04-14T15:18:35.003968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:15.771602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":911,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.747818+0000","last_change":"2024-04-14T15:20:06.794160+0000","last_active":"2024-04-14T15:20:13.747818+0000","last_peered":"2024-04-14T15:20:13.747818+0000","last_clean":"2024-04-14T15:20:13.747818+0000","last_became_active":"2024-04-14T15:09:47.048427+0000","last_became_peered":"2024-04-14T15:09:47.048427+0000","last_unstale":"2024-04-14T15:20:13.747818+0000","last_undegraded":"2024-04-14T15:20:13.747818+0000","last_fullsized":"2024-04-14T15:20:13.747818+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:06.794023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:20:06.794023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:16.849759+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"479'37","reported_seq":966,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.338166+0000","last_change":"2024-04-14T15:17:46.924432+0000","last_active":"2024-04-14T15:20:13.338166+0000","last_peered":"2024-04-14T15:20:13.338166+0000","last_clean":"2024-04-14T15:20:13.338166+0000","last_became_active":"2024-04-14T15:09:45.179182+0000","last_became_peered":"2024-04-14T15:09:45.179182+0000","last_unstale":"2024-04-14T15:20:13.338166+0000","last_undegraded":"2024-04-14T15:20:13.338166+0000","last_fullsized":"2024-04-14T15:20:13.338166+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'27","last_scrub_stamp":"2024-04-14T15:17:46.924330+0000","last_deep_scrub":"102'27","last_deep_scrub_stamp":"2024-04-14T15:17:46.924330+0000","last_clean_scrub_stamp":"2024-04-14T15:17:46.924330+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:17.378210+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":910,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.660736+0000","last_change":"2024-04-14T15:19:14.569862+0000","last_active":"2024-04-14T15:20:13.660736+0000","last_peered":"2024-04-14T15:20:13.660736+0000","last_clean":"2024-04-14T15:20:13.660736+0000","last_became_active":"2024-04-14T15:09:43.021006+0000","last_became_peered":"2024-04-14T15:09:43.021006+0000","last_unstale":"2024-04-14T15:20:13.660736+0000","last_undegraded":"2024-04-14T15:20:13.660736+0000","last_fullsized":"2024-04-14T15:20:13.660736+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:14.569778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:19:14.569778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:38.152634+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":900,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.934578+0000","last_change":"2024-04-14T15:20:00.976745+0000","last_active":"2024-04-14T15:20:12.934578+0000","last_peered":"2024-04-14T15:20:12.934578+0000","last_clean":"2024-04-14T15:20:12.934578+0000","last_became_active":"2024-04-14T15:09:49.128136+0000","last_became_peered":"2024-04-14T15:09:49.128136+0000","last_unstale":"2024-04-14T15:20:12.934578+0000","last_undegraded":"2024-04-14T15:20:12.934578+0000","last_fullsized":"2024-04-14T15:20:12.934578+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:00.976659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:56.206649+0000","last_clean_scrub_stamp":"2024-04-14T15:20:00.976659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:01.483885+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"91'130","reported_seq":981,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.934877+0000","last_change":"2024-04-14T15:19:09.060997+0000","last_active":"2024-04-14T15:20:12.934877+0000","last_peered":"2024-04-14T15:20:12.934877+0000","last_clean":"2024-04-14T15:20:12.934877+0000","last_became_active":"2024-04-14T15:18:40.910133+0000","last_became_peered":"2024-04-14T15:18:40.910133+0000","last_unstale":"2024-04-14T15:20:12.934877+0000","last_undegraded":"2024-04-14T15:20:12.934877+0000","last_fullsized":"2024-04-14T15:20:12.934877+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:09.060906+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:09.060906+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:16.750116+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":520,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.936018+0000","last_change":"2024-04-14T15:19:47.610791+0000","last_active":"2024-04-14T15:20:12.936018+0000","last_peered":"2024-04-14T15:20:12.936018+0000","last_clean":"2024-04-14T15:20:12.936018+0000","last_became_active":"2024-04-14T15:15:21.887769+0000","last_became_peered":"2024-04-14T15:15:21.887769+0000","last_unstale":"2024-04-14T15:20:12.936018+0000","last_undegraded":"2024-04-14T15:20:12.936018+0000","last_fullsized":"2024-04-14T15:20:12.936018+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:47.610667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:19:47.610667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:55.096189+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":462,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.936933+0000","last_change":"2024-04-14T15:17:47.871801+0000","last_active":"2024-04-14T15:20:12.936933+0000","last_peered":"2024-04-14T15:20:12.936933+0000","last_clean":"2024-04-14T15:20:12.936933+0000","last_became_active":"2024-04-14T15:15:44.883694+0000","last_became_peered":"2024-04-14T15:15:44.883694+0000","last_unstale":"2024-04-14T15:20:12.936933+0000","last_undegraded":"2024-04-14T15:20:12.936933+0000","last_fullsized":"2024-04-14T15:20:12.936933+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:47.871722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:17:47.871722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:15.771501+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"479'30","reported_seq":948,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.338258+0000","last_change":"2024-04-14T15:17:44.919445+0000","last_active":"2024-04-14T15:20:13.338258+0000","last_peered":"2024-04-14T15:20:13.338258+0000","last_clean":"2024-04-14T15:20:13.338258+0000","last_became_active":"2024-04-14T15:09:45.037231+0000","last_became_peered":"2024-04-14T15:09:45.037231+0000","last_unstale":"2024-04-14T15:20:13.338258+0000","last_undegraded":"2024-04-14T15:20:13.338258+0000","last_fullsized":"2024-04-14T15:20:13.338258+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'18","last_scrub_stamp":"2024-04-14T15:17:44.919358+0000","last_deep_scrub":"182'18","last_deep_scrub_stamp":"2024-04-14T15:14:10.383906+0000","last_clean_scrub_stamp":"2024-04-14T15:17:44.919358+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:17.378192+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":910,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.659806+0000","last_change":"2024-04-14T15:17:46.542125+0000","last_active":"2024-04-14T15:20:13.659806+0000","last_peered":"2024-04-14T15:20:13.659806+0000","last_clean":"2024-04-14T15:20:13.659806+0000","last_became_active":"2024-04-14T15:09:43.025095+0000","last_became_peered":"2024-04-14T15:09:43.025095+0000","last_unstale":"2024-04-14T15:20:13.659806+0000","last_undegraded":"2024-04-14T15:20:13.659806+0000","last_fullsized":"2024-04-14T15:20:13.659806+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:46.542030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:46.542030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:18.657814+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":918,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.338432+0000","last_change":"2024-04-14T15:17:48.866222+0000","last_active":"2024-04-14T15:20:13.338432+0000","last_peered":"2024-04-14T15:20:13.338432+0000","last_clean":"2024-04-14T15:20:13.338432+0000","last_became_active":"2024-04-14T15:09:43.018509+0000","last_became_peered":"2024-04-14T15:09:43.018509+0000","last_unstale":"2024-04-14T15:20:13.338432+0000","last_undegraded":"2024-04-14T15:20:13.338432+0000","last_fullsized":"2024-04-14T15:20:13.338432+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:48.866146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:48.866146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:17.378227+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0018592680000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"355'2","reported_seq":901,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.935938+0000","last_change":"2024-04-14T15:20:04.842983+0000","last_active":"2024-04-14T15:20:12.935938+0000","last_peered":"2024-04-14T15:20:12.935938+0000","last_clean":"2024-04-14T15:20:12.935938+0000","last_became_active":"2024-04-14T15:09:49.065243+0000","last_became_peered":"2024-04-14T15:09:49.065243+0000","last_unstale":"2024-04-14T15:20:12.935938+0000","last_undegraded":"2024-04-14T15:20:12.935938+0000","last_fullsized":"2024-04-14T15:20:12.935938+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'2","last_scrub_stamp":"2024-04-14T15:20:04.842881+0000","last_deep_scrub":"355'2","last_deep_scrub_stamp":"2024-04-14T15:20:04.842881+0000","last_clean_scrub_stamp":"2024-04-14T15:20:04.842881+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:06.921319+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":504,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.936581+0000","last_change":"2024-04-14T15:18:32.532077+0000","last_active":"2024-04-14T15:20:12.936581+0000","last_peered":"2024-04-14T15:20:12.936581+0000","last_clean":"2024-04-14T15:20:12.936581+0000","last_became_active":"2024-04-14T15:15:21.890115+0000","last_became_peered":"2024-04-14T15:15:21.890115+0000","last_unstale":"2024-04-14T15:20:12.936581+0000","last_undegraded":"2024-04-14T15:20:12.936581+0000","last_fullsized":"2024-04-14T15:20:12.936581+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:32.531993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:18:32.531993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:13.731221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":901,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.660776+0000","last_change":"2024-04-14T15:19:46.631011+0000","last_active":"2024-04-14T15:20:13.660776+0000","last_peered":"2024-04-14T15:20:13.660776+0000","last_clean":"2024-04-14T15:20:13.660776+0000","last_became_active":"2024-04-14T15:09:47.047201+0000","last_became_peered":"2024-04-14T15:09:47.047201+0000","last_unstale":"2024-04-14T15:20:13.660776+0000","last_undegraded":"2024-04-14T15:20:13.660776+0000","last_fullsized":"2024-04-14T15:20:13.660776+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:46.630886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:19:46.630886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:47.188067+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"479'36","reported_seq":1082,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.748119+0000","last_change":"2024-04-14T15:19:02.841789+0000","last_active":"2024-04-14T15:20:13.748119+0000","last_peered":"2024-04-14T15:20:13.748119+0000","last_clean":"2024-04-14T15:20:13.748119+0000","last_became_active":"2024-04-14T15:09:45.179978+0000","last_became_peered":"2024-04-14T15:09:45.179978+0000","last_unstale":"2024-04-14T15:20:13.748119+0000","last_undegraded":"2024-04-14T15:20:13.748119+0000","last_fullsized":"2024-04-14T15:20:13.748119+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'24","last_scrub_stamp":"2024-04-14T15:19:02.841668+0000","last_deep_scrub":"90'24","last_deep_scrub_stamp":"2024-04-14T15:11:25.120692+0000","last_clean_scrub_stamp":"2024-04-14T15:19:02.841668+0000","objects_scrubbed":8,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:20.808249+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"91'130","reported_seq":1191,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.659796+0000","last_change":"2024-04-14T15:19:08.494753+0000","last_active":"2024-04-14T15:20:13.659796+0000","last_peered":"2024-04-14T15:20:13.659796+0000","last_clean":"2024-04-14T15:20:13.659796+0000","last_became_active":"2024-04-14T15:18:40.911206+0000","last_became_peered":"2024-04-14T15:18:40.911206+0000","last_unstale":"2024-04-14T15:20:13.659796+0000","last_undegraded":"2024-04-14T15:20:13.659796+0000","last_fullsized":"2024-04-14T15:20:13.659796+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:08.494666+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:08.494666+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:25.546717+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":467,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.386077+0000","last_change":"2024-04-14T15:17:35.705777+0000","last_active":"2024-04-14T15:20:13.386077+0000","last_peered":"2024-04-14T15:20:13.386077+0000","last_clean":"2024-04-14T15:20:13.386077+0000","last_became_active":"2024-04-14T15:15:44.885315+0000","last_became_peered":"2024-04-14T15:15:44.885315+0000","last_unstale":"2024-04-14T15:20:13.386077+0000","last_undegraded":"2024-04-14T15:20:13.386077+0000","last_fullsized":"2024-04-14T15:20:13.386077+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:35.705645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:47.627305+0000","last_clean_scrub_stamp":"2024-04-14T15:17:35.705645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:15.826530+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"69'2","reported_seq":594,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.936591+0000","last_change":"2024-04-14T15:16:53.972591+0000","last_active":"2024-04-14T15:20:12.936591+0000","last_peered":"2024-04-14T15:20:12.936591+0000","last_clean":"2024-04-14T15:20:12.936591+0000","last_became_active":"2024-04-14T15:15:21.889851+0000","last_became_peered":"2024-04-14T15:15:21.889851+0000","last_unstale":"2024-04-14T15:20:12.936591+0000","last_undegraded":"2024-04-14T15:20:12.936591+0000","last_fullsized":"2024-04-14T15:20:12.936591+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-04-14T15:16:53.972502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:53.972502+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:13.731048+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0058046169999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":467,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.937815+0000","last_change":"2024-04-14T15:20:03.807964+0000","last_active":"2024-04-14T15:20:12.937815+0000","last_peered":"2024-04-14T15:20:12.937815+0000","last_clean":"2024-04-14T15:20:12.937815+0000","last_became_active":"2024-04-14T15:15:44.887934+0000","last_became_peered":"2024-04-14T15:15:44.887934+0000","last_unstale":"2024-04-14T15:20:12.937815+0000","last_undegraded":"2024-04-14T15:20:12.937815+0000","last_fullsized":"2024-04-14T15:20:12.937815+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:03.807844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:20:03.807844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:27.650888+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"91'130","reported_seq":1198,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.659626+0000","last_change":"2024-04-14T15:19:07.501791+0000","last_active":"2024-04-14T15:20:13.659626+0000","last_peered":"2024-04-14T15:20:13.659626+0000","last_clean":"2024-04-14T15:20:13.659626+0000","last_became_active":"2024-04-14T15:18:40.912277+0000","last_became_peered":"2024-04-14T15:18:40.912277+0000","last_unstale":"2024-04-14T15:20:13.659626+0000","last_undegraded":"2024-04-14T15:20:13.659626+0000","last_fullsized":"2024-04-14T15:20:13.659626+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:07.501704+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:07.501704+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:18.658069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":512,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.935536+0000","last_change":"2024-04-14T15:19:46.589280+0000","last_active":"2024-04-14T15:20:12.935536+0000","last_peered":"2024-04-14T15:20:12.935536+0000","last_clean":"2024-04-14T15:20:12.935536+0000","last_became_active":"2024-04-14T15:15:21.889970+0000","last_became_peered":"2024-04-14T15:15:21.889970+0000","last_unstale":"2024-04-14T15:20:12.935536+0000","last_undegraded":"2024-04-14T15:20:12.935536+0000","last_fullsized":"2024-04-14T15:20:12.935536+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:46.589189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:19:46.589189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:15.849303+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":908,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.386489+0000","last_change":"2024-04-14T15:20:00.850190+0000","last_active":"2024-04-14T15:20:13.386489+0000","last_peered":"2024-04-14T15:20:13.386489+0000","last_clean":"2024-04-14T15:20:13.386489+0000","last_became_active":"2024-04-14T15:09:47.049457+0000","last_became_peered":"2024-04-14T15:09:47.049457+0000","last_unstale":"2024-04-14T15:20:13.386489+0000","last_undegraded":"2024-04-14T15:20:13.386489+0000","last_fullsized":"2024-04-14T15:20:13.386489+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:00.850117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:20:00.850117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:29.884274+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"479'22","reported_seq":945,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.936845+0000","last_change":"2024-04-14T15:19:26.078482+0000","last_active":"2024-04-14T15:20:12.936845+0000","last_peered":"2024-04-14T15:20:12.936845+0000","last_clean":"2024-04-14T15:20:12.936845+0000","last_became_active":"2024-04-14T15:09:45.033883+0000","last_became_peered":"2024-04-14T15:09:45.033883+0000","last_unstale":"2024-04-14T15:20:12.936845+0000","last_undegraded":"2024-04-14T15:20:12.936845+0000","last_fullsized":"2024-04-14T15:20:12.936845+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"308'18","last_scrub_stamp":"2024-04-14T15:19:26.078396+0000","last_deep_scrub":"308'18","last_deep_scrub_stamp":"2024-04-14T15:16:16.054032+0000","last_clean_scrub_stamp":"2024-04-14T15:19:26.078396+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:27.837657+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":917,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.385586+0000","last_change":"2024-04-14T15:19:16.788873+0000","last_active":"2024-04-14T15:20:13.385586+0000","last_peered":"2024-04-14T15:20:13.385586+0000","last_clean":"2024-04-14T15:20:13.385586+0000","last_became_active":"2024-04-14T15:09:43.021188+0000","last_became_peered":"2024-04-14T15:09:43.021188+0000","last_unstale":"2024-04-14T15:20:13.385586+0000","last_undegraded":"2024-04-14T15:20:13.385586+0000","last_fullsized":"2024-04-14T15:20:13.385586+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:16.788795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:19:16.788795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:35.846721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"73'5","reported_seq":1546,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.934971+0000","last_change":"2024-04-14T15:18:26.955802+0000","last_active":"2024-04-14T15:20:12.934971+0000","last_peered":"2024-04-14T15:20:12.934971+0000","last_clean":"2024-04-14T15:20:12.934971+0000","last_became_active":"2024-04-14T15:09:43.016544+0000","last_became_peered":"2024-04-14T15:09:43.016544+0000","last_unstale":"2024-04-14T15:20:12.934971+0000","last_undegraded":"2024-04-14T15:20:12.934971+0000","last_fullsized":"2024-04-14T15:20:12.934971+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-04-14T15:18:26.955711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:18:26.955711+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:15.771620+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00056542099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":624,"num_read_kb":619,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":897,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.659167+0000","last_change":"2024-04-14T15:19:22.535232+0000","last_active":"2024-04-14T15:20:13.659167+0000","last_peered":"2024-04-14T15:20:13.659167+0000","last_clean":"2024-04-14T15:20:13.659167+0000","last_became_active":"2024-04-14T15:09:49.130307+0000","last_became_peered":"2024-04-14T15:09:49.130307+0000","last_unstale":"2024-04-14T15:20:13.659167+0000","last_undegraded":"2024-04-14T15:20:13.659167+0000","last_fullsized":"2024-04-14T15:20:13.659167+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:22.535145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:20.827589+0000","last_clean_scrub_stamp":"2024-04-14T15:19:22.535145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:35.538494+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"91'130","reported_seq":812,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.337480+0000","last_change":"2024-04-14T15:19:06.124178+0000","last_active":"2024-04-14T15:20:13.337480+0000","last_peered":"2024-04-14T15:20:13.337480+0000","last_clean":"2024-04-14T15:20:13.337480+0000","last_became_active":"2024-04-14T15:18:40.913215+0000","last_became_peered":"2024-04-14T15:18:40.913215+0000","last_unstale":"2024-04-14T15:20:13.337480+0000","last_undegraded":"2024-04-14T15:20:13.337480+0000","last_fullsized":"2024-04-14T15:20:13.337480+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:06.124071+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:06.124071+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:23.152389+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":512,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.937284+0000","last_change":"2024-04-14T15:19:15.372917+0000","last_active":"2024-04-14T15:20:12.937284+0000","last_peered":"2024-04-14T15:20:12.937284+0000","last_clean":"2024-04-14T15:20:12.937284+0000","last_became_active":"2024-04-14T15:15:21.883717+0000","last_became_peered":"2024-04-14T15:15:21.883717+0000","last_unstale":"2024-04-14T15:20:12.937284+0000","last_undegraded":"2024-04-14T15:20:12.937284+0000","last_fullsized":"2024-04-14T15:20:12.937284+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:15.372837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:19:15.372837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:42.207119+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":903,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.338302+0000","last_change":"2024-04-14T15:19:25.297173+0000","last_active":"2024-04-14T15:20:13.338302+0000","last_peered":"2024-04-14T15:20:13.338302+0000","last_clean":"2024-04-14T15:20:13.338302+0000","last_became_active":"2024-04-14T15:09:47.040185+0000","last_became_peered":"2024-04-14T15:09:47.040185+0000","last_unstale":"2024-04-14T15:20:13.338302+0000","last_undegraded":"2024-04-14T15:20:13.338302+0000","last_fullsized":"2024-04-14T15:20:13.338302+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:25.297044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:53.731610+0000","last_clean_scrub_stamp":"2024-04-14T15:19:25.297044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:25.349894+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"479'40","reported_seq":1101,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.188811+0000","last_change":"2024-04-14T15:19:49.915826+0000","last_active":"2024-04-14T15:20:13.188811+0000","last_peered":"2024-04-14T15:20:13.188811+0000","last_clean":"2024-04-14T15:20:13.188811+0000","last_became_active":"2024-04-14T15:09:45.034168+0000","last_became_peered":"2024-04-14T15:09:45.034168+0000","last_unstale":"2024-04-14T15:20:13.188811+0000","last_undegraded":"2024-04-14T15:20:13.188811+0000","last_fullsized":"2024-04-14T15:20:13.188811+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'28","last_scrub_stamp":"2024-04-14T15:19:49.915672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:19:49.915672+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:54.452998+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":470,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.934726+0000","last_change":"2024-04-14T15:19:24.066365+0000","last_active":"2024-04-14T15:20:12.934726+0000","last_peered":"2024-04-14T15:20:12.934726+0000","last_clean":"2024-04-14T15:20:12.934726+0000","last_became_active":"2024-04-14T15:15:44.885315+0000","last_became_peered":"2024-04-14T15:15:44.885315+0000","last_unstale":"2024-04-14T15:20:12.934726+0000","last_undegraded":"2024-04-14T15:20:12.934726+0000","last_fullsized":"2024-04-14T15:20:12.934726+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:24.066275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:24.066275+0000","last_clean_scrub_stamp":"2024-04-14T15:19:24.066275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:43.166123+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":564,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.937802+0000","last_change":"2024-04-14T15:17:44.084142+0000","last_active":"2024-04-14T15:20:12.937802+0000","last_peered":"2024-04-14T15:20:12.937802+0000","last_clean":"2024-04-14T15:20:12.937802+0000","last_became_active":"2024-04-14T15:15:21.874742+0000","last_became_peered":"2024-04-14T15:15:21.874742+0000","last_unstale":"2024-04-14T15:20:12.937802+0000","last_undegraded":"2024-04-14T15:20:12.937802+0000","last_fullsized":"2024-04-14T15:20:12.937802+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:44.084010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:44.084010+0000","last_clean_scrub_stamp":"2024-04-14T15:17:44.084010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:13.731116+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"91'130","reported_seq":992,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.748382+0000","last_change":"2024-04-14T15:19:07.770220+0000","last_active":"2024-04-14T15:20:13.748382+0000","last_peered":"2024-04-14T15:20:13.748382+0000","last_clean":"2024-04-14T15:20:13.748382+0000","last_became_active":"2024-04-14T15:18:58.189402+0000","last_became_peered":"2024-04-14T15:18:58.189402+0000","last_unstale":"2024-04-14T15:20:13.748382+0000","last_undegraded":"2024-04-14T15:20:13.748382+0000","last_fullsized":"2024-04-14T15:20:13.748382+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:07.770125+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:07.770125+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:14.820318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":813,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.386120+0000","last_change":"2024-04-14T15:19:55.858944+0000","last_active":"2024-04-14T15:20:13.386120+0000","last_peered":"2024-04-14T15:20:13.386120+0000","last_clean":"2024-04-14T15:20:13.386120+0000","last_became_active":"2024-04-14T15:12:56.350703+0000","last_became_peered":"2024-04-14T15:12:56.350703+0000","last_unstale":"2024-04-14T15:20:13.386120+0000","last_undegraded":"2024-04-14T15:20:13.386120+0000","last_fullsized":"2024-04-14T15:20:13.386120+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:55.858835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:19:55.858835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:09.317978+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"72'8","reported_seq":1046,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.337808+0000","last_change":"2024-04-14T15:20:02.298437+0000","last_active":"2024-04-14T15:20:13.337808+0000","last_peered":"2024-04-14T15:20:13.337808+0000","last_clean":"2024-04-14T15:20:13.337808+0000","last_became_active":"2024-04-14T15:09:47.047699+0000","last_became_peered":"2024-04-14T15:09:47.047699+0000","last_unstale":"2024-04-14T15:20:13.337808+0000","last_undegraded":"2024-04-14T15:20:13.337808+0000","last_fullsized":"2024-04-14T15:20:13.337808+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:20:02.298301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:20:02.298301+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:16.534076+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"479'20","reported_seq":913,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.385937+0000","last_change":"2024-04-14T15:18:38.870798+0000","last_active":"2024-04-14T15:20:13.385937+0000","last_peered":"2024-04-14T15:20:13.385937+0000","last_clean":"2024-04-14T15:20:13.385937+0000","last_became_active":"2024-04-14T15:10:07.764042+0000","last_became_peered":"2024-04-14T15:10:07.764042+0000","last_unstale":"2024-04-14T15:20:13.385937+0000","last_undegraded":"2024-04-14T15:20:13.385937+0000","last_fullsized":"2024-04-14T15:20:13.385937+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-04-14T15:18:38.870710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:18:38.870710+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:15.826694+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":904,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.936040+0000","last_change":"2024-04-14T15:17:40.971492+0000","last_active":"2024-04-14T15:20:12.936040+0000","last_peered":"2024-04-14T15:20:12.936040+0000","last_clean":"2024-04-14T15:20:12.936040+0000","last_became_active":"2024-04-14T15:09:43.022597+0000","last_became_peered":"2024-04-14T15:09:43.022597+0000","last_unstale":"2024-04-14T15:20:12.936040+0000","last_undegraded":"2024-04-14T15:20:12.936040+0000","last_fullsized":"2024-04-14T15:20:12.936040+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:40.971412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:40.971412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:15.771450+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":899,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.337164+0000","last_change":"2024-04-14T15:19:16.390131+0000","last_active":"2024-04-14T15:20:13.337164+0000","last_peered":"2024-04-14T15:20:13.337164+0000","last_clean":"2024-04-14T15:20:13.337164+0000","last_became_active":"2024-04-14T15:09:49.066648+0000","last_became_peered":"2024-04-14T15:09:49.066648+0000","last_unstale":"2024-04-14T15:20:13.337164+0000","last_undegraded":"2024-04-14T15:20:13.337164+0000","last_fullsized":"2024-04-14T15:20:13.337164+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:16.390049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:19:16.390049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:36.162522+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.c","version":"91'130","reported_seq":997,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.937212+0000","last_change":"2024-04-14T15:19:04.397426+0000","last_active":"2024-04-14T15:20:12.937212+0000","last_peered":"2024-04-14T15:20:12.937212+0000","last_clean":"2024-04-14T15:20:12.937212+0000","last_became_active":"2024-04-14T15:18:58.186036+0000","last_became_peered":"2024-04-14T15:18:58.186036+0000","last_unstale":"2024-04-14T15:20:12.937212+0000","last_undegraded":"2024-04-14T15:20:12.937212+0000","last_fullsized":"2024-04-14T15:20:12.937212+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:04.397343+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:04.397343+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:31.837520+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":807,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.338162+0000","last_change":"2024-04-14T15:18:27.133391+0000","last_active":"2024-04-14T15:20:13.338162+0000","last_peered":"2024-04-14T15:20:13.338162+0000","last_clean":"2024-04-14T15:20:13.338162+0000","last_became_active":"2024-04-14T15:12:56.563074+0000","last_became_peered":"2024-04-14T15:12:56.563074+0000","last_unstale":"2024-04-14T15:20:13.338162+0000","last_undegraded":"2024-04-14T15:20:13.338162+0000","last_fullsized":"2024-04-14T15:20:13.338162+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:27.133310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:18:27.133310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:17.378260+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":900,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.385382+0000","last_change":"2024-04-14T15:17:56.915660+0000","last_active":"2024-04-14T15:20:13.385382+0000","last_peered":"2024-04-14T15:20:13.385382+0000","last_clean":"2024-04-14T15:20:13.385382+0000","last_became_active":"2024-04-14T15:09:47.049005+0000","last_became_peered":"2024-04-14T15:09:47.049005+0000","last_unstale":"2024-04-14T15:20:13.385382+0000","last_undegraded":"2024-04-14T15:20:13.385382+0000","last_fullsized":"2024-04-14T15:20:13.385382+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:56.915544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:56.915544+0000","last_clean_scrub_stamp":"2024-04-14T15:17:56.915544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:15.826663+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"479'23","reported_seq":950,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.934668+0000","last_change":"2024-04-14T15:19:44.525830+0000","last_active":"2024-04-14T15:20:12.934668+0000","last_peered":"2024-04-14T15:20:12.934668+0000","last_clean":"2024-04-14T15:20:12.934668+0000","last_became_active":"2024-04-14T15:09:45.032892+0000","last_became_peered":"2024-04-14T15:09:45.032892+0000","last_unstale":"2024-04-14T15:20:12.934668+0000","last_undegraded":"2024-04-14T15:20:12.934668+0000","last_fullsized":"2024-04-14T15:20:12.934668+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'17","last_scrub_stamp":"2024-04-14T15:19:44.525712+0000","last_deep_scrub":"183'17","last_deep_scrub_stamp":"2024-04-14T15:14:11.577574+0000","last_clean_scrub_stamp":"2024-04-14T15:19:44.525712+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:47.139793+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"65'3","reported_seq":927,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.934863+0000","last_change":"2024-04-14T15:19:22.603785+0000","last_active":"2024-04-14T15:20:12.934863+0000","last_peered":"2024-04-14T15:20:12.934863+0000","last_clean":"2024-04-14T15:20:12.934863+0000","last_became_active":"2024-04-14T15:09:43.020490+0000","last_became_peered":"2024-04-14T15:09:43.020490+0000","last_unstale":"2024-04-14T15:20:12.934863+0000","last_undegraded":"2024-04-14T15:20:12.934863+0000","last_fullsized":"2024-04-14T15:20:12.934863+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3","last_scrub_stamp":"2024-04-14T15:19:22.603651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:19:22.603651+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:27.010861+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":528,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":564,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.937627+0000","last_change":"2024-04-14T15:16:58.942591+0000","last_active":"2024-04-14T15:20:12.937627+0000","last_peered":"2024-04-14T15:20:12.937627+0000","last_clean":"2024-04-14T15:20:12.937627+0000","last_became_active":"2024-04-14T15:15:21.876410+0000","last_became_peered":"2024-04-14T15:15:21.876410+0000","last_unstale":"2024-04-14T15:20:12.937627+0000","last_undegraded":"2024-04-14T15:20:12.937627+0000","last_fullsized":"2024-04-14T15:20:12.937627+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:58.942499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:58.942499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:13.730968+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"91'130","reported_seq":989,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.934494+0000","last_change":"2024-04-14T15:19:07.091520+0000","last_active":"2024-04-14T15:20:12.934494+0000","last_peered":"2024-04-14T15:20:12.934494+0000","last_clean":"2024-04-14T15:20:12.934494+0000","last_became_active":"2024-04-14T15:18:45.912049+0000","last_became_peered":"2024-04-14T15:18:45.912049+0000","last_unstale":"2024-04-14T15:20:12.934494+0000","last_undegraded":"2024-04-14T15:20:12.934494+0000","last_fullsized":"2024-04-14T15:20:12.934494+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:07.091439+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:07.091439+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:25.022161+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":815,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.338309+0000","last_change":"2024-04-14T15:19:24.289915+0000","last_active":"2024-04-14T15:20:13.338309+0000","last_peered":"2024-04-14T15:20:13.338309+0000","last_clean":"2024-04-14T15:20:13.338309+0000","last_became_active":"2024-04-14T15:12:56.551177+0000","last_became_peered":"2024-04-14T15:12:56.551177+0000","last_unstale":"2024-04-14T15:20:13.338309+0000","last_undegraded":"2024-04-14T15:20:13.338309+0000","last_fullsized":"2024-04-14T15:20:13.338309+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:24.289806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:19:24.289806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:44.967529+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"479'19","reported_seq":935,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.338369+0000","last_change":"2024-04-14T15:19:51.290310+0000","last_active":"2024-04-14T15:20:13.338369+0000","last_peered":"2024-04-14T15:20:13.338369+0000","last_clean":"2024-04-14T15:20:13.338369+0000","last_became_active":"2024-04-14T15:09:45.180375+0000","last_became_peered":"2024-04-14T15:09:45.180375+0000","last_unstale":"2024-04-14T15:20:13.338369+0000","last_undegraded":"2024-04-14T15:20:13.338369+0000","last_fullsized":"2024-04-14T15:20:13.338369+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'19","last_scrub_stamp":"2024-04-14T15:19:51.290211+0000","last_deep_scrub":"347'13","last_deep_scrub_stamp":"2024-04-14T15:16:55.694599+0000","last_clean_scrub_stamp":"2024-04-14T15:19:51.290211+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:18.067434+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":916,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.385380+0000","last_change":"2024-04-14T15:19:21.774315+0000","last_active":"2024-04-14T15:20:13.385380+0000","last_peered":"2024-04-14T15:20:13.385380+0000","last_clean":"2024-04-14T15:20:13.385380+0000","last_became_active":"2024-04-14T15:09:43.016788+0000","last_became_peered":"2024-04-14T15:09:43.016788+0000","last_unstale":"2024-04-14T15:20:13.385380+0000","last_undegraded":"2024-04-14T15:20:13.385380+0000","last_fullsized":"2024-04-14T15:20:13.385380+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:21.774184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:19:21.774184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:48.086427+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":584,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.937831+0000","last_change":"2024-04-14T15:18:40.385814+0000","last_active":"2024-04-14T15:20:12.937831+0000","last_peered":"2024-04-14T15:20:12.937831+0000","last_clean":"2024-04-14T15:20:12.937831+0000","last_became_active":"2024-04-14T15:15:21.873580+0000","last_became_peered":"2024-04-14T15:15:21.873580+0000","last_unstale":"2024-04-14T15:20:12.937831+0000","last_undegraded":"2024-04-14T15:20:12.937831+0000","last_fullsized":"2024-04-14T15:20:12.937831+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:40.385736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:18:40.385736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":904,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.384936+0000","last_change":"2024-04-14T15:19:24.821597+0000","last_active":"2024-04-14T15:20:13.384936+0000","last_peered":"2024-04-14T15:20:13.384936+0000","last_clean":"2024-04-14T15:20:13.384936+0000","last_became_active":"2024-04-14T15:09:49.064296+0000","last_became_peered":"2024-04-14T15:09:49.064296+0000","last_unstale":"2024-04-14T15:20:13.384936+0000","last_undegraded":"2024-04-14T15:20:13.384936+0000","last_fullsized":"2024-04-14T15:20:13.384936+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:24.821514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:19:24.821514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:39.516148+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.12","version":"91'130","reported_seq":461,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.934545+0000","last_change":"2024-04-14T15:19:06.098758+0000","last_active":"2024-04-14T15:20:12.934545+0000","last_peered":"2024-04-14T15:20:12.934545+0000","last_clean":"2024-04-14T15:20:12.934545+0000","last_became_active":"2024-04-14T15:18:45.918413+0000","last_became_peered":"2024-04-14T15:18:45.918413+0000","last_unstale":"2024-04-14T15:20:12.934545+0000","last_undegraded":"2024-04-14T15:20:12.934545+0000","last_fullsized":"2024-04-14T15:20:12.934545+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:06.098672+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:06.098672+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:32.614691+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":805,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.385325+0000","last_change":"2024-04-14T15:17:36.728543+0000","last_active":"2024-04-14T15:20:13.385325+0000","last_peered":"2024-04-14T15:20:13.385325+0000","last_clean":"2024-04-14T15:20:13.385325+0000","last_became_active":"2024-04-14T15:12:56.340066+0000","last_became_peered":"2024-04-14T15:12:56.340066+0000","last_unstale":"2024-04-14T15:20:13.385325+0000","last_undegraded":"2024-04-14T15:20:13.385325+0000","last_fullsized":"2024-04-14T15:20:13.385325+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:36.728465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:17:36.728465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:15.826547+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.17","version":"479'12","reported_seq":921,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.937638+0000","last_change":"2024-04-14T15:17:42.019838+0000","last_active":"2024-04-14T15:20:12.937638+0000","last_peered":"2024-04-14T15:20:12.937638+0000","last_clean":"2024-04-14T15:20:12.937638+0000","last_became_active":"2024-04-14T15:09:45.180037+0000","last_became_peered":"2024-04-14T15:09:45.180037+0000","last_unstale":"2024-04-14T15:20:12.937638+0000","last_undegraded":"2024-04-14T15:20:12.937638+0000","last_fullsized":"2024-04-14T15:20:12.937638+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'6","last_scrub_stamp":"2024-04-14T15:17:42.019717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:17:42.019717+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:13.731066+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":918,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.747390+0000","last_change":"2024-04-14T15:19:23.915584+0000","last_active":"2024-04-14T15:20:13.747390+0000","last_peered":"2024-04-14T15:20:13.747390+0000","last_clean":"2024-04-14T15:20:13.747390+0000","last_became_active":"2024-04-14T15:09:43.018987+0000","last_became_peered":"2024-04-14T15:09:43.018987+0000","last_unstale":"2024-04-14T15:20:13.747390+0000","last_undegraded":"2024-04-14T15:20:13.747390+0000","last_fullsized":"2024-04-14T15:20:13.747390+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:23.915503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:59.640565+0000","last_clean_scrub_stamp":"2024-04-14T15:19:23.915503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:36.396647+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":457,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.937705+0000","last_change":"2024-04-14T15:17:47.072247+0000","last_active":"2024-04-14T15:20:12.937705+0000","last_peered":"2024-04-14T15:20:12.937705+0000","last_clean":"2024-04-14T15:20:12.937705+0000","last_became_active":"2024-04-14T15:15:44.889786+0000","last_became_peered":"2024-04-14T15:15:44.889786+0000","last_unstale":"2024-04-14T15:20:12.937705+0000","last_undegraded":"2024-04-14T15:20:12.937705+0000","last_fullsized":"2024-04-14T15:20:12.937705+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:47.072170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:13.735024+0000","last_clean_scrub_stamp":"2024-04-14T15:17:47.072170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:13.731101+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":897,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.659164+0000","last_change":"2024-04-14T15:19:47.597543+0000","last_active":"2024-04-14T15:20:13.659164+0000","last_peered":"2024-04-14T15:20:13.659164+0000","last_clean":"2024-04-14T15:20:13.659164+0000","last_became_active":"2024-04-14T15:09:49.130438+0000","last_became_peered":"2024-04-14T15:09:49.130438+0000","last_unstale":"2024-04-14T15:20:13.659164+0000","last_undegraded":"2024-04-14T15:20:13.659164+0000","last_fullsized":"2024-04-14T15:20:13.659164+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:47.597268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:19:47.597268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:59.841711+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.11","version":"91'130","reported_seq":819,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.337475+0000","last_change":"2024-04-14T15:19:05.136092+0000","last_active":"2024-04-14T15:20:13.337475+0000","last_peered":"2024-04-14T15:20:13.337475+0000","last_clean":"2024-04-14T15:20:13.337475+0000","last_became_active":"2024-04-14T15:18:45.915385+0000","last_became_peered":"2024-04-14T15:18:45.915385+0000","last_unstale":"2024-04-14T15:20:13.337475+0000","last_undegraded":"2024-04-14T15:20:13.337475+0000","last_fullsized":"2024-04-14T15:20:13.337475+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:05.136001+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:05.136001+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:15.595631+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":805,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.935113+0000","last_change":"2024-04-14T15:17:46.369320+0000","last_active":"2024-04-14T15:20:12.935113+0000","last_peered":"2024-04-14T15:20:12.935113+0000","last_clean":"2024-04-14T15:20:12.935113+0000","last_became_active":"2024-04-14T15:12:56.203577+0000","last_became_peered":"2024-04-14T15:12:56.203577+0000","last_unstale":"2024-04-14T15:20:12.935113+0000","last_undegraded":"2024-04-14T15:20:12.935113+0000","last_fullsized":"2024-04-14T15:20:12.935113+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:46.369231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:46.369231+0000","last_clean_scrub_stamp":"2024-04-14T15:17:46.369231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:14.561258+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"479'20","reported_seq":618,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.937696+0000","last_change":"2024-04-14T15:18:36.534967+0000","last_active":"2024-04-14T15:20:12.937696+0000","last_peered":"2024-04-14T15:20:12.937696+0000","last_clean":"2024-04-14T15:20:12.937696+0000","last_became_active":"2024-04-14T15:15:21.877667+0000","last_became_peered":"2024-04-14T15:15:21.877667+0000","last_unstale":"2024-04-14T15:20:12.937696+0000","last_undegraded":"2024-04-14T15:20:12.937696+0000","last_fullsized":"2024-04-14T15:20:12.937696+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-04-14T15:18:36.534874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:18:36.534874+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:13.731303+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":465,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.936758+0000","last_change":"2024-04-14T15:19:56.775138+0000","last_active":"2024-04-14T15:20:12.936758+0000","last_peered":"2024-04-14T15:20:12.936758+0000","last_clean":"2024-04-14T15:20:12.936758+0000","last_became_active":"2024-04-14T15:15:44.887452+0000","last_became_peered":"2024-04-14T15:15:44.887452+0000","last_unstale":"2024-04-14T15:20:12.936758+0000","last_undegraded":"2024-04-14T15:20:12.936758+0000","last_fullsized":"2024-04-14T15:20:12.936758+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:56.775036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:19:56.775036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:23.210279+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"280'9","reported_seq":540,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.936816+0000","last_change":"2024-04-14T15:17:50.908130+0000","last_active":"2024-04-14T15:20:12.936816+0000","last_peered":"2024-04-14T15:20:12.936816+0000","last_clean":"2024-04-14T15:20:12.936816+0000","last_became_active":"2024-04-14T15:15:44.882916+0000","last_became_peered":"2024-04-14T15:15:44.882916+0000","last_unstale":"2024-04-14T15:20:12.936816+0000","last_undegraded":"2024-04-14T15:20:12.936816+0000","last_fullsized":"2024-04-14T15:20:12.936816+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"280'9","last_scrub_stamp":"2024-04-14T15:17:50.908054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:17:50.908054+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:15.771519+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":890,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.337176+0000","last_change":"2024-04-14T15:16:54.724807+0000","last_active":"2024-04-14T15:20:13.337176+0000","last_peered":"2024-04-14T15:20:13.337176+0000","last_clean":"2024-04-14T15:20:13.337176+0000","last_became_active":"2024-04-14T15:09:49.066068+0000","last_became_peered":"2024-04-14T15:09:49.066068+0000","last_unstale":"2024-04-14T15:20:13.337176+0000","last_undegraded":"2024-04-14T15:20:13.337176+0000","last_fullsized":"2024-04-14T15:20:13.337176+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:54.724681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:54.724681+0000","last_clean_scrub_stamp":"2024-04-14T15:16:54.724681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:17.378084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"91'130","reported_seq":1198,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.659411+0000","last_change":"2024-04-14T15:19:05.496449+0000","last_active":"2024-04-14T15:20:13.659411+0000","last_peered":"2024-04-14T15:20:13.659411+0000","last_clean":"2024-04-14T15:20:13.659411+0000","last_became_active":"2024-04-14T15:18:45.917220+0000","last_became_peered":"2024-04-14T15:18:45.917220+0000","last_unstale":"2024-04-14T15:20:13.659411+0000","last_undegraded":"2024-04-14T15:20:13.659411+0000","last_fullsized":"2024-04-14T15:20:13.659411+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:05.496303+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:05.496303+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:22.805010+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":457,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.936383+0000","last_change":"2024-04-14T15:18:30.438135+0000","last_active":"2024-04-14T15:20:12.936383+0000","last_peered":"2024-04-14T15:20:12.936383+0000","last_clean":"2024-04-14T15:20:12.936383+0000","last_became_active":"2024-04-14T15:15:44.889106+0000","last_became_peered":"2024-04-14T15:15:44.889106+0000","last_unstale":"2024-04-14T15:20:12.936383+0000","last_undegraded":"2024-04-14T15:20:12.936383+0000","last_fullsized":"2024-04-14T15:20:12.936383+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:30.438001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:18:30.438001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:13.731237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"479'19","reported_seq":755,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.660723+0000","last_change":"2024-04-14T15:20:07.673953+0000","last_active":"2024-04-14T15:20:13.660723+0000","last_peered":"2024-04-14T15:20:13.660723+0000","last_clean":"2024-04-14T15:20:13.660723+0000","last_became_active":"2024-04-14T15:15:44.884211+0000","last_became_peered":"2024-04-14T15:15:44.884211+0000","last_unstale":"2024-04-14T15:20:13.660723+0000","last_undegraded":"2024-04-14T15:20:13.660723+0000","last_fullsized":"2024-04-14T15:20:13.660723+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'19","last_scrub_stamp":"2024-04-14T15:20:07.673794+0000","last_deep_scrub":"183'13","last_deep_scrub_stamp":"2024-04-14T15:14:11.755252+0000","last_clean_scrub_stamp":"2024-04-14T15:20:07.673794+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:08.415336+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":910,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.337931+0000","last_change":"2024-04-14T15:17:02.670294+0000","last_active":"2024-04-14T15:20:13.337931+0000","last_peered":"2024-04-14T15:20:13.337931+0000","last_clean":"2024-04-14T15:20:13.337931+0000","last_became_active":"2024-04-14T15:09:43.019301+0000","last_became_peered":"2024-04-14T15:09:43.019301+0000","last_unstale":"2024-04-14T15:20:13.337931+0000","last_undegraded":"2024-04-14T15:20:13.337931+0000","last_fullsized":"2024-04-14T15:20:13.337931+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:02.670213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:02.670213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:17.378105+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"255'10","reported_seq":682,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.937982+0000","last_change":"2024-04-14T15:19:59.811456+0000","last_active":"2024-04-14T15:20:12.937982+0000","last_peered":"2024-04-14T15:20:12.937982+0000","last_clean":"2024-04-14T15:20:12.937982+0000","last_became_active":"2024-04-14T15:15:21.884468+0000","last_became_peered":"2024-04-14T15:15:21.884468+0000","last_unstale":"2024-04-14T15:20:12.937982+0000","last_undegraded":"2024-04-14T15:20:12.937982+0000","last_fullsized":"2024-04-14T15:20:12.937982+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'10","last_scrub_stamp":"2024-04-14T15:19:59.811370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:19:59.811370+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:08.120065+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":897,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.935073+0000","last_change":"2024-04-14T15:19:15.642110+0000","last_active":"2024-04-14T15:20:12.935073+0000","last_peered":"2024-04-14T15:20:12.935073+0000","last_clean":"2024-04-14T15:20:12.935073+0000","last_became_active":"2024-04-14T15:09:49.065242+0000","last_became_peered":"2024-04-14T15:09:49.065242+0000","last_unstale":"2024-04-14T15:20:12.935073+0000","last_undegraded":"2024-04-14T15:20:12.935073+0000","last_fullsized":"2024-04-14T15:20:12.935073+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:15.642017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:19:15.642017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:37.747416+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.17","version":"91'130","reported_seq":431,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.338037+0000","last_change":"2024-04-14T15:19:03.143086+0000","last_active":"2024-04-14T15:20:13.338037+0000","last_peered":"2024-04-14T15:20:13.338037+0000","last_clean":"2024-04-14T15:20:13.338037+0000","last_became_active":"2024-04-14T15:19:02.936495+0000","last_became_peered":"2024-04-14T15:19:02.936495+0000","last_unstale":"2024-04-14T15:20:13.338037+0000","last_undegraded":"2024-04-14T15:20:13.338037+0000","last_fullsized":"2024-04-14T15:20:13.338037+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":446,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:03.143008+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:03.143008+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:17.378318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":805,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.660835+0000","last_change":"2024-04-14T15:18:40.735708+0000","last_active":"2024-04-14T15:20:13.660835+0000","last_peered":"2024-04-14T15:20:13.660835+0000","last_clean":"2024-04-14T15:20:13.660835+0000","last_became_active":"2024-04-14T15:12:56.389204+0000","last_became_peered":"2024-04-14T15:12:56.389204+0000","last_unstale":"2024-04-14T15:20:13.660835+0000","last_undegraded":"2024-04-14T15:20:13.660835+0000","last_fullsized":"2024-04-14T15:20:13.660835+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:40.735623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:18:40.735623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:18.657933+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"479'19","reported_seq":446,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.747568+0000","last_change":"2024-04-14T15:19:10.762280+0000","last_active":"2024-04-14T15:20:13.747568+0000","last_peered":"2024-04-14T15:20:13.747568+0000","last_clean":"2024-04-14T15:20:13.747568+0000","last_became_active":"2024-04-14T15:16:24.993130+0000","last_became_peered":"2024-04-14T15:16:24.993130+0000","last_unstale":"2024-04-14T15:20:13.747568+0000","last_undegraded":"2024-04-14T15:20:13.747568+0000","last_fullsized":"2024-04-14T15:20:13.747568+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"344'13","last_scrub_stamp":"2024-04-14T15:19:10.762147+0000","last_deep_scrub":"344'13","last_deep_scrub_stamp":"2024-04-14T15:16:52.738163+0000","last_clean_scrub_stamp":"2024-04-14T15:19:10.762147+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:28.923046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":918,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.748119+0000","last_change":"2024-04-14T15:19:24.898168+0000","last_active":"2024-04-14T15:20:13.748119+0000","last_peered":"2024-04-14T15:20:13.748119+0000","last_clean":"2024-04-14T15:20:13.748119+0000","last_became_active":"2024-04-14T15:09:43.022480+0000","last_became_peered":"2024-04-14T15:09:43.022480+0000","last_unstale":"2024-04-14T15:20:13.748119+0000","last_undegraded":"2024-04-14T15:20:13.748119+0000","last_fullsized":"2024-04-14T15:20:13.748119+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:24.898086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:19:24.898086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:33.891662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":576,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.937459+0000","last_change":"2024-04-14T15:18:33.485536+0000","last_active":"2024-04-14T15:20:12.937459+0000","last_peered":"2024-04-14T15:20:12.937459+0000","last_clean":"2024-04-14T15:20:12.937459+0000","last_became_active":"2024-04-14T15:15:21.882643+0000","last_became_peered":"2024-04-14T15:15:21.882643+0000","last_unstale":"2024-04-14T15:20:12.937459+0000","last_undegraded":"2024-04-14T15:20:12.937459+0000","last_fullsized":"2024-04-14T15:20:12.937459+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:33.485411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:33.485411+0000","last_clean_scrub_stamp":"2024-04-14T15:18:33.485411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:13.731197+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":901,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.337273+0000","last_change":"2024-04-14T15:18:24.039316+0000","last_active":"2024-04-14T15:20:13.337273+0000","last_peered":"2024-04-14T15:20:13.337273+0000","last_clean":"2024-04-14T15:20:13.337273+0000","last_became_active":"2024-04-14T15:09:49.059711+0000","last_became_peered":"2024-04-14T15:09:49.059711+0000","last_unstale":"2024-04-14T15:20:13.337273+0000","last_undegraded":"2024-04-14T15:20:13.337273+0000","last_fullsized":"2024-04-14T15:20:13.337273+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:24.039237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:18:24.039237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:17.378243+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"91'130","reported_seq":428,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.935843+0000","last_change":"2024-04-14T15:19:04.045598+0000","last_active":"2024-04-14T15:20:12.935843+0000","last_peered":"2024-04-14T15:20:12.935843+0000","last_clean":"2024-04-14T15:20:12.935843+0000","last_became_active":"2024-04-14T15:19:02.938648+0000","last_became_peered":"2024-04-14T15:19:02.938648+0000","last_unstale":"2024-04-14T15:20:12.935843+0000","last_undegraded":"2024-04-14T15:20:12.935843+0000","last_fullsized":"2024-04-14T15:20:12.935843+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":446,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:04.045513+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:04.045513+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:30.423540+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":814,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.748041+0000","last_change":"2024-04-14T15:19:26.891939+0000","last_active":"2024-04-14T15:20:13.748041+0000","last_peered":"2024-04-14T15:20:13.748041+0000","last_clean":"2024-04-14T15:20:13.748041+0000","last_became_active":"2024-04-14T15:12:56.396294+0000","last_became_peered":"2024-04-14T15:12:56.396294+0000","last_unstale":"2024-04-14T15:20:13.748041+0000","last_undegraded":"2024-04-14T15:20:13.748041+0000","last_fullsized":"2024-04-14T15:20:13.748041+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:26.891862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:19:26.891862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:48.097098+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"479'24","reported_seq":941,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.935231+0000","last_change":"2024-04-14T15:19:17.590124+0000","last_active":"2024-04-14T15:20:12.935231+0000","last_peered":"2024-04-14T15:20:12.935231+0000","last_clean":"2024-04-14T15:20:12.935231+0000","last_became_active":"2024-04-14T15:09:45.026065+0000","last_became_peered":"2024-04-14T15:09:45.026065+0000","last_unstale":"2024-04-14T15:20:12.935231+0000","last_undegraded":"2024-04-14T15:20:12.935231+0000","last_fullsized":"2024-04-14T15:20:12.935231+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"464'16","last_scrub_stamp":"2024-04-14T15:19:17.589999+0000","last_deep_scrub":"464'16","last_deep_scrub_stamp":"2024-04-14T15:19:17.589999+0000","last_clean_scrub_stamp":"2024-04-14T15:19:17.589999+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:35.929485+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":911,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.937010+0000","last_change":"2024-04-14T15:17:34.986735+0000","last_active":"2024-04-14T15:20:12.937010+0000","last_peered":"2024-04-14T15:20:12.937010+0000","last_clean":"2024-04-14T15:20:12.937010+0000","last_became_active":"2024-04-14T15:09:43.023583+0000","last_became_peered":"2024-04-14T15:09:43.023583+0000","last_unstale":"2024-04-14T15:20:12.937010+0000","last_undegraded":"2024-04-14T15:20:12.937010+0000","last_fullsized":"2024-04-14T15:20:12.937010+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:34.986655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:34.986655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:15.771376+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":903,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.934872+0000","last_change":"2024-04-14T15:20:01.971662+0000","last_active":"2024-04-14T15:20:12.934872+0000","last_peered":"2024-04-14T15:20:12.934872+0000","last_clean":"2024-04-14T15:20:12.934872+0000","last_became_active":"2024-04-14T15:09:47.041683+0000","last_became_peered":"2024-04-14T15:09:47.041683+0000","last_unstale":"2024-04-14T15:20:12.934872+0000","last_undegraded":"2024-04-14T15:20:12.934872+0000","last_fullsized":"2024-04-14T15:20:12.934872+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:01.971564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:08.486217+0000","last_clean_scrub_stamp":"2024-04-14T15:20:01.971564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:18.567528+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":580,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.935910+0000","last_change":"2024-04-14T15:19:57.757627+0000","last_active":"2024-04-14T15:20:12.935910+0000","last_peered":"2024-04-14T15:20:12.935910+0000","last_clean":"2024-04-14T15:20:12.935910+0000","last_became_active":"2024-04-14T15:15:21.889567+0000","last_became_peered":"2024-04-14T15:15:21.889567+0000","last_unstale":"2024-04-14T15:20:12.935910+0000","last_undegraded":"2024-04-14T15:20:12.935910+0000","last_fullsized":"2024-04-14T15:20:12.935910+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:57.757547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:19:57.757547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:15.034106+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.15","version":"91'130","reported_seq":430,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.660443+0000","last_change":"2024-04-14T15:19:04.485222+0000","last_active":"2024-04-14T15:20:13.660443+0000","last_peered":"2024-04-14T15:20:13.660443+0000","last_clean":"2024-04-14T15:20:13.660443+0000","last_became_active":"2024-04-14T15:19:02.934529+0000","last_became_peered":"2024-04-14T15:19:02.934529+0000","last_unstale":"2024-04-14T15:20:13.660443+0000","last_undegraded":"2024-04-14T15:20:13.660443+0000","last_fullsized":"2024-04-14T15:20:13.660443+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":446,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:04.485114+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:04.485114+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:29.092721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":813,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.386362+0000","last_change":"2024-04-14T15:19:53.922021+0000","last_active":"2024-04-14T15:20:13.386362+0000","last_peered":"2024-04-14T15:20:13.386362+0000","last_clean":"2024-04-14T15:20:13.386362+0000","last_became_active":"2024-04-14T15:12:56.387723+0000","last_became_peered":"2024-04-14T15:12:56.387723+0000","last_unstale":"2024-04-14T15:20:13.386362+0000","last_undegraded":"2024-04-14T15:20:13.386362+0000","last_fullsized":"2024-04-14T15:20:13.386362+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:53.921934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:03.685045+0000","last_clean_scrub_stamp":"2024-04-14T15:19:53.921934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:08.254525+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.10","version":"479'8","reported_seq":915,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.935536+0000","last_change":"2024-04-14T15:17:53.181132+0000","last_active":"2024-04-14T15:20:12.935536+0000","last_peered":"2024-04-14T15:20:12.935536+0000","last_clean":"2024-04-14T15:20:12.935536+0000","last_became_active":"2024-04-14T15:09:45.038815+0000","last_became_peered":"2024-04-14T15:09:45.038815+0000","last_unstale":"2024-04-14T15:20:12.935536+0000","last_undegraded":"2024-04-14T15:20:12.935536+0000","last_fullsized":"2024-04-14T15:20:12.935536+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4","last_scrub_stamp":"2024-04-14T15:17:53.181053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:17:53.181053+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:13.731182+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":460,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.748311+0000","last_change":"2024-04-14T15:17:48.845163+0000","last_active":"2024-04-14T15:20:13.748311+0000","last_peered":"2024-04-14T15:20:13.748311+0000","last_clean":"2024-04-14T15:20:13.748311+0000","last_became_active":"2024-04-14T15:15:44.884730+0000","last_became_peered":"2024-04-14T15:15:44.884730+0000","last_unstale":"2024-04-14T15:20:13.748311+0000","last_undegraded":"2024-04-14T15:20:13.748311+0000","last_fullsized":"2024-04-14T15:20:13.748311+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:48.845081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:08.712289+0000","last_clean_scrub_stamp":"2024-04-14T15:17:48.845081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":918,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.747531+0000","last_change":"2024-04-14T15:19:48.867805+0000","last_active":"2024-04-14T15:20:13.747531+0000","last_peered":"2024-04-14T15:20:13.747531+0000","last_clean":"2024-04-14T15:20:13.747531+0000","last_became_active":"2024-04-14T15:09:47.041074+0000","last_became_peered":"2024-04-14T15:09:47.041074+0000","last_unstale":"2024-04-14T15:20:13.747531+0000","last_undegraded":"2024-04-14T15:20:13.747531+0000","last_fullsized":"2024-04-14T15:20:13.747531+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:48.867704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:19:48.867704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:59.267600+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":897,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.659088+0000","last_change":"2024-04-14T15:19:59.634301+0000","last_active":"2024-04-14T15:20:13.659088+0000","last_peered":"2024-04-14T15:20:13.659088+0000","last_clean":"2024-04-14T15:20:13.659088+0000","last_became_active":"2024-04-14T15:09:49.061359+0000","last_became_peered":"2024-04-14T15:09:49.061359+0000","last_unstale":"2024-04-14T15:20:13.659088+0000","last_undegraded":"2024-04-14T15:20:13.659088+0000","last_fullsized":"2024-04-14T15:20:13.659088+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:59.634212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:19:59.634212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:22.995269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"91'130","reported_seq":980,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.385638+0000","last_change":"2024-04-14T15:19:06.939565+0000","last_active":"2024-04-14T15:20:13.385638+0000","last_peered":"2024-04-14T15:20:13.385638+0000","last_clean":"2024-04-14T15:20:13.385638+0000","last_became_active":"2024-04-14T15:18:45.913983+0000","last_became_peered":"2024-04-14T15:18:45.913983+0000","last_unstale":"2024-04-14T15:20:13.385638+0000","last_undegraded":"2024-04-14T15:20:13.385638+0000","last_fullsized":"2024-04-14T15:20:13.385638+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:06.939444+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:06.939444+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:20.862256+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":806,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.661246+0000","last_change":"2024-04-14T15:18:35.596732+0000","last_active":"2024-04-14T15:20:13.661246+0000","last_peered":"2024-04-14T15:20:13.661246+0000","last_clean":"2024-04-14T15:20:13.661246+0000","last_became_active":"2024-04-14T15:12:56.352528+0000","last_became_peered":"2024-04-14T15:12:56.352528+0000","last_unstale":"2024-04-14T15:20:13.661246+0000","last_undegraded":"2024-04-14T15:20:13.661246+0000","last_fullsized":"2024-04-14T15:20:13.661246+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:35.596656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:15:43.680852+0000","last_clean_scrub_stamp":"2024-04-14T15:18:35.596656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:18.657872+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"479'24","reported_seq":577,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.936239+0000","last_change":"2024-04-14T15:17:43.061551+0000","last_active":"2024-04-14T15:20:12.936239+0000","last_peered":"2024-04-14T15:20:12.936239+0000","last_clean":"2024-04-14T15:20:12.936239+0000","last_became_active":"2024-04-14T15:15:21.884783+0000","last_became_peered":"2024-04-14T15:15:21.884783+0000","last_unstale":"2024-04-14T15:20:12.936239+0000","last_undegraded":"2024-04-14T15:20:12.936239+0000","last_fullsized":"2024-04-14T15:20:12.936239+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'16","last_scrub_stamp":"2024-04-14T15:17:43.061458+0000","last_deep_scrub":"102'16","last_deep_scrub_stamp":"2024-04-14T15:12:35.999326+0000","last_clean_scrub_stamp":"2024-04-14T15:17:43.061458+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:13.731164+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":918,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.748487+0000","last_change":"2024-04-14T15:19:11.750498+0000","last_active":"2024-04-14T15:20:13.748487+0000","last_peered":"2024-04-14T15:20:13.748487+0000","last_clean":"2024-04-14T15:20:13.748487+0000","last_became_active":"2024-04-14T15:09:43.023220+0000","last_became_peered":"2024-04-14T15:09:43.023220+0000","last_unstale":"2024-04-14T15:20:13.748487+0000","last_undegraded":"2024-04-14T15:20:13.748487+0000","last_fullsized":"2024-04-14T15:20:13.748487+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:11.750419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:19:11.750419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:20.900268+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":901,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.659513+0000","last_change":"2024-04-14T15:19:54.634365+0000","last_active":"2024-04-14T15:20:13.659513+0000","last_peered":"2024-04-14T15:20:13.659513+0000","last_clean":"2024-04-14T15:20:13.659513+0000","last_became_active":"2024-04-14T15:09:47.044665+0000","last_became_peered":"2024-04-14T15:09:47.044665+0000","last_unstale":"2024-04-14T15:20:13.659513+0000","last_undegraded":"2024-04-14T15:20:13.659513+0000","last_fullsized":"2024-04-14T15:20:13.659513+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:54.634279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:19:54.634279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:23.855974+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":580,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.937697+0000","last_change":"2024-04-14T15:19:23.462816+0000","last_active":"2024-04-14T15:20:12.937697+0000","last_peered":"2024-04-14T15:20:12.937697+0000","last_clean":"2024-04-14T15:20:12.937697+0000","last_became_active":"2024-04-14T15:15:21.883940+0000","last_became_peered":"2024-04-14T15:15:21.883940+0000","last_unstale":"2024-04-14T15:20:12.937697+0000","last_undegraded":"2024-04-14T15:20:12.937697+0000","last_fullsized":"2024-04-14T15:20:12.937697+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:23.462727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:19:23.462727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:46.221952+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1b","version":"91'130","reported_seq":954,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.934633+0000","last_change":"2024-04-14T15:19:06.558576+0000","last_active":"2024-04-14T15:20:12.934633+0000","last_peered":"2024-04-14T15:20:12.934633+0000","last_clean":"2024-04-14T15:20:12.934633+0000","last_became_active":"2024-04-14T15:19:02.936749+0000","last_became_peered":"2024-04-14T15:19:02.936749+0000","last_unstale":"2024-04-14T15:20:12.934633+0000","last_undegraded":"2024-04-14T15:20:12.934633+0000","last_fullsized":"2024-04-14T15:20:12.934633+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":446,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:06.558465+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:06.558465+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:28.851475+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":805,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.385472+0000","last_change":"2024-04-14T15:18:25.997409+0000","last_active":"2024-04-14T15:20:13.385472+0000","last_peered":"2024-04-14T15:20:13.385472+0000","last_clean":"2024-04-14T15:20:13.385472+0000","last_became_active":"2024-04-14T15:12:56.335230+0000","last_became_peered":"2024-04-14T15:12:56.335230+0000","last_unstale":"2024-04-14T15:20:13.385472+0000","last_undegraded":"2024-04-14T15:20:13.385472+0000","last_fullsized":"2024-04-14T15:20:13.385472+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:25.997333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:18:25.997333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:15.826678+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"479'20","reported_seq":937,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.338472+0000","last_change":"2024-04-14T15:20:06.373257+0000","last_active":"2024-04-14T15:20:13.338472+0000","last_peered":"2024-04-14T15:20:13.338472+0000","last_clean":"2024-04-14T15:20:13.338472+0000","last_became_active":"2024-04-14T15:09:45.181456+0000","last_became_peered":"2024-04-14T15:09:45.181456+0000","last_unstale":"2024-04-14T15:20:13.338472+0000","last_undegraded":"2024-04-14T15:20:13.338472+0000","last_fullsized":"2024-04-14T15:20:13.338472+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'20","last_scrub_stamp":"2024-04-14T15:20:06.373158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:20:06.373158+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:19.959672+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":467,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.933634+0000","last_change":"2024-04-14T15:19:57.575006+0000","last_active":"2024-04-14T15:20:12.933634+0000","last_peered":"2024-04-14T15:20:12.933634+0000","last_clean":"2024-04-14T15:20:12.933634+0000","last_became_active":"2024-04-14T15:15:44.878336+0000","last_became_peered":"2024-04-14T15:15:44.878336+0000","last_unstale":"2024-04-14T15:20:12.933634+0000","last_undegraded":"2024-04-14T15:20:12.933634+0000","last_fullsized":"2024-04-14T15:20:12.933634+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:57.574879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:19:57.574879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:02.545636+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"72'8","reported_seq":1049,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.748650+0000","last_change":"2024-04-14T15:18:28.793061+0000","last_active":"2024-04-14T15:20:13.748650+0000","last_peered":"2024-04-14T15:20:13.748650+0000","last_clean":"2024-04-14T15:20:13.748650+0000","last_became_active":"2024-04-14T15:09:47.048216+0000","last_became_peered":"2024-04-14T15:09:47.048216+0000","last_unstale":"2024-04-14T15:20:13.748650+0000","last_undegraded":"2024-04-14T15:20:13.748650+0000","last_fullsized":"2024-04-14T15:20:13.748650+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:18:28.792975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:18:28.792975+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"72'1","reported_seq":906,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.660946+0000","last_change":"2024-04-14T15:20:04.672008+0000","last_active":"2024-04-14T15:20:13.660946+0000","last_peered":"2024-04-14T15:20:13.660946+0000","last_clean":"2024-04-14T15:20:13.660946+0000","last_became_active":"2024-04-14T15:09:49.062051+0000","last_became_peered":"2024-04-14T15:09:49.062051+0000","last_unstale":"2024-04-14T15:20:13.660946+0000","last_undegraded":"2024-04-14T15:20:13.660946+0000","last_fullsized":"2024-04-14T15:20:13.660946+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-04-14T15:20:04.671896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:20:04.671896+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:09.232031+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"91'130","reported_seq":806,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.933205+0000","last_change":"2024-04-14T15:19:03.537885+0000","last_active":"2024-04-14T15:20:12.933205+0000","last_peered":"2024-04-14T15:20:12.933205+0000","last_clean":"2024-04-14T15:20:12.933205+0000","last_became_active":"2024-04-14T15:19:02.930025+0000","last_became_peered":"2024-04-14T15:19:02.930025+0000","last_unstale":"2024-04-14T15:20:12.933205+0000","last_undegraded":"2024-04-14T15:20:12.933205+0000","last_fullsized":"2024-04-14T15:20:12.933205+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":446,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:03.537803+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:03.537803+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:14.561473+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":807,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.337476+0000","last_change":"2024-04-14T15:19:13.345390+0000","last_active":"2024-04-14T15:20:13.337476+0000","last_peered":"2024-04-14T15:20:13.337476+0000","last_clean":"2024-04-14T15:20:13.337476+0000","last_became_active":"2024-04-14T15:12:56.543793+0000","last_became_peered":"2024-04-14T15:12:56.543793+0000","last_unstale":"2024-04-14T15:20:13.337476+0000","last_undegraded":"2024-04-14T15:20:13.337476+0000","last_fullsized":"2024-04-14T15:20:13.337476+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:13.345316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:19:13.345316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:38.673629+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"479'19","reported_seq":956,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.748625+0000","last_change":"2024-04-14T15:18:31.818435+0000","last_active":"2024-04-14T15:20:13.748625+0000","last_peered":"2024-04-14T15:20:13.748625+0000","last_clean":"2024-04-14T15:20:13.748625+0000","last_became_active":"2024-04-14T15:10:07.764183+0000","last_became_peered":"2024-04-14T15:10:07.764183+0000","last_unstale":"2024-04-14T15:20:13.748625+0000","last_undegraded":"2024-04-14T15:20:13.748625+0000","last_fullsized":"2024-04-14T15:20:13.748625+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-04-14T15:18:31.818311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:18:31.818311+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"65'2","reported_seq":591,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.936053+0000","last_change":"2024-04-14T15:16:56.943047+0000","last_active":"2024-04-14T15:20:12.936053+0000","last_peered":"2024-04-14T15:20:12.936053+0000","last_clean":"2024-04-14T15:20:12.936053+0000","last_became_active":"2024-04-14T15:15:21.885343+0000","last_became_peered":"2024-04-14T15:15:21.885343+0000","last_unstale":"2024-04-14T15:20:12.936053+0000","last_undegraded":"2024-04-14T15:20:12.936053+0000","last_fullsized":"2024-04-14T15:20:12.936053+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-04-14T15:16:56.942840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:56.942840+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":903,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.338541+0000","last_change":"2024-04-14T15:19:21.349524+0000","last_active":"2024-04-14T15:20:13.338541+0000","last_peered":"2024-04-14T15:20:13.338541+0000","last_clean":"2024-04-14T15:20:13.338541+0000","last_became_active":"2024-04-14T15:09:47.045730+0000","last_became_peered":"2024-04-14T15:09:47.045730+0000","last_unstale":"2024-04-14T15:20:13.338541+0000","last_undegraded":"2024-04-14T15:20:13.338541+0000","last_fullsized":"2024-04-14T15:20:13.338541+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:21.349441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:19:21.349441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:47.517518+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":891,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.936620+0000","last_change":"2024-04-14T15:16:57.235339+0000","last_active":"2024-04-14T15:20:12.936620+0000","last_peered":"2024-04-14T15:20:12.936620+0000","last_clean":"2024-04-14T15:20:12.936620+0000","last_became_active":"2024-04-14T15:09:49.057545+0000","last_became_peered":"2024-04-14T15:09:49.057545+0000","last_unstale":"2024-04-14T15:20:12.936620+0000","last_undegraded":"2024-04-14T15:20:12.936620+0000","last_fullsized":"2024-04-14T15:20:12.936620+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:57.235261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:21.247695+0000","last_clean_scrub_stamp":"2024-04-14T15:16:57.235261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:15.771339+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.19","version":"91'130","reported_seq":650,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.935664+0000","last_change":"2024-04-14T15:19:06.383564+0000","last_active":"2024-04-14T15:20:12.935664+0000","last_peered":"2024-04-14T15:20:12.935664+0000","last_clean":"2024-04-14T15:20:12.935664+0000","last_became_active":"2024-04-14T15:19:02.941363+0000","last_became_peered":"2024-04-14T15:19:02.941363+0000","last_unstale":"2024-04-14T15:20:12.935664+0000","last_undegraded":"2024-04-14T15:20:12.935664+0000","last_fullsized":"2024-04-14T15:20:12.935664+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":446,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:06.383478+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:06.383478+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:24.667992+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":459,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.934993+0000","last_change":"2024-04-14T15:17:44.364114+0000","last_active":"2024-04-14T15:20:12.934993+0000","last_peered":"2024-04-14T15:20:12.934993+0000","last_clean":"2024-04-14T15:20:12.934993+0000","last_became_active":"2024-04-14T15:15:44.882604+0000","last_became_peered":"2024-04-14T15:15:44.882604+0000","last_unstale":"2024-04-14T15:20:12.934993+0000","last_undegraded":"2024-04-14T15:20:12.934993+0000","last_fullsized":"2024-04-14T15:20:12.934993+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:44.364013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:17:44.364013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:14.561296+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"479'27","reported_seq":947,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.386500+0000","last_change":"2024-04-14T15:18:32.950805+0000","last_active":"2024-04-14T15:20:13.386500+0000","last_peered":"2024-04-14T15:20:13.386500+0000","last_clean":"2024-04-14T15:20:13.386500+0000","last_became_active":"2024-04-14T15:09:45.032887+0000","last_became_peered":"2024-04-14T15:09:45.032887+0000","last_unstale":"2024-04-14T15:20:13.386500+0000","last_undegraded":"2024-04-14T15:20:13.386500+0000","last_fullsized":"2024-04-14T15:20:13.386500+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-04-14T15:18:32.950706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:18:32.950706+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:15.826724+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":913,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.747707+0000","last_change":"2024-04-14T15:18:27.799078+0000","last_active":"2024-04-14T15:20:13.747707+0000","last_peered":"2024-04-14T15:20:13.747707+0000","last_clean":"2024-04-14T15:20:13.747707+0000","last_became_active":"2024-04-14T15:09:43.021494+0000","last_became_peered":"2024-04-14T15:09:43.021494+0000","last_unstale":"2024-04-14T15:20:13.747707+0000","last_undegraded":"2024-04-14T15:20:13.747707+0000","last_fullsized":"2024-04-14T15:20:13.747707+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:27.798964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:18:27.798964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":903,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.936971+0000","last_change":"2024-04-14T15:18:22.953738+0000","last_active":"2024-04-14T15:20:12.936971+0000","last_peered":"2024-04-14T15:20:12.936971+0000","last_clean":"2024-04-14T15:20:12.936971+0000","last_became_active":"2024-04-14T15:09:47.042733+0000","last_became_peered":"2024-04-14T15:09:47.042733+0000","last_unstale":"2024-04-14T15:20:12.936971+0000","last_undegraded":"2024-04-14T15:20:12.936971+0000","last_fullsized":"2024-04-14T15:20:12.936971+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:22.953661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:18:22.953661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:15.771580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":898,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.934867+0000","last_change":"2024-04-14T15:17:54.453289+0000","last_active":"2024-04-14T15:20:12.934867+0000","last_peered":"2024-04-14T15:20:12.934867+0000","last_clean":"2024-04-14T15:20:12.934867+0000","last_became_active":"2024-04-14T15:09:49.127979+0000","last_became_peered":"2024-04-14T15:09:49.127979+0000","last_unstale":"2024-04-14T15:20:12.934867+0000","last_undegraded":"2024-04-14T15:20:12.934867+0000","last_fullsized":"2024-04-14T15:20:12.934867+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:54.453174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:54.453174+0000","last_clean_scrub_stamp":"2024-04-14T15:17:54.453174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:14.561332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.18","version":"91'130","reported_seq":808,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.385171+0000","last_change":"2024-04-14T15:19:02.957443+0000","last_active":"2024-04-14T15:20:13.385171+0000","last_peered":"2024-04-14T15:20:13.385171+0000","last_clean":"2024-04-14T15:20:13.385171+0000","last_became_active":"2024-04-14T15:19:02.940290+0000","last_became_peered":"2024-04-14T15:19:02.940290+0000","last_unstale":"2024-04-14T15:20:13.385171+0000","last_undegraded":"2024-04-14T15:20:13.385171+0000","last_fullsized":"2024-04-14T15:20:13.385171+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":446,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:02.957360+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:02.957360+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:15.826770+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":806,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.659810+0000","last_change":"2024-04-14T15:18:26.532035+0000","last_active":"2024-04-14T15:20:13.659810+0000","last_peered":"2024-04-14T15:20:13.659810+0000","last_clean":"2024-04-14T15:20:13.659810+0000","last_became_active":"2024-04-14T15:12:56.392352+0000","last_became_peered":"2024-04-14T15:12:56.392352+0000","last_unstale":"2024-04-14T15:20:13.659810+0000","last_undegraded":"2024-04-14T15:20:13.659810+0000","last_fullsized":"2024-04-14T15:20:13.659810+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:26.531955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:18:26.531955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:18.657951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"479'18","reported_seq":619,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.937135+0000","last_change":"2024-04-14T15:19:17.364178+0000","last_active":"2024-04-14T15:20:12.937135+0000","last_peered":"2024-04-14T15:20:12.937135+0000","last_clean":"2024-04-14T15:20:12.937135+0000","last_became_active":"2024-04-14T15:15:21.877838+0000","last_became_peered":"2024-04-14T15:15:21.877838+0000","last_unstale":"2024-04-14T15:20:12.937135+0000","last_undegraded":"2024-04-14T15:20:12.937135+0000","last_fullsized":"2024-04-14T15:20:12.937135+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'12","last_scrub_stamp":"2024-04-14T15:19:17.364059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:19:17.364059+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:22.503752+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":591,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.937556+0000","last_change":"2024-04-14T15:19:25.470235+0000","last_active":"2024-04-14T15:20:12.937556+0000","last_peered":"2024-04-14T15:20:12.937556+0000","last_clean":"2024-04-14T15:20:12.937556+0000","last_became_active":"2024-04-14T15:15:21.883544+0000","last_became_peered":"2024-04-14T15:15:21.883544+0000","last_unstale":"2024-04-14T15:20:12.937556+0000","last_undegraded":"2024-04-14T15:20:12.937556+0000","last_fullsized":"2024-04-14T15:20:12.937556+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:25.470160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:19:25.470160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:47.982067+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":901,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.934044+0000","last_change":"2024-04-14T15:17:01.098681+0000","last_active":"2024-04-14T15:20:12.934044+0000","last_peered":"2024-04-14T15:20:12.934044+0000","last_clean":"2024-04-14T15:20:12.934044+0000","last_became_active":"2024-04-14T15:09:47.048045+0000","last_became_peered":"2024-04-14T15:09:47.048045+0000","last_unstale":"2024-04-14T15:20:12.934044+0000","last_undegraded":"2024-04-14T15:20:12.934044+0000","last_fullsized":"2024-04-14T15:20:12.934044+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:01.098599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:17:01.098599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:14.561163+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":581,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.937629+0000","last_change":"2024-04-14T15:19:19.371279+0000","last_active":"2024-04-14T15:20:12.937629+0000","last_peered":"2024-04-14T15:20:12.937629+0000","last_clean":"2024-04-14T15:20:12.937629+0000","last_became_active":"2024-04-14T15:15:21.889691+0000","last_became_peered":"2024-04-14T15:15:21.889691+0000","last_unstale":"2024-04-14T15:20:12.937629+0000","last_undegraded":"2024-04-14T15:20:12.937629+0000","last_fullsized":"2024-04-14T15:20:12.937629+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:19.371182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:19:19.371182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:48.666091+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":813,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.386246+0000","last_change":"2024-04-14T15:20:01.841015+0000","last_active":"2024-04-14T15:20:13.386246+0000","last_peered":"2024-04-14T15:20:13.386246+0000","last_clean":"2024-04-14T15:20:13.386246+0000","last_became_active":"2024-04-14T15:12:56.375278+0000","last_became_peered":"2024-04-14T15:12:56.375278+0000","last_unstale":"2024-04-14T15:20:13.386246+0000","last_undegraded":"2024-04-14T15:20:13.386246+0000","last_fullsized":"2024-04-14T15:20:13.386246+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:01.840924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:01.840924+0000","last_clean_scrub_stamp":"2024-04-14T15:20:01.840924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:05.629144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"479'19","reported_seq":934,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.933643+0000","last_change":"2024-04-14T15:17:52.445394+0000","last_active":"2024-04-14T15:20:12.933643+0000","last_peered":"2024-04-14T15:20:12.933643+0000","last_clean":"2024-04-14T15:20:12.933643+0000","last_became_active":"2024-04-14T15:09:45.030093+0000","last_became_peered":"2024-04-14T15:09:45.030093+0000","last_unstale":"2024-04-14T15:20:12.933643+0000","last_undegraded":"2024-04-14T15:20:12.933643+0000","last_fullsized":"2024-04-14T15:20:12.933643+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'13","last_scrub_stamp":"2024-04-14T15:17:52.445294+0000","last_deep_scrub":"276'13","last_deep_scrub_stamp":"2024-04-14T15:15:43.969775+0000","last_clean_scrub_stamp":"2024-04-14T15:17:52.445294+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:14.561278+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":615,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.660385+0000","last_change":"2024-04-14T15:20:00.650164+0000","last_active":"2024-04-14T15:20:13.660385+0000","last_peered":"2024-04-14T15:20:13.660385+0000","last_clean":"2024-04-14T15:20:13.660385+0000","last_became_active":"2024-04-14T15:14:30.857727+0000","last_became_peered":"2024-04-14T15:14:30.857727+0000","last_unstale":"2024-04-14T15:20:13.660385+0000","last_undegraded":"2024-04-14T15:20:13.660385+0000","last_fullsized":"2024-04-14T15:20:13.660385+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:00.650062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:00.650062+0000","last_clean_scrub_stamp":"2024-04-14T15:20:00.650062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:25.732113+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":460,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.338258+0000","last_change":"2024-04-14T15:16:59.695423+0000","last_active":"2024-04-14T15:20:13.338258+0000","last_peered":"2024-04-14T15:20:13.338258+0000","last_clean":"2024-04-14T15:20:13.338258+0000","last_became_active":"2024-04-14T15:15:44.883121+0000","last_became_peered":"2024-04-14T15:15:44.883121+0000","last_unstale":"2024-04-14T15:20:13.338258+0000","last_undegraded":"2024-04-14T15:20:13.338258+0000","last_fullsized":"2024-04-14T15:20:13.338258+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:59.695337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:59.695337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:17.378142+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":898,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.337165+0000","last_change":"2024-04-14T15:17:33.952503+0000","last_active":"2024-04-14T15:20:13.337165+0000","last_peered":"2024-04-14T15:20:13.337165+0000","last_clean":"2024-04-14T15:20:13.337165+0000","last_became_active":"2024-04-14T15:09:49.061756+0000","last_became_peered":"2024-04-14T15:09:49.061756+0000","last_unstale":"2024-04-14T15:20:13.337165+0000","last_undegraded":"2024-04-14T15:20:13.337165+0000","last_fullsized":"2024-04-14T15:20:13.337165+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:33.952312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:17:33.952312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:17.378158+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"91'130","reported_seq":438,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.937290+0000","last_change":"2024-04-14T15:19:03.395696+0000","last_active":"2024-04-14T15:20:12.937290+0000","last_peered":"2024-04-14T15:20:12.937290+0000","last_clean":"2024-04-14T15:20:12.937290+0000","last_became_active":"2024-04-14T15:19:02.936733+0000","last_became_peered":"2024-04-14T15:19:02.936733+0000","last_unstale":"2024-04-14T15:20:12.937290+0000","last_undegraded":"2024-04-14T15:20:12.937290+0000","last_fullsized":"2024-04-14T15:20:12.937290+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":446,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:03.395612+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:03.395612+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":815,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.748288+0000","last_change":"2024-04-14T15:19:12.723151+0000","last_active":"2024-04-14T15:20:13.748288+0000","last_peered":"2024-04-14T15:20:13.748288+0000","last_clean":"2024-04-14T15:20:13.748288+0000","last_became_active":"2024-04-14T15:12:56.232831+0000","last_became_peered":"2024-04-14T15:12:56.232831+0000","last_unstale":"2024-04-14T15:20:13.748288+0000","last_undegraded":"2024-04-14T15:20:13.748288+0000","last_fullsized":"2024-04-14T15:20:13.748288+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:12.723073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:19:12.723073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:34.125961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1b","version":"479'9","reported_seq":920,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.935363+0000","last_change":"2024-04-14T15:18:35.481608+0000","last_active":"2024-04-14T15:20:12.935363+0000","last_peered":"2024-04-14T15:20:12.935363+0000","last_clean":"2024-04-14T15:20:12.935363+0000","last_became_active":"2024-04-14T15:09:45.028667+0000","last_became_peered":"2024-04-14T15:09:45.028667+0000","last_unstale":"2024-04-14T15:20:12.935363+0000","last_undegraded":"2024-04-14T15:20:12.935363+0000","last_fullsized":"2024-04-14T15:20:12.935363+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'7","last_scrub_stamp":"2024-04-14T15:18:35.481504+0000","last_deep_scrub":"87'7","last_deep_scrub_stamp":"2024-04-14T15:11:10.510785+0000","last_clean_scrub_stamp":"2024-04-14T15:18:35.481504+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:14.561413+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":385,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.385867+0000","last_change":"2024-04-14T15:19:26.879798+0000","last_active":"2024-04-14T15:20:13.385867+0000","last_peered":"2024-04-14T15:20:13.385867+0000","last_clean":"2024-04-14T15:20:13.385867+0000","last_became_active":"2024-04-14T15:16:24.910246+0000","last_became_peered":"2024-04-14T15:16:24.910246+0000","last_unstale":"2024-04-14T15:20:13.385867+0000","last_undegraded":"2024-04-14T15:20:13.385867+0000","last_fullsized":"2024-04-14T15:20:13.385867+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:26.879685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:08.254601+0000","last_clean_scrub_stamp":"2024-04-14T15:19:26.879685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:37.880480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":901,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.660026+0000","last_change":"2024-04-14T15:19:15.567537+0000","last_active":"2024-04-14T15:20:13.660026+0000","last_peered":"2024-04-14T15:20:13.660026+0000","last_clean":"2024-04-14T15:20:13.660026+0000","last_became_active":"2024-04-14T15:09:47.046517+0000","last_became_peered":"2024-04-14T15:09:47.046517+0000","last_unstale":"2024-04-14T15:20:13.660026+0000","last_undegraded":"2024-04-14T15:20:13.660026+0000","last_fullsized":"2024-04-14T15:20:13.660026+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:15.567432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:19:15.567432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:24.223440+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":462,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.934713+0000","last_change":"2024-04-14T15:18:21.023508+0000","last_active":"2024-04-14T15:20:12.934713+0000","last_peered":"2024-04-14T15:20:12.934713+0000","last_clean":"2024-04-14T15:20:12.934713+0000","last_became_active":"2024-04-14T15:15:44.887227+0000","last_became_peered":"2024-04-14T15:15:44.887227+0000","last_unstale":"2024-04-14T15:20:12.934713+0000","last_undegraded":"2024-04-14T15:20:12.934713+0000","last_fullsized":"2024-04-14T15:20:12.934713+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:21.023426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:56.268086+0000","last_clean_scrub_stamp":"2024-04-14T15:18:21.023426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:15.771549+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1d","version":"91'130","reported_seq":438,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.936243+0000","last_change":"2024-04-14T15:19:03.081107+0000","last_active":"2024-04-14T15:20:12.936243+0000","last_peered":"2024-04-14T15:20:12.936243+0000","last_clean":"2024-04-14T15:20:12.936243+0000","last_became_active":"2024-04-14T15:19:02.934382+0000","last_became_peered":"2024-04-14T15:19:02.934382+0000","last_unstale":"2024-04-14T15:20:12.936243+0000","last_undegraded":"2024-04-14T15:20:12.936243+0000","last_fullsized":"2024-04-14T15:20:12.936243+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":446,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:03.081016+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:03.081016+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:15.771652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":806,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.661349+0000","last_change":"2024-04-14T15:18:34.558447+0000","last_active":"2024-04-14T15:20:13.661349+0000","last_peered":"2024-04-14T15:20:13.661349+0000","last_clean":"2024-04-14T15:20:13.661349+0000","last_became_active":"2024-04-14T15:12:56.189117+0000","last_became_peered":"2024-04-14T15:12:56.189117+0000","last_unstale":"2024-04-14T15:20:13.661349+0000","last_undegraded":"2024-04-14T15:20:13.661349+0000","last_fullsized":"2024-04-14T15:20:13.661349+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:34.558312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:34.558312+0000","last_clean_scrub_stamp":"2024-04-14T15:18:34.558312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:18.657991+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.18","version":"479'15","reported_seq":930,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.933520+0000","last_change":"2024-04-14T15:18:36.441934+0000","last_active":"2024-04-14T15:20:12.933520+0000","last_peered":"2024-04-14T15:20:12.933520+0000","last_clean":"2024-04-14T15:20:12.933520+0000","last_became_active":"2024-04-14T15:09:45.032079+0000","last_became_peered":"2024-04-14T15:09:45.032079+0000","last_unstale":"2024-04-14T15:20:12.933520+0000","last_undegraded":"2024-04-14T15:20:12.933520+0000","last_fullsized":"2024-04-14T15:20:12.933520+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-04-14T15:18:36.441813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:18:36.441813+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:14.561453+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":585,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.937708+0000","last_change":"2024-04-14T15:17:00.948362+0000","last_active":"2024-04-14T15:20:12.937708+0000","last_peered":"2024-04-14T15:20:12.937708+0000","last_clean":"2024-04-14T15:20:12.937708+0000","last_became_active":"2024-04-14T15:15:21.891926+0000","last_became_peered":"2024-04-14T15:15:21.891926+0000","last_unstale":"2024-04-14T15:20:12.937708+0000","last_undegraded":"2024-04-14T15:20:12.937708+0000","last_fullsized":"2024-04-14T15:20:12.937708+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:00.948283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:00.948283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:13.731011+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":903,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.934407+0000","last_change":"2024-04-14T15:19:56.903138+0000","last_active":"2024-04-14T15:20:12.934407+0000","last_peered":"2024-04-14T15:20:12.934407+0000","last_clean":"2024-04-14T15:20:12.934407+0000","last_became_active":"2024-04-14T15:09:47.041486+0000","last_became_peered":"2024-04-14T15:09:47.041486+0000","last_unstale":"2024-04-14T15:20:12.934407+0000","last_undegraded":"2024-04-14T15:20:12.934407+0000","last_fullsized":"2024-04-14T15:20:12.934407+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:56.903059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:19:56.903059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:25.052114+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":889,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.933498+0000","last_change":"2024-04-14T15:16:55.184211+0000","last_active":"2024-04-14T15:20:12.933498+0000","last_peered":"2024-04-14T15:20:12.933498+0000","last_clean":"2024-04-14T15:20:12.933498+0000","last_became_active":"2024-04-14T15:09:49.058904+0000","last_became_peered":"2024-04-14T15:09:49.058904+0000","last_unstale":"2024-04-14T15:20:12.933498+0000","last_undegraded":"2024-04-14T15:20:12.933498+0000","last_fullsized":"2024-04-14T15:20:12.933498+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:55.184094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:55.184094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:14.561120+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1c","version":"91'130","reported_seq":951,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.660262+0000","last_change":"2024-04-14T15:19:03.486816+0000","last_active":"2024-04-14T15:20:13.660262+0000","last_peered":"2024-04-14T15:20:13.660262+0000","last_clean":"2024-04-14T15:20:13.660262+0000","last_became_active":"2024-04-14T15:19:02.934458+0000","last_became_peered":"2024-04-14T15:19:02.934458+0000","last_unstale":"2024-04-14T15:20:13.660262+0000","last_undegraded":"2024-04-14T15:20:13.660262+0000","last_fullsized":"2024-04-14T15:20:13.660262+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":446,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:03.486726+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:03.486726+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:18.658030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":459,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.661176+0000","last_change":"2024-04-14T15:17:52.593706+0000","last_active":"2024-04-14T15:20:13.661176+0000","last_peered":"2024-04-14T15:20:13.661176+0000","last_clean":"2024-04-14T15:20:13.661176+0000","last_became_active":"2024-04-14T15:15:44.889605+0000","last_became_peered":"2024-04-14T15:15:44.889605+0000","last_unstale":"2024-04-14T15:20:13.661176+0000","last_undegraded":"2024-04-14T15:20:13.661176+0000","last_fullsized":"2024-04-14T15:20:13.661176+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:52.593618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:17:52.593618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:18.657854+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"479'34","reported_seq":638,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.936315+0000","last_change":"2024-04-14T15:19:53.781367+0000","last_active":"2024-04-14T15:20:12.936315+0000","last_peered":"2024-04-14T15:20:12.936315+0000","last_clean":"2024-04-14T15:20:12.936315+0000","last_became_active":"2024-04-14T15:15:21.884310+0000","last_became_peered":"2024-04-14T15:15:21.884310+0000","last_unstale":"2024-04-14T15:20:12.936315+0000","last_undegraded":"2024-04-14T15:20:12.936315+0000","last_fullsized":"2024-04-14T15:20:12.936315+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'34","last_scrub_stamp":"2024-04-14T15:19:53.781241+0000","last_deep_scrub":"306'22","last_deep_scrub_stamp":"2024-04-14T15:16:13.898106+0000","last_clean_scrub_stamp":"2024-04-14T15:19:53.781241+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:54.804612+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":911,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:13.338271+0000","last_change":"2024-04-14T15:19:18.347632+0000","last_active":"2024-04-14T15:20:13.338271+0000","last_peered":"2024-04-14T15:20:13.338271+0000","last_clean":"2024-04-14T15:20:13.338271+0000","last_became_active":"2024-04-14T15:09:43.017351+0000","last_became_peered":"2024-04-14T15:09:43.017351+0000","last_unstale":"2024-04-14T15:20:13.338271+0000","last_undegraded":"2024-04-14T15:20:13.338271+0000","last_fullsized":"2024-04-14T15:20:13.338271+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:18.347533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:19:18.347533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:22.403586+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":901,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.934215+0000","last_change":"2024-04-14T15:19:49.631364+0000","last_active":"2024-04-14T15:20:12.934215+0000","last_peered":"2024-04-14T15:20:12.934215+0000","last_clean":"2024-04-14T15:20:12.934215+0000","last_became_active":"2024-04-14T15:09:47.042538+0000","last_became_peered":"2024-04-14T15:09:47.042538+0000","last_unstale":"2024-04-14T15:20:12.934215+0000","last_undegraded":"2024-04-14T15:20:12.934215+0000","last_fullsized":"2024-04-14T15:20:12.934215+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:49.631273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:08.512130+0000","last_clean_scrub_stamp":"2024-04-14T15:19:49.631273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:09.671784+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":895,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-14T15:20:12.937170+0000","last_change":"2024-04-14T15:17:56.237366+0000","last_active":"2024-04-14T15:20:12.937170+0000","last_peered":"2024-04-14T15:20:12.937170+0000","last_clean":"2024-04-14T15:20:12.937170+0000","last_became_active":"2024-04-14T15:09:49.129636+0000","last_became_peered":"2024-04-14T15:09:49.129636+0000","last_unstale":"2024-04-14T15:20:12.937170+0000","last_undegraded":"2024-04-14T15:20:12.937170+0000","last_fullsized":"2024-04-14T15:20:12.937170+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:56.237273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:17:56.237273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:13.731131+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":305,"num_pg":32,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":60,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":110,"num_pg":32,"stat_sum":{"num_bytes":8195937,"num_objects":176,"num_object_clones":10,"num_object_copies":528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":175,"num_whiteouts":0,"num_read":116,"num_read_kb":11,"num_write":632,"num_write_kb":12716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":41,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":20,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":26677248,"data_stored":24587787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1973,"ondisk_log_size":1973,"up":96,"acting":96,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4030,"ondisk_log_size":4030,"up":93,"acting":93,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":624,"num_read_kb":619,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1124,"num_read_kb":881,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":779,"ondisk_log_size":779,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71,"ondisk_log_size":71,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103300,"num_pgs":112,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":61492,"kb_used_data":16416,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93683660,"statfs":{"total":95995035648,"available":95932067840,"internally_reserved":0,"allocated":16809984,"data_stored":13185911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":46135754},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397639,"num_pgs":92,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":75708,"kb_used_data":8424,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93669444,"statfs":{"total":95995035648,"available":95917510656,"internally_reserved":0,"allocated":8626176,"data_stored":5056775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575009,"num_pgs":103,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":75444,"kb_used_data":12320,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93669708,"statfs":{"total":95995035648,"available":95917780992,"internally_reserved":0,"allocated":12615680,"data_stored":8932153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738534,"num_pgs":91,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":88000,"kb_used_data":12396,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93657152,"statfs":{"total":95995035648,"available":95904923648,"internally_reserved":0,"allocated":12693504,"data_stored":9057231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77396423},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411485,"num_pgs":93,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":87996,"kb_used_data":12392,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93657156,"statfs":{"total":95995035648,"available":95904927744,"internally_reserved":0,"allocated":12689408,"data_stored":8999745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182551,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":57680,"kb_used_data":8432,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93687472,"statfs":{"total":95995035648,"available":95935971328,"internally_reserved":0,"allocated":8634368,"data_stored":5065584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986321,"num_pgs":93,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":75368,"kb_used_data":8084,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93669784,"statfs":{"total":95995035648,"available":95917858816,"internally_reserved":0,"allocated":8278016,"data_stored":4737060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659278,"num_pgs":92,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":79720,"kb_used_data":12436,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93665432,"statfs":{"total":95995035648,"available":95913402368,"internally_reserved":0,"allocated":12734464,"data_stored":9185095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4292608,"data_stored":4000650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4509696,"data_stored":4195049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4366336,"data_stored":4000852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4407296,"data_stored":4194794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":229376,"data_stored":534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8482816,"data_stored":8194980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":305,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":305,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":305,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":305,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":20,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":305,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":305,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":20,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":305,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":20,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":305,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-14T15:20:16.464 INFO:tasks.cephadm.ceph_manager.ceph:decrease pool size by 10 2024-04-14T15:20:16.464 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd pool set .mgr pg_num 21 2024-04-14T15:20:16.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:16.468 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:16.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:20:16.569 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:20:16.592 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:16.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:16.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:20:16.694 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:20:16.720 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:16.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:16.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:20:16.822 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:20:16.847 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:16.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:16.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:20:16.920 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:20:16.921 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:20:16.949 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:20:16.973 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:16.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:16.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:20:17.074 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:20:17.098 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:17.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:17.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:20:17.167 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:16 smithi090 ceph-mon[41670]: pgmap v717: 258 pgs: 258 active+clean; 8.4 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 129 B/s wr, 2 op/s 2024-04-14T15:20:17.168 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:16 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:20:17.168 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:16 smithi090 ceph-mon[41670]: from='client.47414 v1:172.21.15.90:0/1440636058' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:20:17.168 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:16 smithi090 ceph-mon[41670]: osdmap e498: 8 total, 8 up, 8 in 2024-04-14T15:20:17.168 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:16 smithi090 ceph-mon[30399]: pgmap v717: 258 pgs: 258 active+clean; 8.4 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 129 B/s wr, 2 op/s 2024-04-14T15:20:17.168 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:16 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:20:17.168 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:16 smithi090 ceph-mon[30399]: from='client.47414 v1:172.21.15.90:0/1440636058' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:20:17.168 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:16 smithi090 ceph-mon[30399]: osdmap e498: 8 total, 8 up, 8 in 2024-04-14T15:20:17.200 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:20:17.222 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:17.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:17.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:20:17.324 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:20:17.349 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:17.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:17.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:20:17.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:16 smithi092 ceph-mon[33966]: pgmap v717: 258 pgs: 258 active+clean; 8.4 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 129 B/s wr, 2 op/s 2024-04-14T15:20:17.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:16 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:20:17.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:16 smithi092 ceph-mon[33966]: from='client.47414 v1:172.21.15.90:0/1440636058' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:20:17.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:16 smithi092 ceph-mon[33966]: osdmap e498: 8 total, 8 up, 8 in 2024-04-14T15:20:17.451 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:20:17.475 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:17.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:17.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:20:17.577 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:20:17.602 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:17.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:17.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:20:17.704 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:20:17.727 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:17.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:17.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:20:17.828 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:20:17.852 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:17.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:17.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:20:17.953 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:20:17.980 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:17.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:17.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:20:18.081 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:20:18.109 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:18.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:18.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:20:18.210 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:18.235 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:18.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:18.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:18.336 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:20:18.364 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:18.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:18.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:20:18.466 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:20:18.493 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:18.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:18.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:20:18.595 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:20:18.621 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:18.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:18.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:20:18.723 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:20:18.748 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:18.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:18.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:20:18.850 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:20:18.878 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:18.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:18.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:20:18.980 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:20:19.004 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:19.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:19.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:20:19.105 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:20:19.129 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:19.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:19.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:20:19.166 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:18 smithi090 ceph-mon[30399]: pgmap v719: 258 pgs: 258 active+clean; 8.4 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 743 B/s rd, 0 op/s 2024-04-14T15:20:19.166 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:18 smithi090 ceph-mon[41670]: pgmap v719: 258 pgs: 258 active+clean; 8.4 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 743 B/s rd, 0 op/s 2024-04-14T15:20:19.231 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:20:19.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:19.255 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:19.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:20:19.356 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:20:19.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:18 smithi092 ceph-mon[33966]: pgmap v719: 258 pgs: 258 active+clean; 8.4 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 743 B/s rd, 0 op/s 2024-04-14T15:20:19.383 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:19.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:19.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:20:19.484 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:20:19.512 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:19.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:19.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:20:19.614 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:20:19.640 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:19.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:19.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:20:19.742 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:20:19.765 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:19.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:19.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:20:19.867 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:20:19.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:19.890 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:19.891 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:20:19.991 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:20:20.019 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:20.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:20.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:20:20.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.946+0000 7fc347179700 1 -- 172.21.15.90:0/2107757799 >> v1:172.21.15.90:6789/0 conn(0x7fc34010a610 legacy=0x7fc34010ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:20.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.946+0000 7fc347179700 1 -- 172.21.15.90:0/2107757799 shutdown_connections 2024-04-14T15:20:20.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.946+0000 7fc347179700 1 -- 172.21.15.90:0/2107757799 >> 172.21.15.90:0/2107757799 conn(0x7fc3400fd290 msgr2=0x7fc3400ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:20:20.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.946+0000 7fc347179700 1 -- 172.21.15.90:0/2107757799 shutdown_connections 2024-04-14T15:20:20.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.946+0000 7fc347179700 1 -- 172.21.15.90:0/2107757799 wait complete. 2024-04-14T15:20:20.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.946+0000 7fc347179700 1 librados: starting msgr at 2024-04-14T15:20:20.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.946+0000 7fc347179700 1 librados: starting objecter 2024-04-14T15:20:20.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.946+0000 7fc347179700 1 Processor -- start 2024-04-14T15:20:20.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.946+0000 7fc347179700 1 -- start start 2024-04-14T15:20:20.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.946+0000 7fc347179700 1 librados: setting wanted keys 2024-04-14T15:20:20.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.946+0000 7fc347179700 1 librados: calling monclient init 2024-04-14T15:20:20.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.947+0000 7fc347179700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc3401a7df0 con 0x7fc34010a610 2024-04-14T15:20:20.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.947+0000 7fc347179700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc3401a8fd0 con 0x7fc34010e1e0 2024-04-14T15:20:20.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.947+0000 7fc347179700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc3401aa1b0 con 0x7fc340107760 2024-04-14T15:20:20.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.947+0000 7fc33ffff700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fc34010a610 0x7fc34010c5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:49374/0 (socket says 172.21.15.90:49374) 2024-04-14T15:20:20.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.947+0000 7fc33ffff700 1 -- 172.21.15.90:0/1193298934 learned_addr learned my addr 172.21.15.90:0/1193298934 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:20:20.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.947+0000 7fc344f15700 1 --1- 172.21.15.90:0/1193298934 >> v1:172.21.15.90:6790/0 conn(0x7fc340107760 0x7fc34010bea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:36574/0 (socket says 172.21.15.90:36574) 2024-04-14T15:20:20.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.947+0000 7fc33dffb700 10 client.?.objecter ms_handle_connect 0x7fc34010a610 2024-04-14T15:20:20.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.947+0000 7fc33dffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:20:20.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.947+0000 7fc33dffb700 10 client.?.objecter ms_handle_connect 0x7fc340107760 2024-04-14T15:20:20.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.947+0000 7fc33dffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:20:20.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.947+0000 7fc33dffb700 10 client.?.objecter ms_handle_connect 0x7fc34010e1e0 2024-04-14T15:20:20.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.947+0000 7fc33dffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:20:20.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.947+0000 7fc33dffb700 1 -- 172.21.15.90:0/1193298934 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3778865917 0 0) 0x7fc3401a7df0 con 0x7fc34010a610 2024-04-14T15:20:20.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.947+0000 7fc33dffb700 1 -- 172.21.15.90:0/1193298934 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3280035b0 con 0x7fc34010a610 2024-04-14T15:20:20.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.947+0000 7fc33dffb700 1 -- 172.21.15.90:0/1193298934 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1302374174 0 0) 0x7fc3401aa1b0 con 0x7fc340107760 2024-04-14T15:20:20.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.947+0000 7fc33dffb700 1 -- 172.21.15.90:0/1193298934 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3401a7df0 con 0x7fc340107760 2024-04-14T15:20:20.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.947+0000 7fc33dffb700 1 -- 172.21.15.90:0/1193298934 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3887741672 0 0) 0x7fc3401a8fd0 con 0x7fc34010e1e0 2024-04-14T15:20:20.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.947+0000 7fc33dffb700 1 -- 172.21.15.90:0/1193298934 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3401aa1b0 con 0x7fc34010e1e0 2024-04-14T15:20:20.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.947+0000 7fc33dffb700 1 -- 172.21.15.90:0/1193298934 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 329559384 0 0) 0x7fc3280035b0 con 0x7fc34010a610 2024-04-14T15:20:20.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.947+0000 7fc33dffb700 1 -- 172.21.15.90:0/1193298934 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc3401a8fd0 con 0x7fc34010a610 2024-04-14T15:20:20.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.947+0000 7fc33dffb700 1 -- 172.21.15.90:0/1193298934 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fc334001970 con 0x7fc34010a610 2024-04-14T15:20:20.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.947+0000 7fc33dffb700 1 -- 172.21.15.90:0/1193298934 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3595261008 0 0) 0x7fc3401a7df0 con 0x7fc340107760 2024-04-14T15:20:20.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.947+0000 7fc33dffb700 1 -- 172.21.15.90:0/1193298934 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc3280035b0 con 0x7fc340107760 2024-04-14T15:20:20.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.947+0000 7fc33dffb700 1 -- 172.21.15.90:0/1193298934 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 287913058 0 0) 0x7fc3401aa1b0 con 0x7fc34010e1e0 2024-04-14T15:20:20.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.947+0000 7fc33dffb700 1 -- 172.21.15.90:0/1193298934 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc3401a7df0 con 0x7fc34010e1e0 2024-04-14T15:20:20.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.947+0000 7fc33dffb700 1 -- 172.21.15.90:0/1193298934 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fc338001bb0 con 0x7fc340107760 2024-04-14T15:20:20.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.947+0000 7fc33dffb700 1 -- 172.21.15.90:0/1193298934 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fc330001830 con 0x7fc34010e1e0 2024-04-14T15:20:20.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.947+0000 7fc33dffb700 1 -- 172.21.15.90:0/1193298934 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3474414656 0 0) 0x7fc3401a8fd0 con 0x7fc34010a610 2024-04-14T15:20:20.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.947+0000 7fc33dffb700 1 -- 172.21.15.90:0/1193298934 >> v1:172.21.15.90:6790/0 conn(0x7fc340107760 legacy=0x7fc34010bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:20.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.947+0000 7fc33dffb700 1 -- 172.21.15.90:0/1193298934 >> v1:172.21.15.92:6789/0 conn(0x7fc34010e1e0 legacy=0x7fc3401a66d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:20.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.947+0000 7fc33dffb700 1 -- 172.21.15.90:0/1193298934 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3401ab390 con 0x7fc34010a610 2024-04-14T15:20:20.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.947+0000 7fc347179700 1 -- 172.21.15.90:0/1193298934 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc3401aa3e0 con 0x7fc34010a610 2024-04-14T15:20:20.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.948+0000 7fc33dffb700 1 -- 172.21.15.90:0/1193298934 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fc334004480 con 0x7fc34010a610 2024-04-14T15:20:20.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.948+0000 7fc33dffb700 1 -- 172.21.15.90:0/1193298934 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fc334004f80 con 0x7fc34010a610 2024-04-14T15:20:20.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.948+0000 7fc347179700 10 client.37467.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:20:20.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.948+0000 7fc347179700 1 -- 172.21.15.90:0/1193298934 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc3401aa8a0 con 0x7fc34010a610 2024-04-14T15:20:20.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.948+0000 7fc347179700 1 librados: init done 2024-04-14T15:20:20.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.949+0000 7fc347179700 1 -- 172.21.15.90:0/1193298934 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fc340102ea0 con 0x7fc34010a610 2024-04-14T15:20:20.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.949+0000 7fc33dffb700 1 -- 172.21.15.90:0/1193298934 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fc3340037f0 con 0x7fc34010a610 2024-04-14T15:20:20.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.949+0000 7fc33dffb700 1 -- 172.21.15.90:0/1193298934 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(498..498 src has 51..498) ==== 8197+0+0 (unknown 2199958775 0 0) 0x7fc33408aa60 con 0x7fc34010a610 2024-04-14T15:20:20.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.949+0000 7fc33dffb700 10 client.37467.objecter ms_dispatch 0x7fc340001550 osd_map(498..498 src has 51..498) 2024-04-14T15:20:20.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.949+0000 7fc33dffb700 3 client.37467.objecter handle_osd_map got epochs [498,498] > 0 2024-04-14T15:20:20.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.949+0000 7fc33dffb700 3 client.37467.objecter handle_osd_map decoding full epoch 498 2024-04-14T15:20:20.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.950+0000 7fc33dffb700 20 client.37467.objecter dump_active .. 0 homeless 2024-04-14T15:20:20.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.950+0000 7fc33dffb700 10 client.37467.objecter ms_handle_connect 0x7fc3280714a0 2024-04-14T15:20:20.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.950+0000 7fc33dffb700 1 -- 172.21.15.90:0/1193298934 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v498) ==== 54+0+16 (unknown 2274996656 0 2420000192) 0x7fc33408ae50 con 0x7fc34010a610 2024-04-14T15:20:20.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.950+0000 7fc347179700 10 client.37467.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:20:20.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.950+0000 7fc347179700 15 client.37467.objecter _session_command_op_assign -1 1 2024-04-14T15:20:20.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.950+0000 7fc347179700 20 client.37467.objecter _get_session s=0x7fc3401121b0 osd=0 3 2024-04-14T15:20:20.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.950+0000 7fc347179700 20 client.37467.objecter put_session s=0x7fc3401121b0 osd=0 3 2024-04-14T15:20:20.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.950+0000 7fc347179700 20 client.37467.objecter _get_session s=0x7fc3401121b0 osd=0 3 2024-04-14T15:20:20.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.950+0000 7fc347179700 15 client.37467.objecter _session_command_op_remove -1 1 2024-04-14T15:20:20.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.950+0000 7fc347179700 20 client.37467.objecter get_session s=0x7fc3401121b0 osd=0 3 2024-04-14T15:20:20.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.950+0000 7fc347179700 15 client.37467.objecter _session_command_op_assign 0 1 2024-04-14T15:20:20.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.950+0000 7fc347179700 20 client.37467.objecter put_session s=0x7fc3401121b0 osd=0 4 2024-04-14T15:20:20.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.950+0000 7fc347179700 10 client.37467.objecter _send_command 1 2024-04-14T15:20:20.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.950+0000 7fc347179700 1 -- 172.21.15.90:0/1193298934 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc34010d3d0 con 0x7fc3401a91a0 2024-04-14T15:20:20.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.950+0000 7fc33dffb700 10 client.37467.objecter ms_handle_connect 0x7fc3401a91a0 2024-04-14T15:20:20.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.951+0000 7fc33dffb700 1 -- 172.21.15.90:0/1193298934 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc34010d3d0 con 0x7fc3401a91a0 2024-04-14T15:20:20.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.951+0000 7fc33dffb700 10 client.37467.objecter ms_dispatch 0x7fc340001550 command_reply(tid 1: 0 ) 2024-04-14T15:20:20.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.951+0000 7fc33dffb700 10 client.37467.objecter _finish_command 1 = system:0 2024-04-14T15:20:20.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.951+0000 7fc33dffb700 20 client.37467.objecter put_session s=0x7fc3401121b0 osd=0 4 2024-04-14T15:20:20.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.951+0000 7fc33dffb700 15 client.37467.objecter _session_command_op_remove 0 1 2024-04-14T15:20:20.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.968+0000 7fc347179700 10 client.37467.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:20:20.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.968+0000 7fc347179700 15 client.37467.objecter _session_command_op_assign -1 2 2024-04-14T15:20:20.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.968+0000 7fc347179700 20 client.37467.objecter _get_session s=0x7fc3401121b0 osd=0 3 2024-04-14T15:20:20.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.968+0000 7fc347179700 20 client.37467.objecter put_session s=0x7fc3401121b0 osd=0 3 2024-04-14T15:20:20.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.968+0000 7fc347179700 20 client.37467.objecter _get_session s=0x7fc3401121b0 osd=0 3 2024-04-14T15:20:20.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.968+0000 7fc347179700 15 client.37467.objecter _session_command_op_remove -1 2 2024-04-14T15:20:20.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.968+0000 7fc347179700 20 client.37467.objecter get_session s=0x7fc3401121b0 osd=0 3 2024-04-14T15:20:20.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.968+0000 7fc347179700 15 client.37467.objecter _session_command_op_assign 0 2 2024-04-14T15:20:20.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.968+0000 7fc347179700 20 client.37467.objecter put_session s=0x7fc3401121b0 osd=0 4 2024-04-14T15:20:20.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.968+0000 7fc347179700 10 client.37467.objecter _send_command 2 2024-04-14T15:20:20.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.968+0000 7fc347179700 1 -- 172.21.15.90:0/1193298934 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fc3400693d0 con 0x7fc3401a91a0 2024-04-14T15:20:20.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.969+0000 7fc33dffb700 1 -- 172.21.15.90:0/1193298934 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc3400693d0 con 0x7fc3401a91a0 2024-04-14T15:20:20.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.969+0000 7fc33dffb700 10 client.37467.objecter ms_dispatch 0x7fc340001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:20:20.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.969+0000 7fc33dffb700 10 client.37467.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:20:20.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.969+0000 7fc33dffb700 20 client.37467.objecter put_session s=0x7fc3401121b0 osd=0 4 2024-04-14T15:20:20.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.969+0000 7fc33dffb700 15 client.37467.objecter _session_command_op_remove 0 2 2024-04-14T15:20:20.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.969+0000 7fc347179700 10 client.37467.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:20:20.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.969+0000 7fc347179700 15 client.37467.objecter _session_command_op_assign -1 3 2024-04-14T15:20:20.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.969+0000 7fc347179700 20 client.37467.objecter _get_session s=0x7fc3401a9ee0 osd=1 3 2024-04-14T15:20:20.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.969+0000 7fc347179700 20 client.37467.objecter put_session s=0x7fc3401a9ee0 osd=1 3 2024-04-14T15:20:20.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.969+0000 7fc347179700 20 client.37467.objecter _get_session s=0x7fc3401a9ee0 osd=1 3 2024-04-14T15:20:20.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.969+0000 7fc347179700 15 client.37467.objecter _session_command_op_remove -1 3 2024-04-14T15:20:20.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.969+0000 7fc347179700 20 client.37467.objecter get_session s=0x7fc3401a9ee0 osd=1 3 2024-04-14T15:20:20.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.969+0000 7fc347179700 15 client.37467.objecter _session_command_op_assign 1 3 2024-04-14T15:20:20.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.969+0000 7fc347179700 20 client.37467.objecter put_session s=0x7fc3401a9ee0 osd=1 4 2024-04-14T15:20:20.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.969+0000 7fc347179700 10 client.37467.objecter _send_command 3 2024-04-14T15:20:20.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.969+0000 7fc347179700 1 -- 172.21.15.90:0/1193298934 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fc34002de30 con 0x7fc3400fdb80 2024-04-14T15:20:20.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.970+0000 7fc33dffb700 10 client.37467.objecter ms_handle_connect 0x7fc3400fdb80 2024-04-14T15:20:20.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.970+0000 7fc33dffb700 1 -- 172.21.15.90:0/1193298934 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc34002de30 con 0x7fc3400fdb80 2024-04-14T15:20:20.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.970+0000 7fc33dffb700 10 client.37467.objecter ms_dispatch 0x7fc340001550 command_reply(tid 3: 0 ) 2024-04-14T15:20:20.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.970+0000 7fc33dffb700 10 client.37467.objecter _finish_command 3 = system:0 2024-04-14T15:20:20.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.970+0000 7fc33dffb700 20 client.37467.objecter put_session s=0x7fc3401a9ee0 osd=1 4 2024-04-14T15:20:20.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.970+0000 7fc33dffb700 15 client.37467.objecter _session_command_op_remove 1 3 2024-04-14T15:20:20.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.987+0000 7fc347179700 10 client.37467.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:20:20.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.987+0000 7fc347179700 15 client.37467.objecter _session_command_op_assign -1 4 2024-04-14T15:20:20.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.987+0000 7fc347179700 20 client.37467.objecter _get_session s=0x7fc3401a9ee0 osd=1 3 2024-04-14T15:20:20.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.987+0000 7fc347179700 20 client.37467.objecter put_session s=0x7fc3401a9ee0 osd=1 3 2024-04-14T15:20:20.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.987+0000 7fc347179700 20 client.37467.objecter _get_session s=0x7fc3401a9ee0 osd=1 3 2024-04-14T15:20:20.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.987+0000 7fc347179700 15 client.37467.objecter _session_command_op_remove -1 4 2024-04-14T15:20:20.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.987+0000 7fc347179700 20 client.37467.objecter get_session s=0x7fc3401a9ee0 osd=1 3 2024-04-14T15:20:20.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.987+0000 7fc347179700 15 client.37467.objecter _session_command_op_assign 1 4 2024-04-14T15:20:20.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.987+0000 7fc347179700 20 client.37467.objecter put_session s=0x7fc3401a9ee0 osd=1 4 2024-04-14T15:20:20.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.987+0000 7fc347179700 10 client.37467.objecter _send_command 4 2024-04-14T15:20:20.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.987+0000 7fc347179700 1 -- 172.21.15.90:0/1193298934 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fc34002e260 con 0x7fc3400fdb80 2024-04-14T15:20:20.079 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:20:20.079 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:20:20.079 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:20:20.079 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:20:20.079 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:20:20.079 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:20:20.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.988+0000 7fc33dffb700 1 -- 172.21.15.90:0/1193298934 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc34002e260 con 0x7fc3400fdb80 2024-04-14T15:20:20.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.988+0000 7fc33dffb700 10 client.37467.objecter ms_dispatch 0x7fc340001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:20:20.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.988+0000 7fc33dffb700 10 client.37467.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:20:20.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.988+0000 7fc33dffb700 20 client.37467.objecter put_session s=0x7fc3401a9ee0 osd=1 4 2024-04-14T15:20:20.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.988+0000 7fc33dffb700 15 client.37467.objecter _session_command_op_remove 1 4 2024-04-14T15:20:20.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.989+0000 7fc347179700 10 client.37467.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:20:20.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.989+0000 7fc347179700 15 client.37467.objecter _session_command_op_assign -1 5 2024-04-14T15:20:20.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.989+0000 7fc347179700 20 client.37467.objecter _get_session s=0x7fc34002e410 osd=2 3 2024-04-14T15:20:20.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.989+0000 7fc347179700 20 client.37467.objecter put_session s=0x7fc34002e410 osd=2 3 2024-04-14T15:20:20.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.989+0000 7fc347179700 20 client.37467.objecter _get_session s=0x7fc34002e410 osd=2 3 2024-04-14T15:20:20.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.989+0000 7fc347179700 15 client.37467.objecter _session_command_op_remove -1 5 2024-04-14T15:20:20.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.989+0000 7fc347179700 20 client.37467.objecter get_session s=0x7fc34002e410 osd=2 3 2024-04-14T15:20:20.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.989+0000 7fc347179700 15 client.37467.objecter _session_command_op_assign 2 5 2024-04-14T15:20:20.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.989+0000 7fc347179700 20 client.37467.objecter put_session s=0x7fc34002e410 osd=2 4 2024-04-14T15:20:20.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.989+0000 7fc347179700 10 client.37467.objecter _send_command 5 2024-04-14T15:20:20.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.989+0000 7fc347179700 1 -- 172.21.15.90:0/1193298934 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fc340063b70 con 0x7fc3400636c0 2024-04-14T15:20:20.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.989+0000 7fc33dffb700 10 client.37467.objecter ms_handle_connect 0x7fc3400636c0 2024-04-14T15:20:20.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.990+0000 7fc33dffb700 1 -- 172.21.15.90:0/1193298934 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc340063b70 con 0x7fc3400636c0 2024-04-14T15:20:20.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.990+0000 7fc33dffb700 10 client.37467.objecter ms_dispatch 0x7fc340001550 command_reply(tid 5: 0 ) 2024-04-14T15:20:20.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.990+0000 7fc33dffb700 10 client.37467.objecter _finish_command 5 = system:0 2024-04-14T15:20:20.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.990+0000 7fc33dffb700 20 client.37467.objecter put_session s=0x7fc34002e410 osd=2 4 2024-04-14T15:20:20.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:19.990+0000 7fc33dffb700 15 client.37467.objecter _session_command_op_remove 2 5 2024-04-14T15:20:20.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.006+0000 7fc347179700 10 client.37467.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:20:20.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.006+0000 7fc347179700 15 client.37467.objecter _session_command_op_assign -1 6 2024-04-14T15:20:20.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.006+0000 7fc347179700 20 client.37467.objecter _get_session s=0x7fc34002e410 osd=2 3 2024-04-14T15:20:20.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.006+0000 7fc347179700 20 client.37467.objecter put_session s=0x7fc34002e410 osd=2 3 2024-04-14T15:20:20.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.006+0000 7fc347179700 20 client.37467.objecter _get_session s=0x7fc34002e410 osd=2 3 2024-04-14T15:20:20.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.006+0000 7fc347179700 15 client.37467.objecter _session_command_op_remove -1 6 2024-04-14T15:20:20.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.006+0000 7fc347179700 20 client.37467.objecter get_session s=0x7fc34002e410 osd=2 3 2024-04-14T15:20:20.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.006+0000 7fc347179700 15 client.37467.objecter _session_command_op_assign 2 6 2024-04-14T15:20:20.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.006+0000 7fc347179700 20 client.37467.objecter put_session s=0x7fc34002e410 osd=2 4 2024-04-14T15:20:20.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.006+0000 7fc347179700 10 client.37467.objecter _send_command 6 2024-04-14T15:20:20.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.006+0000 7fc347179700 1 -- 172.21.15.90:0/1193298934 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fc34010e690 con 0x7fc3400636c0 2024-04-14T15:20:20.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.008+0000 7fc33dffb700 1 -- 172.21.15.90:0/1193298934 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc34010e690 con 0x7fc3400636c0 2024-04-14T15:20:20.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.008+0000 7fc33dffb700 10 client.37467.objecter ms_dispatch 0x7fc340001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:20:20.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.008+0000 7fc33dffb700 10 client.37467.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:20:20.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.008+0000 7fc33dffb700 20 client.37467.objecter put_session s=0x7fc34002e410 osd=2 4 2024-04-14T15:20:20.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.008+0000 7fc33dffb700 15 client.37467.objecter _session_command_op_remove 2 6 2024-04-14T15:20:20.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.009+0000 7fc347179700 10 client.37467.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:20:20.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.009+0000 7fc347179700 15 client.37467.objecter _session_command_op_assign -1 7 2024-04-14T15:20:20.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.009+0000 7fc347179700 20 client.37467.objecter _get_session s=0x7fc34010e690 osd=3 3 2024-04-14T15:20:20.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.009+0000 7fc347179700 20 client.37467.objecter put_session s=0x7fc34010e690 osd=3 3 2024-04-14T15:20:20.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.009+0000 7fc347179700 20 client.37467.objecter _get_session s=0x7fc34010e690 osd=3 3 2024-04-14T15:20:20.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.009+0000 7fc347179700 15 client.37467.objecter _session_command_op_remove -1 7 2024-04-14T15:20:20.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.009+0000 7fc347179700 20 client.37467.objecter get_session s=0x7fc34010e690 osd=3 3 2024-04-14T15:20:20.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.009+0000 7fc347179700 15 client.37467.objecter _session_command_op_assign 3 7 2024-04-14T15:20:20.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.009+0000 7fc347179700 20 client.37467.objecter put_session s=0x7fc34010e690 osd=3 4 2024-04-14T15:20:20.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.009+0000 7fc347179700 10 client.37467.objecter _send_command 7 2024-04-14T15:20:20.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.009+0000 7fc347179700 1 -- 172.21.15.90:0/1193298934 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fc3401b0270 con 0x7fc3401afe10 2024-04-14T15:20:20.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.009+0000 7fc33dffb700 10 client.37467.objecter ms_handle_connect 0x7fc3401afe10 2024-04-14T15:20:20.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.010+0000 7fc33dffb700 1 -- 172.21.15.90:0/1193298934 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc3401b0270 con 0x7fc3401afe10 2024-04-14T15:20:20.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.010+0000 7fc33dffb700 10 client.37467.objecter ms_dispatch 0x7fc340001550 command_reply(tid 7: 0 ) 2024-04-14T15:20:20.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.010+0000 7fc33dffb700 10 client.37467.objecter _finish_command 7 = system:0 2024-04-14T15:20:20.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.010+0000 7fc33dffb700 20 client.37467.objecter put_session s=0x7fc34010e690 osd=3 4 2024-04-14T15:20:20.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.010+0000 7fc33dffb700 15 client.37467.objecter _session_command_op_remove 3 7 2024-04-14T15:20:20.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.027+0000 7fc347179700 10 client.37467.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:20:20.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.027+0000 7fc347179700 15 client.37467.objecter _session_command_op_assign -1 8 2024-04-14T15:20:20.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.027+0000 7fc347179700 20 client.37467.objecter _get_session s=0x7fc34010e690 osd=3 3 2024-04-14T15:20:20.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.027+0000 7fc347179700 20 client.37467.objecter put_session s=0x7fc34010e690 osd=3 3 2024-04-14T15:20:20.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.027+0000 7fc347179700 20 client.37467.objecter _get_session s=0x7fc34010e690 osd=3 3 2024-04-14T15:20:20.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.027+0000 7fc347179700 15 client.37467.objecter _session_command_op_remove -1 8 2024-04-14T15:20:20.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.027+0000 7fc347179700 20 client.37467.objecter get_session s=0x7fc34010e690 osd=3 3 2024-04-14T15:20:20.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.027+0000 7fc347179700 15 client.37467.objecter _session_command_op_assign 3 8 2024-04-14T15:20:20.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.027+0000 7fc347179700 20 client.37467.objecter put_session s=0x7fc34010e690 osd=3 4 2024-04-14T15:20:20.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.027+0000 7fc347179700 10 client.37467.objecter _send_command 8 2024-04-14T15:20:20.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.027+0000 7fc347179700 1 -- 172.21.15.90:0/1193298934 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fc3401b0270 con 0x7fc3401afe10 2024-04-14T15:20:20.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.028+0000 7fc33dffb700 1 -- 172.21.15.90:0/1193298934 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc3401b0270 con 0x7fc3401afe10 2024-04-14T15:20:20.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.028+0000 7fc33dffb700 10 client.37467.objecter ms_dispatch 0x7fc340001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:20:20.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.028+0000 7fc33dffb700 10 client.37467.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:20:20.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.028+0000 7fc33dffb700 20 client.37467.objecter put_session s=0x7fc34010e690 osd=3 4 2024-04-14T15:20:20.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.028+0000 7fc33dffb700 15 client.37467.objecter _session_command_op_remove 3 8 2024-04-14T15:20:20.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.028+0000 7fc347179700 10 client.37467.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:20:20.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.028+0000 7fc347179700 15 client.37467.objecter _session_command_op_assign -1 9 2024-04-14T15:20:20.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.028+0000 7fc347179700 20 client.37467.objecter _get_session s=0x7fc3401b0270 osd=4 3 2024-04-14T15:20:20.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.028+0000 7fc347179700 20 client.37467.objecter put_session s=0x7fc3401b0270 osd=4 3 2024-04-14T15:20:20.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.028+0000 7fc347179700 20 client.37467.objecter _get_session s=0x7fc3401b0270 osd=4 3 2024-04-14T15:20:20.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.028+0000 7fc347179700 15 client.37467.objecter _session_command_op_remove -1 9 2024-04-14T15:20:20.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.028+0000 7fc347179700 20 client.37467.objecter get_session s=0x7fc3401b0270 osd=4 3 2024-04-14T15:20:20.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.028+0000 7fc347179700 15 client.37467.objecter _session_command_op_assign 4 9 2024-04-14T15:20:20.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.028+0000 7fc347179700 20 client.37467.objecter put_session s=0x7fc3401b0270 osd=4 4 2024-04-14T15:20:20.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.028+0000 7fc347179700 10 client.37467.objecter _send_command 9 2024-04-14T15:20:20.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.028+0000 7fc347179700 1 -- 172.21.15.90:0/1193298934 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fc3401106b0 con 0x7fc3401bb320 2024-04-14T15:20:20.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.029+0000 7fc33dffb700 10 client.37467.objecter ms_handle_connect 0x7fc3401bb320 2024-04-14T15:20:20.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.030+0000 7fc33dffb700 1 -- 172.21.15.90:0/1193298934 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc3401106b0 con 0x7fc3401bb320 2024-04-14T15:20:20.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.030+0000 7fc33dffb700 10 client.37467.objecter ms_dispatch 0x7fc340001550 command_reply(tid 9: 0 ) 2024-04-14T15:20:20.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.030+0000 7fc33dffb700 10 client.37467.objecter _finish_command 9 = system:0 2024-04-14T15:20:20.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.030+0000 7fc33dffb700 20 client.37467.objecter put_session s=0x7fc3401b0270 osd=4 4 2024-04-14T15:20:20.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.030+0000 7fc33dffb700 15 client.37467.objecter _session_command_op_remove 4 9 2024-04-14T15:20:20.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.046+0000 7fc347179700 10 client.37467.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:20:20.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.046+0000 7fc347179700 15 client.37467.objecter _session_command_op_assign -1 10 2024-04-14T15:20:20.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.046+0000 7fc347179700 20 client.37467.objecter _get_session s=0x7fc3401b0270 osd=4 3 2024-04-14T15:20:20.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.046+0000 7fc347179700 20 client.37467.objecter put_session s=0x7fc3401b0270 osd=4 3 2024-04-14T15:20:20.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.046+0000 7fc347179700 20 client.37467.objecter _get_session s=0x7fc3401b0270 osd=4 3 2024-04-14T15:20:20.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.046+0000 7fc347179700 15 client.37467.objecter _session_command_op_remove -1 10 2024-04-14T15:20:20.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.046+0000 7fc347179700 20 client.37467.objecter get_session s=0x7fc3401b0270 osd=4 3 2024-04-14T15:20:20.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.046+0000 7fc347179700 15 client.37467.objecter _session_command_op_assign 4 10 2024-04-14T15:20:20.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.046+0000 7fc347179700 20 client.37467.objecter put_session s=0x7fc3401b0270 osd=4 4 2024-04-14T15:20:20.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.046+0000 7fc347179700 10 client.37467.objecter _send_command 10 2024-04-14T15:20:20.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.046+0000 7fc347179700 1 -- 172.21.15.90:0/1193298934 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fc3401c0600 con 0x7fc3401bb320 2024-04-14T15:20:20.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.048+0000 7fc33dffb700 1 -- 172.21.15.90:0/1193298934 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc3401c0600 con 0x7fc3401bb320 2024-04-14T15:20:20.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.048+0000 7fc33dffb700 10 client.37467.objecter ms_dispatch 0x7fc340001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:20:20.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.048+0000 7fc33dffb700 10 client.37467.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:20:20.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.048+0000 7fc33dffb700 20 client.37467.objecter put_session s=0x7fc3401b0270 osd=4 4 2024-04-14T15:20:20.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.048+0000 7fc33dffb700 15 client.37467.objecter _session_command_op_remove 4 10 2024-04-14T15:20:20.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.048+0000 7fc347179700 10 client.37467.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:20:20.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.048+0000 7fc347179700 15 client.37467.objecter _session_command_op_assign -1 11 2024-04-14T15:20:20.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.048+0000 7fc347179700 20 client.37467.objecter _get_session s=0x7fc3401c0600 osd=5 3 2024-04-14T15:20:20.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.048+0000 7fc347179700 20 client.37467.objecter put_session s=0x7fc3401c0600 osd=5 3 2024-04-14T15:20:20.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.048+0000 7fc347179700 20 client.37467.objecter _get_session s=0x7fc3401c0600 osd=5 3 2024-04-14T15:20:20.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.048+0000 7fc347179700 15 client.37467.objecter _session_command_op_remove -1 11 2024-04-14T15:20:20.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.048+0000 7fc347179700 20 client.37467.objecter get_session s=0x7fc3401c0600 osd=5 3 2024-04-14T15:20:20.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.048+0000 7fc347179700 15 client.37467.objecter _session_command_op_assign 5 11 2024-04-14T15:20:20.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.048+0000 7fc347179700 20 client.37467.objecter put_session s=0x7fc3401c0600 osd=5 4 2024-04-14T15:20:20.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.048+0000 7fc347179700 10 client.37467.objecter _send_command 11 2024-04-14T15:20:20.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.048+0000 7fc347179700 1 -- 172.21.15.90:0/1193298934 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fc3401c1170 con 0x7fc3401c0d20 2024-04-14T15:20:20.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.049+0000 7fc33dffb700 10 client.37467.objecter ms_handle_connect 0x7fc3401c0d20 2024-04-14T15:20:20.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.049+0000 7fc33dffb700 1 -- 172.21.15.90:0/1193298934 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc3401c1170 con 0x7fc3401c0d20 2024-04-14T15:20:20.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.049+0000 7fc33dffb700 10 client.37467.objecter ms_dispatch 0x7fc340001550 command_reply(tid 11: 0 ) 2024-04-14T15:20:20.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.049+0000 7fc33dffb700 10 client.37467.objecter _finish_command 11 = system:0 2024-04-14T15:20:20.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.049+0000 7fc33dffb700 20 client.37467.objecter put_session s=0x7fc3401c0600 osd=5 4 2024-04-14T15:20:20.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.049+0000 7fc33dffb700 15 client.37467.objecter _session_command_op_remove 5 11 2024-04-14T15:20:20.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.066+0000 7fc347179700 10 client.37467.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:20:20.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.066+0000 7fc347179700 15 client.37467.objecter _session_command_op_assign -1 12 2024-04-14T15:20:20.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.066+0000 7fc347179700 20 client.37467.objecter _get_session s=0x7fc3401c0600 osd=5 3 2024-04-14T15:20:20.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.066+0000 7fc347179700 20 client.37467.objecter put_session s=0x7fc3401c0600 osd=5 3 2024-04-14T15:20:20.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.066+0000 7fc347179700 20 client.37467.objecter _get_session s=0x7fc3401c0600 osd=5 3 2024-04-14T15:20:20.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.066+0000 7fc347179700 15 client.37467.objecter _session_command_op_remove -1 12 2024-04-14T15:20:20.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.066+0000 7fc347179700 20 client.37467.objecter get_session s=0x7fc3401c0600 osd=5 3 2024-04-14T15:20:20.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.066+0000 7fc347179700 15 client.37467.objecter _session_command_op_assign 5 12 2024-04-14T15:20:20.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.066+0000 7fc347179700 20 client.37467.objecter put_session s=0x7fc3401c0600 osd=5 4 2024-04-14T15:20:20.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.066+0000 7fc347179700 10 client.37467.objecter _send_command 12 2024-04-14T15:20:20.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.066+0000 7fc347179700 1 -- 172.21.15.90:0/1193298934 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fc3401c1170 con 0x7fc3401c0d20 2024-04-14T15:20:20.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.068+0000 7fc33dffb700 1 -- 172.21.15.90:0/1193298934 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc3401c1170 con 0x7fc3401c0d20 2024-04-14T15:20:20.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.068+0000 7fc33dffb700 10 client.37467.objecter ms_dispatch 0x7fc340001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:20:20.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.068+0000 7fc33dffb700 10 client.37467.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:20:20.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.068+0000 7fc33dffb700 20 client.37467.objecter put_session s=0x7fc3401c0600 osd=5 4 2024-04-14T15:20:20.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.068+0000 7fc33dffb700 15 client.37467.objecter _session_command_op_remove 5 12 2024-04-14T15:20:20.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.068+0000 7fc347179700 10 client.37467.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:20:20.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.068+0000 7fc347179700 15 client.37467.objecter _session_command_op_assign -1 13 2024-04-14T15:20:20.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.068+0000 7fc347179700 20 client.37467.objecter _get_session s=0x7fc3401c1170 osd=6 3 2024-04-14T15:20:20.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.068+0000 7fc347179700 20 client.37467.objecter put_session s=0x7fc3401c1170 osd=6 3 2024-04-14T15:20:20.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.068+0000 7fc347179700 20 client.37467.objecter _get_session s=0x7fc3401c1170 osd=6 3 2024-04-14T15:20:20.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.068+0000 7fc347179700 15 client.37467.objecter _session_command_op_remove -1 13 2024-04-14T15:20:20.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.068+0000 7fc347179700 20 client.37467.objecter get_session s=0x7fc3401c1170 osd=6 3 2024-04-14T15:20:20.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.068+0000 7fc347179700 15 client.37467.objecter _session_command_op_assign 6 13 2024-04-14T15:20:20.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.068+0000 7fc347179700 20 client.37467.objecter put_session s=0x7fc3401c1170 osd=6 4 2024-04-14T15:20:20.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.068+0000 7fc347179700 10 client.37467.objecter _send_command 13 2024-04-14T15:20:20.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.068+0000 7fc347179700 1 -- 172.21.15.90:0/1193298934 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fc3401c15a0 con 0x7fc3401c3cf0 2024-04-14T15:20:20.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.069+0000 7fc33dffb700 10 client.37467.objecter ms_handle_connect 0x7fc3401c3cf0 2024-04-14T15:20:20.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.070+0000 7fc33dffb700 1 -- 172.21.15.90:0/1193298934 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc3401c15a0 con 0x7fc3401c3cf0 2024-04-14T15:20:20.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.070+0000 7fc33dffb700 10 client.37467.objecter ms_dispatch 0x7fc340001550 command_reply(tid 13: 0 ) 2024-04-14T15:20:20.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.070+0000 7fc33dffb700 10 client.37467.objecter _finish_command 13 = system:0 2024-04-14T15:20:20.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.070+0000 7fc33dffb700 20 client.37467.objecter put_session s=0x7fc3401c1170 osd=6 4 2024-04-14T15:20:20.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.070+0000 7fc33dffb700 15 client.37467.objecter _session_command_op_remove 6 13 2024-04-14T15:20:20.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.087+0000 7fc347179700 10 client.37467.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:20:20.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.087+0000 7fc347179700 15 client.37467.objecter _session_command_op_assign -1 14 2024-04-14T15:20:20.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.087+0000 7fc347179700 20 client.37467.objecter _get_session s=0x7fc3401c1170 osd=6 3 2024-04-14T15:20:20.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.087+0000 7fc347179700 20 client.37467.objecter put_session s=0x7fc3401c1170 osd=6 3 2024-04-14T15:20:20.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.087+0000 7fc347179700 20 client.37467.objecter _get_session s=0x7fc3401c1170 osd=6 3 2024-04-14T15:20:20.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.087+0000 7fc347179700 15 client.37467.objecter _session_command_op_remove -1 14 2024-04-14T15:20:20.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.087+0000 7fc347179700 20 client.37467.objecter get_session s=0x7fc3401c1170 osd=6 3 2024-04-14T15:20:20.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.087+0000 7fc347179700 15 client.37467.objecter _session_command_op_assign 6 14 2024-04-14T15:20:20.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.087+0000 7fc347179700 20 client.37467.objecter put_session s=0x7fc3401c1170 osd=6 4 2024-04-14T15:20:20.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.087+0000 7fc347179700 10 client.37467.objecter _send_command 14 2024-04-14T15:20:20.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.087+0000 7fc347179700 1 -- 172.21.15.90:0/1193298934 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fc3401c7800 con 0x7fc3401c3cf0 2024-04-14T15:20:20.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.089+0000 7fc33dffb700 1 -- 172.21.15.90:0/1193298934 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc3401c7800 con 0x7fc3401c3cf0 2024-04-14T15:20:20.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.089+0000 7fc33dffb700 10 client.37467.objecter ms_dispatch 0x7fc340001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:20:20.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.089+0000 7fc33dffb700 10 client.37467.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:20:20.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.089+0000 7fc33dffb700 20 client.37467.objecter put_session s=0x7fc3401c1170 osd=6 4 2024-04-14T15:20:20.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.089+0000 7fc33dffb700 15 client.37467.objecter _session_command_op_remove 6 14 2024-04-14T15:20:20.092 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:20:20.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.089+0000 7fc347179700 10 client.37467.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:20:20.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.089+0000 7fc347179700 15 client.37467.objecter _session_command_op_assign -1 15 2024-04-14T15:20:20.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.089+0000 7fc347179700 20 client.37467.objecter _get_session s=0x7fc3401c7800 osd=7 3 2024-04-14T15:20:20.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.089+0000 7fc347179700 20 client.37467.objecter put_session s=0x7fc3401c7800 osd=7 3 2024-04-14T15:20:20.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.089+0000 7fc347179700 20 client.37467.objecter _get_session s=0x7fc3401c7800 osd=7 3 2024-04-14T15:20:20.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.089+0000 7fc347179700 15 client.37467.objecter _session_command_op_remove -1 15 2024-04-14T15:20:20.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.089+0000 7fc347179700 20 client.37467.objecter get_session s=0x7fc3401c7800 osd=7 3 2024-04-14T15:20:20.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.089+0000 7fc347179700 15 client.37467.objecter _session_command_op_assign 7 15 2024-04-14T15:20:20.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.089+0000 7fc347179700 20 client.37467.objecter put_session s=0x7fc3401c7800 osd=7 4 2024-04-14T15:20:20.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.089+0000 7fc347179700 10 client.37467.objecter _send_command 15 2024-04-14T15:20:20.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.089+0000 7fc347179700 1 -- 172.21.15.90:0/1193298934 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fc3401c8ac0 con 0x7fc3401c7f20 2024-04-14T15:20:20.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.090+0000 7fc33dffb700 10 client.37467.objecter ms_handle_connect 0x7fc3401c7f20 2024-04-14T15:20:20.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.091+0000 7fc33dffb700 1 -- 172.21.15.90:0/1193298934 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc3401c8ac0 con 0x7fc3401c7f20 2024-04-14T15:20:20.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.091+0000 7fc33dffb700 10 client.37467.objecter ms_dispatch 0x7fc340001550 command_reply(tid 15: 0 ) 2024-04-14T15:20:20.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.091+0000 7fc33dffb700 10 client.37467.objecter _finish_command 15 = system:0 2024-04-14T15:20:20.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.091+0000 7fc33dffb700 20 client.37467.objecter put_session s=0x7fc3401c7800 osd=7 4 2024-04-14T15:20:20.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.091+0000 7fc33dffb700 15 client.37467.objecter _session_command_op_remove 7 15 2024-04-14T15:20:20.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.107+0000 7fc347179700 10 client.37467.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:20:20.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.107+0000 7fc347179700 15 client.37467.objecter _session_command_op_assign -1 16 2024-04-14T15:20:20.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.107+0000 7fc347179700 20 client.37467.objecter _get_session s=0x7fc3401c7800 osd=7 3 2024-04-14T15:20:20.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.107+0000 7fc347179700 20 client.37467.objecter put_session s=0x7fc3401c7800 osd=7 3 2024-04-14T15:20:20.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.107+0000 7fc347179700 20 client.37467.objecter _get_session s=0x7fc3401c7800 osd=7 3 2024-04-14T15:20:20.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.107+0000 7fc347179700 15 client.37467.objecter _session_command_op_remove -1 16 2024-04-14T15:20:20.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.107+0000 7fc347179700 20 client.37467.objecter get_session s=0x7fc3401c7800 osd=7 3 2024-04-14T15:20:20.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.107+0000 7fc347179700 15 client.37467.objecter _session_command_op_assign 7 16 2024-04-14T15:20:20.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.107+0000 7fc347179700 20 client.37467.objecter put_session s=0x7fc3401c7800 osd=7 4 2024-04-14T15:20:20.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.107+0000 7fc347179700 10 client.37467.objecter _send_command 16 2024-04-14T15:20:20.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.107+0000 7fc347179700 1 -- 172.21.15.90:0/1193298934 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fc3401c8ac0 con 0x7fc3401c7f20 2024-04-14T15:20:20.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.109+0000 7fc33dffb700 1 -- 172.21.15.90:0/1193298934 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc3401c8ac0 con 0x7fc3401c7f20 2024-04-14T15:20:20.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.109+0000 7fc33dffb700 10 client.37467.objecter ms_dispatch 0x7fc340001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:20:20.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.109+0000 7fc33dffb700 10 client.37467.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:20:20.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.109+0000 7fc33dffb700 20 client.37467.objecter put_session s=0x7fc3401c7800 osd=7 4 2024-04-14T15:20:20.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.109+0000 7fc33dffb700 15 client.37467.objecter _session_command_op_remove 7 16 2024-04-14T15:20:20.110 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:20:20.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.110+0000 7fc347179700 10 librados: watch_flush enter 2024-04-14T15:20:20.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.110+0000 7fc347179700 10 librados: watch_flush exit 2024-04-14T15:20:20.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.110+0000 7fc347179700 10 client.37467.objecter close_session for osd.0 2024-04-14T15:20:20.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.110+0000 7fc347179700 1 -- 172.21.15.90:0/1193298934 >> v1:172.21.15.90:6801/3761806885 conn(0x7fc3401a91a0 legacy=0x7fc3401a9580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:20.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.110+0000 7fc347179700 20 client.37467.objecter put_session s=0x7fc3401121b0 osd=0 1 2024-04-14T15:20:20.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.110+0000 7fc347179700 10 client.37467.objecter close_session for osd.1 2024-04-14T15:20:20.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.110+0000 7fc347179700 1 -- 172.21.15.90:0/1193298934 >> v1:172.21.15.90:6805/4170361699 conn(0x7fc3400fdb80 legacy=0x7fc3400fdfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:20.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.110+0000 7fc347179700 20 client.37467.objecter put_session s=0x7fc3401a9ee0 osd=1 1 2024-04-14T15:20:20.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.110+0000 7fc347179700 10 client.37467.objecter close_session for osd.2 2024-04-14T15:20:20.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.110+0000 7fc347179700 1 -- 172.21.15.90:0/1193298934 >> v1:172.21.15.90:6809/1686184251 conn(0x7fc3400636c0 legacy=0x7fc3401b27d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:20.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.110+0000 7fc347179700 20 client.37467.objecter put_session s=0x7fc34002e410 osd=2 1 2024-04-14T15:20:20.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.110+0000 7fc347179700 10 client.37467.objecter close_session for osd.3 2024-04-14T15:20:20.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.110+0000 7fc347179700 1 -- 172.21.15.90:0/1193298934 >> v1:172.21.15.90:6813/2270166774 conn(0x7fc3401afe10 legacy=0x7fc34010fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:20.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.110+0000 7fc347179700 20 client.37467.objecter put_session s=0x7fc34010e690 osd=3 1 2024-04-14T15:20:20.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.110+0000 7fc347179700 10 client.37467.objecter close_session for osd.4 2024-04-14T15:20:20.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.110+0000 7fc347179700 1 -- 172.21.15.90:0/1193298934 >> v1:172.21.15.92:6800/3887549733 conn(0x7fc3401bb320 legacy=0x7fc3401bd710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:20.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.110+0000 7fc347179700 20 client.37467.objecter put_session s=0x7fc3401b0270 osd=4 1 2024-04-14T15:20:20.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.110+0000 7fc347179700 10 client.37467.objecter close_session for osd.5 2024-04-14T15:20:20.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.110+0000 7fc347179700 1 -- 172.21.15.90:0/1193298934 >> v1:172.21.15.92:6804/3244561440 conn(0x7fc3401c0d20 legacy=0x7fc3401bee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:20.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.110+0000 7fc347179700 20 client.37467.objecter put_session s=0x7fc3401c0600 osd=5 1 2024-04-14T15:20:20.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.110+0000 7fc347179700 10 client.37467.objecter close_session for osd.6 2024-04-14T15:20:20.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.110+0000 7fc347179700 1 -- 172.21.15.90:0/1193298934 >> v1:172.21.15.92:6808/636125963 conn(0x7fc3401c3cf0 legacy=0x7fc3401c60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:20.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.110+0000 7fc347179700 20 client.37467.objecter put_session s=0x7fc3401c1170 osd=6 1 2024-04-14T15:20:20.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.110+0000 7fc347179700 10 client.37467.objecter close_session for osd.7 2024-04-14T15:20:20.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.110+0000 7fc347179700 1 -- 172.21.15.90:0/1193298934 >> v1:172.21.15.92:6812/1773706716 conn(0x7fc3401c7f20 legacy=0x7fc3401c8380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:20.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.110+0000 7fc347179700 20 client.37467.objecter put_session s=0x7fc3401c7800 osd=7 1 2024-04-14T15:20:20.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.110+0000 7fc347179700 20 client.37467.objecter shutdown clearing up homeless session... 2024-04-14T15:20:20.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.110+0000 7fc347179700 10 client.37467.objecter successfully canceled tick 2024-04-14T15:20:20.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.110+0000 7fc345716700 1 -- 172.21.15.90:0/1193298934 reap_dead start 2024-04-14T15:20:20.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.110+0000 7fc345716700 1 -- 172.21.15.90:0/1193298934 reap_dead start 2024-04-14T15:20:20.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.110+0000 7fc345716700 1 -- 172.21.15.90:0/1193298934 reap_dead start 2024-04-14T15:20:20.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.110+0000 7fc345716700 1 -- 172.21.15.90:0/1193298934 reap_dead start 2024-04-14T15:20:20.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.110+0000 7fc345716700 1 -- 172.21.15.90:0/1193298934 reap_dead start 2024-04-14T15:20:20.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.110+0000 7fc347179700 1 -- 172.21.15.90:0/1193298934 >> v1:172.21.15.90:6800/420935427 conn(0x7fc3280714a0 legacy=0x7fc328073960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:20.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.110+0000 7fc347179700 1 -- 172.21.15.90:0/1193298934 >> v1:172.21.15.90:6789/0 conn(0x7fc34010a610 legacy=0x7fc34010c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:20.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.110+0000 7fc347179700 1 -- 172.21.15.90:0/1193298934 shutdown_connections 2024-04-14T15:20:20.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.110+0000 7fc347179700 1 -- 172.21.15.90:0/1193298934 >> 172.21.15.90:0/1193298934 conn(0x7fc3400fd290 msgr2=0x7fc340100160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:20:20.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.110+0000 7fc347179700 1 -- 172.21.15.90:0/1193298934 shutdown_connections 2024-04-14T15:20:20.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.110+0000 7fc347179700 1 -- 172.21.15.90:0/1193298934 wait complete. 2024-04-14T15:20:20.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.110+0000 7fc347179700 1 librados: shutdown 2024-04-14T15:20:20.114 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:20:20.114 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:20:20.114 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:20:20.114 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:20:20.114 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:20:20.114 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:20:20.114 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:20:20.114 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:20:20.121 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:20:20.145 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:20.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:20.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:20:20.247 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:20:20.275 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:20.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:20.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:20:20.376 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:20:20.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:20.401 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:20.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:20:20.502 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:20:20.525 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:20.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:20.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:20:20.627 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:20.650 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:20.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:20.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:20.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.719+0000 7fb0caec8700 1 -- 172.21.15.90:0/1748319048 >> v1:172.21.15.90:6789/0 conn(0x7fb0c41077a0 legacy=0x7fb0c4107b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:20.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.719+0000 7fb0caec8700 1 -- 172.21.15.90:0/1748319048 shutdown_connections 2024-04-14T15:20:20.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.719+0000 7fb0caec8700 1 -- 172.21.15.90:0/1748319048 >> 172.21.15.90:0/1748319048 conn(0x7fb0c40fd2b0 msgr2=0x7fb0c40ff6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:20:20.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.720+0000 7fb0caec8700 1 -- 172.21.15.90:0/1748319048 shutdown_connections 2024-04-14T15:20:20.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.720+0000 7fb0caec8700 1 -- 172.21.15.90:0/1748319048 wait complete. 2024-04-14T15:20:20.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.720+0000 7fb0caec8700 1 librados: starting msgr at 2024-04-14T15:20:20.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.720+0000 7fb0caec8700 1 librados: starting objecter 2024-04-14T15:20:20.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.720+0000 7fb0caec8700 1 Processor -- start 2024-04-14T15:20:20.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.720+0000 7fb0caec8700 1 -- start start 2024-04-14T15:20:20.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.720+0000 7fb0caec8700 1 librados: setting wanted keys 2024-04-14T15:20:20.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.720+0000 7fb0caec8700 1 librados: calling monclient init 2024-04-14T15:20:20.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.720+0000 7fb0caec8700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb0c4110470 con 0x7fb0c410e220 2024-04-14T15:20:20.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.720+0000 7fb0caec8700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb0c4110680 con 0x7fb0c41077a0 2024-04-14T15:20:20.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.720+0000 7fb0caec8700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb0c41ac240 con 0x7fb0c410a650 2024-04-14T15:20:20.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.720+0000 7fb0c3fff700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7fb0c410a650 0x7fb0c410eeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:39680/0 (socket says 172.21.15.90:39680) 2024-04-14T15:20:20.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.720+0000 7fb0c3fff700 1 -- 172.21.15.90:0/2640704664 learned_addr learned my addr 172.21.15.90:0/2640704664 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:20:20.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.720+0000 7fb0c1ffb700 10 client.?.objecter ms_handle_connect 0x7fb0c410e220 2024-04-14T15:20:20.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.721+0000 7fb0c1ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:20:20.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.721+0000 7fb0c1ffb700 10 client.?.objecter ms_handle_connect 0x7fb0c410a650 2024-04-14T15:20:20.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.721+0000 7fb0c1ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:20:20.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.721+0000 7fb0c1ffb700 1 -- 172.21.15.90:0/2640704664 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4090686850 0 0) 0x7fb0c4110470 con 0x7fb0c410e220 2024-04-14T15:20:20.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.721+0000 7fb0c1ffb700 1 -- 172.21.15.90:0/2640704664 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb0b40035b0 con 0x7fb0c410e220 2024-04-14T15:20:20.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.721+0000 7fb0c1ffb700 1 -- 172.21.15.90:0/2640704664 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3394570753 0 0) 0x7fb0c41ac240 con 0x7fb0c410a650 2024-04-14T15:20:20.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.721+0000 7fb0c1ffb700 1 -- 172.21.15.90:0/2640704664 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb0c4110470 con 0x7fb0c410a650 2024-04-14T15:20:20.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.721+0000 7fb0c1ffb700 10 client.?.objecter ms_handle_connect 0x7fb0c41077a0 2024-04-14T15:20:20.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.721+0000 7fb0c1ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:20:20.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.721+0000 7fb0c1ffb700 1 -- 172.21.15.90:0/2640704664 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1399677967 0 0) 0x7fb0c4110680 con 0x7fb0c41077a0 2024-04-14T15:20:20.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.721+0000 7fb0c1ffb700 1 -- 172.21.15.90:0/2640704664 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb0c41ac240 con 0x7fb0c41077a0 2024-04-14T15:20:20.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.721+0000 7fb0c1ffb700 1 -- 172.21.15.90:0/2640704664 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 407284684 0 0) 0x7fb0b40035b0 con 0x7fb0c410e220 2024-04-14T15:20:20.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.721+0000 7fb0c1ffb700 1 -- 172.21.15.90:0/2640704664 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb0c4110680 con 0x7fb0c410e220 2024-04-14T15:20:20.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.721+0000 7fb0c1ffb700 1 -- 172.21.15.90:0/2640704664 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fb0bc001ae0 con 0x7fb0c410e220 2024-04-14T15:20:20.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.721+0000 7fb0c1ffb700 1 -- 172.21.15.90:0/2640704664 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2327663452 0 0) 0x7fb0c4110470 con 0x7fb0c410a650 2024-04-14T15:20:20.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.721+0000 7fb0c1ffb700 1 -- 172.21.15.90:0/2640704664 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb0b40035b0 con 0x7fb0c410a650 2024-04-14T15:20:20.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.721+0000 7fb0c1ffb700 1 -- 172.21.15.90:0/2640704664 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fb0b0001850 con 0x7fb0c410a650 2024-04-14T15:20:20.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.721+0000 7fb0c1ffb700 1 -- 172.21.15.90:0/2640704664 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2365362064 0 0) 0x7fb0c4110680 con 0x7fb0c410e220 2024-04-14T15:20:20.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.721+0000 7fb0c1ffb700 1 -- 172.21.15.90:0/2640704664 >> v1:172.21.15.90:6790/0 conn(0x7fb0c410a650 legacy=0x7fb0c410eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:20.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.721+0000 7fb0c1ffb700 1 -- 172.21.15.90:0/2640704664 >> v1:172.21.15.92:6789/0 conn(0x7fb0c41077a0 legacy=0x7fb0c410e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:20.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.721+0000 7fb0c1ffb700 1 -- 172.21.15.90:0/2640704664 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0c41ad460 con 0x7fb0c410e220 2024-04-14T15:20:20.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.721+0000 7fb0caec8700 1 -- 172.21.15.90:0/2640704664 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb0c41ac470 con 0x7fb0c410e220 2024-04-14T15:20:20.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.721+0000 7fb0c1ffb700 1 -- 172.21.15.90:0/2640704664 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fb0bc003ea0 con 0x7fb0c410e220 2024-04-14T15:20:20.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.721+0000 7fb0c1ffb700 1 -- 172.21.15.90:0/2640704664 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fb0bc004e30 con 0x7fb0c410e220 2024-04-14T15:20:20.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.721+0000 7fb0caec8700 10 client.37473.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:20:20.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.721+0000 7fb0caec8700 1 -- 172.21.15.90:0/2640704664 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb0c41ac930 con 0x7fb0c410e220 2024-04-14T15:20:20.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.721+0000 7fb0caec8700 1 librados: init done 2024-04-14T15:20:20.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.722+0000 7fb0caec8700 1 -- 172.21.15.90:0/2640704664 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0c4102ee0 con 0x7fb0c410e220 2024-04-14T15:20:20.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.723+0000 7fb0c1ffb700 1 -- 172.21.15.90:0/2640704664 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fb0bc003a00 con 0x7fb0c410e220 2024-04-14T15:20:20.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.724+0000 7fb0c1ffb700 1 -- 172.21.15.90:0/2640704664 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(498..498 src has 51..498) ==== 8197+0+0 (unknown 2199958775 0 0) 0x7fb0bc089d90 con 0x7fb0c410e220 2024-04-14T15:20:20.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.724+0000 7fb0c1ffb700 10 client.37473.objecter ms_dispatch 0x7fb0c4001550 osd_map(498..498 src has 51..498) 2024-04-14T15:20:20.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.724+0000 7fb0c1ffb700 3 client.37473.objecter handle_osd_map got epochs [498,498] > 0 2024-04-14T15:20:20.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.724+0000 7fb0c1ffb700 3 client.37473.objecter handle_osd_map decoding full epoch 498 2024-04-14T15:20:20.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.724+0000 7fb0c1ffb700 20 client.37473.objecter dump_active .. 0 homeless 2024-04-14T15:20:20.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.724+0000 7fb0c1ffb700 10 client.37473.objecter ms_handle_connect 0x7fb0b4071290 2024-04-14T15:20:20.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.728+0000 7fb0c1ffb700 1 -- 172.21.15.90:0/2640704664 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fb0bc0567a0 con 0x7fb0c410e220 2024-04-14T15:20:20.752 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:20:20.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:20.780 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:20.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:20:20.881 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:20:20.903 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:20.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:20.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:20:20.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:20.960+0000 7fb0caec8700 1 -- 172.21.15.90:0/2640704664 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7fb0c410fcd0 con 0x7fb0c410e220 2024-04-14T15:20:20.989 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-14T15:20:20.989 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-14T15:20:20.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-14T15:20:20.989 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-14T15:20:20.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:20.989 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25009.0:2 32.0 32:0384ab8e:::oid:head [write 18446744073709551615~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-04-14T15:20:20.990 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:12:55.060999+0000", 2024-04-14T15:20:20.990 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 445.92715983900001, 2024-04-14T15:20:20.990 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.494352567, 2024-04-14T15:20:20.990 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:20:20.990 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:20:20.990 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-14T15:20:20.990 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:20:20.990 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25009", 2024-04-14T15:20:20.990 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2850614897", 2024-04-14T15:20:20.990 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2 2024-04-14T15:20:20.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:20.990 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:20:20.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:20.991 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:20:20.991 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.060999+0000", 2024-04-14T15:20:20.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:20:20.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:20.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:20.991 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:20:20.991 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.060999+0000", 2024-04-14T15:20:20.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:20:20.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:20.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:20.991 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:20:20.991 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.061001+0000", 2024-04-14T15:20:20.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0030000000000001e-06 2024-04-14T15:20:20.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:20.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:20.992 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:20:20.992 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.061006+0000", 2024-04-14T15:20:20.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5450000000000003e-06 2024-04-14T15:20:20.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:20.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:20.992 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:20:20.992 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.061017+0000", 2024-04-14T15:20:20.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1090999999999999e-05 2024-04-14T15:20:20.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:20.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:20.992 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:20:20.992 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.061028+0000", 2024-04-14T15:20:20.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0675e-05 2024-04-14T15:20:20.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:20.993 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:20.993 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:20:20.993 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.066663+0000", 2024-04-14T15:20:20.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0056352490000000002 2024-04-14T15:20:20.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:20.993 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:20.993 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:20:20.993 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.066673+0000", 2024-04-14T15:20:20.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3400000000000004e-06 2024-04-14T15:20:20.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:20.993 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:20.993 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:20:20.993 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.470857+0000", 2024-04-14T15:20:20.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.40418453 2024-04-14T15:20:20.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:20.993 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:20.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:20:20.994 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.471681+0000", 2024-04-14T15:20:20.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00082436499999999997 2024-04-14T15:20:20.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:20.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:20.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:20:20.994 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.555351+0000", 2024-04-14T15:20:20.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.083669769000000005 2024-04-14T15:20:20.994 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:20:20.994 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:20:20.994 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:20:20.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:20.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:20.994 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.24965.0:2 30.1a e108/107 30:5c6776db:::0.obj:head v 108'1, mlcod=0'0)", 2024-04-14T15:20:20.994 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:12:56.467908+0000", 2024-04-14T15:20:20.995 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 444.52025070299999, 2024-04-14T15:20:20.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.031681980999999998, 2024-04-14T15:20:20.995 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:20:20.995 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:20:20.995 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:20:20.995 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:20:20.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:20.995 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:20:20.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.467908+0000", 2024-04-14T15:20:20.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:20:20.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:20.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:20.995 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:20:20.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.467908+0000", 2024-04-14T15:20:20.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:20:20.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:20.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:20.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:20:20.996 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.467908+0000", 2024-04-14T15:20:20.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9700000000000003e-07 2024-04-14T15:20:20.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:20.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:20.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:20:20.996 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.467914+0000", 2024-04-14T15:20:20.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4700000000000001e-06 2024-04-14T15:20:20.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:20.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:20.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:20:20.996 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.467920+0000", 2024-04-14T15:20:20.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5509999999999996e-06 2024-04-14T15:20:20.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:20.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:20.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:20:20.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.467928+0000", 2024-04-14T15:20:20.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9729999999999997e-06 2024-04-14T15:20:20.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:20.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:20.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:20:20.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.469047+0000", 2024-04-14T15:20:20.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011187 2024-04-14T15:20:20.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:20.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:20.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:20:20.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.469065+0000", 2024-04-14T15:20:20.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8732e-05 2024-04-14T15:20:20.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:20.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:20.998 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:20:20.998 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.499557+0000", 2024-04-14T15:20:20.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030491082999999999 2024-04-14T15:20:20.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:20.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:20.998 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:20:20.998 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.499590+0000", 2024-04-14T15:20:20.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3175e-05 2024-04-14T15:20:20.998 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:20:20.998 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:20:20.998 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:20:20.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:20.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:20.999 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.25144.0:16 24.10 e108/107 24:09666ec2:::28.obj2:head v 108'13, mlcod=108'11)", 2024-04-14T15:20:20.999 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:12:57.294151+0000", 2024-04-14T15:20:20.999 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 443.69400778400001, 2024-04-14T15:20:20.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.042811227, 2024-04-14T15:20:20.999 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:20:20.999 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:20:20.999 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:20:20.999 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:20:20.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:20.999 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:20:20.999 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.294151+0000", 2024-04-14T15:20:20.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:20:20.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:20.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:20.999 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:20:21.000 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.294151+0000", 2024-04-14T15:20:21.000 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:20:21.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.000 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.000 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:20:21.000 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.294151+0000", 2024-04-14T15:20:21.000 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8200000000000001e-07 2024-04-14T15:20:21.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.000 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.000 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:20:21.000 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.294170+0000", 2024-04-14T15:20:21.000 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8800999999999998e-05 2024-04-14T15:20:21.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:20:21.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.294178+0000", 2024-04-14T15:20:21.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9710000000000004e-06 2024-04-14T15:20:21.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:20:21.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.294192+0000", 2024-04-14T15:20:21.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4623e-05 2024-04-14T15:20:21.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:20:21.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.294994+0000", 2024-04-14T15:20:21.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00080199199999999996 2024-04-14T15:20:21.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.002 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:20:21.002 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.295017+0000", 2024-04-14T15:20:21.002 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2606999999999999e-05 2024-04-14T15:20:21.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.002 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:20:21.002 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.336915+0000", 2024-04-14T15:20:21.002 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.041897917 2024-04-14T15:20:21.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.002 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:20:21.002 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.336962+0000", 2024-04-14T15:20:21.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7033999999999997e-05 2024-04-14T15:20:21.003 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:20:21.003 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:20:21.003 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:20:21.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.004 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.26497.0:35 112.1 e133/122 112:8133d882:test-rados-api-smithi090-93724-10::foo:head v 133'7, mlcod=132'4)", 2024-04-14T15:20:21.004 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:13:20.911793+0000", 2024-04-14T15:20:21.004 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 420.07636557199999, 2024-04-14T15:20:21.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.031446676, 2024-04-14T15:20:21.004 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:20:21.004 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:20:21.004 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:20:21.004 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:20:21.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.004 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:20:21.004 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.911793+0000", 2024-04-14T15:20:21.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:20:21.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.005 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:20:21.005 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.911793+0000", 2024-04-14T15:20:21.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:20:21.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.005 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:20:21.005 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.911793+0000", 2024-04-14T15:20:21.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1e-07 2024-04-14T15:20:21.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.005 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:20:21.005 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.911802+0000", 2024-04-14T15:20:21.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0100000000000001e-06 2024-04-14T15:20:21.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.006 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:20:21.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.911809+0000", 2024-04-14T15:20:21.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3319999999999997e-06 2024-04-14T15:20:21.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.006 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:20:21.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.911818+0000", 2024-04-14T15:20:21.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.166e-06 2024-04-14T15:20:21.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.006 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:20:21.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.912551+0000", 2024-04-14T15:20:21.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00073312300000000002 2024-04-14T15:20:21.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:20:21.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.912572+0000", 2024-04-14T15:20:21.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1274e-05 2024-04-14T15:20:21.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:20:21.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.943196+0000", 2024-04-14T15:20:21.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030623862000000002 2024-04-14T15:20:21.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:20:21.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.943240+0000", 2024-04-14T15:20:21.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3599e-05 2024-04-14T15:20:21.007 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:20:21.007 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:20:21.008 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:20:21.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.008 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.26497.0:36 112.1 e133/122 112:8133d882:test-rados-api-smithi090-93724-10::foo:head v 133'8, mlcod=132'5)", 2024-04-14T15:20:21.008 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:13:20.947031+0000", 2024-04-14T15:20:21.008 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 420.041127571, 2024-04-14T15:20:21.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.026026404999999999, 2024-04-14T15:20:21.008 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:20:21.008 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:20:21.008 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:20:21.008 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:20:21.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:20:21.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.947031+0000", 2024-04-14T15:20:21.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:20:21.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:20:21.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.947031+0000", 2024-04-14T15:20:21.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:20:21.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:20:21.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.947033+0000", 2024-04-14T15:20:21.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9139999999999998e-06 2024-04-14T15:20:21.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:20:21.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.947044+0000", 2024-04-14T15:20:21.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1134999999999999e-05 2024-04-14T15:20:21.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:20:21.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.947063+0000", 2024-04-14T15:20:21.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8853999999999999e-05 2024-04-14T15:20:21.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:20:21.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.947074+0000", 2024-04-14T15:20:21.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0815999999999999e-05 2024-04-14T15:20:21.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:20:21.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.947153+0000", 2024-04-14T15:20:21.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9678999999999994e-05 2024-04-14T15:20:21.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:20:21.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.947177+0000", 2024-04-14T15:20:21.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3332999999999998e-05 2024-04-14T15:20:21.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:20:21.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.972985+0000", 2024-04-14T15:20:21.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.025808122999999999 2024-04-14T15:20:21.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:20:21.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.973057+0000", 2024-04-14T15:20:21.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2551000000000004e-05 2024-04-14T15:20:21.012 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:20:21.012 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:20:21.012 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:20:21.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.012 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.012 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.26497.0:51 112.e 112:75b427dd:test-rados-api-smithi090-93724-11::foo:head [write 0~1048576] snapc 9=[9] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-04-14T15:20:21.012 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:13:25.825582+0000", 2024-04-14T15:20:21.012 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 415.16257621099999, 2024-04-14T15:20:21.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.141706371, 2024-04-14T15:20:21.012 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:20:21.012 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:20:21.012 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:20:21.012 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:20:21.012 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.26497", 2024-04-14T15:20:21.013 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3962832909", 2024-04-14T15:20:21.013 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 51 2024-04-14T15:20:21.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.013 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:20:21.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:20:21.013 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:25.825582+0000", 2024-04-14T15:20:21.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:20:21.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:20:21.013 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:25.825582+0000", 2024-04-14T15:20:21.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:20:21.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:20:21.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:25.825584+0000", 2024-04-14T15:20:21.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9069999999999999e-06 2024-04-14T15:20:21.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:20:21.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:25.826345+0000", 2024-04-14T15:20:21.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076040899999999998 2024-04-14T15:20:21.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:20:21.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:25.826358+0000", 2024-04-14T15:20:21.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2989999999999999e-05 2024-04-14T15:20:21.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:20:21.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:25.826369+0000", 2024-04-14T15:20:21.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.128e-05 2024-04-14T15:20:21.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:20:21.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:25.826434+0000", 2024-04-14T15:20:21.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4833999999999996e-05 2024-04-14T15:20:21.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2024-04-14T15:20:21.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:25.826441+0000", 2024-04-14T15:20:21.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9890000000000003e-06 2024-04-14T15:20:21.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:20:21.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:25.865309+0000", 2024-04-14T15:20:21.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.038867983000000002 2024-04-14T15:20:21.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.016 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:20:21.016 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:25.865442+0000", 2024-04-14T15:20:21.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133158 2024-04-14T15:20:21.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.016 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2024-04-14T15:20:21.016 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:25.865580+0000", 2024-04-14T15:20:21.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000138615 2024-04-14T15:20:21.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.016 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:20:21.016 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:25.872696+0000", 2024-04-14T15:20:21.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0071156220000000003 2024-04-14T15:20:21.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:20:21.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:25.921899+0000", 2024-04-14T15:20:21.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.049203021999999999 2024-04-14T15:20:21.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:20:21.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:25.967182+0000", 2024-04-14T15:20:21.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.045283367999999997 2024-04-14T15:20:21.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:20:21.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:25.967240+0000", 2024-04-14T15:20:21.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7874999999999997e-05 2024-04-14T15:20:21.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:20:21.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:25.967289+0000", 2024-04-14T15:20:21.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8319000000000003e-05 2024-04-14T15:20:21.018 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:20:21.018 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:20:21.018 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:20:21.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.018 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.26497.0:53 112.e 112:75b427dd:test-rados-api-smithi090-93724-11::foo:head [stat,delete] snapc a=[a,9] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-04-14T15:20:21.018 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:13:26.893558+0000", 2024-04-14T15:20:21.018 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 414.09460073100001, 2024-04-14T15:20:21.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.059330368000000001, 2024-04-14T15:20:21.018 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:20:21.019 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:20:21.019 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:20:21.019 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:20:21.019 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.26497", 2024-04-14T15:20:21.019 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3962832909", 2024-04-14T15:20:21.019 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 53 2024-04-14T15:20:21.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.019 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:20:21.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.019 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:20:21.019 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.893558+0000", 2024-04-14T15:20:21.019 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:20:21.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.019 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:20:21.019 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.893558+0000", 2024-04-14T15:20:21.020 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:20:21.020 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.020 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:20:21.020 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.893560+0000", 2024-04-14T15:20:21.020 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.863e-06 2024-04-14T15:20:21.020 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.020 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:20:21.020 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.893565+0000", 2024-04-14T15:20:21.020 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.344e-06 2024-04-14T15:20:21.020 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:20:21.021 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.893576+0000", 2024-04-14T15:20:21.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0743e-05 2024-04-14T15:20:21.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:20:21.021 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.893585+0000", 2024-04-14T15:20:21.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7990000000000005e-06 2024-04-14T15:20:21.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:20:21.021 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.894455+0000", 2024-04-14T15:20:21.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00086904300000000001 2024-04-14T15:20:21.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.022 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:20:21.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.897306+0000", 2024-04-14T15:20:21.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028518160000000001 2024-04-14T15:20:21.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.022 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2024-04-14T15:20:21.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.897553+0000", 2024-04-14T15:20:21.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024705700000000002 2024-04-14T15:20:21.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.022 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:20:21.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.913549+0000", 2024-04-14T15:20:21.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015995939000000001 2024-04-14T15:20:21.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:20:21.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.946641+0000", 2024-04-14T15:20:21.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.033091655999999997 2024-04-14T15:20:21.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:20:21.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.952773+0000", 2024-04-14T15:20:21.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0061320630000000001 2024-04-14T15:20:21.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:20:21.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.952835+0000", 2024-04-14T15:20:21.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2082999999999995e-05 2024-04-14T15:20:21.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.024 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:20:21.024 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.952888+0000", 2024-04-14T15:20:21.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2961999999999999e-05 2024-04-14T15:20:21.024 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:20:21.024 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:20:21.024 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:20:21.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.024 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.25684.0:248 110.1c e139/121 110:3d310c86:test-rados-api-smithi090-93652-12::bar:head v 139'1, mlcod=0'0)", 2024-04-14T15:20:21.024 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:13:26.946390+0000", 2024-04-14T15:20:21.024 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 414.041768225, 2024-04-14T15:20:21.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.026315974999999998, 2024-04-14T15:20:21.024 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:20:21.024 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:20:21.024 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:20:21.025 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:20:21.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:20:21.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.946390+0000", 2024-04-14T15:20:21.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:20:21.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:20:21.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.946390+0000", 2024-04-14T15:20:21.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:20:21.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:20:21.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.946391+0000", 2024-04-14T15:20:21.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9000000000000002e-07 2024-04-14T15:20:21.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:20:21.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.946397+0000", 2024-04-14T15:20:21.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7460000000000003e-06 2024-04-14T15:20:21.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:20:21.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.946403+0000", 2024-04-14T15:20:21.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4480000000000002e-06 2024-04-14T15:20:21.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:20:21.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.946411+0000", 2024-04-14T15:20:21.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2649999999999996e-06 2024-04-14T15:20:21.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:20:21.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.959679+0000", 2024-04-14T15:20:21.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013267582 2024-04-14T15:20:21.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:20:21.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.959698+0000", 2024-04-14T15:20:21.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9508e-05 2024-04-14T15:20:21.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:20:21.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.972667+0000", 2024-04-14T15:20:21.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012968841 2024-04-14T15:20:21.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:20:21.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.972706+0000", 2024-04-14T15:20:21.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9195e-05 2024-04-14T15:20:21.028 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:20:21.028 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:20:21.028 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:20:21.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.028 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(149.6s1 141/136 ECSubWrite(tid=801, reqid=client.38086.0:7, at_version=141'1, trim_to=0'0, roll_forward_to=0'0))", 2024-04-14T15:20:21.028 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:13:28.866071+0000", 2024-04-14T15:20:21.028 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 412.12208700899998, 2024-04-14T15:20:21.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.028275307, 2024-04-14T15:20:21.028 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:20:21.028 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:20:21.028 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-04-14T15:20:21.028 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:20:21.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:20:21.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.866071+0000", 2024-04-14T15:20:21.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:20:21.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:20:21.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.866071+0000", 2024-04-14T15:20:21.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:20:21.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:20:21.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.866072+0000", 2024-04-14T15:20:21.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3500000000000002e-07 2024-04-14T15:20:21.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:20:21.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.866102+0000", 2024-04-14T15:20:21.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9887e-05 2024-04-14T15:20:21.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:20:21.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.866110+0000", 2024-04-14T15:20:21.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0909999999999993e-06 2024-04-14T15:20:21.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:20:21.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.866120+0000", 2024-04-14T15:20:21.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0360000000000001e-05 2024-04-14T15:20:21.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:20:21.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.884475+0000", 2024-04-14T15:20:21.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.018355172999999999 2024-04-14T15:20:21.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2024-04-14T15:20:21.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.884486+0000", 2024-04-14T15:20:21.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0404999999999999e-05 2024-04-14T15:20:21.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2024-04-14T15:20:21.032 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.894318+0000", 2024-04-14T15:20:21.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0098320620000000008 2024-04-14T15:20:21.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:20:21.032 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.894347+0000", 2024-04-14T15:20:21.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8994000000000001e-05 2024-04-14T15:20:21.032 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:20:21.032 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:20:21.032 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:20:21.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.032 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(149.6s1 141/136 ECSubWrite(tid=802, reqid=client.38086.0:7, at_version=0'0, trim_to=0'0, roll_forward_to=141'1))", 2024-04-14T15:20:21.032 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:13:28.900478+0000", 2024-04-14T15:20:21.032 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 412.08768062299998, 2024-04-14T15:20:21.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.027266736999999999, 2024-04-14T15:20:21.033 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:20:21.033 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:20:21.033 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-04-14T15:20:21.033 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:20:21.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:20:21.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.900478+0000", 2024-04-14T15:20:21.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:20:21.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:20:21.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.900478+0000", 2024-04-14T15:20:21.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:20:21.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:20:21.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.900478+0000", 2024-04-14T15:20:21.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3300000000000002e-07 2024-04-14T15:20:21.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:20:21.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.900488+0000", 2024-04-14T15:20:21.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0078000000000001e-05 2024-04-14T15:20:21.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:20:21.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.900495+0000", 2024-04-14T15:20:21.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0369999999999997e-06 2024-04-14T15:20:21.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:20:21.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.900506+0000", 2024-04-14T15:20:21.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0472000000000001e-05 2024-04-14T15:20:21.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:20:21.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.901653+0000", 2024-04-14T15:20:21.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011474359999999999 2024-04-14T15:20:21.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2024-04-14T15:20:21.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.901664+0000", 2024-04-14T15:20:21.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0267e-05 2024-04-14T15:20:21.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2024-04-14T15:20:21.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.927711+0000", 2024-04-14T15:20:21.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.026047229000000002 2024-04-14T15:20:21.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:20:21.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.927745+0000", 2024-04-14T15:20:21.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3685e-05 2024-04-14T15:20:21.036 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:20:21.036 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:20:21.036 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:20:21.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.036 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.26497.0:57 112.1a e142/122 112:5978e4a5:test-rados-api-smithi090-93724-12::foo:head v 142'1, mlcod=0'0)", 2024-04-14T15:20:21.037 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:13:29.900967+0000", 2024-04-14T15:20:21.037 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 411.08719174999999, 2024-04-14T15:20:21.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.105006778, 2024-04-14T15:20:21.037 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:20:21.037 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:20:21.037 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:20:21.037 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:20:21.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:20:21.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.900967+0000", 2024-04-14T15:20:21.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:20:21.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:20:21.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.900967+0000", 2024-04-14T15:20:21.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:20:21.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:20:21.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.900967+0000", 2024-04-14T15:20:21.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0400000000000002e-07 2024-04-14T15:20:21.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:20:21.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.900976+0000", 2024-04-14T15:20:21.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9080000000000001e-06 2024-04-14T15:20:21.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:20:21.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.934576+0000", 2024-04-14T15:20:21.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.033600179000000001 2024-04-14T15:20:21.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:20:21.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.934588+0000", 2024-04-14T15:20:21.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1939e-05 2024-04-14T15:20:21.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:20:21.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.937063+0000", 2024-04-14T15:20:21.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0024749889999999999 2024-04-14T15:20:21.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:20:21.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.937084+0000", 2024-04-14T15:20:21.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1220999999999999e-05 2024-04-14T15:20:21.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.040 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:20:21.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:30.005783+0000", 2024-04-14T15:20:21.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.068699239999999995 2024-04-14T15:20:21.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.040 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:20:21.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:30.005973+0000", 2024-04-14T15:20:21.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018999800000000001 2024-04-14T15:20:21.040 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:20:21.040 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:20:21.041 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:20:21.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.041 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.48565.0:1 239.5s0 239:ab97ba35:::nonexistent:head [read 0~128] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e236)", 2024-04-14T15:20:21.041 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:15:03.863599+0000", 2024-04-14T15:20:21.041 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 317.12455977399998, 2024-04-14T15:20:21.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0325199839999999, 2024-04-14T15:20:21.041 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:20:21.041 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:20:21.041 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-04-14T15:20:21.041 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:20:21.041 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.48565", 2024-04-14T15:20:21.041 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3767512710", 2024-04-14T15:20:21.041 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1 2024-04-14T15:20:21.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.042 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:20:21.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:20:21.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:03.863599+0000", 2024-04-14T15:20:21.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:20:21.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:20:21.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:03.863599+0000", 2024-04-14T15:20:21.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:20:21.043 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:20:21.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:20:21.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:03.863601+0000", 2024-04-14T15:20:21.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.995e-06 2024-04-14T15:20:21.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:20:21.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:03.863607+0000", 2024-04-14T15:20:21.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2700000000000001e-06 2024-04-14T15:20:21.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:20:21.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:03.863618+0000", 2024-04-14T15:20:21.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1368e-05 2024-04-14T15:20:21.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:20:21.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:03.863630+0000", 2024-04-14T15:20:21.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1324e-05 2024-04-14T15:20:21.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:20:21.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:03.863700+0000", 2024-04-14T15:20:21.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0399000000000001e-05 2024-04-14T15:20:21.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:20:21.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:03.863703+0000", 2024-04-14T15:20:21.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2349999999999999e-06 2024-04-14T15:20:21.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:20:21.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:04.896017+0000", 2024-04-14T15:20:21.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0323134599999999 2024-04-14T15:20:21.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:20:21.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:04.896119+0000", 2024-04-14T15:20:21.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000101933 2024-04-14T15:20:21.046 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:20:21.046 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:20:21.046 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:20:21.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.046 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25684.0:2198 247.1f 247:f8000000:%01:::head [pgnls start_epoch 253] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e253)", 2024-04-14T15:20:21.046 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:15:20.904026+0000", 2024-04-14T15:20:21.046 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 300.084132501, 2024-04-14T15:20:21.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.98221185899999996, 2024-04-14T15:20:21.046 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:20:21.046 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:20:21.047 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-14T15:20:21.047 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:20:21.047 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25684", 2024-04-14T15:20:21.047 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1109245683", 2024-04-14T15:20:21.047 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2198 2024-04-14T15:20:21.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.047 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:20:21.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:20:21.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:20.904026+0000", 2024-04-14T15:20:21.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:20:21.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:20:21.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:20.904026+0000", 2024-04-14T15:20:21.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:20:21.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:20:21.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:20.904029+0000", 2024-04-14T15:20:21.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0900000000000001e-06 2024-04-14T15:20:21.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:20:21.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:20.904037+0000", 2024-04-14T15:20:21.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3319999999999992e-06 2024-04-14T15:20:21.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:20:21.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:20.904053+0000", 2024-04-14T15:20:21.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5311e-05 2024-04-14T15:20:21.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:20:21.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:20.904061+0000", 2024-04-14T15:20:21.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1659999999999994e-06 2024-04-14T15:20:21.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:20:21.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:20.904133+0000", 2024-04-14T15:20:21.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2409000000000004e-05 2024-04-14T15:20:21.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:20:21.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:20.904146+0000", 2024-04-14T15:20:21.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2639e-05 2024-04-14T15:20:21.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:20:21.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:21.885405+0000", 2024-04-14T15:20:21.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.981259192 2024-04-14T15:20:21.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:20:21.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:21.885442+0000", 2024-04-14T15:20:21.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7014000000000002e-05 2024-04-14T15:20:21.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:20:21.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:21.886238+0000", 2024-04-14T15:20:21.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00079570599999999997 2024-04-14T15:20:21.051 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:20:21.051 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:20:21.051 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:20:21.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.051 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.36840.0:1 256.6s0 256:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e270)", 2024-04-14T15:20:21.051 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:15:37.877392+0000", 2024-04-14T15:20:21.051 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 283.110766356, 2024-04-14T15:20:21.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0013250600000001, 2024-04-14T15:20:21.051 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:20:21.051 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:20:21.051 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:20:21.051 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:20:21.051 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.36840", 2024-04-14T15:20:21.052 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2015849731", 2024-04-14T15:20:21.052 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1 2024-04-14T15:20:21.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.052 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:20:21.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:20:21.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:37.877392+0000", 2024-04-14T15:20:21.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:20:21.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:20:21.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:37.877392+0000", 2024-04-14T15:20:21.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:20:21.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:20:21.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:37.877394+0000", 2024-04-14T15:20:21.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.702e-06 2024-04-14T15:20:21.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:20:21.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:37.877400+0000", 2024-04-14T15:20:21.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2689999999999996e-06 2024-04-14T15:20:21.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:20:21.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:37.877410+0000", 2024-04-14T15:20:21.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0365e-05 2024-04-14T15:20:21.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:20:21.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:37.877421+0000", 2024-04-14T15:20:21.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.082e-05 2024-04-14T15:20:21.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:20:21.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:37.877468+0000", 2024-04-14T15:20:21.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6479e-05 2024-04-14T15:20:21.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:20:21.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:37.877472+0000", 2024-04-14T15:20:21.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.907e-06 2024-04-14T15:20:21.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:20:21.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:38.875012+0000", 2024-04-14T15:20:21.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99754053399999998 2024-04-14T15:20:21.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:20:21.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:38.875130+0000", 2024-04-14T15:20:21.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011831499999999999 2024-04-14T15:20:21.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2024-04-14T15:20:21.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:38.876401+0000", 2024-04-14T15:20:21.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001270544 2024-04-14T15:20:21.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.056 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2024-04-14T15:20:21.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:38.876968+0000", 2024-04-14T15:20:21.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00056727100000000005 2024-04-14T15:20:21.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.056 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:20:21.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:38.878670+0000", 2024-04-14T15:20:21.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0017013379999999999 2024-04-14T15:20:21.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.056 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:20:21.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:38.878717+0000", 2024-04-14T15:20:21.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7515999999999997e-05 2024-04-14T15:20:21.057 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:20:21.057 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:20:21.057 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:20:21.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.057 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.48691.0:1 257.6s0 257:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-04-14T15:20:21.057 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:15:38.872646+0000", 2024-04-14T15:20:21.057 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 282.11551224499999, 2024-04-14T15:20:21.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.015757738, 2024-04-14T15:20:21.057 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:20:21.057 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:20:21.057 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:20:21.057 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:20:21.057 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.48691", 2024-04-14T15:20:21.057 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3211270109", 2024-04-14T15:20:21.058 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1 2024-04-14T15:20:21.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.058 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:20:21.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:20:21.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:38.872646+0000", 2024-04-14T15:20:21.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:20:21.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:20:21.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:38.872646+0000", 2024-04-14T15:20:21.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:20:21.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:20:21.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:38.872648+0000", 2024-04-14T15:20:21.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8899999999999999e-06 2024-04-14T15:20:21.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:20:21.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:38.872655+0000", 2024-04-14T15:20:21.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6599999999999998e-06 2024-04-14T15:20:21.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:20:21.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:38.872666+0000", 2024-04-14T15:20:21.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1158000000000001e-05 2024-04-14T15:20:21.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:20:21.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:38.872676+0000", 2024-04-14T15:20:21.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0295000000000001e-05 2024-04-14T15:20:21.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:20:21.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:38.873327+0000", 2024-04-14T15:20:21.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00065083600000000001 2024-04-14T15:20:21.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:20:21.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:38.873331+0000", 2024-04-14T15:20:21.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6030000000000001e-06 2024-04-14T15:20:21.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:20:21.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:39.881515+0000", 2024-04-14T15:20:21.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.00818429 2024-04-14T15:20:21.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:20:21.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:39.881635+0000", 2024-04-14T15:20:21.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000119824 2024-04-14T15:20:21.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2024-04-14T15:20:21.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:39.882074+0000", 2024-04-14T15:20:21.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00043970599999999999 2024-04-14T15:20:21.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2024-04-14T15:20:21.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:39.882644+0000", 2024-04-14T15:20:21.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000569251 2024-04-14T15:20:21.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.062 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:20:21.062 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:39.888332+0000", 2024-04-14T15:20:21.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0056887109999999999 2024-04-14T15:20:21.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.062 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:20:21.062 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:39.888404+0000", 2024-04-14T15:20:21.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1513999999999993e-05 2024-04-14T15:20:21.062 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:20:21.062 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:20:21.062 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:20:21.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.063 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.48715.0:1 261.6s0 261:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e278)", 2024-04-14T15:20:21.063 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:15:45.860394+0000", 2024-04-14T15:20:21.063 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 275.12776485400002, 2024-04-14T15:20:21.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0127147990000001, 2024-04-14T15:20:21.063 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:20:21.063 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:20:21.063 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:20:21.063 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:20:21.063 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.48715", 2024-04-14T15:20:21.063 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1788001083", 2024-04-14T15:20:21.063 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1 2024-04-14T15:20:21.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.063 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:20:21.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:20:21.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:45.860394+0000", 2024-04-14T15:20:21.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:20:21.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:20:21.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:45.860394+0000", 2024-04-14T15:20:21.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:20:21.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:20:21.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:45.860396+0000", 2024-04-14T15:20:21.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2979999999999999e-06 2024-04-14T15:20:21.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:20:21.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:45.860404+0000", 2024-04-14T15:20:21.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1429999999999998e-06 2024-04-14T15:20:21.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:20:21.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:45.860419+0000", 2024-04-14T15:20:21.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5013e-05 2024-04-14T15:20:21.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:20:21.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:45.860431+0000", 2024-04-14T15:20:21.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2327000000000001e-05 2024-04-14T15:20:21.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:20:21.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:45.899182+0000", 2024-04-14T15:20:21.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.038750740999999998 2024-04-14T15:20:21.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:20:21.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:45.899187+0000", 2024-04-14T15:20:21.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.515e-06 2024-04-14T15:20:21.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:20:21.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:46.869238+0000", 2024-04-14T15:20:21.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.970051628 2024-04-14T15:20:21.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:20:21.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:46.869425+0000", 2024-04-14T15:20:21.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018649899999999999 2024-04-14T15:20:21.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2024-04-14T15:20:21.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:46.870375+0000", 2024-04-14T15:20:21.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00095067800000000005 2024-04-14T15:20:21.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2024-04-14T15:20:21.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:46.871231+0000", 2024-04-14T15:20:21.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00085567499999999997 2024-04-14T15:20:21.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:20:21.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:46.873042+0000", 2024-04-14T15:20:21.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001810571 2024-04-14T15:20:21.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:20:21.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:46.873108+0000", 2024-04-14T15:20:21.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6711000000000006e-05 2024-04-14T15:20:21.068 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:20:21.068 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:20:21.068 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:20:21.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.068 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25684.0:3739 272.8 272:10000000::::head [pgnls start_epoch 314] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e314)", 2024-04-14T15:20:21.068 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:16:21.890450+0000", 2024-04-14T15:20:21.068 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 239.09770870899999, 2024-04-14T15:20:21.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.044063261999999999, 2024-04-14T15:20:21.068 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:20:21.068 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:20:21.068 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-14T15:20:21.068 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:20:21.068 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25684", 2024-04-14T15:20:21.069 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1109245683", 2024-04-14T15:20:21.069 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3739 2024-04-14T15:20:21.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.069 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:20:21.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:20:21.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:16:21.890450+0000", 2024-04-14T15:20:21.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:20:21.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:20:21.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:16:21.890450+0000", 2024-04-14T15:20:21.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:20:21.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:20:21.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:16:21.890454+0000", 2024-04-14T15:20:21.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1239999999999998e-06 2024-04-14T15:20:21.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:20:21.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:16:21.890468+0000", 2024-04-14T15:20:21.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4154999999999999e-05 2024-04-14T15:20:21.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:20:21.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:16:21.890486+0000", 2024-04-14T15:20:21.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7601e-05 2024-04-14T15:20:21.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:20:21.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:16:21.890496+0000", 2024-04-14T15:20:21.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0723999999999999e-05 2024-04-14T15:20:21.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:20:21.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:16:21.890565+0000", 2024-04-14T15:20:21.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8442000000000006e-05 2024-04-14T15:20:21.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2024-04-14T15:20:21.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:16:21.890570+0000", 2024-04-14T15:20:21.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.972e-06 2024-04-14T15:20:21.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:20:21.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:16:21.934313+0000", 2024-04-14T15:20:21.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.043743734999999999 2024-04-14T15:20:21.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:20:21.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:16:21.934353+0000", 2024-04-14T15:20:21.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9403999999999997e-05 2024-04-14T15:20:21.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:20:21.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:16:21.934513+0000", 2024-04-14T15:20:21.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016010500000000001 2024-04-14T15:20:21.072 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:20:21.073 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:20:21.073 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:20:21.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.073 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.37104.0:2 282.13 e353/352 282:cbb9903e:::test_obj:head v 353'1, mlcod=0'0)", 2024-04-14T15:20:21.073 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:17:00.914225+0000", 2024-04-14T15:20:21.073 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 200.073933732, 2024-04-14T15:20:21.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.025144527999999999, 2024-04-14T15:20:21.073 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:20:21.073 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:20:21.073 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:20:21.073 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:20:21.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:20:21.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:00.914225+0000", 2024-04-14T15:20:21.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:20:21.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.074 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:20:21.074 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:00.914225+0000", 2024-04-14T15:20:21.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:20:21.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.074 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:20:21.074 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:00.914225+0000", 2024-04-14T15:20:21.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9200000000000002e-07 2024-04-14T15:20:21.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.074 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:20:21.074 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:00.914235+0000", 2024-04-14T15:20:21.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0054999999999999e-05 2024-04-14T15:20:21.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:20:21.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:00.914241+0000", 2024-04-14T15:20:21.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5509999999999999e-06 2024-04-14T15:20:21.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:20:21.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:00.914249+0000", 2024-04-14T15:20:21.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4780000000000008e-06 2024-04-14T15:20:21.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:20:21.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:00.914393+0000", 2024-04-14T15:20:21.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014401999999999999 2024-04-14T15:20:21.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:20:21.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:00.914410+0000", 2024-04-14T15:20:21.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6631999999999999e-05 2024-04-14T15:20:21.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:20:21.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:00.938022+0000", 2024-04-14T15:20:21.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.023612016999999999 2024-04-14T15:20:21.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:20:21.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:00.939369+0000", 2024-04-14T15:20:21.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001347483 2024-04-14T15:20:21.076 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:20:21.077 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:20:21.077 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:20:21.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.077 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.24373.0:2579 4.f e479/66 4:f2c84a15:::obj_delete_at_hint.0000000126:head v 479'33, mlcod=479'31)", 2024-04-14T15:20:21.077 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:19:50.734794+0000", 2024-04-14T15:20:21.077 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 30.25336458, 2024-04-14T15:20:21.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.055023135000000001, 2024-04-14T15:20:21.077 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:20:21.077 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:20:21.077 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:20:21.077 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:20:21.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:20:21.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:19:50.734794+0000", 2024-04-14T15:20:21.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:20:21.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:20:21.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:19:50.734794+0000", 2024-04-14T15:20:21.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:20:21.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:20:21.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:19:50.734795+0000", 2024-04-14T15:20:21.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5499999999999997e-07 2024-04-14T15:20:21.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:20:21.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:19:50.734800+0000", 2024-04-14T15:20:21.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9099999999999996e-06 2024-04-14T15:20:21.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:20:21.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:19:50.734806+0000", 2024-04-14T15:20:21.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6579999999999996e-06 2024-04-14T15:20:21.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:20:21.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:19:50.734811+0000", 2024-04-14T15:20:21.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5669999999999996e-06 2024-04-14T15:20:21.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:20:21.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:19:50.734846+0000", 2024-04-14T15:20:21.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4714e-05 2024-04-14T15:20:21.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:20:21.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:19:50.734858+0000", 2024-04-14T15:20:21.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.224e-05 2024-04-14T15:20:21.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:20:21.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:19:50.735075+0000", 2024-04-14T15:20:21.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021727199999999999 2024-04-14T15:20:21.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:20:21.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:19:50.789817+0000", 2024-04-14T15:20:21.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.054741919 2024-04-14T15:20:21.081 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:20:21.081 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:20:21.081 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:20:21.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.081 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25684.0:7352 305.1b 305:da40d66a:test-rados-api-smithi090-93652-80::foo:head [writefull 0~10] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e492)", 2024-04-14T15:20:21.081 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:20:02.944272+0000", 2024-04-14T15:20:21.081 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.043886043000001, 2024-04-14T15:20:21.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19261669200000001, 2024-04-14T15:20:21.081 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:20:21.081 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:20:21.081 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:20:21.081 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:20:21.081 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25684", 2024-04-14T15:20:21.082 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1109245683", 2024-04-14T15:20:21.082 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 7352 2024-04-14T15:20:21.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.082 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:20:21.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:20:21.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:20:02.944272+0000", 2024-04-14T15:20:21.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:20:21.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:20:21.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:20:02.944272+0000", 2024-04-14T15:20:21.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:20:21.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:20:21.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:20:02.944275+0000", 2024-04-14T15:20:21.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1059999999999998e-06 2024-04-14T15:20:21.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:20:21.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:20:02.944282+0000", 2024-04-14T15:20:21.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8120000000000008e-06 2024-04-14T15:20:21.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:20:21.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:20:02.944294+0000", 2024-04-14T15:20:21.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1583000000000001e-05 2024-04-14T15:20:21.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:20:21.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:20:02.944305+0000", 2024-04-14T15:20:21.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1144999999999999e-05 2024-04-14T15:20:21.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:20:21.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:20:02.944359+0000", 2024-04-14T15:20:21.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4249999999999997e-05 2024-04-14T15:20:21.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:20:21.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:20:02.944368+0000", 2024-04-14T15:20:21.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2019999999999993e-06 2024-04-14T15:20:21.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.085 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:20:21.085 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:20:03.134140+0000", 2024-04-14T15:20:21.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.189771103 2024-04-14T15:20:21.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.085 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:20:21.085 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:20:03.134292+0000", 2024-04-14T15:20:21.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015226399999999999 2024-04-14T15:20:21.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.085 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 5,6", 2024-04-14T15:20:21.085 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:20:03.134549+0000", 2024-04-14T15:20:21.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025749699999999998 2024-04-14T15:20:21.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:20:21.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:20:03.135751+0000", 2024-04-14T15:20:21.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0012017530000000001 2024-04-14T15:20:21.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:20:21.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:20:03.136684+0000", 2024-04-14T15:20:21.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00093322700000000001 2024-04-14T15:20:21.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:20:21.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:20:03.136827+0000", 2024-04-14T15:20:21.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014281600000000001 2024-04-14T15:20:21.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:20:21.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:20:03.136860+0000", 2024-04-14T15:20:21.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3229999999999999e-05 2024-04-14T15:20:21.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:20:21.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:20:21.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:20:21.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:20:03.136889+0000", 2024-04-14T15:20:21.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8704000000000002e-05 2024-04-14T15:20:21.087 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:20:21.087 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:20:21.087 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:20:21.087 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:20:21.087 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:20:21.087 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-14T15:20:21.088 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:20:21.114 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:21.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:21.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:20:21.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:20 smithi090 ceph-mon[41670]: pgmap v720: 258 pgs: 258 active+clean; 8.4 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-14T15:20:21.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:20 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1193298934' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:20:21.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:20 smithi090 ceph-mon[30399]: pgmap v720: 258 pgs: 258 active+clean; 8.4 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-14T15:20:21.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:20 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1193298934' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:20:21.216 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:20:21.241 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:21.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:21.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:20:21.343 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:20:21.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:21.369 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:21.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:20:21.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:20 smithi092 ceph-mon[33966]: pgmap v720: 258 pgs: 258 active+clean; 8.4 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-14T15:20:21.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:20 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1193298934' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:20:21.470 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:20:21.495 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:21.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:21.496 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:20:21.596 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:21.624 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:21.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:21.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:21.726 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:20:21.751 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:21.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:21.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:20:21.853 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:20:21.878 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:21.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:21.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:20:21.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:21.939+0000 7fb0c1ffb700 1 -- 172.21.15.90:0/2640704664 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v499) ==== 97+0+0 (unknown 2831217725 0 0) 0x7fb0bc05a450 con 0x7fb0c410e220 2024-04-14T15:20:21.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:21.943+0000 7fb0caec8700 10 librados: watch_flush enter 2024-04-14T15:20:21.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:21.943+0000 7fb0caec8700 10 librados: watch_flush exit 2024-04-14T15:20:21.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:21.943+0000 7fb0caec8700 20 client.37473.objecter shutdown clearing up homeless session... 2024-04-14T15:20:21.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:21.943+0000 7fb0caec8700 10 client.37473.objecter successfully canceled tick 2024-04-14T15:20:21.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:21.943+0000 7fb0caec8700 1 -- 172.21.15.90:0/2640704664 >> v1:172.21.15.90:6800/420935427 conn(0x7fb0b4071290 legacy=0x7fb0b4073750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:21.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:21.943+0000 7fb0caec8700 1 -- 172.21.15.90:0/2640704664 >> v1:172.21.15.90:6789/0 conn(0x7fb0c410e220 legacy=0x7fb0c41a8b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:21.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:21.943+0000 7fb0caec8700 1 -- 172.21.15.90:0/2640704664 shutdown_connections 2024-04-14T15:20:21.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:21.943+0000 7fb0caec8700 1 -- 172.21.15.90:0/2640704664 >> 172.21.15.90:0/2640704664 conn(0x7fb0c40fd2b0 msgr2=0x7fb0c4111000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:20:21.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:21.943+0000 7fb0caec8700 1 -- 172.21.15.90:0/2640704664 shutdown_connections 2024-04-14T15:20:21.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:21.943+0000 7fb0caec8700 1 -- 172.21.15.90:0/2640704664 wait complete. 2024-04-14T15:20:21.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:21.943+0000 7fb0caec8700 1 librados: shutdown 2024-04-14T15:20:21.948 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is set 2024-04-14T15:20:21.981 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:20:22.005 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:22.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:22.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:20:22.107 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:20:22.131 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:22.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:22.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:20:22.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:21 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2640704664' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:20:22.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:21 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2640704664' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:20:22.233 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:20:22.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:22.260 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:22.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:20:22.272 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:21 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2640704664' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:20:22.361 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:20:22.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:22.387 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:22.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:20:22.488 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:20:22.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:22.514 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:22.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:20:22.615 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:20:22.640 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:22.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:22.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:20:22.741 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:20:22.766 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:22.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:22.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:20:22.867 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:20:22.892 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:22.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:22.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:20:22.994 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:20:23.020 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:23.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:23.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:20:23.122 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:23.148 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:23.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:23.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:23.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:22 smithi090 ceph-mon[41670]: pgmap v721: 258 pgs: 258 active+clean; 8.4 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:20:23.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:22 smithi090 ceph-mon[41670]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:20:23.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:22 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2640704664' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-14T15:20:23.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:22 smithi090 ceph-mon[41670]: osdmap e499: 8 total, 8 up, 8 in 2024-04-14T15:20:23.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:22 smithi090 ceph-mon[30399]: pgmap v721: 258 pgs: 258 active+clean; 8.4 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:20:23.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:22 smithi090 ceph-mon[30399]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:20:23.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:22 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2640704664' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-14T15:20:23.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:22 smithi090 ceph-mon[30399]: osdmap e499: 8 total, 8 up, 8 in 2024-04-14T15:20:23.250 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:20:23.276 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:23.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:23.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:20:23.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:22 smithi092 ceph-mon[33966]: pgmap v721: 258 pgs: 258 active+clean; 8.4 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:20:23.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:22 smithi092 ceph-mon[33966]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:20:23.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:22 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2640704664' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-14T15:20:23.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:22 smithi092 ceph-mon[33966]: osdmap e499: 8 total, 8 up, 8 in 2024-04-14T15:20:23.377 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:20:23.401 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:23.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:23.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:20:23.505 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:20:23.530 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:23.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:23.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:20:23.632 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:20:23.659 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:23.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:23.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:20:23.761 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:20:23.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:23.788 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:23.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:20:23.889 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:20:23.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:23.915 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:23.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:20:23.958 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-14T15:20:24.015 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:20:24.042 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:24.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:24.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:20:24.144 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:20:24.168 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:24.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:24.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:20:24.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:23 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:20:24.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:23 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:20:24.192 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:20:23 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:20:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:20:24.269 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:20:24.293 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:24.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:24.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:20:24.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:23 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:20:24.394 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:20:24.422 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:24.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:24.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:20:24.525 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:20:24.554 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:24.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:24.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:20:24.603 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-14T15:20:24.656 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:20:24.681 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:24.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:24.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:20:24.782 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:20:24.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:24.806 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:24.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:20:24.848 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:20:24.907 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:20:24.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:24.934 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:24.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:20:25.035 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:20:25.059 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:25.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:25.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:20:25.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:25.151+0000 7f674a5d7700 1 -- 172.21.15.90:0/664977173 >> v1:172.21.15.90:6789/0 conn(0x7f674410e360 legacy=0x7f6744110820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:25.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:25.151+0000 7f674a5d7700 1 -- 172.21.15.90:0/664977173 shutdown_connections 2024-04-14T15:20:25.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:25.151+0000 7f674a5d7700 1 -- 172.21.15.90:0/664977173 >> 172.21.15.90:0/664977173 conn(0x7f67440fd2c0 msgr2=0x7f67440ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:20:25.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:25.151+0000 7f674a5d7700 1 -- 172.21.15.90:0/664977173 shutdown_connections 2024-04-14T15:20:25.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:25.151+0000 7f674a5d7700 1 -- 172.21.15.90:0/664977173 wait complete. 2024-04-14T15:20:25.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:25.152+0000 7f674a5d7700 1 librados: starting msgr at 2024-04-14T15:20:25.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:25.152+0000 7f674a5d7700 1 librados: starting objecter 2024-04-14T15:20:25.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:25.152+0000 7f674a5d7700 1 Processor -- start 2024-04-14T15:20:25.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:25.152+0000 7f674a5d7700 1 -- start start 2024-04-14T15:20:25.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:25.152+0000 7f674a5d7700 1 librados: setting wanted keys 2024-04-14T15:20:25.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:25.152+0000 7f674a5d7700 1 librados: calling monclient init 2024-04-14T15:20:25.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:25.153+0000 7f674a5d7700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f67441a9d60 con 0x7f67441078e0 2024-04-14T15:20:25.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:25.153+0000 7f674a5d7700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f67441aaf40 con 0x7f674410a790 2024-04-14T15:20:25.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:25.153+0000 7f674a5d7700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f67441ac180 con 0x7f674410e360 2024-04-14T15:20:25.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:25.153+0000 7f6743fff700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f67441078e0 0x7f674410da90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:38330/0 (socket says 172.21.15.90:38330) 2024-04-14T15:20:25.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:25.153+0000 7f6743fff700 1 -- 172.21.15.90:0/2746590224 learned_addr learned my addr 172.21.15.90:0/2746590224 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:20:25.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:25.153+0000 7f67417fa700 10 client.?.objecter ms_handle_connect 0x7f674410a790 2024-04-14T15:20:25.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:25.153+0000 7f67417fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:20:25.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:25.153+0000 7f67417fa700 10 client.?.objecter ms_handle_connect 0x7f67441078e0 2024-04-14T15:20:25.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:25.153+0000 7f67417fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:20:25.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:25.153+0000 7f67417fa700 10 client.?.objecter ms_handle_connect 0x7f674410e360 2024-04-14T15:20:25.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:25.153+0000 7f67417fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:20:25.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:25.153+0000 7f67417fa700 1 -- 172.21.15.90:0/2746590224 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2129739148 0 0) 0x7f67441aaf40 con 0x7f674410a790 2024-04-14T15:20:25.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:25.153+0000 7f67417fa700 1 -- 172.21.15.90:0/2746590224 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f672c0031f0 con 0x7f674410a790 2024-04-14T15:20:25.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:25.153+0000 7f67417fa700 1 -- 172.21.15.90:0/2746590224 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1205048006 0 0) 0x7f67441ac180 con 0x7f674410e360 2024-04-14T15:20:25.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:25.153+0000 7f67417fa700 1 -- 172.21.15.90:0/2746590224 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f67441aaf40 con 0x7f674410e360 2024-04-14T15:20:25.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:25.153+0000 7f67417fa700 1 -- 172.21.15.90:0/2746590224 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 151162243 0 0) 0x7f67441a9d60 con 0x7f67441078e0 2024-04-14T15:20:25.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:25.153+0000 7f67417fa700 1 -- 172.21.15.90:0/2746590224 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f67441ac180 con 0x7f67441078e0 2024-04-14T15:20:25.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:25.153+0000 7f67417fa700 1 -- 172.21.15.90:0/2746590224 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4031951453 0 0) 0x7f672c0031f0 con 0x7f674410a790 2024-04-14T15:20:25.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:25.153+0000 7f67417fa700 1 -- 172.21.15.90:0/2746590224 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f67441a9d60 con 0x7f674410a790 2024-04-14T15:20:25.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:25.153+0000 7f67417fa700 1 -- 172.21.15.90:0/2746590224 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f67380018b0 con 0x7f674410a790 2024-04-14T15:20:25.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:25.153+0000 7f67417fa700 1 -- 172.21.15.90:0/2746590224 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2280969636 0 0) 0x7f67441aaf40 con 0x7f674410e360 2024-04-14T15:20:25.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:25.153+0000 7f67417fa700 1 -- 172.21.15.90:0/2746590224 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f672c0031f0 con 0x7f674410e360 2024-04-14T15:20:25.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:25.153+0000 7f67417fa700 1 -- 172.21.15.90:0/2746590224 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3445949610 0 0) 0x7f67441a9d60 con 0x7f674410a790 2024-04-14T15:20:25.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:25.153+0000 7f67417fa700 1 -- 172.21.15.90:0/2746590224 >> v1:172.21.15.90:6790/0 conn(0x7f674410e360 legacy=0x7f67441a83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:25.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:25.153+0000 7f67417fa700 1 -- 172.21.15.90:0/2746590224 >> v1:172.21.15.90:6789/0 conn(0x7f67441078e0 legacy=0x7f674410da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:25.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:25.153+0000 7f67417fa700 1 -- 172.21.15.90:0/2746590224 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67441ad3c0 con 0x7f674410a790 2024-04-14T15:20:25.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:25.153+0000 7f674a5d7700 1 -- 172.21.15.90:0/2746590224 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f67441a9f90 con 0x7f674410a790 2024-04-14T15:20:25.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:25.154+0000 7f674a5d7700 10 client.49312.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:20:25.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:25.154+0000 7f674a5d7700 1 -- 172.21.15.90:0/2746590224 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f67441aa450 con 0x7f674410a790 2024-04-14T15:20:25.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:25.154+0000 7f674a5d7700 1 librados: init done 2024-04-14T15:20:25.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:25.154+0000 7f67417fa700 1 -- 172.21.15.90:0/2746590224 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f6738001c60 con 0x7f674410a790 2024-04-14T15:20:25.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:25.154+0000 7f67417fa700 1 -- 172.21.15.90:0/2746590224 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f6738004bf0 con 0x7f674410a790 2024-04-14T15:20:25.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:25.155+0000 7f674a5d7700 1 -- 172.21.15.90:0/2746590224 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6730005150 con 0x7f674410a790 2024-04-14T15:20:25.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:25.155+0000 7f67417fa700 1 -- 172.21.15.90:0/2746590224 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f6738004d90 con 0x7f674410a790 2024-04-14T15:20:25.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:25.156+0000 7f67417fa700 1 -- 172.21.15.90:0/2746590224 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(500..500 src has 51..500) ==== 8197+0+0 (unknown 2863954077 0 0) 0x7f6738089840 con 0x7f674410a790 2024-04-14T15:20:25.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:25.156+0000 7f67417fa700 10 client.49312.objecter ms_dispatch 0x7f6744001550 osd_map(500..500 src has 51..500) 2024-04-14T15:20:25.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:25.156+0000 7f67417fa700 3 client.49312.objecter handle_osd_map got epochs [500,500] > 0 2024-04-14T15:20:25.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:25.156+0000 7f67417fa700 3 client.49312.objecter handle_osd_map decoding full epoch 500 2024-04-14T15:20:25.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:25.156+0000 7f67417fa700 20 client.49312.objecter dump_active .. 0 homeless 2024-04-14T15:20:25.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:25.156+0000 7f67417fa700 10 client.49312.objecter ms_handle_connect 0x7f672c0711a0 2024-04-14T15:20:25.160 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:20:25.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:25.160+0000 7f67417fa700 1 -- 172.21.15.90:0/2746590224 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f6738056230 con 0x7f674410a790 2024-04-14T15:20:25.161 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:24 smithi090 ceph-mon[41670]: pgmap v723: 258 pgs: 258 active+clean; 8.4 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-14T15:20:25.161 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:24 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:20:25.161 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:24 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:20:25.162 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:24 smithi090 ceph-mon[30399]: pgmap v723: 258 pgs: 258 active+clean; 8.4 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-14T15:20:25.162 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:24 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:20:25.162 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:24 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:20:25.162 INFO:journalctl@ceph.osd.3.smithi090.stdout:Apr 14 15:20:24 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-3[64642]: 2024-04-14T15:20:24.988+0000 7f57ca508700 -1 snap_mapper.add_oid found existing snaps mapped on 305:da40d66a:test-rados-api-smithi090-93652-80::foo:2, removing 2024-04-14T15:20:25.187 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:25.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:25.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:20:25.193 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:20:25.198 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:24 smithi092 ceph-mon[33966]: pgmap v723: 258 pgs: 258 active+clean; 8.4 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-14T15:20:25.198 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:24 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:20:25.198 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:24 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:20:25.199 INFO:journalctl@ceph.osd.5.smithi092.stdout:Apr 14 15:20:24 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-5[45851]: 2024-04-14T15:20:24.989+0000 7f20d8fc4700 -1 snap_mapper.add_oid found existing snaps mapped on 305:da40d66a:test-rados-api-smithi090-93652-80::foo:2, removing 2024-04-14T15:20:25.199 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:20:24 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6[52258]: 2024-04-14T15:20:24.989+0000 7fb5c28b8700 -1 snap_mapper.add_oid found existing snaps mapped on 305:da40d66a:test-rados-api-smithi090-93652-80::foo:2, removing 2024-04-14T15:20:25.289 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:20:25.314 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:25.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:25.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:20:25.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:25.376+0000 7f674a5d7700 1 -- 172.21.15.90:0/2746590224 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} v 0) -- 0x7f6730005440 con 0x7f674410a790 2024-04-14T15:20:25.416 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:20:25.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:25.440 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:25.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:20:25.541 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:20:25.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:25.566 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:25.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:20:25.622 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:20:25 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:20:25.646 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd unset noscrub 2024-04-14T15:20:25.667 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:20:25.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:25.713 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:25.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:20:25.814 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:25.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:25.838 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:25.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:25.922 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:20:25.938 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:20:25.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:25.959+0000 7f67417fa700 1 -- 172.21.15.90:0/2746590224 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]=0 set pool 1 pg_num to 21 v501) ==== 129+0+0 (unknown 2369143290 0 0) 0x7f6738059ee0 con 0x7f674410a790 2024-04-14T15:20:25.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:25.963+0000 7f674a5d7700 10 librados: watch_flush enter 2024-04-14T15:20:25.964 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:25.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:25.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:20:25.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:25.963+0000 7f674a5d7700 10 librados: watch_flush exit 2024-04-14T15:20:25.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:25.963+0000 7f674a5d7700 20 client.49312.objecter shutdown clearing up homeless session... 2024-04-14T15:20:25.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:25.963+0000 7f674a5d7700 10 client.49312.objecter successfully canceled tick 2024-04-14T15:20:25.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:25.963+0000 7f674a5d7700 1 -- 172.21.15.90:0/2746590224 >> v1:172.21.15.90:6800/420935427 conn(0x7f672c0711a0 legacy=0x7f672c073660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:25.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:25.963+0000 7f674a5d7700 1 -- 172.21.15.90:0/2746590224 >> v1:172.21.15.92:6789/0 conn(0x7f674410a790 legacy=0x7f67441a4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:25.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:25.970+0000 7f674a5d7700 1 -- 172.21.15.90:0/2746590224 shutdown_connections 2024-04-14T15:20:25.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:25.970+0000 7f674a5d7700 1 -- 172.21.15.90:0/2746590224 >> 172.21.15.90:0/2746590224 conn(0x7f67440fd2c0 msgr2=0x7f67441003d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:20:25.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:25.971+0000 7f674a5d7700 1 -- 172.21.15.90:0/2746590224 shutdown_connections 2024-04-14T15:20:25.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:25.971+0000 7f674a5d7700 1 -- 172.21.15.90:0/2746590224 wait complete. 2024-04-14T15:20:25.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:25.971+0000 7f674a5d7700 1 librados: shutdown 2024-04-14T15:20:25.973 INFO:teuthology.orchestra.run.smithi090.stderr:set pool 1 pg_num to 21 2024-04-14T15:20:26.066 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:26.091 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:26.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:26.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:26.193 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:20:26.217 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:26.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:26.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:20:26.288 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:25 smithi092 ceph-mon[33966]: osdmap e500: 8 total, 8 up, 8 in 2024-04-14T15:20:26.288 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:25 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:20:26.288 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:25 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:20:26.288 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:25 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-79"}]: dispatch 2024-04-14T15:20:26.288 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:25 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-79"}]: dispatch 2024-04-14T15:20:26.288 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:25 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2746590224' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-04-14T15:20:26.288 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:25 smithi092 ceph-mon[33966]: from='client.49312 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-04-14T15:20:26.318 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:26.342 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:26.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:26.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:26.435 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:25 smithi090 ceph-mon[41670]: osdmap e500: 8 total, 8 up, 8 in 2024-04-14T15:20:26.435 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:25 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:20:26.435 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:25 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:20:26.435 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:25 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-79"}]: dispatch 2024-04-14T15:20:26.435 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:25 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-79"}]: dispatch 2024-04-14T15:20:26.435 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:25 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2746590224' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-04-14T15:20:26.435 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:25 smithi090 ceph-mon[41670]: from='client.49312 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-04-14T15:20:26.436 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:25 smithi090 ceph-mon[30399]: osdmap e500: 8 total, 8 up, 8 in 2024-04-14T15:20:26.436 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:25 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:20:26.436 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:25 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:20:26.436 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:25 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-79"}]: dispatch 2024-04-14T15:20:26.436 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:25 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-79"}]: dispatch 2024-04-14T15:20:26.436 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:25 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2746590224' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-04-14T15:20:26.436 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:25 smithi090 ceph-mon[30399]: from='client.49312 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-04-14T15:20:26.443 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:20:26.467 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:26.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:26.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:20:26.536 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-04-14T15:20:26.536 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-04-14T15:20:26.536 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 0 2024-04-14T15:20:26.536 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-04-14T15:20:26.569 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:20:26.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:26.595 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:26.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:20:26.696 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:20:26.719 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:26.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:26.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:20:26.821 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:20:26.844 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:26.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:26.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:20:26.946 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:20:26.972 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:26.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:26.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:20:27.074 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:20:27.098 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:27.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:27.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:20:27.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:26 smithi090 ceph-mon[41670]: pgmap v725: 258 pgs: 258 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:20:27.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:26 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:20:27.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:26 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:20:27.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:26 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:20:27.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:26 smithi090 ceph-mon[41670]: from='client.49312 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-04-14T15:20:27.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:26 smithi090 ceph-mon[41670]: osdmap e501: 8 total, 8 up, 8 in 2024-04-14T15:20:27.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:26 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:20:27.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:26 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:20:27.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:26 smithi090 ceph-mon[30399]: pgmap v725: 258 pgs: 258 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:20:27.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:26 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:20:27.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:26 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:20:27.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:26 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:20:27.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:26 smithi090 ceph-mon[30399]: from='client.49312 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-04-14T15:20:27.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:26 smithi090 ceph-mon[30399]: osdmap e501: 8 total, 8 up, 8 in 2024-04-14T15:20:27.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:26 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:20:27.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:26 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:20:27.200 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:20:27.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:27.225 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:27.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:20:27.245 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:26 smithi092 ceph-mon[33966]: pgmap v725: 258 pgs: 258 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:20:27.246 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:26 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:20:27.246 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:26 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:20:27.246 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:26 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:20:27.246 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:26 smithi092 ceph-mon[33966]: from='client.49312 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-04-14T15:20:27.246 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:26 smithi092 ceph-mon[33966]: osdmap e501: 8 total, 8 up, 8 in 2024-04-14T15:20:27.246 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:26 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:20:27.246 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:26 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:20:27.271 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-14T15:20:27.326 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:20:27.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.344+0000 7f726bfd8700 1 -- 172.21.15.90:0/4132967053 >> v1:172.21.15.92:6789/0 conn(0x7f726410f340 legacy=0x7f7264111800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:27.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.344+0000 7f726bfd8700 1 -- 172.21.15.90:0/4132967053 shutdown_connections 2024-04-14T15:20:27.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.344+0000 7f726bfd8700 1 -- 172.21.15.90:0/4132967053 >> 172.21.15.90:0/4132967053 conn(0x7f72640fe3d0 msgr2=0x7f72641007f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:20:27.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.344+0000 7f726bfd8700 1 -- 172.21.15.90:0/4132967053 shutdown_connections 2024-04-14T15:20:27.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.344+0000 7f726bfd8700 1 -- 172.21.15.90:0/4132967053 wait complete. 2024-04-14T15:20:27.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.345+0000 7f726bfd8700 1 librados: starting msgr at 2024-04-14T15:20:27.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.345+0000 7f726bfd8700 1 librados: starting objecter 2024-04-14T15:20:27.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.345+0000 7f726bfd8700 1 Processor -- start 2024-04-14T15:20:27.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.345+0000 7f726bfd8700 1 -- start start 2024-04-14T15:20:27.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.345+0000 7f726bfd8700 1 librados: setting wanted keys 2024-04-14T15:20:27.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.345+0000 7f726bfd8700 1 librados: calling monclient init 2024-04-14T15:20:27.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.345+0000 7f726bfd8700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f72641a9bc0 con 0x7f726410b770 2024-04-14T15:20:27.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.345+0000 7f726bfd8700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f72641aade0 con 0x7f72641088c0 2024-04-14T15:20:27.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.345+0000 7f726bfd8700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f72641ac000 con 0x7f726410f340 2024-04-14T15:20:27.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.345+0000 7f7269573700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f726410b770 0x7f72641a4be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:38350/0 (socket says 172.21.15.90:38350) 2024-04-14T15:20:27.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.345+0000 7f7269573700 1 -- 172.21.15.90:0/937819003 learned_addr learned my addr 172.21.15.90:0/937819003 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:20:27.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.345+0000 7f725affd700 10 client.?.objecter ms_handle_connect 0x7f72641088c0 2024-04-14T15:20:27.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.345+0000 7f725affd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:20:27.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.345+0000 7f725affd700 10 client.?.objecter ms_handle_connect 0x7f726410f340 2024-04-14T15:20:27.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.345+0000 7f725affd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:20:27.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.345+0000 7f725affd700 10 client.?.objecter ms_handle_connect 0x7f726410b770 2024-04-14T15:20:27.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.345+0000 7f725affd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:20:27.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.345+0000 7f725affd700 1 -- 172.21.15.90:0/937819003 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3185733163 0 0) 0x7f72641aade0 con 0x7f72641088c0 2024-04-14T15:20:27.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.345+0000 7f725affd700 1 -- 172.21.15.90:0/937819003 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f72500031f0 con 0x7f72641088c0 2024-04-14T15:20:27.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.345+0000 7f725affd700 1 -- 172.21.15.90:0/937819003 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3138626142 0 0) 0x7f72641ac000 con 0x7f726410f340 2024-04-14T15:20:27.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.345+0000 7f725affd700 1 -- 172.21.15.90:0/937819003 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f72641aade0 con 0x7f726410f340 2024-04-14T15:20:27.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.345+0000 7f725affd700 1 -- 172.21.15.90:0/937819003 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 451835611 0 0) 0x7f72641a9bc0 con 0x7f726410b770 2024-04-14T15:20:27.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.346+0000 7f725affd700 1 -- 172.21.15.90:0/937819003 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f72641ac000 con 0x7f726410b770 2024-04-14T15:20:27.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.346+0000 7f725affd700 1 -- 172.21.15.90:0/937819003 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3669411961 0 0) 0x7f72500031f0 con 0x7f72641088c0 2024-04-14T15:20:27.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.346+0000 7f725affd700 1 -- 172.21.15.90:0/937819003 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f72641a9bc0 con 0x7f72641088c0 2024-04-14T15:20:27.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.346+0000 7f725affd700 1 -- 172.21.15.90:0/937819003 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f7260001870 con 0x7f72641088c0 2024-04-14T15:20:27.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.346+0000 7f725affd700 1 -- 172.21.15.90:0/937819003 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 991861952 0 0) 0x7f72641aade0 con 0x7f726410f340 2024-04-14T15:20:27.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.346+0000 7f725affd700 1 -- 172.21.15.90:0/937819003 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f72500031f0 con 0x7f726410f340 2024-04-14T15:20:27.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.346+0000 7f725affd700 1 -- 172.21.15.90:0/937819003 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f725c0048c0 con 0x7f726410f340 2024-04-14T15:20:27.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.346+0000 7f725affd700 1 -- 172.21.15.90:0/937819003 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2623432322 0 0) 0x7f72641a9bc0 con 0x7f72641088c0 2024-04-14T15:20:27.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.346+0000 7f725affd700 1 -- 172.21.15.90:0/937819003 >> v1:172.21.15.90:6790/0 conn(0x7f726410f340 legacy=0x7f72641a8310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:27.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.346+0000 7f725affd700 1 -- 172.21.15.90:0/937819003 >> v1:172.21.15.90:6789/0 conn(0x7f726410b770 legacy=0x7f72641a4be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:27.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.346+0000 7f725affd700 1 -- 172.21.15.90:0/937819003 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72641ad240 con 0x7f72641088c0 2024-04-14T15:20:27.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.346+0000 7f726bfd8700 1 -- 172.21.15.90:0/937819003 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f72641ac230 con 0x7f72641088c0 2024-04-14T15:20:27.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.346+0000 7f726bfd8700 10 client.49318.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:20:27.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.346+0000 7f726bfd8700 1 -- 172.21.15.90:0/937819003 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f72641ac6f0 con 0x7f72641088c0 2024-04-14T15:20:27.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.346+0000 7f726bfd8700 1 librados: init done 2024-04-14T15:20:27.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.346+0000 7f725affd700 1 -- 172.21.15.90:0/937819003 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f7260001c80 con 0x7f72641088c0 2024-04-14T15:20:27.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.346+0000 7f725affd700 1 -- 172.21.15.90:0/937819003 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f7260004af0 con 0x7f72641088c0 2024-04-14T15:20:27.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.347+0000 7f726bfd8700 1 -- 172.21.15.90:0/937819003 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f7248005130 con 0x7f72641088c0 2024-04-14T15:20:27.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.347+0000 7f725affd700 1 -- 172.21.15.90:0/937819003 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f726001b6a0 con 0x7f72641088c0 2024-04-14T15:20:27.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.348+0000 7f725affd700 1 -- 172.21.15.90:0/937819003 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(502..502 src has 51..502) ==== 8184+0+0 (unknown 1540555052 0 0) 0x7f7260089880 con 0x7f72641088c0 2024-04-14T15:20:27.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.348+0000 7f725affd700 10 client.49318.objecter ms_dispatch 0x7f7264001550 osd_map(502..502 src has 51..502) 2024-04-14T15:20:27.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.348+0000 7f725affd700 3 client.49318.objecter handle_osd_map got epochs [502,502] > 0 2024-04-14T15:20:27.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.348+0000 7f725affd700 3 client.49318.objecter handle_osd_map decoding full epoch 502 2024-04-14T15:20:27.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.348+0000 7f725affd700 20 client.49318.objecter dump_active .. 0 homeless 2024-04-14T15:20:27.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.348+0000 7f725affd700 10 client.49318.objecter ms_handle_connect 0x7f7250071390 2024-04-14T15:20:27.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.348+0000 7f725affd700 1 -- 172.21.15.90:0/937819003 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v502) ==== 54+0+16 (unknown 2888788756 0 2420000192) 0x7f7260089bf0 con 0x7f72641088c0 2024-04-14T15:20:27.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.348+0000 7f726bfd8700 10 client.49318.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:20:27.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.348+0000 7f726bfd8700 15 client.49318.objecter _session_command_op_assign -1 1 2024-04-14T15:20:27.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.348+0000 7f726bfd8700 20 client.49318.objecter _get_session s=0x7f7248005410 osd=0 3 2024-04-14T15:20:27.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.348+0000 7f726bfd8700 20 client.49318.objecter put_session s=0x7f7248005410 osd=0 3 2024-04-14T15:20:27.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.348+0000 7f726bfd8700 20 client.49318.objecter _get_session s=0x7f7248005410 osd=0 3 2024-04-14T15:20:27.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.348+0000 7f726bfd8700 15 client.49318.objecter _session_command_op_remove -1 1 2024-04-14T15:20:27.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.348+0000 7f726bfd8700 20 client.49318.objecter get_session s=0x7f7248005410 osd=0 3 2024-04-14T15:20:27.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.348+0000 7f726bfd8700 15 client.49318.objecter _session_command_op_assign 0 1 2024-04-14T15:20:27.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.348+0000 7f726bfd8700 20 client.49318.objecter put_session s=0x7f7248005410 osd=0 4 2024-04-14T15:20:27.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.348+0000 7f726bfd8700 10 client.49318.objecter _send_command 1 2024-04-14T15:20:27.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.348+0000 7f726bfd8700 1 -- 172.21.15.90:0/937819003 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7248004ba0 con 0x7f7248004760 2024-04-14T15:20:27.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.349+0000 7f725affd700 10 client.49318.objecter ms_handle_connect 0x7f7248004760 2024-04-14T15:20:27.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.349+0000 7f725affd700 1 -- 172.21.15.90:0/937819003 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f7248004ba0 con 0x7f7248004760 2024-04-14T15:20:27.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.349+0000 7f725affd700 10 client.49318.objecter ms_dispatch 0x7f7264001550 command_reply(tid 1: 0 ) 2024-04-14T15:20:27.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.349+0000 7f725affd700 10 client.49318.objecter _finish_command 1 = system:0 2024-04-14T15:20:27.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.349+0000 7f725affd700 20 client.49318.objecter put_session s=0x7f7248005410 osd=0 4 2024-04-14T15:20:27.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.349+0000 7f725affd700 15 client.49318.objecter _session_command_op_remove 0 1 2024-04-14T15:20:27.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:27.353 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:27.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:20:27.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.366+0000 7f726bfd8700 10 client.49318.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:20:27.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.366+0000 7f726bfd8700 15 client.49318.objecter _session_command_op_assign -1 2 2024-04-14T15:20:27.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.366+0000 7f726bfd8700 20 client.49318.objecter _get_session s=0x7f7248005410 osd=0 3 2024-04-14T15:20:27.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.366+0000 7f726bfd8700 20 client.49318.objecter put_session s=0x7f7248005410 osd=0 3 2024-04-14T15:20:27.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.366+0000 7f726bfd8700 20 client.49318.objecter _get_session s=0x7f7248005410 osd=0 3 2024-04-14T15:20:27.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.366+0000 7f726bfd8700 15 client.49318.objecter _session_command_op_remove -1 2 2024-04-14T15:20:27.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.366+0000 7f726bfd8700 20 client.49318.objecter get_session s=0x7f7248005410 osd=0 3 2024-04-14T15:20:27.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.366+0000 7f726bfd8700 15 client.49318.objecter _session_command_op_assign 0 2 2024-04-14T15:20:27.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.366+0000 7f726bfd8700 20 client.49318.objecter put_session s=0x7f7248005410 osd=0 4 2024-04-14T15:20:27.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.366+0000 7f726bfd8700 10 client.49318.objecter _send_command 2 2024-04-14T15:20:27.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.366+0000 7f726bfd8700 1 -- 172.21.15.90:0/937819003 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f7248007a20 con 0x7f7248004760 2024-04-14T15:20:27.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.368+0000 7f725affd700 1 -- 172.21.15.90:0/937819003 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f7248007a20 con 0x7f7248004760 2024-04-14T15:20:27.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.368+0000 7f725affd700 10 client.49318.objecter ms_dispatch 0x7f7264001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:20:27.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.368+0000 7f725affd700 10 client.49318.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:20:27.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.368+0000 7f725affd700 20 client.49318.objecter put_session s=0x7f7248005410 osd=0 4 2024-04-14T15:20:27.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.368+0000 7f725affd700 15 client.49318.objecter _session_command_op_remove 0 2 2024-04-14T15:20:27.369 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:20:27.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.368+0000 7f726bfd8700 10 client.49318.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:20:27.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.368+0000 7f726bfd8700 15 client.49318.objecter _session_command_op_assign -1 3 2024-04-14T15:20:27.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.368+0000 7f726bfd8700 20 client.49318.objecter _get_session s=0x7f7248004160 osd=1 3 2024-04-14T15:20:27.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.368+0000 7f726bfd8700 20 client.49318.objecter put_session s=0x7f7248004160 osd=1 3 2024-04-14T15:20:27.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.368+0000 7f726bfd8700 20 client.49318.objecter _get_session s=0x7f7248004160 osd=1 3 2024-04-14T15:20:27.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.368+0000 7f726bfd8700 15 client.49318.objecter _session_command_op_remove -1 3 2024-04-14T15:20:27.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.368+0000 7f726bfd8700 20 client.49318.objecter get_session s=0x7f7248004160 osd=1 3 2024-04-14T15:20:27.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.368+0000 7f726bfd8700 15 client.49318.objecter _session_command_op_assign 1 3 2024-04-14T15:20:27.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.368+0000 7f726bfd8700 20 client.49318.objecter put_session s=0x7f7248004160 osd=1 4 2024-04-14T15:20:27.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.368+0000 7f726bfd8700 10 client.49318.objecter _send_command 3 2024-04-14T15:20:27.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.368+0000 7f726bfd8700 1 -- 172.21.15.90:0/937819003 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f72480022d0 con 0x7f7248004340 2024-04-14T15:20:27.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.368+0000 7f725affd700 10 client.49318.objecter ms_handle_connect 0x7f7248004340 2024-04-14T15:20:27.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.369+0000 7f725affd700 1 -- 172.21.15.90:0/937819003 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f72480022d0 con 0x7f7248004340 2024-04-14T15:20:27.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.369+0000 7f725affd700 10 client.49318.objecter ms_dispatch 0x7f7264001550 command_reply(tid 3: 0 ) 2024-04-14T15:20:27.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.369+0000 7f725affd700 10 client.49318.objecter _finish_command 3 = system:0 2024-04-14T15:20:27.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.369+0000 7f725affd700 20 client.49318.objecter put_session s=0x7f7248004160 osd=1 4 2024-04-14T15:20:27.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.369+0000 7f725affd700 15 client.49318.objecter _session_command_op_remove 1 3 2024-04-14T15:20:27.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.385+0000 7f726bfd8700 10 client.49318.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:20:27.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.385+0000 7f726bfd8700 15 client.49318.objecter _session_command_op_assign -1 4 2024-04-14T15:20:27.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.385+0000 7f726bfd8700 20 client.49318.objecter _get_session s=0x7f7248004160 osd=1 3 2024-04-14T15:20:27.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.385+0000 7f726bfd8700 20 client.49318.objecter put_session s=0x7f7248004160 osd=1 3 2024-04-14T15:20:27.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.385+0000 7f726bfd8700 20 client.49318.objecter _get_session s=0x7f7248004160 osd=1 3 2024-04-14T15:20:27.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.385+0000 7f726bfd8700 15 client.49318.objecter _session_command_op_remove -1 4 2024-04-14T15:20:27.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.385+0000 7f726bfd8700 20 client.49318.objecter get_session s=0x7f7248004160 osd=1 3 2024-04-14T15:20:27.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.385+0000 7f726bfd8700 15 client.49318.objecter _session_command_op_assign 1 4 2024-04-14T15:20:27.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.385+0000 7f726bfd8700 20 client.49318.objecter put_session s=0x7f7248004160 osd=1 4 2024-04-14T15:20:27.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.385+0000 7f726bfd8700 10 client.49318.objecter _send_command 4 2024-04-14T15:20:27.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.385+0000 7f726bfd8700 1 -- 172.21.15.90:0/937819003 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f7248001750 con 0x7f7248004340 2024-04-14T15:20:27.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.387+0000 7f725affd700 1 -- 172.21.15.90:0/937819003 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f7248001750 con 0x7f7248004340 2024-04-14T15:20:27.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.387+0000 7f725affd700 10 client.49318.objecter ms_dispatch 0x7f7264001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:20:27.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.387+0000 7f725affd700 10 client.49318.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:20:27.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.387+0000 7f725affd700 20 client.49318.objecter put_session s=0x7f7248004160 osd=1 4 2024-04-14T15:20:27.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.387+0000 7f725affd700 15 client.49318.objecter _session_command_op_remove 1 4 2024-04-14T15:20:27.388 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:20:27.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.387+0000 7f726bfd8700 10 client.49318.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:20:27.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.387+0000 7f726bfd8700 15 client.49318.objecter _session_command_op_assign -1 5 2024-04-14T15:20:27.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.387+0000 7f726bfd8700 20 client.49318.objecter _get_session s=0x7f72480019b0 osd=2 3 2024-04-14T15:20:27.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.387+0000 7f726bfd8700 20 client.49318.objecter put_session s=0x7f72480019b0 osd=2 3 2024-04-14T15:20:27.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.387+0000 7f726bfd8700 20 client.49318.objecter _get_session s=0x7f72480019b0 osd=2 3 2024-04-14T15:20:27.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.387+0000 7f726bfd8700 15 client.49318.objecter _session_command_op_remove -1 5 2024-04-14T15:20:27.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.387+0000 7f726bfd8700 20 client.49318.objecter get_session s=0x7f72480019b0 osd=2 3 2024-04-14T15:20:27.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.387+0000 7f726bfd8700 15 client.49318.objecter _session_command_op_assign 2 5 2024-04-14T15:20:27.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.387+0000 7f726bfd8700 20 client.49318.objecter put_session s=0x7f72480019b0 osd=2 4 2024-04-14T15:20:27.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.387+0000 7f726bfd8700 10 client.49318.objecter _send_command 5 2024-04-14T15:20:27.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.387+0000 7f726bfd8700 1 -- 172.21.15.90:0/937819003 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f7248011110 con 0x7f724800d600 2024-04-14T15:20:27.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.388+0000 7f725affd700 10 client.49318.objecter ms_handle_connect 0x7f724800d600 2024-04-14T15:20:27.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.388+0000 7f725affd700 1 -- 172.21.15.90:0/937819003 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f7248011110 con 0x7f724800d600 2024-04-14T15:20:27.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.388+0000 7f725affd700 10 client.49318.objecter ms_dispatch 0x7f7264001550 command_reply(tid 5: 0 ) 2024-04-14T15:20:27.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.388+0000 7f725affd700 10 client.49318.objecter _finish_command 5 = system:0 2024-04-14T15:20:27.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.388+0000 7f725affd700 20 client.49318.objecter put_session s=0x7f72480019b0 osd=2 4 2024-04-14T15:20:27.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.388+0000 7f725affd700 15 client.49318.objecter _session_command_op_remove 2 5 2024-04-14T15:20:27.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.404+0000 7f726bfd8700 10 client.49318.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:20:27.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.404+0000 7f726bfd8700 15 client.49318.objecter _session_command_op_assign -1 6 2024-04-14T15:20:27.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.404+0000 7f726bfd8700 20 client.49318.objecter _get_session s=0x7f72480019b0 osd=2 3 2024-04-14T15:20:27.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.404+0000 7f726bfd8700 20 client.49318.objecter put_session s=0x7f72480019b0 osd=2 3 2024-04-14T15:20:27.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.404+0000 7f726bfd8700 20 client.49318.objecter _get_session s=0x7f72480019b0 osd=2 3 2024-04-14T15:20:27.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.404+0000 7f726bfd8700 15 client.49318.objecter _session_command_op_remove -1 6 2024-04-14T15:20:27.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.404+0000 7f726bfd8700 20 client.49318.objecter get_session s=0x7f72480019b0 osd=2 3 2024-04-14T15:20:27.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.404+0000 7f726bfd8700 15 client.49318.objecter _session_command_op_assign 2 6 2024-04-14T15:20:27.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.404+0000 7f726bfd8700 20 client.49318.objecter put_session s=0x7f72480019b0 osd=2 4 2024-04-14T15:20:27.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.404+0000 7f726bfd8700 10 client.49318.objecter _send_command 6 2024-04-14T15:20:27.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.404+0000 7f726bfd8700 1 -- 172.21.15.90:0/937819003 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f7248008130 con 0x7f724800d600 2024-04-14T15:20:27.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.406+0000 7f725affd700 1 -- 172.21.15.90:0/937819003 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f7248008130 con 0x7f724800d600 2024-04-14T15:20:27.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.406+0000 7f725affd700 10 client.49318.objecter ms_dispatch 0x7f7264001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:20:27.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.406+0000 7f725affd700 10 client.49318.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:20:27.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.406+0000 7f725affd700 20 client.49318.objecter put_session s=0x7f72480019b0 osd=2 4 2024-04-14T15:20:27.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.406+0000 7f725affd700 15 client.49318.objecter _session_command_op_remove 2 6 2024-04-14T15:20:27.407 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:20:27.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.406+0000 7f726bfd8700 10 client.49318.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:20:27.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.406+0000 7f726bfd8700 15 client.49318.objecter _session_command_op_assign -1 7 2024-04-14T15:20:27.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.406+0000 7f726bfd8700 20 client.49318.objecter _get_session s=0x7f72480083c0 osd=3 3 2024-04-14T15:20:27.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.406+0000 7f726bfd8700 20 client.49318.objecter put_session s=0x7f72480083c0 osd=3 3 2024-04-14T15:20:27.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.406+0000 7f726bfd8700 20 client.49318.objecter _get_session s=0x7f72480083c0 osd=3 3 2024-04-14T15:20:27.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.406+0000 7f726bfd8700 15 client.49318.objecter _session_command_op_remove -1 7 2024-04-14T15:20:27.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.406+0000 7f726bfd8700 20 client.49318.objecter get_session s=0x7f72480083c0 osd=3 3 2024-04-14T15:20:27.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.406+0000 7f726bfd8700 15 client.49318.objecter _session_command_op_assign 3 7 2024-04-14T15:20:27.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.406+0000 7f726bfd8700 20 client.49318.objecter put_session s=0x7f72480083c0 osd=3 4 2024-04-14T15:20:27.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.406+0000 7f726bfd8700 10 client.49318.objecter _send_command 7 2024-04-14T15:20:27.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.406+0000 7f726bfd8700 1 -- 172.21.15.90:0/937819003 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f7248013000 con 0x7f7248012440 2024-04-14T15:20:27.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.406+0000 7f725affd700 10 client.49318.objecter ms_handle_connect 0x7f7248012440 2024-04-14T15:20:27.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.407+0000 7f725affd700 1 -- 172.21.15.90:0/937819003 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f7248013000 con 0x7f7248012440 2024-04-14T15:20:27.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.407+0000 7f725affd700 10 client.49318.objecter ms_dispatch 0x7f7264001550 command_reply(tid 7: 0 ) 2024-04-14T15:20:27.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.407+0000 7f725affd700 10 client.49318.objecter _finish_command 7 = system:0 2024-04-14T15:20:27.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.407+0000 7f725affd700 20 client.49318.objecter put_session s=0x7f72480083c0 osd=3 4 2024-04-14T15:20:27.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.407+0000 7f725affd700 15 client.49318.objecter _session_command_op_remove 3 7 2024-04-14T15:20:27.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.424+0000 7f726bfd8700 10 client.49318.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:20:27.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.424+0000 7f726bfd8700 15 client.49318.objecter _session_command_op_assign -1 8 2024-04-14T15:20:27.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.424+0000 7f726bfd8700 20 client.49318.objecter _get_session s=0x7f72480083c0 osd=3 3 2024-04-14T15:20:27.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.424+0000 7f726bfd8700 20 client.49318.objecter put_session s=0x7f72480083c0 osd=3 3 2024-04-14T15:20:27.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.424+0000 7f726bfd8700 20 client.49318.objecter _get_session s=0x7f72480083c0 osd=3 3 2024-04-14T15:20:27.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.424+0000 7f726bfd8700 15 client.49318.objecter _session_command_op_remove -1 8 2024-04-14T15:20:27.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.424+0000 7f726bfd8700 20 client.49318.objecter get_session s=0x7f72480083c0 osd=3 3 2024-04-14T15:20:27.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.424+0000 7f726bfd8700 15 client.49318.objecter _session_command_op_assign 3 8 2024-04-14T15:20:27.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.424+0000 7f726bfd8700 20 client.49318.objecter put_session s=0x7f72480083c0 osd=3 4 2024-04-14T15:20:27.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.424+0000 7f726bfd8700 10 client.49318.objecter _send_command 8 2024-04-14T15:20:27.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.424+0000 7f726bfd8700 1 -- 172.21.15.90:0/937819003 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f7248013000 con 0x7f7248012440 2024-04-14T15:20:27.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.426+0000 7f725affd700 1 -- 172.21.15.90:0/937819003 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f7248013000 con 0x7f7248012440 2024-04-14T15:20:27.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.426+0000 7f725affd700 10 client.49318.objecter ms_dispatch 0x7f7264001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:20:27.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.426+0000 7f725affd700 10 client.49318.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:20:27.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.426+0000 7f725affd700 20 client.49318.objecter put_session s=0x7f72480083c0 osd=3 4 2024-04-14T15:20:27.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.426+0000 7f725affd700 15 client.49318.objecter _session_command_op_remove 3 8 2024-04-14T15:20:27.427 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:20:27.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.426+0000 7f726bfd8700 10 client.49318.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:20:27.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.426+0000 7f726bfd8700 15 client.49318.objecter _session_command_op_assign -1 9 2024-04-14T15:20:27.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.426+0000 7f726bfd8700 20 client.49318.objecter _get_session s=0x7f72480137d0 osd=4 3 2024-04-14T15:20:27.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.426+0000 7f726bfd8700 20 client.49318.objecter put_session s=0x7f72480137d0 osd=4 3 2024-04-14T15:20:27.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.426+0000 7f726bfd8700 20 client.49318.objecter _get_session s=0x7f72480137d0 osd=4 3 2024-04-14T15:20:27.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.426+0000 7f726bfd8700 15 client.49318.objecter _session_command_op_remove -1 9 2024-04-14T15:20:27.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.426+0000 7f726bfd8700 20 client.49318.objecter get_session s=0x7f72480137d0 osd=4 3 2024-04-14T15:20:27.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.426+0000 7f726bfd8700 15 client.49318.objecter _session_command_op_assign 4 9 2024-04-14T15:20:27.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.426+0000 7f726bfd8700 20 client.49318.objecter put_session s=0x7f72480137d0 osd=4 4 2024-04-14T15:20:27.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.426+0000 7f726bfd8700 10 client.49318.objecter _send_command 9 2024-04-14T15:20:27.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.426+0000 7f726bfd8700 1 -- 172.21.15.90:0/937819003 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f724801a780 con 0x7f7248016c70 2024-04-14T15:20:27.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.426+0000 7f725affd700 10 client.49318.objecter ms_handle_connect 0x7f7248016c70 2024-04-14T15:20:27.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.427+0000 7f725affd700 1 -- 172.21.15.90:0/937819003 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f724801a780 con 0x7f7248016c70 2024-04-14T15:20:27.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.427+0000 7f725affd700 10 client.49318.objecter ms_dispatch 0x7f7264001550 command_reply(tid 9: 0 ) 2024-04-14T15:20:27.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.427+0000 7f725affd700 10 client.49318.objecter _finish_command 9 = system:0 2024-04-14T15:20:27.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.427+0000 7f725affd700 20 client.49318.objecter put_session s=0x7f72480137d0 osd=4 4 2024-04-14T15:20:27.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.427+0000 7f725affd700 15 client.49318.objecter _session_command_op_remove 4 9 2024-04-14T15:20:27.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.443+0000 7f726bfd8700 10 client.49318.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:20:27.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.443+0000 7f726bfd8700 15 client.49318.objecter _session_command_op_assign -1 10 2024-04-14T15:20:27.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.443+0000 7f726bfd8700 20 client.49318.objecter _get_session s=0x7f72480137d0 osd=4 3 2024-04-14T15:20:27.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.443+0000 7f726bfd8700 20 client.49318.objecter put_session s=0x7f72480137d0 osd=4 3 2024-04-14T15:20:27.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.443+0000 7f726bfd8700 20 client.49318.objecter _get_session s=0x7f72480137d0 osd=4 3 2024-04-14T15:20:27.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.443+0000 7f726bfd8700 15 client.49318.objecter _session_command_op_remove -1 10 2024-04-14T15:20:27.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.443+0000 7f726bfd8700 20 client.49318.objecter get_session s=0x7f72480137d0 osd=4 3 2024-04-14T15:20:27.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.443+0000 7f726bfd8700 15 client.49318.objecter _session_command_op_assign 4 10 2024-04-14T15:20:27.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.443+0000 7f726bfd8700 20 client.49318.objecter put_session s=0x7f72480137d0 osd=4 4 2024-04-14T15:20:27.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.443+0000 7f726bfd8700 10 client.49318.objecter _send_command 10 2024-04-14T15:20:27.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.443+0000 7f726bfd8700 1 -- 172.21.15.90:0/937819003 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f724800c730 con 0x7f7248016c70 2024-04-14T15:20:27.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.445+0000 7f725affd700 1 -- 172.21.15.90:0/937819003 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f724800c730 con 0x7f7248016c70 2024-04-14T15:20:27.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.445+0000 7f725affd700 10 client.49318.objecter ms_dispatch 0x7f7264001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:20:27.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.445+0000 7f725affd700 10 client.49318.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:20:27.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.445+0000 7f725affd700 20 client.49318.objecter put_session s=0x7f72480137d0 osd=4 4 2024-04-14T15:20:27.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.445+0000 7f725affd700 15 client.49318.objecter _session_command_op_remove 4 10 2024-04-14T15:20:27.446 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:20:27.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.445+0000 7f726bfd8700 10 client.49318.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:20:27.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.445+0000 7f726bfd8700 15 client.49318.objecter _session_command_op_assign -1 11 2024-04-14T15:20:27.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.445+0000 7f726bfd8700 20 client.49318.objecter _get_session s=0x7f724800c730 osd=5 3 2024-04-14T15:20:27.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.445+0000 7f726bfd8700 20 client.49318.objecter put_session s=0x7f724800c730 osd=5 3 2024-04-14T15:20:27.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.445+0000 7f726bfd8700 20 client.49318.objecter _get_session s=0x7f724800c730 osd=5 3 2024-04-14T15:20:27.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.445+0000 7f726bfd8700 15 client.49318.objecter _session_command_op_remove -1 11 2024-04-14T15:20:27.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.445+0000 7f726bfd8700 20 client.49318.objecter get_session s=0x7f724800c730 osd=5 3 2024-04-14T15:20:27.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.445+0000 7f726bfd8700 15 client.49318.objecter _session_command_op_assign 5 11 2024-04-14T15:20:27.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.445+0000 7f726bfd8700 20 client.49318.objecter put_session s=0x7f724800c730 osd=5 4 2024-04-14T15:20:27.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.445+0000 7f726bfd8700 10 client.49318.objecter _send_command 11 2024-04-14T15:20:27.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.445+0000 7f726bfd8700 1 -- 172.21.15.90:0/937819003 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f7248011fb0 con 0x7f7248011a90 2024-04-14T15:20:27.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.446+0000 7f725affd700 10 client.49318.objecter ms_handle_connect 0x7f7248011a90 2024-04-14T15:20:27.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.446+0000 7f725affd700 1 -- 172.21.15.90:0/937819003 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f7248011fb0 con 0x7f7248011a90 2024-04-14T15:20:27.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.446+0000 7f725affd700 10 client.49318.objecter ms_dispatch 0x7f7264001550 command_reply(tid 11: 0 ) 2024-04-14T15:20:27.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.446+0000 7f725affd700 10 client.49318.objecter _finish_command 11 = system:0 2024-04-14T15:20:27.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.446+0000 7f725affd700 20 client.49318.objecter put_session s=0x7f724800c730 osd=5 4 2024-04-14T15:20:27.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.446+0000 7f725affd700 15 client.49318.objecter _session_command_op_remove 5 11 2024-04-14T15:20:27.454 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:27.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.465+0000 7f726bfd8700 10 client.49318.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:20:27.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.465+0000 7f726bfd8700 15 client.49318.objecter _session_command_op_assign -1 12 2024-04-14T15:20:27.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.465+0000 7f726bfd8700 20 client.49318.objecter _get_session s=0x7f724800c730 osd=5 3 2024-04-14T15:20:27.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.465+0000 7f726bfd8700 20 client.49318.objecter put_session s=0x7f724800c730 osd=5 3 2024-04-14T15:20:27.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.465+0000 7f726bfd8700 20 client.49318.objecter _get_session s=0x7f724800c730 osd=5 3 2024-04-14T15:20:27.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.465+0000 7f726bfd8700 15 client.49318.objecter _session_command_op_remove -1 12 2024-04-14T15:20:27.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.465+0000 7f726bfd8700 20 client.49318.objecter get_session s=0x7f724800c730 osd=5 3 2024-04-14T15:20:27.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.465+0000 7f726bfd8700 15 client.49318.objecter _session_command_op_assign 5 12 2024-04-14T15:20:27.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.465+0000 7f726bfd8700 20 client.49318.objecter put_session s=0x7f724800c730 osd=5 4 2024-04-14T15:20:27.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.465+0000 7f726bfd8700 10 client.49318.objecter _send_command 12 2024-04-14T15:20:27.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.465+0000 7f726bfd8700 1 -- 172.21.15.90:0/937819003 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f72480121c0 con 0x7f7248011a90 2024-04-14T15:20:27.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.466+0000 7f725affd700 1 -- 172.21.15.90:0/937819003 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f72480121c0 con 0x7f7248011a90 2024-04-14T15:20:27.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.467+0000 7f725affd700 10 client.49318.objecter ms_dispatch 0x7f7264001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:20:27.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.467+0000 7f725affd700 10 client.49318.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:20:27.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.467+0000 7f725affd700 20 client.49318.objecter put_session s=0x7f724800c730 osd=5 4 2024-04-14T15:20:27.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.467+0000 7f725affd700 15 client.49318.objecter _session_command_op_remove 5 12 2024-04-14T15:20:27.467 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:20:27.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.467+0000 7f726bfd8700 10 client.49318.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:20:27.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.467+0000 7f726bfd8700 15 client.49318.objecter _session_command_op_assign -1 13 2024-04-14T15:20:27.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.467+0000 7f726bfd8700 20 client.49318.objecter _get_session s=0x7f72480121c0 osd=6 3 2024-04-14T15:20:27.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.467+0000 7f726bfd8700 20 client.49318.objecter put_session s=0x7f72480121c0 osd=6 3 2024-04-14T15:20:27.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.467+0000 7f726bfd8700 20 client.49318.objecter _get_session s=0x7f72480121c0 osd=6 3 2024-04-14T15:20:27.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.467+0000 7f726bfd8700 15 client.49318.objecter _session_command_op_remove -1 13 2024-04-14T15:20:27.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.467+0000 7f726bfd8700 20 client.49318.objecter get_session s=0x7f72480121c0 osd=6 3 2024-04-14T15:20:27.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.467+0000 7f726bfd8700 15 client.49318.objecter _session_command_op_assign 6 13 2024-04-14T15:20:27.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.467+0000 7f726bfd8700 20 client.49318.objecter put_session s=0x7f72480121c0 osd=6 4 2024-04-14T15:20:27.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.467+0000 7f726bfd8700 10 client.49318.objecter _send_command 13 2024-04-14T15:20:27.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.467+0000 7f726bfd8700 1 -- 172.21.15.90:0/937819003 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f72480230f0 con 0x7f724801f540 2024-04-14T15:20:27.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.467+0000 7f725affd700 10 client.49318.objecter ms_handle_connect 0x7f724801f540 2024-04-14T15:20:27.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.468+0000 7f725affd700 1 -- 172.21.15.90:0/937819003 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f72480230f0 con 0x7f724801f540 2024-04-14T15:20:27.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.468+0000 7f725affd700 10 client.49318.objecter ms_dispatch 0x7f7264001550 command_reply(tid 13: 0 ) 2024-04-14T15:20:27.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.468+0000 7f725affd700 10 client.49318.objecter _finish_command 13 = system:0 2024-04-14T15:20:27.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.468+0000 7f725affd700 20 client.49318.objecter put_session s=0x7f72480121c0 osd=6 4 2024-04-14T15:20:27.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.468+0000 7f725affd700 15 client.49318.objecter _session_command_op_remove 6 13 2024-04-14T15:20:27.478 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:27.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:27.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:27.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.485+0000 7f726bfd8700 10 client.49318.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:20:27.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.485+0000 7f726bfd8700 15 client.49318.objecter _session_command_op_assign -1 14 2024-04-14T15:20:27.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.485+0000 7f726bfd8700 20 client.49318.objecter _get_session s=0x7f72480121c0 osd=6 3 2024-04-14T15:20:27.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.485+0000 7f726bfd8700 20 client.49318.objecter put_session s=0x7f72480121c0 osd=6 3 2024-04-14T15:20:27.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.485+0000 7f726bfd8700 20 client.49318.objecter _get_session s=0x7f72480121c0 osd=6 3 2024-04-14T15:20:27.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.485+0000 7f726bfd8700 15 client.49318.objecter _session_command_op_remove -1 14 2024-04-14T15:20:27.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.485+0000 7f726bfd8700 20 client.49318.objecter get_session s=0x7f72480121c0 osd=6 3 2024-04-14T15:20:27.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.485+0000 7f726bfd8700 15 client.49318.objecter _session_command_op_assign 6 14 2024-04-14T15:20:27.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.485+0000 7f726bfd8700 20 client.49318.objecter put_session s=0x7f72480121c0 osd=6 4 2024-04-14T15:20:27.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.485+0000 7f726bfd8700 10 client.49318.objecter _send_command 14 2024-04-14T15:20:27.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.485+0000 7f726bfd8700 1 -- 172.21.15.90:0/937819003 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f72480230f0 con 0x7f724801f540 2024-04-14T15:20:27.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.487+0000 7f725affd700 1 -- 172.21.15.90:0/937819003 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f72480230f0 con 0x7f724801f540 2024-04-14T15:20:27.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.487+0000 7f725affd700 10 client.49318.objecter ms_dispatch 0x7f7264001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:20:27.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.487+0000 7f725affd700 10 client.49318.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:20:27.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.487+0000 7f725affd700 20 client.49318.objecter put_session s=0x7f72480121c0 osd=6 4 2024-04-14T15:20:27.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.487+0000 7f725affd700 15 client.49318.objecter _session_command_op_remove 6 14 2024-04-14T15:20:27.488 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:20:27.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.487+0000 7f726bfd8700 10 client.49318.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:20:27.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.487+0000 7f726bfd8700 15 client.49318.objecter _session_command_op_assign -1 15 2024-04-14T15:20:27.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.487+0000 7f726bfd8700 20 client.49318.objecter _get_session s=0x7f72480230f0 osd=7 3 2024-04-14T15:20:27.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.487+0000 7f726bfd8700 20 client.49318.objecter put_session s=0x7f72480230f0 osd=7 3 2024-04-14T15:20:27.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.487+0000 7f726bfd8700 20 client.49318.objecter _get_session s=0x7f72480230f0 osd=7 3 2024-04-14T15:20:27.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.487+0000 7f726bfd8700 15 client.49318.objecter _session_command_op_remove -1 15 2024-04-14T15:20:27.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.487+0000 7f726bfd8700 20 client.49318.objecter get_session s=0x7f72480230f0 osd=7 3 2024-04-14T15:20:27.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.487+0000 7f726bfd8700 15 client.49318.objecter _session_command_op_assign 7 15 2024-04-14T15:20:27.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.487+0000 7f726bfd8700 20 client.49318.objecter put_session s=0x7f72480230f0 osd=7 4 2024-04-14T15:20:27.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.487+0000 7f726bfd8700 10 client.49318.objecter _send_command 15 2024-04-14T15:20:27.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.487+0000 7f726bfd8700 1 -- 172.21.15.90:0/937819003 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f7248024350 con 0x7f7248025010 2024-04-14T15:20:27.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.488+0000 7f725affd700 10 client.49318.objecter ms_handle_connect 0x7f7248025010 2024-04-14T15:20:27.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.488+0000 7f725affd700 1 -- 172.21.15.90:0/937819003 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f7248024350 con 0x7f7248025010 2024-04-14T15:20:27.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.488+0000 7f725affd700 10 client.49318.objecter ms_dispatch 0x7f7264001550 command_reply(tid 15: 0 ) 2024-04-14T15:20:27.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.488+0000 7f725affd700 10 client.49318.objecter _finish_command 15 = system:0 2024-04-14T15:20:27.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.488+0000 7f725affd700 20 client.49318.objecter put_session s=0x7f72480230f0 osd=7 4 2024-04-14T15:20:27.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.488+0000 7f725affd700 15 client.49318.objecter _session_command_op_remove 7 15 2024-04-14T15:20:27.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.505+0000 7f726bfd8700 10 client.49318.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:20:27.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.505+0000 7f726bfd8700 15 client.49318.objecter _session_command_op_assign -1 16 2024-04-14T15:20:27.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.505+0000 7f726bfd8700 20 client.49318.objecter _get_session s=0x7f72480230f0 osd=7 3 2024-04-14T15:20:27.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.505+0000 7f726bfd8700 20 client.49318.objecter put_session s=0x7f72480230f0 osd=7 3 2024-04-14T15:20:27.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.505+0000 7f726bfd8700 20 client.49318.objecter _get_session s=0x7f72480230f0 osd=7 3 2024-04-14T15:20:27.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.505+0000 7f726bfd8700 15 client.49318.objecter _session_command_op_remove -1 16 2024-04-14T15:20:27.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.505+0000 7f726bfd8700 20 client.49318.objecter get_session s=0x7f72480230f0 osd=7 3 2024-04-14T15:20:27.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.505+0000 7f726bfd8700 15 client.49318.objecter _session_command_op_assign 7 16 2024-04-14T15:20:27.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.505+0000 7f726bfd8700 20 client.49318.objecter put_session s=0x7f72480230f0 osd=7 4 2024-04-14T15:20:27.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.505+0000 7f726bfd8700 10 client.49318.objecter _send_command 16 2024-04-14T15:20:27.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.505+0000 7f726bfd8700 1 -- 172.21.15.90:0/937819003 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f7248024350 con 0x7f7248025010 2024-04-14T15:20:27.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.507+0000 7f725affd700 1 -- 172.21.15.90:0/937819003 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f7248024350 con 0x7f7248025010 2024-04-14T15:20:27.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.507+0000 7f725affd700 10 client.49318.objecter ms_dispatch 0x7f7264001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:20:27.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.507+0000 7f725affd700 10 client.49318.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:20:27.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.507+0000 7f725affd700 20 client.49318.objecter put_session s=0x7f72480230f0 osd=7 4 2024-04-14T15:20:27.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.507+0000 7f725affd700 15 client.49318.objecter _session_command_op_remove 7 16 2024-04-14T15:20:27.508 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:20:27.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.507+0000 7f726bfd8700 10 librados: watch_flush enter 2024-04-14T15:20:27.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.507+0000 7f726bfd8700 10 librados: watch_flush exit 2024-04-14T15:20:27.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.507+0000 7f726bfd8700 10 client.49318.objecter close_session for osd.0 2024-04-14T15:20:27.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.507+0000 7f726bfd8700 1 -- 172.21.15.90:0/937819003 >> v1:172.21.15.90:6801/3761806885 conn(0x7f7248004760 legacy=0x7f7248002780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:27.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.507+0000 7f726bfd8700 20 client.49318.objecter put_session s=0x7f7248005410 osd=0 1 2024-04-14T15:20:27.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.507+0000 7f726bfd8700 10 client.49318.objecter close_session for osd.1 2024-04-14T15:20:27.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.507+0000 7f726bfd8700 1 -- 172.21.15.90:0/937819003 >> v1:172.21.15.90:6805/4170361699 conn(0x7f7248004340 legacy=0x7f724800ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:27.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.507+0000 7f726bfd8700 20 client.49318.objecter put_session s=0x7f7248004160 osd=1 1 2024-04-14T15:20:27.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.507+0000 7f726bfd8700 10 client.49318.objecter close_session for osd.2 2024-04-14T15:20:27.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.507+0000 7f726bfd8700 1 -- 172.21.15.90:0/937819003 >> v1:172.21.15.90:6809/1686184251 conn(0x7f724800d600 legacy=0x7f724800f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:27.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.507+0000 7f726bfd8700 20 client.49318.objecter put_session s=0x7f72480019b0 osd=2 1 2024-04-14T15:20:27.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.507+0000 7f726bfd8700 10 client.49318.objecter close_session for osd.3 2024-04-14T15:20:27.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.507+0000 7f726bfd8700 1 -- 172.21.15.90:0/937819003 >> v1:172.21.15.90:6813/2270166774 conn(0x7f7248012440 legacy=0x7f72480128f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:27.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.507+0000 7f726bfd8700 20 client.49318.objecter put_session s=0x7f72480083c0 osd=3 1 2024-04-14T15:20:27.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.507+0000 7f726bfd8700 10 client.49318.objecter close_session for osd.4 2024-04-14T15:20:27.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.507+0000 7f726a575700 1 -- 172.21.15.90:0/937819003 reap_dead start 2024-04-14T15:20:27.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.507+0000 7f726bfd8700 1 -- 172.21.15.90:0/937819003 >> v1:172.21.15.92:6800/3887549733 conn(0x7f7248016c70 legacy=0x7f7248019060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:27.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.507+0000 7f726bfd8700 20 client.49318.objecter put_session s=0x7f72480137d0 osd=4 1 2024-04-14T15:20:27.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.507+0000 7f726a575700 1 -- 172.21.15.90:0/937819003 reap_dead start 2024-04-14T15:20:27.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.507+0000 7f726bfd8700 10 client.49318.objecter close_session for osd.5 2024-04-14T15:20:27.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.507+0000 7f726bfd8700 1 -- 172.21.15.90:0/937819003 >> v1:172.21.15.92:6804/3244561440 conn(0x7f7248011a90 legacy=0x7f724801d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:27.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.507+0000 7f726bfd8700 20 client.49318.objecter put_session s=0x7f724800c730 osd=5 1 2024-04-14T15:20:27.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.507+0000 7f726bfd8700 10 client.49318.objecter close_session for osd.6 2024-04-14T15:20:27.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.507+0000 7f726bfd8700 1 -- 172.21.15.90:0/937819003 >> v1:172.21.15.92:6808/636125963 conn(0x7f724801f540 legacy=0x7f7248021930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:27.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.507+0000 7f726bfd8700 20 client.49318.objecter put_session s=0x7f72480121c0 osd=6 1 2024-04-14T15:20:27.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.507+0000 7f726bfd8700 10 client.49318.objecter close_session for osd.7 2024-04-14T15:20:27.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.507+0000 7f726bfd8700 1 -- 172.21.15.90:0/937819003 >> v1:172.21.15.92:6812/1773706716 conn(0x7f7248025010 legacy=0x7f7248025470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:27.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.507+0000 7f726bfd8700 20 client.49318.objecter put_session s=0x7f72480230f0 osd=7 1 2024-04-14T15:20:27.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.507+0000 7f726bfd8700 20 client.49318.objecter shutdown clearing up homeless session... 2024-04-14T15:20:27.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.507+0000 7f726bfd8700 10 client.49318.objecter successfully canceled tick 2024-04-14T15:20:27.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.508+0000 7f726bfd8700 1 -- 172.21.15.90:0/937819003 >> v1:172.21.15.90:6800/420935427 conn(0x7f7250071390 legacy=0x7f7250073850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:27.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.508+0000 7f726bfd8700 1 -- 172.21.15.90:0/937819003 >> v1:172.21.15.92:6789/0 conn(0x7f72641088c0 legacy=0x7f72640765e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:27.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.508+0000 7f726a575700 1 -- 172.21.15.90:0/937819003 reap_dead start 2024-04-14T15:20:27.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.508+0000 7f726bfd8700 1 -- 172.21.15.90:0/937819003 shutdown_connections 2024-04-14T15:20:27.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.508+0000 7f726bfd8700 1 -- 172.21.15.90:0/937819003 >> 172.21.15.90:0/937819003 conn(0x7f72640fe3d0 msgr2=0x7f7264112860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:20:27.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.508+0000 7f726bfd8700 1 -- 172.21.15.90:0/937819003 shutdown_connections 2024-04-14T15:20:27.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.508+0000 7f726bfd8700 1 -- 172.21.15.90:0/937819003 wait complete. 2024-04-14T15:20:27.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:27.508+0000 7f726bfd8700 1 librados: shutdown 2024-04-14T15:20:27.512 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:20:27.512 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:20:27.512 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:20:27.512 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:20:27.512 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:20:27.512 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:20:27.512 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:20:27.512 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:20:27.513 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:20:27.580 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:20:27.604 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:27.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:27.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:20:27.705 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:20:27.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:27.730 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:27.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:20:27.831 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:20:27.854 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:27.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:27.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:20:27.956 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:20:27.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:27.986 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:27.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:20:28.087 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:20:28.110 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:28.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:28.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:20:28.212 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:20:28.236 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:28.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:28.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:20:28.338 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:28.362 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:28.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:28.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:28.365 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:27 smithi092 ceph-mon[33966]: osdmap e502: 8 total, 8 up, 8 in 2024-04-14T15:20:28.365 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:27 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:20:28.365 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:27 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:20:28.365 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:27 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-04-14T15:20:28.365 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:27 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/937819003' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:20:28.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:27 smithi090 ceph-mon[30399]: osdmap e502: 8 total, 8 up, 8 in 2024-04-14T15:20:28.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:27 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:20:28.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:27 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:20:28.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:27 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-04-14T15:20:28.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:27 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/937819003' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:20:28.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:27 smithi090 ceph-mon[41670]: osdmap e502: 8 total, 8 up, 8 in 2024-04-14T15:20:28.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:27 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:20:28.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:27 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:20:28.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:27 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-04-14T15:20:28.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:27 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/937819003' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:20:28.464 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:20:28.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:28.488 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:28.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:20:28.589 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:20:28.613 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:28.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:28.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:20:28.714 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:28.738 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:28.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:28.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:28.840 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:20:28.864 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:28.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:28.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:20:28.966 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:20:29.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:29.005 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:29.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:20:29.099 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:28 smithi090 ceph-mon[30399]: pgmap v728: 258 pgs: 32 unknown, 226 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 990 B/s rd, 0 op/s 2024-04-14T15:20:29.099 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:28 smithi090 ceph-mon[30399]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:20:29.099 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:28 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:20:29.099 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:28 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-04-14T15:20:29.099 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:28 smithi090 ceph-mon[30399]: osdmap e503: 8 total, 8 up, 8 in 2024-04-14T15:20:29.099 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:28 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-14T15:20:29.099 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:28 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-14T15:20:29.099 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:28 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-14T15:20:29.099 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:28 smithi090 ceph-mon[41670]: pgmap v728: 258 pgs: 32 unknown, 226 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 990 B/s rd, 0 op/s 2024-04-14T15:20:29.100 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:28 smithi090 ceph-mon[41670]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:20:29.100 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:28 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:20:29.100 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:28 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-04-14T15:20:29.100 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:28 smithi090 ceph-mon[41670]: osdmap e503: 8 total, 8 up, 8 in 2024-04-14T15:20:29.100 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:28 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-14T15:20:29.100 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:28 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-14T15:20:29.100 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:28 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-14T15:20:29.106 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:20:29.131 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:29.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:29.132 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:20:29.232 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:20:29.242 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-04-14T15:20:29.243 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:20:29.243 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:28 smithi092 ceph-mon[33966]: pgmap v728: 258 pgs: 32 unknown, 226 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 990 B/s rd, 0 op/s 2024-04-14T15:20:29.243 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:28 smithi092 ceph-mon[33966]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:20:29.243 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:28 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:20:29.243 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:28 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-04-14T15:20:29.243 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:28 smithi092 ceph-mon[33966]: osdmap e503: 8 total, 8 up, 8 in 2024-04-14T15:20:29.243 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:28 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-14T15:20:29.243 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:28 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-14T15:20:29.243 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:28 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-14T15:20:29.255 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:29.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:29.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:20:29.357 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:20:29.380 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:29.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:29.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:20:29.482 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:29.626 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:29.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:29.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:29.728 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:29.751 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:29.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:29.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:29.852 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:20:29.879 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:29.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:29.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:20:29.908 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-14T15:20:29.981 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:20:30.014 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:30.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:30.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:20:30.116 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:20:30.140 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:30.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:30.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:20:30.174 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:29 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-14T15:20:30.174 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:29 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-81","var": "dedup_tier","val": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:20:30.174 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:29 smithi090 ceph-mon[41670]: osdmap e504: 8 total, 8 up, 8 in 2024-04-14T15:20:30.174 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:29 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-81","var": "dedup_tier","val": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:20:30.174 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:29 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3953547519' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:20:30.174 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:29 smithi090 ceph-mon[41670]: from='client.49324 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:20:30.174 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:29 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-14T15:20:30.174 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:29 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-81","var": "dedup_tier","val": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:20:30.175 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:29 smithi090 ceph-mon[30399]: osdmap e504: 8 total, 8 up, 8 in 2024-04-14T15:20:30.175 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:29 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-81","var": "dedup_tier","val": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:20:30.175 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:29 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3953547519' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:20:30.175 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:29 smithi090 ceph-mon[30399]: from='client.49324 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:20:30.194 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:20:30.243 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:20:30.265 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:30.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:30.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:20:30.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.202+0000 7fbf7fa86700 1 -- 172.21.15.90:0/3514309524 >> v1:172.21.15.90:6789/0 conn(0x7fbf7810a630 legacy=0x7fbf7810ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:30.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.202+0000 7fbf7fa86700 1 -- 172.21.15.90:0/3514309524 shutdown_connections 2024-04-14T15:20:30.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.202+0000 7fbf7fa86700 1 -- 172.21.15.90:0/3514309524 >> 172.21.15.90:0/3514309524 conn(0x7fbf780fd2b0 msgr2=0x7fbf780ff6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:20:30.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.202+0000 7fbf7fa86700 1 -- 172.21.15.90:0/3514309524 shutdown_connections 2024-04-14T15:20:30.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.202+0000 7fbf7fa86700 1 -- 172.21.15.90:0/3514309524 wait complete. 2024-04-14T15:20:30.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.203+0000 7fbf7fa86700 1 librados: starting msgr at 2024-04-14T15:20:30.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.203+0000 7fbf7fa86700 1 librados: starting objecter 2024-04-14T15:20:30.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.203+0000 7fbf7fa86700 1 Processor -- start 2024-04-14T15:20:30.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.203+0000 7fbf7fa86700 1 -- start start 2024-04-14T15:20:30.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.203+0000 7fbf7fa86700 1 librados: setting wanted keys 2024-04-14T15:20:30.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.203+0000 7fbf7fa86700 1 librados: calling monclient init 2024-04-14T15:20:30.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.203+0000 7fbf7fa86700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbf781a9d50 con 0x7fbf7810a630 2024-04-14T15:20:30.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.203+0000 7fbf7fa86700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbf781aaf70 con 0x7fbf78107780 2024-04-14T15:20:30.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.203+0000 7fbf7fa86700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbf781ac1b0 con 0x7fbf7810e200 2024-04-14T15:20:30.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.203+0000 7fbf7e023700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7fbf7810e200 0x7fbf781a8410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:39746/0 (socket says 172.21.15.90:39746) 2024-04-14T15:20:30.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.203+0000 7fbf7d021700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fbf7810a630 0x7fbf781a4ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:38366/0 (socket says 172.21.15.90:38366) 2024-04-14T15:20:30.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.203+0000 7fbf7e023700 1 -- 172.21.15.90:0/3953547519 learned_addr learned my addr 172.21.15.90:0/3953547519 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:20:30.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.203+0000 7fbf6effd700 10 client.?.objecter ms_handle_connect 0x7fbf7810e200 2024-04-14T15:20:30.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.203+0000 7fbf6effd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:20:30.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.203+0000 7fbf6effd700 10 client.?.objecter ms_handle_connect 0x7fbf7810a630 2024-04-14T15:20:30.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.203+0000 7fbf6effd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:20:30.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.203+0000 7fbf6effd700 10 client.?.objecter ms_handle_connect 0x7fbf78107780 2024-04-14T15:20:30.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.203+0000 7fbf6effd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:20:30.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.203+0000 7fbf6effd700 1 -- 172.21.15.90:0/3953547519 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 602539934 0 0) 0x7fbf781aaf70 con 0x7fbf78107780 2024-04-14T15:20:30.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.203+0000 7fbf6effd700 1 -- 172.21.15.90:0/3953547519 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf640035b0 con 0x7fbf78107780 2024-04-14T15:20:30.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.203+0000 7fbf6effd700 1 -- 172.21.15.90:0/3953547519 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2486951702 0 0) 0x7fbf781ac1b0 con 0x7fbf7810e200 2024-04-14T15:20:30.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.203+0000 7fbf6effd700 1 -- 172.21.15.90:0/3953547519 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf781aaf70 con 0x7fbf7810e200 2024-04-14T15:20:30.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.203+0000 7fbf6effd700 1 -- 172.21.15.90:0/3953547519 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4124627950 0 0) 0x7fbf781a9d50 con 0x7fbf7810a630 2024-04-14T15:20:30.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.203+0000 7fbf6effd700 1 -- 172.21.15.90:0/3953547519 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf781ac1b0 con 0x7fbf7810a630 2024-04-14T15:20:30.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.204+0000 7fbf6effd700 1 -- 172.21.15.90:0/3953547519 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1909563448 0 0) 0x7fbf640035b0 con 0x7fbf78107780 2024-04-14T15:20:30.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.204+0000 7fbf6effd700 1 -- 172.21.15.90:0/3953547519 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbf781a9d50 con 0x7fbf78107780 2024-04-14T15:20:30.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.204+0000 7fbf6effd700 1 -- 172.21.15.90:0/3953547519 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fbf680018b0 con 0x7fbf78107780 2024-04-14T15:20:30.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.204+0000 7fbf6effd700 1 -- 172.21.15.90:0/3953547519 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 898707262 0 0) 0x7fbf781aaf70 con 0x7fbf7810e200 2024-04-14T15:20:30.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.204+0000 7fbf6effd700 1 -- 172.21.15.90:0/3953547519 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbf640035b0 con 0x7fbf7810e200 2024-04-14T15:20:30.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.204+0000 7fbf6effd700 1 -- 172.21.15.90:0/3953547519 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fbf70001c70 con 0x7fbf7810e200 2024-04-14T15:20:30.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.204+0000 7fbf6effd700 1 -- 172.21.15.90:0/3953547519 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3908249580 0 0) 0x7fbf781ac1b0 con 0x7fbf7810a630 2024-04-14T15:20:30.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.204+0000 7fbf6effd700 1 -- 172.21.15.90:0/3953547519 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbf781aaf70 con 0x7fbf7810a630 2024-04-14T15:20:30.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.204+0000 7fbf6effd700 1 -- 172.21.15.90:0/3953547519 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3928062475 0 0) 0x7fbf781a9d50 con 0x7fbf78107780 2024-04-14T15:20:30.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.204+0000 7fbf6effd700 1 -- 172.21.15.90:0/3953547519 >> v1:172.21.15.90:6790/0 conn(0x7fbf7810e200 legacy=0x7fbf781a8410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:30.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.204+0000 7fbf6effd700 1 -- 172.21.15.90:0/3953547519 >> v1:172.21.15.90:6789/0 conn(0x7fbf7810a630 legacy=0x7fbf781a4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:30.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.204+0000 7fbf6effd700 1 -- 172.21.15.90:0/3953547519 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf781ad3f0 con 0x7fbf78107780 2024-04-14T15:20:30.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.204+0000 7fbf7fa86700 1 -- 172.21.15.90:0/3953547519 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbf781ac3e0 con 0x7fbf78107780 2024-04-14T15:20:30.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.204+0000 7fbf7fa86700 10 client.49324.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:20:30.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.204+0000 7fbf7fa86700 1 -- 172.21.15.90:0/3953547519 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fbf781ac8a0 con 0x7fbf78107780 2024-04-14T15:20:30.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.204+0000 7fbf7fa86700 1 librados: init done 2024-04-14T15:20:30.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.204+0000 7fbf6effd700 1 -- 172.21.15.90:0/3953547519 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fbf68003b40 con 0x7fbf78107780 2024-04-14T15:20:30.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.204+0000 7fbf6effd700 1 -- 172.21.15.90:0/3953547519 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fbf68004a90 con 0x7fbf78107780 2024-04-14T15:20:30.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.205+0000 7fbf7fa86700 1 -- 172.21.15.90:0/3953547519 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf5c005150 con 0x7fbf78107780 2024-04-14T15:20:30.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.205+0000 7fbf6effd700 1 -- 172.21.15.90:0/3953547519 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fbf680036a0 con 0x7fbf78107780 2024-04-14T15:20:30.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.206+0000 7fbf6effd700 1 -- 172.21.15.90:0/3953547519 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(504..504 src has 51..504) ==== 8503+0+0 (unknown 1857505329 0 0) 0x7fbf68089880 con 0x7fbf78107780 2024-04-14T15:20:30.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.206+0000 7fbf6effd700 10 client.49324.objecter ms_dispatch 0x7fbf78001550 osd_map(504..504 src has 51..504) 2024-04-14T15:20:30.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.206+0000 7fbf6effd700 3 client.49324.objecter handle_osd_map got epochs [504,504] > 0 2024-04-14T15:20:30.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.206+0000 7fbf6effd700 3 client.49324.objecter handle_osd_map decoding full epoch 504 2024-04-14T15:20:30.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.206+0000 7fbf6effd700 20 client.49324.objecter dump_active .. 0 homeless 2024-04-14T15:20:30.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.206+0000 7fbf6effd700 10 client.49324.objecter ms_handle_connect 0x7fbf64071440 2024-04-14T15:20:30.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.210+0000 7fbf6effd700 1 -- 172.21.15.90:0/3953547519 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fbf68004a90 con 0x7fbf78107780 2024-04-14T15:20:30.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.425+0000 7fbf7fa86700 1 -- 172.21.15.90:0/3953547519 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7fbf5c005440 con 0x7fbf78107780 2024-04-14T15:20:30.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.983+0000 7fbf6effd700 1 -- 172.21.15.90:0/3953547519 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v505) ==== 91+0+0 (unknown 1236520708 0 0) 0x7fbf68056140 con 0x7fbf78107780 2024-04-14T15:20:30.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.986+0000 7fbf7fa86700 10 librados: watch_flush enter 2024-04-14T15:20:30.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.986+0000 7fbf7fa86700 10 librados: watch_flush exit 2024-04-14T15:20:30.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.986+0000 7fbf7fa86700 20 client.49324.objecter shutdown clearing up homeless session... 2024-04-14T15:20:30.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.986+0000 7fbf7fa86700 10 client.49324.objecter successfully canceled tick 2024-04-14T15:20:30.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.987+0000 7fbf7fa86700 1 -- 172.21.15.90:0/3953547519 >> v1:172.21.15.90:6800/420935427 conn(0x7fbf64071440 legacy=0x7fbf64073900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:30.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.987+0000 7fbf7fa86700 1 -- 172.21.15.90:0/3953547519 >> v1:172.21.15.92:6789/0 conn(0x7fbf78107780 legacy=0x7fbf7810dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:30.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.987+0000 7fbf7fa86700 1 -- 172.21.15.90:0/3953547519 shutdown_connections 2024-04-14T15:20:30.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.987+0000 7fbf7fa86700 1 -- 172.21.15.90:0/3953547519 >> 172.21.15.90:0/3953547519 conn(0x7fbf780fd2b0 msgr2=0x7fbf781116d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:20:30.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.988+0000 7fbf7fa86700 1 -- 172.21.15.90:0/3953547519 shutdown_connections 2024-04-14T15:20:30.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.988+0000 7fbf7fa86700 1 -- 172.21.15.90:0/3953547519 wait complete. 2024-04-14T15:20:30.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:29.988+0000 7fbf7fa86700 1 librados: shutdown 2024-04-14T15:20:30.314 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is unset 2024-04-14T15:20:30.315 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:29 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-14T15:20:30.315 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:29 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-81","var": "dedup_tier","val": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:20:30.315 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:29 smithi092 ceph-mon[33966]: osdmap e504: 8 total, 8 up, 8 in 2024-04-14T15:20:30.315 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:29 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-81","var": "dedup_tier","val": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:20:30.315 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:29 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3953547519' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:20:30.315 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:29 smithi092 ceph-mon[33966]: from='client.49324 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:20:30.368 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:20:30.392 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:30.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:30.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:20:30.493 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:30.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:30.518 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:30.518 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:30.619 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:20:30.642 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:30.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:30.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:20:30.744 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:20:30.769 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:30.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:30.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:20:30.871 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:20:30.895 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:30.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:30.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:20:30.998 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:20:31.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:31.022 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:31.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:20:31.123 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:20:31.147 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:31.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:31.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:20:31.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:30 smithi090 ceph-mon[41670]: pgmap v731: 258 pgs: 1 remapped+peering, 15 unknown, 242 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:20:31.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:30 smithi090 ceph-mon[41670]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-14T15:20:31.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:30 smithi090 ceph-mon[41670]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:20:31.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:30 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-81","var": "dedup_tier","val": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:20:31.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:30 smithi090 ceph-mon[41670]: from='client.49324 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-14T15:20:31.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:30 smithi090 ceph-mon[41670]: osdmap e505: 8 total, 8 up, 8 in 2024-04-14T15:20:31.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:30 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-14T15:20:31.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:30 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-14T15:20:31.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:30 smithi090 ceph-mon[41670]: 6.16 scrub starts 2024-04-14T15:20:31.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:30 smithi090 ceph-mon[41670]: 6.16 scrub ok 2024-04-14T15:20:31.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:30 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-14T15:20:31.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:30 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-14T15:20:31.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:30 smithi090 ceph-mon[41670]: osdmap e506: 8 total, 8 up, 8 in 2024-04-14T15:20:31.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:30 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-14T15:20:31.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:30 smithi090 ceph-mon[30399]: pgmap v731: 258 pgs: 1 remapped+peering, 15 unknown, 242 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:20:31.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:30 smithi090 ceph-mon[30399]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-14T15:20:31.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:30 smithi090 ceph-mon[30399]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:20:31.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:30 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-81","var": "dedup_tier","val": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:20:31.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:30 smithi090 ceph-mon[30399]: from='client.49324 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-14T15:20:31.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:30 smithi090 ceph-mon[30399]: osdmap e505: 8 total, 8 up, 8 in 2024-04-14T15:20:31.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:30 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-14T15:20:31.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:30 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-14T15:20:31.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:30 smithi090 ceph-mon[30399]: 6.16 scrub starts 2024-04-14T15:20:31.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:30 smithi090 ceph-mon[30399]: 6.16 scrub ok 2024-04-14T15:20:31.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:30 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-14T15:20:31.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:30 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-14T15:20:31.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:30 smithi090 ceph-mon[30399]: osdmap e506: 8 total, 8 up, 8 in 2024-04-14T15:20:31.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:30 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-14T15:20:31.249 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:20:31.272 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:31.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:31.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:20:31.296 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:30 smithi092 ceph-mon[33966]: pgmap v731: 258 pgs: 1 remapped+peering, 15 unknown, 242 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:20:31.296 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:30 smithi092 ceph-mon[33966]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-14T15:20:31.296 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:30 smithi092 ceph-mon[33966]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:20:31.296 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:30 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-81","var": "dedup_tier","val": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:20:31.296 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:30 smithi092 ceph-mon[33966]: from='client.49324 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-14T15:20:31.296 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:30 smithi092 ceph-mon[33966]: osdmap e505: 8 total, 8 up, 8 in 2024-04-14T15:20:31.296 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:30 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-14T15:20:31.296 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:30 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-14T15:20:31.296 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:30 smithi092 ceph-mon[33966]: 6.16 scrub starts 2024-04-14T15:20:31.297 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:30 smithi092 ceph-mon[33966]: 6.16 scrub ok 2024-04-14T15:20:31.297 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:30 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-14T15:20:31.297 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:30 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-14T15:20:31.297 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:30 smithi092 ceph-mon[33966]: osdmap e506: 8 total, 8 up, 8 in 2024-04-14T15:20:31.297 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:30 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-14T15:20:31.374 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:20:31.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:31.398 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:31.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:20:31.500 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:31.523 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:31.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:31.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:31.625 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:20:31.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:31.650 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:31.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:20:31.751 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:20:31.774 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:31.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:31.775 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:20:31.827 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-04-14T15:20:31.827 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:20:31.876 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:20:31.899 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:31.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:31.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:20:32.001 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:20:32.025 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:32.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:32.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:20:32.127 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:20:32.150 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:32.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:32.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:20:32.199 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-14T15:20:32.252 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:20:32.275 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:32.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:32.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:20:32.377 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:20:32.402 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:32.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:32.403 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:20:32.430 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-14T15:20:32.442 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:20:32.504 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:32.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:32.529 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:32.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:32.629 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:20:32.653 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:32.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:32.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:20:32.668 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:20:32.755 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:20:32.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:32.780 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:32.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:20:32.881 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:20:32.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:32.906 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:32.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:20:32.973 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-14T15:20:33.006 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:20:33.039 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:33.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:33.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:20:33.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:32 smithi092 ceph-mon[33966]: pgmap v734: 258 pgs: 10 remapped+peering, 248 active+clean; 8.5 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 9.0 KiB/s rd, 55 KiB/s wr, 11 op/s; 39 KiB/s, 0 objects/s recovering 2024-04-14T15:20:33.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:32 smithi092 ceph-mon[33966]: 1.e scrub starts 2024-04-14T15:20:33.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:32 smithi092 ceph-mon[33966]: 1.e scrub ok 2024-04-14T15:20:33.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:32 smithi092 ceph-mon[33966]: 2.3 scrub starts 2024-04-14T15:20:33.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:32 smithi092 ceph-mon[33966]: 2.3 scrub ok 2024-04-14T15:20:33.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:32 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-14T15:20:33.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:32 smithi092 ceph-mon[33966]: osdmap e507: 8 total, 8 up, 8 in 2024-04-14T15:20:33.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:32 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-04-14T15:20:33.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:32 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-04-14T15:20:33.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:32 smithi092 ceph-mon[33966]: 2.13 scrub starts 2024-04-14T15:20:33.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:32 smithi092 ceph-mon[33966]: 2.13 scrub ok 2024-04-14T15:20:33.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:32 smithi092 ceph-mon[33966]: 1.1b scrub starts 2024-04-14T15:20:33.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:32 smithi092 ceph-mon[33966]: 1.1b scrub ok 2024-04-14T15:20:33.141 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:20:33.165 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:33.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:33.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:20:33.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:32 smithi090 ceph-mon[41670]: pgmap v734: 258 pgs: 10 remapped+peering, 248 active+clean; 8.5 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 9.0 KiB/s rd, 55 KiB/s wr, 11 op/s; 39 KiB/s, 0 objects/s recovering 2024-04-14T15:20:33.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:32 smithi090 ceph-mon[41670]: 1.e scrub starts 2024-04-14T15:20:33.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:32 smithi090 ceph-mon[41670]: 1.e scrub ok 2024-04-14T15:20:33.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:32 smithi090 ceph-mon[41670]: 2.3 scrub starts 2024-04-14T15:20:33.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:32 smithi090 ceph-mon[41670]: 2.3 scrub ok 2024-04-14T15:20:33.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:32 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-14T15:20:33.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:32 smithi090 ceph-mon[41670]: osdmap e507: 8 total, 8 up, 8 in 2024-04-14T15:20:33.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:32 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-04-14T15:20:33.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:32 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-04-14T15:20:33.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:32 smithi090 ceph-mon[41670]: 2.13 scrub starts 2024-04-14T15:20:33.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:32 smithi090 ceph-mon[41670]: 2.13 scrub ok 2024-04-14T15:20:33.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:32 smithi090 ceph-mon[41670]: 1.1b scrub starts 2024-04-14T15:20:33.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:32 smithi090 ceph-mon[41670]: 1.1b scrub ok 2024-04-14T15:20:33.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:32 smithi090 ceph-mon[30399]: pgmap v734: 258 pgs: 10 remapped+peering, 248 active+clean; 8.5 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 9.0 KiB/s rd, 55 KiB/s wr, 11 op/s; 39 KiB/s, 0 objects/s recovering 2024-04-14T15:20:33.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:32 smithi090 ceph-mon[30399]: 1.e scrub starts 2024-04-14T15:20:33.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:32 smithi090 ceph-mon[30399]: 1.e scrub ok 2024-04-14T15:20:33.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:32 smithi090 ceph-mon[30399]: 2.3 scrub starts 2024-04-14T15:20:33.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:32 smithi090 ceph-mon[30399]: 2.3 scrub ok 2024-04-14T15:20:33.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:32 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-14T15:20:33.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:32 smithi090 ceph-mon[30399]: osdmap e507: 8 total, 8 up, 8 in 2024-04-14T15:20:33.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:32 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-04-14T15:20:33.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:32 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-04-14T15:20:33.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:32 smithi090 ceph-mon[30399]: 2.13 scrub starts 2024-04-14T15:20:33.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:32 smithi090 ceph-mon[30399]: 2.13 scrub ok 2024-04-14T15:20:33.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:32 smithi090 ceph-mon[30399]: 1.1b scrub starts 2024-04-14T15:20:33.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:32 smithi090 ceph-mon[30399]: 1.1b scrub ok 2024-04-14T15:20:33.267 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:20:33.290 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:33.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:33.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:20:33.391 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:33.416 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:33.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:33.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:33.518 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:20:33.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:33.547 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:33.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:20:33.648 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:20:33.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:33.673 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:33.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:20:33.774 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:20:33.798 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:33.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:33.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:20:33.899 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:20:33.925 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:33.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:33.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:20:34.027 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:34.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:34.050 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:34.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:34.151 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:20:34.168 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:20:33 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:20:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:20:34.169 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:33 smithi090 ceph-mon[41670]: 1.19 scrub starts 2024-04-14T15:20:34.169 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:33 smithi090 ceph-mon[41670]: 1.19 scrub ok 2024-04-14T15:20:34.169 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:33 smithi090 ceph-mon[41670]: 1.15 scrub starts 2024-04-14T15:20:34.169 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:33 smithi090 ceph-mon[41670]: 1.15 scrub ok 2024-04-14T15:20:34.169 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:33 smithi090 ceph-mon[41670]: 1.1a scrub starts 2024-04-14T15:20:34.169 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:33 smithi090 ceph-mon[41670]: 1.1a scrub ok 2024-04-14T15:20:34.169 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:33 smithi090 ceph-mon[41670]: 1.17 scrub starts 2024-04-14T15:20:34.169 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:33 smithi090 ceph-mon[41670]: 1.17 scrub ok 2024-04-14T15:20:34.169 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:33 smithi090 ceph-mon[41670]: 1.1c scrub starts 2024-04-14T15:20:34.169 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:33 smithi090 ceph-mon[41670]: 1.1c scrub ok 2024-04-14T15:20:34.169 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:33 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-81","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-04-14T15:20:34.169 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:33 smithi090 ceph-mon[41670]: osdmap e508: 8 total, 8 up, 8 in 2024-04-14T15:20:34.169 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:33 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-04-14T15:20:34.169 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:33 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-04-14T15:20:34.169 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:33 smithi090 ceph-mon[41670]: 4.b scrub starts 2024-04-14T15:20:34.170 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:33 smithi090 ceph-mon[41670]: 4.b scrub ok 2024-04-14T15:20:34.170 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:33 smithi090 ceph-mon[30399]: 1.19 scrub starts 2024-04-14T15:20:34.170 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:33 smithi090 ceph-mon[30399]: 1.19 scrub ok 2024-04-14T15:20:34.170 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:33 smithi090 ceph-mon[30399]: 1.15 scrub starts 2024-04-14T15:20:34.170 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:33 smithi090 ceph-mon[30399]: 1.15 scrub ok 2024-04-14T15:20:34.170 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:33 smithi090 ceph-mon[30399]: 1.1a scrub starts 2024-04-14T15:20:34.170 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:33 smithi090 ceph-mon[30399]: 1.1a scrub ok 2024-04-14T15:20:34.170 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:33 smithi090 ceph-mon[30399]: 1.17 scrub starts 2024-04-14T15:20:34.170 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:33 smithi090 ceph-mon[30399]: 1.17 scrub ok 2024-04-14T15:20:34.170 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:33 smithi090 ceph-mon[30399]: 1.1c scrub starts 2024-04-14T15:20:34.170 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:33 smithi090 ceph-mon[30399]: 1.1c scrub ok 2024-04-14T15:20:34.170 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:33 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-81","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-04-14T15:20:34.171 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:33 smithi090 ceph-mon[30399]: osdmap e508: 8 total, 8 up, 8 in 2024-04-14T15:20:34.171 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:33 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-04-14T15:20:34.171 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:33 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-04-14T15:20:34.171 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:33 smithi090 ceph-mon[30399]: 4.b scrub starts 2024-04-14T15:20:34.171 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:33 smithi090 ceph-mon[30399]: 4.b scrub ok 2024-04-14T15:20:34.175 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:34.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:34.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:20:34.196 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:20:34.259 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:33 smithi092 ceph-mon[33966]: 1.19 scrub starts 2024-04-14T15:20:34.259 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:33 smithi092 ceph-mon[33966]: 1.19 scrub ok 2024-04-14T15:20:34.259 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:33 smithi092 ceph-mon[33966]: 1.15 scrub starts 2024-04-14T15:20:34.259 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:33 smithi092 ceph-mon[33966]: 1.15 scrub ok 2024-04-14T15:20:34.259 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:33 smithi092 ceph-mon[33966]: 1.1a scrub starts 2024-04-14T15:20:34.259 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:33 smithi092 ceph-mon[33966]: 1.1a scrub ok 2024-04-14T15:20:34.259 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:33 smithi092 ceph-mon[33966]: 1.17 scrub starts 2024-04-14T15:20:34.259 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:33 smithi092 ceph-mon[33966]: 1.17 scrub ok 2024-04-14T15:20:34.259 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:33 smithi092 ceph-mon[33966]: 1.1c scrub starts 2024-04-14T15:20:34.259 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:33 smithi092 ceph-mon[33966]: 1.1c scrub ok 2024-04-14T15:20:34.259 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:33 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-81","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-04-14T15:20:34.259 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:33 smithi092 ceph-mon[33966]: osdmap e508: 8 total, 8 up, 8 in 2024-04-14T15:20:34.260 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:33 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-04-14T15:20:34.260 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:33 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-04-14T15:20:34.260 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:33 smithi092 ceph-mon[33966]: 4.b scrub starts 2024-04-14T15:20:34.260 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:33 smithi092 ceph-mon[33966]: 4.b scrub ok 2024-04-14T15:20:34.277 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:20:34.300 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:34.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:34.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:20:34.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.380+0000 7eff353b5700 1 -- 172.21.15.90:0/992343993 >> v1:172.21.15.90:6789/0 conn(0x7eff30108980 legacy=0x7eff30108d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:34.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.380+0000 7eff353b5700 1 -- 172.21.15.90:0/992343993 shutdown_connections 2024-04-14T15:20:34.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.380+0000 7eff353b5700 1 -- 172.21.15.90:0/992343993 >> 172.21.15.90:0/992343993 conn(0x7eff30076aa0 msgr2=0x7eff30076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:20:34.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.380+0000 7eff353b5700 1 -- 172.21.15.90:0/992343993 shutdown_connections 2024-04-14T15:20:34.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.380+0000 7eff353b5700 1 -- 172.21.15.90:0/992343993 wait complete. 2024-04-14T15:20:34.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.381+0000 7eff353b5700 1 librados: starting msgr at 2024-04-14T15:20:34.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.381+0000 7eff353b5700 1 librados: starting objecter 2024-04-14T15:20:34.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.381+0000 7eff353b5700 1 Processor -- start 2024-04-14T15:20:34.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.381+0000 7eff353b5700 1 -- start start 2024-04-14T15:20:34.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.381+0000 7eff353b5700 1 librados: setting wanted keys 2024-04-14T15:20:34.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.381+0000 7eff353b5700 1 librados: calling monclient init 2024-04-14T15:20:34.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.381+0000 7eff353b5700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7eff30108130 con 0x7eff30108980 2024-04-14T15:20:34.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.381+0000 7eff353b5700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7eff301a8f00 con 0x7eff3010f2f0 2024-04-14T15:20:34.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.381+0000 7eff353b5700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7eff301aa0e0 con 0x7eff3010b6f0 2024-04-14T15:20:34.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.381+0000 7eff2effd700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7eff30108980 0x7eff30104f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:59434/0 (socket says 172.21.15.90:59434) 2024-04-14T15:20:34.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.381+0000 7eff2effd700 1 -- 172.21.15.90:0/546383551 learned_addr learned my addr 172.21.15.90:0/546383551 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:20:34.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.381+0000 7eff17fff700 10 client.?.objecter ms_handle_connect 0x7eff30108980 2024-04-14T15:20:34.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.381+0000 7eff17fff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:20:34.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.381+0000 7eff17fff700 10 client.?.objecter ms_handle_connect 0x7eff3010b6f0 2024-04-14T15:20:34.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.381+0000 7eff17fff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:20:34.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.381+0000 7eff17fff700 10 client.?.objecter ms_handle_connect 0x7eff3010f2f0 2024-04-14T15:20:34.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.381+0000 7eff17fff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:20:34.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.382+0000 7eff17fff700 1 -- 172.21.15.90:0/546383551 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1332060008 0 0) 0x7eff30108130 con 0x7eff30108980 2024-04-14T15:20:34.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.382+0000 7eff17fff700 1 -- 172.21.15.90:0/546383551 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff180035b0 con 0x7eff30108980 2024-04-14T15:20:34.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.382+0000 7eff17fff700 1 -- 172.21.15.90:0/546383551 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3685112289 0 0) 0x7eff301aa0e0 con 0x7eff3010b6f0 2024-04-14T15:20:34.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.382+0000 7eff17fff700 1 -- 172.21.15.90:0/546383551 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff30108130 con 0x7eff3010b6f0 2024-04-14T15:20:34.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.382+0000 7eff17fff700 1 -- 172.21.15.90:0/546383551 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1299564886 0 0) 0x7eff301a8f00 con 0x7eff3010f2f0 2024-04-14T15:20:34.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.382+0000 7eff17fff700 1 -- 172.21.15.90:0/546383551 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff301aa0e0 con 0x7eff3010f2f0 2024-04-14T15:20:34.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.382+0000 7eff17fff700 1 -- 172.21.15.90:0/546383551 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1674006360 0 0) 0x7eff180035b0 con 0x7eff30108980 2024-04-14T15:20:34.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.382+0000 7eff17fff700 1 -- 172.21.15.90:0/546383551 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7eff301a8f00 con 0x7eff30108980 2024-04-14T15:20:34.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.382+0000 7eff17fff700 1 -- 172.21.15.90:0/546383551 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7eff20001430 con 0x7eff30108980 2024-04-14T15:20:34.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.382+0000 7eff17fff700 1 -- 172.21.15.90:0/546383551 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1951767938 0 0) 0x7eff301aa0e0 con 0x7eff3010f2f0 2024-04-14T15:20:34.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.382+0000 7eff17fff700 1 -- 172.21.15.90:0/546383551 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7eff180035b0 con 0x7eff3010f2f0 2024-04-14T15:20:34.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.382+0000 7eff17fff700 1 -- 172.21.15.90:0/546383551 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7eff24001850 con 0x7eff3010f2f0 2024-04-14T15:20:34.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.382+0000 7eff17fff700 1 -- 172.21.15.90:0/546383551 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2007479988 0 0) 0x7eff30108130 con 0x7eff3010b6f0 2024-04-14T15:20:34.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.382+0000 7eff17fff700 1 -- 172.21.15.90:0/546383551 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7eff301aa0e0 con 0x7eff3010b6f0 2024-04-14T15:20:34.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.382+0000 7eff17fff700 1 -- 172.21.15.90:0/546383551 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7eff28001b30 con 0x7eff3010b6f0 2024-04-14T15:20:34.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.382+0000 7eff17fff700 1 -- 172.21.15.90:0/546383551 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2555259398 0 0) 0x7eff301a8f00 con 0x7eff30108980 2024-04-14T15:20:34.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.382+0000 7eff17fff700 1 -- 172.21.15.90:0/546383551 >> v1:172.21.15.90:6790/0 conn(0x7eff3010b6f0 legacy=0x7eff30105690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:34.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.382+0000 7eff17fff700 1 -- 172.21.15.90:0/546383551 >> v1:172.21.15.92:6789/0 conn(0x7eff3010f2f0 legacy=0x7eff301a67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:34.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.382+0000 7eff17fff700 1 -- 172.21.15.90:0/546383551 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff301ab2c0 con 0x7eff30108980 2024-04-14T15:20:34.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.382+0000 7eff353b5700 1 -- 172.21.15.90:0/546383551 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7eff301a7f50 con 0x7eff30108980 2024-04-14T15:20:34.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.382+0000 7eff353b5700 10 client.37497.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:20:34.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.382+0000 7eff353b5700 1 -- 172.21.15.90:0/546383551 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7eff301a83b0 con 0x7eff30108980 2024-04-14T15:20:34.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.382+0000 7eff353b5700 1 librados: init done 2024-04-14T15:20:34.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.382+0000 7eff17fff700 1 -- 172.21.15.90:0/546383551 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7eff20001b30 con 0x7eff30108980 2024-04-14T15:20:34.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.382+0000 7eff17fff700 1 -- 172.21.15.90:0/546383551 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7eff20004e30 con 0x7eff30108980 2024-04-14T15:20:34.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.383+0000 7eff353b5700 1 -- 172.21.15.90:0/546383551 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7eff1c005100 con 0x7eff30108980 2024-04-14T15:20:34.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.384+0000 7eff17fff700 1 -- 172.21.15.90:0/546383551 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7eff20005070 con 0x7eff30108980 2024-04-14T15:20:34.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.384+0000 7eff17fff700 1 -- 172.21.15.90:0/546383551 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(509..509 src has 51..509) ==== 8264+0+0 (unknown 2103874551 0 0) 0x7eff2008ab00 con 0x7eff30108980 2024-04-14T15:20:34.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.384+0000 7eff17fff700 10 client.37497.objecter ms_dispatch 0x7eff30001550 osd_map(509..509 src has 51..509) 2024-04-14T15:20:34.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.384+0000 7eff17fff700 3 client.37497.objecter handle_osd_map got epochs [509,509] > 0 2024-04-14T15:20:34.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.384+0000 7eff17fff700 3 client.37497.objecter handle_osd_map decoding full epoch 509 2024-04-14T15:20:34.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.384+0000 7eff17fff700 20 client.37497.objecter dump_active .. 0 homeless 2024-04-14T15:20:34.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.384+0000 7eff17fff700 1 -- 172.21.15.90:0/546383551 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v509) ==== 54+0+16 (unknown 3832808871 0 2420000192) 0x7eff2008af10 con 0x7eff30108980 2024-04-14T15:20:34.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.384+0000 7eff17fff700 10 client.37497.objecter ms_handle_connect 0x7eff18071380 2024-04-14T15:20:34.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.385+0000 7eff353b5700 10 client.37497.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:20:34.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.385+0000 7eff353b5700 15 client.37497.objecter _session_command_op_assign -1 1 2024-04-14T15:20:34.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.385+0000 7eff353b5700 20 client.37497.objecter _get_session s=0x7eff1c0053e0 osd=0 3 2024-04-14T15:20:34.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.385+0000 7eff353b5700 20 client.37497.objecter put_session s=0x7eff1c0053e0 osd=0 3 2024-04-14T15:20:34.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.385+0000 7eff353b5700 20 client.37497.objecter _get_session s=0x7eff1c0053e0 osd=0 3 2024-04-14T15:20:34.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.385+0000 7eff353b5700 15 client.37497.objecter _session_command_op_remove -1 1 2024-04-14T15:20:34.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.385+0000 7eff353b5700 20 client.37497.objecter get_session s=0x7eff1c0053e0 osd=0 3 2024-04-14T15:20:34.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.385+0000 7eff353b5700 15 client.37497.objecter _session_command_op_assign 0 1 2024-04-14T15:20:34.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.385+0000 7eff353b5700 20 client.37497.objecter put_session s=0x7eff1c0053e0 osd=0 4 2024-04-14T15:20:34.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.385+0000 7eff353b5700 10 client.37497.objecter _send_command 1 2024-04-14T15:20:34.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.385+0000 7eff353b5700 1 -- 172.21.15.90:0/546383551 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7eff1c004b70 con 0x7eff1c004730 2024-04-14T15:20:34.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.385+0000 7eff17fff700 10 client.37497.objecter ms_handle_connect 0x7eff1c004730 2024-04-14T15:20:34.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.385+0000 7eff17fff700 1 -- 172.21.15.90:0/546383551 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7eff1c004b70 con 0x7eff1c004730 2024-04-14T15:20:34.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.385+0000 7eff17fff700 10 client.37497.objecter ms_dispatch 0x7eff30001550 command_reply(tid 1: 0 ) 2024-04-14T15:20:34.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.385+0000 7eff17fff700 10 client.37497.objecter _finish_command 1 = system:0 2024-04-14T15:20:34.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.385+0000 7eff17fff700 20 client.37497.objecter put_session s=0x7eff1c0053e0 osd=0 4 2024-04-14T15:20:34.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.385+0000 7eff17fff700 15 client.37497.objecter _session_command_op_remove 0 1 2024-04-14T15:20:34.402 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:20:34.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.403+0000 7eff353b5700 10 client.37497.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:20:34.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.403+0000 7eff353b5700 15 client.37497.objecter _session_command_op_assign -1 2 2024-04-14T15:20:34.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.403+0000 7eff353b5700 20 client.37497.objecter _get_session s=0x7eff1c0053e0 osd=0 3 2024-04-14T15:20:34.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.403+0000 7eff353b5700 20 client.37497.objecter put_session s=0x7eff1c0053e0 osd=0 3 2024-04-14T15:20:34.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.403+0000 7eff353b5700 20 client.37497.objecter _get_session s=0x7eff1c0053e0 osd=0 3 2024-04-14T15:20:34.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.403+0000 7eff353b5700 15 client.37497.objecter _session_command_op_remove -1 2 2024-04-14T15:20:34.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.403+0000 7eff353b5700 20 client.37497.objecter get_session s=0x7eff1c0053e0 osd=0 3 2024-04-14T15:20:34.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.403+0000 7eff353b5700 15 client.37497.objecter _session_command_op_assign 0 2 2024-04-14T15:20:34.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.403+0000 7eff353b5700 20 client.37497.objecter put_session s=0x7eff1c0053e0 osd=0 4 2024-04-14T15:20:34.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.403+0000 7eff353b5700 10 client.37497.objecter _send_command 2 2024-04-14T15:20:34.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.403+0000 7eff353b5700 1 -- 172.21.15.90:0/546383551 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7eff1c0079f0 con 0x7eff1c004730 2024-04-14T15:20:34.406 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:20:34.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.405+0000 7eff17fff700 1 -- 172.21.15.90:0/546383551 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7eff1c0079f0 con 0x7eff1c004730 2024-04-14T15:20:34.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.405+0000 7eff17fff700 10 client.37497.objecter ms_dispatch 0x7eff30001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:20:34.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.405+0000 7eff17fff700 10 client.37497.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:20:34.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.405+0000 7eff17fff700 20 client.37497.objecter put_session s=0x7eff1c0053e0 osd=0 4 2024-04-14T15:20:34.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.405+0000 7eff17fff700 15 client.37497.objecter _session_command_op_remove 0 2 2024-04-14T15:20:34.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.405+0000 7eff353b5700 10 client.37497.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:20:34.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.405+0000 7eff353b5700 15 client.37497.objecter _session_command_op_assign -1 3 2024-04-14T15:20:34.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.405+0000 7eff353b5700 20 client.37497.objecter _get_session s=0x7eff1c004130 osd=1 3 2024-04-14T15:20:34.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.405+0000 7eff353b5700 20 client.37497.objecter put_session s=0x7eff1c004130 osd=1 3 2024-04-14T15:20:34.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.405+0000 7eff353b5700 20 client.37497.objecter _get_session s=0x7eff1c004130 osd=1 3 2024-04-14T15:20:34.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.405+0000 7eff353b5700 15 client.37497.objecter _session_command_op_remove -1 3 2024-04-14T15:20:34.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.405+0000 7eff353b5700 20 client.37497.objecter get_session s=0x7eff1c004130 osd=1 3 2024-04-14T15:20:34.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.405+0000 7eff353b5700 15 client.37497.objecter _session_command_op_assign 1 3 2024-04-14T15:20:34.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.405+0000 7eff353b5700 20 client.37497.objecter put_session s=0x7eff1c004130 osd=1 4 2024-04-14T15:20:34.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.405+0000 7eff353b5700 10 client.37497.objecter _send_command 3 2024-04-14T15:20:34.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.405+0000 7eff353b5700 1 -- 172.21.15.90:0/546383551 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7eff1c0022a0 con 0x7eff1c004310 2024-04-14T15:20:34.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.405+0000 7eff17fff700 10 client.37497.objecter ms_handle_connect 0x7eff1c004310 2024-04-14T15:20:34.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.406+0000 7eff17fff700 1 -- 172.21.15.90:0/546383551 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7eff1c0022a0 con 0x7eff1c004310 2024-04-14T15:20:34.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.406+0000 7eff17fff700 10 client.37497.objecter ms_dispatch 0x7eff30001550 command_reply(tid 3: 0 ) 2024-04-14T15:20:34.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.406+0000 7eff17fff700 10 client.37497.objecter _finish_command 3 = system:0 2024-04-14T15:20:34.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.406+0000 7eff17fff700 20 client.37497.objecter put_session s=0x7eff1c004130 osd=1 4 2024-04-14T15:20:34.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.406+0000 7eff17fff700 15 client.37497.objecter _session_command_op_remove 1 3 2024-04-14T15:20:34.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.423+0000 7eff353b5700 10 client.37497.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:20:34.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.423+0000 7eff353b5700 15 client.37497.objecter _session_command_op_assign -1 4 2024-04-14T15:20:34.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.423+0000 7eff353b5700 20 client.37497.objecter _get_session s=0x7eff1c004130 osd=1 3 2024-04-14T15:20:34.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.423+0000 7eff353b5700 20 client.37497.objecter put_session s=0x7eff1c004130 osd=1 3 2024-04-14T15:20:34.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.423+0000 7eff353b5700 20 client.37497.objecter _get_session s=0x7eff1c004130 osd=1 3 2024-04-14T15:20:34.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.423+0000 7eff353b5700 15 client.37497.objecter _session_command_op_remove -1 4 2024-04-14T15:20:34.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.423+0000 7eff353b5700 20 client.37497.objecter get_session s=0x7eff1c004130 osd=1 3 2024-04-14T15:20:34.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.423+0000 7eff353b5700 15 client.37497.objecter _session_command_op_assign 1 4 2024-04-14T15:20:34.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.423+0000 7eff353b5700 20 client.37497.objecter put_session s=0x7eff1c004130 osd=1 4 2024-04-14T15:20:34.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.423+0000 7eff353b5700 10 client.37497.objecter _send_command 4 2024-04-14T15:20:34.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.423+0000 7eff353b5700 1 -- 172.21.15.90:0/546383551 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7eff1c001720 con 0x7eff1c004310 2024-04-14T15:20:34.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.425+0000 7eff17fff700 1 -- 172.21.15.90:0/546383551 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7eff1c001720 con 0x7eff1c004310 2024-04-14T15:20:34.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.425+0000 7eff17fff700 10 client.37497.objecter ms_dispatch 0x7eff30001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:20:34.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.425+0000 7eff17fff700 10 client.37497.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:20:34.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.425+0000 7eff17fff700 20 client.37497.objecter put_session s=0x7eff1c004130 osd=1 4 2024-04-14T15:20:34.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.425+0000 7eff17fff700 15 client.37497.objecter _session_command_op_remove 1 4 2024-04-14T15:20:34.426 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:20:34.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.425+0000 7eff353b5700 10 client.37497.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:20:34.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.425+0000 7eff353b5700 15 client.37497.objecter _session_command_op_assign -1 5 2024-04-14T15:20:34.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.425+0000 7eff353b5700 20 client.37497.objecter _get_session s=0x7eff1c001980 osd=2 3 2024-04-14T15:20:34.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.425+0000 7eff353b5700 20 client.37497.objecter put_session s=0x7eff1c001980 osd=2 3 2024-04-14T15:20:34.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.425+0000 7eff353b5700 20 client.37497.objecter _get_session s=0x7eff1c001980 osd=2 3 2024-04-14T15:20:34.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.425+0000 7eff353b5700 15 client.37497.objecter _session_command_op_remove -1 5 2024-04-14T15:20:34.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.425+0000 7eff353b5700 20 client.37497.objecter get_session s=0x7eff1c001980 osd=2 3 2024-04-14T15:20:34.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.425+0000 7eff353b5700 15 client.37497.objecter _session_command_op_assign 2 5 2024-04-14T15:20:34.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.425+0000 7eff353b5700 20 client.37497.objecter put_session s=0x7eff1c001980 osd=2 4 2024-04-14T15:20:34.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.425+0000 7eff353b5700 10 client.37497.objecter _send_command 5 2024-04-14T15:20:34.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.425+0000 7eff353b5700 1 -- 172.21.15.90:0/546383551 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7eff1c0110e0 con 0x7eff1c00d5d0 2024-04-14T15:20:34.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.426+0000 7eff17fff700 10 client.37497.objecter ms_handle_connect 0x7eff1c00d5d0 2024-04-14T15:20:34.428 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:34.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.426+0000 7eff17fff700 1 -- 172.21.15.90:0/546383551 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7eff1c0110e0 con 0x7eff1c00d5d0 2024-04-14T15:20:34.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.426+0000 7eff17fff700 10 client.37497.objecter ms_dispatch 0x7eff30001550 command_reply(tid 5: 0 ) 2024-04-14T15:20:34.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.426+0000 7eff17fff700 10 client.37497.objecter _finish_command 5 = system:0 2024-04-14T15:20:34.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.426+0000 7eff17fff700 20 client.37497.objecter put_session s=0x7eff1c001980 osd=2 4 2024-04-14T15:20:34.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.426+0000 7eff17fff700 15 client.37497.objecter _session_command_op_remove 2 5 2024-04-14T15:20:34.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:34.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:20:34.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.443+0000 7eff353b5700 10 client.37497.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:20:34.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.443+0000 7eff353b5700 15 client.37497.objecter _session_command_op_assign -1 6 2024-04-14T15:20:34.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.443+0000 7eff353b5700 20 client.37497.objecter _get_session s=0x7eff1c001980 osd=2 3 2024-04-14T15:20:34.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.443+0000 7eff353b5700 20 client.37497.objecter put_session s=0x7eff1c001980 osd=2 3 2024-04-14T15:20:34.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.443+0000 7eff353b5700 20 client.37497.objecter _get_session s=0x7eff1c001980 osd=2 3 2024-04-14T15:20:34.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.443+0000 7eff353b5700 15 client.37497.objecter _session_command_op_remove -1 6 2024-04-14T15:20:34.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.443+0000 7eff353b5700 20 client.37497.objecter get_session s=0x7eff1c001980 osd=2 3 2024-04-14T15:20:34.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.443+0000 7eff353b5700 15 client.37497.objecter _session_command_op_assign 2 6 2024-04-14T15:20:34.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.443+0000 7eff353b5700 20 client.37497.objecter put_session s=0x7eff1c001980 osd=2 4 2024-04-14T15:20:34.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.443+0000 7eff353b5700 10 client.37497.objecter _send_command 6 2024-04-14T15:20:34.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.443+0000 7eff353b5700 1 -- 172.21.15.90:0/546383551 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7eff1c008100 con 0x7eff1c00d5d0 2024-04-14T15:20:34.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.444+0000 7eff17fff700 1 -- 172.21.15.90:0/546383551 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7eff1c008100 con 0x7eff1c00d5d0 2024-04-14T15:20:34.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.444+0000 7eff17fff700 10 client.37497.objecter ms_dispatch 0x7eff30001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:20:34.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.444+0000 7eff17fff700 10 client.37497.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:20:34.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.444+0000 7eff17fff700 20 client.37497.objecter put_session s=0x7eff1c001980 osd=2 4 2024-04-14T15:20:34.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.444+0000 7eff17fff700 15 client.37497.objecter _session_command_op_remove 2 6 2024-04-14T15:20:34.445 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:20:34.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.445+0000 7eff353b5700 10 client.37497.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:20:34.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.445+0000 7eff353b5700 15 client.37497.objecter _session_command_op_assign -1 7 2024-04-14T15:20:34.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.445+0000 7eff353b5700 20 client.37497.objecter _get_session s=0x7eff1c008390 osd=3 3 2024-04-14T15:20:34.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.445+0000 7eff353b5700 20 client.37497.objecter put_session s=0x7eff1c008390 osd=3 3 2024-04-14T15:20:34.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.445+0000 7eff353b5700 20 client.37497.objecter _get_session s=0x7eff1c008390 osd=3 3 2024-04-14T15:20:34.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.445+0000 7eff353b5700 15 client.37497.objecter _session_command_op_remove -1 7 2024-04-14T15:20:34.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.445+0000 7eff353b5700 20 client.37497.objecter get_session s=0x7eff1c008390 osd=3 3 2024-04-14T15:20:34.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.445+0000 7eff353b5700 15 client.37497.objecter _session_command_op_assign 3 7 2024-04-14T15:20:34.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.445+0000 7eff353b5700 20 client.37497.objecter put_session s=0x7eff1c008390 osd=3 4 2024-04-14T15:20:34.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.445+0000 7eff353b5700 10 client.37497.objecter _send_command 7 2024-04-14T15:20:34.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.445+0000 7eff353b5700 1 -- 172.21.15.90:0/546383551 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7eff1c012fd0 con 0x7eff1c012410 2024-04-14T15:20:34.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.445+0000 7eff17fff700 10 client.37497.objecter ms_handle_connect 0x7eff1c012410 2024-04-14T15:20:34.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.446+0000 7eff17fff700 1 -- 172.21.15.90:0/546383551 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7eff1c012fd0 con 0x7eff1c012410 2024-04-14T15:20:34.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.446+0000 7eff17fff700 10 client.37497.objecter ms_dispatch 0x7eff30001550 command_reply(tid 7: 0 ) 2024-04-14T15:20:34.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.446+0000 7eff17fff700 10 client.37497.objecter _finish_command 7 = system:0 2024-04-14T15:20:34.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.446+0000 7eff17fff700 20 client.37497.objecter put_session s=0x7eff1c008390 osd=3 4 2024-04-14T15:20:34.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.446+0000 7eff17fff700 15 client.37497.objecter _session_command_op_remove 3 7 2024-04-14T15:20:34.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.462+0000 7eff353b5700 10 client.37497.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:20:34.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.462+0000 7eff353b5700 15 client.37497.objecter _session_command_op_assign -1 8 2024-04-14T15:20:34.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.462+0000 7eff353b5700 20 client.37497.objecter _get_session s=0x7eff1c008390 osd=3 3 2024-04-14T15:20:34.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.462+0000 7eff353b5700 20 client.37497.objecter put_session s=0x7eff1c008390 osd=3 3 2024-04-14T15:20:34.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.462+0000 7eff353b5700 20 client.37497.objecter _get_session s=0x7eff1c008390 osd=3 3 2024-04-14T15:20:34.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.462+0000 7eff353b5700 15 client.37497.objecter _session_command_op_remove -1 8 2024-04-14T15:20:34.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.462+0000 7eff353b5700 20 client.37497.objecter get_session s=0x7eff1c008390 osd=3 3 2024-04-14T15:20:34.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.462+0000 7eff353b5700 15 client.37497.objecter _session_command_op_assign 3 8 2024-04-14T15:20:34.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.462+0000 7eff353b5700 20 client.37497.objecter put_session s=0x7eff1c008390 osd=3 4 2024-04-14T15:20:34.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.462+0000 7eff353b5700 10 client.37497.objecter _send_command 8 2024-04-14T15:20:34.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.462+0000 7eff353b5700 1 -- 172.21.15.90:0/546383551 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7eff1c013980 con 0x7eff1c012410 2024-04-14T15:20:34.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.464+0000 7eff17fff700 1 -- 172.21.15.90:0/546383551 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7eff1c013980 con 0x7eff1c012410 2024-04-14T15:20:34.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.464+0000 7eff17fff700 10 client.37497.objecter ms_dispatch 0x7eff30001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:20:34.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.464+0000 7eff17fff700 10 client.37497.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:20:34.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.464+0000 7eff17fff700 20 client.37497.objecter put_session s=0x7eff1c008390 osd=3 4 2024-04-14T15:20:34.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.464+0000 7eff17fff700 15 client.37497.objecter _session_command_op_remove 3 8 2024-04-14T15:20:34.465 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:20:34.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.464+0000 7eff353b5700 10 client.37497.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:20:34.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.464+0000 7eff353b5700 15 client.37497.objecter _session_command_op_assign -1 9 2024-04-14T15:20:34.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.464+0000 7eff353b5700 20 client.37497.objecter _get_session s=0x7eff1c013980 osd=4 3 2024-04-14T15:20:34.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.464+0000 7eff353b5700 20 client.37497.objecter put_session s=0x7eff1c013980 osd=4 3 2024-04-14T15:20:34.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.464+0000 7eff353b5700 20 client.37497.objecter _get_session s=0x7eff1c013980 osd=4 3 2024-04-14T15:20:34.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.464+0000 7eff353b5700 15 client.37497.objecter _session_command_op_remove -1 9 2024-04-14T15:20:34.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.464+0000 7eff353b5700 20 client.37497.objecter get_session s=0x7eff1c013980 osd=4 3 2024-04-14T15:20:34.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.464+0000 7eff353b5700 15 client.37497.objecter _session_command_op_assign 4 9 2024-04-14T15:20:34.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.464+0000 7eff353b5700 20 client.37497.objecter put_session s=0x7eff1c013980 osd=4 4 2024-04-14T15:20:34.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.464+0000 7eff353b5700 10 client.37497.objecter _send_command 9 2024-04-14T15:20:34.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.464+0000 7eff353b5700 1 -- 172.21.15.90:0/546383551 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7eff1c01a820 con 0x7eff1c016c40 2024-04-14T15:20:34.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.465+0000 7eff17fff700 10 client.37497.objecter ms_handle_connect 0x7eff1c016c40 2024-04-14T15:20:34.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.465+0000 7eff17fff700 1 -- 172.21.15.90:0/546383551 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7eff1c01a820 con 0x7eff1c016c40 2024-04-14T15:20:34.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.465+0000 7eff17fff700 10 client.37497.objecter ms_dispatch 0x7eff30001550 command_reply(tid 9: 0 ) 2024-04-14T15:20:34.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.465+0000 7eff17fff700 10 client.37497.objecter _finish_command 9 = system:0 2024-04-14T15:20:34.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.465+0000 7eff17fff700 20 client.37497.objecter put_session s=0x7eff1c013980 osd=4 4 2024-04-14T15:20:34.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.465+0000 7eff17fff700 15 client.37497.objecter _session_command_op_remove 4 9 2024-04-14T15:20:34.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.482+0000 7eff353b5700 10 client.37497.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:20:34.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.482+0000 7eff353b5700 15 client.37497.objecter _session_command_op_assign -1 10 2024-04-14T15:20:34.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.482+0000 7eff353b5700 20 client.37497.objecter _get_session s=0x7eff1c013980 osd=4 3 2024-04-14T15:20:34.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.482+0000 7eff353b5700 20 client.37497.objecter put_session s=0x7eff1c013980 osd=4 3 2024-04-14T15:20:34.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.482+0000 7eff353b5700 20 client.37497.objecter _get_session s=0x7eff1c013980 osd=4 3 2024-04-14T15:20:34.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.482+0000 7eff353b5700 15 client.37497.objecter _session_command_op_remove -1 10 2024-04-14T15:20:34.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.482+0000 7eff353b5700 20 client.37497.objecter get_session s=0x7eff1c013980 osd=4 3 2024-04-14T15:20:34.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.482+0000 7eff353b5700 15 client.37497.objecter _session_command_op_assign 4 10 2024-04-14T15:20:34.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.482+0000 7eff353b5700 20 client.37497.objecter put_session s=0x7eff1c013980 osd=4 4 2024-04-14T15:20:34.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.482+0000 7eff353b5700 10 client.37497.objecter _send_command 10 2024-04-14T15:20:34.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.482+0000 7eff353b5700 1 -- 172.21.15.90:0/546383551 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7eff1c013400 con 0x7eff1c016c40 2024-04-14T15:20:34.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.484+0000 7eff17fff700 1 -- 172.21.15.90:0/546383551 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7eff1c013400 con 0x7eff1c016c40 2024-04-14T15:20:34.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.484+0000 7eff17fff700 10 client.37497.objecter ms_dispatch 0x7eff30001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:20:34.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.484+0000 7eff17fff700 10 client.37497.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:20:34.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.484+0000 7eff17fff700 20 client.37497.objecter put_session s=0x7eff1c013980 osd=4 4 2024-04-14T15:20:34.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.484+0000 7eff17fff700 15 client.37497.objecter _session_command_op_remove 4 10 2024-04-14T15:20:34.485 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:20:34.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.484+0000 7eff353b5700 10 client.37497.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:20:34.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.484+0000 7eff353b5700 15 client.37497.objecter _session_command_op_assign -1 11 2024-04-14T15:20:34.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.484+0000 7eff353b5700 20 client.37497.objecter _get_session s=0x7eff1c00c760 osd=5 3 2024-04-14T15:20:34.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.484+0000 7eff353b5700 20 client.37497.objecter put_session s=0x7eff1c00c760 osd=5 3 2024-04-14T15:20:34.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.484+0000 7eff353b5700 20 client.37497.objecter _get_session s=0x7eff1c00c760 osd=5 3 2024-04-14T15:20:34.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.484+0000 7eff353b5700 15 client.37497.objecter _session_command_op_remove -1 11 2024-04-14T15:20:34.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.484+0000 7eff353b5700 20 client.37497.objecter get_session s=0x7eff1c00c760 osd=5 3 2024-04-14T15:20:34.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.484+0000 7eff353b5700 15 client.37497.objecter _session_command_op_assign 5 11 2024-04-14T15:20:34.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.484+0000 7eff353b5700 20 client.37497.objecter put_session s=0x7eff1c00c760 osd=5 4 2024-04-14T15:20:34.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.484+0000 7eff353b5700 10 client.37497.objecter _send_command 11 2024-04-14T15:20:34.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.484+0000 7eff353b5700 1 -- 172.21.15.90:0/546383551 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7eff1c01bfe0 con 0x7eff1c0131b0 2024-04-14T15:20:34.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.484+0000 7eff17fff700 10 client.37497.objecter ms_handle_connect 0x7eff1c0131b0 2024-04-14T15:20:34.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.485+0000 7eff17fff700 1 -- 172.21.15.90:0/546383551 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7eff280082e0 con 0x7eff1c0131b0 2024-04-14T15:20:34.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.485+0000 7eff17fff700 10 client.37497.objecter ms_dispatch 0x7eff30001550 command_reply(tid 11: 0 ) 2024-04-14T15:20:34.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.485+0000 7eff17fff700 10 client.37497.objecter _finish_command 11 = system:0 2024-04-14T15:20:34.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.485+0000 7eff17fff700 20 client.37497.objecter put_session s=0x7eff1c00c760 osd=5 4 2024-04-14T15:20:34.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.485+0000 7eff17fff700 15 client.37497.objecter _session_command_op_remove 5 11 2024-04-14T15:20:34.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.501+0000 7eff353b5700 10 client.37497.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:20:34.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.501+0000 7eff353b5700 15 client.37497.objecter _session_command_op_assign -1 12 2024-04-14T15:20:34.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.501+0000 7eff353b5700 20 client.37497.objecter _get_session s=0x7eff1c00c760 osd=5 3 2024-04-14T15:20:34.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.501+0000 7eff353b5700 20 client.37497.objecter put_session s=0x7eff1c00c760 osd=5 3 2024-04-14T15:20:34.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.501+0000 7eff353b5700 20 client.37497.objecter _get_session s=0x7eff1c00c760 osd=5 3 2024-04-14T15:20:34.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.501+0000 7eff353b5700 15 client.37497.objecter _session_command_op_remove -1 12 2024-04-14T15:20:34.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.501+0000 7eff353b5700 20 client.37497.objecter get_session s=0x7eff1c00c760 osd=5 3 2024-04-14T15:20:34.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.501+0000 7eff353b5700 15 client.37497.objecter _session_command_op_assign 5 12 2024-04-14T15:20:34.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.501+0000 7eff353b5700 20 client.37497.objecter put_session s=0x7eff1c00c760 osd=5 4 2024-04-14T15:20:34.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.501+0000 7eff353b5700 10 client.37497.objecter _send_command 12 2024-04-14T15:20:34.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.501+0000 7eff353b5700 1 -- 172.21.15.90:0/546383551 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7eff1c012040 con 0x7eff1c0131b0 2024-04-14T15:20:34.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.503+0000 7eff17fff700 1 -- 172.21.15.90:0/546383551 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7eff1c012040 con 0x7eff1c0131b0 2024-04-14T15:20:34.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.503+0000 7eff17fff700 10 client.37497.objecter ms_dispatch 0x7eff30001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:20:34.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.503+0000 7eff17fff700 10 client.37497.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:20:34.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.503+0000 7eff17fff700 20 client.37497.objecter put_session s=0x7eff1c00c760 osd=5 4 2024-04-14T15:20:34.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.503+0000 7eff17fff700 15 client.37497.objecter _session_command_op_remove 5 12 2024-04-14T15:20:34.504 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:20:34.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.504+0000 7eff353b5700 10 client.37497.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:20:34.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.504+0000 7eff353b5700 15 client.37497.objecter _session_command_op_assign -1 13 2024-04-14T15:20:34.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.504+0000 7eff353b5700 20 client.37497.objecter _get_session s=0x7eff1c012040 osd=6 3 2024-04-14T15:20:34.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.504+0000 7eff353b5700 20 client.37497.objecter put_session s=0x7eff1c012040 osd=6 3 2024-04-14T15:20:34.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.504+0000 7eff353b5700 20 client.37497.objecter _get_session s=0x7eff1c012040 osd=6 3 2024-04-14T15:20:34.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.504+0000 7eff353b5700 15 client.37497.objecter _session_command_op_remove -1 13 2024-04-14T15:20:34.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.504+0000 7eff353b5700 20 client.37497.objecter get_session s=0x7eff1c012040 osd=6 3 2024-04-14T15:20:34.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.504+0000 7eff353b5700 15 client.37497.objecter _session_command_op_assign 6 13 2024-04-14T15:20:34.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.504+0000 7eff353b5700 20 client.37497.objecter put_session s=0x7eff1c012040 osd=6 4 2024-04-14T15:20:34.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.504+0000 7eff353b5700 10 client.37497.objecter _send_command 13 2024-04-14T15:20:34.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.504+0000 7eff353b5700 1 -- 172.21.15.90:0/546383551 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7eff1c0233f0 con 0x7eff1c01f8e0 2024-04-14T15:20:34.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.504+0000 7eff17fff700 10 client.37497.objecter ms_handle_connect 0x7eff1c01f8e0 2024-04-14T15:20:34.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.505+0000 7eff17fff700 1 -- 172.21.15.90:0/546383551 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7eff1c0233f0 con 0x7eff1c01f8e0 2024-04-14T15:20:34.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.505+0000 7eff17fff700 10 client.37497.objecter ms_dispatch 0x7eff30001550 command_reply(tid 13: 0 ) 2024-04-14T15:20:34.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.505+0000 7eff17fff700 10 client.37497.objecter _finish_command 13 = system:0 2024-04-14T15:20:34.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.505+0000 7eff17fff700 20 client.37497.objecter put_session s=0x7eff1c012040 osd=6 4 2024-04-14T15:20:34.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.505+0000 7eff17fff700 15 client.37497.objecter _session_command_op_remove 6 13 2024-04-14T15:20:34.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.521+0000 7eff353b5700 10 client.37497.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:20:34.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.521+0000 7eff353b5700 15 client.37497.objecter _session_command_op_assign -1 14 2024-04-14T15:20:34.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.521+0000 7eff353b5700 20 client.37497.objecter _get_session s=0x7eff1c012040 osd=6 3 2024-04-14T15:20:34.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.521+0000 7eff353b5700 20 client.37497.objecter put_session s=0x7eff1c012040 osd=6 3 2024-04-14T15:20:34.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.521+0000 7eff353b5700 20 client.37497.objecter _get_session s=0x7eff1c012040 osd=6 3 2024-04-14T15:20:34.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.521+0000 7eff353b5700 15 client.37497.objecter _session_command_op_remove -1 14 2024-04-14T15:20:34.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.521+0000 7eff353b5700 20 client.37497.objecter get_session s=0x7eff1c012040 osd=6 3 2024-04-14T15:20:34.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.521+0000 7eff353b5700 15 client.37497.objecter _session_command_op_assign 6 14 2024-04-14T15:20:34.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.521+0000 7eff353b5700 20 client.37497.objecter put_session s=0x7eff1c012040 osd=6 4 2024-04-14T15:20:34.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.521+0000 7eff353b5700 10 client.37497.objecter _send_command 14 2024-04-14T15:20:34.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.521+0000 7eff353b5700 1 -- 172.21.15.90:0/546383551 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7eff1c0235d0 con 0x7eff1c01f8e0 2024-04-14T15:20:34.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.523+0000 7eff17fff700 1 -- 172.21.15.90:0/546383551 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7eff1c0235d0 con 0x7eff1c01f8e0 2024-04-14T15:20:34.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.523+0000 7eff17fff700 10 client.37497.objecter ms_dispatch 0x7eff30001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:20:34.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.523+0000 7eff17fff700 10 client.37497.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:20:34.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.523+0000 7eff17fff700 20 client.37497.objecter put_session s=0x7eff1c012040 osd=6 4 2024-04-14T15:20:34.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.523+0000 7eff17fff700 15 client.37497.objecter _session_command_op_remove 6 14 2024-04-14T15:20:34.524 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:20:34.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.523+0000 7eff353b5700 10 client.37497.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:20:34.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.523+0000 7eff353b5700 15 client.37497.objecter _session_command_op_assign -1 15 2024-04-14T15:20:34.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.523+0000 7eff353b5700 20 client.37497.objecter _get_session s=0x7eff1c0235d0 osd=7 3 2024-04-14T15:20:34.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.523+0000 7eff353b5700 20 client.37497.objecter put_session s=0x7eff1c0235d0 osd=7 3 2024-04-14T15:20:34.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.523+0000 7eff353b5700 20 client.37497.objecter _get_session s=0x7eff1c0235d0 osd=7 3 2024-04-14T15:20:34.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.523+0000 7eff353b5700 15 client.37497.objecter _session_command_op_remove -1 15 2024-04-14T15:20:34.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.523+0000 7eff353b5700 20 client.37497.objecter get_session s=0x7eff1c0235d0 osd=7 3 2024-04-14T15:20:34.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.523+0000 7eff353b5700 15 client.37497.objecter _session_command_op_assign 7 15 2024-04-14T15:20:34.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.523+0000 7eff353b5700 20 client.37497.objecter put_session s=0x7eff1c0235d0 osd=7 4 2024-04-14T15:20:34.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.523+0000 7eff353b5700 10 client.37497.objecter _send_command 15 2024-04-14T15:20:34.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.523+0000 7eff353b5700 1 -- 172.21.15.90:0/546383551 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7eff1c0248d0 con 0x7eff1c023cf0 2024-04-14T15:20:34.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.524+0000 7eff17fff700 10 client.37497.objecter ms_handle_connect 0x7eff1c023cf0 2024-04-14T15:20:34.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.524+0000 7eff17fff700 1 -- 172.21.15.90:0/546383551 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7eff1c0248d0 con 0x7eff1c023cf0 2024-04-14T15:20:34.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.524+0000 7eff17fff700 10 client.37497.objecter ms_dispatch 0x7eff30001550 command_reply(tid 15: 0 ) 2024-04-14T15:20:34.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.524+0000 7eff17fff700 10 client.37497.objecter _finish_command 15 = system:0 2024-04-14T15:20:34.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.524+0000 7eff17fff700 20 client.37497.objecter put_session s=0x7eff1c0235d0 osd=7 4 2024-04-14T15:20:34.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.524+0000 7eff17fff700 15 client.37497.objecter _session_command_op_remove 7 15 2024-04-14T15:20:34.530 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:20:34.533 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-04-14T15:20:34.533 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-04-14T15:20:34.533 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 0 2024-04-14T15:20:34.533 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-04-14T15:20:34.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.541+0000 7eff353b5700 10 client.37497.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:20:34.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.541+0000 7eff353b5700 15 client.37497.objecter _session_command_op_assign -1 16 2024-04-14T15:20:34.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.541+0000 7eff353b5700 20 client.37497.objecter _get_session s=0x7eff1c0235d0 osd=7 3 2024-04-14T15:20:34.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.541+0000 7eff353b5700 20 client.37497.objecter put_session s=0x7eff1c0235d0 osd=7 3 2024-04-14T15:20:34.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.541+0000 7eff353b5700 20 client.37497.objecter _get_session s=0x7eff1c0235d0 osd=7 3 2024-04-14T15:20:34.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.541+0000 7eff353b5700 15 client.37497.objecter _session_command_op_remove -1 16 2024-04-14T15:20:34.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.541+0000 7eff353b5700 20 client.37497.objecter get_session s=0x7eff1c0235d0 osd=7 3 2024-04-14T15:20:34.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.541+0000 7eff353b5700 15 client.37497.objecter _session_command_op_assign 7 16 2024-04-14T15:20:34.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.541+0000 7eff353b5700 20 client.37497.objecter put_session s=0x7eff1c0235d0 osd=7 4 2024-04-14T15:20:34.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.541+0000 7eff353b5700 10 client.37497.objecter _send_command 16 2024-04-14T15:20:34.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.541+0000 7eff353b5700 1 -- 172.21.15.90:0/546383551 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7eff1c0248d0 con 0x7eff1c023cf0 2024-04-14T15:20:34.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.543+0000 7eff17fff700 1 -- 172.21.15.90:0/546383551 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7eff1c0248d0 con 0x7eff1c023cf0 2024-04-14T15:20:34.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.543+0000 7eff17fff700 10 client.37497.objecter ms_dispatch 0x7eff30001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:20:34.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.543+0000 7eff17fff700 10 client.37497.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:20:34.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.543+0000 7eff17fff700 20 client.37497.objecter put_session s=0x7eff1c0235d0 osd=7 4 2024-04-14T15:20:34.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.543+0000 7eff17fff700 15 client.37497.objecter _session_command_op_remove 7 16 2024-04-14T15:20:34.544 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:20:34.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.544+0000 7eff353b5700 10 librados: watch_flush enter 2024-04-14T15:20:34.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.544+0000 7eff353b5700 10 librados: watch_flush exit 2024-04-14T15:20:34.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.544+0000 7eff353b5700 10 client.37497.objecter close_session for osd.0 2024-04-14T15:20:34.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.544+0000 7eff353b5700 1 -- 172.21.15.90:0/546383551 >> v1:172.21.15.90:6801/3761806885 conn(0x7eff1c004730 legacy=0x7eff1c002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:34.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.544+0000 7eff353b5700 20 client.37497.objecter put_session s=0x7eff1c0053e0 osd=0 1 2024-04-14T15:20:34.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.544+0000 7eff353b5700 10 client.37497.objecter close_session for osd.1 2024-04-14T15:20:34.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.544+0000 7eff353b5700 1 -- 172.21.15.90:0/546383551 >> v1:172.21.15.90:6805/4170361699 conn(0x7eff1c004310 legacy=0x7eff1c00ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:34.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.544+0000 7eff353b5700 20 client.37497.objecter put_session s=0x7eff1c004130 osd=1 1 2024-04-14T15:20:34.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.544+0000 7eff353b5700 10 client.37497.objecter close_session for osd.2 2024-04-14T15:20:34.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.544+0000 7eff353b5700 1 -- 172.21.15.90:0/546383551 >> v1:172.21.15.90:6809/1686184251 conn(0x7eff1c00d5d0 legacy=0x7eff1c00f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:34.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.544+0000 7eff353b5700 20 client.37497.objecter put_session s=0x7eff1c001980 osd=2 1 2024-04-14T15:20:34.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.544+0000 7eff353b5700 10 client.37497.objecter close_session for osd.3 2024-04-14T15:20:34.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.544+0000 7eff353b5700 1 -- 172.21.15.90:0/546383551 >> v1:172.21.15.90:6813/2270166774 conn(0x7eff1c012410 legacy=0x7eff1c0128c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:34.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.544+0000 7eff2f7fe700 1 -- 172.21.15.90:0/546383551 reap_dead start 2024-04-14T15:20:34.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.544+0000 7eff353b5700 20 client.37497.objecter put_session s=0x7eff1c008390 osd=3 1 2024-04-14T15:20:34.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.544+0000 7eff353b5700 10 client.37497.objecter close_session for osd.4 2024-04-14T15:20:34.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.544+0000 7eff353b5700 1 -- 172.21.15.90:0/546383551 >> v1:172.21.15.92:6800/3887549733 conn(0x7eff1c016c40 legacy=0x7eff1c019030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:34.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.544+0000 7eff353b5700 20 client.37497.objecter put_session s=0x7eff1c013980 osd=4 1 2024-04-14T15:20:34.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.544+0000 7eff353b5700 10 client.37497.objecter close_session for osd.5 2024-04-14T15:20:34.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.544+0000 7eff353b5700 1 -- 172.21.15.90:0/546383551 >> v1:172.21.15.92:6804/3244561440 conn(0x7eff1c0131b0 legacy=0x7eff1c01a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:34.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.544+0000 7eff353b5700 20 client.37497.objecter put_session s=0x7eff1c00c760 osd=5 1 2024-04-14T15:20:34.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.544+0000 7eff353b5700 10 client.37497.objecter close_session for osd.6 2024-04-14T15:20:34.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.544+0000 7eff353b5700 1 -- 172.21.15.90:0/546383551 >> v1:172.21.15.92:6808/636125963 conn(0x7eff1c01f8e0 legacy=0x7eff1c021cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:34.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.544+0000 7eff353b5700 20 client.37497.objecter put_session s=0x7eff1c012040 osd=6 1 2024-04-14T15:20:34.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.544+0000 7eff353b5700 10 client.37497.objecter close_session for osd.7 2024-04-14T15:20:34.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.544+0000 7eff353b5700 1 -- 172.21.15.90:0/546383551 >> v1:172.21.15.92:6812/1773706716 conn(0x7eff1c023cf0 legacy=0x7eff1c024150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:34.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.544+0000 7eff353b5700 20 client.37497.objecter put_session s=0x7eff1c0235d0 osd=7 1 2024-04-14T15:20:34.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.544+0000 7eff353b5700 20 client.37497.objecter shutdown clearing up homeless session... 2024-04-14T15:20:34.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.544+0000 7eff353b5700 10 client.37497.objecter successfully canceled tick 2024-04-14T15:20:34.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.544+0000 7eff2f7fe700 1 -- 172.21.15.90:0/546383551 reap_dead start 2024-04-14T15:20:34.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.544+0000 7eff353b5700 1 -- 172.21.15.90:0/546383551 >> v1:172.21.15.90:6800/420935427 conn(0x7eff18071380 legacy=0x7eff18073840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:34.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.544+0000 7eff353b5700 1 -- 172.21.15.90:0/546383551 >> v1:172.21.15.90:6789/0 conn(0x7eff30108980 legacy=0x7eff30104f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:34.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.544+0000 7eff353b5700 1 -- 172.21.15.90:0/546383551 shutdown_connections 2024-04-14T15:20:34.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.544+0000 7eff353b5700 1 -- 172.21.15.90:0/546383551 >> 172.21.15.90:0/546383551 conn(0x7eff30076aa0 msgr2=0x7eff301117b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:20:34.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.544+0000 7eff353b5700 1 -- 172.21.15.90:0/546383551 shutdown_connections 2024-04-14T15:20:34.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.544+0000 7eff353b5700 1 -- 172.21.15.90:0/546383551 wait complete. 2024-04-14T15:20:34.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:34.544+0000 7eff353b5700 1 librados: shutdown 2024-04-14T15:20:34.549 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:20:34.549 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:20:34.549 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:20:34.549 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:20:34.549 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:20:34.549 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:20:34.549 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:20:34.549 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:20:34.553 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:34.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:34.555 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:20:34.564 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-14T15:20:34.564 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-14T15:20:34.564 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph pg dump --format=json 2024-04-14T15:20:34.656 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:20:34.680 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:34.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:34.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:20:34.782 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:34.805 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:34.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:34.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:34.907 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:20:34.930 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:34.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:34.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:20:35.032 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:20:35.055 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:35.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:35.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:20:35.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:34 smithi092 ceph-mon[33966]: pgmap v737: 258 pgs: 10 remapped+peering, 248 active+clean; 8.5 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 9.0 KiB/s rd, 55 KiB/s wr, 11 op/s; 39 KiB/s, 0 objects/s recovering 2024-04-14T15:20:35.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:34 smithi092 ceph-mon[33966]: 1.16 scrub starts 2024-04-14T15:20:35.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:34 smithi092 ceph-mon[33966]: 1.16 scrub ok 2024-04-14T15:20:35.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:34 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-81","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-04-14T15:20:35.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:34 smithi092 ceph-mon[33966]: osdmap e509: 8 total, 8 up, 8 in 2024-04-14T15:20:35.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:34 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-14T15:20:35.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:34 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-14T15:20:35.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:34 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/546383551' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:20:35.152 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:34 smithi090 ceph-mon[41670]: pgmap v737: 258 pgs: 10 remapped+peering, 248 active+clean; 8.5 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 9.0 KiB/s rd, 55 KiB/s wr, 11 op/s; 39 KiB/s, 0 objects/s recovering 2024-04-14T15:20:35.152 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:34 smithi090 ceph-mon[41670]: 1.16 scrub starts 2024-04-14T15:20:35.152 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:34 smithi090 ceph-mon[41670]: 1.16 scrub ok 2024-04-14T15:20:35.152 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:34 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-81","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-04-14T15:20:35.152 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:34 smithi090 ceph-mon[41670]: osdmap e509: 8 total, 8 up, 8 in 2024-04-14T15:20:35.152 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:34 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-14T15:20:35.152 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:34 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-14T15:20:35.153 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:34 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/546383551' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:20:35.153 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:34 smithi090 ceph-mon[30399]: pgmap v737: 258 pgs: 10 remapped+peering, 248 active+clean; 8.5 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 9.0 KiB/s rd, 55 KiB/s wr, 11 op/s; 39 KiB/s, 0 objects/s recovering 2024-04-14T15:20:35.153 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:34 smithi090 ceph-mon[30399]: 1.16 scrub starts 2024-04-14T15:20:35.153 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:34 smithi090 ceph-mon[30399]: 1.16 scrub ok 2024-04-14T15:20:35.153 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:34 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-81","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-04-14T15:20:35.153 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:34 smithi090 ceph-mon[30399]: osdmap e509: 8 total, 8 up, 8 in 2024-04-14T15:20:35.153 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:34 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-14T15:20:35.153 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:34 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-14T15:20:35.153 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:34 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/546383551' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:20:35.157 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:20:35.177 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-14T15:20:35.181 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:35.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:35.182 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:20:35.283 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:20:35.306 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:35.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:35.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:20:35.408 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:20:35.413 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:20:35.434 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:35.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:35.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:20:35.535 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:20:35 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:20:35.535 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:20:35.558 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:35.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:35.562 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:20:35.663 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:35.687 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:35.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:35.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:35.788 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:35.812 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:35.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:35.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:35.913 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:35.937 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:35.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:35.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:36.039 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:20:36.062 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:36.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:36.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:20:36.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:35 smithi092 ceph-mon[33966]: 110.1d scrub starts 2024-04-14T15:20:36.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:35 smithi092 ceph-mon[33966]: 110.1d scrub ok 2024-04-14T15:20:36.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:35 smithi092 ceph-mon[33966]: 11.18 scrub starts 2024-04-14T15:20:36.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:35 smithi092 ceph-mon[33966]: 11.18 scrub ok 2024-04-14T15:20:36.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:35 smithi092 ceph-mon[33966]: 110.1a scrub starts 2024-04-14T15:20:36.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:35 smithi092 ceph-mon[33966]: 110.1a scrub ok 2024-04-14T15:20:36.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:35 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-14T15:20:36.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:35 smithi092 ceph-mon[33966]: osdmap e510: 8 total, 8 up, 8 in 2024-04-14T15:20:36.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:35 smithi092 ceph-mon[33966]: 6.b scrub starts 2024-04-14T15:20:36.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:35 smithi092 ceph-mon[33966]: 6.b scrub ok 2024-04-14T15:20:36.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:35 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:20:36.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:35 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:20:36.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:35 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-81"}]: dispatch 2024-04-14T15:20:36.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:35 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-81"}]: dispatch 2024-04-14T15:20:36.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:35 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-04-14T15:20:36.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:35 smithi092 ceph-mon[33966]: Health check update: Reduced data availability: 4 pgs inactive, 7 pgs peering (PG_AVAILABILITY) 2024-04-14T15:20:36.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:35 smithi092 ceph-mon[33966]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:20:36.164 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:20:36.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:36.188 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:36.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:20:36.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:35 smithi090 ceph-mon[30399]: 110.1d scrub starts 2024-04-14T15:20:36.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:35 smithi090 ceph-mon[30399]: 110.1d scrub ok 2024-04-14T15:20:36.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:35 smithi090 ceph-mon[30399]: 11.18 scrub starts 2024-04-14T15:20:36.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:35 smithi090 ceph-mon[30399]: 11.18 scrub ok 2024-04-14T15:20:36.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:35 smithi090 ceph-mon[30399]: 110.1a scrub starts 2024-04-14T15:20:36.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:35 smithi090 ceph-mon[30399]: 110.1a scrub ok 2024-04-14T15:20:36.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:35 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-14T15:20:36.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:35 smithi090 ceph-mon[30399]: osdmap e510: 8 total, 8 up, 8 in 2024-04-14T15:20:36.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:35 smithi090 ceph-mon[30399]: 6.b scrub starts 2024-04-14T15:20:36.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:35 smithi090 ceph-mon[30399]: 6.b scrub ok 2024-04-14T15:20:36.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:35 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:20:36.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:35 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:20:36.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:35 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-81"}]: dispatch 2024-04-14T15:20:36.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:35 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-81"}]: dispatch 2024-04-14T15:20:36.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:35 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-04-14T15:20:36.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:35 smithi090 ceph-mon[30399]: Health check update: Reduced data availability: 4 pgs inactive, 7 pgs peering (PG_AVAILABILITY) 2024-04-14T15:20:36.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:35 smithi090 ceph-mon[30399]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:20:36.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:35 smithi090 ceph-mon[41670]: 110.1d scrub starts 2024-04-14T15:20:36.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:35 smithi090 ceph-mon[41670]: 110.1d scrub ok 2024-04-14T15:20:36.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:35 smithi090 ceph-mon[41670]: 11.18 scrub starts 2024-04-14T15:20:36.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:35 smithi090 ceph-mon[41670]: 11.18 scrub ok 2024-04-14T15:20:36.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:35 smithi090 ceph-mon[41670]: 110.1a scrub starts 2024-04-14T15:20:36.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:35 smithi090 ceph-mon[41670]: 110.1a scrub ok 2024-04-14T15:20:36.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:35 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-14T15:20:36.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:35 smithi090 ceph-mon[41670]: osdmap e510: 8 total, 8 up, 8 in 2024-04-14T15:20:36.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:35 smithi090 ceph-mon[41670]: 6.b scrub starts 2024-04-14T15:20:36.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:35 smithi090 ceph-mon[41670]: 6.b scrub ok 2024-04-14T15:20:36.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:35 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:20:36.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:35 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:20:36.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:35 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-81"}]: dispatch 2024-04-14T15:20:36.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:35 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-81"}]: dispatch 2024-04-14T15:20:36.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:35 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-04-14T15:20:36.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:35 smithi090 ceph-mon[41670]: Health check update: Reduced data availability: 4 pgs inactive, 7 pgs peering (PG_AVAILABILITY) 2024-04-14T15:20:36.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:35 smithi090 ceph-mon[41670]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:20:36.289 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:20:36.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:36.313 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:36.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:20:36.414 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:20:36.438 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:36.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:36.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:20:36.540 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:20:36.563 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:36.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:36.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:20:36.665 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:36.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:36.689 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:36.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:36.721 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:20:36.790 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:36.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:36.814 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:36.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:36.915 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:20:36.939 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:36.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:36.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:20:36.961 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:36 smithi090 ceph-mon[30399]: pgmap v740: 258 pgs: 258 active+clean; 8.4 MiB data, 593 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 B/s wr, 20 op/s; 25 KiB/s, 0 objects/s recovering 2024-04-14T15:20:36.961 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:36 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:20:36.961 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:36 smithi090 ceph-mon[30399]: 2.e scrub starts 2024-04-14T15:20:36.961 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:36 smithi090 ceph-mon[30399]: 2.e scrub ok 2024-04-14T15:20:36.961 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:36 smithi090 ceph-mon[30399]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 7 pgs peering) 2024-04-14T15:20:36.961 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:36 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "30"}]': finished 2024-04-14T15:20:36.961 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:36 smithi090 ceph-mon[30399]: osdmap e511: 8 total, 8 up, 8 in 2024-04-14T15:20:36.961 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:36 smithi090 ceph-mon[30399]: 2.1e scrub starts 2024-04-14T15:20:36.961 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:36 smithi090 ceph-mon[30399]: 2.1e scrub ok 2024-04-14T15:20:36.961 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:36 smithi090 ceph-mon[30399]: 5.1b scrub starts 2024-04-14T15:20:36.961 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:36 smithi090 ceph-mon[30399]: 5.1b scrub ok 2024-04-14T15:20:36.962 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:36 smithi090 ceph-mon[41670]: pgmap v740: 258 pgs: 258 active+clean; 8.4 MiB data, 593 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 B/s wr, 20 op/s; 25 KiB/s, 0 objects/s recovering 2024-04-14T15:20:36.962 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:36 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:20:36.962 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:36 smithi090 ceph-mon[41670]: 2.e scrub starts 2024-04-14T15:20:36.962 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:36 smithi090 ceph-mon[41670]: 2.e scrub ok 2024-04-14T15:20:36.962 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:36 smithi090 ceph-mon[41670]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 7 pgs peering) 2024-04-14T15:20:36.962 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:36 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "30"}]': finished 2024-04-14T15:20:36.962 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:36 smithi090 ceph-mon[41670]: osdmap e511: 8 total, 8 up, 8 in 2024-04-14T15:20:36.962 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:36 smithi090 ceph-mon[41670]: 2.1e scrub starts 2024-04-14T15:20:36.962 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:36 smithi090 ceph-mon[41670]: 2.1e scrub ok 2024-04-14T15:20:36.962 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:36 smithi090 ceph-mon[41670]: 5.1b scrub starts 2024-04-14T15:20:36.962 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:36 smithi090 ceph-mon[41670]: 5.1b scrub ok 2024-04-14T15:20:36.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:36.954+0000 7f3a78c4d700 1 -- 172.21.15.90:0/2023946628 >> v1:172.21.15.90:6789/0 conn(0x7f3a7410d990 legacy=0x7f3a7410fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:36.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:36.955+0000 7f3a78c4d700 1 -- 172.21.15.90:0/2023946628 shutdown_connections 2024-04-14T15:20:36.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:36.955+0000 7f3a78c4d700 1 -- 172.21.15.90:0/2023946628 >> 172.21.15.90:0/2023946628 conn(0x7f3a74076aa0 msgr2=0x7f3a74076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:20:36.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:36.955+0000 7f3a78c4d700 1 -- 172.21.15.90:0/2023946628 shutdown_connections 2024-04-14T15:20:36.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:36.955+0000 7f3a78c4d700 1 -- 172.21.15.90:0/2023946628 wait complete. 2024-04-14T15:20:36.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:36.956+0000 7f3a78c4d700 1 librados: starting msgr at 2024-04-14T15:20:36.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:36.956+0000 7f3a78c4d700 1 librados: starting objecter 2024-04-14T15:20:36.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:36.956+0000 7f3a78c4d700 1 Processor -- start 2024-04-14T15:20:36.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:36.956+0000 7f3a78c4d700 1 -- start start 2024-04-14T15:20:36.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:36.956+0000 7f3a78c4d700 1 librados: setting wanted keys 2024-04-14T15:20:36.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:36.956+0000 7f3a78c4d700 1 librados: calling monclient init 2024-04-14T15:20:36.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:36.957+0000 7f3a78c4d700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3a741a7d00 con 0x7f3a74109df0 2024-04-14T15:20:36.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:36.957+0000 7f3a78c4d700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3a741a8ee0 con 0x7f3a741012e0 2024-04-14T15:20:36.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:36.957+0000 7f3a78c4d700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3a741aa0c0 con 0x7f3a7410d990 2024-04-14T15:20:36.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:36.957+0000 7f3a71d9b700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f3a74109df0 0x7f3a741a2eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:59450/0 (socket says 172.21.15.90:59450) 2024-04-14T15:20:36.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:36.957+0000 7f3a71d9b700 1 -- 172.21.15.90:0/381102000 learned_addr learned my addr 172.21.15.90:0/381102000 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:20:36.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:36.957+0000 7f3a7259c700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f3a741012e0 0x7f3a7419d190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.90:45470/0 (socket says 172.21.15.90:45470) 2024-04-14T15:20:36.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:36.957+0000 7f3a72d9d700 1 --1- 172.21.15.90:0/381102000 >> v1:172.21.15.90:6790/0 conn(0x7f3a7410d990 0x7f3a741a65e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:57250/0 (socket says 172.21.15.90:57250) 2024-04-14T15:20:36.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:36.957+0000 7f3a637fe700 10 client.?.objecter ms_handle_connect 0x7f3a74109df0 2024-04-14T15:20:36.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:36.957+0000 7f3a637fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:20:36.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:36.957+0000 7f3a637fe700 10 client.?.objecter ms_handle_connect 0x7f3a7410d990 2024-04-14T15:20:36.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:36.957+0000 7f3a637fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:20:36.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:36.957+0000 7f3a637fe700 10 client.?.objecter ms_handle_connect 0x7f3a741012e0 2024-04-14T15:20:36.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:36.957+0000 7f3a637fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:20:36.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:36.959+0000 7f3a637fe700 1 -- 172.21.15.90:0/381102000 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3268390264 0 0) 0x7f3a741a8ee0 con 0x7f3a741012e0 2024-04-14T15:20:36.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:36.959+0000 7f3a637fe700 1 -- 172.21.15.90:0/381102000 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3a5c0035b0 con 0x7f3a741012e0 2024-04-14T15:20:36.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:36.959+0000 7f3a637fe700 1 -- 172.21.15.90:0/381102000 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3924993752 0 0) 0x7f3a741aa0c0 con 0x7f3a7410d990 2024-04-14T15:20:36.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:36.959+0000 7f3a637fe700 1 -- 172.21.15.90:0/381102000 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3a741a8ee0 con 0x7f3a7410d990 2024-04-14T15:20:36.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:36.959+0000 7f3a637fe700 1 -- 172.21.15.90:0/381102000 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1710207678 0 0) 0x7f3a5c0035b0 con 0x7f3a741012e0 2024-04-14T15:20:36.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:36.959+0000 7f3a637fe700 1 -- 172.21.15.90:0/381102000 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3a741aa0c0 con 0x7f3a741012e0 2024-04-14T15:20:36.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:36.959+0000 7f3a637fe700 1 -- 172.21.15.90:0/381102000 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f3a640016e0 con 0x7f3a741012e0 2024-04-14T15:20:36.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:36.959+0000 7f3a637fe700 1 -- 172.21.15.90:0/381102000 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4153216655 0 0) 0x7f3a741a8ee0 con 0x7f3a7410d990 2024-04-14T15:20:36.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:36.959+0000 7f3a637fe700 1 -- 172.21.15.90:0/381102000 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3a5c0035b0 con 0x7f3a7410d990 2024-04-14T15:20:36.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:36.959+0000 7f3a637fe700 1 -- 172.21.15.90:0/381102000 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f3a6c001950 con 0x7f3a7410d990 2024-04-14T15:20:36.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:36.959+0000 7f3a637fe700 1 -- 172.21.15.90:0/381102000 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2259651768 0 0) 0x7f3a741aa0c0 con 0x7f3a741012e0 2024-04-14T15:20:36.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:36.959+0000 7f3a637fe700 1 -- 172.21.15.90:0/381102000 >> v1:172.21.15.90:6790/0 conn(0x7f3a7410d990 legacy=0x7f3a741a65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:36.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:36.959+0000 7f3a637fe700 1 -- 172.21.15.90:0/381102000 >> v1:172.21.15.90:6789/0 conn(0x7f3a74109df0 legacy=0x7f3a741a2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:36.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:36.959+0000 7f3a637fe700 1 -- 172.21.15.90:0/381102000 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a741ab2a0 con 0x7f3a741012e0 2024-04-14T15:20:36.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:36.960+0000 7f3a78c4d700 1 -- 172.21.15.90:0/381102000 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3a741a7f30 con 0x7f3a741012e0 2024-04-14T15:20:36.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:36.960+0000 7f3a78c4d700 10 client.49333.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:20:36.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:36.960+0000 7f3a78c4d700 1 -- 172.21.15.90:0/381102000 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f3a741a83f0 con 0x7f3a741012e0 2024-04-14T15:20:36.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:36.960+0000 7f3a78c4d700 1 librados: init done 2024-04-14T15:20:36.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:36.960+0000 7f3a637fe700 1 -- 172.21.15.90:0/381102000 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f3a64001cf0 con 0x7f3a741012e0 2024-04-14T15:20:36.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:36.960+0000 7f3a637fe700 1 -- 172.21.15.90:0/381102000 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f3a64005c20 con 0x7f3a741012e0 2024-04-14T15:20:36.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:36.961+0000 7f3a78c4d700 1 -- 172.21.15.90:0/381102000 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a54005120 con 0x7f3a741012e0 2024-04-14T15:20:36.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:36.961+0000 7f3a637fe700 1 -- 172.21.15.90:0/381102000 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f3a64005dc0 con 0x7f3a741012e0 2024-04-14T15:20:36.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:36.962+0000 7f3a637fe700 1 -- 172.21.15.90:0/381102000 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(511..511 src has 51..511) ==== 7812+0+0 (unknown 2943012820 0 0) 0x7f3a6408a890 con 0x7f3a741012e0 2024-04-14T15:20:36.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:36.962+0000 7f3a637fe700 10 client.49333.objecter ms_dispatch 0x7f3a74001550 osd_map(511..511 src has 51..511) 2024-04-14T15:20:36.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:36.962+0000 7f3a637fe700 3 client.49333.objecter handle_osd_map got epochs [511,511] > 0 2024-04-14T15:20:36.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:36.962+0000 7f3a637fe700 3 client.49333.objecter handle_osd_map decoding full epoch 511 2024-04-14T15:20:36.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:36.962+0000 7f3a637fe700 20 client.49333.objecter dump_active .. 0 homeless 2024-04-14T15:20:36.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:36.962+0000 7f3a637fe700 10 client.49333.objecter ms_handle_connect 0x7f3a5c071330 2024-04-14T15:20:36.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:36.974+0000 7f3a637fe700 1 -- 172.21.15.90:0/381102000 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f3a6405fa50 con 0x7f3a741012e0 2024-04-14T15:20:37.040 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:37.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:37.064 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:37.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:37.098 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:36 smithi092 ceph-mon[33966]: pgmap v740: 258 pgs: 258 active+clean; 8.4 MiB data, 593 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 B/s wr, 20 op/s; 25 KiB/s, 0 objects/s recovering 2024-04-14T15:20:37.098 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:36 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:20:37.098 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:36 smithi092 ceph-mon[33966]: 2.e scrub starts 2024-04-14T15:20:37.098 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:36 smithi092 ceph-mon[33966]: 2.e scrub ok 2024-04-14T15:20:37.099 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:36 smithi092 ceph-mon[33966]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 7 pgs peering) 2024-04-14T15:20:37.099 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:36 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "30"}]': finished 2024-04-14T15:20:37.099 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:36 smithi092 ceph-mon[33966]: osdmap e511: 8 total, 8 up, 8 in 2024-04-14T15:20:37.099 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:36 smithi092 ceph-mon[33966]: 2.1e scrub starts 2024-04-14T15:20:37.099 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:36 smithi092 ceph-mon[33966]: 2.1e scrub ok 2024-04-14T15:20:37.099 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:36 smithi092 ceph-mon[33966]: 5.1b scrub starts 2024-04-14T15:20:37.099 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:36 smithi092 ceph-mon[33966]: 5.1b scrub ok 2024-04-14T15:20:37.165 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:20:37.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:37.189 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:37.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:20:37.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:37.193+0000 7f3a78c4d700 1 -- 172.21.15.90:0/381102000 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7f3a54005410 con 0x7f3a741012e0 2024-04-14T15:20:37.239 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-04-14T15:20:37.239 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-04-14T15:20:37.239 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-04-14T15:20:37.239 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-04-14T15:20:37.239 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-04-14T15:20:37.290 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:20:37.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:37.316 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:37.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:20:37.416 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:20:37.440 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:37.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:37.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:20:37.541 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:37.565 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:37.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:37.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:37.666 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:20:37.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:37.691 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:37.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:20:37.792 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:20:37.815 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:37.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:37.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:20:37.851 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-14T15:20:37.917 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:20:37.938 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-14T15:20:37.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:37.942 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:37.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:20:37.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:37.976+0000 7f3a637fe700 1 -- 172.21.15.90:0/381102000 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v513) ==== 101+0+0 (unknown 492807293 0 0) 0x7f3a64055f20 con 0x7f3a741012e0 2024-04-14T15:20:37.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:37.979+0000 7f3a78c4d700 10 librados: watch_flush enter 2024-04-14T15:20:37.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:37.979+0000 7f3a78c4d700 10 librados: watch_flush exit 2024-04-14T15:20:37.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:37.979+0000 7f3a78c4d700 20 client.49333.objecter shutdown clearing up homeless session... 2024-04-14T15:20:37.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:37.979+0000 7f3a78c4d700 10 client.49333.objecter successfully canceled tick 2024-04-14T15:20:37.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:37.980+0000 7f3a78c4d700 1 -- 172.21.15.90:0/381102000 >> v1:172.21.15.90:6800/420935427 conn(0x7f3a5c071330 legacy=0x7f3a5c0737f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:37.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:37.980+0000 7f3a78c4d700 1 -- 172.21.15.90:0/381102000 >> v1:172.21.15.92:6789/0 conn(0x7f3a741012e0 legacy=0x7f3a7419d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:37.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:37.980+0000 7f3a78c4d700 1 -- 172.21.15.90:0/381102000 shutdown_connections 2024-04-14T15:20:37.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:37.980+0000 7f3a78c4d700 1 -- 172.21.15.90:0/381102000 >> 172.21.15.90:0/381102000 conn(0x7f3a74076aa0 msgr2=0x7f3a74110980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:20:37.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:37.981+0000 7f3a78c4d700 1 -- 172.21.15.90:0/381102000 shutdown_connections 2024-04-14T15:20:37.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:37.982+0000 7f3a78c4d700 1 -- 172.21.15.90:0/381102000 wait complete. 2024-04-14T15:20:37.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:37.982+0000 7f3a78c4d700 1 librados: shutdown 2024-04-14T15:20:37.991 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is unset 2024-04-14T15:20:38.043 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:38.066 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:38.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:38.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:38.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:37 smithi092 ceph-mon[33966]: 5.d scrub starts 2024-04-14T15:20:38.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:37 smithi092 ceph-mon[33966]: 5.d scrub ok 2024-04-14T15:20:38.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:37 smithi092 ceph-mon[33966]: 6.1a scrub starts 2024-04-14T15:20:38.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:37 smithi092 ceph-mon[33966]: 6.1a scrub ok 2024-04-14T15:20:38.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:37 smithi092 ceph-mon[33966]: 6.a scrub starts 2024-04-14T15:20:38.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:37 smithi092 ceph-mon[33966]: 6.a scrub ok 2024-04-14T15:20:38.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:37 smithi092 ceph-mon[33966]: 2.19 scrub starts 2024-04-14T15:20:38.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:37 smithi092 ceph-mon[33966]: 2.19 scrub ok 2024-04-14T15:20:38.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:37 smithi092 ceph-mon[33966]: osdmap e512: 8 total, 8 up, 8 in 2024-04-14T15:20:38.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:37 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:20:38.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:37 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:20:38.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:37 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/381102000' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:20:38.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:37 smithi092 ceph-mon[33966]: from='client.49333 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:20:38.148 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:20:38.168 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:20:38.191 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:38.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:37 smithi090 ceph-mon[30399]: 5.d scrub starts 2024-04-14T15:20:38.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:37 smithi090 ceph-mon[30399]: 5.d scrub ok 2024-04-14T15:20:38.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:37 smithi090 ceph-mon[30399]: 6.1a scrub starts 2024-04-14T15:20:38.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:37 smithi090 ceph-mon[30399]: 6.1a scrub ok 2024-04-14T15:20:38.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:37 smithi090 ceph-mon[30399]: 6.a scrub starts 2024-04-14T15:20:38.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:37 smithi090 ceph-mon[30399]: 6.a scrub ok 2024-04-14T15:20:38.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:37 smithi090 ceph-mon[30399]: 2.19 scrub starts 2024-04-14T15:20:38.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:37 smithi090 ceph-mon[30399]: 2.19 scrub ok 2024-04-14T15:20:38.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:37 smithi090 ceph-mon[30399]: osdmap e512: 8 total, 8 up, 8 in 2024-04-14T15:20:38.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:37 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:20:38.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:37 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:20:38.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:37 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/381102000' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:20:38.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:37 smithi090 ceph-mon[30399]: from='client.49333 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:20:38.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:37 smithi090 ceph-mon[41670]: 5.d scrub starts 2024-04-14T15:20:38.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:37 smithi090 ceph-mon[41670]: 5.d scrub ok 2024-04-14T15:20:38.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:37 smithi090 ceph-mon[41670]: 6.1a scrub starts 2024-04-14T15:20:38.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:37 smithi090 ceph-mon[41670]: 6.1a scrub ok 2024-04-14T15:20:38.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:37 smithi090 ceph-mon[41670]: 6.a scrub starts 2024-04-14T15:20:38.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:37 smithi090 ceph-mon[41670]: 6.a scrub ok 2024-04-14T15:20:38.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:37 smithi090 ceph-mon[41670]: 2.19 scrub starts 2024-04-14T15:20:38.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:37 smithi090 ceph-mon[41670]: 2.19 scrub ok 2024-04-14T15:20:38.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:37 smithi090 ceph-mon[41670]: osdmap e512: 8 total, 8 up, 8 in 2024-04-14T15:20:38.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:37 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:20:38.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:37 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:20:38.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:37 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/381102000' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:20:38.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:37 smithi090 ceph-mon[41670]: from='client.49333 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:20:38.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:38.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:20:38.295 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:20:38.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:38.320 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:38.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:20:38.421 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:20:38.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:38.445 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:38.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:20:38.546 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:20:38.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:38.571 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:38.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:20:38.672 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:20:38.695 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:38.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:38.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:20:38.797 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:20:38.821 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:38.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:38.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:20:38.923 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:20:38.947 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:38.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:38.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:20:38.969 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:38 smithi092 ceph-mon[33966]: pgmap v743: 258 pgs: 32 unknown, 226 active+clean; 8.4 MiB data, 593 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 B/s wr, 19 op/s; 25 KiB/s, 0 objects/s recovering 2024-04-14T15:20:38.969 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:38 smithi092 ceph-mon[33966]: 11.e scrub starts 2024-04-14T15:20:38.969 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:38 smithi092 ceph-mon[33966]: 11.e scrub ok 2024-04-14T15:20:38.969 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:38 smithi092 ceph-mon[33966]: 11.1 scrub starts 2024-04-14T15:20:38.969 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:38 smithi092 ceph-mon[33966]: 11.1 scrub ok 2024-04-14T15:20:38.969 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:38 smithi092 ceph-mon[33966]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-14T15:20:38.969 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:38 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:20:38.969 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:38 smithi092 ceph-mon[33966]: from='client.49333 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-14T15:20:38.969 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:38 smithi092 ceph-mon[33966]: osdmap e513: 8 total, 8 up, 8 in 2024-04-14T15:20:38.969 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:38 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-14T15:20:38.970 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:38 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-14T15:20:38.970 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:38 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-14T15:20:38.970 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:38 smithi092 ceph-mon[33966]: 6.1d scrub starts 2024-04-14T15:20:38.970 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:38 smithi092 ceph-mon[33966]: 6.1d scrub ok 2024-04-14T15:20:38.970 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:38 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:20:38.970 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:38 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:20:39.048 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:20:39.072 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:39.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:39.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:20:39.173 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:20:39.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:38 smithi090 ceph-mon[30399]: pgmap v743: 258 pgs: 32 unknown, 226 active+clean; 8.4 MiB data, 593 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 B/s wr, 19 op/s; 25 KiB/s, 0 objects/s recovering 2024-04-14T15:20:39.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:38 smithi090 ceph-mon[30399]: 11.e scrub starts 2024-04-14T15:20:39.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:38 smithi090 ceph-mon[30399]: 11.e scrub ok 2024-04-14T15:20:39.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:38 smithi090 ceph-mon[30399]: 11.1 scrub starts 2024-04-14T15:20:39.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:38 smithi090 ceph-mon[30399]: 11.1 scrub ok 2024-04-14T15:20:39.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:38 smithi090 ceph-mon[30399]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-14T15:20:39.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:38 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:20:39.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:38 smithi090 ceph-mon[30399]: from='client.49333 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-14T15:20:39.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:38 smithi090 ceph-mon[30399]: osdmap e513: 8 total, 8 up, 8 in 2024-04-14T15:20:39.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:38 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-14T15:20:39.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:38 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-14T15:20:39.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:38 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-14T15:20:39.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:38 smithi090 ceph-mon[30399]: 6.1d scrub starts 2024-04-14T15:20:39.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:38 smithi090 ceph-mon[30399]: 6.1d scrub ok 2024-04-14T15:20:39.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:38 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:20:39.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:38 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:20:39.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:38 smithi090 ceph-mon[41670]: pgmap v743: 258 pgs: 32 unknown, 226 active+clean; 8.4 MiB data, 593 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 B/s wr, 19 op/s; 25 KiB/s, 0 objects/s recovering 2024-04-14T15:20:39.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:38 smithi090 ceph-mon[41670]: 11.e scrub starts 2024-04-14T15:20:39.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:38 smithi090 ceph-mon[41670]: 11.e scrub ok 2024-04-14T15:20:39.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:38 smithi090 ceph-mon[41670]: 11.1 scrub starts 2024-04-14T15:20:39.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:38 smithi090 ceph-mon[41670]: 11.1 scrub ok 2024-04-14T15:20:39.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:38 smithi090 ceph-mon[41670]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-14T15:20:39.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:38 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:20:39.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:38 smithi090 ceph-mon[41670]: from='client.49333 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-14T15:20:39.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:38 smithi090 ceph-mon[41670]: osdmap e513: 8 total, 8 up, 8 in 2024-04-14T15:20:39.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:38 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-14T15:20:39.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:38 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-14T15:20:39.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:38 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-14T15:20:39.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:38 smithi090 ceph-mon[41670]: 6.1d scrub starts 2024-04-14T15:20:39.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:38 smithi090 ceph-mon[41670]: 6.1d scrub ok 2024-04-14T15:20:39.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:38 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:20:39.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:38 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:20:39.198 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:39.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:39.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:20:39.299 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:20:39.323 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:39.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:39.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:20:39.426 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:39.449 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:39.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:39.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:39.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.527+0000 7fa891577700 1 -- 172.21.15.90:0/3323560633 >> v1:172.21.15.90:6789/0 conn(0x7fa88c10f310 legacy=0x7fa88c1117d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:39.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.527+0000 7fa891577700 1 -- 172.21.15.90:0/3323560633 shutdown_connections 2024-04-14T15:20:39.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.527+0000 7fa891577700 1 -- 172.21.15.90:0/3323560633 >> 172.21.15.90:0/3323560633 conn(0x7fa88c0fe380 msgr2=0x7fa88c1007a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:20:39.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.527+0000 7fa891577700 1 -- 172.21.15.90:0/3323560633 shutdown_connections 2024-04-14T15:20:39.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.527+0000 7fa891577700 1 -- 172.21.15.90:0/3323560633 wait complete. 2024-04-14T15:20:39.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.527+0000 7fa891577700 1 librados: starting msgr at 2024-04-14T15:20:39.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.527+0000 7fa891577700 1 librados: starting objecter 2024-04-14T15:20:39.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.527+0000 7fa891577700 1 Processor -- start 2024-04-14T15:20:39.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.527+0000 7fa891577700 1 -- start start 2024-04-14T15:20:39.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.527+0000 7fa891577700 1 librados: setting wanted keys 2024-04-14T15:20:39.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.527+0000 7fa891577700 1 librados: calling monclient init 2024-04-14T15:20:39.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.528+0000 7fa891577700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa88c10f000 con 0x7fa88c108890 2024-04-14T15:20:39.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.528+0000 7fa891577700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa88c1a8f20 con 0x7fa88c10b740 2024-04-14T15:20:39.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.528+0000 7fa891577700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa88c1aa100 con 0x7fa88c10f310 2024-04-14T15:20:39.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.528+0000 7fa88affd700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fa88c108890 0x7fa88c10bfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:59480/0 (socket says 172.21.15.90:59480) 2024-04-14T15:20:39.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.528+0000 7fa88affd700 1 -- 172.21.15.90:0/1954794875 learned_addr learned my addr 172.21.15.90:0/1954794875 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:20:39.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.528+0000 7fa873fff700 10 client.?.objecter ms_handle_connect 0x7fa88c108890 2024-04-14T15:20:39.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.528+0000 7fa873fff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:20:39.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.528+0000 7fa873fff700 10 client.?.objecter ms_handle_connect 0x7fa88c10f310 2024-04-14T15:20:39.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.528+0000 7fa873fff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:20:39.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.528+0000 7fa873fff700 10 client.?.objecter ms_handle_connect 0x7fa88c10b740 2024-04-14T15:20:39.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.528+0000 7fa873fff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:20:39.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.528+0000 7fa873fff700 1 -- 172.21.15.90:0/1954794875 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2245154200 0 0) 0x7fa88c10f000 con 0x7fa88c108890 2024-04-14T15:20:39.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.528+0000 7fa873fff700 1 -- 172.21.15.90:0/1954794875 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa8740035b0 con 0x7fa88c108890 2024-04-14T15:20:39.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.528+0000 7fa873fff700 1 -- 172.21.15.90:0/1954794875 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4010874932 0 0) 0x7fa88c1a8f20 con 0x7fa88c10b740 2024-04-14T15:20:39.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.528+0000 7fa873fff700 1 -- 172.21.15.90:0/1954794875 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa88c10f000 con 0x7fa88c10b740 2024-04-14T15:20:39.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.528+0000 7fa873fff700 1 -- 172.21.15.90:0/1954794875 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3519326297 0 0) 0x7fa88c1aa100 con 0x7fa88c10f310 2024-04-14T15:20:39.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.528+0000 7fa873fff700 1 -- 172.21.15.90:0/1954794875 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa88c1a8f20 con 0x7fa88c10f310 2024-04-14T15:20:39.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.528+0000 7fa873fff700 1 -- 172.21.15.90:0/1954794875 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 185600124 0 0) 0x7fa88c10f000 con 0x7fa88c10b740 2024-04-14T15:20:39.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.528+0000 7fa873fff700 1 -- 172.21.15.90:0/1954794875 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa88c1aa100 con 0x7fa88c10b740 2024-04-14T15:20:39.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.528+0000 7fa873fff700 1 -- 172.21.15.90:0/1954794875 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fa8800017b0 con 0x7fa88c10b740 2024-04-14T15:20:39.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.528+0000 7fa873fff700 1 -- 172.21.15.90:0/1954794875 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2102717884 0 0) 0x7fa8740035b0 con 0x7fa88c108890 2024-04-14T15:20:39.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.528+0000 7fa873fff700 1 -- 172.21.15.90:0/1954794875 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa88c10f000 con 0x7fa88c108890 2024-04-14T15:20:39.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.528+0000 7fa873fff700 1 -- 172.21.15.90:0/1954794875 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 835067362 0 0) 0x7fa88c1a8f20 con 0x7fa88c10f310 2024-04-14T15:20:39.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.528+0000 7fa873fff700 1 -- 172.21.15.90:0/1954794875 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa8740035b0 con 0x7fa88c10f310 2024-04-14T15:20:39.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.528+0000 7fa873fff700 1 -- 172.21.15.90:0/1954794875 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fa87c001850 con 0x7fa88c108890 2024-04-14T15:20:39.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.528+0000 7fa873fff700 1 -- 172.21.15.90:0/1954794875 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fa8840017b0 con 0x7fa88c10f310 2024-04-14T15:20:39.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.528+0000 7fa873fff700 1 -- 172.21.15.90:0/1954794875 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 100646201 0 0) 0x7fa88c1aa100 con 0x7fa88c10b740 2024-04-14T15:20:39.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.528+0000 7fa873fff700 1 -- 172.21.15.90:0/1954794875 >> v1:172.21.15.90:6790/0 conn(0x7fa88c10f310 legacy=0x7fa88c1a67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:39.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.528+0000 7fa873fff700 1 -- 172.21.15.90:0/1954794875 >> v1:172.21.15.90:6789/0 conn(0x7fa88c108890 legacy=0x7fa88c10bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:39.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.529+0000 7fa873fff700 1 -- 172.21.15.90:0/1954794875 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa88c1ab2e0 con 0x7fa88c10b740 2024-04-14T15:20:39.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.529+0000 7fa891577700 1 -- 172.21.15.90:0/1954794875 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa88c1aa330 con 0x7fa88c10b740 2024-04-14T15:20:39.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.529+0000 7fa891577700 10 client.49339.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:20:39.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.529+0000 7fa891577700 1 -- 172.21.15.90:0/1954794875 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa88c1aa7f0 con 0x7fa88c10b740 2024-04-14T15:20:39.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.529+0000 7fa891577700 1 librados: init done 2024-04-14T15:20:39.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.529+0000 7fa873fff700 1 -- 172.21.15.90:0/1954794875 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fa880003b20 con 0x7fa88c10b740 2024-04-14T15:20:39.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.529+0000 7fa873fff700 1 -- 172.21.15.90:0/1954794875 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fa880004970 con 0x7fa88c10b740 2024-04-14T15:20:39.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.529+0000 7fa891577700 1 -- 172.21.15.90:0/1954794875 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa878005150 con 0x7fa88c10b740 2024-04-14T15:20:39.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.530+0000 7fa873fff700 1 -- 172.21.15.90:0/1954794875 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fa880003680 con 0x7fa88c10b740 2024-04-14T15:20:39.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.531+0000 7fa873fff700 1 -- 172.21.15.90:0/1954794875 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(514..514 src has 51..514) ==== 8213+0+0 (unknown 1698249196 0 0) 0x7fa880089730 con 0x7fa88c10b740 2024-04-14T15:20:39.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.531+0000 7fa873fff700 10 client.49339.objecter ms_dispatch 0x7fa88c001550 osd_map(514..514 src has 51..514) 2024-04-14T15:20:39.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.531+0000 7fa873fff700 3 client.49339.objecter handle_osd_map got epochs [514,514] > 0 2024-04-14T15:20:39.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.531+0000 7fa873fff700 3 client.49339.objecter handle_osd_map decoding full epoch 514 2024-04-14T15:20:39.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.531+0000 7fa873fff700 20 client.49339.objecter dump_active .. 0 homeless 2024-04-14T15:20:39.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.531+0000 7fa873fff700 10 client.49339.objecter ms_handle_connect 0x7fa8740715c0 2024-04-14T15:20:39.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.535+0000 7fa873fff700 1 -- 172.21.15.90:0/1954794875 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fa880056110 con 0x7fa88c10b740 2024-04-14T15:20:39.551 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:20:39.575 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:39.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:39.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:20:39.651 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:20:39.676 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:20:39.699 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:39.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:39.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:20:39.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.739+0000 7fa891577700 1 -- 172.21.15.90:0/1954794875 --> v1:172.21.15.90:6800/420935427 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa878000bc0 con 0x7fa8740715c0 2024-04-14T15:20:39.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.755+0000 7fa873fff700 1 -- 172.21.15.90:0/1954794875 <== mgr.14610 v1:172.21.15.90:6800/420935427 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+653895 (unknown 2965378022 0 1286322666) 0x7fa878000bc0 con 0x7fa8740715c0 2024-04-14T15:20:39.764 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:20:39.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.760+0000 7fa891577700 10 librados: watch_flush enter 2024-04-14T15:20:39.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.761+0000 7fa891577700 10 librados: watch_flush exit 2024-04-14T15:20:39.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.761+0000 7fa891577700 20 client.49339.objecter shutdown clearing up homeless session... 2024-04-14T15:20:39.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.761+0000 7fa891577700 10 client.49339.objecter successfully canceled tick 2024-04-14T15:20:39.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.761+0000 7fa891577700 1 -- 172.21.15.90:0/1954794875 >> v1:172.21.15.90:6800/420935427 conn(0x7fa8740715c0 legacy=0x7fa874073a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:39.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.761+0000 7fa891577700 1 -- 172.21.15.90:0/1954794875 >> v1:172.21.15.92:6789/0 conn(0x7fa88c10b740 legacy=0x7fa88c10c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:39.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.761+0000 7fa891577700 1 -- 172.21.15.90:0/1954794875 shutdown_connections 2024-04-14T15:20:39.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.761+0000 7fa891577700 1 -- 172.21.15.90:0/1954794875 >> 172.21.15.90:0/1954794875 conn(0x7fa88c0fe380 msgr2=0x7fa88c112850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:20:39.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.761+0000 7fa891577700 1 -- 172.21.15.90:0/1954794875 shutdown_connections 2024-04-14T15:20:39.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.761+0000 7fa891577700 1 -- 172.21.15.90:0/1954794875 wait complete. 2024-04-14T15:20:39.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:39.761+0000 7fa891577700 1 librados: shutdown 2024-04-14T15:20:39.772 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-14T15:20:39.801 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:20:39.825 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:39.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:39.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:20:39.927 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:39.952 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:39.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:39.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:40.054 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:20:40.072 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-04-14T15:20:40.072 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-04-14T15:20:40.073 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-04-14T15:20:40.073 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-04-14T15:20:40.073 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.073 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25063.0:2 39.0 39:0384ab8e:::oid:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-04-14T15:20:40.073 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:55.066308+0000", 2024-04-14T15:20:40.073 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 465.00498176000002, 2024-04-14T15:20:40.073 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.468718499, 2024-04-14T15:20:40.073 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:20:40.073 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:20:40.073 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:20:40.073 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:20:40.073 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25063", 2024-04-14T15:20:40.073 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/3585118987", 2024-04-14T15:20:40.073 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 2 2024-04-14T15:20:40.074 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.074 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:20:40.074 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.074 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:20:40.074 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.066308+0000", 2024-04-14T15:20:40.074 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:20:40.074 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.074 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.074 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:20:40.074 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.066308+0000", 2024-04-14T15:20:40.074 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:20:40.074 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.074 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.074 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:20:40.074 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.066310+0000", 2024-04-14T15:20:40.075 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.787e-06 2024-04-14T15:20:40.075 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.075 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.075 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:20:40.075 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.066316+0000", 2024-04-14T15:20:40.075 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5149999999999997e-06 2024-04-14T15:20:40.075 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.075 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.075 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:20:40.075 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.066326+0000", 2024-04-14T15:20:40.075 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0193999999999999e-05 2024-04-14T15:20:40.075 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.075 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.075 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:20:40.075 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.066336+0000", 2024-04-14T15:20:40.076 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0071000000000001e-05 2024-04-14T15:20:40.076 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.076 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.076 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:20:40.076 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.096000+0000", 2024-04-14T15:20:40.076 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.029663666000000002 2024-04-14T15:20:40.076 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.076 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.076 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:20:40.076 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.096009+0000", 2024-04-14T15:20:40.076 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.2199999999999998e-06 2024-04-14T15:20:40.076 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.076 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.076 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:20:40.076 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.391307+0000", 2024-04-14T15:20:40.076 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2952977379999999 2024-04-14T15:20:40.077 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.077 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.077 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:20:40.077 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.391405+0000", 2024-04-14T15:20:40.077 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8735999999999998e-05 2024-04-14T15:20:40.077 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.077 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.077 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,4", 2024-04-14T15:20:40.077 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.391512+0000", 2024-04-14T15:20:40.077 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000106436 2024-04-14T15:20:40.077 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.077 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.077 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:20:40.077 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.397856+0000", 2024-04-14T15:20:40.077 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0063441529999999999 2024-04-14T15:20:40.078 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.078 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.078 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:20:40.078 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.411535+0000", 2024-04-14T15:20:40.078 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.013678968 2024-04-14T15:20:40.078 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.078 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.078 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:20:40.078 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.534946+0000", 2024-04-14T15:20:40.078 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.123411443 2024-04-14T15:20:40.078 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.078 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.078 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:20:40.078 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.534986+0000", 2024-04-14T15:20:40.078 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9972999999999998e-05 2024-04-14T15:20:40.079 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.079 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.079 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:20:40.079 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.535027+0000", 2024-04-14T15:20:40.079 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0599000000000002e-05 2024-04-14T15:20:40.079 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:20:40.079 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:20:40.079 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:20:40.079 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.079 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.079 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.24938.0:2 27.1 27:847a3d55:::obj:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-04-14T15:20:40.079 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:55.066927+0000", 2024-04-14T15:20:40.079 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 465.00436292500001, 2024-04-14T15:20:40.079 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4681723280000001, 2024-04-14T15:20:40.079 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:20:40.079 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:20:40.080 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:20:40.080 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:20:40.080 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.24938", 2024-04-14T15:20:40.080 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/3748273494", 2024-04-14T15:20:40.080 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 2 2024-04-14T15:20:40.080 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.080 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:20:40.080 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.080 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:20:40.080 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.066927+0000", 2024-04-14T15:20:40.080 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:20:40.080 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.080 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.080 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:20:40.080 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.066927+0000", 2024-04-14T15:20:40.080 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:20:40.080 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.081 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.081 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:20:40.081 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.066929+0000", 2024-04-14T15:20:40.081 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.528e-06 2024-04-14T15:20:40.081 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.081 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.081 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:20:40.081 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.066959+0000", 2024-04-14T15:20:40.081 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0093000000000001e-05 2024-04-14T15:20:40.081 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.081 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.081 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:20:40.081 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.066979+0000", 2024-04-14T15:20:40.081 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0183e-05 2024-04-14T15:20:40.081 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.081 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.082 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:20:40.082 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.066994+0000", 2024-04-14T15:20:40.082 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5218000000000001e-05 2024-04-14T15:20:40.082 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.082 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.082 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:20:40.082 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.110325+0000", 2024-04-14T15:20:40.082 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.043331135 2024-04-14T15:20:40.082 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.082 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.082 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:20:40.082 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:55.110335+0000", 2024-04-14T15:20:40.082 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.5540000000000003e-06 2024-04-14T15:20:40.082 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.082 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.083 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:20:40.083 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.388426+0000", 2024-04-14T15:20:40.083 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2780914050000001 2024-04-14T15:20:40.083 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.083 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.083 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:20:40.083 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.388545+0000", 2024-04-14T15:20:40.083 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011851800000000001 2024-04-14T15:20:40.083 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.083 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.083 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,6", 2024-04-14T15:20:40.083 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.388667+0000", 2024-04-14T15:20:40.083 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000121766 2024-04-14T15:20:40.083 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.083 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.084 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:20:40.084 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.391689+0000", 2024-04-14T15:20:40.084 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0030224330000000002 2024-04-14T15:20:40.084 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.084 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.084 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:20:40.084 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.432617+0000", 2024-04-14T15:20:40.084 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.040928216000000003 2024-04-14T15:20:40.084 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.084 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.084 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:20:40.084 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.535021+0000", 2024-04-14T15:20:40.084 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.102403801 2024-04-14T15:20:40.084 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.084 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.084 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:20:40.085 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.535062+0000", 2024-04-14T15:20:40.085 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0646999999999999e-05 2024-04-14T15:20:40.085 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.085 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.085 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:20:40.085 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.535100+0000", 2024-04-14T15:20:40.085 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.7830999999999997e-05 2024-04-14T15:20:40.086 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:20:40.086 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:20:40.086 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:20:40.086 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.086 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.086 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.24991.0:3 29.14 29:2a647cd4:::obj0:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e108)", 2024-04-14T15:20:40.086 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:56.074507+0000", 2024-04-14T15:20:40.086 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 463.996783069, 2024-04-14T15:20:40.086 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.17118332999999999, 2024-04-14T15:20:40.086 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:20:40.086 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:20:40.086 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-04-14T15:20:40.086 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:20:40.087 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.24991", 2024-04-14T15:20:40.087 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/3757631150", 2024-04-14T15:20:40.087 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 3 2024-04-14T15:20:40.087 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.087 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:20:40.087 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.087 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:20:40.087 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.074507+0000", 2024-04-14T15:20:40.087 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:20:40.087 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.087 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.087 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:20:40.087 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.074507+0000", 2024-04-14T15:20:40.087 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:20:40.087 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.087 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.088 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:20:40.088 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.074509+0000", 2024-04-14T15:20:40.088 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.984e-06 2024-04-14T15:20:40.088 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.088 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.088 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:20:40.088 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.074516+0000", 2024-04-14T15:20:40.088 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8580000000000002e-06 2024-04-14T15:20:40.088 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.088 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.088 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:20:40.088 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.074530+0000", 2024-04-14T15:20:40.088 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3966e-05 2024-04-14T15:20:40.088 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.088 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.088 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:20:40.089 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.074539+0000", 2024-04-14T15:20:40.089 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.5079999999999994e-06 2024-04-14T15:20:40.089 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.089 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.089 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:20:40.089 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.074616+0000", 2024-04-14T15:20:40.089 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.6228000000000003e-05 2024-04-14T15:20:40.089 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.089 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.089 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op must wait for map", 2024-04-14T15:20:40.089 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.074621+0000", 2024-04-14T15:20:40.089 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6219999999999997e-06 2024-04-14T15:20:40.089 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.089 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.089 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:20:40.089 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.215446+0000", 2024-04-14T15:20:40.090 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.140824962 2024-04-14T15:20:40.090 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.090 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.090 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:20:40.090 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.215454+0000", 2024-04-14T15:20:40.090 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1009999999999992e-06 2024-04-14T15:20:40.090 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.090 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.090 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:20:40.090 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.245596+0000", 2024-04-14T15:20:40.090 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.030142163999999999 2024-04-14T15:20:40.090 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.090 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.090 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:20:40.091 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.245690+0000", 2024-04-14T15:20:40.091 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.3937000000000005e-05 2024-04-14T15:20:40.091 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:20:40.091 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:20:40.091 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:20:40.091 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.091 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.091 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.24955.0:2 26.1e 26:7a7d4169:::src_object.1:head [writefull 0~4096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-04-14T15:20:40.091 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:56.075137+0000", 2024-04-14T15:20:40.091 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 463.99615307300002, 2024-04-14T15:20:40.091 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.18840111900000001, 2024-04-14T15:20:40.091 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:20:40.091 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:20:40.091 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:20:40.092 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:20:40.092 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.24955", 2024-04-14T15:20:40.092 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/117387996", 2024-04-14T15:20:40.092 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 2 2024-04-14T15:20:40.092 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.092 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:20:40.092 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.092 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:20:40.092 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.075137+0000", 2024-04-14T15:20:40.092 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:20:40.092 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.092 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.092 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:20:40.092 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.075137+0000", 2024-04-14T15:20:40.092 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:20:40.093 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.093 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.093 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:20:40.093 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.075138+0000", 2024-04-14T15:20:40.093 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.398e-06 2024-04-14T15:20:40.093 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.093 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.093 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:20:40.093 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.075149+0000", 2024-04-14T15:20:40.093 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.022e-05 2024-04-14T15:20:40.093 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.093 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.093 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:20:40.093 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.075161+0000", 2024-04-14T15:20:40.094 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2126999999999999e-05 2024-04-14T15:20:40.094 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.094 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.094 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:20:40.094 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.075169+0000", 2024-04-14T15:20:40.094 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3799999999999994e-06 2024-04-14T15:20:40.094 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.094 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.094 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:20:40.094 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.075243+0000", 2024-04-14T15:20:40.094 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3870999999999998e-05 2024-04-14T15:20:40.094 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.094 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.094 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op must wait for map", 2024-04-14T15:20:40.094 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.075249+0000", 2024-04-14T15:20:40.095 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9449999999999998e-06 2024-04-14T15:20:40.095 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.095 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.095 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:20:40.095 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.184989+0000", 2024-04-14T15:20:40.095 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.109740052 2024-04-14T15:20:40.095 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.095 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.095 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:20:40.095 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.184999+0000", 2024-04-14T15:20:40.095 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.5449999999999992e-06 2024-04-14T15:20:40.095 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.095 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.095 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:20:40.095 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.238236+0000", 2024-04-14T15:20:40.095 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.053237643000000001 2024-04-14T15:20:40.095 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.096 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.096 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:20:40.096 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.238399+0000", 2024-04-14T15:20:40.096 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016226899999999999 2024-04-14T15:20:40.096 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.096 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.096 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4,7", 2024-04-14T15:20:40.096 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.238556+0000", 2024-04-14T15:20:40.096 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015753499999999999 2024-04-14T15:20:40.096 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.096 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.096 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:20:40.096 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.241212+0000", 2024-04-14T15:20:40.096 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0026559270000000002 2024-04-14T15:20:40.097 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.097 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.097 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:20:40.097 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.252026+0000", 2024-04-14T15:20:40.097 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.010814414 2024-04-14T15:20:40.097 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.097 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.097 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:20:40.097 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.263440+0000", 2024-04-14T15:20:40.097 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.011413637000000001 2024-04-14T15:20:40.097 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.097 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.097 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:20:40.097 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.263493+0000", 2024-04-14T15:20:40.097 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3086e-05 2024-04-14T15:20:40.097 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.097 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.098 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:20:40.098 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.263538+0000", 2024-04-14T15:20:40.098 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.507e-05 2024-04-14T15:20:40.098 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:20:40.098 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:20:40.098 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:20:40.098 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.098 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.098 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.24811.0:3 10.9 10:9294e480:nspace::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-04-14T15:20:40.098 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:56.449492+0000", 2024-04-14T15:20:40.098 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 463.621798367, 2024-04-14T15:20:40.098 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.095302850999999994, 2024-04-14T15:20:40.098 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:20:40.098 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:20:40.098 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:20:40.099 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:20:40.099 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.24811", 2024-04-14T15:20:40.099 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/473680876", 2024-04-14T15:20:40.099 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 3 2024-04-14T15:20:40.099 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.099 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:20:40.099 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.099 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:20:40.099 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.449492+0000", 2024-04-14T15:20:40.099 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:20:40.099 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.099 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.099 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:20:40.099 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.449492+0000", 2024-04-14T15:20:40.099 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:20:40.100 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.100 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.100 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:20:40.100 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.449495+0000", 2024-04-14T15:20:40.100 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2669999999999998e-06 2024-04-14T15:20:40.100 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.100 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.100 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:20:40.100 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.449507+0000", 2024-04-14T15:20:40.100 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1997000000000001e-05 2024-04-14T15:20:40.100 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.100 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.100 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:20:40.100 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.449533+0000", 2024-04-14T15:20:40.101 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6392000000000001e-05 2024-04-14T15:20:40.101 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.101 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.101 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:20:40.101 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.449552+0000", 2024-04-14T15:20:40.101 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8938999999999999e-05 2024-04-14T15:20:40.101 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.101 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.101 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:20:40.101 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.449687+0000", 2024-04-14T15:20:40.101 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000134807 2024-04-14T15:20:40.101 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.101 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.101 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:20:40.102 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.449874+0000", 2024-04-14T15:20:40.102 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000187164 2024-04-14T15:20:40.102 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.102 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.102 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 2,7", 2024-04-14T15:20:40.102 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.449967+0000", 2024-04-14T15:20:40.102 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.3089000000000003e-05 2024-04-14T15:20:40.102 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.102 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.102 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:20:40.102 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.450284+0000", 2024-04-14T15:20:40.102 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00031640299999999999 2024-04-14T15:20:40.102 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.102 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.103 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:20:40.103 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.450580+0000", 2024-04-14T15:20:40.103 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000296011 2024-04-14T15:20:40.103 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.103 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.103 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:20:40.103 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.544700+0000", 2024-04-14T15:20:40.103 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.094120081999999994 2024-04-14T15:20:40.103 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.103 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.103 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:20:40.103 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.544752+0000", 2024-04-14T15:20:40.103 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.1919999999999998e-05 2024-04-14T15:20:40.103 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.103 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.103 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:20:40.104 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.544795+0000", 2024-04-14T15:20:40.104 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.278e-05 2024-04-14T15:20:40.104 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:20:40.104 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:20:40.104 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:20:40.104 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.104 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.104 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.24811.0:17 10.6 10:62743f70:test-rados-api-smithi090-93398-5::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-04-14T15:20:40.104 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:56.552466+0000", 2024-04-14T15:20:40.104 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 463.51882372799997, 2024-04-14T15:20:40.104 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.036935908000000003, 2024-04-14T15:20:40.104 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:20:40.104 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:20:40.104 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:20:40.104 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:20:40.105 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.24811", 2024-04-14T15:20:40.105 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/473680876", 2024-04-14T15:20:40.105 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 17 2024-04-14T15:20:40.105 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.105 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:20:40.105 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.105 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:20:40.105 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.552466+0000", 2024-04-14T15:20:40.105 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:20:40.105 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.105 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.105 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:20:40.105 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.552466+0000", 2024-04-14T15:20:40.105 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:20:40.105 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.105 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.106 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:20:40.106 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.552468+0000", 2024-04-14T15:20:40.106 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.384e-06 2024-04-14T15:20:40.106 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.106 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.106 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:20:40.106 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.552475+0000", 2024-04-14T15:20:40.106 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.6229999999999997e-06 2024-04-14T15:20:40.106 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.106 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.106 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:20:40.106 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.552482+0000", 2024-04-14T15:20:40.107 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9609999999999999e-06 2024-04-14T15:20:40.107 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.107 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.107 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:20:40.107 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.552489+0000", 2024-04-14T15:20:40.107 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1219999999999997e-06 2024-04-14T15:20:40.107 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.107 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.107 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:20:40.107 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.552544+0000", 2024-04-14T15:20:40.107 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4731999999999997e-05 2024-04-14T15:20:40.107 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.107 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.107 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:20:40.108 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.552657+0000", 2024-04-14T15:20:40.108 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011296199999999999 2024-04-14T15:20:40.108 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.108 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.108 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,4", 2024-04-14T15:20:40.108 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.552783+0000", 2024-04-14T15:20:40.108 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012542200000000001 2024-04-14T15:20:40.108 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.108 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.108 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:20:40.108 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.553193+0000", 2024-04-14T15:20:40.108 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000410734 2024-04-14T15:20:40.108 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.108 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.109 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:20:40.109 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.553471+0000", 2024-04-14T15:20:40.109 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00027812099999999999 2024-04-14T15:20:40.109 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.109 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.109 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:20:40.109 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.589310+0000", 2024-04-14T15:20:40.109 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.035838137999999999 2024-04-14T15:20:40.109 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.109 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.109 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:20:40.109 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.589371+0000", 2024-04-14T15:20:40.109 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1273999999999996e-05 2024-04-14T15:20:40.109 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.109 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.109 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:20:40.110 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:56.589402+0000", 2024-04-14T15:20:40.110 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1435e-05 2024-04-14T15:20:40.110 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:20:40.110 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:20:40.110 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:20:40.110 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.110 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.110 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.24853.0:433 16.1b 16:dbd443e1:test-rados-api-smithi090-93601-15::foo:head [setxattr a192 in=4b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-04-14T15:20:40.110 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:57.070215+0000", 2024-04-14T15:20:40.110 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 463.00107514600001, 2024-04-14T15:20:40.110 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.17665454899999999, 2024-04-14T15:20:40.110 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:20:40.110 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:20:40.110 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:20:40.110 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:20:40.110 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.24853", 2024-04-14T15:20:40.111 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/3851923713", 2024-04-14T15:20:40.111 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 433 2024-04-14T15:20:40.111 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.111 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:20:40.111 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.111 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:20:40.111 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.070215+0000", 2024-04-14T15:20:40.111 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:20:40.111 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.111 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.111 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:20:40.111 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.070215+0000", 2024-04-14T15:20:40.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:40.112 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:40.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:20:40.113 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:20:40.114 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.114 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.114 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:20:40.114 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.070217+0000", 2024-04-14T15:20:40.114 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.801e-06 2024-04-14T15:20:40.114 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.114 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.114 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:20:40.114 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.070224+0000", 2024-04-14T15:20:40.114 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8210000000000004e-06 2024-04-14T15:20:40.114 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.115 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.115 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:20:40.115 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.070249+0000", 2024-04-14T15:20:40.115 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.4995999999999999e-05 2024-04-14T15:20:40.115 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.115 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.115 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:20:40.115 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.070260+0000", 2024-04-14T15:20:40.115 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1574e-05 2024-04-14T15:20:40.115 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.115 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.115 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:20:40.115 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.079670+0000", 2024-04-14T15:20:40.116 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0094096590000000008 2024-04-14T15:20:40.116 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.116 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.116 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:20:40.116 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.079767+0000", 2024-04-14T15:20:40.116 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.7183999999999997e-05 2024-04-14T15:20:40.116 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.116 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.116 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1,6", 2024-04-14T15:20:40.116 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.079914+0000", 2024-04-14T15:20:40.116 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014677599999999999 2024-04-14T15:20:40.116 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.116 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.116 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:20:40.116 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.083998+0000", 2024-04-14T15:20:40.117 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0040846600000000004 2024-04-14T15:20:40.117 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.117 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.117 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:20:40.117 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.157521+0000", 2024-04-14T15:20:40.117 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.073522865000000007 2024-04-14T15:20:40.117 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.117 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.117 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:20:40.117 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.246599+0000", 2024-04-14T15:20:40.117 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.089077323 2024-04-14T15:20:40.117 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.117 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.117 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:20:40.117 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.246815+0000", 2024-04-14T15:20:40.117 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000216527 2024-04-14T15:20:40.118 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.118 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.118 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:20:40.118 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.246870+0000", 2024-04-14T15:20:40.118 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4363000000000001e-05 2024-04-14T15:20:40.118 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:20:40.118 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:20:40.118 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:20:40.118 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.118 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.118 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25001.0:1 44.6s0 44:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-04-14T15:20:40.118 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:57.082702+0000", 2024-04-14T15:20:40.118 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 462.98858853199999, 2024-04-14T15:20:40.119 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.118265955, 2024-04-14T15:20:40.119 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:20:40.119 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:20:40.119 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:20:40.119 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:20:40.119 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25001", 2024-04-14T15:20:40.119 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/1465058495", 2024-04-14T15:20:40.119 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 1 2024-04-14T15:20:40.119 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.119 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:20:40.119 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.119 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:20:40.119 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.082702+0000", 2024-04-14T15:20:40.119 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:20:40.120 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.120 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.120 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:20:40.120 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.082702+0000", 2024-04-14T15:20:40.120 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:20:40.120 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.120 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.120 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:20:40.120 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.082703+0000", 2024-04-14T15:20:40.120 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8789999999999999e-06 2024-04-14T15:20:40.120 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.120 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.120 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:20:40.120 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.082709+0000", 2024-04-14T15:20:40.120 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7139999999999998e-06 2024-04-14T15:20:40.120 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.121 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.121 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:20:40.121 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.082718+0000", 2024-04-14T15:20:40.121 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5490000000000006e-06 2024-04-14T15:20:40.121 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.121 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.121 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:20:40.121 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.082727+0000", 2024-04-14T15:20:40.121 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.2709999999999998e-06 2024-04-14T15:20:40.121 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.121 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.121 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:20:40.121 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.083560+0000", 2024-04-14T15:20:40.121 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00083254599999999998 2024-04-14T15:20:40.121 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.121 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.122 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:20:40.122 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.083562+0000", 2024-04-14T15:20:40.122 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6089999999999999e-06 2024-04-14T15:20:40.122 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.122 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.122 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:20:40.122 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.198059+0000", 2024-04-14T15:20:40.122 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.114496822 2024-04-14T15:20:40.122 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.122 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.122 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:20:40.122 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.198174+0000", 2024-04-14T15:20:40.122 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000115389 2024-04-14T15:20:40.122 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.122 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.122 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_started", 2024-04-14T15:20:40.123 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.198510+0000", 2024-04-14T15:20:40.123 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00033594199999999999 2024-04-14T15:20:40.123 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.123 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.123 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_committed", 2024-04-14T15:20:40.123 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.198835+0000", 2024-04-14T15:20:40.123 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000324743 2024-04-14T15:20:40.123 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.123 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.123 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:20:40.123 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.200916+0000", 2024-04-14T15:20:40.123 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0020809399999999999 2024-04-14T15:20:40.123 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.123 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.123 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:20:40.123 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.200968+0000", 2024-04-14T15:20:40.124 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.1551000000000002e-05 2024-04-14T15:20:40.124 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:20:40.124 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:20:40.124 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:20:40.124 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.124 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.124 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.24853.0:435 16.1b 16:dbd443e1:test-rados-api-smithi090-93601-15::foo:head [setxattr a193 in=4b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-04-14T15:20:40.124 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:57.248940+0000", 2024-04-14T15:20:40.124 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 462.82235007600002, 2024-04-14T15:20:40.124 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.094945007999999997, 2024-04-14T15:20:40.124 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:20:40.124 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:20:40.124 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:20:40.124 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:20:40.124 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.24853", 2024-04-14T15:20:40.124 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/3851923713", 2024-04-14T15:20:40.125 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 435 2024-04-14T15:20:40.125 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.125 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:20:40.125 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.125 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:20:40.125 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.248940+0000", 2024-04-14T15:20:40.125 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:20:40.125 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.125 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.125 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:20:40.125 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.248940+0000", 2024-04-14T15:20:40.125 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:20:40.125 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.125 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.125 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:20:40.125 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.248942+0000", 2024-04-14T15:20:40.126 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.646e-06 2024-04-14T15:20:40.126 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.126 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.126 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:20:40.126 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.248949+0000", 2024-04-14T15:20:40.126 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.4889999999999998e-06 2024-04-14T15:20:40.126 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.126 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.126 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:20:40.126 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.248965+0000", 2024-04-14T15:20:40.126 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5620999999999998e-05 2024-04-14T15:20:40.126 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.126 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.126 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:20:40.126 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.248974+0000", 2024-04-14T15:20:40.126 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.1570000000000006e-06 2024-04-14T15:20:40.127 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.127 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.127 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:20:40.127 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.249027+0000", 2024-04-14T15:20:40.127 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3229999999999997e-05 2024-04-14T15:20:40.127 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.127 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.127 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:20:40.127 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.249109+0000", 2024-04-14T15:20:40.127 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1914000000000002e-05 2024-04-14T15:20:40.127 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.127 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.127 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1,6", 2024-04-14T15:20:40.127 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.249252+0000", 2024-04-14T15:20:40.127 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000143356 2024-04-14T15:20:40.127 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.128 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.128 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:20:40.128 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.249903+0000", 2024-04-14T15:20:40.128 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000651007 2024-04-14T15:20:40.128 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.128 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.128 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:20:40.128 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.253012+0000", 2024-04-14T15:20:40.128 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0031087699999999998 2024-04-14T15:20:40.128 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.128 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.128 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:20:40.128 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.343821+0000", 2024-04-14T15:20:40.128 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.090809216999999998 2024-04-14T15:20:40.129 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.129 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.129 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:20:40.129 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.343857+0000", 2024-04-14T15:20:40.129 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5116000000000001e-05 2024-04-14T15:20:40.129 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.129 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.129 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:20:40.129 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:57.343885+0000", 2024-04-14T15:20:40.129 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8484999999999999e-05 2024-04-14T15:20:40.129 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:20:40.129 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:20:40.129 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:20:40.129 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.129 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.129 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25156.0:2 51.0 51:0384ab8e:::oid:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-04-14T15:20:40.130 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:58.094739+0000", 2024-04-14T15:20:40.130 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 461.97655094499999, 2024-04-14T15:20:40.130 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2282847750000001, 2024-04-14T15:20:40.130 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:20:40.130 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:20:40.130 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:20:40.130 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:20:40.130 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25156", 2024-04-14T15:20:40.130 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/2499391840", 2024-04-14T15:20:40.130 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 2 2024-04-14T15:20:40.130 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.130 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:20:40.130 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.130 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:20:40.130 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.094739+0000", 2024-04-14T15:20:40.130 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:20:40.131 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.131 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.131 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:20:40.131 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.094739+0000", 2024-04-14T15:20:40.131 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:20:40.131 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.131 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.131 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:20:40.131 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.094756+0000", 2024-04-14T15:20:40.131 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7054e-05 2024-04-14T15:20:40.131 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.131 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.131 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:20:40.131 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.094764+0000", 2024-04-14T15:20:40.131 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0260000000000007e-06 2024-04-14T15:20:40.132 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.132 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.132 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:20:40.132 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.094774+0000", 2024-04-14T15:20:40.132 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.9080000000000007e-06 2024-04-14T15:20:40.132 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.132 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.132 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:20:40.132 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.094784+0000", 2024-04-14T15:20:40.132 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0139999999999999e-05 2024-04-14T15:20:40.132 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.132 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.132 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:20:40.132 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.095687+0000", 2024-04-14T15:20:40.132 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00090298600000000004 2024-04-14T15:20:40.132 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.133 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.133 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:20:40.133 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.095695+0000", 2024-04-14T15:20:40.133 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1200000000000002e-06 2024-04-14T15:20:40.133 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.133 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.133 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:20:40.133 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.287077+0000", 2024-04-14T15:20:40.133 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.191381926 2024-04-14T15:20:40.133 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.133 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.133 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:20:40.133 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.287203+0000", 2024-04-14T15:20:40.133 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000125635 2024-04-14T15:20:40.133 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.133 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.134 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,2", 2024-04-14T15:20:40.134 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.287329+0000", 2024-04-14T15:20:40.134 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000126105 2024-04-14T15:20:40.134 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.134 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.134 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:20:40.134 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.287810+0000", 2024-04-14T15:20:40.134 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000481387 2024-04-14T15:20:40.134 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.134 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.134 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:20:40.134 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.292936+0000", 2024-04-14T15:20:40.134 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0051251480000000004 2024-04-14T15:20:40.134 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.134 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.135 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:20:40.135 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.322940+0000", 2024-04-14T15:20:40.135 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.030003906 2024-04-14T15:20:40.135 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.135 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.135 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:20:40.135 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.322984+0000", 2024-04-14T15:20:40.135 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.4869e-05 2024-04-14T15:20:40.135 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.135 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.135 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:20:40.135 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.323024+0000", 2024-04-14T15:20:40.135 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9564999999999998e-05 2024-04-14T15:20:40.135 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:20:40.135 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:20:40.136 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:20:40.136 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.136 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.136 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.24853.0:1223 16.1b 16:dbd443e1:test-rados-api-smithi090-93601-15::foo:head [setxattr a587 in=4b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-04-14T15:20:40.136 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:58.099323+0000", 2024-04-14T15:20:40.136 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 461.97196726700003, 2024-04-14T15:20:40.136 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.097390662000000003, 2024-04-14T15:20:40.136 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:20:40.136 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:20:40.136 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:20:40.136 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:20:40.136 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.24853", 2024-04-14T15:20:40.136 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/3851923713", 2024-04-14T15:20:40.136 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 1223 2024-04-14T15:20:40.136 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.136 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:20:40.137 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.137 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:20:40.137 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.099323+0000", 2024-04-14T15:20:40.137 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:20:40.137 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.137 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.137 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:20:40.137 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.099323+0000", 2024-04-14T15:20:40.137 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:20:40.137 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.137 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.137 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:20:40.137 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.099325+0000", 2024-04-14T15:20:40.137 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.945e-06 2024-04-14T15:20:40.137 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.138 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.138 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:20:40.138 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.099333+0000", 2024-04-14T15:20:40.138 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.8369999999999997e-06 2024-04-14T15:20:40.138 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.138 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.138 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:20:40.138 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.099344+0000", 2024-04-14T15:20:40.138 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1307e-05 2024-04-14T15:20:40.138 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.138 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.138 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:20:40.138 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.099354+0000", 2024-04-14T15:20:40.138 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.9199999999999999e-06 2024-04-14T15:20:40.138 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.139 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.139 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:20:40.139 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.099507+0000", 2024-04-14T15:20:40.139 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000153156 2024-04-14T15:20:40.139 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.139 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.139 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:20:40.139 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.099597+0000", 2024-04-14T15:20:40.139 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0169999999999999e-05 2024-04-14T15:20:40.139 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.139 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.139 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1,6", 2024-04-14T15:20:40.139 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.099730+0000", 2024-04-14T15:20:40.139 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000133144 2024-04-14T15:20:40.139 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.140 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.140 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:20:40.140 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.102087+0000", 2024-04-14T15:20:40.140 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002356814 2024-04-14T15:20:40.140 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.140 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.140 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:20:40.140 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.164910+0000", 2024-04-14T15:20:40.140 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.062822391000000005 2024-04-14T15:20:40.140 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.140 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.140 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:20:40.140 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.196609+0000", 2024-04-14T15:20:40.140 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.031699790999999998 2024-04-14T15:20:40.141 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.141 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.141 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:20:40.141 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.196671+0000", 2024-04-14T15:20:40.141 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2024000000000001e-05 2024-04-14T15:20:40.141 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.141 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.141 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:20:40.141 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.196714+0000", 2024-04-14T15:20:40.141 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2163000000000003e-05 2024-04-14T15:20:40.141 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:20:40.141 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:20:40.141 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:20:40.141 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.142 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.142 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.24853.0:1853 16.1b 16:dbd443e1:test-rados-api-smithi090-93601-15::foo:head [setxattr a902 in=4b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-04-14T15:20:40.142 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:58.610106+0000", 2024-04-14T15:20:40.142 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 461.46118374399998, 2024-04-14T15:20:40.142 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.068056710000000006, 2024-04-14T15:20:40.142 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:20:40.142 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:20:40.142 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:20:40.142 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:20:40.142 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.24853", 2024-04-14T15:20:40.142 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/3851923713", 2024-04-14T15:20:40.142 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 1853 2024-04-14T15:20:40.142 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.142 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:20:40.142 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.143 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:20:40.143 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.610106+0000", 2024-04-14T15:20:40.143 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:20:40.143 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.143 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.143 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:20:40.143 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.610106+0000", 2024-04-14T15:20:40.143 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:20:40.143 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.143 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.143 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:20:40.143 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.610107+0000", 2024-04-14T15:20:40.143 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.3500000000000005e-07 2024-04-14T15:20:40.143 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.143 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.143 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:20:40.144 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.610112+0000", 2024-04-14T15:20:40.144 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.87e-06 2024-04-14T15:20:40.144 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.144 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.144 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:20:40.144 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.610122+0000", 2024-04-14T15:20:40.144 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0161e-05 2024-04-14T15:20:40.144 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.144 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.144 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:20:40.144 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.610129+0000", 2024-04-14T15:20:40.144 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8029999999999999e-06 2024-04-14T15:20:40.144 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.144 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.144 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:20:40.144 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.610174+0000", 2024-04-14T15:20:40.145 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.4799999999999998e-05 2024-04-14T15:20:40.145 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.145 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.145 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:20:40.145 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.610234+0000", 2024-04-14T15:20:40.145 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0414999999999997e-05 2024-04-14T15:20:40.145 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.145 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.145 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1,6", 2024-04-14T15:20:40.145 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.610309+0000", 2024-04-14T15:20:40.145 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.4377999999999998e-05 2024-04-14T15:20:40.145 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.145 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.145 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:20:40.145 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.611179+0000", 2024-04-14T15:20:40.145 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00087030300000000001 2024-04-14T15:20:40.146 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.146 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.146 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:20:40.146 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.611328+0000", 2024-04-14T15:20:40.146 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000148999 2024-04-14T15:20:40.146 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.146 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.146 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:20:40.146 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.678095+0000", 2024-04-14T15:20:40.146 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.066766958000000001 2024-04-14T15:20:40.146 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.146 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.146 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:20:40.146 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.678130+0000", 2024-04-14T15:20:40.146 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5423999999999998e-05 2024-04-14T15:20:40.147 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.147 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.147 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:20:40.147 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.678163+0000", 2024-04-14T15:20:40.147 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2663999999999998e-05 2024-04-14T15:20:40.147 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:20:40.147 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:20:40.147 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:20:40.147 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.147 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.147 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.24835.0:15 14.a 14:57934e76:test-rados-api-smithi090-93476-7::foo:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-04-14T15:20:40.147 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:58.632278+0000", 2024-04-14T15:20:40.147 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 461.439012617, 2024-04-14T15:20:40.147 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.046282352999999998, 2024-04-14T15:20:40.147 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:20:40.148 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:20:40.148 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:20:40.148 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:20:40.148 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.24835", 2024-04-14T15:20:40.148 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/1412026529", 2024-04-14T15:20:40.148 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 15 2024-04-14T15:20:40.148 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.148 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:20:40.148 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.148 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:20:40.148 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.632278+0000", 2024-04-14T15:20:40.148 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:20:40.148 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.148 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.148 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:20:40.149 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.632278+0000", 2024-04-14T15:20:40.149 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:20:40.149 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.149 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.149 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:20:40.149 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.632279+0000", 2024-04-14T15:20:40.149 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4670000000000001e-06 2024-04-14T15:20:40.149 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.149 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.149 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:20:40.149 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.632284+0000", 2024-04-14T15:20:40.149 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7079999999999996e-06 2024-04-14T15:20:40.149 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.149 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.149 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:20:40.150 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.632292+0000", 2024-04-14T15:20:40.150 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.9019999999999999e-06 2024-04-14T15:20:40.150 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.150 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.150 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:20:40.150 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.632297+0000", 2024-04-14T15:20:40.150 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6230000000000002e-06 2024-04-14T15:20:40.150 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.150 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.150 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:20:40.150 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.632345+0000", 2024-04-14T15:20:40.150 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7737000000000001e-05 2024-04-14T15:20:40.150 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.150 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.151 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:20:40.151 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.632423+0000", 2024-04-14T15:20:40.151 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.8055999999999999e-05 2024-04-14T15:20:40.151 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.151 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.151 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1,3", 2024-04-14T15:20:40.151 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.632537+0000", 2024-04-14T15:20:40.151 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000113689 2024-04-14T15:20:40.151 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.151 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.151 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:20:40.151 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.632805+0000", 2024-04-14T15:20:40.151 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00026789099999999997 2024-04-14T15:20:40.151 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.151 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.152 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:20:40.152 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.633077+0000", 2024-04-14T15:20:40.152 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000272279 2024-04-14T15:20:40.152 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.152 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.152 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:20:40.152 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.678497+0000", 2024-04-14T15:20:40.152 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.045419757999999998 2024-04-14T15:20:40.152 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.152 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.152 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:20:40.152 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.678526+0000", 2024-04-14T15:20:40.152 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9434e-05 2024-04-14T15:20:40.152 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.152 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.153 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:20:40.153 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:58.678560+0000", 2024-04-14T15:20:40.153 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.3809000000000001e-05 2024-04-14T15:20:40.153 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:20:40.153 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:20:40.153 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:20:40.153 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.153 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.153 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.24853.0:2070 16.12 e111/107 16:4b6e22cc:test-rados-api-smithi090-93601-17::big2:head v 111'7, mlcod=108'5)", 2024-04-14T15:20:40.153 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:59.459220+0000", 2024-04-14T15:20:40.153 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 460.612069663, 2024-04-14T15:20:40.153 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.061580874000000001, 2024-04-14T15:20:40.153 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:20:40.153 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:20:40.153 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:20:40.154 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:20:40.154 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.154 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:20:40.154 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.459220+0000", 2024-04-14T15:20:40.154 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:20:40.154 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.154 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.154 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:20:40.154 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.459220+0000", 2024-04-14T15:20:40.154 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:20:40.154 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.154 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.154 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:20:40.154 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.459222+0000", 2024-04-14T15:20:40.154 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4300000000000001e-06 2024-04-14T15:20:40.155 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.155 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.155 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:20:40.155 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.471887+0000", 2024-04-14T15:20:40.155 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.012665239 2024-04-14T15:20:40.155 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.155 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.155 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:20:40.155 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.471912+0000", 2024-04-14T15:20:40.155 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.4428000000000001e-05 2024-04-14T15:20:40.155 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.155 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.155 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:20:40.155 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.471923+0000", 2024-04-14T15:20:40.155 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1052e-05 2024-04-14T15:20:40.156 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.156 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.156 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:20:40.156 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.471994+0000", 2024-04-14T15:20:40.156 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1526000000000006e-05 2024-04-14T15:20:40.156 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.156 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.156 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:20:40.156 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.472022+0000", 2024-04-14T15:20:40.156 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.7789000000000001e-05 2024-04-14T15:20:40.156 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.156 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.156 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:20:40.156 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.520754+0000", 2024-04-14T15:20:40.157 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.048732005000000002 2024-04-14T15:20:40.157 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.157 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.157 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:20:40.157 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.520801+0000", 2024-04-14T15:20:40.157 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7404999999999998e-05 2024-04-14T15:20:40.157 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:20:40.157 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:20:40.157 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:20:40.157 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.157 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.157 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.24853.0:2074 16.c 16:31d79ce4:test-rados-api-smithi090-93601-17::big3:head [writefull 0~25165828] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-14T15:20:40.157 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:12:59.550683+0000", 2024-04-14T15:20:40.157 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 460.520606638, 2024-04-14T15:20:40.157 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.087008673999999994, 2024-04-14T15:20:40.157 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:20:40.158 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:20:40.158 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:20:40.158 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:20:40.158 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.24853", 2024-04-14T15:20:40.158 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/3851923713", 2024-04-14T15:20:40.158 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 2074 2024-04-14T15:20:40.158 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.158 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:20:40.158 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.158 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:20:40.158 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.550683+0000", 2024-04-14T15:20:40.158 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:20:40.158 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.158 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.158 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:20:40.159 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.550683+0000", 2024-04-14T15:20:40.159 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:20:40.159 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.159 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.159 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:20:40.159 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.550685+0000", 2024-04-14T15:20:40.159 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9e-06 2024-04-14T15:20:40.159 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.159 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.159 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:20:40.159 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.574320+0000", 2024-04-14T15:20:40.159 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.023634131999999999 2024-04-14T15:20:40.159 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.159 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.159 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:20:40.159 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.574340+0000", 2024-04-14T15:20:40.160 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0344999999999999e-05 2024-04-14T15:20:40.160 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.160 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.160 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:20:40.160 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.574352+0000", 2024-04-14T15:20:40.160 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2408999999999999e-05 2024-04-14T15:20:40.160 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.160 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.160 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:20:40.160 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.574415+0000", 2024-04-14T15:20:40.160 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2867000000000006e-05 2024-04-14T15:20:40.160 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.160 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.160 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:20:40.160 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.574521+0000", 2024-04-14T15:20:40.160 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010585 2024-04-14T15:20:40.161 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.161 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.161 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,4", 2024-04-14T15:20:40.161 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.574635+0000", 2024-04-14T15:20:40.161 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000114028 2024-04-14T15:20:40.161 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.161 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.161 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:20:40.161 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.628088+0000", 2024-04-14T15:20:40.161 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.053452491999999997 2024-04-14T15:20:40.161 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.161 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.161 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:20:40.161 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.637609+0000", 2024-04-14T15:20:40.162 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0095212950000000008 2024-04-14T15:20:40.162 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.162 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.162 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:20:40.162 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.637620+0000", 2024-04-14T15:20:40.162 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0696000000000001e-05 2024-04-14T15:20:40.162 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.162 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.162 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:20:40.162 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.637656+0000", 2024-04-14T15:20:40.162 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6294000000000002e-05 2024-04-14T15:20:40.162 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.162 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.162 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:20:40.163 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:12:59.637692+0000", 2024-04-14T15:20:40.163 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6365999999999997e-05 2024-04-14T15:20:40.163 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:20:40.163 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:20:40.163 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:20:40.163 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.163 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.163 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25240.0:1 67.0s0 67:005d9b81:test-rados-api-smithi090-93379-17::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-04-14T15:20:40.163 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:13:00.177410+0000", 2024-04-14T15:20:40.163 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 459.89388032699998, 2024-04-14T15:20:40.163 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.77316005899999996, 2024-04-14T15:20:40.163 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:20:40.163 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:20:40.163 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:20:40.163 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:20:40.163 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25240", 2024-04-14T15:20:40.163 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/3178004355", 2024-04-14T15:20:40.164 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 1 2024-04-14T15:20:40.164 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.164 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:20:40.164 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.164 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:20:40.164 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.177410+0000", 2024-04-14T15:20:40.164 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:20:40.164 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.164 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.164 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:20:40.164 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.177410+0000", 2024-04-14T15:20:40.164 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:20:40.164 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.164 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.165 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:20:40.165 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.177412+0000", 2024-04-14T15:20:40.165 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3939999999999999e-06 2024-04-14T15:20:40.165 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.165 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.165 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:20:40.165 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.177421+0000", 2024-04-14T15:20:40.165 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.1949999999999992e-06 2024-04-14T15:20:40.165 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.165 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.165 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:20:40.165 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.178275+0000", 2024-04-14T15:20:40.165 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00085384199999999995 2024-04-14T15:20:40.165 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.165 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.165 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:20:40.166 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.178289+0000", 2024-04-14T15:20:40.166 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3935e-05 2024-04-14T15:20:40.166 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.166 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.166 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:20:40.166 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.178811+0000", 2024-04-14T15:20:40.166 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00052192000000000004 2024-04-14T15:20:40.166 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.166 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.166 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:20:40.166 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.178814+0000", 2024-04-14T15:20:40.166 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.65e-06 2024-04-14T15:20:40.166 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.166 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.167 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:20:40.167 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.936581+0000", 2024-04-14T15:20:40.167 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.75776740499999995 2024-04-14T15:20:40.167 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.167 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.167 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:20:40.167 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.936707+0000", 2024-04-14T15:20:40.167 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000126119 2024-04-14T15:20:40.167 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.167 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.167 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_started", 2024-04-14T15:20:40.167 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.937166+0000", 2024-04-14T15:20:40.167 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00045856599999999999 2024-04-14T15:20:40.167 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.167 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.167 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_committed", 2024-04-14T15:20:40.168 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.937681+0000", 2024-04-14T15:20:40.168 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00051527300000000001 2024-04-14T15:20:40.168 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.168 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.168 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:20:40.168 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.950484+0000", 2024-04-14T15:20:40.168 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.012803162 2024-04-14T15:20:40.168 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.168 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.168 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:20:40.168 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:00.950570+0000", 2024-04-14T15:20:40.168 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5598000000000003e-05 2024-04-14T15:20:40.168 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:20:40.168 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:20:40.168 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:20:40.168 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.169 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.169 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.36752.0:2 145.0 145:0384ab8e:::oid:head [append 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-04-14T15:20:40.169 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:13:21.821544+0000", 2024-04-14T15:20:40.169 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 438.24974636299999, 2024-04-14T15:20:40.169 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1791939170000001, 2024-04-14T15:20:40.169 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:20:40.169 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:20:40.169 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:20:40.169 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:20:40.169 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.36752", 2024-04-14T15:20:40.169 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/3990091175", 2024-04-14T15:20:40.169 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 2 2024-04-14T15:20:40.169 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.169 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:20:40.169 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.170 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:20:40.170 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:21.821544+0000", 2024-04-14T15:20:40.170 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:20:40.170 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.170 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.170 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:20:40.170 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:21.821544+0000", 2024-04-14T15:20:40.170 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:20:40.170 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.170 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.170 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:20:40.170 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:21.821546+0000", 2024-04-14T15:20:40.170 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.889e-06 2024-04-14T15:20:40.170 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.170 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.171 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:20:40.171 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:21.821554+0000", 2024-04-14T15:20:40.171 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1019999999999997e-06 2024-04-14T15:20:40.171 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.171 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.171 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:20:40.171 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:21.821564+0000", 2024-04-14T15:20:40.171 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0441999999999999e-05 2024-04-14T15:20:40.171 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.171 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.171 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:20:40.171 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:21.822878+0000", 2024-04-14T15:20:40.171 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001313495 2024-04-14T15:20:40.171 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.171 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.172 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:20:40.172 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:21.831489+0000", 2024-04-14T15:20:40.172 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0086117720000000002 2024-04-14T15:20:40.172 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.172 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.172 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:20:40.172 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:21.831499+0000", 2024-04-14T15:20:40.172 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.2949999999999999e-06 2024-04-14T15:20:40.172 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.172 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.172 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:20:40.172 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:22.959687+0000", 2024-04-14T15:20:40.172 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1281882809999999 2024-04-14T15:20:40.172 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.172 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.173 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:20:40.173 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:22.959817+0000", 2024-04-14T15:20:40.173 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012978999999999999 2024-04-14T15:20:40.173 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.173 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.173 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 2,6", 2024-04-14T15:20:40.173 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:22.959961+0000", 2024-04-14T15:20:40.173 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014379200000000001 2024-04-14T15:20:40.173 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.173 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.173 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:20:40.173 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:22.961030+0000", 2024-04-14T15:20:40.173 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0010694400000000001 2024-04-14T15:20:40.173 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.173 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.173 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:20:40.174 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:22.961681+0000", 2024-04-14T15:20:40.174 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00065044799999999998 2024-04-14T15:20:40.174 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.174 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.174 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:20:40.174 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:23.000643+0000", 2024-04-14T15:20:40.174 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.038962055000000002 2024-04-14T15:20:40.174 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.174 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.174 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:20:40.174 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:23.000696+0000", 2024-04-14T15:20:40.174 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3875999999999997e-05 2024-04-14T15:20:40.174 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.175 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.175 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:20:40.175 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:23.000738+0000", 2024-04-14T15:20:40.175 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1239999999999998e-05 2024-04-14T15:20:40.175 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:20:40.175 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:20:40.175 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:20:40.175 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.175 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.175 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.44341.0:2 161.3 161:c656db43:::oid_RoundTripPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-14T15:20:40.175 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:13:28.864198+0000", 2024-04-14T15:20:40.175 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 431.20709249200002, 2024-04-14T15:20:40.175 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.083493583999999996, 2024-04-14T15:20:40.175 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:20:40.175 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:20:40.175 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:20:40.176 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:20:40.176 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.44341", 2024-04-14T15:20:40.176 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/4039543108", 2024-04-14T15:20:40.176 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 2 2024-04-14T15:20:40.176 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.176 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:20:40.176 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.176 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:20:40.176 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.864198+0000", 2024-04-14T15:20:40.176 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:20:40.176 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.176 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.176 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:20:40.176 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.864198+0000", 2024-04-14T15:20:40.176 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:20:40.176 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.177 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.177 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:20:40.177 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.864200+0000", 2024-04-14T15:20:40.177 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.018e-06 2024-04-14T15:20:40.177 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.177 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.177 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:20:40.177 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.864207+0000", 2024-04-14T15:20:40.177 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9280000000000004e-06 2024-04-14T15:20:40.177 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.177 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.177 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:20:40.177 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.864217+0000", 2024-04-14T15:20:40.177 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0249e-05 2024-04-14T15:20:40.177 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.178 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.178 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:20:40.178 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.864226+0000", 2024-04-14T15:20:40.178 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.2240000000000001e-06 2024-04-14T15:20:40.178 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.178 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.178 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:20:40.178 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.864291+0000", 2024-04-14T15:20:40.178 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5009e-05 2024-04-14T15:20:40.178 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.178 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.178 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:20:40.178 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.864299+0000", 2024-04-14T15:20:40.178 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.9570000000000002e-06 2024-04-14T15:20:40.179 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.179 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.179 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:20:40.179 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.906801+0000", 2024-04-14T15:20:40.179 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.042502054999999997 2024-04-14T15:20:40.179 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.179 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.179 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:20:40.179 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.906887+0000", 2024-04-14T15:20:40.179 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6168000000000006e-05 2024-04-14T15:20:40.179 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.179 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.179 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,3", 2024-04-14T15:20:40.180 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.907005+0000", 2024-04-14T15:20:40.180 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000117281 2024-04-14T15:20:40.180 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.180 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.180 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:20:40.180 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.907458+0000", 2024-04-14T15:20:40.180 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00045332400000000001 2024-04-14T15:20:40.180 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.180 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.180 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:20:40.180 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.914412+0000", 2024-04-14T15:20:40.180 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0069537720000000004 2024-04-14T15:20:40.180 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.180 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.180 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:20:40.180 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.947618+0000", 2024-04-14T15:20:40.181 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.033206688999999998 2024-04-14T15:20:40.181 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.181 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.181 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:20:40.181 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.947658+0000", 2024-04-14T15:20:40.181 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9946e-05 2024-04-14T15:20:40.181 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.181 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.181 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:20:40.181 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.947691+0000", 2024-04-14T15:20:40.181 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2963999999999998e-05 2024-04-14T15:20:40.181 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:20:40.181 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:20:40.181 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:20:40.181 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.181 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.181 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.24853.0:2079 16.11 16:8af1a83e:test-rados-api-smithi090-93601-17::big2.copy:head [copy-from ver 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-14T15:20:40.182 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:13:29.771058+0000", 2024-04-14T15:20:40.182 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 430.30023247399998, 2024-04-14T15:20:40.182 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.113103565, 2024-04-14T15:20:40.182 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:20:40.182 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:20:40.182 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:20:40.182 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:20:40.182 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.24853", 2024-04-14T15:20:40.182 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/3851923713", 2024-04-14T15:20:40.182 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 2079 2024-04-14T15:20:40.182 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.182 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:20:40.182 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.183 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:20:40.183 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:29.771058+0000", 2024-04-14T15:20:40.183 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:20:40.183 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.183 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.183 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:20:40.183 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:29.771058+0000", 2024-04-14T15:20:40.183 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:20:40.183 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.183 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.183 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:20:40.183 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:29.771061+0000", 2024-04-14T15:20:40.183 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.292e-06 2024-04-14T15:20:40.183 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.183 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.184 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:20:40.184 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:29.771071+0000", 2024-04-14T15:20:40.184 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0317e-05 2024-04-14T15:20:40.184 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.184 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.184 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:20:40.184 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:29.771093+0000", 2024-04-14T15:20:40.184 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1461999999999999e-05 2024-04-14T15:20:40.184 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.184 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.184 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:20:40.184 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:29.771105+0000", 2024-04-14T15:20:40.184 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2128e-05 2024-04-14T15:20:40.184 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.184 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.185 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:20:40.185 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:29.771208+0000", 2024-04-14T15:20:40.185 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010289300000000001 2024-04-14T15:20:40.185 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.185 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.185 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:20:40.185 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:29.771354+0000", 2024-04-14T15:20:40.185 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014649899999999999 2024-04-14T15:20:40.185 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.185 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.185 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,1", 2024-04-14T15:20:40.185 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:29.832534+0000", 2024-04-14T15:20:40.185 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.061179807000000003 2024-04-14T15:20:40.185 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.185 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.186 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:20:40.186 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:29.839859+0000", 2024-04-14T15:20:40.186 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0073249099999999996 2024-04-14T15:20:40.186 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.186 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.186 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:20:40.186 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:29.867794+0000", 2024-04-14T15:20:40.186 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.027935206000000001 2024-04-14T15:20:40.186 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.186 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.186 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:20:40.186 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:29.884083+0000", 2024-04-14T15:20:40.186 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.016289051999999998 2024-04-14T15:20:40.186 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.187 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.187 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:20:40.187 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:29.884125+0000", 2024-04-14T15:20:40.187 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1943000000000001e-05 2024-04-14T15:20:40.187 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.187 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.187 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:20:40.187 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:29.884161+0000", 2024-04-14T15:20:40.187 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6056000000000002e-05 2024-04-14T15:20:40.187 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:20:40.187 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:20:40.187 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:20:40.187 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.187 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.187 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.46400.0:2 202.11 e184/182 202:8de73fe4:test-rados-api-smithi090-93601-26::foo:head v 184'1, mlcod=0'0)", 2024-04-14T15:20:40.188 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:14:11.852282+0000", 2024-04-14T15:20:40.188 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 388.21900768299997, 2024-04-14T15:20:40.188 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.040243552000000002, 2024-04-14T15:20:40.188 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:20:40.188 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:20:40.188 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:20:40.188 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:20:40.188 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.188 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:20:40.188 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:11.852282+0000", 2024-04-14T15:20:40.188 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:20:40.188 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.188 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.188 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:20:40.188 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:11.852282+0000", 2024-04-14T15:20:40.188 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:20:40.189 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.189 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.189 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:20:40.189 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:11.852283+0000", 2024-04-14T15:20:40.189 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1300000000000001e-07 2024-04-14T15:20:40.189 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.189 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.189 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:20:40.189 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:11.876535+0000", 2024-04-14T15:20:40.189 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.024252234000000001 2024-04-14T15:20:40.189 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.189 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.189 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:20:40.189 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:11.876562+0000", 2024-04-14T15:20:40.189 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.654e-05 2024-04-14T15:20:40.190 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.190 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.190 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:20:40.190 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:11.876577+0000", 2024-04-14T15:20:40.190 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5503999999999999e-05 2024-04-14T15:20:40.190 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.190 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.190 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:20:40.190 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:11.876660+0000", 2024-04-14T15:20:40.190 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2367999999999995e-05 2024-04-14T15:20:40.190 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.190 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.190 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:20:40.191 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:11.876681+0000", 2024-04-14T15:20:40.191 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1022999999999998e-05 2024-04-14T15:20:40.191 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.191 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.191 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:20:40.191 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:11.892462+0000", 2024-04-14T15:20:40.191 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.015781309 2024-04-14T15:20:40.191 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:20:40.191 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:20:40.191 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:20:40.191 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:11.892526+0000", 2024-04-14T15:20:40.191 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4160999999999998e-05 2024-04-14T15:20:40.191 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:20:40.191 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:20:40.191 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:20:40.192 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:20:40.192 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:20:40.192 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-04-14T15:20:40.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:39 smithi090 ceph-mon[41670]: 3.2 scrub starts 2024-04-14T15:20:40.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:39 smithi090 ceph-mon[41670]: 3.2 scrub ok 2024-04-14T15:20:40.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:39 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-14T15:20:40.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:39 smithi090 ceph-mon[41670]: osdmap e514: 8 total, 8 up, 8 in 2024-04-14T15:20:40.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:39 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-83","var": "dedup_tier","val": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:20:40.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:39 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-83","var": "dedup_tier","val": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:20:40.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:39 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-04-14T15:20:40.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:39 smithi090 ceph-mon[30399]: 3.2 scrub starts 2024-04-14T15:20:40.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:39 smithi090 ceph-mon[30399]: 3.2 scrub ok 2024-04-14T15:20:40.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:39 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-14T15:20:40.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:39 smithi090 ceph-mon[30399]: osdmap e514: 8 total, 8 up, 8 in 2024-04-14T15:20:40.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:39 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-83","var": "dedup_tier","val": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:20:40.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:39 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-83","var": "dedup_tier","val": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:20:40.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:39 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-04-14T15:20:40.214 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:40.237 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:40.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:40.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:40.339 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:20:40.363 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:40.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:40.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:20:40.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:39 smithi092 ceph-mon[33966]: 3.2 scrub starts 2024-04-14T15:20:40.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:39 smithi092 ceph-mon[33966]: 3.2 scrub ok 2024-04-14T15:20:40.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:39 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-14T15:20:40.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:39 smithi092 ceph-mon[33966]: osdmap e514: 8 total, 8 up, 8 in 2024-04-14T15:20:40.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:39 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-83","var": "dedup_tier","val": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:20:40.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:39 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-83","var": "dedup_tier","val": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:20:40.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:39 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-04-14T15:20:40.464 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:20:40.488 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:40.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:40.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:20:40.590 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:20:40.613 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:40.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:40.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:20:40.715 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:20:40.738 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:40.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:40.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:20:40.840 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:20:40.864 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:40.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:40.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:20:40.886 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-14T15:20:40.966 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:20:40.990 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:40.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:40.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:20:41.092 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:20:41.116 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:41.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:41.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:20:41.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:40 smithi092 ceph-mon[33966]: pgmap v746: 257 pgs: 16 unknown, 241 active+clean; 8.4 MiB data, 593 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-04-14T15:20:41.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:40 smithi092 ceph-mon[33966]: 110.1b scrub starts 2024-04-14T15:20:41.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:40 smithi092 ceph-mon[33966]: 110.1b scrub ok 2024-04-14T15:20:41.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:40 smithi092 ceph-mon[33966]: 5.1c scrub starts 2024-04-14T15:20:41.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:40 smithi092 ceph-mon[33966]: 5.1c scrub ok 2024-04-14T15:20:41.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:40 smithi092 ceph-mon[33966]: from='client.49339 v1:172.21.15.90:0/1954794875' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:20:41.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:40 smithi092 ceph-mon[33966]: 2.16 scrub starts 2024-04-14T15:20:41.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:40 smithi092 ceph-mon[33966]: 2.16 scrub ok 2024-04-14T15:20:41.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:40 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-83","var": "dedup_tier","val": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:20:41.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:40 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "29"}]': finished 2024-04-14T15:20:41.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:40 smithi092 ceph-mon[33966]: osdmap e515: 8 total, 8 up, 8 in 2024-04-14T15:20:41.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:40 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-14T15:20:41.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:40 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-14T15:20:41.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:40 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-14T15:20:41.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:40 smithi092 ceph-mon[33966]: osdmap e516: 8 total, 8 up, 8 in 2024-04-14T15:20:41.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:40 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-14T15:20:41.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:40 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-14T15:20:41.170 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:20:41.217 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:20:41.241 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:41.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:41.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:20:41.342 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:41.343 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:40 smithi090 ceph-mon[41670]: pgmap v746: 257 pgs: 16 unknown, 241 active+clean; 8.4 MiB data, 593 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-04-14T15:20:41.343 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:40 smithi090 ceph-mon[41670]: 110.1b scrub starts 2024-04-14T15:20:41.343 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:40 smithi090 ceph-mon[41670]: 110.1b scrub ok 2024-04-14T15:20:41.343 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:40 smithi090 ceph-mon[41670]: 5.1c scrub starts 2024-04-14T15:20:41.343 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:40 smithi090 ceph-mon[41670]: 5.1c scrub ok 2024-04-14T15:20:41.343 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:40 smithi090 ceph-mon[41670]: from='client.49339 v1:172.21.15.90:0/1954794875' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:20:41.343 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:40 smithi090 ceph-mon[41670]: 2.16 scrub starts 2024-04-14T15:20:41.344 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:40 smithi090 ceph-mon[41670]: 2.16 scrub ok 2024-04-14T15:20:41.344 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:40 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-83","var": "dedup_tier","val": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:20:41.344 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:40 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "29"}]': finished 2024-04-14T15:20:41.344 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:40 smithi090 ceph-mon[41670]: osdmap e515: 8 total, 8 up, 8 in 2024-04-14T15:20:41.344 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:40 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-14T15:20:41.344 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:40 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-14T15:20:41.344 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:40 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-14T15:20:41.344 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:40 smithi090 ceph-mon[41670]: osdmap e516: 8 total, 8 up, 8 in 2024-04-14T15:20:41.344 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:40 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-14T15:20:41.344 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:40 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-14T15:20:41.344 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:40 smithi090 ceph-mon[30399]: pgmap v746: 257 pgs: 16 unknown, 241 active+clean; 8.4 MiB data, 593 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-04-14T15:20:41.344 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:40 smithi090 ceph-mon[30399]: 110.1b scrub starts 2024-04-14T15:20:41.345 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:40 smithi090 ceph-mon[30399]: 110.1b scrub ok 2024-04-14T15:20:41.345 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:40 smithi090 ceph-mon[30399]: 5.1c scrub starts 2024-04-14T15:20:41.345 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:40 smithi090 ceph-mon[30399]: 5.1c scrub ok 2024-04-14T15:20:41.345 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:40 smithi090 ceph-mon[30399]: from='client.49339 v1:172.21.15.90:0/1954794875' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:20:41.345 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:40 smithi090 ceph-mon[30399]: 2.16 scrub starts 2024-04-14T15:20:41.345 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:40 smithi090 ceph-mon[30399]: 2.16 scrub ok 2024-04-14T15:20:41.345 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:40 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-83","var": "dedup_tier","val": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:20:41.345 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:40 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "29"}]': finished 2024-04-14T15:20:41.345 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:40 smithi090 ceph-mon[30399]: osdmap e515: 8 total, 8 up, 8 in 2024-04-14T15:20:41.345 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:40 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-14T15:20:41.345 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:40 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-14T15:20:41.345 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:40 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-14T15:20:41.345 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:40 smithi090 ceph-mon[30399]: osdmap e516: 8 total, 8 up, 8 in 2024-04-14T15:20:41.345 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:40 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-14T15:20:41.345 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:40 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-14T15:20:41.346 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd set noscrub 2024-04-14T15:20:41.369 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:41.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:41.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:41.374 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":746,"stamp":"2024-04-14T15:20:39.114163+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8800516,"num_objects":375,"num_object_clones":10,"num_object_copies":1125,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":374,"num_whiteouts":0,"num_read":1965,"num_read_kb":1590,"num_write":1429,"num_write_kb":12805,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":263169,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":20,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6665,"ondisk_log_size":6665,"up":723,"acting":723,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":759,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":607324,"kb_used_data":92548,"kb_used_omap":12,"kb_used_meta":514483,"kb_avail":749353892,"statfs":{"total":767960285184,"available":767338385408,"internally_reserved":0,"allocated":94769152,"data_stored":65240891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":526831154},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8192,"num_objects":-1,"num_object_clones":0,"num_object_copies":-3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":0,"num_write_kb":-9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000859"},"pg_stats":[{"pgid":"307.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:20:36.960470+0000","last_change":"2024-04-14T15:20:36.960470+0000","last_active":"2024-04-14T15:20:36.960470+0000","last_peered":"2024-04-14T15:20:36.960470+0000","last_clean":"2024-04-14T15:20:36.960470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:20:36.960470+0000","last_undegraded":"2024-04-14T15:20:36.960470+0000","last_fullsized":"2024-04-14T15:20:36.960470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:36.960470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:36.960470+0000","last_clean_scrub_stamp":"2024-04-14T15:20:36.960470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"307.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:20:36.960470+0000","last_change":"2024-04-14T15:20:36.960470+0000","last_active":"2024-04-14T15:20:36.960470+0000","last_peered":"2024-04-14T15:20:36.960470+0000","last_clean":"2024-04-14T15:20:36.960470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:20:36.960470+0000","last_undegraded":"2024-04-14T15:20:36.960470+0000","last_fullsized":"2024-04-14T15:20:36.960470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:36.960470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:36.960470+0000","last_clean_scrub_stamp":"2024-04-14T15:20:36.960470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"307.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:20:36.960470+0000","last_change":"2024-04-14T15:20:36.960470+0000","last_active":"2024-04-14T15:20:36.960470+0000","last_peered":"2024-04-14T15:20:36.960470+0000","last_clean":"2024-04-14T15:20:36.960470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:20:36.960470+0000","last_undegraded":"2024-04-14T15:20:36.960470+0000","last_fullsized":"2024-04-14T15:20:36.960470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:36.960470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:36.960470+0000","last_clean_scrub_stamp":"2024-04-14T15:20:36.960470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"307.1c","version":"0'0","reported_seq":13,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.982622+0000","last_change":"2024-04-14T15:20:38.109086+0000","last_active":"2024-04-14T15:20:38.982622+0000","last_peered":"2024-04-14T15:20:38.982622+0000","last_clean":"2024-04-14T15:20:38.982622+0000","last_became_active":"2024-04-14T15:20:38.108774+0000","last_became_peered":"2024-04-14T15:20:38.108774+0000","last_unstale":"2024-04-14T15:20:38.982622+0000","last_undegraded":"2024-04-14T15:20:38.982622+0000","last_fullsized":"2024-04-14T15:20:38.982622+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":512,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:36.960470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:36.960470+0000","last_clean_scrub_stamp":"2024-04-14T15:20:36.960470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:57.228252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"307.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:20:36.960470+0000","last_change":"2024-04-14T15:20:36.960470+0000","last_active":"2024-04-14T15:20:36.960470+0000","last_peered":"2024-04-14T15:20:36.960470+0000","last_clean":"2024-04-14T15:20:36.960470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:20:36.960470+0000","last_undegraded":"2024-04-14T15:20:36.960470+0000","last_fullsized":"2024-04-14T15:20:36.960470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:36.960470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:36.960470+0000","last_clean_scrub_stamp":"2024-04-14T15:20:36.960470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"307.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:20:36.960470+0000","last_change":"2024-04-14T15:20:36.960470+0000","last_active":"2024-04-14T15:20:36.960470+0000","last_peered":"2024-04-14T15:20:36.960470+0000","last_clean":"2024-04-14T15:20:36.960470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:20:36.960470+0000","last_undegraded":"2024-04-14T15:20:36.960470+0000","last_fullsized":"2024-04-14T15:20:36.960470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:36.960470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:36.960470+0000","last_clean_scrub_stamp":"2024-04-14T15:20:36.960470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"307.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:20:36.960470+0000","last_change":"2024-04-14T15:20:36.960470+0000","last_active":"2024-04-14T15:20:36.960470+0000","last_peered":"2024-04-14T15:20:36.960470+0000","last_clean":"2024-04-14T15:20:36.960470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:20:36.960470+0000","last_undegraded":"2024-04-14T15:20:36.960470+0000","last_fullsized":"2024-04-14T15:20:36.960470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:36.960470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:36.960470+0000","last_clean_scrub_stamp":"2024-04-14T15:20:36.960470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"307.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:20:36.960470+0000","last_change":"2024-04-14T15:20:36.960470+0000","last_active":"2024-04-14T15:20:36.960470+0000","last_peered":"2024-04-14T15:20:36.960470+0000","last_clean":"2024-04-14T15:20:36.960470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:20:36.960470+0000","last_undegraded":"2024-04-14T15:20:36.960470+0000","last_fullsized":"2024-04-14T15:20:36.960470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:36.960470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:36.960470+0000","last_clean_scrub_stamp":"2024-04-14T15:20:36.960470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"307.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:20:36.960470+0000","last_change":"2024-04-14T15:20:36.960470+0000","last_active":"2024-04-14T15:20:36.960470+0000","last_peered":"2024-04-14T15:20:36.960470+0000","last_clean":"2024-04-14T15:20:36.960470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:20:36.960470+0000","last_undegraded":"2024-04-14T15:20:36.960470+0000","last_fullsized":"2024-04-14T15:20:36.960470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:36.960470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:36.960470+0000","last_clean_scrub_stamp":"2024-04-14T15:20:36.960470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"307.16","version":"0'0","reported_seq":13,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.980606+0000","last_change":"2024-04-14T15:20:37.996231+0000","last_active":"2024-04-14T15:20:38.980606+0000","last_peered":"2024-04-14T15:20:38.980606+0000","last_clean":"2024-04-14T15:20:38.980606+0000","last_became_active":"2024-04-14T15:20:37.995461+0000","last_became_peered":"2024-04-14T15:20:37.995461+0000","last_unstale":"2024-04-14T15:20:38.980606+0000","last_undegraded":"2024-04-14T15:20:38.980606+0000","last_fullsized":"2024-04-14T15:20:38.980606+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":512,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:36.960470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:36.960470+0000","last_clean_scrub_stamp":"2024-04-14T15:20:36.960470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:04.804958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"307.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:20:36.960470+0000","last_change":"2024-04-14T15:20:36.960470+0000","last_active":"2024-04-14T15:20:36.960470+0000","last_peered":"2024-04-14T15:20:36.960470+0000","last_clean":"2024-04-14T15:20:36.960470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:20:36.960470+0000","last_undegraded":"2024-04-14T15:20:36.960470+0000","last_fullsized":"2024-04-14T15:20:36.960470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:36.960470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:36.960470+0000","last_clean_scrub_stamp":"2024-04-14T15:20:36.960470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"307.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:20:36.960470+0000","last_change":"2024-04-14T15:20:36.960470+0000","last_active":"2024-04-14T15:20:36.960470+0000","last_peered":"2024-04-14T15:20:36.960470+0000","last_clean":"2024-04-14T15:20:36.960470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:20:36.960470+0000","last_undegraded":"2024-04-14T15:20:36.960470+0000","last_fullsized":"2024-04-14T15:20:36.960470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:36.960470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:36.960470+0000","last_clean_scrub_stamp":"2024-04-14T15:20:36.960470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"307.13","version":"0'0","reported_seq":10,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-04-14T15:20:37.989045+0000","last_change":"2024-04-14T15:20:37.987559+0000","last_active":"2024-04-14T15:20:37.989045+0000","last_peered":"2024-04-14T15:20:37.989045+0000","last_clean":"2024-04-14T15:20:37.989045+0000","last_became_active":"2024-04-14T15:20:37.987185+0000","last_became_peered":"2024-04-14T15:20:37.987185+0000","last_unstale":"2024-04-14T15:20:37.989045+0000","last_undegraded":"2024-04-14T15:20:37.989045+0000","last_fullsized":"2024-04-14T15:20:37.989045+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":512,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:36.960470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:36.960470+0000","last_clean_scrub_stamp":"2024-04-14T15:20:36.960470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:42.130039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"307.12","version":"0'0","reported_seq":13,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.979762+0000","last_change":"2024-04-14T15:20:37.993821+0000","last_active":"2024-04-14T15:20:38.979762+0000","last_peered":"2024-04-14T15:20:38.979762+0000","last_clean":"2024-04-14T15:20:38.979762+0000","last_became_active":"2024-04-14T15:20:37.993448+0000","last_became_peered":"2024-04-14T15:20:37.993448+0000","last_unstale":"2024-04-14T15:20:38.979762+0000","last_undegraded":"2024-04-14T15:20:38.979762+0000","last_fullsized":"2024-04-14T15:20:38.979762+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":512,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:36.960470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:36.960470+0000","last_clean_scrub_stamp":"2024-04-14T15:20:36.960470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:44.667221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"307.11","version":"0'0","reported_seq":13,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.981878+0000","last_change":"2024-04-14T15:20:37.990862+0000","last_active":"2024-04-14T15:20:38.981878+0000","last_peered":"2024-04-14T15:20:38.981878+0000","last_clean":"2024-04-14T15:20:38.981878+0000","last_became_active":"2024-04-14T15:20:37.990545+0000","last_became_peered":"2024-04-14T15:20:37.990545+0000","last_unstale":"2024-04-14T15:20:38.981878+0000","last_undegraded":"2024-04-14T15:20:38.981878+0000","last_fullsized":"2024-04-14T15:20:38.981878+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":512,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:36.960470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:36.960470+0000","last_clean_scrub_stamp":"2024-04-14T15:20:36.960470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:55.968435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"307.10","version":"0'0","reported_seq":13,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.979222+0000","last_change":"2024-04-14T15:20:37.995478+0000","last_active":"2024-04-14T15:20:38.979222+0000","last_peered":"2024-04-14T15:20:38.979222+0000","last_clean":"2024-04-14T15:20:38.979222+0000","last_became_active":"2024-04-14T15:20:37.995150+0000","last_became_peered":"2024-04-14T15:20:37.995150+0000","last_unstale":"2024-04-14T15:20:38.979222+0000","last_undegraded":"2024-04-14T15:20:38.979222+0000","last_fullsized":"2024-04-14T15:20:38.979222+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":512,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:36.960470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:36.960470+0000","last_clean_scrub_stamp":"2024-04-14T15:20:36.960470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:45.926922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"307.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:20:36.960470+0000","last_change":"2024-04-14T15:20:36.960470+0000","last_active":"2024-04-14T15:20:36.960470+0000","last_peered":"2024-04-14T15:20:36.960470+0000","last_clean":"2024-04-14T15:20:36.960470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:20:36.960470+0000","last_undegraded":"2024-04-14T15:20:36.960470+0000","last_fullsized":"2024-04-14T15:20:36.960470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:36.960470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:36.960470+0000","last_clean_scrub_stamp":"2024-04-14T15:20:36.960470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"307.e","version":"0'0","reported_seq":10,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.112367+0000","last_change":"2024-04-14T15:20:38.112110+0000","last_active":"2024-04-14T15:20:38.112367+0000","last_peered":"2024-04-14T15:20:38.112367+0000","last_clean":"2024-04-14T15:20:38.112367+0000","last_became_active":"2024-04-14T15:20:38.111823+0000","last_became_peered":"2024-04-14T15:20:38.111823+0000","last_unstale":"2024-04-14T15:20:38.112367+0000","last_undegraded":"2024-04-14T15:20:38.112367+0000","last_fullsized":"2024-04-14T15:20:38.112367+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":512,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:36.960470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:36.960470+0000","last_clean_scrub_stamp":"2024-04-14T15:20:36.960470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:43.336576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"307.d","version":"0'0","reported_seq":13,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.982829+0000","last_change":"2024-04-14T15:20:37.994421+0000","last_active":"2024-04-14T15:20:38.982829+0000","last_peered":"2024-04-14T15:20:38.982829+0000","last_clean":"2024-04-14T15:20:38.982829+0000","last_became_active":"2024-04-14T15:20:37.993974+0000","last_became_peered":"2024-04-14T15:20:37.993974+0000","last_unstale":"2024-04-14T15:20:38.982829+0000","last_undegraded":"2024-04-14T15:20:38.982829+0000","last_fullsized":"2024-04-14T15:20:38.982829+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":512,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:36.960470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:36.960470+0000","last_clean_scrub_stamp":"2024-04-14T15:20:36.960470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:02.847673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"307.c","version":"0'0","reported_seq":10,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-04-14T15:20:37.989288+0000","last_change":"2024-04-14T15:20:37.988138+0000","last_active":"2024-04-14T15:20:37.989288+0000","last_peered":"2024-04-14T15:20:37.989288+0000","last_clean":"2024-04-14T15:20:37.989288+0000","last_became_active":"2024-04-14T15:20:37.987835+0000","last_became_peered":"2024-04-14T15:20:37.987835+0000","last_unstale":"2024-04-14T15:20:37.989288+0000","last_undegraded":"2024-04-14T15:20:37.989288+0000","last_fullsized":"2024-04-14T15:20:37.989288+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":512,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:36.960470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:36.960470+0000","last_clean_scrub_stamp":"2024-04-14T15:20:36.960470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:03.257401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"307.b","version":"0'0","reported_seq":13,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.979576+0000","last_change":"2024-04-14T15:20:38.109009+0000","last_active":"2024-04-14T15:20:38.979576+0000","last_peered":"2024-04-14T15:20:38.979576+0000","last_clean":"2024-04-14T15:20:38.979576+0000","last_became_active":"2024-04-14T15:20:38.108702+0000","last_became_peered":"2024-04-14T15:20:38.108702+0000","last_unstale":"2024-04-14T15:20:38.979576+0000","last_undegraded":"2024-04-14T15:20:38.979576+0000","last_fullsized":"2024-04-14T15:20:38.979576+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":512,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:36.960470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:36.960470+0000","last_clean_scrub_stamp":"2024-04-14T15:20:36.960470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:01.851208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"307.a","version":"0'0","reported_seq":10,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-04-14T15:20:37.990284+0000","last_change":"2024-04-14T15:20:37.989924+0000","last_active":"2024-04-14T15:20:37.990284+0000","last_peered":"2024-04-14T15:20:37.990284+0000","last_clean":"2024-04-14T15:20:37.990284+0000","last_became_active":"2024-04-14T15:20:37.989551+0000","last_became_peered":"2024-04-14T15:20:37.989551+0000","last_unstale":"2024-04-14T15:20:37.990284+0000","last_undegraded":"2024-04-14T15:20:37.990284+0000","last_fullsized":"2024-04-14T15:20:37.990284+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":512,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:36.960470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:36.960470+0000","last_clean_scrub_stamp":"2024-04-14T15:20:36.960470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:43.276101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"307.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:20:36.960470+0000","last_change":"2024-04-14T15:20:36.960470+0000","last_active":"2024-04-14T15:20:36.960470+0000","last_peered":"2024-04-14T15:20:36.960470+0000","last_clean":"2024-04-14T15:20:36.960470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:20:36.960470+0000","last_undegraded":"2024-04-14T15:20:36.960470+0000","last_fullsized":"2024-04-14T15:20:36.960470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:36.960470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:36.960470+0000","last_clean_scrub_stamp":"2024-04-14T15:20:36.960470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"307.8","version":"0'0","reported_seq":13,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.979258+0000","last_change":"2024-04-14T15:20:37.996064+0000","last_active":"2024-04-14T15:20:38.979258+0000","last_peered":"2024-04-14T15:20:38.979258+0000","last_clean":"2024-04-14T15:20:38.979258+0000","last_became_active":"2024-04-14T15:20:37.995449+0000","last_became_peered":"2024-04-14T15:20:37.995449+0000","last_unstale":"2024-04-14T15:20:38.979258+0000","last_undegraded":"2024-04-14T15:20:38.979258+0000","last_fullsized":"2024-04-14T15:20:38.979258+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":512,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:36.960470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:36.960470+0000","last_clean_scrub_stamp":"2024-04-14T15:20:36.960470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:05.068361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"307.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:20:36.960470+0000","last_change":"2024-04-14T15:20:36.960470+0000","last_active":"2024-04-14T15:20:36.960470+0000","last_peered":"2024-04-14T15:20:36.960470+0000","last_clean":"2024-04-14T15:20:36.960470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:20:36.960470+0000","last_undegraded":"2024-04-14T15:20:36.960470+0000","last_fullsized":"2024-04-14T15:20:36.960470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:36.960470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:36.960470+0000","last_clean_scrub_stamp":"2024-04-14T15:20:36.960470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"307.6","version":"0'0","reported_seq":13,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.980767+0000","last_change":"2024-04-14T15:20:37.996418+0000","last_active":"2024-04-14T15:20:38.980767+0000","last_peered":"2024-04-14T15:20:38.980767+0000","last_clean":"2024-04-14T15:20:38.980767+0000","last_became_active":"2024-04-14T15:20:37.995887+0000","last_became_peered":"2024-04-14T15:20:37.995887+0000","last_unstale":"2024-04-14T15:20:38.980767+0000","last_undegraded":"2024-04-14T15:20:38.980767+0000","last_fullsized":"2024-04-14T15:20:38.980767+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":512,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:36.960470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:36.960470+0000","last_clean_scrub_stamp":"2024-04-14T15:20:36.960470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:37.488107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"307.5","version":"0'0","reported_seq":13,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.980683+0000","last_change":"2024-04-14T15:20:38.110357+0000","last_active":"2024-04-14T15:20:38.980683+0000","last_peered":"2024-04-14T15:20:38.980683+0000","last_clean":"2024-04-14T15:20:38.980683+0000","last_became_active":"2024-04-14T15:20:38.110035+0000","last_became_peered":"2024-04-14T15:20:38.110035+0000","last_unstale":"2024-04-14T15:20:38.980683+0000","last_undegraded":"2024-04-14T15:20:38.980683+0000","last_fullsized":"2024-04-14T15:20:38.980683+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":512,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:36.960470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:36.960470+0000","last_clean_scrub_stamp":"2024-04-14T15:20:36.960470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:44.006153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"307.4","version":"0'0","reported_seq":13,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.979988+0000","last_change":"2024-04-14T15:20:37.992693+0000","last_active":"2024-04-14T15:20:38.979988+0000","last_peered":"2024-04-14T15:20:38.979988+0000","last_clean":"2024-04-14T15:20:38.979988+0000","last_became_active":"2024-04-14T15:20:37.992294+0000","last_became_peered":"2024-04-14T15:20:37.992294+0000","last_unstale":"2024-04-14T15:20:38.979988+0000","last_undegraded":"2024-04-14T15:20:38.979988+0000","last_fullsized":"2024-04-14T15:20:38.979988+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":512,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:36.960470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:36.960470+0000","last_clean_scrub_stamp":"2024-04-14T15:20:36.960470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:52.659319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"307.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:20:36.960470+0000","last_change":"2024-04-14T15:20:36.960470+0000","last_active":"2024-04-14T15:20:36.960470+0000","last_peered":"2024-04-14T15:20:36.960470+0000","last_clean":"2024-04-14T15:20:36.960470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:20:36.960470+0000","last_undegraded":"2024-04-14T15:20:36.960470+0000","last_fullsized":"2024-04-14T15:20:36.960470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:36.960470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:36.960470+0000","last_clean_scrub_stamp":"2024-04-14T15:20:36.960470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"307.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:20:36.960470+0000","last_change":"2024-04-14T15:20:36.960470+0000","last_active":"2024-04-14T15:20:36.960470+0000","last_peered":"2024-04-14T15:20:36.960470+0000","last_clean":"2024-04-14T15:20:36.960470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:20:36.960470+0000","last_undegraded":"2024-04-14T15:20:36.960470+0000","last_fullsized":"2024-04-14T15:20:36.960470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:36.960470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:36.960470+0000","last_clean_scrub_stamp":"2024-04-14T15:20:36.960470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"307.1","version":"0'0","reported_seq":10,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-04-14T15:20:37.989724+0000","last_change":"2024-04-14T15:20:37.988909+0000","last_active":"2024-04-14T15:20:37.989724+0000","last_peered":"2024-04-14T15:20:37.989724+0000","last_clean":"2024-04-14T15:20:37.989724+0000","last_became_active":"2024-04-14T15:20:37.988622+0000","last_became_peered":"2024-04-14T15:20:37.988622+0000","last_unstale":"2024-04-14T15:20:37.989724+0000","last_undegraded":"2024-04-14T15:20:37.989724+0000","last_fullsized":"2024-04-14T15:20:37.989724+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":512,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:36.960470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:36.960470+0000","last_clean_scrub_stamp":"2024-04-14T15:20:36.960470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:03.565013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"307.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:20:36.960470+0000","last_change":"2024-04-14T15:20:36.960470+0000","last_active":"2024-04-14T15:20:36.960470+0000","last_peered":"2024-04-14T15:20:36.960470+0000","last_clean":"2024-04-14T15:20:36.960470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:20:36.960470+0000","last_undegraded":"2024-04-14T15:20:36.960470+0000","last_fullsized":"2024-04-14T15:20:36.960470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:36.960470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:36.960470+0000","last_clean_scrub_stamp":"2024-04-14T15:20:36.960470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"110.11","version":"509'43","reported_seq":1049,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-04-14T15:20:37.981261+0000","last_change":"2024-04-14T15:20:00.830028+0000","last_active":"2024-04-14T15:20:37.981261+0000","last_peered":"2024-04-14T15:20:37.981261+0000","last_clean":"2024-04-14T15:20:37.981261+0000","last_became_active":"2024-04-14T15:13:09.936619+0000","last_became_peered":"2024-04-14T15:13:09.936619+0000","last_unstale":"2024-04-14T15:20:37.981261+0000","last_undegraded":"2024-04-14T15:20:37.981261+0000","last_fullsized":"2024-04-14T15:20:37.981261+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"403'41","last_scrub_stamp":"2024-04-14T15:20:00.829932+0000","last_deep_scrub":"403'41","last_deep_scrub_stamp":"2024-04-14T15:17:56.911004+0000","last_clean_scrub_stamp":"2024-04-14T15:20:00.829932+0000","objects_scrubbed":2,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:03.495441+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00061958300000000005,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":15,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"110.10","version":"253'158","reported_seq":1123,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.981499+0000","last_change":"2024-04-14T15:18:54.158668+0000","last_active":"2024-04-14T15:20:38.981499+0000","last_peered":"2024-04-14T15:20:38.981499+0000","last_clean":"2024-04-14T15:20:38.981499+0000","last_became_active":"2024-04-14T15:13:09.928915+0000","last_became_peered":"2024-04-14T15:13:09.928915+0000","last_unstale":"2024-04-14T15:20:38.981499+0000","last_undegraded":"2024-04-14T15:20:38.981499+0000","last_fullsized":"2024-04-14T15:20:38.981499+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"253'158","last_scrub_stamp":"2024-04-14T15:17:34.291854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:34.291854+0000","objects_scrubbed":10,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00045995200000000002,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":47,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"110.13","version":"477'53","reported_seq":1043,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.954167+0000","last_change":"2024-04-14T15:18:54.157558+0000","last_active":"2024-04-14T15:20:33.954167+0000","last_peered":"2024-04-14T15:20:33.954167+0000","last_clean":"2024-04-14T15:20:33.954167+0000","last_became_active":"2024-04-14T15:13:09.918152+0000","last_became_peered":"2024-04-14T15:13:09.918152+0000","last_unstale":"2024-04-14T15:20:33.954167+0000","last_undegraded":"2024-04-14T15:20:33.954167+0000","last_fullsized":"2024-04-14T15:20:33.954167+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'39","last_scrub_stamp":"2024-04-14T15:17:44.873766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:44.873766+0000","objects_scrubbed":3,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001942764,"stat_sum":{"num_bytes":57,"num_objects":7,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"110.12","version":"508'79","reported_seq":1076,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.954350+0000","last_change":"2024-04-14T15:19:47.921780+0000","last_active":"2024-04-14T15:20:33.954350+0000","last_peered":"2024-04-14T15:20:33.954350+0000","last_clean":"2024-04-14T15:20:33.954350+0000","last_became_active":"2024-04-14T15:13:09.939566+0000","last_became_peered":"2024-04-14T15:13:09.939566+0000","last_unstale":"2024-04-14T15:20:33.954350+0000","last_undegraded":"2024-04-14T15:20:33.954350+0000","last_fullsized":"2024-04-14T15:20:33.954350+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"409'70","last_scrub_stamp":"2024-04-14T15:19:47.921684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:19:47.921684+0000","objects_scrubbed":9,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:12.085817+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.000970527,"stat_sum":{"num_bytes":89,"num_objects":11,"num_object_clones":1,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":8,"num_read_kb":2,"num_write":30,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"110.15","version":"509'36","reported_seq":1028,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.946757+0000","last_change":"2024-04-14T15:18:54.155028+0000","last_active":"2024-04-14T15:20:33.946757+0000","last_peered":"2024-04-14T15:20:33.946757+0000","last_clean":"2024-04-14T15:20:33.946757+0000","last_became_active":"2024-04-14T15:13:09.960121+0000","last_became_peered":"2024-04-14T15:13:09.960121+0000","last_unstale":"2024-04-14T15:20:33.946757+0000","last_undegraded":"2024-04-14T15:20:33.946757+0000","last_fullsized":"2024-04-14T15:20:33.946757+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'34","last_scrub_stamp":"2024-04-14T15:18:36.013135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:18:36.013135+0000","objects_scrubbed":2,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0029213049999999999,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":13,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"110.14","version":"509'89","reported_seq":1074,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.947134+0000","last_change":"2024-04-14T15:18:54.151534+0000","last_active":"2024-04-14T15:20:33.947134+0000","last_peered":"2024-04-14T15:20:33.947134+0000","last_clean":"2024-04-14T15:20:33.947134+0000","last_became_active":"2024-04-14T15:13:09.899561+0000","last_became_peered":"2024-04-14T15:13:09.899561+0000","last_unstale":"2024-04-14T15:20:33.947134+0000","last_undegraded":"2024-04-14T15:20:33.947134+0000","last_fullsized":"2024-04-14T15:20:33.947134+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'87","last_scrub_stamp":"2024-04-14T15:18:31.117033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:18:31.117033+0000","objects_scrubbed":7,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0014558730000000001,"stat_sum":{"num_bytes":71,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":7,"num_read_kb":1,"num_write":30,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"110.17","version":"509'16","reported_seq":1023,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.955576+0000","last_change":"2024-04-14T15:19:48.225230+0000","last_active":"2024-04-14T15:20:33.955576+0000","last_peered":"2024-04-14T15:20:33.955576+0000","last_clean":"2024-04-14T15:20:33.955576+0000","last_became_active":"2024-04-14T15:13:09.898521+0000","last_became_peered":"2024-04-14T15:13:09.898521+0000","last_unstale":"2024-04-14T15:20:33.955576+0000","last_undegraded":"2024-04-14T15:20:33.955576+0000","last_fullsized":"2024-04-14T15:20:33.955576+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'12","last_scrub_stamp":"2024-04-14T15:19:48.225130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:19:48.225130+0000","objects_scrubbed":4,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:56.459153+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0018681640000000001,"stat_sum":{"num_bytes":24,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":2,"num_write":11,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"110.16","version":"355'35","reported_seq":859,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.961567+0000","last_change":"2024-04-14T15:18:54.160925+0000","last_active":"2024-04-14T15:20:33.961567+0000","last_peered":"2024-04-14T15:20:33.961567+0000","last_clean":"2024-04-14T15:20:33.961567+0000","last_became_active":"2024-04-14T15:15:21.889783+0000","last_became_peered":"2024-04-14T15:15:21.889783+0000","last_unstale":"2024-04-14T15:20:33.961567+0000","last_undegraded":"2024-04-14T15:20:33.961567+0000","last_fullsized":"2024-04-14T15:20:33.961567+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'35","last_scrub_stamp":"2024-04-14T15:17:34.097563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:34.097563+0000","objects_scrubbed":3,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000499903,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"110.19","version":"509'110","reported_seq":1103,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.962204+0000","last_change":"2024-04-14T15:19:12.058233+0000","last_active":"2024-04-14T15:20:33.962204+0000","last_peered":"2024-04-14T15:20:33.962204+0000","last_clean":"2024-04-14T15:20:33.962204+0000","last_became_active":"2024-04-14T15:13:09.998139+0000","last_became_peered":"2024-04-14T15:13:09.998139+0000","last_unstale":"2024-04-14T15:20:33.962204+0000","last_undegraded":"2024-04-14T15:20:33.962204+0000","last_fullsized":"2024-04-14T15:20:33.962204+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'85","last_scrub_stamp":"2024-04-14T15:19:12.058127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:19:12.058127+0000","objects_scrubbed":5,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0.0012903700000000001,"stat_sum":{"num_bytes":85,"num_objects":8,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":43,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"110.18","version":"387'94","reported_seq":1078,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.942543+0000","last_change":"2024-04-14T15:18:54.150377+0000","last_active":"2024-04-14T15:20:33.942543+0000","last_peered":"2024-04-14T15:20:33.942543+0000","last_clean":"2024-04-14T15:20:33.942543+0000","last_became_active":"2024-04-14T15:13:09.998803+0000","last_became_peered":"2024-04-14T15:13:09.998803+0000","last_unstale":"2024-04-14T15:20:33.942543+0000","last_undegraded":"2024-04-14T15:20:33.942543+0000","last_fullsized":"2024-04-14T15:20:33.942543+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'94","last_scrub_stamp":"2024-04-14T15:17:36.005598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:36.005598+0000","objects_scrubbed":8,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0013897930000000001,"stat_sum":{"num_bytes":65,"num_objects":8,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":29,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"110.1b","version":"509'112","reported_seq":1096,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.958438+0000","last_change":"2024-04-14T15:18:54.156913+0000","last_active":"2024-04-14T15:20:33.958438+0000","last_peered":"2024-04-14T15:20:33.958438+0000","last_clean":"2024-04-14T15:20:33.958438+0000","last_became_active":"2024-04-14T15:13:09.946919+0000","last_became_peered":"2024-04-14T15:13:09.946919+0000","last_unstale":"2024-04-14T15:20:33.958438+0000","last_undegraded":"2024-04-14T15:20:33.958438+0000","last_fullsized":"2024-04-14T15:20:33.958438+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'91","last_scrub_stamp":"2024-04-14T15:17:47.889364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:47.889364+0000","objects_scrubbed":6,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001643863,"stat_sum":{"num_bytes":8285,"num_objects":10,"num_object_clones":2,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":14,"num_read_kb":4,"num_write":41,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"110.1a","version":"387'39","reported_seq":1051,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-04-14T15:20:37.980108+0000","last_change":"2024-04-14T15:20:34.830237+0000","last_active":"2024-04-14T15:20:37.980108+0000","last_peered":"2024-04-14T15:20:37.980108+0000","last_clean":"2024-04-14T15:20:37.980108+0000","last_became_active":"2024-04-14T15:13:09.960319+0000","last_became_peered":"2024-04-14T15:13:09.960319+0000","last_unstale":"2024-04-14T15:20:37.980108+0000","last_undegraded":"2024-04-14T15:20:37.980108+0000","last_fullsized":"2024-04-14T15:20:37.980108+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'39","last_scrub_stamp":"2024-04-14T15:20:34.830136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:20:34.830136+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:48.029180+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0018412890000000001,"stat_sum":{"num_bytes":62,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"110.1d","version":"225'85","reported_seq":1088,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.980918+0000","last_change":"2024-04-14T15:20:34.688380+0000","last_active":"2024-04-14T15:20:38.980918+0000","last_peered":"2024-04-14T15:20:38.980918+0000","last_clean":"2024-04-14T15:20:38.980918+0000","last_became_active":"2024-04-14T15:13:09.868082+0000","last_became_peered":"2024-04-14T15:13:09.868082+0000","last_unstale":"2024-04-14T15:20:38.980918+0000","last_undegraded":"2024-04-14T15:20:38.980918+0000","last_fullsized":"2024-04-14T15:20:38.980918+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"225'85","last_scrub_stamp":"2024-04-14T15:20:34.688290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:20:34.688290+0000","objects_scrubbed":5,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:50.948360+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0010543219999999999,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"110.1c","version":"509'97","reported_seq":1084,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.954969+0000","last_change":"2024-04-14T15:19:10.839468+0000","last_active":"2024-04-14T15:20:33.954969+0000","last_peered":"2024-04-14T15:20:33.954969+0000","last_clean":"2024-04-14T15:20:33.954969+0000","last_became_active":"2024-04-14T15:13:09.954091+0000","last_became_peered":"2024-04-14T15:13:09.954091+0000","last_unstale":"2024-04-14T15:20:33.954969+0000","last_undegraded":"2024-04-14T15:20:33.954969+0000","last_fullsized":"2024-04-14T15:20:33.954969+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'95","last_scrub_stamp":"2024-04-14T15:19:10.839339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:19:10.839339+0000","objects_scrubbed":7,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0011789070000000001,"stat_sum":{"num_bytes":65,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":33,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"110.1f","version":"508'29","reported_seq":1076,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.944782+0000","last_change":"2024-04-14T15:19:25.078771+0000","last_active":"2024-04-14T15:20:33.944782+0000","last_peered":"2024-04-14T15:20:33.944782+0000","last_clean":"2024-04-14T15:20:33.944782+0000","last_became_active":"2024-04-14T15:13:09.999629+0000","last_became_peered":"2024-04-14T15:13:09.999629+0000","last_unstale":"2024-04-14T15:20:33.944782+0000","last_undegraded":"2024-04-14T15:20:33.944782+0000","last_fullsized":"2024-04-14T15:20:33.944782+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"466'27","last_scrub_stamp":"2024-04-14T15:19:25.078678+0000","last_deep_scrub":"466'27","last_deep_scrub_stamp":"2024-04-14T15:19:25.078678+0000","last_clean_scrub_stamp":"2024-04-14T15:19:25.078678+0000","objects_scrubbed":2,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:50.898358+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.001392454,"stat_sum":{"num_bytes":4194315,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":11,"num_write_kb":8199,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"110.1e","version":"508'25","reported_seq":1021,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.947453+0000","last_change":"2024-04-14T15:19:17.364851+0000","last_active":"2024-04-14T15:20:33.947453+0000","last_peered":"2024-04-14T15:20:33.947453+0000","last_clean":"2024-04-14T15:20:33.947453+0000","last_became_active":"2024-04-14T15:13:09.955932+0000","last_became_peered":"2024-04-14T15:13:09.955932+0000","last_unstale":"2024-04-14T15:20:33.947453+0000","last_undegraded":"2024-04-14T15:20:33.947453+0000","last_fullsized":"2024-04-14T15:20:33.947453+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'23","last_scrub_stamp":"2024-04-14T15:19:17.364769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:19:17.364769+0000","objects_scrubbed":2,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0019783700000000001,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":11,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"110.1","version":"509'34","reported_seq":1043,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.979519+0000","last_change":"2024-04-14T15:18:54.150210+0000","last_active":"2024-04-14T15:20:38.979519+0000","last_peered":"2024-04-14T15:20:38.979519+0000","last_clean":"2024-04-14T15:20:38.979519+0000","last_became_active":"2024-04-14T15:13:09.929562+0000","last_became_peered":"2024-04-14T15:13:09.929562+0000","last_unstale":"2024-04-14T15:20:38.979519+0000","last_undegraded":"2024-04-14T15:20:38.979519+0000","last_fullsized":"2024-04-14T15:20:38.979519+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"406'32","last_scrub_stamp":"2024-04-14T15:18:38.685469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:18:38.685469+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00089761599999999995,"stat_sum":{"num_bytes":4000050,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":18,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"110.0","version":"219'85","reported_seq":1074,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.981507+0000","last_change":"2024-04-14T15:18:54.159255+0000","last_active":"2024-04-14T15:20:38.981507+0000","last_peered":"2024-04-14T15:20:38.981507+0000","last_clean":"2024-04-14T15:20:38.981507+0000","last_became_active":"2024-04-14T15:13:09.856407+0000","last_became_peered":"2024-04-14T15:13:09.856407+0000","last_unstale":"2024-04-14T15:20:38.981507+0000","last_undegraded":"2024-04-14T15:20:38.981507+0000","last_fullsized":"2024-04-14T15:20:38.981507+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"219'85","last_scrub_stamp":"2024-04-14T15:17:51.438149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:51.438149+0000","objects_scrubbed":5,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00052820199999999997,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"110.3","version":"464'56","reported_seq":1080,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.943317+0000","last_change":"2024-04-14T15:19:35.504089+0000","last_active":"2024-04-14T15:20:33.943317+0000","last_peered":"2024-04-14T15:20:33.943317+0000","last_clean":"2024-04-14T15:20:33.943317+0000","last_became_active":"2024-04-14T15:19:35.486976+0000","last_became_peered":"2024-04-14T15:19:35.486976+0000","last_unstale":"2024-04-14T15:20:33.943317+0000","last_undegraded":"2024-04-14T15:20:33.943317+0000","last_fullsized":"2024-04-14T15:20:33.943317+0000","mapping_epoch":470,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":471,"parent":"0.0","parent_split_bits":0,"last_scrub":"464'56","last_scrub_stamp":"2024-04-14T15:19:17.063716+0000","last_deep_scrub":"464'56","last_deep_scrub_stamp":"2024-04-14T15:19:17.063716+0000","last_clean_scrub_stamp":"2024-04-14T15:19:17.063716+0000","objects_scrubbed":4,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0019291829999999999,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":41,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"110.2","version":"509'57","reported_seq":1048,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.944417+0000","last_change":"2024-04-14T15:18:54.152834+0000","last_active":"2024-04-14T15:20:33.944417+0000","last_peered":"2024-04-14T15:20:33.944417+0000","last_clean":"2024-04-14T15:20:33.944417+0000","last_became_active":"2024-04-14T15:13:09.957187+0000","last_became_peered":"2024-04-14T15:13:09.957187+0000","last_unstale":"2024-04-14T15:20:33.944417+0000","last_undegraded":"2024-04-14T15:20:33.944417+0000","last_fullsized":"2024-04-14T15:20:33.944417+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'55","last_scrub_stamp":"2024-04-14T15:17:37.941589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:17:37.941589+0000","objects_scrubbed":6,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0020075969999999999,"stat_sum":{"num_bytes":49,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":21,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"110.5","version":"510'68","reported_seq":1061,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.983274+0000","last_change":"2024-04-14T15:18:54.162113+0000","last_active":"2024-04-14T15:20:38.983274+0000","last_peered":"2024-04-14T15:20:38.983274+0000","last_clean":"2024-04-14T15:20:38.983274+0000","last_became_active":"2024-04-14T15:13:09.853689+0000","last_became_peered":"2024-04-14T15:13:09.853689+0000","last_unstale":"2024-04-14T15:20:38.983274+0000","last_undegraded":"2024-04-14T15:20:38.983274+0000","last_fullsized":"2024-04-14T15:20:38.983274+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"396'64","last_scrub_stamp":"2024-04-14T15:17:47.418687+0000","last_deep_scrub":"396'64","last_deep_scrub_stamp":"2024-04-14T15:17:47.418687+0000","last_clean_scrub_stamp":"2024-04-14T15:17:47.418687+0000","objects_scrubbed":5,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00099928400000000002,"stat_sum":{"num_bytes":42,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":26,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"110.4","version":"406'55","reported_seq":1043,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.947221+0000","last_change":"2024-04-14T15:19:11.276191+0000","last_active":"2024-04-14T15:20:33.947221+0000","last_peered":"2024-04-14T15:20:33.947221+0000","last_clean":"2024-04-14T15:20:33.947221+0000","last_became_active":"2024-04-14T15:13:09.958847+0000","last_became_peered":"2024-04-14T15:13:09.958847+0000","last_unstale":"2024-04-14T15:20:33.947221+0000","last_undegraded":"2024-04-14T15:20:33.947221+0000","last_fullsized":"2024-04-14T15:20:33.947221+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"406'55","last_scrub_stamp":"2024-04-14T15:19:11.276056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:19:11.276056+0000","objects_scrubbed":4,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001395522,"stat_sum":{"num_bytes":38,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"110.7","version":"355'18","reported_seq":1028,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-04-14T15:20:37.984423+0000","last_change":"2024-04-14T15:19:56.930358+0000","last_active":"2024-04-14T15:20:37.984423+0000","last_peered":"2024-04-14T15:20:37.984423+0000","last_clean":"2024-04-14T15:20:37.984423+0000","last_became_active":"2024-04-14T15:13:09.936631+0000","last_became_peered":"2024-04-14T15:13:09.936631+0000","last_unstale":"2024-04-14T15:20:37.984423+0000","last_undegraded":"2024-04-14T15:20:37.984423+0000","last_fullsized":"2024-04-14T15:20:37.984423+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'18","last_scrub_stamp":"2024-04-14T15:19:56.930258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:19:56.930258+0000","objects_scrubbed":2,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:09.254833+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00050163699999999998,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"110.6","version":"444'90","reported_seq":1079,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.981007+0000","last_change":"2024-04-14T15:19:12.507223+0000","last_active":"2024-04-14T15:20:38.981007+0000","last_peered":"2024-04-14T15:20:38.981007+0000","last_clean":"2024-04-14T15:20:38.981007+0000","last_became_active":"2024-04-14T15:13:09.930125+0000","last_became_peered":"2024-04-14T15:13:09.930125+0000","last_unstale":"2024-04-14T15:20:38.981007+0000","last_undegraded":"2024-04-14T15:20:38.981007+0000","last_fullsized":"2024-04-14T15:20:38.981007+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"444'90","last_scrub_stamp":"2024-04-14T15:19:12.507119+0000","last_deep_scrub":"274'74","last_deep_scrub_stamp":"2024-04-14T15:15:42.641684+0000","last_clean_scrub_stamp":"2024-04-14T15:19:12.507119+0000","objects_scrubbed":7,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":1,"snaptrim_duration":2.0059427689999998,"stat_sum":{"num_bytes":62,"num_objects":7,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":26,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"110.9","version":"508'22","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.943831+0000","last_change":"2024-04-14T15:19:22.004327+0000","last_active":"2024-04-14T15:20:33.943831+0000","last_peered":"2024-04-14T15:20:33.943831+0000","last_clean":"2024-04-14T15:20:33.943831+0000","last_became_active":"2024-04-14T15:13:09.957324+0000","last_became_peered":"2024-04-14T15:13:09.957324+0000","last_unstale":"2024-04-14T15:20:33.943831+0000","last_undegraded":"2024-04-14T15:20:33.943831+0000","last_fullsized":"2024-04-14T15:20:33.943831+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"369'20","last_scrub_stamp":"2024-04-14T15:19:22.004242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:19:22.004242+0000","objects_scrubbed":4,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:42.606912+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00098494500000000009,"stat_sum":{"num_bytes":35,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":11,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"110.8","version":"509'19","reported_seq":765,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.956840+0000","last_change":"2024-04-14T15:18:54.159492+0000","last_active":"2024-04-14T15:20:33.956840+0000","last_peered":"2024-04-14T15:20:33.956840+0000","last_clean":"2024-04-14T15:20:33.956840+0000","last_became_active":"2024-04-14T15:15:44.890057+0000","last_became_peered":"2024-04-14T15:15:44.890057+0000","last_unstale":"2024-04-14T15:20:33.956840+0000","last_undegraded":"2024-04-14T15:20:33.956840+0000","last_fullsized":"2024-04-14T15:20:33.956840+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'17","last_scrub_stamp":"2024-04-14T15:18:34.505521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:18:34.505521+0000","objects_scrubbed":1,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00097299799999999998,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"110.b","version":"412'71","reported_seq":1062,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.954239+0000","last_change":"2024-04-14T15:19:22.798118+0000","last_active":"2024-04-14T15:20:33.954239+0000","last_peered":"2024-04-14T15:20:33.954239+0000","last_clean":"2024-04-14T15:20:33.954239+0000","last_became_active":"2024-04-14T15:13:09.964629+0000","last_became_peered":"2024-04-14T15:13:09.964629+0000","last_unstale":"2024-04-14T15:20:33.954239+0000","last_undegraded":"2024-04-14T15:20:33.954239+0000","last_fullsized":"2024-04-14T15:20:33.954239+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'71","last_scrub_stamp":"2024-04-14T15:19:22.798030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:19:22.798030+0000","objects_scrubbed":5,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:40.892661+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0017521979999999999,"stat_sum":{"num_bytes":54,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":22,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"110.a","version":"387'74","reported_seq":987,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.956972+0000","last_change":"2024-04-14T15:19:14.371878+0000","last_active":"2024-04-14T15:20:33.956972+0000","last_peered":"2024-04-14T15:20:33.956972+0000","last_clean":"2024-04-14T15:20:33.956972+0000","last_became_active":"2024-04-14T15:15:21.885346+0000","last_became_peered":"2024-04-14T15:15:21.885346+0000","last_unstale":"2024-04-14T15:20:33.956972+0000","last_undegraded":"2024-04-14T15:20:33.956972+0000","last_fullsized":"2024-04-14T15:20:33.956972+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'74","last_scrub_stamp":"2024-04-14T15:19:14.371777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:19:14.371777+0000","objects_scrubbed":9,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00095557400000000005,"stat_sum":{"num_bytes":84,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":26,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"110.d","version":"507'98","reported_seq":914,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.958403+0000","last_change":"2024-04-14T15:19:13.327926+0000","last_active":"2024-04-14T15:20:33.958403+0000","last_peered":"2024-04-14T15:20:33.958403+0000","last_clean":"2024-04-14T15:20:33.958403+0000","last_became_active":"2024-04-14T15:15:21.877073+0000","last_became_peered":"2024-04-14T15:15:21.877073+0000","last_unstale":"2024-04-14T15:20:33.958403+0000","last_undegraded":"2024-04-14T15:20:33.958403+0000","last_fullsized":"2024-04-14T15:20:33.958403+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'95","last_scrub_stamp":"2024-04-14T15:19:13.327836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:19:13.327836+0000","objects_scrubbed":8,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00067428099999999999,"stat_sum":{"num_bytes":71,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":7,"num_read_kb":1,"num_write":37,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"110.c","version":"459'25","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.954938+0000","last_change":"2024-04-14T15:18:54.157224+0000","last_active":"2024-04-14T15:20:33.954938+0000","last_peered":"2024-04-14T15:20:33.954938+0000","last_clean":"2024-04-14T15:20:33.954938+0000","last_became_active":"2024-04-14T15:13:09.984609+0000","last_became_peered":"2024-04-14T15:13:09.984609+0000","last_unstale":"2024-04-14T15:20:33.954938+0000","last_undegraded":"2024-04-14T15:20:33.954938+0000","last_fullsized":"2024-04-14T15:20:33.954938+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'20","last_scrub_stamp":"2024-04-14T15:17:52.882509+0000","last_deep_scrub":"398'20","last_deep_scrub_stamp":"2024-04-14T15:17:52.882509+0000","last_clean_scrub_stamp":"2024-04-14T15:17:52.882509+0000","objects_scrubbed":2,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001761999,"stat_sum":{"num_bytes":29,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":9,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"110.f","version":"508'54","reported_seq":1050,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.955902+0000","last_change":"2024-04-14T15:20:03.860729+0000","last_active":"2024-04-14T15:20:33.955902+0000","last_peered":"2024-04-14T15:20:33.955902+0000","last_clean":"2024-04-14T15:20:33.955902+0000","last_became_active":"2024-04-14T15:13:09.964057+0000","last_became_peered":"2024-04-14T15:13:09.964057+0000","last_unstale":"2024-04-14T15:20:33.955902+0000","last_undegraded":"2024-04-14T15:20:33.955902+0000","last_fullsized":"2024-04-14T15:20:33.955902+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'52","last_scrub_stamp":"2024-04-14T15:20:03.860616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:20:03.860616+0000","objects_scrubbed":4,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:21.988635+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001409509,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":19,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"110.e","version":"509'94","reported_seq":1095,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.980907+0000","last_change":"2024-04-14T15:20:02.654434+0000","last_active":"2024-04-14T15:20:38.980907+0000","last_peered":"2024-04-14T15:20:38.980907+0000","last_clean":"2024-04-14T15:20:38.980907+0000","last_became_active":"2024-04-14T15:13:09.984980+0000","last_became_peered":"2024-04-14T15:13:09.984980+0000","last_unstale":"2024-04-14T15:20:38.980907+0000","last_undegraded":"2024-04-14T15:20:38.980907+0000","last_fullsized":"2024-04-14T15:20:38.980907+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'92","last_scrub_stamp":"2024-04-14T15:20:02.654315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:20:02.654315+0000","objects_scrubbed":7,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:18.006902+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0012202070000000001,"stat_sum":{"num_bytes":68,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":8,"num_read_kb":1,"num_write":31,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"91'130","reported_seq":506,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.980757+0000","last_change":"2024-04-14T15:19:09.497449+0000","last_active":"2024-04-14T15:20:38.980757+0000","last_peered":"2024-04-14T15:20:38.980757+0000","last_clean":"2024-04-14T15:20:38.980757+0000","last_became_active":"2024-04-14T15:18:58.187712+0000","last_became_peered":"2024-04-14T15:18:58.187712+0000","last_unstale":"2024-04-14T15:20:38.980757+0000","last_undegraded":"2024-04-14T15:20:38.980757+0000","last_fullsized":"2024-04-14T15:20:38.980757+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:09.497308+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:09.497308+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":486,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.944885+0000","last_change":"2024-04-14T15:19:10.053883+0000","last_active":"2024-04-14T15:20:33.944885+0000","last_peered":"2024-04-14T15:20:33.944885+0000","last_clean":"2024-04-14T15:20:33.944885+0000","last_became_active":"2024-04-14T15:15:44.882941+0000","last_became_peered":"2024-04-14T15:15:44.882941+0000","last_unstale":"2024-04-14T15:20:33.944885+0000","last_undegraded":"2024-04-14T15:20:33.944885+0000","last_fullsized":"2024-04-14T15:20:33.944885+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:10.053804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:19:10.053804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.9","version":"85'9","reported_seq":1076,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.979878+0000","last_change":"2024-04-14T15:17:54.606158+0000","last_active":"2024-04-14T15:20:38.979878+0000","last_peered":"2024-04-14T15:20:38.979878+0000","last_clean":"2024-04-14T15:20:38.979878+0000","last_became_active":"2024-04-14T15:09:47.045441+0000","last_became_peered":"2024-04-14T15:09:47.045441+0000","last_unstale":"2024-04-14T15:20:38.979878+0000","last_undegraded":"2024-04-14T15:20:38.979878+0000","last_fullsized":"2024-04-14T15:20:38.979878+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9","last_scrub_stamp":"2024-04-14T15:17:54.606051+0000","last_deep_scrub":"85'9","last_deep_scrub_stamp":"2024-04-14T15:10:51.950580+0000","last_clean_scrub_stamp":"2024-04-14T15:17:54.606051+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"479'34","reported_seq":563,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.979411+0000","last_change":"2024-04-14T15:18:39.704989+0000","last_active":"2024-04-14T15:20:38.979411+0000","last_peered":"2024-04-14T15:20:38.979411+0000","last_clean":"2024-04-14T15:20:38.979411+0000","last_became_active":"2024-04-14T15:15:44.885409+0000","last_became_peered":"2024-04-14T15:15:44.885409+0000","last_unstale":"2024-04-14T15:20:38.979411+0000","last_undegraded":"2024-04-14T15:20:38.979411+0000","last_fullsized":"2024-04-14T15:20:38.979411+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'22","last_scrub_stamp":"2024-04-14T15:18:39.704883+0000","last_deep_scrub":"430'22","last_deep_scrub_stamp":"2024-04-14T15:18:39.704883+0000","last_clean_scrub_stamp":"2024-04-14T15:18:39.704883+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":932,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.956180+0000","last_change":"2024-04-14T15:17:53.867764+0000","last_active":"2024-04-14T15:20:33.956180+0000","last_peered":"2024-04-14T15:20:33.956180+0000","last_clean":"2024-04-14T15:20:33.956180+0000","last_became_active":"2024-04-14T15:09:43.021460+0000","last_became_peered":"2024-04-14T15:09:43.021460+0000","last_unstale":"2024-04-14T15:20:33.956180+0000","last_undegraded":"2024-04-14T15:20:33.956180+0000","last_fullsized":"2024-04-14T15:20:33.956180+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:53.867696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:53.867696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":502,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-04-14T15:20:37.981977+0000","last_change":"2024-04-14T15:20:36.860516+0000","last_active":"2024-04-14T15:20:37.981977+0000","last_peered":"2024-04-14T15:20:37.981977+0000","last_clean":"2024-04-14T15:20:37.981977+0000","last_became_active":"2024-04-14T15:15:44.888154+0000","last_became_peered":"2024-04-14T15:15:44.888154+0000","last_unstale":"2024-04-14T15:20:37.981977+0000","last_undegraded":"2024-04-14T15:20:37.981977+0000","last_fullsized":"2024-04-14T15:20:37.981977+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:36.860426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:15:40.368219+0000","last_clean_scrub_stamp":"2024-04-14T15:20:36.860426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:54.793195+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"91'130","reported_seq":523,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.984299+0000","last_change":"2024-04-14T15:20:38.984299+0000","last_active":"2024-04-14T15:20:38.984299+0000","last_peered":"2024-04-14T15:20:38.984299+0000","last_clean":"2024-04-14T15:20:38.984299+0000","last_became_active":"2024-04-14T15:20:38.983964+0000","last_became_peered":"2024-04-14T15:20:38.983964+0000","last_unstale":"2024-04-14T15:20:38.984299+0000","last_undegraded":"2024-04-14T15:20:38.984299+0000","last_fullsized":"2024-04-14T15:20:38.984299+0000","mapping_epoch":436,"log_start":"91'130","ondisk_log_start":"91'130","created":431,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:20:31.675561+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:20:31.675561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:45.364743+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":840,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.980421+0000","last_change":"2024-04-14T15:18:37.636498+0000","last_active":"2024-04-14T15:20:38.980421+0000","last_peered":"2024-04-14T15:20:38.980421+0000","last_clean":"2024-04-14T15:20:38.980421+0000","last_became_active":"2024-04-14T15:12:56.355348+0000","last_became_peered":"2024-04-14T15:12:56.355348+0000","last_unstale":"2024-04-14T15:20:38.980421+0000","last_undegraded":"2024-04-14T15:20:38.980421+0000","last_fullsized":"2024-04-14T15:20:38.980421+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:37.636419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:18:37.636419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"479'15","reported_seq":963,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.947046+0000","last_change":"2024-04-14T15:20:33.353872+0000","last_active":"2024-04-14T15:20:33.947046+0000","last_peered":"2024-04-14T15:20:33.947046+0000","last_clean":"2024-04-14T15:20:33.947046+0000","last_became_active":"2024-04-14T15:09:45.037179+0000","last_became_peered":"2024-04-14T15:09:45.037179+0000","last_unstale":"2024-04-14T15:20:33.947046+0000","last_undegraded":"2024-04-14T15:20:33.947046+0000","last_fullsized":"2024-04-14T15:20:33.947046+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'15","last_scrub_stamp":"2024-04-14T15:20:33.353777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:20:33.353777+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:00.604055+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":935,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.945996+0000","last_change":"2024-04-14T15:18:36.029705+0000","last_active":"2024-04-14T15:20:33.945996+0000","last_peered":"2024-04-14T15:20:33.945996+0000","last_clean":"2024-04-14T15:20:33.945996+0000","last_became_active":"2024-04-14T15:09:43.020044+0000","last_became_peered":"2024-04-14T15:09:43.020044+0000","last_unstale":"2024-04-14T15:20:33.945996+0000","last_undegraded":"2024-04-14T15:20:33.945996+0000","last_fullsized":"2024-04-14T15:20:33.945996+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:36.029630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:18:36.029630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":932,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.954448+0000","last_change":"2024-04-14T15:19:11.838298+0000","last_active":"2024-04-14T15:20:33.954448+0000","last_peered":"2024-04-14T15:20:33.954448+0000","last_clean":"2024-04-14T15:20:33.954448+0000","last_became_active":"2024-04-14T15:09:47.046153+0000","last_became_peered":"2024-04-14T15:09:47.046153+0000","last_unstale":"2024-04-14T15:20:33.954448+0000","last_undegraded":"2024-04-14T15:20:33.954448+0000","last_fullsized":"2024-04-14T15:20:33.954448+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:11.838212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:19:11.838212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":922,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.946555+0000","last_change":"2024-04-14T15:17:37.919665+0000","last_active":"2024-04-14T15:20:33.946555+0000","last_peered":"2024-04-14T15:20:33.946555+0000","last_clean":"2024-04-14T15:20:33.946555+0000","last_became_active":"2024-04-14T15:09:49.061495+0000","last_became_peered":"2024-04-14T15:09:49.061495+0000","last_unstale":"2024-04-14T15:20:33.946555+0000","last_undegraded":"2024-04-14T15:20:33.946555+0000","last_fullsized":"2024-04-14T15:20:33.946555+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:37.919595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:17:37.919595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"91'130","reported_seq":1031,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-04-14T15:20:37.984181+0000","last_change":"2024-04-14T15:19:06.754212+0000","last_active":"2024-04-14T15:20:37.984181+0000","last_peered":"2024-04-14T15:20:37.984181+0000","last_clean":"2024-04-14T15:20:37.984181+0000","last_became_active":"2024-04-14T15:18:45.919901+0000","last_became_peered":"2024-04-14T15:18:45.919901+0000","last_unstale":"2024-04-14T15:20:37.984181+0000","last_undegraded":"2024-04-14T15:20:37.984181+0000","last_fullsized":"2024-04-14T15:20:37.984181+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:06.754126+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:06.754126+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":840,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.981171+0000","last_change":"2024-04-14T15:18:25.523604+0000","last_active":"2024-04-14T15:20:38.981171+0000","last_peered":"2024-04-14T15:20:38.981171+0000","last_clean":"2024-04-14T15:20:38.981171+0000","last_became_active":"2024-04-14T15:12:56.246062+0000","last_became_peered":"2024-04-14T15:12:56.246062+0000","last_unstale":"2024-04-14T15:20:38.981171+0000","last_undegraded":"2024-04-14T15:20:38.981171+0000","last_fullsized":"2024-04-14T15:20:38.981171+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:25.523520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:18:25.523520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"479'44","reported_seq":1011,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-04-14T15:20:37.982482+0000","last_change":"2024-04-14T15:19:58.856267+0000","last_active":"2024-04-14T15:20:37.982482+0000","last_peered":"2024-04-14T15:20:37.982482+0000","last_clean":"2024-04-14T15:20:37.982482+0000","last_became_active":"2024-04-14T15:09:45.178782+0000","last_became_peered":"2024-04-14T15:09:45.178782+0000","last_unstale":"2024-04-14T15:20:37.982482+0000","last_undegraded":"2024-04-14T15:20:37.982482+0000","last_fullsized":"2024-04-14T15:20:37.982482+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'44","last_scrub_stamp":"2024-04-14T15:19:58.856115+0000","last_deep_scrub":"479'44","last_deep_scrub_stamp":"2024-04-14T15:19:58.856115+0000","last_clean_scrub_stamp":"2024-04-14T15:19:58.856115+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:10.582292+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":940,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.955466+0000","last_change":"2024-04-14T15:18:36.925573+0000","last_active":"2024-04-14T15:20:33.955466+0000","last_peered":"2024-04-14T15:20:33.955466+0000","last_clean":"2024-04-14T15:20:33.955466+0000","last_became_active":"2024-04-14T15:09:43.020268+0000","last_became_peered":"2024-04-14T15:09:43.020268+0000","last_unstale":"2024-04-14T15:20:33.955466+0000","last_undegraded":"2024-04-14T15:20:33.955466+0000","last_fullsized":"2024-04-14T15:20:33.955466+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:36.925496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:34.342793+0000","last_clean_scrub_stamp":"2024-04-14T15:18:36.925496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":924,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.954269+0000","last_change":"2024-04-14T15:18:31.989052+0000","last_active":"2024-04-14T15:20:33.954269+0000","last_peered":"2024-04-14T15:20:33.954269+0000","last_clean":"2024-04-14T15:20:33.954269+0000","last_became_active":"2024-04-14T15:09:47.045542+0000","last_became_peered":"2024-04-14T15:09:47.045542+0000","last_unstale":"2024-04-14T15:20:33.954269+0000","last_undegraded":"2024-04-14T15:20:33.954269+0000","last_fullsized":"2024-04-14T15:20:33.954269+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:31.988957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:18:31.988957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":932,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.980071+0000","last_change":"2024-04-14T15:19:16.566595+0000","last_active":"2024-04-14T15:20:38.980071+0000","last_peered":"2024-04-14T15:20:38.980071+0000","last_clean":"2024-04-14T15:20:38.980071+0000","last_became_active":"2024-04-14T15:09:49.065397+0000","last_became_peered":"2024-04-14T15:09:49.065397+0000","last_unstale":"2024-04-14T15:20:38.980071+0000","last_undegraded":"2024-04-14T15:20:38.980071+0000","last_fullsized":"2024-04-14T15:20:38.980071+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:16.566496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:19:16.566496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"91'130","reported_seq":822,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.953534+0000","last_change":"2024-04-14T15:19:04.951104+0000","last_active":"2024-04-14T15:20:33.953534+0000","last_peered":"2024-04-14T15:20:33.953534+0000","last_clean":"2024-04-14T15:20:33.953534+0000","last_became_active":"2024-04-14T15:18:58.181194+0000","last_became_peered":"2024-04-14T15:18:58.181194+0000","last_unstale":"2024-04-14T15:20:33.953534+0000","last_undegraded":"2024-04-14T15:20:33.953534+0000","last_fullsized":"2024-04-14T15:20:33.953534+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:04.950991+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:04.950991+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":837,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.954409+0000","last_change":"2024-04-14T15:19:15.778552+0000","last_active":"2024-04-14T15:20:33.954409+0000","last_peered":"2024-04-14T15:20:33.954409+0000","last_clean":"2024-04-14T15:20:33.954409+0000","last_became_active":"2024-04-14T15:12:56.474322+0000","last_became_peered":"2024-04-14T15:12:56.474322+0000","last_unstale":"2024-04-14T15:20:33.954409+0000","last_undegraded":"2024-04-14T15:20:33.954409+0000","last_fullsized":"2024-04-14T15:20:33.954409+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:15.778469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:19:15.778469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:40.510707+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"479'31","reported_seq":1036,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.983579+0000","last_change":"2024-04-14T15:17:48.374589+0000","last_active":"2024-04-14T15:20:38.983579+0000","last_peered":"2024-04-14T15:20:38.983579+0000","last_clean":"2024-04-14T15:20:38.983579+0000","last_became_active":"2024-04-14T15:13:12.933907+0000","last_became_peered":"2024-04-14T15:13:12.933907+0000","last_unstale":"2024-04-14T15:20:38.983579+0000","last_undegraded":"2024-04-14T15:20:38.983579+0000","last_fullsized":"2024-04-14T15:20:38.983579+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'17","last_scrub_stamp":"2024-04-14T15:17:48.374469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:17:48.374469+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":943,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.979661+0000","last_change":"2024-04-14T15:19:26.515133+0000","last_active":"2024-04-14T15:20:38.979661+0000","last_peered":"2024-04-14T15:20:38.979661+0000","last_clean":"2024-04-14T15:20:38.979661+0000","last_became_active":"2024-04-14T15:09:43.022351+0000","last_became_peered":"2024-04-14T15:09:43.022351+0000","last_unstale":"2024-04-14T15:20:38.979661+0000","last_undegraded":"2024-04-14T15:20:38.979661+0000","last_fullsized":"2024-04-14T15:20:38.979661+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:26.515055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:19:26.515055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:51.899280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":919,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.944974+0000","last_change":"2024-04-14T15:17:54.871693+0000","last_active":"2024-04-14T15:20:33.944974+0000","last_peered":"2024-04-14T15:20:33.944974+0000","last_clean":"2024-04-14T15:20:33.944974+0000","last_became_active":"2024-04-14T15:09:47.042810+0000","last_became_peered":"2024-04-14T15:09:47.042810+0000","last_unstale":"2024-04-14T15:20:33.944974+0000","last_undegraded":"2024-04-14T15:20:33.944974+0000","last_fullsized":"2024-04-14T15:20:33.944974+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:54.871593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:17:54.871593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":928,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.956230+0000","last_change":"2024-04-14T15:19:51.902814+0000","last_active":"2024-04-14T15:20:33.956230+0000","last_peered":"2024-04-14T15:20:33.956230+0000","last_clean":"2024-04-14T15:20:33.956230+0000","last_became_active":"2024-04-14T15:09:49.069594+0000","last_became_peered":"2024-04-14T15:09:49.069594+0000","last_unstale":"2024-04-14T15:20:33.956230+0000","last_undegraded":"2024-04-14T15:20:33.956230+0000","last_fullsized":"2024-04-14T15:20:33.956230+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:51.902730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:19:51.902730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:19.807697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"91'130","reported_seq":1236,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.979323+0000","last_change":"2024-04-14T15:19:06.490294+0000","last_active":"2024-04-14T15:20:38.979323+0000","last_peered":"2024-04-14T15:20:38.979323+0000","last_clean":"2024-04-14T15:20:38.979323+0000","last_became_active":"2024-04-14T15:18:58.189483+0000","last_became_peered":"2024-04-14T15:18:58.189483+0000","last_unstale":"2024-04-14T15:20:38.979323+0000","last_undegraded":"2024-04-14T15:20:38.979323+0000","last_fullsized":"2024-04-14T15:20:38.979323+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:06.490208+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:06.490208+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":847,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-04-14T15:20:37.981735+0000","last_change":"2024-04-14T15:20:02.774050+0000","last_active":"2024-04-14T15:20:37.981735+0000","last_peered":"2024-04-14T15:20:37.981735+0000","last_clean":"2024-04-14T15:20:37.981735+0000","last_became_active":"2024-04-14T15:12:56.289745+0000","last_became_peered":"2024-04-14T15:12:56.289745+0000","last_unstale":"2024-04-14T15:20:37.981735+0000","last_undegraded":"2024-04-14T15:20:37.981735+0000","last_fullsized":"2024-04-14T15:20:37.981735+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:02.773961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:20:02.773961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:08.770183+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"479'25","reported_seq":974,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.982707+0000","last_change":"2024-04-14T15:20:03.571298+0000","last_active":"2024-04-14T15:20:38.982707+0000","last_peered":"2024-04-14T15:20:38.982707+0000","last_clean":"2024-04-14T15:20:38.982707+0000","last_became_active":"2024-04-14T15:09:45.029170+0000","last_became_peered":"2024-04-14T15:09:45.029170+0000","last_unstale":"2024-04-14T15:20:38.982707+0000","last_undegraded":"2024-04-14T15:20:38.982707+0000","last_fullsized":"2024-04-14T15:20:38.982707+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'25","last_scrub_stamp":"2024-04-14T15:20:03.571160+0000","last_deep_scrub":"479'25","last_deep_scrub_stamp":"2024-04-14T15:20:03.571160+0000","last_clean_scrub_stamp":"2024-04-14T15:20:03.571160+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:14.606809+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":936,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.944724+0000","last_change":"2024-04-14T15:17:51.854742+0000","last_active":"2024-04-14T15:20:33.944724+0000","last_peered":"2024-04-14T15:20:33.944724+0000","last_clean":"2024-04-14T15:20:33.944724+0000","last_became_active":"2024-04-14T15:09:43.018871+0000","last_became_peered":"2024-04-14T15:09:43.018871+0000","last_unstale":"2024-04-14T15:20:33.944724+0000","last_undegraded":"2024-04-14T15:20:33.944724+0000","last_fullsized":"2024-04-14T15:20:33.944724+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:51.854659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:51.854659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"400'9","reported_seq":1064,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.955669+0000","last_change":"2024-04-14T15:17:54.845967+0000","last_active":"2024-04-14T15:20:33.955669+0000","last_peered":"2024-04-14T15:20:33.955669+0000","last_clean":"2024-04-14T15:20:33.955669+0000","last_became_active":"2024-04-14T15:09:47.044085+0000","last_became_peered":"2024-04-14T15:09:47.044085+0000","last_unstale":"2024-04-14T15:20:33.955669+0000","last_undegraded":"2024-04-14T15:20:33.955669+0000","last_fullsized":"2024-04-14T15:20:33.955669+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'9","last_scrub_stamp":"2024-04-14T15:17:54.845881+0000","last_deep_scrub":"400'9","last_deep_scrub_stamp":"2024-04-14T15:17:54.845881+0000","last_clean_scrub_stamp":"2024-04-14T15:17:54.845881+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":931,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.983310+0000","last_change":"2024-04-14T15:19:18.588681+0000","last_active":"2024-04-14T15:20:38.983310+0000","last_peered":"2024-04-14T15:20:38.983310+0000","last_clean":"2024-04-14T15:20:38.983310+0000","last_became_active":"2024-04-14T15:09:49.065274+0000","last_became_peered":"2024-04-14T15:09:49.065274+0000","last_unstale":"2024-04-14T15:20:38.983310+0000","last_undegraded":"2024-04-14T15:20:38.983310+0000","last_fullsized":"2024-04-14T15:20:38.983310+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:18.588546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:18.588546+0000","last_clean_scrub_stamp":"2024-04-14T15:19:18.588546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"91'130","reported_seq":830,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.954230+0000","last_change":"2024-04-14T15:19:05.972132+0000","last_active":"2024-04-14T15:20:33.954230+0000","last_peered":"2024-04-14T15:20:33.954230+0000","last_clean":"2024-04-14T15:20:33.954230+0000","last_became_active":"2024-04-14T15:18:58.187649+0000","last_became_peered":"2024-04-14T15:18:58.187649+0000","last_unstale":"2024-04-14T15:20:33.954230+0000","last_undegraded":"2024-04-14T15:20:33.954230+0000","last_fullsized":"2024-04-14T15:20:33.954230+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:05.972050+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:05.972050+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":832,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.943158+0000","last_change":"2024-04-14T15:19:11.008294+0000","last_active":"2024-04-14T15:20:33.943158+0000","last_peered":"2024-04-14T15:20:33.943158+0000","last_clean":"2024-04-14T15:20:33.943158+0000","last_became_active":"2024-04-14T15:12:56.487615+0000","last_became_peered":"2024-04-14T15:12:56.487615+0000","last_unstale":"2024-04-14T15:20:33.943158+0000","last_undegraded":"2024-04-14T15:20:33.943158+0000","last_fullsized":"2024-04-14T15:20:33.943158+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:11.008225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:19:11.008225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"479'34","reported_seq":981,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.945039+0000","last_change":"2024-04-14T15:17:52.883984+0000","last_active":"2024-04-14T15:20:33.945039+0000","last_peered":"2024-04-14T15:20:33.945039+0000","last_clean":"2024-04-14T15:20:33.945039+0000","last_became_active":"2024-04-14T15:09:45.033331+0000","last_became_peered":"2024-04-14T15:09:45.033331+0000","last_unstale":"2024-04-14T15:20:33.945039+0000","last_undegraded":"2024-04-14T15:20:33.945039+0000","last_fullsized":"2024-04-14T15:20:33.945039+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'22","last_scrub_stamp":"2024-04-14T15:17:52.883850+0000","last_deep_scrub":"398'22","last_deep_scrub_stamp":"2024-04-14T15:17:52.883850+0000","last_clean_scrub_stamp":"2024-04-14T15:17:52.883850+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":936,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.944316+0000","last_change":"2024-04-14T15:18:25.009179+0000","last_active":"2024-04-14T15:20:33.944316+0000","last_peered":"2024-04-14T15:20:33.944316+0000","last_clean":"2024-04-14T15:20:33.944316+0000","last_became_active":"2024-04-14T15:09:43.020878+0000","last_became_peered":"2024-04-14T15:09:43.020878+0000","last_unstale":"2024-04-14T15:20:33.944316+0000","last_undegraded":"2024-04-14T15:20:33.944316+0000","last_fullsized":"2024-04-14T15:20:33.944316+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:25.009100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:18:25.009100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"72'8","reported_seq":1075,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.983278+0000","last_change":"2024-04-14T15:19:19.596882+0000","last_active":"2024-04-14T15:20:38.983278+0000","last_peered":"2024-04-14T15:20:38.983278+0000","last_clean":"2024-04-14T15:20:38.983278+0000","last_became_active":"2024-04-14T15:09:47.050962+0000","last_became_peered":"2024-04-14T15:09:47.050962+0000","last_unstale":"2024-04-14T15:20:38.983278+0000","last_undegraded":"2024-04-14T15:20:38.983278+0000","last_fullsized":"2024-04-14T15:20:38.983278+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:19:19.596773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:19:19.596773+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:44.419959+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":487,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.945740+0000","last_change":"2024-04-14T15:16:51.199025+0000","last_active":"2024-04-14T15:20:33.945740+0000","last_peered":"2024-04-14T15:20:33.945740+0000","last_clean":"2024-04-14T15:20:33.945740+0000","last_became_active":"2024-04-14T15:15:44.879699+0000","last_became_peered":"2024-04-14T15:15:44.879699+0000","last_unstale":"2024-04-14T15:20:33.945740+0000","last_undegraded":"2024-04-14T15:20:33.945740+0000","last_fullsized":"2024-04-14T15:20:33.945740+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:51.198907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:51.198907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"91'130","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.943135+0000","last_change":"2024-04-14T15:19:08.097458+0000","last_active":"2024-04-14T15:20:33.943135+0000","last_peered":"2024-04-14T15:20:33.943135+0000","last_clean":"2024-04-14T15:20:33.943135+0000","last_became_active":"2024-04-14T15:18:58.179785+0000","last_became_peered":"2024-04-14T15:18:58.179785+0000","last_unstale":"2024-04-14T15:20:33.943135+0000","last_undegraded":"2024-04-14T15:20:33.943135+0000","last_fullsized":"2024-04-14T15:20:33.943135+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:08.097372+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:08.097372+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"233'8","reported_seq":861,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-04-14T15:20:37.981701+0000","last_change":"2024-04-14T15:20:37.855519+0000","last_active":"2024-04-14T15:20:37.981701+0000","last_peered":"2024-04-14T15:20:37.981701+0000","last_clean":"2024-04-14T15:20:37.981701+0000","last_became_active":"2024-04-14T15:12:56.381026+0000","last_became_peered":"2024-04-14T15:12:56.381026+0000","last_unstale":"2024-04-14T15:20:37.981701+0000","last_undegraded":"2024-04-14T15:20:37.981701+0000","last_fullsized":"2024-04-14T15:20:37.981701+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'8","last_scrub_stamp":"2024-04-14T15:20:37.855435+0000","last_deep_scrub":"233'8","last_deep_scrub_stamp":"2024-04-14T15:15:01.295800+0000","last_clean_scrub_stamp":"2024-04-14T15:20:37.855435+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:05.091035+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"479'24","reported_seq":975,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.983122+0000","last_change":"2024-04-14T15:16:53.143329+0000","last_active":"2024-04-14T15:20:38.983122+0000","last_peered":"2024-04-14T15:20:38.983122+0000","last_clean":"2024-04-14T15:20:38.983122+0000","last_became_active":"2024-04-14T15:09:45.028212+0000","last_became_peered":"2024-04-14T15:09:45.028212+0000","last_unstale":"2024-04-14T15:20:38.983122+0000","last_undegraded":"2024-04-14T15:20:38.983122+0000","last_fullsized":"2024-04-14T15:20:38.983122+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-14T15:16:53.143239+0000","last_deep_scrub":"85'16","last_deep_scrub_stamp":"2024-04-14T15:10:47.456169+0000","last_clean_scrub_stamp":"2024-04-14T15:16:53.143239+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":952,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.979932+0000","last_change":"2024-04-14T15:20:03.642215+0000","last_active":"2024-04-14T15:20:38.979932+0000","last_peered":"2024-04-14T15:20:38.979932+0000","last_clean":"2024-04-14T15:20:38.979932+0000","last_became_active":"2024-04-14T15:09:43.018601+0000","last_became_peered":"2024-04-14T15:09:43.018601+0000","last_unstale":"2024-04-14T15:20:38.979932+0000","last_undegraded":"2024-04-14T15:20:38.979932+0000","last_fullsized":"2024-04-14T15:20:38.979932+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:03.642112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:20:03.642112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:09.466843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":502,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.983365+0000","last_change":"2024-04-14T15:19:59.597112+0000","last_active":"2024-04-14T15:20:38.983365+0000","last_peered":"2024-04-14T15:20:38.983365+0000","last_clean":"2024-04-14T15:20:38.983365+0000","last_became_active":"2024-04-14T15:15:44.886834+0000","last_became_peered":"2024-04-14T15:15:44.886834+0000","last_unstale":"2024-04-14T15:20:38.983365+0000","last_undegraded":"2024-04-14T15:20:38.983365+0000","last_fullsized":"2024-04-14T15:20:38.983365+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:59.596964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:19:59.596964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:07.158046+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":596,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.958579+0000","last_change":"2024-04-14T15:17:57.214134+0000","last_active":"2024-04-14T15:20:33.958579+0000","last_peered":"2024-04-14T15:20:33.958579+0000","last_clean":"2024-04-14T15:20:33.958579+0000","last_became_active":"2024-04-14T15:15:21.888587+0000","last_became_peered":"2024-04-14T15:15:21.888587+0000","last_unstale":"2024-04-14T15:20:33.958579+0000","last_undegraded":"2024-04-14T15:20:33.958579+0000","last_fullsized":"2024-04-14T15:20:33.958579+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:57.214046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:57.214046+0000","last_clean_scrub_stamp":"2024-04-14T15:17:57.214046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"91'130","reported_seq":1004,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.958425+0000","last_change":"2024-04-14T15:19:05.396196+0000","last_active":"2024-04-14T15:20:33.958425+0000","last_peered":"2024-04-14T15:20:33.958425+0000","last_clean":"2024-04-14T15:20:33.958425+0000","last_became_active":"2024-04-14T15:18:40.917537+0000","last_became_peered":"2024-04-14T15:18:40.917537+0000","last_unstale":"2024-04-14T15:20:33.958425+0000","last_undegraded":"2024-04-14T15:20:33.958425+0000","last_fullsized":"2024-04-14T15:20:33.958425+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:05.396090+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:05.396090+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":848,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.983059+0000","last_change":"2024-04-14T15:20:37.765993+0000","last_active":"2024-04-14T15:20:38.983059+0000","last_peered":"2024-04-14T15:20:38.983059+0000","last_clean":"2024-04-14T15:20:38.983059+0000","last_became_active":"2024-04-14T15:12:56.427276+0000","last_became_peered":"2024-04-14T15:12:56.427276+0000","last_unstale":"2024-04-14T15:20:38.983059+0000","last_undegraded":"2024-04-14T15:20:38.983059+0000","last_fullsized":"2024-04-14T15:20:38.983059+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:37.765916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:20:37.765916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:45.507528+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"255'10","reported_seq":708,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.957045+0000","last_change":"2024-04-14T15:19:50.669678+0000","last_active":"2024-04-14T15:20:33.957045+0000","last_peered":"2024-04-14T15:20:33.957045+0000","last_clean":"2024-04-14T15:20:33.957045+0000","last_became_active":"2024-04-14T15:15:21.873747+0000","last_became_peered":"2024-04-14T15:15:21.873747+0000","last_unstale":"2024-04-14T15:20:33.957045+0000","last_undegraded":"2024-04-14T15:20:33.957045+0000","last_fullsized":"2024-04-14T15:20:33.957045+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'10","last_scrub_stamp":"2024-04-14T15:19:50.669574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:19:50.669574+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:12.441595+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"479'28","reported_seq":982,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.981906+0000","last_change":"2024-04-14T15:18:22.545734+0000","last_active":"2024-04-14T15:20:38.981906+0000","last_peered":"2024-04-14T15:20:38.981906+0000","last_clean":"2024-04-14T15:20:38.981906+0000","last_became_active":"2024-04-14T15:09:45.025673+0000","last_became_peered":"2024-04-14T15:09:45.025673+0000","last_unstale":"2024-04-14T15:20:38.981906+0000","last_undegraded":"2024-04-14T15:20:38.981906+0000","last_fullsized":"2024-04-14T15:20:38.981906+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'14","last_scrub_stamp":"2024-04-14T15:18:22.545646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:18:22.545646+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":943,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-04-14T15:20:37.984365+0000","last_change":"2024-04-14T15:18:26.824852+0000","last_active":"2024-04-14T15:20:37.984365+0000","last_peered":"2024-04-14T15:20:37.984365+0000","last_clean":"2024-04-14T15:20:37.984365+0000","last_became_active":"2024-04-14T15:09:43.020503+0000","last_became_peered":"2024-04-14T15:09:43.020503+0000","last_unstale":"2024-04-14T15:20:37.984365+0000","last_undegraded":"2024-04-14T15:20:37.984365+0000","last_fullsized":"2024-04-14T15:20:37.984365+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:26.824773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:18:26.824773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":932,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.980391+0000","last_change":"2024-04-14T15:19:50.609813+0000","last_active":"2024-04-14T15:20:38.980391+0000","last_peered":"2024-04-14T15:20:38.980391+0000","last_clean":"2024-04-14T15:20:38.980391+0000","last_became_active":"2024-04-14T15:09:49.129610+0000","last_became_peered":"2024-04-14T15:09:49.129610+0000","last_unstale":"2024-04-14T15:20:38.980391+0000","last_undegraded":"2024-04-14T15:20:38.980391+0000","last_fullsized":"2024-04-14T15:20:38.980391+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:50.609691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:19:50.609691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:58.284777+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"91'130","reported_seq":1023,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.982838+0000","last_change":"2024-04-14T15:19:04.526327+0000","last_active":"2024-04-14T15:20:38.982838+0000","last_peered":"2024-04-14T15:20:38.982838+0000","last_clean":"2024-04-14T15:20:38.982838+0000","last_became_active":"2024-04-14T15:18:58.179181+0000","last_became_peered":"2024-04-14T15:18:58.179181+0000","last_unstale":"2024-04-14T15:20:38.982838+0000","last_undegraded":"2024-04-14T15:20:38.982838+0000","last_fullsized":"2024-04-14T15:20:38.982838+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:04.526214+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:04.526214+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":840,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.980816+0000","last_change":"2024-04-14T15:17:47.553179+0000","last_active":"2024-04-14T15:20:38.980816+0000","last_peered":"2024-04-14T15:20:38.980816+0000","last_clean":"2024-04-14T15:20:38.980816+0000","last_became_active":"2024-04-14T15:12:56.355254+0000","last_became_peered":"2024-04-14T15:12:56.355254+0000","last_unstale":"2024-04-14T15:20:38.980816+0000","last_undegraded":"2024-04-14T15:20:38.980816+0000","last_fullsized":"2024-04-14T15:20:38.980816+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:47.553055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:17:47.553055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":936,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.982042+0000","last_change":"2024-04-14T15:18:39.394736+0000","last_active":"2024-04-14T15:20:38.982042+0000","last_peered":"2024-04-14T15:20:38.982042+0000","last_clean":"2024-04-14T15:20:38.982042+0000","last_became_active":"2024-04-14T15:09:47.044610+0000","last_became_peered":"2024-04-14T15:09:47.044610+0000","last_unstale":"2024-04-14T15:20:38.982042+0000","last_undegraded":"2024-04-14T15:20:38.982042+0000","last_fullsized":"2024-04-14T15:20:38.982042+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:39.394655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:39.394655+0000","last_clean_scrub_stamp":"2024-04-14T15:18:39.394655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"479'30","reported_seq":663,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.956997+0000","last_change":"2024-04-14T15:19:26.509637+0000","last_active":"2024-04-14T15:20:33.956997+0000","last_peered":"2024-04-14T15:20:33.956997+0000","last_clean":"2024-04-14T15:20:33.956997+0000","last_became_active":"2024-04-14T15:15:21.880565+0000","last_became_peered":"2024-04-14T15:15:21.880565+0000","last_unstale":"2024-04-14T15:20:33.956997+0000","last_undegraded":"2024-04-14T15:20:33.956997+0000","last_fullsized":"2024-04-14T15:20:33.956997+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'18","last_scrub_stamp":"2024-04-14T15:19:26.509541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:19:26.509541+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:53.956765+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'2","reported_seq":930,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.944234+0000","last_change":"2024-04-14T15:17:48.850178+0000","last_active":"2024-04-14T15:20:33.944234+0000","last_peered":"2024-04-14T15:20:33.944234+0000","last_clean":"2024-04-14T15:20:33.944234+0000","last_became_active":"2024-04-14T15:09:43.018478+0000","last_became_peered":"2024-04-14T15:09:43.018478+0000","last_unstale":"2024-04-14T15:20:33.944234+0000","last_undegraded":"2024-04-14T15:20:33.944234+0000","last_fullsized":"2024-04-14T15:20:33.944234+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'2","last_scrub_stamp":"2024-04-14T15:17:48.850069+0000","last_deep_scrub":"91'2","last_deep_scrub_stamp":"2024-04-14T15:17:48.850069+0000","last_clean_scrub_stamp":"2024-04-14T15:17:48.850069+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":932,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.982402+0000","last_change":"2024-04-14T15:19:14.609181+0000","last_active":"2024-04-14T15:20:38.982402+0000","last_peered":"2024-04-14T15:20:38.982402+0000","last_clean":"2024-04-14T15:20:38.982402+0000","last_became_active":"2024-04-14T15:09:49.065684+0000","last_became_peered":"2024-04-14T15:09:49.065684+0000","last_unstale":"2024-04-14T15:20:38.982402+0000","last_undegraded":"2024-04-14T15:20:38.982402+0000","last_fullsized":"2024-04-14T15:20:38.982402+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:14.609079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:19:14.609079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"91'130","reported_seq":1028,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-04-14T15:20:37.983458+0000","last_change":"2024-04-14T15:19:05.797940+0000","last_active":"2024-04-14T15:20:37.983458+0000","last_peered":"2024-04-14T15:20:37.983458+0000","last_clean":"2024-04-14T15:20:37.983458+0000","last_became_active":"2024-04-14T15:18:58.182649+0000","last_became_peered":"2024-04-14T15:18:58.182649+0000","last_unstale":"2024-04-14T15:20:37.983458+0000","last_undegraded":"2024-04-14T15:20:37.983458+0000","last_fullsized":"2024-04-14T15:20:37.983458+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:05.797859+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:05.797859+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":832,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.944773+0000","last_change":"2024-04-14T15:18:35.004058+0000","last_active":"2024-04-14T15:20:33.944773+0000","last_peered":"2024-04-14T15:20:33.944773+0000","last_clean":"2024-04-14T15:20:33.944773+0000","last_became_active":"2024-04-14T15:12:56.402023+0000","last_became_peered":"2024-04-14T15:12:56.402023+0000","last_unstale":"2024-04-14T15:20:33.944773+0000","last_undegraded":"2024-04-14T15:20:33.944773+0000","last_fullsized":"2024-04-14T15:20:33.944773+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:35.003968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:35.003968+0000","last_clean_scrub_stamp":"2024-04-14T15:18:35.003968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":943,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-04-14T15:20:37.982351+0000","last_change":"2024-04-14T15:20:06.794160+0000","last_active":"2024-04-14T15:20:37.982351+0000","last_peered":"2024-04-14T15:20:37.982351+0000","last_clean":"2024-04-14T15:20:37.982351+0000","last_became_active":"2024-04-14T15:09:47.048427+0000","last_became_peered":"2024-04-14T15:09:47.048427+0000","last_unstale":"2024-04-14T15:20:37.982351+0000","last_undegraded":"2024-04-14T15:20:37.982351+0000","last_fullsized":"2024-04-14T15:20:37.982351+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:06.794023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:20:06.794023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:16.849759+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"479'37","reported_seq":990,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.947205+0000","last_change":"2024-04-14T15:17:46.924432+0000","last_active":"2024-04-14T15:20:33.947205+0000","last_peered":"2024-04-14T15:20:33.947205+0000","last_clean":"2024-04-14T15:20:33.947205+0000","last_became_active":"2024-04-14T15:09:45.179182+0000","last_became_peered":"2024-04-14T15:09:45.179182+0000","last_unstale":"2024-04-14T15:20:33.947205+0000","last_undegraded":"2024-04-14T15:20:33.947205+0000","last_fullsized":"2024-04-14T15:20:33.947205+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'27","last_scrub_stamp":"2024-04-14T15:17:46.924330+0000","last_deep_scrub":"102'27","last_deep_scrub_stamp":"2024-04-14T15:17:46.924330+0000","last_clean_scrub_stamp":"2024-04-14T15:17:46.924330+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":944,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.980971+0000","last_change":"2024-04-14T15:19:14.569862+0000","last_active":"2024-04-14T15:20:38.980971+0000","last_peered":"2024-04-14T15:20:38.980971+0000","last_clean":"2024-04-14T15:20:38.980971+0000","last_became_active":"2024-04-14T15:09:43.021006+0000","last_became_peered":"2024-04-14T15:09:43.021006+0000","last_unstale":"2024-04-14T15:20:38.980971+0000","last_undegraded":"2024-04-14T15:20:38.980971+0000","last_fullsized":"2024-04-14T15:20:38.980971+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:14.569778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:19:14.569778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":924,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.944369+0000","last_change":"2024-04-14T15:20:00.976745+0000","last_active":"2024-04-14T15:20:33.944369+0000","last_peered":"2024-04-14T15:20:33.944369+0000","last_clean":"2024-04-14T15:20:33.944369+0000","last_became_active":"2024-04-14T15:09:49.128136+0000","last_became_peered":"2024-04-14T15:09:49.128136+0000","last_unstale":"2024-04-14T15:20:33.944369+0000","last_undegraded":"2024-04-14T15:20:33.944369+0000","last_fullsized":"2024-04-14T15:20:33.944369+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:00.976659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:56.206649+0000","last_clean_scrub_stamp":"2024-04-14T15:20:00.976659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:01.483885+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"91'130","reported_seq":1008,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.943225+0000","last_change":"2024-04-14T15:19:09.060997+0000","last_active":"2024-04-14T15:20:33.943225+0000","last_peered":"2024-04-14T15:20:33.943225+0000","last_clean":"2024-04-14T15:20:33.943225+0000","last_became_active":"2024-04-14T15:18:40.910133+0000","last_became_peered":"2024-04-14T15:18:40.910133+0000","last_unstale":"2024-04-14T15:20:33.943225+0000","last_undegraded":"2024-04-14T15:20:33.943225+0000","last_fullsized":"2024-04-14T15:20:33.943225+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:09.060906+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:09.060906+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:37.763955+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":544,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.957102+0000","last_change":"2024-04-14T15:19:47.610791+0000","last_active":"2024-04-14T15:20:33.957102+0000","last_peered":"2024-04-14T15:20:33.957102+0000","last_clean":"2024-04-14T15:20:33.957102+0000","last_became_active":"2024-04-14T15:15:21.887769+0000","last_became_peered":"2024-04-14T15:15:21.887769+0000","last_unstale":"2024-04-14T15:20:33.957102+0000","last_undegraded":"2024-04-14T15:20:33.957102+0000","last_fullsized":"2024-04-14T15:20:33.957102+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:47.610667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:19:47.610667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:55.096189+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":486,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.945479+0000","last_change":"2024-04-14T15:17:47.871801+0000","last_active":"2024-04-14T15:20:33.945479+0000","last_peered":"2024-04-14T15:20:33.945479+0000","last_clean":"2024-04-14T15:20:33.945479+0000","last_became_active":"2024-04-14T15:15:44.883694+0000","last_became_peered":"2024-04-14T15:15:44.883694+0000","last_unstale":"2024-04-14T15:20:33.945479+0000","last_undegraded":"2024-04-14T15:20:33.945479+0000","last_fullsized":"2024-04-14T15:20:33.945479+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:47.871722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:17:47.871722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"479'30","reported_seq":973,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.947602+0000","last_change":"2024-04-14T15:17:44.919445+0000","last_active":"2024-04-14T15:20:33.947602+0000","last_peered":"2024-04-14T15:20:33.947602+0000","last_clean":"2024-04-14T15:20:33.947602+0000","last_became_active":"2024-04-14T15:09:45.037231+0000","last_became_peered":"2024-04-14T15:09:45.037231+0000","last_unstale":"2024-04-14T15:20:33.947602+0000","last_undegraded":"2024-04-14T15:20:33.947602+0000","last_fullsized":"2024-04-14T15:20:33.947602+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'18","last_scrub_stamp":"2024-04-14T15:17:44.919358+0000","last_deep_scrub":"182'18","last_deep_scrub_stamp":"2024-04-14T15:14:10.383906+0000","last_clean_scrub_stamp":"2024-04-14T15:17:44.919358+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":944,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.979955+0000","last_change":"2024-04-14T15:17:46.542125+0000","last_active":"2024-04-14T15:20:38.979955+0000","last_peered":"2024-04-14T15:20:38.979955+0000","last_clean":"2024-04-14T15:20:38.979955+0000","last_became_active":"2024-04-14T15:09:43.025095+0000","last_became_peered":"2024-04-14T15:09:43.025095+0000","last_unstale":"2024-04-14T15:20:38.979955+0000","last_undegraded":"2024-04-14T15:20:38.979955+0000","last_fullsized":"2024-04-14T15:20:38.979955+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:46.542030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:46.542030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":942,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.946138+0000","last_change":"2024-04-14T15:17:48.866222+0000","last_active":"2024-04-14T15:20:33.946138+0000","last_peered":"2024-04-14T15:20:33.946138+0000","last_clean":"2024-04-14T15:20:33.946138+0000","last_became_active":"2024-04-14T15:09:43.018509+0000","last_became_peered":"2024-04-14T15:09:43.018509+0000","last_unstale":"2024-04-14T15:20:33.946138+0000","last_undegraded":"2024-04-14T15:20:33.946138+0000","last_fullsized":"2024-04-14T15:20:33.946138+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:48.866146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:48.866146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0018592680000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"355'2","reported_seq":926,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.945152+0000","last_change":"2024-04-14T15:20:04.842983+0000","last_active":"2024-04-14T15:20:33.945152+0000","last_peered":"2024-04-14T15:20:33.945152+0000","last_clean":"2024-04-14T15:20:33.945152+0000","last_became_active":"2024-04-14T15:09:49.065243+0000","last_became_peered":"2024-04-14T15:09:49.065243+0000","last_unstale":"2024-04-14T15:20:33.945152+0000","last_undegraded":"2024-04-14T15:20:33.945152+0000","last_fullsized":"2024-04-14T15:20:33.945152+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'2","last_scrub_stamp":"2024-04-14T15:20:04.842881+0000","last_deep_scrub":"355'2","last_deep_scrub_stamp":"2024-04-14T15:20:04.842881+0000","last_clean_scrub_stamp":"2024-04-14T15:20:04.842881+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:06.921319+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":528,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.957377+0000","last_change":"2024-04-14T15:18:32.532077+0000","last_active":"2024-04-14T15:20:33.957377+0000","last_peered":"2024-04-14T15:20:33.957377+0000","last_clean":"2024-04-14T15:20:33.957377+0000","last_became_active":"2024-04-14T15:15:21.890115+0000","last_became_peered":"2024-04-14T15:15:21.890115+0000","last_unstale":"2024-04-14T15:20:33.957377+0000","last_undegraded":"2024-04-14T15:20:33.957377+0000","last_fullsized":"2024-04-14T15:20:33.957377+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:32.531993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:18:32.531993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":936,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.980654+0000","last_change":"2024-04-14T15:19:46.631011+0000","last_active":"2024-04-14T15:20:38.980654+0000","last_peered":"2024-04-14T15:20:38.980654+0000","last_clean":"2024-04-14T15:20:38.980654+0000","last_became_active":"2024-04-14T15:09:47.047201+0000","last_became_peered":"2024-04-14T15:09:47.047201+0000","last_unstale":"2024-04-14T15:20:38.980654+0000","last_undegraded":"2024-04-14T15:20:38.980654+0000","last_fullsized":"2024-04-14T15:20:38.980654+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:46.630886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:19:46.630886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:47.188067+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"479'36","reported_seq":1119,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-04-14T15:20:37.982806+0000","last_change":"2024-04-14T15:19:02.841789+0000","last_active":"2024-04-14T15:20:37.982806+0000","last_peered":"2024-04-14T15:20:37.982806+0000","last_clean":"2024-04-14T15:20:37.982806+0000","last_became_active":"2024-04-14T15:09:45.179978+0000","last_became_peered":"2024-04-14T15:09:45.179978+0000","last_unstale":"2024-04-14T15:20:37.982806+0000","last_undegraded":"2024-04-14T15:20:37.982806+0000","last_fullsized":"2024-04-14T15:20:37.982806+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'24","last_scrub_stamp":"2024-04-14T15:19:02.841668+0000","last_deep_scrub":"90'24","last_deep_scrub_stamp":"2024-04-14T15:11:25.120692+0000","last_clean_scrub_stamp":"2024-04-14T15:19:02.841668+0000","objects_scrubbed":8,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"91'130","reported_seq":1229,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.979701+0000","last_change":"2024-04-14T15:19:08.494753+0000","last_active":"2024-04-14T15:20:38.979701+0000","last_peered":"2024-04-14T15:20:38.979701+0000","last_clean":"2024-04-14T15:20:38.979701+0000","last_became_active":"2024-04-14T15:18:40.911206+0000","last_became_peered":"2024-04-14T15:18:40.911206+0000","last_unstale":"2024-04-14T15:20:38.979701+0000","last_undegraded":"2024-04-14T15:20:38.979701+0000","last_fullsized":"2024-04-14T15:20:38.979701+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:08.494666+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:08.494666+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":499,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.954335+0000","last_change":"2024-04-14T15:20:31.776844+0000","last_active":"2024-04-14T15:20:33.954335+0000","last_peered":"2024-04-14T15:20:33.954335+0000","last_clean":"2024-04-14T15:20:33.954335+0000","last_became_active":"2024-04-14T15:15:44.885315+0000","last_became_peered":"2024-04-14T15:15:44.885315+0000","last_unstale":"2024-04-14T15:20:33.954335+0000","last_undegraded":"2024-04-14T15:20:33.954335+0000","last_fullsized":"2024-04-14T15:20:33.954335+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:31.776767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:47.627305+0000","last_clean_scrub_stamp":"2024-04-14T15:20:31.776767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:40.461595+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"69'2","reported_seq":618,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.957501+0000","last_change":"2024-04-14T15:16:53.972591+0000","last_active":"2024-04-14T15:20:33.957501+0000","last_peered":"2024-04-14T15:20:33.957501+0000","last_clean":"2024-04-14T15:20:33.957501+0000","last_became_active":"2024-04-14T15:15:21.889851+0000","last_became_peered":"2024-04-14T15:15:21.889851+0000","last_unstale":"2024-04-14T15:20:33.957501+0000","last_undegraded":"2024-04-14T15:20:33.957501+0000","last_fullsized":"2024-04-14T15:20:33.957501+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-04-14T15:16:53.972502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:53.972502+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0058046169999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":491,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.960229+0000","last_change":"2024-04-14T15:20:03.807964+0000","last_active":"2024-04-14T15:20:33.960229+0000","last_peered":"2024-04-14T15:20:33.960229+0000","last_clean":"2024-04-14T15:20:33.960229+0000","last_became_active":"2024-04-14T15:15:44.887934+0000","last_became_peered":"2024-04-14T15:15:44.887934+0000","last_unstale":"2024-04-14T15:20:33.960229+0000","last_undegraded":"2024-04-14T15:20:33.960229+0000","last_fullsized":"2024-04-14T15:20:33.960229+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:03.807844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:20:03.807844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:27.650888+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"91'130","reported_seq":1237,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.979433+0000","last_change":"2024-04-14T15:19:07.501791+0000","last_active":"2024-04-14T15:20:38.979433+0000","last_peered":"2024-04-14T15:20:38.979433+0000","last_clean":"2024-04-14T15:20:38.979433+0000","last_became_active":"2024-04-14T15:18:40.912277+0000","last_became_peered":"2024-04-14T15:18:40.912277+0000","last_unstale":"2024-04-14T15:20:38.979433+0000","last_undegraded":"2024-04-14T15:20:38.979433+0000","last_fullsized":"2024-04-14T15:20:38.979433+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:07.501704+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:07.501704+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":536,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.956898+0000","last_change":"2024-04-14T15:19:46.589280+0000","last_active":"2024-04-14T15:20:33.956898+0000","last_peered":"2024-04-14T15:20:33.956898+0000","last_clean":"2024-04-14T15:20:33.956898+0000","last_became_active":"2024-04-14T15:15:21.889970+0000","last_became_peered":"2024-04-14T15:15:21.889970+0000","last_unstale":"2024-04-14T15:20:33.956898+0000","last_undegraded":"2024-04-14T15:20:33.956898+0000","last_fullsized":"2024-04-14T15:20:33.956898+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:46.589189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:19:46.589189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:15.849303+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":932,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.956363+0000","last_change":"2024-04-14T15:20:00.850190+0000","last_active":"2024-04-14T15:20:33.956363+0000","last_peered":"2024-04-14T15:20:33.956363+0000","last_clean":"2024-04-14T15:20:33.956363+0000","last_became_active":"2024-04-14T15:09:47.049457+0000","last_became_peered":"2024-04-14T15:09:47.049457+0000","last_unstale":"2024-04-14T15:20:33.956363+0000","last_undegraded":"2024-04-14T15:20:33.956363+0000","last_fullsized":"2024-04-14T15:20:33.956363+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:00.850117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:20:00.850117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:29.884274+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"479'22","reported_seq":969,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.945327+0000","last_change":"2024-04-14T15:19:26.078482+0000","last_active":"2024-04-14T15:20:33.945327+0000","last_peered":"2024-04-14T15:20:33.945327+0000","last_clean":"2024-04-14T15:20:33.945327+0000","last_became_active":"2024-04-14T15:09:45.033883+0000","last_became_peered":"2024-04-14T15:09:45.033883+0000","last_unstale":"2024-04-14T15:20:33.945327+0000","last_undegraded":"2024-04-14T15:20:33.945327+0000","last_fullsized":"2024-04-14T15:20:33.945327+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"308'18","last_scrub_stamp":"2024-04-14T15:19:26.078396+0000","last_deep_scrub":"308'18","last_deep_scrub_stamp":"2024-04-14T15:16:16.054032+0000","last_clean_scrub_stamp":"2024-04-14T15:19:26.078396+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":941,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.954107+0000","last_change":"2024-04-14T15:19:16.788873+0000","last_active":"2024-04-14T15:20:33.954107+0000","last_peered":"2024-04-14T15:20:33.954107+0000","last_clean":"2024-04-14T15:20:33.954107+0000","last_became_active":"2024-04-14T15:09:43.021188+0000","last_became_peered":"2024-04-14T15:09:43.021188+0000","last_unstale":"2024-04-14T15:20:33.954107+0000","last_undegraded":"2024-04-14T15:20:33.954107+0000","last_fullsized":"2024-04-14T15:20:33.954107+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:16.788795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:19:16.788795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:35.846721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"73'5","reported_seq":1590,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.943373+0000","last_change":"2024-04-14T15:18:26.955802+0000","last_active":"2024-04-14T15:20:33.943373+0000","last_peered":"2024-04-14T15:20:33.943373+0000","last_clean":"2024-04-14T15:20:33.943373+0000","last_became_active":"2024-04-14T15:09:43.016544+0000","last_became_peered":"2024-04-14T15:09:43.016544+0000","last_unstale":"2024-04-14T15:20:33.943373+0000","last_undegraded":"2024-04-14T15:20:33.943373+0000","last_fullsized":"2024-04-14T15:20:33.943373+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-04-14T15:18:26.955711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:18:26.955711+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00056542099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":644,"num_read_kb":639,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":932,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.981062+0000","last_change":"2024-04-14T15:19:22.535232+0000","last_active":"2024-04-14T15:20:38.981062+0000","last_peered":"2024-04-14T15:20:38.981062+0000","last_clean":"2024-04-14T15:20:38.981062+0000","last_became_active":"2024-04-14T15:09:49.130307+0000","last_became_peered":"2024-04-14T15:09:49.130307+0000","last_unstale":"2024-04-14T15:20:38.981062+0000","last_undegraded":"2024-04-14T15:20:38.981062+0000","last_fullsized":"2024-04-14T15:20:38.981062+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:22.535145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:20.827589+0000","last_clean_scrub_stamp":"2024-04-14T15:19:22.535145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"91'130","reported_seq":838,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.945781+0000","last_change":"2024-04-14T15:19:06.124178+0000","last_active":"2024-04-14T15:20:33.945781+0000","last_peered":"2024-04-14T15:20:33.945781+0000","last_clean":"2024-04-14T15:20:33.945781+0000","last_became_active":"2024-04-14T15:18:40.913215+0000","last_became_peered":"2024-04-14T15:18:40.913215+0000","last_unstale":"2024-04-14T15:20:33.945781+0000","last_undegraded":"2024-04-14T15:20:33.945781+0000","last_fullsized":"2024-04-14T15:20:33.945781+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:06.124071+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:06.124071+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":536,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.957711+0000","last_change":"2024-04-14T15:19:15.372917+0000","last_active":"2024-04-14T15:20:33.957711+0000","last_peered":"2024-04-14T15:20:33.957711+0000","last_clean":"2024-04-14T15:20:33.957711+0000","last_became_active":"2024-04-14T15:15:21.883717+0000","last_became_peered":"2024-04-14T15:15:21.883717+0000","last_unstale":"2024-04-14T15:20:33.957711+0000","last_undegraded":"2024-04-14T15:20:33.957711+0000","last_fullsized":"2024-04-14T15:20:33.957711+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:15.372837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:19:15.372837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:42.207119+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":927,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.947195+0000","last_change":"2024-04-14T15:19:25.297173+0000","last_active":"2024-04-14T15:20:33.947195+0000","last_peered":"2024-04-14T15:20:33.947195+0000","last_clean":"2024-04-14T15:20:33.947195+0000","last_became_active":"2024-04-14T15:09:47.040185+0000","last_became_peered":"2024-04-14T15:09:47.040185+0000","last_unstale":"2024-04-14T15:20:33.947195+0000","last_undegraded":"2024-04-14T15:20:33.947195+0000","last_fullsized":"2024-04-14T15:20:33.947195+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:25.297044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:53.731610+0000","last_clean_scrub_stamp":"2024-04-14T15:19:25.297044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"479'40","reported_seq":1129,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.944681+0000","last_change":"2024-04-14T15:19:49.915826+0000","last_active":"2024-04-14T15:20:33.944681+0000","last_peered":"2024-04-14T15:20:33.944681+0000","last_clean":"2024-04-14T15:20:33.944681+0000","last_became_active":"2024-04-14T15:09:45.034168+0000","last_became_peered":"2024-04-14T15:09:45.034168+0000","last_unstale":"2024-04-14T15:20:33.944681+0000","last_undegraded":"2024-04-14T15:20:33.944681+0000","last_fullsized":"2024-04-14T15:20:33.944681+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'28","last_scrub_stamp":"2024-04-14T15:19:49.915672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:19:49.915672+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:54.452998+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":494,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.944537+0000","last_change":"2024-04-14T15:19:24.066365+0000","last_active":"2024-04-14T15:20:33.944537+0000","last_peered":"2024-04-14T15:20:33.944537+0000","last_clean":"2024-04-14T15:20:33.944537+0000","last_became_active":"2024-04-14T15:15:44.885315+0000","last_became_peered":"2024-04-14T15:15:44.885315+0000","last_unstale":"2024-04-14T15:20:33.944537+0000","last_undegraded":"2024-04-14T15:20:33.944537+0000","last_fullsized":"2024-04-14T15:20:33.944537+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:24.066275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:24.066275+0000","last_clean_scrub_stamp":"2024-04-14T15:19:24.066275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:43.166123+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":588,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.961996+0000","last_change":"2024-04-14T15:17:44.084142+0000","last_active":"2024-04-14T15:20:33.961996+0000","last_peered":"2024-04-14T15:20:33.961996+0000","last_clean":"2024-04-14T15:20:33.961996+0000","last_became_active":"2024-04-14T15:15:21.874742+0000","last_became_peered":"2024-04-14T15:15:21.874742+0000","last_unstale":"2024-04-14T15:20:33.961996+0000","last_undegraded":"2024-04-14T15:20:33.961996+0000","last_fullsized":"2024-04-14T15:20:33.961996+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:44.084010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:44.084010+0000","last_clean_scrub_stamp":"2024-04-14T15:17:44.084010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"91'130","reported_seq":1028,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-04-14T15:20:37.984242+0000","last_change":"2024-04-14T15:19:07.770220+0000","last_active":"2024-04-14T15:20:37.984242+0000","last_peered":"2024-04-14T15:20:37.984242+0000","last_clean":"2024-04-14T15:20:37.984242+0000","last_became_active":"2024-04-14T15:18:58.189402+0000","last_became_peered":"2024-04-14T15:18:58.189402+0000","last_unstale":"2024-04-14T15:20:37.984242+0000","last_undegraded":"2024-04-14T15:20:37.984242+0000","last_fullsized":"2024-04-14T15:20:37.984242+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:07.770125+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:07.770125+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":837,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.955758+0000","last_change":"2024-04-14T15:19:55.858944+0000","last_active":"2024-04-14T15:20:33.955758+0000","last_peered":"2024-04-14T15:20:33.955758+0000","last_clean":"2024-04-14T15:20:33.955758+0000","last_became_active":"2024-04-14T15:12:56.350703+0000","last_became_peered":"2024-04-14T15:12:56.350703+0000","last_unstale":"2024-04-14T15:20:33.955758+0000","last_undegraded":"2024-04-14T15:20:33.955758+0000","last_fullsized":"2024-04-14T15:20:33.955758+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:55.858835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:19:55.858835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:09.317978+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"72'8","reported_seq":1074,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.946943+0000","last_change":"2024-04-14T15:20:02.298437+0000","last_active":"2024-04-14T15:20:33.946943+0000","last_peered":"2024-04-14T15:20:33.946943+0000","last_clean":"2024-04-14T15:20:33.946943+0000","last_became_active":"2024-04-14T15:09:47.047699+0000","last_became_peered":"2024-04-14T15:09:47.047699+0000","last_unstale":"2024-04-14T15:20:33.946943+0000","last_undegraded":"2024-04-14T15:20:33.946943+0000","last_fullsized":"2024-04-14T15:20:33.946943+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:20:02.298301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:20:02.298301+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:16.534076+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"479'20","reported_seq":938,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.954610+0000","last_change":"2024-04-14T15:18:38.870798+0000","last_active":"2024-04-14T15:20:33.954610+0000","last_peered":"2024-04-14T15:20:33.954610+0000","last_clean":"2024-04-14T15:20:33.954610+0000","last_became_active":"2024-04-14T15:10:07.764042+0000","last_became_peered":"2024-04-14T15:10:07.764042+0000","last_unstale":"2024-04-14T15:20:33.954610+0000","last_undegraded":"2024-04-14T15:20:33.954610+0000","last_fullsized":"2024-04-14T15:20:33.954610+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-04-14T15:18:38.870710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:18:38.870710+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":928,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.944468+0000","last_change":"2024-04-14T15:17:40.971492+0000","last_active":"2024-04-14T15:20:33.944468+0000","last_peered":"2024-04-14T15:20:33.944468+0000","last_clean":"2024-04-14T15:20:33.944468+0000","last_became_active":"2024-04-14T15:09:43.022597+0000","last_became_peered":"2024-04-14T15:09:43.022597+0000","last_unstale":"2024-04-14T15:20:33.944468+0000","last_undegraded":"2024-04-14T15:20:33.944468+0000","last_fullsized":"2024-04-14T15:20:33.944468+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:40.971412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:40.971412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":923,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.945814+0000","last_change":"2024-04-14T15:19:16.390131+0000","last_active":"2024-04-14T15:20:33.945814+0000","last_peered":"2024-04-14T15:20:33.945814+0000","last_clean":"2024-04-14T15:20:33.945814+0000","last_became_active":"2024-04-14T15:09:49.066648+0000","last_became_peered":"2024-04-14T15:09:49.066648+0000","last_unstale":"2024-04-14T15:20:33.945814+0000","last_undegraded":"2024-04-14T15:20:33.945814+0000","last_fullsized":"2024-04-14T15:20:33.945814+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:16.390049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:19:16.390049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:36.162522+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.c","version":"91'130","reported_seq":1023,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.958399+0000","last_change":"2024-04-14T15:19:04.397426+0000","last_active":"2024-04-14T15:20:33.958399+0000","last_peered":"2024-04-14T15:20:33.958399+0000","last_clean":"2024-04-14T15:20:33.958399+0000","last_became_active":"2024-04-14T15:18:58.186036+0000","last_became_peered":"2024-04-14T15:18:58.186036+0000","last_unstale":"2024-04-14T15:20:33.958399+0000","last_undegraded":"2024-04-14T15:20:33.958399+0000","last_fullsized":"2024-04-14T15:20:33.958399+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:04.397343+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:04.397343+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":831,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.947486+0000","last_change":"2024-04-14T15:18:27.133391+0000","last_active":"2024-04-14T15:20:33.947486+0000","last_peered":"2024-04-14T15:20:33.947486+0000","last_clean":"2024-04-14T15:20:33.947486+0000","last_became_active":"2024-04-14T15:12:56.563074+0000","last_became_peered":"2024-04-14T15:12:56.563074+0000","last_unstale":"2024-04-14T15:20:33.947486+0000","last_undegraded":"2024-04-14T15:20:33.947486+0000","last_fullsized":"2024-04-14T15:20:33.947486+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:27.133310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:18:27.133310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":924,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.953752+0000","last_change":"2024-04-14T15:17:56.915660+0000","last_active":"2024-04-14T15:20:33.953752+0000","last_peered":"2024-04-14T15:20:33.953752+0000","last_clean":"2024-04-14T15:20:33.953752+0000","last_became_active":"2024-04-14T15:09:47.049005+0000","last_became_peered":"2024-04-14T15:09:47.049005+0000","last_unstale":"2024-04-14T15:20:33.953752+0000","last_undegraded":"2024-04-14T15:20:33.953752+0000","last_fullsized":"2024-04-14T15:20:33.953752+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:56.915544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:56.915544+0000","last_clean_scrub_stamp":"2024-04-14T15:17:56.915544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"479'23","reported_seq":985,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.982061+0000","last_change":"2024-04-14T15:19:44.525830+0000","last_active":"2024-04-14T15:20:38.982061+0000","last_peered":"2024-04-14T15:20:38.982061+0000","last_clean":"2024-04-14T15:20:38.982061+0000","last_became_active":"2024-04-14T15:09:45.032892+0000","last_became_peered":"2024-04-14T15:09:45.032892+0000","last_unstale":"2024-04-14T15:20:38.982061+0000","last_undegraded":"2024-04-14T15:20:38.982061+0000","last_fullsized":"2024-04-14T15:20:38.982061+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'17","last_scrub_stamp":"2024-04-14T15:19:44.525712+0000","last_deep_scrub":"183'17","last_deep_scrub_stamp":"2024-04-14T15:14:11.577574+0000","last_clean_scrub_stamp":"2024-04-14T15:19:44.525712+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:47.139793+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"65'3","reported_seq":961,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.983258+0000","last_change":"2024-04-14T15:19:22.603785+0000","last_active":"2024-04-14T15:20:38.983258+0000","last_peered":"2024-04-14T15:20:38.983258+0000","last_clean":"2024-04-14T15:20:38.983258+0000","last_became_active":"2024-04-14T15:09:43.020490+0000","last_became_peered":"2024-04-14T15:09:43.020490+0000","last_unstale":"2024-04-14T15:20:38.983258+0000","last_undegraded":"2024-04-14T15:20:38.983258+0000","last_fullsized":"2024-04-14T15:20:38.983258+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3","last_scrub_stamp":"2024-04-14T15:19:22.603651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:19:22.603651+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":528,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":588,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.957966+0000","last_change":"2024-04-14T15:16:58.942591+0000","last_active":"2024-04-14T15:20:33.957966+0000","last_peered":"2024-04-14T15:20:33.957966+0000","last_clean":"2024-04-14T15:20:33.957966+0000","last_became_active":"2024-04-14T15:15:21.876410+0000","last_became_peered":"2024-04-14T15:15:21.876410+0000","last_unstale":"2024-04-14T15:20:33.957966+0000","last_undegraded":"2024-04-14T15:20:33.957966+0000","last_fullsized":"2024-04-14T15:20:33.957966+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:58.942499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:16:58.942499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"91'130","reported_seq":1015,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.943042+0000","last_change":"2024-04-14T15:19:07.091520+0000","last_active":"2024-04-14T15:20:33.943042+0000","last_peered":"2024-04-14T15:20:33.943042+0000","last_clean":"2024-04-14T15:20:33.943042+0000","last_became_active":"2024-04-14T15:18:45.912049+0000","last_became_peered":"2024-04-14T15:18:45.912049+0000","last_unstale":"2024-04-14T15:20:33.943042+0000","last_undegraded":"2024-04-14T15:20:33.943042+0000","last_fullsized":"2024-04-14T15:20:33.943042+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:07.091439+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:07.091439+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":839,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.946011+0000","last_change":"2024-04-14T15:19:24.289915+0000","last_active":"2024-04-14T15:20:33.946011+0000","last_peered":"2024-04-14T15:20:33.946011+0000","last_clean":"2024-04-14T15:20:33.946011+0000","last_became_active":"2024-04-14T15:12:56.551177+0000","last_became_peered":"2024-04-14T15:12:56.551177+0000","last_unstale":"2024-04-14T15:20:33.946011+0000","last_undegraded":"2024-04-14T15:20:33.946011+0000","last_fullsized":"2024-04-14T15:20:33.946011+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:24.289806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:19:24.289806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:44.967529+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"479'19","reported_seq":959,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.947648+0000","last_change":"2024-04-14T15:19:51.290310+0000","last_active":"2024-04-14T15:20:33.947648+0000","last_peered":"2024-04-14T15:20:33.947648+0000","last_clean":"2024-04-14T15:20:33.947648+0000","last_became_active":"2024-04-14T15:09:45.180375+0000","last_became_peered":"2024-04-14T15:09:45.180375+0000","last_unstale":"2024-04-14T15:20:33.947648+0000","last_undegraded":"2024-04-14T15:20:33.947648+0000","last_fullsized":"2024-04-14T15:20:33.947648+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'19","last_scrub_stamp":"2024-04-14T15:19:51.290211+0000","last_deep_scrub":"347'13","last_deep_scrub_stamp":"2024-04-14T15:16:55.694599+0000","last_clean_scrub_stamp":"2024-04-14T15:19:51.290211+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:18.067434+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":940,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.953720+0000","last_change":"2024-04-14T15:19:21.774315+0000","last_active":"2024-04-14T15:20:33.953720+0000","last_peered":"2024-04-14T15:20:33.953720+0000","last_clean":"2024-04-14T15:20:33.953720+0000","last_became_active":"2024-04-14T15:09:43.016788+0000","last_became_peered":"2024-04-14T15:09:43.016788+0000","last_unstale":"2024-04-14T15:20:33.953720+0000","last_undegraded":"2024-04-14T15:20:33.953720+0000","last_fullsized":"2024-04-14T15:20:33.953720+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:21.774184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:19:21.774184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:48.086427+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":608,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.960012+0000","last_change":"2024-04-14T15:18:40.385814+0000","last_active":"2024-04-14T15:20:33.960012+0000","last_peered":"2024-04-14T15:20:33.960012+0000","last_clean":"2024-04-14T15:20:33.960012+0000","last_became_active":"2024-04-14T15:15:21.873580+0000","last_became_peered":"2024-04-14T15:15:21.873580+0000","last_unstale":"2024-04-14T15:20:33.960012+0000","last_undegraded":"2024-04-14T15:20:33.960012+0000","last_fullsized":"2024-04-14T15:20:33.960012+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:40.385736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:18:40.385736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":928,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.955315+0000","last_change":"2024-04-14T15:19:24.821597+0000","last_active":"2024-04-14T15:20:33.955315+0000","last_peered":"2024-04-14T15:20:33.955315+0000","last_clean":"2024-04-14T15:20:33.955315+0000","last_became_active":"2024-04-14T15:09:49.064296+0000","last_became_peered":"2024-04-14T15:09:49.064296+0000","last_unstale":"2024-04-14T15:20:33.955315+0000","last_undegraded":"2024-04-14T15:20:33.955315+0000","last_fullsized":"2024-04-14T15:20:33.955315+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:24.821514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:19:24.821514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:39.516148+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.12","version":"91'130","reported_seq":487,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.944271+0000","last_change":"2024-04-14T15:19:06.098758+0000","last_active":"2024-04-14T15:20:33.944271+0000","last_peered":"2024-04-14T15:20:33.944271+0000","last_clean":"2024-04-14T15:20:33.944271+0000","last_became_active":"2024-04-14T15:18:45.918413+0000","last_became_peered":"2024-04-14T15:18:45.918413+0000","last_unstale":"2024-04-14T15:20:33.944271+0000","last_undegraded":"2024-04-14T15:20:33.944271+0000","last_fullsized":"2024-04-14T15:20:33.944271+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:06.098672+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:06.098672+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":829,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.953655+0000","last_change":"2024-04-14T15:17:36.728543+0000","last_active":"2024-04-14T15:20:33.953655+0000","last_peered":"2024-04-14T15:20:33.953655+0000","last_clean":"2024-04-14T15:20:33.953655+0000","last_became_active":"2024-04-14T15:12:56.340066+0000","last_became_peered":"2024-04-14T15:12:56.340066+0000","last_unstale":"2024-04-14T15:20:33.953655+0000","last_undegraded":"2024-04-14T15:20:33.953655+0000","last_fullsized":"2024-04-14T15:20:33.953655+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:36.728465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:17:36.728465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.17","version":"479'12","reported_seq":945,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.959877+0000","last_change":"2024-04-14T15:17:42.019838+0000","last_active":"2024-04-14T15:20:33.959877+0000","last_peered":"2024-04-14T15:20:33.959877+0000","last_clean":"2024-04-14T15:20:33.959877+0000","last_became_active":"2024-04-14T15:09:45.180037+0000","last_became_peered":"2024-04-14T15:09:45.180037+0000","last_unstale":"2024-04-14T15:20:33.959877+0000","last_undegraded":"2024-04-14T15:20:33.959877+0000","last_fullsized":"2024-04-14T15:20:33.959877+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'6","last_scrub_stamp":"2024-04-14T15:17:42.019717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:17:42.019717+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":950,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-04-14T15:20:37.981532+0000","last_change":"2024-04-14T15:19:23.915584+0000","last_active":"2024-04-14T15:20:37.981532+0000","last_peered":"2024-04-14T15:20:37.981532+0000","last_clean":"2024-04-14T15:20:37.981532+0000","last_became_active":"2024-04-14T15:09:43.018987+0000","last_became_peered":"2024-04-14T15:09:43.018987+0000","last_unstale":"2024-04-14T15:20:37.981532+0000","last_undegraded":"2024-04-14T15:20:37.981532+0000","last_fullsized":"2024-04-14T15:20:37.981532+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:23.915503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:59.640565+0000","last_clean_scrub_stamp":"2024-04-14T15:19:23.915503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":482,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.961962+0000","last_change":"2024-04-14T15:17:47.072247+0000","last_active":"2024-04-14T15:20:33.961962+0000","last_peered":"2024-04-14T15:20:33.961962+0000","last_clean":"2024-04-14T15:20:33.961962+0000","last_became_active":"2024-04-14T15:15:44.889786+0000","last_became_peered":"2024-04-14T15:15:44.889786+0000","last_unstale":"2024-04-14T15:20:33.961962+0000","last_undegraded":"2024-04-14T15:20:33.961962+0000","last_fullsized":"2024-04-14T15:20:33.961962+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:47.072170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:13.735024+0000","last_clean_scrub_stamp":"2024-04-14T15:17:47.072170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":931,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.981272+0000","last_change":"2024-04-14T15:19:47.597543+0000","last_active":"2024-04-14T15:20:38.981272+0000","last_peered":"2024-04-14T15:20:38.981272+0000","last_clean":"2024-04-14T15:20:38.981272+0000","last_became_active":"2024-04-14T15:09:49.130438+0000","last_became_peered":"2024-04-14T15:09:49.130438+0000","last_unstale":"2024-04-14T15:20:38.981272+0000","last_undegraded":"2024-04-14T15:20:38.981272+0000","last_fullsized":"2024-04-14T15:20:38.981272+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:47.597268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:19:47.597268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:59.841711+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.11","version":"91'130","reported_seq":845,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.945670+0000","last_change":"2024-04-14T15:19:05.136092+0000","last_active":"2024-04-14T15:20:33.945670+0000","last_peered":"2024-04-14T15:20:33.945670+0000","last_clean":"2024-04-14T15:20:33.945670+0000","last_became_active":"2024-04-14T15:18:45.915385+0000","last_became_peered":"2024-04-14T15:18:45.915385+0000","last_unstale":"2024-04-14T15:20:33.945670+0000","last_undegraded":"2024-04-14T15:20:33.945670+0000","last_fullsized":"2024-04-14T15:20:33.945670+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:05.136001+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:05.136001+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":840,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.982299+0000","last_change":"2024-04-14T15:17:46.369320+0000","last_active":"2024-04-14T15:20:38.982299+0000","last_peered":"2024-04-14T15:20:38.982299+0000","last_clean":"2024-04-14T15:20:38.982299+0000","last_became_active":"2024-04-14T15:12:56.203577+0000","last_became_peered":"2024-04-14T15:12:56.203577+0000","last_unstale":"2024-04-14T15:20:38.982299+0000","last_undegraded":"2024-04-14T15:20:38.982299+0000","last_fullsized":"2024-04-14T15:20:38.982299+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:46.369231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:46.369231+0000","last_clean_scrub_stamp":"2024-04-14T15:17:46.369231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"479'20","reported_seq":642,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.959236+0000","last_change":"2024-04-14T15:18:36.534967+0000","last_active":"2024-04-14T15:20:33.959236+0000","last_peered":"2024-04-14T15:20:33.959236+0000","last_clean":"2024-04-14T15:20:33.959236+0000","last_became_active":"2024-04-14T15:15:21.877667+0000","last_became_peered":"2024-04-14T15:15:21.877667+0000","last_unstale":"2024-04-14T15:20:33.959236+0000","last_undegraded":"2024-04-14T15:20:33.959236+0000","last_fullsized":"2024-04-14T15:20:33.959236+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-04-14T15:18:36.534874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:18:36.534874+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":490,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.957576+0000","last_change":"2024-04-14T15:19:56.775138+0000","last_active":"2024-04-14T15:20:33.957576+0000","last_peered":"2024-04-14T15:20:33.957576+0000","last_clean":"2024-04-14T15:20:33.957576+0000","last_became_active":"2024-04-14T15:15:44.887452+0000","last_became_peered":"2024-04-14T15:15:44.887452+0000","last_unstale":"2024-04-14T15:20:33.957576+0000","last_undegraded":"2024-04-14T15:20:33.957576+0000","last_fullsized":"2024-04-14T15:20:33.957576+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:56.775036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:19:56.775036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:23.210279+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"280'9","reported_seq":569,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.945899+0000","last_change":"2024-04-14T15:17:50.908130+0000","last_active":"2024-04-14T15:20:33.945899+0000","last_peered":"2024-04-14T15:20:33.945899+0000","last_clean":"2024-04-14T15:20:33.945899+0000","last_became_active":"2024-04-14T15:15:44.882916+0000","last_became_peered":"2024-04-14T15:15:44.882916+0000","last_unstale":"2024-04-14T15:20:33.945899+0000","last_undegraded":"2024-04-14T15:20:33.945899+0000","last_fullsized":"2024-04-14T15:20:33.945899+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"280'9","last_scrub_stamp":"2024-04-14T15:17:50.908054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:17:50.908054+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":922,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.947558+0000","last_change":"2024-04-14T15:20:30.356561+0000","last_active":"2024-04-14T15:20:33.947558+0000","last_peered":"2024-04-14T15:20:33.947558+0000","last_clean":"2024-04-14T15:20:33.947558+0000","last_became_active":"2024-04-14T15:09:49.066068+0000","last_became_peered":"2024-04-14T15:09:49.066068+0000","last_unstale":"2024-04-14T15:20:33.947558+0000","last_undegraded":"2024-04-14T15:20:33.947558+0000","last_fullsized":"2024-04-14T15:20:33.947558+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:30.356481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:54.724681+0000","last_clean_scrub_stamp":"2024-04-14T15:20:30.356481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:51.057447+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"91'130","reported_seq":1236,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.979261+0000","last_change":"2024-04-14T15:19:05.496449+0000","last_active":"2024-04-14T15:20:38.979261+0000","last_peered":"2024-04-14T15:20:38.979261+0000","last_clean":"2024-04-14T15:20:38.979261+0000","last_became_active":"2024-04-14T15:18:45.917220+0000","last_became_peered":"2024-04-14T15:18:45.917220+0000","last_unstale":"2024-04-14T15:20:38.979261+0000","last_undegraded":"2024-04-14T15:20:38.979261+0000","last_fullsized":"2024-04-14T15:20:38.979261+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:05.496303+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:05.496303+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":481,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.957125+0000","last_change":"2024-04-14T15:18:30.438135+0000","last_active":"2024-04-14T15:20:33.957125+0000","last_peered":"2024-04-14T15:20:33.957125+0000","last_clean":"2024-04-14T15:20:33.957125+0000","last_became_active":"2024-04-14T15:15:44.889106+0000","last_became_peered":"2024-04-14T15:15:44.889106+0000","last_unstale":"2024-04-14T15:20:33.957125+0000","last_undegraded":"2024-04-14T15:20:33.957125+0000","last_fullsized":"2024-04-14T15:20:33.957125+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:30.438001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:18:30.438001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"479'19","reported_seq":790,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.981131+0000","last_change":"2024-04-14T15:20:07.673953+0000","last_active":"2024-04-14T15:20:38.981131+0000","last_peered":"2024-04-14T15:20:38.981131+0000","last_clean":"2024-04-14T15:20:38.981131+0000","last_became_active":"2024-04-14T15:15:44.884211+0000","last_became_peered":"2024-04-14T15:15:44.884211+0000","last_unstale":"2024-04-14T15:20:38.981131+0000","last_undegraded":"2024-04-14T15:20:38.981131+0000","last_fullsized":"2024-04-14T15:20:38.981131+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'19","last_scrub_stamp":"2024-04-14T15:20:07.673794+0000","last_deep_scrub":"183'13","last_deep_scrub_stamp":"2024-04-14T15:14:11.755252+0000","last_clean_scrub_stamp":"2024-04-14T15:20:07.673794+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:08.415336+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":942,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.947141+0000","last_change":"2024-04-14T15:20:32.388821+0000","last_active":"2024-04-14T15:20:33.947141+0000","last_peered":"2024-04-14T15:20:33.947141+0000","last_clean":"2024-04-14T15:20:33.947141+0000","last_became_active":"2024-04-14T15:09:43.019301+0000","last_became_peered":"2024-04-14T15:09:43.019301+0000","last_unstale":"2024-04-14T15:20:33.947141+0000","last_undegraded":"2024-04-14T15:20:33.947141+0000","last_fullsized":"2024-04-14T15:20:33.947141+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:32.388728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:20:32.388728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:50.382306+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"255'10","reported_seq":710,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.959170+0000","last_change":"2024-04-14T15:19:59.811456+0000","last_active":"2024-04-14T15:20:33.959170+0000","last_peered":"2024-04-14T15:20:33.959170+0000","last_clean":"2024-04-14T15:20:33.959170+0000","last_became_active":"2024-04-14T15:15:21.884468+0000","last_became_peered":"2024-04-14T15:15:21.884468+0000","last_unstale":"2024-04-14T15:20:33.959170+0000","last_undegraded":"2024-04-14T15:20:33.959170+0000","last_fullsized":"2024-04-14T15:20:33.959170+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'10","last_scrub_stamp":"2024-04-14T15:19:59.811370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:19:59.811370+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:08.120065+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":931,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.983364+0000","last_change":"2024-04-14T15:19:15.642110+0000","last_active":"2024-04-14T15:20:38.983364+0000","last_peered":"2024-04-14T15:20:38.983364+0000","last_clean":"2024-04-14T15:20:38.983364+0000","last_became_active":"2024-04-14T15:09:49.065242+0000","last_became_peered":"2024-04-14T15:09:49.065242+0000","last_unstale":"2024-04-14T15:20:38.983364+0000","last_undegraded":"2024-04-14T15:20:38.983364+0000","last_fullsized":"2024-04-14T15:20:38.983364+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:15.642017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:19:15.642017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.17","version":"91'130","reported_seq":924,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-04-14T15:20:37.984062+0000","last_change":"2024-04-14T15:20:32.779074+0000","last_active":"2024-04-14T15:20:37.984062+0000","last_peered":"2024-04-14T15:20:37.984062+0000","last_clean":"2024-04-14T15:20:37.984062+0000","last_became_active":"2024-04-14T15:20:31.940473+0000","last_became_peered":"2024-04-14T15:20:31.940473+0000","last_unstale":"2024-04-14T15:20:37.984062+0000","last_undegraded":"2024-04-14T15:20:37.984062+0000","last_fullsized":"2024-04-14T15:20:37.984062+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":446,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:20:32.778990+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:20:32.778990+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:46.128531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":840,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.981277+0000","last_change":"2024-04-14T15:18:40.735708+0000","last_active":"2024-04-14T15:20:38.981277+0000","last_peered":"2024-04-14T15:20:38.981277+0000","last_clean":"2024-04-14T15:20:38.981277+0000","last_became_active":"2024-04-14T15:12:56.389204+0000","last_became_peered":"2024-04-14T15:12:56.389204+0000","last_unstale":"2024-04-14T15:20:38.981277+0000","last_undegraded":"2024-04-14T15:20:38.981277+0000","last_fullsized":"2024-04-14T15:20:38.981277+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:40.735623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:18:40.735623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"479'19","reported_seq":478,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-04-14T15:20:37.982088+0000","last_change":"2024-04-14T15:19:10.762280+0000","last_active":"2024-04-14T15:20:37.982088+0000","last_peered":"2024-04-14T15:20:37.982088+0000","last_clean":"2024-04-14T15:20:37.982088+0000","last_became_active":"2024-04-14T15:16:24.993130+0000","last_became_peered":"2024-04-14T15:16:24.993130+0000","last_unstale":"2024-04-14T15:20:37.982088+0000","last_undegraded":"2024-04-14T15:20:37.982088+0000","last_fullsized":"2024-04-14T15:20:37.982088+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"344'13","last_scrub_stamp":"2024-04-14T15:19:10.762147+0000","last_deep_scrub":"344'13","last_deep_scrub_stamp":"2024-04-14T15:16:52.738163+0000","last_clean_scrub_stamp":"2024-04-14T15:19:10.762147+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":950,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-04-14T15:20:37.982656+0000","last_change":"2024-04-14T15:19:24.898168+0000","last_active":"2024-04-14T15:20:37.982656+0000","last_peered":"2024-04-14T15:20:37.982656+0000","last_clean":"2024-04-14T15:20:37.982656+0000","last_became_active":"2024-04-14T15:09:43.022480+0000","last_became_peered":"2024-04-14T15:09:43.022480+0000","last_unstale":"2024-04-14T15:20:37.982656+0000","last_undegraded":"2024-04-14T15:20:37.982656+0000","last_fullsized":"2024-04-14T15:20:37.982656+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:24.898086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:19:24.898086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":600,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.957826+0000","last_change":"2024-04-14T15:18:33.485536+0000","last_active":"2024-04-14T15:20:33.957826+0000","last_peered":"2024-04-14T15:20:33.957826+0000","last_clean":"2024-04-14T15:20:33.957826+0000","last_became_active":"2024-04-14T15:15:21.882643+0000","last_became_peered":"2024-04-14T15:15:21.882643+0000","last_unstale":"2024-04-14T15:20:33.957826+0000","last_undegraded":"2024-04-14T15:20:33.957826+0000","last_fullsized":"2024-04-14T15:20:33.957826+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:33.485411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:33.485411+0000","last_clean_scrub_stamp":"2024-04-14T15:18:33.485411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":925,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.945716+0000","last_change":"2024-04-14T15:18:24.039316+0000","last_active":"2024-04-14T15:20:33.945716+0000","last_peered":"2024-04-14T15:20:33.945716+0000","last_clean":"2024-04-14T15:20:33.945716+0000","last_became_active":"2024-04-14T15:09:49.059711+0000","last_became_peered":"2024-04-14T15:09:49.059711+0000","last_unstale":"2024-04-14T15:20:33.945716+0000","last_undegraded":"2024-04-14T15:20:33.945716+0000","last_fullsized":"2024-04-14T15:20:33.945716+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:24.039237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:18:24.039237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"91'130","reported_seq":361,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-04-14T15:20:37.983320+0000","last_change":"2024-04-14T15:20:33.788929+0000","last_active":"2024-04-14T15:20:37.983320+0000","last_peered":"2024-04-14T15:20:37.983320+0000","last_clean":"2024-04-14T15:20:37.983320+0000","last_became_active":"2024-04-14T15:20:31.942542+0000","last_became_peered":"2024-04-14T15:20:31.942542+0000","last_unstale":"2024-04-14T15:20:37.983320+0000","last_undegraded":"2024-04-14T15:20:37.983320+0000","last_fullsized":"2024-04-14T15:20:37.983320+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":446,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:20:33.788814+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:20:33.788814+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:39.708512+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":847,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-04-14T15:20:37.982509+0000","last_change":"2024-04-14T15:19:26.891939+0000","last_active":"2024-04-14T15:20:37.982509+0000","last_peered":"2024-04-14T15:20:37.982509+0000","last_clean":"2024-04-14T15:20:37.982509+0000","last_became_active":"2024-04-14T15:12:56.396294+0000","last_became_peered":"2024-04-14T15:12:56.396294+0000","last_unstale":"2024-04-14T15:20:37.982509+0000","last_undegraded":"2024-04-14T15:20:37.982509+0000","last_fullsized":"2024-04-14T15:20:37.982509+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:26.891862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:19:26.891862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:48.097098+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"479'24","reported_seq":975,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.982404+0000","last_change":"2024-04-14T15:19:17.590124+0000","last_active":"2024-04-14T15:20:38.982404+0000","last_peered":"2024-04-14T15:20:38.982404+0000","last_clean":"2024-04-14T15:20:38.982404+0000","last_became_active":"2024-04-14T15:09:45.026065+0000","last_became_peered":"2024-04-14T15:09:45.026065+0000","last_unstale":"2024-04-14T15:20:38.982404+0000","last_undegraded":"2024-04-14T15:20:38.982404+0000","last_fullsized":"2024-04-14T15:20:38.982404+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"464'16","last_scrub_stamp":"2024-04-14T15:19:17.589999+0000","last_deep_scrub":"464'16","last_deep_scrub_stamp":"2024-04-14T15:19:17.589999+0000","last_clean_scrub_stamp":"2024-04-14T15:19:17.589999+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":935,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.945950+0000","last_change":"2024-04-14T15:17:34.986735+0000","last_active":"2024-04-14T15:20:33.945950+0000","last_peered":"2024-04-14T15:20:33.945950+0000","last_clean":"2024-04-14T15:20:33.945950+0000","last_became_active":"2024-04-14T15:09:43.023583+0000","last_became_peered":"2024-04-14T15:09:43.023583+0000","last_unstale":"2024-04-14T15:20:33.945950+0000","last_undegraded":"2024-04-14T15:20:33.945950+0000","last_fullsized":"2024-04-14T15:20:33.945950+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:34.986655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:34.986655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":927,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.944727+0000","last_change":"2024-04-14T15:20:01.971662+0000","last_active":"2024-04-14T15:20:33.944727+0000","last_peered":"2024-04-14T15:20:33.944727+0000","last_clean":"2024-04-14T15:20:33.944727+0000","last_became_active":"2024-04-14T15:09:47.041683+0000","last_became_peered":"2024-04-14T15:09:47.041683+0000","last_unstale":"2024-04-14T15:20:33.944727+0000","last_undegraded":"2024-04-14T15:20:33.944727+0000","last_fullsized":"2024-04-14T15:20:33.944727+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:01.971564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:08.486217+0000","last_clean_scrub_stamp":"2024-04-14T15:20:01.971564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:18.567528+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":604,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.956922+0000","last_change":"2024-04-14T15:19:57.757627+0000","last_active":"2024-04-14T15:20:33.956922+0000","last_peered":"2024-04-14T15:20:33.956922+0000","last_clean":"2024-04-14T15:20:33.956922+0000","last_became_active":"2024-04-14T15:15:21.889567+0000","last_became_peered":"2024-04-14T15:15:21.889567+0000","last_unstale":"2024-04-14T15:20:33.956922+0000","last_undegraded":"2024-04-14T15:20:33.956922+0000","last_fullsized":"2024-04-14T15:20:33.956922+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:57.757547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:19:57.757547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:15.034106+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.15","version":"91'130","reported_seq":359,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.983098+0000","last_change":"2024-04-14T15:20:32.705659+0000","last_active":"2024-04-14T15:20:38.983098+0000","last_peered":"2024-04-14T15:20:38.983098+0000","last_clean":"2024-04-14T15:20:38.983098+0000","last_became_active":"2024-04-14T15:20:31.941220+0000","last_became_peered":"2024-04-14T15:20:31.941220+0000","last_unstale":"2024-04-14T15:20:38.983098+0000","last_undegraded":"2024-04-14T15:20:38.983098+0000","last_fullsized":"2024-04-14T15:20:38.983098+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":446,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:20:32.705583+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:20:32.705583+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:34.732006+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":837,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.956049+0000","last_change":"2024-04-14T15:19:53.922021+0000","last_active":"2024-04-14T15:20:33.956049+0000","last_peered":"2024-04-14T15:20:33.956049+0000","last_clean":"2024-04-14T15:20:33.956049+0000","last_became_active":"2024-04-14T15:12:56.387723+0000","last_became_peered":"2024-04-14T15:12:56.387723+0000","last_unstale":"2024-04-14T15:20:33.956049+0000","last_undegraded":"2024-04-14T15:20:33.956049+0000","last_fullsized":"2024-04-14T15:20:33.956049+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:53.921934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:03.685045+0000","last_clean_scrub_stamp":"2024-04-14T15:19:53.921934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:08.254525+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.10","version":"479'8","reported_seq":939,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.956871+0000","last_change":"2024-04-14T15:17:53.181132+0000","last_active":"2024-04-14T15:20:33.956871+0000","last_peered":"2024-04-14T15:20:33.956871+0000","last_clean":"2024-04-14T15:20:33.956871+0000","last_became_active":"2024-04-14T15:09:45.038815+0000","last_became_peered":"2024-04-14T15:09:45.038815+0000","last_unstale":"2024-04-14T15:20:33.956871+0000","last_undegraded":"2024-04-14T15:20:33.956871+0000","last_fullsized":"2024-04-14T15:20:33.956871+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4","last_scrub_stamp":"2024-04-14T15:17:53.181053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:17:53.181053+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":493,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-04-14T15:20:37.983762+0000","last_change":"2024-04-14T15:17:48.845163+0000","last_active":"2024-04-14T15:20:37.983762+0000","last_peered":"2024-04-14T15:20:37.983762+0000","last_clean":"2024-04-14T15:20:37.983762+0000","last_became_active":"2024-04-14T15:15:44.884730+0000","last_became_peered":"2024-04-14T15:15:44.884730+0000","last_unstale":"2024-04-14T15:20:37.983762+0000","last_undegraded":"2024-04-14T15:20:37.983762+0000","last_fullsized":"2024-04-14T15:20:37.983762+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:48.845081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:08.712289+0000","last_clean_scrub_stamp":"2024-04-14T15:17:48.845081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":950,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-04-14T15:20:37.981537+0000","last_change":"2024-04-14T15:19:48.867805+0000","last_active":"2024-04-14T15:20:37.981537+0000","last_peered":"2024-04-14T15:20:37.981537+0000","last_clean":"2024-04-14T15:20:37.981537+0000","last_became_active":"2024-04-14T15:09:47.041074+0000","last_became_peered":"2024-04-14T15:09:47.041074+0000","last_unstale":"2024-04-14T15:20:37.981537+0000","last_undegraded":"2024-04-14T15:20:37.981537+0000","last_fullsized":"2024-04-14T15:20:37.981537+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:48.867704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:19:48.867704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:59.267600+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":931,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.980067+0000","last_change":"2024-04-14T15:19:59.634301+0000","last_active":"2024-04-14T15:20:38.980067+0000","last_peered":"2024-04-14T15:20:38.980067+0000","last_clean":"2024-04-14T15:20:38.980067+0000","last_became_active":"2024-04-14T15:09:49.061359+0000","last_became_peered":"2024-04-14T15:09:49.061359+0000","last_unstale":"2024-04-14T15:20:38.980067+0000","last_undegraded":"2024-04-14T15:20:38.980067+0000","last_fullsized":"2024-04-14T15:20:38.980067+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:59.634212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:19:59.634212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:22.995269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"91'130","reported_seq":1006,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.954807+0000","last_change":"2024-04-14T15:19:06.939565+0000","last_active":"2024-04-14T15:20:33.954807+0000","last_peered":"2024-04-14T15:20:33.954807+0000","last_clean":"2024-04-14T15:20:33.954807+0000","last_became_active":"2024-04-14T15:18:45.913983+0000","last_became_peered":"2024-04-14T15:18:45.913983+0000","last_unstale":"2024-04-14T15:20:33.954807+0000","last_undegraded":"2024-04-14T15:20:33.954807+0000","last_fullsized":"2024-04-14T15:20:33.954807+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:06.939444+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:06.939444+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":840,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.981290+0000","last_change":"2024-04-14T15:18:35.596732+0000","last_active":"2024-04-14T15:20:38.981290+0000","last_peered":"2024-04-14T15:20:38.981290+0000","last_clean":"2024-04-14T15:20:38.981290+0000","last_became_active":"2024-04-14T15:12:56.352528+0000","last_became_peered":"2024-04-14T15:12:56.352528+0000","last_unstale":"2024-04-14T15:20:38.981290+0000","last_undegraded":"2024-04-14T15:20:38.981290+0000","last_fullsized":"2024-04-14T15:20:38.981290+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:35.596656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:15:43.680852+0000","last_clean_scrub_stamp":"2024-04-14T15:18:35.596656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"479'24","reported_seq":601,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.957260+0000","last_change":"2024-04-14T15:17:43.061551+0000","last_active":"2024-04-14T15:20:33.957260+0000","last_peered":"2024-04-14T15:20:33.957260+0000","last_clean":"2024-04-14T15:20:33.957260+0000","last_became_active":"2024-04-14T15:15:21.884783+0000","last_became_peered":"2024-04-14T15:15:21.884783+0000","last_unstale":"2024-04-14T15:20:33.957260+0000","last_undegraded":"2024-04-14T15:20:33.957260+0000","last_fullsized":"2024-04-14T15:20:33.957260+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'16","last_scrub_stamp":"2024-04-14T15:17:43.061458+0000","last_deep_scrub":"102'16","last_deep_scrub_stamp":"2024-04-14T15:12:35.999326+0000","last_clean_scrub_stamp":"2024-04-14T15:17:43.061458+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":950,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-04-14T15:20:37.984349+0000","last_change":"2024-04-14T15:19:11.750498+0000","last_active":"2024-04-14T15:20:37.984349+0000","last_peered":"2024-04-14T15:20:37.984349+0000","last_clean":"2024-04-14T15:20:37.984349+0000","last_became_active":"2024-04-14T15:09:43.023220+0000","last_became_peered":"2024-04-14T15:09:43.023220+0000","last_unstale":"2024-04-14T15:20:37.984349+0000","last_undegraded":"2024-04-14T15:20:37.984349+0000","last_fullsized":"2024-04-14T15:20:37.984349+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:11.750419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:19:11.750419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":935,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.979419+0000","last_change":"2024-04-14T15:19:54.634365+0000","last_active":"2024-04-14T15:20:38.979419+0000","last_peered":"2024-04-14T15:20:38.979419+0000","last_clean":"2024-04-14T15:20:38.979419+0000","last_became_active":"2024-04-14T15:09:47.044665+0000","last_became_peered":"2024-04-14T15:09:47.044665+0000","last_unstale":"2024-04-14T15:20:38.979419+0000","last_undegraded":"2024-04-14T15:20:38.979419+0000","last_fullsized":"2024-04-14T15:20:38.979419+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:54.634279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:19:54.634279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:23.855974+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":604,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.958003+0000","last_change":"2024-04-14T15:19:23.462816+0000","last_active":"2024-04-14T15:20:33.958003+0000","last_peered":"2024-04-14T15:20:33.958003+0000","last_clean":"2024-04-14T15:20:33.958003+0000","last_became_active":"2024-04-14T15:15:21.883940+0000","last_became_peered":"2024-04-14T15:15:21.883940+0000","last_unstale":"2024-04-14T15:20:33.958003+0000","last_undegraded":"2024-04-14T15:20:33.958003+0000","last_fullsized":"2024-04-14T15:20:33.958003+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:23.462727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:19:23.462727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:46.221952+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1b","version":"91'130","reported_seq":877,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.942825+0000","last_change":"2024-04-14T15:20:32.493769+0000","last_active":"2024-04-14T15:20:33.942825+0000","last_peered":"2024-04-14T15:20:33.942825+0000","last_clean":"2024-04-14T15:20:33.942825+0000","last_became_active":"2024-04-14T15:20:31.939956+0000","last_became_peered":"2024-04-14T15:20:31.939956+0000","last_unstale":"2024-04-14T15:20:33.942825+0000","last_undegraded":"2024-04-14T15:20:33.942825+0000","last_fullsized":"2024-04-14T15:20:33.942825+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":446,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:20:32.493686+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:20:32.493686+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:53.355409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":829,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.953997+0000","last_change":"2024-04-14T15:18:25.997409+0000","last_active":"2024-04-14T15:20:33.953997+0000","last_peered":"2024-04-14T15:20:33.953997+0000","last_clean":"2024-04-14T15:20:33.953997+0000","last_became_active":"2024-04-14T15:12:56.335230+0000","last_became_peered":"2024-04-14T15:12:56.335230+0000","last_unstale":"2024-04-14T15:20:33.953997+0000","last_undegraded":"2024-04-14T15:20:33.953997+0000","last_fullsized":"2024-04-14T15:20:33.953997+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:25.997333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:18:25.997333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"479'20","reported_seq":961,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.947746+0000","last_change":"2024-04-14T15:20:06.373257+0000","last_active":"2024-04-14T15:20:33.947746+0000","last_peered":"2024-04-14T15:20:33.947746+0000","last_clean":"2024-04-14T15:20:33.947746+0000","last_became_active":"2024-04-14T15:09:45.181456+0000","last_became_peered":"2024-04-14T15:09:45.181456+0000","last_unstale":"2024-04-14T15:20:33.947746+0000","last_undegraded":"2024-04-14T15:20:33.947746+0000","last_fullsized":"2024-04-14T15:20:33.947746+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'20","last_scrub_stamp":"2024-04-14T15:20:06.373158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:20:06.373158+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:19.959672+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":502,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.981746+0000","last_change":"2024-04-14T15:19:57.575006+0000","last_active":"2024-04-14T15:20:38.981746+0000","last_peered":"2024-04-14T15:20:38.981746+0000","last_clean":"2024-04-14T15:20:38.981746+0000","last_became_active":"2024-04-14T15:15:44.878336+0000","last_became_peered":"2024-04-14T15:15:44.878336+0000","last_unstale":"2024-04-14T15:20:38.981746+0000","last_undegraded":"2024-04-14T15:20:38.981746+0000","last_fullsized":"2024-04-14T15:20:38.981746+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:57.574879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:19:57.574879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:02.545636+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"72'8","reported_seq":1086,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-04-14T15:20:37.984511+0000","last_change":"2024-04-14T15:18:28.793061+0000","last_active":"2024-04-14T15:20:37.984511+0000","last_peered":"2024-04-14T15:20:37.984511+0000","last_clean":"2024-04-14T15:20:37.984511+0000","last_became_active":"2024-04-14T15:09:47.048216+0000","last_became_peered":"2024-04-14T15:09:47.048216+0000","last_unstale":"2024-04-14T15:20:37.984511+0000","last_undegraded":"2024-04-14T15:20:37.984511+0000","last_fullsized":"2024-04-14T15:20:37.984511+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:18:28.792975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:18:28.792975+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"72'1","reported_seq":940,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.980961+0000","last_change":"2024-04-14T15:20:04.672008+0000","last_active":"2024-04-14T15:20:38.980961+0000","last_peered":"2024-04-14T15:20:38.980961+0000","last_clean":"2024-04-14T15:20:38.980961+0000","last_became_active":"2024-04-14T15:09:49.062051+0000","last_became_peered":"2024-04-14T15:09:49.062051+0000","last_unstale":"2024-04-14T15:20:38.980961+0000","last_undegraded":"2024-04-14T15:20:38.980961+0000","last_fullsized":"2024-04-14T15:20:38.980961+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-04-14T15:20:04.671896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:20:04.671896+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:09.232031+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"91'130","reported_seq":722,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.954284+0000","last_change":"2024-04-14T15:20:32.771814+0000","last_active":"2024-04-14T15:20:33.954284+0000","last_peered":"2024-04-14T15:20:33.954284+0000","last_clean":"2024-04-14T15:20:33.954284+0000","last_became_active":"2024-04-14T15:20:31.940241+0000","last_became_peered":"2024-04-14T15:20:31.940241+0000","last_unstale":"2024-04-14T15:20:33.954284+0000","last_undegraded":"2024-04-14T15:20:33.954284+0000","last_fullsized":"2024-04-14T15:20:33.954284+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":446,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:20:32.771731+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:20:32.771731+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:48.378770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":831,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.945902+0000","last_change":"2024-04-14T15:19:13.345390+0000","last_active":"2024-04-14T15:20:33.945902+0000","last_peered":"2024-04-14T15:20:33.945902+0000","last_clean":"2024-04-14T15:20:33.945902+0000","last_became_active":"2024-04-14T15:12:56.543793+0000","last_became_peered":"2024-04-14T15:12:56.543793+0000","last_unstale":"2024-04-14T15:20:33.945902+0000","last_undegraded":"2024-04-14T15:20:33.945902+0000","last_fullsized":"2024-04-14T15:20:33.945902+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:13.345316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:19:13.345316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:38.673629+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"479'19","reported_seq":988,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-04-14T15:20:37.984511+0000","last_change":"2024-04-14T15:18:31.818435+0000","last_active":"2024-04-14T15:20:37.984511+0000","last_peered":"2024-04-14T15:20:37.984511+0000","last_clean":"2024-04-14T15:20:37.984511+0000","last_became_active":"2024-04-14T15:10:07.764183+0000","last_became_peered":"2024-04-14T15:10:07.764183+0000","last_unstale":"2024-04-14T15:20:37.984511+0000","last_undegraded":"2024-04-14T15:20:37.984511+0000","last_fullsized":"2024-04-14T15:20:37.984511+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-04-14T15:18:31.818311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:18:31.818311+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"65'2","reported_seq":615,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.957068+0000","last_change":"2024-04-14T15:16:56.943047+0000","last_active":"2024-04-14T15:20:33.957068+0000","last_peered":"2024-04-14T15:20:33.957068+0000","last_clean":"2024-04-14T15:20:33.957068+0000","last_became_active":"2024-04-14T15:15:21.885343+0000","last_became_peered":"2024-04-14T15:15:21.885343+0000","last_unstale":"2024-04-14T15:20:33.957068+0000","last_undegraded":"2024-04-14T15:20:33.957068+0000","last_fullsized":"2024-04-14T15:20:33.957068+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-04-14T15:16:56.942840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:16:56.942840+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":927,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.947810+0000","last_change":"2024-04-14T15:19:21.349524+0000","last_active":"2024-04-14T15:20:33.947810+0000","last_peered":"2024-04-14T15:20:33.947810+0000","last_clean":"2024-04-14T15:20:33.947810+0000","last_became_active":"2024-04-14T15:09:47.045730+0000","last_became_peered":"2024-04-14T15:09:47.045730+0000","last_unstale":"2024-04-14T15:20:33.947810+0000","last_undegraded":"2024-04-14T15:20:33.947810+0000","last_fullsized":"2024-04-14T15:20:33.947810+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:21.349441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:19:21.349441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:47.517518+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":915,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.945746+0000","last_change":"2024-04-14T15:16:57.235339+0000","last_active":"2024-04-14T15:20:33.945746+0000","last_peered":"2024-04-14T15:20:33.945746+0000","last_clean":"2024-04-14T15:20:33.945746+0000","last_became_active":"2024-04-14T15:09:49.057545+0000","last_became_peered":"2024-04-14T15:09:49.057545+0000","last_unstale":"2024-04-14T15:20:33.945746+0000","last_undegraded":"2024-04-14T15:20:33.945746+0000","last_fullsized":"2024-04-14T15:20:33.945746+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:57.235261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:21.247695+0000","last_clean_scrub_stamp":"2024-04-14T15:16:57.235261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.19","version":"91'130","reported_seq":574,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.979322+0000","last_change":"2024-04-14T15:20:32.703084+0000","last_active":"2024-04-14T15:20:38.979322+0000","last_peered":"2024-04-14T15:20:38.979322+0000","last_clean":"2024-04-14T15:20:38.979322+0000","last_became_active":"2024-04-14T15:20:31.938460+0000","last_became_peered":"2024-04-14T15:20:31.938460+0000","last_unstale":"2024-04-14T15:20:38.979322+0000","last_undegraded":"2024-04-14T15:20:38.979322+0000","last_fullsized":"2024-04-14T15:20:38.979322+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":446,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:20:32.703008+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:20:32.703008+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:55.892322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":494,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.982271+0000","last_change":"2024-04-14T15:17:44.364114+0000","last_active":"2024-04-14T15:20:38.982271+0000","last_peered":"2024-04-14T15:20:38.982271+0000","last_clean":"2024-04-14T15:20:38.982271+0000","last_became_active":"2024-04-14T15:15:44.882604+0000","last_became_peered":"2024-04-14T15:15:44.882604+0000","last_unstale":"2024-04-14T15:20:38.982271+0000","last_undegraded":"2024-04-14T15:20:38.982271+0000","last_fullsized":"2024-04-14T15:20:38.982271+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:44.364013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:17:44.364013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"479'27","reported_seq":971,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.955561+0000","last_change":"2024-04-14T15:18:32.950805+0000","last_active":"2024-04-14T15:20:33.955561+0000","last_peered":"2024-04-14T15:20:33.955561+0000","last_clean":"2024-04-14T15:20:33.955561+0000","last_became_active":"2024-04-14T15:09:45.032887+0000","last_became_peered":"2024-04-14T15:09:45.032887+0000","last_unstale":"2024-04-14T15:20:33.955561+0000","last_undegraded":"2024-04-14T15:20:33.955561+0000","last_fullsized":"2024-04-14T15:20:33.955561+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-04-14T15:18:32.950706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:18:32.950706+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":945,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-04-14T15:20:37.982265+0000","last_change":"2024-04-14T15:18:27.799078+0000","last_active":"2024-04-14T15:20:37.982265+0000","last_peered":"2024-04-14T15:20:37.982265+0000","last_clean":"2024-04-14T15:20:37.982265+0000","last_became_active":"2024-04-14T15:09:43.021494+0000","last_became_peered":"2024-04-14T15:09:43.021494+0000","last_unstale":"2024-04-14T15:20:37.982265+0000","last_undegraded":"2024-04-14T15:20:37.982265+0000","last_fullsized":"2024-04-14T15:20:37.982265+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:27.798964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:18:27.798964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":927,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.945902+0000","last_change":"2024-04-14T15:18:22.953738+0000","last_active":"2024-04-14T15:20:33.945902+0000","last_peered":"2024-04-14T15:20:33.945902+0000","last_clean":"2024-04-14T15:20:33.945902+0000","last_became_active":"2024-04-14T15:09:47.042733+0000","last_became_peered":"2024-04-14T15:09:47.042733+0000","last_unstale":"2024-04-14T15:20:33.945902+0000","last_undegraded":"2024-04-14T15:20:33.945902+0000","last_fullsized":"2024-04-14T15:20:33.945902+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:22.953661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:18:22.953661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":932,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.983173+0000","last_change":"2024-04-14T15:17:54.453289+0000","last_active":"2024-04-14T15:20:38.983173+0000","last_peered":"2024-04-14T15:20:38.983173+0000","last_clean":"2024-04-14T15:20:38.983173+0000","last_became_active":"2024-04-14T15:09:49.127979+0000","last_became_peered":"2024-04-14T15:09:49.127979+0000","last_unstale":"2024-04-14T15:20:38.983173+0000","last_undegraded":"2024-04-14T15:20:38.983173+0000","last_fullsized":"2024-04-14T15:20:38.983173+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:54.453174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:54.453174+0000","last_clean_scrub_stamp":"2024-04-14T15:17:54.453174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.18","version":"91'130","reported_seq":846,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.953526+0000","last_change":"2024-04-14T15:20:31.938853+0000","last_active":"2024-04-14T15:20:33.953526+0000","last_peered":"2024-04-14T15:20:33.953526+0000","last_clean":"2024-04-14T15:20:33.953526+0000","last_became_active":"2024-04-14T15:20:31.938379+0000","last_became_peered":"2024-04-14T15:20:31.938379+0000","last_unstale":"2024-04-14T15:20:33.953526+0000","last_undegraded":"2024-04-14T15:20:33.953526+0000","last_fullsized":"2024-04-14T15:20:33.953526+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":446,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:02.957360+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:02.957360+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":840,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.979690+0000","last_change":"2024-04-14T15:18:26.532035+0000","last_active":"2024-04-14T15:20:38.979690+0000","last_peered":"2024-04-14T15:20:38.979690+0000","last_clean":"2024-04-14T15:20:38.979690+0000","last_became_active":"2024-04-14T15:12:56.392352+0000","last_became_peered":"2024-04-14T15:12:56.392352+0000","last_unstale":"2024-04-14T15:20:38.979690+0000","last_undegraded":"2024-04-14T15:20:38.979690+0000","last_fullsized":"2024-04-14T15:20:38.979690+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:26.531955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:18:26.531955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"479'18","reported_seq":644,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.958415+0000","last_change":"2024-04-14T15:19:17.364178+0000","last_active":"2024-04-14T15:20:33.958415+0000","last_peered":"2024-04-14T15:20:33.958415+0000","last_clean":"2024-04-14T15:20:33.958415+0000","last_became_active":"2024-04-14T15:15:21.877838+0000","last_became_peered":"2024-04-14T15:15:21.877838+0000","last_unstale":"2024-04-14T15:20:33.958415+0000","last_undegraded":"2024-04-14T15:20:33.958415+0000","last_fullsized":"2024-04-14T15:20:33.958415+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'12","last_scrub_stamp":"2024-04-14T15:19:17.364059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:19:17.364059+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":616,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.957909+0000","last_change":"2024-04-14T15:19:25.470235+0000","last_active":"2024-04-14T15:20:33.957909+0000","last_peered":"2024-04-14T15:20:33.957909+0000","last_clean":"2024-04-14T15:20:33.957909+0000","last_became_active":"2024-04-14T15:15:21.883544+0000","last_became_peered":"2024-04-14T15:15:21.883544+0000","last_unstale":"2024-04-14T15:20:33.957909+0000","last_undegraded":"2024-04-14T15:20:33.957909+0000","last_fullsized":"2024-04-14T15:20:33.957909+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:25.470160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:19:25.470160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:47.982067+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":935,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.982781+0000","last_change":"2024-04-14T15:17:01.098681+0000","last_active":"2024-04-14T15:20:38.982781+0000","last_peered":"2024-04-14T15:20:38.982781+0000","last_clean":"2024-04-14T15:20:38.982781+0000","last_became_active":"2024-04-14T15:09:47.048045+0000","last_became_peered":"2024-04-14T15:09:47.048045+0000","last_unstale":"2024-04-14T15:20:38.982781+0000","last_undegraded":"2024-04-14T15:20:38.982781+0000","last_fullsized":"2024-04-14T15:20:38.982781+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:01.098599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:17:01.098599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":605,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.959746+0000","last_change":"2024-04-14T15:19:19.371279+0000","last_active":"2024-04-14T15:20:33.959746+0000","last_peered":"2024-04-14T15:20:33.959746+0000","last_clean":"2024-04-14T15:20:33.959746+0000","last_became_active":"2024-04-14T15:15:21.889691+0000","last_became_peered":"2024-04-14T15:15:21.889691+0000","last_unstale":"2024-04-14T15:20:33.959746+0000","last_undegraded":"2024-04-14T15:20:33.959746+0000","last_fullsized":"2024-04-14T15:20:33.959746+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:19.371182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:19:19.371182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:48.666091+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":837,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.955850+0000","last_change":"2024-04-14T15:20:01.841015+0000","last_active":"2024-04-14T15:20:33.955850+0000","last_peered":"2024-04-14T15:20:33.955850+0000","last_clean":"2024-04-14T15:20:33.955850+0000","last_became_active":"2024-04-14T15:12:56.375278+0000","last_became_peered":"2024-04-14T15:12:56.375278+0000","last_unstale":"2024-04-14T15:20:33.955850+0000","last_undegraded":"2024-04-14T15:20:33.955850+0000","last_fullsized":"2024-04-14T15:20:33.955850+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:01.840924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:01.840924+0000","last_clean_scrub_stamp":"2024-04-14T15:20:01.840924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:05.629144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"479'19","reported_seq":968,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.982212+0000","last_change":"2024-04-14T15:17:52.445394+0000","last_active":"2024-04-14T15:20:38.982212+0000","last_peered":"2024-04-14T15:20:38.982212+0000","last_clean":"2024-04-14T15:20:38.982212+0000","last_became_active":"2024-04-14T15:09:45.030093+0000","last_became_peered":"2024-04-14T15:09:45.030093+0000","last_unstale":"2024-04-14T15:20:38.982212+0000","last_undegraded":"2024-04-14T15:20:38.982212+0000","last_fullsized":"2024-04-14T15:20:38.982212+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'13","last_scrub_stamp":"2024-04-14T15:17:52.445294+0000","last_deep_scrub":"276'13","last_deep_scrub_stamp":"2024-04-14T15:15:43.969775+0000","last_clean_scrub_stamp":"2024-04-14T15:17:52.445294+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":649,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.980147+0000","last_change":"2024-04-14T15:20:00.650164+0000","last_active":"2024-04-14T15:20:38.980147+0000","last_peered":"2024-04-14T15:20:38.980147+0000","last_clean":"2024-04-14T15:20:38.980147+0000","last_became_active":"2024-04-14T15:14:30.857727+0000","last_became_peered":"2024-04-14T15:14:30.857727+0000","last_unstale":"2024-04-14T15:20:38.980147+0000","last_undegraded":"2024-04-14T15:20:38.980147+0000","last_fullsized":"2024-04-14T15:20:38.980147+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:00.650062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:00.650062+0000","last_clean_scrub_stamp":"2024-04-14T15:20:00.650062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:25.732113+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":484,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.947226+0000","last_change":"2024-04-14T15:16:59.695423+0000","last_active":"2024-04-14T15:20:33.947226+0000","last_peered":"2024-04-14T15:20:33.947226+0000","last_clean":"2024-04-14T15:20:33.947226+0000","last_became_active":"2024-04-14T15:15:44.883121+0000","last_became_peered":"2024-04-14T15:15:44.883121+0000","last_unstale":"2024-04-14T15:20:33.947226+0000","last_undegraded":"2024-04-14T15:20:33.947226+0000","last_fullsized":"2024-04-14T15:20:33.947226+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:16:59.695337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:16:59.695337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":922,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.946183+0000","last_change":"2024-04-14T15:17:33.952503+0000","last_active":"2024-04-14T15:20:33.946183+0000","last_peered":"2024-04-14T15:20:33.946183+0000","last_clean":"2024-04-14T15:20:33.946183+0000","last_became_active":"2024-04-14T15:09:49.061756+0000","last_became_peered":"2024-04-14T15:09:49.061756+0000","last_unstale":"2024-04-14T15:20:33.946183+0000","last_undegraded":"2024-04-14T15:20:33.946183+0000","last_fullsized":"2024-04-14T15:20:33.946183+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:33.952312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:17:33.952312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":847,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-04-14T15:20:37.982740+0000","last_change":"2024-04-14T15:19:12.723151+0000","last_active":"2024-04-14T15:20:37.982740+0000","last_peered":"2024-04-14T15:20:37.982740+0000","last_clean":"2024-04-14T15:20:37.982740+0000","last_became_active":"2024-04-14T15:12:56.232831+0000","last_became_peered":"2024-04-14T15:12:56.232831+0000","last_unstale":"2024-04-14T15:20:37.982740+0000","last_undegraded":"2024-04-14T15:20:37.982740+0000","last_fullsized":"2024-04-14T15:20:37.982740+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:12.723073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:19:12.723073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1b","version":"479'9","reported_seq":954,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.982481+0000","last_change":"2024-04-14T15:18:35.481608+0000","last_active":"2024-04-14T15:20:38.982481+0000","last_peered":"2024-04-14T15:20:38.982481+0000","last_clean":"2024-04-14T15:20:38.982481+0000","last_became_active":"2024-04-14T15:09:45.028667+0000","last_became_peered":"2024-04-14T15:09:45.028667+0000","last_unstale":"2024-04-14T15:20:38.982481+0000","last_undegraded":"2024-04-14T15:20:38.982481+0000","last_fullsized":"2024-04-14T15:20:38.982481+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'7","last_scrub_stamp":"2024-04-14T15:18:35.481504+0000","last_deep_scrub":"87'7","last_deep_scrub_stamp":"2024-04-14T15:11:10.510785+0000","last_clean_scrub_stamp":"2024-04-14T15:18:35.481504+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":409,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.955408+0000","last_change":"2024-04-14T15:19:26.879798+0000","last_active":"2024-04-14T15:20:33.955408+0000","last_peered":"2024-04-14T15:20:33.955408+0000","last_clean":"2024-04-14T15:20:33.955408+0000","last_became_active":"2024-04-14T15:16:24.910246+0000","last_became_peered":"2024-04-14T15:16:24.910246+0000","last_unstale":"2024-04-14T15:20:33.955408+0000","last_undegraded":"2024-04-14T15:20:33.955408+0000","last_fullsized":"2024-04-14T15:20:33.955408+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:26.879685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:08.254601+0000","last_clean_scrub_stamp":"2024-04-14T15:19:26.879685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:37.880480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":935,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.980006+0000","last_change":"2024-04-14T15:19:15.567537+0000","last_active":"2024-04-14T15:20:38.980006+0000","last_peered":"2024-04-14T15:20:38.980006+0000","last_clean":"2024-04-14T15:20:38.980006+0000","last_became_active":"2024-04-14T15:09:47.046517+0000","last_became_peered":"2024-04-14T15:09:47.046517+0000","last_unstale":"2024-04-14T15:20:38.980006+0000","last_undegraded":"2024-04-14T15:20:38.980006+0000","last_fullsized":"2024-04-14T15:20:38.980006+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:15.567432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:19:15.567432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":487,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.944616+0000","last_change":"2024-04-14T15:18:21.023508+0000","last_active":"2024-04-14T15:20:33.944616+0000","last_peered":"2024-04-14T15:20:33.944616+0000","last_clean":"2024-04-14T15:20:33.944616+0000","last_became_active":"2024-04-14T15:15:44.887227+0000","last_became_peered":"2024-04-14T15:15:44.887227+0000","last_unstale":"2024-04-14T15:20:33.944616+0000","last_undegraded":"2024-04-14T15:20:33.944616+0000","last_fullsized":"2024-04-14T15:20:33.944616+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:21.023426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:56.268086+0000","last_clean_scrub_stamp":"2024-04-14T15:18:21.023426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1d","version":"91'130","reported_seq":479,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.980751+0000","last_change":"2024-04-14T15:20:31.940207+0000","last_active":"2024-04-14T15:20:38.980751+0000","last_peered":"2024-04-14T15:20:38.980751+0000","last_clean":"2024-04-14T15:20:38.980751+0000","last_became_active":"2024-04-14T15:20:31.939730+0000","last_became_peered":"2024-04-14T15:20:31.939730+0000","last_unstale":"2024-04-14T15:20:38.980751+0000","last_undegraded":"2024-04-14T15:20:38.980751+0000","last_fullsized":"2024-04-14T15:20:38.980751+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":446,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:03.081016+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:03.081016+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":840,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.980786+0000","last_change":"2024-04-14T15:18:34.558447+0000","last_active":"2024-04-14T15:20:38.980786+0000","last_peered":"2024-04-14T15:20:38.980786+0000","last_clean":"2024-04-14T15:20:38.980786+0000","last_became_active":"2024-04-14T15:12:56.189117+0000","last_became_peered":"2024-04-14T15:12:56.189117+0000","last_unstale":"2024-04-14T15:20:38.980786+0000","last_undegraded":"2024-04-14T15:20:38.980786+0000","last_fullsized":"2024-04-14T15:20:38.980786+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:34.558312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:34.558312+0000","last_clean_scrub_stamp":"2024-04-14T15:18:34.558312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.18","version":"479'15","reported_seq":964,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.981644+0000","last_change":"2024-04-14T15:18:36.441934+0000","last_active":"2024-04-14T15:20:38.981644+0000","last_peered":"2024-04-14T15:20:38.981644+0000","last_clean":"2024-04-14T15:20:38.981644+0000","last_became_active":"2024-04-14T15:09:45.032079+0000","last_became_peered":"2024-04-14T15:09:45.032079+0000","last_unstale":"2024-04-14T15:20:38.981644+0000","last_undegraded":"2024-04-14T15:20:38.981644+0000","last_fullsized":"2024-04-14T15:20:38.981644+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-04-14T15:18:36.441813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:18:36.441813+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":609,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.961801+0000","last_change":"2024-04-14T15:17:00.948362+0000","last_active":"2024-04-14T15:20:33.961801+0000","last_peered":"2024-04-14T15:20:33.961801+0000","last_clean":"2024-04-14T15:20:33.961801+0000","last_became_active":"2024-04-14T15:15:21.891926+0000","last_became_peered":"2024-04-14T15:15:21.891926+0000","last_unstale":"2024-04-14T15:20:33.961801+0000","last_undegraded":"2024-04-14T15:20:33.961801+0000","last_fullsized":"2024-04-14T15:20:33.961801+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:00.948283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:00.948283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":927,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.944172+0000","last_change":"2024-04-14T15:19:56.903138+0000","last_active":"2024-04-14T15:20:33.944172+0000","last_peered":"2024-04-14T15:20:33.944172+0000","last_clean":"2024-04-14T15:20:33.944172+0000","last_became_active":"2024-04-14T15:09:47.041486+0000","last_became_peered":"2024-04-14T15:09:47.041486+0000","last_unstale":"2024-04-14T15:20:33.944172+0000","last_undegraded":"2024-04-14T15:20:33.944172+0000","last_fullsized":"2024-04-14T15:20:33.944172+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:56.903059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:19:56.903059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:25.052114+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":931,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.982138+0000","last_change":"2024-04-14T15:20:36.761316+0000","last_active":"2024-04-14T15:20:38.982138+0000","last_peered":"2024-04-14T15:20:38.982138+0000","last_clean":"2024-04-14T15:20:38.982138+0000","last_became_active":"2024-04-14T15:09:49.058904+0000","last_became_peered":"2024-04-14T15:09:49.058904+0000","last_unstale":"2024-04-14T15:20:38.982138+0000","last_undegraded":"2024-04-14T15:20:38.982138+0000","last_fullsized":"2024-04-14T15:20:38.982138+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:36.761246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:20:36.761246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:58.179590+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1c","version":"91'130","reported_seq":868,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.958349+0000","last_change":"2024-04-14T15:20:32.880440+0000","last_active":"2024-04-14T15:20:33.958349+0000","last_peered":"2024-04-14T15:20:33.958349+0000","last_clean":"2024-04-14T15:20:33.958349+0000","last_became_active":"2024-04-14T15:20:31.941012+0000","last_became_peered":"2024-04-14T15:20:31.941012+0000","last_unstale":"2024-04-14T15:20:33.958349+0000","last_undegraded":"2024-04-14T15:20:33.958349+0000","last_fullsized":"2024-04-14T15:20:33.958349+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":446,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:20:32.880362+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:20:32.880362+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:36.780631+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":494,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.981154+0000","last_change":"2024-04-14T15:17:52.593706+0000","last_active":"2024-04-14T15:20:38.981154+0000","last_peered":"2024-04-14T15:20:38.981154+0000","last_clean":"2024-04-14T15:20:38.981154+0000","last_became_active":"2024-04-14T15:15:44.889605+0000","last_became_peered":"2024-04-14T15:15:44.889605+0000","last_unstale":"2024-04-14T15:20:38.981154+0000","last_undegraded":"2024-04-14T15:20:38.981154+0000","last_fullsized":"2024-04-14T15:20:38.981154+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:52.593618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:17:52.593618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"479'34","reported_seq":662,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.957328+0000","last_change":"2024-04-14T15:19:53.781367+0000","last_active":"2024-04-14T15:20:33.957328+0000","last_peered":"2024-04-14T15:20:33.957328+0000","last_clean":"2024-04-14T15:20:33.957328+0000","last_became_active":"2024-04-14T15:15:21.884310+0000","last_became_peered":"2024-04-14T15:15:21.884310+0000","last_unstale":"2024-04-14T15:20:33.957328+0000","last_undegraded":"2024-04-14T15:20:33.957328+0000","last_fullsized":"2024-04-14T15:20:33.957328+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'34","last_scrub_stamp":"2024-04-14T15:19:53.781241+0000","last_deep_scrub":"306'22","last_deep_scrub_stamp":"2024-04-14T15:16:13.898106+0000","last_clean_scrub_stamp":"2024-04-14T15:19:53.781241+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:20:54.804612+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":935,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.947649+0000","last_change":"2024-04-14T15:19:18.347632+0000","last_active":"2024-04-14T15:20:33.947649+0000","last_peered":"2024-04-14T15:20:33.947649+0000","last_clean":"2024-04-14T15:20:33.947649+0000","last_became_active":"2024-04-14T15:09:43.017351+0000","last_became_peered":"2024-04-14T15:09:43.017351+0000","last_unstale":"2024-04-14T15:20:33.947649+0000","last_undegraded":"2024-04-14T15:20:33.947649+0000","last_fullsized":"2024-04-14T15:20:33.947649+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:18.347533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:19:18.347533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":935,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-14T15:20:38.981831+0000","last_change":"2024-04-14T15:19:49.631364+0000","last_active":"2024-04-14T15:20:38.981831+0000","last_peered":"2024-04-14T15:20:38.981831+0000","last_clean":"2024-04-14T15:20:38.981831+0000","last_became_active":"2024-04-14T15:09:47.042538+0000","last_became_peered":"2024-04-14T15:09:47.042538+0000","last_unstale":"2024-04-14T15:20:38.981831+0000","last_undegraded":"2024-04-14T15:20:38.981831+0000","last_fullsized":"2024-04-14T15:20:38.981831+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:49.631273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:08.512130+0000","last_clean_scrub_stamp":"2024-04-14T15:19:49.631273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:09.671784+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":919,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-04-14T15:20:33.957686+0000","last_change":"2024-04-14T15:17:56.237366+0000","last_active":"2024-04-14T15:20:33.957686+0000","last_peered":"2024-04-14T15:20:33.957686+0000","last_clean":"2024-04-14T15:20:33.957686+0000","last_became_active":"2024-04-14T15:09:49.129636+0000","last_became_peered":"2024-04-14T15:09:49.129636+0000","last_unstale":"2024-04-14T15:20:33.957686+0000","last_undegraded":"2024-04-14T15:20:33.957686+0000","last_fullsized":"2024-04-14T15:20:33.957686+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:56.237273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:17:56.237273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":307,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":48,"acting":48,"num_store_stats":4},{"poolid":110,"num_pg":32,"stat_sum":{"num_bytes":8204119,"num_objects":176,"num_object_clones":10,"num_object_copies":528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":175,"num_whiteouts":0,"num_read":167,"num_read_kb":40,"num_write":700,"num_write_kb":12753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":41,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":20,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":26681344,"data_stored":24604141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2020,"ondisk_log_size":2020,"up":96,"acting":96,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":30,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3770,"ondisk_log_size":3770,"up":90,"acting":90,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":644,"num_read_kb":639,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1124,"num_read_kb":881,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":779,"ondisk_log_size":779,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71,"ondisk_log_size":71,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103305,"num_pgs":113,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":61496,"kb_used_data":16396,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93683656,"statfs":{"total":95995035648,"available":95932063744,"internally_reserved":0,"allocated":16789504,"data_stored":13073671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":46135754},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397644,"num_pgs":89,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":75964,"kb_used_data":8656,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93669188,"statfs":{"total":95995035648,"available":95917248512,"internally_reserved":0,"allocated":8863744,"data_stored":5207203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575013,"num_pgs":100,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":75656,"kb_used_data":12508,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93669496,"statfs":{"total":95995035648,"available":95917563904,"internally_reserved":0,"allocated":12808192,"data_stored":9041622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738538,"num_pgs":90,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":88204,"kb_used_data":12588,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93656948,"statfs":{"total":95995035648,"available":95904714752,"internally_reserved":0,"allocated":12890112,"data_stored":9166545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77396423},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411489,"num_pgs":87,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":88172,"kb_used_data":12568,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93656980,"statfs":{"total":95995035648,"available":95904747520,"internally_reserved":0,"allocated":12869632,"data_stored":9107901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182555,"num_pgs":103,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":62264,"kb_used_data":8856,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93682888,"statfs":{"total":95995035648,"available":95931277312,"internally_reserved":0,"allocated":9068544,"data_stored":5429375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986326,"num_pgs":97,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":75620,"kb_used_data":8312,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93669532,"statfs":{"total":95995035648,"available":95917600768,"internally_reserved":0,"allocated":8511488,"data_stored":4879061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659283,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":79948,"kb_used_data":12664,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93665204,"statfs":{"total":95995035648,"available":95913168896,"internally_reserved":0,"allocated":12967936,"data_stored":9335513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4292608,"data_stored":4000650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4509696,"data_stored":4195049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4370432,"data_stored":4009034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":8636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4403200,"data_stored":4194784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":229376,"data_stored":534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8482816,"data_stored":8194980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":307,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":307,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":307,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":307,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-14T15:20:41.471 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:20:41.494 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:41.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:41.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:20:41.596 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:20:41.621 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:41.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:41.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:20:41.722 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:20:41.745 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:41.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:41.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:20:41.847 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:41.870 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:41.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:41.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:41.972 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:20:41.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:41.996 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:41.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:20:42.060 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:41 smithi090 ceph-mon[41670]: 11.f scrub starts 2024-04-14T15:20:42.061 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:41 smithi090 ceph-mon[41670]: 11.f scrub ok 2024-04-14T15:20:42.061 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:41 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-14T15:20:42.061 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:41 smithi090 ceph-mon[41670]: osdmap e517: 8 total, 8 up, 8 in 2024-04-14T15:20:42.061 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:41 smithi090 ceph-mon[30399]: 11.f scrub starts 2024-04-14T15:20:42.061 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:41 smithi090 ceph-mon[30399]: 11.f scrub ok 2024-04-14T15:20:42.061 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:41 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-14T15:20:42.061 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:41 smithi090 ceph-mon[30399]: osdmap e517: 8 total, 8 up, 8 in 2024-04-14T15:20:42.096 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:20:42.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:42.120 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:42.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:20:42.221 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:20:42.244 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:42.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:42.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:20:42.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.301+0000 7fec702f4700 1 -- 172.21.15.90:0/438232351 >> v1:172.21.15.90:6789/0 conn(0x7fec681115e0 legacy=0x7fec68113aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:42.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.301+0000 7fec702f4700 1 -- 172.21.15.90:0/438232351 shutdown_connections 2024-04-14T15:20:42.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.301+0000 7fec702f4700 1 -- 172.21.15.90:0/438232351 >> 172.21.15.90:0/438232351 conn(0x7fec68076aa0 msgr2=0x7fec68076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:20:42.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.301+0000 7fec702f4700 1 -- 172.21.15.90:0/438232351 shutdown_connections 2024-04-14T15:20:42.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.301+0000 7fec702f4700 1 -- 172.21.15.90:0/438232351 wait complete. 2024-04-14T15:20:42.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.302+0000 7fec702f4700 1 librados: starting msgr at 2024-04-14T15:20:42.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.302+0000 7fec702f4700 1 librados: starting objecter 2024-04-14T15:20:42.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.302+0000 7fec702f4700 1 Processor -- start 2024-04-14T15:20:42.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.302+0000 7fec702f4700 1 -- start start 2024-04-14T15:20:42.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.302+0000 7fec702f4700 1 librados: setting wanted keys 2024-04-14T15:20:42.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.302+0000 7fec702f4700 1 librados: calling monclient init 2024-04-14T15:20:42.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.302+0000 7fec702f4700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fec681a9a60 con 0x7fec681115e0 2024-04-14T15:20:42.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.302+0000 7fec702f4700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fec681aac80 con 0x7fec6810d950 2024-04-14T15:20:42.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.302+0000 7fec702f4700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fec681abec0 con 0x7fec6810abe0 2024-04-14T15:20:42.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.302+0000 7fec6e891700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fec681115e0 0x7fec681a81a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:40568/0 (socket says 172.21.15.90:40568) 2024-04-14T15:20:42.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.302+0000 7fec6e891700 1 -- 172.21.15.90:0/3563185833 learned_addr learned my addr 172.21.15.90:0/3563185833 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:20:42.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.302+0000 7fec5f7fe700 10 client.?.objecter ms_handle_connect 0x7fec681115e0 2024-04-14T15:20:42.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.302+0000 7fec5f7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:20:42.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.302+0000 7fec5f7fe700 10 client.?.objecter ms_handle_connect 0x7fec6810d950 2024-04-14T15:20:42.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.302+0000 7fec5f7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:20:42.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.302+0000 7fec5f7fe700 10 client.?.objecter ms_handle_connect 0x7fec6810abe0 2024-04-14T15:20:42.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.302+0000 7fec5f7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:20:42.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.302+0000 7fec5f7fe700 1 -- 172.21.15.90:0/3563185833 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2901077696 0 0) 0x7fec681aac80 con 0x7fec6810d950 2024-04-14T15:20:42.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.303+0000 7fec5f7fe700 1 -- 172.21.15.90:0/3563185833 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec540031f0 con 0x7fec6810d950 2024-04-14T15:20:42.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.303+0000 7fec5f7fe700 1 -- 172.21.15.90:0/3563185833 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3494802412 0 0) 0x7fec681a9a60 con 0x7fec681115e0 2024-04-14T15:20:42.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.303+0000 7fec5f7fe700 1 -- 172.21.15.90:0/3563185833 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec681aac80 con 0x7fec681115e0 2024-04-14T15:20:42.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.303+0000 7fec5f7fe700 1 -- 172.21.15.90:0/3563185833 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2059563513 0 0) 0x7fec681abec0 con 0x7fec6810abe0 2024-04-14T15:20:42.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.303+0000 7fec5f7fe700 1 -- 172.21.15.90:0/3563185833 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec681a9a60 con 0x7fec6810abe0 2024-04-14T15:20:42.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.303+0000 7fec5f7fe700 1 -- 172.21.15.90:0/3563185833 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3815921314 0 0) 0x7fec540031f0 con 0x7fec6810d950 2024-04-14T15:20:42.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.303+0000 7fec5f7fe700 1 -- 172.21.15.90:0/3563185833 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fec681abec0 con 0x7fec6810d950 2024-04-14T15:20:42.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.303+0000 7fec5f7fe700 1 -- 172.21.15.90:0/3563185833 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fec58001a30 con 0x7fec6810d950 2024-04-14T15:20:42.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.303+0000 7fec5f7fe700 1 -- 172.21.15.90:0/3563185833 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2623421274 0 0) 0x7fec681a9a60 con 0x7fec6810abe0 2024-04-14T15:20:42.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.303+0000 7fec5f7fe700 1 -- 172.21.15.90:0/3563185833 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fec540031f0 con 0x7fec6810abe0 2024-04-14T15:20:42.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.303+0000 7fec5f7fe700 1 -- 172.21.15.90:0/3563185833 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4038687980 0 0) 0x7fec681aac80 con 0x7fec681115e0 2024-04-14T15:20:42.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.303+0000 7fec5f7fe700 1 -- 172.21.15.90:0/3563185833 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fec681a9a60 con 0x7fec681115e0 2024-04-14T15:20:42.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.303+0000 7fec5f7fe700 1 -- 172.21.15.90:0/3563185833 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fec640017b0 con 0x7fec6810abe0 2024-04-14T15:20:42.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.303+0000 7fec5f7fe700 1 -- 172.21.15.90:0/3563185833 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1359698234 0 0) 0x7fec681abec0 con 0x7fec6810d950 2024-04-14T15:20:42.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.303+0000 7fec5f7fe700 1 -- 172.21.15.90:0/3563185833 >> v1:172.21.15.90:6790/0 conn(0x7fec6810abe0 legacy=0x7fec68101810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:42.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.303+0000 7fec5f7fe700 1 -- 172.21.15.90:0/3563185833 >> v1:172.21.15.90:6789/0 conn(0x7fec681115e0 legacy=0x7fec681a81a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:42.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.303+0000 7fec5f7fe700 1 -- 172.21.15.90:0/3563185833 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec681ad100 con 0x7fec6810d950 2024-04-14T15:20:42.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.303+0000 7fec702f4700 1 -- 172.21.15.90:0/3563185833 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fec681aaeb0 con 0x7fec6810d950 2024-04-14T15:20:42.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.303+0000 7fec702f4700 10 client.49345.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:20:42.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.303+0000 7fec702f4700 1 -- 172.21.15.90:0/3563185833 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fec681ab370 con 0x7fec6810d950 2024-04-14T15:20:42.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.303+0000 7fec702f4700 1 librados: init done 2024-04-14T15:20:42.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.303+0000 7fec5f7fe700 1 -- 172.21.15.90:0/3563185833 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fec58001ce0 con 0x7fec6810d950 2024-04-14T15:20:42.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.303+0000 7fec5f7fe700 1 -- 172.21.15.90:0/3563185833 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fec58004a70 con 0x7fec6810d950 2024-04-14T15:20:42.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.304+0000 7fec702f4700 1 -- 172.21.15.90:0/3563185833 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fec4c005100 con 0x7fec6810d950 2024-04-14T15:20:42.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.305+0000 7fec5f7fe700 1 -- 172.21.15.90:0/3563185833 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fec5801b620 con 0x7fec6810d950 2024-04-14T15:20:42.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.306+0000 7fec5f7fe700 1 -- 172.21.15.90:0/3563185833 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(517..517 src has 51..517) ==== 8264+0+0 (unknown 3811061181 0 0) 0x7fec58089910 con 0x7fec6810d950 2024-04-14T15:20:42.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.306+0000 7fec5f7fe700 10 client.49345.objecter ms_dispatch 0x7fec68001550 osd_map(517..517 src has 51..517) 2024-04-14T15:20:42.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.306+0000 7fec5f7fe700 3 client.49345.objecter handle_osd_map got epochs [517,517] > 0 2024-04-14T15:20:42.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.306+0000 7fec5f7fe700 3 client.49345.objecter handle_osd_map decoding full epoch 517 2024-04-14T15:20:42.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.306+0000 7fec5f7fe700 20 client.49345.objecter dump_active .. 0 homeless 2024-04-14T15:20:42.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.306+0000 7fec5f7fe700 10 client.49345.objecter ms_handle_connect 0x7fec54071460 2024-04-14T15:20:42.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.306+0000 7fec5f7fe700 1 -- 172.21.15.90:0/3563185833 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v517) ==== 54+0+16 (unknown 3478234358 0 2420000192) 0x7fec58089d00 con 0x7fec6810d950 2024-04-14T15:20:42.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.306+0000 7fec702f4700 10 client.49345.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:20:42.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.306+0000 7fec702f4700 15 client.49345.objecter _session_command_op_assign -1 1 2024-04-14T15:20:42.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.306+0000 7fec702f4700 20 client.49345.objecter _get_session s=0x7fec4c0053e0 osd=0 3 2024-04-14T15:20:42.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.306+0000 7fec702f4700 20 client.49345.objecter put_session s=0x7fec4c0053e0 osd=0 3 2024-04-14T15:20:42.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.306+0000 7fec702f4700 20 client.49345.objecter _get_session s=0x7fec4c0053e0 osd=0 3 2024-04-14T15:20:42.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.306+0000 7fec702f4700 15 client.49345.objecter _session_command_op_remove -1 1 2024-04-14T15:20:42.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.306+0000 7fec702f4700 20 client.49345.objecter get_session s=0x7fec4c0053e0 osd=0 3 2024-04-14T15:20:42.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.306+0000 7fec702f4700 15 client.49345.objecter _session_command_op_assign 0 1 2024-04-14T15:20:42.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.306+0000 7fec702f4700 20 client.49345.objecter put_session s=0x7fec4c0053e0 osd=0 4 2024-04-14T15:20:42.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.306+0000 7fec702f4700 10 client.49345.objecter _send_command 1 2024-04-14T15:20:42.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.306+0000 7fec702f4700 1 -- 172.21.15.90:0/3563185833 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fec4c004b70 con 0x7fec4c004730 2024-04-14T15:20:42.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.306+0000 7fec5f7fe700 10 client.49345.objecter ms_handle_connect 0x7fec4c004730 2024-04-14T15:20:42.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.307+0000 7fec5f7fe700 1 -- 172.21.15.90:0/3563185833 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fec4c004b70 con 0x7fec4c004730 2024-04-14T15:20:42.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.307+0000 7fec5f7fe700 10 client.49345.objecter ms_dispatch 0x7fec68001550 command_reply(tid 1: 0 ) 2024-04-14T15:20:42.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.307+0000 7fec5f7fe700 10 client.49345.objecter _finish_command 1 = system:0 2024-04-14T15:20:42.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.307+0000 7fec5f7fe700 20 client.49345.objecter put_session s=0x7fec4c0053e0 osd=0 4 2024-04-14T15:20:42.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.307+0000 7fec5f7fe700 15 client.49345.objecter _session_command_op_remove 0 1 2024-04-14T15:20:42.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.329+0000 7fec702f4700 10 client.49345.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:20:42.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.329+0000 7fec702f4700 15 client.49345.objecter _session_command_op_assign -1 2 2024-04-14T15:20:42.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.329+0000 7fec702f4700 20 client.49345.objecter _get_session s=0x7fec4c0053e0 osd=0 3 2024-04-14T15:20:42.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.329+0000 7fec702f4700 20 client.49345.objecter put_session s=0x7fec4c0053e0 osd=0 3 2024-04-14T15:20:42.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.329+0000 7fec702f4700 20 client.49345.objecter _get_session s=0x7fec4c0053e0 osd=0 3 2024-04-14T15:20:42.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.329+0000 7fec702f4700 15 client.49345.objecter _session_command_op_remove -1 2 2024-04-14T15:20:42.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.329+0000 7fec702f4700 20 client.49345.objecter get_session s=0x7fec4c0053e0 osd=0 3 2024-04-14T15:20:42.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.329+0000 7fec702f4700 15 client.49345.objecter _session_command_op_assign 0 2 2024-04-14T15:20:42.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.329+0000 7fec702f4700 20 client.49345.objecter put_session s=0x7fec4c0053e0 osd=0 4 2024-04-14T15:20:42.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.329+0000 7fec702f4700 10 client.49345.objecter _send_command 2 2024-04-14T15:20:42.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.329+0000 7fec702f4700 1 -- 172.21.15.90:0/3563185833 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fec4c0079f0 con 0x7fec4c004730 2024-04-14T15:20:42.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.331+0000 7fec5f7fe700 1 -- 172.21.15.90:0/3563185833 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fec4c0079f0 con 0x7fec4c004730 2024-04-14T15:20:42.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.331+0000 7fec5f7fe700 10 client.49345.objecter ms_dispatch 0x7fec68001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:20:42.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.331+0000 7fec5f7fe700 10 client.49345.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:20:42.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.331+0000 7fec5f7fe700 20 client.49345.objecter put_session s=0x7fec4c0053e0 osd=0 4 2024-04-14T15:20:42.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.331+0000 7fec5f7fe700 15 client.49345.objecter _session_command_op_remove 0 2 2024-04-14T15:20:42.332 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:20:42.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.331+0000 7fec702f4700 10 client.49345.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:20:42.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.331+0000 7fec702f4700 15 client.49345.objecter _session_command_op_assign -1 3 2024-04-14T15:20:42.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.331+0000 7fec702f4700 20 client.49345.objecter _get_session s=0x7fec4c004130 osd=1 3 2024-04-14T15:20:42.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.331+0000 7fec702f4700 20 client.49345.objecter put_session s=0x7fec4c004130 osd=1 3 2024-04-14T15:20:42.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.331+0000 7fec702f4700 20 client.49345.objecter _get_session s=0x7fec4c004130 osd=1 3 2024-04-14T15:20:42.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.331+0000 7fec702f4700 15 client.49345.objecter _session_command_op_remove -1 3 2024-04-14T15:20:42.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.331+0000 7fec702f4700 20 client.49345.objecter get_session s=0x7fec4c004130 osd=1 3 2024-04-14T15:20:42.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.331+0000 7fec702f4700 15 client.49345.objecter _session_command_op_assign 1 3 2024-04-14T15:20:42.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.331+0000 7fec702f4700 20 client.49345.objecter put_session s=0x7fec4c004130 osd=1 4 2024-04-14T15:20:42.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.331+0000 7fec702f4700 10 client.49345.objecter _send_command 3 2024-04-14T15:20:42.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.331+0000 7fec702f4700 1 -- 172.21.15.90:0/3563185833 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fec4c0022a0 con 0x7fec4c004310 2024-04-14T15:20:42.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.332+0000 7fec5f7fe700 10 client.49345.objecter ms_handle_connect 0x7fec4c004310 2024-04-14T15:20:42.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.333+0000 7fec5f7fe700 1 -- 172.21.15.90:0/3563185833 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fec4c0022a0 con 0x7fec4c004310 2024-04-14T15:20:42.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.333+0000 7fec5f7fe700 10 client.49345.objecter ms_dispatch 0x7fec68001550 command_reply(tid 3: 0 ) 2024-04-14T15:20:42.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.333+0000 7fec5f7fe700 10 client.49345.objecter _finish_command 3 = system:0 2024-04-14T15:20:42.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.333+0000 7fec5f7fe700 20 client.49345.objecter put_session s=0x7fec4c004130 osd=1 4 2024-04-14T15:20:42.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.333+0000 7fec5f7fe700 15 client.49345.objecter _session_command_op_remove 1 3 2024-04-14T15:20:42.334 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:20:42.346 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:20:42.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.350+0000 7fec702f4700 10 client.49345.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:20:42.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.350+0000 7fec702f4700 15 client.49345.objecter _session_command_op_assign -1 4 2024-04-14T15:20:42.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.350+0000 7fec702f4700 20 client.49345.objecter _get_session s=0x7fec4c004130 osd=1 3 2024-04-14T15:20:42.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.350+0000 7fec702f4700 20 client.49345.objecter put_session s=0x7fec4c004130 osd=1 3 2024-04-14T15:20:42.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.350+0000 7fec702f4700 20 client.49345.objecter _get_session s=0x7fec4c004130 osd=1 3 2024-04-14T15:20:42.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.350+0000 7fec702f4700 15 client.49345.objecter _session_command_op_remove -1 4 2024-04-14T15:20:42.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.350+0000 7fec702f4700 20 client.49345.objecter get_session s=0x7fec4c004130 osd=1 3 2024-04-14T15:20:42.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.350+0000 7fec702f4700 15 client.49345.objecter _session_command_op_assign 1 4 2024-04-14T15:20:42.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.350+0000 7fec702f4700 20 client.49345.objecter put_session s=0x7fec4c004130 osd=1 4 2024-04-14T15:20:42.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.350+0000 7fec702f4700 10 client.49345.objecter _send_command 4 2024-04-14T15:20:42.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.350+0000 7fec702f4700 1 -- 172.21.15.90:0/3563185833 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fec4c001720 con 0x7fec4c004310 2024-04-14T15:20:42.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.352+0000 7fec5f7fe700 1 -- 172.21.15.90:0/3563185833 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fec4c001720 con 0x7fec4c004310 2024-04-14T15:20:42.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.352+0000 7fec5f7fe700 10 client.49345.objecter ms_dispatch 0x7fec68001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:20:42.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.352+0000 7fec5f7fe700 10 client.49345.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:20:42.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.352+0000 7fec5f7fe700 20 client.49345.objecter put_session s=0x7fec4c004130 osd=1 4 2024-04-14T15:20:42.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.352+0000 7fec5f7fe700 15 client.49345.objecter _session_command_op_remove 1 4 2024-04-14T15:20:42.353 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:20:42.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.353+0000 7fec702f4700 10 client.49345.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:20:42.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.353+0000 7fec702f4700 15 client.49345.objecter _session_command_op_assign -1 5 2024-04-14T15:20:42.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.353+0000 7fec702f4700 20 client.49345.objecter _get_session s=0x7fec4c001980 osd=2 3 2024-04-14T15:20:42.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.353+0000 7fec702f4700 20 client.49345.objecter put_session s=0x7fec4c001980 osd=2 3 2024-04-14T15:20:42.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.353+0000 7fec702f4700 20 client.49345.objecter _get_session s=0x7fec4c001980 osd=2 3 2024-04-14T15:20:42.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.353+0000 7fec702f4700 15 client.49345.objecter _session_command_op_remove -1 5 2024-04-14T15:20:42.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.353+0000 7fec702f4700 20 client.49345.objecter get_session s=0x7fec4c001980 osd=2 3 2024-04-14T15:20:42.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.353+0000 7fec702f4700 15 client.49345.objecter _session_command_op_assign 2 5 2024-04-14T15:20:42.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.353+0000 7fec702f4700 20 client.49345.objecter put_session s=0x7fec4c001980 osd=2 4 2024-04-14T15:20:42.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.353+0000 7fec702f4700 10 client.49345.objecter _send_command 5 2024-04-14T15:20:42.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.353+0000 7fec702f4700 1 -- 172.21.15.90:0/3563185833 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fec4c011170 con 0x7fec4c00d5d0 2024-04-14T15:20:42.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.353+0000 7fec5f7fe700 10 client.49345.objecter ms_handle_connect 0x7fec4c00d5d0 2024-04-14T15:20:42.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.354+0000 7fec5f7fe700 1 -- 172.21.15.90:0/3563185833 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fec4c011170 con 0x7fec4c00d5d0 2024-04-14T15:20:42.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.354+0000 7fec5f7fe700 10 client.49345.objecter ms_dispatch 0x7fec68001550 command_reply(tid 5: 0 ) 2024-04-14T15:20:42.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.354+0000 7fec5f7fe700 10 client.49345.objecter _finish_command 5 = system:0 2024-04-14T15:20:42.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.354+0000 7fec5f7fe700 20 client.49345.objecter put_session s=0x7fec4c001980 osd=2 4 2024-04-14T15:20:42.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.354+0000 7fec5f7fe700 15 client.49345.objecter _session_command_op_remove 2 5 2024-04-14T15:20:42.366 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:41 smithi092 ceph-mon[33966]: 11.f scrub starts 2024-04-14T15:20:42.366 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:41 smithi092 ceph-mon[33966]: 11.f scrub ok 2024-04-14T15:20:42.366 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:41 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-14T15:20:42.366 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:41 smithi092 ceph-mon[33966]: osdmap e517: 8 total, 8 up, 8 in 2024-04-14T15:20:42.369 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:42.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:42.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:20:42.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.371+0000 7fec702f4700 10 client.49345.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:20:42.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.371+0000 7fec702f4700 15 client.49345.objecter _session_command_op_assign -1 6 2024-04-14T15:20:42.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.371+0000 7fec702f4700 20 client.49345.objecter _get_session s=0x7fec4c001980 osd=2 3 2024-04-14T15:20:42.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.371+0000 7fec702f4700 20 client.49345.objecter put_session s=0x7fec4c001980 osd=2 3 2024-04-14T15:20:42.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.371+0000 7fec702f4700 20 client.49345.objecter _get_session s=0x7fec4c001980 osd=2 3 2024-04-14T15:20:42.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.371+0000 7fec702f4700 15 client.49345.objecter _session_command_op_remove -1 6 2024-04-14T15:20:42.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.371+0000 7fec702f4700 20 client.49345.objecter get_session s=0x7fec4c001980 osd=2 3 2024-04-14T15:20:42.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.371+0000 7fec702f4700 15 client.49345.objecter _session_command_op_assign 2 6 2024-04-14T15:20:42.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.371+0000 7fec702f4700 20 client.49345.objecter put_session s=0x7fec4c001980 osd=2 4 2024-04-14T15:20:42.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.371+0000 7fec702f4700 10 client.49345.objecter _send_command 6 2024-04-14T15:20:42.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.371+0000 7fec702f4700 1 -- 172.21.15.90:0/3563185833 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fec4c008100 con 0x7fec4c00d5d0 2024-04-14T15:20:42.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.373+0000 7fec5f7fe700 1 -- 172.21.15.90:0/3563185833 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fec4c008100 con 0x7fec4c00d5d0 2024-04-14T15:20:42.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.373+0000 7fec5f7fe700 10 client.49345.objecter ms_dispatch 0x7fec68001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:20:42.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.373+0000 7fec5f7fe700 10 client.49345.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:20:42.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.373+0000 7fec5f7fe700 20 client.49345.objecter put_session s=0x7fec4c001980 osd=2 4 2024-04-14T15:20:42.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.373+0000 7fec5f7fe700 15 client.49345.objecter _session_command_op_remove 2 6 2024-04-14T15:20:42.374 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:20:42.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.373+0000 7fec702f4700 10 client.49345.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:20:42.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.373+0000 7fec702f4700 15 client.49345.objecter _session_command_op_assign -1 7 2024-04-14T15:20:42.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.373+0000 7fec702f4700 20 client.49345.objecter _get_session s=0x7fec4c008390 osd=3 3 2024-04-14T15:20:42.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.373+0000 7fec702f4700 20 client.49345.objecter put_session s=0x7fec4c008390 osd=3 3 2024-04-14T15:20:42.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.373+0000 7fec702f4700 20 client.49345.objecter _get_session s=0x7fec4c008390 osd=3 3 2024-04-14T15:20:42.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.373+0000 7fec702f4700 15 client.49345.objecter _session_command_op_remove -1 7 2024-04-14T15:20:42.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.373+0000 7fec702f4700 20 client.49345.objecter get_session s=0x7fec4c008390 osd=3 3 2024-04-14T15:20:42.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.373+0000 7fec702f4700 15 client.49345.objecter _session_command_op_assign 3 7 2024-04-14T15:20:42.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.373+0000 7fec702f4700 20 client.49345.objecter put_session s=0x7fec4c008390 osd=3 4 2024-04-14T15:20:42.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.373+0000 7fec702f4700 10 client.49345.objecter _send_command 7 2024-04-14T15:20:42.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.373+0000 7fec702f4700 1 -- 172.21.15.90:0/3563185833 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fec4c015fb0 con 0x7fec4c0124a0 2024-04-14T15:20:42.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.373+0000 7fec5f7fe700 10 client.49345.objecter ms_handle_connect 0x7fec4c0124a0 2024-04-14T15:20:42.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.374+0000 7fec5f7fe700 1 -- 172.21.15.90:0/3563185833 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fec4c015fb0 con 0x7fec4c0124a0 2024-04-14T15:20:42.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.374+0000 7fec5f7fe700 10 client.49345.objecter ms_dispatch 0x7fec68001550 command_reply(tid 7: 0 ) 2024-04-14T15:20:42.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.374+0000 7fec5f7fe700 10 client.49345.objecter _finish_command 7 = system:0 2024-04-14T15:20:42.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.374+0000 7fec5f7fe700 20 client.49345.objecter put_session s=0x7fec4c008390 osd=3 4 2024-04-14T15:20:42.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.374+0000 7fec5f7fe700 15 client.49345.objecter _session_command_op_remove 3 7 2024-04-14T15:20:42.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.391+0000 7fec702f4700 10 client.49345.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:20:42.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.391+0000 7fec702f4700 15 client.49345.objecter _session_command_op_assign -1 8 2024-04-14T15:20:42.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.391+0000 7fec702f4700 20 client.49345.objecter _get_session s=0x7fec4c008390 osd=3 3 2024-04-14T15:20:42.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.391+0000 7fec702f4700 20 client.49345.objecter put_session s=0x7fec4c008390 osd=3 3 2024-04-14T15:20:42.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.391+0000 7fec702f4700 20 client.49345.objecter _get_session s=0x7fec4c008390 osd=3 3 2024-04-14T15:20:42.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.391+0000 7fec702f4700 15 client.49345.objecter _session_command_op_remove -1 8 2024-04-14T15:20:42.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.391+0000 7fec702f4700 20 client.49345.objecter get_session s=0x7fec4c008390 osd=3 3 2024-04-14T15:20:42.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.391+0000 7fec702f4700 15 client.49345.objecter _session_command_op_assign 3 8 2024-04-14T15:20:42.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.391+0000 7fec702f4700 20 client.49345.objecter put_session s=0x7fec4c008390 osd=3 4 2024-04-14T15:20:42.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.391+0000 7fec702f4700 10 client.49345.objecter _send_command 8 2024-04-14T15:20:42.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.391+0000 7fec702f4700 1 -- 172.21.15.90:0/3563185833 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fec4c00c9b0 con 0x7fec4c0124a0 2024-04-14T15:20:42.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.392+0000 7fec5f7fe700 1 -- 172.21.15.90:0/3563185833 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fec4c00c9b0 con 0x7fec4c0124a0 2024-04-14T15:20:42.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.392+0000 7fec5f7fe700 10 client.49345.objecter ms_dispatch 0x7fec68001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:20:42.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.392+0000 7fec5f7fe700 10 client.49345.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:20:42.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.392+0000 7fec5f7fe700 20 client.49345.objecter put_session s=0x7fec4c008390 osd=3 4 2024-04-14T15:20:42.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.392+0000 7fec5f7fe700 15 client.49345.objecter _session_command_op_remove 3 8 2024-04-14T15:20:42.393 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:20:42.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.392+0000 7fec702f4700 10 client.49345.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:20:42.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.392+0000 7fec702f4700 15 client.49345.objecter _session_command_op_assign -1 9 2024-04-14T15:20:42.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.392+0000 7fec702f4700 20 client.49345.objecter _get_session s=0x7fec4c00c760 osd=4 3 2024-04-14T15:20:42.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.392+0000 7fec702f4700 20 client.49345.objecter put_session s=0x7fec4c00c760 osd=4 3 2024-04-14T15:20:42.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.392+0000 7fec702f4700 20 client.49345.objecter _get_session s=0x7fec4c00c760 osd=4 3 2024-04-14T15:20:42.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.392+0000 7fec702f4700 15 client.49345.objecter _session_command_op_remove -1 9 2024-04-14T15:20:42.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.392+0000 7fec702f4700 20 client.49345.objecter get_session s=0x7fec4c00c760 osd=4 3 2024-04-14T15:20:42.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.392+0000 7fec702f4700 15 client.49345.objecter _session_command_op_assign 4 9 2024-04-14T15:20:42.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.392+0000 7fec702f4700 20 client.49345.objecter put_session s=0x7fec4c00c760 osd=4 4 2024-04-14T15:20:42.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.392+0000 7fec702f4700 10 client.49345.objecter _send_command 9 2024-04-14T15:20:42.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.392+0000 7fec702f4700 1 -- 172.21.15.90:0/3563185833 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fec4c01a860 con 0x7fec4c012000 2024-04-14T15:20:42.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.393+0000 7fec5f7fe700 10 client.49345.objecter ms_handle_connect 0x7fec4c012000 2024-04-14T15:20:42.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.394+0000 7fec5f7fe700 1 -- 172.21.15.90:0/3563185833 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fec4c01a860 con 0x7fec4c012000 2024-04-14T15:20:42.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.394+0000 7fec5f7fe700 10 client.49345.objecter ms_dispatch 0x7fec68001550 command_reply(tid 9: 0 ) 2024-04-14T15:20:42.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.394+0000 7fec5f7fe700 10 client.49345.objecter _finish_command 9 = system:0 2024-04-14T15:20:42.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.394+0000 7fec5f7fe700 20 client.49345.objecter put_session s=0x7fec4c00c760 osd=4 4 2024-04-14T15:20:42.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.394+0000 7fec5f7fe700 15 client.49345.objecter _session_command_op_remove 4 9 2024-04-14T15:20:42.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.410+0000 7fec702f4700 10 client.49345.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:20:42.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.410+0000 7fec702f4700 15 client.49345.objecter _session_command_op_assign -1 10 2024-04-14T15:20:42.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.410+0000 7fec702f4700 20 client.49345.objecter _get_session s=0x7fec4c00c760 osd=4 3 2024-04-14T15:20:42.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.410+0000 7fec702f4700 20 client.49345.objecter put_session s=0x7fec4c00c760 osd=4 3 2024-04-14T15:20:42.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.410+0000 7fec702f4700 20 client.49345.objecter _get_session s=0x7fec4c00c760 osd=4 3 2024-04-14T15:20:42.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.410+0000 7fec702f4700 15 client.49345.objecter _session_command_op_remove -1 10 2024-04-14T15:20:42.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.410+0000 7fec702f4700 20 client.49345.objecter get_session s=0x7fec4c00c760 osd=4 3 2024-04-14T15:20:42.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.410+0000 7fec702f4700 15 client.49345.objecter _session_command_op_assign 4 10 2024-04-14T15:20:42.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.410+0000 7fec702f4700 20 client.49345.objecter put_session s=0x7fec4c00c760 osd=4 4 2024-04-14T15:20:42.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.410+0000 7fec702f4700 10 client.49345.objecter _send_command 10 2024-04-14T15:20:42.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.410+0000 7fec702f4700 1 -- 172.21.15.90:0/3563185833 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fec4c011570 con 0x7fec4c012000 2024-04-14T15:20:42.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.413+0000 7fec5f7fe700 1 -- 172.21.15.90:0/3563185833 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fec4c011570 con 0x7fec4c012000 2024-04-14T15:20:42.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.413+0000 7fec5f7fe700 10 client.49345.objecter ms_dispatch 0x7fec68001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:20:42.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.413+0000 7fec5f7fe700 10 client.49345.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:20:42.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.413+0000 7fec5f7fe700 20 client.49345.objecter put_session s=0x7fec4c00c760 osd=4 4 2024-04-14T15:20:42.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.413+0000 7fec5f7fe700 15 client.49345.objecter _session_command_op_remove 4 10 2024-04-14T15:20:42.414 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:20:42.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.413+0000 7fec702f4700 10 client.49345.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:20:42.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.413+0000 7fec702f4700 15 client.49345.objecter _session_command_op_assign -1 11 2024-04-14T15:20:42.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.413+0000 7fec702f4700 20 client.49345.objecter _get_session s=0x7fec4c011570 osd=5 3 2024-04-14T15:20:42.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.413+0000 7fec702f4700 20 client.49345.objecter put_session s=0x7fec4c011570 osd=5 3 2024-04-14T15:20:42.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.413+0000 7fec702f4700 20 client.49345.objecter _get_session s=0x7fec4c011570 osd=5 3 2024-04-14T15:20:42.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.413+0000 7fec702f4700 15 client.49345.objecter _session_command_op_remove -1 11 2024-04-14T15:20:42.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.413+0000 7fec702f4700 20 client.49345.objecter get_session s=0x7fec4c011570 osd=5 3 2024-04-14T15:20:42.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.413+0000 7fec702f4700 15 client.49345.objecter _session_command_op_assign 5 11 2024-04-14T15:20:42.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.413+0000 7fec702f4700 20 client.49345.objecter put_session s=0x7fec4c011570 osd=5 4 2024-04-14T15:20:42.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.413+0000 7fec702f4700 10 client.49345.objecter _send_command 11 2024-04-14T15:20:42.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.413+0000 7fec702f4700 1 -- 172.21.15.90:0/3563185833 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fec4c016e20 con 0x7fec4c0116e0 2024-04-14T15:20:42.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.414+0000 7fec5f7fe700 10 client.49345.objecter ms_handle_connect 0x7fec4c0116e0 2024-04-14T15:20:42.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.414+0000 7fec5f7fe700 1 -- 172.21.15.90:0/3563185833 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fec4c016e20 con 0x7fec4c0116e0 2024-04-14T15:20:42.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.414+0000 7fec5f7fe700 10 client.49345.objecter ms_dispatch 0x7fec68001550 command_reply(tid 11: 0 ) 2024-04-14T15:20:42.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.414+0000 7fec5f7fe700 10 client.49345.objecter _finish_command 11 = system:0 2024-04-14T15:20:42.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.414+0000 7fec5f7fe700 20 client.49345.objecter put_session s=0x7fec4c011570 osd=5 4 2024-04-14T15:20:42.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.414+0000 7fec5f7fe700 15 client.49345.objecter _session_command_op_remove 5 11 2024-04-14T15:20:42.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.431+0000 7fec702f4700 10 client.49345.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:20:42.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.431+0000 7fec702f4700 15 client.49345.objecter _session_command_op_assign -1 12 2024-04-14T15:20:42.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.431+0000 7fec702f4700 20 client.49345.objecter _get_session s=0x7fec4c011570 osd=5 3 2024-04-14T15:20:42.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.431+0000 7fec702f4700 20 client.49345.objecter put_session s=0x7fec4c011570 osd=5 3 2024-04-14T15:20:42.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.431+0000 7fec702f4700 20 client.49345.objecter _get_session s=0x7fec4c011570 osd=5 3 2024-04-14T15:20:42.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.431+0000 7fec702f4700 15 client.49345.objecter _session_command_op_remove -1 12 2024-04-14T15:20:42.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.431+0000 7fec702f4700 20 client.49345.objecter get_session s=0x7fec4c011570 osd=5 3 2024-04-14T15:20:42.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.431+0000 7fec702f4700 15 client.49345.objecter _session_command_op_assign 5 12 2024-04-14T15:20:42.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.431+0000 7fec702f4700 20 client.49345.objecter put_session s=0x7fec4c011570 osd=5 4 2024-04-14T15:20:42.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.431+0000 7fec702f4700 10 client.49345.objecter _send_command 12 2024-04-14T15:20:42.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.431+0000 7fec702f4700 1 -- 172.21.15.90:0/3563185833 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fec4c01f110 con 0x7fec4c0116e0 2024-04-14T15:20:42.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.432+0000 7fec5f7fe700 1 -- 172.21.15.90:0/3563185833 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fec4c01f110 con 0x7fec4c0116e0 2024-04-14T15:20:42.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.432+0000 7fec5f7fe700 10 client.49345.objecter ms_dispatch 0x7fec68001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:20:42.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.432+0000 7fec5f7fe700 10 client.49345.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:20:42.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.432+0000 7fec5f7fe700 20 client.49345.objecter put_session s=0x7fec4c011570 osd=5 4 2024-04-14T15:20:42.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.432+0000 7fec5f7fe700 15 client.49345.objecter _session_command_op_remove 5 12 2024-04-14T15:20:42.434 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:20:42.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.433+0000 7fec702f4700 10 client.49345.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:20:42.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.433+0000 7fec702f4700 15 client.49345.objecter _session_command_op_assign -1 13 2024-04-14T15:20:42.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.433+0000 7fec702f4700 20 client.49345.objecter _get_session s=0x7fec4c016e80 osd=6 3 2024-04-14T15:20:42.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.433+0000 7fec702f4700 20 client.49345.objecter put_session s=0x7fec4c016e80 osd=6 3 2024-04-14T15:20:42.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.433+0000 7fec702f4700 20 client.49345.objecter _get_session s=0x7fec4c016e80 osd=6 3 2024-04-14T15:20:42.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.433+0000 7fec702f4700 15 client.49345.objecter _session_command_op_remove -1 13 2024-04-14T15:20:42.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.433+0000 7fec702f4700 20 client.49345.objecter get_session s=0x7fec4c016e80 osd=6 3 2024-04-14T15:20:42.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.433+0000 7fec702f4700 15 client.49345.objecter _session_command_op_assign 6 13 2024-04-14T15:20:42.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.433+0000 7fec702f4700 20 client.49345.objecter put_session s=0x7fec4c016e80 osd=6 4 2024-04-14T15:20:42.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.433+0000 7fec702f4700 10 client.49345.objecter _send_command 13 2024-04-14T15:20:42.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.433+0000 7fec702f4700 1 -- 172.21.15.90:0/3563185833 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fec4c023370 con 0x7fec4c01f830 2024-04-14T15:20:42.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.433+0000 7fec5f7fe700 10 client.49345.objecter ms_handle_connect 0x7fec4c01f830 2024-04-14T15:20:42.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.434+0000 7fec5f7fe700 1 -- 172.21.15.90:0/3563185833 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fec4c023370 con 0x7fec4c01f830 2024-04-14T15:20:42.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.434+0000 7fec5f7fe700 10 client.49345.objecter ms_dispatch 0x7fec68001550 command_reply(tid 13: 0 ) 2024-04-14T15:20:42.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.434+0000 7fec5f7fe700 10 client.49345.objecter _finish_command 13 = system:0 2024-04-14T15:20:42.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.434+0000 7fec5f7fe700 20 client.49345.objecter put_session s=0x7fec4c016e80 osd=6 4 2024-04-14T15:20:42.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.434+0000 7fec5f7fe700 15 client.49345.objecter _session_command_op_remove 6 13 2024-04-14T15:20:42.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.451+0000 7fec702f4700 10 client.49345.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:20:42.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.451+0000 7fec702f4700 15 client.49345.objecter _session_command_op_assign -1 14 2024-04-14T15:20:42.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.451+0000 7fec702f4700 20 client.49345.objecter _get_session s=0x7fec4c016e80 osd=6 3 2024-04-14T15:20:42.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.451+0000 7fec702f4700 20 client.49345.objecter put_session s=0x7fec4c016e80 osd=6 3 2024-04-14T15:20:42.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.451+0000 7fec702f4700 20 client.49345.objecter _get_session s=0x7fec4c016e80 osd=6 3 2024-04-14T15:20:42.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.451+0000 7fec702f4700 15 client.49345.objecter _session_command_op_remove -1 14 2024-04-14T15:20:42.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.451+0000 7fec702f4700 20 client.49345.objecter get_session s=0x7fec4c016e80 osd=6 3 2024-04-14T15:20:42.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.451+0000 7fec702f4700 15 client.49345.objecter _session_command_op_assign 6 14 2024-04-14T15:20:42.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.451+0000 7fec702f4700 20 client.49345.objecter put_session s=0x7fec4c016e80 osd=6 4 2024-04-14T15:20:42.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.451+0000 7fec702f4700 10 client.49345.objecter _send_command 14 2024-04-14T15:20:42.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.451+0000 7fec702f4700 1 -- 172.21.15.90:0/3563185833 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fec4c023370 con 0x7fec4c01f830 2024-04-14T15:20:42.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.453+0000 7fec5f7fe700 1 -- 172.21.15.90:0/3563185833 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fec4c023370 con 0x7fec4c01f830 2024-04-14T15:20:42.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.453+0000 7fec5f7fe700 10 client.49345.objecter ms_dispatch 0x7fec68001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:20:42.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.453+0000 7fec5f7fe700 10 client.49345.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:20:42.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.453+0000 7fec5f7fe700 20 client.49345.objecter put_session s=0x7fec4c016e80 osd=6 4 2024-04-14T15:20:42.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.453+0000 7fec5f7fe700 15 client.49345.objecter _session_command_op_remove 6 14 2024-04-14T15:20:42.454 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:20:42.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.453+0000 7fec702f4700 10 client.49345.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:20:42.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.453+0000 7fec702f4700 15 client.49345.objecter _session_command_op_assign -1 15 2024-04-14T15:20:42.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.453+0000 7fec702f4700 20 client.49345.objecter _get_session s=0x7fec4c023370 osd=7 3 2024-04-14T15:20:42.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.453+0000 7fec702f4700 20 client.49345.objecter put_session s=0x7fec4c023370 osd=7 3 2024-04-14T15:20:42.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.453+0000 7fec702f4700 20 client.49345.objecter _get_session s=0x7fec4c023370 osd=7 3 2024-04-14T15:20:42.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.453+0000 7fec702f4700 15 client.49345.objecter _session_command_op_remove -1 15 2024-04-14T15:20:42.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.453+0000 7fec702f4700 20 client.49345.objecter get_session s=0x7fec4c023370 osd=7 3 2024-04-14T15:20:42.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.453+0000 7fec702f4700 15 client.49345.objecter _session_command_op_assign 7 15 2024-04-14T15:20:42.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.453+0000 7fec702f4700 20 client.49345.objecter put_session s=0x7fec4c023370 osd=7 4 2024-04-14T15:20:42.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.453+0000 7fec702f4700 10 client.49345.objecter _send_command 15 2024-04-14T15:20:42.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.453+0000 7fec702f4700 1 -- 172.21.15.90:0/3563185833 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fec4c0245d0 con 0x7fec4c025290 2024-04-14T15:20:42.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.453+0000 7fec5f7fe700 10 client.49345.objecter ms_handle_connect 0x7fec4c025290 2024-04-14T15:20:42.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.454+0000 7fec5f7fe700 1 -- 172.21.15.90:0/3563185833 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fec4c0245d0 con 0x7fec4c025290 2024-04-14T15:20:42.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.454+0000 7fec5f7fe700 10 client.49345.objecter ms_dispatch 0x7fec68001550 command_reply(tid 15: 0 ) 2024-04-14T15:20:42.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.454+0000 7fec5f7fe700 10 client.49345.objecter _finish_command 15 = system:0 2024-04-14T15:20:42.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.454+0000 7fec5f7fe700 20 client.49345.objecter put_session s=0x7fec4c023370 osd=7 4 2024-04-14T15:20:42.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.454+0000 7fec5f7fe700 15 client.49345.objecter _session_command_op_remove 7 15 2024-04-14T15:20:42.471 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:20:42.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.471+0000 7fec702f4700 10 client.49345.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:20:42.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.471+0000 7fec702f4700 15 client.49345.objecter _session_command_op_assign -1 16 2024-04-14T15:20:42.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.471+0000 7fec702f4700 20 client.49345.objecter _get_session s=0x7fec4c023370 osd=7 3 2024-04-14T15:20:42.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.471+0000 7fec702f4700 20 client.49345.objecter put_session s=0x7fec4c023370 osd=7 3 2024-04-14T15:20:42.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.471+0000 7fec702f4700 20 client.49345.objecter _get_session s=0x7fec4c023370 osd=7 3 2024-04-14T15:20:42.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.471+0000 7fec702f4700 15 client.49345.objecter _session_command_op_remove -1 16 2024-04-14T15:20:42.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.471+0000 7fec702f4700 20 client.49345.objecter get_session s=0x7fec4c023370 osd=7 3 2024-04-14T15:20:42.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.471+0000 7fec702f4700 15 client.49345.objecter _session_command_op_assign 7 16 2024-04-14T15:20:42.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.471+0000 7fec702f4700 20 client.49345.objecter put_session s=0x7fec4c023370 osd=7 4 2024-04-14T15:20:42.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.471+0000 7fec702f4700 10 client.49345.objecter _send_command 16 2024-04-14T15:20:42.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.471+0000 7fec702f4700 1 -- 172.21.15.90:0/3563185833 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fec4c0245d0 con 0x7fec4c025290 2024-04-14T15:20:42.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.472+0000 7fec5f7fe700 1 -- 172.21.15.90:0/3563185833 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fec4c0245d0 con 0x7fec4c025290 2024-04-14T15:20:42.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.472+0000 7fec5f7fe700 10 client.49345.objecter ms_dispatch 0x7fec68001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:20:42.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.472+0000 7fec5f7fe700 10 client.49345.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:20:42.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.472+0000 7fec5f7fe700 20 client.49345.objecter put_session s=0x7fec4c023370 osd=7 4 2024-04-14T15:20:42.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.472+0000 7fec5f7fe700 15 client.49345.objecter _session_command_op_remove 7 16 2024-04-14T15:20:42.474 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:20:42.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.473+0000 7fec702f4700 10 librados: watch_flush enter 2024-04-14T15:20:42.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.473+0000 7fec702f4700 10 librados: watch_flush exit 2024-04-14T15:20:42.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.473+0000 7fec702f4700 10 client.49345.objecter close_session for osd.0 2024-04-14T15:20:42.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.473+0000 7fec702f4700 1 -- 172.21.15.90:0/3563185833 >> v1:172.21.15.90:6801/3761806885 conn(0x7fec4c004730 legacy=0x7fec4c002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:42.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.473+0000 7fec702f4700 20 client.49345.objecter put_session s=0x7fec4c0053e0 osd=0 1 2024-04-14T15:20:42.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.473+0000 7fec702f4700 10 client.49345.objecter close_session for osd.1 2024-04-14T15:20:42.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.473+0000 7fec702f4700 1 -- 172.21.15.90:0/3563185833 >> v1:172.21.15.90:6805/4170361699 conn(0x7fec4c004310 legacy=0x7fec4c00ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:42.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.473+0000 7fec702f4700 20 client.49345.objecter put_session s=0x7fec4c004130 osd=1 1 2024-04-14T15:20:42.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.473+0000 7fec702f4700 10 client.49345.objecter close_session for osd.2 2024-04-14T15:20:42.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.473+0000 7fec702f4700 1 -- 172.21.15.90:0/3563185833 >> v1:172.21.15.90:6809/1686184251 conn(0x7fec4c00d5d0 legacy=0x7fec4c00f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:42.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.473+0000 7fec702f4700 20 client.49345.objecter put_session s=0x7fec4c001980 osd=2 1 2024-04-14T15:20:42.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.473+0000 7fec702f4700 10 client.49345.objecter close_session for osd.3 2024-04-14T15:20:42.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.473+0000 7fec702f4700 1 -- 172.21.15.90:0/3563185833 >> v1:172.21.15.90:6813/2270166774 conn(0x7fec4c0124a0 legacy=0x7fec4c014890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:42.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.473+0000 7fec702f4700 20 client.49345.objecter put_session s=0x7fec4c008390 osd=3 1 2024-04-14T15:20:42.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.473+0000 7fec702f4700 10 client.49345.objecter close_session for osd.4 2024-04-14T15:20:42.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.473+0000 7fec6e891700 1 -- 172.21.15.90:0/3563185833 reap_dead start 2024-04-14T15:20:42.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.473+0000 7fec702f4700 1 -- 172.21.15.90:0/3563185833 >> v1:172.21.15.92:6800/3887549733 conn(0x7fec4c012000 legacy=0x7fec4c019140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:42.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.473+0000 7fec6e891700 1 -- 172.21.15.90:0/3563185833 reap_dead start 2024-04-14T15:20:42.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.473+0000 7fec702f4700 20 client.49345.objecter put_session s=0x7fec4c00c760 osd=4 1 2024-04-14T15:20:42.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.473+0000 7fec702f4700 10 client.49345.objecter close_session for osd.5 2024-04-14T15:20:42.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.473+0000 7fec702f4700 1 -- 172.21.15.90:0/3563185833 >> v1:172.21.15.92:6804/3244561440 conn(0x7fec4c0116e0 legacy=0x7fec4c01d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:42.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.473+0000 7fec702f4700 20 client.49345.objecter put_session s=0x7fec4c011570 osd=5 1 2024-04-14T15:20:42.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.473+0000 7fec702f4700 10 client.49345.objecter close_session for osd.6 2024-04-14T15:20:42.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.473+0000 7fec702f4700 1 -- 172.21.15.90:0/3563185833 >> v1:172.21.15.92:6808/636125963 conn(0x7fec4c01f830 legacy=0x7fec4c021c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:42.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.473+0000 7fec702f4700 20 client.49345.objecter put_session s=0x7fec4c016e80 osd=6 1 2024-04-14T15:20:42.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.473+0000 7fec702f4700 10 client.49345.objecter close_session for osd.7 2024-04-14T15:20:42.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.473+0000 7fec702f4700 1 -- 172.21.15.90:0/3563185833 >> v1:172.21.15.92:6812/1773706716 conn(0x7fec4c025290 legacy=0x7fec4c0256f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:42.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.473+0000 7fec702f4700 20 client.49345.objecter put_session s=0x7fec4c023370 osd=7 1 2024-04-14T15:20:42.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.473+0000 7fec702f4700 20 client.49345.objecter shutdown clearing up homeless session... 2024-04-14T15:20:42.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.473+0000 7fec702f4700 10 client.49345.objecter successfully canceled tick 2024-04-14T15:20:42.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.473+0000 7fec702f4700 1 -- 172.21.15.90:0/3563185833 >> v1:172.21.15.90:6800/420935427 conn(0x7fec54071460 legacy=0x7fec54073920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:42.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.473+0000 7fec702f4700 1 -- 172.21.15.90:0/3563185833 >> v1:172.21.15.92:6789/0 conn(0x7fec6810d950 legacy=0x7fec681a4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:42.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.473+0000 7fec6e891700 1 -- 172.21.15.90:0/3563185833 reap_dead start 2024-04-14T15:20:42.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.473+0000 7fec6e891700 1 -- 172.21.15.90:0/3563185833 reap_dead start 2024-04-14T15:20:42.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.473+0000 7fec702f4700 1 -- 172.21.15.90:0/3563185833 shutdown_connections 2024-04-14T15:20:42.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.473+0000 7fec702f4700 1 -- 172.21.15.90:0/3563185833 >> 172.21.15.90:0/3563185833 conn(0x7fec68076aa0 msgr2=0x7fec68102b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:20:42.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.473+0000 7fec702f4700 1 -- 172.21.15.90:0/3563185833 shutdown_connections 2024-04-14T15:20:42.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.473+0000 7fec702f4700 1 -- 172.21.15.90:0/3563185833 wait complete. 2024-04-14T15:20:42.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:42.473+0000 7fec702f4700 1 librados: shutdown 2024-04-14T15:20:42.478 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:20:42.478 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:20:42.479 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:20:42.479 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:20:42.479 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:20:42.479 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:20:42.479 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:20:42.479 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:20:42.497 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:42.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:42.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:20:42.598 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:20:42.623 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:42.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:42.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:20:42.724 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:20:42.747 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:42.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:42.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:20:42.849 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:20:42.872 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:42.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:42.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:20:42.965 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-04-14T15:20:42.965 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:20:42.973 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:42.998 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:42.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:42.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:43.099 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:20:43.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:43.124 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:43.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:20:43.224 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:20:43.249 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:43.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:43.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:20:43.251 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:42 smithi090 ceph-mon[30399]: pgmap v749: 257 pgs: 257 active+clean; 8.4 MiB data, 593 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.2 KiB/s wr, 11 op/s 2024-04-14T15:20:43.251 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:42 smithi090 ceph-mon[30399]: 110.13 deep-scrub starts 2024-04-14T15:20:43.251 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:42 smithi090 ceph-mon[30399]: 110.13 deep-scrub ok 2024-04-14T15:20:43.251 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:42 smithi090 ceph-mon[30399]: 2.7 scrub starts 2024-04-14T15:20:43.251 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:42 smithi090 ceph-mon[30399]: 2.7 scrub ok 2024-04-14T15:20:43.251 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:42 smithi090 ceph-mon[30399]: 4.17 scrub starts 2024-04-14T15:20:43.251 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:42 smithi090 ceph-mon[30399]: 4.17 scrub ok 2024-04-14T15:20:43.251 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:42 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3563185833' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:20:43.252 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:42 smithi090 ceph-mon[30399]: 6.18 scrub starts 2024-04-14T15:20:43.252 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:42 smithi090 ceph-mon[30399]: 6.18 scrub ok 2024-04-14T15:20:43.252 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:42 smithi090 ceph-mon[30399]: osdmap e518: 8 total, 8 up, 8 in 2024-04-14T15:20:43.252 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:42 smithi090 ceph-mon[41670]: pgmap v749: 257 pgs: 257 active+clean; 8.4 MiB data, 593 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.2 KiB/s wr, 11 op/s 2024-04-14T15:20:43.252 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:42 smithi090 ceph-mon[41670]: 110.13 deep-scrub starts 2024-04-14T15:20:43.252 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:42 smithi090 ceph-mon[41670]: 110.13 deep-scrub ok 2024-04-14T15:20:43.252 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:42 smithi090 ceph-mon[41670]: 2.7 scrub starts 2024-04-14T15:20:43.252 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:42 smithi090 ceph-mon[41670]: 2.7 scrub ok 2024-04-14T15:20:43.252 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:42 smithi090 ceph-mon[41670]: 4.17 scrub starts 2024-04-14T15:20:43.252 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:42 smithi090 ceph-mon[41670]: 4.17 scrub ok 2024-04-14T15:20:43.252 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:42 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3563185833' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:20:43.252 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:42 smithi090 ceph-mon[41670]: 6.18 scrub starts 2024-04-14T15:20:43.252 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:42 smithi090 ceph-mon[41670]: 6.18 scrub ok 2024-04-14T15:20:43.252 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:42 smithi090 ceph-mon[41670]: osdmap e518: 8 total, 8 up, 8 in 2024-04-14T15:20:43.290 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:42 smithi092 ceph-mon[33966]: pgmap v749: 257 pgs: 257 active+clean; 8.4 MiB data, 593 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.2 KiB/s wr, 11 op/s 2024-04-14T15:20:43.290 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:42 smithi092 ceph-mon[33966]: 110.13 deep-scrub starts 2024-04-14T15:20:43.290 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:42 smithi092 ceph-mon[33966]: 110.13 deep-scrub ok 2024-04-14T15:20:43.290 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:42 smithi092 ceph-mon[33966]: 2.7 scrub starts 2024-04-14T15:20:43.290 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:42 smithi092 ceph-mon[33966]: 2.7 scrub ok 2024-04-14T15:20:43.290 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:42 smithi092 ceph-mon[33966]: 4.17 scrub starts 2024-04-14T15:20:43.290 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:42 smithi092 ceph-mon[33966]: 4.17 scrub ok 2024-04-14T15:20:43.290 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:42 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3563185833' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:20:43.291 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:42 smithi092 ceph-mon[33966]: 6.18 scrub starts 2024-04-14T15:20:43.291 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:42 smithi092 ceph-mon[33966]: 6.18 scrub ok 2024-04-14T15:20:43.291 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:42 smithi092 ceph-mon[33966]: osdmap e518: 8 total, 8 up, 8 in 2024-04-14T15:20:43.351 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:20:43.374 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:43.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:43.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:20:43.476 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:43.499 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:43.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:43.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:43.601 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:20:43.625 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-14T15:20:43.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:43.626 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:43.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:20:43.727 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:43.750 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:43.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:43.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:43.851 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:20:43.876 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:43.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:43.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:20:43.903 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:20:43.978 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:20:44.005 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:44.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:44.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:20:44.107 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:20:44.130 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:44.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:44.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:20:44.191 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:20:43 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:20:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:20:44.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:43 smithi090 ceph-mon[41670]: 2.0 deep-scrub starts 2024-04-14T15:20:44.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:43 smithi090 ceph-mon[41670]: 2.0 deep-scrub ok 2024-04-14T15:20:44.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:43 smithi090 ceph-mon[41670]: 2.1a deep-scrub starts 2024-04-14T15:20:44.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:43 smithi090 ceph-mon[41670]: 2.1a deep-scrub ok 2024-04-14T15:20:44.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:43 smithi090 ceph-mon[41670]: 110.16 deep-scrub starts 2024-04-14T15:20:44.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:43 smithi090 ceph-mon[41670]: 110.16 deep-scrub ok 2024-04-14T15:20:44.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:43 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-04-14T15:20:44.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:43 smithi090 ceph-mon[41670]: 6.9 scrub starts 2024-04-14T15:20:44.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:43 smithi090 ceph-mon[41670]: 6.9 scrub ok 2024-04-14T15:20:44.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:43 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "28"}]': finished 2024-04-14T15:20:44.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:43 smithi090 ceph-mon[41670]: osdmap e519: 8 total, 8 up, 8 in 2024-04-14T15:20:44.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:43 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:20:44.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:43 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:20:44.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:43 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-83"}]: dispatch 2024-04-14T15:20:44.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:43 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-83"}]: dispatch 2024-04-14T15:20:44.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:43 smithi090 ceph-mon[30399]: 2.0 deep-scrub starts 2024-04-14T15:20:44.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:43 smithi090 ceph-mon[30399]: 2.0 deep-scrub ok 2024-04-14T15:20:44.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:43 smithi090 ceph-mon[30399]: 2.1a deep-scrub starts 2024-04-14T15:20:44.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:43 smithi090 ceph-mon[30399]: 2.1a deep-scrub ok 2024-04-14T15:20:44.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:43 smithi090 ceph-mon[30399]: 110.16 deep-scrub starts 2024-04-14T15:20:44.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:43 smithi090 ceph-mon[30399]: 110.16 deep-scrub ok 2024-04-14T15:20:44.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:43 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-04-14T15:20:44.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:43 smithi090 ceph-mon[30399]: 6.9 scrub starts 2024-04-14T15:20:44.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:43 smithi090 ceph-mon[30399]: 6.9 scrub ok 2024-04-14T15:20:44.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:43 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "28"}]': finished 2024-04-14T15:20:44.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:43 smithi090 ceph-mon[30399]: osdmap e519: 8 total, 8 up, 8 in 2024-04-14T15:20:44.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:43 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:20:44.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:43 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:20:44.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:43 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-83"}]: dispatch 2024-04-14T15:20:44.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:43 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-83"}]: dispatch 2024-04-14T15:20:44.232 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:20:44.256 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:44.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:44.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:20:44.309 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:43 smithi092 ceph-mon[33966]: 2.0 deep-scrub starts 2024-04-14T15:20:44.309 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:43 smithi092 ceph-mon[33966]: 2.0 deep-scrub ok 2024-04-14T15:20:44.310 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:43 smithi092 ceph-mon[33966]: 2.1a deep-scrub starts 2024-04-14T15:20:44.310 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:43 smithi092 ceph-mon[33966]: 2.1a deep-scrub ok 2024-04-14T15:20:44.310 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:43 smithi092 ceph-mon[33966]: 110.16 deep-scrub starts 2024-04-14T15:20:44.310 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:43 smithi092 ceph-mon[33966]: 110.16 deep-scrub ok 2024-04-14T15:20:44.310 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:43 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-04-14T15:20:44.310 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:43 smithi092 ceph-mon[33966]: 6.9 scrub starts 2024-04-14T15:20:44.310 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:43 smithi092 ceph-mon[33966]: 6.9 scrub ok 2024-04-14T15:20:44.310 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:43 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "28"}]': finished 2024-04-14T15:20:44.310 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:43 smithi092 ceph-mon[33966]: osdmap e519: 8 total, 8 up, 8 in 2024-04-14T15:20:44.310 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:43 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:20:44.310 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:43 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:20:44.310 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:43 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-83"}]: dispatch 2024-04-14T15:20:44.310 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:43 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-83"}]: dispatch 2024-04-14T15:20:44.358 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:20:44.381 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:44.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:44.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:20:44.483 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:20:44.506 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:44.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:44.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:20:44.608 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:20:44.632 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:44.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:44.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:20:44.733 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:20:44.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:44.759 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:44.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:20:44.859 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:20:44.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:44.859+0000 7ff47cd9d700 1 -- 172.21.15.90:0/3382158520 >> v1:172.21.15.90:6789/0 conn(0x7ff47810e200 legacy=0x7ff4781106c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:44.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:44.859+0000 7ff47cd9d700 1 -- 172.21.15.90:0/3382158520 shutdown_connections 2024-04-14T15:20:44.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:44.859+0000 7ff47cd9d700 1 -- 172.21.15.90:0/3382158520 >> 172.21.15.90:0/3382158520 conn(0x7ff4780fd2b0 msgr2=0x7ff4780ff6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:20:44.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:44.859+0000 7ff47cd9d700 1 -- 172.21.15.90:0/3382158520 shutdown_connections 2024-04-14T15:20:44.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:44.859+0000 7ff47cd9d700 1 -- 172.21.15.90:0/3382158520 wait complete. 2024-04-14T15:20:44.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:44.860+0000 7ff47cd9d700 1 librados: starting msgr at 2024-04-14T15:20:44.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:44.860+0000 7ff47cd9d700 1 librados: starting objecter 2024-04-14T15:20:44.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:44.860+0000 7ff47cd9d700 1 Processor -- start 2024-04-14T15:20:44.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:44.860+0000 7ff47cd9d700 1 -- start start 2024-04-14T15:20:44.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:44.860+0000 7ff47cd9d700 1 librados: setting wanted keys 2024-04-14T15:20:44.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:44.860+0000 7ff47cd9d700 1 librados: calling monclient init 2024-04-14T15:20:44.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:44.860+0000 7ff47cd9d700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff47810de90 con 0x7ff47810e200 2024-04-14T15:20:44.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:44.860+0000 7ff47cd9d700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff4781a7e10 con 0x7ff47810a630 2024-04-14T15:20:44.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:44.860+0000 7ff47cd9d700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff4781a8ff0 con 0x7ff478107780 2024-04-14T15:20:44.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:44.860+0000 7ff47659c700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7ff478107780 0x7ff47810c760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:38148/0 (socket says 172.21.15.90:38148) 2024-04-14T15:20:44.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:44.860+0000 7ff47659c700 1 -- 172.21.15.90:0/1376112303 learned_addr learned my addr 172.21.15.90:0/1376112303 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:20:44.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:44.860+0000 7ff476d9d700 1 --1- 172.21.15.90:0/1376112303 >> v1:172.21.15.90:6789/0 conn(0x7ff47810e200 0x7ff4781a56e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:40580/0 (socket says 172.21.15.90:40580) 2024-04-14T15:20:44.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:44.861+0000 7ff4677fe700 10 client.?.objecter ms_handle_connect 0x7ff47810e200 2024-04-14T15:20:44.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:44.861+0000 7ff4677fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:20:44.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:44.861+0000 7ff4677fe700 10 client.?.objecter ms_handle_connect 0x7ff478107780 2024-04-14T15:20:44.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:44.861+0000 7ff4677fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:20:44.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:44.861+0000 7ff4677fe700 10 client.?.objecter ms_handle_connect 0x7ff47810a630 2024-04-14T15:20:44.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:44.861+0000 7ff4677fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:20:44.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:44.861+0000 7ff4677fe700 1 -- 172.21.15.90:0/1376112303 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 942051587 0 0) 0x7ff4781a8ff0 con 0x7ff478107780 2024-04-14T15:20:44.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:44.861+0000 7ff4677fe700 1 -- 172.21.15.90:0/1376112303 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff4600035b0 con 0x7ff478107780 2024-04-14T15:20:44.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:44.861+0000 7ff4677fe700 1 -- 172.21.15.90:0/1376112303 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 257218766 0 0) 0x7ff47810de90 con 0x7ff47810e200 2024-04-14T15:20:44.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:44.861+0000 7ff4677fe700 1 -- 172.21.15.90:0/1376112303 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff4781a8ff0 con 0x7ff47810e200 2024-04-14T15:20:44.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:44.861+0000 7ff4677fe700 1 -- 172.21.15.90:0/1376112303 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2493287616 0 0) 0x7ff4781a7e10 con 0x7ff47810a630 2024-04-14T15:20:44.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:44.861+0000 7ff4677fe700 1 -- 172.21.15.90:0/1376112303 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff47810de90 con 0x7ff47810a630 2024-04-14T15:20:44.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:44.861+0000 7ff4677fe700 1 -- 172.21.15.90:0/1376112303 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3369881146 0 0) 0x7ff4600035b0 con 0x7ff478107780 2024-04-14T15:20:44.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:44.861+0000 7ff4677fe700 1 -- 172.21.15.90:0/1376112303 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff4781a7e10 con 0x7ff478107780 2024-04-14T15:20:44.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:44.861+0000 7ff4677fe700 1 -- 172.21.15.90:0/1376112303 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7ff46c0018b0 con 0x7ff478107780 2024-04-14T15:20:44.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:44.861+0000 7ff4677fe700 1 -- 172.21.15.90:0/1376112303 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 508439962 0 0) 0x7ff4781a8ff0 con 0x7ff47810e200 2024-04-14T15:20:44.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:44.861+0000 7ff4677fe700 1 -- 172.21.15.90:0/1376112303 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff4600035b0 con 0x7ff47810e200 2024-04-14T15:20:44.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:44.861+0000 7ff4677fe700 1 -- 172.21.15.90:0/1376112303 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2866484495 0 0) 0x7ff47810de90 con 0x7ff47810a630 2024-04-14T15:20:44.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:44.861+0000 7ff4677fe700 1 -- 172.21.15.90:0/1376112303 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff4781a8ff0 con 0x7ff47810a630 2024-04-14T15:20:44.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:44.861+0000 7ff4677fe700 1 -- 172.21.15.90:0/1376112303 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7ff4700017b0 con 0x7ff47810e200 2024-04-14T15:20:44.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:44.861+0000 7ff4677fe700 1 -- 172.21.15.90:0/1376112303 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7ff468001830 con 0x7ff47810a630 2024-04-14T15:20:44.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:44.861+0000 7ff4677fe700 1 -- 172.21.15.90:0/1376112303 <== mon.2 v1:172.21.15.90:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 379819307 0 0) 0x7ff4781a7e10 con 0x7ff478107780 2024-04-14T15:20:44.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:44.861+0000 7ff4677fe700 1 -- 172.21.15.90:0/1376112303 >> v1:172.21.15.92:6789/0 conn(0x7ff47810a630 legacy=0x7ff47810ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:44.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:44.861+0000 7ff4677fe700 1 -- 172.21.15.90:0/1376112303 >> v1:172.21.15.90:6789/0 conn(0x7ff47810e200 legacy=0x7ff4781a56e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:44.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:44.861+0000 7ff4677fe700 1 -- 172.21.15.90:0/1376112303 --> v1:172.21.15.90:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4781aa1d0 con 0x7ff478107780 2024-04-14T15:20:44.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:44.861+0000 7ff47cd9d700 1 -- 172.21.15.90:0/1376112303 --> v1:172.21.15.90:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff4781a9220 con 0x7ff478107780 2024-04-14T15:20:44.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:44.861+0000 7ff47cd9d700 10 client.47471.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:20:44.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:44.861+0000 7ff47cd9d700 1 -- 172.21.15.90:0/1376112303 --> v1:172.21.15.90:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7ff4781a96e0 con 0x7ff478107780 2024-04-14T15:20:44.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:44.861+0000 7ff47cd9d700 1 librados: init done 2024-04-14T15:20:44.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:44.861+0000 7ff4677fe700 1 -- 172.21.15.90:0/1376112303 <== mon.2 v1:172.21.15.90:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7ff46c001c50 con 0x7ff478107780 2024-04-14T15:20:44.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:44.861+0000 7ff4677fe700 1 -- 172.21.15.90:0/1376112303 <== mon.2 v1:172.21.15.90:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7ff46c005d10 con 0x7ff478107780 2024-04-14T15:20:44.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:44.862+0000 7ff47cd9d700 1 -- 172.21.15.90:0/1376112303 --> v1:172.21.15.90:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff458005150 con 0x7ff478107780 2024-04-14T15:20:44.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:44.863+0000 7ff4677fe700 1 -- 172.21.15.90:0/1376112303 <== mon.2 v1:172.21.15.90:6790/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7ff46c003950 con 0x7ff478107780 2024-04-14T15:20:44.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:44.864+0000 7ff4677fe700 1 -- 172.21.15.90:0/1376112303 <== mon.2 v1:172.21.15.90:6790/0 8 ==== osd_map(519..519 src has 51..519) ==== 8264+0+0 (unknown 1173169728 0 0) 0x7ff46c08aa20 con 0x7ff478107780 2024-04-14T15:20:44.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:44.864+0000 7ff4677fe700 10 client.47471.objecter ms_dispatch 0x7ff478001550 osd_map(519..519 src has 51..519) 2024-04-14T15:20:44.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:44.864+0000 7ff4677fe700 3 client.47471.objecter handle_osd_map got epochs [519,519] > 0 2024-04-14T15:20:44.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:44.864+0000 7ff4677fe700 3 client.47471.objecter handle_osd_map decoding full epoch 519 2024-04-14T15:20:44.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:44.864+0000 7ff4677fe700 20 client.47471.objecter dump_active .. 0 homeless 2024-04-14T15:20:44.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:44.864+0000 7ff4677fe700 10 client.47471.objecter ms_handle_connect 0x7ff4600757a0 2024-04-14T15:20:44.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:44.869+0000 7ff4677fe700 1 -- 172.21.15.90:0/1376112303 <== mon.2 v1:172.21.15.90:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7ff46c0573f0 con 0x7ff478107780 2024-04-14T15:20:44.884 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:44.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:44.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:20:44.985 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:45.009 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:45.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:45.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:45.096 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:44 smithi092 ceph-mon[33966]: pgmap v752: 256 pgs: 256 active+clean; 8.4 MiB data, 593 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 3.0 KiB/s wr, 10 op/s 2024-04-14T15:20:45.096 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:44 smithi092 ceph-mon[33966]: 4.6 scrub starts 2024-04-14T15:20:45.096 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:44 smithi092 ceph-mon[33966]: 4.6 scrub ok 2024-04-14T15:20:45.096 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:44 smithi092 ceph-mon[33966]: osdmap e520: 8 total, 8 up, 8 in 2024-04-14T15:20:45.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:45.098+0000 7ff47cd9d700 1 -- 172.21.15.90:0/1376112303 --> v1:172.21.15.90:6790/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7ff458005440 con 0x7ff478107780 2024-04-14T15:20:45.099 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:44 smithi090 ceph-mon[41670]: pgmap v752: 256 pgs: 256 active+clean; 8.4 MiB data, 593 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 3.0 KiB/s wr, 10 op/s 2024-04-14T15:20:45.099 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:44 smithi090 ceph-mon[41670]: 4.6 scrub starts 2024-04-14T15:20:45.099 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:44 smithi090 ceph-mon[41670]: 4.6 scrub ok 2024-04-14T15:20:45.099 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:44 smithi090 ceph-mon[41670]: osdmap e520: 8 total, 8 up, 8 in 2024-04-14T15:20:45.099 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:44 smithi090 ceph-mon[30399]: pgmap v752: 256 pgs: 256 active+clean; 8.4 MiB data, 593 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 3.0 KiB/s wr, 10 op/s 2024-04-14T15:20:45.099 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:44 smithi090 ceph-mon[30399]: 4.6 scrub starts 2024-04-14T15:20:45.099 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:44 smithi090 ceph-mon[30399]: 4.6 scrub ok 2024-04-14T15:20:45.099 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:44 smithi090 ceph-mon[30399]: osdmap e520: 8 total, 8 up, 8 in 2024-04-14T15:20:45.111 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:20:45.135 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:45.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:45.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:20:45.236 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:45.260 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:45.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:45.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:45.361 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:20:45.372 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:20:45 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:20:45.386 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:45.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:45.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:20:45.488 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:20:45.512 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:45.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:45.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:20:45.613 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:20:45.637 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:45.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:45.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:20:45.641 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-04-14T15:20:45.641 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:20:45.739 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:45.762 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:45.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:45.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:45.863 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:20:45.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:45.887 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:45.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:20:45.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:45.952+0000 7ff4677fe700 1 -- 172.21.15.90:0/1376112303 <== mon.2 v1:172.21.15.90:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v521) ==== 87+0+0 (unknown 301738605 0 0) 0x7ff46c05b0a0 con 0x7ff478107780 2024-04-14T15:20:45.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:45.956+0000 7ff47cd9d700 10 librados: watch_flush enter 2024-04-14T15:20:45.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:45.956+0000 7ff47cd9d700 10 librados: watch_flush exit 2024-04-14T15:20:45.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:45.956+0000 7ff47cd9d700 20 client.47471.objecter shutdown clearing up homeless session... 2024-04-14T15:20:45.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:45.956+0000 7ff47cd9d700 10 client.47471.objecter successfully canceled tick 2024-04-14T15:20:45.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:45.956+0000 7ff47cd9d700 1 -- 172.21.15.90:0/1376112303 >> v1:172.21.15.90:6800/420935427 conn(0x7ff4600757a0 legacy=0x7ff460077c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:45.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:45.956+0000 7ff47cd9d700 1 -- 172.21.15.90:0/1376112303 >> v1:172.21.15.90:6790/0 conn(0x7ff478107780 legacy=0x7ff47810c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:45.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:45.957+0000 7ff47cd9d700 1 -- 172.21.15.90:0/1376112303 shutdown_connections 2024-04-14T15:20:45.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:45.957+0000 7ff47cd9d700 1 -- 172.21.15.90:0/1376112303 >> 172.21.15.90:0/1376112303 conn(0x7ff4780fd2b0 msgr2=0x7ff478101780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:20:45.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:45.958+0000 7ff47cd9d700 1 -- 172.21.15.90:0/1376112303 shutdown_connections 2024-04-14T15:20:45.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:45.958+0000 7ff47cd9d700 1 -- 172.21.15.90:0/1376112303 wait complete. 2024-04-14T15:20:45.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:45.959+0000 7ff47cd9d700 1 librados: shutdown 2024-04-14T15:20:45.961 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is set 2024-04-14T15:20:45.988 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:20:46.013 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:46.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:46.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:20:46.115 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:20:46.138 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:46.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:46.139 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:20:46.240 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:20:46.245 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:45 smithi092 ceph-mon[33966]: 6.d scrub starts 2024-04-14T15:20:46.245 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:45 smithi092 ceph-mon[33966]: 6.d scrub ok 2024-04-14T15:20:46.245 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:45 smithi092 ceph-mon[33966]: 4.1f deep-scrub starts 2024-04-14T15:20:46.245 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:45 smithi092 ceph-mon[33966]: 4.1f deep-scrub ok 2024-04-14T15:20:46.245 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:45 smithi092 ceph-mon[33966]: 5.16 scrub starts 2024-04-14T15:20:46.246 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:45 smithi092 ceph-mon[33966]: 5.16 scrub ok 2024-04-14T15:20:46.246 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:45 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1376112303' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:20:46.246 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:45 smithi092 ceph-mon[33966]: from='client.47471 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:20:46.246 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:45 smithi092 ceph-mon[33966]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:20:46.246 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:45 smithi092 ceph-mon[33966]: from='client.47471 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-14T15:20:46.246 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:45 smithi092 ceph-mon[33966]: osdmap e521: 8 total, 8 up, 8 in 2024-04-14T15:20:46.246 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:45 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:20:46.246 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:45 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:20:46.266 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:46.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:46.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:20:46.272 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-14T15:20:46.276 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:45 smithi090 ceph-mon[30399]: 6.d scrub starts 2024-04-14T15:20:46.276 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:45 smithi090 ceph-mon[30399]: 6.d scrub ok 2024-04-14T15:20:46.276 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:45 smithi090 ceph-mon[30399]: 4.1f deep-scrub starts 2024-04-14T15:20:46.276 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:45 smithi090 ceph-mon[30399]: 4.1f deep-scrub ok 2024-04-14T15:20:46.276 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:45 smithi090 ceph-mon[30399]: 5.16 scrub starts 2024-04-14T15:20:46.276 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:45 smithi090 ceph-mon[30399]: 5.16 scrub ok 2024-04-14T15:20:46.277 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:45 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1376112303' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:20:46.277 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:45 smithi090 ceph-mon[30399]: from='client.47471 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:20:46.277 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:45 smithi090 ceph-mon[30399]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:20:46.277 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:45 smithi090 ceph-mon[30399]: from='client.47471 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-14T15:20:46.277 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:45 smithi090 ceph-mon[30399]: osdmap e521: 8 total, 8 up, 8 in 2024-04-14T15:20:46.277 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:45 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:20:46.277 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:45 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:20:46.277 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:45 smithi090 ceph-mon[41670]: 6.d scrub starts 2024-04-14T15:20:46.277 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:45 smithi090 ceph-mon[41670]: 6.d scrub ok 2024-04-14T15:20:46.277 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:45 smithi090 ceph-mon[41670]: 4.1f deep-scrub starts 2024-04-14T15:20:46.277 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:45 smithi090 ceph-mon[41670]: 4.1f deep-scrub ok 2024-04-14T15:20:46.277 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:45 smithi090 ceph-mon[41670]: 5.16 scrub starts 2024-04-14T15:20:46.277 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:45 smithi090 ceph-mon[41670]: 5.16 scrub ok 2024-04-14T15:20:46.277 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:45 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1376112303' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:20:46.278 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:45 smithi090 ceph-mon[41670]: from='client.47471 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:20:46.278 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:45 smithi090 ceph-mon[41670]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:20:46.278 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:45 smithi090 ceph-mon[41670]: from='client.47471 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-14T15:20:46.278 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:45 smithi090 ceph-mon[41670]: osdmap e521: 8 total, 8 up, 8 in 2024-04-14T15:20:46.278 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:45 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:20:46.278 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:45 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:20:46.368 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:20:46.384 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-14T15:20:46.384 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-14T15:20:46.384 INFO:tasks.thrashosds.thrasher:Removing osd 6, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-14T15:20:46.384 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd out 6 2024-04-14T15:20:46.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:46.392 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:46.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:20:46.464 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-14T15:20:46.493 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:20:46.516 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:46.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:46.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:20:46.553 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:20:46.618 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:46.643 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:46.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:46.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:46.745 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:20:46.771 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:46.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:46.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:20:46.822 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:20:46.826 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:20:46.873 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:20:46.897 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:46.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:46.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:20:46.999 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:20:47.021 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:47.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:47.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:20:47.123 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:20:47.150 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:47.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:47.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:20:47.252 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:20:47.275 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:47.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:47.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:20:47.315 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:46 smithi090 ceph-mon[41670]: pgmap v755: 224 pgs: 2 peering, 222 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-04-14T15:20:47.315 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:46 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:20:47.315 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:46 smithi090 ceph-mon[41670]: 2.15 scrub starts 2024-04-14T15:20:47.315 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:46 smithi090 ceph-mon[41670]: 2.15 scrub ok 2024-04-14T15:20:47.315 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:46 smithi090 ceph-mon[41670]: 5.9 scrub starts 2024-04-14T15:20:47.315 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:46 smithi090 ceph-mon[41670]: 5.9 scrub ok 2024-04-14T15:20:47.315 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:46 smithi090 ceph-mon[41670]: 5.1f scrub starts 2024-04-14T15:20:47.315 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:46 smithi090 ceph-mon[41670]: 5.1f scrub ok 2024-04-14T15:20:47.315 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:46 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-85","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:20:47.315 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:46 smithi090 ceph-mon[41670]: osdmap e522: 8 total, 8 up, 8 in 2024-04-14T15:20:47.315 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:46 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-14T15:20:47.315 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:46 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-14T15:20:47.315 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:46 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-14T15:20:47.316 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:46 smithi090 ceph-mon[30399]: pgmap v755: 224 pgs: 2 peering, 222 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-04-14T15:20:47.316 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:46 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:20:47.316 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:46 smithi090 ceph-mon[30399]: 2.15 scrub starts 2024-04-14T15:20:47.316 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:46 smithi090 ceph-mon[30399]: 2.15 scrub ok 2024-04-14T15:20:47.316 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:46 smithi090 ceph-mon[30399]: 5.9 scrub starts 2024-04-14T15:20:47.316 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:46 smithi090 ceph-mon[30399]: 5.9 scrub ok 2024-04-14T15:20:47.316 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:46 smithi090 ceph-mon[30399]: 5.1f scrub starts 2024-04-14T15:20:47.316 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:46 smithi090 ceph-mon[30399]: 5.1f scrub ok 2024-04-14T15:20:47.316 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:46 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-85","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:20:47.316 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:46 smithi090 ceph-mon[30399]: osdmap e522: 8 total, 8 up, 8 in 2024-04-14T15:20:47.316 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:46 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-14T15:20:47.316 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:46 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-14T15:20:47.316 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:46 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-14T15:20:47.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:46 smithi092 ceph-mon[33966]: pgmap v755: 224 pgs: 2 peering, 222 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-04-14T15:20:47.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:46 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:20:47.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:46 smithi092 ceph-mon[33966]: 2.15 scrub starts 2024-04-14T15:20:47.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:46 smithi092 ceph-mon[33966]: 2.15 scrub ok 2024-04-14T15:20:47.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:46 smithi092 ceph-mon[33966]: 5.9 scrub starts 2024-04-14T15:20:47.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:46 smithi092 ceph-mon[33966]: 5.9 scrub ok 2024-04-14T15:20:47.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:46 smithi092 ceph-mon[33966]: 5.1f scrub starts 2024-04-14T15:20:47.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:46 smithi092 ceph-mon[33966]: 5.1f scrub ok 2024-04-14T15:20:47.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:46 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-85","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:20:47.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:46 smithi092 ceph-mon[33966]: osdmap e522: 8 total, 8 up, 8 in 2024-04-14T15:20:47.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:46 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-14T15:20:47.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:46 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-14T15:20:47.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:46 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-14T15:20:47.376 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:20:47.399 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:47.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:47.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:20:47.501 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:20:47.525 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:47.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:47.526 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:20:47.626 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:20:47.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:47.651 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:47.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:20:47.752 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:20:47.775 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:47.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:47.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:20:47.877 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:20:47.901 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:47.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:47.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:20:48.003 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:20:48.027 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:48.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:48.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:20:48.129 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:20:48.154 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:48.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:48.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:20:48.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:48 smithi090 ceph-mon[30399]: pgmap v758: 255 pgs: 32 unknown, 1 peering, 222 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-04-14T15:20:48.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:48 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-04-14T15:20:48.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:48 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-85","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-14T15:20:48.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:48 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "27"}]': finished 2024-04-14T15:20:48.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:48 smithi090 ceph-mon[30399]: osdmap e523: 8 total, 8 up, 8 in 2024-04-14T15:20:48.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:48 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-85","var": "dedup_tier","val": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:20:48.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:48 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-85","var": "dedup_tier","val": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:20:48.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:48 smithi090 ceph-mon[41670]: pgmap v758: 255 pgs: 32 unknown, 1 peering, 222 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-04-14T15:20:48.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:48 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-04-14T15:20:48.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:48 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-85","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-14T15:20:48.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:48 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "27"}]': finished 2024-04-14T15:20:48.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:48 smithi090 ceph-mon[41670]: osdmap e523: 8 total, 8 up, 8 in 2024-04-14T15:20:48.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:48 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-85","var": "dedup_tier","val": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:20:48.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:48 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-85","var": "dedup_tier","val": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:20:48.256 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:20:48.265 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-04-14T15:20:48.266 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:20:48.266 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:48 smithi092 ceph-mon[33966]: pgmap v758: 255 pgs: 32 unknown, 1 peering, 222 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-04-14T15:20:48.266 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:48 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-04-14T15:20:48.266 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:48 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-85","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-14T15:20:48.266 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:48 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "27"}]': finished 2024-04-14T15:20:48.266 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:48 smithi092 ceph-mon[33966]: osdmap e523: 8 total, 8 up, 8 in 2024-04-14T15:20:48.267 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:48 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-85","var": "dedup_tier","val": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:20:48.267 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:48 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-85","var": "dedup_tier","val": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:20:48.280 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:48.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:48.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:20:48.381 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:20:48.406 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:48.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:48.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:20:48.507 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:48.532 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:48.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:48.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:48.633 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:20:48.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:48.658 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:48.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:20:48.759 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-14T15:20:48.759 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:20:48.784 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:48.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:48.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:20:48.886 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:20:48.911 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:48.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:48.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:20:49.006 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-14T15:20:49.013 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:20:49.073 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:49.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:49.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:20:49.175 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:20:49.199 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:49.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:49.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:20:49.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.211+0000 7fdbd7d29700 1 -- 172.21.15.90:0/568184230 >> v1:172.21.15.90:6789/0 conn(0x7fdbd0101350 legacy=0x7fdbd0101730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:49.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.211+0000 7fdbd7d29700 1 -- 172.21.15.90:0/568184230 shutdown_connections 2024-04-14T15:20:49.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.211+0000 7fdbd7d29700 1 -- 172.21.15.90:0/568184230 >> 172.21.15.90:0/568184230 conn(0x7fdbd0076ad0 msgr2=0x7fdbd0076ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:20:49.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.211+0000 7fdbd7d29700 1 -- 172.21.15.90:0/568184230 shutdown_connections 2024-04-14T15:20:49.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.211+0000 7fdbd7d29700 1 -- 172.21.15.90:0/568184230 wait complete. 2024-04-14T15:20:49.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.211+0000 7fdbd7d29700 1 librados: starting msgr at 2024-04-14T15:20:49.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.211+0000 7fdbd7d29700 1 librados: starting objecter 2024-04-14T15:20:49.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.211+0000 7fdbd7d29700 1 Processor -- start 2024-04-14T15:20:49.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.211+0000 7fdbd7d29700 1 -- start start 2024-04-14T15:20:49.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.211+0000 7fdbd7d29700 1 librados: setting wanted keys 2024-04-14T15:20:49.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.211+0000 7fdbd7d29700 1 librados: calling monclient init 2024-04-14T15:20:49.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.211+0000 7fdbd7d29700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdbd0108280 con 0x7fdbd010f330 2024-04-14T15:20:49.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.211+0000 7fdbd7d29700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdbd01a8f00 con 0x7fdbd010b7c0 2024-04-14T15:20:49.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.212+0000 7fdbd7d29700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdbd01aa0e0 con 0x7fdbd0101350 2024-04-14T15:20:49.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.212+0000 7fdbd5ac5700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7fdbd0101350 0x7fdbd01050d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:38156/0 (socket says 172.21.15.90:38156) 2024-04-14T15:20:49.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.212+0000 7fdbd5ac5700 1 -- 172.21.15.90:0/3573576483 learned_addr learned my addr 172.21.15.90:0/3573576483 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:20:49.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.212+0000 7fdbc6ffd700 10 client.?.objecter ms_handle_connect 0x7fdbd010f330 2024-04-14T15:20:49.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.212+0000 7fdbc6ffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:20:49.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.212+0000 7fdbc6ffd700 10 client.?.objecter ms_handle_connect 0x7fdbd0101350 2024-04-14T15:20:49.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.212+0000 7fdbc6ffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:20:49.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.212+0000 7fdbc6ffd700 10 client.?.objecter ms_handle_connect 0x7fdbd010b7c0 2024-04-14T15:20:49.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.212+0000 7fdbc6ffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:20:49.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.212+0000 7fdbc6ffd700 1 -- 172.21.15.90:0/3573576483 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3946352571 0 0) 0x7fdbd0108280 con 0x7fdbd010f330 2024-04-14T15:20:49.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.212+0000 7fdbc6ffd700 1 -- 172.21.15.90:0/3573576483 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdbbc0035b0 con 0x7fdbd010f330 2024-04-14T15:20:49.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.212+0000 7fdbc6ffd700 1 -- 172.21.15.90:0/3573576483 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1208997289 0 0) 0x7fdbd01a8f00 con 0x7fdbd010b7c0 2024-04-14T15:20:49.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.212+0000 7fdbc6ffd700 1 -- 172.21.15.90:0/3573576483 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdbd0108280 con 0x7fdbd010b7c0 2024-04-14T15:20:49.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.212+0000 7fdbc6ffd700 1 -- 172.21.15.90:0/3573576483 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 507108900 0 0) 0x7fdbd01aa0e0 con 0x7fdbd0101350 2024-04-14T15:20:49.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.212+0000 7fdbc6ffd700 1 -- 172.21.15.90:0/3573576483 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdbd01a8f00 con 0x7fdbd0101350 2024-04-14T15:20:49.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.212+0000 7fdbc6ffd700 1 -- 172.21.15.90:0/3573576483 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 539365487 0 0) 0x7fdbd0108280 con 0x7fdbd010b7c0 2024-04-14T15:20:49.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.212+0000 7fdbc6ffd700 1 -- 172.21.15.90:0/3573576483 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdbd01aa0e0 con 0x7fdbd010b7c0 2024-04-14T15:20:49.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.212+0000 7fdbc6ffd700 1 -- 172.21.15.90:0/3573576483 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 727558772 0 0) 0x7fdbbc0035b0 con 0x7fdbd010f330 2024-04-14T15:20:49.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.212+0000 7fdbc6ffd700 1 -- 172.21.15.90:0/3573576483 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdbd0108280 con 0x7fdbd010f330 2024-04-14T15:20:49.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.212+0000 7fdbc6ffd700 1 -- 172.21.15.90:0/3573576483 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fdbc0001870 con 0x7fdbd010b7c0 2024-04-14T15:20:49.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.212+0000 7fdbc6ffd700 1 -- 172.21.15.90:0/3573576483 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fdbc8001bb0 con 0x7fdbd010f330 2024-04-14T15:20:49.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.212+0000 7fdbc6ffd700 1 -- 172.21.15.90:0/3573576483 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2692930484 0 0) 0x7fdbd01a8f00 con 0x7fdbd0101350 2024-04-14T15:20:49.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.212+0000 7fdbc6ffd700 1 -- 172.21.15.90:0/3573576483 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdbbc0035b0 con 0x7fdbd0101350 2024-04-14T15:20:49.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.212+0000 7fdbc6ffd700 1 -- 172.21.15.90:0/3573576483 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fdbcc000f60 con 0x7fdbd0101350 2024-04-14T15:20:49.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.212+0000 7fdbc6ffd700 1 -- 172.21.15.90:0/3573576483 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 240530442 0 0) 0x7fdbd01aa0e0 con 0x7fdbd010b7c0 2024-04-14T15:20:49.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.212+0000 7fdbc6ffd700 1 -- 172.21.15.90:0/3573576483 >> v1:172.21.15.90:6790/0 conn(0x7fdbd0101350 legacy=0x7fdbd01050d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:49.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.212+0000 7fdbc6ffd700 1 -- 172.21.15.90:0/3573576483 >> v1:172.21.15.90:6789/0 conn(0x7fdbd010f330 legacy=0x7fdbd01a67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:49.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.212+0000 7fdbc6ffd700 1 -- 172.21.15.90:0/3573576483 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbd01ab2c0 con 0x7fdbd010b7c0 2024-04-14T15:20:49.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.212+0000 7fdbd7d29700 1 -- 172.21.15.90:0/3573576483 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdbd01a9130 con 0x7fdbd010b7c0 2024-04-14T15:20:49.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.212+0000 7fdbd7d29700 10 client.49357.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:20:49.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.212+0000 7fdbd7d29700 1 -- 172.21.15.90:0/3573576483 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fdbd01a95f0 con 0x7fdbd010b7c0 2024-04-14T15:20:49.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.212+0000 7fdbd7d29700 1 librados: init done 2024-04-14T15:20:49.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.212+0000 7fdbc6ffd700 1 -- 172.21.15.90:0/3573576483 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fdbc0003b40 con 0x7fdbd010b7c0 2024-04-14T15:20:49.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.212+0000 7fdbc6ffd700 1 -- 172.21.15.90:0/3573576483 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fdbc0004a20 con 0x7fdbd010b7c0 2024-04-14T15:20:49.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.213+0000 7fdbd7d29700 1 -- 172.21.15.90:0/3573576483 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fdbb4005100 con 0x7fdbd010b7c0 2024-04-14T15:20:49.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.214+0000 7fdbc6ffd700 1 -- 172.21.15.90:0/3573576483 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fdbc0003680 con 0x7fdbd010b7c0 2024-04-14T15:20:49.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.214+0000 7fdbc6ffd700 1 -- 172.21.15.90:0/3573576483 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(524..524 src has 467..524) ==== 8229+0+0 (unknown 2086490208 0 0) 0x7fdbc0089680 con 0x7fdbd010b7c0 2024-04-14T15:20:49.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.214+0000 7fdbc6ffd700 10 client.49357.objecter ms_dispatch 0x7fdbd0001550 osd_map(524..524 src has 467..524) 2024-04-14T15:20:49.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.214+0000 7fdbc6ffd700 3 client.49357.objecter handle_osd_map got epochs [524,524] > 0 2024-04-14T15:20:49.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.214+0000 7fdbc6ffd700 3 client.49357.objecter handle_osd_map decoding full epoch 524 2024-04-14T15:20:49.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.214+0000 7fdbc6ffd700 20 client.49357.objecter dump_active .. 0 homeless 2024-04-14T15:20:49.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.214+0000 7fdbc6ffd700 1 -- 172.21.15.90:0/3573576483 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v524) ==== 54+0+16 (unknown 2463213335 0 2420000192) 0x7fdbc0089a80 con 0x7fdbd010b7c0 2024-04-14T15:20:49.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.214+0000 7fdbc6ffd700 10 client.49357.objecter ms_handle_connect 0x7fdbbc0714b0 2024-04-14T15:20:49.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.215+0000 7fdbd7d29700 10 client.49357.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:20:49.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.215+0000 7fdbd7d29700 15 client.49357.objecter _session_command_op_assign -1 1 2024-04-14T15:20:49.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.215+0000 7fdbd7d29700 20 client.49357.objecter _get_session s=0x7fdbb40053e0 osd=0 3 2024-04-14T15:20:49.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.215+0000 7fdbd7d29700 20 client.49357.objecter put_session s=0x7fdbb40053e0 osd=0 3 2024-04-14T15:20:49.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.215+0000 7fdbd7d29700 20 client.49357.objecter _get_session s=0x7fdbb40053e0 osd=0 3 2024-04-14T15:20:49.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.215+0000 7fdbd7d29700 15 client.49357.objecter _session_command_op_remove -1 1 2024-04-14T15:20:49.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.215+0000 7fdbd7d29700 20 client.49357.objecter get_session s=0x7fdbb40053e0 osd=0 3 2024-04-14T15:20:49.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.215+0000 7fdbd7d29700 15 client.49357.objecter _session_command_op_assign 0 1 2024-04-14T15:20:49.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.215+0000 7fdbd7d29700 20 client.49357.objecter put_session s=0x7fdbb40053e0 osd=0 4 2024-04-14T15:20:49.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.215+0000 7fdbd7d29700 10 client.49357.objecter _send_command 1 2024-04-14T15:20:49.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.215+0000 7fdbd7d29700 1 -- 172.21.15.90:0/3573576483 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdbb4004b50 con 0x7fdbb4004730 2024-04-14T15:20:49.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.215+0000 7fdbc6ffd700 10 client.49357.objecter ms_handle_connect 0x7fdbb4004730 2024-04-14T15:20:49.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.216+0000 7fdbc6ffd700 1 -- 172.21.15.90:0/3573576483 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fdbb4004b50 con 0x7fdbb4004730 2024-04-14T15:20:49.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.216+0000 7fdbc6ffd700 10 client.49357.objecter ms_dispatch 0x7fdbd0001550 command_reply(tid 1: 0 ) 2024-04-14T15:20:49.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.216+0000 7fdbc6ffd700 10 client.49357.objecter _finish_command 1 = system:0 2024-04-14T15:20:49.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.216+0000 7fdbc6ffd700 20 client.49357.objecter put_session s=0x7fdbb40053e0 osd=0 4 2024-04-14T15:20:49.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.216+0000 7fdbc6ffd700 15 client.49357.objecter _session_command_op_remove 0 1 2024-04-14T15:20:49.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.234+0000 7fdbd7d29700 10 client.49357.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:20:49.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.234+0000 7fdbd7d29700 15 client.49357.objecter _session_command_op_assign -1 2 2024-04-14T15:20:49.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.234+0000 7fdbd7d29700 20 client.49357.objecter _get_session s=0x7fdbb40053e0 osd=0 3 2024-04-14T15:20:49.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.234+0000 7fdbd7d29700 20 client.49357.objecter put_session s=0x7fdbb40053e0 osd=0 3 2024-04-14T15:20:49.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.234+0000 7fdbd7d29700 20 client.49357.objecter _get_session s=0x7fdbb40053e0 osd=0 3 2024-04-14T15:20:49.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.234+0000 7fdbd7d29700 15 client.49357.objecter _session_command_op_remove -1 2 2024-04-14T15:20:49.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.234+0000 7fdbd7d29700 20 client.49357.objecter get_session s=0x7fdbb40053e0 osd=0 3 2024-04-14T15:20:49.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.234+0000 7fdbd7d29700 15 client.49357.objecter _session_command_op_assign 0 2 2024-04-14T15:20:49.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.234+0000 7fdbd7d29700 20 client.49357.objecter put_session s=0x7fdbb40053e0 osd=0 4 2024-04-14T15:20:49.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.234+0000 7fdbd7d29700 10 client.49357.objecter _send_command 2 2024-04-14T15:20:49.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.234+0000 7fdbd7d29700 1 -- 172.21.15.90:0/3573576483 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdbb40079f0 con 0x7fdbb4004730 2024-04-14T15:20:49.235 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:49 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:20:49.235 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:49 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:20:49.236 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:49 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-85","var": "dedup_tier","val": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:20:49.236 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:49 smithi090 ceph-mon[30399]: osdmap e524: 8 total, 8 up, 8 in 2024-04-14T15:20:49.236 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:49 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-14T15:20:49.236 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:49 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-14T15:20:49.236 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:49 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:20:49.236 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:49 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:20:49.236 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:49 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-85","var": "dedup_tier","val": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:20:49.236 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:49 smithi090 ceph-mon[41670]: osdmap e524: 8 total, 8 up, 8 in 2024-04-14T15:20:49.236 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:49 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-14T15:20:49.236 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:49 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-14T15:20:49.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.236+0000 7fdbc6ffd700 1 -- 172.21.15.90:0/3573576483 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fdbb40079f0 con 0x7fdbb4004730 2024-04-14T15:20:49.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.236+0000 7fdbc6ffd700 10 client.49357.objecter ms_dispatch 0x7fdbd0001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:20:49.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.236+0000 7fdbc6ffd700 10 client.49357.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:20:49.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.236+0000 7fdbc6ffd700 20 client.49357.objecter put_session s=0x7fdbb40053e0 osd=0 4 2024-04-14T15:20:49.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.236+0000 7fdbc6ffd700 15 client.49357.objecter _session_command_op_remove 0 2 2024-04-14T15:20:49.237 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:20:49.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.236+0000 7fdbd7d29700 10 client.49357.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:20:49.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.236+0000 7fdbd7d29700 15 client.49357.objecter _session_command_op_assign -1 3 2024-04-14T15:20:49.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.236+0000 7fdbd7d29700 20 client.49357.objecter _get_session s=0x7fdbb4004110 osd=1 3 2024-04-14T15:20:49.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.236+0000 7fdbd7d29700 20 client.49357.objecter put_session s=0x7fdbb4004110 osd=1 3 2024-04-14T15:20:49.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.236+0000 7fdbd7d29700 20 client.49357.objecter _get_session s=0x7fdbb4004110 osd=1 3 2024-04-14T15:20:49.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.236+0000 7fdbd7d29700 15 client.49357.objecter _session_command_op_remove -1 3 2024-04-14T15:20:49.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.236+0000 7fdbd7d29700 20 client.49357.objecter get_session s=0x7fdbb4004110 osd=1 3 2024-04-14T15:20:49.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.236+0000 7fdbd7d29700 15 client.49357.objecter _session_command_op_assign 1 3 2024-04-14T15:20:49.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.236+0000 7fdbd7d29700 20 client.49357.objecter put_session s=0x7fdbb4004110 osd=1 4 2024-04-14T15:20:49.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.236+0000 7fdbd7d29700 10 client.49357.objecter _send_command 3 2024-04-14T15:20:49.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.236+0000 7fdbd7d29700 1 -- 172.21.15.90:0/3573576483 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fdbb40020b0 con 0x7fdbb4004280 2024-04-14T15:20:49.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.237+0000 7fdbc6ffd700 10 client.49357.objecter ms_handle_connect 0x7fdbb4004280 2024-04-14T15:20:49.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.238+0000 7fdbc6ffd700 1 -- 172.21.15.90:0/3573576483 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fdbb40020b0 con 0x7fdbb4004280 2024-04-14T15:20:49.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.238+0000 7fdbc6ffd700 10 client.49357.objecter ms_dispatch 0x7fdbd0001550 command_reply(tid 3: 0 ) 2024-04-14T15:20:49.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.238+0000 7fdbc6ffd700 10 client.49357.objecter _finish_command 3 = system:0 2024-04-14T15:20:49.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.238+0000 7fdbc6ffd700 20 client.49357.objecter put_session s=0x7fdbb4004110 osd=1 4 2024-04-14T15:20:49.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.238+0000 7fdbc6ffd700 15 client.49357.objecter _session_command_op_remove 1 3 2024-04-14T15:20:49.249 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:20:49.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.260+0000 7fdbd7d29700 10 client.49357.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:20:49.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.260+0000 7fdbd7d29700 15 client.49357.objecter _session_command_op_assign -1 4 2024-04-14T15:20:49.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.260+0000 7fdbd7d29700 20 client.49357.objecter _get_session s=0x7fdbb4004110 osd=1 3 2024-04-14T15:20:49.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.260+0000 7fdbd7d29700 20 client.49357.objecter put_session s=0x7fdbb4004110 osd=1 3 2024-04-14T15:20:49.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.260+0000 7fdbd7d29700 20 client.49357.objecter _get_session s=0x7fdbb4004110 osd=1 3 2024-04-14T15:20:49.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.260+0000 7fdbd7d29700 15 client.49357.objecter _session_command_op_remove -1 4 2024-04-14T15:20:49.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.260+0000 7fdbd7d29700 20 client.49357.objecter get_session s=0x7fdbb4004110 osd=1 3 2024-04-14T15:20:49.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.260+0000 7fdbd7d29700 15 client.49357.objecter _session_command_op_assign 1 4 2024-04-14T15:20:49.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.260+0000 7fdbd7d29700 20 client.49357.objecter put_session s=0x7fdbb4004110 osd=1 4 2024-04-14T15:20:49.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.260+0000 7fdbd7d29700 10 client.49357.objecter _send_command 4 2024-04-14T15:20:49.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.260+0000 7fdbd7d29700 1 -- 172.21.15.90:0/3573576483 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdbb4001600 con 0x7fdbb4004280 2024-04-14T15:20:49.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.261+0000 7fdbc6ffd700 1 -- 172.21.15.90:0/3573576483 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fdbb4001600 con 0x7fdbb4004280 2024-04-14T15:20:49.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.261+0000 7fdbc6ffd700 10 client.49357.objecter ms_dispatch 0x7fdbd0001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:20:49.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.261+0000 7fdbc6ffd700 10 client.49357.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:20:49.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.261+0000 7fdbc6ffd700 20 client.49357.objecter put_session s=0x7fdbb4004110 osd=1 4 2024-04-14T15:20:49.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.261+0000 7fdbc6ffd700 15 client.49357.objecter _session_command_op_remove 1 4 2024-04-14T15:20:49.262 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:20:49.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.261+0000 7fdbd7d29700 10 client.49357.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:20:49.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.261+0000 7fdbd7d29700 15 client.49357.objecter _session_command_op_assign -1 5 2024-04-14T15:20:49.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.261+0000 7fdbd7d29700 20 client.49357.objecter _get_session s=0x7fdbb4001a00 osd=2 3 2024-04-14T15:20:49.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.261+0000 7fdbd7d29700 20 client.49357.objecter put_session s=0x7fdbb4001a00 osd=2 3 2024-04-14T15:20:49.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.261+0000 7fdbd7d29700 20 client.49357.objecter _get_session s=0x7fdbb4001a00 osd=2 3 2024-04-14T15:20:49.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.261+0000 7fdbd7d29700 15 client.49357.objecter _session_command_op_remove -1 5 2024-04-14T15:20:49.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.261+0000 7fdbd7d29700 20 client.49357.objecter get_session s=0x7fdbb4001a00 osd=2 3 2024-04-14T15:20:49.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.261+0000 7fdbd7d29700 15 client.49357.objecter _session_command_op_assign 2 5 2024-04-14T15:20:49.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.261+0000 7fdbd7d29700 20 client.49357.objecter put_session s=0x7fdbb4001a00 osd=2 4 2024-04-14T15:20:49.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.261+0000 7fdbd7d29700 10 client.49357.objecter _send_command 5 2024-04-14T15:20:49.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.261+0000 7fdbd7d29700 1 -- 172.21.15.90:0/3573576483 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fdbb40153c0 con 0x7fdbb4008880 2024-04-14T15:20:49.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.262+0000 7fdbc6ffd700 10 client.49357.objecter ms_handle_connect 0x7fdbb4008880 2024-04-14T15:20:49.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.262+0000 7fdbc6ffd700 1 -- 172.21.15.90:0/3573576483 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fdbb40153c0 con 0x7fdbb4008880 2024-04-14T15:20:49.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.262+0000 7fdbc6ffd700 10 client.49357.objecter ms_dispatch 0x7fdbd0001550 command_reply(tid 5: 0 ) 2024-04-14T15:20:49.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.262+0000 7fdbc6ffd700 10 client.49357.objecter _finish_command 5 = system:0 2024-04-14T15:20:49.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.262+0000 7fdbc6ffd700 20 client.49357.objecter put_session s=0x7fdbb4001a00 osd=2 4 2024-04-14T15:20:49.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.262+0000 7fdbc6ffd700 15 client.49357.objecter _session_command_op_remove 2 5 2024-04-14T15:20:49.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.279+0000 7fdbd7d29700 10 client.49357.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:20:49.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.279+0000 7fdbd7d29700 15 client.49357.objecter _session_command_op_assign -1 6 2024-04-14T15:20:49.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.279+0000 7fdbd7d29700 20 client.49357.objecter _get_session s=0x7fdbb4001a00 osd=2 3 2024-04-14T15:20:49.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.279+0000 7fdbd7d29700 20 client.49357.objecter put_session s=0x7fdbb4001a00 osd=2 3 2024-04-14T15:20:49.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.279+0000 7fdbd7d29700 20 client.49357.objecter _get_session s=0x7fdbb4001a00 osd=2 3 2024-04-14T15:20:49.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.279+0000 7fdbd7d29700 15 client.49357.objecter _session_command_op_remove -1 6 2024-04-14T15:20:49.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.279+0000 7fdbd7d29700 20 client.49357.objecter get_session s=0x7fdbb4001a00 osd=2 3 2024-04-14T15:20:49.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.279+0000 7fdbd7d29700 15 client.49357.objecter _session_command_op_assign 2 6 2024-04-14T15:20:49.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.279+0000 7fdbd7d29700 20 client.49357.objecter put_session s=0x7fdbb4001a00 osd=2 4 2024-04-14T15:20:49.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.279+0000 7fdbd7d29700 10 client.49357.objecter _send_command 6 2024-04-14T15:20:49.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.279+0000 7fdbd7d29700 1 -- 172.21.15.90:0/3573576483 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdbb4007f10 con 0x7fdbb4008880 2024-04-14T15:20:49.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.280+0000 7fdbc6ffd700 1 -- 172.21.15.90:0/3573576483 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fdbb4007f10 con 0x7fdbb4008880 2024-04-14T15:20:49.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.280+0000 7fdbc6ffd700 10 client.49357.objecter ms_dispatch 0x7fdbd0001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:20:49.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.280+0000 7fdbc6ffd700 10 client.49357.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:20:49.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.280+0000 7fdbc6ffd700 20 client.49357.objecter put_session s=0x7fdbb4001a00 osd=2 4 2024-04-14T15:20:49.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.280+0000 7fdbc6ffd700 15 client.49357.objecter _session_command_op_remove 2 6 2024-04-14T15:20:49.281 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:20:49.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.281+0000 7fdbd7d29700 10 client.49357.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:20:49.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.281+0000 7fdbd7d29700 15 client.49357.objecter _session_command_op_assign -1 7 2024-04-14T15:20:49.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.281+0000 7fdbd7d29700 20 client.49357.objecter _get_session s=0x7fdbb4008170 osd=3 3 2024-04-14T15:20:49.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.281+0000 7fdbd7d29700 20 client.49357.objecter put_session s=0x7fdbb4008170 osd=3 3 2024-04-14T15:20:49.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.281+0000 7fdbd7d29700 20 client.49357.objecter _get_session s=0x7fdbb4008170 osd=3 3 2024-04-14T15:20:49.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.281+0000 7fdbd7d29700 15 client.49357.objecter _session_command_op_remove -1 7 2024-04-14T15:20:49.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.281+0000 7fdbd7d29700 20 client.49357.objecter get_session s=0x7fdbb4008170 osd=3 3 2024-04-14T15:20:49.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.281+0000 7fdbd7d29700 15 client.49357.objecter _session_command_op_assign 3 7 2024-04-14T15:20:49.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.281+0000 7fdbd7d29700 20 client.49357.objecter put_session s=0x7fdbb4008170 osd=3 4 2024-04-14T15:20:49.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.281+0000 7fdbd7d29700 10 client.49357.objecter _send_command 7 2024-04-14T15:20:49.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.281+0000 7fdbd7d29700 1 -- 172.21.15.90:0/3573576483 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fdbb40117f0 con 0x7fdbb4011360 2024-04-14T15:20:49.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.281+0000 7fdbc6ffd700 10 client.49357.objecter ms_handle_connect 0x7fdbb4011360 2024-04-14T15:20:49.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.282+0000 7fdbc6ffd700 1 -- 172.21.15.90:0/3573576483 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fdbb40117f0 con 0x7fdbb4011360 2024-04-14T15:20:49.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.282+0000 7fdbc6ffd700 10 client.49357.objecter ms_dispatch 0x7fdbd0001550 command_reply(tid 7: 0 ) 2024-04-14T15:20:49.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.282+0000 7fdbc6ffd700 10 client.49357.objecter _finish_command 7 = system:0 2024-04-14T15:20:49.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.282+0000 7fdbc6ffd700 20 client.49357.objecter put_session s=0x7fdbb4008170 osd=3 4 2024-04-14T15:20:49.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.282+0000 7fdbc6ffd700 15 client.49357.objecter _session_command_op_remove 3 7 2024-04-14T15:20:49.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.298+0000 7fdbd7d29700 10 client.49357.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:20:49.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.298+0000 7fdbd7d29700 15 client.49357.objecter _session_command_op_assign -1 8 2024-04-14T15:20:49.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.298+0000 7fdbd7d29700 20 client.49357.objecter _get_session s=0x7fdbb4008170 osd=3 3 2024-04-14T15:20:49.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.298+0000 7fdbd7d29700 20 client.49357.objecter put_session s=0x7fdbb4008170 osd=3 3 2024-04-14T15:20:49.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.298+0000 7fdbd7d29700 20 client.49357.objecter _get_session s=0x7fdbb4008170 osd=3 3 2024-04-14T15:20:49.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.298+0000 7fdbd7d29700 15 client.49357.objecter _session_command_op_remove -1 8 2024-04-14T15:20:49.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.298+0000 7fdbd7d29700 20 client.49357.objecter get_session s=0x7fdbb4008170 osd=3 3 2024-04-14T15:20:49.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.298+0000 7fdbd7d29700 15 client.49357.objecter _session_command_op_assign 3 8 2024-04-14T15:20:49.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.298+0000 7fdbd7d29700 20 client.49357.objecter put_session s=0x7fdbb4008170 osd=3 4 2024-04-14T15:20:49.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.298+0000 7fdbd7d29700 10 client.49357.objecter _send_command 8 2024-04-14T15:20:49.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.298+0000 7fdbd7d29700 1 -- 172.21.15.90:0/3573576483 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdbb4010de0 con 0x7fdbb4011360 2024-04-14T15:20:49.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.300+0000 7fdbc6ffd700 1 -- 172.21.15.90:0/3573576483 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fdbb4010de0 con 0x7fdbb4011360 2024-04-14T15:20:49.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.300+0000 7fdbc6ffd700 10 client.49357.objecter ms_dispatch 0x7fdbd0001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:20:49.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.300+0000 7fdbc6ffd700 10 client.49357.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:20:49.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.300+0000 7fdbc6ffd700 20 client.49357.objecter put_session s=0x7fdbb4008170 osd=3 4 2024-04-14T15:20:49.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.300+0000 7fdbc6ffd700 15 client.49357.objecter _session_command_op_remove 3 8 2024-04-14T15:20:49.301 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:20:49.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.300+0000 7fdbd7d29700 10 client.49357.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:20:49.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.300+0000 7fdbd7d29700 15 client.49357.objecter _session_command_op_assign -1 9 2024-04-14T15:20:49.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.300+0000 7fdbd7d29700 20 client.49357.objecter _get_session s=0x7fdbb4010b90 osd=4 3 2024-04-14T15:20:49.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.300+0000 7fdbd7d29700 20 client.49357.objecter put_session s=0x7fdbb4010b90 osd=4 3 2024-04-14T15:20:49.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.300+0000 7fdbd7d29700 20 client.49357.objecter _get_session s=0x7fdbb4010b90 osd=4 3 2024-04-14T15:20:49.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.300+0000 7fdbd7d29700 15 client.49357.objecter _session_command_op_remove -1 9 2024-04-14T15:20:49.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.300+0000 7fdbd7d29700 20 client.49357.objecter get_session s=0x7fdbb4010b90 osd=4 3 2024-04-14T15:20:49.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.300+0000 7fdbd7d29700 15 client.49357.objecter _session_command_op_assign 4 9 2024-04-14T15:20:49.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.300+0000 7fdbd7d29700 20 client.49357.objecter put_session s=0x7fdbb4010b90 osd=4 4 2024-04-14T15:20:49.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.300+0000 7fdbd7d29700 10 client.49357.objecter _send_command 9 2024-04-14T15:20:49.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.300+0000 7fdbd7d29700 1 -- 172.21.15.90:0/3573576483 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fdbb4010da0 con 0x7fdbb4016250 2024-04-14T15:20:49.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.301+0000 7fdbc6ffd700 10 client.49357.objecter ms_handle_connect 0x7fdbb4016250 2024-04-14T15:20:49.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.301+0000 7fdbc6ffd700 1 -- 172.21.15.90:0/3573576483 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fdbb4010da0 con 0x7fdbb4016250 2024-04-14T15:20:49.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.301+0000 7fdbc6ffd700 10 client.49357.objecter ms_dispatch 0x7fdbd0001550 command_reply(tid 9: 0 ) 2024-04-14T15:20:49.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.301+0000 7fdbc6ffd700 10 client.49357.objecter _finish_command 9 = system:0 2024-04-14T15:20:49.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.301+0000 7fdbc6ffd700 20 client.49357.objecter put_session s=0x7fdbb4010b90 osd=4 4 2024-04-14T15:20:49.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.301+0000 7fdbc6ffd700 15 client.49357.objecter _session_command_op_remove 4 9 2024-04-14T15:20:49.303 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:20:49.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.318+0000 7fdbd7d29700 10 client.49357.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:20:49.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.318+0000 7fdbd7d29700 15 client.49357.objecter _session_command_op_assign -1 10 2024-04-14T15:20:49.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.318+0000 7fdbd7d29700 20 client.49357.objecter _get_session s=0x7fdbb4010b90 osd=4 3 2024-04-14T15:20:49.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.318+0000 7fdbd7d29700 20 client.49357.objecter put_session s=0x7fdbb4010b90 osd=4 3 2024-04-14T15:20:49.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.318+0000 7fdbd7d29700 20 client.49357.objecter _get_session s=0x7fdbb4010b90 osd=4 3 2024-04-14T15:20:49.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.318+0000 7fdbd7d29700 15 client.49357.objecter _session_command_op_remove -1 10 2024-04-14T15:20:49.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.318+0000 7fdbd7d29700 20 client.49357.objecter get_session s=0x7fdbb4010b90 osd=4 3 2024-04-14T15:20:49.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.318+0000 7fdbd7d29700 15 client.49357.objecter _session_command_op_assign 4 10 2024-04-14T15:20:49.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.318+0000 7fdbd7d29700 20 client.49357.objecter put_session s=0x7fdbb4010b90 osd=4 4 2024-04-14T15:20:49.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.318+0000 7fdbd7d29700 10 client.49357.objecter _send_command 10 2024-04-14T15:20:49.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.318+0000 7fdbd7d29700 1 -- 172.21.15.90:0/3573576483 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdbb401ed50 con 0x7fdbb4016250 2024-04-14T15:20:49.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.320+0000 7fdbc6ffd700 1 -- 172.21.15.90:0/3573576483 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fdbb401ed50 con 0x7fdbb4016250 2024-04-14T15:20:49.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.320+0000 7fdbc6ffd700 10 client.49357.objecter ms_dispatch 0x7fdbd0001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:20:49.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.320+0000 7fdbc6ffd700 10 client.49357.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:20:49.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.320+0000 7fdbc6ffd700 20 client.49357.objecter put_session s=0x7fdbb4010b90 osd=4 4 2024-04-14T15:20:49.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.320+0000 7fdbc6ffd700 15 client.49357.objecter _session_command_op_remove 4 10 2024-04-14T15:20:49.321 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:20:49.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.320+0000 7fdbd7d29700 10 client.49357.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:20:49.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.320+0000 7fdbd7d29700 15 client.49357.objecter _session_command_op_assign -1 11 2024-04-14T15:20:49.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.320+0000 7fdbd7d29700 20 client.49357.objecter _get_session s=0x7fdbb401ed50 osd=5 3 2024-04-14T15:20:49.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.320+0000 7fdbd7d29700 20 client.49357.objecter put_session s=0x7fdbb401ed50 osd=5 3 2024-04-14T15:20:49.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.320+0000 7fdbd7d29700 20 client.49357.objecter _get_session s=0x7fdbb401ed50 osd=5 3 2024-04-14T15:20:49.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.320+0000 7fdbd7d29700 15 client.49357.objecter _session_command_op_remove -1 11 2024-04-14T15:20:49.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.320+0000 7fdbd7d29700 20 client.49357.objecter get_session s=0x7fdbb401ed50 osd=5 3 2024-04-14T15:20:49.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.320+0000 7fdbd7d29700 15 client.49357.objecter _session_command_op_assign 5 11 2024-04-14T15:20:49.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.320+0000 7fdbd7d29700 20 client.49357.objecter put_session s=0x7fdbb401ed50 osd=5 4 2024-04-14T15:20:49.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.320+0000 7fdbd7d29700 10 client.49357.objecter _send_command 11 2024-04-14T15:20:49.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.320+0000 7fdbd7d29700 1 -- 172.21.15.90:0/3573576483 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fdbb4021070 con 0x7fdbb401f470 2024-04-14T15:20:49.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.321+0000 7fdbc6ffd700 10 client.49357.objecter ms_handle_connect 0x7fdbb401f470 2024-04-14T15:20:49.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.322+0000 7fdbc6ffd700 1 -- 172.21.15.90:0/3573576483 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fdbb4021070 con 0x7fdbb401f470 2024-04-14T15:20:49.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.322+0000 7fdbc6ffd700 10 client.49357.objecter ms_dispatch 0x7fdbd0001550 command_reply(tid 11: 0 ) 2024-04-14T15:20:49.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.322+0000 7fdbc6ffd700 10 client.49357.objecter _finish_command 11 = system:0 2024-04-14T15:20:49.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.322+0000 7fdbc6ffd700 20 client.49357.objecter put_session s=0x7fdbb401ed50 osd=5 4 2024-04-14T15:20:49.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.322+0000 7fdbc6ffd700 15 client.49357.objecter _session_command_op_remove 5 11 2024-04-14T15:20:49.327 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:49.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:49.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:20:49.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.338+0000 7fdbd7d29700 10 client.49357.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:20:49.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.338+0000 7fdbd7d29700 15 client.49357.objecter _session_command_op_assign -1 12 2024-04-14T15:20:49.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.338+0000 7fdbd7d29700 20 client.49357.objecter _get_session s=0x7fdbb401ed50 osd=5 3 2024-04-14T15:20:49.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.338+0000 7fdbd7d29700 20 client.49357.objecter put_session s=0x7fdbb401ed50 osd=5 3 2024-04-14T15:20:49.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.338+0000 7fdbd7d29700 20 client.49357.objecter _get_session s=0x7fdbb401ed50 osd=5 3 2024-04-14T15:20:49.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.338+0000 7fdbd7d29700 15 client.49357.objecter _session_command_op_remove -1 12 2024-04-14T15:20:49.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.338+0000 7fdbd7d29700 20 client.49357.objecter get_session s=0x7fdbb401ed50 osd=5 3 2024-04-14T15:20:49.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.338+0000 7fdbd7d29700 15 client.49357.objecter _session_command_op_assign 5 12 2024-04-14T15:20:49.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.338+0000 7fdbd7d29700 20 client.49357.objecter put_session s=0x7fdbb401ed50 osd=5 4 2024-04-14T15:20:49.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.338+0000 7fdbd7d29700 10 client.49357.objecter _send_command 12 2024-04-14T15:20:49.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.338+0000 7fdbd7d29700 1 -- 172.21.15.90:0/3573576483 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdbb4021070 con 0x7fdbb401f470 2024-04-14T15:20:49.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.339+0000 7fdbc6ffd700 1 -- 172.21.15.90:0/3573576483 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fdbb4021070 con 0x7fdbb401f470 2024-04-14T15:20:49.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.339+0000 7fdbc6ffd700 10 client.49357.objecter ms_dispatch 0x7fdbd0001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:20:49.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.339+0000 7fdbc6ffd700 10 client.49357.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:20:49.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.339+0000 7fdbc6ffd700 20 client.49357.objecter put_session s=0x7fdbb401ed50 osd=5 4 2024-04-14T15:20:49.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.339+0000 7fdbc6ffd700 15 client.49357.objecter _session_command_op_remove 5 12 2024-04-14T15:20:49.340 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:20:49.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.340+0000 7fdbd7d29700 10 client.49357.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:20:49.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.340+0000 7fdbd7d29700 15 client.49357.objecter _session_command_op_assign -1 13 2024-04-14T15:20:49.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.340+0000 7fdbd7d29700 20 client.49357.objecter _get_session s=0x7fdbb4021070 osd=6 3 2024-04-14T15:20:49.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.340+0000 7fdbd7d29700 20 client.49357.objecter put_session s=0x7fdbb4021070 osd=6 3 2024-04-14T15:20:49.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.340+0000 7fdbd7d29700 20 client.49357.objecter _get_session s=0x7fdbb4021070 osd=6 3 2024-04-14T15:20:49.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.340+0000 7fdbd7d29700 15 client.49357.objecter _session_command_op_remove -1 13 2024-04-14T15:20:49.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.340+0000 7fdbd7d29700 20 client.49357.objecter get_session s=0x7fdbb4021070 osd=6 3 2024-04-14T15:20:49.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.340+0000 7fdbd7d29700 15 client.49357.objecter _session_command_op_assign 6 13 2024-04-14T15:20:49.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.340+0000 7fdbd7d29700 20 client.49357.objecter put_session s=0x7fdbb4021070 osd=6 4 2024-04-14T15:20:49.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.340+0000 7fdbd7d29700 10 client.49357.objecter _send_command 13 2024-04-14T15:20:49.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.340+0000 7fdbd7d29700 1 -- 172.21.15.90:0/3573576483 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fdbb4021480 con 0x7fdbb4023be0 2024-04-14T15:20:49.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.340+0000 7fdbc6ffd700 10 client.49357.objecter ms_handle_connect 0x7fdbb4023be0 2024-04-14T15:20:49.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.341+0000 7fdbc6ffd700 1 -- 172.21.15.90:0/3573576483 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fdbb4021480 con 0x7fdbb4023be0 2024-04-14T15:20:49.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.341+0000 7fdbc6ffd700 10 client.49357.objecter ms_dispatch 0x7fdbd0001550 command_reply(tid 13: 0 ) 2024-04-14T15:20:49.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.341+0000 7fdbc6ffd700 10 client.49357.objecter _finish_command 13 = system:0 2024-04-14T15:20:49.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.341+0000 7fdbc6ffd700 20 client.49357.objecter put_session s=0x7fdbb4021070 osd=6 4 2024-04-14T15:20:49.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.341+0000 7fdbc6ffd700 15 client.49357.objecter _session_command_op_remove 6 13 2024-04-14T15:20:49.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.359+0000 7fdbd7d29700 10 client.49357.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:20:49.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.359+0000 7fdbd7d29700 15 client.49357.objecter _session_command_op_assign -1 14 2024-04-14T15:20:49.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.359+0000 7fdbd7d29700 20 client.49357.objecter _get_session s=0x7fdbb4021070 osd=6 3 2024-04-14T15:20:49.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.359+0000 7fdbd7d29700 20 client.49357.objecter put_session s=0x7fdbb4021070 osd=6 3 2024-04-14T15:20:49.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.359+0000 7fdbd7d29700 20 client.49357.objecter _get_session s=0x7fdbb4021070 osd=6 3 2024-04-14T15:20:49.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.359+0000 7fdbd7d29700 15 client.49357.objecter _session_command_op_remove -1 14 2024-04-14T15:20:49.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.359+0000 7fdbd7d29700 20 client.49357.objecter get_session s=0x7fdbb4021070 osd=6 3 2024-04-14T15:20:49.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.359+0000 7fdbd7d29700 15 client.49357.objecter _session_command_op_assign 6 14 2024-04-14T15:20:49.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.359+0000 7fdbd7d29700 20 client.49357.objecter put_session s=0x7fdbb4021070 osd=6 4 2024-04-14T15:20:49.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.359+0000 7fdbd7d29700 10 client.49357.objecter _send_command 14 2024-04-14T15:20:49.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.359+0000 7fdbd7d29700 1 -- 172.21.15.90:0/3573576483 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdbb4028ec0 con 0x7fdbb4023be0 2024-04-14T15:20:49.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.361+0000 7fdbc6ffd700 1 -- 172.21.15.90:0/3573576483 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fdbb4028ec0 con 0x7fdbb4023be0 2024-04-14T15:20:49.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.361+0000 7fdbc6ffd700 10 client.49357.objecter ms_dispatch 0x7fdbd0001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:20:49.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.361+0000 7fdbc6ffd700 10 client.49357.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:20:49.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.361+0000 7fdbc6ffd700 20 client.49357.objecter put_session s=0x7fdbb4021070 osd=6 4 2024-04-14T15:20:49.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.361+0000 7fdbc6ffd700 15 client.49357.objecter _session_command_op_remove 6 14 2024-04-14T15:20:49.362 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:20:49.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.361+0000 7fdbd7d29700 10 client.49357.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:20:49.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.361+0000 7fdbd7d29700 15 client.49357.objecter _session_command_op_assign -1 15 2024-04-14T15:20:49.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.361+0000 7fdbd7d29700 20 client.49357.objecter _get_session s=0x7fdbb4028ec0 osd=7 3 2024-04-14T15:20:49.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.361+0000 7fdbd7d29700 20 client.49357.objecter put_session s=0x7fdbb4028ec0 osd=7 3 2024-04-14T15:20:49.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.361+0000 7fdbd7d29700 20 client.49357.objecter _get_session s=0x7fdbb4028ec0 osd=7 3 2024-04-14T15:20:49.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.361+0000 7fdbd7d29700 15 client.49357.objecter _session_command_op_remove -1 15 2024-04-14T15:20:49.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.361+0000 7fdbd7d29700 20 client.49357.objecter get_session s=0x7fdbb4028ec0 osd=7 3 2024-04-14T15:20:49.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.361+0000 7fdbd7d29700 15 client.49357.objecter _session_command_op_assign 7 15 2024-04-14T15:20:49.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.361+0000 7fdbd7d29700 20 client.49357.objecter put_session s=0x7fdbb4028ec0 osd=7 4 2024-04-14T15:20:49.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.361+0000 7fdbd7d29700 10 client.49357.objecter _send_command 15 2024-04-14T15:20:49.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.361+0000 7fdbd7d29700 1 -- 172.21.15.90:0/3573576483 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fdbb4029a40 con 0x7fdbb40295e0 2024-04-14T15:20:49.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.362+0000 7fdbc6ffd700 10 client.49357.objecter ms_handle_connect 0x7fdbb40295e0 2024-04-14T15:20:49.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.362+0000 7fdbc6ffd700 1 -- 172.21.15.90:0/3573576483 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fdbb4029a40 con 0x7fdbb40295e0 2024-04-14T15:20:49.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.362+0000 7fdbc6ffd700 10 client.49357.objecter ms_dispatch 0x7fdbd0001550 command_reply(tid 15: 0 ) 2024-04-14T15:20:49.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.362+0000 7fdbc6ffd700 10 client.49357.objecter _finish_command 15 = system:0 2024-04-14T15:20:49.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.362+0000 7fdbc6ffd700 20 client.49357.objecter put_session s=0x7fdbb4028ec0 osd=7 4 2024-04-14T15:20:49.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.362+0000 7fdbc6ffd700 15 client.49357.objecter _session_command_op_remove 7 15 2024-04-14T15:20:49.370 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:49 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:20:49.371 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:49 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:20:49.371 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:49 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-85","var": "dedup_tier","val": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:20:49.371 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:49 smithi092 ceph-mon[33966]: osdmap e524: 8 total, 8 up, 8 in 2024-04-14T15:20:49.371 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:49 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-14T15:20:49.371 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:49 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-14T15:20:49.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.379+0000 7fdbd7d29700 10 client.49357.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:20:49.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.379+0000 7fdbd7d29700 15 client.49357.objecter _session_command_op_assign -1 16 2024-04-14T15:20:49.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.379+0000 7fdbd7d29700 20 client.49357.objecter _get_session s=0x7fdbb4028ec0 osd=7 3 2024-04-14T15:20:49.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.379+0000 7fdbd7d29700 20 client.49357.objecter put_session s=0x7fdbb4028ec0 osd=7 3 2024-04-14T15:20:49.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.379+0000 7fdbd7d29700 20 client.49357.objecter _get_session s=0x7fdbb4028ec0 osd=7 3 2024-04-14T15:20:49.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.379+0000 7fdbd7d29700 15 client.49357.objecter _session_command_op_remove -1 16 2024-04-14T15:20:49.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.379+0000 7fdbd7d29700 20 client.49357.objecter get_session s=0x7fdbb4028ec0 osd=7 3 2024-04-14T15:20:49.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.379+0000 7fdbd7d29700 15 client.49357.objecter _session_command_op_assign 7 16 2024-04-14T15:20:49.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.379+0000 7fdbd7d29700 20 client.49357.objecter put_session s=0x7fdbb4028ec0 osd=7 4 2024-04-14T15:20:49.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.379+0000 7fdbd7d29700 10 client.49357.objecter _send_command 16 2024-04-14T15:20:49.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.379+0000 7fdbd7d29700 1 -- 172.21.15.90:0/3573576483 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdbb4029a40 con 0x7fdbb40295e0 2024-04-14T15:20:49.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.381+0000 7fdbc6ffd700 1 -- 172.21.15.90:0/3573576483 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fdbb4029a40 con 0x7fdbb40295e0 2024-04-14T15:20:49.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.381+0000 7fdbc6ffd700 10 client.49357.objecter ms_dispatch 0x7fdbd0001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:20:49.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.381+0000 7fdbc6ffd700 10 client.49357.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:20:49.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.381+0000 7fdbc6ffd700 20 client.49357.objecter put_session s=0x7fdbb4028ec0 osd=7 4 2024-04-14T15:20:49.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.381+0000 7fdbc6ffd700 15 client.49357.objecter _session_command_op_remove 7 16 2024-04-14T15:20:49.381 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:20:49.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.381+0000 7fdbd7d29700 10 librados: watch_flush enter 2024-04-14T15:20:49.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.381+0000 7fdbd7d29700 10 librados: watch_flush exit 2024-04-14T15:20:49.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.381+0000 7fdbd7d29700 10 client.49357.objecter close_session for osd.0 2024-04-14T15:20:49.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.381+0000 7fdbd7d29700 1 -- 172.21.15.90:0/3573576483 >> v1:172.21.15.90:6801/3761806885 conn(0x7fdbb4004730 legacy=0x7fdbb4002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:49.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.381+0000 7fdbd7d29700 20 client.49357.objecter put_session s=0x7fdbb40053e0 osd=0 1 2024-04-14T15:20:49.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.381+0000 7fdbd7d29700 10 client.49357.objecter close_session for osd.1 2024-04-14T15:20:49.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.381+0000 7fdbd7d29700 1 -- 172.21.15.90:0/3573576483 >> v1:172.21.15.90:6805/4170361699 conn(0x7fdbb4004280 legacy=0x7fdbb4008d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:49.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.381+0000 7fdbd7d29700 20 client.49357.objecter put_session s=0x7fdbb4004110 osd=1 1 2024-04-14T15:20:49.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.381+0000 7fdbd7d29700 10 client.49357.objecter close_session for osd.2 2024-04-14T15:20:49.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.381+0000 7fdbd7d29700 1 -- 172.21.15.90:0/3573576483 >> v1:172.21.15.90:6809/1686184251 conn(0x7fdbb4008880 legacy=0x7fdbb4013ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:49.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.381+0000 7fdbd7d29700 20 client.49357.objecter put_session s=0x7fdbb4001a00 osd=2 1 2024-04-14T15:20:49.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.381+0000 7fdbd7d29700 10 client.49357.objecter close_session for osd.3 2024-04-14T15:20:49.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.381+0000 7fdbd7d29700 1 -- 172.21.15.90:0/3573576483 >> v1:172.21.15.90:6813/2270166774 conn(0x7fdbb4011360 legacy=0x7fdbb4018d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:49.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.381+0000 7fdbd7d29700 20 client.49357.objecter put_session s=0x7fdbb4008170 osd=3 1 2024-04-14T15:20:49.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.381+0000 7fdbd7d29700 10 client.49357.objecter close_session for osd.4 2024-04-14T15:20:49.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.381+0000 7fdbd7d29700 1 -- 172.21.15.90:0/3573576483 >> v1:172.21.15.92:6800/3887549733 conn(0x7fdbb4016250 legacy=0x7fdbb401d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:49.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.381+0000 7fdbd7d29700 20 client.49357.objecter put_session s=0x7fdbb4010b90 osd=4 1 2024-04-14T15:20:49.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.381+0000 7fdbd7d29700 10 client.49357.objecter close_session for osd.5 2024-04-14T15:20:49.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.381+0000 7fdbd7d29700 1 -- 172.21.15.90:0/3573576483 >> v1:172.21.15.92:6804/3244561440 conn(0x7fdbb401f470 legacy=0x7fdbb401f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:49.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.381+0000 7fdbd7d29700 20 client.49357.objecter put_session s=0x7fdbb401ed50 osd=5 1 2024-04-14T15:20:49.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.381+0000 7fdbd7d29700 10 client.49357.objecter close_session for osd.6 2024-04-14T15:20:49.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.381+0000 7fdbd7d29700 1 -- 172.21.15.90:0/3573576483 >> v1:172.21.15.92:6808/636125963 conn(0x7fdbb4023be0 legacy=0x7fdbb4025fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:49.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.381+0000 7fdbd7d29700 20 client.49357.objecter put_session s=0x7fdbb4021070 osd=6 1 2024-04-14T15:20:49.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.381+0000 7fdbd7d29700 10 client.49357.objecter close_session for osd.7 2024-04-14T15:20:49.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.381+0000 7fdbd7d29700 1 -- 172.21.15.90:0/3573576483 >> v1:172.21.15.92:6812/1773706716 conn(0x7fdbb40295e0 legacy=0x7fdbb40276f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:49.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.381+0000 7fdbd7d29700 20 client.49357.objecter put_session s=0x7fdbb4028ec0 osd=7 1 2024-04-14T15:20:49.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.381+0000 7fdbd7d29700 20 client.49357.objecter shutdown clearing up homeless session... 2024-04-14T15:20:49.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.381+0000 7fdbd7d29700 10 client.49357.objecter successfully canceled tick 2024-04-14T15:20:49.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.381+0000 7fdbd62c6700 1 -- 172.21.15.90:0/3573576483 reap_dead start 2024-04-14T15:20:49.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.381+0000 7fdbd62c6700 1 -- 172.21.15.90:0/3573576483 reap_dead start 2024-04-14T15:20:49.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.381+0000 7fdbd7d29700 1 -- 172.21.15.90:0/3573576483 >> v1:172.21.15.90:6800/420935427 conn(0x7fdbbc0714b0 legacy=0x7fdbbc073970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:49.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.381+0000 7fdbd7d29700 1 -- 172.21.15.90:0/3573576483 >> v1:172.21.15.92:6789/0 conn(0x7fdbd010b7c0 legacy=0x7fdbd01057e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:49.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.381+0000 7fdbd62c6700 1 -- 172.21.15.90:0/3573576483 reap_dead start 2024-04-14T15:20:49.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.381+0000 7fdbd62c6700 1 -- 172.21.15.90:0/3573576483 reap_dead start 2024-04-14T15:20:49.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.381+0000 7fdbd62c6700 1 -- 172.21.15.90:0/3573576483 reap_dead start 2024-04-14T15:20:49.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.381+0000 7fdbd7d29700 1 -- 172.21.15.90:0/3573576483 shutdown_connections 2024-04-14T15:20:49.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.381+0000 7fdbd7d29700 1 -- 172.21.15.90:0/3573576483 >> 172.21.15.90:0/3573576483 conn(0x7fdbd0076ad0 msgr2=0x7fdbd01117d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:20:49.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.381+0000 7fdbd7d29700 1 -- 172.21.15.90:0/3573576483 shutdown_connections 2024-04-14T15:20:49.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.381+0000 7fdbd7d29700 1 -- 172.21.15.90:0/3573576483 wait complete. 2024-04-14T15:20:49.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:49.381+0000 7fdbd7d29700 1 librados: shutdown 2024-04-14T15:20:49.385 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:20:49.385 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:20:49.385 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:20:49.385 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:20:49.385 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:20:49.385 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:20:49.385 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:20:49.385 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:20:49.429 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:20:49.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:49.453 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:49.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:20:49.553 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:20:49.577 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:49.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:49.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:20:49.678 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:49.702 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:49.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:49.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:49.804 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:20:49.828 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:49.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:49.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:20:49.927 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:20:49.929 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:20:49.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:49.961 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:49.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:20:50.062 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:20:50.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:50.087 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:50.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:20:50.188 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:20:50.211 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:50.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:50.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:20:50.313 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:20:50.333 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:50 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:20:50.333 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:50 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:20:50.333 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:50 smithi090 ceph-mon[30399]: pgmap v761: 255 pgs: 16 unknown, 1 peering, 238 active+clean; 8.4 MiB data, 593 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:20:50.334 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:50 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3573576483' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:20:50.334 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:50 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:20:50.334 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:50 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:20:50.334 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:50 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-14T15:20:50.334 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:50 smithi090 ceph-mon[30399]: osdmap e525: 8 total, 8 up, 8 in 2024-04-14T15:20:50.334 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:50 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-14T15:20:50.334 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:50 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-14T15:20:50.334 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:50 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:20:50.334 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:50 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:20:50.334 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:50 smithi090 ceph-mon[41670]: pgmap v761: 255 pgs: 16 unknown, 1 peering, 238 active+clean; 8.4 MiB data, 593 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:20:50.334 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:50 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3573576483' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:20:50.335 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:50 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:20:50.335 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:50 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:20:50.335 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:50 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-14T15:20:50.335 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:50 smithi090 ceph-mon[41670]: osdmap e525: 8 total, 8 up, 8 in 2024-04-14T15:20:50.335 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:50 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-14T15:20:50.335 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:50 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-14T15:20:50.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:50.337 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:50.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:20:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:50 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:20:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:50 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:20:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:50 smithi092 ceph-mon[33966]: pgmap v761: 255 pgs: 16 unknown, 1 peering, 238 active+clean; 8.4 MiB data, 593 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:20:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:50 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3573576483' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:20:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:50 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:20:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:50 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:20:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:50 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-14T15:20:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:50 smithi092 ceph-mon[33966]: osdmap e525: 8 total, 8 up, 8 in 2024-04-14T15:20:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:50 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-14T15:20:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:50 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-14T15:20:50.438 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:20:50.462 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:50.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:50.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:20:50.566 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:50.592 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:50.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:50.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:50.694 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:20:50.718 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:50.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:50.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:20:50.820 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:20:50.844 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:50.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:50.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:20:50.946 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:20:50.975 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:50.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:50.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:20:51.004 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-04-14T15:20:51.004 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-04-14T15:20:51.004 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 0 2024-04-14T15:20:51.004 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-04-14T15:20:51.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:51.037+0000 7ff534c26700 1 -- 172.21.15.90:0/1294518161 >> v1:172.21.15.90:6789/0 conn(0x7ff530108980 legacy=0x7ff530108d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:51.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:51.037+0000 7ff534c26700 1 -- 172.21.15.90:0/1294518161 shutdown_connections 2024-04-14T15:20:51.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:51.037+0000 7ff534c26700 1 -- 172.21.15.90:0/1294518161 >> 172.21.15.90:0/1294518161 conn(0x7ff530076aa0 msgr2=0x7ff530076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:20:51.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:51.037+0000 7ff534c26700 1 -- 172.21.15.90:0/1294518161 shutdown_connections 2024-04-14T15:20:51.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:51.037+0000 7ff534c26700 1 -- 172.21.15.90:0/1294518161 wait complete. 2024-04-14T15:20:51.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:51.037+0000 7ff534c26700 1 librados: starting msgr at 2024-04-14T15:20:51.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:51.037+0000 7ff534c26700 1 librados: starting objecter 2024-04-14T15:20:51.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:51.037+0000 7ff534c26700 1 Processor -- start 2024-04-14T15:20:51.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:51.038+0000 7ff534c26700 1 -- start start 2024-04-14T15:20:51.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:51.038+0000 7ff534c26700 1 librados: setting wanted keys 2024-04-14T15:20:51.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:51.038+0000 7ff534c26700 1 librados: calling monclient init 2024-04-14T15:20:51.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:51.038+0000 7ff534c26700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff5301a9b60 con 0x7ff53010f2f0 2024-04-14T15:20:51.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:51.038+0000 7ff534c26700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff5301aad80 con 0x7ff53010b6f0 2024-04-14T15:20:51.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:51.038+0000 7ff534c26700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff5301abfc0 con 0x7ff530108980 2024-04-14T15:20:51.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:51.038+0000 7ff52ed9d700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7ff53010f2f0 0x7ff5301a82c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:35920/0 (socket says 172.21.15.90:35920) 2024-04-14T15:20:51.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:51.038+0000 7ff52ed9d700 1 -- 172.21.15.90:0/1708639068 learned_addr learned my addr 172.21.15.90:0/1708639068 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:20:51.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:51.038+0000 7ff51f7fe700 10 client.?.objecter ms_handle_connect 0x7ff53010b6f0 2024-04-14T15:20:51.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:51.038+0000 7ff51f7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:20:51.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:51.038+0000 7ff51f7fe700 10 client.?.objecter ms_handle_connect 0x7ff53010f2f0 2024-04-14T15:20:51.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:51.038+0000 7ff51f7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:20:51.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:51.038+0000 7ff51f7fe700 10 client.?.objecter ms_handle_connect 0x7ff530108980 2024-04-14T15:20:51.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:51.038+0000 7ff51f7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:20:51.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:51.038+0000 7ff51f7fe700 1 -- 172.21.15.90:0/1708639068 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2623295906 0 0) 0x7ff5301aad80 con 0x7ff53010b6f0 2024-04-14T15:20:51.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:51.038+0000 7ff51f7fe700 1 -- 172.21.15.90:0/1708639068 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff5180031f0 con 0x7ff53010b6f0 2024-04-14T15:20:51.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:51.038+0000 7ff51f7fe700 1 -- 172.21.15.90:0/1708639068 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3132129936 0 0) 0x7ff5301abfc0 con 0x7ff530108980 2024-04-14T15:20:51.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:51.038+0000 7ff51f7fe700 1 -- 172.21.15.90:0/1708639068 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff5301aad80 con 0x7ff530108980 2024-04-14T15:20:51.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:51.038+0000 7ff51f7fe700 1 -- 172.21.15.90:0/1708639068 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3004837395 0 0) 0x7ff5301a9b60 con 0x7ff53010f2f0 2024-04-14T15:20:51.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:51.038+0000 7ff51f7fe700 1 -- 172.21.15.90:0/1708639068 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff5301abfc0 con 0x7ff53010f2f0 2024-04-14T15:20:51.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:51.038+0000 7ff51f7fe700 1 -- 172.21.15.90:0/1708639068 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3659216263 0 0) 0x7ff5180031f0 con 0x7ff53010b6f0 2024-04-14T15:20:51.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:51.038+0000 7ff51f7fe700 1 -- 172.21.15.90:0/1708639068 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff5301a9b60 con 0x7ff53010b6f0 2024-04-14T15:20:51.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:51.038+0000 7ff51f7fe700 1 -- 172.21.15.90:0/1708639068 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7ff524001110 con 0x7ff53010b6f0 2024-04-14T15:20:51.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:51.039+0000 7ff51f7fe700 1 -- 172.21.15.90:0/1708639068 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3522322319 0 0) 0x7ff5301abfc0 con 0x7ff53010f2f0 2024-04-14T15:20:51.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:51.039+0000 7ff51f7fe700 1 -- 172.21.15.90:0/1708639068 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff5180031f0 con 0x7ff53010f2f0 2024-04-14T15:20:51.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:51.039+0000 7ff51f7fe700 1 -- 172.21.15.90:0/1708639068 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7ff528001c30 con 0x7ff53010f2f0 2024-04-14T15:20:51.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:51.039+0000 7ff51f7fe700 1 -- 172.21.15.90:0/1708639068 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 144594024 0 0) 0x7ff5301a9b60 con 0x7ff53010b6f0 2024-04-14T15:20:51.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:51.039+0000 7ff51f7fe700 1 -- 172.21.15.90:0/1708639068 >> v1:172.21.15.90:6790/0 conn(0x7ff530108980 legacy=0x7ff530107ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:51.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:51.039+0000 7ff51f7fe700 1 -- 172.21.15.90:0/1708639068 >> v1:172.21.15.90:6789/0 conn(0x7ff53010f2f0 legacy=0x7ff5301a82c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:51.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:51.039+0000 7ff51f7fe700 1 -- 172.21.15.90:0/1708639068 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5301ad200 con 0x7ff53010b6f0 2024-04-14T15:20:51.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:51.039+0000 7ff534c26700 1 -- 172.21.15.90:0/1708639068 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff5301a9d90 con 0x7ff53010b6f0 2024-04-14T15:20:51.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:51.039+0000 7ff534c26700 10 client.49363.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:20:51.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:51.039+0000 7ff534c26700 1 -- 172.21.15.90:0/1708639068 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff5301aa250 con 0x7ff53010b6f0 2024-04-14T15:20:51.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:51.039+0000 7ff534c26700 1 librados: init done 2024-04-14T15:20:51.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:51.039+0000 7ff51f7fe700 1 -- 172.21.15.90:0/1708639068 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7ff524003b20 con 0x7ff53010b6f0 2024-04-14T15:20:51.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:51.039+0000 7ff51f7fe700 1 -- 172.21.15.90:0/1708639068 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7ff5240049a0 con 0x7ff53010b6f0 2024-04-14T15:20:51.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:51.040+0000 7ff534c26700 1 -- 172.21.15.90:0/1708639068 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff510005120 con 0x7ff53010b6f0 2024-04-14T15:20:51.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:51.040+0000 7ff51f7fe700 1 -- 172.21.15.90:0/1708639068 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7ff524003680 con 0x7ff53010b6f0 2024-04-14T15:20:51.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:51.041+0000 7ff51f7fe700 1 -- 172.21.15.90:0/1708639068 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(526..526 src has 467..526) ==== 8264+0+0 (unknown 209632739 0 0) 0x7ff5240896c0 con 0x7ff53010b6f0 2024-04-14T15:20:51.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:51.041+0000 7ff51f7fe700 10 client.49363.objecter ms_dispatch 0x7ff530001550 osd_map(526..526 src has 467..526) 2024-04-14T15:20:51.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:51.041+0000 7ff51f7fe700 3 client.49363.objecter handle_osd_map got epochs [526,526] > 0 2024-04-14T15:20:51.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:51.041+0000 7ff51f7fe700 3 client.49363.objecter handle_osd_map decoding full epoch 526 2024-04-14T15:20:51.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:51.041+0000 7ff51f7fe700 20 client.49363.objecter dump_active .. 0 homeless 2024-04-14T15:20:51.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:51.041+0000 7ff51f7fe700 10 client.49363.objecter ms_handle_connect 0x7ff5180710d0 2024-04-14T15:20:51.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:51.045+0000 7ff51f7fe700 1 -- 172.21.15.90:0/1708639068 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7ff524056070 con 0x7ff53010b6f0 2024-04-14T15:20:51.077 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:20:51.100 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:51.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:51.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:20:51.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:51 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:20:51.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:51 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-85","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-14T15:20:51.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:51 smithi090 ceph-mon[41670]: osdmap e526: 8 total, 8 up, 8 in 2024-04-14T15:20:51.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:51 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:20:51.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:51 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-85","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-14T15:20:51.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:51 smithi090 ceph-mon[30399]: osdmap e526: 8 total, 8 up, 8 in 2024-04-14T15:20:51.202 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:20:51.226 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:51.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:51.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:20:51.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:51.296+0000 7ff534c26700 1 -- 172.21.15.90:0/1708639068 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd out", "ids": ["6"]} v 0) -- 0x7ff510005410 con 0x7ff53010b6f0 2024-04-14T15:20:51.328 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:20:51.331 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:51 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:20:51.331 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:51 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-85","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-14T15:20:51.331 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:51 smithi092 ceph-mon[33966]: osdmap e526: 8 total, 8 up, 8 in 2024-04-14T15:20:51.352 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:51.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:51.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:20:51.454 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:20:51.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:51.478 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:51.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:20:51.579 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:20:51.597 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:51 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mon-a[30376]: 2024-04-14T15:20:51.296+0000 7f0166ba8700 -1 mon.a@0(leader).osd e526 definitely_dead 0 2024-04-14T15:20:51.603 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:51.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:51.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:20:51.656 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-14T15:20:51.704 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:20:51.728 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:51.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:51.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:20:51.830 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:20:51.853 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:51.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:51.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:20:51.902 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:20:51.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:51.953+0000 7ff51f7fe700 1 -- 172.21.15.90:0/1708639068 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd out", "ids": ["6"]}]=0 marked out osd.6. v527) ==== 87+0+0 (unknown 924598994 0 0) 0x7ff524059d20 con 0x7ff53010b6f0 2024-04-14T15:20:51.955 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:20:51.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:51.957+0000 7ff534c26700 10 librados: watch_flush enter 2024-04-14T15:20:51.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:51.957+0000 7ff534c26700 10 librados: watch_flush exit 2024-04-14T15:20:51.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:51.957+0000 7ff534c26700 20 client.49363.objecter shutdown clearing up homeless session... 2024-04-14T15:20:51.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:51.957+0000 7ff534c26700 10 client.49363.objecter successfully canceled tick 2024-04-14T15:20:51.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:51.957+0000 7ff534c26700 1 -- 172.21.15.90:0/1708639068 >> v1:172.21.15.90:6800/420935427 conn(0x7ff5180710d0 legacy=0x7ff518073590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:51.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:51.957+0000 7ff534c26700 1 -- 172.21.15.90:0/1708639068 >> v1:172.21.15.92:6789/0 conn(0x7ff53010b6f0 legacy=0x7ff5301a4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:51.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:51.960+0000 7ff534c26700 1 -- 172.21.15.90:0/1708639068 shutdown_connections 2024-04-14T15:20:51.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:51.960+0000 7ff534c26700 1 -- 172.21.15.90:0/1708639068 >> 172.21.15.90:0/1708639068 conn(0x7ff530076aa0 msgr2=0x7ff530109620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:20:51.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:51.960+0000 7ff534c26700 1 -- 172.21.15.90:0/1708639068 shutdown_connections 2024-04-14T15:20:51.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:51.960+0000 7ff534c26700 1 -- 172.21.15.90:0/1708639068 wait complete. 2024-04-14T15:20:51.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:51.961+0000 7ff534c26700 1 librados: shutdown 2024-04-14T15:20:51.966 INFO:teuthology.orchestra.run.smithi090.stderr:marked out osd.6. 2024-04-14T15:20:51.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:51.998 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:51.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:20:52.101 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:20:52.124 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:52.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:52.126 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:20:52.226 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:52.251 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:52.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:52.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:52.301 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:52 smithi092 ceph-mon[33966]: pgmap v764: 254 pgs: 1 clean+premerge+peered, 253 active+clean; 8.4 MiB data, 597 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:20:52.301 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:52 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-04-14T15:20:52.301 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:52 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1708639068' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-04-14T15:20:52.302 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:52 smithi092 ceph-mon[33966]: from='client.49363 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-04-14T15:20:52.302 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:52 smithi092 ceph-mon[33966]: Client client.admin marked osd.6 out, while it was still marked up 2024-04-14T15:20:52.302 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:52 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:20:52.302 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:52 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:20:52.302 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:52 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:20:52.302 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:52 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:20:52.302 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:52 smithi092 ceph-mon[33966]: Adjusting osd_memory_target on smithi090 to 3248M 2024-04-14T15:20:52.302 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:52 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:20:52.302 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:52 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:20:52.302 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:52 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:20:52.302 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:52 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:20:52.302 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:52 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:20:52.302 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:52 smithi092 ceph-mon[33966]: Adjusting osd_memory_target on smithi092 to 3248M 2024-04-14T15:20:52.303 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:52 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:20:52.303 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:52 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:20:52.303 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:52 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:20:52.303 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:52 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:20:52.303 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:52 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "26"}]': finished 2024-04-14T15:20:52.303 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:52 smithi092 ceph-mon[33966]: from='client.49363 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-04-14T15:20:52.303 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:52 smithi092 ceph-mon[33966]: osdmap e527: 8 total, 8 up, 7 in 2024-04-14T15:20:52.331 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:52 smithi090 ceph-mon[41670]: pgmap v764: 254 pgs: 1 clean+premerge+peered, 253 active+clean; 8.4 MiB data, 597 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:20:52.331 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:52 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-04-14T15:20:52.331 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:52 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1708639068' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-04-14T15:20:52.331 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:52 smithi090 ceph-mon[41670]: from='client.49363 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-04-14T15:20:52.331 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:52 smithi090 ceph-mon[41670]: Client client.admin marked osd.6 out, while it was still marked up 2024-04-14T15:20:52.331 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:52 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:20:52.331 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:52 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:20:52.331 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:52 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:20:52.331 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:52 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:20:52.331 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:52 smithi090 ceph-mon[41670]: Adjusting osd_memory_target on smithi090 to 3248M 2024-04-14T15:20:52.331 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:52 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:20:52.331 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:52 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:20:52.331 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:52 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:20:52.331 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:52 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:20:52.331 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:52 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:20:52.332 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:52 smithi090 ceph-mon[41670]: Adjusting osd_memory_target on smithi092 to 3248M 2024-04-14T15:20:52.332 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:52 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:20:52.332 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:52 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:20:52.332 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:52 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:20:52.332 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:52 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:20:52.332 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:52 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "26"}]': finished 2024-04-14T15:20:52.332 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:52 smithi090 ceph-mon[41670]: from='client.49363 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-04-14T15:20:52.332 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:52 smithi090 ceph-mon[41670]: osdmap e527: 8 total, 8 up, 7 in 2024-04-14T15:20:52.332 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:52 smithi090 ceph-mon[30399]: pgmap v764: 254 pgs: 1 clean+premerge+peered, 253 active+clean; 8.4 MiB data, 597 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:20:52.332 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:52 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-04-14T15:20:52.332 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:52 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1708639068' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-04-14T15:20:52.332 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:52 smithi090 ceph-mon[30399]: from='client.49363 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-04-14T15:20:52.332 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:52 smithi090 ceph-mon[30399]: Client client.admin marked osd.6 out, while it was still marked up 2024-04-14T15:20:52.332 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:52 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:20:52.333 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:52 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:20:52.333 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:52 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:20:52.333 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:52 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:20:52.333 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:52 smithi090 ceph-mon[30399]: Adjusting osd_memory_target on smithi090 to 3248M 2024-04-14T15:20:52.333 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:52 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:20:52.333 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:52 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:20:52.333 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:52 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:20:52.333 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:52 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:20:52.333 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:52 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:20:52.333 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:52 smithi090 ceph-mon[30399]: Adjusting osd_memory_target on smithi092 to 3248M 2024-04-14T15:20:52.333 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:52 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:20:52.333 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:52 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:20:52.333 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:52 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:20:52.333 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:52 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:20:52.333 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:52 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "26"}]': finished 2024-04-14T15:20:52.333 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:52 smithi090 ceph-mon[30399]: from='client.49363 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-04-14T15:20:52.334 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:52 smithi090 ceph-mon[30399]: osdmap e527: 8 total, 8 up, 7 in 2024-04-14T15:20:52.353 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:20:52.360 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-14T15:20:52.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:52.378 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:52.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:20:52.479 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:20:52.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:52.504 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:52.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:20:52.605 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:20:52.628 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:52.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:52.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:20:52.730 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:20:52.755 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:52.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:52.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:20:52.856 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:20:52.880 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:52.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:52.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:20:52.981 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:20:53.019 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:53.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:53.020 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:20:53.121 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:20:53.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.146+0000 7eff2dbbd700 1 -- 172.21.15.90:0/2891955529 >> v1:172.21.15.90:6789/0 conn(0x7eff2810b750 legacy=0x7eff2810db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:53.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.146+0000 7eff2dbbd700 1 -- 172.21.15.90:0/2891955529 shutdown_connections 2024-04-14T15:20:53.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.146+0000 7eff2dbbd700 1 -- 172.21.15.90:0/2891955529 >> 172.21.15.90:0/2891955529 conn(0x7eff28076aa0 msgr2=0x7eff28076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:20:53.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.146+0000 7eff2dbbd700 1 -- 172.21.15.90:0/2891955529 shutdown_connections 2024-04-14T15:20:53.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.146+0000 7eff2dbbd700 1 -- 172.21.15.90:0/2891955529 wait complete. 2024-04-14T15:20:53.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.147+0000 7eff2dbbd700 1 librados: starting msgr at 2024-04-14T15:20:53.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.147+0000 7eff2dbbd700 1 librados: starting objecter 2024-04-14T15:20:53.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.147+0000 7eff2dbbd700 1 Processor -- start 2024-04-14T15:20:53.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.147+0000 7eff2dbbd700 1 -- start start 2024-04-14T15:20:53.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.147+0000 7eff2dbbd700 1 librados: setting wanted keys 2024-04-14T15:20:53.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.147+0000 7eff2dbbd700 1 librados: calling monclient init 2024-04-14T15:20:53.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.147+0000 7eff2dbbd700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7eff28101ce0 con 0x7eff2810b750 2024-04-14T15:20:53.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.147+0000 7eff2dbbd700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7eff281a8f00 con 0x7eff2810f320 2024-04-14T15:20:53.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.147+0000 7eff2dbbd700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7eff281aa0e0 con 0x7eff28107880 2024-04-14T15:20:53.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.147+0000 7eff277fe700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7eff28107880 0x7eff280feb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:59198/0 (socket says 172.21.15.90:59198) 2024-04-14T15:20:53.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.147+0000 7eff277fe700 1 -- 172.21.15.90:0/1471623840 learned_addr learned my addr 172.21.15.90:0/1471623840 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:20:53.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.148+0000 7eff24ff9700 10 client.?.objecter ms_handle_connect 0x7eff28107880 2024-04-14T15:20:53.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.148+0000 7eff24ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:20:53.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.148+0000 7eff24ff9700 10 client.?.objecter ms_handle_connect 0x7eff2810b750 2024-04-14T15:20:53.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.148+0000 7eff24ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:20:53.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.148+0000 7eff24ff9700 10 client.?.objecter ms_handle_connect 0x7eff2810f320 2024-04-14T15:20:53.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.148+0000 7eff24ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:20:53.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.148+0000 7eff24ff9700 1 -- 172.21.15.90:0/1471623840 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 651504200 0 0) 0x7eff28101ce0 con 0x7eff2810b750 2024-04-14T15:20:53.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.148+0000 7eff24ff9700 1 -- 172.21.15.90:0/1471623840 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff100035b0 con 0x7eff2810b750 2024-04-14T15:20:53.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.148+0000 7eff24ff9700 1 -- 172.21.15.90:0/1471623840 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 477713841 0 0) 0x7eff281a8f00 con 0x7eff2810f320 2024-04-14T15:20:53.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.148+0000 7eff24ff9700 1 -- 172.21.15.90:0/1471623840 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff28101ce0 con 0x7eff2810f320 2024-04-14T15:20:53.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.148+0000 7eff24ff9700 1 -- 172.21.15.90:0/1471623840 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4046894374 0 0) 0x7eff281aa0e0 con 0x7eff28107880 2024-04-14T15:20:53.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.148+0000 7eff24ff9700 1 -- 172.21.15.90:0/1471623840 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff281a8f00 con 0x7eff28107880 2024-04-14T15:20:53.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.148+0000 7eff24ff9700 1 -- 172.21.15.90:0/1471623840 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4046807852 0 0) 0x7eff28101ce0 con 0x7eff2810f320 2024-04-14T15:20:53.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.148+0000 7eff24ff9700 1 -- 172.21.15.90:0/1471623840 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7eff281aa0e0 con 0x7eff2810f320 2024-04-14T15:20:53.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.148+0000 7eff24ff9700 1 -- 172.21.15.90:0/1471623840 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4015985386 0 0) 0x7eff100035b0 con 0x7eff2810b750 2024-04-14T15:20:53.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.148+0000 7eff24ff9700 1 -- 172.21.15.90:0/1471623840 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28101ce0 con 0x7eff2810b750 2024-04-14T15:20:53.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.148+0000 7eff24ff9700 1 -- 172.21.15.90:0/1471623840 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7eff20001aa0 con 0x7eff2810f320 2024-04-14T15:20:53.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.148+0000 7eff24ff9700 1 -- 172.21.15.90:0/1471623840 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4131210400 0 0) 0x7eff281a8f00 con 0x7eff28107880 2024-04-14T15:20:53.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.148+0000 7eff24ff9700 1 -- 172.21.15.90:0/1471623840 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7eff100035b0 con 0x7eff28107880 2024-04-14T15:20:53.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.148+0000 7eff24ff9700 1 -- 172.21.15.90:0/1471623840 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7eff1c0015d0 con 0x7eff2810b750 2024-04-14T15:20:53.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.148+0000 7eff24ff9700 1 -- 172.21.15.90:0/1471623840 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7eff18001830 con 0x7eff28107880 2024-04-14T15:20:53.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.148+0000 7eff24ff9700 1 -- 172.21.15.90:0/1471623840 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 754822905 0 0) 0x7eff281aa0e0 con 0x7eff2810f320 2024-04-14T15:20:53.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.148+0000 7eff24ff9700 1 -- 172.21.15.90:0/1471623840 >> v1:172.21.15.90:6790/0 conn(0x7eff28107880 legacy=0x7eff280feb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:53.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.148+0000 7eff24ff9700 1 -- 172.21.15.90:0/1471623840 >> v1:172.21.15.90:6789/0 conn(0x7eff2810b750 legacy=0x7eff280ff240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:53.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.148+0000 7eff24ff9700 1 -- 172.21.15.90:0/1471623840 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff281ab2c0 con 0x7eff2810f320 2024-04-14T15:20:53.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.148+0000 7eff2dbbd700 1 -- 172.21.15.90:0/1471623840 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7eff281aa310 con 0x7eff2810f320 2024-04-14T15:20:53.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.148+0000 7eff2dbbd700 10 client.49369.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:20:53.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.148+0000 7eff2dbbd700 1 -- 172.21.15.90:0/1471623840 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7eff281aa850 con 0x7eff2810f320 2024-04-14T15:20:53.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.148+0000 7eff2dbbd700 1 librados: init done 2024-04-14T15:20:53.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.148+0000 7eff24ff9700 1 -- 172.21.15.90:0/1471623840 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7eff20001d10 con 0x7eff2810f320 2024-04-14T15:20:53.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.148+0000 7eff24ff9700 1 -- 172.21.15.90:0/1471623840 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7eff20004db0 con 0x7eff2810f320 2024-04-14T15:20:53.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:53.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.149+0000 7eff2dbbd700 1 -- 172.21.15.90:0/1471623840 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff14005120 con 0x7eff2810f320 2024-04-14T15:20:53.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.150+0000 7eff24ff9700 1 -- 172.21.15.90:0/1471623840 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7eff20003a00 con 0x7eff2810f320 2024-04-14T15:20:53.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.150+0000 7eff24ff9700 1 -- 172.21.15.90:0/1471623840 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(528..528 src has 467..528) ==== 8674+0+0 (unknown 207271150 0 0) 0x7eff20089c90 con 0x7eff2810f320 2024-04-14T15:20:53.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.150+0000 7eff24ff9700 10 client.49369.objecter ms_dispatch 0x7eff28001550 osd_map(528..528 src has 467..528) 2024-04-14T15:20:53.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.150+0000 7eff24ff9700 3 client.49369.objecter handle_osd_map got epochs [528,528] > 0 2024-04-14T15:20:53.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.150+0000 7eff24ff9700 3 client.49369.objecter handle_osd_map decoding full epoch 528 2024-04-14T15:20:53.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.150+0000 7eff24ff9700 20 client.49369.objecter dump_active .. 0 homeless 2024-04-14T15:20:53.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.150+0000 7eff24ff9700 10 client.49369.objecter ms_handle_connect 0x7eff10071410 2024-04-14T15:20:53.155 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:53.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:20:53.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.155+0000 7eff24ff9700 1 -- 172.21.15.90:0/1471623840 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7eff200564a0 con 0x7eff2810f320 2024-04-14T15:20:53.256 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:20:53.258 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:20:53.279 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:53.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:53.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:20:53.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.379+0000 7eff2dbbd700 1 -- 172.21.15.90:0/1471623840 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7eff14005410 con 0x7eff2810f320 2024-04-14T15:20:53.381 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:20:53.406 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:53.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:53.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:20:53.507 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:53.532 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:53.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:53.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:53.613 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-04-14T15:20:53.613 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-04-14T15:20:53.613 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-04-14T15:20:53.613 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-04-14T15:20:53.613 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-04-14T15:20:53.633 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:20:53.658 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:53.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:53.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:20:53.760 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:20:53.784 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:53.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:53.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:20:53.885 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:20:53.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:53.909 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:53.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:20:53.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.967+0000 7eff24ff9700 1 -- 172.21.15.90:0/1471623840 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v529) ==== 97+0+0 (unknown 1732369464 0 0) 0x7eff2005a150 con 0x7eff2810f320 2024-04-14T15:20:53.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.970+0000 7eff2dbbd700 10 librados: watch_flush enter 2024-04-14T15:20:53.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.970+0000 7eff2dbbd700 10 librados: watch_flush exit 2024-04-14T15:20:53.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.970+0000 7eff2dbbd700 20 client.49369.objecter shutdown clearing up homeless session... 2024-04-14T15:20:53.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.970+0000 7eff2dbbd700 10 client.49369.objecter successfully canceled tick 2024-04-14T15:20:53.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.973+0000 7eff2dbbd700 1 -- 172.21.15.90:0/1471623840 >> v1:172.21.15.90:6800/420935427 conn(0x7eff10071410 legacy=0x7eff100738d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:53.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.973+0000 7eff2dbbd700 1 -- 172.21.15.90:0/1471623840 >> v1:172.21.15.92:6789/0 conn(0x7eff2810f320 legacy=0x7eff281a67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:53.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.974+0000 7eff2dbbd700 1 -- 172.21.15.90:0/1471623840 shutdown_connections 2024-04-14T15:20:53.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.974+0000 7eff2dbbd700 1 -- 172.21.15.90:0/1471623840 >> 172.21.15.90:0/1471623840 conn(0x7eff28076aa0 msgr2=0x7eff281117e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:20:53.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.975+0000 7eff2dbbd700 1 -- 172.21.15.90:0/1471623840 shutdown_connections 2024-04-14T15:20:53.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.976+0000 7eff2dbbd700 1 -- 172.21.15.90:0/1471623840 wait complete. 2024-04-14T15:20:53.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:53.977+0000 7eff2dbbd700 1 librados: shutdown 2024-04-14T15:20:53.981 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is set 2024-04-14T15:20:54.010 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:20:54.034 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:54.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:54.035 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:20:54.135 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:20:54.159 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:54.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:54.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:20:54.191 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:20:53 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:20:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:20:54.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:53 smithi090 ceph-mon[30399]: osdmap e528: 8 total, 8 up, 7 in 2024-04-14T15:20:54.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:53 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:20:54.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:53 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:20:54.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:53 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-85"}]: dispatch 2024-04-14T15:20:54.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:53 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-85"}]: dispatch 2024-04-14T15:20:54.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:53 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1471623840' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:20:54.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:53 smithi090 ceph-mon[30399]: from='client.49369 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:20:54.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:53 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:20:54.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:53 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:20:54.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:53 smithi090 ceph-mon[41670]: osdmap e528: 8 total, 8 up, 7 in 2024-04-14T15:20:54.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:53 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:20:54.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:53 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:20:54.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:53 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-85"}]: dispatch 2024-04-14T15:20:54.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:53 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-85"}]: dispatch 2024-04-14T15:20:54.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:53 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1471623840' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:20:54.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:53 smithi090 ceph-mon[41670]: from='client.49369 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:20:54.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:53 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:20:54.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:53 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:20:54.261 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:20:54.262 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:53 smithi092 ceph-mon[33966]: osdmap e528: 8 total, 8 up, 7 in 2024-04-14T15:20:54.262 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:53 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:20:54.262 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:53 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:20:54.262 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:53 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-85"}]: dispatch 2024-04-14T15:20:54.263 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:53 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-85"}]: dispatch 2024-04-14T15:20:54.263 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:53 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1471623840' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:20:54.263 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:53 smithi092 ceph-mon[33966]: from='client.49369 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:20:54.263 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:53 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:20:54.263 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:53 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:20:54.287 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:54.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:54.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:20:54.292 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-14T15:20:54.388 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:20:54.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:54.413 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:54.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:20:54.514 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:20:54.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:54.541 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:54.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:20:54.579 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:20:54.642 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:20:54.666 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:54.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:54.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:20:54.767 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:20:54.791 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:54.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:54.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:20:54.892 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:20:54.916 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:54.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:54.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:20:55.018 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:20:55.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:55.042 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:55.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:20:55.143 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:20:55.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:55.167 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:55.167 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:20:55.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:54 smithi090 ceph-mon[30399]: pgmap v767: 254 pgs: 1 clean+premerge+peered, 253 active+clean; 8.4 MiB data, 523 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:20:55.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:54 smithi090 ceph-mon[30399]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:20:55.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:54 smithi090 ceph-mon[30399]: from='client.49369 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-14T15:20:55.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:54 smithi090 ceph-mon[30399]: osdmap e529: 8 total, 8 up, 7 in 2024-04-14T15:20:55.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:54 smithi090 ceph-mon[30399]: 1.c starting backfill to osd.4 from (0'0,0'0] MAX to 91'130 2024-04-14T15:20:55.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:54 smithi090 ceph-mon[41670]: pgmap v767: 254 pgs: 1 clean+premerge+peered, 253 active+clean; 8.4 MiB data, 523 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:20:55.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:54 smithi090 ceph-mon[41670]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:20:55.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:54 smithi090 ceph-mon[41670]: from='client.49369 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-14T15:20:55.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:54 smithi090 ceph-mon[41670]: osdmap e529: 8 total, 8 up, 7 in 2024-04-14T15:20:55.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:54 smithi090 ceph-mon[41670]: 1.c starting backfill to osd.4 from (0'0,0'0] MAX to 91'130 2024-04-14T15:20:55.231 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:54 smithi092 ceph-mon[33966]: pgmap v767: 254 pgs: 1 clean+premerge+peered, 253 active+clean; 8.4 MiB data, 523 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:20:55.231 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:54 smithi092 ceph-mon[33966]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:20:55.231 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:54 smithi092 ceph-mon[33966]: from='client.49369 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-14T15:20:55.231 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:54 smithi092 ceph-mon[33966]: osdmap e529: 8 total, 8 up, 7 in 2024-04-14T15:20:55.231 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:54 smithi092 ceph-mon[33966]: 1.c starting backfill to osd.4 from (0'0,0'0] MAX to 91'130 2024-04-14T15:20:55.268 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:20:55.291 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:55.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:55.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:20:55.392 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:20:55.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:55.417 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:55.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:20:55.518 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:20:55.541 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:55.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:55.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:20:55.619 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:20:55 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:20:55.643 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:20:55.667 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:55.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:55.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:20:55.769 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:55.792 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:55.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:55.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:55.894 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:20:55.918 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:55.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:55.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:20:55.985 INFO:tasks.workunit.client.0.smithi090.stdout: OK ] LibRadosTwoPoolsPP.ProxyRead (18032 ms) 2024-04-14T15:20:55.985 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2024-04-14T15:20:55.985 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (22048 ms) 2024-04-14T15:20:55.985 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2024-04-14T15:20:55.986 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (2924 ms) 2024-04-14T15:20:55.986 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2024-04-14T15:20:55.986 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: require_osd_release = squid 2024-04-14T15:20:55.986 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (3005 ms) 2024-04-14T15:20:55.986 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2024-04-14T15:20:55.986 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (3038 ms) 2024-04-14T15:20:55.986 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2024-04-14T15:20:55.986 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: require_osd_release = squid 2024-04-14T15:20:55.986 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (2962 ms) 2024-04-14T15:20:55.986 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2024-04-14T15:20:55.986 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: require_osd_release = squid 2024-04-14T15:20:55.986 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (4022 ms) 2024-04-14T15:20:55.986 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2024-04-14T15:20:55.986 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: require_osd_release = squid 2024-04-14T15:20:55.986 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (37180 ms) 2024-04-14T15:20:55.987 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2024-04-14T15:20:55.987 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: require_osd_release = squid 2024-04-14T15:20:55.987 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (16835 ms) 2024-04-14T15:20:55.987 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2024-04-14T15:20:55.987 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: require_osd_release = squid 2024-04-14T15:20:55.987 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (4023 ms) 2024-04-14T15:20:55.987 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2024-04-14T15:20:55.987 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: require_osd_release = squid 2024-04-14T15:20:55.987 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (3027 ms) 2024-04-14T15:20:55.987 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2024-04-14T15:20:55.987 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: require_osd_release = squid 2024-04-14T15:20:55.987 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (24967 ms) 2024-04-14T15:20:55.987 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2024-04-14T15:20:55.987 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: require_osd_release = squid 2024-04-14T15:20:55.987 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (14975 ms) 2024-04-14T15:20:55.987 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2024-04-14T15:20:55.988 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: require_osd_release = squid 2024-04-14T15:20:55.988 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (5040 ms) 2024-04-14T15:20:55.988 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictPromote 2024-04-14T15:20:55.988 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: require_osd_release = squid 2024-04-14T15:20:55.988 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictPromote (3952 ms) 2024-04-14T15:20:55.988 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2024-04-14T15:20:55.988 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: require_osd_release = squid 2024-04-14T15:20:55.988 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: waiting for scrubs... 2024-04-14T15:20:55.988 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: done waiting 2024-04-14T15:20:55.988 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (25041 ms) 2024-04-14T15:20:55.988 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2024-04-14T15:20:55.988 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: require_osd_release = squid 2024-04-14T15:20:55.988 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (9999 ms) 2024-04-14T15:20:55.988 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2024-04-14T15:20:55.988 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: require_osd_release = squid 2024-04-14T15:20:55.989 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (8977 ms) 2024-04-14T15:20:55.989 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2024-04-14T15:20:55.989 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: require_osd_release = squid 2024-04-14T15:20:55.989 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (9028 ms) 2024-04-14T15:20:55.989 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlush 2024-04-14T15:20:56.020 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:20:56.044 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:56.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:56.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:20:56.145 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:20:56.169 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:56.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:56.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:20:56.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:55 smithi090 ceph-mon[41670]: osdmap e530: 8 total, 8 up, 7 in 2024-04-14T15:20:56.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:55 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:20:56.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:55 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:20:56.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:55 smithi090 ceph-mon[30399]: osdmap e530: 8 total, 8 up, 7 in 2024-04-14T15:20:56.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:55 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:20:56.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:55 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:20:56.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.170+0000 7f7838bea700 1 -- 172.21.15.90:0/4030399905 >> v1:172.21.15.92:6789/0 conn(0x7f783410a630 legacy=0x7f783410ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:56.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.170+0000 7f7838bea700 1 -- 172.21.15.90:0/4030399905 shutdown_connections 2024-04-14T15:20:56.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.170+0000 7f7838bea700 1 -- 172.21.15.90:0/4030399905 >> 172.21.15.90:0/4030399905 conn(0x7f78340fd2b0 msgr2=0x7f78340ff6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:20:56.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.170+0000 7f7838bea700 1 -- 172.21.15.90:0/4030399905 shutdown_connections 2024-04-14T15:20:56.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.170+0000 7f7838bea700 1 -- 172.21.15.90:0/4030399905 wait complete. 2024-04-14T15:20:56.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.171+0000 7f7838bea700 1 librados: starting msgr at 2024-04-14T15:20:56.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.171+0000 7f7838bea700 1 librados: starting objecter 2024-04-14T15:20:56.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.171+0000 7f7838bea700 1 Processor -- start 2024-04-14T15:20:56.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.171+0000 7f7838bea700 1 -- start start 2024-04-14T15:20:56.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.171+0000 7f7838bea700 1 librados: setting wanted keys 2024-04-14T15:20:56.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.171+0000 7f7838bea700 1 librados: calling monclient init 2024-04-14T15:20:56.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.171+0000 7f7838bea700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f78341a9b00 con 0x7f783410a630 2024-04-14T15:20:56.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.171+0000 7f7838bea700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f78341aad20 con 0x7f783410e200 2024-04-14T15:20:56.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.171+0000 7f7838bea700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f78341abf60 con 0x7f7834107780 2024-04-14T15:20:56.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.171+0000 7f783259c700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f7834107780 0x7f783410da10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:59218/0 (socket says 172.21.15.90:59218) 2024-04-14T15:20:56.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.171+0000 7f783259c700 1 -- 172.21.15.90:0/1032290602 learned_addr learned my addr 172.21.15.90:0/1032290602 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:20:56.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.171+0000 7f782b7fe700 10 client.?.objecter ms_handle_connect 0x7f783410a630 2024-04-14T15:20:56.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.171+0000 7f782b7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:20:56.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.171+0000 7f782b7fe700 10 client.?.objecter ms_handle_connect 0x7f7834107780 2024-04-14T15:20:56.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.171+0000 7f782b7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:20:56.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.171+0000 7f782b7fe700 1 -- 172.21.15.90:0/1032290602 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1875388634 0 0) 0x7f78341a9b00 con 0x7f783410a630 2024-04-14T15:20:56.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.171+0000 7f782b7fe700 1 -- 172.21.15.90:0/1032290602 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78200031f0 con 0x7f783410a630 2024-04-14T15:20:56.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.172+0000 7f782b7fe700 10 client.?.objecter ms_handle_connect 0x7f783410e200 2024-04-14T15:20:56.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.172+0000 7f782b7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:20:56.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.172+0000 7f782b7fe700 1 -- 172.21.15.90:0/1032290602 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2782164191 0 0) 0x7f78341abf60 con 0x7f7834107780 2024-04-14T15:20:56.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.172+0000 7f782b7fe700 1 -- 172.21.15.90:0/1032290602 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78341a9b00 con 0x7f7834107780 2024-04-14T15:20:56.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.172+0000 7f782b7fe700 1 -- 172.21.15.90:0/1032290602 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1856574881 0 0) 0x7f78341aad20 con 0x7f783410e200 2024-04-14T15:20:56.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.172+0000 7f782b7fe700 1 -- 172.21.15.90:0/1032290602 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78341abf60 con 0x7f783410e200 2024-04-14T15:20:56.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.172+0000 7f782b7fe700 1 -- 172.21.15.90:0/1032290602 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1774506203 0 0) 0x7f78200031f0 con 0x7f783410a630 2024-04-14T15:20:56.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.172+0000 7f782b7fe700 1 -- 172.21.15.90:0/1032290602 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f78341aad20 con 0x7f783410a630 2024-04-14T15:20:56.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.172+0000 7f782b7fe700 1 -- 172.21.15.90:0/1032290602 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f781c0017f0 con 0x7f783410a630 2024-04-14T15:20:56.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.172+0000 7f782b7fe700 1 -- 172.21.15.90:0/1032290602 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 417324934 0 0) 0x7f78341a9b00 con 0x7f7834107780 2024-04-14T15:20:56.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.172+0000 7f782b7fe700 1 -- 172.21.15.90:0/1032290602 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f78200031f0 con 0x7f7834107780 2024-04-14T15:20:56.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.172+0000 7f782b7fe700 1 -- 172.21.15.90:0/1032290602 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f782c0013f0 con 0x7f7834107780 2024-04-14T15:20:56.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.172+0000 7f782b7fe700 1 -- 172.21.15.90:0/1032290602 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3254902422 0 0) 0x7f78341abf60 con 0x7f783410e200 2024-04-14T15:20:56.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.172+0000 7f782b7fe700 1 -- 172.21.15.90:0/1032290602 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f78341a9b00 con 0x7f783410e200 2024-04-14T15:20:56.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.172+0000 7f782b7fe700 1 -- 172.21.15.90:0/1032290602 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f7824001720 con 0x7f783410e200 2024-04-14T15:20:56.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.172+0000 7f782b7fe700 1 -- 172.21.15.90:0/1032290602 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1228612954 0 0) 0x7f78341aad20 con 0x7f783410a630 2024-04-14T15:20:56.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.172+0000 7f782b7fe700 1 -- 172.21.15.90:0/1032290602 >> v1:172.21.15.90:6790/0 conn(0x7f7834107780 legacy=0x7f783410da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:56.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.172+0000 7f782b7fe700 1 -- 172.21.15.90:0/1032290602 >> v1:172.21.15.92:6789/0 conn(0x7f783410e200 legacy=0x7f78341a81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:56.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.172+0000 7f782b7fe700 1 -- 172.21.15.90:0/1032290602 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78341ad1a0 con 0x7f783410a630 2024-04-14T15:20:56.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.172+0000 7f7838bea700 1 -- 172.21.15.90:0/1032290602 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f78341ac190 con 0x7f783410a630 2024-04-14T15:20:56.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.172+0000 7f7838bea700 10 client.37542.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:20:56.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.172+0000 7f7838bea700 1 -- 172.21.15.90:0/1032290602 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f78341ac650 con 0x7f783410a630 2024-04-14T15:20:56.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.172+0000 7f7838bea700 1 librados: init done 2024-04-14T15:20:56.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.172+0000 7f782b7fe700 1 -- 172.21.15.90:0/1032290602 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f781c001af0 con 0x7f783410a630 2024-04-14T15:20:56.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.172+0000 7f782b7fe700 1 -- 172.21.15.90:0/1032290602 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f781c004dd0 con 0x7f783410a630 2024-04-14T15:20:56.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.173+0000 7f7838bea700 1 -- 172.21.15.90:0/1032290602 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f7814005100 con 0x7f783410a630 2024-04-14T15:20:56.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.174+0000 7f782b7fe700 1 -- 172.21.15.90:0/1032290602 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f781c0039c0 con 0x7f783410a630 2024-04-14T15:20:56.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.175+0000 7f782b7fe700 1 -- 172.21.15.90:0/1032290602 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(531..531 src has 467..531) ==== 8230+0+0 (unknown 3786832762 0 0) 0x7f781c0898d0 con 0x7f783410a630 2024-04-14T15:20:56.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.175+0000 7f782b7fe700 10 client.37542.objecter ms_dispatch 0x7f7834001550 osd_map(531..531 src has 467..531) 2024-04-14T15:20:56.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.175+0000 7f782b7fe700 3 client.37542.objecter handle_osd_map got epochs [531,531] > 0 2024-04-14T15:20:56.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.175+0000 7f782b7fe700 3 client.37542.objecter handle_osd_map decoding full epoch 531 2024-04-14T15:20:56.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.175+0000 7f782b7fe700 20 client.37542.objecter dump_active .. 0 homeless 2024-04-14T15:20:56.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.175+0000 7f782b7fe700 1 -- 172.21.15.90:0/1032290602 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v531) ==== 54+0+16 (unknown 1609005456 0 2420000192) 0x7f781c08a060 con 0x7f783410a630 2024-04-14T15:20:56.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.175+0000 7f782b7fe700 10 client.37542.objecter ms_handle_connect 0x7f7820071390 2024-04-14T15:20:56.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.175+0000 7f7838bea700 10 client.37542.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:20:56.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.175+0000 7f7838bea700 15 client.37542.objecter _session_command_op_assign -1 1 2024-04-14T15:20:56.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.175+0000 7f7838bea700 20 client.37542.objecter _get_session s=0x7f78140053e0 osd=0 3 2024-04-14T15:20:56.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.175+0000 7f7838bea700 20 client.37542.objecter put_session s=0x7f78140053e0 osd=0 3 2024-04-14T15:20:56.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.175+0000 7f7838bea700 20 client.37542.objecter _get_session s=0x7f78140053e0 osd=0 3 2024-04-14T15:20:56.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.175+0000 7f7838bea700 15 client.37542.objecter _session_command_op_remove -1 1 2024-04-14T15:20:56.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.175+0000 7f7838bea700 20 client.37542.objecter get_session s=0x7f78140053e0 osd=0 3 2024-04-14T15:20:56.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.175+0000 7f7838bea700 15 client.37542.objecter _session_command_op_assign 0 1 2024-04-14T15:20:56.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.175+0000 7f7838bea700 20 client.37542.objecter put_session s=0x7f78140053e0 osd=0 4 2024-04-14T15:20:56.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.175+0000 7f7838bea700 10 client.37542.objecter _send_command 1 2024-04-14T15:20:56.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.175+0000 7f7838bea700 1 -- 172.21.15.90:0/1032290602 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7814004b50 con 0x7f7814004730 2024-04-14T15:20:56.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.175+0000 7f782b7fe700 10 client.37542.objecter ms_handle_connect 0x7f7814004730 2024-04-14T15:20:56.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.176+0000 7f782b7fe700 1 -- 172.21.15.90:0/1032290602 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f7814004b50 con 0x7f7814004730 2024-04-14T15:20:56.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.176+0000 7f782b7fe700 10 client.37542.objecter ms_dispatch 0x7f7834001550 command_reply(tid 1: 0 ) 2024-04-14T15:20:56.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.176+0000 7f782b7fe700 10 client.37542.objecter _finish_command 1 = system:0 2024-04-14T15:20:56.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.176+0000 7f782b7fe700 20 client.37542.objecter put_session s=0x7f78140053e0 osd=0 4 2024-04-14T15:20:56.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.176+0000 7f782b7fe700 15 client.37542.objecter _session_command_op_remove 0 1 2024-04-14T15:20:56.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.194+0000 7f7838bea700 10 client.37542.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:20:56.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.194+0000 7f7838bea700 15 client.37542.objecter _session_command_op_assign -1 2 2024-04-14T15:20:56.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.194+0000 7f7838bea700 20 client.37542.objecter _get_session s=0x7f78140053e0 osd=0 3 2024-04-14T15:20:56.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.194+0000 7f7838bea700 20 client.37542.objecter put_session s=0x7f78140053e0 osd=0 3 2024-04-14T15:20:56.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.194+0000 7f7838bea700 20 client.37542.objecter _get_session s=0x7f78140053e0 osd=0 3 2024-04-14T15:20:56.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.194+0000 7f7838bea700 15 client.37542.objecter _session_command_op_remove -1 2 2024-04-14T15:20:56.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.194+0000 7f7838bea700 20 client.37542.objecter get_session s=0x7f78140053e0 osd=0 3 2024-04-14T15:20:56.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.194+0000 7f7838bea700 15 client.37542.objecter _session_command_op_assign 0 2 2024-04-14T15:20:56.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.194+0000 7f7838bea700 20 client.37542.objecter put_session s=0x7f78140053e0 osd=0 4 2024-04-14T15:20:56.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.194+0000 7f7838bea700 10 client.37542.objecter _send_command 2 2024-04-14T15:20:56.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.194+0000 7f7838bea700 1 -- 172.21.15.90:0/1032290602 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f78140079f0 con 0x7f7814004730 2024-04-14T15:20:56.201 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:55 smithi092 ceph-mon[33966]: osdmap e530: 8 total, 8 up, 7 in 2024-04-14T15:20:56.201 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:55 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:20:56.201 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:55 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:20:56.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.196+0000 7f782b7fe700 1 -- 172.21.15.90:0/1032290602 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f78140079f0 con 0x7f7814004730 2024-04-14T15:20:56.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.196+0000 7f782b7fe700 10 client.37542.objecter ms_dispatch 0x7f7834001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:20:56.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.196+0000 7f782b7fe700 10 client.37542.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:20:56.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.196+0000 7f782b7fe700 20 client.37542.objecter put_session s=0x7f78140053e0 osd=0 4 2024-04-14T15:20:56.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.196+0000 7f782b7fe700 15 client.37542.objecter _session_command_op_remove 0 2 2024-04-14T15:20:56.202 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:20:56.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.196+0000 7f7838bea700 10 client.37542.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:20:56.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.196+0000 7f7838bea700 15 client.37542.objecter _session_command_op_assign -1 3 2024-04-14T15:20:56.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.196+0000 7f7838bea700 20 client.37542.objecter _get_session s=0x7f7814004110 osd=1 3 2024-04-14T15:20:56.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.196+0000 7f7838bea700 20 client.37542.objecter put_session s=0x7f7814004110 osd=1 3 2024-04-14T15:20:56.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.196+0000 7f7838bea700 20 client.37542.objecter _get_session s=0x7f7814004110 osd=1 3 2024-04-14T15:20:56.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.196+0000 7f7838bea700 15 client.37542.objecter _session_command_op_remove -1 3 2024-04-14T15:20:56.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.196+0000 7f7838bea700 20 client.37542.objecter get_session s=0x7f7814004110 osd=1 3 2024-04-14T15:20:56.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.196+0000 7f7838bea700 15 client.37542.objecter _session_command_op_assign 1 3 2024-04-14T15:20:56.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.196+0000 7f7838bea700 20 client.37542.objecter put_session s=0x7f7814004110 osd=1 4 2024-04-14T15:20:56.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.196+0000 7f7838bea700 10 client.37542.objecter _send_command 3 2024-04-14T15:20:56.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.196+0000 7f7838bea700 1 -- 172.21.15.90:0/1032290602 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f7814002180 con 0x7f7814004280 2024-04-14T15:20:56.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.197+0000 7f782b7fe700 10 client.37542.objecter ms_handle_connect 0x7f7814004280 2024-04-14T15:20:56.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.197+0000 7f782b7fe700 1 -- 172.21.15.90:0/1032290602 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f7814002180 con 0x7f7814004280 2024-04-14T15:20:56.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.197+0000 7f782b7fe700 10 client.37542.objecter ms_dispatch 0x7f7834001550 command_reply(tid 3: 0 ) 2024-04-14T15:20:56.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.197+0000 7f782b7fe700 10 client.37542.objecter _finish_command 3 = system:0 2024-04-14T15:20:56.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.197+0000 7f782b7fe700 20 client.37542.objecter put_session s=0x7f7814004110 osd=1 4 2024-04-14T15:20:56.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.197+0000 7f782b7fe700 15 client.37542.objecter _session_command_op_remove 1 3 2024-04-14T15:20:56.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.214+0000 7f7838bea700 10 client.37542.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:20:56.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.214+0000 7f7838bea700 15 client.37542.objecter _session_command_op_assign -1 4 2024-04-14T15:20:56.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.214+0000 7f7838bea700 20 client.37542.objecter _get_session s=0x7f7814004110 osd=1 3 2024-04-14T15:20:56.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.214+0000 7f7838bea700 20 client.37542.objecter put_session s=0x7f7814004110 osd=1 3 2024-04-14T15:20:56.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.214+0000 7f7838bea700 20 client.37542.objecter _get_session s=0x7f7814004110 osd=1 3 2024-04-14T15:20:56.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.214+0000 7f7838bea700 15 client.37542.objecter _session_command_op_remove -1 4 2024-04-14T15:20:56.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.214+0000 7f7838bea700 20 client.37542.objecter get_session s=0x7f7814004110 osd=1 3 2024-04-14T15:20:56.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.214+0000 7f7838bea700 15 client.37542.objecter _session_command_op_assign 1 4 2024-04-14T15:20:56.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.214+0000 7f7838bea700 20 client.37542.objecter put_session s=0x7f7814004110 osd=1 4 2024-04-14T15:20:56.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.214+0000 7f7838bea700 10 client.37542.objecter _send_command 4 2024-04-14T15:20:56.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.214+0000 7f7838bea700 1 -- 172.21.15.90:0/1032290602 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f7814001600 con 0x7f7814004280 2024-04-14T15:20:56.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.216+0000 7f782b7fe700 1 -- 172.21.15.90:0/1032290602 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f7814001600 con 0x7f7814004280 2024-04-14T15:20:56.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.216+0000 7f782b7fe700 10 client.37542.objecter ms_dispatch 0x7f7834001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:20:56.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.216+0000 7f782b7fe700 10 client.37542.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:20:56.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.216+0000 7f782b7fe700 20 client.37542.objecter put_session s=0x7f7814004110 osd=1 4 2024-04-14T15:20:56.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.216+0000 7f782b7fe700 15 client.37542.objecter _session_command_op_remove 1 4 2024-04-14T15:20:56.217 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:20:56.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.216+0000 7f7838bea700 10 client.37542.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:20:56.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.216+0000 7f7838bea700 15 client.37542.objecter _session_command_op_assign -1 5 2024-04-14T15:20:56.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.216+0000 7f7838bea700 20 client.37542.objecter _get_session s=0x7f7814008370 osd=2 3 2024-04-14T15:20:56.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.216+0000 7f7838bea700 20 client.37542.objecter put_session s=0x7f7814008370 osd=2 3 2024-04-14T15:20:56.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.216+0000 7f7838bea700 20 client.37542.objecter _get_session s=0x7f7814008370 osd=2 3 2024-04-14T15:20:56.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.216+0000 7f7838bea700 15 client.37542.objecter _session_command_op_remove -1 5 2024-04-14T15:20:56.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.216+0000 7f7838bea700 20 client.37542.objecter get_session s=0x7f7814008370 osd=2 3 2024-04-14T15:20:56.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.216+0000 7f7838bea700 15 client.37542.objecter _session_command_op_assign 2 5 2024-04-14T15:20:56.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.216+0000 7f7838bea700 20 client.37542.objecter put_session s=0x7f7814008370 osd=2 4 2024-04-14T15:20:56.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.216+0000 7f7838bea700 10 client.37542.objecter _send_command 5 2024-04-14T15:20:56.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.216+0000 7f7838bea700 1 -- 172.21.15.90:0/1032290602 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f78140110e0 con 0x7f781400d5d0 2024-04-14T15:20:56.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.216+0000 7f782b7fe700 10 client.37542.objecter ms_handle_connect 0x7f781400d5d0 2024-04-14T15:20:56.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.217+0000 7f782b7fe700 1 -- 172.21.15.90:0/1032290602 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f78140110e0 con 0x7f781400d5d0 2024-04-14T15:20:56.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.217+0000 7f782b7fe700 10 client.37542.objecter ms_dispatch 0x7f7834001550 command_reply(tid 5: 0 ) 2024-04-14T15:20:56.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.217+0000 7f782b7fe700 10 client.37542.objecter _finish_command 5 = system:0 2024-04-14T15:20:56.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.217+0000 7f782b7fe700 20 client.37542.objecter put_session s=0x7f7814008370 osd=2 4 2024-04-14T15:20:56.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.217+0000 7f782b7fe700 15 client.37542.objecter _session_command_op_remove 2 5 2024-04-14T15:20:56.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.233+0000 7f7838bea700 10 client.37542.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:20:56.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.233+0000 7f7838bea700 15 client.37542.objecter _session_command_op_assign -1 6 2024-04-14T15:20:56.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.233+0000 7f7838bea700 20 client.37542.objecter _get_session s=0x7f7814008370 osd=2 3 2024-04-14T15:20:56.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.233+0000 7f7838bea700 20 client.37542.objecter put_session s=0x7f7814008370 osd=2 3 2024-04-14T15:20:56.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.233+0000 7f7838bea700 20 client.37542.objecter _get_session s=0x7f7814008370 osd=2 3 2024-04-14T15:20:56.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.233+0000 7f7838bea700 15 client.37542.objecter _session_command_op_remove -1 6 2024-04-14T15:20:56.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.233+0000 7f7838bea700 20 client.37542.objecter get_session s=0x7f7814008370 osd=2 3 2024-04-14T15:20:56.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.233+0000 7f7838bea700 15 client.37542.objecter _session_command_op_assign 2 6 2024-04-14T15:20:56.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.233+0000 7f7838bea700 20 client.37542.objecter put_session s=0x7f7814008370 osd=2 4 2024-04-14T15:20:56.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.233+0000 7f7838bea700 10 client.37542.objecter _send_command 6 2024-04-14T15:20:56.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.233+0000 7f7838bea700 1 -- 172.21.15.90:0/1032290602 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f7814007eb0 con 0x7f781400d5d0 2024-04-14T15:20:56.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.235+0000 7f782b7fe700 1 -- 172.21.15.90:0/1032290602 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f7814007eb0 con 0x7f781400d5d0 2024-04-14T15:20:56.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.235+0000 7f782b7fe700 10 client.37542.objecter ms_dispatch 0x7f7834001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:20:56.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.235+0000 7f782b7fe700 10 client.37542.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:20:56.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.235+0000 7f782b7fe700 20 client.37542.objecter put_session s=0x7f7814008370 osd=2 4 2024-04-14T15:20:56.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.235+0000 7f782b7fe700 15 client.37542.objecter _session_command_op_remove 2 6 2024-04-14T15:20:56.236 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:20:56.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.235+0000 7f7838bea700 10 client.37542.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:20:56.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.235+0000 7f7838bea700 15 client.37542.objecter _session_command_op_assign -1 7 2024-04-14T15:20:56.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.235+0000 7f7838bea700 20 client.37542.objecter _get_session s=0x7f781400cc20 osd=3 3 2024-04-14T15:20:56.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.235+0000 7f7838bea700 20 client.37542.objecter put_session s=0x7f781400cc20 osd=3 3 2024-04-14T15:20:56.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.235+0000 7f7838bea700 20 client.37542.objecter _get_session s=0x7f781400cc20 osd=3 3 2024-04-14T15:20:56.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.235+0000 7f7838bea700 15 client.37542.objecter _session_command_op_remove -1 7 2024-04-14T15:20:56.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.235+0000 7f7838bea700 20 client.37542.objecter get_session s=0x7f781400cc20 osd=3 3 2024-04-14T15:20:56.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.235+0000 7f7838bea700 15 client.37542.objecter _session_command_op_assign 3 7 2024-04-14T15:20:56.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.235+0000 7f7838bea700 20 client.37542.objecter put_session s=0x7f781400cc20 osd=3 4 2024-04-14T15:20:56.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.235+0000 7f7838bea700 10 client.37542.objecter _send_command 7 2024-04-14T15:20:56.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.235+0000 7f7838bea700 1 -- 172.21.15.90:0/1032290602 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f7814015f20 con 0x7f7814012410 2024-04-14T15:20:56.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.236+0000 7f782b7fe700 10 client.37542.objecter ms_handle_connect 0x7f7814012410 2024-04-14T15:20:56.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.236+0000 7f782b7fe700 1 -- 172.21.15.90:0/1032290602 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f7814015f20 con 0x7f7814012410 2024-04-14T15:20:56.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.236+0000 7f782b7fe700 10 client.37542.objecter ms_dispatch 0x7f7834001550 command_reply(tid 7: 0 ) 2024-04-14T15:20:56.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.236+0000 7f782b7fe700 10 client.37542.objecter _finish_command 7 = system:0 2024-04-14T15:20:56.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.236+0000 7f782b7fe700 20 client.37542.objecter put_session s=0x7f781400cc20 osd=3 4 2024-04-14T15:20:56.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.236+0000 7f782b7fe700 15 client.37542.objecter _session_command_op_remove 3 7 2024-04-14T15:20:56.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.253+0000 7f7838bea700 10 client.37542.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:20:56.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.253+0000 7f7838bea700 15 client.37542.objecter _session_command_op_assign -1 8 2024-04-14T15:20:56.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.253+0000 7f7838bea700 20 client.37542.objecter _get_session s=0x7f781400cc20 osd=3 3 2024-04-14T15:20:56.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.253+0000 7f7838bea700 20 client.37542.objecter put_session s=0x7f781400cc20 osd=3 3 2024-04-14T15:20:56.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.253+0000 7f7838bea700 20 client.37542.objecter _get_session s=0x7f781400cc20 osd=3 3 2024-04-14T15:20:56.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.253+0000 7f7838bea700 15 client.37542.objecter _session_command_op_remove -1 8 2024-04-14T15:20:56.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.253+0000 7f7838bea700 20 client.37542.objecter get_session s=0x7f781400cc20 osd=3 3 2024-04-14T15:20:56.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.253+0000 7f7838bea700 15 client.37542.objecter _session_command_op_assign 3 8 2024-04-14T15:20:56.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.253+0000 7f7838bea700 20 client.37542.objecter put_session s=0x7f781400cc20 osd=3 4 2024-04-14T15:20:56.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.253+0000 7f7838bea700 10 client.37542.objecter _send_command 8 2024-04-14T15:20:56.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.253+0000 7f7838bea700 1 -- 172.21.15.90:0/1032290602 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f781400c700 con 0x7f7814012410 2024-04-14T15:20:56.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.254+0000 7f782b7fe700 1 -- 172.21.15.90:0/1032290602 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f781400c700 con 0x7f7814012410 2024-04-14T15:20:56.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.254+0000 7f782b7fe700 10 client.37542.objecter ms_dispatch 0x7f7834001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:20:56.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.254+0000 7f782b7fe700 10 client.37542.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:20:56.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.254+0000 7f782b7fe700 20 client.37542.objecter put_session s=0x7f781400cc20 osd=3 4 2024-04-14T15:20:56.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.254+0000 7f782b7fe700 15 client.37542.objecter _session_command_op_remove 3 8 2024-04-14T15:20:56.255 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:20:56.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.255+0000 7f7838bea700 10 client.37542.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:20:56.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.255+0000 7f7838bea700 15 client.37542.objecter _session_command_op_assign -1 9 2024-04-14T15:20:56.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.255+0000 7f7838bea700 20 client.37542.objecter _get_session s=0x7f781400c700 osd=4 3 2024-04-14T15:20:56.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.255+0000 7f7838bea700 20 client.37542.objecter put_session s=0x7f781400c700 osd=4 3 2024-04-14T15:20:56.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.255+0000 7f7838bea700 20 client.37542.objecter _get_session s=0x7f781400c700 osd=4 3 2024-04-14T15:20:56.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.255+0000 7f7838bea700 15 client.37542.objecter _session_command_op_remove -1 9 2024-04-14T15:20:56.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.255+0000 7f7838bea700 20 client.37542.objecter get_session s=0x7f781400c700 osd=4 3 2024-04-14T15:20:56.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.255+0000 7f7838bea700 15 client.37542.objecter _session_command_op_assign 4 9 2024-04-14T15:20:56.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.255+0000 7f7838bea700 20 client.37542.objecter put_session s=0x7f781400c700 osd=4 4 2024-04-14T15:20:56.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.255+0000 7f7838bea700 10 client.37542.objecter _send_command 9 2024-04-14T15:20:56.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.255+0000 7f7838bea700 1 -- 172.21.15.90:0/1032290602 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f781401a9f0 con 0x7f7814011f70 2024-04-14T15:20:56.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.255+0000 7f782b7fe700 10 client.37542.objecter ms_handle_connect 0x7f7814011f70 2024-04-14T15:20:56.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.256+0000 7f782b7fe700 1 -- 172.21.15.90:0/1032290602 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f781401a9f0 con 0x7f7814011f70 2024-04-14T15:20:56.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.256+0000 7f782b7fe700 10 client.37542.objecter ms_dispatch 0x7f7834001550 command_reply(tid 9: 0 ) 2024-04-14T15:20:56.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.256+0000 7f782b7fe700 10 client.37542.objecter _finish_command 9 = system:0 2024-04-14T15:20:56.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.256+0000 7f782b7fe700 20 client.37542.objecter put_session s=0x7f781400c700 osd=4 4 2024-04-14T15:20:56.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.256+0000 7f782b7fe700 15 client.37542.objecter _session_command_op_remove 4 9 2024-04-14T15:20:56.270 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:20:56.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.272+0000 7f7838bea700 10 client.37542.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:20:56.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.272+0000 7f7838bea700 15 client.37542.objecter _session_command_op_assign -1 10 2024-04-14T15:20:56.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.272+0000 7f7838bea700 20 client.37542.objecter _get_session s=0x7f781400c700 osd=4 3 2024-04-14T15:20:56.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.272+0000 7f7838bea700 20 client.37542.objecter put_session s=0x7f781400c700 osd=4 3 2024-04-14T15:20:56.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.272+0000 7f7838bea700 20 client.37542.objecter _get_session s=0x7f781400c700 osd=4 3 2024-04-14T15:20:56.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.272+0000 7f7838bea700 15 client.37542.objecter _session_command_op_remove -1 10 2024-04-14T15:20:56.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.272+0000 7f7838bea700 20 client.37542.objecter get_session s=0x7f781400c700 osd=4 3 2024-04-14T15:20:56.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.272+0000 7f7838bea700 15 client.37542.objecter _session_command_op_assign 4 10 2024-04-14T15:20:56.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.272+0000 7f7838bea700 20 client.37542.objecter put_session s=0x7f781400c700 osd=4 4 2024-04-14T15:20:56.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.272+0000 7f7838bea700 10 client.37542.objecter _send_command 10 2024-04-14T15:20:56.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.272+0000 7f7838bea700 1 -- 172.21.15.90:0/1032290602 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f781401c3a0 con 0x7f7814011f70 2024-04-14T15:20:56.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.274+0000 7f782b7fe700 1 -- 172.21.15.90:0/1032290602 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f781401c3a0 con 0x7f7814011f70 2024-04-14T15:20:56.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.274+0000 7f782b7fe700 10 client.37542.objecter ms_dispatch 0x7f7834001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:20:56.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.274+0000 7f782b7fe700 10 client.37542.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:20:56.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.274+0000 7f782b7fe700 20 client.37542.objecter put_session s=0x7f781400c700 osd=4 4 2024-04-14T15:20:56.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.274+0000 7f782b7fe700 15 client.37542.objecter _session_command_op_remove 4 10 2024-04-14T15:20:56.275 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:20:56.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.274+0000 7f7838bea700 10 client.37542.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:20:56.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.274+0000 7f7838bea700 15 client.37542.objecter _session_command_op_assign -1 11 2024-04-14T15:20:56.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.274+0000 7f7838bea700 20 client.37542.objecter _get_session s=0x7f781401c3a0 osd=5 3 2024-04-14T15:20:56.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.274+0000 7f7838bea700 20 client.37542.objecter put_session s=0x7f781401c3a0 osd=5 3 2024-04-14T15:20:56.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.274+0000 7f7838bea700 20 client.37542.objecter _get_session s=0x7f781401c3a0 osd=5 3 2024-04-14T15:20:56.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.274+0000 7f7838bea700 15 client.37542.objecter _session_command_op_remove -1 11 2024-04-14T15:20:56.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.274+0000 7f7838bea700 20 client.37542.objecter get_session s=0x7f781401c3a0 osd=5 3 2024-04-14T15:20:56.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.274+0000 7f7838bea700 15 client.37542.objecter _session_command_op_assign 5 11 2024-04-14T15:20:56.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.274+0000 7f7838bea700 20 client.37542.objecter put_session s=0x7f781401c3a0 osd=5 4 2024-04-14T15:20:56.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.274+0000 7f7838bea700 10 client.37542.objecter _send_command 11 2024-04-14T15:20:56.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.274+0000 7f7838bea700 1 -- 172.21.15.90:0/1032290602 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f781401cf70 con 0x7f781401cac0 2024-04-14T15:20:56.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.274+0000 7f782b7fe700 10 client.37542.objecter ms_handle_connect 0x7f781401cac0 2024-04-14T15:20:56.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.275+0000 7f782b7fe700 1 -- 172.21.15.90:0/1032290602 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f781401cf70 con 0x7f781401cac0 2024-04-14T15:20:56.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.275+0000 7f782b7fe700 10 client.37542.objecter ms_dispatch 0x7f7834001550 command_reply(tid 11: 0 ) 2024-04-14T15:20:56.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.275+0000 7f782b7fe700 10 client.37542.objecter _finish_command 11 = system:0 2024-04-14T15:20:56.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.275+0000 7f782b7fe700 20 client.37542.objecter put_session s=0x7f781401c3a0 osd=5 4 2024-04-14T15:20:56.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.275+0000 7f782b7fe700 15 client.37542.objecter _session_command_op_remove 5 11 2024-04-14T15:20:56.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.292+0000 7f7838bea700 10 client.37542.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:20:56.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.292+0000 7f7838bea700 15 client.37542.objecter _session_command_op_assign -1 12 2024-04-14T15:20:56.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.292+0000 7f7838bea700 20 client.37542.objecter _get_session s=0x7f781401c3a0 osd=5 3 2024-04-14T15:20:56.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.292+0000 7f7838bea700 20 client.37542.objecter put_session s=0x7f781401c3a0 osd=5 3 2024-04-14T15:20:56.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.292+0000 7f7838bea700 20 client.37542.objecter _get_session s=0x7f781401c3a0 osd=5 3 2024-04-14T15:20:56.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.292+0000 7f7838bea700 15 client.37542.objecter _session_command_op_remove -1 12 2024-04-14T15:20:56.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.292+0000 7f7838bea700 20 client.37542.objecter get_session s=0x7f781401c3a0 osd=5 3 2024-04-14T15:20:56.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.292+0000 7f7838bea700 15 client.37542.objecter _session_command_op_assign 5 12 2024-04-14T15:20:56.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.292+0000 7f7838bea700 20 client.37542.objecter put_session s=0x7f781401c3a0 osd=5 4 2024-04-14T15:20:56.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.292+0000 7f7838bea700 10 client.37542.objecter _send_command 12 2024-04-14T15:20:56.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.292+0000 7f7838bea700 1 -- 172.21.15.90:0/1032290602 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f781401d1d0 con 0x7f781401cac0 2024-04-14T15:20:56.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.293+0000 7f782b7fe700 1 -- 172.21.15.90:0/1032290602 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f781401d1d0 con 0x7f781401cac0 2024-04-14T15:20:56.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.293+0000 7f782b7fe700 10 client.37542.objecter ms_dispatch 0x7f7834001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:20:56.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.293+0000 7f782b7fe700 10 client.37542.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:20:56.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.293+0000 7f782b7fe700 20 client.37542.objecter put_session s=0x7f781401c3a0 osd=5 4 2024-04-14T15:20:56.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.293+0000 7f782b7fe700 15 client.37542.objecter _session_command_op_remove 5 12 2024-04-14T15:20:56.295 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:20:56.295 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:56.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.294+0000 7f7838bea700 10 client.37542.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:20:56.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.294+0000 7f7838bea700 15 client.37542.objecter _session_command_op_assign -1 13 2024-04-14T15:20:56.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.294+0000 7f7838bea700 20 client.37542.objecter _get_session s=0x7f781401d1d0 osd=6 3 2024-04-14T15:20:56.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.294+0000 7f7838bea700 20 client.37542.objecter put_session s=0x7f781401d1d0 osd=6 3 2024-04-14T15:20:56.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.294+0000 7f7838bea700 20 client.37542.objecter _get_session s=0x7f781401d1d0 osd=6 3 2024-04-14T15:20:56.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.294+0000 7f7838bea700 15 client.37542.objecter _session_command_op_remove -1 13 2024-04-14T15:20:56.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.294+0000 7f7838bea700 20 client.37542.objecter get_session s=0x7f781401d1d0 osd=6 3 2024-04-14T15:20:56.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.294+0000 7f7838bea700 15 client.37542.objecter _session_command_op_assign 6 13 2024-04-14T15:20:56.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.294+0000 7f7838bea700 20 client.37542.objecter put_session s=0x7f781401d1d0 osd=6 4 2024-04-14T15:20:56.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.294+0000 7f7838bea700 10 client.37542.objecter _send_command 13 2024-04-14T15:20:56.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.294+0000 7f7838bea700 1 -- 172.21.15.90:0/1032290602 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f78140235c0 con 0x7f781401fab0 2024-04-14T15:20:56.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:56.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:20:56.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.294+0000 7f782b7fe700 10 client.37542.objecter ms_handle_connect 0x7f781401fab0 2024-04-14T15:20:56.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.295+0000 7f782b7fe700 1 -- 172.21.15.90:0/1032290602 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f78140235c0 con 0x7f781401fab0 2024-04-14T15:20:56.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.295+0000 7f782b7fe700 10 client.37542.objecter ms_dispatch 0x7f7834001550 command_reply(tid 13: 0 ) 2024-04-14T15:20:56.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.295+0000 7f782b7fe700 10 client.37542.objecter _finish_command 13 = system:0 2024-04-14T15:20:56.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.295+0000 7f782b7fe700 20 client.37542.objecter put_session s=0x7f781401d1d0 osd=6 4 2024-04-14T15:20:56.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.295+0000 7f782b7fe700 15 client.37542.objecter _session_command_op_remove 6 13 2024-04-14T15:20:56.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.311+0000 7f7838bea700 10 client.37542.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:20:56.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.311+0000 7f7838bea700 15 client.37542.objecter _session_command_op_assign -1 14 2024-04-14T15:20:56.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.311+0000 7f7838bea700 20 client.37542.objecter _get_session s=0x7f781401d1d0 osd=6 3 2024-04-14T15:20:56.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.311+0000 7f7838bea700 20 client.37542.objecter put_session s=0x7f781401d1d0 osd=6 3 2024-04-14T15:20:56.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.311+0000 7f7838bea700 20 client.37542.objecter _get_session s=0x7f781401d1d0 osd=6 3 2024-04-14T15:20:56.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.311+0000 7f7838bea700 15 client.37542.objecter _session_command_op_remove -1 14 2024-04-14T15:20:56.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.311+0000 7f7838bea700 20 client.37542.objecter get_session s=0x7f781401d1d0 osd=6 3 2024-04-14T15:20:56.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.311+0000 7f7838bea700 15 client.37542.objecter _session_command_op_assign 6 14 2024-04-14T15:20:56.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.311+0000 7f7838bea700 20 client.37542.objecter put_session s=0x7f781401d1d0 osd=6 4 2024-04-14T15:20:56.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.311+0000 7f7838bea700 10 client.37542.objecter _send_command 14 2024-04-14T15:20:56.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.311+0000 7f7838bea700 1 -- 172.21.15.90:0/1032290602 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f781400c910 con 0x7f781401fab0 2024-04-14T15:20:56.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.313+0000 7f782b7fe700 1 -- 172.21.15.90:0/1032290602 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f781400c910 con 0x7f781401fab0 2024-04-14T15:20:56.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.313+0000 7f782b7fe700 10 client.37542.objecter ms_dispatch 0x7f7834001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:20:56.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.313+0000 7f782b7fe700 10 client.37542.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:20:56.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.313+0000 7f782b7fe700 20 client.37542.objecter put_session s=0x7f781401d1d0 osd=6 4 2024-04-14T15:20:56.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.313+0000 7f782b7fe700 15 client.37542.objecter _session_command_op_remove 6 14 2024-04-14T15:20:56.314 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:20:56.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.314+0000 7f7838bea700 10 client.37542.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:20:56.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.314+0000 7f7838bea700 15 client.37542.objecter _session_command_op_assign -1 15 2024-04-14T15:20:56.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.314+0000 7f7838bea700 20 client.37542.objecter _get_session s=0x7f78140235c0 osd=7 3 2024-04-14T15:20:56.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.314+0000 7f7838bea700 20 client.37542.objecter put_session s=0x7f78140235c0 osd=7 3 2024-04-14T15:20:56.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.314+0000 7f7838bea700 20 client.37542.objecter _get_session s=0x7f78140235c0 osd=7 3 2024-04-14T15:20:56.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.314+0000 7f7838bea700 15 client.37542.objecter _session_command_op_remove -1 15 2024-04-14T15:20:56.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.314+0000 7f7838bea700 20 client.37542.objecter get_session s=0x7f78140235c0 osd=7 3 2024-04-14T15:20:56.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.314+0000 7f7838bea700 15 client.37542.objecter _session_command_op_assign 7 15 2024-04-14T15:20:56.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.314+0000 7f7838bea700 20 client.37542.objecter put_session s=0x7f78140235c0 osd=7 4 2024-04-14T15:20:56.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.314+0000 7f7838bea700 10 client.37542.objecter _send_command 15 2024-04-14T15:20:56.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.314+0000 7f7838bea700 1 -- 172.21.15.90:0/1032290602 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f7814027900 con 0x7f7814023c40 2024-04-14T15:20:56.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.314+0000 7f782b7fe700 10 client.37542.objecter ms_handle_connect 0x7f7814023c40 2024-04-14T15:20:56.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.315+0000 7f782b7fe700 1 -- 172.21.15.90:0/1032290602 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f7814027900 con 0x7f7814023c40 2024-04-14T15:20:56.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.315+0000 7f782b7fe700 10 client.37542.objecter ms_dispatch 0x7f7834001550 command_reply(tid 15: 0 ) 2024-04-14T15:20:56.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.315+0000 7f782b7fe700 10 client.37542.objecter _finish_command 15 = system:0 2024-04-14T15:20:56.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.315+0000 7f782b7fe700 20 client.37542.objecter put_session s=0x7f78140235c0 osd=7 4 2024-04-14T15:20:56.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.315+0000 7f782b7fe700 15 client.37542.objecter _session_command_op_remove 7 15 2024-04-14T15:20:56.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.331+0000 7f7838bea700 10 client.37542.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:20:56.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.331+0000 7f7838bea700 15 client.37542.objecter _session_command_op_assign -1 16 2024-04-14T15:20:56.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.331+0000 7f7838bea700 20 client.37542.objecter _get_session s=0x7f78140235c0 osd=7 3 2024-04-14T15:20:56.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.331+0000 7f7838bea700 20 client.37542.objecter put_session s=0x7f78140235c0 osd=7 3 2024-04-14T15:20:56.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.331+0000 7f7838bea700 20 client.37542.objecter _get_session s=0x7f78140235c0 osd=7 3 2024-04-14T15:20:56.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.331+0000 7f7838bea700 15 client.37542.objecter _session_command_op_remove -1 16 2024-04-14T15:20:56.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.331+0000 7f7838bea700 20 client.37542.objecter get_session s=0x7f78140235c0 osd=7 3 2024-04-14T15:20:56.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.331+0000 7f7838bea700 15 client.37542.objecter _session_command_op_assign 7 16 2024-04-14T15:20:56.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.331+0000 7f7838bea700 20 client.37542.objecter put_session s=0x7f78140235c0 osd=7 4 2024-04-14T15:20:56.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.331+0000 7f7838bea700 10 client.37542.objecter _send_command 16 2024-04-14T15:20:56.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.331+0000 7f7838bea700 1 -- 172.21.15.90:0/1032290602 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f78140292b0 con 0x7f7814023c40 2024-04-14T15:20:56.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.333+0000 7f782b7fe700 1 -- 172.21.15.90:0/1032290602 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f78140292b0 con 0x7f7814023c40 2024-04-14T15:20:56.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.333+0000 7f782b7fe700 10 client.37542.objecter ms_dispatch 0x7f7834001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:20:56.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.333+0000 7f782b7fe700 10 client.37542.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:20:56.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.333+0000 7f782b7fe700 20 client.37542.objecter put_session s=0x7f78140235c0 osd=7 4 2024-04-14T15:20:56.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.333+0000 7f782b7fe700 15 client.37542.objecter _session_command_op_remove 7 16 2024-04-14T15:20:56.334 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:20:56.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.333+0000 7f7838bea700 10 librados: watch_flush enter 2024-04-14T15:20:56.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.333+0000 7f7838bea700 10 librados: watch_flush exit 2024-04-14T15:20:56.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.333+0000 7f7838bea700 10 client.37542.objecter close_session for osd.0 2024-04-14T15:20:56.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.333+0000 7f7838bea700 1 -- 172.21.15.90:0/1032290602 >> v1:172.21.15.90:6801/3761806885 conn(0x7f7814004730 legacy=0x7f7814002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:56.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.333+0000 7f7838bea700 20 client.37542.objecter put_session s=0x7f78140053e0 osd=0 1 2024-04-14T15:20:56.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.333+0000 7f7838bea700 10 client.37542.objecter close_session for osd.1 2024-04-14T15:20:56.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.333+0000 7f7838bea700 1 -- 172.21.15.90:0/1032290602 >> v1:172.21.15.90:6805/4170361699 conn(0x7f7814004280 legacy=0x7f781400ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:56.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.333+0000 7f7838bea700 20 client.37542.objecter put_session s=0x7f7814004110 osd=1 1 2024-04-14T15:20:56.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.333+0000 7f7838bea700 10 client.37542.objecter close_session for osd.2 2024-04-14T15:20:56.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.333+0000 7f7838bea700 1 -- 172.21.15.90:0/1032290602 >> v1:172.21.15.90:6809/1686184251 conn(0x7f781400d5d0 legacy=0x7f781400f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:56.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.333+0000 7f7838bea700 20 client.37542.objecter put_session s=0x7f7814008370 osd=2 1 2024-04-14T15:20:56.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.333+0000 7f7838bea700 10 client.37542.objecter close_session for osd.3 2024-04-14T15:20:56.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.333+0000 7f7838bea700 1 -- 172.21.15.90:0/1032290602 >> v1:172.21.15.90:6813/2270166774 conn(0x7f7814012410 legacy=0x7f7814014800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:56.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.333+0000 7f7838bea700 20 client.37542.objecter put_session s=0x7f781400cc20 osd=3 1 2024-04-14T15:20:56.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.333+0000 7f7838bea700 10 client.37542.objecter close_session for osd.4 2024-04-14T15:20:56.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.333+0000 7f7838bea700 1 -- 172.21.15.90:0/1032290602 >> v1:172.21.15.92:6800/3887549733 conn(0x7f7814011f70 legacy=0x7f7814019260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:56.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.333+0000 7f7832d9d700 1 -- 172.21.15.90:0/1032290602 reap_dead start 2024-04-14T15:20:56.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.333+0000 7f7832d9d700 1 -- 172.21.15.90:0/1032290602 reap_dead start 2024-04-14T15:20:56.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.334+0000 7f7838bea700 20 client.37542.objecter put_session s=0x7f781400c700 osd=4 1 2024-04-14T15:20:56.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.334+0000 7f7838bea700 10 client.37542.objecter close_session for osd.5 2024-04-14T15:20:56.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.334+0000 7f7838bea700 1 -- 172.21.15.90:0/1032290602 >> v1:172.21.15.92:6804/3244561440 conn(0x7f781401cac0 legacy=0x7f781401abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:56.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.334+0000 7f7838bea700 20 client.37542.objecter put_session s=0x7f781401c3a0 osd=5 1 2024-04-14T15:20:56.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.334+0000 7f7838bea700 10 client.37542.objecter close_session for osd.6 2024-04-14T15:20:56.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.334+0000 7f7838bea700 1 -- 172.21.15.90:0/1032290602 >> v1:172.21.15.92:6808/636125963 conn(0x7f781401fab0 legacy=0x7f7814021ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:56.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.334+0000 7f7838bea700 20 client.37542.objecter put_session s=0x7f781401d1d0 osd=6 1 2024-04-14T15:20:56.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.334+0000 7f7838bea700 10 client.37542.objecter close_session for osd.7 2024-04-14T15:20:56.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.334+0000 7f7838bea700 1 -- 172.21.15.90:0/1032290602 >> v1:172.21.15.92:6812/1773706716 conn(0x7f7814023c40 legacy=0x7f7814026090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:56.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.334+0000 7f7838bea700 20 client.37542.objecter put_session s=0x7f78140235c0 osd=7 1 2024-04-14T15:20:56.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.334+0000 7f7838bea700 20 client.37542.objecter shutdown clearing up homeless session... 2024-04-14T15:20:56.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.334+0000 7f7838bea700 10 client.37542.objecter successfully canceled tick 2024-04-14T15:20:56.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.334+0000 7f7838bea700 1 -- 172.21.15.90:0/1032290602 >> v1:172.21.15.90:6800/420935427 conn(0x7f7820071390 legacy=0x7f7820073850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:56.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.334+0000 7f7838bea700 1 -- 172.21.15.90:0/1032290602 >> v1:172.21.15.90:6789/0 conn(0x7f783410a630 legacy=0x7f78341a4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:20:56.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.334+0000 7f7832d9d700 1 -- 172.21.15.90:0/1032290602 reap_dead start 2024-04-14T15:20:56.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.334+0000 7f7832d9d700 1 -- 172.21.15.90:0/1032290602 reap_dead start 2024-04-14T15:20:56.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.334+0000 7f7838bea700 1 -- 172.21.15.90:0/1032290602 shutdown_connections 2024-04-14T15:20:56.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.334+0000 7f7838bea700 1 -- 172.21.15.90:0/1032290602 >> 172.21.15.90:0/1032290602 conn(0x7f78340fd2b0 msgr2=0x7f7834111720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:20:56.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.334+0000 7f7838bea700 1 -- 172.21.15.90:0/1032290602 shutdown_connections 2024-04-14T15:20:56.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.334+0000 7f7838bea700 1 -- 172.21.15.90:0/1032290602 wait complete. 2024-04-14T15:20:56.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:20:56.334+0000 7f7838bea700 1 librados: shutdown 2024-04-14T15:20:56.338 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:20:56.338 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:20:56.338 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:20:56.338 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:20:56.338 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:20:56.338 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:20:56.338 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:20:56.338 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:20:56.398 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:20:56.421 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:56.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:56.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:20:56.461 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-04-14T15:20:56.461 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:20:56.523 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:56.547 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:56.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:56.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:56.649 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:20:56.672 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:56.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:56.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:20:56.774 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:20:56.797 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:56.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:56.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:20:56.899 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:20:56.923 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:56.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:56.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:20:57.025 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:20:57.034 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:56 smithi092 ceph-mon[33966]: pgmap v770: 254 pgs: 1 active+remapped+backfill_toofull, 2 undersized+remapped+premerge+peered, 32 unknown, 11 active+undersized+remapped, 208 active+clean; 8.4 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 6.7 KiB/s rd, 46 KiB/s wr, 16 op/s; 1/1170 objects misplaced (0.085%); 31 KiB/s, 69 objects/s recovering 2024-04-14T15:20:57.034 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:56 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:20:57.034 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:56 smithi092 ceph-mon[33966]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-04-14T15:20:57.035 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:56 smithi092 ceph-mon[33966]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:20:57.035 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:56 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-87","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:20:57.035 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:56 smithi092 ceph-mon[33966]: osdmap e531: 8 total, 8 up, 7 in 2024-04-14T15:20:57.035 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:56 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-14T15:20:57.035 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:56 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1032290602' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:20:57.050 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:57.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:57.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:20:57.063 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-14T15:20:57.151 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:20:57.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:57.176 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:57.176 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:20:57.277 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:20:57.301 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:57.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:57.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:20:57.402 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:20:57.426 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:57.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:57.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:20:57.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:56 smithi090 ceph-mon[30399]: pgmap v770: 254 pgs: 1 active+remapped+backfill_toofull, 2 undersized+remapped+premerge+peered, 32 unknown, 11 active+undersized+remapped, 208 active+clean; 8.4 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 6.7 KiB/s rd, 46 KiB/s wr, 16 op/s; 1/1170 objects misplaced (0.085%); 31 KiB/s, 69 objects/s recovering 2024-04-14T15:20:57.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:56 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:20:57.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:56 smithi090 ceph-mon[30399]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-04-14T15:20:57.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:56 smithi090 ceph-mon[30399]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:20:57.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:56 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-87","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:20:57.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:56 smithi090 ceph-mon[30399]: osdmap e531: 8 total, 8 up, 7 in 2024-04-14T15:20:57.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:56 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-14T15:20:57.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:56 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1032290602' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:20:57.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:56 smithi090 ceph-mon[41670]: pgmap v770: 254 pgs: 1 active+remapped+backfill_toofull, 2 undersized+remapped+premerge+peered, 32 unknown, 11 active+undersized+remapped, 208 active+clean; 8.4 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 6.7 KiB/s rd, 46 KiB/s wr, 16 op/s; 1/1170 objects misplaced (0.085%); 31 KiB/s, 69 objects/s recovering 2024-04-14T15:20:57.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:56 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:20:57.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:56 smithi090 ceph-mon[41670]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-04-14T15:20:57.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:56 smithi090 ceph-mon[41670]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:20:57.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:56 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-87","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:20:57.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:56 smithi090 ceph-mon[41670]: osdmap e531: 8 total, 8 up, 7 in 2024-04-14T15:20:57.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:56 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-14T15:20:57.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:56 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1032290602' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:20:57.528 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:57.534 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:20:57.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:57.553 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:57.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:57.654 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:20:57.677 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:57.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:57.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:20:57.710 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd unset noscrub 2024-04-14T15:20:57.780 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:20:57.804 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:57.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:57.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:20:57.906 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:20:57.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:57.933 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:57.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:20:57.942 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-14T15:20:57.942 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-14T15:20:57.942 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-04-14T15:20:57.942 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd dump -f json-pretty 2024-04-14T15:20:58.035 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:20:58.058 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:58.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:58.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:20:58.160 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:20:58.164 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:57 smithi090 ceph-mon[41670]: osdmap e532: 8 total, 8 up, 7 in 2024-04-14T15:20:58.164 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:57 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-14T15:20:58.165 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:57 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-14T15:20:58.165 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:57 smithi090 ceph-mon[30399]: osdmap e532: 8 total, 8 up, 7 in 2024-04-14T15:20:58.165 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:57 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-14T15:20:58.165 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:57 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-14T15:20:58.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:58.184 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:58.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:20:58.285 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:20:58.312 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:58.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:58.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:20:58.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:57 smithi092 ceph-mon[33966]: osdmap e532: 8 total, 8 up, 7 in 2024-04-14T15:20:58.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:57 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-14T15:20:58.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:57 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-14T15:20:58.414 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:20:58.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:58.438 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:58.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:20:58.539 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:20:58.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:58.564 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:58.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:20:58.665 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:20:58.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:58.689 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:58.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:20:58.790 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:20:58.814 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:58.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:58.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:20:58.915 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:20:58.939 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:58.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:58.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:20:59.041 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:20:59.069 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:59.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:59.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:20:59.170 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:20:59.196 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:59.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:59.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:20:59.265 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:20:59.297 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:20:59.321 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:59.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:59.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:20:59.364 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:58 smithi090 ceph-mon[41670]: pgmap v773: 286 pgs: 1 active+remapped+backfill_toofull, 2 undersized+remapped+premerge+peered, 64 unknown, 11 active+undersized+remapped, 208 active+clean; 8.4 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 6.7 KiB/s rd, 46 KiB/s wr, 16 op/s; 1/1170 objects misplaced (0.085%); 31 KiB/s, 69 objects/s recovering 2024-04-14T15:20:59.364 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:58 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-87","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-14T15:20:59.364 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:58 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-87","var": "dedup_tier","val": "test-rados-api-smithi090-93652-89-test-flush"}]: dispatch 2024-04-14T15:20:59.364 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:58 smithi090 ceph-mon[41670]: osdmap e533: 8 total, 8 up, 7 in 2024-04-14T15:20:59.364 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:58 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-87","var": "dedup_tier","val": "test-rados-api-smithi090-93652-89-test-flush"}]: dispatch 2024-04-14T15:20:59.364 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:58 smithi090 ceph-mon[30399]: pgmap v773: 286 pgs: 1 active+remapped+backfill_toofull, 2 undersized+remapped+premerge+peered, 64 unknown, 11 active+undersized+remapped, 208 active+clean; 8.4 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 6.7 KiB/s rd, 46 KiB/s wr, 16 op/s; 1/1170 objects misplaced (0.085%); 31 KiB/s, 69 objects/s recovering 2024-04-14T15:20:59.364 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:58 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-87","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-14T15:20:59.364 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:58 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-87","var": "dedup_tier","val": "test-rados-api-smithi090-93652-89-test-flush"}]: dispatch 2024-04-14T15:20:59.364 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:58 smithi090 ceph-mon[30399]: osdmap e533: 8 total, 8 up, 7 in 2024-04-14T15:20:59.364 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:58 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-87","var": "dedup_tier","val": "test-rados-api-smithi090-93652-89-test-flush"}]: dispatch 2024-04-14T15:20:59.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:58 smithi092 ceph-mon[33966]: pgmap v773: 286 pgs: 1 active+remapped+backfill_toofull, 2 undersized+remapped+premerge+peered, 64 unknown, 11 active+undersized+remapped, 208 active+clean; 8.4 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 6.7 KiB/s rd, 46 KiB/s wr, 16 op/s; 1/1170 objects misplaced (0.085%); 31 KiB/s, 69 objects/s recovering 2024-04-14T15:20:59.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:58 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-87","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-14T15:20:59.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:58 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-87","var": "dedup_tier","val": "test-rados-api-smithi090-93652-89-test-flush"}]: dispatch 2024-04-14T15:20:59.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:58 smithi092 ceph-mon[33966]: osdmap e533: 8 total, 8 up, 7 in 2024-04-14T15:20:59.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:58 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-87","var": "dedup_tier","val": "test-rados-api-smithi090-93652-89-test-flush"}]: dispatch 2024-04-14T15:20:59.423 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:20:59.451 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:59.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:59.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:20:59.489 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-14T15:20:59.553 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:20:59.580 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:59.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:59.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:20:59.682 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:20:59.710 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:20:59.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:59.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:20:59.812 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:20:59.839 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:59.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:59.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:20:59.940 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:20:59.966 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:20:59.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:20:59.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:21:00.063 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-14T15:21:00.063 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:21:00.068 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:21:00.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:00.093 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:00.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:21:00.194 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:21:00.217 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:00.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:00.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:21:00.297 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:59 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-87","var": "dedup_tier","val": "test-rados-api-smithi090-93652-89-test-flush"}]': finished 2024-04-14T15:21:00.298 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:59 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-14T15:21:00.298 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:59 smithi090 ceph-mon[41670]: osdmap e534: 8 total, 8 up, 7 in 2024-04-14T15:21:00.298 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:59 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-14T15:21:00.298 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:20:59 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-14T15:21:00.298 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:59 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-87","var": "dedup_tier","val": "test-rados-api-smithi090-93652-89-test-flush"}]': finished 2024-04-14T15:21:00.298 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:59 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-14T15:21:00.298 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:59 smithi090 ceph-mon[30399]: osdmap e534: 8 total, 8 up, 7 in 2024-04-14T15:21:00.298 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:59 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-14T15:21:00.298 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:20:59 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-14T15:21:00.319 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:00.345 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:00.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:00.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:00.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:59 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-87","var": "dedup_tier","val": "test-rados-api-smithi090-93652-89-test-flush"}]': finished 2024-04-14T15:21:00.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:59 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-14T15:21:00.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:59 smithi092 ceph-mon[33966]: osdmap e534: 8 total, 8 up, 7 in 2024-04-14T15:21:00.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:59 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-14T15:21:00.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:20:59 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-14T15:21:00.448 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:00.474 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:00.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:00.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:00.576 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:21:00.603 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:00.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:00.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:21:00.705 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:21:00.731 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:00.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:00.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:21:00.833 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:21:00.860 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:00.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:00.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:21:00.962 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:21:00.990 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:00.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:00.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:21:01.092 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:21:01.116 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:01.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:01.117 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:21:01.218 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:21:01.243 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:01.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:01.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:21:01.344 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:21:01.369 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:01.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:01.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:21:01.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:01 smithi092 ceph-mon[33966]: pgmap v776: 286 pgs: 1 active+remapped+backfill_toofull, 2 undersized+remapped+premerge+peered, 43 unknown, 6 active+undersized+remapped, 234 active+clean; 8.4 MiB data, 516 MiB used, 625 GiB / 626 GiB avail; 1/1170 objects misplaced (0.085%) 2024-04-14T15:21:01.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:01 smithi092 ceph-mon[33966]: osdmap e535: 8 total, 8 up, 7 in 2024-04-14T15:21:01.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:01 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-14T15:21:01.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:01 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-14T15:21:01.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:01 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-87","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-14T15:21:01.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:01 smithi092 ceph-mon[33966]: osdmap e536: 8 total, 8 up, 7 in 2024-04-14T15:21:01.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:01 smithi090 ceph-mon[41670]: pgmap v776: 286 pgs: 1 active+remapped+backfill_toofull, 2 undersized+remapped+premerge+peered, 43 unknown, 6 active+undersized+remapped, 234 active+clean; 8.4 MiB data, 516 MiB used, 625 GiB / 626 GiB avail; 1/1170 objects misplaced (0.085%) 2024-04-14T15:21:01.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:01 smithi090 ceph-mon[41670]: osdmap e535: 8 total, 8 up, 7 in 2024-04-14T15:21:01.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:01 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-14T15:21:01.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:01 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-14T15:21:01.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:01 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-87","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-14T15:21:01.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:01 smithi090 ceph-mon[41670]: osdmap e536: 8 total, 8 up, 7 in 2024-04-14T15:21:01.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:01 smithi090 ceph-mon[30399]: pgmap v776: 286 pgs: 1 active+remapped+backfill_toofull, 2 undersized+remapped+premerge+peered, 43 unknown, 6 active+undersized+remapped, 234 active+clean; 8.4 MiB data, 516 MiB used, 625 GiB / 626 GiB avail; 1/1170 objects misplaced (0.085%) 2024-04-14T15:21:01.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:01 smithi090 ceph-mon[30399]: osdmap e535: 8 total, 8 up, 7 in 2024-04-14T15:21:01.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:01 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-14T15:21:01.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:01 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-14T15:21:01.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:01 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-87","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-14T15:21:01.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:01 smithi090 ceph-mon[30399]: osdmap e536: 8 total, 8 up, 7 in 2024-04-14T15:21:01.471 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:21:01.495 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:01.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:01.496 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:21:01.578 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:21:01.579 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:21:01.596 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:21:01.622 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:01.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:01.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:21:01.724 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:21:01.748 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:01.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:01.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:21:01.850 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:21:01.874 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:01.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:01.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:21:01.975 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:21:01.999 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:02.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:02.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:21:02.101 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:21:02.124 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:02.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:02.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:21:02.226 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:21:02.251 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:02.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:02.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:21:02.352 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:21:02.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:02 smithi092 ceph-mon[33966]: pgmap v779: 286 pgs: 1 active+remapped+backfill_toofull, 285 active+clean; 8.4 MiB data, 512 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/1170 objects misplaced (0.085%) 2024-04-14T15:21:02.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:02 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-04-14T15:21:02.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:02 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "26"}]': finished 2024-04-14T15:21:02.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:02 smithi092 ceph-mon[33966]: osdmap e537: 8 total, 8 up, 7 in 2024-04-14T15:21:02.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:02 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:21:02.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:02 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:21:02.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:02 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-87"}]: dispatch 2024-04-14T15:21:02.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:02 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-87"}]: dispatch 2024-04-14T15:21:02.377 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:02.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:02.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:21:02.430 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:02 smithi090 ceph-mon[30399]: pgmap v779: 286 pgs: 1 active+remapped+backfill_toofull, 285 active+clean; 8.4 MiB data, 512 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/1170 objects misplaced (0.085%) 2024-04-14T15:21:02.431 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:02 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-04-14T15:21:02.431 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:02 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "26"}]': finished 2024-04-14T15:21:02.431 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:02 smithi090 ceph-mon[30399]: osdmap e537: 8 total, 8 up, 7 in 2024-04-14T15:21:02.431 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:02 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:21:02.431 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:02 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:21:02.431 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:02 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-87"}]: dispatch 2024-04-14T15:21:02.431 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:02 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-87"}]: dispatch 2024-04-14T15:21:02.431 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:02 smithi090 ceph-mon[41670]: pgmap v779: 286 pgs: 1 active+remapped+backfill_toofull, 285 active+clean; 8.4 MiB data, 512 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/1170 objects misplaced (0.085%) 2024-04-14T15:21:02.431 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:02 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-04-14T15:21:02.431 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:02 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "26"}]': finished 2024-04-14T15:21:02.431 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:02 smithi090 ceph-mon[41670]: osdmap e537: 8 total, 8 up, 7 in 2024-04-14T15:21:02.431 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:02 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:21:02.431 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:02 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:21:02.432 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:02 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-87"}]: dispatch 2024-04-14T15:21:02.432 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:02 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-87"}]: dispatch 2024-04-14T15:21:02.479 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:21:02.504 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:02.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:02.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:21:02.606 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:21:02.631 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:02.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:02.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:21:02.644 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-14T15:21:02.732 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:21:02.758 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:02.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:02.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:21:02.860 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:21:02.883 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:02.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:02.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:21:02.984 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:21:03.011 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:03.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:03.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:21:03.112 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:03.139 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:03.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:03.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:03.240 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:21:03.264 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:03.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:03.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:21:03.366 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:21:03.393 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:03.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:03.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:21:03.495 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:03.523 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:03.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:03.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:03.624 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:21:03.648 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:03.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:03.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:21:03.750 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:21:03.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:03.773 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:03.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:21:03.874 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:21:03.900 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:03.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:03.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:21:04.002 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:21:04.002 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:03 smithi090 ceph-mon[30399]: osdmap e538: 8 total, 8 up, 7 in 2024-04-14T15:21:04.003 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:21:03 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:21:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:21:04.003 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:03 smithi090 ceph-mon[41670]: osdmap e538: 8 total, 8 up, 7 in 2024-04-14T15:21:04.026 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:04.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:04.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:21:04.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:03 smithi092 ceph-mon[33966]: osdmap e538: 8 total, 8 up, 7 in 2024-04-14T15:21:04.128 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:21:04.153 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:04.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:04.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:21:04.254 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:21:04.278 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:04.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:04.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:21:04.379 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:21:04.404 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:04.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:04.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:21:04.505 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:21:04.529 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:04.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:04.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:21:04.630 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:21:04.656 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:04.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:04.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:21:04.757 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:21:04.782 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:04.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:04.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:21:04.884 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:21:04.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:04.908 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:04.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:21:05.009 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:05.033 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:05.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:05.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:05.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.082+0000 7fdc3e5a3700 1 -- 172.21.15.90:0/2041985027 >> v1:172.21.15.90:6789/0 conn(0x7fdc3810a6d0 legacy=0x7fdc3810cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:05.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.082+0000 7fdc3e5a3700 1 -- 172.21.15.90:0/2041985027 shutdown_connections 2024-04-14T15:21:05.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.082+0000 7fdc3e5a3700 1 -- 172.21.15.90:0/2041985027 >> 172.21.15.90:0/2041985027 conn(0x7fdc38076aa0 msgr2=0x7fdc38076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:21:05.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.082+0000 7fdc3e5a3700 1 -- 172.21.15.90:0/2041985027 shutdown_connections 2024-04-14T15:21:05.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.082+0000 7fdc3e5a3700 1 -- 172.21.15.90:0/2041985027 wait complete. 2024-04-14T15:21:05.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.083+0000 7fdc3e5a3700 1 librados: starting msgr at 2024-04-14T15:21:05.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.083+0000 7fdc3e5a3700 1 librados: starting objecter 2024-04-14T15:21:05.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.083+0000 7fdc3e5a3700 1 Processor -- start 2024-04-14T15:21:05.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.083+0000 7fdc3e5a3700 1 -- start start 2024-04-14T15:21:05.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.083+0000 7fdc3e5a3700 1 librados: setting wanted keys 2024-04-14T15:21:05.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.083+0000 7fdc3e5a3700 1 librados: calling monclient init 2024-04-14T15:21:05.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.083+0000 7fdc3e5a3700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdc381a79c0 con 0x7fdc381023f0 2024-04-14T15:21:05.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.083+0000 7fdc3e5a3700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdc381a8be0 con 0x7fdc3810e270 2024-04-14T15:21:05.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.083+0000 7fdc3e5a3700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdc381a9e20 con 0x7fdc3810a6d0 2024-04-14T15:21:05.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.083+0000 7fdc37fff700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fdc381023f0 0x7fdc38106d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:60010/0 (socket says 172.21.15.90:60010) 2024-04-14T15:21:05.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.083+0000 7fdc37fff700 1 -- 172.21.15.90:0/4101025460 learned_addr learned my addr 172.21.15.90:0/4101025460 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:21:05.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.083+0000 7fdc357fa700 10 client.?.objecter ms_handle_connect 0x7fdc381023f0 2024-04-14T15:21:05.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.083+0000 7fdc357fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:21:05.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.083+0000 7fdc357fa700 10 client.?.objecter ms_handle_connect 0x7fdc3810a6d0 2024-04-14T15:21:05.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.083+0000 7fdc357fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:21:05.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.083+0000 7fdc357fa700 10 client.?.objecter ms_handle_connect 0x7fdc3810e270 2024-04-14T15:21:05.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.083+0000 7fdc357fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:21:05.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.084+0000 7fdc357fa700 1 -- 172.21.15.90:0/4101025460 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3696845017 0 0) 0x7fdc381a9e20 con 0x7fdc3810a6d0 2024-04-14T15:21:05.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.084+0000 7fdc357fa700 1 -- 172.21.15.90:0/4101025460 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc200035b0 con 0x7fdc3810a6d0 2024-04-14T15:21:05.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.084+0000 7fdc357fa700 1 -- 172.21.15.90:0/4101025460 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 770118275 0 0) 0x7fdc381a79c0 con 0x7fdc381023f0 2024-04-14T15:21:05.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.084+0000 7fdc357fa700 1 -- 172.21.15.90:0/4101025460 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc381a9e20 con 0x7fdc381023f0 2024-04-14T15:21:05.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.084+0000 7fdc357fa700 1 -- 172.21.15.90:0/4101025460 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3172174614 0 0) 0x7fdc381a8be0 con 0x7fdc3810e270 2024-04-14T15:21:05.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.084+0000 7fdc357fa700 1 -- 172.21.15.90:0/4101025460 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc381a79c0 con 0x7fdc3810e270 2024-04-14T15:21:05.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.084+0000 7fdc357fa700 1 -- 172.21.15.90:0/4101025460 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2735612549 0 0) 0x7fdc200035b0 con 0x7fdc3810a6d0 2024-04-14T15:21:05.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.084+0000 7fdc357fa700 1 -- 172.21.15.90:0/4101025460 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdc381a8be0 con 0x7fdc3810a6d0 2024-04-14T15:21:05.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.084+0000 7fdc357fa700 1 -- 172.21.15.90:0/4101025460 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fdc2c004540 con 0x7fdc3810a6d0 2024-04-14T15:21:05.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.084+0000 7fdc357fa700 1 -- 172.21.15.90:0/4101025460 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1151698608 0 0) 0x7fdc381a79c0 con 0x7fdc3810e270 2024-04-14T15:21:05.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.084+0000 7fdc357fa700 1 -- 172.21.15.90:0/4101025460 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdc200035b0 con 0x7fdc3810e270 2024-04-14T15:21:05.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.084+0000 7fdc357fa700 1 -- 172.21.15.90:0/4101025460 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fdc30001470 con 0x7fdc3810e270 2024-04-14T15:21:05.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.084+0000 7fdc357fa700 1 -- 172.21.15.90:0/4101025460 <== mon.2 v1:172.21.15.90:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 487942306 0 0) 0x7fdc381a8be0 con 0x7fdc3810a6d0 2024-04-14T15:21:05.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.084+0000 7fdc357fa700 1 -- 172.21.15.90:0/4101025460 >> v1:172.21.15.92:6789/0 conn(0x7fdc3810e270 legacy=0x7fdc381a6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:05.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.084+0000 7fdc357fa700 1 -- 172.21.15.90:0/4101025460 >> v1:172.21.15.90:6789/0 conn(0x7fdc381023f0 legacy=0x7fdc38106d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:05.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.084+0000 7fdc357fa700 1 -- 172.21.15.90:0/4101025460 --> v1:172.21.15.90:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc381ab060 con 0x7fdc3810a6d0 2024-04-14T15:21:05.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.084+0000 7fdc3e5a3700 1 -- 172.21.15.90:0/4101025460 --> v1:172.21.15.90:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdc381aa050 con 0x7fdc3810a6d0 2024-04-14T15:21:05.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.084+0000 7fdc3e5a3700 10 client.47501.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:21:05.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.084+0000 7fdc3e5a3700 1 -- 172.21.15.90:0/4101025460 --> v1:172.21.15.90:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fdc381aa510 con 0x7fdc3810a6d0 2024-04-14T15:21:05.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.084+0000 7fdc3e5a3700 1 librados: init done 2024-04-14T15:21:05.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.084+0000 7fdc357fa700 1 -- 172.21.15.90:0/4101025460 <== mon.2 v1:172.21.15.90:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fdc2c001b70 con 0x7fdc3810a6d0 2024-04-14T15:21:05.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.084+0000 7fdc357fa700 1 -- 172.21.15.90:0/4101025460 <== mon.2 v1:172.21.15.90:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fdc2c004dc0 con 0x7fdc3810a6d0 2024-04-14T15:21:05.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.085+0000 7fdc3e5a3700 1 -- 172.21.15.90:0/4101025460 --> v1:172.21.15.90:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fdc24005100 con 0x7fdc3810a6d0 2024-04-14T15:21:05.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.086+0000 7fdc357fa700 1 -- 172.21.15.90:0/4101025460 <== mon.2 v1:172.21.15.90:6790/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fdc2c01b9f0 con 0x7fdc3810a6d0 2024-04-14T15:21:05.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.086+0000 7fdc357fa700 1 -- 172.21.15.90:0/4101025460 <== mon.2 v1:172.21.15.90:6790/0 8 ==== osd_map(540..540 src has 467..540) ==== 8230+0+0 (unknown 1164497643 0 0) 0x7fdc2c089a70 con 0x7fdc3810a6d0 2024-04-14T15:21:05.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.086+0000 7fdc357fa700 10 client.47501.objecter ms_dispatch 0x7fdc38001550 osd_map(540..540 src has 467..540) 2024-04-14T15:21:05.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.086+0000 7fdc357fa700 3 client.47501.objecter handle_osd_map got epochs [540,540] > 0 2024-04-14T15:21:05.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.086+0000 7fdc357fa700 3 client.47501.objecter handle_osd_map decoding full epoch 540 2024-04-14T15:21:05.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.086+0000 7fdc357fa700 20 client.47501.objecter dump_active .. 0 homeless 2024-04-14T15:21:05.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.086+0000 7fdc357fa700 1 -- 172.21.15.90:0/4101025460 <== mon.2 v1:172.21.15.90:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v540) ==== 54+0+16 (unknown 1007340935 0 2420000192) 0x7fdc2c089eb0 con 0x7fdc3810a6d0 2024-04-14T15:21:05.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.086+0000 7fdc357fa700 10 client.47501.objecter ms_handle_connect 0x7fdc200714a0 2024-04-14T15:21:05.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.087+0000 7fdc3e5a3700 10 client.47501.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:21:05.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.087+0000 7fdc3e5a3700 15 client.47501.objecter _session_command_op_assign -1 1 2024-04-14T15:21:05.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.087+0000 7fdc3e5a3700 20 client.47501.objecter _get_session s=0x7fdc240053e0 osd=0 3 2024-04-14T15:21:05.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.087+0000 7fdc3e5a3700 20 client.47501.objecter put_session s=0x7fdc240053e0 osd=0 3 2024-04-14T15:21:05.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.087+0000 7fdc3e5a3700 20 client.47501.objecter _get_session s=0x7fdc240053e0 osd=0 3 2024-04-14T15:21:05.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.087+0000 7fdc3e5a3700 15 client.47501.objecter _session_command_op_remove -1 1 2024-04-14T15:21:05.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.087+0000 7fdc3e5a3700 20 client.47501.objecter get_session s=0x7fdc240053e0 osd=0 3 2024-04-14T15:21:05.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.087+0000 7fdc3e5a3700 15 client.47501.objecter _session_command_op_assign 0 1 2024-04-14T15:21:05.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.087+0000 7fdc3e5a3700 20 client.47501.objecter put_session s=0x7fdc240053e0 osd=0 4 2024-04-14T15:21:05.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.087+0000 7fdc3e5a3700 10 client.47501.objecter _send_command 1 2024-04-14T15:21:05.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.087+0000 7fdc3e5a3700 1 -- 172.21.15.90:0/4101025460 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdc24004b70 con 0x7fdc24004730 2024-04-14T15:21:05.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.087+0000 7fdc357fa700 10 client.47501.objecter ms_handle_connect 0x7fdc24004730 2024-04-14T15:21:05.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.087+0000 7fdc357fa700 1 -- 172.21.15.90:0/4101025460 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fdc24004b70 con 0x7fdc24004730 2024-04-14T15:21:05.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.087+0000 7fdc357fa700 10 client.47501.objecter ms_dispatch 0x7fdc38001550 command_reply(tid 1: 0 ) 2024-04-14T15:21:05.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.087+0000 7fdc357fa700 10 client.47501.objecter _finish_command 1 = system:0 2024-04-14T15:21:05.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.087+0000 7fdc357fa700 20 client.47501.objecter put_session s=0x7fdc240053e0 osd=0 4 2024-04-14T15:21:05.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.087+0000 7fdc357fa700 15 client.47501.objecter _session_command_op_remove 0 1 2024-04-14T15:21:05.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.104+0000 7fdc3e5a3700 10 client.47501.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:21:05.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.104+0000 7fdc3e5a3700 15 client.47501.objecter _session_command_op_assign -1 2 2024-04-14T15:21:05.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.104+0000 7fdc3e5a3700 20 client.47501.objecter _get_session s=0x7fdc240053e0 osd=0 3 2024-04-14T15:21:05.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.104+0000 7fdc3e5a3700 20 client.47501.objecter put_session s=0x7fdc240053e0 osd=0 3 2024-04-14T15:21:05.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.104+0000 7fdc3e5a3700 20 client.47501.objecter _get_session s=0x7fdc240053e0 osd=0 3 2024-04-14T15:21:05.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.104+0000 7fdc3e5a3700 15 client.47501.objecter _session_command_op_remove -1 2 2024-04-14T15:21:05.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.104+0000 7fdc3e5a3700 20 client.47501.objecter get_session s=0x7fdc240053e0 osd=0 3 2024-04-14T15:21:05.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.104+0000 7fdc3e5a3700 15 client.47501.objecter _session_command_op_assign 0 2 2024-04-14T15:21:05.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.104+0000 7fdc3e5a3700 20 client.47501.objecter put_session s=0x7fdc240053e0 osd=0 4 2024-04-14T15:21:05.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.104+0000 7fdc3e5a3700 10 client.47501.objecter _send_command 2 2024-04-14T15:21:05.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.104+0000 7fdc3e5a3700 1 -- 172.21.15.90:0/4101025460 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdc240079f0 con 0x7fdc24004730 2024-04-14T15:21:05.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.106+0000 7fdc357fa700 1 -- 172.21.15.90:0/4101025460 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fdc240079f0 con 0x7fdc24004730 2024-04-14T15:21:05.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.106+0000 7fdc357fa700 10 client.47501.objecter ms_dispatch 0x7fdc38001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:21:05.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.106+0000 7fdc357fa700 10 client.47501.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:21:05.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.106+0000 7fdc357fa700 20 client.47501.objecter put_session s=0x7fdc240053e0 osd=0 4 2024-04-14T15:21:05.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.106+0000 7fdc357fa700 15 client.47501.objecter _session_command_op_remove 0 2 2024-04-14T15:21:05.107 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:21:05.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.106+0000 7fdc3e5a3700 10 client.47501.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:21:05.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.106+0000 7fdc3e5a3700 15 client.47501.objecter _session_command_op_assign -1 3 2024-04-14T15:21:05.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.106+0000 7fdc3e5a3700 20 client.47501.objecter _get_session s=0x7fdc24004130 osd=1 3 2024-04-14T15:21:05.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.106+0000 7fdc3e5a3700 20 client.47501.objecter put_session s=0x7fdc24004130 osd=1 3 2024-04-14T15:21:05.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.106+0000 7fdc3e5a3700 20 client.47501.objecter _get_session s=0x7fdc24004130 osd=1 3 2024-04-14T15:21:05.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.106+0000 7fdc3e5a3700 15 client.47501.objecter _session_command_op_remove -1 3 2024-04-14T15:21:05.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.106+0000 7fdc3e5a3700 20 client.47501.objecter get_session s=0x7fdc24004130 osd=1 3 2024-04-14T15:21:05.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.106+0000 7fdc3e5a3700 15 client.47501.objecter _session_command_op_assign 1 3 2024-04-14T15:21:05.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.106+0000 7fdc3e5a3700 20 client.47501.objecter put_session s=0x7fdc24004130 osd=1 4 2024-04-14T15:21:05.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.106+0000 7fdc3e5a3700 10 client.47501.objecter _send_command 3 2024-04-14T15:21:05.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.106+0000 7fdc3e5a3700 1 -- 172.21.15.90:0/4101025460 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fdc240022a0 con 0x7fdc24004310 2024-04-14T15:21:05.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.106+0000 7fdc357fa700 10 client.47501.objecter ms_handle_connect 0x7fdc24004310 2024-04-14T15:21:05.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.107+0000 7fdc357fa700 1 -- 172.21.15.90:0/4101025460 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fdc240022a0 con 0x7fdc24004310 2024-04-14T15:21:05.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.107+0000 7fdc357fa700 10 client.47501.objecter ms_dispatch 0x7fdc38001550 command_reply(tid 3: 0 ) 2024-04-14T15:21:05.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.107+0000 7fdc357fa700 10 client.47501.objecter _finish_command 3 = system:0 2024-04-14T15:21:05.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.107+0000 7fdc357fa700 20 client.47501.objecter put_session s=0x7fdc24004130 osd=1 4 2024-04-14T15:21:05.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.107+0000 7fdc357fa700 15 client.47501.objecter _session_command_op_remove 1 3 2024-04-14T15:21:05.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.124+0000 7fdc3e5a3700 10 client.47501.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:21:05.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.124+0000 7fdc3e5a3700 15 client.47501.objecter _session_command_op_assign -1 4 2024-04-14T15:21:05.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.124+0000 7fdc3e5a3700 20 client.47501.objecter _get_session s=0x7fdc24004130 osd=1 3 2024-04-14T15:21:05.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.124+0000 7fdc3e5a3700 20 client.47501.objecter put_session s=0x7fdc24004130 osd=1 3 2024-04-14T15:21:05.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.124+0000 7fdc3e5a3700 20 client.47501.objecter _get_session s=0x7fdc24004130 osd=1 3 2024-04-14T15:21:05.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.124+0000 7fdc3e5a3700 15 client.47501.objecter _session_command_op_remove -1 4 2024-04-14T15:21:05.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.124+0000 7fdc3e5a3700 20 client.47501.objecter get_session s=0x7fdc24004130 osd=1 3 2024-04-14T15:21:05.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.124+0000 7fdc3e5a3700 15 client.47501.objecter _session_command_op_assign 1 4 2024-04-14T15:21:05.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.124+0000 7fdc3e5a3700 20 client.47501.objecter put_session s=0x7fdc24004130 osd=1 4 2024-04-14T15:21:05.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.124+0000 7fdc3e5a3700 10 client.47501.objecter _send_command 4 2024-04-14T15:21:05.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.124+0000 7fdc3e5a3700 1 -- 172.21.15.90:0/4101025460 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdc24001720 con 0x7fdc24004310 2024-04-14T15:21:05.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.125+0000 7fdc357fa700 1 -- 172.21.15.90:0/4101025460 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fdc24001720 con 0x7fdc24004310 2024-04-14T15:21:05.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.125+0000 7fdc357fa700 10 client.47501.objecter ms_dispatch 0x7fdc38001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:21:05.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.125+0000 7fdc357fa700 10 client.47501.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:21:05.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.125+0000 7fdc357fa700 20 client.47501.objecter put_session s=0x7fdc24004130 osd=1 4 2024-04-14T15:21:05.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.125+0000 7fdc357fa700 15 client.47501.objecter _session_command_op_remove 1 4 2024-04-14T15:21:05.126 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:21:05.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.126+0000 7fdc3e5a3700 10 client.47501.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:21:05.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.126+0000 7fdc3e5a3700 15 client.47501.objecter _session_command_op_assign -1 5 2024-04-14T15:21:05.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.126+0000 7fdc3e5a3700 20 client.47501.objecter _get_session s=0x7fdc24001980 osd=2 3 2024-04-14T15:21:05.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.126+0000 7fdc3e5a3700 20 client.47501.objecter put_session s=0x7fdc24001980 osd=2 3 2024-04-14T15:21:05.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.126+0000 7fdc3e5a3700 20 client.47501.objecter _get_session s=0x7fdc24001980 osd=2 3 2024-04-14T15:21:05.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.126+0000 7fdc3e5a3700 15 client.47501.objecter _session_command_op_remove -1 5 2024-04-14T15:21:05.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.126+0000 7fdc3e5a3700 20 client.47501.objecter get_session s=0x7fdc24001980 osd=2 3 2024-04-14T15:21:05.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.126+0000 7fdc3e5a3700 15 client.47501.objecter _session_command_op_assign 2 5 2024-04-14T15:21:05.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.126+0000 7fdc3e5a3700 20 client.47501.objecter put_session s=0x7fdc24001980 osd=2 4 2024-04-14T15:21:05.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.126+0000 7fdc3e5a3700 10 client.47501.objecter _send_command 5 2024-04-14T15:21:05.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.126+0000 7fdc3e5a3700 1 -- 172.21.15.90:0/4101025460 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fdc240154f0 con 0x7fdc240119e0 2024-04-14T15:21:05.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.126+0000 7fdc357fa700 10 client.47501.objecter ms_handle_connect 0x7fdc240119e0 2024-04-14T15:21:05.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.127+0000 7fdc357fa700 1 -- 172.21.15.90:0/4101025460 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fdc240154f0 con 0x7fdc240119e0 2024-04-14T15:21:05.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.127+0000 7fdc357fa700 10 client.47501.objecter ms_dispatch 0x7fdc38001550 command_reply(tid 5: 0 ) 2024-04-14T15:21:05.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.127+0000 7fdc357fa700 10 client.47501.objecter _finish_command 5 = system:0 2024-04-14T15:21:05.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.127+0000 7fdc357fa700 20 client.47501.objecter put_session s=0x7fdc24001980 osd=2 4 2024-04-14T15:21:05.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.127+0000 7fdc357fa700 15 client.47501.objecter _session_command_op_remove 2 5 2024-04-14T15:21:05.135 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:21:05.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.143+0000 7fdc3e5a3700 10 client.47501.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:21:05.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.143+0000 7fdc3e5a3700 15 client.47501.objecter _session_command_op_assign -1 6 2024-04-14T15:21:05.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.143+0000 7fdc3e5a3700 20 client.47501.objecter _get_session s=0x7fdc24001980 osd=2 3 2024-04-14T15:21:05.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.143+0000 7fdc3e5a3700 20 client.47501.objecter put_session s=0x7fdc24001980 osd=2 3 2024-04-14T15:21:05.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.143+0000 7fdc3e5a3700 20 client.47501.objecter _get_session s=0x7fdc24001980 osd=2 3 2024-04-14T15:21:05.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.143+0000 7fdc3e5a3700 15 client.47501.objecter _session_command_op_remove -1 6 2024-04-14T15:21:05.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.143+0000 7fdc3e5a3700 20 client.47501.objecter get_session s=0x7fdc24001980 osd=2 3 2024-04-14T15:21:05.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.143+0000 7fdc3e5a3700 15 client.47501.objecter _session_command_op_assign 2 6 2024-04-14T15:21:05.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.143+0000 7fdc3e5a3700 20 client.47501.objecter put_session s=0x7fdc24001980 osd=2 4 2024-04-14T15:21:05.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.143+0000 7fdc3e5a3700 10 client.47501.objecter _send_command 6 2024-04-14T15:21:05.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.143+0000 7fdc3e5a3700 1 -- 172.21.15.90:0/4101025460 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdc24008100 con 0x7fdc240119e0 2024-04-14T15:21:05.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.145+0000 7fdc357fa700 1 -- 172.21.15.90:0/4101025460 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fdc24008100 con 0x7fdc240119e0 2024-04-14T15:21:05.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.145+0000 7fdc357fa700 10 client.47501.objecter ms_dispatch 0x7fdc38001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:21:05.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.145+0000 7fdc357fa700 10 client.47501.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:21:05.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.145+0000 7fdc357fa700 20 client.47501.objecter put_session s=0x7fdc24001980 osd=2 4 2024-04-14T15:21:05.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.145+0000 7fdc357fa700 15 client.47501.objecter _session_command_op_remove 2 6 2024-04-14T15:21:05.146 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:21:05.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.145+0000 7fdc3e5a3700 10 client.47501.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:21:05.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.145+0000 7fdc3e5a3700 15 client.47501.objecter _session_command_op_assign -1 7 2024-04-14T15:21:05.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.145+0000 7fdc3e5a3700 20 client.47501.objecter _get_session s=0x7fdc2400cc20 osd=3 3 2024-04-14T15:21:05.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.145+0000 7fdc3e5a3700 20 client.47501.objecter put_session s=0x7fdc2400cc20 osd=3 3 2024-04-14T15:21:05.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.145+0000 7fdc3e5a3700 20 client.47501.objecter _get_session s=0x7fdc2400cc20 osd=3 3 2024-04-14T15:21:05.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.145+0000 7fdc3e5a3700 15 client.47501.objecter _session_command_op_remove -1 7 2024-04-14T15:21:05.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.145+0000 7fdc3e5a3700 20 client.47501.objecter get_session s=0x7fdc2400cc20 osd=3 3 2024-04-14T15:21:05.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.145+0000 7fdc3e5a3700 15 client.47501.objecter _session_command_op_assign 3 7 2024-04-14T15:21:05.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.145+0000 7fdc3e5a3700 20 client.47501.objecter put_session s=0x7fdc2400cc20 osd=3 4 2024-04-14T15:21:05.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.145+0000 7fdc3e5a3700 10 client.47501.objecter _send_command 7 2024-04-14T15:21:05.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.145+0000 7fdc3e5a3700 1 -- 172.21.15.90:0/4101025460 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fdc240161d0 con 0x7fdc240156a0 2024-04-14T15:21:05.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.146+0000 7fdc357fa700 10 client.47501.objecter ms_handle_connect 0x7fdc240156a0 2024-04-14T15:21:05.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.146+0000 7fdc357fa700 1 -- 172.21.15.90:0/4101025460 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fdc240161d0 con 0x7fdc240156a0 2024-04-14T15:21:05.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.146+0000 7fdc357fa700 10 client.47501.objecter ms_dispatch 0x7fdc38001550 command_reply(tid 7: 0 ) 2024-04-14T15:21:05.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.146+0000 7fdc357fa700 10 client.47501.objecter _finish_command 7 = system:0 2024-04-14T15:21:05.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.146+0000 7fdc357fa700 20 client.47501.objecter put_session s=0x7fdc2400cc20 osd=3 4 2024-04-14T15:21:05.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.146+0000 7fdc357fa700 15 client.47501.objecter _session_command_op_remove 3 7 2024-04-14T15:21:05.158 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:05.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:05.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:21:05.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.164+0000 7fdc3e5a3700 10 client.47501.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:21:05.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.164+0000 7fdc3e5a3700 15 client.47501.objecter _session_command_op_assign -1 8 2024-04-14T15:21:05.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.164+0000 7fdc3e5a3700 20 client.47501.objecter _get_session s=0x7fdc2400cc20 osd=3 3 2024-04-14T15:21:05.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.164+0000 7fdc3e5a3700 20 client.47501.objecter put_session s=0x7fdc2400cc20 osd=3 3 2024-04-14T15:21:05.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.164+0000 7fdc3e5a3700 20 client.47501.objecter _get_session s=0x7fdc2400cc20 osd=3 3 2024-04-14T15:21:05.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.164+0000 7fdc3e5a3700 15 client.47501.objecter _session_command_op_remove -1 8 2024-04-14T15:21:05.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.164+0000 7fdc3e5a3700 20 client.47501.objecter get_session s=0x7fdc2400cc20 osd=3 3 2024-04-14T15:21:05.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.164+0000 7fdc3e5a3700 15 client.47501.objecter _session_command_op_assign 3 8 2024-04-14T15:21:05.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.164+0000 7fdc3e5a3700 20 client.47501.objecter put_session s=0x7fdc2400cc20 osd=3 4 2024-04-14T15:21:05.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.164+0000 7fdc3e5a3700 10 client.47501.objecter _send_command 8 2024-04-14T15:21:05.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.164+0000 7fdc3e5a3700 1 -- 172.21.15.90:0/4101025460 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdc240161d0 con 0x7fdc240156a0 2024-04-14T15:21:05.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.165+0000 7fdc357fa700 1 -- 172.21.15.90:0/4101025460 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fdc240161d0 con 0x7fdc240156a0 2024-04-14T15:21:05.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.165+0000 7fdc357fa700 10 client.47501.objecter ms_dispatch 0x7fdc38001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:21:05.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.166+0000 7fdc357fa700 10 client.47501.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:21:05.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.166+0000 7fdc357fa700 20 client.47501.objecter put_session s=0x7fdc2400cc20 osd=3 4 2024-04-14T15:21:05.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.166+0000 7fdc357fa700 15 client.47501.objecter _session_command_op_remove 3 8 2024-04-14T15:21:05.166 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:21:05.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.166+0000 7fdc3e5a3700 10 client.47501.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:21:05.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.166+0000 7fdc3e5a3700 15 client.47501.objecter _session_command_op_assign -1 9 2024-04-14T15:21:05.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.166+0000 7fdc3e5a3700 20 client.47501.objecter _get_session s=0x7fdc240161d0 osd=4 3 2024-04-14T15:21:05.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.166+0000 7fdc3e5a3700 20 client.47501.objecter put_session s=0x7fdc240161d0 osd=4 3 2024-04-14T15:21:05.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.166+0000 7fdc3e5a3700 20 client.47501.objecter _get_session s=0x7fdc240161d0 osd=4 3 2024-04-14T15:21:05.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.166+0000 7fdc3e5a3700 15 client.47501.objecter _session_command_op_remove -1 9 2024-04-14T15:21:05.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.166+0000 7fdc3e5a3700 20 client.47501.objecter get_session s=0x7fdc240161d0 osd=4 3 2024-04-14T15:21:05.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.166+0000 7fdc3e5a3700 15 client.47501.objecter _session_command_op_assign 4 9 2024-04-14T15:21:05.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.166+0000 7fdc3e5a3700 20 client.47501.objecter put_session s=0x7fdc240161d0 osd=4 4 2024-04-14T15:21:05.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.166+0000 7fdc3e5a3700 10 client.47501.objecter _send_command 9 2024-04-14T15:21:05.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.166+0000 7fdc3e5a3700 1 -- 172.21.15.90:0/4101025460 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fdc2401ebc0 con 0x7fdc2401b0b0 2024-04-14T15:21:05.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.166+0000 7fdc357fa700 10 client.47501.objecter ms_handle_connect 0x7fdc2401b0b0 2024-04-14T15:21:05.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.167+0000 7fdc357fa700 1 -- 172.21.15.90:0/4101025460 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fdc2401ebc0 con 0x7fdc2401b0b0 2024-04-14T15:21:05.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.167+0000 7fdc357fa700 10 client.47501.objecter ms_dispatch 0x7fdc38001550 command_reply(tid 9: 0 ) 2024-04-14T15:21:05.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.167+0000 7fdc357fa700 10 client.47501.objecter _finish_command 9 = system:0 2024-04-14T15:21:05.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.167+0000 7fdc357fa700 20 client.47501.objecter put_session s=0x7fdc240161d0 osd=4 4 2024-04-14T15:21:05.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.167+0000 7fdc357fa700 15 client.47501.objecter _session_command_op_remove 4 9 2024-04-14T15:21:05.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.183+0000 7fdc3e5a3700 10 client.47501.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:21:05.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.183+0000 7fdc3e5a3700 15 client.47501.objecter _session_command_op_assign -1 10 2024-04-14T15:21:05.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.183+0000 7fdc3e5a3700 20 client.47501.objecter _get_session s=0x7fdc240161d0 osd=4 3 2024-04-14T15:21:05.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.183+0000 7fdc3e5a3700 20 client.47501.objecter put_session s=0x7fdc240161d0 osd=4 3 2024-04-14T15:21:05.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.183+0000 7fdc3e5a3700 20 client.47501.objecter _get_session s=0x7fdc240161d0 osd=4 3 2024-04-14T15:21:05.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.183+0000 7fdc3e5a3700 15 client.47501.objecter _session_command_op_remove -1 10 2024-04-14T15:21:05.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.183+0000 7fdc3e5a3700 20 client.47501.objecter get_session s=0x7fdc240161d0 osd=4 3 2024-04-14T15:21:05.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.183+0000 7fdc3e5a3700 15 client.47501.objecter _session_command_op_assign 4 10 2024-04-14T15:21:05.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.183+0000 7fdc3e5a3700 20 client.47501.objecter put_session s=0x7fdc240161d0 osd=4 4 2024-04-14T15:21:05.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.183+0000 7fdc3e5a3700 10 client.47501.objecter _send_command 10 2024-04-14T15:21:05.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.183+0000 7fdc3e5a3700 1 -- 172.21.15.90:0/4101025460 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdc24000930 con 0x7fdc2401b0b0 2024-04-14T15:21:05.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.185+0000 7fdc357fa700 1 -- 172.21.15.90:0/4101025460 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fdc24000930 con 0x7fdc2401b0b0 2024-04-14T15:21:05.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.186+0000 7fdc357fa700 10 client.47501.objecter ms_dispatch 0x7fdc38001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:21:05.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.186+0000 7fdc357fa700 10 client.47501.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:21:05.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.186+0000 7fdc357fa700 20 client.47501.objecter put_session s=0x7fdc240161d0 osd=4 4 2024-04-14T15:21:05.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.186+0000 7fdc357fa700 15 client.47501.objecter _session_command_op_remove 4 10 2024-04-14T15:21:05.186 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:21:05.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.186+0000 7fdc3e5a3700 10 client.47501.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:21:05.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.186+0000 7fdc3e5a3700 15 client.47501.objecter _session_command_op_assign -1 11 2024-04-14T15:21:05.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.186+0000 7fdc3e5a3700 20 client.47501.objecter _get_session s=0x7fdc24000930 osd=5 3 2024-04-14T15:21:05.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.186+0000 7fdc3e5a3700 20 client.47501.objecter put_session s=0x7fdc24000930 osd=5 3 2024-04-14T15:21:05.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.186+0000 7fdc3e5a3700 20 client.47501.objecter _get_session s=0x7fdc24000930 osd=5 3 2024-04-14T15:21:05.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.186+0000 7fdc3e5a3700 15 client.47501.objecter _session_command_op_remove -1 11 2024-04-14T15:21:05.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.186+0000 7fdc3e5a3700 20 client.47501.objecter get_session s=0x7fdc24000930 osd=5 3 2024-04-14T15:21:05.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.186+0000 7fdc3e5a3700 15 client.47501.objecter _session_command_op_assign 5 11 2024-04-14T15:21:05.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.186+0000 7fdc3e5a3700 20 client.47501.objecter put_session s=0x7fdc24000930 osd=5 4 2024-04-14T15:21:05.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.186+0000 7fdc3e5a3700 10 client.47501.objecter _send_command 11 2024-04-14T15:21:05.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.186+0000 7fdc3e5a3700 1 -- 172.21.15.90:0/4101025460 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fdc24017c00 con 0x7fdc24017670 2024-04-14T15:21:05.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.186+0000 7fdc357fa700 10 client.47501.objecter ms_handle_connect 0x7fdc24017670 2024-04-14T15:21:05.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.187+0000 7fdc357fa700 1 -- 172.21.15.90:0/4101025460 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fdc24017c00 con 0x7fdc24017670 2024-04-14T15:21:05.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.187+0000 7fdc357fa700 10 client.47501.objecter ms_dispatch 0x7fdc38001550 command_reply(tid 11: 0 ) 2024-04-14T15:21:05.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.187+0000 7fdc357fa700 10 client.47501.objecter _finish_command 11 = system:0 2024-04-14T15:21:05.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.187+0000 7fdc357fa700 20 client.47501.objecter put_session s=0x7fdc24000930 osd=5 4 2024-04-14T15:21:05.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.187+0000 7fdc357fa700 15 client.47501.objecter _session_command_op_remove 5 11 2024-04-14T15:21:05.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.204+0000 7fdc3e5a3700 10 client.47501.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:21:05.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.204+0000 7fdc3e5a3700 15 client.47501.objecter _session_command_op_assign -1 12 2024-04-14T15:21:05.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.204+0000 7fdc3e5a3700 20 client.47501.objecter _get_session s=0x7fdc24000930 osd=5 3 2024-04-14T15:21:05.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.204+0000 7fdc3e5a3700 20 client.47501.objecter put_session s=0x7fdc24000930 osd=5 3 2024-04-14T15:21:05.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.204+0000 7fdc3e5a3700 20 client.47501.objecter _get_session s=0x7fdc24000930 osd=5 3 2024-04-14T15:21:05.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.204+0000 7fdc3e5a3700 15 client.47501.objecter _session_command_op_remove -1 12 2024-04-14T15:21:05.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.204+0000 7fdc3e5a3700 20 client.47501.objecter get_session s=0x7fdc24000930 osd=5 3 2024-04-14T15:21:05.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.204+0000 7fdc3e5a3700 15 client.47501.objecter _session_command_op_assign 5 12 2024-04-14T15:21:05.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.204+0000 7fdc3e5a3700 20 client.47501.objecter put_session s=0x7fdc24000930 osd=5 4 2024-04-14T15:21:05.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.204+0000 7fdc3e5a3700 10 client.47501.objecter _send_command 12 2024-04-14T15:21:05.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.204+0000 7fdc3e5a3700 1 -- 172.21.15.90:0/4101025460 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdc24016760 con 0x7fdc24017670 2024-04-14T15:21:05.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.206+0000 7fdc357fa700 1 -- 172.21.15.90:0/4101025460 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fdc2c056460 con 0x7fdc24017670 2024-04-14T15:21:05.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.206+0000 7fdc357fa700 10 client.47501.objecter ms_dispatch 0x7fdc38001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:21:05.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.206+0000 7fdc357fa700 10 client.47501.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:21:05.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.206+0000 7fdc357fa700 20 client.47501.objecter put_session s=0x7fdc24000930 osd=5 4 2024-04-14T15:21:05.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.206+0000 7fdc357fa700 15 client.47501.objecter _session_command_op_remove 5 12 2024-04-14T15:21:05.207 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:21:05.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.207+0000 7fdc3e5a3700 10 client.47501.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:21:05.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.207+0000 7fdc3e5a3700 15 client.47501.objecter _session_command_op_assign -1 13 2024-04-14T15:21:05.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.207+0000 7fdc3e5a3700 20 client.47501.objecter _get_session s=0x7fdc24017e10 osd=6 3 2024-04-14T15:21:05.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.207+0000 7fdc3e5a3700 20 client.47501.objecter put_session s=0x7fdc24017e10 osd=6 3 2024-04-14T15:21:05.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.207+0000 7fdc3e5a3700 20 client.47501.objecter _get_session s=0x7fdc24017e10 osd=6 3 2024-04-14T15:21:05.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.207+0000 7fdc3e5a3700 15 client.47501.objecter _session_command_op_remove -1 13 2024-04-14T15:21:05.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.207+0000 7fdc3e5a3700 20 client.47501.objecter get_session s=0x7fdc24017e10 osd=6 3 2024-04-14T15:21:05.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.207+0000 7fdc3e5a3700 15 client.47501.objecter _session_command_op_assign 6 13 2024-04-14T15:21:05.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.207+0000 7fdc3e5a3700 20 client.47501.objecter put_session s=0x7fdc24017e10 osd=6 4 2024-04-14T15:21:05.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.207+0000 7fdc3e5a3700 10 client.47501.objecter _send_command 13 2024-04-14T15:21:05.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.207+0000 7fdc3e5a3700 1 -- 172.21.15.90:0/4101025460 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fdc24027640 con 0x7fdc24023980 2024-04-14T15:21:05.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.207+0000 7fdc357fa700 10 client.47501.objecter ms_handle_connect 0x7fdc24023980 2024-04-14T15:21:05.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.208+0000 7fdc357fa700 1 -- 172.21.15.90:0/4101025460 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fdc24027640 con 0x7fdc24023980 2024-04-14T15:21:05.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.208+0000 7fdc357fa700 10 client.47501.objecter ms_dispatch 0x7fdc38001550 command_reply(tid 13: 0 ) 2024-04-14T15:21:05.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.208+0000 7fdc357fa700 10 client.47501.objecter _finish_command 13 = system:0 2024-04-14T15:21:05.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.208+0000 7fdc357fa700 20 client.47501.objecter put_session s=0x7fdc24017e10 osd=6 4 2024-04-14T15:21:05.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.208+0000 7fdc357fa700 15 client.47501.objecter _session_command_op_remove 6 13 2024-04-14T15:21:05.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.225+0000 7fdc3e5a3700 10 client.47501.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:21:05.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.225+0000 7fdc3e5a3700 15 client.47501.objecter _session_command_op_assign -1 14 2024-04-14T15:21:05.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.225+0000 7fdc3e5a3700 20 client.47501.objecter _get_session s=0x7fdc24017e10 osd=6 3 2024-04-14T15:21:05.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.225+0000 7fdc3e5a3700 20 client.47501.objecter put_session s=0x7fdc24017e10 osd=6 3 2024-04-14T15:21:05.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.225+0000 7fdc3e5a3700 20 client.47501.objecter _get_session s=0x7fdc24017e10 osd=6 3 2024-04-14T15:21:05.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.225+0000 7fdc3e5a3700 15 client.47501.objecter _session_command_op_remove -1 14 2024-04-14T15:21:05.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.225+0000 7fdc3e5a3700 20 client.47501.objecter get_session s=0x7fdc24017e10 osd=6 3 2024-04-14T15:21:05.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.225+0000 7fdc3e5a3700 15 client.47501.objecter _session_command_op_assign 6 14 2024-04-14T15:21:05.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.225+0000 7fdc3e5a3700 20 client.47501.objecter put_session s=0x7fdc24017e10 osd=6 4 2024-04-14T15:21:05.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.225+0000 7fdc3e5a3700 10 client.47501.objecter _send_command 14 2024-04-14T15:21:05.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.225+0000 7fdc3e5a3700 1 -- 172.21.15.90:0/4101025460 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdc24028ff0 con 0x7fdc24023980 2024-04-14T15:21:05.226 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:04 smithi090 ceph-mon[41670]: pgmap v782: 222 pgs: 1 active+remapped+backfill_toofull, 221 active+clean; 8.4 MiB data, 512 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/1170 objects misplaced (0.085%) 2024-04-14T15:21:05.226 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:04 smithi090 ceph-mon[41670]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:21:05.226 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:04 smithi090 ceph-mon[41670]: osdmap e539: 8 total, 8 up, 7 in 2024-04-14T15:21:05.226 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:04 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:21:05.227 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:04 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:21:05.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:04 smithi090 ceph-mon[30399]: pgmap v782: 222 pgs: 1 active+remapped+backfill_toofull, 221 active+clean; 8.4 MiB data, 512 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/1170 objects misplaced (0.085%) 2024-04-14T15:21:05.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:04 smithi090 ceph-mon[30399]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:21:05.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:04 smithi090 ceph-mon[30399]: osdmap e539: 8 total, 8 up, 7 in 2024-04-14T15:21:05.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:04 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:21:05.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:04 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:21:05.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.227+0000 7fdc357fa700 1 -- 172.21.15.90:0/4101025460 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fdc24028ff0 con 0x7fdc24023980 2024-04-14T15:21:05.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.227+0000 7fdc357fa700 10 client.47501.objecter ms_dispatch 0x7fdc38001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:21:05.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.227+0000 7fdc357fa700 10 client.47501.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:21:05.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.227+0000 7fdc357fa700 20 client.47501.objecter put_session s=0x7fdc24017e10 osd=6 4 2024-04-14T15:21:05.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.227+0000 7fdc357fa700 15 client.47501.objecter _session_command_op_remove 6 14 2024-04-14T15:21:05.228 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:21:05.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.227+0000 7fdc3e5a3700 10 client.47501.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:21:05.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.227+0000 7fdc3e5a3700 15 client.47501.objecter _session_command_op_assign -1 15 2024-04-14T15:21:05.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.227+0000 7fdc3e5a3700 20 client.47501.objecter _get_session s=0x7fdc24028ff0 osd=7 3 2024-04-14T15:21:05.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.227+0000 7fdc3e5a3700 20 client.47501.objecter put_session s=0x7fdc24028ff0 osd=7 3 2024-04-14T15:21:05.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.227+0000 7fdc3e5a3700 20 client.47501.objecter _get_session s=0x7fdc24028ff0 osd=7 3 2024-04-14T15:21:05.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.227+0000 7fdc3e5a3700 15 client.47501.objecter _session_command_op_remove -1 15 2024-04-14T15:21:05.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.227+0000 7fdc3e5a3700 20 client.47501.objecter get_session s=0x7fdc24028ff0 osd=7 3 2024-04-14T15:21:05.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.227+0000 7fdc3e5a3700 15 client.47501.objecter _session_command_op_assign 7 15 2024-04-14T15:21:05.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.227+0000 7fdc3e5a3700 20 client.47501.objecter put_session s=0x7fdc24028ff0 osd=7 4 2024-04-14T15:21:05.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.227+0000 7fdc3e5a3700 10 client.47501.objecter _send_command 15 2024-04-14T15:21:05.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.227+0000 7fdc3e5a3700 1 -- 172.21.15.90:0/4101025460 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fdc24029b70 con 0x7fdc24029710 2024-04-14T15:21:05.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.227+0000 7fdc357fa700 10 client.47501.objecter ms_handle_connect 0x7fdc24029710 2024-04-14T15:21:05.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.228+0000 7fdc357fa700 1 -- 172.21.15.90:0/4101025460 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fdc24029b70 con 0x7fdc24029710 2024-04-14T15:21:05.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.228+0000 7fdc357fa700 10 client.47501.objecter ms_dispatch 0x7fdc38001550 command_reply(tid 15: 0 ) 2024-04-14T15:21:05.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.228+0000 7fdc357fa700 10 client.47501.objecter _finish_command 15 = system:0 2024-04-14T15:21:05.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.228+0000 7fdc357fa700 20 client.47501.objecter put_session s=0x7fdc24028ff0 osd=7 4 2024-04-14T15:21:05.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.228+0000 7fdc357fa700 15 client.47501.objecter _session_command_op_remove 7 15 2024-04-14T15:21:05.235 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:04 smithi092 ceph-mon[33966]: pgmap v782: 222 pgs: 1 active+remapped+backfill_toofull, 221 active+clean; 8.4 MiB data, 512 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/1170 objects misplaced (0.085%) 2024-04-14T15:21:05.235 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:04 smithi092 ceph-mon[33966]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:21:05.235 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:04 smithi092 ceph-mon[33966]: osdmap e539: 8 total, 8 up, 7 in 2024-04-14T15:21:05.235 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:04 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:21:05.235 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:04 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:21:05.259 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:21:05.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.263+0000 7fdc3e5a3700 10 client.47501.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:21:05.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.263+0000 7fdc3e5a3700 15 client.47501.objecter _session_command_op_assign -1 16 2024-04-14T15:21:05.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.263+0000 7fdc3e5a3700 20 client.47501.objecter _get_session s=0x7fdc24028ff0 osd=7 3 2024-04-14T15:21:05.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.263+0000 7fdc3e5a3700 20 client.47501.objecter put_session s=0x7fdc24028ff0 osd=7 3 2024-04-14T15:21:05.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.263+0000 7fdc3e5a3700 20 client.47501.objecter _get_session s=0x7fdc24028ff0 osd=7 3 2024-04-14T15:21:05.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.263+0000 7fdc3e5a3700 15 client.47501.objecter _session_command_op_remove -1 16 2024-04-14T15:21:05.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.263+0000 7fdc3e5a3700 20 client.47501.objecter get_session s=0x7fdc24028ff0 osd=7 3 2024-04-14T15:21:05.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.263+0000 7fdc3e5a3700 15 client.47501.objecter _session_command_op_assign 7 16 2024-04-14T15:21:05.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.263+0000 7fdc3e5a3700 20 client.47501.objecter put_session s=0x7fdc24028ff0 osd=7 4 2024-04-14T15:21:05.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.263+0000 7fdc3e5a3700 10 client.47501.objecter _send_command 16 2024-04-14T15:21:05.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.263+0000 7fdc3e5a3700 1 -- 172.21.15.90:0/4101025460 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdc24029b70 con 0x7fdc24029710 2024-04-14T15:21:05.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.265+0000 7fdc357fa700 1 -- 172.21.15.90:0/4101025460 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fdc24029b70 con 0x7fdc24029710 2024-04-14T15:21:05.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.265+0000 7fdc357fa700 10 client.47501.objecter ms_dispatch 0x7fdc38001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:21:05.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.265+0000 7fdc357fa700 10 client.47501.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:21:05.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.265+0000 7fdc357fa700 20 client.47501.objecter put_session s=0x7fdc24028ff0 osd=7 4 2024-04-14T15:21:05.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.265+0000 7fdc357fa700 15 client.47501.objecter _session_command_op_remove 7 16 2024-04-14T15:21:05.266 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:21:05.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.266+0000 7fdc3e5a3700 10 librados: watch_flush enter 2024-04-14T15:21:05.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.266+0000 7fdc3e5a3700 10 librados: watch_flush exit 2024-04-14T15:21:05.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.266+0000 7fdc3e5a3700 10 client.47501.objecter close_session for osd.0 2024-04-14T15:21:05.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.266+0000 7fdc3e5a3700 1 -- 172.21.15.90:0/4101025460 >> v1:172.21.15.90:6801/3761806885 conn(0x7fdc24004730 legacy=0x7fdc24002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:05.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.266+0000 7fdc3e5a3700 20 client.47501.objecter put_session s=0x7fdc240053e0 osd=0 1 2024-04-14T15:21:05.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.266+0000 7fdc3e5a3700 10 client.47501.objecter close_session for osd.1 2024-04-14T15:21:05.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.266+0000 7fdc3e5a3700 1 -- 172.21.15.90:0/4101025460 >> v1:172.21.15.90:6805/4170361699 conn(0x7fdc24004310 legacy=0x7fdc2400ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:05.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.266+0000 7fdc3e5a3700 20 client.47501.objecter put_session s=0x7fdc24004130 osd=1 1 2024-04-14T15:21:05.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.266+0000 7fdc3e5a3700 10 client.47501.objecter close_session for osd.2 2024-04-14T15:21:05.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.266+0000 7fdc3e5a3700 1 -- 172.21.15.90:0/4101025460 >> v1:172.21.15.90:6809/1686184251 conn(0x7fdc240119e0 legacy=0x7fdc24013dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:05.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.266+0000 7fdc3e5a3700 20 client.47501.objecter put_session s=0x7fdc24001980 osd=2 1 2024-04-14T15:21:05.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.266+0000 7fdc3e5a3700 10 client.47501.objecter close_session for osd.3 2024-04-14T15:21:05.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.266+0000 7fdc3e5a3700 1 -- 172.21.15.90:0/4101025460 >> v1:172.21.15.90:6813/2270166774 conn(0x7fdc240156a0 legacy=0x7fdc24015ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:05.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.266+0000 7fdc3e5a3700 20 client.47501.objecter put_session s=0x7fdc2400cc20 osd=3 1 2024-04-14T15:21:05.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.266+0000 7fdc3cb40700 1 -- 172.21.15.90:0/4101025460 reap_dead start 2024-04-14T15:21:05.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.266+0000 7fdc3e5a3700 10 client.47501.objecter close_session for osd.4 2024-04-14T15:21:05.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.266+0000 7fdc3e5a3700 1 -- 172.21.15.90:0/4101025460 >> v1:172.21.15.92:6800/3887549733 conn(0x7fdc2401b0b0 legacy=0x7fdc2401d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:05.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.266+0000 7fdc3cb40700 1 -- 172.21.15.90:0/4101025460 reap_dead start 2024-04-14T15:21:05.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.266+0000 7fdc3e5a3700 20 client.47501.objecter put_session s=0x7fdc240161d0 osd=4 1 2024-04-14T15:21:05.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.266+0000 7fdc3e5a3700 10 client.47501.objecter close_session for osd.5 2024-04-14T15:21:05.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.266+0000 7fdc3e5a3700 1 -- 172.21.15.90:0/4101025460 >> v1:172.21.15.92:6804/3244561440 conn(0x7fdc24017670 legacy=0x7fdc24021d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:05.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.266+0000 7fdc3e5a3700 20 client.47501.objecter put_session s=0x7fdc24000930 osd=5 1 2024-04-14T15:21:05.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.266+0000 7fdc3e5a3700 10 client.47501.objecter close_session for osd.6 2024-04-14T15:21:05.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.266+0000 7fdc3e5a3700 1 -- 172.21.15.90:0/4101025460 >> v1:172.21.15.92:6808/636125963 conn(0x7fdc24023980 legacy=0x7fdc24025d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:05.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.266+0000 7fdc3e5a3700 20 client.47501.objecter put_session s=0x7fdc24017e10 osd=6 1 2024-04-14T15:21:05.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.266+0000 7fdc3e5a3700 10 client.47501.objecter close_session for osd.7 2024-04-14T15:21:05.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.266+0000 7fdc3e5a3700 1 -- 172.21.15.90:0/4101025460 >> v1:172.21.15.92:6812/1773706716 conn(0x7fdc24029710 legacy=0x7fdc24027820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:05.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.266+0000 7fdc3e5a3700 20 client.47501.objecter put_session s=0x7fdc24028ff0 osd=7 1 2024-04-14T15:21:05.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.266+0000 7fdc3e5a3700 20 client.47501.objecter shutdown clearing up homeless session... 2024-04-14T15:21:05.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.266+0000 7fdc3e5a3700 10 client.47501.objecter successfully canceled tick 2024-04-14T15:21:05.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.266+0000 7fdc3e5a3700 1 -- 172.21.15.90:0/4101025460 >> v1:172.21.15.90:6800/420935427 conn(0x7fdc200714a0 legacy=0x7fdc20073960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:05.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.266+0000 7fdc3e5a3700 1 -- 172.21.15.90:0/4101025460 >> v1:172.21.15.90:6790/0 conn(0x7fdc3810a6d0 legacy=0x7fdc381a26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:05.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.266+0000 7fdc3cb40700 1 -- 172.21.15.90:0/4101025460 reap_dead start 2024-04-14T15:21:05.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.266+0000 7fdc3e5a3700 1 -- 172.21.15.90:0/4101025460 shutdown_connections 2024-04-14T15:21:05.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.266+0000 7fdc3e5a3700 1 -- 172.21.15.90:0/4101025460 >> 172.21.15.90:0/4101025460 conn(0x7fdc38076aa0 msgr2=0x7fdc38100070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:21:05.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.267+0000 7fdc3e5a3700 1 -- 172.21.15.90:0/4101025460 shutdown_connections 2024-04-14T15:21:05.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.267+0000 7fdc3e5a3700 1 -- 172.21.15.90:0/4101025460 wait complete. 2024-04-14T15:21:05.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.267+0000 7fdc3e5a3700 1 librados: shutdown 2024-04-14T15:21:05.270 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:21:05.270 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:21:05.270 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:21:05.270 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:21:05.270 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:21:05.270 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:21:05.270 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:21:05.270 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:21:05.283 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:05.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:05.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:21:05.385 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:21:05.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.330+0000 7f6c3bb55700 1 -- 172.21.15.90:0/3357889742 >> v1:172.21.15.90:6789/0 conn(0x7f6c3410e310 legacy=0x7f6c341107d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:05.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.330+0000 7f6c3bb55700 1 -- 172.21.15.90:0/3357889742 shutdown_connections 2024-04-14T15:21:05.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.330+0000 7f6c3bb55700 1 -- 172.21.15.90:0/3357889742 >> 172.21.15.90:0/3357889742 conn(0x7f6c340fd290 msgr2=0x7f6c340ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:21:05.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.331+0000 7f6c3bb55700 1 -- 172.21.15.90:0/3357889742 shutdown_connections 2024-04-14T15:21:05.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.331+0000 7f6c3bb55700 1 -- 172.21.15.90:0/3357889742 wait complete. 2024-04-14T15:21:05.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.331+0000 7f6c3bb55700 1 librados: starting msgr at 2024-04-14T15:21:05.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.331+0000 7f6c3bb55700 1 librados: starting objecter 2024-04-14T15:21:05.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.331+0000 7f6c3bb55700 1 Processor -- start 2024-04-14T15:21:05.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.331+0000 7f6c3bb55700 1 -- start start 2024-04-14T15:21:05.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.331+0000 7f6c3bb55700 1 librados: setting wanted keys 2024-04-14T15:21:05.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.331+0000 7f6c3bb55700 1 librados: calling monclient init 2024-04-14T15:21:05.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.331+0000 7f6c3bb55700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6c3410df50 con 0x7f6c3410e310 2024-04-14T15:21:05.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.331+0000 7f6c3bb55700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6c341a8f30 con 0x7f6c3410a740 2024-04-14T15:21:05.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.331+0000 7f6c3bb55700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6c341aa110 con 0x7f6c34107890 2024-04-14T15:21:05.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.331+0000 7f6c398f1700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f6c34107890 0x7f6c3410bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:46964/0 (socket says 172.21.15.90:46964) 2024-04-14T15:21:05.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.331+0000 7f6c398f1700 1 -- 172.21.15.90:0/2956839545 learned_addr learned my addr 172.21.15.90:0/2956839545 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:21:05.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.331+0000 7f6c3a0f2700 1 --1- 172.21.15.90:0/2956839545 >> v1:172.21.15.90:6789/0 conn(0x7f6c3410e310 0x7f6c341a6800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:60028/0 (socket says 172.21.15.90:60028) 2024-04-14T15:21:05.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.332+0000 7f6c32ffd700 10 client.?.objecter ms_handle_connect 0x7f6c34107890 2024-04-14T15:21:05.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.332+0000 7f6c32ffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:21:05.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.332+0000 7f6c32ffd700 10 client.?.objecter ms_handle_connect 0x7f6c3410e310 2024-04-14T15:21:05.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.332+0000 7f6c32ffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:21:05.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.332+0000 7f6c32ffd700 10 client.?.objecter ms_handle_connect 0x7f6c3410a740 2024-04-14T15:21:05.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.332+0000 7f6c32ffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:21:05.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.332+0000 7f6c32ffd700 1 -- 172.21.15.90:0/2956839545 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2425172655 0 0) 0x7f6c341a8f30 con 0x7f6c3410a740 2024-04-14T15:21:05.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.332+0000 7f6c32ffd700 1 -- 172.21.15.90:0/2956839545 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c1c0031f0 con 0x7f6c3410a740 2024-04-14T15:21:05.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.332+0000 7f6c32ffd700 1 -- 172.21.15.90:0/2956839545 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 232822093 0 0) 0x7f6c341aa110 con 0x7f6c34107890 2024-04-14T15:21:05.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.332+0000 7f6c32ffd700 1 -- 172.21.15.90:0/2956839545 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c341a8f30 con 0x7f6c34107890 2024-04-14T15:21:05.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.332+0000 7f6c32ffd700 1 -- 172.21.15.90:0/2956839545 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2604716529 0 0) 0x7f6c3410df50 con 0x7f6c3410e310 2024-04-14T15:21:05.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.332+0000 7f6c32ffd700 1 -- 172.21.15.90:0/2956839545 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c341aa110 con 0x7f6c3410e310 2024-04-14T15:21:05.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.332+0000 7f6c32ffd700 1 -- 172.21.15.90:0/2956839545 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2121172503 0 0) 0x7f6c1c0031f0 con 0x7f6c3410a740 2024-04-14T15:21:05.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.332+0000 7f6c32ffd700 1 -- 172.21.15.90:0/2956839545 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6c3410df50 con 0x7f6c3410a740 2024-04-14T15:21:05.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.332+0000 7f6c32ffd700 1 -- 172.21.15.90:0/2956839545 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f6c280014c0 con 0x7f6c3410a740 2024-04-14T15:21:05.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.332+0000 7f6c32ffd700 1 -- 172.21.15.90:0/2956839545 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1150280659 0 0) 0x7f6c341aa110 con 0x7f6c3410e310 2024-04-14T15:21:05.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.332+0000 7f6c32ffd700 1 -- 172.21.15.90:0/2956839545 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6c1c0031f0 con 0x7f6c3410e310 2024-04-14T15:21:05.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.332+0000 7f6c32ffd700 1 -- 172.21.15.90:0/2956839545 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 473269092 0 0) 0x7f6c341a8f30 con 0x7f6c34107890 2024-04-14T15:21:05.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.332+0000 7f6c32ffd700 1 -- 172.21.15.90:0/2956839545 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6c341aa110 con 0x7f6c34107890 2024-04-14T15:21:05.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.332+0000 7f6c32ffd700 1 -- 172.21.15.90:0/2956839545 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f6c240043b0 con 0x7f6c3410e310 2024-04-14T15:21:05.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.332+0000 7f6c32ffd700 1 -- 172.21.15.90:0/2956839545 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f6c2c001930 con 0x7f6c34107890 2024-04-14T15:21:05.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.332+0000 7f6c32ffd700 1 -- 172.21.15.90:0/2956839545 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1870551599 0 0) 0x7f6c3410df50 con 0x7f6c3410a740 2024-04-14T15:21:05.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.332+0000 7f6c32ffd700 1 -- 172.21.15.90:0/2956839545 >> v1:172.21.15.90:6790/0 conn(0x7f6c34107890 legacy=0x7f6c3410bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:05.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.332+0000 7f6c32ffd700 1 -- 172.21.15.90:0/2956839545 >> v1:172.21.15.90:6789/0 conn(0x7f6c3410e310 legacy=0x7f6c341a6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:05.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.332+0000 7f6c32ffd700 1 -- 172.21.15.90:0/2956839545 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c341ab2f0 con 0x7f6c3410a740 2024-04-14T15:21:05.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.332+0000 7f6c3bb55700 1 -- 172.21.15.90:0/2956839545 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6c341a7f80 con 0x7f6c3410a740 2024-04-14T15:21:05.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.332+0000 7f6c3bb55700 10 client.49387.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:21:05.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.332+0000 7f6c3bb55700 1 -- 172.21.15.90:0/2956839545 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f6c341a8440 con 0x7f6c3410a740 2024-04-14T15:21:05.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.332+0000 7f6c3bb55700 1 librados: init done 2024-04-14T15:21:05.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.332+0000 7f6c32ffd700 1 -- 172.21.15.90:0/2956839545 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f6c28003ea0 con 0x7f6c3410a740 2024-04-14T15:21:05.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.332+0000 7f6c32ffd700 1 -- 172.21.15.90:0/2956839545 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f6c28004d50 con 0x7f6c3410a740 2024-04-14T15:21:05.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.333+0000 7f6c3bb55700 1 -- 172.21.15.90:0/2956839545 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c20005120 con 0x7f6c3410a740 2024-04-14T15:21:05.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.334+0000 7f6c32ffd700 1 -- 172.21.15.90:0/2956839545 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f6c28003a00 con 0x7f6c3410a740 2024-04-14T15:21:05.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.334+0000 7f6c32ffd700 1 -- 172.21.15.90:0/2956839545 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(540..540 src has 467..540) ==== 8230+0+0 (unknown 1164497643 0 0) 0x7f6c28089ca0 con 0x7f6c3410a740 2024-04-14T15:21:05.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.334+0000 7f6c32ffd700 10 client.49387.objecter ms_dispatch 0x7f6c34001550 osd_map(540..540 src has 467..540) 2024-04-14T15:21:05.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.334+0000 7f6c32ffd700 3 client.49387.objecter handle_osd_map got epochs [540,540] > 0 2024-04-14T15:21:05.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.334+0000 7f6c32ffd700 3 client.49387.objecter handle_osd_map decoding full epoch 540 2024-04-14T15:21:05.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.334+0000 7f6c32ffd700 20 client.49387.objecter dump_active .. 0 homeless 2024-04-14T15:21:05.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.334+0000 7f6c32ffd700 10 client.49387.objecter ms_handle_connect 0x7f6c1c0713c0 2024-04-14T15:21:05.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.339+0000 7f6c32ffd700 1 -- 172.21.15.90:0/2956839545 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f6c28056670 con 0x7f6c3410a740 2024-04-14T15:21:05.408 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:05.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:05.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:21:05.510 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:21:05.535 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:05.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:05.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:21:05.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.551+0000 7f6c3bb55700 1 -- 172.21.15.90:0/2956839545 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) -- 0x7f6c20005410 con 0x7f6c3410a740 2024-04-14T15:21:05.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.554+0000 7f6c32ffd700 1 -- 172.21.15.90:0/2956839545 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v540) ==== 81+0+53399 (unknown 4289514357 0 46882378) 0x7f6c2805a320 con 0x7f6c3410a740 2024-04-14T15:21:05.555 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:21:05.555 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-14T15:21:05.556 INFO:teuthology.orchestra.run.smithi090.stdout: "epoch": 540, 2024-04-14T15:21:05.556 INFO:teuthology.orchestra.run.smithi090.stdout: "fsid": "0e288098-fa70-11ee-bc8e-c7b262605968", 2024-04-14T15:21:05.556 INFO:teuthology.orchestra.run.smithi090.stdout: "created": "2024-04-14T15:04:05.808313+0000", 2024-04-14T15:21:05.556 INFO:teuthology.orchestra.run.smithi090.stdout: "modified": "2024-04-14T15:21:04.955113+0000", 2024-04-14T15:21:05.556 INFO:teuthology.orchestra.run.smithi090.stdout: "last_up_change": "2024-04-14T15:09:29.769352+0000", 2024-04-14T15:21:05.556 INFO:teuthology.orchestra.run.smithi090.stdout: "last_in_change": "2024-04-14T15:20:51.950363+0000", 2024-04-14T15:21:05.556 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-14T15:21:05.556 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_num": 5806080, 2024-04-14T15:21:05.556 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_set": [ 2024-04-14T15:21:05.556 INFO:teuthology.orchestra.run.smithi090.stdout: "nodeep-scrub", 2024-04-14T15:21:05.556 INFO:teuthology.orchestra.run.smithi090.stdout: "noscrub", 2024-04-14T15:21:05.556 INFO:teuthology.orchestra.run.smithi090.stdout: "pglog_hardlimit", 2024-04-14T15:21:05.556 INFO:teuthology.orchestra.run.smithi090.stdout: "purged_snapdirs", 2024-04-14T15:21:05.556 INFO:teuthology.orchestra.run.smithi090.stdout: "recovery_deletes", 2024-04-14T15:21:05.557 INFO:teuthology.orchestra.run.smithi090.stdout: "sortbitwise" 2024-04-14T15:21:05.557 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-14T15:21:05.557 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_version": 113, 2024-04-14T15:21:05.557 INFO:teuthology.orchestra.run.smithi090.stdout: "full_ratio": 0.94999998807907104, 2024-04-14T15:21:05.557 INFO:teuthology.orchestra.run.smithi090.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-14T15:21:05.557 INFO:teuthology.orchestra.run.smithi090.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-14T15:21:05.557 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_snapshot": "", 2024-04-14T15:21:05.557 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_max": 311, 2024-04-14T15:21:05.557 INFO:teuthology.orchestra.run.smithi090.stdout: "max_osd": 8, 2024-04-14T15:21:05.557 INFO:teuthology.orchestra.run.smithi090.stdout: "require_min_compat_client": "luminous", 2024-04-14T15:21:05.557 INFO:teuthology.orchestra.run.smithi090.stdout: "min_compat_client": "luminous", 2024-04-14T15:21:05.557 INFO:teuthology.orchestra.run.smithi090.stdout: "require_osd_release": "squid", 2024-04-14T15:21:05.557 INFO:teuthology.orchestra.run.smithi090.stdout: "allow_crimson": false, 2024-04-14T15:21:05.557 INFO:teuthology.orchestra.run.smithi090.stdout: "pools": [ 2024-04-14T15:21:05.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.558 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 1, 2024-04-14T15:21:05.558 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": ".mgr", 2024-04-14T15:21:05.558 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:07:23.959628+0000", 2024-04-14T15:21:05.558 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-14T15:21:05.558 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool", 2024-04-14T15:21:05.558 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:21:05.558 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:21:05.558 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:21:05.558 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:21:05.558 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:21:05.558 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:21:05.558 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:21:05.559 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:21:05.559 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:21:05.559 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:21:05.559 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 26, 2024-04-14T15:21:05.559 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 21, 2024-04-14T15:21:05.559 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 21, 2024-04-14T15:21:05.559 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 21, 2024-04-14T15:21:05.559 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 26, 2024-04-14T15:21:05.559 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:21:05.559 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "1.1a", 2024-04-14T15:21:05.559 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 539, 2024-04-14T15:21:05.559 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 538, 2024-04-14T15:21:05.559 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 538, 2024-04-14T15:21:05.559 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "91'130", 2024-04-14T15:21:05.559 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "91'130" 2024-04-14T15:21:05.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.560 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "539", 2024-04-14T15:21:05.560 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:21:05.560 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "539", 2024-04-14T15:21:05.560 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "537", 2024-04-14T15:21:05.560 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:21:05.560 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:21:05.560 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:21:05.560 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:21:05.560 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:21:05.560 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:21:05.560 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:21:05.560 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:21:05.560 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:21:05.561 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:21:05.561 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:21:05.561 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:21:05.561 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:21:05.561 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:21:05.561 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:21:05.561 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:21:05.561 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:21:05.561 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:21:05.561 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:21:05.561 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:21:05.561 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:21:05.561 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:21:05.561 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:21:05.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.562 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:21:05.562 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:21:05.562 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:21:05.562 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:21:05.562 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:21:05.562 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:21:05.562 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:21:05.562 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:21:05.562 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:21:05.562 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:21:05.562 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:21:05.562 INFO:teuthology.orchestra.run.smithi090.stdout: "options": { 2024-04-14T15:21:05.562 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_max": 32, 2024-04-14T15:21:05.562 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_min": 1 2024-04-14T15:21:05.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.563 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:21:05.563 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": {} 2024-04-14T15:21:05.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.563 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:21:05.563 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:21:05.563 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 1.6200000047683716, 2024-04-14T15:21:05.563 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 1.6200000047683716, 2024-04-14T15:21:05.563 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:21:05.563 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 1.6200000047683716, 2024-04-14T15:21:05.563 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 1.6200000047683716, 2024-04-14T15:21:05.563 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.875, 2024-04-14T15:21:05.563 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 1, 2024-04-14T15:21:05.563 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-04-14T15:21:05.564 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.564 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 2, 2024-04-14T15:21:05.564 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": ".rgw.root", 2024-04-14T15:21:05.564 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:09:41.608420+0000", 2024-04-14T15:21:05.564 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-14T15:21:05.564 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool", 2024-04-14T15:21:05.564 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:21:05.564 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:21:05.564 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:21:05.564 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:21:05.564 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:21:05.564 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:21:05.564 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:21:05.565 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:21:05.565 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:21:05.565 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:21:05.565 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:21:05.565 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:21:05.565 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:21:05.565 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:21:05.565 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:21:05.565 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:21:05.565 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:21:05.565 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:21:05.565 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:21:05.565 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:21:05.565 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:21:05.565 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:21:05.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.566 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "66", 2024-04-14T15:21:05.566 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:21:05.566 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:21:05.566 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:21:05.566 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:21:05.566 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:21:05.566 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:21:05.566 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:21:05.566 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:21:05.566 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:21:05.566 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:21:05.566 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:21:05.566 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:21:05.567 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:21:05.567 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:21:05.567 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:21:05.567 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:21:05.567 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:21:05.567 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:21:05.567 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:21:05.567 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:21:05.567 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:21:05.567 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:21:05.567 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:21:05.567 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:21:05.567 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:21:05.567 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:21:05.567 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.568 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:21:05.568 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:21:05.568 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:21:05.568 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:21:05.568 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:21:05.568 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:21:05.568 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:21:05.568 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:21:05.568 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:21:05.568 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:21:05.568 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:21:05.568 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:21:05.568 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:21:05.568 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw": {} 2024-04-14T15:21:05.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.569 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:21:05.569 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:21:05.569 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 1.75, 2024-04-14T15:21:05.569 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 1.75, 2024-04-14T15:21:05.569 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:21:05.569 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 1.75, 2024-04-14T15:21:05.569 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 1.75, 2024-04-14T15:21:05.569 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.875, 2024-04-14T15:21:05.569 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 1, 2024-04-14T15:21:05.569 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-04-14T15:21:05.569 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.569 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 3, 2024-04-14T15:21:05.569 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "datapool", 2024-04-14T15:21:05.570 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:09:41.744391+0000", 2024-04-14T15:21:05.570 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 8193, 2024-04-14T15:21:05.570 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-14T15:21:05.570 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:21:05.570 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:21:05.570 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:21:05.570 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:21:05.570 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:21:05.570 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:21:05.570 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:21:05.570 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:21:05.570 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:21:05.570 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:21:05.570 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 3, 2024-04-14T15:21:05.571 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 3, 2024-04-14T15:21:05.571 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 3, 2024-04-14T15:21:05.571 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 3, 2024-04-14T15:21:05.571 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 3, 2024-04-14T15:21:05.571 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:21:05.571 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:21:05.571 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:21:05.571 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:21:05.571 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:21:05.571 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:21:05.572 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:21:05.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.572 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "69", 2024-04-14T15:21:05.572 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:21:05.572 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:21:05.572 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:21:05.572 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:21:05.572 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:21:05.572 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 2, 2024-04-14T15:21:05.573 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 69, 2024-04-14T15:21:05.573 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:21:05.573 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:21:05.573 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:21:05.573 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:21:05.573 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:21:05.573 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:21:05.573 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:21:05.573 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:21:05.573 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:21:05.573 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:21:05.573 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:21:05.573 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:21:05.573 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:21:05.574 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:21:05.574 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:21:05.574 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:21:05.574 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:21:05.574 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:21:05.574 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:21:05.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.574 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:21:05.574 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:21:05.574 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:21:05.574 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:21:05.574 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:21:05.574 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:21:05.574 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:21:05.575 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:21:05.575 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:21:05.575 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:21:05.575 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:21:05.575 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:21:05.575 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:21:05.575 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd": {} 2024-04-14T15:21:05.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.575 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:21:05.575 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:21:05.575 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 2.3299999237060547, 2024-04-14T15:21:05.575 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 2.3299999237060547, 2024-04-14T15:21:05.575 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 0.86000001430511475, 2024-04-14T15:21:05.575 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 2, 2024-04-14T15:21:05.575 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 2, 2024-04-14T15:21:05.576 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.875, 2024-04-14T15:21:05.576 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 1, 2024-04-14T15:21:05.576 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-04-14T15:21:05.576 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.576 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 4, 2024-04-14T15:21:05.576 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "default.rgw.log", 2024-04-14T15:21:05.576 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:09:43.088157+0000", 2024-04-14T15:21:05.576 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-14T15:21:05.576 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool", 2024-04-14T15:21:05.576 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:21:05.576 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:21:05.576 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:21:05.577 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:21:05.577 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:21:05.577 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:21:05.577 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:21:05.577 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:21:05.577 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:21:05.577 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:21:05.577 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:21:05.577 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:21:05.577 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:21:05.577 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:21:05.577 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:21:05.577 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:21:05.577 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:21:05.577 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:21:05.578 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:21:05.578 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:21:05.578 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:21:05.578 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:21:05.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.578 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "68", 2024-04-14T15:21:05.578 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:21:05.578 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:21:05.578 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:21:05.578 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:21:05.578 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:21:05.578 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:21:05.578 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:21:05.578 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:21:05.579 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:21:05.579 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:21:05.579 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:21:05.579 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:21:05.579 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:21:05.579 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:21:05.579 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:21:05.579 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:21:05.579 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:21:05.579 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:21:05.579 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:21:05.579 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:21:05.579 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:21:05.579 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:21:05.579 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:21:05.580 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:21:05.580 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:21:05.580 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:21:05.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.580 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:21:05.580 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:21:05.580 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:21:05.580 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:21:05.580 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:21:05.580 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:21:05.580 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:21:05.580 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:21:05.580 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:21:05.580 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:21:05.580 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:21:05.580 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:21:05.581 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:21:05.581 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw": {} 2024-04-14T15:21:05.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.581 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:21:05.581 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:21:05.581 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 2.190000057220459, 2024-04-14T15:21:05.581 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 2.190000057220459, 2024-04-14T15:21:05.581 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:21:05.581 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 2.190000057220459, 2024-04-14T15:21:05.581 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 2.190000057220459, 2024-04-14T15:21:05.581 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.875, 2024-04-14T15:21:05.581 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 1, 2024-04-14T15:21:05.581 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-04-14T15:21:05.581 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.582 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 5, 2024-04-14T15:21:05.582 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "default.rgw.control", 2024-04-14T15:21:05.582 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:09:45.042722+0000", 2024-04-14T15:21:05.582 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-14T15:21:05.582 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool", 2024-04-14T15:21:05.582 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:21:05.582 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:21:05.582 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:21:05.582 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:21:05.582 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:21:05.582 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:21:05.582 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:21:05.583 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:21:05.583 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:21:05.583 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:21:05.583 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:21:05.583 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:21:05.583 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:21:05.583 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:21:05.583 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:21:05.583 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:21:05.583 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:21:05.583 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:21:05.583 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:21:05.583 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:21:05.583 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:21:05.584 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:21:05.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.584 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "70", 2024-04-14T15:21:05.584 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:21:05.584 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:21:05.584 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:21:05.584 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:21:05.584 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:21:05.584 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:21:05.584 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:21:05.584 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:21:05.584 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:21:05.584 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:21:05.584 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:21:05.584 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:21:05.584 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:21:05.585 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:21:05.585 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:21:05.585 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:21:05.585 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:21:05.585 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:21:05.585 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:21:05.585 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:21:05.585 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:21:05.585 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:21:05.585 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:21:05.585 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:21:05.585 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:21:05.585 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:21:05.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.586 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:21:05.586 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:21:05.586 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:21:05.586 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:21:05.586 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:21:05.586 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:21:05.586 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:21:05.586 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:21:05.586 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:21:05.586 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:21:05.586 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:21:05.586 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:21:05.586 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:21:05.586 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw": {} 2024-04-14T15:21:05.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.587 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:21:05.587 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:21:05.587 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 1.5299999713897705, 2024-04-14T15:21:05.587 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 1.5299999713897705, 2024-04-14T15:21:05.587 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:21:05.587 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 1.5299999713897705, 2024-04-14T15:21:05.587 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 1.5299999713897705, 2024-04-14T15:21:05.587 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.875, 2024-04-14T15:21:05.587 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 1, 2024-04-14T15:21:05.587 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-04-14T15:21:05.587 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.587 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 6, 2024-04-14T15:21:05.588 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "default.rgw.meta", 2024-04-14T15:21:05.588 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:09:47.111336+0000", 2024-04-14T15:21:05.588 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-14T15:21:05.588 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool", 2024-04-14T15:21:05.588 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:21:05.588 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:21:05.588 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:21:05.588 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:21:05.588 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:21:05.588 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:21:05.588 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:21:05.588 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:21:05.588 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:21:05.588 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:21:05.588 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:21:05.589 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:21:05.589 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:21:05.589 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:21:05.589 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:21:05.589 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:21:05.589 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:21:05.589 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:21:05.589 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:21:05.589 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:21:05.589 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:21:05.589 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:21:05.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.589 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "72", 2024-04-14T15:21:05.589 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:21:05.590 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:21:05.590 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:21:05.590 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:21:05.590 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:21:05.590 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:21:05.590 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:21:05.590 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:21:05.590 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:21:05.590 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:21:05.590 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:21:05.590 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:21:05.590 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:21:05.590 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:21:05.590 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:21:05.591 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:21:05.591 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:21:05.591 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:21:05.591 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:21:05.591 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:21:05.591 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:21:05.591 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:21:05.591 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:21:05.591 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:21:05.591 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:21:05.591 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:21:05.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.591 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:21:05.592 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:21:05.592 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:21:05.592 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:21:05.592 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:21:05.592 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:21:05.592 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:21:05.592 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:21:05.592 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:21:05.592 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:21:05.592 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:21:05.592 INFO:teuthology.orchestra.run.smithi090.stdout: "options": { 2024-04-14T15:21:05.592 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_bias": 4 2024-04-14T15:21:05.592 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.592 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:21:05.592 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw": {} 2024-04-14T15:21:05.593 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.593 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:21:05.593 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:21:05.593 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 1.75, 2024-04-14T15:21:05.593 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 1.75, 2024-04-14T15:21:05.593 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:21:05.593 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 1.75, 2024-04-14T15:21:05.593 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 1.75, 2024-04-14T15:21:05.593 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.875, 2024-04-14T15:21:05.593 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 1, 2024-04-14T15:21:05.593 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-04-14T15:21:05.593 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.593 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.593 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.593 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 11, 2024-04-14T15:21:05.593 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "ceph_test_rados_api_asio", 2024-04-14T15:21:05.594 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:12:54.225671+0000", 2024-04-14T15:21:05.594 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-14T15:21:05.594 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool", 2024-04-14T15:21:05.594 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:21:05.594 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:21:05.594 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:21:05.594 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:21:05.594 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:21:05.594 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:21:05.594 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:21:05.594 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:21:05.594 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:21:05.594 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:21:05.594 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:21:05.595 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:21:05.595 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:21:05.595 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:21:05.595 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:21:05.595 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:21:05.595 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:21:05.595 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:21:05.595 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:21:05.595 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:21:05.595 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:21:05.595 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:21:05.595 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.595 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "109", 2024-04-14T15:21:05.595 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:21:05.596 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:21:05.596 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:21:05.596 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:21:05.596 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:21:05.596 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:21:05.596 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:21:05.596 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:21:05.596 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:21:05.596 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:21:05.596 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:21:05.596 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:21:05.596 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:21:05.596 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:21:05.596 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:21:05.596 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:21:05.596 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:21:05.597 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:21:05.597 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:21:05.597 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:21:05.597 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:21:05.597 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:21:05.597 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:21:05.597 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:21:05.597 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:21:05.597 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:21:05.597 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.597 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:21:05.597 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:21:05.597 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:21:05.597 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:21:05.598 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:21:05.598 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:21:05.598 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:21:05.598 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:21:05.598 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:21:05.598 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:21:05.598 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:21:05.598 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:21:05.598 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": {}, 2024-04-14T15:21:05.598 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:21:05.598 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:21:05.598 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 2.190000057220459, 2024-04-14T15:21:05.598 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 2.190000057220459, 2024-04-14T15:21:05.598 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:21:05.599 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 2.190000057220459, 2024-04-14T15:21:05.599 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 2.190000057220459, 2024-04-14T15:21:05.599 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.875, 2024-04-14T15:21:05.599 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 1, 2024-04-14T15:21:05.599 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-04-14T15:21:05.599 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.599 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.599 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.599 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 110, 2024-04-14T15:21:05.599 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "test-rados-api-smithi090-93652-6", 2024-04-14T15:21:05.599 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:13:07.960794+0000", 2024-04-14T15:21:05.599 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 8193, 2024-04-14T15:21:05.599 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-14T15:21:05.599 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:21:05.599 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:21:05.600 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:21:05.600 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:21:05.600 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:21:05.600 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:21:05.600 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:21:05.600 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:21:05.600 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:21:05.600 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:21:05.600 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:21:05.600 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:21:05.600 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:21:05.600 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:21:05.600 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:21:05.600 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:21:05.600 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:21:05.601 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:21:05.601 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:21:05.601 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:21:05.601 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:21:05.601 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:21:05.601 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.601 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "489", 2024-04-14T15:21:05.601 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "399", 2024-04-14T15:21:05.601 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "399", 2024-04-14T15:21:05.601 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "399", 2024-04-14T15:21:05.601 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:21:05.601 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:21:05.601 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 26, 2024-04-14T15:21:05.601 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 489, 2024-04-14T15:21:05.601 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:21:05.602 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:21:05.602 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:21:05.602 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:21:05.602 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:21:05.602 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:21:05.602 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:21:05.602 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:21:05.602 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:21:05.602 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:21:05.602 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:21:05.602 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:21:05.602 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:21:05.602 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:21:05.602 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:21:05.603 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:21:05.603 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:21:05.603 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:21:05.603 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:21:05.603 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.603 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:21:05.603 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:21:05.603 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:21:05.603 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:21:05.603 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:21:05.603 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:21:05.603 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:21:05.603 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:21:05.603 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:21:05.604 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:21:05.604 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:21:05.604 INFO:teuthology.orchestra.run.smithi090.stdout: "options": { 2024-04-14T15:21:05.604 INFO:teuthology.orchestra.run.smithi090.stdout: "fingerprint_algorithm": "sha1" 2024-04-14T15:21:05.604 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.604 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:21:05.604 INFO:teuthology.orchestra.run.smithi090.stdout: "rados": {} 2024-04-14T15:21:05.604 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.604 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:21:05.604 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:21:05.604 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 1.5299999713897705, 2024-04-14T15:21:05.604 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 1.5299999713897705, 2024-04-14T15:21:05.604 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:21:05.604 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 1.5299999713897705, 2024-04-14T15:21:05.604 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 1.5299999713897705, 2024-04-14T15:21:05.604 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.875, 2024-04-14T15:21:05.605 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 1, 2024-04-14T15:21:05.605 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-04-14T15:21:05.605 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.605 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.605 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.605 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 311, 2024-04-14T15:21:05.605 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "test-rados-api-smithi090-93652-90", 2024-04-14T15:21:05.605 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-14T15:21:02.969894+0000", 2024-04-14T15:21:05.605 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 32769, 2024-04-14T15:21:05.605 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,creating", 2024-04-14T15:21:05.605 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-14T15:21:05.605 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2024-04-14T15:21:05.605 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2024-04-14T15:21:05.605 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-14T15:21:05.605 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-14T15:21:05.606 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-14T15:21:05.606 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-14T15:21:05.606 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-14T15:21:05.606 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-14T15:21:05.606 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-14T15:21:05.606 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 32, 2024-04-14T15:21:05.606 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 32, 2024-04-14T15:21:05.606 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 32, 2024-04-14T15:21:05.606 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 32, 2024-04-14T15:21:05.606 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 32, 2024-04-14T15:21:05.606 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-14T15:21:05.606 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-14T15:21:05.606 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-14T15:21:05.606 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-14T15:21:05.607 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-14T15:21:05.607 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-14T15:21:05.607 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-14T15:21:05.607 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.607 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "540", 2024-04-14T15:21:05.607 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-14T15:21:05.607 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-14T15:21:05.607 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-14T15:21:05.607 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-14T15:21:05.607 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-14T15:21:05.607 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-14T15:21:05.607 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-14T15:21:05.607 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-14T15:21:05.607 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-14T15:21:05.607 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-14T15:21:05.608 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-14T15:21:05.608 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-14T15:21:05.608 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-14T15:21:05.608 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-14T15:21:05.608 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-14T15:21:05.608 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-14T15:21:05.608 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-14T15:21:05.608 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-14T15:21:05.608 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-14T15:21:05.608 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-14T15:21:05.608 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-14T15:21:05.608 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-14T15:21:05.608 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-14T15:21:05.608 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-14T15:21:05.609 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-14T15:21:05.609 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-14T15:21:05.609 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.609 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-14T15:21:05.609 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-14T15:21:05.609 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-14T15:21:05.609 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-14T15:21:05.609 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-14T15:21:05.609 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-14T15:21:05.609 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-14T15:21:05.609 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-14T15:21:05.609 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-14T15:21:05.609 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-14T15:21:05.609 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-14T15:21:05.609 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-14T15:21:05.610 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-14T15:21:05.610 INFO:teuthology.orchestra.run.smithi090.stdout: "rados": {} 2024-04-14T15:21:05.610 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.610 INFO:teuthology.orchestra.run.smithi090.stdout: "read_balance": { 2024-04-14T15:21:05.610 INFO:teuthology.orchestra.run.smithi090.stdout: "score_type": "Fair distribution", 2024-04-14T15:21:05.610 INFO:teuthology.orchestra.run.smithi090.stdout: "score_acting": 2.190000057220459, 2024-04-14T15:21:05.610 INFO:teuthology.orchestra.run.smithi090.stdout: "score_stable": 2.190000057220459, 2024-04-14T15:21:05.610 INFO:teuthology.orchestra.run.smithi090.stdout: "optimal_score": 1, 2024-04-14T15:21:05.610 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_acting": 2.190000057220459, 2024-04-14T15:21:05.610 INFO:teuthology.orchestra.run.smithi090.stdout: "raw_score_stable": 2.190000057220459, 2024-04-14T15:21:05.610 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity_weighted": 0.875, 2024-04-14T15:21:05.610 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity": 1, 2024-04-14T15:21:05.610 INFO:teuthology.orchestra.run.smithi090.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-04-14T15:21:05.610 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.611 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.611 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-14T15:21:05.611 INFO:teuthology.orchestra.run.smithi090.stdout: "osds": [ 2024-04-14T15:21:05.611 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.611 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 0, 2024-04-14T15:21:05.611 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "76772cfc-89d1-4013-b617-32284ed8dfd9", 2024-04-14T15:21:05.611 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-14T15:21:05.611 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-14T15:21:05.611 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-14T15:21:05.611 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-14T15:21:05.611 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-14T15:21:05.611 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-14T15:21:05.611 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 8, 2024-04-14T15:21:05.611 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 539, 2024-04-14T15:21:05.611 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-14T15:21:05.612 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-14T15:21:05.612 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-14T15:21:05.612 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:21:05.612 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.612 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:21:05.612 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6801", 2024-04-14T15:21:05.612 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3761806885 2024-04-14T15:21:05.612 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.612 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.612 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.612 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-14T15:21:05.612 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:21:05.612 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.612 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:21:05.613 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6802", 2024-04-14T15:21:05.613 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3761806885 2024-04-14T15:21:05.613 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.613 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.613 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.613 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-14T15:21:05.613 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:21:05.613 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.613 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:21:05.613 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6804", 2024-04-14T15:21:05.613 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3761806885 2024-04-14T15:21:05.613 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.613 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.613 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.614 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-14T15:21:05.614 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:21:05.614 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.614 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:21:05.614 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6803", 2024-04-14T15:21:05.614 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3761806885 2024-04-14T15:21:05.614 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.614 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.614 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.614 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6801/3761806885", 2024-04-14T15:21:05.614 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6802/3761806885", 2024-04-14T15:21:05.614 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6804/3761806885", 2024-04-14T15:21:05.614 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6803/3761806885", 2024-04-14T15:21:05.614 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-14T15:21:05.614 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-14T15:21:05.615 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-14T15:21:05.615 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.615 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.615 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.615 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 1, 2024-04-14T15:21:05.615 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "de4a7fa2-7d57-4ff3-a4fa-28c0cde8ef6c", 2024-04-14T15:21:05.615 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-14T15:21:05.615 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-14T15:21:05.615 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-14T15:21:05.615 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-14T15:21:05.615 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-14T15:21:05.615 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-14T15:21:05.615 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2024-04-14T15:21:05.615 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 539, 2024-04-14T15:21:05.615 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-14T15:21:05.616 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-14T15:21:05.616 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-14T15:21:05.616 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:21:05.616 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.616 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:21:05.616 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6805", 2024-04-14T15:21:05.616 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 4170361699 2024-04-14T15:21:05.616 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.616 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.616 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.616 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-14T15:21:05.616 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:21:05.616 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.616 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:21:05.616 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6806", 2024-04-14T15:21:05.616 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 4170361699 2024-04-14T15:21:05.617 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.617 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.617 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.617 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-14T15:21:05.617 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:21:05.617 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.617 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:21:05.617 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6808", 2024-04-14T15:21:05.617 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 4170361699 2024-04-14T15:21:05.617 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.617 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.617 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.617 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-14T15:21:05.617 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:21:05.617 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.618 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:21:05.618 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6807", 2024-04-14T15:21:05.618 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 4170361699 2024-04-14T15:21:05.618 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.618 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.618 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.618 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6805/4170361699", 2024-04-14T15:21:05.618 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6806/4170361699", 2024-04-14T15:21:05.618 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6808/4170361699", 2024-04-14T15:21:05.618 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6807/4170361699", 2024-04-14T15:21:05.618 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-14T15:21:05.618 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-14T15:21:05.618 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-14T15:21:05.618 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.618 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.619 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.619 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 2, 2024-04-14T15:21:05.619 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "9c39b5ff-6175-46ba-9d71-370581c6ff3d", 2024-04-14T15:21:05.619 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-14T15:21:05.619 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-14T15:21:05.619 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-14T15:21:05.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.556+0000 7f6c3bb55700 10 librados: watch_flush enter 2024-04-14T15:21:05.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.556+0000 7f6c3bb55700 10 librados: watch_flush exit 2024-04-14T15:21:05.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.556+0000 7f6c3bb55700 20 client.49387.objecter shutdown clearing up homeless session... 2024-04-14T15:21:05.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.556+0000 7f6c3bb55700 10 client.49387.objecter successfully canceled tick 2024-04-14T15:21:05.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.557+0000 7f6c3bb55700 1 -- 172.21.15.90:0/2956839545 >> v1:172.21.15.90:6800/420935427 conn(0x7f6c1c0713c0 legacy=0x7f6c1c073880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:05.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.557+0000 7f6c3bb55700 1 -- 172.21.15.90:0/2956839545 >> v1:172.21.15.92:6789/0 conn(0x7f6c3410a740 legacy=0x7f6c3410c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:05.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.557+0000 7f6c3bb55700 1 -- 172.21.15.90:0/2956839545 shutdown_connections 2024-04-14T15:21:05.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.557+0000 7f6c3bb55700 1 -- 172.21.15.90:0/2956839545 >> 172.21.15.90:0/2956839545 conn(0x7f6c340fd290 msgr2=0x7f6c34101780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:21:05.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.557+0000 7f6c3bb55700 1 -- 172.21.15.90:0/2956839545 shutdown_connections 2024-04-14T15:21:05.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.557+0000 7f6c3bb55700 1 -- 172.21.15.90:0/2956839545 wait complete. 2024-04-14T15:21:05.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:05.557+0000 7f6c3bb55700 1 librados: shutdown 2024-04-14T15:21:05.620 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-14T15:21:05.620 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-14T15:21:05.621 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-14T15:21:05.621 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 18, 2024-04-14T15:21:05.621 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 539, 2024-04-14T15:21:05.621 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-14T15:21:05.621 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-14T15:21:05.621 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-14T15:21:05.621 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:21:05.621 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.621 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:21:05.621 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6809", 2024-04-14T15:21:05.621 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 1686184251 2024-04-14T15:21:05.621 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.621 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.621 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.622 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-14T15:21:05.622 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:21:05.622 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.622 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:21:05.622 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6810", 2024-04-14T15:21:05.622 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 1686184251 2024-04-14T15:21:05.622 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.622 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.622 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.622 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-14T15:21:05.622 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:21:05.622 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.622 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:21:05.623 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6812", 2024-04-14T15:21:05.623 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 1686184251 2024-04-14T15:21:05.623 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.623 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.623 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.623 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-14T15:21:05.623 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:21:05.623 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.623 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:21:05.623 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6811", 2024-04-14T15:21:05.623 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 1686184251 2024-04-14T15:21:05.623 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.623 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.623 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.624 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6809/1686184251", 2024-04-14T15:21:05.624 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6810/1686184251", 2024-04-14T15:21:05.624 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6812/1686184251", 2024-04-14T15:21:05.624 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6811/1686184251", 2024-04-14T15:21:05.624 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-14T15:21:05.624 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-14T15:21:05.624 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-14T15:21:05.624 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.624 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.624 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.624 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 3, 2024-04-14T15:21:05.624 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "efd2ca45-5b80-4013-a571-499e2cb5b2e8", 2024-04-14T15:21:05.624 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-14T15:21:05.624 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-14T15:21:05.625 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-14T15:21:05.625 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-14T15:21:05.625 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-14T15:21:05.625 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-14T15:21:05.625 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 25, 2024-04-14T15:21:05.625 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 539, 2024-04-14T15:21:05.625 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-14T15:21:05.625 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-14T15:21:05.625 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-14T15:21:05.625 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:21:05.625 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.625 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:21:05.625 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6813", 2024-04-14T15:21:05.625 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 2270166774 2024-04-14T15:21:05.625 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.626 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.626 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.626 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-14T15:21:05.626 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:21:05.626 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.626 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:21:05.626 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6814", 2024-04-14T15:21:05.626 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 2270166774 2024-04-14T15:21:05.626 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.626 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.626 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.626 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-14T15:21:05.626 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:21:05.626 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.627 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:21:05.627 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6816", 2024-04-14T15:21:05.627 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 2270166774 2024-04-14T15:21:05.627 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.627 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.627 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.627 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-14T15:21:05.627 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:21:05.627 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.627 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:21:05.627 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6815", 2024-04-14T15:21:05.627 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 2270166774 2024-04-14T15:21:05.627 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.627 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.627 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.628 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6813/2270166774", 2024-04-14T15:21:05.628 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6814/2270166774", 2024-04-14T15:21:05.628 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6816/2270166774", 2024-04-14T15:21:05.628 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6815/2270166774", 2024-04-14T15:21:05.628 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-14T15:21:05.628 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-14T15:21:05.628 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-14T15:21:05.628 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.628 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.628 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.628 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 4, 2024-04-14T15:21:05.628 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "e3d9013d-5eb6-433c-96b3-3fc38ba7ac8d", 2024-04-14T15:21:05.628 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-14T15:21:05.628 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-14T15:21:05.629 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-14T15:21:05.629 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-14T15:21:05.629 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-14T15:21:05.629 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-14T15:21:05.629 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 31, 2024-04-14T15:21:05.629 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 539, 2024-04-14T15:21:05.629 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-14T15:21:05.629 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-14T15:21:05.629 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-14T15:21:05.629 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:21:05.629 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.629 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:21:05.629 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6800", 2024-04-14T15:21:05.629 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3887549733 2024-04-14T15:21:05.630 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.630 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.630 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.630 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-14T15:21:05.630 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:21:05.630 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.630 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:21:05.630 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6801", 2024-04-14T15:21:05.630 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3887549733 2024-04-14T15:21:05.630 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.630 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.630 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.630 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-14T15:21:05.630 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:21:05.630 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.631 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:21:05.631 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6803", 2024-04-14T15:21:05.631 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3887549733 2024-04-14T15:21:05.631 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.631 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.631 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.631 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-14T15:21:05.631 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:21:05.631 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.631 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:21:05.631 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6802", 2024-04-14T15:21:05.631 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3887549733 2024-04-14T15:21:05.631 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.631 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.631 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.632 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.92:6800/3887549733", 2024-04-14T15:21:05.632 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.92:6801/3887549733", 2024-04-14T15:21:05.632 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.92:6803/3887549733", 2024-04-14T15:21:05.632 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.92:6802/3887549733", 2024-04-14T15:21:05.632 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-14T15:21:05.632 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-14T15:21:05.632 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-14T15:21:05.632 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.632 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.632 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.632 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 5, 2024-04-14T15:21:05.632 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "1ec41805-6f82-4139-81bc-75305ba9e958", 2024-04-14T15:21:05.632 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-14T15:21:05.632 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-14T15:21:05.633 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-14T15:21:05.633 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 0, 2024-04-14T15:21:05.633 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-14T15:21:05.633 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-14T15:21:05.633 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 41, 2024-04-14T15:21:05.633 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 271, 2024-04-14T15:21:05.633 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-14T15:21:05.633 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-14T15:21:05.633 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-14T15:21:05.633 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:21:05.633 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.633 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:21:05.633 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6804", 2024-04-14T15:21:05.633 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3244561440 2024-04-14T15:21:05.634 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.634 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.634 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.634 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-14T15:21:05.634 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:21:05.634 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.634 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:21:05.634 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6805", 2024-04-14T15:21:05.634 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3244561440 2024-04-14T15:21:05.634 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.634 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.634 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.634 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-14T15:21:05.634 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:21:05.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.635 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:21:05.635 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6807", 2024-04-14T15:21:05.635 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3244561440 2024-04-14T15:21:05.635 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.635 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.635 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.635 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-14T15:21:05.635 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:21:05.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.635 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:21:05.635 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6806", 2024-04-14T15:21:05.635 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3244561440 2024-04-14T15:21:05.635 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.636 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.636 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.92:6804/3244561440", 2024-04-14T15:21:05.636 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.92:6805/3244561440", 2024-04-14T15:21:05.636 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.92:6807/3244561440", 2024-04-14T15:21:05.636 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.92:6806/3244561440", 2024-04-14T15:21:05.636 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-14T15:21:05.636 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-14T15:21:05.636 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-14T15:21:05.636 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.636 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.636 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 6, 2024-04-14T15:21:05.636 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "ca977ab2-bf8d-4b3a-ae5e-0efa5bc7a3ad", 2024-04-14T15:21:05.636 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-14T15:21:05.636 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 0, 2024-04-14T15:21:05.637 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 0, 2024-04-14T15:21:05.637 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-14T15:21:05.637 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-14T15:21:05.637 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-14T15:21:05.637 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 49, 2024-04-14T15:21:05.637 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 521, 2024-04-14T15:21:05.637 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-14T15:21:05.637 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-14T15:21:05.637 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-14T15:21:05.637 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:21:05.637 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.637 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:21:05.637 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6808", 2024-04-14T15:21:05.637 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 636125963 2024-04-14T15:21:05.637 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.638 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.638 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-14T15:21:05.638 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:21:05.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.638 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:21:05.638 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6809", 2024-04-14T15:21:05.638 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 636125963 2024-04-14T15:21:05.638 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.638 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.638 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-14T15:21:05.638 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:21:05.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.638 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:21:05.639 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6811", 2024-04-14T15:21:05.639 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 636125963 2024-04-14T15:21:05.639 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.639 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.639 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.639 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-14T15:21:05.639 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:21:05.639 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.639 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:21:05.639 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6810", 2024-04-14T15:21:05.639 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 636125963 2024-04-14T15:21:05.639 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.639 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.639 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.639 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.92:6808/636125963", 2024-04-14T15:21:05.640 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.92:6809/636125963", 2024-04-14T15:21:05.640 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.92:6811/636125963", 2024-04-14T15:21:05.640 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.92:6810/636125963", 2024-04-14T15:21:05.640 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-14T15:21:05.640 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-14T15:21:05.640 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-14T15:21:05.640 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.640 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.640 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.640 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 7, 2024-04-14T15:21:05.640 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "e5354645-79ec-4d4c-b6e5-f9b0653915ee", 2024-04-14T15:21:05.640 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-14T15:21:05.640 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-14T15:21:05.640 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-14T15:21:05.641 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-14T15:21:05.641 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-14T15:21:05.641 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-14T15:21:05.641 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 58, 2024-04-14T15:21:05.641 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 539, 2024-04-14T15:21:05.641 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-14T15:21:05.641 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-14T15:21:05.641 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-14T15:21:05.641 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:21:05.641 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.641 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:21:05.641 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6812", 2024-04-14T15:21:05.641 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 1773706716 2024-04-14T15:21:05.641 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.641 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.642 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-14T15:21:05.642 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:21:05.642 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.642 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:21:05.642 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6813", 2024-04-14T15:21:05.642 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 1773706716 2024-04-14T15:21:05.642 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.642 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.642 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-14T15:21:05.642 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:21:05.642 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.643 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:21:05.643 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6815", 2024-04-14T15:21:05.643 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 1773706716 2024-04-14T15:21:05.643 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.643 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.643 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.643 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-14T15:21:05.643 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-14T15:21:05.643 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.643 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-14T15:21:05.643 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.92:6814", 2024-04-14T15:21:05.643 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 1773706716 2024-04-14T15:21:05.643 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.643 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.643 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.644 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.92:6812/1773706716", 2024-04-14T15:21:05.644 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.92:6813/1773706716", 2024-04-14T15:21:05.644 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.92:6815/1773706716", 2024-04-14T15:21:05.644 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.92:6814/1773706716", 2024-04-14T15:21:05.644 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-14T15:21:05.644 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-14T15:21:05.644 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-14T15:21:05.644 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.644 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.644 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-14T15:21:05.644 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_xinfo": [ 2024-04-14T15:21:05.644 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.644 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 0, 2024-04-14T15:21:05.644 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-14T15:21:05.644 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-14T15:21:05.645 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-14T15:21:05.645 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540701547738038271, 2024-04-14T15:21:05.645 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-14T15:21:05.645 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-14T15:20:25.995394+0000", 2024-04-14T15:21:05.645 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-14T15:21:05.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.645 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.645 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 1, 2024-04-14T15:21:05.645 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-14T15:21:05.645 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-14T15:21:05.645 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-14T15:21:05.645 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540701547738038271, 2024-04-14T15:21:05.645 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-14T15:21:05.645 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-14T15:20:48.161602+0000", 2024-04-14T15:21:05.645 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-14T15:21:05.646 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.646 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.646 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 2, 2024-04-14T15:21:05.646 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-14T15:21:05.646 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-14T15:21:05.646 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-14T15:21:05.646 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540701547738038271, 2024-04-14T15:21:05.646 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-14T15:21:05.646 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-14T15:20:48.864326+0000", 2024-04-14T15:21:05.646 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-14T15:21:05.646 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.646 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.646 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 3, 2024-04-14T15:21:05.646 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-14T15:21:05.647 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-14T15:21:05.647 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-14T15:21:05.647 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540701547738038271, 2024-04-14T15:21:05.647 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-14T15:21:05.647 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-14T15:20:25.493644+0000", 2024-04-14T15:21:05.647 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-14T15:21:05.647 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.647 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.647 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 4, 2024-04-14T15:21:05.647 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-14T15:21:05.647 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-14T15:21:05.647 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-14T15:21:05.647 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540701547738038271, 2024-04-14T15:21:05.647 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-14T15:21:05.647 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-14T15:20:24.298175+0000", 2024-04-14T15:21:05.648 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-14T15:21:05.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.648 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.648 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 5, 2024-04-14T15:21:05.648 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-14T15:21:05.648 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-14T15:21:05.648 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-14T15:21:05.648 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540701547738038271, 2024-04-14T15:21:05.648 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-14T15:21:05.648 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-14T15:20:12.532132+0000", 2024-04-14T15:21:05.648 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-14T15:21:05.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.648 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.648 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 6, 2024-04-14T15:21:05.649 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-14T15:21:05.649 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-14T15:21:05.649 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-14T15:21:05.649 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540701547738038271, 2024-04-14T15:21:05.649 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 65536, 2024-04-14T15:21:05.649 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-14T15:20:10.183706+0000", 2024-04-14T15:21:05.649 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-14T15:21:05.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.649 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 7, 2024-04-14T15:21:05.649 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-14T15:21:05.649 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-14T15:21:05.649 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-14T15:21:05.649 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540701547738038271, 2024-04-14T15:21:05.650 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-14T15:21:05.650 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-14T15:20:49.257144+0000", 2024-04-14T15:21:05.650 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-14T15:21:05.650 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.650 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-14T15:21:05.650 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_upmap": [], 2024-04-14T15:21:05.650 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_upmap_items": [ 2024-04-14T15:21:05.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.650 INFO:teuthology.orchestra.run.smithi090.stdout: "pgid": "2.1d", 2024-04-14T15:21:05.650 INFO:teuthology.orchestra.run.smithi090.stdout: "mappings": [ 2024-04-14T15:21:05.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.650 INFO:teuthology.orchestra.run.smithi090.stdout: "from": 7, 2024-04-14T15:21:05.650 INFO:teuthology.orchestra.run.smithi090.stdout: "to": 2 2024-04-14T15:21:05.650 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.650 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.650 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.651 INFO:teuthology.orchestra.run.smithi090.stdout: "pgid": "4.2", 2024-04-14T15:21:05.651 INFO:teuthology.orchestra.run.smithi090.stdout: "mappings": [ 2024-04-14T15:21:05.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.651 INFO:teuthology.orchestra.run.smithi090.stdout: "from": 1, 2024-04-14T15:21:05.651 INFO:teuthology.orchestra.run.smithi090.stdout: "to": 2 2024-04-14T15:21:05.651 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.651 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.651 INFO:teuthology.orchestra.run.smithi090.stdout: "pgid": "4.12", 2024-04-14T15:21:05.651 INFO:teuthology.orchestra.run.smithi090.stdout: "mappings": [ 2024-04-14T15:21:05.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.651 INFO:teuthology.orchestra.run.smithi090.stdout: "from": 1, 2024-04-14T15:21:05.652 INFO:teuthology.orchestra.run.smithi090.stdout: "to": 5 2024-04-14T15:21:05.652 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.652 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.652 INFO:teuthology.orchestra.run.smithi090.stdout: "pgid": "4.1f", 2024-04-14T15:21:05.652 INFO:teuthology.orchestra.run.smithi090.stdout: "mappings": [ 2024-04-14T15:21:05.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.652 INFO:teuthology.orchestra.run.smithi090.stdout: "from": 1, 2024-04-14T15:21:05.652 INFO:teuthology.orchestra.run.smithi090.stdout: "to": 2 2024-04-14T15:21:05.652 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.652 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.652 INFO:teuthology.orchestra.run.smithi090.stdout: "pgid": "110.3", 2024-04-14T15:21:05.652 INFO:teuthology.orchestra.run.smithi090.stdout: "mappings": [ 2024-04-14T15:21:05.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.653 INFO:teuthology.orchestra.run.smithi090.stdout: "from": 5, 2024-04-14T15:21:05.653 INFO:teuthology.orchestra.run.smithi090.stdout: "to": 7 2024-04-14T15:21:05.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.653 INFO:teuthology.orchestra.run.smithi090.stdout: "from": 2, 2024-04-14T15:21:05.653 INFO:teuthology.orchestra.run.smithi090.stdout: "to": 1 2024-04-14T15:21:05.653 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.653 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.653 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.653 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-14T15:21:05.653 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_upmap_primaries": [], 2024-04-14T15:21:05.653 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_temp": [ 2024-04-14T15:21:05.654 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.654 INFO:teuthology.orchestra.run.smithi090.stdout: "pgid": "1.c", 2024-04-14T15:21:05.654 INFO:teuthology.orchestra.run.smithi090.stdout: "osds": [ 2024-04-14T15:21:05.654 INFO:teuthology.orchestra.run.smithi090.stdout: 3, 2024-04-14T15:21:05.654 INFO:teuthology.orchestra.run.smithi090.stdout: 0, 2024-04-14T15:21:05.654 INFO:teuthology.orchestra.run.smithi090.stdout: 6 2024-04-14T15:21:05.654 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.654 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.654 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-14T15:21:05.654 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_temp": [], 2024-04-14T15:21:05.654 INFO:teuthology.orchestra.run.smithi090.stdout: "blocklist": { 2024-04-14T15:21:05.654 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:0/508768479": "2024-04-15T15:10:22.918674+0000", 2024-04-14T15:21:05.654 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:0/1823072952": "2024-04-15T15:10:22.918674+0000", 2024-04-14T15:21:05.654 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:6800/491071118": "2024-04-15T15:10:22.918674+0000", 2024-04-14T15:21:05.654 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:0/1337064035": "2024-04-15T15:10:22.918674+0000", 2024-04-14T15:21:05.655 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:0/769462306": "2024-04-15T15:10:22.918674+0000", 2024-04-14T15:21:05.655 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:0/1139868610": "2024-04-15T15:05:05.882361+0000", 2024-04-14T15:21:05.655 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:0/93763408": "2024-04-15T15:10:22.918674+0000", 2024-04-14T15:21:05.655 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:0/977818204": "2024-04-15T15:04:39.251711+0000", 2024-04-14T15:21:05.655 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:6800/941091171": "2024-04-15T15:05:05.882361+0000", 2024-04-14T15:21:05.655 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:0/3671453561": "2024-04-15T15:05:05.882361+0000", 2024-04-14T15:21:05.655 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:0/2611856200": "2024-04-15T15:05:05.882361+0000", 2024-04-14T15:21:05.655 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:6800/2306258962": "2024-04-15T15:04:39.251711+0000", 2024-04-14T15:21:05.655 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:0/4051692380": "2024-04-15T15:04:39.251711+0000", 2024-04-14T15:21:05.655 INFO:teuthology.orchestra.run.smithi090.stdout: "172.21.15.90:0/2681502561": "2024-04-15T15:04:39.251711+0000" 2024-04-14T15:21:05.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.655 INFO:teuthology.orchestra.run.smithi090.stdout: "range_blocklist": {}, 2024-04-14T15:21:05.655 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profiles": { 2024-04-14T15:21:05.655 INFO:teuthology.orchestra.run.smithi090.stdout: "default": { 2024-04-14T15:21:05.656 INFO:teuthology.orchestra.run.smithi090.stdout: "crush-failure-domain": "osd", 2024-04-14T15:21:05.656 INFO:teuthology.orchestra.run.smithi090.stdout: "k": "2", 2024-04-14T15:21:05.656 INFO:teuthology.orchestra.run.smithi090.stdout: "m": "1", 2024-04-14T15:21:05.656 INFO:teuthology.orchestra.run.smithi090.stdout: "plugin": "jerasure", 2024-04-14T15:21:05.656 INFO:teuthology.orchestra.run.smithi090.stdout: "technique": "reed_sol_van" 2024-04-14T15:21:05.656 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.656 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps_queue": [ 2024-04-14T15:21:05.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.656 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 17, 2024-04-14T15:21:05.656 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:21:05.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.656 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 2, 2024-04-14T15:21:05.656 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 5 2024-04-14T15:21:05.656 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.656 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.657 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 18, 2024-04-14T15:21:05.657 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:21:05.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.657 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 2, 2024-04-14T15:21:05.657 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 3 2024-04-14T15:21:05.657 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.657 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.657 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 39, 2024-04-14T15:21:05.657 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:21:05.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.657 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 1, 2024-04-14T15:21:05.658 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2024-04-14T15:21:05.658 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.658 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.658 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.658 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.658 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 72, 2024-04-14T15:21:05.658 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:21:05.658 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.658 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 1, 2024-04-14T15:21:05.658 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2024-04-14T15:21:05.658 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.658 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.658 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.658 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.658 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 99, 2024-04-14T15:21:05.659 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:21:05.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.659 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 4, 2024-04-14T15:21:05.659 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 2 2024-04-14T15:21:05.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.659 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 7, 2024-04-14T15:21:05.659 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 2 2024-04-14T15:21:05.659 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.659 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.659 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 109, 2024-04-14T15:21:05.659 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:21:05.660 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.660 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 1, 2024-04-14T15:21:05.660 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2024-04-14T15:21:05.660 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.660 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.660 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.660 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 128, 2024-04-14T15:21:05.660 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:21:05.660 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.660 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 1, 2024-04-14T15:21:05.660 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 2 2024-04-14T15:21:05.660 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.660 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.661 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 142, 2024-04-14T15:21:05.661 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:21:05.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.661 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 3, 2024-04-14T15:21:05.661 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2024-04-14T15:21:05.661 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.661 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.661 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 149, 2024-04-14T15:21:05.661 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:21:05.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.661 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 1, 2024-04-14T15:21:05.661 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 4 2024-04-14T15:21:05.662 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.662 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.662 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 156, 2024-04-14T15:21:05.662 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:21:05.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.662 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 2, 2024-04-14T15:21:05.662 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2024-04-14T15:21:05.662 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.662 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.662 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 157, 2024-04-14T15:21:05.663 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:21:05.663 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.663 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 2, 2024-04-14T15:21:05.663 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 2 2024-04-14T15:21:05.663 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.663 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.663 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.663 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.663 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 167, 2024-04-14T15:21:05.663 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:21:05.663 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.663 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 2, 2024-04-14T15:21:05.663 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 3 2024-04-14T15:21:05.663 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.663 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.663 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.664 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 172, 2024-04-14T15:21:05.664 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:21:05.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.664 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 3, 2024-04-14T15:21:05.664 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 4 2024-04-14T15:21:05.664 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.664 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.664 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 175, 2024-04-14T15:21:05.664 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:21:05.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.664 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 2, 2024-04-14T15:21:05.664 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 4 2024-04-14T15:21:05.665 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.665 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.665 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 176, 2024-04-14T15:21:05.665 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:21:05.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.665 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 2, 2024-04-14T15:21:05.665 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 2 2024-04-14T15:21:05.665 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.665 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.665 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 187, 2024-04-14T15:21:05.665 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:21:05.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.666 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 2, 2024-04-14T15:21:05.666 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2024-04-14T15:21:05.666 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.666 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.666 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 192, 2024-04-14T15:21:05.666 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:21:05.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.666 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 1, 2024-04-14T15:21:05.666 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 4 2024-04-14T15:21:05.666 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.666 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.667 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.667 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 205, 2024-04-14T15:21:05.667 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:21:05.667 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.667 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 2, 2024-04-14T15:21:05.667 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 6 2024-04-14T15:21:05.667 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.667 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.667 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.667 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 213, 2024-04-14T15:21:05.667 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:21:05.667 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.667 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 7, 2024-04-14T15:21:05.668 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2024-04-14T15:21:05.668 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.668 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.668 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 234, 2024-04-14T15:21:05.668 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:21:05.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.668 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 8, 2024-04-14T15:21:05.668 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2024-04-14T15:21:05.668 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.668 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.668 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 247, 2024-04-14T15:21:05.668 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:21:05.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.669 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 10, 2024-04-14T15:21:05.669 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2024-04-14T15:21:05.669 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.669 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.669 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 252, 2024-04-14T15:21:05.669 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:21:05.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.669 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 11, 2024-04-14T15:21:05.669 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 2 2024-04-14T15:21:05.669 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.669 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:05.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.670 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 265, 2024-04-14T15:21:05.670 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-14T15:21:05.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:05.670 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 14, 2024-04-14T15:21:05.670 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2024-04-14T15:21:05.670 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.670 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:05.670 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.670 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-14T15:21:05.670 INFO:teuthology.orchestra.run.smithi090.stdout: "new_removed_snaps": [], 2024-04-14T15:21:05.670 INFO:teuthology.orchestra.run.smithi090.stdout: "new_purged_snaps": [], 2024-04-14T15:21:05.670 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_node_flags": {}, 2024-04-14T15:21:05.670 INFO:teuthology.orchestra.run.smithi090.stdout: "device_class_flags": {}, 2024-04-14T15:21:05.671 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_mode": { 2024-04-14T15:21:05.671 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_mode_enabled": false, 2024-04-14T15:21:05.671 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_bucket_count": 0, 2024-04-14T15:21:05.671 INFO:teuthology.orchestra.run.smithi090.stdout: "degraded_stretch_mode": 0, 2024-04-14T15:21:05.671 INFO:teuthology.orchestra.run.smithi090.stdout: "recovering_stretch_mode": 0, 2024-04-14T15:21:05.671 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_mode_bucket": 0 2024-04-14T15:21:05.671 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:05.671 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-14T15:21:05.671 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:05.672 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:21:05 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:21:05.696 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:05.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:05.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:05.798 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:05.824 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:05.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:05.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:05.926 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:21:05.950 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:05.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:05.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:21:06.051 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:06.078 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:06.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:06.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:06.180 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:21:06.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:05 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:21:06.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:05 smithi090 ceph-mon[41670]: osdmap e540: 8 total, 8 up, 7 in 2024-04-14T15:21:06.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:05 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-14T15:21:06.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:05 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-14T15:21:06.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:05 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-14T15:21:06.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:05 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/4101025460' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:21:06.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:05 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-04-14T15:21:06.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:05 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2956839545' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-14T15:21:06.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:05 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:21:06.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:05 smithi090 ceph-mon[30399]: osdmap e540: 8 total, 8 up, 7 in 2024-04-14T15:21:06.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:05 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-14T15:21:06.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:05 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-14T15:21:06.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:05 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-14T15:21:06.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:05 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/4101025460' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:21:06.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:05 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-04-14T15:21:06.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:05 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2956839545' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-14T15:21:06.207 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:06.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:06.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:21:06.308 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:21:06.331 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:06.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:06.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:21:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:05 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:21:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:05 smithi092 ceph-mon[33966]: osdmap e540: 8 total, 8 up, 7 in 2024-04-14T15:21:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:05 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-14T15:21:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:05 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-14T15:21:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:05 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-14T15:21:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:05 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/4101025460' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:21:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:05 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-04-14T15:21:06.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:05 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2956839545' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-14T15:21:06.433 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:21:06.460 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:06.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:06.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:21:06.562 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:21:06.586 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:06.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:06.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:21:06.687 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:21:06.715 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:06.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:06.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:21:06.761 INFO:tasks.thrashosds.thrasher:j is {'epoch': 540, 'fsid': '0e288098-fa70-11ee-bc8e-c7b262605968', 'created': '2024-04-14T15:04:05.808313+0000', 'modified': '2024-04-14T15:21:04.955113+0000', 'last_up_change': '2024-04-14T15:09:29.769352+0000', 'last_in_change': '2024-04-14T15:20:51.950363+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 113, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 311, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-14T15:07:23.959628+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 26, 'pg_placement_num': 21, 'pg_placement_num_target': 21, 'pg_num_target': 21, 'pg_num_pending': 26, 'last_pg_merge_meta': {'source_pgid': '1.1a', 'ready_epoch': 539, 'last_epoch_started': 538, 'last_epoch_clean': 538, 'source_version': "91'130", 'target_version': "91'130"}, 'last_change': '539', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '539', 'last_force_op_resend_preluminous': '537', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6200000047683716, 'score_stable': 1.6200000047683716, 'optimal_score': 1, 'raw_score_acting': 1.6200000047683716, 'raw_score_stable': 1.6200000047683716, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-04-14T15:09:41.608420+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 3, 'pool_name': 'datapool', 'create_time': '2024-04-14T15:09:41.744391+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 69, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.3299999237060547, 'score_stable': 2.3299999237060547, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-04-14T15:09:43.088157+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.190000057220459, 'score_stable': 2.190000057220459, 'optimal_score': 1, 'raw_score_acting': 2.190000057220459, 'raw_score_stable': 2.190000057220459, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-04-14T15:09:45.042722+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5299999713897705, 'score_stable': 1.5299999713897705, 'optimal_score': 1, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-04-14T15:09:47.111336+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 11, 'pool_name': 'ceph_test_rados_api_asio', 'create_time': '2024-04-14T15:12:54.225671+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.190000057220459, 'score_stable': 2.190000057220459, 'optimal_score': 1, 'raw_score_acting': 2.190000057220459, 'raw_score_stable': 2.190000057220459, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 110, 'pool_name': 'test-rados-api-smithi090-93652-6', 'create_time': '2024-04-14T15:13:07.960794+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '489', 'last_force_op_resend': '399', 'last_force_op_resend_prenautilus': '399', 'last_force_op_resend_preluminous': '399', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 26, 'snap_epoch': 489, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'fingerprint_algorithm': 'sha1'}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5299999713897705, 'score_stable': 1.5299999713897705, 'optimal_score': 1, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 311, 'pool_name': 'test-rados-api-smithi090-93652-90', 'create_time': '2024-04-14T15:21:02.969894+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '540', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.190000057220459, 'score_stable': 2.190000057220459, 'optimal_score': 1, 'raw_score_acting': 2.190000057220459, 'raw_score_stable': 2.190000057220459, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}], 'osds': [{'osd': 0, 'uuid': '76772cfc-89d1-4013-b617-32284ed8dfd9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 539, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6801', 'nonce': 3761806885}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6802', 'nonce': 3761806885}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6804', 'nonce': 3761806885}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6803', 'nonce': 3761806885}]}, 'public_addr': '172.21.15.90:6801/3761806885', 'cluster_addr': '172.21.15.90:6802/3761806885', 'heartbeat_back_addr': '172.21.15.90:6804/3761806885', 'heartbeat_front_addr': '172.21.15.90:6803/3761806885', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'de4a7fa2-7d57-4ff3-a4fa-28c0cde8ef6c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 539, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6805', 'nonce': 4170361699}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6806', 'nonce': 4170361699}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6808', 'nonce': 4170361699}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6807', 'nonce': 4170361699}]}, 'public_addr': '172.21.15.90:6805/4170361699', 'cluster_addr': '172.21.15.90:6806/4170361699', 'heartbeat_back_addr': '172.21.15.90:6808/4170361699', 'heartbeat_front_addr': '172.21.15.90:6807/4170361699', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '9c39b5ff-6175-46ba-9d71-370581c6ff3d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 539, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6809', 'nonce': 1686184251}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6810', 'nonce': 1686184251}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6812', 'nonce': 1686184251}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6811', 'nonce': 1686184251}]}, 'public_addr': '172.21.15.90:6809/1686184251', 'cluster_addr': '172.21.15.90:6810/1686184251', 'heartbeat_back_addr': '172.21.15.90:6812/1686184251', 'heartbeat_front_addr': '172.21.15.90:6811/1686184251', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'efd2ca45-5b80-4013-a571-499e2cb5b2e8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 539, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6813', 'nonce': 2270166774}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6814', 'nonce': 2270166774}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6816', 'nonce': 2270166774}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6815', 'nonce': 2270166774}]}, 'public_addr': '172.21.15.90:6813/2270166774', 'cluster_addr': '172.21.15.90:6814/2270166774', 'heartbeat_back_addr': '172.21.15.90:6816/2270166774', 'heartbeat_front_addr': '172.21.15.90:6815/2270166774', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'e3d9013d-5eb6-433c-96b3-3fc38ba7ac8d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 539, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6800', 'nonce': 3887549733}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6801', 'nonce': 3887549733}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6803', 'nonce': 3887549733}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6802', 'nonce': 3887549733}]}, 'public_addr': '172.21.15.92:6800/3887549733', 'cluster_addr': '172.21.15.92:6801/3887549733', 'heartbeat_back_addr': '172.21.15.92:6803/3887549733', 'heartbeat_front_addr': '172.21.15.92:6802/3887549733', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '1ec41805-6f82-4139-81bc-75305ba9e958', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 41, 'up_thru': 271, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6804', 'nonce': 3244561440}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6805', 'nonce': 3244561440}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6807', 'nonce': 3244561440}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6806', 'nonce': 3244561440}]}, 'public_addr': '172.21.15.92:6804/3244561440', 'cluster_addr': '172.21.15.92:6805/3244561440', 'heartbeat_back_addr': '172.21.15.92:6807/3244561440', 'heartbeat_front_addr': '172.21.15.92:6806/3244561440', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'ca977ab2-bf8d-4b3a-ae5e-0efa5bc7a3ad', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 49, 'up_thru': 521, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6808', 'nonce': 636125963}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6809', 'nonce': 636125963}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6811', 'nonce': 636125963}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6810', 'nonce': 636125963}]}, 'public_addr': '172.21.15.92:6808/636125963', 'cluster_addr': '172.21.15.92:6809/636125963', 'heartbeat_back_addr': '172.21.15.92:6811/636125963', 'heartbeat_front_addr': '172.21.15.92:6810/636125963', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'e5354645-79ec-4d4c-b6e5-f9b0653915ee', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 58, 'up_thru': 539, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6812', 'nonce': 1773706716}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6813', 'nonce': 1773706716}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6815', 'nonce': 1773706716}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6814', 'nonce': 1773706716}]}, 'public_addr': '172.21.15.92:6812/1773706716', 'cluster_addr': '172.21.15.92:6813/1773706716', 'heartbeat_back_addr': '172.21.15.92:6815/1773706716', 'heartbeat_front_addr': '172.21.15.92:6814/1773706716', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-14T15:20:25.995394+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-14T15:20:48.161602+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-14T15:20:48.864326+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-14T15:20:25.493644+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-14T15:20:24.298175+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-14T15:20:12.532132+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-14T15:20:10.183706+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-14T15:20:49.257144+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '2.1d', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '4.2', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.12', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.1f', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '110.3', 'mappings': [{'from': 5, 'to': 7}, {'from': 2, 'to': 1}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '1.c', 'osds': [3, 0, 6]}], 'primary_temp': [], 'blocklist': {'172.21.15.90:0/508768479': '2024-04-15T15:10:22.918674+0000', '172.21.15.90:0/1823072952': '2024-04-15T15:10:22.918674+0000', '172.21.15.90:6800/491071118': '2024-04-15T15:10:22.918674+0000', '172.21.15.90:0/1337064035': '2024-04-15T15:10:22.918674+0000', '172.21.15.90:0/769462306': '2024-04-15T15:10:22.918674+0000', '172.21.15.90:0/1139868610': '2024-04-15T15:05:05.882361+0000', '172.21.15.90:0/93763408': '2024-04-15T15:10:22.918674+0000', '172.21.15.90:0/977818204': '2024-04-15T15:04:39.251711+0000', '172.21.15.90:6800/941091171': '2024-04-15T15:05:05.882361+0000', '172.21.15.90:0/3671453561': '2024-04-15T15:05:05.882361+0000', '172.21.15.90:0/2611856200': '2024-04-15T15:05:05.882361+0000', '172.21.15.90:6800/2306258962': '2024-04-15T15:04:39.251711+0000', '172.21.15.90:0/4051692380': '2024-04-15T15:04:39.251711+0000', '172.21.15.90:0/2681502561': '2024-04-15T15:04:39.251711+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 17, 'snaps': [{'begin': 2, 'length': 5}]}, {'pool': 18, 'snaps': [{'begin': 2, 'length': 3}]}, {'pool': 39, 'snaps': [{'begin': 1, 'length': 1}]}, {'pool': 72, 'snaps': [{'begin': 1, 'length': 1}]}, {'pool': 99, 'snaps': [{'begin': 4, 'length': 2}, {'begin': 7, 'length': 2}]}, {'pool': 109, 'snaps': [{'begin': 1, 'length': 1}]}, {'pool': 128, 'snaps': [{'begin': 1, 'length': 2}]}, {'pool': 142, 'snaps': [{'begin': 3, 'length': 1}]}, {'pool': 149, 'snaps': [{'begin': 1, 'length': 4}]}, {'pool': 156, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 157, 'snaps': [{'begin': 2, 'length': 2}]}, {'pool': 167, 'snaps': [{'begin': 2, 'length': 3}]}, {'pool': 172, 'snaps': [{'begin': 3, 'length': 4}]}, {'pool': 175, 'snaps': [{'begin': 2, 'length': 4}]}, {'pool': 176, 'snaps': [{'begin': 2, 'length': 2}]}, {'pool': 187, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 192, 'snaps': [{'begin': 1, 'length': 4}]}, {'pool': 205, 'snaps': [{'begin': 2, 'length': 6}]}, {'pool': 213, 'snaps': [{'begin': 7, 'length': 1}]}, {'pool': 234, 'snaps': [{'begin': 8, 'length': 1}]}, {'pool': 247, 'snaps': [{'begin': 10, 'length': 1}]}, {'pool': 252, 'snaps': [{'begin': 11, 'length': 2}]}, {'pool': 265, 'snaps': [{'begin': 14, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-14T15:21:06.761 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 4.2 2024-04-14T15:21:06.761 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd rm-pg-upmap-items 4.2 2024-04-14T15:21:06.816 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:21:06.840 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:06.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:06.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:21:06.941 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:21:06.969 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:06.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:06.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:21:07.060 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:06 smithi090 ceph-mon[41670]: pgmap v785: 253 pgs: 32 creating+peering, 1 active+remapped+backfill_toofull, 1 peering, 219 active+clean; 8.4 MiB data, 504 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s; 1/1170 objects misplaced (0.085%) 2024-04-14T15:21:07.060 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:06 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:21:07.060 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:06 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-14T15:21:07.060 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:06 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "25"}]': finished 2024-04-14T15:21:07.060 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:06 smithi090 ceph-mon[41670]: osdmap e541: 8 total, 8 up, 7 in 2024-04-14T15:21:07.060 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:06 smithi090 ceph-mon[30399]: pgmap v785: 253 pgs: 32 creating+peering, 1 active+remapped+backfill_toofull, 1 peering, 219 active+clean; 8.4 MiB data, 504 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s; 1/1170 objects misplaced (0.085%) 2024-04-14T15:21:07.060 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:06 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:21:07.060 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:06 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-14T15:21:07.060 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:06 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "25"}]': finished 2024-04-14T15:21:07.061 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:06 smithi090 ceph-mon[30399]: osdmap e541: 8 total, 8 up, 7 in 2024-04-14T15:21:07.070 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:21:07.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:07.095 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:07.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:21:07.196 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:21:07.220 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:07.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:07.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:21:07.284 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:21:07.323 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:07.351 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:07.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:07.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:07.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:06 smithi092 ceph-mon[33966]: pgmap v785: 253 pgs: 32 creating+peering, 1 active+remapped+backfill_toofull, 1 peering, 219 active+clean; 8.4 MiB data, 504 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s; 1/1170 objects misplaced (0.085%) 2024-04-14T15:21:07.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:06 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:21:07.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:06 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-14T15:21:07.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:06 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "25"}]': finished 2024-04-14T15:21:07.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:06 smithi092 ceph-mon[33966]: osdmap e541: 8 total, 8 up, 7 in 2024-04-14T15:21:07.453 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:21:07.476 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:07.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:07.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:21:07.578 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:21:07.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:07.602 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:07.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:21:07.703 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:07.731 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:07.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:07.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:07.832 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:07.859 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:07.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:07.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:07.961 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:21:07.985 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:07.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:07.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:21:08.086 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:21:08.113 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:08.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:08.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:21:08.161 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:07 smithi090 ceph-mon[30399]: osdmap e542: 8 total, 8 up, 7 in 2024-04-14T15:21:08.161 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:07 smithi090 ceph-mon[41670]: osdmap e542: 8 total, 8 up, 7 in 2024-04-14T15:21:08.215 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:21:08.239 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:08.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:08.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:21:08.341 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:08.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:08.369 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:08.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:08.371 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:07 smithi092 ceph-mon[33966]: osdmap e542: 8 total, 8 up, 7 in 2024-04-14T15:21:08.470 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:21:08.497 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:08.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:08.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:21:08.598 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:21:08.623 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:08.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:08.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:21:08.724 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:21:08.752 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:08.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:08.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:21:08.853 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:21:08.878 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:08.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:08.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:21:08.980 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:21:09.016 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:09.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:09.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:21:09.111 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:08 smithi090 ceph-mon[41670]: pgmap v788: 253 pgs: 32 creating+peering, 1 active+remapped+backfill_toofull, 1 peering, 219 active+clean; 8.4 MiB data, 504 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s; 1/1170 objects misplaced (0.085%) 2024-04-14T15:21:09.111 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:08 smithi090 ceph-mon[41670]: osdmap e543: 8 total, 8 up, 7 in 2024-04-14T15:21:09.111 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:08 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:21:09.111 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:08 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:21:09.111 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:08 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-90"}]: dispatch 2024-04-14T15:21:09.112 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:08 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-90"}]: dispatch 2024-04-14T15:21:09.112 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:08 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:21:09.112 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:08 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:21:09.112 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:08 smithi090 ceph-mon[30399]: pgmap v788: 253 pgs: 32 creating+peering, 1 active+remapped+backfill_toofull, 1 peering, 219 active+clean; 8.4 MiB data, 504 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s; 1/1170 objects misplaced (0.085%) 2024-04-14T15:21:09.112 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:08 smithi090 ceph-mon[30399]: osdmap e543: 8 total, 8 up, 7 in 2024-04-14T15:21:09.112 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:08 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:21:09.112 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:08 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:21:09.112 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:08 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-90"}]: dispatch 2024-04-14T15:21:09.112 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:08 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-90"}]: dispatch 2024-04-14T15:21:09.112 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:08 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:21:09.113 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:08 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:21:09.118 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:21:09.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:09.142+0000 7faacfd23700 1 -- 172.21.15.90:0/453962181 >> v1:172.21.15.90:6790/0 conn(0x7faac810f350 legacy=0x7faac81117e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:09.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:09.142+0000 7faacfd23700 1 -- 172.21.15.90:0/453962181 shutdown_connections 2024-04-14T15:21:09.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:09.142+0000 7faacfd23700 1 -- 172.21.15.90:0/453962181 >> 172.21.15.90:0/453962181 conn(0x7faac80fd280 msgr2=0x7faac80ff6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:21:09.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:09.142+0000 7faacfd23700 1 -- 172.21.15.90:0/453962181 shutdown_connections 2024-04-14T15:21:09.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:09.142+0000 7faacfd23700 1 -- 172.21.15.90:0/453962181 wait complete. 2024-04-14T15:21:09.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:09.142+0000 7faacfd23700 1 librados: starting msgr at 2024-04-14T15:21:09.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:09.142+0000 7faacfd23700 1 librados: starting objecter 2024-04-14T15:21:09.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:09.142+0000 7faacfd23700 1 Processor -- start 2024-04-14T15:21:09.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:09.143+0000 7faacfd23700 1 -- start start 2024-04-14T15:21:09.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:09.143+0000 7faacfd23700 1 librados: setting wanted keys 2024-04-14T15:21:09.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:09.143+0000 7faacfd23700 1 librados: calling monclient init 2024-04-14T15:21:09.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:09.143+0000 7faacfd23700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7faac81a9b10 con 0x7faac810b840 2024-04-14T15:21:09.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:09.143+0000 7faacfd23700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7faac81aad30 con 0x7faac8103a30 2024-04-14T15:21:09.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:09.143+0000 7faacfd23700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7faac81abf50 con 0x7faac810f350 2024-04-14T15:21:09.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:09.143+0000 7faacd2be700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7faac810b840 0x7faac81a4b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:60040/0 (socket says 172.21.15.90:60040) 2024-04-14T15:21:09.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:09.143+0000 7faacd2be700 1 -- 172.21.15.90:0/463847464 learned_addr learned my addr 172.21.15.90:0/463847464 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:21:09.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:09.143+0000 7faabeffd700 10 client.?.objecter ms_handle_connect 0x7faac810f350 2024-04-14T15:21:09.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:09.143+0000 7faabeffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:21:09.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:09.143+0000 7faabeffd700 10 client.?.objecter ms_handle_connect 0x7faac810b840 2024-04-14T15:21:09.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:09.143+0000 7faabeffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:21:09.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:09.143+0000 7faabeffd700 10 client.?.objecter ms_handle_connect 0x7faac8103a30 2024-04-14T15:21:09.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:09.143+0000 7faabeffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:21:09.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:09.143+0000 7faabeffd700 1 -- 172.21.15.90:0/463847464 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4097065890 0 0) 0x7faac81abf50 con 0x7faac810f350 2024-04-14T15:21:09.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:09.143+0000 7faabeffd700 1 -- 172.21.15.90:0/463847464 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faab40035b0 con 0x7faac810f350 2024-04-14T15:21:09.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:09.143+0000 7faabeffd700 1 -- 172.21.15.90:0/463847464 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2723021510 0 0) 0x7faac81a9b10 con 0x7faac810b840 2024-04-14T15:21:09.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:09.143+0000 7faabeffd700 1 -- 172.21.15.90:0/463847464 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faac81abf50 con 0x7faac810b840 2024-04-14T15:21:09.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:09.143+0000 7faabeffd700 1 -- 172.21.15.90:0/463847464 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2027954747 0 0) 0x7faac81aad30 con 0x7faac8103a30 2024-04-14T15:21:09.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:09.143+0000 7faabeffd700 1 -- 172.21.15.90:0/463847464 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faac81a9b10 con 0x7faac8103a30 2024-04-14T15:21:09.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:09.144+0000 7faabeffd700 1 -- 172.21.15.90:0/463847464 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1748404379 0 0) 0x7faac81abf50 con 0x7faac810b840 2024-04-14T15:21:09.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:09.144+0000 7faabeffd700 1 -- 172.21.15.90:0/463847464 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faac81aad30 con 0x7faac810b840 2024-04-14T15:21:09.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:09.144+0000 7faabeffd700 1 -- 172.21.15.90:0/463847464 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2727542410 0 0) 0x7faab40035b0 con 0x7faac810f350 2024-04-14T15:21:09.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:09.144+0000 7faabeffd700 1 -- 172.21.15.90:0/463847464 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faac81abf50 con 0x7faac810f350 2024-04-14T15:21:09.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:09.144+0000 7faabeffd700 1 -- 172.21.15.90:0/463847464 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7faab80018a0 con 0x7faac810b840 2024-04-14T15:21:09.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:09.144+0000 7faabeffd700 1 -- 172.21.15.90:0/463847464 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7faac0003c90 con 0x7faac810f350 2024-04-14T15:21:09.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:09.144+0000 7faabeffd700 1 -- 172.21.15.90:0/463847464 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1395693285 0 0) 0x7faac81a9b10 con 0x7faac8103a30 2024-04-14T15:21:09.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:09.144+0000 7faabeffd700 1 -- 172.21.15.90:0/463847464 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faab40035b0 con 0x7faac8103a30 2024-04-14T15:21:09.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:09.144+0000 7faabeffd700 1 -- 172.21.15.90:0/463847464 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7faac4001850 con 0x7faac8103a30 2024-04-14T15:21:09.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:09.144+0000 7faabeffd700 1 -- 172.21.15.90:0/463847464 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3146534340 0 0) 0x7faac81aad30 con 0x7faac810b840 2024-04-14T15:21:09.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:09.144+0000 7faabeffd700 1 -- 172.21.15.90:0/463847464 >> v1:172.21.15.90:6790/0 conn(0x7faac810f350 legacy=0x7faac81a8290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:09.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:09.144+0000 7faabeffd700 1 -- 172.21.15.90:0/463847464 >> v1:172.21.15.92:6789/0 conn(0x7faac8103a30 legacy=0x7faac810afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:09.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:09.144+0000 7faabeffd700 1 -- 172.21.15.90:0/463847464 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faac81ad190 con 0x7faac810b840 2024-04-14T15:21:09.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:09.144+0000 7faacfd23700 1 -- 172.21.15.90:0/463847464 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7faac81a9d40 con 0x7faac810b840 2024-04-14T15:21:09.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:09.144+0000 7faacfd23700 10 client.37557.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:21:09.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:09.144+0000 7faacfd23700 1 -- 172.21.15.90:0/463847464 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7faac81aa200 con 0x7faac810b840 2024-04-14T15:21:09.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:09.144+0000 7faacfd23700 1 librados: init done 2024-04-14T15:21:09.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:09.144+0000 7faabeffd700 1 -- 172.21.15.90:0/463847464 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7faab8001cd0 con 0x7faac810b840 2024-04-14T15:21:09.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:09.144+0000 7faabeffd700 1 -- 172.21.15.90:0/463847464 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7faab8005d10 con 0x7faac810b840 2024-04-14T15:21:09.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:09.145+0000 7faacfd23700 1 -- 172.21.15.90:0/463847464 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faaac005120 con 0x7faac810b840 2024-04-14T15:21:09.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:09.146+0000 7faabeffd700 1 -- 172.21.15.90:0/463847464 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7faab8003930 con 0x7faac810b840 2024-04-14T15:21:09.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:09.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:09.146+0000 7faabeffd700 1 -- 172.21.15.90:0/463847464 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(544..544 src has 467..544) ==== 7845+0+0 (unknown 3772899367 0 0) 0x7faab808a860 con 0x7faac810b840 2024-04-14T15:21:09.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:09.146+0000 7faabeffd700 10 client.37557.objecter ms_dispatch 0x7faac8001550 osd_map(544..544 src has 467..544) 2024-04-14T15:21:09.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:09.146+0000 7faabeffd700 3 client.37557.objecter handle_osd_map got epochs [544,544] > 0 2024-04-14T15:21:09.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:09.146+0000 7faabeffd700 3 client.37557.objecter handle_osd_map decoding full epoch 544 2024-04-14T15:21:09.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:09.146+0000 7faabeffd700 20 client.37557.objecter dump_active .. 0 homeless 2024-04-14T15:21:09.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:09.146+0000 7faabeffd700 10 client.37557.objecter ms_handle_connect 0x7faab4071330 2024-04-14T15:21:09.151 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:09.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:21:09.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:09.151+0000 7faabeffd700 1 -- 172.21.15.90:0/463847464 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7faab8005d10 con 0x7faac810b840 2024-04-14T15:21:09.252 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:21:09.279 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:09.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:09.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:21:09.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:09.364+0000 7faacfd23700 1 -- 172.21.15.90:0/463847464 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7faaac005410 con 0x7faac810b840 2024-04-14T15:21:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:08 smithi092 ceph-mon[33966]: pgmap v788: 253 pgs: 32 creating+peering, 1 active+remapped+backfill_toofull, 1 peering, 219 active+clean; 8.4 MiB data, 504 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s; 1/1170 objects misplaced (0.085%) 2024-04-14T15:21:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:08 smithi092 ceph-mon[33966]: osdmap e543: 8 total, 8 up, 7 in 2024-04-14T15:21:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:08 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:21:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:08 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:21:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:08 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-90"}]: dispatch 2024-04-14T15:21:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:08 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-90"}]: dispatch 2024-04-14T15:21:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:08 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:21:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:08 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:21:09.382 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:09.408 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:09.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:09.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:09.452 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:21:09.510 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:21:09.534 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:09.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:09.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:21:09.636 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:21:09.660 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:09.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:09.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:21:09.761 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:09.789 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:09.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:09.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:09.890 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:21:09.914 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:09.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:09.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:21:09.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:09.998+0000 7faabeffd700 1 -- 172.21.15.90:0/463847464 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v545) ==== 91+0+0 (unknown 4160089944 0 0) 0x7faab80573f0 con 0x7faac810b840 2024-04-14T15:21:10.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:10.001+0000 7faacfd23700 10 librados: watch_flush enter 2024-04-14T15:21:10.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:10.001+0000 7faacfd23700 10 librados: watch_flush exit 2024-04-14T15:21:10.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:10.001+0000 7faacfd23700 20 client.37557.objecter shutdown clearing up homeless session... 2024-04-14T15:21:10.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:10.001+0000 7faacfd23700 10 client.37557.objecter successfully canceled tick 2024-04-14T15:21:10.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:10.001+0000 7faacfd23700 1 -- 172.21.15.90:0/463847464 >> v1:172.21.15.90:6800/420935427 conn(0x7faab4071330 legacy=0x7faab40737f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:10.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:10.002+0000 7faacfd23700 1 -- 172.21.15.90:0/463847464 >> v1:172.21.15.90:6789/0 conn(0x7faac810b840 legacy=0x7faac81a4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:10.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:10.003+0000 7faacfd23700 1 -- 172.21.15.90:0/463847464 shutdown_connections 2024-04-14T15:21:10.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:10.003+0000 7faacfd23700 1 -- 172.21.15.90:0/463847464 >> 172.21.15.90:0/463847464 conn(0x7faac80fd280 msgr2=0x7faac81117a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:21:10.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:10.006+0000 7faacfd23700 1 -- 172.21.15.90:0/463847464 shutdown_connections 2024-04-14T15:21:10.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:10.006+0000 7faacfd23700 1 -- 172.21.15.90:0/463847464 wait complete. 2024-04-14T15:21:10.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:10.006+0000 7faacfd23700 1 librados: shutdown 2024-04-14T15:21:10.015 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is unset 2024-04-14T15:21:10.015 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:21:10.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:10.040 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:10.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:21:10.142 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:21:10.166 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:10.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:10.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:21:10.268 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:21:10.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:10.296 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:10.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:21:10.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:10 smithi092 ceph-mon[33966]: osdmap e544: 8 total, 8 up, 7 in 2024-04-14T15:21:10.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:10 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-04-14T15:21:10.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:10 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/463847464' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:21:10.396 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:21:10.399 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:10 smithi090 ceph-mon[30399]: osdmap e544: 8 total, 8 up, 7 in 2024-04-14T15:21:10.399 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:10 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-04-14T15:21:10.399 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:10 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/463847464' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:21:10.399 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:10 smithi090 ceph-mon[41670]: osdmap e544: 8 total, 8 up, 7 in 2024-04-14T15:21:10.399 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:10 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-04-14T15:21:10.400 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:10 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/463847464' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:21:10.420 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:10.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:10.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:21:10.521 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:10.548 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:10.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:10.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:10.650 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:21:10.677 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:10.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:10.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:21:10.778 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:10.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:10.806 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:10.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:10.827 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-14T15:21:10.907 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:21:10.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:10.931 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:10.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:21:11.032 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:21:11.055 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:11.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:11.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:21:11.060 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:11 smithi090 ceph-mon[41670]: pgmap v791: 220 pgs: 1 active+remapped+backfill_toofull, 1 activating, 1 peering, 217 active+clean; 8.4 MiB data, 501 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s wr, 1 op/s; 1/1179 objects misplaced (0.085%) 2024-04-14T15:21:11.060 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:11 smithi090 ceph-mon[41670]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:21:11.060 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:11 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "24"}]': finished 2024-04-14T15:21:11.061 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:11 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/463847464' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-14T15:21:11.061 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:11 smithi090 ceph-mon[41670]: osdmap e545: 8 total, 8 up, 7 in 2024-04-14T15:21:11.061 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:11 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:21:11.061 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:11 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:21:11.061 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:11 smithi090 ceph-mon[41670]: 1.17 scrub starts 2024-04-14T15:21:11.061 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:11 smithi090 ceph-mon[41670]: 1.17 scrub ok 2024-04-14T15:21:11.061 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:11 smithi090 ceph-mon[41670]: 1.16 scrub starts 2024-04-14T15:21:11.061 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:11 smithi090 ceph-mon[41670]: 1.16 scrub ok 2024-04-14T15:21:11.061 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:11 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:21:11.061 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:11 smithi090 ceph-mon[41670]: osdmap e546: 8 total, 8 up, 7 in 2024-04-14T15:21:11.061 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:11 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-14T15:21:11.061 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:11 smithi090 ceph-mon[30399]: pgmap v791: 220 pgs: 1 active+remapped+backfill_toofull, 1 activating, 1 peering, 217 active+clean; 8.4 MiB data, 501 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s wr, 1 op/s; 1/1179 objects misplaced (0.085%) 2024-04-14T15:21:11.062 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:11 smithi090 ceph-mon[30399]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:21:11.062 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:11 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "24"}]': finished 2024-04-14T15:21:11.062 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:11 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/463847464' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-14T15:21:11.062 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:11 smithi090 ceph-mon[30399]: osdmap e545: 8 total, 8 up, 7 in 2024-04-14T15:21:11.062 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:11 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:21:11.062 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:11 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:21:11.062 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:11 smithi090 ceph-mon[30399]: 1.17 scrub starts 2024-04-14T15:21:11.062 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:11 smithi090 ceph-mon[30399]: 1.17 scrub ok 2024-04-14T15:21:11.062 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:11 smithi090 ceph-mon[30399]: 1.16 scrub starts 2024-04-14T15:21:11.062 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:11 smithi090 ceph-mon[30399]: 1.16 scrub ok 2024-04-14T15:21:11.062 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:11 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:21:11.062 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:11 smithi090 ceph-mon[30399]: osdmap e546: 8 total, 8 up, 7 in 2024-04-14T15:21:11.062 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:11 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-14T15:21:11.157 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:21:11.184 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:11.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:11.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:21:11.286 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:21:11.313 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:11.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:11.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:21:11.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:11 smithi092 ceph-mon[33966]: pgmap v791: 220 pgs: 1 active+remapped+backfill_toofull, 1 activating, 1 peering, 217 active+clean; 8.4 MiB data, 501 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s wr, 1 op/s; 1/1179 objects misplaced (0.085%) 2024-04-14T15:21:11.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:11 smithi092 ceph-mon[33966]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:21:11.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:11 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "24"}]': finished 2024-04-14T15:21:11.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:11 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/463847464' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-14T15:21:11.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:11 smithi092 ceph-mon[33966]: osdmap e545: 8 total, 8 up, 7 in 2024-04-14T15:21:11.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:11 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:21:11.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:11 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:21:11.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:11 smithi092 ceph-mon[33966]: 1.17 scrub starts 2024-04-14T15:21:11.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:11 smithi092 ceph-mon[33966]: 1.17 scrub ok 2024-04-14T15:21:11.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:11 smithi092 ceph-mon[33966]: 1.16 scrub starts 2024-04-14T15:21:11.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:11 smithi092 ceph-mon[33966]: 1.16 scrub ok 2024-04-14T15:21:11.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:11 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:21:11.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:11 smithi092 ceph-mon[33966]: osdmap e546: 8 total, 8 up, 7 in 2024-04-14T15:21:11.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:11 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-14T15:21:11.415 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:11.442 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:11.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:11.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:11.544 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:21:11.570 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:11.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:11.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:21:11.672 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:21:11.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:11.699 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:11.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:21:11.800 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:21:11.827 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:11.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:11.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:21:11.929 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:21:11.957 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:11.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:11.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:21:12.059 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:21:12.083 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:12.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:12.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:21:12.185 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:12.212 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:12.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:12.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:12.251 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:21:12.290 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-14T15:21:12.290 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-14T15:21:12.291 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-14T15:21:12.291 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-14T15:21:12.291 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-14T15:21:12.291 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:12 smithi090 ceph-mon[30399]: 1.f scrub starts 2024-04-14T15:21:12.291 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:12 smithi090 ceph-mon[30399]: 1.f scrub ok 2024-04-14T15:21:12.291 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:12 smithi090 ceph-mon[30399]: 1.7 scrub starts 2024-04-14T15:21:12.291 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:12 smithi090 ceph-mon[30399]: 1.7 scrub ok 2024-04-14T15:21:12.291 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:12 smithi090 ceph-mon[30399]: pgmap v794: 252 pgs: 32 unknown, 1 active+remapped+backfill_toofull, 1 activating, 218 active+clean; 8.4 MiB data, 497 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s; 1/1179 objects misplaced (0.085%) 2024-04-14T15:21:12.291 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:12 smithi090 ceph-mon[30399]: 1.6 scrub starts 2024-04-14T15:21:12.291 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:12 smithi090 ceph-mon[30399]: 1.6 scrub ok 2024-04-14T15:21:12.292 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:12 smithi090 ceph-mon[30399]: osdmap e547: 8 total, 8 up, 7 in 2024-04-14T15:21:12.292 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:12 smithi090 ceph-mon[41670]: 1.f scrub starts 2024-04-14T15:21:12.292 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:12 smithi090 ceph-mon[41670]: 1.f scrub ok 2024-04-14T15:21:12.292 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:12 smithi090 ceph-mon[41670]: 1.7 scrub starts 2024-04-14T15:21:12.292 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:12 smithi090 ceph-mon[41670]: 1.7 scrub ok 2024-04-14T15:21:12.292 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:12 smithi090 ceph-mon[41670]: pgmap v794: 252 pgs: 32 unknown, 1 active+remapped+backfill_toofull, 1 activating, 218 active+clean; 8.4 MiB data, 497 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s; 1/1179 objects misplaced (0.085%) 2024-04-14T15:21:12.292 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:12 smithi090 ceph-mon[41670]: 1.6 scrub starts 2024-04-14T15:21:12.292 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:12 smithi090 ceph-mon[41670]: 1.6 scrub ok 2024-04-14T15:21:12.292 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:12 smithi090 ceph-mon[41670]: osdmap e547: 8 total, 8 up, 7 in 2024-04-14T15:21:12.313 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:21:12.336 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:12.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:12.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:21:12.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:12 smithi092 ceph-mon[33966]: 1.f scrub starts 2024-04-14T15:21:12.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:12 smithi092 ceph-mon[33966]: 1.f scrub ok 2024-04-14T15:21:12.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:12 smithi092 ceph-mon[33966]: 1.7 scrub starts 2024-04-14T15:21:12.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:12 smithi092 ceph-mon[33966]: 1.7 scrub ok 2024-04-14T15:21:12.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:12 smithi092 ceph-mon[33966]: pgmap v794: 252 pgs: 32 unknown, 1 active+remapped+backfill_toofull, 1 activating, 218 active+clean; 8.4 MiB data, 497 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s; 1/1179 objects misplaced (0.085%) 2024-04-14T15:21:12.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:12 smithi092 ceph-mon[33966]: 1.6 scrub starts 2024-04-14T15:21:12.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:12 smithi092 ceph-mon[33966]: 1.6 scrub ok 2024-04-14T15:21:12.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:12 smithi092 ceph-mon[33966]: osdmap e547: 8 total, 8 up, 7 in 2024-04-14T15:21:12.438 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:21:12.465 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:12.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:12.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:21:12.567 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:12.593 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:12.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:12.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:12.694 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:21:12.718 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:12.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:12.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:21:12.810 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-14T15:21:12.820 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:21:12.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:12.877 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:12.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:21:12.978 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:21:13.003 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:13.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:13.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:21:13.106 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:21:13.131 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:13.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:13.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:21:13.232 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:21:13.257 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:13.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:13.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:21:13.358 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:21:13.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:13 smithi092 ceph-mon[33966]: 4.3 scrub starts 2024-04-14T15:21:13.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:13 smithi092 ceph-mon[33966]: 4.3 scrub ok 2024-04-14T15:21:13.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:13 smithi092 ceph-mon[33966]: osdmap e548: 8 total, 8 up, 7 in 2024-04-14T15:21:13.382 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:13.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:13.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:21:13.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:13 smithi090 ceph-mon[30399]: 4.3 scrub starts 2024-04-14T15:21:13.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:13 smithi090 ceph-mon[30399]: 4.3 scrub ok 2024-04-14T15:21:13.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:13 smithi090 ceph-mon[30399]: osdmap e548: 8 total, 8 up, 7 in 2024-04-14T15:21:13.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:13 smithi090 ceph-mon[41670]: 4.3 scrub starts 2024-04-14T15:21:13.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:13 smithi090 ceph-mon[41670]: 4.3 scrub ok 2024-04-14T15:21:13.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:13 smithi090 ceph-mon[41670]: osdmap e548: 8 total, 8 up, 7 in 2024-04-14T15:21:13.484 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:21:13.491 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-14T15:21:13.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:13.509 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:13.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:21:13.610 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:21:13.635 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:13.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:13.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:21:13.736 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:21:13.759 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:13.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:13.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:21:13.861 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:13.888 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:13.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:13.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:13.989 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:21:14.015 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:14.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:14.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:21:14.117 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:21:14.146 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:14.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:14.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:21:14.184 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:21:13 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:21:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:21:14.185 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:14 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:21:14.185 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:14 smithi090 ceph-mon[30399]: 4.11 scrub starts 2024-04-14T15:21:14.185 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:14 smithi090 ceph-mon[30399]: 4.11 scrub ok 2024-04-14T15:21:14.185 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:14 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:21:14.185 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:14 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-92"}]: dispatch 2024-04-14T15:21:14.185 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:14 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-92"}]: dispatch 2024-04-14T15:21:14.185 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:14 smithi090 ceph-mon[30399]: pgmap v797: 251 pgs: 32 unknown, 1 active+remapped+backfill_toofull, 1 activating, 217 active+clean; 8.2 MiB data, 497 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/1176 objects misplaced (0.085%) 2024-04-14T15:21:14.185 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:14 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-04-14T15:21:14.186 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:14 smithi090 ceph-mon[30399]: 4.f scrub starts 2024-04-14T15:21:14.186 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:14 smithi090 ceph-mon[30399]: 4.f scrub ok 2024-04-14T15:21:14.186 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:14 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:21:14.186 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:14 smithi090 ceph-mon[41670]: 4.11 scrub starts 2024-04-14T15:21:14.186 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:14 smithi090 ceph-mon[41670]: 4.11 scrub ok 2024-04-14T15:21:14.186 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:14 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:21:14.186 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:14 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-92"}]: dispatch 2024-04-14T15:21:14.186 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:14 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-92"}]: dispatch 2024-04-14T15:21:14.186 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:14 smithi090 ceph-mon[41670]: pgmap v797: 251 pgs: 32 unknown, 1 active+remapped+backfill_toofull, 1 activating, 217 active+clean; 8.2 MiB data, 497 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/1176 objects misplaced (0.085%) 2024-04-14T15:21:14.186 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:14 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-04-14T15:21:14.187 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:14 smithi090 ceph-mon[41670]: 4.f scrub starts 2024-04-14T15:21:14.187 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:14 smithi090 ceph-mon[41670]: 4.f scrub ok 2024-04-14T15:21:14.248 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:21:14.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:14.275 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:14.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:21:14.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:13.939+0000 7f2673bd5700 1 -- 172.21.15.90:0/934340074 >> v1:172.21.15.90:6789/0 conn(0x7f266c107760 legacy=0x7f266c107b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:14.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:13.939+0000 7f2673bd5700 1 -- 172.21.15.90:0/934340074 shutdown_connections 2024-04-14T15:21:14.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:13.939+0000 7f2673bd5700 1 -- 172.21.15.90:0/934340074 >> 172.21.15.90:0/934340074 conn(0x7f266c0fd290 msgr2=0x7f266c0ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:21:14.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:13.939+0000 7f2673bd5700 1 -- 172.21.15.90:0/934340074 shutdown_connections 2024-04-14T15:21:14.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:13.939+0000 7f2673bd5700 1 -- 172.21.15.90:0/934340074 wait complete. 2024-04-14T15:21:14.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:13.940+0000 7f2673bd5700 1 librados: starting msgr at 2024-04-14T15:21:14.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:13.940+0000 7f2673bd5700 1 librados: starting objecter 2024-04-14T15:21:14.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:13.940+0000 7f2673bd5700 1 Processor -- start 2024-04-14T15:21:14.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:13.940+0000 7f2673bd5700 1 -- start start 2024-04-14T15:21:14.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:13.940+0000 7f2673bd5700 1 librados: setting wanted keys 2024-04-14T15:21:14.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:13.940+0000 7f2673bd5700 1 librados: calling monclient init 2024-04-14T15:21:14.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:13.940+0000 7f2673bd5700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f266c1a7830 con 0x7f266c10e1e0 2024-04-14T15:21:14.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:13.940+0000 7f2673bd5700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f266c1a8a50 con 0x7f266c10a610 2024-04-14T15:21:14.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:13.940+0000 7f2673bd5700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f266c1a9c90 con 0x7f266c107760 2024-04-14T15:21:14.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:13.940+0000 7f2672172700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f266c10e1e0 0x7f266c1a5f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:36866/0 (socket says 172.21.15.90:36866) 2024-04-14T15:21:14.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:13.940+0000 7f2671971700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f266c107760 0x7f266c10d910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:33122/0 (socket says 172.21.15.90:33122) 2024-04-14T15:21:14.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:13.940+0000 7f2672172700 1 -- 172.21.15.90:0/2649042674 learned_addr learned my addr 172.21.15.90:0/2649042674 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:21:14.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:13.940+0000 7f2662ffd700 10 client.?.objecter ms_handle_connect 0x7f266c10e1e0 2024-04-14T15:21:14.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:13.940+0000 7f2662ffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:21:14.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:13.940+0000 7f2662ffd700 10 client.?.objecter ms_handle_connect 0x7f266c107760 2024-04-14T15:21:14.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:13.940+0000 7f2662ffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:21:14.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:13.940+0000 7f2662ffd700 1 -- 172.21.15.90:0/2649042674 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1722695687 0 0) 0x7f266c1a7830 con 0x7f266c10e1e0 2024-04-14T15:21:14.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:13.940+0000 7f2662ffd700 1 -- 172.21.15.90:0/2649042674 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f26580031f0 con 0x7f266c10e1e0 2024-04-14T15:21:14.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:13.940+0000 7f2662ffd700 10 client.?.objecter ms_handle_connect 0x7f266c10a610 2024-04-14T15:21:14.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:13.940+0000 7f2662ffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:21:14.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:13.940+0000 7f2662ffd700 1 -- 172.21.15.90:0/2649042674 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2732206156 0 0) 0x7f266c1a9c90 con 0x7f266c107760 2024-04-14T15:21:14.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:13.940+0000 7f2662ffd700 1 -- 172.21.15.90:0/2649042674 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f266c1a7830 con 0x7f266c107760 2024-04-14T15:21:14.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:13.941+0000 7f2662ffd700 1 -- 172.21.15.90:0/2649042674 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 700806459 0 0) 0x7f266c1a8a50 con 0x7f266c10a610 2024-04-14T15:21:14.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:13.941+0000 7f2662ffd700 1 -- 172.21.15.90:0/2649042674 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f266c1a9c90 con 0x7f266c10a610 2024-04-14T15:21:14.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:13.941+0000 7f2662ffd700 1 -- 172.21.15.90:0/2649042674 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 143959794 0 0) 0x7f26580031f0 con 0x7f266c10e1e0 2024-04-14T15:21:14.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:13.941+0000 7f2662ffd700 1 -- 172.21.15.90:0/2649042674 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f266c1a8a50 con 0x7f266c10e1e0 2024-04-14T15:21:14.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:13.941+0000 7f2662ffd700 1 -- 172.21.15.90:0/2649042674 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f2664001d50 con 0x7f266c10e1e0 2024-04-14T15:21:14.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:13.941+0000 7f2662ffd700 1 -- 172.21.15.90:0/2649042674 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2962306036 0 0) 0x7f266c1a7830 con 0x7f266c107760 2024-04-14T15:21:14.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:13.941+0000 7f2662ffd700 1 -- 172.21.15.90:0/2649042674 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f26580031f0 con 0x7f266c107760 2024-04-14T15:21:14.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:13.941+0000 7f2662ffd700 1 -- 172.21.15.90:0/2649042674 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f2668004550 con 0x7f266c107760 2024-04-14T15:21:14.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:13.941+0000 7f2662ffd700 1 -- 172.21.15.90:0/2649042674 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1605498939 0 0) 0x7f266c1a9c90 con 0x7f266c10a610 2024-04-14T15:21:14.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:13.941+0000 7f2662ffd700 1 -- 172.21.15.90:0/2649042674 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f266c1a7830 con 0x7f266c10a610 2024-04-14T15:21:14.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:13.941+0000 7f2662ffd700 1 -- 172.21.15.90:0/2649042674 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f265c001130 con 0x7f266c10a610 2024-04-14T15:21:14.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:13.941+0000 7f2662ffd700 1 -- 172.21.15.90:0/2649042674 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2965404162 0 0) 0x7f266c1a8a50 con 0x7f266c10e1e0 2024-04-14T15:21:14.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:13.941+0000 7f2662ffd700 1 -- 172.21.15.90:0/2649042674 >> v1:172.21.15.90:6790/0 conn(0x7f266c107760 legacy=0x7f266c10d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:14.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:13.941+0000 7f2662ffd700 1 -- 172.21.15.90:0/2649042674 >> v1:172.21.15.92:6789/0 conn(0x7f266c10a610 legacy=0x7f266c1a26a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:14.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:13.941+0000 7f2662ffd700 1 -- 172.21.15.90:0/2649042674 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f266c1aaed0 con 0x7f266c10e1e0 2024-04-14T15:21:14.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:13.941+0000 7f2673bd5700 1 -- 172.21.15.90:0/2649042674 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f266c1a9ec0 con 0x7f266c10e1e0 2024-04-14T15:21:14.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:13.941+0000 7f2673bd5700 10 client.37563.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:21:14.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:13.941+0000 7f2673bd5700 1 -- 172.21.15.90:0/2649042674 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f266c1aa380 con 0x7f266c10e1e0 2024-04-14T15:21:14.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:13.941+0000 7f2673bd5700 1 librados: init done 2024-04-14T15:21:14.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:13.941+0000 7f2662ffd700 1 -- 172.21.15.90:0/2649042674 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f2664002080 con 0x7f266c10e1e0 2024-04-14T15:21:14.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:13.941+0000 7f2662ffd700 1 -- 172.21.15.90:0/2649042674 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f2664006140 con 0x7f266c10e1e0 2024-04-14T15:21:14.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:13.942+0000 7f2673bd5700 1 -- 172.21.15.90:0/2649042674 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2650005150 con 0x7f266c10e1e0 2024-04-14T15:21:14.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:13.942+0000 7f2662ffd700 1 -- 172.21.15.90:0/2649042674 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f2664003cb0 con 0x7f266c10e1e0 2024-04-14T15:21:14.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:13.943+0000 7f2662ffd700 1 -- 172.21.15.90:0/2649042674 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(548..548 src has 467..548) ==== 8230+0+0 (unknown 2458245730 0 0) 0x7f266408add0 con 0x7f266c10e1e0 2024-04-14T15:21:14.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:13.943+0000 7f2662ffd700 10 client.37563.objecter ms_dispatch 0x7f266c001550 osd_map(548..548 src has 467..548) 2024-04-14T15:21:14.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:13.943+0000 7f2662ffd700 3 client.37563.objecter handle_osd_map got epochs [548,548] > 0 2024-04-14T15:21:14.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:13.943+0000 7f2662ffd700 3 client.37563.objecter handle_osd_map decoding full epoch 548 2024-04-14T15:21:14.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:13.943+0000 7f2662ffd700 20 client.37563.objecter dump_active .. 0 homeless 2024-04-14T15:21:14.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:13.943+0000 7f2662ffd700 10 client.37563.objecter ms_handle_connect 0x7f26580713e0 2024-04-14T15:21:14.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:13.947+0000 7f2662ffd700 1 -- 172.21.15.90:0/2649042674 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f26640bbfa0 con 0x7f266c10e1e0 2024-04-14T15:21:14.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:14.184+0000 7f2673bd5700 1 -- 172.21.15.90:0/2649042674 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd rm-pg-upmap-items", "pgid": "4.2"} v 0) -- 0x7f2650005440 con 0x7f266c10e1e0 2024-04-14T15:21:14.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:14 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:21:14.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:14 smithi092 ceph-mon[33966]: 4.11 scrub starts 2024-04-14T15:21:14.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:14 smithi092 ceph-mon[33966]: 4.11 scrub ok 2024-04-14T15:21:14.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:14 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:21:14.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:14 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-92"}]: dispatch 2024-04-14T15:21:14.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:14 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-92"}]: dispatch 2024-04-14T15:21:14.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:14 smithi092 ceph-mon[33966]: pgmap v797: 251 pgs: 32 unknown, 1 active+remapped+backfill_toofull, 1 activating, 217 active+clean; 8.2 MiB data, 497 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/1176 objects misplaced (0.085%) 2024-04-14T15:21:14.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:14 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-04-14T15:21:14.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:14 smithi092 ceph-mon[33966]: 4.f scrub starts 2024-04-14T15:21:14.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:14 smithi092 ceph-mon[33966]: 4.f scrub ok 2024-04-14T15:21:14.376 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:21:14.400 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:14.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:14.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:21:14.501 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:14.528 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:14.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:14.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:14.630 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:21:14.653 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:14.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:14.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:21:14.755 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:21:14.779 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:14.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:14.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:21:14.844 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:21:14.881 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:21:14.904 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:14.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:14.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:21:15.006 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:21:15.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:15.017+0000 7f2662ffd700 1 -- 172.21.15.90:0/2649042674 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd rm-pg-upmap-items", "pgid": "4.2"}]=0 clear 4.2 pg_upmap_items mapping v550) ==== 116+0+0 (unknown 1298022493 0 0) 0x7f2664006140 con 0x7f266c10e1e0 2024-04-14T15:21:15.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:15.020+0000 7f2673bd5700 10 librados: watch_flush enter 2024-04-14T15:21:15.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:15.022+0000 7f2673bd5700 10 librados: watch_flush exit 2024-04-14T15:21:15.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:15.022+0000 7f2673bd5700 20 client.37563.objecter shutdown clearing up homeless session... 2024-04-14T15:21:15.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:15.022+0000 7f2673bd5700 10 client.37563.objecter successfully canceled tick 2024-04-14T15:21:15.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:15.023+0000 7f2673bd5700 1 -- 172.21.15.90:0/2649042674 >> v1:172.21.15.90:6800/420935427 conn(0x7f26580713e0 legacy=0x7f26580738a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:15.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:15.023+0000 7f2673bd5700 1 -- 172.21.15.90:0/2649042674 >> v1:172.21.15.90:6789/0 conn(0x7f266c10e1e0 legacy=0x7f266c1a5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:15.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:15.023+0000 7f2673bd5700 1 -- 172.21.15.90:0/2649042674 shutdown_connections 2024-04-14T15:21:15.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:15.023+0000 7f2673bd5700 1 -- 172.21.15.90:0/2649042674 >> 172.21.15.90:0/2649042674 conn(0x7f266c0fd290 msgr2=0x7f266c0ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:21:15.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:15.023+0000 7f2673bd5700 1 -- 172.21.15.90:0/2649042674 shutdown_connections 2024-04-14T15:21:15.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:15.023+0000 7f2673bd5700 1 -- 172.21.15.90:0/2649042674 wait complete. 2024-04-14T15:21:15.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:15.024+0000 7f2673bd5700 1 librados: shutdown 2024-04-14T15:21:15.032 INFO:teuthology.orchestra.run.smithi090.stderr:clear 4.2 pg_upmap_items mapping 2024-04-14T15:21:15.039 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:15.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:15.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:21:15.075 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:15 smithi090 ceph-mon[30399]: 110.c scrub starts 2024-04-14T15:21:15.075 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:15 smithi090 ceph-mon[30399]: 110.c scrub ok 2024-04-14T15:21:15.075 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:15 smithi090 ceph-mon[30399]: 4.e scrub starts 2024-04-14T15:21:15.075 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:15 smithi090 ceph-mon[30399]: 4.e scrub ok 2024-04-14T15:21:15.075 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:15 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "23"}]': finished 2024-04-14T15:21:15.075 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:15 smithi090 ceph-mon[30399]: osdmap e549: 8 total, 8 up, 7 in 2024-04-14T15:21:15.075 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:15 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2649042674' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.2"}]: dispatch 2024-04-14T15:21:15.075 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:15 smithi090 ceph-mon[41670]: 110.c scrub starts 2024-04-14T15:21:15.076 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:15 smithi090 ceph-mon[41670]: 110.c scrub ok 2024-04-14T15:21:15.076 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:15 smithi090 ceph-mon[41670]: 4.e scrub starts 2024-04-14T15:21:15.076 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:15 smithi090 ceph-mon[41670]: 4.e scrub ok 2024-04-14T15:21:15.076 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:15 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "23"}]': finished 2024-04-14T15:21:15.076 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:15 smithi090 ceph-mon[41670]: osdmap e549: 8 total, 8 up, 7 in 2024-04-14T15:21:15.076 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:15 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2649042674' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.2"}]: dispatch 2024-04-14T15:21:15.141 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:21:15.165 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:15.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:15.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:21:15.267 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:21:15.290 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:15.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:15.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:21:15.372 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:21:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:21:15.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:15 smithi092 ceph-mon[33966]: 110.c scrub starts 2024-04-14T15:21:15.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:15 smithi092 ceph-mon[33966]: 110.c scrub ok 2024-04-14T15:21:15.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:15 smithi092 ceph-mon[33966]: 4.e scrub starts 2024-04-14T15:21:15.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:15 smithi092 ceph-mon[33966]: 4.e scrub ok 2024-04-14T15:21:15.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:15 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "23"}]': finished 2024-04-14T15:21:15.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:15 smithi092 ceph-mon[33966]: osdmap e549: 8 total, 8 up, 7 in 2024-04-14T15:21:15.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:15 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2649042674' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.2"}]: dispatch 2024-04-14T15:21:15.392 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:21:15.419 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:15.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:15.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:21:15.521 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:21:15.545 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:15.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:15.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:21:15.647 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:21:15.674 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:15.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:15.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:21:15.776 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:15.802 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:15.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:15.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:15.904 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:21:15.928 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:15.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:15.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:21:16.030 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:21:16.067 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:16.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:16.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:21:16.122 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:21:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=cleanup t=2024-04-14T15:21:15.865720252Z level=info msg="Completed cleanup jobs" duration=2.347087ms 2024-04-14T15:21:16.122 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:21:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=sqlstore.transactions t=2024-04-14T15:21:15.87376472Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-04-14T15:21:16.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:16 smithi092 ceph-mon[33966]: 11.16 scrub starts 2024-04-14T15:21:16.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:16 smithi092 ceph-mon[33966]: 11.16 scrub ok 2024-04-14T15:21:16.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:16 smithi092 ceph-mon[33966]: 5.8 scrub starts 2024-04-14T15:21:16.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:16 smithi092 ceph-mon[33966]: 5.8 scrub ok 2024-04-14T15:21:16.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:16 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2649042674' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.2"}]': finished 2024-04-14T15:21:16.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:16 smithi092 ceph-mon[33966]: osdmap e550: 8 total, 8 up, 7 in 2024-04-14T15:21:16.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:16 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:21:16.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:16 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:21:16.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:16 smithi092 ceph-mon[33966]: pgmap v800: 251 pgs: 32 unknown, 1 active+remapped+backfill_toofull, 218 active+clean; 8.4 MiB data, 494 MiB used, 625 GiB / 626 GiB avail; 1/1188 objects misplaced (0.084%) 2024-04-14T15:21:16.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:16 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:21:16.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.154+0000 7fafee4a9700 1 -- 172.21.15.90:0/4182299691 >> v1:172.21.15.90:6789/0 conn(0x7fafe810b760 legacy=0x7fafe810db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:16.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.154+0000 7fafee4a9700 1 -- 172.21.15.90:0/4182299691 shutdown_connections 2024-04-14T15:21:16.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.154+0000 7fafee4a9700 1 -- 172.21.15.90:0/4182299691 >> 172.21.15.90:0/4182299691 conn(0x7fafe80fe3c0 msgr2=0x7fafe81007e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:21:16.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.154+0000 7fafee4a9700 1 -- 172.21.15.90:0/4182299691 shutdown_connections 2024-04-14T15:21:16.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.154+0000 7fafee4a9700 1 -- 172.21.15.90:0/4182299691 wait complete. 2024-04-14T15:21:16.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.155+0000 7fafee4a9700 1 librados: starting msgr at 2024-04-14T15:21:16.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.155+0000 7fafee4a9700 1 librados: starting objecter 2024-04-14T15:21:16.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.155+0000 7fafee4a9700 1 Processor -- start 2024-04-14T15:21:16.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.155+0000 7fafee4a9700 1 -- start start 2024-04-14T15:21:16.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.155+0000 7fafee4a9700 1 librados: setting wanted keys 2024-04-14T15:21:16.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.155+0000 7fafee4a9700 1 librados: calling monclient init 2024-04-14T15:21:16.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.155+0000 7fafee4a9700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fafe80769e0 con 0x7fafe810f330 2024-04-14T15:21:16.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.155+0000 7fafee4a9700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fafe8076bf0 con 0x7fafe81088b0 2024-04-14T15:21:16.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.155+0000 7fafee4a9700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fafe81ac340 con 0x7fafe810b760 2024-04-14T15:21:16.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.155+0000 7fafe77fe700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7fafe810b760 0x7fafe80754c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:33136/0 (socket says 172.21.15.90:33136) 2024-04-14T15:21:16.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.155+0000 7fafe77fe700 1 -- 172.21.15.90:0/3877361749 learned_addr learned my addr 172.21.15.90:0/3877361749 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:21:16.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.155+0000 7fafe57fa700 10 client.?.objecter ms_handle_connect 0x7fafe810f330 2024-04-14T15:21:16.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.155+0000 7fafe57fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:21:16.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.155+0000 7fafe57fa700 10 client.?.objecter ms_handle_connect 0x7fafe810b760 2024-04-14T15:21:16.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.155+0000 7fafe57fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:21:16.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.156+0000 7fafe57fa700 1 -- 172.21.15.90:0/3877361749 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2365855887 0 0) 0x7fafe80769e0 con 0x7fafe810f330 2024-04-14T15:21:16.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.156+0000 7fafe57fa700 1 -- 172.21.15.90:0/3877361749 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fafd00035b0 con 0x7fafe810f330 2024-04-14T15:21:16.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.156+0000 7fafe57fa700 1 -- 172.21.15.90:0/3877361749 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2026886289 0 0) 0x7fafe81ac340 con 0x7fafe810b760 2024-04-14T15:21:16.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.156+0000 7fafe57fa700 1 -- 172.21.15.90:0/3877361749 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fafe80769e0 con 0x7fafe810b760 2024-04-14T15:21:16.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.156+0000 7fafe57fa700 10 client.?.objecter ms_handle_connect 0x7fafe81088b0 2024-04-14T15:21:16.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.156+0000 7fafe57fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:21:16.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.156+0000 7fafe57fa700 1 -- 172.21.15.90:0/3877361749 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1836342228 0 0) 0x7fafe8076bf0 con 0x7fafe81088b0 2024-04-14T15:21:16.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.156+0000 7fafe57fa700 1 -- 172.21.15.90:0/3877361749 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fafe81ac340 con 0x7fafe81088b0 2024-04-14T15:21:16.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.156+0000 7fafe57fa700 1 -- 172.21.15.90:0/3877361749 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2745956331 0 0) 0x7fafd00035b0 con 0x7fafe810f330 2024-04-14T15:21:16.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.156+0000 7fafe57fa700 1 -- 172.21.15.90:0/3877361749 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fafe8076bf0 con 0x7fafe810f330 2024-04-14T15:21:16.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.156+0000 7fafe57fa700 1 -- 172.21.15.90:0/3877361749 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fafe0001c30 con 0x7fafe810f330 2024-04-14T15:21:16.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.156+0000 7fafe57fa700 1 -- 172.21.15.90:0/3877361749 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1674693827 0 0) 0x7fafe80769e0 con 0x7fafe810b760 2024-04-14T15:21:16.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.156+0000 7fafe57fa700 1 -- 172.21.15.90:0/3877361749 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fafd00035b0 con 0x7fafe810b760 2024-04-14T15:21:16.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.156+0000 7fafe57fa700 1 -- 172.21.15.90:0/3877361749 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fafdc001430 con 0x7fafe810b760 2024-04-14T15:21:16.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.156+0000 7fafe57fa700 1 -- 172.21.15.90:0/3877361749 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 648307407 0 0) 0x7fafe81ac340 con 0x7fafe81088b0 2024-04-14T15:21:16.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.156+0000 7fafe57fa700 1 -- 172.21.15.90:0/3877361749 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fafe80769e0 con 0x7fafe81088b0 2024-04-14T15:21:16.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.156+0000 7fafe57fa700 1 -- 172.21.15.90:0/3877361749 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fafd80018b0 con 0x7fafe81088b0 2024-04-14T15:21:16.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.156+0000 7fafe57fa700 1 -- 172.21.15.90:0/3877361749 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3948440925 0 0) 0x7fafe8076bf0 con 0x7fafe810f330 2024-04-14T15:21:16.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.156+0000 7fafe57fa700 1 -- 172.21.15.90:0/3877361749 >> v1:172.21.15.90:6790/0 conn(0x7fafe810b760 legacy=0x7fafe80754c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:16.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.156+0000 7fafe57fa700 1 -- 172.21.15.90:0/3877361749 >> v1:172.21.15.92:6789/0 conn(0x7fafe81088b0 legacy=0x7fafe8074db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:16.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.156+0000 7fafe57fa700 1 -- 172.21.15.90:0/3877361749 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafe81ad520 con 0x7fafe810f330 2024-04-14T15:21:16.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.156+0000 7fafee4a9700 1 -- 172.21.15.90:0/3877361749 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fafe81ac570 con 0x7fafe810f330 2024-04-14T15:21:16.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.156+0000 7fafee4a9700 10 client.37569.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:21:16.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.156+0000 7fafee4a9700 1 -- 172.21.15.90:0/3877361749 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fafe81aca30 con 0x7fafe810f330 2024-04-14T15:21:16.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.156+0000 7fafee4a9700 1 librados: init done 2024-04-14T15:21:16.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.156+0000 7fafe57fa700 1 -- 172.21.15.90:0/3877361749 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fafe0003ec0 con 0x7fafe810f330 2024-04-14T15:21:16.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.156+0000 7fafe57fa700 1 -- 172.21.15.90:0/3877361749 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fafe0004d80 con 0x7fafe810f330 2024-04-14T15:21:16.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.157+0000 7fafee4a9700 1 -- 172.21.15.90:0/3877361749 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fafd4005100 con 0x7fafe810f330 2024-04-14T15:21:16.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.158+0000 7fafe57fa700 1 -- 172.21.15.90:0/3877361749 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fafe0003a20 con 0x7fafe810f330 2024-04-14T15:21:16.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.158+0000 7fafe57fa700 1 -- 172.21.15.90:0/3877361749 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(551..551 src has 467..551) ==== 8201+0+0 (unknown 3563501404 0 0) 0x7fafe0089940 con 0x7fafe810f330 2024-04-14T15:21:16.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.158+0000 7fafe57fa700 10 client.37569.objecter ms_dispatch 0x7fafe8001550 osd_map(551..551 src has 467..551) 2024-04-14T15:21:16.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.158+0000 7fafe57fa700 3 client.37569.objecter handle_osd_map got epochs [551,551] > 0 2024-04-14T15:21:16.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.158+0000 7fafe57fa700 3 client.37569.objecter handle_osd_map decoding full epoch 551 2024-04-14T15:21:16.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.158+0000 7fafe57fa700 20 client.37569.objecter dump_active .. 0 homeless 2024-04-14T15:21:16.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.158+0000 7fafe57fa700 1 -- 172.21.15.90:0/3877361749 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v551) ==== 54+0+16 (unknown 2194696821 0 2420000192) 0x7fafe008a060 con 0x7fafe810f330 2024-04-14T15:21:16.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.159+0000 7fafe57fa700 10 client.37569.objecter ms_handle_connect 0x7fafd0071330 2024-04-14T15:21:16.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.159+0000 7fafee4a9700 10 client.37569.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:21:16.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.159+0000 7fafee4a9700 15 client.37569.objecter _session_command_op_assign -1 1 2024-04-14T15:21:16.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.159+0000 7fafee4a9700 20 client.37569.objecter _get_session s=0x7fafd40053e0 osd=0 3 2024-04-14T15:21:16.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.159+0000 7fafee4a9700 20 client.37569.objecter put_session s=0x7fafd40053e0 osd=0 3 2024-04-14T15:21:16.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.159+0000 7fafee4a9700 20 client.37569.objecter _get_session s=0x7fafd40053e0 osd=0 3 2024-04-14T15:21:16.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.159+0000 7fafee4a9700 15 client.37569.objecter _session_command_op_remove -1 1 2024-04-14T15:21:16.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.159+0000 7fafee4a9700 20 client.37569.objecter get_session s=0x7fafd40053e0 osd=0 3 2024-04-14T15:21:16.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.159+0000 7fafee4a9700 15 client.37569.objecter _session_command_op_assign 0 1 2024-04-14T15:21:16.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.159+0000 7fafee4a9700 20 client.37569.objecter put_session s=0x7fafd40053e0 osd=0 4 2024-04-14T15:21:16.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.159+0000 7fafee4a9700 10 client.37569.objecter _send_command 1 2024-04-14T15:21:16.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.159+0000 7fafee4a9700 1 -- 172.21.15.90:0/3877361749 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fafd4004b50 con 0x7fafd4004730 2024-04-14T15:21:16.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.159+0000 7fafe57fa700 10 client.37569.objecter ms_handle_connect 0x7fafd4004730 2024-04-14T15:21:16.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.160+0000 7fafe57fa700 1 -- 172.21.15.90:0/3877361749 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fafd4004b50 con 0x7fafd4004730 2024-04-14T15:21:16.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.160+0000 7fafe57fa700 10 client.37569.objecter ms_dispatch 0x7fafe8001550 command_reply(tid 1: 0 ) 2024-04-14T15:21:16.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.160+0000 7fafe57fa700 10 client.37569.objecter _finish_command 1 = system:0 2024-04-14T15:21:16.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.160+0000 7fafe57fa700 20 client.37569.objecter put_session s=0x7fafd40053e0 osd=0 4 2024-04-14T15:21:16.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.160+0000 7fafe57fa700 15 client.37569.objecter _session_command_op_remove 0 1 2024-04-14T15:21:16.169 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:21:16.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.176+0000 7fafee4a9700 10 client.37569.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:21:16.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.176+0000 7fafee4a9700 15 client.37569.objecter _session_command_op_assign -1 2 2024-04-14T15:21:16.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.176+0000 7fafee4a9700 20 client.37569.objecter _get_session s=0x7fafd40053e0 osd=0 3 2024-04-14T15:21:16.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.176+0000 7fafee4a9700 20 client.37569.objecter put_session s=0x7fafd40053e0 osd=0 3 2024-04-14T15:21:16.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.176+0000 7fafee4a9700 20 client.37569.objecter _get_session s=0x7fafd40053e0 osd=0 3 2024-04-14T15:21:16.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.176+0000 7fafee4a9700 15 client.37569.objecter _session_command_op_remove -1 2 2024-04-14T15:21:16.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.176+0000 7fafee4a9700 20 client.37569.objecter get_session s=0x7fafd40053e0 osd=0 3 2024-04-14T15:21:16.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.176+0000 7fafee4a9700 15 client.37569.objecter _session_command_op_assign 0 2 2024-04-14T15:21:16.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.176+0000 7fafee4a9700 20 client.37569.objecter put_session s=0x7fafd40053e0 osd=0 4 2024-04-14T15:21:16.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.176+0000 7fafee4a9700 10 client.37569.objecter _send_command 2 2024-04-14T15:21:16.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.176+0000 7fafee4a9700 1 -- 172.21.15.90:0/3877361749 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fafd40079f0 con 0x7fafd4004730 2024-04-14T15:21:16.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.178+0000 7fafe57fa700 1 -- 172.21.15.90:0/3877361749 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fafd40079f0 con 0x7fafd4004730 2024-04-14T15:21:16.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.178+0000 7fafe57fa700 10 client.37569.objecter ms_dispatch 0x7fafe8001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:21:16.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.178+0000 7fafe57fa700 10 client.37569.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:21:16.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.178+0000 7fafe57fa700 20 client.37569.objecter put_session s=0x7fafd40053e0 osd=0 4 2024-04-14T15:21:16.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.178+0000 7fafe57fa700 15 client.37569.objecter _session_command_op_remove 0 2 2024-04-14T15:21:16.179 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:21:16.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.179+0000 7fafee4a9700 10 client.37569.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:21:16.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.179+0000 7fafee4a9700 15 client.37569.objecter _session_command_op_assign -1 3 2024-04-14T15:21:16.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.179+0000 7fafee4a9700 20 client.37569.objecter _get_session s=0x7fafd4004110 osd=1 3 2024-04-14T15:21:16.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.179+0000 7fafee4a9700 20 client.37569.objecter put_session s=0x7fafd4004110 osd=1 3 2024-04-14T15:21:16.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.179+0000 7fafee4a9700 20 client.37569.objecter _get_session s=0x7fafd4004110 osd=1 3 2024-04-14T15:21:16.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.179+0000 7fafee4a9700 15 client.37569.objecter _session_command_op_remove -1 3 2024-04-14T15:21:16.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.179+0000 7fafee4a9700 20 client.37569.objecter get_session s=0x7fafd4004110 osd=1 3 2024-04-14T15:21:16.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.179+0000 7fafee4a9700 15 client.37569.objecter _session_command_op_assign 1 3 2024-04-14T15:21:16.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.179+0000 7fafee4a9700 20 client.37569.objecter put_session s=0x7fafd4004110 osd=1 4 2024-04-14T15:21:16.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.179+0000 7fafee4a9700 10 client.37569.objecter _send_command 3 2024-04-14T15:21:16.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.179+0000 7fafee4a9700 1 -- 172.21.15.90:0/3877361749 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fafd4002130 con 0x7fafd4004280 2024-04-14T15:21:16.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.179+0000 7fafe57fa700 10 client.37569.objecter ms_handle_connect 0x7fafd4004280 2024-04-14T15:21:16.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.180+0000 7fafe57fa700 1 -- 172.21.15.90:0/3877361749 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fafd4002130 con 0x7fafd4004280 2024-04-14T15:21:16.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.180+0000 7fafe57fa700 10 client.37569.objecter ms_dispatch 0x7fafe8001550 command_reply(tid 3: 0 ) 2024-04-14T15:21:16.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.180+0000 7fafe57fa700 10 client.37569.objecter _finish_command 3 = system:0 2024-04-14T15:21:16.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.180+0000 7fafe57fa700 20 client.37569.objecter put_session s=0x7fafd4004110 osd=1 4 2024-04-14T15:21:16.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.180+0000 7fafe57fa700 15 client.37569.objecter _session_command_op_remove 1 3 2024-04-14T15:21:16.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:16 smithi090 ceph-mon[41670]: 11.16 scrub starts 2024-04-14T15:21:16.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:16 smithi090 ceph-mon[41670]: 11.16 scrub ok 2024-04-14T15:21:16.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:16 smithi090 ceph-mon[41670]: 5.8 scrub starts 2024-04-14T15:21:16.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:16 smithi090 ceph-mon[41670]: 5.8 scrub ok 2024-04-14T15:21:16.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:16 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2649042674' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.2"}]': finished 2024-04-14T15:21:16.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:16 smithi090 ceph-mon[41670]: osdmap e550: 8 total, 8 up, 7 in 2024-04-14T15:21:16.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:16 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:21:16.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:16 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:21:16.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:16 smithi090 ceph-mon[41670]: pgmap v800: 251 pgs: 32 unknown, 1 active+remapped+backfill_toofull, 218 active+clean; 8.4 MiB data, 494 MiB used, 625 GiB / 626 GiB avail; 1/1188 objects misplaced (0.084%) 2024-04-14T15:21:16.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:16 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:21:16.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:16 smithi090 ceph-mon[30399]: 11.16 scrub starts 2024-04-14T15:21:16.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:16 smithi090 ceph-mon[30399]: 11.16 scrub ok 2024-04-14T15:21:16.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:16 smithi090 ceph-mon[30399]: 5.8 scrub starts 2024-04-14T15:21:16.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:16 smithi090 ceph-mon[30399]: 5.8 scrub ok 2024-04-14T15:21:16.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:16 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2649042674' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.2"}]': finished 2024-04-14T15:21:16.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:16 smithi090 ceph-mon[30399]: osdmap e550: 8 total, 8 up, 7 in 2024-04-14T15:21:16.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:16 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:21:16.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:16 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:21:16.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:16 smithi090 ceph-mon[30399]: pgmap v800: 251 pgs: 32 unknown, 1 active+remapped+backfill_toofull, 218 active+clean; 8.4 MiB data, 494 MiB used, 625 GiB / 626 GiB avail; 1/1188 objects misplaced (0.084%) 2024-04-14T15:21:16.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:16 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:21:16.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:16.194 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:16.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:21:16.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.197+0000 7fafee4a9700 10 client.37569.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:21:16.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.197+0000 7fafee4a9700 15 client.37569.objecter _session_command_op_assign -1 4 2024-04-14T15:21:16.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.197+0000 7fafee4a9700 20 client.37569.objecter _get_session s=0x7fafd4004110 osd=1 3 2024-04-14T15:21:16.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.197+0000 7fafee4a9700 20 client.37569.objecter put_session s=0x7fafd4004110 osd=1 3 2024-04-14T15:21:16.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.197+0000 7fafee4a9700 20 client.37569.objecter _get_session s=0x7fafd4004110 osd=1 3 2024-04-14T15:21:16.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.197+0000 7fafee4a9700 15 client.37569.objecter _session_command_op_remove -1 4 2024-04-14T15:21:16.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.197+0000 7fafee4a9700 20 client.37569.objecter get_session s=0x7fafd4004110 osd=1 3 2024-04-14T15:21:16.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.197+0000 7fafee4a9700 15 client.37569.objecter _session_command_op_assign 1 4 2024-04-14T15:21:16.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.197+0000 7fafee4a9700 20 client.37569.objecter put_session s=0x7fafd4004110 osd=1 4 2024-04-14T15:21:16.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.197+0000 7fafee4a9700 10 client.37569.objecter _send_command 4 2024-04-14T15:21:16.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.197+0000 7fafee4a9700 1 -- 172.21.15.90:0/3877361749 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fafd4001600 con 0x7fafd4004280 2024-04-14T15:21:16.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.198+0000 7fafe57fa700 1 -- 172.21.15.90:0/3877361749 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fafd4001600 con 0x7fafd4004280 2024-04-14T15:21:16.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.198+0000 7fafe57fa700 10 client.37569.objecter ms_dispatch 0x7fafe8001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:21:16.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.198+0000 7fafe57fa700 10 client.37569.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:21:16.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.198+0000 7fafe57fa700 20 client.37569.objecter put_session s=0x7fafd4004110 osd=1 4 2024-04-14T15:21:16.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.198+0000 7fafe57fa700 15 client.37569.objecter _session_command_op_remove 1 4 2024-04-14T15:21:16.199 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:21:16.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.199+0000 7fafee4a9700 10 client.37569.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:21:16.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.199+0000 7fafee4a9700 15 client.37569.objecter _session_command_op_assign -1 5 2024-04-14T15:21:16.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.199+0000 7fafee4a9700 20 client.37569.objecter _get_session s=0x7fafd4001a00 osd=2 3 2024-04-14T15:21:16.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.199+0000 7fafee4a9700 20 client.37569.objecter put_session s=0x7fafd4001a00 osd=2 3 2024-04-14T15:21:16.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.199+0000 7fafee4a9700 20 client.37569.objecter _get_session s=0x7fafd4001a00 osd=2 3 2024-04-14T15:21:16.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.199+0000 7fafee4a9700 15 client.37569.objecter _session_command_op_remove -1 5 2024-04-14T15:21:16.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.199+0000 7fafee4a9700 20 client.37569.objecter get_session s=0x7fafd4001a00 osd=2 3 2024-04-14T15:21:16.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.199+0000 7fafee4a9700 15 client.37569.objecter _session_command_op_assign 2 5 2024-04-14T15:21:16.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.199+0000 7fafee4a9700 20 client.37569.objecter put_session s=0x7fafd4001a00 osd=2 4 2024-04-14T15:21:16.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.199+0000 7fafee4a9700 10 client.37569.objecter _send_command 5 2024-04-14T15:21:16.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.199+0000 7fafee4a9700 1 -- 172.21.15.90:0/3877361749 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fafd4010f40 con 0x7fafd40088b0 2024-04-14T15:21:16.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.199+0000 7fafe57fa700 10 client.37569.objecter ms_handle_connect 0x7fafd40088b0 2024-04-14T15:21:16.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.200+0000 7fafe57fa700 1 -- 172.21.15.90:0/3877361749 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fafd4010f40 con 0x7fafd40088b0 2024-04-14T15:21:16.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.200+0000 7fafe57fa700 10 client.37569.objecter ms_dispatch 0x7fafe8001550 command_reply(tid 5: 0 ) 2024-04-14T15:21:16.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.200+0000 7fafe57fa700 10 client.37569.objecter _finish_command 5 = system:0 2024-04-14T15:21:16.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.200+0000 7fafe57fa700 20 client.37569.objecter put_session s=0x7fafd4001a00 osd=2 4 2024-04-14T15:21:16.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.200+0000 7fafe57fa700 15 client.37569.objecter _session_command_op_remove 2 5 2024-04-14T15:21:16.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.216+0000 7fafee4a9700 10 client.37569.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:21:16.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.216+0000 7fafee4a9700 15 client.37569.objecter _session_command_op_assign -1 6 2024-04-14T15:21:16.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.216+0000 7fafee4a9700 20 client.37569.objecter _get_session s=0x7fafd4001a00 osd=2 3 2024-04-14T15:21:16.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.216+0000 7fafee4a9700 20 client.37569.objecter put_session s=0x7fafd4001a00 osd=2 3 2024-04-14T15:21:16.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.216+0000 7fafee4a9700 20 client.37569.objecter _get_session s=0x7fafd4001a00 osd=2 3 2024-04-14T15:21:16.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.216+0000 7fafee4a9700 15 client.37569.objecter _session_command_op_remove -1 6 2024-04-14T15:21:16.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.216+0000 7fafee4a9700 20 client.37569.objecter get_session s=0x7fafd4001a00 osd=2 3 2024-04-14T15:21:16.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.216+0000 7fafee4a9700 15 client.37569.objecter _session_command_op_assign 2 6 2024-04-14T15:21:16.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.216+0000 7fafee4a9700 20 client.37569.objecter put_session s=0x7fafd4001a00 osd=2 4 2024-04-14T15:21:16.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.216+0000 7fafee4a9700 10 client.37569.objecter _send_command 6 2024-04-14T15:21:16.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.216+0000 7fafee4a9700 1 -- 172.21.15.90:0/3877361749 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fafd4007f10 con 0x7fafd40088b0 2024-04-14T15:21:16.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.218+0000 7fafe57fa700 1 -- 172.21.15.90:0/3877361749 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fafd4007f10 con 0x7fafd40088b0 2024-04-14T15:21:16.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.218+0000 7fafe57fa700 10 client.37569.objecter ms_dispatch 0x7fafe8001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:21:16.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.218+0000 7fafe57fa700 10 client.37569.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:21:16.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.218+0000 7fafe57fa700 20 client.37569.objecter put_session s=0x7fafd4001a00 osd=2 4 2024-04-14T15:21:16.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.218+0000 7fafe57fa700 15 client.37569.objecter _session_command_op_remove 2 6 2024-04-14T15:21:16.219 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:21:16.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.218+0000 7fafee4a9700 10 client.37569.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:21:16.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.218+0000 7fafee4a9700 15 client.37569.objecter _session_command_op_assign -1 7 2024-04-14T15:21:16.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.218+0000 7fafee4a9700 20 client.37569.objecter _get_session s=0x7fafd4008170 osd=3 3 2024-04-14T15:21:16.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.218+0000 7fafee4a9700 20 client.37569.objecter put_session s=0x7fafd4008170 osd=3 3 2024-04-14T15:21:16.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.218+0000 7fafee4a9700 20 client.37569.objecter _get_session s=0x7fafd4008170 osd=3 3 2024-04-14T15:21:16.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.218+0000 7fafee4a9700 15 client.37569.objecter _session_command_op_remove -1 7 2024-04-14T15:21:16.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.218+0000 7fafee4a9700 20 client.37569.objecter get_session s=0x7fafd4008170 osd=3 3 2024-04-14T15:21:16.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.218+0000 7fafee4a9700 15 client.37569.objecter _session_command_op_assign 3 7 2024-04-14T15:21:16.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.218+0000 7fafee4a9700 20 client.37569.objecter put_session s=0x7fafd4008170 osd=3 4 2024-04-14T15:21:16.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.218+0000 7fafee4a9700 10 client.37569.objecter _send_command 7 2024-04-14T15:21:16.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.218+0000 7fafee4a9700 1 -- 172.21.15.90:0/3877361749 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fafd400d300 con 0x7fafd400ce70 2024-04-14T15:21:16.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.218+0000 7fafe57fa700 10 client.37569.objecter ms_handle_connect 0x7fafd400ce70 2024-04-14T15:21:16.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.219+0000 7fafe57fa700 1 -- 172.21.15.90:0/3877361749 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fafd400d300 con 0x7fafd400ce70 2024-04-14T15:21:16.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.219+0000 7fafe57fa700 10 client.37569.objecter ms_dispatch 0x7fafe8001550 command_reply(tid 7: 0 ) 2024-04-14T15:21:16.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.219+0000 7fafe57fa700 10 client.37569.objecter _finish_command 7 = system:0 2024-04-14T15:21:16.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.219+0000 7fafe57fa700 20 client.37569.objecter put_session s=0x7fafd4008170 osd=3 4 2024-04-14T15:21:16.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.219+0000 7fafe57fa700 15 client.37569.objecter _session_command_op_remove 3 7 2024-04-14T15:21:16.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.235+0000 7fafee4a9700 10 client.37569.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:21:16.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.235+0000 7fafee4a9700 15 client.37569.objecter _session_command_op_assign -1 8 2024-04-14T15:21:16.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.235+0000 7fafee4a9700 20 client.37569.objecter _get_session s=0x7fafd4008170 osd=3 3 2024-04-14T15:21:16.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.235+0000 7fafee4a9700 20 client.37569.objecter put_session s=0x7fafd4008170 osd=3 3 2024-04-14T15:21:16.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.235+0000 7fafee4a9700 20 client.37569.objecter _get_session s=0x7fafd4008170 osd=3 3 2024-04-14T15:21:16.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.235+0000 7fafee4a9700 15 client.37569.objecter _session_command_op_remove -1 8 2024-04-14T15:21:16.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.235+0000 7fafee4a9700 20 client.37569.objecter get_session s=0x7fafd4008170 osd=3 3 2024-04-14T15:21:16.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.235+0000 7fafee4a9700 15 client.37569.objecter _session_command_op_assign 3 8 2024-04-14T15:21:16.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.235+0000 7fafee4a9700 20 client.37569.objecter put_session s=0x7fafd4008170 osd=3 4 2024-04-14T15:21:16.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.235+0000 7fafee4a9700 10 client.37569.objecter _send_command 8 2024-04-14T15:21:16.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.235+0000 7fafee4a9700 1 -- 172.21.15.90:0/3877361749 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fafd400c950 con 0x7fafd400ce70 2024-04-14T15:21:16.237 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:21:16.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.237+0000 7fafe57fa700 1 -- 172.21.15.90:0/3877361749 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fafd400c950 con 0x7fafd400ce70 2024-04-14T15:21:16.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.237+0000 7fafe57fa700 10 client.37569.objecter ms_dispatch 0x7fafe8001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:21:16.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.237+0000 7fafe57fa700 10 client.37569.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:21:16.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.237+0000 7fafe57fa700 20 client.37569.objecter put_session s=0x7fafd4008170 osd=3 4 2024-04-14T15:21:16.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.237+0000 7fafe57fa700 15 client.37569.objecter _session_command_op_remove 3 8 2024-04-14T15:21:16.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.237+0000 7fafee4a9700 10 client.37569.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:21:16.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.237+0000 7fafee4a9700 15 client.37569.objecter _session_command_op_assign -1 9 2024-04-14T15:21:16.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.237+0000 7fafee4a9700 20 client.37569.objecter _get_session s=0x7fafd400c700 osd=4 3 2024-04-14T15:21:16.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.237+0000 7fafee4a9700 20 client.37569.objecter put_session s=0x7fafd400c700 osd=4 3 2024-04-14T15:21:16.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.237+0000 7fafee4a9700 20 client.37569.objecter _get_session s=0x7fafd400c700 osd=4 3 2024-04-14T15:21:16.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.237+0000 7fafee4a9700 15 client.37569.objecter _session_command_op_remove -1 9 2024-04-14T15:21:16.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.237+0000 7fafee4a9700 20 client.37569.objecter get_session s=0x7fafd400c700 osd=4 3 2024-04-14T15:21:16.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.237+0000 7fafee4a9700 15 client.37569.objecter _session_command_op_assign 4 9 2024-04-14T15:21:16.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.237+0000 7fafee4a9700 20 client.37569.objecter put_session s=0x7fafd400c700 osd=4 4 2024-04-14T15:21:16.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.237+0000 7fafee4a9700 10 client.37569.objecter _send_command 9 2024-04-14T15:21:16.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.237+0000 7fafee4a9700 1 -- 172.21.15.90:0/3877361749 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fafd401a8d0 con 0x7fafd4011dd0 2024-04-14T15:21:16.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.238+0000 7fafe57fa700 10 client.37569.objecter ms_handle_connect 0x7fafd4011dd0 2024-04-14T15:21:16.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.238+0000 7fafe57fa700 1 -- 172.21.15.90:0/3877361749 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fafd401a8d0 con 0x7fafd4011dd0 2024-04-14T15:21:16.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.238+0000 7fafe57fa700 10 client.37569.objecter ms_dispatch 0x7fafe8001550 command_reply(tid 9: 0 ) 2024-04-14T15:21:16.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.238+0000 7fafe57fa700 10 client.37569.objecter _finish_command 9 = system:0 2024-04-14T15:21:16.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.238+0000 7fafe57fa700 20 client.37569.objecter put_session s=0x7fafd400c700 osd=4 4 2024-04-14T15:21:16.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.238+0000 7fafe57fa700 15 client.37569.objecter _session_command_op_remove 4 9 2024-04-14T15:21:16.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.254+0000 7fafee4a9700 10 client.37569.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:21:16.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.254+0000 7fafee4a9700 15 client.37569.objecter _session_command_op_assign -1 10 2024-04-14T15:21:16.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.254+0000 7fafee4a9700 20 client.37569.objecter _get_session s=0x7fafd400c700 osd=4 3 2024-04-14T15:21:16.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.254+0000 7fafee4a9700 20 client.37569.objecter put_session s=0x7fafd400c700 osd=4 3 2024-04-14T15:21:16.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.254+0000 7fafee4a9700 20 client.37569.objecter _get_session s=0x7fafd400c700 osd=4 3 2024-04-14T15:21:16.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.254+0000 7fafee4a9700 15 client.37569.objecter _session_command_op_remove -1 10 2024-04-14T15:21:16.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.254+0000 7fafee4a9700 20 client.37569.objecter get_session s=0x7fafd400c700 osd=4 3 2024-04-14T15:21:16.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.254+0000 7fafee4a9700 15 client.37569.objecter _session_command_op_assign 4 10 2024-04-14T15:21:16.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.254+0000 7fafee4a9700 20 client.37569.objecter put_session s=0x7fafd400c700 osd=4 4 2024-04-14T15:21:16.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.254+0000 7fafee4a9700 10 client.37569.objecter _send_command 10 2024-04-14T15:21:16.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.254+0000 7fafee4a9700 1 -- 172.21.15.90:0/3877361749 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fafd4011340 con 0x7fafd4011dd0 2024-04-14T15:21:16.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.257+0000 7fafe57fa700 1 -- 172.21.15.90:0/3877361749 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fafd4011340 con 0x7fafd4011dd0 2024-04-14T15:21:16.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.257+0000 7fafe57fa700 10 client.37569.objecter ms_dispatch 0x7fafe8001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:21:16.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.257+0000 7fafe57fa700 10 client.37569.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:21:16.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.257+0000 7fafe57fa700 20 client.37569.objecter put_session s=0x7fafd400c700 osd=4 4 2024-04-14T15:21:16.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.257+0000 7fafe57fa700 15 client.37569.objecter _session_command_op_remove 4 10 2024-04-14T15:21:16.258 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:21:16.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.257+0000 7fafee4a9700 10 client.37569.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:21:16.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.257+0000 7fafee4a9700 15 client.37569.objecter _session_command_op_assign -1 11 2024-04-14T15:21:16.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.257+0000 7fafee4a9700 20 client.37569.objecter _get_session s=0x7fafd4011340 osd=5 3 2024-04-14T15:21:16.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.257+0000 7fafee4a9700 20 client.37569.objecter put_session s=0x7fafd4011340 osd=5 3 2024-04-14T15:21:16.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.257+0000 7fafee4a9700 20 client.37569.objecter _get_session s=0x7fafd4011340 osd=5 3 2024-04-14T15:21:16.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.257+0000 7fafee4a9700 15 client.37569.objecter _session_command_op_remove -1 11 2024-04-14T15:21:16.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.257+0000 7fafee4a9700 20 client.37569.objecter get_session s=0x7fafd4011340 osd=5 3 2024-04-14T15:21:16.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.257+0000 7fafee4a9700 15 client.37569.objecter _session_command_op_assign 5 11 2024-04-14T15:21:16.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.257+0000 7fafee4a9700 20 client.37569.objecter put_session s=0x7fafd4011340 osd=5 4 2024-04-14T15:21:16.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.257+0000 7fafee4a9700 10 client.37569.objecter _send_command 11 2024-04-14T15:21:16.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.257+0000 7fafee4a9700 1 -- 172.21.15.90:0/3877361749 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fafd4016fa0 con 0x7fafd40114e0 2024-04-14T15:21:16.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.257+0000 7fafe57fa700 10 client.37569.objecter ms_handle_connect 0x7fafd40114e0 2024-04-14T15:21:16.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.258+0000 7fafe57fa700 1 -- 172.21.15.90:0/3877361749 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fafd4016fa0 con 0x7fafd40114e0 2024-04-14T15:21:16.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.258+0000 7fafe57fa700 10 client.37569.objecter ms_dispatch 0x7fafe8001550 command_reply(tid 11: 0 ) 2024-04-14T15:21:16.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.258+0000 7fafe57fa700 10 client.37569.objecter _finish_command 11 = system:0 2024-04-14T15:21:16.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.258+0000 7fafe57fa700 20 client.37569.objecter put_session s=0x7fafd4011340 osd=5 4 2024-04-14T15:21:16.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.258+0000 7fafe57fa700 15 client.37569.objecter _session_command_op_remove 5 11 2024-04-14T15:21:16.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.274+0000 7fafee4a9700 10 client.37569.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:21:16.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.274+0000 7fafee4a9700 15 client.37569.objecter _session_command_op_assign -1 12 2024-04-14T15:21:16.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.274+0000 7fafee4a9700 20 client.37569.objecter _get_session s=0x7fafd4011340 osd=5 3 2024-04-14T15:21:16.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.274+0000 7fafee4a9700 20 client.37569.objecter put_session s=0x7fafd4011340 osd=5 3 2024-04-14T15:21:16.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.274+0000 7fafee4a9700 20 client.37569.objecter _get_session s=0x7fafd4011340 osd=5 3 2024-04-14T15:21:16.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.274+0000 7fafee4a9700 15 client.37569.objecter _session_command_op_remove -1 12 2024-04-14T15:21:16.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.274+0000 7fafee4a9700 20 client.37569.objecter get_session s=0x7fafd4011340 osd=5 3 2024-04-14T15:21:16.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.274+0000 7fafee4a9700 15 client.37569.objecter _session_command_op_assign 5 12 2024-04-14T15:21:16.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.274+0000 7fafee4a9700 20 client.37569.objecter put_session s=0x7fafd4011340 osd=5 4 2024-04-14T15:21:16.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.274+0000 7fafee4a9700 10 client.37569.objecter _send_command 12 2024-04-14T15:21:16.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.274+0000 7fafee4a9700 1 -- 172.21.15.90:0/3877361749 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fafd401f180 con 0x7fafd40114e0 2024-04-14T15:21:16.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.276+0000 7fafe57fa700 1 -- 172.21.15.90:0/3877361749 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fafd401f180 con 0x7fafd40114e0 2024-04-14T15:21:16.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.276+0000 7fafe57fa700 10 client.37569.objecter ms_dispatch 0x7fafe8001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:21:16.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.276+0000 7fafe57fa700 10 client.37569.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:21:16.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.276+0000 7fafe57fa700 20 client.37569.objecter put_session s=0x7fafd4011340 osd=5 4 2024-04-14T15:21:16.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.276+0000 7fafe57fa700 15 client.37569.objecter _session_command_op_remove 5 12 2024-04-14T15:21:16.277 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:21:16.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.276+0000 7fafee4a9700 10 client.37569.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:21:16.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.276+0000 7fafee4a9700 15 client.37569.objecter _session_command_op_assign -1 13 2024-04-14T15:21:16.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.276+0000 7fafee4a9700 20 client.37569.objecter _get_session s=0x7fafd401f180 osd=6 3 2024-04-14T15:21:16.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.276+0000 7fafee4a9700 20 client.37569.objecter put_session s=0x7fafd401f180 osd=6 3 2024-04-14T15:21:16.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.276+0000 7fafee4a9700 20 client.37569.objecter _get_session s=0x7fafd401f180 osd=6 3 2024-04-14T15:21:16.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.276+0000 7fafee4a9700 15 client.37569.objecter _session_command_op_remove -1 13 2024-04-14T15:21:16.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.276+0000 7fafee4a9700 20 client.37569.objecter get_session s=0x7fafd401f180 osd=6 3 2024-04-14T15:21:16.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.276+0000 7fafee4a9700 15 client.37569.objecter _session_command_op_assign 6 13 2024-04-14T15:21:16.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.276+0000 7fafee4a9700 20 client.37569.objecter put_session s=0x7fafd401f180 osd=6 4 2024-04-14T15:21:16.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.276+0000 7fafee4a9700 10 client.37569.objecter _send_command 13 2024-04-14T15:21:16.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.276+0000 7fafee4a9700 1 -- 172.21.15.90:0/3877361749 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fafd40203f0 con 0x7fafd401f8a0 2024-04-14T15:21:16.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.277+0000 7fafe57fa700 10 client.37569.objecter ms_handle_connect 0x7fafd401f8a0 2024-04-14T15:21:16.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.278+0000 7fafe57fa700 1 -- 172.21.15.90:0/3877361749 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fafd40203f0 con 0x7fafd401f8a0 2024-04-14T15:21:16.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.278+0000 7fafe57fa700 10 client.37569.objecter ms_dispatch 0x7fafe8001550 command_reply(tid 13: 0 ) 2024-04-14T15:21:16.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.278+0000 7fafe57fa700 10 client.37569.objecter _finish_command 13 = system:0 2024-04-14T15:21:16.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.278+0000 7fafe57fa700 20 client.37569.objecter put_session s=0x7fafd401f180 osd=6 4 2024-04-14T15:21:16.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.278+0000 7fafe57fa700 15 client.37569.objecter _session_command_op_remove 6 13 2024-04-14T15:21:16.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.294+0000 7fafee4a9700 10 client.37569.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:21:16.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.294+0000 7fafee4a9700 15 client.37569.objecter _session_command_op_assign -1 14 2024-04-14T15:21:16.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.294+0000 7fafee4a9700 20 client.37569.objecter _get_session s=0x7fafd401f180 osd=6 3 2024-04-14T15:21:16.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.294+0000 7fafee4a9700 20 client.37569.objecter put_session s=0x7fafd401f180 osd=6 3 2024-04-14T15:21:16.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.294+0000 7fafee4a9700 20 client.37569.objecter _get_session s=0x7fafd401f180 osd=6 3 2024-04-14T15:21:16.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.294+0000 7fafee4a9700 15 client.37569.objecter _session_command_op_remove -1 14 2024-04-14T15:21:16.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.294+0000 7fafee4a9700 20 client.37569.objecter get_session s=0x7fafd401f180 osd=6 3 2024-04-14T15:21:16.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.294+0000 7fafee4a9700 15 client.37569.objecter _session_command_op_assign 6 14 2024-04-14T15:21:16.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.294+0000 7fafee4a9700 20 client.37569.objecter put_session s=0x7fafd401f180 osd=6 4 2024-04-14T15:21:16.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.294+0000 7fafee4a9700 10 client.37569.objecter _send_command 14 2024-04-14T15:21:16.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.294+0000 7fafee4a9700 1 -- 172.21.15.90:0/3877361749 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fafd40203f0 con 0x7fafd401f8a0 2024-04-14T15:21:16.295 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:21:16.296 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:21:16.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.296+0000 7fafe57fa700 1 -- 172.21.15.90:0/3877361749 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fafd40203f0 con 0x7fafd401f8a0 2024-04-14T15:21:16.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.296+0000 7fafe57fa700 10 client.37569.objecter ms_dispatch 0x7fafe8001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:21:16.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.296+0000 7fafe57fa700 10 client.37569.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:21:16.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.296+0000 7fafe57fa700 20 client.37569.objecter put_session s=0x7fafd401f180 osd=6 4 2024-04-14T15:21:16.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.296+0000 7fafe57fa700 15 client.37569.objecter _session_command_op_remove 6 14 2024-04-14T15:21:16.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.296+0000 7fafee4a9700 10 client.37569.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:21:16.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.296+0000 7fafee4a9700 15 client.37569.objecter _session_command_op_assign -1 15 2024-04-14T15:21:16.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.296+0000 7fafee4a9700 20 client.37569.objecter _get_session s=0x7fafd40203f0 osd=7 3 2024-04-14T15:21:16.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.296+0000 7fafee4a9700 20 client.37569.objecter put_session s=0x7fafd40203f0 osd=7 3 2024-04-14T15:21:16.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.296+0000 7fafee4a9700 20 client.37569.objecter _get_session s=0x7fafd40203f0 osd=7 3 2024-04-14T15:21:16.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.296+0000 7fafee4a9700 15 client.37569.objecter _session_command_op_remove -1 15 2024-04-14T15:21:16.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.296+0000 7fafee4a9700 20 client.37569.objecter get_session s=0x7fafd40203f0 osd=7 3 2024-04-14T15:21:16.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.296+0000 7fafee4a9700 15 client.37569.objecter _session_command_op_assign 7 15 2024-04-14T15:21:16.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.296+0000 7fafee4a9700 20 client.37569.objecter put_session s=0x7fafd40203f0 osd=7 4 2024-04-14T15:21:16.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.296+0000 7fafee4a9700 10 client.37569.objecter _send_command 15 2024-04-14T15:21:16.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.296+0000 7fafee4a9700 1 -- 172.21.15.90:0/3877361749 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fafd4027990 con 0x7fafd4023e80 2024-04-14T15:21:16.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.296+0000 7fafe57fa700 10 client.37569.objecter ms_handle_connect 0x7fafd4023e80 2024-04-14T15:21:16.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.297+0000 7fafe57fa700 1 -- 172.21.15.90:0/3877361749 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fafd4027990 con 0x7fafd4023e80 2024-04-14T15:21:16.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.297+0000 7fafe57fa700 10 client.37569.objecter ms_dispatch 0x7fafe8001550 command_reply(tid 15: 0 ) 2024-04-14T15:21:16.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.297+0000 7fafe57fa700 10 client.37569.objecter _finish_command 15 = system:0 2024-04-14T15:21:16.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.297+0000 7fafe57fa700 20 client.37569.objecter put_session s=0x7fafd40203f0 osd=7 4 2024-04-14T15:21:16.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.297+0000 7fafe57fa700 15 client.37569.objecter _session_command_op_remove 7 15 2024-04-14T15:21:16.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.314+0000 7fafee4a9700 10 client.37569.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:21:16.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.314+0000 7fafee4a9700 15 client.37569.objecter _session_command_op_assign -1 16 2024-04-14T15:21:16.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.314+0000 7fafee4a9700 20 client.37569.objecter _get_session s=0x7fafd40203f0 osd=7 3 2024-04-14T15:21:16.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.314+0000 7fafee4a9700 20 client.37569.objecter put_session s=0x7fafd40203f0 osd=7 3 2024-04-14T15:21:16.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.314+0000 7fafee4a9700 20 client.37569.objecter _get_session s=0x7fafd40203f0 osd=7 3 2024-04-14T15:21:16.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.314+0000 7fafee4a9700 15 client.37569.objecter _session_command_op_remove -1 16 2024-04-14T15:21:16.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.314+0000 7fafee4a9700 20 client.37569.objecter get_session s=0x7fafd40203f0 osd=7 3 2024-04-14T15:21:16.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.314+0000 7fafee4a9700 15 client.37569.objecter _session_command_op_assign 7 16 2024-04-14T15:21:16.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.314+0000 7fafee4a9700 20 client.37569.objecter put_session s=0x7fafd40203f0 osd=7 4 2024-04-14T15:21:16.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.314+0000 7fafee4a9700 10 client.37569.objecter _send_command 16 2024-04-14T15:21:16.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.314+0000 7fafee4a9700 1 -- 172.21.15.90:0/3877361749 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fafd4027990 con 0x7fafd4023e80 2024-04-14T15:21:16.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.316+0000 7fafe57fa700 1 -- 172.21.15.90:0/3877361749 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fafd4027990 con 0x7fafd4023e80 2024-04-14T15:21:16.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.316+0000 7fafe57fa700 10 client.37569.objecter ms_dispatch 0x7fafe8001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:21:16.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.316+0000 7fafe57fa700 10 client.37569.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:21:16.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.316+0000 7fafe57fa700 20 client.37569.objecter put_session s=0x7fafd40203f0 osd=7 4 2024-04-14T15:21:16.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.316+0000 7fafe57fa700 15 client.37569.objecter _session_command_op_remove 7 16 2024-04-14T15:21:16.317 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:21:16.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.316+0000 7fafee4a9700 10 librados: watch_flush enter 2024-04-14T15:21:16.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.316+0000 7fafee4a9700 10 librados: watch_flush exit 2024-04-14T15:21:16.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.316+0000 7fafee4a9700 10 client.37569.objecter close_session for osd.0 2024-04-14T15:21:16.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.316+0000 7fafee4a9700 1 -- 172.21.15.90:0/3877361749 >> v1:172.21.15.90:6801/3761806885 conn(0x7fafd4004730 legacy=0x7fafd4002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:16.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.316+0000 7fafee4a9700 20 client.37569.objecter put_session s=0x7fafd40053e0 osd=0 1 2024-04-14T15:21:16.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.316+0000 7fafee4a9700 10 client.37569.objecter close_session for osd.1 2024-04-14T15:21:16.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.316+0000 7fafee4a9700 1 -- 172.21.15.90:0/3877361749 >> v1:172.21.15.90:6805/4170361699 conn(0x7fafd4004280 legacy=0x7fafd400ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:16.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.316+0000 7fafee4a9700 20 client.37569.objecter put_session s=0x7fafd4004110 osd=1 1 2024-04-14T15:21:16.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.316+0000 7fafee4a9700 10 client.37569.objecter close_session for osd.2 2024-04-14T15:21:16.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.316+0000 7fafee4a9700 1 -- 172.21.15.90:0/3877361749 >> v1:172.21.15.90:6809/1686184251 conn(0x7fafd40088b0 legacy=0x7fafd400f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:16.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.316+0000 7fafeca46700 1 -- 172.21.15.90:0/3877361749 reap_dead start 2024-04-14T15:21:16.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.316+0000 7fafee4a9700 20 client.37569.objecter put_session s=0x7fafd4001a00 osd=2 1 2024-04-14T15:21:16.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.316+0000 7fafee4a9700 10 client.37569.objecter close_session for osd.3 2024-04-14T15:21:16.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.316+0000 7fafee4a9700 1 -- 172.21.15.90:0/3877361749 >> v1:172.21.15.90:6813/2270166774 conn(0x7fafd400ce70 legacy=0x7fafd4014900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:16.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.316+0000 7fafee4a9700 20 client.37569.objecter put_session s=0x7fafd4008170 osd=3 1 2024-04-14T15:21:16.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.316+0000 7fafee4a9700 10 client.37569.objecter close_session for osd.4 2024-04-14T15:21:16.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.316+0000 7fafee4a9700 1 -- 172.21.15.90:0/3877361749 >> v1:172.21.15.92:6800/3887549733 conn(0x7fafd4011dd0 legacy=0x7fafd40191b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:16.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.316+0000 7fafee4a9700 20 client.37569.objecter put_session s=0x7fafd400c700 osd=4 1 2024-04-14T15:21:16.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.316+0000 7fafee4a9700 10 client.37569.objecter close_session for osd.5 2024-04-14T15:21:16.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.316+0000 7fafee4a9700 1 -- 172.21.15.90:0/3877361749 >> v1:172.21.15.92:6804/3244561440 conn(0x7fafd40114e0 legacy=0x7fafd401da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:16.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.316+0000 7fafee4a9700 20 client.37569.objecter put_session s=0x7fafd4011340 osd=5 1 2024-04-14T15:21:16.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.316+0000 7fafee4a9700 10 client.37569.objecter close_session for osd.6 2024-04-14T15:21:16.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.316+0000 7fafee4a9700 1 -- 172.21.15.90:0/3877361749 >> v1:172.21.15.92:6808/636125963 conn(0x7fafd401f8a0 legacy=0x7fafd401fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:16.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.316+0000 7fafee4a9700 20 client.37569.objecter put_session s=0x7fafd401f180 osd=6 1 2024-04-14T15:21:16.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.316+0000 7fafee4a9700 10 client.37569.objecter close_session for osd.7 2024-04-14T15:21:16.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.316+0000 7fafee4a9700 1 -- 172.21.15.90:0/3877361749 >> v1:172.21.15.92:6812/1773706716 conn(0x7fafd4023e80 legacy=0x7fafd4026270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:16.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.316+0000 7fafee4a9700 20 client.37569.objecter put_session s=0x7fafd40203f0 osd=7 1 2024-04-14T15:21:16.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.316+0000 7fafee4a9700 20 client.37569.objecter shutdown clearing up homeless session... 2024-04-14T15:21:16.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.316+0000 7fafee4a9700 10 client.37569.objecter successfully canceled tick 2024-04-14T15:21:16.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.316+0000 7fafeca46700 1 -- 172.21.15.90:0/3877361749 reap_dead start 2024-04-14T15:21:16.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.316+0000 7fafee4a9700 1 -- 172.21.15.90:0/3877361749 >> v1:172.21.15.90:6800/420935427 conn(0x7fafd0071330 legacy=0x7fafd00737f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:16.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.316+0000 7fafee4a9700 1 -- 172.21.15.90:0/3877361749 >> v1:172.21.15.90:6789/0 conn(0x7fafe810f330 legacy=0x7fafe81a8c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:16.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.317+0000 7fafee4a9700 1 -- 172.21.15.90:0/3877361749 shutdown_connections 2024-04-14T15:21:16.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.317+0000 7fafee4a9700 1 -- 172.21.15.90:0/3877361749 >> 172.21.15.90:0/3877361749 conn(0x7fafe80fe3c0 msgr2=0x7fafe8112830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:21:16.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.317+0000 7fafee4a9700 1 -- 172.21.15.90:0/3877361749 shutdown_connections 2024-04-14T15:21:16.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.317+0000 7fafee4a9700 1 -- 172.21.15.90:0/3877361749 wait complete. 2024-04-14T15:21:16.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:16.317+0000 7fafee4a9700 1 librados: shutdown 2024-04-14T15:21:16.321 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:21:16.321 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:21:16.321 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:21:16.321 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:21:16.321 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:21:16.321 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:21:16.321 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:21:16.321 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:21:16.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:16.322 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:16.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:21:16.423 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:21:16.451 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:16.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:16.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:21:16.552 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:21:16.579 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:16.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:16.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:21:16.681 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:21:16.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:16.706 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:16.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:21:16.807 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:21:16.830 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:16.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:16.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:21:16.932 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:21:16.959 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:16.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:16.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:21:17.017 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:21:17.061 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:21:17.085 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:17.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:17.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:21:17.187 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:21:17.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:17.211 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:17.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:21:17.312 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:17.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:17.338 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:17.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:17.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:17 smithi092 ceph-mon[33966]: 11.11 scrub starts 2024-04-14T15:21:17.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:17 smithi092 ceph-mon[33966]: 11.11 scrub ok 2024-04-14T15:21:17.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:17 smithi092 ceph-mon[33966]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:21:17.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:17 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-94","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:21:17.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:17 smithi092 ceph-mon[33966]: osdmap e551: 8 total, 8 up, 7 in 2024-04-14T15:21:17.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:17 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-14T15:21:17.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:17 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3877361749' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:21:17.439 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:21:17.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:17 smithi090 ceph-mon[30399]: 11.11 scrub starts 2024-04-14T15:21:17.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:17 smithi090 ceph-mon[30399]: 11.11 scrub ok 2024-04-14T15:21:17.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:17 smithi090 ceph-mon[30399]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:21:17.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:17 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-94","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:21:17.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:17 smithi090 ceph-mon[30399]: osdmap e551: 8 total, 8 up, 7 in 2024-04-14T15:21:17.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:17 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-14T15:21:17.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:17 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3877361749' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:21:17.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:17 smithi090 ceph-mon[41670]: 11.11 scrub starts 2024-04-14T15:21:17.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:17 smithi090 ceph-mon[41670]: 11.11 scrub ok 2024-04-14T15:21:17.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:17 smithi090 ceph-mon[41670]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:21:17.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:17 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-94","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:21:17.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:17 smithi090 ceph-mon[41670]: osdmap e551: 8 total, 8 up, 7 in 2024-04-14T15:21:17.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:17 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-14T15:21:17.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:17 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3877361749' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:21:17.463 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:17.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:17.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:21:17.565 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:21:17.590 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:17.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:17.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:21:17.692 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:21:17.716 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:17.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:17.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:21:17.818 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:21:17.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:17.846 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:17.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:21:17.947 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:21:17.971 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:17.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:17.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:21:18.073 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:21:18.099 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:18.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:18.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:21:18.201 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:21:18.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:18.228 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:18.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:21:18.329 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:21:18.355 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:18.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:18.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:21:18.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:18 smithi092 ceph-mon[33966]: 11.1d scrub starts 2024-04-14T15:21:18.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:18 smithi092 ceph-mon[33966]: 11.1d scrub ok 2024-04-14T15:21:18.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:18 smithi092 ceph-mon[33966]: 110.10 scrub starts 2024-04-14T15:21:18.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:18 smithi092 ceph-mon[33966]: 110.10 scrub ok 2024-04-14T15:21:18.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:18 smithi092 ceph-mon[33966]: osdmap e552: 8 total, 8 up, 7 in 2024-04-14T15:21:18.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:18 smithi092 ceph-mon[33966]: pgmap v803: 250 pgs: 32 unknown, 1 active+remapped+backfill_toofull, 217 active+clean; 8.4 MiB data, 494 MiB used, 625 GiB / 626 GiB avail; 1/1188 objects misplaced (0.084%); 35 KiB/s, 0 objects/s recovering 2024-04-14T15:21:18.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:18 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-04-14T15:21:18.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:18 smithi092 ceph-mon[33966]: 3.1 scrub starts 2024-04-14T15:21:18.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:18 smithi092 ceph-mon[33966]: 3.1 scrub ok 2024-04-14T15:21:18.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:18 smithi092 ceph-mon[33966]: 110.18 scrub starts 2024-04-14T15:21:18.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:18 smithi092 ceph-mon[33966]: 110.18 scrub ok 2024-04-14T15:21:18.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:18 smithi090 ceph-mon[30399]: 11.1d scrub starts 2024-04-14T15:21:18.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:18 smithi090 ceph-mon[30399]: 11.1d scrub ok 2024-04-14T15:21:18.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:18 smithi090 ceph-mon[30399]: 110.10 scrub starts 2024-04-14T15:21:18.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:18 smithi090 ceph-mon[30399]: 110.10 scrub ok 2024-04-14T15:21:18.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:18 smithi090 ceph-mon[30399]: osdmap e552: 8 total, 8 up, 7 in 2024-04-14T15:21:18.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:18 smithi090 ceph-mon[30399]: pgmap v803: 250 pgs: 32 unknown, 1 active+remapped+backfill_toofull, 217 active+clean; 8.4 MiB data, 494 MiB used, 625 GiB / 626 GiB avail; 1/1188 objects misplaced (0.084%); 35 KiB/s, 0 objects/s recovering 2024-04-14T15:21:18.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:18 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-04-14T15:21:18.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:18 smithi090 ceph-mon[30399]: 3.1 scrub starts 2024-04-14T15:21:18.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:18 smithi090 ceph-mon[30399]: 3.1 scrub ok 2024-04-14T15:21:18.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:18 smithi090 ceph-mon[30399]: 110.18 scrub starts 2024-04-14T15:21:18.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:18 smithi090 ceph-mon[30399]: 110.18 scrub ok 2024-04-14T15:21:18.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:18 smithi090 ceph-mon[41670]: 11.1d scrub starts 2024-04-14T15:21:18.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:18 smithi090 ceph-mon[41670]: 11.1d scrub ok 2024-04-14T15:21:18.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:18 smithi090 ceph-mon[41670]: 110.10 scrub starts 2024-04-14T15:21:18.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:18 smithi090 ceph-mon[41670]: 110.10 scrub ok 2024-04-14T15:21:18.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:18 smithi090 ceph-mon[41670]: osdmap e552: 8 total, 8 up, 7 in 2024-04-14T15:21:18.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:18 smithi090 ceph-mon[41670]: pgmap v803: 250 pgs: 32 unknown, 1 active+remapped+backfill_toofull, 217 active+clean; 8.4 MiB data, 494 MiB used, 625 GiB / 626 GiB avail; 1/1188 objects misplaced (0.084%); 35 KiB/s, 0 objects/s recovering 2024-04-14T15:21:18.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:18 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-04-14T15:21:18.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:18 smithi090 ceph-mon[41670]: 3.1 scrub starts 2024-04-14T15:21:18.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:18 smithi090 ceph-mon[41670]: 3.1 scrub ok 2024-04-14T15:21:18.445 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:18 smithi090 ceph-mon[41670]: 110.18 scrub starts 2024-04-14T15:21:18.445 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:18 smithi090 ceph-mon[41670]: 110.18 scrub ok 2024-04-14T15:21:18.457 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:21:18.481 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:18.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:18.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:21:18.583 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:21:18.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:18.608 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:18.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:21:18.709 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:18.737 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:18.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:18.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:18.839 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:21:18.863 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:18.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:18.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:21:18.964 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:21:18.987 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:18.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:18.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:21:19.089 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:21:19.113 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:19.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:19.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:21:19.215 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:21:19.242 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:19.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:19.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:21:19.343 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:21:19.367 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:19.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:19.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:21:19.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:19 smithi092 ceph-mon[33966]: 11.12 scrub starts 2024-04-14T15:21:19.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:19 smithi092 ceph-mon[33966]: 11.12 scrub ok 2024-04-14T15:21:19.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:19 smithi092 ceph-mon[33966]: 2.c scrub starts 2024-04-14T15:21:19.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:19 smithi092 ceph-mon[33966]: 2.c scrub ok 2024-04-14T15:21:19.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:19 smithi092 ceph-mon[33966]: 6.c scrub starts 2024-04-14T15:21:19.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:19 smithi092 ceph-mon[33966]: 6.c scrub ok 2024-04-14T15:21:19.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:19 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "22"}]': finished 2024-04-14T15:21:19.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:19 smithi092 ceph-mon[33966]: osdmap e553: 8 total, 8 up, 7 in 2024-04-14T15:21:19.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:19 smithi090 ceph-mon[41670]: 11.12 scrub starts 2024-04-14T15:21:19.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:19 smithi090 ceph-mon[41670]: 11.12 scrub ok 2024-04-14T15:21:19.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:19 smithi090 ceph-mon[41670]: 2.c scrub starts 2024-04-14T15:21:19.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:19 smithi090 ceph-mon[41670]: 2.c scrub ok 2024-04-14T15:21:19.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:19 smithi090 ceph-mon[41670]: 6.c scrub starts 2024-04-14T15:21:19.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:19 smithi090 ceph-mon[41670]: 6.c scrub ok 2024-04-14T15:21:19.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:19 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "22"}]': finished 2024-04-14T15:21:19.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:19 smithi090 ceph-mon[41670]: osdmap e553: 8 total, 8 up, 7 in 2024-04-14T15:21:19.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:19 smithi090 ceph-mon[30399]: 11.12 scrub starts 2024-04-14T15:21:19.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:19 smithi090 ceph-mon[30399]: 11.12 scrub ok 2024-04-14T15:21:19.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:19 smithi090 ceph-mon[30399]: 2.c scrub starts 2024-04-14T15:21:19.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:19 smithi090 ceph-mon[30399]: 2.c scrub ok 2024-04-14T15:21:19.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:19 smithi090 ceph-mon[30399]: 6.c scrub starts 2024-04-14T15:21:19.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:19 smithi090 ceph-mon[30399]: 6.c scrub ok 2024-04-14T15:21:19.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:19 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "22"}]': finished 2024-04-14T15:21:19.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:19 smithi090 ceph-mon[30399]: osdmap e553: 8 total, 8 up, 7 in 2024-04-14T15:21:19.469 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:21:19.492 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:19.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:19.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:21:19.594 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:19.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:19.623 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:19.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:19.724 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:19.750 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:19.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:19.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:19.852 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:21:19.878 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:19.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:19.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:21:19.980 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:20.007 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:20.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:20.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:20.109 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:21:20.136 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:20.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:20.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:21:20.238 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:21:20.264 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:20.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:20.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:21:20.320 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:20 smithi090 ceph-mon[41670]: 110.1 scrub starts 2024-04-14T15:21:20.321 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:20 smithi090 ceph-mon[41670]: 110.1 scrub ok 2024-04-14T15:21:20.321 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:20 smithi090 ceph-mon[41670]: osdmap e554: 8 total, 8 up, 7 in 2024-04-14T15:21:20.321 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:20 smithi090 ceph-mon[41670]: pgmap v806: 250 pgs: 19 unknown, 1 active+remapped+backfill_toofull, 230 active+clean; 8.4 MiB data, 493 MiB used, 625 GiB / 626 GiB avail; 767 B/s rd, 1.5 KiB/s wr, 4 op/s; 1/1200 objects misplaced (0.083%) 2024-04-14T15:21:20.321 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:20 smithi090 ceph-mon[30399]: 110.1 scrub starts 2024-04-14T15:21:20.321 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:20 smithi090 ceph-mon[30399]: 110.1 scrub ok 2024-04-14T15:21:20.321 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:20 smithi090 ceph-mon[30399]: osdmap e554: 8 total, 8 up, 7 in 2024-04-14T15:21:20.321 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:20 smithi090 ceph-mon[30399]: pgmap v806: 250 pgs: 19 unknown, 1 active+remapped+backfill_toofull, 230 active+clean; 8.4 MiB data, 493 MiB used, 625 GiB / 626 GiB avail; 767 B/s rd, 1.5 KiB/s wr, 4 op/s; 1/1200 objects misplaced (0.083%) 2024-04-14T15:21:20.366 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:21:20.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:20 smithi092 ceph-mon[33966]: 110.1 scrub starts 2024-04-14T15:21:20.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:20 smithi092 ceph-mon[33966]: 110.1 scrub ok 2024-04-14T15:21:20.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:20 smithi092 ceph-mon[33966]: osdmap e554: 8 total, 8 up, 7 in 2024-04-14T15:21:20.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:20 smithi092 ceph-mon[33966]: pgmap v806: 250 pgs: 19 unknown, 1 active+remapped+backfill_toofull, 230 active+clean; 8.4 MiB data, 493 MiB used, 625 GiB / 626 GiB avail; 767 B/s rd, 1.5 KiB/s wr, 4 op/s; 1/1200 objects misplaced (0.083%) 2024-04-14T15:21:20.389 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:20.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:20.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:21:20.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.425+0000 7f18efda6700 1 -- 172.21.15.90:0/1762097889 >> v1:172.21.15.90:6789/0 conn(0x7f18e810e200 legacy=0x7f18e81106c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:20.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.425+0000 7f18efda6700 1 -- 172.21.15.90:0/1762097889 shutdown_connections 2024-04-14T15:21:20.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.425+0000 7f18efda6700 1 -- 172.21.15.90:0/1762097889 >> 172.21.15.90:0/1762097889 conn(0x7f18e80fd2b0 msgr2=0x7f18e80ff6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:21:20.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.425+0000 7f18efda6700 1 -- 172.21.15.90:0/1762097889 shutdown_connections 2024-04-14T15:21:20.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.425+0000 7f18efda6700 1 -- 172.21.15.90:0/1762097889 wait complete. 2024-04-14T15:21:20.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.425+0000 7f18efda6700 1 librados: starting msgr at 2024-04-14T15:21:20.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.425+0000 7f18efda6700 1 librados: starting objecter 2024-04-14T15:21:20.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.425+0000 7f18efda6700 1 Processor -- start 2024-04-14T15:21:20.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.425+0000 7f18efda6700 1 -- start start 2024-04-14T15:21:20.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.425+0000 7f18efda6700 1 librados: setting wanted keys 2024-04-14T15:21:20.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.425+0000 7f18efda6700 1 librados: calling monclient init 2024-04-14T15:21:20.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.426+0000 7f18efda6700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f18e80ff4d0 con 0x7f18e8107780 2024-04-14T15:21:20.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.426+0000 7f18efda6700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f18e81ab210 con 0x7f18e810e200 2024-04-14T15:21:20.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.426+0000 7f18efda6700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f18e81ac3f0 con 0x7f18e810a630 2024-04-14T15:21:20.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.426+0000 7f18edb42700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f18e8107780 0x7f18e80fd760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:36904/0 (socket says 172.21.15.90:36904) 2024-04-14T15:21:20.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.426+0000 7f18ed341700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f18e810a630 0x7f18e80fde70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:33154/0 (socket says 172.21.15.90:33154) 2024-04-14T15:21:20.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.426+0000 7f18edb42700 1 -- 172.21.15.90:0/3389928330 learned_addr learned my addr 172.21.15.90:0/3389928330 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:21:20.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.426+0000 7f18deffd700 10 client.?.objecter ms_handle_connect 0x7f18e810a630 2024-04-14T15:21:20.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.426+0000 7f18deffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:21:20.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.426+0000 7f18deffd700 10 client.?.objecter ms_handle_connect 0x7f18e8107780 2024-04-14T15:21:20.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.426+0000 7f18deffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:21:20.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.426+0000 7f18deffd700 10 client.?.objecter ms_handle_connect 0x7f18e810e200 2024-04-14T15:21:20.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.426+0000 7f18deffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:21:20.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.426+0000 7f18deffd700 1 -- 172.21.15.90:0/3389928330 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 600057981 0 0) 0x7f18e81ac3f0 con 0x7f18e810a630 2024-04-14T15:21:20.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.426+0000 7f18deffd700 1 -- 172.21.15.90:0/3389928330 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f18d40035b0 con 0x7f18e810a630 2024-04-14T15:21:20.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.426+0000 7f18deffd700 1 -- 172.21.15.90:0/3389928330 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2480804188 0 0) 0x7f18e80ff4d0 con 0x7f18e8107780 2024-04-14T15:21:20.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.426+0000 7f18deffd700 1 -- 172.21.15.90:0/3389928330 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f18e81ac3f0 con 0x7f18e8107780 2024-04-14T15:21:20.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.426+0000 7f18deffd700 1 -- 172.21.15.90:0/3389928330 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2804900112 0 0) 0x7f18e81ab210 con 0x7f18e810e200 2024-04-14T15:21:20.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.426+0000 7f18deffd700 1 -- 172.21.15.90:0/3389928330 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f18e80ff4d0 con 0x7f18e810e200 2024-04-14T15:21:20.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.426+0000 7f18deffd700 1 -- 172.21.15.90:0/3389928330 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3067646296 0 0) 0x7f18d40035b0 con 0x7f18e810a630 2024-04-14T15:21:20.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.426+0000 7f18deffd700 1 -- 172.21.15.90:0/3389928330 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f18e81ab210 con 0x7f18e810a630 2024-04-14T15:21:20.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.426+0000 7f18deffd700 1 -- 172.21.15.90:0/3389928330 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 631253172 0 0) 0x7f18e81ac3f0 con 0x7f18e8107780 2024-04-14T15:21:20.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.426+0000 7f18deffd700 1 -- 172.21.15.90:0/3389928330 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f18d40035b0 con 0x7f18e8107780 2024-04-14T15:21:20.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.426+0000 7f18deffd700 1 -- 172.21.15.90:0/3389928330 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f18d80018b0 con 0x7f18e810a630 2024-04-14T15:21:20.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.426+0000 7f18deffd700 1 -- 172.21.15.90:0/3389928330 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4085937676 0 0) 0x7f18e80ff4d0 con 0x7f18e810e200 2024-04-14T15:21:20.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.426+0000 7f18deffd700 1 -- 172.21.15.90:0/3389928330 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f18e81ac3f0 con 0x7f18e810e200 2024-04-14T15:21:20.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.426+0000 7f18deffd700 1 -- 172.21.15.90:0/3389928330 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f18e40018b0 con 0x7f18e8107780 2024-04-14T15:21:20.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.426+0000 7f18deffd700 1 -- 172.21.15.90:0/3389928330 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f18e00017b0 con 0x7f18e810e200 2024-04-14T15:21:20.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.427+0000 7f18deffd700 1 -- 172.21.15.90:0/3389928330 <== mon.2 v1:172.21.15.90:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2596197355 0 0) 0x7f18e81ab210 con 0x7f18e810a630 2024-04-14T15:21:20.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.427+0000 7f18deffd700 1 -- 172.21.15.90:0/3389928330 >> v1:172.21.15.92:6789/0 conn(0x7f18e810e200 legacy=0x7f18e81a8ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:20.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.427+0000 7f18deffd700 1 -- 172.21.15.90:0/3389928330 >> v1:172.21.15.90:6789/0 conn(0x7f18e8107780 legacy=0x7f18e80fd760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:20.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.427+0000 7f18deffd700 1 -- 172.21.15.90:0/3389928330 --> v1:172.21.15.90:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18e81ad5d0 con 0x7f18e810a630 2024-04-14T15:21:20.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.427+0000 7f18efda6700 1 -- 172.21.15.90:0/3389928330 --> v1:172.21.15.90:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f18e81aa200 con 0x7f18e810a630 2024-04-14T15:21:20.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.427+0000 7f18efda6700 10 client.47531.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:21:20.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.427+0000 7f18efda6700 1 -- 172.21.15.90:0/3389928330 --> v1:172.21.15.90:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f18e81aa6c0 con 0x7f18e810a630 2024-04-14T15:21:20.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.427+0000 7f18efda6700 1 librados: init done 2024-04-14T15:21:20.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.427+0000 7f18deffd700 1 -- 172.21.15.90:0/3389928330 <== mon.2 v1:172.21.15.90:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f18d8001c90 con 0x7f18e810a630 2024-04-14T15:21:20.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.427+0000 7f18deffd700 1 -- 172.21.15.90:0/3389928330 <== mon.2 v1:172.21.15.90:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f18d8005ca0 con 0x7f18e810a630 2024-04-14T15:21:20.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.428+0000 7f18efda6700 1 -- 172.21.15.90:0/3389928330 --> v1:172.21.15.90:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18cc005150 con 0x7f18e810a630 2024-04-14T15:21:20.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.428+0000 7f18deffd700 1 -- 172.21.15.90:0/3389928330 <== mon.2 v1:172.21.15.90:6790/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f18d8003950 con 0x7f18e810a630 2024-04-14T15:21:20.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.429+0000 7f18deffd700 1 -- 172.21.15.90:0/3389928330 <== mon.2 v1:172.21.15.90:6790/0 8 ==== osd_map(555..555 src has 467..555) ==== 8201+0+0 (unknown 3879273898 0 0) 0x7f18d808a9a0 con 0x7f18e810a630 2024-04-14T15:21:20.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.429+0000 7f18deffd700 10 client.47531.objecter ms_dispatch 0x7f18e8001550 osd_map(555..555 src has 467..555) 2024-04-14T15:21:20.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.429+0000 7f18deffd700 3 client.47531.objecter handle_osd_map got epochs [555,555] > 0 2024-04-14T15:21:20.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.429+0000 7f18deffd700 3 client.47531.objecter handle_osd_map decoding full epoch 555 2024-04-14T15:21:20.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.429+0000 7f18deffd700 20 client.47531.objecter dump_active .. 0 homeless 2024-04-14T15:21:20.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.429+0000 7f18deffd700 10 client.47531.objecter ms_handle_connect 0x7f18d4071310 2024-04-14T15:21:20.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.433+0000 7f18deffd700 1 -- 172.21.15.90:0/3389928330 <== mon.2 v1:172.21.15.90:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f18d80573a0 con 0x7f18e810a630 2024-04-14T15:21:20.491 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:21:20.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:20.515 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:20.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:21:20.571 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-14T15:21:20.617 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:20.644 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:20.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:20.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:20.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.657+0000 7f18efda6700 1 -- 172.21.15.90:0/3389928330 --> v1:172.21.15.90:6790/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7f18cc005440 con 0x7f18e810a630 2024-04-14T15:21:20.693 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-14T15:21:20.693 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-14T15:21:20.693 INFO:tasks.thrashosds.thrasher:inject_pause on osd.7 2024-04-14T15:21:20.693 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-14T15:21:20.693 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-14T15:21:20.694 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.7 config set filestore_inject_stall 3 2024-04-14T15:21:20.746 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:21:20.772 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:20.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:20.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:21:20.874 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:20.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:20.900 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:20.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:20.943 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:21:20.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.946+0000 7f18deffd700 1 -- 172.21.15.90:0/3389928330 <== mon.2 v1:172.21.15.90:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v556) ==== 101+0+0 (unknown 2184015217 0 0) 0x7f18d805b050 con 0x7f18e810a630 2024-04-14T15:21:20.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.950+0000 7f18efda6700 10 librados: watch_flush enter 2024-04-14T15:21:20.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.950+0000 7f18efda6700 10 librados: watch_flush exit 2024-04-14T15:21:20.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.950+0000 7f18efda6700 20 client.47531.objecter shutdown clearing up homeless session... 2024-04-14T15:21:20.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.950+0000 7f18efda6700 10 client.47531.objecter successfully canceled tick 2024-04-14T15:21:20.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.950+0000 7f18efda6700 1 -- 172.21.15.90:0/3389928330 >> v1:172.21.15.90:6800/420935427 conn(0x7f18d4071310 legacy=0x7f18d40737d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:20.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.950+0000 7f18efda6700 1 -- 172.21.15.90:0/3389928330 >> v1:172.21.15.90:6790/0 conn(0x7f18e810a630 legacy=0x7f18e80fde70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:20.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.950+0000 7f18efda6700 1 -- 172.21.15.90:0/3389928330 shutdown_connections 2024-04-14T15:21:20.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.950+0000 7f18efda6700 1 -- 172.21.15.90:0/3389928330 >> 172.21.15.90:0/3389928330 conn(0x7f18e80fd2b0 msgr2=0x7f18e8111120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:21:20.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.952+0000 7f18efda6700 1 -- 172.21.15.90:0/3389928330 shutdown_connections 2024-04-14T15:21:20.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.952+0000 7f18efda6700 1 -- 172.21.15.90:0/3389928330 wait complete. 2024-04-14T15:21:20.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:20.952+0000 7f18efda6700 1 librados: shutdown 2024-04-14T15:21:20.953 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is unset 2024-04-14T15:21:21.001 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:21:21.024 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:21.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:21.025 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:21:21.025 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-14T15:21:21.025 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:21:21.126 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:21:21.149 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:21.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:21.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:21:21.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:21 smithi090 ceph-mon[30399]: 11.1b scrub starts 2024-04-14T15:21:21.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:21 smithi090 ceph-mon[30399]: 11.1b scrub ok 2024-04-14T15:21:21.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:21 smithi090 ceph-mon[30399]: 4.10 scrub starts 2024-04-14T15:21:21.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:21 smithi090 ceph-mon[30399]: 4.10 scrub ok 2024-04-14T15:21:21.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:21 smithi090 ceph-mon[30399]: osdmap e555: 8 total, 8 up, 7 in 2024-04-14T15:21:21.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:21 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3389928330' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:21:21.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:21 smithi090 ceph-mon[30399]: from='client.47531 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:21:21.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:21 smithi090 ceph-mon[30399]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-14T15:21:21.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:21 smithi090 ceph-mon[30399]: from='client.47531 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-14T15:21:21.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:21 smithi090 ceph-mon[30399]: osdmap e556: 8 total, 8 up, 7 in 2024-04-14T15:21:21.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:21 smithi090 ceph-mon[41670]: 11.1b scrub starts 2024-04-14T15:21:21.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:21 smithi090 ceph-mon[41670]: 11.1b scrub ok 2024-04-14T15:21:21.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:21 smithi090 ceph-mon[41670]: 4.10 scrub starts 2024-04-14T15:21:21.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:21 smithi090 ceph-mon[41670]: 4.10 scrub ok 2024-04-14T15:21:21.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:21 smithi090 ceph-mon[41670]: osdmap e555: 8 total, 8 up, 7 in 2024-04-14T15:21:21.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:21 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3389928330' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:21:21.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:21 smithi090 ceph-mon[41670]: from='client.47531 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:21:21.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:21 smithi090 ceph-mon[41670]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-14T15:21:21.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:21 smithi090 ceph-mon[41670]: from='client.47531 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-14T15:21:21.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:21 smithi090 ceph-mon[41670]: osdmap e556: 8 total, 8 up, 7 in 2024-04-14T15:21:21.251 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:21:21.275 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:21.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:21.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:21:21.371 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:21 smithi092 ceph-mon[33966]: 11.1b scrub starts 2024-04-14T15:21:21.371 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:21 smithi092 ceph-mon[33966]: 11.1b scrub ok 2024-04-14T15:21:21.371 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:21 smithi092 ceph-mon[33966]: 4.10 scrub starts 2024-04-14T15:21:21.371 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:21 smithi092 ceph-mon[33966]: 4.10 scrub ok 2024-04-14T15:21:21.371 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:21 smithi092 ceph-mon[33966]: osdmap e555: 8 total, 8 up, 7 in 2024-04-14T15:21:21.371 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:21 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3389928330' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:21:21.371 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:21 smithi092 ceph-mon[33966]: from='client.47531 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:21:21.371 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:21 smithi092 ceph-mon[33966]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-14T15:21:21.371 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:21 smithi092 ceph-mon[33966]: from='client.47531 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-14T15:21:21.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:21 smithi092 ceph-mon[33966]: osdmap e556: 8 total, 8 up, 7 in 2024-04-14T15:21:21.377 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:21:21.401 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:21.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:21.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:21:21.503 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:21:21.526 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:21.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:21.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:21:21.627 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:21:21.649 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:21.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:21.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:21:21.751 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:21.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:21.775 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:21.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:21.876 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:21:21.901 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:21.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:21.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:21:22.002 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:21:22.026 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:22.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:22.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:21:22.128 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:21:22.153 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-14T15:21:22.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:22.154 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:22.155 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:21:22.256 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:21:22.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:22.281 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:22.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:21:22.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:22 smithi092 ceph-mon[33966]: 110.2 scrub starts 2024-04-14T15:21:22.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:22 smithi092 ceph-mon[33966]: 110.2 scrub ok 2024-04-14T15:21:22.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:22 smithi092 ceph-mon[33966]: 11.5 scrub starts 2024-04-14T15:21:22.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:22 smithi092 ceph-mon[33966]: 11.5 scrub ok 2024-04-14T15:21:22.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:22 smithi092 ceph-mon[33966]: 4.1c scrub starts 2024-04-14T15:21:22.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:22 smithi092 ceph-mon[33966]: 4.1c scrub ok 2024-04-14T15:21:22.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:22 smithi092 ceph-mon[33966]: 110.0 scrub starts 2024-04-14T15:21:22.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:22 smithi092 ceph-mon[33966]: 110.0 scrub ok 2024-04-14T15:21:22.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:22 smithi092 ceph-mon[33966]: 6.6 scrub starts 2024-04-14T15:21:22.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:22 smithi092 ceph-mon[33966]: 6.6 scrub ok 2024-04-14T15:21:22.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:22 smithi092 ceph-mon[33966]: pgmap v809: 249 pgs: 1 active+remapped+backfill_toofull, 1 clean+premerge+peered, 247 active+clean; 8.4 MiB data, 493 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 2.0 KiB/s wr, 7 op/s; 1/1206 objects misplaced (0.083%); 0 B/s, 1 objects/s recovering 2024-04-14T15:21:22.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:22 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-04-14T15:21:22.383 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:22.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:22.408 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:22.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:22.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:22 smithi090 ceph-mon[41670]: 110.2 scrub starts 2024-04-14T15:21:22.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:22 smithi090 ceph-mon[41670]: 110.2 scrub ok 2024-04-14T15:21:22.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:22 smithi090 ceph-mon[41670]: 11.5 scrub starts 2024-04-14T15:21:22.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:22 smithi090 ceph-mon[41670]: 11.5 scrub ok 2024-04-14T15:21:22.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:22 smithi090 ceph-mon[41670]: 4.1c scrub starts 2024-04-14T15:21:22.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:22 smithi090 ceph-mon[41670]: 4.1c scrub ok 2024-04-14T15:21:22.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:22 smithi090 ceph-mon[41670]: 110.0 scrub starts 2024-04-14T15:21:22.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:22 smithi090 ceph-mon[41670]: 110.0 scrub ok 2024-04-14T15:21:22.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:22 smithi090 ceph-mon[41670]: 6.6 scrub starts 2024-04-14T15:21:22.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:22 smithi090 ceph-mon[41670]: 6.6 scrub ok 2024-04-14T15:21:22.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:22 smithi090 ceph-mon[41670]: pgmap v809: 249 pgs: 1 active+remapped+backfill_toofull, 1 clean+premerge+peered, 247 active+clean; 8.4 MiB data, 493 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 2.0 KiB/s wr, 7 op/s; 1/1206 objects misplaced (0.083%); 0 B/s, 1 objects/s recovering 2024-04-14T15:21:22.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:22 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-04-14T15:21:22.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:22 smithi090 ceph-mon[30399]: 110.2 scrub starts 2024-04-14T15:21:22.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:22 smithi090 ceph-mon[30399]: 110.2 scrub ok 2024-04-14T15:21:22.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:22 smithi090 ceph-mon[30399]: 11.5 scrub starts 2024-04-14T15:21:22.445 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:22 smithi090 ceph-mon[30399]: 11.5 scrub ok 2024-04-14T15:21:22.445 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:22 smithi090 ceph-mon[30399]: 4.1c scrub starts 2024-04-14T15:21:22.445 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:22 smithi090 ceph-mon[30399]: 4.1c scrub ok 2024-04-14T15:21:22.445 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:22 smithi090 ceph-mon[30399]: 110.0 scrub starts 2024-04-14T15:21:22.445 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:22 smithi090 ceph-mon[30399]: 110.0 scrub ok 2024-04-14T15:21:22.445 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:22 smithi090 ceph-mon[30399]: 6.6 scrub starts 2024-04-14T15:21:22.446 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:22 smithi090 ceph-mon[30399]: 6.6 scrub ok 2024-04-14T15:21:22.446 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:22 smithi090 ceph-mon[30399]: pgmap v809: 249 pgs: 1 active+remapped+backfill_toofull, 1 clean+premerge+peered, 247 active+clean; 8.4 MiB data, 493 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 2.0 KiB/s wr, 7 op/s; 1/1206 objects misplaced (0.083%); 0 B/s, 1 objects/s recovering 2024-04-14T15:21:22.446 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:22 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-04-14T15:21:22.510 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:21:22.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:22.536 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:22.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:21:22.586 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:21:22.637 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:21:22.661 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:22.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:22.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:21:22.763 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:22.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:22.787 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:22.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:22.888 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:22.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:22.913 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:22.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:23.014 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:21:23.023 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-04-14T15:21:23.023 INFO:teuthology.orchestra.run.smithi092.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-14T15:21:23.023 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-04-14T15:21:23.039 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:23.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:23.040 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:21:23.141 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:21:23.165 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:23.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:23.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:21:23.267 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:21:23.285 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:23 smithi092 ceph-mon[33966]: 11.17 scrub starts 2024-04-14T15:21:23.285 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:23 smithi092 ceph-mon[33966]: 11.17 scrub ok 2024-04-14T15:21:23.285 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:23 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-04-14T15:21:23.285 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:23 smithi092 ceph-mon[33966]: osdmap e557: 8 total, 8 up, 7 in 2024-04-14T15:21:23.285 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:23 smithi092 ceph-mon[33966]: 4.1a scrub starts 2024-04-14T15:21:23.285 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:23 smithi092 ceph-mon[33966]: 4.1a scrub ok 2024-04-14T15:21:23.285 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:23 smithi092 ceph-mon[33966]: 6.1b deep-scrub starts 2024-04-14T15:21:23.285 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:23 smithi092 ceph-mon[33966]: 6.1b deep-scrub ok 2024-04-14T15:21:23.290 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:23.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:23.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:21:23.392 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:23.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:23.415 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:23.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:23.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:23 smithi090 ceph-mon[30399]: 11.17 scrub starts 2024-04-14T15:21:23.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:23 smithi090 ceph-mon[30399]: 11.17 scrub ok 2024-04-14T15:21:23.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:23 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-04-14T15:21:23.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:23 smithi090 ceph-mon[30399]: osdmap e557: 8 total, 8 up, 7 in 2024-04-14T15:21:23.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:23 smithi090 ceph-mon[30399]: 4.1a scrub starts 2024-04-14T15:21:23.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:23 smithi090 ceph-mon[30399]: 4.1a scrub ok 2024-04-14T15:21:23.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:23 smithi090 ceph-mon[30399]: 6.1b deep-scrub starts 2024-04-14T15:21:23.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:23 smithi090 ceph-mon[30399]: 6.1b deep-scrub ok 2024-04-14T15:21:23.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:23 smithi090 ceph-mon[41670]: 11.17 scrub starts 2024-04-14T15:21:23.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:23 smithi090 ceph-mon[41670]: 11.17 scrub ok 2024-04-14T15:21:23.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:23 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-04-14T15:21:23.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:23 smithi090 ceph-mon[41670]: osdmap e557: 8 total, 8 up, 7 in 2024-04-14T15:21:23.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:23 smithi090 ceph-mon[41670]: 4.1a scrub starts 2024-04-14T15:21:23.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:23 smithi090 ceph-mon[41670]: 4.1a scrub ok 2024-04-14T15:21:23.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:23 smithi090 ceph-mon[41670]: 6.1b deep-scrub starts 2024-04-14T15:21:23.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:23 smithi090 ceph-mon[41670]: 6.1b deep-scrub ok 2024-04-14T15:21:23.516 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:21:23.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:23.542 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:23.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:21:23.643 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:21:23.669 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:23.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:23.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:21:23.771 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:21:23.795 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:23.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:23.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:21:23.896 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:21:23.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:23.920 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:23.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:21:24.021 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:21:24.045 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:24.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:24.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:21:24.147 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:21:24.171 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:24.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:24.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:21:24.191 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:21:23 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:21:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:21:24.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:24 smithi090 ceph-mon[41670]: osdmap e558: 8 total, 8 up, 7 in 2024-04-14T15:21:24.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:24 smithi090 ceph-mon[41670]: pgmap v812: 249 pgs: 1 active+remapped+backfill_toofull, 1 clean+premerge+peered, 247 active+clean; 8.4 MiB data, 493 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 3 op/s; 1/1206 objects misplaced (0.083%); 0 B/s, 1 objects/s recovering 2024-04-14T15:21:24.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:24 smithi090 ceph-mon[41670]: 5.13 scrub starts 2024-04-14T15:21:24.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:24 smithi090 ceph-mon[41670]: 5.13 scrub ok 2024-04-14T15:21:24.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:24 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:21:24.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:24 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:21:24.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:24 smithi090 ceph-mon[30399]: osdmap e558: 8 total, 8 up, 7 in 2024-04-14T15:21:24.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:24 smithi090 ceph-mon[30399]: pgmap v812: 249 pgs: 1 active+remapped+backfill_toofull, 1 clean+premerge+peered, 247 active+clean; 8.4 MiB data, 493 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 3 op/s; 1/1206 objects misplaced (0.083%); 0 B/s, 1 objects/s recovering 2024-04-14T15:21:24.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:24 smithi090 ceph-mon[30399]: 5.13 scrub starts 2024-04-14T15:21:24.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:24 smithi090 ceph-mon[30399]: 5.13 scrub ok 2024-04-14T15:21:24.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:24 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:21:24.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:24 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:21:24.274 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:21:24.298 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:24.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:24.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:21:24.304 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:21:24.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:24 smithi092 ceph-mon[33966]: osdmap e558: 8 total, 8 up, 7 in 2024-04-14T15:21:24.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:24 smithi092 ceph-mon[33966]: pgmap v812: 249 pgs: 1 active+remapped+backfill_toofull, 1 clean+premerge+peered, 247 active+clean; 8.4 MiB data, 493 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 3 op/s; 1/1206 objects misplaced (0.083%); 0 B/s, 1 objects/s recovering 2024-04-14T15:21:24.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:24 smithi092 ceph-mon[33966]: 5.13 scrub starts 2024-04-14T15:21:24.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:24 smithi092 ceph-mon[33966]: 5.13 scrub ok 2024-04-14T15:21:24.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:24 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:21:24.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:24 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:21:24.401 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:21:24.425 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:24.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:24.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:21:24.527 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:21:24.551 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:24.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:24.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:21:24.653 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:21:24.677 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:24.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:24.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:21:24.779 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:21:24.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:24.803 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:24.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:21:24.904 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:21:24.929 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:24.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:24.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:21:25.030 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:21:25.054 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:25.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:25.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:21:25.155 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:21:25.179 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:25.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:25.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:21:25.280 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:21:25.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:25.304 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:25.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:21:25.372 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:21:25 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:21:25.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:25 smithi092 ceph-mon[33966]: 5.b scrub starts 2024-04-14T15:21:25.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:25 smithi092 ceph-mon[33966]: 5.b scrub ok 2024-04-14T15:21:25.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:25 smithi092 ceph-mon[33966]: osdmap e559: 8 total, 8 up, 7 in 2024-04-14T15:21:25.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:25 smithi092 ceph-mon[33966]: 11.1a scrub starts 2024-04-14T15:21:25.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:25 smithi092 ceph-mon[33966]: 11.1a scrub ok 2024-04-14T15:21:25.385 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:25 smithi090 ceph-mon[41670]: 5.b scrub starts 2024-04-14T15:21:25.385 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:25 smithi090 ceph-mon[41670]: 5.b scrub ok 2024-04-14T15:21:25.385 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:25 smithi090 ceph-mon[41670]: osdmap e559: 8 total, 8 up, 7 in 2024-04-14T15:21:25.385 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:25 smithi090 ceph-mon[41670]: 11.1a scrub starts 2024-04-14T15:21:25.385 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:25 smithi090 ceph-mon[41670]: 11.1a scrub ok 2024-04-14T15:21:25.385 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:25 smithi090 ceph-mon[30399]: 5.b scrub starts 2024-04-14T15:21:25.386 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:25 smithi090 ceph-mon[30399]: 5.b scrub ok 2024-04-14T15:21:25.386 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:25 smithi090 ceph-mon[30399]: osdmap e559: 8 total, 8 up, 7 in 2024-04-14T15:21:25.386 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:25 smithi090 ceph-mon[30399]: 11.1a scrub starts 2024-04-14T15:21:25.386 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:25 smithi090 ceph-mon[30399]: 11.1a scrub ok 2024-04-14T15:21:25.405 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:21:25.429 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:25.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:25.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:21:25.530 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:21:25.554 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:25.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:25.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:21:25.657 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:21:25.658 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd set noscrub 2024-04-14T15:21:25.683 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:25.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:25.684 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:21:25.785 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:21:25.808 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:25.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:25.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:21:25.911 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:25.933 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:21:25.935 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:25.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:25.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:26.036 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:21:26.060 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:26.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:26.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:21:26.162 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:21:26.185 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:26.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:26.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:21:26.286 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:21:26.310 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:26.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:26.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:21:26.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:26 smithi092 ceph-mon[33966]: 11.4 scrub starts 2024-04-14T15:21:26.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:26 smithi092 ceph-mon[33966]: 11.4 scrub ok 2024-04-14T15:21:26.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:26 smithi092 ceph-mon[33966]: osdmap e560: 8 total, 8 up, 7 in 2024-04-14T15:21:26.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:26 smithi092 ceph-mon[33966]: pgmap v815: 248 pgs: 1 peering, 1 clean+premerge+peered, 246 active+clean; 8.4 MiB data, 493 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 80 KiB/s, 0 objects/s recovering 2024-04-14T15:21:26.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:26 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:21:26.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:26 smithi092 ceph-mon[33966]: 6.10 scrub starts 2024-04-14T15:21:26.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:26 smithi092 ceph-mon[33966]: 6.10 scrub ok 2024-04-14T15:21:26.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:26 smithi092 ceph-mon[33966]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:21:26.412 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:21:26.435 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:26.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:26.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:21:26.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:26 smithi090 ceph-mon[30399]: 11.4 scrub starts 2024-04-14T15:21:26.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:26 smithi090 ceph-mon[30399]: 11.4 scrub ok 2024-04-14T15:21:26.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:26 smithi090 ceph-mon[30399]: osdmap e560: 8 total, 8 up, 7 in 2024-04-14T15:21:26.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:26 smithi090 ceph-mon[30399]: pgmap v815: 248 pgs: 1 peering, 1 clean+premerge+peered, 246 active+clean; 8.4 MiB data, 493 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 80 KiB/s, 0 objects/s recovering 2024-04-14T15:21:26.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:26 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:21:26.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:26 smithi090 ceph-mon[30399]: 6.10 scrub starts 2024-04-14T15:21:26.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:26 smithi090 ceph-mon[30399]: 6.10 scrub ok 2024-04-14T15:21:26.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:26 smithi090 ceph-mon[30399]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:21:26.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:26 smithi090 ceph-mon[41670]: 11.4 scrub starts 2024-04-14T15:21:26.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:26 smithi090 ceph-mon[41670]: 11.4 scrub ok 2024-04-14T15:21:26.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:26 smithi090 ceph-mon[41670]: osdmap e560: 8 total, 8 up, 7 in 2024-04-14T15:21:26.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:26 smithi090 ceph-mon[41670]: pgmap v815: 248 pgs: 1 peering, 1 clean+premerge+peered, 246 active+clean; 8.4 MiB data, 493 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 80 KiB/s, 0 objects/s recovering 2024-04-14T15:21:26.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:26 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:21:26.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:26 smithi090 ceph-mon[41670]: 6.10 scrub starts 2024-04-14T15:21:26.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:26 smithi090 ceph-mon[41670]: 6.10 scrub ok 2024-04-14T15:21:26.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:26 smithi090 ceph-mon[41670]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:21:26.537 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:21:26.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:26.561 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:26.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:21:26.662 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:26.687 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:26.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:26.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:26.788 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:21:26.812 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:26.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:26.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:21:26.915 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:21:26.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:26.940 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:26.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:21:27.041 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:21:27.065 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:27.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:27.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:21:27.164 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:27 smithi090 ceph-mon[41670]: 2.6 scrub starts 2024-04-14T15:21:27.164 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:27 smithi090 ceph-mon[41670]: 2.6 scrub ok 2024-04-14T15:21:27.164 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:27 smithi090 ceph-mon[41670]: 4.12 scrub starts 2024-04-14T15:21:27.164 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:27 smithi090 ceph-mon[41670]: 4.12 scrub ok 2024-04-14T15:21:27.164 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:27 smithi090 ceph-mon[41670]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-04-14T15:21:27.164 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:27 smithi090 ceph-mon[41670]: 11.6 scrub starts 2024-04-14T15:21:27.165 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:27 smithi090 ceph-mon[41670]: 11.6 scrub ok 2024-04-14T15:21:27.165 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:27 smithi090 ceph-mon[30399]: 2.6 scrub starts 2024-04-14T15:21:27.165 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:27 smithi090 ceph-mon[30399]: 2.6 scrub ok 2024-04-14T15:21:27.165 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:27 smithi090 ceph-mon[30399]: 4.12 scrub starts 2024-04-14T15:21:27.165 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:27 smithi090 ceph-mon[30399]: 4.12 scrub ok 2024-04-14T15:21:27.165 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:27 smithi090 ceph-mon[30399]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-04-14T15:21:27.165 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:27 smithi090 ceph-mon[30399]: 11.6 scrub starts 2024-04-14T15:21:27.165 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:27 smithi090 ceph-mon[30399]: 11.6 scrub ok 2024-04-14T15:21:27.168 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:21:27.191 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:27.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:27.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:21:27.293 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:21:27.317 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:27.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:27.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:21:27.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:27 smithi092 ceph-mon[33966]: 2.6 scrub starts 2024-04-14T15:21:27.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:27 smithi092 ceph-mon[33966]: 2.6 scrub ok 2024-04-14T15:21:27.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:27 smithi092 ceph-mon[33966]: 4.12 scrub starts 2024-04-14T15:21:27.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:27 smithi092 ceph-mon[33966]: 4.12 scrub ok 2024-04-14T15:21:27.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:27 smithi092 ceph-mon[33966]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-04-14T15:21:27.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:27 smithi092 ceph-mon[33966]: 11.6 scrub starts 2024-04-14T15:21:27.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:27 smithi092 ceph-mon[33966]: 11.6 scrub ok 2024-04-14T15:21:27.419 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:21:27.443 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:27.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:27.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:21:27.546 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:21:27.569 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:27.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:27.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:21:27.671 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:21:27.698 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:27.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:27.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:21:27.800 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:21:27.824 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:27.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:27.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:21:27.925 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:21:27.947 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-14T15:21:27.947 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:21:27.949 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:27.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:27.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:21:28.051 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:28.077 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:28.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:28.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:28.179 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:28.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:28 smithi090 ceph-mon[30399]: pgmap v816: 248 pgs: 1 peering, 1 clean+premerge+peered, 246 active+clean; 8.4 MiB data, 493 MiB used, 625 GiB / 626 GiB avail; 1011 B/s rd, 0 op/s; 63 KiB/s, 0 objects/s recovering 2024-04-14T15:21:28.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:28 smithi090 ceph-mon[30399]: 110.8 deep-scrub starts 2024-04-14T15:21:28.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:28 smithi090 ceph-mon[30399]: 110.8 deep-scrub ok 2024-04-14T15:21:28.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:28 smithi090 ceph-mon[41670]: pgmap v816: 248 pgs: 1 peering, 1 clean+premerge+peered, 246 active+clean; 8.4 MiB data, 493 MiB used, 625 GiB / 626 GiB avail; 1011 B/s rd, 0 op/s; 63 KiB/s, 0 objects/s recovering 2024-04-14T15:21:28.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:28 smithi090 ceph-mon[41670]: 110.8 deep-scrub starts 2024-04-14T15:21:28.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:28 smithi090 ceph-mon[41670]: 110.8 deep-scrub ok 2024-04-14T15:21:28.204 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:28.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:28.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:28.306 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:21:28.329 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:28.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:28.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:21:28.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:28 smithi092 ceph-mon[33966]: pgmap v816: 248 pgs: 1 peering, 1 clean+premerge+peered, 246 active+clean; 8.4 MiB data, 493 MiB used, 625 GiB / 626 GiB avail; 1011 B/s rd, 0 op/s; 63 KiB/s, 0 objects/s recovering 2024-04-14T15:21:28.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:28 smithi092 ceph-mon[33966]: 110.8 deep-scrub starts 2024-04-14T15:21:28.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:28 smithi092 ceph-mon[33966]: 110.8 deep-scrub ok 2024-04-14T15:21:28.430 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:21:28.454 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:28.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:28.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:21:28.556 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:21:28.579 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:28.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:28.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:21:28.671 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-14T15:21:28.671 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-14T15:21:28.672 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-04-14T15:21:28.672 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-04-14T15:21:28.672 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph pg dump --format=json 2024-04-14T15:21:28.681 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:21:28.738 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:28.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:28.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:21:28.840 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:21:28.868 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:28.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:28.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:21:28.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.939+0000 7fdafbf3b700 1 -- 172.21.15.90:0/51300273 >> v1:172.21.15.90:6789/0 conn(0x7fdaf4102260 legacy=0x7fdaf410fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:28.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.939+0000 7fdafbf3b700 1 -- 172.21.15.90:0/51300273 shutdown_connections 2024-04-14T15:21:28.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.939+0000 7fdafbf3b700 1 -- 172.21.15.90:0/51300273 >> 172.21.15.90:0/51300273 conn(0x7fdaf40fd2a0 msgr2=0x7fdaf40ff6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:21:28.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.939+0000 7fdafbf3b700 1 -- 172.21.15.90:0/51300273 shutdown_connections 2024-04-14T15:21:28.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.939+0000 7fdafbf3b700 1 -- 172.21.15.90:0/51300273 wait complete. 2024-04-14T15:21:28.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.940+0000 7fdafbf3b700 1 librados: starting msgr at 2024-04-14T15:21:28.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.940+0000 7fdafbf3b700 1 librados: starting objecter 2024-04-14T15:21:28.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.940+0000 7fdafbf3b700 1 Processor -- start 2024-04-14T15:21:28.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.940+0000 7fdafbf3b700 1 -- start start 2024-04-14T15:21:28.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.940+0000 7fdafbf3b700 1 librados: setting wanted keys 2024-04-14T15:21:28.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.940+0000 7fdafbf3b700 1 librados: calling monclient init 2024-04-14T15:21:28.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.940+0000 7fdafbf3b700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdaf4110010 con 0x7fdaf4102260 2024-04-14T15:21:28.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.940+0000 7fdafbf3b700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdaf4110220 con 0x7fdaf4101680 2024-04-14T15:21:28.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.940+0000 7fdafbf3b700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdaf41ac170 con 0x7fdaf4111550 2024-04-14T15:21:28.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.940+0000 7fdaf94d6700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fdaf4102260 0x7fdaf410d570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:42636/0 (socket says 172.21.15.90:42636) 2024-04-14T15:21:28.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.940+0000 7fdaf94d6700 1 -- 172.21.15.90:0/3239286385 learned_addr learned my addr 172.21.15.90:0/3239286385 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:21:28.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.940+0000 7fdaeaffd700 10 client.?.objecter ms_handle_connect 0x7fdaf4102260 2024-04-14T15:21:28.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.940+0000 7fdaeaffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:21:28.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.940+0000 7fdaeaffd700 10 client.?.objecter ms_handle_connect 0x7fdaf4111550 2024-04-14T15:21:28.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.940+0000 7fdaeaffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:21:28.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.940+0000 7fdaeaffd700 10 client.?.objecter ms_handle_connect 0x7fdaf4101680 2024-04-14T15:21:28.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.940+0000 7fdaeaffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:21:28.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.941+0000 7fdaeaffd700 1 -- 172.21.15.90:0/3239286385 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2004992925 0 0) 0x7fdaf4110010 con 0x7fdaf4102260 2024-04-14T15:21:28.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.941+0000 7fdaeaffd700 1 -- 172.21.15.90:0/3239286385 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdae00035b0 con 0x7fdaf4102260 2024-04-14T15:21:28.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.941+0000 7fdaeaffd700 1 -- 172.21.15.90:0/3239286385 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 889865952 0 0) 0x7fdaf4110220 con 0x7fdaf4101680 2024-04-14T15:21:28.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.941+0000 7fdaeaffd700 1 -- 172.21.15.90:0/3239286385 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdaf4110010 con 0x7fdaf4101680 2024-04-14T15:21:28.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.941+0000 7fdaeaffd700 1 -- 172.21.15.90:0/3239286385 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4021902268 0 0) 0x7fdaf41ac170 con 0x7fdaf4111550 2024-04-14T15:21:28.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.941+0000 7fdaeaffd700 1 -- 172.21.15.90:0/3239286385 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdaf4110220 con 0x7fdaf4111550 2024-04-14T15:21:28.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.941+0000 7fdaeaffd700 1 -- 172.21.15.90:0/3239286385 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1321843265 0 0) 0x7fdaf4110010 con 0x7fdaf4101680 2024-04-14T15:21:28.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.941+0000 7fdaeaffd700 1 -- 172.21.15.90:0/3239286385 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdaf41ac170 con 0x7fdaf4101680 2024-04-14T15:21:28.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.941+0000 7fdaeaffd700 1 -- 172.21.15.90:0/3239286385 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fdaf00018b0 con 0x7fdaf4101680 2024-04-14T15:21:28.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.941+0000 7fdaeaffd700 1 -- 172.21.15.90:0/3239286385 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1110403874 0 0) 0x7fdae00035b0 con 0x7fdaf4102260 2024-04-14T15:21:28.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.941+0000 7fdaeaffd700 1 -- 172.21.15.90:0/3239286385 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdaf4110010 con 0x7fdaf4102260 2024-04-14T15:21:28.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.941+0000 7fdaeaffd700 1 -- 172.21.15.90:0/3239286385 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fdae4001430 con 0x7fdaf4102260 2024-04-14T15:21:28.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.941+0000 7fdaeaffd700 1 -- 172.21.15.90:0/3239286385 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1800816469 0 0) 0x7fdaf4110220 con 0x7fdaf4111550 2024-04-14T15:21:28.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.941+0000 7fdaeaffd700 1 -- 172.21.15.90:0/3239286385 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdae00035b0 con 0x7fdaf4111550 2024-04-14T15:21:28.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.941+0000 7fdaeaffd700 1 -- 172.21.15.90:0/3239286385 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fdaec001bb0 con 0x7fdaf4111550 2024-04-14T15:21:28.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.941+0000 7fdaeaffd700 1 -- 172.21.15.90:0/3239286385 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2432726666 0 0) 0x7fdaf41ac170 con 0x7fdaf4101680 2024-04-14T15:21:28.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.941+0000 7fdaeaffd700 1 -- 172.21.15.90:0/3239286385 >> v1:172.21.15.90:6790/0 conn(0x7fdaf4111550 legacy=0x7fdaf41a89f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:28.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.941+0000 7fdaeaffd700 1 -- 172.21.15.90:0/3239286385 >> v1:172.21.15.90:6789/0 conn(0x7fdaf4102260 legacy=0x7fdaf410d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:28.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.941+0000 7fdaeaffd700 1 -- 172.21.15.90:0/3239286385 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdaf41ad3b0 con 0x7fdaf4101680 2024-04-14T15:21:28.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.941+0000 7fdafbf3b700 1 -- 172.21.15.90:0/3239286385 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdaf41ac3a0 con 0x7fdaf4101680 2024-04-14T15:21:28.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.941+0000 7fdafbf3b700 10 client.49417.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:21:28.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.941+0000 7fdafbf3b700 1 -- 172.21.15.90:0/3239286385 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fdaf41ac860 con 0x7fdaf4101680 2024-04-14T15:21:28.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.941+0000 7fdafbf3b700 1 librados: init done 2024-04-14T15:21:28.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.941+0000 7fdaeaffd700 1 -- 172.21.15.90:0/3239286385 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fdaf0001c80 con 0x7fdaf4101680 2024-04-14T15:21:28.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.941+0000 7fdaeaffd700 1 -- 172.21.15.90:0/3239286385 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fdaf0004be0 con 0x7fdaf4101680 2024-04-14T15:21:28.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.942+0000 7fdafbf3b700 1 -- 172.21.15.90:0/3239286385 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fdad8005100 con 0x7fdaf4101680 2024-04-14T15:21:28.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.943+0000 7fdaeaffd700 1 -- 172.21.15.90:0/3239286385 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fdaf0003950 con 0x7fdaf4101680 2024-04-14T15:21:28.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.943+0000 7fdaeaffd700 1 -- 172.21.15.90:0/3239286385 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(561..561 src has 467..561) ==== 8224+0+0 (unknown 1548988565 0 0) 0x7fdaf0089830 con 0x7fdaf4101680 2024-04-14T15:21:28.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.943+0000 7fdaeaffd700 10 client.49417.objecter ms_dispatch 0x7fdaf4001550 osd_map(561..561 src has 467..561) 2024-04-14T15:21:28.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.943+0000 7fdaeaffd700 3 client.49417.objecter handle_osd_map got epochs [561,561] > 0 2024-04-14T15:21:28.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.943+0000 7fdaeaffd700 3 client.49417.objecter handle_osd_map decoding full epoch 561 2024-04-14T15:21:28.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.944+0000 7fdaeaffd700 20 client.49417.objecter dump_active .. 0 homeless 2024-04-14T15:21:28.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.944+0000 7fdaeaffd700 1 -- 172.21.15.90:0/3239286385 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v561) ==== 54+0+16 (unknown 308721427 0 2420000192) 0x7fdaf008a060 con 0x7fdaf4101680 2024-04-14T15:21:28.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.944+0000 7fdaeaffd700 10 client.49417.objecter ms_handle_connect 0x7fdae0071390 2024-04-14T15:21:28.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.944+0000 7fdafbf3b700 10 client.49417.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:21:28.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.944+0000 7fdafbf3b700 15 client.49417.objecter _session_command_op_assign -1 1 2024-04-14T15:21:28.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.944+0000 7fdafbf3b700 20 client.49417.objecter _get_session s=0x7fdad80053e0 osd=0 3 2024-04-14T15:21:28.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.944+0000 7fdafbf3b700 20 client.49417.objecter put_session s=0x7fdad80053e0 osd=0 3 2024-04-14T15:21:28.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.944+0000 7fdafbf3b700 20 client.49417.objecter _get_session s=0x7fdad80053e0 osd=0 3 2024-04-14T15:21:28.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.944+0000 7fdafbf3b700 15 client.49417.objecter _session_command_op_remove -1 1 2024-04-14T15:21:28.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.944+0000 7fdafbf3b700 20 client.49417.objecter get_session s=0x7fdad80053e0 osd=0 3 2024-04-14T15:21:28.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.944+0000 7fdafbf3b700 15 client.49417.objecter _session_command_op_assign 0 1 2024-04-14T15:21:28.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.944+0000 7fdafbf3b700 20 client.49417.objecter put_session s=0x7fdad80053e0 osd=0 4 2024-04-14T15:21:28.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.944+0000 7fdafbf3b700 10 client.49417.objecter _send_command 1 2024-04-14T15:21:28.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.944+0000 7fdafbf3b700 1 -- 172.21.15.90:0/3239286385 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdad8004b50 con 0x7fdad8004730 2024-04-14T15:21:28.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.944+0000 7fdaeaffd700 10 client.49417.objecter ms_handle_connect 0x7fdad8004730 2024-04-14T15:21:28.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.945+0000 7fdaeaffd700 1 -- 172.21.15.90:0/3239286385 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fdad8004b50 con 0x7fdad8004730 2024-04-14T15:21:28.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.945+0000 7fdaeaffd700 10 client.49417.objecter ms_dispatch 0x7fdaf4001550 command_reply(tid 1: 0 ) 2024-04-14T15:21:28.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.945+0000 7fdaeaffd700 10 client.49417.objecter _finish_command 1 = system:0 2024-04-14T15:21:28.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.945+0000 7fdaeaffd700 20 client.49417.objecter put_session s=0x7fdad80053e0 osd=0 4 2024-04-14T15:21:28.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.945+0000 7fdaeaffd700 15 client.49417.objecter _session_command_op_remove 0 1 2024-04-14T15:21:28.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.962+0000 7fdafbf3b700 10 client.49417.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:21:28.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.962+0000 7fdafbf3b700 15 client.49417.objecter _session_command_op_assign -1 2 2024-04-14T15:21:28.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.962+0000 7fdafbf3b700 20 client.49417.objecter _get_session s=0x7fdad80053e0 osd=0 3 2024-04-14T15:21:28.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.962+0000 7fdafbf3b700 20 client.49417.objecter put_session s=0x7fdad80053e0 osd=0 3 2024-04-14T15:21:28.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.962+0000 7fdafbf3b700 20 client.49417.objecter _get_session s=0x7fdad80053e0 osd=0 3 2024-04-14T15:21:28.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.962+0000 7fdafbf3b700 15 client.49417.objecter _session_command_op_remove -1 2 2024-04-14T15:21:28.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.962+0000 7fdafbf3b700 20 client.49417.objecter get_session s=0x7fdad80053e0 osd=0 3 2024-04-14T15:21:28.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.962+0000 7fdafbf3b700 15 client.49417.objecter _session_command_op_assign 0 2 2024-04-14T15:21:28.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.962+0000 7fdafbf3b700 20 client.49417.objecter put_session s=0x7fdad80053e0 osd=0 4 2024-04-14T15:21:28.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.962+0000 7fdafbf3b700 10 client.49417.objecter _send_command 2 2024-04-14T15:21:28.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.962+0000 7fdafbf3b700 1 -- 172.21.15.90:0/3239286385 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdad80079f0 con 0x7fdad8004730 2024-04-14T15:21:28.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.964+0000 7fdaeaffd700 1 -- 172.21.15.90:0/3239286385 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fdad80079f0 con 0x7fdad8004730 2024-04-14T15:21:28.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.964+0000 7fdaeaffd700 10 client.49417.objecter ms_dispatch 0x7fdaf4001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:21:28.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.964+0000 7fdaeaffd700 10 client.49417.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:21:28.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.964+0000 7fdaeaffd700 20 client.49417.objecter put_session s=0x7fdad80053e0 osd=0 4 2024-04-14T15:21:28.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.964+0000 7fdaeaffd700 15 client.49417.objecter _session_command_op_remove 0 2 2024-04-14T15:21:28.965 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:21:28.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.964+0000 7fdafbf3b700 10 client.49417.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:21:28.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.964+0000 7fdafbf3b700 15 client.49417.objecter _session_command_op_assign -1 3 2024-04-14T15:21:28.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.964+0000 7fdafbf3b700 20 client.49417.objecter _get_session s=0x7fdad8004110 osd=1 3 2024-04-14T15:21:28.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.964+0000 7fdafbf3b700 20 client.49417.objecter put_session s=0x7fdad8004110 osd=1 3 2024-04-14T15:21:28.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.964+0000 7fdafbf3b700 20 client.49417.objecter _get_session s=0x7fdad8004110 osd=1 3 2024-04-14T15:21:28.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.964+0000 7fdafbf3b700 15 client.49417.objecter _session_command_op_remove -1 3 2024-04-14T15:21:28.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.964+0000 7fdafbf3b700 20 client.49417.objecter get_session s=0x7fdad8004110 osd=1 3 2024-04-14T15:21:28.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.964+0000 7fdafbf3b700 15 client.49417.objecter _session_command_op_assign 1 3 2024-04-14T15:21:28.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.964+0000 7fdafbf3b700 20 client.49417.objecter put_session s=0x7fdad8004110 osd=1 4 2024-04-14T15:21:28.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.964+0000 7fdafbf3b700 10 client.49417.objecter _send_command 3 2024-04-14T15:21:28.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.964+0000 7fdafbf3b700 1 -- 172.21.15.90:0/3239286385 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fdad8002180 con 0x7fdad8004280 2024-04-14T15:21:28.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.964+0000 7fdaeaffd700 10 client.49417.objecter ms_handle_connect 0x7fdad8004280 2024-04-14T15:21:28.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.965+0000 7fdaeaffd700 1 -- 172.21.15.90:0/3239286385 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fdad8002180 con 0x7fdad8004280 2024-04-14T15:21:28.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.965+0000 7fdaeaffd700 10 client.49417.objecter ms_dispatch 0x7fdaf4001550 command_reply(tid 3: 0 ) 2024-04-14T15:21:28.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.965+0000 7fdaeaffd700 10 client.49417.objecter _finish_command 3 = system:0 2024-04-14T15:21:28.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.965+0000 7fdaeaffd700 20 client.49417.objecter put_session s=0x7fdad8004110 osd=1 4 2024-04-14T15:21:28.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.965+0000 7fdaeaffd700 15 client.49417.objecter _session_command_op_remove 1 3 2024-04-14T15:21:28.969 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:21:28.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.982+0000 7fdafbf3b700 10 client.49417.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:21:28.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.982+0000 7fdafbf3b700 15 client.49417.objecter _session_command_op_assign -1 4 2024-04-14T15:21:28.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.982+0000 7fdafbf3b700 20 client.49417.objecter _get_session s=0x7fdad8004110 osd=1 3 2024-04-14T15:21:28.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.982+0000 7fdafbf3b700 20 client.49417.objecter put_session s=0x7fdad8004110 osd=1 3 2024-04-14T15:21:28.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.982+0000 7fdafbf3b700 20 client.49417.objecter _get_session s=0x7fdad8004110 osd=1 3 2024-04-14T15:21:28.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.982+0000 7fdafbf3b700 15 client.49417.objecter _session_command_op_remove -1 4 2024-04-14T15:21:28.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.982+0000 7fdafbf3b700 20 client.49417.objecter get_session s=0x7fdad8004110 osd=1 3 2024-04-14T15:21:28.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.982+0000 7fdafbf3b700 15 client.49417.objecter _session_command_op_assign 1 4 2024-04-14T15:21:28.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.982+0000 7fdafbf3b700 20 client.49417.objecter put_session s=0x7fdad8004110 osd=1 4 2024-04-14T15:21:28.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.982+0000 7fdafbf3b700 10 client.49417.objecter _send_command 4 2024-04-14T15:21:28.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.982+0000 7fdafbf3b700 1 -- 172.21.15.90:0/3239286385 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdad8001600 con 0x7fdad8004280 2024-04-14T15:21:28.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.984+0000 7fdaeaffd700 1 -- 172.21.15.90:0/3239286385 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fdad8001600 con 0x7fdad8004280 2024-04-14T15:21:28.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.984+0000 7fdaeaffd700 10 client.49417.objecter ms_dispatch 0x7fdaf4001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:21:28.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.984+0000 7fdaeaffd700 10 client.49417.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:21:28.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.984+0000 7fdaeaffd700 20 client.49417.objecter put_session s=0x7fdad8004110 osd=1 4 2024-04-14T15:21:28.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.984+0000 7fdaeaffd700 15 client.49417.objecter _session_command_op_remove 1 4 2024-04-14T15:21:28.985 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:21:28.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.984+0000 7fdafbf3b700 10 client.49417.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:21:28.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.984+0000 7fdafbf3b700 15 client.49417.objecter _session_command_op_assign -1 5 2024-04-14T15:21:28.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.984+0000 7fdafbf3b700 20 client.49417.objecter _get_session s=0x7fdad8001a00 osd=2 3 2024-04-14T15:21:28.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.984+0000 7fdafbf3b700 20 client.49417.objecter put_session s=0x7fdad8001a00 osd=2 3 2024-04-14T15:21:28.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.984+0000 7fdafbf3b700 20 client.49417.objecter _get_session s=0x7fdad8001a00 osd=2 3 2024-04-14T15:21:28.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.984+0000 7fdafbf3b700 15 client.49417.objecter _session_command_op_remove -1 5 2024-04-14T15:21:28.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.984+0000 7fdafbf3b700 20 client.49417.objecter get_session s=0x7fdad8001a00 osd=2 3 2024-04-14T15:21:28.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.984+0000 7fdafbf3b700 15 client.49417.objecter _session_command_op_assign 2 5 2024-04-14T15:21:28.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.984+0000 7fdafbf3b700 20 client.49417.objecter put_session s=0x7fdad8001a00 osd=2 4 2024-04-14T15:21:28.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.984+0000 7fdafbf3b700 10 client.49417.objecter _send_command 5 2024-04-14T15:21:28.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.984+0000 7fdafbf3b700 1 -- 172.21.15.90:0/3239286385 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fdad8010f20 con 0x7fdad8008880 2024-04-14T15:21:28.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.984+0000 7fdaeaffd700 10 client.49417.objecter ms_handle_connect 0x7fdad8008880 2024-04-14T15:21:28.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.985+0000 7fdaeaffd700 1 -- 172.21.15.90:0/3239286385 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fdad8010f20 con 0x7fdad8008880 2024-04-14T15:21:28.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.985+0000 7fdaeaffd700 10 client.49417.objecter ms_dispatch 0x7fdaf4001550 command_reply(tid 5: 0 ) 2024-04-14T15:21:28.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.985+0000 7fdaeaffd700 10 client.49417.objecter _finish_command 5 = system:0 2024-04-14T15:21:28.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.985+0000 7fdaeaffd700 20 client.49417.objecter put_session s=0x7fdad8001a00 osd=2 4 2024-04-14T15:21:28.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:28.985+0000 7fdaeaffd700 15 client.49417.objecter _session_command_op_remove 2 5 2024-04-14T15:21:28.995 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:28.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:28.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:21:29.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.001+0000 7fdafbf3b700 10 client.49417.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:21:29.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.001+0000 7fdafbf3b700 15 client.49417.objecter _session_command_op_assign -1 6 2024-04-14T15:21:29.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.001+0000 7fdafbf3b700 20 client.49417.objecter _get_session s=0x7fdad8001a00 osd=2 3 2024-04-14T15:21:29.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.001+0000 7fdafbf3b700 20 client.49417.objecter put_session s=0x7fdad8001a00 osd=2 3 2024-04-14T15:21:29.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.001+0000 7fdafbf3b700 20 client.49417.objecter _get_session s=0x7fdad8001a00 osd=2 3 2024-04-14T15:21:29.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.001+0000 7fdafbf3b700 15 client.49417.objecter _session_command_op_remove -1 6 2024-04-14T15:21:29.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.001+0000 7fdafbf3b700 20 client.49417.objecter get_session s=0x7fdad8001a00 osd=2 3 2024-04-14T15:21:29.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.001+0000 7fdafbf3b700 15 client.49417.objecter _session_command_op_assign 2 6 2024-04-14T15:21:29.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.001+0000 7fdafbf3b700 20 client.49417.objecter put_session s=0x7fdad8001a00 osd=2 4 2024-04-14T15:21:29.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.001+0000 7fdafbf3b700 10 client.49417.objecter _send_command 6 2024-04-14T15:21:29.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.001+0000 7fdafbf3b700 1 -- 172.21.15.90:0/3239286385 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdad8007f10 con 0x7fdad8008880 2024-04-14T15:21:29.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.003+0000 7fdaeaffd700 1 -- 172.21.15.90:0/3239286385 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fdad8007f10 con 0x7fdad8008880 2024-04-14T15:21:29.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.003+0000 7fdaeaffd700 10 client.49417.objecter ms_dispatch 0x7fdaf4001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:21:29.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.003+0000 7fdaeaffd700 10 client.49417.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:21:29.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.003+0000 7fdaeaffd700 20 client.49417.objecter put_session s=0x7fdad8001a00 osd=2 4 2024-04-14T15:21:29.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.003+0000 7fdaeaffd700 15 client.49417.objecter _session_command_op_remove 2 6 2024-04-14T15:21:29.004 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:21:29.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.003+0000 7fdafbf3b700 10 client.49417.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:21:29.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.003+0000 7fdafbf3b700 15 client.49417.objecter _session_command_op_assign -1 7 2024-04-14T15:21:29.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.003+0000 7fdafbf3b700 20 client.49417.objecter _get_session s=0x7fdad80081a0 osd=3 3 2024-04-14T15:21:29.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.003+0000 7fdafbf3b700 20 client.49417.objecter put_session s=0x7fdad80081a0 osd=3 3 2024-04-14T15:21:29.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.003+0000 7fdafbf3b700 20 client.49417.objecter _get_session s=0x7fdad80081a0 osd=3 3 2024-04-14T15:21:29.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.003+0000 7fdafbf3b700 15 client.49417.objecter _session_command_op_remove -1 7 2024-04-14T15:21:29.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.003+0000 7fdafbf3b700 20 client.49417.objecter get_session s=0x7fdad80081a0 osd=3 3 2024-04-14T15:21:29.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.003+0000 7fdafbf3b700 15 client.49417.objecter _session_command_op_assign 3 7 2024-04-14T15:21:29.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.003+0000 7fdafbf3b700 20 client.49417.objecter put_session s=0x7fdad80081a0 osd=3 4 2024-04-14T15:21:29.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.003+0000 7fdafbf3b700 10 client.49417.objecter _send_command 7 2024-04-14T15:21:29.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.003+0000 7fdafbf3b700 1 -- 172.21.15.90:0/3239286385 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fdad8011c70 con 0x7fdad8011100 2024-04-14T15:21:29.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.004+0000 7fdaeaffd700 10 client.49417.objecter ms_handle_connect 0x7fdad8011100 2024-04-14T15:21:29.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.004+0000 7fdaeaffd700 1 -- 172.21.15.90:0/3239286385 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fdad8011c70 con 0x7fdad8011100 2024-04-14T15:21:29.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.004+0000 7fdaeaffd700 10 client.49417.objecter ms_dispatch 0x7fdaf4001550 command_reply(tid 7: 0 ) 2024-04-14T15:21:29.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.004+0000 7fdaeaffd700 10 client.49417.objecter _finish_command 7 = system:0 2024-04-14T15:21:29.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.004+0000 7fdaeaffd700 20 client.49417.objecter put_session s=0x7fdad80081a0 osd=3 4 2024-04-14T15:21:29.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.004+0000 7fdaeaffd700 15 client.49417.objecter _session_command_op_remove 3 7 2024-04-14T15:21:29.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.022+0000 7fdafbf3b700 10 client.49417.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:21:29.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.022+0000 7fdafbf3b700 15 client.49417.objecter _session_command_op_assign -1 8 2024-04-14T15:21:29.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.022+0000 7fdafbf3b700 20 client.49417.objecter _get_session s=0x7fdad80081a0 osd=3 3 2024-04-14T15:21:29.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.022+0000 7fdafbf3b700 20 client.49417.objecter put_session s=0x7fdad80081a0 osd=3 3 2024-04-14T15:21:29.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.022+0000 7fdafbf3b700 20 client.49417.objecter _get_session s=0x7fdad80081a0 osd=3 3 2024-04-14T15:21:29.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.022+0000 7fdafbf3b700 15 client.49417.objecter _session_command_op_remove -1 8 2024-04-14T15:21:29.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.022+0000 7fdafbf3b700 20 client.49417.objecter get_session s=0x7fdad80081a0 osd=3 3 2024-04-14T15:21:29.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.022+0000 7fdafbf3b700 15 client.49417.objecter _session_command_op_assign 3 8 2024-04-14T15:21:29.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.022+0000 7fdafbf3b700 20 client.49417.objecter put_session s=0x7fdad80081a0 osd=3 4 2024-04-14T15:21:29.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.022+0000 7fdafbf3b700 10 client.49417.objecter _send_command 8 2024-04-14T15:21:29.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.022+0000 7fdafbf3b700 1 -- 172.21.15.90:0/3239286385 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdad8011c70 con 0x7fdad8011100 2024-04-14T15:21:29.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.023+0000 7fdaeaffd700 1 -- 172.21.15.90:0/3239286385 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fdad8011c70 con 0x7fdad8011100 2024-04-14T15:21:29.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.023+0000 7fdaeaffd700 10 client.49417.objecter ms_dispatch 0x7fdaf4001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:21:29.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.023+0000 7fdaeaffd700 10 client.49417.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:21:29.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.023+0000 7fdaeaffd700 20 client.49417.objecter put_session s=0x7fdad80081a0 osd=3 4 2024-04-14T15:21:29.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.023+0000 7fdaeaffd700 15 client.49417.objecter _session_command_op_remove 3 8 2024-04-14T15:21:29.024 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:21:29.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.024+0000 7fdafbf3b700 10 client.49417.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:21:29.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.024+0000 7fdafbf3b700 15 client.49417.objecter _session_command_op_assign -1 9 2024-04-14T15:21:29.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.024+0000 7fdafbf3b700 20 client.49417.objecter _get_session s=0x7fdad8011c70 osd=4 3 2024-04-14T15:21:29.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.024+0000 7fdafbf3b700 20 client.49417.objecter put_session s=0x7fdad8011c70 osd=4 3 2024-04-14T15:21:29.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.024+0000 7fdafbf3b700 20 client.49417.objecter _get_session s=0x7fdad8011c70 osd=4 3 2024-04-14T15:21:29.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.024+0000 7fdafbf3b700 15 client.49417.objecter _session_command_op_remove -1 9 2024-04-14T15:21:29.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.024+0000 7fdafbf3b700 20 client.49417.objecter get_session s=0x7fdad8011c70 osd=4 3 2024-04-14T15:21:29.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.024+0000 7fdafbf3b700 15 client.49417.objecter _session_command_op_assign 4 9 2024-04-14T15:21:29.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.024+0000 7fdafbf3b700 20 client.49417.objecter put_session s=0x7fdad8011c70 osd=4 4 2024-04-14T15:21:29.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.024+0000 7fdafbf3b700 10 client.49417.objecter _send_command 9 2024-04-14T15:21:29.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.024+0000 7fdafbf3b700 1 -- 172.21.15.90:0/3239286385 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fdad801a5f0 con 0x7fdad8016ae0 2024-04-14T15:21:29.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.024+0000 7fdaeaffd700 10 client.49417.objecter ms_handle_connect 0x7fdad8016ae0 2024-04-14T15:21:29.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.025+0000 7fdaeaffd700 1 -- 172.21.15.90:0/3239286385 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fdad801a5f0 con 0x7fdad8016ae0 2024-04-14T15:21:29.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.025+0000 7fdaeaffd700 10 client.49417.objecter ms_dispatch 0x7fdaf4001550 command_reply(tid 9: 0 ) 2024-04-14T15:21:29.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.025+0000 7fdaeaffd700 10 client.49417.objecter _finish_command 9 = system:0 2024-04-14T15:21:29.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.025+0000 7fdaeaffd700 20 client.49417.objecter put_session s=0x7fdad8011c70 osd=4 4 2024-04-14T15:21:29.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.025+0000 7fdaeaffd700 15 client.49417.objecter _session_command_op_remove 4 9 2024-04-14T15:21:29.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.041+0000 7fdafbf3b700 10 client.49417.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:21:29.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.041+0000 7fdafbf3b700 15 client.49417.objecter _session_command_op_assign -1 10 2024-04-14T15:21:29.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.041+0000 7fdafbf3b700 20 client.49417.objecter _get_session s=0x7fdad8011c70 osd=4 3 2024-04-14T15:21:29.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.041+0000 7fdafbf3b700 20 client.49417.objecter put_session s=0x7fdad8011c70 osd=4 3 2024-04-14T15:21:29.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.041+0000 7fdafbf3b700 20 client.49417.objecter _get_session s=0x7fdad8011c70 osd=4 3 2024-04-14T15:21:29.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.041+0000 7fdafbf3b700 15 client.49417.objecter _session_command_op_remove -1 10 2024-04-14T15:21:29.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.041+0000 7fdafbf3b700 20 client.49417.objecter get_session s=0x7fdad8011c70 osd=4 3 2024-04-14T15:21:29.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.041+0000 7fdafbf3b700 15 client.49417.objecter _session_command_op_assign 4 10 2024-04-14T15:21:29.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.041+0000 7fdafbf3b700 20 client.49417.objecter put_session s=0x7fdad8011c70 osd=4 4 2024-04-14T15:21:29.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.041+0000 7fdafbf3b700 10 client.49417.objecter _send_command 10 2024-04-14T15:21:29.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.041+0000 7fdafbf3b700 1 -- 172.21.15.90:0/3239286385 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdad800c700 con 0x7fdad8016ae0 2024-04-14T15:21:29.043 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:21:29.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.043+0000 7fdaeaffd700 1 -- 172.21.15.90:0/3239286385 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fdad800c700 con 0x7fdad8016ae0 2024-04-14T15:21:29.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.043+0000 7fdaeaffd700 10 client.49417.objecter ms_dispatch 0x7fdaf4001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:21:29.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.043+0000 7fdaeaffd700 10 client.49417.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:21:29.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.043+0000 7fdaeaffd700 20 client.49417.objecter put_session s=0x7fdad8011c70 osd=4 4 2024-04-14T15:21:29.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.043+0000 7fdaeaffd700 15 client.49417.objecter _session_command_op_remove 4 10 2024-04-14T15:21:29.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.043+0000 7fdafbf3b700 10 client.49417.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:21:29.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.043+0000 7fdafbf3b700 15 client.49417.objecter _session_command_op_assign -1 11 2024-04-14T15:21:29.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.043+0000 7fdafbf3b700 20 client.49417.objecter _get_session s=0x7fdad801a5f0 osd=5 3 2024-04-14T15:21:29.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.043+0000 7fdafbf3b700 20 client.49417.objecter put_session s=0x7fdad801a5f0 osd=5 3 2024-04-14T15:21:29.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.043+0000 7fdafbf3b700 20 client.49417.objecter _get_session s=0x7fdad801a5f0 osd=5 3 2024-04-14T15:21:29.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.043+0000 7fdafbf3b700 15 client.49417.objecter _session_command_op_remove -1 11 2024-04-14T15:21:29.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.043+0000 7fdafbf3b700 20 client.49417.objecter get_session s=0x7fdad801a5f0 osd=5 3 2024-04-14T15:21:29.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.043+0000 7fdafbf3b700 15 client.49417.objecter _session_command_op_assign 5 11 2024-04-14T15:21:29.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.043+0000 7fdafbf3b700 20 client.49417.objecter put_session s=0x7fdad801a5f0 osd=5 4 2024-04-14T15:21:29.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.043+0000 7fdafbf3b700 10 client.49417.objecter _send_command 11 2024-04-14T15:21:29.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.043+0000 7fdafbf3b700 1 -- 172.21.15.90:0/3239286385 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fdad801e9f0 con 0x7fdad801aca0 2024-04-14T15:21:29.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.043+0000 7fdaeaffd700 10 client.49417.objecter ms_handle_connect 0x7fdad801aca0 2024-04-14T15:21:29.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.044+0000 7fdaeaffd700 1 -- 172.21.15.90:0/3239286385 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fdad801e9f0 con 0x7fdad801aca0 2024-04-14T15:21:29.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.044+0000 7fdaeaffd700 10 client.49417.objecter ms_dispatch 0x7fdaf4001550 command_reply(tid 11: 0 ) 2024-04-14T15:21:29.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.044+0000 7fdaeaffd700 10 client.49417.objecter _finish_command 11 = system:0 2024-04-14T15:21:29.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.044+0000 7fdaeaffd700 20 client.49417.objecter put_session s=0x7fdad801a5f0 osd=5 4 2024-04-14T15:21:29.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.044+0000 7fdaeaffd700 15 client.49417.objecter _session_command_op_remove 5 11 2024-04-14T15:21:29.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.062+0000 7fdafbf3b700 10 client.49417.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:21:29.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.062+0000 7fdafbf3b700 15 client.49417.objecter _session_command_op_assign -1 12 2024-04-14T15:21:29.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.062+0000 7fdafbf3b700 20 client.49417.objecter _get_session s=0x7fdad801a5f0 osd=5 3 2024-04-14T15:21:29.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.062+0000 7fdafbf3b700 20 client.49417.objecter put_session s=0x7fdad801a5f0 osd=5 3 2024-04-14T15:21:29.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.062+0000 7fdafbf3b700 20 client.49417.objecter _get_session s=0x7fdad801a5f0 osd=5 3 2024-04-14T15:21:29.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.062+0000 7fdafbf3b700 15 client.49417.objecter _session_command_op_remove -1 12 2024-04-14T15:21:29.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.062+0000 7fdafbf3b700 20 client.49417.objecter get_session s=0x7fdad801a5f0 osd=5 3 2024-04-14T15:21:29.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.062+0000 7fdafbf3b700 15 client.49417.objecter _session_command_op_assign 5 12 2024-04-14T15:21:29.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.062+0000 7fdafbf3b700 20 client.49417.objecter put_session s=0x7fdad801a5f0 osd=5 4 2024-04-14T15:21:29.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.062+0000 7fdafbf3b700 10 client.49417.objecter _send_command 12 2024-04-14T15:21:29.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.062+0000 7fdafbf3b700 1 -- 172.21.15.90:0/3239286385 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdad801fd50 con 0x7fdad801aca0 2024-04-14T15:21:29.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.063+0000 7fdaeaffd700 1 -- 172.21.15.90:0/3239286385 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fdad801fd50 con 0x7fdad801aca0 2024-04-14T15:21:29.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.063+0000 7fdaeaffd700 10 client.49417.objecter ms_dispatch 0x7fdaf4001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:21:29.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.063+0000 7fdaeaffd700 10 client.49417.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:21:29.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.063+0000 7fdaeaffd700 20 client.49417.objecter put_session s=0x7fdad801a5f0 osd=5 4 2024-04-14T15:21:29.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.063+0000 7fdaeaffd700 15 client.49417.objecter _session_command_op_remove 5 12 2024-04-14T15:21:29.064 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:21:29.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.064+0000 7fdafbf3b700 10 client.49417.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:21:29.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.064+0000 7fdafbf3b700 15 client.49417.objecter _session_command_op_assign -1 13 2024-04-14T15:21:29.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.064+0000 7fdafbf3b700 20 client.49417.objecter _get_session s=0x7fdad801fd50 osd=6 3 2024-04-14T15:21:29.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.064+0000 7fdafbf3b700 20 client.49417.objecter put_session s=0x7fdad801fd50 osd=6 3 2024-04-14T15:21:29.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.064+0000 7fdafbf3b700 20 client.49417.objecter _get_session s=0x7fdad801fd50 osd=6 3 2024-04-14T15:21:29.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.064+0000 7fdafbf3b700 15 client.49417.objecter _session_command_op_remove -1 13 2024-04-14T15:21:29.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.064+0000 7fdafbf3b700 20 client.49417.objecter get_session s=0x7fdad801fd50 osd=6 3 2024-04-14T15:21:29.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.064+0000 7fdafbf3b700 15 client.49417.objecter _session_command_op_assign 6 13 2024-04-14T15:21:29.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.064+0000 7fdafbf3b700 20 client.49417.objecter put_session s=0x7fdad801fd50 osd=6 4 2024-04-14T15:21:29.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.064+0000 7fdafbf3b700 10 client.49417.objecter _send_command 13 2024-04-14T15:21:29.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.064+0000 7fdafbf3b700 1 -- 172.21.15.90:0/3239286385 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fdad80135a0 con 0x7fdad801ff60 2024-04-14T15:21:29.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.064+0000 7fdaeaffd700 10 client.49417.objecter ms_handle_connect 0x7fdad801ff60 2024-04-14T15:21:29.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.065+0000 7fdaeaffd700 1 -- 172.21.15.90:0/3239286385 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fdad80135a0 con 0x7fdad801ff60 2024-04-14T15:21:29.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.065+0000 7fdaeaffd700 10 client.49417.objecter ms_dispatch 0x7fdaf4001550 command_reply(tid 13: 0 ) 2024-04-14T15:21:29.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.065+0000 7fdaeaffd700 10 client.49417.objecter _finish_command 13 = system:0 2024-04-14T15:21:29.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.065+0000 7fdaeaffd700 20 client.49417.objecter put_session s=0x7fdad801fd50 osd=6 4 2024-04-14T15:21:29.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.065+0000 7fdaeaffd700 15 client.49417.objecter _session_command_op_remove 6 13 2024-04-14T15:21:29.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.081+0000 7fdafbf3b700 10 client.49417.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:21:29.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.081+0000 7fdafbf3b700 15 client.49417.objecter _session_command_op_assign -1 14 2024-04-14T15:21:29.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.081+0000 7fdafbf3b700 20 client.49417.objecter _get_session s=0x7fdad801fd50 osd=6 3 2024-04-14T15:21:29.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.081+0000 7fdafbf3b700 20 client.49417.objecter put_session s=0x7fdad801fd50 osd=6 3 2024-04-14T15:21:29.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.081+0000 7fdafbf3b700 20 client.49417.objecter _get_session s=0x7fdad801fd50 osd=6 3 2024-04-14T15:21:29.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.081+0000 7fdafbf3b700 15 client.49417.objecter _session_command_op_remove -1 14 2024-04-14T15:21:29.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.081+0000 7fdafbf3b700 20 client.49417.objecter get_session s=0x7fdad801fd50 osd=6 3 2024-04-14T15:21:29.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.081+0000 7fdafbf3b700 15 client.49417.objecter _session_command_op_assign 6 14 2024-04-14T15:21:29.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.081+0000 7fdafbf3b700 20 client.49417.objecter put_session s=0x7fdad801fd50 osd=6 4 2024-04-14T15:21:29.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.081+0000 7fdafbf3b700 10 client.49417.objecter _send_command 14 2024-04-14T15:21:29.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.081+0000 7fdafbf3b700 1 -- 172.21.15.90:0/3239286385 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdad80135a0 con 0x7fdad801ff60 2024-04-14T15:21:29.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.083+0000 7fdaeaffd700 1 -- 172.21.15.90:0/3239286385 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fdad80135a0 con 0x7fdad801ff60 2024-04-14T15:21:29.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.083+0000 7fdaeaffd700 10 client.49417.objecter ms_dispatch 0x7fdaf4001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:21:29.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.083+0000 7fdaeaffd700 10 client.49417.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:21:29.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.083+0000 7fdaeaffd700 20 client.49417.objecter put_session s=0x7fdad801fd50 osd=6 4 2024-04-14T15:21:29.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.083+0000 7fdaeaffd700 15 client.49417.objecter _session_command_op_remove 6 14 2024-04-14T15:21:29.084 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:21:29.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.083+0000 7fdafbf3b700 10 client.49417.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:21:29.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.083+0000 7fdafbf3b700 15 client.49417.objecter _session_command_op_assign -1 15 2024-04-14T15:21:29.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.083+0000 7fdafbf3b700 20 client.49417.objecter _get_session s=0x7fdad80135a0 osd=7 3 2024-04-14T15:21:29.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.083+0000 7fdafbf3b700 20 client.49417.objecter put_session s=0x7fdad80135a0 osd=7 3 2024-04-14T15:21:29.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.083+0000 7fdafbf3b700 20 client.49417.objecter _get_session s=0x7fdad80135a0 osd=7 3 2024-04-14T15:21:29.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.083+0000 7fdafbf3b700 15 client.49417.objecter _session_command_op_remove -1 15 2024-04-14T15:21:29.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.083+0000 7fdafbf3b700 20 client.49417.objecter get_session s=0x7fdad80135a0 osd=7 3 2024-04-14T15:21:29.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.083+0000 7fdafbf3b700 15 client.49417.objecter _session_command_op_assign 7 15 2024-04-14T15:21:29.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.083+0000 7fdafbf3b700 20 client.49417.objecter put_session s=0x7fdad80135a0 osd=7 4 2024-04-14T15:21:29.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.083+0000 7fdafbf3b700 10 client.49417.objecter _send_command 15 2024-04-14T15:21:29.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.083+0000 7fdafbf3b700 1 -- 172.21.15.90:0/3239286385 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fdad80275c0 con 0x7fdad8023ab0 2024-04-14T15:21:29.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.084+0000 7fdaeaffd700 10 client.49417.objecter ms_handle_connect 0x7fdad8023ab0 2024-04-14T15:21:29.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.085+0000 7fdaeaffd700 1 -- 172.21.15.90:0/3239286385 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fdad80275c0 con 0x7fdad8023ab0 2024-04-14T15:21:29.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.085+0000 7fdaeaffd700 10 client.49417.objecter ms_dispatch 0x7fdaf4001550 command_reply(tid 15: 0 ) 2024-04-14T15:21:29.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.085+0000 7fdaeaffd700 10 client.49417.objecter _finish_command 15 = system:0 2024-04-14T15:21:29.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.085+0000 7fdaeaffd700 20 client.49417.objecter put_session s=0x7fdad80135a0 osd=7 4 2024-04-14T15:21:29.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.085+0000 7fdaeaffd700 15 client.49417.objecter _session_command_op_remove 7 15 2024-04-14T15:21:29.097 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:21:29.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.102+0000 7fdafbf3b700 10 client.49417.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:21:29.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.102+0000 7fdafbf3b700 15 client.49417.objecter _session_command_op_assign -1 16 2024-04-14T15:21:29.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.102+0000 7fdafbf3b700 20 client.49417.objecter _get_session s=0x7fdad80135a0 osd=7 3 2024-04-14T15:21:29.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.102+0000 7fdafbf3b700 20 client.49417.objecter put_session s=0x7fdad80135a0 osd=7 3 2024-04-14T15:21:29.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.102+0000 7fdafbf3b700 20 client.49417.objecter _get_session s=0x7fdad80135a0 osd=7 3 2024-04-14T15:21:29.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.102+0000 7fdafbf3b700 15 client.49417.objecter _session_command_op_remove -1 16 2024-04-14T15:21:29.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.102+0000 7fdafbf3b700 20 client.49417.objecter get_session s=0x7fdad80135a0 osd=7 3 2024-04-14T15:21:29.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.102+0000 7fdafbf3b700 15 client.49417.objecter _session_command_op_assign 7 16 2024-04-14T15:21:29.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.102+0000 7fdafbf3b700 20 client.49417.objecter put_session s=0x7fdad80135a0 osd=7 4 2024-04-14T15:21:29.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.102+0000 7fdafbf3b700 10 client.49417.objecter _send_command 16 2024-04-14T15:21:29.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.102+0000 7fdafbf3b700 1 -- 172.21.15.90:0/3239286385 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdad80275c0 con 0x7fdad8023ab0 2024-04-14T15:21:29.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.104+0000 7fdaeaffd700 1 -- 172.21.15.90:0/3239286385 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fdaf0056e50 con 0x7fdad8023ab0 2024-04-14T15:21:29.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.104+0000 7fdaeaffd700 10 client.49417.objecter ms_dispatch 0x7fdaf4001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:21:29.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.104+0000 7fdaeaffd700 10 client.49417.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:21:29.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.104+0000 7fdaeaffd700 20 client.49417.objecter put_session s=0x7fdad80135a0 osd=7 4 2024-04-14T15:21:29.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.104+0000 7fdaeaffd700 15 client.49417.objecter _session_command_op_remove 7 16 2024-04-14T15:21:29.106 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:21:29.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.104+0000 7fdafbf3b700 10 librados: watch_flush enter 2024-04-14T15:21:29.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.104+0000 7fdafbf3b700 10 librados: watch_flush exit 2024-04-14T15:21:29.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.104+0000 7fdafbf3b700 10 client.49417.objecter close_session for osd.0 2024-04-14T15:21:29.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.104+0000 7fdafbf3b700 1 -- 172.21.15.90:0/3239286385 >> v1:172.21.15.90:6801/3761806885 conn(0x7fdad8004730 legacy=0x7fdad8002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:29.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.104+0000 7fdafbf3b700 20 client.49417.objecter put_session s=0x7fdad80053e0 osd=0 1 2024-04-14T15:21:29.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.104+0000 7fdafbf3b700 10 client.49417.objecter close_session for osd.1 2024-04-14T15:21:29.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.104+0000 7fdafbf3b700 1 -- 172.21.15.90:0/3239286385 >> v1:172.21.15.90:6805/4170361699 conn(0x7fdad8004280 legacy=0x7fdad800ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:29.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.104+0000 7fdafbf3b700 20 client.49417.objecter put_session s=0x7fdad8004110 osd=1 1 2024-04-14T15:21:29.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.104+0000 7fdafbf3b700 10 client.49417.objecter close_session for osd.2 2024-04-14T15:21:29.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.104+0000 7fdafbf3b700 1 -- 172.21.15.90:0/3239286385 >> v1:172.21.15.90:6809/1686184251 conn(0x7fdad8008880 legacy=0x7fdad800f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:29.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.104+0000 7fdafbf3b700 20 client.49417.objecter put_session s=0x7fdad8001a00 osd=2 1 2024-04-14T15:21:29.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.104+0000 7fdafbf3b700 10 client.49417.objecter close_session for osd.3 2024-04-14T15:21:29.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.104+0000 7fdafbf3b700 1 -- 172.21.15.90:0/3239286385 >> v1:172.21.15.90:6813/2270166774 conn(0x7fdad8011100 legacy=0x7fdad8011560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:29.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.104+0000 7fdafbf3b700 20 client.49417.objecter put_session s=0x7fdad80081a0 osd=3 1 2024-04-14T15:21:29.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.104+0000 7fdafbf3b700 10 client.49417.objecter close_session for osd.4 2024-04-14T15:21:29.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.104+0000 7fdafbf3b700 1 -- 172.21.15.90:0/3239286385 >> v1:172.21.15.92:6800/3887549733 conn(0x7fdad8016ae0 legacy=0x7fdad8018ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:29.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.104+0000 7fdafbf3b700 20 client.49417.objecter put_session s=0x7fdad8011c70 osd=4 1 2024-04-14T15:21:29.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.104+0000 7fdafbf3b700 10 client.49417.objecter close_session for osd.5 2024-04-14T15:21:29.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.104+0000 7fdafbf3b700 1 -- 172.21.15.90:0/3239286385 >> v1:172.21.15.92:6804/3244561440 conn(0x7fdad801aca0 legacy=0x7fdad801d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:29.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.104+0000 7fdafbf3b700 20 client.49417.objecter put_session s=0x7fdad801a5f0 osd=5 1 2024-04-14T15:21:29.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.104+0000 7fdafbf3b700 10 client.49417.objecter close_session for osd.6 2024-04-14T15:21:29.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.104+0000 7fdafbf3b700 1 -- 172.21.15.90:0/3239286385 >> v1:172.21.15.92:6808/636125963 conn(0x7fdad801ff60 legacy=0x7fdad8011e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:29.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.104+0000 7fdafbf3b700 20 client.49417.objecter put_session s=0x7fdad801fd50 osd=6 1 2024-04-14T15:21:29.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.104+0000 7fdafbf3b700 10 client.49417.objecter close_session for osd.7 2024-04-14T15:21:29.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.104+0000 7fdafa4d8700 1 -- 172.21.15.90:0/3239286385 reap_dead start 2024-04-14T15:21:29.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.104+0000 7fdafbf3b700 1 -- 172.21.15.90:0/3239286385 >> v1:172.21.15.92:6812/1773706716 conn(0x7fdad8023ab0 legacy=0x7fdad8025ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:29.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.104+0000 7fdafa4d8700 1 -- 172.21.15.90:0/3239286385 reap_dead start 2024-04-14T15:21:29.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.104+0000 7fdafbf3b700 20 client.49417.objecter put_session s=0x7fdad80135a0 osd=7 1 2024-04-14T15:21:29.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.104+0000 7fdafbf3b700 20 client.49417.objecter shutdown clearing up homeless session... 2024-04-14T15:21:29.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.104+0000 7fdafbf3b700 10 client.49417.objecter successfully canceled tick 2024-04-14T15:21:29.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.104+0000 7fdafa4d8700 1 -- 172.21.15.90:0/3239286385 reap_dead start 2024-04-14T15:21:29.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.104+0000 7fdafa4d8700 1 -- 172.21.15.90:0/3239286385 reap_dead start 2024-04-14T15:21:29.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.104+0000 7fdafbf3b700 1 -- 172.21.15.90:0/3239286385 >> v1:172.21.15.90:6800/420935427 conn(0x7fdae0071390 legacy=0x7fdae0073850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:29.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.104+0000 7fdafbf3b700 1 -- 172.21.15.90:0/3239286385 >> v1:172.21.15.92:6789/0 conn(0x7fdaf4101680 legacy=0x7fdaf410ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:29.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.105+0000 7fdafbf3b700 1 -- 172.21.15.90:0/3239286385 shutdown_connections 2024-04-14T15:21:29.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.105+0000 7fdafbf3b700 1 -- 172.21.15.90:0/3239286385 >> 172.21.15.90:0/3239286385 conn(0x7fdaf40fd2a0 msgr2=0x7fdaf41104b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:21:29.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.105+0000 7fdafbf3b700 1 -- 172.21.15.90:0/3239286385 shutdown_connections 2024-04-14T15:21:29.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.105+0000 7fdafbf3b700 1 -- 172.21.15.90:0/3239286385 wait complete. 2024-04-14T15:21:29.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:29.105+0000 7fdafbf3b700 1 librados: shutdown 2024-04-14T15:21:29.110 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:21:29.110 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:21:29.110 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:21:29.110 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:21:29.110 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:21:29.110 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:21:29.110 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:21:29.111 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:21:29.122 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:29.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:29.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:21:29.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:29 smithi090 ceph-mon[41670]: 4.8 scrub starts 2024-04-14T15:21:29.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:29 smithi090 ceph-mon[41670]: 4.8 scrub ok 2024-04-14T15:21:29.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:29 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:21:29.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:29 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:21:29.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:29 smithi090 ceph-mon[41670]: osdmap e561: 8 total, 8 up, 7 in 2024-04-14T15:21:29.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:29 smithi090 ceph-mon[41670]: 11.a scrub starts 2024-04-14T15:21:29.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:29 smithi090 ceph-mon[41670]: 11.a scrub ok 2024-04-14T15:21:29.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:29 smithi090 ceph-mon[41670]: 110.14 scrub starts 2024-04-14T15:21:29.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:29 smithi090 ceph-mon[41670]: 110.14 scrub ok 2024-04-14T15:21:29.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:29 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3239286385' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:21:29.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:29 smithi090 ceph-mon[30399]: 4.8 scrub starts 2024-04-14T15:21:29.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:29 smithi090 ceph-mon[30399]: 4.8 scrub ok 2024-04-14T15:21:29.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:29 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:21:29.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:29 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:21:29.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:29 smithi090 ceph-mon[30399]: osdmap e561: 8 total, 8 up, 7 in 2024-04-14T15:21:29.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:29 smithi090 ceph-mon[30399]: 11.a scrub starts 2024-04-14T15:21:29.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:29 smithi090 ceph-mon[30399]: 11.a scrub ok 2024-04-14T15:21:29.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:29 smithi090 ceph-mon[30399]: 110.14 scrub starts 2024-04-14T15:21:29.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:29 smithi090 ceph-mon[30399]: 110.14 scrub ok 2024-04-14T15:21:29.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:29 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3239286385' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:21:29.223 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:21:29.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:29.248 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:29.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:21:29.349 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:29.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:29 smithi092 ceph-mon[33966]: 4.8 scrub starts 2024-04-14T15:21:29.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:29 smithi092 ceph-mon[33966]: 4.8 scrub ok 2024-04-14T15:21:29.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:29 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:21:29.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:29 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:21:29.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:29 smithi092 ceph-mon[33966]: osdmap e561: 8 total, 8 up, 7 in 2024-04-14T15:21:29.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:29 smithi092 ceph-mon[33966]: 11.a scrub starts 2024-04-14T15:21:29.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:29 smithi092 ceph-mon[33966]: 11.a scrub ok 2024-04-14T15:21:29.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:29 smithi092 ceph-mon[33966]: 110.14 scrub starts 2024-04-14T15:21:29.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:29 smithi092 ceph-mon[33966]: 110.14 scrub ok 2024-04-14T15:21:29.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:29 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3239286385' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:21:29.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:29.373 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:29.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:29.474 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:21:29.498 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:29.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:29.499 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:21:29.599 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:21:29.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:29.624 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:29.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:21:29.725 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:21:29.748 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:29.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:29.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:21:29.770 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-14T15:21:29.850 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:29.873 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:29.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:29.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:29.975 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:29.998 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:29.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:29.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:30.100 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:21:30.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:30.126 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:30.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:21:30.227 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:30.251 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:30.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:30.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:30.352 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:21:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:30 smithi092 ceph-mon[33966]: 2.4 deep-scrub starts 2024-04-14T15:21:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:30 smithi092 ceph-mon[33966]: 2.4 deep-scrub ok 2024-04-14T15:21:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:30 smithi092 ceph-mon[33966]: 11.1e scrub starts 2024-04-14T15:21:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:30 smithi092 ceph-mon[33966]: 11.1e scrub ok 2024-04-14T15:21:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:30 smithi092 ceph-mon[33966]: 5.a scrub starts 2024-04-14T15:21:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:30 smithi092 ceph-mon[33966]: 5.a scrub ok 2024-04-14T15:21:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:30 smithi092 ceph-mon[33966]: 5.17 scrub starts 2024-04-14T15:21:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:30 smithi092 ceph-mon[33966]: 5.17 scrub ok 2024-04-14T15:21:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:30 smithi092 ceph-mon[33966]: pgmap v818: 248 pgs: 1 active+clean+snaptrim, 1 peering, 246 active+clean; 8.4 MiB data, 498 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s; 53 KiB/s, 0 objects/s recovering 2024-04-14T15:21:30.376 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:30.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:30.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:21:30.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:30 smithi090 ceph-mon[41670]: 2.4 deep-scrub starts 2024-04-14T15:21:30.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:30 smithi090 ceph-mon[41670]: 2.4 deep-scrub ok 2024-04-14T15:21:30.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:30 smithi090 ceph-mon[41670]: 11.1e scrub starts 2024-04-14T15:21:30.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:30 smithi090 ceph-mon[41670]: 11.1e scrub ok 2024-04-14T15:21:30.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:30 smithi090 ceph-mon[41670]: 5.a scrub starts 2024-04-14T15:21:30.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:30 smithi090 ceph-mon[41670]: 5.a scrub ok 2024-04-14T15:21:30.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:30 smithi090 ceph-mon[41670]: 5.17 scrub starts 2024-04-14T15:21:30.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:30 smithi090 ceph-mon[41670]: 5.17 scrub ok 2024-04-14T15:21:30.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:30 smithi090 ceph-mon[41670]: pgmap v818: 248 pgs: 1 active+clean+snaptrim, 1 peering, 246 active+clean; 8.4 MiB data, 498 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s; 53 KiB/s, 0 objects/s recovering 2024-04-14T15:21:30.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:30 smithi090 ceph-mon[30399]: 2.4 deep-scrub starts 2024-04-14T15:21:30.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:30 smithi090 ceph-mon[30399]: 2.4 deep-scrub ok 2024-04-14T15:21:30.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:30 smithi090 ceph-mon[30399]: 11.1e scrub starts 2024-04-14T15:21:30.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:30 smithi090 ceph-mon[30399]: 11.1e scrub ok 2024-04-14T15:21:30.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:30 smithi090 ceph-mon[30399]: 5.a scrub starts 2024-04-14T15:21:30.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:30 smithi090 ceph-mon[30399]: 5.a scrub ok 2024-04-14T15:21:30.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:30 smithi090 ceph-mon[30399]: 5.17 scrub starts 2024-04-14T15:21:30.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:30 smithi090 ceph-mon[30399]: 5.17 scrub ok 2024-04-14T15:21:30.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:30 smithi090 ceph-mon[30399]: pgmap v818: 248 pgs: 1 active+clean+snaptrim, 1 peering, 246 active+clean; 8.4 MiB data, 498 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s; 53 KiB/s, 0 objects/s recovering 2024-04-14T15:21:30.477 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:21:30.501 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:30.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:30.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:21:30.602 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:21:30.626 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:30.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:30.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:21:30.728 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:21:30.753 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:30.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:30.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:21:30.856 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:30.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:30.881 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:30.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:30.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:30.926+0000 7f65776c1700 1 -- 172.21.15.90:0/1390466966 >> v1:172.21.15.90:6789/0 conn(0x7f657010e270 legacy=0x7f6570110710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:30.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:30.926+0000 7f65776c1700 1 -- 172.21.15.90:0/1390466966 shutdown_connections 2024-04-14T15:21:30.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:30.926+0000 7f65776c1700 1 -- 172.21.15.90:0/1390466966 >> 172.21.15.90:0/1390466966 conn(0x7f6570076aa0 msgr2=0x7f6570076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:21:30.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:30.927+0000 7f65776c1700 1 -- 172.21.15.90:0/1390466966 shutdown_connections 2024-04-14T15:21:30.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:30.927+0000 7f65776c1700 1 -- 172.21.15.90:0/1390466966 wait complete. 2024-04-14T15:21:30.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:30.927+0000 7f65776c1700 1 librados: starting msgr at 2024-04-14T15:21:30.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:30.927+0000 7f65776c1700 1 librados: starting objecter 2024-04-14T15:21:30.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:30.928+0000 7f65776c1700 1 Processor -- start 2024-04-14T15:21:30.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:30.928+0000 7f65776c1700 1 -- start start 2024-04-14T15:21:30.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:30.928+0000 7f65776c1700 1 librados: setting wanted keys 2024-04-14T15:21:30.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:30.928+0000 7f65776c1700 1 librados: calling monclient init 2024-04-14T15:21:30.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:30.928+0000 7f65776c1700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f65701a9a60 con 0x7f65701023f0 2024-04-14T15:21:30.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:30.928+0000 7f65776c1700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f65701aac80 con 0x7f657010e270 2024-04-14T15:21:30.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:30.928+0000 7f65776c1700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f65701abec0 con 0x7f657010a6d0 2024-04-14T15:21:30.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:30.928+0000 7f657545d700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f65701023f0 0x7f65701018d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:45664/0 (socket says 172.21.15.90:45664) 2024-04-14T15:21:30.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:30.928+0000 7f6574c5c700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f657010a6d0 0x7f65701a4ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:50184/0 (socket says 172.21.15.90:50184) 2024-04-14T15:21:30.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:30.928+0000 7f657545d700 1 -- 172.21.15.90:0/3391229622 learned_addr learned my addr 172.21.15.90:0/3391229622 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:21:30.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:30.928+0000 7f65667fc700 10 client.?.objecter ms_handle_connect 0x7f657010a6d0 2024-04-14T15:21:30.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:30.928+0000 7f65667fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:21:30.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:30.928+0000 7f65667fc700 10 client.?.objecter ms_handle_connect 0x7f65701023f0 2024-04-14T15:21:30.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:30.928+0000 7f65667fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:21:30.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:30.928+0000 7f65667fc700 10 client.?.objecter ms_handle_connect 0x7f657010e270 2024-04-14T15:21:30.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:30.928+0000 7f65667fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:21:30.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:30.928+0000 7f65667fc700 1 -- 172.21.15.90:0/3391229622 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2151431895 0 0) 0x7f65701abec0 con 0x7f657010a6d0 2024-04-14T15:21:30.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:30.928+0000 7f65667fc700 1 -- 172.21.15.90:0/3391229622 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f655c0035b0 con 0x7f657010a6d0 2024-04-14T15:21:30.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:30.928+0000 7f65667fc700 1 -- 172.21.15.90:0/3391229622 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2411892419 0 0) 0x7f65701a9a60 con 0x7f65701023f0 2024-04-14T15:21:30.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:30.928+0000 7f65667fc700 1 -- 172.21.15.90:0/3391229622 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f65701abec0 con 0x7f65701023f0 2024-04-14T15:21:30.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:30.928+0000 7f65667fc700 1 -- 172.21.15.90:0/3391229622 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3486098936 0 0) 0x7f65701aac80 con 0x7f657010e270 2024-04-14T15:21:30.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:30.928+0000 7f65667fc700 1 -- 172.21.15.90:0/3391229622 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f65701a9a60 con 0x7f657010e270 2024-04-14T15:21:30.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:30.929+0000 7f65667fc700 1 -- 172.21.15.90:0/3391229622 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2524195546 0 0) 0x7f65701abec0 con 0x7f65701023f0 2024-04-14T15:21:30.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:30.929+0000 7f65667fc700 1 -- 172.21.15.90:0/3391229622 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f65701aac80 con 0x7f65701023f0 2024-04-14T15:21:30.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:30.929+0000 7f65667fc700 1 -- 172.21.15.90:0/3391229622 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3835631998 0 0) 0x7f65701a9a60 con 0x7f657010e270 2024-04-14T15:21:30.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:30.929+0000 7f65667fc700 1 -- 172.21.15.90:0/3391229622 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f65701abec0 con 0x7f657010e270 2024-04-14T15:21:30.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:30.929+0000 7f65667fc700 1 -- 172.21.15.90:0/3391229622 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f656c004050 con 0x7f65701023f0 2024-04-14T15:21:30.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:30.929+0000 7f65667fc700 1 -- 172.21.15.90:0/3391229622 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2800194343 0 0) 0x7f655c0035b0 con 0x7f657010a6d0 2024-04-14T15:21:30.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:30.929+0000 7f65667fc700 1 -- 172.21.15.90:0/3391229622 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f65701a9a60 con 0x7f657010a6d0 2024-04-14T15:21:30.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:30.929+0000 7f65667fc700 1 -- 172.21.15.90:0/3391229622 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f6568004b00 con 0x7f657010e270 2024-04-14T15:21:30.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:30.929+0000 7f65667fc700 1 -- 172.21.15.90:0/3391229622 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f65600017a0 con 0x7f657010a6d0 2024-04-14T15:21:30.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:30.929+0000 7f65667fc700 1 -- 172.21.15.90:0/3391229622 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1982637733 0 0) 0x7f65701aac80 con 0x7f65701023f0 2024-04-14T15:21:30.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:30.929+0000 7f65667fc700 1 -- 172.21.15.90:0/3391229622 >> v1:172.21.15.90:6790/0 conn(0x7f657010a6d0 legacy=0x7f65701a4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:30.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:30.929+0000 7f65667fc700 1 -- 172.21.15.90:0/3391229622 >> v1:172.21.15.92:6789/0 conn(0x7f657010e270 legacy=0x7f65701a8210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:30.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:30.929+0000 7f65667fc700 1 -- 172.21.15.90:0/3391229622 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65701ad100 con 0x7f65701023f0 2024-04-14T15:21:30.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:30.929+0000 7f65776c1700 1 -- 172.21.15.90:0/3391229622 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f65701a9c90 con 0x7f65701023f0 2024-04-14T15:21:30.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:30.929+0000 7f65776c1700 10 client.37587.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:21:30.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:30.929+0000 7f65776c1700 1 -- 172.21.15.90:0/3391229622 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f65701aa150 con 0x7f65701023f0 2024-04-14T15:21:30.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:30.929+0000 7f65776c1700 1 librados: init done 2024-04-14T15:21:30.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:30.929+0000 7f65667fc700 1 -- 172.21.15.90:0/3391229622 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f656c004430 con 0x7f65701023f0 2024-04-14T15:21:30.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:30.929+0000 7f65667fc700 1 -- 172.21.15.90:0/3391229622 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f656c005ce0 con 0x7f65701023f0 2024-04-14T15:21:30.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:30.930+0000 7f65776c1700 1 -- 172.21.15.90:0/3391229622 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6554005120 con 0x7f65701023f0 2024-04-14T15:21:30.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:30.930+0000 7f65667fc700 1 -- 172.21.15.90:0/3391229622 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f656c01c6f0 con 0x7f65701023f0 2024-04-14T15:21:30.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:30.931+0000 7f65667fc700 1 -- 172.21.15.90:0/3391229622 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(561..561 src has 467..561) ==== 8224+0+0 (unknown 1548988565 0 0) 0x7f656c08aa00 con 0x7f65701023f0 2024-04-14T15:21:30.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:30.931+0000 7f65667fc700 10 client.37587.objecter ms_dispatch 0x7f6570001550 osd_map(561..561 src has 467..561) 2024-04-14T15:21:30.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:30.931+0000 7f65667fc700 3 client.37587.objecter handle_osd_map got epochs [561,561] > 0 2024-04-14T15:21:30.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:30.931+0000 7f65667fc700 3 client.37587.objecter handle_osd_map decoding full epoch 561 2024-04-14T15:21:30.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:30.931+0000 7f65667fc700 20 client.37587.objecter dump_active .. 0 homeless 2024-04-14T15:21:30.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:30.931+0000 7f65667fc700 10 client.37587.objecter ms_handle_connect 0x7f655c0713b0 2024-04-14T15:21:30.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:30.937+0000 7f65667fc700 1 -- 172.21.15.90:0/3391229622 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f656c0573f0 con 0x7f65701023f0 2024-04-14T15:21:30.952 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:21:30.973 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:21:30.981 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:21:31.007 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:31.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:31.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:21:31.109 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:21:31.132 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:31.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:31.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:21:31.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:31.167+0000 7f65776c1700 1 -- 172.21.15.90:0/3391229622 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7f6554005410 con 0x7f65701023f0 2024-04-14T15:21:31.235 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:31.259 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:31.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:31.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:31.360 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:21:31.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:31 smithi092 ceph-mon[33966]: 110.15 scrub starts 2024-04-14T15:21:31.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:31 smithi092 ceph-mon[33966]: 110.15 scrub ok 2024-04-14T15:21:31.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:31 smithi092 ceph-mon[33966]: osdmap e562: 8 total, 8 up, 7 in 2024-04-14T15:21:31.384 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:31.384 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:31 smithi090 ceph-mon[41670]: 110.15 scrub starts 2024-04-14T15:21:31.384 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:31 smithi090 ceph-mon[41670]: 110.15 scrub ok 2024-04-14T15:21:31.384 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:31 smithi090 ceph-mon[41670]: osdmap e562: 8 total, 8 up, 7 in 2024-04-14T15:21:31.384 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:31 smithi090 ceph-mon[30399]: 110.15 scrub starts 2024-04-14T15:21:31.384 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:31 smithi090 ceph-mon[30399]: 110.15 scrub ok 2024-04-14T15:21:31.385 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:31 smithi090 ceph-mon[30399]: osdmap e562: 8 total, 8 up, 7 in 2024-04-14T15:21:31.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:31.385 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:21:31.486 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:21:31.510 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:31.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:31.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:21:31.611 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:21:31.634 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:31.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:31.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:21:31.736 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:21:31.761 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:31.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:31.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:21:31.862 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:21:31.886 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:31.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:31.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:21:31.987 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:32.010 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:32.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:32.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:32.112 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:21:32.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:32.120+0000 7f65667fc700 1 -- 172.21.15.90:0/3391229622 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v563) ==== 87+0+0 (unknown 588096026 0 0) 0x7f656c05b0a0 con 0x7f65701023f0 2024-04-14T15:21:32.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:32.122+0000 7f65776c1700 10 librados: watch_flush enter 2024-04-14T15:21:32.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:32.122+0000 7f65776c1700 10 librados: watch_flush exit 2024-04-14T15:21:32.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:32.122+0000 7f65776c1700 20 client.37587.objecter shutdown clearing up homeless session... 2024-04-14T15:21:32.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:32.122+0000 7f65776c1700 10 client.37587.objecter successfully canceled tick 2024-04-14T15:21:32.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:32.122+0000 7f65776c1700 1 -- 172.21.15.90:0/3391229622 >> v1:172.21.15.90:6800/420935427 conn(0x7f655c0713b0 legacy=0x7f655c073870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:32.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:32.122+0000 7f65776c1700 1 -- 172.21.15.90:0/3391229622 >> v1:172.21.15.90:6789/0 conn(0x7f65701023f0 legacy=0x7f65701018d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:32.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:32.122+0000 7f65776c1700 1 -- 172.21.15.90:0/3391229622 shutdown_connections 2024-04-14T15:21:32.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:32.122+0000 7f65776c1700 1 -- 172.21.15.90:0/3391229622 >> 172.21.15.90:0/3391229622 conn(0x7f6570076aa0 msgr2=0x7f6570107de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:21:32.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:32.122+0000 7f65776c1700 1 -- 172.21.15.90:0/3391229622 shutdown_connections 2024-04-14T15:21:32.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:32.122+0000 7f65776c1700 1 -- 172.21.15.90:0/3391229622 wait complete. 2024-04-14T15:21:32.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:32.122+0000 7f65776c1700 1 librados: shutdown 2024-04-14T15:21:32.125 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is set 2024-04-14T15:21:32.140 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:32.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:32.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:21:32.242 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:21:32.265 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:32.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:32.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:21:32.369 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:21:32.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:32 smithi092 ceph-mon[33966]: 1.14 scrub starts 2024-04-14T15:21:32.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:32 smithi092 ceph-mon[33966]: 1.14 scrub ok 2024-04-14T15:21:32.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:32 smithi092 ceph-mon[33966]: pgmap v820: 248 pgs: 1 active+clean+snaptrim, 247 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-14T15:21:32.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:32 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3391229622' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:21:32.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:32 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:21:32.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:32 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:21:32.393 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:32.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:32.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:21:32.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:32 smithi090 ceph-mon[41670]: 1.14 scrub starts 2024-04-14T15:21:32.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:32 smithi090 ceph-mon[41670]: 1.14 scrub ok 2024-04-14T15:21:32.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:32 smithi090 ceph-mon[41670]: pgmap v820: 248 pgs: 1 active+clean+snaptrim, 247 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-14T15:21:32.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:32 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3391229622' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:21:32.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:32 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:21:32.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:32 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:21:32.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:32 smithi090 ceph-mon[30399]: 1.14 scrub starts 2024-04-14T15:21:32.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:32 smithi090 ceph-mon[30399]: 1.14 scrub ok 2024-04-14T15:21:32.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:32 smithi090 ceph-mon[30399]: pgmap v820: 248 pgs: 1 active+clean+snaptrim, 247 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-14T15:21:32.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:32 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3391229622' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:21:32.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:32 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:21:32.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:32 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:21:32.496 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:32.519 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:32.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:32.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:32.621 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:21:32.644 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:32.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:32.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:21:32.746 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:32.764 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-14T15:21:32.769 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:32.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:32.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:32.871 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:21:32.894 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:32.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:32.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:21:32.996 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:21:33.019 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:33.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:33.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:21:33.121 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:21:33.144 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:33.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:33.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:21:33.246 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:21:33.269 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:33.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:33.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:21:33.371 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:21:33.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:33 smithi092 ceph-mon[33966]: 2.a scrub starts 2024-04-14T15:21:33.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:33 smithi092 ceph-mon[33966]: 2.a scrub ok 2024-04-14T15:21:33.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:33 smithi092 ceph-mon[33966]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:21:33.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:33 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3391229622' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-14T15:21:33.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:33 smithi092 ceph-mon[33966]: osdmap e563: 8 total, 8 up, 7 in 2024-04-14T15:21:33.380 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:33 smithi090 ceph-mon[41670]: 2.a scrub starts 2024-04-14T15:21:33.380 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:33 smithi090 ceph-mon[41670]: 2.a scrub ok 2024-04-14T15:21:33.381 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:33 smithi090 ceph-mon[41670]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:21:33.381 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:33 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3391229622' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-14T15:21:33.381 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:33 smithi090 ceph-mon[41670]: osdmap e563: 8 total, 8 up, 7 in 2024-04-14T15:21:33.381 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:33 smithi090 ceph-mon[30399]: 2.a scrub starts 2024-04-14T15:21:33.381 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:33 smithi090 ceph-mon[30399]: 2.a scrub ok 2024-04-14T15:21:33.381 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:33 smithi090 ceph-mon[30399]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:21:33.381 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:33 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3391229622' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-14T15:21:33.381 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:33 smithi090 ceph-mon[30399]: osdmap e563: 8 total, 8 up, 7 in 2024-04-14T15:21:33.395 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:33.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:33.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:21:33.497 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:21:33.520 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:33.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:33.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:21:33.622 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:33.646 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:33.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:33.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:33.748 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:21:33.771 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:33.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:33.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:21:33.873 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:21:33.896 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:33.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:33.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:21:33.998 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:21:34.022 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:34.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:34.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:21:34.124 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:21:34.147 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:34.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:34.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:21:34.228 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:34 smithi090 ceph-mon[41670]: pgmap v822: 248 pgs: 1 active+clean+snaptrim, 247 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-14T15:21:34.229 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:21:33 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:21:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:21:34.229 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:34 smithi090 ceph-mon[30399]: pgmap v822: 248 pgs: 1 active+clean+snaptrim, 247 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-14T15:21:34.249 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:21:34.273 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:34.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:34.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:21:34.375 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:21:34.399 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:34.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:34.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:21:34.501 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:21:34.525 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:34.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:34.526 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:21:34.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:34 smithi092 ceph-mon[33966]: pgmap v822: 248 pgs: 1 active+clean+snaptrim, 247 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-14T15:21:34.627 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:21:34.650 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:34.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:34.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:21:34.752 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:34.777 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:34.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:34.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:34.879 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:21:34.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:34.887+0000 7f6534731700 1 -- 172.21.15.90:0/4158768338 >> v1:172.21.15.90:6789/0 conn(0x7f652c10b6f0 legacy=0x7f652c10dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:34.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:34.888+0000 7f6534731700 1 -- 172.21.15.90:0/4158768338 shutdown_connections 2024-04-14T15:21:34.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:34.888+0000 7f6534731700 1 -- 172.21.15.90:0/4158768338 >> 172.21.15.90:0/4158768338 conn(0x7f652c076aa0 msgr2=0x7f652c076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:21:34.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:34.888+0000 7f6534731700 1 -- 172.21.15.90:0/4158768338 shutdown_connections 2024-04-14T15:21:34.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:34.888+0000 7f6534731700 1 -- 172.21.15.90:0/4158768338 wait complete. 2024-04-14T15:21:34.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:34.888+0000 7f6534731700 1 librados: starting msgr at 2024-04-14T15:21:34.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:34.888+0000 7f6534731700 1 librados: starting objecter 2024-04-14T15:21:34.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:34.888+0000 7f6534731700 1 Processor -- start 2024-04-14T15:21:34.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:34.888+0000 7f6534731700 1 -- start start 2024-04-14T15:21:34.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:34.888+0000 7f6534731700 1 librados: setting wanted keys 2024-04-14T15:21:34.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:34.888+0000 7f6534731700 1 librados: calling monclient init 2024-04-14T15:21:34.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:34.889+0000 7f6534731700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f652c103e90 con 0x7f652c10f2f0 2024-04-14T15:21:34.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:34.889+0000 7f6534731700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f652c1040a0 con 0x7f652c108980 2024-04-14T15:21:34.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:34.889+0000 7f6534731700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f652c1ac300 con 0x7f652c10b6f0 2024-04-14T15:21:34.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:34.889+0000 7f6532cce700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f652c10f2f0 0x7f652c1a8bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:45676/0 (socket says 172.21.15.90:45676) 2024-04-14T15:21:34.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:34.889+0000 7f6532cce700 1 -- 172.21.15.90:0/1507913373 learned_addr learned my addr 172.21.15.90:0/1507913373 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:21:34.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:34.889+0000 7f65237fe700 10 client.?.objecter ms_handle_connect 0x7f652c10f2f0 2024-04-14T15:21:34.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:34.889+0000 7f65237fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:21:34.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:34.889+0000 7f65237fe700 10 client.?.objecter ms_handle_connect 0x7f652c10b6f0 2024-04-14T15:21:34.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:34.889+0000 7f65237fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:21:34.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:34.889+0000 7f65237fe700 10 client.?.objecter ms_handle_connect 0x7f652c108980 2024-04-14T15:21:34.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:34.889+0000 7f65237fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:21:34.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:34.889+0000 7f65237fe700 1 -- 172.21.15.90:0/1507913373 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2030348685 0 0) 0x7f652c103e90 con 0x7f652c10f2f0 2024-04-14T15:21:34.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:34.889+0000 7f65237fe700 1 -- 172.21.15.90:0/1507913373 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f65180035b0 con 0x7f652c10f2f0 2024-04-14T15:21:34.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:34.889+0000 7f65237fe700 1 -- 172.21.15.90:0/1507913373 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2251767406 0 0) 0x7f652c1ac300 con 0x7f652c10b6f0 2024-04-14T15:21:34.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:34.889+0000 7f65237fe700 1 -- 172.21.15.90:0/1507913373 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f652c103e90 con 0x7f652c10b6f0 2024-04-14T15:21:34.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:34.889+0000 7f65237fe700 1 -- 172.21.15.90:0/1507913373 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1358590499 0 0) 0x7f652c1040a0 con 0x7f652c108980 2024-04-14T15:21:34.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:34.889+0000 7f65237fe700 1 -- 172.21.15.90:0/1507913373 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f652c1ac300 con 0x7f652c108980 2024-04-14T15:21:34.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:34.890+0000 7f65237fe700 1 -- 172.21.15.90:0/1507913373 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 170522684 0 0) 0x7f652c1ac300 con 0x7f652c108980 2024-04-14T15:21:34.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:34.890+0000 7f65237fe700 1 -- 172.21.15.90:0/1507913373 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f652c1040a0 con 0x7f652c108980 2024-04-14T15:21:34.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:34.890+0000 7f65237fe700 1 -- 172.21.15.90:0/1507913373 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f6528001870 con 0x7f652c108980 2024-04-14T15:21:34.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:34.890+0000 7f65237fe700 1 -- 172.21.15.90:0/1507913373 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1216183492 0 0) 0x7f652c103e90 con 0x7f652c10b6f0 2024-04-14T15:21:34.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:34.890+0000 7f65237fe700 1 -- 172.21.15.90:0/1507913373 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f652c1ac300 con 0x7f652c10b6f0 2024-04-14T15:21:34.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:34.890+0000 7f65237fe700 1 -- 172.21.15.90:0/1507913373 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 648426639 0 0) 0x7f65180035b0 con 0x7f652c10f2f0 2024-04-14T15:21:34.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:34.890+0000 7f65237fe700 1 -- 172.21.15.90:0/1507913373 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f652c103e90 con 0x7f652c10f2f0 2024-04-14T15:21:34.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:34.890+0000 7f65237fe700 1 -- 172.21.15.90:0/1507913373 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f651c001430 con 0x7f652c10b6f0 2024-04-14T15:21:34.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:34.890+0000 7f65237fe700 1 -- 172.21.15.90:0/1507913373 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f6524001bb0 con 0x7f652c10f2f0 2024-04-14T15:21:34.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:34.890+0000 7f65237fe700 1 -- 172.21.15.90:0/1507913373 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1296530343 0 0) 0x7f652c1040a0 con 0x7f652c108980 2024-04-14T15:21:34.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:34.890+0000 7f65237fe700 1 -- 172.21.15.90:0/1507913373 >> v1:172.21.15.90:6790/0 conn(0x7f652c10b6f0 legacy=0x7f652c102970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:34.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:34.890+0000 7f65237fe700 1 -- 172.21.15.90:0/1507913373 >> v1:172.21.15.90:6789/0 conn(0x7f652c10f2f0 legacy=0x7f652c1a8bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:34.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:34.890+0000 7f65237fe700 1 -- 172.21.15.90:0/1507913373 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f652c1ad4e0 con 0x7f652c108980 2024-04-14T15:21:34.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:34.890+0000 7f6534731700 1 -- 172.21.15.90:0/1507913373 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f652c1ac530 con 0x7f652c108980 2024-04-14T15:21:34.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:34.890+0000 7f6534731700 10 client.49426.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:21:34.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:34.890+0000 7f6534731700 1 -- 172.21.15.90:0/1507913373 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f652c1ac9f0 con 0x7f652c108980 2024-04-14T15:21:34.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:34.890+0000 7f6534731700 1 librados: init done 2024-04-14T15:21:34.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:34.890+0000 7f65237fe700 1 -- 172.21.15.90:0/1507913373 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f6528003b40 con 0x7f652c108980 2024-04-14T15:21:34.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:34.890+0000 7f65237fe700 1 -- 172.21.15.90:0/1507913373 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f6528004b60 con 0x7f652c108980 2024-04-14T15:21:34.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:34.891+0000 7f6534731700 1 -- 172.21.15.90:0/1507913373 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6510005120 con 0x7f652c108980 2024-04-14T15:21:34.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:34.892+0000 7f65237fe700 1 -- 172.21.15.90:0/1507913373 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f652801b570 con 0x7f652c108980 2024-04-14T15:21:34.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:34.892+0000 7f65237fe700 1 -- 172.21.15.90:0/1507913373 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(563..563 src has 467..563) ==== 8196+0+0 (unknown 4161504076 0 0) 0x7f6528089690 con 0x7f652c108980 2024-04-14T15:21:34.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:34.892+0000 7f65237fe700 10 client.49426.objecter ms_dispatch 0x7f652c001550 osd_map(563..563 src has 467..563) 2024-04-14T15:21:34.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:34.892+0000 7f65237fe700 3 client.49426.objecter handle_osd_map got epochs [563,563] > 0 2024-04-14T15:21:34.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:34.892+0000 7f65237fe700 3 client.49426.objecter handle_osd_map decoding full epoch 563 2024-04-14T15:21:34.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:34.892+0000 7f65237fe700 20 client.49426.objecter dump_active .. 0 homeless 2024-04-14T15:21:34.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:34.892+0000 7f65237fe700 10 client.49426.objecter ms_handle_connect 0x7f6518071460 2024-04-14T15:21:34.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:34.897+0000 7f65237fe700 1 -- 172.21.15.90:0/1507913373 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f652805e5f0 con 0x7f652c108980 2024-04-14T15:21:34.908 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:34.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:34.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:21:35.010 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:21:35.033 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:35.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:35.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:21:35.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:35.103+0000 7f6534731700 1 -- 172.21.15.90:0/1507913373 --> v1:172.21.15.90:6800/420935427 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6510000b90 con 0x7f6518071460 2024-04-14T15:21:35.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:35.122+0000 7f65237fe700 1 -- 172.21.15.90:0/1507913373 <== mgr.14610 v1:172.21.15.90:6800/420935427 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+636017 (unknown 2965378022 0 3526781628) 0x7f6510000b90 con 0x7f6518071460 2024-04-14T15:21:35.130 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:21:35.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:35.126+0000 7f6534731700 10 librados: watch_flush enter 2024-04-14T15:21:35.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:35.126+0000 7f6534731700 10 librados: watch_flush exit 2024-04-14T15:21:35.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:35.126+0000 7f6534731700 20 client.49426.objecter shutdown clearing up homeless session... 2024-04-14T15:21:35.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:35.126+0000 7f6534731700 10 client.49426.objecter successfully canceled tick 2024-04-14T15:21:35.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:35.126+0000 7f6534731700 1 -- 172.21.15.90:0/1507913373 >> v1:172.21.15.90:6800/420935427 conn(0x7f6518071460 legacy=0x7f6518073920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:35.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:35.126+0000 7f6534731700 1 -- 172.21.15.90:0/1507913373 >> v1:172.21.15.92:6789/0 conn(0x7f652c108980 legacy=0x7f652c102260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:35.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:35.126+0000 7f6534731700 1 -- 172.21.15.90:0/1507913373 shutdown_connections 2024-04-14T15:21:35.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:35.126+0000 7f6534731700 1 -- 172.21.15.90:0/1507913373 >> 172.21.15.90:0/1507913373 conn(0x7f652c076aa0 msgr2=0x7f652c111750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:21:35.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:35.126+0000 7f6534731700 1 -- 172.21.15.90:0/1507913373 shutdown_connections 2024-04-14T15:21:35.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:35.126+0000 7f6534731700 1 -- 172.21.15.90:0/1507913373 wait complete. 2024-04-14T15:21:35.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:35.126+0000 7f6534731700 1 librados: shutdown 2024-04-14T15:21:35.137 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-14T15:21:35.137 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:21:35.164 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:35.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:35.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:21:35.266 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:21:35.289 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:35.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:35.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:21:35.391 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:21:35.408 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-14T15:21:35.418 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:35.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:35.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:21:35.520 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:21:35.545 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:35.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:35.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:21:35.622 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:21:35 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:21:35.647 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:21:35.673 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:35.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:35.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:21:35.774 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:21:35.798 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:35.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:35.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:21:35.900 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:21:35.924 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:35.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:35.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:21:35.985 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:21:36.025 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:21:36.048 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:36.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:36.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:21:36.150 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:21:36.173 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:36.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:36.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:21:36.275 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:21:36.298 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:36.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:36.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:21:36.400 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:36.428 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:36.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:36.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:36.530 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:21:36.557 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:36.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:36.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:21:36.558 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:21:36.566 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":822,"stamp":"2024-04-14T15:21:33.127643+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8811231,"num_objects":401,"num_object_clones":15,"num_object_copies":1203,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":400,"num_whiteouts":0,"num_read":2077,"num_read_kb":1669,"num_write":1531,"num_write_kb":12871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":332,"num_bytes_recovered":597473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":27,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4447,"ondisk_log_size":4447,"up":744,"acting":744,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":744,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":518032,"kb_used_data":45756,"kb_used_omap":10,"kb_used_meta":472181,"kb_avail":655698032,"statfs":{"total":671965249536,"available":671434784768,"internally_reserved":0,"allocated":46854144,"data_stored":33199150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11156,"internal_metadata":483513452},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-10,"num_objects":-1,"num_object_clones":-1,"num_object_copies":-3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001364"},"pg_stats":[{"pgid":"313.1f","version":"0'0","reported_seq":31,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.114763+0000","last_change":"2024-04-14T15:21:16.047112+0000","last_active":"2024-04-14T15:21:28.114763+0000","last_peered":"2024-04-14T15:21:28.114763+0000","last_clean":"2024-04-14T15:21:28.114763+0000","last_became_active":"2024-04-14T15:21:16.046674+0000","last_became_peered":"2024-04-14T15:21:16.046674+0000","last_unstale":"2024-04-14T15:21:28.114763+0000","last_undegraded":"2024-04-14T15:21:28.114763+0000","last_fullsized":"2024-04-14T15:21:28.114763+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":550,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:15.012837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:21:15.012837+0000","last_clean_scrub_stamp":"2024-04-14T15:21:15.012837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:31.229033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"313.1e","version":"0'0","reported_seq":31,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.114530+0000","last_change":"2024-04-14T15:21:16.051048+0000","last_active":"2024-04-14T15:21:28.114530+0000","last_peered":"2024-04-14T15:21:28.114530+0000","last_clean":"2024-04-14T15:21:28.114530+0000","last_became_active":"2024-04-14T15:21:16.046970+0000","last_became_peered":"2024-04-14T15:21:16.046970+0000","last_unstale":"2024-04-14T15:21:28.114530+0000","last_undegraded":"2024-04-14T15:21:28.114530+0000","last_fullsized":"2024-04-14T15:21:28.114530+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":550,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:15.012837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:21:15.012837+0000","last_clean_scrub_stamp":"2024-04-14T15:21:15.012837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:34.015626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"313.1d","version":"553'2","reported_seq":33,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.114521+0000","last_change":"2024-04-14T15:21:16.051705+0000","last_active":"2024-04-14T15:21:28.114521+0000","last_peered":"2024-04-14T15:21:28.114521+0000","last_clean":"2024-04-14T15:21:28.114521+0000","last_became_active":"2024-04-14T15:21:16.051371+0000","last_became_peered":"2024-04-14T15:21:16.051371+0000","last_unstale":"2024-04-14T15:21:28.114521+0000","last_undegraded":"2024-04-14T15:21:28.114521+0000","last_fullsized":"2024-04-14T15:21:28.114521+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":550,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:15.012837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:21:15.012837+0000","last_clean_scrub_stamp":"2024-04-14T15:21:15.012837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:16.651843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"313.1c","version":"0'0","reported_seq":31,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.113461+0000","last_change":"2024-04-14T15:21:16.047842+0000","last_active":"2024-04-14T15:21:28.113461+0000","last_peered":"2024-04-14T15:21:28.113461+0000","last_clean":"2024-04-14T15:21:28.113461+0000","last_became_active":"2024-04-14T15:21:16.047491+0000","last_became_peered":"2024-04-14T15:21:16.047491+0000","last_unstale":"2024-04-14T15:21:28.113461+0000","last_undegraded":"2024-04-14T15:21:28.113461+0000","last_fullsized":"2024-04-14T15:21:28.113461+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":550,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:15.012837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:21:15.012837+0000","last_clean_scrub_stamp":"2024-04-14T15:21:15.012837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:26.140266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"313.1b","version":"0'0","reported_seq":31,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.107566+0000","last_change":"2024-04-14T15:21:16.052349+0000","last_active":"2024-04-14T15:21:28.107566+0000","last_peered":"2024-04-14T15:21:28.107566+0000","last_clean":"2024-04-14T15:21:28.107566+0000","last_became_active":"2024-04-14T15:21:16.050722+0000","last_became_peered":"2024-04-14T15:21:16.050722+0000","last_unstale":"2024-04-14T15:21:28.107566+0000","last_undegraded":"2024-04-14T15:21:28.107566+0000","last_fullsized":"2024-04-14T15:21:28.107566+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":550,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:15.012837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:21:15.012837+0000","last_clean_scrub_stamp":"2024-04-14T15:21:15.012837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:40.762144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"313.1a","version":"0'0","reported_seq":31,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.114003+0000","last_change":"2024-04-14T15:21:16.036609+0000","last_active":"2024-04-14T15:21:28.114003+0000","last_peered":"2024-04-14T15:21:28.114003+0000","last_clean":"2024-04-14T15:21:28.114003+0000","last_became_active":"2024-04-14T15:21:16.036345+0000","last_became_peered":"2024-04-14T15:21:16.036345+0000","last_unstale":"2024-04-14T15:21:28.114003+0000","last_undegraded":"2024-04-14T15:21:28.114003+0000","last_fullsized":"2024-04-14T15:21:28.114003+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":550,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:15.012837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:21:15.012837+0000","last_clean_scrub_stamp":"2024-04-14T15:21:15.012837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:42.920794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"313.19","version":"553'3","reported_seq":34,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.105104+0000","last_change":"2024-04-14T15:21:16.040501+0000","last_active":"2024-04-14T15:21:28.105104+0000","last_peered":"2024-04-14T15:21:28.105104+0000","last_clean":"2024-04-14T15:21:28.105104+0000","last_became_active":"2024-04-14T15:21:16.040173+0000","last_became_peered":"2024-04-14T15:21:16.040173+0000","last_unstale":"2024-04-14T15:21:28.105104+0000","last_undegraded":"2024-04-14T15:21:28.105104+0000","last_fullsized":"2024-04-14T15:21:28.105104+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":550,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:15.012837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:21:15.012837+0000","last_clean_scrub_stamp":"2024-04-14T15:21:15.012837+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:34.551749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"313.18","version":"0'0","reported_seq":31,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.104994+0000","last_change":"2024-04-14T15:21:16.049922+0000","last_active":"2024-04-14T15:21:28.104994+0000","last_peered":"2024-04-14T15:21:28.104994+0000","last_clean":"2024-04-14T15:21:28.104994+0000","last_became_active":"2024-04-14T15:21:16.049602+0000","last_became_peered":"2024-04-14T15:21:16.049602+0000","last_unstale":"2024-04-14T15:21:28.104994+0000","last_undegraded":"2024-04-14T15:21:28.104994+0000","last_fullsized":"2024-04-14T15:21:28.104994+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":550,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:15.012837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:21:15.012837+0000","last_clean_scrub_stamp":"2024-04-14T15:21:15.012837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:31.750898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"313.17","version":"553'3","reported_seq":34,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.106684+0000","last_change":"2024-04-14T15:21:16.049934+0000","last_active":"2024-04-14T15:21:28.106684+0000","last_peered":"2024-04-14T15:21:28.106684+0000","last_clean":"2024-04-14T15:21:28.106684+0000","last_became_active":"2024-04-14T15:21:16.049608+0000","last_became_peered":"2024-04-14T15:21:16.049608+0000","last_unstale":"2024-04-14T15:21:28.106684+0000","last_undegraded":"2024-04-14T15:21:28.106684+0000","last_fullsized":"2024-04-14T15:21:28.106684+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":550,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:15.012837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:21:15.012837+0000","last_clean_scrub_stamp":"2024-04-14T15:21:15.012837+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:40.372832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"313.16","version":"0'0","reported_seq":31,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.115845+0000","last_change":"2024-04-14T15:21:16.043082+0000","last_active":"2024-04-14T15:21:28.115845+0000","last_peered":"2024-04-14T15:21:28.115845+0000","last_clean":"2024-04-14T15:21:28.115845+0000","last_became_active":"2024-04-14T15:21:16.042758+0000","last_became_peered":"2024-04-14T15:21:16.042758+0000","last_unstale":"2024-04-14T15:21:28.115845+0000","last_undegraded":"2024-04-14T15:21:28.115845+0000","last_fullsized":"2024-04-14T15:21:28.115845+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":550,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:15.012837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:21:15.012837+0000","last_clean_scrub_stamp":"2024-04-14T15:21:15.012837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:34.586307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"313.15","version":"561'4","reported_seq":35,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.115234+0000","last_change":"2024-04-14T15:21:16.047475+0000","last_active":"2024-04-14T15:21:28.115234+0000","last_peered":"2024-04-14T15:21:28.115234+0000","last_clean":"2024-04-14T15:21:28.115234+0000","last_became_active":"2024-04-14T15:21:16.047121+0000","last_became_peered":"2024-04-14T15:21:16.047121+0000","last_unstale":"2024-04-14T15:21:28.115234+0000","last_undegraded":"2024-04-14T15:21:28.115234+0000","last_fullsized":"2024-04-14T15:21:28.115234+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":550,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:15.012837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:21:15.012837+0000","last_clean_scrub_stamp":"2024-04-14T15:21:15.012837+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:30.161337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"313.14","version":"0'0","reported_seq":31,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.117330+0000","last_change":"2024-04-14T15:21:16.046470+0000","last_active":"2024-04-14T15:21:28.117330+0000","last_peered":"2024-04-14T15:21:28.117330+0000","last_clean":"2024-04-14T15:21:28.117330+0000","last_became_active":"2024-04-14T15:21:16.046156+0000","last_became_peered":"2024-04-14T15:21:16.046156+0000","last_unstale":"2024-04-14T15:21:28.117330+0000","last_undegraded":"2024-04-14T15:21:28.117330+0000","last_fullsized":"2024-04-14T15:21:28.117330+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":550,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:15.012837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:21:15.012837+0000","last_clean_scrub_stamp":"2024-04-14T15:21:15.012837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:33.195971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"313.13","version":"0'0","reported_seq":31,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.117490+0000","last_change":"2024-04-14T15:21:16.053763+0000","last_active":"2024-04-14T15:21:28.117490+0000","last_peered":"2024-04-14T15:21:28.117490+0000","last_clean":"2024-04-14T15:21:28.117490+0000","last_became_active":"2024-04-14T15:21:16.053159+0000","last_became_peered":"2024-04-14T15:21:16.053159+0000","last_unstale":"2024-04-14T15:21:28.117490+0000","last_undegraded":"2024-04-14T15:21:28.117490+0000","last_fullsized":"2024-04-14T15:21:28.117490+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":550,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:15.012837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:21:15.012837+0000","last_clean_scrub_stamp":"2024-04-14T15:21:15.012837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:35.327285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"313.12","version":"0'0","reported_seq":31,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.117278+0000","last_change":"2024-04-14T15:21:16.050167+0000","last_active":"2024-04-14T15:21:28.117278+0000","last_peered":"2024-04-14T15:21:28.117278+0000","last_clean":"2024-04-14T15:21:28.117278+0000","last_became_active":"2024-04-14T15:21:16.049937+0000","last_became_peered":"2024-04-14T15:21:16.049937+0000","last_unstale":"2024-04-14T15:21:28.117278+0000","last_undegraded":"2024-04-14T15:21:28.117278+0000","last_fullsized":"2024-04-14T15:21:28.117278+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":550,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:15.012837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:21:15.012837+0000","last_clean_scrub_stamp":"2024-04-14T15:21:15.012837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:27.705142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"313.11","version":"0'0","reported_seq":31,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.105127+0000","last_change":"2024-04-14T15:21:16.035697+0000","last_active":"2024-04-14T15:21:28.105127+0000","last_peered":"2024-04-14T15:21:28.105127+0000","last_clean":"2024-04-14T15:21:28.105127+0000","last_became_active":"2024-04-14T15:21:16.035399+0000","last_became_peered":"2024-04-14T15:21:16.035399+0000","last_unstale":"2024-04-14T15:21:28.105127+0000","last_undegraded":"2024-04-14T15:21:28.105127+0000","last_fullsized":"2024-04-14T15:21:28.105127+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":550,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:15.012837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:21:15.012837+0000","last_clean_scrub_stamp":"2024-04-14T15:21:15.012837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:29.520146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"313.10","version":"0'0","reported_seq":31,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.107203+0000","last_change":"2024-04-14T15:21:16.052626+0000","last_active":"2024-04-14T15:21:28.107203+0000","last_peered":"2024-04-14T15:21:28.107203+0000","last_clean":"2024-04-14T15:21:28.107203+0000","last_became_active":"2024-04-14T15:21:16.050625+0000","last_became_peered":"2024-04-14T15:21:16.050625+0000","last_unstale":"2024-04-14T15:21:28.107203+0000","last_undegraded":"2024-04-14T15:21:28.107203+0000","last_fullsized":"2024-04-14T15:21:28.107203+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":550,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:15.012837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:21:15.012837+0000","last_clean_scrub_stamp":"2024-04-14T15:21:15.012837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:28.805595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"313.f","version":"0'0","reported_seq":31,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.117847+0000","last_change":"2024-04-14T15:21:16.042184+0000","last_active":"2024-04-14T15:21:28.117847+0000","last_peered":"2024-04-14T15:21:28.117847+0000","last_clean":"2024-04-14T15:21:28.117847+0000","last_became_active":"2024-04-14T15:21:16.040868+0000","last_became_peered":"2024-04-14T15:21:16.040868+0000","last_unstale":"2024-04-14T15:21:28.117847+0000","last_undegraded":"2024-04-14T15:21:28.117847+0000","last_fullsized":"2024-04-14T15:21:28.117847+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":550,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:15.012837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:21:15.012837+0000","last_clean_scrub_stamp":"2024-04-14T15:21:15.012837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:37.846103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"313.e","version":"0'0","reported_seq":31,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.106725+0000","last_change":"2024-04-14T15:21:16.042697+0000","last_active":"2024-04-14T15:21:28.106725+0000","last_peered":"2024-04-14T15:21:28.106725+0000","last_clean":"2024-04-14T15:21:28.106725+0000","last_became_active":"2024-04-14T15:21:16.042075+0000","last_became_peered":"2024-04-14T15:21:16.042075+0000","last_unstale":"2024-04-14T15:21:28.106725+0000","last_undegraded":"2024-04-14T15:21:28.106725+0000","last_fullsized":"2024-04-14T15:21:28.106725+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":550,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:15.012837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:21:15.012837+0000","last_clean_scrub_stamp":"2024-04-14T15:21:15.012837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:38.586128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"313.d","version":"0'0","reported_seq":31,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.117553+0000","last_change":"2024-04-14T15:21:16.056940+0000","last_active":"2024-04-14T15:21:28.117553+0000","last_peered":"2024-04-14T15:21:28.117553+0000","last_clean":"2024-04-14T15:21:28.117553+0000","last_became_active":"2024-04-14T15:21:16.056686+0000","last_became_peered":"2024-04-14T15:21:16.056686+0000","last_unstale":"2024-04-14T15:21:28.117553+0000","last_undegraded":"2024-04-14T15:21:28.117553+0000","last_fullsized":"2024-04-14T15:21:28.117553+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":550,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:15.012837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:21:15.012837+0000","last_clean_scrub_stamp":"2024-04-14T15:21:15.012837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:23.007227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"313.c","version":"0'0","reported_seq":31,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.117354+0000","last_change":"2024-04-14T15:21:16.049807+0000","last_active":"2024-04-14T15:21:28.117354+0000","last_peered":"2024-04-14T15:21:28.117354+0000","last_clean":"2024-04-14T15:21:28.117354+0000","last_became_active":"2024-04-14T15:21:16.049468+0000","last_became_peered":"2024-04-14T15:21:16.049468+0000","last_unstale":"2024-04-14T15:21:28.117354+0000","last_undegraded":"2024-04-14T15:21:28.117354+0000","last_fullsized":"2024-04-14T15:21:28.117354+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":550,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:15.012837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:21:15.012837+0000","last_clean_scrub_stamp":"2024-04-14T15:21:15.012837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:34.013607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"313.b","version":"0'0","reported_seq":31,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.117537+0000","last_change":"2024-04-14T15:21:16.053472+0000","last_active":"2024-04-14T15:21:28.117537+0000","last_peered":"2024-04-14T15:21:28.117537+0000","last_clean":"2024-04-14T15:21:28.117537+0000","last_became_active":"2024-04-14T15:21:16.052594+0000","last_became_peered":"2024-04-14T15:21:16.052594+0000","last_unstale":"2024-04-14T15:21:28.117537+0000","last_undegraded":"2024-04-14T15:21:28.117537+0000","last_fullsized":"2024-04-14T15:21:28.117537+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":550,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:15.012837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:21:15.012837+0000","last_clean_scrub_stamp":"2024-04-14T15:21:15.012837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:38.025711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"313.a","version":"0'0","reported_seq":31,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.105326+0000","last_change":"2024-04-14T15:21:16.041809+0000","last_active":"2024-04-14T15:21:28.105326+0000","last_peered":"2024-04-14T15:21:28.105326+0000","last_clean":"2024-04-14T15:21:28.105326+0000","last_became_active":"2024-04-14T15:21:16.041486+0000","last_became_peered":"2024-04-14T15:21:16.041486+0000","last_unstale":"2024-04-14T15:21:28.105326+0000","last_undegraded":"2024-04-14T15:21:28.105326+0000","last_fullsized":"2024-04-14T15:21:28.105326+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":550,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:15.012837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:21:15.012837+0000","last_clean_scrub_stamp":"2024-04-14T15:21:15.012837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:20.854906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"313.9","version":"561'4","reported_seq":35,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.105292+0000","last_change":"2024-04-14T15:21:16.059880+0000","last_active":"2024-04-14T15:21:28.105292+0000","last_peered":"2024-04-14T15:21:28.105292+0000","last_clean":"2024-04-14T15:21:28.105292+0000","last_became_active":"2024-04-14T15:21:16.059586+0000","last_became_peered":"2024-04-14T15:21:16.059586+0000","last_unstale":"2024-04-14T15:21:28.105292+0000","last_undegraded":"2024-04-14T15:21:28.105292+0000","last_fullsized":"2024-04-14T15:21:28.105292+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":550,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:15.012837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:21:15.012837+0000","last_clean_scrub_stamp":"2024-04-14T15:21:15.012837+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:18.467768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"313.8","version":"0'0","reported_seq":31,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.105003+0000","last_change":"2024-04-14T15:21:16.056095+0000","last_active":"2024-04-14T15:21:28.105003+0000","last_peered":"2024-04-14T15:21:28.105003+0000","last_clean":"2024-04-14T15:21:28.105003+0000","last_became_active":"2024-04-14T15:21:16.055804+0000","last_became_peered":"2024-04-14T15:21:16.055804+0000","last_unstale":"2024-04-14T15:21:28.105003+0000","last_undegraded":"2024-04-14T15:21:28.105003+0000","last_fullsized":"2024-04-14T15:21:28.105003+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":550,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:15.012837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:21:15.012837+0000","last_clean_scrub_stamp":"2024-04-14T15:21:15.012837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:30.303283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"313.7","version":"0'0","reported_seq":31,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.106689+0000","last_change":"2024-04-14T15:21:16.046538+0000","last_active":"2024-04-14T15:21:28.106689+0000","last_peered":"2024-04-14T15:21:28.106689+0000","last_clean":"2024-04-14T15:21:28.106689+0000","last_became_active":"2024-04-14T15:21:16.046101+0000","last_became_peered":"2024-04-14T15:21:16.046101+0000","last_unstale":"2024-04-14T15:21:28.106689+0000","last_undegraded":"2024-04-14T15:21:28.106689+0000","last_fullsized":"2024-04-14T15:21:28.106689+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":550,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:15.012837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:21:15.012837+0000","last_clean_scrub_stamp":"2024-04-14T15:21:15.012837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:24.721771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"313.6","version":"0'0","reported_seq":31,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.120186+0000","last_change":"2024-04-14T15:21:16.048958+0000","last_active":"2024-04-14T15:21:28.120186+0000","last_peered":"2024-04-14T15:21:28.120186+0000","last_clean":"2024-04-14T15:21:28.120186+0000","last_became_active":"2024-04-14T15:21:16.048676+0000","last_became_peered":"2024-04-14T15:21:16.048676+0000","last_unstale":"2024-04-14T15:21:28.120186+0000","last_undegraded":"2024-04-14T15:21:28.120186+0000","last_fullsized":"2024-04-14T15:21:28.120186+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":550,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:15.012837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:21:15.012837+0000","last_clean_scrub_stamp":"2024-04-14T15:21:15.012837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:23.224940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"313.5","version":"0'0","reported_seq":31,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.115289+0000","last_change":"2024-04-14T15:21:16.038501+0000","last_active":"2024-04-14T15:21:28.115289+0000","last_peered":"2024-04-14T15:21:28.115289+0000","last_clean":"2024-04-14T15:21:28.115289+0000","last_became_active":"2024-04-14T15:21:16.038205+0000","last_became_peered":"2024-04-14T15:21:16.038205+0000","last_unstale":"2024-04-14T15:21:28.115289+0000","last_undegraded":"2024-04-14T15:21:28.115289+0000","last_fullsized":"2024-04-14T15:21:28.115289+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":550,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:15.012837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:21:15.012837+0000","last_clean_scrub_stamp":"2024-04-14T15:21:15.012837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:39.752935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"313.4","version":"0'0","reported_seq":31,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.111710+0000","last_change":"2024-04-14T15:21:16.053256+0000","last_active":"2024-04-14T15:21:28.111710+0000","last_peered":"2024-04-14T15:21:28.111710+0000","last_clean":"2024-04-14T15:21:28.111710+0000","last_became_active":"2024-04-14T15:21:16.052964+0000","last_became_peered":"2024-04-14T15:21:16.052964+0000","last_unstale":"2024-04-14T15:21:28.111710+0000","last_undegraded":"2024-04-14T15:21:28.111710+0000","last_fullsized":"2024-04-14T15:21:28.111710+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":550,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:15.012837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:21:15.012837+0000","last_clean_scrub_stamp":"2024-04-14T15:21:15.012837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:39.261834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"313.3","version":"0'0","reported_seq":31,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.110335+0000","last_change":"2024-04-14T15:21:16.048203+0000","last_active":"2024-04-14T15:21:28.110335+0000","last_peered":"2024-04-14T15:21:28.110335+0000","last_clean":"2024-04-14T15:21:28.110335+0000","last_became_active":"2024-04-14T15:21:16.047924+0000","last_became_peered":"2024-04-14T15:21:16.047924+0000","last_unstale":"2024-04-14T15:21:28.110335+0000","last_undegraded":"2024-04-14T15:21:28.110335+0000","last_fullsized":"2024-04-14T15:21:28.110335+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":550,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:15.012837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:21:15.012837+0000","last_clean_scrub_stamp":"2024-04-14T15:21:15.012837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:38.162177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"313.2","version":"0'0","reported_seq":31,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.116515+0000","last_change":"2024-04-14T15:21:16.046759+0000","last_active":"2024-04-14T15:21:28.116515+0000","last_peered":"2024-04-14T15:21:28.116515+0000","last_clean":"2024-04-14T15:21:28.116515+0000","last_became_active":"2024-04-14T15:21:16.046464+0000","last_became_peered":"2024-04-14T15:21:16.046464+0000","last_unstale":"2024-04-14T15:21:28.116515+0000","last_undegraded":"2024-04-14T15:21:28.116515+0000","last_fullsized":"2024-04-14T15:21:28.116515+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":550,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:15.012837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:21:15.012837+0000","last_clean_scrub_stamp":"2024-04-14T15:21:15.012837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:33.936263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"313.1","version":"0'0","reported_seq":31,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.117039+0000","last_change":"2024-04-14T15:21:16.058891+0000","last_active":"2024-04-14T15:21:28.117039+0000","last_peered":"2024-04-14T15:21:28.117039+0000","last_clean":"2024-04-14T15:21:28.117039+0000","last_became_active":"2024-04-14T15:21:16.057543+0000","last_became_peered":"2024-04-14T15:21:16.057543+0000","last_unstale":"2024-04-14T15:21:28.117039+0000","last_undegraded":"2024-04-14T15:21:28.117039+0000","last_fullsized":"2024-04-14T15:21:28.117039+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":550,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:15.012837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:21:15.012837+0000","last_clean_scrub_stamp":"2024-04-14T15:21:15.012837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:33.905016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"313.0","version":"0'0","reported_seq":31,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.116704+0000","last_change":"2024-04-14T15:21:16.057854+0000","last_active":"2024-04-14T15:21:28.116704+0000","last_peered":"2024-04-14T15:21:28.116704+0000","last_clean":"2024-04-14T15:21:28.116704+0000","last_became_active":"2024-04-14T15:21:16.057265+0000","last_became_peered":"2024-04-14T15:21:16.057265+0000","last_unstale":"2024-04-14T15:21:28.116704+0000","last_undegraded":"2024-04-14T15:21:28.116704+0000","last_fullsized":"2024-04-14T15:21:28.116704+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":550,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:15.012837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:21:15.012837+0000","last_clean_scrub_stamp":"2024-04-14T15:21:15.012837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:41.152289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"110.11","version":"528'45","reported_seq":1181,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.106354+0000","last_change":"2024-04-14T15:21:28.107050+0000","last_active":"2024-04-14T15:21:28.106354+0000","last_peered":"2024-04-14T15:21:28.106354+0000","last_clean":"2024-04-14T15:21:28.106354+0000","last_became_active":"2024-04-14T15:20:52.978090+0000","last_became_peered":"2024-04-14T15:20:52.978090+0000","last_unstale":"2024-04-14T15:21:28.106354+0000","last_undegraded":"2024-04-14T15:21:28.106354+0000","last_fullsized":"2024-04-14T15:21:28.106354+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"403'41","last_scrub_stamp":"2024-04-14T15:20:00.829932+0000","last_deep_scrub":"403'41","last_deep_scrub_stamp":"2024-04-14T15:17:56.911004+0000","last_clean_scrub_stamp":"2024-04-14T15:20:00.829932+0000","objects_scrubbed":2,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0016224849999999999,"stat_sum":{"num_bytes":2574,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":19,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":1359,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1f","length":"1"}]},{"pgid":"110.10","version":"519'159","reported_seq":1238,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.106279+0000","last_change":"2024-04-14T15:21:28.107055+0000","last_active":"2024-04-14T15:21:28.106279+0000","last_peered":"2024-04-14T15:21:28.106279+0000","last_clean":"2024-04-14T15:21:28.106279+0000","last_became_active":"2024-04-14T15:13:09.928915+0000","last_became_peered":"2024-04-14T15:13:09.928915+0000","last_unstale":"2024-04-14T15:21:28.106279+0000","last_undegraded":"2024-04-14T15:21:28.106279+0000","last_fullsized":"2024-04-14T15:21:28.106279+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"519'159","last_scrub_stamp":"2024-04-14T15:21:16.933615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:21:16.933615+0000","objects_scrubbed":11,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:20.502812+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0017170659999999999,"stat_sum":{"num_bytes":1646,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":49,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1f","length":"1"}]},{"pgid":"110.13","version":"516'60","reported_seq":1174,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.113176+0000","last_change":"2024-04-14T15:21:28.113540+0000","last_active":"2024-04-14T15:21:28.113176+0000","last_peered":"2024-04-14T15:21:28.113176+0000","last_clean":"2024-04-14T15:21:28.113176+0000","last_became_active":"2024-04-14T15:13:09.918152+0000","last_became_peered":"2024-04-14T15:13:09.918152+0000","last_unstale":"2024-04-14T15:21:28.113176+0000","last_undegraded":"2024-04-14T15:21:28.113176+0000","last_fullsized":"2024-04-14T15:21:28.113176+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'60","last_scrub_stamp":"2024-04-14T15:20:41.757524+0000","last_deep_scrub":"516'60","last_deep_scrub_stamp":"2024-04-14T15:20:41.757524+0000","last_clean_scrub_stamp":"2024-04-14T15:20:41.757524+0000","objects_scrubbed":7,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:00.735958+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.001946665,"stat_sum":{"num_bytes":57,"num_objects":7,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1f","length":"1"}]},{"pgid":"110.12","version":"510'81","reported_seq":1194,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.116508+0000","last_change":"2024-04-14T15:21:28.118180+0000","last_active":"2024-04-14T15:21:28.116508+0000","last_peered":"2024-04-14T15:21:28.116508+0000","last_clean":"2024-04-14T15:21:28.116508+0000","last_became_active":"2024-04-14T15:13:09.939566+0000","last_became_peered":"2024-04-14T15:13:09.939566+0000","last_unstale":"2024-04-14T15:21:28.116508+0000","last_undegraded":"2024-04-14T15:21:28.116508+0000","last_fullsized":"2024-04-14T15:21:28.116508+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"409'70","last_scrub_stamp":"2024-04-14T15:19:47.921684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:19:47.921684+0000","objects_scrubbed":9,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0025337519999999998,"stat_sum":{"num_bytes":89,"num_objects":11,"num_object_clones":1,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":10,"num_read_kb":3,"num_write":33,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1f","length":"1"}]},{"pgid":"110.15","version":"519'37","reported_seq":1145,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.120573+0000","last_change":"2024-04-14T15:21:28.121068+0000","last_active":"2024-04-14T15:21:28.120573+0000","last_peered":"2024-04-14T15:21:28.120573+0000","last_clean":"2024-04-14T15:21:28.120573+0000","last_became_active":"2024-04-14T15:13:09.960121+0000","last_became_peered":"2024-04-14T15:13:09.960121+0000","last_unstale":"2024-04-14T15:21:28.120573+0000","last_undegraded":"2024-04-14T15:21:28.120573+0000","last_fullsized":"2024-04-14T15:21:28.120573+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'34","last_scrub_stamp":"2024-04-14T15:18:36.013135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:18:36.013135+0000","objects_scrubbed":2,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00066767900000000002,"stat_sum":{"num_bytes":2332,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":15,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1f","length":"1"}]},{"pgid":"110.14","version":"510'91","reported_seq":1266,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.432668+0000","last_change":"2024-04-14T15:21:28.432705+0000","last_active":"2024-04-14T15:21:28.432668+0000","last_peered":"2024-04-14T15:21:28.432668+0000","last_clean":"2024-04-14T15:21:28.432668+0000","last_became_active":"2024-04-14T15:20:52.985628+0000","last_became_peered":"2024-04-14T15:20:52.985628+0000","last_unstale":"2024-04-14T15:21:28.432668+0000","last_undegraded":"2024-04-14T15:21:28.432668+0000","last_fullsized":"2024-04-14T15:21:28.432668+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"510'91","last_scrub_stamp":"2024-04-14T15:21:28.432608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:21:28.432608+0000","objects_scrubbed":7,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:39.761680+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00074881999999999998,"stat_sum":{"num_bytes":71,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":9,"num_read_kb":2,"num_write":33,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":71,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1f","length":"1"}]},{"pgid":"110.17","version":"519'17","reported_seq":1141,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.121112+0000","last_change":"2024-04-14T15:21:28.121372+0000","last_active":"2024-04-14T15:21:28.121112+0000","last_peered":"2024-04-14T15:21:28.121112+0000","last_clean":"2024-04-14T15:21:28.121112+0000","last_became_active":"2024-04-14T15:13:09.898521+0000","last_became_peered":"2024-04-14T15:13:09.898521+0000","last_unstale":"2024-04-14T15:21:28.121112+0000","last_undegraded":"2024-04-14T15:21:28.121112+0000","last_fullsized":"2024-04-14T15:21:28.121112+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'12","last_scrub_stamp":"2024-04-14T15:19:48.225130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:19:48.225130+0000","objects_scrubbed":4,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00075242500000000003,"stat_sum":{"num_bytes":724,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":3,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1f","length":"1"}]},{"pgid":"110.16","version":"528'41","reported_seq":989,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.112231+0000","last_change":"2024-04-14T15:21:28.112825+0000","last_active":"2024-04-14T15:21:28.112231+0000","last_peered":"2024-04-14T15:21:28.112231+0000","last_clean":"2024-04-14T15:21:28.112231+0000","last_became_active":"2024-04-14T15:15:21.889783+0000","last_became_peered":"2024-04-14T15:15:21.889783+0000","last_unstale":"2024-04-14T15:21:28.112231+0000","last_undegraded":"2024-04-14T15:21:28.112231+0000","last_fullsized":"2024-04-14T15:21:28.112231+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"518'38","last_scrub_stamp":"2024-04-14T15:20:42.941590+0000","last_deep_scrub":"518'38","last_deep_scrub_stamp":"2024-04-14T15:20:42.941590+0000","last_clean_scrub_stamp":"2024-04-14T15:20:42.941590+0000","objects_scrubbed":3,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:47.870613+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.00096776800000000004,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":14,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1f","length":"1"}]},{"pgid":"110.19","version":"510'112","reported_seq":1222,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.110146+0000","last_change":"2024-04-14T15:21:28.110457+0000","last_active":"2024-04-14T15:21:28.110146+0000","last_peered":"2024-04-14T15:21:28.110146+0000","last_clean":"2024-04-14T15:21:28.110146+0000","last_became_active":"2024-04-14T15:13:09.998139+0000","last_became_peered":"2024-04-14T15:13:09.998139+0000","last_unstale":"2024-04-14T15:21:28.110146+0000","last_undegraded":"2024-04-14T15:21:28.110146+0000","last_fullsized":"2024-04-14T15:21:28.110146+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'85","last_scrub_stamp":"2024-04-14T15:19:12.058127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:19:12.058127+0000","objects_scrubbed":5,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0.0014077130000000001,"stat_sum":{"num_bytes":85,"num_objects":8,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":18,"num_read_kb":9,"num_write":46,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1f","length":"1"}]},{"pgid":"110.18","version":"528'95","reported_seq":1203,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.110202+0000","last_change":"2024-04-14T15:21:28.110478+0000","last_active":"2024-04-14T15:21:28.110202+0000","last_peered":"2024-04-14T15:21:28.110202+0000","last_clean":"2024-04-14T15:21:28.110202+0000","last_became_active":"2024-04-14T15:13:09.998803+0000","last_became_peered":"2024-04-14T15:13:09.998803+0000","last_unstale":"2024-04-14T15:21:28.110202+0000","last_undegraded":"2024-04-14T15:21:28.110202+0000","last_fullsized":"2024-04-14T15:21:28.110202+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"528'95","last_scrub_stamp":"2024-04-14T15:21:17.545263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:21:17.545263+0000","objects_scrubbed":9,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:42.462798+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001114603,"stat_sum":{"num_bytes":2375,"num_objects":9,"num_object_clones":1,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":8,"num_read_kb":3,"num_write":31,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1f","length":"1"}]},{"pgid":"110.1b","version":"519'114","reported_seq":1223,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.113117+0000","last_change":"2024-04-14T15:21:28.113485+0000","last_active":"2024-04-14T15:21:28.113117+0000","last_peered":"2024-04-14T15:21:28.113117+0000","last_clean":"2024-04-14T15:21:28.113117+0000","last_became_active":"2024-04-14T15:13:09.946919+0000","last_became_peered":"2024-04-14T15:13:09.946919+0000","last_unstale":"2024-04-14T15:21:28.113117+0000","last_undegraded":"2024-04-14T15:21:28.113117+0000","last_fullsized":"2024-04-14T15:21:28.113117+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"510'113","last_scrub_stamp":"2024-04-14T15:20:39.682795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:20:39.682795+0000","objects_scrubbed":9,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:06.500184+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.002156477,"stat_sum":{"num_bytes":1308,"num_objects":10,"num_object_clones":2,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":16,"num_read_kb":5,"num_write":44,"num_write_kb":42,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1f","length":"1"}]},{"pgid":"110.1a","version":"528'41","reported_seq":864,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.111237+0000","last_change":"2024-04-14T15:21:28.112109+0000","last_active":"2024-04-14T15:21:28.111237+0000","last_peered":"2024-04-14T15:21:28.111237+0000","last_clean":"2024-04-14T15:21:28.111237+0000","last_became_active":"2024-04-14T15:20:52.979115+0000","last_became_peered":"2024-04-14T15:20:52.979115+0000","last_unstale":"2024-04-14T15:21:28.111237+0000","last_undegraded":"2024-04-14T15:21:28.111237+0000","last_fullsized":"2024-04-14T15:21:28.111237+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'39","last_scrub_stamp":"2024-04-14T15:20:34.830136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:20:34.830136+0000","objects_scrubbed":2,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:57.181076+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001107278,"stat_sum":{"num_bytes":1977,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":19,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":62,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1f","length":"1"}]},{"pgid":"110.1d","version":"537'89","reported_seq":1258,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.118798+0000","last_change":"2024-04-14T15:21:28.119149+0000","last_active":"2024-04-14T15:21:28.118798+0000","last_peered":"2024-04-14T15:21:28.118798+0000","last_clean":"2024-04-14T15:21:28.118798+0000","last_became_active":"2024-04-14T15:20:52.994805+0000","last_became_peered":"2024-04-14T15:20:52.994805+0000","last_unstale":"2024-04-14T15:21:28.118798+0000","last_undegraded":"2024-04-14T15:21:28.118798+0000","last_fullsized":"2024-04-14T15:21:28.118798+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"225'85","last_scrub_stamp":"2024-04-14T15:20:34.688290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:20:34.688290+0000","objects_scrubbed":5,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:57.740524+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001345036,"stat_sum":{"num_bytes":1270,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":8,"num_read_kb":1,"num_write":30,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1f","length":"1"}]},{"pgid":"110.1c","version":"509'97","reported_seq":1201,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.114367+0000","last_change":"2024-04-14T15:21:28.116704+0000","last_active":"2024-04-14T15:21:28.114367+0000","last_peered":"2024-04-14T15:21:28.114367+0000","last_clean":"2024-04-14T15:21:28.114367+0000","last_became_active":"2024-04-14T15:13:09.954091+0000","last_became_peered":"2024-04-14T15:13:09.954091+0000","last_unstale":"2024-04-14T15:21:28.114367+0000","last_undegraded":"2024-04-14T15:21:28.114367+0000","last_fullsized":"2024-04-14T15:21:28.114367+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'95","last_scrub_stamp":"2024-04-14T15:19:10.839339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:19:10.839339+0000","objects_scrubbed":7,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0022805350000000002,"stat_sum":{"num_bytes":65,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":33,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1f","length":"1"}]},{"pgid":"110.1f","version":"519'36","reported_seq":1200,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.115705+0000","last_change":"2024-04-14T15:21:28.115705+0000","last_active":"2024-04-14T15:21:28.115705+0000","last_peered":"2024-04-14T15:21:28.115705+0000","last_clean":"2024-04-14T15:21:28.115705+0000","last_became_active":"2024-04-14T15:13:09.999629+0000","last_became_peered":"2024-04-14T15:13:09.999629+0000","last_unstale":"2024-04-14T15:21:28.115705+0000","last_undegraded":"2024-04-14T15:21:28.115705+0000","last_fullsized":"2024-04-14T15:21:28.115705+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"466'27","last_scrub_stamp":"2024-04-14T15:19:25.078678+0000","last_deep_scrub":"466'27","last_deep_scrub_stamp":"2024-04-14T15:19:25.078678+0000","last_clean_scrub_stamp":"2024-04-14T15:19:25.078678+0000","objects_scrubbed":2,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.00062799999999999998,"stat_sum":{"num_bytes":4194315,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":19,"num_write_kb":8201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1f","length":"1"}]},{"pgid":"110.1e","version":"510'27","reported_seq":1140,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.120714+0000","last_change":"2024-04-14T15:21:28.121108+0000","last_active":"2024-04-14T15:21:28.120714+0000","last_peered":"2024-04-14T15:21:28.120714+0000","last_clean":"2024-04-14T15:21:28.120714+0000","last_became_active":"2024-04-14T15:13:09.955932+0000","last_became_peered":"2024-04-14T15:13:09.955932+0000","last_unstale":"2024-04-14T15:21:28.120714+0000","last_undegraded":"2024-04-14T15:21:28.120714+0000","last_fullsized":"2024-04-14T15:21:28.120714+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'23","last_scrub_stamp":"2024-04-14T15:19:17.364769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:19:17.364769+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00059742599999999997,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":14,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1f","length":"1"}]},{"pgid":"110.1","version":"509'34","reported_seq":1158,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.117501+0000","last_change":"2024-04-14T15:21:28.117501+0000","last_active":"2024-04-14T15:21:28.117501+0000","last_peered":"2024-04-14T15:21:28.117501+0000","last_clean":"2024-04-14T15:21:28.117501+0000","last_became_active":"2024-04-14T15:13:09.929562+0000","last_became_peered":"2024-04-14T15:13:09.929562+0000","last_unstale":"2024-04-14T15:21:28.117501+0000","last_undegraded":"2024-04-14T15:21:28.117501+0000","last_fullsized":"2024-04-14T15:21:28.117501+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"509'34","last_scrub_stamp":"2024-04-14T15:21:18.701670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:21:18.701670+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:47.433293+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00082309099999999997,"stat_sum":{"num_bytes":4000050,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":18,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1f","length":"1"}]},{"pgid":"110.0","version":"561'98","reported_seq":1256,"reported_epoch":561,"state":"active+clean+snaptrim","last_fresh":"2024-04-14T15:21:28.107030+0000","last_change":"2024-04-14T15:21:28.107030+0000","last_active":"2024-04-14T15:21:28.107030+0000","last_peered":"2024-04-14T15:21:28.107030+0000","last_clean":"2024-04-14T15:21:28.107030+0000","last_became_active":"2024-04-14T15:20:52.986208+0000","last_became_peered":"2024-04-14T15:20:52.986208+0000","last_unstale":"2024-04-14T15:21:28.107030+0000","last_undegraded":"2024-04-14T15:21:28.107030+0000","last_fullsized":"2024-04-14T15:21:28.107030+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"553'96","last_scrub_stamp":"2024-04-14T15:21:21.040842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:21:21.040842+0000","objects_scrubbed":8,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:23.190445+0000","scrub_duration":2,"objects_trimmed":1,"snaptrim_duration":0.00052820199999999997,"stat_sum":{"num_bytes":75,"num_objects":7,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":28,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":55,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"110.3","version":"464'56","reported_seq":1197,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.112445+0000","last_change":"2024-04-14T15:21:28.112445+0000","last_active":"2024-04-14T15:21:28.112445+0000","last_peered":"2024-04-14T15:21:28.112445+0000","last_clean":"2024-04-14T15:21:28.112445+0000","last_became_active":"2024-04-14T15:19:35.486976+0000","last_became_peered":"2024-04-14T15:19:35.486976+0000","last_unstale":"2024-04-14T15:21:28.112445+0000","last_undegraded":"2024-04-14T15:21:28.112445+0000","last_fullsized":"2024-04-14T15:21:28.112445+0000","mapping_epoch":470,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":471,"parent":"0.0","parent_split_bits":0,"last_scrub":"464'56","last_scrub_stamp":"2024-04-14T15:19:17.063716+0000","last_deep_scrub":"464'56","last_deep_scrub_stamp":"2024-04-14T15:19:17.063716+0000","last_clean_scrub_stamp":"2024-04-14T15:19:17.063716+0000","objects_scrubbed":4,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00089879200000000004,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":41,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1f","length":"1"}]},{"pgid":"110.2","version":"548'67","reported_seq":1229,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.112046+0000","last_change":"2024-04-14T15:21:28.112046+0000","last_active":"2024-04-14T15:21:28.112046+0000","last_peered":"2024-04-14T15:21:28.112046+0000","last_clean":"2024-04-14T15:21:28.112046+0000","last_became_active":"2024-04-14T15:20:52.982822+0000","last_became_peered":"2024-04-14T15:20:52.982822+0000","last_unstale":"2024-04-14T15:21:28.112046+0000","last_undegraded":"2024-04-14T15:21:28.112046+0000","last_fullsized":"2024-04-14T15:21:28.112046+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"548'67","last_scrub_stamp":"2024-04-14T15:21:20.607257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:21:20.607257+0000","objects_scrubbed":9,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:22.407872+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00087365499999999998,"stat_sum":{"num_bytes":79,"num_objects":9,"num_object_clones":3,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":8,"num_read_kb":3,"num_write":24,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":49,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1f","length":"1"}]},{"pgid":"110.5","version":"510'68","reported_seq":1216,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.108306+0000","last_change":"2024-04-14T15:21:28.108306+0000","last_active":"2024-04-14T15:21:28.108306+0000","last_peered":"2024-04-14T15:21:28.108306+0000","last_clean":"2024-04-14T15:21:28.108306+0000","last_became_active":"2024-04-14T15:20:52.987602+0000","last_became_peered":"2024-04-14T15:20:52.987602+0000","last_unstale":"2024-04-14T15:21:28.108306+0000","last_undegraded":"2024-04-14T15:21:28.108306+0000","last_fullsized":"2024-04-14T15:21:28.108306+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"396'64","last_scrub_stamp":"2024-04-14T15:17:47.418687+0000","last_deep_scrub":"396'64","last_deep_scrub_stamp":"2024-04-14T15:17:47.418687+0000","last_clean_scrub_stamp":"2024-04-14T15:17:47.418687+0000","objects_scrubbed":5,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00077287999999999999,"stat_sum":{"num_bytes":42,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":26,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":42,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1f","length":"1"}]},{"pgid":"110.4","version":"528'56","reported_seq":1161,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.119953+0000","last_change":"2024-04-14T15:21:28.119953+0000","last_active":"2024-04-14T15:21:28.119953+0000","last_peered":"2024-04-14T15:21:28.119953+0000","last_clean":"2024-04-14T15:21:28.119953+0000","last_became_active":"2024-04-14T15:13:09.958847+0000","last_became_peered":"2024-04-14T15:13:09.958847+0000","last_unstale":"2024-04-14T15:21:28.119953+0000","last_undegraded":"2024-04-14T15:21:28.119953+0000","last_fullsized":"2024-04-14T15:21:28.119953+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"406'55","last_scrub_stamp":"2024-04-14T15:19:11.276056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:19:11.276056+0000","objects_scrubbed":4,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00082073699999999999,"stat_sum":{"num_bytes":1375,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":19,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1f","length":"1"}]},{"pgid":"110.7","version":"543'32","reported_seq":765,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.119386+0000","last_change":"2024-04-14T15:21:28.119609+0000","last_active":"2024-04-14T15:21:28.119386+0000","last_peered":"2024-04-14T15:21:28.119386+0000","last_clean":"2024-04-14T15:21:28.119386+0000","last_became_active":"2024-04-14T15:20:52.992363+0000","last_became_peered":"2024-04-14T15:20:52.992363+0000","last_unstale":"2024-04-14T15:21:28.119386+0000","last_undegraded":"2024-04-14T15:21:28.119386+0000","last_fullsized":"2024-04-14T15:21:28.119386+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'18","last_scrub_stamp":"2024-04-14T15:19:56.930258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:19:56.930258+0000","objects_scrubbed":1,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00045557300000000002,"stat_sum":{"num_bytes":32,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1f","length":"1"}]},{"pgid":"110.6","version":"528'91","reported_seq":1186,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.119330+0000","last_change":"2024-04-14T15:21:28.119665+0000","last_active":"2024-04-14T15:21:28.119330+0000","last_peered":"2024-04-14T15:21:28.119330+0000","last_clean":"2024-04-14T15:21:28.119330+0000","last_became_active":"2024-04-14T15:13:09.930125+0000","last_became_peered":"2024-04-14T15:13:09.930125+0000","last_unstale":"2024-04-14T15:21:28.119330+0000","last_undegraded":"2024-04-14T15:21:28.119330+0000","last_fullsized":"2024-04-14T15:21:28.119330+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"444'90","last_scrub_stamp":"2024-04-14T15:19:12.507119+0000","last_deep_scrub":"274'74","last_deep_scrub_stamp":"2024-04-14T15:15:42.641684+0000","last_clean_scrub_stamp":"2024-04-14T15:19:12.507119+0000","objects_scrubbed":7,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0016818549999999999,"stat_sum":{"num_bytes":629,"num_objects":8,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":28,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1f","length":"1"}]},{"pgid":"110.9","version":"508'22","reported_seq":1160,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.109103+0000","last_change":"2024-04-14T15:21:28.110046+0000","last_active":"2024-04-14T15:21:28.109103+0000","last_peered":"2024-04-14T15:21:28.109103+0000","last_clean":"2024-04-14T15:21:28.109103+0000","last_became_active":"2024-04-14T15:20:52.983058+0000","last_became_peered":"2024-04-14T15:20:52.983058+0000","last_unstale":"2024-04-14T15:21:28.109103+0000","last_undegraded":"2024-04-14T15:21:28.109103+0000","last_fullsized":"2024-04-14T15:21:28.109103+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"369'20","last_scrub_stamp":"2024-04-14T15:19:22.004242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:19:22.004242+0000","objects_scrubbed":4,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001222725,"stat_sum":{"num_bytes":35,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":11,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":35,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1f","length":"1"}]},{"pgid":"110.8","version":"560'22","reported_seq":914,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.107648+0000","last_change":"2024-04-14T15:21:28.111923+0000","last_active":"2024-04-14T15:21:28.107648+0000","last_peered":"2024-04-14T15:21:28.107648+0000","last_clean":"2024-04-14T15:21:28.107648+0000","last_became_active":"2024-04-14T15:20:52.996553+0000","last_became_peered":"2024-04-14T15:20:52.996553+0000","last_unstale":"2024-04-14T15:21:28.107648+0000","last_undegraded":"2024-04-14T15:21:28.107648+0000","last_fullsized":"2024-04-14T15:21:28.107648+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"560'22","last_scrub_stamp":"2024-04-14T15:21:27.155912+0000","last_deep_scrub":"560'22","last_deep_scrub_stamp":"2024-04-14T15:21:27.155912+0000","last_clean_scrub_stamp":"2024-04-14T15:21:27.155912+0000","objects_scrubbed":1,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:34.042712+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0044512240000000002,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":11,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":11,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1f","length":"1"}]},{"pgid":"110.b","version":"412'71","reported_seq":1178,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.113778+0000","last_change":"2024-04-14T15:21:28.114185+0000","last_active":"2024-04-14T15:21:28.113778+0000","last_peered":"2024-04-14T15:21:28.113778+0000","last_clean":"2024-04-14T15:21:28.113778+0000","last_became_active":"2024-04-14T15:13:09.964629+0000","last_became_peered":"2024-04-14T15:13:09.964629+0000","last_unstale":"2024-04-14T15:21:28.113778+0000","last_undegraded":"2024-04-14T15:21:28.113778+0000","last_fullsized":"2024-04-14T15:21:28.113778+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'71","last_scrub_stamp":"2024-04-14T15:19:22.798030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:19:22.798030+0000","objects_scrubbed":5,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.002284211,"stat_sum":{"num_bytes":54,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":22,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1f","length":"1"}]},{"pgid":"110.a","version":"528'79","reported_seq":1170,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.109755+0000","last_change":"2024-04-14T15:21:28.110273+0000","last_active":"2024-04-14T15:21:28.109755+0000","last_peered":"2024-04-14T15:21:28.109755+0000","last_clean":"2024-04-14T15:21:28.109755+0000","last_became_active":"2024-04-14T15:20:52.997626+0000","last_became_peered":"2024-04-14T15:20:52.997626+0000","last_unstale":"2024-04-14T15:21:28.109755+0000","last_undegraded":"2024-04-14T15:21:28.109755+0000","last_fullsized":"2024-04-14T15:21:28.109755+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'74","last_scrub_stamp":"2024-04-14T15:19:14.371777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:19:14.371777+0000","objects_scrubbed":9,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0026197989999999999,"stat_sum":{"num_bytes":1693,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":10,"num_read_kb":1,"num_write":35,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":1172,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1f","length":"1"}]},{"pgid":"110.d","version":"510'101","reported_seq":1033,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.110408+0000","last_change":"2024-04-14T15:21:28.110730+0000","last_active":"2024-04-14T15:21:28.110408+0000","last_peered":"2024-04-14T15:21:28.110408+0000","last_clean":"2024-04-14T15:21:28.110408+0000","last_became_active":"2024-04-14T15:15:21.877073+0000","last_became_peered":"2024-04-14T15:15:21.877073+0000","last_unstale":"2024-04-14T15:21:28.110408+0000","last_undegraded":"2024-04-14T15:21:28.110408+0000","last_fullsized":"2024-04-14T15:21:28.110408+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'95","last_scrub_stamp":"2024-04-14T15:19:13.327836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:19:13.327836+0000","objects_scrubbed":8,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0024031410000000001,"stat_sum":{"num_bytes":71,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":9,"num_read_kb":2,"num_write":40,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1f","length":"1"}]},{"pgid":"110.c","version":"528'27","reported_seq":1166,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.115216+0000","last_change":"2024-04-14T15:21:28.116716+0000","last_active":"2024-04-14T15:21:28.115216+0000","last_peered":"2024-04-14T15:21:28.115216+0000","last_clean":"2024-04-14T15:21:28.115216+0000","last_became_active":"2024-04-14T15:20:52.977993+0000","last_became_peered":"2024-04-14T15:20:52.977993+0000","last_unstale":"2024-04-14T15:21:28.115216+0000","last_undegraded":"2024-04-14T15:21:28.115216+0000","last_fullsized":"2024-04-14T15:21:28.115216+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"528'27","last_scrub_stamp":"2024-04-14T15:21:13.763125+0000","last_deep_scrub":"398'20","last_deep_scrub_stamp":"2024-04-14T15:17:52.882509+0000","last_clean_scrub_stamp":"2024-04-14T15:21:13.763125+0000","objects_scrubbed":3,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:30.015286+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0020729160000000002,"stat_sum":{"num_bytes":29,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":12,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":29,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1f","length":"1"}]},{"pgid":"110.f","version":"510'56","reported_seq":1168,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.118862+0000","last_change":"2024-04-14T15:21:28.119222+0000","last_active":"2024-04-14T15:21:28.118862+0000","last_peered":"2024-04-14T15:21:28.118862+0000","last_clean":"2024-04-14T15:21:28.118862+0000","last_became_active":"2024-04-14T15:13:09.964057+0000","last_became_peered":"2024-04-14T15:13:09.964057+0000","last_unstale":"2024-04-14T15:21:28.118862+0000","last_undegraded":"2024-04-14T15:21:28.118862+0000","last_fullsized":"2024-04-14T15:21:28.118862+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'52","last_scrub_stamp":"2024-04-14T15:20:03.860616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:20:03.860616+0000","objects_scrubbed":4,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00054261999999999995,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":2,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1f","length":"1"}]},{"pgid":"110.e","version":"528'95","reported_seq":1202,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.118881+0000","last_change":"2024-04-14T15:21:28.119198+0000","last_active":"2024-04-14T15:21:28.118881+0000","last_peered":"2024-04-14T15:21:28.118881+0000","last_clean":"2024-04-14T15:21:28.118881+0000","last_became_active":"2024-04-14T15:13:09.984980+0000","last_became_peered":"2024-04-14T15:13:09.984980+0000","last_unstale":"2024-04-14T15:21:28.118881+0000","last_undegraded":"2024-04-14T15:21:28.118881+0000","last_fullsized":"2024-04-14T15:21:28.118881+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'92","last_scrub_stamp":"2024-04-14T15:20:02.654315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:08.799672+0000","last_clean_scrub_stamp":"2024-04-14T15:20:02.654315+0000","objects_scrubbed":7,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001283347,"stat_sum":{"num_bytes":1610,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":9,"num_read_kb":1,"num_write":33,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1f","length":"1"}]},{"pgid":"1.d","version":"91'130","reported_seq":631,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.117635+0000","last_change":"2024-04-14T15:20:42.941695+0000","last_active":"2024-04-14T15:21:28.117635+0000","last_peered":"2024-04-14T15:21:28.117635+0000","last_clean":"2024-04-14T15:21:28.117635+0000","last_became_active":"2024-04-14T15:20:42.941300+0000","last_became_peered":"2024-04-14T15:20:42.941300+0000","last_unstale":"2024-04-14T15:21:28.117635+0000","last_undegraded":"2024-04-14T15:21:28.117635+0000","last_fullsized":"2024-04-14T15:21:28.117635+0000","mapping_epoch":436,"log_start":"91'130","ondisk_log_start":"91'130","created":431,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:09.497308+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:09.497308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":590,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.115054+0000","last_change":"2024-04-14T15:19:10.053883+0000","last_active":"2024-04-14T15:21:28.115054+0000","last_peered":"2024-04-14T15:21:28.115054+0000","last_clean":"2024-04-14T15:21:28.115054+0000","last_became_active":"2024-04-14T15:15:44.882941+0000","last_became_peered":"2024-04-14T15:15:44.882941+0000","last_unstale":"2024-04-14T15:21:28.115054+0000","last_undegraded":"2024-04-14T15:21:28.115054+0000","last_fullsized":"2024-04-14T15:21:28.115054+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:10.053804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:19:10.053804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.9","version":"528'10","reported_seq":1199,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.117239+0000","last_change":"2024-04-14T15:20:52.998038+0000","last_active":"2024-04-14T15:21:28.117239+0000","last_peered":"2024-04-14T15:21:28.117239+0000","last_clean":"2024-04-14T15:21:28.117239+0000","last_became_active":"2024-04-14T15:20:52.984943+0000","last_became_peered":"2024-04-14T15:20:52.984943+0000","last_unstale":"2024-04-14T15:21:28.117239+0000","last_undegraded":"2024-04-14T15:21:28.117239+0000","last_fullsized":"2024-04-14T15:21:28.117239+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9","last_scrub_stamp":"2024-04-14T15:20:45.737773+0000","last_deep_scrub":"85'9","last_deep_scrub_stamp":"2024-04-14T15:10:51.950580+0000","last_clean_scrub_stamp":"2024-04-14T15:20:45.737773+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:07.463837+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"479'34","reported_seq":687,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.116376+0000","last_change":"2024-04-14T15:21:27.711513+0000","last_active":"2024-04-14T15:21:28.116376+0000","last_peered":"2024-04-14T15:21:28.116376+0000","last_clean":"2024-04-14T15:21:28.116376+0000","last_became_active":"2024-04-14T15:20:52.979204+0000","last_became_peered":"2024-04-14T15:20:52.979204+0000","last_unstale":"2024-04-14T15:21:28.116376+0000","last_undegraded":"2024-04-14T15:21:28.116376+0000","last_fullsized":"2024-04-14T15:21:28.116376+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'34","last_scrub_stamp":"2024-04-14T15:21:27.711357+0000","last_deep_scrub":"430'22","last_deep_scrub_stamp":"2024-04-14T15:18:39.704883+0000","last_clean_scrub_stamp":"2024-04-14T15:21:27.711357+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:40.998286+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":1045,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.116469+0000","last_change":"2024-04-14T15:20:35.731883+0000","last_active":"2024-04-14T15:21:28.116469+0000","last_peered":"2024-04-14T15:21:28.116469+0000","last_clean":"2024-04-14T15:21:28.116469+0000","last_became_active":"2024-04-14T15:09:43.021460+0000","last_became_peered":"2024-04-14T15:09:43.021460+0000","last_unstale":"2024-04-14T15:21:28.116469+0000","last_undegraded":"2024-04-14T15:21:28.116469+0000","last_fullsized":"2024-04-14T15:21:28.116469+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:35.731808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:20:35.731808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:37.156866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":82,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.117671+0000","last_change":"2024-04-14T15:20:52.989978+0000","last_active":"2024-04-14T15:21:28.117671+0000","last_peered":"2024-04-14T15:21:28.117671+0000","last_clean":"2024-04-14T15:21:28.117671+0000","last_became_active":"2024-04-14T15:20:52.989666+0000","last_became_peered":"2024-04-14T15:20:52.989666+0000","last_unstale":"2024-04-14T15:21:28.117671+0000","last_undegraded":"2024-04-14T15:21:28.117671+0000","last_fullsized":"2024-04-14T15:21:28.117671+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:36.860426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:15:40.368219+0000","last_clean_scrub_stamp":"2024-04-14T15:20:36.860426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:46.129984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"91'130","reported_seq":638,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.106718+0000","last_change":"2024-04-14T15:20:38.984299+0000","last_active":"2024-04-14T15:21:28.106718+0000","last_peered":"2024-04-14T15:21:28.106718+0000","last_clean":"2024-04-14T15:21:28.106718+0000","last_became_active":"2024-04-14T15:20:38.983964+0000","last_became_peered":"2024-04-14T15:20:38.983964+0000","last_unstale":"2024-04-14T15:21:28.106718+0000","last_undegraded":"2024-04-14T15:21:28.106718+0000","last_fullsized":"2024-04-14T15:21:28.106718+0000","mapping_epoch":436,"log_start":"91'130","ondisk_log_start":"91'130","created":431,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:20:31.675561+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:20:31.675561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:46.311472+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":949,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.117867+0000","last_change":"2024-04-14T15:21:24.713987+0000","last_active":"2024-04-14T15:21:28.117867+0000","last_peered":"2024-04-14T15:21:28.117867+0000","last_clean":"2024-04-14T15:21:28.117867+0000","last_became_active":"2024-04-14T15:20:52.991952+0000","last_became_peered":"2024-04-14T15:20:52.991952+0000","last_unstale":"2024-04-14T15:21:28.117867+0000","last_undegraded":"2024-04-14T15:21:28.117867+0000","last_fullsized":"2024-04-14T15:21:28.117867+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:24.713877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:21:24.713877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:32.425724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"479'15","reported_seq":1068,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.117946+0000","last_change":"2024-04-14T15:20:33.353872+0000","last_active":"2024-04-14T15:21:28.117946+0000","last_peered":"2024-04-14T15:21:28.117946+0000","last_clean":"2024-04-14T15:21:28.117946+0000","last_became_active":"2024-04-14T15:09:45.037179+0000","last_became_peered":"2024-04-14T15:09:45.037179+0000","last_unstale":"2024-04-14T15:21:28.117946+0000","last_undegraded":"2024-04-14T15:21:28.117946+0000","last_fullsized":"2024-04-14T15:21:28.117946+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'15","last_scrub_stamp":"2024-04-14T15:20:33.353777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:20:33.353777+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:00.604055+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":1040,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.114344+0000","last_change":"2024-04-14T15:18:36.029705+0000","last_active":"2024-04-14T15:21:28.114344+0000","last_peered":"2024-04-14T15:21:28.114344+0000","last_clean":"2024-04-14T15:21:28.114344+0000","last_became_active":"2024-04-14T15:09:43.020044+0000","last_became_peered":"2024-04-14T15:09:43.020044+0000","last_unstale":"2024-04-14T15:21:28.114344+0000","last_undegraded":"2024-04-14T15:21:28.114344+0000","last_fullsized":"2024-04-14T15:21:28.114344+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:36.029630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:18:36.029630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1045,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.774856+0000","last_change":"2024-04-14T15:21:28.774876+0000","last_active":"2024-04-14T15:21:28.774856+0000","last_peered":"2024-04-14T15:21:28.774856+0000","last_clean":"2024-04-14T15:21:28.774856+0000","last_became_active":"2024-04-14T15:09:47.046153+0000","last_became_peered":"2024-04-14T15:09:47.046153+0000","last_unstale":"2024-04-14T15:21:28.774856+0000","last_undegraded":"2024-04-14T15:21:28.774856+0000","last_fullsized":"2024-04-14T15:21:28.774856+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:28.774798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:21:28.774798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:48.686194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1035,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.117874+0000","last_change":"2024-04-14T15:20:43.480669+0000","last_active":"2024-04-14T15:21:28.117874+0000","last_peered":"2024-04-14T15:21:28.117874+0000","last_clean":"2024-04-14T15:21:28.117874+0000","last_became_active":"2024-04-14T15:09:49.061495+0000","last_became_peered":"2024-04-14T15:09:49.061495+0000","last_unstale":"2024-04-14T15:21:28.117874+0000","last_undegraded":"2024-04-14T15:21:28.117874+0000","last_fullsized":"2024-04-14T15:21:28.117874+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:43.480592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:20:43.480592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:12.444433+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"91'130","reported_seq":948,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.106811+0000","last_change":"2024-04-14T15:21:10.884106+0000","last_active":"2024-04-14T15:21:28.106811+0000","last_peered":"2024-04-14T15:21:28.106811+0000","last_clean":"2024-04-14T15:21:28.106811+0000","last_became_active":"2024-04-14T15:20:55.999101+0000","last_became_peered":"2024-04-14T15:20:55.999101+0000","last_unstale":"2024-04-14T15:21:28.106811+0000","last_undegraded":"2024-04-14T15:21:28.106811+0000","last_fullsized":"2024-04-14T15:21:28.106811+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:21:10.883960+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:21:10.883960+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:18.257172+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":949,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.118789+0000","last_change":"2024-04-14T15:21:20.684941+0000","last_active":"2024-04-14T15:21:28.118789+0000","last_peered":"2024-04-14T15:21:28.118789+0000","last_clean":"2024-04-14T15:21:28.118789+0000","last_became_active":"2024-04-14T15:20:52.999845+0000","last_became_peered":"2024-04-14T15:20:52.999845+0000","last_unstale":"2024-04-14T15:21:28.118789+0000","last_undegraded":"2024-04-14T15:21:28.118789+0000","last_fullsized":"2024-04-14T15:21:28.118789+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:20.684813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:21:20.684813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:47.282917+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"479'44","reported_seq":1030,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.111534+0000","last_change":"2024-04-14T15:20:53.012885+0000","last_active":"2024-04-14T15:21:28.111534+0000","last_peered":"2024-04-14T15:21:28.111534+0000","last_clean":"2024-04-14T15:21:28.111534+0000","last_became_active":"2024-04-14T15:20:52.980461+0000","last_became_peered":"2024-04-14T15:20:52.980461+0000","last_unstale":"2024-04-14T15:21:28.111534+0000","last_undegraded":"2024-04-14T15:21:28.111534+0000","last_fullsized":"2024-04-14T15:21:28.111534+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'44","last_scrub_stamp":"2024-04-14T15:19:58.856115+0000","last_deep_scrub":"479'44","last_deep_scrub_stamp":"2024-04-14T15:19:58.856115+0000","last_clean_scrub_stamp":"2024-04-14T15:19:58.856115+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":1053,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.115464+0000","last_change":"2024-04-14T15:21:17.771545+0000","last_active":"2024-04-14T15:21:28.115464+0000","last_peered":"2024-04-14T15:21:28.115464+0000","last_clean":"2024-04-14T15:21:28.115464+0000","last_became_active":"2024-04-14T15:09:43.020268+0000","last_became_peered":"2024-04-14T15:09:43.020268+0000","last_unstale":"2024-04-14T15:21:28.115464+0000","last_undegraded":"2024-04-14T15:21:28.115464+0000","last_fullsized":"2024-04-14T15:21:28.115464+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:17.771462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:34.342793+0000","last_clean_scrub_stamp":"2024-04-14T15:21:17.771462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:23.777280+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1037,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.112944+0000","last_change":"2024-04-14T15:21:23.738251+0000","last_active":"2024-04-14T15:21:28.112944+0000","last_peered":"2024-04-14T15:21:28.112944+0000","last_clean":"2024-04-14T15:21:28.112944+0000","last_became_active":"2024-04-14T15:09:47.045542+0000","last_became_peered":"2024-04-14T15:09:47.045542+0000","last_unstale":"2024-04-14T15:21:28.112944+0000","last_undegraded":"2024-04-14T15:21:28.112944+0000","last_fullsized":"2024-04-14T15:21:28.112944+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:23.738176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:21:23.738176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:27.527252+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1026,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.116886+0000","last_change":"2024-04-14T15:19:16.566595+0000","last_active":"2024-04-14T15:21:28.116886+0000","last_peered":"2024-04-14T15:21:28.116886+0000","last_clean":"2024-04-14T15:21:28.116886+0000","last_became_active":"2024-04-14T15:09:49.065397+0000","last_became_peered":"2024-04-14T15:09:49.065397+0000","last_unstale":"2024-04-14T15:21:28.116886+0000","last_undegraded":"2024-04-14T15:21:28.116886+0000","last_fullsized":"2024-04-14T15:21:28.116886+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:16.566496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:19:16.566496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"91'130","reported_seq":958,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.109712+0000","last_change":"2024-04-14T15:21:12.961821+0000","last_active":"2024-04-14T15:21:28.109712+0000","last_peered":"2024-04-14T15:21:28.109712+0000","last_clean":"2024-04-14T15:21:28.109712+0000","last_became_active":"2024-04-14T15:21:12.961419+0000","last_became_peered":"2024-04-14T15:21:12.961419+0000","last_unstale":"2024-04-14T15:21:28.109712+0000","last_undegraded":"2024-04-14T15:21:28.109712+0000","last_fullsized":"2024-04-14T15:21:28.109712+0000","mapping_epoch":276,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:04.950991+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:04.950991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":70,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":942,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.116515+0000","last_change":"2024-04-14T15:19:15.778552+0000","last_active":"2024-04-14T15:21:28.116515+0000","last_peered":"2024-04-14T15:21:28.116515+0000","last_clean":"2024-04-14T15:21:28.116515+0000","last_became_active":"2024-04-14T15:12:56.474322+0000","last_became_peered":"2024-04-14T15:12:56.474322+0000","last_unstale":"2024-04-14T15:21:28.116515+0000","last_undegraded":"2024-04-14T15:21:28.116515+0000","last_fullsized":"2024-04-14T15:21:28.116515+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:15.778469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:19:15.778469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"479'31","reported_seq":1131,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.107866+0000","last_change":"2024-04-14T15:17:48.374589+0000","last_active":"2024-04-14T15:21:28.107866+0000","last_peered":"2024-04-14T15:21:28.107866+0000","last_clean":"2024-04-14T15:21:28.107866+0000","last_became_active":"2024-04-14T15:13:12.933907+0000","last_became_peered":"2024-04-14T15:13:12.933907+0000","last_unstale":"2024-04-14T15:21:28.107866+0000","last_undegraded":"2024-04-14T15:21:28.107866+0000","last_fullsized":"2024-04-14T15:21:28.107866+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'17","last_scrub_stamp":"2024-04-14T15:17:48.374469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:17:48.374469+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":1038,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.117291+0000","last_change":"2024-04-14T15:19:26.515133+0000","last_active":"2024-04-14T15:21:28.117291+0000","last_peered":"2024-04-14T15:21:28.117291+0000","last_clean":"2024-04-14T15:21:28.117291+0000","last_became_active":"2024-04-14T15:09:43.022351+0000","last_became_peered":"2024-04-14T15:09:43.022351+0000","last_unstale":"2024-04-14T15:21:28.117291+0000","last_undegraded":"2024-04-14T15:21:28.117291+0000","last_fullsized":"2024-04-14T15:21:28.117291+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:26.515055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:19:26.515055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1024,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.113905+0000","last_change":"2024-04-14T15:17:54.871693+0000","last_active":"2024-04-14T15:21:28.113905+0000","last_peered":"2024-04-14T15:21:28.113905+0000","last_clean":"2024-04-14T15:21:28.113905+0000","last_became_active":"2024-04-14T15:09:47.042810+0000","last_became_peered":"2024-04-14T15:09:47.042810+0000","last_unstale":"2024-04-14T15:21:28.113905+0000","last_undegraded":"2024-04-14T15:21:28.113905+0000","last_fullsized":"2024-04-14T15:21:28.113905+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:54.871593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:17:54.871593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1033,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.118319+0000","last_change":"2024-04-14T15:19:51.902814+0000","last_active":"2024-04-14T15:21:28.118319+0000","last_peered":"2024-04-14T15:21:28.118319+0000","last_clean":"2024-04-14T15:21:28.118319+0000","last_became_active":"2024-04-14T15:09:49.069594+0000","last_became_peered":"2024-04-14T15:09:49.069594+0000","last_unstale":"2024-04-14T15:21:28.118319+0000","last_undegraded":"2024-04-14T15:21:28.118319+0000","last_fullsized":"2024-04-14T15:21:28.118319+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:51.902730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:19:51.902730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"91'130","reported_seq":1361,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.116504+0000","last_change":"2024-04-14T15:21:09.020536+0000","last_active":"2024-04-14T15:21:28.116504+0000","last_peered":"2024-04-14T15:21:28.116504+0000","last_clean":"2024-04-14T15:21:28.116504+0000","last_became_active":"2024-04-14T15:21:09.020300+0000","last_became_peered":"2024-04-14T15:21:09.020300+0000","last_unstale":"2024-04-14T15:21:28.116504+0000","last_undegraded":"2024-04-14T15:21:28.116504+0000","last_fullsized":"2024-04-14T15:21:28.116504+0000","mapping_epoch":189,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":544,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:06.490208+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:06.490208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":78,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.117143+0000","last_change":"2024-04-14T15:20:52.980707+0000","last_active":"2024-04-14T15:21:28.117143+0000","last_peered":"2024-04-14T15:21:28.117143+0000","last_clean":"2024-04-14T15:21:28.117143+0000","last_became_active":"2024-04-14T15:20:52.980375+0000","last_became_peered":"2024-04-14T15:20:52.980375+0000","last_unstale":"2024-04-14T15:21:28.117143+0000","last_undegraded":"2024-04-14T15:21:28.117143+0000","last_fullsized":"2024-04-14T15:21:28.117143+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:02.773961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:20:02.773961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"479'25","reported_seq":1087,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.106298+0000","last_change":"2024-04-14T15:20:53.030926+0000","last_active":"2024-04-14T15:21:28.106298+0000","last_peered":"2024-04-14T15:21:28.106298+0000","last_clean":"2024-04-14T15:21:28.106298+0000","last_became_active":"2024-04-14T15:20:52.987699+0000","last_became_peered":"2024-04-14T15:20:52.987699+0000","last_unstale":"2024-04-14T15:21:28.106298+0000","last_undegraded":"2024-04-14T15:21:28.106298+0000","last_fullsized":"2024-04-14T15:21:28.106298+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'25","last_scrub_stamp":"2024-04-14T15:20:03.571160+0000","last_deep_scrub":"479'25","last_deep_scrub_stamp":"2024-04-14T15:20:03.571160+0000","last_clean_scrub_stamp":"2024-04-14T15:20:03.571160+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":1040,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.111633+0000","last_change":"2024-04-14T15:17:51.854742+0000","last_active":"2024-04-14T15:21:28.111633+0000","last_peered":"2024-04-14T15:21:28.111633+0000","last_clean":"2024-04-14T15:21:28.111633+0000","last_became_active":"2024-04-14T15:09:43.018871+0000","last_became_peered":"2024-04-14T15:09:43.018871+0000","last_unstale":"2024-04-14T15:21:28.111633+0000","last_undegraded":"2024-04-14T15:21:28.111633+0000","last_fullsized":"2024-04-14T15:21:28.111633+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:51.854659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:17:51.854659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"400'9","reported_seq":1188,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.116458+0000","last_change":"2024-04-14T15:20:36.702189+0000","last_active":"2024-04-14T15:21:28.116458+0000","last_peered":"2024-04-14T15:21:28.116458+0000","last_clean":"2024-04-14T15:21:28.116458+0000","last_became_active":"2024-04-14T15:09:47.044085+0000","last_became_peered":"2024-04-14T15:09:47.044085+0000","last_unstale":"2024-04-14T15:21:28.116458+0000","last_undegraded":"2024-04-14T15:21:28.116458+0000","last_fullsized":"2024-04-14T15:21:28.116458+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'9","last_scrub_stamp":"2024-04-14T15:20:36.702101+0000","last_deep_scrub":"400'9","last_deep_scrub_stamp":"2024-04-14T15:17:54.845881+0000","last_clean_scrub_stamp":"2024-04-14T15:20:36.702101+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:46.454244+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1026,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.107652+0000","last_change":"2024-04-14T15:19:18.588681+0000","last_active":"2024-04-14T15:21:28.107652+0000","last_peered":"2024-04-14T15:21:28.107652+0000","last_clean":"2024-04-14T15:21:28.107652+0000","last_became_active":"2024-04-14T15:09:49.065274+0000","last_became_peered":"2024-04-14T15:09:49.065274+0000","last_unstale":"2024-04-14T15:21:28.107652+0000","last_undegraded":"2024-04-14T15:21:28.107652+0000","last_fullsized":"2024-04-14T15:21:28.107652+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:18.588546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:18.588546+0000","last_clean_scrub_stamp":"2024-04-14T15:19:18.588546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"91'130","reported_seq":973,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.115358+0000","last_change":"2024-04-14T15:21:04.971858+0000","last_active":"2024-04-14T15:21:28.115358+0000","last_peered":"2024-04-14T15:21:28.115358+0000","last_clean":"2024-04-14T15:21:28.115358+0000","last_became_active":"2024-04-14T15:21:04.971515+0000","last_became_peered":"2024-04-14T15:21:04.971515+0000","last_unstale":"2024-04-14T15:21:28.115358+0000","last_undegraded":"2024-04-14T15:21:28.115358+0000","last_fullsized":"2024-04-14T15:21:28.115358+0000","mapping_epoch":530,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":540,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:05.972050+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:05.972050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":937,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.109936+0000","last_change":"2024-04-14T15:19:11.008294+0000","last_active":"2024-04-14T15:21:28.109936+0000","last_peered":"2024-04-14T15:21:28.109936+0000","last_clean":"2024-04-14T15:21:28.109936+0000","last_became_active":"2024-04-14T15:12:56.487615+0000","last_became_peered":"2024-04-14T15:12:56.487615+0000","last_unstale":"2024-04-14T15:21:28.109936+0000","last_undegraded":"2024-04-14T15:21:28.109936+0000","last_fullsized":"2024-04-14T15:21:28.109936+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:11.008225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:19:11.008225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"479'34","reported_seq":1094,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.115175+0000","last_change":"2024-04-14T15:21:13.577045+0000","last_active":"2024-04-14T15:21:28.115175+0000","last_peered":"2024-04-14T15:21:28.115175+0000","last_clean":"2024-04-14T15:21:28.115175+0000","last_became_active":"2024-04-14T15:09:45.033331+0000","last_became_peered":"2024-04-14T15:09:45.033331+0000","last_unstale":"2024-04-14T15:21:28.115175+0000","last_undegraded":"2024-04-14T15:21:28.115175+0000","last_fullsized":"2024-04-14T15:21:28.115175+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'34","last_scrub_stamp":"2024-04-14T15:21:13.576916+0000","last_deep_scrub":"398'22","last_deep_scrub_stamp":"2024-04-14T15:17:52.883850+0000","last_clean_scrub_stamp":"2024-04-14T15:21:13.576916+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:27.429874+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":1040,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.109489+0000","last_change":"2024-04-14T15:18:25.009179+0000","last_active":"2024-04-14T15:21:28.109489+0000","last_peered":"2024-04-14T15:21:28.109489+0000","last_clean":"2024-04-14T15:21:28.109489+0000","last_became_active":"2024-04-14T15:09:43.020878+0000","last_became_peered":"2024-04-14T15:09:43.020878+0000","last_unstale":"2024-04-14T15:21:28.109489+0000","last_undegraded":"2024-04-14T15:21:28.109489+0000","last_fullsized":"2024-04-14T15:21:28.109489+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:25.009100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:18:25.009100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"72'8","reported_seq":1180,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.107680+0000","last_change":"2024-04-14T15:19:19.596882+0000","last_active":"2024-04-14T15:21:28.107680+0000","last_peered":"2024-04-14T15:21:28.107680+0000","last_clean":"2024-04-14T15:21:28.107680+0000","last_became_active":"2024-04-14T15:09:47.050962+0000","last_became_peered":"2024-04-14T15:09:47.050962+0000","last_unstale":"2024-04-14T15:21:28.107680+0000","last_undegraded":"2024-04-14T15:21:28.107680+0000","last_fullsized":"2024-04-14T15:21:28.107680+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:19:19.596773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:19:19.596773+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":599,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.113847+0000","last_change":"2024-04-14T15:20:44.583925+0000","last_active":"2024-04-14T15:21:28.113847+0000","last_peered":"2024-04-14T15:21:28.113847+0000","last_clean":"2024-04-14T15:21:28.113847+0000","last_became_active":"2024-04-14T15:15:44.879699+0000","last_became_peered":"2024-04-14T15:15:44.879699+0000","last_unstale":"2024-04-14T15:21:28.113847+0000","last_undegraded":"2024-04-14T15:21:28.113847+0000","last_fullsized":"2024-04-14T15:21:28.113847+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:44.583849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:20:44.583849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:46.909706+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"91'130","reported_seq":1157,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.111183+0000","last_change":"2024-04-14T15:20:50.960054+0000","last_active":"2024-04-14T15:21:28.111183+0000","last_peered":"2024-04-14T15:21:28.111183+0000","last_clean":"2024-04-14T15:21:28.111183+0000","last_became_active":"2024-04-14T15:20:50.959654+0000","last_became_peered":"2024-04-14T15:20:50.959654+0000","last_unstale":"2024-04-14T15:21:28.111183+0000","last_undegraded":"2024-04-14T15:21:28.111183+0000","last_fullsized":"2024-04-14T15:21:28.111183+0000","mapping_epoch":436,"log_start":"91'130","ondisk_log_start":"91'130","created":431,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:08.097372+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:08.097372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"233'8","reported_seq":359,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.107730+0000","last_change":"2024-04-14T15:20:53.048741+0000","last_active":"2024-04-14T15:21:28.107730+0000","last_peered":"2024-04-14T15:21:28.107730+0000","last_clean":"2024-04-14T15:21:28.107730+0000","last_became_active":"2024-04-14T15:20:52.995129+0000","last_became_peered":"2024-04-14T15:20:52.995129+0000","last_unstale":"2024-04-14T15:21:28.107730+0000","last_undegraded":"2024-04-14T15:21:28.107730+0000","last_fullsized":"2024-04-14T15:21:28.107730+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'8","last_scrub_stamp":"2024-04-14T15:20:37.855435+0000","last_deep_scrub":"233'8","last_deep_scrub_stamp":"2024-04-14T15:15:01.295800+0000","last_clean_scrub_stamp":"2024-04-14T15:20:37.855435+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:43.189446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":5,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"479'24","reported_seq":1096,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.106939+0000","last_change":"2024-04-14T15:21:13.939265+0000","last_active":"2024-04-14T15:21:28.106939+0000","last_peered":"2024-04-14T15:21:28.106939+0000","last_clean":"2024-04-14T15:21:28.106939+0000","last_became_active":"2024-04-14T15:20:52.998853+0000","last_became_peered":"2024-04-14T15:20:52.998853+0000","last_unstale":"2024-04-14T15:21:28.106939+0000","last_undegraded":"2024-04-14T15:21:28.106939+0000","last_fullsized":"2024-04-14T15:21:28.106939+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'24","last_scrub_stamp":"2024-04-14T15:21:13.939121+0000","last_deep_scrub":"85'16","last_deep_scrub_stamp":"2024-04-14T15:10:47.456169+0000","last_clean_scrub_stamp":"2024-04-14T15:21:13.939121+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:15.725824+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":1046,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.116491+0000","last_change":"2024-04-14T15:20:03.642215+0000","last_active":"2024-04-14T15:21:28.116491+0000","last_peered":"2024-04-14T15:21:28.116491+0000","last_clean":"2024-04-14T15:21:28.116491+0000","last_became_active":"2024-04-14T15:09:43.018601+0000","last_became_peered":"2024-04-14T15:09:43.018601+0000","last_unstale":"2024-04-14T15:21:28.116491+0000","last_undegraded":"2024-04-14T15:21:28.116491+0000","last_fullsized":"2024-04-14T15:21:28.116491+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:03.642112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:20:03.642112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":602,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.107703+0000","last_change":"2024-04-14T15:20:52.974819+0000","last_active":"2024-04-14T15:21:28.107703+0000","last_peered":"2024-04-14T15:21:28.107703+0000","last_clean":"2024-04-14T15:21:28.107703+0000","last_became_active":"2024-04-14T15:20:52.970831+0000","last_became_peered":"2024-04-14T15:20:52.970831+0000","last_unstale":"2024-04-14T15:21:28.107703+0000","last_undegraded":"2024-04-14T15:21:28.107703+0000","last_fullsized":"2024-04-14T15:21:28.107703+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:59.596964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:19:59.596964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:29.441153+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":714,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.112181+0000","last_change":"2024-04-14T15:21:17.980121+0000","last_active":"2024-04-14T15:21:28.112181+0000","last_peered":"2024-04-14T15:21:28.112181+0000","last_clean":"2024-04-14T15:21:28.112181+0000","last_became_active":"2024-04-14T15:20:52.999771+0000","last_became_peered":"2024-04-14T15:20:52.999771+0000","last_unstale":"2024-04-14T15:21:28.112181+0000","last_undegraded":"2024-04-14T15:21:28.112181+0000","last_fullsized":"2024-04-14T15:21:28.112181+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:17.980009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:57.214046+0000","last_clean_scrub_stamp":"2024-04-14T15:21:17.980009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:45.495930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"91'130","reported_seq":1130,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.111916+0000","last_change":"2024-04-14T15:19:05.396196+0000","last_active":"2024-04-14T15:21:28.111916+0000","last_peered":"2024-04-14T15:21:28.111916+0000","last_clean":"2024-04-14T15:21:28.111916+0000","last_became_active":"2024-04-14T15:18:40.917537+0000","last_became_peered":"2024-04-14T15:18:40.917537+0000","last_unstale":"2024-04-14T15:21:28.111916+0000","last_undegraded":"2024-04-14T15:21:28.111916+0000","last_fullsized":"2024-04-14T15:21:28.111916+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:05.396090+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:05.396090+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":942,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.106864+0000","last_change":"2024-04-14T15:20:37.765993+0000","last_active":"2024-04-14T15:21:28.106864+0000","last_peered":"2024-04-14T15:21:28.106864+0000","last_clean":"2024-04-14T15:21:28.106864+0000","last_became_active":"2024-04-14T15:12:56.427276+0000","last_became_peered":"2024-04-14T15:12:56.427276+0000","last_unstale":"2024-04-14T15:21:28.106864+0000","last_undegraded":"2024-04-14T15:21:28.106864+0000","last_fullsized":"2024-04-14T15:21:28.106864+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:37.765916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:20:37.765916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:45.507528+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"255'10","reported_seq":824,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.108849+0000","last_change":"2024-04-14T15:19:50.669678+0000","last_active":"2024-04-14T15:21:28.108849+0000","last_peered":"2024-04-14T15:21:28.108849+0000","last_clean":"2024-04-14T15:21:28.108849+0000","last_became_active":"2024-04-14T15:15:21.873747+0000","last_became_peered":"2024-04-14T15:15:21.873747+0000","last_unstale":"2024-04-14T15:21:28.108849+0000","last_undegraded":"2024-04-14T15:21:28.108849+0000","last_fullsized":"2024-04-14T15:21:28.108849+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'10","last_scrub_stamp":"2024-04-14T15:19:50.669574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:19:50.669574+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"479'28","reported_seq":1098,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.105504+0000","last_change":"2024-04-14T15:20:53.086781+0000","last_active":"2024-04-14T15:21:28.105504+0000","last_peered":"2024-04-14T15:21:28.105504+0000","last_clean":"2024-04-14T15:21:28.105504+0000","last_became_active":"2024-04-14T15:20:52.989635+0000","last_became_peered":"2024-04-14T15:20:52.989635+0000","last_unstale":"2024-04-14T15:21:28.105504+0000","last_undegraded":"2024-04-14T15:21:28.105504+0000","last_fullsized":"2024-04-14T15:21:28.105504+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'14","last_scrub_stamp":"2024-04-14T15:18:22.545646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:18:22.545646+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":78,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.118998+0000","last_change":"2024-04-14T15:20:52.999728+0000","last_active":"2024-04-14T15:21:28.118998+0000","last_peered":"2024-04-14T15:21:28.118998+0000","last_clean":"2024-04-14T15:21:28.118998+0000","last_became_active":"2024-04-14T15:20:52.999379+0000","last_became_peered":"2024-04-14T15:20:52.999379+0000","last_unstale":"2024-04-14T15:21:28.118998+0000","last_undegraded":"2024-04-14T15:21:28.118998+0000","last_fullsized":"2024-04-14T15:21:28.118998+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:41.799129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:20:41.799129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:07.383275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1032,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.117599+0000","last_change":"2024-04-14T15:20:52.991724+0000","last_active":"2024-04-14T15:21:28.117599+0000","last_peered":"2024-04-14T15:21:28.117599+0000","last_clean":"2024-04-14T15:21:28.117599+0000","last_became_active":"2024-04-14T15:20:52.991388+0000","last_became_peered":"2024-04-14T15:20:52.991388+0000","last_unstale":"2024-04-14T15:21:28.117599+0000","last_undegraded":"2024-04-14T15:21:28.117599+0000","last_fullsized":"2024-04-14T15:21:28.117599+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:50.609691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:19:50.609691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"91'130","reported_seq":1154,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.106521+0000","last_change":"2024-04-14T15:21:25.091353+0000","last_active":"2024-04-14T15:21:28.106521+0000","last_peered":"2024-04-14T15:21:28.106521+0000","last_clean":"2024-04-14T15:21:28.106521+0000","last_became_active":"2024-04-14T15:21:25.090964+0000","last_became_peered":"2024-04-14T15:21:25.090964+0000","last_unstale":"2024-04-14T15:21:28.106521+0000","last_undegraded":"2024-04-14T15:21:28.106521+0000","last_fullsized":"2024-04-14T15:21:28.106521+0000","mapping_epoch":530,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:04.526214+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:04.526214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":942,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.118886+0000","last_change":"2024-04-14T15:20:40.724743+0000","last_active":"2024-04-14T15:21:28.118886+0000","last_peered":"2024-04-14T15:21:28.118886+0000","last_clean":"2024-04-14T15:21:28.118886+0000","last_became_active":"2024-04-14T15:12:56.355254+0000","last_became_peered":"2024-04-14T15:12:56.355254+0000","last_unstale":"2024-04-14T15:21:28.118886+0000","last_undegraded":"2024-04-14T15:21:28.118886+0000","last_fullsized":"2024-04-14T15:21:28.118886+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:40.724658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:20:40.724658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:45.821791+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1030,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.105749+0000","last_change":"2024-04-14T15:18:39.394736+0000","last_active":"2024-04-14T15:21:28.105749+0000","last_peered":"2024-04-14T15:21:28.105749+0000","last_clean":"2024-04-14T15:21:28.105749+0000","last_became_active":"2024-04-14T15:09:47.044610+0000","last_became_peered":"2024-04-14T15:09:47.044610+0000","last_unstale":"2024-04-14T15:21:28.105749+0000","last_undegraded":"2024-04-14T15:21:28.105749+0000","last_fullsized":"2024-04-14T15:21:28.105749+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:39.394655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:39.394655+0000","last_clean_scrub_stamp":"2024-04-14T15:18:39.394655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"479'30","reported_seq":768,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.107720+0000","last_change":"2024-04-14T15:19:26.509637+0000","last_active":"2024-04-14T15:21:28.107720+0000","last_peered":"2024-04-14T15:21:28.107720+0000","last_clean":"2024-04-14T15:21:28.107720+0000","last_became_active":"2024-04-14T15:15:21.880565+0000","last_became_peered":"2024-04-14T15:15:21.880565+0000","last_unstale":"2024-04-14T15:21:28.107720+0000","last_undegraded":"2024-04-14T15:21:28.107720+0000","last_fullsized":"2024-04-14T15:21:28.107720+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'18","last_scrub_stamp":"2024-04-14T15:19:26.509541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:19:26.509541+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'2","reported_seq":1052,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.114757+0000","last_change":"2024-04-14T15:21:25.663010+0000","last_active":"2024-04-14T15:21:28.114757+0000","last_peered":"2024-04-14T15:21:28.114757+0000","last_clean":"2024-04-14T15:21:28.114757+0000","last_became_active":"2024-04-14T15:20:52.987576+0000","last_became_peered":"2024-04-14T15:20:52.987576+0000","last_unstale":"2024-04-14T15:21:28.114757+0000","last_undegraded":"2024-04-14T15:21:28.114757+0000","last_fullsized":"2024-04-14T15:21:28.114757+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'2","last_scrub_stamp":"2024-04-14T15:21:25.662901+0000","last_deep_scrub":"91'2","last_deep_scrub_stamp":"2024-04-14T15:17:48.850069+0000","last_clean_scrub_stamp":"2024-04-14T15:21:25.662901+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:46.779473+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1026,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.106052+0000","last_change":"2024-04-14T15:19:14.609181+0000","last_active":"2024-04-14T15:21:28.106052+0000","last_peered":"2024-04-14T15:21:28.106052+0000","last_clean":"2024-04-14T15:21:28.106052+0000","last_became_active":"2024-04-14T15:09:49.065684+0000","last_became_peered":"2024-04-14T15:09:49.065684+0000","last_unstale":"2024-04-14T15:21:28.106052+0000","last_undegraded":"2024-04-14T15:21:28.106052+0000","last_fullsized":"2024-04-14T15:21:28.106052+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:14.609079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:19:14.609079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"91'130","reported_seq":426,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.120297+0000","last_change":"2024-04-14T15:21:20.955422+0000","last_active":"2024-04-14T15:21:28.120297+0000","last_peered":"2024-04-14T15:21:28.120297+0000","last_clean":"2024-04-14T15:21:28.120297+0000","last_became_active":"2024-04-14T15:21:20.955039+0000","last_became_peered":"2024-04-14T15:21:20.955039+0000","last_unstale":"2024-04-14T15:21:28.120297+0000","last_undegraded":"2024-04-14T15:21:28.120297+0000","last_fullsized":"2024-04-14T15:21:28.120297+0000","mapping_epoch":530,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":556,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:21:11.531805+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:21:11.531805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:33.320815+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":936,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.113687+0000","last_change":"2024-04-14T15:18:35.004058+0000","last_active":"2024-04-14T15:21:28.113687+0000","last_peered":"2024-04-14T15:21:28.113687+0000","last_clean":"2024-04-14T15:21:28.113687+0000","last_became_active":"2024-04-14T15:12:56.402023+0000","last_became_peered":"2024-04-14T15:12:56.402023+0000","last_unstale":"2024-04-14T15:21:28.113687+0000","last_undegraded":"2024-04-14T15:21:28.113687+0000","last_fullsized":"2024-04-14T15:21:28.113687+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:35.003968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:35.003968+0000","last_clean_scrub_stamp":"2024-04-14T15:18:35.003968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":78,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.117571+0000","last_change":"2024-04-14T15:20:52.988230+0000","last_active":"2024-04-14T15:21:28.117571+0000","last_peered":"2024-04-14T15:21:28.117571+0000","last_clean":"2024-04-14T15:21:28.117571+0000","last_became_active":"2024-04-14T15:20:52.987883+0000","last_became_peered":"2024-04-14T15:20:52.987883+0000","last_unstale":"2024-04-14T15:21:28.117571+0000","last_undegraded":"2024-04-14T15:21:28.117571+0000","last_fullsized":"2024-04-14T15:21:28.117571+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:06.794023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:20:06.794023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"479'37","reported_seq":1102,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.118480+0000","last_change":"2024-04-14T15:21:12.559656+0000","last_active":"2024-04-14T15:21:28.118480+0000","last_peered":"2024-04-14T15:21:28.118480+0000","last_clean":"2024-04-14T15:21:28.118480+0000","last_became_active":"2024-04-14T15:09:45.179182+0000","last_became_peered":"2024-04-14T15:09:45.179182+0000","last_unstale":"2024-04-14T15:21:28.118480+0000","last_undegraded":"2024-04-14T15:21:28.118480+0000","last_fullsized":"2024-04-14T15:21:28.118480+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'37","last_scrub_stamp":"2024-04-14T15:21:12.559525+0000","last_deep_scrub":"102'27","last_deep_scrub_stamp":"2024-04-14T15:17:46.924330+0000","last_clean_scrub_stamp":"2024-04-14T15:21:12.559525+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:17.398104+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1038,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.117946+0000","last_change":"2024-04-14T15:19:14.569862+0000","last_active":"2024-04-14T15:21:28.117946+0000","last_peered":"2024-04-14T15:21:28.117946+0000","last_clean":"2024-04-14T15:21:28.117946+0000","last_became_active":"2024-04-14T15:09:43.021006+0000","last_became_peered":"2024-04-14T15:09:43.021006+0000","last_unstale":"2024-04-14T15:21:28.117946+0000","last_undegraded":"2024-04-14T15:21:28.117946+0000","last_fullsized":"2024-04-14T15:21:28.117946+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:14.569778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:19:14.569778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1034,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.109575+0000","last_change":"2024-04-14T15:20:52.979079+0000","last_active":"2024-04-14T15:21:28.109575+0000","last_peered":"2024-04-14T15:21:28.109575+0000","last_clean":"2024-04-14T15:21:28.109575+0000","last_became_active":"2024-04-14T15:20:52.978765+0000","last_became_peered":"2024-04-14T15:20:52.978765+0000","last_unstale":"2024-04-14T15:21:28.109575+0000","last_undegraded":"2024-04-14T15:21:28.109575+0000","last_fullsized":"2024-04-14T15:21:28.109575+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:00.976659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:56.206649+0000","last_clean_scrub_stamp":"2024-04-14T15:20:00.976659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"91'130","reported_seq":1141,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.111332+0000","last_change":"2024-04-14T15:20:55.990169+0000","last_active":"2024-04-14T15:21:28.111332+0000","last_peered":"2024-04-14T15:21:28.111332+0000","last_clean":"2024-04-14T15:21:28.111332+0000","last_became_active":"2024-04-14T15:20:55.989721+0000","last_became_peered":"2024-04-14T15:20:55.989721+0000","last_unstale":"2024-04-14T15:21:28.111332+0000","last_undegraded":"2024-04-14T15:21:28.111332+0000","last_fullsized":"2024-04-14T15:21:28.111332+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:09.060906+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:09.060906+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":654,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.107946+0000","last_change":"2024-04-14T15:20:53.000096+0000","last_active":"2024-04-14T15:21:28.107946+0000","last_peered":"2024-04-14T15:21:28.107946+0000","last_clean":"2024-04-14T15:21:28.107946+0000","last_became_active":"2024-04-14T15:20:52.999776+0000","last_became_peered":"2024-04-14T15:20:52.999776+0000","last_unstale":"2024-04-14T15:21:28.107946+0000","last_undegraded":"2024-04-14T15:21:28.107946+0000","last_fullsized":"2024-04-14T15:21:28.107946+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:47.610667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:19:47.610667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":590,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.115394+0000","last_change":"2024-04-14T15:17:47.871801+0000","last_active":"2024-04-14T15:21:28.115394+0000","last_peered":"2024-04-14T15:21:28.115394+0000","last_clean":"2024-04-14T15:21:28.115394+0000","last_became_active":"2024-04-14T15:15:44.883694+0000","last_became_peered":"2024-04-14T15:15:44.883694+0000","last_unstale":"2024-04-14T15:21:28.115394+0000","last_undegraded":"2024-04-14T15:21:28.115394+0000","last_fullsized":"2024-04-14T15:21:28.115394+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:47.871722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:17:47.871722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"479'30","reported_seq":1085,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.120696+0000","last_change":"2024-04-14T15:20:44.479671+0000","last_active":"2024-04-14T15:21:28.120696+0000","last_peered":"2024-04-14T15:21:28.120696+0000","last_clean":"2024-04-14T15:21:28.120696+0000","last_became_active":"2024-04-14T15:09:45.037231+0000","last_became_peered":"2024-04-14T15:09:45.037231+0000","last_unstale":"2024-04-14T15:21:28.120696+0000","last_undegraded":"2024-04-14T15:21:28.120696+0000","last_fullsized":"2024-04-14T15:21:28.120696+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'30","last_scrub_stamp":"2024-04-14T15:20:44.479555+0000","last_deep_scrub":"182'18","last_deep_scrub_stamp":"2024-04-14T15:14:10.383906+0000","last_clean_scrub_stamp":"2024-04-14T15:20:44.479555+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:53.234896+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1046,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.116677+0000","last_change":"2024-04-14T15:20:42.694332+0000","last_active":"2024-04-14T15:21:28.116677+0000","last_peered":"2024-04-14T15:21:28.116677+0000","last_clean":"2024-04-14T15:21:28.116677+0000","last_became_active":"2024-04-14T15:09:43.025095+0000","last_became_peered":"2024-04-14T15:09:43.025095+0000","last_unstale":"2024-04-14T15:21:28.116677+0000","last_undegraded":"2024-04-14T15:21:28.116677+0000","last_fullsized":"2024-04-14T15:21:28.116677+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:42.694243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:42.694243+0000","last_clean_scrub_stamp":"2024-04-14T15:20:42.694243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:47.503311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1054,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.117594+0000","last_change":"2024-04-14T15:21:17.524105+0000","last_active":"2024-04-14T15:21:28.117594+0000","last_peered":"2024-04-14T15:21:28.117594+0000","last_clean":"2024-04-14T15:21:28.117594+0000","last_became_active":"2024-04-14T15:09:43.018509+0000","last_became_peered":"2024-04-14T15:09:43.018509+0000","last_unstale":"2024-04-14T15:21:28.117594+0000","last_undegraded":"2024-04-14T15:21:28.117594+0000","last_fullsized":"2024-04-14T15:21:28.117594+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:17.524022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:21:17.524022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:30.574704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018592680000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"355'2","reported_seq":1030,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.113983+0000","last_change":"2024-04-14T15:20:04.842983+0000","last_active":"2024-04-14T15:21:28.113983+0000","last_peered":"2024-04-14T15:21:28.113983+0000","last_clean":"2024-04-14T15:21:28.113983+0000","last_became_active":"2024-04-14T15:09:49.065243+0000","last_became_peered":"2024-04-14T15:09:49.065243+0000","last_unstale":"2024-04-14T15:21:28.113983+0000","last_undegraded":"2024-04-14T15:21:28.113983+0000","last_fullsized":"2024-04-14T15:21:28.113983+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'2","last_scrub_stamp":"2024-04-14T15:20:04.842881+0000","last_deep_scrub":"355'2","last_deep_scrub_stamp":"2024-04-14T15:20:04.842881+0000","last_clean_scrub_stamp":"2024-04-14T15:20:04.842881+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":646,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.122700+0000","last_change":"2024-04-14T15:21:28.122721+0000","last_active":"2024-04-14T15:21:28.122700+0000","last_peered":"2024-04-14T15:21:28.122700+0000","last_clean":"2024-04-14T15:21:28.122700+0000","last_became_active":"2024-04-14T15:20:53.000058+0000","last_became_peered":"2024-04-14T15:20:53.000058+0000","last_unstale":"2024-04-14T15:21:28.122700+0000","last_undegraded":"2024-04-14T15:21:28.122700+0000","last_fullsized":"2024-04-14T15:21:28.122700+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:28.122647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:21:28.122647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:40.258966+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1030,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.118364+0000","last_change":"2024-04-14T15:19:46.631011+0000","last_active":"2024-04-14T15:21:28.118364+0000","last_peered":"2024-04-14T15:21:28.118364+0000","last_clean":"2024-04-14T15:21:28.118364+0000","last_became_active":"2024-04-14T15:09:47.047201+0000","last_became_peered":"2024-04-14T15:09:47.047201+0000","last_unstale":"2024-04-14T15:21:28.118364+0000","last_undegraded":"2024-04-14T15:21:28.118364+0000","last_fullsized":"2024-04-14T15:21:28.118364+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:46.630886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:19:46.630886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"528'37","reported_seq":1142,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.107880+0000","last_change":"2024-04-14T15:20:53.087600+0000","last_active":"2024-04-14T15:21:28.107880+0000","last_peered":"2024-04-14T15:21:28.107880+0000","last_clean":"2024-04-14T15:21:28.107880+0000","last_became_active":"2024-04-14T15:20:53.002282+0000","last_became_peered":"2024-04-14T15:20:53.002282+0000","last_unstale":"2024-04-14T15:21:28.107880+0000","last_undegraded":"2024-04-14T15:21:28.107880+0000","last_fullsized":"2024-04-14T15:21:28.107880+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'24","last_scrub_stamp":"2024-04-14T15:19:02.841668+0000","last_deep_scrub":"90'24","last_deep_scrub_stamp":"2024-04-14T15:11:25.120692+0000","last_clean_scrub_stamp":"2024-04-14T15:19:02.841668+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"91'130","reported_seq":1350,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.116582+0000","last_change":"2024-04-14T15:20:55.985118+0000","last_active":"2024-04-14T15:21:28.116582+0000","last_peered":"2024-04-14T15:21:28.116582+0000","last_clean":"2024-04-14T15:21:28.116582+0000","last_became_active":"2024-04-14T15:20:55.984832+0000","last_became_peered":"2024-04-14T15:20:55.984832+0000","last_unstale":"2024-04-14T15:21:28.116582+0000","last_undegraded":"2024-04-14T15:21:28.116582+0000","last_fullsized":"2024-04-14T15:21:28.116582+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:08.494666+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:08.494666+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":603,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.113002+0000","last_change":"2024-04-14T15:20:31.776844+0000","last_active":"2024-04-14T15:21:28.113002+0000","last_peered":"2024-04-14T15:21:28.113002+0000","last_clean":"2024-04-14T15:21:28.113002+0000","last_became_active":"2024-04-14T15:15:44.885315+0000","last_became_peered":"2024-04-14T15:15:44.885315+0000","last_unstale":"2024-04-14T15:21:28.113002+0000","last_undegraded":"2024-04-14T15:21:28.113002+0000","last_fullsized":"2024-04-14T15:21:28.113002+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:31.776767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:47.627305+0000","last_clean_scrub_stamp":"2024-04-14T15:20:31.776767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:40.461595+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"69'2","reported_seq":740,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.109334+0000","last_change":"2024-04-14T15:20:53.084362+0000","last_active":"2024-04-14T15:21:28.109334+0000","last_peered":"2024-04-14T15:21:28.109334+0000","last_clean":"2024-04-14T15:21:28.109334+0000","last_became_active":"2024-04-14T15:20:53.002545+0000","last_became_peered":"2024-04-14T15:20:53.002545+0000","last_unstale":"2024-04-14T15:21:28.109334+0000","last_undegraded":"2024-04-14T15:21:28.109334+0000","last_fullsized":"2024-04-14T15:21:28.109334+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-04-14T15:20:38.921819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:20:38.921819+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:54.930916+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0058046169999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":595,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.112583+0000","last_change":"2024-04-14T15:20:03.807964+0000","last_active":"2024-04-14T15:21:28.112583+0000","last_peered":"2024-04-14T15:21:28.112583+0000","last_clean":"2024-04-14T15:21:28.112583+0000","last_became_active":"2024-04-14T15:15:44.887934+0000","last_became_peered":"2024-04-14T15:15:44.887934+0000","last_unstale":"2024-04-14T15:21:28.112583+0000","last_undegraded":"2024-04-14T15:21:28.112583+0000","last_fullsized":"2024-04-14T15:21:28.112583+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:03.807844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:20:03.807844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"91'130","reported_seq":1351,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.116731+0000","last_change":"2024-04-14T15:19:07.501791+0000","last_active":"2024-04-14T15:21:28.116731+0000","last_peered":"2024-04-14T15:21:28.116731+0000","last_clean":"2024-04-14T15:21:28.116731+0000","last_became_active":"2024-04-14T15:18:40.912277+0000","last_became_peered":"2024-04-14T15:18:40.912277+0000","last_unstale":"2024-04-14T15:21:28.116731+0000","last_undegraded":"2024-04-14T15:21:28.116731+0000","last_fullsized":"2024-04-14T15:21:28.116731+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:07.501704+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:07.501704+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":646,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.107882+0000","last_change":"2024-04-14T15:20:52.986649+0000","last_active":"2024-04-14T15:21:28.107882+0000","last_peered":"2024-04-14T15:21:28.107882+0000","last_clean":"2024-04-14T15:21:28.107882+0000","last_became_active":"2024-04-14T15:20:52.986308+0000","last_became_peered":"2024-04-14T15:20:52.986308+0000","last_unstale":"2024-04-14T15:21:28.107882+0000","last_undegraded":"2024-04-14T15:21:28.107882+0000","last_fullsized":"2024-04-14T15:21:28.107882+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:46.589189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:19:46.589189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1037,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.117683+0000","last_change":"2024-04-14T15:20:00.850190+0000","last_active":"2024-04-14T15:21:28.117683+0000","last_peered":"2024-04-14T15:21:28.117683+0000","last_clean":"2024-04-14T15:21:28.117683+0000","last_became_active":"2024-04-14T15:09:47.049457+0000","last_became_peered":"2024-04-14T15:09:47.049457+0000","last_unstale":"2024-04-14T15:21:28.117683+0000","last_undegraded":"2024-04-14T15:21:28.117683+0000","last_fullsized":"2024-04-14T15:21:28.117683+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:00.850117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:20:00.850117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:29.884274+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"479'22","reported_seq":1074,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.115762+0000","last_change":"2024-04-14T15:19:26.078482+0000","last_active":"2024-04-14T15:21:28.115762+0000","last_peered":"2024-04-14T15:21:28.115762+0000","last_clean":"2024-04-14T15:21:28.115762+0000","last_became_active":"2024-04-14T15:09:45.033883+0000","last_became_peered":"2024-04-14T15:09:45.033883+0000","last_unstale":"2024-04-14T15:21:28.115762+0000","last_undegraded":"2024-04-14T15:21:28.115762+0000","last_fullsized":"2024-04-14T15:21:28.115762+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"308'18","last_scrub_stamp":"2024-04-14T15:19:26.078396+0000","last_deep_scrub":"308'18","last_deep_scrub_stamp":"2024-04-14T15:16:16.054032+0000","last_clean_scrub_stamp":"2024-04-14T15:19:26.078396+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1045,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.113782+0000","last_change":"2024-04-14T15:19:16.788873+0000","last_active":"2024-04-14T15:21:28.113782+0000","last_peered":"2024-04-14T15:21:28.113782+0000","last_clean":"2024-04-14T15:21:28.113782+0000","last_became_active":"2024-04-14T15:09:43.021188+0000","last_became_peered":"2024-04-14T15:09:43.021188+0000","last_unstale":"2024-04-14T15:21:28.113782+0000","last_undegraded":"2024-04-14T15:21:28.113782+0000","last_fullsized":"2024-04-14T15:21:28.113782+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:16.788795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:19:16.788795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"73'5","reported_seq":1759,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.110214+0000","last_change":"2024-04-14T15:20:53.001880+0000","last_active":"2024-04-14T15:21:28.110214+0000","last_peered":"2024-04-14T15:21:28.110214+0000","last_clean":"2024-04-14T15:21:28.110214+0000","last_became_active":"2024-04-14T15:20:52.975473+0000","last_became_peered":"2024-04-14T15:20:52.975473+0000","last_unstale":"2024-04-14T15:21:28.110214+0000","last_undegraded":"2024-04-14T15:21:28.110214+0000","last_fullsized":"2024-04-14T15:21:28.110214+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-04-14T15:18:26.955711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:18:26.955711+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00056542099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":698,"num_read_kb":693,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1032,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.118042+0000","last_change":"2024-04-14T15:20:53.000335+0000","last_active":"2024-04-14T15:21:28.118042+0000","last_peered":"2024-04-14T15:21:28.118042+0000","last_clean":"2024-04-14T15:21:28.118042+0000","last_became_active":"2024-04-14T15:20:52.999464+0000","last_became_peered":"2024-04-14T15:20:52.999464+0000","last_unstale":"2024-04-14T15:21:28.118042+0000","last_undegraded":"2024-04-14T15:21:28.118042+0000","last_fullsized":"2024-04-14T15:21:28.118042+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:22.535145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:20.827589+0000","last_clean_scrub_stamp":"2024-04-14T15:19:22.535145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"91'130","reported_seq":964,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.117264+0000","last_change":"2024-04-14T15:19:06.124178+0000","last_active":"2024-04-14T15:21:28.117264+0000","last_peered":"2024-04-14T15:21:28.117264+0000","last_clean":"2024-04-14T15:21:28.117264+0000","last_became_active":"2024-04-14T15:18:40.913215+0000","last_became_peered":"2024-04-14T15:18:40.913215+0000","last_unstale":"2024-04-14T15:21:28.117264+0000","last_undegraded":"2024-04-14T15:21:28.117264+0000","last_fullsized":"2024-04-14T15:21:28.117264+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:06.124071+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:06.124071+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":640,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.108818+0000","last_change":"2024-04-14T15:19:15.372917+0000","last_active":"2024-04-14T15:21:28.108818+0000","last_peered":"2024-04-14T15:21:28.108818+0000","last_clean":"2024-04-14T15:21:28.108818+0000","last_became_active":"2024-04-14T15:15:21.883717+0000","last_became_peered":"2024-04-14T15:15:21.883717+0000","last_unstale":"2024-04-14T15:21:28.108818+0000","last_undegraded":"2024-04-14T15:21:28.108818+0000","last_fullsized":"2024-04-14T15:21:28.108818+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:15.372837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:19:15.372837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1031,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.120802+0000","last_change":"2024-04-14T15:19:25.297173+0000","last_active":"2024-04-14T15:21:28.120802+0000","last_peered":"2024-04-14T15:21:28.120802+0000","last_clean":"2024-04-14T15:21:28.120802+0000","last_became_active":"2024-04-14T15:09:47.040185+0000","last_became_peered":"2024-04-14T15:09:47.040185+0000","last_unstale":"2024-04-14T15:21:28.120802+0000","last_undegraded":"2024-04-14T15:21:28.120802+0000","last_fullsized":"2024-04-14T15:21:28.120802+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:25.297044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:53.731610+0000","last_clean_scrub_stamp":"2024-04-14T15:19:25.297044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"479'40","reported_seq":1244,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.113716+0000","last_change":"2024-04-14T15:19:49.915826+0000","last_active":"2024-04-14T15:21:28.113716+0000","last_peered":"2024-04-14T15:21:28.113716+0000","last_clean":"2024-04-14T15:21:28.113716+0000","last_became_active":"2024-04-14T15:09:45.034168+0000","last_became_peered":"2024-04-14T15:09:45.034168+0000","last_unstale":"2024-04-14T15:21:28.113716+0000","last_undegraded":"2024-04-14T15:21:28.113716+0000","last_fullsized":"2024-04-14T15:21:28.113716+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'28","last_scrub_stamp":"2024-04-14T15:19:49.915672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:19:49.915672+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":604,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.111366+0000","last_change":"2024-04-14T15:20:52.983423+0000","last_active":"2024-04-14T15:21:28.111366+0000","last_peered":"2024-04-14T15:21:28.111366+0000","last_clean":"2024-04-14T15:21:28.111366+0000","last_became_active":"2024-04-14T15:20:52.979943+0000","last_became_peered":"2024-04-14T15:20:52.979943+0000","last_unstale":"2024-04-14T15:21:28.111366+0000","last_undegraded":"2024-04-14T15:21:28.111366+0000","last_fullsized":"2024-04-14T15:21:28.111366+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:24.066275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:24.066275+0000","last_clean_scrub_stamp":"2024-04-14T15:19:24.066275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":700,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.112368+0000","last_change":"2024-04-14T15:21:21.103290+0000","last_active":"2024-04-14T15:21:28.112368+0000","last_peered":"2024-04-14T15:21:28.112368+0000","last_clean":"2024-04-14T15:21:28.112368+0000","last_became_active":"2024-04-14T15:15:21.874742+0000","last_became_peered":"2024-04-14T15:15:21.874742+0000","last_unstale":"2024-04-14T15:21:28.112368+0000","last_undegraded":"2024-04-14T15:21:28.112368+0000","last_fullsized":"2024-04-14T15:21:28.112368+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:21.103213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:44.084010+0000","last_clean_scrub_stamp":"2024-04-14T15:21:21.103213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:42.553038+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"91'130","reported_seq":430,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.112068+0000","last_change":"2024-04-14T15:21:17.052747+0000","last_active":"2024-04-14T15:21:28.112068+0000","last_peered":"2024-04-14T15:21:28.112068+0000","last_clean":"2024-04-14T15:21:28.112068+0000","last_became_active":"2024-04-14T15:21:17.052430+0000","last_became_peered":"2024-04-14T15:21:17.052430+0000","last_unstale":"2024-04-14T15:21:28.112068+0000","last_undegraded":"2024-04-14T15:21:28.112068+0000","last_fullsized":"2024-04-14T15:21:28.112068+0000","mapping_epoch":530,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":552,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:21:11.055159+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:21:11.055159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:28.557272+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":947,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.116014+0000","last_change":"2024-04-14T15:20:52.989592+0000","last_active":"2024-04-14T15:21:28.116014+0000","last_peered":"2024-04-14T15:21:28.116014+0000","last_clean":"2024-04-14T15:21:28.116014+0000","last_became_active":"2024-04-14T15:20:52.989272+0000","last_became_peered":"2024-04-14T15:20:52.989272+0000","last_unstale":"2024-04-14T15:21:28.116014+0000","last_undegraded":"2024-04-14T15:21:28.116014+0000","last_fullsized":"2024-04-14T15:21:28.116014+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:55.858835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:19:55.858835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"528'9","reported_seq":1199,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.117857+0000","last_change":"2024-04-14T15:20:53.041034+0000","last_active":"2024-04-14T15:21:28.117857+0000","last_peered":"2024-04-14T15:21:28.117857+0000","last_clean":"2024-04-14T15:21:28.117857+0000","last_became_active":"2024-04-14T15:20:53.000669+0000","last_became_peered":"2024-04-14T15:20:53.000669+0000","last_unstale":"2024-04-14T15:21:28.117857+0000","last_undegraded":"2024-04-14T15:21:28.117857+0000","last_fullsized":"2024-04-14T15:21:28.117857+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:20:02.298301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:20:02.298301+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"479'20","reported_seq":43,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.110649+0000","last_change":"2024-04-14T15:21:16.061154+0000","last_active":"2024-04-14T15:21:28.110649+0000","last_peered":"2024-04-14T15:21:28.110649+0000","last_clean":"2024-04-14T15:21:28.110649+0000","last_became_active":"2024-04-14T15:21:16.034027+0000","last_became_peered":"2024-04-14T15:21:16.034027+0000","last_unstale":"2024-04-14T15:21:28.110649+0000","last_undegraded":"2024-04-14T15:21:28.110649+0000","last_fullsized":"2024-04-14T15:21:28.110649+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-04-14T15:18:38.870710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:18:38.870710+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1040,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.684921+0000","last_change":"2024-04-14T15:21:28.684960+0000","last_active":"2024-04-14T15:21:28.684921+0000","last_peered":"2024-04-14T15:21:28.684921+0000","last_clean":"2024-04-14T15:21:28.684921+0000","last_became_active":"2024-04-14T15:09:43.022597+0000","last_became_peered":"2024-04-14T15:09:43.022597+0000","last_unstale":"2024-04-14T15:21:28.684921+0000","last_undegraded":"2024-04-14T15:21:28.684921+0000","last_fullsized":"2024-04-14T15:21:28.684921+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:28.684818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:21:28.684818+0000","last_clean_scrub_stamp":"2024-04-14T15:21:28.684818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:50.508963+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1027,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.116930+0000","last_change":"2024-04-14T15:19:16.390131+0000","last_active":"2024-04-14T15:21:28.116930+0000","last_peered":"2024-04-14T15:21:28.116930+0000","last_clean":"2024-04-14T15:21:28.116930+0000","last_became_active":"2024-04-14T15:09:49.066648+0000","last_became_peered":"2024-04-14T15:09:49.066648+0000","last_unstale":"2024-04-14T15:21:28.116930+0000","last_undegraded":"2024-04-14T15:21:28.116930+0000","last_fullsized":"2024-04-14T15:21:28.116930+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:16.390049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:19:16.390049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.c","version":"91'130","reported_seq":1179,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.111813+0000","last_change":"2024-04-14T15:21:25.094760+0000","last_active":"2024-04-14T15:21:28.111813+0000","last_peered":"2024-04-14T15:21:28.111813+0000","last_clean":"2024-04-14T15:21:28.111813+0000","last_became_active":"2024-04-14T15:21:25.094286+0000","last_became_peered":"2024-04-14T15:21:25.094286+0000","last_unstale":"2024-04-14T15:21:28.111813+0000","last_undegraded":"2024-04-14T15:21:28.111813+0000","last_fullsized":"2024-04-14T15:21:28.111813+0000","mapping_epoch":559,"log_start":"91'130","ondisk_log_start":"91'130","created":431,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:04.397343+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:04.397343+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":60,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":943,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.120601+0000","last_change":"2024-04-14T15:21:26.419871+0000","last_active":"2024-04-14T15:21:28.120601+0000","last_peered":"2024-04-14T15:21:28.120601+0000","last_clean":"2024-04-14T15:21:28.120601+0000","last_became_active":"2024-04-14T15:12:56.563074+0000","last_became_peered":"2024-04-14T15:12:56.563074+0000","last_unstale":"2024-04-14T15:21:28.120601+0000","last_undegraded":"2024-04-14T15:21:28.120601+0000","last_fullsized":"2024-04-14T15:21:28.120601+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:26.419794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:21:26.419794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:42.513424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1037,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.112937+0000","last_change":"2024-04-14T15:21:14.769138+0000","last_active":"2024-04-14T15:21:28.112937+0000","last_peered":"2024-04-14T15:21:28.112937+0000","last_clean":"2024-04-14T15:21:28.112937+0000","last_became_active":"2024-04-14T15:09:47.049005+0000","last_became_peered":"2024-04-14T15:09:47.049005+0000","last_unstale":"2024-04-14T15:21:28.112937+0000","last_undegraded":"2024-04-14T15:21:28.112937+0000","last_fullsized":"2024-04-14T15:21:28.112937+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:14.769006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:56.915544+0000","last_clean_scrub_stamp":"2024-04-14T15:21:14.769006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:25.852822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"479'23","reported_seq":1079,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.105684+0000","last_change":"2024-04-14T15:19:44.525830+0000","last_active":"2024-04-14T15:21:28.105684+0000","last_peered":"2024-04-14T15:21:28.105684+0000","last_clean":"2024-04-14T15:21:28.105684+0000","last_became_active":"2024-04-14T15:09:45.032892+0000","last_became_peered":"2024-04-14T15:09:45.032892+0000","last_unstale":"2024-04-14T15:21:28.105684+0000","last_undegraded":"2024-04-14T15:21:28.105684+0000","last_fullsized":"2024-04-14T15:21:28.105684+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'17","last_scrub_stamp":"2024-04-14T15:19:44.525712+0000","last_deep_scrub":"183'17","last_deep_scrub_stamp":"2024-04-14T15:14:11.577574+0000","last_clean_scrub_stamp":"2024-04-14T15:19:44.525712+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"65'3","reported_seq":1056,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.107040+0000","last_change":"2024-04-14T15:19:22.603785+0000","last_active":"2024-04-14T15:21:28.107040+0000","last_peered":"2024-04-14T15:21:28.107040+0000","last_clean":"2024-04-14T15:21:28.107040+0000","last_became_active":"2024-04-14T15:09:43.020490+0000","last_became_peered":"2024-04-14T15:09:43.020490+0000","last_unstale":"2024-04-14T15:21:28.107040+0000","last_undegraded":"2024-04-14T15:21:28.107040+0000","last_fullsized":"2024-04-14T15:21:28.107040+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3","last_scrub_stamp":"2024-04-14T15:19:22.603651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:19:22.603651+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":528,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":700,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.109099+0000","last_change":"2024-04-14T15:20:34.966215+0000","last_active":"2024-04-14T15:21:28.109099+0000","last_peered":"2024-04-14T15:21:28.109099+0000","last_clean":"2024-04-14T15:21:28.109099+0000","last_became_active":"2024-04-14T15:15:21.876410+0000","last_became_peered":"2024-04-14T15:15:21.876410+0000","last_unstale":"2024-04-14T15:21:28.109099+0000","last_undegraded":"2024-04-14T15:21:28.109099+0000","last_fullsized":"2024-04-14T15:21:28.109099+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:34.966140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:20:34.966140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:42.982226+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"91'130","reported_seq":1142,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.110666+0000","last_change":"2024-04-14T15:19:07.091520+0000","last_active":"2024-04-14T15:21:28.110666+0000","last_peered":"2024-04-14T15:21:28.110666+0000","last_clean":"2024-04-14T15:21:28.110666+0000","last_became_active":"2024-04-14T15:18:45.912049+0000","last_became_peered":"2024-04-14T15:18:45.912049+0000","last_unstale":"2024-04-14T15:21:28.110666+0000","last_undegraded":"2024-04-14T15:21:28.110666+0000","last_fullsized":"2024-04-14T15:21:28.110666+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:07.091439+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:07.091439+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":943,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.117479+0000","last_change":"2024-04-14T15:19:24.289915+0000","last_active":"2024-04-14T15:21:28.117479+0000","last_peered":"2024-04-14T15:21:28.117479+0000","last_clean":"2024-04-14T15:21:28.117479+0000","last_became_active":"2024-04-14T15:12:56.551177+0000","last_became_peered":"2024-04-14T15:12:56.551177+0000","last_unstale":"2024-04-14T15:21:28.117479+0000","last_undegraded":"2024-04-14T15:21:28.117479+0000","last_fullsized":"2024-04-14T15:21:28.117479+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:24.289806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:19:24.289806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"479'19","reported_seq":1063,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.120789+0000","last_change":"2024-04-14T15:19:51.290310+0000","last_active":"2024-04-14T15:21:28.120789+0000","last_peered":"2024-04-14T15:21:28.120789+0000","last_clean":"2024-04-14T15:21:28.120789+0000","last_became_active":"2024-04-14T15:09:45.180375+0000","last_became_peered":"2024-04-14T15:09:45.180375+0000","last_unstale":"2024-04-14T15:21:28.120789+0000","last_undegraded":"2024-04-14T15:21:28.120789+0000","last_fullsized":"2024-04-14T15:21:28.120789+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'19","last_scrub_stamp":"2024-04-14T15:19:51.290211+0000","last_deep_scrub":"347'13","last_deep_scrub_stamp":"2024-04-14T15:16:55.694599+0000","last_clean_scrub_stamp":"2024-04-14T15:19:51.290211+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":1045,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.110395+0000","last_change":"2024-04-14T15:19:21.774315+0000","last_active":"2024-04-14T15:21:28.110395+0000","last_peered":"2024-04-14T15:21:28.110395+0000","last_clean":"2024-04-14T15:21:28.110395+0000","last_became_active":"2024-04-14T15:09:43.016788+0000","last_became_peered":"2024-04-14T15:09:43.016788+0000","last_unstale":"2024-04-14T15:21:28.110395+0000","last_undegraded":"2024-04-14T15:21:28.110395+0000","last_fullsized":"2024-04-14T15:21:28.110395+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:21.774184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:19:21.774184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":720,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:29.095614+0000","last_change":"2024-04-14T15:21:29.095634+0000","last_active":"2024-04-14T15:21:29.095614+0000","last_peered":"2024-04-14T15:21:29.095614+0000","last_clean":"2024-04-14T15:21:29.095614+0000","last_became_active":"2024-04-14T15:15:21.873580+0000","last_became_peered":"2024-04-14T15:15:21.873580+0000","last_unstale":"2024-04-14T15:21:29.095614+0000","last_undegraded":"2024-04-14T15:21:29.095614+0000","last_fullsized":"2024-04-14T15:21:29.095614+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:29.095557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:21:29.095557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:48.625664+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1032,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.115318+0000","last_change":"2024-04-14T15:19:24.821597+0000","last_active":"2024-04-14T15:21:28.115318+0000","last_peered":"2024-04-14T15:21:28.115318+0000","last_clean":"2024-04-14T15:21:28.115318+0000","last_became_active":"2024-04-14T15:09:49.064296+0000","last_became_peered":"2024-04-14T15:09:49.064296+0000","last_unstale":"2024-04-14T15:21:28.115318+0000","last_undegraded":"2024-04-14T15:21:28.115318+0000","last_fullsized":"2024-04-14T15:21:28.115318+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:24.821514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:19:24.821514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.12","version":"91'130","reported_seq":621,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.110824+0000","last_change":"2024-04-14T15:20:55.986603+0000","last_active":"2024-04-14T15:21:28.110824+0000","last_peered":"2024-04-14T15:21:28.110824+0000","last_clean":"2024-04-14T15:21:28.110824+0000","last_became_active":"2024-04-14T15:20:55.986287+0000","last_became_peered":"2024-04-14T15:20:55.986287+0000","last_unstale":"2024-04-14T15:21:28.110824+0000","last_undegraded":"2024-04-14T15:21:28.110824+0000","last_fullsized":"2024-04-14T15:21:28.110824+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:06.098672+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:06.098672+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":947,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.111409+0000","last_change":"2024-04-14T15:20:52.978238+0000","last_active":"2024-04-14T15:21:28.111409+0000","last_peered":"2024-04-14T15:21:28.111409+0000","last_clean":"2024-04-14T15:21:28.111409+0000","last_became_active":"2024-04-14T15:20:52.977894+0000","last_became_peered":"2024-04-14T15:20:52.977894+0000","last_unstale":"2024-04-14T15:21:28.111409+0000","last_undegraded":"2024-04-14T15:21:28.111409+0000","last_fullsized":"2024-04-14T15:21:28.111409+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:34.771789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:20:34.771789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:46.916331+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.17","version":"479'12","reported_seq":1057,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.112381+0000","last_change":"2024-04-14T15:20:41.939842+0000","last_active":"2024-04-14T15:21:28.112381+0000","last_peered":"2024-04-14T15:21:28.112381+0000","last_clean":"2024-04-14T15:21:28.112381+0000","last_became_active":"2024-04-14T15:09:45.180037+0000","last_became_peered":"2024-04-14T15:09:45.180037+0000","last_unstale":"2024-04-14T15:21:28.112381+0000","last_undegraded":"2024-04-14T15:21:28.112381+0000","last_fullsized":"2024-04-14T15:21:28.112381+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'12","last_scrub_stamp":"2024-04-14T15:20:41.939761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:20:41.939761+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:57.924063+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":78,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.105545+0000","last_change":"2024-04-14T15:20:52.987269+0000","last_active":"2024-04-14T15:21:28.105545+0000","last_peered":"2024-04-14T15:21:28.105545+0000","last_clean":"2024-04-14T15:21:28.105545+0000","last_became_active":"2024-04-14T15:20:52.986974+0000","last_became_peered":"2024-04-14T15:20:52.986974+0000","last_unstale":"2024-04-14T15:21:28.105545+0000","last_undegraded":"2024-04-14T15:21:28.105545+0000","last_fullsized":"2024-04-14T15:21:28.105545+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:23.915503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:59.640565+0000","last_clean_scrub_stamp":"2024-04-14T15:19:23.915503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":594,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.112131+0000","last_change":"2024-04-14T15:20:44.996397+0000","last_active":"2024-04-14T15:21:28.112131+0000","last_peered":"2024-04-14T15:21:28.112131+0000","last_clean":"2024-04-14T15:21:28.112131+0000","last_became_active":"2024-04-14T15:15:44.889786+0000","last_became_peered":"2024-04-14T15:15:44.889786+0000","last_unstale":"2024-04-14T15:21:28.112131+0000","last_undegraded":"2024-04-14T15:21:28.112131+0000","last_fullsized":"2024-04-14T15:21:28.112131+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:44.996260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:13.735024+0000","last_clean_scrub_stamp":"2024-04-14T15:20:44.996260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:58.629892+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1031,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.118823+0000","last_change":"2024-04-14T15:20:52.985470+0000","last_active":"2024-04-14T15:21:28.118823+0000","last_peered":"2024-04-14T15:21:28.118823+0000","last_clean":"2024-04-14T15:21:28.118823+0000","last_became_active":"2024-04-14T15:20:52.985136+0000","last_became_peered":"2024-04-14T15:20:52.985136+0000","last_unstale":"2024-04-14T15:21:28.118823+0000","last_undegraded":"2024-04-14T15:21:28.118823+0000","last_fullsized":"2024-04-14T15:21:28.118823+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:47.597268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:19:47.597268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.11","version":"91'130","reported_seq":972,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.117355+0000","last_change":"2024-04-14T15:19:05.136092+0000","last_active":"2024-04-14T15:21:28.117355+0000","last_peered":"2024-04-14T15:21:28.117355+0000","last_clean":"2024-04-14T15:21:28.117355+0000","last_became_active":"2024-04-14T15:18:45.915385+0000","last_became_peered":"2024-04-14T15:18:45.915385+0000","last_unstale":"2024-04-14T15:21:28.117355+0000","last_undegraded":"2024-04-14T15:21:28.117355+0000","last_fullsized":"2024-04-14T15:21:28.117355+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:05.136001+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:05.136001+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":942,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.105915+0000","last_change":"2024-04-14T15:21:20.009834+0000","last_active":"2024-04-14T15:21:28.105915+0000","last_peered":"2024-04-14T15:21:28.105915+0000","last_clean":"2024-04-14T15:21:28.105915+0000","last_became_active":"2024-04-14T15:12:56.203577+0000","last_became_peered":"2024-04-14T15:12:56.203577+0000","last_unstale":"2024-04-14T15:21:28.105915+0000","last_undegraded":"2024-04-14T15:21:28.105915+0000","last_fullsized":"2024-04-14T15:21:28.105915+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:20.009719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:46.369231+0000","last_clean_scrub_stamp":"2024-04-14T15:21:20.009719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:35.320776+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"479'20","reported_seq":746,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.112280+0000","last_change":"2024-04-14T15:18:36.534967+0000","last_active":"2024-04-14T15:21:28.112280+0000","last_peered":"2024-04-14T15:21:28.112280+0000","last_clean":"2024-04-14T15:21:28.112280+0000","last_became_active":"2024-04-14T15:15:21.877667+0000","last_became_peered":"2024-04-14T15:15:21.877667+0000","last_unstale":"2024-04-14T15:21:28.112280+0000","last_undegraded":"2024-04-14T15:21:28.112280+0000","last_fullsized":"2024-04-14T15:21:28.112280+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-04-14T15:18:36.534874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:18:36.534874+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":594,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.109523+0000","last_change":"2024-04-14T15:19:56.775138+0000","last_active":"2024-04-14T15:21:28.109523+0000","last_peered":"2024-04-14T15:21:28.109523+0000","last_clean":"2024-04-14T15:21:28.109523+0000","last_became_active":"2024-04-14T15:15:44.887452+0000","last_became_peered":"2024-04-14T15:15:44.887452+0000","last_unstale":"2024-04-14T15:21:28.109523+0000","last_undegraded":"2024-04-14T15:21:28.109523+0000","last_fullsized":"2024-04-14T15:21:28.109523+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:56.775036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:19:56.775036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"280'9","reported_seq":684,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.114260+0000","last_change":"2024-04-14T15:17:50.908130+0000","last_active":"2024-04-14T15:21:28.114260+0000","last_peered":"2024-04-14T15:21:28.114260+0000","last_clean":"2024-04-14T15:21:28.114260+0000","last_became_active":"2024-04-14T15:15:44.882916+0000","last_became_peered":"2024-04-14T15:15:44.882916+0000","last_unstale":"2024-04-14T15:21:28.114260+0000","last_undegraded":"2024-04-14T15:21:28.114260+0000","last_fullsized":"2024-04-14T15:21:28.114260+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"280'9","last_scrub_stamp":"2024-04-14T15:17:50.908054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:17:50.908054+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1026,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.120687+0000","last_change":"2024-04-14T15:20:30.356561+0000","last_active":"2024-04-14T15:21:28.120687+0000","last_peered":"2024-04-14T15:21:28.120687+0000","last_clean":"2024-04-14T15:21:28.120687+0000","last_became_active":"2024-04-14T15:09:49.066068+0000","last_became_peered":"2024-04-14T15:09:49.066068+0000","last_unstale":"2024-04-14T15:21:28.120687+0000","last_undegraded":"2024-04-14T15:21:28.120687+0000","last_fullsized":"2024-04-14T15:21:28.120687+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:30.356481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:54.724681+0000","last_clean_scrub_stamp":"2024-04-14T15:20:30.356481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:51.057447+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"91'130","reported_seq":1358,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.116213+0000","last_change":"2024-04-14T15:20:55.986015+0000","last_active":"2024-04-14T15:21:28.116213+0000","last_peered":"2024-04-14T15:21:28.116213+0000","last_clean":"2024-04-14T15:21:28.116213+0000","last_became_active":"2024-04-14T15:20:55.985433+0000","last_became_peered":"2024-04-14T15:20:55.985433+0000","last_unstale":"2024-04-14T15:21:28.116213+0000","last_undegraded":"2024-04-14T15:21:28.116213+0000","last_fullsized":"2024-04-14T15:21:28.116213+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:05.496303+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:05.496303+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":594,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.107839+0000","last_change":"2024-04-14T15:21:24.134615+0000","last_active":"2024-04-14T15:21:28.107839+0000","last_peered":"2024-04-14T15:21:28.107839+0000","last_clean":"2024-04-14T15:21:28.107839+0000","last_became_active":"2024-04-14T15:15:44.889106+0000","last_became_peered":"2024-04-14T15:15:44.889106+0000","last_unstale":"2024-04-14T15:21:28.107839+0000","last_undegraded":"2024-04-14T15:21:28.107839+0000","last_fullsized":"2024-04-14T15:21:28.107839+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:24.134540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:21:24.134540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:30.655006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"479'19","reported_seq":884,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.118634+0000","last_change":"2024-04-14T15:20:07.673953+0000","last_active":"2024-04-14T15:21:28.118634+0000","last_peered":"2024-04-14T15:21:28.118634+0000","last_clean":"2024-04-14T15:21:28.118634+0000","last_became_active":"2024-04-14T15:15:44.884211+0000","last_became_peered":"2024-04-14T15:15:44.884211+0000","last_unstale":"2024-04-14T15:21:28.118634+0000","last_undegraded":"2024-04-14T15:21:28.118634+0000","last_fullsized":"2024-04-14T15:21:28.118634+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'19","last_scrub_stamp":"2024-04-14T15:20:07.673794+0000","last_deep_scrub":"183'13","last_deep_scrub_stamp":"2024-04-14T15:14:11.755252+0000","last_clean_scrub_stamp":"2024-04-14T15:20:07.673794+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1047,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.118427+0000","last_change":"2024-04-14T15:20:32.388821+0000","last_active":"2024-04-14T15:21:28.118427+0000","last_peered":"2024-04-14T15:21:28.118427+0000","last_clean":"2024-04-14T15:21:28.118427+0000","last_became_active":"2024-04-14T15:09:43.019301+0000","last_became_peered":"2024-04-14T15:09:43.019301+0000","last_unstale":"2024-04-14T15:21:28.118427+0000","last_undegraded":"2024-04-14T15:21:28.118427+0000","last_fullsized":"2024-04-14T15:21:28.118427+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:32.388728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:20:32.388728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:50.382306+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"255'10","reported_seq":825,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.112742+0000","last_change":"2024-04-14T15:19:59.811456+0000","last_active":"2024-04-14T15:21:28.112742+0000","last_peered":"2024-04-14T15:21:28.112742+0000","last_clean":"2024-04-14T15:21:28.112742+0000","last_became_active":"2024-04-14T15:15:21.884468+0000","last_became_peered":"2024-04-14T15:15:21.884468+0000","last_unstale":"2024-04-14T15:21:28.112742+0000","last_undegraded":"2024-04-14T15:21:28.112742+0000","last_fullsized":"2024-04-14T15:21:28.112742+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'10","last_scrub_stamp":"2024-04-14T15:19:59.811370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:19:59.811370+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1025,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.107716+0000","last_change":"2024-04-14T15:19:15.642110+0000","last_active":"2024-04-14T15:21:28.107716+0000","last_peered":"2024-04-14T15:21:28.107716+0000","last_clean":"2024-04-14T15:21:28.107716+0000","last_became_active":"2024-04-14T15:09:49.065242+0000","last_became_peered":"2024-04-14T15:09:49.065242+0000","last_unstale":"2024-04-14T15:21:28.107716+0000","last_undegraded":"2024-04-14T15:21:28.107716+0000","last_fullsized":"2024-04-14T15:21:28.107716+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:15.642017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:19:15.642017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":942,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.118888+0000","last_change":"2024-04-14T15:21:16.642973+0000","last_active":"2024-04-14T15:21:28.118888+0000","last_peered":"2024-04-14T15:21:28.118888+0000","last_clean":"2024-04-14T15:21:28.118888+0000","last_became_active":"2024-04-14T15:12:56.389204+0000","last_became_peered":"2024-04-14T15:12:56.389204+0000","last_unstale":"2024-04-14T15:21:28.118888+0000","last_undegraded":"2024-04-14T15:21:28.118888+0000","last_fullsized":"2024-04-14T15:21:28.118888+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:16.642880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:21:16.642880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:20.606098+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"479'19","reported_seq":504,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.116063+0000","last_change":"2024-04-14T15:21:25.737152+0000","last_active":"2024-04-14T15:21:28.116063+0000","last_peered":"2024-04-14T15:21:28.116063+0000","last_clean":"2024-04-14T15:21:28.116063+0000","last_became_active":"2024-04-14T15:20:52.990622+0000","last_became_peered":"2024-04-14T15:20:52.990622+0000","last_unstale":"2024-04-14T15:21:28.116063+0000","last_undegraded":"2024-04-14T15:21:28.116063+0000","last_fullsized":"2024-04-14T15:21:28.116063+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'19","last_scrub_stamp":"2024-04-14T15:21:25.737057+0000","last_deep_scrub":"344'13","last_deep_scrub_stamp":"2024-04-14T15:16:52.738163+0000","last_clean_scrub_stamp":"2024-04-14T15:21:25.737057+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:33.539332+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":78,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.112431+0000","last_change":"2024-04-14T15:20:52.998638+0000","last_active":"2024-04-14T15:21:28.112431+0000","last_peered":"2024-04-14T15:21:28.112431+0000","last_clean":"2024-04-14T15:21:28.112431+0000","last_became_active":"2024-04-14T15:20:52.996548+0000","last_became_peered":"2024-04-14T15:20:52.996548+0000","last_unstale":"2024-04-14T15:21:28.112431+0000","last_undegraded":"2024-04-14T15:21:28.112431+0000","last_fullsized":"2024-04-14T15:21:28.112431+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:24.898086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:19:24.898086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":712,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.108737+0000","last_change":"2024-04-14T15:21:23.131643+0000","last_active":"2024-04-14T15:21:28.108737+0000","last_peered":"2024-04-14T15:21:28.108737+0000","last_clean":"2024-04-14T15:21:28.108737+0000","last_became_active":"2024-04-14T15:15:21.882643+0000","last_became_peered":"2024-04-14T15:15:21.882643+0000","last_unstale":"2024-04-14T15:21:28.108737+0000","last_undegraded":"2024-04-14T15:21:28.108737+0000","last_fullsized":"2024-04-14T15:21:28.108737+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:23.131512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:33.485411+0000","last_clean_scrub_stamp":"2024-04-14T15:21:23.131512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:44.884317+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1038,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.116950+0000","last_change":"2024-04-14T15:21:25.392328+0000","last_active":"2024-04-14T15:21:28.116950+0000","last_peered":"2024-04-14T15:21:28.116950+0000","last_clean":"2024-04-14T15:21:28.116950+0000","last_became_active":"2024-04-14T15:09:49.059711+0000","last_became_peered":"2024-04-14T15:09:49.059711+0000","last_unstale":"2024-04-14T15:21:28.116950+0000","last_undegraded":"2024-04-14T15:21:28.116950+0000","last_fullsized":"2024-04-14T15:21:28.116950+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:25.392246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:21:25.392246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:44.024700+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":78,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.117797+0000","last_change":"2024-04-14T15:20:52.992281+0000","last_active":"2024-04-14T15:21:28.117797+0000","last_peered":"2024-04-14T15:21:28.117797+0000","last_clean":"2024-04-14T15:21:28.117797+0000","last_became_active":"2024-04-14T15:20:52.991608+0000","last_became_peered":"2024-04-14T15:20:52.991608+0000","last_unstale":"2024-04-14T15:21:28.117797+0000","last_undegraded":"2024-04-14T15:21:28.117797+0000","last_fullsized":"2024-04-14T15:21:28.117797+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:26.891862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:19:26.891862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"479'24","reported_seq":1088,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.106104+0000","last_change":"2024-04-14T15:20:53.071526+0000","last_active":"2024-04-14T15:21:28.106104+0000","last_peered":"2024-04-14T15:21:28.106104+0000","last_clean":"2024-04-14T15:21:28.106104+0000","last_became_active":"2024-04-14T15:20:52.989300+0000","last_became_peered":"2024-04-14T15:20:52.989300+0000","last_unstale":"2024-04-14T15:21:28.106104+0000","last_undegraded":"2024-04-14T15:21:28.106104+0000","last_fullsized":"2024-04-14T15:21:28.106104+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"464'16","last_scrub_stamp":"2024-04-14T15:19:17.589999+0000","last_deep_scrub":"464'16","last_deep_scrub_stamp":"2024-04-14T15:19:17.589999+0000","last_clean_scrub_stamp":"2024-04-14T15:19:17.589999+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":1047,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.114314+0000","last_change":"2024-04-14T15:20:45.587983+0000","last_active":"2024-04-14T15:21:28.114314+0000","last_peered":"2024-04-14T15:21:28.114314+0000","last_clean":"2024-04-14T15:21:28.114314+0000","last_became_active":"2024-04-14T15:09:43.023583+0000","last_became_peered":"2024-04-14T15:09:43.023583+0000","last_unstale":"2024-04-14T15:21:28.114314+0000","last_undegraded":"2024-04-14T15:21:28.114314+0000","last_fullsized":"2024-04-14T15:21:28.114314+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:45.587898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:20:45.587898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:56.691537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1032,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.111752+0000","last_change":"2024-04-14T15:20:01.971662+0000","last_active":"2024-04-14T15:21:28.111752+0000","last_peered":"2024-04-14T15:21:28.111752+0000","last_clean":"2024-04-14T15:21:28.111752+0000","last_became_active":"2024-04-14T15:09:47.041683+0000","last_became_peered":"2024-04-14T15:09:47.041683+0000","last_unstale":"2024-04-14T15:21:28.111752+0000","last_undegraded":"2024-04-14T15:21:28.111752+0000","last_fullsized":"2024-04-14T15:21:28.111752+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:01.971564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:08.486217+0000","last_clean_scrub_stamp":"2024-04-14T15:20:01.971564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":708,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.107666+0000","last_change":"2024-04-14T15:19:57.757627+0000","last_active":"2024-04-14T15:21:28.107666+0000","last_peered":"2024-04-14T15:21:28.107666+0000","last_clean":"2024-04-14T15:21:28.107666+0000","last_became_active":"2024-04-14T15:15:21.889567+0000","last_became_peered":"2024-04-14T15:15:21.889567+0000","last_unstale":"2024-04-14T15:21:28.107666+0000","last_undegraded":"2024-04-14T15:21:28.107666+0000","last_fullsized":"2024-04-14T15:21:28.107666+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:57.757547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:19:57.757547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":942,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.118078+0000","last_change":"2024-04-14T15:19:53.922021+0000","last_active":"2024-04-14T15:21:28.118078+0000","last_peered":"2024-04-14T15:21:28.118078+0000","last_clean":"2024-04-14T15:21:28.118078+0000","last_became_active":"2024-04-14T15:12:56.387723+0000","last_became_peered":"2024-04-14T15:12:56.387723+0000","last_unstale":"2024-04-14T15:21:28.118078+0000","last_undegraded":"2024-04-14T15:21:28.118078+0000","last_fullsized":"2024-04-14T15:21:28.118078+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:53.921934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:03.685045+0000","last_clean_scrub_stamp":"2024-04-14T15:19:53.921934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.10","version":"479'8","reported_seq":1064,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.107584+0000","last_change":"2024-04-14T15:21:20.054070+0000","last_active":"2024-04-14T15:21:28.107584+0000","last_peered":"2024-04-14T15:21:28.107584+0000","last_clean":"2024-04-14T15:21:28.107584+0000","last_became_active":"2024-04-14T15:20:53.001300+0000","last_became_peered":"2024-04-14T15:20:53.001300+0000","last_unstale":"2024-04-14T15:21:28.107584+0000","last_undegraded":"2024-04-14T15:21:28.107584+0000","last_fullsized":"2024-04-14T15:21:28.107584+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'8","last_scrub_stamp":"2024-04-14T15:21:20.053929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:21:20.053929+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:22.446916+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":81,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.116574+0000","last_change":"2024-04-14T15:20:52.979797+0000","last_active":"2024-04-14T15:21:28.116574+0000","last_peered":"2024-04-14T15:21:28.116574+0000","last_clean":"2024-04-14T15:21:28.116574+0000","last_became_active":"2024-04-14T15:20:52.979496+0000","last_became_peered":"2024-04-14T15:20:52.979496+0000","last_unstale":"2024-04-14T15:21:28.116574+0000","last_undegraded":"2024-04-14T15:21:28.116574+0000","last_fullsized":"2024-04-14T15:21:28.116574+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:39.844329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:08.712289+0000","last_clean_scrub_stamp":"2024-04-14T15:20:39.844329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:54.092229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":78,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.105423+0000","last_change":"2024-04-14T15:20:52.991440+0000","last_active":"2024-04-14T15:21:28.105423+0000","last_peered":"2024-04-14T15:21:28.105423+0000","last_clean":"2024-04-14T15:21:28.105423+0000","last_became_active":"2024-04-14T15:20:52.990347+0000","last_became_peered":"2024-04-14T15:20:52.990347+0000","last_unstale":"2024-04-14T15:21:28.105423+0000","last_undegraded":"2024-04-14T15:21:28.105423+0000","last_fullsized":"2024-04-14T15:21:28.105423+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:48.867704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:19:48.867704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1026,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.117314+0000","last_change":"2024-04-14T15:19:59.634301+0000","last_active":"2024-04-14T15:21:28.117314+0000","last_peered":"2024-04-14T15:21:28.117314+0000","last_clean":"2024-04-14T15:21:28.117314+0000","last_became_active":"2024-04-14T15:09:49.061359+0000","last_became_peered":"2024-04-14T15:09:49.061359+0000","last_unstale":"2024-04-14T15:21:28.117314+0000","last_undegraded":"2024-04-14T15:21:28.117314+0000","last_fullsized":"2024-04-14T15:21:28.117314+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:59.634212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:19:59.634212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"91'130","reported_seq":1133,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.113973+0000","last_change":"2024-04-14T15:19:06.939565+0000","last_active":"2024-04-14T15:21:28.113973+0000","last_peered":"2024-04-14T15:21:28.113973+0000","last_clean":"2024-04-14T15:21:28.113973+0000","last_became_active":"2024-04-14T15:18:45.913983+0000","last_became_peered":"2024-04-14T15:18:45.913983+0000","last_unstale":"2024-04-14T15:21:28.113973+0000","last_undegraded":"2024-04-14T15:21:28.113973+0000","last_fullsized":"2024-04-14T15:21:28.113973+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:19:06.939444+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:19:06.939444+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":949,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.711336+0000","last_change":"2024-04-14T15:21:28.711361+0000","last_active":"2024-04-14T15:21:28.711336+0000","last_peered":"2024-04-14T15:21:28.711336+0000","last_clean":"2024-04-14T15:21:28.711336+0000","last_became_active":"2024-04-14T15:20:52.999623+0000","last_became_peered":"2024-04-14T15:20:52.999623+0000","last_unstale":"2024-04-14T15:21:28.711336+0000","last_undegraded":"2024-04-14T15:21:28.711336+0000","last_fullsized":"2024-04-14T15:21:28.711336+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:28.711266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:15:43.680852+0000","last_clean_scrub_stamp":"2024-04-14T15:21:28.711266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:34.716705+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"479'24","reported_seq":731,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.108621+0000","last_change":"2024-04-14T15:21:13.011149+0000","last_active":"2024-04-14T15:21:28.108621+0000","last_peered":"2024-04-14T15:21:28.108621+0000","last_clean":"2024-04-14T15:21:28.108621+0000","last_became_active":"2024-04-14T15:20:52.996917+0000","last_became_peered":"2024-04-14T15:20:52.996917+0000","last_unstale":"2024-04-14T15:21:28.108621+0000","last_undegraded":"2024-04-14T15:21:28.108621+0000","last_fullsized":"2024-04-14T15:21:28.108621+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'24","last_scrub_stamp":"2024-04-14T15:21:13.011012+0000","last_deep_scrub":"102'16","last_deep_scrub_stamp":"2024-04-14T15:12:35.999326+0000","last_clean_scrub_stamp":"2024-04-14T15:21:13.011012+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:41.142654+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":78,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.118171+0000","last_change":"2024-04-14T15:20:52.971272+0000","last_active":"2024-04-14T15:21:28.118171+0000","last_peered":"2024-04-14T15:21:28.118171+0000","last_clean":"2024-04-14T15:21:28.118171+0000","last_became_active":"2024-04-14T15:20:52.970959+0000","last_became_peered":"2024-04-14T15:20:52.970959+0000","last_unstale":"2024-04-14T15:21:28.118171+0000","last_undegraded":"2024-04-14T15:21:28.118171+0000","last_fullsized":"2024-04-14T15:21:28.118171+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:11.750419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:19:11.750419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1036,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.116372+0000","last_change":"2024-04-14T15:20:52.984259+0000","last_active":"2024-04-14T15:21:28.116372+0000","last_peered":"2024-04-14T15:21:28.116372+0000","last_clean":"2024-04-14T15:21:28.116372+0000","last_became_active":"2024-04-14T15:20:52.983919+0000","last_became_peered":"2024-04-14T15:20:52.983919+0000","last_unstale":"2024-04-14T15:21:28.116372+0000","last_undegraded":"2024-04-14T15:21:28.116372+0000","last_fullsized":"2024-04-14T15:21:28.116372+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:54.634279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:19:54.634279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":714,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.109036+0000","last_change":"2024-04-14T15:20:52.994265+0000","last_active":"2024-04-14T15:21:28.109036+0000","last_peered":"2024-04-14T15:21:28.109036+0000","last_clean":"2024-04-14T15:21:28.109036+0000","last_became_active":"2024-04-14T15:20:52.993918+0000","last_became_peered":"2024-04-14T15:20:52.993918+0000","last_unstale":"2024-04-14T15:21:28.109036+0000","last_undegraded":"2024-04-14T15:21:28.109036+0000","last_fullsized":"2024-04-14T15:21:28.109036+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:23.462727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:19:23.462727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":941,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.111069+0000","last_change":"2024-04-14T15:21:15.764125+0000","last_active":"2024-04-14T15:21:28.111069+0000","last_peered":"2024-04-14T15:21:28.111069+0000","last_clean":"2024-04-14T15:21:28.111069+0000","last_became_active":"2024-04-14T15:12:56.335230+0000","last_became_peered":"2024-04-14T15:12:56.335230+0000","last_unstale":"2024-04-14T15:21:28.111069+0000","last_undegraded":"2024-04-14T15:21:28.111069+0000","last_fullsized":"2024-04-14T15:21:28.111069+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:15.764017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:21:15.764017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:39.038735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"479'20","reported_seq":1065,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.120887+0000","last_change":"2024-04-14T15:20:06.373257+0000","last_active":"2024-04-14T15:21:28.120887+0000","last_peered":"2024-04-14T15:21:28.120887+0000","last_clean":"2024-04-14T15:21:28.120887+0000","last_became_active":"2024-04-14T15:09:45.181456+0000","last_became_peered":"2024-04-14T15:09:45.181456+0000","last_unstale":"2024-04-14T15:21:28.120887+0000","last_undegraded":"2024-04-14T15:21:28.120887+0000","last_fullsized":"2024-04-14T15:21:28.120887+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'20","last_scrub_stamp":"2024-04-14T15:20:06.373158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:20:06.373158+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":596,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.106242+0000","last_change":"2024-04-14T15:19:57.575006+0000","last_active":"2024-04-14T15:21:28.106242+0000","last_peered":"2024-04-14T15:21:28.106242+0000","last_clean":"2024-04-14T15:21:28.106242+0000","last_became_active":"2024-04-14T15:15:44.878336+0000","last_became_peered":"2024-04-14T15:15:44.878336+0000","last_unstale":"2024-04-14T15:21:28.106242+0000","last_undegraded":"2024-04-14T15:21:28.106242+0000","last_fullsized":"2024-04-14T15:21:28.106242+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:57.574879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:19:57.574879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"528'9","reported_seq":118,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.107881+0000","last_change":"2024-04-14T15:20:53.001282+0000","last_active":"2024-04-14T15:21:28.107881+0000","last_peered":"2024-04-14T15:21:28.107881+0000","last_clean":"2024-04-14T15:21:28.107881+0000","last_became_active":"2024-04-14T15:20:52.982112+0000","last_became_peered":"2024-04-14T15:20:52.982112+0000","last_unstale":"2024-04-14T15:21:28.107881+0000","last_undegraded":"2024-04-14T15:21:28.107881+0000","last_fullsized":"2024-04-14T15:21:28.107881+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:20:45.809489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:20:45.809489+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:53.411092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"72'1","reported_seq":1034,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.118525+0000","last_change":"2024-04-14T15:20:04.672008+0000","last_active":"2024-04-14T15:21:28.118525+0000","last_peered":"2024-04-14T15:21:28.118525+0000","last_clean":"2024-04-14T15:21:28.118525+0000","last_became_active":"2024-04-14T15:09:49.062051+0000","last_became_peered":"2024-04-14T15:09:49.062051+0000","last_unstale":"2024-04-14T15:21:28.118525+0000","last_undegraded":"2024-04-14T15:21:28.118525+0000","last_fullsized":"2024-04-14T15:21:28.118525+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-04-14T15:20:04.671896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:20:04.671896+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":941,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.117019+0000","last_change":"2024-04-14T15:20:52.986956+0000","last_active":"2024-04-14T15:21:28.117019+0000","last_peered":"2024-04-14T15:21:28.117019+0000","last_clean":"2024-04-14T15:21:28.117019+0000","last_became_active":"2024-04-14T15:20:52.986657+0000","last_became_peered":"2024-04-14T15:20:52.986657+0000","last_unstale":"2024-04-14T15:21:28.117019+0000","last_undegraded":"2024-04-14T15:21:28.117019+0000","last_fullsized":"2024-04-14T15:21:28.117019+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:13.345316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:19:13.345316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"519'24","reported_seq":1093,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.118498+0000","last_change":"2024-04-14T15:20:53.032992+0000","last_active":"2024-04-14T15:21:28.118498+0000","last_peered":"2024-04-14T15:21:28.118498+0000","last_clean":"2024-04-14T15:21:28.118498+0000","last_became_active":"2024-04-14T15:20:52.987333+0000","last_became_peered":"2024-04-14T15:20:52.987333+0000","last_unstale":"2024-04-14T15:21:28.118498+0000","last_undegraded":"2024-04-14T15:21:28.118498+0000","last_fullsized":"2024-04-14T15:21:28.118498+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"519'24","last_scrub_stamp":"2024-04-14T15:20:44.763417+0000","last_deep_scrub":"519'24","last_deep_scrub_stamp":"2024-04-14T15:20:44.763417+0000","last_clean_scrub_stamp":"2024-04-14T15:20:44.763417+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:48.205621+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.19","version":"65'2","reported_seq":739,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.107839+0000","last_change":"2024-04-14T15:20:53.064772+0000","last_active":"2024-04-14T15:21:28.107839+0000","last_peered":"2024-04-14T15:21:28.107839+0000","last_clean":"2024-04-14T15:21:28.107839+0000","last_became_active":"2024-04-14T15:20:52.995628+0000","last_became_peered":"2024-04-14T15:20:52.995628+0000","last_unstale":"2024-04-14T15:21:28.107839+0000","last_undegraded":"2024-04-14T15:21:28.107839+0000","last_fullsized":"2024-04-14T15:21:28.107839+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-04-14T15:20:36.939845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:20:36.939845+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:56.679088+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":957,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1031,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.120897+0000","last_change":"2024-04-14T15:19:21.349524+0000","last_active":"2024-04-14T15:21:28.120897+0000","last_peered":"2024-04-14T15:21:28.120897+0000","last_clean":"2024-04-14T15:21:28.120897+0000","last_became_active":"2024-04-14T15:09:47.045730+0000","last_became_peered":"2024-04-14T15:09:47.045730+0000","last_unstale":"2024-04-14T15:21:28.120897+0000","last_undegraded":"2024-04-14T15:21:28.120897+0000","last_fullsized":"2024-04-14T15:21:28.120897+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:21.349441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:19:21.349441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1027,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.113976+0000","last_change":"2024-04-14T15:20:38.518996+0000","last_active":"2024-04-14T15:21:28.113976+0000","last_peered":"2024-04-14T15:21:28.113976+0000","last_clean":"2024-04-14T15:21:28.113976+0000","last_became_active":"2024-04-14T15:09:49.057545+0000","last_became_peered":"2024-04-14T15:09:49.057545+0000","last_unstale":"2024-04-14T15:21:28.113976+0000","last_undegraded":"2024-04-14T15:21:28.113976+0000","last_fullsized":"2024-04-14T15:21:28.113976+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:38.518885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:21.247695+0000","last_clean_scrub_stamp":"2024-04-14T15:20:38.518885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:53.962959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":588,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.105859+0000","last_change":"2024-04-14T15:17:44.364114+0000","last_active":"2024-04-14T15:21:28.105859+0000","last_peered":"2024-04-14T15:21:28.105859+0000","last_clean":"2024-04-14T15:21:28.105859+0000","last_became_active":"2024-04-14T15:15:44.882604+0000","last_became_peered":"2024-04-14T15:15:44.882604+0000","last_unstale":"2024-04-14T15:21:28.105859+0000","last_undegraded":"2024-04-14T15:21:28.105859+0000","last_fullsized":"2024-04-14T15:21:28.105859+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:44.364013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:17:44.364013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"479'27","reported_seq":1083,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.115550+0000","last_change":"2024-04-14T15:21:20.766716+0000","last_active":"2024-04-14T15:21:28.115550+0000","last_peered":"2024-04-14T15:21:28.115550+0000","last_clean":"2024-04-14T15:21:28.115550+0000","last_became_active":"2024-04-14T15:09:45.032887+0000","last_became_peered":"2024-04-14T15:09:45.032887+0000","last_unstale":"2024-04-14T15:21:28.115550+0000","last_undegraded":"2024-04-14T15:21:28.115550+0000","last_fullsized":"2024-04-14T15:21:28.115550+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'27","last_scrub_stamp":"2024-04-14T15:21:20.766623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:21:20.766623+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:23.939904+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":78,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.105794+0000","last_change":"2024-04-14T15:20:52.990991+0000","last_active":"2024-04-14T15:21:28.105794+0000","last_peered":"2024-04-14T15:21:28.105794+0000","last_clean":"2024-04-14T15:21:28.105794+0000","last_became_active":"2024-04-14T15:20:52.990634+0000","last_became_peered":"2024-04-14T15:20:52.990634+0000","last_unstale":"2024-04-14T15:21:28.105794+0000","last_undegraded":"2024-04-14T15:21:28.105794+0000","last_fullsized":"2024-04-14T15:21:28.105794+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:42.754721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:42.754721+0000","last_clean_scrub_stamp":"2024-04-14T15:20:42.754721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:10.447793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1031,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.114254+0000","last_change":"2024-04-14T15:18:22.953738+0000","last_active":"2024-04-14T15:21:28.114254+0000","last_peered":"2024-04-14T15:21:28.114254+0000","last_clean":"2024-04-14T15:21:28.114254+0000","last_became_active":"2024-04-14T15:09:47.042733+0000","last_became_peered":"2024-04-14T15:09:47.042733+0000","last_unstale":"2024-04-14T15:21:28.114254+0000","last_undegraded":"2024-04-14T15:21:28.114254+0000","last_fullsized":"2024-04-14T15:21:28.114254+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:22.953661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:18:22.953661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1032,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.106996+0000","last_change":"2024-04-14T15:20:52.998739+0000","last_active":"2024-04-14T15:21:28.106996+0000","last_peered":"2024-04-14T15:21:28.106996+0000","last_clean":"2024-04-14T15:21:28.106996+0000","last_became_active":"2024-04-14T15:20:52.998253+0000","last_became_peered":"2024-04-14T15:20:52.998253+0000","last_unstale":"2024-04-14T15:21:28.106996+0000","last_undegraded":"2024-04-14T15:21:28.106996+0000","last_fullsized":"2024-04-14T15:21:28.106996+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:17:54.453174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:54.453174+0000","last_clean_scrub_stamp":"2024-04-14T15:17:54.453174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":942,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.116938+0000","last_change":"2024-04-14T15:21:17.675180+0000","last_active":"2024-04-14T15:21:28.116938+0000","last_peered":"2024-04-14T15:21:28.116938+0000","last_clean":"2024-04-14T15:21:28.116938+0000","last_became_active":"2024-04-14T15:12:56.392352+0000","last_became_peered":"2024-04-14T15:12:56.392352+0000","last_unstale":"2024-04-14T15:21:28.116938+0000","last_undegraded":"2024-04-14T15:21:28.116938+0000","last_fullsized":"2024-04-14T15:21:28.116938+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:17.675093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:21:17.675093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:34.933651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"479'18","reported_seq":748,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.108293+0000","last_change":"2024-04-14T15:19:17.364178+0000","last_active":"2024-04-14T15:21:28.108293+0000","last_peered":"2024-04-14T15:21:28.108293+0000","last_clean":"2024-04-14T15:21:28.108293+0000","last_became_active":"2024-04-14T15:15:21.877838+0000","last_became_peered":"2024-04-14T15:15:21.877838+0000","last_unstale":"2024-04-14T15:21:28.108293+0000","last_undegraded":"2024-04-14T15:21:28.108293+0000","last_fullsized":"2024-04-14T15:21:28.108293+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'12","last_scrub_stamp":"2024-04-14T15:19:17.364059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:19:17.364059+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":726,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.108904+0000","last_change":"2024-04-14T15:20:53.001854+0000","last_active":"2024-04-14T15:21:28.108904+0000","last_peered":"2024-04-14T15:21:28.108904+0000","last_clean":"2024-04-14T15:21:28.108904+0000","last_became_active":"2024-04-14T15:20:53.001534+0000","last_became_peered":"2024-04-14T15:20:53.001534+0000","last_unstale":"2024-04-14T15:21:28.108904+0000","last_undegraded":"2024-04-14T15:21:28.108904+0000","last_fullsized":"2024-04-14T15:21:28.108904+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:25.470160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:19:25.470160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1037,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.106337+0000","last_change":"2024-04-14T15:20:39.717630+0000","last_active":"2024-04-14T15:21:28.106337+0000","last_peered":"2024-04-14T15:21:28.106337+0000","last_clean":"2024-04-14T15:21:28.106337+0000","last_became_active":"2024-04-14T15:09:47.048045+0000","last_became_peered":"2024-04-14T15:09:47.048045+0000","last_unstale":"2024-04-14T15:21:28.106337+0000","last_undegraded":"2024-04-14T15:21:28.106337+0000","last_fullsized":"2024-04-14T15:21:28.106337+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:39.717522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:20:39.717522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:08.064104+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":715,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.112287+0000","last_change":"2024-04-14T15:20:52.999103+0000","last_active":"2024-04-14T15:21:28.112287+0000","last_peered":"2024-04-14T15:21:28.112287+0000","last_clean":"2024-04-14T15:21:28.112287+0000","last_became_active":"2024-04-14T15:20:52.998775+0000","last_became_peered":"2024-04-14T15:20:52.998775+0000","last_unstale":"2024-04-14T15:21:28.112287+0000","last_undegraded":"2024-04-14T15:21:28.112287+0000","last_fullsized":"2024-04-14T15:21:28.112287+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:19.371182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:19:19.371182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":941,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.116106+0000","last_change":"2024-04-14T15:20:01.841015+0000","last_active":"2024-04-14T15:21:28.116106+0000","last_peered":"2024-04-14T15:21:28.116106+0000","last_clean":"2024-04-14T15:21:28.116106+0000","last_became_active":"2024-04-14T15:12:56.375278+0000","last_became_peered":"2024-04-14T15:12:56.375278+0000","last_unstale":"2024-04-14T15:21:28.116106+0000","last_undegraded":"2024-04-14T15:21:28.116106+0000","last_fullsized":"2024-04-14T15:21:28.116106+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:01.840924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:01.840924+0000","last_clean_scrub_stamp":"2024-04-14T15:20:01.840924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"479'19","reported_seq":1070,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.105864+0000","last_change":"2024-04-14T15:21:22.085022+0000","last_active":"2024-04-14T15:21:28.105864+0000","last_peered":"2024-04-14T15:21:28.105864+0000","last_clean":"2024-04-14T15:21:28.105864+0000","last_became_active":"2024-04-14T15:09:45.030093+0000","last_became_peered":"2024-04-14T15:09:45.030093+0000","last_unstale":"2024-04-14T15:21:28.105864+0000","last_undegraded":"2024-04-14T15:21:28.105864+0000","last_fullsized":"2024-04-14T15:21:28.105864+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'19","last_scrub_stamp":"2024-04-14T15:21:22.084927+0000","last_deep_scrub":"276'13","last_deep_scrub_stamp":"2024-04-14T15:15:43.969775+0000","last_clean_scrub_stamp":"2024-04-14T15:21:22.084927+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:51.437329+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":743,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.117642+0000","last_change":"2024-04-14T15:20:00.650164+0000","last_active":"2024-04-14T15:21:28.117642+0000","last_peered":"2024-04-14T15:21:28.117642+0000","last_clean":"2024-04-14T15:21:28.117642+0000","last_became_active":"2024-04-14T15:14:30.857727+0000","last_became_peered":"2024-04-14T15:14:30.857727+0000","last_unstale":"2024-04-14T15:21:28.117642+0000","last_undegraded":"2024-04-14T15:21:28.117642+0000","last_fullsized":"2024-04-14T15:21:28.117642+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:00.650062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:00.650062+0000","last_clean_scrub_stamp":"2024-04-14T15:20:00.650062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":597,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.118504+0000","last_change":"2024-04-14T15:20:36.422786+0000","last_active":"2024-04-14T15:21:28.118504+0000","last_peered":"2024-04-14T15:21:28.118504+0000","last_clean":"2024-04-14T15:21:28.118504+0000","last_became_active":"2024-04-14T15:15:44.883121+0000","last_became_peered":"2024-04-14T15:15:44.883121+0000","last_unstale":"2024-04-14T15:21:28.118504+0000","last_undegraded":"2024-04-14T15:21:28.118504+0000","last_fullsized":"2024-04-14T15:21:28.118504+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:36.422671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:20:36.422671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:45.547619+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1034,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.117259+0000","last_change":"2024-04-14T15:20:42.444544+0000","last_active":"2024-04-14T15:21:28.117259+0000","last_peered":"2024-04-14T15:21:28.117259+0000","last_clean":"2024-04-14T15:21:28.117259+0000","last_became_active":"2024-04-14T15:09:49.061756+0000","last_became_peered":"2024-04-14T15:09:49.061756+0000","last_unstale":"2024-04-14T15:21:28.117259+0000","last_undegraded":"2024-04-14T15:21:28.117259+0000","last_fullsized":"2024-04-14T15:21:28.117259+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:42.444464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:20:42.444464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:44.158027+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":78,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.106134+0000","last_change":"2024-04-14T15:20:52.991133+0000","last_active":"2024-04-14T15:21:28.106134+0000","last_peered":"2024-04-14T15:21:28.106134+0000","last_clean":"2024-04-14T15:21:28.106134+0000","last_became_active":"2024-04-14T15:20:52.990838+0000","last_became_peered":"2024-04-14T15:20:52.990838+0000","last_unstale":"2024-04-14T15:21:28.106134+0000","last_undegraded":"2024-04-14T15:21:28.106134+0000","last_fullsized":"2024-04-14T15:21:28.106134+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:12.723073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:19:12.723073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"479'9","reported_seq":1048,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.106112+0000","last_change":"2024-04-14T15:18:35.481608+0000","last_active":"2024-04-14T15:21:28.106112+0000","last_peered":"2024-04-14T15:21:28.106112+0000","last_clean":"2024-04-14T15:21:28.106112+0000","last_became_active":"2024-04-14T15:09:45.028667+0000","last_became_peered":"2024-04-14T15:09:45.028667+0000","last_unstale":"2024-04-14T15:21:28.106112+0000","last_undegraded":"2024-04-14T15:21:28.106112+0000","last_fullsized":"2024-04-14T15:21:28.106112+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'7","last_scrub_stamp":"2024-04-14T15:18:35.481504+0000","last_deep_scrub":"87'7","last_deep_scrub_stamp":"2024-04-14T15:11:10.510785+0000","last_clean_scrub_stamp":"2024-04-14T15:18:35.481504+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":520,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.115506+0000","last_change":"2024-04-14T15:20:52.989677+0000","last_active":"2024-04-14T15:21:28.115506+0000","last_peered":"2024-04-14T15:21:28.115506+0000","last_clean":"2024-04-14T15:21:28.115506+0000","last_became_active":"2024-04-14T15:20:52.989307+0000","last_became_peered":"2024-04-14T15:20:52.989307+0000","last_unstale":"2024-04-14T15:21:28.115506+0000","last_undegraded":"2024-04-14T15:21:28.115506+0000","last_fullsized":"2024-04-14T15:21:28.115506+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:26.879685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:08.254601+0000","last_clean_scrub_stamp":"2024-04-14T15:19:26.879685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1029,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.117173+0000","last_change":"2024-04-14T15:19:15.567537+0000","last_active":"2024-04-14T15:21:28.117173+0000","last_peered":"2024-04-14T15:21:28.117173+0000","last_clean":"2024-04-14T15:21:28.117173+0000","last_became_active":"2024-04-14T15:09:47.046517+0000","last_became_peered":"2024-04-14T15:09:47.046517+0000","last_unstale":"2024-04-14T15:21:28.117173+0000","last_undegraded":"2024-04-14T15:21:28.117173+0000","last_fullsized":"2024-04-14T15:21:28.117173+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:15.567432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:19:15.567432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":591,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.109666+0000","last_change":"2024-04-14T15:18:21.023508+0000","last_active":"2024-04-14T15:21:28.109666+0000","last_peered":"2024-04-14T15:21:28.109666+0000","last_clean":"2024-04-14T15:21:28.109666+0000","last_became_active":"2024-04-14T15:15:44.887227+0000","last_became_peered":"2024-04-14T15:15:44.887227+0000","last_unstale":"2024-04-14T15:21:28.109666+0000","last_undegraded":"2024-04-14T15:21:28.109666+0000","last_fullsized":"2024-04-14T15:21:28.109666+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:18:21.023426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:56.268086+0000","last_clean_scrub_stamp":"2024-04-14T15:18:21.023426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":948,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.118722+0000","last_change":"2024-04-14T15:21:21.668496+0000","last_active":"2024-04-14T15:21:28.118722+0000","last_peered":"2024-04-14T15:21:28.118722+0000","last_clean":"2024-04-14T15:21:28.118722+0000","last_became_active":"2024-04-14T15:20:52.984387+0000","last_became_peered":"2024-04-14T15:20:52.984387+0000","last_unstale":"2024-04-14T15:21:28.118722+0000","last_undegraded":"2024-04-14T15:21:28.118722+0000","last_fullsized":"2024-04-14T15:21:28.118722+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:21.668407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:34.558312+0000","last_clean_scrub_stamp":"2024-04-14T15:21:21.668407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:22.229729+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.18","version":"479'15","reported_seq":1074,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.106044+0000","last_change":"2024-04-14T15:20:53.012987+0000","last_active":"2024-04-14T15:21:28.106044+0000","last_peered":"2024-04-14T15:21:28.106044+0000","last_clean":"2024-04-14T15:21:28.106044+0000","last_became_active":"2024-04-14T15:20:52.985210+0000","last_became_peered":"2024-04-14T15:20:52.985210+0000","last_unstale":"2024-04-14T15:21:28.106044+0000","last_undegraded":"2024-04-14T15:21:28.106044+0000","last_fullsized":"2024-04-14T15:21:28.106044+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-04-14T15:18:36.441813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:18:36.441813+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":722,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.112883+0000","last_change":"2024-04-14T15:20:35.984146+0000","last_active":"2024-04-14T15:21:28.112883+0000","last_peered":"2024-04-14T15:21:28.112883+0000","last_clean":"2024-04-14T15:21:28.112883+0000","last_became_active":"2024-04-14T15:15:21.891926+0000","last_became_peered":"2024-04-14T15:15:21.891926+0000","last_unstale":"2024-04-14T15:21:28.112883+0000","last_undegraded":"2024-04-14T15:21:28.112883+0000","last_fullsized":"2024-04-14T15:21:28.112883+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:35.984075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:20:35.984075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:01.610043+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1031,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.109400+0000","last_change":"2024-04-14T15:19:56.903138+0000","last_active":"2024-04-14T15:21:28.109400+0000","last_peered":"2024-04-14T15:21:28.109400+0000","last_clean":"2024-04-14T15:21:28.109400+0000","last_became_active":"2024-04-14T15:09:47.041486+0000","last_became_peered":"2024-04-14T15:09:47.041486+0000","last_unstale":"2024-04-14T15:21:28.109400+0000","last_undegraded":"2024-04-14T15:21:28.109400+0000","last_fullsized":"2024-04-14T15:21:28.109400+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:56.903059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:19:56.903059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1025,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.105636+0000","last_change":"2024-04-14T15:20:36.761316+0000","last_active":"2024-04-14T15:21:28.105636+0000","last_peered":"2024-04-14T15:21:28.105636+0000","last_clean":"2024-04-14T15:21:28.105636+0000","last_became_active":"2024-04-14T15:09:49.058904+0000","last_became_peered":"2024-04-14T15:09:49.058904+0000","last_unstale":"2024-04-14T15:21:28.105636+0000","last_undegraded":"2024-04-14T15:21:28.105636+0000","last_fullsized":"2024-04-14T15:21:28.105636+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:36.761246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:20:36.761246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:21:58.179590+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":596,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.118315+0000","last_change":"2024-04-14T15:21:14.673688+0000","last_active":"2024-04-14T15:21:28.118315+0000","last_peered":"2024-04-14T15:21:28.118315+0000","last_clean":"2024-04-14T15:21:28.118315+0000","last_became_active":"2024-04-14T15:15:44.889605+0000","last_became_peered":"2024-04-14T15:15:44.889605+0000","last_unstale":"2024-04-14T15:21:28.118315+0000","last_undegraded":"2024-04-14T15:21:28.118315+0000","last_fullsized":"2024-04-14T15:21:28.118315+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:14.673591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:21:14.673591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:23.061298+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"479'34","reported_seq":766,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.108716+0000","last_change":"2024-04-14T15:19:53.781367+0000","last_active":"2024-04-14T15:21:28.108716+0000","last_peered":"2024-04-14T15:21:28.108716+0000","last_clean":"2024-04-14T15:21:28.108716+0000","last_became_active":"2024-04-14T15:15:21.884310+0000","last_became_peered":"2024-04-14T15:15:21.884310+0000","last_unstale":"2024-04-14T15:21:28.108716+0000","last_undegraded":"2024-04-14T15:21:28.108716+0000","last_fullsized":"2024-04-14T15:21:28.108716+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'34","last_scrub_stamp":"2024-04-14T15:19:53.781241+0000","last_deep_scrub":"306'22","last_deep_scrub_stamp":"2024-04-14T15:16:13.898106+0000","last_clean_scrub_stamp":"2024-04-14T15:19:53.781241+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":1039,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.120679+0000","last_change":"2024-04-14T15:19:18.347632+0000","last_active":"2024-04-14T15:21:28.120679+0000","last_peered":"2024-04-14T15:21:28.120679+0000","last_clean":"2024-04-14T15:21:28.120679+0000","last_became_active":"2024-04-14T15:09:43.017351+0000","last_became_peered":"2024-04-14T15:09:43.017351+0000","last_unstale":"2024-04-14T15:21:28.120679+0000","last_undegraded":"2024-04-14T15:21:28.120679+0000","last_fullsized":"2024-04-14T15:21:28.120679+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:18.347533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:19:18.347533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1035,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.106352+0000","last_change":"2024-04-14T15:20:52.976225+0000","last_active":"2024-04-14T15:21:28.106352+0000","last_peered":"2024-04-14T15:21:28.106352+0000","last_clean":"2024-04-14T15:21:28.106352+0000","last_became_active":"2024-04-14T15:20:52.974360+0000","last_became_peered":"2024-04-14T15:20:52.974360+0000","last_unstale":"2024-04-14T15:21:28.106352+0000","last_undegraded":"2024-04-14T15:21:28.106352+0000","last_fullsized":"2024-04-14T15:21:28.106352+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:49.631273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:08.512130+0000","last_clean_scrub_stamp":"2024-04-14T15:19:49.631273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1037,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-04-14T15:21:28.107980+0000","last_change":"2024-04-14T15:21:22.119138+0000","last_active":"2024-04-14T15:21:28.107980+0000","last_peered":"2024-04-14T15:21:28.107980+0000","last_clean":"2024-04-14T15:21:28.107980+0000","last_became_active":"2024-04-14T15:20:52.998938+0000","last_became_peered":"2024-04-14T15:20:52.998938+0000","last_unstale":"2024-04-14T15:21:28.107980+0000","last_undegraded":"2024-04-14T15:21:28.107980+0000","last_fullsized":"2024-04-14T15:21:28.107980+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:22.119061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:21:22.119061+0000","last_clean_scrub_stamp":"2024-04-14T15:21:22.119061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:35.653564+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":313,"num_pg":32,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":11,"num_read_kb":6,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16,"ondisk_log_size":16,"up":96,"acting":96,"num_store_stats":7},{"poolid":110,"num_pg":32,"stat_sum":{"num_bytes":8214804,"num_objects":199,"num_object_clones":15,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":217,"num_read_kb":62,"num_write":786,"num_write_kb":12814,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":220,"num_bytes_recovered":4215,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":27,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":26947584,"data_stored":24644439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2117,"ondisk_log_size":2117,"up":96,"acting":96,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":5,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1003,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2109440,"data_stored":2098784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1430,"ondisk_log_size":1430,"up":63,"acting":63,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":698,"num_read_kb":693,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1121,"num_read_kb":878,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":1474,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":785,"ondisk_log_size":785,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103315,"num_pgs":115,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":60104,"kb_used_data":10876,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93685048,"statfs":{"total":95995035648,"available":95933489152,"internally_reserved":0,"allocated":11137024,"data_stored":9172943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":210453397654,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575024,"num_pgs":108,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":74276,"kb_used_data":7000,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93670876,"statfs":{"total":95995035648,"available":95918977024,"internally_reserved":0,"allocated":7168000,"data_stored":5178937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738549,"num_pgs":102,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":86836,"kb_used_data":7080,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":93658316,"statfs":{"total":95995035648,"available":95906115584,"internally_reserved":0,"allocated":7249920,"data_stored":5308443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":81656263},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411500,"num_pgs":106,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":86900,"kb_used_data":7144,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":93658252,"statfs":{"total":95995035648,"available":95906050048,"internally_reserved":0,"allocated":7315456,"data_stored":5242182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81656261},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182566,"num_pgs":105,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":56768,"kb_used_data":3384,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93688384,"statfs":{"total":95995035648,"available":95936905216,"internally_reserved":0,"allocated":3465216,"data_stored":1564675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986336,"num_pgs":104,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":74504,"kb_used_data":3064,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93670648,"statfs":{"total":95995035648,"available":95918743552,"internally_reserved":0,"allocated":3137536,"data_stored":1294616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659293,"num_pgs":104,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":78644,"kb_used_data":7208,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93666508,"statfs":{"total":95995035648,"available":95914504192,"internally_reserved":0,"allocated":7380992,"data_stored":5437354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4317184,"data_stored":4009416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4567040,"data_stored":4208617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4476928,"data_stored":4009091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":253952,"data_stored":3385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":229376,"data_stored":5833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4538368,"data_stored":4205070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":110,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8564736,"data_stored":8203027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":313,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":20,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":313,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":313,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":20,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":313,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":313,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":20,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":313,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":313,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-14T15:21:36.576 INFO:tasks.cephadm.ceph_manager.ceph:decrease pool size by 10 2024-04-14T15:21:36.577 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd pool set .mgr pg_num 11 2024-04-14T15:21:36.659 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:21:36.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:36.684 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:36.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:21:36.785 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:21:36.812 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:36.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:36.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:21:36.914 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:21:36.938 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:36.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:36.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:21:37.040 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:37.065 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:37.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:37.066 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:37.167 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:21:37.183 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:36 smithi090 ceph-mon[41670]: from='client.49426 v1:172.21.15.90:0/1507913373' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:21:37.183 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:36 smithi090 ceph-mon[41670]: pgmap v823: 248 pgs: 248 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-14T15:21:37.183 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:36 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:21:37.183 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:36 smithi090 ceph-mon[41670]: osdmap e564: 8 total, 8 up, 7 in 2024-04-14T15:21:37.184 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:36 smithi090 ceph-mon[30399]: from='client.49426 v1:172.21.15.90:0/1507913373' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:21:37.184 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:36 smithi090 ceph-mon[30399]: pgmap v823: 248 pgs: 248 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-14T15:21:37.184 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:36 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:21:37.184 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:36 smithi090 ceph-mon[30399]: osdmap e564: 8 total, 8 up, 7 in 2024-04-14T15:21:37.193 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:37.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:37.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:21:37.294 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:21:37.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:37.320 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:37.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:21:37.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:36 smithi092 ceph-mon[33966]: from='client.49426 v1:172.21.15.90:0/1507913373' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:21:37.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:36 smithi092 ceph-mon[33966]: pgmap v823: 248 pgs: 248 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-14T15:21:37.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:36 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:21:37.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:36 smithi092 ceph-mon[33966]: osdmap e564: 8 total, 8 up, 7 in 2024-04-14T15:21:37.422 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:21:37.447 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:37.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:37.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:21:37.549 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:21:37.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:37.573 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:37.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:21:37.674 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:21:37.699 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:37.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:37.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:21:37.801 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:37.827 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:37.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:37.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:37.929 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:37.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:37.956 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:37.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:38.057 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:21:38.083 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:38.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:38.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:21:38.185 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:21:38.211 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:38.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:38.212 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:21:38.313 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:21:38.336 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:38.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:38.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:21:38.438 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:21:38.461 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:38.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:38.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:21:38.562 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:21:38.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:38.587 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:38.587 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:21:38.688 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:38.715 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:38.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:38.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:38.818 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:21:38.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:38.846 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:38.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:21:38.947 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:21:38.977 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:38.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:38.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:21:39.079 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:21:39.104 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:39.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:39.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:21:39.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:38 smithi090 ceph-mon[41670]: pgmap v825: 248 pgs: 248 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 828 B/s rd, 0 op/s 2024-04-14T15:21:39.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:38 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:21:39.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:38 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:21:39.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:38 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-94"}]: dispatch 2024-04-14T15:21:39.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:38 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-94"}]: dispatch 2024-04-14T15:21:39.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:38 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:21:39.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:38 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:21:39.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:38 smithi090 ceph-mon[30399]: pgmap v825: 248 pgs: 248 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 828 B/s rd, 0 op/s 2024-04-14T15:21:39.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:38 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:21:39.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:38 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:21:39.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:38 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-94"}]: dispatch 2024-04-14T15:21:39.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:38 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-94"}]: dispatch 2024-04-14T15:21:39.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:38 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:21:39.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:38 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:21:39.206 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:21:39.230 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:39.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:39.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:21:39.332 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:21:39.357 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:39.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:39.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:21:39.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:38 smithi092 ceph-mon[33966]: pgmap v825: 248 pgs: 248 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 828 B/s rd, 0 op/s 2024-04-14T15:21:39.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:38 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:21:39.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:38 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:21:39.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:38 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-94"}]: dispatch 2024-04-14T15:21:39.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:38 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-94"}]: dispatch 2024-04-14T15:21:39.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:38 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:21:39.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:38 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:21:39.460 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:21:39.485 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:39.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:39.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:21:39.587 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:21:39.611 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:39.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:39.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:21:39.713 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:21:39.740 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:39.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:39.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:21:39.842 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:21:39.865 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:39.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:39.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:21:39.966 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:21:40.005 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:40.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:40.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:21:40.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.042+0000 7fc6342a0700 1 -- 172.21.15.90:0/2993852712 >> v1:172.21.15.90:6790/0 conn(0x7fc62c108a30 legacy=0x7fc62c108e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:40.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.043+0000 7fc6342a0700 1 -- 172.21.15.90:0/2993852712 shutdown_connections 2024-04-14T15:21:40.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.043+0000 7fc6342a0700 1 -- 172.21.15.90:0/2993852712 >> 172.21.15.90:0/2993852712 conn(0x7fc62c076ab0 msgr2=0x7fc62c076ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:21:40.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.043+0000 7fc6342a0700 1 -- 172.21.15.90:0/2993852712 shutdown_connections 2024-04-14T15:21:40.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.043+0000 7fc6342a0700 1 -- 172.21.15.90:0/2993852712 wait complete. 2024-04-14T15:21:40.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.043+0000 7fc6342a0700 1 librados: starting msgr at 2024-04-14T15:21:40.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.043+0000 7fc6342a0700 1 librados: starting objecter 2024-04-14T15:21:40.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.043+0000 7fc6342a0700 1 Processor -- start 2024-04-14T15:21:40.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.043+0000 7fc6342a0700 1 -- start start 2024-04-14T15:21:40.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.043+0000 7fc6342a0700 1 librados: setting wanted keys 2024-04-14T15:21:40.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.043+0000 7fc6342a0700 1 librados: calling monclient init 2024-04-14T15:21:40.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.043+0000 7fc6342a0700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc62c1a7a40 con 0x7fc62c10b830 2024-04-14T15:21:40.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.043+0000 7fc6342a0700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc62c1a8c60 con 0x7fc62c108a30 2024-04-14T15:21:40.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.043+0000 7fc6342a0700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc62c1a9ea0 con 0x7fc62c10f430 2024-04-14T15:21:40.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.043+0000 7fc63183b700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fc62c10b830 0x7fc62c1a28c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:45694/0 (socket says 172.21.15.90:45694) 2024-04-14T15:21:40.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.043+0000 7fc63183b700 1 -- 172.21.15.90:0/2217948461 learned_addr learned my addr 172.21.15.90:0/2217948461 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:21:40.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.044+0000 7fc6237fe700 10 client.?.objecter ms_handle_connect 0x7fc62c10b830 2024-04-14T15:21:40.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.044+0000 7fc6237fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:21:40.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.044+0000 7fc6237fe700 10 client.?.objecter ms_handle_connect 0x7fc62c108a30 2024-04-14T15:21:40.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.044+0000 7fc6237fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:21:40.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.044+0000 7fc6237fe700 10 client.?.objecter ms_handle_connect 0x7fc62c10f430 2024-04-14T15:21:40.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.044+0000 7fc6237fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:21:40.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.044+0000 7fc6237fe700 1 -- 172.21.15.90:0/2217948461 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3117910752 0 0) 0x7fc62c1a7a40 con 0x7fc62c10b830 2024-04-14T15:21:40.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.044+0000 7fc6237fe700 1 -- 172.21.15.90:0/2217948461 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc618003aa0 con 0x7fc62c10b830 2024-04-14T15:21:40.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.044+0000 7fc6237fe700 1 -- 172.21.15.90:0/2217948461 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 44734199 0 0) 0x7fc62c1a8c60 con 0x7fc62c108a30 2024-04-14T15:21:40.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.044+0000 7fc6237fe700 1 -- 172.21.15.90:0/2217948461 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc62c1a7a40 con 0x7fc62c108a30 2024-04-14T15:21:40.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.044+0000 7fc6237fe700 1 -- 172.21.15.90:0/2217948461 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1925996944 0 0) 0x7fc62c1a9ea0 con 0x7fc62c10f430 2024-04-14T15:21:40.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.044+0000 7fc6237fe700 1 -- 172.21.15.90:0/2217948461 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc62c1a8c60 con 0x7fc62c10f430 2024-04-14T15:21:40.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.044+0000 7fc6237fe700 1 -- 172.21.15.90:0/2217948461 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4029171821 0 0) 0x7fc618003aa0 con 0x7fc62c10b830 2024-04-14T15:21:40.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.044+0000 7fc6237fe700 1 -- 172.21.15.90:0/2217948461 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc62c1a9ea0 con 0x7fc62c10b830 2024-04-14T15:21:40.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.044+0000 7fc6237fe700 1 -- 172.21.15.90:0/2217948461 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fc61c001230 con 0x7fc62c10b830 2024-04-14T15:21:40.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.044+0000 7fc6237fe700 1 -- 172.21.15.90:0/2217948461 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1906045676 0 0) 0x7fc62c1a7a40 con 0x7fc62c108a30 2024-04-14T15:21:40.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.044+0000 7fc6237fe700 1 -- 172.21.15.90:0/2217948461 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc618003aa0 con 0x7fc62c108a30 2024-04-14T15:21:40.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.044+0000 7fc6237fe700 1 -- 172.21.15.90:0/2217948461 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fc628004470 con 0x7fc62c108a30 2024-04-14T15:21:40.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.044+0000 7fc6237fe700 1 -- 172.21.15.90:0/2217948461 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 858970008 0 0) 0x7fc62c1a8c60 con 0x7fc62c10f430 2024-04-14T15:21:40.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.044+0000 7fc6237fe700 1 -- 172.21.15.90:0/2217948461 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc62c1a7a40 con 0x7fc62c10f430 2024-04-14T15:21:40.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.044+0000 7fc6237fe700 1 -- 172.21.15.90:0/2217948461 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fc624001af0 con 0x7fc62c10f430 2024-04-14T15:21:40.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.044+0000 7fc6237fe700 1 -- 172.21.15.90:0/2217948461 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1361228290 0 0) 0x7fc62c1a9ea0 con 0x7fc62c10b830 2024-04-14T15:21:40.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.044+0000 7fc6237fe700 1 -- 172.21.15.90:0/2217948461 >> v1:172.21.15.90:6790/0 conn(0x7fc62c10f430 legacy=0x7fc62c1a6100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:40.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.044+0000 7fc6237fe700 1 -- 172.21.15.90:0/2217948461 >> v1:172.21.15.92:6789/0 conn(0x7fc62c108a30 legacy=0x7fc62c101870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:40.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.044+0000 7fc6237fe700 1 -- 172.21.15.90:0/2217948461 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc62c1ab0e0 con 0x7fc62c10b830 2024-04-14T15:21:40.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.044+0000 7fc6342a0700 1 -- 172.21.15.90:0/2217948461 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc62c1a8e90 con 0x7fc62c10b830 2024-04-14T15:21:40.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.044+0000 7fc6342a0700 10 client.37599.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:21:40.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.044+0000 7fc6342a0700 1 -- 172.21.15.90:0/2217948461 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc62c1a9350 con 0x7fc62c10b830 2024-04-14T15:21:40.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.044+0000 7fc6342a0700 1 librados: init done 2024-04-14T15:21:40.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.045+0000 7fc6237fe700 1 -- 172.21.15.90:0/2217948461 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fc61c003b20 con 0x7fc62c10b830 2024-04-14T15:21:40.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.045+0000 7fc6237fe700 1 -- 172.21.15.90:0/2217948461 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fc61c005cd0 con 0x7fc62c10b830 2024-04-14T15:21:40.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.045+0000 7fc6342a0700 1 -- 172.21.15.90:0/2217948461 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fc610005100 con 0x7fc62c10b830 2024-04-14T15:21:40.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.046+0000 7fc6237fe700 1 -- 172.21.15.90:0/2217948461 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fc61c003680 con 0x7fc62c10b830 2024-04-14T15:21:40.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.047+0000 7fc6237fe700 1 -- 172.21.15.90:0/2217948461 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(566..566 src has 467..566) ==== 8155+0+0 (unknown 4063064441 0 0) 0x7fc61c08aa20 con 0x7fc62c10b830 2024-04-14T15:21:40.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.047+0000 7fc6237fe700 10 client.37599.objecter ms_dispatch 0x7fc62c001550 osd_map(566..566 src has 467..566) 2024-04-14T15:21:40.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.047+0000 7fc6237fe700 3 client.37599.objecter handle_osd_map got epochs [566,566] > 0 2024-04-14T15:21:40.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.047+0000 7fc6237fe700 3 client.37599.objecter handle_osd_map decoding full epoch 566 2024-04-14T15:21:40.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.047+0000 7fc6237fe700 20 client.37599.objecter dump_active .. 0 homeless 2024-04-14T15:21:40.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.047+0000 7fc6237fe700 1 -- 172.21.15.90:0/2217948461 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v566) ==== 54+0+16 (unknown 652654156 0 2420000192) 0x7fc61c08b060 con 0x7fc62c10b830 2024-04-14T15:21:40.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.047+0000 7fc6237fe700 10 client.37599.objecter ms_handle_connect 0x7fc618071380 2024-04-14T15:21:40.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.047+0000 7fc6342a0700 10 client.37599.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:21:40.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.047+0000 7fc6342a0700 15 client.37599.objecter _session_command_op_assign -1 1 2024-04-14T15:21:40.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.047+0000 7fc6342a0700 20 client.37599.objecter _get_session s=0x7fc6100053e0 osd=0 3 2024-04-14T15:21:40.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.047+0000 7fc6342a0700 20 client.37599.objecter put_session s=0x7fc6100053e0 osd=0 3 2024-04-14T15:21:40.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.047+0000 7fc6342a0700 20 client.37599.objecter _get_session s=0x7fc6100053e0 osd=0 3 2024-04-14T15:21:40.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.047+0000 7fc6342a0700 15 client.37599.objecter _session_command_op_remove -1 1 2024-04-14T15:21:40.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.047+0000 7fc6342a0700 20 client.37599.objecter get_session s=0x7fc6100053e0 osd=0 3 2024-04-14T15:21:40.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.047+0000 7fc6342a0700 15 client.37599.objecter _session_command_op_assign 0 1 2024-04-14T15:21:40.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.047+0000 7fc6342a0700 20 client.37599.objecter put_session s=0x7fc6100053e0 osd=0 4 2024-04-14T15:21:40.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.047+0000 7fc6342a0700 10 client.37599.objecter _send_command 1 2024-04-14T15:21:40.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.047+0000 7fc6342a0700 1 -- 172.21.15.90:0/2217948461 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc610004b50 con 0x7fc610004730 2024-04-14T15:21:40.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.047+0000 7fc6237fe700 10 client.37599.objecter ms_handle_connect 0x7fc610004730 2024-04-14T15:21:40.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.048+0000 7fc6237fe700 1 -- 172.21.15.90:0/2217948461 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc610004b50 con 0x7fc610004730 2024-04-14T15:21:40.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.048+0000 7fc6237fe700 10 client.37599.objecter ms_dispatch 0x7fc62c001550 command_reply(tid 1: 0 ) 2024-04-14T15:21:40.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.048+0000 7fc6237fe700 10 client.37599.objecter _finish_command 1 = system:0 2024-04-14T15:21:40.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.048+0000 7fc6237fe700 20 client.37599.objecter put_session s=0x7fc6100053e0 osd=0 4 2024-04-14T15:21:40.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.048+0000 7fc6237fe700 15 client.37599.objecter _session_command_op_remove 0 1 2024-04-14T15:21:40.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.064+0000 7fc6342a0700 10 client.37599.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:21:40.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.064+0000 7fc6342a0700 15 client.37599.objecter _session_command_op_assign -1 2 2024-04-14T15:21:40.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.064+0000 7fc6342a0700 20 client.37599.objecter _get_session s=0x7fc6100053e0 osd=0 3 2024-04-14T15:21:40.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.064+0000 7fc6342a0700 20 client.37599.objecter put_session s=0x7fc6100053e0 osd=0 3 2024-04-14T15:21:40.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.064+0000 7fc6342a0700 20 client.37599.objecter _get_session s=0x7fc6100053e0 osd=0 3 2024-04-14T15:21:40.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.064+0000 7fc6342a0700 15 client.37599.objecter _session_command_op_remove -1 2 2024-04-14T15:21:40.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.064+0000 7fc6342a0700 20 client.37599.objecter get_session s=0x7fc6100053e0 osd=0 3 2024-04-14T15:21:40.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.064+0000 7fc6342a0700 15 client.37599.objecter _session_command_op_assign 0 2 2024-04-14T15:21:40.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.064+0000 7fc6342a0700 20 client.37599.objecter put_session s=0x7fc6100053e0 osd=0 4 2024-04-14T15:21:40.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.064+0000 7fc6342a0700 10 client.37599.objecter _send_command 2 2024-04-14T15:21:40.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.064+0000 7fc6342a0700 1 -- 172.21.15.90:0/2217948461 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fc61000be00 con 0x7fc610004730 2024-04-14T15:21:40.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.066+0000 7fc6237fe700 1 -- 172.21.15.90:0/2217948461 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc61000be00 con 0x7fc610004730 2024-04-14T15:21:40.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.066+0000 7fc6237fe700 10 client.37599.objecter ms_dispatch 0x7fc62c001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:21:40.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.066+0000 7fc6237fe700 10 client.37599.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:21:40.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.066+0000 7fc6237fe700 20 client.37599.objecter put_session s=0x7fc6100053e0 osd=0 4 2024-04-14T15:21:40.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.066+0000 7fc6237fe700 15 client.37599.objecter _session_command_op_remove 0 2 2024-04-14T15:21:40.067 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:21:40.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.066+0000 7fc6342a0700 10 client.37599.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:21:40.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.066+0000 7fc6342a0700 15 client.37599.objecter _session_command_op_assign -1 3 2024-04-14T15:21:40.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.066+0000 7fc6342a0700 20 client.37599.objecter _get_session s=0x7fc610004110 osd=1 3 2024-04-14T15:21:40.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.066+0000 7fc6342a0700 20 client.37599.objecter put_session s=0x7fc610004110 osd=1 3 2024-04-14T15:21:40.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.066+0000 7fc6342a0700 20 client.37599.objecter _get_session s=0x7fc610004110 osd=1 3 2024-04-14T15:21:40.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.066+0000 7fc6342a0700 15 client.37599.objecter _session_command_op_remove -1 3 2024-04-14T15:21:40.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.066+0000 7fc6342a0700 20 client.37599.objecter get_session s=0x7fc610004110 osd=1 3 2024-04-14T15:21:40.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.066+0000 7fc6342a0700 15 client.37599.objecter _session_command_op_assign 1 3 2024-04-14T15:21:40.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.066+0000 7fc6342a0700 20 client.37599.objecter put_session s=0x7fc610004110 osd=1 4 2024-04-14T15:21:40.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.066+0000 7fc6342a0700 10 client.37599.objecter _send_command 3 2024-04-14T15:21:40.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.066+0000 7fc6342a0700 1 -- 172.21.15.90:0/2217948461 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fc610002180 con 0x7fc610004280 2024-04-14T15:21:40.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.067+0000 7fc6237fe700 10 client.37599.objecter ms_handle_connect 0x7fc610004280 2024-04-14T15:21:40.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.067+0000 7fc6237fe700 1 -- 172.21.15.90:0/2217948461 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc610002180 con 0x7fc610004280 2024-04-14T15:21:40.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.067+0000 7fc6237fe700 10 client.37599.objecter ms_dispatch 0x7fc62c001550 command_reply(tid 3: 0 ) 2024-04-14T15:21:40.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.067+0000 7fc6237fe700 10 client.37599.objecter _finish_command 3 = system:0 2024-04-14T15:21:40.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.067+0000 7fc6237fe700 20 client.37599.objecter put_session s=0x7fc610004110 osd=1 4 2024-04-14T15:21:40.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.067+0000 7fc6237fe700 15 client.37599.objecter _session_command_op_remove 1 3 2024-04-14T15:21:40.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.083+0000 7fc6342a0700 10 client.37599.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:21:40.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.083+0000 7fc6342a0700 15 client.37599.objecter _session_command_op_assign -1 4 2024-04-14T15:21:40.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.083+0000 7fc6342a0700 20 client.37599.objecter _get_session s=0x7fc610004110 osd=1 3 2024-04-14T15:21:40.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.083+0000 7fc6342a0700 20 client.37599.objecter put_session s=0x7fc610004110 osd=1 3 2024-04-14T15:21:40.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.083+0000 7fc6342a0700 20 client.37599.objecter _get_session s=0x7fc610004110 osd=1 3 2024-04-14T15:21:40.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.083+0000 7fc6342a0700 15 client.37599.objecter _session_command_op_remove -1 4 2024-04-14T15:21:40.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.083+0000 7fc6342a0700 20 client.37599.objecter get_session s=0x7fc610004110 osd=1 3 2024-04-14T15:21:40.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.083+0000 7fc6342a0700 15 client.37599.objecter _session_command_op_assign 1 4 2024-04-14T15:21:40.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.083+0000 7fc6342a0700 20 client.37599.objecter put_session s=0x7fc610004110 osd=1 4 2024-04-14T15:21:40.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.083+0000 7fc6342a0700 10 client.37599.objecter _send_command 4 2024-04-14T15:21:40.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.083+0000 7fc6342a0700 1 -- 172.21.15.90:0/2217948461 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fc610001600 con 0x7fc610004280 2024-04-14T15:21:40.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.085+0000 7fc6237fe700 1 -- 172.21.15.90:0/2217948461 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc610001600 con 0x7fc610004280 2024-04-14T15:21:40.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.085+0000 7fc6237fe700 10 client.37599.objecter ms_dispatch 0x7fc62c001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:21:40.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.085+0000 7fc6237fe700 10 client.37599.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:21:40.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.085+0000 7fc6237fe700 20 client.37599.objecter put_session s=0x7fc610004110 osd=1 4 2024-04-14T15:21:40.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.085+0000 7fc6237fe700 15 client.37599.objecter _session_command_op_remove 1 4 2024-04-14T15:21:40.086 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:21:40.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.085+0000 7fc6342a0700 10 client.37599.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:21:40.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.085+0000 7fc6342a0700 15 client.37599.objecter _session_command_op_assign -1 5 2024-04-14T15:21:40.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.085+0000 7fc6342a0700 20 client.37599.objecter _get_session s=0x7fc61000c780 osd=2 3 2024-04-14T15:21:40.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.085+0000 7fc6342a0700 20 client.37599.objecter put_session s=0x7fc61000c780 osd=2 3 2024-04-14T15:21:40.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.085+0000 7fc6342a0700 20 client.37599.objecter _get_session s=0x7fc61000c780 osd=2 3 2024-04-14T15:21:40.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.085+0000 7fc6342a0700 15 client.37599.objecter _session_command_op_remove -1 5 2024-04-14T15:21:40.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.085+0000 7fc6342a0700 20 client.37599.objecter get_session s=0x7fc61000c780 osd=2 3 2024-04-14T15:21:40.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.085+0000 7fc6342a0700 15 client.37599.objecter _session_command_op_assign 2 5 2024-04-14T15:21:40.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.085+0000 7fc6342a0700 20 client.37599.objecter put_session s=0x7fc61000c780 osd=2 4 2024-04-14T15:21:40.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.085+0000 7fc6342a0700 10 client.37599.objecter _send_command 5 2024-04-14T15:21:40.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.085+0000 7fc6342a0700 1 -- 172.21.15.90:0/2217948461 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fc6100154f0 con 0x7fc6100119e0 2024-04-14T15:21:40.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.086+0000 7fc6237fe700 10 client.37599.objecter ms_handle_connect 0x7fc6100119e0 2024-04-14T15:21:40.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.086+0000 7fc6237fe700 1 -- 172.21.15.90:0/2217948461 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc6100154f0 con 0x7fc6100119e0 2024-04-14T15:21:40.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.086+0000 7fc6237fe700 10 client.37599.objecter ms_dispatch 0x7fc62c001550 command_reply(tid 5: 0 ) 2024-04-14T15:21:40.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.086+0000 7fc6237fe700 10 client.37599.objecter _finish_command 5 = system:0 2024-04-14T15:21:40.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.086+0000 7fc6237fe700 20 client.37599.objecter put_session s=0x7fc61000c780 osd=2 4 2024-04-14T15:21:40.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.086+0000 7fc6237fe700 15 client.37599.objecter _session_command_op_remove 2 5 2024-04-14T15:21:40.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.103+0000 7fc6342a0700 10 client.37599.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:21:40.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.103+0000 7fc6342a0700 15 client.37599.objecter _session_command_op_assign -1 6 2024-04-14T15:21:40.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.103+0000 7fc6342a0700 20 client.37599.objecter _get_session s=0x7fc61000c780 osd=2 3 2024-04-14T15:21:40.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.103+0000 7fc6342a0700 20 client.37599.objecter put_session s=0x7fc61000c780 osd=2 3 2024-04-14T15:21:40.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.103+0000 7fc6342a0700 20 client.37599.objecter _get_session s=0x7fc61000c780 osd=2 3 2024-04-14T15:21:40.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.103+0000 7fc6342a0700 15 client.37599.objecter _session_command_op_remove -1 6 2024-04-14T15:21:40.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.103+0000 7fc6342a0700 20 client.37599.objecter get_session s=0x7fc61000c780 osd=2 3 2024-04-14T15:21:40.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.103+0000 7fc6342a0700 15 client.37599.objecter _session_command_op_assign 2 6 2024-04-14T15:21:40.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.103+0000 7fc6342a0700 20 client.37599.objecter put_session s=0x7fc61000c780 osd=2 4 2024-04-14T15:21:40.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.103+0000 7fc6342a0700 10 client.37599.objecter _send_command 6 2024-04-14T15:21:40.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.103+0000 7fc6342a0700 1 -- 172.21.15.90:0/2217948461 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fc61000c2c0 con 0x7fc6100119e0 2024-04-14T15:21:40.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.104+0000 7fc6237fe700 1 -- 172.21.15.90:0/2217948461 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc61000c2c0 con 0x7fc6100119e0 2024-04-14T15:21:40.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.104+0000 7fc6237fe700 10 client.37599.objecter ms_dispatch 0x7fc62c001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:21:40.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.104+0000 7fc6237fe700 10 client.37599.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:21:40.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.104+0000 7fc6237fe700 20 client.37599.objecter put_session s=0x7fc61000c780 osd=2 4 2024-04-14T15:21:40.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.104+0000 7fc6237fe700 15 client.37599.objecter _session_command_op_remove 2 6 2024-04-14T15:21:40.105 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:21:40.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.104+0000 7fc6342a0700 10 client.37599.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:21:40.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.104+0000 7fc6342a0700 15 client.37599.objecter _session_command_op_assign -1 7 2024-04-14T15:21:40.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.104+0000 7fc6342a0700 20 client.37599.objecter _get_session s=0x7fc610011030 osd=3 3 2024-04-14T15:21:40.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.104+0000 7fc6342a0700 20 client.37599.objecter put_session s=0x7fc610011030 osd=3 3 2024-04-14T15:21:40.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.104+0000 7fc6342a0700 20 client.37599.objecter _get_session s=0x7fc610011030 osd=3 3 2024-04-14T15:21:40.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.104+0000 7fc6342a0700 15 client.37599.objecter _session_command_op_remove -1 7 2024-04-14T15:21:40.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.104+0000 7fc6342a0700 20 client.37599.objecter get_session s=0x7fc610011030 osd=3 3 2024-04-14T15:21:40.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.104+0000 7fc6342a0700 15 client.37599.objecter _session_command_op_assign 3 7 2024-04-14T15:21:40.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.104+0000 7fc6342a0700 20 client.37599.objecter put_session s=0x7fc610011030 osd=3 4 2024-04-14T15:21:40.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.104+0000 7fc6342a0700 10 client.37599.objecter _send_command 7 2024-04-14T15:21:40.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.104+0000 7fc6342a0700 1 -- 172.21.15.90:0/2217948461 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fc61001a330 con 0x7fc610016820 2024-04-14T15:21:40.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.105+0000 7fc6237fe700 10 client.37599.objecter ms_handle_connect 0x7fc610016820 2024-04-14T15:21:40.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.105+0000 7fc6237fe700 1 -- 172.21.15.90:0/2217948461 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc61001a330 con 0x7fc610016820 2024-04-14T15:21:40.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.105+0000 7fc6237fe700 10 client.37599.objecter ms_dispatch 0x7fc62c001550 command_reply(tid 7: 0 ) 2024-04-14T15:21:40.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.105+0000 7fc6237fe700 10 client.37599.objecter _finish_command 7 = system:0 2024-04-14T15:21:40.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.105+0000 7fc6237fe700 20 client.37599.objecter put_session s=0x7fc610011030 osd=3 4 2024-04-14T15:21:40.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.105+0000 7fc6237fe700 15 client.37599.objecter _session_command_op_remove 3 7 2024-04-14T15:21:40.107 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:21:40.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.122+0000 7fc6342a0700 10 client.37599.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:21:40.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.122+0000 7fc6342a0700 15 client.37599.objecter _session_command_op_assign -1 8 2024-04-14T15:21:40.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.122+0000 7fc6342a0700 20 client.37599.objecter _get_session s=0x7fc610011030 osd=3 3 2024-04-14T15:21:40.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.122+0000 7fc6342a0700 20 client.37599.objecter put_session s=0x7fc610011030 osd=3 3 2024-04-14T15:21:40.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.122+0000 7fc6342a0700 20 client.37599.objecter _get_session s=0x7fc610011030 osd=3 3 2024-04-14T15:21:40.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.122+0000 7fc6342a0700 15 client.37599.objecter _session_command_op_remove -1 8 2024-04-14T15:21:40.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.122+0000 7fc6342a0700 20 client.37599.objecter get_session s=0x7fc610011030 osd=3 3 2024-04-14T15:21:40.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.122+0000 7fc6342a0700 15 client.37599.objecter _session_command_op_assign 3 8 2024-04-14T15:21:40.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.122+0000 7fc6342a0700 20 client.37599.objecter put_session s=0x7fc610011030 osd=3 4 2024-04-14T15:21:40.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.122+0000 7fc6342a0700 10 client.37599.objecter _send_command 8 2024-04-14T15:21:40.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.122+0000 7fc6342a0700 1 -- 172.21.15.90:0/2217948461 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fc610010b10 con 0x7fc610016820 2024-04-14T15:21:40.124 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:21:40.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.123+0000 7fc6237fe700 1 -- 172.21.15.90:0/2217948461 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc610010b10 con 0x7fc610016820 2024-04-14T15:21:40.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.123+0000 7fc6237fe700 10 client.37599.objecter ms_dispatch 0x7fc62c001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:21:40.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.123+0000 7fc6237fe700 10 client.37599.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:21:40.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.123+0000 7fc6237fe700 20 client.37599.objecter put_session s=0x7fc610011030 osd=3 4 2024-04-14T15:21:40.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.123+0000 7fc6237fe700 15 client.37599.objecter _session_command_op_remove 3 8 2024-04-14T15:21:40.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.124+0000 7fc6342a0700 10 client.37599.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:21:40.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.124+0000 7fc6342a0700 15 client.37599.objecter _session_command_op_assign -1 9 2024-04-14T15:21:40.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.124+0000 7fc6342a0700 20 client.37599.objecter _get_session s=0x7fc610010d60 osd=4 3 2024-04-14T15:21:40.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.124+0000 7fc6342a0700 20 client.37599.objecter put_session s=0x7fc610010d60 osd=4 3 2024-04-14T15:21:40.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.124+0000 7fc6342a0700 20 client.37599.objecter _get_session s=0x7fc610010d60 osd=4 3 2024-04-14T15:21:40.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.124+0000 7fc6342a0700 15 client.37599.objecter _session_command_op_remove -1 9 2024-04-14T15:21:40.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.124+0000 7fc6342a0700 20 client.37599.objecter get_session s=0x7fc610010d60 osd=4 3 2024-04-14T15:21:40.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.124+0000 7fc6342a0700 15 client.37599.objecter _session_command_op_assign 4 9 2024-04-14T15:21:40.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.124+0000 7fc6342a0700 20 client.37599.objecter put_session s=0x7fc610010d60 osd=4 4 2024-04-14T15:21:40.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.124+0000 7fc6342a0700 10 client.37599.objecter _send_command 9 2024-04-14T15:21:40.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.124+0000 7fc6342a0700 1 -- 172.21.15.90:0/2217948461 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fc61001ec10 con 0x7fc610016380 2024-04-14T15:21:40.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.124+0000 7fc6237fe700 10 client.37599.objecter ms_handle_connect 0x7fc610016380 2024-04-14T15:21:40.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.125+0000 7fc6237fe700 1 -- 172.21.15.90:0/2217948461 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc61001ec10 con 0x7fc610016380 2024-04-14T15:21:40.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.125+0000 7fc6237fe700 10 client.37599.objecter ms_dispatch 0x7fc62c001550 command_reply(tid 9: 0 ) 2024-04-14T15:21:40.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.125+0000 7fc6237fe700 10 client.37599.objecter _finish_command 9 = system:0 2024-04-14T15:21:40.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.125+0000 7fc6237fe700 20 client.37599.objecter put_session s=0x7fc610010d60 osd=4 4 2024-04-14T15:21:40.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.125+0000 7fc6237fe700 15 client.37599.objecter _session_command_op_remove 4 9 2024-04-14T15:21:40.134 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:40.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:40.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:21:40.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.142+0000 7fc6342a0700 10 client.37599.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:21:40.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.142+0000 7fc6342a0700 15 client.37599.objecter _session_command_op_assign -1 10 2024-04-14T15:21:40.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.142+0000 7fc6342a0700 20 client.37599.objecter _get_session s=0x7fc610010d60 osd=4 3 2024-04-14T15:21:40.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.142+0000 7fc6342a0700 20 client.37599.objecter put_session s=0x7fc610010d60 osd=4 3 2024-04-14T15:21:40.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.142+0000 7fc6342a0700 20 client.37599.objecter _get_session s=0x7fc610010d60 osd=4 3 2024-04-14T15:21:40.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.142+0000 7fc6342a0700 15 client.37599.objecter _session_command_op_remove -1 10 2024-04-14T15:21:40.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.142+0000 7fc6342a0700 20 client.37599.objecter get_session s=0x7fc610010d60 osd=4 3 2024-04-14T15:21:40.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.142+0000 7fc6342a0700 15 client.37599.objecter _session_command_op_assign 4 10 2024-04-14T15:21:40.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.142+0000 7fc6342a0700 20 client.37599.objecter put_session s=0x7fc610010d60 osd=4 4 2024-04-14T15:21:40.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.142+0000 7fc6342a0700 10 client.37599.objecter _send_command 10 2024-04-14T15:21:40.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.142+0000 7fc6342a0700 1 -- 172.21.15.90:0/2217948461 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fc6100205c0 con 0x7fc610016380 2024-04-14T15:21:40.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.144+0000 7fc6237fe700 1 -- 172.21.15.90:0/2217948461 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc6100205c0 con 0x7fc610016380 2024-04-14T15:21:40.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.144+0000 7fc6237fe700 10 client.37599.objecter ms_dispatch 0x7fc62c001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:21:40.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.144+0000 7fc6237fe700 10 client.37599.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:21:40.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.145+0000 7fc6237fe700 20 client.37599.objecter put_session s=0x7fc610010d60 osd=4 4 2024-04-14T15:21:40.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.145+0000 7fc6237fe700 15 client.37599.objecter _session_command_op_remove 4 10 2024-04-14T15:21:40.147 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:21:40.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.145+0000 7fc6342a0700 10 client.37599.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:21:40.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.145+0000 7fc6342a0700 15 client.37599.objecter _session_command_op_assign -1 11 2024-04-14T15:21:40.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.145+0000 7fc6342a0700 20 client.37599.objecter _get_session s=0x7fc6100205c0 osd=5 3 2024-04-14T15:21:40.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.145+0000 7fc6342a0700 20 client.37599.objecter put_session s=0x7fc6100205c0 osd=5 3 2024-04-14T15:21:40.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.145+0000 7fc6342a0700 20 client.37599.objecter _get_session s=0x7fc6100205c0 osd=5 3 2024-04-14T15:21:40.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.145+0000 7fc6342a0700 15 client.37599.objecter _session_command_op_remove -1 11 2024-04-14T15:21:40.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.145+0000 7fc6342a0700 20 client.37599.objecter get_session s=0x7fc6100205c0 osd=5 3 2024-04-14T15:21:40.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.145+0000 7fc6342a0700 15 client.37599.objecter _session_command_op_assign 5 11 2024-04-14T15:21:40.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.145+0000 7fc6342a0700 20 client.37599.objecter put_session s=0x7fc6100205c0 osd=5 4 2024-04-14T15:21:40.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.145+0000 7fc6342a0700 10 client.37599.objecter _send_command 11 2024-04-14T15:21:40.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.145+0000 7fc6342a0700 1 -- 172.21.15.90:0/2217948461 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fc6100211b0 con 0x7fc610020ce0 2024-04-14T15:21:40.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.145+0000 7fc6237fe700 10 client.37599.objecter ms_handle_connect 0x7fc610020ce0 2024-04-14T15:21:40.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.146+0000 7fc6237fe700 1 -- 172.21.15.90:0/2217948461 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc6100211b0 con 0x7fc610020ce0 2024-04-14T15:21:40.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.146+0000 7fc6237fe700 10 client.37599.objecter ms_dispatch 0x7fc62c001550 command_reply(tid 11: 0 ) 2024-04-14T15:21:40.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.146+0000 7fc6237fe700 10 client.37599.objecter _finish_command 11 = system:0 2024-04-14T15:21:40.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.146+0000 7fc6237fe700 20 client.37599.objecter put_session s=0x7fc6100205c0 osd=5 4 2024-04-14T15:21:40.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.146+0000 7fc6237fe700 15 client.37599.objecter _session_command_op_remove 5 11 2024-04-14T15:21:40.149 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:39 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:21:40.149 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:39 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:21:40.149 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:39 smithi090 ceph-mon[41670]: osdmap e565: 8 total, 8 up, 7 in 2024-04-14T15:21:40.149 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:39 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:21:40.149 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:39 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:21:40.149 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:39 smithi090 ceph-mon[30399]: osdmap e565: 8 total, 8 up, 7 in 2024-04-14T15:21:40.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.164+0000 7fc6342a0700 10 client.37599.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:21:40.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.164+0000 7fc6342a0700 15 client.37599.objecter _session_command_op_assign -1 12 2024-04-14T15:21:40.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.164+0000 7fc6342a0700 20 client.37599.objecter _get_session s=0x7fc6100205c0 osd=5 3 2024-04-14T15:21:40.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.164+0000 7fc6342a0700 20 client.37599.objecter put_session s=0x7fc6100205c0 osd=5 3 2024-04-14T15:21:40.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.164+0000 7fc6342a0700 20 client.37599.objecter _get_session s=0x7fc6100205c0 osd=5 3 2024-04-14T15:21:40.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.164+0000 7fc6342a0700 15 client.37599.objecter _session_command_op_remove -1 12 2024-04-14T15:21:40.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.164+0000 7fc6342a0700 20 client.37599.objecter get_session s=0x7fc6100205c0 osd=5 3 2024-04-14T15:21:40.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.164+0000 7fc6342a0700 15 client.37599.objecter _session_command_op_assign 5 12 2024-04-14T15:21:40.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.164+0000 7fc6342a0700 20 client.37599.objecter put_session s=0x7fc6100205c0 osd=5 4 2024-04-14T15:21:40.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.164+0000 7fc6342a0700 10 client.37599.objecter _send_command 12 2024-04-14T15:21:40.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.164+0000 7fc6342a0700 1 -- 172.21.15.90:0/2217948461 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fc610021410 con 0x7fc610020ce0 2024-04-14T15:21:40.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.166+0000 7fc6237fe700 1 -- 172.21.15.90:0/2217948461 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc610021410 con 0x7fc610020ce0 2024-04-14T15:21:40.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.166+0000 7fc6237fe700 10 client.37599.objecter ms_dispatch 0x7fc62c001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:21:40.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.166+0000 7fc6237fe700 10 client.37599.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:21:40.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.166+0000 7fc6237fe700 20 client.37599.objecter put_session s=0x7fc6100205c0 osd=5 4 2024-04-14T15:21:40.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.166+0000 7fc6237fe700 15 client.37599.objecter _session_command_op_remove 5 12 2024-04-14T15:21:40.167 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:21:40.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.166+0000 7fc6342a0700 10 client.37599.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:21:40.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.166+0000 7fc6342a0700 15 client.37599.objecter _session_command_op_assign -1 13 2024-04-14T15:21:40.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.166+0000 7fc6342a0700 20 client.37599.objecter _get_session s=0x7fc610021210 osd=6 3 2024-04-14T15:21:40.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.166+0000 7fc6342a0700 20 client.37599.objecter put_session s=0x7fc610021210 osd=6 3 2024-04-14T15:21:40.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.166+0000 7fc6342a0700 20 client.37599.objecter _get_session s=0x7fc610021210 osd=6 3 2024-04-14T15:21:40.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.166+0000 7fc6342a0700 15 client.37599.objecter _session_command_op_remove -1 13 2024-04-14T15:21:40.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.166+0000 7fc6342a0700 20 client.37599.objecter get_session s=0x7fc610021210 osd=6 3 2024-04-14T15:21:40.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.166+0000 7fc6342a0700 15 client.37599.objecter _session_command_op_assign 6 13 2024-04-14T15:21:40.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.166+0000 7fc6342a0700 20 client.37599.objecter put_session s=0x7fc610021210 osd=6 4 2024-04-14T15:21:40.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.166+0000 7fc6342a0700 10 client.37599.objecter _send_command 13 2024-04-14T15:21:40.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.166+0000 7fc6342a0700 1 -- 172.21.15.90:0/2217948461 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fc6100277e0 con 0x7fc610023cd0 2024-04-14T15:21:40.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.167+0000 7fc6237fe700 10 client.37599.objecter ms_handle_connect 0x7fc610023cd0 2024-04-14T15:21:40.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.167+0000 7fc6237fe700 1 -- 172.21.15.90:0/2217948461 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc6100277e0 con 0x7fc610023cd0 2024-04-14T15:21:40.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.167+0000 7fc6237fe700 10 client.37599.objecter ms_dispatch 0x7fc62c001550 command_reply(tid 13: 0 ) 2024-04-14T15:21:40.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.167+0000 7fc6237fe700 10 client.37599.objecter _finish_command 13 = system:0 2024-04-14T15:21:40.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.167+0000 7fc6237fe700 20 client.37599.objecter put_session s=0x7fc610021210 osd=6 4 2024-04-14T15:21:40.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.167+0000 7fc6237fe700 15 client.37599.objecter _session_command_op_remove 6 13 2024-04-14T15:21:40.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.184+0000 7fc6342a0700 10 client.37599.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:21:40.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.184+0000 7fc6342a0700 15 client.37599.objecter _session_command_op_assign -1 14 2024-04-14T15:21:40.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.184+0000 7fc6342a0700 20 client.37599.objecter _get_session s=0x7fc610021210 osd=6 3 2024-04-14T15:21:40.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.184+0000 7fc6342a0700 20 client.37599.objecter put_session s=0x7fc610021210 osd=6 3 2024-04-14T15:21:40.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.184+0000 7fc6342a0700 20 client.37599.objecter _get_session s=0x7fc610021210 osd=6 3 2024-04-14T15:21:40.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.184+0000 7fc6342a0700 15 client.37599.objecter _session_command_op_remove -1 14 2024-04-14T15:21:40.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.184+0000 7fc6342a0700 20 client.37599.objecter get_session s=0x7fc610021210 osd=6 3 2024-04-14T15:21:40.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.184+0000 7fc6342a0700 15 client.37599.objecter _session_command_op_assign 6 14 2024-04-14T15:21:40.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.184+0000 7fc6342a0700 20 client.37599.objecter put_session s=0x7fc610021210 osd=6 4 2024-04-14T15:21:40.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.184+0000 7fc6342a0700 10 client.37599.objecter _send_command 14 2024-04-14T15:21:40.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.184+0000 7fc6342a0700 1 -- 172.21.15.90:0/2217948461 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fc610010b70 con 0x7fc610023cd0 2024-04-14T15:21:40.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.186+0000 7fc6237fe700 1 -- 172.21.15.90:0/2217948461 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc610010b70 con 0x7fc610023cd0 2024-04-14T15:21:40.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.186+0000 7fc6237fe700 10 client.37599.objecter ms_dispatch 0x7fc62c001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:21:40.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.186+0000 7fc6237fe700 10 client.37599.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:21:40.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.186+0000 7fc6237fe700 20 client.37599.objecter put_session s=0x7fc610021210 osd=6 4 2024-04-14T15:21:40.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.186+0000 7fc6237fe700 15 client.37599.objecter _session_command_op_remove 6 14 2024-04-14T15:21:40.187 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:21:40.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.186+0000 7fc6342a0700 10 client.37599.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:21:40.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.186+0000 7fc6342a0700 15 client.37599.objecter _session_command_op_assign -1 15 2024-04-14T15:21:40.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.186+0000 7fc6342a0700 20 client.37599.objecter _get_session s=0x7fc6100277e0 osd=7 3 2024-04-14T15:21:40.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.186+0000 7fc6342a0700 20 client.37599.objecter put_session s=0x7fc6100277e0 osd=7 3 2024-04-14T15:21:40.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.186+0000 7fc6342a0700 20 client.37599.objecter _get_session s=0x7fc6100277e0 osd=7 3 2024-04-14T15:21:40.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.186+0000 7fc6342a0700 15 client.37599.objecter _session_command_op_remove -1 15 2024-04-14T15:21:40.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.186+0000 7fc6342a0700 20 client.37599.objecter get_session s=0x7fc6100277e0 osd=7 3 2024-04-14T15:21:40.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.186+0000 7fc6342a0700 15 client.37599.objecter _session_command_op_assign 7 15 2024-04-14T15:21:40.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.186+0000 7fc6342a0700 20 client.37599.objecter put_session s=0x7fc6100277e0 osd=7 4 2024-04-14T15:21:40.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.186+0000 7fc6342a0700 10 client.37599.objecter _send_command 15 2024-04-14T15:21:40.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.186+0000 7fc6342a0700 1 -- 172.21.15.90:0/2217948461 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fc61002bb20 con 0x7fc610027e60 2024-04-14T15:21:40.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.186+0000 7fc6237fe700 10 client.37599.objecter ms_handle_connect 0x7fc610027e60 2024-04-14T15:21:40.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.187+0000 7fc6237fe700 1 -- 172.21.15.90:0/2217948461 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc61002bb20 con 0x7fc610027e60 2024-04-14T15:21:40.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.187+0000 7fc6237fe700 10 client.37599.objecter ms_dispatch 0x7fc62c001550 command_reply(tid 15: 0 ) 2024-04-14T15:21:40.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.187+0000 7fc6237fe700 10 client.37599.objecter _finish_command 15 = system:0 2024-04-14T15:21:40.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.187+0000 7fc6237fe700 20 client.37599.objecter put_session s=0x7fc6100277e0 osd=7 4 2024-04-14T15:21:40.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.187+0000 7fc6237fe700 15 client.37599.objecter _session_command_op_remove 7 15 2024-04-14T15:21:40.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.203+0000 7fc6342a0700 10 client.37599.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:21:40.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.203+0000 7fc6342a0700 15 client.37599.objecter _session_command_op_assign -1 16 2024-04-14T15:21:40.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.203+0000 7fc6342a0700 20 client.37599.objecter _get_session s=0x7fc6100277e0 osd=7 3 2024-04-14T15:21:40.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.203+0000 7fc6342a0700 20 client.37599.objecter put_session s=0x7fc6100277e0 osd=7 3 2024-04-14T15:21:40.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.203+0000 7fc6342a0700 20 client.37599.objecter _get_session s=0x7fc6100277e0 osd=7 3 2024-04-14T15:21:40.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.203+0000 7fc6342a0700 15 client.37599.objecter _session_command_op_remove -1 16 2024-04-14T15:21:40.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.203+0000 7fc6342a0700 20 client.37599.objecter get_session s=0x7fc6100277e0 osd=7 3 2024-04-14T15:21:40.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.203+0000 7fc6342a0700 15 client.37599.objecter _session_command_op_assign 7 16 2024-04-14T15:21:40.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.203+0000 7fc6342a0700 20 client.37599.objecter put_session s=0x7fc6100277e0 osd=7 4 2024-04-14T15:21:40.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.203+0000 7fc6342a0700 10 client.37599.objecter _send_command 16 2024-04-14T15:21:40.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.203+0000 7fc6342a0700 1 -- 172.21.15.90:0/2217948461 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fc61002d4d0 con 0x7fc610027e60 2024-04-14T15:21:40.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.205+0000 7fc6237fe700 1 -- 172.21.15.90:0/2217948461 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc61002d4d0 con 0x7fc610027e60 2024-04-14T15:21:40.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.205+0000 7fc6237fe700 10 client.37599.objecter ms_dispatch 0x7fc62c001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:21:40.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.205+0000 7fc6237fe700 10 client.37599.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:21:40.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.205+0000 7fc6237fe700 20 client.37599.objecter put_session s=0x7fc6100277e0 osd=7 4 2024-04-14T15:21:40.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.205+0000 7fc6237fe700 15 client.37599.objecter _session_command_op_remove 7 16 2024-04-14T15:21:40.206 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:21:40.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.205+0000 7fc6342a0700 10 librados: watch_flush enter 2024-04-14T15:21:40.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.205+0000 7fc6342a0700 10 librados: watch_flush exit 2024-04-14T15:21:40.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.205+0000 7fc6342a0700 10 client.37599.objecter close_session for osd.0 2024-04-14T15:21:40.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.205+0000 7fc6342a0700 1 -- 172.21.15.90:0/2217948461 >> v1:172.21.15.90:6801/3761806885 conn(0x7fc610004730 legacy=0x7fc610002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:40.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.205+0000 7fc6342a0700 20 client.37599.objecter put_session s=0x7fc6100053e0 osd=0 1 2024-04-14T15:21:40.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.205+0000 7fc6342a0700 10 client.37599.objecter close_session for osd.1 2024-04-14T15:21:40.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.205+0000 7fc6342a0700 1 -- 172.21.15.90:0/2217948461 >> v1:172.21.15.90:6805/4170361699 conn(0x7fc610004280 legacy=0x7fc61000f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:40.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.205+0000 7fc6342a0700 20 client.37599.objecter put_session s=0x7fc610004110 osd=1 1 2024-04-14T15:21:40.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.205+0000 7fc6342a0700 10 client.37599.objecter close_session for osd.2 2024-04-14T15:21:40.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.205+0000 7fc6342a0700 1 -- 172.21.15.90:0/2217948461 >> v1:172.21.15.90:6809/1686184251 conn(0x7fc6100119e0 legacy=0x7fc610013dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:40.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.205+0000 7fc63283d700 1 -- 172.21.15.90:0/2217948461 reap_dead start 2024-04-14T15:21:40.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.206+0000 7fc6342a0700 20 client.37599.objecter put_session s=0x7fc61000c780 osd=2 1 2024-04-14T15:21:40.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.206+0000 7fc6342a0700 10 client.37599.objecter close_session for osd.3 2024-04-14T15:21:40.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.206+0000 7fc6342a0700 1 -- 172.21.15.90:0/2217948461 >> v1:172.21.15.90:6813/2270166774 conn(0x7fc610016820 legacy=0x7fc610018c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:40.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.206+0000 7fc6342a0700 20 client.37599.objecter put_session s=0x7fc610011030 osd=3 1 2024-04-14T15:21:40.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.206+0000 7fc6342a0700 10 client.37599.objecter close_session for osd.4 2024-04-14T15:21:40.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.206+0000 7fc6342a0700 1 -- 172.21.15.90:0/2217948461 >> v1:172.21.15.92:6800/3887549733 conn(0x7fc610016380 legacy=0x7fc61001d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:40.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.206+0000 7fc6342a0700 20 client.37599.objecter put_session s=0x7fc610010d60 osd=4 1 2024-04-14T15:21:40.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.206+0000 7fc6342a0700 10 client.37599.objecter close_session for osd.5 2024-04-14T15:21:40.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.206+0000 7fc6342a0700 1 -- 172.21.15.90:0/2217948461 >> v1:172.21.15.92:6804/3244561440 conn(0x7fc610020ce0 legacy=0x7fc61001edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:40.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.206+0000 7fc6342a0700 20 client.37599.objecter put_session s=0x7fc6100205c0 osd=5 1 2024-04-14T15:21:40.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.206+0000 7fc6342a0700 10 client.37599.objecter close_session for osd.6 2024-04-14T15:21:40.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.206+0000 7fc6342a0700 1 -- 172.21.15.90:0/2217948461 >> v1:172.21.15.92:6808/636125963 conn(0x7fc610023cd0 legacy=0x7fc6100260c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:40.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.206+0000 7fc6342a0700 20 client.37599.objecter put_session s=0x7fc610021210 osd=6 1 2024-04-14T15:21:40.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.206+0000 7fc6342a0700 10 client.37599.objecter close_session for osd.7 2024-04-14T15:21:40.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.206+0000 7fc6342a0700 1 -- 172.21.15.90:0/2217948461 >> v1:172.21.15.92:6812/1773706716 conn(0x7fc610027e60 legacy=0x7fc61002a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:40.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.206+0000 7fc6342a0700 20 client.37599.objecter put_session s=0x7fc6100277e0 osd=7 1 2024-04-14T15:21:40.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.206+0000 7fc6342a0700 20 client.37599.objecter shutdown clearing up homeless session... 2024-04-14T15:21:40.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.206+0000 7fc6342a0700 10 client.37599.objecter successfully canceled tick 2024-04-14T15:21:40.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.206+0000 7fc63283d700 1 -- 172.21.15.90:0/2217948461 reap_dead start 2024-04-14T15:21:40.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.206+0000 7fc6342a0700 1 -- 172.21.15.90:0/2217948461 >> v1:172.21.15.90:6800/420935427 conn(0x7fc618071380 legacy=0x7fc618073840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:40.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.206+0000 7fc6342a0700 1 -- 172.21.15.90:0/2217948461 >> v1:172.21.15.90:6789/0 conn(0x7fc62c10b830 legacy=0x7fc62c1a28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:40.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.206+0000 7fc6342a0700 1 -- 172.21.15.90:0/2217948461 shutdown_connections 2024-04-14T15:21:40.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.206+0000 7fc6342a0700 1 -- 172.21.15.90:0/2217948461 >> 172.21.15.90:0/2217948461 conn(0x7fc62c076ab0 msgr2=0x7fc62c112b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:21:40.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.206+0000 7fc6342a0700 1 -- 172.21.15.90:0/2217948461 shutdown_connections 2024-04-14T15:21:40.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.206+0000 7fc6342a0700 1 -- 172.21.15.90:0/2217948461 wait complete. 2024-04-14T15:21:40.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:40.206+0000 7fc6342a0700 1 librados: shutdown 2024-04-14T15:21:40.210 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:21:40.210 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:21:40.210 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:21:40.210 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:21:40.210 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:21:40.210 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:21:40.210 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:21:40.211 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:21:40.236 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:21:40.259 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:40.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:40.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:21:40.361 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:21:40.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:39 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:21:40.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:39 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:21:40.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:39 smithi092 ceph-mon[33966]: osdmap e565: 8 total, 8 up, 7 in 2024-04-14T15:21:40.384 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:40.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:40.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:21:40.486 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:21:40.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:40.510 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:40.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:21:40.610 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:21:40.638 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:40.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:40.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:21:40.740 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:21:40.764 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:40.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:40.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:21:40.865 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:21:40.893 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:40.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:40.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:21:40.994 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:21:41.025 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:41.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:41.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:21:41.126 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:21:41.151 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:41.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:41.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:21:41.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:40 smithi090 ceph-mon[30399]: pgmap v827: 216 pgs: 216 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:21:41.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:40 smithi090 ceph-mon[30399]: osdmap e566: 8 total, 8 up, 7 in 2024-04-14T15:21:41.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:40 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:21:41.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:40 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:21:41.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:40 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2217948461' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:21:41.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:40 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:21:41.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:40 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:21:41.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:40 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:21:41.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:40 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:21:41.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:40 smithi090 ceph-mon[41670]: pgmap v827: 216 pgs: 216 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:21:41.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:40 smithi090 ceph-mon[41670]: osdmap e566: 8 total, 8 up, 7 in 2024-04-14T15:21:41.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:40 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:21:41.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:40 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:21:41.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:40 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2217948461' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:21:41.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:40 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:21:41.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:40 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:21:41.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:40 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:21:41.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:40 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:21:41.253 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:21:41.276 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:41.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:41.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:21:41.358 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-14T15:21:41.358 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-14T15:21:41.358 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-14T15:21:41.358 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-14T15:21:41.358 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-14T15:21:41.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:40 smithi092 ceph-mon[33966]: pgmap v827: 216 pgs: 216 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:21:41.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:40 smithi092 ceph-mon[33966]: osdmap e566: 8 total, 8 up, 7 in 2024-04-14T15:21:41.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:40 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:21:41.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:40 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:21:41.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:40 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2217948461' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:21:41.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:40 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:21:41.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:40 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:21:41.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:40 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:21:41.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:40 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:21:41.378 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:21:41.403 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:41.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:41.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:21:41.504 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:21:41.508 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:21:41.529 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:41.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:41.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:21:41.630 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:21:41.654 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:41.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:41.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:21:41.755 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:41.782 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:41.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:41.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:41.883 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:21:41.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:41.907 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:41.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:21:42.008 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:21:42.035 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:42.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:42.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:21:42.137 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:21:42.163 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:42.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:42.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:21:42.264 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:42.291 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:42.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:42.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:42.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:42 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-96","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:21:42.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:42 smithi092 ceph-mon[33966]: osdmap e567: 8 total, 8 up, 7 in 2024-04-14T15:21:42.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:42 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-14T15:21:42.393 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:42.394 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:42 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-96","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:21:42.394 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:42 smithi090 ceph-mon[41670]: osdmap e567: 8 total, 8 up, 7 in 2024-04-14T15:21:42.394 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:42 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-14T15:21:42.394 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:42 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-96","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:21:42.394 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:42 smithi090 ceph-mon[30399]: osdmap e567: 8 total, 8 up, 7 in 2024-04-14T15:21:42.394 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:42 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-14T15:21:42.421 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:42.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:42.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:42.523 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:21:42.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:42.550 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:42.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:21:42.651 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:21:42.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:42.676 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:42.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:21:42.777 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:21:42.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:42.802 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:42.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:21:42.903 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:21:42.927 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:42.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:42.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:21:43.028 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-14T15:21:43.028 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:21:43.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:43.052 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:43.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:21:43.153 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:21:43.181 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:43.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:43.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:21:43.282 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:21:43.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:43.306 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:43.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:21:43.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:43 smithi092 ceph-mon[33966]: pgmap v830: 248 pgs: 32 unknown, 216 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:21:43.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:43 smithi092 ceph-mon[33966]: osdmap e568: 8 total, 8 up, 7 in 2024-04-14T15:21:43.407 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:21:43.435 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:43.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:43.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:21:43.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:43 smithi090 ceph-mon[41670]: pgmap v830: 248 pgs: 32 unknown, 216 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:21:43.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:43 smithi090 ceph-mon[41670]: osdmap e568: 8 total, 8 up, 7 in 2024-04-14T15:21:43.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:43 smithi090 ceph-mon[30399]: pgmap v830: 248 pgs: 32 unknown, 216 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:21:43.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:43 smithi090 ceph-mon[30399]: osdmap e568: 8 total, 8 up, 7 in 2024-04-14T15:21:43.537 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:21:43.564 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:43.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:43.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:21:43.666 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:43.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:43.693 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:43.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:43.794 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:21:43.820 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:43.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:43.821 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:21:43.922 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:21:43.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:43.948 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:43.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:21:44.049 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:21:44.075 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:44.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:44.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:21:44.177 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:21:44.202 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:44.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:44.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:21:44.304 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:21:44.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:44.330 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:44.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:21:44.372 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:44 smithi090 ceph-mon[41670]: pgmap v832: 248 pgs: 32 unknown, 216 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 245 B/s wr, 1 op/s 2024-04-14T15:21:44.372 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:21:43 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:21:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:21:44.372 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:44 smithi090 ceph-mon[30399]: pgmap v832: 248 pgs: 32 unknown, 216 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 245 B/s wr, 1 op/s 2024-04-14T15:21:44.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:44 smithi092 ceph-mon[33966]: pgmap v832: 248 pgs: 32 unknown, 216 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 245 B/s wr, 1 op/s 2024-04-14T15:21:44.433 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:21:44.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:44.458 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:44.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:21:44.559 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:44.585 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:44.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:44.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:44.687 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:21:44.712 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:44.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:44.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:21:44.814 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:21:44.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:44.669+0000 7fa90380f700 1 -- 172.21.15.90:0/3446239950 >> v1:172.21.15.90:6789/0 conn(0x7fa8fc103ac0 legacy=0x7fa8fc103ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:44.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:44.669+0000 7fa90380f700 1 -- 172.21.15.90:0/3446239950 shutdown_connections 2024-04-14T15:21:44.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:44.669+0000 7fa90380f700 1 -- 172.21.15.90:0/3446239950 >> 172.21.15.90:0/3446239950 conn(0x7fa8fc0fd290 msgr2=0x7fa8fc0ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:21:44.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:44.669+0000 7fa90380f700 1 -- 172.21.15.90:0/3446239950 shutdown_connections 2024-04-14T15:21:44.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:44.669+0000 7fa90380f700 1 -- 172.21.15.90:0/3446239950 wait complete. 2024-04-14T15:21:44.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:44.670+0000 7fa90380f700 1 librados: starting msgr at 2024-04-14T15:21:44.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:44.670+0000 7fa90380f700 1 librados: starting objecter 2024-04-14T15:21:44.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:44.670+0000 7fa90380f700 1 Processor -- start 2024-04-14T15:21:44.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:44.670+0000 7fa90380f700 1 -- start start 2024-04-14T15:21:44.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:44.670+0000 7fa90380f700 1 librados: setting wanted keys 2024-04-14T15:21:44.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:44.670+0000 7fa90380f700 1 librados: calling monclient init 2024-04-14T15:21:44.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:44.670+0000 7fa90380f700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa8fc1032f0 con 0x7fa8fc10b8d0 2024-04-14T15:21:44.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:44.670+0000 7fa90380f700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa8fc103500 con 0x7fa8fc103ac0 2024-04-14T15:21:44.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:44.670+0000 7fa90380f700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa8fc1ac2c0 con 0x7fa8fc10f3e0 2024-04-14T15:21:44.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:44.670+0000 7fa901dac700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7fa8fc10f3e0 0x7fa8fc1a8b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:53380/0 (socket says 172.21.15.90:53380) 2024-04-14T15:21:44.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:44.670+0000 7fa900daa700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fa8fc10b8d0 0x7fa8fc101e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:47788/0 (socket says 172.21.15.90:47788) 2024-04-14T15:21:44.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:44.670+0000 7fa901dac700 1 -- 172.21.15.90:0/4062140390 learned_addr learned my addr 172.21.15.90:0/4062140390 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:21:44.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:44.670+0000 7fa8fa7fc700 10 client.?.objecter ms_handle_connect 0x7fa8fc10b8d0 2024-04-14T15:21:44.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:44.670+0000 7fa8fa7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:21:44.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:44.670+0000 7fa8fa7fc700 10 client.?.objecter ms_handle_connect 0x7fa8fc10f3e0 2024-04-14T15:21:44.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:44.670+0000 7fa8fa7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:21:44.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:44.670+0000 7fa8fa7fc700 10 client.?.objecter ms_handle_connect 0x7fa8fc103ac0 2024-04-14T15:21:44.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:44.670+0000 7fa8fa7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:21:44.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:44.670+0000 7fa8fa7fc700 1 -- 172.21.15.90:0/4062140390 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 276216308 0 0) 0x7fa8fc1ac2c0 con 0x7fa8fc10f3e0 2024-04-14T15:21:44.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:44.670+0000 7fa8fa7fc700 1 -- 172.21.15.90:0/4062140390 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa8e40035b0 con 0x7fa8fc10f3e0 2024-04-14T15:21:44.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:44.670+0000 7fa8fa7fc700 1 -- 172.21.15.90:0/4062140390 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 446702427 0 0) 0x7fa8fc1032f0 con 0x7fa8fc10b8d0 2024-04-14T15:21:44.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:44.670+0000 7fa8fa7fc700 1 -- 172.21.15.90:0/4062140390 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa8fc1ac2c0 con 0x7fa8fc10b8d0 2024-04-14T15:21:44.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:44.671+0000 7fa8fa7fc700 1 -- 172.21.15.90:0/4062140390 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2023386264 0 0) 0x7fa8fc103500 con 0x7fa8fc103ac0 2024-04-14T15:21:44.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:44.671+0000 7fa8fa7fc700 1 -- 172.21.15.90:0/4062140390 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa8fc1032f0 con 0x7fa8fc103ac0 2024-04-14T15:21:44.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:44.671+0000 7fa8fa7fc700 1 -- 172.21.15.90:0/4062140390 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3278334222 0 0) 0x7fa8e40035b0 con 0x7fa8fc10f3e0 2024-04-14T15:21:44.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:44.671+0000 7fa8fa7fc700 1 -- 172.21.15.90:0/4062140390 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa8fc103500 con 0x7fa8fc10f3e0 2024-04-14T15:21:44.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:44.671+0000 7fa8fa7fc700 1 -- 172.21.15.90:0/4062140390 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3529752423 0 0) 0x7fa8fc1ac2c0 con 0x7fa8fc10b8d0 2024-04-14T15:21:44.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:44.671+0000 7fa8fa7fc700 1 -- 172.21.15.90:0/4062140390 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa8e40035b0 con 0x7fa8fc10b8d0 2024-04-14T15:21:44.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:44.671+0000 7fa8fa7fc700 1 -- 172.21.15.90:0/4062140390 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fa8ec001bb0 con 0x7fa8fc10f3e0 2024-04-14T15:21:44.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:44.671+0000 7fa8fa7fc700 1 -- 172.21.15.90:0/4062140390 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fa8f00018b0 con 0x7fa8fc10b8d0 2024-04-14T15:21:44.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:44.671+0000 7fa8fa7fc700 1 -- 172.21.15.90:0/4062140390 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2406241348 0 0) 0x7fa8e40035b0 con 0x7fa8fc10b8d0 2024-04-14T15:21:44.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:44.671+0000 7fa8fa7fc700 1 -- 172.21.15.90:0/4062140390 >> v1:172.21.15.90:6790/0 conn(0x7fa8fc10f3e0 legacy=0x7fa8fc1a8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:44.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:44.671+0000 7fa8fa7fc700 1 -- 172.21.15.90:0/4062140390 >> v1:172.21.15.92:6789/0 conn(0x7fa8fc103ac0 legacy=0x7fa8fc101720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:44.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:44.671+0000 7fa8fa7fc700 1 -- 172.21.15.90:0/4062140390 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8fc1ad4a0 con 0x7fa8fc10b8d0 2024-04-14T15:21:44.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:44.671+0000 7fa90380f700 1 -- 172.21.15.90:0/4062140390 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa8fc1ac4f0 con 0x7fa8fc10b8d0 2024-04-14T15:21:44.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:44.671+0000 7fa90380f700 10 client.37605.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:21:44.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:44.671+0000 7fa90380f700 1 -- 172.21.15.90:0/4062140390 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa8fc1ac9b0 con 0x7fa8fc10b8d0 2024-04-14T15:21:44.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:44.671+0000 7fa90380f700 1 librados: init done 2024-04-14T15:21:44.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:44.671+0000 7fa8fa7fc700 1 -- 172.21.15.90:0/4062140390 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fa8f0003b20 con 0x7fa8fc10b8d0 2024-04-14T15:21:44.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:44.671+0000 7fa8fa7fc700 1 -- 172.21.15.90:0/4062140390 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fa8f0005c00 con 0x7fa8fc10b8d0 2024-04-14T15:21:44.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:44.672+0000 7fa90380f700 1 -- 172.21.15.90:0/4062140390 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8e8005120 con 0x7fa8fc10b8d0 2024-04-14T15:21:44.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:44.672+0000 7fa8fa7fc700 1 -- 172.21.15.90:0/4062140390 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fa8f0003680 con 0x7fa8fc10b8d0 2024-04-14T15:21:44.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:44.673+0000 7fa8fa7fc700 1 -- 172.21.15.90:0/4062140390 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(568..568 src has 467..568) ==== 8168+0+0 (unknown 2087003471 0 0) 0x7fa8f008ab20 con 0x7fa8fc10b8d0 2024-04-14T15:21:44.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:44.673+0000 7fa8fa7fc700 10 client.37605.objecter ms_dispatch 0x7fa8fc001550 osd_map(568..568 src has 467..568) 2024-04-14T15:21:44.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:44.673+0000 7fa8fa7fc700 3 client.37605.objecter handle_osd_map got epochs [568,568] > 0 2024-04-14T15:21:44.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:44.673+0000 7fa8fa7fc700 3 client.37605.objecter handle_osd_map decoding full epoch 568 2024-04-14T15:21:44.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:44.673+0000 7fa8fa7fc700 20 client.37605.objecter dump_active .. 0 homeless 2024-04-14T15:21:44.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:44.673+0000 7fa8fa7fc700 10 client.37605.objecter ms_handle_connect 0x7fa8e40712f0 2024-04-14T15:21:44.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:44.677+0000 7fa8fa7fc700 1 -- 172.21.15.90:0/4062140390 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fa8f0057530 con 0x7fa8fc10b8d0 2024-04-14T15:21:44.837 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:44.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:44.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:21:44.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:44.888+0000 7fa90380f700 1 -- 172.21.15.90:0/4062140390 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7fa8e8005410 con 0x7fa8fc10b8d0 2024-04-14T15:21:44.942 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:21:44.969 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:44.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:44.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:21:45.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:45.057+0000 7fa8fa7fc700 1 -- 172.21.15.90:0/4062140390 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v569) ==== 97+0+0 (unknown 553416395 0 0) 0x7fa8f005b1e0 con 0x7fa8fc10b8d0 2024-04-14T15:21:45.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:45.060+0000 7fa90380f700 10 librados: watch_flush enter 2024-04-14T15:21:45.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:45.060+0000 7fa90380f700 10 librados: watch_flush exit 2024-04-14T15:21:45.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:45.060+0000 7fa90380f700 20 client.37605.objecter shutdown clearing up homeless session... 2024-04-14T15:21:45.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:45.060+0000 7fa90380f700 10 client.37605.objecter successfully canceled tick 2024-04-14T15:21:45.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:45.060+0000 7fa90380f700 1 -- 172.21.15.90:0/4062140390 >> v1:172.21.15.90:6800/420935427 conn(0x7fa8e40712f0 legacy=0x7fa8e40737b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:45.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:45.060+0000 7fa90380f700 1 -- 172.21.15.90:0/4062140390 >> v1:172.21.15.90:6789/0 conn(0x7fa8fc10b8d0 legacy=0x7fa8fc101e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:45.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:45.063+0000 7fa90380f700 1 -- 172.21.15.90:0/4062140390 shutdown_connections 2024-04-14T15:21:45.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:45.063+0000 7fa90380f700 1 -- 172.21.15.90:0/4062140390 >> 172.21.15.90:0/4062140390 conn(0x7fa8fc0fd290 msgr2=0x7fa8fc0ff680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:21:45.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:45.063+0000 7fa90380f700 1 -- 172.21.15.90:0/4062140390 shutdown_connections 2024-04-14T15:21:45.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:45.063+0000 7fa90380f700 1 -- 172.21.15.90:0/4062140390 wait complete. 2024-04-14T15:21:45.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:45.063+0000 7fa90380f700 1 librados: shutdown 2024-04-14T15:21:45.065 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is set 2024-04-14T15:21:45.070 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:21:45.096 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:45.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:45.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:21:45.197 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:21:45.220 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:45.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:45.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:21:45.322 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:21:45.345 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:45.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:45.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:21:45.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:45 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/4062140390' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:21:45.372 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:21:45 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:21:45.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:45 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/4062140390' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:21:45.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:45 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/4062140390' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:21:45.447 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:21:45.471 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:45.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:45.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:21:45.573 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:21:45.596 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:45.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:45.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:21:45.652 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-14T15:21:45.698 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:21:45.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:45.726 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:45.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:21:45.827 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:21:45.850 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:45.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:45.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:21:45.951 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:21:45.976 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:45.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:45.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:21:46.078 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:21:46.103 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:46.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:46.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:21:46.204 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:21:46.232 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:46.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:46.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:21:46.334 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:21:46.368 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:46 smithi090 ceph-mon[41670]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:21:46.368 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:46 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/4062140390' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-14T15:21:46.368 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:46 smithi090 ceph-mon[41670]: osdmap e569: 8 total, 8 up, 7 in 2024-04-14T15:21:46.368 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:46 smithi090 ceph-mon[41670]: pgmap v834: 248 pgs: 248 active+clean; 8.4 MiB data, 507 MiB used, 625 GiB / 626 GiB avail; 991 B/s rd, 991 B/s wr, 3 op/s 2024-04-14T15:21:46.368 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:46 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:21:46.368 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:46 smithi090 ceph-mon[41670]: osdmap e570: 8 total, 8 up, 7 in 2024-04-14T15:21:46.369 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:46 smithi090 ceph-mon[30399]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:21:46.369 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:46 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/4062140390' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-14T15:21:46.369 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:46 smithi090 ceph-mon[30399]: osdmap e569: 8 total, 8 up, 7 in 2024-04-14T15:21:46.369 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:46 smithi090 ceph-mon[30399]: pgmap v834: 248 pgs: 248 active+clean; 8.4 MiB data, 507 MiB used, 625 GiB / 626 GiB avail; 991 B/s rd, 991 B/s wr, 3 op/s 2024-04-14T15:21:46.369 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:46 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:21:46.369 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:46 smithi090 ceph-mon[30399]: osdmap e570: 8 total, 8 up, 7 in 2024-04-14T15:21:46.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:46.367+0000 7fa167458700 1 -- 172.21.15.90:0/4281095197 >> v1:172.21.15.90:6789/0 conn(0x7fa16010f320 legacy=0x7fa1601117e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:46.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:46.367+0000 7fa167458700 1 -- 172.21.15.90:0/4281095197 shutdown_connections 2024-04-14T15:21:46.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:46.367+0000 7fa167458700 1 -- 172.21.15.90:0/4281095197 >> 172.21.15.90:0/4281095197 conn(0x7fa1600fe3d0 msgr2=0x7fa1601007f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:21:46.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:46.367+0000 7fa167458700 1 -- 172.21.15.90:0/4281095197 shutdown_connections 2024-04-14T15:21:46.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:46.367+0000 7fa167458700 1 -- 172.21.15.90:0/4281095197 wait complete. 2024-04-14T15:21:46.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:46.368+0000 7fa167458700 1 librados: starting msgr at 2024-04-14T15:21:46.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:46.368+0000 7fa167458700 1 librados: starting objecter 2024-04-14T15:21:46.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:46.368+0000 7fa167458700 1 Processor -- start 2024-04-14T15:21:46.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:46.368+0000 7fa167458700 1 -- start start 2024-04-14T15:21:46.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:46.368+0000 7fa167458700 1 librados: setting wanted keys 2024-04-14T15:21:46.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:46.368+0000 7fa167458700 1 librados: calling monclient init 2024-04-14T15:21:46.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:46.368+0000 7fa167458700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa16010f070 con 0x7fa16010b750 2024-04-14T15:21:46.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:46.368+0000 7fa167458700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa1601a8f30 con 0x7fa1601088a0 2024-04-14T15:21:46.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:46.368+0000 7fa167458700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa1601aa110 con 0x7fa16010f320 2024-04-14T15:21:46.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:46.368+0000 7fa1649f3700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fa16010b750 0x7fa16010df40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:47796/0 (socket says 172.21.15.90:47796) 2024-04-14T15:21:46.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:46.368+0000 7fa1649f3700 1 -- 172.21.15.90:0/880421666 learned_addr learned my addr 172.21.15.90:0/880421666 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:21:46.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:46.369+0000 7fa1567fc700 10 client.?.objecter ms_handle_connect 0x7fa16010b750 2024-04-14T15:21:46.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:46.369+0000 7fa1567fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:21:46.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:46.369+0000 7fa1567fc700 10 client.?.objecter ms_handle_connect 0x7fa16010f320 2024-04-14T15:21:46.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:46.369+0000 7fa1567fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:21:46.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:46.369+0000 7fa1567fc700 1 -- 172.21.15.90:0/880421666 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3626553503 0 0) 0x7fa16010f070 con 0x7fa16010b750 2024-04-14T15:21:46.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:46.369+0000 7fa1567fc700 1 -- 172.21.15.90:0/880421666 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa14c0031f0 con 0x7fa16010b750 2024-04-14T15:21:46.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:46.369+0000 7fa1567fc700 10 client.?.objecter ms_handle_connect 0x7fa1601088a0 2024-04-14T15:21:46.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:46.369+0000 7fa1567fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:21:46.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:46.369+0000 7fa1567fc700 1 -- 172.21.15.90:0/880421666 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3859303878 0 0) 0x7fa1601aa110 con 0x7fa16010f320 2024-04-14T15:21:46.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:46.369+0000 7fa1567fc700 1 -- 172.21.15.90:0/880421666 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa16010f070 con 0x7fa16010f320 2024-04-14T15:21:46.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:46.369+0000 7fa1567fc700 1 -- 172.21.15.90:0/880421666 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 243009275 0 0) 0x7fa1601a8f30 con 0x7fa1601088a0 2024-04-14T15:21:46.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:46.369+0000 7fa1567fc700 1 -- 172.21.15.90:0/880421666 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa1601aa110 con 0x7fa1601088a0 2024-04-14T15:21:46.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:46.369+0000 7fa1567fc700 1 -- 172.21.15.90:0/880421666 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3236195347 0 0) 0x7fa16010f070 con 0x7fa16010f320 2024-04-14T15:21:46.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:46.369+0000 7fa1567fc700 1 -- 172.21.15.90:0/880421666 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa1601a8f30 con 0x7fa16010f320 2024-04-14T15:21:46.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:46.369+0000 7fa1567fc700 1 -- 172.21.15.90:0/880421666 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fa1580017b0 con 0x7fa16010f320 2024-04-14T15:21:46.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:46.369+0000 7fa1567fc700 1 -- 172.21.15.90:0/880421666 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4266156874 0 0) 0x7fa14c0031f0 con 0x7fa16010b750 2024-04-14T15:21:46.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:46.369+0000 7fa1567fc700 1 -- 172.21.15.90:0/880421666 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa16010f070 con 0x7fa16010b750 2024-04-14T15:21:46.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:46.369+0000 7fa1567fc700 1 -- 172.21.15.90:0/880421666 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fa1500018b0 con 0x7fa16010b750 2024-04-14T15:21:46.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:46.369+0000 7fa1567fc700 1 -- 172.21.15.90:0/880421666 <== mon.2 v1:172.21.15.90:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4024113109 0 0) 0x7fa1601a8f30 con 0x7fa16010f320 2024-04-14T15:21:46.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:46.369+0000 7fa1567fc700 1 -- 172.21.15.90:0/880421666 >> v1:172.21.15.92:6789/0 conn(0x7fa1601088a0 legacy=0x7fa16010d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:46.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:46.369+0000 7fa1567fc700 1 -- 172.21.15.90:0/880421666 >> v1:172.21.15.90:6789/0 conn(0x7fa16010b750 legacy=0x7fa16010df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:46.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:46.369+0000 7fa1567fc700 1 -- 172.21.15.90:0/880421666 --> v1:172.21.15.90:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1601ab350 con 0x7fa16010f320 2024-04-14T15:21:46.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:46.369+0000 7fa167458700 1 -- 172.21.15.90:0/880421666 --> v1:172.21.15.90:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa1601aa340 con 0x7fa16010f320 2024-04-14T15:21:46.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:46.369+0000 7fa167458700 10 client.47564.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:21:46.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:46.369+0000 7fa167458700 1 -- 172.21.15.90:0/880421666 --> v1:172.21.15.90:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fa1601aa800 con 0x7fa16010f320 2024-04-14T15:21:46.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:46.369+0000 7fa167458700 1 librados: init done 2024-04-14T15:21:46.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:46.370+0000 7fa1567fc700 1 -- 172.21.15.90:0/880421666 <== mon.2 v1:172.21.15.90:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fa158001950 con 0x7fa16010f320 2024-04-14T15:21:46.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:46.370+0000 7fa1567fc700 1 -- 172.21.15.90:0/880421666 <== mon.2 v1:172.21.15.90:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fa158004f40 con 0x7fa16010f320 2024-04-14T15:21:46.376 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:46 smithi092 ceph-mon[33966]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:21:46.376 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:46 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/4062140390' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-14T15:21:46.376 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:46 smithi092 ceph-mon[33966]: osdmap e569: 8 total, 8 up, 7 in 2024-04-14T15:21:46.376 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:46 smithi092 ceph-mon[33966]: pgmap v834: 248 pgs: 248 active+clean; 8.4 MiB data, 507 MiB used, 625 GiB / 626 GiB avail; 991 B/s rd, 991 B/s wr, 3 op/s 2024-04-14T15:21:46.376 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:46 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:21:46.376 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:46 smithi092 ceph-mon[33966]: osdmap e570: 8 total, 8 up, 7 in 2024-04-14T15:21:46.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:46.371+0000 7fa167458700 1 -- 172.21.15.90:0/880421666 --> v1:172.21.15.90:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa144005150 con 0x7fa16010f320 2024-04-14T15:21:46.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:46.371+0000 7fa1567fc700 1 -- 172.21.15.90:0/880421666 <== mon.2 v1:172.21.15.90:6790/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fa15801bb70 con 0x7fa16010f320 2024-04-14T15:21:46.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:46.372+0000 7fa1567fc700 1 -- 172.21.15.90:0/880421666 <== mon.2 v1:172.21.15.90:6790/0 8 ==== osd_map(570..570 src has 467..570) ==== 8168+0+0 (unknown 3819831024 0 0) 0x7fa158089c70 con 0x7fa16010f320 2024-04-14T15:21:46.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:46.372+0000 7fa1567fc700 10 client.47564.objecter ms_dispatch 0x7fa160001550 osd_map(570..570 src has 467..570) 2024-04-14T15:21:46.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:46.372+0000 7fa1567fc700 3 client.47564.objecter handle_osd_map got epochs [570,570] > 0 2024-04-14T15:21:46.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:46.372+0000 7fa1567fc700 3 client.47564.objecter handle_osd_map decoding full epoch 570 2024-04-14T15:21:46.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:46.372+0000 7fa1567fc700 20 client.47564.objecter dump_active .. 0 homeless 2024-04-14T15:21:46.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:46.372+0000 7fa1567fc700 10 client.47564.objecter ms_handle_connect 0x7fa14c079c80 2024-04-14T15:21:46.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:46.377+0000 7fa1567fc700 1 -- 172.21.15.90:0/880421666 <== mon.2 v1:172.21.15.90:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fa1580566c0 con 0x7fa16010f320 2024-04-14T15:21:46.394 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:46.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:46.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:21:46.397 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:21:46.496 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:21:46.519 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:46.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:46.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:21:46.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:46.597+0000 7fa167458700 1 -- 172.21.15.90:0/880421666 --> v1:172.21.15.90:6790/0 -- mon_command({"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} v 0) -- 0x7fa144005440 con 0x7fa16010f320 2024-04-14T15:21:46.620 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:21:46.649 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:46.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:46.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:21:46.750 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:21:46.778 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:46.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:46.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:21:46.873 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:21:46.880 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:46.907 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:46.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:46.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:47.009 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:21:47.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:47.038 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:47.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:21:47.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:47.078+0000 7fa1567fc700 1 -- 172.21.15.90:0/880421666 <== mon.2 v1:172.21.15.90:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]=0 set pool 1 pg_num to 11 v571) ==== 129+0+0 (unknown 3402647585 0 0) 0x7fa15805a370 con 0x7fa16010f320 2024-04-14T15:21:47.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:47.081+0000 7fa167458700 10 librados: watch_flush enter 2024-04-14T15:21:47.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:47.081+0000 7fa167458700 10 librados: watch_flush exit 2024-04-14T15:21:47.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:47.081+0000 7fa167458700 20 client.47564.objecter shutdown clearing up homeless session... 2024-04-14T15:21:47.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:47.081+0000 7fa167458700 10 client.47564.objecter successfully canceled tick 2024-04-14T15:21:47.082 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:47 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/880421666' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-04-14T15:21:47.082 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:47 smithi090 ceph-mon[41670]: from='client.47564 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-04-14T15:21:47.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:47.081+0000 7fa167458700 1 -- 172.21.15.90:0/880421666 >> v1:172.21.15.90:6800/420935427 conn(0x7fa14c079c80 legacy=0x7fa14c07c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:47.085 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:47 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/880421666' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-04-14T15:21:47.086 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:47 smithi090 ceph-mon[30399]: from='client.47564 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-04-14T15:21:47.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:47.081+0000 7fa167458700 1 -- 172.21.15.90:0/880421666 >> v1:172.21.15.90:6790/0 conn(0x7fa16010f320 legacy=0x7fa1601a6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:47.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:47.081+0000 7fa167458700 1 -- 172.21.15.90:0/880421666 shutdown_connections 2024-04-14T15:21:47.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:47.081+0000 7fa167458700 1 -- 172.21.15.90:0/880421666 >> 172.21.15.90:0/880421666 conn(0x7fa1600fe3d0 msgr2=0x7fa1601127c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:21:47.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:47.081+0000 7fa167458700 1 -- 172.21.15.90:0/880421666 shutdown_connections 2024-04-14T15:21:47.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:47.081+0000 7fa167458700 1 -- 172.21.15.90:0/880421666 wait complete. 2024-04-14T15:21:47.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:47.081+0000 7fa167458700 1 librados: shutdown 2024-04-14T15:21:47.087 INFO:teuthology.orchestra.run.smithi090.stderr:set pool 1 pg_num to 11 2024-04-14T15:21:47.140 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:21:47.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:47.165 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:47.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:21:47.266 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:21:47.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:47.291 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:47.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:21:47.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:47 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/880421666' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-04-14T15:21:47.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:47 smithi092 ceph-mon[33966]: from='client.47564 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-04-14T15:21:47.392 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:21:47.419 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:47.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:47.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:21:47.521 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:21:47.544 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:47.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:47.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:21:47.646 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:47.673 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:47.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:47.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:47.775 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:21:47.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:47.800 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:47.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:21:47.901 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:21:47.924 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:47.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:47.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:21:48.026 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:21:48.050 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:48.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:48.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:21:48.151 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd unset noscrub 2024-04-14T15:21:48.151 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:21:48.176 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:48.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:48.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:21:48.277 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:21:48.305 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:48.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:48.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:21:48.348 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:48 smithi090 ceph-mon[30399]: from='client.47564 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-04-14T15:21:48.348 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:48 smithi090 ceph-mon[30399]: osdmap e571: 8 total, 8 up, 7 in 2024-04-14T15:21:48.348 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:48 smithi090 ceph-mon[30399]: pgmap v837: 248 pgs: 248 active+clean; 8.4 MiB data, 507 MiB used, 625 GiB / 626 GiB avail; 997 B/s rd, 997 B/s wr, 3 op/s 2024-04-14T15:21:48.348 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:48 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-04-14T15:21:48.349 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:48 smithi090 ceph-mon[41670]: from='client.47564 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-04-14T15:21:48.349 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:48 smithi090 ceph-mon[41670]: osdmap e571: 8 total, 8 up, 7 in 2024-04-14T15:21:48.349 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:48 smithi090 ceph-mon[41670]: pgmap v837: 248 pgs: 248 active+clean; 8.4 MiB data, 507 MiB used, 625 GiB / 626 GiB avail; 997 B/s rd, 997 B/s wr, 3 op/s 2024-04-14T15:21:48.349 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:48 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-04-14T15:21:48.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:48 smithi092 ceph-mon[33966]: from='client.47564 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-04-14T15:21:48.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:48 smithi092 ceph-mon[33966]: osdmap e571: 8 total, 8 up, 7 in 2024-04-14T15:21:48.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:48 smithi092 ceph-mon[33966]: pgmap v837: 248 pgs: 248 active+clean; 8.4 MiB data, 507 MiB used, 625 GiB / 626 GiB avail; 997 B/s rd, 997 B/s wr, 3 op/s 2024-04-14T15:21:48.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:48 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-04-14T15:21:48.407 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:21:48.434 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:48.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:48.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:21:48.536 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:21:48.560 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:48.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:48.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:21:48.661 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:48.689 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:48.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:48.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:48.791 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:21:48.815 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:48.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:48.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:21:48.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.896+0000 7f42beb44700 1 -- 172.21.15.90:0/3777545027 >> v1:172.21.15.90:6789/0 conn(0x7f42b810b740 legacy=0x7f42b810db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:48.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.896+0000 7f42beb44700 1 -- 172.21.15.90:0/3777545027 shutdown_connections 2024-04-14T15:21:48.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.896+0000 7f42beb44700 1 -- 172.21.15.90:0/3777545027 >> 172.21.15.90:0/3777545027 conn(0x7f42b80fe380 msgr2=0x7f42b81007a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:21:48.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.896+0000 7f42beb44700 1 -- 172.21.15.90:0/3777545027 shutdown_connections 2024-04-14T15:21:48.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.896+0000 7f42beb44700 1 -- 172.21.15.90:0/3777545027 wait complete. 2024-04-14T15:21:48.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.897+0000 7f42beb44700 1 librados: starting msgr at 2024-04-14T15:21:48.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.897+0000 7f42beb44700 1 librados: starting objecter 2024-04-14T15:21:48.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.897+0000 7f42beb44700 1 Processor -- start 2024-04-14T15:21:48.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.897+0000 7f42beb44700 1 -- start start 2024-04-14T15:21:48.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.897+0000 7f42beb44700 1 librados: setting wanted keys 2024-04-14T15:21:48.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.897+0000 7f42beb44700 1 librados: calling monclient init 2024-04-14T15:21:48.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.897+0000 7f42beb44700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f42b81a9ba0 con 0x7f42b810f310 2024-04-14T15:21:48.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.897+0000 7f42beb44700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f42b81aadc0 con 0x7f42b810b740 2024-04-14T15:21:48.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.897+0000 7f42beb44700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f42b81ac000 con 0x7f42b8108890 2024-04-14T15:21:48.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.897+0000 7f42bc8e0700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f42b8108890 0x7f42b810eb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:53420/0 (socket says 172.21.15.90:53420) 2024-04-14T15:21:48.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.897+0000 7f42bc8e0700 1 -- 172.21.15.90:0/1929515481 learned_addr learned my addr 172.21.15.90:0/1929515481 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:21:48.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.897+0000 7f42b5ffb700 10 client.?.objecter ms_handle_connect 0x7f42b810f310 2024-04-14T15:21:48.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.897+0000 7f42b5ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:21:48.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.897+0000 7f42b5ffb700 10 client.?.objecter ms_handle_connect 0x7f42b8108890 2024-04-14T15:21:48.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.897+0000 7f42b5ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:21:48.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.897+0000 7f42b5ffb700 1 -- 172.21.15.90:0/1929515481 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 830887130 0 0) 0x7f42b81ac000 con 0x7f42b8108890 2024-04-14T15:21:48.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.897+0000 7f42b5ffb700 1 -- 172.21.15.90:0/1929515481 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42a40031f0 con 0x7f42b8108890 2024-04-14T15:21:48.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.897+0000 7f42b5ffb700 1 -- 172.21.15.90:0/1929515481 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 151122734 0 0) 0x7f42b81a9ba0 con 0x7f42b810f310 2024-04-14T15:21:48.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.897+0000 7f42b5ffb700 1 -- 172.21.15.90:0/1929515481 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42b81ac000 con 0x7f42b810f310 2024-04-14T15:21:48.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.897+0000 7f42b5ffb700 10 client.?.objecter ms_handle_connect 0x7f42b810b740 2024-04-14T15:21:48.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.898+0000 7f42b5ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:21:48.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.898+0000 7f42b5ffb700 1 -- 172.21.15.90:0/1929515481 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2859003022 0 0) 0x7f42b81aadc0 con 0x7f42b810b740 2024-04-14T15:21:48.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.898+0000 7f42b5ffb700 1 -- 172.21.15.90:0/1929515481 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42b81a9ba0 con 0x7f42b810b740 2024-04-14T15:21:48.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.898+0000 7f42b5ffb700 1 -- 172.21.15.90:0/1929515481 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3500324931 0 0) 0x7f42b81ac000 con 0x7f42b810f310 2024-04-14T15:21:48.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.898+0000 7f42b5ffb700 1 -- 172.21.15.90:0/1929515481 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f42b81aadc0 con 0x7f42b810f310 2024-04-14T15:21:48.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.898+0000 7f42b5ffb700 1 -- 172.21.15.90:0/1929515481 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f42a80018b0 con 0x7f42b810f310 2024-04-14T15:21:48.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.898+0000 7f42b5ffb700 1 -- 172.21.15.90:0/1929515481 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2774115889 0 0) 0x7f42a40031f0 con 0x7f42b8108890 2024-04-14T15:21:48.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.898+0000 7f42b5ffb700 1 -- 172.21.15.90:0/1929515481 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f42b81ac000 con 0x7f42b8108890 2024-04-14T15:21:48.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.898+0000 7f42b5ffb700 1 -- 172.21.15.90:0/1929515481 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f42b0001650 con 0x7f42b8108890 2024-04-14T15:21:48.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.898+0000 7f42b5ffb700 1 -- 172.21.15.90:0/1929515481 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3958001644 0 0) 0x7f42b81a9ba0 con 0x7f42b810b740 2024-04-14T15:21:48.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.898+0000 7f42b5ffb700 1 -- 172.21.15.90:0/1929515481 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f42a40031f0 con 0x7f42b810b740 2024-04-14T15:21:48.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.898+0000 7f42b5ffb700 1 -- 172.21.15.90:0/1929515481 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f42a0004540 con 0x7f42b810b740 2024-04-14T15:21:48.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.898+0000 7f42b5ffb700 1 -- 172.21.15.90:0/1929515481 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2229261674 0 0) 0x7f42b81aadc0 con 0x7f42b810f310 2024-04-14T15:21:48.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.898+0000 7f42b5ffb700 1 -- 172.21.15.90:0/1929515481 >> v1:172.21.15.90:6790/0 conn(0x7f42b8108890 legacy=0x7f42b810eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:48.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.898+0000 7f42b5ffb700 1 -- 172.21.15.90:0/1929515481 >> v1:172.21.15.92:6789/0 conn(0x7f42b810b740 legacy=0x7f42b81a4bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:48.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.898+0000 7f42b5ffb700 1 -- 172.21.15.90:0/1929515481 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42b81ad240 con 0x7f42b810f310 2024-04-14T15:21:48.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.898+0000 7f42beb44700 1 -- 172.21.15.90:0/1929515481 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f42b81ac230 con 0x7f42b810f310 2024-04-14T15:21:48.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.898+0000 7f42b5ffb700 1 -- 172.21.15.90:0/1929515481 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f42a8003b40 con 0x7f42b810f310 2024-04-14T15:21:48.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.898+0000 7f42b5ffb700 1 -- 172.21.15.90:0/1929515481 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f42a8004b10 con 0x7f42b810f310 2024-04-14T15:21:48.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.898+0000 7f42beb44700 10 client.37617.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:21:48.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.898+0000 7f42beb44700 1 -- 172.21.15.90:0/1929515481 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f42b81ac6f0 con 0x7f42b810f310 2024-04-14T15:21:48.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.898+0000 7f42beb44700 1 librados: init done 2024-04-14T15:21:48.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.899+0000 7f42beb44700 1 -- 172.21.15.90:0/1929515481 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f42b8103fd0 con 0x7f42b810f310 2024-04-14T15:21:48.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.900+0000 7f42b5ffb700 1 -- 172.21.15.90:0/1929515481 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f42a80036a0 con 0x7f42b810f310 2024-04-14T15:21:48.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.900+0000 7f42b5ffb700 1 -- 172.21.15.90:0/1929515481 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(572..572 src has 467..572) ==== 8168+0+0 (unknown 1176036544 0 0) 0x7f42a8089a70 con 0x7f42b810f310 2024-04-14T15:21:48.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.900+0000 7f42b5ffb700 10 client.37617.objecter ms_dispatch 0x7f42b8001550 osd_map(572..572 src has 467..572) 2024-04-14T15:21:48.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.900+0000 7f42b5ffb700 3 client.37617.objecter handle_osd_map got epochs [572,572] > 0 2024-04-14T15:21:48.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.900+0000 7f42b5ffb700 3 client.37617.objecter handle_osd_map decoding full epoch 572 2024-04-14T15:21:48.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.900+0000 7f42b5ffb700 20 client.37617.objecter dump_active .. 0 homeless 2024-04-14T15:21:48.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.900+0000 7f42b5ffb700 10 client.37617.objecter ms_handle_connect 0x7f42a4071340 2024-04-14T15:21:48.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.900+0000 7f42b5ffb700 1 -- 172.21.15.90:0/1929515481 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v572) ==== 54+0+16 (unknown 1682992726 0 2420000192) 0x7f42a8089ea0 con 0x7f42b810f310 2024-04-14T15:21:48.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.900+0000 7f42beb44700 10 client.37617.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:21:48.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.900+0000 7f42beb44700 15 client.37617.objecter _session_command_op_assign -1 1 2024-04-14T15:21:48.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.900+0000 7f42beb44700 20 client.37617.objecter _get_session s=0x7f42b80693d0 osd=0 3 2024-04-14T15:21:48.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.900+0000 7f42beb44700 20 client.37617.objecter put_session s=0x7f42b80693d0 osd=0 3 2024-04-14T15:21:48.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.901+0000 7f42beb44700 20 client.37617.objecter _get_session s=0x7f42b80693d0 osd=0 3 2024-04-14T15:21:48.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.901+0000 7f42beb44700 15 client.37617.objecter _session_command_op_remove -1 1 2024-04-14T15:21:48.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.901+0000 7f42beb44700 20 client.37617.objecter get_session s=0x7f42b80693d0 osd=0 3 2024-04-14T15:21:48.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.901+0000 7f42beb44700 15 client.37617.objecter _session_command_op_assign 0 1 2024-04-14T15:21:48.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.901+0000 7f42beb44700 20 client.37617.objecter put_session s=0x7f42b80693d0 osd=0 4 2024-04-14T15:21:48.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.901+0000 7f42beb44700 10 client.37617.objecter _send_command 1 2024-04-14T15:21:48.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.901+0000 7f42beb44700 1 -- 172.21.15.90:0/1929515481 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f42b8076250 con 0x7f42b81ab4a0 2024-04-14T15:21:48.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.901+0000 7f42b5ffb700 10 client.37617.objecter ms_handle_connect 0x7f42b81ab4a0 2024-04-14T15:21:48.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.901+0000 7f42b5ffb700 1 -- 172.21.15.90:0/1929515481 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f42b8076250 con 0x7f42b81ab4a0 2024-04-14T15:21:48.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.901+0000 7f42b5ffb700 10 client.37617.objecter ms_dispatch 0x7f42b8001550 command_reply(tid 1: 0 ) 2024-04-14T15:21:48.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.901+0000 7f42b5ffb700 10 client.37617.objecter _finish_command 1 = system:0 2024-04-14T15:21:48.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.901+0000 7f42b5ffb700 20 client.37617.objecter put_session s=0x7f42b80693d0 osd=0 4 2024-04-14T15:21:48.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.901+0000 7f42b5ffb700 15 client.37617.objecter _session_command_op_remove 0 1 2024-04-14T15:21:48.918 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:21:48.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.918+0000 7f42beb44700 10 client.37617.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:21:48.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.918+0000 7f42beb44700 15 client.37617.objecter _session_command_op_assign -1 2 2024-04-14T15:21:48.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.918+0000 7f42beb44700 20 client.37617.objecter _get_session s=0x7f42b80693d0 osd=0 3 2024-04-14T15:21:48.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.918+0000 7f42beb44700 20 client.37617.objecter put_session s=0x7f42b80693d0 osd=0 3 2024-04-14T15:21:48.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.918+0000 7f42beb44700 20 client.37617.objecter _get_session s=0x7f42b80693d0 osd=0 3 2024-04-14T15:21:48.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.918+0000 7f42beb44700 15 client.37617.objecter _session_command_op_remove -1 2 2024-04-14T15:21:48.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.918+0000 7f42beb44700 20 client.37617.objecter get_session s=0x7f42b80693d0 osd=0 3 2024-04-14T15:21:48.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.918+0000 7f42beb44700 15 client.37617.objecter _session_command_op_assign 0 2 2024-04-14T15:21:48.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.918+0000 7f42beb44700 20 client.37617.objecter put_session s=0x7f42b80693d0 osd=0 4 2024-04-14T15:21:48.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.918+0000 7f42beb44700 10 client.37617.objecter _send_command 2 2024-04-14T15:21:48.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.918+0000 7f42beb44700 1 -- 172.21.15.90:0/1929515481 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f42b81acc30 con 0x7f42b81ab4a0 2024-04-14T15:21:48.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.920+0000 7f42b5ffb700 1 -- 172.21.15.90:0/1929515481 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f42b81acc30 con 0x7f42b81ab4a0 2024-04-14T15:21:48.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.920+0000 7f42b5ffb700 10 client.37617.objecter ms_dispatch 0x7f42b8001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:21:48.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.920+0000 7f42b5ffb700 10 client.37617.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:21:48.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.920+0000 7f42b5ffb700 20 client.37617.objecter put_session s=0x7f42b80693d0 osd=0 4 2024-04-14T15:21:48.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.920+0000 7f42b5ffb700 15 client.37617.objecter _session_command_op_remove 0 2 2024-04-14T15:21:48.925 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:21:48.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.920+0000 7f42beb44700 10 client.37617.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:21:48.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.920+0000 7f42beb44700 15 client.37617.objecter _session_command_op_assign -1 3 2024-04-14T15:21:48.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.920+0000 7f42beb44700 20 client.37617.objecter _get_session s=0x7f42b81ad030 osd=1 3 2024-04-14T15:21:48.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.920+0000 7f42beb44700 20 client.37617.objecter put_session s=0x7f42b81ad030 osd=1 3 2024-04-14T15:21:48.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.920+0000 7f42beb44700 20 client.37617.objecter _get_session s=0x7f42b81ad030 osd=1 3 2024-04-14T15:21:48.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.920+0000 7f42beb44700 15 client.37617.objecter _session_command_op_remove -1 3 2024-04-14T15:21:48.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.920+0000 7f42beb44700 20 client.37617.objecter get_session s=0x7f42b81ad030 osd=1 3 2024-04-14T15:21:48.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.920+0000 7f42beb44700 15 client.37617.objecter _session_command_op_assign 1 3 2024-04-14T15:21:48.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.920+0000 7f42beb44700 20 client.37617.objecter put_session s=0x7f42b81ad030 osd=1 4 2024-04-14T15:21:48.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.920+0000 7f42beb44700 10 client.37617.objecter _send_command 3 2024-04-14T15:21:48.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.920+0000 7f42beb44700 1 -- 172.21.15.90:0/1929515481 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f42b8110ee0 con 0x7f42b80636c0 2024-04-14T15:21:48.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.920+0000 7f42b5ffb700 10 client.37617.objecter ms_handle_connect 0x7f42b80636c0 2024-04-14T15:21:48.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.921+0000 7f42b5ffb700 1 -- 172.21.15.90:0/1929515481 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f42b8110ee0 con 0x7f42b80636c0 2024-04-14T15:21:48.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.921+0000 7f42b5ffb700 10 client.37617.objecter ms_dispatch 0x7f42b8001550 command_reply(tid 3: 0 ) 2024-04-14T15:21:48.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.921+0000 7f42b5ffb700 10 client.37617.objecter _finish_command 3 = system:0 2024-04-14T15:21:48.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.921+0000 7f42b5ffb700 20 client.37617.objecter put_session s=0x7f42b81ad030 osd=1 4 2024-04-14T15:21:48.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.921+0000 7f42b5ffb700 15 client.37617.objecter _session_command_op_remove 1 3 2024-04-14T15:21:48.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.937+0000 7f42beb44700 10 client.37617.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:21:48.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.938+0000 7f42beb44700 15 client.37617.objecter _session_command_op_assign -1 4 2024-04-14T15:21:48.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.938+0000 7f42beb44700 20 client.37617.objecter _get_session s=0x7f42b81ad030 osd=1 3 2024-04-14T15:21:48.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.938+0000 7f42beb44700 20 client.37617.objecter put_session s=0x7f42b81ad030 osd=1 3 2024-04-14T15:21:48.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.938+0000 7f42beb44700 20 client.37617.objecter _get_session s=0x7f42b81ad030 osd=1 3 2024-04-14T15:21:48.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.938+0000 7f42beb44700 15 client.37617.objecter _session_command_op_remove -1 4 2024-04-14T15:21:48.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.938+0000 7f42beb44700 20 client.37617.objecter get_session s=0x7f42b81ad030 osd=1 3 2024-04-14T15:21:48.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.938+0000 7f42beb44700 15 client.37617.objecter _session_command_op_assign 1 4 2024-04-14T15:21:48.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.938+0000 7f42beb44700 20 client.37617.objecter put_session s=0x7f42b81ad030 osd=1 4 2024-04-14T15:21:48.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.938+0000 7f42beb44700 10 client.37617.objecter _send_command 4 2024-04-14T15:21:48.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.938+0000 7f42beb44700 1 -- 172.21.15.90:0/1929515481 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f42b8063d70 con 0x7f42b80636c0 2024-04-14T15:21:48.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.939+0000 7f42b5ffb700 1 -- 172.21.15.90:0/1929515481 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f42b8063d70 con 0x7f42b80636c0 2024-04-14T15:21:48.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.939+0000 7f42b5ffb700 10 client.37617.objecter ms_dispatch 0x7f42b8001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:21:48.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.939+0000 7f42b5ffb700 10 client.37617.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:21:48.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.939+0000 7f42b5ffb700 20 client.37617.objecter put_session s=0x7f42b81ad030 osd=1 4 2024-04-14T15:21:48.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.939+0000 7f42b5ffb700 15 client.37617.objecter _session_command_op_remove 1 4 2024-04-14T15:21:48.940 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:21:48.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.939+0000 7f42beb44700 10 client.37617.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:21:48.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.939+0000 7f42beb44700 15 client.37617.objecter _session_command_op_assign -1 5 2024-04-14T15:21:48.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.939+0000 7f42beb44700 20 client.37617.objecter _get_session s=0x7f42b8063140 osd=2 3 2024-04-14T15:21:48.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.939+0000 7f42beb44700 20 client.37617.objecter put_session s=0x7f42b8063140 osd=2 3 2024-04-14T15:21:48.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.939+0000 7f42beb44700 20 client.37617.objecter _get_session s=0x7f42b8063140 osd=2 3 2024-04-14T15:21:48.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.939+0000 7f42beb44700 15 client.37617.objecter _session_command_op_remove -1 5 2024-04-14T15:21:48.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.939+0000 7f42beb44700 20 client.37617.objecter get_session s=0x7f42b8063140 osd=2 3 2024-04-14T15:21:48.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.939+0000 7f42beb44700 15 client.37617.objecter _session_command_op_assign 2 5 2024-04-14T15:21:48.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.939+0000 7f42beb44700 20 client.37617.objecter put_session s=0x7f42b8063140 osd=2 4 2024-04-14T15:21:48.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.939+0000 7f42beb44700 10 client.37617.objecter _send_command 5 2024-04-14T15:21:48.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.939+0000 7f42beb44700 1 -- 172.21.15.90:0/1929515481 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f42b802db00 con 0x7f42b80632b0 2024-04-14T15:21:48.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.940+0000 7f42b5ffb700 10 client.37617.objecter ms_handle_connect 0x7f42b80632b0 2024-04-14T15:21:48.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.940+0000 7f42b5ffb700 1 -- 172.21.15.90:0/1929515481 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f42b802db00 con 0x7f42b80632b0 2024-04-14T15:21:48.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.940+0000 7f42b5ffb700 10 client.37617.objecter ms_dispatch 0x7f42b8001550 command_reply(tid 5: 0 ) 2024-04-14T15:21:48.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.940+0000 7f42b5ffb700 10 client.37617.objecter _finish_command 5 = system:0 2024-04-14T15:21:48.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.940+0000 7f42b5ffb700 20 client.37617.objecter put_session s=0x7f42b8063140 osd=2 4 2024-04-14T15:21:48.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.940+0000 7f42b5ffb700 15 client.37617.objecter _session_command_op_remove 2 5 2024-04-14T15:21:48.949 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:48.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:48.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:21:48.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.957+0000 7f42beb44700 10 client.37617.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:21:48.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.957+0000 7f42beb44700 15 client.37617.objecter _session_command_op_assign -1 6 2024-04-14T15:21:48.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.957+0000 7f42beb44700 20 client.37617.objecter _get_session s=0x7f42b8063140 osd=2 3 2024-04-14T15:21:48.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.957+0000 7f42beb44700 20 client.37617.objecter put_session s=0x7f42b8063140 osd=2 3 2024-04-14T15:21:48.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.957+0000 7f42beb44700 20 client.37617.objecter _get_session s=0x7f42b8063140 osd=2 3 2024-04-14T15:21:48.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.957+0000 7f42beb44700 15 client.37617.objecter _session_command_op_remove -1 6 2024-04-14T15:21:48.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.957+0000 7f42beb44700 20 client.37617.objecter get_session s=0x7f42b8063140 osd=2 3 2024-04-14T15:21:48.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.957+0000 7f42beb44700 15 client.37617.objecter _session_command_op_assign 2 6 2024-04-14T15:21:48.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.957+0000 7f42beb44700 20 client.37617.objecter put_session s=0x7f42b8063140 osd=2 4 2024-04-14T15:21:48.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.957+0000 7f42beb44700 10 client.37617.objecter _send_command 6 2024-04-14T15:21:48.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.957+0000 7f42beb44700 1 -- 172.21.15.90:0/1929515481 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f42b81b5da0 con 0x7f42b80632b0 2024-04-14T15:21:48.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.958+0000 7f42b5ffb700 1 -- 172.21.15.90:0/1929515481 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f42b81b5da0 con 0x7f42b80632b0 2024-04-14T15:21:48.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.958+0000 7f42b5ffb700 10 client.37617.objecter ms_dispatch 0x7f42b8001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:21:48.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.958+0000 7f42b5ffb700 10 client.37617.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:21:48.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.958+0000 7f42b5ffb700 20 client.37617.objecter put_session s=0x7f42b8063140 osd=2 4 2024-04-14T15:21:48.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.958+0000 7f42b5ffb700 15 client.37617.objecter _session_command_op_remove 2 6 2024-04-14T15:21:48.959 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:21:48.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.959+0000 7f42beb44700 10 client.37617.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:21:48.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.959+0000 7f42beb44700 15 client.37617.objecter _session_command_op_assign -1 7 2024-04-14T15:21:48.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.959+0000 7f42beb44700 20 client.37617.objecter _get_session s=0x7f42b802d070 osd=3 3 2024-04-14T15:21:48.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.959+0000 7f42beb44700 20 client.37617.objecter put_session s=0x7f42b802d070 osd=3 3 2024-04-14T15:21:48.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.959+0000 7f42beb44700 20 client.37617.objecter _get_session s=0x7f42b802d070 osd=3 3 2024-04-14T15:21:48.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.959+0000 7f42beb44700 15 client.37617.objecter _session_command_op_remove -1 7 2024-04-14T15:21:48.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.959+0000 7f42beb44700 20 client.37617.objecter get_session s=0x7f42b802d070 osd=3 3 2024-04-14T15:21:48.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.959+0000 7f42beb44700 15 client.37617.objecter _session_command_op_assign 3 7 2024-04-14T15:21:48.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.959+0000 7f42beb44700 20 client.37617.objecter put_session s=0x7f42b802d070 osd=3 4 2024-04-14T15:21:48.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.959+0000 7f42beb44700 10 client.37617.objecter _send_command 7 2024-04-14T15:21:48.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.959+0000 7f42beb44700 1 -- 172.21.15.90:0/1929515481 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f42b81b2200 con 0x7f42b802d1e0 2024-04-14T15:21:48.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.959+0000 7f42b5ffb700 10 client.37617.objecter ms_handle_connect 0x7f42b802d1e0 2024-04-14T15:21:48.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.960+0000 7f42b5ffb700 1 -- 172.21.15.90:0/1929515481 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f42b81b2200 con 0x7f42b802d1e0 2024-04-14T15:21:48.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.960+0000 7f42b5ffb700 10 client.37617.objecter ms_dispatch 0x7f42b8001550 command_reply(tid 7: 0 ) 2024-04-14T15:21:48.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.960+0000 7f42b5ffb700 10 client.37617.objecter _finish_command 7 = system:0 2024-04-14T15:21:48.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.960+0000 7f42b5ffb700 20 client.37617.objecter put_session s=0x7f42b802d070 osd=3 4 2024-04-14T15:21:48.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.960+0000 7f42b5ffb700 15 client.37617.objecter _session_command_op_remove 3 7 2024-04-14T15:21:48.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.976+0000 7f42beb44700 10 client.37617.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:21:48.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.976+0000 7f42beb44700 15 client.37617.objecter _session_command_op_assign -1 8 2024-04-14T15:21:48.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.976+0000 7f42beb44700 20 client.37617.objecter _get_session s=0x7f42b802d070 osd=3 3 2024-04-14T15:21:48.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.976+0000 7f42beb44700 20 client.37617.objecter put_session s=0x7f42b802d070 osd=3 3 2024-04-14T15:21:48.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.976+0000 7f42beb44700 20 client.37617.objecter _get_session s=0x7f42b802d070 osd=3 3 2024-04-14T15:21:48.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.976+0000 7f42beb44700 15 client.37617.objecter _session_command_op_remove -1 8 2024-04-14T15:21:48.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.976+0000 7f42beb44700 20 client.37617.objecter get_session s=0x7f42b802d070 osd=3 3 2024-04-14T15:21:48.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.976+0000 7f42beb44700 15 client.37617.objecter _session_command_op_assign 3 8 2024-04-14T15:21:48.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.976+0000 7f42beb44700 20 client.37617.objecter put_session s=0x7f42b802d070 osd=3 4 2024-04-14T15:21:48.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.976+0000 7f42beb44700 10 client.37617.objecter _send_command 8 2024-04-14T15:21:48.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.976+0000 7f42beb44700 1 -- 172.21.15.90:0/1929515481 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f42b81b2480 con 0x7f42b802d1e0 2024-04-14T15:21:48.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.978+0000 7f42b5ffb700 1 -- 172.21.15.90:0/1929515481 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f42b81b2480 con 0x7f42b802d1e0 2024-04-14T15:21:48.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.978+0000 7f42b5ffb700 10 client.37617.objecter ms_dispatch 0x7f42b8001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:21:48.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.978+0000 7f42b5ffb700 10 client.37617.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:21:48.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.978+0000 7f42b5ffb700 20 client.37617.objecter put_session s=0x7f42b802d070 osd=3 4 2024-04-14T15:21:48.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.978+0000 7f42b5ffb700 15 client.37617.objecter _session_command_op_remove 3 8 2024-04-14T15:21:48.980 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:21:48.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.978+0000 7f42beb44700 10 client.37617.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:21:48.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.978+0000 7f42beb44700 15 client.37617.objecter _session_command_op_assign -1 9 2024-04-14T15:21:48.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.978+0000 7f42beb44700 20 client.37617.objecter _get_session s=0x7f42b81b2480 osd=4 3 2024-04-14T15:21:48.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.978+0000 7f42beb44700 20 client.37617.objecter put_session s=0x7f42b81b2480 osd=4 3 2024-04-14T15:21:48.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.978+0000 7f42beb44700 20 client.37617.objecter _get_session s=0x7f42b81b2480 osd=4 3 2024-04-14T15:21:48.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.978+0000 7f42beb44700 15 client.37617.objecter _session_command_op_remove -1 9 2024-04-14T15:21:48.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.978+0000 7f42beb44700 20 client.37617.objecter get_session s=0x7f42b81b2480 osd=4 3 2024-04-14T15:21:48.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.978+0000 7f42beb44700 15 client.37617.objecter _session_command_op_assign 4 9 2024-04-14T15:21:48.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.978+0000 7f42beb44700 20 client.37617.objecter put_session s=0x7f42b81b2480 osd=4 4 2024-04-14T15:21:48.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.978+0000 7f42beb44700 10 client.37617.objecter _send_command 9 2024-04-14T15:21:48.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.978+0000 7f42beb44700 1 -- 172.21.15.90:0/1929515481 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f42b81b6ba0 con 0x7f42b81b6720 2024-04-14T15:21:48.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.980+0000 7f42b5ffb700 10 client.37617.objecter ms_handle_connect 0x7f42b81b6720 2024-04-14T15:21:48.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.981+0000 7f42b5ffb700 1 -- 172.21.15.90:0/1929515481 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f42b81b6ba0 con 0x7f42b81b6720 2024-04-14T15:21:48.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.981+0000 7f42b5ffb700 10 client.37617.objecter ms_dispatch 0x7f42b8001550 command_reply(tid 9: 0 ) 2024-04-14T15:21:48.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.981+0000 7f42b5ffb700 10 client.37617.objecter _finish_command 9 = system:0 2024-04-14T15:21:48.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.981+0000 7f42b5ffb700 20 client.37617.objecter put_session s=0x7f42b81b2480 osd=4 4 2024-04-14T15:21:48.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:48.981+0000 7f42b5ffb700 15 client.37617.objecter _session_command_op_remove 4 9 2024-04-14T15:21:49.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.002+0000 7f42beb44700 10 client.37617.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:21:49.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.002+0000 7f42beb44700 15 client.37617.objecter _session_command_op_assign -1 10 2024-04-14T15:21:49.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.002+0000 7f42beb44700 20 client.37617.objecter _get_session s=0x7f42b81b2480 osd=4 3 2024-04-14T15:21:49.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.002+0000 7f42beb44700 20 client.37617.objecter put_session s=0x7f42b81b2480 osd=4 3 2024-04-14T15:21:49.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.002+0000 7f42beb44700 20 client.37617.objecter _get_session s=0x7f42b81b2480 osd=4 3 2024-04-14T15:21:49.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.002+0000 7f42beb44700 15 client.37617.objecter _session_command_op_remove -1 10 2024-04-14T15:21:49.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.002+0000 7f42beb44700 20 client.37617.objecter get_session s=0x7f42b81b2480 osd=4 3 2024-04-14T15:21:49.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.002+0000 7f42beb44700 15 client.37617.objecter _session_command_op_assign 4 10 2024-04-14T15:21:49.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.002+0000 7f42beb44700 20 client.37617.objecter put_session s=0x7f42b81b2480 osd=4 4 2024-04-14T15:21:49.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.002+0000 7f42beb44700 10 client.37617.objecter _send_command 10 2024-04-14T15:21:49.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.002+0000 7f42beb44700 1 -- 172.21.15.90:0/1929515481 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f42b81b6e20 con 0x7f42b81b6720 2024-04-14T15:21:49.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.004+0000 7f42b5ffb700 1 -- 172.21.15.90:0/1929515481 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f42b81b6e20 con 0x7f42b81b6720 2024-04-14T15:21:49.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.004+0000 7f42b5ffb700 10 client.37617.objecter ms_dispatch 0x7f42b8001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:21:49.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.004+0000 7f42b5ffb700 10 client.37617.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:21:49.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.004+0000 7f42b5ffb700 20 client.37617.objecter put_session s=0x7f42b81b2480 osd=4 4 2024-04-14T15:21:49.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.004+0000 7f42b5ffb700 15 client.37617.objecter _session_command_op_remove 4 10 2024-04-14T15:21:49.004 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:21:49.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.004+0000 7f42beb44700 10 client.37617.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:21:49.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.004+0000 7f42beb44700 15 client.37617.objecter _session_command_op_assign -1 11 2024-04-14T15:21:49.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.004+0000 7f42beb44700 20 client.37617.objecter _get_session s=0x7f42b81b6e20 osd=5 3 2024-04-14T15:21:49.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.004+0000 7f42beb44700 20 client.37617.objecter put_session s=0x7f42b81b6e20 osd=5 3 2024-04-14T15:21:49.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.004+0000 7f42beb44700 20 client.37617.objecter _get_session s=0x7f42b81b6e20 osd=5 3 2024-04-14T15:21:49.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.004+0000 7f42beb44700 15 client.37617.objecter _session_command_op_remove -1 11 2024-04-14T15:21:49.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.004+0000 7f42beb44700 20 client.37617.objecter get_session s=0x7f42b81b6e20 osd=5 3 2024-04-14T15:21:49.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.004+0000 7f42beb44700 15 client.37617.objecter _session_command_op_assign 5 11 2024-04-14T15:21:49.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.004+0000 7f42beb44700 20 client.37617.objecter put_session s=0x7f42b81b6e20 osd=5 4 2024-04-14T15:21:49.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.004+0000 7f42beb44700 10 client.37617.objecter _send_command 11 2024-04-14T15:21:49.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.004+0000 7f42beb44700 1 -- 172.21.15.90:0/1929515481 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f42b81c31c0 con 0x7f42b81bf5c0 2024-04-14T15:21:49.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.004+0000 7f42b5ffb700 10 client.37617.objecter ms_handle_connect 0x7f42b81bf5c0 2024-04-14T15:21:49.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.005+0000 7f42b5ffb700 1 -- 172.21.15.90:0/1929515481 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f42b81c31c0 con 0x7f42b81bf5c0 2024-04-14T15:21:49.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.005+0000 7f42b5ffb700 10 client.37617.objecter ms_dispatch 0x7f42b8001550 command_reply(tid 11: 0 ) 2024-04-14T15:21:49.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.005+0000 7f42b5ffb700 10 client.37617.objecter _finish_command 11 = system:0 2024-04-14T15:21:49.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.005+0000 7f42b5ffb700 20 client.37617.objecter put_session s=0x7f42b81b6e20 osd=5 4 2024-04-14T15:21:49.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.005+0000 7f42b5ffb700 15 client.37617.objecter _session_command_op_remove 5 11 2024-04-14T15:21:49.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.022+0000 7f42beb44700 10 client.37617.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:21:49.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.022+0000 7f42beb44700 15 client.37617.objecter _session_command_op_assign -1 12 2024-04-14T15:21:49.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.022+0000 7f42beb44700 20 client.37617.objecter _get_session s=0x7f42b81b6e20 osd=5 3 2024-04-14T15:21:49.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.022+0000 7f42beb44700 20 client.37617.objecter put_session s=0x7f42b81b6e20 osd=5 3 2024-04-14T15:21:49.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.022+0000 7f42beb44700 20 client.37617.objecter _get_session s=0x7f42b81b6e20 osd=5 3 2024-04-14T15:21:49.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.022+0000 7f42beb44700 15 client.37617.objecter _session_command_op_remove -1 12 2024-04-14T15:21:49.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.022+0000 7f42beb44700 20 client.37617.objecter get_session s=0x7f42b81b6e20 osd=5 3 2024-04-14T15:21:49.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.022+0000 7f42beb44700 15 client.37617.objecter _session_command_op_assign 5 12 2024-04-14T15:21:49.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.022+0000 7f42beb44700 20 client.37617.objecter put_session s=0x7f42b81b6e20 osd=5 4 2024-04-14T15:21:49.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.022+0000 7f42beb44700 10 client.37617.objecter _send_command 12 2024-04-14T15:21:49.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.022+0000 7f42beb44700 1 -- 172.21.15.90:0/1929515481 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f42b81c4b70 con 0x7f42b81bf5c0 2024-04-14T15:21:49.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.024+0000 7f42b5ffb700 1 -- 172.21.15.90:0/1929515481 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f42b81c4b70 con 0x7f42b81bf5c0 2024-04-14T15:21:49.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.024+0000 7f42b5ffb700 10 client.37617.objecter ms_dispatch 0x7f42b8001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:21:49.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.024+0000 7f42b5ffb700 10 client.37617.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:21:49.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.024+0000 7f42b5ffb700 20 client.37617.objecter put_session s=0x7f42b81b6e20 osd=5 4 2024-04-14T15:21:49.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.024+0000 7f42b5ffb700 15 client.37617.objecter _session_command_op_remove 5 12 2024-04-14T15:21:49.025 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:21:49.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.024+0000 7f42beb44700 10 client.37617.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:21:49.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.024+0000 7f42beb44700 15 client.37617.objecter _session_command_op_assign -1 13 2024-04-14T15:21:49.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.024+0000 7f42beb44700 20 client.37617.objecter _get_session s=0x7f42b81c4b70 osd=6 3 2024-04-14T15:21:49.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.024+0000 7f42beb44700 20 client.37617.objecter put_session s=0x7f42b81c4b70 osd=6 3 2024-04-14T15:21:49.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.024+0000 7f42beb44700 20 client.37617.objecter _get_session s=0x7f42b81c4b70 osd=6 3 2024-04-14T15:21:49.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.024+0000 7f42beb44700 15 client.37617.objecter _session_command_op_remove -1 13 2024-04-14T15:21:49.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.024+0000 7f42beb44700 20 client.37617.objecter get_session s=0x7f42b81c4b70 osd=6 3 2024-04-14T15:21:49.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.024+0000 7f42beb44700 15 client.37617.objecter _session_command_op_assign 6 13 2024-04-14T15:21:49.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.024+0000 7f42beb44700 20 client.37617.objecter put_session s=0x7f42b81c4b70 osd=6 4 2024-04-14T15:21:49.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.024+0000 7f42beb44700 10 client.37617.objecter _send_command 13 2024-04-14T15:21:49.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.024+0000 7f42beb44700 1 -- 172.21.15.90:0/1929515481 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f42b81c56f0 con 0x7f42b81c5290 2024-04-14T15:21:49.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.025+0000 7f42b5ffb700 10 client.37617.objecter ms_handle_connect 0x7f42b81c5290 2024-04-14T15:21:49.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.025+0000 7f42b5ffb700 1 -- 172.21.15.90:0/1929515481 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f42b81c56f0 con 0x7f42b81c5290 2024-04-14T15:21:49.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.025+0000 7f42b5ffb700 10 client.37617.objecter ms_dispatch 0x7f42b8001550 command_reply(tid 13: 0 ) 2024-04-14T15:21:49.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.025+0000 7f42b5ffb700 10 client.37617.objecter _finish_command 13 = system:0 2024-04-14T15:21:49.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.025+0000 7f42b5ffb700 20 client.37617.objecter put_session s=0x7f42b81c4b70 osd=6 4 2024-04-14T15:21:49.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.025+0000 7f42b5ffb700 15 client.37617.objecter _session_command_op_remove 6 13 2024-04-14T15:21:49.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.042+0000 7f42beb44700 10 client.37617.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:21:49.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.042+0000 7f42beb44700 15 client.37617.objecter _session_command_op_assign -1 14 2024-04-14T15:21:49.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.042+0000 7f42beb44700 20 client.37617.objecter _get_session s=0x7f42b81c4b70 osd=6 3 2024-04-14T15:21:49.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.042+0000 7f42beb44700 20 client.37617.objecter put_session s=0x7f42b81c4b70 osd=6 3 2024-04-14T15:21:49.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.042+0000 7f42beb44700 20 client.37617.objecter _get_session s=0x7f42b81c4b70 osd=6 3 2024-04-14T15:21:49.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.042+0000 7f42beb44700 15 client.37617.objecter _session_command_op_remove -1 14 2024-04-14T15:21:49.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.042+0000 7f42beb44700 20 client.37617.objecter get_session s=0x7f42b81c4b70 osd=6 3 2024-04-14T15:21:49.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.042+0000 7f42beb44700 15 client.37617.objecter _session_command_op_assign 6 14 2024-04-14T15:21:49.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.042+0000 7f42beb44700 20 client.37617.objecter put_session s=0x7f42b81c4b70 osd=6 4 2024-04-14T15:21:49.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.042+0000 7f42beb44700 10 client.37617.objecter _send_command 14 2024-04-14T15:21:49.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.042+0000 7f42beb44700 1 -- 172.21.15.90:0/1929515481 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f42b81c56f0 con 0x7f42b81c5290 2024-04-14T15:21:49.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.044+0000 7f42b5ffb700 1 -- 172.21.15.90:0/1929515481 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f42b81c56f0 con 0x7f42b81c5290 2024-04-14T15:21:49.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.044+0000 7f42b5ffb700 10 client.37617.objecter ms_dispatch 0x7f42b8001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:21:49.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.044+0000 7f42b5ffb700 10 client.37617.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:21:49.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.044+0000 7f42b5ffb700 20 client.37617.objecter put_session s=0x7f42b81c4b70 osd=6 4 2024-04-14T15:21:49.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.044+0000 7f42b5ffb700 15 client.37617.objecter _session_command_op_remove 6 14 2024-04-14T15:21:49.044 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:21:49.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.044+0000 7f42beb44700 10 client.37617.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:21:49.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.044+0000 7f42beb44700 15 client.37617.objecter _session_command_op_assign -1 15 2024-04-14T15:21:49.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.044+0000 7f42beb44700 20 client.37617.objecter _get_session s=0x7f42b81c56f0 osd=7 3 2024-04-14T15:21:49.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.044+0000 7f42beb44700 20 client.37617.objecter put_session s=0x7f42b81c56f0 osd=7 3 2024-04-14T15:21:49.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.044+0000 7f42beb44700 20 client.37617.objecter _get_session s=0x7f42b81c56f0 osd=7 3 2024-04-14T15:21:49.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.044+0000 7f42beb44700 15 client.37617.objecter _session_command_op_remove -1 15 2024-04-14T15:21:49.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.044+0000 7f42beb44700 20 client.37617.objecter get_session s=0x7f42b81c56f0 osd=7 3 2024-04-14T15:21:49.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.044+0000 7f42beb44700 15 client.37617.objecter _session_command_op_assign 7 15 2024-04-14T15:21:49.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.044+0000 7f42beb44700 20 client.37617.objecter put_session s=0x7f42b81c56f0 osd=7 4 2024-04-14T15:21:49.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.044+0000 7f42beb44700 10 client.37617.objecter _send_command 15 2024-04-14T15:21:49.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.044+0000 7f42beb44700 1 -- 172.21.15.90:0/1929515481 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f42b81c5af0 con 0x7f42b81c8280 2024-04-14T15:21:49.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.044+0000 7f42b5ffb700 10 client.37617.objecter ms_handle_connect 0x7f42b81c8280 2024-04-14T15:21:49.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.045+0000 7f42b5ffb700 1 -- 172.21.15.90:0/1929515481 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f42b81c5af0 con 0x7f42b81c8280 2024-04-14T15:21:49.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.045+0000 7f42b5ffb700 10 client.37617.objecter ms_dispatch 0x7f42b8001550 command_reply(tid 15: 0 ) 2024-04-14T15:21:49.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.045+0000 7f42b5ffb700 10 client.37617.objecter _finish_command 15 = system:0 2024-04-14T15:21:49.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.045+0000 7f42b5ffb700 20 client.37617.objecter put_session s=0x7f42b81c56f0 osd=7 4 2024-04-14T15:21:49.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.045+0000 7f42b5ffb700 15 client.37617.objecter _session_command_op_remove 7 15 2024-04-14T15:21:49.051 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:49.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.061+0000 7f42beb44700 10 client.37617.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:21:49.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.061+0000 7f42beb44700 15 client.37617.objecter _session_command_op_assign -1 16 2024-04-14T15:21:49.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.061+0000 7f42beb44700 20 client.37617.objecter _get_session s=0x7f42b81c56f0 osd=7 3 2024-04-14T15:21:49.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.061+0000 7f42beb44700 20 client.37617.objecter put_session s=0x7f42b81c56f0 osd=7 3 2024-04-14T15:21:49.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.061+0000 7f42beb44700 20 client.37617.objecter _get_session s=0x7f42b81c56f0 osd=7 3 2024-04-14T15:21:49.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.061+0000 7f42beb44700 15 client.37617.objecter _session_command_op_remove -1 16 2024-04-14T15:21:49.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.061+0000 7f42beb44700 20 client.37617.objecter get_session s=0x7f42b81c56f0 osd=7 3 2024-04-14T15:21:49.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.061+0000 7f42beb44700 15 client.37617.objecter _session_command_op_assign 7 16 2024-04-14T15:21:49.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.061+0000 7f42beb44700 20 client.37617.objecter put_session s=0x7f42b81c56f0 osd=7 4 2024-04-14T15:21:49.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.061+0000 7f42beb44700 10 client.37617.objecter _send_command 16 2024-04-14T15:21:49.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.061+0000 7f42beb44700 1 -- 172.21.15.90:0/1929515481 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f42b81cbd90 con 0x7f42b81c8280 2024-04-14T15:21:49.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.063+0000 7f42b5ffb700 1 -- 172.21.15.90:0/1929515481 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f42b81cbd90 con 0x7f42b81c8280 2024-04-14T15:21:49.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.063+0000 7f42b5ffb700 10 client.37617.objecter ms_dispatch 0x7f42b8001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:21:49.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.063+0000 7f42b5ffb700 10 client.37617.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:21:49.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.063+0000 7f42b5ffb700 20 client.37617.objecter put_session s=0x7f42b81c56f0 osd=7 4 2024-04-14T15:21:49.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.063+0000 7f42b5ffb700 15 client.37617.objecter _session_command_op_remove 7 16 2024-04-14T15:21:49.065 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:21:49.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.064+0000 7f42beb44700 10 librados: watch_flush enter 2024-04-14T15:21:49.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.064+0000 7f42beb44700 10 librados: watch_flush exit 2024-04-14T15:21:49.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.064+0000 7f42beb44700 10 client.37617.objecter close_session for osd.0 2024-04-14T15:21:49.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.064+0000 7f42beb44700 1 -- 172.21.15.90:0/1929515481 >> v1:172.21.15.90:6801/3761806885 conn(0x7f42b81ab4a0 legacy=0x7f42b81ab880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:49.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.064+0000 7f42beb44700 20 client.37617.objecter put_session s=0x7f42b80693d0 osd=0 1 2024-04-14T15:21:49.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.064+0000 7f42beb44700 10 client.37617.objecter close_session for osd.1 2024-04-14T15:21:49.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.064+0000 7f42beb44700 1 -- 172.21.15.90:0/1929515481 >> v1:172.21.15.90:6805/4170361699 conn(0x7f42b80636c0 legacy=0x7f42b810f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:49.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.064+0000 7f42beb44700 20 client.37617.objecter put_session s=0x7f42b81ad030 osd=1 1 2024-04-14T15:21:49.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.064+0000 7f42beb44700 10 client.37617.objecter close_session for osd.2 2024-04-14T15:21:49.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.064+0000 7f42beb44700 1 -- 172.21.15.90:0/1929515481 >> v1:172.21.15.90:6809/1686184251 conn(0x7f42b80632b0 legacy=0x7f42b81b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:49.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.064+0000 7f42bd0e1700 1 -- 172.21.15.90:0/1929515481 reap_dead start 2024-04-14T15:21:49.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.064+0000 7f42beb44700 20 client.37617.objecter put_session s=0x7f42b8063140 osd=2 1 2024-04-14T15:21:49.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.064+0000 7f42beb44700 10 client.37617.objecter close_session for osd.3 2024-04-14T15:21:49.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.064+0000 7f42beb44700 1 -- 172.21.15.90:0/1929515481 >> v1:172.21.15.90:6813/2270166774 conn(0x7f42b802d1e0 legacy=0x7f42b81b90e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:49.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.064+0000 7f42beb44700 20 client.37617.objecter put_session s=0x7f42b802d070 osd=3 1 2024-04-14T15:21:49.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.064+0000 7f42beb44700 10 client.37617.objecter close_session for osd.4 2024-04-14T15:21:49.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.064+0000 7f42beb44700 1 -- 172.21.15.90:0/1929515481 >> v1:172.21.15.92:6800/3887549733 conn(0x7f42b81b6720 legacy=0x7f42b81bd990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:49.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.064+0000 7f42beb44700 20 client.37617.objecter put_session s=0x7f42b81b2480 osd=4 1 2024-04-14T15:21:49.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.064+0000 7f42beb44700 10 client.37617.objecter close_session for osd.5 2024-04-14T15:21:49.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.064+0000 7f42beb44700 1 -- 172.21.15.90:0/1929515481 >> v1:172.21.15.92:6804/3244561440 conn(0x7f42b81bf5c0 legacy=0x7f42b81c19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:49.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.064+0000 7f42beb44700 20 client.37617.objecter put_session s=0x7f42b81b6e20 osd=5 1 2024-04-14T15:21:49.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.064+0000 7f42beb44700 10 client.37617.objecter close_session for osd.6 2024-04-14T15:21:49.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.064+0000 7f42beb44700 1 -- 172.21.15.90:0/1929515481 >> v1:172.21.15.92:6808/636125963 conn(0x7f42b81c5290 legacy=0x7f42b81c33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:49.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.064+0000 7f42beb44700 20 client.37617.objecter put_session s=0x7f42b81c4b70 osd=6 1 2024-04-14T15:21:49.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.064+0000 7f42beb44700 10 client.37617.objecter close_session for osd.7 2024-04-14T15:21:49.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.064+0000 7f42beb44700 1 -- 172.21.15.90:0/1929515481 >> v1:172.21.15.92:6812/1773706716 conn(0x7f42b81c8280 legacy=0x7f42b81ca670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:49.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.064+0000 7f42beb44700 20 client.37617.objecter put_session s=0x7f42b81c56f0 osd=7 1 2024-04-14T15:21:49.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.064+0000 7f42beb44700 20 client.37617.objecter shutdown clearing up homeless session... 2024-04-14T15:21:49.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.064+0000 7f42beb44700 10 client.37617.objecter successfully canceled tick 2024-04-14T15:21:49.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.064+0000 7f42bd0e1700 1 -- 172.21.15.90:0/1929515481 reap_dead start 2024-04-14T15:21:49.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.064+0000 7f42beb44700 1 -- 172.21.15.90:0/1929515481 >> v1:172.21.15.90:6800/420935427 conn(0x7f42a4071340 legacy=0x7f42a4073800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:49.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.064+0000 7f42beb44700 1 -- 172.21.15.90:0/1929515481 >> v1:172.21.15.90:6789/0 conn(0x7f42b810f310 legacy=0x7f42b81a82e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:49.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.064+0000 7f42beb44700 1 -- 172.21.15.90:0/1929515481 shutdown_connections 2024-04-14T15:21:49.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.064+0000 7f42beb44700 1 -- 172.21.15.90:0/1929515481 >> 172.21.15.90:0/1929515481 conn(0x7f42b80fe380 msgr2=0x7f42b8101430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:21:49.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.064+0000 7f42beb44700 1 -- 172.21.15.90:0/1929515481 shutdown_connections 2024-04-14T15:21:49.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.064+0000 7f42beb44700 1 -- 172.21.15.90:0/1929515481 wait complete. 2024-04-14T15:21:49.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:49.064+0000 7f42beb44700 1 librados: shutdown 2024-04-14T15:21:49.069 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:21:49.069 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:21:49.069 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:21:49.069 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:21:49.069 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:21:49.069 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:21:49.069 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:21:49.069 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:21:49.075 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:49.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:49.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:49.177 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:49.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:49 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-04-14T15:21:49.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:49 smithi090 ceph-mon[41670]: osdmap e572: 8 total, 8 up, 7 in 2024-04-14T15:21:49.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:49 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1929515481' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:21:49.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:49 smithi090 ceph-mon[41670]: osdmap e573: 8 total, 8 up, 7 in 2024-04-14T15:21:49.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:49 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-04-14T15:21:49.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:49 smithi090 ceph-mon[30399]: osdmap e572: 8 total, 8 up, 7 in 2024-04-14T15:21:49.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:49 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1929515481' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:21:49.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:49 smithi090 ceph-mon[30399]: osdmap e573: 8 total, 8 up, 7 in 2024-04-14T15:21:49.201 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:49.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:49.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:49.303 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:21:49.328 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:49.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:49.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:21:49.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:49 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-04-14T15:21:49.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:49 smithi092 ceph-mon[33966]: osdmap e572: 8 total, 8 up, 7 in 2024-04-14T15:21:49.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:49 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1929515481' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:21:49.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:49 smithi092 ceph-mon[33966]: osdmap e573: 8 total, 8 up, 7 in 2024-04-14T15:21:49.430 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:21:49.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:49.456 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:49.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:21:49.558 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:49.584 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:49.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:49.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:49.686 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:21:49.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:49.715 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:49.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:21:49.816 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:21:49.842 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:49.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:49.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:21:49.945 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:21:49.971 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:49.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:49.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:21:50.074 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:50.108 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:50.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:50.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:50.210 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:50.236 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:50.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:50.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:50.253 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:50 smithi090 ceph-mon[41670]: pgmap v840: 248 pgs: 1 peering, 2 unknown, 245 active+clean; 8.4 MiB data, 507 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:21:50.254 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:50 smithi090 ceph-mon[41670]: osdmap e574: 8 total, 8 up, 7 in 2024-04-14T15:21:50.254 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:50 smithi090 ceph-mon[30399]: pgmap v840: 248 pgs: 1 peering, 2 unknown, 245 active+clean; 8.4 MiB data, 507 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:21:50.254 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:50 smithi090 ceph-mon[30399]: osdmap e574: 8 total, 8 up, 7 in 2024-04-14T15:21:50.337 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:21:50.363 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:50.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:50.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:21:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:50 smithi092 ceph-mon[33966]: pgmap v840: 248 pgs: 1 peering, 2 unknown, 245 active+clean; 8.4 MiB data, 507 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:21:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:50 smithi092 ceph-mon[33966]: osdmap e574: 8 total, 8 up, 7 in 2024-04-14T15:21:50.465 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:50.491 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:50.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:50.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:50.593 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:21:50.617 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:50.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:50.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:21:50.719 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:21:50.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:50.746 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:50.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:21:50.847 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:50.872 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:50.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:50.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:50.973 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:21:50.998 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:50.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:50.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:21:51.099 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:21:51.123 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:51.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:51.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:21:51.224 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:21:51.248 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:51.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:51.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:21:51.350 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:21:51.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:51 smithi092 ceph-mon[33966]: 1.b starting backfill to osd.4 from (0'0,0'0] MAX to 91'130 2024-04-14T15:21:51.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:51 smithi092 ceph-mon[33966]: 1.c starting backfill to osd.5 from (0'0,0'0] MAX to 91'130 2024-04-14T15:21:51.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:51 smithi092 ceph-mon[33966]: 1.c starting backfill to osd.7 from (0'0,0'0] MAX to 91'130 2024-04-14T15:21:51.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:51 smithi092 ceph-mon[33966]: 1.d starting backfill to osd.2 from (0'0,0'0] MAX to 91'130 2024-04-14T15:21:51.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:51 smithi092 ceph-mon[33966]: 1.d starting backfill to osd.5 from (0'0,0'0] MAX to 91'130 2024-04-14T15:21:51.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:51 smithi092 ceph-mon[33966]: 1.e starting backfill to osd.0 from (0'0,0'0] MAX to 91'130 2024-04-14T15:21:51.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:51 smithi092 ceph-mon[33966]: osdmap e575: 8 total, 8 up, 7 in 2024-04-14T15:21:51.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:51.375 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:51.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:21:51.399 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:21:51.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:51 smithi090 ceph-mon[41670]: 1.b starting backfill to osd.4 from (0'0,0'0] MAX to 91'130 2024-04-14T15:21:51.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:51 smithi090 ceph-mon[41670]: 1.c starting backfill to osd.5 from (0'0,0'0] MAX to 91'130 2024-04-14T15:21:51.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:51 smithi090 ceph-mon[41670]: 1.c starting backfill to osd.7 from (0'0,0'0] MAX to 91'130 2024-04-14T15:21:51.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:51 smithi090 ceph-mon[41670]: 1.d starting backfill to osd.2 from (0'0,0'0] MAX to 91'130 2024-04-14T15:21:51.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:51 smithi090 ceph-mon[41670]: 1.d starting backfill to osd.5 from (0'0,0'0] MAX to 91'130 2024-04-14T15:21:51.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:51 smithi090 ceph-mon[41670]: 1.e starting backfill to osd.0 from (0'0,0'0] MAX to 91'130 2024-04-14T15:21:51.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:51 smithi090 ceph-mon[41670]: osdmap e575: 8 total, 8 up, 7 in 2024-04-14T15:21:51.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:51 smithi090 ceph-mon[30399]: 1.b starting backfill to osd.4 from (0'0,0'0] MAX to 91'130 2024-04-14T15:21:51.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:51 smithi090 ceph-mon[30399]: 1.c starting backfill to osd.5 from (0'0,0'0] MAX to 91'130 2024-04-14T15:21:51.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:51 smithi090 ceph-mon[30399]: 1.c starting backfill to osd.7 from (0'0,0'0] MAX to 91'130 2024-04-14T15:21:51.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:51 smithi090 ceph-mon[30399]: 1.d starting backfill to osd.2 from (0'0,0'0] MAX to 91'130 2024-04-14T15:21:51.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:51 smithi090 ceph-mon[30399]: 1.d starting backfill to osd.5 from (0'0,0'0] MAX to 91'130 2024-04-14T15:21:51.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:51 smithi090 ceph-mon[30399]: 1.e starting backfill to osd.0 from (0'0,0'0] MAX to 91'130 2024-04-14T15:21:51.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:51 smithi090 ceph-mon[30399]: osdmap e575: 8 total, 8 up, 7 in 2024-04-14T15:21:51.476 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:21:51.504 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:51.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:51.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:21:51.606 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:21:51.634 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:51.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:51.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:21:51.736 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:51.763 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:51.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:51.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:51.865 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:21:51.892 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:51.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:51.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:21:51.993 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:21:52.018 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:52.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:52.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:21:52.120 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:21:52.143 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:52.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:52.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:21:52.245 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:21:52.268 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:52.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:52.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:21:52.287 INFO:journalctl@ceph.osd.5.smithi092.stdout:Apr 14 15:21:52 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-5[45851]: 2024-04-14T15:21:52.024+0000 7f20d8fc4700 -1 snap_mapper.add_oid found existing snaps mapped on 110:d236bce0:test-rados-api-smithi090-93652-97::foo:21, removing 2024-04-14T15:21:52.288 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:52 smithi092 ceph-mon[33966]: pgmap v843: 248 pgs: 7 remapped+peering, 1 peering, 2 unknown, 238 active+clean; 8.4 MiB data, 507 MiB used, 625 GiB / 626 GiB avail; 3.0 KiB/s rd, 14 KiB/s wr, 3 op/s; 9.8 KiB/s, 0 objects/s recovering 2024-04-14T15:21:52.288 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:52 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:21:52.288 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:52 smithi092 ceph-mon[33966]: osdmap e576: 8 total, 8 up, 7 in 2024-04-14T15:21:52.288 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:52 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:21:52.288 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:52 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:21:52.288 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:52 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-96"}]: dispatch 2024-04-14T15:21:52.288 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:52 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-96"}]: dispatch 2024-04-14T15:21:52.294 INFO:journalctl@ceph.osd.2.smithi090.stdout:Apr 14 15:21:52 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-2[58450]: 2024-04-14T15:21:52.024+0000 7f791103e700 -1 snap_mapper.add_oid found existing snaps mapped on 110:d236bce0:test-rados-api-smithi090-93652-97::foo:21, removing 2024-04-14T15:21:52.294 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:52 smithi090 ceph-mon[41670]: pgmap v843: 248 pgs: 7 remapped+peering, 1 peering, 2 unknown, 238 active+clean; 8.4 MiB data, 507 MiB used, 625 GiB / 626 GiB avail; 3.0 KiB/s rd, 14 KiB/s wr, 3 op/s; 9.8 KiB/s, 0 objects/s recovering 2024-04-14T15:21:52.295 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:52 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:21:52.295 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:52 smithi090 ceph-mon[41670]: osdmap e576: 8 total, 8 up, 7 in 2024-04-14T15:21:52.295 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:52 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:21:52.295 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:52 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:21:52.295 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:52 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-96"}]: dispatch 2024-04-14T15:21:52.295 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:52 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-96"}]: dispatch 2024-04-14T15:21:52.295 INFO:journalctl@ceph.osd.1.smithi090.stdout:Apr 14 15:21:52 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-1[52165]: 2024-04-14T15:21:52.024+0000 7f18a8097700 -1 snap_mapper.add_oid found existing snaps mapped on 110:d236bce0:test-rados-api-smithi090-93652-97::foo:21, removing 2024-04-14T15:21:52.295 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:52 smithi090 ceph-mon[30399]: pgmap v843: 248 pgs: 7 remapped+peering, 1 peering, 2 unknown, 238 active+clean; 8.4 MiB data, 507 MiB used, 625 GiB / 626 GiB avail; 3.0 KiB/s rd, 14 KiB/s wr, 3 op/s; 9.8 KiB/s, 0 objects/s recovering 2024-04-14T15:21:52.295 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:52 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:21:52.296 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:52 smithi090 ceph-mon[30399]: osdmap e576: 8 total, 8 up, 7 in 2024-04-14T15:21:52.296 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:52 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:21:52.296 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:52 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:21:52.296 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:52 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-96"}]: dispatch 2024-04-14T15:21:52.296 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:52 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-96"}]: dispatch 2024-04-14T15:21:52.370 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:21:52.395 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:52.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:52.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:21:52.496 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:21:52.519 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:52.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:52.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:21:52.621 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:21:52.645 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:52.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:52.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:21:52.747 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:21:52.770 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:52.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:52.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:21:52.872 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:21:52.895 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:52.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:52.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:21:52.997 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:21:53.020 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:53.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:53.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:21:53.122 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:21:53.145 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:53.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:53.146 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:21:53.247 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:21:53.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:53.271 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:53.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:21:53.351 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:53 smithi090 ceph-mon[41670]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-04-14T15:21:53.352 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:53 smithi090 ceph-mon[41670]: osdmap e577: 8 total, 8 up, 7 in 2024-04-14T15:21:53.352 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:53 smithi090 ceph-mon[30399]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-04-14T15:21:53.352 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:53 smithi090 ceph-mon[30399]: osdmap e577: 8 total, 8 up, 7 in 2024-04-14T15:21:53.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:53 smithi092 ceph-mon[33966]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-04-14T15:21:53.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:53 smithi092 ceph-mon[33966]: osdmap e577: 8 total, 8 up, 7 in 2024-04-14T15:21:53.372 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:21:53.373 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-14T15:21:53.373 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-14T15:21:53.373 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-14T15:21:53.373 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-14T15:21:53.373 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.373 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24928.0:2 24.1a 24:5c6776db:::0.obj:head [write 0~103] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-04-14T15:21:53.373 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:12:55.060209+0000", 2024-04-14T15:21:53.374 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 538.31103593199998, 2024-04-14T15:21:53.374 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.508190039, 2024-04-14T15:21:53.374 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:21:53.374 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:21:53.374 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:21:53.374 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:21:53.374 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24928", 2024-04-14T15:21:53.374 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/346751907", 2024-04-14T15:21:53.374 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2 2024-04-14T15:21:53.374 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.374 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:21:53.374 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.374 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:21:53.375 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.060209+0000", 2024-04-14T15:21:53.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:21:53.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.375 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.375 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:21:53.375 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.060209+0000", 2024-04-14T15:21:53.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:21:53.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.375 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.375 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:21:53.375 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.060211+0000", 2024-04-14T15:21:53.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0090000000000002e-06 2024-04-14T15:21:53.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.375 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.375 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:21:53.376 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.060220+0000", 2024-04-14T15:21:53.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7870000000000007e-06 2024-04-14T15:21:53.376 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.376 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.376 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:21:53.376 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.060231+0000", 2024-04-14T15:21:53.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1317e-05 2024-04-14T15:21:53.376 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.376 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.376 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:21:53.376 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.060243+0000", 2024-04-14T15:21:53.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.13e-05 2024-04-14T15:21:53.376 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.376 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.376 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:21:53.377 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.107072+0000", 2024-04-14T15:21:53.377 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.046829279000000001 2024-04-14T15:21:53.377 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.377 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.377 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:21:53.377 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.107080+0000", 2024-04-14T15:21:53.377 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4400000000000005e-06 2024-04-14T15:21:53.377 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.377 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.377 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:21:53.377 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.502476+0000", 2024-04-14T15:21:53.377 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.395395623 2024-04-14T15:21:53.377 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.377 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.378 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:21:53.378 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.502603+0000", 2024-04-14T15:21:53.378 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000127508 2024-04-14T15:21:53.378 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.378 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.378 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,3", 2024-04-14T15:21:53.378 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.502746+0000", 2024-04-14T15:21:53.378 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000142817 2024-04-14T15:21:53.378 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.378 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.378 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:21:53.378 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.514091+0000", 2024-04-14T15:21:53.378 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011344468999999999 2024-04-14T15:21:53.378 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.378 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.379 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:21:53.379 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.559343+0000", 2024-04-14T15:21:53.379 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.045251922999999999 2024-04-14T15:21:53.379 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.379 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.379 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:21:53.379 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.568307+0000", 2024-04-14T15:21:53.379 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0089640250000000005 2024-04-14T15:21:53.379 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.379 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.379 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:21:53.379 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.568354+0000", 2024-04-14T15:21:53.379 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7246000000000001e-05 2024-04-14T15:21:53.379 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.379 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.380 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:21:53.380 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.568399+0000", 2024-04-14T15:21:53.380 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5296000000000001e-05 2024-04-14T15:21:53.380 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:53.380 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:53.380 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:53.380 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.380 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.380 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24965.0:2 30.1a 30:5c6776db:::0.obj:head [write 0~103] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-04-14T15:21:53.380 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:12:55.063803+0000", 2024-04-14T15:21:53.380 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 538.30744252099998, 2024-04-14T15:21:53.380 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4453814549999999, 2024-04-14T15:21:53.380 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:21:53.380 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:21:53.380 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:21:53.380 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:21:53.381 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24965", 2024-04-14T15:21:53.381 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1840281572", 2024-04-14T15:21:53.381 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2 2024-04-14T15:21:53.381 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.381 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:21:53.381 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.381 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:21:53.381 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.063803+0000", 2024-04-14T15:21:53.381 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:21:53.381 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.381 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.381 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:21:53.381 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.063803+0000", 2024-04-14T15:21:53.381 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:21:53.381 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.381 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.382 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:21:53.382 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.063805+0000", 2024-04-14T15:21:53.382 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4990000000000001e-06 2024-04-14T15:21:53.382 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.382 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.382 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:21:53.382 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.063813+0000", 2024-04-14T15:21:53.382 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0299999999999994e-06 2024-04-14T15:21:53.382 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.382 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.382 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:21:53.382 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.063826+0000", 2024-04-14T15:21:53.382 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2480000000000001e-05 2024-04-14T15:21:53.382 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.383 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.383 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:21:53.383 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.063837+0000", 2024-04-14T15:21:53.383 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.119e-05 2024-04-14T15:21:53.383 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.383 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.383 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:21:53.383 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.120883+0000", 2024-04-14T15:21:53.383 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.057046615000000002 2024-04-14T15:21:53.383 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.383 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.383 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:21:53.383 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:55.120892+0000", 2024-04-14T15:21:53.383 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2530000000000004e-06 2024-04-14T15:21:53.383 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.383 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.383 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:21:53.384 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.446585+0000", 2024-04-14T15:21:53.384 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.325693403 2024-04-14T15:21:53.384 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.384 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.384 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:21:53.384 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.446706+0000", 2024-04-14T15:21:53.384 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000121396 2024-04-14T15:21:53.384 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.384 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.384 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,5", 2024-04-14T15:21:53.384 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.446847+0000", 2024-04-14T15:21:53.384 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014026300000000001 2024-04-14T15:21:53.384 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.384 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.384 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:21:53.384 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.479690+0000", 2024-04-14T15:21:53.385 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.032843226000000003 2024-04-14T15:21:53.385 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.385 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.385 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:21:53.385 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.493251+0000", 2024-04-14T15:21:53.385 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013560588 2024-04-14T15:21:53.385 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.385 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.385 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:21:53.385 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.509099+0000", 2024-04-14T15:21:53.385 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015848240999999999 2024-04-14T15:21:53.385 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.385 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.385 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:21:53.385 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.509147+0000", 2024-04-14T15:21:53.385 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8028999999999997e-05 2024-04-14T15:21:53.385 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.386 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.386 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:21:53.386 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.509184+0000", 2024-04-14T15:21:53.386 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7242e-05 2024-04-14T15:21:53.386 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:53.386 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:53.386 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:53.386 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.386 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.386 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24860.0:2 21.d 21:b2a9e82d:test-rados-api-smithi090-93864-2::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-04-14T15:21:53.386 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:12:56.070980+0000", 2024-04-14T15:21:53.386 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 537.30026507900004, 2024-04-14T15:21:53.386 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.42151502099999999, 2024-04-14T15:21:53.387 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:21:53.387 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:21:53.387 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:21:53.387 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:21:53.387 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24860", 2024-04-14T15:21:53.387 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/766142007", 2024-04-14T15:21:53.387 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2 2024-04-14T15:21:53.387 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.387 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:21:53.387 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.387 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:21:53.387 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.070980+0000", 2024-04-14T15:21:53.387 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:21:53.387 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.387 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.388 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:21:53.388 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.070980+0000", 2024-04-14T15:21:53.388 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:21:53.388 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.388 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.388 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:21:53.388 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.070982+0000", 2024-04-14T15:21:53.388 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7099999999999999e-06 2024-04-14T15:21:53.388 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.388 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.388 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:21:53.388 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.071000+0000", 2024-04-14T15:21:53.388 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8391999999999999e-05 2024-04-14T15:21:53.388 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.388 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.388 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:21:53.389 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.071010+0000", 2024-04-14T15:21:53.389 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4020000000000008e-06 2024-04-14T15:21:53.389 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.389 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.389 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:21:53.389 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.071027+0000", 2024-04-14T15:21:53.389 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7200000000000001e-05 2024-04-14T15:21:53.389 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.389 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.389 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:21:53.389 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.187255+0000", 2024-04-14T15:21:53.389 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.116227975 2024-04-14T15:21:53.389 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.389 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.389 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:21:53.390 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.187264+0000", 2024-04-14T15:21:53.390 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0750000000000004e-06 2024-04-14T15:21:53.390 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.390 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.390 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:21:53.390 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.464111+0000", 2024-04-14T15:21:53.390 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.27684678200000001 2024-04-14T15:21:53.390 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.390 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.390 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:21:53.390 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.464239+0000", 2024-04-14T15:21:53.390 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012847999999999999 2024-04-14T15:21:53.390 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.391 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.391 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,6", 2024-04-14T15:21:53.391 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.464380+0000", 2024-04-14T15:21:53.391 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000141209 2024-04-14T15:21:53.391 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.391 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.391 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:21:53.391 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.469522+0000", 2024-04-14T15:21:53.391 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0051416109999999999 2024-04-14T15:21:53.391 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.391 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.391 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:21:53.391 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.473583+0000", 2024-04-14T15:21:53.391 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040609499999999998 2024-04-14T15:21:53.392 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.392 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.392 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:21:53.392 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.492397+0000", 2024-04-14T15:21:53.392 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.018814000000000001 2024-04-14T15:21:53.392 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.392 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.392 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:21:53.392 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.492454+0000", 2024-04-14T15:21:53.392 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7510000000000003e-05 2024-04-14T15:21:53.392 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.392 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.392 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:21:53.392 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.492495+0000", 2024-04-14T15:21:53.392 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0725000000000001e-05 2024-04-14T15:21:53.392 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:53.393 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:53.393 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:53.393 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.393 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.393 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.14955.0:2 19.1 19:8584e8fe:test-rados-api-smithi090-93731-2::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-04-14T15:21:53.393 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:12:56.109199+0000", 2024-04-14T15:21:53.393 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 537.26204588999997, 2024-04-14T15:21:53.393 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.44945317499999998, 2024-04-14T15:21:53.393 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:21:53.393 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:21:53.393 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:21:53.393 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:21:53.393 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.14955", 2024-04-14T15:21:53.393 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2763816171", 2024-04-14T15:21:53.394 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2 2024-04-14T15:21:53.394 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.394 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:21:53.394 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.394 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:21:53.394 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.109199+0000", 2024-04-14T15:21:53.394 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:21:53.394 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.394 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.394 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:21:53.394 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.109199+0000", 2024-04-14T15:21:53.394 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:21:53.394 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.395 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.395 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:21:53.395 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.109201+0000", 2024-04-14T15:21:53.395 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8679999999999999e-06 2024-04-14T15:21:53.395 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.395 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.395 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:21:53.395 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.109209+0000", 2024-04-14T15:21:53.395 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1820000000000006e-06 2024-04-14T15:21:53.395 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.395 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.395 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:21:53.395 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.109223+0000", 2024-04-14T15:21:53.396 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4149e-05 2024-04-14T15:21:53.396 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.396 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.396 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:21:53.396 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.109234+0000", 2024-04-14T15:21:53.396 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0883999999999999e-05 2024-04-14T15:21:53.396 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.396 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.396 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:21:53.396 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.109301+0000", 2024-04-14T15:21:53.396 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7136000000000002e-05 2024-04-14T15:21:53.396 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.396 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.396 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:21:53.396 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.109311+0000", 2024-04-14T15:21:53.397 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4269999999999997e-06 2024-04-14T15:21:53.397 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.397 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.397 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:21:53.397 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.476527+0000", 2024-04-14T15:21:53.397 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.36721647699999999 2024-04-14T15:21:53.397 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.397 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.397 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:21:53.397 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.476630+0000", 2024-04-14T15:21:53.397 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000102748 2024-04-14T15:21:53.397 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.397 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.397 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,7", 2024-04-14T15:21:53.397 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.476756+0000", 2024-04-14T15:21:53.398 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000125489 2024-04-14T15:21:53.398 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.398 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.398 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:21:53.398 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.493558+0000", 2024-04-14T15:21:53.398 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016802699000000001 2024-04-14T15:21:53.398 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.398 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.398 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:21:53.398 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.495245+0000", 2024-04-14T15:21:53.398 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016863609999999999 2024-04-14T15:21:53.398 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.398 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.398 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:21:53.398 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.558554+0000", 2024-04-14T15:21:53.399 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.063309655000000006 2024-04-14T15:21:53.399 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.399 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.399 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:21:53.399 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.558606+0000", 2024-04-14T15:21:53.399 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2160000000000002e-05 2024-04-14T15:21:53.400 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.400 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.400 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:21:53.400 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.558652+0000", 2024-04-14T15:21:53.400 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5939999999999997e-05 2024-04-14T15:21:53.401 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:53.401 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:53.401 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:53.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.401 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.401 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.25030.0:2 35.6 e108/107 35:602f83fe:::foo:head v 108'1, mlcod=0'0)", 2024-04-14T15:21:53.401 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:12:56.452112+0000", 2024-04-14T15:21:53.401 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 536.91913350499999, 2024-04-14T15:21:53.401 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.061498231, 2024-04-14T15:21:53.401 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:21:53.401 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:21:53.401 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:21:53.401 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:21:53.401 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.402 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:21:53.402 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.452112+0000", 2024-04-14T15:21:53.402 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:21:53.402 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.402 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.402 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:21:53.402 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.452112+0000", 2024-04-14T15:21:53.402 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:21:53.402 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.402 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.402 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:21:53.402 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.452112+0000", 2024-04-14T15:21:53.402 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5100000000000001e-07 2024-04-14T15:21:53.402 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.403 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.403 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:21:53.403 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.452118+0000", 2024-04-14T15:21:53.403 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7339999999999996e-06 2024-04-14T15:21:53.403 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.403 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.403 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:21:53.403 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.452123+0000", 2024-04-14T15:21:53.403 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1379999999999998e-06 2024-04-14T15:21:53.403 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.403 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.403 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:21:53.403 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.452131+0000", 2024-04-14T15:21:53.404 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4290000000000001e-06 2024-04-14T15:21:53.404 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.404 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.404 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:21:53.404 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.497771+0000", 2024-04-14T15:21:53.404 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.045639800000000001 2024-04-14T15:21:53.404 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.404 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.404 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:21:53.404 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.497790+0000", 2024-04-14T15:21:53.404 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9224000000000001e-05 2024-04-14T15:21:53.404 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.404 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.404 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:21:53.404 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.513569+0000", 2024-04-14T15:21:53.404 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015779202999999999 2024-04-14T15:21:53.405 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.405 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.405 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:21:53.405 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.513610+0000", 2024-04-14T15:21:53.405 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0451999999999998e-05 2024-04-14T15:21:53.405 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:53.405 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:53.405 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:53.405 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.405 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.405 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24955.0:3 26.f 26:f7483bc4:::src_object.2:head [writefull 0~4096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-04-14T15:21:53.405 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:12:56.472685+0000", 2024-04-14T15:21:53.405 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 536.89855996999995, 2024-04-14T15:21:53.405 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.086134901999999999, 2024-04-14T15:21:53.405 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:21:53.406 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:21:53.406 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:21:53.406 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:21:53.406 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24955", 2024-04-14T15:21:53.406 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/117387996", 2024-04-14T15:21:53.406 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3 2024-04-14T15:21:53.406 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.406 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:21:53.406 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.406 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:21:53.406 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.472685+0000", 2024-04-14T15:21:53.406 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:21:53.406 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.407 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.407 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:21:53.407 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.472685+0000", 2024-04-14T15:21:53.407 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:21:53.407 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.407 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.407 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:21:53.407 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.472687+0000", 2024-04-14T15:21:53.407 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8300000000000001e-06 2024-04-14T15:21:53.407 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.407 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.407 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:21:53.407 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.472697+0000", 2024-04-14T15:21:53.407 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8190000000000003e-06 2024-04-14T15:21:53.408 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.408 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.408 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:21:53.408 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.472705+0000", 2024-04-14T15:21:53.408 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8299999999999996e-06 2024-04-14T15:21:53.408 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.408 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.408 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:21:53.408 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.472711+0000", 2024-04-14T15:21:53.408 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5409999999999997e-06 2024-04-14T15:21:53.408 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.408 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.408 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:21:53.408 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.475204+0000", 2024-04-14T15:21:53.408 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0024926380000000001 2024-04-14T15:21:53.408 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.409 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.409 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:21:53.409 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.475324+0000", 2024-04-14T15:21:53.409 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000119669 2024-04-14T15:21:53.409 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.409 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.409 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,7", 2024-04-14T15:21:53.409 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.475458+0000", 2024-04-14T15:21:53.409 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013436200000000001 2024-04-14T15:21:53.409 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.409 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.409 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:21:53.409 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.486202+0000", 2024-04-14T15:21:53.409 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010744448 2024-04-14T15:21:53.409 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.409 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.409 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:21:53.410 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.494871+0000", 2024-04-14T15:21:53.410 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0086686590000000004 2024-04-14T15:21:53.410 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.410 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.410 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:21:53.410 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.558724+0000", 2024-04-14T15:21:53.410 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.063852781999999997 2024-04-14T15:21:53.410 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.410 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.410 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:21:53.410 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.558777+0000", 2024-04-14T15:21:53.410 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2805e-05 2024-04-14T15:21:53.410 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.410 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.411 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:21:53.411 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.558820+0000", 2024-04-14T15:21:53.411 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3519000000000001e-05 2024-04-14T15:21:53.411 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:53.411 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:53.411 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:53.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.411 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.411 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.24853.0:433 16.1b e108/107 16:dbd443e1:test-rados-api-smithi090-93601-15::foo:head v 108'194, mlcod=108'192)", 2024-04-14T15:21:53.411 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:12:57.109500+0000", 2024-04-14T15:21:53.411 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 536.26174562699998, 2024-04-14T15:21:53.411 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.097327853000000006, 2024-04-14T15:21:53.411 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:21:53.411 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:21:53.411 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:21:53.412 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:21:53.412 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.412 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:21:53.412 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.109500+0000", 2024-04-14T15:21:53.412 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:21:53.412 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.412 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.412 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:21:53.412 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.109500+0000", 2024-04-14T15:21:53.412 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:21:53.412 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.412 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.412 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:21:53.412 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.109500+0000", 2024-04-14T15:21:53.412 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1199999999999999e-07 2024-04-14T15:21:53.412 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.413 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.413 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:21:53.413 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.109506+0000", 2024-04-14T15:21:53.413 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0179999999999998e-06 2024-04-14T15:21:53.413 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.413 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.413 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:21:53.413 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.109511+0000", 2024-04-14T15:21:53.413 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7609999999999998e-06 2024-04-14T15:21:53.413 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.413 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.413 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:21:53.413 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.109520+0000", 2024-04-14T15:21:53.413 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9449999999999999e-06 2024-04-14T15:21:53.414 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.414 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.414 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:21:53.414 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.109898+0000", 2024-04-14T15:21:53.414 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037795899999999998 2024-04-14T15:21:53.414 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.414 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.414 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:21:53.414 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.109922+0000", 2024-04-14T15:21:53.414 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4073000000000001e-05 2024-04-14T15:21:53.414 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.414 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.414 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:21:53.414 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.206793+0000", 2024-04-14T15:21:53.415 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.096871041000000005 2024-04-14T15:21:53.415 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.415 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.415 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:21:53.415 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.206827+0000", 2024-04-14T15:21:53.415 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4743999999999998e-05 2024-04-14T15:21:53.415 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:53.415 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:53.415 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:53.415 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.415 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.415 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.14973.0:1 42.1s0 42:8981422e:test-rados-api-smithi090-93998-2::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-04-14T15:21:53.415 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:12:57.328942+0000", 2024-04-14T15:21:53.417 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 536.04230347800001, 2024-04-14T15:21:53.417 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.87141143300000001, 2024-04-14T15:21:53.417 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:21:53.417 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:21:53.417 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:21:53.417 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:21:53.417 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.14973", 2024-04-14T15:21:53.417 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/309942895", 2024-04-14T15:21:53.417 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1 2024-04-14T15:21:53.417 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.418 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:21:53.418 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.418 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:21:53.418 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.328942+0000", 2024-04-14T15:21:53.418 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:21:53.418 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.418 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.418 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:21:53.418 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.328942+0000", 2024-04-14T15:21:53.418 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:21:53.418 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.418 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.418 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:21:53.418 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.328943+0000", 2024-04-14T15:21:53.419 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6819999999999999e-06 2024-04-14T15:21:53.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.419 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.419 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:21:53.419 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.328949+0000", 2024-04-14T15:21:53.419 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6470000000000003e-06 2024-04-14T15:21:53.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.419 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.419 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:21:53.419 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.328957+0000", 2024-04-14T15:21:53.419 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2439999999999993e-06 2024-04-14T15:21:53.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.419 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.419 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:21:53.419 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.328965+0000", 2024-04-14T15:21:53.420 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7989999999999994e-06 2024-04-14T15:21:53.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.420 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:21:53.420 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.329261+0000", 2024-04-14T15:21:53.420 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029585899999999999 2024-04-14T15:21:53.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.420 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:21:53.420 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:57.329265+0000", 2024-04-14T15:21:53.420 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9339999999999999e-06 2024-04-14T15:21:53.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.420 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:21:53.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.196185+0000", 2024-04-14T15:21:53.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.86692026799999999 2024-04-14T15:21:53.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:21:53.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.196319+0000", 2024-04-14T15:21:53.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013423299999999999 2024-04-14T15:21:53.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2024-04-14T15:21:53.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.197772+0000", 2024-04-14T15:21:53.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001452395 2024-04-14T15:21:53.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.422 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2024-04-14T15:21:53.422 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.198662+0000", 2024-04-14T15:21:53.422 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00089075099999999996 2024-04-14T15:21:53.422 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.422 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:21:53.422 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.200283+0000", 2024-04-14T15:21:53.422 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016206040000000001 2024-04-14T15:21:53.422 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.422 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:21:53.422 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.200353+0000", 2024-04-14T15:21:53.422 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0017000000000004e-05 2024-04-14T15:21:53.422 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:53.423 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:53.423 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:53.423 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.423 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.423 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.24853.0:1853 16.1b e110/107 16:dbd443e1:test-rados-api-smithi090-93601-15::foo:head v 110'904, mlcod=110'902)", 2024-04-14T15:21:53.423 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:12:58.610337+0000", 2024-04-14T15:21:53.423 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 534.76090818399996, 2024-04-14T15:21:53.423 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.067447942999999996, 2024-04-14T15:21:53.423 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:21:53.423 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:21:53.423 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:21:53.423 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:21:53.423 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.423 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:21:53.423 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.610337+0000", 2024-04-14T15:21:53.424 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:21:53.424 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.424 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.424 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:21:53.424 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.610337+0000", 2024-04-14T15:21:53.424 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:21:53.424 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.424 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.424 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:21:53.424 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.610338+0000", 2024-04-14T15:21:53.424 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9700000000000005e-07 2024-04-14T15:21:53.424 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.424 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.424 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:21:53.424 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.610342+0000", 2024-04-14T15:21:53.424 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1280000000000001e-06 2024-04-14T15:21:53.425 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.425 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.425 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:21:53.425 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.610348+0000", 2024-04-14T15:21:53.425 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4679999999999997e-06 2024-04-14T15:21:53.425 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.425 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.425 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:21:53.425 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.610352+0000", 2024-04-14T15:21:53.425 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8809999999999998e-06 2024-04-14T15:21:53.425 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.425 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.425 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:21:53.425 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.610377+0000", 2024-04-14T15:21:53.425 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5080999999999999e-05 2024-04-14T15:21:53.425 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.426 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.426 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:21:53.426 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.610388+0000", 2024-04-14T15:21:53.426 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0139000000000001e-05 2024-04-14T15:21:53.426 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.426 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.426 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:21:53.426 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.610725+0000", 2024-04-14T15:21:53.426 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000337502 2024-04-14T15:21:53.426 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.426 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.426 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:21:53.426 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.677785+0000", 2024-04-14T15:21:53.426 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.067059847000000006 2024-04-14T15:21:53.427 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:53.427 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:53.427 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:53.427 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.427 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.427 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25258.0:2 58.15 58:ad551702:::test:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-14T15:21:53.427 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:12:59.091942+0000", 2024-04-14T15:21:53.427 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 534.27930284299998, 2024-04-14T15:21:53.427 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.178728945, 2024-04-14T15:21:53.427 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:21:53.427 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:21:53.427 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:21:53.427 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:21:53.427 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25258", 2024-04-14T15:21:53.427 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1067094721", 2024-04-14T15:21:53.428 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2 2024-04-14T15:21:53.428 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.428 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:21:53.428 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.428 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:21:53.428 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.091942+0000", 2024-04-14T15:21:53.428 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:21:53.428 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.428 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.428 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:21:53.428 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.091942+0000", 2024-04-14T15:21:53.428 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:21:53.429 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.429 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.429 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:21:53.429 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.091944+0000", 2024-04-14T15:21:53.429 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.784e-06 2024-04-14T15:21:53.429 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.429 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.429 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:21:53.429 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.091949+0000", 2024-04-14T15:21:53.429 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2510000000000002e-06 2024-04-14T15:21:53.429 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.429 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.429 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:21:53.429 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.091960+0000", 2024-04-14T15:21:53.429 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0958000000000001e-05 2024-04-14T15:21:53.430 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.430 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.430 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:21:53.430 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.091967+0000", 2024-04-14T15:21:53.430 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8199999999999999e-06 2024-04-14T15:21:53.430 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.430 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.430 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:21:53.430 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.092028+0000", 2024-04-14T15:21:53.430 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0875000000000002e-05 2024-04-14T15:21:53.430 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.430 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.430 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2024-04-14T15:21:53.431 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.092036+0000", 2024-04-14T15:21:53.431 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9410000000000007e-06 2024-04-14T15:21:53.431 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.431 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.431 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:21:53.431 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.119696+0000", 2024-04-14T15:21:53.431 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.027659777 2024-04-14T15:21:53.431 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.431 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.431 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:21:53.431 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.119705+0000", 2024-04-14T15:21:53.431 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0280000000000006e-06 2024-04-14T15:21:53.431 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.431 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.431 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:21:53.431 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.252689+0000", 2024-04-14T15:21:53.432 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.132983924 2024-04-14T15:21:53.432 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.432 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.432 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:21:53.432 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.252798+0000", 2024-04-14T15:21:53.432 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000109275 2024-04-14T15:21:53.432 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.432 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.432 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4,5", 2024-04-14T15:21:53.432 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.252921+0000", 2024-04-14T15:21:53.432 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000122583 2024-04-14T15:21:53.432 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.432 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.432 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:21:53.432 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.257824+0000", 2024-04-14T15:21:53.432 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0049038320000000003 2024-04-14T15:21:53.433 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.433 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.433 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:21:53.433 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.269059+0000", 2024-04-14T15:21:53.433 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011234545 2024-04-14T15:21:53.433 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.433 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.433 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:21:53.433 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.270587+0000", 2024-04-14T15:21:53.433 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001528223 2024-04-14T15:21:53.433 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.433 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.433 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:21:53.433 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.270631+0000", 2024-04-14T15:21:53.433 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3869999999999998e-05 2024-04-14T15:21:53.434 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.434 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.434 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:21:53.434 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.270671+0000", 2024-04-14T15:21:53.434 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0259000000000002e-05 2024-04-14T15:21:53.434 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:53.434 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:53.434 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:53.434 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.434 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.434 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.15264.0:3 64.14 64:2a647cd4:::obj0:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-14T15:21:53.434 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:12:59.285619+0000", 2024-04-14T15:21:53.434 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 534.08562583200001, 2024-04-14T15:21:53.434 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99102734800000003, 2024-04-14T15:21:53.435 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:21:53.435 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:21:53.435 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-04-14T15:21:53.435 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:21:53.435 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.15264", 2024-04-14T15:21:53.435 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1337739815", 2024-04-14T15:21:53.435 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3 2024-04-14T15:21:53.435 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.435 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:21:53.435 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.435 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:21:53.435 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.285619+0000", 2024-04-14T15:21:53.435 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:21:53.435 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.435 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.435 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:21:53.436 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.285619+0000", 2024-04-14T15:21:53.436 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:21:53.436 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.436 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.436 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:21:53.436 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.285621+0000", 2024-04-14T15:21:53.436 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.339e-06 2024-04-14T15:21:53.436 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.436 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.436 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:21:53.436 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.285625+0000", 2024-04-14T15:21:53.436 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4429999999999996e-06 2024-04-14T15:21:53.436 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.436 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.437 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:21:53.437 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.285636+0000", 2024-04-14T15:21:53.437 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0769e-05 2024-04-14T15:21:53.437 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.437 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.437 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:21:53.437 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.285644+0000", 2024-04-14T15:21:53.437 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5029999999999997e-06 2024-04-14T15:21:53.437 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.437 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.437 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:21:53.437 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.286065+0000", 2024-04-14T15:21:53.437 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00042017999999999998 2024-04-14T15:21:53.437 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.437 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.437 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:21:53.437 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.286072+0000", 2024-04-14T15:21:53.438 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6000000000000001e-06 2024-04-14T15:21:53.438 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.438 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.438 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:21:53.438 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.276569+0000", 2024-04-14T15:21:53.438 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99049650199999995 2024-04-14T15:21:53.438 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.438 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.438 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:21:53.438 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.276647+0000", 2024-04-14T15:21:53.438 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8011999999999998e-05 2024-04-14T15:21:53.438 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:53.438 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:53.439 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:53.439 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.439 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.439 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24871.0:8 18.1c 18:3cd39c19:test-rados-api-smithi090-93709-4::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-04-14T15:21:53.439 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:13:00.205084+0000", 2024-04-14T15:21:53.439 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 533.16616082600001, 2024-04-14T15:21:53.439 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.071432922999999995, 2024-04-14T15:21:53.439 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:21:53.439 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:21:53.439 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:21:53.439 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:21:53.439 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24871", 2024-04-14T15:21:53.439 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1680088077", 2024-04-14T15:21:53.440 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 8 2024-04-14T15:21:53.440 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.440 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:21:53.440 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.440 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:21:53.440 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.205084+0000", 2024-04-14T15:21:53.440 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:21:53.440 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.440 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.440 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:21:53.440 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.205084+0000", 2024-04-14T15:21:53.440 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:21:53.440 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.440 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.441 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:21:53.441 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.205087+0000", 2024-04-14T15:21:53.441 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3829999999999999e-06 2024-04-14T15:21:53.441 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.441 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.441 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:21:53.441 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.205096+0000", 2024-04-14T15:21:53.441 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5259999999999999e-06 2024-04-14T15:21:53.441 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.441 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.441 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:21:53.441 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.205109+0000", 2024-04-14T15:21:53.441 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2473000000000001e-05 2024-04-14T15:21:53.441 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.441 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.442 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:21:53.442 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.205119+0000", 2024-04-14T15:21:53.442 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.077e-05 2024-04-14T15:21:53.442 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.442 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.442 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:21:53.442 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.206813+0000", 2024-04-14T15:21:53.442 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001693684 2024-04-14T15:21:53.442 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.442 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.442 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:21:53.442 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.206943+0000", 2024-04-14T15:21:53.442 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129667 2024-04-14T15:21:53.443 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.443 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.443 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,7", 2024-04-14T15:21:53.443 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.210425+0000", 2024-04-14T15:21:53.443 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0034817469999999999 2024-04-14T15:21:53.443 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.443 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.443 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:21:53.443 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.212326+0000", 2024-04-14T15:21:53.443 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0019013420000000001 2024-04-14T15:21:53.443 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.443 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.443 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:21:53.444 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.255071+0000", 2024-04-14T15:21:53.444 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.042745584000000003 2024-04-14T15:21:53.444 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.444 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.444 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:21:53.444 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.276425+0000", 2024-04-14T15:21:53.444 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.021353556999999999 2024-04-14T15:21:53.444 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.444 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.444 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:21:53.444 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.276476+0000", 2024-04-14T15:21:53.444 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1041000000000001e-05 2024-04-14T15:21:53.444 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.444 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.444 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:21:53.445 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.276517+0000", 2024-04-14T15:21:53.445 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1149000000000002e-05 2024-04-14T15:21:53.445 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:53.445 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:53.445 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:53.445 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.445 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.445 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24871.0:10 18.1c 18:3cd39c19:test-rados-api-smithi090-93709-4::foo:head [writefull 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-04-14T15:21:53.445 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:13:00.870154+0000", 2024-04-14T15:21:53.445 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 532.50109080799996, 2024-04-14T15:21:53.445 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.067196143, 2024-04-14T15:21:53.445 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:21:53.445 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:21:53.445 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:21:53.446 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:21:53.446 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24871", 2024-04-14T15:21:53.446 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1680088077", 2024-04-14T15:21:53.446 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 10 2024-04-14T15:21:53.446 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.446 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:21:53.446 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.446 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:21:53.446 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.870154+0000", 2024-04-14T15:21:53.446 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:21:53.446 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.446 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.446 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:21:53.446 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.870154+0000", 2024-04-14T15:21:53.447 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:21:53.447 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.447 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.447 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:21:53.447 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.870156+0000", 2024-04-14T15:21:53.447 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.75e-06 2024-04-14T15:21:53.447 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.447 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.447 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:21:53.447 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.870165+0000", 2024-04-14T15:21:53.447 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7490000000000004e-06 2024-04-14T15:21:53.447 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.447 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.447 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:21:53.447 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.870177+0000", 2024-04-14T15:21:53.447 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1686e-05 2024-04-14T15:21:53.448 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.448 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.448 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:21:53.448 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.870188+0000", 2024-04-14T15:21:53.448 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1654000000000001e-05 2024-04-14T15:21:53.448 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.448 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.448 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:21:53.448 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.880646+0000", 2024-04-14T15:21:53.448 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010457664 2024-04-14T15:21:53.448 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.448 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.448 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:21:53.448 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.880744+0000", 2024-04-14T15:21:53.448 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7684999999999997e-05 2024-04-14T15:21:53.449 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.449 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.449 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,7", 2024-04-14T15:21:53.449 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.880916+0000", 2024-04-14T15:21:53.449 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000172518 2024-04-14T15:21:53.449 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.449 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.449 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:21:53.449 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.902443+0000", 2024-04-14T15:21:53.449 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.021526851999999999 2024-04-14T15:21:53.449 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.449 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.449 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:21:53.449 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.913208+0000", 2024-04-14T15:21:53.449 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010764785000000001 2024-04-14T15:21:53.450 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.450 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.450 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:21:53.450 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.937249+0000", 2024-04-14T15:21:53.450 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.024041565000000001 2024-04-14T15:21:53.450 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.450 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.450 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:21:53.450 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.937305+0000", 2024-04-14T15:21:53.450 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5503e-05 2024-04-14T15:21:53.450 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.450 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.450 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:21:53.450 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.937350+0000", 2024-04-14T15:21:53.450 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5732000000000002e-05 2024-04-14T15:21:53.451 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:53.451 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:53.451 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:53.451 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.451 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.451 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.26497.0:35 112.1 112:8133d882:test-rados-api-smithi090-93724-10::foo:head [write 128~128] snapc 7=[7,6] ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-04-14T15:21:53.451 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:13:20.864703+0000", 2024-04-14T15:21:53.451 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 512.50654169500001, 2024-04-14T15:21:53.451 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.080590538000000003, 2024-04-14T15:21:53.451 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:21:53.451 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:21:53.451 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:21:53.451 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:21:53.451 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.26497", 2024-04-14T15:21:53.451 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3962832909", 2024-04-14T15:21:53.452 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 35 2024-04-14T15:21:53.452 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.452 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:21:53.452 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.452 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:21:53.452 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.864703+0000", 2024-04-14T15:21:53.452 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:21:53.452 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.452 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.452 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:21:53.452 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.864703+0000", 2024-04-14T15:21:53.452 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:21:53.452 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.452 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.452 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:21:53.452 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.864706+0000", 2024-04-14T15:21:53.453 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2139999999999999e-06 2024-04-14T15:21:53.453 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.453 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.453 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:21:53.453 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.864714+0000", 2024-04-14T15:21:53.453 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4610000000000008e-06 2024-04-14T15:21:53.453 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.453 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.453 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:21:53.453 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.864726+0000", 2024-04-14T15:21:53.453 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1395e-05 2024-04-14T15:21:53.453 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.453 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.453 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:21:53.453 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.864736+0000", 2024-04-14T15:21:53.454 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0335e-05 2024-04-14T15:21:53.454 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.454 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.454 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:21:53.454 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.869283+0000", 2024-04-14T15:21:53.454 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0045468080000000003 2024-04-14T15:21:53.454 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.454 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.454 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:21:53.454 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.869389+0000", 2024-04-14T15:21:53.454 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010623799999999999 2024-04-14T15:21:53.454 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.454 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.454 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,5", 2024-04-14T15:21:53.455 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.869590+0000", 2024-04-14T15:21:53.455 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020072800000000001 2024-04-14T15:21:53.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.455 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.455 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:21:53.455 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.878172+0000", 2024-04-14T15:21:53.455 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0085828139999999994 2024-04-14T15:21:53.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.455 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.455 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:21:53.455 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.897188+0000", 2024-04-14T15:21:53.455 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019015964 2024-04-14T15:21:53.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.455 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.455 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:21:53.456 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.945179+0000", 2024-04-14T15:21:53.456 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.047990789999999998 2024-04-14T15:21:53.456 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.456 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.456 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:21:53.456 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.945239+0000", 2024-04-14T15:21:53.456 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9553999999999999e-05 2024-04-14T15:21:53.456 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.456 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.456 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:21:53.456 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:20.945294+0000", 2024-04-14T15:21:53.456 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5237e-05 2024-04-14T15:21:53.456 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:53.456 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:53.456 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:53.456 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.457 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.457 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.38086.0:1 149.4s0 149:2d0198b2:test-rados-api-smithi090-93709-11::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-04-14T15:21:53.457 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:13:23.915322+0000", 2024-04-14T15:21:53.457 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 509.45592288300003, 2024-04-14T15:21:53.457 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.98550727800000004, 2024-04-14T15:21:53.457 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:21:53.457 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:21:53.457 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:21:53.457 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:21:53.457 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.38086", 2024-04-14T15:21:53.457 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2598250370", 2024-04-14T15:21:53.457 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1 2024-04-14T15:21:53.457 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.457 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:21:53.457 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.457 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:21:53.458 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:23.915322+0000", 2024-04-14T15:21:53.458 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:21:53.458 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.458 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.458 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:21:53.458 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:23.915322+0000", 2024-04-14T15:21:53.458 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:21:53.458 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.458 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.458 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:21:53.458 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:23.915330+0000", 2024-04-14T15:21:53.458 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3350000000000001e-06 2024-04-14T15:21:53.458 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.458 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.459 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:21:53.459 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:23.915345+0000", 2024-04-14T15:21:53.459 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4956e-05 2024-04-14T15:21:53.459 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.459 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.459 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:21:53.459 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:23.915361+0000", 2024-04-14T15:21:53.459 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6750999999999999e-05 2024-04-14T15:21:53.459 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.459 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.459 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:21:53.459 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:23.915374+0000", 2024-04-14T15:21:53.459 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.309e-05 2024-04-14T15:21:53.459 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.459 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.459 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:21:53.460 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:23.933322+0000", 2024-04-14T15:21:53.460 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017947874999999999 2024-04-14T15:21:53.460 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.460 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.460 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:21:53.460 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:23.933327+0000", 2024-04-14T15:21:53.460 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5399999999999997e-06 2024-04-14T15:21:53.460 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.460 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.460 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:21:53.460 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:24.887691+0000", 2024-04-14T15:21:53.460 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.95436462 2024-04-14T15:21:53.460 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.460 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.460 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:21:53.460 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:24.887825+0000", 2024-04-14T15:21:53.460 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133387 2024-04-14T15:21:53.461 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.461 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.461 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2024-04-14T15:21:53.461 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:24.888636+0000", 2024-04-14T15:21:53.461 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00081147700000000001 2024-04-14T15:21:53.461 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.461 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.461 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2024-04-14T15:21:53.461 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:24.896179+0000", 2024-04-14T15:21:53.461 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0075423469999999996 2024-04-14T15:21:53.461 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.461 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.461 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:21:53.461 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:24.900761+0000", 2024-04-14T15:21:53.461 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0045823230000000001 2024-04-14T15:21:53.461 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.462 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.462 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:21:53.462 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:24.900830+0000", 2024-04-14T15:21:53.462 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8577000000000004e-05 2024-04-14T15:21:53.462 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:53.462 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:53.462 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:53.462 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.462 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.462 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.42758.0:2 160.1b 160:d96a01e0:::testobj:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-04-14T15:21:53.462 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:13:27.843225+0000", 2024-04-14T15:21:53.462 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 505.52801973599998, 2024-04-14T15:21:53.462 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.150888965, 2024-04-14T15:21:53.462 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:21:53.463 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:21:53.463 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:21:53.463 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:21:53.463 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.42758", 2024-04-14T15:21:53.463 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2966476037", 2024-04-14T15:21:53.463 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2 2024-04-14T15:21:53.463 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.463 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:21:53.463 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.463 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:21:53.463 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:27.843225+0000", 2024-04-14T15:21:53.463 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:21:53.463 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.463 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.464 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:21:53.464 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:27.843225+0000", 2024-04-14T15:21:53.464 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:21:53.464 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.464 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.464 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:21:53.464 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:27.843228+0000", 2024-04-14T15:21:53.464 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3140000000000002e-06 2024-04-14T15:21:53.464 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.464 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.464 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:21:53.464 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:27.843237+0000", 2024-04-14T15:21:53.464 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9099999999999994e-06 2024-04-14T15:21:53.464 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.464 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.464 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:21:53.465 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:27.843248+0000", 2024-04-14T15:21:53.465 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1494999999999999e-05 2024-04-14T15:21:53.465 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.465 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.465 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:21:53.465 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:27.843261+0000", 2024-04-14T15:21:53.465 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3067e-05 2024-04-14T15:21:53.465 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.465 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.465 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:21:53.465 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:27.878933+0000", 2024-04-14T15:21:53.465 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.035672263000000003 2024-04-14T15:21:53.465 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.465 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.465 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:21:53.465 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:27.878942+0000", 2024-04-14T15:21:53.466 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3860000000000007e-06 2024-04-14T15:21:53.466 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.466 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.466 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:21:53.466 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.961457+0000", 2024-04-14T15:21:53.466 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0825154450000001 2024-04-14T15:21:53.466 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.466 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.466 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:21:53.466 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.961594+0000", 2024-04-14T15:21:53.466 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000136687 2024-04-14T15:21:53.466 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.466 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.466 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2024-04-14T15:21:53.467 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.961729+0000", 2024-04-14T15:21:53.467 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000134914 2024-04-14T15:21:53.467 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.467 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.467 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:21:53.467 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.962866+0000", 2024-04-14T15:21:53.468 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011373519999999999 2024-04-14T15:21:53.468 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.468 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.468 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:21:53.468 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.968752+0000", 2024-04-14T15:21:53.468 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0058854370000000003 2024-04-14T15:21:53.468 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.468 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.468 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:21:53.469 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.994007+0000", 2024-04-14T15:21:53.469 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.025255248000000001 2024-04-14T15:21:53.469 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.469 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.469 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:21:53.469 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.994064+0000", 2024-04-14T15:21:53.469 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6789000000000001e-05 2024-04-14T15:21:53.469 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.469 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.469 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:21:53.469 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.994114+0000", 2024-04-14T15:21:53.469 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0658000000000002e-05 2024-04-14T15:21:53.469 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:53.469 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:53.469 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:53.469 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.470 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.470 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.44332.0:2 159.0 159:0384ab8e:::oid:head [append 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-04-14T15:21:53.470 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:13:27.899663+0000", 2024-04-14T15:21:53.470 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 505.47158183699997, 2024-04-14T15:21:53.470 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0793927670000001, 2024-04-14T15:21:53.470 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:21:53.470 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:21:53.470 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:21:53.470 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:21:53.470 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.44332", 2024-04-14T15:21:53.470 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/4025981533", 2024-04-14T15:21:53.470 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2 2024-04-14T15:21:53.470 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.470 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:21:53.471 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.471 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:21:53.471 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:27.899663+0000", 2024-04-14T15:21:53.471 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:21:53.471 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.471 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.471 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:21:53.471 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:27.899663+0000", 2024-04-14T15:21:53.471 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:21:53.471 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.471 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.471 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:21:53.471 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:27.899665+0000", 2024-04-14T15:21:53.471 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.44e-06 2024-04-14T15:21:53.472 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.472 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.472 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:21:53.472 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:27.899670+0000", 2024-04-14T15:21:53.472 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7200000000000003e-06 2024-04-14T15:21:53.472 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.472 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.472 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:21:53.472 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:27.899680+0000", 2024-04-14T15:21:53.472 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7049999999999994e-06 2024-04-14T15:21:53.472 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.472 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.472 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:21:53.472 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:27.899691+0000", 2024-04-14T15:21:53.473 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0336999999999999e-05 2024-04-14T15:21:53.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.473 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.473 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:21:53.473 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:27.899995+0000", 2024-04-14T15:21:53.473 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030434700000000001 2024-04-14T15:21:53.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.473 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.473 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:21:53.473 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:27.900004+0000", 2024-04-14T15:21:53.473 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2420000000000006e-06 2024-04-14T15:21:53.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.473 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.473 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:21:53.473 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.969216+0000", 2024-04-14T15:21:53.474 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.069211793 2024-04-14T15:21:53.474 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.474 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.474 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:21:53.474 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.969339+0000", 2024-04-14T15:21:53.474 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012265 2024-04-14T15:21:53.474 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.474 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.474 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,5", 2024-04-14T15:21:53.474 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.969485+0000", 2024-04-14T15:21:53.474 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014675699999999999 2024-04-14T15:21:53.474 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.474 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.474 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:21:53.474 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.971325+0000", 2024-04-14T15:21:53.475 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0018392720000000001 2024-04-14T15:21:53.475 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.475 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.475 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:21:53.475 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.971713+0000", 2024-04-14T15:21:53.475 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038802299999999998 2024-04-14T15:21:53.475 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.475 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.475 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:21:53.475 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.978938+0000", 2024-04-14T15:21:53.475 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0072256710000000004 2024-04-14T15:21:53.475 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.475 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.475 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:21:53.475 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.979007+0000", 2024-04-14T15:21:53.475 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9220000000000005e-05 2024-04-14T15:21:53.476 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.476 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.476 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:21:53.476 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:28.979056+0000", 2024-04-14T15:21:53.476 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8590000000000001e-05 2024-04-14T15:21:53.476 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:53.476 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:53.476 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:53.476 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.476 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.476 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.35916.0:2 164.1b 164:d96a01e0:::testobj:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-04-14T15:21:53.476 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:13:30.848008+0000", 2024-04-14T15:21:53.476 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 502.52323755200001, 2024-04-14T15:21:53.476 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.012766968, 2024-04-14T15:21:53.476 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:21:53.476 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:21:53.477 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:21:53.477 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:21:53.477 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.35916", 2024-04-14T15:21:53.477 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2231042766", 2024-04-14T15:21:53.477 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2 2024-04-14T15:21:53.477 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.477 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:21:53.477 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.477 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:21:53.477 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:30.848008+0000", 2024-04-14T15:21:53.477 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:21:53.477 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.477 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.477 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:21:53.477 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:30.848008+0000", 2024-04-14T15:21:53.477 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:21:53.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.478 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:21:53.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:30.848009+0000", 2024-04-14T15:21:53.478 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5e-06 2024-04-14T15:21:53.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.478 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:21:53.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:30.848015+0000", 2024-04-14T15:21:53.478 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2439999999999998e-06 2024-04-14T15:21:53.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.478 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:21:53.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:30.848024+0000", 2024-04-14T15:21:53.479 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2560000000000002e-06 2024-04-14T15:21:53.479 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.479 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.479 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:21:53.479 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:30.848032+0000", 2024-04-14T15:21:53.479 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0630000000000006e-06 2024-04-14T15:21:53.479 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.479 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.479 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:21:53.479 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:30.848081+0000", 2024-04-14T15:21:53.479 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9628e-05 2024-04-14T15:21:53.479 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.479 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.479 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:21:53.479 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:30.848089+0000", 2024-04-14T15:21:53.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.362e-06 2024-04-14T15:21:53.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.480 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:21:53.480 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:31.850888+0000", 2024-04-14T15:21:53.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.002799182 2024-04-14T15:21:53.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.480 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:21:53.480 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:31.850999+0000", 2024-04-14T15:21:53.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000111294 2024-04-14T15:21:53.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.480 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,6", 2024-04-14T15:21:53.481 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:31.851133+0000", 2024-04-14T15:21:53.481 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133678 2024-04-14T15:21:53.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.481 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:21:53.481 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:31.852376+0000", 2024-04-14T15:21:53.481 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0012427479999999999 2024-04-14T15:21:53.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.481 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:21:53.481 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:31.859603+0000", 2024-04-14T15:21:53.481 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0072276409999999999 2024-04-14T15:21:53.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.482 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:21:53.482 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:31.860679+0000", 2024-04-14T15:21:53.482 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001076123 2024-04-14T15:21:53.482 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.482 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.482 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:21:53.482 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:31.860733+0000", 2024-04-14T15:21:53.482 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3341999999999999e-05 2024-04-14T15:21:53.482 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.482 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.482 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:21:53.482 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:31.860775+0000", 2024-04-14T15:21:53.482 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1906999999999997e-05 2024-04-14T15:21:53.482 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:53.482 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:53.483 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:53.483 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.483 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.483 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.48652.0:1 251.6s0 251:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e257)", 2024-04-14T15:21:53.483 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:15:24.884068+0000", 2024-04-14T15:21:53.483 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 388.48717747900002, 2024-04-14T15:21:53.483 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.98326544100000002, 2024-04-14T15:21:53.483 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:21:53.483 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:21:53.483 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:21:53.483 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:21:53.483 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.48652", 2024-04-14T15:21:53.483 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/747873163", 2024-04-14T15:21:53.483 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1 2024-04-14T15:21:53.483 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.483 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:21:53.484 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.484 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:21:53.484 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:24.884068+0000", 2024-04-14T15:21:53.484 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:21:53.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.484 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.484 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:21:53.484 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:24.884068+0000", 2024-04-14T15:21:53.484 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:21:53.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.484 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.484 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:21:53.484 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:24.884070+0000", 2024-04-14T15:21:53.484 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.996e-06 2024-04-14T15:21:53.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.484 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.485 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:21:53.485 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:24.884080+0000", 2024-04-14T15:21:53.485 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0229e-05 2024-04-14T15:21:53.485 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.485 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.485 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:21:53.485 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:24.884092+0000", 2024-04-14T15:21:53.485 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1758e-05 2024-04-14T15:21:53.485 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.485 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.485 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:21:53.485 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:24.884102+0000", 2024-04-14T15:21:53.485 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0344e-05 2024-04-14T15:21:53.485 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.485 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.485 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:21:53.486 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:24.885058+0000", 2024-04-14T15:21:53.486 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00095602199999999995 2024-04-14T15:21:53.486 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.486 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.486 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:21:53.486 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:24.885061+0000", 2024-04-14T15:21:53.486 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.433e-06 2024-04-14T15:21:53.486 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.486 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.486 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:21:53.486 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:25.865679+0000", 2024-04-14T15:21:53.486 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.98061781100000001 2024-04-14T15:21:53.486 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.486 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.487 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:21:53.487 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:25.865804+0000", 2024-04-14T15:21:53.487 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000124344 2024-04-14T15:21:53.487 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.487 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.487 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2024-04-14T15:21:53.487 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:25.866199+0000", 2024-04-14T15:21:53.487 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039497799999999998 2024-04-14T15:21:53.487 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.487 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.487 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2024-04-14T15:21:53.487 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:25.867043+0000", 2024-04-14T15:21:53.487 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00084418399999999999 2024-04-14T15:21:53.487 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.487 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.488 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:21:53.488 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:25.867267+0000", 2024-04-14T15:21:53.488 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022428300000000001 2024-04-14T15:21:53.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.488 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:21:53.488 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:15:25.867333+0000", 2024-04-14T15:21:53.488 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6058999999999999e-05 2024-04-14T15:21:53.488 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:53.488 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:53.488 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:53.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.488 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24437.0:657 3.0 3:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e526)", 2024-04-14T15:21:53.488 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:20:52.848110+0000", 2024-04-14T15:21:53.488 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 60.523135322000002, 2024-04-14T15:21:53.489 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12803794499999999, 2024-04-14T15:21:53.489 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:21:53.489 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:21:53.489 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-14T15:21:53.489 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:21:53.489 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24437", 2024-04-14T15:21:53.489 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.92:0/2783975543", 2024-04-14T15:21:53.489 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 657 2024-04-14T15:21:53.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.489 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:21:53.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.489 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:21:53.489 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:20:52.848110+0000", 2024-04-14T15:21:53.489 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:21:53.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.490 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:21:53.490 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:20:52.848110+0000", 2024-04-14T15:21:53.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:21:53.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.490 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.490 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:21:53.490 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:20:52.848112+0000", 2024-04-14T15:21:53.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.554e-06 2024-04-14T15:21:53.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.490 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.490 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:21:53.490 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:20:52.848121+0000", 2024-04-14T15:21:53.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9209999999999998e-06 2024-04-14T15:21:53.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.491 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.491 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:21:53.491 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:20:52.848135+0000", 2024-04-14T15:21:53.491 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3215e-05 2024-04-14T15:21:53.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.491 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.491 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:21:53.491 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:20:52.848145+0000", 2024-04-14T15:21:53.491 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0767999999999999e-05 2024-04-14T15:21:53.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.491 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.491 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:21:53.491 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:20:52.848259+0000", 2024-04-14T15:21:53.491 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113909 2024-04-14T15:21:53.492 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.492 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.492 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:21:53.492 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:20:52.848271+0000", 2024-04-14T15:21:53.492 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1606999999999999e-05 2024-04-14T15:21:53.492 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.492 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.492 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:21:53.492 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:20:52.975869+0000", 2024-04-14T15:21:53.492 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12759867499999999 2024-04-14T15:21:53.492 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.492 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.492 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:21:53.492 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:20:52.976008+0000", 2024-04-14T15:21:53.492 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013836600000000001 2024-04-14T15:21:53.492 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:21:53.492 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:21:53.493 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:21:53.493 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:20:52.976148+0000", 2024-04-14T15:21:53.493 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013993 2024-04-14T15:21:53.493 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:53.493 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:53.493 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:53.493 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:21:53.493 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:21:53.493 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-14T15:21:53.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:53.494 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:53.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:21:53.595 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:53.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:53.620 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:53.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:53.720 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:21:53.745 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:53.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:53.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:21:53.847 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:21:53.870 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:53.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:53.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:21:53.972 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:21:53.995 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:53.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:53.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:21:54.029 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-14T15:21:54.097 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:21:54.120 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:54.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:54.121 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:21:54.222 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:21:54.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:54.246 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:54.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:21:54.346 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:21:54.370 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:54.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:54.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:21:54.441 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:21:53 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:21:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:21:54.471 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:54.496 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:54.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:54.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:54.598 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:21:54.621 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:54.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:54.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:21:54.723 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:21:54.746 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:54.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:54.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:21:54.847 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:21:54.874 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:54.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:54.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:21:54.976 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-14T15:21:54.976 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-14T15:21:54.976 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-04-14T15:21:54.976 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph tell osd.0 reset_purged_snaps_last 2024-04-14T15:21:54.977 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:55.005 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:55.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:55.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:55.107 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:21:55.131 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:55.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:55.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:21:55.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:54 smithi090 ceph-mon[30399]: pgmap v846: 216 pgs: 7 remapped+peering, 1 peering, 2 unknown, 206 active+clean; 8.4 MiB data, 507 MiB used, 625 GiB / 626 GiB avail; 3.0 KiB/s rd, 1 op/s; 9.8 KiB/s, 0 objects/s recovering 2024-04-14T15:21:55.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:54 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:21:55.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:54 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:21:55.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:54 smithi090 ceph-mon[30399]: osdmap e578: 8 total, 8 up, 7 in 2024-04-14T15:21:55.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:54 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:21:55.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:54 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:21:55.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:54 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:21:55.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:54 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:21:55.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:54 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:21:55.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:54 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:21:55.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:54 smithi090 ceph-mon[41670]: pgmap v846: 216 pgs: 7 remapped+peering, 1 peering, 2 unknown, 206 active+clean; 8.4 MiB data, 507 MiB used, 625 GiB / 626 GiB avail; 3.0 KiB/s rd, 1 op/s; 9.8 KiB/s, 0 objects/s recovering 2024-04-14T15:21:55.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:54 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:21:55.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:54 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:21:55.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:54 smithi090 ceph-mon[41670]: osdmap e578: 8 total, 8 up, 7 in 2024-04-14T15:21:55.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:54 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:21:55.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:54 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:21:55.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:54 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:21:55.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:54 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:21:55.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:54 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:21:55.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:54 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:21:55.233 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:21:55.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:55.258 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:55.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:21:55.275 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:54 smithi092 ceph-mon[33966]: pgmap v846: 216 pgs: 7 remapped+peering, 1 peering, 2 unknown, 206 active+clean; 8.4 MiB data, 507 MiB used, 625 GiB / 626 GiB avail; 3.0 KiB/s rd, 1 op/s; 9.8 KiB/s, 0 objects/s recovering 2024-04-14T15:21:55.275 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:54 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:21:55.275 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:54 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:21:55.275 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:54 smithi092 ceph-mon[33966]: osdmap e578: 8 total, 8 up, 7 in 2024-04-14T15:21:55.275 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:54 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:21:55.275 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:54 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:21:55.275 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:54 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:21:55.275 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:54 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:21:55.275 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:54 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:21:55.275 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:54 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:21:55.359 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:21:55.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:55.385 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:55.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:21:55.486 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:21:55.510 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:55.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:55.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:21:55.612 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:21:55.622 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:21:55 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:21:55.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:55.636 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:55.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:21:55.737 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:21:55.762 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:55.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:55.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:21:55.780 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-14T15:21:55.865 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:21:55.893 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:55.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:55.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:21:55.994 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:21:56.020 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:56.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:56.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:21:56.062 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:55 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-98","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:21:56.062 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:55 smithi090 ceph-mon[30399]: osdmap e579: 8 total, 8 up, 7 in 2024-04-14T15:21:56.062 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:55 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:21:56.062 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:55 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:21:56.062 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:55 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-04-14T15:21:56.062 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:55 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-98", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:21:56.062 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:55 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]': finished 2024-04-14T15:21:56.063 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:55 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-98","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:21:56.063 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:55 smithi090 ceph-mon[41670]: osdmap e579: 8 total, 8 up, 7 in 2024-04-14T15:21:56.063 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:55 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:21:56.063 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:55 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:21:56.063 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:55 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-04-14T15:21:56.063 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:55 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-98", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:21:56.063 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:55 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]': finished 2024-04-14T15:21:56.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:56.068+0000 7fa97f74d700 1 -- 172.21.15.90:0/3111386027 >> v1:172.21.15.92:6789/0 conn(0x7fa97810f340 legacy=0x7fa9781117d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:56.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:56.068+0000 7fa97f74d700 1 -- 172.21.15.90:0/3111386027 shutdown_connections 2024-04-14T15:21:56.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:56.068+0000 7fa97f74d700 1 -- 172.21.15.90:0/3111386027 >> 172.21.15.90:0/3111386027 conn(0x7fa9780fd290 msgr2=0x7fa9780ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:21:56.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:56.068+0000 7fa97f74d700 1 -- 172.21.15.90:0/3111386027 shutdown_connections 2024-04-14T15:21:56.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:56.069+0000 7fa97f74d700 1 -- 172.21.15.90:0/3111386027 wait complete. 2024-04-14T15:21:56.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:56.069+0000 7fa97f74d700 1 librados: starting msgr at 2024-04-14T15:21:56.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:56.069+0000 7fa97f74d700 1 librados: starting objecter 2024-04-14T15:21:56.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:56.069+0000 7fa97f74d700 1 Processor -- start 2024-04-14T15:21:56.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:56.069+0000 7fa97f74d700 1 -- start start 2024-04-14T15:21:56.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:56.069+0000 7fa97f74d700 1 librados: setting wanted keys 2024-04-14T15:21:56.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:56.069+0000 7fa97f74d700 1 librados: calling monclient init 2024-04-14T15:21:56.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:56.070+0000 7fa97f74d700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa9781a77f0 con 0x7fa97810b830 2024-04-14T15:21:56.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:56.070+0000 7fa97f74d700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa9781a8a10 con 0x7fa978103a20 2024-04-14T15:21:56.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:56.070+0000 7fa97f74d700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa9781a9c50 con 0x7fa97810f340 2024-04-14T15:21:56.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:56.070+0000 7fa97cce8700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fa97810b830 0x7fa9781a2760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:36598/0 (socket says 172.21.15.90:36598) 2024-04-14T15:21:56.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:56.070+0000 7fa97cce8700 1 -- 172.21.15.90:0/199288222 learned_addr learned my addr 172.21.15.90:0/199288222 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:21:56.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:56.070+0000 7fa96e7fc700 10 client.?.objecter ms_handle_connect 0x7fa97810f340 2024-04-14T15:21:56.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:56.070+0000 7fa96e7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:21:56.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:56.070+0000 7fa96e7fc700 10 client.?.objecter ms_handle_connect 0x7fa97810b830 2024-04-14T15:21:56.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:56.070+0000 7fa96e7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:21:56.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:56.070+0000 7fa96e7fc700 1 -- 172.21.15.90:0/199288222 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2592425492 0 0) 0x7fa9781a9c50 con 0x7fa97810f340 2024-04-14T15:21:56.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:56.070+0000 7fa96e7fc700 1 -- 172.21.15.90:0/199288222 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa9640039c0 con 0x7fa97810f340 2024-04-14T15:21:56.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:56.070+0000 7fa96e7fc700 1 -- 172.21.15.90:0/199288222 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2379332084 0 0) 0x7fa9781a77f0 con 0x7fa97810b830 2024-04-14T15:21:56.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:56.070+0000 7fa96e7fc700 1 -- 172.21.15.90:0/199288222 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa9781a9c50 con 0x7fa97810b830 2024-04-14T15:21:56.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:56.070+0000 7fa96e7fc700 10 client.?.objecter ms_handle_connect 0x7fa978103a20 2024-04-14T15:21:56.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:56.070+0000 7fa96e7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:21:56.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:56.071+0000 7fa96e7fc700 1 -- 172.21.15.90:0/199288222 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1537474030 0 0) 0x7fa9640039c0 con 0x7fa97810f340 2024-04-14T15:21:56.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:56.071+0000 7fa96e7fc700 1 -- 172.21.15.90:0/199288222 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa9781a77f0 con 0x7fa97810f340 2024-04-14T15:21:56.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:56.071+0000 7fa96e7fc700 1 -- 172.21.15.90:0/199288222 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fa970001ac0 con 0x7fa97810f340 2024-04-14T15:21:56.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:56.071+0000 7fa96e7fc700 1 -- 172.21.15.90:0/199288222 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3604910008 0 0) 0x7fa9781a8a10 con 0x7fa978103a20 2024-04-14T15:21:56.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:56.071+0000 7fa96e7fc700 1 -- 172.21.15.90:0/199288222 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa9640039c0 con 0x7fa978103a20 2024-04-14T15:21:56.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:56.071+0000 7fa96e7fc700 1 -- 172.21.15.90:0/199288222 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3845167009 0 0) 0x7fa9781a9c50 con 0x7fa97810b830 2024-04-14T15:21:56.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:56.071+0000 7fa96e7fc700 1 -- 172.21.15.90:0/199288222 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa9781a8a10 con 0x7fa97810b830 2024-04-14T15:21:56.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:56.071+0000 7fa96e7fc700 1 -- 172.21.15.90:0/199288222 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fa968001a30 con 0x7fa97810b830 2024-04-14T15:21:56.074 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:21:56.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:56.072+0000 7fa96e7fc700 1 -- 172.21.15.90:0/199288222 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3120501134 0 0) 0x7fa9781a8a10 con 0x7fa97810b830 2024-04-14T15:21:56.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:56.072+0000 7fa96e7fc700 1 -- 172.21.15.90:0/199288222 >> v1:172.21.15.90:6790/0 conn(0x7fa97810f340 legacy=0x7fa9781a5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:56.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:56.072+0000 7fa96e7fc700 1 -- 172.21.15.90:0/199288222 >> v1:172.21.15.92:6789/0 conn(0x7fa978103a20 legacy=0x7fa97810ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:56.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:56.072+0000 7fa96e7fc700 1 -- 172.21.15.90:0/199288222 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9781aae90 con 0x7fa97810b830 2024-04-14T15:21:56.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:56.072+0000 7fa97f74d700 1 -- 172.21.15.90:0/199288222 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa9781a7a20 con 0x7fa97810b830 2024-04-14T15:21:56.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:56.072+0000 7fa97f74d700 10 client.37623.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:21:56.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:56.072+0000 7fa97f74d700 1 -- 172.21.15.90:0/199288222 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa9781a7ee0 con 0x7fa97810b830 2024-04-14T15:21:56.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:56.072+0000 7fa97f74d700 1 librados: init done 2024-04-14T15:21:56.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:56.072+0000 7fa96e7fc700 1 -- 172.21.15.90:0/199288222 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fa968001c90 con 0x7fa97810b830 2024-04-14T15:21:56.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:56.072+0000 7fa96e7fc700 1 -- 172.21.15.90:0/199288222 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fa968005ca0 con 0x7fa97810b830 2024-04-14T15:21:56.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:56.073+0000 7fa97f74d700 1 -- 172.21.15.90:0/199288222 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa95c005120 con 0x7fa97810b830 2024-04-14T15:21:56.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:56.074+0000 7fa96e7fc700 1 -- 172.21.15.90:0/199288222 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fa96801c6b0 con 0x7fa97810b830 2024-04-14T15:21:56.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:56.074+0000 7fa96e7fc700 1 -- 172.21.15.90:0/199288222 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(580..580 src has 467..580) ==== 8308+0+0 (unknown 2946689253 0 0) 0x7fa96808a9c0 con 0x7fa97810b830 2024-04-14T15:21:56.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:56.074+0000 7fa96e7fc700 10 client.37623.objecter ms_dispatch 0x7fa978001550 osd_map(580..580 src has 467..580) 2024-04-14T15:21:56.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:56.074+0000 7fa96e7fc700 3 client.37623.objecter handle_osd_map got epochs [580,580] > 0 2024-04-14T15:21:56.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:56.074+0000 7fa96e7fc700 3 client.37623.objecter handle_osd_map decoding full epoch 580 2024-04-14T15:21:56.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:56.075+0000 7fa96e7fc700 20 client.37623.objecter dump_active .. 0 homeless 2024-04-14T15:21:56.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:56.075+0000 7fa96e7fc700 10 client.37623.objecter ms_handle_connect 0x7fa9640754d0 2024-04-14T15:21:56.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:56.080+0000 7fa96e7fc700 1 -- 172.21.15.90:0/199288222 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fa96808ce20 con 0x7fa97810b830 2024-04-14T15:21:56.115 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:21:56.121 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:21:56.146 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:56.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:56.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:21:56.248 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:56.272 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:56.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:56.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:56.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:56.306+0000 7fa97f74d700 1 -- 172.21.15.90:0/199288222 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7fa95c005410 con 0x7fa97810b830 2024-04-14T15:21:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:55 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-98","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:21:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:55 smithi092 ceph-mon[33966]: osdmap e579: 8 total, 8 up, 7 in 2024-04-14T15:21:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:55 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:21:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:55 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:21:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:55 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-04-14T15:21:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:55 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-98", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:21:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:55 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]': finished 2024-04-14T15:21:56.374 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:21:56.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:56.399 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:56.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:21:56.500 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:21:56.524 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:56.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:56.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:21:56.626 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:21:56.653 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:56.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:56.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:21:56.755 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:21:56.781 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:56.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:56.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:21:56.883 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:21:56.907 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:56.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:56.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:21:56.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:56.953+0000 7fa96e7fc700 1 -- 172.21.15.90:0/199288222 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v581) ==== 91+0+0 (unknown 722281653 0 0) 0x7fa968057360 con 0x7fa97810b830 2024-04-14T15:21:56.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:56.955+0000 7fa97f74d700 10 librados: watch_flush enter 2024-04-14T15:21:56.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:56.955+0000 7fa97f74d700 10 librados: watch_flush exit 2024-04-14T15:21:56.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:56.955+0000 7fa97f74d700 20 client.37623.objecter shutdown clearing up homeless session... 2024-04-14T15:21:56.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:56.955+0000 7fa97f74d700 10 client.37623.objecter successfully canceled tick 2024-04-14T15:21:56.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:56.955+0000 7fa97f74d700 1 -- 172.21.15.90:0/199288222 >> v1:172.21.15.90:6800/420935427 conn(0x7fa9640754d0 legacy=0x7fa964077990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:56.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:56.955+0000 7fa97f74d700 1 -- 172.21.15.90:0/199288222 >> v1:172.21.15.90:6789/0 conn(0x7fa97810b830 legacy=0x7fa9781a2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:21:56.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:56.956+0000 7fa97f74d700 1 -- 172.21.15.90:0/199288222 shutdown_connections 2024-04-14T15:21:56.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:56.956+0000 7fa97f74d700 1 -- 172.21.15.90:0/199288222 >> 172.21.15.90:0/199288222 conn(0x7fa9780fd290 msgr2=0x7fa978111770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:21:56.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:56.956+0000 7fa97f74d700 1 -- 172.21.15.90:0/199288222 shutdown_connections 2024-04-14T15:21:56.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:56.956+0000 7fa97f74d700 1 -- 172.21.15.90:0/199288222 wait complete. 2024-04-14T15:21:56.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:21:56.956+0000 7fa97f74d700 1 librados: shutdown 2024-04-14T15:21:56.957 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is unset 2024-04-14T15:21:57.009 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:21:57.036 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:57.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:57.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:21:57.138 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:21:57.161 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:57.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:57.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:21:57.262 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:21:57.265 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:21:57.287 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:57.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:57.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:21:57.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:56 smithi092 ceph-mon[33966]: pgmap v849: 248 pgs: 32 creating+peering, 4 active+remapped+backfill_toofull, 212 active+clean; 8.4 MiB data, 495 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 op/s; 2/1200 objects misplaced (0.167%) 2024-04-14T15:21:57.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:56 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:21:57.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:56 smithi092 ceph-mon[33966]: osdmap e580: 8 total, 8 up, 7 in 2024-04-14T15:21:57.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:56 smithi092 ceph-mon[33966]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-04-14T15:21:57.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:56 smithi092 ceph-mon[33966]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 2 pgs peering) 2024-04-14T15:21:57.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:56 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-98", "mode": "writeback"}]: dispatch 2024-04-14T15:21:57.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:56 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-98", "mode": "writeback"}]: dispatch 2024-04-14T15:21:57.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:56 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:21:57.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:56 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:21:57.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:56 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:21:57.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:56 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/199288222' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:21:57.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:56 smithi092 ceph-mon[33966]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:21:57.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:56 smithi092 ceph-mon[33966]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:21:57.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:56 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-98", "mode": "writeback"}]': finished 2024-04-14T15:21:57.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:56 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/199288222' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-14T15:21:57.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:56 smithi092 ceph-mon[33966]: osdmap e581: 8 total, 8 up, 7 in 2024-04-14T15:21:57.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:56 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-98"}]: dispatch 2024-04-14T15:21:57.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:56 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-98"}]: dispatch 2024-04-14T15:21:57.374 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:56 smithi090 ceph-mon[30399]: pgmap v849: 248 pgs: 32 creating+peering, 4 active+remapped+backfill_toofull, 212 active+clean; 8.4 MiB data, 495 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 op/s; 2/1200 objects misplaced (0.167%) 2024-04-14T15:21:57.374 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:56 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:21:57.374 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:56 smithi090 ceph-mon[30399]: osdmap e580: 8 total, 8 up, 7 in 2024-04-14T15:21:57.374 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:56 smithi090 ceph-mon[30399]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-04-14T15:21:57.374 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:56 smithi090 ceph-mon[30399]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 2 pgs peering) 2024-04-14T15:21:57.374 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:56 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-98", "mode": "writeback"}]: dispatch 2024-04-14T15:21:57.374 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:56 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-98", "mode": "writeback"}]: dispatch 2024-04-14T15:21:57.375 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:56 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:21:57.375 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:56 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:21:57.375 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:56 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:21:57.375 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:56 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/199288222' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:21:57.375 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:56 smithi090 ceph-mon[30399]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:21:57.375 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:56 smithi090 ceph-mon[30399]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:21:57.375 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:56 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-98", "mode": "writeback"}]': finished 2024-04-14T15:21:57.375 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:56 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/199288222' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-14T15:21:57.375 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:56 smithi090 ceph-mon[30399]: osdmap e581: 8 total, 8 up, 7 in 2024-04-14T15:21:57.375 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:56 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-98"}]: dispatch 2024-04-14T15:21:57.375 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:56 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-98"}]: dispatch 2024-04-14T15:21:57.375 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:56 smithi090 ceph-mon[41670]: pgmap v849: 248 pgs: 32 creating+peering, 4 active+remapped+backfill_toofull, 212 active+clean; 8.4 MiB data, 495 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 op/s; 2/1200 objects misplaced (0.167%) 2024-04-14T15:21:57.375 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:56 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:21:57.376 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:56 smithi090 ceph-mon[41670]: osdmap e580: 8 total, 8 up, 7 in 2024-04-14T15:21:57.376 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:56 smithi090 ceph-mon[41670]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-04-14T15:21:57.376 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:56 smithi090 ceph-mon[41670]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 2 pgs peering) 2024-04-14T15:21:57.376 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:56 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-98", "mode": "writeback"}]: dispatch 2024-04-14T15:21:57.376 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:56 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-98", "mode": "writeback"}]: dispatch 2024-04-14T15:21:57.376 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:56 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:21:57.376 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:56 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:21:57.376 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:56 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:21:57.376 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:56 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/199288222' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:21:57.376 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:56 smithi090 ceph-mon[41670]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:21:57.376 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:56 smithi090 ceph-mon[41670]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:21:57.376 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:56 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-98", "mode": "writeback"}]': finished 2024-04-14T15:21:57.376 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:56 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/199288222' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-14T15:21:57.376 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:56 smithi090 ceph-mon[41670]: osdmap e581: 8 total, 8 up, 7 in 2024-04-14T15:21:57.376 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:56 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-98"}]: dispatch 2024-04-14T15:21:57.377 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:56 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-98"}]: dispatch 2024-04-14T15:21:57.389 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:21:57.412 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:57.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:57.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:21:57.514 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:21:57.542 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:57.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:57.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:21:57.643 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:21:57.670 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:57.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:57.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:21:57.772 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:21:57.796 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:57.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:57.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:21:57.898 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:21:57.921 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:57.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:57.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:21:58.023 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:21:58.046 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:58.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:58.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:21:58.148 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:58.175 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:58.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:58.176 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:58.276 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:21:58.303 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:58.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:58.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:21:58.405 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:21:58.428 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:58.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:58.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:21:58.530 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:21:58.554 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:58.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:58.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:21:58.655 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:21:58.682 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:58.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:58.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:21:58.784 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:21:58.807 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:58.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:58.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:21:58.909 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:21:58.936 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:58.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:58.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:21:59.037 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:59.063 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:59.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:59.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:59.110 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:58 smithi090 ceph-mon[41670]: pgmap v852: 248 pgs: 32 creating+peering, 4 active+remapped+backfill_toofull, 212 active+clean; 8.4 MiB data, 495 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s; 2/1200 objects misplaced (0.167%) 2024-04-14T15:21:59.110 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:58 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-98"}]': finished 2024-04-14T15:21:59.111 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:58 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-14T15:21:59.111 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:58 smithi090 ceph-mon[41670]: osdmap e582: 8 total, 8 up, 7 in 2024-04-14T15:21:59.111 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:58 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-14T15:21:59.111 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:58 smithi090 ceph-mon[41670]: 11.13 scrub starts 2024-04-14T15:21:59.111 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:58 smithi090 ceph-mon[41670]: 11.13 scrub ok 2024-04-14T15:21:59.111 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:58 smithi090 ceph-mon[41670]: 1.f scrub starts 2024-04-14T15:21:59.111 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:58 smithi090 ceph-mon[41670]: 1.f scrub ok 2024-04-14T15:21:59.111 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:58 smithi090 ceph-mon[41670]: 1.1 scrub starts 2024-04-14T15:21:59.111 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:58 smithi090 ceph-mon[41670]: 1.1 scrub ok 2024-04-14T15:21:59.111 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:58 smithi090 ceph-mon[30399]: pgmap v852: 248 pgs: 32 creating+peering, 4 active+remapped+backfill_toofull, 212 active+clean; 8.4 MiB data, 495 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s; 2/1200 objects misplaced (0.167%) 2024-04-14T15:21:59.111 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:58 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-98"}]': finished 2024-04-14T15:21:59.111 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:58 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-14T15:21:59.112 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:58 smithi090 ceph-mon[30399]: osdmap e582: 8 total, 8 up, 7 in 2024-04-14T15:21:59.112 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:58 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-14T15:21:59.112 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:58 smithi090 ceph-mon[30399]: 11.13 scrub starts 2024-04-14T15:21:59.112 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:58 smithi090 ceph-mon[30399]: 11.13 scrub ok 2024-04-14T15:21:59.112 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:58 smithi090 ceph-mon[30399]: 1.f scrub starts 2024-04-14T15:21:59.112 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:58 smithi090 ceph-mon[30399]: 1.f scrub ok 2024-04-14T15:21:59.112 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:58 smithi090 ceph-mon[30399]: 1.1 scrub starts 2024-04-14T15:21:59.112 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:58 smithi090 ceph-mon[30399]: 1.1 scrub ok 2024-04-14T15:21:59.164 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:21:59.189 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:59.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:59.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:21:59.291 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:21:59.316 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:21:59.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:59.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:21:59.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:58 smithi092 ceph-mon[33966]: pgmap v852: 248 pgs: 32 creating+peering, 4 active+remapped+backfill_toofull, 212 active+clean; 8.4 MiB data, 495 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s; 2/1200 objects misplaced (0.167%) 2024-04-14T15:21:59.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:58 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-98"}]': finished 2024-04-14T15:21:59.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:58 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-14T15:21:59.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:58 smithi092 ceph-mon[33966]: osdmap e582: 8 total, 8 up, 7 in 2024-04-14T15:21:59.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:58 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-14T15:21:59.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:58 smithi092 ceph-mon[33966]: 11.13 scrub starts 2024-04-14T15:21:59.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:58 smithi092 ceph-mon[33966]: 11.13 scrub ok 2024-04-14T15:21:59.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:58 smithi092 ceph-mon[33966]: 1.f scrub starts 2024-04-14T15:21:59.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:58 smithi092 ceph-mon[33966]: 1.f scrub ok 2024-04-14T15:21:59.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:58 smithi092 ceph-mon[33966]: 1.1 scrub starts 2024-04-14T15:21:59.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:58 smithi092 ceph-mon[33966]: 1.1 scrub ok 2024-04-14T15:21:59.418 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:21:59.442 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:59.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:59.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:21:59.543 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:21:59.567 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:59.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:59.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:21:59.668 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:21:59.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:59.692 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:59.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:21:59.793 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:21:59.816 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:59.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:59.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:21:59.858 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-14T15:21:59.918 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:21:59.941 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:21:59.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:21:59.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:22:00.043 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:22:00.071 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:00.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:00.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:22:00.173 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:22:00.200 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:00.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:00.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:22:00.222 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:59 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:22:00.222 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:59 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:22:00.222 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:59 smithi090 ceph-mon[30399]: 1.8 scrub starts 2024-04-14T15:22:00.222 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:59 smithi090 ceph-mon[30399]: 1.8 scrub ok 2024-04-14T15:22:00.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:59 smithi090 ceph-mon[30399]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:22:00.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:59 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-98","var": "hit_set_type","val": "bloom"}]': finished 2024-04-14T15:22:00.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:59 smithi090 ceph-mon[30399]: osdmap e583: 8 total, 8 up, 7 in 2024-04-14T15:22:00.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:59 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-98","var": "hit_set_count","val": "1"}]: dispatch 2024-04-14T15:22:00.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:59 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-98","var": "hit_set_count","val": "1"}]: dispatch 2024-04-14T15:22:00.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:59 smithi090 ceph-mon[30399]: 5.7 scrub starts 2024-04-14T15:22:00.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:59 smithi090 ceph-mon[30399]: 5.7 scrub ok 2024-04-14T15:22:00.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:59 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-04-14T15:22:00.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:59 smithi090 ceph-mon[30399]: 1.12 scrub starts 2024-04-14T15:22:00.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:59 smithi090 ceph-mon[30399]: 1.12 scrub ok 2024-04-14T15:22:00.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:59 smithi090 ceph-mon[30399]: 110.4 scrub starts 2024-04-14T15:22:00.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:59 smithi090 ceph-mon[30399]: 110.4 scrub ok 2024-04-14T15:22:00.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:59 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-98","var": "hit_set_count","val": "1"}]': finished 2024-04-14T15:22:00.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:59 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]': finished 2024-04-14T15:22:00.224 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:59 smithi090 ceph-mon[30399]: osdmap e584: 8 total, 8 up, 7 in 2024-04-14T15:22:00.224 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:59 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-98","var": "hit_set_period","val": "600"}]: dispatch 2024-04-14T15:22:00.224 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:21:59 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-98","var": "hit_set_period","val": "600"}]: dispatch 2024-04-14T15:22:00.224 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:59 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:22:00.224 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:59 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:22:00.224 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:59 smithi090 ceph-mon[41670]: 1.8 scrub starts 2024-04-14T15:22:00.224 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:59 smithi090 ceph-mon[41670]: 1.8 scrub ok 2024-04-14T15:22:00.224 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:59 smithi090 ceph-mon[41670]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:22:00.224 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:59 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-98","var": "hit_set_type","val": "bloom"}]': finished 2024-04-14T15:22:00.224 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:59 smithi090 ceph-mon[41670]: osdmap e583: 8 total, 8 up, 7 in 2024-04-14T15:22:00.224 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:59 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-98","var": "hit_set_count","val": "1"}]: dispatch 2024-04-14T15:22:00.224 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:59 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-98","var": "hit_set_count","val": "1"}]: dispatch 2024-04-14T15:22:00.225 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:59 smithi090 ceph-mon[41670]: 5.7 scrub starts 2024-04-14T15:22:00.225 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:59 smithi090 ceph-mon[41670]: 5.7 scrub ok 2024-04-14T15:22:00.225 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:59 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-04-14T15:22:00.225 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:59 smithi090 ceph-mon[41670]: 1.12 scrub starts 2024-04-14T15:22:00.225 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:59 smithi090 ceph-mon[41670]: 1.12 scrub ok 2024-04-14T15:22:00.225 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:59 smithi090 ceph-mon[41670]: 110.4 scrub starts 2024-04-14T15:22:00.225 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:59 smithi090 ceph-mon[41670]: 110.4 scrub ok 2024-04-14T15:22:00.225 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:59 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-98","var": "hit_set_count","val": "1"}]': finished 2024-04-14T15:22:00.225 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:59 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]': finished 2024-04-14T15:22:00.225 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:59 smithi090 ceph-mon[41670]: osdmap e584: 8 total, 8 up, 7 in 2024-04-14T15:22:00.225 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:59 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-98","var": "hit_set_period","val": "600"}]: dispatch 2024-04-14T15:22:00.225 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:21:59 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-98","var": "hit_set_period","val": "600"}]: dispatch 2024-04-14T15:22:00.302 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:22:00.328 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:00.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:00.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:22:00.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:59 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:22:00.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:59 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:22:00.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:59 smithi092 ceph-mon[33966]: 1.8 scrub starts 2024-04-14T15:22:00.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:59 smithi092 ceph-mon[33966]: 1.8 scrub ok 2024-04-14T15:22:00.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:59 smithi092 ceph-mon[33966]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:22:00.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:59 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-98","var": "hit_set_type","val": "bloom"}]': finished 2024-04-14T15:22:00.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:59 smithi092 ceph-mon[33966]: osdmap e583: 8 total, 8 up, 7 in 2024-04-14T15:22:00.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:59 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-98","var": "hit_set_count","val": "1"}]: dispatch 2024-04-14T15:22:00.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:59 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-98","var": "hit_set_count","val": "1"}]: dispatch 2024-04-14T15:22:00.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:59 smithi092 ceph-mon[33966]: 5.7 scrub starts 2024-04-14T15:22:00.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:59 smithi092 ceph-mon[33966]: 5.7 scrub ok 2024-04-14T15:22:00.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:59 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-04-14T15:22:00.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:59 smithi092 ceph-mon[33966]: 1.12 scrub starts 2024-04-14T15:22:00.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:59 smithi092 ceph-mon[33966]: 1.12 scrub ok 2024-04-14T15:22:00.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:59 smithi092 ceph-mon[33966]: 110.4 scrub starts 2024-04-14T15:22:00.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:59 smithi092 ceph-mon[33966]: 110.4 scrub ok 2024-04-14T15:22:00.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:59 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-98","var": "hit_set_count","val": "1"}]': finished 2024-04-14T15:22:00.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:59 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]': finished 2024-04-14T15:22:00.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:59 smithi092 ceph-mon[33966]: osdmap e584: 8 total, 8 up, 7 in 2024-04-14T15:22:00.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:59 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-98","var": "hit_set_period","val": "600"}]: dispatch 2024-04-14T15:22:00.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:21:59 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-98","var": "hit_set_period","val": "600"}]: dispatch 2024-04-14T15:22:00.430 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:00.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.442+0000 7fda21a75700 1 -- 172.21.15.90:0/2371435288 >> v1:172.21.15.90:6789/0 conn(0x7fda1c108980 legacy=0x7fda1c108d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:00.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.442+0000 7fda21a75700 1 -- 172.21.15.90:0/2371435288 shutdown_connections 2024-04-14T15:22:00.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.442+0000 7fda21a75700 1 -- 172.21.15.90:0/2371435288 >> 172.21.15.90:0/2371435288 conn(0x7fda1c076aa0 msgr2=0x7fda1c076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:22:00.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.442+0000 7fda21a75700 1 -- 172.21.15.90:0/2371435288 shutdown_connections 2024-04-14T15:22:00.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.442+0000 7fda21a75700 1 -- 172.21.15.90:0/2371435288 wait complete. 2024-04-14T15:22:00.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.442+0000 7fda21a75700 1 librados: starting msgr at 2024-04-14T15:22:00.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.443+0000 7fda21a75700 1 librados: starting objecter 2024-04-14T15:22:00.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.443+0000 7fda21a75700 1 Processor -- start 2024-04-14T15:22:00.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.443+0000 7fda21a75700 1 -- start start 2024-04-14T15:22:00.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.443+0000 7fda21a75700 1 librados: setting wanted keys 2024-04-14T15:22:00.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.443+0000 7fda21a75700 1 librados: calling monclient init 2024-04-14T15:22:00.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.443+0000 7fda21a75700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fda1c108170 con 0x7fda1c10b6f0 2024-04-14T15:22:00.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.443+0000 7fda21a75700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fda1c1a8f00 con 0x7fda1c10f2f0 2024-04-14T15:22:00.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.443+0000 7fda21a75700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fda1c1aa0e0 con 0x7fda1c108980 2024-04-14T15:22:00.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.443+0000 7fda1affd700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fda1c10b6f0 0x7fda1c1056d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:36618/0 (socket says 172.21.15.90:36618) 2024-04-14T15:22:00.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.443+0000 7fda1affd700 1 -- 172.21.15.90:0/1116329828 learned_addr learned my addr 172.21.15.90:0/1116329828 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:22:00.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.443+0000 7fda18ff9700 10 client.?.objecter ms_handle_connect 0x7fda1c10b6f0 2024-04-14T15:22:00.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.443+0000 7fda18ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:22:00.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.443+0000 7fda18ff9700 10 client.?.objecter ms_handle_connect 0x7fda1c10f2f0 2024-04-14T15:22:00.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.443+0000 7fda18ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:22:00.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.443+0000 7fda18ff9700 10 client.?.objecter ms_handle_connect 0x7fda1c108980 2024-04-14T15:22:00.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.443+0000 7fda18ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:22:00.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.444+0000 7fda18ff9700 1 -- 172.21.15.90:0/1116329828 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4190233598 0 0) 0x7fda1c108170 con 0x7fda1c10b6f0 2024-04-14T15:22:00.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.444+0000 7fda18ff9700 1 -- 172.21.15.90:0/1116329828 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda040035b0 con 0x7fda1c10b6f0 2024-04-14T15:22:00.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.444+0000 7fda18ff9700 1 -- 172.21.15.90:0/1116329828 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 433170480 0 0) 0x7fda1c1a8f00 con 0x7fda1c10f2f0 2024-04-14T15:22:00.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.444+0000 7fda18ff9700 1 -- 172.21.15.90:0/1116329828 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda1c108170 con 0x7fda1c10f2f0 2024-04-14T15:22:00.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.444+0000 7fda18ff9700 1 -- 172.21.15.90:0/1116329828 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 152523324 0 0) 0x7fda1c1aa0e0 con 0x7fda1c108980 2024-04-14T15:22:00.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.444+0000 7fda18ff9700 1 -- 172.21.15.90:0/1116329828 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda1c1a8f00 con 0x7fda1c108980 2024-04-14T15:22:00.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.444+0000 7fda18ff9700 1 -- 172.21.15.90:0/1116329828 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4012540227 0 0) 0x7fda040035b0 con 0x7fda1c10b6f0 2024-04-14T15:22:00.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.444+0000 7fda18ff9700 1 -- 172.21.15.90:0/1116329828 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fda1c1aa0e0 con 0x7fda1c10b6f0 2024-04-14T15:22:00.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.444+0000 7fda18ff9700 1 -- 172.21.15.90:0/1116329828 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fda10001930 con 0x7fda1c10b6f0 2024-04-14T15:22:00.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.444+0000 7fda18ff9700 1 -- 172.21.15.90:0/1116329828 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1858946691 0 0) 0x7fda1c108170 con 0x7fda1c10f2f0 2024-04-14T15:22:00.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.444+0000 7fda18ff9700 1 -- 172.21.15.90:0/1116329828 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fda040035b0 con 0x7fda1c10f2f0 2024-04-14T15:22:00.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.444+0000 7fda18ff9700 1 -- 172.21.15.90:0/1116329828 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fda14001bb0 con 0x7fda1c10f2f0 2024-04-14T15:22:00.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.444+0000 7fda18ff9700 1 -- 172.21.15.90:0/1116329828 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 37304806 0 0) 0x7fda1c1a8f00 con 0x7fda1c108980 2024-04-14T15:22:00.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.444+0000 7fda18ff9700 1 -- 172.21.15.90:0/1116329828 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fda1c108170 con 0x7fda1c108980 2024-04-14T15:22:00.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.444+0000 7fda18ff9700 1 -- 172.21.15.90:0/1116329828 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fda0c001430 con 0x7fda1c108980 2024-04-14T15:22:00.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.444+0000 7fda18ff9700 1 -- 172.21.15.90:0/1116329828 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1665743008 0 0) 0x7fda1c1aa0e0 con 0x7fda1c10b6f0 2024-04-14T15:22:00.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.444+0000 7fda18ff9700 1 -- 172.21.15.90:0/1116329828 >> v1:172.21.15.90:6790/0 conn(0x7fda1c108980 legacy=0x7fda1c104fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:00.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.444+0000 7fda18ff9700 1 -- 172.21.15.90:0/1116329828 >> v1:172.21.15.92:6789/0 conn(0x7fda1c10f2f0 legacy=0x7fda1c1a67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:00.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.444+0000 7fda18ff9700 1 -- 172.21.15.90:0/1116329828 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda1c1ab2c0 con 0x7fda1c10b6f0 2024-04-14T15:22:00.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.444+0000 7fda21a75700 1 -- 172.21.15.90:0/1116329828 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fda1c1a9130 con 0x7fda1c10b6f0 2024-04-14T15:22:00.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.444+0000 7fda21a75700 10 client.37629.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:22:00.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.444+0000 7fda21a75700 1 -- 172.21.15.90:0/1116329828 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fda1c1a95f0 con 0x7fda1c10b6f0 2024-04-14T15:22:00.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.444+0000 7fda21a75700 1 librados: init done 2024-04-14T15:22:00.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.444+0000 7fda18ff9700 1 -- 172.21.15.90:0/1116329828 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fda10001c50 con 0x7fda1c10b6f0 2024-04-14T15:22:00.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.444+0000 7fda18ff9700 1 -- 172.21.15.90:0/1116329828 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fda10005ca0 con 0x7fda1c10b6f0 2024-04-14T15:22:00.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.445+0000 7fda21a75700 10 client.37629.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:22:00.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.445+0000 7fda21a75700 15 client.37629.objecter _session_command_op_assign -1 1 2024-04-14T15:22:00.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.445+0000 7fda21a75700 20 client.37629.objecter _get_session osd=-1 returning homeless 2024-04-14T15:22:00.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.445+0000 7fda21a75700 10 client.37629.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:22:00.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.445+0000 7fda21a75700 1 -- 172.21.15.90:0/1116329828 --> v1:172.21.15.90:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fda08002920 con 0x7fda1c10b6f0 2024-04-14T15:22:00.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.446+0000 7fda18ff9700 1 -- 172.21.15.90:0/1116329828 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fda10003950 con 0x7fda1c10b6f0 2024-04-14T15:22:00.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.446+0000 7fda18ff9700 1 -- 172.21.15.90:0/1116329828 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(584..584 src has 467..584) ==== 8334+0+0 (unknown 2329834852 0 0) 0x7fda1008aac0 con 0x7fda1c10b6f0 2024-04-14T15:22:00.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.446+0000 7fda18ff9700 10 client.37629.objecter ms_dispatch 0x7fda1c001550 osd_map(584..584 src has 467..584) 2024-04-14T15:22:00.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.446+0000 7fda18ff9700 3 client.37629.objecter handle_osd_map got epochs [584,584] > 0 2024-04-14T15:22:00.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.446+0000 7fda18ff9700 3 client.37629.objecter handle_osd_map decoding full epoch 584 2024-04-14T15:22:00.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.447+0000 7fda18ff9700 10 client.37629.objecter checking command 1 2024-04-14T15:22:00.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.447+0000 7fda18ff9700 20 client.37629.objecter _get_session s=0x7fda0407ef90 osd=0 3 2024-04-14T15:22:00.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.447+0000 7fda18ff9700 20 client.37629.objecter put_session s=0x7fda0407ef90 osd=0 3 2024-04-14T15:22:00.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.447+0000 7fda18ff9700 15 client.37629.objecter _session_command_op_remove -1 1 2024-04-14T15:22:00.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.447+0000 7fda18ff9700 20 client.37629.objecter _get_session s=0x7fda0407ef90 osd=0 3 2024-04-14T15:22:00.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.447+0000 7fda18ff9700 20 client.37629.objecter get_session s=0x7fda0407ef90 osd=0 3 2024-04-14T15:22:00.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.447+0000 7fda18ff9700 15 client.37629.objecter _session_command_op_assign 0 1 2024-04-14T15:22:00.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.447+0000 7fda18ff9700 20 client.37629.objecter put_session s=0x7fda0407ef90 osd=0 4 2024-04-14T15:22:00.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.447+0000 7fda18ff9700 10 client.37629.objecter _send_command 1 2024-04-14T15:22:00.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.447+0000 7fda18ff9700 1 -- 172.21.15.90:0/1116329828 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fda04083350 con 0x7fda0407f640 2024-04-14T15:22:00.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.447+0000 7fda18ff9700 20 client.37629.objecter dump_active .. 0 homeless 2024-04-14T15:22:00.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.447+0000 7fda18ff9700 1 -- 172.21.15.90:0/1116329828 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_get_version_reply(handle=1 version=584) ==== 24+0+0 (unknown 3172944404 0 0) 0x7fda1008aea0 con 0x7fda1c10b6f0 2024-04-14T15:22:00.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.447+0000 7fda18ff9700 10 client.37629.objecter ms_handle_connect 0x7fda04071330 2024-04-14T15:22:00.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.447+0000 7fda18ff9700 10 client.37629.objecter ms_handle_connect 0x7fda0407f640 2024-04-14T15:22:00.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.447+0000 7fda18ff9700 1 -- 172.21.15.90:0/1116329828 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fda04083350 con 0x7fda0407f640 2024-04-14T15:22:00.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.448+0000 7fda18ff9700 10 client.37629.objecter ms_dispatch 0x7fda1c001550 command_reply(tid 1: 0 ) 2024-04-14T15:22:00.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.448+0000 7fda18ff9700 10 client.37629.objecter _finish_command 1 = system:0 2024-04-14T15:22:00.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.448+0000 7fda18ff9700 20 client.37629.objecter put_session s=0x7fda0407ef90 osd=0 4 2024-04-14T15:22:00.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.448+0000 7fda18ff9700 15 client.37629.objecter _session_command_op_remove 0 1 2024-04-14T15:22:00.456 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:00.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:00.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:00.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.467+0000 7fda21a75700 10 client.37629.objecter _submit_command 2 [{"prefix": "reset_purged_snaps_last"}] 2024-04-14T15:22:00.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.467+0000 7fda21a75700 15 client.37629.objecter _session_command_op_assign -1 2 2024-04-14T15:22:00.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.467+0000 7fda21a75700 20 client.37629.objecter _get_session s=0x7fda0407ef90 osd=0 3 2024-04-14T15:22:00.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.467+0000 7fda21a75700 20 client.37629.objecter put_session s=0x7fda0407ef90 osd=0 3 2024-04-14T15:22:00.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.467+0000 7fda21a75700 20 client.37629.objecter _get_session s=0x7fda0407ef90 osd=0 3 2024-04-14T15:22:00.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.467+0000 7fda21a75700 15 client.37629.objecter _session_command_op_remove -1 2 2024-04-14T15:22:00.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.467+0000 7fda21a75700 20 client.37629.objecter get_session s=0x7fda0407ef90 osd=0 3 2024-04-14T15:22:00.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.467+0000 7fda21a75700 15 client.37629.objecter _session_command_op_assign 0 2 2024-04-14T15:22:00.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.467+0000 7fda21a75700 20 client.37629.objecter put_session s=0x7fda0407ef90 osd=0 4 2024-04-14T15:22:00.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.467+0000 7fda21a75700 10 client.37629.objecter _send_command 2 2024-04-14T15:22:00.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.467+0000 7fda21a75700 1 -- 172.21.15.90:0/1116329828 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) -- 0x7fda080011f0 con 0x7fda0407f640 2024-04-14T15:22:00.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.467+0000 7fda18ff9700 1 -- 172.21.15.90:0/1116329828 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 ) ==== 8+0+0 (unknown 0 0 0) 0x7fda080011f0 con 0x7fda0407f640 2024-04-14T15:22:00.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.467+0000 7fda18ff9700 10 client.37629.objecter ms_dispatch 0x7fda1c001550 command_reply(tid 2: 0 ) 2024-04-14T15:22:00.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.467+0000 7fda18ff9700 10 client.37629.objecter _finish_command 2 = system:0 2024-04-14T15:22:00.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.467+0000 7fda18ff9700 20 client.37629.objecter put_session s=0x7fda0407ef90 osd=0 4 2024-04-14T15:22:00.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.467+0000 7fda18ff9700 15 client.37629.objecter _session_command_op_remove 0 2 2024-04-14T15:22:00.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.468+0000 7fda21a75700 10 librados: watch_flush enter 2024-04-14T15:22:00.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.468+0000 7fda21a75700 10 librados: watch_flush exit 2024-04-14T15:22:00.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.468+0000 7fda21a75700 10 client.37629.objecter close_session for osd.0 2024-04-14T15:22:00.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.468+0000 7fda21a75700 1 -- 172.21.15.90:0/1116329828 >> v1:172.21.15.90:6801/3761806885 conn(0x7fda0407f640 legacy=0x7fda04081ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:00.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.468+0000 7fda21a75700 20 client.37629.objecter put_session s=0x7fda0407ef90 osd=0 1 2024-04-14T15:22:00.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.468+0000 7fda21a75700 20 client.37629.objecter shutdown clearing up homeless session... 2024-04-14T15:22:00.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.468+0000 7fda21a75700 10 client.37629.objecter successfully canceled tick 2024-04-14T15:22:00.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.468+0000 7fda21a75700 1 -- 172.21.15.90:0/1116329828 >> v1:172.21.15.90:6800/420935427 conn(0x7fda04071330 legacy=0x7fda040737f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:00.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.468+0000 7fda21a75700 1 -- 172.21.15.90:0/1116329828 >> v1:172.21.15.90:6789/0 conn(0x7fda1c10b6f0 legacy=0x7fda1c1056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:00.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.468+0000 7fda1bfff700 1 -- 172.21.15.90:0/1116329828 reap_dead start 2024-04-14T15:22:00.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.468+0000 7fda21a75700 1 -- 172.21.15.90:0/1116329828 shutdown_connections 2024-04-14T15:22:00.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.468+0000 7fda21a75700 1 -- 172.21.15.90:0/1116329828 >> 172.21.15.90:0/1116329828 conn(0x7fda1c076aa0 msgr2=0x7fda1c109fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:22:00.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.468+0000 7fda21a75700 1 -- 172.21.15.90:0/1116329828 shutdown_connections 2024-04-14T15:22:00.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.468+0000 7fda21a75700 1 -- 172.21.15.90:0/1116329828 wait complete. 2024-04-14T15:22:00.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:00.468+0000 7fda21a75700 1 librados: shutdown 2024-04-14T15:22:00.558 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:22:00.585 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:00.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:00.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:22:00.687 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:22:00.711 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:00.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:00.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:22:00.813 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:22:00.840 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:00.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:00.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:22:00.942 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:22:00.970 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:00.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:00.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:22:01.072 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:22:01.095 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:01.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:01.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:22:01.197 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:22:01.223 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:01.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:01.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:22:01.324 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:22:01.336 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:00 smithi090 ceph-mon[41670]: pgmap v855: 247 pgs: 23 creating+peering, 4 active+remapped+backfill_toofull, 220 active+clean; 8.4 MiB data, 513 MiB used, 625 GiB / 626 GiB avail; 2/1200 objects misplaced (0.167%) 2024-04-14T15:22:01.336 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:00 smithi090 ceph-mon[41670]: 110.a scrub starts 2024-04-14T15:22:01.336 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:00 smithi090 ceph-mon[41670]: 110.a scrub ok 2024-04-14T15:22:01.336 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:00 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-98","var": "hit_set_period","val": "600"}]': finished 2024-04-14T15:22:01.336 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:00 smithi090 ceph-mon[41670]: osdmap e585: 8 total, 8 up, 7 in 2024-04-14T15:22:01.336 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:00 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-98","var": "target_max_objects","val": "250"}]: dispatch 2024-04-14T15:22:01.336 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:00 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-98","var": "target_max_objects","val": "250"}]: dispatch 2024-04-14T15:22:01.336 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:00 smithi090 ceph-mon[30399]: pgmap v855: 247 pgs: 23 creating+peering, 4 active+remapped+backfill_toofull, 220 active+clean; 8.4 MiB data, 513 MiB used, 625 GiB / 626 GiB avail; 2/1200 objects misplaced (0.167%) 2024-04-14T15:22:01.336 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:00 smithi090 ceph-mon[30399]: 110.a scrub starts 2024-04-14T15:22:01.337 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:00 smithi090 ceph-mon[30399]: 110.a scrub ok 2024-04-14T15:22:01.337 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:00 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-98","var": "hit_set_period","val": "600"}]': finished 2024-04-14T15:22:01.337 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:00 smithi090 ceph-mon[30399]: osdmap e585: 8 total, 8 up, 7 in 2024-04-14T15:22:01.337 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:00 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-98","var": "target_max_objects","val": "250"}]: dispatch 2024-04-14T15:22:01.337 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:00 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-98","var": "target_max_objects","val": "250"}]: dispatch 2024-04-14T15:22:01.349 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:01.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:01.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:22:01.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:00 smithi092 ceph-mon[33966]: pgmap v855: 247 pgs: 23 creating+peering, 4 active+remapped+backfill_toofull, 220 active+clean; 8.4 MiB data, 513 MiB used, 625 GiB / 626 GiB avail; 2/1200 objects misplaced (0.167%) 2024-04-14T15:22:01.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:00 smithi092 ceph-mon[33966]: 110.a scrub starts 2024-04-14T15:22:01.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:00 smithi092 ceph-mon[33966]: 110.a scrub ok 2024-04-14T15:22:01.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:00 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-98","var": "hit_set_period","val": "600"}]': finished 2024-04-14T15:22:01.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:00 smithi092 ceph-mon[33966]: osdmap e585: 8 total, 8 up, 7 in 2024-04-14T15:22:01.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:00 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-98","var": "target_max_objects","val": "250"}]: dispatch 2024-04-14T15:22:01.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:00 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-98","var": "target_max_objects","val": "250"}]: dispatch 2024-04-14T15:22:01.451 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:22:01.475 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:01.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:01.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:22:01.577 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:22:01.602 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:01.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:01.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:22:01.704 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:22:01.729 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:01.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:01.730 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:22:01.831 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:22:01.855 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:01.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:01.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:22:01.957 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:22:01.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:01.990 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:01.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:22:02.091 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:22:02.110 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:01 smithi090 ceph-mon[41670]: 1.10 scrub starts 2024-04-14T15:22:02.110 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:01 smithi090 ceph-mon[41670]: 1.10 scrub ok 2024-04-14T15:22:02.110 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:01 smithi090 ceph-mon[41670]: 110.1c scrub starts 2024-04-14T15:22:02.110 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:01 smithi090 ceph-mon[41670]: 110.1c scrub ok 2024-04-14T15:22:02.110 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:01 smithi090 ceph-mon[41670]: 5.15 scrub starts 2024-04-14T15:22:02.111 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:01 smithi090 ceph-mon[41670]: 5.15 scrub ok 2024-04-14T15:22:02.111 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:01 smithi090 ceph-mon[41670]: 4.14 scrub starts 2024-04-14T15:22:02.111 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:01 smithi090 ceph-mon[41670]: 4.14 scrub ok 2024-04-14T15:22:02.111 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:01 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-98","var": "target_max_objects","val": "250"}]': finished 2024-04-14T15:22:02.111 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:01 smithi090 ceph-mon[41670]: osdmap e586: 8 total, 8 up, 7 in 2024-04-14T15:22:02.111 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:01 smithi090 ceph-mon[30399]: 1.10 scrub starts 2024-04-14T15:22:02.111 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:01 smithi090 ceph-mon[30399]: 1.10 scrub ok 2024-04-14T15:22:02.111 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:01 smithi090 ceph-mon[30399]: 110.1c scrub starts 2024-04-14T15:22:02.111 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:01 smithi090 ceph-mon[30399]: 110.1c scrub ok 2024-04-14T15:22:02.111 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:01 smithi090 ceph-mon[30399]: 5.15 scrub starts 2024-04-14T15:22:02.111 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:01 smithi090 ceph-mon[30399]: 5.15 scrub ok 2024-04-14T15:22:02.111 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:01 smithi090 ceph-mon[30399]: 4.14 scrub starts 2024-04-14T15:22:02.111 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:01 smithi090 ceph-mon[30399]: 4.14 scrub ok 2024-04-14T15:22:02.112 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:01 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-98","var": "target_max_objects","val": "250"}]': finished 2024-04-14T15:22:02.112 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:01 smithi090 ceph-mon[30399]: osdmap e586: 8 total, 8 up, 7 in 2024-04-14T15:22:02.116 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:02.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:02.117 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:22:02.218 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:02.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:02.243 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:02.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:02.344 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:22:02.369 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:02.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:02.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:22:02.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:01 smithi092 ceph-mon[33966]: 1.10 scrub starts 2024-04-14T15:22:02.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:01 smithi092 ceph-mon[33966]: 1.10 scrub ok 2024-04-14T15:22:02.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:01 smithi092 ceph-mon[33966]: 110.1c scrub starts 2024-04-14T15:22:02.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:01 smithi092 ceph-mon[33966]: 110.1c scrub ok 2024-04-14T15:22:02.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:01 smithi092 ceph-mon[33966]: 5.15 scrub starts 2024-04-14T15:22:02.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:01 smithi092 ceph-mon[33966]: 5.15 scrub ok 2024-04-14T15:22:02.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:01 smithi092 ceph-mon[33966]: 4.14 scrub starts 2024-04-14T15:22:02.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:01 smithi092 ceph-mon[33966]: 4.14 scrub ok 2024-04-14T15:22:02.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:01 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-98","var": "target_max_objects","val": "250"}]': finished 2024-04-14T15:22:02.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:01 smithi092 ceph-mon[33966]: osdmap e586: 8 total, 8 up, 7 in 2024-04-14T15:22:02.471 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:22:02.496 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:02.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:02.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:22:02.597 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:22:02.626 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:02.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:02.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:22:02.728 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:02.752 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:02.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:02.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:02.854 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:22:02.882 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:02.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:02.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:22:02.983 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:22:03.010 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:03.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:03.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:22:03.111 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:22:03.137 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:03.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:03.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:22:03.146 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:02 smithi090 ceph-mon[41670]: pgmap v858: 247 pgs: 4 active+remapped+backfill_toofull, 243 active+clean; 8.4 MiB data, 497 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 2/1203 objects misplaced (0.166%) 2024-04-14T15:22:03.146 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:02 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:22:03.147 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:02 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:22:03.147 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:02 smithi090 ceph-mon[41670]: 4.d scrub starts 2024-04-14T15:22:03.147 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:02 smithi090 ceph-mon[41670]: 4.d scrub ok 2024-04-14T15:22:03.147 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:02 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:22:03.147 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:02 smithi090 ceph-mon[41670]: osdmap e587: 8 total, 8 up, 7 in 2024-04-14T15:22:03.147 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:02 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-98"}]: dispatch 2024-04-14T15:22:03.147 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:02 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-98"}]: dispatch 2024-04-14T15:22:03.147 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:02 smithi090 ceph-mon[30399]: pgmap v858: 247 pgs: 4 active+remapped+backfill_toofull, 243 active+clean; 8.4 MiB data, 497 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 2/1203 objects misplaced (0.166%) 2024-04-14T15:22:03.147 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:02 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:22:03.147 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:02 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:22:03.147 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:02 smithi090 ceph-mon[30399]: 4.d scrub starts 2024-04-14T15:22:03.147 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:02 smithi090 ceph-mon[30399]: 4.d scrub ok 2024-04-14T15:22:03.147 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:02 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:22:03.148 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:02 smithi090 ceph-mon[30399]: osdmap e587: 8 total, 8 up, 7 in 2024-04-14T15:22:03.148 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:02 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-98"}]: dispatch 2024-04-14T15:22:03.148 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:02 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-98"}]: dispatch 2024-04-14T15:22:03.172 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph tell osd.1 reset_purged_snaps_last 2024-04-14T15:22:03.239 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:22:03.264 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:03.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:03.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:22:03.366 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:22:03.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:02 smithi092 ceph-mon[33966]: pgmap v858: 247 pgs: 4 active+remapped+backfill_toofull, 243 active+clean; 8.4 MiB data, 497 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 2/1203 objects misplaced (0.166%) 2024-04-14T15:22:03.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:02 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:22:03.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:02 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:22:03.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:02 smithi092 ceph-mon[33966]: 4.d scrub starts 2024-04-14T15:22:03.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:02 smithi092 ceph-mon[33966]: 4.d scrub ok 2024-04-14T15:22:03.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:02 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:22:03.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:02 smithi092 ceph-mon[33966]: osdmap e587: 8 total, 8 up, 7 in 2024-04-14T15:22:03.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:02 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-98"}]: dispatch 2024-04-14T15:22:03.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:02 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-98"}]: dispatch 2024-04-14T15:22:03.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:03.391 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:03.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:22:03.492 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:22:03.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:03.517 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:03.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:22:03.618 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:22:03.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:03.643 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:03.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:22:03.673 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-14T15:22:03.673 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-14T15:22:03.673 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-14T15:22:03.673 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-14T15:22:03.744 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:22:03.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:03.768 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:03.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:22:03.869 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:22:03.892 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:03.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:03.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:22:03.994 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:22:04.017 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:04.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:04.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:22:04.119 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:04.144 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:04.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:04.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:04.245 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:22:04.268 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:04.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:04.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:22:04.370 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:22:04.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:03 smithi092 ceph-mon[33966]: 1.4 scrub starts 2024-04-14T15:22:04.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:03 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-04-14T15:22:04.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:03 smithi092 ceph-mon[33966]: 1.4 scrub ok 2024-04-14T15:22:04.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:03 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-98"}]': finished 2024-04-14T15:22:04.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:03 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]': finished 2024-04-14T15:22:04.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:03 smithi092 ceph-mon[33966]: osdmap e588: 8 total, 8 up, 7 in 2024-04-14T15:22:04.393 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:04.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:04.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:22:04.442 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:22:03 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:22:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:22:04.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:03 smithi090 ceph-mon[30399]: 1.4 scrub starts 2024-04-14T15:22:04.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:03 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-04-14T15:22:04.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:03 smithi090 ceph-mon[30399]: 1.4 scrub ok 2024-04-14T15:22:04.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:03 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-98"}]': finished 2024-04-14T15:22:04.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:03 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]': finished 2024-04-14T15:22:04.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:03 smithi090 ceph-mon[30399]: osdmap e588: 8 total, 8 up, 7 in 2024-04-14T15:22:04.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:03 smithi090 ceph-mon[41670]: 1.4 scrub starts 2024-04-14T15:22:04.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:03 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-04-14T15:22:04.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:03 smithi090 ceph-mon[41670]: 1.4 scrub ok 2024-04-14T15:22:04.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:03 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-98"}]': finished 2024-04-14T15:22:04.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:03 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]': finished 2024-04-14T15:22:04.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:03 smithi090 ceph-mon[41670]: osdmap e588: 8 total, 8 up, 7 in 2024-04-14T15:22:04.495 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:04.515 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-14T15:22:04.519 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:04.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:04.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:04.522 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:22:04.622 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:22:04.646 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:04.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:04.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:22:04.748 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:22:04.773 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:04.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:04.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:22:04.875 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:22:04.899 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:04.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:04.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:22:05.000 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:22:05.024 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:05.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:05.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:22:05.126 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:22:05.149 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:05.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:05.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:22:05.251 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:22:05.259 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:05 smithi090 ceph-mon[41670]: pgmap v861: 246 pgs: 4 active+remapped+backfill_toofull, 242 active+clean; 8.4 MiB data, 497 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 2/1203 objects misplaced (0.166%) 2024-04-14T15:22:05.259 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:05 smithi090 ceph-mon[41670]: osdmap e589: 8 total, 8 up, 7 in 2024-04-14T15:22:05.259 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:05 smithi090 ceph-mon[30399]: pgmap v861: 246 pgs: 4 active+remapped+backfill_toofull, 242 active+clean; 8.4 MiB data, 497 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 2/1203 objects misplaced (0.166%) 2024-04-14T15:22:05.259 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:05 smithi090 ceph-mon[30399]: osdmap e589: 8 total, 8 up, 7 in 2024-04-14T15:22:05.276 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:05.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:05.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:22:05.285 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:05 smithi092 ceph-mon[33966]: pgmap v861: 246 pgs: 4 active+remapped+backfill_toofull, 242 active+clean; 8.4 MiB data, 497 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 2/1203 objects misplaced (0.166%) 2024-04-14T15:22:05.285 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:05 smithi092 ceph-mon[33966]: osdmap e589: 8 total, 8 up, 7 in 2024-04-14T15:22:05.378 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:22:05.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:05.402 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:05.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:22:05.503 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:22:05.526 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:05.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:05.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:22:05.622 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:22:05 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:22:05.628 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:22:05.651 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:05.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:05.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:22:05.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.589+0000 7f3973647700 1 -- 172.21.15.90:0/1559542049 >> v1:172.21.15.90:6789/0 conn(0x7f396c10e270 legacy=0x7f396c110710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:05.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.589+0000 7f3973647700 1 -- 172.21.15.90:0/1559542049 shutdown_connections 2024-04-14T15:22:05.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.589+0000 7f3973647700 1 -- 172.21.15.90:0/1559542049 >> 172.21.15.90:0/1559542049 conn(0x7f396c076aa0 msgr2=0x7f396c076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:22:05.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.589+0000 7f3973647700 1 -- 172.21.15.90:0/1559542049 shutdown_connections 2024-04-14T15:22:05.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.589+0000 7f3973647700 1 -- 172.21.15.90:0/1559542049 wait complete. 2024-04-14T15:22:05.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.590+0000 7f3973647700 1 librados: starting msgr at 2024-04-14T15:22:05.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.590+0000 7f3973647700 1 librados: starting objecter 2024-04-14T15:22:05.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.590+0000 7f3973647700 1 Processor -- start 2024-04-14T15:22:05.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.590+0000 7f3973647700 1 -- start start 2024-04-14T15:22:05.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.590+0000 7f3973647700 1 librados: setting wanted keys 2024-04-14T15:22:05.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.590+0000 7f3973647700 1 librados: calling monclient init 2024-04-14T15:22:05.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.590+0000 7f3973647700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f396c1a9e20 con 0x7f396c10a6d0 2024-04-14T15:22:05.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.590+0000 7f3973647700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f396c1ab060 con 0x7f396c1023f0 2024-04-14T15:22:05.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.590+0000 7f3973647700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f396c1ac2a0 con 0x7f396c10e270 2024-04-14T15:22:05.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.590+0000 7f3970be2700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f396c10a6d0 0x7f396c1a4cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:43166/0 (socket says 172.21.15.90:43166) 2024-04-14T15:22:05.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.590+0000 7f3970be2700 1 -- 172.21.15.90:0/2361563628 learned_addr learned my addr 172.21.15.90:0/2361563628 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:22:05.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.590+0000 7f39627fc700 10 client.?.objecter ms_handle_connect 0x7f396c1023f0 2024-04-14T15:22:05.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.590+0000 7f39627fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:22:05.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.590+0000 7f39627fc700 10 client.?.objecter ms_handle_connect 0x7f396c10a6d0 2024-04-14T15:22:05.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.590+0000 7f39627fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:22:05.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.591+0000 7f39627fc700 1 -- 172.21.15.90:0/2361563628 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4022543276 0 0) 0x7f396c1ab060 con 0x7f396c1023f0 2024-04-14T15:22:05.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.591+0000 7f39627fc700 1 -- 172.21.15.90:0/2361563628 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f39580031f0 con 0x7f396c1023f0 2024-04-14T15:22:05.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.591+0000 7f39627fc700 1 -- 172.21.15.90:0/2361563628 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1525305522 0 0) 0x7f396c1a9e20 con 0x7f396c10a6d0 2024-04-14T15:22:05.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.591+0000 7f39627fc700 1 -- 172.21.15.90:0/2361563628 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f396c1ab060 con 0x7f396c10a6d0 2024-04-14T15:22:05.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.591+0000 7f39627fc700 1 -- 172.21.15.90:0/2361563628 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2639791095 0 0) 0x7f39580031f0 con 0x7f396c1023f0 2024-04-14T15:22:05.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.591+0000 7f39627fc700 1 -- 172.21.15.90:0/2361563628 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f396c1a9e20 con 0x7f396c1023f0 2024-04-14T15:22:05.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.591+0000 7f39627fc700 10 client.?.objecter ms_handle_connect 0x7f396c10e270 2024-04-14T15:22:05.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.591+0000 7f39627fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:22:05.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.591+0000 7f39627fc700 1 -- 172.21.15.90:0/2361563628 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f39680018b0 con 0x7f396c1023f0 2024-04-14T15:22:05.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.591+0000 7f39627fc700 1 -- 172.21.15.90:0/2361563628 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1822134811 0 0) 0x7f396c1ab060 con 0x7f396c10a6d0 2024-04-14T15:22:05.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.591+0000 7f39627fc700 1 -- 172.21.15.90:0/2361563628 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f39580031f0 con 0x7f396c10a6d0 2024-04-14T15:22:05.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.591+0000 7f39627fc700 1 -- 172.21.15.90:0/2361563628 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f395c001870 con 0x7f396c10a6d0 2024-04-14T15:22:05.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.591+0000 7f39627fc700 1 -- 172.21.15.90:0/2361563628 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 130212129 0 0) 0x7f396c1a9e20 con 0x7f396c1023f0 2024-04-14T15:22:05.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.591+0000 7f39627fc700 1 -- 172.21.15.90:0/2361563628 >> v1:172.21.15.90:6790/0 conn(0x7f396c10e270 legacy=0x7f396c1a84e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:05.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.591+0000 7f39627fc700 1 -- 172.21.15.90:0/2361563628 >> v1:172.21.15.90:6789/0 conn(0x7f396c10a6d0 legacy=0x7f396c1a4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:05.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.591+0000 7f39627fc700 1 -- 172.21.15.90:0/2361563628 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f396c1ad4e0 con 0x7f396c1023f0 2024-04-14T15:22:05.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.591+0000 7f3973647700 1 -- 172.21.15.90:0/2361563628 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f396c1aa050 con 0x7f396c1023f0 2024-04-14T15:22:05.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.591+0000 7f3973647700 10 client.49468.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:22:05.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.591+0000 7f3973647700 1 -- 172.21.15.90:0/2361563628 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f396c1aa550 con 0x7f396c1023f0 2024-04-14T15:22:05.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.591+0000 7f3973647700 1 librados: init done 2024-04-14T15:22:05.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.591+0000 7f39627fc700 1 -- 172.21.15.90:0/2361563628 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f3968003b40 con 0x7f396c1023f0 2024-04-14T15:22:05.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.591+0000 7f39627fc700 1 -- 172.21.15.90:0/2361563628 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f3968004a70 con 0x7f396c1023f0 2024-04-14T15:22:05.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.592+0000 7f3973647700 1 -- 172.21.15.90:0/2361563628 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f3950005100 con 0x7f396c1023f0 2024-04-14T15:22:05.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.593+0000 7f39627fc700 1 -- 172.21.15.90:0/2361563628 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f396801b640 con 0x7f396c1023f0 2024-04-14T15:22:05.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.593+0000 7f39627fc700 1 -- 172.21.15.90:0/2361563628 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(589..589 src has 467..589) ==== 7915+0+0 (unknown 3407461246 0 0) 0x7f3968089790 con 0x7f396c1023f0 2024-04-14T15:22:05.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.593+0000 7f39627fc700 10 client.49468.objecter ms_dispatch 0x7f396c001550 osd_map(589..589 src has 467..589) 2024-04-14T15:22:05.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.593+0000 7f39627fc700 3 client.49468.objecter handle_osd_map got epochs [589,589] > 0 2024-04-14T15:22:05.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.593+0000 7f39627fc700 3 client.49468.objecter handle_osd_map decoding full epoch 589 2024-04-14T15:22:05.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.593+0000 7f39627fc700 20 client.49468.objecter dump_active .. 0 homeless 2024-04-14T15:22:05.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.593+0000 7f39627fc700 1 -- 172.21.15.90:0/2361563628 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v589) ==== 54+0+16 (unknown 680089116 0 2420000192) 0x7f396808bbc0 con 0x7f396c1023f0 2024-04-14T15:22:05.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.593+0000 7f39627fc700 10 client.49468.objecter ms_handle_connect 0x7f3958071340 2024-04-14T15:22:05.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.594+0000 7f3973647700 10 client.49468.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:22:05.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.594+0000 7f3973647700 15 client.49468.objecter _session_command_op_assign -1 1 2024-04-14T15:22:05.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.594+0000 7f3973647700 20 client.49468.objecter _get_session s=0x7f39500053e0 osd=0 3 2024-04-14T15:22:05.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.594+0000 7f3973647700 20 client.49468.objecter put_session s=0x7f39500053e0 osd=0 3 2024-04-14T15:22:05.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.594+0000 7f3973647700 20 client.49468.objecter _get_session s=0x7f39500053e0 osd=0 3 2024-04-14T15:22:05.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.594+0000 7f3973647700 15 client.49468.objecter _session_command_op_remove -1 1 2024-04-14T15:22:05.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.594+0000 7f3973647700 20 client.49468.objecter get_session s=0x7f39500053e0 osd=0 3 2024-04-14T15:22:05.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.594+0000 7f3973647700 15 client.49468.objecter _session_command_op_assign 0 1 2024-04-14T15:22:05.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.594+0000 7f3973647700 20 client.49468.objecter put_session s=0x7f39500053e0 osd=0 4 2024-04-14T15:22:05.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.594+0000 7f3973647700 10 client.49468.objecter _send_command 1 2024-04-14T15:22:05.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.594+0000 7f3973647700 1 -- 172.21.15.90:0/2361563628 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3950004b70 con 0x7f3950004730 2024-04-14T15:22:05.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.594+0000 7f39627fc700 10 client.49468.objecter ms_handle_connect 0x7f3950004730 2024-04-14T15:22:05.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.595+0000 7f39627fc700 1 -- 172.21.15.90:0/2361563628 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f3950004b70 con 0x7f3950004730 2024-04-14T15:22:05.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.595+0000 7f39627fc700 10 client.49468.objecter ms_dispatch 0x7f396c001550 command_reply(tid 1: 0 ) 2024-04-14T15:22:05.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.595+0000 7f39627fc700 10 client.49468.objecter _finish_command 1 = system:0 2024-04-14T15:22:05.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.595+0000 7f39627fc700 20 client.49468.objecter put_session s=0x7f39500053e0 osd=0 4 2024-04-14T15:22:05.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.595+0000 7f39627fc700 15 client.49468.objecter _session_command_op_remove 0 1 2024-04-14T15:22:05.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.614+0000 7f3973647700 10 client.49468.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:22:05.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.614+0000 7f3973647700 15 client.49468.objecter _session_command_op_assign -1 2 2024-04-14T15:22:05.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.614+0000 7f3973647700 20 client.49468.objecter _get_session s=0x7f39500053e0 osd=0 3 2024-04-14T15:22:05.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.614+0000 7f3973647700 20 client.49468.objecter put_session s=0x7f39500053e0 osd=0 3 2024-04-14T15:22:05.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.614+0000 7f3973647700 20 client.49468.objecter _get_session s=0x7f39500053e0 osd=0 3 2024-04-14T15:22:05.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.614+0000 7f3973647700 15 client.49468.objecter _session_command_op_remove -1 2 2024-04-14T15:22:05.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.614+0000 7f3973647700 20 client.49468.objecter get_session s=0x7f39500053e0 osd=0 3 2024-04-14T15:22:05.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.614+0000 7f3973647700 15 client.49468.objecter _session_command_op_assign 0 2 2024-04-14T15:22:05.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.614+0000 7f3973647700 20 client.49468.objecter put_session s=0x7f39500053e0 osd=0 4 2024-04-14T15:22:05.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.614+0000 7f3973647700 10 client.49468.objecter _send_command 2 2024-04-14T15:22:05.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.614+0000 7f3973647700 1 -- 172.21.15.90:0/2361563628 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f39500079f0 con 0x7f3950004730 2024-04-14T15:22:05.686 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:22:05.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.616+0000 7f39627fc700 1 -- 172.21.15.90:0/2361563628 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f39500079f0 con 0x7f3950004730 2024-04-14T15:22:05.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.616+0000 7f39627fc700 10 client.49468.objecter ms_dispatch 0x7f396c001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:22:05.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.616+0000 7f39627fc700 10 client.49468.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:22:05.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.616+0000 7f39627fc700 20 client.49468.objecter put_session s=0x7f39500053e0 osd=0 4 2024-04-14T15:22:05.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.616+0000 7f39627fc700 15 client.49468.objecter _session_command_op_remove 0 2 2024-04-14T15:22:05.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.616+0000 7f3973647700 10 client.49468.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:22:05.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.616+0000 7f3973647700 15 client.49468.objecter _session_command_op_assign -1 3 2024-04-14T15:22:05.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.616+0000 7f3973647700 20 client.49468.objecter _get_session s=0x7f3950004130 osd=1 3 2024-04-14T15:22:05.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.616+0000 7f3973647700 20 client.49468.objecter put_session s=0x7f3950004130 osd=1 3 2024-04-14T15:22:05.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.616+0000 7f3973647700 20 client.49468.objecter _get_session s=0x7f3950004130 osd=1 3 2024-04-14T15:22:05.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.616+0000 7f3973647700 15 client.49468.objecter _session_command_op_remove -1 3 2024-04-14T15:22:05.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.616+0000 7f3973647700 20 client.49468.objecter get_session s=0x7f3950004130 osd=1 3 2024-04-14T15:22:05.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.616+0000 7f3973647700 15 client.49468.objecter _session_command_op_assign 1 3 2024-04-14T15:22:05.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.616+0000 7f3973647700 20 client.49468.objecter put_session s=0x7f3950004130 osd=1 4 2024-04-14T15:22:05.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.616+0000 7f3973647700 10 client.49468.objecter _send_command 3 2024-04-14T15:22:05.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.616+0000 7f3973647700 1 -- 172.21.15.90:0/2361563628 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f39500022a0 con 0x7f3950004310 2024-04-14T15:22:05.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.616+0000 7f39627fc700 10 client.49468.objecter ms_handle_connect 0x7f3950004310 2024-04-14T15:22:05.688 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:22:05.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.617+0000 7f39627fc700 1 -- 172.21.15.90:0/2361563628 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f39500022a0 con 0x7f3950004310 2024-04-14T15:22:05.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.617+0000 7f39627fc700 10 client.49468.objecter ms_dispatch 0x7f396c001550 command_reply(tid 3: 0 ) 2024-04-14T15:22:05.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.617+0000 7f39627fc700 10 client.49468.objecter _finish_command 3 = system:0 2024-04-14T15:22:05.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.617+0000 7f39627fc700 20 client.49468.objecter put_session s=0x7f3950004130 osd=1 4 2024-04-14T15:22:05.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.617+0000 7f39627fc700 15 client.49468.objecter _session_command_op_remove 1 3 2024-04-14T15:22:05.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.634+0000 7f3973647700 10 client.49468.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:22:05.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.634+0000 7f3973647700 15 client.49468.objecter _session_command_op_assign -1 4 2024-04-14T15:22:05.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.634+0000 7f3973647700 20 client.49468.objecter _get_session s=0x7f3950004130 osd=1 3 2024-04-14T15:22:05.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.634+0000 7f3973647700 20 client.49468.objecter put_session s=0x7f3950004130 osd=1 3 2024-04-14T15:22:05.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.634+0000 7f3973647700 20 client.49468.objecter _get_session s=0x7f3950004130 osd=1 3 2024-04-14T15:22:05.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.634+0000 7f3973647700 15 client.49468.objecter _session_command_op_remove -1 4 2024-04-14T15:22:05.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.634+0000 7f3973647700 20 client.49468.objecter get_session s=0x7f3950004130 osd=1 3 2024-04-14T15:22:05.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.634+0000 7f3973647700 15 client.49468.objecter _session_command_op_assign 1 4 2024-04-14T15:22:05.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.634+0000 7f3973647700 20 client.49468.objecter put_session s=0x7f3950004130 osd=1 4 2024-04-14T15:22:05.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.634+0000 7f3973647700 10 client.49468.objecter _send_command 4 2024-04-14T15:22:05.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.634+0000 7f3973647700 1 -- 172.21.15.90:0/2361563628 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f3950001720 con 0x7f3950004310 2024-04-14T15:22:05.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.635+0000 7f39627fc700 1 -- 172.21.15.90:0/2361563628 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f3950001720 con 0x7f3950004310 2024-04-14T15:22:05.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.635+0000 7f39627fc700 10 client.49468.objecter ms_dispatch 0x7f396c001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:22:05.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.635+0000 7f39627fc700 10 client.49468.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:22:05.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.635+0000 7f39627fc700 20 client.49468.objecter put_session s=0x7f3950004130 osd=1 4 2024-04-14T15:22:05.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.635+0000 7f39627fc700 15 client.49468.objecter _session_command_op_remove 1 4 2024-04-14T15:22:05.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.635+0000 7f3973647700 10 client.49468.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:22:05.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.635+0000 7f3973647700 15 client.49468.objecter _session_command_op_assign -1 5 2024-04-14T15:22:05.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.635+0000 7f3973647700 20 client.49468.objecter _get_session s=0x7f3950001980 osd=2 3 2024-04-14T15:22:05.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.635+0000 7f3973647700 20 client.49468.objecter put_session s=0x7f3950001980 osd=2 3 2024-04-14T15:22:05.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.635+0000 7f3973647700 20 client.49468.objecter _get_session s=0x7f3950001980 osd=2 3 2024-04-14T15:22:05.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.635+0000 7f3973647700 15 client.49468.objecter _session_command_op_remove -1 5 2024-04-14T15:22:05.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.635+0000 7f3973647700 20 client.49468.objecter get_session s=0x7f3950001980 osd=2 3 2024-04-14T15:22:05.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.635+0000 7f3973647700 15 client.49468.objecter _session_command_op_assign 2 5 2024-04-14T15:22:05.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.635+0000 7f3973647700 20 client.49468.objecter put_session s=0x7f3950001980 osd=2 4 2024-04-14T15:22:05.691 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:22:05.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.635+0000 7f3973647700 10 client.49468.objecter _send_command 5 2024-04-14T15:22:05.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.635+0000 7f3973647700 1 -- 172.21.15.90:0/2361563628 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f39500110e0 con 0x7f395000d5d0 2024-04-14T15:22:05.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.636+0000 7f39627fc700 10 client.49468.objecter ms_handle_connect 0x7f395000d5d0 2024-04-14T15:22:05.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.636+0000 7f39627fc700 1 -- 172.21.15.90:0/2361563628 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f39500110e0 con 0x7f395000d5d0 2024-04-14T15:22:05.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.636+0000 7f39627fc700 10 client.49468.objecter ms_dispatch 0x7f396c001550 command_reply(tid 5: 0 ) 2024-04-14T15:22:05.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.636+0000 7f39627fc700 10 client.49468.objecter _finish_command 5 = system:0 2024-04-14T15:22:05.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.636+0000 7f39627fc700 20 client.49468.objecter put_session s=0x7f3950001980 osd=2 4 2024-04-14T15:22:05.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.636+0000 7f39627fc700 15 client.49468.objecter _session_command_op_remove 2 5 2024-04-14T15:22:05.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.652+0000 7f3973647700 10 client.49468.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:22:05.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.653+0000 7f3973647700 15 client.49468.objecter _session_command_op_assign -1 6 2024-04-14T15:22:05.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.653+0000 7f3973647700 20 client.49468.objecter _get_session s=0x7f3950001980 osd=2 3 2024-04-14T15:22:05.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.653+0000 7f3973647700 20 client.49468.objecter put_session s=0x7f3950001980 osd=2 3 2024-04-14T15:22:05.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.653+0000 7f3973647700 20 client.49468.objecter _get_session s=0x7f3950001980 osd=2 3 2024-04-14T15:22:05.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.653+0000 7f3973647700 15 client.49468.objecter _session_command_op_remove -1 6 2024-04-14T15:22:05.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.653+0000 7f3973647700 20 client.49468.objecter get_session s=0x7f3950001980 osd=2 3 2024-04-14T15:22:05.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.653+0000 7f3973647700 15 client.49468.objecter _session_command_op_assign 2 6 2024-04-14T15:22:05.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.653+0000 7f3973647700 20 client.49468.objecter put_session s=0x7f3950001980 osd=2 4 2024-04-14T15:22:05.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.653+0000 7f3973647700 10 client.49468.objecter _send_command 6 2024-04-14T15:22:05.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.653+0000 7f3973647700 1 -- 172.21.15.90:0/2361563628 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f3950008100 con 0x7f395000d5d0 2024-04-14T15:22:05.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.654+0000 7f39627fc700 1 -- 172.21.15.90:0/2361563628 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f3950008100 con 0x7f395000d5d0 2024-04-14T15:22:05.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.654+0000 7f39627fc700 10 client.49468.objecter ms_dispatch 0x7f396c001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:22:05.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.654+0000 7f39627fc700 10 client.49468.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:22:05.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.654+0000 7f39627fc700 20 client.49468.objecter put_session s=0x7f3950001980 osd=2 4 2024-04-14T15:22:05.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.654+0000 7f39627fc700 15 client.49468.objecter _session_command_op_remove 2 6 2024-04-14T15:22:05.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.654+0000 7f3973647700 10 client.49468.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:22:05.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.654+0000 7f3973647700 15 client.49468.objecter _session_command_op_assign -1 7 2024-04-14T15:22:05.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.654+0000 7f3973647700 20 client.49468.objecter _get_session s=0x7f3950008390 osd=3 3 2024-04-14T15:22:05.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.654+0000 7f3973647700 20 client.49468.objecter put_session s=0x7f3950008390 osd=3 3 2024-04-14T15:22:05.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.654+0000 7f3973647700 20 client.49468.objecter _get_session s=0x7f3950008390 osd=3 3 2024-04-14T15:22:05.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.654+0000 7f3973647700 15 client.49468.objecter _session_command_op_remove -1 7 2024-04-14T15:22:05.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.654+0000 7f3973647700 20 client.49468.objecter get_session s=0x7f3950008390 osd=3 3 2024-04-14T15:22:05.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.654+0000 7f3973647700 15 client.49468.objecter _session_command_op_assign 3 7 2024-04-14T15:22:05.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.654+0000 7f3973647700 20 client.49468.objecter put_session s=0x7f3950008390 osd=3 4 2024-04-14T15:22:05.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.654+0000 7f3973647700 10 client.49468.objecter _send_command 7 2024-04-14T15:22:05.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.654+0000 7f3973647700 1 -- 172.21.15.90:0/2361563628 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f3950012fd0 con 0x7f3950012410 2024-04-14T15:22:05.694 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:22:05.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.655+0000 7f39627fc700 10 client.49468.objecter ms_handle_connect 0x7f3950012410 2024-04-14T15:22:05.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.655+0000 7f39627fc700 1 -- 172.21.15.90:0/2361563628 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f3950012fd0 con 0x7f3950012410 2024-04-14T15:22:05.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.655+0000 7f39627fc700 10 client.49468.objecter ms_dispatch 0x7f396c001550 command_reply(tid 7: 0 ) 2024-04-14T15:22:05.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.655+0000 7f39627fc700 10 client.49468.objecter _finish_command 7 = system:0 2024-04-14T15:22:05.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.655+0000 7f39627fc700 20 client.49468.objecter put_session s=0x7f3950008390 osd=3 4 2024-04-14T15:22:05.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.655+0000 7f39627fc700 15 client.49468.objecter _session_command_op_remove 3 7 2024-04-14T15:22:05.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.672+0000 7f3973647700 10 client.49468.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:22:05.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.672+0000 7f3973647700 15 client.49468.objecter _session_command_op_assign -1 8 2024-04-14T15:22:05.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.672+0000 7f3973647700 20 client.49468.objecter _get_session s=0x7f3950008390 osd=3 3 2024-04-14T15:22:05.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.672+0000 7f3973647700 20 client.49468.objecter put_session s=0x7f3950008390 osd=3 3 2024-04-14T15:22:05.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.672+0000 7f3973647700 20 client.49468.objecter _get_session s=0x7f3950008390 osd=3 3 2024-04-14T15:22:05.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.672+0000 7f3973647700 15 client.49468.objecter _session_command_op_remove -1 8 2024-04-14T15:22:05.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.672+0000 7f3973647700 20 client.49468.objecter get_session s=0x7f3950008390 osd=3 3 2024-04-14T15:22:05.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.672+0000 7f3973647700 15 client.49468.objecter _session_command_op_assign 3 8 2024-04-14T15:22:05.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.672+0000 7f3973647700 20 client.49468.objecter put_session s=0x7f3950008390 osd=3 4 2024-04-14T15:22:05.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.672+0000 7f3973647700 10 client.49468.objecter _send_command 8 2024-04-14T15:22:05.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.672+0000 7f3973647700 1 -- 172.21.15.90:0/2361563628 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f3950013980 con 0x7f3950012410 2024-04-14T15:22:05.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.674+0000 7f39627fc700 1 -- 172.21.15.90:0/2361563628 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f3950013980 con 0x7f3950012410 2024-04-14T15:22:05.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.674+0000 7f39627fc700 10 client.49468.objecter ms_dispatch 0x7f396c001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:22:05.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.674+0000 7f39627fc700 10 client.49468.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:22:05.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.674+0000 7f39627fc700 20 client.49468.objecter put_session s=0x7f3950008390 osd=3 4 2024-04-14T15:22:05.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.674+0000 7f39627fc700 15 client.49468.objecter _session_command_op_remove 3 8 2024-04-14T15:22:05.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.674+0000 7f3973647700 10 client.49468.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:22:05.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.674+0000 7f3973647700 15 client.49468.objecter _session_command_op_assign -1 9 2024-04-14T15:22:05.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.674+0000 7f3973647700 20 client.49468.objecter _get_session s=0x7f3950013950 osd=4 3 2024-04-14T15:22:05.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.674+0000 7f3973647700 20 client.49468.objecter put_session s=0x7f3950013950 osd=4 3 2024-04-14T15:22:05.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.674+0000 7f3973647700 20 client.49468.objecter _get_session s=0x7f3950013950 osd=4 3 2024-04-14T15:22:05.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.674+0000 7f3973647700 15 client.49468.objecter _session_command_op_remove -1 9 2024-04-14T15:22:05.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.674+0000 7f3973647700 20 client.49468.objecter get_session s=0x7f3950013950 osd=4 3 2024-04-14T15:22:05.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.674+0000 7f3973647700 15 client.49468.objecter _session_command_op_assign 4 9 2024-04-14T15:22:05.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.674+0000 7f3973647700 20 client.49468.objecter put_session s=0x7f3950013950 osd=4 4 2024-04-14T15:22:05.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.674+0000 7f3973647700 10 client.49468.objecter _send_command 9 2024-04-14T15:22:05.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.674+0000 7f3973647700 1 -- 172.21.15.90:0/2361563628 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f395001a7f0 con 0x7f3950016c40 2024-04-14T15:22:05.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.674+0000 7f39627fc700 10 client.49468.objecter ms_handle_connect 0x7f3950016c40 2024-04-14T15:22:05.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.675+0000 7f39627fc700 1 -- 172.21.15.90:0/2361563628 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f395001a7f0 con 0x7f3950016c40 2024-04-14T15:22:05.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.675+0000 7f39627fc700 10 client.49468.objecter ms_dispatch 0x7f396c001550 command_reply(tid 9: 0 ) 2024-04-14T15:22:05.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.675+0000 7f39627fc700 10 client.49468.objecter _finish_command 9 = system:0 2024-04-14T15:22:05.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.675+0000 7f39627fc700 20 client.49468.objecter put_session s=0x7f3950013950 osd=4 4 2024-04-14T15:22:05.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.675+0000 7f39627fc700 15 client.49468.objecter _session_command_op_remove 4 9 2024-04-14T15:22:05.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.691+0000 7f3973647700 10 client.49468.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:22:05.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.691+0000 7f3973647700 15 client.49468.objecter _session_command_op_assign -1 10 2024-04-14T15:22:05.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.691+0000 7f3973647700 20 client.49468.objecter _get_session s=0x7f3950013950 osd=4 3 2024-04-14T15:22:05.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.691+0000 7f3973647700 20 client.49468.objecter put_session s=0x7f3950013950 osd=4 3 2024-04-14T15:22:05.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.691+0000 7f3973647700 20 client.49468.objecter _get_session s=0x7f3950013950 osd=4 3 2024-04-14T15:22:05.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.691+0000 7f3973647700 15 client.49468.objecter _session_command_op_remove -1 10 2024-04-14T15:22:05.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.691+0000 7f3973647700 20 client.49468.objecter get_session s=0x7f3950013950 osd=4 3 2024-04-14T15:22:05.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.691+0000 7f3973647700 15 client.49468.objecter _session_command_op_assign 4 10 2024-04-14T15:22:05.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.691+0000 7f3973647700 20 client.49468.objecter put_session s=0x7f3950013950 osd=4 4 2024-04-14T15:22:05.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.691+0000 7f3973647700 10 client.49468.objecter _send_command 10 2024-04-14T15:22:05.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.691+0000 7f3973647700 1 -- 172.21.15.90:0/2361563628 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f39500133d0 con 0x7f3950016c40 2024-04-14T15:22:05.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.693+0000 7f39627fc700 1 -- 172.21.15.90:0/2361563628 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f39500133d0 con 0x7f3950016c40 2024-04-14T15:22:05.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.693+0000 7f39627fc700 10 client.49468.objecter ms_dispatch 0x7f396c001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:22:05.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.693+0000 7f39627fc700 10 client.49468.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:22:05.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.693+0000 7f39627fc700 20 client.49468.objecter put_session s=0x7f3950013950 osd=4 4 2024-04-14T15:22:05.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.693+0000 7f39627fc700 15 client.49468.objecter _session_command_op_remove 4 10 2024-04-14T15:22:05.699 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:22:05.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.693+0000 7f3973647700 10 client.49468.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:22:05.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.693+0000 7f3973647700 15 client.49468.objecter _session_command_op_assign -1 11 2024-04-14T15:22:05.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.693+0000 7f3973647700 20 client.49468.objecter _get_session s=0x7f395000c760 osd=5 3 2024-04-14T15:22:05.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.693+0000 7f3973647700 20 client.49468.objecter put_session s=0x7f395000c760 osd=5 3 2024-04-14T15:22:05.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.693+0000 7f3973647700 20 client.49468.objecter _get_session s=0x7f395000c760 osd=5 3 2024-04-14T15:22:05.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.693+0000 7f3973647700 15 client.49468.objecter _session_command_op_remove -1 11 2024-04-14T15:22:05.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.693+0000 7f3973647700 20 client.49468.objecter get_session s=0x7f395000c760 osd=5 3 2024-04-14T15:22:05.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.693+0000 7f3973647700 15 client.49468.objecter _session_command_op_assign 5 11 2024-04-14T15:22:05.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.693+0000 7f3973647700 20 client.49468.objecter put_session s=0x7f395000c760 osd=5 4 2024-04-14T15:22:05.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.693+0000 7f3973647700 10 client.49468.objecter _send_command 11 2024-04-14T15:22:05.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.693+0000 7f3973647700 1 -- 172.21.15.90:0/2361563628 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f395001bfb0 con 0x7f3950013180 2024-04-14T15:22:05.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.694+0000 7f39627fc700 10 client.49468.objecter ms_handle_connect 0x7f3950013180 2024-04-14T15:22:05.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.694+0000 7f39627fc700 1 -- 172.21.15.90:0/2361563628 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f395c009470 con 0x7f3950013180 2024-04-14T15:22:05.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.694+0000 7f39627fc700 10 client.49468.objecter ms_dispatch 0x7f396c001550 command_reply(tid 11: 0 ) 2024-04-14T15:22:05.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.694+0000 7f39627fc700 10 client.49468.objecter _finish_command 11 = system:0 2024-04-14T15:22:05.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.694+0000 7f39627fc700 20 client.49468.objecter put_session s=0x7f395000c760 osd=5 4 2024-04-14T15:22:05.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.694+0000 7f39627fc700 15 client.49468.objecter _session_command_op_remove 5 11 2024-04-14T15:22:05.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.711+0000 7f3973647700 10 client.49468.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:22:05.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.711+0000 7f3973647700 15 client.49468.objecter _session_command_op_assign -1 12 2024-04-14T15:22:05.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.711+0000 7f3973647700 20 client.49468.objecter _get_session s=0x7f395000c760 osd=5 3 2024-04-14T15:22:05.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.711+0000 7f3973647700 20 client.49468.objecter put_session s=0x7f395000c760 osd=5 3 2024-04-14T15:22:05.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.711+0000 7f3973647700 20 client.49468.objecter _get_session s=0x7f395000c760 osd=5 3 2024-04-14T15:22:05.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.711+0000 7f3973647700 15 client.49468.objecter _session_command_op_remove -1 12 2024-04-14T15:22:05.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.711+0000 7f3973647700 20 client.49468.objecter get_session s=0x7f395000c760 osd=5 3 2024-04-14T15:22:05.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.711+0000 7f3973647700 15 client.49468.objecter _session_command_op_assign 5 12 2024-04-14T15:22:05.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.711+0000 7f3973647700 20 client.49468.objecter put_session s=0x7f395000c760 osd=5 4 2024-04-14T15:22:05.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.711+0000 7f3973647700 10 client.49468.objecter _send_command 12 2024-04-14T15:22:05.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.711+0000 7f3973647700 1 -- 172.21.15.90:0/2361563628 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f3950012040 con 0x7f3950013180 2024-04-14T15:22:05.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.713+0000 7f39627fc700 1 -- 172.21.15.90:0/2361563628 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f3950012040 con 0x7f3950013180 2024-04-14T15:22:05.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.713+0000 7f39627fc700 10 client.49468.objecter ms_dispatch 0x7f396c001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:22:05.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.713+0000 7f39627fc700 10 client.49468.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:22:05.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.713+0000 7f39627fc700 20 client.49468.objecter put_session s=0x7f395000c760 osd=5 4 2024-04-14T15:22:05.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.713+0000 7f39627fc700 15 client.49468.objecter _session_command_op_remove 5 12 2024-04-14T15:22:05.714 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:22:05.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.713+0000 7f3973647700 10 client.49468.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:22:05.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.713+0000 7f3973647700 15 client.49468.objecter _session_command_op_assign -1 13 2024-04-14T15:22:05.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.713+0000 7f3973647700 20 client.49468.objecter _get_session s=0x7f3950012040 osd=6 3 2024-04-14T15:22:05.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.713+0000 7f3973647700 20 client.49468.objecter put_session s=0x7f3950012040 osd=6 3 2024-04-14T15:22:05.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.713+0000 7f3973647700 20 client.49468.objecter _get_session s=0x7f3950012040 osd=6 3 2024-04-14T15:22:05.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.713+0000 7f3973647700 15 client.49468.objecter _session_command_op_remove -1 13 2024-04-14T15:22:05.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.713+0000 7f3973647700 20 client.49468.objecter get_session s=0x7f3950012040 osd=6 3 2024-04-14T15:22:05.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.713+0000 7f3973647700 15 client.49468.objecter _session_command_op_assign 6 13 2024-04-14T15:22:05.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.713+0000 7f3973647700 20 client.49468.objecter put_session s=0x7f3950012040 osd=6 4 2024-04-14T15:22:05.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.713+0000 7f3973647700 10 client.49468.objecter _send_command 13 2024-04-14T15:22:05.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.713+0000 7f3973647700 1 -- 172.21.15.90:0/2361563628 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f39500233c0 con 0x7f395001f8b0 2024-04-14T15:22:05.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.713+0000 7f39627fc700 10 client.49468.objecter ms_handle_connect 0x7f395001f8b0 2024-04-14T15:22:05.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.714+0000 7f39627fc700 1 -- 172.21.15.90:0/2361563628 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f39500233c0 con 0x7f395001f8b0 2024-04-14T15:22:05.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.714+0000 7f39627fc700 10 client.49468.objecter ms_dispatch 0x7f396c001550 command_reply(tid 13: 0 ) 2024-04-14T15:22:05.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.714+0000 7f39627fc700 10 client.49468.objecter _finish_command 13 = system:0 2024-04-14T15:22:05.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.714+0000 7f39627fc700 20 client.49468.objecter put_session s=0x7f3950012040 osd=6 4 2024-04-14T15:22:05.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.714+0000 7f39627fc700 15 client.49468.objecter _session_command_op_remove 6 13 2024-04-14T15:22:05.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.730+0000 7f3973647700 10 client.49468.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:22:05.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.730+0000 7f3973647700 15 client.49468.objecter _session_command_op_assign -1 14 2024-04-14T15:22:05.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.730+0000 7f3973647700 20 client.49468.objecter _get_session s=0x7f3950012040 osd=6 3 2024-04-14T15:22:05.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.730+0000 7f3973647700 20 client.49468.objecter put_session s=0x7f3950012040 osd=6 3 2024-04-14T15:22:05.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.730+0000 7f3973647700 20 client.49468.objecter _get_session s=0x7f3950012040 osd=6 3 2024-04-14T15:22:05.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.730+0000 7f3973647700 15 client.49468.objecter _session_command_op_remove -1 14 2024-04-14T15:22:05.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.730+0000 7f3973647700 20 client.49468.objecter get_session s=0x7f3950012040 osd=6 3 2024-04-14T15:22:05.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.730+0000 7f3973647700 15 client.49468.objecter _session_command_op_assign 6 14 2024-04-14T15:22:05.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.730+0000 7f3973647700 20 client.49468.objecter put_session s=0x7f3950012040 osd=6 4 2024-04-14T15:22:05.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.730+0000 7f3973647700 10 client.49468.objecter _send_command 14 2024-04-14T15:22:05.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.730+0000 7f3973647700 1 -- 172.21.15.90:0/2361563628 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f3950024d70 con 0x7f395001f8b0 2024-04-14T15:22:05.733 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:22:05.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.732+0000 7f39627fc700 1 -- 172.21.15.90:0/2361563628 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f3950024d70 con 0x7f395001f8b0 2024-04-14T15:22:05.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.732+0000 7f39627fc700 10 client.49468.objecter ms_dispatch 0x7f396c001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:22:05.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.732+0000 7f39627fc700 10 client.49468.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:22:05.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.732+0000 7f39627fc700 20 client.49468.objecter put_session s=0x7f3950012040 osd=6 4 2024-04-14T15:22:05.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.732+0000 7f39627fc700 15 client.49468.objecter _session_command_op_remove 6 14 2024-04-14T15:22:05.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.733+0000 7f3973647700 10 client.49468.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:22:05.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.733+0000 7f3973647700 15 client.49468.objecter _session_command_op_assign -1 15 2024-04-14T15:22:05.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.733+0000 7f3973647700 20 client.49468.objecter _get_session s=0x7f3950024d70 osd=7 3 2024-04-14T15:22:05.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.733+0000 7f3973647700 20 client.49468.objecter put_session s=0x7f3950024d70 osd=7 3 2024-04-14T15:22:05.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.733+0000 7f3973647700 20 client.49468.objecter _get_session s=0x7f3950024d70 osd=7 3 2024-04-14T15:22:05.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.733+0000 7f3973647700 15 client.49468.objecter _session_command_op_remove -1 15 2024-04-14T15:22:05.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.733+0000 7f3973647700 20 client.49468.objecter get_session s=0x7f3950024d70 osd=7 3 2024-04-14T15:22:05.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.733+0000 7f3973647700 15 client.49468.objecter _session_command_op_assign 7 15 2024-04-14T15:22:05.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.733+0000 7f3973647700 20 client.49468.objecter put_session s=0x7f3950024d70 osd=7 4 2024-04-14T15:22:05.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.733+0000 7f3973647700 10 client.49468.objecter _send_command 15 2024-04-14T15:22:05.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.733+0000 7f3973647700 1 -- 172.21.15.90:0/2361563628 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f39500258f0 con 0x7f3950025490 2024-04-14T15:22:05.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.733+0000 7f39627fc700 10 client.49468.objecter ms_handle_connect 0x7f3950025490 2024-04-14T15:22:05.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.734+0000 7f39627fc700 1 -- 172.21.15.90:0/2361563628 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f39500258f0 con 0x7f3950025490 2024-04-14T15:22:05.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.734+0000 7f39627fc700 10 client.49468.objecter ms_dispatch 0x7f396c001550 command_reply(tid 15: 0 ) 2024-04-14T15:22:05.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.734+0000 7f39627fc700 10 client.49468.objecter _finish_command 15 = system:0 2024-04-14T15:22:05.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.734+0000 7f39627fc700 20 client.49468.objecter put_session s=0x7f3950024d70 osd=7 4 2024-04-14T15:22:05.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.734+0000 7f39627fc700 15 client.49468.objecter _session_command_op_remove 7 15 2024-04-14T15:22:05.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.750+0000 7f3973647700 10 client.49468.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:22:05.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.750+0000 7f3973647700 15 client.49468.objecter _session_command_op_assign -1 16 2024-04-14T15:22:05.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.750+0000 7f3973647700 20 client.49468.objecter _get_session s=0x7f3950024d70 osd=7 3 2024-04-14T15:22:05.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.750+0000 7f3973647700 20 client.49468.objecter put_session s=0x7f3950024d70 osd=7 3 2024-04-14T15:22:05.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.750+0000 7f3973647700 20 client.49468.objecter _get_session s=0x7f3950024d70 osd=7 3 2024-04-14T15:22:05.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.750+0000 7f3973647700 15 client.49468.objecter _session_command_op_remove -1 16 2024-04-14T15:22:05.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.750+0000 7f3973647700 20 client.49468.objecter get_session s=0x7f3950024d70 osd=7 3 2024-04-14T15:22:05.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.750+0000 7f3973647700 15 client.49468.objecter _session_command_op_assign 7 16 2024-04-14T15:22:05.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.750+0000 7f3973647700 20 client.49468.objecter put_session s=0x7f3950024d70 osd=7 4 2024-04-14T15:22:05.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.750+0000 7f3973647700 10 client.49468.objecter _send_command 16 2024-04-14T15:22:05.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.750+0000 7f3973647700 1 -- 172.21.15.90:0/2361563628 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f39500258b0 con 0x7f3950025490 2024-04-14T15:22:05.752 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:22:05.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.752+0000 7f39627fc700 1 -- 172.21.15.90:0/2361563628 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f39500258b0 con 0x7f3950025490 2024-04-14T15:22:05.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.752+0000 7f39627fc700 10 client.49468.objecter ms_dispatch 0x7f396c001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:22:05.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.752+0000 7f39627fc700 10 client.49468.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:22:05.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.752+0000 7f39627fc700 20 client.49468.objecter put_session s=0x7f3950024d70 osd=7 4 2024-04-14T15:22:05.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.752+0000 7f39627fc700 15 client.49468.objecter _session_command_op_remove 7 16 2024-04-14T15:22:05.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.752+0000 7f3973647700 10 librados: watch_flush enter 2024-04-14T15:22:05.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.752+0000 7f3973647700 10 librados: watch_flush exit 2024-04-14T15:22:05.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.752+0000 7f3973647700 10 client.49468.objecter close_session for osd.0 2024-04-14T15:22:05.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.752+0000 7f3973647700 1 -- 172.21.15.90:0/2361563628 >> v1:172.21.15.90:6801/3761806885 conn(0x7f3950004730 legacy=0x7f3950002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:05.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.752+0000 7f3973647700 20 client.49468.objecter put_session s=0x7f39500053e0 osd=0 1 2024-04-14T15:22:05.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.752+0000 7f3973647700 10 client.49468.objecter close_session for osd.1 2024-04-14T15:22:05.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.752+0000 7f3973647700 1 -- 172.21.15.90:0/2361563628 >> v1:172.21.15.90:6805/4170361699 conn(0x7f3950004310 legacy=0x7f395000ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:05.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.752+0000 7f3973647700 20 client.49468.objecter put_session s=0x7f3950004130 osd=1 1 2024-04-14T15:22:05.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.752+0000 7f3973647700 10 client.49468.objecter close_session for osd.2 2024-04-14T15:22:05.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.752+0000 7f3973647700 1 -- 172.21.15.90:0/2361563628 >> v1:172.21.15.90:6809/1686184251 conn(0x7f395000d5d0 legacy=0x7f395000f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:05.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.752+0000 7f3973647700 20 client.49468.objecter put_session s=0x7f3950001980 osd=2 1 2024-04-14T15:22:05.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.752+0000 7f3973647700 10 client.49468.objecter close_session for osd.3 2024-04-14T15:22:05.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.752+0000 7f3973647700 1 -- 172.21.15.90:0/2361563628 >> v1:172.21.15.90:6813/2270166774 conn(0x7f3950012410 legacy=0x7f39500128c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:05.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.752+0000 7f3973647700 20 client.49468.objecter put_session s=0x7f3950008390 osd=3 1 2024-04-14T15:22:05.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.752+0000 7f3973647700 10 client.49468.objecter close_session for osd.4 2024-04-14T15:22:05.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.752+0000 7f3973647700 1 -- 172.21.15.90:0/2361563628 >> v1:172.21.15.92:6800/3887549733 conn(0x7f3950016c40 legacy=0x7f3950019030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:05.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.752+0000 7f3973647700 20 client.49468.objecter put_session s=0x7f3950013950 osd=4 1 2024-04-14T15:22:05.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.752+0000 7f3973647700 10 client.49468.objecter close_session for osd.5 2024-04-14T15:22:05.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.752+0000 7f3973647700 1 -- 172.21.15.90:0/2361563628 >> v1:172.21.15.92:6804/3244561440 conn(0x7f3950013180 legacy=0x7f395001a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:05.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.752+0000 7f3973647700 20 client.49468.objecter put_session s=0x7f395000c760 osd=5 1 2024-04-14T15:22:05.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.752+0000 7f3973647700 10 client.49468.objecter close_session for osd.6 2024-04-14T15:22:05.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.752+0000 7f3973647700 1 -- 172.21.15.90:0/2361563628 >> v1:172.21.15.92:6808/636125963 conn(0x7f395001f8b0 legacy=0x7f3950021ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:05.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.752+0000 7f3973647700 20 client.49468.objecter put_session s=0x7f3950012040 osd=6 1 2024-04-14T15:22:05.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.752+0000 7f3973647700 10 client.49468.objecter close_session for osd.7 2024-04-14T15:22:05.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.752+0000 7f3971be4700 1 -- 172.21.15.90:0/2361563628 reap_dead start 2024-04-14T15:22:05.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.752+0000 7f3973647700 1 -- 172.21.15.90:0/2361563628 >> v1:172.21.15.92:6812/1773706716 conn(0x7f3950025490 legacy=0x7f39500235a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:05.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.752+0000 7f3971be4700 1 -- 172.21.15.90:0/2361563628 reap_dead start 2024-04-14T15:22:05.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.752+0000 7f3973647700 20 client.49468.objecter put_session s=0x7f3950024d70 osd=7 1 2024-04-14T15:22:05.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.752+0000 7f3973647700 20 client.49468.objecter shutdown clearing up homeless session... 2024-04-14T15:22:05.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.752+0000 7f3973647700 10 client.49468.objecter successfully canceled tick 2024-04-14T15:22:05.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.752+0000 7f3971be4700 1 -- 172.21.15.90:0/2361563628 reap_dead start 2024-04-14T15:22:05.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.752+0000 7f3971be4700 1 -- 172.21.15.90:0/2361563628 reap_dead start 2024-04-14T15:22:05.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.752+0000 7f3973647700 1 -- 172.21.15.90:0/2361563628 >> v1:172.21.15.90:6800/420935427 conn(0x7f3958071340 legacy=0x7f3958073800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:05.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.752+0000 7f3973647700 1 -- 172.21.15.90:0/2361563628 >> v1:172.21.15.92:6789/0 conn(0x7f396c1023f0 legacy=0x7f396c1172a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:05.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.753+0000 7f3973647700 1 -- 172.21.15.90:0/2361563628 shutdown_connections 2024-04-14T15:22:05.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.753+0000 7f3973647700 1 -- 172.21.15.90:0/2361563628 >> 172.21.15.90:0/2361563628 conn(0x7f396c076aa0 msgr2=0x7f396c10e6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:22:05.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.753+0000 7f3973647700 1 -- 172.21.15.90:0/2361563628 shutdown_connections 2024-04-14T15:22:05.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.753+0000 7f3973647700 1 -- 172.21.15.90:0/2361563628 wait complete. 2024-04-14T15:22:05.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:05.753+0000 7f3973647700 1 librados: shutdown 2024-04-14T15:22:05.756 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:22:05.757 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:22:05.757 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:22:05.757 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:22:05.757 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:22:05.757 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:22:05.757 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:22:05.757 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:22:05.757 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:22:05.780 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:05.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:05.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:22:05.882 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:05.907 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:05.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:05.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:06.009 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:22:06.032 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:06.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:06.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:22:06.133 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:22:06.157 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:06.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:06.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:22:06.258 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:22:06.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:06.284 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:06.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:22:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:06 smithi092 ceph-mon[33966]: 1.9 scrub starts 2024-04-14T15:22:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:06 smithi092 ceph-mon[33966]: 1.9 scrub ok 2024-04-14T15:22:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:06 smithi092 ceph-mon[33966]: 110.5 scrub starts 2024-04-14T15:22:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:06 smithi092 ceph-mon[33966]: 110.5 scrub ok 2024-04-14T15:22:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:06 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2361563628' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:22:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:06 smithi092 ceph-mon[33966]: osdmap e590: 8 total, 8 up, 7 in 2024-04-14T15:22:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:06 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:22:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:06 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:22:06.385 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:06.408 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:06.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:06.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:06.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:06 smithi090 ceph-mon[41670]: 1.9 scrub starts 2024-04-14T15:22:06.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:06 smithi090 ceph-mon[41670]: 1.9 scrub ok 2024-04-14T15:22:06.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:06 smithi090 ceph-mon[41670]: 110.5 scrub starts 2024-04-14T15:22:06.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:06 smithi090 ceph-mon[41670]: 110.5 scrub ok 2024-04-14T15:22:06.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:06 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2361563628' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:22:06.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:06 smithi090 ceph-mon[41670]: osdmap e590: 8 total, 8 up, 7 in 2024-04-14T15:22:06.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:06 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:22:06.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:06 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:22:06.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:06 smithi090 ceph-mon[30399]: 1.9 scrub starts 2024-04-14T15:22:06.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:06 smithi090 ceph-mon[30399]: 1.9 scrub ok 2024-04-14T15:22:06.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:06 smithi090 ceph-mon[30399]: 110.5 scrub starts 2024-04-14T15:22:06.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:06 smithi090 ceph-mon[30399]: 110.5 scrub ok 2024-04-14T15:22:06.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:06 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2361563628' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:22:06.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:06 smithi090 ceph-mon[30399]: osdmap e590: 8 total, 8 up, 7 in 2024-04-14T15:22:06.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:06 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:22:06.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:06 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:22:06.509 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:22:06.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:06.534 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:06.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:22:06.635 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:22:06.658 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:06.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:06.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:22:06.760 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:22:06.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:06.784 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:06.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:22:06.885 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:22:06.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:06.911 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:06.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:22:07.012 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:22:07.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:07.037 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:07.037 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:22:07.138 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:22:07.162 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:07.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:07.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:22:07.264 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:22:07.287 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:07.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:07.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:22:07.366 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:07 smithi090 ceph-mon[41670]: pgmap v864: 214 pgs: 2 peering, 4 active+remapped+backfill_toofull, 208 active+clean; 8.4 MiB data, 498 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/1203 objects misplaced (0.166%) 2024-04-14T15:22:07.366 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:07 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:22:07.366 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:07 smithi090 ceph-mon[41670]: 5.1a scrub starts 2024-04-14T15:22:07.367 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:07 smithi090 ceph-mon[41670]: 5.1a scrub ok 2024-04-14T15:22:07.367 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:07 smithi090 ceph-mon[41670]: 4.1d scrub starts 2024-04-14T15:22:07.367 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:07 smithi090 ceph-mon[41670]: 4.1d scrub ok 2024-04-14T15:22:07.367 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:07 smithi090 ceph-mon[41670]: 5.c scrub starts 2024-04-14T15:22:07.367 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:07 smithi090 ceph-mon[41670]: 5.c scrub ok 2024-04-14T15:22:07.367 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:07 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-100","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:22:07.367 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:07 smithi090 ceph-mon[41670]: osdmap e591: 8 total, 8 up, 7 in 2024-04-14T15:22:07.367 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:07 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:22:07.367 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:07 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:22:07.367 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:07 smithi090 ceph-mon[30399]: pgmap v864: 214 pgs: 2 peering, 4 active+remapped+backfill_toofull, 208 active+clean; 8.4 MiB data, 498 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/1203 objects misplaced (0.166%) 2024-04-14T15:22:07.367 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:07 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:22:07.368 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:07 smithi090 ceph-mon[30399]: 5.1a scrub starts 2024-04-14T15:22:07.368 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:07 smithi090 ceph-mon[30399]: 5.1a scrub ok 2024-04-14T15:22:07.368 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:07 smithi090 ceph-mon[30399]: 4.1d scrub starts 2024-04-14T15:22:07.368 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:07 smithi090 ceph-mon[30399]: 4.1d scrub ok 2024-04-14T15:22:07.368 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:07 smithi090 ceph-mon[30399]: 5.c scrub starts 2024-04-14T15:22:07.368 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:07 smithi090 ceph-mon[30399]: 5.c scrub ok 2024-04-14T15:22:07.368 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:07 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-100","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:22:07.368 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:07 smithi090 ceph-mon[30399]: osdmap e591: 8 total, 8 up, 7 in 2024-04-14T15:22:07.368 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:07 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:22:07.368 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:07 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:22:07.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:07 smithi092 ceph-mon[33966]: pgmap v864: 214 pgs: 2 peering, 4 active+remapped+backfill_toofull, 208 active+clean; 8.4 MiB data, 498 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/1203 objects misplaced (0.166%) 2024-04-14T15:22:07.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:07 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:22:07.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:07 smithi092 ceph-mon[33966]: 5.1a scrub starts 2024-04-14T15:22:07.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:07 smithi092 ceph-mon[33966]: 5.1a scrub ok 2024-04-14T15:22:07.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:07 smithi092 ceph-mon[33966]: 4.1d scrub starts 2024-04-14T15:22:07.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:07 smithi092 ceph-mon[33966]: 4.1d scrub ok 2024-04-14T15:22:07.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:07 smithi092 ceph-mon[33966]: 5.c scrub starts 2024-04-14T15:22:07.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:07 smithi092 ceph-mon[33966]: 5.c scrub ok 2024-04-14T15:22:07.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:07 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-100","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:22:07.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:07 smithi092 ceph-mon[33966]: osdmap e591: 8 total, 8 up, 7 in 2024-04-14T15:22:07.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:07 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:22:07.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:07 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:22:07.389 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:22:07.412 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:07.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:07.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:22:07.514 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:22:07.537 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:07.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:07.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:22:07.639 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:22:07.663 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:07.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:07.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:22:07.765 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:22:07.788 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:07.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:07.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:22:07.890 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:22:07.907 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:22:07.913 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:07.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:07.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:22:07.947 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:22:08.015 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:22:08.038 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:08.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:08.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:22:08.139 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:08.164 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:08.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:08.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:08.177 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:08 smithi090 ceph-mon[41670]: 1.a scrub starts 2024-04-14T15:22:08.177 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:08 smithi090 ceph-mon[41670]: 1.a scrub ok 2024-04-14T15:22:08.177 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:08 smithi090 ceph-mon[41670]: pgmap v867: 245 pgs: 32 unknown, 1 peering, 4 active+remapped+backfill_toofull, 208 active+clean; 8.4 MiB data, 498 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/1203 objects misplaced (0.166%) 2024-04-14T15:22:08.177 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:08 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-04-14T15:22:08.177 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:08 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-100", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:22:08.177 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:08 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]': finished 2024-04-14T15:22:08.177 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:08 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-100"}]: dispatch 2024-04-14T15:22:08.177 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:08 smithi090 ceph-mon[41670]: osdmap e592: 8 total, 8 up, 7 in 2024-04-14T15:22:08.177 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:08 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-100"}]: dispatch 2024-04-14T15:22:08.177 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:08 smithi090 ceph-mon[30399]: 1.a scrub starts 2024-04-14T15:22:08.177 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:08 smithi090 ceph-mon[30399]: 1.a scrub ok 2024-04-14T15:22:08.178 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:08 smithi090 ceph-mon[30399]: pgmap v867: 245 pgs: 32 unknown, 1 peering, 4 active+remapped+backfill_toofull, 208 active+clean; 8.4 MiB data, 498 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/1203 objects misplaced (0.166%) 2024-04-14T15:22:08.178 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:08 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-04-14T15:22:08.178 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:08 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-100", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:22:08.178 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:08 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]': finished 2024-04-14T15:22:08.178 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:08 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-100"}]: dispatch 2024-04-14T15:22:08.178 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:08 smithi090 ceph-mon[30399]: osdmap e592: 8 total, 8 up, 7 in 2024-04-14T15:22:08.178 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:08 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-100"}]: dispatch 2024-04-14T15:22:08.266 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:22:08.290 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:08.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:08.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:22:08.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:08 smithi092 ceph-mon[33966]: 1.a scrub starts 2024-04-14T15:22:08.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:08 smithi092 ceph-mon[33966]: 1.a scrub ok 2024-04-14T15:22:08.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:08 smithi092 ceph-mon[33966]: pgmap v867: 245 pgs: 32 unknown, 1 peering, 4 active+remapped+backfill_toofull, 208 active+clean; 8.4 MiB data, 498 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/1203 objects misplaced (0.166%) 2024-04-14T15:22:08.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:08 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-04-14T15:22:08.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:08 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-100", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:22:08.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:08 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]': finished 2024-04-14T15:22:08.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:08 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-100"}]: dispatch 2024-04-14T15:22:08.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:08 smithi092 ceph-mon[33966]: osdmap e592: 8 total, 8 up, 7 in 2024-04-14T15:22:08.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:08 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-100"}]: dispatch 2024-04-14T15:22:08.392 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:22:08.416 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:08.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:08.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:22:08.518 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:22:08.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:08.543 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:08.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:22:08.644 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:22:08.668 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:08.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:08.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:22:08.769 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:22:08.793 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:08.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:08.794 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:22:08.895 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:22:08.918 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:08.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:08.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:22:09.020 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:22:09.044 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:09.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:09.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:22:09.122 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-14T15:22:09.146 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:22:09.170 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:09.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:09.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:22:09.271 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:22:09.295 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:09.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:09.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:22:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:09 smithi092 ceph-mon[33966]: 6.1e scrub starts 2024-04-14T15:22:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:09 smithi092 ceph-mon[33966]: 6.1e scrub ok 2024-04-14T15:22:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:09 smithi092 ceph-mon[33966]: 6.19 scrub starts 2024-04-14T15:22:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:09 smithi092 ceph-mon[33966]: 6.19 scrub ok 2024-04-14T15:22:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:09 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:22:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:09 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:22:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:09 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-100"}]': finished 2024-04-14T15:22:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:09 smithi092 ceph-mon[33966]: osdmap e593: 8 total, 8 up, 7 in 2024-04-14T15:22:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:09 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-100", "mode": "writeback"}]: dispatch 2024-04-14T15:22:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:09 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-100", "mode": "writeback"}]: dispatch 2024-04-14T15:22:09.397 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:22:09.420 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:09.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:09.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:22:09.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:09 smithi090 ceph-mon[41670]: 6.1e scrub starts 2024-04-14T15:22:09.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:09 smithi090 ceph-mon[41670]: 6.1e scrub ok 2024-04-14T15:22:09.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:09 smithi090 ceph-mon[41670]: 6.19 scrub starts 2024-04-14T15:22:09.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:09 smithi090 ceph-mon[41670]: 6.19 scrub ok 2024-04-14T15:22:09.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:09 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:22:09.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:09 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:22:09.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:09 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-100"}]': finished 2024-04-14T15:22:09.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:09 smithi090 ceph-mon[41670]: osdmap e593: 8 total, 8 up, 7 in 2024-04-14T15:22:09.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:09 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-100", "mode": "writeback"}]: dispatch 2024-04-14T15:22:09.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:09 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-100", "mode": "writeback"}]: dispatch 2024-04-14T15:22:09.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:09 smithi090 ceph-mon[30399]: 6.1e scrub starts 2024-04-14T15:22:09.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:09 smithi090 ceph-mon[30399]: 6.1e scrub ok 2024-04-14T15:22:09.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:09 smithi090 ceph-mon[30399]: 6.19 scrub starts 2024-04-14T15:22:09.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:09 smithi090 ceph-mon[30399]: 6.19 scrub ok 2024-04-14T15:22:09.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:09 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:22:09.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:09 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:22:09.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:09 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-6", "overlaypool": "test-rados-api-smithi090-93652-100"}]': finished 2024-04-14T15:22:09.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:09 smithi090 ceph-mon[30399]: osdmap e593: 8 total, 8 up, 7 in 2024-04-14T15:22:09.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:09 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-100", "mode": "writeback"}]: dispatch 2024-04-14T15:22:09.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:09 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-100", "mode": "writeback"}]: dispatch 2024-04-14T15:22:09.522 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:22:09.545 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:09.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:09.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:22:09.647 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:22:09.670 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:09.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:09.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:22:09.772 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:22:09.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:09.797 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:09.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:22:09.899 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:22:09.922 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:09.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:09.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:22:10.024 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:10.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:09.994+0000 7feb8d3fc700 1 -- 172.21.15.90:0/1991605854 >> v1:172.21.15.90:6789/0 conn(0x7feb88108880 legacy=0x7feb88108c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:10.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:09.995+0000 7feb8d3fc700 1 -- 172.21.15.90:0/1991605854 shutdown_connections 2024-04-14T15:22:10.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:09.995+0000 7feb8d3fc700 1 -- 172.21.15.90:0/1991605854 >> 172.21.15.90:0/1991605854 conn(0x7feb880fe3d0 msgr2=0x7feb881007f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:22:10.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:09.995+0000 7feb8d3fc700 1 -- 172.21.15.90:0/1991605854 shutdown_connections 2024-04-14T15:22:10.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:09.995+0000 7feb8d3fc700 1 -- 172.21.15.90:0/1991605854 wait complete. 2024-04-14T15:22:10.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:09.995+0000 7feb8d3fc700 1 librados: starting msgr at 2024-04-14T15:22:10.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:09.995+0000 7feb8d3fc700 1 librados: starting objecter 2024-04-14T15:22:10.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:09.995+0000 7feb8d3fc700 1 Processor -- start 2024-04-14T15:22:10.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:09.995+0000 7feb8d3fc700 1 -- start start 2024-04-14T15:22:10.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:09.995+0000 7feb8d3fc700 1 librados: setting wanted keys 2024-04-14T15:22:10.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:09.995+0000 7feb8d3fc700 1 librados: calling monclient init 2024-04-14T15:22:10.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:09.995+0000 7feb8d3fc700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7feb881a9bf0 con 0x7feb8810f300 2024-04-14T15:22:10.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:09.996+0000 7feb8d3fc700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7feb881aae10 con 0x7feb8810b730 2024-04-14T15:22:10.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:09.996+0000 7feb8d3fc700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7feb881ac050 con 0x7feb88108880 2024-04-14T15:22:10.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:09.996+0000 7feb86ffd700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7feb88108880 0x7feb8810eb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:51496/0 (socket says 172.21.15.90:51496) 2024-04-14T15:22:10.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:09.996+0000 7feb877fe700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7feb8810f300 0x7feb881a82d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:43194/0 (socket says 172.21.15.90:43194) 2024-04-14T15:22:10.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:09.996+0000 7feb86ffd700 1 -- 172.21.15.90:0/4271440367 learned_addr learned my addr 172.21.15.90:0/4271440367 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:22:10.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:09.996+0000 7feb6ffff700 10 client.?.objecter ms_handle_connect 0x7feb88108880 2024-04-14T15:22:10.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:09.996+0000 7feb6ffff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:22:10.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:09.996+0000 7feb6ffff700 10 client.?.objecter ms_handle_connect 0x7feb8810f300 2024-04-14T15:22:10.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:09.996+0000 7feb6ffff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:22:10.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:09.996+0000 7feb6ffff700 10 client.?.objecter ms_handle_connect 0x7feb8810b730 2024-04-14T15:22:10.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:09.996+0000 7feb6ffff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:22:10.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:09.996+0000 7feb6ffff700 1 -- 172.21.15.90:0/4271440367 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 43702288 0 0) 0x7feb881a9bf0 con 0x7feb8810f300 2024-04-14T15:22:10.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:09.996+0000 7feb6ffff700 1 -- 172.21.15.90:0/4271440367 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb700031f0 con 0x7feb8810f300 2024-04-14T15:22:10.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:09.996+0000 7feb6ffff700 1 -- 172.21.15.90:0/4271440367 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 912746271 0 0) 0x7feb881aae10 con 0x7feb8810b730 2024-04-14T15:22:10.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:09.996+0000 7feb6ffff700 1 -- 172.21.15.90:0/4271440367 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb881a9bf0 con 0x7feb8810b730 2024-04-14T15:22:10.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:09.996+0000 7feb6ffff700 1 -- 172.21.15.90:0/4271440367 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3025228462 0 0) 0x7feb881ac050 con 0x7feb88108880 2024-04-14T15:22:10.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:09.996+0000 7feb6ffff700 1 -- 172.21.15.90:0/4271440367 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb881aae10 con 0x7feb88108880 2024-04-14T15:22:10.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:09.996+0000 7feb6ffff700 1 -- 172.21.15.90:0/4271440367 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1293999613 0 0) 0x7feb700031f0 con 0x7feb8810f300 2024-04-14T15:22:10.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:09.996+0000 7feb6ffff700 1 -- 172.21.15.90:0/4271440367 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feb881ac050 con 0x7feb8810f300 2024-04-14T15:22:10.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:09.996+0000 7feb6ffff700 1 -- 172.21.15.90:0/4271440367 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7feb800014b0 con 0x7feb8810f300 2024-04-14T15:22:10.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:09.996+0000 7feb6ffff700 1 -- 172.21.15.90:0/4271440367 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2139277514 0 0) 0x7feb881a9bf0 con 0x7feb8810b730 2024-04-14T15:22:10.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:09.996+0000 7feb6ffff700 1 -- 172.21.15.90:0/4271440367 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feb700031f0 con 0x7feb8810b730 2024-04-14T15:22:10.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:09.996+0000 7feb6ffff700 1 -- 172.21.15.90:0/4271440367 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7feb7c0017f0 con 0x7feb8810b730 2024-04-14T15:22:10.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:09.996+0000 7feb6ffff700 1 -- 172.21.15.90:0/4271440367 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 678242525 0 0) 0x7feb881aae10 con 0x7feb88108880 2024-04-14T15:22:10.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:09.996+0000 7feb6ffff700 1 -- 172.21.15.90:0/4271440367 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feb881a9bf0 con 0x7feb88108880 2024-04-14T15:22:10.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:09.996+0000 7feb6ffff700 1 -- 172.21.15.90:0/4271440367 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2952228204 0 0) 0x7feb881ac050 con 0x7feb8810f300 2024-04-14T15:22:10.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:09.996+0000 7feb6ffff700 1 -- 172.21.15.90:0/4271440367 >> v1:172.21.15.90:6790/0 conn(0x7feb88108880 legacy=0x7feb8810eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:10.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:09.996+0000 7feb6ffff700 1 -- 172.21.15.90:0/4271440367 >> v1:172.21.15.92:6789/0 conn(0x7feb8810b730 legacy=0x7feb881a4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:10.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:09.996+0000 7feb6ffff700 1 -- 172.21.15.90:0/4271440367 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb881ad290 con 0x7feb8810f300 2024-04-14T15:22:10.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:09.996+0000 7feb8d3fc700 1 -- 172.21.15.90:0/4271440367 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7feb881a9e20 con 0x7feb8810f300 2024-04-14T15:22:10.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:09.996+0000 7feb8d3fc700 10 client.37641.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:22:10.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:09.996+0000 7feb8d3fc700 1 -- 172.21.15.90:0/4271440367 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7feb881aa2e0 con 0x7feb8810f300 2024-04-14T15:22:10.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:09.996+0000 7feb8d3fc700 1 librados: init done 2024-04-14T15:22:10.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:09.997+0000 7feb6ffff700 1 -- 172.21.15.90:0/4271440367 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7feb80003ea0 con 0x7feb8810f300 2024-04-14T15:22:10.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:09.997+0000 7feb6ffff700 1 -- 172.21.15.90:0/4271440367 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7feb80005f60 con 0x7feb8810f300 2024-04-14T15:22:10.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:09.998+0000 7feb8d3fc700 1 -- 172.21.15.90:0/4271440367 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb74005120 con 0x7feb8810f300 2024-04-14T15:22:10.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:09.998+0000 7feb6ffff700 1 -- 172.21.15.90:0/4271440367 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7feb80003a00 con 0x7feb8810f300 2024-04-14T15:22:10.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:09.998+0000 7feb6ffff700 1 -- 172.21.15.90:0/4271440367 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(594..594 src has 467..594) ==== 8309+0+0 (unknown 4182502956 0 0) 0x7feb8008ac80 con 0x7feb8810f300 2024-04-14T15:22:10.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:09.998+0000 7feb6ffff700 10 client.37641.objecter ms_dispatch 0x7feb88001550 osd_map(594..594 src has 467..594) 2024-04-14T15:22:10.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:09.998+0000 7feb6ffff700 3 client.37641.objecter handle_osd_map got epochs [594,594] > 0 2024-04-14T15:22:10.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:09.998+0000 7feb6ffff700 3 client.37641.objecter handle_osd_map decoding full epoch 594 2024-04-14T15:22:10.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:09.999+0000 7feb6ffff700 20 client.37641.objecter dump_active .. 0 homeless 2024-04-14T15:22:10.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:09.999+0000 7feb6ffff700 10 client.37641.objecter ms_handle_connect 0x7feb70071370 2024-04-14T15:22:10.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.003+0000 7feb6ffff700 1 -- 172.21.15.90:0/4271440367 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7feb80057600 con 0x7feb8810f300 2024-04-14T15:22:10.048 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:10.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:10.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:10.150 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:22:10.174 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:10.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:10.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:22:10.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.220+0000 7feb8d3fc700 1 -- 172.21.15.90:0/4271440367 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7feb74005410 con 0x7feb8810f300 2024-04-14T15:22:10.275 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:22:10.299 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:10.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:10.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:22:10.361 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:10 smithi090 ceph-mon[41670]: 2.1 scrub starts 2024-04-14T15:22:10.361 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:10 smithi090 ceph-mon[41670]: 2.1 scrub ok 2024-04-14T15:22:10.362 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:10 smithi090 ceph-mon[41670]: 6.13 scrub starts 2024-04-14T15:22:10.362 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:10 smithi090 ceph-mon[41670]: 6.13 scrub ok 2024-04-14T15:22:10.362 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:10 smithi090 ceph-mon[41670]: 4.1 scrub starts 2024-04-14T15:22:10.362 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:10 smithi090 ceph-mon[41670]: 4.1 scrub ok 2024-04-14T15:22:10.362 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:10 smithi090 ceph-mon[41670]: pgmap v870: 245 pgs: 21 unknown, 4 active+remapped+backfill_toofull, 220 active+clean; 8.4 MiB data, 498 MiB used, 625 GiB / 626 GiB avail; 2/1203 objects misplaced (0.166%) 2024-04-14T15:22:10.362 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:10 smithi090 ceph-mon[41670]: 3.0 scrub starts 2024-04-14T15:22:10.362 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:10 smithi090 ceph-mon[41670]: 3.0 scrub ok 2024-04-14T15:22:10.362 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:10 smithi090 ceph-mon[41670]: 2.1f scrub starts 2024-04-14T15:22:10.362 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:10 smithi090 ceph-mon[41670]: 2.1f scrub ok 2024-04-14T15:22:10.362 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:10 smithi090 ceph-mon[41670]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:22:10.362 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:10 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-100", "mode": "writeback"}]': finished 2024-04-14T15:22:10.362 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:10 smithi090 ceph-mon[41670]: osdmap e594: 8 total, 8 up, 7 in 2024-04-14T15:22:10.362 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:10 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-100","var": "hit_set_count","val": "2"}]: dispatch 2024-04-14T15:22:10.362 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:10 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-100","var": "hit_set_count","val": "2"}]: dispatch 2024-04-14T15:22:10.363 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:10 smithi090 ceph-mon[30399]: 2.1 scrub starts 2024-04-14T15:22:10.363 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:10 smithi090 ceph-mon[30399]: 2.1 scrub ok 2024-04-14T15:22:10.363 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:10 smithi090 ceph-mon[30399]: 6.13 scrub starts 2024-04-14T15:22:10.363 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:10 smithi090 ceph-mon[30399]: 6.13 scrub ok 2024-04-14T15:22:10.363 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:10 smithi090 ceph-mon[30399]: 4.1 scrub starts 2024-04-14T15:22:10.363 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:10 smithi090 ceph-mon[30399]: 4.1 scrub ok 2024-04-14T15:22:10.363 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:10 smithi090 ceph-mon[30399]: pgmap v870: 245 pgs: 21 unknown, 4 active+remapped+backfill_toofull, 220 active+clean; 8.4 MiB data, 498 MiB used, 625 GiB / 626 GiB avail; 2/1203 objects misplaced (0.166%) 2024-04-14T15:22:10.363 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:10 smithi090 ceph-mon[30399]: 3.0 scrub starts 2024-04-14T15:22:10.363 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:10 smithi090 ceph-mon[30399]: 3.0 scrub ok 2024-04-14T15:22:10.363 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:10 smithi090 ceph-mon[30399]: 2.1f scrub starts 2024-04-14T15:22:10.363 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:10 smithi090 ceph-mon[30399]: 2.1f scrub ok 2024-04-14T15:22:10.363 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:10 smithi090 ceph-mon[30399]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:22:10.363 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:10 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-100", "mode": "writeback"}]': finished 2024-04-14T15:22:10.363 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:10 smithi090 ceph-mon[30399]: osdmap e594: 8 total, 8 up, 7 in 2024-04-14T15:22:10.364 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:10 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-100","var": "hit_set_count","val": "2"}]: dispatch 2024-04-14T15:22:10.364 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:10 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-100","var": "hit_set_count","val": "2"}]: dispatch 2024-04-14T15:22:10.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:10 smithi092 ceph-mon[33966]: 2.1 scrub starts 2024-04-14T15:22:10.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:10 smithi092 ceph-mon[33966]: 2.1 scrub ok 2024-04-14T15:22:10.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:10 smithi092 ceph-mon[33966]: 6.13 scrub starts 2024-04-14T15:22:10.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:10 smithi092 ceph-mon[33966]: 6.13 scrub ok 2024-04-14T15:22:10.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:10 smithi092 ceph-mon[33966]: 4.1 scrub starts 2024-04-14T15:22:10.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:10 smithi092 ceph-mon[33966]: 4.1 scrub ok 2024-04-14T15:22:10.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:10 smithi092 ceph-mon[33966]: pgmap v870: 245 pgs: 21 unknown, 4 active+remapped+backfill_toofull, 220 active+clean; 8.4 MiB data, 498 MiB used, 625 GiB / 626 GiB avail; 2/1203 objects misplaced (0.166%) 2024-04-14T15:22:10.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:10 smithi092 ceph-mon[33966]: 3.0 scrub starts 2024-04-14T15:22:10.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:10 smithi092 ceph-mon[33966]: 3.0 scrub ok 2024-04-14T15:22:10.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:10 smithi092 ceph-mon[33966]: 2.1f scrub starts 2024-04-14T15:22:10.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:10 smithi092 ceph-mon[33966]: 2.1f scrub ok 2024-04-14T15:22:10.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:10 smithi092 ceph-mon[33966]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:22:10.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:10 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-100", "mode": "writeback"}]': finished 2024-04-14T15:22:10.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:10 smithi092 ceph-mon[33966]: osdmap e594: 8 total, 8 up, 7 in 2024-04-14T15:22:10.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:10 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-100","var": "hit_set_count","val": "2"}]: dispatch 2024-04-14T15:22:10.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:10 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-100","var": "hit_set_count","val": "2"}]: dispatch 2024-04-14T15:22:10.400 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:22:10.424 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:10.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:10.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:22:10.526 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:22:10.550 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:10.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:10.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:22:10.652 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:22:10.675 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:10.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:10.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:22:10.777 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:22:10.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.761+0000 7f34f17d2700 1 -- 172.21.15.90:0/1217126510 >> v1:172.21.15.90:6789/0 conn(0x7f34ec101830 legacy=0x7f34ec101c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:10.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.761+0000 7f34f17d2700 1 -- 172.21.15.90:0/1217126510 shutdown_connections 2024-04-14T15:22:10.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.761+0000 7f34f17d2700 1 -- 172.21.15.90:0/1217126510 >> 172.21.15.90:0/1217126510 conn(0x7f34ec0fd290 msgr2=0x7f34ec0ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:22:10.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.761+0000 7f34f17d2700 1 -- 172.21.15.90:0/1217126510 shutdown_connections 2024-04-14T15:22:10.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.761+0000 7f34f17d2700 1 -- 172.21.15.90:0/1217126510 wait complete. 2024-04-14T15:22:10.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.761+0000 7f34f17d2700 1 librados: starting msgr at 2024-04-14T15:22:10.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.761+0000 7f34f17d2700 1 librados: starting objecter 2024-04-14T15:22:10.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.761+0000 7f34f17d2700 1 Processor -- start 2024-04-14T15:22:10.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.761+0000 7f34f17d2700 1 -- start start 2024-04-14T15:22:10.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.761+0000 7f34f17d2700 1 librados: setting wanted keys 2024-04-14T15:22:10.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.761+0000 7f34f17d2700 1 librados: calling monclient init 2024-04-14T15:22:10.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.761+0000 7f34f17d2700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f34ec115740 con 0x7f34ec115b40 2024-04-14T15:22:10.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.761+0000 7f34f17d2700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f34ec115950 con 0x7f34ec101830 2024-04-14T15:22:10.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.761+0000 7f34f17d2700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f34ec1b0550 con 0x7f34ec107630 2024-04-14T15:22:10.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.762+0000 7f34e3fff700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f34ec107630 0x7f34ec112cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:39146/0 (socket says 172.21.15.90:39146) 2024-04-14T15:22:10.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.762+0000 7f34e3fff700 1 -- 172.21.15.90:0/3068770865 learned_addr learned my addr 172.21.15.90:0/3068770865 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:22:10.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.762+0000 7f34eb7fe700 1 --1- 172.21.15.90:0/3068770865 >> v1:172.21.15.90:6789/0 conn(0x7f34ec115b40 0x7f34ec1acdd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:42000/0 (socket says 172.21.15.90:42000) 2024-04-14T15:22:10.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.762+0000 7f34e8ff9700 10 client.?.objecter ms_handle_connect 0x7f34ec115b40 2024-04-14T15:22:10.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.762+0000 7f34e8ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:22:10.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.762+0000 7f34e8ff9700 10 client.?.objecter ms_handle_connect 0x7f34ec107630 2024-04-14T15:22:10.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.762+0000 7f34e8ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:22:10.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.762+0000 7f34e8ff9700 10 client.?.objecter ms_handle_connect 0x7f34ec101830 2024-04-14T15:22:10.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.762+0000 7f34e8ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:22:10.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.762+0000 7f34e8ff9700 1 -- 172.21.15.90:0/3068770865 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 651334569 0 0) 0x7f34ec115740 con 0x7f34ec115b40 2024-04-14T15:22:10.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.762+0000 7f34e8ff9700 1 -- 172.21.15.90:0/3068770865 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f34d80035b0 con 0x7f34ec115b40 2024-04-14T15:22:10.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.762+0000 7f34e8ff9700 1 -- 172.21.15.90:0/3068770865 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2002350421 0 0) 0x7f34ec115950 con 0x7f34ec101830 2024-04-14T15:22:10.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.762+0000 7f34e8ff9700 1 -- 172.21.15.90:0/3068770865 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f34ec115740 con 0x7f34ec101830 2024-04-14T15:22:10.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.762+0000 7f34e8ff9700 1 -- 172.21.15.90:0/3068770865 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2282757643 0 0) 0x7f34ec1b0550 con 0x7f34ec107630 2024-04-14T15:22:10.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.762+0000 7f34e8ff9700 1 -- 172.21.15.90:0/3068770865 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f34ec115950 con 0x7f34ec107630 2024-04-14T15:22:10.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.762+0000 7f34e8ff9700 1 -- 172.21.15.90:0/3068770865 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 819355118 0 0) 0x7f34ec115740 con 0x7f34ec101830 2024-04-14T15:22:10.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.762+0000 7f34e8ff9700 1 -- 172.21.15.90:0/3068770865 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f34ec1b0550 con 0x7f34ec101830 2024-04-14T15:22:10.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.762+0000 7f34e8ff9700 1 -- 172.21.15.90:0/3068770865 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f34dc0015d0 con 0x7f34ec101830 2024-04-14T15:22:10.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.762+0000 7f34e8ff9700 1 -- 172.21.15.90:0/3068770865 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4084219282 0 0) 0x7f34d80035b0 con 0x7f34ec115b40 2024-04-14T15:22:10.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.762+0000 7f34e8ff9700 1 -- 172.21.15.90:0/3068770865 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f34ec115740 con 0x7f34ec115b40 2024-04-14T15:22:10.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.762+0000 7f34e8ff9700 1 -- 172.21.15.90:0/3068770865 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f34e4001bb0 con 0x7f34ec115b40 2024-04-14T15:22:10.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.762+0000 7f34e8ff9700 1 -- 172.21.15.90:0/3068770865 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3690055565 0 0) 0x7f34ec115950 con 0x7f34ec107630 2024-04-14T15:22:10.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.762+0000 7f34e8ff9700 1 -- 172.21.15.90:0/3068770865 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f34d80035b0 con 0x7f34ec107630 2024-04-14T15:22:10.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.762+0000 7f34e8ff9700 1 -- 172.21.15.90:0/3068770865 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f34d4001830 con 0x7f34ec107630 2024-04-14T15:22:10.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.762+0000 7f34e8ff9700 1 -- 172.21.15.90:0/3068770865 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 672458892 0 0) 0x7f34ec1b0550 con 0x7f34ec101830 2024-04-14T15:22:10.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.762+0000 7f34e8ff9700 1 -- 172.21.15.90:0/3068770865 >> v1:172.21.15.90:6790/0 conn(0x7f34ec107630 legacy=0x7f34ec112cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:10.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.762+0000 7f34e8ff9700 1 -- 172.21.15.90:0/3068770865 >> v1:172.21.15.90:6789/0 conn(0x7f34ec115b40 legacy=0x7f34ec1acdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:10.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.762+0000 7f34e8ff9700 1 -- 172.21.15.90:0/3068770865 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34ec1b1790 con 0x7f34ec101830 2024-04-14T15:22:10.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.762+0000 7f34f17d2700 1 -- 172.21.15.90:0/3068770865 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f34ec1b0780 con 0x7f34ec101830 2024-04-14T15:22:10.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.762+0000 7f34f17d2700 10 client.49480.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:22:10.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.762+0000 7f34f17d2700 1 -- 172.21.15.90:0/3068770865 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f34ec1b0c40 con 0x7f34ec101830 2024-04-14T15:22:10.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.762+0000 7f34f17d2700 1 librados: init done 2024-04-14T15:22:10.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.762+0000 7f34e8ff9700 1 -- 172.21.15.90:0/3068770865 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f34dc0011c0 con 0x7f34ec101830 2024-04-14T15:22:10.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.763+0000 7f34e8ff9700 1 -- 172.21.15.90:0/3068770865 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f34dc004de0 con 0x7f34ec101830 2024-04-14T15:22:10.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.763+0000 7f34f17d2700 10 client.49480.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:22:10.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.763+0000 7f34f17d2700 15 client.49480.objecter _session_command_op_assign -1 1 2024-04-14T15:22:10.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.763+0000 7f34f17d2700 20 client.49480.objecter _get_session osd=-1 returning homeless 2024-04-14T15:22:10.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.763+0000 7f34f17d2700 10 client.49480.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:22:10.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.763+0000 7f34f17d2700 1 -- 172.21.15.90:0/3068770865 --> v1:172.21.15.92:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f34cc002920 con 0x7f34ec101830 2024-04-14T15:22:10.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.764+0000 7f34e8ff9700 1 -- 172.21.15.90:0/3068770865 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f34dc001b70 con 0x7f34ec101830 2024-04-14T15:22:10.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.764+0000 7f34e8ff9700 1 -- 172.21.15.90:0/3068770865 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(594..594 src has 467..594) ==== 8309+0+0 (unknown 4182502956 0 0) 0x7f34dc0898a0 con 0x7f34ec101830 2024-04-14T15:22:10.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.764+0000 7f34e8ff9700 10 client.49480.objecter ms_dispatch 0x7f34ec001550 osd_map(594..594 src has 467..594) 2024-04-14T15:22:10.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.764+0000 7f34e8ff9700 3 client.49480.objecter handle_osd_map got epochs [594,594] > 0 2024-04-14T15:22:10.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.764+0000 7f34e8ff9700 3 client.49480.objecter handle_osd_map decoding full epoch 594 2024-04-14T15:22:10.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.764+0000 7f34e8ff9700 10 client.49480.objecter checking command 1 2024-04-14T15:22:10.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.764+0000 7f34e8ff9700 20 client.49480.objecter _get_session s=0x7f34d807f1d0 osd=1 3 2024-04-14T15:22:10.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.764+0000 7f34e8ff9700 20 client.49480.objecter put_session s=0x7f34d807f1d0 osd=1 3 2024-04-14T15:22:10.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.764+0000 7f34e8ff9700 15 client.49480.objecter _session_command_op_remove -1 1 2024-04-14T15:22:10.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.764+0000 7f34e8ff9700 20 client.49480.objecter _get_session s=0x7f34d807f1d0 osd=1 3 2024-04-14T15:22:10.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.764+0000 7f34e8ff9700 20 client.49480.objecter get_session s=0x7f34d807f1d0 osd=1 3 2024-04-14T15:22:10.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.764+0000 7f34e8ff9700 15 client.49480.objecter _session_command_op_assign 1 1 2024-04-14T15:22:10.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.764+0000 7f34e8ff9700 20 client.49480.objecter put_session s=0x7f34d807f1d0 osd=1 4 2024-04-14T15:22:10.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.764+0000 7f34e8ff9700 10 client.49480.objecter _send_command 1 2024-04-14T15:22:10.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.764+0000 7f34e8ff9700 1 -- 172.21.15.90:0/3068770865 --> v1:172.21.15.90:6805/4170361699 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f34d8083590 con 0x7f34d807f880 2024-04-14T15:22:10.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.764+0000 7f34e8ff9700 20 client.49480.objecter dump_active .. 0 homeless 2024-04-14T15:22:10.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.764+0000 7f34e8ff9700 1 -- 172.21.15.90:0/3068770865 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_get_version_reply(handle=1 version=594) ==== 24+0+0 (unknown 3274186881 0 0) 0x7f34dc089c80 con 0x7f34ec101830 2024-04-14T15:22:10.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.765+0000 7f34e8ff9700 10 client.49480.objecter ms_handle_connect 0x7f34d80714f0 2024-04-14T15:22:10.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.765+0000 7f34e8ff9700 10 client.49480.objecter ms_handle_connect 0x7f34d807f880 2024-04-14T15:22:10.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.765+0000 7f34e8ff9700 1 -- 172.21.15.90:0/3068770865 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f34d8083590 con 0x7f34d807f880 2024-04-14T15:22:10.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.765+0000 7f34e8ff9700 10 client.49480.objecter ms_dispatch 0x7f34ec001550 command_reply(tid 1: 0 ) 2024-04-14T15:22:10.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.765+0000 7f34e8ff9700 10 client.49480.objecter _finish_command 1 = system:0 2024-04-14T15:22:10.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.765+0000 7f34e8ff9700 20 client.49480.objecter put_session s=0x7f34d807f1d0 osd=1 4 2024-04-14T15:22:10.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.765+0000 7f34e8ff9700 15 client.49480.objecter _session_command_op_remove 1 1 2024-04-14T15:22:10.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.784+0000 7f34f17d2700 10 client.49480.objecter _submit_command 2 [{"prefix": "reset_purged_snaps_last"}] 2024-04-14T15:22:10.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.784+0000 7f34f17d2700 15 client.49480.objecter _session_command_op_assign -1 2 2024-04-14T15:22:10.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.784+0000 7f34f17d2700 20 client.49480.objecter _get_session s=0x7f34d807f1d0 osd=1 3 2024-04-14T15:22:10.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.784+0000 7f34f17d2700 20 client.49480.objecter put_session s=0x7f34d807f1d0 osd=1 3 2024-04-14T15:22:10.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.784+0000 7f34f17d2700 20 client.49480.objecter _get_session s=0x7f34d807f1d0 osd=1 3 2024-04-14T15:22:10.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.784+0000 7f34f17d2700 15 client.49480.objecter _session_command_op_remove -1 2 2024-04-14T15:22:10.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.784+0000 7f34f17d2700 20 client.49480.objecter get_session s=0x7f34d807f1d0 osd=1 3 2024-04-14T15:22:10.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.784+0000 7f34f17d2700 15 client.49480.objecter _session_command_op_assign 1 2 2024-04-14T15:22:10.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.784+0000 7f34f17d2700 20 client.49480.objecter put_session s=0x7f34d807f1d0 osd=1 4 2024-04-14T15:22:10.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.784+0000 7f34f17d2700 10 client.49480.objecter _send_command 2 2024-04-14T15:22:10.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.784+0000 7f34f17d2700 1 -- 172.21.15.90:0/3068770865 --> v1:172.21.15.90:6805/4170361699 -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) -- 0x7f34cc0011f0 con 0x7f34d807f880 2024-04-14T15:22:10.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.785+0000 7f34e8ff9700 1 -- 172.21.15.90:0/3068770865 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 2: 0 ) ==== 8+0+0 (unknown 0 0 0) 0x7f34cc0011f0 con 0x7f34d807f880 2024-04-14T15:22:10.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.785+0000 7f34e8ff9700 10 client.49480.objecter ms_dispatch 0x7f34ec001550 command_reply(tid 2: 0 ) 2024-04-14T15:22:10.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.785+0000 7f34e8ff9700 10 client.49480.objecter _finish_command 2 = system:0 2024-04-14T15:22:10.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.785+0000 7f34e8ff9700 20 client.49480.objecter put_session s=0x7f34d807f1d0 osd=1 4 2024-04-14T15:22:10.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.785+0000 7f34e8ff9700 15 client.49480.objecter _session_command_op_remove 1 2 2024-04-14T15:22:10.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.785+0000 7f34f17d2700 10 librados: watch_flush enter 2024-04-14T15:22:10.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.786+0000 7f34f17d2700 10 librados: watch_flush exit 2024-04-14T15:22:10.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.786+0000 7f34f17d2700 10 client.49480.objecter close_session for osd.1 2024-04-14T15:22:10.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.786+0000 7f34f17d2700 1 -- 172.21.15.90:0/3068770865 >> v1:172.21.15.90:6805/4170361699 conn(0x7f34d807f880 legacy=0x7f34d8081d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:10.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.786+0000 7f34f17d2700 20 client.49480.objecter put_session s=0x7f34d807f1d0 osd=1 1 2024-04-14T15:22:10.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.786+0000 7f34f17d2700 20 client.49480.objecter shutdown clearing up homeless session... 2024-04-14T15:22:10.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.786+0000 7f34f17d2700 10 client.49480.objecter successfully canceled tick 2024-04-14T15:22:10.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.786+0000 7f34f17d2700 1 -- 172.21.15.90:0/3068770865 >> v1:172.21.15.90:6800/420935427 conn(0x7f34d80714f0 legacy=0x7f34d80739b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:10.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.786+0000 7f34f17d2700 1 -- 172.21.15.90:0/3068770865 >> v1:172.21.15.92:6789/0 conn(0x7f34ec101830 legacy=0x7f34ec1125e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:10.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.786+0000 7f34eb7fe700 1 -- 172.21.15.90:0/3068770865 reap_dead start 2024-04-14T15:22:10.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.786+0000 7f34f17d2700 1 -- 172.21.15.90:0/3068770865 shutdown_connections 2024-04-14T15:22:10.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.786+0000 7f34f17d2700 1 -- 172.21.15.90:0/3068770865 >> 172.21.15.90:0/3068770865 conn(0x7f34ec0fd290 msgr2=0x7f34ec0fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:22:10.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.786+0000 7f34f17d2700 1 -- 172.21.15.90:0/3068770865 shutdown_connections 2024-04-14T15:22:10.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.786+0000 7f34f17d2700 1 -- 172.21.15.90:0/3068770865 wait complete. 2024-04-14T15:22:10.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.786+0000 7f34f17d2700 1 librados: shutdown 2024-04-14T15:22:10.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:10.811 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:10.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:22:10.912 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:22:10.936 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:10.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:10.937 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:22:10.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.964+0000 7feb6ffff700 1 -- 172.21.15.90:0/4271440367 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v595) ==== 101+0+0 (unknown 3405838325 0 0) 0x7feb8005b2b0 con 0x7feb8810f300 2024-04-14T15:22:10.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.967+0000 7feb8d3fc700 10 librados: watch_flush enter 2024-04-14T15:22:10.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.967+0000 7feb8d3fc700 10 librados: watch_flush exit 2024-04-14T15:22:10.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.967+0000 7feb8d3fc700 20 client.37641.objecter shutdown clearing up homeless session... 2024-04-14T15:22:10.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.967+0000 7feb8d3fc700 10 client.37641.objecter successfully canceled tick 2024-04-14T15:22:10.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.967+0000 7feb8d3fc700 1 -- 172.21.15.90:0/4271440367 >> v1:172.21.15.90:6800/420935427 conn(0x7feb70071370 legacy=0x7feb70073830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:10.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.967+0000 7feb8d3fc700 1 -- 172.21.15.90:0/4271440367 >> v1:172.21.15.90:6789/0 conn(0x7feb8810f300 legacy=0x7feb881a82d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:10.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.967+0000 7feb8d3fc700 1 -- 172.21.15.90:0/4271440367 shutdown_connections 2024-04-14T15:22:10.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.967+0000 7feb8d3fc700 1 -- 172.21.15.90:0/4271440367 >> 172.21.15.90:0/4271440367 conn(0x7feb880fe3d0 msgr2=0x7feb88102880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:22:10.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.967+0000 7feb8d3fc700 1 -- 172.21.15.90:0/4271440367 shutdown_connections 2024-04-14T15:22:10.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.967+0000 7feb8d3fc700 1 -- 172.21.15.90:0/4271440367 wait complete. 2024-04-14T15:22:10.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:10.967+0000 7feb8d3fc700 1 librados: shutdown 2024-04-14T15:22:10.969 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is unset 2024-04-14T15:22:11.038 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:22:11.062 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:11.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:11.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:22:11.164 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:22:11.188 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:11.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:11.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:22:11.289 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:22:11.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:11.314 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:11.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:22:11.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:11 smithi092 ceph-mon[33966]: 110.b scrub starts 2024-04-14T15:22:11.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:11 smithi092 ceph-mon[33966]: 110.b scrub ok 2024-04-14T15:22:11.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:11 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/4271440367' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:22:11.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:11 smithi092 ceph-mon[33966]: 5.5 scrub starts 2024-04-14T15:22:11.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:11 smithi092 ceph-mon[33966]: 5.5 scrub ok 2024-04-14T15:22:11.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:11 smithi092 ceph-mon[33966]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-14T15:22:11.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:11 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-100","var": "hit_set_count","val": "2"}]': finished 2024-04-14T15:22:11.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:11 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/4271440367' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-14T15:22:11.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:11 smithi092 ceph-mon[33966]: osdmap e595: 8 total, 8 up, 7 in 2024-04-14T15:22:11.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:11 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-100","var": "hit_set_period","val": "600"}]: dispatch 2024-04-14T15:22:11.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:11 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-100","var": "hit_set_period","val": "600"}]: dispatch 2024-04-14T15:22:11.415 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:11.438 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:11.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:11.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:11.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:11 smithi090 ceph-mon[30399]: 110.b scrub starts 2024-04-14T15:22:11.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:11 smithi090 ceph-mon[30399]: 110.b scrub ok 2024-04-14T15:22:11.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:11 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/4271440367' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:22:11.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:11 smithi090 ceph-mon[30399]: 5.5 scrub starts 2024-04-14T15:22:11.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:11 smithi090 ceph-mon[30399]: 5.5 scrub ok 2024-04-14T15:22:11.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:11 smithi090 ceph-mon[30399]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-14T15:22:11.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:11 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-100","var": "hit_set_count","val": "2"}]': finished 2024-04-14T15:22:11.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:11 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/4271440367' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-14T15:22:11.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:11 smithi090 ceph-mon[30399]: osdmap e595: 8 total, 8 up, 7 in 2024-04-14T15:22:11.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:11 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-100","var": "hit_set_period","val": "600"}]: dispatch 2024-04-14T15:22:11.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:11 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-100","var": "hit_set_period","val": "600"}]: dispatch 2024-04-14T15:22:11.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:11 smithi090 ceph-mon[41670]: 110.b scrub starts 2024-04-14T15:22:11.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:11 smithi090 ceph-mon[41670]: 110.b scrub ok 2024-04-14T15:22:11.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:11 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/4271440367' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:22:11.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:11 smithi090 ceph-mon[41670]: 5.5 scrub starts 2024-04-14T15:22:11.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:11 smithi090 ceph-mon[41670]: 5.5 scrub ok 2024-04-14T15:22:11.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:11 smithi090 ceph-mon[41670]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-14T15:22:11.445 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:11 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-100","var": "hit_set_count","val": "2"}]': finished 2024-04-14T15:22:11.445 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:11 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/4271440367' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-14T15:22:11.445 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:11 smithi090 ceph-mon[41670]: osdmap e595: 8 total, 8 up, 7 in 2024-04-14T15:22:11.445 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:11 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-100","var": "hit_set_period","val": "600"}]: dispatch 2024-04-14T15:22:11.445 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:11 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-100","var": "hit_set_period","val": "600"}]: dispatch 2024-04-14T15:22:11.540 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:22:11.566 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:11.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:11.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:22:11.669 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:22:11.692 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:11.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:11.693 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:22:11.794 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:22:11.824 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:11.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:11.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:22:11.926 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:22:11.954 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:11.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:11.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:22:12.055 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:22:12.081 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:12.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:12.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:22:12.182 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:22:12.206 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:12.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:12.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:22:12.307 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:22:12.331 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:12.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:12.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:22:12.352 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:12 smithi090 ceph-mon[41670]: pgmap v873: 244 pgs: 1 clean+premerge+peered, 4 active+remapped+backfill_toofull, 239 active+clean; 8.4 MiB data, 498 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/1203 objects misplaced (0.166%) 2024-04-14T15:22:12.352 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:12 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-04-14T15:22:12.352 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:12 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-100","var": "hit_set_period","val": "600"}]': finished 2024-04-14T15:22:12.352 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:12 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]': finished 2024-04-14T15:22:12.352 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:12 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-14T15:22:12.352 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:12 smithi090 ceph-mon[41670]: osdmap e596: 8 total, 8 up, 7 in 2024-04-14T15:22:12.352 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:12 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-14T15:22:12.352 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:12 smithi090 ceph-mon[30399]: pgmap v873: 244 pgs: 1 clean+premerge+peered, 4 active+remapped+backfill_toofull, 239 active+clean; 8.4 MiB data, 498 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/1203 objects misplaced (0.166%) 2024-04-14T15:22:12.353 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:12 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-04-14T15:22:12.353 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:12 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-100","var": "hit_set_period","val": "600"}]': finished 2024-04-14T15:22:12.353 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:12 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]': finished 2024-04-14T15:22:12.353 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:12 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-14T15:22:12.353 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:12 smithi090 ceph-mon[30399]: osdmap e596: 8 total, 8 up, 7 in 2024-04-14T15:22:12.353 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:12 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-14T15:22:12.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:12 smithi092 ceph-mon[33966]: pgmap v873: 244 pgs: 1 clean+premerge+peered, 4 active+remapped+backfill_toofull, 239 active+clean; 8.4 MiB data, 498 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/1203 objects misplaced (0.166%) 2024-04-14T15:22:12.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:12 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-04-14T15:22:12.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:12 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-100","var": "hit_set_period","val": "600"}]': finished 2024-04-14T15:22:12.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:12 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]': finished 2024-04-14T15:22:12.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:12 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-14T15:22:12.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:12 smithi092 ceph-mon[33966]: osdmap e596: 8 total, 8 up, 7 in 2024-04-14T15:22:12.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:12 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-14T15:22:12.433 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:12.457 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:12.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:12.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:12.558 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:22:12.582 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:12.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:12.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:22:12.684 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:22:12.707 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:12.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:12.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:22:12.809 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:22:12.837 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:12.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:12.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:22:12.939 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:22:12.963 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:12.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:12.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:22:13.065 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:22:13.088 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:13.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:13.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:22:13.190 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:22:13.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:13 smithi090 ceph-mon[30399]: 2.17 scrub starts 2024-04-14T15:22:13.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:13 smithi090 ceph-mon[30399]: 2.17 scrub ok 2024-04-14T15:22:13.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:13 smithi090 ceph-mon[30399]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:22:13.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:13 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-100","var": "hit_set_type","val": "explicit_object"}]': finished 2024-04-14T15:22:13.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:13 smithi090 ceph-mon[30399]: osdmap e597: 8 total, 8 up, 7 in 2024-04-14T15:22:13.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:13 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-04-14T15:22:13.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:13 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-04-14T15:22:13.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:13 smithi090 ceph-mon[41670]: 2.17 scrub starts 2024-04-14T15:22:13.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:13 smithi090 ceph-mon[41670]: 2.17 scrub ok 2024-04-14T15:22:13.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:13 smithi090 ceph-mon[41670]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:22:13.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:13 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-100","var": "hit_set_type","val": "explicit_object"}]': finished 2024-04-14T15:22:13.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:13 smithi090 ceph-mon[41670]: osdmap e597: 8 total, 8 up, 7 in 2024-04-14T15:22:13.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:13 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-04-14T15:22:13.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:13 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-04-14T15:22:13.213 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:13.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:13.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:22:13.314 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:22:13.342 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:13.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:13.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:22:13.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:13 smithi092 ceph-mon[33966]: 2.17 scrub starts 2024-04-14T15:22:13.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:13 smithi092 ceph-mon[33966]: 2.17 scrub ok 2024-04-14T15:22:13.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:13 smithi092 ceph-mon[33966]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:22:13.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:13 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-100","var": "hit_set_type","val": "explicit_object"}]': finished 2024-04-14T15:22:13.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:13 smithi092 ceph-mon[33966]: osdmap e597: 8 total, 8 up, 7 in 2024-04-14T15:22:13.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:13 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-04-14T15:22:13.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:13 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-04-14T15:22:13.445 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:22:13.468 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:13.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:13.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:22:13.569 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:13.593 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:13.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:13.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:13.695 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:13.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:13.720 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:13.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:13.821 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:22:13.844 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:13.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:13.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:22:13.946 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:13.970 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:13.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:13.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:14.034 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd set noscrub 2024-04-14T15:22:14.071 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:14.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:14.101 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:14.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:14.202 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:14.215 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:22:13 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:22:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:22:14.215 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:14 smithi090 ceph-mon[30399]: 6.14 scrub starts 2024-04-14T15:22:14.215 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:14 smithi090 ceph-mon[30399]: 6.14 scrub ok 2024-04-14T15:22:14.215 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:14 smithi090 ceph-mon[30399]: 110.d scrub starts 2024-04-14T15:22:14.215 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:14 smithi090 ceph-mon[30399]: 110.d scrub ok 2024-04-14T15:22:14.216 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:14 smithi090 ceph-mon[30399]: pgmap v876: 244 pgs: 1 clean+premerge+peered, 4 active+remapped+backfill_toofull, 239 active+clean; 8.4 MiB data, 498 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/1203 objects misplaced (0.166%) 2024-04-14T15:22:14.216 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:14 smithi090 ceph-mon[30399]: 4.2 scrub starts 2024-04-14T15:22:14.216 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:14 smithi090 ceph-mon[30399]: 4.2 scrub ok 2024-04-14T15:22:14.216 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:14 smithi090 ceph-mon[30399]: 6.0 scrub starts 2024-04-14T15:22:14.216 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:14 smithi090 ceph-mon[30399]: 6.0 scrub ok 2024-04-14T15:22:14.216 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:14 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-100","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-04-14T15:22:14.216 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:14 smithi090 ceph-mon[30399]: osdmap e598: 8 total, 8 up, 7 in 2024-04-14T15:22:14.216 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:14 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:22:14.216 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:14 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:22:14.216 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:14 smithi090 ceph-mon[41670]: 6.14 scrub starts 2024-04-14T15:22:14.216 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:14 smithi090 ceph-mon[41670]: 6.14 scrub ok 2024-04-14T15:22:14.216 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:14 smithi090 ceph-mon[41670]: 110.d scrub starts 2024-04-14T15:22:14.217 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:14 smithi090 ceph-mon[41670]: 110.d scrub ok 2024-04-14T15:22:14.217 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:14 smithi090 ceph-mon[41670]: pgmap v876: 244 pgs: 1 clean+premerge+peered, 4 active+remapped+backfill_toofull, 239 active+clean; 8.4 MiB data, 498 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/1203 objects misplaced (0.166%) 2024-04-14T15:22:14.217 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:14 smithi090 ceph-mon[41670]: 4.2 scrub starts 2024-04-14T15:22:14.217 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:14 smithi090 ceph-mon[41670]: 4.2 scrub ok 2024-04-14T15:22:14.217 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:14 smithi090 ceph-mon[41670]: 6.0 scrub starts 2024-04-14T15:22:14.217 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:14 smithi090 ceph-mon[41670]: 6.0 scrub ok 2024-04-14T15:22:14.217 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:14 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-100","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-04-14T15:22:14.217 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:14 smithi090 ceph-mon[41670]: osdmap e598: 8 total, 8 up, 7 in 2024-04-14T15:22:14.217 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:14 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:22:14.217 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:14 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:22:14.228 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:14.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:14.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:14.247 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph tell osd.2 reset_purged_snaps_last 2024-04-14T15:22:14.330 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:22:14.354 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:14.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:14.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:22:14.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:14 smithi092 ceph-mon[33966]: 6.14 scrub starts 2024-04-14T15:22:14.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:14 smithi092 ceph-mon[33966]: 6.14 scrub ok 2024-04-14T15:22:14.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:14 smithi092 ceph-mon[33966]: 110.d scrub starts 2024-04-14T15:22:14.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:14 smithi092 ceph-mon[33966]: 110.d scrub ok 2024-04-14T15:22:14.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:14 smithi092 ceph-mon[33966]: pgmap v876: 244 pgs: 1 clean+premerge+peered, 4 active+remapped+backfill_toofull, 239 active+clean; 8.4 MiB data, 498 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/1203 objects misplaced (0.166%) 2024-04-14T15:22:14.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:14 smithi092 ceph-mon[33966]: 4.2 scrub starts 2024-04-14T15:22:14.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:14 smithi092 ceph-mon[33966]: 4.2 scrub ok 2024-04-14T15:22:14.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:14 smithi092 ceph-mon[33966]: 6.0 scrub starts 2024-04-14T15:22:14.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:14 smithi092 ceph-mon[33966]: 6.0 scrub ok 2024-04-14T15:22:14.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:14 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-100","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-04-14T15:22:14.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:14 smithi092 ceph-mon[33966]: osdmap e598: 8 total, 8 up, 7 in 2024-04-14T15:22:14.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:14 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:22:14.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:14 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:22:14.456 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:22:14.484 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:14.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:14.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:22:14.585 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:22:14.609 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:14.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:14.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:22:14.624 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:22:14.662 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:22:14.668 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-14T15:22:14.668 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:22:14.711 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:22:14.738 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:14.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:14.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:22:14.840 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:22:14.865 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:14.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:14.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:22:14.967 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:22:14.997 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:14.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:14.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:22:15.099 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:22:15.122 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:15.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:15.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:22:15.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:15 smithi090 ceph-mon[41670]: 11.10 scrub starts 2024-04-14T15:22:15.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:15 smithi090 ceph-mon[41670]: 11.10 scrub ok 2024-04-14T15:22:15.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:15 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:22:15.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:15 smithi090 ceph-mon[41670]: osdmap e599: 8 total, 8 up, 7 in 2024-04-14T15:22:15.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:15 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-100"}]: dispatch 2024-04-14T15:22:15.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:15 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-100"}]: dispatch 2024-04-14T15:22:15.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:15 smithi090 ceph-mon[30399]: 11.10 scrub starts 2024-04-14T15:22:15.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:15 smithi090 ceph-mon[30399]: 11.10 scrub ok 2024-04-14T15:22:15.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:15 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:22:15.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:15 smithi090 ceph-mon[30399]: osdmap e599: 8 total, 8 up, 7 in 2024-04-14T15:22:15.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:15 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-100"}]: dispatch 2024-04-14T15:22:15.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:15 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-100"}]: dispatch 2024-04-14T15:22:15.224 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:22:15.247 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:15.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:15.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:22:15.296 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:15 smithi092 ceph-mon[33966]: 11.10 scrub starts 2024-04-14T15:22:15.296 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:15 smithi092 ceph-mon[33966]: 11.10 scrub ok 2024-04-14T15:22:15.296 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:15 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]': finished 2024-04-14T15:22:15.296 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:15 smithi092 ceph-mon[33966]: osdmap e599: 8 total, 8 up, 7 in 2024-04-14T15:22:15.296 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:15 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-100"}]: dispatch 2024-04-14T15:22:15.296 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:15 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-100"}]: dispatch 2024-04-14T15:22:15.349 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:22:15.376 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:15.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:15.377 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:22:15.478 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:22:15.501 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:15.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:15.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:22:15.603 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:22:15.622 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:22:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:22:15.630 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:15.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:15.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:22:15.732 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:22:15.758 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:15.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:15.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:22:15.860 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:22:15.885 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:15.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:15.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:22:15.986 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:22:16.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:16.013 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:16.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:22:16.114 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:22:16.137 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:16.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:16.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:22:16.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:16 smithi090 ceph-mon[41670]: 11.2 scrub starts 2024-04-14T15:22:16.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:16 smithi090 ceph-mon[41670]: 11.2 scrub ok 2024-04-14T15:22:16.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:16 smithi090 ceph-mon[41670]: pgmap v879: 243 pgs: 1 peering, 4 active+remapped+backfill_toofull, 238 active+clean; 8.4 MiB data, 516 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s; 2/1206 objects misplaced (0.166%) 2024-04-14T15:22:16.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:16 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-04-14T15:22:16.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:16 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:22:16.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:16 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-100"}]': finished 2024-04-14T15:22:16.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:16 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]': finished 2024-04-14T15:22:16.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:16 smithi090 ceph-mon[41670]: osdmap e600: 8 total, 8 up, 7 in 2024-04-14T15:22:16.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:16 smithi090 ceph-mon[30399]: 11.2 scrub starts 2024-04-14T15:22:16.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:16 smithi090 ceph-mon[30399]: 11.2 scrub ok 2024-04-14T15:22:16.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:16 smithi090 ceph-mon[30399]: pgmap v879: 243 pgs: 1 peering, 4 active+remapped+backfill_toofull, 238 active+clean; 8.4 MiB data, 516 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s; 2/1206 objects misplaced (0.166%) 2024-04-14T15:22:16.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:16 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-04-14T15:22:16.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:16 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:22:16.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:16 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-100"}]': finished 2024-04-14T15:22:16.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:16 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]': finished 2024-04-14T15:22:16.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:16 smithi090 ceph-mon[30399]: osdmap e600: 8 total, 8 up, 7 in 2024-04-14T15:22:16.239 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:22:16.267 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:16.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:16.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:22:16.368 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:22:16.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:16 smithi092 ceph-mon[33966]: 11.2 scrub starts 2024-04-14T15:22:16.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:16 smithi092 ceph-mon[33966]: 11.2 scrub ok 2024-04-14T15:22:16.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:16 smithi092 ceph-mon[33966]: pgmap v879: 243 pgs: 1 peering, 4 active+remapped+backfill_toofull, 238 active+clean; 8.4 MiB data, 516 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s; 2/1206 objects misplaced (0.166%) 2024-04-14T15:22:16.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:16 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-04-14T15:22:16.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:16 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:22:16.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:16 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-100"}]': finished 2024-04-14T15:22:16.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:16 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]': finished 2024-04-14T15:22:16.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:16 smithi092 ceph-mon[33966]: osdmap e600: 8 total, 8 up, 7 in 2024-04-14T15:22:16.396 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:16.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:16.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:22:16.497 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:22:16.521 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:16.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:16.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:22:16.622 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:22:16.647 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:16.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:16.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:22:16.749 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:22:16.772 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:16.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:16.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:22:16.874 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:22:16.897 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:16.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:16.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:22:16.999 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:22:17.085 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:17.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:17.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:22:17.187 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:22:17.210 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:17.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:17.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:22:17.313 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:22:17.336 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:17.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:17.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:22:17.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:17 smithi092 ceph-mon[33966]: 2.1d deep-scrub starts 2024-04-14T15:22:17.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:17 smithi092 ceph-mon[33966]: 2.1d deep-scrub ok 2024-04-14T15:22:17.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:17 smithi092 ceph-mon[33966]: 110.1e deep-scrub starts 2024-04-14T15:22:17.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:17 smithi092 ceph-mon[33966]: 110.1e deep-scrub ok 2024-04-14T15:22:17.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:17 smithi092 ceph-mon[33966]: osdmap e601: 8 total, 8 up, 7 in 2024-04-14T15:22:17.438 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:22:17.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:17 smithi090 ceph-mon[30399]: 2.1d deep-scrub starts 2024-04-14T15:22:17.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:17 smithi090 ceph-mon[30399]: 2.1d deep-scrub ok 2024-04-14T15:22:17.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:17 smithi090 ceph-mon[30399]: 110.1e deep-scrub starts 2024-04-14T15:22:17.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:17 smithi090 ceph-mon[30399]: 110.1e deep-scrub ok 2024-04-14T15:22:17.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:17 smithi090 ceph-mon[30399]: osdmap e601: 8 total, 8 up, 7 in 2024-04-14T15:22:17.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:17 smithi090 ceph-mon[41670]: 2.1d deep-scrub starts 2024-04-14T15:22:17.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:17 smithi090 ceph-mon[41670]: 2.1d deep-scrub ok 2024-04-14T15:22:17.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:17 smithi090 ceph-mon[41670]: 110.1e deep-scrub starts 2024-04-14T15:22:17.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:17 smithi090 ceph-mon[41670]: 110.1e deep-scrub ok 2024-04-14T15:22:17.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:17 smithi090 ceph-mon[41670]: osdmap e601: 8 total, 8 up, 7 in 2024-04-14T15:22:17.466 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:17.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:17.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:22:17.567 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:22:17.591 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:17.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:17.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:22:17.675 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-14T15:22:17.693 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:22:17.741 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:17.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:17.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:22:17.842 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:22:17.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:17.867 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:17.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:22:17.968 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:22:17.998 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:17.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:17.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:22:18.100 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:22:18.125 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:18.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:18.126 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:22:18.227 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:18.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:18.251 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:18.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:18.305 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:18 smithi090 ceph-mon[30399]: 2.10 scrub starts 2024-04-14T15:22:18.305 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:18 smithi090 ceph-mon[30399]: 2.10 scrub ok 2024-04-14T15:22:18.305 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:18 smithi090 ceph-mon[30399]: pgmap v882: 211 pgs: 1 peering, 4 active+remapped+backfill_toofull, 206 active+clean; 8.4 MiB data, 516 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s; 2/1206 objects misplaced (0.166%) 2024-04-14T15:22:18.305 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:18 smithi090 ceph-mon[30399]: 2.d scrub starts 2024-04-14T15:22:18.305 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:18 smithi090 ceph-mon[30399]: 2.d scrub ok 2024-04-14T15:22:18.305 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:18 smithi090 ceph-mon[30399]: osdmap e602: 8 total, 8 up, 7 in 2024-04-14T15:22:18.305 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:18 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:22:18.305 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:18 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:22:18.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:18 smithi090 ceph-mon[41670]: 2.10 scrub starts 2024-04-14T15:22:18.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:18 smithi090 ceph-mon[41670]: 2.10 scrub ok 2024-04-14T15:22:18.306 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:18 smithi090 ceph-mon[41670]: pgmap v882: 211 pgs: 1 peering, 4 active+remapped+backfill_toofull, 206 active+clean; 8.4 MiB data, 516 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s; 2/1206 objects misplaced (0.166%) 2024-04-14T15:22:18.306 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:18 smithi090 ceph-mon[41670]: 2.d scrub starts 2024-04-14T15:22:18.306 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:18 smithi090 ceph-mon[41670]: 2.d scrub ok 2024-04-14T15:22:18.306 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:18 smithi090 ceph-mon[41670]: osdmap e602: 8 total, 8 up, 7 in 2024-04-14T15:22:18.306 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:18 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:22:18.306 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:18 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:22:18.352 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:22:18.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:18 smithi092 ceph-mon[33966]: 2.10 scrub starts 2024-04-14T15:22:18.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:18 smithi092 ceph-mon[33966]: 2.10 scrub ok 2024-04-14T15:22:18.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:18 smithi092 ceph-mon[33966]: pgmap v882: 211 pgs: 1 peering, 4 active+remapped+backfill_toofull, 206 active+clean; 8.4 MiB data, 516 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s; 2/1206 objects misplaced (0.166%) 2024-04-14T15:22:18.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:18 smithi092 ceph-mon[33966]: 2.d scrub starts 2024-04-14T15:22:18.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:18 smithi092 ceph-mon[33966]: 2.d scrub ok 2024-04-14T15:22:18.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:18 smithi092 ceph-mon[33966]: osdmap e602: 8 total, 8 up, 7 in 2024-04-14T15:22:18.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:18 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:22:18.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:18 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:22:18.380 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:18.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:18.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:22:18.482 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:22:18.505 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:18.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:18.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:22:18.607 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:22:18.630 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:18.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:18.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:22:18.732 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:22:18.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:18.758 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:18.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:22:18.859 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:22:18.887 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:18.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:18.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:22:18.989 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:22:19.016 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:19.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:19.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:22:19.118 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:19.144 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:19.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:19.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:19.246 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:22:19.270 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:19.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:19.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:22:19.328 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:19 smithi090 ceph-mon[30399]: 1.2 deep-scrub starts 2024-04-14T15:22:19.328 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:19 smithi090 ceph-mon[30399]: 1.2 deep-scrub ok 2024-04-14T15:22:19.328 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:19 smithi090 ceph-mon[30399]: 110.12 scrub starts 2024-04-14T15:22:19.328 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:19 smithi090 ceph-mon[30399]: 110.12 scrub ok 2024-04-14T15:22:19.329 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:19 smithi090 ceph-mon[30399]: 11.19 deep-scrub starts 2024-04-14T15:22:19.329 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:19 smithi090 ceph-mon[30399]: 11.19 deep-scrub ok 2024-04-14T15:22:19.329 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:19 smithi090 ceph-mon[30399]: 5.1d scrub starts 2024-04-14T15:22:19.329 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:19 smithi090 ceph-mon[30399]: 5.1d scrub ok 2024-04-14T15:22:19.329 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:19 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:22:19.329 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:19 smithi090 ceph-mon[30399]: osdmap e603: 8 total, 8 up, 7 in 2024-04-14T15:22:19.329 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:19 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-14T15:22:19.329 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:19 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-14T15:22:19.329 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:19 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-14T15:22:19.329 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:19 smithi090 ceph-mon[41670]: 1.2 deep-scrub starts 2024-04-14T15:22:19.329 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:19 smithi090 ceph-mon[41670]: 1.2 deep-scrub ok 2024-04-14T15:22:19.329 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:19 smithi090 ceph-mon[41670]: 110.12 scrub starts 2024-04-14T15:22:19.330 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:19 smithi090 ceph-mon[41670]: 110.12 scrub ok 2024-04-14T15:22:19.330 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:19 smithi090 ceph-mon[41670]: 11.19 deep-scrub starts 2024-04-14T15:22:19.330 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:19 smithi090 ceph-mon[41670]: 11.19 deep-scrub ok 2024-04-14T15:22:19.330 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:19 smithi090 ceph-mon[41670]: 5.1d scrub starts 2024-04-14T15:22:19.330 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:19 smithi090 ceph-mon[41670]: 5.1d scrub ok 2024-04-14T15:22:19.330 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:19 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:22:19.330 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:19 smithi090 ceph-mon[41670]: osdmap e603: 8 total, 8 up, 7 in 2024-04-14T15:22:19.330 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:19 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-14T15:22:19.330 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:19 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-14T15:22:19.330 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:19 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-14T15:22:19.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.330+0000 7fa875bfc700 1 -- 172.21.15.90:0/1013705331 >> v1:172.21.15.90:6789/0 conn(0x7fa87010f3a0 legacy=0x7fa8701117f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:19.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.330+0000 7fa875bfc700 1 -- 172.21.15.90:0/1013705331 shutdown_connections 2024-04-14T15:22:19.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.330+0000 7fa875bfc700 1 -- 172.21.15.90:0/1013705331 >> 172.21.15.90:0/1013705331 conn(0x7fa8700fd290 msgr2=0x7fa8700ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:22:19.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.330+0000 7fa875bfc700 1 -- 172.21.15.90:0/1013705331 shutdown_connections 2024-04-14T15:22:19.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.330+0000 7fa875bfc700 1 -- 172.21.15.90:0/1013705331 wait complete. 2024-04-14T15:22:19.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.330+0000 7fa875bfc700 1 librados: starting msgr at 2024-04-14T15:22:19.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.330+0000 7fa875bfc700 1 librados: starting objecter 2024-04-14T15:22:19.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.331+0000 7fa875bfc700 1 Processor -- start 2024-04-14T15:22:19.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.331+0000 7fa875bfc700 1 -- start start 2024-04-14T15:22:19.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.331+0000 7fa875bfc700 1 librados: setting wanted keys 2024-04-14T15:22:19.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.331+0000 7fa875bfc700 1 librados: calling monclient init 2024-04-14T15:22:19.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.331+0000 7fa875bfc700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa8701a99c0 con 0x7fa87010b890 2024-04-14T15:22:19.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.331+0000 7fa875bfc700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa8701aabe0 con 0x7fa870103a80 2024-04-14T15:22:19.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.331+0000 7fa875bfc700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa8701abe00 con 0x7fa87010f3a0 2024-04-14T15:22:19.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.331+0000 7fa86effd700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fa87010b890 0x7fa8701a4b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:42028/0 (socket says 172.21.15.90:42028) 2024-04-14T15:22:19.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.331+0000 7fa86effd700 1 -- 172.21.15.90:0/3093729176 learned_addr learned my addr 172.21.15.90:0/3093729176 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:22:19.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.331+0000 7fa86cff9700 10 client.?.objecter ms_handle_connect 0x7fa87010b890 2024-04-14T15:22:19.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.331+0000 7fa86cff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:22:19.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.331+0000 7fa86cff9700 10 client.?.objecter ms_handle_connect 0x7fa870103a80 2024-04-14T15:22:19.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.331+0000 7fa86cff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:22:19.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.331+0000 7fa86cff9700 10 client.?.objecter ms_handle_connect 0x7fa87010f3a0 2024-04-14T15:22:19.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.331+0000 7fa86cff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:22:19.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.331+0000 7fa86cff9700 1 -- 172.21.15.90:0/3093729176 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3668281422 0 0) 0x7fa8701a99c0 con 0x7fa87010b890 2024-04-14T15:22:19.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.332+0000 7fa86cff9700 1 -- 172.21.15.90:0/3093729176 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa85c0031f0 con 0x7fa87010b890 2024-04-14T15:22:19.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.332+0000 7fa86cff9700 1 -- 172.21.15.90:0/3093729176 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2645743152 0 0) 0x7fa8701aabe0 con 0x7fa870103a80 2024-04-14T15:22:19.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.332+0000 7fa86cff9700 1 -- 172.21.15.90:0/3093729176 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa8701a99c0 con 0x7fa870103a80 2024-04-14T15:22:19.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.332+0000 7fa86cff9700 1 -- 172.21.15.90:0/3093729176 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3167669417 0 0) 0x7fa8701abe00 con 0x7fa87010f3a0 2024-04-14T15:22:19.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.332+0000 7fa86cff9700 1 -- 172.21.15.90:0/3093729176 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa8701aabe0 con 0x7fa87010f3a0 2024-04-14T15:22:19.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.332+0000 7fa86cff9700 1 -- 172.21.15.90:0/3093729176 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2798326604 0 0) 0x7fa85c0031f0 con 0x7fa87010b890 2024-04-14T15:22:19.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.332+0000 7fa86cff9700 1 -- 172.21.15.90:0/3093729176 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa8701abe00 con 0x7fa87010b890 2024-04-14T15:22:19.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.332+0000 7fa86cff9700 1 -- 172.21.15.90:0/3093729176 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fa858001950 con 0x7fa87010b890 2024-04-14T15:22:19.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.332+0000 7fa86cff9700 1 -- 172.21.15.90:0/3093729176 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1508451392 0 0) 0x7fa8701a99c0 con 0x7fa870103a80 2024-04-14T15:22:19.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.332+0000 7fa86cff9700 1 -- 172.21.15.90:0/3093729176 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa85c0031f0 con 0x7fa870103a80 2024-04-14T15:22:19.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.332+0000 7fa86cff9700 1 -- 172.21.15.90:0/3093729176 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fa8600017b0 con 0x7fa870103a80 2024-04-14T15:22:19.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.332+0000 7fa86cff9700 1 -- 172.21.15.90:0/3093729176 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 240193992 0 0) 0x7fa8701aabe0 con 0x7fa87010f3a0 2024-04-14T15:22:19.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.332+0000 7fa86cff9700 1 -- 172.21.15.90:0/3093729176 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa8701a99c0 con 0x7fa87010f3a0 2024-04-14T15:22:19.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.332+0000 7fa86cff9700 1 -- 172.21.15.90:0/3093729176 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fa868004910 con 0x7fa87010f3a0 2024-04-14T15:22:19.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.332+0000 7fa86cff9700 1 -- 172.21.15.90:0/3093729176 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2725318160 0 0) 0x7fa8701abe00 con 0x7fa87010b890 2024-04-14T15:22:19.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.332+0000 7fa86cff9700 1 -- 172.21.15.90:0/3093729176 >> v1:172.21.15.90:6790/0 conn(0x7fa87010f3a0 legacy=0x7fa8701a82a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:19.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.332+0000 7fa86cff9700 1 -- 172.21.15.90:0/3093729176 >> v1:172.21.15.92:6789/0 conn(0x7fa870103a80 legacy=0x7fa87010aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:19.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.332+0000 7fa86cff9700 1 -- 172.21.15.90:0/3093729176 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8701ad040 con 0x7fa87010b890 2024-04-14T15:22:19.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.332+0000 7fa875bfc700 1 -- 172.21.15.90:0/3093729176 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa8701aae10 con 0x7fa87010b890 2024-04-14T15:22:19.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.332+0000 7fa875bfc700 10 client.37653.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:22:19.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.332+0000 7fa875bfc700 1 -- 172.21.15.90:0/3093729176 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa8701ab2d0 con 0x7fa87010b890 2024-04-14T15:22:19.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.332+0000 7fa875bfc700 1 librados: init done 2024-04-14T15:22:19.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.332+0000 7fa86cff9700 1 -- 172.21.15.90:0/3093729176 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fa858001d20 con 0x7fa87010b890 2024-04-14T15:22:19.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.332+0000 7fa86cff9700 1 -- 172.21.15.90:0/3093729176 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fa858005d70 con 0x7fa87010b890 2024-04-14T15:22:19.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.333+0000 7fa875bfc700 1 -- 172.21.15.90:0/3093729176 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fa850005130 con 0x7fa87010b890 2024-04-14T15:22:19.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.334+0000 7fa86cff9700 1 -- 172.21.15.90:0/3093729176 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fa858003950 con 0x7fa87010b890 2024-04-14T15:22:19.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.335+0000 7fa86cff9700 1 -- 172.21.15.90:0/3093729176 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(603..603 src has 467..603) ==== 8301+0+0 (unknown 1748645378 0 0) 0x7fa85808a940 con 0x7fa87010b890 2024-04-14T15:22:19.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.335+0000 7fa86cff9700 10 client.37653.objecter ms_dispatch 0x7fa870001550 osd_map(603..603 src has 467..603) 2024-04-14T15:22:19.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.335+0000 7fa86cff9700 3 client.37653.objecter handle_osd_map got epochs [603,603] > 0 2024-04-14T15:22:19.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.335+0000 7fa86cff9700 3 client.37653.objecter handle_osd_map decoding full epoch 603 2024-04-14T15:22:19.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.335+0000 7fa86cff9700 20 client.37653.objecter dump_active .. 0 homeless 2024-04-14T15:22:19.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.335+0000 7fa86cff9700 1 -- 172.21.15.90:0/3093729176 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v603) ==== 54+0+16 (unknown 3091172218 0 2420000192) 0x7fa85808ad80 con 0x7fa87010b890 2024-04-14T15:22:19.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.335+0000 7fa86cff9700 10 client.37653.objecter ms_handle_connect 0x7fa85c079d10 2024-04-14T15:22:19.337 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:22:19.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.335+0000 7fa875bfc700 10 client.37653.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:22:19.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.335+0000 7fa875bfc700 15 client.37653.objecter _session_command_op_assign -1 1 2024-04-14T15:22:19.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.335+0000 7fa875bfc700 20 client.37653.objecter _get_session s=0x7fa850005410 osd=0 3 2024-04-14T15:22:19.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.335+0000 7fa875bfc700 20 client.37653.objecter put_session s=0x7fa850005410 osd=0 3 2024-04-14T15:22:19.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.335+0000 7fa875bfc700 20 client.37653.objecter _get_session s=0x7fa850005410 osd=0 3 2024-04-14T15:22:19.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.335+0000 7fa875bfc700 15 client.37653.objecter _session_command_op_remove -1 1 2024-04-14T15:22:19.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.335+0000 7fa875bfc700 20 client.37653.objecter get_session s=0x7fa850005410 osd=0 3 2024-04-14T15:22:19.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.335+0000 7fa875bfc700 15 client.37653.objecter _session_command_op_assign 0 1 2024-04-14T15:22:19.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.335+0000 7fa875bfc700 20 client.37653.objecter put_session s=0x7fa850005410 osd=0 4 2024-04-14T15:22:19.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.335+0000 7fa875bfc700 10 client.37653.objecter _send_command 1 2024-04-14T15:22:19.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.335+0000 7fa875bfc700 1 -- 172.21.15.90:0/3093729176 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa850004ba0 con 0x7fa850004760 2024-04-14T15:22:19.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.336+0000 7fa86cff9700 10 client.37653.objecter ms_handle_connect 0x7fa850004760 2024-04-14T15:22:19.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.336+0000 7fa86cff9700 1 -- 172.21.15.90:0/3093729176 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fa850004ba0 con 0x7fa850004760 2024-04-14T15:22:19.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.336+0000 7fa86cff9700 10 client.37653.objecter ms_dispatch 0x7fa870001550 command_reply(tid 1: 0 ) 2024-04-14T15:22:19.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.336+0000 7fa86cff9700 10 client.37653.objecter _finish_command 1 = system:0 2024-04-14T15:22:19.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.336+0000 7fa86cff9700 20 client.37653.objecter put_session s=0x7fa850005410 osd=0 4 2024-04-14T15:22:19.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.336+0000 7fa86cff9700 15 client.37653.objecter _session_command_op_remove 0 1 2024-04-14T15:22:19.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.363+0000 7fa875bfc700 10 client.37653.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:22:19.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.363+0000 7fa875bfc700 15 client.37653.objecter _session_command_op_assign -1 2 2024-04-14T15:22:19.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.363+0000 7fa875bfc700 20 client.37653.objecter _get_session s=0x7fa850005410 osd=0 3 2024-04-14T15:22:19.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.363+0000 7fa875bfc700 20 client.37653.objecter put_session s=0x7fa850005410 osd=0 3 2024-04-14T15:22:19.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.363+0000 7fa875bfc700 20 client.37653.objecter _get_session s=0x7fa850005410 osd=0 3 2024-04-14T15:22:19.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.363+0000 7fa875bfc700 15 client.37653.objecter _session_command_op_remove -1 2 2024-04-14T15:22:19.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.363+0000 7fa875bfc700 20 client.37653.objecter get_session s=0x7fa850005410 osd=0 3 2024-04-14T15:22:19.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.363+0000 7fa875bfc700 15 client.37653.objecter _session_command_op_assign 0 2 2024-04-14T15:22:19.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.363+0000 7fa875bfc700 20 client.37653.objecter put_session s=0x7fa850005410 osd=0 4 2024-04-14T15:22:19.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.363+0000 7fa875bfc700 10 client.37653.objecter _send_command 2 2024-04-14T15:22:19.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.363+0000 7fa875bfc700 1 -- 172.21.15.90:0/3093729176 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa850007a20 con 0x7fa850004760 2024-04-14T15:22:19.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.365+0000 7fa86cff9700 1 -- 172.21.15.90:0/3093729176 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fa850007a20 con 0x7fa850004760 2024-04-14T15:22:19.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.365+0000 7fa86cff9700 10 client.37653.objecter ms_dispatch 0x7fa870001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:22:19.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.365+0000 7fa86cff9700 10 client.37653.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:22:19.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.365+0000 7fa86cff9700 20 client.37653.objecter put_session s=0x7fa850005410 osd=0 4 2024-04-14T15:22:19.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.365+0000 7fa86cff9700 15 client.37653.objecter _session_command_op_remove 0 2 2024-04-14T15:22:19.366 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:22:19.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.365+0000 7fa875bfc700 10 client.37653.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:22:19.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.365+0000 7fa875bfc700 15 client.37653.objecter _session_command_op_assign -1 3 2024-04-14T15:22:19.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.365+0000 7fa875bfc700 20 client.37653.objecter _get_session s=0x7fa850004160 osd=1 3 2024-04-14T15:22:19.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.365+0000 7fa875bfc700 20 client.37653.objecter put_session s=0x7fa850004160 osd=1 3 2024-04-14T15:22:19.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.365+0000 7fa875bfc700 20 client.37653.objecter _get_session s=0x7fa850004160 osd=1 3 2024-04-14T15:22:19.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.365+0000 7fa875bfc700 15 client.37653.objecter _session_command_op_remove -1 3 2024-04-14T15:22:19.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.365+0000 7fa875bfc700 20 client.37653.objecter get_session s=0x7fa850004160 osd=1 3 2024-04-14T15:22:19.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.365+0000 7fa875bfc700 15 client.37653.objecter _session_command_op_assign 1 3 2024-04-14T15:22:19.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.365+0000 7fa875bfc700 20 client.37653.objecter put_session s=0x7fa850004160 osd=1 4 2024-04-14T15:22:19.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.365+0000 7fa875bfc700 10 client.37653.objecter _send_command 3 2024-04-14T15:22:19.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.365+0000 7fa875bfc700 1 -- 172.21.15.90:0/3093729176 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fa8500022d0 con 0x7fa850004340 2024-04-14T15:22:19.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.366+0000 7fa86cff9700 10 client.37653.objecter ms_handle_connect 0x7fa850004340 2024-04-14T15:22:19.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.367+0000 7fa86cff9700 1 -- 172.21.15.90:0/3093729176 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fa8500022d0 con 0x7fa850004340 2024-04-14T15:22:19.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.367+0000 7fa86cff9700 10 client.37653.objecter ms_dispatch 0x7fa870001550 command_reply(tid 3: 0 ) 2024-04-14T15:22:19.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.367+0000 7fa86cff9700 10 client.37653.objecter _finish_command 3 = system:0 2024-04-14T15:22:19.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.367+0000 7fa86cff9700 20 client.37653.objecter put_session s=0x7fa850004160 osd=1 4 2024-04-14T15:22:19.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.367+0000 7fa86cff9700 15 client.37653.objecter _session_command_op_remove 1 3 2024-04-14T15:22:19.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:19 smithi092 ceph-mon[33966]: 1.2 deep-scrub starts 2024-04-14T15:22:19.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:19 smithi092 ceph-mon[33966]: 1.2 deep-scrub ok 2024-04-14T15:22:19.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:19 smithi092 ceph-mon[33966]: 110.12 scrub starts 2024-04-14T15:22:19.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:19 smithi092 ceph-mon[33966]: 110.12 scrub ok 2024-04-14T15:22:19.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:19 smithi092 ceph-mon[33966]: 11.19 deep-scrub starts 2024-04-14T15:22:19.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:19 smithi092 ceph-mon[33966]: 11.19 deep-scrub ok 2024-04-14T15:22:19.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:19 smithi092 ceph-mon[33966]: 5.1d scrub starts 2024-04-14T15:22:19.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:19 smithi092 ceph-mon[33966]: 5.1d scrub ok 2024-04-14T15:22:19.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:19 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:22:19.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:19 smithi092 ceph-mon[33966]: osdmap e603: 8 total, 8 up, 7 in 2024-04-14T15:22:19.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:19 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-14T15:22:19.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:19 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-14T15:22:19.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:19 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-14T15:22:19.373 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:19.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.384+0000 7fa875bfc700 10 client.37653.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:22:19.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.384+0000 7fa875bfc700 15 client.37653.objecter _session_command_op_assign -1 4 2024-04-14T15:22:19.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.384+0000 7fa875bfc700 20 client.37653.objecter _get_session s=0x7fa850004160 osd=1 3 2024-04-14T15:22:19.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.384+0000 7fa875bfc700 20 client.37653.objecter put_session s=0x7fa850004160 osd=1 3 2024-04-14T15:22:19.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.384+0000 7fa875bfc700 20 client.37653.objecter _get_session s=0x7fa850004160 osd=1 3 2024-04-14T15:22:19.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.384+0000 7fa875bfc700 15 client.37653.objecter _session_command_op_remove -1 4 2024-04-14T15:22:19.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.384+0000 7fa875bfc700 20 client.37653.objecter get_session s=0x7fa850004160 osd=1 3 2024-04-14T15:22:19.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.384+0000 7fa875bfc700 15 client.37653.objecter _session_command_op_assign 1 4 2024-04-14T15:22:19.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.384+0000 7fa875bfc700 20 client.37653.objecter put_session s=0x7fa850004160 osd=1 4 2024-04-14T15:22:19.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.384+0000 7fa875bfc700 10 client.37653.objecter _send_command 4 2024-04-14T15:22:19.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.384+0000 7fa875bfc700 1 -- 172.21.15.90:0/3093729176 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa850001750 con 0x7fa850004340 2024-04-14T15:22:19.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.386+0000 7fa86cff9700 1 -- 172.21.15.90:0/3093729176 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fa850001750 con 0x7fa850004340 2024-04-14T15:22:19.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.386+0000 7fa86cff9700 10 client.37653.objecter ms_dispatch 0x7fa870001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:22:19.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.386+0000 7fa86cff9700 10 client.37653.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:22:19.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.386+0000 7fa86cff9700 20 client.37653.objecter put_session s=0x7fa850004160 osd=1 4 2024-04-14T15:22:19.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.386+0000 7fa86cff9700 15 client.37653.objecter _session_command_op_remove 1 4 2024-04-14T15:22:19.387 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:22:19.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.387+0000 7fa875bfc700 10 client.37653.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:22:19.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.387+0000 7fa875bfc700 15 client.37653.objecter _session_command_op_assign -1 5 2024-04-14T15:22:19.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.387+0000 7fa875bfc700 20 client.37653.objecter _get_session s=0x7fa8500019e0 osd=2 3 2024-04-14T15:22:19.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.387+0000 7fa875bfc700 20 client.37653.objecter put_session s=0x7fa8500019e0 osd=2 3 2024-04-14T15:22:19.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.387+0000 7fa875bfc700 20 client.37653.objecter _get_session s=0x7fa8500019e0 osd=2 3 2024-04-14T15:22:19.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.387+0000 7fa875bfc700 15 client.37653.objecter _session_command_op_remove -1 5 2024-04-14T15:22:19.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.387+0000 7fa875bfc700 20 client.37653.objecter get_session s=0x7fa8500019e0 osd=2 3 2024-04-14T15:22:19.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.387+0000 7fa875bfc700 15 client.37653.objecter _session_command_op_assign 2 5 2024-04-14T15:22:19.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.387+0000 7fa875bfc700 20 client.37653.objecter put_session s=0x7fa8500019e0 osd=2 4 2024-04-14T15:22:19.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.387+0000 7fa875bfc700 10 client.37653.objecter _send_command 5 2024-04-14T15:22:19.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.387+0000 7fa875bfc700 1 -- 172.21.15.90:0/3093729176 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fa850011110 con 0x7fa85000d600 2024-04-14T15:22:19.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.387+0000 7fa86cff9700 10 client.37653.objecter ms_handle_connect 0x7fa85000d600 2024-04-14T15:22:19.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.388+0000 7fa86cff9700 1 -- 172.21.15.90:0/3093729176 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fa850011110 con 0x7fa85000d600 2024-04-14T15:22:19.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.388+0000 7fa86cff9700 10 client.37653.objecter ms_dispatch 0x7fa870001550 command_reply(tid 5: 0 ) 2024-04-14T15:22:19.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.388+0000 7fa86cff9700 10 client.37653.objecter _finish_command 5 = system:0 2024-04-14T15:22:19.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.388+0000 7fa86cff9700 20 client.37653.objecter put_session s=0x7fa8500019e0 osd=2 4 2024-04-14T15:22:19.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.388+0000 7fa86cff9700 15 client.37653.objecter _session_command_op_remove 2 5 2024-04-14T15:22:19.397 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:19.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:19.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:19.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.405+0000 7fa875bfc700 10 client.37653.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:22:19.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.405+0000 7fa875bfc700 15 client.37653.objecter _session_command_op_assign -1 6 2024-04-14T15:22:19.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.405+0000 7fa875bfc700 20 client.37653.objecter _get_session s=0x7fa8500019e0 osd=2 3 2024-04-14T15:22:19.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.405+0000 7fa875bfc700 20 client.37653.objecter put_session s=0x7fa8500019e0 osd=2 3 2024-04-14T15:22:19.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.405+0000 7fa875bfc700 20 client.37653.objecter _get_session s=0x7fa8500019e0 osd=2 3 2024-04-14T15:22:19.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.405+0000 7fa875bfc700 15 client.37653.objecter _session_command_op_remove -1 6 2024-04-14T15:22:19.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.405+0000 7fa875bfc700 20 client.37653.objecter get_session s=0x7fa8500019e0 osd=2 3 2024-04-14T15:22:19.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.405+0000 7fa875bfc700 15 client.37653.objecter _session_command_op_assign 2 6 2024-04-14T15:22:19.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.405+0000 7fa875bfc700 20 client.37653.objecter put_session s=0x7fa8500019e0 osd=2 4 2024-04-14T15:22:19.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.405+0000 7fa875bfc700 10 client.37653.objecter _send_command 6 2024-04-14T15:22:19.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.405+0000 7fa875bfc700 1 -- 172.21.15.90:0/3093729176 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa850008190 con 0x7fa85000d600 2024-04-14T15:22:19.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.407+0000 7fa86cff9700 1 -- 172.21.15.90:0/3093729176 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fa850008190 con 0x7fa85000d600 2024-04-14T15:22:19.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.407+0000 7fa86cff9700 10 client.37653.objecter ms_dispatch 0x7fa870001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:22:19.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.407+0000 7fa86cff9700 10 client.37653.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:22:19.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.407+0000 7fa86cff9700 20 client.37653.objecter put_session s=0x7fa8500019e0 osd=2 4 2024-04-14T15:22:19.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.407+0000 7fa86cff9700 15 client.37653.objecter _session_command_op_remove 2 6 2024-04-14T15:22:19.407 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:22:19.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.407+0000 7fa875bfc700 10 client.37653.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:22:19.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.407+0000 7fa875bfc700 15 client.37653.objecter _session_command_op_assign -1 7 2024-04-14T15:22:19.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.407+0000 7fa875bfc700 20 client.37653.objecter _get_session s=0x7fa850008440 osd=3 3 2024-04-14T15:22:19.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.407+0000 7fa875bfc700 20 client.37653.objecter put_session s=0x7fa850008440 osd=3 3 2024-04-14T15:22:19.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.407+0000 7fa875bfc700 20 client.37653.objecter _get_session s=0x7fa850008440 osd=3 3 2024-04-14T15:22:19.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.407+0000 7fa875bfc700 15 client.37653.objecter _session_command_op_remove -1 7 2024-04-14T15:22:19.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.407+0000 7fa875bfc700 20 client.37653.objecter get_session s=0x7fa850008440 osd=3 3 2024-04-14T15:22:19.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.407+0000 7fa875bfc700 15 client.37653.objecter _session_command_op_assign 3 7 2024-04-14T15:22:19.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.407+0000 7fa875bfc700 20 client.37653.objecter put_session s=0x7fa850008440 osd=3 4 2024-04-14T15:22:19.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.407+0000 7fa875bfc700 10 client.37653.objecter _send_command 7 2024-04-14T15:22:19.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.407+0000 7fa875bfc700 1 -- 172.21.15.90:0/3093729176 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fa85000d370 con 0x7fa85000cea0 2024-04-14T15:22:19.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.407+0000 7fa86cff9700 10 client.37653.objecter ms_handle_connect 0x7fa85000cea0 2024-04-14T15:22:19.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.408+0000 7fa86cff9700 1 -- 172.21.15.90:0/3093729176 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fa85000d370 con 0x7fa85000cea0 2024-04-14T15:22:19.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.408+0000 7fa86cff9700 10 client.37653.objecter ms_dispatch 0x7fa870001550 command_reply(tid 7: 0 ) 2024-04-14T15:22:19.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.408+0000 7fa86cff9700 10 client.37653.objecter _finish_command 7 = system:0 2024-04-14T15:22:19.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.408+0000 7fa86cff9700 20 client.37653.objecter put_session s=0x7fa850008440 osd=3 4 2024-04-14T15:22:19.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.408+0000 7fa86cff9700 15 client.37653.objecter _session_command_op_remove 3 7 2024-04-14T15:22:19.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.424+0000 7fa875bfc700 10 client.37653.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:22:19.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.425+0000 7fa875bfc700 15 client.37653.objecter _session_command_op_assign -1 8 2024-04-14T15:22:19.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.425+0000 7fa875bfc700 20 client.37653.objecter _get_session s=0x7fa850008440 osd=3 3 2024-04-14T15:22:19.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.425+0000 7fa875bfc700 20 client.37653.objecter put_session s=0x7fa850008440 osd=3 3 2024-04-14T15:22:19.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.425+0000 7fa875bfc700 20 client.37653.objecter _get_session s=0x7fa850008440 osd=3 3 2024-04-14T15:22:19.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.425+0000 7fa875bfc700 15 client.37653.objecter _session_command_op_remove -1 8 2024-04-14T15:22:19.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.425+0000 7fa875bfc700 20 client.37653.objecter get_session s=0x7fa850008440 osd=3 3 2024-04-14T15:22:19.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.425+0000 7fa875bfc700 15 client.37653.objecter _session_command_op_assign 3 8 2024-04-14T15:22:19.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.425+0000 7fa875bfc700 20 client.37653.objecter put_session s=0x7fa850008440 osd=3 4 2024-04-14T15:22:19.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.425+0000 7fa875bfc700 10 client.37653.objecter _send_command 8 2024-04-14T15:22:19.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.425+0000 7fa875bfc700 1 -- 172.21.15.90:0/3093729176 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa85000c9e0 con 0x7fa85000cea0 2024-04-14T15:22:19.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.426+0000 7fa86cff9700 1 -- 172.21.15.90:0/3093729176 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fa85000c9e0 con 0x7fa85000cea0 2024-04-14T15:22:19.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.426+0000 7fa86cff9700 10 client.37653.objecter ms_dispatch 0x7fa870001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:22:19.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.426+0000 7fa86cff9700 10 client.37653.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:22:19.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.426+0000 7fa86cff9700 20 client.37653.objecter put_session s=0x7fa850008440 osd=3 4 2024-04-14T15:22:19.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.426+0000 7fa86cff9700 15 client.37653.objecter _session_command_op_remove 3 8 2024-04-14T15:22:19.427 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:22:19.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.427+0000 7fa875bfc700 10 client.37653.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:22:19.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.427+0000 7fa875bfc700 15 client.37653.objecter _session_command_op_assign -1 9 2024-04-14T15:22:19.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.427+0000 7fa875bfc700 20 client.37653.objecter _get_session s=0x7fa85000c790 osd=4 3 2024-04-14T15:22:19.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.427+0000 7fa875bfc700 20 client.37653.objecter put_session s=0x7fa85000c790 osd=4 3 2024-04-14T15:22:19.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.427+0000 7fa875bfc700 20 client.37653.objecter _get_session s=0x7fa85000c790 osd=4 3 2024-04-14T15:22:19.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.427+0000 7fa875bfc700 15 client.37653.objecter _session_command_op_remove -1 9 2024-04-14T15:22:19.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.427+0000 7fa875bfc700 20 client.37653.objecter get_session s=0x7fa85000c790 osd=4 3 2024-04-14T15:22:19.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.427+0000 7fa875bfc700 15 client.37653.objecter _session_command_op_assign 4 9 2024-04-14T15:22:19.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.427+0000 7fa875bfc700 20 client.37653.objecter put_session s=0x7fa85000c790 osd=4 4 2024-04-14T15:22:19.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.427+0000 7fa875bfc700 10 client.37653.objecter _send_command 9 2024-04-14T15:22:19.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.427+0000 7fa875bfc700 1 -- 172.21.15.90:0/3093729176 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fa85001aa70 con 0x7fa850011fa0 2024-04-14T15:22:19.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.427+0000 7fa86cff9700 10 client.37653.objecter ms_handle_connect 0x7fa850011fa0 2024-04-14T15:22:19.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.428+0000 7fa86cff9700 1 -- 172.21.15.90:0/3093729176 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fa85001aa70 con 0x7fa850011fa0 2024-04-14T15:22:19.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.428+0000 7fa86cff9700 10 client.37653.objecter ms_dispatch 0x7fa870001550 command_reply(tid 9: 0 ) 2024-04-14T15:22:19.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.428+0000 7fa86cff9700 10 client.37653.objecter _finish_command 9 = system:0 2024-04-14T15:22:19.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.428+0000 7fa86cff9700 20 client.37653.objecter put_session s=0x7fa85000c790 osd=4 4 2024-04-14T15:22:19.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.428+0000 7fa86cff9700 15 client.37653.objecter _session_command_op_remove 4 9 2024-04-14T15:22:19.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.444+0000 7fa875bfc700 10 client.37653.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:22:19.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.444+0000 7fa875bfc700 15 client.37653.objecter _session_command_op_assign -1 10 2024-04-14T15:22:19.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.444+0000 7fa875bfc700 20 client.37653.objecter _get_session s=0x7fa85000c790 osd=4 3 2024-04-14T15:22:19.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.444+0000 7fa875bfc700 20 client.37653.objecter put_session s=0x7fa85000c790 osd=4 3 2024-04-14T15:22:19.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.444+0000 7fa875bfc700 20 client.37653.objecter _get_session s=0x7fa85000c790 osd=4 3 2024-04-14T15:22:19.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.444+0000 7fa875bfc700 15 client.37653.objecter _session_command_op_remove -1 10 2024-04-14T15:22:19.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.444+0000 7fa875bfc700 20 client.37653.objecter get_session s=0x7fa85000c790 osd=4 3 2024-04-14T15:22:19.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.444+0000 7fa875bfc700 15 client.37653.objecter _session_command_op_assign 4 10 2024-04-14T15:22:19.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.444+0000 7fa875bfc700 20 client.37653.objecter put_session s=0x7fa85000c790 osd=4 4 2024-04-14T15:22:19.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.444+0000 7fa875bfc700 10 client.37653.objecter _send_command 10 2024-04-14T15:22:19.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.444+0000 7fa875bfc700 1 -- 172.21.15.90:0/3093729176 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa85001aa70 con 0x7fa850011fa0 2024-04-14T15:22:19.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.446+0000 7fa86cff9700 1 -- 172.21.15.90:0/3093729176 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fa85001aa70 con 0x7fa850011fa0 2024-04-14T15:22:19.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.446+0000 7fa86cff9700 10 client.37653.objecter ms_dispatch 0x7fa870001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:22:19.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.446+0000 7fa86cff9700 10 client.37653.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:22:19.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.446+0000 7fa86cff9700 20 client.37653.objecter put_session s=0x7fa85000c790 osd=4 4 2024-04-14T15:22:19.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.446+0000 7fa86cff9700 15 client.37653.objecter _session_command_op_remove 4 10 2024-04-14T15:22:19.447 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:22:19.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.446+0000 7fa875bfc700 10 client.37653.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:22:19.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.446+0000 7fa875bfc700 15 client.37653.objecter _session_command_op_assign -1 11 2024-04-14T15:22:19.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.446+0000 7fa875bfc700 20 client.37653.objecter _get_session s=0x7fa85001aa70 osd=5 3 2024-04-14T15:22:19.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.446+0000 7fa875bfc700 20 client.37653.objecter put_session s=0x7fa85001aa70 osd=5 3 2024-04-14T15:22:19.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.446+0000 7fa875bfc700 20 client.37653.objecter _get_session s=0x7fa85001aa70 osd=5 3 2024-04-14T15:22:19.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.446+0000 7fa875bfc700 15 client.37653.objecter _session_command_op_remove -1 11 2024-04-14T15:22:19.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.446+0000 7fa875bfc700 20 client.37653.objecter get_session s=0x7fa85001aa70 osd=5 3 2024-04-14T15:22:19.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.446+0000 7fa875bfc700 15 client.37653.objecter _session_command_op_assign 5 11 2024-04-14T15:22:19.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.446+0000 7fa875bfc700 20 client.37653.objecter put_session s=0x7fa85001aa70 osd=5 4 2024-04-14T15:22:19.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.446+0000 7fa875bfc700 10 client.37653.objecter _send_command 11 2024-04-14T15:22:19.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.446+0000 7fa875bfc700 1 -- 172.21.15.90:0/3093729176 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fa85001ee10 con 0x7fa85001b0f0 2024-04-14T15:22:19.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.447+0000 7fa86cff9700 10 client.37653.objecter ms_handle_connect 0x7fa85001b0f0 2024-04-14T15:22:19.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.447+0000 7fa86cff9700 1 -- 172.21.15.90:0/3093729176 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fa85001ee10 con 0x7fa85001b0f0 2024-04-14T15:22:19.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.447+0000 7fa86cff9700 10 client.37653.objecter ms_dispatch 0x7fa870001550 command_reply(tid 11: 0 ) 2024-04-14T15:22:19.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.447+0000 7fa86cff9700 10 client.37653.objecter _finish_command 11 = system:0 2024-04-14T15:22:19.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.447+0000 7fa86cff9700 20 client.37653.objecter put_session s=0x7fa85001aa70 osd=5 4 2024-04-14T15:22:19.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.447+0000 7fa86cff9700 15 client.37653.objecter _session_command_op_remove 5 11 2024-04-14T15:22:19.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.464+0000 7fa875bfc700 10 client.37653.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:22:19.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.464+0000 7fa875bfc700 15 client.37653.objecter _session_command_op_assign -1 12 2024-04-14T15:22:19.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.464+0000 7fa875bfc700 20 client.37653.objecter _get_session s=0x7fa85001aa70 osd=5 3 2024-04-14T15:22:19.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.464+0000 7fa875bfc700 20 client.37653.objecter put_session s=0x7fa85001aa70 osd=5 3 2024-04-14T15:22:19.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.464+0000 7fa875bfc700 20 client.37653.objecter _get_session s=0x7fa85001aa70 osd=5 3 2024-04-14T15:22:19.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.464+0000 7fa875bfc700 15 client.37653.objecter _session_command_op_remove -1 12 2024-04-14T15:22:19.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.464+0000 7fa875bfc700 20 client.37653.objecter get_session s=0x7fa85001aa70 osd=5 3 2024-04-14T15:22:19.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.464+0000 7fa875bfc700 15 client.37653.objecter _session_command_op_assign 5 12 2024-04-14T15:22:19.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.464+0000 7fa875bfc700 20 client.37653.objecter put_session s=0x7fa85001aa70 osd=5 4 2024-04-14T15:22:19.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.464+0000 7fa875bfc700 10 client.37653.objecter _send_command 12 2024-04-14T15:22:19.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.464+0000 7fa875bfc700 1 -- 172.21.15.90:0/3093729176 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa850011510 con 0x7fa85001b0f0 2024-04-14T15:22:19.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.465+0000 7fa86cff9700 1 -- 172.21.15.90:0/3093729176 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fa850011510 con 0x7fa85001b0f0 2024-04-14T15:22:19.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.465+0000 7fa86cff9700 10 client.37653.objecter ms_dispatch 0x7fa870001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:22:19.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.465+0000 7fa86cff9700 10 client.37653.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:22:19.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.465+0000 7fa86cff9700 20 client.37653.objecter put_session s=0x7fa85001aa70 osd=5 4 2024-04-14T15:22:19.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.465+0000 7fa86cff9700 15 client.37653.objecter _session_command_op_remove 5 12 2024-04-14T15:22:19.466 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:22:19.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.466+0000 7fa875bfc700 10 client.37653.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:22:19.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.466+0000 7fa875bfc700 15 client.37653.objecter _session_command_op_assign -1 13 2024-04-14T15:22:19.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.466+0000 7fa875bfc700 20 client.37653.objecter _get_session s=0x7fa8500112c0 osd=6 3 2024-04-14T15:22:19.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.466+0000 7fa875bfc700 20 client.37653.objecter put_session s=0x7fa8500112c0 osd=6 3 2024-04-14T15:22:19.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.466+0000 7fa875bfc700 20 client.37653.objecter _get_session s=0x7fa8500112c0 osd=6 3 2024-04-14T15:22:19.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.466+0000 7fa875bfc700 15 client.37653.objecter _session_command_op_remove -1 13 2024-04-14T15:22:19.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.466+0000 7fa875bfc700 20 client.37653.objecter get_session s=0x7fa8500112c0 osd=6 3 2024-04-14T15:22:19.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.466+0000 7fa875bfc700 15 client.37653.objecter _session_command_op_assign 6 13 2024-04-14T15:22:19.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.466+0000 7fa875bfc700 20 client.37653.objecter put_session s=0x7fa8500112c0 osd=6 4 2024-04-14T15:22:19.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.466+0000 7fa875bfc700 10 client.37653.objecter _send_command 13 2024-04-14T15:22:19.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.466+0000 7fa875bfc700 1 -- 172.21.15.90:0/3093729176 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fa850011430 con 0x7fa850016990 2024-04-14T15:22:19.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.466+0000 7fa86cff9700 10 client.37653.objecter ms_handle_connect 0x7fa850016990 2024-04-14T15:22:19.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.467+0000 7fa86cff9700 1 -- 172.21.15.90:0/3093729176 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fa850011430 con 0x7fa850016990 2024-04-14T15:22:19.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.467+0000 7fa86cff9700 10 client.37653.objecter ms_dispatch 0x7fa870001550 command_reply(tid 13: 0 ) 2024-04-14T15:22:19.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.467+0000 7fa86cff9700 10 client.37653.objecter _finish_command 13 = system:0 2024-04-14T15:22:19.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.467+0000 7fa86cff9700 20 client.37653.objecter put_session s=0x7fa8500112c0 osd=6 4 2024-04-14T15:22:19.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.467+0000 7fa86cff9700 15 client.37653.objecter _session_command_op_remove 6 13 2024-04-14T15:22:19.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.485+0000 7fa875bfc700 10 client.37653.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:22:19.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.485+0000 7fa875bfc700 15 client.37653.objecter _session_command_op_assign -1 14 2024-04-14T15:22:19.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.485+0000 7fa875bfc700 20 client.37653.objecter _get_session s=0x7fa8500112c0 osd=6 3 2024-04-14T15:22:19.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.485+0000 7fa875bfc700 20 client.37653.objecter put_session s=0x7fa8500112c0 osd=6 3 2024-04-14T15:22:19.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.485+0000 7fa875bfc700 20 client.37653.objecter _get_session s=0x7fa8500112c0 osd=6 3 2024-04-14T15:22:19.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.485+0000 7fa875bfc700 15 client.37653.objecter _session_command_op_remove -1 14 2024-04-14T15:22:19.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.485+0000 7fa875bfc700 20 client.37653.objecter get_session s=0x7fa8500112c0 osd=6 3 2024-04-14T15:22:19.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.485+0000 7fa875bfc700 15 client.37653.objecter _session_command_op_assign 6 14 2024-04-14T15:22:19.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.485+0000 7fa875bfc700 20 client.37653.objecter put_session s=0x7fa8500112c0 osd=6 4 2024-04-14T15:22:19.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.485+0000 7fa875bfc700 10 client.37653.objecter _send_command 14 2024-04-14T15:22:19.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.485+0000 7fa875bfc700 1 -- 172.21.15.90:0/3093729176 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa850011430 con 0x7fa850016990 2024-04-14T15:22:19.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.487+0000 7fa86cff9700 1 -- 172.21.15.90:0/3093729176 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fa850011430 con 0x7fa850016990 2024-04-14T15:22:19.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.487+0000 7fa86cff9700 10 client.37653.objecter ms_dispatch 0x7fa870001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:22:19.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.487+0000 7fa86cff9700 10 client.37653.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:22:19.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.487+0000 7fa86cff9700 20 client.37653.objecter put_session s=0x7fa8500112c0 osd=6 4 2024-04-14T15:22:19.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.487+0000 7fa86cff9700 15 client.37653.objecter _session_command_op_remove 6 14 2024-04-14T15:22:19.488 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:22:19.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.487+0000 7fa875bfc700 10 client.37653.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:22:19.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.487+0000 7fa875bfc700 15 client.37653.objecter _session_command_op_assign -1 15 2024-04-14T15:22:19.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.487+0000 7fa875bfc700 20 client.37653.objecter _get_session s=0x7fa850016e10 osd=7 3 2024-04-14T15:22:19.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.487+0000 7fa875bfc700 20 client.37653.objecter put_session s=0x7fa850016e10 osd=7 3 2024-04-14T15:22:19.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.487+0000 7fa875bfc700 20 client.37653.objecter _get_session s=0x7fa850016e10 osd=7 3 2024-04-14T15:22:19.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.487+0000 7fa875bfc700 15 client.37653.objecter _session_command_op_remove -1 15 2024-04-14T15:22:19.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.487+0000 7fa875bfc700 20 client.37653.objecter get_session s=0x7fa850016e10 osd=7 3 2024-04-14T15:22:19.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.487+0000 7fa875bfc700 15 client.37653.objecter _session_command_op_assign 7 15 2024-04-14T15:22:19.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.487+0000 7fa875bfc700 20 client.37653.objecter put_session s=0x7fa850016e10 osd=7 4 2024-04-14T15:22:19.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.487+0000 7fa875bfc700 10 client.37653.objecter _send_command 15 2024-04-14T15:22:19.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.487+0000 7fa875bfc700 1 -- 172.21.15.90:0/3093729176 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fa8500170c0 con 0x7fa850023ef0 2024-04-14T15:22:19.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.488+0000 7fa86cff9700 10 client.37653.objecter ms_handle_connect 0x7fa850023ef0 2024-04-14T15:22:19.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.489+0000 7fa86cff9700 1 -- 172.21.15.90:0/3093729176 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fa8500170c0 con 0x7fa850023ef0 2024-04-14T15:22:19.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.489+0000 7fa86cff9700 10 client.37653.objecter ms_dispatch 0x7fa870001550 command_reply(tid 15: 0 ) 2024-04-14T15:22:19.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.489+0000 7fa86cff9700 10 client.37653.objecter _finish_command 15 = system:0 2024-04-14T15:22:19.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.489+0000 7fa86cff9700 20 client.37653.objecter put_session s=0x7fa850016e10 osd=7 4 2024-04-14T15:22:19.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.489+0000 7fa86cff9700 15 client.37653.objecter _session_command_op_remove 7 15 2024-04-14T15:22:19.499 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:22:19.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.505+0000 7fa875bfc700 10 client.37653.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:22:19.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.505+0000 7fa875bfc700 15 client.37653.objecter _session_command_op_assign -1 16 2024-04-14T15:22:19.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.505+0000 7fa875bfc700 20 client.37653.objecter _get_session s=0x7fa850016e10 osd=7 3 2024-04-14T15:22:19.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.505+0000 7fa875bfc700 20 client.37653.objecter put_session s=0x7fa850016e10 osd=7 3 2024-04-14T15:22:19.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.505+0000 7fa875bfc700 20 client.37653.objecter _get_session s=0x7fa850016e10 osd=7 3 2024-04-14T15:22:19.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.505+0000 7fa875bfc700 15 client.37653.objecter _session_command_op_remove -1 16 2024-04-14T15:22:19.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.505+0000 7fa875bfc700 20 client.37653.objecter get_session s=0x7fa850016e10 osd=7 3 2024-04-14T15:22:19.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.505+0000 7fa875bfc700 15 client.37653.objecter _session_command_op_assign 7 16 2024-04-14T15:22:19.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.505+0000 7fa875bfc700 20 client.37653.objecter put_session s=0x7fa850016e10 osd=7 4 2024-04-14T15:22:19.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.505+0000 7fa875bfc700 10 client.37653.objecter _send_command 16 2024-04-14T15:22:19.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.505+0000 7fa875bfc700 1 -- 172.21.15.90:0/3093729176 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa850027a00 con 0x7fa850023ef0 2024-04-14T15:22:19.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.507+0000 7fa86cff9700 1 -- 172.21.15.90:0/3093729176 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fa850027a00 con 0x7fa850023ef0 2024-04-14T15:22:19.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.507+0000 7fa86cff9700 10 client.37653.objecter ms_dispatch 0x7fa870001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:22:19.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.507+0000 7fa86cff9700 10 client.37653.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:22:19.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.507+0000 7fa86cff9700 20 client.37653.objecter put_session s=0x7fa850016e10 osd=7 4 2024-04-14T15:22:19.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.507+0000 7fa86cff9700 15 client.37653.objecter _session_command_op_remove 7 16 2024-04-14T15:22:19.508 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:22:19.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.507+0000 7fa875bfc700 10 librados: watch_flush enter 2024-04-14T15:22:19.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.507+0000 7fa875bfc700 10 librados: watch_flush exit 2024-04-14T15:22:19.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.507+0000 7fa875bfc700 10 client.37653.objecter close_session for osd.0 2024-04-14T15:22:19.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.507+0000 7fa875bfc700 1 -- 172.21.15.90:0/3093729176 >> v1:172.21.15.90:6801/3761806885 conn(0x7fa850004760 legacy=0x7fa850002780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:19.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.507+0000 7fa875bfc700 20 client.37653.objecter put_session s=0x7fa850005410 osd=0 1 2024-04-14T15:22:19.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.507+0000 7fa875bfc700 10 client.37653.objecter close_session for osd.1 2024-04-14T15:22:19.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.507+0000 7fa875bfc700 1 -- 172.21.15.90:0/3093729176 >> v1:172.21.15.90:6805/4170361699 conn(0x7fa850004340 legacy=0x7fa85000ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:19.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.507+0000 7fa875bfc700 20 client.37653.objecter put_session s=0x7fa850004160 osd=1 1 2024-04-14T15:22:19.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.507+0000 7fa875bfc700 10 client.37653.objecter close_session for osd.2 2024-04-14T15:22:19.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.507+0000 7fa875bfc700 1 -- 172.21.15.90:0/3093729176 >> v1:172.21.15.90:6809/1686184251 conn(0x7fa85000d600 legacy=0x7fa85000f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:19.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.507+0000 7fa875bfc700 20 client.37653.objecter put_session s=0x7fa8500019e0 osd=2 1 2024-04-14T15:22:19.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.508+0000 7fa875bfc700 10 client.37653.objecter close_session for osd.3 2024-04-14T15:22:19.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.508+0000 7fa875bfc700 1 -- 172.21.15.90:0/3093729176 >> v1:172.21.15.90:6813/2270166774 conn(0x7fa85000cea0 legacy=0x7fa850014aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:19.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.508+0000 7fa86ffff700 1 -- 172.21.15.90:0/3093729176 reap_dead start 2024-04-14T15:22:19.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.508+0000 7fa875bfc700 20 client.37653.objecter put_session s=0x7fa850008440 osd=3 1 2024-04-14T15:22:19.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.508+0000 7fa875bfc700 10 client.37653.objecter close_session for osd.4 2024-04-14T15:22:19.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.508+0000 7fa875bfc700 1 -- 172.21.15.90:0/3093729176 >> v1:172.21.15.92:6800/3887549733 conn(0x7fa850011fa0 legacy=0x7fa850019350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:19.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.508+0000 7fa875bfc700 20 client.37653.objecter put_session s=0x7fa85000c790 osd=4 1 2024-04-14T15:22:19.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.508+0000 7fa875bfc700 10 client.37653.objecter close_session for osd.5 2024-04-14T15:22:19.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.508+0000 7fa875bfc700 1 -- 172.21.15.90:0/3093729176 >> v1:172.21.15.92:6804/3244561440 conn(0x7fa85001b0f0 legacy=0x7fa85001d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:19.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.508+0000 7fa875bfc700 20 client.37653.objecter put_session s=0x7fa85001aa70 osd=5 1 2024-04-14T15:22:19.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.508+0000 7fa875bfc700 10 client.37653.objecter close_session for osd.6 2024-04-14T15:22:19.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.508+0000 7fa875bfc700 1 -- 172.21.15.90:0/3093729176 >> v1:172.21.15.92:6808/636125963 conn(0x7fa850016990 legacy=0x7fa8500227d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:19.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.508+0000 7fa875bfc700 20 client.37653.objecter put_session s=0x7fa8500112c0 osd=6 1 2024-04-14T15:22:19.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.508+0000 7fa875bfc700 10 client.37653.objecter close_session for osd.7 2024-04-14T15:22:19.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.508+0000 7fa875bfc700 1 -- 172.21.15.90:0/3093729176 >> v1:172.21.15.92:6812/1773706716 conn(0x7fa850023ef0 legacy=0x7fa8500262e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:19.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.508+0000 7fa875bfc700 20 client.37653.objecter put_session s=0x7fa850016e10 osd=7 1 2024-04-14T15:22:19.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.508+0000 7fa875bfc700 20 client.37653.objecter shutdown clearing up homeless session... 2024-04-14T15:22:19.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.508+0000 7fa875bfc700 10 client.37653.objecter successfully canceled tick 2024-04-14T15:22:19.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.508+0000 7fa86ffff700 1 -- 172.21.15.90:0/3093729176 reap_dead start 2024-04-14T15:22:19.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.508+0000 7fa875bfc700 1 -- 172.21.15.90:0/3093729176 >> v1:172.21.15.90:6800/420935427 conn(0x7fa85c079d10 legacy=0x7fa85c07c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:19.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.508+0000 7fa875bfc700 1 -- 172.21.15.90:0/3093729176 >> v1:172.21.15.90:6789/0 conn(0x7fa87010b890 legacy=0x7fa8701a4b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:19.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.508+0000 7fa875bfc700 1 -- 172.21.15.90:0/3093729176 shutdown_connections 2024-04-14T15:22:19.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.508+0000 7fa875bfc700 1 -- 172.21.15.90:0/3093729176 >> 172.21.15.90:0/3093729176 conn(0x7fa8700fd290 msgr2=0x7fa8701117b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:22:19.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.508+0000 7fa875bfc700 1 -- 172.21.15.90:0/3093729176 shutdown_connections 2024-04-14T15:22:19.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.508+0000 7fa875bfc700 1 -- 172.21.15.90:0/3093729176 wait complete. 2024-04-14T15:22:19.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.508+0000 7fa875bfc700 1 librados: shutdown 2024-04-14T15:22:19.512 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:22:19.512 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:22:19.512 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:22:19.512 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:22:19.512 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:22:19.512 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:22:19.512 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:22:19.512 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:22:19.523 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:19.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:19.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:22:19.625 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:22:19.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:19.649 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:19.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:22:19.750 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:22:19.773 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:19.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:19.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:22:19.874 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:22:19.902 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:19.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:19.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:22:20.003 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:22:20.027 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:20.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:20.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:22:20.129 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:22:20.152 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:20.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:20.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:22:20.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:20 smithi090 ceph-mon[30399]: 11.15 scrub starts 2024-04-14T15:22:20.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:20 smithi090 ceph-mon[30399]: 11.15 scrub ok 2024-04-14T15:22:20.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:20 smithi090 ceph-mon[30399]: pgmap v885: 242 pgs: 23 unknown, 4 active+remapped+backfill_toofull, 215 active+clean; 8.4 MiB data, 517 MiB used, 625 GiB / 626 GiB avail; 2/1206 objects misplaced (0.166%) 2024-04-14T15:22:20.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:20 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3093729176' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:22:20.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:20 smithi090 ceph-mon[30399]: 11.c scrub starts 2024-04-14T15:22:20.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:20 smithi090 ceph-mon[30399]: 11.c scrub ok 2024-04-14T15:22:20.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:20 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-102","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-14T15:22:20.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:20 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-14T15:22:20.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:20 smithi090 ceph-mon[30399]: osdmap e604: 8 total, 8 up, 7 in 2024-04-14T15:22:20.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:20 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-14T15:22:20.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:20 smithi090 ceph-mon[41670]: 11.15 scrub starts 2024-04-14T15:22:20.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:20 smithi090 ceph-mon[41670]: 11.15 scrub ok 2024-04-14T15:22:20.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:20 smithi090 ceph-mon[41670]: pgmap v885: 242 pgs: 23 unknown, 4 active+remapped+backfill_toofull, 215 active+clean; 8.4 MiB data, 517 MiB used, 625 GiB / 626 GiB avail; 2/1206 objects misplaced (0.166%) 2024-04-14T15:22:20.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:20 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3093729176' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:22:20.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:20 smithi090 ceph-mon[41670]: 11.c scrub starts 2024-04-14T15:22:20.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:20 smithi090 ceph-mon[41670]: 11.c scrub ok 2024-04-14T15:22:20.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:20 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-102","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-14T15:22:20.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:20 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-14T15:22:20.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:20 smithi090 ceph-mon[41670]: osdmap e604: 8 total, 8 up, 7 in 2024-04-14T15:22:20.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:20 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-14T15:22:20.253 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:22:20.282 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:20.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:20.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:22:20.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:20 smithi092 ceph-mon[33966]: 11.15 scrub starts 2024-04-14T15:22:20.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:20 smithi092 ceph-mon[33966]: 11.15 scrub ok 2024-04-14T15:22:20.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:20 smithi092 ceph-mon[33966]: pgmap v885: 242 pgs: 23 unknown, 4 active+remapped+backfill_toofull, 215 active+clean; 8.4 MiB data, 517 MiB used, 625 GiB / 626 GiB avail; 2/1206 objects misplaced (0.166%) 2024-04-14T15:22:20.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:20 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3093729176' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:22:20.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:20 smithi092 ceph-mon[33966]: 11.c scrub starts 2024-04-14T15:22:20.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:20 smithi092 ceph-mon[33966]: 11.c scrub ok 2024-04-14T15:22:20.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:20 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-102","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-14T15:22:20.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:20 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-14T15:22:20.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:20 smithi092 ceph-mon[33966]: osdmap e604: 8 total, 8 up, 7 in 2024-04-14T15:22:20.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:20 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-14T15:22:20.383 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:22:20.410 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:20.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:20.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:22:20.511 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:20.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.902+0000 7f10680d3700 1 -- 172.21.15.90:0/2776353697 >> v1:172.21.15.90:6789/0 conn(0x7f1060108980 legacy=0x7f1060108d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:20.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.902+0000 7f10680d3700 1 -- 172.21.15.90:0/2776353697 shutdown_connections 2024-04-14T15:22:20.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.902+0000 7f10680d3700 1 -- 172.21.15.90:0/2776353697 >> 172.21.15.90:0/2776353697 conn(0x7f1060076aa0 msgr2=0x7f1060076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:22:20.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.902+0000 7f10680d3700 1 -- 172.21.15.90:0/2776353697 shutdown_connections 2024-04-14T15:22:20.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.902+0000 7f10680d3700 1 -- 172.21.15.90:0/2776353697 wait complete. 2024-04-14T15:22:20.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.902+0000 7f10680d3700 1 librados: starting msgr at 2024-04-14T15:22:20.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.902+0000 7f10680d3700 1 librados: starting objecter 2024-04-14T15:22:20.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.902+0000 7f10680d3700 1 Processor -- start 2024-04-14T15:22:20.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.903+0000 7f10680d3700 1 -- start start 2024-04-14T15:22:20.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.903+0000 7f10680d3700 1 librados: setting wanted keys 2024-04-14T15:22:20.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.903+0000 7f10680d3700 1 librados: calling monclient init 2024-04-14T15:22:20.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.903+0000 7f10680d3700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f10601a7790 con 0x7f106010b6f0 2024-04-14T15:22:20.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.903+0000 7f10680d3700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f10601a89b0 con 0x7f106010f2f0 2024-04-14T15:22:20.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.903+0000 7f10680d3700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f10601a9bf0 con 0x7f1060108980 2024-04-14T15:22:20.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.903+0000 7f106566e700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f106010b6f0 0x7f10601a27a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:42042/0 (socket says 172.21.15.90:42042) 2024-04-14T15:22:20.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.903+0000 7f106566e700 1 -- 172.21.15.90:0/3038639602 learned_addr learned my addr 172.21.15.90:0/3038639602 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:22:20.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.903+0000 7f1056ffd700 10 client.?.objecter ms_handle_connect 0x7f106010b6f0 2024-04-14T15:22:20.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.903+0000 7f1056ffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:22:20.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.903+0000 7f1056ffd700 10 client.?.objecter ms_handle_connect 0x7f1060108980 2024-04-14T15:22:20.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.903+0000 7f1056ffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:22:20.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.903+0000 7f1056ffd700 10 client.?.objecter ms_handle_connect 0x7f106010f2f0 2024-04-14T15:22:20.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.903+0000 7f1056ffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:22:20.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.903+0000 7f1056ffd700 1 -- 172.21.15.90:0/3038639602 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3128704253 0 0) 0x7f10601a7790 con 0x7f106010b6f0 2024-04-14T15:22:20.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.903+0000 7f1056ffd700 1 -- 172.21.15.90:0/3038639602 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f104c0031f0 con 0x7f106010b6f0 2024-04-14T15:22:20.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.903+0000 7f1056ffd700 1 -- 172.21.15.90:0/3038639602 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 745276123 0 0) 0x7f10601a9bf0 con 0x7f1060108980 2024-04-14T15:22:20.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.903+0000 7f1056ffd700 1 -- 172.21.15.90:0/3038639602 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f10601a7790 con 0x7f1060108980 2024-04-14T15:22:20.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.903+0000 7f1056ffd700 1 -- 172.21.15.90:0/3038639602 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2537462047 0 0) 0x7f10601a89b0 con 0x7f106010f2f0 2024-04-14T15:22:20.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.903+0000 7f1056ffd700 1 -- 172.21.15.90:0/3038639602 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f10601a9bf0 con 0x7f106010f2f0 2024-04-14T15:22:20.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.903+0000 7f1056ffd700 1 -- 172.21.15.90:0/3038639602 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 415201105 0 0) 0x7f104c0031f0 con 0x7f106010b6f0 2024-04-14T15:22:20.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.903+0000 7f1056ffd700 1 -- 172.21.15.90:0/3038639602 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f10601a89b0 con 0x7f106010b6f0 2024-04-14T15:22:20.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.903+0000 7f1056ffd700 1 -- 172.21.15.90:0/3038639602 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2855584864 0 0) 0x7f10601a7790 con 0x7f1060108980 2024-04-14T15:22:20.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.903+0000 7f1056ffd700 1 -- 172.21.15.90:0/3038639602 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f104c0031f0 con 0x7f1060108980 2024-04-14T15:22:20.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.903+0000 7f1056ffd700 1 -- 172.21.15.90:0/3038639602 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f1050001910 con 0x7f106010b6f0 2024-04-14T15:22:20.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.903+0000 7f1056ffd700 1 -- 172.21.15.90:0/3038639602 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f105c0044b0 con 0x7f1060108980 2024-04-14T15:22:20.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.903+0000 7f1056ffd700 1 -- 172.21.15.90:0/3038639602 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1929554266 0 0) 0x7f10601a9bf0 con 0x7f106010f2f0 2024-04-14T15:22:20.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.903+0000 7f1056ffd700 1 -- 172.21.15.90:0/3038639602 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f10601a7790 con 0x7f106010f2f0 2024-04-14T15:22:20.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.903+0000 7f1056ffd700 1 -- 172.21.15.90:0/3038639602 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f1058001550 con 0x7f106010f2f0 2024-04-14T15:22:20.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.904+0000 7f1056ffd700 1 -- 172.21.15.90:0/3038639602 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2873756674 0 0) 0x7f10601a89b0 con 0x7f106010b6f0 2024-04-14T15:22:20.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.904+0000 7f1056ffd700 1 -- 172.21.15.90:0/3038639602 >> v1:172.21.15.90:6790/0 conn(0x7f1060108980 legacy=0x7f1060107b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:20.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.904+0000 7f1056ffd700 1 -- 172.21.15.90:0/3038639602 >> v1:172.21.15.92:6789/0 conn(0x7f106010f2f0 legacy=0x7f10601a5ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:20.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.904+0000 7f1056ffd700 1 -- 172.21.15.90:0/3038639602 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10601aae30 con 0x7f106010b6f0 2024-04-14T15:22:20.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.904+0000 7f10680d3700 1 -- 172.21.15.90:0/3038639602 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f10601a9e20 con 0x7f106010b6f0 2024-04-14T15:22:20.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.904+0000 7f10680d3700 10 client.37659.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:22:20.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.904+0000 7f10680d3700 1 -- 172.21.15.90:0/3038639602 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f10601aa2e0 con 0x7f106010b6f0 2024-04-14T15:22:20.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.904+0000 7f10680d3700 1 librados: init done 2024-04-14T15:22:20.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.904+0000 7f1056ffd700 1 -- 172.21.15.90:0/3038639602 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f1050001c90 con 0x7f106010b6f0 2024-04-14T15:22:20.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.904+0000 7f1056ffd700 1 -- 172.21.15.90:0/3038639602 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f1050005d10 con 0x7f106010b6f0 2024-04-14T15:22:20.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.904+0000 7f10680d3700 1 -- 172.21.15.90:0/3038639602 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1044005120 con 0x7f106010b6f0 2024-04-14T15:22:20.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.905+0000 7f1056ffd700 1 -- 172.21.15.90:0/3038639602 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f1050003950 con 0x7f106010b6f0 2024-04-14T15:22:20.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.906+0000 7f1056ffd700 1 -- 172.21.15.90:0/3038639602 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(603..603 src has 467..603) ==== 8301+0+0 (unknown 1748645378 0 0) 0x7f105008aa50 con 0x7f106010b6f0 2024-04-14T15:22:20.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.906+0000 7f1056ffd700 10 client.37659.objecter ms_dispatch 0x7f1060001550 osd_map(603..603 src has 467..603) 2024-04-14T15:22:20.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.906+0000 7f1056ffd700 3 client.37659.objecter handle_osd_map got epochs [603,603] > 0 2024-04-14T15:22:20.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.906+0000 7f1056ffd700 3 client.37659.objecter handle_osd_map decoding full epoch 603 2024-04-14T15:22:20.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.906+0000 7f1056ffd700 20 client.37659.objecter dump_active .. 0 homeless 2024-04-14T15:22:20.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.906+0000 7f1056ffd700 10 client.37659.objecter ms_handle_connect 0x7f104c071410 2024-04-14T15:22:20.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:19.910+0000 7f1056ffd700 1 -- 172.21.15.90:0/3038639602 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f10500573f0 con 0x7f106010b6f0 2024-04-14T15:22:20.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:20.135+0000 7f10680d3700 1 -- 172.21.15.90:0/3038639602 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7f1044005410 con 0x7f106010b6f0 2024-04-14T15:22:20.536 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:20.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:20.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:20.638 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:22:20.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:20.662 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:20.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:22:20.763 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:22:20.791 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:20.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:20.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:22:20.893 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:22:20.902 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:22:20.920 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:20.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:20.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:22:20.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:20.959+0000 7f1056ffd700 1 -- 172.21.15.90:0/3038639602 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v605) ==== 87+0+0 (unknown 4170266258 0 0) 0x7f105005b0a0 con 0x7f106010b6f0 2024-04-14T15:22:20.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:20.961+0000 7f10680d3700 10 librados: watch_flush enter 2024-04-14T15:22:20.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:20.962+0000 7f10680d3700 10 librados: watch_flush exit 2024-04-14T15:22:20.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:20.962+0000 7f10680d3700 20 client.37659.objecter shutdown clearing up homeless session... 2024-04-14T15:22:20.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:20.962+0000 7f10680d3700 10 client.37659.objecter successfully canceled tick 2024-04-14T15:22:20.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:20.962+0000 7f10680d3700 1 -- 172.21.15.90:0/3038639602 >> v1:172.21.15.90:6800/420935427 conn(0x7f104c071410 legacy=0x7f104c0738d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:20.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:20.962+0000 7f10680d3700 1 -- 172.21.15.90:0/3038639602 >> v1:172.21.15.90:6789/0 conn(0x7f106010b6f0 legacy=0x7f10601a27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:20.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:20.962+0000 7f10680d3700 1 -- 172.21.15.90:0/3038639602 shutdown_connections 2024-04-14T15:22:20.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:20.962+0000 7f10680d3700 1 -- 172.21.15.90:0/3038639602 >> 172.21.15.90:0/3038639602 conn(0x7f1060076aa0 msgr2=0x7f1060109620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:22:20.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:20.964+0000 7f10680d3700 1 -- 172.21.15.90:0/3038639602 shutdown_connections 2024-04-14T15:22:20.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:20.965+0000 7f10680d3700 1 -- 172.21.15.90:0/3038639602 wait complete. 2024-04-14T15:22:20.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:20.965+0000 7f10680d3700 1 librados: shutdown 2024-04-14T15:22:20.966 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is set 2024-04-14T15:22:21.022 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:22:21.047 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:21.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:21.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:22:21.148 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:21.172 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:21.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:21.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:21.274 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:21.297 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:21.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:21.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:21.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:21 smithi092 ceph-mon[33966]: 1.0 scrub starts 2024-04-14T15:22:21.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:21 smithi092 ceph-mon[33966]: 1.0 scrub ok 2024-04-14T15:22:21.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:21 smithi092 ceph-mon[33966]: 11.1f scrub starts 2024-04-14T15:22:21.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:21 smithi092 ceph-mon[33966]: 11.1f scrub ok 2024-04-14T15:22:21.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:21 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3038639602' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:22:21.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:21 smithi092 ceph-mon[33966]: 5.1e scrub starts 2024-04-14T15:22:21.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:21 smithi092 ceph-mon[33966]: 5.1e scrub ok 2024-04-14T15:22:21.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:21 smithi092 ceph-mon[33966]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:22:21.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:21 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-14T15:22:21.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:21 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3038639602' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-14T15:22:21.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:21 smithi092 ceph-mon[33966]: osdmap e605: 8 total, 8 up, 7 in 2024-04-14T15:22:21.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:21 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-14T15:22:21.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:21 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-14T15:22:21.398 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:22:21.426 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:21.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:21.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:22:21.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:21 smithi090 ceph-mon[41670]: 1.0 scrub starts 2024-04-14T15:22:21.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:21 smithi090 ceph-mon[41670]: 1.0 scrub ok 2024-04-14T15:22:21.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:21 smithi090 ceph-mon[41670]: 11.1f scrub starts 2024-04-14T15:22:21.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:21 smithi090 ceph-mon[41670]: 11.1f scrub ok 2024-04-14T15:22:21.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:21 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3038639602' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:22:21.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:21 smithi090 ceph-mon[41670]: 5.1e scrub starts 2024-04-14T15:22:21.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:21 smithi090 ceph-mon[41670]: 5.1e scrub ok 2024-04-14T15:22:21.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:21 smithi090 ceph-mon[41670]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:22:21.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:21 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-14T15:22:21.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:21 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3038639602' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-14T15:22:21.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:21 smithi090 ceph-mon[41670]: osdmap e605: 8 total, 8 up, 7 in 2024-04-14T15:22:21.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:21 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-14T15:22:21.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:21 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-14T15:22:21.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:21 smithi090 ceph-mon[30399]: 1.0 scrub starts 2024-04-14T15:22:21.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:21 smithi090 ceph-mon[30399]: 1.0 scrub ok 2024-04-14T15:22:21.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:21 smithi090 ceph-mon[30399]: 11.1f scrub starts 2024-04-14T15:22:21.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:21 smithi090 ceph-mon[30399]: 11.1f scrub ok 2024-04-14T15:22:21.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:21 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3038639602' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:22:21.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:21 smithi090 ceph-mon[30399]: 5.1e scrub starts 2024-04-14T15:22:21.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:21 smithi090 ceph-mon[30399]: 5.1e scrub ok 2024-04-14T15:22:21.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:21 smithi090 ceph-mon[30399]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:22:21.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:21 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-14T15:22:21.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:21 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3038639602' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-14T15:22:21.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:21 smithi090 ceph-mon[30399]: osdmap e605: 8 total, 8 up, 7 in 2024-04-14T15:22:21.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:21 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-14T15:22:21.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:21 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-14T15:22:21.528 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:22:21.556 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:21.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:21.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:22:21.657 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:22:21.685 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:21.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:21.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:22:21.787 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:21.810 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:21.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:21.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:21.912 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:21.935 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:21.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:21.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:22.037 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:22:22.062 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:22.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:22.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:22:22.164 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:22:22.189 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:22.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:22.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:22:22.291 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:22:22.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:22.319 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:22.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:22:22.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:22 smithi092 ceph-mon[33966]: 6.1f deep-scrub starts 2024-04-14T15:22:22.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:22 smithi092 ceph-mon[33966]: 6.1f deep-scrub ok 2024-04-14T15:22:22.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:22 smithi092 ceph-mon[33966]: pgmap v888: 242 pgs: 4 active+remapped+backfill_toofull, 238 active+clean; 8.4 MiB data, 517 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/1206 objects misplaced (0.166%) 2024-04-14T15:22:22.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:22 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-102","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-14T15:22:22.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:22 smithi092 ceph-mon[33966]: osdmap e606: 8 total, 8 up, 7 in 2024-04-14T15:22:22.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:22 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:22:22.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:22 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:22:22.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:22 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-102"}]: dispatch 2024-04-14T15:22:22.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:22 smithi092 ceph-mon[33966]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-102"}]: dispatch 2024-04-14T15:22:22.385 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:22 smithi090 ceph-mon[41670]: 6.1f deep-scrub starts 2024-04-14T15:22:22.385 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:22 smithi090 ceph-mon[41670]: 6.1f deep-scrub ok 2024-04-14T15:22:22.385 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:22 smithi090 ceph-mon[41670]: pgmap v888: 242 pgs: 4 active+remapped+backfill_toofull, 238 active+clean; 8.4 MiB data, 517 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/1206 objects misplaced (0.166%) 2024-04-14T15:22:22.385 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:22 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-102","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-14T15:22:22.385 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:22 smithi090 ceph-mon[41670]: osdmap e606: 8 total, 8 up, 7 in 2024-04-14T15:22:22.385 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:22 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:22:22.385 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:22 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:22:22.385 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:22 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-102"}]: dispatch 2024-04-14T15:22:22.385 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:22 smithi090 ceph-mon[41670]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-102"}]: dispatch 2024-04-14T15:22:22.386 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:22 smithi090 ceph-mon[30399]: 6.1f deep-scrub starts 2024-04-14T15:22:22.386 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:22 smithi090 ceph-mon[30399]: 6.1f deep-scrub ok 2024-04-14T15:22:22.386 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:22 smithi090 ceph-mon[30399]: pgmap v888: 242 pgs: 4 active+remapped+backfill_toofull, 238 active+clean; 8.4 MiB data, 517 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/1206 objects misplaced (0.166%) 2024-04-14T15:22:22.386 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:22 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-102","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-14T15:22:22.386 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:22 smithi090 ceph-mon[30399]: osdmap e606: 8 total, 8 up, 7 in 2024-04-14T15:22:22.386 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:22 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:22:22.386 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:22 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-6"}]: dispatch 2024-04-14T15:22:22.386 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:22 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1109245683' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-102"}]: dispatch 2024-04-14T15:22:22.386 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:22 smithi090 ceph-mon[30399]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-6", "tierpool": "test-rados-api-smithi090-93652-102"}]: dispatch 2024-04-14T15:22:22.420 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:22:22.444 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:22.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:22.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:22:22.546 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:22:22.569 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:22.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:22.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:22:22.670 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:22:22.693 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:22.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:22.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:22:22.795 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:22:22.818 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:22.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:22.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:22:22.919 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:22:22.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:22.944 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:22.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:22:23.045 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:22:23.071 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:23.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:23.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:22:23.172 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:23.195 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:23.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:23.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:23.297 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:22:23.320 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:23.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:23.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:22:23.422 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:23.445 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:23.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:23.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:23.547 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:22:23.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:23.571 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:23.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:22:23.589 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-14T15:22:23.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.614+0000 7f5704910700 1 -- 172.21.15.90:0/1919477960 >> v1:172.21.15.90:6789/0 conn(0x7f570010b730 legacy=0x7f570010db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:23.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.614+0000 7f5704910700 1 -- 172.21.15.90:0/1919477960 shutdown_connections 2024-04-14T15:22:23.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.614+0000 7f5704910700 1 -- 172.21.15.90:0/1919477960 >> 172.21.15.90:0/1919477960 conn(0x7f57000fe3d0 msgr2=0x7f57001007f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:22:23.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.615+0000 7f5704910700 1 -- 172.21.15.90:0/1919477960 shutdown_connections 2024-04-14T15:22:23.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.615+0000 7f5704910700 1 -- 172.21.15.90:0/1919477960 wait complete. 2024-04-14T15:22:23.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.615+0000 7f5704910700 1 librados: starting msgr at 2024-04-14T15:22:23.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.615+0000 7f5704910700 1 librados: starting objecter 2024-04-14T15:22:23.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.615+0000 7f5704910700 1 Processor -- start 2024-04-14T15:22:23.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.615+0000 7f5704910700 1 -- start start 2024-04-14T15:22:23.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.615+0000 7f5704910700 1 librados: setting wanted keys 2024-04-14T15:22:23.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.615+0000 7f5704910700 1 librados: calling monclient init 2024-04-14T15:22:23.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.616+0000 7f5704910700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f57001a9d80 con 0x7f570010b730 2024-04-14T15:22:23.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.616+0000 7f5704910700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f57001aafa0 con 0x7f5700108880 2024-04-14T15:22:23.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.616+0000 7f5704910700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f57001ac1e0 con 0x7f570010f300 2024-04-14T15:22:23.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.616+0000 7f56fed9d700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f570010f300 0x7f57001a8440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:54642/0 (socket says 172.21.15.90:54642) 2024-04-14T15:22:23.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.616+0000 7f56fdd9b700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f570010b730 0x7f57001a4ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:53248/0 (socket says 172.21.15.90:53248) 2024-04-14T15:22:23.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.616+0000 7f56fed9d700 1 -- 172.21.15.90:0/348550871 learned_addr learned my addr 172.21.15.90:0/348550871 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:22:23.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.616+0000 7f56ef7fe700 10 client.?.objecter ms_handle_connect 0x7f570010b730 2024-04-14T15:22:23.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.616+0000 7f56ef7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:22:23.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.616+0000 7f56ef7fe700 10 client.?.objecter ms_handle_connect 0x7f5700108880 2024-04-14T15:22:23.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.616+0000 7f56ef7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:22:23.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.616+0000 7f56ef7fe700 10 client.?.objecter ms_handle_connect 0x7f570010f300 2024-04-14T15:22:23.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.616+0000 7f56ef7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:22:23.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.616+0000 7f56ef7fe700 1 -- 172.21.15.90:0/348550871 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 184457147 0 0) 0x7f57001aafa0 con 0x7f5700108880 2024-04-14T15:22:23.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.616+0000 7f56ef7fe700 1 -- 172.21.15.90:0/348550871 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f56e80035b0 con 0x7f5700108880 2024-04-14T15:22:23.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.616+0000 7f56ef7fe700 1 -- 172.21.15.90:0/348550871 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1436951850 0 0) 0x7f57001a9d80 con 0x7f570010b730 2024-04-14T15:22:23.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.616+0000 7f56ef7fe700 1 -- 172.21.15.90:0/348550871 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f57001aafa0 con 0x7f570010b730 2024-04-14T15:22:23.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.616+0000 7f56ef7fe700 1 -- 172.21.15.90:0/348550871 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2385040403 0 0) 0x7f57001ac1e0 con 0x7f570010f300 2024-04-14T15:22:23.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.616+0000 7f56ef7fe700 1 -- 172.21.15.90:0/348550871 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f57001a9d80 con 0x7f570010f300 2024-04-14T15:22:23.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.616+0000 7f56ef7fe700 1 -- 172.21.15.90:0/348550871 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3311742383 0 0) 0x7f56e80035b0 con 0x7f5700108880 2024-04-14T15:22:23.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.616+0000 7f56ef7fe700 1 -- 172.21.15.90:0/348550871 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f57001ac1e0 con 0x7f5700108880 2024-04-14T15:22:23.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.616+0000 7f56ef7fe700 1 -- 172.21.15.90:0/348550871 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f56f00018b0 con 0x7f5700108880 2024-04-14T15:22:23.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.617+0000 7f56ef7fe700 1 -- 172.21.15.90:0/348550871 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 18322059 0 0) 0x7f57001aafa0 con 0x7f570010b730 2024-04-14T15:22:23.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.617+0000 7f56ef7fe700 1 -- 172.21.15.90:0/348550871 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f56e80035b0 con 0x7f570010b730 2024-04-14T15:22:23.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.617+0000 7f56ef7fe700 1 -- 172.21.15.90:0/348550871 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f56f40015d0 con 0x7f570010b730 2024-04-14T15:22:23.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.617+0000 7f56ef7fe700 1 -- 172.21.15.90:0/348550871 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 370364858 0 0) 0x7f57001ac1e0 con 0x7f5700108880 2024-04-14T15:22:23.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.617+0000 7f56ef7fe700 1 -- 172.21.15.90:0/348550871 >> v1:172.21.15.90:6790/0 conn(0x7f570010f300 legacy=0x7f57001a8440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:23.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.617+0000 7f56ef7fe700 1 -- 172.21.15.90:0/348550871 >> v1:172.21.15.90:6789/0 conn(0x7f570010b730 legacy=0x7f57001a4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:23.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.617+0000 7f56ef7fe700 1 -- 172.21.15.90:0/348550871 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57001ad420 con 0x7f5700108880 2024-04-14T15:22:23.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.617+0000 7f5704910700 1 -- 172.21.15.90:0/348550871 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f57001ab1d0 con 0x7f5700108880 2024-04-14T15:22:23.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.617+0000 7f5704910700 10 client.49498.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:22:23.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.617+0000 7f5704910700 1 -- 172.21.15.90:0/348550871 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f57001ab690 con 0x7f5700108880 2024-04-14T15:22:23.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.617+0000 7f5704910700 1 librados: init done 2024-04-14T15:22:23.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.617+0000 7f56ef7fe700 1 -- 172.21.15.90:0/348550871 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f56f0003b40 con 0x7f5700108880 2024-04-14T15:22:23.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.617+0000 7f56ef7fe700 1 -- 172.21.15.90:0/348550871 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f56f0004a20 con 0x7f5700108880 2024-04-14T15:22:23.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.617+0000 7f5704910700 10 client.49498.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:22:23.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.617+0000 7f5704910700 15 client.49498.objecter _session_command_op_assign -1 1 2024-04-14T15:22:23.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.617+0000 7f5704910700 20 client.49498.objecter _get_session osd=-1 returning homeless 2024-04-14T15:22:23.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.617+0000 7f5704910700 10 client.49498.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:22:23.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.618+0000 7f5704910700 1 -- 172.21.15.90:0/348550871 --> v1:172.21.15.92:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f56e0002950 con 0x7f5700108880 2024-04-14T15:22:23.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.619+0000 7f56ef7fe700 1 -- 172.21.15.90:0/348550871 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f56f0005dd0 con 0x7f5700108880 2024-04-14T15:22:23.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.619+0000 7f56ef7fe700 1 -- 172.21.15.90:0/348550871 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(607..607 src has 467..607) ==== 7882+0+0 (unknown 799719173 0 0) 0x7f56f0089720 con 0x7f5700108880 2024-04-14T15:22:23.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.619+0000 7f56ef7fe700 10 client.49498.objecter ms_dispatch 0x7f5700001550 osd_map(607..607 src has 467..607) 2024-04-14T15:22:23.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.619+0000 7f56ef7fe700 3 client.49498.objecter handle_osd_map got epochs [607,607] > 0 2024-04-14T15:22:23.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.619+0000 7f56ef7fe700 3 client.49498.objecter handle_osd_map decoding full epoch 607 2024-04-14T15:22:23.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.619+0000 7f56ef7fe700 10 client.49498.objecter checking command 1 2024-04-14T15:22:23.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.619+0000 7f56ef7fe700 20 client.49498.objecter _get_session s=0x7f56e807df70 osd=2 3 2024-04-14T15:22:23.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.619+0000 7f56ef7fe700 20 client.49498.objecter put_session s=0x7f56e807df70 osd=2 3 2024-04-14T15:22:23.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.619+0000 7f56ef7fe700 15 client.49498.objecter _session_command_op_remove -1 1 2024-04-14T15:22:23.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.619+0000 7f56ef7fe700 20 client.49498.objecter _get_session s=0x7f56e807df70 osd=2 3 2024-04-14T15:22:23.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.619+0000 7f56ef7fe700 20 client.49498.objecter get_session s=0x7f56e807df70 osd=2 3 2024-04-14T15:22:23.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.619+0000 7f56ef7fe700 15 client.49498.objecter _session_command_op_assign 2 1 2024-04-14T15:22:23.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.619+0000 7f56ef7fe700 20 client.49498.objecter put_session s=0x7f56e807df70 osd=2 4 2024-04-14T15:22:23.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.619+0000 7f56ef7fe700 10 client.49498.objecter _send_command 1 2024-04-14T15:22:23.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.619+0000 7f56ef7fe700 1 -- 172.21.15.90:0/348550871 --> v1:172.21.15.90:6809/1686184251 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f56e8082330 con 0x7f56e807e620 2024-04-14T15:22:23.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.619+0000 7f56ef7fe700 20 client.49498.objecter dump_active .. 0 homeless 2024-04-14T15:22:23.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.619+0000 7f56ef7fe700 1 -- 172.21.15.90:0/348550871 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_get_version_reply(handle=1 version=607) ==== 24+0+0 (unknown 2126884531 0 0) 0x7f56f0089a80 con 0x7f5700108880 2024-04-14T15:22:23.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.619+0000 7f56ef7fe700 10 client.49498.objecter ms_handle_connect 0x7f56e8071290 2024-04-14T15:22:23.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.620+0000 7f56ef7fe700 10 client.49498.objecter ms_handle_connect 0x7f56e807e620 2024-04-14T15:22:23.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.621+0000 7f56ef7fe700 1 -- 172.21.15.90:0/348550871 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f56e8082330 con 0x7f56e807e620 2024-04-14T15:22:23.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.621+0000 7f56ef7fe700 10 client.49498.objecter ms_dispatch 0x7f5700001550 command_reply(tid 1: 0 ) 2024-04-14T15:22:23.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.621+0000 7f56ef7fe700 10 client.49498.objecter _finish_command 1 = system:0 2024-04-14T15:22:23.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.621+0000 7f56ef7fe700 20 client.49498.objecter put_session s=0x7f56e807df70 osd=2 4 2024-04-14T15:22:23.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.621+0000 7f56ef7fe700 15 client.49498.objecter _session_command_op_remove 2 1 2024-04-14T15:22:23.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.640+0000 7f5704910700 10 client.49498.objecter _submit_command 2 [{"prefix": "reset_purged_snaps_last"}] 2024-04-14T15:22:23.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.640+0000 7f5704910700 15 client.49498.objecter _session_command_op_assign -1 2 2024-04-14T15:22:23.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.640+0000 7f5704910700 20 client.49498.objecter _get_session s=0x7f56e807df70 osd=2 3 2024-04-14T15:22:23.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.640+0000 7f5704910700 20 client.49498.objecter put_session s=0x7f56e807df70 osd=2 3 2024-04-14T15:22:23.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.640+0000 7f5704910700 20 client.49498.objecter _get_session s=0x7f56e807df70 osd=2 3 2024-04-14T15:22:23.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.640+0000 7f5704910700 15 client.49498.objecter _session_command_op_remove -1 2 2024-04-14T15:22:23.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.640+0000 7f5704910700 20 client.49498.objecter get_session s=0x7f56e807df70 osd=2 3 2024-04-14T15:22:23.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.640+0000 7f5704910700 15 client.49498.objecter _session_command_op_assign 2 2 2024-04-14T15:22:23.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.640+0000 7f5704910700 20 client.49498.objecter put_session s=0x7f56e807df70 osd=2 4 2024-04-14T15:22:23.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.640+0000 7f5704910700 10 client.49498.objecter _send_command 2 2024-04-14T15:22:23.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.640+0000 7f5704910700 1 -- 172.21.15.90:0/348550871 --> v1:172.21.15.90:6809/1686184251 -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) -- 0x7f56e0001220 con 0x7f56e807e620 2024-04-14T15:22:23.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.640+0000 7f56ef7fe700 1 -- 172.21.15.90:0/348550871 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 2: 0 ) ==== 8+0+0 (unknown 0 0 0) 0x7f56e0001220 con 0x7f56e807e620 2024-04-14T15:22:23.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.640+0000 7f56ef7fe700 10 client.49498.objecter ms_dispatch 0x7f5700001550 command_reply(tid 2: 0 ) 2024-04-14T15:22:23.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.640+0000 7f56ef7fe700 10 client.49498.objecter _finish_command 2 = system:0 2024-04-14T15:22:23.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.640+0000 7f56ef7fe700 20 client.49498.objecter put_session s=0x7f56e807df70 osd=2 4 2024-04-14T15:22:23.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.640+0000 7f56ef7fe700 15 client.49498.objecter _session_command_op_remove 2 2 2024-04-14T15:22:23.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.641+0000 7f5704910700 10 librados: watch_flush enter 2024-04-14T15:22:23.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.641+0000 7f5704910700 10 librados: watch_flush exit 2024-04-14T15:22:23.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.641+0000 7f5704910700 10 client.49498.objecter close_session for osd.2 2024-04-14T15:22:23.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.641+0000 7f5704910700 1 -- 172.21.15.90:0/348550871 >> v1:172.21.15.90:6809/1686184251 conn(0x7f56e807e620 legacy=0x7f56e8080ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:23.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.641+0000 7f5704910700 20 client.49498.objecter put_session s=0x7f56e807df70 osd=2 1 2024-04-14T15:22:23.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.641+0000 7f5704910700 20 client.49498.objecter shutdown clearing up homeless session... 2024-04-14T15:22:23.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.641+0000 7f5704910700 10 client.49498.objecter successfully canceled tick 2024-04-14T15:22:23.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.641+0000 7f5704910700 1 -- 172.21.15.90:0/348550871 >> v1:172.21.15.90:6800/420935427 conn(0x7f56e8071290 legacy=0x7f56e8073750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:23.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.641+0000 7f5704910700 1 -- 172.21.15.90:0/348550871 >> v1:172.21.15.92:6789/0 conn(0x7f5700108880 legacy=0x7f5700076760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:23.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.641+0000 7f56fed9d700 1 -- 172.21.15.90:0/348550871 reap_dead start 2024-04-14T15:22:23.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.641+0000 7f5704910700 1 -- 172.21.15.90:0/348550871 shutdown_connections 2024-04-14T15:22:23.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.641+0000 7f5704910700 1 -- 172.21.15.90:0/348550871 >> 172.21.15.90:0/348550871 conn(0x7f57000fe3d0 msgr2=0x7f5700110ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:22:23.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.641+0000 7f5704910700 1 -- 172.21.15.90:0/348550871 shutdown_connections 2024-04-14T15:22:23.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.641+0000 7f5704910700 1 -- 172.21.15.90:0/348550871 wait complete. 2024-04-14T15:22:23.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:23.641+0000 7f5704910700 1 librados: shutdown 2024-04-14T15:22:23.672 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:22:23.695 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:23.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:23.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:22:23.797 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:22:23.819 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:23.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:23.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:22:23.921 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:22:23.945 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:23.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:23.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:22:24.047 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:24.072 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:24.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:24.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:24.174 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:24.191 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:22:23 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:22:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:22:24.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:23 smithi090 ceph-mon[41670]: osdmap e607: 8 total, 8 up, 7 in 2024-04-14T15:22:24.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:23 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:22:24.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:23 smithi090 ceph-mon[30399]: osdmap e607: 8 total, 8 up, 7 in 2024-04-14T15:22:24.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:23 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:22:24.198 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:24.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:24.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:24.300 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:22:24.323 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:24.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:24.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:22:24.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:23 smithi092 ceph-mon[33966]: osdmap e607: 8 total, 8 up, 7 in 2024-04-14T15:22:24.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:23 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:22:24.424 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:24.448 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:24.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:24.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:24.550 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:22:24.573 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:24.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:24.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:22:24.675 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:22:24.698 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:24.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:24.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:22:24.800 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:22:24.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:24.827 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:24.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:22:24.928 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:24.952 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:24.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:24.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:25.053 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:22:25.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:25.079 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:25.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:22:25.127 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph tell osd.3 reset_purged_snaps_last 2024-04-14T15:22:25.180 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:22:25.205 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:25.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:25.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:22:25.245 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:24 smithi090 ceph-mon[41670]: pgmap v891: 210 pgs: 4 active+remapped+backfill_toofull, 206 active+clean; 8.4 MiB data, 517 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/1206 objects misplaced (0.166%) 2024-04-14T15:22:25.245 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:24 smithi090 ceph-mon[41670]: osdmap e608: 8 total, 8 up, 7 in 2024-04-14T15:22:25.246 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:24 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-04-14T15:22:25.246 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:24 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]}]: dispatch 2024-04-14T15:22:25.246 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:24 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "11.1c", "id": [2, 4]}]: dispatch 2024-04-14T15:22:25.246 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:24 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1673052523' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi090-93652-104"}]: dispatch 2024-04-14T15:22:25.246 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:24 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1673052523' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi090-93652-104"}]: dispatch 2024-04-14T15:22:25.246 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:24 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1673052523' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi090-93652-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:22:25.246 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:24 smithi090 ceph-mon[30399]: pgmap v891: 210 pgs: 4 active+remapped+backfill_toofull, 206 active+clean; 8.4 MiB data, 517 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/1206 objects misplaced (0.166%) 2024-04-14T15:22:25.246 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:24 smithi090 ceph-mon[30399]: osdmap e608: 8 total, 8 up, 7 in 2024-04-14T15:22:25.246 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:24 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-04-14T15:22:25.246 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:24 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]}]: dispatch 2024-04-14T15:22:25.246 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:24 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "11.1c", "id": [2, 4]}]: dispatch 2024-04-14T15:22:25.247 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:24 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1673052523' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi090-93652-104"}]: dispatch 2024-04-14T15:22:25.247 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:24 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1673052523' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi090-93652-104"}]: dispatch 2024-04-14T15:22:25.247 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:24 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1673052523' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi090-93652-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:22:25.299 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:24 smithi092 ceph-mon[33966]: pgmap v891: 210 pgs: 4 active+remapped+backfill_toofull, 206 active+clean; 8.4 MiB data, 517 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/1206 objects misplaced (0.166%) 2024-04-14T15:22:25.300 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:24 smithi092 ceph-mon[33966]: osdmap e608: 8 total, 8 up, 7 in 2024-04-14T15:22:25.300 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:24 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-04-14T15:22:25.300 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:24 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]}]: dispatch 2024-04-14T15:22:25.300 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:24 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "11.1c", "id": [2, 4]}]: dispatch 2024-04-14T15:22:25.300 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:24 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1673052523' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi090-93652-104"}]: dispatch 2024-04-14T15:22:25.300 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:24 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1673052523' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi090-93652-104"}]: dispatch 2024-04-14T15:22:25.300 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:24 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1673052523' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi090-93652-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:22:25.306 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:25.329 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:25.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:25.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:25.366 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-14T15:22:25.431 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:22:25.458 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:25.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:25.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:22:25.560 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:22:25.583 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:25.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:25.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:22:25.622 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:22:25 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:22:25.684 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:22:25.704 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:22:25.711 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:25.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:25.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:22:25.813 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:22:25.836 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:25.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:25.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:22:25.938 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:22:25.968 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:25.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:25.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:22:26.069 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:22:26.093 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:26.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:26.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:22:26.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:25 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2024-04-14T15:22:26.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:25 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]}]': finished 2024-04-14T15:22:26.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:25 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "11.1c", "id": [2, 4]}]': finished 2024-04-14T15:22:26.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:25 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1673052523' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi090-93652-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:22:26.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:25 smithi090 ceph-mon[41670]: osdmap e609: 8 total, 8 up, 7 in 2024-04-14T15:22:26.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:25 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1673052523' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi090-93652-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi090-93652-104"}]: dispatch 2024-04-14T15:22:26.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:25 smithi090 ceph-mon[41670]: osdmap e610: 8 total, 8 up, 7 in 2024-04-14T15:22:26.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:25 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2024-04-14T15:22:26.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:25 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]}]': finished 2024-04-14T15:22:26.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:25 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "11.1c", "id": [2, 4]}]': finished 2024-04-14T15:22:26.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:25 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1673052523' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi090-93652-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:22:26.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:25 smithi090 ceph-mon[30399]: osdmap e609: 8 total, 8 up, 7 in 2024-04-14T15:22:26.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:25 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1673052523' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi090-93652-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi090-93652-104"}]: dispatch 2024-04-14T15:22:26.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:25 smithi090 ceph-mon[30399]: osdmap e610: 8 total, 8 up, 7 in 2024-04-14T15:22:26.195 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:22:26.223 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:26.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:26.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:22:26.325 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:22:26.349 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:26.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:26.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:22:26.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:25 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2024-04-14T15:22:26.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:25 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]}]': finished 2024-04-14T15:22:26.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:25 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "11.1c", "id": [2, 4]}]': finished 2024-04-14T15:22:26.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:25 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1673052523' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi090-93652-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:22:26.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:25 smithi092 ceph-mon[33966]: osdmap e609: 8 total, 8 up, 7 in 2024-04-14T15:22:26.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:25 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1673052523' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi090-93652-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi090-93652-104"}]: dispatch 2024-04-14T15:22:26.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:25 smithi092 ceph-mon[33966]: osdmap e610: 8 total, 8 up, 7 in 2024-04-14T15:22:26.450 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:22:26.457 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:22:26.477 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:26.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:26.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:22:26.579 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:22:26.607 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:26.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:26.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:22:26.708 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:22:26.736 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:26.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:26.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:22:26.837 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:22:26.865 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:26.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:26.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:22:26.966 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:26.993 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:26.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:26.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:27.095 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:22:27.122 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:27.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:27.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:22:27.224 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:22:27.251 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:27.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:27.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:22:27.255 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:26 smithi090 ceph-mon[41670]: pgmap v894: 178 pgs: 3 active+remapped+backfill_toofull, 175 active+clean; 596 KiB data, 522 MiB used, 625 GiB / 626 GiB avail; 2/597 objects misplaced (0.335%) 2024-04-14T15:22:27.256 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:26 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:22:27.256 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:26 smithi090 ceph-mon[41670]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-04-14T15:22:27.256 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:26 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1673052523' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi090-93652-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi090-93652-104"}]': finished 2024-04-14T15:22:27.256 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:26 smithi090 ceph-mon[41670]: osdmap e611: 8 total, 8 up, 7 in 2024-04-14T15:22:27.256 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:26 smithi090 ceph-mon[30399]: pgmap v894: 178 pgs: 3 active+remapped+backfill_toofull, 175 active+clean; 596 KiB data, 522 MiB used, 625 GiB / 626 GiB avail; 2/597 objects misplaced (0.335%) 2024-04-14T15:22:27.256 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:26 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:22:27.256 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:26 smithi090 ceph-mon[30399]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-04-14T15:22:27.256 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:26 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1673052523' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi090-93652-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi090-93652-104"}]': finished 2024-04-14T15:22:27.256 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:26 smithi090 ceph-mon[30399]: osdmap e611: 8 total, 8 up, 7 in 2024-04-14T15:22:27.353 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:27.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:26 smithi092 ceph-mon[33966]: pgmap v894: 178 pgs: 3 active+remapped+backfill_toofull, 175 active+clean; 596 KiB data, 522 MiB used, 625 GiB / 626 GiB avail; 2/597 objects misplaced (0.335%) 2024-04-14T15:22:27.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:26 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:22:27.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:26 smithi092 ceph-mon[33966]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-04-14T15:22:27.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:26 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1673052523' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi090-93652-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi090-93652-104"}]': finished 2024-04-14T15:22:27.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:26 smithi092 ceph-mon[33966]: osdmap e611: 8 total, 8 up, 7 in 2024-04-14T15:22:27.378 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:27.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:27.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:27.480 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:22:27.503 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:27.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:27.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:22:27.605 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:22:27.633 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:27.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:27.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:22:27.735 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:22:27.758 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:27.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:27.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:22:27.860 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:22:27.887 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:27.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:27.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:22:27.971 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-14T15:22:27.971 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-14T15:22:27.971 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-14T15:22:27.972 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-14T15:22:27.972 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.972 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24889.0:2 22.15 22:ac175842:test-rados-api-smithi090-93930-2::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-04-14T15:22:27.972 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:12:56.066705+0000", 2024-04-14T15:22:27.972 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 571.86402822000002, 2024-04-14T15:22:27.972 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.43969079999999999, 2024-04-14T15:22:27.972 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:22:27.972 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:22:27.972 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:22:27.972 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:22:27.972 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24889", 2024-04-14T15:22:27.972 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3182938272", 2024-04-14T15:22:27.972 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2 2024-04-14T15:22:27.972 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.973 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:22:27.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.973 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:22:27.973 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.066705+0000", 2024-04-14T15:22:27.973 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:22:27.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.973 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:22:27.973 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.066705+0000", 2024-04-14T15:22:27.973 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:22:27.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.973 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:22:27.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.066708+0000", 2024-04-14T15:22:27.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4320000000000002e-06 2024-04-14T15:22:27.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.974 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:22:27.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.066756+0000", 2024-04-14T15:22:27.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7888e-05 2024-04-14T15:22:27.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.974 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:22:27.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.066775+0000", 2024-04-14T15:22:27.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9391000000000001e-05 2024-04-14T15:22:27.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:22:27.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.066788+0000", 2024-04-14T15:22:27.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2519e-05 2024-04-14T15:22:27.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:22:27.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.066860+0000", 2024-04-14T15:22:27.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1919e-05 2024-04-14T15:22:27.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2024-04-14T15:22:27.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.066867+0000", 2024-04-14T15:22:27.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1130000000000003e-06 2024-04-14T15:22:27.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:22:27.976 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.145373+0000", 2024-04-14T15:22:27.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.078505825000000001 2024-04-14T15:22:27.976 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.976 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:22:27.976 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.145383+0000", 2024-04-14T15:22:27.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0196e-05 2024-04-14T15:22:27.976 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.976 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:22:27.976 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.288000+0000", 2024-04-14T15:22:27.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.14261763599999999 2024-04-14T15:22:27.976 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.976 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:22:27.976 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.288166+0000", 2024-04-14T15:22:27.977 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016534000000000001 2024-04-14T15:22:27.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.977 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4,7", 2024-04-14T15:22:27.977 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.288314+0000", 2024-04-14T15:22:27.977 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000148293 2024-04-14T15:22:27.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.977 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:22:27.977 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.337996+0000", 2024-04-14T15:22:27.977 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.049681625 2024-04-14T15:22:27.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.977 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:22:27.977 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.344006+0000", 2024-04-14T15:22:27.977 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0060103960000000003 2024-04-14T15:22:27.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.978 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:22:27.978 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.504829+0000", 2024-04-14T15:22:27.978 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.16082263299999999 2024-04-14T15:22:27.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.978 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:22:27.978 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.504878+0000", 2024-04-14T15:22:27.978 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9257e-05 2024-04-14T15:22:27.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.978 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:22:27.978 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.506396+0000", 2024-04-14T15:22:27.978 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001518337 2024-04-14T15:22:27.979 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:22:27.979 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:22:27.979 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:22:27.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.979 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24923.0:2 25.6 25:65dba387:test-rados-api-smithi090-94131-3::testobj:head [write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-04-14T15:22:27.979 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:12:56.071392+0000", 2024-04-14T15:22:27.979 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 571.85934195200002, 2024-04-14T15:22:27.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49329716000000001, 2024-04-14T15:22:27.979 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:22:27.979 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:22:27.979 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:22:27.979 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:22:27.979 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24923", 2024-04-14T15:22:27.979 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/4254723003", 2024-04-14T15:22:27.979 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2 2024-04-14T15:22:27.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.980 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:22:27.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:22:27.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.071392+0000", 2024-04-14T15:22:27.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:22:27.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:22:27.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.071392+0000", 2024-04-14T15:22:27.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:22:27.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:22:27.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.071394+0000", 2024-04-14T15:22:27.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.294e-06 2024-04-14T15:22:27.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:22:27.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.071403+0000", 2024-04-14T15:22:27.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8999999999999995e-06 2024-04-14T15:22:27.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:22:27.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.071416+0000", 2024-04-14T15:22:27.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3366999999999999e-05 2024-04-14T15:22:27.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:22:27.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.071426+0000", 2024-04-14T15:22:27.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5030000000000003e-06 2024-04-14T15:22:27.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:22:27.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.072369+0000", 2024-04-14T15:22:27.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00094297499999999998 2024-04-14T15:22:27.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2024-04-14T15:22:27.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.072375+0000", 2024-04-14T15:22:27.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5400000000000001e-06 2024-04-14T15:22:27.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:22:27.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.238121+0000", 2024-04-14T15:22:27.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.165745485 2024-04-14T15:22:27.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.983 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:22:27.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.238129+0000", 2024-04-14T15:22:27.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.49e-06 2024-04-14T15:22:27.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.983 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:22:27.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.380419+0000", 2024-04-14T15:22:27.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.142289742 2024-04-14T15:22:27.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.983 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:22:27.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.380539+0000", 2024-04-14T15:22:27.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000120221 2024-04-14T15:22:27.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,5", 2024-04-14T15:22:27.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.380682+0000", 2024-04-14T15:22:27.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000142924 2024-04-14T15:22:27.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:22:27.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.408898+0000", 2024-04-14T15:22:27.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.028216187 2024-04-14T15:22:27.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:22:27.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.564447+0000", 2024-04-14T15:22:27.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15554912700000001 2024-04-14T15:22:27.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:22:27.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.564550+0000", 2024-04-14T15:22:27.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000102725 2024-04-14T15:22:27.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:22:27.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.564602+0000", 2024-04-14T15:22:27.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1891999999999997e-05 2024-04-14T15:22:27.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:22:27.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.564689+0000", 2024-04-14T15:22:27.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6787999999999996e-05 2024-04-14T15:22:27.985 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:22:27.985 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:22:27.986 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:22:27.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.986 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24853.0:2 16.1d 16:b9b4aa4b:test-rados-api-smithi090-93601-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-04-14T15:22:27.986 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:12:56.373038+0000", 2024-04-14T15:22:27.986 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 571.55769590600005, 2024-04-14T15:22:27.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.193584592, 2024-04-14T15:22:27.986 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:22:27.986 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:22:27.986 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:22:27.986 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:22:27.986 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24853", 2024-04-14T15:22:27.986 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3851923713", 2024-04-14T15:22:27.987 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2 2024-04-14T15:22:27.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.987 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:22:27.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:22:27.987 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.373038+0000", 2024-04-14T15:22:27.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:22:27.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:22:27.987 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.373038+0000", 2024-04-14T15:22:27.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:22:27.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.988 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:22:27.988 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.373039+0000", 2024-04-14T15:22:27.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.711e-06 2024-04-14T15:22:27.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.988 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:22:27.988 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.373047+0000", 2024-04-14T15:22:27.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1300000000000003e-06 2024-04-14T15:22:27.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.988 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:22:27.988 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.373064+0000", 2024-04-14T15:22:27.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7603000000000001e-05 2024-04-14T15:22:27.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.989 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:22:27.989 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.373080+0000", 2024-04-14T15:22:27.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5344000000000001e-05 2024-04-14T15:22:27.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.989 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:22:27.989 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.404205+0000", 2024-04-14T15:22:27.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.03112556 2024-04-14T15:22:27.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.989 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:22:27.989 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.404429+0000", 2024-04-14T15:22:27.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022366199999999999 2024-04-14T15:22:27.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.990 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,7", 2024-04-14T15:22:27.990 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.404619+0000", 2024-04-14T15:22:27.990 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019027099999999999 2024-04-14T15:22:27.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.990 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:22:27.990 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.406595+0000", 2024-04-14T15:22:27.990 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0019755699999999998 2024-04-14T15:22:27.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.990 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:22:27.990 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.564357+0000", 2024-04-14T15:22:27.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15776261699999999 2024-04-14T15:22:27.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.991 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:22:27.991 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.566534+0000", 2024-04-14T15:22:27.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0021764670000000001 2024-04-14T15:22:27.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.991 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:22:27.991 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.566583+0000", 2024-04-14T15:22:27.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9251e-05 2024-04-14T15:22:27.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.991 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:22:27.992 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.566622+0000", 2024-04-14T15:22:27.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9406000000000002e-05 2024-04-14T15:22:27.992 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:22:27.992 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:22:27.992 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:22:27.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.992 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24860.0:10 21.15 21:abf5dd0a:test-rados-api-smithi090-93864-5::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-04-14T15:22:27.992 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:12:56.544723+0000", 2024-04-14T15:22:27.992 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 571.38601031799999, 2024-04-14T15:22:27.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.049845436, 2024-04-14T15:22:27.992 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:22:27.993 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:22:27.993 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:22:27.993 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:22:27.993 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24860", 2024-04-14T15:22:27.993 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/766142007", 2024-04-14T15:22:27.993 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 10 2024-04-14T15:22:27.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.993 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:22:27.993 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.993 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:22:27.993 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.544723+0000", 2024-04-14T15:22:27.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:22:27.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.993 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.993 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:22:27.994 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.544723+0000", 2024-04-14T15:22:27.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:22:27.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:22:27.994 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.544725+0000", 2024-04-14T15:22:27.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2359999999999999e-06 2024-04-14T15:22:27.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:22:27.994 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.544730+0000", 2024-04-14T15:22:27.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6439999999999997e-06 2024-04-14T15:22:27.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:22:27.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.544737+0000", 2024-04-14T15:22:27.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2509999999999997e-06 2024-04-14T15:22:27.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.995 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:22:27.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.544745+0000", 2024-04-14T15:22:27.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1029999999999996e-06 2024-04-14T15:22:27.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.995 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:22:27.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.544796+0000", 2024-04-14T15:22:27.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1461000000000001e-05 2024-04-14T15:22:27.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.995 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:22:27.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.544899+0000", 2024-04-14T15:22:27.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000102927 2024-04-14T15:22:27.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2024-04-14T15:22:27.996 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.545067+0000", 2024-04-14T15:22:27.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016825400000000001 2024-04-14T15:22:27.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:22:27.996 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.546327+0000", 2024-04-14T15:22:27.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001259713 2024-04-14T15:22:27.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:22:27.996 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.546461+0000", 2024-04-14T15:22:27.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013432799999999999 2024-04-14T15:22:27.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:22:27.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.594465+0000", 2024-04-14T15:22:27.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.048003571000000002 2024-04-14T15:22:27.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:22:27.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.594519+0000", 2024-04-14T15:22:27.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4620000000000002e-05 2024-04-14T15:22:27.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:22:27.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:56.594569+0000", 2024-04-14T15:22:27.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9327999999999999e-05 2024-04-14T15:22:27.998 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:22:27.998 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:22:27.998 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:22:27.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.998 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24853.0:2059 16.1 16:85b90e3d:test-rados-api-smithi090-93601-16::big:head [writefull 0~25165836] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-04-14T15:22:27.998 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:12:58.834624+0000", 2024-04-14T15:22:27.998 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 569.09610970300002, 2024-04-14T15:22:27.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.072638431000000003, 2024-04-14T15:22:27.998 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:22:27.998 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:22:27.998 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:22:27.998 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:22:27.998 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24853", 2024-04-14T15:22:27.999 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3851923713", 2024-04-14T15:22:27.999 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2059 2024-04-14T15:22:27.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.999 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:22:27.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.999 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:22:27.999 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.834624+0000", 2024-04-14T15:22:27.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:22:27.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:27.999 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:22:27.999 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.834624+0000", 2024-04-14T15:22:27.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:22:27.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:27.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.000 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:22:28.000 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.834628+0000", 2024-04-14T15:22:28.000 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0409999999999999e-06 2024-04-14T15:22:28.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.000 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.000 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:22:28.000 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.845804+0000", 2024-04-14T15:22:28.000 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011176303 2024-04-14T15:22:28.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.000 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.000 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:22:28.000 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.845830+0000", 2024-04-14T15:22:28.000 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5939999999999999e-05 2024-04-14T15:22:28.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:22:28.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.845841+0000", 2024-04-14T15:22:28.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0954999999999999e-05 2024-04-14T15:22:28.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:22:28.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.845938+0000", 2024-04-14T15:22:28.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7158999999999996e-05 2024-04-14T15:22:28.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:22:28.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.846060+0000", 2024-04-14T15:22:28.002 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012211700000000001 2024-04-14T15:22:28.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.002 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,6", 2024-04-14T15:22:28.002 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.846167+0000", 2024-04-14T15:22:28.002 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010677 2024-04-14T15:22:28.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.002 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:22:28.002 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.903809+0000", 2024-04-14T15:22:28.002 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.057642081999999997 2024-04-14T15:22:28.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:22:28.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.906802+0000", 2024-04-14T15:22:28.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002992279 2024-04-14T15:22:28.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:22:28.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.907187+0000", 2024-04-14T15:22:28.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038566799999999998 2024-04-14T15:22:28.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:22:28.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.907226+0000", 2024-04-14T15:22:28.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9014999999999998e-05 2024-04-14T15:22:28.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.004 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:22:28.004 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:58.907262+0000", 2024-04-14T15:22:28.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6102000000000001e-05 2024-04-14T15:22:28.004 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:22:28.004 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:22:28.004 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:22:28.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.004 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24853.0:2064 16.e 16:722084bd:test-rados-api-smithi090-93601-16::big.copy2:head [copy-from ver 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-04-14T15:22:28.004 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:12:59.026890+0000", 2024-04-14T15:22:28.004 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 568.90384356799996, 2024-04-14T15:22:28.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.178520341, 2024-04-14T15:22:28.004 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:22:28.004 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:22:28.004 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:22:28.004 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:22:28.005 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24853", 2024-04-14T15:22:28.005 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3851923713", 2024-04-14T15:22:28.005 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2064 2024-04-14T15:22:28.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.005 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:22:28.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.005 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:22:28.005 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.026890+0000", 2024-04-14T15:22:28.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:22:28.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.005 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:22:28.005 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.026890+0000", 2024-04-14T15:22:28.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:22:28.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.006 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:22:28.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.026891+0000", 2024-04-14T15:22:28.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6599999999999994e-07 2024-04-14T15:22:28.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.006 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:22:28.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.026897+0000", 2024-04-14T15:22:28.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6520000000000003e-06 2024-04-14T15:22:28.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.006 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:22:28.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.026904+0000", 2024-04-14T15:22:28.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1169999999999998e-06 2024-04-14T15:22:28.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:22:28.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.026909+0000", 2024-04-14T15:22:28.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0590000000000002e-06 2024-04-14T15:22:28.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:22:28.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.026951+0000", 2024-04-14T15:22:28.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1758999999999998e-05 2024-04-14T15:22:28.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:22:28.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.027040+0000", 2024-04-14T15:22:28.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9747999999999995e-05 2024-04-14T15:22:28.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2024-04-14T15:22:28.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.090575+0000", 2024-04-14T15:22:28.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.063534532000000005 2024-04-14T15:22:28.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:22:28.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.102440+0000", 2024-04-14T15:22:28.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0118651 2024-04-14T15:22:28.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:22:28.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.102518+0000", 2024-04-14T15:22:28.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7830000000000005e-05 2024-04-14T15:22:28.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:22:28.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.205327+0000", 2024-04-14T15:22:28.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10280921 2024-04-14T15:22:28.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:22:28.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.205373+0000", 2024-04-14T15:22:28.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6254999999999997e-05 2024-04-14T15:22:28.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:22:28.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.205410+0000", 2024-04-14T15:22:28.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7113000000000002e-05 2024-04-14T15:22:28.010 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:22:28.010 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:22:28.010 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:22:28.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.010 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25336.0:2 65.c 65:368a68c5:::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [writefull 0~7] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-14T15:22:28.010 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:12:59.135291+0000", 2024-04-14T15:22:28.010 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 568.79544302299996, 2024-04-14T15:22:28.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.183077902, 2024-04-14T15:22:28.010 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:22:28.011 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:22:28.011 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:22:28.011 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:22:28.011 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25336", 2024-04-14T15:22:28.011 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1239204013", 2024-04-14T15:22:28.011 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2 2024-04-14T15:22:28.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.011 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:22:28.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:22:28.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.135291+0000", 2024-04-14T15:22:28.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:22:28.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:22:28.012 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.135291+0000", 2024-04-14T15:22:28.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:22:28.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.012 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.012 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:22:28.012 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.135293+0000", 2024-04-14T15:22:28.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6390000000000001e-06 2024-04-14T15:22:28.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.012 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.012 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:22:28.012 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.135305+0000", 2024-04-14T15:22:28.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1826e-05 2024-04-14T15:22:28.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.012 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.012 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:22:28.012 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.159193+0000", 2024-04-14T15:22:28.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.023888105999999999 2024-04-14T15:22:28.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:22:28.013 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.159208+0000", 2024-04-14T15:22:28.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4513e-05 2024-04-14T15:22:28.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:22:28.013 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.241684+0000", 2024-04-14T15:22:28.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.082476771000000004 2024-04-14T15:22:28.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:22:28.013 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.241693+0000", 2024-04-14T15:22:28.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7229999999999993e-06 2024-04-14T15:22:28.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:22:28.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.273020+0000", 2024-04-14T15:22:28.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.031326838 2024-04-14T15:22:28.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:22:28.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.273177+0000", 2024-04-14T15:22:28.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015696899999999999 2024-04-14T15:22:28.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,7", 2024-04-14T15:22:28.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.273346+0000", 2024-04-14T15:22:28.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016890699999999999 2024-04-14T15:22:28.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:22:28.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.288442+0000", 2024-04-14T15:22:28.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015095844000000001 2024-04-14T15:22:28.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:22:28.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.289944+0000", 2024-04-14T15:22:28.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00150186 2024-04-14T15:22:28.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:22:28.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.318262+0000", 2024-04-14T15:22:28.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.028318073999999999 2024-04-14T15:22:28.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.016 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:22:28.016 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.318324+0000", 2024-04-14T15:22:28.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2242000000000005e-05 2024-04-14T15:22:28.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.016 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:22:28.016 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.318369+0000", 2024-04-14T15:22:28.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4589999999999998e-05 2024-04-14T15:22:28.016 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:22:28.016 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:22:28.016 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:22:28.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.017 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24853.0:2069 16.e 16:70021a00:test-rados-api-smithi090-93601-17::big:head [writefull 0~25165828] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-14T15:22:28.017 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:12:59.365160+0000", 2024-04-14T15:22:28.017 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 568.56557414700001, 2024-04-14T15:22:28.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.069670120000000002, 2024-04-14T15:22:28.017 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:22:28.017 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:22:28.017 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:22:28.017 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:22:28.017 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24853", 2024-04-14T15:22:28.017 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3851923713", 2024-04-14T15:22:28.017 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2069 2024-04-14T15:22:28.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.018 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:22:28.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:22:28.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.365160+0000", 2024-04-14T15:22:28.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:22:28.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:22:28.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.365160+0000", 2024-04-14T15:22:28.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:22:28.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.019 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:22:28.019 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.365162+0000", 2024-04-14T15:22:28.019 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1009999999999999e-06 2024-04-14T15:22:28.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.019 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:22:28.019 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.377376+0000", 2024-04-14T15:22:28.019 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012214849 2024-04-14T15:22:28.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.019 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:22:28.019 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.377400+0000", 2024-04-14T15:22:28.019 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3288000000000001e-05 2024-04-14T15:22:28.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.020 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:22:28.020 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.377410+0000", 2024-04-14T15:22:28.020 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0678e-05 2024-04-14T15:22:28.020 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.020 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:22:28.020 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.377478+0000", 2024-04-14T15:22:28.020 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7850999999999998e-05 2024-04-14T15:22:28.020 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.020 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:22:28.020 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.377615+0000", 2024-04-14T15:22:28.020 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000136907 2024-04-14T15:22:28.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2024-04-14T15:22:28.021 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.377710+0000", 2024-04-14T15:22:28.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5137999999999996e-05 2024-04-14T15:22:28.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:22:28.021 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.428359+0000", 2024-04-14T15:22:28.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.050648276999999998 2024-04-14T15:22:28.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:22:28.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.434675+0000", 2024-04-14T15:22:28.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0063159740000000002 2024-04-14T15:22:28.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.022 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:22:28.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.434736+0000", 2024-04-14T15:22:28.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.101e-05 2024-04-14T15:22:28.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.022 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:22:28.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.434787+0000", 2024-04-14T15:22:28.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1093e-05 2024-04-14T15:22:28.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:22:28.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:12:59.434830+0000", 2024-04-14T15:22:28.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2954000000000002e-05 2024-04-14T15:22:28.023 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:22:28.023 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:22:28.023 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:22:28.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.023 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25267.0:1 69.3s0 69:d43101ce:test-rados-api-smithi090-93864-8::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-04-14T15:22:28.023 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:13:00.272277+0000", 2024-04-14T15:22:28.023 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 567.658456581, 2024-04-14T15:22:28.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.68897849099999997, 2024-04-14T15:22:28.024 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:22:28.024 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:22:28.024 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:22:28.024 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:22:28.024 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25267", 2024-04-14T15:22:28.024 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1324117929", 2024-04-14T15:22:28.024 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1 2024-04-14T15:22:28.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.024 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:22:28.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.024 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:22:28.024 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.272277+0000", 2024-04-14T15:22:28.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:22:28.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:22:28.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.272277+0000", 2024-04-14T15:22:28.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:22:28.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:22:28.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.272280+0000", 2024-04-14T15:22:28.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6060000000000001e-06 2024-04-14T15:22:28.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:22:28.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.272289+0000", 2024-04-14T15:22:28.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3619999999999995e-06 2024-04-14T15:22:28.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:22:28.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.272299+0000", 2024-04-14T15:22:28.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6579999999999997e-06 2024-04-14T15:22:28.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:22:28.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.272309+0000", 2024-04-14T15:22:28.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0550999999999999e-05 2024-04-14T15:22:28.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:22:28.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.294275+0000", 2024-04-14T15:22:28.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.021965506999999999 2024-04-14T15:22:28.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:22:28.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.294279+0000", 2024-04-14T15:22:28.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8089999999999999e-06 2024-04-14T15:22:28.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:22:28.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.938589+0000", 2024-04-14T15:22:28.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.64431073299999997 2024-04-14T15:22:28.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:22:28.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.941384+0000", 2024-04-14T15:22:28.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002795021 2024-04-14T15:22:28.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2024-04-14T15:22:28.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.941783+0000", 2024-04-14T15:22:28.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039845599999999999 2024-04-14T15:22:28.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2024-04-14T15:22:28.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.955583+0000", 2024-04-14T15:22:28.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013800415 2024-04-14T15:22:28.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:22:28.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.961184+0000", 2024-04-14T15:22:28.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0056007729999999999 2024-04-14T15:22:28.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:22:28.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:00.961256+0000", 2024-04-14T15:22:28.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1600000000000006e-05 2024-04-14T15:22:28.029 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:22:28.029 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:22:28.029 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:22:28.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.030 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.41473.0:2 150.1b 150:d96a01e0:::testobj:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-04-14T15:22:28.030 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:13:24.876020+0000", 2024-04-14T15:22:28.030 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 543.05471391699996, 2024-04-14T15:22:28.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.107669923, 2024-04-14T15:22:28.030 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:22:28.030 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:22:28.030 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:22:28.030 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:22:28.030 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.41473", 2024-04-14T15:22:28.030 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1386986747", 2024-04-14T15:22:28.030 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2 2024-04-14T15:22:28.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.030 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:22:28.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:22:28.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:24.876020+0000", 2024-04-14T15:22:28.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:22:28.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:22:28.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:24.876020+0000", 2024-04-14T15:22:28.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:22:28.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:22:28.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:24.876022+0000", 2024-04-14T15:22:28.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8840000000000001e-06 2024-04-14T15:22:28.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:22:28.032 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:24.876030+0000", 2024-04-14T15:22:28.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9740000000000002e-06 2024-04-14T15:22:28.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:22:28.032 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:24.876041+0000", 2024-04-14T15:22:28.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.112e-05 2024-04-14T15:22:28.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:22:28.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:24.876052+0000", 2024-04-14T15:22:28.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0798999999999999e-05 2024-04-14T15:22:28.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:22:28.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:24.877471+0000", 2024-04-14T15:22:28.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0014197459999999999 2024-04-14T15:22:28.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:22:28.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:24.877480+0000", 2024-04-14T15:22:28.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.02e-06 2024-04-14T15:22:28.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:22:28.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:25.965298+0000", 2024-04-14T15:22:28.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0878175809999999 2024-04-14T15:22:28.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:22:28.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:25.965416+0000", 2024-04-14T15:22:28.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000118453 2024-04-14T15:22:28.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2024-04-14T15:22:28.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:25.965546+0000", 2024-04-14T15:22:28.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012987200000000001 2024-04-14T15:22:28.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:22:28.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:25.969566+0000", 2024-04-14T15:22:28.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040198910000000003 2024-04-14T15:22:28.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:22:28.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:25.980352+0000", 2024-04-14T15:22:28.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010785846 2024-04-14T15:22:28.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:22:28.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:25.983575+0000", 2024-04-14T15:22:28.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032233140000000001 2024-04-14T15:22:28.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:22:28.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:25.983636+0000", 2024-04-14T15:22:28.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0291000000000003e-05 2024-04-14T15:22:28.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:22:28.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:25.983690+0000", 2024-04-14T15:22:28.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4132000000000003e-05 2024-04-14T15:22:28.036 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:22:28.036 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:22:28.036 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:22:28.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.036 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25684.0:248 110.1c 110:3d310c86:test-rados-api-smithi090-93652-12::bar:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-04-14T15:22:28.037 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:13:26.901575+0000", 2024-04-14T15:22:28.037 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 541.02915902899997, 2024-04-14T15:22:28.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.096119394999999996, 2024-04-14T15:22:28.037 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:22:28.037 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:22:28.037 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:22:28.037 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:22:28.037 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25684", 2024-04-14T15:22:28.037 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1109245683", 2024-04-14T15:22:28.037 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 248 2024-04-14T15:22:28.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.037 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:22:28.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:22:28.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.901575+0000", 2024-04-14T15:22:28.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:22:28.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:22:28.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.901575+0000", 2024-04-14T15:22:28.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:22:28.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:22:28.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.901576+0000", 2024-04-14T15:22:28.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.542e-06 2024-04-14T15:22:28.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:22:28.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.901583+0000", 2024-04-14T15:22:28.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6869999999999997e-06 2024-04-14T15:22:28.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:22:28.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.901590+0000", 2024-04-14T15:22:28.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6199999999999999e-06 2024-04-14T15:22:28.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:22:28.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.901601+0000", 2024-04-14T15:22:28.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0037999999999999e-05 2024-04-14T15:22:28.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:22:28.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.903042+0000", 2024-04-14T15:22:28.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001440993 2024-04-14T15:22:28.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.040 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:22:28.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.903165+0000", 2024-04-14T15:22:28.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000123201 2024-04-14T15:22:28.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.040 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,3", 2024-04-14T15:22:28.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.903317+0000", 2024-04-14T15:22:28.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015206799999999999 2024-04-14T15:22:28.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.040 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:22:28.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.910951+0000", 2024-04-14T15:22:28.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0076343510000000002 2024-04-14T15:22:28.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:22:28.041 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.974802+0000", 2024-04-14T15:22:28.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.063850738000000004 2024-04-14T15:22:28.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:22:28.041 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.997592+0000", 2024-04-14T15:22:28.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022790299 2024-04-14T15:22:28.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:22:28.041 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.997648+0000", 2024-04-14T15:22:28.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5745999999999998e-05 2024-04-14T15:22:28.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:22:28.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:26.997694+0000", 2024-04-14T15:22:28.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6112000000000002e-05 2024-04-14T15:22:28.042 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:22:28.042 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:22:28.042 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:22:28.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.042 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24853.0:2078 16.1 16:81eb58c7:test-rados-api-smithi090-93601-17::big.copy:head [copy-from ver 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-14T15:22:28.042 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:13:29.666343+0000", 2024-04-14T15:22:28.042 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 538.26439027699996, 2024-04-14T15:22:28.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10434288999999999, 2024-04-14T15:22:28.042 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:22:28.042 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:22:28.042 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:22:28.043 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:22:28.043 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24853", 2024-04-14T15:22:28.043 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3851923713", 2024-04-14T15:22:28.043 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2078 2024-04-14T15:22:28.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.043 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:22:28.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:22:28.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.666343+0000", 2024-04-14T15:22:28.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:22:28.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:22:28.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.666343+0000", 2024-04-14T15:22:28.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:22:28.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:22:28.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.666347+0000", 2024-04-14T15:22:28.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0500000000000002e-06 2024-04-14T15:22:28.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:22:28.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.666363+0000", 2024-04-14T15:22:28.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5131e-05 2024-04-14T15:22:28.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:22:28.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.666388+0000", 2024-04-14T15:22:28.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5803e-05 2024-04-14T15:22:28.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:22:28.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.666403+0000", 2024-04-14T15:22:28.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5013e-05 2024-04-14T15:22:28.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:22:28.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.666497+0000", 2024-04-14T15:22:28.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3468999999999996e-05 2024-04-14T15:22:28.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:22:28.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.666664+0000", 2024-04-14T15:22:28.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016748699999999999 2024-04-14T15:22:28.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,6", 2024-04-14T15:22:28.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.755301+0000", 2024-04-14T15:22:28.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.088636912999999998 2024-04-14T15:22:28.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:22:28.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.770454+0000", 2024-04-14T15:22:28.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015152528 2024-04-14T15:22:28.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:22:28.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.770527+0000", 2024-04-14T15:22:28.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3238999999999995e-05 2024-04-14T15:22:28.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:22:28.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.770597+0000", 2024-04-14T15:22:28.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0469999999999994e-05 2024-04-14T15:22:28.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:22:28.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.770649+0000", 2024-04-14T15:22:28.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1309e-05 2024-04-14T15:22:28.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:22:28.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.770686+0000", 2024-04-14T15:22:28.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7478000000000003e-05 2024-04-14T15:22:28.047 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:22:28.047 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:22:28.047 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:22:28.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.048 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.26497.0:57 112.1a 112:5978e4a5:test-rados-api-smithi090-93724-12::foo:head [write 0~128] snapc b=[b] ondisk+write+ordersnap+known_if_redirected+supports_pool_eio e142)", 2024-04-14T15:22:28.048 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:13:29.880875+0000", 2024-04-14T15:22:28.048 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 538.04985868599999, 2024-04-14T15:22:28.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13447140699999999, 2024-04-14T15:22:28.048 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:22:28.048 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:22:28.048 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:22:28.048 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:22:28.048 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.26497", 2024-04-14T15:22:28.048 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3962832909", 2024-04-14T15:22:28.048 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 57 2024-04-14T15:22:28.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.048 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:22:28.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:22:28.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.880875+0000", 2024-04-14T15:22:28.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:22:28.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:22:28.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.880875+0000", 2024-04-14T15:22:28.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:22:28.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:22:28.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.880877+0000", 2024-04-14T15:22:28.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.776e-06 2024-04-14T15:22:28.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:22:28.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.880885+0000", 2024-04-14T15:22:28.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1130000000000001e-06 2024-04-14T15:22:28.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:22:28.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.880896+0000", 2024-04-14T15:22:28.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0868e-05 2024-04-14T15:22:28.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:22:28.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.880907+0000", 2024-04-14T15:22:28.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1119e-05 2024-04-14T15:22:28.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:22:28.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.880980+0000", 2024-04-14T15:22:28.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3484000000000003e-05 2024-04-14T15:22:28.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2024-04-14T15:22:28.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.880996+0000", 2024-04-14T15:22:28.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6120000000000002e-05 2024-04-14T15:22:28.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:22:28.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.900278+0000", 2024-04-14T15:22:28.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019281436999999998 2024-04-14T15:22:28.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:22:28.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.900405+0000", 2024-04-14T15:22:28.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000127463 2024-04-14T15:22:28.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,6", 2024-04-14T15:22:28.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.900555+0000", 2024-04-14T15:22:28.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014966799999999999 2024-04-14T15:22:28.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:22:28.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.928367+0000", 2024-04-14T15:22:28.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02781177 2024-04-14T15:22:28.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:22:28.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.930228+0000", 2024-04-14T15:22:28.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0018616939999999999 2024-04-14T15:22:28.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:22:28.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:30.015227+0000", 2024-04-14T15:22:28.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.084998517999999995 2024-04-14T15:22:28.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:22:28.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:30.015290+0000", 2024-04-14T15:22:28.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3338999999999998e-05 2024-04-14T15:22:28.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:22:28.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:30.015346+0000", 2024-04-14T15:22:28.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6038000000000001e-05 2024-04-14T15:22:28.054 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:22:28.054 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:22:28.054 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:22:28.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.054 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24853.0:2080 16.e 16:770790e7:test-rados-api-smithi090-93601-17::big3.copy:head [copy-from ver 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-14T15:22:28.054 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:13:29.886267+0000", 2024-04-14T15:22:28.054 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 538.04446655699996, 2024-04-14T15:22:28.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.25823401099999999, 2024-04-14T15:22:28.054 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:22:28.054 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:22:28.054 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:22:28.054 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:22:28.054 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24853", 2024-04-14T15:22:28.055 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3851923713", 2024-04-14T15:22:28.055 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2080 2024-04-14T15:22:28.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.055 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:22:28.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:22:28.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.886267+0000", 2024-04-14T15:22:28.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:22:28.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:22:28.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.886267+0000", 2024-04-14T15:22:28.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:22:28.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.056 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:22:28.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.886269+0000", 2024-04-14T15:22:28.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0820000000000001e-06 2024-04-14T15:22:28.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.056 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:22:28.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.886278+0000", 2024-04-14T15:22:28.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0550000000000005e-06 2024-04-14T15:22:28.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.056 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:22:28.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.890009+0000", 2024-04-14T15:22:28.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037304180000000001 2024-04-14T15:22:28.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:22:28.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.890026+0000", 2024-04-14T15:22:28.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7244999999999998e-05 2024-04-14T15:22:28.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:22:28.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.890095+0000", 2024-04-14T15:22:28.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8621000000000006e-05 2024-04-14T15:22:28.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:22:28.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:29.890254+0000", 2024-04-14T15:22:28.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015991900000000001 2024-04-14T15:22:28.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2024-04-14T15:22:28.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:30.073652+0000", 2024-04-14T15:22:28.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.183397687 2024-04-14T15:22:28.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:22:28.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:30.085273+0000", 2024-04-14T15:22:28.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011620966 2024-04-14T15:22:28.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:22:28.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:30.088107+0000", 2024-04-14T15:22:28.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028342459999999999 2024-04-14T15:22:28.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:22:28.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:30.144398+0000", 2024-04-14T15:22:28.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.056290426999999997 2024-04-14T15:22:28.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:22:28.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:30.144458+0000", 2024-04-14T15:22:28.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0164999999999997e-05 2024-04-14T15:22:28.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:22:28.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:13:30.144501+0000", 2024-04-14T15:22:28.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3180000000000003e-05 2024-04-14T15:22:28.059 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:22:28.059 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:22:28.060 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:22:28.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.060 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.46400.0:2 202.11 202:8de73fe4:test-rados-api-smithi090-93601-26::foo:head [writefull 0~8388608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e184)", 2024-04-14T15:22:28.060 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:14:11.840817+0000", 2024-04-14T15:22:28.060 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 496.089916722, 2024-04-14T15:22:28.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.052008092999999998, 2024-04-14T15:22:28.060 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:22:28.060 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:22:28.060 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:22:28.060 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:22:28.060 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.46400", 2024-04-14T15:22:28.060 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/844371530", 2024-04-14T15:22:28.060 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2 2024-04-14T15:22:28.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.061 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:22:28.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:22:28.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:11.840817+0000", 2024-04-14T15:22:28.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:22:28.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:22:28.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:11.840817+0000", 2024-04-14T15:22:28.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:22:28.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:22:28.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:11.840821+0000", 2024-04-14T15:22:28.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8920000000000002e-06 2024-04-14T15:22:28.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.062 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:22:28.062 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:11.847066+0000", 2024-04-14T15:22:28.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0062451310000000001 2024-04-14T15:22:28.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.062 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:22:28.062 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:11.847083+0000", 2024-04-14T15:22:28.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7419e-05 2024-04-14T15:22:28.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.062 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:22:28.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:11.847095+0000", 2024-04-14T15:22:28.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1341e-05 2024-04-14T15:22:28.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:22:28.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:11.848854+0000", 2024-04-14T15:22:28.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001759755 2024-04-14T15:22:28.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:22:28.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:11.848974+0000", 2024-04-14T15:22:28.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012000100000000001 2024-04-14T15:22:28.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 5,7", 2024-04-14T15:22:28.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:11.849110+0000", 2024-04-14T15:22:28.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013593399999999999 2024-04-14T15:22:28.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:22:28.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:11.866374+0000", 2024-04-14T15:22:28.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017263341000000001 2024-04-14T15:22:28.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:22:28.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:11.892695+0000", 2024-04-14T15:22:28.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.026321297 2024-04-14T15:22:28.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:22:28.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:11.892733+0000", 2024-04-14T15:22:28.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8220000000000003e-05 2024-04-14T15:22:28.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:22:28.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:11.892771+0000", 2024-04-14T15:22:28.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8169e-05 2024-04-14T15:22:28.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:22:28.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:11.892825+0000", 2024-04-14T15:22:28.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3593e-05 2024-04-14T15:22:28.065 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:22:28.065 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:22:28.065 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:22:28.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.066 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.36564.0:1 217.2s0 217:47f32b6d:test-rados-api-smithi090-93601-37::foo:head [write 0~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-04-14T15:22:28.066 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:14:27.854266+0000", 2024-04-14T15:22:28.066 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 480.07646805100001, 2024-04-14T15:22:28.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.007350245, 2024-04-14T15:22:28.066 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:22:28.066 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:22:28.066 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:22:28.066 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:22:28.066 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.36564", 2024-04-14T15:22:28.066 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3214396702", 2024-04-14T15:22:28.066 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1 2024-04-14T15:22:28.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.066 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:22:28.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:22:28.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:27.854266+0000", 2024-04-14T15:22:28.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:22:28.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:22:28.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:27.854266+0000", 2024-04-14T15:22:28.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:22:28.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:22:28.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:27.854268+0000", 2024-04-14T15:22:28.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1090000000000001e-06 2024-04-14T15:22:28.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:22:28.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:27.854277+0000", 2024-04-14T15:22:28.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9639999999999992e-06 2024-04-14T15:22:28.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:22:28.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:27.854288+0000", 2024-04-14T15:22:28.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1589e-05 2024-04-14T15:22:28.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:22:28.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:27.854297+0000", 2024-04-14T15:22:28.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1370000000000008e-06 2024-04-14T15:22:28.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:22:28.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:27.854384+0000", 2024-04-14T15:22:28.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6551999999999994e-05 2024-04-14T15:22:28.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:22:28.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:27.854388+0000", 2024-04-14T15:22:28.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6559999999999998e-06 2024-04-14T15:22:28.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:22:28.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:28.850918+0000", 2024-04-14T15:22:28.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99653077300000004 2024-04-14T15:22:28.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:22:28.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:28.851047+0000", 2024-04-14T15:22:28.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000128508 2024-04-14T15:22:28.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2024-04-14T15:22:28.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:28.851452+0000", 2024-04-14T15:22:28.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00040467600000000003 2024-04-14T15:22:28.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2024-04-14T15:22:28.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:28.852774+0000", 2024-04-14T15:22:28.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0013221179999999999 2024-04-14T15:22:28.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:22:28.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:28.861533+0000", 2024-04-14T15:22:28.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0087598079999999991 2024-04-14T15:22:28.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:22:28.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:28.861616+0000", 2024-04-14T15:22:28.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2354999999999993e-05 2024-04-14T15:22:28.071 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:22:28.071 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:22:28.071 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:22:28.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.071 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.48424.0:1 219.6s0 219:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-04-14T15:22:28.072 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:14:28.846506+0000", 2024-04-14T15:22:28.072 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 479.08422811999998, 2024-04-14T15:22:28.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0258293709999999, 2024-04-14T15:22:28.072 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:22:28.072 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:22:28.072 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:22:28.072 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:22:28.072 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.48424", 2024-04-14T15:22:28.072 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2359757960", 2024-04-14T15:22:28.072 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1 2024-04-14T15:22:28.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.072 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:22:28.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:22:28.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:28.846506+0000", 2024-04-14T15:22:28.073 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:22:28.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:22:28.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.074 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:22:28.074 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:28.846506+0000", 2024-04-14T15:22:28.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:22:28.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.074 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:22:28.074 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:28.846507+0000", 2024-04-14T15:22:28.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6160000000000001e-06 2024-04-14T15:22:28.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:22:28.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:28.846512+0000", 2024-04-14T15:22:28.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.22e-06 2024-04-14T15:22:28.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:22:28.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:28.846521+0000", 2024-04-14T15:22:28.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0089999999999996e-06 2024-04-14T15:22:28.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:22:28.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:28.846532+0000", 2024-04-14T15:22:28.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0229e-05 2024-04-14T15:22:28.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:22:28.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:28.846598+0000", 2024-04-14T15:22:28.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6029000000000001e-05 2024-04-14T15:22:28.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:22:28.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:28.846601+0000", 2024-04-14T15:22:28.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2349999999999999e-06 2024-04-14T15:22:28.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:22:28.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:29.869910+0000", 2024-04-14T15:22:28.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.023308635 2024-04-14T15:22:28.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:22:28.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:29.870033+0000", 2024-04-14T15:22:28.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000123795 2024-04-14T15:22:28.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2024-04-14T15:22:28.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:29.870480+0000", 2024-04-14T15:22:28.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00044620199999999998 2024-04-14T15:22:28.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2024-04-14T15:22:28.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:29.871280+0000", 2024-04-14T15:22:28.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00080055000000000002 2024-04-14T15:22:28.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:22:28.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:29.872259+0000", 2024-04-14T15:22:28.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00097881199999999991 2024-04-14T15:22:28.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:22:28.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:29.872335+0000", 2024-04-14T15:22:28.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6038999999999995e-05 2024-04-14T15:22:28.078 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:22:28.078 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:22:28.078 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:22:28.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.078 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.48793.0:1 269.0s0 269:1e0071ee:::oid_RoundTripSparseReadPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-04-14T15:22:28.079 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:16:05.868323+0000", 2024-04-14T15:22:28.079 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 382.06241106300001, 2024-04-14T15:22:28.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.007858315, 2024-04-14T15:22:28.079 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:22:28.079 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:22:28.079 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:22:28.079 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:22:28.079 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.48793", 2024-04-14T15:22:28.079 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3028951022", 2024-04-14T15:22:28.079 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1 2024-04-14T15:22:28.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.079 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:22:28.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:22:28.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:16:05.868323+0000", 2024-04-14T15:22:28.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:22:28.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:22:28.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:16:05.868323+0000", 2024-04-14T15:22:28.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:22:28.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:22:28.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:16:05.868324+0000", 2024-04-14T15:22:28.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6899999999999999e-06 2024-04-14T15:22:28.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:22:28.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:16:05.868332+0000", 2024-04-14T15:22:28.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5900000000000002e-06 2024-04-14T15:22:28.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:22:28.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:16:05.868341+0000", 2024-04-14T15:22:28.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5869999999999999e-06 2024-04-14T15:22:28.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:22:28.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:16:05.868352+0000", 2024-04-14T15:22:28.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0900000000000001e-05 2024-04-14T15:22:28.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:22:28.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:16:05.871436+0000", 2024-04-14T15:22:28.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00308348 2024-04-14T15:22:28.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:22:28.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:16:05.871440+0000", 2024-04-14T15:22:28.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3970000000000004e-06 2024-04-14T15:22:28.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:22:28.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:16:06.871794+0000", 2024-04-14T15:22:28.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.00035395 2024-04-14T15:22:28.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:22:28.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:16:06.871908+0000", 2024-04-14T15:22:28.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113795 2024-04-14T15:22:28.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2024-04-14T15:22:28.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:16:06.872654+0000", 2024-04-14T15:22:28.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00074600299999999997 2024-04-14T15:22:28.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2024-04-14T15:22:28.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:16:06.875274+0000", 2024-04-14T15:22:28.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026198860000000001 2024-04-14T15:22:28.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:22:28.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:16:06.876114+0000", 2024-04-14T15:22:28.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00084004299999999995 2024-04-14T15:22:28.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:22:28.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:16:06.876181+0000", 2024-04-14T15:22:28.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6994000000000003e-05 2024-04-14T15:22:28.084 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:22:28.084 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:22:28.084 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:22:28.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.084 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.37110.0:1 284.2s0 284:482bd08b:::oid_SimpleStatPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-04-14T15:22:28.084 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:17:04.907943+0000", 2024-04-14T15:22:28.085 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 323.02279041000003, 2024-04-14T15:22:28.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8548804969999999, 2024-04-14T15:22:28.085 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:22:28.085 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:22:28.085 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:22:28.085 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:22:28.085 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.37110", 2024-04-14T15:22:28.085 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/784308691", 2024-04-14T15:22:28.085 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1 2024-04-14T15:22:28.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.085 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:22:28.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.085 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:22:28.085 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:04.907943+0000", 2024-04-14T15:22:28.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:22:28.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:22:28.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:04.907943+0000", 2024-04-14T15:22:28.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:22:28.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:22:28.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:04.907945+0000", 2024-04-14T15:22:28.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.629e-06 2024-04-14T15:22:28.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:22:28.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:04.907952+0000", 2024-04-14T15:22:28.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1420000000000004e-06 2024-04-14T15:22:28.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:22:28.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:04.907962+0000", 2024-04-14T15:22:28.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.012e-05 2024-04-14T15:22:28.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:22:28.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:04.907973+0000", 2024-04-14T15:22:28.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0541e-05 2024-04-14T15:22:28.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:22:28.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:04.908034+0000", 2024-04-14T15:22:28.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1426000000000004e-05 2024-04-14T15:22:28.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:22:28.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:04.908038+0000", 2024-04-14T15:22:28.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5200000000000002e-06 2024-04-14T15:22:28.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:22:28.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:06.760349+0000", 2024-04-14T15:22:28.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8523112479999999 2024-04-14T15:22:28.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:22:28.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:06.760477+0000", 2024-04-14T15:22:28.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000128052 2024-04-14T15:22:28.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2024-04-14T15:22:28.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:06.760924+0000", 2024-04-14T15:22:28.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00044702499999999998 2024-04-14T15:22:28.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2024-04-14T15:22:28.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:06.761350+0000", 2024-04-14T15:22:28.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00042607799999999998 2024-04-14T15:22:28.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:22:28.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:06.762752+0000", 2024-04-14T15:22:28.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0014017599999999999 2024-04-14T15:22:28.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:22:28.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:06.762824+0000", 2024-04-14T15:22:28.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1956e-05 2024-04-14T15:22:28.090 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:22:28.090 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:22:28.090 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:22:28.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.090 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.47114.0:1 288.4s0 288:311cbe94:::oid_ExecuteClassPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-04-14T15:22:28.090 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:17:25.890728+0000", 2024-04-14T15:22:28.090 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 302.04000613599999, 2024-04-14T15:22:28.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0120479069999999, 2024-04-14T15:22:28.090 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:22:28.090 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:22:28.091 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:22:28.091 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:22:28.091 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.47114", 2024-04-14T15:22:28.091 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1312471356", 2024-04-14T15:22:28.091 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1 2024-04-14T15:22:28.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.091 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:22:28.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:22:28.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:25.890728+0000", 2024-04-14T15:22:28.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:22:28.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:22:28.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:25.890728+0000", 2024-04-14T15:22:28.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:22:28.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:22:28.092 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:25.890729+0000", 2024-04-14T15:22:28.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6029999999999999e-06 2024-04-14T15:22:28.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:22:28.092 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:25.890736+0000", 2024-04-14T15:22:28.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8360000000000003e-06 2024-04-14T15:22:28.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:22:28.092 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:25.890746+0000", 2024-04-14T15:22:28.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7729999999999994e-06 2024-04-14T15:22:28.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:22:28.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:25.890756+0000", 2024-04-14T15:22:28.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9410000000000002e-06 2024-04-14T15:22:28.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:22:28.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:25.890812+0000", 2024-04-14T15:22:28.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6164e-05 2024-04-14T15:22:28.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:22:28.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:25.890816+0000", 2024-04-14T15:22:28.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.856e-06 2024-04-14T15:22:28.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:22:28.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:26.901241+0000", 2024-04-14T15:22:28.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.010425718 2024-04-14T15:22:28.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:22:28.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:26.901363+0000", 2024-04-14T15:22:28.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000121956 2024-04-14T15:22:28.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2024-04-14T15:22:28.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:26.901787+0000", 2024-04-14T15:22:28.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00042398300000000001 2024-04-14T15:22:28.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2024-04-14T15:22:28.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:26.902091+0000", 2024-04-14T15:22:28.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000303689 2024-04-14T15:22:28.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:22:28.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:26.902721+0000", 2024-04-14T15:22:28.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00062978299999999997 2024-04-14T15:22:28.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:22:28.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:22:28.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:22:28.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:17:26.902775+0000", 2024-04-14T15:22:28.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4605000000000003e-05 2024-04-14T15:22:28.096 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:22:28.096 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:22:28.096 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:22:28.096 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:22:28.096 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:22:28.096 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-14T15:22:28.099 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:28.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:28.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:22:28.200 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:22:28.226 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:28.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:28.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:22:28.328 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:28.352 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:28.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:28.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:28.454 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:22:28.482 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:28.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:28.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:22:28.583 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:22:28.607 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:28.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:28.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:22:28.709 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:22:28.737 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:28.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:28.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:22:28.839 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:22:28.866 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:28.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:28.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:22:28.968 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:22:29.000 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:29.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:29.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:22:29.102 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:22:29.125 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:29.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:29.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:22:29.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:28 smithi090 ceph-mon[30399]: pgmap v897: 186 pgs: 8 unknown, 3 active+remapped+backfill_toofull, 175 active+clean; 596 KiB data, 522 MiB used, 625 GiB / 626 GiB avail; 2/597 objects misplaced (0.335%) 2024-04-14T15:22:29.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:28 smithi090 ceph-mon[30399]: osdmap e612: 8 total, 8 up, 7 in 2024-04-14T15:22:29.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:28 smithi090 ceph-mon[30399]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:22:29.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:28 smithi090 ceph-mon[41670]: pgmap v897: 186 pgs: 8 unknown, 3 active+remapped+backfill_toofull, 175 active+clean; 596 KiB data, 522 MiB used, 625 GiB / 626 GiB avail; 2/597 objects misplaced (0.335%) 2024-04-14T15:22:29.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:28 smithi090 ceph-mon[41670]: osdmap e612: 8 total, 8 up, 7 in 2024-04-14T15:22:29.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:28 smithi090 ceph-mon[41670]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:22:29.227 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:22:29.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:29.252 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:29.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:22:29.353 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:22:29.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:28 smithi092 ceph-mon[33966]: pgmap v897: 186 pgs: 8 unknown, 3 active+remapped+backfill_toofull, 175 active+clean; 596 KiB data, 522 MiB used, 625 GiB / 626 GiB avail; 2/597 objects misplaced (0.335%) 2024-04-14T15:22:29.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:28 smithi092 ceph-mon[33966]: osdmap e612: 8 total, 8 up, 7 in 2024-04-14T15:22:29.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:28 smithi092 ceph-mon[33966]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:22:29.379 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-14T15:22:29.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:29.380 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:29.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:22:29.481 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:22:29.510 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:29.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:29.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:22:29.611 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:22:29.635 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:29.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:29.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:22:29.737 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:22:29.760 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:29.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:29.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:22:29.861 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:22:29.885 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:29.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:29.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:22:29.987 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:22:30.012 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:29 smithi090 ceph-mon[41670]: osdmap e613: 8 total, 8 up, 7 in 2024-04-14T15:22:30.012 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:29 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1465271679' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:22:30.012 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:29 smithi090 ceph-mon[30399]: osdmap e613: 8 total, 8 up, 7 in 2024-04-14T15:22:30.012 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:29 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1465271679' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:22:30.014 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:30.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:30.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:22:30.116 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:22:30.143 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:30.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:30.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:22:30.218 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:22:30.245 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:22:30.273 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:30.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:30.274 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:22:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:29 smithi092 ceph-mon[33966]: osdmap e613: 8 total, 8 up, 7 in 2024-04-14T15:22:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:29 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1465271679' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:22:30.375 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:22:30.399 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:30.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:30.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:22:30.501 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:22:30.532 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:30.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:30.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:22:30.634 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:22:30.662 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:30.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:30.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:22:30.764 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:22:30.791 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:30.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:30.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:22:30.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.863+0000 7f839318e700 1 -- 172.21.15.90:0/3675176852 >> v1:172.21.15.90:6790/0 conn(0x7f838c10f330 legacy=0x7f838c1117f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:30.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.863+0000 7f839318e700 1 -- 172.21.15.90:0/3675176852 shutdown_connections 2024-04-14T15:22:30.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.863+0000 7f839318e700 1 -- 172.21.15.90:0/3675176852 >> 172.21.15.90:0/3675176852 conn(0x7f838c0fe3c0 msgr2=0x7f838c1007e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:22:30.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.863+0000 7f839318e700 1 -- 172.21.15.90:0/3675176852 shutdown_connections 2024-04-14T15:22:30.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.863+0000 7f839318e700 1 -- 172.21.15.90:0/3675176852 wait complete. 2024-04-14T15:22:30.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.863+0000 7f839318e700 1 librados: starting msgr at 2024-04-14T15:22:30.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.863+0000 7f839318e700 1 librados: starting objecter 2024-04-14T15:22:30.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.863+0000 7f839318e700 1 Processor -- start 2024-04-14T15:22:30.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.863+0000 7f839318e700 1 -- start start 2024-04-14T15:22:30.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.863+0000 7f839318e700 1 librados: setting wanted keys 2024-04-14T15:22:30.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.863+0000 7f839318e700 1 librados: calling monclient init 2024-04-14T15:22:30.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.864+0000 7f839318e700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f838c1a79e0 con 0x7f838c1088b0 2024-04-14T15:22:30.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.864+0000 7f839318e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f838c1a8c00 con 0x7f838c10b760 2024-04-14T15:22:30.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.864+0000 7f839318e700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f838c1a9e40 con 0x7f838c10f330 2024-04-14T15:22:30.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.864+0000 7f8390f2a700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f838c1088b0 0x7f838c10eb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:51332/0 (socket says 172.21.15.90:51332) 2024-04-14T15:22:30.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.864+0000 7f8390f2a700 1 -- 172.21.15.90:0/3957578272 learned_addr learned my addr 172.21.15.90:0/3957578272 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:22:30.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.864+0000 7f839172b700 1 --1- 172.21.15.90:0/3957578272 >> v1:172.21.15.90:6790/0 conn(0x7f838c10f330 0x7f838c1a60a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:56146/0 (socket says 172.21.15.90:56146) 2024-04-14T15:22:30.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.864+0000 7f8389ffb700 10 client.?.objecter ms_handle_connect 0x7f838c1088b0 2024-04-14T15:22:30.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.864+0000 7f8389ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:22:30.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.864+0000 7f8389ffb700 10 client.?.objecter ms_handle_connect 0x7f838c10f330 2024-04-14T15:22:30.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.864+0000 7f8389ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:22:30.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.864+0000 7f8389ffb700 10 client.?.objecter ms_handle_connect 0x7f838c10b760 2024-04-14T15:22:30.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.864+0000 7f8389ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:22:30.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.864+0000 7f8389ffb700 1 -- 172.21.15.90:0/3957578272 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2403706518 0 0) 0x7f838c1a79e0 con 0x7f838c1088b0 2024-04-14T15:22:30.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.864+0000 7f8389ffb700 1 -- 172.21.15.90:0/3957578272 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f837c003aa0 con 0x7f838c1088b0 2024-04-14T15:22:30.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.864+0000 7f8389ffb700 1 -- 172.21.15.90:0/3957578272 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 487034638 0 0) 0x7f838c1a9e40 con 0x7f838c10f330 2024-04-14T15:22:30.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.864+0000 7f8389ffb700 1 -- 172.21.15.90:0/3957578272 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f838c1a79e0 con 0x7f838c10f330 2024-04-14T15:22:30.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.864+0000 7f8389ffb700 1 -- 172.21.15.90:0/3957578272 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1991753330 0 0) 0x7f838c1a8c00 con 0x7f838c10b760 2024-04-14T15:22:30.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.864+0000 7f8389ffb700 1 -- 172.21.15.90:0/3957578272 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f838c1a9e40 con 0x7f838c10b760 2024-04-14T15:22:30.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.864+0000 7f8389ffb700 1 -- 172.21.15.90:0/3957578272 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1529925073 0 0) 0x7f837c003aa0 con 0x7f838c1088b0 2024-04-14T15:22:30.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.864+0000 7f8389ffb700 1 -- 172.21.15.90:0/3957578272 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f838c1a8c00 con 0x7f838c1088b0 2024-04-14T15:22:30.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.864+0000 7f8389ffb700 1 -- 172.21.15.90:0/3957578272 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f8380001a30 con 0x7f838c1088b0 2024-04-14T15:22:30.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.864+0000 7f8389ffb700 1 -- 172.21.15.90:0/3957578272 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2849986337 0 0) 0x7f838c1a79e0 con 0x7f838c10f330 2024-04-14T15:22:30.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.864+0000 7f8389ffb700 1 -- 172.21.15.90:0/3957578272 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f837c003aa0 con 0x7f838c10f330 2024-04-14T15:22:30.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.864+0000 7f8389ffb700 1 -- 172.21.15.90:0/3957578272 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f83840048d0 con 0x7f838c10f330 2024-04-14T15:22:30.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.865+0000 7f8389ffb700 1 -- 172.21.15.90:0/3957578272 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1809675158 0 0) 0x7f838c1a9e40 con 0x7f838c10b760 2024-04-14T15:22:30.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.865+0000 7f8389ffb700 1 -- 172.21.15.90:0/3957578272 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f838c1a79e0 con 0x7f838c10b760 2024-04-14T15:22:30.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.865+0000 7f8389ffb700 1 -- 172.21.15.90:0/3957578272 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f83780017b0 con 0x7f838c10b760 2024-04-14T15:22:30.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.865+0000 7f8389ffb700 1 -- 172.21.15.90:0/3957578272 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1243049644 0 0) 0x7f838c1a8c00 con 0x7f838c1088b0 2024-04-14T15:22:30.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.865+0000 7f8389ffb700 1 -- 172.21.15.90:0/3957578272 >> v1:172.21.15.90:6790/0 conn(0x7f838c10f330 legacy=0x7f838c1a60a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:30.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.865+0000 7f8389ffb700 1 -- 172.21.15.90:0/3957578272 >> v1:172.21.15.92:6789/0 conn(0x7f838c10b760 legacy=0x7f838c1a27e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:30.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.865+0000 7f8389ffb700 1 -- 172.21.15.90:0/3957578272 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f838c1ab080 con 0x7f838c1088b0 2024-04-14T15:22:30.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.865+0000 7f839318e700 1 -- 172.21.15.90:0/3957578272 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f838c1a7c10 con 0x7f838c1088b0 2024-04-14T15:22:30.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.865+0000 7f8389ffb700 1 -- 172.21.15.90:0/3957578272 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f8380001ce0 con 0x7f838c1088b0 2024-04-14T15:22:30.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.865+0000 7f8389ffb700 1 -- 172.21.15.90:0/3957578272 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f8380005d80 con 0x7f838c1088b0 2024-04-14T15:22:30.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.865+0000 7f839318e700 10 client.37683.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:22:30.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.865+0000 7f839318e700 1 -- 172.21.15.90:0/3957578272 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f838c1a80d0 con 0x7f838c1088b0 2024-04-14T15:22:30.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.865+0000 7f839318e700 1 librados: init done 2024-04-14T15:22:30.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.866+0000 7f839318e700 1 -- 172.21.15.90:0/3957578272 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f838c103ff0 con 0x7f838c1088b0 2024-04-14T15:22:30.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.867+0000 7f8389ffb700 1 -- 172.21.15.90:0/3957578272 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f8380004140 con 0x7f838c1088b0 2024-04-14T15:22:30.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.867+0000 7f8389ffb700 1 -- 172.21.15.90:0/3957578272 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(614..614 src has 467..614) ==== 8785+0+0 (unknown 1784875221 0 0) 0x7f838008ace0 con 0x7f838c1088b0 2024-04-14T15:22:30.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.867+0000 7f8389ffb700 10 client.37683.objecter ms_dispatch 0x7f838c001550 osd_map(614..614 src has 467..614) 2024-04-14T15:22:30.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.867+0000 7f8389ffb700 3 client.37683.objecter handle_osd_map got epochs [614,614] > 0 2024-04-14T15:22:30.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.867+0000 7f8389ffb700 3 client.37683.objecter handle_osd_map decoding full epoch 614 2024-04-14T15:22:30.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.867+0000 7f8389ffb700 20 client.37683.objecter dump_active .. 0 homeless 2024-04-14T15:22:30.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.867+0000 7f8389ffb700 1 -- 172.21.15.90:0/3957578272 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v614) ==== 54+0+16 (unknown 948476798 0 2420000192) 0x7f838008b0e0 con 0x7f838c1088b0 2024-04-14T15:22:30.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.867+0000 7f839318e700 10 client.37683.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:22:30.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.867+0000 7f839318e700 15 client.37683.objecter _session_command_op_assign -1 1 2024-04-14T15:22:30.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.867+0000 7f839318e700 20 client.37683.objecter _get_session s=0x7f838c10d130 osd=0 3 2024-04-14T15:22:30.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.867+0000 7f839318e700 20 client.37683.objecter put_session s=0x7f838c10d130 osd=0 3 2024-04-14T15:22:30.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.867+0000 7f839318e700 20 client.37683.objecter _get_session s=0x7f838c10d130 osd=0 3 2024-04-14T15:22:30.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.867+0000 7f839318e700 15 client.37683.objecter _session_command_op_remove -1 1 2024-04-14T15:22:30.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.867+0000 7f839318e700 20 client.37683.objecter get_session s=0x7f838c10d130 osd=0 3 2024-04-14T15:22:30.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.867+0000 7f839318e700 15 client.37683.objecter _session_command_op_assign 0 1 2024-04-14T15:22:30.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.867+0000 7f839318e700 20 client.37683.objecter put_session s=0x7f838c10d130 osd=0 4 2024-04-14T15:22:30.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.867+0000 7f839318e700 10 client.37683.objecter _send_command 1 2024-04-14T15:22:30.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.867+0000 7f8389ffb700 10 client.37683.objecter ms_handle_connect 0x7f837c071450 2024-04-14T15:22:30.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.867+0000 7f839318e700 1 -- 172.21.15.90:0/3957578272 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f838c0693d0 con 0x7f838c1a8dd0 2024-04-14T15:22:30.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.868+0000 7f8389ffb700 10 client.37683.objecter ms_handle_connect 0x7f838c1a8dd0 2024-04-14T15:22:30.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.868+0000 7f8389ffb700 1 -- 172.21.15.90:0/3957578272 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f838c0693d0 con 0x7f838c1a8dd0 2024-04-14T15:22:30.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.868+0000 7f8389ffb700 10 client.37683.objecter ms_dispatch 0x7f838c001550 command_reply(tid 1: 0 ) 2024-04-14T15:22:30.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.868+0000 7f8389ffb700 10 client.37683.objecter _finish_command 1 = system:0 2024-04-14T15:22:30.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.868+0000 7f8389ffb700 20 client.37683.objecter put_session s=0x7f838c10d130 osd=0 4 2024-04-14T15:22:30.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.868+0000 7f8389ffb700 15 client.37683.objecter _session_command_op_remove 0 1 2024-04-14T15:22:30.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.886+0000 7f839318e700 10 client.37683.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:22:30.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.886+0000 7f839318e700 15 client.37683.objecter _session_command_op_assign -1 2 2024-04-14T15:22:30.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.886+0000 7f839318e700 20 client.37683.objecter _get_session s=0x7f838c10d130 osd=0 3 2024-04-14T15:22:30.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.886+0000 7f839318e700 20 client.37683.objecter put_session s=0x7f838c10d130 osd=0 3 2024-04-14T15:22:30.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.886+0000 7f839318e700 20 client.37683.objecter _get_session s=0x7f838c10d130 osd=0 3 2024-04-14T15:22:30.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.886+0000 7f839318e700 15 client.37683.objecter _session_command_op_remove -1 2 2024-04-14T15:22:30.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.886+0000 7f839318e700 20 client.37683.objecter get_session s=0x7f838c10d130 osd=0 3 2024-04-14T15:22:30.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.886+0000 7f839318e700 15 client.37683.objecter _session_command_op_assign 0 2 2024-04-14T15:22:30.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.886+0000 7f839318e700 20 client.37683.objecter put_session s=0x7f838c10d130 osd=0 4 2024-04-14T15:22:30.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.886+0000 7f839318e700 10 client.37683.objecter _send_command 2 2024-04-14T15:22:30.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.886+0000 7f839318e700 1 -- 172.21.15.90:0/3957578272 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f838c1a98c0 con 0x7f838c1a8dd0 2024-04-14T15:22:30.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.888+0000 7f8389ffb700 1 -- 172.21.15.90:0/3957578272 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f838c1a98c0 con 0x7f838c1a8dd0 2024-04-14T15:22:30.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.888+0000 7f8389ffb700 10 client.37683.objecter ms_dispatch 0x7f838c001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:22:30.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.888+0000 7f8389ffb700 10 client.37683.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:22:30.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.888+0000 7f8389ffb700 20 client.37683.objecter put_session s=0x7f838c10d130 osd=0 4 2024-04-14T15:22:30.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.888+0000 7f8389ffb700 15 client.37683.objecter _session_command_op_remove 0 2 2024-04-14T15:22:30.889 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:22:30.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.888+0000 7f839318e700 10 client.37683.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:22:30.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.888+0000 7f839318e700 15 client.37683.objecter _session_command_op_assign -1 3 2024-04-14T15:22:30.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.888+0000 7f839318e700 20 client.37683.objecter _get_session s=0x7f838c0ff7b0 osd=1 3 2024-04-14T15:22:30.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.888+0000 7f839318e700 20 client.37683.objecter put_session s=0x7f838c0ff7b0 osd=1 3 2024-04-14T15:22:30.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.888+0000 7f839318e700 20 client.37683.objecter _get_session s=0x7f838c0ff7b0 osd=1 3 2024-04-14T15:22:30.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.888+0000 7f839318e700 15 client.37683.objecter _session_command_op_remove -1 3 2024-04-14T15:22:30.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.888+0000 7f839318e700 20 client.37683.objecter get_session s=0x7f838c0ff7b0 osd=1 3 2024-04-14T15:22:30.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.888+0000 7f839318e700 15 client.37683.objecter _session_command_op_assign 1 3 2024-04-14T15:22:30.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.888+0000 7f839318e700 20 client.37683.objecter put_session s=0x7f838c0ff7b0 osd=1 4 2024-04-14T15:22:30.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.888+0000 7f839318e700 10 client.37683.objecter _send_command 3 2024-04-14T15:22:30.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.888+0000 7f839318e700 1 -- 172.21.15.90:0/3957578272 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f838c100270 con 0x7f838c0ffe30 2024-04-14T15:22:30.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.889+0000 7f8389ffb700 10 client.37683.objecter ms_handle_connect 0x7f838c0ffe30 2024-04-14T15:22:30.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.889+0000 7f8389ffb700 1 -- 172.21.15.90:0/3957578272 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f838c100270 con 0x7f838c0ffe30 2024-04-14T15:22:30.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.889+0000 7f8389ffb700 10 client.37683.objecter ms_dispatch 0x7f838c001550 command_reply(tid 3: 0 ) 2024-04-14T15:22:30.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.889+0000 7f8389ffb700 10 client.37683.objecter _finish_command 3 = system:0 2024-04-14T15:22:30.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.889+0000 7f8389ffb700 20 client.37683.objecter put_session s=0x7f838c0ff7b0 osd=1 4 2024-04-14T15:22:30.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.889+0000 7f8389ffb700 15 client.37683.objecter _session_command_op_remove 1 3 2024-04-14T15:22:30.893 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:22:30.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.906+0000 7f839318e700 10 client.37683.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:22:30.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.906+0000 7f839318e700 15 client.37683.objecter _session_command_op_assign -1 4 2024-04-14T15:22:30.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.906+0000 7f839318e700 20 client.37683.objecter _get_session s=0x7f838c0ff7b0 osd=1 3 2024-04-14T15:22:30.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.906+0000 7f839318e700 20 client.37683.objecter put_session s=0x7f838c0ff7b0 osd=1 3 2024-04-14T15:22:30.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.906+0000 7f839318e700 20 client.37683.objecter _get_session s=0x7f838c0ff7b0 osd=1 3 2024-04-14T15:22:30.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.906+0000 7f839318e700 15 client.37683.objecter _session_command_op_remove -1 4 2024-04-14T15:22:30.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.906+0000 7f839318e700 20 client.37683.objecter get_session s=0x7f838c0ff7b0 osd=1 3 2024-04-14T15:22:30.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.906+0000 7f839318e700 15 client.37683.objecter _session_command_op_assign 1 4 2024-04-14T15:22:30.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.906+0000 7f839318e700 20 client.37683.objecter put_session s=0x7f838c0ff7b0 osd=1 4 2024-04-14T15:22:30.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.906+0000 7f839318e700 10 client.37683.objecter _send_command 4 2024-04-14T15:22:30.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.906+0000 7f839318e700 1 -- 172.21.15.90:0/3957578272 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f838c100670 con 0x7f838c0ffe30 2024-04-14T15:22:30.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.908+0000 7f8389ffb700 1 -- 172.21.15.90:0/3957578272 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f838c100670 con 0x7f838c0ffe30 2024-04-14T15:22:30.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.908+0000 7f8389ffb700 10 client.37683.objecter ms_dispatch 0x7f838c001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:22:30.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.908+0000 7f8389ffb700 10 client.37683.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:22:30.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.908+0000 7f8389ffb700 20 client.37683.objecter put_session s=0x7f838c0ff7b0 osd=1 4 2024-04-14T15:22:30.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.908+0000 7f8389ffb700 15 client.37683.objecter _session_command_op_remove 1 4 2024-04-14T15:22:30.908 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:22:30.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.908+0000 7f839318e700 10 client.37683.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:22:30.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.908+0000 7f839318e700 15 client.37683.objecter _session_command_op_assign -1 5 2024-04-14T15:22:30.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.908+0000 7f839318e700 20 client.37683.objecter _get_session s=0x7f838c112920 osd=2 3 2024-04-14T15:22:30.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.908+0000 7f839318e700 20 client.37683.objecter put_session s=0x7f838c112920 osd=2 3 2024-04-14T15:22:30.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.908+0000 7f839318e700 20 client.37683.objecter _get_session s=0x7f838c112920 osd=2 3 2024-04-14T15:22:30.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.908+0000 7f839318e700 15 client.37683.objecter _session_command_op_remove -1 5 2024-04-14T15:22:30.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.908+0000 7f839318e700 20 client.37683.objecter get_session s=0x7f838c112920 osd=2 3 2024-04-14T15:22:30.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.908+0000 7f839318e700 15 client.37683.objecter _session_command_op_assign 2 5 2024-04-14T15:22:30.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.908+0000 7f839318e700 20 client.37683.objecter put_session s=0x7f838c112920 osd=2 4 2024-04-14T15:22:30.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.908+0000 7f839318e700 10 client.37683.objecter _send_command 5 2024-04-14T15:22:30.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.908+0000 7f839318e700 1 -- 172.21.15.90:0/3957578272 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f838c063bd0 con 0x7f838c112a90 2024-04-14T15:22:30.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.908+0000 7f8389ffb700 10 client.37683.objecter ms_handle_connect 0x7f838c112a90 2024-04-14T15:22:30.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.909+0000 7f8389ffb700 1 -- 172.21.15.90:0/3957578272 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f838c063bd0 con 0x7f838c112a90 2024-04-14T15:22:30.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.909+0000 7f8389ffb700 10 client.37683.objecter ms_dispatch 0x7f838c001550 command_reply(tid 5: 0 ) 2024-04-14T15:22:30.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.909+0000 7f8389ffb700 10 client.37683.objecter _finish_command 5 = system:0 2024-04-14T15:22:30.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.909+0000 7f8389ffb700 20 client.37683.objecter put_session s=0x7f838c112920 osd=2 4 2024-04-14T15:22:30.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.909+0000 7f8389ffb700 15 client.37683.objecter _session_command_op_remove 2 5 2024-04-14T15:22:30.916 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:30.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:30.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:22:30.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.927+0000 7f839318e700 10 client.37683.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:22:30.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.927+0000 7f839318e700 15 client.37683.objecter _session_command_op_assign -1 6 2024-04-14T15:22:30.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.927+0000 7f839318e700 20 client.37683.objecter _get_session s=0x7f838c112920 osd=2 3 2024-04-14T15:22:30.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.927+0000 7f839318e700 20 client.37683.objecter put_session s=0x7f838c112920 osd=2 3 2024-04-14T15:22:30.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.927+0000 7f839318e700 20 client.37683.objecter _get_session s=0x7f838c112920 osd=2 3 2024-04-14T15:22:30.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.927+0000 7f839318e700 15 client.37683.objecter _session_command_op_remove -1 6 2024-04-14T15:22:30.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.927+0000 7f839318e700 20 client.37683.objecter get_session s=0x7f838c112920 osd=2 3 2024-04-14T15:22:30.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.927+0000 7f839318e700 15 client.37683.objecter _session_command_op_assign 2 6 2024-04-14T15:22:30.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.927+0000 7f839318e700 20 client.37683.objecter put_session s=0x7f838c112920 osd=2 4 2024-04-14T15:22:30.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.927+0000 7f839318e700 10 client.37683.objecter _send_command 6 2024-04-14T15:22:30.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.927+0000 7f839318e700 1 -- 172.21.15.90:0/3957578272 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f838c1b3be0 con 0x7f838c112a90 2024-04-14T15:22:30.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.929+0000 7f8389ffb700 1 -- 172.21.15.90:0/3957578272 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f838c1b3be0 con 0x7f838c112a90 2024-04-14T15:22:30.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.929+0000 7f8389ffb700 10 client.37683.objecter ms_dispatch 0x7f838c001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:22:30.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.929+0000 7f8389ffb700 10 client.37683.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:22:30.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.929+0000 7f8389ffb700 20 client.37683.objecter put_session s=0x7f838c112920 osd=2 4 2024-04-14T15:22:30.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.929+0000 7f8389ffb700 15 client.37683.objecter _session_command_op_remove 2 6 2024-04-14T15:22:30.929 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:22:30.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.929+0000 7f839318e700 10 client.37683.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:22:30.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.929+0000 7f839318e700 15 client.37683.objecter _session_command_op_assign -1 7 2024-04-14T15:22:30.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.929+0000 7f839318e700 20 client.37683.objecter _get_session s=0x7f838c063140 osd=3 3 2024-04-14T15:22:30.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.929+0000 7f839318e700 20 client.37683.objecter put_session s=0x7f838c063140 osd=3 3 2024-04-14T15:22:30.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.929+0000 7f839318e700 20 client.37683.objecter _get_session s=0x7f838c063140 osd=3 3 2024-04-14T15:22:30.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.929+0000 7f839318e700 15 client.37683.objecter _session_command_op_remove -1 7 2024-04-14T15:22:30.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.929+0000 7f839318e700 20 client.37683.objecter get_session s=0x7f838c063140 osd=3 3 2024-04-14T15:22:30.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.929+0000 7f839318e700 15 client.37683.objecter _session_command_op_assign 3 7 2024-04-14T15:22:30.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.929+0000 7f839318e700 20 client.37683.objecter put_session s=0x7f838c063140 osd=3 4 2024-04-14T15:22:30.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.929+0000 7f839318e700 10 client.37683.objecter _send_command 7 2024-04-14T15:22:30.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.929+0000 7f839318e700 1 -- 172.21.15.90:0/3957578272 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f838c1b0010 con 0x7f838c0632b0 2024-04-14T15:22:30.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.929+0000 7f8389ffb700 10 client.37683.objecter ms_handle_connect 0x7f838c0632b0 2024-04-14T15:22:30.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.930+0000 7f8389ffb700 1 -- 172.21.15.90:0/3957578272 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f838c1b0010 con 0x7f838c0632b0 2024-04-14T15:22:30.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.930+0000 7f8389ffb700 10 client.37683.objecter ms_dispatch 0x7f838c001550 command_reply(tid 7: 0 ) 2024-04-14T15:22:30.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.930+0000 7f8389ffb700 10 client.37683.objecter _finish_command 7 = system:0 2024-04-14T15:22:30.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.930+0000 7f8389ffb700 20 client.37683.objecter put_session s=0x7f838c063140 osd=3 4 2024-04-14T15:22:30.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.930+0000 7f8389ffb700 15 client.37683.objecter _session_command_op_remove 3 7 2024-04-14T15:22:30.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.947+0000 7f839318e700 10 client.37683.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:22:30.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.947+0000 7f839318e700 15 client.37683.objecter _session_command_op_assign -1 8 2024-04-14T15:22:30.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.947+0000 7f839318e700 20 client.37683.objecter _get_session s=0x7f838c063140 osd=3 3 2024-04-14T15:22:30.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.947+0000 7f839318e700 20 client.37683.objecter put_session s=0x7f838c063140 osd=3 3 2024-04-14T15:22:30.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.947+0000 7f839318e700 20 client.37683.objecter _get_session s=0x7f838c063140 osd=3 3 2024-04-14T15:22:30.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.947+0000 7f839318e700 15 client.37683.objecter _session_command_op_remove -1 8 2024-04-14T15:22:30.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.947+0000 7f839318e700 20 client.37683.objecter get_session s=0x7f838c063140 osd=3 3 2024-04-14T15:22:30.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.947+0000 7f839318e700 15 client.37683.objecter _session_command_op_assign 3 8 2024-04-14T15:22:30.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.947+0000 7f839318e700 20 client.37683.objecter put_session s=0x7f838c063140 osd=3 4 2024-04-14T15:22:30.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.947+0000 7f839318e700 10 client.37683.objecter _send_command 8 2024-04-14T15:22:30.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.947+0000 7f839318e700 1 -- 172.21.15.90:0/3957578272 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f838c1b8640 con 0x7f838c0632b0 2024-04-14T15:22:30.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.948+0000 7f8389ffb700 1 -- 172.21.15.90:0/3957578272 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f838c1b8640 con 0x7f838c0632b0 2024-04-14T15:22:30.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.948+0000 7f8389ffb700 10 client.37683.objecter ms_dispatch 0x7f838c001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:22:30.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.948+0000 7f8389ffb700 10 client.37683.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:22:30.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.948+0000 7f8389ffb700 20 client.37683.objecter put_session s=0x7f838c063140 osd=3 4 2024-04-14T15:22:30.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.948+0000 7f8389ffb700 15 client.37683.objecter _session_command_op_remove 3 8 2024-04-14T15:22:30.949 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:22:30.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.949+0000 7f839318e700 10 client.37683.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:22:30.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.949+0000 7f839318e700 15 client.37683.objecter _session_command_op_assign -1 9 2024-04-14T15:22:30.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.949+0000 7f839318e700 20 client.37683.objecter _get_session s=0x7f838c1b8640 osd=4 3 2024-04-14T15:22:30.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.949+0000 7f839318e700 20 client.37683.objecter put_session s=0x7f838c1b8640 osd=4 3 2024-04-14T15:22:30.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.949+0000 7f839318e700 20 client.37683.objecter _get_session s=0x7f838c1b8640 osd=4 3 2024-04-14T15:22:30.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.949+0000 7f839318e700 15 client.37683.objecter _session_command_op_remove -1 9 2024-04-14T15:22:30.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.949+0000 7f839318e700 20 client.37683.objecter get_session s=0x7f838c1b8640 osd=4 3 2024-04-14T15:22:30.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.949+0000 7f839318e700 15 client.37683.objecter _session_command_op_assign 4 9 2024-04-14T15:22:30.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.949+0000 7f839318e700 20 client.37683.objecter put_session s=0x7f838c1b8640 osd=4 4 2024-04-14T15:22:30.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.949+0000 7f839318e700 10 client.37683.objecter _send_command 9 2024-04-14T15:22:30.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.949+0000 7f839318e700 1 -- 172.21.15.90:0/3957578272 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f838c1b4a60 con 0x7f838c1b4560 2024-04-14T15:22:30.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.949+0000 7f8389ffb700 10 client.37683.objecter ms_handle_connect 0x7f838c1b4560 2024-04-14T15:22:30.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.950+0000 7f8389ffb700 1 -- 172.21.15.90:0/3957578272 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f838c1b4a60 con 0x7f838c1b4560 2024-04-14T15:22:30.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.950+0000 7f8389ffb700 10 client.37683.objecter ms_dispatch 0x7f838c001550 command_reply(tid 9: 0 ) 2024-04-14T15:22:30.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.950+0000 7f8389ffb700 10 client.37683.objecter _finish_command 9 = system:0 2024-04-14T15:22:30.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.950+0000 7f8389ffb700 20 client.37683.objecter put_session s=0x7f838c1b8640 osd=4 4 2024-04-14T15:22:30.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.950+0000 7f8389ffb700 15 client.37683.objecter _session_command_op_remove 4 9 2024-04-14T15:22:30.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.968+0000 7f839318e700 10 client.37683.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:22:30.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.968+0000 7f839318e700 15 client.37683.objecter _session_command_op_assign -1 10 2024-04-14T15:22:30.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.968+0000 7f839318e700 20 client.37683.objecter _get_session s=0x7f838c1b8640 osd=4 3 2024-04-14T15:22:30.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.968+0000 7f839318e700 20 client.37683.objecter put_session s=0x7f838c1b8640 osd=4 3 2024-04-14T15:22:30.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.968+0000 7f839318e700 20 client.37683.objecter _get_session s=0x7f838c1b8640 osd=4 3 2024-04-14T15:22:30.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.968+0000 7f839318e700 15 client.37683.objecter _session_command_op_remove -1 10 2024-04-14T15:22:30.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.968+0000 7f839318e700 20 client.37683.objecter get_session s=0x7f838c1b8640 osd=4 3 2024-04-14T15:22:30.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.968+0000 7f839318e700 15 client.37683.objecter _session_command_op_assign 4 10 2024-04-14T15:22:30.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.968+0000 7f839318e700 20 client.37683.objecter put_session s=0x7f838c1b8640 osd=4 4 2024-04-14T15:22:30.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.968+0000 7f839318e700 10 client.37683.objecter _send_command 10 2024-04-14T15:22:30.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.968+0000 7f839318e700 1 -- 172.21.15.90:0/3957578272 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f838c1b4a60 con 0x7f838c1b4560 2024-04-14T15:22:30.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.970+0000 7f8389ffb700 1 -- 172.21.15.90:0/3957578272 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f838c1b4a60 con 0x7f838c1b4560 2024-04-14T15:22:30.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.970+0000 7f8389ffb700 10 client.37683.objecter ms_dispatch 0x7f838c001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:22:30.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.970+0000 7f8389ffb700 10 client.37683.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:22:30.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.970+0000 7f8389ffb700 20 client.37683.objecter put_session s=0x7f838c1b8640 osd=4 4 2024-04-14T15:22:30.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.970+0000 7f8389ffb700 15 client.37683.objecter _session_command_op_remove 4 10 2024-04-14T15:22:30.971 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:22:30.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.971+0000 7f839318e700 10 client.37683.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:22:30.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.971+0000 7f839318e700 15 client.37683.objecter _session_command_op_assign -1 11 2024-04-14T15:22:30.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.971+0000 7f839318e700 20 client.37683.objecter _get_session s=0x7f838c1b4a60 osd=5 3 2024-04-14T15:22:30.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.971+0000 7f839318e700 20 client.37683.objecter put_session s=0x7f838c1b4a60 osd=5 3 2024-04-14T15:22:30.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.971+0000 7f839318e700 20 client.37683.objecter _get_session s=0x7f838c1b4a60 osd=5 3 2024-04-14T15:22:30.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.971+0000 7f839318e700 15 client.37683.objecter _session_command_op_remove -1 11 2024-04-14T15:22:30.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.971+0000 7f839318e700 20 client.37683.objecter get_session s=0x7f838c1b4a60 osd=5 3 2024-04-14T15:22:30.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.971+0000 7f839318e700 15 client.37683.objecter _session_command_op_assign 5 11 2024-04-14T15:22:30.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.971+0000 7f839318e700 20 client.37683.objecter put_session s=0x7f838c1b4a60 osd=5 4 2024-04-14T15:22:30.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.971+0000 7f839318e700 10 client.37683.objecter _send_command 11 2024-04-14T15:22:30.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.971+0000 7f839318e700 1 -- 172.21.15.90:0/3957578272 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f838c1c11f0 con 0x7f838c1bd6e0 2024-04-14T15:22:30.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.971+0000 7f8389ffb700 10 client.37683.objecter ms_handle_connect 0x7f838c1bd6e0 2024-04-14T15:22:30.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.972+0000 7f8389ffb700 1 -- 172.21.15.90:0/3957578272 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f838c1c11f0 con 0x7f838c1bd6e0 2024-04-14T15:22:30.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.972+0000 7f8389ffb700 10 client.37683.objecter ms_dispatch 0x7f838c001550 command_reply(tid 11: 0 ) 2024-04-14T15:22:30.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.972+0000 7f8389ffb700 10 client.37683.objecter _finish_command 11 = system:0 2024-04-14T15:22:30.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.972+0000 7f8389ffb700 20 client.37683.objecter put_session s=0x7f838c1b4a60 osd=5 4 2024-04-14T15:22:30.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.972+0000 7f8389ffb700 15 client.37683.objecter _session_command_op_remove 5 11 2024-04-14T15:22:30.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.990+0000 7f839318e700 10 client.37683.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:22:30.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.990+0000 7f839318e700 15 client.37683.objecter _session_command_op_assign -1 12 2024-04-14T15:22:30.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.990+0000 7f839318e700 20 client.37683.objecter _get_session s=0x7f838c1b4a60 osd=5 3 2024-04-14T15:22:30.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.990+0000 7f839318e700 20 client.37683.objecter put_session s=0x7f838c1b4a60 osd=5 3 2024-04-14T15:22:30.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.990+0000 7f839318e700 20 client.37683.objecter _get_session s=0x7f838c1b4a60 osd=5 3 2024-04-14T15:22:30.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.990+0000 7f839318e700 15 client.37683.objecter _session_command_op_remove -1 12 2024-04-14T15:22:30.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.990+0000 7f839318e700 20 client.37683.objecter get_session s=0x7f838c1b4a60 osd=5 3 2024-04-14T15:22:30.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.990+0000 7f839318e700 15 client.37683.objecter _session_command_op_assign 5 12 2024-04-14T15:22:30.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.990+0000 7f839318e700 20 client.37683.objecter put_session s=0x7f838c1b4a60 osd=5 4 2024-04-14T15:22:30.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.990+0000 7f839318e700 10 client.37683.objecter _send_command 12 2024-04-14T15:22:30.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.990+0000 7f839318e700 1 -- 172.21.15.90:0/3957578272 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f838c1b3fe0 con 0x7f838c1bd6e0 2024-04-14T15:22:30.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.992+0000 7f8389ffb700 1 -- 172.21.15.90:0/3957578272 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f838c1b3fe0 con 0x7f838c1bd6e0 2024-04-14T15:22:30.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.992+0000 7f8389ffb700 10 client.37683.objecter ms_dispatch 0x7f838c001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:22:30.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.992+0000 7f8389ffb700 10 client.37683.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:22:30.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.992+0000 7f8389ffb700 20 client.37683.objecter put_session s=0x7f838c1b4a60 osd=5 4 2024-04-14T15:22:30.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.992+0000 7f8389ffb700 15 client.37683.objecter _session_command_op_remove 5 12 2024-04-14T15:22:30.993 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:22:30.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.993+0000 7f839318e700 10 client.37683.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:22:30.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.993+0000 7f839318e700 15 client.37683.objecter _session_command_op_assign -1 13 2024-04-14T15:22:30.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.993+0000 7f839318e700 20 client.37683.objecter _get_session s=0x7f838c1af390 osd=6 3 2024-04-14T15:22:30.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.993+0000 7f839318e700 20 client.37683.objecter put_session s=0x7f838c1af390 osd=6 3 2024-04-14T15:22:30.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.993+0000 7f839318e700 20 client.37683.objecter _get_session s=0x7f838c1af390 osd=6 3 2024-04-14T15:22:30.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.993+0000 7f839318e700 15 client.37683.objecter _session_command_op_remove -1 13 2024-04-14T15:22:30.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.993+0000 7f839318e700 20 client.37683.objecter get_session s=0x7f838c1af390 osd=6 3 2024-04-14T15:22:30.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.993+0000 7f839318e700 15 client.37683.objecter _session_command_op_assign 6 13 2024-04-14T15:22:30.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.993+0000 7f839318e700 20 client.37683.objecter put_session s=0x7f838c1af390 osd=6 4 2024-04-14T15:22:30.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.993+0000 7f839318e700 10 client.37683.objecter _send_command 13 2024-04-14T15:22:30.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.993+0000 7f839318e700 1 -- 172.21.15.90:0/3957578272 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f838c1b9530 con 0x7f838c1b8fc0 2024-04-14T15:22:30.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.993+0000 7f8389ffb700 10 client.37683.objecter ms_handle_connect 0x7f838c1b8fc0 2024-04-14T15:22:30.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.994+0000 7f8389ffb700 1 -- 172.21.15.90:0/3957578272 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f838c1b9530 con 0x7f838c1b8fc0 2024-04-14T15:22:30.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.994+0000 7f8389ffb700 10 client.37683.objecter ms_dispatch 0x7f838c001550 command_reply(tid 13: 0 ) 2024-04-14T15:22:30.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.994+0000 7f8389ffb700 10 client.37683.objecter _finish_command 13 = system:0 2024-04-14T15:22:30.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.994+0000 7f8389ffb700 20 client.37683.objecter put_session s=0x7f838c1af390 osd=6 4 2024-04-14T15:22:30.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:30.994+0000 7f8389ffb700 15 client.37683.objecter _session_command_op_remove 6 13 2024-04-14T15:22:31.018 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:22:31.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.018+0000 7f839318e700 10 client.37683.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:22:31.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.018+0000 7f839318e700 15 client.37683.objecter _session_command_op_assign -1 14 2024-04-14T15:22:31.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.018+0000 7f839318e700 20 client.37683.objecter _get_session s=0x7f838c1af390 osd=6 3 2024-04-14T15:22:31.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.018+0000 7f839318e700 20 client.37683.objecter put_session s=0x7f838c1af390 osd=6 3 2024-04-14T15:22:31.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.018+0000 7f839318e700 20 client.37683.objecter _get_session s=0x7f838c1af390 osd=6 3 2024-04-14T15:22:31.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.018+0000 7f839318e700 15 client.37683.objecter _session_command_op_remove -1 14 2024-04-14T15:22:31.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.018+0000 7f839318e700 20 client.37683.objecter get_session s=0x7f838c1af390 osd=6 3 2024-04-14T15:22:31.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.018+0000 7f839318e700 15 client.37683.objecter _session_command_op_assign 6 14 2024-04-14T15:22:31.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.018+0000 7f839318e700 20 client.37683.objecter put_session s=0x7f838c1af390 osd=6 4 2024-04-14T15:22:31.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.018+0000 7f839318e700 10 client.37683.objecter _send_command 14 2024-04-14T15:22:31.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.018+0000 7f839318e700 1 -- 172.21.15.90:0/3957578272 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f838c1b9740 con 0x7f838c1b8fc0 2024-04-14T15:22:31.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.021+0000 7f8389ffb700 1 -- 172.21.15.90:0/3957578272 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f838c1b9740 con 0x7f838c1b8fc0 2024-04-14T15:22:31.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.021+0000 7f8389ffb700 10 client.37683.objecter ms_dispatch 0x7f838c001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:22:31.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.021+0000 7f8389ffb700 10 client.37683.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:22:31.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.021+0000 7f8389ffb700 20 client.37683.objecter put_session s=0x7f838c1af390 osd=6 4 2024-04-14T15:22:31.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.021+0000 7f8389ffb700 15 client.37683.objecter _session_command_op_remove 6 14 2024-04-14T15:22:31.022 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:22:31.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.021+0000 7f839318e700 10 client.37683.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:22:31.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.021+0000 7f839318e700 15 client.37683.objecter _session_command_op_assign -1 15 2024-04-14T15:22:31.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.021+0000 7f839318e700 20 client.37683.objecter _get_session s=0x7f838c1b9590 osd=7 3 2024-04-14T15:22:31.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.021+0000 7f839318e700 20 client.37683.objecter put_session s=0x7f838c1b9590 osd=7 3 2024-04-14T15:22:31.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.021+0000 7f839318e700 20 client.37683.objecter _get_session s=0x7f838c1b9590 osd=7 3 2024-04-14T15:22:31.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.021+0000 7f839318e700 15 client.37683.objecter _session_command_op_remove -1 15 2024-04-14T15:22:31.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.021+0000 7f839318e700 20 client.37683.objecter get_session s=0x7f838c1b9590 osd=7 3 2024-04-14T15:22:31.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.021+0000 7f839318e700 15 client.37683.objecter _session_command_op_assign 7 15 2024-04-14T15:22:31.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.021+0000 7f839318e700 20 client.37683.objecter put_session s=0x7f838c1b9590 osd=7 4 2024-04-14T15:22:31.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.021+0000 7f839318e700 10 client.37683.objecter _send_command 15 2024-04-14T15:22:31.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.021+0000 7f839318e700 1 -- 172.21.15.90:0/3957578272 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f838c074ce0 con 0x7f838c1c5fb0 2024-04-14T15:22:31.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.022+0000 7f8389ffb700 10 client.37683.objecter ms_handle_connect 0x7f838c1c5fb0 2024-04-14T15:22:31.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.022+0000 7f8389ffb700 1 -- 172.21.15.90:0/3957578272 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f838c074ce0 con 0x7f838c1c5fb0 2024-04-14T15:22:31.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.022+0000 7f8389ffb700 10 client.37683.objecter ms_dispatch 0x7f838c001550 command_reply(tid 15: 0 ) 2024-04-14T15:22:31.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.022+0000 7f8389ffb700 10 client.37683.objecter _finish_command 15 = system:0 2024-04-14T15:22:31.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.022+0000 7f8389ffb700 20 client.37683.objecter put_session s=0x7f838c1b9590 osd=7 4 2024-04-14T15:22:31.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.022+0000 7f8389ffb700 15 client.37683.objecter _session_command_op_remove 7 15 2024-04-14T15:22:31.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.039+0000 7f839318e700 10 client.37683.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:22:31.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.039+0000 7f839318e700 15 client.37683.objecter _session_command_op_assign -1 16 2024-04-14T15:22:31.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.039+0000 7f839318e700 20 client.37683.objecter _get_session s=0x7f838c1b9590 osd=7 3 2024-04-14T15:22:31.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.039+0000 7f839318e700 20 client.37683.objecter put_session s=0x7f838c1b9590 osd=7 3 2024-04-14T15:22:31.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.039+0000 7f839318e700 20 client.37683.objecter _get_session s=0x7f838c1b9590 osd=7 3 2024-04-14T15:22:31.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.039+0000 7f839318e700 15 client.37683.objecter _session_command_op_remove -1 16 2024-04-14T15:22:31.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.039+0000 7f839318e700 20 client.37683.objecter get_session s=0x7f838c1b9590 osd=7 3 2024-04-14T15:22:31.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.039+0000 7f839318e700 15 client.37683.objecter _session_command_op_assign 7 16 2024-04-14T15:22:31.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.039+0000 7f839318e700 20 client.37683.objecter put_session s=0x7f838c1b9590 osd=7 4 2024-04-14T15:22:31.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.039+0000 7f839318e700 10 client.37683.objecter _send_command 16 2024-04-14T15:22:31.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.039+0000 7f839318e700 1 -- 172.21.15.90:0/3957578272 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f838c1b3df0 con 0x7f838c1c5fb0 2024-04-14T15:22:31.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.041+0000 7f8389ffb700 1 -- 172.21.15.90:0/3957578272 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f838c1b3df0 con 0x7f838c1c5fb0 2024-04-14T15:22:31.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.041+0000 7f8389ffb700 10 client.37683.objecter ms_dispatch 0x7f838c001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:22:31.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.041+0000 7f8389ffb700 10 client.37683.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:22:31.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.041+0000 7f8389ffb700 20 client.37683.objecter put_session s=0x7f838c1b9590 osd=7 4 2024-04-14T15:22:31.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.041+0000 7f8389ffb700 15 client.37683.objecter _session_command_op_remove 7 16 2024-04-14T15:22:31.042 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:22:31.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.041+0000 7f839318e700 10 librados: watch_flush enter 2024-04-14T15:22:31.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.041+0000 7f839318e700 10 librados: watch_flush exit 2024-04-14T15:22:31.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.041+0000 7f839318e700 10 client.37683.objecter close_session for osd.0 2024-04-14T15:22:31.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.041+0000 7f839318e700 1 -- 172.21.15.90:0/3957578272 >> v1:172.21.15.90:6801/3761806885 conn(0x7f838c1a8dd0 legacy=0x7f838c1a91b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:31.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.041+0000 7f839318e700 20 client.37683.objecter put_session s=0x7f838c10d130 osd=0 1 2024-04-14T15:22:31.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.041+0000 7f839318e700 10 client.37683.objecter close_session for osd.1 2024-04-14T15:22:31.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.041+0000 7f839318e700 1 -- 172.21.15.90:0/3957578272 >> v1:172.21.15.90:6805/4170361699 conn(0x7f838c0ffe30 legacy=0x7f838c02ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:31.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.041+0000 7f839318e700 20 client.37683.objecter put_session s=0x7f838c0ff7b0 osd=1 1 2024-04-14T15:22:31.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.041+0000 7f839318e700 10 client.37683.objecter close_session for osd.2 2024-04-14T15:22:31.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.041+0000 7f839318e700 1 -- 172.21.15.90:0/3957578272 >> v1:172.21.15.90:6809/1686184251 conn(0x7f838c112a90 legacy=0x7f838c1b24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:31.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.041+0000 7f839318e700 20 client.37683.objecter put_session s=0x7f838c112920 osd=2 1 2024-04-14T15:22:31.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.041+0000 7f839318e700 10 client.37683.objecter close_session for osd.3 2024-04-14T15:22:31.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.041+0000 7f839318e700 1 -- 172.21.15.90:0/3957578272 >> v1:172.21.15.90:6813/2270166774 conn(0x7f838c0632b0 legacy=0x7f838c1b6f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:31.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.041+0000 7f839318e700 20 client.37683.objecter put_session s=0x7f838c063140 osd=3 1 2024-04-14T15:22:31.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.041+0000 7f839318e700 10 client.37683.objecter close_session for osd.4 2024-04-14T15:22:31.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.041+0000 7f839318e700 1 -- 172.21.15.90:0/3957578272 >> v1:172.21.15.92:6800/3887549733 conn(0x7f838c1b4560 legacy=0x7f838c1b99d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:31.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.041+0000 7f839318e700 20 client.37683.objecter put_session s=0x7f838c1b8640 osd=4 1 2024-04-14T15:22:31.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.041+0000 7f839318e700 10 client.37683.objecter close_session for osd.5 2024-04-14T15:22:31.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.041+0000 7f839318e700 1 -- 172.21.15.90:0/3957578272 >> v1:172.21.15.92:6804/3244561440 conn(0x7f838c1bd6e0 legacy=0x7f838c1bfad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:31.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.041+0000 7f839318e700 20 client.37683.objecter put_session s=0x7f838c1b4a60 osd=5 1 2024-04-14T15:22:31.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.041+0000 7f839318e700 10 client.37683.objecter close_session for osd.6 2024-04-14T15:22:31.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.041+0000 7f839318e700 1 -- 172.21.15.90:0/3957578272 >> v1:172.21.15.92:6808/636125963 conn(0x7f838c1b8fc0 legacy=0x7f838c1c4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:31.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.041+0000 7f839318e700 20 client.37683.objecter put_session s=0x7f838c1af390 osd=6 1 2024-04-14T15:22:31.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.041+0000 7f839318e700 10 client.37683.objecter close_session for osd.7 2024-04-14T15:22:31.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.041+0000 7f839318e700 1 -- 172.21.15.90:0/3957578272 >> v1:172.21.15.92:6812/1773706716 conn(0x7f838c1c5fb0 legacy=0x7f838c1c83a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:31.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.041+0000 7f839172b700 1 -- 172.21.15.90:0/3957578272 reap_dead start 2024-04-14T15:22:31.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.041+0000 7f839172b700 1 -- 172.21.15.90:0/3957578272 reap_dead start 2024-04-14T15:22:31.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.041+0000 7f839318e700 20 client.37683.objecter put_session s=0x7f838c1b9590 osd=7 1 2024-04-14T15:22:31.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.041+0000 7f839318e700 20 client.37683.objecter shutdown clearing up homeless session... 2024-04-14T15:22:31.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.041+0000 7f839318e700 10 client.37683.objecter successfully canceled tick 2024-04-14T15:22:31.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.041+0000 7f839172b700 1 -- 172.21.15.90:0/3957578272 reap_dead start 2024-04-14T15:22:31.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.041+0000 7f839172b700 1 -- 172.21.15.90:0/3957578272 reap_dead start 2024-04-14T15:22:31.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.042+0000 7f839318e700 1 -- 172.21.15.90:0/3957578272 >> v1:172.21.15.90:6800/420935427 conn(0x7f837c071450 legacy=0x7f837c073910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:31.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.042+0000 7f839318e700 1 -- 172.21.15.90:0/3957578272 >> v1:172.21.15.90:6789/0 conn(0x7f838c1088b0 legacy=0x7f838c10eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:31.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.042+0000 7f839318e700 1 -- 172.21.15.90:0/3957578272 shutdown_connections 2024-04-14T15:22:31.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.042+0000 7f839318e700 1 -- 172.21.15.90:0/3957578272 >> 172.21.15.90:0/3957578272 conn(0x7f838c0fe3c0 msgr2=0x7f838c111fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:22:31.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.042+0000 7f839318e700 1 -- 172.21.15.90:0/3957578272 shutdown_connections 2024-04-14T15:22:31.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.042+0000 7f839318e700 1 -- 172.21.15.90:0/3957578272 wait complete. 2024-04-14T15:22:31.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:31.042+0000 7f839318e700 1 librados: shutdown 2024-04-14T15:22:31.046 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:22:31.046 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:22:31.046 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:22:31.046 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:22:31.046 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:22:31.046 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:22:31.046 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:22:31.046 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:22:31.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:31.047 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:31.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:22:31.148 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:22:31.171 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:31.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:31.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:22:31.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:30 smithi090 ceph-mon[30399]: pgmap v900: 218 pgs: 13 creating+peering, 24 unknown, 3 active+remapped+backfill_toofull, 178 active+clean; 596 KiB data, 509 MiB used, 625 GiB / 626 GiB avail; 2/597 objects misplaced (0.335%); 27 B/s, 0 objects/s recovering 2024-04-14T15:22:31.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:30 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1465271679' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-107","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:22:31.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:30 smithi090 ceph-mon[30399]: osdmap e614: 8 total, 8 up, 7 in 2024-04-14T15:22:31.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:30 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3957578272' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:22:31.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:30 smithi090 ceph-mon[41670]: pgmap v900: 218 pgs: 13 creating+peering, 24 unknown, 3 active+remapped+backfill_toofull, 178 active+clean; 596 KiB data, 509 MiB used, 625 GiB / 626 GiB avail; 2/597 objects misplaced (0.335%); 27 B/s, 0 objects/s recovering 2024-04-14T15:22:31.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:30 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1465271679' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-107","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:22:31.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:30 smithi090 ceph-mon[41670]: osdmap e614: 8 total, 8 up, 7 in 2024-04-14T15:22:31.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:30 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3957578272' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:22:31.274 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:22:31.301 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:31.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:31.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:22:31.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:30 smithi092 ceph-mon[33966]: pgmap v900: 218 pgs: 13 creating+peering, 24 unknown, 3 active+remapped+backfill_toofull, 178 active+clean; 596 KiB data, 509 MiB used, 625 GiB / 626 GiB avail; 2/597 objects misplaced (0.335%); 27 B/s, 0 objects/s recovering 2024-04-14T15:22:31.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:30 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1465271679' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-107","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:22:31.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:30 smithi092 ceph-mon[33966]: osdmap e614: 8 total, 8 up, 7 in 2024-04-14T15:22:31.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:30 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3957578272' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:22:31.403 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:31.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:31.428 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:31.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:31.528 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:22:31.551 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:31.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:31.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:22:31.653 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:22:31.676 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:31.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:31.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:22:31.777 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:22:31.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:31.805 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:31.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:22:31.906 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:22:31.930 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:31.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:31.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:22:32.031 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:22:32.051 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:32 smithi090 ceph-mon[41670]: osdmap e615: 8 total, 8 up, 7 in 2024-04-14T15:22:32.051 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:32 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1465271679' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:22:32.051 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:32 smithi090 ceph-mon[30399]: osdmap e615: 8 total, 8 up, 7 in 2024-04-14T15:22:32.051 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:32 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1465271679' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:22:32.058 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:32.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:32.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:22:32.160 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:22:32.184 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:32.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:32.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:22:32.286 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:22:32.313 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:32.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:32.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:22:32.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:32 smithi092 ceph-mon[33966]: osdmap e615: 8 total, 8 up, 7 in 2024-04-14T15:22:32.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:32 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1465271679' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:22:32.415 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:22:32.443 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:32.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:32.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:22:32.545 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:22:32.572 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:32.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:32.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:22:32.674 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:22:32.697 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:32.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:32.698 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:22:32.799 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:22:32.827 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:32.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:32.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:22:32.928 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:32.951 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:32.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:32.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:33.004 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:22:33.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.004+0000 7f5024eff700 1 -- 172.21.15.90:0/3944767351 >> v1:172.21.15.90:6790/0 conn(0x7f502010e310 legacy=0x7f50201107d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:33.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.004+0000 7f5024eff700 1 -- 172.21.15.90:0/3944767351 shutdown_connections 2024-04-14T15:22:33.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.004+0000 7f5024eff700 1 -- 172.21.15.90:0/3944767351 >> 172.21.15.90:0/3944767351 conn(0x7f50200fd290 msgr2=0x7f50200ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:22:33.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.004+0000 7f5024eff700 1 -- 172.21.15.90:0/3944767351 shutdown_connections 2024-04-14T15:22:33.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.004+0000 7f5024eff700 1 -- 172.21.15.90:0/3944767351 wait complete. 2024-04-14T15:22:33.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.004+0000 7f5024eff700 1 librados: starting msgr at 2024-04-14T15:22:33.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.004+0000 7f5024eff700 1 librados: starting objecter 2024-04-14T15:22:33.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.005+0000 7f5024eff700 1 Processor -- start 2024-04-14T15:22:33.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.005+0000 7f5024eff700 1 -- start start 2024-04-14T15:22:33.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.005+0000 7f5024eff700 1 librados: setting wanted keys 2024-04-14T15:22:33.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.005+0000 7f5024eff700 1 librados: calling monclient init 2024-04-14T15:22:33.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.005+0000 7f5024eff700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f502010e010 con 0x7f502010e310 2024-04-14T15:22:33.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.005+0000 7f5024eff700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f50201a8f30 con 0x7f5020107890 2024-04-14T15:22:33.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.005+0000 7f5024eff700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f50201aa110 con 0x7f502010a740 2024-04-14T15:22:33.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.005+0000 7f501ed9d700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f502010e310 0x7f50201a6800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:51354/0 (socket says 172.21.15.90:51354) 2024-04-14T15:22:33.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.005+0000 7f501ed9d700 1 -- 172.21.15.90:0/1986732019 learned_addr learned my addr 172.21.15.90:0/1986732019 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:22:33.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.005+0000 7f500f7fe700 10 client.?.objecter ms_handle_connect 0x7f502010a740 2024-04-14T15:22:33.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.005+0000 7f500f7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:22:33.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.005+0000 7f500f7fe700 10 client.?.objecter ms_handle_connect 0x7f502010e310 2024-04-14T15:22:33.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.005+0000 7f500f7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:22:33.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.005+0000 7f500f7fe700 10 client.?.objecter ms_handle_connect 0x7f5020107890 2024-04-14T15:22:33.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.005+0000 7f500f7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:22:33.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.005+0000 7f500f7fe700 1 -- 172.21.15.90:0/1986732019 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3514841315 0 0) 0x7f50201aa110 con 0x7f502010a740 2024-04-14T15:22:33.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.005+0000 7f500f7fe700 1 -- 172.21.15.90:0/1986732019 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f50080039c0 con 0x7f502010a740 2024-04-14T15:22:33.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.005+0000 7f500f7fe700 1 -- 172.21.15.90:0/1986732019 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2330647057 0 0) 0x7f502010e010 con 0x7f502010e310 2024-04-14T15:22:33.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.005+0000 7f500f7fe700 1 -- 172.21.15.90:0/1986732019 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f50201aa110 con 0x7f502010e310 2024-04-14T15:22:33.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.005+0000 7f500f7fe700 1 -- 172.21.15.90:0/1986732019 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3197024259 0 0) 0x7f50201a8f30 con 0x7f5020107890 2024-04-14T15:22:33.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.005+0000 7f500f7fe700 1 -- 172.21.15.90:0/1986732019 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f502010e010 con 0x7f5020107890 2024-04-14T15:22:33.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.006+0000 7f500f7fe700 1 -- 172.21.15.90:0/1986732019 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4083115317 0 0) 0x7f502010e010 con 0x7f5020107890 2024-04-14T15:22:33.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.006+0000 7f500f7fe700 1 -- 172.21.15.90:0/1986732019 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f50201a8f30 con 0x7f5020107890 2024-04-14T15:22:33.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.006+0000 7f500f7fe700 1 -- 172.21.15.90:0/1986732019 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f50100017f0 con 0x7f5020107890 2024-04-14T15:22:33.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.006+0000 7f500f7fe700 1 -- 172.21.15.90:0/1986732019 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 87504818 0 0) 0x7f50080039c0 con 0x7f502010a740 2024-04-14T15:22:33.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.006+0000 7f500f7fe700 1 -- 172.21.15.90:0/1986732019 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f502010e010 con 0x7f502010a740 2024-04-14T15:22:33.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.006+0000 7f500f7fe700 1 -- 172.21.15.90:0/1986732019 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f5014001890 con 0x7f502010a740 2024-04-14T15:22:33.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.006+0000 7f500f7fe700 1 -- 172.21.15.90:0/1986732019 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1045866492 0 0) 0x7f50201aa110 con 0x7f502010e310 2024-04-14T15:22:33.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.006+0000 7f500f7fe700 1 -- 172.21.15.90:0/1986732019 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f50080039c0 con 0x7f502010e310 2024-04-14T15:22:33.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.006+0000 7f500f7fe700 1 -- 172.21.15.90:0/1986732019 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f5018004780 con 0x7f502010e310 2024-04-14T15:22:33.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.006+0000 7f500f7fe700 1 -- 172.21.15.90:0/1986732019 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1041714856 0 0) 0x7f50201a8f30 con 0x7f5020107890 2024-04-14T15:22:33.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.006+0000 7f500f7fe700 1 -- 172.21.15.90:0/1986732019 >> v1:172.21.15.90:6790/0 conn(0x7f502010a740 legacy=0x7f502006b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:33.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.006+0000 7f500f7fe700 1 -- 172.21.15.90:0/1986732019 >> v1:172.21.15.90:6789/0 conn(0x7f502010e310 legacy=0x7f50201a6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:33.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.006+0000 7f500f7fe700 1 -- 172.21.15.90:0/1986732019 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50201ab2f0 con 0x7f5020107890 2024-04-14T15:22:33.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.006+0000 7f5024eff700 1 -- 172.21.15.90:0/1986732019 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f50201a7f80 con 0x7f5020107890 2024-04-14T15:22:33.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.006+0000 7f5024eff700 10 client.49522.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:22:33.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.006+0000 7f5024eff700 1 -- 172.21.15.90:0/1986732019 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f50201a8440 con 0x7f5020107890 2024-04-14T15:22:33.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.006+0000 7f5024eff700 1 librados: init done 2024-04-14T15:22:33.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.006+0000 7f500f7fe700 1 -- 172.21.15.90:0/1986732019 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f5010001d00 con 0x7f5020107890 2024-04-14T15:22:33.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.006+0000 7f500f7fe700 1 -- 172.21.15.90:0/1986732019 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f50100048b0 con 0x7f5020107890 2024-04-14T15:22:33.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.007+0000 7f5024eff700 10 client.49522.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:22:33.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.007+0000 7f5024eff700 15 client.49522.objecter _session_command_op_assign -1 1 2024-04-14T15:22:33.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.007+0000 7f5024eff700 20 client.49522.objecter _get_session osd=-1 returning homeless 2024-04-14T15:22:33.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.007+0000 7f5024eff700 10 client.49522.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:22:33.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.007+0000 7f5024eff700 1 -- 172.21.15.90:0/1986732019 --> v1:172.21.15.92:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f5000002920 con 0x7f5020107890 2024-04-14T15:22:33.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.008+0000 7f500f7fe700 1 -- 172.21.15.90:0/1986732019 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f50100036a0 con 0x7f5020107890 2024-04-14T15:22:33.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.008+0000 7f500f7fe700 1 -- 172.21.15.90:0/1986732019 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(616..616 src has 467..616) ==== 9177+0+0 (unknown 3432665064 0 0) 0x7f5010089a60 con 0x7f5020107890 2024-04-14T15:22:33.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.008+0000 7f500f7fe700 10 client.49522.objecter ms_dispatch 0x7f5020001550 osd_map(616..616 src has 467..616) 2024-04-14T15:22:33.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.008+0000 7f500f7fe700 3 client.49522.objecter handle_osd_map got epochs [616,616] > 0 2024-04-14T15:22:33.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.008+0000 7f500f7fe700 3 client.49522.objecter handle_osd_map decoding full epoch 616 2024-04-14T15:22:33.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.008+0000 7f500f7fe700 10 client.49522.objecter checking command 1 2024-04-14T15:22:33.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.008+0000 7f500f7fe700 20 client.49522.objecter _get_session s=0x7f500807ff10 osd=3 3 2024-04-14T15:22:33.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.008+0000 7f500f7fe700 20 client.49522.objecter put_session s=0x7f500807ff10 osd=3 3 2024-04-14T15:22:33.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.008+0000 7f500f7fe700 15 client.49522.objecter _session_command_op_remove -1 1 2024-04-14T15:22:33.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.008+0000 7f500f7fe700 20 client.49522.objecter _get_session s=0x7f500807ff10 osd=3 3 2024-04-14T15:22:33.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.008+0000 7f500f7fe700 20 client.49522.objecter get_session s=0x7f500807ff10 osd=3 3 2024-04-14T15:22:33.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.008+0000 7f500f7fe700 15 client.49522.objecter _session_command_op_assign 3 1 2024-04-14T15:22:33.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.008+0000 7f500f7fe700 20 client.49522.objecter put_session s=0x7f500807ff10 osd=3 4 2024-04-14T15:22:33.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.008+0000 7f500f7fe700 10 client.49522.objecter _send_command 1 2024-04-14T15:22:33.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.008+0000 7f500f7fe700 1 -- 172.21.15.90:0/1986732019 --> v1:172.21.15.90:6813/2270166774 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f50080842d0 con 0x7f50080805c0 2024-04-14T15:22:33.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.009+0000 7f500f7fe700 20 client.49522.objecter dump_active .. 0 homeless 2024-04-14T15:22:33.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.009+0000 7f500f7fe700 1 -- 172.21.15.90:0/1986732019 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_get_version_reply(handle=1 version=616) ==== 24+0+0 (unknown 2438142826 0 0) 0x7f5010089e40 con 0x7f5020107890 2024-04-14T15:22:33.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.009+0000 7f500f7fe700 10 client.49522.objecter ms_handle_connect 0x7f5008071520 2024-04-14T15:22:33.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.009+0000 7f500f7fe700 10 client.49522.objecter ms_handle_connect 0x7f50080805c0 2024-04-14T15:22:33.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.009+0000 7f500f7fe700 1 -- 172.21.15.90:0/1986732019 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f50080842d0 con 0x7f50080805c0 2024-04-14T15:22:33.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.009+0000 7f500f7fe700 10 client.49522.objecter ms_dispatch 0x7f5020001550 command_reply(tid 1: 0 ) 2024-04-14T15:22:33.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.009+0000 7f500f7fe700 10 client.49522.objecter _finish_command 1 = system:0 2024-04-14T15:22:33.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.009+0000 7f500f7fe700 20 client.49522.objecter put_session s=0x7f500807ff10 osd=3 4 2024-04-14T15:22:33.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.009+0000 7f500f7fe700 15 client.49522.objecter _session_command_op_remove 3 1 2024-04-14T15:22:33.023 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:33 smithi090 ceph-mon[30399]: pgmap v903: 250 pgs: 32 unknown, 32 creating+peering, 3 active+remapped+backfill_toofull, 183 active+clean; 596 KiB data, 496 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s; 2/600 objects misplaced (0.333%); 54 B/s, 3 objects/s recovering 2024-04-14T15:22:33.024 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:33 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1465271679' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-107-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:22:33.024 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:33 smithi090 ceph-mon[30399]: osdmap e616: 8 total, 8 up, 7 in 2024-04-14T15:22:33.024 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:33 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1465271679' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-107", "tierpool": "test-rados-api-smithi090-93652-107-cache"}]: dispatch 2024-04-14T15:22:33.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.039+0000 7f5024eff700 10 client.49522.objecter _submit_command 2 [{"prefix": "reset_purged_snaps_last"}] 2024-04-14T15:22:33.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.039+0000 7f5024eff700 15 client.49522.objecter _session_command_op_assign -1 2 2024-04-14T15:22:33.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.039+0000 7f5024eff700 20 client.49522.objecter _get_session s=0x7f500807ff10 osd=3 3 2024-04-14T15:22:33.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.039+0000 7f5024eff700 20 client.49522.objecter put_session s=0x7f500807ff10 osd=3 3 2024-04-14T15:22:33.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.039+0000 7f5024eff700 20 client.49522.objecter _get_session s=0x7f500807ff10 osd=3 3 2024-04-14T15:22:33.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.039+0000 7f5024eff700 15 client.49522.objecter _session_command_op_remove -1 2 2024-04-14T15:22:33.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.039+0000 7f5024eff700 20 client.49522.objecter get_session s=0x7f500807ff10 osd=3 3 2024-04-14T15:22:33.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.039+0000 7f5024eff700 15 client.49522.objecter _session_command_op_assign 3 2 2024-04-14T15:22:33.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.039+0000 7f5024eff700 20 client.49522.objecter put_session s=0x7f500807ff10 osd=3 4 2024-04-14T15:22:33.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.039+0000 7f5024eff700 10 client.49522.objecter _send_command 2 2024-04-14T15:22:33.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.039+0000 7f5024eff700 1 -- 172.21.15.90:0/1986732019 --> v1:172.21.15.90:6813/2270166774 -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) -- 0x7f50000011f0 con 0x7f50080805c0 2024-04-14T15:22:33.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.039+0000 7f500f7fe700 1 -- 172.21.15.90:0/1986732019 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 2: 0 ) ==== 8+0+0 (unknown 0 0 0) 0x7f50000011f0 con 0x7f50080805c0 2024-04-14T15:22:33.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.039+0000 7f500f7fe700 10 client.49522.objecter ms_dispatch 0x7f5020001550 command_reply(tid 2: 0 ) 2024-04-14T15:22:33.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.039+0000 7f500f7fe700 10 client.49522.objecter _finish_command 2 = system:0 2024-04-14T15:22:33.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.039+0000 7f500f7fe700 20 client.49522.objecter put_session s=0x7f500807ff10 osd=3 4 2024-04-14T15:22:33.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.039+0000 7f500f7fe700 15 client.49522.objecter _session_command_op_remove 3 2 2024-04-14T15:22:33.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.040+0000 7f5024eff700 10 librados: watch_flush enter 2024-04-14T15:22:33.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.040+0000 7f5024eff700 10 librados: watch_flush exit 2024-04-14T15:22:33.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.040+0000 7f5024eff700 10 client.49522.objecter close_session for osd.3 2024-04-14T15:22:33.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.040+0000 7f5024eff700 1 -- 172.21.15.90:0/1986732019 >> v1:172.21.15.90:6813/2270166774 conn(0x7f50080805c0 legacy=0x7f5008082a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:33.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.040+0000 7f5024eff700 20 client.49522.objecter put_session s=0x7f500807ff10 osd=3 1 2024-04-14T15:22:33.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.040+0000 7f5024eff700 20 client.49522.objecter shutdown clearing up homeless session... 2024-04-14T15:22:33.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.040+0000 7f5024eff700 10 client.49522.objecter successfully canceled tick 2024-04-14T15:22:33.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.040+0000 7f5024eff700 1 -- 172.21.15.90:0/1986732019 >> v1:172.21.15.90:6800/420935427 conn(0x7f5008071520 legacy=0x7f50080739e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:33.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.040+0000 7f5024eff700 1 -- 172.21.15.90:0/1986732019 >> v1:172.21.15.92:6789/0 conn(0x7f5020107890 legacy=0x7f502006acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:33.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.040+0000 7f501ed9d700 1 -- 172.21.15.90:0/1986732019 reap_dead start 2024-04-14T15:22:33.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.040+0000 7f5024eff700 1 -- 172.21.15.90:0/1986732019 shutdown_connections 2024-04-14T15:22:33.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.040+0000 7f5024eff700 1 -- 172.21.15.90:0/1986732019 >> 172.21.15.90:0/1986732019 conn(0x7f50200fd290 msgr2=0x7f5020100360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:22:33.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.040+0000 7f5024eff700 1 -- 172.21.15.90:0/1986732019 shutdown_connections 2024-04-14T15:22:33.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.040+0000 7f5024eff700 1 -- 172.21.15.90:0/1986732019 wait complete. 2024-04-14T15:22:33.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:33.040+0000 7f5024eff700 1 librados: shutdown 2024-04-14T15:22:33.053 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:22:33.078 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:33.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:33.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:22:33.180 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:22:33.204 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:33.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:33.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:22:33.306 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:22:33.331 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:33.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:33.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:22:33.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:33 smithi092 ceph-mon[33966]: pgmap v903: 250 pgs: 32 unknown, 32 creating+peering, 3 active+remapped+backfill_toofull, 183 active+clean; 596 KiB data, 496 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s; 2/600 objects misplaced (0.333%); 54 B/s, 3 objects/s recovering 2024-04-14T15:22:33.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:33 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1465271679' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-107-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:22:33.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:33 smithi092 ceph-mon[33966]: osdmap e616: 8 total, 8 up, 7 in 2024-04-14T15:22:33.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:33 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1465271679' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-107", "tierpool": "test-rados-api-smithi090-93652-107-cache"}]: dispatch 2024-04-14T15:22:33.432 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:22:33.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:33 smithi090 ceph-mon[41670]: pgmap v903: 250 pgs: 32 unknown, 32 creating+peering, 3 active+remapped+backfill_toofull, 183 active+clean; 596 KiB data, 496 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s; 2/600 objects misplaced (0.333%); 54 B/s, 3 objects/s recovering 2024-04-14T15:22:33.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:33 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1465271679' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-107-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:22:33.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:33 smithi090 ceph-mon[41670]: osdmap e616: 8 total, 8 up, 7 in 2024-04-14T15:22:33.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:33 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1465271679' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-107", "tierpool": "test-rados-api-smithi090-93652-107-cache"}]: dispatch 2024-04-14T15:22:33.457 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:33.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:33.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:22:33.558 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:22:33.581 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:33.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:33.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:22:33.683 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:22:33.706 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:33.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:33.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:22:33.808 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:22:33.831 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:33.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:33.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:22:33.933 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:22:33.955 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:33.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:33.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:22:34.057 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:34.081 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:34.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:34.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:34.183 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:22:34.191 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:22:33 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:22:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:22:34.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:34 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1465271679' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-107", "tierpool": "test-rados-api-smithi090-93652-107-cache"}]': finished 2024-04-14T15:22:34.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:34 smithi090 ceph-mon[41670]: osdmap e617: 8 total, 8 up, 7 in 2024-04-14T15:22:34.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:34 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1465271679' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-107", "overlaypool": "test-rados-api-smithi090-93652-107-cache"}]: dispatch 2024-04-14T15:22:34.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:34 smithi090 ceph-mon[41670]: pgmap v906: 250 pgs: 32 unknown, 32 creating+peering, 3 active+remapped+backfill_toofull, 183 active+clean; 596 KiB data, 496 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s; 2/600 objects misplaced (0.333%); 27 B/s, 2 objects/s recovering 2024-04-14T15:22:34.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:34 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1465271679' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-107", "overlaypool": "test-rados-api-smithi090-93652-107-cache"}]': finished 2024-04-14T15:22:34.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:34 smithi090 ceph-mon[41670]: osdmap e618: 8 total, 8 up, 7 in 2024-04-14T15:22:34.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:34 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1465271679' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-107-cache", "mode": "writeback"}]: dispatch 2024-04-14T15:22:34.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:34 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1465271679' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-107", "tierpool": "test-rados-api-smithi090-93652-107-cache"}]': finished 2024-04-14T15:22:34.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:34 smithi090 ceph-mon[30399]: osdmap e617: 8 total, 8 up, 7 in 2024-04-14T15:22:34.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:34 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1465271679' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-107", "overlaypool": "test-rados-api-smithi090-93652-107-cache"}]: dispatch 2024-04-14T15:22:34.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:34 smithi090 ceph-mon[30399]: pgmap v906: 250 pgs: 32 unknown, 32 creating+peering, 3 active+remapped+backfill_toofull, 183 active+clean; 596 KiB data, 496 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s; 2/600 objects misplaced (0.333%); 27 B/s, 2 objects/s recovering 2024-04-14T15:22:34.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:34 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1465271679' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-107", "overlaypool": "test-rados-api-smithi090-93652-107-cache"}]': finished 2024-04-14T15:22:34.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:34 smithi090 ceph-mon[30399]: osdmap e618: 8 total, 8 up, 7 in 2024-04-14T15:22:34.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:34 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1465271679' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-107-cache", "mode": "writeback"}]: dispatch 2024-04-14T15:22:34.206 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:34.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:34.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:22:34.308 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:22:34.331 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:34.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:34.332 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:22:34.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:34 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1465271679' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-107", "tierpool": "test-rados-api-smithi090-93652-107-cache"}]': finished 2024-04-14T15:22:34.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:34 smithi092 ceph-mon[33966]: osdmap e617: 8 total, 8 up, 7 in 2024-04-14T15:22:34.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:34 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1465271679' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-107", "overlaypool": "test-rados-api-smithi090-93652-107-cache"}]: dispatch 2024-04-14T15:22:34.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:34 smithi092 ceph-mon[33966]: pgmap v906: 250 pgs: 32 unknown, 32 creating+peering, 3 active+remapped+backfill_toofull, 183 active+clean; 596 KiB data, 496 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s; 2/600 objects misplaced (0.333%); 27 B/s, 2 objects/s recovering 2024-04-14T15:22:34.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:34 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1465271679' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-107", "overlaypool": "test-rados-api-smithi090-93652-107-cache"}]': finished 2024-04-14T15:22:34.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:34 smithi092 ceph-mon[33966]: osdmap e618: 8 total, 8 up, 7 in 2024-04-14T15:22:34.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:34 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1465271679' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-107-cache", "mode": "writeback"}]: dispatch 2024-04-14T15:22:34.433 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:22:34.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:34.458 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:34.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:22:34.558 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:22:34.582 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:34.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:34.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:22:34.685 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:22:34.709 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:34.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:34.710 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:22:34.797 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-14T15:22:34.811 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:22:34.865 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:34.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:34.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:22:34.966 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:34.990 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:34.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:34.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:35.093 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:22:35.117 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:35.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:35.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:22:35.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:35 smithi090 ceph-mon[30399]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:22:35.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:35 smithi090 ceph-mon[30399]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:22:35.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:35 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1465271679' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-107-cache", "mode": "writeback"}]': finished 2024-04-14T15:22:35.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:35 smithi090 ceph-mon[30399]: osdmap e619: 8 total, 8 up, 7 in 2024-04-14T15:22:35.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:35 smithi090 ceph-mon[41670]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:22:35.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:35 smithi090 ceph-mon[41670]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:22:35.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:35 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1465271679' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-107-cache", "mode": "writeback"}]': finished 2024-04-14T15:22:35.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:35 smithi090 ceph-mon[41670]: osdmap e619: 8 total, 8 up, 7 in 2024-04-14T15:22:35.219 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:22:35.243 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:35.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:35.244 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:22:35.253 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph tell osd.4 reset_purged_snaps_last 2024-04-14T15:22:35.302 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:35 smithi092 ceph-mon[33966]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:22:35.302 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:35 smithi092 ceph-mon[33966]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:22:35.303 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:35 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1465271679' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-107-cache", "mode": "writeback"}]': finished 2024-04-14T15:22:35.303 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:35 smithi092 ceph-mon[33966]: osdmap e619: 8 total, 8 up, 7 in 2024-04-14T15:22:35.345 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:22:35.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:35.369 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:35.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:22:35.470 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:22:35.497 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:35.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:35.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:22:35.598 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:22:35.622 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:22:35 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:22:35.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:35.623 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:35.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:22:35.724 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:22:35.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:35.745+0000 7fe73c1dd700 1 -- 172.21.15.90:0/2014278151 >> v1:172.21.15.90:6789/0 conn(0x7fe73410e1e0 legacy=0x7fe7341106a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:35.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:35.745+0000 7fe73c1dd700 1 -- 172.21.15.90:0/2014278151 shutdown_connections 2024-04-14T15:22:35.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:35.745+0000 7fe73c1dd700 1 -- 172.21.15.90:0/2014278151 >> 172.21.15.90:0/2014278151 conn(0x7fe7340fd290 msgr2=0x7fe7340ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:22:35.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:35.745+0000 7fe73c1dd700 1 -- 172.21.15.90:0/2014278151 shutdown_connections 2024-04-14T15:22:35.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:35.745+0000 7fe73c1dd700 1 -- 172.21.15.90:0/2014278151 wait complete. 2024-04-14T15:22:35.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:35.746+0000 7fe73c1dd700 1 librados: starting msgr at 2024-04-14T15:22:35.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:35.746+0000 7fe73c1dd700 1 librados: starting objecter 2024-04-14T15:22:35.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:35.746+0000 7fe73c1dd700 1 Processor -- start 2024-04-14T15:22:35.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:35.746+0000 7fe73c1dd700 1 -- start start 2024-04-14T15:22:35.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:35.746+0000 7fe73c1dd700 1 librados: setting wanted keys 2024-04-14T15:22:35.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:35.746+0000 7fe73c1dd700 1 librados: calling monclient init 2024-04-14T15:22:35.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:35.746+0000 7fe73c1dd700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe73410df30 con 0x7fe734107760 2024-04-14T15:22:35.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:35.746+0000 7fe73c1dd700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe7341a7df0 con 0x7fe73410e1e0 2024-04-14T15:22:35.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:35.746+0000 7fe73c1dd700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe7341a8fd0 con 0x7fe73410a610 2024-04-14T15:22:35.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:35.746+0000 7fe739f79700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fe734107760 0x7fe73410c6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:51372/0 (socket says 172.21.15.90:51372) 2024-04-14T15:22:35.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:35.746+0000 7fe739778700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7fe73410a610 0x7fe73410ce00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:56198/0 (socket says 172.21.15.90:56198) 2024-04-14T15:22:35.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:35.746+0000 7fe739f79700 1 -- 172.21.15.90:0/3988629816 learned_addr learned my addr 172.21.15.90:0/3988629816 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:22:35.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:35.746+0000 7fe72affd700 10 client.?.objecter ms_handle_connect 0x7fe73410a610 2024-04-14T15:22:35.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:35.746+0000 7fe72affd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:22:35.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:35.746+0000 7fe72affd700 10 client.?.objecter ms_handle_connect 0x7fe734107760 2024-04-14T15:22:35.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:35.746+0000 7fe72affd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:22:35.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:35.746+0000 7fe72affd700 10 client.?.objecter ms_handle_connect 0x7fe73410e1e0 2024-04-14T15:22:35.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:35.746+0000 7fe72affd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:22:35.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:35.746+0000 7fe72affd700 1 -- 172.21.15.90:0/3988629816 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2448324659 0 0) 0x7fe7341a8fd0 con 0x7fe73410a610 2024-04-14T15:22:35.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:35.746+0000 7fe72affd700 1 -- 172.21.15.90:0/3988629816 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe7200035b0 con 0x7fe73410a610 2024-04-14T15:22:35.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:35.747+0000 7fe72affd700 1 -- 172.21.15.90:0/3988629816 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3281739484 0 0) 0x7fe73410df30 con 0x7fe734107760 2024-04-14T15:22:35.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:35.747+0000 7fe72affd700 1 -- 172.21.15.90:0/3988629816 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe7341a8fd0 con 0x7fe734107760 2024-04-14T15:22:35.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:35.747+0000 7fe72affd700 1 -- 172.21.15.90:0/3988629816 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2443082118 0 0) 0x7fe7341a7df0 con 0x7fe73410e1e0 2024-04-14T15:22:35.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:35.747+0000 7fe72affd700 1 -- 172.21.15.90:0/3988629816 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe73410df30 con 0x7fe73410e1e0 2024-04-14T15:22:35.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:35.747+0000 7fe72affd700 1 -- 172.21.15.90:0/3988629816 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4213710959 0 0) 0x7fe7200035b0 con 0x7fe73410a610 2024-04-14T15:22:35.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:35.747+0000 7fe72affd700 1 -- 172.21.15.90:0/3988629816 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe7341a7df0 con 0x7fe73410a610 2024-04-14T15:22:35.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:35.747+0000 7fe72affd700 1 -- 172.21.15.90:0/3988629816 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fe7300018b0 con 0x7fe73410a610 2024-04-14T15:22:35.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:35.747+0000 7fe72affd700 1 -- 172.21.15.90:0/3988629816 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1358997142 0 0) 0x7fe7341a8fd0 con 0x7fe734107760 2024-04-14T15:22:35.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:35.747+0000 7fe72affd700 1 -- 172.21.15.90:0/3988629816 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe7200035b0 con 0x7fe734107760 2024-04-14T15:22:35.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:35.747+0000 7fe72affd700 1 -- 172.21.15.90:0/3988629816 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3871498358 0 0) 0x7fe73410df30 con 0x7fe73410e1e0 2024-04-14T15:22:35.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:35.747+0000 7fe72affd700 1 -- 172.21.15.90:0/3988629816 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe7341a8fd0 con 0x7fe73410e1e0 2024-04-14T15:22:35.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:35.747+0000 7fe72affd700 1 -- 172.21.15.90:0/3988629816 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fe724001870 con 0x7fe734107760 2024-04-14T15:22:35.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:35.747+0000 7fe72affd700 1 -- 172.21.15.90:0/3988629816 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fe72c001bb0 con 0x7fe73410e1e0 2024-04-14T15:22:35.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:35.747+0000 7fe72affd700 1 -- 172.21.15.90:0/3988629816 <== mon.2 v1:172.21.15.90:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4192337503 0 0) 0x7fe7341a7df0 con 0x7fe73410a610 2024-04-14T15:22:35.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:35.747+0000 7fe72affd700 1 -- 172.21.15.90:0/3988629816 >> v1:172.21.15.92:6789/0 conn(0x7fe73410e1e0 legacy=0x7fe7341a56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:35.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:35.747+0000 7fe72affd700 1 -- 172.21.15.90:0/3988629816 >> v1:172.21.15.90:6789/0 conn(0x7fe734107760 legacy=0x7fe73410c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:35.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:35.747+0000 7fe72affd700 1 -- 172.21.15.90:0/3988629816 --> v1:172.21.15.90:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7341aa210 con 0x7fe73410a610 2024-04-14T15:22:35.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:35.747+0000 7fe73c1dd700 1 -- 172.21.15.90:0/3988629816 --> v1:172.21.15.90:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe7341a6e40 con 0x7fe73410a610 2024-04-14T15:22:35.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:35.747+0000 7fe73c1dd700 10 client.47645.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:22:35.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:35.747+0000 7fe73c1dd700 1 -- 172.21.15.90:0/3988629816 --> v1:172.21.15.90:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fe7341a7300 con 0x7fe73410a610 2024-04-14T15:22:35.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:35.747+0000 7fe73c1dd700 1 librados: init done 2024-04-14T15:22:35.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:35.747+0000 7fe72affd700 1 -- 172.21.15.90:0/3988629816 <== mon.2 v1:172.21.15.90:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fe730003b20 con 0x7fe73410a610 2024-04-14T15:22:35.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:35.747+0000 7fe72affd700 1 -- 172.21.15.90:0/3988629816 <== mon.2 v1:172.21.15.90:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fe730005c30 con 0x7fe73410a610 2024-04-14T15:22:35.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:35.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:35.748+0000 7fe73c1dd700 1 -- 172.21.15.90:0/3988629816 --> v1:172.21.15.90:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe718005150 con 0x7fe73410a610 2024-04-14T15:22:35.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:35.749+0000 7fe72affd700 1 -- 172.21.15.90:0/3988629816 <== mon.2 v1:172.21.15.90:6790/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fe730003680 con 0x7fe73410a610 2024-04-14T15:22:35.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:35.749+0000 7fe72affd700 1 -- 172.21.15.90:0/3988629816 <== mon.2 v1:172.21.15.90:6790/0 8 ==== osd_map(619..619 src has 467..619) ==== 9185+0+0 (unknown 1161791052 0 0) 0x7fe73008ad10 con 0x7fe73410a610 2024-04-14T15:22:35.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:35.749+0000 7fe72affd700 10 client.47645.objecter ms_dispatch 0x7fe734001550 osd_map(619..619 src has 467..619) 2024-04-14T15:22:35.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:35.749+0000 7fe72affd700 3 client.47645.objecter handle_osd_map got epochs [619,619] > 0 2024-04-14T15:22:35.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:35.749+0000 7fe72affd700 3 client.47645.objecter handle_osd_map decoding full epoch 619 2024-04-14T15:22:35.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:35.749+0000 7fe72affd700 20 client.47645.objecter dump_active .. 0 homeless 2024-04-14T15:22:35.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:35.749+0000 7fe72affd700 10 client.47645.objecter ms_handle_connect 0x7fe7200736c0 2024-04-14T15:22:35.754 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:35.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:22:35.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:35.754+0000 7fe72affd700 1 -- 172.21.15.90:0/3988629816 <== mon.2 v1:172.21.15.90:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fe730057340 con 0x7fe73410a610 2024-04-14T15:22:35.855 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:35.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:35.879 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:35.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:35.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:35.973+0000 7fe73c1dd700 1 -- 172.21.15.90:0/3988629816 --> v1:172.21.15.90:6790/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7fe718005440 con 0x7fe73410a610 2024-04-14T15:22:35.980 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:22:36.004 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:36.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:36.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:22:36.105 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:22:36.129 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:36.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:36.130 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:22:36.231 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:22:36.254 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:36.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:36.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:22:36.356 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:22:36.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:36 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1465271679' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-107"}]: dispatch 2024-04-14T15:22:36.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:36 smithi092 ceph-mon[33966]: pgmap v909: 250 pgs: 3 active+remapped+backfill_toofull, 247 active+clean; 596 KiB data, 519 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/600 objects misplaced (0.333%) 2024-04-14T15:22:36.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:36 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:22:36.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:36 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1465271679' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-107"}]': finished 2024-04-14T15:22:36.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:36 smithi092 ceph-mon[33966]: osdmap e620: 8 total, 8 up, 7 in 2024-04-14T15:22:36.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:36 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1465271679' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-107", "tierpool": "test-rados-api-smithi090-93652-107-cache"}]: dispatch 2024-04-14T15:22:36.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:36 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3988629816' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:22:36.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:36 smithi092 ceph-mon[33966]: from='client.47645 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:22:36.375 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:36 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1465271679' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-107"}]: dispatch 2024-04-14T15:22:36.375 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:36 smithi090 ceph-mon[41670]: pgmap v909: 250 pgs: 3 active+remapped+backfill_toofull, 247 active+clean; 596 KiB data, 519 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/600 objects misplaced (0.333%) 2024-04-14T15:22:36.375 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:36 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:22:36.375 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:36 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1465271679' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-107"}]': finished 2024-04-14T15:22:36.375 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:36 smithi090 ceph-mon[41670]: osdmap e620: 8 total, 8 up, 7 in 2024-04-14T15:22:36.375 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:36 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1465271679' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-107", "tierpool": "test-rados-api-smithi090-93652-107-cache"}]: dispatch 2024-04-14T15:22:36.375 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:36 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3988629816' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:22:36.375 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:36 smithi090 ceph-mon[41670]: from='client.47645 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:22:36.376 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:36 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1465271679' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-107"}]: dispatch 2024-04-14T15:22:36.376 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:36 smithi090 ceph-mon[30399]: pgmap v909: 250 pgs: 3 active+remapped+backfill_toofull, 247 active+clean; 596 KiB data, 519 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/600 objects misplaced (0.333%) 2024-04-14T15:22:36.376 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:36 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:22:36.376 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:36 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1465271679' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-107"}]': finished 2024-04-14T15:22:36.376 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:36 smithi090 ceph-mon[30399]: osdmap e620: 8 total, 8 up, 7 in 2024-04-14T15:22:36.376 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:36 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1465271679' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-107", "tierpool": "test-rados-api-smithi090-93652-107-cache"}]: dispatch 2024-04-14T15:22:36.376 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:36 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3988629816' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:22:36.376 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:36 smithi090 ceph-mon[30399]: from='client.47645 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:22:36.380 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:36.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:36.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:22:36.482 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:22:36.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:36.508 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:36.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:22:36.609 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:22:36.633 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:36.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:36.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:22:36.735 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:22:36.759 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:36.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:36.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:22:36.860 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:22:36.869 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:22:36.869 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:22:36.886 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:36.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:36.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:22:36.915 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-14T15:22:36.915 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:22:36.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:36.971+0000 7fe72affd700 1 -- 172.21.15.90:0/3988629816 <== mon.2 v1:172.21.15.90:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v621) ==== 97+0+0 (unknown 1937665495 0 0) 0x7fe73005aff0 con 0x7fe73410a610 2024-04-14T15:22:36.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:36.974+0000 7fe73c1dd700 10 librados: watch_flush enter 2024-04-14T15:22:36.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:36.974+0000 7fe73c1dd700 10 librados: watch_flush exit 2024-04-14T15:22:36.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:36.974+0000 7fe73c1dd700 20 client.47645.objecter shutdown clearing up homeless session... 2024-04-14T15:22:36.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:36.974+0000 7fe73c1dd700 10 client.47645.objecter successfully canceled tick 2024-04-14T15:22:36.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:36.974+0000 7fe73c1dd700 1 -- 172.21.15.90:0/3988629816 >> v1:172.21.15.90:6800/420935427 conn(0x7fe7200736c0 legacy=0x7fe720075b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:36.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:36.974+0000 7fe73c1dd700 1 -- 172.21.15.90:0/3988629816 >> v1:172.21.15.90:6790/0 conn(0x7fe73410a610 legacy=0x7fe73410ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:36.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:36.974+0000 7fe73c1dd700 1 -- 172.21.15.90:0/3988629816 shutdown_connections 2024-04-14T15:22:36.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:36.974+0000 7fe73c1dd700 1 -- 172.21.15.90:0/3988629816 >> 172.21.15.90:0/3988629816 conn(0x7fe7340fd290 msgr2=0x7fe734101760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:22:36.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:36.975+0000 7fe73c1dd700 1 -- 172.21.15.90:0/3988629816 shutdown_connections 2024-04-14T15:22:36.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:36.975+0000 7fe73c1dd700 1 -- 172.21.15.90:0/3988629816 wait complete. 2024-04-14T15:22:36.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:36.975+0000 7fe73c1dd700 1 librados: shutdown 2024-04-14T15:22:36.976 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is set 2024-04-14T15:22:36.988 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:22:37.014 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:37.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:37.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:22:37.115 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:22:37.140 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:37.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:37.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:22:37.166 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:37 smithi090 ceph-mon[41670]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:22:37.166 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:37 smithi090 ceph-mon[41670]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:22:37.166 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:37 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1465271679' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-107", "tierpool": "test-rados-api-smithi090-93652-107-cache"}]': finished 2024-04-14T15:22:37.166 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:37 smithi090 ceph-mon[41670]: from='client.47645 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-14T15:22:37.166 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:37 smithi090 ceph-mon[41670]: osdmap e621: 8 total, 8 up, 7 in 2024-04-14T15:22:37.166 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:37 smithi090 ceph-mon[30399]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:22:37.166 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:37 smithi090 ceph-mon[30399]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:22:37.167 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:37 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1465271679' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-107", "tierpool": "test-rados-api-smithi090-93652-107-cache"}]': finished 2024-04-14T15:22:37.167 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:37 smithi090 ceph-mon[30399]: from='client.47645 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-14T15:22:37.167 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:37 smithi090 ceph-mon[30399]: osdmap e621: 8 total, 8 up, 7 in 2024-04-14T15:22:37.242 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:22:37.267 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:37.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:37.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:22:37.368 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:22:37.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:37 smithi092 ceph-mon[33966]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:22:37.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:37 smithi092 ceph-mon[33966]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:22:37.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:37 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1465271679' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-107", "tierpool": "test-rados-api-smithi090-93652-107-cache"}]': finished 2024-04-14T15:22:37.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:37 smithi092 ceph-mon[33966]: from='client.47645 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-14T15:22:37.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:37 smithi092 ceph-mon[33966]: osdmap e621: 8 total, 8 up, 7 in 2024-04-14T15:22:37.393 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:37.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:37.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:22:37.495 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:37.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:37.520 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:37.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:37.621 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:37.644 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:37.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:37.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:37.746 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:22:37.774 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:37.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:37.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:22:37.876 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:37.901 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:37.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:37.902 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:38.002 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:22:38.030 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:38.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:38.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:22:38.131 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:22:38.154 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:38.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:38.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:22:38.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:38 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:22:38.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:38 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:22:38.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:38 smithi090 ceph-mon[41670]: pgmap v912: 250 pgs: 3 active+remapped+backfill_toofull, 247 active+clean; 596 KiB data, 519 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/600 objects misplaced (0.333%) 2024-04-14T15:22:38.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:38 smithi090 ceph-mon[41670]: osdmap e622: 8 total, 8 up, 7 in 2024-04-14T15:22:38.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:38 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:22:38.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:38 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:22:38.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:38 smithi090 ceph-mon[30399]: pgmap v912: 250 pgs: 3 active+remapped+backfill_toofull, 247 active+clean; 596 KiB data, 519 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/600 objects misplaced (0.333%) 2024-04-14T15:22:38.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:38 smithi090 ceph-mon[30399]: osdmap e622: 8 total, 8 up, 7 in 2024-04-14T15:22:38.256 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:38.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:38.280 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:38.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:38.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:38 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:22:38.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:38 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:22:38.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:38 smithi092 ceph-mon[33966]: pgmap v912: 250 pgs: 3 active+remapped+backfill_toofull, 247 active+clean; 596 KiB data, 519 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/600 objects misplaced (0.333%) 2024-04-14T15:22:38.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:38 smithi092 ceph-mon[33966]: osdmap e622: 8 total, 8 up, 7 in 2024-04-14T15:22:38.381 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:22:38.408 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:38.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:38.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:22:38.510 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:22:38.539 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:38.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:38.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:22:38.641 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:22:38.669 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:38.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:38.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:22:38.770 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:22:38.798 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:38.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:38.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:22:38.900 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:22:38.925 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:38.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:38.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:22:38.935 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-14T15:22:39.028 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:22:39.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:39.053 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:39.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:22:39.153 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:22:39.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:39.178 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:39.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:22:39.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:39 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:22:39.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:39 smithi090 ceph-mon[41670]: osdmap e623: 8 total, 8 up, 7 in 2024-04-14T15:22:39.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:39 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi090-93652-109"}]: dispatch 2024-04-14T15:22:39.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:39 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi090-93652-109"}]: dispatch 2024-04-14T15:22:39.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:39 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi090-93652-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:22:39.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:39 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:22:39.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:39 smithi090 ceph-mon[30399]: osdmap e623: 8 total, 8 up, 7 in 2024-04-14T15:22:39.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:39 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi090-93652-109"}]: dispatch 2024-04-14T15:22:39.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:39 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi090-93652-109"}]: dispatch 2024-04-14T15:22:39.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:39 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi090-93652-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:22:39.279 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:22:39.306 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:39.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:39.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:22:39.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:39 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:22:39.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:39 smithi092 ceph-mon[33966]: osdmap e623: 8 total, 8 up, 7 in 2024-04-14T15:22:39.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:39 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi090-93652-109"}]: dispatch 2024-04-14T15:22:39.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:39 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi090-93652-109"}]: dispatch 2024-04-14T15:22:39.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:39 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi090-93652-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:22:39.407 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:22:39.435 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:39.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:39.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:22:39.536 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:22:39.564 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:39.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:39.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:22:39.666 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:22:39.693 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:39.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:39.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:22:39.794 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:22:39.822 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:39.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:39.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:22:39.924 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:22:39.949 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:39.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:39.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:22:40.050 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:22:40.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:40.075 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:40.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:22:40.176 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:22:40.199 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:40.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:40.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:22:40.300 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:40.327 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:40.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:40.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:40.429 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:22:40.457 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:40.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:40.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:22:40.559 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:22:40.582 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:40.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:40.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:22:40.684 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:22:40.712 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:40.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:40.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:22:40.813 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:22:40.836 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:40.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:40.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:22:40.938 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:22:40.962 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:40.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:40.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:22:41.063 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:41.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:41.088 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:41.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:41.089 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:41 smithi090 ceph-mon[41670]: pgmap v915: 186 pgs: 3 active+remapped+backfill_toofull, 183 active+clean; 596 KiB data, 519 MiB used, 625 GiB / 626 GiB avail; 2/600 objects misplaced (0.333%) 2024-04-14T15:22:41.089 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:41 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi090-93652-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:22:41.089 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:41 smithi090 ceph-mon[41670]: osdmap e624: 8 total, 8 up, 7 in 2024-04-14T15:22:41.089 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:41 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi090-93652-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi090-93652-109"}]: dispatch 2024-04-14T15:22:41.089 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:41 smithi090 ceph-mon[30399]: pgmap v915: 186 pgs: 3 active+remapped+backfill_toofull, 183 active+clean; 596 KiB data, 519 MiB used, 625 GiB / 626 GiB avail; 2/600 objects misplaced (0.333%) 2024-04-14T15:22:41.089 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:41 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi090-93652-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:22:41.089 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:41 smithi090 ceph-mon[30399]: osdmap e624: 8 total, 8 up, 7 in 2024-04-14T15:22:41.089 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:41 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi090-93652-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi090-93652-109"}]: dispatch 2024-04-14T15:22:41.189 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:22:41.215 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:41.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:41.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:22:41.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.289+0000 7f4568f11700 1 -- 172.21.15.90:0/975176541 >> v1:172.21.15.90:6789/0 conn(0x7f456410f300 legacy=0x7f45641117c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:41.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.289+0000 7f4568f11700 1 -- 172.21.15.90:0/975176541 shutdown_connections 2024-04-14T15:22:41.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.289+0000 7f4568f11700 1 -- 172.21.15.90:0/975176541 >> 172.21.15.90:0/975176541 conn(0x7f45640fe390 msgr2=0x7f45641007b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:22:41.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.289+0000 7f4568f11700 1 -- 172.21.15.90:0/975176541 shutdown_connections 2024-04-14T15:22:41.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.289+0000 7f4568f11700 1 -- 172.21.15.90:0/975176541 wait complete. 2024-04-14T15:22:41.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.290+0000 7f4568f11700 1 librados: starting msgr at 2024-04-14T15:22:41.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.290+0000 7f4568f11700 1 librados: starting objecter 2024-04-14T15:22:41.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.290+0000 7f4568f11700 1 Processor -- start 2024-04-14T15:22:41.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.290+0000 7f4568f11700 1 -- start start 2024-04-14T15:22:41.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.290+0000 7f4568f11700 1 librados: setting wanted keys 2024-04-14T15:22:41.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.290+0000 7f4568f11700 1 librados: calling monclient init 2024-04-14T15:22:41.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.290+0000 7f4568f11700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f45641a78f0 con 0x7f456410b730 2024-04-14T15:22:41.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.290+0000 7f4568f11700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f45641a8b30 con 0x7f456410f300 2024-04-14T15:22:41.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.290+0000 7f4568f11700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f45641a9d70 con 0x7f4564108880 2024-04-14T15:22:41.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.291+0000 7f456259c700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f4564108880 0x7f456410e9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:35208/0 (socket says 172.21.15.90:35208) 2024-04-14T15:22:41.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.291+0000 7f456259c700 1 -- 172.21.15.90:0/2464633671 learned_addr learned my addr 172.21.15.90:0/2464633671 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:22:41.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.291+0000 7f4561d9b700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f456410b730 0x7f45641a27b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50144/0 (socket says 172.21.15.90:50144) 2024-04-14T15:22:41.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.291+0000 7f45537fe700 10 client.?.objecter ms_handle_connect 0x7f456410b730 2024-04-14T15:22:41.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.291+0000 7f45537fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:22:41.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.291+0000 7f45537fe700 10 client.?.objecter ms_handle_connect 0x7f4564108880 2024-04-14T15:22:41.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.291+0000 7f45537fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:22:41.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.291+0000 7f45537fe700 1 -- 172.21.15.90:0/2464633671 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 147464379 0 0) 0x7f45641a78f0 con 0x7f456410b730 2024-04-14T15:22:41.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.291+0000 7f45537fe700 1 -- 172.21.15.90:0/2464633671 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f454c0031f0 con 0x7f456410b730 2024-04-14T15:22:41.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.291+0000 7f45537fe700 1 -- 172.21.15.90:0/2464633671 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2415066912 0 0) 0x7f45641a9d70 con 0x7f4564108880 2024-04-14T15:22:41.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.291+0000 7f45537fe700 1 -- 172.21.15.90:0/2464633671 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f45641a78f0 con 0x7f4564108880 2024-04-14T15:22:41.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.291+0000 7f45537fe700 10 client.?.objecter ms_handle_connect 0x7f456410f300 2024-04-14T15:22:41.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.291+0000 7f45537fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:22:41.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.291+0000 7f45537fe700 1 -- 172.21.15.90:0/2464633671 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 166663379 0 0) 0x7f454c0031f0 con 0x7f456410b730 2024-04-14T15:22:41.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.291+0000 7f45537fe700 1 -- 172.21.15.90:0/2464633671 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f45641a9d70 con 0x7f456410b730 2024-04-14T15:22:41.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.291+0000 7f45537fe700 1 -- 172.21.15.90:0/2464633671 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f4558001910 con 0x7f456410b730 2024-04-14T15:22:41.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.291+0000 7f45537fe700 1 -- 172.21.15.90:0/2464633671 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3940467107 0 0) 0x7f45641a8b30 con 0x7f456410f300 2024-04-14T15:22:41.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.291+0000 7f45537fe700 1 -- 172.21.15.90:0/2464633671 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f454c0031f0 con 0x7f456410f300 2024-04-14T15:22:41.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.291+0000 7f45537fe700 1 -- 172.21.15.90:0/2464633671 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2110782259 0 0) 0x7f45641a78f0 con 0x7f4564108880 2024-04-14T15:22:41.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.291+0000 7f45537fe700 1 -- 172.21.15.90:0/2464633671 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f45641a8b30 con 0x7f4564108880 2024-04-14T15:22:41.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.291+0000 7f45537fe700 1 -- 172.21.15.90:0/2464633671 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f4554001230 con 0x7f4564108880 2024-04-14T15:22:41.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.291+0000 7f45537fe700 1 -- 172.21.15.90:0/2464633671 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2892528195 0 0) 0x7f45641a9d70 con 0x7f456410b730 2024-04-14T15:22:41.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.291+0000 7f45537fe700 1 -- 172.21.15.90:0/2464633671 >> v1:172.21.15.90:6790/0 conn(0x7f4564108880 legacy=0x7f456410e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:41.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.291+0000 7f45537fe700 1 -- 172.21.15.90:0/2464633671 >> v1:172.21.15.92:6789/0 conn(0x7f456410f300 legacy=0x7f45641a5ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:41.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.291+0000 7f45537fe700 1 -- 172.21.15.90:0/2464633671 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45641aafb0 con 0x7f456410b730 2024-04-14T15:22:41.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.291+0000 7f4568f11700 1 -- 172.21.15.90:0/2464633671 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f45641a9fa0 con 0x7f456410b730 2024-04-14T15:22:41.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.291+0000 7f4568f11700 10 client.37707.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:22:41.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.291+0000 7f4568f11700 1 -- 172.21.15.90:0/2464633671 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f45641aa460 con 0x7f456410b730 2024-04-14T15:22:41.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.291+0000 7f4568f11700 1 librados: init done 2024-04-14T15:22:41.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.291+0000 7f45537fe700 1 -- 172.21.15.90:0/2464633671 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f4558003b20 con 0x7f456410b730 2024-04-14T15:22:41.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.292+0000 7f45537fe700 1 -- 172.21.15.90:0/2464633671 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f4558005c80 con 0x7f456410b730 2024-04-14T15:22:41.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.292+0000 7f4568f11700 10 client.37707.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:22:41.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.292+0000 7f4568f11700 15 client.37707.objecter _session_command_op_assign -1 1 2024-04-14T15:22:41.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.292+0000 7f4568f11700 20 client.37707.objecter _get_session osd=-1 returning homeless 2024-04-14T15:22:41.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.292+0000 7f4568f11700 10 client.37707.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:22:41.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.292+0000 7f4568f11700 1 -- 172.21.15.90:0/2464633671 --> v1:172.21.15.90:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f4544002920 con 0x7f456410b730 2024-04-14T15:22:41.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.293+0000 7f45537fe700 1 -- 172.21.15.90:0/2464633671 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f4558003680 con 0x7f456410b730 2024-04-14T15:22:41.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.294+0000 7f45537fe700 1 -- 172.21.15.90:0/2464633671 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(625..625 src has 467..625) ==== 8839+0+0 (unknown 2615881034 0 0) 0x7f455808ad70 con 0x7f456410b730 2024-04-14T15:22:41.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.294+0000 7f45537fe700 10 client.37707.objecter ms_dispatch 0x7f4564001550 osd_map(625..625 src has 467..625) 2024-04-14T15:22:41.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.294+0000 7f45537fe700 3 client.37707.objecter handle_osd_map got epochs [625,625] > 0 2024-04-14T15:22:41.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.294+0000 7f45537fe700 3 client.37707.objecter handle_osd_map decoding full epoch 625 2024-04-14T15:22:41.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.294+0000 7f45537fe700 10 client.37707.objecter checking command 1 2024-04-14T15:22:41.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.294+0000 7f45537fe700 20 client.37707.objecter _get_session s=0x7f454c07e8c0 osd=4 3 2024-04-14T15:22:41.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.294+0000 7f45537fe700 20 client.37707.objecter put_session s=0x7f454c07e8c0 osd=4 3 2024-04-14T15:22:41.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.294+0000 7f45537fe700 15 client.37707.objecter _session_command_op_remove -1 1 2024-04-14T15:22:41.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.294+0000 7f45537fe700 20 client.37707.objecter _get_session s=0x7f454c07e8c0 osd=4 3 2024-04-14T15:22:41.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.294+0000 7f45537fe700 20 client.37707.objecter get_session s=0x7f454c07e8c0 osd=4 3 2024-04-14T15:22:41.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.294+0000 7f45537fe700 15 client.37707.objecter _session_command_op_assign 4 1 2024-04-14T15:22:41.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.294+0000 7f45537fe700 20 client.37707.objecter put_session s=0x7f454c07e8c0 osd=4 4 2024-04-14T15:22:41.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.294+0000 7f45537fe700 10 client.37707.objecter _send_command 1 2024-04-14T15:22:41.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.294+0000 7f45537fe700 1 -- 172.21.15.90:0/2464633671 --> v1:172.21.15.92:6800/3887549733 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f454c087090 con 0x7f454c07ef70 2024-04-14T15:22:41.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.294+0000 7f45537fe700 20 client.37707.objecter dump_active .. 0 homeless 2024-04-14T15:22:41.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.294+0000 7f45537fe700 1 -- 172.21.15.90:0/2464633671 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_get_version_reply(handle=1 version=625) ==== 24+0+0 (unknown 4238086412 0 0) 0x7f455808b180 con 0x7f456410b730 2024-04-14T15:22:41.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.294+0000 7f45537fe700 10 client.37707.objecter ms_handle_connect 0x7f454c071380 2024-04-14T15:22:41.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.294+0000 7f45537fe700 10 client.37707.objecter ms_handle_connect 0x7f454c07ef70 2024-04-14T15:22:41.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.295+0000 7f45537fe700 1 -- 172.21.15.90:0/2464633671 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f454c087090 con 0x7f454c07ef70 2024-04-14T15:22:41.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.295+0000 7f45537fe700 10 client.37707.objecter ms_dispatch 0x7f4564001550 command_reply(tid 1: 0 ) 2024-04-14T15:22:41.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.295+0000 7f45537fe700 10 client.37707.objecter _finish_command 1 = system:0 2024-04-14T15:22:41.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.295+0000 7f45537fe700 20 client.37707.objecter put_session s=0x7f454c07e8c0 osd=4 4 2024-04-14T15:22:41.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.295+0000 7f45537fe700 15 client.37707.objecter _session_command_op_remove 4 1 2024-04-14T15:22:41.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.314+0000 7f4568f11700 10 client.37707.objecter _submit_command 2 [{"prefix": "reset_purged_snaps_last"}] 2024-04-14T15:22:41.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.314+0000 7f4568f11700 15 client.37707.objecter _session_command_op_assign -1 2 2024-04-14T15:22:41.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.314+0000 7f4568f11700 20 client.37707.objecter _get_session s=0x7f454c07e8c0 osd=4 3 2024-04-14T15:22:41.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.314+0000 7f4568f11700 20 client.37707.objecter put_session s=0x7f454c07e8c0 osd=4 3 2024-04-14T15:22:41.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.314+0000 7f4568f11700 20 client.37707.objecter _get_session s=0x7f454c07e8c0 osd=4 3 2024-04-14T15:22:41.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.314+0000 7f4568f11700 15 client.37707.objecter _session_command_op_remove -1 2 2024-04-14T15:22:41.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.314+0000 7f4568f11700 20 client.37707.objecter get_session s=0x7f454c07e8c0 osd=4 3 2024-04-14T15:22:41.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.314+0000 7f4568f11700 15 client.37707.objecter _session_command_op_assign 4 2 2024-04-14T15:22:41.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.314+0000 7f4568f11700 20 client.37707.objecter put_session s=0x7f454c07e8c0 osd=4 4 2024-04-14T15:22:41.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.314+0000 7f4568f11700 10 client.37707.objecter _send_command 2 2024-04-14T15:22:41.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.314+0000 7f4568f11700 1 -- 172.21.15.90:0/2464633671 --> v1:172.21.15.92:6800/3887549733 -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) -- 0x7f45440011f0 con 0x7f454c07ef70 2024-04-14T15:22:41.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.315+0000 7f45537fe700 1 -- 172.21.15.90:0/2464633671 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 2: 0 ) ==== 8+0+0 (unknown 0 0 0) 0x7f45440011f0 con 0x7f454c07ef70 2024-04-14T15:22:41.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.315+0000 7f45537fe700 10 client.37707.objecter ms_dispatch 0x7f4564001550 command_reply(tid 2: 0 ) 2024-04-14T15:22:41.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.315+0000 7f45537fe700 10 client.37707.objecter _finish_command 2 = system:0 2024-04-14T15:22:41.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.315+0000 7f45537fe700 20 client.37707.objecter put_session s=0x7f454c07e8c0 osd=4 4 2024-04-14T15:22:41.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.315+0000 7f45537fe700 15 client.37707.objecter _session_command_op_remove 4 2 2024-04-14T15:22:41.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.315+0000 7f4568f11700 10 librados: watch_flush enter 2024-04-14T15:22:41.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.315+0000 7f4568f11700 10 librados: watch_flush exit 2024-04-14T15:22:41.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.315+0000 7f4568f11700 10 client.37707.objecter close_session for osd.4 2024-04-14T15:22:41.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.315+0000 7f4568f11700 1 -- 172.21.15.90:0/2464633671 >> v1:172.21.15.92:6800/3887549733 conn(0x7f454c07ef70 legacy=0x7f454c081410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:41.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.315+0000 7f4568f11700 20 client.37707.objecter put_session s=0x7f454c07e8c0 osd=4 1 2024-04-14T15:22:41.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.315+0000 7f4568f11700 20 client.37707.objecter shutdown clearing up homeless session... 2024-04-14T15:22:41.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.315+0000 7f4568f11700 10 client.37707.objecter successfully canceled tick 2024-04-14T15:22:41.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.315+0000 7f4568f11700 1 -- 172.21.15.90:0/2464633671 >> v1:172.21.15.90:6800/420935427 conn(0x7f454c071380 legacy=0x7f454c073840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:41.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.315+0000 7f4568f11700 1 -- 172.21.15.90:0/2464633671 >> v1:172.21.15.90:6789/0 conn(0x7f456410b730 legacy=0x7f45641a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:41.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.315+0000 7f4562d9d700 1 -- 172.21.15.90:0/2464633671 reap_dead start 2024-04-14T15:22:41.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.315+0000 7f4568f11700 1 -- 172.21.15.90:0/2464633671 shutdown_connections 2024-04-14T15:22:41.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.315+0000 7f4568f11700 1 -- 172.21.15.90:0/2464633671 >> 172.21.15.90:0/2464633671 conn(0x7f45640fe390 msgr2=0x7f4564111f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:22:41.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.315+0000 7f4568f11700 1 -- 172.21.15.90:0/2464633671 shutdown_connections 2024-04-14T15:22:41.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.315+0000 7f4568f11700 1 -- 172.21.15.90:0/2464633671 wait complete. 2024-04-14T15:22:41.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.315+0000 7f4568f11700 1 librados: shutdown 2024-04-14T15:22:41.318 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:22:41.343 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:41.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:41.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:22:41.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:41 smithi092 ceph-mon[33966]: pgmap v915: 186 pgs: 3 active+remapped+backfill_toofull, 183 active+clean; 596 KiB data, 519 MiB used, 625 GiB / 626 GiB avail; 2/600 objects misplaced (0.333%) 2024-04-14T15:22:41.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:41 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi090-93652-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:22:41.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:41 smithi092 ceph-mon[33966]: osdmap e624: 8 total, 8 up, 7 in 2024-04-14T15:22:41.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:41 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi090-93652-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi090-93652-109"}]: dispatch 2024-04-14T15:22:41.445 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:22:41.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:41.469 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:41.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:22:41.570 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:22:41.596 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:41.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:41.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:22:41.630 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd unset noscrub 2024-04-14T15:22:41.698 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:22:41.722 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:41.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:41.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:22:41.824 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:41.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:41.848 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:41.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:41.949 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:22:41.977 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:41.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:41.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:22:42.079 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:22:42.104 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:42.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:42.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:22:42.152 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph tell osd.5 reset_purged_snaps_last 2024-04-14T15:22:42.206 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:22:42.231 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:42.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:42.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:22:42.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.505+0000 7feabdb9a700 1 -- 172.21.15.90:0/3817763132 >> v1:172.21.15.90:6789/0 conn(0x7feab810a610 legacy=0x7feab810ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:42.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.505+0000 7feabdb9a700 1 -- 172.21.15.90:0/3817763132 shutdown_connections 2024-04-14T15:22:42.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.505+0000 7feabdb9a700 1 -- 172.21.15.90:0/3817763132 >> 172.21.15.90:0/3817763132 conn(0x7feab80fd290 msgr2=0x7feab80ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:22:42.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.505+0000 7feabdb9a700 1 -- 172.21.15.90:0/3817763132 shutdown_connections 2024-04-14T15:22:42.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.505+0000 7feabdb9a700 1 -- 172.21.15.90:0/3817763132 wait complete. 2024-04-14T15:22:42.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.505+0000 7feabdb9a700 1 librados: starting msgr at 2024-04-14T15:22:42.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.505+0000 7feabdb9a700 1 librados: starting objecter 2024-04-14T15:22:42.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.505+0000 7feabdb9a700 1 Processor -- start 2024-04-14T15:22:42.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.506+0000 7feabdb9a700 1 -- start start 2024-04-14T15:22:42.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.506+0000 7feabdb9a700 1 librados: setting wanted keys 2024-04-14T15:22:42.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.506+0000 7feabdb9a700 1 librados: calling monclient init 2024-04-14T15:22:42.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.506+0000 7feabdb9a700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7feab81a9b00 con 0x7feab810e1e0 2024-04-14T15:22:42.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.506+0000 7feabdb9a700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7feab81aad20 con 0x7feab810a610 2024-04-14T15:22:42.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.506+0000 7feabdb9a700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7feab81abf60 con 0x7feab8107760 2024-04-14T15:22:42.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.506+0000 7feab77fe700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7feab8107760 0x7feab810d880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:35228/0 (socket says 172.21.15.90:35228) 2024-04-14T15:22:42.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.506+0000 7feab77fe700 1 -- 172.21.15.90:0/3862467937 learned_addr learned my addr 172.21.15.90:0/3862467937 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:22:42.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.506+0000 7feab4ff9700 10 client.?.objecter ms_handle_connect 0x7feab810e1e0 2024-04-14T15:22:42.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.506+0000 7feab4ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:22:42.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.506+0000 7feab4ff9700 10 client.?.objecter ms_handle_connect 0x7feab8107760 2024-04-14T15:22:42.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.506+0000 7feab4ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:22:42.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.506+0000 7feab4ff9700 10 client.?.objecter ms_handle_connect 0x7feab810a610 2024-04-14T15:22:42.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.506+0000 7feab4ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:22:42.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.506+0000 7feab4ff9700 1 -- 172.21.15.90:0/3862467937 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1130702078 0 0) 0x7feab81a9b00 con 0x7feab810e1e0 2024-04-14T15:22:42.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.506+0000 7feab4ff9700 1 -- 172.21.15.90:0/3862467937 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feaa00035b0 con 0x7feab810e1e0 2024-04-14T15:22:42.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.506+0000 7feab4ff9700 1 -- 172.21.15.90:0/3862467937 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 573891860 0 0) 0x7feab81abf60 con 0x7feab8107760 2024-04-14T15:22:42.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.506+0000 7feab4ff9700 1 -- 172.21.15.90:0/3862467937 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feab81a9b00 con 0x7feab8107760 2024-04-14T15:22:42.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.506+0000 7feab4ff9700 1 -- 172.21.15.90:0/3862467937 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3060135391 0 0) 0x7feab81aad20 con 0x7feab810a610 2024-04-14T15:22:42.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.506+0000 7feab4ff9700 1 -- 172.21.15.90:0/3862467937 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feab81abf60 con 0x7feab810a610 2024-04-14T15:22:42.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.506+0000 7feab4ff9700 1 -- 172.21.15.90:0/3862467937 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1648892644 0 0) 0x7feaa00035b0 con 0x7feab810e1e0 2024-04-14T15:22:42.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.506+0000 7feab4ff9700 1 -- 172.21.15.90:0/3862467937 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feab81aad20 con 0x7feab810e1e0 2024-04-14T15:22:42.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.506+0000 7feab4ff9700 1 -- 172.21.15.90:0/3862467937 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7feab0001bb0 con 0x7feab810e1e0 2024-04-14T15:22:42.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.506+0000 7feab4ff9700 1 -- 172.21.15.90:0/3862467937 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1634843618 0 0) 0x7feab81a9b00 con 0x7feab8107760 2024-04-14T15:22:42.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.506+0000 7feab4ff9700 1 -- 172.21.15.90:0/3862467937 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feaa00035b0 con 0x7feab8107760 2024-04-14T15:22:42.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.506+0000 7feab4ff9700 1 -- 172.21.15.90:0/3862467937 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7feaa80018f0 con 0x7feab8107760 2024-04-14T15:22:42.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.506+0000 7feab4ff9700 1 -- 172.21.15.90:0/3862467937 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2211384834 0 0) 0x7feab81abf60 con 0x7feab810a610 2024-04-14T15:22:42.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.507+0000 7feab4ff9700 1 -- 172.21.15.90:0/3862467937 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feab81a9b00 con 0x7feab810a610 2024-04-14T15:22:42.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.507+0000 7feab4ff9700 1 -- 172.21.15.90:0/3862467937 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7feaac004540 con 0x7feab810a610 2024-04-14T15:22:42.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.507+0000 7feab4ff9700 1 -- 172.21.15.90:0/3862467937 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1870912814 0 0) 0x7feab81aad20 con 0x7feab810e1e0 2024-04-14T15:22:42.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.507+0000 7feab4ff9700 1 -- 172.21.15.90:0/3862467937 >> v1:172.21.15.90:6790/0 conn(0x7feab8107760 legacy=0x7feab810d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:42.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.507+0000 7feab4ff9700 1 -- 172.21.15.90:0/3862467937 >> v1:172.21.15.92:6789/0 conn(0x7feab810a610 legacy=0x7feab81a4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:42.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.507+0000 7feab4ff9700 1 -- 172.21.15.90:0/3862467937 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feab81ad1a0 con 0x7feab810e1e0 2024-04-14T15:22:42.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.507+0000 7feabdb9a700 1 -- 172.21.15.90:0/3862467937 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7feab81ac190 con 0x7feab810e1e0 2024-04-14T15:22:42.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.507+0000 7feabdb9a700 10 client.37713.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:22:42.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.507+0000 7feabdb9a700 1 -- 172.21.15.90:0/3862467937 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7feab81ac650 con 0x7feab810e1e0 2024-04-14T15:22:42.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.507+0000 7feabdb9a700 1 librados: init done 2024-04-14T15:22:42.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.507+0000 7feab4ff9700 1 -- 172.21.15.90:0/3862467937 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7feab0003ec0 con 0x7feab810e1e0 2024-04-14T15:22:42.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.507+0000 7feab4ff9700 1 -- 172.21.15.90:0/3862467937 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7feab0004d70 con 0x7feab810e1e0 2024-04-14T15:22:42.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.508+0000 7feabdb9a700 1 -- 172.21.15.90:0/3862467937 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7feaa4005100 con 0x7feab810e1e0 2024-04-14T15:22:42.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.508+0000 7feab4ff9700 1 -- 172.21.15.90:0/3862467937 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7feab0003a20 con 0x7feab810e1e0 2024-04-14T15:22:42.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.509+0000 7feab4ff9700 1 -- 172.21.15.90:0/3862467937 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(625..625 src has 467..625) ==== 8839+0+0 (unknown 2615881034 0 0) 0x7feab0089e10 con 0x7feab810e1e0 2024-04-14T15:22:42.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.509+0000 7feab4ff9700 10 client.37713.objecter ms_dispatch 0x7feab8001550 osd_map(625..625 src has 467..625) 2024-04-14T15:22:42.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.509+0000 7feab4ff9700 3 client.37713.objecter handle_osd_map got epochs [625,625] > 0 2024-04-14T15:22:42.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.509+0000 7feab4ff9700 3 client.37713.objecter handle_osd_map decoding full epoch 625 2024-04-14T15:22:42.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.509+0000 7feab4ff9700 20 client.37713.objecter dump_active .. 0 homeless 2024-04-14T15:22:42.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.509+0000 7feab4ff9700 1 -- 172.21.15.90:0/3862467937 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v625) ==== 54+0+16 (unknown 2731798705 0 2420000192) 0x7feab008c1f0 con 0x7feab810e1e0 2024-04-14T15:22:42.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.509+0000 7feab4ff9700 10 client.37713.objecter ms_handle_connect 0x7feaa0071380 2024-04-14T15:22:42.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.509+0000 7feabdb9a700 10 client.37713.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:22:42.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.509+0000 7feabdb9a700 15 client.37713.objecter _session_command_op_assign -1 1 2024-04-14T15:22:42.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.509+0000 7feabdb9a700 20 client.37713.objecter _get_session s=0x7feaa40053e0 osd=0 3 2024-04-14T15:22:42.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.509+0000 7feabdb9a700 20 client.37713.objecter put_session s=0x7feaa40053e0 osd=0 3 2024-04-14T15:22:42.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.509+0000 7feabdb9a700 20 client.37713.objecter _get_session s=0x7feaa40053e0 osd=0 3 2024-04-14T15:22:42.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.509+0000 7feabdb9a700 15 client.37713.objecter _session_command_op_remove -1 1 2024-04-14T15:22:42.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.509+0000 7feabdb9a700 20 client.37713.objecter get_session s=0x7feaa40053e0 osd=0 3 2024-04-14T15:22:42.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.509+0000 7feabdb9a700 15 client.37713.objecter _session_command_op_assign 0 1 2024-04-14T15:22:42.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.509+0000 7feabdb9a700 20 client.37713.objecter put_session s=0x7feaa40053e0 osd=0 4 2024-04-14T15:22:42.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.509+0000 7feabdb9a700 10 client.37713.objecter _send_command 1 2024-04-14T15:22:42.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.509+0000 7feabdb9a700 1 -- 172.21.15.90:0/3862467937 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7feaa4004b50 con 0x7feaa4004730 2024-04-14T15:22:42.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.509+0000 7feab4ff9700 10 client.37713.objecter ms_handle_connect 0x7feaa4004730 2024-04-14T15:22:42.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.510+0000 7feab4ff9700 1 -- 172.21.15.90:0/3862467937 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7feaa4004b50 con 0x7feaa4004730 2024-04-14T15:22:42.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.510+0000 7feab4ff9700 10 client.37713.objecter ms_dispatch 0x7feab8001550 command_reply(tid 1: 0 ) 2024-04-14T15:22:42.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.510+0000 7feab4ff9700 10 client.37713.objecter _finish_command 1 = system:0 2024-04-14T15:22:42.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.510+0000 7feab4ff9700 20 client.37713.objecter put_session s=0x7feaa40053e0 osd=0 4 2024-04-14T15:22:42.249 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:22:42.249 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:22:42.249 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:22:42.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.510+0000 7feab4ff9700 15 client.37713.objecter _session_command_op_remove 0 1 2024-04-14T15:22:42.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.527+0000 7feabdb9a700 10 client.37713.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:22:42.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.527+0000 7feabdb9a700 15 client.37713.objecter _session_command_op_assign -1 2 2024-04-14T15:22:42.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.527+0000 7feabdb9a700 20 client.37713.objecter _get_session s=0x7feaa40053e0 osd=0 3 2024-04-14T15:22:42.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.527+0000 7feabdb9a700 20 client.37713.objecter put_session s=0x7feaa40053e0 osd=0 3 2024-04-14T15:22:42.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.527+0000 7feabdb9a700 20 client.37713.objecter _get_session s=0x7feaa40053e0 osd=0 3 2024-04-14T15:22:42.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.527+0000 7feabdb9a700 15 client.37713.objecter _session_command_op_remove -1 2 2024-04-14T15:22:42.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.527+0000 7feabdb9a700 20 client.37713.objecter get_session s=0x7feaa40053e0 osd=0 3 2024-04-14T15:22:42.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.527+0000 7feabdb9a700 15 client.37713.objecter _session_command_op_assign 0 2 2024-04-14T15:22:42.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.527+0000 7feabdb9a700 20 client.37713.objecter put_session s=0x7feaa40053e0 osd=0 4 2024-04-14T15:22:42.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.527+0000 7feabdb9a700 10 client.37713.objecter _send_command 2 2024-04-14T15:22:42.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.527+0000 7feabdb9a700 1 -- 172.21.15.90:0/3862467937 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7feaa4007b00 con 0x7feaa4004730 2024-04-14T15:22:42.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.529+0000 7feab4ff9700 1 -- 172.21.15.90:0/3862467937 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7feaa4007b00 con 0x7feaa4004730 2024-04-14T15:22:42.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.529+0000 7feab4ff9700 10 client.37713.objecter ms_dispatch 0x7feab8001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:22:42.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.529+0000 7feab4ff9700 10 client.37713.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:22:42.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.529+0000 7feab4ff9700 20 client.37713.objecter put_session s=0x7feaa40053e0 osd=0 4 2024-04-14T15:22:42.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.529+0000 7feab4ff9700 15 client.37713.objecter _session_command_op_remove 0 2 2024-04-14T15:22:42.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.529+0000 7feabdb9a700 10 client.37713.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:22:42.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.529+0000 7feabdb9a700 15 client.37713.objecter _session_command_op_assign -1 3 2024-04-14T15:22:42.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.529+0000 7feabdb9a700 20 client.37713.objecter _get_session s=0x7feaa4004f90 osd=1 3 2024-04-14T15:22:42.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.529+0000 7feabdb9a700 20 client.37713.objecter put_session s=0x7feaa4004f90 osd=1 3 2024-04-14T15:22:42.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.529+0000 7feabdb9a700 20 client.37713.objecter _get_session s=0x7feaa4004f90 osd=1 3 2024-04-14T15:22:42.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.529+0000 7feabdb9a700 15 client.37713.objecter _session_command_op_remove -1 3 2024-04-14T15:22:42.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.529+0000 7feabdb9a700 20 client.37713.objecter get_session s=0x7feaa4004f90 osd=1 3 2024-04-14T15:22:42.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.529+0000 7feabdb9a700 15 client.37713.objecter _session_command_op_assign 1 3 2024-04-14T15:22:42.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.529+0000 7feabdb9a700 20 client.37713.objecter put_session s=0x7feaa4004f90 osd=1 4 2024-04-14T15:22:42.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.529+0000 7feabdb9a700 10 client.37713.objecter _send_command 3 2024-04-14T15:22:42.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.529+0000 7feabdb9a700 1 -- 172.21.15.90:0/3862467937 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7feaa4002100 con 0x7feaa4001b80 2024-04-14T15:22:42.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.529+0000 7feab4ff9700 10 client.37713.objecter ms_handle_connect 0x7feaa4001b80 2024-04-14T15:22:42.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.530+0000 7feab4ff9700 1 -- 172.21.15.90:0/3862467937 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7feaa4002100 con 0x7feaa4001b80 2024-04-14T15:22:42.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.530+0000 7feab4ff9700 10 client.37713.objecter ms_dispatch 0x7feab8001550 command_reply(tid 3: 0 ) 2024-04-14T15:22:42.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.530+0000 7feab4ff9700 10 client.37713.objecter _finish_command 3 = system:0 2024-04-14T15:22:42.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.530+0000 7feab4ff9700 20 client.37713.objecter put_session s=0x7feaa4004f90 osd=1 4 2024-04-14T15:22:42.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.530+0000 7feab4ff9700 15 client.37713.objecter _session_command_op_remove 1 3 2024-04-14T15:22:42.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.546+0000 7feabdb9a700 10 client.37713.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:22:42.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.546+0000 7feabdb9a700 15 client.37713.objecter _session_command_op_assign -1 4 2024-04-14T15:22:42.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.546+0000 7feabdb9a700 20 client.37713.objecter _get_session s=0x7feaa4004f90 osd=1 3 2024-04-14T15:22:42.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.546+0000 7feabdb9a700 20 client.37713.objecter put_session s=0x7feaa4004f90 osd=1 3 2024-04-14T15:22:42.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.546+0000 7feabdb9a700 20 client.37713.objecter _get_session s=0x7feaa4004f90 osd=1 3 2024-04-14T15:22:42.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.546+0000 7feabdb9a700 15 client.37713.objecter _session_command_op_remove -1 4 2024-04-14T15:22:42.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.546+0000 7feabdb9a700 20 client.37713.objecter get_session s=0x7feaa4004f90 osd=1 3 2024-04-14T15:22:42.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.546+0000 7feabdb9a700 15 client.37713.objecter _session_command_op_assign 1 4 2024-04-14T15:22:42.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.546+0000 7feabdb9a700 20 client.37713.objecter put_session s=0x7feaa4004f90 osd=1 4 2024-04-14T15:22:42.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.546+0000 7feabdb9a700 10 client.37713.objecter _send_command 4 2024-04-14T15:22:42.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.546+0000 7feabdb9a700 1 -- 172.21.15.90:0/3862467937 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7feaa4001600 con 0x7feaa4001b80 2024-04-14T15:22:42.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.548+0000 7feab4ff9700 1 -- 172.21.15.90:0/3862467937 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7feaa4001600 con 0x7feaa4001b80 2024-04-14T15:22:42.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.548+0000 7feab4ff9700 10 client.37713.objecter ms_dispatch 0x7feab8001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:22:42.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.548+0000 7feab4ff9700 10 client.37713.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:22:42.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.548+0000 7feab4ff9700 20 client.37713.objecter put_session s=0x7feaa4004f90 osd=1 4 2024-04-14T15:22:42.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.548+0000 7feab4ff9700 15 client.37713.objecter _session_command_op_remove 1 4 2024-04-14T15:22:42.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.548+0000 7feabdb9a700 10 client.37713.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:22:42.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.548+0000 7feabdb9a700 15 client.37713.objecter _session_command_op_assign -1 5 2024-04-14T15:22:42.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.548+0000 7feabdb9a700 20 client.37713.objecter _get_session s=0x7feaa4001a00 osd=2 3 2024-04-14T15:22:42.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.548+0000 7feabdb9a700 20 client.37713.objecter put_session s=0x7feaa4001a00 osd=2 3 2024-04-14T15:22:42.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.548+0000 7feabdb9a700 20 client.37713.objecter _get_session s=0x7feaa4001a00 osd=2 3 2024-04-14T15:22:42.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.548+0000 7feabdb9a700 15 client.37713.objecter _session_command_op_remove -1 5 2024-04-14T15:22:42.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.548+0000 7feabdb9a700 20 client.37713.objecter get_session s=0x7feaa4001a00 osd=2 3 2024-04-14T15:22:42.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.548+0000 7feabdb9a700 15 client.37713.objecter _session_command_op_assign 2 5 2024-04-14T15:22:42.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.548+0000 7feabdb9a700 20 client.37713.objecter put_session s=0x7feaa4001a00 osd=2 4 2024-04-14T15:22:42.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.548+0000 7feabdb9a700 10 client.37713.objecter _send_command 5 2024-04-14T15:22:42.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.548+0000 7feabdb9a700 1 -- 172.21.15.90:0/3862467937 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7feaa4010fb0 con 0x7feaa4008990 2024-04-14T15:22:42.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.548+0000 7feab4ff9700 10 client.37713.objecter ms_handle_connect 0x7feaa4008990 2024-04-14T15:22:42.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.549+0000 7feab4ff9700 1 -- 172.21.15.90:0/3862467937 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7feaa4010fb0 con 0x7feaa4008990 2024-04-14T15:22:42.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.549+0000 7feab4ff9700 10 client.37713.objecter ms_dispatch 0x7feab8001550 command_reply(tid 5: 0 ) 2024-04-14T15:22:42.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.549+0000 7feab4ff9700 10 client.37713.objecter _finish_command 5 = system:0 2024-04-14T15:22:42.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.549+0000 7feab4ff9700 20 client.37713.objecter put_session s=0x7feaa4001a00 osd=2 4 2024-04-14T15:22:42.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.549+0000 7feab4ff9700 15 client.37713.objecter _session_command_op_remove 2 5 2024-04-14T15:22:42.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.565+0000 7feabdb9a700 10 client.37713.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:22:42.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.565+0000 7feabdb9a700 15 client.37713.objecter _session_command_op_assign -1 6 2024-04-14T15:22:42.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.565+0000 7feabdb9a700 20 client.37713.objecter _get_session s=0x7feaa4001a00 osd=2 3 2024-04-14T15:22:42.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.565+0000 7feabdb9a700 20 client.37713.objecter put_session s=0x7feaa4001a00 osd=2 3 2024-04-14T15:22:42.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.565+0000 7feabdb9a700 20 client.37713.objecter _get_session s=0x7feaa4001a00 osd=2 3 2024-04-14T15:22:42.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.565+0000 7feabdb9a700 15 client.37713.objecter _session_command_op_remove -1 6 2024-04-14T15:22:42.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.565+0000 7feabdb9a700 20 client.37713.objecter get_session s=0x7feaa4001a00 osd=2 3 2024-04-14T15:22:42.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.565+0000 7feabdb9a700 15 client.37713.objecter _session_command_op_assign 2 6 2024-04-14T15:22:42.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.565+0000 7feabdb9a700 20 client.37713.objecter put_session s=0x7feaa4001a00 osd=2 4 2024-04-14T15:22:42.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.565+0000 7feabdb9a700 10 client.37713.objecter _send_command 6 2024-04-14T15:22:42.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.565+0000 7feabdb9a700 1 -- 172.21.15.90:0/3862467937 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7feaa4007f00 con 0x7feaa4008990 2024-04-14T15:22:42.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.567+0000 7feab4ff9700 1 -- 172.21.15.90:0/3862467937 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7feaa4007f00 con 0x7feaa4008990 2024-04-14T15:22:42.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.567+0000 7feab4ff9700 10 client.37713.objecter ms_dispatch 0x7feab8001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:22:42.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.567+0000 7feab4ff9700 10 client.37713.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:22:42.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.567+0000 7feab4ff9700 20 client.37713.objecter put_session s=0x7feaa4001a00 osd=2 4 2024-04-14T15:22:42.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.567+0000 7feab4ff9700 15 client.37713.objecter _session_command_op_remove 2 6 2024-04-14T15:22:42.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.567+0000 7feabdb9a700 10 client.37713.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:22:42.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.567+0000 7feabdb9a700 15 client.37713.objecter _session_command_op_assign -1 7 2024-04-14T15:22:42.255 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:22:42.255 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:22:42.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.567+0000 7feabdb9a700 20 client.37713.objecter _get_session s=0x7feaa400cd30 osd=3 3 2024-04-14T15:22:42.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.567+0000 7feabdb9a700 20 client.37713.objecter put_session s=0x7feaa400cd30 osd=3 3 2024-04-14T15:22:42.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.567+0000 7feabdb9a700 20 client.37713.objecter _get_session s=0x7feaa400cd30 osd=3 3 2024-04-14T15:22:42.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.567+0000 7feabdb9a700 15 client.37713.objecter _session_command_op_remove -1 7 2024-04-14T15:22:42.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.567+0000 7feabdb9a700 20 client.37713.objecter get_session s=0x7feaa400cd30 osd=3 3 2024-04-14T15:22:42.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.567+0000 7feabdb9a700 15 client.37713.objecter _session_command_op_assign 3 7 2024-04-14T15:22:42.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.567+0000 7feabdb9a700 20 client.37713.objecter put_session s=0x7feaa400cd30 osd=3 4 2024-04-14T15:22:42.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.567+0000 7feabdb9a700 10 client.37713.objecter _send_command 7 2024-04-14T15:22:42.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.567+0000 7feabdb9a700 1 -- 172.21.15.90:0/3862467937 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7feaa400d520 con 0x7feaa4011160 2024-04-14T15:22:42.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.567+0000 7feab4ff9700 10 client.37713.objecter ms_handle_connect 0x7feaa4011160 2024-04-14T15:22:42.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.568+0000 7feab4ff9700 1 -- 172.21.15.90:0/3862467937 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7feab00565a0 con 0x7feaa4011160 2024-04-14T15:22:42.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.568+0000 7feab4ff9700 10 client.37713.objecter ms_dispatch 0x7feab8001550 command_reply(tid 7: 0 ) 2024-04-14T15:22:42.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.568+0000 7feab4ff9700 10 client.37713.objecter _finish_command 7 = system:0 2024-04-14T15:22:42.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.568+0000 7feab4ff9700 20 client.37713.objecter put_session s=0x7feaa400cd30 osd=3 4 2024-04-14T15:22:42.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.568+0000 7feab4ff9700 15 client.37713.objecter _session_command_op_remove 3 7 2024-04-14T15:22:42.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.585+0000 7feabdb9a700 10 client.37713.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:22:42.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.585+0000 7feabdb9a700 15 client.37713.objecter _session_command_op_assign -1 8 2024-04-14T15:22:42.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.585+0000 7feabdb9a700 20 client.37713.objecter _get_session s=0x7feaa400cd30 osd=3 3 2024-04-14T15:22:42.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.585+0000 7feabdb9a700 20 client.37713.objecter put_session s=0x7feaa400cd30 osd=3 3 2024-04-14T15:22:42.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.585+0000 7feabdb9a700 20 client.37713.objecter _get_session s=0x7feaa400cd30 osd=3 3 2024-04-14T15:22:42.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.585+0000 7feabdb9a700 15 client.37713.objecter _session_command_op_remove -1 8 2024-04-14T15:22:42.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.585+0000 7feabdb9a700 20 client.37713.objecter get_session s=0x7feaa400cd30 osd=3 3 2024-04-14T15:22:42.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.585+0000 7feabdb9a700 15 client.37713.objecter _session_command_op_assign 3 8 2024-04-14T15:22:42.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.585+0000 7feabdb9a700 20 client.37713.objecter put_session s=0x7feaa400cd30 osd=3 4 2024-04-14T15:22:42.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.585+0000 7feabdb9a700 10 client.37713.objecter _send_command 8 2024-04-14T15:22:42.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.585+0000 7feabdb9a700 1 -- 172.21.15.90:0/3862467937 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7feaa40124e0 con 0x7feaa4011160 2024-04-14T15:22:42.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.586+0000 7feab4ff9700 1 -- 172.21.15.90:0/3862467937 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7feaa40124e0 con 0x7feaa4011160 2024-04-14T15:22:42.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.586+0000 7feab4ff9700 10 client.37713.objecter ms_dispatch 0x7feab8001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:22:42.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.586+0000 7feab4ff9700 10 client.37713.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:22:42.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.586+0000 7feab4ff9700 20 client.37713.objecter put_session s=0x7feaa400cd30 osd=3 4 2024-04-14T15:22:42.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.586+0000 7feab4ff9700 15 client.37713.objecter _session_command_op_remove 3 8 2024-04-14T15:22:42.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.586+0000 7feabdb9a700 10 client.37713.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:22:42.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.586+0000 7feabdb9a700 15 client.37713.objecter _session_command_op_assign -1 9 2024-04-14T15:22:42.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.586+0000 7feabdb9a700 20 client.37713.objecter _get_session s=0x7feaa4013950 osd=4 3 2024-04-14T15:22:42.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.586+0000 7feabdb9a700 20 client.37713.objecter put_session s=0x7feaa4013950 osd=4 3 2024-04-14T15:22:42.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.586+0000 7feabdb9a700 20 client.37713.objecter _get_session s=0x7feaa4013950 osd=4 3 2024-04-14T15:22:42.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.586+0000 7feabdb9a700 15 client.37713.objecter _session_command_op_remove -1 9 2024-04-14T15:22:42.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.586+0000 7feabdb9a700 20 client.37713.objecter get_session s=0x7feaa4013950 osd=4 3 2024-04-14T15:22:42.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.586+0000 7feabdb9a700 15 client.37713.objecter _session_command_op_assign 4 9 2024-04-14T15:22:42.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.586+0000 7feabdb9a700 20 client.37713.objecter put_session s=0x7feaa4013950 osd=4 4 2024-04-14T15:22:42.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.586+0000 7feabdb9a700 10 client.37713.objecter _send_command 9 2024-04-14T15:22:42.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.586+0000 7feabdb9a700 1 -- 172.21.15.90:0/3862467937 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7feaa401a7b0 con 0x7feaa400c8d0 2024-04-14T15:22:42.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.587+0000 7feab4ff9700 10 client.37713.objecter ms_handle_connect 0x7feaa400c8d0 2024-04-14T15:22:42.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.588+0000 7feab4ff9700 1 -- 172.21.15.90:0/3862467937 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7feaa401a7b0 con 0x7feaa400c8d0 2024-04-14T15:22:42.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.588+0000 7feab4ff9700 10 client.37713.objecter ms_dispatch 0x7feab8001550 command_reply(tid 9: 0 ) 2024-04-14T15:22:42.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.588+0000 7feab4ff9700 10 client.37713.objecter _finish_command 9 = system:0 2024-04-14T15:22:42.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.588+0000 7feab4ff9700 20 client.37713.objecter put_session s=0x7feaa4013950 osd=4 4 2024-04-14T15:22:42.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.588+0000 7feab4ff9700 15 client.37713.objecter _session_command_op_remove 4 9 2024-04-14T15:22:42.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.603+0000 7feabdb9a700 10 client.37713.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:22:42.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.603+0000 7feabdb9a700 15 client.37713.objecter _session_command_op_assign -1 10 2024-04-14T15:22:42.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.603+0000 7feabdb9a700 20 client.37713.objecter _get_session s=0x7feaa4013950 osd=4 3 2024-04-14T15:22:42.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.603+0000 7feabdb9a700 20 client.37713.objecter put_session s=0x7feaa4013950 osd=4 3 2024-04-14T15:22:42.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.603+0000 7feabdb9a700 20 client.37713.objecter _get_session s=0x7feaa4013950 osd=4 3 2024-04-14T15:22:42.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.603+0000 7feabdb9a700 15 client.37713.objecter _session_command_op_remove -1 10 2024-04-14T15:22:42.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.603+0000 7feabdb9a700 20 client.37713.objecter get_session s=0x7feaa4013950 osd=4 3 2024-04-14T15:22:42.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.603+0000 7feabdb9a700 15 client.37713.objecter _session_command_op_assign 4 10 2024-04-14T15:22:42.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.603+0000 7feabdb9a700 20 client.37713.objecter put_session s=0x7feaa4013950 osd=4 4 2024-04-14T15:22:42.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.603+0000 7feabdb9a700 10 client.37713.objecter _send_command 10 2024-04-14T15:22:42.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.603+0000 7feabdb9a700 1 -- 172.21.15.90:0/3862467937 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7feaa4011f00 con 0x7feaa400c8d0 2024-04-14T15:22:42.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.605+0000 7feab4ff9700 1 -- 172.21.15.90:0/3862467937 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7feaa4011f00 con 0x7feaa400c8d0 2024-04-14T15:22:42.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.605+0000 7feab4ff9700 10 client.37713.objecter ms_dispatch 0x7feab8001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:22:42.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.605+0000 7feab4ff9700 10 client.37713.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:22:42.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.605+0000 7feab4ff9700 20 client.37713.objecter put_session s=0x7feaa4013950 osd=4 4 2024-04-14T15:22:42.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.605+0000 7feab4ff9700 15 client.37713.objecter _session_command_op_remove 4 10 2024-04-14T15:22:42.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.606+0000 7feabdb9a700 10 client.37713.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:22:42.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.606+0000 7feabdb9a700 15 client.37713.objecter _session_command_op_assign -1 11 2024-04-14T15:22:42.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.606+0000 7feabdb9a700 20 client.37713.objecter _get_session s=0x7feaa4011cb0 osd=5 3 2024-04-14T15:22:42.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.606+0000 7feabdb9a700 20 client.37713.objecter put_session s=0x7feaa4011cb0 osd=5 3 2024-04-14T15:22:42.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.606+0000 7feabdb9a700 20 client.37713.objecter _get_session s=0x7feaa4011cb0 osd=5 3 2024-04-14T15:22:42.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.606+0000 7feabdb9a700 15 client.37713.objecter _session_command_op_remove -1 11 2024-04-14T15:22:42.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.606+0000 7feabdb9a700 20 client.37713.objecter get_session s=0x7feaa4011cb0 osd=5 3 2024-04-14T15:22:42.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.606+0000 7feabdb9a700 15 client.37713.objecter _session_command_op_assign 5 11 2024-04-14T15:22:42.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.606+0000 7feabdb9a700 20 client.37713.objecter put_session s=0x7feaa4011cb0 osd=5 4 2024-04-14T15:22:42.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.606+0000 7feabdb9a700 10 client.37713.objecter _send_command 11 2024-04-14T15:22:42.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.606+0000 7feabdb9a700 1 -- 172.21.15.90:0/3862467937 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7feaa401d050 con 0x7feaa4013470 2024-04-14T15:22:42.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.606+0000 7feab4ff9700 10 client.37713.objecter ms_handle_connect 0x7feaa4013470 2024-04-14T15:22:42.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.607+0000 7feab4ff9700 1 -- 172.21.15.90:0/3862467937 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7feaa401d050 con 0x7feaa4013470 2024-04-14T15:22:42.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.607+0000 7feab4ff9700 10 client.37713.objecter ms_dispatch 0x7feab8001550 command_reply(tid 11: 0 ) 2024-04-14T15:22:42.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.607+0000 7feab4ff9700 10 client.37713.objecter _finish_command 11 = system:0 2024-04-14T15:22:42.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.607+0000 7feab4ff9700 20 client.37713.objecter put_session s=0x7feaa4011cb0 osd=5 4 2024-04-14T15:22:42.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.607+0000 7feab4ff9700 15 client.37713.objecter _session_command_op_remove 5 11 2024-04-14T15:22:42.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.623+0000 7feabdb9a700 10 client.37713.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:22:42.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.623+0000 7feabdb9a700 15 client.37713.objecter _session_command_op_assign -1 12 2024-04-14T15:22:42.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.623+0000 7feabdb9a700 20 client.37713.objecter _get_session s=0x7feaa4011cb0 osd=5 3 2024-04-14T15:22:42.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.623+0000 7feabdb9a700 20 client.37713.objecter put_session s=0x7feaa4011cb0 osd=5 3 2024-04-14T15:22:42.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.623+0000 7feabdb9a700 20 client.37713.objecter _get_session s=0x7feaa4011cb0 osd=5 3 2024-04-14T15:22:42.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.623+0000 7feabdb9a700 15 client.37713.objecter _session_command_op_remove -1 12 2024-04-14T15:22:42.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.623+0000 7feabdb9a700 20 client.37713.objecter get_session s=0x7feaa4011cb0 osd=5 3 2024-04-14T15:22:42.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.623+0000 7feabdb9a700 15 client.37713.objecter _session_command_op_assign 5 12 2024-04-14T15:22:42.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.623+0000 7feabdb9a700 20 client.37713.objecter put_session s=0x7feaa4011cb0 osd=5 4 2024-04-14T15:22:42.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.623+0000 7feabdb9a700 10 client.37713.objecter _send_command 12 2024-04-14T15:22:42.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.623+0000 7feabdb9a700 1 -- 172.21.15.90:0/3862467937 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7feaa401d050 con 0x7feaa4013470 2024-04-14T15:22:42.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.625+0000 7feab4ff9700 1 -- 172.21.15.90:0/3862467937 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7feaa401d050 con 0x7feaa4013470 2024-04-14T15:22:42.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.625+0000 7feab4ff9700 10 client.37713.objecter ms_dispatch 0x7feab8001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:22:42.262 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:22:42.262 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:22:42.262 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:22:42.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.625+0000 7feab4ff9700 10 client.37713.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:22:42.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.625+0000 7feab4ff9700 20 client.37713.objecter put_session s=0x7feaa4011cb0 osd=5 4 2024-04-14T15:22:42.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.625+0000 7feab4ff9700 15 client.37713.objecter _session_command_op_remove 5 12 2024-04-14T15:22:42.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.625+0000 7feabdb9a700 10 client.37713.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:22:42.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.625+0000 7feabdb9a700 15 client.37713.objecter _session_command_op_assign -1 13 2024-04-14T15:22:42.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.625+0000 7feabdb9a700 20 client.37713.objecter _get_session s=0x7feaa401d050 osd=6 3 2024-04-14T15:22:42.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.625+0000 7feabdb9a700 20 client.37713.objecter put_session s=0x7feaa401d050 osd=6 3 2024-04-14T15:22:42.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.625+0000 7feabdb9a700 20 client.37713.objecter _get_session s=0x7feaa401d050 osd=6 3 2024-04-14T15:22:42.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.625+0000 7feabdb9a700 15 client.37713.objecter _session_command_op_remove -1 13 2024-04-14T15:22:42.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.625+0000 7feabdb9a700 20 client.37713.objecter get_session s=0x7feaa401d050 osd=6 3 2024-04-14T15:22:42.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.625+0000 7feabdb9a700 15 client.37713.objecter _session_command_op_assign 6 13 2024-04-14T15:22:42.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.625+0000 7feabdb9a700 20 client.37713.objecter put_session s=0x7feaa401d050 osd=6 4 2024-04-14T15:22:42.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.625+0000 7feabdb9a700 10 client.37713.objecter _send_command 13 2024-04-14T15:22:42.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.625+0000 7feabdb9a700 1 -- 172.21.15.90:0/3862467937 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7feaa4023300 con 0x7feaa401f7f0 2024-04-14T15:22:42.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.626+0000 7feab4ff9700 10 client.37713.objecter ms_handle_connect 0x7feaa401f7f0 2024-04-14T15:22:42.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.626+0000 7feab4ff9700 1 -- 172.21.15.90:0/3862467937 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7feaa4023300 con 0x7feaa401f7f0 2024-04-14T15:22:42.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.626+0000 7feab4ff9700 10 client.37713.objecter ms_dispatch 0x7feab8001550 command_reply(tid 13: 0 ) 2024-04-14T15:22:42.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.626+0000 7feab4ff9700 10 client.37713.objecter _finish_command 13 = system:0 2024-04-14T15:22:42.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.626+0000 7feab4ff9700 20 client.37713.objecter put_session s=0x7feaa401d050 osd=6 4 2024-04-14T15:22:42.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.626+0000 7feab4ff9700 15 client.37713.objecter _session_command_op_remove 6 13 2024-04-14T15:22:42.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.644+0000 7feabdb9a700 10 client.37713.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:22:42.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.644+0000 7feabdb9a700 15 client.37713.objecter _session_command_op_assign -1 14 2024-04-14T15:22:42.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.644+0000 7feabdb9a700 20 client.37713.objecter _get_session s=0x7feaa401d050 osd=6 3 2024-04-14T15:22:42.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.644+0000 7feabdb9a700 20 client.37713.objecter put_session s=0x7feaa401d050 osd=6 3 2024-04-14T15:22:42.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.644+0000 7feabdb9a700 20 client.37713.objecter _get_session s=0x7feaa401d050 osd=6 3 2024-04-14T15:22:42.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.644+0000 7feabdb9a700 15 client.37713.objecter _session_command_op_remove -1 14 2024-04-14T15:22:42.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.644+0000 7feabdb9a700 20 client.37713.objecter get_session s=0x7feaa401d050 osd=6 3 2024-04-14T15:22:42.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.644+0000 7feabdb9a700 15 client.37713.objecter _session_command_op_assign 6 14 2024-04-14T15:22:42.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.644+0000 7feabdb9a700 20 client.37713.objecter put_session s=0x7feaa401d050 osd=6 4 2024-04-14T15:22:42.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.644+0000 7feabdb9a700 10 client.37713.objecter _send_command 14 2024-04-14T15:22:42.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.644+0000 7feabdb9a700 1 -- 172.21.15.90:0/3862467937 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7feaa4011e80 con 0x7feaa401f7f0 2024-04-14T15:22:42.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.646+0000 7feab4ff9700 1 -- 172.21.15.90:0/3862467937 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7feaa4011e80 con 0x7feaa401f7f0 2024-04-14T15:22:42.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.646+0000 7feab4ff9700 10 client.37713.objecter ms_dispatch 0x7feab8001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:22:42.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.646+0000 7feab4ff9700 10 client.37713.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:22:42.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.646+0000 7feab4ff9700 20 client.37713.objecter put_session s=0x7feaa401d050 osd=6 4 2024-04-14T15:22:42.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.646+0000 7feab4ff9700 15 client.37713.objecter _session_command_op_remove 6 14 2024-04-14T15:22:42.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.647+0000 7feabdb9a700 10 client.37713.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:22:42.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.647+0000 7feabdb9a700 15 client.37713.objecter _session_command_op_assign -1 15 2024-04-14T15:22:42.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.647+0000 7feabdb9a700 20 client.37713.objecter _get_session s=0x7feaa40129e0 osd=7 3 2024-04-14T15:22:42.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.647+0000 7feabdb9a700 20 client.37713.objecter put_session s=0x7feaa40129e0 osd=7 3 2024-04-14T15:22:42.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.647+0000 7feabdb9a700 20 client.37713.objecter _get_session s=0x7feaa40129e0 osd=7 3 2024-04-14T15:22:42.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.647+0000 7feabdb9a700 15 client.37713.objecter _session_command_op_remove -1 15 2024-04-14T15:22:42.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.647+0000 7feabdb9a700 20 client.37713.objecter get_session s=0x7feaa40129e0 osd=7 3 2024-04-14T15:22:42.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.647+0000 7feabdb9a700 15 client.37713.objecter _session_command_op_assign 7 15 2024-04-14T15:22:42.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.647+0000 7feabdb9a700 20 client.37713.objecter put_session s=0x7feaa40129e0 osd=7 4 2024-04-14T15:22:42.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.647+0000 7feabdb9a700 10 client.37713.objecter _send_command 15 2024-04-14T15:22:42.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.647+0000 7feabdb9a700 1 -- 172.21.15.90:0/3862467937 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7feaa401b660 con 0x7feaa4012b50 2024-04-14T15:22:42.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.647+0000 7feab4ff9700 10 client.37713.objecter ms_handle_connect 0x7feaa4012b50 2024-04-14T15:22:42.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.648+0000 7feab4ff9700 1 -- 172.21.15.90:0/3862467937 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7feaa401b660 con 0x7feaa4012b50 2024-04-14T15:22:42.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.648+0000 7feab4ff9700 10 client.37713.objecter ms_dispatch 0x7feab8001550 command_reply(tid 15: 0 ) 2024-04-14T15:22:42.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.648+0000 7feab4ff9700 10 client.37713.objecter _finish_command 15 = system:0 2024-04-14T15:22:42.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.648+0000 7feab4ff9700 20 client.37713.objecter put_session s=0x7feaa40129e0 osd=7 4 2024-04-14T15:22:42.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.648+0000 7feab4ff9700 15 client.37713.objecter _session_command_op_remove 7 15 2024-04-14T15:22:42.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.670+0000 7feabdb9a700 10 client.37713.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:22:42.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.670+0000 7feabdb9a700 15 client.37713.objecter _session_command_op_assign -1 16 2024-04-14T15:22:42.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.670+0000 7feabdb9a700 20 client.37713.objecter _get_session s=0x7feaa40129e0 osd=7 3 2024-04-14T15:22:42.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.670+0000 7feabdb9a700 20 client.37713.objecter put_session s=0x7feaa40129e0 osd=7 3 2024-04-14T15:22:42.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.670+0000 7feabdb9a700 20 client.37713.objecter _get_session s=0x7feaa40129e0 osd=7 3 2024-04-14T15:22:42.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.670+0000 7feabdb9a700 15 client.37713.objecter _session_command_op_remove -1 16 2024-04-14T15:22:42.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.670+0000 7feabdb9a700 20 client.37713.objecter get_session s=0x7feaa40129e0 osd=7 3 2024-04-14T15:22:42.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.670+0000 7feabdb9a700 15 client.37713.objecter _session_command_op_assign 7 16 2024-04-14T15:22:42.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.670+0000 7feabdb9a700 20 client.37713.objecter put_session s=0x7feaa40129e0 osd=7 4 2024-04-14T15:22:42.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.670+0000 7feabdb9a700 10 client.37713.objecter _send_command 16 2024-04-14T15:22:42.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.670+0000 7feabdb9a700 1 -- 172.21.15.90:0/3862467937 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7feaa4027bb0 con 0x7feaa4012b50 2024-04-14T15:22:42.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.672+0000 7feab4ff9700 1 -- 172.21.15.90:0/3862467937 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7feaa4027bb0 con 0x7feaa4012b50 2024-04-14T15:22:42.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.672+0000 7feab4ff9700 10 client.37713.objecter ms_dispatch 0x7feab8001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:22:42.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.672+0000 7feab4ff9700 10 client.37713.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:22:42.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.672+0000 7feab4ff9700 20 client.37713.objecter put_session s=0x7feaa40129e0 osd=7 4 2024-04-14T15:22:42.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.672+0000 7feab4ff9700 15 client.37713.objecter _session_command_op_remove 7 16 2024-04-14T15:22:42.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.672+0000 7feabdb9a700 10 librados: watch_flush enter 2024-04-14T15:22:42.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.672+0000 7feabdb9a700 10 librados: watch_flush exit 2024-04-14T15:22:42.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.672+0000 7feabdb9a700 10 client.37713.objecter close_session for osd.0 2024-04-14T15:22:42.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.672+0000 7feabdb9a700 1 -- 172.21.15.90:0/3862467937 >> v1:172.21.15.90:6801/3761806885 conn(0x7feaa4004730 legacy=0x7feaa4002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:42.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.672+0000 7feabdb9a700 20 client.37713.objecter put_session s=0x7feaa40053e0 osd=0 1 2024-04-14T15:22:42.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.672+0000 7feabdb9a700 10 client.37713.objecter close_session for osd.1 2024-04-14T15:22:42.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.672+0000 7feabdb9a700 1 -- 172.21.15.90:0/3862467937 >> v1:172.21.15.90:6805/4170361699 conn(0x7feaa4001b80 legacy=0x7feaa400ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:42.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.672+0000 7feabdb9a700 20 client.37713.objecter put_session s=0x7feaa4004f90 osd=1 1 2024-04-14T15:22:42.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.672+0000 7feabdb9a700 10 client.37713.objecter close_session for osd.2 2024-04-14T15:22:42.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.672+0000 7feabdb9a700 1 -- 172.21.15.90:0/3862467937 >> v1:172.21.15.90:6809/1686184251 conn(0x7feaa4008990 legacy=0x7feaa400f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:42.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.672+0000 7feabdb9a700 20 client.37713.objecter put_session s=0x7feaa4001a00 osd=2 1 2024-04-14T15:22:42.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.672+0000 7feabdb9a700 10 client.37713.objecter close_session for osd.3 2024-04-14T15:22:42.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.672+0000 7feabdb9a700 1 -- 172.21.15.90:0/3862467937 >> v1:172.21.15.90:6813/2270166774 conn(0x7feaa4011160 legacy=0x7feaa40115a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:42.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.672+0000 7feabdb9a700 20 client.37713.objecter put_session s=0x7feaa400cd30 osd=3 1 2024-04-14T15:22:42.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.672+0000 7feabdb9a700 10 client.37713.objecter close_session for osd.4 2024-04-14T15:22:42.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.672+0000 7feabdb9a700 1 -- 172.21.15.90:0/3862467937 >> v1:172.21.15.92:6800/3887549733 conn(0x7feaa400c8d0 legacy=0x7feaa4019090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:42.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.672+0000 7feabdb9a700 20 client.37713.objecter put_session s=0x7feaa4013950 osd=4 1 2024-04-14T15:22:42.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.672+0000 7feabdb9a700 10 client.37713.objecter close_session for osd.5 2024-04-14T15:22:42.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.672+0000 7feabdb9a700 1 -- 172.21.15.90:0/3862467937 >> v1:172.21.15.92:6804/3244561440 conn(0x7feaa4013470 legacy=0x7feaa401b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:42.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.672+0000 7feabdb9a700 20 client.37713.objecter put_session s=0x7feaa4011cb0 osd=5 1 2024-04-14T15:22:42.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.672+0000 7feabdb9a700 10 client.37713.objecter close_session for osd.6 2024-04-14T15:22:42.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.672+0000 7feabdb9a700 1 -- 172.21.15.90:0/3862467937 >> v1:172.21.15.92:6808/636125963 conn(0x7feaa401f7f0 legacy=0x7feaa4021be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:42.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.672+0000 7feabdb9a700 20 client.37713.objecter put_session s=0x7feaa401d050 osd=6 1 2024-04-14T15:22:42.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.672+0000 7feabdb9a700 10 client.37713.objecter close_session for osd.7 2024-04-14T15:22:42.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.672+0000 7feabdb9a700 1 -- 172.21.15.90:0/3862467937 >> v1:172.21.15.92:6812/1773706716 conn(0x7feaa4012b50 legacy=0x7feaa4026490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:42.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.672+0000 7feabdb9a700 20 client.37713.objecter put_session s=0x7feaa40129e0 osd=7 1 2024-04-14T15:22:42.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.672+0000 7feabdb9a700 20 client.37713.objecter shutdown clearing up homeless session... 2024-04-14T15:22:42.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.672+0000 7feabdb9a700 10 client.37713.objecter successfully canceled tick 2024-04-14T15:22:42.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.672+0000 7feab7fff700 1 -- 172.21.15.90:0/3862467937 reap_dead start 2024-04-14T15:22:42.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.672+0000 7feab7fff700 1 -- 172.21.15.90:0/3862467937 reap_dead start 2024-04-14T15:22:42.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.672+0000 7feab7fff700 1 -- 172.21.15.90:0/3862467937 reap_dead start 2024-04-14T15:22:42.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.672+0000 7feab7fff700 1 -- 172.21.15.90:0/3862467937 reap_dead start 2024-04-14T15:22:42.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.673+0000 7feab7fff700 1 -- 172.21.15.90:0/3862467937 reap_dead start 2024-04-14T15:22:42.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.673+0000 7feabdb9a700 1 -- 172.21.15.90:0/3862467937 >> v1:172.21.15.90:6800/420935427 conn(0x7feaa0071380 legacy=0x7feaa0073840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:42.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.673+0000 7feabdb9a700 1 -- 172.21.15.90:0/3862467937 >> v1:172.21.15.90:6789/0 conn(0x7feab810e1e0 legacy=0x7feab81a81c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:42.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.673+0000 7feabdb9a700 1 -- 172.21.15.90:0/3862467937 shutdown_connections 2024-04-14T15:22:42.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.673+0000 7feabdb9a700 1 -- 172.21.15.90:0/3862467937 >> 172.21.15.90:0/3862467937 conn(0x7feab80fd290 msgr2=0x7feab81006f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:22:42.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.673+0000 7feabdb9a700 1 -- 172.21.15.90:0/3862467937 shutdown_connections 2024-04-14T15:22:42.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.673+0000 7feabdb9a700 1 -- 172.21.15.90:0/3862467937 wait complete. 2024-04-14T15:22:42.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:41.673+0000 7feabdb9a700 1 librados: shutdown 2024-04-14T15:22:42.270 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:22:42.270 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:22:42.270 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:22:42.270 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:22:42.270 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:22:42.270 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:22:42.270 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:22:42.270 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:22:42.333 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:22:42.356 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:42.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:42.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:22:42.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:42 smithi092 ceph-mon[33966]: osdmap e625: 8 total, 8 up, 7 in 2024-04-14T15:22:42.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:42 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3862467937' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:22:42.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:42 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi090-93652-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi090-93652-109"}]': finished 2024-04-14T15:22:42.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:42 smithi092 ceph-mon[33966]: osdmap e626: 8 total, 8 up, 7 in 2024-04-14T15:22:42.415 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:22:42.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:42 smithi090 ceph-mon[41670]: osdmap e625: 8 total, 8 up, 7 in 2024-04-14T15:22:42.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:42 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3862467937' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:22:42.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:42 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi090-93652-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi090-93652-109"}]': finished 2024-04-14T15:22:42.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:42 smithi090 ceph-mon[41670]: osdmap e626: 8 total, 8 up, 7 in 2024-04-14T15:22:42.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:42 smithi090 ceph-mon[30399]: osdmap e625: 8 total, 8 up, 7 in 2024-04-14T15:22:42.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:42 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3862467937' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:22:42.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:42 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi090-93652-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi090-93652-109"}]': finished 2024-04-14T15:22:42.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:42 smithi090 ceph-mon[30399]: osdmap e626: 8 total, 8 up, 7 in 2024-04-14T15:22:42.458 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:42.481 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:42.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:42.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:42.582 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:22:42.606 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:42.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:42.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:22:42.708 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:22:42.731 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:42.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:42.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:22:42.832 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:22:42.859 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:42.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:42.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:22:42.962 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:22:42.996 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:42.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:42.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:22:43.098 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:22:43.121 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:43.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:43.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:22:43.223 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:22:43.250 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:43.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:43.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:22:43.286 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:43 smithi090 ceph-mon[41670]: pgmap v918: 186 pgs: 3 active+remapped+backfill_toofull, 183 active+clean; 596 KiB data, 519 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s; 2/600 objects misplaced (0.333%) 2024-04-14T15:22:43.286 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:43 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:22:43.286 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:43 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:22:43.286 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:43 smithi090 ceph-mon[41670]: osdmap e627: 8 total, 8 up, 7 in 2024-04-14T15:22:43.286 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:43 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:22:43.287 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:43 smithi090 ceph-mon[30399]: pgmap v918: 186 pgs: 3 active+remapped+backfill_toofull, 183 active+clean; 596 KiB data, 519 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s; 2/600 objects misplaced (0.333%) 2024-04-14T15:22:43.287 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:43 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:22:43.287 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:43 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:22:43.287 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:43 smithi090 ceph-mon[30399]: osdmap e627: 8 total, 8 up, 7 in 2024-04-14T15:22:43.287 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:43 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:22:43.352 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:22:43.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:43 smithi092 ceph-mon[33966]: pgmap v918: 186 pgs: 3 active+remapped+backfill_toofull, 183 active+clean; 596 KiB data, 519 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s; 2/600 objects misplaced (0.333%) 2024-04-14T15:22:43.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:43 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:22:43.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:43 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:22:43.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:43 smithi092 ceph-mon[33966]: osdmap e627: 8 total, 8 up, 7 in 2024-04-14T15:22:43.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:43 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:22:43.380 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:43.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:43.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:22:43.482 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:22:43.509 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:43.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:43.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:22:43.611 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:22:43.638 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:43.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:43.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:22:43.740 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:22:43.767 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:43.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:43.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:22:43.868 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:22:43.893 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:43.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:43.894 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:22:43.994 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:22:44.027 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:44.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:44.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:22:44.128 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:22:44.156 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:44.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:44.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:22:44.211 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:22:43 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:22:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:22:44.258 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:22:44.286 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:44.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:44.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:22:44.387 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:22:44.415 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:44.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:44.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:22:44.518 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:22:44.545 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:44.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:44.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:22:44.605 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:22:44.605 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:22:44.616 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-14T15:22:44.616 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-14T15:22:44.616 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-14T15:22:44.616 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-14T15:22:44.616 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-14T15:22:44.647 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:44.671 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:44.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:44.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:44.772 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:22:44.800 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:44.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:44.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:22:44.901 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:22:44.929 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:44.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:44.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:22:45.030 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:22:45.057 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:45.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:45.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:22:45.158 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:22:45.183 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:45.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:45.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:22:45.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:44 smithi090 ceph-mon[41670]: pgmap v921: 226 pgs: 40 unknown, 3 active+remapped+backfill_toofull, 183 active+clean; 596 KiB data, 519 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s; 2/600 objects misplaced (0.333%) 2024-04-14T15:22:45.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:44 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-109-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:22:45.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:44 smithi090 ceph-mon[41670]: osdmap e628: 8 total, 8 up, 7 in 2024-04-14T15:22:45.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:44 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-109", "tierpool": "test-rados-api-smithi090-93652-109-cache"}]: dispatch 2024-04-14T15:22:45.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:44 smithi090 ceph-mon[41670]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:22:45.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:44 smithi090 ceph-mon[30399]: pgmap v921: 226 pgs: 40 unknown, 3 active+remapped+backfill_toofull, 183 active+clean; 596 KiB data, 519 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s; 2/600 objects misplaced (0.333%) 2024-04-14T15:22:45.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:44 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-109-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:22:45.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:44 smithi090 ceph-mon[30399]: osdmap e628: 8 total, 8 up, 7 in 2024-04-14T15:22:45.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:44 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-109", "tierpool": "test-rados-api-smithi090-93652-109-cache"}]: dispatch 2024-04-14T15:22:45.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:44 smithi090 ceph-mon[30399]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:22:45.246 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-14T15:22:45.285 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:22:45.310 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:45.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:45.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:22:45.313 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:44 smithi092 ceph-mon[33966]: pgmap v921: 226 pgs: 40 unknown, 3 active+remapped+backfill_toofull, 183 active+clean; 596 KiB data, 519 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s; 2/600 objects misplaced (0.333%) 2024-04-14T15:22:45.313 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:44 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-109-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:22:45.313 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:44 smithi092 ceph-mon[33966]: osdmap e628: 8 total, 8 up, 7 in 2024-04-14T15:22:45.313 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:44 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-109", "tierpool": "test-rados-api-smithi090-93652-109-cache"}]: dispatch 2024-04-14T15:22:45.313 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:44 smithi092 ceph-mon[33966]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:22:45.412 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:22:45.436 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:45.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:45.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:22:45.538 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:22:45.564 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:45.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:45.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:22:45.622 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:22:45 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:22:45.666 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:22:45.692 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:45.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:45.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:22:45.794 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:22:45.819 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:45.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:45.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:22:45.921 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:22:45.947 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:45.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:45.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:22:46.049 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:22:46.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:46.074 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:46.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:22:46.110 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-14T15:22:46.138 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:46 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-109", "tierpool": "test-rados-api-smithi090-93652-109-cache"}]': finished 2024-04-14T15:22:46.138 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:46 smithi090 ceph-mon[41670]: osdmap e629: 8 total, 8 up, 7 in 2024-04-14T15:22:46.138 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:46 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-109", "overlaypool": "test-rados-api-smithi090-93652-109-cache"}]: dispatch 2024-04-14T15:22:46.138 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:46 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-109", "overlaypool": "test-rados-api-smithi090-93652-109-cache"}]': finished 2024-04-14T15:22:46.139 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:46 smithi090 ceph-mon[41670]: osdmap e630: 8 total, 8 up, 7 in 2024-04-14T15:22:46.139 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:46 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-109-cache", "mode": "writeback"}]: dispatch 2024-04-14T15:22:46.139 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:46 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-109", "tierpool": "test-rados-api-smithi090-93652-109-cache"}]': finished 2024-04-14T15:22:46.139 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:46 smithi090 ceph-mon[30399]: osdmap e629: 8 total, 8 up, 7 in 2024-04-14T15:22:46.139 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:46 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-109", "overlaypool": "test-rados-api-smithi090-93652-109-cache"}]: dispatch 2024-04-14T15:22:46.139 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:46 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-109", "overlaypool": "test-rados-api-smithi090-93652-109-cache"}]': finished 2024-04-14T15:22:46.139 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:46 smithi090 ceph-mon[30399]: osdmap e630: 8 total, 8 up, 7 in 2024-04-14T15:22:46.139 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:46 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-109-cache", "mode": "writeback"}]: dispatch 2024-04-14T15:22:46.176 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:22:46.199 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:46.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:46.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:22:46.301 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:46.323 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:46.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:46.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:46.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:46 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-109", "tierpool": "test-rados-api-smithi090-93652-109-cache"}]': finished 2024-04-14T15:22:46.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:46 smithi092 ceph-mon[33966]: osdmap e629: 8 total, 8 up, 7 in 2024-04-14T15:22:46.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:46 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-109", "overlaypool": "test-rados-api-smithi090-93652-109-cache"}]: dispatch 2024-04-14T15:22:46.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:46 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-109", "overlaypool": "test-rados-api-smithi090-93652-109-cache"}]': finished 2024-04-14T15:22:46.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:46 smithi092 ceph-mon[33966]: osdmap e630: 8 total, 8 up, 7 in 2024-04-14T15:22:46.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:46 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-109-cache", "mode": "writeback"}]: dispatch 2024-04-14T15:22:46.425 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:22:46.450 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:46.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:46.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:22:46.552 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:22:46.576 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:46.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:46.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:22:46.678 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:22:46.705 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:46.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:46.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:22:46.807 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:46.830 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:46.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:46.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:46.932 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:22:46.955 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:46.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:46.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:22:47.057 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:22:47.085 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:47.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:47.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:22:47.186 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:22:47.210 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:47.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:47.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:22:47.299 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:47 smithi090 ceph-mon[41670]: pgmap v924: 226 pgs: 12 creating+activating, 3 active+remapped+backfill_toofull, 211 active+clean; 596 KiB data, 520 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/600 objects misplaced (0.333%) 2024-04-14T15:22:47.299 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:47 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:22:47.299 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:47 smithi090 ceph-mon[41670]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:22:47.300 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:47 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-109-cache", "mode": "writeback"}]': finished 2024-04-14T15:22:47.300 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:47 smithi090 ceph-mon[41670]: osdmap e631: 8 total, 8 up, 7 in 2024-04-14T15:22:47.300 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:47 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-04-14T15:22:47.300 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:47 smithi090 ceph-mon[30399]: pgmap v924: 226 pgs: 12 creating+activating, 3 active+remapped+backfill_toofull, 211 active+clean; 596 KiB data, 520 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/600 objects misplaced (0.333%) 2024-04-14T15:22:47.300 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:47 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:22:47.300 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:47 smithi090 ceph-mon[30399]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:22:47.300 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:47 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-109-cache", "mode": "writeback"}]': finished 2024-04-14T15:22:47.300 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:47 smithi090 ceph-mon[30399]: osdmap e631: 8 total, 8 up, 7 in 2024-04-14T15:22:47.300 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:47 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-04-14T15:22:47.312 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:22:47.337 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:47.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:47.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:22:47.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:47 smithi092 ceph-mon[33966]: pgmap v924: 226 pgs: 12 creating+activating, 3 active+remapped+backfill_toofull, 211 active+clean; 596 KiB data, 520 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/600 objects misplaced (0.333%) 2024-04-14T15:22:47.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:47 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:22:47.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:47 smithi092 ceph-mon[33966]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:22:47.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:47 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-109-cache", "mode": "writeback"}]': finished 2024-04-14T15:22:47.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:47 smithi092 ceph-mon[33966]: osdmap e631: 8 total, 8 up, 7 in 2024-04-14T15:22:47.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:47 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-04-14T15:22:47.438 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:22:47.462 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:47.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:47.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:22:47.563 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:47.586 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:47.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:47.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:47.688 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:22:47.716 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:47.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:47.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:22:47.873 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:22:47.896 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:47.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:47.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:22:47.997 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:48.021 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:48.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:48.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:48.123 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:22:48.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:47.958+0000 7f34b26d5700 1 -- 172.21.15.90:0/1012659306 >> v1:172.21.15.90:6789/0 conn(0x7f34ac10e1e0 legacy=0x7f34ac1106a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:48.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:47.958+0000 7f34b26d5700 1 -- 172.21.15.90:0/1012659306 shutdown_connections 2024-04-14T15:22:48.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:47.958+0000 7f34b26d5700 1 -- 172.21.15.90:0/1012659306 >> 172.21.15.90:0/1012659306 conn(0x7f34ac0fd290 msgr2=0x7f34ac0ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:22:48.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:47.959+0000 7f34b26d5700 1 -- 172.21.15.90:0/1012659306 shutdown_connections 2024-04-14T15:22:48.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:47.959+0000 7f34b26d5700 1 -- 172.21.15.90:0/1012659306 wait complete. 2024-04-14T15:22:48.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:47.959+0000 7f34b26d5700 1 librados: starting msgr at 2024-04-14T15:22:48.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:47.959+0000 7f34b26d5700 1 librados: starting objecter 2024-04-14T15:22:48.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:47.959+0000 7f34b26d5700 1 Processor -- start 2024-04-14T15:22:48.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:47.959+0000 7f34b26d5700 1 -- start start 2024-04-14T15:22:48.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:47.959+0000 7f34b26d5700 1 librados: setting wanted keys 2024-04-14T15:22:48.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:47.959+0000 7f34b26d5700 1 librados: calling monclient init 2024-04-14T15:22:48.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:47.959+0000 7f34b26d5700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f34ac10ded0 con 0x7f34ac107760 2024-04-14T15:22:48.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:47.959+0000 7f34b26d5700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f34ac1a7df0 con 0x7f34ac10a610 2024-04-14T15:22:48.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:47.959+0000 7f34b26d5700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f34ac1a8fd0 con 0x7f34ac10e1e0 2024-04-14T15:22:48.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:47.960+0000 7f34abfff700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f34ac107760 0x7f34ac10aeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50172/0 (socket says 172.21.15.90:50172) 2024-04-14T15:22:48.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:47.960+0000 7f34abfff700 1 -- 172.21.15.90:0/1921335352 learned_addr learned my addr 172.21.15.90:0/1921335352 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:22:48.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:47.960+0000 7f34a97fa700 10 client.?.objecter ms_handle_connect 0x7f34ac107760 2024-04-14T15:22:48.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:47.960+0000 7f34a97fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:22:48.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:47.960+0000 7f34a97fa700 10 client.?.objecter ms_handle_connect 0x7f34ac10e1e0 2024-04-14T15:22:48.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:47.960+0000 7f34a97fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:22:48.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:47.960+0000 7f34a97fa700 1 -- 172.21.15.90:0/1921335352 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3989906232 0 0) 0x7f34ac10ded0 con 0x7f34ac107760 2024-04-14T15:22:48.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:47.960+0000 7f34a97fa700 1 -- 172.21.15.90:0/1921335352 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f34940035b0 con 0x7f34ac107760 2024-04-14T15:22:48.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:47.960+0000 7f34a97fa700 1 -- 172.21.15.90:0/1921335352 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2471874718 0 0) 0x7f34ac1a8fd0 con 0x7f34ac10e1e0 2024-04-14T15:22:48.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:47.960+0000 7f34a97fa700 1 -- 172.21.15.90:0/1921335352 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f34ac10ded0 con 0x7f34ac10e1e0 2024-04-14T15:22:48.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:47.960+0000 7f34a97fa700 10 client.?.objecter ms_handle_connect 0x7f34ac10a610 2024-04-14T15:22:48.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:47.960+0000 7f34a97fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:22:48.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:47.960+0000 7f34a97fa700 1 -- 172.21.15.90:0/1921335352 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2665207570 0 0) 0x7f34ac10ded0 con 0x7f34ac10e1e0 2024-04-14T15:22:48.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:47.960+0000 7f34a97fa700 1 -- 172.21.15.90:0/1921335352 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f34ac1a8fd0 con 0x7f34ac10e1e0 2024-04-14T15:22:48.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:47.960+0000 7f34a97fa700 1 -- 172.21.15.90:0/1921335352 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f34a40017b0 con 0x7f34ac10e1e0 2024-04-14T15:22:48.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:47.960+0000 7f34a97fa700 1 -- 172.21.15.90:0/1921335352 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2412297158 0 0) 0x7f34940035b0 con 0x7f34ac107760 2024-04-14T15:22:48.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:47.960+0000 7f34a97fa700 1 -- 172.21.15.90:0/1921335352 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f34ac10ded0 con 0x7f34ac107760 2024-04-14T15:22:48.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:47.960+0000 7f34a97fa700 1 -- 172.21.15.90:0/1921335352 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2366984692 0 0) 0x7f34ac1a7df0 con 0x7f34ac10a610 2024-04-14T15:22:48.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:47.960+0000 7f34a97fa700 1 -- 172.21.15.90:0/1921335352 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f34940035b0 con 0x7f34ac10a610 2024-04-14T15:22:48.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:47.960+0000 7f34a97fa700 1 -- 172.21.15.90:0/1921335352 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f349c0018b0 con 0x7f34ac107760 2024-04-14T15:22:48.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:47.960+0000 7f34a97fa700 1 -- 172.21.15.90:0/1921335352 <== mon.2 v1:172.21.15.90:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2306479892 0 0) 0x7f34ac1a8fd0 con 0x7f34ac10e1e0 2024-04-14T15:22:48.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:47.960+0000 7f34a97fa700 1 -- 172.21.15.90:0/1921335352 >> v1:172.21.15.92:6789/0 conn(0x7f34ac10a610 legacy=0x7f34ac10b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:48.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:47.960+0000 7f34a97fa700 1 -- 172.21.15.90:0/1921335352 >> v1:172.21.15.90:6789/0 conn(0x7f34ac107760 legacy=0x7f34ac10aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:48.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:47.960+0000 7f34a97fa700 1 -- 172.21.15.90:0/1921335352 --> v1:172.21.15.90:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34ac1aa1b0 con 0x7f34ac10e1e0 2024-04-14T15:22:48.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:47.960+0000 7f34b26d5700 1 -- 172.21.15.90:0/1921335352 --> v1:172.21.15.90:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f34ac1a9200 con 0x7f34ac10e1e0 2024-04-14T15:22:48.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:47.960+0000 7f34b26d5700 10 client.47669.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:22:48.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:47.960+0000 7f34b26d5700 1 -- 172.21.15.90:0/1921335352 --> v1:172.21.15.90:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f34ac1a96c0 con 0x7f34ac10e1e0 2024-04-14T15:22:48.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:47.960+0000 7f34b26d5700 1 librados: init done 2024-04-14T15:22:48.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:47.960+0000 7f34a97fa700 1 -- 172.21.15.90:0/1921335352 <== mon.2 v1:172.21.15.90:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f34a40019d0 con 0x7f34ac10e1e0 2024-04-14T15:22:48.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:47.960+0000 7f34a97fa700 1 -- 172.21.15.90:0/1921335352 <== mon.2 v1:172.21.15.90:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f34a4005090 con 0x7f34ac10e1e0 2024-04-14T15:22:48.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:47.961+0000 7f34b26d5700 1 -- 172.21.15.90:0/1921335352 --> v1:172.21.15.90:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3498005120 con 0x7f34ac10e1e0 2024-04-14T15:22:48.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:47.962+0000 7f34a97fa700 1 -- 172.21.15.90:0/1921335352 <== mon.2 v1:172.21.15.90:6790/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f34a4001eb0 con 0x7f34ac10e1e0 2024-04-14T15:22:48.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:47.962+0000 7f34a97fa700 1 -- 172.21.15.90:0/1921335352 <== mon.2 v1:172.21.15.90:6790/0 8 ==== osd_map(631..631 src has 467..631) ==== 9645+0+0 (unknown 1921274430 0 0) 0x7f34a408a350 con 0x7f34ac10e1e0 2024-04-14T15:22:48.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:47.962+0000 7f34a97fa700 10 client.47669.objecter ms_dispatch 0x7f34ac001550 osd_map(631..631 src has 467..631) 2024-04-14T15:22:48.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:47.962+0000 7f34a97fa700 3 client.47669.objecter handle_osd_map got epochs [631,631] > 0 2024-04-14T15:22:48.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:47.962+0000 7f34a97fa700 3 client.47669.objecter handle_osd_map decoding full epoch 631 2024-04-14T15:22:48.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:47.963+0000 7f34a97fa700 20 client.47669.objecter dump_active .. 0 homeless 2024-04-14T15:22:48.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:47.963+0000 7f34a97fa700 10 client.47669.objecter ms_handle_connect 0x7f34940712a0 2024-04-14T15:22:48.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:47.967+0000 7f34a97fa700 1 -- 172.21.15.90:0/1921335352 <== mon.2 v1:172.21.15.90:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f34a40567b0 con 0x7f34ac10e1e0 2024-04-14T15:22:48.143 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:22:48.148 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:48.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:48.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:22:48.179 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:22:48.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:48.209+0000 7f34b26d5700 1 -- 172.21.15.90:0/1921335352 --> v1:172.21.15.90:6790/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7f3498005410 con 0x7f34ac10e1e0 2024-04-14T15:22:48.249 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:22:48.277 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:48.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:48.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:22:48.379 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:22:48.404 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:48.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:48.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:22:48.506 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:22:48.534 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:48.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:48.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:22:48.636 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:48.659 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:48.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:48.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:48.761 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:22:48.786 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:48.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:48.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:22:48.887 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:22:48.911 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:48.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:48.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:22:48.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:48.976+0000 7f34a97fa700 1 -- 172.21.15.90:0/1921335352 <== mon.2 v1:172.21.15.90:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v633) ==== 91+0+0 (unknown 2919217982 0 0) 0x7f34a405a460 con 0x7f34ac10e1e0 2024-04-14T15:22:48.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:48.979+0000 7f34b26d5700 10 librados: watch_flush enter 2024-04-14T15:22:48.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:48.979+0000 7f34b26d5700 10 librados: watch_flush exit 2024-04-14T15:22:48.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:48.979+0000 7f34b26d5700 20 client.47669.objecter shutdown clearing up homeless session... 2024-04-14T15:22:48.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:48.979+0000 7f34b26d5700 10 client.47669.objecter successfully canceled tick 2024-04-14T15:22:48.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:48.979+0000 7f34b26d5700 1 -- 172.21.15.90:0/1921335352 >> v1:172.21.15.90:6800/420935427 conn(0x7f34940712a0 legacy=0x7f3494073760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:48.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:48.979+0000 7f34b26d5700 1 -- 172.21.15.90:0/1921335352 >> v1:172.21.15.90:6790/0 conn(0x7f34ac10e1e0 legacy=0x7f34ac1a56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:48.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:48.980+0000 7f34b26d5700 1 -- 172.21.15.90:0/1921335352 shutdown_connections 2024-04-14T15:22:48.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:48.980+0000 7f34b26d5700 1 -- 172.21.15.90:0/1921335352 >> 172.21.15.90:0/1921335352 conn(0x7f34ac0fd290 msgr2=0x7f34ac1117d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:22:48.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:48.980+0000 7f34b26d5700 1 -- 172.21.15.90:0/1921335352 shutdown_connections 2024-04-14T15:22:48.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:48.980+0000 7f34b26d5700 1 -- 172.21.15.90:0/1921335352 wait complete. 2024-04-14T15:22:48.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:48.980+0000 7f34b26d5700 1 librados: shutdown 2024-04-14T15:22:48.981 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is unset 2024-04-14T15:22:49.013 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:22:49.041 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:49.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:49.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:22:49.143 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:22:49.170 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:49.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:49.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:22:49.272 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:22:49.299 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:49.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:49.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:22:49.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:48 smithi092 ceph-mon[33966]: pgmap v927: 226 pgs: 12 creating+activating, 3 active+remapped+backfill_toofull, 211 active+clean; 596 KiB data, 520 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/600 objects misplaced (0.333%) 2024-04-14T15:22:49.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:48 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-109-cache","var": "hit_set_count","val": "2"}]': finished 2024-04-14T15:22:49.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:48 smithi092 ceph-mon[33966]: osdmap e632: 8 total, 8 up, 7 in 2024-04-14T15:22:49.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:48 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-04-14T15:22:49.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:48 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1921335352' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:22:49.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:48 smithi092 ceph-mon[33966]: from='client.47669 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:22:49.401 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:22:49.424 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:49.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:49.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:22:49.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:48 smithi090 ceph-mon[30399]: pgmap v927: 226 pgs: 12 creating+activating, 3 active+remapped+backfill_toofull, 211 active+clean; 596 KiB data, 520 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/600 objects misplaced (0.333%) 2024-04-14T15:22:49.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:48 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-109-cache","var": "hit_set_count","val": "2"}]': finished 2024-04-14T15:22:49.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:48 smithi090 ceph-mon[30399]: osdmap e632: 8 total, 8 up, 7 in 2024-04-14T15:22:49.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:48 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-04-14T15:22:49.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:48 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1921335352' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:22:49.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:48 smithi090 ceph-mon[30399]: from='client.47669 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:22:49.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:48 smithi090 ceph-mon[41670]: pgmap v927: 226 pgs: 12 creating+activating, 3 active+remapped+backfill_toofull, 211 active+clean; 596 KiB data, 520 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/600 objects misplaced (0.333%) 2024-04-14T15:22:49.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:48 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-109-cache","var": "hit_set_count","val": "2"}]': finished 2024-04-14T15:22:49.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:48 smithi090 ceph-mon[41670]: osdmap e632: 8 total, 8 up, 7 in 2024-04-14T15:22:49.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:48 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-04-14T15:22:49.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:48 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1921335352' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:22:49.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:48 smithi090 ceph-mon[41670]: from='client.47669 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:22:49.526 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:49.549 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:49.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:49.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:49.650 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:49.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:49.674 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:49.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:49.775 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:22:49.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:49.804 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:49.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:22:49.905 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:22:49.932 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:49.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:49.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:22:50.034 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:22:50.061 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:50.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:50.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:22:50.163 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:22:50.191 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:50.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:50.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:22:50.238 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:49 smithi090 ceph-mon[41670]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:22:50.238 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:49 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-109-cache","var": "hit_set_period","val": "600"}]': finished 2024-04-14T15:22:50.239 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:49 smithi090 ceph-mon[41670]: from='client.47669 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-14T15:22:50.239 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:49 smithi090 ceph-mon[41670]: osdmap e633: 8 total, 8 up, 7 in 2024-04-14T15:22:50.239 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:49 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-14T15:22:50.239 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:49 smithi090 ceph-mon[41670]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:22:50.239 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:49 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-109-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-04-14T15:22:50.239 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:49 smithi090 ceph-mon[41670]: osdmap e634: 8 total, 8 up, 7 in 2024-04-14T15:22:50.239 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:49 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-04-14T15:22:50.239 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:49 smithi090 ceph-mon[30399]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:22:50.239 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:49 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-109-cache","var": "hit_set_period","val": "600"}]': finished 2024-04-14T15:22:50.239 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:49 smithi090 ceph-mon[30399]: from='client.47669 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-14T15:22:50.239 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:49 smithi090 ceph-mon[30399]: osdmap e633: 8 total, 8 up, 7 in 2024-04-14T15:22:50.239 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:49 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-14T15:22:50.240 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:49 smithi090 ceph-mon[30399]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:22:50.240 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:49 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-109-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-04-14T15:22:50.240 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:49 smithi090 ceph-mon[30399]: osdmap e634: 8 total, 8 up, 7 in 2024-04-14T15:22:50.240 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:49 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-04-14T15:22:50.293 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:22:50.321 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:50.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:50.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:22:50.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.341+0000 7f097fffd700 1 -- 172.21.15.90:0/216000736 >> v1:172.21.15.90:6789/0 conn(0x7f0978103b10 legacy=0x7f0978103ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:50.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.341+0000 7f097fffd700 1 -- 172.21.15.90:0/216000736 shutdown_connections 2024-04-14T15:22:50.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.341+0000 7f097fffd700 1 -- 172.21.15.90:0/216000736 >> 172.21.15.90:0/216000736 conn(0x7f09780fd2c0 msgr2=0x7f09780ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:22:50.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.341+0000 7f097fffd700 1 -- 172.21.15.90:0/216000736 shutdown_connections 2024-04-14T15:22:50.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.341+0000 7f097fffd700 1 -- 172.21.15.90:0/216000736 wait complete. 2024-04-14T15:22:50.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.341+0000 7f097fffd700 1 librados: starting msgr at 2024-04-14T15:22:50.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.341+0000 7f097fffd700 1 librados: starting objecter 2024-04-14T15:22:50.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.341+0000 7f097fffd700 1 Processor -- start 2024-04-14T15:22:50.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.341+0000 7f097fffd700 1 -- start start 2024-04-14T15:22:50.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.341+0000 7f097fffd700 1 librados: setting wanted keys 2024-04-14T15:22:50.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.341+0000 7f097fffd700 1 librados: calling monclient init 2024-04-14T15:22:50.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.342+0000 7f097fffd700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f09781a79c0 con 0x7f097810f430 2024-04-14T15:22:50.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.342+0000 7f097fffd700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f09781a8be0 con 0x7f097810b920 2024-04-14T15:22:50.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.342+0000 7f097fffd700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f09781a9e20 con 0x7f0978103b10 2024-04-14T15:22:50.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.342+0000 7f097dd99700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f0978103b10 0x7f097810b180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:35266/0 (socket says 172.21.15.90:35266) 2024-04-14T15:22:50.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.342+0000 7f097dd99700 1 -- 172.21.15.90:0/1013056523 learned_addr learned my addr 172.21.15.90:0/1013056523 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:22:50.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.342+0000 7f096effd700 10 client.?.objecter ms_handle_connect 0x7f097810f430 2024-04-14T15:22:50.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.342+0000 7f096effd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:22:50.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.342+0000 7f096effd700 10 client.?.objecter ms_handle_connect 0x7f0978103b10 2024-04-14T15:22:50.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.342+0000 7f096effd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:22:50.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.342+0000 7f096effd700 10 client.?.objecter ms_handle_connect 0x7f097810b920 2024-04-14T15:22:50.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.342+0000 7f096effd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:22:50.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.342+0000 7f096effd700 1 -- 172.21.15.90:0/1013056523 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 157251224 0 0) 0x7f09781a9e20 con 0x7f0978103b10 2024-04-14T15:22:50.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.342+0000 7f096effd700 1 -- 172.21.15.90:0/1013056523 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f09640035b0 con 0x7f0978103b10 2024-04-14T15:22:50.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.342+0000 7f096effd700 1 -- 172.21.15.90:0/1013056523 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 553386732 0 0) 0x7f09781a79c0 con 0x7f097810f430 2024-04-14T15:22:50.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.342+0000 7f096effd700 1 -- 172.21.15.90:0/1013056523 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f09781a9e20 con 0x7f097810f430 2024-04-14T15:22:50.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.342+0000 7f096effd700 1 -- 172.21.15.90:0/1013056523 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 823556952 0 0) 0x7f09781a8be0 con 0x7f097810b920 2024-04-14T15:22:50.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.342+0000 7f096effd700 1 -- 172.21.15.90:0/1013056523 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f09781a79c0 con 0x7f097810b920 2024-04-14T15:22:50.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.342+0000 7f096effd700 1 -- 172.21.15.90:0/1013056523 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3102265745 0 0) 0x7f09781a9e20 con 0x7f097810f430 2024-04-14T15:22:50.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.342+0000 7f096effd700 1 -- 172.21.15.90:0/1013056523 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f09781a8be0 con 0x7f097810f430 2024-04-14T15:22:50.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.342+0000 7f096effd700 1 -- 172.21.15.90:0/1013056523 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f0970001c30 con 0x7f097810f430 2024-04-14T15:22:50.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.342+0000 7f096effd700 1 -- 172.21.15.90:0/1013056523 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1099235681 0 0) 0x7f09640035b0 con 0x7f0978103b10 2024-04-14T15:22:50.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.342+0000 7f096effd700 1 -- 172.21.15.90:0/1013056523 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f09781a9e20 con 0x7f0978103b10 2024-04-14T15:22:50.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.342+0000 7f096effd700 1 -- 172.21.15.90:0/1013056523 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 900036412 0 0) 0x7f09781a79c0 con 0x7f097810b920 2024-04-14T15:22:50.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.342+0000 7f096effd700 1 -- 172.21.15.90:0/1013056523 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f09640035b0 con 0x7f097810b920 2024-04-14T15:22:50.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.342+0000 7f096effd700 1 -- 172.21.15.90:0/1013056523 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f09740044e0 con 0x7f0978103b10 2024-04-14T15:22:50.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.342+0000 7f096effd700 1 -- 172.21.15.90:0/1013056523 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f0968001830 con 0x7f097810b920 2024-04-14T15:22:50.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.342+0000 7f096effd700 1 -- 172.21.15.90:0/1013056523 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2489997787 0 0) 0x7f09781a8be0 con 0x7f097810f430 2024-04-14T15:22:50.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.342+0000 7f096effd700 1 -- 172.21.15.90:0/1013056523 >> v1:172.21.15.90:6790/0 conn(0x7f0978103b10 legacy=0x7f097810b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:50.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.342+0000 7f096effd700 1 -- 172.21.15.90:0/1013056523 >> v1:172.21.15.92:6789/0 conn(0x7f097810b920 legacy=0x7f09781a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:50.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.342+0000 7f096effd700 1 -- 172.21.15.90:0/1013056523 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09781ab060 con 0x7f097810f430 2024-04-14T15:22:50.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.342+0000 7f097fffd700 1 -- 172.21.15.90:0/1013056523 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f09781aa050 con 0x7f097810f430 2024-04-14T15:22:50.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.342+0000 7f097fffd700 10 client.37725.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:22:50.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.342+0000 7f097fffd700 1 -- 172.21.15.90:0/1013056523 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f09781aa510 con 0x7f097810f430 2024-04-14T15:22:50.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.343+0000 7f097fffd700 1 librados: init done 2024-04-14T15:22:50.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.343+0000 7f096effd700 1 -- 172.21.15.90:0/1013056523 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f0970003ec0 con 0x7f097810f430 2024-04-14T15:22:50.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.343+0000 7f096effd700 1 -- 172.21.15.90:0/1013056523 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f0970004d20 con 0x7f097810f430 2024-04-14T15:22:50.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.343+0000 7f097fffd700 10 client.37725.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:22:50.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.343+0000 7f097fffd700 15 client.37725.objecter _session_command_op_assign -1 1 2024-04-14T15:22:50.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.343+0000 7f097fffd700 20 client.37725.objecter _get_session osd=-1 returning homeless 2024-04-14T15:22:50.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.343+0000 7f097fffd700 10 client.37725.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:22:50.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.343+0000 7f097fffd700 1 -- 172.21.15.90:0/1013056523 --> v1:172.21.15.90:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f095c002920 con 0x7f097810f430 2024-04-14T15:22:50.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.344+0000 7f096effd700 1 -- 172.21.15.90:0/1013056523 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f0970003a20 con 0x7f097810f430 2024-04-14T15:22:50.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.345+0000 7f096effd700 1 -- 172.21.15.90:0/1013056523 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(634..634 src has 467..634) ==== 9645+0+0 (unknown 4160382754 0 0) 0x7f097008a020 con 0x7f097810f430 2024-04-14T15:22:50.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.345+0000 7f096effd700 10 client.37725.objecter ms_dispatch 0x7f0978001550 osd_map(634..634 src has 467..634) 2024-04-14T15:22:50.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.345+0000 7f096effd700 3 client.37725.objecter handle_osd_map got epochs [634,634] > 0 2024-04-14T15:22:50.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.345+0000 7f096effd700 3 client.37725.objecter handle_osd_map decoding full epoch 634 2024-04-14T15:22:50.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.345+0000 7f096effd700 10 client.37725.objecter checking command 1 2024-04-14T15:22:50.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.345+0000 7f096effd700 20 client.37725.objecter _get_session s=0x7f096407fd90 osd=5 3 2024-04-14T15:22:50.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.345+0000 7f096effd700 20 client.37725.objecter put_session s=0x7f096407fd90 osd=5 3 2024-04-14T15:22:50.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.345+0000 7f096effd700 15 client.37725.objecter _session_command_op_remove -1 1 2024-04-14T15:22:50.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.345+0000 7f096effd700 20 client.37725.objecter _get_session s=0x7f096407fd90 osd=5 3 2024-04-14T15:22:50.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.345+0000 7f096effd700 20 client.37725.objecter get_session s=0x7f096407fd90 osd=5 3 2024-04-14T15:22:50.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.345+0000 7f096effd700 15 client.37725.objecter _session_command_op_assign 5 1 2024-04-14T15:22:50.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.345+0000 7f096effd700 20 client.37725.objecter put_session s=0x7f096407fd90 osd=5 4 2024-04-14T15:22:50.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.345+0000 7f096effd700 10 client.37725.objecter _send_command 1 2024-04-14T15:22:50.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.345+0000 7f096effd700 1 -- 172.21.15.90:0/1013056523 --> v1:172.21.15.92:6804/3244561440 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f09640841d0 con 0x7f0964080440 2024-04-14T15:22:50.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.345+0000 7f096effd700 20 client.37725.objecter dump_active .. 0 homeless 2024-04-14T15:22:50.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.345+0000 7f096effd700 1 -- 172.21.15.90:0/1013056523 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_get_version_reply(handle=1 version=634) ==== 24+0+0 (unknown 1719754456 0 0) 0x7f097008a430 con 0x7f097810f430 2024-04-14T15:22:50.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.345+0000 7f096effd700 10 client.37725.objecter ms_handle_connect 0x7f0964071480 2024-04-14T15:22:50.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.345+0000 7f096effd700 10 client.37725.objecter ms_handle_connect 0x7f0964080440 2024-04-14T15:22:50.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.346+0000 7f096effd700 1 -- 172.21.15.90:0/1013056523 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f09640841d0 con 0x7f0964080440 2024-04-14T15:22:50.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.346+0000 7f096effd700 10 client.37725.objecter ms_dispatch 0x7f0978001550 command_reply(tid 1: 0 ) 2024-04-14T15:22:50.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.346+0000 7f096effd700 10 client.37725.objecter _finish_command 1 = system:0 2024-04-14T15:22:50.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.346+0000 7f096effd700 20 client.37725.objecter put_session s=0x7f096407fd90 osd=5 4 2024-04-14T15:22:50.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.346+0000 7f096effd700 15 client.37725.objecter _session_command_op_remove 5 1 2024-04-14T15:22:50.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.365+0000 7f097fffd700 10 client.37725.objecter _submit_command 2 [{"prefix": "reset_purged_snaps_last"}] 2024-04-14T15:22:50.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.365+0000 7f097fffd700 15 client.37725.objecter _session_command_op_assign -1 2 2024-04-14T15:22:50.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.365+0000 7f097fffd700 20 client.37725.objecter _get_session s=0x7f096407fd90 osd=5 3 2024-04-14T15:22:50.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.365+0000 7f097fffd700 20 client.37725.objecter put_session s=0x7f096407fd90 osd=5 3 2024-04-14T15:22:50.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.365+0000 7f097fffd700 20 client.37725.objecter _get_session s=0x7f096407fd90 osd=5 3 2024-04-14T15:22:50.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.365+0000 7f097fffd700 15 client.37725.objecter _session_command_op_remove -1 2 2024-04-14T15:22:50.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.365+0000 7f097fffd700 20 client.37725.objecter get_session s=0x7f096407fd90 osd=5 3 2024-04-14T15:22:50.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.365+0000 7f097fffd700 15 client.37725.objecter _session_command_op_assign 5 2 2024-04-14T15:22:50.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.365+0000 7f097fffd700 20 client.37725.objecter put_session s=0x7f096407fd90 osd=5 4 2024-04-14T15:22:50.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.365+0000 7f097fffd700 10 client.37725.objecter _send_command 2 2024-04-14T15:22:50.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.365+0000 7f097fffd700 1 -- 172.21.15.90:0/1013056523 --> v1:172.21.15.92:6804/3244561440 -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) -- 0x7f095c0011f0 con 0x7f0964080440 2024-04-14T15:22:50.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.366+0000 7f096effd700 1 -- 172.21.15.90:0/1013056523 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 2: 0 ) ==== 8+0+0 (unknown 0 0 0) 0x7f095c0011f0 con 0x7f0964080440 2024-04-14T15:22:50.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.366+0000 7f096effd700 10 client.37725.objecter ms_dispatch 0x7f0978001550 command_reply(tid 2: 0 ) 2024-04-14T15:22:50.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.366+0000 7f096effd700 10 client.37725.objecter _finish_command 2 = system:0 2024-04-14T15:22:50.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.366+0000 7f096effd700 20 client.37725.objecter put_session s=0x7f096407fd90 osd=5 4 2024-04-14T15:22:50.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.366+0000 7f096effd700 15 client.37725.objecter _session_command_op_remove 5 2 2024-04-14T15:22:50.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.366+0000 7f097fffd700 10 librados: watch_flush enter 2024-04-14T15:22:50.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.366+0000 7f097fffd700 10 librados: watch_flush exit 2024-04-14T15:22:50.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.366+0000 7f097fffd700 10 client.37725.objecter close_session for osd.5 2024-04-14T15:22:50.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.366+0000 7f097fffd700 1 -- 172.21.15.90:0/1013056523 >> v1:172.21.15.92:6804/3244561440 conn(0x7f0964080440 legacy=0x7f09640828e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:50.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.366+0000 7f097fffd700 20 client.37725.objecter put_session s=0x7f096407fd90 osd=5 1 2024-04-14T15:22:50.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.366+0000 7f097fffd700 20 client.37725.objecter shutdown clearing up homeless session... 2024-04-14T15:22:50.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.366+0000 7f097fffd700 10 client.37725.objecter successfully canceled tick 2024-04-14T15:22:50.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.366+0000 7f097fffd700 1 -- 172.21.15.90:0/1013056523 >> v1:172.21.15.90:6800/420935427 conn(0x7f0964071480 legacy=0x7f0964073940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:50.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.366+0000 7f097fffd700 1 -- 172.21.15.90:0/1013056523 >> v1:172.21.15.90:6789/0 conn(0x7f097810f430 legacy=0x7f09781a6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:50.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.366+0000 7f097e59a700 1 -- 172.21.15.90:0/1013056523 reap_dead start 2024-04-14T15:22:50.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.366+0000 7f097fffd700 1 -- 172.21.15.90:0/1013056523 shutdown_connections 2024-04-14T15:22:50.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.366+0000 7f097fffd700 1 -- 172.21.15.90:0/1013056523 >> 172.21.15.90:0/1013056523 conn(0x7f09780fd2c0 msgr2=0x7f09780ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:22:50.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.366+0000 7f097fffd700 1 -- 172.21.15.90:0/1013056523 shutdown_connections 2024-04-14T15:22:50.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.367+0000 7f097fffd700 1 -- 172.21.15.90:0/1013056523 wait complete. 2024-04-14T15:22:50.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:50.367+0000 7f097fffd700 1 librados: shutdown 2024-04-14T15:22:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:49 smithi092 ceph-mon[33966]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:22:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:49 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-109-cache","var": "hit_set_period","val": "600"}]': finished 2024-04-14T15:22:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:49 smithi092 ceph-mon[33966]: from='client.47669 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-14T15:22:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:49 smithi092 ceph-mon[33966]: osdmap e633: 8 total, 8 up, 7 in 2024-04-14T15:22:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:49 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-14T15:22:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:49 smithi092 ceph-mon[33966]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:22:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:49 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-109-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-04-14T15:22:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:49 smithi092 ceph-mon[33966]: osdmap e634: 8 total, 8 up, 7 in 2024-04-14T15:22:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:49 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-04-14T15:22:50.423 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:22:50.450 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:50.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:50.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:22:50.552 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:22:50.580 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:50.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:50.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:22:50.681 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:22:50.705 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:50.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:50.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:22:50.807 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:22:50.834 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:50.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:50.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:22:50.936 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:22:50.963 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:50.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:50.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:22:51.065 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:22:51.091 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:51.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:51.092 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:22:51.193 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:22:51.220 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:51.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:51.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:22:51.270 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:50 smithi090 ceph-mon[41670]: pgmap v930: 226 pgs: 10 creating+activating, 3 active+remapped+backfill_toofull, 213 active+clean; 596 KiB data, 559 MiB used, 625 GiB / 626 GiB avail; 2/600 objects misplaced (0.333%) 2024-04-14T15:22:51.271 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:50 smithi090 ceph-mon[41670]: 1.d scrub starts 2024-04-14T15:22:51.271 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:50 smithi090 ceph-mon[41670]: 1.d scrub ok 2024-04-14T15:22:51.271 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:50 smithi090 ceph-mon[41670]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:22:51.271 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:50 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-109-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-04-14T15:22:51.271 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:50 smithi090 ceph-mon[41670]: osdmap e635: 8 total, 8 up, 7 in 2024-04-14T15:22:51.271 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:50 smithi090 ceph-mon[30399]: pgmap v930: 226 pgs: 10 creating+activating, 3 active+remapped+backfill_toofull, 213 active+clean; 596 KiB data, 559 MiB used, 625 GiB / 626 GiB avail; 2/600 objects misplaced (0.333%) 2024-04-14T15:22:51.271 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:50 smithi090 ceph-mon[30399]: 1.d scrub starts 2024-04-14T15:22:51.271 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:50 smithi090 ceph-mon[30399]: 1.d scrub ok 2024-04-14T15:22:51.271 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:50 smithi090 ceph-mon[30399]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:22:51.272 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:50 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-109-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-04-14T15:22:51.272 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:50 smithi090 ceph-mon[30399]: osdmap e635: 8 total, 8 up, 7 in 2024-04-14T15:22:51.322 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:22:51.349 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:51.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:51.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:22:51.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:50 smithi092 ceph-mon[33966]: pgmap v930: 226 pgs: 10 creating+activating, 3 active+remapped+backfill_toofull, 213 active+clean; 596 KiB data, 559 MiB used, 625 GiB / 626 GiB avail; 2/600 objects misplaced (0.333%) 2024-04-14T15:22:51.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:50 smithi092 ceph-mon[33966]: 1.d scrub starts 2024-04-14T15:22:51.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:50 smithi092 ceph-mon[33966]: 1.d scrub ok 2024-04-14T15:22:51.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:50 smithi092 ceph-mon[33966]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:22:51.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:50 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-109-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-04-14T15:22:51.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:50 smithi092 ceph-mon[33966]: osdmap e635: 8 total, 8 up, 7 in 2024-04-14T15:22:51.451 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:22:51.479 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:51.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:51.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:22:51.581 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:22:51.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:51.610 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:51.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:22:51.710 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:22:51.738 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:51.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:51.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:22:51.840 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:22:51.867 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:51.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:51.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:22:51.969 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:52.000 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:52.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:52.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:52.018 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-14T15:22:52.067 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:51 smithi090 ceph-mon[41670]: 1.e scrub starts 2024-04-14T15:22:52.067 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:51 smithi090 ceph-mon[41670]: 1.e scrub ok 2024-04-14T15:22:52.067 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:51 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-109"}]: dispatch 2024-04-14T15:22:52.067 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:51 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-109"}]': finished 2024-04-14T15:22:52.067 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:51 smithi090 ceph-mon[41670]: osdmap e636: 8 total, 8 up, 7 in 2024-04-14T15:22:52.067 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:51 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-109", "tierpool": "test-rados-api-smithi090-93652-109-cache"}]: dispatch 2024-04-14T15:22:52.067 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:51 smithi090 ceph-mon[30399]: 1.e scrub starts 2024-04-14T15:22:52.068 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:51 smithi090 ceph-mon[30399]: 1.e scrub ok 2024-04-14T15:22:52.068 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:51 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-109"}]: dispatch 2024-04-14T15:22:52.068 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:51 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-109"}]': finished 2024-04-14T15:22:52.068 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:51 smithi090 ceph-mon[30399]: osdmap e636: 8 total, 8 up, 7 in 2024-04-14T15:22:52.068 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:51 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-109", "tierpool": "test-rados-api-smithi090-93652-109-cache"}]: dispatch 2024-04-14T15:22:52.102 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:52.126 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:52.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:52.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:52.228 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:22:52.256 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:52.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:52.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:22:52.273 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:51 smithi092 ceph-mon[33966]: 1.e scrub starts 2024-04-14T15:22:52.273 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:51 smithi092 ceph-mon[33966]: 1.e scrub ok 2024-04-14T15:22:52.273 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:51 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-109"}]: dispatch 2024-04-14T15:22:52.273 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:51 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-109"}]': finished 2024-04-14T15:22:52.273 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:51 smithi092 ceph-mon[33966]: osdmap e636: 8 total, 8 up, 7 in 2024-04-14T15:22:52.273 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:51 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-109", "tierpool": "test-rados-api-smithi090-93652-109-cache"}]: dispatch 2024-04-14T15:22:52.359 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:22:52.387 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:52.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:52.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:22:52.488 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:52.512 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:52.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:52.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:52.614 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:22:52.637 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:52.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:52.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:22:52.739 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:22:52.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:52.763 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:52.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:22:52.836 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph tell osd.7 reset_purged_snaps_last 2024-04-14T15:22:52.864 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:22:52.902 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:52.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:52.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:22:53.004 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:22:53.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:53.028 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:53.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:22:53.129 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:53.153 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:53.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:53.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:53.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:52 smithi090 ceph-mon[41670]: pgmap v933: 226 pgs: 3 active+remapped+backfill_toofull, 223 active+clean; 596 KiB data, 577 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/600 objects misplaced (0.333%) 2024-04-14T15:22:53.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:52 smithi090 ceph-mon[41670]: 11.d scrub starts 2024-04-14T15:22:53.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:52 smithi090 ceph-mon[41670]: 11.d scrub ok 2024-04-14T15:22:53.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:52 smithi090 ceph-mon[41670]: 1.e scrub starts 2024-04-14T15:22:53.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:52 smithi090 ceph-mon[41670]: 1.e scrub ok 2024-04-14T15:22:53.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:52 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-109", "tierpool": "test-rados-api-smithi090-93652-109-cache"}]': finished 2024-04-14T15:22:53.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:52 smithi090 ceph-mon[41670]: osdmap e637: 8 total, 8 up, 7 in 2024-04-14T15:22:53.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:52 smithi090 ceph-mon[30399]: pgmap v933: 226 pgs: 3 active+remapped+backfill_toofull, 223 active+clean; 596 KiB data, 577 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/600 objects misplaced (0.333%) 2024-04-14T15:22:53.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:52 smithi090 ceph-mon[30399]: 11.d scrub starts 2024-04-14T15:22:53.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:52 smithi090 ceph-mon[30399]: 11.d scrub ok 2024-04-14T15:22:53.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:52 smithi090 ceph-mon[30399]: 1.e scrub starts 2024-04-14T15:22:53.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:52 smithi090 ceph-mon[30399]: 1.e scrub ok 2024-04-14T15:22:53.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:52 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-109", "tierpool": "test-rados-api-smithi090-93652-109-cache"}]': finished 2024-04-14T15:22:53.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:52 smithi090 ceph-mon[30399]: osdmap e637: 8 total, 8 up, 7 in 2024-04-14T15:22:53.255 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:22:53.282 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:53.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:53.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:22:53.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:52 smithi092 ceph-mon[33966]: pgmap v933: 226 pgs: 3 active+remapped+backfill_toofull, 223 active+clean; 596 KiB data, 577 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/600 objects misplaced (0.333%) 2024-04-14T15:22:53.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:52 smithi092 ceph-mon[33966]: 11.d scrub starts 2024-04-14T15:22:53.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:52 smithi092 ceph-mon[33966]: 11.d scrub ok 2024-04-14T15:22:53.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:52 smithi092 ceph-mon[33966]: 1.e scrub starts 2024-04-14T15:22:53.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:52 smithi092 ceph-mon[33966]: 1.e scrub ok 2024-04-14T15:22:53.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:52 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-109", "tierpool": "test-rados-api-smithi090-93652-109-cache"}]': finished 2024-04-14T15:22:53.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:52 smithi092 ceph-mon[33966]: osdmap e637: 8 total, 8 up, 7 in 2024-04-14T15:22:53.384 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:53.407 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:53.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:53.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:53.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.444+0000 7fd393061700 1 -- 172.21.15.90:0/2926071807 >> v1:172.21.15.90:6789/0 conn(0x7fd38c10a740 legacy=0x7fd38c10cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:53.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.444+0000 7fd393061700 1 -- 172.21.15.90:0/2926071807 shutdown_connections 2024-04-14T15:22:53.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.444+0000 7fd393061700 1 -- 172.21.15.90:0/2926071807 >> 172.21.15.90:0/2926071807 conn(0x7fd38c0fd290 msgr2=0x7fd38c0ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:22:53.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.444+0000 7fd393061700 1 -- 172.21.15.90:0/2926071807 shutdown_connections 2024-04-14T15:22:53.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.444+0000 7fd393061700 1 -- 172.21.15.90:0/2926071807 wait complete. 2024-04-14T15:22:53.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.444+0000 7fd393061700 1 librados: starting msgr at 2024-04-14T15:22:53.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.444+0000 7fd393061700 1 librados: starting objecter 2024-04-14T15:22:53.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.444+0000 7fd393061700 1 Processor -- start 2024-04-14T15:22:53.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.444+0000 7fd393061700 1 -- start start 2024-04-14T15:22:53.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.444+0000 7fd393061700 1 librados: setting wanted keys 2024-04-14T15:22:53.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.444+0000 7fd393061700 1 librados: calling monclient init 2024-04-14T15:22:53.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.444+0000 7fd393061700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd38c1a7f20 con 0x7fd38c107890 2024-04-14T15:22:53.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.444+0000 7fd393061700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd38c1a9100 con 0x7fd38c10e310 2024-04-14T15:22:53.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.444+0000 7fd393061700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd38c1aa2e0 con 0x7fd38c10a740 2024-04-14T15:22:53.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.444+0000 7fd390dfd700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fd38c107890 0x7fd38c06acf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:41718/0 (socket says 172.21.15.90:41718) 2024-04-14T15:22:53.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.444+0000 7fd390dfd700 1 -- 172.21.15.90:0/4161790804 learned_addr learned my addr 172.21.15.90:0/4161790804 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:22:53.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.445+0000 7fd389ffb700 10 client.?.objecter ms_handle_connect 0x7fd38c107890 2024-04-14T15:22:53.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.445+0000 7fd389ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:22:53.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.445+0000 7fd389ffb700 10 client.?.objecter ms_handle_connect 0x7fd38c10a740 2024-04-14T15:22:53.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.445+0000 7fd389ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:22:53.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.445+0000 7fd389ffb700 10 client.?.objecter ms_handle_connect 0x7fd38c10e310 2024-04-14T15:22:53.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.445+0000 7fd389ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:22:53.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.445+0000 7fd389ffb700 1 -- 172.21.15.90:0/4161790804 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1932950574 0 0) 0x7fd38c1a7f20 con 0x7fd38c107890 2024-04-14T15:22:53.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.445+0000 7fd389ffb700 1 -- 172.21.15.90:0/4161790804 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd37c0035b0 con 0x7fd38c107890 2024-04-14T15:22:53.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.445+0000 7fd389ffb700 1 -- 172.21.15.90:0/4161790804 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1126026017 0 0) 0x7fd38c1aa2e0 con 0x7fd38c10a740 2024-04-14T15:22:53.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.445+0000 7fd389ffb700 1 -- 172.21.15.90:0/4161790804 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd38c1a7f20 con 0x7fd38c10a740 2024-04-14T15:22:53.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.445+0000 7fd389ffb700 1 -- 172.21.15.90:0/4161790804 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4248887243 0 0) 0x7fd38c1a9100 con 0x7fd38c10e310 2024-04-14T15:22:53.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.445+0000 7fd389ffb700 1 -- 172.21.15.90:0/4161790804 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd38c1aa2e0 con 0x7fd38c10e310 2024-04-14T15:22:53.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.445+0000 7fd389ffb700 1 -- 172.21.15.90:0/4161790804 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2926257635 0 0) 0x7fd37c0035b0 con 0x7fd38c107890 2024-04-14T15:22:53.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.445+0000 7fd389ffb700 1 -- 172.21.15.90:0/4161790804 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd38c1a9100 con 0x7fd38c107890 2024-04-14T15:22:53.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.445+0000 7fd389ffb700 1 -- 172.21.15.90:0/4161790804 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fd380001910 con 0x7fd38c107890 2024-04-14T15:22:53.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.445+0000 7fd389ffb700 1 -- 172.21.15.90:0/4161790804 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2536248268 0 0) 0x7fd38c1a7f20 con 0x7fd38c10a740 2024-04-14T15:22:53.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.445+0000 7fd389ffb700 1 -- 172.21.15.90:0/4161790804 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd37c0035b0 con 0x7fd38c10a740 2024-04-14T15:22:53.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.445+0000 7fd389ffb700 1 -- 172.21.15.90:0/4161790804 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fd3780015d0 con 0x7fd38c10a740 2024-04-14T15:22:53.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.445+0000 7fd389ffb700 1 -- 172.21.15.90:0/4161790804 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2227356868 0 0) 0x7fd38c1aa2e0 con 0x7fd38c10e310 2024-04-14T15:22:53.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.445+0000 7fd389ffb700 1 -- 172.21.15.90:0/4161790804 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd38c1a7f20 con 0x7fd38c10e310 2024-04-14T15:22:53.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.445+0000 7fd389ffb700 1 -- 172.21.15.90:0/4161790804 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fd384001ae0 con 0x7fd38c10e310 2024-04-14T15:22:53.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.445+0000 7fd389ffb700 1 -- 172.21.15.90:0/4161790804 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2526479642 0 0) 0x7fd38c1a9100 con 0x7fd38c107890 2024-04-14T15:22:53.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.445+0000 7fd389ffb700 1 -- 172.21.15.90:0/4161790804 >> v1:172.21.15.90:6790/0 conn(0x7fd38c10a740 legacy=0x7fd38c06b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:53.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.445+0000 7fd389ffb700 1 -- 172.21.15.90:0/4161790804 >> v1:172.21.15.92:6789/0 conn(0x7fd38c10e310 legacy=0x7fd38c1a6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:53.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.445+0000 7fd389ffb700 1 -- 172.21.15.90:0/4161790804 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd38c1ab4c0 con 0x7fd38c107890 2024-04-14T15:22:53.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.445+0000 7fd393061700 1 -- 172.21.15.90:0/4161790804 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd38c1aa510 con 0x7fd38c107890 2024-04-14T15:22:53.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.445+0000 7fd389ffb700 1 -- 172.21.15.90:0/4161790804 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fd380003b40 con 0x7fd38c107890 2024-04-14T15:22:53.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.445+0000 7fd389ffb700 1 -- 172.21.15.90:0/4161790804 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fd380005cb0 con 0x7fd38c107890 2024-04-14T15:22:53.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.446+0000 7fd393061700 10 client.37731.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:22:53.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.446+0000 7fd393061700 1 -- 172.21.15.90:0/4161790804 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd38c1aa9d0 con 0x7fd38c107890 2024-04-14T15:22:53.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.446+0000 7fd393061700 1 librados: init done 2024-04-14T15:22:53.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.446+0000 7fd393061700 1 -- 172.21.15.90:0/4161790804 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fd38c0693d0 con 0x7fd38c107890 2024-04-14T15:22:53.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.447+0000 7fd389ffb700 1 -- 172.21.15.90:0/4161790804 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fd380005c90 con 0x7fd38c107890 2024-04-14T15:22:53.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.447+0000 7fd389ffb700 1 -- 172.21.15.90:0/4161790804 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(637..637 src has 467..637) ==== 9571+0+0 (unknown 3560858064 0 0) 0x7fd38008af80 con 0x7fd38c107890 2024-04-14T15:22:53.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.447+0000 7fd389ffb700 10 client.37731.objecter ms_dispatch 0x7fd38c001550 osd_map(637..637 src has 467..637) 2024-04-14T15:22:53.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.447+0000 7fd389ffb700 3 client.37731.objecter handle_osd_map got epochs [637,637] > 0 2024-04-14T15:22:53.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.447+0000 7fd389ffb700 3 client.37731.objecter handle_osd_map decoding full epoch 637 2024-04-14T15:22:53.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.448+0000 7fd389ffb700 20 client.37731.objecter dump_active .. 0 homeless 2024-04-14T15:22:53.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.448+0000 7fd389ffb700 1 -- 172.21.15.90:0/4161790804 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v637) ==== 54+0+16 (unknown 3725107037 0 2420000192) 0x7fd38008d3e0 con 0x7fd38c107890 2024-04-14T15:22:53.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.448+0000 7fd389ffb700 10 client.37731.objecter ms_handle_connect 0x7fd37c071440 2024-04-14T15:22:53.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.448+0000 7fd393061700 10 client.37731.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:22:53.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.448+0000 7fd393061700 15 client.37731.objecter _session_command_op_assign -1 1 2024-04-14T15:22:53.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.448+0000 7fd393061700 20 client.37731.objecter _get_session s=0x7fd38c1122e0 osd=0 3 2024-04-14T15:22:53.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.448+0000 7fd393061700 20 client.37731.objecter put_session s=0x7fd38c1122e0 osd=0 3 2024-04-14T15:22:53.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.448+0000 7fd393061700 20 client.37731.objecter _get_session s=0x7fd38c1122e0 osd=0 3 2024-04-14T15:22:53.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.448+0000 7fd393061700 15 client.37731.objecter _session_command_op_remove -1 1 2024-04-14T15:22:53.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.448+0000 7fd393061700 20 client.37731.objecter get_session s=0x7fd38c1122e0 osd=0 3 2024-04-14T15:22:53.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.448+0000 7fd393061700 15 client.37731.objecter _session_command_op_assign 0 1 2024-04-14T15:22:53.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.448+0000 7fd393061700 20 client.37731.objecter put_session s=0x7fd38c1122e0 osd=0 4 2024-04-14T15:22:53.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.448+0000 7fd393061700 10 client.37731.objecter _send_command 1 2024-04-14T15:22:53.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.448+0000 7fd393061700 1 -- 172.21.15.90:0/4161790804 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd38c10d4d0 con 0x7fd38c1a97e0 2024-04-14T15:22:53.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.448+0000 7fd389ffb700 10 client.37731.objecter ms_handle_connect 0x7fd38c1a97e0 2024-04-14T15:22:53.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.449+0000 7fd389ffb700 1 -- 172.21.15.90:0/4161790804 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd38c10d4d0 con 0x7fd38c1a97e0 2024-04-14T15:22:53.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.449+0000 7fd389ffb700 10 client.37731.objecter ms_dispatch 0x7fd38c001550 command_reply(tid 1: 0 ) 2024-04-14T15:22:53.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.449+0000 7fd389ffb700 10 client.37731.objecter _finish_command 1 = system:0 2024-04-14T15:22:53.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.449+0000 7fd389ffb700 20 client.37731.objecter put_session s=0x7fd38c1122e0 osd=0 4 2024-04-14T15:22:53.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.449+0000 7fd389ffb700 15 client.37731.objecter _session_command_op_remove 0 1 2024-04-14T15:22:53.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.466+0000 7fd393061700 10 client.37731.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:22:53.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.466+0000 7fd393061700 15 client.37731.objecter _session_command_op_assign -1 2 2024-04-14T15:22:53.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.466+0000 7fd393061700 20 client.37731.objecter _get_session s=0x7fd38c1122e0 osd=0 3 2024-04-14T15:22:53.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.466+0000 7fd393061700 20 client.37731.objecter put_session s=0x7fd38c1122e0 osd=0 3 2024-04-14T15:22:53.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.466+0000 7fd393061700 20 client.37731.objecter _get_session s=0x7fd38c1122e0 osd=0 3 2024-04-14T15:22:53.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.466+0000 7fd393061700 15 client.37731.objecter _session_command_op_remove -1 2 2024-04-14T15:22:53.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.466+0000 7fd393061700 20 client.37731.objecter get_session s=0x7fd38c1122e0 osd=0 3 2024-04-14T15:22:53.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.466+0000 7fd393061700 15 client.37731.objecter _session_command_op_assign 0 2 2024-04-14T15:22:53.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.466+0000 7fd393061700 20 client.37731.objecter put_session s=0x7fd38c1122e0 osd=0 4 2024-04-14T15:22:53.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.466+0000 7fd393061700 10 client.37731.objecter _send_command 2 2024-04-14T15:22:53.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.466+0000 7fd393061700 1 -- 172.21.15.90:0/4161790804 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd38c02ce20 con 0x7fd38c1a97e0 2024-04-14T15:22:53.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.468+0000 7fd389ffb700 1 -- 172.21.15.90:0/4161790804 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd38c02ce20 con 0x7fd38c1a97e0 2024-04-14T15:22:53.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.468+0000 7fd389ffb700 10 client.37731.objecter ms_dispatch 0x7fd38c001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:22:53.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.468+0000 7fd389ffb700 10 client.37731.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:22:53.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.468+0000 7fd389ffb700 20 client.37731.objecter put_session s=0x7fd38c1122e0 osd=0 4 2024-04-14T15:22:53.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.468+0000 7fd389ffb700 15 client.37731.objecter _session_command_op_remove 0 2 2024-04-14T15:22:53.469 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:22:53.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.468+0000 7fd393061700 10 client.37731.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:22:53.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.468+0000 7fd393061700 15 client.37731.objecter _session_command_op_assign -1 3 2024-04-14T15:22:53.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.468+0000 7fd393061700 20 client.37731.objecter _get_session s=0x7fd38c02e150 osd=1 3 2024-04-14T15:22:53.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.468+0000 7fd393061700 20 client.37731.objecter put_session s=0x7fd38c02e150 osd=1 3 2024-04-14T15:22:53.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.468+0000 7fd393061700 20 client.37731.objecter _get_session s=0x7fd38c02e150 osd=1 3 2024-04-14T15:22:53.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.468+0000 7fd393061700 15 client.37731.objecter _session_command_op_remove -1 3 2024-04-14T15:22:53.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.468+0000 7fd393061700 20 client.37731.objecter get_session s=0x7fd38c02e150 osd=1 3 2024-04-14T15:22:53.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.468+0000 7fd393061700 15 client.37731.objecter _session_command_op_assign 1 3 2024-04-14T15:22:53.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.468+0000 7fd393061700 20 client.37731.objecter put_session s=0x7fd38c02e150 osd=1 4 2024-04-14T15:22:53.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.468+0000 7fd393061700 10 client.37731.objecter _send_command 3 2024-04-14T15:22:53.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.468+0000 7fd393061700 1 -- 172.21.15.90:0/4161790804 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fd38c10cf00 con 0x7fd38c0636c0 2024-04-14T15:22:53.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.469+0000 7fd389ffb700 10 client.37731.objecter ms_handle_connect 0x7fd38c0636c0 2024-04-14T15:22:53.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.469+0000 7fd389ffb700 1 -- 172.21.15.90:0/4161790804 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd38c10cf00 con 0x7fd38c0636c0 2024-04-14T15:22:53.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.469+0000 7fd389ffb700 10 client.37731.objecter ms_dispatch 0x7fd38c001550 command_reply(tid 3: 0 ) 2024-04-14T15:22:53.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.469+0000 7fd389ffb700 10 client.37731.objecter _finish_command 3 = system:0 2024-04-14T15:22:53.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.469+0000 7fd389ffb700 20 client.37731.objecter put_session s=0x7fd38c02e150 osd=1 4 2024-04-14T15:22:53.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.469+0000 7fd389ffb700 15 client.37731.objecter _session_command_op_remove 1 3 2024-04-14T15:22:53.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.488+0000 7fd393061700 10 client.37731.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:22:53.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.488+0000 7fd393061700 15 client.37731.objecter _session_command_op_assign -1 4 2024-04-14T15:22:53.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.488+0000 7fd393061700 20 client.37731.objecter _get_session s=0x7fd38c02e150 osd=1 3 2024-04-14T15:22:53.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.488+0000 7fd393061700 20 client.37731.objecter put_session s=0x7fd38c02e150 osd=1 3 2024-04-14T15:22:53.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.488+0000 7fd393061700 20 client.37731.objecter _get_session s=0x7fd38c02e150 osd=1 3 2024-04-14T15:22:53.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.488+0000 7fd393061700 15 client.37731.objecter _session_command_op_remove -1 4 2024-04-14T15:22:53.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.488+0000 7fd393061700 20 client.37731.objecter get_session s=0x7fd38c02e150 osd=1 3 2024-04-14T15:22:53.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.488+0000 7fd393061700 15 client.37731.objecter _session_command_op_assign 1 4 2024-04-14T15:22:53.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.488+0000 7fd393061700 20 client.37731.objecter put_session s=0x7fd38c02e150 osd=1 4 2024-04-14T15:22:53.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.488+0000 7fd393061700 10 client.37731.objecter _send_command 4 2024-04-14T15:22:53.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.488+0000 7fd393061700 1 -- 172.21.15.90:0/4161790804 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd38c109e30 con 0x7fd38c0636c0 2024-04-14T15:22:53.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.490+0000 7fd389ffb700 1 -- 172.21.15.90:0/4161790804 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd38c109e30 con 0x7fd38c0636c0 2024-04-14T15:22:53.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.490+0000 7fd389ffb700 10 client.37731.objecter ms_dispatch 0x7fd38c001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:22:53.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.490+0000 7fd389ffb700 10 client.37731.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:22:53.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.490+0000 7fd389ffb700 20 client.37731.objecter put_session s=0x7fd38c02e150 osd=1 4 2024-04-14T15:22:53.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.490+0000 7fd389ffb700 15 client.37731.objecter _session_command_op_remove 1 4 2024-04-14T15:22:53.491 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:22:53.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.490+0000 7fd393061700 10 client.37731.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:22:53.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.490+0000 7fd393061700 15 client.37731.objecter _session_command_op_assign -1 5 2024-04-14T15:22:53.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.490+0000 7fd393061700 20 client.37731.objecter _get_session s=0x7fd38c063d90 osd=2 3 2024-04-14T15:22:53.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.490+0000 7fd393061700 20 client.37731.objecter put_session s=0x7fd38c063d90 osd=2 3 2024-04-14T15:22:53.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.490+0000 7fd393061700 20 client.37731.objecter _get_session s=0x7fd38c063d90 osd=2 3 2024-04-14T15:22:53.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.490+0000 7fd393061700 15 client.37731.objecter _session_command_op_remove -1 5 2024-04-14T15:22:53.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.490+0000 7fd393061700 20 client.37731.objecter get_session s=0x7fd38c063d90 osd=2 3 2024-04-14T15:22:53.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.490+0000 7fd393061700 15 client.37731.objecter _session_command_op_assign 2 5 2024-04-14T15:22:53.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.490+0000 7fd393061700 20 client.37731.objecter put_session s=0x7fd38c063d90 osd=2 4 2024-04-14T15:22:53.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.490+0000 7fd393061700 10 client.37731.objecter _send_command 5 2024-04-14T15:22:53.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.490+0000 7fd393061700 1 -- 172.21.15.90:0/4161790804 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fd38c1b5b90 con 0x7fd38c1b2080 2024-04-14T15:22:53.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.491+0000 7fd389ffb700 10 client.37731.objecter ms_handle_connect 0x7fd38c1b2080 2024-04-14T15:22:53.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.492+0000 7fd389ffb700 1 -- 172.21.15.90:0/4161790804 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd38c1b5b90 con 0x7fd38c1b2080 2024-04-14T15:22:53.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.492+0000 7fd389ffb700 10 client.37731.objecter ms_dispatch 0x7fd38c001550 command_reply(tid 5: 0 ) 2024-04-14T15:22:53.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.492+0000 7fd389ffb700 10 client.37731.objecter _finish_command 5 = system:0 2024-04-14T15:22:53.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.492+0000 7fd389ffb700 20 client.37731.objecter put_session s=0x7fd38c063d90 osd=2 4 2024-04-14T15:22:53.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.492+0000 7fd389ffb700 15 client.37731.objecter _session_command_op_remove 2 5 2024-04-14T15:22:53.509 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:22:53.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.510+0000 7fd393061700 10 client.37731.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:22:53.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.510+0000 7fd393061700 15 client.37731.objecter _session_command_op_assign -1 6 2024-04-14T15:22:53.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.510+0000 7fd393061700 20 client.37731.objecter _get_session s=0x7fd38c063d90 osd=2 3 2024-04-14T15:22:53.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.510+0000 7fd393061700 20 client.37731.objecter put_session s=0x7fd38c063d90 osd=2 3 2024-04-14T15:22:53.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.510+0000 7fd393061700 20 client.37731.objecter _get_session s=0x7fd38c063d90 osd=2 3 2024-04-14T15:22:53.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.510+0000 7fd393061700 15 client.37731.objecter _session_command_op_remove -1 6 2024-04-14T15:22:53.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.510+0000 7fd393061700 20 client.37731.objecter get_session s=0x7fd38c063d90 osd=2 3 2024-04-14T15:22:53.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.510+0000 7fd393061700 15 client.37731.objecter _session_command_op_assign 2 6 2024-04-14T15:22:53.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.510+0000 7fd393061700 20 client.37731.objecter put_session s=0x7fd38c063d90 osd=2 4 2024-04-14T15:22:53.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.510+0000 7fd393061700 10 client.37731.objecter _send_command 6 2024-04-14T15:22:53.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.510+0000 7fd393061700 1 -- 172.21.15.90:0/4161790804 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd38c063140 con 0x7fd38c1b2080 2024-04-14T15:22:53.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.512+0000 7fd389ffb700 1 -- 172.21.15.90:0/4161790804 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd38c063140 con 0x7fd38c1b2080 2024-04-14T15:22:53.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.512+0000 7fd389ffb700 10 client.37731.objecter ms_dispatch 0x7fd38c001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:22:53.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.512+0000 7fd389ffb700 10 client.37731.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:22:53.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.512+0000 7fd389ffb700 20 client.37731.objecter put_session s=0x7fd38c063d90 osd=2 4 2024-04-14T15:22:53.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.512+0000 7fd389ffb700 15 client.37731.objecter _session_command_op_remove 2 6 2024-04-14T15:22:53.513 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:22:53.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.512+0000 7fd393061700 10 client.37731.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:22:53.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.512+0000 7fd393061700 15 client.37731.objecter _session_command_op_assign -1 7 2024-04-14T15:22:53.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.512+0000 7fd393061700 20 client.37731.objecter _get_session s=0x7fd38c063540 osd=3 3 2024-04-14T15:22:53.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.512+0000 7fd393061700 20 client.37731.objecter put_session s=0x7fd38c063540 osd=3 3 2024-04-14T15:22:53.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.512+0000 7fd393061700 20 client.37731.objecter _get_session s=0x7fd38c063540 osd=3 3 2024-04-14T15:22:53.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.512+0000 7fd393061700 15 client.37731.objecter _session_command_op_remove -1 7 2024-04-14T15:22:53.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.512+0000 7fd393061700 20 client.37731.objecter get_session s=0x7fd38c063540 osd=3 3 2024-04-14T15:22:53.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.512+0000 7fd393061700 15 client.37731.objecter _session_command_op_assign 3 7 2024-04-14T15:22:53.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.512+0000 7fd393061700 20 client.37731.objecter put_session s=0x7fd38c063540 osd=3 4 2024-04-14T15:22:53.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.512+0000 7fd393061700 10 client.37731.objecter _send_command 7 2024-04-14T15:22:53.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.512+0000 7fd393061700 1 -- 172.21.15.90:0/4161790804 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fd38c1ba690 con 0x7fd38c02dcb0 2024-04-14T15:22:53.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.513+0000 7fd389ffb700 10 client.37731.objecter ms_handle_connect 0x7fd38c02dcb0 2024-04-14T15:22:53.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.513+0000 7fd389ffb700 1 -- 172.21.15.90:0/4161790804 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd38c1ba690 con 0x7fd38c02dcb0 2024-04-14T15:22:53.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.513+0000 7fd389ffb700 10 client.37731.objecter ms_dispatch 0x7fd38c001550 command_reply(tid 7: 0 ) 2024-04-14T15:22:53.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.513+0000 7fd389ffb700 10 client.37731.objecter _finish_command 7 = system:0 2024-04-14T15:22:53.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.513+0000 7fd389ffb700 20 client.37731.objecter put_session s=0x7fd38c063540 osd=3 4 2024-04-14T15:22:53.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.513+0000 7fd389ffb700 15 client.37731.objecter _session_command_op_remove 3 7 2024-04-14T15:22:53.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.530+0000 7fd393061700 10 client.37731.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:22:53.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.530+0000 7fd393061700 15 client.37731.objecter _session_command_op_assign -1 8 2024-04-14T15:22:53.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.530+0000 7fd393061700 20 client.37731.objecter _get_session s=0x7fd38c063540 osd=3 3 2024-04-14T15:22:53.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.530+0000 7fd393061700 20 client.37731.objecter put_session s=0x7fd38c063540 osd=3 3 2024-04-14T15:22:53.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.530+0000 7fd393061700 20 client.37731.objecter _get_session s=0x7fd38c063540 osd=3 3 2024-04-14T15:22:53.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.530+0000 7fd393061700 15 client.37731.objecter _session_command_op_remove -1 8 2024-04-14T15:22:53.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.530+0000 7fd393061700 20 client.37731.objecter get_session s=0x7fd38c063540 osd=3 3 2024-04-14T15:22:53.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.530+0000 7fd393061700 15 client.37731.objecter _session_command_op_assign 3 8 2024-04-14T15:22:53.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.530+0000 7fd393061700 20 client.37731.objecter put_session s=0x7fd38c063540 osd=3 4 2024-04-14T15:22:53.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.530+0000 7fd393061700 10 client.37731.objecter _send_command 8 2024-04-14T15:22:53.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.530+0000 7fd393061700 1 -- 172.21.15.90:0/4161790804 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd38c02d2e0 con 0x7fd38c02dcb0 2024-04-14T15:22:53.532 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:53.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.532+0000 7fd389ffb700 1 -- 172.21.15.90:0/4161790804 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd38c02d2e0 con 0x7fd38c02dcb0 2024-04-14T15:22:53.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.532+0000 7fd389ffb700 10 client.37731.objecter ms_dispatch 0x7fd38c001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:22:53.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.532+0000 7fd389ffb700 10 client.37731.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:22:53.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.532+0000 7fd389ffb700 20 client.37731.objecter put_session s=0x7fd38c063540 osd=3 4 2024-04-14T15:22:53.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.532+0000 7fd389ffb700 15 client.37731.objecter _session_command_op_remove 3 8 2024-04-14T15:22:53.533 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:22:53.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.532+0000 7fd393061700 10 client.37731.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:22:53.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.532+0000 7fd393061700 15 client.37731.objecter _session_command_op_assign -1 9 2024-04-14T15:22:53.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.532+0000 7fd393061700 20 client.37731.objecter _get_session s=0x7fd38c02d320 osd=4 3 2024-04-14T15:22:53.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.532+0000 7fd393061700 20 client.37731.objecter put_session s=0x7fd38c02d320 osd=4 3 2024-04-14T15:22:53.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.532+0000 7fd393061700 20 client.37731.objecter _get_session s=0x7fd38c02d320 osd=4 3 2024-04-14T15:22:53.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.532+0000 7fd393061700 15 client.37731.objecter _session_command_op_remove -1 9 2024-04-14T15:22:53.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.532+0000 7fd393061700 20 client.37731.objecter get_session s=0x7fd38c02d320 osd=4 3 2024-04-14T15:22:53.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.532+0000 7fd393061700 15 client.37731.objecter _session_command_op_assign 4 9 2024-04-14T15:22:53.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.532+0000 7fd393061700 20 client.37731.objecter put_session s=0x7fd38c02d320 osd=4 4 2024-04-14T15:22:53.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.532+0000 7fd393061700 10 client.37731.objecter _send_command 9 2024-04-14T15:22:53.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.532+0000 7fd393061700 1 -- 172.21.15.90:0/4161790804 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fd38c1bf220 con 0x7fd38c1b6a20 2024-04-14T15:22:53.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:53.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:22:53.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.533+0000 7fd389ffb700 10 client.37731.objecter ms_handle_connect 0x7fd38c1b6a20 2024-04-14T15:22:53.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.534+0000 7fd389ffb700 1 -- 172.21.15.90:0/4161790804 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd38c1bf220 con 0x7fd38c1b6a20 2024-04-14T15:22:53.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.534+0000 7fd389ffb700 10 client.37731.objecter ms_dispatch 0x7fd38c001550 command_reply(tid 9: 0 ) 2024-04-14T15:22:53.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.534+0000 7fd389ffb700 10 client.37731.objecter _finish_command 9 = system:0 2024-04-14T15:22:53.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.534+0000 7fd389ffb700 20 client.37731.objecter put_session s=0x7fd38c02d320 osd=4 4 2024-04-14T15:22:53.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.534+0000 7fd389ffb700 15 client.37731.objecter _session_command_op_remove 4 9 2024-04-14T15:22:53.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.550+0000 7fd393061700 10 client.37731.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:22:53.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.550+0000 7fd393061700 15 client.37731.objecter _session_command_op_assign -1 10 2024-04-14T15:22:53.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.550+0000 7fd393061700 20 client.37731.objecter _get_session s=0x7fd38c02d320 osd=4 3 2024-04-14T15:22:53.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.550+0000 7fd393061700 20 client.37731.objecter put_session s=0x7fd38c02d320 osd=4 3 2024-04-14T15:22:53.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.550+0000 7fd393061700 20 client.37731.objecter _get_session s=0x7fd38c02d320 osd=4 3 2024-04-14T15:22:53.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.550+0000 7fd393061700 15 client.37731.objecter _session_command_op_remove -1 10 2024-04-14T15:22:53.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.550+0000 7fd393061700 20 client.37731.objecter get_session s=0x7fd38c02d320 osd=4 3 2024-04-14T15:22:53.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.550+0000 7fd393061700 15 client.37731.objecter _session_command_op_assign 4 10 2024-04-14T15:22:53.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.550+0000 7fd393061700 20 client.37731.objecter put_session s=0x7fd38c02d320 osd=4 4 2024-04-14T15:22:53.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.550+0000 7fd393061700 10 client.37731.objecter _send_command 10 2024-04-14T15:22:53.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.550+0000 7fd393061700 1 -- 172.21.15.90:0/4161790804 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd38c1bf220 con 0x7fd38c1b6a20 2024-04-14T15:22:53.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.552+0000 7fd389ffb700 1 -- 172.21.15.90:0/4161790804 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd38c1bf220 con 0x7fd38c1b6a20 2024-04-14T15:22:53.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.552+0000 7fd389ffb700 10 client.37731.objecter ms_dispatch 0x7fd38c001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:22:53.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.552+0000 7fd389ffb700 10 client.37731.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:22:53.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.552+0000 7fd389ffb700 20 client.37731.objecter put_session s=0x7fd38c02d320 osd=4 4 2024-04-14T15:22:53.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.552+0000 7fd389ffb700 15 client.37731.objecter _session_command_op_remove 4 10 2024-04-14T15:22:53.554 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:22:53.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.553+0000 7fd393061700 10 client.37731.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:22:53.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.553+0000 7fd393061700 15 client.37731.objecter _session_command_op_assign -1 11 2024-04-14T15:22:53.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.553+0000 7fd393061700 20 client.37731.objecter _get_session s=0x7fd38c1bf220 osd=5 3 2024-04-14T15:22:53.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.553+0000 7fd393061700 20 client.37731.objecter put_session s=0x7fd38c1bf220 osd=5 3 2024-04-14T15:22:53.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.553+0000 7fd393061700 20 client.37731.objecter _get_session s=0x7fd38c1bf220 osd=5 3 2024-04-14T15:22:53.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.553+0000 7fd393061700 15 client.37731.objecter _session_command_op_remove -1 11 2024-04-14T15:22:53.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.553+0000 7fd393061700 20 client.37731.objecter get_session s=0x7fd38c1bf220 osd=5 3 2024-04-14T15:22:53.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.553+0000 7fd393061700 15 client.37731.objecter _session_command_op_assign 5 11 2024-04-14T15:22:53.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.553+0000 7fd393061700 20 client.37731.objecter put_session s=0x7fd38c1bf220 osd=5 4 2024-04-14T15:22:53.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.553+0000 7fd393061700 10 client.37731.objecter _send_command 11 2024-04-14T15:22:53.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.553+0000 7fd393061700 1 -- 172.21.15.90:0/4161790804 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fd38c1c35e0 con 0x7fd38c1bf8a0 2024-04-14T15:22:53.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.553+0000 7fd389ffb700 10 client.37731.objecter ms_handle_connect 0x7fd38c1bf8a0 2024-04-14T15:22:53.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.554+0000 7fd389ffb700 1 -- 172.21.15.90:0/4161790804 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd38c1c35e0 con 0x7fd38c1bf8a0 2024-04-14T15:22:53.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.554+0000 7fd389ffb700 10 client.37731.objecter ms_dispatch 0x7fd38c001550 command_reply(tid 11: 0 ) 2024-04-14T15:22:53.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.554+0000 7fd389ffb700 10 client.37731.objecter _finish_command 11 = system:0 2024-04-14T15:22:53.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.554+0000 7fd389ffb700 20 client.37731.objecter put_session s=0x7fd38c1bf220 osd=5 4 2024-04-14T15:22:53.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.554+0000 7fd389ffb700 15 client.37731.objecter _session_command_op_remove 5 11 2024-04-14T15:22:53.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.570+0000 7fd393061700 10 client.37731.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:22:53.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.570+0000 7fd393061700 15 client.37731.objecter _session_command_op_assign -1 12 2024-04-14T15:22:53.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.570+0000 7fd393061700 20 client.37731.objecter _get_session s=0x7fd38c1bf220 osd=5 3 2024-04-14T15:22:53.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.570+0000 7fd393061700 20 client.37731.objecter put_session s=0x7fd38c1bf220 osd=5 3 2024-04-14T15:22:53.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.570+0000 7fd393061700 20 client.37731.objecter _get_session s=0x7fd38c1bf220 osd=5 3 2024-04-14T15:22:53.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.570+0000 7fd393061700 15 client.37731.objecter _session_command_op_remove -1 12 2024-04-14T15:22:53.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.570+0000 7fd393061700 20 client.37731.objecter get_session s=0x7fd38c1bf220 osd=5 3 2024-04-14T15:22:53.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.570+0000 7fd393061700 15 client.37731.objecter _session_command_op_assign 5 12 2024-04-14T15:22:53.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.570+0000 7fd393061700 20 client.37731.objecter put_session s=0x7fd38c1bf220 osd=5 4 2024-04-14T15:22:53.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.570+0000 7fd393061700 10 client.37731.objecter _send_command 12 2024-04-14T15:22:53.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.570+0000 7fd393061700 1 -- 172.21.15.90:0/4161790804 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd38c1c37c0 con 0x7fd38c1bf8a0 2024-04-14T15:22:53.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.573+0000 7fd389ffb700 1 -- 172.21.15.90:0/4161790804 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd38c1c37c0 con 0x7fd38c1bf8a0 2024-04-14T15:22:53.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.573+0000 7fd389ffb700 10 client.37731.objecter ms_dispatch 0x7fd38c001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:22:53.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.573+0000 7fd389ffb700 10 client.37731.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:22:53.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.573+0000 7fd389ffb700 20 client.37731.objecter put_session s=0x7fd38c1bf220 osd=5 4 2024-04-14T15:22:53.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.573+0000 7fd389ffb700 15 client.37731.objecter _session_command_op_remove 5 12 2024-04-14T15:22:53.573 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:22:53.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.573+0000 7fd393061700 10 client.37731.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:22:53.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.573+0000 7fd393061700 15 client.37731.objecter _session_command_op_assign -1 13 2024-04-14T15:22:53.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.573+0000 7fd393061700 20 client.37731.objecter _get_session s=0x7fd38c1c3600 osd=6 3 2024-04-14T15:22:53.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.573+0000 7fd393061700 20 client.37731.objecter put_session s=0x7fd38c1c3600 osd=6 3 2024-04-14T15:22:53.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.573+0000 7fd393061700 20 client.37731.objecter _get_session s=0x7fd38c1c3600 osd=6 3 2024-04-14T15:22:53.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.573+0000 7fd393061700 15 client.37731.objecter _session_command_op_remove -1 13 2024-04-14T15:22:53.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.573+0000 7fd393061700 20 client.37731.objecter get_session s=0x7fd38c1c3600 osd=6 3 2024-04-14T15:22:53.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.573+0000 7fd393061700 15 client.37731.objecter _session_command_op_assign 6 13 2024-04-14T15:22:53.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.573+0000 7fd393061700 20 client.37731.objecter put_session s=0x7fd38c1c3600 osd=6 4 2024-04-14T15:22:53.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.573+0000 7fd393061700 10 client.37731.objecter _send_command 13 2024-04-14T15:22:53.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.573+0000 7fd393061700 1 -- 172.21.15.90:0/4161790804 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fd38c1c4a80 con 0x7fd38c1c3ee0 2024-04-14T15:22:53.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.573+0000 7fd389ffb700 10 client.37731.objecter ms_handle_connect 0x7fd38c1c3ee0 2024-04-14T15:22:53.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.574+0000 7fd389ffb700 1 -- 172.21.15.90:0/4161790804 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd38c1c4a80 con 0x7fd38c1c3ee0 2024-04-14T15:22:53.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.574+0000 7fd389ffb700 10 client.37731.objecter ms_dispatch 0x7fd38c001550 command_reply(tid 13: 0 ) 2024-04-14T15:22:53.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.574+0000 7fd389ffb700 10 client.37731.objecter _finish_command 13 = system:0 2024-04-14T15:22:53.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.574+0000 7fd389ffb700 20 client.37731.objecter put_session s=0x7fd38c1c3600 osd=6 4 2024-04-14T15:22:53.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.574+0000 7fd389ffb700 15 client.37731.objecter _session_command_op_remove 6 13 2024-04-14T15:22:53.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.590+0000 7fd393061700 10 client.37731.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:22:53.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.590+0000 7fd393061700 15 client.37731.objecter _session_command_op_assign -1 14 2024-04-14T15:22:53.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.590+0000 7fd393061700 20 client.37731.objecter _get_session s=0x7fd38c1c3600 osd=6 3 2024-04-14T15:22:53.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.590+0000 7fd393061700 20 client.37731.objecter put_session s=0x7fd38c1c3600 osd=6 3 2024-04-14T15:22:53.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.590+0000 7fd393061700 20 client.37731.objecter _get_session s=0x7fd38c1c3600 osd=6 3 2024-04-14T15:22:53.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.590+0000 7fd393061700 15 client.37731.objecter _session_command_op_remove -1 14 2024-04-14T15:22:53.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.590+0000 7fd393061700 20 client.37731.objecter get_session s=0x7fd38c1c3600 osd=6 3 2024-04-14T15:22:53.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.590+0000 7fd393061700 15 client.37731.objecter _session_command_op_assign 6 14 2024-04-14T15:22:53.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.590+0000 7fd393061700 20 client.37731.objecter put_session s=0x7fd38c1c3600 osd=6 4 2024-04-14T15:22:53.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.590+0000 7fd393061700 10 client.37731.objecter _send_command 14 2024-04-14T15:22:53.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.590+0000 7fd393061700 1 -- 172.21.15.90:0/4161790804 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd38c1c4a80 con 0x7fd38c1c3ee0 2024-04-14T15:22:53.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.592+0000 7fd389ffb700 1 -- 172.21.15.90:0/4161790804 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd38c1c4a80 con 0x7fd38c1c3ee0 2024-04-14T15:22:53.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.592+0000 7fd389ffb700 10 client.37731.objecter ms_dispatch 0x7fd38c001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:22:53.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.592+0000 7fd389ffb700 10 client.37731.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:22:53.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.592+0000 7fd389ffb700 20 client.37731.objecter put_session s=0x7fd38c1c3600 osd=6 4 2024-04-14T15:22:53.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.592+0000 7fd389ffb700 15 client.37731.objecter _session_command_op_remove 6 14 2024-04-14T15:22:53.593 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:22:53.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.593+0000 7fd393061700 10 client.37731.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:22:53.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.593+0000 7fd393061700 15 client.37731.objecter _session_command_op_assign -1 15 2024-04-14T15:22:53.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.593+0000 7fd393061700 20 client.37731.objecter _get_session s=0x7fd38c1c4a80 osd=7 3 2024-04-14T15:22:53.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.593+0000 7fd393061700 20 client.37731.objecter put_session s=0x7fd38c1c4a80 osd=7 3 2024-04-14T15:22:53.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.593+0000 7fd393061700 20 client.37731.objecter _get_session s=0x7fd38c1c4a80 osd=7 3 2024-04-14T15:22:53.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.593+0000 7fd393061700 15 client.37731.objecter _session_command_op_remove -1 15 2024-04-14T15:22:53.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.593+0000 7fd393061700 20 client.37731.objecter get_session s=0x7fd38c1c4a80 osd=7 3 2024-04-14T15:22:53.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.593+0000 7fd393061700 15 client.37731.objecter _session_command_op_assign 7 15 2024-04-14T15:22:53.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.593+0000 7fd393061700 20 client.37731.objecter put_session s=0x7fd38c1c4a80 osd=7 4 2024-04-14T15:22:53.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.593+0000 7fd393061700 10 client.37731.objecter _send_command 15 2024-04-14T15:22:53.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.593+0000 7fd393061700 1 -- 172.21.15.90:0/4161790804 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fd38c1cbfd0 con 0x7fd38c1c84c0 2024-04-14T15:22:53.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.593+0000 7fd389ffb700 10 client.37731.objecter ms_handle_connect 0x7fd38c1c84c0 2024-04-14T15:22:53.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.594+0000 7fd389ffb700 1 -- 172.21.15.90:0/4161790804 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd38c1cbfd0 con 0x7fd38c1c84c0 2024-04-14T15:22:53.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.594+0000 7fd389ffb700 10 client.37731.objecter ms_dispatch 0x7fd38c001550 command_reply(tid 15: 0 ) 2024-04-14T15:22:53.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.594+0000 7fd389ffb700 10 client.37731.objecter _finish_command 15 = system:0 2024-04-14T15:22:53.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.594+0000 7fd389ffb700 20 client.37731.objecter put_session s=0x7fd38c1c4a80 osd=7 4 2024-04-14T15:22:53.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.594+0000 7fd389ffb700 15 client.37731.objecter _session_command_op_remove 7 15 2024-04-14T15:22:53.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.611+0000 7fd393061700 10 client.37731.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:22:53.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.611+0000 7fd393061700 15 client.37731.objecter _session_command_op_assign -1 16 2024-04-14T15:22:53.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.611+0000 7fd393061700 20 client.37731.objecter _get_session s=0x7fd38c1c4a80 osd=7 3 2024-04-14T15:22:53.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.611+0000 7fd393061700 20 client.37731.objecter put_session s=0x7fd38c1c4a80 osd=7 3 2024-04-14T15:22:53.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.611+0000 7fd393061700 20 client.37731.objecter _get_session s=0x7fd38c1c4a80 osd=7 3 2024-04-14T15:22:53.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.611+0000 7fd393061700 15 client.37731.objecter _session_command_op_remove -1 16 2024-04-14T15:22:53.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.611+0000 7fd393061700 20 client.37731.objecter get_session s=0x7fd38c1c4a80 osd=7 3 2024-04-14T15:22:53.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.611+0000 7fd393061700 15 client.37731.objecter _session_command_op_assign 7 16 2024-04-14T15:22:53.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.611+0000 7fd393061700 20 client.37731.objecter put_session s=0x7fd38c1c4a80 osd=7 4 2024-04-14T15:22:53.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.611+0000 7fd393061700 10 client.37731.objecter _send_command 16 2024-04-14T15:22:53.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.611+0000 7fd393061700 1 -- 172.21.15.90:0/4161790804 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd38c1c4dc0 con 0x7fd38c1c84c0 2024-04-14T15:22:53.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.613+0000 7fd389ffb700 1 -- 172.21.15.90:0/4161790804 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd38c1c4dc0 con 0x7fd38c1c84c0 2024-04-14T15:22:53.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.613+0000 7fd389ffb700 10 client.37731.objecter ms_dispatch 0x7fd38c001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:22:53.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.613+0000 7fd389ffb700 10 client.37731.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:22:53.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.613+0000 7fd389ffb700 20 client.37731.objecter put_session s=0x7fd38c1c4a80 osd=7 4 2024-04-14T15:22:53.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.613+0000 7fd389ffb700 15 client.37731.objecter _session_command_op_remove 7 16 2024-04-14T15:22:53.614 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:22:53.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.613+0000 7fd393061700 10 librados: watch_flush enter 2024-04-14T15:22:53.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.613+0000 7fd393061700 10 librados: watch_flush exit 2024-04-14T15:22:53.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.613+0000 7fd393061700 10 client.37731.objecter close_session for osd.0 2024-04-14T15:22:53.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.613+0000 7fd393061700 1 -- 172.21.15.90:0/4161790804 >> v1:172.21.15.90:6801/3761806885 conn(0x7fd38c1a97e0 legacy=0x7fd38c1a9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:53.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.613+0000 7fd393061700 20 client.37731.objecter put_session s=0x7fd38c1122e0 osd=0 1 2024-04-14T15:22:53.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.613+0000 7fd393061700 10 client.37731.objecter close_session for osd.1 2024-04-14T15:22:53.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.613+0000 7fd393061700 1 -- 172.21.15.90:0/4161790804 >> v1:172.21.15.90:6805/4170361699 conn(0x7fd38c0636c0 legacy=0x7fd38c108710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:53.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.613+0000 7fd393061700 20 client.37731.objecter put_session s=0x7fd38c02e150 osd=1 1 2024-04-14T15:22:53.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.613+0000 7fd393061700 10 client.37731.objecter close_session for osd.2 2024-04-14T15:22:53.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.613+0000 7fd393061700 1 -- 172.21.15.90:0/4161790804 >> v1:172.21.15.90:6809/1686184251 conn(0x7fd38c1b2080 legacy=0x7fd38c1b4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:53.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.613+0000 7fd393061700 20 client.37731.objecter put_session s=0x7fd38c063d90 osd=2 1 2024-04-14T15:22:53.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.613+0000 7fd393061700 10 client.37731.objecter close_session for osd.3 2024-04-14T15:22:53.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.613+0000 7fd393061700 1 -- 172.21.15.90:0/4161790804 >> v1:172.21.15.90:6813/2270166774 conn(0x7fd38c02dcb0 legacy=0x7fd38c1b8ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:53.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.613+0000 7fd393061700 20 client.37731.objecter put_session s=0x7fd38c063540 osd=3 1 2024-04-14T15:22:53.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.613+0000 7fd393061700 10 client.37731.objecter close_session for osd.4 2024-04-14T15:22:53.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.613+0000 7fd393061700 1 -- 172.21.15.90:0/4161790804 >> v1:172.21.15.92:6800/3887549733 conn(0x7fd38c1b6a20 legacy=0x7fd38c1ba870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:53.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.613+0000 7fd393061700 20 client.37731.objecter put_session s=0x7fd38c02d320 osd=4 1 2024-04-14T15:22:53.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.613+0000 7fd393061700 10 client.37731.objecter close_session for osd.5 2024-04-14T15:22:53.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.613+0000 7fd393061700 1 -- 172.21.15.90:0/4161790804 >> v1:172.21.15.92:6804/3244561440 conn(0x7fd38c1bf8a0 legacy=0x7fd38c1c1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:53.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.613+0000 7fd393061700 20 client.37731.objecter put_session s=0x7fd38c1bf220 osd=5 1 2024-04-14T15:22:53.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.613+0000 7fd393061700 10 client.37731.objecter close_session for osd.6 2024-04-14T15:22:53.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.613+0000 7fd393061700 1 -- 172.21.15.90:0/4161790804 >> v1:172.21.15.92:6808/636125963 conn(0x7fd38c1c3ee0 legacy=0x7fd38c1c4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:53.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.613+0000 7fd393061700 20 client.37731.objecter put_session s=0x7fd38c1c3600 osd=6 1 2024-04-14T15:22:53.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.613+0000 7fd393061700 10 client.37731.objecter close_session for osd.7 2024-04-14T15:22:53.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.613+0000 7fd393061700 1 -- 172.21.15.90:0/4161790804 >> v1:172.21.15.92:6812/1773706716 conn(0x7fd38c1c84c0 legacy=0x7fd38c1ca8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:53.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.613+0000 7fd393061700 20 client.37731.objecter put_session s=0x7fd38c1c4a80 osd=7 1 2024-04-14T15:22:53.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.613+0000 7fd393061700 20 client.37731.objecter shutdown clearing up homeless session... 2024-04-14T15:22:53.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.613+0000 7fd393061700 10 client.37731.objecter successfully canceled tick 2024-04-14T15:22:53.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.613+0000 7fd3915fe700 1 -- 172.21.15.90:0/4161790804 reap_dead start 2024-04-14T15:22:53.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.613+0000 7fd3915fe700 1 -- 172.21.15.90:0/4161790804 reap_dead start 2024-04-14T15:22:53.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.613+0000 7fd3915fe700 1 -- 172.21.15.90:0/4161790804 reap_dead start 2024-04-14T15:22:53.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.613+0000 7fd3915fe700 1 -- 172.21.15.90:0/4161790804 reap_dead start 2024-04-14T15:22:53.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.613+0000 7fd3915fe700 1 -- 172.21.15.90:0/4161790804 reap_dead start 2024-04-14T15:22:53.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.614+0000 7fd393061700 1 -- 172.21.15.90:0/4161790804 >> v1:172.21.15.90:6800/420935427 conn(0x7fd37c071440 legacy=0x7fd37c073900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:53.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.614+0000 7fd393061700 1 -- 172.21.15.90:0/4161790804 >> v1:172.21.15.90:6789/0 conn(0x7fd38c107890 legacy=0x7fd38c06acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:53.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.614+0000 7fd393061700 1 -- 172.21.15.90:0/4161790804 shutdown_connections 2024-04-14T15:22:53.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.614+0000 7fd393061700 1 -- 172.21.15.90:0/4161790804 >> 172.21.15.90:0/4161790804 conn(0x7fd38c0fd290 msgr2=0x7fd38c1118f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:22:53.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.614+0000 7fd393061700 1 -- 172.21.15.90:0/4161790804 shutdown_connections 2024-04-14T15:22:53.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.614+0000 7fd393061700 1 -- 172.21.15.90:0/4161790804 wait complete. 2024-04-14T15:22:53.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:53.614+0000 7fd393061700 1 librados: shutdown 2024-04-14T15:22:53.618 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:22:53.618 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:22:53.618 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:22:53.618 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:22:53.618 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:22:53.618 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:22:53.618 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:22:53.618 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:22:53.637 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:22:53.660 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:53.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:53.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:22:53.761 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:53.784 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:53.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:53.786 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:53.887 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:22:53.915 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:53.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:53.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:22:54.016 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:22:54.040 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:54.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:54.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:22:54.142 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:22:54.169 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:54.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:54.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:22:54.271 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:22:54.299 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:54.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:54.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:22:54.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:54 smithi092 ceph-mon[33966]: 2.1b scrub starts 2024-04-14T15:22:54.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:54 smithi092 ceph-mon[33966]: 2.1b scrub ok 2024-04-14T15:22:54.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:54 smithi092 ceph-mon[33966]: 11.1b scrub starts 2024-04-14T15:22:54.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:54 smithi092 ceph-mon[33966]: 11.1b scrub ok 2024-04-14T15:22:54.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:54 smithi092 ceph-mon[33966]: 5.3 scrub starts 2024-04-14T15:22:54.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:54 smithi092 ceph-mon[33966]: 5.3 scrub ok 2024-04-14T15:22:54.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:54 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:22:54.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:54 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:22:54.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:54 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/4161790804' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:22:54.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:54 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:22:54.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:54 smithi092 ceph-mon[33966]: osdmap e638: 8 total, 8 up, 7 in 2024-04-14T15:22:54.401 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:22:54.426 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:54.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:54.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:22:54.441 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:22:53 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:22:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:22:54.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:53 smithi090 ceph-mon[41670]: 2.1b scrub starts 2024-04-14T15:22:54.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:53 smithi090 ceph-mon[41670]: 2.1b scrub ok 2024-04-14T15:22:54.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:53 smithi090 ceph-mon[41670]: 11.1b scrub starts 2024-04-14T15:22:54.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:53 smithi090 ceph-mon[41670]: 11.1b scrub ok 2024-04-14T15:22:54.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:53 smithi090 ceph-mon[41670]: 5.3 scrub starts 2024-04-14T15:22:54.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:53 smithi090 ceph-mon[41670]: 5.3 scrub ok 2024-04-14T15:22:54.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:53 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:22:54.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:53 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:22:54.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:53 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/4161790804' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:22:54.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:53 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:22:54.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:53 smithi090 ceph-mon[41670]: osdmap e638: 8 total, 8 up, 7 in 2024-04-14T15:22:54.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:53 smithi090 ceph-mon[30399]: 2.1b scrub starts 2024-04-14T15:22:54.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:53 smithi090 ceph-mon[30399]: 2.1b scrub ok 2024-04-14T15:22:54.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:53 smithi090 ceph-mon[30399]: 11.1b scrub starts 2024-04-14T15:22:54.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:53 smithi090 ceph-mon[30399]: 11.1b scrub ok 2024-04-14T15:22:54.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:53 smithi090 ceph-mon[30399]: 5.3 scrub starts 2024-04-14T15:22:54.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:53 smithi090 ceph-mon[30399]: 5.3 scrub ok 2024-04-14T15:22:54.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:53 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:22:54.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:53 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:22:54.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:53 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/4161790804' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:22:54.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:53 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:22:54.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:53 smithi090 ceph-mon[30399]: osdmap e638: 8 total, 8 up, 7 in 2024-04-14T15:22:54.502 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:22:54.503 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:22:54.527 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:22:54.555 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:54.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:54.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:22:54.657 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:22:54.681 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:54.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:54.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:22:54.782 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:22:54.810 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:54.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:54.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:22:54.911 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:22:54.936 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:54.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:54.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:22:55.038 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:22:55.063 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:55.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:55.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:22:55.165 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:22:55.189 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:55.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:55.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:22:55.264 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:55 smithi090 ceph-mon[41670]: pgmap v936: 226 pgs: 3 active+remapped+backfill_toofull, 223 active+clean; 596 KiB data, 577 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/600 objects misplaced (0.333%) 2024-04-14T15:22:55.264 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:55 smithi090 ceph-mon[41670]: 11.b scrub starts 2024-04-14T15:22:55.264 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:55 smithi090 ceph-mon[41670]: 11.b scrub ok 2024-04-14T15:22:55.264 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:55 smithi090 ceph-mon[41670]: 4.1b scrub starts 2024-04-14T15:22:55.264 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:55 smithi090 ceph-mon[41670]: 4.1b scrub ok 2024-04-14T15:22:55.264 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:55 smithi090 ceph-mon[41670]: osdmap e639: 8 total, 8 up, 7 in 2024-04-14T15:22:55.264 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:55 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi090-93652-109"}]: dispatch 2024-04-14T15:22:55.265 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:55 smithi090 ceph-mon[30399]: pgmap v936: 226 pgs: 3 active+remapped+backfill_toofull, 223 active+clean; 596 KiB data, 577 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/600 objects misplaced (0.333%) 2024-04-14T15:22:55.265 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:55 smithi090 ceph-mon[30399]: 11.b scrub starts 2024-04-14T15:22:55.265 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:55 smithi090 ceph-mon[30399]: 11.b scrub ok 2024-04-14T15:22:55.265 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:55 smithi090 ceph-mon[30399]: 4.1b scrub starts 2024-04-14T15:22:55.265 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:55 smithi090 ceph-mon[30399]: 4.1b scrub ok 2024-04-14T15:22:55.265 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:55 smithi090 ceph-mon[30399]: osdmap e639: 8 total, 8 up, 7 in 2024-04-14T15:22:55.265 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:55 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi090-93652-109"}]: dispatch 2024-04-14T15:22:55.291 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:22:55.314 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:55.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:55.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:22:55.323 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:55 smithi092 ceph-mon[33966]: pgmap v936: 226 pgs: 3 active+remapped+backfill_toofull, 223 active+clean; 596 KiB data, 577 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/600 objects misplaced (0.333%) 2024-04-14T15:22:55.323 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:55 smithi092 ceph-mon[33966]: 11.b scrub starts 2024-04-14T15:22:55.323 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:55 smithi092 ceph-mon[33966]: 11.b scrub ok 2024-04-14T15:22:55.323 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:55 smithi092 ceph-mon[33966]: 4.1b scrub starts 2024-04-14T15:22:55.323 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:55 smithi092 ceph-mon[33966]: 4.1b scrub ok 2024-04-14T15:22:55.323 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:55 smithi092 ceph-mon[33966]: osdmap e639: 8 total, 8 up, 7 in 2024-04-14T15:22:55.324 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:55 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi090-93652-109"}]: dispatch 2024-04-14T15:22:55.416 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:22:55.440 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:55.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:55.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:22:55.541 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:22:55.564 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:55.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:55.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:22:55.622 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:22:55 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:22:55.666 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:22:55.690 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:55.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:55.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:22:55.791 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:22:55.815 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:55.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:55.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:22:55.917 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:22:55.944 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:55.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:55.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:22:56.004 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-14T15:22:56.004 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:22:56.046 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:22:56.074 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:56.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:56.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:22:56.175 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:22:56.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:56 smithi090 ceph-mon[41670]: 2.14 scrub starts 2024-04-14T15:22:56.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:56 smithi090 ceph-mon[41670]: 2.14 scrub ok 2024-04-14T15:22:56.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:56 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-04-14T15:22:56.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:56 smithi090 ceph-mon[41670]: 5.1 scrub starts 2024-04-14T15:22:56.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:56 smithi090 ceph-mon[41670]: 5.1 scrub ok 2024-04-14T15:22:56.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:56 smithi090 ceph-mon[41670]: 4.16 scrub starts 2024-04-14T15:22:56.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:56 smithi090 ceph-mon[41670]: 4.16 scrub ok 2024-04-14T15:22:56.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:56 smithi090 ceph-mon[41670]: 2.9 scrub starts 2024-04-14T15:22:56.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:56 smithi090 ceph-mon[41670]: 2.9 scrub ok 2024-04-14T15:22:56.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:56 smithi090 ceph-mon[41670]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:22:56.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:56 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi090-93652-109"}]': finished 2024-04-14T15:22:56.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:56 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]': finished 2024-04-14T15:22:56.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:56 smithi090 ceph-mon[41670]: osdmap e640: 8 total, 8 up, 7 in 2024-04-14T15:22:56.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:56 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi090-93652-109"}]: dispatch 2024-04-14T15:22:56.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:56 smithi090 ceph-mon[30399]: 2.14 scrub starts 2024-04-14T15:22:56.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:56 smithi090 ceph-mon[30399]: 2.14 scrub ok 2024-04-14T15:22:56.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:56 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-04-14T15:22:56.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:56 smithi090 ceph-mon[30399]: 5.1 scrub starts 2024-04-14T15:22:56.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:56 smithi090 ceph-mon[30399]: 5.1 scrub ok 2024-04-14T15:22:56.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:56 smithi090 ceph-mon[30399]: 4.16 scrub starts 2024-04-14T15:22:56.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:56 smithi090 ceph-mon[30399]: 4.16 scrub ok 2024-04-14T15:22:56.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:56 smithi090 ceph-mon[30399]: 2.9 scrub starts 2024-04-14T15:22:56.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:56 smithi090 ceph-mon[30399]: 2.9 scrub ok 2024-04-14T15:22:56.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:56 smithi090 ceph-mon[30399]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:22:56.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:56 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi090-93652-109"}]': finished 2024-04-14T15:22:56.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:56 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]': finished 2024-04-14T15:22:56.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:56 smithi090 ceph-mon[30399]: osdmap e640: 8 total, 8 up, 7 in 2024-04-14T15:22:56.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:56 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi090-93652-109"}]: dispatch 2024-04-14T15:22:56.202 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:56.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:56.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:22:56.304 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:56.328 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:56.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:56.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:56 smithi092 ceph-mon[33966]: 2.14 scrub starts 2024-04-14T15:22:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:56 smithi092 ceph-mon[33966]: 2.14 scrub ok 2024-04-14T15:22:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:56 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-04-14T15:22:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:56 smithi092 ceph-mon[33966]: 5.1 scrub starts 2024-04-14T15:22:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:56 smithi092 ceph-mon[33966]: 5.1 scrub ok 2024-04-14T15:22:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:56 smithi092 ceph-mon[33966]: 4.16 scrub starts 2024-04-14T15:22:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:56 smithi092 ceph-mon[33966]: 4.16 scrub ok 2024-04-14T15:22:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:56 smithi092 ceph-mon[33966]: 2.9 scrub starts 2024-04-14T15:22:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:56 smithi092 ceph-mon[33966]: 2.9 scrub ok 2024-04-14T15:22:56.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:56 smithi092 ceph-mon[33966]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:22:56.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:56 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi090-93652-109"}]': finished 2024-04-14T15:22:56.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:56 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]': finished 2024-04-14T15:22:56.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:56 smithi092 ceph-mon[33966]: osdmap e640: 8 total, 8 up, 7 in 2024-04-14T15:22:56.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:56 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi090-93652-109"}]: dispatch 2024-04-14T15:22:56.429 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:22:56.453 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:56.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:56.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:22:56.555 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:22:56.581 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:56.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:56.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:22:56.683 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:22:56.706 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:56.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:56.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:22:56.808 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:22:56.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:56.833 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:56.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:22:56.934 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:22:56.958 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:56.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:56.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:22:57.060 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:22:57.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:57.085 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:57.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:22:57.186 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:22:57.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:57 smithi090 ceph-mon[30399]: pgmap v939: 186 pgs: 1 active+remapped+backfill_toofull, 185 active+clean; 582 KiB data, 578 MiB used, 625 GiB / 626 GiB avail; 2/600 objects misplaced (0.333%) 2024-04-14T15:22:57.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:57 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:22:57.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:57 smithi090 ceph-mon[30399]: 6.8 scrub starts 2024-04-14T15:22:57.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:57 smithi090 ceph-mon[30399]: 6.8 scrub ok 2024-04-14T15:22:57.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:57 smithi090 ceph-mon[30399]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-04-14T15:22:57.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:57 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:22:57.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:57 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi090-93652-109"}]': finished 2024-04-14T15:22:57.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:57 smithi090 ceph-mon[30399]: osdmap e641: 8 total, 8 up, 7 in 2024-04-14T15:22:57.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:57 smithi090 ceph-mon[41670]: pgmap v939: 186 pgs: 1 active+remapped+backfill_toofull, 185 active+clean; 582 KiB data, 578 MiB used, 625 GiB / 626 GiB avail; 2/600 objects misplaced (0.333%) 2024-04-14T15:22:57.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:57 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:22:57.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:57 smithi090 ceph-mon[41670]: 6.8 scrub starts 2024-04-14T15:22:57.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:57 smithi090 ceph-mon[41670]: 6.8 scrub ok 2024-04-14T15:22:57.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:57 smithi090 ceph-mon[41670]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-04-14T15:22:57.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:57 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:22:57.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:57 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi090-93652-109"}]': finished 2024-04-14T15:22:57.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:57 smithi090 ceph-mon[41670]: osdmap e641: 8 total, 8 up, 7 in 2024-04-14T15:22:57.209 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:57.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:57.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:22:57.310 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:22:57.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:57.335 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:57.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:22:57.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:57 smithi092 ceph-mon[33966]: pgmap v939: 186 pgs: 1 active+remapped+backfill_toofull, 185 active+clean; 582 KiB data, 578 MiB used, 625 GiB / 626 GiB avail; 2/600 objects misplaced (0.333%) 2024-04-14T15:22:57.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:57 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:22:57.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:57 smithi092 ceph-mon[33966]: 6.8 scrub starts 2024-04-14T15:22:57.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:57 smithi092 ceph-mon[33966]: 6.8 scrub ok 2024-04-14T15:22:57.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:57 smithi092 ceph-mon[33966]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-04-14T15:22:57.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:57 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:22:57.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:57 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3730104738' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi090-93652-109"}]': finished 2024-04-14T15:22:57.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:57 smithi092 ceph-mon[33966]: osdmap e641: 8 total, 8 up, 7 in 2024-04-14T15:22:57.437 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:22:57.461 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:57.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:57.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:22:57.562 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:22:57.586 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:57.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:57.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:22:57.687 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:22:57.712 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:57.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:57.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:22:57.814 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:57.838 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:57.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:57.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:57.940 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:22:57.962 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:57.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:57.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:22:58.064 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:22:58.087 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:58.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:58.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:22:58.189 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:22:58.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:58 smithi090 ceph-mon[30399]: 2.5 scrub starts 2024-04-14T15:22:58.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:58 smithi090 ceph-mon[30399]: 2.5 scrub ok 2024-04-14T15:22:58.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:58 smithi090 ceph-mon[30399]: 4.18 scrub starts 2024-04-14T15:22:58.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:58 smithi090 ceph-mon[30399]: 4.18 scrub ok 2024-04-14T15:22:58.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:58 smithi090 ceph-mon[30399]: 1.3 scrub starts 2024-04-14T15:22:58.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:58 smithi090 ceph-mon[30399]: 1.3 scrub ok 2024-04-14T15:22:58.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:58 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:22:58.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:58 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:22:58.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:58 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:22:58.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:58 smithi090 ceph-mon[30399]: osdmap e642: 8 total, 8 up, 7 in 2024-04-14T15:22:58.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:58 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1673052523' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi090-93652-104"}]: dispatch 2024-04-14T15:22:58.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:58 smithi090 ceph-mon[41670]: 2.5 scrub starts 2024-04-14T15:22:58.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:58 smithi090 ceph-mon[41670]: 2.5 scrub ok 2024-04-14T15:22:58.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:58 smithi090 ceph-mon[41670]: 4.18 scrub starts 2024-04-14T15:22:58.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:58 smithi090 ceph-mon[41670]: 4.18 scrub ok 2024-04-14T15:22:58.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:58 smithi090 ceph-mon[41670]: 1.3 scrub starts 2024-04-14T15:22:58.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:58 smithi090 ceph-mon[41670]: 1.3 scrub ok 2024-04-14T15:22:58.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:58 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:22:58.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:58 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:22:58.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:58 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:22:58.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:58 smithi090 ceph-mon[41670]: osdmap e642: 8 total, 8 up, 7 in 2024-04-14T15:22:58.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:58 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1673052523' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi090-93652-104"}]: dispatch 2024-04-14T15:22:58.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:58.214 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:58.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:22:58.315 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:22:58.339 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:58.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:58.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:22:58.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:58 smithi092 ceph-mon[33966]: 2.5 scrub starts 2024-04-14T15:22:58.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:58 smithi092 ceph-mon[33966]: 2.5 scrub ok 2024-04-14T15:22:58.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:58 smithi092 ceph-mon[33966]: 4.18 scrub starts 2024-04-14T15:22:58.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:58 smithi092 ceph-mon[33966]: 4.18 scrub ok 2024-04-14T15:22:58.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:58 smithi092 ceph-mon[33966]: 1.3 scrub starts 2024-04-14T15:22:58.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:58 smithi092 ceph-mon[33966]: 1.3 scrub ok 2024-04-14T15:22:58.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:58 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:22:58.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:58 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:22:58.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:58 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:22:58.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:58 smithi092 ceph-mon[33966]: osdmap e642: 8 total, 8 up, 7 in 2024-04-14T15:22:58.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:58 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1673052523' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi090-93652-104"}]: dispatch 2024-04-14T15:22:58.440 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:22:58.464 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:58.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:58.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:22:58.565 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:22:58.589 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:58.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:58.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:22:58.690 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:58.715 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:58.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:58.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:58.816 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:22:58.840 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:58.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:58.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:22:58.941 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:22:58.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:58.966 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:58.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:22:59.067 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:59.091 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:59.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:59.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:59.159 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:59 smithi090 ceph-mon[41670]: pgmap v942: 186 pgs: 1 active+remapped+backfill_toofull, 185 active+clean; 582 KiB data, 578 MiB used, 625 GiB / 626 GiB avail; 2/600 objects misplaced (0.333%) 2024-04-14T15:22:59.159 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:59 smithi090 ceph-mon[41670]: 4.0 scrub starts 2024-04-14T15:22:59.159 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:59 smithi090 ceph-mon[41670]: 4.0 scrub ok 2024-04-14T15:22:59.159 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:59 smithi090 ceph-mon[41670]: 4.1e scrub starts 2024-04-14T15:22:59.159 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:59 smithi090 ceph-mon[41670]: 4.1e scrub ok 2024-04-14T15:22:59.159 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:59 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1673052523' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi090-93652-104"}]': finished 2024-04-14T15:22:59.159 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:59 smithi090 ceph-mon[41670]: osdmap e643: 8 total, 8 up, 7 in 2024-04-14T15:22:59.159 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:22:59 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1673052523' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi090-93652-104"}]: dispatch 2024-04-14T15:22:59.159 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:59 smithi090 ceph-mon[30399]: pgmap v942: 186 pgs: 1 active+remapped+backfill_toofull, 185 active+clean; 582 KiB data, 578 MiB used, 625 GiB / 626 GiB avail; 2/600 objects misplaced (0.333%) 2024-04-14T15:22:59.159 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:59 smithi090 ceph-mon[30399]: 4.0 scrub starts 2024-04-14T15:22:59.159 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:59 smithi090 ceph-mon[30399]: 4.0 scrub ok 2024-04-14T15:22:59.160 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:59 smithi090 ceph-mon[30399]: 4.1e scrub starts 2024-04-14T15:22:59.160 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:59 smithi090 ceph-mon[30399]: 4.1e scrub ok 2024-04-14T15:22:59.160 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:59 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1673052523' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi090-93652-104"}]': finished 2024-04-14T15:22:59.160 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:59 smithi090 ceph-mon[30399]: osdmap e643: 8 total, 8 up, 7 in 2024-04-14T15:22:59.160 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:22:59 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1673052523' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi090-93652-104"}]: dispatch 2024-04-14T15:22:59.186 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-14T15:22:59.193 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:22:59.253 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:59.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:59.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:22:59.354 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:59.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:59 smithi092 ceph-mon[33966]: pgmap v942: 186 pgs: 1 active+remapped+backfill_toofull, 185 active+clean; 582 KiB data, 578 MiB used, 625 GiB / 626 GiB avail; 2/600 objects misplaced (0.333%) 2024-04-14T15:22:59.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:59 smithi092 ceph-mon[33966]: 4.0 scrub starts 2024-04-14T15:22:59.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:59 smithi092 ceph-mon[33966]: 4.0 scrub ok 2024-04-14T15:22:59.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:59 smithi092 ceph-mon[33966]: 4.1e scrub starts 2024-04-14T15:22:59.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:59 smithi092 ceph-mon[33966]: 4.1e scrub ok 2024-04-14T15:22:59.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:59 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1673052523' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi090-93652-104"}]': finished 2024-04-14T15:22:59.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:59 smithi092 ceph-mon[33966]: osdmap e643: 8 total, 8 up, 7 in 2024-04-14T15:22:59.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:22:59 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1673052523' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi090-93652-104"}]: dispatch 2024-04-14T15:22:59.378 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:59.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:59.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:59.426 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:22:59.479 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:59.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:59.503 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:59.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:59.603 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:22:59.626 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:59.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:59.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:22:59.728 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:22:59.752 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:22:59.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:59.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:22:59.854 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:22:59.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:22:59.879 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:22:59.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:22:59.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:59.943+0000 7fc13a879700 1 -- 172.21.15.90:0/991333299 >> v1:172.21.15.90:6789/0 conn(0x7fc13410b730 legacy=0x7fc13410db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:59.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:59.943+0000 7fc13a879700 1 -- 172.21.15.90:0/991333299 shutdown_connections 2024-04-14T15:22:59.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:59.943+0000 7fc13a879700 1 -- 172.21.15.90:0/991333299 >> 172.21.15.90:0/991333299 conn(0x7fc1340fe3d0 msgr2=0x7fc1341007f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:22:59.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:59.943+0000 7fc13a879700 1 -- 172.21.15.90:0/991333299 shutdown_connections 2024-04-14T15:22:59.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:59.943+0000 7fc13a879700 1 -- 172.21.15.90:0/991333299 wait complete. 2024-04-14T15:22:59.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:59.943+0000 7fc13a879700 1 librados: starting msgr at 2024-04-14T15:22:59.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:59.943+0000 7fc13a879700 1 librados: starting objecter 2024-04-14T15:22:59.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:59.943+0000 7fc13a879700 1 Processor -- start 2024-04-14T15:22:59.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:59.944+0000 7fc13a879700 1 -- start start 2024-04-14T15:22:59.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:59.944+0000 7fc13a879700 1 librados: setting wanted keys 2024-04-14T15:22:59.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:59.944+0000 7fc13a879700 1 librados: calling monclient init 2024-04-14T15:22:59.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:59.944+0000 7fc13a879700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc13410eff0 con 0x7fc13410f300 2024-04-14T15:22:59.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:59.944+0000 7fc13a879700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc1341a8f10 con 0x7fc134108880 2024-04-14T15:22:59.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:59.944+0000 7fc13a879700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc1341aa0f0 con 0x7fc13410b730 2024-04-14T15:22:59.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:59.944+0000 7fc138e16700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fc13410f300 0x7fc1341a67e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:41736/0 (socket says 172.21.15.90:41736) 2024-04-14T15:22:59.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:59.944+0000 7fc1337fe700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7fc13410b730 0x7fc13410dfc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:34558/0 (socket says 172.21.15.90:34558) 2024-04-14T15:22:59.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:59.944+0000 7fc138e16700 1 -- 172.21.15.90:0/3145247990 learned_addr learned my addr 172.21.15.90:0/3145247990 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:22:59.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:59.944+0000 7fc1317fa700 10 client.?.objecter ms_handle_connect 0x7fc13410f300 2024-04-14T15:22:59.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:59.944+0000 7fc1317fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:22:59.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:59.944+0000 7fc1317fa700 10 client.?.objecter ms_handle_connect 0x7fc13410b730 2024-04-14T15:22:59.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:59.944+0000 7fc1317fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:22:59.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:59.944+0000 7fc1317fa700 10 client.?.objecter ms_handle_connect 0x7fc134108880 2024-04-14T15:22:59.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:59.944+0000 7fc1317fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:22:59.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:59.944+0000 7fc1317fa700 1 -- 172.21.15.90:0/3145247990 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3100838972 0 0) 0x7fc13410eff0 con 0x7fc13410f300 2024-04-14T15:22:59.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:59.944+0000 7fc1317fa700 1 -- 172.21.15.90:0/3145247990 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc11c0035b0 con 0x7fc13410f300 2024-04-14T15:22:59.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:59.944+0000 7fc1317fa700 1 -- 172.21.15.90:0/3145247990 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3166460574 0 0) 0x7fc1341a8f10 con 0x7fc134108880 2024-04-14T15:22:59.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:59.944+0000 7fc1317fa700 1 -- 172.21.15.90:0/3145247990 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc13410eff0 con 0x7fc134108880 2024-04-14T15:22:59.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:59.944+0000 7fc1317fa700 1 -- 172.21.15.90:0/3145247990 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1030303985 0 0) 0x7fc1341aa0f0 con 0x7fc13410b730 2024-04-14T15:22:59.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:59.944+0000 7fc1317fa700 1 -- 172.21.15.90:0/3145247990 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc1341a8f10 con 0x7fc13410b730 2024-04-14T15:22:59.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:59.944+0000 7fc1317fa700 1 -- 172.21.15.90:0/3145247990 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1363067401 0 0) 0x7fc13410eff0 con 0x7fc134108880 2024-04-14T15:22:59.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:59.944+0000 7fc1317fa700 1 -- 172.21.15.90:0/3145247990 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc1341aa0f0 con 0x7fc134108880 2024-04-14T15:22:59.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:59.944+0000 7fc1317fa700 1 -- 172.21.15.90:0/3145247990 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fc1240017f0 con 0x7fc134108880 2024-04-14T15:22:59.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:59.945+0000 7fc1317fa700 1 -- 172.21.15.90:0/3145247990 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2755417982 0 0) 0x7fc11c0035b0 con 0x7fc13410f300 2024-04-14T15:22:59.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:59.945+0000 7fc1317fa700 1 -- 172.21.15.90:0/3145247990 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc13410eff0 con 0x7fc13410f300 2024-04-14T15:22:59.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:59.945+0000 7fc1317fa700 1 -- 172.21.15.90:0/3145247990 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3929552520 0 0) 0x7fc1341a8f10 con 0x7fc13410b730 2024-04-14T15:22:59.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:59.945+0000 7fc1317fa700 1 -- 172.21.15.90:0/3145247990 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc11c0035b0 con 0x7fc13410b730 2024-04-14T15:22:59.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:59.945+0000 7fc1317fa700 1 -- 172.21.15.90:0/3145247990 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fc12c001bb0 con 0x7fc13410f300 2024-04-14T15:22:59.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:59.945+0000 7fc1317fa700 1 -- 172.21.15.90:0/3145247990 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fc128001430 con 0x7fc13410b730 2024-04-14T15:22:59.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:59.945+0000 7fc1317fa700 1 -- 172.21.15.90:0/3145247990 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 960103254 0 0) 0x7fc1341aa0f0 con 0x7fc134108880 2024-04-14T15:22:59.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:59.945+0000 7fc1317fa700 1 -- 172.21.15.90:0/3145247990 >> v1:172.21.15.90:6790/0 conn(0x7fc13410b730 legacy=0x7fc13410dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:59.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:59.945+0000 7fc1317fa700 1 -- 172.21.15.90:0/3145247990 >> v1:172.21.15.90:6789/0 conn(0x7fc13410f300 legacy=0x7fc1341a67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:22:59.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:59.945+0000 7fc1317fa700 1 -- 172.21.15.90:0/3145247990 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1341ab2d0 con 0x7fc134108880 2024-04-14T15:22:59.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:59.945+0000 7fc13a879700 1 -- 172.21.15.90:0/3145247990 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc1341aa320 con 0x7fc134108880 2024-04-14T15:22:59.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:59.945+0000 7fc13a879700 10 client.49570.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:22:59.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:59.945+0000 7fc13a879700 1 -- 172.21.15.90:0/3145247990 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc1341aa7e0 con 0x7fc134108880 2024-04-14T15:22:59.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:59.945+0000 7fc13a879700 1 librados: init done 2024-04-14T15:22:59.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:59.945+0000 7fc1317fa700 1 -- 172.21.15.90:0/3145247990 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fc124003b40 con 0x7fc134108880 2024-04-14T15:22:59.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:59.945+0000 7fc1317fa700 1 -- 172.21.15.90:0/3145247990 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fc124004a00 con 0x7fc134108880 2024-04-14T15:22:59.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:59.946+0000 7fc13a879700 1 -- 172.21.15.90:0/3145247990 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc120005120 con 0x7fc134108880 2024-04-14T15:22:59.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:59.946+0000 7fc1317fa700 1 -- 172.21.15.90:0/3145247990 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fc124003680 con 0x7fc134108880 2024-04-14T15:22:59.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:59.947+0000 7fc1317fa700 1 -- 172.21.15.90:0/3145247990 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(643..643 src has 467..643) ==== 7578+0+0 (unknown 4102732544 0 0) 0x7fc124089400 con 0x7fc134108880 2024-04-14T15:22:59.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:59.947+0000 7fc1317fa700 10 client.49570.objecter ms_dispatch 0x7fc134001550 osd_map(643..643 src has 467..643) 2024-04-14T15:22:59.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:59.947+0000 7fc1317fa700 3 client.49570.objecter handle_osd_map got epochs [643,643] > 0 2024-04-14T15:22:59.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:59.947+0000 7fc1317fa700 3 client.49570.objecter handle_osd_map decoding full epoch 643 2024-04-14T15:22:59.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:59.947+0000 7fc1317fa700 20 client.49570.objecter dump_active .. 0 homeless 2024-04-14T15:22:59.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:59.947+0000 7fc1317fa700 10 client.49570.objecter ms_handle_connect 0x7fc11c071450 2024-04-14T15:22:59.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:22:59.952+0000 7fc1317fa700 1 -- 172.21.15.90:0/3145247990 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fc124056060 con 0x7fc134108880 2024-04-14T15:22:59.979 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:23:00.008 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:00.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:00.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:23:00.110 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:00.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:00 smithi092 ceph-mon[33966]: 6.f scrub starts 2024-04-14T15:23:00.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:00 smithi092 ceph-mon[33966]: 6.f scrub ok 2024-04-14T15:23:00.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:00 smithi092 ceph-mon[33966]: pgmap v945: 177 pgs: 1 active+remapped+backfill_toofull, 176 active+clean; 582 KiB data, 578 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 op/s; 2/597 objects misplaced (0.335%) 2024-04-14T15:23:00.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:00 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-04-14T15:23:00.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:00 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1673052523' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi090-93652-104"}]': finished 2024-04-14T15:23:00.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:00 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]': finished 2024-04-14T15:23:00.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:00 smithi092 ceph-mon[33966]: osdmap e644: 8 total, 8 up, 7 in 2024-04-14T15:23:00.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:00 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:23:00.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:00 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:23:00.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:00 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:23:00.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:00 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:23:00.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:00 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi090-93652-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:23:00.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:00 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi090-93652-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:23:00.134 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:00.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:00.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:00.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.179+0000 7fc13a879700 1 -- 172.21.15.90:0/3145247990 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7fc120005410 con 0x7fc134108880 2024-04-14T15:23:00.215 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:00 smithi090 ceph-mon[41670]: 6.f scrub starts 2024-04-14T15:23:00.216 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:00 smithi090 ceph-mon[41670]: 6.f scrub ok 2024-04-14T15:23:00.216 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:00 smithi090 ceph-mon[41670]: pgmap v945: 177 pgs: 1 active+remapped+backfill_toofull, 176 active+clean; 582 KiB data, 578 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 op/s; 2/597 objects misplaced (0.335%) 2024-04-14T15:23:00.216 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:00 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-04-14T15:23:00.216 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:00 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1673052523' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi090-93652-104"}]': finished 2024-04-14T15:23:00.216 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:00 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]': finished 2024-04-14T15:23:00.216 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:00 smithi090 ceph-mon[41670]: osdmap e644: 8 total, 8 up, 7 in 2024-04-14T15:23:00.216 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:00 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:23:00.216 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:00 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:23:00.216 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:00 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:23:00.216 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:00 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:23:00.216 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:00 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi090-93652-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:23:00.216 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:00 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi090-93652-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:23:00.217 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:00 smithi090 ceph-mon[30399]: 6.f scrub starts 2024-04-14T15:23:00.217 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:00 smithi090 ceph-mon[30399]: 6.f scrub ok 2024-04-14T15:23:00.217 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:00 smithi090 ceph-mon[30399]: pgmap v945: 177 pgs: 1 active+remapped+backfill_toofull, 176 active+clean; 582 KiB data, 578 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 op/s; 2/597 objects misplaced (0.335%) 2024-04-14T15:23:00.217 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:00 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-04-14T15:23:00.217 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:00 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1673052523' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi090-93652-104"}]': finished 2024-04-14T15:23:00.217 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:00 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]': finished 2024-04-14T15:23:00.217 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:00 smithi090 ceph-mon[30399]: osdmap e644: 8 total, 8 up, 7 in 2024-04-14T15:23:00.217 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:00 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:23:00.217 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:00 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:23:00.217 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:00 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:23:00.217 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:00 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:23:00.217 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:00 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi090-93652-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:23:00.218 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:00 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi090-93652-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-14T15:23:00.236 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:23:00.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:00.262 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:00.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:23:00.334 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-14T15:23:00.364 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:23:00.388 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:00.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:00.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:23:00.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.451+0000 7f223e911700 1 -- 172.21.15.90:0/1850788308 >> v1:172.21.15.90:6789/0 conn(0x7f223810d940 legacy=0x7f223810fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:00.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.451+0000 7f223e911700 1 -- 172.21.15.90:0/1850788308 shutdown_connections 2024-04-14T15:23:00.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.451+0000 7f223e911700 1 -- 172.21.15.90:0/1850788308 >> 172.21.15.90:0/1850788308 conn(0x7f2238076aa0 msgr2=0x7f2238076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:23:00.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.451+0000 7f223e911700 1 -- 172.21.15.90:0/1850788308 shutdown_connections 2024-04-14T15:23:00.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.451+0000 7f223e911700 1 -- 172.21.15.90:0/1850788308 wait complete. 2024-04-14T15:23:00.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.451+0000 7f223e911700 1 librados: starting msgr at 2024-04-14T15:23:00.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.451+0000 7f223e911700 1 librados: starting objecter 2024-04-14T15:23:00.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.451+0000 7f223e911700 1 Processor -- start 2024-04-14T15:23:00.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.451+0000 7f223e911700 1 -- start start 2024-04-14T15:23:00.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.451+0000 7f223e911700 1 librados: setting wanted keys 2024-04-14T15:23:00.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.451+0000 7f223e911700 1 librados: calling monclient init 2024-04-14T15:23:00.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.452+0000 7f223e911700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f2238101330 con 0x7f2238109da0 2024-04-14T15:23:00.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.452+0000 7f223e911700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f22381a8de0 con 0x7f2238106f80 2024-04-14T15:23:00.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.452+0000 7f223e911700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f22381aa000 con 0x7f223810d940 2024-04-14T15:23:00.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.452+0000 7f22377fe700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f2238109da0 0x7f22381a2f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:41774/0 (socket says 172.21.15.90:41774) 2024-04-14T15:23:00.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.452+0000 7f22377fe700 1 -- 172.21.15.90:0/1924992346 learned_addr learned my addr 172.21.15.90:0/1924992346 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:23:00.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.452+0000 7f2237fff700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f2238106f80 0x7f2238100740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.90:39732/0 (socket says 172.21.15.90:39732) 2024-04-14T15:23:00.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.452+0000 7f22357fa700 10 client.?.objecter ms_handle_connect 0x7f2238109da0 2024-04-14T15:23:00.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.452+0000 7f22357fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:23:00.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.452+0000 7f22357fa700 10 client.?.objecter ms_handle_connect 0x7f2238106f80 2024-04-14T15:23:00.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.452+0000 7f22357fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:23:00.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.452+0000 7f22357fa700 10 client.?.objecter ms_handle_connect 0x7f223810d940 2024-04-14T15:23:00.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.452+0000 7f22357fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:23:00.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.452+0000 7f22357fa700 1 -- 172.21.15.90:0/1924992346 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1788822262 0 0) 0x7f22381a8de0 con 0x7f2238106f80 2024-04-14T15:23:00.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.452+0000 7f22357fa700 1 -- 172.21.15.90:0/1924992346 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22200035b0 con 0x7f2238106f80 2024-04-14T15:23:00.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.452+0000 7f22357fa700 1 -- 172.21.15.90:0/1924992346 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2021916995 0 0) 0x7f2238101330 con 0x7f2238109da0 2024-04-14T15:23:00.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.452+0000 7f22357fa700 1 -- 172.21.15.90:0/1924992346 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22381a8de0 con 0x7f2238109da0 2024-04-14T15:23:00.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.452+0000 7f22357fa700 1 -- 172.21.15.90:0/1924992346 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1858978781 0 0) 0x7f22381aa000 con 0x7f223810d940 2024-04-14T15:23:00.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.452+0000 7f22357fa700 1 -- 172.21.15.90:0/1924992346 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2238101330 con 0x7f223810d940 2024-04-14T15:23:00.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.452+0000 7f22357fa700 1 -- 172.21.15.90:0/1924992346 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3447108941 0 0) 0x7f22200035b0 con 0x7f2238106f80 2024-04-14T15:23:00.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.452+0000 7f22357fa700 1 -- 172.21.15.90:0/1924992346 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f22381aa000 con 0x7f2238106f80 2024-04-14T15:23:00.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.452+0000 7f22357fa700 1 -- 172.21.15.90:0/1924992346 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f22280018b0 con 0x7f2238106f80 2024-04-14T15:23:00.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.452+0000 7f22357fa700 1 -- 172.21.15.90:0/1924992346 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4280371027 0 0) 0x7f22381a8de0 con 0x7f2238109da0 2024-04-14T15:23:00.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.452+0000 7f22357fa700 1 -- 172.21.15.90:0/1924992346 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f22200035b0 con 0x7f2238109da0 2024-04-14T15:23:00.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.452+0000 7f22357fa700 1 -- 172.21.15.90:0/1924992346 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f222c0018f0 con 0x7f2238109da0 2024-04-14T15:23:00.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.453+0000 7f22357fa700 1 -- 172.21.15.90:0/1924992346 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1797576885 0 0) 0x7f2238101330 con 0x7f223810d940 2024-04-14T15:23:00.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.453+0000 7f22357fa700 1 -- 172.21.15.90:0/1924992346 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f22381a8de0 con 0x7f223810d940 2024-04-14T15:23:00.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.453+0000 7f22357fa700 1 -- 172.21.15.90:0/1924992346 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f22300017b0 con 0x7f223810d940 2024-04-14T15:23:00.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.453+0000 7f22357fa700 1 -- 172.21.15.90:0/1924992346 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 21109942 0 0) 0x7f22200035b0 con 0x7f2238109da0 2024-04-14T15:23:00.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.453+0000 7f22357fa700 1 -- 172.21.15.90:0/1924992346 >> v1:172.21.15.90:6790/0 conn(0x7f223810d940 legacy=0x7f22381a66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:00.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.453+0000 7f22357fa700 1 -- 172.21.15.90:0/1924992346 >> v1:172.21.15.92:6789/0 conn(0x7f2238106f80 legacy=0x7f2238100740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:00.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.453+0000 7f22357fa700 1 -- 172.21.15.90:0/1924992346 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22381ab240 con 0x7f2238109da0 2024-04-14T15:23:00.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.453+0000 7f223e911700 1 -- 172.21.15.90:0/1924992346 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f22381a7e30 con 0x7f2238109da0 2024-04-14T15:23:00.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.453+0000 7f223e911700 10 client.37749.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:23:00.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.453+0000 7f223e911700 1 -- 172.21.15.90:0/1924992346 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f22381a82f0 con 0x7f2238109da0 2024-04-14T15:23:00.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.453+0000 7f223e911700 1 librados: init done 2024-04-14T15:23:00.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.453+0000 7f22357fa700 1 -- 172.21.15.90:0/1924992346 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f222c003b40 con 0x7f2238109da0 2024-04-14T15:23:00.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.453+0000 7f22357fa700 1 -- 172.21.15.90:0/1924992346 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f222c005d30 con 0x7f2238109da0 2024-04-14T15:23:00.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.454+0000 7f223e911700 10 client.37749.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:23:00.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.454+0000 7f223e911700 15 client.37749.objecter _session_command_op_assign -1 1 2024-04-14T15:23:00.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.454+0000 7f223e911700 20 client.37749.objecter _get_session osd=-1 returning homeless 2024-04-14T15:23:00.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.454+0000 7f223e911700 10 client.37749.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:23:00.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.454+0000 7f223e911700 1 -- 172.21.15.90:0/1924992346 --> v1:172.21.15.90:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f2224002920 con 0x7f2238109da0 2024-04-14T15:23:00.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.455+0000 7f22357fa700 1 -- 172.21.15.90:0/1924992346 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f222c0036a0 con 0x7f2238109da0 2024-04-14T15:23:00.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.455+0000 7f22357fa700 1 -- 172.21.15.90:0/1924992346 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(644..644 src has 467..644) ==== 7466+0+0 (unknown 495029275 0 0) 0x7f222c08a6c0 con 0x7f2238109da0 2024-04-14T15:23:00.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.455+0000 7f22357fa700 10 client.37749.objecter ms_dispatch 0x7f2238001550 osd_map(644..644 src has 467..644) 2024-04-14T15:23:00.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.455+0000 7f22357fa700 3 client.37749.objecter handle_osd_map got epochs [644,644] > 0 2024-04-14T15:23:00.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.455+0000 7f22357fa700 3 client.37749.objecter handle_osd_map decoding full epoch 644 2024-04-14T15:23:00.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.456+0000 7f22357fa700 10 client.37749.objecter checking command 1 2024-04-14T15:23:00.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.456+0000 7f22357fa700 20 client.37749.objecter _get_session s=0x7f222007d2c0 osd=7 3 2024-04-14T15:23:00.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.456+0000 7f22357fa700 20 client.37749.objecter put_session s=0x7f222007d2c0 osd=7 3 2024-04-14T15:23:00.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.456+0000 7f22357fa700 15 client.37749.objecter _session_command_op_remove -1 1 2024-04-14T15:23:00.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.456+0000 7f22357fa700 20 client.37749.objecter _get_session s=0x7f222007d2c0 osd=7 3 2024-04-14T15:23:00.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.456+0000 7f22357fa700 20 client.37749.objecter get_session s=0x7f222007d2c0 osd=7 3 2024-04-14T15:23:00.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.456+0000 7f22357fa700 15 client.37749.objecter _session_command_op_assign 7 1 2024-04-14T15:23:00.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.456+0000 7f22357fa700 20 client.37749.objecter put_session s=0x7f222007d2c0 osd=7 4 2024-04-14T15:23:00.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.456+0000 7f22357fa700 10 client.37749.objecter _send_command 1 2024-04-14T15:23:00.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.456+0000 7f22357fa700 1 -- 172.21.15.90:0/1924992346 --> v1:172.21.15.92:6812/1773706716 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2220081680 con 0x7f222007d970 2024-04-14T15:23:00.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.456+0000 7f22357fa700 20 client.37749.objecter dump_active .. 0 homeless 2024-04-14T15:23:00.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.456+0000 7f22357fa700 1 -- 172.21.15.90:0/1924992346 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_get_version_reply(handle=1 version=644) ==== 24+0+0 (unknown 458698460 0 0) 0x7f222c08ab90 con 0x7f2238109da0 2024-04-14T15:23:00.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.456+0000 7f22357fa700 10 client.37749.objecter ms_handle_connect 0x7f2220071390 2024-04-14T15:23:00.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.456+0000 7f22357fa700 10 client.37749.objecter ms_handle_connect 0x7f222007d970 2024-04-14T15:23:00.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.457+0000 7f22357fa700 1 -- 172.21.15.90:0/1924992346 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2220081680 con 0x7f222007d970 2024-04-14T15:23:00.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.457+0000 7f22357fa700 10 client.37749.objecter ms_dispatch 0x7f2238001550 command_reply(tid 1: 0 ) 2024-04-14T15:23:00.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.457+0000 7f22357fa700 10 client.37749.objecter _finish_command 1 = system:0 2024-04-14T15:23:00.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.457+0000 7f22357fa700 20 client.37749.objecter put_session s=0x7f222007d2c0 osd=7 4 2024-04-14T15:23:00.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.457+0000 7f22357fa700 15 client.37749.objecter _session_command_op_remove 7 1 2024-04-14T15:23:00.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.476+0000 7f223e911700 10 client.37749.objecter _submit_command 2 [{"prefix": "reset_purged_snaps_last"}] 2024-04-14T15:23:00.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.476+0000 7f223e911700 15 client.37749.objecter _session_command_op_assign -1 2 2024-04-14T15:23:00.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.476+0000 7f223e911700 20 client.37749.objecter _get_session s=0x7f222007d2c0 osd=7 3 2024-04-14T15:23:00.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.476+0000 7f223e911700 20 client.37749.objecter put_session s=0x7f222007d2c0 osd=7 3 2024-04-14T15:23:00.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.476+0000 7f223e911700 20 client.37749.objecter _get_session s=0x7f222007d2c0 osd=7 3 2024-04-14T15:23:00.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.476+0000 7f223e911700 15 client.37749.objecter _session_command_op_remove -1 2 2024-04-14T15:23:00.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.476+0000 7f223e911700 20 client.37749.objecter get_session s=0x7f222007d2c0 osd=7 3 2024-04-14T15:23:00.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.476+0000 7f223e911700 15 client.37749.objecter _session_command_op_assign 7 2 2024-04-14T15:23:00.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.476+0000 7f223e911700 20 client.37749.objecter put_session s=0x7f222007d2c0 osd=7 4 2024-04-14T15:23:00.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.476+0000 7f223e911700 10 client.37749.objecter _send_command 2 2024-04-14T15:23:00.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.476+0000 7f223e911700 1 -- 172.21.15.90:0/1924992346 --> v1:172.21.15.92:6812/1773706716 -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) -- 0x7f22240011f0 con 0x7f222007d970 2024-04-14T15:23:00.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.477+0000 7f22357fa700 1 -- 172.21.15.90:0/1924992346 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 2: 0 ) ==== 8+0+0 (unknown 0 0 0) 0x7f22240011f0 con 0x7f222007d970 2024-04-14T15:23:00.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.477+0000 7f22357fa700 10 client.37749.objecter ms_dispatch 0x7f2238001550 command_reply(tid 2: 0 ) 2024-04-14T15:23:00.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.477+0000 7f22357fa700 10 client.37749.objecter _finish_command 2 = system:0 2024-04-14T15:23:00.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.477+0000 7f22357fa700 20 client.37749.objecter put_session s=0x7f222007d2c0 osd=7 4 2024-04-14T15:23:00.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.477+0000 7f22357fa700 15 client.37749.objecter _session_command_op_remove 7 2 2024-04-14T15:23:00.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.477+0000 7f223e911700 10 librados: watch_flush enter 2024-04-14T15:23:00.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.477+0000 7f223e911700 10 librados: watch_flush exit 2024-04-14T15:23:00.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.477+0000 7f223e911700 10 client.37749.objecter close_session for osd.7 2024-04-14T15:23:00.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.477+0000 7f223e911700 1 -- 172.21.15.90:0/1924992346 >> v1:172.21.15.92:6812/1773706716 conn(0x7f222007d970 legacy=0x7f222007fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:00.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.477+0000 7f223e911700 20 client.37749.objecter put_session s=0x7f222007d2c0 osd=7 1 2024-04-14T15:23:00.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.477+0000 7f223e911700 20 client.37749.objecter shutdown clearing up homeless session... 2024-04-14T15:23:00.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.477+0000 7f223e911700 10 client.37749.objecter successfully canceled tick 2024-04-14T15:23:00.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.477+0000 7f223e911700 1 -- 172.21.15.90:0/1924992346 >> v1:172.21.15.90:6800/420935427 conn(0x7f2220071390 legacy=0x7f2220073850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:00.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.477+0000 7f223e911700 1 -- 172.21.15.90:0/1924992346 >> v1:172.21.15.90:6789/0 conn(0x7f2238109da0 legacy=0x7f22381a2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:00.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.477+0000 7f223ceae700 1 -- 172.21.15.90:0/1924992346 reap_dead start 2024-04-14T15:23:00.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.477+0000 7f223e911700 1 -- 172.21.15.90:0/1924992346 shutdown_connections 2024-04-14T15:23:00.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.477+0000 7f223e911700 1 -- 172.21.15.90:0/1924992346 >> 172.21.15.90:0/1924992346 conn(0x7f2238076aa0 msgr2=0x7f2238108de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:23:00.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.477+0000 7f223e911700 1 -- 172.21.15.90:0/1924992346 shutdown_connections 2024-04-14T15:23:00.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.477+0000 7f223e911700 1 -- 172.21.15.90:0/1924992346 wait complete. 2024-04-14T15:23:00.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.477+0000 7f223e911700 1 librados: shutdown 2024-04-14T15:23:00.489 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:23:00.512 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:00.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:00.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:23:00.614 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:23:00.637 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:00.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:00.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:23:00.739 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:23:00.762 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:00.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:00.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:23:00.864 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:00.888 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:00.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:00.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:00.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.984+0000 7fc1317fa700 1 -- 172.21.15.90:0/3145247990 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v645) ==== 101+0+0 (unknown 258159371 0 0) 0x7fc124059d10 con 0x7fc134108880 2024-04-14T15:23:00.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.987+0000 7fc13a879700 10 librados: watch_flush enter 2024-04-14T15:23:00.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.987+0000 7fc13a879700 10 librados: watch_flush exit 2024-04-14T15:23:00.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.987+0000 7fc13a879700 20 client.49570.objecter shutdown clearing up homeless session... 2024-04-14T15:23:00.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.987+0000 7fc13a879700 10 client.49570.objecter successfully canceled tick 2024-04-14T15:23:00.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.988+0000 7fc13a879700 1 -- 172.21.15.90:0/3145247990 >> v1:172.21.15.90:6800/420935427 conn(0x7fc11c071450 legacy=0x7fc11c073910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:00.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.988+0000 7fc13a879700 1 -- 172.21.15.90:0/3145247990 >> v1:172.21.15.92:6789/0 conn(0x7fc134108880 legacy=0x7fc13410d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:00.990 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:23:00.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.989+0000 7fc13a879700 1 -- 172.21.15.90:0/3145247990 shutdown_connections 2024-04-14T15:23:00.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.989+0000 7fc13a879700 1 -- 172.21.15.90:0/3145247990 >> 172.21.15.90:0/3145247990 conn(0x7fc1340fe3d0 msgr2=0x7fc134112880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:23:00.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.989+0000 7fc13a879700 1 -- 172.21.15.90:0/3145247990 shutdown_connections 2024-04-14T15:23:00.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.990+0000 7fc13a879700 1 -- 172.21.15.90:0/3145247990 wait complete. 2024-04-14T15:23:00.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:00.990+0000 7fc13a879700 1 librados: shutdown 2024-04-14T15:23:00.992 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is unset 2024-04-14T15:23:01.018 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:01.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:01.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:23:01.120 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:01.144 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:01.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:01.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:01.246 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:23:01.261 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:01 smithi092 ceph-mon[33966]: 4.19 scrub starts 2024-04-14T15:23:01.261 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:01 smithi092 ceph-mon[33966]: 4.19 scrub ok 2024-04-14T15:23:01.261 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:01 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3145247990' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:23:01.261 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:01 smithi092 ceph-mon[33966]: from='client.49570 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:23:01.261 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:01 smithi092 ceph-mon[33966]: 1.b scrub starts 2024-04-14T15:23:01.261 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:01 smithi092 ceph-mon[33966]: 1.b scrub ok 2024-04-14T15:23:01.261 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:01 smithi092 ceph-mon[33966]: 5.2 scrub starts 2024-04-14T15:23:01.262 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:01 smithi092 ceph-mon[33966]: 5.2 scrub ok 2024-04-14T15:23:01.262 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:01 smithi092 ceph-mon[33966]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:23:01.262 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:01 smithi092 ceph-mon[33966]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-14T15:23:01.262 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:01 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi090-93652-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:23:01.262 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:01 smithi092 ceph-mon[33966]: from='client.49570 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-14T15:23:01.262 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:01 smithi092 ceph-mon[33966]: osdmap e645: 8 total, 8 up, 7 in 2024-04-14T15:23:01.262 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:01 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi090-93652-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:23:01.262 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:01 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi090-93652-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:23:01.270 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:01.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:01.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:23:01.371 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:23:01.386 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-04-14T15:23:01.387 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:23:01.395 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:01.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:01.396 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:23:01.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:01 smithi090 ceph-mon[41670]: 4.19 scrub starts 2024-04-14T15:23:01.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:01 smithi090 ceph-mon[41670]: 4.19 scrub ok 2024-04-14T15:23:01.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:01 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3145247990' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:23:01.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:01 smithi090 ceph-mon[41670]: from='client.49570 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:23:01.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:01 smithi090 ceph-mon[41670]: 1.b scrub starts 2024-04-14T15:23:01.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:01 smithi090 ceph-mon[41670]: 1.b scrub ok 2024-04-14T15:23:01.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:01 smithi090 ceph-mon[41670]: 5.2 scrub starts 2024-04-14T15:23:01.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:01 smithi090 ceph-mon[41670]: 5.2 scrub ok 2024-04-14T15:23:01.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:01 smithi090 ceph-mon[41670]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:23:01.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:01 smithi090 ceph-mon[41670]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-14T15:23:01.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:01 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi090-93652-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:23:01.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:01 smithi090 ceph-mon[41670]: from='client.49570 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-14T15:23:01.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:01 smithi090 ceph-mon[41670]: osdmap e645: 8 total, 8 up, 7 in 2024-04-14T15:23:01.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:01 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi090-93652-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:23:01.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:01 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi090-93652-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:23:01.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:01 smithi090 ceph-mon[30399]: 4.19 scrub starts 2024-04-14T15:23:01.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:01 smithi090 ceph-mon[30399]: 4.19 scrub ok 2024-04-14T15:23:01.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:01 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3145247990' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:23:01.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:01 smithi090 ceph-mon[30399]: from='client.49570 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:23:01.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:01 smithi090 ceph-mon[30399]: 1.b scrub starts 2024-04-14T15:23:01.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:01 smithi090 ceph-mon[30399]: 1.b scrub ok 2024-04-14T15:23:01.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:01 smithi090 ceph-mon[30399]: 5.2 scrub starts 2024-04-14T15:23:01.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:01 smithi090 ceph-mon[30399]: 5.2 scrub ok 2024-04-14T15:23:01.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:01 smithi090 ceph-mon[30399]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:23:01.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:01 smithi090 ceph-mon[30399]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-14T15:23:01.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:01 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi090-93652-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-14T15:23:01.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:01 smithi090 ceph-mon[30399]: from='client.49570 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-14T15:23:01.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:01 smithi090 ceph-mon[30399]: osdmap e645: 8 total, 8 up, 7 in 2024-04-14T15:23:01.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:01 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi090-93652-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:23:01.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:01 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi090-93652-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:23:01.497 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:23:01.520 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:01.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:01.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:23:01.622 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:23:01.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:01.646 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:01.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:23:01.747 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:23:01.771 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:01.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:01.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:23:01.872 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:23:01.896 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:01.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:01.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:23:01.997 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:23:02.021 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:02.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:02.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:23:02.058 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-14T15:23:02.107 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:02 smithi090 ceph-mon[41670]: pgmap v948: 177 pgs: 1 active+remapped+backfill_toofull, 176 active+clean; 582 KiB data, 578 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 op/s; 2/597 objects misplaced (0.335%) 2024-04-14T15:23:02.107 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:02 smithi090 ceph-mon[41670]: 6.18 scrub starts 2024-04-14T15:23:02.107 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:02 smithi090 ceph-mon[41670]: 6.18 scrub ok 2024-04-14T15:23:02.107 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:02 smithi090 ceph-mon[41670]: osdmap e646: 8 total, 8 up, 7 in 2024-04-14T15:23:02.107 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:02 smithi090 ceph-mon[30399]: pgmap v948: 177 pgs: 1 active+remapped+backfill_toofull, 176 active+clean; 582 KiB data, 578 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 op/s; 2/597 objects misplaced (0.335%) 2024-04-14T15:23:02.107 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:02 smithi090 ceph-mon[30399]: 6.18 scrub starts 2024-04-14T15:23:02.107 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:02 smithi090 ceph-mon[30399]: 6.18 scrub ok 2024-04-14T15:23:02.107 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:02 smithi090 ceph-mon[30399]: osdmap e646: 8 total, 8 up, 7 in 2024-04-14T15:23:02.123 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:23:02.146 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:02.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:02.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:23:02.185 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:23:02.248 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:23:02.271 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:02.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:02.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:23:02.334 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:23:02.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:02 smithi092 ceph-mon[33966]: pgmap v948: 177 pgs: 1 active+remapped+backfill_toofull, 176 active+clean; 582 KiB data, 578 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 op/s; 2/597 objects misplaced (0.335%) 2024-04-14T15:23:02.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:02 smithi092 ceph-mon[33966]: 6.18 scrub starts 2024-04-14T15:23:02.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:02 smithi092 ceph-mon[33966]: 6.18 scrub ok 2024-04-14T15:23:02.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:02 smithi092 ceph-mon[33966]: osdmap e646: 8 total, 8 up, 7 in 2024-04-14T15:23:02.373 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:23:02.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:02.397 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:02.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:23:02.498 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:02.523 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:02.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:02.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:02.624 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:23:02.648 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:02.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:02.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:23:02.750 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:23:02.774 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:02.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:02.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:23:02.876 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:23:02.900 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:02.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:02.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:23:03.001 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:23:03.024 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:03.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:03.025 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:23:03.126 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:23:03.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:03.150 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:03.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:23:03.250 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:23:03.274 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:03.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:03.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:23:03.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:03 smithi092 ceph-mon[33966]: 6.5 scrub starts 2024-04-14T15:23:03.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:03 smithi092 ceph-mon[33966]: 6.5 scrub ok 2024-04-14T15:23:03.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:03 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi090-93652-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi090-93652-111"}]': finished 2024-04-14T15:23:03.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:03 smithi092 ceph-mon[33966]: osdmap e647: 8 total, 8 up, 7 in 2024-04-14T15:23:03.375 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:23:03.383 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:03 smithi090 ceph-mon[41670]: 6.5 scrub starts 2024-04-14T15:23:03.384 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:03 smithi090 ceph-mon[41670]: 6.5 scrub ok 2024-04-14T15:23:03.384 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:03 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi090-93652-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi090-93652-111"}]': finished 2024-04-14T15:23:03.384 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:03 smithi090 ceph-mon[41670]: osdmap e647: 8 total, 8 up, 7 in 2024-04-14T15:23:03.384 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:03 smithi090 ceph-mon[30399]: 6.5 scrub starts 2024-04-14T15:23:03.384 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:03 smithi090 ceph-mon[30399]: 6.5 scrub ok 2024-04-14T15:23:03.384 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:03 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi090-93652-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi090-93652-111"}]': finished 2024-04-14T15:23:03.384 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:03 smithi090 ceph-mon[30399]: osdmap e647: 8 total, 8 up, 7 in 2024-04-14T15:23:03.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:03.399 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:03.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:23:03.500 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:23:03.526 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:03.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:03.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:23:03.628 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:23:03.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:03.652 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:03.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:23:03.753 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:23:03.777 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:03.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:03.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:23:03.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.871+0000 7f87bd598700 1 -- 172.21.15.90:0/1348332505 >> v1:172.21.15.90:6789/0 conn(0x7f87b810f4d0 legacy=0x7f87b8111990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:03.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.871+0000 7f87bd598700 1 -- 172.21.15.90:0/1348332505 shutdown_connections 2024-04-14T15:23:03.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.871+0000 7f87bd598700 1 -- 172.21.15.90:0/1348332505 >> 172.21.15.90:0/1348332505 conn(0x7f87b80fd290 msgr2=0x7f87b80ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:23:03.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.871+0000 7f87bd598700 1 -- 172.21.15.90:0/1348332505 shutdown_connections 2024-04-14T15:23:03.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.871+0000 7f87bd598700 1 -- 172.21.15.90:0/1348332505 wait complete. 2024-04-14T15:23:03.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.871+0000 7f87bd598700 1 librados: starting msgr at 2024-04-14T15:23:03.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.871+0000 7f87bd598700 1 librados: starting objecter 2024-04-14T15:23:03.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.871+0000 7f87bd598700 1 Processor -- start 2024-04-14T15:23:03.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.871+0000 7f87bd598700 1 -- start start 2024-04-14T15:23:03.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.871+0000 7f87bd598700 1 librados: setting wanted keys 2024-04-14T15:23:03.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.871+0000 7f87bd598700 1 librados: calling monclient init 2024-04-14T15:23:03.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.872+0000 7f87bd598700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f87b8103290 con 0x7f87b8103a40 2024-04-14T15:23:03.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.872+0000 7f87bd598700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f87b81034a0 con 0x7f87b810f4d0 2024-04-14T15:23:03.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.872+0000 7f87bd598700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f87b81ac470 con 0x7f87b810b930 2024-04-14T15:23:03.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.872+0000 7f87b6ffd700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f87b8103a40 0x7f87b81016a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:43158/0 (socket says 172.21.15.90:43158) 2024-04-14T15:23:03.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.872+0000 7f87b6ffd700 1 -- 172.21.15.90:0/2385759697 learned_addr learned my addr 172.21.15.90:0/2385759697 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:23:03.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.872+0000 7f879ffff700 10 client.?.objecter ms_handle_connect 0x7f87b8103a40 2024-04-14T15:23:03.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.872+0000 7f879ffff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:23:03.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.872+0000 7f879ffff700 10 client.?.objecter ms_handle_connect 0x7f87b810b930 2024-04-14T15:23:03.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.872+0000 7f879ffff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:23:03.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.872+0000 7f879ffff700 10 client.?.objecter ms_handle_connect 0x7f87b810f4d0 2024-04-14T15:23:03.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.872+0000 7f879ffff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:23:03.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.872+0000 7f879ffff700 1 -- 172.21.15.90:0/2385759697 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2639876716 0 0) 0x7f87b8103290 con 0x7f87b8103a40 2024-04-14T15:23:03.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.872+0000 7f879ffff700 1 -- 172.21.15.90:0/2385759697 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f87a00031f0 con 0x7f87b8103a40 2024-04-14T15:23:03.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.872+0000 7f879ffff700 1 -- 172.21.15.90:0/2385759697 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3121103528 0 0) 0x7f87b81034a0 con 0x7f87b810f4d0 2024-04-14T15:23:03.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.872+0000 7f879ffff700 1 -- 172.21.15.90:0/2385759697 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f87b8103290 con 0x7f87b810f4d0 2024-04-14T15:23:03.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.872+0000 7f879ffff700 1 -- 172.21.15.90:0/2385759697 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3764143388 0 0) 0x7f87b81ac470 con 0x7f87b810b930 2024-04-14T15:23:03.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.872+0000 7f879ffff700 1 -- 172.21.15.90:0/2385759697 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f87b81034a0 con 0x7f87b810b930 2024-04-14T15:23:03.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.872+0000 7f879ffff700 1 -- 172.21.15.90:0/2385759697 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4030923013 0 0) 0x7f87b8103290 con 0x7f87b810f4d0 2024-04-14T15:23:03.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.872+0000 7f879ffff700 1 -- 172.21.15.90:0/2385759697 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f87b81ac470 con 0x7f87b810f4d0 2024-04-14T15:23:03.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.872+0000 7f879ffff700 1 -- 172.21.15.90:0/2385759697 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f87b00017b0 con 0x7f87b810f4d0 2024-04-14T15:23:03.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.872+0000 7f879ffff700 1 -- 172.21.15.90:0/2385759697 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1793413587 0 0) 0x7f87a00031f0 con 0x7f87b8103a40 2024-04-14T15:23:03.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.872+0000 7f879ffff700 1 -- 172.21.15.90:0/2385759697 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f87b8103290 con 0x7f87b8103a40 2024-04-14T15:23:03.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.872+0000 7f879ffff700 1 -- 172.21.15.90:0/2385759697 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f87ac001830 con 0x7f87b8103a40 2024-04-14T15:23:03.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.872+0000 7f879ffff700 1 -- 172.21.15.90:0/2385759697 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3126730611 0 0) 0x7f87b81034a0 con 0x7f87b810b930 2024-04-14T15:23:03.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.872+0000 7f879ffff700 1 -- 172.21.15.90:0/2385759697 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f87a00031f0 con 0x7f87b810b930 2024-04-14T15:23:03.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.872+0000 7f879ffff700 1 -- 172.21.15.90:0/2385759697 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f87a80017f0 con 0x7f87b810b930 2024-04-14T15:23:03.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.872+0000 7f879ffff700 1 -- 172.21.15.90:0/2385759697 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2598936899 0 0) 0x7f87b81ac470 con 0x7f87b810f4d0 2024-04-14T15:23:03.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.872+0000 7f879ffff700 1 -- 172.21.15.90:0/2385759697 >> v1:172.21.15.90:6790/0 conn(0x7f87b810b930 legacy=0x7f87b8101db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:03.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.873+0000 7f879ffff700 1 -- 172.21.15.90:0/2385759697 >> v1:172.21.15.90:6789/0 conn(0x7f87b8103a40 legacy=0x7f87b81016a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:03.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.873+0000 7f879ffff700 1 -- 172.21.15.90:0/2385759697 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87b81ad650 con 0x7f87b810f4d0 2024-04-14T15:23:03.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.873+0000 7f87bd598700 1 -- 172.21.15.90:0/2385759697 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f87b81ac6a0 con 0x7f87b810f4d0 2024-04-14T15:23:03.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.873+0000 7f87bd598700 10 client.49588.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:23:03.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.873+0000 7f87bd598700 1 -- 172.21.15.90:0/2385759697 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f87b81acb60 con 0x7f87b810f4d0 2024-04-14T15:23:03.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.873+0000 7f87bd598700 1 librados: init done 2024-04-14T15:23:03.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.873+0000 7f879ffff700 1 -- 172.21.15.90:0/2385759697 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f87b0001ef0 con 0x7f87b810f4d0 2024-04-14T15:23:03.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.873+0000 7f879ffff700 1 -- 172.21.15.90:0/2385759697 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f87b0005400 con 0x7f87b810f4d0 2024-04-14T15:23:03.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.874+0000 7f87bd598700 1 -- 172.21.15.90:0/2385759697 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f87a4005100 con 0x7f87b810f4d0 2024-04-14T15:23:03.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.874+0000 7f879ffff700 1 -- 172.21.15.90:0/2385759697 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f87b0005670 con 0x7f87b810f4d0 2024-04-14T15:23:03.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.875+0000 7f879ffff700 1 -- 172.21.15.90:0/2385759697 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(647..647 src has 467..647) ==== 8325+0+0 (unknown 3606969390 0 0) 0x7f87b0089c80 con 0x7f87b810f4d0 2024-04-14T15:23:03.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.875+0000 7f879ffff700 10 client.49588.objecter ms_dispatch 0x7f87b8001550 osd_map(647..647 src has 467..647) 2024-04-14T15:23:03.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.875+0000 7f879ffff700 3 client.49588.objecter handle_osd_map got epochs [647,647] > 0 2024-04-14T15:23:03.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.875+0000 7f879ffff700 3 client.49588.objecter handle_osd_map decoding full epoch 647 2024-04-14T15:23:03.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.875+0000 7f879ffff700 20 client.49588.objecter dump_active .. 0 homeless 2024-04-14T15:23:03.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.875+0000 7f879ffff700 1 -- 172.21.15.90:0/2385759697 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v647) ==== 54+0+16 (unknown 3188469777 0 2420000192) 0x7f87b008c0b0 con 0x7f87b810f4d0 2024-04-14T15:23:03.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.875+0000 7f879ffff700 10 client.49588.objecter ms_handle_connect 0x7f87a0071470 2024-04-14T15:23:03.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.875+0000 7f87bd598700 10 client.49588.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:23:03.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.875+0000 7f87bd598700 15 client.49588.objecter _session_command_op_assign -1 1 2024-04-14T15:23:03.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.875+0000 7f87bd598700 20 client.49588.objecter _get_session s=0x7f87a40053e0 osd=0 3 2024-04-14T15:23:03.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.875+0000 7f87bd598700 20 client.49588.objecter put_session s=0x7f87a40053e0 osd=0 3 2024-04-14T15:23:03.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.875+0000 7f87bd598700 20 client.49588.objecter _get_session s=0x7f87a40053e0 osd=0 3 2024-04-14T15:23:03.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.875+0000 7f87bd598700 15 client.49588.objecter _session_command_op_remove -1 1 2024-04-14T15:23:03.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.875+0000 7f87bd598700 20 client.49588.objecter get_session s=0x7f87a40053e0 osd=0 3 2024-04-14T15:23:03.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.875+0000 7f87bd598700 15 client.49588.objecter _session_command_op_assign 0 1 2024-04-14T15:23:03.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.875+0000 7f87bd598700 20 client.49588.objecter put_session s=0x7f87a40053e0 osd=0 4 2024-04-14T15:23:03.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.875+0000 7f87bd598700 10 client.49588.objecter _send_command 1 2024-04-14T15:23:03.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.875+0000 7f87bd598700 1 -- 172.21.15.90:0/2385759697 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f87a4004b50 con 0x7f87a4004730 2024-04-14T15:23:03.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.875+0000 7f879ffff700 10 client.49588.objecter ms_handle_connect 0x7f87a4004730 2024-04-14T15:23:03.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.876+0000 7f879ffff700 1 -- 172.21.15.90:0/2385759697 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f87a4004b50 con 0x7f87a4004730 2024-04-14T15:23:03.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.876+0000 7f879ffff700 10 client.49588.objecter ms_dispatch 0x7f87b8001550 command_reply(tid 1: 0 ) 2024-04-14T15:23:03.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.876+0000 7f879ffff700 10 client.49588.objecter _finish_command 1 = system:0 2024-04-14T15:23:03.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.876+0000 7f879ffff700 20 client.49588.objecter put_session s=0x7f87a40053e0 osd=0 4 2024-04-14T15:23:03.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.876+0000 7f879ffff700 15 client.49588.objecter _session_command_op_remove 0 1 2024-04-14T15:23:03.880 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:23:03.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.893+0000 7f87bd598700 10 client.49588.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:23:03.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.893+0000 7f87bd598700 15 client.49588.objecter _session_command_op_assign -1 2 2024-04-14T15:23:03.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.893+0000 7f87bd598700 20 client.49588.objecter _get_session s=0x7f87a40053e0 osd=0 3 2024-04-14T15:23:03.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.893+0000 7f87bd598700 20 client.49588.objecter put_session s=0x7f87a40053e0 osd=0 3 2024-04-14T15:23:03.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.893+0000 7f87bd598700 20 client.49588.objecter _get_session s=0x7f87a40053e0 osd=0 3 2024-04-14T15:23:03.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.893+0000 7f87bd598700 15 client.49588.objecter _session_command_op_remove -1 2 2024-04-14T15:23:03.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.893+0000 7f87bd598700 20 client.49588.objecter get_session s=0x7f87a40053e0 osd=0 3 2024-04-14T15:23:03.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.893+0000 7f87bd598700 15 client.49588.objecter _session_command_op_assign 0 2 2024-04-14T15:23:03.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.893+0000 7f87bd598700 20 client.49588.objecter put_session s=0x7f87a40053e0 osd=0 4 2024-04-14T15:23:03.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.893+0000 7f87bd598700 10 client.49588.objecter _send_command 2 2024-04-14T15:23:03.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.893+0000 7f87bd598700 1 -- 172.21.15.90:0/2385759697 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f87a4007b00 con 0x7f87a4004730 2024-04-14T15:23:03.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.894+0000 7f879ffff700 1 -- 172.21.15.90:0/2385759697 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f87a4007b00 con 0x7f87a4004730 2024-04-14T15:23:03.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.894+0000 7f879ffff700 10 client.49588.objecter ms_dispatch 0x7f87b8001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:23:03.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.894+0000 7f879ffff700 10 client.49588.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:23:03.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.894+0000 7f879ffff700 20 client.49588.objecter put_session s=0x7f87a40053e0 osd=0 4 2024-04-14T15:23:03.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.894+0000 7f879ffff700 15 client.49588.objecter _session_command_op_remove 0 2 2024-04-14T15:23:03.895 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:23:03.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.895+0000 7f87bd598700 10 client.49588.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:23:03.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.895+0000 7f87bd598700 15 client.49588.objecter _session_command_op_assign -1 3 2024-04-14T15:23:03.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.895+0000 7f87bd598700 20 client.49588.objecter _get_session s=0x7f87a4004130 osd=1 3 2024-04-14T15:23:03.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.895+0000 7f87bd598700 20 client.49588.objecter put_session s=0x7f87a4004130 osd=1 3 2024-04-14T15:23:03.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.895+0000 7f87bd598700 20 client.49588.objecter _get_session s=0x7f87a4004130 osd=1 3 2024-04-14T15:23:03.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.895+0000 7f87bd598700 15 client.49588.objecter _session_command_op_remove -1 3 2024-04-14T15:23:03.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.895+0000 7f87bd598700 20 client.49588.objecter get_session s=0x7f87a4004130 osd=1 3 2024-04-14T15:23:03.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.895+0000 7f87bd598700 15 client.49588.objecter _session_command_op_assign 1 3 2024-04-14T15:23:03.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.895+0000 7f87bd598700 20 client.49588.objecter put_session s=0x7f87a4004130 osd=1 4 2024-04-14T15:23:03.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.895+0000 7f87bd598700 10 client.49588.objecter _send_command 3 2024-04-14T15:23:03.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.895+0000 7f87bd598700 1 -- 172.21.15.90:0/2385759697 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f87a4002160 con 0x7f87a40042a0 2024-04-14T15:23:03.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.895+0000 7f879ffff700 10 client.49588.objecter ms_handle_connect 0x7f87a40042a0 2024-04-14T15:23:03.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.896+0000 7f879ffff700 1 -- 172.21.15.90:0/2385759697 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f87a4002160 con 0x7f87a40042a0 2024-04-14T15:23:03.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.896+0000 7f879ffff700 10 client.49588.objecter ms_dispatch 0x7f87b8001550 command_reply(tid 3: 0 ) 2024-04-14T15:23:03.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.896+0000 7f879ffff700 10 client.49588.objecter _finish_command 3 = system:0 2024-04-14T15:23:03.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.896+0000 7f879ffff700 20 client.49588.objecter put_session s=0x7f87a4004130 osd=1 4 2024-04-14T15:23:03.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.896+0000 7f879ffff700 15 client.49588.objecter _session_command_op_remove 1 3 2024-04-14T15:23:03.903 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:03.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:03.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:23:03.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.912+0000 7f87bd598700 10 client.49588.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:23:03.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.912+0000 7f87bd598700 15 client.49588.objecter _session_command_op_assign -1 4 2024-04-14T15:23:03.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.912+0000 7f87bd598700 20 client.49588.objecter _get_session s=0x7f87a4004130 osd=1 3 2024-04-14T15:23:03.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.912+0000 7f87bd598700 20 client.49588.objecter put_session s=0x7f87a4004130 osd=1 3 2024-04-14T15:23:03.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.912+0000 7f87bd598700 20 client.49588.objecter _get_session s=0x7f87a4004130 osd=1 3 2024-04-14T15:23:03.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.912+0000 7f87bd598700 15 client.49588.objecter _session_command_op_remove -1 4 2024-04-14T15:23:03.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.912+0000 7f87bd598700 20 client.49588.objecter get_session s=0x7f87a4004130 osd=1 3 2024-04-14T15:23:03.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.912+0000 7f87bd598700 15 client.49588.objecter _session_command_op_assign 1 4 2024-04-14T15:23:03.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.912+0000 7f87bd598700 20 client.49588.objecter put_session s=0x7f87a4004130 osd=1 4 2024-04-14T15:23:03.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.912+0000 7f87bd598700 10 client.49588.objecter _send_command 4 2024-04-14T15:23:03.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.912+0000 7f87bd598700 1 -- 172.21.15.90:0/2385759697 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f87a4001600 con 0x7f87a40042a0 2024-04-14T15:23:03.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.914+0000 7f879ffff700 1 -- 172.21.15.90:0/2385759697 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f87a4001600 con 0x7f87a40042a0 2024-04-14T15:23:03.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.914+0000 7f879ffff700 10 client.49588.objecter ms_dispatch 0x7f87b8001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:23:03.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.914+0000 7f879ffff700 10 client.49588.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:23:03.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.914+0000 7f879ffff700 20 client.49588.objecter put_session s=0x7f87a4004130 osd=1 4 2024-04-14T15:23:03.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.914+0000 7f879ffff700 15 client.49588.objecter _session_command_op_remove 1 4 2024-04-14T15:23:03.915 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:23:03.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.914+0000 7f87bd598700 10 client.49588.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:23:03.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.914+0000 7f87bd598700 15 client.49588.objecter _session_command_op_assign -1 5 2024-04-14T15:23:03.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.914+0000 7f87bd598700 20 client.49588.objecter _get_session s=0x7f87a4001a00 osd=2 3 2024-04-14T15:23:03.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.914+0000 7f87bd598700 20 client.49588.objecter put_session s=0x7f87a4001a00 osd=2 3 2024-04-14T15:23:03.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.914+0000 7f87bd598700 20 client.49588.objecter _get_session s=0x7f87a4001a00 osd=2 3 2024-04-14T15:23:03.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.914+0000 7f87bd598700 15 client.49588.objecter _session_command_op_remove -1 5 2024-04-14T15:23:03.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.914+0000 7f87bd598700 20 client.49588.objecter get_session s=0x7f87a4001a00 osd=2 3 2024-04-14T15:23:03.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.914+0000 7f87bd598700 15 client.49588.objecter _session_command_op_assign 2 5 2024-04-14T15:23:03.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.914+0000 7f87bd598700 20 client.49588.objecter put_session s=0x7f87a4001a00 osd=2 4 2024-04-14T15:23:03.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.914+0000 7f87bd598700 10 client.49588.objecter _send_command 5 2024-04-14T15:23:03.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.914+0000 7f87bd598700 1 -- 172.21.15.90:0/2385759697 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f87a4011010 con 0x7f87a4008990 2024-04-14T15:23:03.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.915+0000 7f879ffff700 10 client.49588.objecter ms_handle_connect 0x7f87a4008990 2024-04-14T15:23:03.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.915+0000 7f879ffff700 1 -- 172.21.15.90:0/2385759697 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f87a4011010 con 0x7f87a4008990 2024-04-14T15:23:03.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.915+0000 7f879ffff700 10 client.49588.objecter ms_dispatch 0x7f87b8001550 command_reply(tid 5: 0 ) 2024-04-14T15:23:03.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.915+0000 7f879ffff700 10 client.49588.objecter _finish_command 5 = system:0 2024-04-14T15:23:03.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.915+0000 7f879ffff700 20 client.49588.objecter put_session s=0x7f87a4001a00 osd=2 4 2024-04-14T15:23:03.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.915+0000 7f879ffff700 15 client.49588.objecter _session_command_op_remove 2 5 2024-04-14T15:23:03.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.931+0000 7f87bd598700 10 client.49588.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:23:03.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.931+0000 7f87bd598700 15 client.49588.objecter _session_command_op_assign -1 6 2024-04-14T15:23:03.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.931+0000 7f87bd598700 20 client.49588.objecter _get_session s=0x7f87a4001a00 osd=2 3 2024-04-14T15:23:03.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.931+0000 7f87bd598700 20 client.49588.objecter put_session s=0x7f87a4001a00 osd=2 3 2024-04-14T15:23:03.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.931+0000 7f87bd598700 20 client.49588.objecter _get_session s=0x7f87a4001a00 osd=2 3 2024-04-14T15:23:03.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.931+0000 7f87bd598700 15 client.49588.objecter _session_command_op_remove -1 6 2024-04-14T15:23:03.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.931+0000 7f87bd598700 20 client.49588.objecter get_session s=0x7f87a4001a00 osd=2 3 2024-04-14T15:23:03.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.931+0000 7f87bd598700 15 client.49588.objecter _session_command_op_assign 2 6 2024-04-14T15:23:03.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.931+0000 7f87bd598700 20 client.49588.objecter put_session s=0x7f87a4001a00 osd=2 4 2024-04-14T15:23:03.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.931+0000 7f87bd598700 10 client.49588.objecter _send_command 6 2024-04-14T15:23:03.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.931+0000 7f87bd598700 1 -- 172.21.15.90:0/2385759697 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f87a4007fc0 con 0x7f87a4008990 2024-04-14T15:23:03.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.933+0000 7f879ffff700 1 -- 172.21.15.90:0/2385759697 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f87a4007fc0 con 0x7f87a4008990 2024-04-14T15:23:03.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.933+0000 7f879ffff700 10 client.49588.objecter ms_dispatch 0x7f87b8001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:23:03.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.933+0000 7f879ffff700 10 client.49588.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:23:03.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.933+0000 7f879ffff700 20 client.49588.objecter put_session s=0x7f87a4001a00 osd=2 4 2024-04-14T15:23:03.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.933+0000 7f879ffff700 15 client.49588.objecter _session_command_op_remove 2 6 2024-04-14T15:23:03.934 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:23:03.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.933+0000 7f87bd598700 10 client.49588.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:23:03.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.933+0000 7f87bd598700 15 client.49588.objecter _session_command_op_assign -1 7 2024-04-14T15:23:03.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.933+0000 7f87bd598700 20 client.49588.objecter _get_session s=0x7f87a400cd30 osd=3 3 2024-04-14T15:23:03.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.933+0000 7f87bd598700 20 client.49588.objecter put_session s=0x7f87a400cd30 osd=3 3 2024-04-14T15:23:03.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.933+0000 7f87bd598700 20 client.49588.objecter _get_session s=0x7f87a400cd30 osd=3 3 2024-04-14T15:23:03.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.933+0000 7f87bd598700 15 client.49588.objecter _session_command_op_remove -1 7 2024-04-14T15:23:03.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.933+0000 7f87bd598700 20 client.49588.objecter get_session s=0x7f87a400cd30 osd=3 3 2024-04-14T15:23:03.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.933+0000 7f87bd598700 15 client.49588.objecter _session_command_op_assign 3 7 2024-04-14T15:23:03.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.933+0000 7f87bd598700 20 client.49588.objecter put_session s=0x7f87a400cd30 osd=3 4 2024-04-14T15:23:03.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.933+0000 7f87bd598700 10 client.49588.objecter _send_command 7 2024-04-14T15:23:03.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.933+0000 7f87bd598700 1 -- 172.21.15.90:0/2385759697 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f87a400d510 con 0x7f87a40111f0 2024-04-14T15:23:03.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.934+0000 7f879ffff700 10 client.49588.objecter ms_handle_connect 0x7f87a40111f0 2024-04-14T15:23:03.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.934+0000 7f879ffff700 1 -- 172.21.15.90:0/2385759697 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f87a400d510 con 0x7f87a40111f0 2024-04-14T15:23:03.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.934+0000 7f879ffff700 10 client.49588.objecter ms_dispatch 0x7f87b8001550 command_reply(tid 7: 0 ) 2024-04-14T15:23:03.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.934+0000 7f879ffff700 10 client.49588.objecter _finish_command 7 = system:0 2024-04-14T15:23:03.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.934+0000 7f879ffff700 20 client.49588.objecter put_session s=0x7f87a400cd30 osd=3 4 2024-04-14T15:23:03.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.934+0000 7f879ffff700 15 client.49588.objecter _session_command_op_remove 3 7 2024-04-14T15:23:03.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.951+0000 7f87bd598700 10 client.49588.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:23:03.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.951+0000 7f87bd598700 15 client.49588.objecter _session_command_op_assign -1 8 2024-04-14T15:23:03.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.951+0000 7f87bd598700 20 client.49588.objecter _get_session s=0x7f87a400cd30 osd=3 3 2024-04-14T15:23:03.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.951+0000 7f87bd598700 20 client.49588.objecter put_session s=0x7f87a400cd30 osd=3 3 2024-04-14T15:23:03.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.951+0000 7f87bd598700 20 client.49588.objecter _get_session s=0x7f87a400cd30 osd=3 3 2024-04-14T15:23:03.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.951+0000 7f87bd598700 15 client.49588.objecter _session_command_op_remove -1 8 2024-04-14T15:23:03.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.951+0000 7f87bd598700 20 client.49588.objecter get_session s=0x7f87a400cd30 osd=3 3 2024-04-14T15:23:03.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.951+0000 7f87bd598700 15 client.49588.objecter _session_command_op_assign 3 8 2024-04-14T15:23:03.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.951+0000 7f87bd598700 20 client.49588.objecter put_session s=0x7f87a400cd30 osd=3 4 2024-04-14T15:23:03.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.951+0000 7f87bd598700 10 client.49588.objecter _send_command 8 2024-04-14T15:23:03.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.951+0000 7f87bd598700 1 -- 172.21.15.90:0/2385759697 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f87a4013740 con 0x7f87a40111f0 2024-04-14T15:23:03.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.953+0000 7f879ffff700 1 -- 172.21.15.90:0/2385759697 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f87a4013740 con 0x7f87a40111f0 2024-04-14T15:23:03.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.953+0000 7f879ffff700 10 client.49588.objecter ms_dispatch 0x7f87b8001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:23:03.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.953+0000 7f879ffff700 10 client.49588.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:23:03.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.953+0000 7f879ffff700 20 client.49588.objecter put_session s=0x7f87a400cd30 osd=3 4 2024-04-14T15:23:03.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.953+0000 7f879ffff700 15 client.49588.objecter _session_command_op_remove 3 8 2024-04-14T15:23:03.954 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:23:03.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.953+0000 7f87bd598700 10 client.49588.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:23:03.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.953+0000 7f87bd598700 15 client.49588.objecter _session_command_op_assign -1 9 2024-04-14T15:23:03.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.953+0000 7f87bd598700 20 client.49588.objecter _get_session s=0x7f87a4013740 osd=4 3 2024-04-14T15:23:03.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.953+0000 7f87bd598700 20 client.49588.objecter put_session s=0x7f87a4013740 osd=4 3 2024-04-14T15:23:03.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.953+0000 7f87bd598700 20 client.49588.objecter _get_session s=0x7f87a4013740 osd=4 3 2024-04-14T15:23:03.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.953+0000 7f87bd598700 15 client.49588.objecter _session_command_op_remove -1 9 2024-04-14T15:23:03.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.953+0000 7f87bd598700 20 client.49588.objecter get_session s=0x7f87a4013740 osd=4 3 2024-04-14T15:23:03.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.953+0000 7f87bd598700 15 client.49588.objecter _session_command_op_assign 4 9 2024-04-14T15:23:03.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.953+0000 7f87bd598700 20 client.49588.objecter put_session s=0x7f87a4013740 osd=4 4 2024-04-14T15:23:03.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.953+0000 7f87bd598700 10 client.49588.objecter _send_command 9 2024-04-14T15:23:03.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.953+0000 7f87bd598700 1 -- 172.21.15.90:0/2385759697 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f87a401a6e0 con 0x7f87a4016bd0 2024-04-14T15:23:03.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.953+0000 7f879ffff700 10 client.49588.objecter ms_handle_connect 0x7f87a4016bd0 2024-04-14T15:23:03.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.954+0000 7f879ffff700 1 -- 172.21.15.90:0/2385759697 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f87a401a6e0 con 0x7f87a4016bd0 2024-04-14T15:23:03.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.954+0000 7f879ffff700 10 client.49588.objecter ms_dispatch 0x7f87b8001550 command_reply(tid 9: 0 ) 2024-04-14T15:23:03.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.954+0000 7f879ffff700 10 client.49588.objecter _finish_command 9 = system:0 2024-04-14T15:23:03.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.954+0000 7f879ffff700 20 client.49588.objecter put_session s=0x7f87a4013740 osd=4 4 2024-04-14T15:23:03.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.954+0000 7f879ffff700 15 client.49588.objecter _session_command_op_remove 4 9 2024-04-14T15:23:03.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.970+0000 7f87bd598700 10 client.49588.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:23:03.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.970+0000 7f87bd598700 15 client.49588.objecter _session_command_op_assign -1 10 2024-04-14T15:23:03.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.970+0000 7f87bd598700 20 client.49588.objecter _get_session s=0x7f87a4013740 osd=4 3 2024-04-14T15:23:03.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.970+0000 7f87bd598700 20 client.49588.objecter put_session s=0x7f87a4013740 osd=4 3 2024-04-14T15:23:03.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.970+0000 7f87bd598700 20 client.49588.objecter _get_session s=0x7f87a4013740 osd=4 3 2024-04-14T15:23:03.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.970+0000 7f87bd598700 15 client.49588.objecter _session_command_op_remove -1 10 2024-04-14T15:23:03.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.970+0000 7f87bd598700 20 client.49588.objecter get_session s=0x7f87a4013740 osd=4 3 2024-04-14T15:23:03.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.970+0000 7f87bd598700 15 client.49588.objecter _session_command_op_assign 4 10 2024-04-14T15:23:03.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.970+0000 7f87bd598700 20 client.49588.objecter put_session s=0x7f87a4013740 osd=4 4 2024-04-14T15:23:03.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.970+0000 7f87bd598700 10 client.49588.objecter _send_command 10 2024-04-14T15:23:03.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.970+0000 7f87bd598700 1 -- 172.21.15.90:0/2385759697 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f87a400c5c0 con 0x7f87a4016bd0 2024-04-14T15:23:03.972 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:23:03.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.972+0000 7f879ffff700 1 -- 172.21.15.90:0/2385759697 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f87a400c5c0 con 0x7f87a4016bd0 2024-04-14T15:23:03.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.972+0000 7f879ffff700 10 client.49588.objecter ms_dispatch 0x7f87b8001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:23:03.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.972+0000 7f879ffff700 10 client.49588.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:23:03.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.972+0000 7f879ffff700 20 client.49588.objecter put_session s=0x7f87a4013740 osd=4 4 2024-04-14T15:23:03.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.972+0000 7f879ffff700 15 client.49588.objecter _session_command_op_remove 4 10 2024-04-14T15:23:03.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.972+0000 7f87bd598700 10 client.49588.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:23:03.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.972+0000 7f87bd598700 15 client.49588.objecter _session_command_op_assign -1 11 2024-04-14T15:23:03.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.972+0000 7f87bd598700 20 client.49588.objecter _get_session s=0x7f87a400c5c0 osd=5 3 2024-04-14T15:23:03.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.972+0000 7f87bd598700 20 client.49588.objecter put_session s=0x7f87a400c5c0 osd=5 3 2024-04-14T15:23:03.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.972+0000 7f87bd598700 20 client.49588.objecter _get_session s=0x7f87a400c5c0 osd=5 3 2024-04-14T15:23:03.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.972+0000 7f87bd598700 15 client.49588.objecter _session_command_op_remove -1 11 2024-04-14T15:23:03.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.972+0000 7f87bd598700 20 client.49588.objecter get_session s=0x7f87a400c5c0 osd=5 3 2024-04-14T15:23:03.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.972+0000 7f87bd598700 15 client.49588.objecter _session_command_op_assign 5 11 2024-04-14T15:23:03.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.972+0000 7f87bd598700 20 client.49588.objecter put_session s=0x7f87a400c5c0 osd=5 4 2024-04-14T15:23:03.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.972+0000 7f87bd598700 10 client.49588.objecter _send_command 11 2024-04-14T15:23:03.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.972+0000 7f87bd598700 1 -- 172.21.15.90:0/2385759697 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f87a40132f0 con 0x7f87a4012d20 2024-04-14T15:23:03.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.973+0000 7f879ffff700 10 client.49588.objecter ms_handle_connect 0x7f87a4012d20 2024-04-14T15:23:03.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.973+0000 7f879ffff700 1 -- 172.21.15.90:0/2385759697 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f87a40132f0 con 0x7f87a4012d20 2024-04-14T15:23:03.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.973+0000 7f879ffff700 10 client.49588.objecter ms_dispatch 0x7f87b8001550 command_reply(tid 11: 0 ) 2024-04-14T15:23:03.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.973+0000 7f879ffff700 10 client.49588.objecter _finish_command 11 = system:0 2024-04-14T15:23:03.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.973+0000 7f879ffff700 20 client.49588.objecter put_session s=0x7f87a400c5c0 osd=5 4 2024-04-14T15:23:03.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.973+0000 7f879ffff700 15 client.49588.objecter _session_command_op_remove 5 11 2024-04-14T15:23:04.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.997+0000 7f87bd598700 10 client.49588.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:23:04.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.997+0000 7f87bd598700 15 client.49588.objecter _session_command_op_assign -1 12 2024-04-14T15:23:04.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.997+0000 7f87bd598700 20 client.49588.objecter _get_session s=0x7f87a400c5c0 osd=5 3 2024-04-14T15:23:04.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.997+0000 7f87bd598700 20 client.49588.objecter put_session s=0x7f87a400c5c0 osd=5 3 2024-04-14T15:23:04.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.997+0000 7f87bd598700 20 client.49588.objecter _get_session s=0x7f87a400c5c0 osd=5 3 2024-04-14T15:23:04.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.997+0000 7f87bd598700 15 client.49588.objecter _session_command_op_remove -1 12 2024-04-14T15:23:04.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.997+0000 7f87bd598700 20 client.49588.objecter get_session s=0x7f87a400c5c0 osd=5 3 2024-04-14T15:23:04.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.997+0000 7f87bd598700 15 client.49588.objecter _session_command_op_assign 5 12 2024-04-14T15:23:04.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.997+0000 7f87bd598700 20 client.49588.objecter put_session s=0x7f87a400c5c0 osd=5 4 2024-04-14T15:23:04.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.997+0000 7f87bd598700 10 client.49588.objecter _send_command 12 2024-04-14T15:23:04.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:03.997+0000 7f87bd598700 1 -- 172.21.15.90:0/2385759697 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f87a4013500 con 0x7f87a4012d20 2024-04-14T15:23:04.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.002+0000 7f879ffff700 1 -- 172.21.15.90:0/2385759697 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f87a4013500 con 0x7f87a4012d20 2024-04-14T15:23:04.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.002+0000 7f879ffff700 10 client.49588.objecter ms_dispatch 0x7f87b8001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:23:04.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.002+0000 7f879ffff700 10 client.49588.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:23:04.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.002+0000 7f879ffff700 20 client.49588.objecter put_session s=0x7f87a400c5c0 osd=5 4 2024-04-14T15:23:04.003 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:23:04.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.002+0000 7f879ffff700 15 client.49588.objecter _session_command_op_remove 5 12 2024-04-14T15:23:04.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.002+0000 7f87bd598700 10 client.49588.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:23:04.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.002+0000 7f87bd598700 15 client.49588.objecter _session_command_op_assign -1 13 2024-04-14T15:23:04.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.002+0000 7f87bd598700 20 client.49588.objecter _get_session s=0x7f87a4013560 osd=6 3 2024-04-14T15:23:04.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.002+0000 7f87bd598700 20 client.49588.objecter put_session s=0x7f87a4013560 osd=6 3 2024-04-14T15:23:04.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.002+0000 7f87bd598700 20 client.49588.objecter _get_session s=0x7f87a4013560 osd=6 3 2024-04-14T15:23:04.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.002+0000 7f87bd598700 15 client.49588.objecter _session_command_op_remove -1 13 2024-04-14T15:23:04.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.002+0000 7f87bd598700 20 client.49588.objecter get_session s=0x7f87a4013560 osd=6 3 2024-04-14T15:23:04.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.002+0000 7f87bd598700 15 client.49588.objecter _session_command_op_assign 6 13 2024-04-14T15:23:04.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.002+0000 7f87bd598700 20 client.49588.objecter put_session s=0x7f87a4013560 osd=6 4 2024-04-14T15:23:04.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.002+0000 7f87bd598700 10 client.49588.objecter _send_command 13 2024-04-14T15:23:04.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.002+0000 7f87bd598700 1 -- 172.21.15.90:0/2385759697 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f87a4022fb0 con 0x7f87a401f4a0 2024-04-14T15:23:04.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.004+0000 7f879ffff700 10 client.49588.objecter ms_handle_connect 0x7f87a401f4a0 2024-04-14T15:23:04.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.004+0000 7f879ffff700 1 -- 172.21.15.90:0/2385759697 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f87a4022fb0 con 0x7f87a401f4a0 2024-04-14T15:23:04.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.004+0000 7f879ffff700 10 client.49588.objecter ms_dispatch 0x7f87b8001550 command_reply(tid 13: 0 ) 2024-04-14T15:23:04.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.004+0000 7f879ffff700 10 client.49588.objecter _finish_command 13 = system:0 2024-04-14T15:23:04.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.004+0000 7f879ffff700 20 client.49588.objecter put_session s=0x7f87a4013560 osd=6 4 2024-04-14T15:23:04.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.004+0000 7f879ffff700 15 client.49588.objecter _session_command_op_remove 6 13 2024-04-14T15:23:04.006 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:23:04.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.022+0000 7f87bd598700 10 client.49588.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:23:04.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.022+0000 7f87bd598700 15 client.49588.objecter _session_command_op_assign -1 14 2024-04-14T15:23:04.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.022+0000 7f87bd598700 20 client.49588.objecter _get_session s=0x7f87a4013560 osd=6 3 2024-04-14T15:23:04.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.022+0000 7f87bd598700 20 client.49588.objecter put_session s=0x7f87a4013560 osd=6 3 2024-04-14T15:23:04.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.022+0000 7f87bd598700 20 client.49588.objecter _get_session s=0x7f87a4013560 osd=6 3 2024-04-14T15:23:04.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.022+0000 7f87bd598700 15 client.49588.objecter _session_command_op_remove -1 14 2024-04-14T15:23:04.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.022+0000 7f87bd598700 20 client.49588.objecter get_session s=0x7f87a4013560 osd=6 3 2024-04-14T15:23:04.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.022+0000 7f87bd598700 15 client.49588.objecter _session_command_op_assign 6 14 2024-04-14T15:23:04.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.022+0000 7f87bd598700 20 client.49588.objecter put_session s=0x7f87a4013560 osd=6 4 2024-04-14T15:23:04.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.022+0000 7f87bd598700 10 client.49588.objecter _send_command 14 2024-04-14T15:23:04.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.022+0000 7f87bd598700 1 -- 172.21.15.90:0/2385759697 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f87a4011ff0 con 0x7f87a401f4a0 2024-04-14T15:23:04.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.024+0000 7f879ffff700 1 -- 172.21.15.90:0/2385759697 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f87a4011ff0 con 0x7f87a401f4a0 2024-04-14T15:23:04.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.024+0000 7f879ffff700 10 client.49588.objecter ms_dispatch 0x7f87b8001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:23:04.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.024+0000 7f879ffff700 10 client.49588.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:23:04.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.024+0000 7f879ffff700 20 client.49588.objecter put_session s=0x7f87a4013560 osd=6 4 2024-04-14T15:23:04.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.024+0000 7f879ffff700 15 client.49588.objecter _session_command_op_remove 6 14 2024-04-14T15:23:04.025 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:23:04.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.024+0000 7f87bd598700 10 client.49588.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:23:04.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.024+0000 7f87bd598700 15 client.49588.objecter _session_command_op_assign -1 15 2024-04-14T15:23:04.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.024+0000 7f87bd598700 20 client.49588.objecter _get_session s=0x7f87a4012450 osd=7 3 2024-04-14T15:23:04.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.024+0000 7f87bd598700 20 client.49588.objecter put_session s=0x7f87a4012450 osd=7 3 2024-04-14T15:23:04.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.024+0000 7f87bd598700 20 client.49588.objecter _get_session s=0x7f87a4012450 osd=7 3 2024-04-14T15:23:04.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.024+0000 7f87bd598700 15 client.49588.objecter _session_command_op_remove -1 15 2024-04-14T15:23:04.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.024+0000 7f87bd598700 20 client.49588.objecter get_session s=0x7f87a4012450 osd=7 3 2024-04-14T15:23:04.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.024+0000 7f87bd598700 15 client.49588.objecter _session_command_op_assign 7 15 2024-04-14T15:23:04.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.024+0000 7f87bd598700 20 client.49588.objecter put_session s=0x7f87a4012450 osd=7 4 2024-04-14T15:23:04.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.024+0000 7f87bd598700 10 client.49588.objecter _send_command 15 2024-04-14T15:23:04.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.024+0000 7f87bd598700 1 -- 172.21.15.90:0/2385759697 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f87a4024740 con 0x7f87a4011da0 2024-04-14T15:23:04.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.025+0000 7f879ffff700 10 client.49588.objecter ms_handle_connect 0x7f87a4011da0 2024-04-14T15:23:04.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.025+0000 7f879ffff700 1 -- 172.21.15.90:0/2385759697 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f87ac008470 con 0x7f87a4011da0 2024-04-14T15:23:04.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.026+0000 7f879ffff700 10 client.49588.objecter ms_dispatch 0x7f87b8001550 command_reply(tid 15: 0 ) 2024-04-14T15:23:04.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.026+0000 7f879ffff700 10 client.49588.objecter _finish_command 15 = system:0 2024-04-14T15:23:04.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.026+0000 7f879ffff700 20 client.49588.objecter put_session s=0x7f87a4012450 osd=7 4 2024-04-14T15:23:04.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.026+0000 7f879ffff700 15 client.49588.objecter _session_command_op_remove 7 15 2024-04-14T15:23:04.030 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:04.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:04.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:23:04.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.043+0000 7f87bd598700 10 client.49588.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:23:04.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.043+0000 7f87bd598700 15 client.49588.objecter _session_command_op_assign -1 16 2024-04-14T15:23:04.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.043+0000 7f87bd598700 20 client.49588.objecter _get_session s=0x7f87a4012450 osd=7 3 2024-04-14T15:23:04.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.043+0000 7f87bd598700 20 client.49588.objecter put_session s=0x7f87a4012450 osd=7 3 2024-04-14T15:23:04.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.043+0000 7f87bd598700 20 client.49588.objecter _get_session s=0x7f87a4012450 osd=7 3 2024-04-14T15:23:04.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.043+0000 7f87bd598700 15 client.49588.objecter _session_command_op_remove -1 16 2024-04-14T15:23:04.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.043+0000 7f87bd598700 20 client.49588.objecter get_session s=0x7f87a4012450 osd=7 3 2024-04-14T15:23:04.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.043+0000 7f87bd598700 15 client.49588.objecter _session_command_op_assign 7 16 2024-04-14T15:23:04.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.043+0000 7f87bd598700 20 client.49588.objecter put_session s=0x7f87a4012450 osd=7 4 2024-04-14T15:23:04.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.043+0000 7f87bd598700 10 client.49588.objecter _send_command 16 2024-04-14T15:23:04.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.043+0000 7f87bd598700 1 -- 172.21.15.90:0/2385759697 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f87a401b490 con 0x7f87a4011da0 2024-04-14T15:23:04.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.045+0000 7f879ffff700 1 -- 172.21.15.90:0/2385759697 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f87a401b490 con 0x7f87a4011da0 2024-04-14T15:23:04.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.045+0000 7f879ffff700 10 client.49588.objecter ms_dispatch 0x7f87b8001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:23:04.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.045+0000 7f879ffff700 10 client.49588.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:23:04.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.045+0000 7f879ffff700 20 client.49588.objecter put_session s=0x7f87a4012450 osd=7 4 2024-04-14T15:23:04.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.045+0000 7f879ffff700 15 client.49588.objecter _session_command_op_remove 7 16 2024-04-14T15:23:04.046 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:23:04.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.045+0000 7f87bd598700 10 librados: watch_flush enter 2024-04-14T15:23:04.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.045+0000 7f87bd598700 10 librados: watch_flush exit 2024-04-14T15:23:04.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.045+0000 7f87bd598700 10 client.49588.objecter close_session for osd.0 2024-04-14T15:23:04.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.045+0000 7f87bd598700 1 -- 172.21.15.90:0/2385759697 >> v1:172.21.15.90:6801/3761806885 conn(0x7f87a4004730 legacy=0x7f87a4002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:04.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.045+0000 7f87bd598700 20 client.49588.objecter put_session s=0x7f87a40053e0 osd=0 1 2024-04-14T15:23:04.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.045+0000 7f87bd598700 10 client.49588.objecter close_session for osd.1 2024-04-14T15:23:04.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.045+0000 7f87bd598700 1 -- 172.21.15.90:0/2385759697 >> v1:172.21.15.90:6805/4170361699 conn(0x7f87a40042a0 legacy=0x7f87a400ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:04.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.045+0000 7f87bd598700 20 client.49588.objecter put_session s=0x7f87a4004130 osd=1 1 2024-04-14T15:23:04.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.045+0000 7f87bd598700 10 client.49588.objecter close_session for osd.2 2024-04-14T15:23:04.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.045+0000 7f87bd598700 1 -- 172.21.15.90:0/2385759697 >> v1:172.21.15.90:6809/1686184251 conn(0x7f87a4008990 legacy=0x7f87a400f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:04.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.045+0000 7f87bd598700 20 client.49588.objecter put_session s=0x7f87a4001a00 osd=2 1 2024-04-14T15:23:04.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.045+0000 7f87bd598700 10 client.49588.objecter close_session for osd.3 2024-04-14T15:23:04.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.045+0000 7f87bd598700 1 -- 172.21.15.90:0/2385759697 >> v1:172.21.15.90:6813/2270166774 conn(0x7f87a40111f0 legacy=0x7f87a4011630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:04.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.045+0000 7f87bd598700 20 client.49588.objecter put_session s=0x7f87a400cd30 osd=3 1 2024-04-14T15:23:04.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.045+0000 7f87bd598700 10 client.49588.objecter close_session for osd.4 2024-04-14T15:23:04.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.045+0000 7f87bd598700 1 -- 172.21.15.90:0/2385759697 >> v1:172.21.15.92:6800/3887549733 conn(0x7f87a4016bd0 legacy=0x7f87a4018fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:04.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.045+0000 7f87bd598700 20 client.49588.objecter put_session s=0x7f87a4013740 osd=4 1 2024-04-14T15:23:04.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.045+0000 7f87bd598700 10 client.49588.objecter close_session for osd.5 2024-04-14T15:23:04.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.045+0000 7f87bd598700 1 -- 172.21.15.90:0/2385759697 >> v1:172.21.15.92:6804/3244561440 conn(0x7f87a4012d20 legacy=0x7f87a401d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:04.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.045+0000 7f87bd598700 20 client.49588.objecter put_session s=0x7f87a400c5c0 osd=5 1 2024-04-14T15:23:04.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.045+0000 7f87bd598700 10 client.49588.objecter close_session for osd.6 2024-04-14T15:23:04.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.045+0000 7f87bd598700 1 -- 172.21.15.90:0/2385759697 >> v1:172.21.15.92:6808/636125963 conn(0x7f87a401f4a0 legacy=0x7f87a4021890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:04.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.045+0000 7f87bd598700 20 client.49588.objecter put_session s=0x7f87a4013560 osd=6 1 2024-04-14T15:23:04.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.045+0000 7f87bd598700 10 client.49588.objecter close_session for osd.7 2024-04-14T15:23:04.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.045+0000 7f87bd598700 1 -- 172.21.15.90:0/2385759697 >> v1:172.21.15.92:6812/1773706716 conn(0x7f87a4011da0 legacy=0x7f87a4022fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:04.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.045+0000 7f87bd598700 20 client.49588.objecter put_session s=0x7f87a4012450 osd=7 1 2024-04-14T15:23:04.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.045+0000 7f87bd598700 20 client.49588.objecter shutdown clearing up homeless session... 2024-04-14T15:23:04.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.045+0000 7f87bd598700 10 client.49588.objecter successfully canceled tick 2024-04-14T15:23:04.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.045+0000 7f87b77fe700 1 -- 172.21.15.90:0/2385759697 reap_dead start 2024-04-14T15:23:04.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.045+0000 7f87b77fe700 1 -- 172.21.15.90:0/2385759697 reap_dead start 2024-04-14T15:23:04.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.045+0000 7f87b77fe700 1 -- 172.21.15.90:0/2385759697 reap_dead start 2024-04-14T15:23:04.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.045+0000 7f87b77fe700 1 -- 172.21.15.90:0/2385759697 reap_dead start 2024-04-14T15:23:04.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.045+0000 7f87b77fe700 1 -- 172.21.15.90:0/2385759697 reap_dead start 2024-04-14T15:23:04.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.045+0000 7f87bd598700 1 -- 172.21.15.90:0/2385759697 >> v1:172.21.15.90:6800/420935427 conn(0x7f87a0071470 legacy=0x7f87a0073930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:04.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.045+0000 7f87bd598700 1 -- 172.21.15.90:0/2385759697 >> v1:172.21.15.92:6789/0 conn(0x7f87b810f4d0 legacy=0x7f87b81a8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:04.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.045+0000 7f87bd598700 1 -- 172.21.15.90:0/2385759697 shutdown_connections 2024-04-14T15:23:04.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.045+0000 7f87bd598700 1 -- 172.21.15.90:0/2385759697 >> 172.21.15.90:0/2385759697 conn(0x7f87b80fd290 msgr2=0x7f87b8111950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:23:04.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.046+0000 7f87bd598700 1 -- 172.21.15.90:0/2385759697 shutdown_connections 2024-04-14T15:23:04.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.046+0000 7f87bd598700 1 -- 172.21.15.90:0/2385759697 wait complete. 2024-04-14T15:23:04.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:04.046+0000 7f87bd598700 1 librados: shutdown 2024-04-14T15:23:04.051 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:23:04.051 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:23:04.051 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:23:04.051 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:23:04.052 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:23:04.052 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:23:04.052 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:23:04.052 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:23:04.109 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-04-14T15:23:04.109 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-04-14T15:23:04.109 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-04-14T15:23:04.109 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-04-14T15:23:04.109 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-04-14T15:23:04.109 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:04 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:23:04.109 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:04 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:23:04.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:04 smithi092 ceph-mon[33966]: 11.8 scrub starts 2024-04-14T15:23:04.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:04 smithi092 ceph-mon[33966]: 11.8 scrub ok 2024-04-14T15:23:04.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:04 smithi092 ceph-mon[33966]: pgmap v951: 184 pgs: 8 unknown, 1 active+remapped+backfill_toofull, 175 active+clean; 582 KiB data, 578 MiB used, 625 GiB / 626 GiB avail; 2/597 objects misplaced (0.335%) 2024-04-14T15:23:04.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:04 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2385759697' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:23:04.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:04 smithi092 ceph-mon[33966]: osdmap e648: 8 total, 8 up, 7 in 2024-04-14T15:23:04.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:04 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:23:04.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:04 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:23:04.132 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:23:04.132 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd set noscrub 2024-04-14T15:23:04.156 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:04.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:04.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:23:04.191 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:23:03 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:23:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:23:04.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:04 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:23:04.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:04 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:23:04.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:04 smithi090 ceph-mon[41670]: 11.8 scrub starts 2024-04-14T15:23:04.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:04 smithi090 ceph-mon[41670]: 11.8 scrub ok 2024-04-14T15:23:04.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:04 smithi090 ceph-mon[41670]: pgmap v951: 184 pgs: 8 unknown, 1 active+remapped+backfill_toofull, 175 active+clean; 582 KiB data, 578 MiB used, 625 GiB / 626 GiB avail; 2/597 objects misplaced (0.335%) 2024-04-14T15:23:04.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:04 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2385759697' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:23:04.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:04 smithi090 ceph-mon[41670]: osdmap e648: 8 total, 8 up, 7 in 2024-04-14T15:23:04.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:04 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:23:04.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:04 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:23:04.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:04 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:23:04.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:04 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:23:04.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:04 smithi090 ceph-mon[30399]: 11.8 scrub starts 2024-04-14T15:23:04.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:04 smithi090 ceph-mon[30399]: 11.8 scrub ok 2024-04-14T15:23:04.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:04 smithi090 ceph-mon[30399]: pgmap v951: 184 pgs: 8 unknown, 1 active+remapped+backfill_toofull, 175 active+clean; 582 KiB data, 578 MiB used, 625 GiB / 626 GiB avail; 2/597 objects misplaced (0.335%) 2024-04-14T15:23:04.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:04 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2385759697' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:23:04.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:04 smithi090 ceph-mon[30399]: osdmap e648: 8 total, 8 up, 7 in 2024-04-14T15:23:04.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:04 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:23:04.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:04 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:23:04.258 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:23:04.282 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:04.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:04.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:23:04.384 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:23:04.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:04.410 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:04.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:23:04.511 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:04.534 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:04.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:04.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:04.609 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:23:04.636 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:23:04.660 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:04.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:04.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:23:04.756 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-14T15:23:04.761 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:23:04.822 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:04.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:04.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:23:04.924 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:23:04.948 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:04.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:04.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:23:05.002 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:23:05.049 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:23:05.073 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:05.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:05.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:23:05.175 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:23:05.198 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:05.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:05.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:23:05.299 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:23:05.305 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:05 smithi092 ceph-mon[33966]: 11.1c scrub starts 2024-04-14T15:23:05.305 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:05 smithi092 ceph-mon[33966]: 11.1c scrub ok 2024-04-14T15:23:05.306 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:05 smithi092 ceph-mon[33966]: 4.a scrub starts 2024-04-14T15:23:05.306 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:05 smithi092 ceph-mon[33966]: 4.a scrub ok 2024-04-14T15:23:05.306 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:05 smithi092 ceph-mon[33966]: 5.14 scrub starts 2024-04-14T15:23:05.306 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:05 smithi092 ceph-mon[33966]: 5.14 scrub ok 2024-04-14T15:23:05.306 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:05 smithi092 ceph-mon[33966]: 5.1b scrub starts 2024-04-14T15:23:05.306 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:05 smithi092 ceph-mon[33966]: 5.1b scrub ok 2024-04-14T15:23:05.306 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:05 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:23:05.306 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:05 smithi092 ceph-mon[33966]: osdmap e649: 8 total, 8 up, 7 in 2024-04-14T15:23:05.306 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:05 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:23:05.306 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:05 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:23:05.323 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:05.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:05.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:23:05.362 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:05 smithi090 ceph-mon[41670]: 11.1c scrub starts 2024-04-14T15:23:05.363 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:05 smithi090 ceph-mon[41670]: 11.1c scrub ok 2024-04-14T15:23:05.363 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:05 smithi090 ceph-mon[41670]: 4.a scrub starts 2024-04-14T15:23:05.363 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:05 smithi090 ceph-mon[41670]: 4.a scrub ok 2024-04-14T15:23:05.363 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:05 smithi090 ceph-mon[41670]: 5.14 scrub starts 2024-04-14T15:23:05.363 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:05 smithi090 ceph-mon[41670]: 5.14 scrub ok 2024-04-14T15:23:05.363 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:05 smithi090 ceph-mon[41670]: 5.1b scrub starts 2024-04-14T15:23:05.363 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:05 smithi090 ceph-mon[41670]: 5.1b scrub ok 2024-04-14T15:23:05.363 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:05 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:23:05.363 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:05 smithi090 ceph-mon[41670]: osdmap e649: 8 total, 8 up, 7 in 2024-04-14T15:23:05.363 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:05 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:23:05.363 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:05 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:23:05.363 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:05 smithi090 ceph-mon[30399]: 11.1c scrub starts 2024-04-14T15:23:05.363 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:05 smithi090 ceph-mon[30399]: 11.1c scrub ok 2024-04-14T15:23:05.364 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:05 smithi090 ceph-mon[30399]: 4.a scrub starts 2024-04-14T15:23:05.364 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:05 smithi090 ceph-mon[30399]: 4.a scrub ok 2024-04-14T15:23:05.364 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:05 smithi090 ceph-mon[30399]: 5.14 scrub starts 2024-04-14T15:23:05.364 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:05 smithi090 ceph-mon[30399]: 5.14 scrub ok 2024-04-14T15:23:05.364 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:05 smithi090 ceph-mon[30399]: 5.1b scrub starts 2024-04-14T15:23:05.364 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:05 smithi090 ceph-mon[30399]: 5.1b scrub ok 2024-04-14T15:23:05.364 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:05 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:23:05.364 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:05 smithi090 ceph-mon[30399]: osdmap e649: 8 total, 8 up, 7 in 2024-04-14T15:23:05.364 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:05 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:23:05.364 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:05 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:23:05.425 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:23:05.448 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:05.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:05.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:23:05.550 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:23:05.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:05.574 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:05.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:23:05.604 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:23:05 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:23:05.675 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:23:05.698 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:05.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:05.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:23:05.800 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:23:05.825 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:05.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:05.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:23:05.927 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:23:05.951 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:05.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:05.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:23:06.052 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:23:06.076 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:06.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:06.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:23:06.177 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:23:06.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:06.202 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:06.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:23:06.303 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:06.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:06.327 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:06.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:06.353 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:06 smithi092 ceph-mon[33966]: pgmap v954: 216 pgs: 32 creating+peering, 1 active+remapped+backfill_toofull, 183 active+clean; 582 KiB data, 579 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/597 objects misplaced (0.335%) 2024-04-14T15:23:06.353 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:06 smithi092 ceph-mon[33966]: 4.7 scrub starts 2024-04-14T15:23:06.353 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:06 smithi092 ceph-mon[33966]: 4.7 scrub ok 2024-04-14T15:23:06.353 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:06 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:23:06.353 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:06 smithi092 ceph-mon[33966]: 6.a scrub starts 2024-04-14T15:23:06.353 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:06 smithi092 ceph-mon[33966]: 6.a scrub ok 2024-04-14T15:23:06.354 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:06 smithi092 ceph-mon[33966]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:23:06.354 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:06 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-112", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:23:06.354 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:06 smithi092 ceph-mon[33966]: osdmap e650: 8 total, 8 up, 7 in 2024-04-14T15:23:06.354 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:06 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-112"}]: dispatch 2024-04-14T15:23:06.354 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:06 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-112"}]: dispatch 2024-04-14T15:23:06.428 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:23:06.431 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:06 smithi090 ceph-mon[30399]: pgmap v954: 216 pgs: 32 creating+peering, 1 active+remapped+backfill_toofull, 183 active+clean; 582 KiB data, 579 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/597 objects misplaced (0.335%) 2024-04-14T15:23:06.431 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:06 smithi090 ceph-mon[30399]: 4.7 scrub starts 2024-04-14T15:23:06.431 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:06 smithi090 ceph-mon[30399]: 4.7 scrub ok 2024-04-14T15:23:06.431 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:06 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:23:06.432 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:06 smithi090 ceph-mon[30399]: 6.a scrub starts 2024-04-14T15:23:06.432 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:06 smithi090 ceph-mon[30399]: 6.a scrub ok 2024-04-14T15:23:06.432 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:06 smithi090 ceph-mon[30399]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:23:06.432 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:06 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-112", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:23:06.432 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:06 smithi090 ceph-mon[30399]: osdmap e650: 8 total, 8 up, 7 in 2024-04-14T15:23:06.432 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:06 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-112"}]: dispatch 2024-04-14T15:23:06.432 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:06 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-112"}]: dispatch 2024-04-14T15:23:06.432 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:06 smithi090 ceph-mon[41670]: pgmap v954: 216 pgs: 32 creating+peering, 1 active+remapped+backfill_toofull, 183 active+clean; 582 KiB data, 579 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/597 objects misplaced (0.335%) 2024-04-14T15:23:06.432 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:06 smithi090 ceph-mon[41670]: 4.7 scrub starts 2024-04-14T15:23:06.432 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:06 smithi090 ceph-mon[41670]: 4.7 scrub ok 2024-04-14T15:23:06.432 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:06 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:23:06.432 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:06 smithi090 ceph-mon[41670]: 6.a scrub starts 2024-04-14T15:23:06.432 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:06 smithi090 ceph-mon[41670]: 6.a scrub ok 2024-04-14T15:23:06.433 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:06 smithi090 ceph-mon[41670]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:23:06.433 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:06 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-112", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:23:06.433 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:06 smithi090 ceph-mon[41670]: osdmap e650: 8 total, 8 up, 7 in 2024-04-14T15:23:06.433 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:06 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-112"}]: dispatch 2024-04-14T15:23:06.433 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:06 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-112"}]: dispatch 2024-04-14T15:23:06.452 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:06.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:06.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:23:06.487 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-14T15:23:06.487 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-14T15:23:06.487 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-04-14T15:23:06.487 INFO:tasks.thrashosds.thrasher:Setting osd 1 primary_affinity to 1.000000 2024-04-14T15:23:06.487 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd primary-affinity 1 1 2024-04-14T15:23:06.554 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:23:06.579 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:06.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:06.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:23:06.680 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:23:06.703 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:06.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:06.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:23:06.805 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:23:06.828 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:06.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:06.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:23:06.843 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-04-14T15:23:06.843 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-04-14T15:23:06.843 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-04-14T15:23:06.843 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-04-14T15:23:06.843 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.843 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25684.0:110 135.17 135:eb4a7a6a:test-rados-api-smithi090-93652-10::foo:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e134)", 2024-04-14T15:23:06.843 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:13:21.824891+0000", 2024-04-14T15:23:06.843 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 585.01721218800003, 2024-04-14T15:23:06.843 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.14413627700000001, 2024-04-14T15:23:06.843 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:23:06.844 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:23:06.844 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-04-14T15:23:06.844 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:23:06.844 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25684", 2024-04-14T15:23:06.844 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/1109245683", 2024-04-14T15:23:06.844 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 110 2024-04-14T15:23:06.844 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.844 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:23:06.844 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.844 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:23:06.844 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:21.824891+0000", 2024-04-14T15:23:06.844 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:23:06.844 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.845 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.845 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:23:06.845 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:21.824891+0000", 2024-04-14T15:23:06.845 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:23:06.845 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.845 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.845 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:23:06.845 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:21.824892+0000", 2024-04-14T15:23:06.845 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.404e-06 2024-04-14T15:23:06.845 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.845 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.845 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:23:06.845 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:21.824898+0000", 2024-04-14T15:23:06.845 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5029999999999997e-06 2024-04-14T15:23:06.845 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.846 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.846 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:23:06.846 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:21.824906+0000", 2024-04-14T15:23:06.846 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8589999999999994e-06 2024-04-14T15:23:06.846 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.846 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.846 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:23:06.846 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:21.824916+0000", 2024-04-14T15:23:06.846 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.6479999999999998e-06 2024-04-14T15:23:06.846 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.846 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.846 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:23:06.846 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:21.824971+0000", 2024-04-14T15:23:06.846 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4805999999999997e-05 2024-04-14T15:23:06.846 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.847 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.847 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for blocked object", 2024-04-14T15:23:06.847 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:21.825199+0000", 2024-04-14T15:23:06.847 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00022818299999999999 2024-04-14T15:23:06.847 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.847 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.847 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:23:06.847 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:21.905705+0000", 2024-04-14T15:23:06.847 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.080505987000000001 2024-04-14T15:23:06.847 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.847 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.847 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for rw locks", 2024-04-14T15:23:06.847 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:21.905814+0000", 2024-04-14T15:23:06.847 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010869 2024-04-14T15:23:06.848 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.848 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.848 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:23:06.848 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:21.968732+0000", 2024-04-14T15:23:06.848 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.062918098000000006 2024-04-14T15:23:06.848 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.848 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.848 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:23:06.848 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:21.968825+0000", 2024-04-14T15:23:06.848 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.2849000000000005e-05 2024-04-14T15:23:06.848 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.848 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.848 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:23:06.848 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:21.969027+0000", 2024-04-14T15:23:06.848 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00020225000000000001 2024-04-14T15:23:06.848 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:23:06.849 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:23:06.849 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:23:06.849 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.849 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.849 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.36767.0:2 147.6 147:602f83fe:::foo:head [append 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-04-14T15:23:06.849 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:13:22.868465+0000", 2024-04-14T15:23:06.849 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 583.97363760899998, 2024-04-14T15:23:06.849 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.10357522299999999, 2024-04-14T15:23:06.849 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:23:06.849 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:23:06.849 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:23:06.849 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:23:06.849 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.36767", 2024-04-14T15:23:06.849 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/902879548", 2024-04-14T15:23:06.849 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 2 2024-04-14T15:23:06.850 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.850 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:23:06.850 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.850 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:23:06.850 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:22.868465+0000", 2024-04-14T15:23:06.850 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:23:06.850 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.850 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.850 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:23:06.850 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:22.868465+0000", 2024-04-14T15:23:06.850 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:23:06.850 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.850 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.850 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:23:06.850 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:22.868467+0000", 2024-04-14T15:23:06.851 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9130000000000001e-06 2024-04-14T15:23:06.851 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.851 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.851 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:23:06.851 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:22.868474+0000", 2024-04-14T15:23:06.851 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2899999999999997e-06 2024-04-14T15:23:06.851 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.851 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.851 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:23:06.851 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:22.868486+0000", 2024-04-14T15:23:06.851 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1823e-05 2024-04-14T15:23:06.851 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.851 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.851 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:23:06.852 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:22.868497+0000", 2024-04-14T15:23:06.852 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0927000000000001e-05 2024-04-14T15:23:06.852 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.852 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.852 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:23:06.852 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:22.868557+0000", 2024-04-14T15:23:06.852 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9996e-05 2024-04-14T15:23:06.852 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.852 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.852 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:23:06.852 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:22.868566+0000", 2024-04-14T15:23:06.852 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3720000000000005e-06 2024-04-14T15:23:06.852 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.852 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.852 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:23:06.852 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:22.917084+0000", 2024-04-14T15:23:06.853 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.048518110000000003 2024-04-14T15:23:06.853 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.853 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.853 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:23:06.853 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:22.917206+0000", 2024-04-14T15:23:06.853 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012209699999999999 2024-04-14T15:23:06.853 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.853 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.853 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 2,6", 2024-04-14T15:23:06.853 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:22.917361+0000", 2024-04-14T15:23:06.853 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000155511 2024-04-14T15:23:06.853 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.853 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.853 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:23:06.853 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:22.917927+0000", 2024-04-14T15:23:06.854 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00056553500000000004 2024-04-14T15:23:06.854 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.854 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.854 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:23:06.854 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:22.918182+0000", 2024-04-14T15:23:06.854 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00025522199999999999 2024-04-14T15:23:06.854 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.854 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.854 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:23:06.854 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:22.971948+0000", 2024-04-14T15:23:06.854 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.053765598999999997 2024-04-14T15:23:06.854 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.854 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.854 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:23:06.855 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:22.971997+0000", 2024-04-14T15:23:06.855 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8860000000000003e-05 2024-04-14T15:23:06.855 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.855 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.855 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:23:06.855 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:22.972041+0000", 2024-04-14T15:23:06.855 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3968000000000003e-05 2024-04-14T15:23:06.855 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:23:06.855 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:23:06.855 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:23:06.855 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.855 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.855 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.41476.0:2 151.0 151:0384ab8e:::oid:head [zero 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-04-14T15:23:06.855 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:13:24.859716+0000", 2024-04-14T15:23:06.855 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 581.98238676300002, 2024-04-14T15:23:06.856 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1201019210000001, 2024-04-14T15:23:06.856 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:23:06.856 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:23:06.856 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-04-14T15:23:06.856 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:23:06.856 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.41476", 2024-04-14T15:23:06.856 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/1075568278", 2024-04-14T15:23:06.856 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 2 2024-04-14T15:23:06.856 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.856 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:23:06.856 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.856 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:23:06.856 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:24.859716+0000", 2024-04-14T15:23:06.856 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:23:06.856 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.857 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.857 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:23:06.857 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:24.859716+0000", 2024-04-14T15:23:06.857 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:23:06.857 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.857 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.857 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:23:06.857 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:24.859718+0000", 2024-04-14T15:23:06.857 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.683e-06 2024-04-14T15:23:06.857 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.857 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.857 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:23:06.857 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:24.859723+0000", 2024-04-14T15:23:06.857 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3279999999999996e-06 2024-04-14T15:23:06.857 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.858 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.858 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:23:06.858 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:24.859732+0000", 2024-04-14T15:23:06.858 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.1810000000000007e-06 2024-04-14T15:23:06.858 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.858 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.858 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:23:06.858 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:24.859742+0000", 2024-04-14T15:23:06.858 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.2709999999999998e-06 2024-04-14T15:23:06.858 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.858 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.858 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:23:06.858 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:24.859790+0000", 2024-04-14T15:23:06.858 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8547000000000001e-05 2024-04-14T15:23:06.858 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.858 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.859 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:23:06.859 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:24.859799+0000", 2024-04-14T15:23:06.859 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.3109999999999995e-06 2024-04-14T15:23:06.859 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.859 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.859 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:23:06.859 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.887558+0000", 2024-04-14T15:23:06.859 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.027758723 2024-04-14T15:23:06.859 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.859 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.859 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:23:06.859 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.887660+0000", 2024-04-14T15:23:06.859 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010152799999999999 2024-04-14T15:23:06.859 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.860 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.860 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:23:06.860 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.979818+0000", 2024-04-14T15:23:06.860 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.092158349000000001 2024-04-14T15:23:06.860 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:23:06.860 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:23:06.860 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:23:06.860 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.860 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.860 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.41515.0:2 154.6 154:602f83fe:::foo:head [append 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-04-14T15:23:06.860 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:13:25.843012+0000", 2024-04-14T15:23:06.860 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 580.99909051400005, 2024-04-14T15:23:06.860 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.136860596, 2024-04-14T15:23:06.860 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:23:06.860 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:23:06.861 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:23:06.861 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:23:06.861 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.41515", 2024-04-14T15:23:06.861 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/1865301891", 2024-04-14T15:23:06.861 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 2 2024-04-14T15:23:06.861 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.861 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:23:06.861 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.861 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:23:06.861 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.843012+0000", 2024-04-14T15:23:06.861 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:23:06.861 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.861 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.861 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:23:06.861 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.843012+0000", 2024-04-14T15:23:06.862 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:23:06.862 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.862 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.862 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:23:06.862 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.843015+0000", 2024-04-14T15:23:06.862 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.159e-06 2024-04-14T15:23:06.862 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.862 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.862 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:23:06.862 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.843024+0000", 2024-04-14T15:23:06.862 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.1330000000000005e-06 2024-04-14T15:23:06.862 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.862 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.862 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:23:06.862 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.843033+0000", 2024-04-14T15:23:06.863 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.5149999999999995e-06 2024-04-14T15:23:06.863 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.863 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.863 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:23:06.863 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.843043+0000", 2024-04-14T15:23:06.863 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0268000000000001e-05 2024-04-14T15:23:06.863 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.863 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.863 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:23:06.863 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.843356+0000", 2024-04-14T15:23:06.863 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00031297200000000001 2024-04-14T15:23:06.863 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.863 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.863 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:23:06.863 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.843366+0000", 2024-04-14T15:23:06.864 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.2240000000000001e-06 2024-04-14T15:23:06.864 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.864 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.864 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:23:06.864 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.903587+0000", 2024-04-14T15:23:06.864 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.060221471999999998 2024-04-14T15:23:06.864 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.864 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.864 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:23:06.864 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.903709+0000", 2024-04-14T15:23:06.864 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012226500000000001 2024-04-14T15:23:06.864 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.864 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.864 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 2,3", 2024-04-14T15:23:06.864 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.903851+0000", 2024-04-14T15:23:06.864 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014190700000000001 2024-04-14T15:23:06.865 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.865 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.865 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:23:06.865 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.904351+0000", 2024-04-14T15:23:06.865 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00049993900000000003 2024-04-14T15:23:06.865 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.865 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.865 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:23:06.865 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.940363+0000", 2024-04-14T15:23:06.865 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.036011941999999998 2024-04-14T15:23:06.865 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.865 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.865 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:23:06.865 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.979696+0000", 2024-04-14T15:23:06.865 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.039332371999999997 2024-04-14T15:23:06.865 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.866 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.866 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:23:06.866 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.979838+0000", 2024-04-14T15:23:06.866 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014200000000000001 2024-04-14T15:23:06.866 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.866 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.866 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:23:06.866 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.979873+0000", 2024-04-14T15:23:06.866 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5428e-05 2024-04-14T15:23:06.866 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:23:06.866 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:23:06.866 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:23:06.866 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.866 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.867 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.42382.0:3 157.0 157:0384ab8e:::oid:head [write 0~128] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-04-14T15:23:06.867 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:13:26.844811+0000", 2024-04-14T15:23:06.867 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 579.99729199000001, 2024-04-14T15:23:06.867 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.152799729, 2024-04-14T15:23:06.867 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:23:06.867 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:23:06.867 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:23:06.867 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:23:06.867 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.42382", 2024-04-14T15:23:06.867 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/1837092947", 2024-04-14T15:23:06.867 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 3 2024-04-14T15:23:06.867 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.868 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:23:06.868 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.868 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:23:06.868 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.844811+0000", 2024-04-14T15:23:06.868 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:23:06.868 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.868 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.868 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:23:06.868 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.844811+0000", 2024-04-14T15:23:06.868 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:23:06.868 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.868 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.868 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:23:06.868 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.847180+0000", 2024-04-14T15:23:06.869 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0023693730000000001 2024-04-14T15:23:06.869 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.869 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.869 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:23:06.869 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.847195+0000", 2024-04-14T15:23:06.869 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.489e-05 2024-04-14T15:23:06.869 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.869 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.870 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:23:06.870 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.847214+0000", 2024-04-14T15:23:06.870 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8564e-05 2024-04-14T15:23:06.870 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.870 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.870 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:23:06.870 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.847225+0000", 2024-04-14T15:23:06.870 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0786e-05 2024-04-14T15:23:06.871 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.871 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.871 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:23:06.871 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.868351+0000", 2024-04-14T15:23:06.871 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.021126823999999999 2024-04-14T15:23:06.871 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.871 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.871 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:23:06.871 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.868370+0000", 2024-04-14T15:23:06.871 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8508e-05 2024-04-14T15:23:06.871 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.871 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.871 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:23:06.872 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.961204+0000", 2024-04-14T15:23:06.872 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.092834633999999999 2024-04-14T15:23:06.872 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.872 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.872 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:23:06.872 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.961327+0000", 2024-04-14T15:23:06.872 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012294199999999999 2024-04-14T15:23:06.872 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.872 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.872 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,6", 2024-04-14T15:23:06.872 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.961485+0000", 2024-04-14T15:23:06.872 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015743699999999999 2024-04-14T15:23:06.872 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.872 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.873 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:23:06.873 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.961970+0000", 2024-04-14T15:23:06.873 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00048496900000000003 2024-04-14T15:23:06.873 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.873 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.873 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:23:06.873 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.962400+0000", 2024-04-14T15:23:06.873 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00043008599999999999 2024-04-14T15:23:06.873 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.873 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.873 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:23:06.873 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.997503+0000", 2024-04-14T15:23:06.873 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.035103500000000003 2024-04-14T15:23:06.873 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.873 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.874 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:23:06.874 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.997559+0000", 2024-04-14T15:23:06.874 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5831000000000001e-05 2024-04-14T15:23:06.874 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.874 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.874 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:23:06.874 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.997611+0000", 2024-04-14T15:23:06.874 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.1384999999999997e-05 2024-04-14T15:23:06.874 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:23:06.874 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:23:06.874 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:23:06.874 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.874 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.874 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.42382.0:8 157.0 157:0384ab8e:::oid:head [write 128~128] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-04-14T15:23:06.875 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:13:27.847441+0000", 2024-04-14T15:23:06.875 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 578.99466231700001, 2024-04-14T15:23:06.875 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.059003879000000002, 2024-04-14T15:23:06.875 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:23:06.875 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:23:06.875 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:23:06.875 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:23:06.875 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.42382", 2024-04-14T15:23:06.875 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/1837092947", 2024-04-14T15:23:06.875 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 8 2024-04-14T15:23:06.875 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.875 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:23:06.875 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.875 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:23:06.875 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:27.847441+0000", 2024-04-14T15:23:06.875 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:23:06.876 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.876 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.876 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:23:06.876 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:27.847441+0000", 2024-04-14T15:23:06.876 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:23:06.876 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.876 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.876 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:23:06.876 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:27.847443+0000", 2024-04-14T15:23:06.876 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0590000000000001e-06 2024-04-14T15:23:06.876 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.876 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.876 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:23:06.876 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:27.847449+0000", 2024-04-14T15:23:06.876 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.7129999999999999e-06 2024-04-14T15:23:06.877 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.877 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.877 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:23:06.877 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:27.847460+0000", 2024-04-14T15:23:06.877 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0359e-05 2024-04-14T15:23:06.877 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.877 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.877 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:23:06.877 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:27.847472+0000", 2024-04-14T15:23:06.877 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.184e-05 2024-04-14T15:23:06.877 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.877 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.877 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:23:06.877 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:27.854881+0000", 2024-04-14T15:23:06.878 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0074090730000000004 2024-04-14T15:23:06.878 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.878 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.878 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:23:06.878 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:27.854956+0000", 2024-04-14T15:23:06.878 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5054000000000003e-05 2024-04-14T15:23:06.878 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.878 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.878 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,6", 2024-04-14T15:23:06.878 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:27.856348+0000", 2024-04-14T15:23:06.878 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001392643 2024-04-14T15:23:06.878 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.878 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.878 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:23:06.879 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:27.857818+0000", 2024-04-14T15:23:06.879 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0014697 2024-04-14T15:23:06.879 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.879 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.879 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:23:06.879 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:27.858388+0000", 2024-04-14T15:23:06.879 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00057031499999999995 2024-04-14T15:23:06.879 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.879 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.879 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:23:06.879 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:27.906371+0000", 2024-04-14T15:23:06.879 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.047982524999999998 2024-04-14T15:23:06.879 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.879 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.879 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:23:06.879 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:27.906406+0000", 2024-04-14T15:23:06.880 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5475999999999997e-05 2024-04-14T15:23:06.880 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.880 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.880 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:23:06.880 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:27.906444+0000", 2024-04-14T15:23:06.880 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8121999999999999e-05 2024-04-14T15:23:06.880 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:23:06.880 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:23:06.880 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:23:06.880 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.880 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.880 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.36864.0:1 259.7s0 259:e732be06:::oid_WaitForSafePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e274)", 2024-04-14T15:23:06.880 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:15:41.861974+0000", 2024-04-14T15:23:06.880 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 444.98012841600001, 2024-04-14T15:23:06.880 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.009675004, 2024-04-14T15:23:06.881 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:23:06.881 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:23:06.881 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:23:06.881 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:23:06.881 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.36864", 2024-04-14T15:23:06.881 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/3688286685", 2024-04-14T15:23:06.881 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 1 2024-04-14T15:23:06.881 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.881 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:23:06.881 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.881 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:23:06.881 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:41.861974+0000", 2024-04-14T15:23:06.881 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:23:06.881 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.881 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.881 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:23:06.881 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:41.861974+0000", 2024-04-14T15:23:06.882 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:23:06.882 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.882 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.882 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:23:06.882 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:41.861976+0000", 2024-04-14T15:23:06.882 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5090000000000001e-06 2024-04-14T15:23:06.882 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.882 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.882 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:23:06.882 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:41.861982+0000", 2024-04-14T15:23:06.882 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6659999999999996e-06 2024-04-14T15:23:06.882 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.882 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.882 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:23:06.883 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:41.861990+0000", 2024-04-14T15:23:06.883 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2789999999999998e-06 2024-04-14T15:23:06.883 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.883 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.883 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:23:06.883 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:41.861999+0000", 2024-04-14T15:23:06.883 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0040000000000005e-06 2024-04-14T15:23:06.883 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.883 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.883 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:23:06.883 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:41.862045+0000", 2024-04-14T15:23:06.883 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5679000000000001e-05 2024-04-14T15:23:06.883 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.883 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.884 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:23:06.884 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:41.862048+0000", 2024-04-14T15:23:06.884 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0639999999999998e-06 2024-04-14T15:23:06.884 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.884 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.884 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:23:06.884 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:42.869700+0000", 2024-04-14T15:23:06.884 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0076524060000001 2024-04-14T15:23:06.884 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.884 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.884 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:23:06.884 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:42.869825+0000", 2024-04-14T15:23:06.884 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012442599999999999 2024-04-14T15:23:06.884 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.884 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.885 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_started", 2024-04-14T15:23:06.885 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:42.870232+0000", 2024-04-14T15:23:06.885 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00040733700000000002 2024-04-14T15:23:06.885 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.885 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.885 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_committed", 2024-04-14T15:23:06.885 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:42.870790+0000", 2024-04-14T15:23:06.885 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00055832199999999998 2024-04-14T15:23:06.885 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.885 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.885 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:23:06.885 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:42.871585+0000", 2024-04-14T15:23:06.885 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00079456100000000005 2024-04-14T15:23:06.885 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.886 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.886 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:23:06.886 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:42.871649+0000", 2024-04-14T15:23:06.886 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4751000000000004e-05 2024-04-14T15:23:06.886 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:23:06.886 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:23:06.886 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:23:06.886 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.886 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.886 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.37038.0:1 277.5s1 277:ab97ba35:::nonexistent:head [read 0~128] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e326)", 2024-04-14T15:23:06.886 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:16:33.929634+0000", 2024-04-14T15:23:06.886 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 392.91246905999998, 2024-04-14T15:23:06.886 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.473047293, 2024-04-14T15:23:06.886 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:23:06.887 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:23:06.887 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-04-14T15:23:06.887 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:23:06.887 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.37038", 2024-04-14T15:23:06.887 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/1251688097", 2024-04-14T15:23:06.887 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 1 2024-04-14T15:23:06.887 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.887 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:23:06.887 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.887 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:23:06.887 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:16:33.929634+0000", 2024-04-14T15:23:06.887 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:23:06.887 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.887 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.887 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:23:06.888 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:16:33.929634+0000", 2024-04-14T15:23:06.888 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:23:06.888 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.888 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.888 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:23:06.888 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:16:33.929636+0000", 2024-04-14T15:23:06.888 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.996e-06 2024-04-14T15:23:06.888 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.888 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.888 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:23:06.888 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:16:33.929642+0000", 2024-04-14T15:23:06.888 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3300000000000004e-06 2024-04-14T15:23:06.888 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.888 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.888 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:23:06.889 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:16:33.929653+0000", 2024-04-14T15:23:06.889 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1212e-05 2024-04-14T15:23:06.889 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.889 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.889 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:23:06.889 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:16:33.929664+0000", 2024-04-14T15:23:06.889 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.024e-05 2024-04-14T15:23:06.889 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.889 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.889 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:23:06.889 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:16:33.929724+0000", 2024-04-14T15:23:06.889 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0699999999999998e-05 2024-04-14T15:23:06.889 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.889 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.890 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:23:06.890 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:16:33.929728+0000", 2024-04-14T15:23:06.890 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.7759999999999999e-06 2024-04-14T15:23:06.890 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.890 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.890 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:23:06.890 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:16:35.402540+0000", 2024-04-14T15:23:06.890 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4728114589999999 2024-04-14T15:23:06.890 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.890 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.890 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:23:06.890 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:16:35.402681+0000", 2024-04-14T15:23:06.890 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014158000000000001 2024-04-14T15:23:06.890 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:23:06.891 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:23:06.891 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:23:06.891 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.891 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.891 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.49576.0:3 324.19 e649/648 324:9d1934b2:test-rados-api-smithi090-93652-113::foo:head v 649'1, mlcod=0'0)", 2024-04-14T15:23:06.891 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:23:05.016594+0000", 2024-04-14T15:23:06.891 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 1.8255087429999999, 2024-04-14T15:23:06.891 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00055741099999999995, 2024-04-14T15:23:06.891 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:23:06.891 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:23:06.891 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:23:06.891 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:23:06.891 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.891 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:23:06.891 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:05.016594+0000", 2024-04-14T15:23:06.892 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:23:06.892 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.892 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.892 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:23:06.892 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:05.016594+0000", 2024-04-14T15:23:06.892 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:23:06.892 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.892 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.892 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:23:06.892 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:05.016595+0000", 2024-04-14T15:23:06.892 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.2800000000000005e-07 2024-04-14T15:23:06.892 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.892 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.892 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:23:06.892 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:05.016602+0000", 2024-04-14T15:23:06.893 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.221e-06 2024-04-14T15:23:06.893 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.893 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.893 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:23:06.893 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:05.016611+0000", 2024-04-14T15:23:06.893 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.9770000000000006e-06 2024-04-14T15:23:06.893 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.893 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.893 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:23:06.893 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:05.016617+0000", 2024-04-14T15:23:06.893 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.981e-06 2024-04-14T15:23:06.893 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.893 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.893 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:23:06.893 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:05.016664+0000", 2024-04-14T15:23:06.894 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7098000000000002e-05 2024-04-14T15:23:06.894 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.894 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.894 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:23:06.894 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:05.016706+0000", 2024-04-14T15:23:06.894 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1409000000000003e-05 2024-04-14T15:23:06.894 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.894 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.894 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:23:06.894 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:05.017095+0000", 2024-04-14T15:23:06.894 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00038902400000000002 2024-04-14T15:23:06.894 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.894 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.894 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:23:06.895 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:05.017152+0000", 2024-04-14T15:23:06.895 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6773e-05 2024-04-14T15:23:06.895 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:23:06.895 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:23:06.895 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:23:06.895 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.895 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.895 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDScrubReserve(1.5,GRANT,e:650,from:2,reservation_nonce:0)", 2024-04-14T15:23:06.895 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:23:06.152037+0000", 2024-04-14T15:23:06.895 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.69006558799999995, 2024-04-14T15:23:06.895 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000109747, 2024-04-14T15:23:06.895 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:23:06.895 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:23:06.895 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-04-14T15:23:06.895 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:23:06.896 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.896 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:23:06.896 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:06.152037+0000", 2024-04-14T15:23:06.896 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:23:06.896 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.896 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.896 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:23:06.896 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:06.152037+0000", 2024-04-14T15:23:06.896 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:23:06.896 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.896 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.896 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:23:06.896 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:06.152039+0000", 2024-04-14T15:23:06.896 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3e-06 2024-04-14T15:23:06.896 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.897 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.897 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:23:06.897 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:06.152044+0000", 2024-04-14T15:23:06.897 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5400000000000003e-06 2024-04-14T15:23:06.897 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.897 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.897 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:23:06.897 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:06.152054+0000", 2024-04-14T15:23:06.897 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0081000000000001e-05 2024-04-14T15:23:06.897 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.897 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.897 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:23:06.897 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:06.152062+0000", 2024-04-14T15:23:06.897 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.6480000000000003e-06 2024-04-14T15:23:06.898 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.898 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.898 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:23:06.898 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:06.152107+0000", 2024-04-14T15:23:06.898 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5250000000000002e-05 2024-04-14T15:23:06.898 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.898 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.898 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:23:06.898 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:06.152121+0000", 2024-04-14T15:23:06.898 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3399e-05 2024-04-14T15:23:06.899 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.899 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.899 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:23:06.899 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:06.152147+0000", 2024-04-14T15:23:06.899 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6529e-05 2024-04-14T15:23:06.899 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:23:06.899 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:23:06.899 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:23:06.899 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.899 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.899 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDScrubReserve(1.5,GRANT,e:650,from:5,reservation_nonce:0)", 2024-04-14T15:23:06.899 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:23:06.152374+0000", 2024-04-14T15:23:06.899 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.68972920800000004, 2024-04-14T15:23:06.899 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000152585, 2024-04-14T15:23:06.900 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:23:06.900 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:23:06.900 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-04-14T15:23:06.900 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:23:06.900 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.900 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:23:06.900 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:06.152374+0000", 2024-04-14T15:23:06.900 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:23:06.900 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.900 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.900 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:23:06.900 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:06.152374+0000", 2024-04-14T15:23:06.900 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:23:06.900 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.900 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.901 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:23:06.901 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:06.152375+0000", 2024-04-14T15:23:06.901 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.9800000000000002e-07 2024-04-14T15:23:06.901 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.901 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.901 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:23:06.901 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:06.152377+0000", 2024-04-14T15:23:06.901 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5129999999999999e-06 2024-04-14T15:23:06.901 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.901 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.901 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:23:06.901 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:06.152384+0000", 2024-04-14T15:23:06.901 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8700000000000003e-06 2024-04-14T15:23:06.901 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.902 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.902 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:23:06.902 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:06.152388+0000", 2024-04-14T15:23:06.902 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0609999999999997e-06 2024-04-14T15:23:06.902 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.902 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.902 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:23:06.902 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:06.152417+0000", 2024-04-14T15:23:06.902 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8719999999999999e-05 2024-04-14T15:23:06.902 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.902 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.902 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:23:06.902 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:06.152429+0000", 2024-04-14T15:23:06.902 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1895999999999999e-05 2024-04-14T15:23:06.903 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.903 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.903 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:23:06.903 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:06.152526+0000", 2024-04-14T15:23:06.903 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.7627000000000006e-05 2024-04-14T15:23:06.903 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:23:06.903 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:23:06.903 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:23:06.903 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.903 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.903 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "rep_scrubmap(1.5 e646 from shard 5)", 2024-04-14T15:23:06.903 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:23:06.153046+0000", 2024-04-14T15:23:06.903 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.68905662499999998, 2024-04-14T15:23:06.903 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3416000000000003e-05, 2024-04-14T15:23:06.903 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:23:06.904 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:23:06.904 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-04-14T15:23:06.904 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:23:06.904 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.904 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:23:06.904 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:06.153046+0000", 2024-04-14T15:23:06.904 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:23:06.904 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.904 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.904 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:23:06.904 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:06.153046+0000", 2024-04-14T15:23:06.904 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:23:06.904 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.905 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.905 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:23:06.905 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:06.153047+0000", 2024-04-14T15:23:06.905 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.16e-07 2024-04-14T15:23:06.905 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.905 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.905 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:23:06.905 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:06.153051+0000", 2024-04-14T15:23:06.905 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2729999999999996e-06 2024-04-14T15:23:06.905 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.905 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.905 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:23:06.905 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:06.153059+0000", 2024-04-14T15:23:06.905 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.4630000000000004e-06 2024-04-14T15:23:06.905 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.906 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.906 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:23:06.906 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:06.153063+0000", 2024-04-14T15:23:06.906 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.527e-06 2024-04-14T15:23:06.906 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.906 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.906 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:23:06.906 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:06.153099+0000", 2024-04-14T15:23:06.906 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5788999999999999e-05 2024-04-14T15:23:06.906 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.906 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.906 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:23:06.906 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:06.153107+0000", 2024-04-14T15:23:06.906 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0790000000000001e-06 2024-04-14T15:23:06.907 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.907 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.907 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:23:06.907 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:06.153120+0000", 2024-04-14T15:23:06.907 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2469e-05 2024-04-14T15:23:06.907 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:23:06.907 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:23:06.907 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:23:06.907 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.907 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.907 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "rep_scrubmap(1.5 e646 from shard 2)", 2024-04-14T15:23:06.907 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:23:06.153089+0000", 2024-04-14T15:23:06.907 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.68901390699999998, 2024-04-14T15:23:06.907 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1909000000000006e-05, 2024-04-14T15:23:06.907 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:23:06.908 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:23:06.908 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-04-14T15:23:06.908 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:23:06.908 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.908 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:23:06.908 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:06.153089+0000", 2024-04-14T15:23:06.908 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:23:06.908 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.908 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.908 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:23:06.908 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:06.153089+0000", 2024-04-14T15:23:06.908 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:23:06.908 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.908 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.908 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:23:06.909 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:06.153089+0000", 2024-04-14T15:23:06.909 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.27e-07 2024-04-14T15:23:06.909 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.909 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.909 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:23:06.909 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:06.153092+0000", 2024-04-14T15:23:06.909 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1749999999999999e-06 2024-04-14T15:23:06.909 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.909 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.909 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:23:06.909 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:06.153095+0000", 2024-04-14T15:23:06.909 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.7570000000000002e-06 2024-04-14T15:23:06.909 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.909 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.910 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:23:06.910 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:06.153098+0000", 2024-04-14T15:23:06.910 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1200000000000002e-06 2024-04-14T15:23:06.910 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.910 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.910 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:23:06.910 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:06.153138+0000", 2024-04-14T15:23:06.910 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9079999999999999e-05 2024-04-14T15:23:06.910 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.910 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.910 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:23:06.910 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:06.153144+0000", 2024-04-14T15:23:06.910 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2099999999999998e-06 2024-04-14T15:23:06.910 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:06.911 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:06.911 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:23:06.911 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:06.153161+0000", 2024-04-14T15:23:06.911 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7139999999999999e-05 2024-04-14T15:23:06.911 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:23:06.911 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:23:06.911 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:23:06.911 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:23:06.911 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:23:06.911 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-04-14T15:23:06.929 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:23:06.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:06.953 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:06.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:23:07.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.040+0000 7f9a6dd63700 1 -- 172.21.15.90:0/474336009 >> v1:172.21.15.92:6789/0 conn(0x7f9a68103aa0 legacy=0x7f9a68103e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:07.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.040+0000 7f9a6dd63700 1 -- 172.21.15.90:0/474336009 shutdown_connections 2024-04-14T15:23:07.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.040+0000 7f9a6dd63700 1 -- 172.21.15.90:0/474336009 >> 172.21.15.90:0/474336009 conn(0x7f9a680fd290 msgr2=0x7f9a680ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:23:07.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.040+0000 7f9a6dd63700 1 -- 172.21.15.90:0/474336009 shutdown_connections 2024-04-14T15:23:07.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.040+0000 7f9a6dd63700 1 -- 172.21.15.90:0/474336009 wait complete. 2024-04-14T15:23:07.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.041+0000 7f9a6dd63700 1 librados: starting msgr at 2024-04-14T15:23:07.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.041+0000 7f9a6dd63700 1 librados: starting objecter 2024-04-14T15:23:07.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.041+0000 7f9a6dd63700 1 Processor -- start 2024-04-14T15:23:07.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.041+0000 7f9a6dd63700 1 -- start start 2024-04-14T15:23:07.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.041+0000 7f9a6dd63700 1 librados: setting wanted keys 2024-04-14T15:23:07.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.041+0000 7f9a6dd63700 1 librados: calling monclient init 2024-04-14T15:23:07.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.042+0000 7f9a6dd63700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f9a681a7b00 con 0x7f9a6810b8b0 2024-04-14T15:23:07.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.042+0000 7f9a6dd63700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f9a681a8d20 con 0x7f9a6810f450 2024-04-14T15:23:07.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.042+0000 7f9a677fe700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f9a68103aa0 0x7f9a6810b160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:36770/0 (socket says 172.21.15.90:36770) 2024-04-14T15:23:07.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.042+0000 7f9a677fe700 1 -- 172.21.15.90:0/3849088408 learned_addr learned my addr 172.21.15.90:0/3849088408 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:23:07.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.042+0000 7f9a66ffd700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f9a6810b8b0 0x7f9a681a2890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:43168/0 (socket says 172.21.15.90:43168) 2024-04-14T15:23:07.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.042+0000 7f9a64ff9700 10 client.?.objecter ms_handle_connect 0x7f9a6810b8b0 2024-04-14T15:23:07.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.042+0000 7f9a64ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:23:07.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.042+0000 7f9a64ff9700 10 client.?.objecter ms_handle_connect 0x7f9a68103aa0 2024-04-14T15:23:07.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.042+0000 7f9a64ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:23:07.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.042+0000 7f9a6dd63700 1 -- 172.21.15.90:0/3849088408 --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f9a681a9f60 con 0x7f9a68103aa0 2024-04-14T15:23:07.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.042+0000 7f9a64ff9700 1 -- 172.21.15.90:0/3849088408 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3273424090 0 0) 0x7f9a681a7b00 con 0x7f9a6810b8b0 2024-04-14T15:23:07.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.042+0000 7f9a64ff9700 1 -- 172.21.15.90:0/3849088408 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9a540039c0 con 0x7f9a6810b8b0 2024-04-14T15:23:07.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.042+0000 7f9a64ff9700 10 client.?.objecter ms_handle_connect 0x7f9a6810f450 2024-04-14T15:23:07.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.042+0000 7f9a64ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:23:07.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.042+0000 7f9a64ff9700 1 -- 172.21.15.90:0/3849088408 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1109771520 0 0) 0x7f9a681a9f60 con 0x7f9a68103aa0 2024-04-14T15:23:07.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.042+0000 7f9a64ff9700 1 -- 172.21.15.90:0/3849088408 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9a681a7b00 con 0x7f9a68103aa0 2024-04-14T15:23:07.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.042+0000 7f9a64ff9700 1 -- 172.21.15.90:0/3849088408 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3488699811 0 0) 0x7f9a681a8d20 con 0x7f9a6810f450 2024-04-14T15:23:07.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.042+0000 7f9a64ff9700 1 -- 172.21.15.90:0/3849088408 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9a681a9f60 con 0x7f9a6810f450 2024-04-14T15:23:07.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.042+0000 7f9a64ff9700 1 -- 172.21.15.90:0/3849088408 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 245096289 0 0) 0x7f9a540039c0 con 0x7f9a6810b8b0 2024-04-14T15:23:07.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.042+0000 7f9a64ff9700 1 -- 172.21.15.90:0/3849088408 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9a681a8d20 con 0x7f9a6810b8b0 2024-04-14T15:23:07.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.042+0000 7f9a64ff9700 1 -- 172.21.15.90:0/3849088408 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f9a500018a0 con 0x7f9a6810b8b0 2024-04-14T15:23:07.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.042+0000 7f9a64ff9700 1 -- 172.21.15.90:0/3849088408 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1157513345 0 0) 0x7f9a681a9f60 con 0x7f9a6810f450 2024-04-14T15:23:07.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.042+0000 7f9a64ff9700 1 -- 172.21.15.90:0/3849088408 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9a540039c0 con 0x7f9a6810f450 2024-04-14T15:23:07.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.042+0000 7f9a64ff9700 1 -- 172.21.15.90:0/3849088408 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3697068082 0 0) 0x7f9a681a7b00 con 0x7f9a68103aa0 2024-04-14T15:23:07.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.042+0000 7f9a64ff9700 1 -- 172.21.15.90:0/3849088408 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9a681a9f60 con 0x7f9a68103aa0 2024-04-14T15:23:07.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.042+0000 7f9a64ff9700 1 -- 172.21.15.90:0/3849088408 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f9a58001070 con 0x7f9a6810f450 2024-04-14T15:23:07.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.042+0000 7f9a64ff9700 1 -- 172.21.15.90:0/3849088408 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f9a60001b90 con 0x7f9a68103aa0 2024-04-14T15:23:07.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.043+0000 7f9a64ff9700 1 -- 172.21.15.90:0/3849088408 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2901099066 0 0) 0x7f9a540039c0 con 0x7f9a6810f450 2024-04-14T15:23:07.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.043+0000 7f9a64ff9700 1 -- 172.21.15.90:0/3849088408 >> v1:172.21.15.90:6790/0 conn(0x7f9a68103aa0 legacy=0x7f9a6810b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:07.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.043+0000 7f9a64ff9700 1 -- 172.21.15.90:0/3849088408 >> v1:172.21.15.90:6789/0 conn(0x7f9a6810b8b0 legacy=0x7f9a681a2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:07.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.043+0000 7f9a64ff9700 1 -- 172.21.15.90:0/3849088408 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a681a7d10 con 0x7f9a6810f450 2024-04-14T15:23:07.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.043+0000 7f9a6dd63700 1 -- 172.21.15.90:0/3849088408 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9a681a8000 con 0x7f9a6810f450 2024-04-14T15:23:07.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.043+0000 7f9a64ff9700 1 -- 172.21.15.90:0/3849088408 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f9a580015e0 con 0x7f9a6810f450 2024-04-14T15:23:07.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.043+0000 7f9a6dd63700 10 client.49594.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:23:07.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.043+0000 7f9a64ff9700 1 -- 172.21.15.90:0/3849088408 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f9a58004920 con 0x7f9a6810f450 2024-04-14T15:23:07.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.043+0000 7f9a6dd63700 1 -- 172.21.15.90:0/3849088408 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f9a681a84c0 con 0x7f9a6810f450 2024-04-14T15:23:07.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.043+0000 7f9a6dd63700 1 librados: init done 2024-04-14T15:23:07.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.044+0000 7f9a6dd63700 1 -- 172.21.15.90:0/3849088408 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a6806b570 con 0x7f9a6810f450 2024-04-14T15:23:07.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.044+0000 7f9a64ff9700 1 -- 172.21.15.90:0/3849088408 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f9a5801b4f0 con 0x7f9a6810f450 2024-04-14T15:23:07.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.045+0000 7f9a64ff9700 1 -- 172.21.15.90:0/3849088408 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(651..651 src has 467..651) ==== 8706+0+0 (unknown 2122373616 0 0) 0x7f9a58089a30 con 0x7f9a6810f450 2024-04-14T15:23:07.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.045+0000 7f9a64ff9700 10 client.49594.objecter ms_dispatch 0x7f9a68001550 osd_map(651..651 src has 467..651) 2024-04-14T15:23:07.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.045+0000 7f9a64ff9700 3 client.49594.objecter handle_osd_map got epochs [651,651] > 0 2024-04-14T15:23:07.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.045+0000 7f9a64ff9700 3 client.49594.objecter handle_osd_map decoding full epoch 651 2024-04-14T15:23:07.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.045+0000 7f9a64ff9700 20 client.49594.objecter dump_active .. 0 homeless 2024-04-14T15:23:07.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.045+0000 7f9a64ff9700 10 client.49594.objecter ms_handle_connect 0x7f9a54071430 2024-04-14T15:23:07.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.049+0000 7f9a64ff9700 1 -- 172.21.15.90:0/3849088408 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f9a58056220 con 0x7f9a6810f450 2024-04-14T15:23:07.054 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:07.058 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:23:07.077 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:07.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:07.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:07.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:07 smithi092 ceph-mon[33966]: 5.4 scrub starts 2024-04-14T15:23:07.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:07 smithi092 ceph-mon[33966]: 5.4 scrub ok 2024-04-14T15:23:07.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:07 smithi092 ceph-mon[33966]: 5.6 scrub starts 2024-04-14T15:23:07.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:07 smithi092 ceph-mon[33966]: 5.6 scrub ok 2024-04-14T15:23:07.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:07 smithi092 ceph-mon[33966]: 1.5 scrub starts 2024-04-14T15:23:07.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:07 smithi092 ceph-mon[33966]: 1.5 scrub ok 2024-04-14T15:23:07.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:07 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-112"}]': finished 2024-04-14T15:23:07.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:07 smithi092 ceph-mon[33966]: osdmap e651: 8 total, 8 up, 7 in 2024-04-14T15:23:07.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:07 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:23:07.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:07 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:23:07.130 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-14T15:23:07.179 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:07.202 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:07.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:07.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:07.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.279+0000 7f9a6dd63700 1 -- 172.21.15.90:0/3849088408 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7f9a680693d0 con 0x7f9a6810f450 2024-04-14T15:23:07.279 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:07 smithi090 ceph-mon[41670]: 5.4 scrub starts 2024-04-14T15:23:07.279 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:07 smithi090 ceph-mon[41670]: 5.4 scrub ok 2024-04-14T15:23:07.280 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:07 smithi090 ceph-mon[41670]: 5.6 scrub starts 2024-04-14T15:23:07.280 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:07 smithi090 ceph-mon[41670]: 5.6 scrub ok 2024-04-14T15:23:07.280 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:07 smithi090 ceph-mon[41670]: 1.5 scrub starts 2024-04-14T15:23:07.280 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:07 smithi090 ceph-mon[41670]: 1.5 scrub ok 2024-04-14T15:23:07.280 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:07 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-112"}]': finished 2024-04-14T15:23:07.280 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:07 smithi090 ceph-mon[41670]: osdmap e651: 8 total, 8 up, 7 in 2024-04-14T15:23:07.280 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:07 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:23:07.280 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:07 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:23:07.280 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:07 smithi090 ceph-mon[30399]: 5.4 scrub starts 2024-04-14T15:23:07.280 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:07 smithi090 ceph-mon[30399]: 5.4 scrub ok 2024-04-14T15:23:07.280 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:07 smithi090 ceph-mon[30399]: 5.6 scrub starts 2024-04-14T15:23:07.280 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:07 smithi090 ceph-mon[30399]: 5.6 scrub ok 2024-04-14T15:23:07.280 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:07 smithi090 ceph-mon[30399]: 1.5 scrub starts 2024-04-14T15:23:07.281 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:07 smithi090 ceph-mon[30399]: 1.5 scrub ok 2024-04-14T15:23:07.281 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:07 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-112"}]': finished 2024-04-14T15:23:07.281 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:07 smithi090 ceph-mon[30399]: osdmap e651: 8 total, 8 up, 7 in 2024-04-14T15:23:07.281 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:07 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:23:07.281 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:07 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:23:07.303 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:23:07.327 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:07.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:07.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:23:07.429 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:23:07.454 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:07.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:07.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:23:07.556 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:23:07.580 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:07.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:07.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:23:07.681 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:23:07.690 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-14T15:23:07.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:07.707 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:07.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:23:07.786 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:23:07.808 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:23:07.831 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:07.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:07.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:23:07.932 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:23:07.943 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:23:07.955 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:07.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:07.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:23:07.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.977+0000 7f9a64ff9700 1 -- 172.21.15.90:0/3849088408 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v652) ==== 87+0+0 (unknown 360426304 0 0) 0x7f9a58059ed0 con 0x7f9a6810f450 2024-04-14T15:23:07.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.981+0000 7f9a6dd63700 10 librados: watch_flush enter 2024-04-14T15:23:07.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.981+0000 7f9a6dd63700 10 librados: watch_flush exit 2024-04-14T15:23:07.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.981+0000 7f9a6dd63700 20 client.49594.objecter shutdown clearing up homeless session... 2024-04-14T15:23:07.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.981+0000 7f9a6dd63700 10 client.49594.objecter successfully canceled tick 2024-04-14T15:23:07.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.981+0000 7f9a6dd63700 1 -- 172.21.15.90:0/3849088408 >> v1:172.21.15.90:6800/420935427 conn(0x7f9a54071430 legacy=0x7f9a540738f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:07.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.981+0000 7f9a6dd63700 1 -- 172.21.15.90:0/3849088408 >> v1:172.21.15.92:6789/0 conn(0x7f9a6810f450 legacy=0x7f9a681a6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:07.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.981+0000 7f9a6dd63700 1 -- 172.21.15.90:0/3849088408 shutdown_connections 2024-04-14T15:23:07.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.981+0000 7f9a6dd63700 1 -- 172.21.15.90:0/3849088408 >> 172.21.15.90:0/3849088408 conn(0x7f9a680fd290 msgr2=0x7f9a681118d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:23:07.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.981+0000 7f9a6dd63700 1 -- 172.21.15.90:0/3849088408 shutdown_connections 2024-04-14T15:23:07.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.981+0000 7f9a6dd63700 1 -- 172.21.15.90:0/3849088408 wait complete. 2024-04-14T15:23:07.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:07.981+0000 7f9a6dd63700 1 librados: shutdown 2024-04-14T15:23:07.983 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is set 2024-04-14T15:23:08.057 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:08.080 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:08.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:08.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:08.182 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:23:08.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:08 smithi090 ceph-mon[30399]: pgmap v957: 216 pgs: 32 creating+peering, 1 active+remapped+backfill_toofull, 183 active+clean; 582 KiB data, 579 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/597 objects misplaced (0.335%) 2024-04-14T15:23:08.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:08 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3849088408' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:23:08.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:08 smithi090 ceph-mon[30399]: from='client.49594 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:23:08.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:08 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:23:08.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:08 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:23:08.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:08 smithi090 ceph-mon[30399]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:23:08.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:08 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]': finished 2024-04-14T15:23:08.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:08 smithi090 ceph-mon[30399]: from='client.49594 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-14T15:23:08.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:08 smithi090 ceph-mon[30399]: osdmap e652: 8 total, 8 up, 7 in 2024-04-14T15:23:08.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:08 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-112"}]: dispatch 2024-04-14T15:23:08.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:08 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-112"}]: dispatch 2024-04-14T15:23:08.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:08 smithi090 ceph-mon[41670]: pgmap v957: 216 pgs: 32 creating+peering, 1 active+remapped+backfill_toofull, 183 active+clean; 582 KiB data, 579 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/597 objects misplaced (0.335%) 2024-04-14T15:23:08.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:08 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3849088408' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:23:08.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:08 smithi090 ceph-mon[41670]: from='client.49594 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:23:08.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:08 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:23:08.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:08 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:23:08.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:08 smithi090 ceph-mon[41670]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:23:08.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:08 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]': finished 2024-04-14T15:23:08.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:08 smithi090 ceph-mon[41670]: from='client.49594 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-14T15:23:08.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:08 smithi090 ceph-mon[41670]: osdmap e652: 8 total, 8 up, 7 in 2024-04-14T15:23:08.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:08 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-112"}]: dispatch 2024-04-14T15:23:08.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:08 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-112"}]: dispatch 2024-04-14T15:23:08.207 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:08.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:08.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:23:08.309 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:08.333 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:08.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:08.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:08.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:08 smithi092 ceph-mon[33966]: pgmap v957: 216 pgs: 32 creating+peering, 1 active+remapped+backfill_toofull, 183 active+clean; 582 KiB data, 579 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/597 objects misplaced (0.335%) 2024-04-14T15:23:08.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:08 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3849088408' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:23:08.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:08 smithi092 ceph-mon[33966]: from='client.49594 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:23:08.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:08 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:23:08.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:08 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:23:08.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:08 smithi092 ceph-mon[33966]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:23:08.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:08 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]': finished 2024-04-14T15:23:08.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:08 smithi092 ceph-mon[33966]: from='client.49594 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-14T15:23:08.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:08 smithi092 ceph-mon[33966]: osdmap e652: 8 total, 8 up, 7 in 2024-04-14T15:23:08.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:08 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-112"}]: dispatch 2024-04-14T15:23:08.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:08 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-112"}]: dispatch 2024-04-14T15:23:08.434 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:08.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:08.459 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:08.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:08.560 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:23:08.583 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:08.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:08.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:23:08.685 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:23:08.709 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:08.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:08.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:23:08.810 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:23:08.834 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:08.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:08.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:23:08.936 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:08.959 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:08.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:08.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:09.060 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:23:09.085 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:09.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:09.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:23:09.103 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:09 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:23:09.103 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:09 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-112"}]': finished 2024-04-14T15:23:09.104 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:09 smithi090 ceph-mon[41670]: osdmap e653: 8 total, 8 up, 7 in 2024-04-14T15:23:09.104 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:09 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:23:09.104 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:09 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-112"}]': finished 2024-04-14T15:23:09.104 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:09 smithi090 ceph-mon[30399]: osdmap e653: 8 total, 8 up, 7 in 2024-04-14T15:23:09.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:09 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:23:09.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:09 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-112"}]': finished 2024-04-14T15:23:09.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:09 smithi092 ceph-mon[33966]: osdmap e653: 8 total, 8 up, 7 in 2024-04-14T15:23:09.186 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:23:09.210 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:09.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:09.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:23:09.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.211+0000 7fe9cdf76700 1 -- 172.21.15.90:0/4237546304 >> v1:172.21.15.90:6789/0 conn(0x7fe9c810b720 legacy=0x7fe9c810db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:09.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.211+0000 7fe9cdf76700 1 -- 172.21.15.90:0/4237546304 shutdown_connections 2024-04-14T15:23:09.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.211+0000 7fe9cdf76700 1 -- 172.21.15.90:0/4237546304 >> 172.21.15.90:0/4237546304 conn(0x7fe9c80fe3c0 msgr2=0x7fe9c81007e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:23:09.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.211+0000 7fe9cdf76700 1 -- 172.21.15.90:0/4237546304 shutdown_connections 2024-04-14T15:23:09.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.211+0000 7fe9cdf76700 1 -- 172.21.15.90:0/4237546304 wait complete. 2024-04-14T15:23:09.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.211+0000 7fe9cdf76700 1 librados: starting msgr at 2024-04-14T15:23:09.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.211+0000 7fe9cdf76700 1 librados: starting objecter 2024-04-14T15:23:09.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.211+0000 7fe9cdf76700 1 Processor -- start 2024-04-14T15:23:09.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.212+0000 7fe9cdf76700 1 -- start start 2024-04-14T15:23:09.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.212+0000 7fe9cdf76700 1 librados: setting wanted keys 2024-04-14T15:23:09.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.212+0000 7fe9cdf76700 1 librados: calling monclient init 2024-04-14T15:23:09.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.212+0000 7fe9cdf76700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe9c80769e0 con 0x7fe9c8108870 2024-04-14T15:23:09.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.212+0000 7fe9cdf76700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe9c8076bf0 con 0x7fe9c810f2f0 2024-04-14T15:23:09.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.212+0000 7fe9cdf76700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe9c81ac300 con 0x7fe9c810b720 2024-04-14T15:23:09.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.212+0000 7fe9c77fe700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fe9c8108870 0x7fe9c8074ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:43182/0 (socket says 172.21.15.90:43182) 2024-04-14T15:23:09.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.212+0000 7fe9c77fe700 1 -- 172.21.15.90:0/3887117836 learned_addr learned my addr 172.21.15.90:0/3887117836 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:23:09.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.212+0000 7fe9c4ff9700 10 client.?.objecter ms_handle_connect 0x7fe9c810b720 2024-04-14T15:23:09.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.212+0000 7fe9c4ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:23:09.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.212+0000 7fe9c4ff9700 10 client.?.objecter ms_handle_connect 0x7fe9c8108870 2024-04-14T15:23:09.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.212+0000 7fe9c4ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:23:09.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.212+0000 7fe9c4ff9700 10 client.?.objecter ms_handle_connect 0x7fe9c810f2f0 2024-04-14T15:23:09.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.212+0000 7fe9c4ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:23:09.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.212+0000 7fe9c4ff9700 1 -- 172.21.15.90:0/3887117836 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1236285923 0 0) 0x7fe9c8076bf0 con 0x7fe9c810f2f0 2024-04-14T15:23:09.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.212+0000 7fe9c4ff9700 1 -- 172.21.15.90:0/3887117836 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe9b40031f0 con 0x7fe9c810f2f0 2024-04-14T15:23:09.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.212+0000 7fe9c4ff9700 1 -- 172.21.15.90:0/3887117836 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2009311522 0 0) 0x7fe9c81ac300 con 0x7fe9c810b720 2024-04-14T15:23:09.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.212+0000 7fe9c4ff9700 1 -- 172.21.15.90:0/3887117836 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe9c8076bf0 con 0x7fe9c810b720 2024-04-14T15:23:09.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.212+0000 7fe9c4ff9700 1 -- 172.21.15.90:0/3887117836 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2684989752 0 0) 0x7fe9c80769e0 con 0x7fe9c8108870 2024-04-14T15:23:09.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.212+0000 7fe9c4ff9700 1 -- 172.21.15.90:0/3887117836 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe9c81ac300 con 0x7fe9c8108870 2024-04-14T15:23:09.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.212+0000 7fe9c4ff9700 1 -- 172.21.15.90:0/3887117836 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 799958569 0 0) 0x7fe9b40031f0 con 0x7fe9c810f2f0 2024-04-14T15:23:09.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.212+0000 7fe9c4ff9700 1 -- 172.21.15.90:0/3887117836 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe9c80769e0 con 0x7fe9c810f2f0 2024-04-14T15:23:09.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.212+0000 7fe9c4ff9700 1 -- 172.21.15.90:0/3887117836 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fe9b8001830 con 0x7fe9c810f2f0 2024-04-14T15:23:09.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.212+0000 7fe9c4ff9700 1 -- 172.21.15.90:0/3887117836 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 158668239 0 0) 0x7fe9c8076bf0 con 0x7fe9c810b720 2024-04-14T15:23:09.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.212+0000 7fe9c4ff9700 1 -- 172.21.15.90:0/3887117836 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe9b40031f0 con 0x7fe9c810b720 2024-04-14T15:23:09.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.212+0000 7fe9c4ff9700 1 -- 172.21.15.90:0/3887117836 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2685758996 0 0) 0x7fe9c81ac300 con 0x7fe9c8108870 2024-04-14T15:23:09.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.212+0000 7fe9c4ff9700 1 -- 172.21.15.90:0/3887117836 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe9c8076bf0 con 0x7fe9c8108870 2024-04-14T15:23:09.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.212+0000 7fe9c4ff9700 1 -- 172.21.15.90:0/3887117836 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fe9c0001bf0 con 0x7fe9c8108870 2024-04-14T15:23:09.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.212+0000 7fe9c4ff9700 1 -- 172.21.15.90:0/3887117836 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fe9b00015d0 con 0x7fe9c810b720 2024-04-14T15:23:09.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.213+0000 7fe9c4ff9700 1 -- 172.21.15.90:0/3887117836 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2765398263 0 0) 0x7fe9c80769e0 con 0x7fe9c810f2f0 2024-04-14T15:23:09.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.213+0000 7fe9c4ff9700 1 -- 172.21.15.90:0/3887117836 >> v1:172.21.15.90:6790/0 conn(0x7fe9c810b720 legacy=0x7fe9c80753f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:09.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.213+0000 7fe9c4ff9700 1 -- 172.21.15.90:0/3887117836 >> v1:172.21.15.90:6789/0 conn(0x7fe9c8108870 legacy=0x7fe9c8074ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:09.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.213+0000 7fe9c4ff9700 1 -- 172.21.15.90:0/3887117836 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9c81ad4e0 con 0x7fe9c810f2f0 2024-04-14T15:23:09.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.213+0000 7fe9cdf76700 1 -- 172.21.15.90:0/3887117836 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe9c81ac530 con 0x7fe9c810f2f0 2024-04-14T15:23:09.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.213+0000 7fe9cdf76700 10 client.49600.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:23:09.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.213+0000 7fe9cdf76700 1 -- 172.21.15.90:0/3887117836 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe9c81ac9f0 con 0x7fe9c810f2f0 2024-04-14T15:23:09.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.213+0000 7fe9cdf76700 1 librados: init done 2024-04-14T15:23:09.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.213+0000 7fe9c4ff9700 1 -- 172.21.15.90:0/3887117836 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fe9b8003b40 con 0x7fe9c810f2f0 2024-04-14T15:23:09.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.213+0000 7fe9c4ff9700 1 -- 172.21.15.90:0/3887117836 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fe9b80049a0 con 0x7fe9c810f2f0 2024-04-14T15:23:09.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.214+0000 7fe9cdf76700 1 -- 172.21.15.90:0/3887117836 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9a8005150 con 0x7fe9c810f2f0 2024-04-14T15:23:09.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.214+0000 7fe9c4ff9700 1 -- 172.21.15.90:0/3887117836 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fe9b80036a0 con 0x7fe9c810f2f0 2024-04-14T15:23:09.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.215+0000 7fe9c4ff9700 1 -- 172.21.15.90:0/3887117836 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(653..653 src has 467..653) ==== 8698+0+0 (unknown 2443847066 0 0) 0x7fe9b80898b0 con 0x7fe9c810f2f0 2024-04-14T15:23:09.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.215+0000 7fe9c4ff9700 10 client.49600.objecter ms_dispatch 0x7fe9c8001550 osd_map(653..653 src has 467..653) 2024-04-14T15:23:09.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.215+0000 7fe9c4ff9700 3 client.49600.objecter handle_osd_map got epochs [653,653] > 0 2024-04-14T15:23:09.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.215+0000 7fe9c4ff9700 3 client.49600.objecter handle_osd_map decoding full epoch 653 2024-04-14T15:23:09.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.215+0000 7fe9c4ff9700 20 client.49600.objecter dump_active .. 0 homeless 2024-04-14T15:23:09.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.215+0000 7fe9c4ff9700 10 client.49600.objecter ms_handle_connect 0x7fe9b4071320 2024-04-14T15:23:09.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.219+0000 7fe9c4ff9700 1 -- 172.21.15.90:0/3887117836 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fe9b80560a0 con 0x7fe9c810f2f0 2024-04-14T15:23:09.312 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:23:09.335 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:09.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:09.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:23:09.437 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:23:09.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.441+0000 7fe9cdf76700 1 -- 172.21.15.90:0/3887117836 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} v 0) -- 0x7fe9a8005c50 con 0x7fe9c810f2f0 2024-04-14T15:23:09.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:09.460 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:09.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:23:09.561 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:23:09.586 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:09.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:09.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:23:09.688 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:23:09.712 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:09.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:09.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:23:09.813 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:23:09.822 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-04-14T15:23:09.822 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-04-14T15:23:09.822 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 0 2024-04-14T15:23:09.822 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-04-14T15:23:09.838 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:09.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:09.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:23:09.939 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:09.963 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:09.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:09.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:09.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.980+0000 7fe9c4ff9700 1 -- 172.21.15.90:0/3887117836 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]=0 set osd.1 primary-affinity to 1 (8655362) v654) ==== 133+0+0 (unknown 91828868 0 0) 0x7fe9b8059d50 con 0x7fe9c810f2f0 2024-04-14T15:23:09.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.983+0000 7fe9cdf76700 10 librados: watch_flush enter 2024-04-14T15:23:09.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.984+0000 7fe9cdf76700 10 librados: watch_flush exit 2024-04-14T15:23:09.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.984+0000 7fe9cdf76700 20 client.49600.objecter shutdown clearing up homeless session... 2024-04-14T15:23:09.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.984+0000 7fe9cdf76700 10 client.49600.objecter successfully canceled tick 2024-04-14T15:23:09.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.984+0000 7fe9cdf76700 1 -- 172.21.15.90:0/3887117836 >> v1:172.21.15.90:6800/420935427 conn(0x7fe9b4071320 legacy=0x7fe9b40737e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:09.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.984+0000 7fe9cdf76700 1 -- 172.21.15.90:0/3887117836 >> v1:172.21.15.92:6789/0 conn(0x7fe9c810f2f0 legacy=0x7fe9c81a8bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:09.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.986+0000 7fe9cdf76700 1 -- 172.21.15.90:0/3887117836 shutdown_connections 2024-04-14T15:23:09.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.986+0000 7fe9cdf76700 1 -- 172.21.15.90:0/3887117836 >> 172.21.15.90:0/3887117836 conn(0x7fe9c80fe3c0 msgr2=0x7fe9c8102870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:23:09.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.987+0000 7fe9cdf76700 1 -- 172.21.15.90:0/3887117836 shutdown_connections 2024-04-14T15:23:09.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.987+0000 7fe9cdf76700 1 -- 172.21.15.90:0/3887117836 wait complete. 2024-04-14T15:23:09.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:09.987+0000 7fe9cdf76700 1 librados: shutdown 2024-04-14T15:23:09.989 INFO:teuthology.orchestra.run.smithi090.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-04-14T15:23:10.064 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:10.087 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:10.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:10.088 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:10.189 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:10.212 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:10.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:10.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:10.314 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:23:10.337 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:10.338 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:10 smithi090 ceph-mon[41670]: pgmap v960: 216 pgs: 15 creating+peering, 1 active+remapped+backfill_toofull, 200 active+clean; 582 KiB data, 583 MiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s; 2/600 objects misplaced (0.333%) 2024-04-14T15:23:10.338 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:10 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3887117836' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-04-14T15:23:10.338 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:10 smithi090 ceph-mon[41670]: from='client.49600 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-04-14T15:23:10.338 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:10 smithi090 ceph-mon[41670]: from='client.49600 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-04-14T15:23:10.338 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:10 smithi090 ceph-mon[41670]: osdmap e654: 8 total, 8 up, 7 in 2024-04-14T15:23:10.339 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:10 smithi090 ceph-mon[30399]: pgmap v960: 216 pgs: 15 creating+peering, 1 active+remapped+backfill_toofull, 200 active+clean; 582 KiB data, 583 MiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s; 2/600 objects misplaced (0.333%) 2024-04-14T15:23:10.339 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:10 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3887117836' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-04-14T15:23:10.339 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:10 smithi090 ceph-mon[30399]: from='client.49600 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-04-14T15:23:10.339 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:10 smithi090 ceph-mon[30399]: from='client.49600 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-04-14T15:23:10.339 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:10 smithi090 ceph-mon[30399]: osdmap e654: 8 total, 8 up, 7 in 2024-04-14T15:23:10.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:10.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:23:10.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:10 smithi092 ceph-mon[33966]: pgmap v960: 216 pgs: 15 creating+peering, 1 active+remapped+backfill_toofull, 200 active+clean; 582 KiB data, 583 MiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s; 2/600 objects misplaced (0.333%) 2024-04-14T15:23:10.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:10 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3887117836' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-04-14T15:23:10.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:10 smithi092 ceph-mon[33966]: from='client.49600 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-04-14T15:23:10.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:10 smithi092 ceph-mon[33966]: from='client.49600 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-04-14T15:23:10.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:10 smithi092 ceph-mon[33966]: osdmap e654: 8 total, 8 up, 7 in 2024-04-14T15:23:10.441 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:23:10.464 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:10.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:10.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:23:10.500 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-14T15:23:10.566 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:10.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:10.607 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:10.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:10.708 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:10.732 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:10.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:10.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:10.779 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:23:10.834 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:23:10.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:10.858 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:10.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:23:10.959 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:23:10.985 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:10.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:10.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:23:11.088 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:23:11.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:11.112 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:11.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:23:11.196 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:11 smithi092 ceph-mon[33966]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:23:11.196 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:11 smithi092 ceph-mon[33966]: osdmap e655: 8 total, 8 up, 7 in 2024-04-14T15:23:11.196 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:11 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:23:11.196 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:11 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:23:11.213 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:11.236 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:11.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:11.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:11.321 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:11 smithi090 ceph-mon[41670]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:23:11.321 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:11 smithi090 ceph-mon[41670]: osdmap e655: 8 total, 8 up, 7 in 2024-04-14T15:23:11.322 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:11 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:23:11.322 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:11 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:23:11.322 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:11 smithi090 ceph-mon[30399]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:23:11.322 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:11 smithi090 ceph-mon[30399]: osdmap e655: 8 total, 8 up, 7 in 2024-04-14T15:23:11.322 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:11 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:23:11.322 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:11 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:23:11.338 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:23:11.361 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:11.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:11.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:23:11.463 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:23:11.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:11.487 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:11.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:23:11.588 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:23:11.612 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:11.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:11.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:23:11.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.668+0000 7f4971a34700 1 -- 172.21.15.90:0/3458720883 >> v1:172.21.15.90:6789/0 conn(0x7f496c10b740 legacy=0x7f496c10db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:11.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.668+0000 7f4971a34700 1 -- 172.21.15.90:0/3458720883 shutdown_connections 2024-04-14T15:23:11.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.668+0000 7f4971a34700 1 -- 172.21.15.90:0/3458720883 >> 172.21.15.90:0/3458720883 conn(0x7f496c0fe380 msgr2=0x7f496c1007a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:23:11.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.668+0000 7f4971a34700 1 -- 172.21.15.90:0/3458720883 shutdown_connections 2024-04-14T15:23:11.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.668+0000 7f4971a34700 1 -- 172.21.15.90:0/3458720883 wait complete. 2024-04-14T15:23:11.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.668+0000 7f4971a34700 1 librados: starting msgr at 2024-04-14T15:23:11.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.668+0000 7f4971a34700 1 librados: starting objecter 2024-04-14T15:23:11.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.668+0000 7f4971a34700 1 Processor -- start 2024-04-14T15:23:11.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.668+0000 7f4971a34700 1 -- start start 2024-04-14T15:23:11.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.668+0000 7f4971a34700 1 librados: setting wanted keys 2024-04-14T15:23:11.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.668+0000 7f4971a34700 1 librados: calling monclient init 2024-04-14T15:23:11.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.669+0000 7f4971a34700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f496c1a9d40 con 0x7f496c10f310 2024-04-14T15:23:11.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.669+0000 7f4971a34700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f496c1aaf60 con 0x7f496c108890 2024-04-14T15:23:11.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.669+0000 7f4971a34700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f496c1ac1a0 con 0x7f496c10b740 2024-04-14T15:23:11.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.669+0000 7f496b7fe700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f496c10f310 0x7f496c1a8400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:39132/0 (socket says 172.21.15.90:39132) 2024-04-14T15:23:11.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.669+0000 7f496b7fe700 1 -- 172.21.15.90:0/2614318357 learned_addr learned my addr 172.21.15.90:0/2614318357 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:23:11.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.669+0000 7f496affd700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f496c108890 0x7f496c076750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.90:34274/0 (socket says 172.21.15.90:34274) 2024-04-14T15:23:11.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.669+0000 7f4970a32700 10 client.?.objecter ms_handle_connect 0x7f496c10f310 2024-04-14T15:23:11.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.669+0000 7f4970a32700 10 client.?.objecter resend_mon_ops 2024-04-14T15:23:11.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.669+0000 7f4970a32700 10 client.?.objecter ms_handle_connect 0x7f496c108890 2024-04-14T15:23:11.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.669+0000 7f4970a32700 10 client.?.objecter resend_mon_ops 2024-04-14T15:23:11.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.669+0000 7f4970a32700 10 client.?.objecter ms_handle_connect 0x7f496c10b740 2024-04-14T15:23:11.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.669+0000 7f4970a32700 10 client.?.objecter resend_mon_ops 2024-04-14T15:23:11.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.669+0000 7f4970a32700 1 -- 172.21.15.90:0/2614318357 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2833195598 0 0) 0x7f496c1a9d40 con 0x7f496c10f310 2024-04-14T15:23:11.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.669+0000 7f4970a32700 1 -- 172.21.15.90:0/2614318357 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f49540035b0 con 0x7f496c10f310 2024-04-14T15:23:11.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.669+0000 7f4970a32700 1 -- 172.21.15.90:0/2614318357 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2779019367 0 0) 0x7f496c1aaf60 con 0x7f496c108890 2024-04-14T15:23:11.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.669+0000 7f4970a32700 1 -- 172.21.15.90:0/2614318357 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f496c1a9d40 con 0x7f496c108890 2024-04-14T15:23:11.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.669+0000 7f4970a32700 1 -- 172.21.15.90:0/2614318357 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 584219674 0 0) 0x7f496c1ac1a0 con 0x7f496c10b740 2024-04-14T15:23:11.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.669+0000 7f4970a32700 1 -- 172.21.15.90:0/2614318357 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f496c1aaf60 con 0x7f496c10b740 2024-04-14T15:23:11.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.669+0000 7f4970a32700 1 -- 172.21.15.90:0/2614318357 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3389247650 0 0) 0x7f496c1a9d40 con 0x7f496c108890 2024-04-14T15:23:11.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.669+0000 7f4970a32700 1 -- 172.21.15.90:0/2614318357 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f496c1ac1a0 con 0x7f496c108890 2024-04-14T15:23:11.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.669+0000 7f4970a32700 1 -- 172.21.15.90:0/2614318357 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f49600018d0 con 0x7f496c108890 2024-04-14T15:23:11.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.669+0000 7f4970a32700 1 -- 172.21.15.90:0/2614318357 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 775354352 0 0) 0x7f49540035b0 con 0x7f496c10f310 2024-04-14T15:23:11.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.669+0000 7f4970a32700 1 -- 172.21.15.90:0/2614318357 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f496c1a9d40 con 0x7f496c10f310 2024-04-14T15:23:11.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.669+0000 7f4970a32700 1 -- 172.21.15.90:0/2614318357 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f4964001c70 con 0x7f496c10f310 2024-04-14T15:23:11.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.669+0000 7f4970a32700 1 -- 172.21.15.90:0/2614318357 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3361074092 0 0) 0x7f496c1aaf60 con 0x7f496c10b740 2024-04-14T15:23:11.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.669+0000 7f4970a32700 1 -- 172.21.15.90:0/2614318357 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f49540035b0 con 0x7f496c10b740 2024-04-14T15:23:11.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.669+0000 7f4970a32700 1 -- 172.21.15.90:0/2614318357 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f495c001430 con 0x7f496c10b740 2024-04-14T15:23:11.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.669+0000 7f4970a32700 1 -- 172.21.15.90:0/2614318357 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3845042963 0 0) 0x7f496c1ac1a0 con 0x7f496c108890 2024-04-14T15:23:11.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.669+0000 7f4970a32700 1 -- 172.21.15.90:0/2614318357 >> v1:172.21.15.90:6790/0 conn(0x7f496c10b740 legacy=0x7f496c1a4bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:11.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.669+0000 7f4970a32700 1 -- 172.21.15.90:0/2614318357 >> v1:172.21.15.90:6789/0 conn(0x7f496c10f310 legacy=0x7f496c1a8400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:11.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.669+0000 7f4970a32700 1 -- 172.21.15.90:0/2614318357 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f496c1ad3e0 con 0x7f496c108890 2024-04-14T15:23:11.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.670+0000 7f4971a34700 1 -- 172.21.15.90:0/2614318357 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f496c1ab190 con 0x7f496c108890 2024-04-14T15:23:11.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.670+0000 7f4970a32700 1 -- 172.21.15.90:0/2614318357 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f4960003b40 con 0x7f496c108890 2024-04-14T15:23:11.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.670+0000 7f4970a32700 1 -- 172.21.15.90:0/2614318357 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f4960005ca0 con 0x7f496c108890 2024-04-14T15:23:11.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.670+0000 7f4971a34700 10 client.49606.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:23:11.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.670+0000 7f4971a34700 1 -- 172.21.15.90:0/2614318357 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f496c1ab650 con 0x7f496c108890 2024-04-14T15:23:11.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.670+0000 7f4971a34700 1 librados: init done 2024-04-14T15:23:11.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.670+0000 7f4971a34700 1 -- 172.21.15.90:0/2614318357 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f496c103fd0 con 0x7f496c108890 2024-04-14T15:23:11.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.671+0000 7f4970a32700 1 -- 172.21.15.90:0/2614318357 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f49600036a0 con 0x7f496c108890 2024-04-14T15:23:11.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.672+0000 7f4970a32700 1 -- 172.21.15.90:0/2614318357 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(655..655 src has 467..655) ==== 8698+0+0 (unknown 2471904097 0 0) 0x7f496008ab30 con 0x7f496c108890 2024-04-14T15:23:11.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.672+0000 7f4970a32700 10 client.49606.objecter ms_dispatch 0x7f496c001550 osd_map(655..655 src has 467..655) 2024-04-14T15:23:11.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.672+0000 7f4970a32700 3 client.49606.objecter handle_osd_map got epochs [655,655] > 0 2024-04-14T15:23:11.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.672+0000 7f4970a32700 3 client.49606.objecter handle_osd_map decoding full epoch 655 2024-04-14T15:23:11.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.672+0000 7f4970a32700 20 client.49606.objecter dump_active .. 0 homeless 2024-04-14T15:23:11.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.672+0000 7f4970a32700 1 -- 172.21.15.90:0/2614318357 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v655) ==== 54+0+16 (unknown 3915482457 0 2420000192) 0x7f496008b060 con 0x7f496c108890 2024-04-14T15:23:11.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.672+0000 7f4970a32700 10 client.49606.objecter ms_handle_connect 0x7f49540713f0 2024-04-14T15:23:11.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.672+0000 7f4971a34700 10 client.49606.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:23:11.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.672+0000 7f4971a34700 15 client.49606.objecter _session_command_op_assign -1 1 2024-04-14T15:23:11.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.672+0000 7f4971a34700 20 client.49606.objecter _get_session s=0x7f496c1132e0 osd=0 3 2024-04-14T15:23:11.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.672+0000 7f4971a34700 20 client.49606.objecter put_session s=0x7f496c1132e0 osd=0 3 2024-04-14T15:23:11.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.672+0000 7f4971a34700 20 client.49606.objecter _get_session s=0x7f496c1132e0 osd=0 3 2024-04-14T15:23:11.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.672+0000 7f4971a34700 15 client.49606.objecter _session_command_op_remove -1 1 2024-04-14T15:23:11.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.672+0000 7f4971a34700 20 client.49606.objecter get_session s=0x7f496c1132e0 osd=0 3 2024-04-14T15:23:11.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.672+0000 7f4971a34700 15 client.49606.objecter _session_command_op_assign 0 1 2024-04-14T15:23:11.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.672+0000 7f4971a34700 20 client.49606.objecter put_session s=0x7f496c1132e0 osd=0 4 2024-04-14T15:23:11.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.672+0000 7f4971a34700 10 client.49606.objecter _send_command 1 2024-04-14T15:23:11.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.672+0000 7f4971a34700 1 -- 172.21.15.90:0/2614318357 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f496c076210 con 0x7f496c1ac880 2024-04-14T15:23:11.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.672+0000 7f4970a32700 10 client.49606.objecter ms_handle_connect 0x7f496c1ac880 2024-04-14T15:23:11.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.673+0000 7f4970a32700 1 -- 172.21.15.90:0/2614318357 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f496c076210 con 0x7f496c1ac880 2024-04-14T15:23:11.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.673+0000 7f4970a32700 10 client.49606.objecter ms_dispatch 0x7f496c001550 command_reply(tid 1: 0 ) 2024-04-14T15:23:11.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.673+0000 7f4970a32700 10 client.49606.objecter _finish_command 1 = system:0 2024-04-14T15:23:11.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.673+0000 7f4970a32700 20 client.49606.objecter put_session s=0x7f496c1132e0 osd=0 4 2024-04-14T15:23:11.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.673+0000 7f4970a32700 15 client.49606.objecter _session_command_op_remove 0 1 2024-04-14T15:23:11.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.690+0000 7f4971a34700 10 client.49606.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:23:11.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.690+0000 7f4971a34700 15 client.49606.objecter _session_command_op_assign -1 2 2024-04-14T15:23:11.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.690+0000 7f4971a34700 20 client.49606.objecter _get_session s=0x7f496c1132e0 osd=0 3 2024-04-14T15:23:11.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.690+0000 7f4971a34700 20 client.49606.objecter put_session s=0x7f496c1132e0 osd=0 3 2024-04-14T15:23:11.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.690+0000 7f4971a34700 20 client.49606.objecter _get_session s=0x7f496c1132e0 osd=0 3 2024-04-14T15:23:11.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.690+0000 7f4971a34700 15 client.49606.objecter _session_command_op_remove -1 2 2024-04-14T15:23:11.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.690+0000 7f4971a34700 20 client.49606.objecter get_session s=0x7f496c1132e0 osd=0 3 2024-04-14T15:23:11.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.690+0000 7f4971a34700 15 client.49606.objecter _session_command_op_assign 0 2 2024-04-14T15:23:11.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.690+0000 7f4971a34700 20 client.49606.objecter put_session s=0x7f496c1132e0 osd=0 4 2024-04-14T15:23:11.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.690+0000 7f4971a34700 10 client.49606.objecter _send_command 2 2024-04-14T15:23:11.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.690+0000 7f4971a34700 1 -- 172.21.15.90:0/2614318357 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f496c0693d0 con 0x7f496c1ac880 2024-04-14T15:23:11.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.691+0000 7f4970a32700 1 -- 172.21.15.90:0/2614318357 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f496c0693d0 con 0x7f496c1ac880 2024-04-14T15:23:11.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.691+0000 7f4970a32700 10 client.49606.objecter ms_dispatch 0x7f496c001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:23:11.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.691+0000 7f4970a32700 10 client.49606.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:23:11.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.691+0000 7f4970a32700 20 client.49606.objecter put_session s=0x7f496c1132e0 osd=0 4 2024-04-14T15:23:11.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.691+0000 7f4970a32700 15 client.49606.objecter _session_command_op_remove 0 2 2024-04-14T15:23:11.692 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:23:11.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.692+0000 7f4971a34700 10 client.49606.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:23:11.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.692+0000 7f4971a34700 15 client.49606.objecter _session_command_op_assign -1 3 2024-04-14T15:23:11.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.692+0000 7f4971a34700 20 client.49606.objecter _get_session s=0x7f496c1abde0 osd=1 3 2024-04-14T15:23:11.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.692+0000 7f4971a34700 20 client.49606.objecter put_session s=0x7f496c1abde0 osd=1 3 2024-04-14T15:23:11.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.692+0000 7f4971a34700 20 client.49606.objecter _get_session s=0x7f496c1abde0 osd=1 3 2024-04-14T15:23:11.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.692+0000 7f4971a34700 15 client.49606.objecter _session_command_op_remove -1 3 2024-04-14T15:23:11.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.692+0000 7f4971a34700 20 client.49606.objecter get_session s=0x7f496c1abde0 osd=1 3 2024-04-14T15:23:11.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.692+0000 7f4971a34700 15 client.49606.objecter _session_command_op_assign 1 3 2024-04-14T15:23:11.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.692+0000 7f4971a34700 20 client.49606.objecter put_session s=0x7f496c1abde0 osd=1 4 2024-04-14T15:23:11.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.692+0000 7f4971a34700 10 client.49606.objecter _send_command 3 2024-04-14T15:23:11.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.692+0000 7f4971a34700 1 -- 172.21.15.90:0/2614318357 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f496c1abf50 con 0x7f496c0636c0 2024-04-14T15:23:11.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.692+0000 7f4970a32700 10 client.49606.objecter ms_handle_connect 0x7f496c0636c0 2024-04-14T15:23:11.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.692+0000 7f4970a32700 1 -- 172.21.15.90:0/2614318357 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f4960005ca0 con 0x7f496c0636c0 2024-04-14T15:23:11.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.692+0000 7f4970a32700 10 client.49606.objecter ms_dispatch 0x7f496c001550 command_reply(tid 3: 0 ) 2024-04-14T15:23:11.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.692+0000 7f4970a32700 10 client.49606.objecter _finish_command 3 = system:0 2024-04-14T15:23:11.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.692+0000 7f4970a32700 20 client.49606.objecter put_session s=0x7f496c1abde0 osd=1 4 2024-04-14T15:23:11.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.692+0000 7f4970a32700 15 client.49606.objecter _session_command_op_remove 1 3 2024-04-14T15:23:11.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.709+0000 7f4971a34700 10 client.49606.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:23:11.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.709+0000 7f4971a34700 15 client.49606.objecter _session_command_op_assign -1 4 2024-04-14T15:23:11.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.709+0000 7f4971a34700 20 client.49606.objecter _get_session s=0x7f496c1abde0 osd=1 3 2024-04-14T15:23:11.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.709+0000 7f4971a34700 20 client.49606.objecter put_session s=0x7f496c1abde0 osd=1 3 2024-04-14T15:23:11.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.709+0000 7f4971a34700 20 client.49606.objecter _get_session s=0x7f496c1abde0 osd=1 3 2024-04-14T15:23:11.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.709+0000 7f4971a34700 15 client.49606.objecter _session_command_op_remove -1 4 2024-04-14T15:23:11.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.709+0000 7f4971a34700 20 client.49606.objecter get_session s=0x7f496c1abde0 osd=1 3 2024-04-14T15:23:11.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.709+0000 7f4971a34700 15 client.49606.objecter _session_command_op_assign 1 4 2024-04-14T15:23:11.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.709+0000 7f4971a34700 20 client.49606.objecter put_session s=0x7f496c1abde0 osd=1 4 2024-04-14T15:23:11.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.709+0000 7f4971a34700 10 client.49606.objecter _send_command 4 2024-04-14T15:23:11.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.709+0000 7f4971a34700 1 -- 172.21.15.90:0/2614318357 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f496c063ae0 con 0x7f496c0636c0 2024-04-14T15:23:11.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.710+0000 7f4970a32700 1 -- 172.21.15.90:0/2614318357 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f496c063ae0 con 0x7f496c0636c0 2024-04-14T15:23:11.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.710+0000 7f4970a32700 10 client.49606.objecter ms_dispatch 0x7f496c001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:23:11.711 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:23:11.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.710+0000 7f4970a32700 10 client.49606.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:23:11.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.710+0000 7f4970a32700 20 client.49606.objecter put_session s=0x7f496c1abde0 osd=1 4 2024-04-14T15:23:11.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.710+0000 7f4970a32700 15 client.49606.objecter _session_command_op_remove 1 4 2024-04-14T15:23:11.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.711+0000 7f4971a34700 10 client.49606.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:23:11.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.711+0000 7f4971a34700 15 client.49606.objecter _session_command_op_assign -1 5 2024-04-14T15:23:11.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.711+0000 7f4971a34700 20 client.49606.objecter _get_session s=0x7f496c063ee0 osd=2 3 2024-04-14T15:23:11.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.711+0000 7f4971a34700 20 client.49606.objecter put_session s=0x7f496c063ee0 osd=2 3 2024-04-14T15:23:11.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.711+0000 7f4971a34700 20 client.49606.objecter _get_session s=0x7f496c063ee0 osd=2 3 2024-04-14T15:23:11.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.711+0000 7f4971a34700 15 client.49606.objecter _session_command_op_remove -1 5 2024-04-14T15:23:11.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.711+0000 7f4971a34700 20 client.49606.objecter get_session s=0x7f496c063ee0 osd=2 3 2024-04-14T15:23:11.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.711+0000 7f4971a34700 15 client.49606.objecter _session_command_op_assign 2 5 2024-04-14T15:23:11.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.711+0000 7f4971a34700 20 client.49606.objecter put_session s=0x7f496c063ee0 osd=2 4 2024-04-14T15:23:11.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.711+0000 7f4971a34700 10 client.49606.objecter _send_command 5 2024-04-14T15:23:11.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.711+0000 7f4971a34700 1 -- 172.21.15.90:0/2614318357 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f496c1b5740 con 0x7f496c1b1ba0 2024-04-14T15:23:11.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.711+0000 7f4970a32700 10 client.49606.objecter ms_handle_connect 0x7f496c1b1ba0 2024-04-14T15:23:11.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.712+0000 7f4970a32700 1 -- 172.21.15.90:0/2614318357 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f496c1b5740 con 0x7f496c1b1ba0 2024-04-14T15:23:11.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.712+0000 7f4970a32700 10 client.49606.objecter ms_dispatch 0x7f496c001550 command_reply(tid 5: 0 ) 2024-04-14T15:23:11.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.712+0000 7f4970a32700 10 client.49606.objecter _finish_command 5 = system:0 2024-04-14T15:23:11.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.712+0000 7f4970a32700 20 client.49606.objecter put_session s=0x7f496c063ee0 osd=2 4 2024-04-14T15:23:11.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.712+0000 7f4970a32700 15 client.49606.objecter _session_command_op_remove 2 5 2024-04-14T15:23:11.714 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:23:11.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.728+0000 7f4971a34700 10 client.49606.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:23:11.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.728+0000 7f4971a34700 15 client.49606.objecter _session_command_op_assign -1 6 2024-04-14T15:23:11.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.728+0000 7f4971a34700 20 client.49606.objecter _get_session s=0x7f496c063ee0 osd=2 3 2024-04-14T15:23:11.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.728+0000 7f4971a34700 20 client.49606.objecter put_session s=0x7f496c063ee0 osd=2 3 2024-04-14T15:23:11.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.728+0000 7f4971a34700 20 client.49606.objecter _get_session s=0x7f496c063ee0 osd=2 3 2024-04-14T15:23:11.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.728+0000 7f4971a34700 15 client.49606.objecter _session_command_op_remove -1 6 2024-04-14T15:23:11.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.728+0000 7f4971a34700 20 client.49606.objecter get_session s=0x7f496c063ee0 osd=2 3 2024-04-14T15:23:11.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.728+0000 7f4971a34700 15 client.49606.objecter _session_command_op_assign 2 6 2024-04-14T15:23:11.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.728+0000 7f4971a34700 20 client.49606.objecter put_session s=0x7f496c063ee0 osd=2 4 2024-04-14T15:23:11.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.728+0000 7f4971a34700 10 client.49606.objecter _send_command 6 2024-04-14T15:23:11.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.728+0000 7f4971a34700 1 -- 172.21.15.90:0/2614318357 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f496c063200 con 0x7f496c1b1ba0 2024-04-14T15:23:11.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.730+0000 7f4970a32700 1 -- 172.21.15.90:0/2614318357 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f496c063200 con 0x7f496c1b1ba0 2024-04-14T15:23:11.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.730+0000 7f4970a32700 10 client.49606.objecter ms_dispatch 0x7f496c001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:23:11.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.730+0000 7f4970a32700 10 client.49606.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:23:11.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.730+0000 7f4970a32700 20 client.49606.objecter put_session s=0x7f496c063ee0 osd=2 4 2024-04-14T15:23:11.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.730+0000 7f4970a32700 15 client.49606.objecter _session_command_op_remove 2 6 2024-04-14T15:23:11.731 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:23:11.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.730+0000 7f4971a34700 10 client.49606.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:23:11.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.730+0000 7f4971a34700 15 client.49606.objecter _session_command_op_assign -1 7 2024-04-14T15:23:11.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.730+0000 7f4971a34700 20 client.49606.objecter _get_session s=0x7f496c02d5f0 osd=3 3 2024-04-14T15:23:11.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.730+0000 7f4971a34700 20 client.49606.objecter put_session s=0x7f496c02d5f0 osd=3 3 2024-04-14T15:23:11.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.730+0000 7f4971a34700 20 client.49606.objecter _get_session s=0x7f496c02d5f0 osd=3 3 2024-04-14T15:23:11.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.730+0000 7f4971a34700 15 client.49606.objecter _session_command_op_remove -1 7 2024-04-14T15:23:11.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.730+0000 7f4971a34700 20 client.49606.objecter get_session s=0x7f496c02d5f0 osd=3 3 2024-04-14T15:23:11.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.730+0000 7f4971a34700 15 client.49606.objecter _session_command_op_assign 3 7 2024-04-14T15:23:11.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.730+0000 7f4971a34700 20 client.49606.objecter put_session s=0x7f496c02d5f0 osd=3 4 2024-04-14T15:23:11.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.730+0000 7f4971a34700 10 client.49606.objecter _send_command 7 2024-04-14T15:23:11.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.730+0000 7f4971a34700 1 -- 172.21.15.90:0/2614318357 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f496c02dd90 con 0x7f496c1b5920 2024-04-14T15:23:11.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.730+0000 7f4970a32700 10 client.49606.objecter ms_handle_connect 0x7f496c1b5920 2024-04-14T15:23:11.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.731+0000 7f4970a32700 1 -- 172.21.15.90:0/2614318357 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f496c02dd90 con 0x7f496c1b5920 2024-04-14T15:23:11.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.731+0000 7f4970a32700 10 client.49606.objecter ms_dispatch 0x7f496c001550 command_reply(tid 7: 0 ) 2024-04-14T15:23:11.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.731+0000 7f4970a32700 10 client.49606.objecter _finish_command 7 = system:0 2024-04-14T15:23:11.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.731+0000 7f4970a32700 20 client.49606.objecter put_session s=0x7f496c02d5f0 osd=3 4 2024-04-14T15:23:11.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.731+0000 7f4970a32700 15 client.49606.objecter _session_command_op_remove 3 7 2024-04-14T15:23:11.737 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:11.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:11.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:23:11.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.747+0000 7f4971a34700 10 client.49606.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:23:11.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.747+0000 7f4971a34700 15 client.49606.objecter _session_command_op_assign -1 8 2024-04-14T15:23:11.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.747+0000 7f4971a34700 20 client.49606.objecter _get_session s=0x7f496c02d5f0 osd=3 3 2024-04-14T15:23:11.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.747+0000 7f4971a34700 20 client.49606.objecter put_session s=0x7f496c02d5f0 osd=3 3 2024-04-14T15:23:11.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.747+0000 7f4971a34700 20 client.49606.objecter _get_session s=0x7f496c02d5f0 osd=3 3 2024-04-14T15:23:11.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.747+0000 7f4971a34700 15 client.49606.objecter _session_command_op_remove -1 8 2024-04-14T15:23:11.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.747+0000 7f4971a34700 20 client.49606.objecter get_session s=0x7f496c02d5f0 osd=3 3 2024-04-14T15:23:11.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.747+0000 7f4971a34700 15 client.49606.objecter _session_command_op_assign 3 8 2024-04-14T15:23:11.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.747+0000 7f4971a34700 20 client.49606.objecter put_session s=0x7f496c02d5f0 osd=3 4 2024-04-14T15:23:11.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.747+0000 7f4971a34700 10 client.49606.objecter _send_command 8 2024-04-14T15:23:11.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.747+0000 7f4971a34700 1 -- 172.21.15.90:0/2614318357 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f496c1b7e10 con 0x7f496c1b5920 2024-04-14T15:23:11.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.749+0000 7f4970a32700 1 -- 172.21.15.90:0/2614318357 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f496c1b7e10 con 0x7f496c1b5920 2024-04-14T15:23:11.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.749+0000 7f4970a32700 10 client.49606.objecter ms_dispatch 0x7f496c001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:23:11.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.749+0000 7f4970a32700 10 client.49606.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:23:11.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.749+0000 7f4970a32700 20 client.49606.objecter put_session s=0x7f496c02d5f0 osd=3 4 2024-04-14T15:23:11.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.749+0000 7f4970a32700 15 client.49606.objecter _session_command_op_remove 3 8 2024-04-14T15:23:11.750 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:23:11.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.749+0000 7f4971a34700 10 client.49606.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:23:11.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.749+0000 7f4971a34700 15 client.49606.objecter _session_command_op_assign -1 9 2024-04-14T15:23:11.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.749+0000 7f4971a34700 20 client.49606.objecter _get_session s=0x7f496c02d380 osd=4 3 2024-04-14T15:23:11.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.749+0000 7f4971a34700 20 client.49606.objecter put_session s=0x7f496c02d380 osd=4 3 2024-04-14T15:23:11.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.749+0000 7f4971a34700 20 client.49606.objecter _get_session s=0x7f496c02d380 osd=4 3 2024-04-14T15:23:11.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.749+0000 7f4971a34700 15 client.49606.objecter _session_command_op_remove -1 9 2024-04-14T15:23:11.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.749+0000 7f4971a34700 20 client.49606.objecter get_session s=0x7f496c02d380 osd=4 3 2024-04-14T15:23:11.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.749+0000 7f4971a34700 15 client.49606.objecter _session_command_op_assign 4 9 2024-04-14T15:23:11.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.749+0000 7f4971a34700 20 client.49606.objecter put_session s=0x7f496c02d380 osd=4 4 2024-04-14T15:23:11.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.749+0000 7f4971a34700 10 client.49606.objecter _send_command 9 2024-04-14T15:23:11.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.749+0000 7f4971a34700 1 -- 172.21.15.90:0/2614318357 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f496c1bf030 con 0x7f496c1b7e10 2024-04-14T15:23:11.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.749+0000 7f4970a32700 10 client.49606.objecter ms_handle_connect 0x7f496c1b7e10 2024-04-14T15:23:11.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.750+0000 7f4970a32700 1 -- 172.21.15.90:0/2614318357 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f496c1bf030 con 0x7f496c1b7e10 2024-04-14T15:23:11.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.750+0000 7f4970a32700 10 client.49606.objecter ms_dispatch 0x7f496c001550 command_reply(tid 9: 0 ) 2024-04-14T15:23:11.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.750+0000 7f4970a32700 10 client.49606.objecter _finish_command 9 = system:0 2024-04-14T15:23:11.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.750+0000 7f4970a32700 20 client.49606.objecter put_session s=0x7f496c02d380 osd=4 4 2024-04-14T15:23:11.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.750+0000 7f4970a32700 15 client.49606.objecter _session_command_op_remove 4 9 2024-04-14T15:23:11.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.766+0000 7f4971a34700 10 client.49606.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:23:11.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.766+0000 7f4971a34700 15 client.49606.objecter _session_command_op_assign -1 10 2024-04-14T15:23:11.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.766+0000 7f4971a34700 20 client.49606.objecter _get_session s=0x7f496c02d380 osd=4 3 2024-04-14T15:23:11.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.766+0000 7f4971a34700 20 client.49606.objecter put_session s=0x7f496c02d380 osd=4 3 2024-04-14T15:23:11.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.766+0000 7f4971a34700 20 client.49606.objecter _get_session s=0x7f496c02d380 osd=4 3 2024-04-14T15:23:11.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.766+0000 7f4971a34700 15 client.49606.objecter _session_command_op_remove -1 10 2024-04-14T15:23:11.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.766+0000 7f4971a34700 20 client.49606.objecter get_session s=0x7f496c02d380 osd=4 3 2024-04-14T15:23:11.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.766+0000 7f4971a34700 15 client.49606.objecter _session_command_op_assign 4 10 2024-04-14T15:23:11.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.766+0000 7f4971a34700 20 client.49606.objecter put_session s=0x7f496c02d380 osd=4 4 2024-04-14T15:23:11.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.766+0000 7f4971a34700 10 client.49606.objecter _send_command 10 2024-04-14T15:23:11.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.766+0000 7f4971a34700 1 -- 172.21.15.90:0/2614318357 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f496c1beff0 con 0x7f496c1b7e10 2024-04-14T15:23:11.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.768+0000 7f4970a32700 1 -- 172.21.15.90:0/2614318357 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f496c1beff0 con 0x7f496c1b7e10 2024-04-14T15:23:11.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.768+0000 7f4970a32700 10 client.49606.objecter ms_dispatch 0x7f496c001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:23:11.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.768+0000 7f4970a32700 10 client.49606.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:23:11.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.768+0000 7f4970a32700 20 client.49606.objecter put_session s=0x7f496c02d380 osd=4 4 2024-04-14T15:23:11.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.768+0000 7f4970a32700 15 client.49606.objecter _session_command_op_remove 4 10 2024-04-14T15:23:11.769 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:23:11.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.768+0000 7f4971a34700 10 client.49606.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:23:11.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.768+0000 7f4971a34700 15 client.49606.objecter _session_command_op_assign -1 11 2024-04-14T15:23:11.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.768+0000 7f4971a34700 20 client.49606.objecter _get_session s=0x7f496c1beff0 osd=5 3 2024-04-14T15:23:11.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.768+0000 7f4971a34700 20 client.49606.objecter put_session s=0x7f496c1beff0 osd=5 3 2024-04-14T15:23:11.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.768+0000 7f4971a34700 20 client.49606.objecter _get_session s=0x7f496c1beff0 osd=5 3 2024-04-14T15:23:11.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.768+0000 7f4971a34700 15 client.49606.objecter _session_command_op_remove -1 11 2024-04-14T15:23:11.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.768+0000 7f4971a34700 20 client.49606.objecter get_session s=0x7f496c1beff0 osd=5 3 2024-04-14T15:23:11.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.768+0000 7f4971a34700 15 client.49606.objecter _session_command_op_assign 5 11 2024-04-14T15:23:11.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.768+0000 7f4971a34700 20 client.49606.objecter put_session s=0x7f496c1beff0 osd=5 4 2024-04-14T15:23:11.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.768+0000 7f4971a34700 10 client.49606.objecter _send_command 11 2024-04-14T15:23:11.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.768+0000 7f4971a34700 1 -- 172.21.15.90:0/2614318357 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f496c1c07c0 con 0x7f496c1bfc20 2024-04-14T15:23:11.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.769+0000 7f4970a32700 10 client.49606.objecter ms_handle_connect 0x7f496c1bfc20 2024-04-14T15:23:11.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.770+0000 7f4970a32700 1 -- 172.21.15.90:0/2614318357 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f496c1c07c0 con 0x7f496c1bfc20 2024-04-14T15:23:11.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.770+0000 7f4970a32700 10 client.49606.objecter ms_dispatch 0x7f496c001550 command_reply(tid 11: 0 ) 2024-04-14T15:23:11.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.770+0000 7f4970a32700 10 client.49606.objecter _finish_command 11 = system:0 2024-04-14T15:23:11.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.770+0000 7f4970a32700 20 client.49606.objecter put_session s=0x7f496c1beff0 osd=5 4 2024-04-14T15:23:11.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.770+0000 7f4970a32700 15 client.49606.objecter _session_command_op_remove 5 11 2024-04-14T15:23:11.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.786+0000 7f4971a34700 10 client.49606.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:23:11.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.786+0000 7f4971a34700 15 client.49606.objecter _session_command_op_assign -1 12 2024-04-14T15:23:11.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.786+0000 7f4971a34700 20 client.49606.objecter _get_session s=0x7f496c1beff0 osd=5 3 2024-04-14T15:23:11.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.786+0000 7f4971a34700 20 client.49606.objecter put_session s=0x7f496c1beff0 osd=5 3 2024-04-14T15:23:11.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.786+0000 7f4971a34700 20 client.49606.objecter _get_session s=0x7f496c1beff0 osd=5 3 2024-04-14T15:23:11.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.786+0000 7f4971a34700 15 client.49606.objecter _session_command_op_remove -1 12 2024-04-14T15:23:11.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.786+0000 7f4971a34700 20 client.49606.objecter get_session s=0x7f496c1beff0 osd=5 3 2024-04-14T15:23:11.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.786+0000 7f4971a34700 15 client.49606.objecter _session_command_op_assign 5 12 2024-04-14T15:23:11.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.786+0000 7f4971a34700 20 client.49606.objecter put_session s=0x7f496c1beff0 osd=5 4 2024-04-14T15:23:11.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.786+0000 7f4971a34700 10 client.49606.objecter _send_command 12 2024-04-14T15:23:11.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.786+0000 7f4971a34700 1 -- 172.21.15.90:0/2614318357 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f496c1b7bd0 con 0x7f496c1bfc20 2024-04-14T15:23:11.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.788+0000 7f4970a32700 1 -- 172.21.15.90:0/2614318357 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f496c1b7bd0 con 0x7f496c1bfc20 2024-04-14T15:23:11.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.788+0000 7f4970a32700 10 client.49606.objecter ms_dispatch 0x7f496c001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:23:11.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.788+0000 7f4970a32700 10 client.49606.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:23:11.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.788+0000 7f4970a32700 20 client.49606.objecter put_session s=0x7f496c1beff0 osd=5 4 2024-04-14T15:23:11.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.788+0000 7f4970a32700 15 client.49606.objecter _session_command_op_remove 5 12 2024-04-14T15:23:11.789 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:23:11.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.788+0000 7f4971a34700 10 client.49606.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:23:11.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.788+0000 7f4971a34700 15 client.49606.objecter _session_command_op_assign -1 13 2024-04-14T15:23:11.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.788+0000 7f4971a34700 20 client.49606.objecter _get_session s=0x7f496c1c0820 osd=6 3 2024-04-14T15:23:11.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.788+0000 7f4971a34700 20 client.49606.objecter put_session s=0x7f496c1c0820 osd=6 3 2024-04-14T15:23:11.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.788+0000 7f4971a34700 20 client.49606.objecter _get_session s=0x7f496c1c0820 osd=6 3 2024-04-14T15:23:11.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.788+0000 7f4971a34700 15 client.49606.objecter _session_command_op_remove -1 13 2024-04-14T15:23:11.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.788+0000 7f4971a34700 20 client.49606.objecter get_session s=0x7f496c1c0820 osd=6 3 2024-04-14T15:23:11.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.788+0000 7f4971a34700 15 client.49606.objecter _session_command_op_assign 6 13 2024-04-14T15:23:11.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.788+0000 7f4971a34700 20 client.49606.objecter put_session s=0x7f496c1c0820 osd=6 4 2024-04-14T15:23:11.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.788+0000 7f4971a34700 10 client.49606.objecter _send_command 13 2024-04-14T15:23:11.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.788+0000 7f4971a34700 1 -- 172.21.15.90:0/2614318357 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f496c1c7a20 con 0x7f496c1c3f10 2024-04-14T15:23:11.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.788+0000 7f4970a32700 10 client.49606.objecter ms_handle_connect 0x7f496c1c3f10 2024-04-14T15:23:11.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.789+0000 7f4970a32700 1 -- 172.21.15.90:0/2614318357 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f496c1c7a20 con 0x7f496c1c3f10 2024-04-14T15:23:11.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.789+0000 7f4970a32700 10 client.49606.objecter ms_dispatch 0x7f496c001550 command_reply(tid 13: 0 ) 2024-04-14T15:23:11.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.789+0000 7f4970a32700 10 client.49606.objecter _finish_command 13 = system:0 2024-04-14T15:23:11.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.789+0000 7f4970a32700 20 client.49606.objecter put_session s=0x7f496c1c0820 osd=6 4 2024-04-14T15:23:11.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.789+0000 7f4970a32700 15 client.49606.objecter _session_command_op_remove 6 13 2024-04-14T15:23:11.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.805+0000 7f4971a34700 10 client.49606.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:23:11.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.805+0000 7f4971a34700 15 client.49606.objecter _session_command_op_assign -1 14 2024-04-14T15:23:11.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.805+0000 7f4971a34700 20 client.49606.objecter _get_session s=0x7f496c1c0820 osd=6 3 2024-04-14T15:23:11.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.805+0000 7f4971a34700 20 client.49606.objecter put_session s=0x7f496c1c0820 osd=6 3 2024-04-14T15:23:11.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.805+0000 7f4971a34700 20 client.49606.objecter _get_session s=0x7f496c1c0820 osd=6 3 2024-04-14T15:23:11.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.805+0000 7f4971a34700 15 client.49606.objecter _session_command_op_remove -1 14 2024-04-14T15:23:11.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.805+0000 7f4971a34700 20 client.49606.objecter get_session s=0x7f496c1c0820 osd=6 3 2024-04-14T15:23:11.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.805+0000 7f4971a34700 15 client.49606.objecter _session_command_op_assign 6 14 2024-04-14T15:23:11.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.805+0000 7f4971a34700 20 client.49606.objecter put_session s=0x7f496c1c0820 osd=6 4 2024-04-14T15:23:11.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.805+0000 7f4971a34700 10 client.49606.objecter _send_command 14 2024-04-14T15:23:11.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.805+0000 7f4971a34700 1 -- 172.21.15.90:0/2614318357 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f496c1bf4b0 con 0x7f496c1c3f10 2024-04-14T15:23:11.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.807+0000 7f4970a32700 1 -- 172.21.15.90:0/2614318357 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f496c1bf4b0 con 0x7f496c1c3f10 2024-04-14T15:23:11.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.807+0000 7f4970a32700 10 client.49606.objecter ms_dispatch 0x7f496c001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:23:11.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.807+0000 7f4970a32700 10 client.49606.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:23:11.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.807+0000 7f4970a32700 20 client.49606.objecter put_session s=0x7f496c1c0820 osd=6 4 2024-04-14T15:23:11.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.807+0000 7f4970a32700 15 client.49606.objecter _session_command_op_remove 6 14 2024-04-14T15:23:11.808 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:23:11.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.807+0000 7f4971a34700 10 client.49606.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:23:11.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.807+0000 7f4971a34700 15 client.49606.objecter _session_command_op_assign -1 15 2024-04-14T15:23:11.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.807+0000 7f4971a34700 20 client.49606.objecter _get_session s=0x7f496c1bf4b0 osd=7 3 2024-04-14T15:23:11.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.807+0000 7f4971a34700 20 client.49606.objecter put_session s=0x7f496c1bf4b0 osd=7 3 2024-04-14T15:23:11.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.807+0000 7f4971a34700 20 client.49606.objecter _get_session s=0x7f496c1bf4b0 osd=7 3 2024-04-14T15:23:11.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.807+0000 7f4971a34700 15 client.49606.objecter _session_command_op_remove -1 15 2024-04-14T15:23:11.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.807+0000 7f4971a34700 20 client.49606.objecter get_session s=0x7f496c1bf4b0 osd=7 3 2024-04-14T15:23:11.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.807+0000 7f4971a34700 15 client.49606.objecter _session_command_op_assign 7 15 2024-04-14T15:23:11.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.807+0000 7f4971a34700 20 client.49606.objecter put_session s=0x7f496c1bf4b0 osd=7 4 2024-04-14T15:23:11.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.807+0000 7f4971a34700 10 client.49606.objecter _send_command 15 2024-04-14T15:23:11.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.807+0000 7f4971a34700 1 -- 172.21.15.90:0/2614318357 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f496c1c9170 con 0x7f496c1ca080 2024-04-14T15:23:11.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.808+0000 7f4970a32700 10 client.49606.objecter ms_handle_connect 0x7f496c1ca080 2024-04-14T15:23:11.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.809+0000 7f4970a32700 1 -- 172.21.15.90:0/2614318357 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f496c1c9170 con 0x7f496c1ca080 2024-04-14T15:23:11.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.809+0000 7f4970a32700 10 client.49606.objecter ms_dispatch 0x7f496c001550 command_reply(tid 15: 0 ) 2024-04-14T15:23:11.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.809+0000 7f4970a32700 10 client.49606.objecter _finish_command 15 = system:0 2024-04-14T15:23:11.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.809+0000 7f4970a32700 20 client.49606.objecter put_session s=0x7f496c1bf4b0 osd=7 4 2024-04-14T15:23:11.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.809+0000 7f4970a32700 15 client.49606.objecter _session_command_op_remove 7 15 2024-04-14T15:23:11.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.825+0000 7f4971a34700 10 client.49606.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:23:11.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.825+0000 7f4971a34700 15 client.49606.objecter _session_command_op_assign -1 16 2024-04-14T15:23:11.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.825+0000 7f4971a34700 20 client.49606.objecter _get_session s=0x7f496c1bf4b0 osd=7 3 2024-04-14T15:23:11.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.825+0000 7f4971a34700 20 client.49606.objecter put_session s=0x7f496c1bf4b0 osd=7 3 2024-04-14T15:23:11.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.825+0000 7f4971a34700 20 client.49606.objecter _get_session s=0x7f496c1bf4b0 osd=7 3 2024-04-14T15:23:11.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.825+0000 7f4971a34700 15 client.49606.objecter _session_command_op_remove -1 16 2024-04-14T15:23:11.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.825+0000 7f4971a34700 20 client.49606.objecter get_session s=0x7f496c1bf4b0 osd=7 3 2024-04-14T15:23:11.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.825+0000 7f4971a34700 15 client.49606.objecter _session_command_op_assign 7 16 2024-04-14T15:23:11.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.825+0000 7f4971a34700 20 client.49606.objecter put_session s=0x7f496c1bf4b0 osd=7 4 2024-04-14T15:23:11.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.825+0000 7f4971a34700 10 client.49606.objecter _send_command 16 2024-04-14T15:23:11.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.825+0000 7f4971a34700 1 -- 172.21.15.90:0/2614318357 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f496c1c9940 con 0x7f496c1ca080 2024-04-14T15:23:11.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.827+0000 7f4970a32700 1 -- 172.21.15.90:0/2614318357 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f496c1c9940 con 0x7f496c1ca080 2024-04-14T15:23:11.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.827+0000 7f4970a32700 10 client.49606.objecter ms_dispatch 0x7f496c001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:23:11.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.827+0000 7f4970a32700 10 client.49606.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:23:11.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.827+0000 7f4970a32700 20 client.49606.objecter put_session s=0x7f496c1bf4b0 osd=7 4 2024-04-14T15:23:11.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.827+0000 7f4970a32700 15 client.49606.objecter _session_command_op_remove 7 16 2024-04-14T15:23:11.827 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:23:11.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.827+0000 7f4971a34700 10 librados: watch_flush enter 2024-04-14T15:23:11.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.827+0000 7f4971a34700 10 librados: watch_flush exit 2024-04-14T15:23:11.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.827+0000 7f4971a34700 10 client.49606.objecter close_session for osd.0 2024-04-14T15:23:11.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.827+0000 7f4971a34700 1 -- 172.21.15.90:0/2614318357 >> v1:172.21.15.90:6801/3761806885 conn(0x7f496c1ac880 legacy=0x7f496c1acc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:11.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.827+0000 7f4971a34700 20 client.49606.objecter put_session s=0x7f496c1132e0 osd=0 1 2024-04-14T15:23:11.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.827+0000 7f4971a34700 10 client.49606.objecter close_session for osd.1 2024-04-14T15:23:11.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.827+0000 7f4971a34700 1 -- 172.21.15.90:0/2614318357 >> v1:172.21.15.90:6805/4170361699 conn(0x7f496c0636c0 legacy=0x7f496c109680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:11.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.827+0000 7f4971a34700 20 client.49606.objecter put_session s=0x7f496c1abde0 osd=1 1 2024-04-14T15:23:11.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.827+0000 7f4971a34700 10 client.49606.objecter close_session for osd.2 2024-04-14T15:23:11.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.827+0000 7f4971a34700 1 -- 172.21.15.90:0/2614318357 >> v1:172.21.15.90:6809/1686184251 conn(0x7f496c1b1ba0 legacy=0x7f496c1b3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:11.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.827+0000 7f496b7fe700 1 -- 172.21.15.90:0/2614318357 reap_dead start 2024-04-14T15:23:11.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.827+0000 7f4971a34700 20 client.49606.objecter put_session s=0x7f496c063ee0 osd=2 1 2024-04-14T15:23:11.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.827+0000 7f4971a34700 10 client.49606.objecter close_session for osd.3 2024-04-14T15:23:11.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.827+0000 7f4971a34700 1 -- 172.21.15.90:0/2614318357 >> v1:172.21.15.90:6813/2270166774 conn(0x7f496c1b5920 legacy=0x7f496c1b5d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:11.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.827+0000 7f4971a34700 20 client.49606.objecter put_session s=0x7f496c02d5f0 osd=3 1 2024-04-14T15:23:11.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.827+0000 7f4971a34700 10 client.49606.objecter close_session for osd.4 2024-04-14T15:23:11.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.827+0000 7f4971a34700 1 -- 172.21.15.90:0/2614318357 >> v1:172.21.15.92:6800/3887549733 conn(0x7f496c1b7e10 legacy=0x7f496c1bd800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:11.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.827+0000 7f4971a34700 20 client.49606.objecter put_session s=0x7f496c02d380 osd=4 1 2024-04-14T15:23:11.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.827+0000 7f4971a34700 10 client.49606.objecter close_session for osd.5 2024-04-14T15:23:11.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.827+0000 7f4971a34700 1 -- 172.21.15.90:0/2614318357 >> v1:172.21.15.92:6804/3244561440 conn(0x7f496c1bfc20 legacy=0x7f496c1c0080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:11.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.827+0000 7f4971a34700 20 client.49606.objecter put_session s=0x7f496c1beff0 osd=5 1 2024-04-14T15:23:11.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.827+0000 7f4971a34700 10 client.49606.objecter close_session for osd.6 2024-04-14T15:23:11.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.827+0000 7f4971a34700 1 -- 172.21.15.90:0/2614318357 >> v1:172.21.15.92:6808/636125963 conn(0x7f496c1c3f10 legacy=0x7f496c1c6300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:11.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.827+0000 7f4971a34700 20 client.49606.objecter put_session s=0x7f496c1c0820 osd=6 1 2024-04-14T15:23:11.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.827+0000 7f4971a34700 10 client.49606.objecter close_session for osd.7 2024-04-14T15:23:11.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.827+0000 7f4971a34700 1 -- 172.21.15.90:0/2614318357 >> v1:172.21.15.92:6812/1773706716 conn(0x7f496c1ca080 legacy=0x7f496c1c7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:11.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.827+0000 7f4971a34700 20 client.49606.objecter put_session s=0x7f496c1bf4b0 osd=7 1 2024-04-14T15:23:11.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.827+0000 7f4971a34700 20 client.49606.objecter shutdown clearing up homeless session... 2024-04-14T15:23:11.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.827+0000 7f4971a34700 10 client.49606.objecter successfully canceled tick 2024-04-14T15:23:11.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.827+0000 7f496b7fe700 1 -- 172.21.15.90:0/2614318357 reap_dead start 2024-04-14T15:23:11.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.827+0000 7f4971a34700 1 -- 172.21.15.90:0/2614318357 >> v1:172.21.15.90:6800/420935427 conn(0x7f49540713f0 legacy=0x7f49540738b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:11.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.827+0000 7f4971a34700 1 -- 172.21.15.90:0/2614318357 >> v1:172.21.15.92:6789/0 conn(0x7f496c108890 legacy=0x7f496c076750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:11.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.827+0000 7f4971a34700 1 -- 172.21.15.90:0/2614318357 shutdown_connections 2024-04-14T15:23:11.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.827+0000 7f4971a34700 1 -- 172.21.15.90:0/2614318357 >> 172.21.15.90:0/2614318357 conn(0x7f496c0fe380 msgr2=0x7f496c1127a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:23:11.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.827+0000 7f4971a34700 1 -- 172.21.15.90:0/2614318357 shutdown_connections 2024-04-14T15:23:11.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.827+0000 7f4971a34700 1 -- 172.21.15.90:0/2614318357 wait complete. 2024-04-14T15:23:11.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:11.828+0000 7f4971a34700 1 librados: shutdown 2024-04-14T15:23:11.831 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:23:11.831 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:23:11.832 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:23:11.832 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:23:11.832 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:23:11.832 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:23:11.832 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:23:11.832 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:23:11.838 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:23:11.862 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:11.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:11.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:23:11.963 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:23:11.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:11.990 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:11.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:23:12.091 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:23:12.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:12.118 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:12.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:23:12.162 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:12 smithi090 ceph-mon[41670]: pgmap v963: 216 pgs: 32 unknown, 1 active+remapped+backfill_toofull, 183 active+clean; 582 KiB data, 583 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s; 2/600 objects misplaced (0.333%) 2024-04-14T15:23:12.163 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:12 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2614318357' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:23:12.163 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:12 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:23:12.163 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:12 smithi090 ceph-mon[41670]: osdmap e656: 8 total, 8 up, 7 in 2024-04-14T15:23:12.163 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:12 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:23:12.163 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:12 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:23:12.163 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:12 smithi090 ceph-mon[30399]: pgmap v963: 216 pgs: 32 unknown, 1 active+remapped+backfill_toofull, 183 active+clean; 582 KiB data, 583 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s; 2/600 objects misplaced (0.333%) 2024-04-14T15:23:12.163 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:12 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2614318357' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:23:12.163 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:12 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:23:12.163 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:12 smithi090 ceph-mon[30399]: osdmap e656: 8 total, 8 up, 7 in 2024-04-14T15:23:12.163 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:12 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:23:12.163 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:12 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:23:12.220 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:23:12.243 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:12.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:12.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:23:12.345 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:23:12.369 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:12.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:12.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:23:12.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:12 smithi092 ceph-mon[33966]: pgmap v963: 216 pgs: 32 unknown, 1 active+remapped+backfill_toofull, 183 active+clean; 582 KiB data, 583 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s; 2/600 objects misplaced (0.333%) 2024-04-14T15:23:12.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:12 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2614318357' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:23:12.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:12 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:23:12.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:12 smithi092 ceph-mon[33966]: osdmap e656: 8 total, 8 up, 7 in 2024-04-14T15:23:12.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:12 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:23:12.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:12 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:23:12.471 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:23:12.496 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:12.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:12.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:23:12.597 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:23:12.619 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-04-14T15:23:12.620 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:23:12.621 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:12.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:12.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:23:12.723 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:23:12.746 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:12.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:12.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:23:12.848 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:23:12.873 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:12.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:12.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:23:12.974 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:23:13.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:13.001 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:13.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:23:13.102 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:23:13.126 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:13.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:13.127 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:23:13.205 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-14T15:23:13.228 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:23:13.252 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:13.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:13.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:23:13.317 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:13 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:23:13.384 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:13 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:23:13.384 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:13 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:23:13.384 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:13 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:23:13.384 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:13 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-114", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:23:13.384 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:13 smithi092 ceph-mon[33966]: osdmap e657: 8 total, 8 up, 7 in 2024-04-14T15:23:13.384 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:13 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-114"}]: dispatch 2024-04-14T15:23:13.384 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:13 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-114"}]: dispatch 2024-04-14T15:23:13.385 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:23:13.386 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-14T15:23:13.410 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:13.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:13.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:23:13.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:13 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:23:13.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:13 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:23:13.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:13 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:23:13.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:13 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:23:13.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:13 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-114", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:23:13.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:13 smithi090 ceph-mon[41670]: osdmap e657: 8 total, 8 up, 7 in 2024-04-14T15:23:13.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:13 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-114"}]: dispatch 2024-04-14T15:23:13.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:13 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-114"}]: dispatch 2024-04-14T15:23:13.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:13 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:23:13.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:13 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:23:13.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:13 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:23:13.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:13 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:23:13.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:13 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-114", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:23:13.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:13 smithi090 ceph-mon[30399]: osdmap e657: 8 total, 8 up, 7 in 2024-04-14T15:23:13.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:13 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-114"}]: dispatch 2024-04-14T15:23:13.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:13 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-114"}]: dispatch 2024-04-14T15:23:13.449 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:23:13.512 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:23:13.535 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:13.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:13.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:23:13.624 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:23:13.637 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:23:13.661 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:13.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:13.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:23:13.763 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:13.787 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:13.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:13.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:13.888 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:23:13.913 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:13.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:13.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:23:14.015 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:23:14.039 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:14.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:14.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:23:14.141 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:23:14.164 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:14.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:14.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:23:14.191 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:23:13 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:23:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:23:14.265 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:14.289 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:14.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:14.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:14.391 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:23:14.415 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:14.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:14.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:23:14.517 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:23:14.540 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:14.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:14.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:23:14.625 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-14T15:23:14.642 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:23:14.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:14.667 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:14.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:23:14.768 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:23:14.792 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:14.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:14.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:23:14.893 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:14.917 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:14.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:14.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:15.019 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:23:15.043 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:15.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:15.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:23:15.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.130+0000 7f89b7a90700 1 -- 172.21.15.90:0/653242705 >> v1:172.21.15.90:6789/0 conn(0x7f89b0108870 legacy=0x7f89b0108c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:15.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.130+0000 7f89b7a90700 1 -- 172.21.15.90:0/653242705 shutdown_connections 2024-04-14T15:23:15.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.130+0000 7f89b7a90700 1 -- 172.21.15.90:0/653242705 >> 172.21.15.90:0/653242705 conn(0x7f89b00fe3a0 msgr2=0x7f89b01007c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:23:15.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.130+0000 7f89b7a90700 1 -- 172.21.15.90:0/653242705 shutdown_connections 2024-04-14T15:23:15.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.130+0000 7f89b7a90700 1 -- 172.21.15.90:0/653242705 wait complete. 2024-04-14T15:23:15.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.130+0000 7f89b7a90700 1 librados: starting msgr at 2024-04-14T15:23:15.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.130+0000 7f89b7a90700 1 librados: starting objecter 2024-04-14T15:23:15.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.130+0000 7f89b7a90700 1 Processor -- start 2024-04-14T15:23:15.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.131+0000 7f89b7a90700 1 -- start start 2024-04-14T15:23:15.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.131+0000 7f89b7a90700 1 librados: setting wanted keys 2024-04-14T15:23:15.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.131+0000 7f89b7a90700 1 librados: calling monclient init 2024-04-14T15:23:15.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.131+0000 7f89b7a90700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f89b01a7980 con 0x7f89b0108870 2024-04-14T15:23:15.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.131+0000 7f89b7a90700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f89b01a8ba0 con 0x7f89b010f2f0 2024-04-14T15:23:15.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.131+0000 7f89b7a90700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f89b01a9de0 con 0x7f89b010b720 2024-04-14T15:23:15.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.131+0000 7f89b602d700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f89b010f2f0 0x7f89b01a6040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.90:34294/0 (socket says 172.21.15.90:34294) 2024-04-14T15:23:15.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.131+0000 7f89b602d700 1 -- 172.21.15.90:0/2205523318 learned_addr learned my addr 172.21.15.90:0/2205523318 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:23:15.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.131+0000 7f89b582c700 1 --1- 172.21.15.90:0/2205523318 >> v1:172.21.15.90:6789/0 conn(0x7f89b0108870 0x7f89b010eb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:39142/0 (socket says 172.21.15.90:39142) 2024-04-14T15:23:15.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.131+0000 7f89a6ffd700 10 client.?.objecter ms_handle_connect 0x7f89b010f2f0 2024-04-14T15:23:15.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.131+0000 7f89a6ffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:23:15.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.131+0000 7f89a6ffd700 10 client.?.objecter ms_handle_connect 0x7f89b010b720 2024-04-14T15:23:15.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.131+0000 7f89a6ffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:23:15.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.131+0000 7f89a6ffd700 10 client.?.objecter ms_handle_connect 0x7f89b0108870 2024-04-14T15:23:15.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.131+0000 7f89a6ffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:23:15.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.131+0000 7f89a6ffd700 1 -- 172.21.15.90:0/2205523318 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2568515822 0 0) 0x7f89b01a8ba0 con 0x7f89b010f2f0 2024-04-14T15:23:15.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.131+0000 7f89a6ffd700 1 -- 172.21.15.90:0/2205523318 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f899c0035b0 con 0x7f89b010f2f0 2024-04-14T15:23:15.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.131+0000 7f89a6ffd700 1 -- 172.21.15.90:0/2205523318 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1004471717 0 0) 0x7f89b01a7980 con 0x7f89b0108870 2024-04-14T15:23:15.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.131+0000 7f89a6ffd700 1 -- 172.21.15.90:0/2205523318 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89b01a8ba0 con 0x7f89b0108870 2024-04-14T15:23:15.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.131+0000 7f89a6ffd700 1 -- 172.21.15.90:0/2205523318 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2605933659 0 0) 0x7f89b01a9de0 con 0x7f89b010b720 2024-04-14T15:23:15.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.131+0000 7f89a6ffd700 1 -- 172.21.15.90:0/2205523318 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89b01a7980 con 0x7f89b010b720 2024-04-14T15:23:15.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.131+0000 7f89a6ffd700 1 -- 172.21.15.90:0/2205523318 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 630733227 0 0) 0x7f899c0035b0 con 0x7f89b010f2f0 2024-04-14T15:23:15.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.131+0000 7f89a6ffd700 1 -- 172.21.15.90:0/2205523318 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f89b01a9de0 con 0x7f89b010f2f0 2024-04-14T15:23:15.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.131+0000 7f89a6ffd700 1 -- 172.21.15.90:0/2205523318 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f89a8001c90 con 0x7f89b010f2f0 2024-04-14T15:23:15.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.132+0000 7f89a6ffd700 1 -- 172.21.15.90:0/2205523318 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1769174611 0 0) 0x7f89b01a8ba0 con 0x7f89b0108870 2024-04-14T15:23:15.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.132+0000 7f89a6ffd700 1 -- 172.21.15.90:0/2205523318 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f899c0035b0 con 0x7f89b0108870 2024-04-14T15:23:15.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.132+0000 7f89a6ffd700 1 -- 172.21.15.90:0/2205523318 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f89ac0044f0 con 0x7f89b0108870 2024-04-14T15:23:15.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.132+0000 7f89a6ffd700 1 -- 172.21.15.90:0/2205523318 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4122171178 0 0) 0x7f89b01a7980 con 0x7f89b010b720 2024-04-14T15:23:15.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.132+0000 7f89a6ffd700 1 -- 172.21.15.90:0/2205523318 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f89b01a8ba0 con 0x7f89b010b720 2024-04-14T15:23:15.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.132+0000 7f89a6ffd700 1 -- 172.21.15.90:0/2205523318 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4116337813 0 0) 0x7f89b01a9de0 con 0x7f89b010f2f0 2024-04-14T15:23:15.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.132+0000 7f89a6ffd700 1 -- 172.21.15.90:0/2205523318 >> v1:172.21.15.90:6790/0 conn(0x7f89b010b720 legacy=0x7f89b01a27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:15.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.132+0000 7f89a6ffd700 1 -- 172.21.15.90:0/2205523318 >> v1:172.21.15.90:6789/0 conn(0x7f89b0108870 legacy=0x7f89b010eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:15.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.132+0000 7f89a6ffd700 1 -- 172.21.15.90:0/2205523318 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89b01ab020 con 0x7f89b010f2f0 2024-04-14T15:23:15.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.132+0000 7f89b7a90700 1 -- 172.21.15.90:0/2205523318 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f89b01aa010 con 0x7f89b010f2f0 2024-04-14T15:23:15.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.132+0000 7f89b7a90700 10 client.49612.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:23:15.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.132+0000 7f89b7a90700 1 -- 172.21.15.90:0/2205523318 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f89b01aa4d0 con 0x7f89b010f2f0 2024-04-14T15:23:15.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.132+0000 7f89b7a90700 1 librados: init done 2024-04-14T15:23:15.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.132+0000 7f89a6ffd700 1 -- 172.21.15.90:0/2205523318 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f89a8001fd0 con 0x7f89b010f2f0 2024-04-14T15:23:15.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.132+0000 7f89a6ffd700 1 -- 172.21.15.90:0/2205523318 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f89a8006010 con 0x7f89b010f2f0 2024-04-14T15:23:15.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.133+0000 7f89b7a90700 1 -- 172.21.15.90:0/2205523318 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8994005150 con 0x7f89b010f2f0 2024-04-14T15:23:15.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.133+0000 7f89a6ffd700 1 -- 172.21.15.90:0/2205523318 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f89a801ca20 con 0x7f89b010f2f0 2024-04-14T15:23:15.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.134+0000 7f89a6ffd700 1 -- 172.21.15.90:0/2205523318 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(659..659 src has 467..659) ==== 8706+0+0 (unknown 488492162 0 0) 0x7f89a808af40 con 0x7f89b010f2f0 2024-04-14T15:23:15.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.134+0000 7f89a6ffd700 10 client.49612.objecter ms_dispatch 0x7f89b0001550 osd_map(659..659 src has 467..659) 2024-04-14T15:23:15.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.134+0000 7f89a6ffd700 3 client.49612.objecter handle_osd_map got epochs [659,659] > 0 2024-04-14T15:23:15.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.134+0000 7f89a6ffd700 3 client.49612.objecter handle_osd_map decoding full epoch 659 2024-04-14T15:23:15.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.134+0000 7f89a6ffd700 20 client.49612.objecter dump_active .. 0 homeless 2024-04-14T15:23:15.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.134+0000 7f89a6ffd700 10 client.49612.objecter ms_handle_connect 0x7f899c071490 2024-04-14T15:23:15.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.139+0000 7f89a6ffd700 1 -- 172.21.15.90:0/2205523318 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f89a8057730 con 0x7f89b010f2f0 2024-04-14T15:23:15.146 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:23:15.158 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:23:15.170 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:15.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:15.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:23:15.187 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:14 smithi092 ceph-mon[33966]: pgmap v966: 216 pgs: 32 unknown, 1 active+remapped+backfill_toofull, 183 active+clean; 582 KiB data, 583 MiB used, 625 GiB / 626 GiB avail; 2/600 objects misplaced (0.333%) 2024-04-14T15:23:15.188 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:14 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-114"}]': finished 2024-04-14T15:23:15.188 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:14 smithi092 ceph-mon[33966]: osdmap e658: 8 total, 8 up, 7 in 2024-04-14T15:23:15.188 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:14 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-114", "mode": "writeback"}]: dispatch 2024-04-14T15:23:15.188 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:14 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-114", "mode": "writeback"}]: dispatch 2024-04-14T15:23:15.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:14 smithi090 ceph-mon[41670]: pgmap v966: 216 pgs: 32 unknown, 1 active+remapped+backfill_toofull, 183 active+clean; 582 KiB data, 583 MiB used, 625 GiB / 626 GiB avail; 2/600 objects misplaced (0.333%) 2024-04-14T15:23:15.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:14 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-114"}]': finished 2024-04-14T15:23:15.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:14 smithi090 ceph-mon[41670]: osdmap e658: 8 total, 8 up, 7 in 2024-04-14T15:23:15.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:14 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-114", "mode": "writeback"}]: dispatch 2024-04-14T15:23:15.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:14 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-114", "mode": "writeback"}]: dispatch 2024-04-14T15:23:15.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:14 smithi090 ceph-mon[30399]: pgmap v966: 216 pgs: 32 unknown, 1 active+remapped+backfill_toofull, 183 active+clean; 582 KiB data, 583 MiB used, 625 GiB / 626 GiB avail; 2/600 objects misplaced (0.333%) 2024-04-14T15:23:15.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:14 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-114"}]': finished 2024-04-14T15:23:15.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:14 smithi090 ceph-mon[30399]: osdmap e658: 8 total, 8 up, 7 in 2024-04-14T15:23:15.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:14 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-114", "mode": "writeback"}]: dispatch 2024-04-14T15:23:15.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:14 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-114", "mode": "writeback"}]: dispatch 2024-04-14T15:23:15.271 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:23:15.294 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:15.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:15.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:23:15.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.366+0000 7f89b7a90700 1 -- 172.21.15.90:0/2205523318 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7f8994005440 con 0x7f89b010f2f0 2024-04-14T15:23:15.396 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:23:15.420 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:15.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:15.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:23:15.444 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-04-14T15:23:15.444 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:23:15.444 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:23:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:23:15.522 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:23:15.545 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:15.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:15.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:23:15.647 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:23:15.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:15.671 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:15.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:23:15.772 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:23:15.785 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-14T15:23:15.785 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-14T15:23:15.785 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-04-14T15:23:15.785 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph tell osd.0 'trim stale osdmaps' 2024-04-14T15:23:15.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:15.797 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:15.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:23:15.898 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:23:15.922 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:15.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:15.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:23:15.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.977+0000 7f89a6ffd700 1 -- 172.21.15.90:0/2205523318 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v660) ==== 97+0+0 (unknown 4006176463 0 0) 0x7f89a805b3e0 con 0x7f89b010f2f0 2024-04-14T15:23:15.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.980+0000 7f89b7a90700 10 librados: watch_flush enter 2024-04-14T15:23:15.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.980+0000 7f89b7a90700 10 librados: watch_flush exit 2024-04-14T15:23:15.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.980+0000 7f89b7a90700 20 client.49612.objecter shutdown clearing up homeless session... 2024-04-14T15:23:15.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.980+0000 7f89b7a90700 10 client.49612.objecter successfully canceled tick 2024-04-14T15:23:15.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.980+0000 7f89b7a90700 1 -- 172.21.15.90:0/2205523318 >> v1:172.21.15.90:6800/420935427 conn(0x7f899c071490 legacy=0x7f899c073950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:15.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.980+0000 7f89b7a90700 1 -- 172.21.15.90:0/2205523318 >> v1:172.21.15.92:6789/0 conn(0x7f89b010f2f0 legacy=0x7f89b01a6040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:15.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.980+0000 7f89b7a90700 1 -- 172.21.15.90:0/2205523318 shutdown_connections 2024-04-14T15:23:15.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.981+0000 7f89b7a90700 1 -- 172.21.15.90:0/2205523318 >> 172.21.15.90:0/2205523318 conn(0x7f89b00fe3a0 msgr2=0x7f89b0112510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:23:15.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.981+0000 7f89b7a90700 1 -- 172.21.15.90:0/2205523318 shutdown_connections 2024-04-14T15:23:15.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.981+0000 7f89b7a90700 1 -- 172.21.15.90:0/2205523318 wait complete. 2024-04-14T15:23:15.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:15.981+0000 7f89b7a90700 1 librados: shutdown 2024-04-14T15:23:15.982 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is set 2024-04-14T15:23:16.024 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:23:16.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:16.047 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:16.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:23:16.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:15 smithi092 ceph-mon[33966]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:23:16.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:15 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-114", "mode": "writeback"}]': finished 2024-04-14T15:23:16.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:15 smithi092 ceph-mon[33966]: osdmap e659: 8 total, 8 up, 7 in 2024-04-14T15:23:16.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:15 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:23:16.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:15 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:23:16.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:15 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2205523318' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:23:16.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:15 smithi092 ceph-mon[33966]: from='client.49612 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:23:16.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:15 smithi092 ceph-mon[33966]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:23:16.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:15 smithi092 ceph-mon[33966]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:23:16.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:15 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]': finished 2024-04-14T15:23:16.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:15 smithi092 ceph-mon[33966]: from='client.49612 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-14T15:23:16.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:15 smithi092 ceph-mon[33966]: osdmap e660: 8 total, 8 up, 7 in 2024-04-14T15:23:16.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:15 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-114"}]: dispatch 2024-04-14T15:23:16.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:15 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-114"}]: dispatch 2024-04-14T15:23:16.149 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:16.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:16.174 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:16.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:16.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:15 smithi090 ceph-mon[30399]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:23:16.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:15 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-114", "mode": "writeback"}]': finished 2024-04-14T15:23:16.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:15 smithi090 ceph-mon[30399]: osdmap e659: 8 total, 8 up, 7 in 2024-04-14T15:23:16.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:15 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:23:16.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:15 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:23:16.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:15 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2205523318' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:23:16.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:15 smithi090 ceph-mon[30399]: from='client.49612 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:23:16.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:15 smithi090 ceph-mon[30399]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:23:16.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:15 smithi090 ceph-mon[30399]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:23:16.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:15 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]': finished 2024-04-14T15:23:16.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:15 smithi090 ceph-mon[30399]: from='client.49612 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-14T15:23:16.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:15 smithi090 ceph-mon[30399]: osdmap e660: 8 total, 8 up, 7 in 2024-04-14T15:23:16.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:15 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-114"}]: dispatch 2024-04-14T15:23:16.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:15 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-114"}]: dispatch 2024-04-14T15:23:16.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:15 smithi090 ceph-mon[41670]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:23:16.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:15 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-114", "mode": "writeback"}]': finished 2024-04-14T15:23:16.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:15 smithi090 ceph-mon[41670]: osdmap e659: 8 total, 8 up, 7 in 2024-04-14T15:23:16.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:15 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:23:16.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:15 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:23:16.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:15 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2205523318' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:23:16.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:15 smithi090 ceph-mon[41670]: from='client.49612 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:23:16.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:15 smithi090 ceph-mon[41670]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:23:16.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:15 smithi090 ceph-mon[41670]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:23:16.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:15 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]': finished 2024-04-14T15:23:16.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:15 smithi090 ceph-mon[41670]: from='client.49612 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-14T15:23:16.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:15 smithi090 ceph-mon[41670]: osdmap e660: 8 total, 8 up, 7 in 2024-04-14T15:23:16.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:15 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-114"}]: dispatch 2024-04-14T15:23:16.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:15 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-114"}]: dispatch 2024-04-14T15:23:16.209 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-14T15:23:16.276 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:23:16.316 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:16.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:16.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:23:16.417 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:23:16.441 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:16.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:16.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:23:16.490 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:23:16.544 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:23:16.567 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:16.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:16.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:23:16.669 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:23:16.693 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:16.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:16.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:23:16.795 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:23:16.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:16.819 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:16.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:23:16.920 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:23:16.943 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:16.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:16.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:23:17.044 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:17.054 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:16 smithi090 ceph-mon[30399]: pgmap v969: 216 pgs: 1 active+remapped+backfill_toofull, 215 active+clean; 582 KiB data, 584 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 2/603 objects misplaced (0.332%) 2024-04-14T15:23:17.055 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:16 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:23:17.055 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:16 smithi090 ceph-mon[30399]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:23:17.055 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:16 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-114"}]': finished 2024-04-14T15:23:17.055 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:16 smithi090 ceph-mon[30399]: osdmap e661: 8 total, 8 up, 7 in 2024-04-14T15:23:17.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:16 smithi090 ceph-mon[41670]: pgmap v969: 216 pgs: 1 active+remapped+backfill_toofull, 215 active+clean; 582 KiB data, 584 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 2/603 objects misplaced (0.332%) 2024-04-14T15:23:17.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:16 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:23:17.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:16 smithi090 ceph-mon[41670]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:23:17.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:16 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-114"}]': finished 2024-04-14T15:23:17.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:16 smithi090 ceph-mon[41670]: osdmap e661: 8 total, 8 up, 7 in 2024-04-14T15:23:17.069 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:17.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:17.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:17.171 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:23:17.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:17.195 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:17.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:23:17.296 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:23:17.321 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:17.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:17.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:23:17.365 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:16 smithi092 ceph-mon[33966]: pgmap v969: 216 pgs: 1 active+remapped+backfill_toofull, 215 active+clean; 582 KiB data, 584 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 2/603 objects misplaced (0.332%) 2024-04-14T15:23:17.366 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:16 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:23:17.366 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:16 smithi092 ceph-mon[33966]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:23:17.366 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:16 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-114"}]': finished 2024-04-14T15:23:17.366 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:16 smithi092 ceph-mon[33966]: osdmap e661: 8 total, 8 up, 7 in 2024-04-14T15:23:17.422 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:23:17.446 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:17.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:17.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:23:17.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.526+0000 7fd863326700 1 -- 172.21.15.90:0/1300019580 >> v1:172.21.15.90:6789/0 conn(0x7fd85c10f340 legacy=0x7fd85c1117d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:17.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.526+0000 7fd863326700 1 -- 172.21.15.90:0/1300019580 shutdown_connections 2024-04-14T15:23:17.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.526+0000 7fd863326700 1 -- 172.21.15.90:0/1300019580 >> 172.21.15.90:0/1300019580 conn(0x7fd85c0fd290 msgr2=0x7fd85c0ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:23:17.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.526+0000 7fd863326700 1 -- 172.21.15.90:0/1300019580 shutdown_connections 2024-04-14T15:23:17.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.526+0000 7fd863326700 1 -- 172.21.15.90:0/1300019580 wait complete. 2024-04-14T15:23:17.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.527+0000 7fd863326700 1 librados: starting msgr at 2024-04-14T15:23:17.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.527+0000 7fd863326700 1 librados: starting objecter 2024-04-14T15:23:17.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.527+0000 7fd863326700 1 Processor -- start 2024-04-14T15:23:17.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.527+0000 7fd863326700 1 -- start start 2024-04-14T15:23:17.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.527+0000 7fd863326700 1 librados: setting wanted keys 2024-04-14T15:23:17.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.527+0000 7fd863326700 1 librados: calling monclient init 2024-04-14T15:23:17.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.528+0000 7fd863326700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd85c1a9bc0 con 0x7fd85c10f340 2024-04-14T15:23:17.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.528+0000 7fd863326700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd85c1aade0 con 0x7fd85c10b830 2024-04-14T15:23:17.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.528+0000 7fd863326700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd85c1ac020 con 0x7fd85c103a20 2024-04-14T15:23:17.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.528+0000 7fd8610c2700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7fd85c103a20 0x7fd85c10b070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:43384/0 (socket says 172.21.15.90:43384) 2024-04-14T15:23:17.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.528+0000 7fd8618c3700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fd85c10f340 0x7fd85c1a8280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:39158/0 (socket says 172.21.15.90:39158) 2024-04-14T15:23:17.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.528+0000 7fd8610c2700 1 -- 172.21.15.90:0/2254210489 learned_addr learned my addr 172.21.15.90:0/2254210489 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:23:17.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.528+0000 7fd85a7fc700 10 client.?.objecter ms_handle_connect 0x7fd85c10f340 2024-04-14T15:23:17.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.528+0000 7fd85a7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:23:17.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.528+0000 7fd85a7fc700 10 client.?.objecter ms_handle_connect 0x7fd85c103a20 2024-04-14T15:23:17.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.528+0000 7fd85a7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:23:17.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.528+0000 7fd85a7fc700 10 client.?.objecter ms_handle_connect 0x7fd85c10b830 2024-04-14T15:23:17.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.528+0000 7fd85a7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:23:17.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.528+0000 7fd85a7fc700 1 -- 172.21.15.90:0/2254210489 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1883011208 0 0) 0x7fd85c1a9bc0 con 0x7fd85c10f340 2024-04-14T15:23:17.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.528+0000 7fd85a7fc700 1 -- 172.21.15.90:0/2254210489 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd8440031f0 con 0x7fd85c10f340 2024-04-14T15:23:17.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.528+0000 7fd85a7fc700 1 -- 172.21.15.90:0/2254210489 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2166450508 0 0) 0x7fd85c1ac020 con 0x7fd85c103a20 2024-04-14T15:23:17.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.528+0000 7fd85a7fc700 1 -- 172.21.15.90:0/2254210489 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd85c1a9bc0 con 0x7fd85c103a20 2024-04-14T15:23:17.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.528+0000 7fd85a7fc700 1 -- 172.21.15.90:0/2254210489 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1223025821 0 0) 0x7fd85c1aade0 con 0x7fd85c10b830 2024-04-14T15:23:17.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.528+0000 7fd85a7fc700 1 -- 172.21.15.90:0/2254210489 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd85c1ac020 con 0x7fd85c10b830 2024-04-14T15:23:17.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.528+0000 7fd85a7fc700 1 -- 172.21.15.90:0/2254210489 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 328097053 0 0) 0x7fd85c1ac020 con 0x7fd85c10b830 2024-04-14T15:23:17.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.528+0000 7fd85a7fc700 1 -- 172.21.15.90:0/2254210489 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd85c1aade0 con 0x7fd85c10b830 2024-04-14T15:23:17.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.528+0000 7fd85a7fc700 1 -- 172.21.15.90:0/2254210489 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fd850001870 con 0x7fd85c10b830 2024-04-14T15:23:17.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.528+0000 7fd85a7fc700 1 -- 172.21.15.90:0/2254210489 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1118215236 0 0) 0x7fd8440031f0 con 0x7fd85c10f340 2024-04-14T15:23:17.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.528+0000 7fd85a7fc700 1 -- 172.21.15.90:0/2254210489 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd85c1ac020 con 0x7fd85c10f340 2024-04-14T15:23:17.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.528+0000 7fd85a7fc700 1 -- 172.21.15.90:0/2254210489 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fd84c0048c0 con 0x7fd85c10f340 2024-04-14T15:23:17.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.528+0000 7fd85a7fc700 1 -- 172.21.15.90:0/2254210489 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2674715945 0 0) 0x7fd85c1a9bc0 con 0x7fd85c103a20 2024-04-14T15:23:17.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.528+0000 7fd85a7fc700 1 -- 172.21.15.90:0/2254210489 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd8440031f0 con 0x7fd85c103a20 2024-04-14T15:23:17.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.528+0000 7fd85a7fc700 1 -- 172.21.15.90:0/2254210489 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 388196104 0 0) 0x7fd85c1aade0 con 0x7fd85c10b830 2024-04-14T15:23:17.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.529+0000 7fd85a7fc700 1 -- 172.21.15.90:0/2254210489 >> v1:172.21.15.90:6790/0 conn(0x7fd85c103a20 legacy=0x7fd85c10b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:17.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.529+0000 7fd85a7fc700 1 -- 172.21.15.90:0/2254210489 >> v1:172.21.15.90:6789/0 conn(0x7fd85c10f340 legacy=0x7fd85c1a8280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:17.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.529+0000 7fd85a7fc700 1 -- 172.21.15.90:0/2254210489 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd85c1ad260 con 0x7fd85c10b830 2024-04-14T15:23:17.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.529+0000 7fd863326700 1 -- 172.21.15.90:0/2254210489 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd85c1ac250 con 0x7fd85c10b830 2024-04-14T15:23:17.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.529+0000 7fd863326700 10 client.49618.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:23:17.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.529+0000 7fd863326700 1 -- 172.21.15.90:0/2254210489 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd85c1ac710 con 0x7fd85c10b830 2024-04-14T15:23:17.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.529+0000 7fd863326700 1 librados: init done 2024-04-14T15:23:17.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.529+0000 7fd85a7fc700 1 -- 172.21.15.90:0/2254210489 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fd850003b40 con 0x7fd85c10b830 2024-04-14T15:23:17.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.529+0000 7fd85a7fc700 1 -- 172.21.15.90:0/2254210489 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fd850004a20 con 0x7fd85c10b830 2024-04-14T15:23:17.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.530+0000 7fd863326700 1 -- 172.21.15.90:0/2254210489 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fd848005100 con 0x7fd85c10b830 2024-04-14T15:23:17.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.530+0000 7fd85a7fc700 1 -- 172.21.15.90:0/2254210489 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fd85001b5d0 con 0x7fd85c10b830 2024-04-14T15:23:17.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.531+0000 7fd85a7fc700 1 -- 172.21.15.90:0/2254210489 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(661..661 src has 467..661) ==== 8698+0+0 (unknown 2377681326 0 0) 0x7fd8500899e0 con 0x7fd85c10b830 2024-04-14T15:23:17.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.531+0000 7fd85a7fc700 10 client.49618.objecter ms_dispatch 0x7fd85c001550 osd_map(661..661 src has 467..661) 2024-04-14T15:23:17.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.531+0000 7fd85a7fc700 3 client.49618.objecter handle_osd_map got epochs [661,661] > 0 2024-04-14T15:23:17.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.531+0000 7fd85a7fc700 3 client.49618.objecter handle_osd_map decoding full epoch 661 2024-04-14T15:23:17.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.531+0000 7fd85a7fc700 20 client.49618.objecter dump_active .. 0 homeless 2024-04-14T15:23:17.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.531+0000 7fd85a7fc700 1 -- 172.21.15.90:0/2254210489 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v661) ==== 54+0+16 (unknown 84686803 0 2420000192) 0x7fd850089dd0 con 0x7fd85c10b830 2024-04-14T15:23:17.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.531+0000 7fd85a7fc700 10 client.49618.objecter ms_handle_connect 0x7fd844071300 2024-04-14T15:23:17.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.531+0000 7fd863326700 10 client.49618.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:23:17.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.531+0000 7fd863326700 15 client.49618.objecter _session_command_op_assign -1 1 2024-04-14T15:23:17.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.531+0000 7fd863326700 20 client.49618.objecter _get_session s=0x7fd8480053e0 osd=0 3 2024-04-14T15:23:17.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.531+0000 7fd863326700 20 client.49618.objecter put_session s=0x7fd8480053e0 osd=0 3 2024-04-14T15:23:17.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.531+0000 7fd863326700 20 client.49618.objecter _get_session s=0x7fd8480053e0 osd=0 3 2024-04-14T15:23:17.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.531+0000 7fd863326700 15 client.49618.objecter _session_command_op_remove -1 1 2024-04-14T15:23:17.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.531+0000 7fd863326700 20 client.49618.objecter get_session s=0x7fd8480053e0 osd=0 3 2024-04-14T15:23:17.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.531+0000 7fd863326700 15 client.49618.objecter _session_command_op_assign 0 1 2024-04-14T15:23:17.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.531+0000 7fd863326700 20 client.49618.objecter put_session s=0x7fd8480053e0 osd=0 4 2024-04-14T15:23:17.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.531+0000 7fd863326700 10 client.49618.objecter _send_command 1 2024-04-14T15:23:17.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.531+0000 7fd863326700 1 -- 172.21.15.90:0/2254210489 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd848004b50 con 0x7fd848004730 2024-04-14T15:23:17.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.531+0000 7fd85a7fc700 10 client.49618.objecter ms_handle_connect 0x7fd848004730 2024-04-14T15:23:17.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.532+0000 7fd85a7fc700 1 -- 172.21.15.90:0/2254210489 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd848004b50 con 0x7fd848004730 2024-04-14T15:23:17.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.532+0000 7fd85a7fc700 10 client.49618.objecter ms_dispatch 0x7fd85c001550 command_reply(tid 1: 0 ) 2024-04-14T15:23:17.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.532+0000 7fd85a7fc700 10 client.49618.objecter _finish_command 1 = system:0 2024-04-14T15:23:17.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.532+0000 7fd85a7fc700 20 client.49618.objecter put_session s=0x7fd8480053e0 osd=0 4 2024-04-14T15:23:17.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.532+0000 7fd85a7fc700 15 client.49618.objecter _session_command_op_remove 0 1 2024-04-14T15:23:17.551 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:23:17.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.558+0000 7fd863326700 10 client.49618.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:23:17.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.558+0000 7fd863326700 15 client.49618.objecter _session_command_op_assign -1 2 2024-04-14T15:23:17.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.558+0000 7fd863326700 20 client.49618.objecter _get_session s=0x7fd8480053e0 osd=0 3 2024-04-14T15:23:17.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.558+0000 7fd863326700 20 client.49618.objecter put_session s=0x7fd8480053e0 osd=0 3 2024-04-14T15:23:17.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.558+0000 7fd863326700 20 client.49618.objecter _get_session s=0x7fd8480053e0 osd=0 3 2024-04-14T15:23:17.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.558+0000 7fd863326700 15 client.49618.objecter _session_command_op_remove -1 2 2024-04-14T15:23:17.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.558+0000 7fd863326700 20 client.49618.objecter get_session s=0x7fd8480053e0 osd=0 3 2024-04-14T15:23:17.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.558+0000 7fd863326700 15 client.49618.objecter _session_command_op_assign 0 2 2024-04-14T15:23:17.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.558+0000 7fd863326700 20 client.49618.objecter put_session s=0x7fd8480053e0 osd=0 4 2024-04-14T15:23:17.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.558+0000 7fd863326700 10 client.49618.objecter _send_command 2 2024-04-14T15:23:17.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.558+0000 7fd863326700 1 -- 172.21.15.90:0/2254210489 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fd848007b00 con 0x7fd848004730 2024-04-14T15:23:17.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.560+0000 7fd85a7fc700 1 -- 172.21.15.90:0/2254210489 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd848007b00 con 0x7fd848004730 2024-04-14T15:23:17.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.560+0000 7fd85a7fc700 10 client.49618.objecter ms_dispatch 0x7fd85c001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:23:17.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.560+0000 7fd85a7fc700 10 client.49618.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:23:17.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.560+0000 7fd85a7fc700 20 client.49618.objecter put_session s=0x7fd8480053e0 osd=0 4 2024-04-14T15:23:17.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.560+0000 7fd85a7fc700 15 client.49618.objecter _session_command_op_remove 0 2 2024-04-14T15:23:17.561 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:23:17.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.560+0000 7fd863326700 10 client.49618.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:23:17.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.560+0000 7fd863326700 15 client.49618.objecter _session_command_op_assign -1 3 2024-04-14T15:23:17.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.560+0000 7fd863326700 20 client.49618.objecter _get_session s=0x7fd848004130 osd=1 3 2024-04-14T15:23:17.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.560+0000 7fd863326700 20 client.49618.objecter put_session s=0x7fd848004130 osd=1 3 2024-04-14T15:23:17.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.560+0000 7fd863326700 20 client.49618.objecter _get_session s=0x7fd848004130 osd=1 3 2024-04-14T15:23:17.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.560+0000 7fd863326700 15 client.49618.objecter _session_command_op_remove -1 3 2024-04-14T15:23:17.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.560+0000 7fd863326700 20 client.49618.objecter get_session s=0x7fd848004130 osd=1 3 2024-04-14T15:23:17.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.560+0000 7fd863326700 15 client.49618.objecter _session_command_op_assign 1 3 2024-04-14T15:23:17.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.560+0000 7fd863326700 20 client.49618.objecter put_session s=0x7fd848004130 osd=1 4 2024-04-14T15:23:17.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.560+0000 7fd863326700 10 client.49618.objecter _send_command 3 2024-04-14T15:23:17.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.560+0000 7fd863326700 1 -- 172.21.15.90:0/2254210489 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fd848002130 con 0x7fd8480042a0 2024-04-14T15:23:17.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.561+0000 7fd85a7fc700 10 client.49618.objecter ms_handle_connect 0x7fd8480042a0 2024-04-14T15:23:17.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.562+0000 7fd85a7fc700 1 -- 172.21.15.90:0/2254210489 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd848002130 con 0x7fd8480042a0 2024-04-14T15:23:17.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.562+0000 7fd85a7fc700 10 client.49618.objecter ms_dispatch 0x7fd85c001550 command_reply(tid 3: 0 ) 2024-04-14T15:23:17.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.562+0000 7fd85a7fc700 10 client.49618.objecter _finish_command 3 = system:0 2024-04-14T15:23:17.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.562+0000 7fd85a7fc700 20 client.49618.objecter put_session s=0x7fd848004130 osd=1 4 2024-04-14T15:23:17.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.562+0000 7fd85a7fc700 15 client.49618.objecter _session_command_op_remove 1 3 2024-04-14T15:23:17.571 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:23:17.576 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:17.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:17.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:23:17.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.579+0000 7fd863326700 10 client.49618.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:23:17.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.579+0000 7fd863326700 15 client.49618.objecter _session_command_op_assign -1 4 2024-04-14T15:23:17.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.579+0000 7fd863326700 20 client.49618.objecter _get_session s=0x7fd848004130 osd=1 3 2024-04-14T15:23:17.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.579+0000 7fd863326700 20 client.49618.objecter put_session s=0x7fd848004130 osd=1 3 2024-04-14T15:23:17.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.579+0000 7fd863326700 20 client.49618.objecter _get_session s=0x7fd848004130 osd=1 3 2024-04-14T15:23:17.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.579+0000 7fd863326700 15 client.49618.objecter _session_command_op_remove -1 4 2024-04-14T15:23:17.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.579+0000 7fd863326700 20 client.49618.objecter get_session s=0x7fd848004130 osd=1 3 2024-04-14T15:23:17.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.579+0000 7fd863326700 15 client.49618.objecter _session_command_op_assign 1 4 2024-04-14T15:23:17.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.579+0000 7fd863326700 20 client.49618.objecter put_session s=0x7fd848004130 osd=1 4 2024-04-14T15:23:17.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.579+0000 7fd863326700 10 client.49618.objecter _send_command 4 2024-04-14T15:23:17.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.579+0000 7fd863326700 1 -- 172.21.15.90:0/2254210489 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fd848001600 con 0x7fd8480042a0 2024-04-14T15:23:17.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.581+0000 7fd85a7fc700 1 -- 172.21.15.90:0/2254210489 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd848001600 con 0x7fd8480042a0 2024-04-14T15:23:17.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.581+0000 7fd85a7fc700 10 client.49618.objecter ms_dispatch 0x7fd85c001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:23:17.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.581+0000 7fd85a7fc700 10 client.49618.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:23:17.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.581+0000 7fd85a7fc700 20 client.49618.objecter put_session s=0x7fd848004130 osd=1 4 2024-04-14T15:23:17.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.581+0000 7fd85a7fc700 15 client.49618.objecter _session_command_op_remove 1 4 2024-04-14T15:23:17.582 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:23:17.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.581+0000 7fd863326700 10 client.49618.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:23:17.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.581+0000 7fd863326700 15 client.49618.objecter _session_command_op_assign -1 5 2024-04-14T15:23:17.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.581+0000 7fd863326700 20 client.49618.objecter _get_session s=0x7fd848001a00 osd=2 3 2024-04-14T15:23:17.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.581+0000 7fd863326700 20 client.49618.objecter put_session s=0x7fd848001a00 osd=2 3 2024-04-14T15:23:17.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.581+0000 7fd863326700 20 client.49618.objecter _get_session s=0x7fd848001a00 osd=2 3 2024-04-14T15:23:17.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.581+0000 7fd863326700 15 client.49618.objecter _session_command_op_remove -1 5 2024-04-14T15:23:17.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.581+0000 7fd863326700 20 client.49618.objecter get_session s=0x7fd848001a00 osd=2 3 2024-04-14T15:23:17.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.581+0000 7fd863326700 15 client.49618.objecter _session_command_op_assign 2 5 2024-04-14T15:23:17.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.581+0000 7fd863326700 20 client.49618.objecter put_session s=0x7fd848001a00 osd=2 4 2024-04-14T15:23:17.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.581+0000 7fd863326700 10 client.49618.objecter _send_command 5 2024-04-14T15:23:17.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.581+0000 7fd863326700 1 -- 172.21.15.90:0/2254210489 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fd848010fe0 con 0x7fd848008990 2024-04-14T15:23:17.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.582+0000 7fd85a7fc700 10 client.49618.objecter ms_handle_connect 0x7fd848008990 2024-04-14T15:23:17.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.583+0000 7fd85a7fc700 1 -- 172.21.15.90:0/2254210489 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd848010fe0 con 0x7fd848008990 2024-04-14T15:23:17.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.583+0000 7fd85a7fc700 10 client.49618.objecter ms_dispatch 0x7fd85c001550 command_reply(tid 5: 0 ) 2024-04-14T15:23:17.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.583+0000 7fd85a7fc700 10 client.49618.objecter _finish_command 5 = system:0 2024-04-14T15:23:17.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.583+0000 7fd85a7fc700 20 client.49618.objecter put_session s=0x7fd848001a00 osd=2 4 2024-04-14T15:23:17.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.583+0000 7fd85a7fc700 15 client.49618.objecter _session_command_op_remove 2 5 2024-04-14T15:23:17.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.601+0000 7fd863326700 10 client.49618.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:23:17.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.601+0000 7fd863326700 15 client.49618.objecter _session_command_op_assign -1 6 2024-04-14T15:23:17.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.601+0000 7fd863326700 20 client.49618.objecter _get_session s=0x7fd848001a00 osd=2 3 2024-04-14T15:23:17.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.601+0000 7fd863326700 20 client.49618.objecter put_session s=0x7fd848001a00 osd=2 3 2024-04-14T15:23:17.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.601+0000 7fd863326700 20 client.49618.objecter _get_session s=0x7fd848001a00 osd=2 3 2024-04-14T15:23:17.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.601+0000 7fd863326700 15 client.49618.objecter _session_command_op_remove -1 6 2024-04-14T15:23:17.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.601+0000 7fd863326700 20 client.49618.objecter get_session s=0x7fd848001a00 osd=2 3 2024-04-14T15:23:17.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.601+0000 7fd863326700 15 client.49618.objecter _session_command_op_assign 2 6 2024-04-14T15:23:17.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.601+0000 7fd863326700 20 client.49618.objecter put_session s=0x7fd848001a00 osd=2 4 2024-04-14T15:23:17.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.601+0000 7fd863326700 10 client.49618.objecter _send_command 6 2024-04-14T15:23:17.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.601+0000 7fd863326700 1 -- 172.21.15.90:0/2254210489 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fd848007fa0 con 0x7fd848008990 2024-04-14T15:23:17.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.602+0000 7fd85a7fc700 1 -- 172.21.15.90:0/2254210489 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd848007fa0 con 0x7fd848008990 2024-04-14T15:23:17.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.602+0000 7fd85a7fc700 10 client.49618.objecter ms_dispatch 0x7fd85c001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:23:17.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.602+0000 7fd85a7fc700 10 client.49618.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:23:17.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.602+0000 7fd85a7fc700 20 client.49618.objecter put_session s=0x7fd848001a00 osd=2 4 2024-04-14T15:23:17.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.602+0000 7fd85a7fc700 15 client.49618.objecter _session_command_op_remove 2 6 2024-04-14T15:23:17.603 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:23:17.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.602+0000 7fd863326700 10 client.49618.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:23:17.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.603+0000 7fd863326700 15 client.49618.objecter _session_command_op_assign -1 7 2024-04-14T15:23:17.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.603+0000 7fd863326700 20 client.49618.objecter _get_session s=0x7fd848008200 osd=3 3 2024-04-14T15:23:17.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.603+0000 7fd863326700 20 client.49618.objecter put_session s=0x7fd848008200 osd=3 3 2024-04-14T15:23:17.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.603+0000 7fd863326700 20 client.49618.objecter _get_session s=0x7fd848008200 osd=3 3 2024-04-14T15:23:17.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.603+0000 7fd863326700 15 client.49618.objecter _session_command_op_remove -1 7 2024-04-14T15:23:17.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.603+0000 7fd863326700 20 client.49618.objecter get_session s=0x7fd848008200 osd=3 3 2024-04-14T15:23:17.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.603+0000 7fd863326700 15 client.49618.objecter _session_command_op_assign 3 7 2024-04-14T15:23:17.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.603+0000 7fd863326700 20 client.49618.objecter put_session s=0x7fd848008200 osd=3 4 2024-04-14T15:23:17.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.603+0000 7fd863326700 10 client.49618.objecter _send_command 7 2024-04-14T15:23:17.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.603+0000 7fd863326700 1 -- 172.21.15.90:0/2254210489 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fd84800d500 con 0x7fd848011190 2024-04-14T15:23:17.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.603+0000 7fd85a7fc700 10 client.49618.objecter ms_handle_connect 0x7fd848011190 2024-04-14T15:23:17.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.603+0000 7fd85a7fc700 1 -- 172.21.15.90:0/2254210489 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd84800d500 con 0x7fd848011190 2024-04-14T15:23:17.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.603+0000 7fd85a7fc700 10 client.49618.objecter ms_dispatch 0x7fd85c001550 command_reply(tid 7: 0 ) 2024-04-14T15:23:17.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.603+0000 7fd85a7fc700 10 client.49618.objecter _finish_command 7 = system:0 2024-04-14T15:23:17.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.603+0000 7fd85a7fc700 20 client.49618.objecter put_session s=0x7fd848008200 osd=3 4 2024-04-14T15:23:17.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.603+0000 7fd85a7fc700 15 client.49618.objecter _session_command_op_remove 3 7 2024-04-14T15:23:17.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.620+0000 7fd863326700 10 client.49618.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:23:17.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.620+0000 7fd863326700 15 client.49618.objecter _session_command_op_assign -1 8 2024-04-14T15:23:17.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.620+0000 7fd863326700 20 client.49618.objecter _get_session s=0x7fd848008200 osd=3 3 2024-04-14T15:23:17.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.620+0000 7fd863326700 20 client.49618.objecter put_session s=0x7fd848008200 osd=3 3 2024-04-14T15:23:17.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.620+0000 7fd863326700 20 client.49618.objecter _get_session s=0x7fd848008200 osd=3 3 2024-04-14T15:23:17.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.620+0000 7fd863326700 15 client.49618.objecter _session_command_op_remove -1 8 2024-04-14T15:23:17.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.620+0000 7fd863326700 20 client.49618.objecter get_session s=0x7fd848008200 osd=3 3 2024-04-14T15:23:17.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.620+0000 7fd863326700 15 client.49618.objecter _session_command_op_assign 3 8 2024-04-14T15:23:17.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.620+0000 7fd863326700 20 client.49618.objecter put_session s=0x7fd848008200 osd=3 4 2024-04-14T15:23:17.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.620+0000 7fd863326700 10 client.49618.objecter _send_command 8 2024-04-14T15:23:17.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.620+0000 7fd863326700 1 -- 172.21.15.90:0/2254210489 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fd848012570 con 0x7fd848011190 2024-04-14T15:23:17.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.622+0000 7fd85a7fc700 1 -- 172.21.15.90:0/2254210489 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd848012570 con 0x7fd848011190 2024-04-14T15:23:17.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.622+0000 7fd85a7fc700 10 client.49618.objecter ms_dispatch 0x7fd85c001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:23:17.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.622+0000 7fd85a7fc700 10 client.49618.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:23:17.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.622+0000 7fd85a7fc700 20 client.49618.objecter put_session s=0x7fd848008200 osd=3 4 2024-04-14T15:23:17.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.622+0000 7fd85a7fc700 15 client.49618.objecter _session_command_op_remove 3 8 2024-04-14T15:23:17.623 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:23:17.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.622+0000 7fd863326700 10 client.49618.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:23:17.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.622+0000 7fd863326700 15 client.49618.objecter _session_command_op_assign -1 9 2024-04-14T15:23:17.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.622+0000 7fd863326700 20 client.49618.objecter _get_session s=0x7fd8480136f0 osd=4 3 2024-04-14T15:23:17.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.622+0000 7fd863326700 20 client.49618.objecter put_session s=0x7fd8480136f0 osd=4 3 2024-04-14T15:23:17.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.622+0000 7fd863326700 20 client.49618.objecter _get_session s=0x7fd8480136f0 osd=4 3 2024-04-14T15:23:17.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.622+0000 7fd863326700 15 client.49618.objecter _session_command_op_remove -1 9 2024-04-14T15:23:17.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.622+0000 7fd863326700 20 client.49618.objecter get_session s=0x7fd8480136f0 osd=4 3 2024-04-14T15:23:17.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.622+0000 7fd863326700 15 client.49618.objecter _session_command_op_assign 4 9 2024-04-14T15:23:17.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.622+0000 7fd863326700 20 client.49618.objecter put_session s=0x7fd8480136f0 osd=4 4 2024-04-14T15:23:17.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.622+0000 7fd863326700 10 client.49618.objecter _send_command 9 2024-04-14T15:23:17.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.622+0000 7fd863326700 1 -- 172.21.15.90:0/2254210489 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fd84801a6b0 con 0x7fd848016ba0 2024-04-14T15:23:17.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.622+0000 7fd85a7fc700 10 client.49618.objecter ms_handle_connect 0x7fd848016ba0 2024-04-14T15:23:17.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.623+0000 7fd85a7fc700 1 -- 172.21.15.90:0/2254210489 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd84801a6b0 con 0x7fd848016ba0 2024-04-14T15:23:17.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.623+0000 7fd85a7fc700 10 client.49618.objecter ms_dispatch 0x7fd85c001550 command_reply(tid 9: 0 ) 2024-04-14T15:23:17.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.623+0000 7fd85a7fc700 10 client.49618.objecter _finish_command 9 = system:0 2024-04-14T15:23:17.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.623+0000 7fd85a7fc700 20 client.49618.objecter put_session s=0x7fd8480136f0 osd=4 4 2024-04-14T15:23:17.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.623+0000 7fd85a7fc700 15 client.49618.objecter _session_command_op_remove 4 9 2024-04-14T15:23:17.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.639+0000 7fd863326700 10 client.49618.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:23:17.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.639+0000 7fd863326700 15 client.49618.objecter _session_command_op_assign -1 10 2024-04-14T15:23:17.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.639+0000 7fd863326700 20 client.49618.objecter _get_session s=0x7fd8480136f0 osd=4 3 2024-04-14T15:23:17.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.639+0000 7fd863326700 20 client.49618.objecter put_session s=0x7fd8480136f0 osd=4 3 2024-04-14T15:23:17.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.639+0000 7fd863326700 20 client.49618.objecter _get_session s=0x7fd8480136f0 osd=4 3 2024-04-14T15:23:17.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.639+0000 7fd863326700 15 client.49618.objecter _session_command_op_remove -1 10 2024-04-14T15:23:17.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.639+0000 7fd863326700 20 client.49618.objecter get_session s=0x7fd8480136f0 osd=4 3 2024-04-14T15:23:17.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.639+0000 7fd863326700 15 client.49618.objecter _session_command_op_assign 4 10 2024-04-14T15:23:17.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.639+0000 7fd863326700 20 client.49618.objecter put_session s=0x7fd8480136f0 osd=4 4 2024-04-14T15:23:17.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.639+0000 7fd863326700 10 client.49618.objecter _send_command 10 2024-04-14T15:23:17.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.639+0000 7fd863326700 1 -- 172.21.15.90:0/2254210489 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fd8480127c0 con 0x7fd848016ba0 2024-04-14T15:23:17.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.641+0000 7fd85a7fc700 1 -- 172.21.15.90:0/2254210489 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd8480127c0 con 0x7fd848016ba0 2024-04-14T15:23:17.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.641+0000 7fd85a7fc700 10 client.49618.objecter ms_dispatch 0x7fd85c001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:23:17.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.641+0000 7fd85a7fc700 10 client.49618.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:23:17.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.641+0000 7fd85a7fc700 20 client.49618.objecter put_session s=0x7fd8480136f0 osd=4 4 2024-04-14T15:23:17.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.641+0000 7fd85a7fc700 15 client.49618.objecter _session_command_op_remove 4 10 2024-04-14T15:23:17.642 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:23:17.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.641+0000 7fd863326700 10 client.49618.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:23:17.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.641+0000 7fd863326700 15 client.49618.objecter _session_command_op_assign -1 11 2024-04-14T15:23:17.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.641+0000 7fd863326700 20 client.49618.objecter _get_session s=0x7fd84800c870 osd=5 3 2024-04-14T15:23:17.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.641+0000 7fd863326700 20 client.49618.objecter put_session s=0x7fd84800c870 osd=5 3 2024-04-14T15:23:17.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.641+0000 7fd863326700 20 client.49618.objecter _get_session s=0x7fd84800c870 osd=5 3 2024-04-14T15:23:17.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.641+0000 7fd863326700 15 client.49618.objecter _session_command_op_remove -1 11 2024-04-14T15:23:17.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.641+0000 7fd863326700 20 client.49618.objecter get_session s=0x7fd84800c870 osd=5 3 2024-04-14T15:23:17.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.641+0000 7fd863326700 15 client.49618.objecter _session_command_op_assign 5 11 2024-04-14T15:23:17.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.641+0000 7fd863326700 20 client.49618.objecter put_session s=0x7fd84800c870 osd=5 4 2024-04-14T15:23:17.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.641+0000 7fd863326700 10 client.49618.objecter _send_command 11 2024-04-14T15:23:17.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.641+0000 7fd863326700 1 -- 172.21.15.90:0/2254210489 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fd848013250 con 0x7fd848012cd0 2024-04-14T15:23:17.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.642+0000 7fd85a7fc700 10 client.49618.objecter ms_handle_connect 0x7fd848012cd0 2024-04-14T15:23:17.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.643+0000 7fd85a7fc700 1 -- 172.21.15.90:0/2254210489 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd848013250 con 0x7fd848012cd0 2024-04-14T15:23:17.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.643+0000 7fd85a7fc700 10 client.49618.objecter ms_dispatch 0x7fd85c001550 command_reply(tid 11: 0 ) 2024-04-14T15:23:17.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.643+0000 7fd85a7fc700 10 client.49618.objecter _finish_command 11 = system:0 2024-04-14T15:23:17.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.643+0000 7fd85a7fc700 20 client.49618.objecter put_session s=0x7fd84800c870 osd=5 4 2024-04-14T15:23:17.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.643+0000 7fd85a7fc700 15 client.49618.objecter _session_command_op_remove 5 11 2024-04-14T15:23:17.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.659+0000 7fd863326700 10 client.49618.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:23:17.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.659+0000 7fd863326700 15 client.49618.objecter _session_command_op_assign -1 12 2024-04-14T15:23:17.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.659+0000 7fd863326700 20 client.49618.objecter _get_session s=0x7fd84800c870 osd=5 3 2024-04-14T15:23:17.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.659+0000 7fd863326700 20 client.49618.objecter put_session s=0x7fd84800c870 osd=5 3 2024-04-14T15:23:17.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.659+0000 7fd863326700 20 client.49618.objecter _get_session s=0x7fd84800c870 osd=5 3 2024-04-14T15:23:17.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.659+0000 7fd863326700 15 client.49618.objecter _session_command_op_remove -1 12 2024-04-14T15:23:17.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.659+0000 7fd863326700 20 client.49618.objecter get_session s=0x7fd84800c870 osd=5 3 2024-04-14T15:23:17.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.659+0000 7fd863326700 15 client.49618.objecter _session_command_op_assign 5 12 2024-04-14T15:23:17.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.659+0000 7fd863326700 20 client.49618.objecter put_session s=0x7fd84800c870 osd=5 4 2024-04-14T15:23:17.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.659+0000 7fd863326700 10 client.49618.objecter _send_command 12 2024-04-14T15:23:17.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.659+0000 7fd863326700 1 -- 172.21.15.90:0/2254210489 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fd8480134d0 con 0x7fd848012cd0 2024-04-14T15:23:17.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.661+0000 7fd85a7fc700 1 -- 172.21.15.90:0/2254210489 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd8480134d0 con 0x7fd848012cd0 2024-04-14T15:23:17.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.661+0000 7fd85a7fc700 10 client.49618.objecter ms_dispatch 0x7fd85c001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:23:17.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.661+0000 7fd85a7fc700 10 client.49618.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:23:17.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.661+0000 7fd85a7fc700 20 client.49618.objecter put_session s=0x7fd84800c870 osd=5 4 2024-04-14T15:23:17.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.661+0000 7fd85a7fc700 15 client.49618.objecter _session_command_op_remove 5 12 2024-04-14T15:23:17.661 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:23:17.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.661+0000 7fd863326700 10 client.49618.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:23:17.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.661+0000 7fd863326700 15 client.49618.objecter _session_command_op_assign -1 13 2024-04-14T15:23:17.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.661+0000 7fd863326700 20 client.49618.objecter _get_session s=0x7fd848013310 osd=6 3 2024-04-14T15:23:17.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.661+0000 7fd863326700 20 client.49618.objecter put_session s=0x7fd848013310 osd=6 3 2024-04-14T15:23:17.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.661+0000 7fd863326700 20 client.49618.objecter _get_session s=0x7fd848013310 osd=6 3 2024-04-14T15:23:17.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.661+0000 7fd863326700 15 client.49618.objecter _session_command_op_remove -1 13 2024-04-14T15:23:17.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.661+0000 7fd863326700 20 client.49618.objecter get_session s=0x7fd848013310 osd=6 3 2024-04-14T15:23:17.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.661+0000 7fd863326700 15 client.49618.objecter _session_command_op_assign 6 13 2024-04-14T15:23:17.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.661+0000 7fd863326700 20 client.49618.objecter put_session s=0x7fd848013310 osd=6 4 2024-04-14T15:23:17.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.661+0000 7fd863326700 10 client.49618.objecter _send_command 13 2024-04-14T15:23:17.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.661+0000 7fd863326700 1 -- 172.21.15.90:0/2254210489 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fd848023050 con 0x7fd84801f470 2024-04-14T15:23:17.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.661+0000 7fd85a7fc700 10 client.49618.objecter ms_handle_connect 0x7fd84801f470 2024-04-14T15:23:17.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.662+0000 7fd85a7fc700 1 -- 172.21.15.90:0/2254210489 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd848023050 con 0x7fd84801f470 2024-04-14T15:23:17.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.662+0000 7fd85a7fc700 10 client.49618.objecter ms_dispatch 0x7fd85c001550 command_reply(tid 13: 0 ) 2024-04-14T15:23:17.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.662+0000 7fd85a7fc700 10 client.49618.objecter _finish_command 13 = system:0 2024-04-14T15:23:17.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.662+0000 7fd85a7fc700 20 client.49618.objecter put_session s=0x7fd848013310 osd=6 4 2024-04-14T15:23:17.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.662+0000 7fd85a7fc700 15 client.49618.objecter _session_command_op_remove 6 13 2024-04-14T15:23:17.678 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:23:17.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.679+0000 7fd863326700 10 client.49618.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:23:17.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.679+0000 7fd863326700 15 client.49618.objecter _session_command_op_assign -1 14 2024-04-14T15:23:17.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.679+0000 7fd863326700 20 client.49618.objecter _get_session s=0x7fd848013310 osd=6 3 2024-04-14T15:23:17.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.679+0000 7fd863326700 20 client.49618.objecter put_session s=0x7fd848013310 osd=6 3 2024-04-14T15:23:17.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.679+0000 7fd863326700 20 client.49618.objecter _get_session s=0x7fd848013310 osd=6 3 2024-04-14T15:23:17.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.679+0000 7fd863326700 15 client.49618.objecter _session_command_op_remove -1 14 2024-04-14T15:23:17.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.679+0000 7fd863326700 20 client.49618.objecter get_session s=0x7fd848013310 osd=6 3 2024-04-14T15:23:17.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.679+0000 7fd863326700 15 client.49618.objecter _session_command_op_assign 6 14 2024-04-14T15:23:17.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.679+0000 7fd863326700 20 client.49618.objecter put_session s=0x7fd848013310 osd=6 4 2024-04-14T15:23:17.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.679+0000 7fd863326700 10 client.49618.objecter _send_command 14 2024-04-14T15:23:17.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.679+0000 7fd863326700 1 -- 172.21.15.90:0/2254210489 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fd848012060 con 0x7fd84801f470 2024-04-14T15:23:17.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.681+0000 7fd85a7fc700 1 -- 172.21.15.90:0/2254210489 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd848012060 con 0x7fd84801f470 2024-04-14T15:23:17.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.681+0000 7fd85a7fc700 10 client.49618.objecter ms_dispatch 0x7fd85c001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:23:17.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.681+0000 7fd85a7fc700 10 client.49618.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:23:17.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.681+0000 7fd85a7fc700 20 client.49618.objecter put_session s=0x7fd848013310 osd=6 4 2024-04-14T15:23:17.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.681+0000 7fd85a7fc700 15 client.49618.objecter _session_command_op_remove 6 14 2024-04-14T15:23:17.681 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:23:17.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.681+0000 7fd863326700 10 client.49618.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:23:17.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.681+0000 7fd863326700 15 client.49618.objecter _session_command_op_assign -1 15 2024-04-14T15:23:17.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.681+0000 7fd863326700 20 client.49618.objecter _get_session s=0x7fd848024840 osd=7 3 2024-04-14T15:23:17.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.681+0000 7fd863326700 20 client.49618.objecter put_session s=0x7fd848024840 osd=7 3 2024-04-14T15:23:17.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.681+0000 7fd863326700 20 client.49618.objecter _get_session s=0x7fd848024840 osd=7 3 2024-04-14T15:23:17.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.681+0000 7fd863326700 15 client.49618.objecter _session_command_op_remove -1 15 2024-04-14T15:23:17.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.681+0000 7fd863326700 20 client.49618.objecter get_session s=0x7fd848024840 osd=7 3 2024-04-14T15:23:17.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.681+0000 7fd863326700 15 client.49618.objecter _session_command_op_assign 7 15 2024-04-14T15:23:17.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.681+0000 7fd863326700 20 client.49618.objecter put_session s=0x7fd848024840 osd=7 4 2024-04-14T15:23:17.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.681+0000 7fd863326700 10 client.49618.objecter _send_command 15 2024-04-14T15:23:17.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.681+0000 7fd863326700 1 -- 172.21.15.90:0/2254210489 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fd84801b2e0 con 0x7fd84801ae80 2024-04-14T15:23:17.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.681+0000 7fd85a7fc700 10 client.49618.objecter ms_handle_connect 0x7fd84801ae80 2024-04-14T15:23:17.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.682+0000 7fd85a7fc700 1 -- 172.21.15.90:0/2254210489 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd84801b2e0 con 0x7fd84801ae80 2024-04-14T15:23:17.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.682+0000 7fd85a7fc700 10 client.49618.objecter ms_dispatch 0x7fd85c001550 command_reply(tid 15: 0 ) 2024-04-14T15:23:17.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.682+0000 7fd85a7fc700 10 client.49618.objecter _finish_command 15 = system:0 2024-04-14T15:23:17.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.682+0000 7fd85a7fc700 20 client.49618.objecter put_session s=0x7fd848024840 osd=7 4 2024-04-14T15:23:17.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.682+0000 7fd85a7fc700 15 client.49618.objecter _session_command_op_remove 7 15 2024-04-14T15:23:17.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.699+0000 7fd863326700 10 client.49618.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:23:17.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.699+0000 7fd863326700 15 client.49618.objecter _session_command_op_assign -1 16 2024-04-14T15:23:17.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.699+0000 7fd863326700 20 client.49618.objecter _get_session s=0x7fd848024840 osd=7 3 2024-04-14T15:23:17.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.699+0000 7fd863326700 20 client.49618.objecter put_session s=0x7fd848024840 osd=7 3 2024-04-14T15:23:17.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.699+0000 7fd863326700 20 client.49618.objecter _get_session s=0x7fd848024840 osd=7 3 2024-04-14T15:23:17.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.699+0000 7fd863326700 15 client.49618.objecter _session_command_op_remove -1 16 2024-04-14T15:23:17.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.699+0000 7fd863326700 20 client.49618.objecter get_session s=0x7fd848024840 osd=7 3 2024-04-14T15:23:17.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.699+0000 7fd863326700 15 client.49618.objecter _session_command_op_assign 7 16 2024-04-14T15:23:17.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.699+0000 7fd863326700 20 client.49618.objecter put_session s=0x7fd848024840 osd=7 4 2024-04-14T15:23:17.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.699+0000 7fd863326700 10 client.49618.objecter _send_command 16 2024-04-14T15:23:17.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.699+0000 7fd863326700 1 -- 172.21.15.90:0/2254210489 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fd84801b2e0 con 0x7fd84801ae80 2024-04-14T15:23:17.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.700+0000 7fd85a7fc700 1 -- 172.21.15.90:0/2254210489 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd84801b2e0 con 0x7fd84801ae80 2024-04-14T15:23:17.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.700+0000 7fd85a7fc700 10 client.49618.objecter ms_dispatch 0x7fd85c001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:23:17.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.700+0000 7fd85a7fc700 10 client.49618.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:23:17.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.700+0000 7fd85a7fc700 20 client.49618.objecter put_session s=0x7fd848024840 osd=7 4 2024-04-14T15:23:17.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.700+0000 7fd85a7fc700 15 client.49618.objecter _session_command_op_remove 7 16 2024-04-14T15:23:17.701 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:23:17.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.701+0000 7fd863326700 10 librados: watch_flush enter 2024-04-14T15:23:17.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.701+0000 7fd863326700 10 librados: watch_flush exit 2024-04-14T15:23:17.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.701+0000 7fd863326700 10 client.49618.objecter close_session for osd.0 2024-04-14T15:23:17.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.701+0000 7fd863326700 1 -- 172.21.15.90:0/2254210489 >> v1:172.21.15.90:6801/3761806885 conn(0x7fd848004730 legacy=0x7fd848002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:17.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.701+0000 7fd863326700 20 client.49618.objecter put_session s=0x7fd8480053e0 osd=0 1 2024-04-14T15:23:17.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.701+0000 7fd863326700 10 client.49618.objecter close_session for osd.1 2024-04-14T15:23:17.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.701+0000 7fd863326700 1 -- 172.21.15.90:0/2254210489 >> v1:172.21.15.90:6805/4170361699 conn(0x7fd8480042a0 legacy=0x7fd84800ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:17.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.701+0000 7fd863326700 20 client.49618.objecter put_session s=0x7fd848004130 osd=1 1 2024-04-14T15:23:17.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.701+0000 7fd863326700 10 client.49618.objecter close_session for osd.2 2024-04-14T15:23:17.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.701+0000 7fd863326700 1 -- 172.21.15.90:0/2254210489 >> v1:172.21.15.90:6809/1686184251 conn(0x7fd848008990 legacy=0x7fd84800f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:17.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.701+0000 7fd863326700 20 client.49618.objecter put_session s=0x7fd848001a00 osd=2 1 2024-04-14T15:23:17.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.701+0000 7fd863326700 10 client.49618.objecter close_session for osd.3 2024-04-14T15:23:17.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.701+0000 7fd863326700 1 -- 172.21.15.90:0/2254210489 >> v1:172.21.15.90:6813/2270166774 conn(0x7fd848011190 legacy=0x7fd8480115f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:17.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.701+0000 7fd863326700 20 client.49618.objecter put_session s=0x7fd848008200 osd=3 1 2024-04-14T15:23:17.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.701+0000 7fd863326700 10 client.49618.objecter close_session for osd.4 2024-04-14T15:23:17.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.701+0000 7fd863326700 1 -- 172.21.15.90:0/2254210489 >> v1:172.21.15.92:6800/3887549733 conn(0x7fd848016ba0 legacy=0x7fd848018f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:17.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.701+0000 7fd863326700 20 client.49618.objecter put_session s=0x7fd8480136f0 osd=4 1 2024-04-14T15:23:17.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.701+0000 7fd863326700 10 client.49618.objecter close_session for osd.5 2024-04-14T15:23:17.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.701+0000 7fd863326700 1 -- 172.21.15.90:0/2254210489 >> v1:172.21.15.92:6804/3244561440 conn(0x7fd848012cd0 legacy=0x7fd84801d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:17.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.701+0000 7fd863326700 20 client.49618.objecter put_session s=0x7fd84800c870 osd=5 1 2024-04-14T15:23:17.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.701+0000 7fd863326700 10 client.49618.objecter close_session for osd.6 2024-04-14T15:23:17.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.701+0000 7fd863326700 1 -- 172.21.15.90:0/2254210489 >> v1:172.21.15.92:6808/636125963 conn(0x7fd84801f470 legacy=0x7fd848021860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:17.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.701+0000 7fd863326700 20 client.49618.objecter put_session s=0x7fd848013310 osd=6 1 2024-04-14T15:23:17.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.701+0000 7fd863326700 10 client.49618.objecter close_session for osd.7 2024-04-14T15:23:17.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.701+0000 7fd8618c3700 1 -- 172.21.15.90:0/2254210489 reap_dead start 2024-04-14T15:23:17.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.701+0000 7fd863326700 1 -- 172.21.15.90:0/2254210489 >> v1:172.21.15.92:6812/1773706716 conn(0x7fd84801ae80 legacy=0x7fd848023050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:17.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.701+0000 7fd8618c3700 1 -- 172.21.15.90:0/2254210489 reap_dead start 2024-04-14T15:23:17.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.701+0000 7fd863326700 20 client.49618.objecter put_session s=0x7fd848024840 osd=7 1 2024-04-14T15:23:17.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.701+0000 7fd863326700 20 client.49618.objecter shutdown clearing up homeless session... 2024-04-14T15:23:17.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.701+0000 7fd863326700 10 client.49618.objecter successfully canceled tick 2024-04-14T15:23:17.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.701+0000 7fd8618c3700 1 -- 172.21.15.90:0/2254210489 reap_dead start 2024-04-14T15:23:17.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.701+0000 7fd8618c3700 1 -- 172.21.15.90:0/2254210489 reap_dead start 2024-04-14T15:23:17.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.701+0000 7fd863326700 1 -- 172.21.15.90:0/2254210489 >> v1:172.21.15.90:6800/420935427 conn(0x7fd844071300 legacy=0x7fd8440737c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:17.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.701+0000 7fd863326700 1 -- 172.21.15.90:0/2254210489 >> v1:172.21.15.92:6789/0 conn(0x7fd85c10b830 legacy=0x7fd85c1a4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:17.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.701+0000 7fd863326700 1 -- 172.21.15.90:0/2254210489 shutdown_connections 2024-04-14T15:23:17.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.701+0000 7fd863326700 1 -- 172.21.15.90:0/2254210489 >> 172.21.15.90:0/2254210489 conn(0x7fd85c0fd290 msgr2=0x7fd85c111790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:23:17.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.701+0000 7fd863326700 1 -- 172.21.15.90:0/2254210489 shutdown_connections 2024-04-14T15:23:17.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.701+0000 7fd863326700 1 -- 172.21.15.90:0/2254210489 wait complete. 2024-04-14T15:23:17.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:17.701+0000 7fd863326700 1 librados: shutdown 2024-04-14T15:23:17.706 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:23:17.707 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:23:17.707 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:23:17.707 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:23:17.707 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:23:17.707 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:23:17.707 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:23:17.707 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:23:17.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:17.708 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:17.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:23:17.809 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:23:17.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:17.835 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:17.835 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:23:17.936 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:23:17.959 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:17.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:17.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:23:18.045 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:18 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2254210489' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:23:18.061 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:23:18.084 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:18.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:18.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:23:18.186 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:23:18.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:17 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2254210489' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:23:18.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:17 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2254210489' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:23:18.210 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:18.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:18.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:23:18.312 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:23:18.313 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-04-14T15:23:18.313 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-04-14T15:23:18.313 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 0 2024-04-14T15:23:18.313 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-04-14T15:23:18.337 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:18.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:18.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:23:18.439 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:23:18.463 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:18.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:18.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:23:18.565 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:18.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:18.590 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:18.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:18.691 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:23:18.715 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:18.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:18.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:23:18.816 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:23:18.840 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:18.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:18.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:23:18.941 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:23:18.965 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:18.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:18.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:23:19.067 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:23:19.083 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd unset noscrub 2024-04-14T15:23:19.091 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:19.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:19.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:23:19.116 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-14T15:23:19.193 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:19.223 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:19.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:19.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:19.325 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:23:19.334 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:19 smithi090 ceph-mon[41670]: pgmap v972: 216 pgs: 1 active+remapped+backfill_toofull, 215 active+clean; 582 KiB data, 584 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 2/603 objects misplaced (0.332%) 2024-04-14T15:23:19.334 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:19 smithi090 ceph-mon[41670]: osdmap e662: 8 total, 8 up, 7 in 2024-04-14T15:23:19.334 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:19 smithi090 ceph-mon[30399]: pgmap v972: 216 pgs: 1 active+remapped+backfill_toofull, 215 active+clean; 582 KiB data, 584 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 2/603 objects misplaced (0.332%) 2024-04-14T15:23:19.335 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:19 smithi090 ceph-mon[30399]: osdmap e662: 8 total, 8 up, 7 in 2024-04-14T15:23:19.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:19.350 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:19.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:23:19.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:19 smithi092 ceph-mon[33966]: pgmap v972: 216 pgs: 1 active+remapped+backfill_toofull, 215 active+clean; 582 KiB data, 584 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 2/603 objects misplaced (0.332%) 2024-04-14T15:23:19.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:19 smithi092 ceph-mon[33966]: osdmap e662: 8 total, 8 up, 7 in 2024-04-14T15:23:19.395 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:23:19.451 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:23:19.474 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:19.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:19.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:23:19.576 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:23:19.599 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:19.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:19.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:23:19.701 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:23:19.724 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:19.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:19.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:23:19.825 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:23:19.848 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:19.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:19.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:23:19.950 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:19.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:19.975 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:19.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:20.076 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:20.099 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:20.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:20.100 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:20.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:20 smithi092 ceph-mon[33966]: osdmap e663: 8 total, 8 up, 7 in 2024-04-14T15:23:20.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:20 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:23:20.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:20 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:23:20.201 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:23:20.225 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:20.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:20.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:23:20.249 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:20 smithi090 ceph-mon[41670]: osdmap e663: 8 total, 8 up, 7 in 2024-04-14T15:23:20.249 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:20 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:23:20.249 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:20 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:23:20.249 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:20 smithi090 ceph-mon[30399]: osdmap e663: 8 total, 8 up, 7 in 2024-04-14T15:23:20.249 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:20 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:23:20.249 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:20 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:23:20.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.253+0000 7ff029676700 1 -- 172.21.15.90:0/1162510420 >> v1:172.21.15.90:6789/0 conn(0x7ff02410e360 legacy=0x7ff024110820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:20.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.253+0000 7ff029676700 1 -- 172.21.15.90:0/1162510420 shutdown_connections 2024-04-14T15:23:20.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.253+0000 7ff029676700 1 -- 172.21.15.90:0/1162510420 >> 172.21.15.90:0/1162510420 conn(0x7ff0240fd2c0 msgr2=0x7ff0240ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:23:20.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.253+0000 7ff029676700 1 -- 172.21.15.90:0/1162510420 shutdown_connections 2024-04-14T15:23:20.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.253+0000 7ff029676700 1 -- 172.21.15.90:0/1162510420 wait complete. 2024-04-14T15:23:20.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.254+0000 7ff029676700 1 librados: starting msgr at 2024-04-14T15:23:20.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.254+0000 7ff029676700 1 librados: starting objecter 2024-04-14T15:23:20.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.254+0000 7ff029676700 1 Processor -- start 2024-04-14T15:23:20.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.254+0000 7ff029676700 1 -- start start 2024-04-14T15:23:20.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.254+0000 7ff029676700 1 librados: setting wanted keys 2024-04-14T15:23:20.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.254+0000 7ff029676700 1 librados: calling monclient init 2024-04-14T15:23:20.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.254+0000 7ff029676700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff02406bd30 con 0x7ff0241078e0 2024-04-14T15:23:20.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.254+0000 7ff029676700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff0241a8fc0 con 0x7ff02410a790 2024-04-14T15:23:20.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.254+0000 7ff029676700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff0241aa200 con 0x7ff02410e360 2024-04-14T15:23:20.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.254+0000 7ff022ffd700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7ff0241078e0 0x7ff02410cf80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:39162/0 (socket says 172.21.15.90:39162) 2024-04-14T15:23:20.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.255+0000 7ff022ffd700 1 -- 172.21.15.90:0/4086940941 learned_addr learned my addr 172.21.15.90:0/4086940941 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:23:20.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.255+0000 7ff00bfff700 10 client.?.objecter ms_handle_connect 0x7ff02410a790 2024-04-14T15:23:20.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.255+0000 7ff00bfff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:23:20.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.255+0000 7ff00bfff700 10 client.?.objecter ms_handle_connect 0x7ff0241078e0 2024-04-14T15:23:20.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.255+0000 7ff00bfff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:23:20.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.255+0000 7ff00bfff700 10 client.?.objecter ms_handle_connect 0x7ff02410e360 2024-04-14T15:23:20.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.255+0000 7ff00bfff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:23:20.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.255+0000 7ff00bfff700 1 -- 172.21.15.90:0/4086940941 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4011504957 0 0) 0x7ff0241a8fc0 con 0x7ff02410a790 2024-04-14T15:23:20.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.255+0000 7ff00bfff700 1 -- 172.21.15.90:0/4086940941 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff00c0035b0 con 0x7ff02410a790 2024-04-14T15:23:20.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.255+0000 7ff00bfff700 1 -- 172.21.15.90:0/4086940941 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1326718260 0 0) 0x7ff02406bd30 con 0x7ff0241078e0 2024-04-14T15:23:20.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.255+0000 7ff00bfff700 1 -- 172.21.15.90:0/4086940941 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0241a8fc0 con 0x7ff0241078e0 2024-04-14T15:23:20.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.255+0000 7ff00bfff700 1 -- 172.21.15.90:0/4086940941 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 357427558 0 0) 0x7ff0241aa200 con 0x7ff02410e360 2024-04-14T15:23:20.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.255+0000 7ff00bfff700 1 -- 172.21.15.90:0/4086940941 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff02406bd30 con 0x7ff02410e360 2024-04-14T15:23:20.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.255+0000 7ff00bfff700 1 -- 172.21.15.90:0/4086940941 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4223079316 0 0) 0x7ff00c0035b0 con 0x7ff02410a790 2024-04-14T15:23:20.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.255+0000 7ff00bfff700 1 -- 172.21.15.90:0/4086940941 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0241aa200 con 0x7ff02410a790 2024-04-14T15:23:20.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.255+0000 7ff00bfff700 1 -- 172.21.15.90:0/4086940941 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7ff0180018b0 con 0x7ff02410a790 2024-04-14T15:23:20.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.255+0000 7ff00bfff700 1 -- 172.21.15.90:0/4086940941 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3877126966 0 0) 0x7ff0241a8fc0 con 0x7ff0241078e0 2024-04-14T15:23:20.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.255+0000 7ff00bfff700 1 -- 172.21.15.90:0/4086940941 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff00c0035b0 con 0x7ff0241078e0 2024-04-14T15:23:20.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.255+0000 7ff00bfff700 1 -- 172.21.15.90:0/4086940941 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7ff0140018b0 con 0x7ff0241078e0 2024-04-14T15:23:20.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.255+0000 7ff00bfff700 1 -- 172.21.15.90:0/4086940941 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2691938101 0 0) 0x7ff0241aa200 con 0x7ff02410a790 2024-04-14T15:23:20.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.255+0000 7ff00bfff700 1 -- 172.21.15.90:0/4086940941 >> v1:172.21.15.90:6790/0 conn(0x7ff02410e360 legacy=0x7ff0241a6850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:20.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.255+0000 7ff00bfff700 1 -- 172.21.15.90:0/4086940941 >> v1:172.21.15.90:6789/0 conn(0x7ff0241078e0 legacy=0x7ff02410cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:20.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.255+0000 7ff00bfff700 1 -- 172.21.15.90:0/4086940941 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0241ab440 con 0x7ff02410a790 2024-04-14T15:23:20.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.255+0000 7ff029676700 1 -- 172.21.15.90:0/4086940941 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff0241aa430 con 0x7ff02410a790 2024-04-14T15:23:20.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.255+0000 7ff029676700 10 client.49624.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:23:20.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.255+0000 7ff029676700 1 -- 172.21.15.90:0/4086940941 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff0241aa8f0 con 0x7ff02410a790 2024-04-14T15:23:20.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.255+0000 7ff029676700 1 librados: init done 2024-04-14T15:23:20.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.255+0000 7ff00bfff700 1 -- 172.21.15.90:0/4086940941 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7ff018003b40 con 0x7ff02410a790 2024-04-14T15:23:20.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.256+0000 7ff00bfff700 1 -- 172.21.15.90:0/4086940941 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7ff018004aa0 con 0x7ff02410a790 2024-04-14T15:23:20.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.256+0000 7ff029676700 10 client.49624.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:23:20.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.256+0000 7ff029676700 15 client.49624.objecter _session_command_op_assign -1 1 2024-04-14T15:23:20.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.256+0000 7ff029676700 20 client.49624.objecter _get_session osd=-1 returning homeless 2024-04-14T15:23:20.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.256+0000 7ff029676700 10 client.49624.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:23:20.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.256+0000 7ff029676700 1 -- 172.21.15.90:0/4086940941 --> v1:172.21.15.92:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7ff010002920 con 0x7ff02410a790 2024-04-14T15:23:20.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.257+0000 7ff00bfff700 1 -- 172.21.15.90:0/4086940941 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7ff018004db0 con 0x7ff02410a790 2024-04-14T15:23:20.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.257+0000 7ff00bfff700 1 -- 172.21.15.90:0/4086940941 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(664..664 src has 467..664) ==== 8711+0+0 (unknown 1274738607 0 0) 0x7ff018089c80 con 0x7ff02410a790 2024-04-14T15:23:20.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.257+0000 7ff00bfff700 10 client.49624.objecter ms_dispatch 0x7ff024001550 osd_map(664..664 src has 467..664) 2024-04-14T15:23:20.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.257+0000 7ff00bfff700 3 client.49624.objecter handle_osd_map got epochs [664,664] > 0 2024-04-14T15:23:20.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.257+0000 7ff00bfff700 3 client.49624.objecter handle_osd_map decoding full epoch 664 2024-04-14T15:23:20.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.258+0000 7ff00bfff700 10 client.49624.objecter checking command 1 2024-04-14T15:23:20.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.258+0000 7ff00bfff700 20 client.49624.objecter _get_session s=0x7ff00c07ee10 osd=0 3 2024-04-14T15:23:20.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.258+0000 7ff00bfff700 20 client.49624.objecter put_session s=0x7ff00c07ee10 osd=0 3 2024-04-14T15:23:20.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.258+0000 7ff00bfff700 15 client.49624.objecter _session_command_op_remove -1 1 2024-04-14T15:23:20.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.258+0000 7ff00bfff700 20 client.49624.objecter _get_session s=0x7ff00c07ee10 osd=0 3 2024-04-14T15:23:20.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.258+0000 7ff00bfff700 20 client.49624.objecter get_session s=0x7ff00c07ee10 osd=0 3 2024-04-14T15:23:20.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.258+0000 7ff00bfff700 15 client.49624.objecter _session_command_op_assign 0 1 2024-04-14T15:23:20.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.258+0000 7ff00bfff700 20 client.49624.objecter put_session s=0x7ff00c07ee10 osd=0 4 2024-04-14T15:23:20.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.258+0000 7ff00bfff700 10 client.49624.objecter _send_command 1 2024-04-14T15:23:20.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.258+0000 7ff00bfff700 1 -- 172.21.15.90:0/4086940941 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff00c0831d0 con 0x7ff00c07f4c0 2024-04-14T15:23:20.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.258+0000 7ff00bfff700 20 client.49624.objecter dump_active .. 0 homeless 2024-04-14T15:23:20.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.258+0000 7ff00bfff700 1 -- 172.21.15.90:0/4086940941 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_get_version_reply(handle=1 version=664) ==== 24+0+0 (unknown 1116767663 0 0) 0x7ff01808a060 con 0x7ff02410a790 2024-04-14T15:23:20.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.258+0000 7ff00bfff700 10 client.49624.objecter ms_handle_connect 0x7ff00c071350 2024-04-14T15:23:20.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.258+0000 7ff00bfff700 10 client.49624.objecter ms_handle_connect 0x7ff00c07f4c0 2024-04-14T15:23:20.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.259+0000 7ff00bfff700 1 -- 172.21.15.90:0/4086940941 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7ff00c0831d0 con 0x7ff00c07f4c0 2024-04-14T15:23:20.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.259+0000 7ff00bfff700 10 client.49624.objecter ms_dispatch 0x7ff024001550 command_reply(tid 1: 0 ) 2024-04-14T15:23:20.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.259+0000 7ff00bfff700 10 client.49624.objecter _finish_command 1 = system:0 2024-04-14T15:23:20.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.259+0000 7ff00bfff700 20 client.49624.objecter put_session s=0x7ff00c07ee10 osd=0 4 2024-04-14T15:23:20.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:20.259+0000 7ff00bfff700 15 client.49624.objecter _session_command_op_remove 0 1 2024-04-14T15:23:20.291 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:23:20.296 INFO:teuthology.orchestra.run.smithi090.stderr:no valid command found; 10 closest matches: 2024-04-14T15:23:20.296 INFO:teuthology.orchestra.run.smithi090.stderr:0 2024-04-14T15:23:20.297 INFO:teuthology.orchestra.run.smithi090.stderr:1 2024-04-14T15:23:20.297 INFO:teuthology.orchestra.run.smithi090.stderr:2 2024-04-14T15:23:20.297 INFO:teuthology.orchestra.run.smithi090.stderr:abort 2024-04-14T15:23:20.297 INFO:teuthology.orchestra.run.smithi090.stderr:assert 2024-04-14T15:23:20.297 INFO:teuthology.orchestra.run.smithi090.stderr:bench [] [] [] [] 2024-04-14T15:23:20.297 INFO:teuthology.orchestra.run.smithi090.stderr:bluefs debug_inject_read_zeros 2024-04-14T15:23:20.297 INFO:teuthology.orchestra.run.smithi090.stderr:bluefs files list 2024-04-14T15:23:20.297 INFO:teuthology.orchestra.run.smithi090.stderr:bluefs stats 2024-04-14T15:23:20.297 INFO:teuthology.orchestra.run.smithi090.stderr:bluestore allocator dump block 2024-04-14T15:23:20.297 INFO:teuthology.orchestra.run.smithi090.stderr:Error EINVAL: invalid command 2024-04-14T15:23:20.328 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:23:20.351 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:20.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:20.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:23:20.453 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:23:20.477 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:20.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:20.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:23:20.578 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:23:20.603 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:20.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:20.604 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:23:20.705 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:23:20.729 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:20.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:20.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:23:20.831 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:23:20.855 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:20.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:20.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:23:20.956 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:23:20.983 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:20.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:20.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:23:21.085 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:21.110 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:21.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:21.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:21.195 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-04-14T15:23:21.196 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-04-14T15:23:21.196 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-04-14T15:23:21.196 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-04-14T15:23:21.196 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-04-14T15:23:21.205 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:21 smithi092 ceph-mon[33966]: pgmap v975: 216 pgs: 26 creating+peering, 6 unknown, 1 active+remapped+backfill_toofull, 183 active+clean; 582 KiB data, 584 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/603 objects misplaced (0.332%) 2024-04-14T15:23:21.205 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:21 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:23:21.205 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:21 smithi092 ceph-mon[33966]: osdmap e664: 8 total, 8 up, 7 in 2024-04-14T15:23:21.205 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:21 smithi092 ceph-mon[33966]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:23:21.205 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:21 smithi092 ceph-mon[33966]: osdmap e665: 8 total, 8 up, 7 in 2024-04-14T15:23:21.205 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:21 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:23:21.205 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:21 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:23:21.212 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:23:21.235 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:21.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:21.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:23:21.337 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:23:21.351 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:21 smithi090 ceph-mon[41670]: pgmap v975: 216 pgs: 26 creating+peering, 6 unknown, 1 active+remapped+backfill_toofull, 183 active+clean; 582 KiB data, 584 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/603 objects misplaced (0.332%) 2024-04-14T15:23:21.351 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:21 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:23:21.351 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:21 smithi090 ceph-mon[41670]: osdmap e664: 8 total, 8 up, 7 in 2024-04-14T15:23:21.351 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:21 smithi090 ceph-mon[41670]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:23:21.351 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:21 smithi090 ceph-mon[41670]: osdmap e665: 8 total, 8 up, 7 in 2024-04-14T15:23:21.351 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:21 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:23:21.351 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:21 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:23:21.351 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:21 smithi090 ceph-mon[30399]: pgmap v975: 216 pgs: 26 creating+peering, 6 unknown, 1 active+remapped+backfill_toofull, 183 active+clean; 582 KiB data, 584 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/603 objects misplaced (0.332%) 2024-04-14T15:23:21.352 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:21 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:23:21.352 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:21 smithi090 ceph-mon[30399]: osdmap e664: 8 total, 8 up, 7 in 2024-04-14T15:23:21.352 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:21 smithi090 ceph-mon[30399]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:23:21.352 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:21 smithi090 ceph-mon[30399]: osdmap e665: 8 total, 8 up, 7 in 2024-04-14T15:23:21.352 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:21 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:23:21.352 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:21 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:23:21.361 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:21.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:21.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:23:21.463 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:23:21.487 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:21.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:21.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:23:21.588 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:21.612 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:21.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:21.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:21.714 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:23:21.737 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-14T15:23:21.737 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:21.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:21.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:23:21.839 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:23:21.862 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:21.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:21.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:23:21.964 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:23:21.990 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:21.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:21.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:23:22.050 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-14T15:23:22.092 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:23:22.115 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:22.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:22.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:23:22.217 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:22.242 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:22.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:22.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:22.286 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:23:22.344 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:23:22.367 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:22.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:22.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:23:22.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:22 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:23:22.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:22 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:23:22.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:22 smithi092 ceph-mon[33966]: pgmap v978: 216 pgs: 1 active+remapped+backfill_toofull, 32 creating+peering, 183 active+clean; 582 KiB data, 584 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/603 objects misplaced (0.332%) 2024-04-14T15:23:22.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:22 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-116", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:23:22.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:22 smithi092 ceph-mon[33966]: osdmap e666: 8 total, 8 up, 7 in 2024-04-14T15:23:22.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:22 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-116"}]: dispatch 2024-04-14T15:23:22.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:22 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-116"}]: dispatch 2024-04-14T15:23:22.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:22 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:23:22.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:22 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:23:22.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:22 smithi090 ceph-mon[30399]: pgmap v978: 216 pgs: 1 active+remapped+backfill_toofull, 32 creating+peering, 183 active+clean; 582 KiB data, 584 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/603 objects misplaced (0.332%) 2024-04-14T15:23:22.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:22 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-116", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:23:22.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:22 smithi090 ceph-mon[30399]: osdmap e666: 8 total, 8 up, 7 in 2024-04-14T15:23:22.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:22 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-116"}]: dispatch 2024-04-14T15:23:22.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:22 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-116"}]: dispatch 2024-04-14T15:23:22.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:22 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:23:22.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:22 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:23:22.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:22 smithi090 ceph-mon[41670]: pgmap v978: 216 pgs: 1 active+remapped+backfill_toofull, 32 creating+peering, 183 active+clean; 582 KiB data, 584 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/603 objects misplaced (0.332%) 2024-04-14T15:23:22.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:22 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-116", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:23:22.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:22 smithi090 ceph-mon[41670]: osdmap e666: 8 total, 8 up, 7 in 2024-04-14T15:23:22.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:22 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-116"}]: dispatch 2024-04-14T15:23:22.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:22 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-116"}]: dispatch 2024-04-14T15:23:22.469 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:23:22.494 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:22.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:22.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:23:22.539 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-14T15:23:22.539 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-14T15:23:22.539 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph tell osd.1 'trim stale osdmaps' 2024-04-14T15:23:22.596 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:23:22.619 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:22.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:22.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:23:22.721 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:23:22.729 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:23:22.752 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:22.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:22.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:23:22.854 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:23:22.877 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:22.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:22.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:23:22.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:22.959+0000 7febf3688700 1 -- 172.21.15.90:0/3389453892 >> v1:172.21.15.90:6789/0 conn(0x7febec108880 legacy=0x7febec108c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:22.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:22.959+0000 7febf3688700 1 -- 172.21.15.90:0/3389453892 shutdown_connections 2024-04-14T15:23:22.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:22.959+0000 7febf3688700 1 -- 172.21.15.90:0/3389453892 >> 172.21.15.90:0/3389453892 conn(0x7febec0fe390 msgr2=0x7febec1007b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:23:22.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:22.959+0000 7febf3688700 1 -- 172.21.15.90:0/3389453892 shutdown_connections 2024-04-14T15:23:22.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:22.959+0000 7febf3688700 1 -- 172.21.15.90:0/3389453892 wait complete. 2024-04-14T15:23:22.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:22.960+0000 7febf3688700 1 librados: starting msgr at 2024-04-14T15:23:22.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:22.960+0000 7febf3688700 1 librados: starting objecter 2024-04-14T15:23:22.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:22.960+0000 7febf3688700 1 Processor -- start 2024-04-14T15:23:22.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:22.960+0000 7febf3688700 1 -- start start 2024-04-14T15:23:22.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:22.960+0000 7febf3688700 1 librados: setting wanted keys 2024-04-14T15:23:22.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:22.960+0000 7febf3688700 1 librados: calling monclient init 2024-04-14T15:23:22.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:22.960+0000 7febf3688700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7febec10f030 con 0x7febec108880 2024-04-14T15:23:22.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:22.960+0000 7febf3688700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7febec1a8f10 con 0x7febec10f300 2024-04-14T15:23:22.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:22.960+0000 7febf3688700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7febec1aa0f0 con 0x7febec10b730 2024-04-14T15:23:22.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:22.960+0000 7febf1424700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7febec108880 0x7febec10bf90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:46264/0 (socket says 172.21.15.90:46264) 2024-04-14T15:23:22.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:22.960+0000 7febf1424700 1 -- 172.21.15.90:0/2378226622 learned_addr learned my addr 172.21.15.90:0/2378226622 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:23:22.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:22.960+0000 7febe27fc700 10 client.?.objecter ms_handle_connect 0x7febec108880 2024-04-14T15:23:22.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:22.960+0000 7febe27fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:23:22.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:22.960+0000 7febe27fc700 10 client.?.objecter ms_handle_connect 0x7febec10f300 2024-04-14T15:23:22.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:22.960+0000 7febe27fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:23:22.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:22.961+0000 7febe27fc700 10 client.?.objecter ms_handle_connect 0x7febec10b730 2024-04-14T15:23:22.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:22.961+0000 7febe27fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:23:22.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:22.961+0000 7febe27fc700 1 -- 172.21.15.90:0/2378226622 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1339665386 0 0) 0x7febec10f030 con 0x7febec108880 2024-04-14T15:23:22.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:22.961+0000 7febe27fc700 1 -- 172.21.15.90:0/2378226622 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7febd80035b0 con 0x7febec108880 2024-04-14T15:23:22.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:22.961+0000 7febe27fc700 1 -- 172.21.15.90:0/2378226622 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1850313202 0 0) 0x7febec1aa0f0 con 0x7febec10b730 2024-04-14T15:23:22.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:22.961+0000 7febe27fc700 1 -- 172.21.15.90:0/2378226622 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7febec10f030 con 0x7febec10b730 2024-04-14T15:23:22.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:22.961+0000 7febe27fc700 1 -- 172.21.15.90:0/2378226622 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2494826659 0 0) 0x7febec1a8f10 con 0x7febec10f300 2024-04-14T15:23:22.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:22.961+0000 7febe27fc700 1 -- 172.21.15.90:0/2378226622 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7febec1aa0f0 con 0x7febec10f300 2024-04-14T15:23:22.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:22.961+0000 7febe27fc700 1 -- 172.21.15.90:0/2378226622 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2405973898 0 0) 0x7febec1aa0f0 con 0x7febec10f300 2024-04-14T15:23:22.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:22.961+0000 7febe27fc700 1 -- 172.21.15.90:0/2378226622 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7febec1a8f10 con 0x7febec10f300 2024-04-14T15:23:22.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:22.961+0000 7febe27fc700 1 -- 172.21.15.90:0/2378226622 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2213221649 0 0) 0x7febd80035b0 con 0x7febec108880 2024-04-14T15:23:22.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:22.961+0000 7febe27fc700 1 -- 172.21.15.90:0/2378226622 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7febec1aa0f0 con 0x7febec108880 2024-04-14T15:23:22.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:22.961+0000 7febe27fc700 1 -- 172.21.15.90:0/2378226622 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 368906034 0 0) 0x7febec10f030 con 0x7febec10b730 2024-04-14T15:23:22.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:22.961+0000 7febe27fc700 1 -- 172.21.15.90:0/2378226622 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7febd80035b0 con 0x7febec10b730 2024-04-14T15:23:22.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:22.961+0000 7febe27fc700 1 -- 172.21.15.90:0/2378226622 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7febe4001bb0 con 0x7febec10f300 2024-04-14T15:23:22.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:22.961+0000 7febe27fc700 1 -- 172.21.15.90:0/2378226622 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7febe80015d0 con 0x7febec108880 2024-04-14T15:23:22.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:22.961+0000 7febe27fc700 1 -- 172.21.15.90:0/2378226622 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7febdc001720 con 0x7febec10b730 2024-04-14T15:23:22.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:22.961+0000 7febe27fc700 1 -- 172.21.15.90:0/2378226622 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1441669456 0 0) 0x7febec1a8f10 con 0x7febec10f300 2024-04-14T15:23:22.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:22.961+0000 7febe27fc700 1 -- 172.21.15.90:0/2378226622 >> v1:172.21.15.90:6790/0 conn(0x7febec10b730 legacy=0x7febec10c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:22.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:22.961+0000 7febe27fc700 1 -- 172.21.15.90:0/2378226622 >> v1:172.21.15.90:6789/0 conn(0x7febec108880 legacy=0x7febec10bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:22.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:22.961+0000 7febe27fc700 1 -- 172.21.15.90:0/2378226622 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febec1ab2d0 con 0x7febec10f300 2024-04-14T15:23:22.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:22.962+0000 7febf3688700 1 -- 172.21.15.90:0/2378226622 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7febec1a9140 con 0x7febec10f300 2024-04-14T15:23:22.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:22.962+0000 7febf3688700 10 client.49630.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:23:22.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:22.962+0000 7febf3688700 1 -- 172.21.15.90:0/2378226622 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7febec1a9600 con 0x7febec10f300 2024-04-14T15:23:22.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:22.962+0000 7febf3688700 1 librados: init done 2024-04-14T15:23:22.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:22.962+0000 7febe27fc700 1 -- 172.21.15.90:0/2378226622 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7febe4003ec0 con 0x7febec10f300 2024-04-14T15:23:22.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:22.962+0000 7febe27fc700 1 -- 172.21.15.90:0/2378226622 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7febe4004d10 con 0x7febec10f300 2024-04-14T15:23:22.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:22.963+0000 7febf3688700 1 -- 172.21.15.90:0/2378226622 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febd0005150 con 0x7febec10f300 2024-04-14T15:23:22.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:22.963+0000 7febe27fc700 1 -- 172.21.15.90:0/2378226622 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7febe4003a20 con 0x7febec10f300 2024-04-14T15:23:22.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:22.964+0000 7febe27fc700 1 -- 172.21.15.90:0/2378226622 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(666..666 src has 467..666) ==== 8673+0+0 (unknown 3447766639 0 0) 0x7febe4089b50 con 0x7febec10f300 2024-04-14T15:23:22.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:22.964+0000 7febe27fc700 10 client.49630.objecter ms_dispatch 0x7febec001550 osd_map(666..666 src has 467..666) 2024-04-14T15:23:22.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:22.964+0000 7febe27fc700 3 client.49630.objecter handle_osd_map got epochs [666,666] > 0 2024-04-14T15:23:22.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:22.964+0000 7febe27fc700 3 client.49630.objecter handle_osd_map decoding full epoch 666 2024-04-14T15:23:22.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:22.964+0000 7febe27fc700 20 client.49630.objecter dump_active .. 0 homeless 2024-04-14T15:23:22.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:22.964+0000 7febe27fc700 10 client.49630.objecter ms_handle_connect 0x7febd8071490 2024-04-14T15:23:22.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:22.972+0000 7febe27fc700 1 -- 172.21.15.90:0/2378226622 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7febe4056360 con 0x7febec10f300 2024-04-14T15:23:22.979 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:23:23.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:23.008 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:23.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:23:23.109 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:23:23.133 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:23.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:23.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:23:23.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:23.203+0000 7febf3688700 1 -- 172.21.15.90:0/2378226622 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7febd0005440 con 0x7febec10f300 2024-04-14T15:23:23.235 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:23:23.258 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:23.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:23.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:23:23.360 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:23:23.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:23.385 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:23.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:23:23.486 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:23:23.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:23.511 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:23.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:23:23.612 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:23:23.630 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:23:23.636 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:23.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:23.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:23:23.737 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:23:23.761 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:23.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:23.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:23:23.863 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:23:23.887 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:23.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:23.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:23:23.989 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:23:23.992 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:23 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-116"}]': finished 2024-04-14T15:23:23.992 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:23 smithi092 ceph-mon[33966]: osdmap e667: 8 total, 8 up, 7 in 2024-04-14T15:23:23.992 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:23 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-116", "mode": "writeback"}]: dispatch 2024-04-14T15:23:23.992 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:23 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-116", "mode": "writeback"}]: dispatch 2024-04-14T15:23:23.992 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:23 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2378226622' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:23:23.992 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:23 smithi092 ceph-mon[33966]: from='client.49630 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:23:23.994 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:23:23 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:23:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:23:23.995 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:23 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-116"}]': finished 2024-04-14T15:23:23.995 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:23 smithi090 ceph-mon[41670]: osdmap e667: 8 total, 8 up, 7 in 2024-04-14T15:23:23.995 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:23 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-116", "mode": "writeback"}]: dispatch 2024-04-14T15:23:23.995 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:23 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-116", "mode": "writeback"}]: dispatch 2024-04-14T15:23:23.995 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:23 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2378226622' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:23:23.995 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:23 smithi090 ceph-mon[41670]: from='client.49630 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:23:23.996 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:23 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-116"}]': finished 2024-04-14T15:23:23.996 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:23 smithi090 ceph-mon[30399]: osdmap e667: 8 total, 8 up, 7 in 2024-04-14T15:23:23.996 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:23 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-116", "mode": "writeback"}]: dispatch 2024-04-14T15:23:23.996 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:23 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-116", "mode": "writeback"}]: dispatch 2024-04-14T15:23:23.996 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:23 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2378226622' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:23:23.997 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:23 smithi090 ceph-mon[30399]: from='client.49630 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:23:23.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:23.996+0000 7febe27fc700 1 -- 172.21.15.90:0/2378226622 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v668) ==== 91+0+0 (unknown 3031314437 0 0) 0x7febe405a010 con 0x7febec10f300 2024-04-14T15:23:24.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:24.000+0000 7febf3688700 10 librados: watch_flush enter 2024-04-14T15:23:24.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:24.000+0000 7febf3688700 10 librados: watch_flush exit 2024-04-14T15:23:24.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:24.000+0000 7febf3688700 20 client.49630.objecter shutdown clearing up homeless session... 2024-04-14T15:23:24.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:24.000+0000 7febf3688700 10 client.49630.objecter successfully canceled tick 2024-04-14T15:23:24.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:24.000+0000 7febf3688700 1 -- 172.21.15.90:0/2378226622 >> v1:172.21.15.90:6800/420935427 conn(0x7febd8071490 legacy=0x7febd8073950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:24.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:24.000+0000 7febf3688700 1 -- 172.21.15.90:0/2378226622 >> v1:172.21.15.92:6789/0 conn(0x7febec10f300 legacy=0x7febec1a67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:24.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:24.000+0000 7febf3688700 1 -- 172.21.15.90:0/2378226622 shutdown_connections 2024-04-14T15:23:24.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:24.000+0000 7febf3688700 1 -- 172.21.15.90:0/2378226622 >> 172.21.15.90:0/2378226622 conn(0x7febec0fe390 msgr2=0x7febec1128d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:23:24.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:24.001+0000 7febf3688700 1 -- 172.21.15.90:0/2378226622 shutdown_connections 2024-04-14T15:23:24.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:24.001+0000 7febf3688700 1 -- 172.21.15.90:0/2378226622 wait complete. 2024-04-14T15:23:24.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:24.001+0000 7febf3688700 1 librados: shutdown 2024-04-14T15:23:24.002 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is unset 2024-04-14T15:23:24.020 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:24.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:24.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:23:24.042 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-04-14T15:23:24.042 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-04-14T15:23:24.043 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-04-14T15:23:24.043 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-04-14T15:23:24.043 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.043 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.41446.0:2 152.0 152:0384ab8e:::oid:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-04-14T15:23:24.043 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:13:24.858105+0000", 2024-04-14T15:23:24.043 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 599.18376330700005, 2024-04-14T15:23:24.043 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0159813529999999, 2024-04-14T15:23:24.043 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:23:24.043 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:23:24.043 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:23:24.043 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:23:24.043 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.41446", 2024-04-14T15:23:24.043 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/3264560952", 2024-04-14T15:23:24.043 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 2 2024-04-14T15:23:24.043 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.044 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:23:24.044 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.044 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:23:24.044 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:24.858105+0000", 2024-04-14T15:23:24.044 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:23:24.044 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.044 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.044 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:23:24.044 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:24.858105+0000", 2024-04-14T15:23:24.044 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:23:24.044 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.044 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.044 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:23:24.045 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:24.858107+0000", 2024-04-14T15:23:24.045 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0679999999999999e-06 2024-04-14T15:23:24.045 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.045 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.045 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:23:24.045 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:24.858113+0000", 2024-04-14T15:23:24.045 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.327e-06 2024-04-14T15:23:24.045 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.045 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.045 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:23:24.045 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:24.858123+0000", 2024-04-14T15:23:24.045 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0560999999999999e-05 2024-04-14T15:23:24.045 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.045 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.045 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:23:24.045 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:24.858134+0000", 2024-04-14T15:23:24.046 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0522e-05 2024-04-14T15:23:24.046 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.046 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.046 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:23:24.046 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:24.861997+0000", 2024-04-14T15:23:24.046 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0038629200000000002 2024-04-14T15:23:24.046 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.046 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.046 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:23:24.046 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:24.862004+0000", 2024-04-14T15:23:24.046 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5240000000000003e-06 2024-04-14T15:23:24.046 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.046 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.046 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:23:24.047 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.853997+0000", 2024-04-14T15:23:24.047 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.99199313700000002 2024-04-14T15:23:24.047 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.047 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.047 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:23:24.047 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.855171+0000", 2024-04-14T15:23:24.047 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0011740120000000001 2024-04-14T15:23:24.047 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.047 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.047 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,4", 2024-04-14T15:23:24.047 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.855307+0000", 2024-04-14T15:23:24.047 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013530200000000001 2024-04-14T15:23:24.047 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.047 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.047 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:23:24.048 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.857608+0000", 2024-04-14T15:23:24.048 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0023010320000000002 2024-04-14T15:23:24.048 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.048 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.048 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:23:24.048 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.858583+0000", 2024-04-14T15:23:24.048 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00097578499999999996 2024-04-14T15:23:24.048 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.048 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.048 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:23:24.048 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.873837+0000", 2024-04-14T15:23:24.048 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.015253697 2024-04-14T15:23:24.048 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.048 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.048 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:23:24.048 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.874020+0000", 2024-04-14T15:23:24.049 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00018252399999999999 2024-04-14T15:23:24.049 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.049 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.049 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:23:24.049 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.874087+0000", 2024-04-14T15:23:24.049 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.6941999999999997e-05 2024-04-14T15:23:24.049 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:23:24.049 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:23:24.049 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:23:24.049 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.049 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.049 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.41446.0:4 152.0 152:0384ab8e:::oid:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-04-14T15:23:24.049 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:13:25.879604+0000", 2024-04-14T15:23:24.050 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 598.16226457300002, 2024-04-14T15:23:24.050 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.033643864000000002, 2024-04-14T15:23:24.050 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:23:24.050 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:23:24.050 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:23:24.050 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:23:24.050 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.41446", 2024-04-14T15:23:24.050 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/3264560952", 2024-04-14T15:23:24.050 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 4 2024-04-14T15:23:24.050 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.050 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:23:24.050 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.050 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:23:24.050 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.879604+0000", 2024-04-14T15:23:24.051 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:23:24.051 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.051 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.051 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:23:24.051 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.879604+0000", 2024-04-14T15:23:24.051 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:23:24.051 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.051 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.051 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:23:24.051 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.879605+0000", 2024-04-14T15:23:24.051 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.7300000000000005e-07 2024-04-14T15:23:24.051 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.051 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.052 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:23:24.052 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.879608+0000", 2024-04-14T15:23:24.052 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8909999999999999e-06 2024-04-14T15:23:24.052 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.052 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.052 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:23:24.052 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.879616+0000", 2024-04-14T15:23:24.052 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.4030000000000007e-06 2024-04-14T15:23:24.052 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.052 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.052 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:23:24.052 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.879621+0000", 2024-04-14T15:23:24.052 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.057e-06 2024-04-14T15:23:24.053 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.053 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.053 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:23:24.053 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.879661+0000", 2024-04-14T15:23:24.053 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0209000000000001e-05 2024-04-14T15:23:24.053 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.053 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.053 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:23:24.053 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.879714+0000", 2024-04-14T15:23:24.053 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2561e-05 2024-04-14T15:23:24.054 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.054 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.054 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,4", 2024-04-14T15:23:24.054 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.879820+0000", 2024-04-14T15:23:24.054 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000105525 2024-04-14T15:23:24.054 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.054 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.054 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:23:24.054 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.880121+0000", 2024-04-14T15:23:24.054 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00030120100000000002 2024-04-14T15:23:24.054 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.054 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.054 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:23:24.055 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.880415+0000", 2024-04-14T15:23:24.055 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00029442999999999998 2024-04-14T15:23:24.055 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.055 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.055 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:23:24.055 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.913166+0000", 2024-04-14T15:23:24.055 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.032750385 2024-04-14T15:23:24.055 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.055 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.055 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:23:24.055 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.913212+0000", 2024-04-14T15:23:24.055 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6768999999999999e-05 2024-04-14T15:23:24.055 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.055 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.055 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:23:24.056 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:25.913248+0000", 2024-04-14T15:23:24.056 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5559999999999998e-05 2024-04-14T15:23:24.056 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:23:24.056 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:23:24.056 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:23:24.056 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.056 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.056 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.38086.0:4 149.1s0 149:83e8722a:test-rados-api-smithi090-93709-12::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-04-14T15:23:24.056 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:13:26.870731+0000", 2024-04-14T15:23:24.057 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 597.17113717799998, 2024-04-14T15:23:24.057 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.089545462000000006, 2024-04-14T15:23:24.057 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:23:24.057 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:23:24.057 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:23:24.057 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:23:24.057 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.38086", 2024-04-14T15:23:24.057 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/2598250370", 2024-04-14T15:23:24.057 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 4 2024-04-14T15:23:24.058 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.058 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:23:24.058 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.058 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:23:24.058 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.870731+0000", 2024-04-14T15:23:24.058 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:23:24.058 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.058 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.058 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:23:24.058 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.870731+0000", 2024-04-14T15:23:24.058 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:23:24.058 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.058 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.059 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:23:24.059 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.870733+0000", 2024-04-14T15:23:24.059 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4869999999999999e-06 2024-04-14T15:23:24.059 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.059 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.059 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:23:24.059 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.870740+0000", 2024-04-14T15:23:24.059 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0609999999999998e-06 2024-04-14T15:23:24.059 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.059 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.059 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:23:24.059 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.870749+0000", 2024-04-14T15:23:24.059 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.9330000000000007e-06 2024-04-14T15:23:24.059 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.060 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.060 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:23:24.060 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.870759+0000", 2024-04-14T15:23:24.060 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0057999999999999e-05 2024-04-14T15:23:24.060 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.060 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.060 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:23:24.060 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.871129+0000", 2024-04-14T15:23:24.060 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00037049300000000001 2024-04-14T15:23:24.060 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.060 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.060 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:23:24.060 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.871258+0000", 2024-04-14T15:23:24.060 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012877 2024-04-14T15:23:24.060 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.060 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.061 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_started", 2024-04-14T15:23:24.061 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.873490+0000", 2024-04-14T15:23:24.061 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002231813 2024-04-14T15:23:24.061 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.061 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.061 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_committed", 2024-04-14T15:23:24.061 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.876587+0000", 2024-04-14T15:23:24.061 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.003097074 2024-04-14T15:23:24.061 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.061 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.061 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:23:24.061 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.960175+0000", 2024-04-14T15:23:24.061 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.083587794000000007 2024-04-14T15:23:24.061 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.062 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.062 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:23:24.062 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:26.960277+0000", 2024-04-14T15:23:24.062 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010197899999999999 2024-04-14T15:23:24.062 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:23:24.062 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:23:24.062 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:23:24.062 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.062 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.062 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.38086.0:7 149.6s0 149:7a93d93e:test-rados-api-smithi090-93709-13::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-14T15:23:24.062 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:13:28.848462+0000", 2024-04-14T15:23:24.062 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 595.19340630199997, 2024-04-14T15:23:24.062 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.051448817000000001, 2024-04-14T15:23:24.063 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:23:24.063 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:23:24.063 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:23:24.063 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:23:24.063 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.38086", 2024-04-14T15:23:24.063 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/2598250370", 2024-04-14T15:23:24.063 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 7 2024-04-14T15:23:24.063 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.063 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:23:24.063 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.063 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:23:24.063 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.848462+0000", 2024-04-14T15:23:24.063 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:23:24.063 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.063 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.063 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:23:24.064 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.848462+0000", 2024-04-14T15:23:24.064 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:23:24.064 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.064 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.064 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:23:24.064 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.848464+0000", 2024-04-14T15:23:24.064 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9420000000000002e-06 2024-04-14T15:23:24.064 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.064 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.064 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:23:24.064 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.848471+0000", 2024-04-14T15:23:24.064 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1099999999999997e-06 2024-04-14T15:23:24.064 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.064 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.064 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:23:24.065 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.850165+0000", 2024-04-14T15:23:24.065 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001693214 2024-04-14T15:23:24.065 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.065 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.065 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:23:24.065 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.854375+0000", 2024-04-14T15:23:24.065 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0042108400000000004 2024-04-14T15:23:24.065 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.065 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.065 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:23:24.065 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.855368+0000", 2024-04-14T15:23:24.065 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0009924650000000001 2024-04-14T15:23:24.065 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.065 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.065 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op must wait for map", 2024-04-14T15:23:24.066 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.855375+0000", 2024-04-14T15:23:24.066 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1389999999999998e-06 2024-04-14T15:23:24.066 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.066 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.066 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:23:24.066 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.864124+0000", 2024-04-14T15:23:24.066 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0087492400000000001 2024-04-14T15:23:24.066 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.066 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.066 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:23:24.066 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.865419+0000", 2024-04-14T15:23:24.066 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0012949000000000001 2024-04-14T15:23:24.066 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.066 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.066 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_started", 2024-04-14T15:23:24.066 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.866179+0000", 2024-04-14T15:23:24.067 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00076017100000000002 2024-04-14T15:23:24.067 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.067 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.067 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_committed", 2024-04-14T15:23:24.067 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.866887+0000", 2024-04-14T15:23:24.067 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00070775799999999995 2024-04-14T15:23:24.067 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.067 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.067 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:23:24.067 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.899834+0000", 2024-04-14T15:23:24.067 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.032947392999999998 2024-04-14T15:23:24.067 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.067 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.067 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:23:24.067 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:28.899911+0000", 2024-04-14T15:23:24.067 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.6644999999999995e-05 2024-04-14T15:23:24.068 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:23:24.068 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:23:24.068 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:23:24.068 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.068 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.068 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.47521.0:2 163.0 163:0384ab8e:::oid:head [append 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-04-14T15:23:24.068 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:13:30.817720+0000", 2024-04-14T15:23:24.068 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 593.22414864300004, 2024-04-14T15:23:24.068 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.074209873, 2024-04-14T15:23:24.068 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:23:24.068 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:23:24.068 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:23:24.068 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:23:24.068 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.47521", 2024-04-14T15:23:24.068 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/3376207688", 2024-04-14T15:23:24.068 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 2 2024-04-14T15:23:24.069 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.069 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:23:24.069 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.069 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:23:24.069 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:30.817720+0000", 2024-04-14T15:23:24.069 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:23:24.069 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.069 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.069 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:23:24.069 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:30.817720+0000", 2024-04-14T15:23:24.069 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:23:24.069 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.069 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.070 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:23:24.070 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:30.817722+0000", 2024-04-14T15:23:24.070 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2419999999999999e-06 2024-04-14T15:23:24.070 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.070 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.070 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:23:24.070 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:30.817730+0000", 2024-04-14T15:23:24.070 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5259999999999996e-06 2024-04-14T15:23:24.070 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.070 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.070 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:23:24.070 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:30.817739+0000", 2024-04-14T15:23:24.071 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.2029999999999998e-06 2024-04-14T15:23:24.071 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.071 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.071 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:23:24.071 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:30.817766+0000", 2024-04-14T15:23:24.071 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.72e-05 2024-04-14T15:23:24.071 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.071 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.071 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:23:24.071 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:30.820672+0000", 2024-04-14T15:23:24.071 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029057979999999998 2024-04-14T15:23:24.071 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.071 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.071 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:23:24.072 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:30.820681+0000", 2024-04-14T15:23:24.072 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0240000000000003e-06 2024-04-14T15:23:24.072 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.072 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.072 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:23:24.072 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:31.877336+0000", 2024-04-14T15:23:24.072 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0566553460000001 2024-04-14T15:23:24.072 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.072 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.072 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:23:24.072 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:31.877442+0000", 2024-04-14T15:23:24.072 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000105764 2024-04-14T15:23:24.072 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.072 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.072 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,4", 2024-04-14T15:23:24.073 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:31.877591+0000", 2024-04-14T15:23:24.073 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014861599999999999 2024-04-14T15:23:24.073 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.073 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.073 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:23:24.073 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:31.878073+0000", 2024-04-14T15:23:24.073 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00048190800000000002 2024-04-14T15:23:24.073 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.073 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.073 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:23:24.073 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:31.878967+0000", 2024-04-14T15:23:24.073 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00089447500000000004 2024-04-14T15:23:24.073 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.073 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.073 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:23:24.074 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:31.891870+0000", 2024-04-14T15:23:24.074 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.012903402 2024-04-14T15:23:24.074 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.074 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.074 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:23:24.074 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:31.891901+0000", 2024-04-14T15:23:24.074 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1012000000000001e-05 2024-04-14T15:23:24.074 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.074 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.074 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:23:24.074 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:31.891930+0000", 2024-04-14T15:23:24.074 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8357e-05 2024-04-14T15:23:24.074 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:23:24.074 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:23:24.074 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:23:24.075 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.075 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.075 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.47545.0:2 165.6 165:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-04-14T15:23:24.075 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:13:31.832019+0000", 2024-04-14T15:23:24.075 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 592.20984938100003, 2024-04-14T15:23:24.075 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.052401587999999999, 2024-04-14T15:23:24.075 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:23:24.075 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:23:24.075 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:23:24.075 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:23:24.075 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.47545", 2024-04-14T15:23:24.075 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/3859057618", 2024-04-14T15:23:24.075 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 2 2024-04-14T15:23:24.075 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.075 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:23:24.075 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.076 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:23:24.076 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:31.832019+0000", 2024-04-14T15:23:24.076 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:23:24.076 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.076 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.076 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:23:24.076 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:31.832019+0000", 2024-04-14T15:23:24.076 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:23:24.076 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.076 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.076 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:23:24.076 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:31.832021+0000", 2024-04-14T15:23:24.076 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.092e-06 2024-04-14T15:23:24.076 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.076 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.077 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:23:24.077 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:31.832029+0000", 2024-04-14T15:23:24.077 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.6090000000000003e-06 2024-04-14T15:23:24.077 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.077 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.077 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:23:24.077 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:31.834577+0000", 2024-04-14T15:23:24.077 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0025480989999999999 2024-04-14T15:23:24.077 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.077 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.077 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:23:24.077 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:31.834592+0000", 2024-04-14T15:23:24.077 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4891e-05 2024-04-14T15:23:24.077 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.077 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.077 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:23:24.078 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:31.837068+0000", 2024-04-14T15:23:24.078 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0024762460000000001 2024-04-14T15:23:24.078 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.078 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.078 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:23:24.078 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:31.837077+0000", 2024-04-14T15:23:24.078 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.7669999999999992e-06 2024-04-14T15:23:24.078 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.078 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.078 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:23:24.078 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:31.872224+0000", 2024-04-14T15:23:24.078 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.035147014999999997 2024-04-14T15:23:24.078 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.078 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.079 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:23:24.079 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:31.872317+0000", 2024-04-14T15:23:24.079 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.3546000000000002e-05 2024-04-14T15:23:24.079 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.079 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.079 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,4", 2024-04-14T15:23:24.079 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:31.872441+0000", 2024-04-14T15:23:24.079 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012318400000000001 2024-04-14T15:23:24.079 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.079 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.079 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:23:24.079 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:31.873715+0000", 2024-04-14T15:23:24.079 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0012740150000000001 2024-04-14T15:23:24.079 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.079 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.079 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:23:24.080 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:31.874641+0000", 2024-04-14T15:23:24.080 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00092663700000000001 2024-04-14T15:23:24.080 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.080 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.080 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:23:24.080 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:31.884289+0000", 2024-04-14T15:23:24.080 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0096479400000000007 2024-04-14T15:23:24.080 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.080 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.080 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:23:24.080 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:31.884375+0000", 2024-04-14T15:23:24.080 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5916999999999997e-05 2024-04-14T15:23:24.080 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.080 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.080 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:23:24.080 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:13:31.884421+0000", 2024-04-14T15:23:24.081 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5630000000000002e-05 2024-04-14T15:23:24.081 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:23:24.081 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:23:24.081 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:23:24.081 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.081 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.081 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.48388.0:2 214.c 214:311cbe94:::oid_ExecuteClassPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e195)", 2024-04-14T15:23:24.081 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:14:22.850377+0000", 2024-04-14T15:23:24.081 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 541.19149115200003, 2024-04-14T15:23:24.081 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.052484141999999998, 2024-04-14T15:23:24.081 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:23:24.081 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:23:24.081 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:23:24.081 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:23:24.081 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.48388", 2024-04-14T15:23:24.082 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/679904944", 2024-04-14T15:23:24.082 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 2 2024-04-14T15:23:24.082 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.082 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:23:24.082 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.082 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:23:24.082 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:22.850377+0000", 2024-04-14T15:23:24.082 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:23:24.082 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.082 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.082 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:23:24.082 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:22.850377+0000", 2024-04-14T15:23:24.082 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:23:24.083 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.083 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.083 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:23:24.083 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:22.850379+0000", 2024-04-14T15:23:24.083 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6780000000000001e-06 2024-04-14T15:23:24.083 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.083 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.083 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:23:24.083 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:22.850386+0000", 2024-04-14T15:23:24.083 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.55e-06 2024-04-14T15:23:24.083 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.083 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.083 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:23:24.083 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:22.850395+0000", 2024-04-14T15:23:24.083 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.1519999999999998e-06 2024-04-14T15:23:24.083 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.084 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.084 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:23:24.084 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:22.850405+0000", 2024-04-14T15:23:24.084 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0655999999999999e-05 2024-04-14T15:23:24.084 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.084 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.084 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:23:24.084 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:22.850469+0000", 2024-04-14T15:23:24.084 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3652000000000007e-05 2024-04-14T15:23:24.084 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.084 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.084 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op must wait for map", 2024-04-14T15:23:24.084 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:22.850477+0000", 2024-04-14T15:23:24.084 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0979999999999994e-06 2024-04-14T15:23:24.084 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.084 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.085 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:23:24.085 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:22.856127+0000", 2024-04-14T15:23:24.085 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0056495089999999996 2024-04-14T15:23:24.085 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.085 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.085 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:23:24.085 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:22.856136+0000", 2024-04-14T15:23:24.085 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.9849999999999995e-06 2024-04-14T15:23:24.085 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.085 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.085 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:23:24.085 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:22.895778+0000", 2024-04-14T15:23:24.085 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.039642150000000001 2024-04-14T15:23:24.085 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.085 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.085 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:23:24.086 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:22.895888+0000", 2024-04-14T15:23:24.086 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011000100000000001 2024-04-14T15:23:24.086 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.086 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.086 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 2,7", 2024-04-14T15:23:24.086 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:22.896012+0000", 2024-04-14T15:23:24.086 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012449999999999999 2024-04-14T15:23:24.086 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.086 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.086 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:23:24.086 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:22.896739+0000", 2024-04-14T15:23:24.086 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00072682200000000001 2024-04-14T15:23:24.086 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.086 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.086 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:23:24.086 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:22.898060+0000", 2024-04-14T15:23:24.087 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0013207939999999999 2024-04-14T15:23:24.087 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.087 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.087 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:23:24.087 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:22.902770+0000", 2024-04-14T15:23:24.087 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0047102969999999996 2024-04-14T15:23:24.087 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.087 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.087 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:23:24.087 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:22.902823+0000", 2024-04-14T15:23:24.087 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2902999999999998e-05 2024-04-14T15:23:24.087 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.087 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.087 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:23:24.087 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:22.902862+0000", 2024-04-14T15:23:24.087 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8395000000000001e-05 2024-04-14T15:23:24.087 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:23:24.088 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:23:24.088 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:23:24.088 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.088 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.088 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.48481.0:1 227.6s0 227:602f83fe:::foo:head [append 0~8192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-04-14T15:23:24.088 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:14:42.851421+0000", 2024-04-14T15:23:24.088 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 521.19044725900005, 2024-04-14T15:23:24.088 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0204730660000001, 2024-04-14T15:23:24.088 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:23:24.088 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:23:24.088 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:23:24.088 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:23:24.088 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.48481", 2024-04-14T15:23:24.088 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/3131004652", 2024-04-14T15:23:24.089 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 1 2024-04-14T15:23:24.089 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.089 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:23:24.089 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.089 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:23:24.089 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:42.851421+0000", 2024-04-14T15:23:24.089 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:23:24.089 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.089 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.089 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:23:24.089 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:42.851421+0000", 2024-04-14T15:23:24.089 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:23:24.089 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.089 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.090 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:23:24.090 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:42.851423+0000", 2024-04-14T15:23:24.090 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.84e-06 2024-04-14T15:23:24.090 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.090 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.090 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:23:24.090 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:42.851437+0000", 2024-04-14T15:23:24.090 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3652000000000001e-05 2024-04-14T15:23:24.090 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.090 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.090 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:23:24.090 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:42.851447+0000", 2024-04-14T15:23:24.090 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8889999999999997e-06 2024-04-14T15:23:24.091 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.091 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.091 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:23:24.091 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:42.851457+0000", 2024-04-14T15:23:24.091 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0175e-05 2024-04-14T15:23:24.091 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.091 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.091 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:23:24.091 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:42.851514+0000", 2024-04-14T15:23:24.091 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7463999999999997e-05 2024-04-14T15:23:24.091 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.091 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.091 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:23:24.092 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:42.851518+0000", 2024-04-14T15:23:24.092 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.7850000000000002e-06 2024-04-14T15:23:24.092 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.092 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.092 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:23:24.092 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:43.866076+0000", 2024-04-14T15:23:24.092 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.014558361 2024-04-14T15:23:24.092 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.092 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.092 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:23:24.092 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:43.866170+0000", 2024-04-14T15:23:24.092 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.3744000000000002e-05 2024-04-14T15:23:24.092 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.092 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.092 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_started", 2024-04-14T15:23:24.093 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:43.866491+0000", 2024-04-14T15:23:24.093 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00032043099999999998 2024-04-14T15:23:24.093 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.093 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.093 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_committed", 2024-04-14T15:23:24.093 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:43.866747+0000", 2024-04-14T15:23:24.093 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00025597499999999997 2024-04-14T15:23:24.093 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.093 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.093 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:23:24.093 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:43.871842+0000", 2024-04-14T15:23:24.093 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0050949180000000004 2024-04-14T15:23:24.093 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.093 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.093 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:23:24.094 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:14:43.871894+0000", 2024-04-14T15:23:24.094 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2831999999999998e-05 2024-04-14T15:23:24.094 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:23:24.094 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:23:24.094 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:23:24.094 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.094 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.094 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.36780.0:2 249.a 249:57d91c0c:::oid_RemoveTestPP_:head [append 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-04-14T15:23:24.094 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:15:20.877445+0000", 2024-04-14T15:23:24.094 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 483.16442325100002, 2024-04-14T15:23:24.094 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.033481961999999997, 2024-04-14T15:23:24.094 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:23:24.094 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:23:24.094 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:23:24.094 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:23:24.095 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.36780", 2024-04-14T15:23:24.095 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/289305254", 2024-04-14T15:23:24.095 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 2 2024-04-14T15:23:24.095 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.095 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:23:24.095 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.095 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:23:24.095 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:20.877445+0000", 2024-04-14T15:23:24.095 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:23:24.095 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.095 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.095 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:23:24.095 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:20.877445+0000", 2024-04-14T15:23:24.095 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:23:24.095 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.096 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.096 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:23:24.096 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:20.877448+0000", 2024-04-14T15:23:24.096 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3630000000000001e-06 2024-04-14T15:23:24.096 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.096 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.096 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:23:24.096 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:20.877456+0000", 2024-04-14T15:23:24.096 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.7579999999999998e-06 2024-04-14T15:23:24.096 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.096 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.096 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:23:24.096 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:20.877468+0000", 2024-04-14T15:23:24.097 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1815e-05 2024-04-14T15:23:24.097 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.097 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.097 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:23:24.097 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:20.877479+0000", 2024-04-14T15:23:24.097 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0895e-05 2024-04-14T15:23:24.097 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.097 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.097 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:23:24.097 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:20.877541+0000", 2024-04-14T15:23:24.097 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1975000000000002e-05 2024-04-14T15:23:24.097 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.097 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.097 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:23:24.098 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:20.877551+0000", 2024-04-14T15:23:24.098 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8039999999999996e-06 2024-04-14T15:23:24.098 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.098 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.098 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:23:24.098 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:20.886958+0000", 2024-04-14T15:23:24.098 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0094066889999999993 2024-04-14T15:23:24.098 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.098 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.098 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:23:24.098 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:20.887102+0000", 2024-04-14T15:23:24.098 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000144182 2024-04-14T15:23:24.098 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.099 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.099 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 2,4", 2024-04-14T15:23:24.099 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:20.887330+0000", 2024-04-14T15:23:24.099 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000228567 2024-04-14T15:23:24.099 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.099 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.099 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:23:24.099 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:20.893884+0000", 2024-04-14T15:23:24.099 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0065540110000000002 2024-04-14T15:23:24.099 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.099 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.099 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:23:24.099 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:20.894871+0000", 2024-04-14T15:23:24.099 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00098655899999999992 2024-04-14T15:23:24.099 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.100 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.100 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:23:24.100 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:20.910850+0000", 2024-04-14T15:23:24.100 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.015978946000000001 2024-04-14T15:23:24.100 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.100 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.100 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:23:24.100 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:20.910891+0000", 2024-04-14T15:23:24.100 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0988000000000001e-05 2024-04-14T15:23:24.100 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:23:24.100 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:23:24.100 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:23:24.100 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:20.910927+0000", 2024-04-14T15:23:24.101 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6409999999999999e-05 2024-04-14T15:23:24.101 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:23:24.101 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:23:24.101 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:23:24.101 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:23:24.101 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:23:24.101 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-04-14T15:23:24.121 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:23:24.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:24.147 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:24.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:23:24.248 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:23:24.272 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:24.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:24.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:23:24.374 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:23:24.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:24.400 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:24.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:23:24.500 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:23:24.524 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:24.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:24.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:23:24.626 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:23:24.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:24.651 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:24.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:23:24.752 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:23:24.776 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:24.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:24.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:23:24.878 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:23:24.902 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:24.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:24.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:23:24.941 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-14T15:23:25.005 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:23:25.029 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:25.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:25.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:23:25.131 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:23:25.156 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:25.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:25.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:23:25.190 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:23:25.258 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:23:25.276 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:24 smithi090 ceph-mon[41670]: pgmap v981: 216 pgs: 1 active+remapped+backfill_toofull, 32 creating+peering, 183 active+clean; 582 KiB data, 584 MiB used, 625 GiB / 626 GiB avail; 2/603 objects misplaced (0.332%) 2024-04-14T15:23:25.276 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:24 smithi090 ceph-mon[41670]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:23:25.276 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:24 smithi090 ceph-mon[41670]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:23:25.276 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:24 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:23:25.276 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:24 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-116", "mode": "writeback"}]': finished 2024-04-14T15:23:25.276 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:24 smithi090 ceph-mon[41670]: from='client.49630 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-14T15:23:25.276 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:24 smithi090 ceph-mon[41670]: osdmap e668: 8 total, 8 up, 7 in 2024-04-14T15:23:25.276 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:24 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "326.5"}]: dispatch 2024-04-14T15:23:25.277 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:24 smithi090 ceph-mon[41670]: 11.c scrub starts 2024-04-14T15:23:25.277 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:24 smithi090 ceph-mon[41670]: 11.c scrub ok 2024-04-14T15:23:25.277 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:24 smithi090 ceph-mon[30399]: pgmap v981: 216 pgs: 1 active+remapped+backfill_toofull, 32 creating+peering, 183 active+clean; 582 KiB data, 584 MiB used, 625 GiB / 626 GiB avail; 2/603 objects misplaced (0.332%) 2024-04-14T15:23:25.277 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:24 smithi090 ceph-mon[30399]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:23:25.277 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:24 smithi090 ceph-mon[30399]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:23:25.277 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:24 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:23:25.277 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:24 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-116", "mode": "writeback"}]': finished 2024-04-14T15:23:25.277 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:24 smithi090 ceph-mon[30399]: from='client.49630 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-14T15:23:25.277 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:24 smithi090 ceph-mon[30399]: osdmap e668: 8 total, 8 up, 7 in 2024-04-14T15:23:25.277 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:24 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "326.5"}]: dispatch 2024-04-14T15:23:25.277 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:24 smithi090 ceph-mon[30399]: 11.c scrub starts 2024-04-14T15:23:25.277 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:24 smithi090 ceph-mon[30399]: 11.c scrub ok 2024-04-14T15:23:25.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:25.283 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:25.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:23:25.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.281+0000 7f4ab637b700 1 -- 172.21.15.90:0/2239469756 >> v1:172.21.15.90:6789/0 conn(0x7f4ab010a6d0 legacy=0x7f4ab010cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:25.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.281+0000 7f4ab637b700 1 -- 172.21.15.90:0/2239469756 shutdown_connections 2024-04-14T15:23:25.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.281+0000 7f4ab637b700 1 -- 172.21.15.90:0/2239469756 >> 172.21.15.90:0/2239469756 conn(0x7f4ab0076aa0 msgr2=0x7f4ab0076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:23:25.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.282+0000 7f4ab637b700 1 -- 172.21.15.90:0/2239469756 shutdown_connections 2024-04-14T15:23:25.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.282+0000 7f4ab637b700 1 -- 172.21.15.90:0/2239469756 wait complete. 2024-04-14T15:23:25.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.282+0000 7f4ab637b700 1 librados: starting msgr at 2024-04-14T15:23:25.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.282+0000 7f4ab637b700 1 librados: starting objecter 2024-04-14T15:23:25.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.282+0000 7f4ab637b700 1 Processor -- start 2024-04-14T15:23:25.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.282+0000 7f4ab637b700 1 -- start start 2024-04-14T15:23:25.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.282+0000 7f4ab637b700 1 librados: setting wanted keys 2024-04-14T15:23:25.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.282+0000 7f4ab637b700 1 librados: calling monclient init 2024-04-14T15:23:25.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.283+0000 7f4ab637b700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f4ab0101fc0 con 0x7f4ab010e270 2024-04-14T15:23:25.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.283+0000 7f4ab637b700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f4ab01a8e70 con 0x7f4ab01023f0 2024-04-14T15:23:25.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.283+0000 7f4ab637b700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f4ab01aa0b0 con 0x7f4ab010a6d0 2024-04-14T15:23:25.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.283+0000 7f4ab4918700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f4ab010e270 0x7f4ab01a6720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:46276/0 (socket says 172.21.15.90:46276) 2024-04-14T15:23:25.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.283+0000 7f4ab4918700 1 -- 172.21.15.90:0/36940600 learned_addr learned my addr 172.21.15.90:0/36940600 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:23:25.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.283+0000 7f4aad7fa700 10 client.?.objecter ms_handle_connect 0x7f4ab010a6d0 2024-04-14T15:23:25.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.283+0000 7f4aad7fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:23:25.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.283+0000 7f4aad7fa700 10 client.?.objecter ms_handle_connect 0x7f4ab010e270 2024-04-14T15:23:25.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.283+0000 7f4aad7fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:23:25.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.283+0000 7f4aad7fa700 10 client.?.objecter ms_handle_connect 0x7f4ab01023f0 2024-04-14T15:23:25.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.283+0000 7f4aad7fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:23:25.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.283+0000 7f4aad7fa700 1 -- 172.21.15.90:0/36940600 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3417318676 0 0) 0x7f4ab01aa0b0 con 0x7f4ab010a6d0 2024-04-14T15:23:25.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.283+0000 7f4aad7fa700 1 -- 172.21.15.90:0/36940600 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a980035b0 con 0x7f4ab010a6d0 2024-04-14T15:23:25.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.283+0000 7f4aad7fa700 1 -- 172.21.15.90:0/36940600 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3130184810 0 0) 0x7f4ab0101fc0 con 0x7f4ab010e270 2024-04-14T15:23:25.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.283+0000 7f4aad7fa700 1 -- 172.21.15.90:0/36940600 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ab01aa0b0 con 0x7f4ab010e270 2024-04-14T15:23:25.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.283+0000 7f4aad7fa700 1 -- 172.21.15.90:0/36940600 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 551016197 0 0) 0x7f4ab01a8e70 con 0x7f4ab01023f0 2024-04-14T15:23:25.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.283+0000 7f4aad7fa700 1 -- 172.21.15.90:0/36940600 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ab0101fc0 con 0x7f4ab01023f0 2024-04-14T15:23:25.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.283+0000 7f4aad7fa700 1 -- 172.21.15.90:0/36940600 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1750990894 0 0) 0x7f4ab0101fc0 con 0x7f4ab01023f0 2024-04-14T15:23:25.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.283+0000 7f4aad7fa700 1 -- 172.21.15.90:0/36940600 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4ab01a8e70 con 0x7f4ab01023f0 2024-04-14T15:23:25.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.283+0000 7f4aad7fa700 1 -- 172.21.15.90:0/36940600 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f4aa00018b0 con 0x7f4ab01023f0 2024-04-14T15:23:25.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.283+0000 7f4aad7fa700 1 -- 172.21.15.90:0/36940600 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 985055245 0 0) 0x7f4a980035b0 con 0x7f4ab010a6d0 2024-04-14T15:23:25.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.283+0000 7f4aad7fa700 1 -- 172.21.15.90:0/36940600 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4ab0101fc0 con 0x7f4ab010a6d0 2024-04-14T15:23:25.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.283+0000 7f4aad7fa700 1 -- 172.21.15.90:0/36940600 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2728217549 0 0) 0x7f4ab01aa0b0 con 0x7f4ab010e270 2024-04-14T15:23:25.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.283+0000 7f4aad7fa700 1 -- 172.21.15.90:0/36940600 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4a980035b0 con 0x7f4ab010e270 2024-04-14T15:23:25.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.283+0000 7f4aad7fa700 1 -- 172.21.15.90:0/36940600 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f4aa40015d0 con 0x7f4ab010a6d0 2024-04-14T15:23:25.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.283+0000 7f4aad7fa700 1 -- 172.21.15.90:0/36940600 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f4aa8001bf0 con 0x7f4ab010e270 2024-04-14T15:23:25.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.283+0000 7f4aad7fa700 1 -- 172.21.15.90:0/36940600 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 172937279 0 0) 0x7f4ab01a8e70 con 0x7f4ab01023f0 2024-04-14T15:23:25.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.283+0000 7f4aad7fa700 1 -- 172.21.15.90:0/36940600 >> v1:172.21.15.90:6790/0 conn(0x7f4ab010a6d0 legacy=0x7f4ab0106850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:25.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.283+0000 7f4aad7fa700 1 -- 172.21.15.90:0/36940600 >> v1:172.21.15.90:6789/0 conn(0x7f4ab010e270 legacy=0x7f4ab01a6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:25.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.283+0000 7f4aad7fa700 1 -- 172.21.15.90:0/36940600 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ab01ab2f0 con 0x7f4ab01023f0 2024-04-14T15:23:25.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.283+0000 7f4ab637b700 1 -- 172.21.15.90:0/36940600 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4ab01a7ea0 con 0x7f4ab01023f0 2024-04-14T15:23:25.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.284+0000 7f4ab637b700 10 client.49636.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:23:25.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.284+0000 7f4ab637b700 1 -- 172.21.15.90:0/36940600 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f4ab01a8360 con 0x7f4ab01023f0 2024-04-14T15:23:25.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.284+0000 7f4ab637b700 1 librados: init done 2024-04-14T15:23:25.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.284+0000 7f4aad7fa700 1 -- 172.21.15.90:0/36940600 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f4aa0003b40 con 0x7f4ab01023f0 2024-04-14T15:23:25.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.284+0000 7f4aad7fa700 1 -- 172.21.15.90:0/36940600 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f4aa0004a10 con 0x7f4ab01023f0 2024-04-14T15:23:25.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.284+0000 7f4ab637b700 1 -- 172.21.15.90:0/36940600 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f4a9c005100 con 0x7f4ab01023f0 2024-04-14T15:23:25.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.285+0000 7f4aad7fa700 1 -- 172.21.15.90:0/36940600 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f4aa0003680 con 0x7f4ab01023f0 2024-04-14T15:23:25.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.285+0000 7f4aad7fa700 1 -- 172.21.15.90:0/36940600 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(668..668 src has 467..668) ==== 8673+0+0 (unknown 3380578365 0 0) 0x7f4aa0089790 con 0x7f4ab01023f0 2024-04-14T15:23:25.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.285+0000 7f4aad7fa700 10 client.49636.objecter ms_dispatch 0x7f4ab0001550 osd_map(668..668 src has 467..668) 2024-04-14T15:23:25.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.285+0000 7f4aad7fa700 3 client.49636.objecter handle_osd_map got epochs [668,668] > 0 2024-04-14T15:23:25.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.285+0000 7f4aad7fa700 3 client.49636.objecter handle_osd_map decoding full epoch 668 2024-04-14T15:23:25.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.286+0000 7f4aad7fa700 20 client.49636.objecter dump_active .. 0 homeless 2024-04-14T15:23:25.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.286+0000 7f4aad7fa700 1 -- 172.21.15.90:0/36940600 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v668) ==== 54+0+16 (unknown 1485583410 0 2420000192) 0x7f4aa0089b90 con 0x7f4ab01023f0 2024-04-14T15:23:25.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.286+0000 7f4aad7fa700 10 client.49636.objecter ms_handle_connect 0x7f4a980714b0 2024-04-14T15:23:25.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.286+0000 7f4ab637b700 10 client.49636.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:23:25.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.286+0000 7f4ab637b700 15 client.49636.objecter _session_command_op_assign -1 1 2024-04-14T15:23:25.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.286+0000 7f4ab637b700 20 client.49636.objecter _get_session s=0x7f4a9c0053e0 osd=0 3 2024-04-14T15:23:25.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.286+0000 7f4ab637b700 20 client.49636.objecter put_session s=0x7f4a9c0053e0 osd=0 3 2024-04-14T15:23:25.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.286+0000 7f4ab637b700 20 client.49636.objecter _get_session s=0x7f4a9c0053e0 osd=0 3 2024-04-14T15:23:25.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.286+0000 7f4ab637b700 15 client.49636.objecter _session_command_op_remove -1 1 2024-04-14T15:23:25.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.286+0000 7f4ab637b700 20 client.49636.objecter get_session s=0x7f4a9c0053e0 osd=0 3 2024-04-14T15:23:25.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.286+0000 7f4ab637b700 15 client.49636.objecter _session_command_op_assign 0 1 2024-04-14T15:23:25.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.286+0000 7f4ab637b700 20 client.49636.objecter put_session s=0x7f4a9c0053e0 osd=0 4 2024-04-14T15:23:25.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.286+0000 7f4ab637b700 10 client.49636.objecter _send_command 1 2024-04-14T15:23:25.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.286+0000 7f4ab637b700 1 -- 172.21.15.90:0/36940600 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4a9c004b50 con 0x7f4a9c004730 2024-04-14T15:23:25.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.286+0000 7f4aad7fa700 10 client.49636.objecter ms_handle_connect 0x7f4a9c004730 2024-04-14T15:23:25.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.287+0000 7f4aad7fa700 1 -- 172.21.15.90:0/36940600 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f4a9c004b50 con 0x7f4a9c004730 2024-04-14T15:23:25.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.287+0000 7f4aad7fa700 10 client.49636.objecter ms_dispatch 0x7f4ab0001550 command_reply(tid 1: 0 ) 2024-04-14T15:23:25.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.287+0000 7f4aad7fa700 10 client.49636.objecter _finish_command 1 = system:0 2024-04-14T15:23:25.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.287+0000 7f4aad7fa700 20 client.49636.objecter put_session s=0x7f4a9c0053e0 osd=0 4 2024-04-14T15:23:25.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.287+0000 7f4aad7fa700 15 client.49636.objecter _session_command_op_remove 0 1 2024-04-14T15:23:25.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.303+0000 7f4ab637b700 10 client.49636.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:23:25.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.303+0000 7f4ab637b700 15 client.49636.objecter _session_command_op_assign -1 2 2024-04-14T15:23:25.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.303+0000 7f4ab637b700 20 client.49636.objecter _get_session s=0x7f4a9c0053e0 osd=0 3 2024-04-14T15:23:25.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.303+0000 7f4ab637b700 20 client.49636.objecter put_session s=0x7f4a9c0053e0 osd=0 3 2024-04-14T15:23:25.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.303+0000 7f4ab637b700 20 client.49636.objecter _get_session s=0x7f4a9c0053e0 osd=0 3 2024-04-14T15:23:25.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.303+0000 7f4ab637b700 15 client.49636.objecter _session_command_op_remove -1 2 2024-04-14T15:23:25.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.303+0000 7f4ab637b700 20 client.49636.objecter get_session s=0x7f4a9c0053e0 osd=0 3 2024-04-14T15:23:25.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.303+0000 7f4ab637b700 15 client.49636.objecter _session_command_op_assign 0 2 2024-04-14T15:23:25.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.303+0000 7f4ab637b700 20 client.49636.objecter put_session s=0x7f4a9c0053e0 osd=0 4 2024-04-14T15:23:25.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.303+0000 7f4ab637b700 10 client.49636.objecter _send_command 2 2024-04-14T15:23:25.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.303+0000 7f4ab637b700 1 -- 172.21.15.90:0/36940600 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4a9c00be00 con 0x7f4a9c004730 2024-04-14T15:23:25.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.305+0000 7f4aad7fa700 1 -- 172.21.15.90:0/36940600 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f4a9c00be00 con 0x7f4a9c004730 2024-04-14T15:23:25.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.305+0000 7f4aad7fa700 10 client.49636.objecter ms_dispatch 0x7f4ab0001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:23:25.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.305+0000 7f4aad7fa700 10 client.49636.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:23:25.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.305+0000 7f4aad7fa700 20 client.49636.objecter put_session s=0x7f4a9c0053e0 osd=0 4 2024-04-14T15:23:25.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.305+0000 7f4aad7fa700 15 client.49636.objecter _session_command_op_remove 0 2 2024-04-14T15:23:25.306 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:23:25.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.305+0000 7f4ab637b700 10 client.49636.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:23:25.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.305+0000 7f4ab637b700 15 client.49636.objecter _session_command_op_assign -1 3 2024-04-14T15:23:25.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.305+0000 7f4ab637b700 20 client.49636.objecter _get_session s=0x7f4a9c004110 osd=1 3 2024-04-14T15:23:25.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.305+0000 7f4ab637b700 20 client.49636.objecter put_session s=0x7f4a9c004110 osd=1 3 2024-04-14T15:23:25.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.305+0000 7f4ab637b700 20 client.49636.objecter _get_session s=0x7f4a9c004110 osd=1 3 2024-04-14T15:23:25.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.305+0000 7f4ab637b700 15 client.49636.objecter _session_command_op_remove -1 3 2024-04-14T15:23:25.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.305+0000 7f4ab637b700 20 client.49636.objecter get_session s=0x7f4a9c004110 osd=1 3 2024-04-14T15:23:25.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.305+0000 7f4ab637b700 15 client.49636.objecter _session_command_op_assign 1 3 2024-04-14T15:23:25.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.305+0000 7f4ab637b700 20 client.49636.objecter put_session s=0x7f4a9c004110 osd=1 4 2024-04-14T15:23:25.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.305+0000 7f4ab637b700 10 client.49636.objecter _send_command 3 2024-04-14T15:23:25.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.305+0000 7f4ab637b700 1 -- 172.21.15.90:0/36940600 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f4a9c002180 con 0x7f4a9c004280 2024-04-14T15:23:25.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.306+0000 7f4aad7fa700 10 client.49636.objecter ms_handle_connect 0x7f4a9c004280 2024-04-14T15:23:25.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.306+0000 7f4aad7fa700 1 -- 172.21.15.90:0/36940600 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f4a9c002180 con 0x7f4a9c004280 2024-04-14T15:23:25.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.306+0000 7f4aad7fa700 10 client.49636.objecter ms_dispatch 0x7f4ab0001550 command_reply(tid 3: 0 ) 2024-04-14T15:23:25.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.306+0000 7f4aad7fa700 10 client.49636.objecter _finish_command 3 = system:0 2024-04-14T15:23:25.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.306+0000 7f4aad7fa700 20 client.49636.objecter put_session s=0x7f4a9c004110 osd=1 4 2024-04-14T15:23:25.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.306+0000 7f4aad7fa700 15 client.49636.objecter _session_command_op_remove 1 3 2024-04-14T15:23:25.312 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:24 smithi092 ceph-mon[33966]: pgmap v981: 216 pgs: 1 active+remapped+backfill_toofull, 32 creating+peering, 183 active+clean; 582 KiB data, 584 MiB used, 625 GiB / 626 GiB avail; 2/603 objects misplaced (0.332%) 2024-04-14T15:23:25.312 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:24 smithi092 ceph-mon[33966]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:23:25.312 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:24 smithi092 ceph-mon[33966]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:23:25.312 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:24 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:23:25.313 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:24 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-116", "mode": "writeback"}]': finished 2024-04-14T15:23:25.313 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:24 smithi092 ceph-mon[33966]: from='client.49630 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-14T15:23:25.313 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:24 smithi092 ceph-mon[33966]: osdmap e668: 8 total, 8 up, 7 in 2024-04-14T15:23:25.313 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:24 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "326.5"}]: dispatch 2024-04-14T15:23:25.313 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:24 smithi092 ceph-mon[33966]: 11.c scrub starts 2024-04-14T15:23:25.313 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:24 smithi092 ceph-mon[33966]: 11.c scrub ok 2024-04-14T15:23:25.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.323+0000 7f4ab637b700 10 client.49636.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:23:25.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.323+0000 7f4ab637b700 15 client.49636.objecter _session_command_op_assign -1 4 2024-04-14T15:23:25.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.323+0000 7f4ab637b700 20 client.49636.objecter _get_session s=0x7f4a9c004110 osd=1 3 2024-04-14T15:23:25.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.323+0000 7f4ab637b700 20 client.49636.objecter put_session s=0x7f4a9c004110 osd=1 3 2024-04-14T15:23:25.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.323+0000 7f4ab637b700 20 client.49636.objecter _get_session s=0x7f4a9c004110 osd=1 3 2024-04-14T15:23:25.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.323+0000 7f4ab637b700 15 client.49636.objecter _session_command_op_remove -1 4 2024-04-14T15:23:25.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.323+0000 7f4ab637b700 20 client.49636.objecter get_session s=0x7f4a9c004110 osd=1 3 2024-04-14T15:23:25.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.323+0000 7f4ab637b700 15 client.49636.objecter _session_command_op_assign 1 4 2024-04-14T15:23:25.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.323+0000 7f4ab637b700 20 client.49636.objecter put_session s=0x7f4a9c004110 osd=1 4 2024-04-14T15:23:25.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.323+0000 7f4ab637b700 10 client.49636.objecter _send_command 4 2024-04-14T15:23:25.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.323+0000 7f4ab637b700 1 -- 172.21.15.90:0/36940600 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4a9c001600 con 0x7f4a9c004280 2024-04-14T15:23:25.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.324+0000 7f4aad7fa700 1 -- 172.21.15.90:0/36940600 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f4a9c001600 con 0x7f4a9c004280 2024-04-14T15:23:25.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.324+0000 7f4aad7fa700 10 client.49636.objecter ms_dispatch 0x7f4ab0001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:23:25.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.324+0000 7f4aad7fa700 10 client.49636.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:23:25.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.324+0000 7f4aad7fa700 20 client.49636.objecter put_session s=0x7f4a9c004110 osd=1 4 2024-04-14T15:23:25.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.324+0000 7f4aad7fa700 15 client.49636.objecter _session_command_op_remove 1 4 2024-04-14T15:23:25.325 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:23:25.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.324+0000 7f4ab637b700 10 client.49636.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:23:25.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.324+0000 7f4ab637b700 15 client.49636.objecter _session_command_op_assign -1 5 2024-04-14T15:23:25.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.324+0000 7f4ab637b700 20 client.49636.objecter _get_session s=0x7f4a9c001a00 osd=2 3 2024-04-14T15:23:25.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.324+0000 7f4ab637b700 20 client.49636.objecter put_session s=0x7f4a9c001a00 osd=2 3 2024-04-14T15:23:25.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.324+0000 7f4ab637b700 20 client.49636.objecter _get_session s=0x7f4a9c001a00 osd=2 3 2024-04-14T15:23:25.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.324+0000 7f4ab637b700 15 client.49636.objecter _session_command_op_remove -1 5 2024-04-14T15:23:25.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.324+0000 7f4ab637b700 20 client.49636.objecter get_session s=0x7f4a9c001a00 osd=2 3 2024-04-14T15:23:25.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.324+0000 7f4ab637b700 15 client.49636.objecter _session_command_op_assign 2 5 2024-04-14T15:23:25.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.324+0000 7f4ab637b700 20 client.49636.objecter put_session s=0x7f4a9c001a00 osd=2 4 2024-04-14T15:23:25.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.324+0000 7f4ab637b700 10 client.49636.objecter _send_command 5 2024-04-14T15:23:25.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.324+0000 7f4ab637b700 1 -- 172.21.15.90:0/36940600 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f4a9c015330 con 0x7f4a9c00cc90 2024-04-14T15:23:25.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.325+0000 7f4aad7fa700 10 client.49636.objecter ms_handle_connect 0x7f4a9c00cc90 2024-04-14T15:23:25.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.325+0000 7f4aad7fa700 1 -- 172.21.15.90:0/36940600 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f4a9c015330 con 0x7f4a9c00cc90 2024-04-14T15:23:25.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.325+0000 7f4aad7fa700 10 client.49636.objecter ms_dispatch 0x7f4ab0001550 command_reply(tid 5: 0 ) 2024-04-14T15:23:25.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.325+0000 7f4aad7fa700 10 client.49636.objecter _finish_command 5 = system:0 2024-04-14T15:23:25.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.325+0000 7f4aad7fa700 20 client.49636.objecter put_session s=0x7f4a9c001a00 osd=2 4 2024-04-14T15:23:25.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.325+0000 7f4aad7fa700 15 client.49636.objecter _session_command_op_remove 2 5 2024-04-14T15:23:25.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.343+0000 7f4ab637b700 10 client.49636.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:23:25.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.343+0000 7f4ab637b700 15 client.49636.objecter _session_command_op_assign -1 6 2024-04-14T15:23:25.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.343+0000 7f4ab637b700 20 client.49636.objecter _get_session s=0x7f4a9c001a00 osd=2 3 2024-04-14T15:23:25.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.343+0000 7f4ab637b700 20 client.49636.objecter put_session s=0x7f4a9c001a00 osd=2 3 2024-04-14T15:23:25.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.343+0000 7f4ab637b700 20 client.49636.objecter _get_session s=0x7f4a9c001a00 osd=2 3 2024-04-14T15:23:25.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.343+0000 7f4ab637b700 15 client.49636.objecter _session_command_op_remove -1 6 2024-04-14T15:23:25.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.343+0000 7f4ab637b700 20 client.49636.objecter get_session s=0x7f4a9c001a00 osd=2 3 2024-04-14T15:23:25.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.343+0000 7f4ab637b700 15 client.49636.objecter _session_command_op_assign 2 6 2024-04-14T15:23:25.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.343+0000 7f4ab637b700 20 client.49636.objecter put_session s=0x7f4a9c001a00 osd=2 4 2024-04-14T15:23:25.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.343+0000 7f4ab637b700 10 client.49636.objecter _send_command 6 2024-04-14T15:23:25.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.343+0000 7f4ab637b700 1 -- 172.21.15.90:0/36940600 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4a9c00c320 con 0x7f4a9c00cc90 2024-04-14T15:23:25.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.345+0000 7f4aad7fa700 1 -- 172.21.15.90:0/36940600 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f4a9c00c320 con 0x7f4a9c00cc90 2024-04-14T15:23:25.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.345+0000 7f4aad7fa700 10 client.49636.objecter ms_dispatch 0x7f4ab0001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:23:25.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.345+0000 7f4aad7fa700 10 client.49636.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:23:25.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.345+0000 7f4aad7fa700 20 client.49636.objecter put_session s=0x7f4a9c001a00 osd=2 4 2024-04-14T15:23:25.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.345+0000 7f4aad7fa700 15 client.49636.objecter _session_command_op_remove 2 6 2024-04-14T15:23:25.346 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:23:25.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.345+0000 7f4ab637b700 10 client.49636.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:23:25.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.345+0000 7f4ab637b700 15 client.49636.objecter _session_command_op_assign -1 7 2024-04-14T15:23:25.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.345+0000 7f4ab637b700 20 client.49636.objecter _get_session s=0x7f4a9c00c580 osd=3 3 2024-04-14T15:23:25.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.345+0000 7f4ab637b700 20 client.49636.objecter put_session s=0x7f4a9c00c580 osd=3 3 2024-04-14T15:23:25.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.345+0000 7f4ab637b700 20 client.49636.objecter _get_session s=0x7f4a9c00c580 osd=3 3 2024-04-14T15:23:25.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.345+0000 7f4ab637b700 15 client.49636.objecter _session_command_op_remove -1 7 2024-04-14T15:23:25.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.345+0000 7f4ab637b700 20 client.49636.objecter get_session s=0x7f4a9c00c580 osd=3 3 2024-04-14T15:23:25.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.345+0000 7f4ab637b700 15 client.49636.objecter _session_command_op_assign 3 7 2024-04-14T15:23:25.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.345+0000 7f4ab637b700 20 client.49636.objecter put_session s=0x7f4a9c00c580 osd=3 4 2024-04-14T15:23:25.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.345+0000 7f4ab637b700 10 client.49636.objecter _send_command 7 2024-04-14T15:23:25.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.345+0000 7f4ab637b700 1 -- 172.21.15.90:0/36940600 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f4a9c016080 con 0x7f4a9c0154e0 2024-04-14T15:23:25.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.345+0000 7f4aad7fa700 10 client.49636.objecter ms_handle_connect 0x7f4a9c0154e0 2024-04-14T15:23:25.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.346+0000 7f4aad7fa700 1 -- 172.21.15.90:0/36940600 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f4a9c016080 con 0x7f4a9c0154e0 2024-04-14T15:23:25.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.346+0000 7f4aad7fa700 10 client.49636.objecter ms_dispatch 0x7f4ab0001550 command_reply(tid 7: 0 ) 2024-04-14T15:23:25.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.346+0000 7f4aad7fa700 10 client.49636.objecter _finish_command 7 = system:0 2024-04-14T15:23:25.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.346+0000 7f4aad7fa700 20 client.49636.objecter put_session s=0x7f4a9c00c580 osd=3 4 2024-04-14T15:23:25.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.346+0000 7f4aad7fa700 15 client.49636.objecter _session_command_op_remove 3 7 2024-04-14T15:23:25.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.363+0000 7f4ab637b700 10 client.49636.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:23:25.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.363+0000 7f4ab637b700 15 client.49636.objecter _session_command_op_assign -1 8 2024-04-14T15:23:25.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.363+0000 7f4ab637b700 20 client.49636.objecter _get_session s=0x7f4a9c00c580 osd=3 3 2024-04-14T15:23:25.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.363+0000 7f4ab637b700 20 client.49636.objecter put_session s=0x7f4a9c00c580 osd=3 3 2024-04-14T15:23:25.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.363+0000 7f4ab637b700 20 client.49636.objecter _get_session s=0x7f4a9c00c580 osd=3 3 2024-04-14T15:23:25.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.363+0000 7f4ab637b700 15 client.49636.objecter _session_command_op_remove -1 8 2024-04-14T15:23:25.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.363+0000 7f4ab637b700 20 client.49636.objecter get_session s=0x7f4a9c00c580 osd=3 3 2024-04-14T15:23:25.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.363+0000 7f4ab637b700 15 client.49636.objecter _session_command_op_assign 3 8 2024-04-14T15:23:25.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.363+0000 7f4ab637b700 20 client.49636.objecter put_session s=0x7f4a9c00c580 osd=3 4 2024-04-14T15:23:25.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.363+0000 7f4ab637b700 10 client.49636.objecter _send_command 8 2024-04-14T15:23:25.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.363+0000 7f4ab637b700 1 -- 172.21.15.90:0/36940600 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4a9c016a30 con 0x7f4a9c0154e0 2024-04-14T15:23:25.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.365+0000 7f4aad7fa700 1 -- 172.21.15.90:0/36940600 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f4a9c016a30 con 0x7f4a9c0154e0 2024-04-14T15:23:25.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.365+0000 7f4aad7fa700 10 client.49636.objecter ms_dispatch 0x7f4ab0001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:23:25.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.365+0000 7f4aad7fa700 10 client.49636.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:23:25.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.365+0000 7f4aad7fa700 20 client.49636.objecter put_session s=0x7f4a9c00c580 osd=3 4 2024-04-14T15:23:25.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.365+0000 7f4aad7fa700 15 client.49636.objecter _session_command_op_remove 3 8 2024-04-14T15:23:25.365 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:23:25.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.365+0000 7f4ab637b700 10 client.49636.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:23:25.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.365+0000 7f4ab637b700 15 client.49636.objecter _session_command_op_assign -1 9 2024-04-14T15:23:25.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.365+0000 7f4ab637b700 20 client.49636.objecter _get_session s=0x7f4a9c017bb0 osd=4 3 2024-04-14T15:23:25.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.365+0000 7f4ab637b700 20 client.49636.objecter put_session s=0x7f4a9c017bb0 osd=4 3 2024-04-14T15:23:25.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.365+0000 7f4ab637b700 20 client.49636.objecter _get_session s=0x7f4a9c017bb0 osd=4 3 2024-04-14T15:23:25.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.365+0000 7f4ab637b700 15 client.49636.objecter _session_command_op_remove -1 9 2024-04-14T15:23:25.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.365+0000 7f4ab637b700 20 client.49636.objecter get_session s=0x7f4a9c017bb0 osd=4 3 2024-04-14T15:23:25.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.365+0000 7f4ab637b700 15 client.49636.objecter _session_command_op_assign 4 9 2024-04-14T15:23:25.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.365+0000 7f4ab637b700 20 client.49636.objecter put_session s=0x7f4a9c017bb0 osd=4 4 2024-04-14T15:23:25.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.365+0000 7f4ab637b700 10 client.49636.objecter _send_command 9 2024-04-14T15:23:25.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.365+0000 7f4ab637b700 1 -- 172.21.15.90:0/36940600 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f4a9c01eb30 con 0x7f4a9c01aef0 2024-04-14T15:23:25.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.365+0000 7f4aad7fa700 10 client.49636.objecter ms_handle_connect 0x7f4a9c01aef0 2024-04-14T15:23:25.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.366+0000 7f4aad7fa700 1 -- 172.21.15.90:0/36940600 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f4a9c01eb30 con 0x7f4a9c01aef0 2024-04-14T15:23:25.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.366+0000 7f4aad7fa700 10 client.49636.objecter ms_dispatch 0x7f4ab0001550 command_reply(tid 9: 0 ) 2024-04-14T15:23:25.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.366+0000 7f4aad7fa700 10 client.49636.objecter _finish_command 9 = system:0 2024-04-14T15:23:25.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.366+0000 7f4aad7fa700 20 client.49636.objecter put_session s=0x7f4a9c017bb0 osd=4 4 2024-04-14T15:23:25.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.366+0000 7f4aad7fa700 15 client.49636.objecter _session_command_op_remove 4 9 2024-04-14T15:23:25.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.382+0000 7f4ab637b700 10 client.49636.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:23:25.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.382+0000 7f4ab637b700 15 client.49636.objecter _session_command_op_assign -1 10 2024-04-14T15:23:25.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.382+0000 7f4ab637b700 20 client.49636.objecter _get_session s=0x7f4a9c017bb0 osd=4 3 2024-04-14T15:23:25.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.382+0000 7f4ab637b700 20 client.49636.objecter put_session s=0x7f4a9c017bb0 osd=4 3 2024-04-14T15:23:25.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.382+0000 7f4ab637b700 20 client.49636.objecter _get_session s=0x7f4a9c017bb0 osd=4 3 2024-04-14T15:23:25.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.382+0000 7f4ab637b700 15 client.49636.objecter _session_command_op_remove -1 10 2024-04-14T15:23:25.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.382+0000 7f4ab637b700 20 client.49636.objecter get_session s=0x7f4a9c017bb0 osd=4 3 2024-04-14T15:23:25.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.382+0000 7f4ab637b700 15 client.49636.objecter _session_command_op_assign 4 10 2024-04-14T15:23:25.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.382+0000 7f4ab637b700 20 client.49636.objecter put_session s=0x7f4a9c017bb0 osd=4 4 2024-04-14T15:23:25.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.382+0000 7f4ab637b700 10 client.49636.objecter _send_command 10 2024-04-14T15:23:25.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.382+0000 7f4ab637b700 1 -- 172.21.15.90:0/36940600 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4a9c01eaf0 con 0x7f4a9c01aef0 2024-04-14T15:23:25.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.384+0000 7f4aad7fa700 1 -- 172.21.15.90:0/36940600 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f4aa001de50 con 0x7f4a9c01aef0 2024-04-14T15:23:25.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.384+0000 7f4aad7fa700 10 client.49636.objecter ms_dispatch 0x7f4ab0001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:23:25.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.384+0000 7f4aad7fa700 10 client.49636.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:23:25.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.384+0000 7f4aad7fa700 20 client.49636.objecter put_session s=0x7f4a9c017bb0 osd=4 4 2024-04-14T15:23:25.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.384+0000 7f4aad7fa700 15 client.49636.objecter _session_command_op_remove 4 10 2024-04-14T15:23:25.384 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:23:25.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.384+0000 7f4ab637b700 10 client.49636.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:23:25.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.384+0000 7f4ab637b700 15 client.49636.objecter _session_command_op_assign -1 11 2024-04-14T15:23:25.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.384+0000 7f4ab637b700 20 client.49636.objecter _get_session s=0x7f4a9c010b70 osd=5 3 2024-04-14T15:23:25.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.384+0000 7f4ab637b700 20 client.49636.objecter put_session s=0x7f4a9c010b70 osd=5 3 2024-04-14T15:23:25.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.384+0000 7f4ab637b700 20 client.49636.objecter _get_session s=0x7f4a9c010b70 osd=5 3 2024-04-14T15:23:25.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.384+0000 7f4ab637b700 15 client.49636.objecter _session_command_op_remove -1 11 2024-04-14T15:23:25.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.384+0000 7f4ab637b700 20 client.49636.objecter get_session s=0x7f4a9c010b70 osd=5 3 2024-04-14T15:23:25.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.384+0000 7f4ab637b700 15 client.49636.objecter _session_command_op_assign 5 11 2024-04-14T15:23:25.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.384+0000 7f4ab637b700 20 client.49636.objecter put_session s=0x7f4a9c010b70 osd=5 4 2024-04-14T15:23:25.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.384+0000 7f4ab637b700 10 client.49636.objecter _send_command 11 2024-04-14T15:23:25.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.384+0000 7f4ab637b700 1 -- 172.21.15.90:0/36940600 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f4a9c017930 con 0x7f4a9c01f720 2024-04-14T15:23:25.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.385+0000 7f4aad7fa700 10 client.49636.objecter ms_handle_connect 0x7f4a9c01f720 2024-04-14T15:23:25.386 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:25.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.385+0000 7f4aad7fa700 1 -- 172.21.15.90:0/36940600 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f4a9c017930 con 0x7f4a9c01f720 2024-04-14T15:23:25.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.385+0000 7f4aad7fa700 10 client.49636.objecter ms_dispatch 0x7f4ab0001550 command_reply(tid 11: 0 ) 2024-04-14T15:23:25.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.385+0000 7f4aad7fa700 10 client.49636.objecter _finish_command 11 = system:0 2024-04-14T15:23:25.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.385+0000 7f4aad7fa700 20 client.49636.objecter put_session s=0x7f4a9c010b70 osd=5 4 2024-04-14T15:23:25.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.385+0000 7f4aad7fa700 15 client.49636.objecter _session_command_op_remove 5 11 2024-04-14T15:23:25.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.402+0000 7f4ab637b700 10 client.49636.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:23:25.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.402+0000 7f4ab637b700 15 client.49636.objecter _session_command_op_assign -1 12 2024-04-14T15:23:25.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.402+0000 7f4ab637b700 20 client.49636.objecter _get_session s=0x7f4a9c010b70 osd=5 3 2024-04-14T15:23:25.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.402+0000 7f4ab637b700 20 client.49636.objecter put_session s=0x7f4a9c010b70 osd=5 3 2024-04-14T15:23:25.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.402+0000 7f4ab637b700 20 client.49636.objecter _get_session s=0x7f4a9c010b70 osd=5 3 2024-04-14T15:23:25.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.402+0000 7f4ab637b700 15 client.49636.objecter _session_command_op_remove -1 12 2024-04-14T15:23:25.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.402+0000 7f4ab637b700 20 client.49636.objecter get_session s=0x7f4a9c010b70 osd=5 3 2024-04-14T15:23:25.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.402+0000 7f4ab637b700 15 client.49636.objecter _session_command_op_assign 5 12 2024-04-14T15:23:25.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.402+0000 7f4ab637b700 20 client.49636.objecter put_session s=0x7f4a9c010b70 osd=5 4 2024-04-14T15:23:25.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.402+0000 7f4ab637b700 10 client.49636.objecter _send_command 12 2024-04-14T15:23:25.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.402+0000 7f4ab637b700 1 -- 172.21.15.90:0/36940600 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4a9c017930 con 0x7f4a9c01f720 2024-04-14T15:23:25.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.403+0000 7f4aad7fa700 1 -- 172.21.15.90:0/36940600 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f4a9c017930 con 0x7f4a9c01f720 2024-04-14T15:23:25.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.403+0000 7f4aad7fa700 10 client.49636.objecter ms_dispatch 0x7f4ab0001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:23:25.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.403+0000 7f4aad7fa700 10 client.49636.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:23:25.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.403+0000 7f4aad7fa700 20 client.49636.objecter put_session s=0x7f4a9c010b70 osd=5 4 2024-04-14T15:23:25.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.403+0000 7f4aad7fa700 15 client.49636.objecter _session_command_op_remove 5 12 2024-04-14T15:23:25.404 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:23:25.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.404+0000 7f4ab637b700 10 client.49636.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:23:25.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.404+0000 7f4ab637b700 15 client.49636.objecter _session_command_op_assign -1 13 2024-04-14T15:23:25.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.404+0000 7f4ab637b700 20 client.49636.objecter _get_session s=0x7f4a9c017930 osd=6 3 2024-04-14T15:23:25.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.404+0000 7f4ab637b700 20 client.49636.objecter put_session s=0x7f4a9c017930 osd=6 3 2024-04-14T15:23:25.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.404+0000 7f4ab637b700 20 client.49636.objecter _get_session s=0x7f4a9c017930 osd=6 3 2024-04-14T15:23:25.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.404+0000 7f4ab637b700 15 client.49636.objecter _session_command_op_remove -1 13 2024-04-14T15:23:25.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.404+0000 7f4ab637b700 20 client.49636.objecter get_session s=0x7f4a9c017930 osd=6 3 2024-04-14T15:23:25.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.404+0000 7f4ab637b700 15 client.49636.objecter _session_command_op_assign 6 13 2024-04-14T15:23:25.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.404+0000 7f4ab637b700 20 client.49636.objecter put_session s=0x7f4a9c017930 osd=6 4 2024-04-14T15:23:25.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.404+0000 7f4ab637b700 10 client.49636.objecter _send_command 13 2024-04-14T15:23:25.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.404+0000 7f4ab637b700 1 -- 172.21.15.90:0/36940600 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f4a9c027690 con 0x7f4a9c023a10 2024-04-14T15:23:25.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.404+0000 7f4aad7fa700 10 client.49636.objecter ms_handle_connect 0x7f4a9c023a10 2024-04-14T15:23:25.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.405+0000 7f4aad7fa700 1 -- 172.21.15.90:0/36940600 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f4a9c027690 con 0x7f4a9c023a10 2024-04-14T15:23:25.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.405+0000 7f4aad7fa700 10 client.49636.objecter ms_dispatch 0x7f4ab0001550 command_reply(tid 13: 0 ) 2024-04-14T15:23:25.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.405+0000 7f4aad7fa700 10 client.49636.objecter _finish_command 13 = system:0 2024-04-14T15:23:25.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.405+0000 7f4aad7fa700 20 client.49636.objecter put_session s=0x7f4a9c017930 osd=6 4 2024-04-14T15:23:25.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.405+0000 7f4aad7fa700 15 client.49636.objecter _session_command_op_remove 6 13 2024-04-14T15:23:25.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:25.411 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:25.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:25.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.421+0000 7f4ab637b700 10 client.49636.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:23:25.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.421+0000 7f4ab637b700 15 client.49636.objecter _session_command_op_assign -1 14 2024-04-14T15:23:25.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.421+0000 7f4ab637b700 20 client.49636.objecter _get_session s=0x7f4a9c017930 osd=6 3 2024-04-14T15:23:25.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.421+0000 7f4ab637b700 20 client.49636.objecter put_session s=0x7f4a9c017930 osd=6 3 2024-04-14T15:23:25.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.421+0000 7f4ab637b700 20 client.49636.objecter _get_session s=0x7f4a9c017930 osd=6 3 2024-04-14T15:23:25.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.421+0000 7f4ab637b700 15 client.49636.objecter _session_command_op_remove -1 14 2024-04-14T15:23:25.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.421+0000 7f4ab637b700 20 client.49636.objecter get_session s=0x7f4a9c017930 osd=6 3 2024-04-14T15:23:25.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.421+0000 7f4ab637b700 15 client.49636.objecter _session_command_op_assign 6 14 2024-04-14T15:23:25.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.421+0000 7f4ab637b700 20 client.49636.objecter put_session s=0x7f4a9c017930 osd=6 4 2024-04-14T15:23:25.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.421+0000 7f4ab637b700 10 client.49636.objecter _send_command 14 2024-04-14T15:23:25.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.421+0000 7f4ab637b700 1 -- 172.21.15.90:0/36940600 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4a9c029040 con 0x7f4a9c023a10 2024-04-14T15:23:25.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.423+0000 7f4aad7fa700 1 -- 172.21.15.90:0/36940600 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f4a9c029040 con 0x7f4a9c023a10 2024-04-14T15:23:25.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.423+0000 7f4aad7fa700 10 client.49636.objecter ms_dispatch 0x7f4ab0001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:23:25.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.423+0000 7f4aad7fa700 10 client.49636.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:23:25.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.423+0000 7f4aad7fa700 20 client.49636.objecter put_session s=0x7f4a9c017930 osd=6 4 2024-04-14T15:23:25.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.423+0000 7f4aad7fa700 15 client.49636.objecter _session_command_op_remove 6 14 2024-04-14T15:23:25.423 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:23:25.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.423+0000 7f4ab637b700 10 client.49636.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:23:25.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.423+0000 7f4ab637b700 15 client.49636.objecter _session_command_op_assign -1 15 2024-04-14T15:23:25.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.423+0000 7f4ab637b700 20 client.49636.objecter _get_session s=0x7f4a9c029040 osd=7 3 2024-04-14T15:23:25.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.423+0000 7f4ab637b700 20 client.49636.objecter put_session s=0x7f4a9c029040 osd=7 3 2024-04-14T15:23:25.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.423+0000 7f4ab637b700 20 client.49636.objecter _get_session s=0x7f4a9c029040 osd=7 3 2024-04-14T15:23:25.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.423+0000 7f4ab637b700 15 client.49636.objecter _session_command_op_remove -1 15 2024-04-14T15:23:25.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.423+0000 7f4ab637b700 20 client.49636.objecter get_session s=0x7f4a9c029040 osd=7 3 2024-04-14T15:23:25.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.423+0000 7f4ab637b700 15 client.49636.objecter _session_command_op_assign 7 15 2024-04-14T15:23:25.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.423+0000 7f4ab637b700 20 client.49636.objecter put_session s=0x7f4a9c029040 osd=7 4 2024-04-14T15:23:25.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.423+0000 7f4ab637b700 10 client.49636.objecter _send_command 15 2024-04-14T15:23:25.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.423+0000 7f4ab637b700 1 -- 172.21.15.90:0/36940600 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f4a9c029ba0 con 0x7f4a9c029760 2024-04-14T15:23:25.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.423+0000 7f4aad7fa700 10 client.49636.objecter ms_handle_connect 0x7f4a9c029760 2024-04-14T15:23:25.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.424+0000 7f4aad7fa700 1 -- 172.21.15.90:0/36940600 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f4a9c029ba0 con 0x7f4a9c029760 2024-04-14T15:23:25.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.424+0000 7f4aad7fa700 10 client.49636.objecter ms_dispatch 0x7f4ab0001550 command_reply(tid 15: 0 ) 2024-04-14T15:23:25.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.424+0000 7f4aad7fa700 10 client.49636.objecter _finish_command 15 = system:0 2024-04-14T15:23:25.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.424+0000 7f4aad7fa700 20 client.49636.objecter put_session s=0x7f4a9c029040 osd=7 4 2024-04-14T15:23:25.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.424+0000 7f4aad7fa700 15 client.49636.objecter _session_command_op_remove 7 15 2024-04-14T15:23:25.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.441+0000 7f4ab637b700 10 client.49636.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:23:25.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.441+0000 7f4ab637b700 15 client.49636.objecter _session_command_op_assign -1 16 2024-04-14T15:23:25.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.441+0000 7f4ab637b700 20 client.49636.objecter _get_session s=0x7f4a9c029040 osd=7 3 2024-04-14T15:23:25.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.441+0000 7f4ab637b700 20 client.49636.objecter put_session s=0x7f4a9c029040 osd=7 3 2024-04-14T15:23:25.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.441+0000 7f4ab637b700 20 client.49636.objecter _get_session s=0x7f4a9c029040 osd=7 3 2024-04-14T15:23:25.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.441+0000 7f4ab637b700 15 client.49636.objecter _session_command_op_remove -1 16 2024-04-14T15:23:25.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.441+0000 7f4ab637b700 20 client.49636.objecter get_session s=0x7f4a9c029040 osd=7 3 2024-04-14T15:23:25.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.441+0000 7f4ab637b700 15 client.49636.objecter _session_command_op_assign 7 16 2024-04-14T15:23:25.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.441+0000 7f4ab637b700 20 client.49636.objecter put_session s=0x7f4a9c029040 osd=7 4 2024-04-14T15:23:25.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.441+0000 7f4ab637b700 10 client.49636.objecter _send_command 16 2024-04-14T15:23:25.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.441+0000 7f4ab637b700 1 -- 172.21.15.90:0/36940600 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4a9c029ba0 con 0x7f4a9c029760 2024-04-14T15:23:25.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.443+0000 7f4aad7fa700 1 -- 172.21.15.90:0/36940600 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f4a9c029ba0 con 0x7f4a9c029760 2024-04-14T15:23:25.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.443+0000 7f4aad7fa700 10 client.49636.objecter ms_dispatch 0x7f4ab0001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:23:25.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.443+0000 7f4aad7fa700 10 client.49636.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:23:25.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.443+0000 7f4aad7fa700 20 client.49636.objecter put_session s=0x7f4a9c029040 osd=7 4 2024-04-14T15:23:25.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.443+0000 7f4aad7fa700 15 client.49636.objecter _session_command_op_remove 7 16 2024-04-14T15:23:25.444 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:23:25.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.444+0000 7f4ab637b700 10 librados: watch_flush enter 2024-04-14T15:23:25.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.444+0000 7f4ab637b700 10 librados: watch_flush exit 2024-04-14T15:23:25.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.444+0000 7f4ab637b700 10 client.49636.objecter close_session for osd.0 2024-04-14T15:23:25.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.444+0000 7f4ab637b700 1 -- 172.21.15.90:0/36940600 >> v1:172.21.15.90:6801/3761806885 conn(0x7f4a9c004730 legacy=0x7f4a9c002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:25.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.444+0000 7f4ab637b700 20 client.49636.objecter put_session s=0x7f4a9c0053e0 osd=0 1 2024-04-14T15:23:25.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.444+0000 7f4ab637b700 10 client.49636.objecter close_session for osd.1 2024-04-14T15:23:25.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.444+0000 7f4ab637b700 1 -- 172.21.15.90:0/36940600 >> v1:172.21.15.90:6805/4170361699 conn(0x7f4a9c004280 legacy=0x7f4a9c00f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:25.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.444+0000 7f4ab637b700 20 client.49636.objecter put_session s=0x7f4a9c004110 osd=1 1 2024-04-14T15:23:25.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.444+0000 7f4ab637b700 10 client.49636.objecter close_session for osd.2 2024-04-14T15:23:25.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.444+0000 7f4ab637b700 1 -- 172.21.15.90:0/36940600 >> v1:172.21.15.90:6809/1686184251 conn(0x7f4a9c00cc90 legacy=0x7f4a9c0139f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:25.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.444+0000 7f4ab637b700 20 client.49636.objecter put_session s=0x7f4a9c001a00 osd=2 1 2024-04-14T15:23:25.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.444+0000 7f4ab637b700 10 client.49636.objecter close_session for osd.3 2024-04-14T15:23:25.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.444+0000 7f4ab637b700 1 -- 172.21.15.90:0/36940600 >> v1:172.21.15.90:6813/2270166774 conn(0x7f4a9c0154e0 legacy=0x7f4a9c015970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:25.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.444+0000 7f4ab637b700 20 client.49636.objecter put_session s=0x7f4a9c00c580 osd=3 1 2024-04-14T15:23:25.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.444+0000 7f4ab637b700 10 client.49636.objecter close_session for osd.4 2024-04-14T15:23:25.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.444+0000 7f4ab637b700 1 -- 172.21.15.90:0/36940600 >> v1:172.21.15.92:6800/3887549733 conn(0x7f4a9c01aef0 legacy=0x7f4a9c01d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:25.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.444+0000 7f4ab637b700 20 client.49636.objecter put_session s=0x7f4a9c017bb0 osd=4 1 2024-04-14T15:23:25.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.444+0000 7f4ab637b700 10 client.49636.objecter close_session for osd.5 2024-04-14T15:23:25.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.444+0000 7f4ab637b700 1 -- 172.21.15.90:0/36940600 >> v1:172.21.15.92:6804/3244561440 conn(0x7f4a9c01f720 legacy=0x7f4a9c01fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:25.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.444+0000 7f4ab637b700 20 client.49636.objecter put_session s=0x7f4a9c010b70 osd=5 1 2024-04-14T15:23:25.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.444+0000 7f4ab637b700 10 client.49636.objecter close_session for osd.6 2024-04-14T15:23:25.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.444+0000 7f4ab637b700 1 -- 172.21.15.90:0/36940600 >> v1:172.21.15.92:6808/636125963 conn(0x7f4a9c023a10 legacy=0x7f4a9c025e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:25.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.444+0000 7f4ab637b700 20 client.49636.objecter put_session s=0x7f4a9c017930 osd=6 1 2024-04-14T15:23:25.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.444+0000 7f4ab4918700 1 -- 172.21.15.90:0/36940600 reap_dead start 2024-04-14T15:23:25.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.444+0000 7f4ab637b700 10 client.49636.objecter close_session for osd.7 2024-04-14T15:23:25.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.444+0000 7f4ab637b700 1 -- 172.21.15.90:0/36940600 >> v1:172.21.15.92:6812/1773706716 conn(0x7f4a9c029760 legacy=0x7f4a9c027870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:25.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.444+0000 7f4ab4918700 1 -- 172.21.15.90:0/36940600 reap_dead start 2024-04-14T15:23:25.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.444+0000 7f4ab4918700 1 -- 172.21.15.90:0/36940600 reap_dead start 2024-04-14T15:23:25.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.444+0000 7f4ab4918700 1 -- 172.21.15.90:0/36940600 reap_dead start 2024-04-14T15:23:25.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.444+0000 7f4ab637b700 20 client.49636.objecter put_session s=0x7f4a9c029040 osd=7 1 2024-04-14T15:23:25.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.444+0000 7f4ab637b700 20 client.49636.objecter shutdown clearing up homeless session... 2024-04-14T15:23:25.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.444+0000 7f4ab637b700 10 client.49636.objecter successfully canceled tick 2024-04-14T15:23:25.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.444+0000 7f4ab637b700 1 -- 172.21.15.90:0/36940600 >> v1:172.21.15.90:6800/420935427 conn(0x7f4a980714b0 legacy=0x7f4a98073970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:25.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.444+0000 7f4ab637b700 1 -- 172.21.15.90:0/36940600 >> v1:172.21.15.92:6789/0 conn(0x7f4ab01023f0 legacy=0x7f4ab0106140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:25.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.444+0000 7f4ab637b700 1 -- 172.21.15.90:0/36940600 shutdown_connections 2024-04-14T15:23:25.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.444+0000 7f4ab637b700 1 -- 172.21.15.90:0/36940600 >> 172.21.15.90:0/36940600 conn(0x7f4ab0076aa0 msgr2=0x7f4ab0100070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:23:25.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.444+0000 7f4ab637b700 1 -- 172.21.15.90:0/36940600 shutdown_connections 2024-04-14T15:23:25.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.444+0000 7f4ab637b700 1 -- 172.21.15.90:0/36940600 wait complete. 2024-04-14T15:23:25.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:25.444+0000 7f4ab637b700 1 librados: shutdown 2024-04-14T15:23:25.448 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:23:25.448 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:23:25.448 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:23:25.448 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:23:25.448 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:23:25.448 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:23:25.448 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:23:25.448 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:23:25.512 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:23:25.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:25.537 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:25.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:23:25.609 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:23:25 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:23:25.638 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:23:25.662 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:25.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:25.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:23:25.764 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:23:25.787 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:25.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:25.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:23:25.889 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:25.912 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:25.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:25.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:26.014 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:23:26.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:26.043 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:26.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:23:26.144 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:26.168 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:26.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:26.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:26.269 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:23:26.293 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:26.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:26.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:23:26.311 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:26 smithi090 ceph-mon[41670]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "326.5"}]: dispatch 2024-04-14T15:23:26.312 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:26 smithi090 ceph-mon[41670]: 326.5 scrub starts 2024-04-14T15:23:26.312 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:26 smithi090 ceph-mon[41670]: 326.5 scrub ok 2024-04-14T15:23:26.312 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:26 smithi090 ceph-mon[41670]: 2.e scrub starts 2024-04-14T15:23:26.312 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:26 smithi090 ceph-mon[41670]: 2.e scrub ok 2024-04-14T15:23:26.312 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:26 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-04-14T15:23:26.312 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:26 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/36940600' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:23:26.312 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:26 smithi090 ceph-mon[41670]: 5.c scrub starts 2024-04-14T15:23:26.312 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:26 smithi090 ceph-mon[41670]: 5.c scrub ok 2024-04-14T15:23:26.312 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:26 smithi090 ceph-mon[41670]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:23:26.312 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:26 smithi090 ceph-mon[30399]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "326.5"}]: dispatch 2024-04-14T15:23:26.312 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:26 smithi090 ceph-mon[30399]: 326.5 scrub starts 2024-04-14T15:23:26.312 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:26 smithi090 ceph-mon[30399]: 326.5 scrub ok 2024-04-14T15:23:26.313 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:26 smithi090 ceph-mon[30399]: 2.e scrub starts 2024-04-14T15:23:26.313 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:26 smithi090 ceph-mon[30399]: 2.e scrub ok 2024-04-14T15:23:26.313 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:26 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-04-14T15:23:26.313 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:26 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/36940600' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:23:26.313 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:26 smithi090 ceph-mon[30399]: 5.c scrub starts 2024-04-14T15:23:26.313 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:26 smithi090 ceph-mon[30399]: 5.c scrub ok 2024-04-14T15:23:26.313 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:26 smithi090 ceph-mon[30399]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:23:26.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:26 smithi092 ceph-mon[33966]: from='mon.? v1:172.21.15.92:0/1106270647' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "326.5"}]: dispatch 2024-04-14T15:23:26.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:26 smithi092 ceph-mon[33966]: 326.5 scrub starts 2024-04-14T15:23:26.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:26 smithi092 ceph-mon[33966]: 326.5 scrub ok 2024-04-14T15:23:26.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:26 smithi092 ceph-mon[33966]: 2.e scrub starts 2024-04-14T15:23:26.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:26 smithi092 ceph-mon[33966]: 2.e scrub ok 2024-04-14T15:23:26.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:26 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-04-14T15:23:26.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:26 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/36940600' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:23:26.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:26 smithi092 ceph-mon[33966]: 5.c scrub starts 2024-04-14T15:23:26.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:26 smithi092 ceph-mon[33966]: 5.c scrub ok 2024-04-14T15:23:26.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:26 smithi092 ceph-mon[33966]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:23:26.395 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:26.418 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:26.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:26.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:26.520 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:23:26.544 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:26.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:26.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:23:26.647 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:23:26.670 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:26.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:26.671 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:23:26.772 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:23:26.794 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:26.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:26.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:23:26.896 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:23:26.919 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:26.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:26.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:23:26.949 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-04-14T15:23:26.949 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:23:27.021 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:23:27.047 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:27.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:27.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:23:27.149 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:23:27.174 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:27.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:27.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:23:27.276 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:23:27.299 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:27.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:27.300 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:23:27.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:27 smithi092 ceph-mon[33966]: pgmap v983: 216 pgs: 216 active+clean; 582 KiB data, 589 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 4 op/s; 77 KiB/s, 0 objects/s recovering 2024-04-14T15:23:27.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:27 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:23:27.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:27 smithi092 ceph-mon[33966]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-04-14T15:23:27.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:27 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]': finished 2024-04-14T15:23:27.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:27 smithi092 ceph-mon[33966]: osdmap e669: 8 total, 8 up, 7 in 2024-04-14T15:23:27.401 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:23:27.423 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:27.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:27.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:23:27.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:27 smithi090 ceph-mon[30399]: pgmap v983: 216 pgs: 216 active+clean; 582 KiB data, 589 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 4 op/s; 77 KiB/s, 0 objects/s recovering 2024-04-14T15:23:27.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:27 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:23:27.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:27 smithi090 ceph-mon[30399]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-04-14T15:23:27.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:27 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]': finished 2024-04-14T15:23:27.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:27 smithi090 ceph-mon[30399]: osdmap e669: 8 total, 8 up, 7 in 2024-04-14T15:23:27.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:27 smithi090 ceph-mon[41670]: pgmap v983: 216 pgs: 216 active+clean; 582 KiB data, 589 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 4 op/s; 77 KiB/s, 0 objects/s recovering 2024-04-14T15:23:27.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:27 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:23:27.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:27 smithi090 ceph-mon[41670]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-04-14T15:23:27.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:27 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]': finished 2024-04-14T15:23:27.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:27 smithi090 ceph-mon[41670]: osdmap e669: 8 total, 8 up, 7 in 2024-04-14T15:23:27.525 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:23:27.548 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-14T15:23:27.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:27.549 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:27.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:23:27.650 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:27.674 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:27.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:27.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:27.775 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:23:27.790 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:23:27.800 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:27.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:27.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:23:27.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.892+0000 7f8e3e38b700 1 -- 172.21.15.90:0/1332185206 >> v1:172.21.15.90:6789/0 conn(0x7f8e3810e1e0 legacy=0x7f8e381106a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:27.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.892+0000 7f8e3e38b700 1 -- 172.21.15.90:0/1332185206 shutdown_connections 2024-04-14T15:23:27.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.892+0000 7f8e3e38b700 1 -- 172.21.15.90:0/1332185206 >> 172.21.15.90:0/1332185206 conn(0x7f8e380fd290 msgr2=0x7f8e380ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:23:27.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.892+0000 7f8e3e38b700 1 -- 172.21.15.90:0/1332185206 shutdown_connections 2024-04-14T15:23:27.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.892+0000 7f8e3e38b700 1 -- 172.21.15.90:0/1332185206 wait complete. 2024-04-14T15:23:27.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.892+0000 7f8e3e38b700 1 librados: starting msgr at 2024-04-14T15:23:27.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.892+0000 7f8e3e38b700 1 librados: starting objecter 2024-04-14T15:23:27.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.893+0000 7f8e3e38b700 1 Processor -- start 2024-04-14T15:23:27.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.893+0000 7f8e3e38b700 1 -- start start 2024-04-14T15:23:27.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.893+0000 7f8e3e38b700 1 librados: setting wanted keys 2024-04-14T15:23:27.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.893+0000 7f8e3e38b700 1 librados: calling monclient init 2024-04-14T15:23:27.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.893+0000 7f8e3e38b700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f8e3810dec0 con 0x7f8e3810a610 2024-04-14T15:23:27.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.893+0000 7f8e3e38b700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f8e381ab1f0 con 0x7f8e3810e1e0 2024-04-14T15:23:27.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.893+0000 7f8e3e38b700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f8e381ac3d0 con 0x7f8e38107760 2024-04-14T15:23:27.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.893+0000 7f8e37fff700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f8e38107760 0x7f8e3806a460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:39910/0 (socket says 172.21.15.90:39910) 2024-04-14T15:23:27.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.893+0000 7f8e37fff700 1 -- 172.21.15.90:0/2105001113 learned_addr learned my addr 172.21.15.90:0/2105001113 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:23:27.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.893+0000 7f8e357fa700 10 client.?.objecter ms_handle_connect 0x7f8e3810a610 2024-04-14T15:23:27.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.893+0000 7f8e357fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:23:27.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.893+0000 7f8e357fa700 10 client.?.objecter ms_handle_connect 0x7f8e38107760 2024-04-14T15:23:27.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.893+0000 7f8e357fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:23:27.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.893+0000 7f8e357fa700 10 client.?.objecter ms_handle_connect 0x7f8e3810e1e0 2024-04-14T15:23:27.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.893+0000 7f8e357fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:23:27.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.893+0000 7f8e357fa700 1 -- 172.21.15.90:0/2105001113 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3175114256 0 0) 0x7f8e3810dec0 con 0x7f8e3810a610 2024-04-14T15:23:27.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.893+0000 7f8e357fa700 1 -- 172.21.15.90:0/2105001113 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e200031f0 con 0x7f8e3810a610 2024-04-14T15:23:27.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.893+0000 7f8e357fa700 1 -- 172.21.15.90:0/2105001113 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1502698444 0 0) 0x7f8e381ac3d0 con 0x7f8e38107760 2024-04-14T15:23:27.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.893+0000 7f8e357fa700 1 -- 172.21.15.90:0/2105001113 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e3810dec0 con 0x7f8e38107760 2024-04-14T15:23:27.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.893+0000 7f8e357fa700 1 -- 172.21.15.90:0/2105001113 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2590843114 0 0) 0x7f8e381ab1f0 con 0x7f8e3810e1e0 2024-04-14T15:23:27.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.893+0000 7f8e357fa700 1 -- 172.21.15.90:0/2105001113 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e381ac3d0 con 0x7f8e3810e1e0 2024-04-14T15:23:27.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.894+0000 7f8e357fa700 1 -- 172.21.15.90:0/2105001113 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3119798369 0 0) 0x7f8e200031f0 con 0x7f8e3810a610 2024-04-14T15:23:27.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.894+0000 7f8e357fa700 1 -- 172.21.15.90:0/2105001113 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8e381ab1f0 con 0x7f8e3810a610 2024-04-14T15:23:27.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.894+0000 7f8e357fa700 1 -- 172.21.15.90:0/2105001113 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f8e280018b0 con 0x7f8e3810a610 2024-04-14T15:23:27.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.894+0000 7f8e357fa700 1 -- 172.21.15.90:0/2105001113 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 217741222 0 0) 0x7f8e381ac3d0 con 0x7f8e3810e1e0 2024-04-14T15:23:27.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.894+0000 7f8e357fa700 1 -- 172.21.15.90:0/2105001113 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8e200031f0 con 0x7f8e3810e1e0 2024-04-14T15:23:27.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.894+0000 7f8e357fa700 1 -- 172.21.15.90:0/2105001113 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2865602353 0 0) 0x7f8e3810dec0 con 0x7f8e38107760 2024-04-14T15:23:27.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.894+0000 7f8e357fa700 1 -- 172.21.15.90:0/2105001113 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8e381ac3d0 con 0x7f8e38107760 2024-04-14T15:23:27.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.894+0000 7f8e357fa700 1 -- 172.21.15.90:0/2105001113 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f8e300017b0 con 0x7f8e3810e1e0 2024-04-14T15:23:27.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.894+0000 7f8e357fa700 1 -- 172.21.15.90:0/2105001113 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f8e2c001830 con 0x7f8e38107760 2024-04-14T15:23:27.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.894+0000 7f8e357fa700 1 -- 172.21.15.90:0/2105001113 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3915063837 0 0) 0x7f8e381ab1f0 con 0x7f8e3810a610 2024-04-14T15:23:27.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.894+0000 7f8e357fa700 1 -- 172.21.15.90:0/2105001113 >> v1:172.21.15.90:6790/0 conn(0x7f8e38107760 legacy=0x7f8e3806a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:27.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.894+0000 7f8e357fa700 1 -- 172.21.15.90:0/2105001113 >> v1:172.21.15.92:6789/0 conn(0x7f8e3810e1e0 legacy=0x7f8e381a8ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:27.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.894+0000 7f8e357fa700 1 -- 172.21.15.90:0/2105001113 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e381ad5b0 con 0x7f8e3810a610 2024-04-14T15:23:27.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.894+0000 7f8e3e38b700 1 -- 172.21.15.90:0/2105001113 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8e381ac600 con 0x7f8e3810a610 2024-04-14T15:23:27.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.894+0000 7f8e3e38b700 10 client.37809.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:23:27.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.894+0000 7f8e3e38b700 1 -- 172.21.15.90:0/2105001113 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f8e381acac0 con 0x7f8e3810a610 2024-04-14T15:23:27.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.894+0000 7f8e3e38b700 1 librados: init done 2024-04-14T15:23:27.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.894+0000 7f8e357fa700 1 -- 172.21.15.90:0/2105001113 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f8e28003b40 con 0x7f8e3810a610 2024-04-14T15:23:27.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.895+0000 7f8e3e38b700 10 client.37809.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:23:27.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.895+0000 7f8e3e38b700 15 client.37809.objecter _session_command_op_assign -1 1 2024-04-14T15:23:27.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.895+0000 7f8e3e38b700 20 client.37809.objecter _get_session osd=-1 returning homeless 2024-04-14T15:23:27.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.895+0000 7f8e3e38b700 10 client.37809.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:23:27.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.895+0000 7f8e357fa700 1 -- 172.21.15.90:0/2105001113 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f8e28004a60 con 0x7f8e3810a610 2024-04-14T15:23:27.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.895+0000 7f8e3e38b700 1 -- 172.21.15.90:0/2105001113 --> v1:172.21.15.90:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f8e38102ea0 con 0x7f8e3810a610 2024-04-14T15:23:27.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.896+0000 7f8e357fa700 1 -- 172.21.15.90:0/2105001113 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f8e28005dc0 con 0x7f8e3810a610 2024-04-14T15:23:27.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.896+0000 7f8e357fa700 1 -- 172.21.15.90:0/2105001113 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(670..670 src has 467..670) ==== 8673+0+0 (unknown 913393756 0 0) 0x7f8e280899a0 con 0x7f8e3810a610 2024-04-14T15:23:27.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.896+0000 7f8e357fa700 10 client.37809.objecter ms_dispatch 0x7f8e38001550 osd_map(670..670 src has 467..670) 2024-04-14T15:23:27.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.896+0000 7f8e357fa700 3 client.37809.objecter handle_osd_map got epochs [670,670] > 0 2024-04-14T15:23:27.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.896+0000 7f8e357fa700 3 client.37809.objecter handle_osd_map decoding full epoch 670 2024-04-14T15:23:27.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.896+0000 7f8e357fa700 10 client.37809.objecter checking command 1 2024-04-14T15:23:27.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.896+0000 7f8e357fa700 20 client.37809.objecter _get_session s=0x7f8e2007ec70 osd=1 3 2024-04-14T15:23:27.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.896+0000 7f8e357fa700 20 client.37809.objecter put_session s=0x7f8e2007ec70 osd=1 3 2024-04-14T15:23:27.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.896+0000 7f8e357fa700 15 client.37809.objecter _session_command_op_remove -1 1 2024-04-14T15:23:27.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.896+0000 7f8e357fa700 20 client.37809.objecter _get_session s=0x7f8e2007ec70 osd=1 3 2024-04-14T15:23:27.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.896+0000 7f8e357fa700 20 client.37809.objecter get_session s=0x7f8e2007ec70 osd=1 3 2024-04-14T15:23:27.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.896+0000 7f8e357fa700 15 client.37809.objecter _session_command_op_assign 1 1 2024-04-14T15:23:27.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.896+0000 7f8e357fa700 20 client.37809.objecter put_session s=0x7f8e2007ec70 osd=1 4 2024-04-14T15:23:27.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.896+0000 7f8e357fa700 10 client.37809.objecter _send_command 1 2024-04-14T15:23:27.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.896+0000 7f8e357fa700 1 -- 172.21.15.90:0/2105001113 --> v1:172.21.15.90:6805/4170361699 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8e20083030 con 0x7f8e2007f320 2024-04-14T15:23:27.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.896+0000 7f8e357fa700 20 client.37809.objecter dump_active .. 0 homeless 2024-04-14T15:23:27.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.896+0000 7f8e357fa700 10 client.37809.objecter ms_handle_connect 0x7f8e20071360 2024-04-14T15:23:27.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.896+0000 7f8e357fa700 1 -- 172.21.15.90:0/2105001113 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_get_version_reply(handle=1 version=670) ==== 24+0+0 (unknown 1700807753 0 0) 0x7f8e28089db0 con 0x7f8e3810a610 2024-04-14T15:23:27.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.897+0000 7f8e357fa700 10 client.37809.objecter ms_handle_connect 0x7f8e2007f320 2024-04-14T15:23:27.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.897+0000 7f8e357fa700 1 -- 172.21.15.90:0/2105001113 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f8e20083030 con 0x7f8e2007f320 2024-04-14T15:23:27.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.897+0000 7f8e357fa700 10 client.37809.objecter ms_dispatch 0x7f8e38001550 command_reply(tid 1: 0 ) 2024-04-14T15:23:27.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.897+0000 7f8e357fa700 10 client.37809.objecter _finish_command 1 = system:0 2024-04-14T15:23:27.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.897+0000 7f8e357fa700 20 client.37809.objecter put_session s=0x7f8e2007ec70 osd=1 4 2024-04-14T15:23:27.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:27.897+0000 7f8e357fa700 15 client.37809.objecter _session_command_op_remove 1 1 2024-04-14T15:23:27.901 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:23:27.925 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:27.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:27.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:23:27.926 INFO:teuthology.orchestra.run.smithi090.stderr:no valid command found; 10 closest matches: 2024-04-14T15:23:27.927 INFO:teuthology.orchestra.run.smithi090.stderr:0 2024-04-14T15:23:27.927 INFO:teuthology.orchestra.run.smithi090.stderr:1 2024-04-14T15:23:27.927 INFO:teuthology.orchestra.run.smithi090.stderr:2 2024-04-14T15:23:27.927 INFO:teuthology.orchestra.run.smithi090.stderr:abort 2024-04-14T15:23:27.927 INFO:teuthology.orchestra.run.smithi090.stderr:assert 2024-04-14T15:23:27.927 INFO:teuthology.orchestra.run.smithi090.stderr:bench [] [] [] [] 2024-04-14T15:23:27.927 INFO:teuthology.orchestra.run.smithi090.stderr:bluefs debug_inject_read_zeros 2024-04-14T15:23:27.927 INFO:teuthology.orchestra.run.smithi090.stderr:bluefs files list 2024-04-14T15:23:27.927 INFO:teuthology.orchestra.run.smithi090.stderr:bluefs stats 2024-04-14T15:23:27.927 INFO:teuthology.orchestra.run.smithi090.stderr:bluestore allocator dump block 2024-04-14T15:23:27.927 INFO:teuthology.orchestra.run.smithi090.stderr:Error EINVAL: invalid command 2024-04-14T15:23:28.026 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:28.052 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:28.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:28.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:28.154 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:23:28.178 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:28.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:28.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:23:28.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:28 smithi090 ceph-mon[30399]: 2.b scrub starts 2024-04-14T15:23:28.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:28 smithi090 ceph-mon[30399]: 2.b scrub ok 2024-04-14T15:23:28.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:28 smithi090 ceph-mon[30399]: 5.0 scrub starts 2024-04-14T15:23:28.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:28 smithi090 ceph-mon[30399]: 5.0 scrub ok 2024-04-14T15:23:28.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:28 smithi090 ceph-mon[30399]: osdmap e670: 8 total, 8 up, 7 in 2024-04-14T15:23:28.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:28 smithi090 ceph-mon[30399]: pgmap v986: 216 pgs: 216 active+clean; 582 KiB data, 589 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 4 op/s; 77 KiB/s, 0 objects/s recovering 2024-04-14T15:23:28.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:28 smithi090 ceph-mon[30399]: 11.7 scrub starts 2024-04-14T15:23:28.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:28 smithi090 ceph-mon[30399]: 11.7 scrub ok 2024-04-14T15:23:28.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:28 smithi090 ceph-mon[41670]: 2.b scrub starts 2024-04-14T15:23:28.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:28 smithi090 ceph-mon[41670]: 2.b scrub ok 2024-04-14T15:23:28.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:28 smithi090 ceph-mon[41670]: 5.0 scrub starts 2024-04-14T15:23:28.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:28 smithi090 ceph-mon[41670]: 5.0 scrub ok 2024-04-14T15:23:28.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:28 smithi090 ceph-mon[41670]: osdmap e670: 8 total, 8 up, 7 in 2024-04-14T15:23:28.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:28 smithi090 ceph-mon[41670]: pgmap v986: 216 pgs: 216 active+clean; 582 KiB data, 589 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 4 op/s; 77 KiB/s, 0 objects/s recovering 2024-04-14T15:23:28.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:28 smithi090 ceph-mon[41670]: 11.7 scrub starts 2024-04-14T15:23:28.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:28 smithi090 ceph-mon[41670]: 11.7 scrub ok 2024-04-14T15:23:28.198 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:28 smithi092 ceph-mon[33966]: 2.b scrub starts 2024-04-14T15:23:28.198 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:28 smithi092 ceph-mon[33966]: 2.b scrub ok 2024-04-14T15:23:28.198 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:28 smithi092 ceph-mon[33966]: 5.0 scrub starts 2024-04-14T15:23:28.198 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:28 smithi092 ceph-mon[33966]: 5.0 scrub ok 2024-04-14T15:23:28.198 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:28 smithi092 ceph-mon[33966]: osdmap e670: 8 total, 8 up, 7 in 2024-04-14T15:23:28.198 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:28 smithi092 ceph-mon[33966]: pgmap v986: 216 pgs: 216 active+clean; 582 KiB data, 589 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 4 op/s; 77 KiB/s, 0 objects/s recovering 2024-04-14T15:23:28.199 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:28 smithi092 ceph-mon[33966]: 11.7 scrub starts 2024-04-14T15:23:28.199 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:28 smithi092 ceph-mon[33966]: 11.7 scrub ok 2024-04-14T15:23:28.280 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:28.304 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:28.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:28.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:28.405 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:28.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:28.429 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:28.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:28.530 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:28.553 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:28.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:28.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:28.594 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-14T15:23:28.641 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-14T15:23:28.641 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-14T15:23:28.642 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph tell osd.2 'trim stale osdmaps' 2024-04-14T15:23:28.655 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:23:28.679 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:28.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:28.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:23:28.780 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:23:28.806 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:28.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:28.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:23:28.849 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:23:28.908 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:23:28.931 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:28.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:28.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:23:29.032 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:29.058 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:29.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:29.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:29.159 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:23:29.184 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:29.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:29.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:23:29.256 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:29 smithi090 ceph-mon[41670]: 6.11 scrub starts 2024-04-14T15:23:29.257 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:29 smithi090 ceph-mon[41670]: 6.11 scrub ok 2024-04-14T15:23:29.257 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:29 smithi090 ceph-mon[41670]: osdmap e671: 8 total, 8 up, 7 in 2024-04-14T15:23:29.257 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:29 smithi090 ceph-mon[41670]: 2.13 scrub starts 2024-04-14T15:23:29.257 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:29 smithi090 ceph-mon[41670]: 2.13 scrub ok 2024-04-14T15:23:29.257 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:29 smithi090 ceph-mon[30399]: 6.11 scrub starts 2024-04-14T15:23:29.257 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:29 smithi090 ceph-mon[30399]: 6.11 scrub ok 2024-04-14T15:23:29.257 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:29 smithi090 ceph-mon[30399]: osdmap e671: 8 total, 8 up, 7 in 2024-04-14T15:23:29.257 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:29 smithi090 ceph-mon[30399]: 2.13 scrub starts 2024-04-14T15:23:29.258 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:29 smithi090 ceph-mon[30399]: 2.13 scrub ok 2024-04-14T15:23:29.285 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:23:29.309 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:29.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:29.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:23:29.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:29 smithi092 ceph-mon[33966]: 6.11 scrub starts 2024-04-14T15:23:29.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:29 smithi092 ceph-mon[33966]: 6.11 scrub ok 2024-04-14T15:23:29.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:29 smithi092 ceph-mon[33966]: osdmap e671: 8 total, 8 up, 7 in 2024-04-14T15:23:29.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:29 smithi092 ceph-mon[33966]: 2.13 scrub starts 2024-04-14T15:23:29.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:29 smithi092 ceph-mon[33966]: 2.13 scrub ok 2024-04-14T15:23:29.411 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:23:29.435 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:29.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:29.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:23:29.537 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:23:29.560 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:29.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:29.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:23:29.647 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-04-14T15:23:29.647 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:23:29.647 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:23:29.654 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-14T15:23:29.661 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:23:29.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:29.687 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:29.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:23:29.788 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:29.811 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:29.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:29.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:29.913 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:29.936 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:29.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:29.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:30.038 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:23:30.064 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:30.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:30.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:23:30.166 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:23:30.189 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:30.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:30.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:23:30.222 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:30 smithi090 ceph-mon[41670]: 6.3 scrub starts 2024-04-14T15:23:30.222 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:30 smithi090 ceph-mon[41670]: 6.3 scrub ok 2024-04-14T15:23:30.222 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:30 smithi090 ceph-mon[41670]: osdmap e672: 8 total, 8 up, 7 in 2024-04-14T15:23:30.222 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:30 smithi090 ceph-mon[41670]: pgmap v989: 215 pgs: 215 active+clean; 262 KiB data, 590 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:23:30.222 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:30 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-04-14T15:23:30.223 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:30 smithi090 ceph-mon[41670]: 11.0 scrub starts 2024-04-14T15:23:30.223 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:30 smithi090 ceph-mon[41670]: 11.0 scrub ok 2024-04-14T15:23:30.223 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:30 smithi090 ceph-mon[41670]: 6.16 scrub starts 2024-04-14T15:23:30.223 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:30 smithi090 ceph-mon[41670]: 6.16 scrub ok 2024-04-14T15:23:30.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:30 smithi090 ceph-mon[30399]: 6.3 scrub starts 2024-04-14T15:23:30.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:30 smithi090 ceph-mon[30399]: 6.3 scrub ok 2024-04-14T15:23:30.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:30 smithi090 ceph-mon[30399]: osdmap e672: 8 total, 8 up, 7 in 2024-04-14T15:23:30.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:30 smithi090 ceph-mon[30399]: pgmap v989: 215 pgs: 215 active+clean; 262 KiB data, 590 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:23:30.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:30 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-04-14T15:23:30.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:30 smithi090 ceph-mon[30399]: 11.0 scrub starts 2024-04-14T15:23:30.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:30 smithi090 ceph-mon[30399]: 11.0 scrub ok 2024-04-14T15:23:30.224 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:30 smithi090 ceph-mon[30399]: 6.16 scrub starts 2024-04-14T15:23:30.224 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:30 smithi090 ceph-mon[30399]: 6.16 scrub ok 2024-04-14T15:23:30.291 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:23:30.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:30.318 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:30.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:23:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:30 smithi092 ceph-mon[33966]: 6.3 scrub starts 2024-04-14T15:23:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:30 smithi092 ceph-mon[33966]: 6.3 scrub ok 2024-04-14T15:23:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:30 smithi092 ceph-mon[33966]: osdmap e672: 8 total, 8 up, 7 in 2024-04-14T15:23:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:30 smithi092 ceph-mon[33966]: pgmap v989: 215 pgs: 215 active+clean; 262 KiB data, 590 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:23:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:30 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-04-14T15:23:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:30 smithi092 ceph-mon[33966]: 11.0 scrub starts 2024-04-14T15:23:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:30 smithi092 ceph-mon[33966]: 11.0 scrub ok 2024-04-14T15:23:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:30 smithi092 ceph-mon[33966]: 6.16 scrub starts 2024-04-14T15:23:30.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:30 smithi092 ceph-mon[33966]: 6.16 scrub ok 2024-04-14T15:23:30.419 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:23:30.421 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-14T15:23:30.444 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:30.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:30.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:23:30.545 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:23:30.569 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:30.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:30.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:23:30.660 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:23:30.671 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:23:30.695 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:30.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:30.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:23:30.796 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:23:30.819 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:30.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:30.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:23:30.921 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:23:30.945 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:30.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:30.946 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:23:31.047 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:23:31.071 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:31.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:31.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:23:31.173 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:23:31.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:31 smithi090 ceph-mon[30399]: 6.15 scrub starts 2024-04-14T15:23:31.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:31 smithi090 ceph-mon[30399]: 6.15 scrub ok 2024-04-14T15:23:31.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:31 smithi090 ceph-mon[30399]: 2.3 scrub starts 2024-04-14T15:23:31.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:31 smithi090 ceph-mon[30399]: 2.3 scrub ok 2024-04-14T15:23:31.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:31 smithi090 ceph-mon[30399]: 4.5 scrub starts 2024-04-14T15:23:31.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:31 smithi090 ceph-mon[30399]: 4.5 scrub ok 2024-04-14T15:23:31.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:31 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-04-14T15:23:31.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:31 smithi090 ceph-mon[30399]: osdmap e673: 8 total, 8 up, 7 in 2024-04-14T15:23:31.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:31 smithi090 ceph-mon[30399]: 2.2 scrub starts 2024-04-14T15:23:31.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:31 smithi090 ceph-mon[30399]: 2.2 scrub ok 2024-04-14T15:23:31.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:31 smithi090 ceph-mon[30399]: 4.6 scrub starts 2024-04-14T15:23:31.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:31 smithi090 ceph-mon[30399]: 4.6 scrub ok 2024-04-14T15:23:31.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:31 smithi090 ceph-mon[30399]: osdmap e674: 8 total, 8 up, 7 in 2024-04-14T15:23:31.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:31 smithi090 ceph-mon[41670]: 6.15 scrub starts 2024-04-14T15:23:31.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:31 smithi090 ceph-mon[41670]: 6.15 scrub ok 2024-04-14T15:23:31.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:31 smithi090 ceph-mon[41670]: 2.3 scrub starts 2024-04-14T15:23:31.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:31 smithi090 ceph-mon[41670]: 2.3 scrub ok 2024-04-14T15:23:31.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:31 smithi090 ceph-mon[41670]: 4.5 scrub starts 2024-04-14T15:23:31.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:31 smithi090 ceph-mon[41670]: 4.5 scrub ok 2024-04-14T15:23:31.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:31 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-04-14T15:23:31.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:31 smithi090 ceph-mon[41670]: osdmap e673: 8 total, 8 up, 7 in 2024-04-14T15:23:31.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:31 smithi090 ceph-mon[41670]: 2.2 scrub starts 2024-04-14T15:23:31.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:31 smithi090 ceph-mon[41670]: 2.2 scrub ok 2024-04-14T15:23:31.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:31 smithi090 ceph-mon[41670]: 4.6 scrub starts 2024-04-14T15:23:31.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:31 smithi090 ceph-mon[41670]: 4.6 scrub ok 2024-04-14T15:23:31.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:31 smithi090 ceph-mon[41670]: osdmap e674: 8 total, 8 up, 7 in 2024-04-14T15:23:31.196 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:31.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:31.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:23:31.298 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:23:31.321 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:31.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:31.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:23:31.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:31 smithi092 ceph-mon[33966]: 6.15 scrub starts 2024-04-14T15:23:31.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:31 smithi092 ceph-mon[33966]: 6.15 scrub ok 2024-04-14T15:23:31.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:31 smithi092 ceph-mon[33966]: 2.3 scrub starts 2024-04-14T15:23:31.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:31 smithi092 ceph-mon[33966]: 2.3 scrub ok 2024-04-14T15:23:31.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:31 smithi092 ceph-mon[33966]: 4.5 scrub starts 2024-04-14T15:23:31.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:31 smithi092 ceph-mon[33966]: 4.5 scrub ok 2024-04-14T15:23:31.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:31 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-04-14T15:23:31.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:31 smithi092 ceph-mon[33966]: osdmap e673: 8 total, 8 up, 7 in 2024-04-14T15:23:31.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:31 smithi092 ceph-mon[33966]: 2.2 scrub starts 2024-04-14T15:23:31.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:31 smithi092 ceph-mon[33966]: 2.2 scrub ok 2024-04-14T15:23:31.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:31 smithi092 ceph-mon[33966]: 4.6 scrub starts 2024-04-14T15:23:31.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:31 smithi092 ceph-mon[33966]: 4.6 scrub ok 2024-04-14T15:23:31.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:31 smithi092 ceph-mon[33966]: osdmap e674: 8 total, 8 up, 7 in 2024-04-14T15:23:31.423 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:23:31.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:31.448 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:31.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:23:31.549 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:23:31.572 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:31.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:31.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:23:31.673 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:23:31.697 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:31.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:31.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:23:31.798 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:23:31.821 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:31.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:31.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:23:31.884 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:23:31.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.885+0000 7f1d54e55700 1 -- 172.21.15.90:0/2995040507 >> v1:172.21.15.90:6790/0 conn(0x7f1d5010f4d0 legacy=0x7f1d50111990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:31.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.885+0000 7f1d54e55700 1 -- 172.21.15.90:0/2995040507 shutdown_connections 2024-04-14T15:23:31.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.885+0000 7f1d54e55700 1 -- 172.21.15.90:0/2995040507 >> 172.21.15.90:0/2995040507 conn(0x7f1d500fd290 msgr2=0x7f1d500ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:23:31.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.885+0000 7f1d54e55700 1 -- 172.21.15.90:0/2995040507 shutdown_connections 2024-04-14T15:23:31.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.885+0000 7f1d54e55700 1 -- 172.21.15.90:0/2995040507 wait complete. 2024-04-14T15:23:31.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.886+0000 7f1d54e55700 1 librados: starting msgr at 2024-04-14T15:23:31.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.886+0000 7f1d54e55700 1 librados: starting objecter 2024-04-14T15:23:31.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.886+0000 7f1d54e55700 1 Processor -- start 2024-04-14T15:23:31.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.886+0000 7f1d54e55700 1 -- start start 2024-04-14T15:23:31.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.886+0000 7f1d54e55700 1 librados: setting wanted keys 2024-04-14T15:23:31.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.886+0000 7f1d54e55700 1 librados: calling monclient init 2024-04-14T15:23:31.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.886+0000 7f1d54e55700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1d501a9d50 con 0x7f1d5010b930 2024-04-14T15:23:31.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.886+0000 7f1d54e55700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1d501aaf70 con 0x7f1d50103a40 2024-04-14T15:23:31.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.886+0000 7f1d54e55700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1d501ac1b0 con 0x7f1d5010f4d0 2024-04-14T15:23:31.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.886+0000 7f1d4ed9d700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f1d5010f4d0 0x7f1d501a8430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:54512/0 (socket says 172.21.15.90:54512) 2024-04-14T15:23:31.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.886+0000 7f1d4ed9d700 1 -- 172.21.15.90:0/2410683567 learned_addr learned my addr 172.21.15.90:0/2410683567 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:23:31.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.886+0000 7f1d3f7fe700 10 client.?.objecter ms_handle_connect 0x7f1d50103a40 2024-04-14T15:23:31.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.887+0000 7f1d3f7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:23:31.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.887+0000 7f1d3f7fe700 10 client.?.objecter ms_handle_connect 0x7f1d5010b930 2024-04-14T15:23:31.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.887+0000 7f1d3f7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:23:31.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.887+0000 7f1d3f7fe700 10 client.?.objecter ms_handle_connect 0x7f1d5010f4d0 2024-04-14T15:23:31.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.887+0000 7f1d3f7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:23:31.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.887+0000 7f1d3f7fe700 1 -- 172.21.15.90:0/2410683567 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 593308206 0 0) 0x7f1d501aaf70 con 0x7f1d50103a40 2024-04-14T15:23:31.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.887+0000 7f1d3f7fe700 1 -- 172.21.15.90:0/2410683567 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1d380039c0 con 0x7f1d50103a40 2024-04-14T15:23:31.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.887+0000 7f1d3f7fe700 1 -- 172.21.15.90:0/2410683567 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3563301955 0 0) 0x7f1d501a9d50 con 0x7f1d5010b930 2024-04-14T15:23:31.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.887+0000 7f1d3f7fe700 1 -- 172.21.15.90:0/2410683567 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1d501aaf70 con 0x7f1d5010b930 2024-04-14T15:23:31.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.887+0000 7f1d3f7fe700 1 -- 172.21.15.90:0/2410683567 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1508787699 0 0) 0x7f1d501ac1b0 con 0x7f1d5010f4d0 2024-04-14T15:23:31.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.887+0000 7f1d3f7fe700 1 -- 172.21.15.90:0/2410683567 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1d501a9d50 con 0x7f1d5010f4d0 2024-04-14T15:23:31.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.887+0000 7f1d3f7fe700 1 -- 172.21.15.90:0/2410683567 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1627574472 0 0) 0x7f1d380039c0 con 0x7f1d50103a40 2024-04-14T15:23:31.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.887+0000 7f1d3f7fe700 1 -- 172.21.15.90:0/2410683567 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1d501ac1b0 con 0x7f1d50103a40 2024-04-14T15:23:31.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.887+0000 7f1d3f7fe700 1 -- 172.21.15.90:0/2410683567 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f1d400018b0 con 0x7f1d50103a40 2024-04-14T15:23:31.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.887+0000 7f1d3f7fe700 1 -- 172.21.15.90:0/2410683567 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 667318695 0 0) 0x7f1d501aaf70 con 0x7f1d5010b930 2024-04-14T15:23:31.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.887+0000 7f1d3f7fe700 1 -- 172.21.15.90:0/2410683567 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1d380039c0 con 0x7f1d5010b930 2024-04-14T15:23:31.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.887+0000 7f1d3f7fe700 1 -- 172.21.15.90:0/2410683567 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f1d44001890 con 0x7f1d5010b930 2024-04-14T15:23:31.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.887+0000 7f1d3f7fe700 1 -- 172.21.15.90:0/2410683567 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2194522307 0 0) 0x7f1d501ac1b0 con 0x7f1d50103a40 2024-04-14T15:23:31.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.887+0000 7f1d3f7fe700 1 -- 172.21.15.90:0/2410683567 >> v1:172.21.15.90:6790/0 conn(0x7f1d5010f4d0 legacy=0x7f1d501a8430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:31.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.887+0000 7f1d3f7fe700 1 -- 172.21.15.90:0/2410683567 >> v1:172.21.15.90:6789/0 conn(0x7f1d5010b930 legacy=0x7f1d501a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:31.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.887+0000 7f1d3f7fe700 1 -- 172.21.15.90:0/2410683567 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d501ad3f0 con 0x7f1d50103a40 2024-04-14T15:23:31.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.887+0000 7f1d54e55700 1 -- 172.21.15.90:0/2410683567 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1d501a9f80 con 0x7f1d50103a40 2024-04-14T15:23:31.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.887+0000 7f1d54e55700 10 client.49648.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:23:31.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.887+0000 7f1d54e55700 1 -- 172.21.15.90:0/2410683567 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f1d501aa440 con 0x7f1d50103a40 2024-04-14T15:23:31.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.887+0000 7f1d54e55700 1 librados: init done 2024-04-14T15:23:31.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.887+0000 7f1d3f7fe700 1 -- 172.21.15.90:0/2410683567 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f1d40003b40 con 0x7f1d50103a40 2024-04-14T15:23:31.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.887+0000 7f1d3f7fe700 1 -- 172.21.15.90:0/2410683567 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f1d40004ad0 con 0x7f1d50103a40 2024-04-14T15:23:31.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.888+0000 7f1d54e55700 10 client.49648.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:23:31.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.888+0000 7f1d54e55700 15 client.49648.objecter _session_command_op_assign -1 1 2024-04-14T15:23:31.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.888+0000 7f1d54e55700 20 client.49648.objecter _get_session osd=-1 returning homeless 2024-04-14T15:23:31.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.888+0000 7f1d54e55700 10 client.49648.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:23:31.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.888+0000 7f1d54e55700 1 -- 172.21.15.90:0/2410683567 --> v1:172.21.15.92:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f1d30002920 con 0x7f1d50103a40 2024-04-14T15:23:31.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.889+0000 7f1d3f7fe700 1 -- 172.21.15.90:0/2410683567 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f1d4001b6a0 con 0x7f1d50103a40 2024-04-14T15:23:31.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.889+0000 7f1d3f7fe700 1 -- 172.21.15.90:0/2410683567 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(674..674 src has 467..674) ==== 8673+0+0 (unknown 410424936 0 0) 0x7f1d40089bf0 con 0x7f1d50103a40 2024-04-14T15:23:31.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.889+0000 7f1d3f7fe700 10 client.49648.objecter ms_dispatch 0x7f1d50001550 osd_map(674..674 src has 467..674) 2024-04-14T15:23:31.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.889+0000 7f1d3f7fe700 3 client.49648.objecter handle_osd_map got epochs [674,674] > 0 2024-04-14T15:23:31.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.889+0000 7f1d3f7fe700 3 client.49648.objecter handle_osd_map decoding full epoch 674 2024-04-14T15:23:31.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.889+0000 7f1d3f7fe700 10 client.49648.objecter checking command 1 2024-04-14T15:23:31.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.889+0000 7f1d3f7fe700 20 client.49648.objecter _get_session s=0x7f1d3807ede0 osd=2 3 2024-04-14T15:23:31.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.889+0000 7f1d3f7fe700 20 client.49648.objecter put_session s=0x7f1d3807ede0 osd=2 3 2024-04-14T15:23:31.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.889+0000 7f1d3f7fe700 15 client.49648.objecter _session_command_op_remove -1 1 2024-04-14T15:23:31.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.889+0000 7f1d3f7fe700 20 client.49648.objecter _get_session s=0x7f1d3807ede0 osd=2 3 2024-04-14T15:23:31.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.889+0000 7f1d3f7fe700 20 client.49648.objecter get_session s=0x7f1d3807ede0 osd=2 3 2024-04-14T15:23:31.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.889+0000 7f1d3f7fe700 15 client.49648.objecter _session_command_op_assign 2 1 2024-04-14T15:23:31.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.889+0000 7f1d3f7fe700 20 client.49648.objecter put_session s=0x7f1d3807ede0 osd=2 4 2024-04-14T15:23:31.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.889+0000 7f1d3f7fe700 10 client.49648.objecter _send_command 1 2024-04-14T15:23:31.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.889+0000 7f1d3f7fe700 1 -- 172.21.15.90:0/2410683567 --> v1:172.21.15.90:6809/1686184251 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1d380831a0 con 0x7f1d3807f490 2024-04-14T15:23:31.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.889+0000 7f1d3f7fe700 20 client.49648.objecter dump_active .. 0 homeless 2024-04-14T15:23:31.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.889+0000 7f1d3f7fe700 1 -- 172.21.15.90:0/2410683567 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_get_version_reply(handle=1 version=674) ==== 24+0+0 (unknown 283885124 0 0) 0x7f1d4008a000 con 0x7f1d50103a40 2024-04-14T15:23:31.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.890+0000 7f1d3f7fe700 10 client.49648.objecter ms_handle_connect 0x7f1d38071410 2024-04-14T15:23:31.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.890+0000 7f1d3f7fe700 10 client.49648.objecter ms_handle_connect 0x7f1d3807f490 2024-04-14T15:23:31.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.890+0000 7f1d3f7fe700 1 -- 172.21.15.90:0/2410683567 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1d380831a0 con 0x7f1d3807f490 2024-04-14T15:23:31.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.890+0000 7f1d3f7fe700 10 client.49648.objecter ms_dispatch 0x7f1d50001550 command_reply(tid 1: 0 ) 2024-04-14T15:23:31.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.890+0000 7f1d3f7fe700 10 client.49648.objecter _finish_command 1 = system:0 2024-04-14T15:23:31.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.890+0000 7f1d3f7fe700 20 client.49648.objecter put_session s=0x7f1d3807ede0 osd=2 4 2024-04-14T15:23:31.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:31.890+0000 7f1d3f7fe700 15 client.49648.objecter _session_command_op_remove 2 1 2024-04-14T15:23:31.922 INFO:teuthology.orchestra.run.smithi090.stderr:no valid command found; 10 closest matches: 2024-04-14T15:23:31.922 INFO:teuthology.orchestra.run.smithi090.stderr:0 2024-04-14T15:23:31.922 INFO:teuthology.orchestra.run.smithi090.stderr:1 2024-04-14T15:23:31.922 INFO:teuthology.orchestra.run.smithi090.stderr:2 2024-04-14T15:23:31.922 INFO:teuthology.orchestra.run.smithi090.stderr:abort 2024-04-14T15:23:31.922 INFO:teuthology.orchestra.run.smithi090.stderr:assert 2024-04-14T15:23:31.922 INFO:teuthology.orchestra.run.smithi090.stderr:bench [] [] [] [] 2024-04-14T15:23:31.922 INFO:teuthology.orchestra.run.smithi090.stderr:bluefs debug_inject_read_zeros 2024-04-14T15:23:31.922 INFO:teuthology.orchestra.run.smithi090.stderr:bluefs files list 2024-04-14T15:23:31.922 INFO:teuthology.orchestra.run.smithi090.stderr:bluefs stats 2024-04-14T15:23:31.922 INFO:teuthology.orchestra.run.smithi090.stderr:bluestore allocator dump block 2024-04-14T15:23:31.922 INFO:teuthology.orchestra.run.smithi090.stderr:Error EINVAL: invalid command 2024-04-14T15:23:31.923 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:23:31.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:31.947 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:31.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:23:32.048 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:23:32.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:32.073 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:32.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:23:32.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:32 smithi092 ceph-mon[33966]: 5.d scrub starts 2024-04-14T15:23:32.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:32 smithi092 ceph-mon[33966]: 5.d scrub ok 2024-04-14T15:23:32.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:32 smithi092 ceph-mon[33966]: 11.9 scrub starts 2024-04-14T15:23:32.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:32 smithi092 ceph-mon[33966]: 11.9 scrub ok 2024-04-14T15:23:32.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:32 smithi092 ceph-mon[33966]: pgmap v992: 215 pgs: 215 active+clean; 582 KiB data, 590 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:23:32.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:32 smithi092 ceph-mon[33966]: 4.4 scrub starts 2024-04-14T15:23:32.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:32 smithi092 ceph-mon[33966]: 4.4 scrub ok 2024-04-14T15:23:32.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:32 smithi092 ceph-mon[33966]: osdmap e675: 8 total, 8 up, 7 in 2024-04-14T15:23:32.153 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:32 smithi090 ceph-mon[30399]: 5.d scrub starts 2024-04-14T15:23:32.153 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:32 smithi090 ceph-mon[30399]: 5.d scrub ok 2024-04-14T15:23:32.153 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:32 smithi090 ceph-mon[30399]: 11.9 scrub starts 2024-04-14T15:23:32.154 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:32 smithi090 ceph-mon[30399]: 11.9 scrub ok 2024-04-14T15:23:32.154 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:32 smithi090 ceph-mon[30399]: pgmap v992: 215 pgs: 215 active+clean; 582 KiB data, 590 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:23:32.154 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:32 smithi090 ceph-mon[30399]: 4.4 scrub starts 2024-04-14T15:23:32.154 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:32 smithi090 ceph-mon[30399]: 4.4 scrub ok 2024-04-14T15:23:32.154 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:32 smithi090 ceph-mon[30399]: osdmap e675: 8 total, 8 up, 7 in 2024-04-14T15:23:32.154 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:32 smithi090 ceph-mon[41670]: 5.d scrub starts 2024-04-14T15:23:32.154 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:32 smithi090 ceph-mon[41670]: 5.d scrub ok 2024-04-14T15:23:32.154 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:32 smithi090 ceph-mon[41670]: 11.9 scrub starts 2024-04-14T15:23:32.154 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:32 smithi090 ceph-mon[41670]: 11.9 scrub ok 2024-04-14T15:23:32.154 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:32 smithi090 ceph-mon[41670]: pgmap v992: 215 pgs: 215 active+clean; 582 KiB data, 590 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:23:32.154 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:32 smithi090 ceph-mon[41670]: 4.4 scrub starts 2024-04-14T15:23:32.154 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:32 smithi090 ceph-mon[41670]: 4.4 scrub ok 2024-04-14T15:23:32.155 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:32 smithi090 ceph-mon[41670]: osdmap e675: 8 total, 8 up, 7 in 2024-04-14T15:23:32.175 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:23:32.197 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:32.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:32.198 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:23:32.299 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:23:32.324 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:32.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:32.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:23:32.425 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:23:32.450 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:32.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:32.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:23:32.549 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-04-14T15:23:32.549 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:23:32.551 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:23:32.574 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:32.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:32.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:23:32.676 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:32.700 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:32.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:32.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:32.801 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:23:32.826 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:32.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:32.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:23:32.928 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:23:32.951 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:32.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:32.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:23:33.053 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:23:33.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:33.077 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:33.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:23:33.178 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:33.201 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:33.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:33.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:33.266 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:33 smithi090 ceph-mon[41670]: 2.12 scrub starts 2024-04-14T15:23:33.267 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:33 smithi090 ceph-mon[41670]: 2.12 scrub ok 2024-04-14T15:23:33.267 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:33 smithi090 ceph-mon[41670]: 4.b scrub starts 2024-04-14T15:23:33.267 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:33 smithi090 ceph-mon[41670]: 4.b scrub ok 2024-04-14T15:23:33.267 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:33 smithi090 ceph-mon[41670]: osdmap e676: 8 total, 8 up, 7 in 2024-04-14T15:23:33.267 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:33 smithi090 ceph-mon[30399]: 2.12 scrub starts 2024-04-14T15:23:33.267 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:33 smithi090 ceph-mon[30399]: 2.12 scrub ok 2024-04-14T15:23:33.267 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:33 smithi090 ceph-mon[30399]: 4.b scrub starts 2024-04-14T15:23:33.268 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:33 smithi090 ceph-mon[30399]: 4.b scrub ok 2024-04-14T15:23:33.268 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:33 smithi090 ceph-mon[30399]: osdmap e676: 8 total, 8 up, 7 in 2024-04-14T15:23:33.275 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:33 smithi092 ceph-mon[33966]: 2.12 scrub starts 2024-04-14T15:23:33.276 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:33 smithi092 ceph-mon[33966]: 2.12 scrub ok 2024-04-14T15:23:33.276 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:33 smithi092 ceph-mon[33966]: 4.b scrub starts 2024-04-14T15:23:33.276 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:33 smithi092 ceph-mon[33966]: 4.b scrub ok 2024-04-14T15:23:33.276 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:33 smithi092 ceph-mon[33966]: osdmap e676: 8 total, 8 up, 7 in 2024-04-14T15:23:33.301 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-14T15:23:33.303 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:23:33.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:33.327 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:33.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:23:33.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.368+0000 7f306e489700 1 -- 172.21.15.90:0/1055820833 >> v1:172.21.15.90:6789/0 conn(0x7f306810e1e0 legacy=0x7f30681106a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:33.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.368+0000 7f306e489700 1 -- 172.21.15.90:0/1055820833 shutdown_connections 2024-04-14T15:23:33.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.368+0000 7f306e489700 1 -- 172.21.15.90:0/1055820833 >> 172.21.15.90:0/1055820833 conn(0x7f30680fd290 msgr2=0x7f30680ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:23:33.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.368+0000 7f306e489700 1 -- 172.21.15.90:0/1055820833 shutdown_connections 2024-04-14T15:23:33.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.368+0000 7f306e489700 1 -- 172.21.15.90:0/1055820833 wait complete. 2024-04-14T15:23:33.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.369+0000 7f306e489700 1 librados: starting msgr at 2024-04-14T15:23:33.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.369+0000 7f306e489700 1 librados: starting objecter 2024-04-14T15:23:33.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.369+0000 7f306e489700 1 Processor -- start 2024-04-14T15:23:33.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.369+0000 7f306e489700 1 -- start start 2024-04-14T15:23:33.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.369+0000 7f306e489700 1 librados: setting wanted keys 2024-04-14T15:23:33.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.369+0000 7f306e489700 1 librados: calling monclient init 2024-04-14T15:23:33.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.369+0000 7f306e489700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f30681a7df0 con 0x7f306810e1e0 2024-04-14T15:23:33.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.369+0000 7f306e489700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f30681a8fd0 con 0x7f3068107760 2024-04-14T15:23:33.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.369+0000 7f306e489700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f30681aa1b0 con 0x7f306810a610 2024-04-14T15:23:33.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.369+0000 7f306ca26700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f306810e1e0 0x7f30681a66d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:44324/0 (socket says 172.21.15.90:44324) 2024-04-14T15:23:33.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.369+0000 7f30677fe700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f306810a610 0x7f306810c570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:54526/0 (socket says 172.21.15.90:54526) 2024-04-14T15:23:33.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.369+0000 7f306ca26700 1 -- 172.21.15.90:0/2622333994 learned_addr learned my addr 172.21.15.90:0/2622333994 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:23:33.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.369+0000 7f30657fa700 10 client.?.objecter ms_handle_connect 0x7f306810e1e0 2024-04-14T15:23:33.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.369+0000 7f30657fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:23:33.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.369+0000 7f30657fa700 10 client.?.objecter ms_handle_connect 0x7f3068107760 2024-04-14T15:23:33.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.369+0000 7f30657fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:23:33.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.369+0000 7f30657fa700 10 client.?.objecter ms_handle_connect 0x7f306810a610 2024-04-14T15:23:33.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.369+0000 7f30657fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:23:33.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.370+0000 7f30657fa700 1 -- 172.21.15.90:0/2622333994 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3424274994 0 0) 0x7f30681a7df0 con 0x7f306810e1e0 2024-04-14T15:23:33.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.370+0000 7f30657fa700 1 -- 172.21.15.90:0/2622333994 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f30500031f0 con 0x7f306810e1e0 2024-04-14T15:23:33.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.370+0000 7f30657fa700 1 -- 172.21.15.90:0/2622333994 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1807343702 0 0) 0x7f30681a8fd0 con 0x7f3068107760 2024-04-14T15:23:33.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.370+0000 7f30657fa700 1 -- 172.21.15.90:0/2622333994 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f30681a7df0 con 0x7f3068107760 2024-04-14T15:23:33.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.370+0000 7f30657fa700 1 -- 172.21.15.90:0/2622333994 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 115877159 0 0) 0x7f30681aa1b0 con 0x7f306810a610 2024-04-14T15:23:33.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.370+0000 7f30657fa700 1 -- 172.21.15.90:0/2622333994 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f30681a8fd0 con 0x7f306810a610 2024-04-14T15:23:33.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.370+0000 7f30657fa700 1 -- 172.21.15.90:0/2622333994 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1402435677 0 0) 0x7f30681a7df0 con 0x7f3068107760 2024-04-14T15:23:33.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.370+0000 7f30657fa700 1 -- 172.21.15.90:0/2622333994 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f30681aa1b0 con 0x7f3068107760 2024-04-14T15:23:33.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.370+0000 7f30657fa700 1 -- 172.21.15.90:0/2622333994 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 308696735 0 0) 0x7f30500031f0 con 0x7f306810e1e0 2024-04-14T15:23:33.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.370+0000 7f30657fa700 1 -- 172.21.15.90:0/2622333994 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f30681a7df0 con 0x7f306810e1e0 2024-04-14T15:23:33.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.370+0000 7f30657fa700 1 -- 172.21.15.90:0/2622333994 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f30580018b0 con 0x7f3068107760 2024-04-14T15:23:33.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.370+0000 7f30657fa700 1 -- 172.21.15.90:0/2622333994 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f3060001950 con 0x7f306810e1e0 2024-04-14T15:23:33.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.370+0000 7f30657fa700 1 -- 172.21.15.90:0/2622333994 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2112326783 0 0) 0x7f30681a8fd0 con 0x7f306810a610 2024-04-14T15:23:33.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.370+0000 7f30657fa700 1 -- 172.21.15.90:0/2622333994 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f30500031f0 con 0x7f306810a610 2024-04-14T15:23:33.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.370+0000 7f30657fa700 1 -- 172.21.15.90:0/2622333994 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f305c0018b0 con 0x7f306810a610 2024-04-14T15:23:33.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.370+0000 7f30657fa700 1 -- 172.21.15.90:0/2622333994 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2803969740 0 0) 0x7f30681aa1b0 con 0x7f3068107760 2024-04-14T15:23:33.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.370+0000 7f30657fa700 1 -- 172.21.15.90:0/2622333994 >> v1:172.21.15.90:6790/0 conn(0x7f306810a610 legacy=0x7f306810c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:33.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.370+0000 7f30657fa700 1 -- 172.21.15.90:0/2622333994 >> v1:172.21.15.90:6789/0 conn(0x7f306810e1e0 legacy=0x7f30681a66d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:33.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.370+0000 7f30657fa700 1 -- 172.21.15.90:0/2622333994 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30681ab390 con 0x7f3068107760 2024-04-14T15:23:33.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.370+0000 7f306e489700 1 -- 172.21.15.90:0/2622333994 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f30681a9200 con 0x7f3068107760 2024-04-14T15:23:33.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.370+0000 7f306e489700 10 client.49654.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:23:33.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.370+0000 7f306e489700 1 -- 172.21.15.90:0/2622333994 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f30681a96c0 con 0x7f3068107760 2024-04-14T15:23:33.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.370+0000 7f30657fa700 1 -- 172.21.15.90:0/2622333994 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f3058003b40 con 0x7f3068107760 2024-04-14T15:23:33.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.370+0000 7f306e489700 1 librados: init done 2024-04-14T15:23:33.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.370+0000 7f30657fa700 1 -- 172.21.15.90:0/2622333994 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f30580049c0 con 0x7f3068107760 2024-04-14T15:23:33.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.371+0000 7f306e489700 1 -- 172.21.15.90:0/2622333994 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f3068102ea0 con 0x7f3068107760 2024-04-14T15:23:33.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.372+0000 7f30657fa700 1 -- 172.21.15.90:0/2622333994 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f30580036a0 con 0x7f3068107760 2024-04-14T15:23:33.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.372+0000 7f30657fa700 1 -- 172.21.15.90:0/2622333994 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(676..676 src has 467..676) ==== 8673+0+0 (unknown 478781498 0 0) 0x7f30580898e0 con 0x7f3068107760 2024-04-14T15:23:33.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.372+0000 7f30657fa700 10 client.49654.objecter ms_dispatch 0x7f3068001550 osd_map(676..676 src has 467..676) 2024-04-14T15:23:33.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.372+0000 7f30657fa700 3 client.49654.objecter handle_osd_map got epochs [676,676] > 0 2024-04-14T15:23:33.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.372+0000 7f30657fa700 3 client.49654.objecter handle_osd_map decoding full epoch 676 2024-04-14T15:23:33.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.372+0000 7f30657fa700 20 client.49654.objecter dump_active .. 0 homeless 2024-04-14T15:23:33.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.372+0000 7f30657fa700 1 -- 172.21.15.90:0/2622333994 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v676) ==== 54+0+16 (unknown 4183836731 0 2420000192) 0x7f305808a060 con 0x7f3068107760 2024-04-14T15:23:33.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.372+0000 7f30657fa700 10 client.49654.objecter ms_handle_connect 0x7f30500713c0 2024-04-14T15:23:33.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.373+0000 7f306e489700 10 client.49654.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:23:33.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.373+0000 7f306e489700 15 client.49654.objecter _session_command_op_assign -1 1 2024-04-14T15:23:33.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.373+0000 7f306e489700 20 client.49654.objecter _get_session s=0x7f306810d390 osd=0 3 2024-04-14T15:23:33.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.373+0000 7f306e489700 20 client.49654.objecter put_session s=0x7f306810d390 osd=0 3 2024-04-14T15:23:33.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.373+0000 7f306e489700 20 client.49654.objecter _get_session s=0x7f306810d390 osd=0 3 2024-04-14T15:23:33.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.373+0000 7f306e489700 15 client.49654.objecter _session_command_op_remove -1 1 2024-04-14T15:23:33.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.373+0000 7f306e489700 20 client.49654.objecter get_session s=0x7f306810d390 osd=0 3 2024-04-14T15:23:33.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.373+0000 7f306e489700 15 client.49654.objecter _session_command_op_assign 0 1 2024-04-14T15:23:33.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.373+0000 7f306e489700 20 client.49654.objecter put_session s=0x7f306810d390 osd=0 4 2024-04-14T15:23:33.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.373+0000 7f306e489700 10 client.49654.objecter _send_command 1 2024-04-14T15:23:33.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.373+0000 7f306e489700 1 -- 172.21.15.90:0/2622333994 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f30680693d0 con 0x7f30681aa380 2024-04-14T15:23:33.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.373+0000 7f30657fa700 10 client.49654.objecter ms_handle_connect 0x7f30681aa380 2024-04-14T15:23:33.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.374+0000 7f30657fa700 1 -- 172.21.15.90:0/2622333994 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f30680693d0 con 0x7f30681aa380 2024-04-14T15:23:33.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.374+0000 7f30657fa700 10 client.49654.objecter ms_dispatch 0x7f3068001550 command_reply(tid 1: 0 ) 2024-04-14T15:23:33.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.374+0000 7f30657fa700 10 client.49654.objecter _finish_command 1 = system:0 2024-04-14T15:23:33.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.374+0000 7f30657fa700 20 client.49654.objecter put_session s=0x7f306810d390 osd=0 4 2024-04-14T15:23:33.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.374+0000 7f30657fa700 15 client.49654.objecter _session_command_op_remove 0 1 2024-04-14T15:23:33.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.395+0000 7f306e489700 10 client.49654.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:23:33.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.395+0000 7f306e489700 15 client.49654.objecter _session_command_op_assign -1 2 2024-04-14T15:23:33.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.395+0000 7f306e489700 20 client.49654.objecter _get_session s=0x7f306810d390 osd=0 3 2024-04-14T15:23:33.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.395+0000 7f306e489700 20 client.49654.objecter put_session s=0x7f306810d390 osd=0 3 2024-04-14T15:23:33.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.395+0000 7f306e489700 20 client.49654.objecter _get_session s=0x7f306810d390 osd=0 3 2024-04-14T15:23:33.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.395+0000 7f306e489700 15 client.49654.objecter _session_command_op_remove -1 2 2024-04-14T15:23:33.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.395+0000 7f306e489700 20 client.49654.objecter get_session s=0x7f306810d390 osd=0 3 2024-04-14T15:23:33.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.395+0000 7f306e489700 15 client.49654.objecter _session_command_op_assign 0 2 2024-04-14T15:23:33.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.395+0000 7f306e489700 20 client.49654.objecter put_session s=0x7f306810d390 osd=0 4 2024-04-14T15:23:33.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.395+0000 7f306e489700 10 client.49654.objecter _send_command 2 2024-04-14T15:23:33.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.395+0000 7f306e489700 1 -- 172.21.15.90:0/2622333994 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f30681aae70 con 0x7f30681aa380 2024-04-14T15:23:33.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.398+0000 7f30657fa700 1 -- 172.21.15.90:0/2622333994 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f30681aae70 con 0x7f30681aa380 2024-04-14T15:23:33.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.398+0000 7f30657fa700 10 client.49654.objecter ms_dispatch 0x7f3068001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:23:33.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.398+0000 7f30657fa700 10 client.49654.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:23:33.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.398+0000 7f30657fa700 20 client.49654.objecter put_session s=0x7f306810d390 osd=0 4 2024-04-14T15:23:33.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.398+0000 7f30657fa700 15 client.49654.objecter _session_command_op_remove 0 2 2024-04-14T15:23:33.399 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:23:33.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.398+0000 7f306e489700 10 client.49654.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:23:33.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.398+0000 7f306e489700 15 client.49654.objecter _session_command_op_assign -1 3 2024-04-14T15:23:33.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.398+0000 7f306e489700 20 client.49654.objecter _get_session s=0x7f30681085e0 osd=1 3 2024-04-14T15:23:33.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.398+0000 7f306e489700 20 client.49654.objecter put_session s=0x7f30681085e0 osd=1 3 2024-04-14T15:23:33.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.398+0000 7f306e489700 20 client.49654.objecter _get_session s=0x7f30681085e0 osd=1 3 2024-04-14T15:23:33.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.398+0000 7f306e489700 15 client.49654.objecter _session_command_op_remove -1 3 2024-04-14T15:23:33.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.398+0000 7f306e489700 20 client.49654.objecter get_session s=0x7f30681085e0 osd=1 3 2024-04-14T15:23:33.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.398+0000 7f306e489700 15 client.49654.objecter _session_command_op_assign 1 3 2024-04-14T15:23:33.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.398+0000 7f306e489700 20 client.49654.objecter put_session s=0x7f30681085e0 osd=1 4 2024-04-14T15:23:33.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.398+0000 7f306e489700 10 client.49654.objecter _send_command 3 2024-04-14T15:23:33.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.398+0000 7f306e489700 1 -- 172.21.15.90:0/2622333994 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f30681090e0 con 0x7f3068108c60 2024-04-14T15:23:33.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.398+0000 7f30657fa700 10 client.49654.objecter ms_handle_connect 0x7f3068108c60 2024-04-14T15:23:33.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.399+0000 7f30657fa700 1 -- 172.21.15.90:0/2622333994 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f30681090e0 con 0x7f3068108c60 2024-04-14T15:23:33.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.400+0000 7f30657fa700 10 client.49654.objecter ms_dispatch 0x7f3068001550 command_reply(tid 3: 0 ) 2024-04-14T15:23:33.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.400+0000 7f30657fa700 10 client.49654.objecter _finish_command 3 = system:0 2024-04-14T15:23:33.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.400+0000 7f30657fa700 20 client.49654.objecter put_session s=0x7f30681085e0 osd=1 4 2024-04-14T15:23:33.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.400+0000 7f30657fa700 15 client.49654.objecter _session_command_op_remove 1 3 2024-04-14T15:23:33.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.417+0000 7f306e489700 10 client.49654.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:23:33.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.417+0000 7f306e489700 15 client.49654.objecter _session_command_op_assign -1 4 2024-04-14T15:23:33.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.417+0000 7f306e489700 20 client.49654.objecter _get_session s=0x7f30681085e0 osd=1 3 2024-04-14T15:23:33.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.417+0000 7f306e489700 20 client.49654.objecter put_session s=0x7f30681085e0 osd=1 3 2024-04-14T15:23:33.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.417+0000 7f306e489700 20 client.49654.objecter _get_session s=0x7f30681085e0 osd=1 3 2024-04-14T15:23:33.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.417+0000 7f306e489700 15 client.49654.objecter _session_command_op_remove -1 4 2024-04-14T15:23:33.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.417+0000 7f306e489700 20 client.49654.objecter get_session s=0x7f30681085e0 osd=1 3 2024-04-14T15:23:33.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.417+0000 7f306e489700 15 client.49654.objecter _session_command_op_assign 1 4 2024-04-14T15:23:33.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.417+0000 7f306e489700 20 client.49654.objecter put_session s=0x7f30681085e0 osd=1 4 2024-04-14T15:23:33.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.417+0000 7f306e489700 10 client.49654.objecter _send_command 4 2024-04-14T15:23:33.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.417+0000 7f306e489700 1 -- 172.21.15.90:0/2622333994 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f30681094e0 con 0x7f3068108c60 2024-04-14T15:23:33.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.419+0000 7f30657fa700 1 -- 172.21.15.90:0/2622333994 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f30681094e0 con 0x7f3068108c60 2024-04-14T15:23:33.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.419+0000 7f30657fa700 10 client.49654.objecter ms_dispatch 0x7f3068001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:23:33.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.419+0000 7f30657fa700 10 client.49654.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:23:33.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.419+0000 7f30657fa700 20 client.49654.objecter put_session s=0x7f30681085e0 osd=1 4 2024-04-14T15:23:33.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.419+0000 7f30657fa700 15 client.49654.objecter _session_command_op_remove 1 4 2024-04-14T15:23:33.420 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:23:33.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.420+0000 7f306e489700 10 client.49654.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:23:33.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.420+0000 7f306e489700 15 client.49654.objecter _session_command_op_assign -1 5 2024-04-14T15:23:33.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.420+0000 7f306e489700 20 client.49654.objecter _get_session s=0x7f30681000b0 osd=2 3 2024-04-14T15:23:33.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.420+0000 7f306e489700 20 client.49654.objecter put_session s=0x7f30681000b0 osd=2 3 2024-04-14T15:23:33.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.420+0000 7f306e489700 20 client.49654.objecter _get_session s=0x7f30681000b0 osd=2 3 2024-04-14T15:23:33.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.420+0000 7f306e489700 15 client.49654.objecter _session_command_op_remove -1 5 2024-04-14T15:23:33.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.420+0000 7f306e489700 20 client.49654.objecter get_session s=0x7f30681000b0 osd=2 3 2024-04-14T15:23:33.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.420+0000 7f306e489700 15 client.49654.objecter _session_command_op_assign 2 5 2024-04-14T15:23:33.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.420+0000 7f306e489700 20 client.49654.objecter put_session s=0x7f30681000b0 osd=2 4 2024-04-14T15:23:33.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.420+0000 7f306e489700 10 client.49654.objecter _send_command 5 2024-04-14T15:23:33.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.420+0000 7f306e489700 1 -- 172.21.15.90:0/2622333994 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f3068063bd0 con 0x7f3068100220 2024-04-14T15:23:33.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.420+0000 7f30657fa700 10 client.49654.objecter ms_handle_connect 0x7f3068100220 2024-04-14T15:23:33.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.421+0000 7f30657fa700 1 -- 172.21.15.90:0/2622333994 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f3068063bd0 con 0x7f3068100220 2024-04-14T15:23:33.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.421+0000 7f30657fa700 10 client.49654.objecter ms_dispatch 0x7f3068001550 command_reply(tid 5: 0 ) 2024-04-14T15:23:33.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.421+0000 7f30657fa700 10 client.49654.objecter _finish_command 5 = system:0 2024-04-14T15:23:33.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.421+0000 7f30657fa700 20 client.49654.objecter put_session s=0x7f30681000b0 osd=2 4 2024-04-14T15:23:33.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.421+0000 7f30657fa700 15 client.49654.objecter _session_command_op_remove 2 5 2024-04-14T15:23:33.428 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:33.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.438+0000 7f306e489700 10 client.49654.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:23:33.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.438+0000 7f306e489700 15 client.49654.objecter _session_command_op_assign -1 6 2024-04-14T15:23:33.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.438+0000 7f306e489700 20 client.49654.objecter _get_session s=0x7f30681000b0 osd=2 3 2024-04-14T15:23:33.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.438+0000 7f306e489700 20 client.49654.objecter put_session s=0x7f30681000b0 osd=2 3 2024-04-14T15:23:33.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.438+0000 7f306e489700 20 client.49654.objecter _get_session s=0x7f30681000b0 osd=2 3 2024-04-14T15:23:33.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.438+0000 7f306e489700 15 client.49654.objecter _session_command_op_remove -1 6 2024-04-14T15:23:33.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.438+0000 7f306e489700 20 client.49654.objecter get_session s=0x7f30681000b0 osd=2 3 2024-04-14T15:23:33.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.438+0000 7f306e489700 15 client.49654.objecter _session_command_op_assign 2 6 2024-04-14T15:23:33.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.438+0000 7f306e489700 20 client.49654.objecter put_session s=0x7f30681000b0 osd=2 4 2024-04-14T15:23:33.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.438+0000 7f306e489700 10 client.49654.objecter _send_command 6 2024-04-14T15:23:33.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.438+0000 7f306e489700 1 -- 172.21.15.90:0/2622333994 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f30681b3ef0 con 0x7f3068100220 2024-04-14T15:23:33.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.439+0000 7f30657fa700 1 -- 172.21.15.90:0/2622333994 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f30681b3ef0 con 0x7f3068100220 2024-04-14T15:23:33.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.439+0000 7f30657fa700 10 client.49654.objecter ms_dispatch 0x7f3068001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:23:33.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.439+0000 7f30657fa700 10 client.49654.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:23:33.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.439+0000 7f30657fa700 20 client.49654.objecter put_session s=0x7f30681000b0 osd=2 4 2024-04-14T15:23:33.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.439+0000 7f30657fa700 15 client.49654.objecter _session_command_op_remove 2 6 2024-04-14T15:23:33.440 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:23:33.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.439+0000 7f306e489700 10 client.49654.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:23:33.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.440+0000 7f306e489700 15 client.49654.objecter _session_command_op_assign -1 7 2024-04-14T15:23:33.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.440+0000 7f306e489700 20 client.49654.objecter _get_session s=0x7f3068063140 osd=3 3 2024-04-14T15:23:33.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.440+0000 7f306e489700 20 client.49654.objecter put_session s=0x7f3068063140 osd=3 3 2024-04-14T15:23:33.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.440+0000 7f306e489700 20 client.49654.objecter _get_session s=0x7f3068063140 osd=3 3 2024-04-14T15:23:33.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.440+0000 7f306e489700 15 client.49654.objecter _session_command_op_remove -1 7 2024-04-14T15:23:33.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.440+0000 7f306e489700 20 client.49654.objecter get_session s=0x7f3068063140 osd=3 3 2024-04-14T15:23:33.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.440+0000 7f306e489700 15 client.49654.objecter _session_command_op_assign 3 7 2024-04-14T15:23:33.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.440+0000 7f306e489700 20 client.49654.objecter put_session s=0x7f3068063140 osd=3 4 2024-04-14T15:23:33.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.440+0000 7f306e489700 10 client.49654.objecter _send_command 7 2024-04-14T15:23:33.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.440+0000 7f306e489700 1 -- 172.21.15.90:0/2622333994 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f30681b6820 con 0x7f30680632b0 2024-04-14T15:23:33.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.440+0000 7f30657fa700 10 client.49654.objecter ms_handle_connect 0x7f30680632b0 2024-04-14T15:23:33.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.441+0000 7f30657fa700 1 -- 172.21.15.90:0/2622333994 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f30681b6820 con 0x7f30680632b0 2024-04-14T15:23:33.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.441+0000 7f30657fa700 10 client.49654.objecter ms_dispatch 0x7f3068001550 command_reply(tid 7: 0 ) 2024-04-14T15:23:33.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.441+0000 7f30657fa700 10 client.49654.objecter _finish_command 7 = system:0 2024-04-14T15:23:33.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.441+0000 7f30657fa700 20 client.49654.objecter put_session s=0x7f3068063140 osd=3 4 2024-04-14T15:23:33.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.441+0000 7f30657fa700 15 client.49654.objecter _session_command_op_remove 3 7 2024-04-14T15:23:33.452 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:33.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:33.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:33.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.458+0000 7f306e489700 10 client.49654.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:23:33.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.458+0000 7f306e489700 15 client.49654.objecter _session_command_op_assign -1 8 2024-04-14T15:23:33.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.458+0000 7f306e489700 20 client.49654.objecter _get_session s=0x7f3068063140 osd=3 3 2024-04-14T15:23:33.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.458+0000 7f306e489700 20 client.49654.objecter put_session s=0x7f3068063140 osd=3 3 2024-04-14T15:23:33.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.458+0000 7f306e489700 20 client.49654.objecter _get_session s=0x7f3068063140 osd=3 3 2024-04-14T15:23:33.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.458+0000 7f306e489700 15 client.49654.objecter _session_command_op_remove -1 8 2024-04-14T15:23:33.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.458+0000 7f306e489700 20 client.49654.objecter get_session s=0x7f3068063140 osd=3 3 2024-04-14T15:23:33.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.458+0000 7f306e489700 15 client.49654.objecter _session_command_op_assign 3 8 2024-04-14T15:23:33.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.458+0000 7f306e489700 20 client.49654.objecter put_session s=0x7f3068063140 osd=3 4 2024-04-14T15:23:33.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.458+0000 7f306e489700 10 client.49654.objecter _send_command 8 2024-04-14T15:23:33.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.458+0000 7f306e489700 1 -- 172.21.15.90:0/2622333994 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f30681b0570 con 0x7f30680632b0 2024-04-14T15:23:33.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.459+0000 7f30657fa700 1 -- 172.21.15.90:0/2622333994 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f30681b0570 con 0x7f30680632b0 2024-04-14T15:23:33.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.459+0000 7f30657fa700 10 client.49654.objecter ms_dispatch 0x7f3068001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:23:33.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.459+0000 7f30657fa700 10 client.49654.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:23:33.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.459+0000 7f30657fa700 20 client.49654.objecter put_session s=0x7f3068063140 osd=3 4 2024-04-14T15:23:33.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.459+0000 7f30657fa700 15 client.49654.objecter _session_command_op_remove 3 8 2024-04-14T15:23:33.461 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:23:33.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.460+0000 7f306e489700 10 client.49654.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:23:33.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.460+0000 7f306e489700 15 client.49654.objecter _session_command_op_assign -1 9 2024-04-14T15:23:33.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.460+0000 7f306e489700 20 client.49654.objecter _get_session s=0x7f30681b0570 osd=4 3 2024-04-14T15:23:33.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.460+0000 7f306e489700 20 client.49654.objecter put_session s=0x7f30681b0570 osd=4 3 2024-04-14T15:23:33.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.460+0000 7f306e489700 20 client.49654.objecter _get_session s=0x7f30681b0570 osd=4 3 2024-04-14T15:23:33.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.460+0000 7f306e489700 15 client.49654.objecter _session_command_op_remove -1 9 2024-04-14T15:23:33.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.460+0000 7f306e489700 20 client.49654.objecter get_session s=0x7f30681b0570 osd=4 3 2024-04-14T15:23:33.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.460+0000 7f306e489700 15 client.49654.objecter _session_command_op_assign 4 9 2024-04-14T15:23:33.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.460+0000 7f306e489700 20 client.49654.objecter put_session s=0x7f30681b0570 osd=4 4 2024-04-14T15:23:33.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.460+0000 7f306e489700 10 client.49654.objecter _send_command 9 2024-04-14T15:23:33.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.460+0000 7f306e489700 1 -- 172.21.15.90:0/2622333994 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f30681beec0 con 0x7f30681bb340 2024-04-14T15:23:33.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.460+0000 7f30657fa700 10 client.49654.objecter ms_handle_connect 0x7f30681bb340 2024-04-14T15:23:33.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.461+0000 7f30657fa700 1 -- 172.21.15.90:0/2622333994 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f30681beec0 con 0x7f30681bb340 2024-04-14T15:23:33.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.461+0000 7f30657fa700 10 client.49654.objecter ms_dispatch 0x7f3068001550 command_reply(tid 9: 0 ) 2024-04-14T15:23:33.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.461+0000 7f30657fa700 10 client.49654.objecter _finish_command 9 = system:0 2024-04-14T15:23:33.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.461+0000 7f30657fa700 20 client.49654.objecter put_session s=0x7f30681b0570 osd=4 4 2024-04-14T15:23:33.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.461+0000 7f30657fa700 15 client.49654.objecter _session_command_op_remove 4 9 2024-04-14T15:23:33.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.477+0000 7f306e489700 10 client.49654.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:23:33.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.477+0000 7f306e489700 15 client.49654.objecter _session_command_op_assign -1 10 2024-04-14T15:23:33.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.477+0000 7f306e489700 20 client.49654.objecter _get_session s=0x7f30681b0570 osd=4 3 2024-04-14T15:23:33.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.477+0000 7f306e489700 20 client.49654.objecter put_session s=0x7f30681b0570 osd=4 3 2024-04-14T15:23:33.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.477+0000 7f306e489700 20 client.49654.objecter _get_session s=0x7f30681b0570 osd=4 3 2024-04-14T15:23:33.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.477+0000 7f306e489700 15 client.49654.objecter _session_command_op_remove -1 10 2024-04-14T15:23:33.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.477+0000 7f306e489700 20 client.49654.objecter get_session s=0x7f30681b0570 osd=4 3 2024-04-14T15:23:33.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.477+0000 7f306e489700 15 client.49654.objecter _session_command_op_assign 4 10 2024-04-14T15:23:33.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.477+0000 7f306e489700 20 client.49654.objecter put_session s=0x7f30681b0570 osd=4 4 2024-04-14T15:23:33.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.477+0000 7f306e489700 10 client.49654.objecter _send_command 10 2024-04-14T15:23:33.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.477+0000 7f306e489700 1 -- 172.21.15.90:0/2622333994 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f30681c08a0 con 0x7f30681bb340 2024-04-14T15:23:33.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.479+0000 7f30657fa700 1 -- 172.21.15.90:0/2622333994 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f30681c08a0 con 0x7f30681bb340 2024-04-14T15:23:33.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.479+0000 7f30657fa700 10 client.49654.objecter ms_dispatch 0x7f3068001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:23:33.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.479+0000 7f30657fa700 10 client.49654.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:23:33.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.479+0000 7f30657fa700 20 client.49654.objecter put_session s=0x7f30681b0570 osd=4 4 2024-04-14T15:23:33.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.479+0000 7f30657fa700 15 client.49654.objecter _session_command_op_remove 4 10 2024-04-14T15:23:33.480 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:23:33.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.480+0000 7f306e489700 10 client.49654.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:23:33.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.480+0000 7f306e489700 15 client.49654.objecter _session_command_op_assign -1 11 2024-04-14T15:23:33.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.480+0000 7f306e489700 20 client.49654.objecter _get_session s=0x7f30681c08a0 osd=5 3 2024-04-14T15:23:33.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.480+0000 7f306e489700 20 client.49654.objecter put_session s=0x7f30681c08a0 osd=5 3 2024-04-14T15:23:33.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.480+0000 7f306e489700 20 client.49654.objecter _get_session s=0x7f30681c08a0 osd=5 3 2024-04-14T15:23:33.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.480+0000 7f306e489700 15 client.49654.objecter _session_command_op_remove -1 11 2024-04-14T15:23:33.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.480+0000 7f306e489700 20 client.49654.objecter get_session s=0x7f30681c08a0 osd=5 3 2024-04-14T15:23:33.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.480+0000 7f306e489700 15 client.49654.objecter _session_command_op_assign 5 11 2024-04-14T15:23:33.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.480+0000 7f306e489700 20 client.49654.objecter put_session s=0x7f30681c08a0 osd=5 4 2024-04-14T15:23:33.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.480+0000 7f306e489700 10 client.49654.objecter _send_command 11 2024-04-14T15:23:33.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.480+0000 7f306e489700 1 -- 172.21.15.90:0/2622333994 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f30681c1400 con 0x7f30681c0fc0 2024-04-14T15:23:33.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.480+0000 7f30657fa700 10 client.49654.objecter ms_handle_connect 0x7f30681c0fc0 2024-04-14T15:23:33.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.481+0000 7f30657fa700 1 -- 172.21.15.90:0/2622333994 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f30681c1400 con 0x7f30681c0fc0 2024-04-14T15:23:33.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.481+0000 7f30657fa700 10 client.49654.objecter ms_dispatch 0x7f3068001550 command_reply(tid 11: 0 ) 2024-04-14T15:23:33.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.481+0000 7f30657fa700 10 client.49654.objecter _finish_command 11 = system:0 2024-04-14T15:23:33.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.481+0000 7f30657fa700 20 client.49654.objecter put_session s=0x7f30681c08a0 osd=5 4 2024-04-14T15:23:33.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.481+0000 7f30657fa700 15 client.49654.objecter _session_command_op_remove 5 11 2024-04-14T15:23:33.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.499+0000 7f306e489700 10 client.49654.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:23:33.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.499+0000 7f306e489700 15 client.49654.objecter _session_command_op_assign -1 12 2024-04-14T15:23:33.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.499+0000 7f306e489700 20 client.49654.objecter _get_session s=0x7f30681c08a0 osd=5 3 2024-04-14T15:23:33.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.499+0000 7f306e489700 20 client.49654.objecter put_session s=0x7f30681c08a0 osd=5 3 2024-04-14T15:23:33.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.499+0000 7f306e489700 20 client.49654.objecter _get_session s=0x7f30681c08a0 osd=5 3 2024-04-14T15:23:33.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.499+0000 7f306e489700 15 client.49654.objecter _session_command_op_remove -1 12 2024-04-14T15:23:33.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.499+0000 7f306e489700 20 client.49654.objecter get_session s=0x7f30681c08a0 osd=5 3 2024-04-14T15:23:33.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.499+0000 7f306e489700 15 client.49654.objecter _session_command_op_assign 5 12 2024-04-14T15:23:33.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.499+0000 7f306e489700 20 client.49654.objecter put_session s=0x7f30681c08a0 osd=5 4 2024-04-14T15:23:33.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.499+0000 7f306e489700 10 client.49654.objecter _send_command 12 2024-04-14T15:23:33.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.499+0000 7f306e489700 1 -- 172.21.15.90:0/2622333994 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f30681c1630 con 0x7f30681c0fc0 2024-04-14T15:23:33.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.501+0000 7f30657fa700 1 -- 172.21.15.90:0/2622333994 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f30681c1630 con 0x7f30681c0fc0 2024-04-14T15:23:33.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.501+0000 7f30657fa700 10 client.49654.objecter ms_dispatch 0x7f3068001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:23:33.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.501+0000 7f30657fa700 10 client.49654.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:23:33.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.501+0000 7f30657fa700 20 client.49654.objecter put_session s=0x7f30681c08a0 osd=5 4 2024-04-14T15:23:33.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.501+0000 7f30657fa700 15 client.49654.objecter _session_command_op_remove 5 12 2024-04-14T15:23:33.502 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:23:33.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.501+0000 7f306e489700 10 client.49654.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:23:33.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.501+0000 7f306e489700 15 client.49654.objecter _session_command_op_assign -1 13 2024-04-14T15:23:33.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.501+0000 7f306e489700 20 client.49654.objecter _get_session s=0x7f30681c1630 osd=6 3 2024-04-14T15:23:33.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.501+0000 7f306e489700 20 client.49654.objecter put_session s=0x7f30681c1630 osd=6 3 2024-04-14T15:23:33.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.501+0000 7f306e489700 20 client.49654.objecter _get_session s=0x7f30681c1630 osd=6 3 2024-04-14T15:23:33.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.501+0000 7f306e489700 15 client.49654.objecter _session_command_op_remove -1 13 2024-04-14T15:23:33.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.501+0000 7f306e489700 20 client.49654.objecter get_session s=0x7f30681c1630 osd=6 3 2024-04-14T15:23:33.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.501+0000 7f306e489700 15 client.49654.objecter _session_command_op_assign 6 13 2024-04-14T15:23:33.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.501+0000 7f306e489700 20 client.49654.objecter put_session s=0x7f30681c1630 osd=6 4 2024-04-14T15:23:33.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.501+0000 7f306e489700 10 client.49654.objecter _send_command 13 2024-04-14T15:23:33.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.501+0000 7f306e489700 1 -- 172.21.15.90:0/2622333994 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f30681c1890 con 0x7f30681c3fb0 2024-04-14T15:23:33.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.501+0000 7f30657fa700 10 client.49654.objecter ms_handle_connect 0x7f30681c3fb0 2024-04-14T15:23:33.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.502+0000 7f30657fa700 1 -- 172.21.15.90:0/2622333994 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f30681c1890 con 0x7f30681c3fb0 2024-04-14T15:23:33.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.502+0000 7f30657fa700 10 client.49654.objecter ms_dispatch 0x7f3068001550 command_reply(tid 13: 0 ) 2024-04-14T15:23:33.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.502+0000 7f30657fa700 10 client.49654.objecter _finish_command 13 = system:0 2024-04-14T15:23:33.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.502+0000 7f30657fa700 20 client.49654.objecter put_session s=0x7f30681c1630 osd=6 4 2024-04-14T15:23:33.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.502+0000 7f30657fa700 15 client.49654.objecter _session_command_op_remove 6 13 2024-04-14T15:23:33.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.520+0000 7f306e489700 10 client.49654.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:23:33.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.520+0000 7f306e489700 15 client.49654.objecter _session_command_op_assign -1 14 2024-04-14T15:23:33.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.520+0000 7f306e489700 20 client.49654.objecter _get_session s=0x7f30681c1630 osd=6 3 2024-04-14T15:23:33.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.520+0000 7f306e489700 20 client.49654.objecter put_session s=0x7f30681c1630 osd=6 3 2024-04-14T15:23:33.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.520+0000 7f306e489700 20 client.49654.objecter _get_session s=0x7f30681c1630 osd=6 3 2024-04-14T15:23:33.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.520+0000 7f306e489700 15 client.49654.objecter _session_command_op_remove -1 14 2024-04-14T15:23:33.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.520+0000 7f306e489700 20 client.49654.objecter get_session s=0x7f30681c1630 osd=6 3 2024-04-14T15:23:33.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.520+0000 7f306e489700 15 client.49654.objecter _session_command_op_assign 6 14 2024-04-14T15:23:33.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.520+0000 7f306e489700 20 client.49654.objecter put_session s=0x7f30681c1630 osd=6 4 2024-04-14T15:23:33.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.520+0000 7f306e489700 10 client.49654.objecter _send_command 14 2024-04-14T15:23:33.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.520+0000 7f306e489700 1 -- 172.21.15.90:0/2622333994 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f30681c7ac0 con 0x7f30681c3fb0 2024-04-14T15:23:33.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.522+0000 7f30657fa700 1 -- 172.21.15.90:0/2622333994 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f30681c7ac0 con 0x7f30681c3fb0 2024-04-14T15:23:33.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.522+0000 7f30657fa700 10 client.49654.objecter ms_dispatch 0x7f3068001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:23:33.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.522+0000 7f30657fa700 10 client.49654.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:23:33.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.522+0000 7f30657fa700 20 client.49654.objecter put_session s=0x7f30681c1630 osd=6 4 2024-04-14T15:23:33.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.522+0000 7f30657fa700 15 client.49654.objecter _session_command_op_remove 6 14 2024-04-14T15:23:33.523 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:23:33.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.522+0000 7f306e489700 10 client.49654.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:23:33.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.522+0000 7f306e489700 15 client.49654.objecter _session_command_op_assign -1 15 2024-04-14T15:23:33.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.522+0000 7f306e489700 20 client.49654.objecter _get_session s=0x7f30681c7ac0 osd=7 3 2024-04-14T15:23:33.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.522+0000 7f306e489700 20 client.49654.objecter put_session s=0x7f30681c7ac0 osd=7 3 2024-04-14T15:23:33.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.522+0000 7f306e489700 20 client.49654.objecter _get_session s=0x7f30681c7ac0 osd=7 3 2024-04-14T15:23:33.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.522+0000 7f306e489700 15 client.49654.objecter _session_command_op_remove -1 15 2024-04-14T15:23:33.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.522+0000 7f306e489700 20 client.49654.objecter get_session s=0x7f30681c7ac0 osd=7 3 2024-04-14T15:23:33.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.522+0000 7f306e489700 15 client.49654.objecter _session_command_op_assign 7 15 2024-04-14T15:23:33.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.522+0000 7f306e489700 20 client.49654.objecter put_session s=0x7f30681c7ac0 osd=7 4 2024-04-14T15:23:33.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.522+0000 7f306e489700 10 client.49654.objecter _send_command 15 2024-04-14T15:23:33.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.522+0000 7f306e489700 1 -- 172.21.15.90:0/2622333994 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f30681c8df0 con 0x7f30681c81e0 2024-04-14T15:23:33.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.523+0000 7f30657fa700 10 client.49654.objecter ms_handle_connect 0x7f30681c81e0 2024-04-14T15:23:33.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.523+0000 7f30657fa700 1 -- 172.21.15.90:0/2622333994 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f30681c8df0 con 0x7f30681c81e0 2024-04-14T15:23:33.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.523+0000 7f30657fa700 10 client.49654.objecter ms_dispatch 0x7f3068001550 command_reply(tid 15: 0 ) 2024-04-14T15:23:33.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.523+0000 7f30657fa700 10 client.49654.objecter _finish_command 15 = system:0 2024-04-14T15:23:33.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.523+0000 7f30657fa700 20 client.49654.objecter put_session s=0x7f30681c7ac0 osd=7 4 2024-04-14T15:23:33.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.523+0000 7f30657fa700 15 client.49654.objecter _session_command_op_remove 7 15 2024-04-14T15:23:33.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.540+0000 7f306e489700 10 client.49654.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:23:33.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.540+0000 7f306e489700 15 client.49654.objecter _session_command_op_assign -1 16 2024-04-14T15:23:33.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.540+0000 7f306e489700 20 client.49654.objecter _get_session s=0x7f30681c7ac0 osd=7 3 2024-04-14T15:23:33.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.540+0000 7f306e489700 20 client.49654.objecter put_session s=0x7f30681c7ac0 osd=7 3 2024-04-14T15:23:33.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.540+0000 7f306e489700 20 client.49654.objecter _get_session s=0x7f30681c7ac0 osd=7 3 2024-04-14T15:23:33.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.540+0000 7f306e489700 15 client.49654.objecter _session_command_op_remove -1 16 2024-04-14T15:23:33.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.540+0000 7f306e489700 20 client.49654.objecter get_session s=0x7f30681c7ac0 osd=7 3 2024-04-14T15:23:33.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.540+0000 7f306e489700 15 client.49654.objecter _session_command_op_assign 7 16 2024-04-14T15:23:33.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.540+0000 7f306e489700 20 client.49654.objecter put_session s=0x7f30681c7ac0 osd=7 4 2024-04-14T15:23:33.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.540+0000 7f306e489700 10 client.49654.objecter _send_command 16 2024-04-14T15:23:33.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.540+0000 7f306e489700 1 -- 172.21.15.90:0/2622333994 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f30681c8df0 con 0x7f30681c81e0 2024-04-14T15:23:33.542 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:23:33.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.542+0000 7f30657fa700 1 -- 172.21.15.90:0/2622333994 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f30681c8df0 con 0x7f30681c81e0 2024-04-14T15:23:33.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.542+0000 7f30657fa700 10 client.49654.objecter ms_dispatch 0x7f3068001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:23:33.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.542+0000 7f30657fa700 10 client.49654.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:23:33.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.542+0000 7f30657fa700 20 client.49654.objecter put_session s=0x7f30681c7ac0 osd=7 4 2024-04-14T15:23:33.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.542+0000 7f30657fa700 15 client.49654.objecter _session_command_op_remove 7 16 2024-04-14T15:23:33.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.542+0000 7f306e489700 10 librados: watch_flush enter 2024-04-14T15:23:33.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.542+0000 7f306e489700 10 librados: watch_flush exit 2024-04-14T15:23:33.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.542+0000 7f306e489700 10 client.49654.objecter close_session for osd.0 2024-04-14T15:23:33.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.542+0000 7f306e489700 1 -- 172.21.15.90:0/2622333994 >> v1:172.21.15.90:6801/3761806885 conn(0x7f30681aa380 legacy=0x7f30681aa760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:33.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.542+0000 7f306e489700 20 client.49654.objecter put_session s=0x7f306810d390 osd=0 1 2024-04-14T15:23:33.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.542+0000 7f306e489700 10 client.49654.objecter close_session for osd.1 2024-04-14T15:23:33.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.542+0000 7f306e489700 1 -- 172.21.15.90:0/2622333994 >> v1:172.21.15.90:6805/4170361699 conn(0x7f3068108c60 legacy=0x7f306802ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:33.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.542+0000 7f306e489700 20 client.49654.objecter put_session s=0x7f30681085e0 osd=1 1 2024-04-14T15:23:33.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.542+0000 7f306e489700 10 client.49654.objecter close_session for osd.2 2024-04-14T15:23:33.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.542+0000 7f306e489700 1 -- 172.21.15.90:0/2622333994 >> v1:172.21.15.90:6809/1686184251 conn(0x7f3068100220 legacy=0x7f30681b27d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:33.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.542+0000 7f306ca26700 1 -- 172.21.15.90:0/2622333994 reap_dead start 2024-04-14T15:23:33.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.542+0000 7f306e489700 20 client.49654.objecter put_session s=0x7f30681000b0 osd=2 1 2024-04-14T15:23:33.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.543+0000 7f306e489700 10 client.49654.objecter close_session for osd.3 2024-04-14T15:23:33.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.543+0000 7f306e489700 1 -- 172.21.15.90:0/2622333994 >> v1:172.21.15.90:6813/2270166774 conn(0x7f30680632b0 legacy=0x7f30681b6110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:33.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.543+0000 7f306e489700 20 client.49654.objecter put_session s=0x7f3068063140 osd=3 1 2024-04-14T15:23:33.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.543+0000 7f306e489700 10 client.49654.objecter close_session for osd.4 2024-04-14T15:23:33.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.543+0000 7f306e489700 1 -- 172.21.15.90:0/2622333994 >> v1:172.21.15.92:6800/3887549733 conn(0x7f30681bb340 legacy=0x7f30681bd730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:33.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.543+0000 7f306e489700 20 client.49654.objecter put_session s=0x7f30681b0570 osd=4 1 2024-04-14T15:23:33.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.543+0000 7f306e489700 10 client.49654.objecter close_session for osd.5 2024-04-14T15:23:33.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.543+0000 7f306e489700 1 -- 172.21.15.90:0/2622333994 >> v1:172.21.15.92:6804/3244561440 conn(0x7f30681c0fc0 legacy=0x7f30681bf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:33.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.543+0000 7f306e489700 20 client.49654.objecter put_session s=0x7f30681c08a0 osd=5 1 2024-04-14T15:23:33.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.543+0000 7f306e489700 10 client.49654.objecter close_session for osd.6 2024-04-14T15:23:33.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.543+0000 7f306e489700 1 -- 172.21.15.90:0/2622333994 >> v1:172.21.15.92:6808/636125963 conn(0x7f30681c3fb0 legacy=0x7f30681c63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:33.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.543+0000 7f306e489700 20 client.49654.objecter put_session s=0x7f30681c1630 osd=6 1 2024-04-14T15:23:33.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.543+0000 7f306e489700 10 client.49654.objecter close_session for osd.7 2024-04-14T15:23:33.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.543+0000 7f306e489700 1 -- 172.21.15.90:0/2622333994 >> v1:172.21.15.92:6812/1773706716 conn(0x7f30681c81e0 legacy=0x7f30681c8670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:33.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.543+0000 7f306e489700 20 client.49654.objecter put_session s=0x7f30681c7ac0 osd=7 1 2024-04-14T15:23:33.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.543+0000 7f306e489700 20 client.49654.objecter shutdown clearing up homeless session... 2024-04-14T15:23:33.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.543+0000 7f306e489700 10 client.49654.objecter successfully canceled tick 2024-04-14T15:23:33.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.543+0000 7f306ca26700 1 -- 172.21.15.90:0/2622333994 reap_dead start 2024-04-14T15:23:33.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.543+0000 7f306e489700 1 -- 172.21.15.90:0/2622333994 >> v1:172.21.15.90:6800/420935427 conn(0x7f30500713c0 legacy=0x7f3050073880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:33.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.543+0000 7f306e489700 1 -- 172.21.15.90:0/2622333994 >> v1:172.21.15.92:6789/0 conn(0x7f3068107760 legacy=0x7f306810be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:33.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.543+0000 7f306e489700 1 -- 172.21.15.90:0/2622333994 shutdown_connections 2024-04-14T15:23:33.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.543+0000 7f306e489700 1 -- 172.21.15.90:0/2622333994 >> 172.21.15.90:0/2622333994 conn(0x7f30680fd290 msgr2=0x7f3068111050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:23:33.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.543+0000 7f306e489700 1 -- 172.21.15.90:0/2622333994 shutdown_connections 2024-04-14T15:23:33.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.543+0000 7f306e489700 1 -- 172.21.15.90:0/2622333994 wait complete. 2024-04-14T15:23:33.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:33.543+0000 7f306e489700 1 librados: shutdown 2024-04-14T15:23:33.548 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:23:33.548 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:23:33.548 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:23:33.548 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:23:33.548 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:23:33.548 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:23:33.548 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:23:33.548 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:23:33.554 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:23:33.578 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:33.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:33.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:23:33.680 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:23:33.704 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:33.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:33.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:23:33.805 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:23:33.930 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:33.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:33.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:23:34.031 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:23:34.056 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:34.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:34.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:23:34.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:34 smithi092 ceph-mon[33966]: 6.1c scrub starts 2024-04-14T15:23:34.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:34 smithi092 ceph-mon[33966]: 6.1c scrub ok 2024-04-14T15:23:34.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:34 smithi092 ceph-mon[33966]: 11.18 scrub starts 2024-04-14T15:23:34.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:34 smithi092 ceph-mon[33966]: 11.18 scrub ok 2024-04-14T15:23:34.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:34 smithi092 ceph-mon[33966]: pgmap v995: 214 pgs: 214 active+clean; 582 KiB data, 590 MiB used, 625 GiB / 626 GiB avail; 160 KiB/s, 0 objects/s recovering 2024-04-14T15:23:34.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:34 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2622333994' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:23:34.158 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:34.174 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-14T15:23:34.175 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-14T15:23:34.175 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph tell osd.3 'trim stale osdmaps' 2024-04-14T15:23:34.181 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:34.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:34.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:34.283 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:34.306 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:34.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:34.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:34.408 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:23:34.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:34.432 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:34.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:23:34.441 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:23:33 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:23:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:23:34.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:34 smithi090 ceph-mon[41670]: 6.1c scrub starts 2024-04-14T15:23:34.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:34 smithi090 ceph-mon[41670]: 6.1c scrub ok 2024-04-14T15:23:34.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:34 smithi090 ceph-mon[41670]: 11.18 scrub starts 2024-04-14T15:23:34.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:34 smithi090 ceph-mon[41670]: 11.18 scrub ok 2024-04-14T15:23:34.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:34 smithi090 ceph-mon[41670]: pgmap v995: 214 pgs: 214 active+clean; 582 KiB data, 590 MiB used, 625 GiB / 626 GiB avail; 160 KiB/s, 0 objects/s recovering 2024-04-14T15:23:34.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:34 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2622333994' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:23:34.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:34 smithi090 ceph-mon[30399]: 6.1c scrub starts 2024-04-14T15:23:34.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:34 smithi090 ceph-mon[30399]: 6.1c scrub ok 2024-04-14T15:23:34.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:34 smithi090 ceph-mon[30399]: 11.18 scrub starts 2024-04-14T15:23:34.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:34 smithi090 ceph-mon[30399]: 11.18 scrub ok 2024-04-14T15:23:34.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:34 smithi090 ceph-mon[30399]: pgmap v995: 214 pgs: 214 active+clean; 582 KiB data, 590 MiB used, 625 GiB / 626 GiB avail; 160 KiB/s, 0 objects/s recovering 2024-04-14T15:23:34.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:34 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2622333994' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:23:34.533 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:23:34.556 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:34.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:34.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:23:34.658 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:23:34.682 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:34.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:34.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:23:34.783 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:23:34.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:34.808 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:34.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:23:34.909 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:23:34.935 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:34.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:34.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:23:35.037 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:23:35.062 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:35.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:35.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:23:35.164 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:23:35.173 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:35 smithi090 ceph-mon[30399]: 6.2 scrub starts 2024-04-14T15:23:35.173 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:35 smithi090 ceph-mon[30399]: 6.2 scrub ok 2024-04-14T15:23:35.173 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:35 smithi090 ceph-mon[30399]: 6.7 scrub starts 2024-04-14T15:23:35.174 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:35 smithi090 ceph-mon[30399]: 6.7 scrub ok 2024-04-14T15:23:35.174 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:35 smithi090 ceph-mon[30399]: 6.1 scrub starts 2024-04-14T15:23:35.174 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:35 smithi090 ceph-mon[30399]: 6.1 scrub ok 2024-04-14T15:23:35.174 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:35 smithi090 ceph-mon[41670]: 6.2 scrub starts 2024-04-14T15:23:35.174 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:35 smithi090 ceph-mon[41670]: 6.2 scrub ok 2024-04-14T15:23:35.174 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:35 smithi090 ceph-mon[41670]: 6.7 scrub starts 2024-04-14T15:23:35.174 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:35 smithi090 ceph-mon[41670]: 6.7 scrub ok 2024-04-14T15:23:35.174 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:35 smithi090 ceph-mon[41670]: 6.1 scrub starts 2024-04-14T15:23:35.174 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:35 smithi090 ceph-mon[41670]: 6.1 scrub ok 2024-04-14T15:23:35.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:35.190 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:35.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:23:35.291 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:23:35.315 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:35.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:35.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:23:35.367 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:35 smithi092 ceph-mon[33966]: 6.2 scrub starts 2024-04-14T15:23:35.367 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:35 smithi092 ceph-mon[33966]: 6.2 scrub ok 2024-04-14T15:23:35.367 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:35 smithi092 ceph-mon[33966]: 6.7 scrub starts 2024-04-14T15:23:35.367 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:35 smithi092 ceph-mon[33966]: 6.7 scrub ok 2024-04-14T15:23:35.367 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:35 smithi092 ceph-mon[33966]: 6.1 scrub starts 2024-04-14T15:23:35.367 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:35 smithi092 ceph-mon[33966]: 6.1 scrub ok 2024-04-14T15:23:35.417 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:23:35.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:35.422+0000 7f6165543700 1 -- 172.21.15.90:0/2414305211 >> v1:172.21.15.90:6789/0 conn(0x7f616010a740 legacy=0x7f616010cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:35.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:35.422+0000 7f6165543700 1 -- 172.21.15.90:0/2414305211 shutdown_connections 2024-04-14T15:23:35.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:35.422+0000 7f6165543700 1 -- 172.21.15.90:0/2414305211 >> 172.21.15.90:0/2414305211 conn(0x7f61600fd290 msgr2=0x7f61600ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:23:35.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:35.422+0000 7f6165543700 1 -- 172.21.15.90:0/2414305211 shutdown_connections 2024-04-14T15:23:35.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:35.422+0000 7f6165543700 1 -- 172.21.15.90:0/2414305211 wait complete. 2024-04-14T15:23:35.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:35.422+0000 7f6165543700 1 librados: starting msgr at 2024-04-14T15:23:35.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:35.422+0000 7f6165543700 1 librados: starting objecter 2024-04-14T15:23:35.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:35.423+0000 7f6165543700 1 Processor -- start 2024-04-14T15:23:35.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:35.423+0000 7f6165543700 1 -- start start 2024-04-14T15:23:35.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:35.423+0000 7f6165543700 1 librados: setting wanted keys 2024-04-14T15:23:35.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:35.423+0000 7f6165543700 1 librados: calling monclient init 2024-04-14T15:23:35.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:35.423+0000 7f6165543700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6160110600 con 0x7f6160107890 2024-04-14T15:23:35.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:35.423+0000 7f6165543700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6160110810 con 0x7f616010e310 2024-04-14T15:23:35.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:35.423+0000 7f6165543700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f61601ac370 con 0x7f616010a740 2024-04-14T15:23:35.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:35.423+0000 7f615effd700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f6160107890 0x7f616010e850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:44338/0 (socket says 172.21.15.90:44338) 2024-04-14T15:23:35.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:35.423+0000 7f615effd700 1 -- 172.21.15.90:0/1728517424 learned_addr learned my addr 172.21.15.90:0/1728517424 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:23:35.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:35.423+0000 7f615e7fc700 1 --1- 172.21.15.90:0/1728517424 >> v1:172.21.15.90:6790/0 conn(0x7f616010a740 0x7f616010ef60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:54544/0 (socket says 172.21.15.90:54544) 2024-04-14T15:23:35.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:35.423+0000 7f6147fff700 10 client.?.objecter ms_handle_connect 0x7f6160107890 2024-04-14T15:23:35.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:35.423+0000 7f6147fff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:23:35.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:35.423+0000 7f6147fff700 10 client.?.objecter ms_handle_connect 0x7f616010a740 2024-04-14T15:23:35.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:35.423+0000 7f6147fff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:23:35.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:35.423+0000 7f6147fff700 10 client.?.objecter ms_handle_connect 0x7f616010e310 2024-04-14T15:23:35.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:35.423+0000 7f6147fff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:23:35.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:35.424+0000 7f6147fff700 1 -- 172.21.15.90:0/1728517424 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2123322920 0 0) 0x7f6160110810 con 0x7f616010e310 2024-04-14T15:23:35.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:35.424+0000 7f6147fff700 1 -- 172.21.15.90:0/1728517424 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f61480035b0 con 0x7f616010e310 2024-04-14T15:23:35.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:35.424+0000 7f6147fff700 1 -- 172.21.15.90:0/1728517424 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3680404488 0 0) 0x7f61601ac370 con 0x7f616010a740 2024-04-14T15:23:35.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:35.424+0000 7f6147fff700 1 -- 172.21.15.90:0/1728517424 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6160110810 con 0x7f616010a740 2024-04-14T15:23:35.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:35.424+0000 7f6147fff700 1 -- 172.21.15.90:0/1728517424 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3094459222 0 0) 0x7f6160110600 con 0x7f6160107890 2024-04-14T15:23:35.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:35.424+0000 7f6147fff700 1 -- 172.21.15.90:0/1728517424 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f61601ac370 con 0x7f6160107890 2024-04-14T15:23:35.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:35.424+0000 7f6147fff700 1 -- 172.21.15.90:0/1728517424 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 784760507 0 0) 0x7f61480035b0 con 0x7f616010e310 2024-04-14T15:23:35.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:35.424+0000 7f6147fff700 1 -- 172.21.15.90:0/1728517424 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6160110600 con 0x7f616010e310 2024-04-14T15:23:35.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:35.424+0000 7f6147fff700 1 -- 172.21.15.90:0/1728517424 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f6158001bb0 con 0x7f616010e310 2024-04-14T15:23:35.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:35.424+0000 7f6147fff700 1 -- 172.21.15.90:0/1728517424 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4141280014 0 0) 0x7f6160110810 con 0x7f616010a740 2024-04-14T15:23:35.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:35.424+0000 7f6147fff700 1 -- 172.21.15.90:0/1728517424 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f61480035b0 con 0x7f616010a740 2024-04-14T15:23:35.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:35.424+0000 7f6147fff700 1 -- 172.21.15.90:0/1728517424 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f6154001370 con 0x7f616010a740 2024-04-14T15:23:35.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:35.424+0000 7f6147fff700 1 -- 172.21.15.90:0/1728517424 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3253163011 0 0) 0x7f61601ac370 con 0x7f6160107890 2024-04-14T15:23:35.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:35.424+0000 7f6147fff700 1 -- 172.21.15.90:0/1728517424 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6160110810 con 0x7f6160107890 2024-04-14T15:23:35.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:35.424+0000 7f6147fff700 1 -- 172.21.15.90:0/1728517424 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f61500017f0 con 0x7f6160107890 2024-04-14T15:23:35.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:35.424+0000 7f6147fff700 1 -- 172.21.15.90:0/1728517424 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2881529545 0 0) 0x7f6160110600 con 0x7f616010e310 2024-04-14T15:23:35.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:35.424+0000 7f6147fff700 1 -- 172.21.15.90:0/1728517424 >> v1:172.21.15.90:6790/0 conn(0x7f616010a740 legacy=0x7f616010ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:35.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:35.424+0000 7f6147fff700 1 -- 172.21.15.90:0/1728517424 >> v1:172.21.15.90:6789/0 conn(0x7f6160107890 legacy=0x7f616010e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:35.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:35.424+0000 7f6147fff700 1 -- 172.21.15.90:0/1728517424 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61601ad5b0 con 0x7f616010e310 2024-04-14T15:23:35.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:35.424+0000 7f6165543700 1 -- 172.21.15.90:0/1728517424 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f61601ac5a0 con 0x7f616010e310 2024-04-14T15:23:35.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:35.424+0000 7f6165543700 10 client.49660.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:23:35.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:35.424+0000 7f6165543700 1 -- 172.21.15.90:0/1728517424 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f61601aca60 con 0x7f616010e310 2024-04-14T15:23:35.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:35.424+0000 7f6165543700 1 librados: init done 2024-04-14T15:23:35.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:35.424+0000 7f6147fff700 1 -- 172.21.15.90:0/1728517424 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f6158003ea0 con 0x7f616010e310 2024-04-14T15:23:35.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:35.424+0000 7f6147fff700 1 -- 172.21.15.90:0/1728517424 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f6158004d70 con 0x7f616010e310 2024-04-14T15:23:35.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:35.426+0000 7f6165543700 1 -- 172.21.15.90:0/1728517424 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f614c005120 con 0x7f616010e310 2024-04-14T15:23:35.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:35.426+0000 7f6147fff700 1 -- 172.21.15.90:0/1728517424 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f6158003a00 con 0x7f616010e310 2024-04-14T15:23:35.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:35.426+0000 7f6147fff700 1 -- 172.21.15.90:0/1728517424 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(676..676 src has 467..676) ==== 8673+0+0 (unknown 478781498 0 0) 0x7f6158089d20 con 0x7f616010e310 2024-04-14T15:23:35.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:35.426+0000 7f6147fff700 10 client.49660.objecter ms_dispatch 0x7f6160001550 osd_map(676..676 src has 467..676) 2024-04-14T15:23:35.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:35.426+0000 7f6147fff700 3 client.49660.objecter handle_osd_map got epochs [676,676] > 0 2024-04-14T15:23:35.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:35.426+0000 7f6147fff700 3 client.49660.objecter handle_osd_map decoding full epoch 676 2024-04-14T15:23:35.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:35.427+0000 7f6147fff700 20 client.49660.objecter dump_active .. 0 homeless 2024-04-14T15:23:35.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:35.427+0000 7f6147fff700 10 client.49660.objecter ms_handle_connect 0x7f6148071500 2024-04-14T15:23:35.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:35.431+0000 7f6147fff700 1 -- 172.21.15.90:0/1728517424 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f6158056530 con 0x7f616010e310 2024-04-14T15:23:35.441 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:35.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:35.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:23:35.543 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:23:35.571 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:35.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:35.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:23:35.622 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:23:35 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:23:35.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:35.645+0000 7f6165543700 1 -- 172.21.15.90:0/1728517424 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7f614c005410 con 0x7f616010e310 2024-04-14T15:23:35.672 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:23:35.700 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:35.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:35.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:23:35.801 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:23:35.828 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:35.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:35.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:23:35.930 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:23:35.956 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:35.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:35.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:23:36.004 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:23:36.047 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:23:36.058 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:23:36.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:36.063+0000 7f6147fff700 1 -- 172.21.15.90:0/1728517424 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v677) ==== 101+0+0 (unknown 3020780956 0 0) 0x7f615805a1e0 con 0x7f616010e310 2024-04-14T15:23:36.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:36.066+0000 7f6165543700 10 librados: watch_flush enter 2024-04-14T15:23:36.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:36.066+0000 7f6165543700 10 librados: watch_flush exit 2024-04-14T15:23:36.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:36.066+0000 7f6165543700 20 client.49660.objecter shutdown clearing up homeless session... 2024-04-14T15:23:36.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:36.066+0000 7f6165543700 10 client.49660.objecter successfully canceled tick 2024-04-14T15:23:36.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:36.067+0000 7f6165543700 1 -- 172.21.15.90:0/1728517424 >> v1:172.21.15.90:6800/420935427 conn(0x7f6148071500 legacy=0x7f61480739c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:36.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:36.067+0000 7f6165543700 1 -- 172.21.15.90:0/1728517424 >> v1:172.21.15.92:6789/0 conn(0x7f616010e310 legacy=0x7f61601a8bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:36.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:36.067+0000 7f6165543700 1 -- 172.21.15.90:0/1728517424 shutdown_connections 2024-04-14T15:23:36.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:36.067+0000 7f6165543700 1 -- 172.21.15.90:0/1728517424 >> 172.21.15.90:0/1728517424 conn(0x7f61600fd290 msgr2=0x7f6160111130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:23:36.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:36.067+0000 7f6165543700 1 -- 172.21.15.90:0/1728517424 shutdown_connections 2024-04-14T15:23:36.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:36.067+0000 7f6165543700 1 -- 172.21.15.90:0/1728517424 wait complete. 2024-04-14T15:23:36.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:36.068+0000 7f6165543700 1 librados: shutdown 2024-04-14T15:23:36.069 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is unset 2024-04-14T15:23:36.084 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:36.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:36.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:23:36.186 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:36.210 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:36.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:36.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:36.259 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:36 smithi090 ceph-mon[41670]: pgmap v996: 214 pgs: 214 active+clean; 582 KiB data, 590 MiB used, 625 GiB / 626 GiB avail; 1000 B/s rd, 0 op/s 2024-04-14T15:23:36.259 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:36 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:23:36.259 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:36 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1728517424' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:23:36.259 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:36 smithi090 ceph-mon[41670]: from='client.49660 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:23:36.259 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:36 smithi090 ceph-mon[30399]: pgmap v996: 214 pgs: 214 active+clean; 582 KiB data, 590 MiB used, 625 GiB / 626 GiB avail; 1000 B/s rd, 0 op/s 2024-04-14T15:23:36.259 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:36 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:23:36.259 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:36 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1728517424' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:23:36.259 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:36 smithi090 ceph-mon[30399]: from='client.49660 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:23:36.312 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:23:36.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:36.338 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:36.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:23:36.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:36 smithi092 ceph-mon[33966]: pgmap v996: 214 pgs: 214 active+clean; 582 KiB data, 590 MiB used, 625 GiB / 626 GiB avail; 1000 B/s rd, 0 op/s 2024-04-14T15:23:36.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:36 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:23:36.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:36 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1728517424' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:23:36.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:36 smithi092 ceph-mon[33966]: from='client.49660 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:23:36.439 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:23:36.464 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:36.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:36.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:23:36.566 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:23:36.590 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:36.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:36.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:23:36.692 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:23:36.718 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:36.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:36.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:23:36.819 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:23:36.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:36.844 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:36.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:23:36.945 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:23:36.973 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:36.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:36.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:23:37.075 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:23:37.098 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:37.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:37.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:23:37.200 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:37.228 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:37.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:37.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:37.329 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:23:37.342 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:37 smithi090 ceph-mon[41670]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-14T15:23:37.342 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:37 smithi090 ceph-mon[41670]: from='client.49660 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-14T15:23:37.342 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:37 smithi090 ceph-mon[41670]: osdmap e677: 8 total, 8 up, 7 in 2024-04-14T15:23:37.342 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:37 smithi090 ceph-mon[30399]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-14T15:23:37.342 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:37 smithi090 ceph-mon[30399]: from='client.49660 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-14T15:23:37.342 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:37 smithi090 ceph-mon[30399]: osdmap e677: 8 total, 8 up, 7 in 2024-04-14T15:23:37.356 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:37.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:37.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:23:37.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:37 smithi092 ceph-mon[33966]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-14T15:23:37.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:37 smithi092 ceph-mon[33966]: from='client.49660 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-14T15:23:37.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:37 smithi092 ceph-mon[33966]: osdmap e677: 8 total, 8 up, 7 in 2024-04-14T15:23:37.458 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:23:37.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:37.482 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:37.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:23:37.583 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:23:37.611 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:37.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:37.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:23:37.713 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:37.740 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:37.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:37.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:37.842 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:23:37.868 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:37.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:37.869 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:23:37.969 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:23:37.993 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:37.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:37.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:23:38.036 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-14T15:23:38.095 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:23:38.121 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:38.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:38.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:23:38.223 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:23:38.247 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:38.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:38.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:23:38.349 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:23:38.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:38 smithi092 ceph-mon[33966]: 2.8 scrub starts 2024-04-14T15:23:38.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:38 smithi092 ceph-mon[33966]: 2.8 scrub ok 2024-04-14T15:23:38.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:38 smithi092 ceph-mon[33966]: pgmap v998: 214 pgs: 214 active+clean; 582 KiB data, 590 MiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:23:38.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:38 smithi092 ceph-mon[33966]: 6.9 deep-scrub starts 2024-04-14T15:23:38.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:38 smithi092 ceph-mon[33966]: 6.9 deep-scrub ok 2024-04-14T15:23:38.372 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:38.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:38.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:23:38.399 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:38 smithi090 ceph-mon[41670]: 2.8 scrub starts 2024-04-14T15:23:38.399 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:38 smithi090 ceph-mon[41670]: 2.8 scrub ok 2024-04-14T15:23:38.399 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:38 smithi090 ceph-mon[41670]: pgmap v998: 214 pgs: 214 active+clean; 582 KiB data, 590 MiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:23:38.399 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:38 smithi090 ceph-mon[41670]: 6.9 deep-scrub starts 2024-04-14T15:23:38.400 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:38 smithi090 ceph-mon[41670]: 6.9 deep-scrub ok 2024-04-14T15:23:38.400 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:38 smithi090 ceph-mon[30399]: 2.8 scrub starts 2024-04-14T15:23:38.400 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:38 smithi090 ceph-mon[30399]: 2.8 scrub ok 2024-04-14T15:23:38.400 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:38 smithi090 ceph-mon[30399]: pgmap v998: 214 pgs: 214 active+clean; 582 KiB data, 590 MiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:23:38.400 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:38 smithi090 ceph-mon[30399]: 6.9 deep-scrub starts 2024-04-14T15:23:38.400 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:38 smithi090 ceph-mon[30399]: 6.9 deep-scrub ok 2024-04-14T15:23:38.475 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:23:38.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:38.499 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:38.499 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:23:38.600 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:23:38.622 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:38.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:38.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:23:38.724 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:23:38.748 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:38.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:38.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:23:38.849 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:23:38.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:38.873 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:38.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:23:38.974 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:23:38.998 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:38.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:38.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:23:39.099 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:23:39.122 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:39.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:39.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:23:39.224 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:23:39.252 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:39.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:39.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:23:39.297 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:23:39.354 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:23:39.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:39 smithi092 ceph-mon[33966]: 4.3 scrub starts 2024-04-14T15:23:39.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:39 smithi092 ceph-mon[33966]: 4.3 scrub ok 2024-04-14T15:23:39.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:39 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:23:39.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:39 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:23:39.377 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:39.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:39.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:23:39.419 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:39 smithi090 ceph-mon[41670]: 4.3 scrub starts 2024-04-14T15:23:39.419 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:39 smithi090 ceph-mon[41670]: 4.3 scrub ok 2024-04-14T15:23:39.419 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:39 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:23:39.419 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:39 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:23:39.419 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:39 smithi090 ceph-mon[30399]: 4.3 scrub starts 2024-04-14T15:23:39.419 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:39 smithi090 ceph-mon[30399]: 4.3 scrub ok 2024-04-14T15:23:39.419 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:39 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:23:39.419 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:39 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:23:39.479 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:23:39.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:39.503 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:39.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:23:39.604 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:23:39.631 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:39.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:39.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:23:39.733 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:23:39.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:39.761 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:39.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:23:39.862 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:23:39.886 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:39.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:39.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:23:39.988 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:23:40.016 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:40.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:40.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:23:40.118 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:40.144 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:40.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:40.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:40.160 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd set noscrub 2024-04-14T15:23:40.245 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:23:40.270 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:40.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:40.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:23:40.326 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:40 smithi090 ceph-mon[30399]: 2.f deep-scrub starts 2024-04-14T15:23:40.326 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:40 smithi090 ceph-mon[30399]: 2.f deep-scrub ok 2024-04-14T15:23:40.326 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:40 smithi090 ceph-mon[30399]: pgmap v999: 214 pgs: 214 active+clean; 583 KiB data, 590 MiB used, 625 GiB / 626 GiB avail; 1.9 KiB/s rd, 2 op/s 2024-04-14T15:23:40.326 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:40 smithi090 ceph-mon[30399]: osdmap e678: 8 total, 8 up, 7 in 2024-04-14T15:23:40.326 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:40 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:23:40.326 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:40 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:23:40.326 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:40 smithi090 ceph-mon[41670]: 2.f deep-scrub starts 2024-04-14T15:23:40.326 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:40 smithi090 ceph-mon[41670]: 2.f deep-scrub ok 2024-04-14T15:23:40.326 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:40 smithi090 ceph-mon[41670]: pgmap v999: 214 pgs: 214 active+clean; 583 KiB data, 590 MiB used, 625 GiB / 626 GiB avail; 1.9 KiB/s rd, 2 op/s 2024-04-14T15:23:40.326 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:40 smithi090 ceph-mon[41670]: osdmap e678: 8 total, 8 up, 7 in 2024-04-14T15:23:40.327 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:40 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:23:40.327 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:40 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:23:40.371 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:23:40.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:40 smithi092 ceph-mon[33966]: 2.f deep-scrub starts 2024-04-14T15:23:40.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:40 smithi092 ceph-mon[33966]: 2.f deep-scrub ok 2024-04-14T15:23:40.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:40 smithi092 ceph-mon[33966]: pgmap v999: 214 pgs: 214 active+clean; 583 KiB data, 590 MiB used, 625 GiB / 626 GiB avail; 1.9 KiB/s rd, 2 op/s 2024-04-14T15:23:40.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:40 smithi092 ceph-mon[33966]: osdmap e678: 8 total, 8 up, 7 in 2024-04-14T15:23:40.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:40 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:23:40.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:40 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:23:40.395 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:40.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:40.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:23:40.498 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:23:40.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.325+0000 7f04b78c8700 1 -- 172.21.15.90:0/2779822678 >> v1:172.21.15.90:6790/0 conn(0x7f04b0107890 legacy=0x7f04b0107c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:40.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.325+0000 7f04b78c8700 1 -- 172.21.15.90:0/2779822678 shutdown_connections 2024-04-14T15:23:40.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.325+0000 7f04b78c8700 1 -- 172.21.15.90:0/2779822678 >> 172.21.15.90:0/2779822678 conn(0x7f04b00fd290 msgr2=0x7f04b00ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:23:40.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.325+0000 7f04b78c8700 1 -- 172.21.15.90:0/2779822678 shutdown_connections 2024-04-14T15:23:40.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.325+0000 7f04b78c8700 1 -- 172.21.15.90:0/2779822678 wait complete. 2024-04-14T15:23:40.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.325+0000 7f04b78c8700 1 librados: starting msgr at 2024-04-14T15:23:40.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.325+0000 7f04b78c8700 1 librados: starting objecter 2024-04-14T15:23:40.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.325+0000 7f04b78c8700 1 Processor -- start 2024-04-14T15:23:40.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.325+0000 7f04b78c8700 1 -- start start 2024-04-14T15:23:40.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.325+0000 7f04b78c8700 1 librados: setting wanted keys 2024-04-14T15:23:40.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.325+0000 7f04b78c8700 1 librados: calling monclient init 2024-04-14T15:23:40.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.326+0000 7f04b78c8700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f04b01a6a20 con 0x7f04b0107890 2024-04-14T15:23:40.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.326+0000 7f04b78c8700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f04b01a7c60 con 0x7f04b010e310 2024-04-14T15:23:40.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.326+0000 7f04b78c8700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f04b01a8ea0 con 0x7f04b010a740 2024-04-14T15:23:40.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.326+0000 7f04b4e63700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f04b010a740 0x7f04b01a17c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:54562/0 (socket says 172.21.15.90:54562) 2024-04-14T15:23:40.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.326+0000 7f04b4e63700 1 -- 172.21.15.90:0/2122646156 learned_addr learned my addr 172.21.15.90:0/2122646156 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:23:40.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.326+0000 7f04b5664700 1 --1- 172.21.15.90:0/2122646156 >> v1:172.21.15.90:6789/0 conn(0x7f04b0107890 0x7f04b010dae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:44366/0 (socket says 172.21.15.90:44366) 2024-04-14T15:23:40.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.326+0000 7f04a67fc700 10 client.?.objecter ms_handle_connect 0x7f04b0107890 2024-04-14T15:23:40.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.326+0000 7f04a67fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:23:40.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.326+0000 7f04a67fc700 10 client.?.objecter ms_handle_connect 0x7f04b010e310 2024-04-14T15:23:40.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.326+0000 7f04a67fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:23:40.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.326+0000 7f04a67fc700 10 client.?.objecter ms_handle_connect 0x7f04b010a740 2024-04-14T15:23:40.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.326+0000 7f04a67fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:23:40.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.326+0000 7f04a67fc700 1 -- 172.21.15.90:0/2122646156 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1015932874 0 0) 0x7f04b01a6a20 con 0x7f04b0107890 2024-04-14T15:23:40.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.326+0000 7f04a67fc700 1 -- 172.21.15.90:0/2122646156 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f049c0039c0 con 0x7f04b0107890 2024-04-14T15:23:40.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.326+0000 7f04a67fc700 1 -- 172.21.15.90:0/2122646156 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2006642995 0 0) 0x7f04b01a7c60 con 0x7f04b010e310 2024-04-14T15:23:40.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.326+0000 7f04a67fc700 1 -- 172.21.15.90:0/2122646156 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f04b01a6a20 con 0x7f04b010e310 2024-04-14T15:23:40.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.326+0000 7f04a67fc700 1 -- 172.21.15.90:0/2122646156 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 163900052 0 0) 0x7f04b01a8ea0 con 0x7f04b010a740 2024-04-14T15:23:40.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.326+0000 7f04a67fc700 1 -- 172.21.15.90:0/2122646156 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f04b01a7c60 con 0x7f04b010a740 2024-04-14T15:23:40.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.326+0000 7f04a67fc700 1 -- 172.21.15.90:0/2122646156 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 78569722 0 0) 0x7f04b01a6a20 con 0x7f04b010e310 2024-04-14T15:23:40.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.326+0000 7f04a67fc700 1 -- 172.21.15.90:0/2122646156 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f04b01a8ea0 con 0x7f04b010e310 2024-04-14T15:23:40.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.326+0000 7f04a67fc700 1 -- 172.21.15.90:0/2122646156 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3916089534 0 0) 0x7f049c0039c0 con 0x7f04b0107890 2024-04-14T15:23:40.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.326+0000 7f04a67fc700 1 -- 172.21.15.90:0/2122646156 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f04b01a6a20 con 0x7f04b0107890 2024-04-14T15:23:40.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.326+0000 7f04a67fc700 1 -- 172.21.15.90:0/2122646156 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f04a80013f0 con 0x7f04b010e310 2024-04-14T15:23:40.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.326+0000 7f04a67fc700 1 -- 172.21.15.90:0/2122646156 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f04ac004400 con 0x7f04b0107890 2024-04-14T15:23:40.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.326+0000 7f04a67fc700 1 -- 172.21.15.90:0/2122646156 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1238390106 0 0) 0x7f04b01a7c60 con 0x7f04b010a740 2024-04-14T15:23:40.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.326+0000 7f04a67fc700 1 -- 172.21.15.90:0/2122646156 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f049c0039c0 con 0x7f04b010a740 2024-04-14T15:23:40.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.326+0000 7f04a67fc700 1 -- 172.21.15.90:0/2122646156 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f04a0001830 con 0x7f04b010a740 2024-04-14T15:23:40.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.327+0000 7f04a67fc700 1 -- 172.21.15.90:0/2122646156 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1993553456 0 0) 0x7f04b01a8ea0 con 0x7f04b010e310 2024-04-14T15:23:40.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.327+0000 7f04a67fc700 1 -- 172.21.15.90:0/2122646156 >> v1:172.21.15.90:6790/0 conn(0x7f04b010a740 legacy=0x7f04b01a17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:40.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.327+0000 7f04a67fc700 1 -- 172.21.15.90:0/2122646156 >> v1:172.21.15.90:6789/0 conn(0x7f04b0107890 legacy=0x7f04b010dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:40.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.327+0000 7f04a67fc700 1 -- 172.21.15.90:0/2122646156 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04b01aa0e0 con 0x7f04b010e310 2024-04-14T15:23:40.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.327+0000 7f04b78c8700 1 -- 172.21.15.90:0/2122646156 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f04b01a7e90 con 0x7f04b010e310 2024-04-14T15:23:40.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.327+0000 7f04b78c8700 10 client.49666.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:23:40.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.327+0000 7f04b78c8700 1 -- 172.21.15.90:0/2122646156 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f04b01a8350 con 0x7f04b010e310 2024-04-14T15:23:40.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.327+0000 7f04b78c8700 1 librados: init done 2024-04-14T15:23:40.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.327+0000 7f04a67fc700 1 -- 172.21.15.90:0/2122646156 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f04a8003ea0 con 0x7f04b010e310 2024-04-14T15:23:40.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.327+0000 7f04a67fc700 1 -- 172.21.15.90:0/2122646156 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f04a8004da0 con 0x7f04b010e310 2024-04-14T15:23:40.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.328+0000 7f04b78c8700 10 client.49666.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:23:40.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.328+0000 7f04b78c8700 15 client.49666.objecter _session_command_op_assign -1 1 2024-04-14T15:23:40.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.328+0000 7f04b78c8700 20 client.49666.objecter _get_session osd=-1 returning homeless 2024-04-14T15:23:40.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.328+0000 7f04b78c8700 10 client.49666.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:23:40.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.328+0000 7f04b78c8700 1 -- 172.21.15.90:0/2122646156 --> v1:172.21.15.92:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f0494002920 con 0x7f04b010e310 2024-04-14T15:23:40.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.328+0000 7f04a67fc700 1 -- 172.21.15.90:0/2122646156 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f04a801ba30 con 0x7f04b010e310 2024-04-14T15:23:40.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.329+0000 7f04a67fc700 1 -- 172.21.15.90:0/2122646156 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(678..678 src has 467..678) ==== 8785+0+0 (unknown 1955197104 0 0) 0x7f04a8089d00 con 0x7f04b010e310 2024-04-14T15:23:40.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.329+0000 7f04a67fc700 10 client.49666.objecter ms_dispatch 0x7f04b0001550 osd_map(678..678 src has 467..678) 2024-04-14T15:23:40.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.329+0000 7f04a67fc700 3 client.49666.objecter handle_osd_map got epochs [678,678] > 0 2024-04-14T15:23:40.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.329+0000 7f04a67fc700 3 client.49666.objecter handle_osd_map decoding full epoch 678 2024-04-14T15:23:40.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.329+0000 7f04a67fc700 10 client.49666.objecter checking command 1 2024-04-14T15:23:40.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.329+0000 7f04a67fc700 20 client.49666.objecter _get_session s=0x7f049c07eec0 osd=3 3 2024-04-14T15:23:40.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.329+0000 7f04a67fc700 20 client.49666.objecter put_session s=0x7f049c07eec0 osd=3 3 2024-04-14T15:23:40.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.329+0000 7f04a67fc700 15 client.49666.objecter _session_command_op_remove -1 1 2024-04-14T15:23:40.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.329+0000 7f04a67fc700 20 client.49666.objecter _get_session s=0x7f049c07eec0 osd=3 3 2024-04-14T15:23:40.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.329+0000 7f04a67fc700 20 client.49666.objecter get_session s=0x7f049c07eec0 osd=3 3 2024-04-14T15:23:40.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.329+0000 7f04a67fc700 15 client.49666.objecter _session_command_op_assign 3 1 2024-04-14T15:23:40.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.329+0000 7f04a67fc700 20 client.49666.objecter put_session s=0x7f049c07eec0 osd=3 4 2024-04-14T15:23:40.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.329+0000 7f04a67fc700 10 client.49666.objecter _send_command 1 2024-04-14T15:23:40.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.329+0000 7f04a67fc700 1 -- 172.21.15.90:0/2122646156 --> v1:172.21.15.90:6813/2270166774 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f049c083280 con 0x7f049c07f570 2024-04-14T15:23:40.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.329+0000 7f04a67fc700 20 client.49666.objecter dump_active .. 0 homeless 2024-04-14T15:23:40.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.329+0000 7f04a67fc700 1 -- 172.21.15.90:0/2122646156 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_get_version_reply(handle=1 version=678) ==== 24+0+0 (unknown 3605992111 0 0) 0x7f04a808a110 con 0x7f04b010e310 2024-04-14T15:23:40.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.329+0000 7f04a67fc700 10 client.49666.objecter ms_handle_connect 0x7f049c0713c0 2024-04-14T15:23:40.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.329+0000 7f04a67fc700 10 client.49666.objecter ms_handle_connect 0x7f049c07f570 2024-04-14T15:23:40.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.330+0000 7f04a67fc700 1 -- 172.21.15.90:0/2122646156 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f049c083280 con 0x7f049c07f570 2024-04-14T15:23:40.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.330+0000 7f04a67fc700 10 client.49666.objecter ms_dispatch 0x7f04b0001550 command_reply(tid 1: 0 ) 2024-04-14T15:23:40.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.330+0000 7f04a67fc700 10 client.49666.objecter _finish_command 1 = system:0 2024-04-14T15:23:40.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.330+0000 7f04a67fc700 20 client.49666.objecter put_session s=0x7f049c07eec0 osd=3 4 2024-04-14T15:23:40.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:40.330+0000 7f04a67fc700 15 client.49666.objecter _session_command_op_remove 3 1 2024-04-14T15:23:40.507 INFO:teuthology.orchestra.run.smithi090.stderr:no valid command found; 10 closest matches: 2024-04-14T15:23:40.507 INFO:teuthology.orchestra.run.smithi090.stderr:0 2024-04-14T15:23:40.507 INFO:teuthology.orchestra.run.smithi090.stderr:1 2024-04-14T15:23:40.507 INFO:teuthology.orchestra.run.smithi090.stderr:2 2024-04-14T15:23:40.507 INFO:teuthology.orchestra.run.smithi090.stderr:abort 2024-04-14T15:23:40.507 INFO:teuthology.orchestra.run.smithi090.stderr:assert 2024-04-14T15:23:40.507 INFO:teuthology.orchestra.run.smithi090.stderr:bench [] [] [] [] 2024-04-14T15:23:40.508 INFO:teuthology.orchestra.run.smithi090.stderr:bluefs debug_inject_read_zeros 2024-04-14T15:23:40.508 INFO:teuthology.orchestra.run.smithi090.stderr:bluefs files list 2024-04-14T15:23:40.508 INFO:teuthology.orchestra.run.smithi090.stderr:bluefs stats 2024-04-14T15:23:40.508 INFO:teuthology.orchestra.run.smithi090.stderr:bluestore allocator dump block 2024-04-14T15:23:40.508 INFO:teuthology.orchestra.run.smithi090.stderr:Error EINVAL: invalid command 2024-04-14T15:23:40.525 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:40.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:40.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:23:40.626 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:23:40.654 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:40.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:40.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:23:40.756 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:23:40.783 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:40.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:40.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:23:40.885 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:23:40.909 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:40.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:40.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:23:41.011 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:23:41.037 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:41.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:41.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:23:41.139 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:23:41.162 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:41.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:41.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:23:41.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:41 smithi090 ceph-mon[41670]: 3.1 scrub starts 2024-04-14T15:23:41.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:41 smithi090 ceph-mon[41670]: 3.1 scrub ok 2024-04-14T15:23:41.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:41 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]': finished 2024-04-14T15:23:41.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:41 smithi090 ceph-mon[41670]: osdmap e679: 8 total, 8 up, 7 in 2024-04-14T15:23:41.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:41 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-116"}]: dispatch 2024-04-14T15:23:41.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:41 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-116"}]: dispatch 2024-04-14T15:23:41.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:41 smithi090 ceph-mon[30399]: 3.1 scrub starts 2024-04-14T15:23:41.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:41 smithi090 ceph-mon[30399]: 3.1 scrub ok 2024-04-14T15:23:41.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:41 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]': finished 2024-04-14T15:23:41.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:41 smithi090 ceph-mon[30399]: osdmap e679: 8 total, 8 up, 7 in 2024-04-14T15:23:41.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:41 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-116"}]: dispatch 2024-04-14T15:23:41.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:41 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-116"}]: dispatch 2024-04-14T15:23:41.264 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:23:41.267 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-14T15:23:41.267 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-14T15:23:41.268 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph tell osd.4 'trim stale osdmaps' 2024-04-14T15:23:41.288 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:41.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:41.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:23:41.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:41 smithi092 ceph-mon[33966]: 3.1 scrub starts 2024-04-14T15:23:41.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:41 smithi092 ceph-mon[33966]: 3.1 scrub ok 2024-04-14T15:23:41.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:41 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]': finished 2024-04-14T15:23:41.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:41 smithi092 ceph-mon[33966]: osdmap e679: 8 total, 8 up, 7 in 2024-04-14T15:23:41.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:41 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-116"}]: dispatch 2024-04-14T15:23:41.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:41 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-116"}]: dispatch 2024-04-14T15:23:41.390 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:41.416 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:41.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:41.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:41.518 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:23:41.544 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:41.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:41.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:23:41.644 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-14T15:23:41.644 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-14T15:23:41.644 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-14T15:23:41.644 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-14T15:23:41.646 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:41.673 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:41.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:41.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:41.775 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:23:41.802 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:41.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:41.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:23:41.904 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:23:41.928 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:41.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:41.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:23:42.030 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:23:42.055 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:42.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:42.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:23:42.157 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:42.184 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:42.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:42.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:42.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:42 smithi090 ceph-mon[41670]: 4.1f scrub starts 2024-04-14T15:23:42.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:42 smithi090 ceph-mon[41670]: 4.1f scrub ok 2024-04-14T15:23:42.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:42 smithi090 ceph-mon[41670]: 2.11 scrub starts 2024-04-14T15:23:42.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:42 smithi090 ceph-mon[41670]: 2.11 scrub ok 2024-04-14T15:23:42.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:42 smithi090 ceph-mon[41670]: 6.b scrub starts 2024-04-14T15:23:42.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:42 smithi090 ceph-mon[41670]: 6.b scrub ok 2024-04-14T15:23:42.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:42 smithi090 ceph-mon[41670]: pgmap v1002: 214 pgs: 214 active+clean; 583 KiB data, 590 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-14T15:23:42.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:42 smithi090 ceph-mon[41670]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:23:42.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:42 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-116"}]': finished 2024-04-14T15:23:42.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:42 smithi090 ceph-mon[41670]: osdmap e680: 8 total, 8 up, 7 in 2024-04-14T15:23:42.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:42 smithi090 ceph-mon[30399]: 4.1f scrub starts 2024-04-14T15:23:42.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:42 smithi090 ceph-mon[30399]: 4.1f scrub ok 2024-04-14T15:23:42.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:42 smithi090 ceph-mon[30399]: 2.11 scrub starts 2024-04-14T15:23:42.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:42 smithi090 ceph-mon[30399]: 2.11 scrub ok 2024-04-14T15:23:42.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:42 smithi090 ceph-mon[30399]: 6.b scrub starts 2024-04-14T15:23:42.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:42 smithi090 ceph-mon[30399]: 6.b scrub ok 2024-04-14T15:23:42.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:42 smithi090 ceph-mon[30399]: pgmap v1002: 214 pgs: 214 active+clean; 583 KiB data, 590 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-14T15:23:42.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:42 smithi090 ceph-mon[30399]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:23:42.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:42 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-116"}]': finished 2024-04-14T15:23:42.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:42 smithi090 ceph-mon[30399]: osdmap e680: 8 total, 8 up, 7 in 2024-04-14T15:23:42.286 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:23:42.313 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:42.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:42.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:23:42.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:42 smithi092 ceph-mon[33966]: 4.1f scrub starts 2024-04-14T15:23:42.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:42 smithi092 ceph-mon[33966]: 4.1f scrub ok 2024-04-14T15:23:42.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:42 smithi092 ceph-mon[33966]: 2.11 scrub starts 2024-04-14T15:23:42.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:42 smithi092 ceph-mon[33966]: 2.11 scrub ok 2024-04-14T15:23:42.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:42 smithi092 ceph-mon[33966]: 6.b scrub starts 2024-04-14T15:23:42.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:42 smithi092 ceph-mon[33966]: 6.b scrub ok 2024-04-14T15:23:42.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:42 smithi092 ceph-mon[33966]: pgmap v1002: 214 pgs: 214 active+clean; 583 KiB data, 590 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-14T15:23:42.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:42 smithi092 ceph-mon[33966]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:23:42.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:42 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-116"}]': finished 2024-04-14T15:23:42.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:42 smithi092 ceph-mon[33966]: osdmap e680: 8 total, 8 up, 7 in 2024-04-14T15:23:42.415 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:42.442 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:42.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:42.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:42.544 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:23:42.573 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:42.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:42.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:23:42.675 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:23:42.698 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:42.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:42.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:23:42.800 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:23:42.828 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:42.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:42.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:23:42.929 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:23:42.956 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:42.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:42.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:23:43.058 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:43.084 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:43.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:43.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:43.186 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:23:43.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:43 smithi090 ceph-mon[41670]: 11.3 scrub starts 2024-04-14T15:23:43.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:43 smithi090 ceph-mon[41670]: 11.3 scrub ok 2024-04-14T15:23:43.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:43 smithi090 ceph-mon[41670]: osdmap e681: 8 total, 8 up, 7 in 2024-04-14T15:23:43.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:43 smithi090 ceph-mon[30399]: 11.3 scrub starts 2024-04-14T15:23:43.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:43 smithi090 ceph-mon[30399]: 11.3 scrub ok 2024-04-14T15:23:43.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:43 smithi090 ceph-mon[30399]: osdmap e681: 8 total, 8 up, 7 in 2024-04-14T15:23:43.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:43.211 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:43.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:23:43.312 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:23:43.338 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:43.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:43.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:23:43.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:43 smithi092 ceph-mon[33966]: 11.3 scrub starts 2024-04-14T15:23:43.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:43 smithi092 ceph-mon[33966]: 11.3 scrub ok 2024-04-14T15:23:43.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:43 smithi092 ceph-mon[33966]: osdmap e681: 8 total, 8 up, 7 in 2024-04-14T15:23:43.440 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:23:43.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:43.466 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:43.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:23:43.511 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-14T15:23:43.523 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:23:43.524 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:23:43.566 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:23:43.590 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:43.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:43.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:23:43.691 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:23:43.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:43.715 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:43.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:23:43.816 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:23:43.840 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:43.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:43.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:23:43.941 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:43.969 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:43.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:43.970 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:44.070 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:23:44.101 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:44.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:44.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:23:44.202 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:23:44.228 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:44.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:44.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:23:44.242 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:23:43 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:23:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:23:44.243 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:44 smithi090 ceph-mon[41670]: 6.4 scrub starts 2024-04-14T15:23:44.243 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:44 smithi090 ceph-mon[41670]: 6.4 scrub ok 2024-04-14T15:23:44.243 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:44 smithi090 ceph-mon[41670]: pgmap v1005: 182 pgs: 182 active+clean; 582 KiB data, 590 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:23:44.243 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:44 smithi090 ceph-mon[41670]: 1.6 scrub starts 2024-04-14T15:23:44.243 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:44 smithi090 ceph-mon[41670]: 1.6 scrub ok 2024-04-14T15:23:44.243 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:44 smithi090 ceph-mon[41670]: osdmap e682: 8 total, 8 up, 7 in 2024-04-14T15:23:44.243 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:44 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:23:44.243 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:44 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:23:44.243 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:44 smithi090 ceph-mon[30399]: 6.4 scrub starts 2024-04-14T15:23:44.243 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:44 smithi090 ceph-mon[30399]: 6.4 scrub ok 2024-04-14T15:23:44.244 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:44 smithi090 ceph-mon[30399]: pgmap v1005: 182 pgs: 182 active+clean; 582 KiB data, 590 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:23:44.244 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:44 smithi090 ceph-mon[30399]: 1.6 scrub starts 2024-04-14T15:23:44.244 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:44 smithi090 ceph-mon[30399]: 1.6 scrub ok 2024-04-14T15:23:44.244 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:44 smithi090 ceph-mon[30399]: osdmap e682: 8 total, 8 up, 7 in 2024-04-14T15:23:44.244 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:44 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:23:44.244 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:44 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:23:44.330 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:23:44.355 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:44.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:44.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:23:44.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:44 smithi092 ceph-mon[33966]: 6.4 scrub starts 2024-04-14T15:23:44.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:44 smithi092 ceph-mon[33966]: 6.4 scrub ok 2024-04-14T15:23:44.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:44 smithi092 ceph-mon[33966]: pgmap v1005: 182 pgs: 182 active+clean; 582 KiB data, 590 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:23:44.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:44 smithi092 ceph-mon[33966]: 1.6 scrub starts 2024-04-14T15:23:44.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:44 smithi092 ceph-mon[33966]: 1.6 scrub ok 2024-04-14T15:23:44.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:44 smithi092 ceph-mon[33966]: osdmap e682: 8 total, 8 up, 7 in 2024-04-14T15:23:44.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:44 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:23:44.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:44 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:23:44.457 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:23:44.482 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:44.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:44.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:23:44.584 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:44.609 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:44.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:44.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:44.711 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:23:44.736 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:44.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:44.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:23:44.838 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:23:44.863 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:44.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:44.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:23:44.965 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:23:44.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:44.991 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:44.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:23:45.092 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:23:45.116 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:45.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:45.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:23:45.218 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:23:45.241 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:45.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:45.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:23:45.343 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:23:45.366 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:45.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:45.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:23:45.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:45 smithi092 ceph-mon[33966]: 2.16 scrub starts 2024-04-14T15:23:45.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:45 smithi092 ceph-mon[33966]: 2.16 scrub ok 2024-04-14T15:23:45.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:45 smithi092 ceph-mon[33966]: 11.1 scrub starts 2024-04-14T15:23:45.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:45 smithi092 ceph-mon[33966]: 11.1 scrub ok 2024-04-14T15:23:45.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:45 smithi092 ceph-mon[33966]: 11.6 scrub starts 2024-04-14T15:23:45.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:45 smithi092 ceph-mon[33966]: 11.6 scrub ok 2024-04-14T15:23:45.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:45 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:23:45.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:45 smithi092 ceph-mon[33966]: osdmap e683: 8 total, 8 up, 7 in 2024-04-14T15:23:45.399 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:45 smithi090 ceph-mon[41670]: 2.16 scrub starts 2024-04-14T15:23:45.399 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:45 smithi090 ceph-mon[41670]: 2.16 scrub ok 2024-04-14T15:23:45.400 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:45 smithi090 ceph-mon[41670]: 11.1 scrub starts 2024-04-14T15:23:45.400 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:45 smithi090 ceph-mon[41670]: 11.1 scrub ok 2024-04-14T15:23:45.400 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:45 smithi090 ceph-mon[41670]: 11.6 scrub starts 2024-04-14T15:23:45.400 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:45 smithi090 ceph-mon[41670]: 11.6 scrub ok 2024-04-14T15:23:45.400 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:45 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:23:45.400 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:45 smithi090 ceph-mon[41670]: osdmap e683: 8 total, 8 up, 7 in 2024-04-14T15:23:45.400 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:45 smithi090 ceph-mon[30399]: 2.16 scrub starts 2024-04-14T15:23:45.400 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:45 smithi090 ceph-mon[30399]: 2.16 scrub ok 2024-04-14T15:23:45.400 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:45 smithi090 ceph-mon[30399]: 11.1 scrub starts 2024-04-14T15:23:45.400 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:45 smithi090 ceph-mon[30399]: 11.1 scrub ok 2024-04-14T15:23:45.400 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:45 smithi090 ceph-mon[30399]: 11.6 scrub starts 2024-04-14T15:23:45.401 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:45 smithi090 ceph-mon[30399]: 11.6 scrub ok 2024-04-14T15:23:45.401 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:45 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:23:45.401 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:45 smithi090 ceph-mon[30399]: osdmap e683: 8 total, 8 up, 7 in 2024-04-14T15:23:45.468 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:23:45.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:45.492 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:45.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:23:45.593 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:23:45.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:45.621 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:45.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:23:45.722 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:23:45.749 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:45.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:45.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:23:45.851 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:23:45.872 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:23:45 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:23:45.874 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:45.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:45.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:23:45.976 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:23:46.004 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:46.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:46.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:23:46.106 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:23:46.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:46.130 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:46.131 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:23:46.231 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:46.259 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:46.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:46.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:46.360 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:23:46.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:46 smithi092 ceph-mon[33966]: 2.c scrub starts 2024-04-14T15:23:46.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:46 smithi092 ceph-mon[33966]: 2.c scrub ok 2024-04-14T15:23:46.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:46 smithi092 ceph-mon[33966]: 3.2 scrub starts 2024-04-14T15:23:46.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:46 smithi092 ceph-mon[33966]: 3.2 scrub ok 2024-04-14T15:23:46.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:46 smithi092 ceph-mon[33966]: pgmap v1008: 214 pgs: 32 creating+peering, 182 active+clean; 582 KiB data, 608 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:23:46.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:46 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:23:46.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:46 smithi092 ceph-mon[33966]: osdmap e684: 8 total, 8 up, 7 in 2024-04-14T15:23:46.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:46 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:23:46.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:46 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:23:46.384 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:46.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:46.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:23:46.387 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:46 smithi090 ceph-mon[41670]: 2.c scrub starts 2024-04-14T15:23:46.387 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:46 smithi090 ceph-mon[41670]: 2.c scrub ok 2024-04-14T15:23:46.387 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:46 smithi090 ceph-mon[41670]: 3.2 scrub starts 2024-04-14T15:23:46.388 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:46 smithi090 ceph-mon[41670]: 3.2 scrub ok 2024-04-14T15:23:46.388 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:46 smithi090 ceph-mon[41670]: pgmap v1008: 214 pgs: 32 creating+peering, 182 active+clean; 582 KiB data, 608 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:23:46.388 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:46 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:23:46.388 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:46 smithi090 ceph-mon[41670]: osdmap e684: 8 total, 8 up, 7 in 2024-04-14T15:23:46.388 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:46 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:23:46.388 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:46 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:23:46.388 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:46 smithi090 ceph-mon[30399]: 2.c scrub starts 2024-04-14T15:23:46.388 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:46 smithi090 ceph-mon[30399]: 2.c scrub ok 2024-04-14T15:23:46.388 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:46 smithi090 ceph-mon[30399]: 3.2 scrub starts 2024-04-14T15:23:46.388 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:46 smithi090 ceph-mon[30399]: 3.2 scrub ok 2024-04-14T15:23:46.388 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:46 smithi090 ceph-mon[30399]: pgmap v1008: 214 pgs: 32 creating+peering, 182 active+clean; 582 KiB data, 608 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:23:46.388 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:46 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:23:46.389 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:46 smithi090 ceph-mon[30399]: osdmap e684: 8 total, 8 up, 7 in 2024-04-14T15:23:46.389 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:46 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:23:46.389 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:46 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:23:46.486 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:23:46.510 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:46.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:46.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:23:46.611 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:23:46.638 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:46.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:46.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:23:46.740 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:23:46.766 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:46.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:46.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:23:46.868 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:23:46.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:46.893 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:46.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:23:46.993 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:23:47.017 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:47.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:47.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:23:47.119 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:23:47.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:47.143 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:47.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:23:47.244 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:23:47.272 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:47.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:47.273 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:23:47.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.298+0000 7f9708e0c700 1 -- 172.21.15.90:0/3897543778 >> v1:172.21.15.90:6789/0 conn(0x7f970410b730 legacy=0x7f970410db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:47.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.298+0000 7f9708e0c700 1 -- 172.21.15.90:0/3897543778 shutdown_connections 2024-04-14T15:23:47.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.298+0000 7f9708e0c700 1 -- 172.21.15.90:0/3897543778 >> 172.21.15.90:0/3897543778 conn(0x7f97040fe3d0 msgr2=0x7f97041007f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:23:47.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.299+0000 7f9708e0c700 1 -- 172.21.15.90:0/3897543778 shutdown_connections 2024-04-14T15:23:47.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.299+0000 7f9708e0c700 1 -- 172.21.15.90:0/3897543778 wait complete. 2024-04-14T15:23:47.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.299+0000 7f9708e0c700 1 librados: starting msgr at 2024-04-14T15:23:47.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.299+0000 7f9708e0c700 1 librados: starting objecter 2024-04-14T15:23:47.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.299+0000 7f9708e0c700 1 Processor -- start 2024-04-14T15:23:47.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.299+0000 7f9708e0c700 1 -- start start 2024-04-14T15:23:47.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.299+0000 7f9708e0c700 1 librados: setting wanted keys 2024-04-14T15:23:47.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.299+0000 7f9708e0c700 1 librados: calling monclient init 2024-04-14T15:23:47.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.300+0000 7f9708e0c700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f970410ef70 con 0x7f9704108880 2024-04-14T15:23:47.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.300+0000 7f9708e0c700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f97041a8f10 con 0x7f970410b730 2024-04-14T15:23:47.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.300+0000 7f9708e0c700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f97041aa0f0 con 0x7f970410f300 2024-04-14T15:23:47.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.300+0000 7f970259c700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f9704108880 0x7f970410bdc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:57128/0 (socket says 172.21.15.90:57128) 2024-04-14T15:23:47.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.300+0000 7f970259c700 1 -- 172.21.15.90:0/3985084990 learned_addr learned my addr 172.21.15.90:0/3985084990 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:23:47.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.300+0000 7f96f37fe700 10 client.?.objecter ms_handle_connect 0x7f9704108880 2024-04-14T15:23:47.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.300+0000 7f96f37fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:23:47.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.300+0000 7f96f37fe700 10 client.?.objecter ms_handle_connect 0x7f970410f300 2024-04-14T15:23:47.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.300+0000 7f96f37fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:23:47.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.300+0000 7f96f37fe700 10 client.?.objecter ms_handle_connect 0x7f970410b730 2024-04-14T15:23:47.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.300+0000 7f96f37fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:23:47.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.300+0000 7f96f37fe700 1 -- 172.21.15.90:0/3985084990 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4277058033 0 0) 0x7f970410ef70 con 0x7f9704108880 2024-04-14T15:23:47.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.300+0000 7f96f37fe700 1 -- 172.21.15.90:0/3985084990 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f96ec0035b0 con 0x7f9704108880 2024-04-14T15:23:47.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.300+0000 7f96f37fe700 1 -- 172.21.15.90:0/3985084990 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 817594112 0 0) 0x7f97041aa0f0 con 0x7f970410f300 2024-04-14T15:23:47.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.300+0000 7f96f37fe700 1 -- 172.21.15.90:0/3985084990 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f970410ef70 con 0x7f970410f300 2024-04-14T15:23:47.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.300+0000 7f96f37fe700 1 -- 172.21.15.90:0/3985084990 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 619289011 0 0) 0x7f97041a8f10 con 0x7f970410b730 2024-04-14T15:23:47.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.300+0000 7f96f37fe700 1 -- 172.21.15.90:0/3985084990 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f97041aa0f0 con 0x7f970410b730 2024-04-14T15:23:47.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.300+0000 7f96f37fe700 1 -- 172.21.15.90:0/3985084990 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1723360431 0 0) 0x7f96ec0035b0 con 0x7f9704108880 2024-04-14T15:23:47.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.300+0000 7f96f37fe700 1 -- 172.21.15.90:0/3985084990 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f97041a8f10 con 0x7f9704108880 2024-04-14T15:23:47.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.300+0000 7f96f37fe700 1 -- 172.21.15.90:0/3985084990 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f96f8001930 con 0x7f9704108880 2024-04-14T15:23:47.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.300+0000 7f96f37fe700 1 -- 172.21.15.90:0/3985084990 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2127125927 0 0) 0x7f970410ef70 con 0x7f970410f300 2024-04-14T15:23:47.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.300+0000 7f96f37fe700 1 -- 172.21.15.90:0/3985084990 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f96ec0035b0 con 0x7f970410f300 2024-04-14T15:23:47.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.300+0000 7f96f37fe700 1 -- 172.21.15.90:0/3985084990 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f96fc001c50 con 0x7f970410f300 2024-04-14T15:23:47.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.300+0000 7f96f37fe700 1 -- 172.21.15.90:0/3985084990 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3744154954 0 0) 0x7f97041aa0f0 con 0x7f970410b730 2024-04-14T15:23:47.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.300+0000 7f96f37fe700 1 -- 172.21.15.90:0/3985084990 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f970410ef70 con 0x7f970410b730 2024-04-14T15:23:47.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.300+0000 7f96f37fe700 1 -- 172.21.15.90:0/3985084990 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f96f4001430 con 0x7f970410b730 2024-04-14T15:23:47.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.300+0000 7f96f37fe700 1 -- 172.21.15.90:0/3985084990 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3159214019 0 0) 0x7f97041a8f10 con 0x7f9704108880 2024-04-14T15:23:47.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.300+0000 7f96f37fe700 1 -- 172.21.15.90:0/3985084990 >> v1:172.21.15.90:6790/0 conn(0x7f970410f300 legacy=0x7f97041a67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:47.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.300+0000 7f96f37fe700 1 -- 172.21.15.90:0/3985084990 >> v1:172.21.15.92:6789/0 conn(0x7f970410b730 legacy=0x7f970410c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:47.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.300+0000 7f96f37fe700 1 -- 172.21.15.90:0/3985084990 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97041ab2d0 con 0x7f9704108880 2024-04-14T15:23:47.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.301+0000 7f9708e0c700 1 -- 172.21.15.90:0/3985084990 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f97041aa320 con 0x7f9704108880 2024-04-14T15:23:47.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.301+0000 7f9708e0c700 10 client.37839.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:23:47.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.301+0000 7f9708e0c700 1 -- 172.21.15.90:0/3985084990 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f97041aa7e0 con 0x7f9704108880 2024-04-14T15:23:47.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.301+0000 7f9708e0c700 1 librados: init done 2024-04-14T15:23:47.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.301+0000 7f96f37fe700 1 -- 172.21.15.90:0/3985084990 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f96f8001d00 con 0x7f9704108880 2024-04-14T15:23:47.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.301+0000 7f96f37fe700 1 -- 172.21.15.90:0/3985084990 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f96f8005d90 con 0x7f9704108880 2024-04-14T15:23:47.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.302+0000 7f9708e0c700 1 -- 172.21.15.90:0/3985084990 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96e4005150 con 0x7f9704108880 2024-04-14T15:23:47.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.302+0000 7f96f37fe700 1 -- 172.21.15.90:0/3985084990 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f96f8003950 con 0x7f9704108880 2024-04-14T15:23:47.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.303+0000 7f96f37fe700 1 -- 172.21.15.90:0/3985084990 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(685..685 src has 467..685) ==== 8729+0+0 (unknown 3028141388 0 0) 0x7f96f808abb0 con 0x7f9704108880 2024-04-14T15:23:47.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.303+0000 7f96f37fe700 10 client.37839.objecter ms_dispatch 0x7f9704001550 osd_map(685..685 src has 467..685) 2024-04-14T15:23:47.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.303+0000 7f96f37fe700 3 client.37839.objecter handle_osd_map got epochs [685,685] > 0 2024-04-14T15:23:47.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.303+0000 7f96f37fe700 3 client.37839.objecter handle_osd_map decoding full epoch 685 2024-04-14T15:23:47.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.303+0000 7f96f37fe700 20 client.37839.objecter dump_active .. 0 homeless 2024-04-14T15:23:47.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.303+0000 7f96f37fe700 10 client.37839.objecter ms_handle_connect 0x7f96ec0713f0 2024-04-14T15:23:47.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.307+0000 7f96f37fe700 1 -- 172.21.15.90:0/3985084990 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f96f8057380 con 0x7f9704108880 2024-04-14T15:23:47.374 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:23:47.401 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:47.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:47.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:23:47.503 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:23:47.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.519+0000 7f9708e0c700 1 -- 172.21.15.90:0/3985084990 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7f96e4005440 con 0x7f9704108880 2024-04-14T15:23:47.527 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:47.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:47.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:23:47.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.557+0000 7f30dcfb5700 1 -- 172.21.15.90:0/2904179863 >> v1:172.21.15.90:6789/0 conn(0x7f30d8107780 legacy=0x7f30d8107b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:47.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.557+0000 7f30dcfb5700 1 -- 172.21.15.90:0/2904179863 shutdown_connections 2024-04-14T15:23:47.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.557+0000 7f30dcfb5700 1 -- 172.21.15.90:0/2904179863 >> 172.21.15.90:0/2904179863 conn(0x7f30d80fd2b0 msgr2=0x7f30d80ff6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:23:47.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.557+0000 7f30dcfb5700 1 -- 172.21.15.90:0/2904179863 shutdown_connections 2024-04-14T15:23:47.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.558+0000 7f30dcfb5700 1 -- 172.21.15.90:0/2904179863 wait complete. 2024-04-14T15:23:47.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.558+0000 7f30dcfb5700 1 librados: starting msgr at 2024-04-14T15:23:47.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.558+0000 7f30dcfb5700 1 librados: starting objecter 2024-04-14T15:23:47.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.558+0000 7f30dcfb5700 1 Processor -- start 2024-04-14T15:23:47.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.558+0000 7f30dcfb5700 1 -- start start 2024-04-14T15:23:47.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.558+0000 7f30dcfb5700 1 librados: setting wanted keys 2024-04-14T15:23:47.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.558+0000 7f30dcfb5700 1 librados: calling monclient init 2024-04-14T15:23:47.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.559+0000 7f30dcfb5700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f30d810dec0 con 0x7f30d810a630 2024-04-14T15:23:47.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.559+0000 7f30dcfb5700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f30d81a7e10 con 0x7f30d810e200 2024-04-14T15:23:47.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.559+0000 7f30dcfb5700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f30d81a8ff0 con 0x7f30d8107780 2024-04-14T15:23:47.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.559+0000 7f30d659c700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f30d8107780 0x7f30d810aed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:36750/0 (socket says 172.21.15.90:36750) 2024-04-14T15:23:47.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.559+0000 7f30d659c700 1 -- 172.21.15.90:0/2302147943 learned_addr learned my addr 172.21.15.90:0/2302147943 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:23:47.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.559+0000 7f30cf7fe700 10 client.?.objecter ms_handle_connect 0x7f30d8107780 2024-04-14T15:23:47.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.559+0000 7f30cf7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:23:47.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.559+0000 7f30cf7fe700 10 client.?.objecter ms_handle_connect 0x7f30d810a630 2024-04-14T15:23:47.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.559+0000 7f30cf7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:23:47.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.559+0000 7f30cf7fe700 1 -- 172.21.15.90:0/2302147943 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 299806296 0 0) 0x7f30d81a8ff0 con 0x7f30d8107780 2024-04-14T15:23:47.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.559+0000 7f30cf7fe700 1 -- 172.21.15.90:0/2302147943 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f30c40035b0 con 0x7f30d8107780 2024-04-14T15:23:47.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.559+0000 7f30cf7fe700 10 client.?.objecter ms_handle_connect 0x7f30d810e200 2024-04-14T15:23:47.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.559+0000 7f30cf7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:23:47.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.559+0000 7f30cf7fe700 1 -- 172.21.15.90:0/2302147943 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3418426492 0 0) 0x7f30d810dec0 con 0x7f30d810a630 2024-04-14T15:23:47.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.559+0000 7f30cf7fe700 1 -- 172.21.15.90:0/2302147943 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f30d81a8ff0 con 0x7f30d810a630 2024-04-14T15:23:47.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.559+0000 7f30cf7fe700 1 -- 172.21.15.90:0/2302147943 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2632582247 0 0) 0x7f30d81a7e10 con 0x7f30d810e200 2024-04-14T15:23:47.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.559+0000 7f30cf7fe700 1 -- 172.21.15.90:0/2302147943 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f30c4005960 con 0x7f30d810e200 2024-04-14T15:23:47.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.559+0000 7f30cf7fe700 1 -- 172.21.15.90:0/2302147943 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2312747002 0 0) 0x7f30c40035b0 con 0x7f30d8107780 2024-04-14T15:23:47.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.559+0000 7f30cf7fe700 1 -- 172.21.15.90:0/2302147943 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f30d81a7e10 con 0x7f30d8107780 2024-04-14T15:23:47.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.559+0000 7f30cf7fe700 1 -- 172.21.15.90:0/2302147943 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f30d0001950 con 0x7f30d8107780 2024-04-14T15:23:47.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.559+0000 7f30cf7fe700 1 -- 172.21.15.90:0/2302147943 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3815808192 0 0) 0x7f30d81a8ff0 con 0x7f30d810a630 2024-04-14T15:23:47.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.559+0000 7f30cf7fe700 1 -- 172.21.15.90:0/2302147943 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f30c40035b0 con 0x7f30d810a630 2024-04-14T15:23:47.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.559+0000 7f30cf7fe700 1 -- 172.21.15.90:0/2302147943 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f30c00018b0 con 0x7f30d810a630 2024-04-14T15:23:47.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.559+0000 7f30cf7fe700 1 -- 172.21.15.90:0/2302147943 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4287922024 0 0) 0x7f30c4005960 con 0x7f30d810e200 2024-04-14T15:23:47.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.559+0000 7f30cf7fe700 1 -- 172.21.15.90:0/2302147943 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f30d81a8ff0 con 0x7f30d810e200 2024-04-14T15:23:47.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.559+0000 7f30cf7fe700 1 -- 172.21.15.90:0/2302147943 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f30c80018b0 con 0x7f30d810e200 2024-04-14T15:23:47.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.560+0000 7f30cf7fe700 1 -- 172.21.15.90:0/2302147943 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4098004759 0 0) 0x7f30c40035b0 con 0x7f30d810a630 2024-04-14T15:23:47.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.560+0000 7f30cf7fe700 1 -- 172.21.15.90:0/2302147943 >> v1:172.21.15.90:6790/0 conn(0x7f30d8107780 legacy=0x7f30d810aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:47.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.560+0000 7f30cf7fe700 1 -- 172.21.15.90:0/2302147943 >> v1:172.21.15.92:6789/0 conn(0x7f30d810e200 legacy=0x7f30d81a56e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:47.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.560+0000 7f30cf7fe700 1 -- 172.21.15.90:0/2302147943 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30d81aa1d0 con 0x7f30d810a630 2024-04-14T15:23:47.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.560+0000 7f30dcfb5700 1 -- 172.21.15.90:0/2302147943 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f30d81a6e60 con 0x7f30d810a630 2024-04-14T15:23:47.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.560+0000 7f30dcfb5700 10 client.37845.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:23:47.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.560+0000 7f30dcfb5700 1 -- 172.21.15.90:0/2302147943 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f30d81a7320 con 0x7f30d810a630 2024-04-14T15:23:47.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.560+0000 7f30dcfb5700 1 librados: init done 2024-04-14T15:23:47.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.560+0000 7f30cf7fe700 1 -- 172.21.15.90:0/2302147943 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f30c0003b40 con 0x7f30d810a630 2024-04-14T15:23:47.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.560+0000 7f30cf7fe700 1 -- 172.21.15.90:0/2302147943 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f30c0004960 con 0x7f30d810a630 2024-04-14T15:23:47.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.561+0000 7f30dcfb5700 1 -- 172.21.15.90:0/2302147943 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f30b8005130 con 0x7f30d810a630 2024-04-14T15:23:47.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.562+0000 7f30cf7fe700 1 -- 172.21.15.90:0/2302147943 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f30c00036a0 con 0x7f30d810a630 2024-04-14T15:23:47.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.562+0000 7f30cf7fe700 1 -- 172.21.15.90:0/2302147943 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(685..685 src has 467..685) ==== 8729+0+0 (unknown 3028141388 0 0) 0x7f30c00897a0 con 0x7f30d810a630 2024-04-14T15:23:47.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.562+0000 7f30cf7fe700 10 client.37845.objecter ms_dispatch 0x7f30d8001550 osd_map(685..685 src has 467..685) 2024-04-14T15:23:47.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.562+0000 7f30cf7fe700 3 client.37845.objecter handle_osd_map got epochs [685,685] > 0 2024-04-14T15:23:47.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.562+0000 7f30cf7fe700 3 client.37845.objecter handle_osd_map decoding full epoch 685 2024-04-14T15:23:47.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.563+0000 7f30cf7fe700 20 client.37845.objecter dump_active .. 0 homeless 2024-04-14T15:23:47.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.563+0000 7f30cf7fe700 10 client.37845.objecter ms_handle_connect 0x7f30c4073710 2024-04-14T15:23:47.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.563+0000 7f30cf7fe700 1 -- 172.21.15.90:0/2302147943 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v685) ==== 54+0+16 (unknown 2766162906 0 2420000192) 0x7f30c0089be0 con 0x7f30d810a630 2024-04-14T15:23:47.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.563+0000 7f30dcfb5700 10 client.37845.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:23:47.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.563+0000 7f30dcfb5700 15 client.37845.objecter _session_command_op_assign -1 1 2024-04-14T15:23:47.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.563+0000 7f30dcfb5700 20 client.37845.objecter _get_session s=0x7f30b8005410 osd=0 3 2024-04-14T15:23:47.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.563+0000 7f30dcfb5700 20 client.37845.objecter put_session s=0x7f30b8005410 osd=0 3 2024-04-14T15:23:47.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.563+0000 7f30dcfb5700 20 client.37845.objecter _get_session s=0x7f30b8005410 osd=0 3 2024-04-14T15:23:47.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.563+0000 7f30dcfb5700 15 client.37845.objecter _session_command_op_remove -1 1 2024-04-14T15:23:47.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.563+0000 7f30dcfb5700 20 client.37845.objecter get_session s=0x7f30b8005410 osd=0 3 2024-04-14T15:23:47.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.563+0000 7f30dcfb5700 15 client.37845.objecter _session_command_op_assign 0 1 2024-04-14T15:23:47.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.563+0000 7f30dcfb5700 20 client.37845.objecter put_session s=0x7f30b8005410 osd=0 4 2024-04-14T15:23:47.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.563+0000 7f30dcfb5700 10 client.37845.objecter _send_command 1 2024-04-14T15:23:47.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.563+0000 7f30dcfb5700 1 -- 172.21.15.90:0/2302147943 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f30b8004ba0 con 0x7f30b8004760 2024-04-14T15:23:47.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.563+0000 7f30cf7fe700 10 client.37845.objecter ms_handle_connect 0x7f30b8004760 2024-04-14T15:23:47.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.564+0000 7f30cf7fe700 1 -- 172.21.15.90:0/2302147943 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f30b8004ba0 con 0x7f30b8004760 2024-04-14T15:23:47.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.564+0000 7f30cf7fe700 10 client.37845.objecter ms_dispatch 0x7f30d8001550 command_reply(tid 1: 0 ) 2024-04-14T15:23:47.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.564+0000 7f30cf7fe700 10 client.37845.objecter _finish_command 1 = system:0 2024-04-14T15:23:47.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.564+0000 7f30cf7fe700 20 client.37845.objecter put_session s=0x7f30b8005410 osd=0 4 2024-04-14T15:23:47.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.564+0000 7f30cf7fe700 15 client.37845.objecter _session_command_op_remove 0 1 2024-04-14T15:23:47.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.582+0000 7f30dcfb5700 10 client.37845.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:23:47.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.582+0000 7f30dcfb5700 15 client.37845.objecter _session_command_op_assign -1 2 2024-04-14T15:23:47.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.582+0000 7f30dcfb5700 20 client.37845.objecter _get_session s=0x7f30b8005410 osd=0 3 2024-04-14T15:23:47.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.582+0000 7f30dcfb5700 20 client.37845.objecter put_session s=0x7f30b8005410 osd=0 3 2024-04-14T15:23:47.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.582+0000 7f30dcfb5700 20 client.37845.objecter _get_session s=0x7f30b8005410 osd=0 3 2024-04-14T15:23:47.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.582+0000 7f30dcfb5700 15 client.37845.objecter _session_command_op_remove -1 2 2024-04-14T15:23:47.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.582+0000 7f30dcfb5700 20 client.37845.objecter get_session s=0x7f30b8005410 osd=0 3 2024-04-14T15:23:47.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.582+0000 7f30dcfb5700 15 client.37845.objecter _session_command_op_assign 0 2 2024-04-14T15:23:47.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.582+0000 7f30dcfb5700 20 client.37845.objecter put_session s=0x7f30b8005410 osd=0 4 2024-04-14T15:23:47.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.582+0000 7f30dcfb5700 10 client.37845.objecter _send_command 2 2024-04-14T15:23:47.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.582+0000 7f30dcfb5700 1 -- 172.21.15.90:0/2302147943 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f30b8007a20 con 0x7f30b8004760 2024-04-14T15:23:47.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.583+0000 7f30cf7fe700 1 -- 172.21.15.90:0/2302147943 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f30b8007a20 con 0x7f30b8004760 2024-04-14T15:23:47.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.583+0000 7f30cf7fe700 10 client.37845.objecter ms_dispatch 0x7f30d8001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:23:47.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.583+0000 7f30cf7fe700 10 client.37845.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:23:47.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.583+0000 7f30cf7fe700 20 client.37845.objecter put_session s=0x7f30b8005410 osd=0 4 2024-04-14T15:23:47.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.583+0000 7f30cf7fe700 15 client.37845.objecter _session_command_op_remove 0 2 2024-04-14T15:23:47.584 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:23:47.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.584+0000 7f30dcfb5700 10 client.37845.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:23:47.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.584+0000 7f30dcfb5700 15 client.37845.objecter _session_command_op_assign -1 3 2024-04-14T15:23:47.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.584+0000 7f30dcfb5700 20 client.37845.objecter _get_session s=0x7f30b8004160 osd=1 3 2024-04-14T15:23:47.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.584+0000 7f30dcfb5700 20 client.37845.objecter put_session s=0x7f30b8004160 osd=1 3 2024-04-14T15:23:47.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.584+0000 7f30dcfb5700 20 client.37845.objecter _get_session s=0x7f30b8004160 osd=1 3 2024-04-14T15:23:47.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.584+0000 7f30dcfb5700 15 client.37845.objecter _session_command_op_remove -1 3 2024-04-14T15:23:47.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.584+0000 7f30dcfb5700 20 client.37845.objecter get_session s=0x7f30b8004160 osd=1 3 2024-04-14T15:23:47.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.584+0000 7f30dcfb5700 15 client.37845.objecter _session_command_op_assign 1 3 2024-04-14T15:23:47.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.584+0000 7f30dcfb5700 20 client.37845.objecter put_session s=0x7f30b8004160 osd=1 4 2024-04-14T15:23:47.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.584+0000 7f30dcfb5700 10 client.37845.objecter _send_command 3 2024-04-14T15:23:47.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.584+0000 7f30dcfb5700 1 -- 172.21.15.90:0/2302147943 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f30b80022d0 con 0x7f30b8004340 2024-04-14T15:23:47.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.584+0000 7f30cf7fe700 10 client.37845.objecter ms_handle_connect 0x7f30b8004340 2024-04-14T15:23:47.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.584+0000 7f30cf7fe700 1 -- 172.21.15.90:0/2302147943 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f30b80022d0 con 0x7f30b8004340 2024-04-14T15:23:47.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.585+0000 7f30cf7fe700 10 client.37845.objecter ms_dispatch 0x7f30d8001550 command_reply(tid 3: 0 ) 2024-04-14T15:23:47.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.585+0000 7f30cf7fe700 10 client.37845.objecter _finish_command 3 = system:0 2024-04-14T15:23:47.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.585+0000 7f30cf7fe700 20 client.37845.objecter put_session s=0x7f30b8004160 osd=1 4 2024-04-14T15:23:47.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.585+0000 7f30cf7fe700 15 client.37845.objecter _session_command_op_remove 1 3 2024-04-14T15:23:47.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.601+0000 7f30dcfb5700 10 client.37845.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:23:47.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.601+0000 7f30dcfb5700 15 client.37845.objecter _session_command_op_assign -1 4 2024-04-14T15:23:47.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.601+0000 7f30dcfb5700 20 client.37845.objecter _get_session s=0x7f30b8004160 osd=1 3 2024-04-14T15:23:47.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.601+0000 7f30dcfb5700 20 client.37845.objecter put_session s=0x7f30b8004160 osd=1 3 2024-04-14T15:23:47.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.601+0000 7f30dcfb5700 20 client.37845.objecter _get_session s=0x7f30b8004160 osd=1 3 2024-04-14T15:23:47.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.601+0000 7f30dcfb5700 15 client.37845.objecter _session_command_op_remove -1 4 2024-04-14T15:23:47.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.601+0000 7f30dcfb5700 20 client.37845.objecter get_session s=0x7f30b8004160 osd=1 3 2024-04-14T15:23:47.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.601+0000 7f30dcfb5700 15 client.37845.objecter _session_command_op_assign 1 4 2024-04-14T15:23:47.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.601+0000 7f30dcfb5700 20 client.37845.objecter put_session s=0x7f30b8004160 osd=1 4 2024-04-14T15:23:47.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.601+0000 7f30dcfb5700 10 client.37845.objecter _send_command 4 2024-04-14T15:23:47.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.601+0000 7f30dcfb5700 1 -- 172.21.15.90:0/2302147943 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f30b8001750 con 0x7f30b8004340 2024-04-14T15:23:47.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.603+0000 7f30cf7fe700 1 -- 172.21.15.90:0/2302147943 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f30b8001750 con 0x7f30b8004340 2024-04-14T15:23:47.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.603+0000 7f30cf7fe700 10 client.37845.objecter ms_dispatch 0x7f30d8001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:23:47.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.603+0000 7f30cf7fe700 10 client.37845.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:23:47.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.603+0000 7f30cf7fe700 20 client.37845.objecter put_session s=0x7f30b8004160 osd=1 4 2024-04-14T15:23:47.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.603+0000 7f30cf7fe700 15 client.37845.objecter _session_command_op_remove 1 4 2024-04-14T15:23:47.604 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:23:47.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.603+0000 7f30dcfb5700 10 client.37845.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:23:47.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.603+0000 7f30dcfb5700 15 client.37845.objecter _session_command_op_assign -1 5 2024-04-14T15:23:47.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.603+0000 7f30dcfb5700 20 client.37845.objecter _get_session s=0x7f30b80019b0 osd=2 3 2024-04-14T15:23:47.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.603+0000 7f30dcfb5700 20 client.37845.objecter put_session s=0x7f30b80019b0 osd=2 3 2024-04-14T15:23:47.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.603+0000 7f30dcfb5700 20 client.37845.objecter _get_session s=0x7f30b80019b0 osd=2 3 2024-04-14T15:23:47.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.603+0000 7f30dcfb5700 15 client.37845.objecter _session_command_op_remove -1 5 2024-04-14T15:23:47.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.603+0000 7f30dcfb5700 20 client.37845.objecter get_session s=0x7f30b80019b0 osd=2 3 2024-04-14T15:23:47.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.603+0000 7f30dcfb5700 15 client.37845.objecter _session_command_op_assign 2 5 2024-04-14T15:23:47.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.603+0000 7f30dcfb5700 20 client.37845.objecter put_session s=0x7f30b80019b0 osd=2 4 2024-04-14T15:23:47.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.603+0000 7f30dcfb5700 10 client.37845.objecter _send_command 5 2024-04-14T15:23:47.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.603+0000 7f30dcfb5700 1 -- 172.21.15.90:0/2302147943 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f30b800e1a0 con 0x7f30b800d600 2024-04-14T15:23:47.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.603+0000 7f30cf7fe700 10 client.37845.objecter ms_handle_connect 0x7f30b800d600 2024-04-14T15:23:47.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.604+0000 7f30cf7fe700 1 -- 172.21.15.90:0/2302147943 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f30b800e1a0 con 0x7f30b800d600 2024-04-14T15:23:47.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.604+0000 7f30cf7fe700 10 client.37845.objecter ms_dispatch 0x7f30d8001550 command_reply(tid 5: 0 ) 2024-04-14T15:23:47.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.604+0000 7f30cf7fe700 10 client.37845.objecter _finish_command 5 = system:0 2024-04-14T15:23:47.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.604+0000 7f30cf7fe700 20 client.37845.objecter put_session s=0x7f30b80019b0 osd=2 4 2024-04-14T15:23:47.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.604+0000 7f30cf7fe700 15 client.37845.objecter _session_command_op_remove 2 5 2024-04-14T15:23:47.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.620+0000 7f30dcfb5700 10 client.37845.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:23:47.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.620+0000 7f30dcfb5700 15 client.37845.objecter _session_command_op_assign -1 6 2024-04-14T15:23:47.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.620+0000 7f30dcfb5700 20 client.37845.objecter _get_session s=0x7f30b80019b0 osd=2 3 2024-04-14T15:23:47.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.620+0000 7f30dcfb5700 20 client.37845.objecter put_session s=0x7f30b80019b0 osd=2 3 2024-04-14T15:23:47.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.620+0000 7f30dcfb5700 20 client.37845.objecter _get_session s=0x7f30b80019b0 osd=2 3 2024-04-14T15:23:47.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.620+0000 7f30dcfb5700 15 client.37845.objecter _session_command_op_remove -1 6 2024-04-14T15:23:47.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.620+0000 7f30dcfb5700 20 client.37845.objecter get_session s=0x7f30b80019b0 osd=2 3 2024-04-14T15:23:47.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.620+0000 7f30dcfb5700 15 client.37845.objecter _session_command_op_assign 2 6 2024-04-14T15:23:47.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.620+0000 7f30dcfb5700 20 client.37845.objecter put_session s=0x7f30b80019b0 osd=2 4 2024-04-14T15:23:47.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.620+0000 7f30dcfb5700 10 client.37845.objecter _send_command 6 2024-04-14T15:23:47.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.620+0000 7f30dcfb5700 1 -- 172.21.15.90:0/2302147943 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f30b800eb20 con 0x7f30b800d600 2024-04-14T15:23:47.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.621+0000 7f30cf7fe700 1 -- 172.21.15.90:0/2302147943 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f30b800eb20 con 0x7f30b800d600 2024-04-14T15:23:47.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.621+0000 7f30cf7fe700 10 client.37845.objecter ms_dispatch 0x7f30d8001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:23:47.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.621+0000 7f30cf7fe700 10 client.37845.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:23:47.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.622+0000 7f30cf7fe700 20 client.37845.objecter put_session s=0x7f30b80019b0 osd=2 4 2024-04-14T15:23:47.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.622+0000 7f30cf7fe700 15 client.37845.objecter _session_command_op_remove 2 6 2024-04-14T15:23:47.622 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:23:47.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.622+0000 7f30dcfb5700 10 client.37845.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:23:47.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.622+0000 7f30dcfb5700 15 client.37845.objecter _session_command_op_assign -1 7 2024-04-14T15:23:47.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.622+0000 7f30dcfb5700 20 client.37845.objecter _get_session s=0x7f30b8008380 osd=3 3 2024-04-14T15:23:47.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.622+0000 7f30dcfb5700 20 client.37845.objecter put_session s=0x7f30b8008380 osd=3 3 2024-04-14T15:23:47.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.622+0000 7f30dcfb5700 20 client.37845.objecter _get_session s=0x7f30b8008380 osd=3 3 2024-04-14T15:23:47.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.622+0000 7f30dcfb5700 15 client.37845.objecter _session_command_op_remove -1 7 2024-04-14T15:23:47.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.622+0000 7f30dcfb5700 20 client.37845.objecter get_session s=0x7f30b8008380 osd=3 3 2024-04-14T15:23:47.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.622+0000 7f30dcfb5700 15 client.37845.objecter _session_command_op_assign 3 7 2024-04-14T15:23:47.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.622+0000 7f30dcfb5700 20 client.37845.objecter put_session s=0x7f30b8008380 osd=3 4 2024-04-14T15:23:47.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.622+0000 7f30dcfb5700 10 client.37845.objecter _send_command 7 2024-04-14T15:23:47.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.622+0000 7f30dcfb5700 1 -- 172.21.15.90:0/2302147943 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f30b801a300 con 0x7f30b80166e0 2024-04-14T15:23:47.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.622+0000 7f30cf7fe700 10 client.37845.objecter ms_handle_connect 0x7f30b80166e0 2024-04-14T15:23:47.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.623+0000 7f30cf7fe700 1 -- 172.21.15.90:0/2302147943 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f30b801a300 con 0x7f30b80166e0 2024-04-14T15:23:47.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.623+0000 7f30cf7fe700 10 client.37845.objecter ms_dispatch 0x7f30d8001550 command_reply(tid 7: 0 ) 2024-04-14T15:23:47.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.623+0000 7f30cf7fe700 10 client.37845.objecter _finish_command 7 = system:0 2024-04-14T15:23:47.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.623+0000 7f30cf7fe700 20 client.37845.objecter put_session s=0x7f30b8008380 osd=3 4 2024-04-14T15:23:47.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.623+0000 7f30cf7fe700 15 client.37845.objecter _session_command_op_remove 3 7 2024-04-14T15:23:47.629 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:23:47.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.639+0000 7f30dcfb5700 10 client.37845.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:23:47.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.639+0000 7f30dcfb5700 15 client.37845.objecter _session_command_op_assign -1 8 2024-04-14T15:23:47.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.639+0000 7f30dcfb5700 20 client.37845.objecter _get_session s=0x7f30b8008380 osd=3 3 2024-04-14T15:23:47.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.639+0000 7f30dcfb5700 20 client.37845.objecter put_session s=0x7f30b8008380 osd=3 3 2024-04-14T15:23:47.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.639+0000 7f30dcfb5700 20 client.37845.objecter _get_session s=0x7f30b8008380 osd=3 3 2024-04-14T15:23:47.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.639+0000 7f30dcfb5700 15 client.37845.objecter _session_command_op_remove -1 8 2024-04-14T15:23:47.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.639+0000 7f30dcfb5700 20 client.37845.objecter get_session s=0x7f30b8008380 osd=3 3 2024-04-14T15:23:47.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.639+0000 7f30dcfb5700 15 client.37845.objecter _session_command_op_assign 3 8 2024-04-14T15:23:47.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.639+0000 7f30dcfb5700 20 client.37845.objecter put_session s=0x7f30b8008380 osd=3 4 2024-04-14T15:23:47.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.639+0000 7f30dcfb5700 10 client.37845.objecter _send_command 8 2024-04-14T15:23:47.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.639+0000 7f30dcfb5700 1 -- 172.21.15.90:0/2302147943 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f30b800e5a0 con 0x7f30b80166e0 2024-04-14T15:23:47.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.641+0000 7f30cf7fe700 1 -- 172.21.15.90:0/2302147943 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f30b800e5a0 con 0x7f30b80166e0 2024-04-14T15:23:47.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.641+0000 7f30cf7fe700 10 client.37845.objecter ms_dispatch 0x7f30d8001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:23:47.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.641+0000 7f30cf7fe700 10 client.37845.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:23:47.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.641+0000 7f30cf7fe700 20 client.37845.objecter put_session s=0x7f30b8008380 osd=3 4 2024-04-14T15:23:47.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.641+0000 7f30cf7fe700 15 client.37845.objecter _session_command_op_remove 3 8 2024-04-14T15:23:47.641 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:23:47.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.641+0000 7f30dcfb5700 10 client.37845.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:23:47.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.641+0000 7f30dcfb5700 15 client.37845.objecter _session_command_op_assign -1 9 2024-04-14T15:23:47.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.641+0000 7f30dcfb5700 20 client.37845.objecter _get_session s=0x7f30b800e350 osd=4 3 2024-04-14T15:23:47.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.641+0000 7f30dcfb5700 20 client.37845.objecter put_session s=0x7f30b800e350 osd=4 3 2024-04-14T15:23:47.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.641+0000 7f30dcfb5700 20 client.37845.objecter _get_session s=0x7f30b800e350 osd=4 3 2024-04-14T15:23:47.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.641+0000 7f30dcfb5700 15 client.37845.objecter _session_command_op_remove -1 9 2024-04-14T15:23:47.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.641+0000 7f30dcfb5700 20 client.37845.objecter get_session s=0x7f30b800e350 osd=4 3 2024-04-14T15:23:47.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.641+0000 7f30dcfb5700 15 client.37845.objecter _session_command_op_assign 4 9 2024-04-14T15:23:47.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.641+0000 7f30dcfb5700 20 client.37845.objecter put_session s=0x7f30b800e350 osd=4 4 2024-04-14T15:23:47.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.641+0000 7f30dcfb5700 10 client.37845.objecter _send_command 9 2024-04-14T15:23:47.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.641+0000 7f30dcfb5700 1 -- 172.21.15.90:0/2302147943 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f30b800e4c0 con 0x7f30b800d160 2024-04-14T15:23:47.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.641+0000 7f30cf7fe700 10 client.37845.objecter ms_handle_connect 0x7f30b800d160 2024-04-14T15:23:47.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.642+0000 7f30cf7fe700 1 -- 172.21.15.90:0/2302147943 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f30b800e4c0 con 0x7f30b800d160 2024-04-14T15:23:47.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.642+0000 7f30cf7fe700 10 client.37845.objecter ms_dispatch 0x7f30d8001550 command_reply(tid 9: 0 ) 2024-04-14T15:23:47.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.642+0000 7f30cf7fe700 10 client.37845.objecter _finish_command 9 = system:0 2024-04-14T15:23:47.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.642+0000 7f30cf7fe700 20 client.37845.objecter put_session s=0x7f30b800e350 osd=4 4 2024-04-14T15:23:47.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.642+0000 7f30cf7fe700 15 client.37845.objecter _session_command_op_remove 4 9 2024-04-14T15:23:47.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:47.657 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:47.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:23:47.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.658+0000 7f30dcfb5700 10 client.37845.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:23:47.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.658+0000 7f30dcfb5700 15 client.37845.objecter _session_command_op_assign -1 10 2024-04-14T15:23:47.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.658+0000 7f30dcfb5700 20 client.37845.objecter _get_session s=0x7f30b800e350 osd=4 3 2024-04-14T15:23:47.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.658+0000 7f30dcfb5700 20 client.37845.objecter put_session s=0x7f30b800e350 osd=4 3 2024-04-14T15:23:47.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.658+0000 7f30dcfb5700 20 client.37845.objecter _get_session s=0x7f30b800e350 osd=4 3 2024-04-14T15:23:47.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.658+0000 7f30dcfb5700 15 client.37845.objecter _session_command_op_remove -1 10 2024-04-14T15:23:47.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.658+0000 7f30dcfb5700 20 client.37845.objecter get_session s=0x7f30b800e350 osd=4 3 2024-04-14T15:23:47.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.658+0000 7f30dcfb5700 15 client.37845.objecter _session_command_op_assign 4 10 2024-04-14T15:23:47.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.658+0000 7f30dcfb5700 20 client.37845.objecter put_session s=0x7f30b800e350 osd=4 4 2024-04-14T15:23:47.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.658+0000 7f30dcfb5700 10 client.37845.objecter _send_command 10 2024-04-14T15:23:47.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.658+0000 7f30dcfb5700 1 -- 172.21.15.90:0/2302147943 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f30b800e4c0 con 0x7f30b800d160 2024-04-14T15:23:47.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.660+0000 7f30cf7fe700 1 -- 172.21.15.90:0/2302147943 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f30d000f020 con 0x7f30b800d160 2024-04-14T15:23:47.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.660+0000 7f30cf7fe700 10 client.37845.objecter ms_dispatch 0x7f30d8001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:23:47.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.660+0000 7f30cf7fe700 10 client.37845.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:23:47.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.660+0000 7f30cf7fe700 20 client.37845.objecter put_session s=0x7f30b800e350 osd=4 4 2024-04-14T15:23:47.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.660+0000 7f30cf7fe700 15 client.37845.objecter _session_command_op_remove 4 10 2024-04-14T15:23:47.661 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:23:47.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.661+0000 7f30dcfb5700 10 client.37845.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:23:47.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.661+0000 7f30dcfb5700 15 client.37845.objecter _session_command_op_assign -1 11 2024-04-14T15:23:47.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.661+0000 7f30dcfb5700 20 client.37845.objecter _get_session s=0x7f30b801eeb0 osd=5 3 2024-04-14T15:23:47.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.661+0000 7f30dcfb5700 20 client.37845.objecter put_session s=0x7f30b801eeb0 osd=5 3 2024-04-14T15:23:47.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.661+0000 7f30dcfb5700 20 client.37845.objecter _get_session s=0x7f30b801eeb0 osd=5 3 2024-04-14T15:23:47.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.661+0000 7f30dcfb5700 15 client.37845.objecter _session_command_op_remove -1 11 2024-04-14T15:23:47.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.661+0000 7f30dcfb5700 20 client.37845.objecter get_session s=0x7f30b801eeb0 osd=5 3 2024-04-14T15:23:47.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.661+0000 7f30dcfb5700 15 client.37845.objecter _session_command_op_assign 5 11 2024-04-14T15:23:47.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.661+0000 7f30dcfb5700 20 client.37845.objecter put_session s=0x7f30b801eeb0 osd=5 4 2024-04-14T15:23:47.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.661+0000 7f30dcfb5700 10 client.37845.objecter _send_command 11 2024-04-14T15:23:47.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.661+0000 7f30dcfb5700 1 -- 172.21.15.90:0/2302147943 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f30b8023210 con 0x7f30b801f530 2024-04-14T15:23:47.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.661+0000 7f30cf7fe700 10 client.37845.objecter ms_handle_connect 0x7f30b801f530 2024-04-14T15:23:47.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.662+0000 7f30cf7fe700 1 -- 172.21.15.90:0/2302147943 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f30b8023210 con 0x7f30b801f530 2024-04-14T15:23:47.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.662+0000 7f30cf7fe700 10 client.37845.objecter ms_dispatch 0x7f30d8001550 command_reply(tid 11: 0 ) 2024-04-14T15:23:47.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.662+0000 7f30cf7fe700 10 client.37845.objecter _finish_command 11 = system:0 2024-04-14T15:23:47.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.662+0000 7f30cf7fe700 20 client.37845.objecter put_session s=0x7f30b801eeb0 osd=5 4 2024-04-14T15:23:47.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.662+0000 7f30cf7fe700 15 client.37845.objecter _session_command_op_remove 5 11 2024-04-14T15:23:47.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.678+0000 7f30dcfb5700 10 client.37845.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:23:47.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.678+0000 7f30dcfb5700 15 client.37845.objecter _session_command_op_assign -1 12 2024-04-14T15:23:47.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.678+0000 7f30dcfb5700 20 client.37845.objecter _get_session s=0x7f30b801eeb0 osd=5 3 2024-04-14T15:23:47.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.678+0000 7f30dcfb5700 20 client.37845.objecter put_session s=0x7f30b801eeb0 osd=5 3 2024-04-14T15:23:47.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.678+0000 7f30dcfb5700 20 client.37845.objecter _get_session s=0x7f30b801eeb0 osd=5 3 2024-04-14T15:23:47.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.678+0000 7f30dcfb5700 15 client.37845.objecter _session_command_op_remove -1 12 2024-04-14T15:23:47.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.678+0000 7f30dcfb5700 20 client.37845.objecter get_session s=0x7f30b801eeb0 osd=5 3 2024-04-14T15:23:47.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.678+0000 7f30dcfb5700 15 client.37845.objecter _session_command_op_assign 5 12 2024-04-14T15:23:47.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.678+0000 7f30dcfb5700 20 client.37845.objecter put_session s=0x7f30b801eeb0 osd=5 4 2024-04-14T15:23:47.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.678+0000 7f30dcfb5700 10 client.37845.objecter _send_command 12 2024-04-14T15:23:47.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.678+0000 7f30dcfb5700 1 -- 172.21.15.90:0/2302147943 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f30b8024bc0 con 0x7f30b801f530 2024-04-14T15:23:47.681 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:23:47.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.680+0000 7f30cf7fe700 1 -- 172.21.15.90:0/2302147943 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f30b8024bc0 con 0x7f30b801f530 2024-04-14T15:23:47.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.680+0000 7f30cf7fe700 10 client.37845.objecter ms_dispatch 0x7f30d8001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:23:47.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.680+0000 7f30cf7fe700 10 client.37845.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:23:47.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.680+0000 7f30cf7fe700 20 client.37845.objecter put_session s=0x7f30b801eeb0 osd=5 4 2024-04-14T15:23:47.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.680+0000 7f30cf7fe700 15 client.37845.objecter _session_command_op_remove 5 12 2024-04-14T15:23:47.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.681+0000 7f30dcfb5700 10 client.37845.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:23:47.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.681+0000 7f30dcfb5700 15 client.37845.objecter _session_command_op_assign -1 13 2024-04-14T15:23:47.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.681+0000 7f30dcfb5700 20 client.37845.objecter _get_session s=0x7f30b8024bc0 osd=6 3 2024-04-14T15:23:47.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.681+0000 7f30dcfb5700 20 client.37845.objecter put_session s=0x7f30b8024bc0 osd=6 3 2024-04-14T15:23:47.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.681+0000 7f30dcfb5700 20 client.37845.objecter _get_session s=0x7f30b8024bc0 osd=6 3 2024-04-14T15:23:47.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.681+0000 7f30dcfb5700 15 client.37845.objecter _session_command_op_remove -1 13 2024-04-14T15:23:47.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.681+0000 7f30dcfb5700 20 client.37845.objecter get_session s=0x7f30b8024bc0 osd=6 3 2024-04-14T15:23:47.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.681+0000 7f30dcfb5700 15 client.37845.objecter _session_command_op_assign 6 13 2024-04-14T15:23:47.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.681+0000 7f30dcfb5700 20 client.37845.objecter put_session s=0x7f30b8024bc0 osd=6 4 2024-04-14T15:23:47.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.681+0000 7f30dcfb5700 10 client.37845.objecter _send_command 13 2024-04-14T15:23:47.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.681+0000 7f30dcfb5700 1 -- 172.21.15.90:0/2302147943 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f30b80249d0 con 0x7f30b80252e0 2024-04-14T15:23:47.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.681+0000 7f30cf7fe700 10 client.37845.objecter ms_handle_connect 0x7f30b80252e0 2024-04-14T15:23:47.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.682+0000 7f30cf7fe700 1 -- 172.21.15.90:0/2302147943 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f30c800cb70 con 0x7f30b80252e0 2024-04-14T15:23:47.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.682+0000 7f30cf7fe700 10 client.37845.objecter ms_dispatch 0x7f30d8001550 command_reply(tid 13: 0 ) 2024-04-14T15:23:47.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.682+0000 7f30cf7fe700 10 client.37845.objecter _finish_command 13 = system:0 2024-04-14T15:23:47.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.682+0000 7f30cf7fe700 20 client.37845.objecter put_session s=0x7f30b8024bc0 osd=6 4 2024-04-14T15:23:47.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.682+0000 7f30cf7fe700 15 client.37845.objecter _session_command_op_remove 6 13 2024-04-14T15:23:47.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.698+0000 7f30dcfb5700 10 client.37845.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:23:47.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.698+0000 7f30dcfb5700 15 client.37845.objecter _session_command_op_assign -1 14 2024-04-14T15:23:47.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.698+0000 7f30dcfb5700 20 client.37845.objecter _get_session s=0x7f30b8024bc0 osd=6 3 2024-04-14T15:23:47.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.698+0000 7f30dcfb5700 20 client.37845.objecter put_session s=0x7f30b8024bc0 osd=6 3 2024-04-14T15:23:47.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.698+0000 7f30dcfb5700 20 client.37845.objecter _get_session s=0x7f30b8024bc0 osd=6 3 2024-04-14T15:23:47.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.698+0000 7f30dcfb5700 15 client.37845.objecter _session_command_op_remove -1 14 2024-04-14T15:23:47.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.698+0000 7f30dcfb5700 20 client.37845.objecter get_session s=0x7f30b8024bc0 osd=6 3 2024-04-14T15:23:47.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.698+0000 7f30dcfb5700 15 client.37845.objecter _session_command_op_assign 6 14 2024-04-14T15:23:47.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.698+0000 7f30dcfb5700 20 client.37845.objecter put_session s=0x7f30b8024bc0 osd=6 4 2024-04-14T15:23:47.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.698+0000 7f30dcfb5700 10 client.37845.objecter _send_command 14 2024-04-14T15:23:47.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.698+0000 7f30dcfb5700 1 -- 172.21.15.90:0/2302147943 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f30b80258d0 con 0x7f30b80252e0 2024-04-14T15:23:47.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.700+0000 7f30cf7fe700 1 -- 172.21.15.90:0/2302147943 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f30b80258d0 con 0x7f30b80252e0 2024-04-14T15:23:47.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.700+0000 7f30cf7fe700 10 client.37845.objecter ms_dispatch 0x7f30d8001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:23:47.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.700+0000 7f30cf7fe700 10 client.37845.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:23:47.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.700+0000 7f30cf7fe700 20 client.37845.objecter put_session s=0x7f30b8024bc0 osd=6 4 2024-04-14T15:23:47.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.700+0000 7f30cf7fe700 15 client.37845.objecter _session_command_op_remove 6 14 2024-04-14T15:23:47.701 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:23:47.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.701+0000 7f30dcfb5700 10 client.37845.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:23:47.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.701+0000 7f30dcfb5700 15 client.37845.objecter _session_command_op_assign -1 15 2024-04-14T15:23:47.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.701+0000 7f30dcfb5700 20 client.37845.objecter _get_session s=0x7f30b80258d0 osd=7 3 2024-04-14T15:23:47.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.701+0000 7f30dcfb5700 20 client.37845.objecter put_session s=0x7f30b80258d0 osd=7 3 2024-04-14T15:23:47.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.701+0000 7f30dcfb5700 20 client.37845.objecter _get_session s=0x7f30b80258d0 osd=7 3 2024-04-14T15:23:47.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.701+0000 7f30dcfb5700 15 client.37845.objecter _session_command_op_remove -1 15 2024-04-14T15:23:47.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.701+0000 7f30dcfb5700 20 client.37845.objecter get_session s=0x7f30b80258d0 osd=7 3 2024-04-14T15:23:47.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.701+0000 7f30dcfb5700 15 client.37845.objecter _session_command_op_assign 7 15 2024-04-14T15:23:47.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.701+0000 7f30dcfb5700 20 client.37845.objecter put_session s=0x7f30b80258d0 osd=7 4 2024-04-14T15:23:47.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.701+0000 7f30dcfb5700 10 client.37845.objecter _send_command 15 2024-04-14T15:23:47.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.701+0000 7f30dcfb5700 1 -- 172.21.15.90:0/2302147943 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f30b802bde0 con 0x7f30b80282d0 2024-04-14T15:23:47.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.701+0000 7f30cf7fe700 10 client.37845.objecter ms_handle_connect 0x7f30b80282d0 2024-04-14T15:23:47.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.702+0000 7f30cf7fe700 1 -- 172.21.15.90:0/2302147943 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f30b802bde0 con 0x7f30b80282d0 2024-04-14T15:23:47.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.702+0000 7f30cf7fe700 10 client.37845.objecter ms_dispatch 0x7f30d8001550 command_reply(tid 15: 0 ) 2024-04-14T15:23:47.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.702+0000 7f30cf7fe700 10 client.37845.objecter _finish_command 15 = system:0 2024-04-14T15:23:47.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.702+0000 7f30cf7fe700 20 client.37845.objecter put_session s=0x7f30b80258d0 osd=7 4 2024-04-14T15:23:47.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.702+0000 7f30cf7fe700 15 client.37845.objecter _session_command_op_remove 7 15 2024-04-14T15:23:47.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.718+0000 7f30dcfb5700 10 client.37845.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:23:47.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.718+0000 7f30dcfb5700 15 client.37845.objecter _session_command_op_assign -1 16 2024-04-14T15:23:47.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.718+0000 7f30dcfb5700 20 client.37845.objecter _get_session s=0x7f30b80258d0 osd=7 3 2024-04-14T15:23:47.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.718+0000 7f30dcfb5700 20 client.37845.objecter put_session s=0x7f30b80258d0 osd=7 3 2024-04-14T15:23:47.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.718+0000 7f30dcfb5700 20 client.37845.objecter _get_session s=0x7f30b80258d0 osd=7 3 2024-04-14T15:23:47.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.718+0000 7f30dcfb5700 15 client.37845.objecter _session_command_op_remove -1 16 2024-04-14T15:23:47.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.718+0000 7f30dcfb5700 20 client.37845.objecter get_session s=0x7f30b80258d0 osd=7 3 2024-04-14T15:23:47.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.718+0000 7f30dcfb5700 15 client.37845.objecter _session_command_op_assign 7 16 2024-04-14T15:23:47.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.718+0000 7f30dcfb5700 20 client.37845.objecter put_session s=0x7f30b80258d0 osd=7 4 2024-04-14T15:23:47.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.718+0000 7f30dcfb5700 10 client.37845.objecter _send_command 16 2024-04-14T15:23:47.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.718+0000 7f30dcfb5700 1 -- 172.21.15.90:0/2302147943 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f30b800e730 con 0x7f30b80282d0 2024-04-14T15:23:47.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.720+0000 7f30cf7fe700 1 -- 172.21.15.90:0/2302147943 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f30b800e730 con 0x7f30b80282d0 2024-04-14T15:23:47.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.720+0000 7f30cf7fe700 10 client.37845.objecter ms_dispatch 0x7f30d8001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:23:47.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.720+0000 7f30cf7fe700 10 client.37845.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:23:47.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.720+0000 7f30cf7fe700 20 client.37845.objecter put_session s=0x7f30b80258d0 osd=7 4 2024-04-14T15:23:47.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.720+0000 7f30cf7fe700 15 client.37845.objecter _session_command_op_remove 7 16 2024-04-14T15:23:47.721 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:23:47.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.720+0000 7f30dcfb5700 10 librados: watch_flush enter 2024-04-14T15:23:47.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.720+0000 7f30dcfb5700 10 librados: watch_flush exit 2024-04-14T15:23:47.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.720+0000 7f30dcfb5700 10 client.37845.objecter close_session for osd.0 2024-04-14T15:23:47.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.720+0000 7f30dcfb5700 1 -- 172.21.15.90:0/2302147943 >> v1:172.21.15.90:6801/3761806885 conn(0x7f30b8004760 legacy=0x7f30b8002780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:47.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.720+0000 7f30dcfb5700 20 client.37845.objecter put_session s=0x7f30b8005410 osd=0 1 2024-04-14T15:23:47.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.720+0000 7f30dcfb5700 10 client.37845.objecter close_session for osd.1 2024-04-14T15:23:47.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.720+0000 7f30dcfb5700 1 -- 172.21.15.90:0/2302147943 >> v1:172.21.15.90:6805/4170361699 conn(0x7f30b8004340 legacy=0x7f30b800ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:47.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.720+0000 7f30dcfb5700 20 client.37845.objecter put_session s=0x7f30b8004160 osd=1 1 2024-04-14T15:23:47.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.720+0000 7f30dcfb5700 10 client.37845.objecter close_session for osd.2 2024-04-14T15:23:47.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.720+0000 7f30dcfb5700 1 -- 172.21.15.90:0/2302147943 >> v1:172.21.15.90:6809/1686184251 conn(0x7f30b800d600 legacy=0x7f30b800da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:47.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.720+0000 7f30dcfb5700 20 client.37845.objecter put_session s=0x7f30b80019b0 osd=2 1 2024-04-14T15:23:47.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.720+0000 7f30dcfb5700 10 client.37845.objecter close_session for osd.3 2024-04-14T15:23:47.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.720+0000 7f30dcfb5700 1 -- 172.21.15.90:0/2302147943 >> v1:172.21.15.90:6813/2270166774 conn(0x7f30b80166e0 legacy=0x7f30b8018ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:47.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.720+0000 7f30dcfb5700 20 client.37845.objecter put_session s=0x7f30b8008380 osd=3 1 2024-04-14T15:23:47.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.720+0000 7f30dcfb5700 10 client.37845.objecter close_session for osd.4 2024-04-14T15:23:47.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.720+0000 7f30dcfb5700 1 -- 172.21.15.90:0/2302147943 >> v1:172.21.15.92:6800/3887549733 conn(0x7f30b800d160 legacy=0x7f30b801a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:47.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.720+0000 7f30dcfb5700 20 client.37845.objecter put_session s=0x7f30b800e350 osd=4 1 2024-04-14T15:23:47.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.720+0000 7f30dcfb5700 10 client.37845.objecter close_session for osd.5 2024-04-14T15:23:47.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.720+0000 7f30dcfb5700 1 -- 172.21.15.90:0/2302147943 >> v1:172.21.15.92:6804/3244561440 conn(0x7f30b801f530 legacy=0x7f30b8021960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:47.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.720+0000 7f30dcfb5700 20 client.37845.objecter put_session s=0x7f30b801eeb0 osd=5 1 2024-04-14T15:23:47.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.720+0000 7f30dcfb5700 10 client.37845.objecter close_session for osd.6 2024-04-14T15:23:47.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.720+0000 7f30dcfb5700 1 -- 172.21.15.90:0/2302147943 >> v1:172.21.15.92:6808/636125963 conn(0x7f30b80252e0 legacy=0x7f30b8023210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:47.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.720+0000 7f30dcfb5700 20 client.37845.objecter put_session s=0x7f30b8024bc0 osd=6 1 2024-04-14T15:23:47.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.720+0000 7f30dcfb5700 10 client.37845.objecter close_session for osd.7 2024-04-14T15:23:47.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.720+0000 7f30dcfb5700 1 -- 172.21.15.90:0/2302147943 >> v1:172.21.15.92:6812/1773706716 conn(0x7f30b80282d0 legacy=0x7f30b802a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:47.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.720+0000 7f30dcfb5700 20 client.37845.objecter put_session s=0x7f30b80258d0 osd=7 1 2024-04-14T15:23:47.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.720+0000 7f30dcfb5700 20 client.37845.objecter shutdown clearing up homeless session... 2024-04-14T15:23:47.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.720+0000 7f30dcfb5700 10 client.37845.objecter successfully canceled tick 2024-04-14T15:23:47.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.721+0000 7f30d6d9d700 1 -- 172.21.15.90:0/2302147943 reap_dead start 2024-04-14T15:23:47.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.721+0000 7f30dcfb5700 1 -- 172.21.15.90:0/2302147943 >> v1:172.21.15.90:6800/420935427 conn(0x7f30c4073710 legacy=0x7f30c4075bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:47.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.721+0000 7f30d6d9d700 1 -- 172.21.15.90:0/2302147943 reap_dead start 2024-04-14T15:23:47.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.721+0000 7f30dcfb5700 1 -- 172.21.15.90:0/2302147943 >> v1:172.21.15.90:6789/0 conn(0x7f30d810a630 legacy=0x7f30d810b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:47.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.721+0000 7f30d6d9d700 1 -- 172.21.15.90:0/2302147943 reap_dead start 2024-04-14T15:23:47.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.721+0000 7f30d6d9d700 1 -- 172.21.15.90:0/2302147943 reap_dead start 2024-04-14T15:23:47.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.721+0000 7f30d6d9d700 1 -- 172.21.15.90:0/2302147943 reap_dead start 2024-04-14T15:23:47.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.721+0000 7f30dcfb5700 1 -- 172.21.15.90:0/2302147943 shutdown_connections 2024-04-14T15:23:47.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.721+0000 7f30dcfb5700 1 -- 172.21.15.90:0/2302147943 >> 172.21.15.90:0/2302147943 conn(0x7f30d80fd2b0 msgr2=0x7f30d8111720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:23:47.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.721+0000 7f30dcfb5700 1 -- 172.21.15.90:0/2302147943 shutdown_connections 2024-04-14T15:23:47.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.721+0000 7f30dcfb5700 1 -- 172.21.15.90:0/2302147943 wait complete. 2024-04-14T15:23:47.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.721+0000 7f30dcfb5700 1 librados: shutdown 2024-04-14T15:23:47.725 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:23:47.725 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:23:47.725 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:23:47.725 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:23:47.725 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:23:47.725 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:23:47.725 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:23:47.725 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:23:47.758 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:23:47.781 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:47.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:47.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:23:47.883 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:23:47.906 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:47.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:47.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:23:47.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.993+0000 7f96f37fe700 1 -- 172.21.15.90:0/3985084990 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v686) ==== 87+0+0 (unknown 2170781951 0 0) 0x7f96f805b030 con 0x7f9704108880 2024-04-14T15:23:47.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.997+0000 7f9708e0c700 10 librados: watch_flush enter 2024-04-14T15:23:47.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.997+0000 7f9708e0c700 10 librados: watch_flush exit 2024-04-14T15:23:47.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.997+0000 7f9708e0c700 20 client.37839.objecter shutdown clearing up homeless session... 2024-04-14T15:23:47.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.997+0000 7f9708e0c700 10 client.37839.objecter successfully canceled tick 2024-04-14T15:23:47.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.997+0000 7f9708e0c700 1 -- 172.21.15.90:0/3985084990 >> v1:172.21.15.90:6800/420935427 conn(0x7f96ec0713f0 legacy=0x7f96ec0738b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:47.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.997+0000 7f9708e0c700 1 -- 172.21.15.90:0/3985084990 >> v1:172.21.15.90:6789/0 conn(0x7f9704108880 legacy=0x7f970410bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:47.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.997+0000 7f9708e0c700 1 -- 172.21.15.90:0/3985084990 shutdown_connections 2024-04-14T15:23:47.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.997+0000 7f9708e0c700 1 -- 172.21.15.90:0/3985084990 >> 172.21.15.90:0/3985084990 conn(0x7f97040fe3d0 msgr2=0x7f9704112840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:23:47.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.998+0000 7f9708e0c700 1 -- 172.21.15.90:0/3985084990 shutdown_connections 2024-04-14T15:23:47.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.999+0000 7f9708e0c700 1 -- 172.21.15.90:0/3985084990 wait complete. 2024-04-14T15:23:47.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:47.999+0000 7f9708e0c700 1 librados: shutdown 2024-04-14T15:23:48.002 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is set 2024-04-14T15:23:48.008 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:23:48.033 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:48.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:48.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:23:48.135 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:23:48.164 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:48.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:48.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:23:48.265 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:23:48.293 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:48.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:48.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:23:48.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:47 smithi092 ceph-mon[33966]: 4.15 scrub starts 2024-04-14T15:23:48.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:47 smithi092 ceph-mon[33966]: 4.15 scrub ok 2024-04-14T15:23:48.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:47 smithi092 ceph-mon[33966]: 2.19 deep-scrub starts 2024-04-14T15:23:48.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:47 smithi092 ceph-mon[33966]: 2.19 deep-scrub ok 2024-04-14T15:23:48.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:47 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-118", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:23:48.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:47 smithi092 ceph-mon[33966]: osdmap e685: 8 total, 8 up, 7 in 2024-04-14T15:23:48.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:47 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-118"}]: dispatch 2024-04-14T15:23:48.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:47 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-118"}]: dispatch 2024-04-14T15:23:48.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:47 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3985084990' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:23:48.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:47 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2302147943' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:23:48.395 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:23:48.418 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:48.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:48.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:23:48.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:47 smithi090 ceph-mon[41670]: 4.15 scrub starts 2024-04-14T15:23:48.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:47 smithi090 ceph-mon[41670]: 4.15 scrub ok 2024-04-14T15:23:48.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:47 smithi090 ceph-mon[41670]: 2.19 deep-scrub starts 2024-04-14T15:23:48.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:47 smithi090 ceph-mon[41670]: 2.19 deep-scrub ok 2024-04-14T15:23:48.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:47 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-118", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:23:48.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:47 smithi090 ceph-mon[41670]: osdmap e685: 8 total, 8 up, 7 in 2024-04-14T15:23:48.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:47 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-118"}]: dispatch 2024-04-14T15:23:48.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:47 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-118"}]: dispatch 2024-04-14T15:23:48.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:47 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3985084990' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:23:48.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:47 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2302147943' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:23:48.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:47 smithi090 ceph-mon[30399]: 4.15 scrub starts 2024-04-14T15:23:48.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:47 smithi090 ceph-mon[30399]: 4.15 scrub ok 2024-04-14T15:23:48.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:47 smithi090 ceph-mon[30399]: 2.19 deep-scrub starts 2024-04-14T15:23:48.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:47 smithi090 ceph-mon[30399]: 2.19 deep-scrub ok 2024-04-14T15:23:48.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:47 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-118", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:23:48.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:47 smithi090 ceph-mon[30399]: osdmap e685: 8 total, 8 up, 7 in 2024-04-14T15:23:48.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:47 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-118"}]: dispatch 2024-04-14T15:23:48.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:47 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-118"}]: dispatch 2024-04-14T15:23:48.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:47 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3985084990' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:23:48.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:47 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2302147943' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:23:48.520 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:23:48.547 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:48.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:48.548 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:23:48.649 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:23:48.675 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:48.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:48.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:23:48.776 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:23:48.801 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:48.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:48.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:23:48.903 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:23:48.927 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:48.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:48.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:23:49.030 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:23:49.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:49.059 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:49.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:23:49.117 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:49 smithi090 ceph-mon[30399]: pgmap v1011: 214 pgs: 32 creating+peering, 182 active+clean; 582 KiB data, 608 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:23:49.117 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:49 smithi090 ceph-mon[30399]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:23:49.118 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:49 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-118"}]': finished 2024-04-14T15:23:49.118 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:49 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3985084990' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-14T15:23:49.118 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:49 smithi090 ceph-mon[30399]: osdmap e686: 8 total, 8 up, 7 in 2024-04-14T15:23:49.118 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:49 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-118", "mode": "writeback"}]: dispatch 2024-04-14T15:23:49.118 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:49 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-118", "mode": "writeback"}]: dispatch 2024-04-14T15:23:49.118 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:49 smithi090 ceph-mon[41670]: pgmap v1011: 214 pgs: 32 creating+peering, 182 active+clean; 582 KiB data, 608 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:23:49.118 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:49 smithi090 ceph-mon[41670]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:23:49.118 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:49 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-118"}]': finished 2024-04-14T15:23:49.118 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:49 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3985084990' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-14T15:23:49.118 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:49 smithi090 ceph-mon[41670]: osdmap e686: 8 total, 8 up, 7 in 2024-04-14T15:23:49.118 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:49 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-118", "mode": "writeback"}]: dispatch 2024-04-14T15:23:49.118 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:49 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-118", "mode": "writeback"}]: dispatch 2024-04-14T15:23:49.159 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:23:49.183 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:49.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:49.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:23:49.284 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:23:49.312 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:49.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:49.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:23:49.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:49 smithi092 ceph-mon[33966]: pgmap v1011: 214 pgs: 32 creating+peering, 182 active+clean; 582 KiB data, 608 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:23:49.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:49 smithi092 ceph-mon[33966]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:23:49.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:49 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-118"}]': finished 2024-04-14T15:23:49.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:49 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3985084990' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-14T15:23:49.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:49 smithi092 ceph-mon[33966]: osdmap e686: 8 total, 8 up, 7 in 2024-04-14T15:23:49.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:49 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-118", "mode": "writeback"}]: dispatch 2024-04-14T15:23:49.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:49 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-118", "mode": "writeback"}]: dispatch 2024-04-14T15:23:49.414 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:23:49.441 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:49.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:49.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:23:49.543 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:23:49.567 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:49.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:49.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:23:49.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.586+0000 7f5deeed8700 1 -- 172.21.15.90:0/3908305525 >> v1:172.21.15.90:6789/0 conn(0x7f5de8074e50 legacy=0x7f5de8075230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:49.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.586+0000 7f5deeed8700 1 -- 172.21.15.90:0/3908305525 shutdown_connections 2024-04-14T15:23:49.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.586+0000 7f5deeed8700 1 -- 172.21.15.90:0/3908305525 >> 172.21.15.90:0/3908305525 conn(0x7f5de80fe3c0 msgr2=0x7f5de81007e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:23:49.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.586+0000 7f5deeed8700 1 -- 172.21.15.90:0/3908305525 shutdown_connections 2024-04-14T15:23:49.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.586+0000 7f5deeed8700 1 -- 172.21.15.90:0/3908305525 wait complete. 2024-04-14T15:23:49.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.586+0000 7f5deeed8700 1 librados: starting msgr at 2024-04-14T15:23:49.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.586+0000 7f5deeed8700 1 librados: starting objecter 2024-04-14T15:23:49.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.586+0000 7f5deeed8700 1 Processor -- start 2024-04-14T15:23:49.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.586+0000 7f5deeed8700 1 -- start start 2024-04-14T15:23:49.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.586+0000 7f5deeed8700 1 librados: setting wanted keys 2024-04-14T15:23:49.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.586+0000 7f5deeed8700 1 librados: calling monclient init 2024-04-14T15:23:49.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.587+0000 7f5deeed8700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5de81adeb0 con 0x7f5de8075940 2024-04-14T15:23:49.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.587+0000 7f5deeed8700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5de81af0d0 con 0x7f5de8074e50 2024-04-14T15:23:49.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.587+0000 7f5deeed8700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5de81b0310 con 0x7f5de8115ab0 2024-04-14T15:23:49.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.587+0000 7f5de7fff700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f5de8075940 0x7f5de81a8dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:57170/0 (socket says 172.21.15.90:57170) 2024-04-14T15:23:49.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.587+0000 7f5de7fff700 1 -- 172.21.15.90:0/258070968 learned_addr learned my addr 172.21.15.90:0/258070968 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:23:49.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.587+0000 7f5de5ffb700 10 client.?.objecter ms_handle_connect 0x7f5de8115ab0 2024-04-14T15:23:49.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.587+0000 7f5de5ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:23:49.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.587+0000 7f5de5ffb700 10 client.?.objecter ms_handle_connect 0x7f5de8075940 2024-04-14T15:23:49.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.587+0000 7f5de5ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:23:49.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.587+0000 7f5de5ffb700 1 -- 172.21.15.90:0/258070968 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 359358844 0 0) 0x7f5de81adeb0 con 0x7f5de8075940 2024-04-14T15:23:49.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.587+0000 7f5de5ffb700 1 -- 172.21.15.90:0/258070968 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5dd80031f0 con 0x7f5de8075940 2024-04-14T15:23:49.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.587+0000 7f5de5ffb700 10 client.?.objecter ms_handle_connect 0x7f5de8074e50 2024-04-14T15:23:49.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.587+0000 7f5de5ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:23:49.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.587+0000 7f5de5ffb700 1 -- 172.21.15.90:0/258070968 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3325879424 0 0) 0x7f5de81b0310 con 0x7f5de8115ab0 2024-04-14T15:23:49.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.587+0000 7f5de5ffb700 1 -- 172.21.15.90:0/258070968 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5de81adeb0 con 0x7f5de8115ab0 2024-04-14T15:23:49.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.587+0000 7f5de5ffb700 1 -- 172.21.15.90:0/258070968 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4281259956 0 0) 0x7f5de81af0d0 con 0x7f5de8074e50 2024-04-14T15:23:49.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.587+0000 7f5de5ffb700 1 -- 172.21.15.90:0/258070968 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5de81b0310 con 0x7f5de8074e50 2024-04-14T15:23:49.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.587+0000 7f5de5ffb700 1 -- 172.21.15.90:0/258070968 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 693012727 0 0) 0x7f5dd80031f0 con 0x7f5de8075940 2024-04-14T15:23:49.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.587+0000 7f5de5ffb700 1 -- 172.21.15.90:0/258070968 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5de81af0d0 con 0x7f5de8075940 2024-04-14T15:23:49.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.587+0000 7f5de5ffb700 1 -- 172.21.15.90:0/258070968 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f5dd40018d0 con 0x7f5de8075940 2024-04-14T15:23:49.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.587+0000 7f5de5ffb700 1 -- 172.21.15.90:0/258070968 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2505897066 0 0) 0x7f5de81adeb0 con 0x7f5de8115ab0 2024-04-14T15:23:49.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.587+0000 7f5de5ffb700 1 -- 172.21.15.90:0/258070968 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5dd80031f0 con 0x7f5de8115ab0 2024-04-14T15:23:49.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.587+0000 7f5de5ffb700 1 -- 172.21.15.90:0/258070968 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f5de0001850 con 0x7f5de8115ab0 2024-04-14T15:23:49.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.587+0000 7f5de5ffb700 1 -- 172.21.15.90:0/258070968 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2658659802 0 0) 0x7f5de81b0310 con 0x7f5de8074e50 2024-04-14T15:23:49.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.587+0000 7f5de5ffb700 1 -- 172.21.15.90:0/258070968 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5de81adeb0 con 0x7f5de8074e50 2024-04-14T15:23:49.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.587+0000 7f5de5ffb700 1 -- 172.21.15.90:0/258070968 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f5ddc0044b0 con 0x7f5de8074e50 2024-04-14T15:23:49.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.588+0000 7f5de5ffb700 1 -- 172.21.15.90:0/258070968 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2079327941 0 0) 0x7f5de81af0d0 con 0x7f5de8075940 2024-04-14T15:23:49.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.588+0000 7f5de5ffb700 1 -- 172.21.15.90:0/258070968 >> v1:172.21.15.90:6790/0 conn(0x7f5de8115ab0 legacy=0x7f5de81ac570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:49.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.588+0000 7f5de5ffb700 1 -- 172.21.15.90:0/258070968 >> v1:172.21.15.92:6789/0 conn(0x7f5de8074e50 legacy=0x7f5de81151e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:49.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.588+0000 7f5de5ffb700 1 -- 172.21.15.90:0/258070968 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5de81b1550 con 0x7f5de8075940 2024-04-14T15:23:49.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.588+0000 7f5deeed8700 1 -- 172.21.15.90:0/258070968 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5de81b0540 con 0x7f5de8075940 2024-04-14T15:23:49.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.588+0000 7f5de5ffb700 1 -- 172.21.15.90:0/258070968 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f5dd4003b40 con 0x7f5de8075940 2024-04-14T15:23:49.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.588+0000 7f5de5ffb700 1 -- 172.21.15.90:0/258070968 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f5dd4005cd0 con 0x7f5de8075940 2024-04-14T15:23:49.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.588+0000 7f5deeed8700 10 client.37851.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:23:49.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.588+0000 7f5deeed8700 1 -- 172.21.15.90:0/258070968 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f5de81b0a00 con 0x7f5de8075940 2024-04-14T15:23:49.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.588+0000 7f5deeed8700 1 librados: init done 2024-04-14T15:23:49.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.589+0000 7f5deeed8700 10 client.37851.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:23:49.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.589+0000 7f5deeed8700 15 client.37851.objecter _session_command_op_assign -1 1 2024-04-14T15:23:49.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.589+0000 7f5deeed8700 20 client.37851.objecter _get_session osd=-1 returning homeless 2024-04-14T15:23:49.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.589+0000 7f5deeed8700 10 client.37851.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:23:49.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.589+0000 7f5deeed8700 1 -- 172.21.15.90:0/258070968 --> v1:172.21.15.90:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f5de8076590 con 0x7f5de8075940 2024-04-14T15:23:49.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.589+0000 7f5de5ffb700 1 -- 172.21.15.90:0/258070968 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f5dd40036a0 con 0x7f5de8075940 2024-04-14T15:23:49.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.590+0000 7f5de5ffb700 1 -- 172.21.15.90:0/258070968 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(687..687 src has 467..687) ==== 8729+0+0 (unknown 1993930958 0 0) 0x7f5dd408ac20 con 0x7f5de8075940 2024-04-14T15:23:49.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.590+0000 7f5de5ffb700 10 client.37851.objecter ms_dispatch 0x7f5de8001550 osd_map(687..687 src has 467..687) 2024-04-14T15:23:49.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.590+0000 7f5de5ffb700 3 client.37851.objecter handle_osd_map got epochs [687,687] > 0 2024-04-14T15:23:49.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.590+0000 7f5de5ffb700 3 client.37851.objecter handle_osd_map decoding full epoch 687 2024-04-14T15:23:49.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.590+0000 7f5de5ffb700 10 client.37851.objecter checking command 1 2024-04-14T15:23:49.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.590+0000 7f5de5ffb700 20 client.37851.objecter _get_session s=0x7f5dd807eeb0 osd=4 3 2024-04-14T15:23:49.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.590+0000 7f5de5ffb700 20 client.37851.objecter put_session s=0x7f5dd807eeb0 osd=4 3 2024-04-14T15:23:49.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.590+0000 7f5de5ffb700 15 client.37851.objecter _session_command_op_remove -1 1 2024-04-14T15:23:49.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.590+0000 7f5de5ffb700 20 client.37851.objecter _get_session s=0x7f5dd807eeb0 osd=4 3 2024-04-14T15:23:49.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.590+0000 7f5de5ffb700 20 client.37851.objecter get_session s=0x7f5dd807eeb0 osd=4 3 2024-04-14T15:23:49.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.590+0000 7f5de5ffb700 15 client.37851.objecter _session_command_op_assign 4 1 2024-04-14T15:23:49.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.590+0000 7f5de5ffb700 20 client.37851.objecter put_session s=0x7f5dd807eeb0 osd=4 4 2024-04-14T15:23:49.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.590+0000 7f5de5ffb700 10 client.37851.objecter _send_command 1 2024-04-14T15:23:49.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.590+0000 7f5de5ffb700 1 -- 172.21.15.90:0/258070968 --> v1:172.21.15.92:6800/3887549733 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5dd8083270 con 0x7f5dd807f560 2024-04-14T15:23:49.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.590+0000 7f5de5ffb700 20 client.37851.objecter dump_active .. 0 homeless 2024-04-14T15:23:49.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.590+0000 7f5de5ffb700 1 -- 172.21.15.90:0/258070968 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_get_version_reply(handle=1 version=687) ==== 24+0+0 (unknown 2902863990 0 0) 0x7f5dd408b030 con 0x7f5de8075940 2024-04-14T15:23:49.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.590+0000 7f5de5ffb700 10 client.37851.objecter ms_handle_connect 0x7f5dd80714b0 2024-04-14T15:23:49.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.591+0000 7f5de5ffb700 10 client.37851.objecter ms_handle_connect 0x7f5dd807f560 2024-04-14T15:23:49.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.591+0000 7f5de5ffb700 1 -- 172.21.15.90:0/258070968 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f5dd8083270 con 0x7f5dd807f560 2024-04-14T15:23:49.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.591+0000 7f5de5ffb700 10 client.37851.objecter ms_dispatch 0x7f5de8001550 command_reply(tid 1: 0 ) 2024-04-14T15:23:49.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.591+0000 7f5de5ffb700 10 client.37851.objecter _finish_command 1 = system:0 2024-04-14T15:23:49.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.591+0000 7f5de5ffb700 20 client.37851.objecter put_session s=0x7f5dd807eeb0 osd=4 4 2024-04-14T15:23:49.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:49.591+0000 7f5de5ffb700 15 client.37851.objecter _session_command_op_remove 4 1 2024-04-14T15:23:49.624 INFO:teuthology.orchestra.run.smithi090.stderr:no valid command found; 10 closest matches: 2024-04-14T15:23:49.624 INFO:teuthology.orchestra.run.smithi090.stderr:0 2024-04-14T15:23:49.624 INFO:teuthology.orchestra.run.smithi090.stderr:1 2024-04-14T15:23:49.624 INFO:teuthology.orchestra.run.smithi090.stderr:2 2024-04-14T15:23:49.624 INFO:teuthology.orchestra.run.smithi090.stderr:abort 2024-04-14T15:23:49.624 INFO:teuthology.orchestra.run.smithi090.stderr:assert 2024-04-14T15:23:49.624 INFO:teuthology.orchestra.run.smithi090.stderr:bench [] [] [] [] 2024-04-14T15:23:49.625 INFO:teuthology.orchestra.run.smithi090.stderr:bluefs debug_inject_read_zeros 2024-04-14T15:23:49.625 INFO:teuthology.orchestra.run.smithi090.stderr:bluefs files list 2024-04-14T15:23:49.625 INFO:teuthology.orchestra.run.smithi090.stderr:bluefs stats 2024-04-14T15:23:49.625 INFO:teuthology.orchestra.run.smithi090.stderr:bluestore allocator dump block 2024-04-14T15:23:49.625 INFO:teuthology.orchestra.run.smithi090.stderr:Error EINVAL: invalid command 2024-04-14T15:23:49.669 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:49.696 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:49.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:49.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:49.798 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:23:49.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:49.826 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:49.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:23:49.927 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:23:49.951 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:49.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:49.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:23:50.053 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:23:50.077 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:50.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:50.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:23:50.179 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:23:50.206 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:50.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:50.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:23:50.308 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:23:50.334 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:50.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:50.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:23:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:50 smithi092 ceph-mon[33966]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:23:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:50 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-118", "mode": "writeback"}]': finished 2024-04-14T15:23:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:50 smithi092 ceph-mon[33966]: osdmap e687: 8 total, 8 up, 7 in 2024-04-14T15:23:50.436 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:23:50.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:50 smithi090 ceph-mon[41670]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:23:50.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:50 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-118", "mode": "writeback"}]': finished 2024-04-14T15:23:50.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:50 smithi090 ceph-mon[41670]: osdmap e687: 8 total, 8 up, 7 in 2024-04-14T15:23:50.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:50 smithi090 ceph-mon[30399]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:23:50.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:50 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-118", "mode": "writeback"}]': finished 2024-04-14T15:23:50.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:50 smithi090 ceph-mon[30399]: osdmap e687: 8 total, 8 up, 7 in 2024-04-14T15:23:50.461 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:50.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:50.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:23:50.469 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:23:50.563 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:23:50.590 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:50.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:50.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:23:50.692 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:23:50.719 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:50.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:50.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:23:50.820 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:23:50.846 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:50.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:50.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:23:50.948 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:23:50.974 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:50.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:50.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:23:51.039 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-14T15:23:51.039 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-14T15:23:51.039 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph tell osd.5 'trim stale osdmaps' 2024-04-14T15:23:51.076 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:23:51.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:51.105 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:51.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:23:51.148 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-14T15:23:51.206 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:23:51.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:51.231 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:51.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:23:51.279 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:51 smithi090 ceph-mon[41670]: pgmap v1014: 214 pgs: 214 active+clean; 582 KiB data, 635 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-04-14T15:23:51.279 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:51 smithi090 ceph-mon[41670]: osdmap e688: 8 total, 8 up, 7 in 2024-04-14T15:23:51.279 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:51 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:23:51.279 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:51 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:23:51.279 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:51 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]': finished 2024-04-14T15:23:51.279 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:51 smithi090 ceph-mon[41670]: osdmap e689: 8 total, 8 up, 7 in 2024-04-14T15:23:51.279 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:51 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-118"}]: dispatch 2024-04-14T15:23:51.279 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:51 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-118"}]: dispatch 2024-04-14T15:23:51.279 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:51 smithi090 ceph-mon[30399]: pgmap v1014: 214 pgs: 214 active+clean; 582 KiB data, 635 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-04-14T15:23:51.279 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:51 smithi090 ceph-mon[30399]: osdmap e688: 8 total, 8 up, 7 in 2024-04-14T15:23:51.279 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:51 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:23:51.280 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:51 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:23:51.280 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:51 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]': finished 2024-04-14T15:23:51.280 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:51 smithi090 ceph-mon[30399]: osdmap e689: 8 total, 8 up, 7 in 2024-04-14T15:23:51.280 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:51 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-118"}]: dispatch 2024-04-14T15:23:51.280 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:51 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-118"}]: dispatch 2024-04-14T15:23:51.333 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:23:51.359 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:51.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:51.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:23:51.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:51 smithi092 ceph-mon[33966]: pgmap v1014: 214 pgs: 214 active+clean; 582 KiB data, 635 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-04-14T15:23:51.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:51 smithi092 ceph-mon[33966]: osdmap e688: 8 total, 8 up, 7 in 2024-04-14T15:23:51.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:51 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:23:51.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:51 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:23:51.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:51 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]': finished 2024-04-14T15:23:51.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:51 smithi092 ceph-mon[33966]: osdmap e689: 8 total, 8 up, 7 in 2024-04-14T15:23:51.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:51 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-118"}]: dispatch 2024-04-14T15:23:51.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:51 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-118"}]: dispatch 2024-04-14T15:23:51.462 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:23:51.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:51.486 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:51.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:23:51.587 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:23:51.613 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:51.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:51.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:23:51.715 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:23:51.716 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:23:51.742 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:51.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:51.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:23:51.843 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:51.868 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:51.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:51.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:51.970 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:23:51.998 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:51.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:52.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:23:52.100 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:52.126 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:52.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:52.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:52.137 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-14T15:23:52.150 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:52 smithi090 ceph-mon[41670]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:23:52.151 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:52 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-118"}]': finished 2024-04-14T15:23:52.151 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:52 smithi090 ceph-mon[41670]: osdmap e690: 8 total, 8 up, 7 in 2024-04-14T15:23:52.151 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:52 smithi090 ceph-mon[30399]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:23:52.151 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:52 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-118"}]': finished 2024-04-14T15:23:52.151 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:52 smithi090 ceph-mon[30399]: osdmap e690: 8 total, 8 up, 7 in 2024-04-14T15:23:52.227 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:23:52.250 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:52.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:52.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:23:52.273 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:52 smithi092 ceph-mon[33966]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:23:52.273 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:52 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-118"}]': finished 2024-04-14T15:23:52.274 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:52 smithi092 ceph-mon[33966]: osdmap e690: 8 total, 8 up, 7 in 2024-04-14T15:23:52.352 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:23:52.376 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:52.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:52.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:23:52.477 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:23:52.502 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:52.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:52.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:23:52.604 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:23:52.627 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:52.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:52.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:23:52.728 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:23:52.752 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:52.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:52.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:23:52.854 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:23:52.877 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:52.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:52.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:23:52.979 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:23:53.003 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:53.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:53.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:23:53.104 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:23:53.128 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:53.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:53.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:23:53.229 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:23:53.252 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:53.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:53.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:23:53.354 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:23:53.368 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:53 smithi090 ceph-mon[41670]: pgmap v1017: 214 pgs: 214 active+clean; 582 KiB data, 635 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-04-14T15:23:53.368 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:53 smithi090 ceph-mon[30399]: pgmap v1017: 214 pgs: 214 active+clean; 582 KiB data, 635 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-04-14T15:23:53.371 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:53 smithi092 ceph-mon[33966]: pgmap v1017: 214 pgs: 214 active+clean; 582 KiB data, 635 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-04-14T15:23:53.377 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:53.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:53.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:23:53.479 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:53.502 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:53.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:53.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:53.603 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:23:53.627 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:53.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:53.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:23:53.729 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:23:53.752 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:53.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:53.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:23:53.853 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:23:53.877 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:53.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:53.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:23:53.979 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:23:54.003 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:54.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:54.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:23:54.025 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:23:53 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:23:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:23:54.105 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:23:54.131 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:54.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:54.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:23:54.232 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:23:54.256 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:54.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:54.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:23:54.291 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:54 smithi090 ceph-mon[30399]: osdmap e691: 8 total, 8 up, 7 in 2024-04-14T15:23:54.352 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:54 smithi090 ceph-mon[30399]: pgmap v1020: 182 pgs: 182 active+clean; 582 KiB data, 635 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:23:54.353 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:54 smithi090 ceph-mon[41670]: osdmap e691: 8 total, 8 up, 7 in 2024-04-14T15:23:54.353 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:54 smithi090 ceph-mon[41670]: pgmap v1020: 182 pgs: 182 active+clean; 582 KiB data, 635 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:23:54.358 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:23:54.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:54 smithi092 ceph-mon[33966]: osdmap e691: 8 total, 8 up, 7 in 2024-04-14T15:23:54.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:54 smithi092 ceph-mon[33966]: pgmap v1020: 182 pgs: 182 active+clean; 582 KiB data, 635 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:23:54.381 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:54.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:54.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:23:54.483 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:23:54.507 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:54.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:54.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:23:54.609 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:54.633 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:54.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:54.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:54.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.396+0000 7fc6907f8700 1 -- 172.21.15.90:0/3036722779 >> v1:172.21.15.90:6789/0 conn(0x7fc688107780 legacy=0x7fc688107b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:54.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.396+0000 7fc6907f8700 1 -- 172.21.15.90:0/3036722779 shutdown_connections 2024-04-14T15:23:54.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.396+0000 7fc6907f8700 1 -- 172.21.15.90:0/3036722779 >> 172.21.15.90:0/3036722779 conn(0x7fc6880fd290 msgr2=0x7fc6880ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:23:54.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.397+0000 7fc6907f8700 1 -- 172.21.15.90:0/3036722779 shutdown_connections 2024-04-14T15:23:54.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.397+0000 7fc6907f8700 1 -- 172.21.15.90:0/3036722779 wait complete. 2024-04-14T15:23:54.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.397+0000 7fc6907f8700 1 librados: starting msgr at 2024-04-14T15:23:54.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.397+0000 7fc6907f8700 1 librados: starting objecter 2024-04-14T15:23:54.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.397+0000 7fc6907f8700 1 Processor -- start 2024-04-14T15:23:54.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.397+0000 7fc6907f8700 1 -- start start 2024-04-14T15:23:54.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.397+0000 7fc6907f8700 1 librados: setting wanted keys 2024-04-14T15:23:54.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.397+0000 7fc6907f8700 1 librados: calling monclient init 2024-04-14T15:23:54.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.397+0000 7fc6907f8700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc6881103e0 con 0x7fc688107780 2024-04-14T15:23:54.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.397+0000 7fc6907f8700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc6881105f0 con 0x7fc68810e200 2024-04-14T15:23:54.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.397+0000 7fc6907f8700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc6881ac220 con 0x7fc68810a630 2024-04-14T15:23:54.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.397+0000 7fc68dd93700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7fc68810a630 0x7fc68810ee90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:51388/0 (socket says 172.21.15.90:51388) 2024-04-14T15:23:54.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.397+0000 7fc68dd93700 1 -- 172.21.15.90:0/664989256 learned_addr learned my addr 172.21.15.90:0/664989256 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:23:54.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.397+0000 7fc67f7fe700 10 client.?.objecter ms_handle_connect 0x7fc688107780 2024-04-14T15:23:54.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.397+0000 7fc67f7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:23:54.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.398+0000 7fc67f7fe700 10 client.?.objecter ms_handle_connect 0x7fc68810a630 2024-04-14T15:23:54.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.398+0000 7fc67f7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:23:54.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.398+0000 7fc67f7fe700 1 -- 172.21.15.90:0/664989256 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3693743172 0 0) 0x7fc6881103e0 con 0x7fc688107780 2024-04-14T15:23:54.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.398+0000 7fc67f7fe700 1 -- 172.21.15.90:0/664989256 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc6740035b0 con 0x7fc688107780 2024-04-14T15:23:54.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.398+0000 7fc67f7fe700 10 client.?.objecter ms_handle_connect 0x7fc68810e200 2024-04-14T15:23:54.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.398+0000 7fc67f7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:23:54.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.398+0000 7fc67f7fe700 1 -- 172.21.15.90:0/664989256 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1998480552 0 0) 0x7fc6881ac220 con 0x7fc68810a630 2024-04-14T15:23:54.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.398+0000 7fc67f7fe700 1 -- 172.21.15.90:0/664989256 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc6881103e0 con 0x7fc68810a630 2024-04-14T15:23:54.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.398+0000 7fc67f7fe700 1 -- 172.21.15.90:0/664989256 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2522149478 0 0) 0x7fc6881105f0 con 0x7fc68810e200 2024-04-14T15:23:54.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.398+0000 7fc67f7fe700 1 -- 172.21.15.90:0/664989256 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc6881ac220 con 0x7fc68810e200 2024-04-14T15:23:54.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.398+0000 7fc67f7fe700 1 -- 172.21.15.90:0/664989256 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1501445363 0 0) 0x7fc6740035b0 con 0x7fc688107780 2024-04-14T15:23:54.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.398+0000 7fc67f7fe700 1 -- 172.21.15.90:0/664989256 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc6881105f0 con 0x7fc688107780 2024-04-14T15:23:54.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.398+0000 7fc67f7fe700 1 -- 172.21.15.90:0/664989256 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fc684001430 con 0x7fc688107780 2024-04-14T15:23:54.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.398+0000 7fc67f7fe700 1 -- 172.21.15.90:0/664989256 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 806842955 0 0) 0x7fc6881103e0 con 0x7fc68810a630 2024-04-14T15:23:54.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.398+0000 7fc67f7fe700 1 -- 172.21.15.90:0/664989256 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc6740035b0 con 0x7fc68810a630 2024-04-14T15:23:54.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.398+0000 7fc67f7fe700 1 -- 172.21.15.90:0/664989256 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fc678001880 con 0x7fc68810a630 2024-04-14T15:23:54.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.398+0000 7fc67f7fe700 1 -- 172.21.15.90:0/664989256 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4096975682 0 0) 0x7fc6881ac220 con 0x7fc68810e200 2024-04-14T15:23:54.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.398+0000 7fc67f7fe700 1 -- 172.21.15.90:0/664989256 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc6881103e0 con 0x7fc68810e200 2024-04-14T15:23:54.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.398+0000 7fc67f7fe700 1 -- 172.21.15.90:0/664989256 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fc680001c50 con 0x7fc68810e200 2024-04-14T15:23:54.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.398+0000 7fc67f7fe700 1 -- 172.21.15.90:0/664989256 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 825982454 0 0) 0x7fc6881105f0 con 0x7fc688107780 2024-04-14T15:23:54.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.398+0000 7fc67f7fe700 1 -- 172.21.15.90:0/664989256 >> v1:172.21.15.90:6790/0 conn(0x7fc68810a630 legacy=0x7fc68810ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:54.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.398+0000 7fc67f7fe700 1 -- 172.21.15.90:0/664989256 >> v1:172.21.15.92:6789/0 conn(0x7fc68810e200 legacy=0x7fc6881a8ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:23:54.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.398+0000 7fc67f7fe700 1 -- 172.21.15.90:0/664989256 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6881ad400 con 0x7fc688107780 2024-04-14T15:23:54.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.398+0000 7fc6907f8700 1 -- 172.21.15.90:0/664989256 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc6881ac450 con 0x7fc688107780 2024-04-14T15:23:54.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.398+0000 7fc6907f8700 10 client.37857.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:23:54.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.398+0000 7fc6907f8700 1 -- 172.21.15.90:0/664989256 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc6881ac910 con 0x7fc688107780 2024-04-14T15:23:54.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.398+0000 7fc6907f8700 1 librados: init done 2024-04-14T15:23:54.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.398+0000 7fc67f7fe700 1 -- 172.21.15.90:0/664989256 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fc684001b30 con 0x7fc688107780 2024-04-14T15:23:54.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.398+0000 7fc67f7fe700 1 -- 172.21.15.90:0/664989256 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fc684004da0 con 0x7fc688107780 2024-04-14T15:23:54.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.399+0000 7fc6907f8700 10 client.37857.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:23:54.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.399+0000 7fc6907f8700 15 client.37857.objecter _session_command_op_assign -1 1 2024-04-14T15:23:54.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.399+0000 7fc6907f8700 20 client.37857.objecter _get_session osd=-1 returning homeless 2024-04-14T15:23:54.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.399+0000 7fc6907f8700 10 client.37857.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:23:54.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.399+0000 7fc6907f8700 1 -- 172.21.15.90:0/664989256 --> v1:172.21.15.90:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fc66c002920 con 0x7fc688107780 2024-04-14T15:23:54.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.400+0000 7fc67f7fe700 1 -- 172.21.15.90:0/664989256 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fc6840037c0 con 0x7fc688107780 2024-04-14T15:23:54.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.400+0000 7fc67f7fe700 1 -- 172.21.15.90:0/664989256 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(692..692 src has 467..692) ==== 8749+0+0 (unknown 226500907 0 0) 0x7fc684089920 con 0x7fc688107780 2024-04-14T15:23:54.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.400+0000 7fc67f7fe700 10 client.37857.objecter ms_dispatch 0x7fc688001550 osd_map(692..692 src has 467..692) 2024-04-14T15:23:54.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.400+0000 7fc67f7fe700 3 client.37857.objecter handle_osd_map got epochs [692,692] > 0 2024-04-14T15:23:54.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.400+0000 7fc67f7fe700 3 client.37857.objecter handle_osd_map decoding full epoch 692 2024-04-14T15:23:54.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.400+0000 7fc67f7fe700 10 client.37857.objecter checking command 1 2024-04-14T15:23:54.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.400+0000 7fc67f7fe700 20 client.37857.objecter _get_session s=0x7fc67407ecb0 osd=5 3 2024-04-14T15:23:54.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.401+0000 7fc67f7fe700 20 client.37857.objecter put_session s=0x7fc67407ecb0 osd=5 3 2024-04-14T15:23:54.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.401+0000 7fc67f7fe700 15 client.37857.objecter _session_command_op_remove -1 1 2024-04-14T15:23:54.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.401+0000 7fc67f7fe700 20 client.37857.objecter _get_session s=0x7fc67407ecb0 osd=5 3 2024-04-14T15:23:54.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.401+0000 7fc67f7fe700 20 client.37857.objecter get_session s=0x7fc67407ecb0 osd=5 3 2024-04-14T15:23:54.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.401+0000 7fc67f7fe700 15 client.37857.objecter _session_command_op_assign 5 1 2024-04-14T15:23:54.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.401+0000 7fc67f7fe700 20 client.37857.objecter put_session s=0x7fc67407ecb0 osd=5 4 2024-04-14T15:23:54.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.401+0000 7fc67f7fe700 10 client.37857.objecter _send_command 1 2024-04-14T15:23:54.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.401+0000 7fc67f7fe700 1 -- 172.21.15.90:0/664989256 --> v1:172.21.15.92:6804/3244561440 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc674083070 con 0x7fc67407f360 2024-04-14T15:23:54.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.401+0000 7fc67f7fe700 20 client.37857.objecter dump_active .. 0 homeless 2024-04-14T15:23:54.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.401+0000 7fc67f7fe700 1 -- 172.21.15.90:0/664989256 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_get_version_reply(handle=1 version=692) ==== 24+0+0 (unknown 557668125 0 0) 0x7fc684089cb0 con 0x7fc688107780 2024-04-14T15:23:54.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.401+0000 7fc67f7fe700 10 client.37857.objecter ms_handle_connect 0x7fc6740712e0 2024-04-14T15:23:54.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.401+0000 7fc67f7fe700 10 client.37857.objecter ms_handle_connect 0x7fc67407f360 2024-04-14T15:23:54.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.402+0000 7fc67f7fe700 1 -- 172.21.15.90:0/664989256 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc674083070 con 0x7fc67407f360 2024-04-14T15:23:54.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.402+0000 7fc67f7fe700 10 client.37857.objecter ms_dispatch 0x7fc688001550 command_reply(tid 1: 0 ) 2024-04-14T15:23:54.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.402+0000 7fc67f7fe700 10 client.37857.objecter _finish_command 1 = system:0 2024-04-14T15:23:54.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.402+0000 7fc67f7fe700 20 client.37857.objecter put_session s=0x7fc67407ecb0 osd=5 4 2024-04-14T15:23:54.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:23:54.402+0000 7fc67f7fe700 15 client.37857.objecter _session_command_op_remove 5 1 2024-04-14T15:23:54.658 INFO:teuthology.orchestra.run.smithi090.stderr:no valid command found; 10 closest matches: 2024-04-14T15:23:54.658 INFO:teuthology.orchestra.run.smithi090.stderr:0 2024-04-14T15:23:54.658 INFO:teuthology.orchestra.run.smithi090.stderr:1 2024-04-14T15:23:54.658 INFO:teuthology.orchestra.run.smithi090.stderr:2 2024-04-14T15:23:54.658 INFO:teuthology.orchestra.run.smithi090.stderr:abort 2024-04-14T15:23:54.658 INFO:teuthology.orchestra.run.smithi090.stderr:assert 2024-04-14T15:23:54.658 INFO:teuthology.orchestra.run.smithi090.stderr:bench [] [] [] [] 2024-04-14T15:23:54.658 INFO:teuthology.orchestra.run.smithi090.stderr:bluefs debug_inject_read_zeros 2024-04-14T15:23:54.659 INFO:teuthology.orchestra.run.smithi090.stderr:bluefs files list 2024-04-14T15:23:54.659 INFO:teuthology.orchestra.run.smithi090.stderr:bluefs stats 2024-04-14T15:23:54.659 INFO:teuthology.orchestra.run.smithi090.stderr:bluestore allocator dump block 2024-04-14T15:23:54.659 INFO:teuthology.orchestra.run.smithi090.stderr:Error EINVAL: invalid command 2024-04-14T15:23:54.735 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:23:54.759 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:54.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:54.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:23:54.861 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:23:54.885 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:54.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:54.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:23:54.894 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:23:54.987 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:23:55.010 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:55.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:55.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:23:55.112 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:23:55.135 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:55.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:55.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:23:55.237 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:23:55.261 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:55.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:55.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:23:55.363 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:23:55.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:55 smithi092 ceph-mon[33966]: osdmap e692: 8 total, 8 up, 7 in 2024-04-14T15:23:55.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:55 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:23:55.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:55 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:23:55.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:55 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:23:55.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:55 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:23:55.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:55.387 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:55.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:23:55.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:55 smithi090 ceph-mon[41670]: osdmap e692: 8 total, 8 up, 7 in 2024-04-14T15:23:55.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:55 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:23:55.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:55 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:23:55.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:55 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:23:55.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:55 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:23:55.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:55 smithi090 ceph-mon[30399]: osdmap e692: 8 total, 8 up, 7 in 2024-04-14T15:23:55.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:55 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:23:55.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:55 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:23:55.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:55 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:23:55.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:55 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:23:55.488 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:23:55.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:55.512 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:55.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:23:55.592 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-14T15:23:55.593 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-14T15:23:55.593 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph tell osd.7 'trim stale osdmaps' 2024-04-14T15:23:55.613 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:23:55.660 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:55.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:55.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:23:55.761 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:23:55.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:55.787 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:55.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:23:55.872 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:23:55 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:23:55.887 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:23:55.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:55.913 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:55.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:23:55.991 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:23:56.013 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:23:56.028 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-14T15:23:56.029 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:23:56.041 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:56.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:56.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:23:56.137 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:56 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:23:56.138 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:56 smithi090 ceph-mon[41670]: osdmap e693: 8 total, 8 up, 7 in 2024-04-14T15:23:56.138 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:56 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:23:56.138 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:56 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:23:56.138 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:56 smithi090 ceph-mon[41670]: pgmap v1023: 214 pgs: 32 creating+peering, 182 active+clean; 582 KiB data, 635 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:23:56.138 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:56 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:23:56.138 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:56 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-120", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:23:56.138 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:56 smithi090 ceph-mon[41670]: osdmap e694: 8 total, 8 up, 7 in 2024-04-14T15:23:56.138 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:56 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-120"}]: dispatch 2024-04-14T15:23:56.138 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:56 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-120"}]: dispatch 2024-04-14T15:23:56.138 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:56 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:23:56.139 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:56 smithi090 ceph-mon[30399]: osdmap e693: 8 total, 8 up, 7 in 2024-04-14T15:23:56.139 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:56 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:23:56.139 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:56 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:23:56.139 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:56 smithi090 ceph-mon[30399]: pgmap v1023: 214 pgs: 32 creating+peering, 182 active+clean; 582 KiB data, 635 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:23:56.139 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:56 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:23:56.139 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:56 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-120", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:23:56.139 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:56 smithi090 ceph-mon[30399]: osdmap e694: 8 total, 8 up, 7 in 2024-04-14T15:23:56.139 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:56 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-120"}]: dispatch 2024-04-14T15:23:56.139 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:56 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-120"}]: dispatch 2024-04-14T15:23:56.142 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:23:56.169 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:56.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:56.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:23:56.271 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:23:56.294 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:56.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:56.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:23:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:56 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:23:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:56 smithi092 ceph-mon[33966]: osdmap e693: 8 total, 8 up, 7 in 2024-04-14T15:23:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:56 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:23:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:56 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:23:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:56 smithi092 ceph-mon[33966]: pgmap v1023: 214 pgs: 32 creating+peering, 182 active+clean; 582 KiB data, 635 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:23:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:56 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:23:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:56 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-120", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:23:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:56 smithi092 ceph-mon[33966]: osdmap e694: 8 total, 8 up, 7 in 2024-04-14T15:23:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:56 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-120"}]: dispatch 2024-04-14T15:23:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:56 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-120"}]: dispatch 2024-04-14T15:23:56.396 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:23:56.419 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:56.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:56.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:23:56.521 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:23:56.544 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:56.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:56.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:23:56.646 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:23:56.669 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:56.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:56.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:23:56.771 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:23:56.798 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:56.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:56.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:23:56.900 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:23:56.926 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:56.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:56.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:23:57.028 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:23:57.053 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:57.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:57.054 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:23:57.155 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:23:57.179 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:57.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:57.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:23:57.281 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:57.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:57.305 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:57.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:57.406 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:23:57.430 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:57.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:57.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:23:57.532 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:57.556 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:57.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:57.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:57.658 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:23:57.687 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:57.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:57.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:23:57.788 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:23:57.815 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:57.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:57.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:23:57.828 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:23:57.917 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:23:57.942 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:57.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:57.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:23:58.043 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:23:58.068 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:58.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:58.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:23:58.153 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:57 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-120"}]': finished 2024-04-14T15:23:58.153 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:57 smithi090 ceph-mon[41670]: osdmap e695: 8 total, 8 up, 7 in 2024-04-14T15:23:58.154 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:57 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-120", "mode": "writeback"}]: dispatch 2024-04-14T15:23:58.154 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:57 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-120", "mode": "writeback"}]: dispatch 2024-04-14T15:23:58.154 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:57 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:23:58.154 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:57 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-120"}]': finished 2024-04-14T15:23:58.154 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:57 smithi090 ceph-mon[30399]: osdmap e695: 8 total, 8 up, 7 in 2024-04-14T15:23:58.154 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:57 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-120", "mode": "writeback"}]: dispatch 2024-04-14T15:23:58.154 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:57 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-120", "mode": "writeback"}]: dispatch 2024-04-14T15:23:58.154 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:57 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:23:58.170 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:23:58.192 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:58.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:58.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:23:58.294 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:23:58.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:58.318 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:58.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:23:58.371 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:57 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-120"}]': finished 2024-04-14T15:23:58.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:57 smithi092 ceph-mon[33966]: osdmap e695: 8 total, 8 up, 7 in 2024-04-14T15:23:58.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:57 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-120", "mode": "writeback"}]: dispatch 2024-04-14T15:23:58.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:57 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-120", "mode": "writeback"}]: dispatch 2024-04-14T15:23:58.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:57 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:23:58.419 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:23:58.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:58.446 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:58.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:23:58.546 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:58.569 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:58.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:58.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:58.671 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:23:58.696 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:58.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:58.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:23:58.715 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-14T15:23:58.799 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:23:58.823 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:58.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:58.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:23:58.924 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:58.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:58.948 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:58.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:23:59.049 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:23:59.072 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:59.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:59.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:23:59.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:58 smithi092 ceph-mon[33966]: pgmap v1026: 214 pgs: 32 creating+peering, 182 active+clean; 582 KiB data, 635 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:23:59.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:58 smithi092 ceph-mon[33966]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:23:59.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:58 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-120", "mode": "writeback"}]': finished 2024-04-14T15:23:59.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:58 smithi092 ceph-mon[33966]: osdmap e696: 8 total, 8 up, 7 in 2024-04-14T15:23:59.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:58 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:23:59.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:23:58 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:23:59.174 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:23:59.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:58 smithi090 ceph-mon[41670]: pgmap v1026: 214 pgs: 32 creating+peering, 182 active+clean; 582 KiB data, 635 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:23:59.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:58 smithi090 ceph-mon[41670]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:23:59.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:58 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-120", "mode": "writeback"}]': finished 2024-04-14T15:23:59.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:58 smithi090 ceph-mon[41670]: osdmap e696: 8 total, 8 up, 7 in 2024-04-14T15:23:59.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:58 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:23:59.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:23:58 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:23:59.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:58 smithi090 ceph-mon[30399]: pgmap v1026: 214 pgs: 32 creating+peering, 182 active+clean; 582 KiB data, 635 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:23:59.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:58 smithi090 ceph-mon[30399]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:23:59.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:58 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-120", "mode": "writeback"}]': finished 2024-04-14T15:23:59.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:58 smithi090 ceph-mon[30399]: osdmap e696: 8 total, 8 up, 7 in 2024-04-14T15:23:59.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:58 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:23:59.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:23:58 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:23:59.197 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:59.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:59.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:23:59.299 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:23:59.322 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:59.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:59.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:23:59.424 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:23:59.447 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:59.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:59.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:23:59.548 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:23:59.572 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:59.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:59.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:23:59.673 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:23:59.696 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:23:59.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:59.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:23:59.798 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:23:59.822 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:59.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:59.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:23:59.923 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:23:59.946 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:23:59.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:23:59.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:24:00.048 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:24:00.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:00.072 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:00.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:24:00.173 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:24:00.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:00 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]': finished 2024-04-14T15:24:00.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:00 smithi090 ceph-mon[30399]: osdmap e697: 8 total, 8 up, 7 in 2024-04-14T15:24:00.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:00 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-120"}]: dispatch 2024-04-14T15:24:00.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:00 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-120"}]: dispatch 2024-04-14T15:24:00.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:00 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:24:00.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:00 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:24:00.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:00 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]': finished 2024-04-14T15:24:00.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:00 smithi090 ceph-mon[41670]: osdmap e697: 8 total, 8 up, 7 in 2024-04-14T15:24:00.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:00 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-120"}]: dispatch 2024-04-14T15:24:00.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:00 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-120"}]: dispatch 2024-04-14T15:24:00.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:00 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:24:00.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:00 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:24:00.196 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:00.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:00.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:24:00.298 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:24:00.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:00.322 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:00.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:24:00.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:00 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]': finished 2024-04-14T15:24:00.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:00 smithi092 ceph-mon[33966]: osdmap e697: 8 total, 8 up, 7 in 2024-04-14T15:24:00.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:00 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-120"}]: dispatch 2024-04-14T15:24:00.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:00 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-120"}]: dispatch 2024-04-14T15:24:00.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:00 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:24:00.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:00 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:24:00.424 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:24:00.448 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:00.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:00.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:24:00.549 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:24:00.573 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:00.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:00.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:24:00.675 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:24:00.698 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:00.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:00.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:24:00.799 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:24:00.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:00.824 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:00.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:24:00.925 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:24:00.948 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:00.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:00.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:24:01.050 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:24:01.073 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:01.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:01.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:24:01.177 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:24:01.201 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:01.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:01.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:24:01.303 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:24:01.327 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:01.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:01.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:24:01.364 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:01 smithi092 ceph-mon[33966]: pgmap v1029: 214 pgs: 214 active+clean; 582 KiB data, 653 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-04-14T15:24:01.364 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:01 smithi092 ceph-mon[33966]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:24:01.364 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:01 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-120"}]': finished 2024-04-14T15:24:01.364 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:01 smithi092 ceph-mon[33966]: osdmap e698: 8 total, 8 up, 7 in 2024-04-14T15:24:01.364 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:01 smithi092 ceph-mon[33966]: osdmap e699: 8 total, 8 up, 7 in 2024-04-14T15:24:01.430 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:24:01.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:01 smithi090 ceph-mon[41670]: pgmap v1029: 214 pgs: 214 active+clean; 582 KiB data, 653 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-04-14T15:24:01.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:01 smithi090 ceph-mon[41670]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:24:01.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:01 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-120"}]': finished 2024-04-14T15:24:01.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:01 smithi090 ceph-mon[41670]: osdmap e698: 8 total, 8 up, 7 in 2024-04-14T15:24:01.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:01 smithi090 ceph-mon[41670]: osdmap e699: 8 total, 8 up, 7 in 2024-04-14T15:24:01.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:01 smithi090 ceph-mon[30399]: pgmap v1029: 214 pgs: 214 active+clean; 582 KiB data, 653 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-04-14T15:24:01.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:01 smithi090 ceph-mon[30399]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:24:01.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:01 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-120"}]': finished 2024-04-14T15:24:01.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:01 smithi090 ceph-mon[30399]: osdmap e698: 8 total, 8 up, 7 in 2024-04-14T15:24:01.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:01 smithi090 ceph-mon[30399]: osdmap e699: 8 total, 8 up, 7 in 2024-04-14T15:24:01.453 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:01.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:01.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:24:01.555 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:24:01.579 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:01.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:01.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:24:01.680 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:24:01.705 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:01.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:01.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:24:01.807 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:24:01.831 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:01.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:01.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:24:01.932 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:24:01.956 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:01.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:01.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:24:02.058 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:24:02.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:02.082 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:02.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:24:02.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.116+0000 7fe4d7539700 1 -- 172.21.15.90:0/1992692586 >> v1:172.21.15.90:6789/0 conn(0x7fe4d0103a20 legacy=0x7fe4d0103e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:02.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.116+0000 7fe4d7539700 1 -- 172.21.15.90:0/1992692586 shutdown_connections 2024-04-14T15:24:02.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.116+0000 7fe4d7539700 1 -- 172.21.15.90:0/1992692586 >> 172.21.15.90:0/1992692586 conn(0x7fe4d00fd290 msgr2=0x7fe4d00ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:24:02.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.116+0000 7fe4d7539700 1 -- 172.21.15.90:0/1992692586 shutdown_connections 2024-04-14T15:24:02.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.116+0000 7fe4d7539700 1 -- 172.21.15.90:0/1992692586 wait complete. 2024-04-14T15:24:02.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.117+0000 7fe4d7539700 1 librados: starting msgr at 2024-04-14T15:24:02.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.117+0000 7fe4d7539700 1 librados: starting objecter 2024-04-14T15:24:02.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.117+0000 7fe4d7539700 1 Processor -- start 2024-04-14T15:24:02.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.117+0000 7fe4d7539700 1 -- start start 2024-04-14T15:24:02.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.117+0000 7fe4d7539700 1 librados: setting wanted keys 2024-04-14T15:24:02.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.117+0000 7fe4d7539700 1 librados: calling monclient init 2024-04-14T15:24:02.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.117+0000 7fe4d7539700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe4d0103250 con 0x7fe4d010f340 2024-04-14T15:24:02.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.117+0000 7fe4d7539700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe4d0103420 con 0x7fe4d0103a20 2024-04-14T15:24:02.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.117+0000 7fe4d7539700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe4d01ac2c0 con 0x7fe4d010b830 2024-04-14T15:24:02.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.117+0000 7fe4d4ad4700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7fe4d010b830 0x7fe4d0101d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:39432/0 (socket says 172.21.15.90:39432) 2024-04-14T15:24:02.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.117+0000 7fe4d4ad4700 1 -- 172.21.15.90:0/2032705009 learned_addr learned my addr 172.21.15.90:0/2032705009 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:24:02.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.117+0000 7fe4c67fc700 10 client.?.objecter ms_handle_connect 0x7fe4d010f340 2024-04-14T15:24:02.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.117+0000 7fe4c67fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:24:02.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.117+0000 7fe4c67fc700 10 client.?.objecter ms_handle_connect 0x7fe4d010b830 2024-04-14T15:24:02.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.117+0000 7fe4c67fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:24:02.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.117+0000 7fe4c67fc700 1 -- 172.21.15.90:0/2032705009 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1846926547 0 0) 0x7fe4d0103250 con 0x7fe4d010f340 2024-04-14T15:24:02.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.117+0000 7fe4c67fc700 1 -- 172.21.15.90:0/2032705009 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe4bc0031f0 con 0x7fe4d010f340 2024-04-14T15:24:02.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.117+0000 7fe4c67fc700 10 client.?.objecter ms_handle_connect 0x7fe4d0103a20 2024-04-14T15:24:02.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.117+0000 7fe4c67fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:24:02.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.117+0000 7fe4c67fc700 1 -- 172.21.15.90:0/2032705009 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 511875106 0 0) 0x7fe4d01ac2c0 con 0x7fe4d010b830 2024-04-14T15:24:02.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.117+0000 7fe4c67fc700 1 -- 172.21.15.90:0/2032705009 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe4d0103250 con 0x7fe4d010b830 2024-04-14T15:24:02.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.117+0000 7fe4c67fc700 1 -- 172.21.15.90:0/2032705009 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2243664094 0 0) 0x7fe4d0103420 con 0x7fe4d0103a20 2024-04-14T15:24:02.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.118+0000 7fe4c67fc700 1 -- 172.21.15.90:0/2032705009 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe4d01ac2c0 con 0x7fe4d0103a20 2024-04-14T15:24:02.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.118+0000 7fe4c67fc700 1 -- 172.21.15.90:0/2032705009 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2817908087 0 0) 0x7fe4bc0031f0 con 0x7fe4d010f340 2024-04-14T15:24:02.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.118+0000 7fe4c67fc700 1 -- 172.21.15.90:0/2032705009 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe4d0103420 con 0x7fe4d010f340 2024-04-14T15:24:02.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.118+0000 7fe4c67fc700 1 -- 172.21.15.90:0/2032705009 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fe4c8001c30 con 0x7fe4d010f340 2024-04-14T15:24:02.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.118+0000 7fe4c67fc700 1 -- 172.21.15.90:0/2032705009 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1984987386 0 0) 0x7fe4d0103250 con 0x7fe4d010b830 2024-04-14T15:24:02.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.118+0000 7fe4c67fc700 1 -- 172.21.15.90:0/2032705009 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe4bc0031f0 con 0x7fe4d010b830 2024-04-14T15:24:02.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.118+0000 7fe4c67fc700 1 -- 172.21.15.90:0/2032705009 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3215294591 0 0) 0x7fe4d01ac2c0 con 0x7fe4d0103a20 2024-04-14T15:24:02.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.118+0000 7fe4c67fc700 1 -- 172.21.15.90:0/2032705009 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe4d0103250 con 0x7fe4d0103a20 2024-04-14T15:24:02.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.118+0000 7fe4c67fc700 1 -- 172.21.15.90:0/2032705009 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fe4c0001870 con 0x7fe4d010b830 2024-04-14T15:24:02.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.118+0000 7fe4c67fc700 1 -- 172.21.15.90:0/2032705009 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fe4cc001370 con 0x7fe4d0103a20 2024-04-14T15:24:02.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.118+0000 7fe4c67fc700 1 -- 172.21.15.90:0/2032705009 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3581065945 0 0) 0x7fe4d0103420 con 0x7fe4d010f340 2024-04-14T15:24:02.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.118+0000 7fe4c67fc700 1 -- 172.21.15.90:0/2032705009 >> v1:172.21.15.90:6790/0 conn(0x7fe4d010b830 legacy=0x7fe4d0101d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:02.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.118+0000 7fe4c67fc700 1 -- 172.21.15.90:0/2032705009 >> v1:172.21.15.92:6789/0 conn(0x7fe4d0103a20 legacy=0x7fe4d0101680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:02.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.118+0000 7fe4c67fc700 1 -- 172.21.15.90:0/2032705009 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4d01ad4a0 con 0x7fe4d010f340 2024-04-14T15:24:02.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.118+0000 7fe4d7539700 1 -- 172.21.15.90:0/2032705009 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe4d01ac490 con 0x7fe4d010f340 2024-04-14T15:24:02.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.118+0000 7fe4d7539700 10 client.37863.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:24:02.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.118+0000 7fe4d7539700 1 -- 172.21.15.90:0/2032705009 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe4d01ac950 con 0x7fe4d010f340 2024-04-14T15:24:02.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.118+0000 7fe4d7539700 1 librados: init done 2024-04-14T15:24:02.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.118+0000 7fe4c67fc700 1 -- 172.21.15.90:0/2032705009 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fe4c8001fc0 con 0x7fe4d010f340 2024-04-14T15:24:02.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.118+0000 7fe4c67fc700 1 -- 172.21.15.90:0/2032705009 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fe4c8004eb0 con 0x7fe4d010f340 2024-04-14T15:24:02.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.119+0000 7fe4d7539700 1 -- 172.21.15.90:0/2032705009 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4b4005120 con 0x7fe4d010f340 2024-04-14T15:24:02.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.120+0000 7fe4c67fc700 1 -- 172.21.15.90:0/2032705009 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fe4c8003cb0 con 0x7fe4d010f340 2024-04-14T15:24:02.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.120+0000 7fe4c67fc700 1 -- 172.21.15.90:0/2032705009 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(700..700 src has 467..700) ==== 8721+0+0 (unknown 2834057299 0 0) 0x7fe4c8089d80 con 0x7fe4d010f340 2024-04-14T15:24:02.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.120+0000 7fe4c67fc700 10 client.37863.objecter ms_dispatch 0x7fe4d0001550 osd_map(700..700 src has 467..700) 2024-04-14T15:24:02.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.120+0000 7fe4c67fc700 3 client.37863.objecter handle_osd_map got epochs [700,700] > 0 2024-04-14T15:24:02.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.120+0000 7fe4c67fc700 3 client.37863.objecter handle_osd_map decoding full epoch 700 2024-04-14T15:24:02.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.120+0000 7fe4c67fc700 20 client.37863.objecter dump_active .. 0 homeless 2024-04-14T15:24:02.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.121+0000 7fe4c67fc700 10 client.37863.objecter ms_handle_connect 0x7fe4bc071330 2024-04-14T15:24:02.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.126+0000 7fe4c67fc700 1 -- 172.21.15.90:0/2032705009 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fe4c8056580 con 0x7fe4d010f340 2024-04-14T15:24:02.183 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:24:02.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:02.208 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:02.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:24:02.309 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:24:02.332 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:02.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:02.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:24:02.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.340+0000 7fe4d7539700 1 -- 172.21.15.90:0/2032705009 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7fe4b4005410 con 0x7fe4d010f340 2024-04-14T15:24:02.434 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:24:02.457 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:02.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:02.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:24:02.559 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:24:02.582 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:02.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:02.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:24:02.684 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:24:02.707 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:02.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:02.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:24:02.809 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:24:02.833 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:02.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:02.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:24:02.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.483+0000 7fabc868b700 1 -- 172.21.15.90:0/3001705451 >> v1:172.21.15.90:6789/0 conn(0x7fabc0107770 legacy=0x7fabc0107b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:02.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.484+0000 7fabc868b700 1 -- 172.21.15.90:0/3001705451 shutdown_connections 2024-04-14T15:24:02.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.484+0000 7fabc868b700 1 -- 172.21.15.90:0/3001705451 >> 172.21.15.90:0/3001705451 conn(0x7fabc00fd280 msgr2=0x7fabc00ff6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:24:02.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.484+0000 7fabc868b700 1 -- 172.21.15.90:0/3001705451 shutdown_connections 2024-04-14T15:24:02.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.484+0000 7fabc868b700 1 -- 172.21.15.90:0/3001705451 wait complete. 2024-04-14T15:24:02.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.484+0000 7fabc868b700 1 librados: starting msgr at 2024-04-14T15:24:02.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.484+0000 7fabc868b700 1 librados: starting objecter 2024-04-14T15:24:02.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.484+0000 7fabc868b700 1 Processor -- start 2024-04-14T15:24:02.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.484+0000 7fabc868b700 1 -- start start 2024-04-14T15:24:02.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.484+0000 7fabc868b700 1 librados: setting wanted keys 2024-04-14T15:24:02.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.484+0000 7fabc868b700 1 librados: calling monclient init 2024-04-14T15:24:02.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.485+0000 7fabc868b700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fabc01a67d0 con 0x7fabc010e1f0 2024-04-14T15:24:02.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.485+0000 7fabc868b700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fabc01a79d0 con 0x7fabc010a620 2024-04-14T15:24:02.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.485+0000 7fabc868b700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fabc01a8bd0 con 0x7fabc0107770 2024-04-14T15:24:02.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.485+0000 7fabc6c28700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fabc010e1f0 0x7fabc01a4ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:38346/0 (socket says 172.21.15.90:38346) 2024-04-14T15:24:02.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.485+0000 7fabc6427700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7fabc0107770 0x7fabc010d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:39440/0 (socket says 172.21.15.90:39440) 2024-04-14T15:24:02.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.485+0000 7fabc5c26700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fabc010a620 0x7fabc01a16a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.90:46918/0 (socket says 172.21.15.90:46918) 2024-04-14T15:24:02.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.485+0000 7fabc6c28700 1 -- 172.21.15.90:0/366417754 learned_addr learned my addr 172.21.15.90:0/366417754 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:24:02.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.485+0000 7fabbf7fe700 10 client.?.objecter ms_handle_connect 0x7fabc010e1f0 2024-04-14T15:24:02.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.485+0000 7fabbf7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:24:02.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.485+0000 7fabbf7fe700 10 client.?.objecter ms_handle_connect 0x7fabc0107770 2024-04-14T15:24:02.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.485+0000 7fabbf7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:24:02.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.485+0000 7fabbf7fe700 10 client.?.objecter ms_handle_connect 0x7fabc010a620 2024-04-14T15:24:02.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.485+0000 7fabbf7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:24:02.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.485+0000 7fabbf7fe700 1 -- 172.21.15.90:0/366417754 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3923812497 0 0) 0x7fabc01a79d0 con 0x7fabc010a620 2024-04-14T15:24:02.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.485+0000 7fabbf7fe700 1 -- 172.21.15.90:0/366417754 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faba80031f0 con 0x7fabc010a620 2024-04-14T15:24:02.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.485+0000 7fabbf7fe700 1 -- 172.21.15.90:0/366417754 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2946945857 0 0) 0x7fabc01a67d0 con 0x7fabc010e1f0 2024-04-14T15:24:02.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.485+0000 7fabbf7fe700 1 -- 172.21.15.90:0/366417754 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fabc01a79d0 con 0x7fabc010e1f0 2024-04-14T15:24:02.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.485+0000 7fabbf7fe700 1 -- 172.21.15.90:0/366417754 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3496860544 0 0) 0x7fabc01a8bd0 con 0x7fabc0107770 2024-04-14T15:24:02.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.485+0000 7fabbf7fe700 1 -- 172.21.15.90:0/366417754 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fabc01a67d0 con 0x7fabc0107770 2024-04-14T15:24:02.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.485+0000 7fabbf7fe700 1 -- 172.21.15.90:0/366417754 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1114340333 0 0) 0x7faba80031f0 con 0x7fabc010a620 2024-04-14T15:24:02.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.486+0000 7fabbf7fe700 1 -- 172.21.15.90:0/366417754 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fabc01a8bd0 con 0x7fabc010a620 2024-04-14T15:24:02.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.486+0000 7fabbf7fe700 1 -- 172.21.15.90:0/366417754 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3861621644 0 0) 0x7fabc01a79d0 con 0x7fabc010e1f0 2024-04-14T15:24:02.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.486+0000 7fabbf7fe700 1 -- 172.21.15.90:0/366417754 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faba80031f0 con 0x7fabc010e1f0 2024-04-14T15:24:02.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.486+0000 7fabbf7fe700 1 -- 172.21.15.90:0/366417754 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fabb0001a10 con 0x7fabc010e1f0 2024-04-14T15:24:02.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.486+0000 7fabbf7fe700 1 -- 172.21.15.90:0/366417754 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fabb40019d0 con 0x7fabc010a620 2024-04-14T15:24:02.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.486+0000 7fabbf7fe700 1 -- 172.21.15.90:0/366417754 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 261228555 0 0) 0x7fabc01a8bd0 con 0x7fabc010a620 2024-04-14T15:24:02.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.486+0000 7fabbf7fe700 1 -- 172.21.15.90:0/366417754 >> v1:172.21.15.90:6790/0 conn(0x7fabc0107770 legacy=0x7fabc010d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:02.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.486+0000 7fabbf7fe700 1 -- 172.21.15.90:0/366417754 >> v1:172.21.15.90:6789/0 conn(0x7fabc010e1f0 legacy=0x7fabc01a4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:02.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.486+0000 7fabbf7fe700 1 -- 172.21.15.90:0/366417754 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabc01a9e10 con 0x7fabc010a620 2024-04-14T15:24:02.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.486+0000 7fabc868b700 1 -- 172.21.15.90:0/366417754 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fabc01a8e00 con 0x7fabc010a620 2024-04-14T15:24:02.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.486+0000 7fabc868b700 10 client.49699.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:24:02.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.486+0000 7fabc868b700 1 -- 172.21.15.90:0/366417754 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fabc01a92c0 con 0x7fabc010a620 2024-04-14T15:24:02.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.486+0000 7fabc868b700 1 librados: init done 2024-04-14T15:24:02.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.486+0000 7fabbf7fe700 1 -- 172.21.15.90:0/366417754 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fabb4003b20 con 0x7fabc010a620 2024-04-14T15:24:02.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.487+0000 7fabbf7fe700 1 -- 172.21.15.90:0/366417754 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fabb4005bb0 con 0x7fabc010a620 2024-04-14T15:24:02.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.487+0000 7fabc868b700 1 -- 172.21.15.90:0/366417754 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fabac005130 con 0x7fabc010a620 2024-04-14T15:24:02.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.488+0000 7fabbf7fe700 1 -- 172.21.15.90:0/366417754 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fabb4003680 con 0x7fabc010a620 2024-04-14T15:24:02.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.488+0000 7fabbf7fe700 1 -- 172.21.15.90:0/366417754 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(700..700 src has 467..700) ==== 8721+0+0 (unknown 2834057299 0 0) 0x7fabb408aba0 con 0x7fabc010a620 2024-04-14T15:24:02.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.488+0000 7fabbf7fe700 10 client.49699.objecter ms_dispatch 0x7fabc0001550 osd_map(700..700 src has 467..700) 2024-04-14T15:24:02.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.488+0000 7fabbf7fe700 3 client.49699.objecter handle_osd_map got epochs [700,700] > 0 2024-04-14T15:24:02.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.488+0000 7fabbf7fe700 3 client.49699.objecter handle_osd_map decoding full epoch 700 2024-04-14T15:24:02.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.488+0000 7fabbf7fe700 20 client.49699.objecter dump_active .. 0 homeless 2024-04-14T15:24:02.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.488+0000 7fabbf7fe700 1 -- 172.21.15.90:0/366417754 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v700) ==== 54+0+16 (unknown 14063587 0 2420000192) 0x7fabb408af90 con 0x7fabc010a620 2024-04-14T15:24:02.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.488+0000 7fabbf7fe700 10 client.49699.objecter ms_handle_connect 0x7faba80757c0 2024-04-14T15:24:02.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.488+0000 7fabc868b700 10 client.49699.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:24:02.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.488+0000 7fabc868b700 15 client.49699.objecter _session_command_op_assign -1 1 2024-04-14T15:24:02.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.488+0000 7fabc868b700 20 client.49699.objecter _get_session s=0x7fabac005410 osd=0 3 2024-04-14T15:24:02.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.488+0000 7fabc868b700 20 client.49699.objecter put_session s=0x7fabac005410 osd=0 3 2024-04-14T15:24:02.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.488+0000 7fabc868b700 20 client.49699.objecter _get_session s=0x7fabac005410 osd=0 3 2024-04-14T15:24:02.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.488+0000 7fabc868b700 15 client.49699.objecter _session_command_op_remove -1 1 2024-04-14T15:24:02.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.488+0000 7fabc868b700 20 client.49699.objecter get_session s=0x7fabac005410 osd=0 3 2024-04-14T15:24:02.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.488+0000 7fabc868b700 15 client.49699.objecter _session_command_op_assign 0 1 2024-04-14T15:24:02.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.488+0000 7fabc868b700 20 client.49699.objecter put_session s=0x7fabac005410 osd=0 4 2024-04-14T15:24:02.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.488+0000 7fabc868b700 10 client.49699.objecter _send_command 1 2024-04-14T15:24:02.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.488+0000 7fabc868b700 1 -- 172.21.15.90:0/366417754 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fabac004b80 con 0x7fabac004760 2024-04-14T15:24:02.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.489+0000 7fabbf7fe700 10 client.49699.objecter ms_handle_connect 0x7fabac004760 2024-04-14T15:24:02.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.489+0000 7fabbf7fe700 1 -- 172.21.15.90:0/366417754 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fabac004b80 con 0x7fabac004760 2024-04-14T15:24:02.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.489+0000 7fabbf7fe700 10 client.49699.objecter ms_dispatch 0x7fabc0001550 command_reply(tid 1: 0 ) 2024-04-14T15:24:02.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.489+0000 7fabbf7fe700 10 client.49699.objecter _finish_command 1 = system:0 2024-04-14T15:24:02.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.489+0000 7fabbf7fe700 20 client.49699.objecter put_session s=0x7fabac005410 osd=0 4 2024-04-14T15:24:02.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.489+0000 7fabbf7fe700 15 client.49699.objecter _session_command_op_remove 0 1 2024-04-14T15:24:02.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.506+0000 7fabc868b700 10 client.49699.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:24:02.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.506+0000 7fabc868b700 15 client.49699.objecter _session_command_op_assign -1 2 2024-04-14T15:24:02.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.506+0000 7fabc868b700 20 client.49699.objecter _get_session s=0x7fabac005410 osd=0 3 2024-04-14T15:24:02.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.506+0000 7fabc868b700 20 client.49699.objecter put_session s=0x7fabac005410 osd=0 3 2024-04-14T15:24:02.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.506+0000 7fabc868b700 20 client.49699.objecter _get_session s=0x7fabac005410 osd=0 3 2024-04-14T15:24:02.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.506+0000 7fabc868b700 15 client.49699.objecter _session_command_op_remove -1 2 2024-04-14T15:24:02.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.506+0000 7fabc868b700 20 client.49699.objecter get_session s=0x7fabac005410 osd=0 3 2024-04-14T15:24:02.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.506+0000 7fabc868b700 15 client.49699.objecter _session_command_op_assign 0 2 2024-04-14T15:24:02.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.506+0000 7fabc868b700 20 client.49699.objecter put_session s=0x7fabac005410 osd=0 4 2024-04-14T15:24:02.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.506+0000 7fabc868b700 10 client.49699.objecter _send_command 2 2024-04-14T15:24:02.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.506+0000 7fabc868b700 1 -- 172.21.15.90:0/366417754 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fabac007b30 con 0x7fabac004760 2024-04-14T15:24:02.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.508+0000 7fabbf7fe700 1 -- 172.21.15.90:0/366417754 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fabac007b30 con 0x7fabac004760 2024-04-14T15:24:02.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.508+0000 7fabbf7fe700 10 client.49699.objecter ms_dispatch 0x7fabc0001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:24:02.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.508+0000 7fabbf7fe700 10 client.49699.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:24:02.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.508+0000 7fabbf7fe700 20 client.49699.objecter put_session s=0x7fabac005410 osd=0 4 2024-04-14T15:24:02.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.508+0000 7fabbf7fe700 15 client.49699.objecter _session_command_op_remove 0 2 2024-04-14T15:24:02.897 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:24:02.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.508+0000 7fabc868b700 10 client.49699.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:24:02.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.508+0000 7fabc868b700 15 client.49699.objecter _session_command_op_assign -1 3 2024-04-14T15:24:02.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.508+0000 7fabc868b700 20 client.49699.objecter _get_session s=0x7fabac004160 osd=1 3 2024-04-14T15:24:02.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.508+0000 7fabc868b700 20 client.49699.objecter put_session s=0x7fabac004160 osd=1 3 2024-04-14T15:24:02.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.508+0000 7fabc868b700 20 client.49699.objecter _get_session s=0x7fabac004160 osd=1 3 2024-04-14T15:24:02.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.508+0000 7fabc868b700 15 client.49699.objecter _session_command_op_remove -1 3 2024-04-14T15:24:02.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.508+0000 7fabc868b700 20 client.49699.objecter get_session s=0x7fabac004160 osd=1 3 2024-04-14T15:24:02.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.508+0000 7fabc868b700 15 client.49699.objecter _session_command_op_assign 1 3 2024-04-14T15:24:02.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.508+0000 7fabc868b700 20 client.49699.objecter put_session s=0x7fabac004160 osd=1 4 2024-04-14T15:24:02.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.508+0000 7fabc868b700 10 client.49699.objecter _send_command 3 2024-04-14T15:24:02.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.508+0000 7fabc868b700 1 -- 172.21.15.90:0/366417754 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fabac002190 con 0x7fabac0042d0 2024-04-14T15:24:02.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.508+0000 7fabbf7fe700 10 client.49699.objecter ms_handle_connect 0x7fabac0042d0 2024-04-14T15:24:02.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.509+0000 7fabbf7fe700 1 -- 172.21.15.90:0/366417754 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fabac002190 con 0x7fabac0042d0 2024-04-14T15:24:02.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.509+0000 7fabbf7fe700 10 client.49699.objecter ms_dispatch 0x7fabc0001550 command_reply(tid 3: 0 ) 2024-04-14T15:24:02.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.509+0000 7fabbf7fe700 10 client.49699.objecter _finish_command 3 = system:0 2024-04-14T15:24:02.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.509+0000 7fabbf7fe700 20 client.49699.objecter put_session s=0x7fabac004160 osd=1 4 2024-04-14T15:24:02.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.509+0000 7fabbf7fe700 15 client.49699.objecter _session_command_op_remove 1 3 2024-04-14T15:24:02.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.526+0000 7fabc868b700 10 client.49699.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:24:02.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.526+0000 7fabc868b700 15 client.49699.objecter _session_command_op_assign -1 4 2024-04-14T15:24:02.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.526+0000 7fabc868b700 20 client.49699.objecter _get_session s=0x7fabac004160 osd=1 3 2024-04-14T15:24:02.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.526+0000 7fabc868b700 20 client.49699.objecter put_session s=0x7fabac004160 osd=1 3 2024-04-14T15:24:02.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.526+0000 7fabc868b700 20 client.49699.objecter _get_session s=0x7fabac004160 osd=1 3 2024-04-14T15:24:02.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.526+0000 7fabc868b700 15 client.49699.objecter _session_command_op_remove -1 4 2024-04-14T15:24:02.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.526+0000 7fabc868b700 20 client.49699.objecter get_session s=0x7fabac004160 osd=1 3 2024-04-14T15:24:02.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.526+0000 7fabc868b700 15 client.49699.objecter _session_command_op_assign 1 4 2024-04-14T15:24:02.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.526+0000 7fabc868b700 20 client.49699.objecter put_session s=0x7fabac004160 osd=1 4 2024-04-14T15:24:02.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.526+0000 7fabc868b700 10 client.49699.objecter _send_command 4 2024-04-14T15:24:02.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.526+0000 7fabc868b700 1 -- 172.21.15.90:0/366417754 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fabac001630 con 0x7fabac0042d0 2024-04-14T15:24:02.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.527+0000 7fabbf7fe700 1 -- 172.21.15.90:0/366417754 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fabac001630 con 0x7fabac0042d0 2024-04-14T15:24:02.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.527+0000 7fabbf7fe700 10 client.49699.objecter ms_dispatch 0x7fabc0001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:24:02.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.527+0000 7fabbf7fe700 10 client.49699.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:24:02.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.527+0000 7fabbf7fe700 20 client.49699.objecter put_session s=0x7fabac004160 osd=1 4 2024-04-14T15:24:02.900 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:24:02.900 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:24:02.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.527+0000 7fabbf7fe700 15 client.49699.objecter _session_command_op_remove 1 4 2024-04-14T15:24:02.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.527+0000 7fabc868b700 10 client.49699.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:24:02.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.527+0000 7fabc868b700 15 client.49699.objecter _session_command_op_assign -1 5 2024-04-14T15:24:02.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.527+0000 7fabc868b700 20 client.49699.objecter _get_session s=0x7fabac001a30 osd=2 3 2024-04-14T15:24:02.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.527+0000 7fabc868b700 20 client.49699.objecter put_session s=0x7fabac001a30 osd=2 3 2024-04-14T15:24:02.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.527+0000 7fabc868b700 20 client.49699.objecter _get_session s=0x7fabac001a30 osd=2 3 2024-04-14T15:24:02.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.527+0000 7fabc868b700 15 client.49699.objecter _session_command_op_remove -1 5 2024-04-14T15:24:02.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.527+0000 7fabc868b700 20 client.49699.objecter get_session s=0x7fabac001a30 osd=2 3 2024-04-14T15:24:02.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.527+0000 7fabc868b700 15 client.49699.objecter _session_command_op_assign 2 5 2024-04-14T15:24:02.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.527+0000 7fabc868b700 20 client.49699.objecter put_session s=0x7fabac001a30 osd=2 4 2024-04-14T15:24:02.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.527+0000 7fabc868b700 10 client.49699.objecter _send_command 5 2024-04-14T15:24:02.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.527+0000 7fabc868b700 1 -- 172.21.15.90:0/366417754 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fabac011040 con 0x7fabac0089c0 2024-04-14T15:24:02.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.528+0000 7fabbf7fe700 10 client.49699.objecter ms_handle_connect 0x7fabac0089c0 2024-04-14T15:24:02.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.528+0000 7fabbf7fe700 1 -- 172.21.15.90:0/366417754 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fabac011040 con 0x7fabac0089c0 2024-04-14T15:24:02.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.528+0000 7fabbf7fe700 10 client.49699.objecter ms_dispatch 0x7fabc0001550 command_reply(tid 5: 0 ) 2024-04-14T15:24:02.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.528+0000 7fabbf7fe700 10 client.49699.objecter _finish_command 5 = system:0 2024-04-14T15:24:02.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.528+0000 7fabbf7fe700 20 client.49699.objecter put_session s=0x7fabac001a30 osd=2 4 2024-04-14T15:24:02.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.528+0000 7fabbf7fe700 15 client.49699.objecter _session_command_op_remove 2 5 2024-04-14T15:24:02.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.545+0000 7fabc868b700 10 client.49699.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:24:02.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.545+0000 7fabc868b700 15 client.49699.objecter _session_command_op_assign -1 6 2024-04-14T15:24:02.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.545+0000 7fabc868b700 20 client.49699.objecter _get_session s=0x7fabac001a30 osd=2 3 2024-04-14T15:24:02.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.545+0000 7fabc868b700 20 client.49699.objecter put_session s=0x7fabac001a30 osd=2 3 2024-04-14T15:24:02.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.545+0000 7fabc868b700 20 client.49699.objecter _get_session s=0x7fabac001a30 osd=2 3 2024-04-14T15:24:02.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.545+0000 7fabc868b700 15 client.49699.objecter _session_command_op_remove -1 6 2024-04-14T15:24:02.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.545+0000 7fabc868b700 20 client.49699.objecter get_session s=0x7fabac001a30 osd=2 3 2024-04-14T15:24:02.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.545+0000 7fabc868b700 15 client.49699.objecter _session_command_op_assign 2 6 2024-04-14T15:24:02.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.545+0000 7fabc868b700 20 client.49699.objecter put_session s=0x7fabac001a30 osd=2 4 2024-04-14T15:24:02.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.545+0000 7fabc868b700 10 client.49699.objecter _send_command 6 2024-04-14T15:24:02.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.545+0000 7fabc868b700 1 -- 172.21.15.90:0/366417754 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fabac008050 con 0x7fabac0089c0 2024-04-14T15:24:02.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.546+0000 7fabbf7fe700 1 -- 172.21.15.90:0/366417754 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fabac008050 con 0x7fabac0089c0 2024-04-14T15:24:02.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.546+0000 7fabbf7fe700 10 client.49699.objecter ms_dispatch 0x7fabc0001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:24:02.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.546+0000 7fabbf7fe700 10 client.49699.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:24:02.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.546+0000 7fabbf7fe700 20 client.49699.objecter put_session s=0x7fabac001a30 osd=2 4 2024-04-14T15:24:02.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.546+0000 7fabbf7fe700 15 client.49699.objecter _session_command_op_remove 2 6 2024-04-14T15:24:02.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.546+0000 7fabc868b700 10 client.49699.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:24:02.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.546+0000 7fabc868b700 15 client.49699.objecter _session_command_op_assign -1 7 2024-04-14T15:24:02.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.546+0000 7fabc868b700 20 client.49699.objecter _get_session s=0x7fabac0082a0 osd=3 3 2024-04-14T15:24:02.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.546+0000 7fabc868b700 20 client.49699.objecter put_session s=0x7fabac0082a0 osd=3 3 2024-04-14T15:24:02.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.546+0000 7fabc868b700 20 client.49699.objecter _get_session s=0x7fabac0082a0 osd=3 3 2024-04-14T15:24:02.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.546+0000 7fabc868b700 15 client.49699.objecter _session_command_op_remove -1 7 2024-04-14T15:24:02.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.546+0000 7fabc868b700 20 client.49699.objecter get_session s=0x7fabac0082a0 osd=3 3 2024-04-14T15:24:02.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.546+0000 7fabc868b700 15 client.49699.objecter _session_command_op_assign 3 7 2024-04-14T15:24:02.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.546+0000 7fabc868b700 20 client.49699.objecter put_session s=0x7fabac0082a0 osd=3 4 2024-04-14T15:24:02.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.546+0000 7fabc868b700 10 client.49699.objecter _send_command 7 2024-04-14T15:24:02.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.547+0000 7fabc868b700 1 -- 172.21.15.90:0/366417754 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fabac011d20 con 0x7fabac0111f0 2024-04-14T15:24:02.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.547+0000 7fabbf7fe700 10 client.49699.objecter ms_handle_connect 0x7fabac0111f0 2024-04-14T15:24:02.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.547+0000 7fabbf7fe700 1 -- 172.21.15.90:0/366417754 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fabac011d20 con 0x7fabac0111f0 2024-04-14T15:24:02.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.547+0000 7fabbf7fe700 10 client.49699.objecter ms_dispatch 0x7fabc0001550 command_reply(tid 7: 0 ) 2024-04-14T15:24:02.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.547+0000 7fabbf7fe700 10 client.49699.objecter _finish_command 7 = system:0 2024-04-14T15:24:02.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.547+0000 7fabbf7fe700 20 client.49699.objecter put_session s=0x7fabac0082a0 osd=3 4 2024-04-14T15:24:02.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.547+0000 7fabbf7fe700 15 client.49699.objecter _session_command_op_remove 3 7 2024-04-14T15:24:02.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.564+0000 7fabc868b700 10 client.49699.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:24:02.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.564+0000 7fabc868b700 15 client.49699.objecter _session_command_op_assign -1 8 2024-04-14T15:24:02.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.564+0000 7fabc868b700 20 client.49699.objecter _get_session s=0x7fabac0082a0 osd=3 3 2024-04-14T15:24:02.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.564+0000 7fabc868b700 20 client.49699.objecter put_session s=0x7fabac0082a0 osd=3 3 2024-04-14T15:24:02.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.564+0000 7fabc868b700 20 client.49699.objecter _get_session s=0x7fabac0082a0 osd=3 3 2024-04-14T15:24:02.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.564+0000 7fabc868b700 15 client.49699.objecter _session_command_op_remove -1 8 2024-04-14T15:24:02.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.564+0000 7fabc868b700 20 client.49699.objecter get_session s=0x7fabac0082a0 osd=3 3 2024-04-14T15:24:02.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.564+0000 7fabc868b700 15 client.49699.objecter _session_command_op_assign 3 8 2024-04-14T15:24:02.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.564+0000 7fabc868b700 20 client.49699.objecter put_session s=0x7fabac0082a0 osd=3 4 2024-04-14T15:24:02.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.564+0000 7fabc868b700 10 client.49699.objecter _send_command 8 2024-04-14T15:24:02.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.564+0000 7fabc868b700 1 -- 172.21.15.90:0/366417754 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fabac0126d0 con 0x7fabac0111f0 2024-04-14T15:24:02.905 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:24:02.905 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:24:02.905 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:24:02.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.566+0000 7fabbf7fe700 1 -- 172.21.15.90:0/366417754 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fabac0126d0 con 0x7fabac0111f0 2024-04-14T15:24:02.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.566+0000 7fabbf7fe700 10 client.49699.objecter ms_dispatch 0x7fabc0001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:24:02.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.566+0000 7fabbf7fe700 10 client.49699.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:24:02.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.566+0000 7fabbf7fe700 20 client.49699.objecter put_session s=0x7fabac0082a0 osd=3 4 2024-04-14T15:24:02.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.566+0000 7fabbf7fe700 15 client.49699.objecter _session_command_op_remove 3 8 2024-04-14T15:24:02.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.566+0000 7fabc868b700 10 client.49699.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:24:02.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.566+0000 7fabc868b700 15 client.49699.objecter _session_command_op_assign -1 9 2024-04-14T15:24:02.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.566+0000 7fabc868b700 20 client.49699.objecter _get_session s=0x7fabac0126d0 osd=4 3 2024-04-14T15:24:02.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.566+0000 7fabc868b700 20 client.49699.objecter put_session s=0x7fabac0126d0 osd=4 3 2024-04-14T15:24:02.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.566+0000 7fabc868b700 20 client.49699.objecter _get_session s=0x7fabac0126d0 osd=4 3 2024-04-14T15:24:02.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.566+0000 7fabc868b700 15 client.49699.objecter _session_command_op_remove -1 9 2024-04-14T15:24:02.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.566+0000 7fabc868b700 20 client.49699.objecter get_session s=0x7fabac0126d0 osd=4 3 2024-04-14T15:24:02.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.566+0000 7fabc868b700 15 client.49699.objecter _session_command_op_assign 4 9 2024-04-14T15:24:02.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.566+0000 7fabc868b700 20 client.49699.objecter put_session s=0x7fabac0126d0 osd=4 4 2024-04-14T15:24:02.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.566+0000 7fabc868b700 10 client.49699.objecter _send_command 9 2024-04-14T15:24:02.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.566+0000 7fabc868b700 1 -- 172.21.15.90:0/366417754 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fabac01a840 con 0x7fabac016c00 2024-04-14T15:24:02.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.566+0000 7fabbf7fe700 10 client.49699.objecter ms_handle_connect 0x7fabac016c00 2024-04-14T15:24:02.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.567+0000 7fabbf7fe700 1 -- 172.21.15.90:0/366417754 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fabac01a840 con 0x7fabac016c00 2024-04-14T15:24:02.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.567+0000 7fabbf7fe700 10 client.49699.objecter ms_dispatch 0x7fabc0001550 command_reply(tid 9: 0 ) 2024-04-14T15:24:02.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.567+0000 7fabbf7fe700 10 client.49699.objecter _finish_command 9 = system:0 2024-04-14T15:24:02.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.567+0000 7fabbf7fe700 20 client.49699.objecter put_session s=0x7fabac0126d0 osd=4 4 2024-04-14T15:24:02.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.567+0000 7fabbf7fe700 15 client.49699.objecter _session_command_op_remove 4 9 2024-04-14T15:24:02.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.585+0000 7fabc868b700 10 client.49699.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:24:02.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.585+0000 7fabc868b700 15 client.49699.objecter _session_command_op_assign -1 10 2024-04-14T15:24:02.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.585+0000 7fabc868b700 20 client.49699.objecter _get_session s=0x7fabac0126d0 osd=4 3 2024-04-14T15:24:02.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.585+0000 7fabc868b700 20 client.49699.objecter put_session s=0x7fabac0126d0 osd=4 3 2024-04-14T15:24:02.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.585+0000 7fabc868b700 20 client.49699.objecter _get_session s=0x7fabac0126d0 osd=4 3 2024-04-14T15:24:02.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.585+0000 7fabc868b700 15 client.49699.objecter _session_command_op_remove -1 10 2024-04-14T15:24:02.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.585+0000 7fabc868b700 20 client.49699.objecter get_session s=0x7fabac0126d0 osd=4 3 2024-04-14T15:24:02.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.585+0000 7fabc868b700 15 client.49699.objecter _session_command_op_assign 4 10 2024-04-14T15:24:02.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.585+0000 7fabc868b700 20 client.49699.objecter put_session s=0x7fabac0126d0 osd=4 4 2024-04-14T15:24:02.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.585+0000 7fabc868b700 10 client.49699.objecter _send_command 10 2024-04-14T15:24:02.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.585+0000 7fabc868b700 1 -- 172.21.15.90:0/366417754 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fabac01aa20 con 0x7fabac016c00 2024-04-14T15:24:02.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.586+0000 7fabbf7fe700 1 -- 172.21.15.90:0/366417754 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fabac01aa20 con 0x7fabac016c00 2024-04-14T15:24:02.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.586+0000 7fabbf7fe700 10 client.49699.objecter ms_dispatch 0x7fabc0001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:24:02.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.586+0000 7fabbf7fe700 10 client.49699.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:24:02.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.587+0000 7fabbf7fe700 20 client.49699.objecter put_session s=0x7fabac0126d0 osd=4 4 2024-04-14T15:24:02.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.587+0000 7fabbf7fe700 15 client.49699.objecter _session_command_op_remove 4 10 2024-04-14T15:24:02.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.587+0000 7fabc868b700 10 client.49699.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:24:02.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.587+0000 7fabc868b700 15 client.49699.objecter _session_command_op_assign -1 11 2024-04-14T15:24:02.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.587+0000 7fabc868b700 20 client.49699.objecter _get_session s=0x7fabac01aa20 osd=5 3 2024-04-14T15:24:02.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.587+0000 7fabc868b700 20 client.49699.objecter put_session s=0x7fabac01aa20 osd=5 3 2024-04-14T15:24:02.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.587+0000 7fabc868b700 20 client.49699.objecter _get_session s=0x7fabac01aa20 osd=5 3 2024-04-14T15:24:02.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.587+0000 7fabc868b700 15 client.49699.objecter _session_command_op_remove -1 11 2024-04-14T15:24:02.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.587+0000 7fabc868b700 20 client.49699.objecter get_session s=0x7fabac01aa20 osd=5 3 2024-04-14T15:24:02.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.587+0000 7fabc868b700 15 client.49699.objecter _session_command_op_assign 5 11 2024-04-14T15:24:02.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.587+0000 7fabc868b700 20 client.49699.objecter put_session s=0x7fabac01aa20 osd=5 4 2024-04-14T15:24:02.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.587+0000 7fabc868b700 10 client.49699.objecter _send_command 11 2024-04-14T15:24:02.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.587+0000 7fabc868b700 1 -- 172.21.15.90:0/366417754 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fabac01be10 con 0x7fabac01b140 2024-04-14T15:24:02.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.587+0000 7fabbf7fe700 10 client.49699.objecter ms_handle_connect 0x7fabac01b140 2024-04-14T15:24:02.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.588+0000 7fabbf7fe700 1 -- 172.21.15.90:0/366417754 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fabac01be10 con 0x7fabac01b140 2024-04-14T15:24:02.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.588+0000 7fabbf7fe700 10 client.49699.objecter ms_dispatch 0x7fabc0001550 command_reply(tid 11: 0 ) 2024-04-14T15:24:02.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.588+0000 7fabbf7fe700 10 client.49699.objecter _finish_command 11 = system:0 2024-04-14T15:24:02.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.588+0000 7fabbf7fe700 20 client.49699.objecter put_session s=0x7fabac01aa20 osd=5 4 2024-04-14T15:24:02.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.588+0000 7fabbf7fe700 15 client.49699.objecter _session_command_op_remove 5 11 2024-04-14T15:24:02.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.606+0000 7fabc868b700 10 client.49699.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:24:02.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.606+0000 7fabc868b700 15 client.49699.objecter _session_command_op_assign -1 12 2024-04-14T15:24:02.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.606+0000 7fabc868b700 20 client.49699.objecter _get_session s=0x7fabac01aa20 osd=5 3 2024-04-14T15:24:02.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.606+0000 7fabc868b700 20 client.49699.objecter put_session s=0x7fabac01aa20 osd=5 3 2024-04-14T15:24:02.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.606+0000 7fabc868b700 20 client.49699.objecter _get_session s=0x7fabac01aa20 osd=5 3 2024-04-14T15:24:02.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.606+0000 7fabc868b700 15 client.49699.objecter _session_command_op_remove -1 12 2024-04-14T15:24:02.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.606+0000 7fabc868b700 20 client.49699.objecter get_session s=0x7fabac01aa20 osd=5 3 2024-04-14T15:24:02.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.606+0000 7fabc868b700 15 client.49699.objecter _session_command_op_assign 5 12 2024-04-14T15:24:02.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.606+0000 7fabc868b700 20 client.49699.objecter put_session s=0x7fabac01aa20 osd=5 4 2024-04-14T15:24:02.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.606+0000 7fabc868b700 10 client.49699.objecter _send_command 12 2024-04-14T15:24:02.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.606+0000 7fabc868b700 1 -- 172.21.15.90:0/366417754 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fabac01be10 con 0x7fabac01b140 2024-04-14T15:24:02.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.608+0000 7fabbf7fe700 1 -- 172.21.15.90:0/366417754 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fabac01be10 con 0x7fabac01b140 2024-04-14T15:24:02.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.608+0000 7fabbf7fe700 10 client.49699.objecter ms_dispatch 0x7fabc0001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:24:02.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.608+0000 7fabbf7fe700 10 client.49699.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:24:02.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.608+0000 7fabbf7fe700 20 client.49699.objecter put_session s=0x7fabac01aa20 osd=5 4 2024-04-14T15:24:02.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.608+0000 7fabbf7fe700 15 client.49699.objecter _session_command_op_remove 5 12 2024-04-14T15:24:02.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.609+0000 7fabc868b700 10 client.49699.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:24:02.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.609+0000 7fabc868b700 15 client.49699.objecter _session_command_op_assign -1 13 2024-04-14T15:24:02.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.609+0000 7fabc868b700 20 client.49699.objecter _get_session s=0x7fabac01c070 osd=6 3 2024-04-14T15:24:02.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.609+0000 7fabc868b700 20 client.49699.objecter put_session s=0x7fabac01c070 osd=6 3 2024-04-14T15:24:02.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.609+0000 7fabc868b700 20 client.49699.objecter _get_session s=0x7fabac01c070 osd=6 3 2024-04-14T15:24:02.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.609+0000 7fabc868b700 15 client.49699.objecter _session_command_op_remove -1 13 2024-04-14T15:24:02.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.609+0000 7fabc868b700 20 client.49699.objecter get_session s=0x7fabac01c070 osd=6 3 2024-04-14T15:24:02.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.609+0000 7fabc868b700 15 client.49699.objecter _session_command_op_assign 6 13 2024-04-14T15:24:02.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.609+0000 7fabc868b700 20 client.49699.objecter put_session s=0x7fabac01c070 osd=6 4 2024-04-14T15:24:02.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.609+0000 7fabc868b700 10 client.49699.objecter _send_command 13 2024-04-14T15:24:02.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.609+0000 7fabc868b700 1 -- 172.21.15.90:0/366417754 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fabac023300 con 0x7fabac01f720 2024-04-14T15:24:02.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.609+0000 7fabbf7fe700 10 client.49699.objecter ms_handle_connect 0x7fabac01f720 2024-04-14T15:24:02.911 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:24:02.912 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:24:02.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.610+0000 7fabbf7fe700 1 -- 172.21.15.90:0/366417754 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fabac023300 con 0x7fabac01f720 2024-04-14T15:24:02.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.610+0000 7fabbf7fe700 10 client.49699.objecter ms_dispatch 0x7fabc0001550 command_reply(tid 13: 0 ) 2024-04-14T15:24:02.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.610+0000 7fabbf7fe700 10 client.49699.objecter _finish_command 13 = system:0 2024-04-14T15:24:02.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.610+0000 7fabbf7fe700 20 client.49699.objecter put_session s=0x7fabac01c070 osd=6 4 2024-04-14T15:24:02.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.610+0000 7fabbf7fe700 15 client.49699.objecter _session_command_op_remove 6 13 2024-04-14T15:24:02.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.627+0000 7fabc868b700 10 client.49699.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:24:02.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.627+0000 7fabc868b700 15 client.49699.objecter _session_command_op_assign -1 14 2024-04-14T15:24:02.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.627+0000 7fabc868b700 20 client.49699.objecter _get_session s=0x7fabac01c070 osd=6 3 2024-04-14T15:24:02.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.627+0000 7fabc868b700 20 client.49699.objecter put_session s=0x7fabac01c070 osd=6 3 2024-04-14T15:24:02.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.627+0000 7fabc868b700 20 client.49699.objecter _get_session s=0x7fabac01c070 osd=6 3 2024-04-14T15:24:02.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.627+0000 7fabc868b700 15 client.49699.objecter _session_command_op_remove -1 14 2024-04-14T15:24:02.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.627+0000 7fabc868b700 20 client.49699.objecter get_session s=0x7fabac01c070 osd=6 3 2024-04-14T15:24:02.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.627+0000 7fabc868b700 15 client.49699.objecter _session_command_op_assign 6 14 2024-04-14T15:24:02.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.627+0000 7fabc868b700 20 client.49699.objecter put_session s=0x7fabac01c070 osd=6 4 2024-04-14T15:24:02.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.627+0000 7fabc868b700 10 client.49699.objecter _send_command 14 2024-04-14T15:24:02.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.627+0000 7fabc868b700 1 -- 172.21.15.90:0/366417754 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fabac023300 con 0x7fabac01f720 2024-04-14T15:24:02.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.629+0000 7fabbf7fe700 1 -- 172.21.15.90:0/366417754 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fabac023300 con 0x7fabac01f720 2024-04-14T15:24:02.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.629+0000 7fabbf7fe700 10 client.49699.objecter ms_dispatch 0x7fabc0001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:24:02.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.629+0000 7fabbf7fe700 10 client.49699.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:24:02.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.629+0000 7fabbf7fe700 20 client.49699.objecter put_session s=0x7fabac01c070 osd=6 4 2024-04-14T15:24:02.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.629+0000 7fabbf7fe700 15 client.49699.objecter _session_command_op_remove 6 14 2024-04-14T15:24:02.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.629+0000 7fabc868b700 10 client.49699.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:24:02.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.629+0000 7fabc868b700 15 client.49699.objecter _session_command_op_assign -1 15 2024-04-14T15:24:02.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.629+0000 7fabc868b700 20 client.49699.objecter _get_session s=0x7fabac023340 osd=7 3 2024-04-14T15:24:02.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.629+0000 7fabc868b700 20 client.49699.objecter put_session s=0x7fabac023340 osd=7 3 2024-04-14T15:24:02.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.629+0000 7fabc868b700 20 client.49699.objecter _get_session s=0x7fabac023340 osd=7 3 2024-04-14T15:24:02.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.629+0000 7fabc868b700 15 client.49699.objecter _session_command_op_remove -1 15 2024-04-14T15:24:02.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.629+0000 7fabc868b700 20 client.49699.objecter get_session s=0x7fabac023340 osd=7 3 2024-04-14T15:24:02.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.629+0000 7fabc868b700 15 client.49699.objecter _session_command_op_assign 7 15 2024-04-14T15:24:02.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.629+0000 7fabc868b700 20 client.49699.objecter put_session s=0x7fabac023340 osd=7 4 2024-04-14T15:24:02.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.629+0000 7fabc868b700 10 client.49699.objecter _send_command 15 2024-04-14T15:24:02.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.629+0000 7fabc868b700 1 -- 172.21.15.90:0/366417754 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fabac024590 con 0x7fabac025200 2024-04-14T15:24:02.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.630+0000 7fabbf7fe700 10 client.49699.objecter ms_handle_connect 0x7fabac025200 2024-04-14T15:24:02.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.631+0000 7fabbf7fe700 1 -- 172.21.15.90:0/366417754 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fabac024590 con 0x7fabac025200 2024-04-14T15:24:02.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.631+0000 7fabbf7fe700 10 client.49699.objecter ms_dispatch 0x7fabc0001550 command_reply(tid 15: 0 ) 2024-04-14T15:24:02.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.631+0000 7fabbf7fe700 10 client.49699.objecter _finish_command 15 = system:0 2024-04-14T15:24:02.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.631+0000 7fabbf7fe700 20 client.49699.objecter put_session s=0x7fabac023340 osd=7 4 2024-04-14T15:24:02.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.631+0000 7fabbf7fe700 15 client.49699.objecter _session_command_op_remove 7 15 2024-04-14T15:24:02.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.647+0000 7fabc868b700 10 client.49699.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:24:02.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.647+0000 7fabc868b700 15 client.49699.objecter _session_command_op_assign -1 16 2024-04-14T15:24:02.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.647+0000 7fabc868b700 20 client.49699.objecter _get_session s=0x7fabac023340 osd=7 3 2024-04-14T15:24:02.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.647+0000 7fabc868b700 20 client.49699.objecter put_session s=0x7fabac023340 osd=7 3 2024-04-14T15:24:02.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.647+0000 7fabc868b700 20 client.49699.objecter _get_session s=0x7fabac023340 osd=7 3 2024-04-14T15:24:02.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.647+0000 7fabc868b700 15 client.49699.objecter _session_command_op_remove -1 16 2024-04-14T15:24:02.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.647+0000 7fabc868b700 20 client.49699.objecter get_session s=0x7fabac023340 osd=7 3 2024-04-14T15:24:02.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.647+0000 7fabc868b700 15 client.49699.objecter _session_command_op_assign 7 16 2024-04-14T15:24:02.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.647+0000 7fabc868b700 20 client.49699.objecter put_session s=0x7fabac023340 osd=7 4 2024-04-14T15:24:02.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.647+0000 7fabc868b700 10 client.49699.objecter _send_command 16 2024-04-14T15:24:02.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.647+0000 7fabc868b700 1 -- 172.21.15.90:0/366417754 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fabac024590 con 0x7fabac025200 2024-04-14T15:24:02.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.650+0000 7fabbf7fe700 1 -- 172.21.15.90:0/366417754 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fabac024590 con 0x7fabac025200 2024-04-14T15:24:02.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.650+0000 7fabbf7fe700 10 client.49699.objecter ms_dispatch 0x7fabc0001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:24:02.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.650+0000 7fabbf7fe700 10 client.49699.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:24:02.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.650+0000 7fabbf7fe700 20 client.49699.objecter put_session s=0x7fabac023340 osd=7 4 2024-04-14T15:24:02.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.650+0000 7fabbf7fe700 15 client.49699.objecter _session_command_op_remove 7 16 2024-04-14T15:24:02.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.650+0000 7fabc868b700 10 librados: watch_flush enter 2024-04-14T15:24:02.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.650+0000 7fabc868b700 10 librados: watch_flush exit 2024-04-14T15:24:02.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.650+0000 7fabc868b700 10 client.49699.objecter close_session for osd.0 2024-04-14T15:24:02.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.650+0000 7fabc868b700 1 -- 172.21.15.90:0/366417754 >> v1:172.21.15.90:6801/3761806885 conn(0x7fabac004760 legacy=0x7fabac002780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:02.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.650+0000 7fabc868b700 20 client.49699.objecter put_session s=0x7fabac005410 osd=0 1 2024-04-14T15:24:02.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.650+0000 7fabc868b700 10 client.49699.objecter close_session for osd.1 2024-04-14T15:24:02.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.650+0000 7fabc868b700 1 -- 172.21.15.90:0/366417754 >> v1:172.21.15.90:6805/4170361699 conn(0x7fabac0042d0 legacy=0x7fabac00ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:02.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.650+0000 7fabc868b700 20 client.49699.objecter put_session s=0x7fabac004160 osd=1 1 2024-04-14T15:24:02.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.650+0000 7fabc868b700 10 client.49699.objecter close_session for osd.2 2024-04-14T15:24:02.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.650+0000 7fabc868b700 1 -- 172.21.15.90:0/366417754 >> v1:172.21.15.90:6809/1686184251 conn(0x7fabac0089c0 legacy=0x7fabac00f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:02.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.650+0000 7fabc868b700 20 client.49699.objecter put_session s=0x7fabac001a30 osd=2 1 2024-04-14T15:24:02.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.650+0000 7fabc868b700 10 client.49699.objecter close_session for osd.3 2024-04-14T15:24:02.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.650+0000 7fabc868b700 1 -- 172.21.15.90:0/366417754 >> v1:172.21.15.90:6813/2270166774 conn(0x7fabac0111f0 legacy=0x7fabac011610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:02.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.650+0000 7fabc868b700 20 client.49699.objecter put_session s=0x7fabac0082a0 osd=3 1 2024-04-14T15:24:02.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.650+0000 7fabc868b700 10 client.49699.objecter close_session for osd.4 2024-04-14T15:24:02.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.650+0000 7fabc868b700 1 -- 172.21.15.90:0/366417754 >> v1:172.21.15.92:6800/3887549733 conn(0x7fabac016c00 legacy=0x7fabac018ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:02.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.650+0000 7fabc868b700 20 client.49699.objecter put_session s=0x7fabac0126d0 osd=4 1 2024-04-14T15:24:02.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.650+0000 7fabc6c28700 1 -- 172.21.15.90:0/366417754 reap_dead start 2024-04-14T15:24:02.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.650+0000 7fabc868b700 10 client.49699.objecter close_session for osd.5 2024-04-14T15:24:02.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.650+0000 7fabc868b700 1 -- 172.21.15.90:0/366417754 >> v1:172.21.15.92:6804/3244561440 conn(0x7fabac01b140 legacy=0x7fabac01b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:02.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.650+0000 7fabc6c28700 1 -- 172.21.15.90:0/366417754 reap_dead start 2024-04-14T15:24:02.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.650+0000 7fabc868b700 20 client.49699.objecter put_session s=0x7fabac01aa20 osd=5 1 2024-04-14T15:24:02.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.650+0000 7fabc868b700 10 client.49699.objecter close_session for osd.6 2024-04-14T15:24:02.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.650+0000 7fabc868b700 1 -- 172.21.15.90:0/366417754 >> v1:172.21.15.92:6808/636125963 conn(0x7fabac01f720 legacy=0x7fabac021b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:02.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.650+0000 7fabc868b700 20 client.49699.objecter put_session s=0x7fabac01c070 osd=6 1 2024-04-14T15:24:02.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.650+0000 7fabc868b700 10 client.49699.objecter close_session for osd.7 2024-04-14T15:24:02.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.650+0000 7fabc868b700 1 -- 172.21.15.90:0/366417754 >> v1:172.21.15.92:6812/1773706716 conn(0x7fabac025200 legacy=0x7fabac025660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:02.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.650+0000 7fabc868b700 20 client.49699.objecter put_session s=0x7fabac023340 osd=7 1 2024-04-14T15:24:02.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.650+0000 7fabc868b700 20 client.49699.objecter shutdown clearing up homeless session... 2024-04-14T15:24:02.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.650+0000 7fabc868b700 10 client.49699.objecter successfully canceled tick 2024-04-14T15:24:02.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.650+0000 7fabc6c28700 1 -- 172.21.15.90:0/366417754 reap_dead start 2024-04-14T15:24:02.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.650+0000 7fabc868b700 1 -- 172.21.15.90:0/366417754 >> v1:172.21.15.90:6800/420935427 conn(0x7faba80757c0 legacy=0x7faba8077c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:02.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.650+0000 7fabc868b700 1 -- 172.21.15.90:0/366417754 >> v1:172.21.15.92:6789/0 conn(0x7fabc010a620 legacy=0x7fabc01a16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:02.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.650+0000 7fabc868b700 1 -- 172.21.15.90:0/366417754 shutdown_connections 2024-04-14T15:24:02.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.650+0000 7fabc868b700 1 -- 172.21.15.90:0/366417754 >> 172.21.15.90:0/366417754 conn(0x7fabc00fd280 msgr2=0x7fabc0110e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:24:02.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.650+0000 7fabc868b700 1 -- 172.21.15.90:0/366417754 shutdown_connections 2024-04-14T15:24:02.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.650+0000 7fabc868b700 1 -- 172.21.15.90:0/366417754 wait complete. 2024-04-14T15:24:02.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.650+0000 7fabc868b700 1 librados: shutdown 2024-04-14T15:24:02.918 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:24:02.918 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:24:02.918 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:24:02.918 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:24:02.918 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:24:02.918 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:24:02.918 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:24:02.919 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:24:02.935 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:24:02.960 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:02.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:02.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:24:02.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.995+0000 7fe4c67fc700 1 -- 172.21.15.90:0/2032705009 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v701) ==== 97+0+0 (unknown 482125050 0 0) 0x7fe4c805a230 con 0x7fe4d010f340 2024-04-14T15:24:02.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.997+0000 7fe4d7539700 10 librados: watch_flush enter 2024-04-14T15:24:02.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.997+0000 7fe4d7539700 10 librados: watch_flush exit 2024-04-14T15:24:02.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.997+0000 7fe4d7539700 20 client.37863.objecter shutdown clearing up homeless session... 2024-04-14T15:24:02.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.997+0000 7fe4d7539700 10 client.37863.objecter successfully canceled tick 2024-04-14T15:24:02.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.997+0000 7fe4d7539700 1 -- 172.21.15.90:0/2032705009 >> v1:172.21.15.90:6800/420935427 conn(0x7fe4bc071330 legacy=0x7fe4bc0737f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:02.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.998+0000 7fe4d7539700 1 -- 172.21.15.90:0/2032705009 >> v1:172.21.15.90:6789/0 conn(0x7fe4d010f340 legacy=0x7fe4d01a8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:02.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.998+0000 7fe4d7539700 1 -- 172.21.15.90:0/2032705009 shutdown_connections 2024-04-14T15:24:02.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.998+0000 7fe4d7539700 1 -- 172.21.15.90:0/2032705009 >> 172.21.15.90:0/2032705009 conn(0x7fe4d00fd290 msgr2=0x7fe4d0111770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:24:02.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.998+0000 7fe4d7539700 1 -- 172.21.15.90:0/2032705009 shutdown_connections 2024-04-14T15:24:02.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.998+0000 7fe4d7539700 1 -- 172.21.15.90:0/2032705009 wait complete. 2024-04-14T15:24:02.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:02.998+0000 7fe4d7539700 1 librados: shutdown 2024-04-14T15:24:03.002 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is set 2024-04-14T15:24:03.061 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:24:03.085 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:03.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:03.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:24:03.186 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:24:03.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:03 smithi090 ceph-mon[41670]: pgmap v1032: 182 pgs: 182 active+clean; 582 KiB data, 653 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-04-14T15:24:03.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:03 smithi090 ceph-mon[41670]: osdmap e700: 8 total, 8 up, 7 in 2024-04-14T15:24:03.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:03 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:24:03.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:03 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:24:03.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:03 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2032705009' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:24:03.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:03 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:24:03.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:03 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:24:03.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:03 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/366417754' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:24:03.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:03 smithi090 ceph-mon[30399]: pgmap v1032: 182 pgs: 182 active+clean; 582 KiB data, 653 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-04-14T15:24:03.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:03 smithi090 ceph-mon[30399]: osdmap e700: 8 total, 8 up, 7 in 2024-04-14T15:24:03.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:03 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:24:03.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:03 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:24:03.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:03 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2032705009' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:24:03.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:03 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:24:03.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:03 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:24:03.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:03 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/366417754' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:24:03.210 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:03.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:03.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:24:03.312 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:24:03.336 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:03.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:03.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:24:03.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:03 smithi092 ceph-mon[33966]: pgmap v1032: 182 pgs: 182 active+clean; 582 KiB data, 653 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-04-14T15:24:03.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:03 smithi092 ceph-mon[33966]: osdmap e700: 8 total, 8 up, 7 in 2024-04-14T15:24:03.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:03 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:24:03.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:03 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:24:03.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:03 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2032705009' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:24:03.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:03 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:24:03.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:03 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:24:03.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:03 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/366417754' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:24:03.438 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:24:03.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:03.462 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:03.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:24:03.563 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:24:03.588 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:03.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:03.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:24:03.689 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:24:03.712 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:03.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:03.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:24:03.814 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:24:03.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:03.838 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:03.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:24:03.939 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:24:03.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:03.964 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:03.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:24:04.065 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:24:04.089 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:04.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:04.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:24:04.191 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:24:04.215 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:04.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:04.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:24:04.248 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:24:03 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:24:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:24:04.249 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:04 smithi090 ceph-mon[41670]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:24:04.249 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:04 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:24:04.249 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:04 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2032705009' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-14T15:24:04.249 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:04 smithi090 ceph-mon[41670]: osdmap e701: 8 total, 8 up, 7 in 2024-04-14T15:24:04.249 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:04 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:24:04.249 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:04 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:24:04.249 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:04 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-122", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:24:04.249 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:04 smithi090 ceph-mon[41670]: osdmap e702: 8 total, 8 up, 7 in 2024-04-14T15:24:04.249 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:04 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-122"}]: dispatch 2024-04-14T15:24:04.249 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:04 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-122"}]: dispatch 2024-04-14T15:24:04.250 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:04 smithi090 ceph-mon[30399]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:24:04.250 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:04 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:24:04.250 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:04 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2032705009' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-14T15:24:04.250 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:04 smithi090 ceph-mon[30399]: osdmap e701: 8 total, 8 up, 7 in 2024-04-14T15:24:04.250 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:04 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:24:04.250 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:04 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:24:04.250 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:04 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-122", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:24:04.250 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:04 smithi090 ceph-mon[30399]: osdmap e702: 8 total, 8 up, 7 in 2024-04-14T15:24:04.250 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:04 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-122"}]: dispatch 2024-04-14T15:24:04.250 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:04 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-122"}]: dispatch 2024-04-14T15:24:04.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.248+0000 7f047afb2700 1 -- 172.21.15.90:0/2227010194 >> v1:172.21.15.90:6789/0 conn(0x7f047410a740 legacy=0x7f047410cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:04.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.248+0000 7f047afb2700 1 -- 172.21.15.90:0/2227010194 shutdown_connections 2024-04-14T15:24:04.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.248+0000 7f047afb2700 1 -- 172.21.15.90:0/2227010194 >> 172.21.15.90:0/2227010194 conn(0x7f04740fd290 msgr2=0x7f04740ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:24:04.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.248+0000 7f047afb2700 1 -- 172.21.15.90:0/2227010194 shutdown_connections 2024-04-14T15:24:04.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.248+0000 7f047afb2700 1 -- 172.21.15.90:0/2227010194 wait complete. 2024-04-14T15:24:04.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.249+0000 7f047afb2700 1 librados: starting msgr at 2024-04-14T15:24:04.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.249+0000 7f047afb2700 1 librados: starting objecter 2024-04-14T15:24:04.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.249+0000 7f047afb2700 1 Processor -- start 2024-04-14T15:24:04.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.249+0000 7f047afb2700 1 -- start start 2024-04-14T15:24:04.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.249+0000 7f047afb2700 1 librados: setting wanted keys 2024-04-14T15:24:04.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.249+0000 7f047afb2700 1 librados: calling monclient init 2024-04-14T15:24:04.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.249+0000 7f047afb2700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f047410df10 con 0x7f0474107890 2024-04-14T15:24:04.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.249+0000 7f047afb2700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f047410e100 con 0x7f047410a740 2024-04-14T15:24:04.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.249+0000 7f047afb2700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f04741a9f80 con 0x7f047410e310 2024-04-14T15:24:04.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.249+0000 7f0478d4e700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f0474107890 0x7f047406acf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:38366/0 (socket says 172.21.15.90:38366) 2024-04-14T15:24:04.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.249+0000 7f0478d4e700 1 -- 172.21.15.90:0/3988596348 learned_addr learned my addr 172.21.15.90:0/3988596348 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:24:04.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.249+0000 7f0471ffb700 10 client.?.objecter ms_handle_connect 0x7f0474107890 2024-04-14T15:24:04.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.249+0000 7f0471ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:24:04.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.249+0000 7f0471ffb700 10 client.?.objecter ms_handle_connect 0x7f047410e310 2024-04-14T15:24:04.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.249+0000 7f0471ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:24:04.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.249+0000 7f0471ffb700 10 client.?.objecter ms_handle_connect 0x7f047410a740 2024-04-14T15:24:04.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.249+0000 7f0471ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:24:04.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.249+0000 7f0471ffb700 1 -- 172.21.15.90:0/3988596348 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2223645239 0 0) 0x7f047410df10 con 0x7f0474107890 2024-04-14T15:24:04.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.249+0000 7f0471ffb700 1 -- 172.21.15.90:0/3988596348 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f04640031f0 con 0x7f0474107890 2024-04-14T15:24:04.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.249+0000 7f0471ffb700 1 -- 172.21.15.90:0/3988596348 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3803618139 0 0) 0x7f047410e100 con 0x7f047410a740 2024-04-14T15:24:04.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.249+0000 7f0471ffb700 1 -- 172.21.15.90:0/3988596348 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f047410df10 con 0x7f047410a740 2024-04-14T15:24:04.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.249+0000 7f0471ffb700 1 -- 172.21.15.90:0/3988596348 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2326995918 0 0) 0x7f04741a9f80 con 0x7f047410e310 2024-04-14T15:24:04.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.250+0000 7f0471ffb700 1 -- 172.21.15.90:0/3988596348 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f047410e100 con 0x7f047410e310 2024-04-14T15:24:04.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.250+0000 7f0471ffb700 1 -- 172.21.15.90:0/3988596348 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 638780886 0 0) 0x7f047410df10 con 0x7f047410a740 2024-04-14T15:24:04.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.250+0000 7f0471ffb700 1 -- 172.21.15.90:0/3988596348 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f04741a9f80 con 0x7f047410a740 2024-04-14T15:24:04.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.250+0000 7f0471ffb700 1 -- 172.21.15.90:0/3988596348 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f0460001370 con 0x7f047410a740 2024-04-14T15:24:04.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.250+0000 7f0471ffb700 1 -- 172.21.15.90:0/3988596348 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 783139643 0 0) 0x7f04640031f0 con 0x7f0474107890 2024-04-14T15:24:04.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.250+0000 7f0471ffb700 1 -- 172.21.15.90:0/3988596348 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f047410df10 con 0x7f0474107890 2024-04-14T15:24:04.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.250+0000 7f0471ffb700 1 -- 172.21.15.90:0/3988596348 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f04680018b0 con 0x7f0474107890 2024-04-14T15:24:04.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.250+0000 7f0471ffb700 1 -- 172.21.15.90:0/3988596348 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4221603314 0 0) 0x7f047410e100 con 0x7f047410e310 2024-04-14T15:24:04.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.250+0000 7f0471ffb700 1 -- 172.21.15.90:0/3988596348 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f04640031f0 con 0x7f047410e310 2024-04-14T15:24:04.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.250+0000 7f0471ffb700 1 -- 172.21.15.90:0/3988596348 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f046c001bf0 con 0x7f047410e310 2024-04-14T15:24:04.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.250+0000 7f0471ffb700 1 -- 172.21.15.90:0/3988596348 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2744142548 0 0) 0x7f04741a9f80 con 0x7f047410a740 2024-04-14T15:24:04.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.250+0000 7f0471ffb700 1 -- 172.21.15.90:0/3988596348 >> v1:172.21.15.90:6790/0 conn(0x7f047410e310 legacy=0x7f04741a6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:04.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.250+0000 7f0471ffb700 1 -- 172.21.15.90:0/3988596348 >> v1:172.21.15.90:6789/0 conn(0x7f0474107890 legacy=0x7f047406acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:04.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.250+0000 7f0471ffb700 1 -- 172.21.15.90:0/3988596348 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04741ab1a0 con 0x7f047410a740 2024-04-14T15:24:04.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.250+0000 7f047afb2700 1 -- 172.21.15.90:0/3988596348 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f04741aa1b0 con 0x7f047410a740 2024-04-14T15:24:04.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.250+0000 7f0471ffb700 1 -- 172.21.15.90:0/3988596348 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f0460001af0 con 0x7f047410a740 2024-04-14T15:24:04.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.250+0000 7f0471ffb700 1 -- 172.21.15.90:0/3988596348 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f0460005010 con 0x7f047410a740 2024-04-14T15:24:04.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.250+0000 7f047afb2700 10 client.49705.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:24:04.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.250+0000 7f047afb2700 1 -- 172.21.15.90:0/3988596348 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f04741aa670 con 0x7f047410a740 2024-04-14T15:24:04.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.250+0000 7f047afb2700 1 librados: init done 2024-04-14T15:24:04.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.251+0000 7f047afb2700 10 client.49705.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:24:04.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.251+0000 7f047afb2700 15 client.49705.objecter _session_command_op_assign -1 1 2024-04-14T15:24:04.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.251+0000 7f047afb2700 20 client.49705.objecter _get_session osd=-1 returning homeless 2024-04-14T15:24:04.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.251+0000 7f047afb2700 10 client.49705.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:24:04.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.251+0000 7f047afb2700 1 -- 172.21.15.90:0/3988596348 --> v1:172.21.15.92:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f04740693d0 con 0x7f047410a740 2024-04-14T15:24:04.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.252+0000 7f0471ffb700 1 -- 172.21.15.90:0/3988596348 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f0460003860 con 0x7f047410a740 2024-04-14T15:24:04.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.252+0000 7f0471ffb700 1 -- 172.21.15.90:0/3988596348 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(702..702 src has 467..702) ==== 8729+0+0 (unknown 516602937 0 0) 0x7f0460089c60 con 0x7f047410a740 2024-04-14T15:24:04.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.252+0000 7f0471ffb700 10 client.49705.objecter ms_dispatch 0x7f0474001550 osd_map(702..702 src has 467..702) 2024-04-14T15:24:04.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.252+0000 7f0471ffb700 3 client.49705.objecter handle_osd_map got epochs [702,702] > 0 2024-04-14T15:24:04.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.252+0000 7f0471ffb700 3 client.49705.objecter handle_osd_map decoding full epoch 702 2024-04-14T15:24:04.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.252+0000 7f0471ffb700 10 client.49705.objecter checking command 1 2024-04-14T15:24:04.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.252+0000 7f0471ffb700 20 client.49705.objecter _get_session s=0x7f046407ee00 osd=7 3 2024-04-14T15:24:04.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.252+0000 7f0471ffb700 20 client.49705.objecter put_session s=0x7f046407ee00 osd=7 3 2024-04-14T15:24:04.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.252+0000 7f0471ffb700 15 client.49705.objecter _session_command_op_remove -1 1 2024-04-14T15:24:04.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.252+0000 7f0471ffb700 20 client.49705.objecter _get_session s=0x7f046407ee00 osd=7 3 2024-04-14T15:24:04.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.252+0000 7f0471ffb700 20 client.49705.objecter get_session s=0x7f046407ee00 osd=7 3 2024-04-14T15:24:04.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.252+0000 7f0471ffb700 15 client.49705.objecter _session_command_op_assign 7 1 2024-04-14T15:24:04.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.252+0000 7f0471ffb700 20 client.49705.objecter put_session s=0x7f046407ee00 osd=7 4 2024-04-14T15:24:04.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.252+0000 7f0471ffb700 10 client.49705.objecter _send_command 1 2024-04-14T15:24:04.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.252+0000 7f0471ffb700 1 -- 172.21.15.90:0/3988596348 --> v1:172.21.15.92:6812/1773706716 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f04640831c0 con 0x7f046407f4b0 2024-04-14T15:24:04.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.252+0000 7f0471ffb700 20 client.49705.objecter dump_active .. 0 homeless 2024-04-14T15:24:04.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.252+0000 7f0471ffb700 1 -- 172.21.15.90:0/3988596348 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_get_version_reply(handle=1 version=702) ==== 24+0+0 (unknown 1227625783 0 0) 0x7f046008a070 con 0x7f047410a740 2024-04-14T15:24:04.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.252+0000 7f0471ffb700 10 client.49705.objecter ms_handle_connect 0x7f0464071400 2024-04-14T15:24:04.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.253+0000 7f0471ffb700 10 client.49705.objecter ms_handle_connect 0x7f046407f4b0 2024-04-14T15:24:04.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.253+0000 7f0471ffb700 1 -- 172.21.15.90:0/3988596348 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f04640831c0 con 0x7f046407f4b0 2024-04-14T15:24:04.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.253+0000 7f0471ffb700 10 client.49705.objecter ms_dispatch 0x7f0474001550 command_reply(tid 1: 0 ) 2024-04-14T15:24:04.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.253+0000 7f0471ffb700 10 client.49705.objecter _finish_command 1 = system:0 2024-04-14T15:24:04.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.253+0000 7f0471ffb700 20 client.49705.objecter put_session s=0x7f046407ee00 osd=7 4 2024-04-14T15:24:04.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:04.253+0000 7f0471ffb700 15 client.49705.objecter _session_command_op_remove 7 1 2024-04-14T15:24:04.284 INFO:teuthology.orchestra.run.smithi090.stderr:no valid command found; 10 closest matches: 2024-04-14T15:24:04.284 INFO:teuthology.orchestra.run.smithi090.stderr:0 2024-04-14T15:24:04.284 INFO:teuthology.orchestra.run.smithi090.stderr:1 2024-04-14T15:24:04.284 INFO:teuthology.orchestra.run.smithi090.stderr:2 2024-04-14T15:24:04.284 INFO:teuthology.orchestra.run.smithi090.stderr:abort 2024-04-14T15:24:04.284 INFO:teuthology.orchestra.run.smithi090.stderr:assert 2024-04-14T15:24:04.284 INFO:teuthology.orchestra.run.smithi090.stderr:bench [] [] [] [] 2024-04-14T15:24:04.284 INFO:teuthology.orchestra.run.smithi090.stderr:bluefs debug_inject_read_zeros 2024-04-14T15:24:04.284 INFO:teuthology.orchestra.run.smithi090.stderr:bluefs files list 2024-04-14T15:24:04.284 INFO:teuthology.orchestra.run.smithi090.stderr:bluefs stats 2024-04-14T15:24:04.284 INFO:teuthology.orchestra.run.smithi090.stderr:bluestore allocator dump block 2024-04-14T15:24:04.284 INFO:teuthology.orchestra.run.smithi090.stderr:Error EINVAL: invalid command 2024-04-14T15:24:04.317 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:24:04.341 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:04.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:04.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:24:04.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:04 smithi092 ceph-mon[33966]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:24:04.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:04 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:24:04.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:04 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2032705009' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-14T15:24:04.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:04 smithi092 ceph-mon[33966]: osdmap e701: 8 total, 8 up, 7 in 2024-04-14T15:24:04.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:04 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:24:04.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:04 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:24:04.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:04 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-122", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:24:04.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:04 smithi092 ceph-mon[33966]: osdmap e702: 8 total, 8 up, 7 in 2024-04-14T15:24:04.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:04 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-122"}]: dispatch 2024-04-14T15:24:04.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:04 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-122"}]: dispatch 2024-04-14T15:24:04.442 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:24:04.465 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:04.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:04.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:24:04.567 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:24:04.591 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:04.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:04.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:24:04.693 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:24:04.716 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:04.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:04.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:24:04.818 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:24:04.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:04.842 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:04.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:24:04.943 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:24:04.970 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:04.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:04.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:24:05.072 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:24:05.098 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:05.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:05.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:24:05.199 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:24:05.223 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:05.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:05.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:24:05.282 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:05 smithi090 ceph-mon[41670]: pgmap v1035: 214 pgs: 32 unknown, 182 active+clean; 582 KiB data, 653 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:24:05.282 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:05 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-122"}]': finished 2024-04-14T15:24:05.282 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:05 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-122", "mode": "writeback"}]: dispatch 2024-04-14T15:24:05.283 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:05 smithi090 ceph-mon[41670]: osdmap e703: 8 total, 8 up, 7 in 2024-04-14T15:24:05.283 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:05 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-122", "mode": "writeback"}]: dispatch 2024-04-14T15:24:05.283 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:05 smithi090 ceph-mon[30399]: pgmap v1035: 214 pgs: 32 unknown, 182 active+clean; 582 KiB data, 653 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:24:05.283 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:05 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-122"}]': finished 2024-04-14T15:24:05.283 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:05 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-122", "mode": "writeback"}]: dispatch 2024-04-14T15:24:05.283 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:05 smithi090 ceph-mon[30399]: osdmap e703: 8 total, 8 up, 7 in 2024-04-14T15:24:05.283 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:05 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-122", "mode": "writeback"}]: dispatch 2024-04-14T15:24:05.292 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:24:05.325 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:24:05.347 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:05.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:05.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:24:05.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:05 smithi092 ceph-mon[33966]: pgmap v1035: 214 pgs: 32 unknown, 182 active+clean; 582 KiB data, 653 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:24:05.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:05 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-122"}]': finished 2024-04-14T15:24:05.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:05 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-122", "mode": "writeback"}]: dispatch 2024-04-14T15:24:05.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:05 smithi092 ceph-mon[33966]: osdmap e703: 8 total, 8 up, 7 in 2024-04-14T15:24:05.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:05 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-122", "mode": "writeback"}]: dispatch 2024-04-14T15:24:05.449 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:24:05.477 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:05.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:05.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:24:05.579 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:24:05.605 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:05.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:05.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:24:05.707 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:24:05.731 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:05.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:05.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:24:05.832 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:24:05.860 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:05.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:05.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:24:05.872 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:24:05 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:24:05.906 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-14T15:24:05.906 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-14T15:24:05.962 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:24:05.989 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:05.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:05.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:24:06.062 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd unset noscrub 2024-04-14T15:24:06.091 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:24:06.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:06.130 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:06.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:24:06.182 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:06 smithi090 ceph-mon[41670]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:24:06.182 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:06 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-122", "mode": "writeback"}]': finished 2024-04-14T15:24:06.182 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:06 smithi090 ceph-mon[41670]: osdmap e704: 8 total, 8 up, 7 in 2024-04-14T15:24:06.182 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:06 smithi090 ceph-mon[30399]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:24:06.183 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:06 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-122", "mode": "writeback"}]': finished 2024-04-14T15:24:06.183 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:06 smithi090 ceph-mon[30399]: osdmap e704: 8 total, 8 up, 7 in 2024-04-14T15:24:06.230 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:24:06.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:06.257 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:06.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:24:06.358 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:24:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:06 smithi092 ceph-mon[33966]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:24:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:06 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-122", "mode": "writeback"}]': finished 2024-04-14T15:24:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:06 smithi092 ceph-mon[33966]: osdmap e704: 8 total, 8 up, 7 in 2024-04-14T15:24:06.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:06.383 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:06.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:24:06.485 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:24:06.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:06.513 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:06.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:24:06.613 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:24:06.637 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:06.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:06.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:24:06.739 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:24:06.766 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:06.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:06.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:24:06.868 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:24:06.891 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:06.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:06.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:24:06.993 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:24:07.016 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:07.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:07.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:24:07.034 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:07 smithi090 ceph-mon[41670]: pgmap v1038: 214 pgs: 214 active+clean; 582 KiB data, 672 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-14T15:24:07.034 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:07 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:24:07.034 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:07 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:24:07.034 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:07 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:24:07.034 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:07 smithi090 ceph-mon[30399]: pgmap v1038: 214 pgs: 214 active+clean; 582 KiB data, 672 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-14T15:24:07.035 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:07 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:24:07.035 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:07 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:24:07.035 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:07 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:24:07.117 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:24:07.144 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:07.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:07.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:24:07.245 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:24:07.271 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:07.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:07.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:24:07.285 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-14T15:24:07.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:07 smithi092 ceph-mon[33966]: pgmap v1038: 214 pgs: 214 active+clean; 582 KiB data, 672 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-14T15:24:07.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:07 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:24:07.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:07 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:24:07.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:07 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:24:07.373 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:24:07.399 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:07.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:07.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:24:07.501 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:24:07.524 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:07.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:07.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:24:07.626 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:24:07.652 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:07.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:07.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:24:07.754 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:24:07.777 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:07.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:07.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:24:07.875 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:24:07.879 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:24:07.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:07.907 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:07.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:24:08.008 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:24:08.034 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:08.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:08.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:24:08.137 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:24:08.162 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:08.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:08.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:24:08.264 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:24:08.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:08.289 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:08.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:24:08.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:08 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]': finished 2024-04-14T15:24:08.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:08 smithi092 ceph-mon[33966]: osdmap e705: 8 total, 8 up, 7 in 2024-04-14T15:24:08.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:08 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-122"}]: dispatch 2024-04-14T15:24:08.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:08 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-122"}]: dispatch 2024-04-14T15:24:08.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:08 smithi092 ceph-mon[33966]: pgmap v1041: 214 pgs: 214 active+clean; 582 KiB data, 672 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-14T15:24:08.389 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:24:08.415 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:08.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:08.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:24:08.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:08 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]': finished 2024-04-14T15:24:08.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:08 smithi090 ceph-mon[30399]: osdmap e705: 8 total, 8 up, 7 in 2024-04-14T15:24:08.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:08 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-122"}]: dispatch 2024-04-14T15:24:08.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:08 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-122"}]: dispatch 2024-04-14T15:24:08.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:08 smithi090 ceph-mon[30399]: pgmap v1041: 214 pgs: 214 active+clean; 582 KiB data, 672 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-14T15:24:08.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:08 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]': finished 2024-04-14T15:24:08.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:08 smithi090 ceph-mon[41670]: osdmap e705: 8 total, 8 up, 7 in 2024-04-14T15:24:08.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:08 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-122"}]: dispatch 2024-04-14T15:24:08.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:08 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-122"}]: dispatch 2024-04-14T15:24:08.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:08 smithi090 ceph-mon[41670]: pgmap v1041: 214 pgs: 214 active+clean; 582 KiB data, 672 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-14T15:24:08.517 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:24:08.543 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:08.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:08.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:24:08.645 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:24:08.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:08.669 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:08.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:24:08.770 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:24:08.797 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:08.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:08.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:24:08.899 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:24:08.925 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:08.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:08.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:24:09.026 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:24:09.051 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:09.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:09.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:24:09.153 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:24:09.175 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:09 smithi090 ceph-mon[41670]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:24:09.175 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:09 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-122"}]': finished 2024-04-14T15:24:09.175 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:09 smithi090 ceph-mon[41670]: osdmap e706: 8 total, 8 up, 7 in 2024-04-14T15:24:09.175 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:09 smithi090 ceph-mon[30399]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:24:09.175 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:09 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-122"}]': finished 2024-04-14T15:24:09.176 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:09 smithi090 ceph-mon[30399]: osdmap e706: 8 total, 8 up, 7 in 2024-04-14T15:24:09.181 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:09.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:09.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:24:09.282 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:24:09.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:09.307 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:09.307 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:24:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:09 smithi092 ceph-mon[33966]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:24:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:09 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-122"}]': finished 2024-04-14T15:24:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:09 smithi092 ceph-mon[33966]: osdmap e706: 8 total, 8 up, 7 in 2024-04-14T15:24:09.409 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:24:09.435 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:09.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:09.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:24:09.499 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:24:09.536 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:24:09.559 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:09.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:09.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:24:09.661 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:24:09.684 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:09.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:09.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:24:09.786 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:24:09.812 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:09.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:09.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:24:09.913 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:24:09.938 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:09.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:09.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:24:10.039 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:24:10.071 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:10.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:10.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:24:10.173 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:24:10.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:10 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:24:10.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:10 smithi090 ceph-mon[41670]: osdmap e707: 8 total, 8 up, 7 in 2024-04-14T15:24:10.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:10 smithi090 ceph-mon[41670]: pgmap v1044: 182 pgs: 182 active+clean; 582 KiB data, 672 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-14T15:24:10.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:10 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:24:10.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:10 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:24:10.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:10 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:24:10.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:10 smithi090 ceph-mon[30399]: osdmap e707: 8 total, 8 up, 7 in 2024-04-14T15:24:10.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:10 smithi090 ceph-mon[30399]: pgmap v1044: 182 pgs: 182 active+clean; 582 KiB data, 672 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-14T15:24:10.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:10 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:24:10.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:10 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:24:10.200 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:10.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:10.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:24:10.254 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-14T15:24:10.254 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-14T15:24:10.254 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-14T15:24:10.254 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-14T15:24:10.254 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.254 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.48511.0:1 231.6s0 231:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e222)", 2024-04-14T15:24:10.254 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:14:49.856632+0000", 2024-04-14T15:24:10.254 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 560.395687404, 2024-04-14T15:24:10.254 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.016754532, 2024-04-14T15:24:10.254 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:24:10.254 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:24:10.255 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:24:10.255 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:24:10.255 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.48511", 2024-04-14T15:24:10.255 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1251627847", 2024-04-14T15:24:10.255 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1 2024-04-14T15:24:10.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.255 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:24:10.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.255 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:24:10.255 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:49.856632+0000", 2024-04-14T15:24:10.255 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:24:10.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.255 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:24:10.256 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:49.856632+0000", 2024-04-14T15:24:10.256 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:24:10.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.256 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:24:10.256 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:49.856635+0000", 2024-04-14T15:24:10.256 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2249999999999999e-06 2024-04-14T15:24:10.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.256 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:24:10.256 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:49.856643+0000", 2024-04-14T15:24:10.256 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6789999999999994e-06 2024-04-14T15:24:10.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:24:10.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:49.856655+0000", 2024-04-14T15:24:10.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2109e-05 2024-04-14T15:24:10.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:24:10.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:49.856667+0000", 2024-04-14T15:24:10.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1644999999999999e-05 2024-04-14T15:24:10.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:24:10.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:49.856757+0000", 2024-04-14T15:24:10.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9755999999999999e-05 2024-04-14T15:24:10.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:24:10.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:49.856761+0000", 2024-04-14T15:24:10.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4839999999999998e-06 2024-04-14T15:24:10.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:24:10.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:50.861976+0000", 2024-04-14T15:24:10.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.005215129 2024-04-14T15:24:10.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:24:10.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:50.862200+0000", 2024-04-14T15:24:10.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000223375 2024-04-14T15:24:10.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.259 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2024-04-14T15:24:10.259 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:50.862806+0000", 2024-04-14T15:24:10.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060652700000000004 2024-04-14T15:24:10.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.259 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2024-04-14T15:24:10.259 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:50.863880+0000", 2024-04-14T15:24:10.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001073854 2024-04-14T15:24:10.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.259 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:24:10.259 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:50.873310+0000", 2024-04-14T15:24:10.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0094293629999999996 2024-04-14T15:24:10.260 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.260 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.260 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:24:10.260 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:14:50.873387+0000", 2024-04-14T15:24:10.260 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7385999999999994e-05 2024-04-14T15:24:10.260 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:24:10.260 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:24:10.260 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:24:10.260 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.260 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.260 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.49576.0:3 324.19 e649/648 324:9d1934b2:test-rados-api-smithi090-93652-113::foo:head v 649'1, mlcod=0'0)", 2024-04-14T15:24:10.260 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:23:05.016753+0000", 2024-04-14T15:24:10.260 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 65.235566457000004, 2024-04-14T15:24:10.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00073054000000000001, 2024-04-14T15:24:10.261 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:24:10.261 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:24:10.261 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:24:10.261 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:24:10.261 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.261 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:24:10.261 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:05.016753+0000", 2024-04-14T15:24:10.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:24:10.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.261 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.261 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:24:10.261 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:05.016753+0000", 2024-04-14T15:24:10.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:24:10.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.262 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:24:10.262 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:05.016754+0000", 2024-04-14T15:24:10.262 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9899999999999999e-07 2024-04-14T15:24:10.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.262 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:24:10.262 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:05.016761+0000", 2024-04-14T15:24:10.262 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3810000000000001e-06 2024-04-14T15:24:10.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.262 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:24:10.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:05.016769+0000", 2024-04-14T15:24:10.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7659999999999998e-06 2024-04-14T15:24:10.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:24:10.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:05.016779+0000", 2024-04-14T15:24:10.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6809999999999993e-06 2024-04-14T15:24:10.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:24:10.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:05.017094+0000", 2024-04-14T15:24:10.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031467799999999998 2024-04-14T15:24:10.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:24:10.264 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:05.017113+0000", 2024-04-14T15:24:10.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9814999999999999e-05 2024-04-14T15:24:10.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.264 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:24:10.264 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:05.017452+0000", 2024-04-14T15:24:10.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033882699999999999 2024-04-14T15:24:10.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.264 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:24:10.264 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:05.017484+0000", 2024-04-14T15:24:10.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1593000000000001e-05 2024-04-14T15:24:10.264 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:24:10.264 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:24:10.265 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:24:10.265 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.265 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.265 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.49576.0:17 324.a 324:50000000:%01:::head [pgnls start_epoch 651] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e651)", 2024-04-14T15:24:10.265 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:23:06.992978+0000", 2024-04-14T15:24:10.265 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 63.259341296000002, 2024-04-14T15:24:10.265 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0019701190000000002, 2024-04-14T15:24:10.265 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:24:10.265 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:24:10.265 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-14T15:24:10.265 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:24:10.265 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.49576", 2024-04-14T15:24:10.265 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2125385135", 2024-04-14T15:24:10.265 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 17 2024-04-14T15:24:10.265 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.266 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:24:10.266 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.266 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:24:10.266 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:06.992978+0000", 2024-04-14T15:24:10.266 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:24:10.266 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.266 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.266 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:24:10.266 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:06.992978+0000", 2024-04-14T15:24:10.266 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:24:10.266 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.266 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.266 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:24:10.266 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:06.992980+0000", 2024-04-14T15:24:10.266 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.595e-06 2024-04-14T15:24:10.267 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.267 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.267 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:24:10.267 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:06.992995+0000", 2024-04-14T15:24:10.267 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4439e-05 2024-04-14T15:24:10.267 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.267 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.267 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:24:10.267 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:06.993003+0000", 2024-04-14T15:24:10.267 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0059999999999992e-06 2024-04-14T15:24:10.267 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.267 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.267 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:24:10.267 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:06.993011+0000", 2024-04-14T15:24:10.268 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2439999999999993e-06 2024-04-14T15:24:10.268 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.268 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.268 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:24:10.268 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:06.993060+0000", 2024-04-14T15:24:10.268 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8785000000000001e-05 2024-04-14T15:24:10.269 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.269 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.269 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2024-04-14T15:24:10.269 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:06.993063+0000", 2024-04-14T15:24:10.269 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8340000000000001e-06 2024-04-14T15:24:10.269 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.269 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.269 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:24:10.269 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:06.994817+0000", 2024-04-14T15:24:10.269 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0017540500000000001 2024-04-14T15:24:10.269 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.269 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.269 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:24:10.269 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:06.994859+0000", 2024-04-14T15:24:10.269 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1764999999999997e-05 2024-04-14T15:24:10.269 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.270 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.270 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:24:10.270 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:06.994949+0000", 2024-04-14T15:24:10.270 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9400999999999993e-05 2024-04-14T15:24:10.270 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:24:10.270 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:24:10.270 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:24:10.270 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.270 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.270 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(323.0s2 665/647 ECSubWrite(tid=1243, reqid=client.49576.0:252, at_version=665'3, trim_to=0'0, roll_forward_to=664'1))", 2024-04-14T15:24:10.270 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:23:20.986331+0000", 2024-04-14T15:24:10.270 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 49.26598843, 2024-04-14T15:24:10.270 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00086042400000000004, 2024-04-14T15:24:10.270 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:24:10.271 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:24:10.271 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-04-14T15:24:10.271 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:24:10.271 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.271 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:24:10.271 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:20.986331+0000", 2024-04-14T15:24:10.271 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:24:10.271 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.271 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.271 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:24:10.271 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:20.986331+0000", 2024-04-14T15:24:10.271 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:24:10.271 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.271 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.271 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:24:10.272 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:20.986332+0000", 2024-04-14T15:24:10.272 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.96e-07 2024-04-14T15:24:10.272 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.272 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.272 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:24:10.272 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:20.986368+0000", 2024-04-14T15:24:10.272 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6421000000000002e-05 2024-04-14T15:24:10.272 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.272 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.272 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:24:10.272 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:20.986380+0000", 2024-04-14T15:24:10.272 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2328e-05 2024-04-14T15:24:10.272 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.273 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.273 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:24:10.273 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:20.986393+0000", 2024-04-14T15:24:10.273 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2041e-05 2024-04-14T15:24:10.273 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.273 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.273 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:24:10.273 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:20.986551+0000", 2024-04-14T15:24:10.273 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015855 2024-04-14T15:24:10.273 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.273 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.273 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2024-04-14T15:24:10.273 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:20.986561+0000", 2024-04-14T15:24:10.273 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0162e-05 2024-04-14T15:24:10.273 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.273 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.273 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2024-04-14T15:24:10.274 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:20.987113+0000", 2024-04-14T15:24:10.274 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00055155600000000003 2024-04-14T15:24:10.274 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.274 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.274 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:24:10.274 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:20.987192+0000", 2024-04-14T15:24:10.274 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8969999999999998e-05 2024-04-14T15:24:10.274 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:24:10.274 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:24:10.274 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:24:10.274 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.274 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.274 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(323.0s2 665/647 ECSubWrite(tid=1244, reqid=client.49576.0:252, at_version=0'0, trim_to=0'0, roll_forward_to=665'3))", 2024-04-14T15:24:10.274 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:23:20.992034+0000", 2024-04-14T15:24:10.275 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 49.260286086000001, 2024-04-14T15:24:10.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060275900000000002, 2024-04-14T15:24:10.275 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:24:10.275 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:24:10.275 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-04-14T15:24:10.275 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:24:10.275 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.275 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:24:10.275 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:20.992034+0000", 2024-04-14T15:24:10.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:24:10.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.275 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.275 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:24:10.275 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:20.992034+0000", 2024-04-14T15:24:10.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:24:10.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.276 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:24:10.276 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:20.992035+0000", 2024-04-14T15:24:10.276 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.742e-06 2024-04-14T15:24:10.276 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.276 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:24:10.276 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:20.992053+0000", 2024-04-14T15:24:10.276 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7965000000000002e-05 2024-04-14T15:24:10.276 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.276 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:24:10.276 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:20.992072+0000", 2024-04-14T15:24:10.276 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8770000000000002e-05 2024-04-14T15:24:10.276 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.276 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:24:10.277 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:20.992083+0000", 2024-04-14T15:24:10.277 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1334e-05 2024-04-14T15:24:10.277 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.277 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.277 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:24:10.277 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:20.992183+0000", 2024-04-14T15:24:10.277 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9996999999999998e-05 2024-04-14T15:24:10.277 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.277 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.277 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2024-04-14T15:24:10.277 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:20.992194+0000", 2024-04-14T15:24:10.277 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0776e-05 2024-04-14T15:24:10.277 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.277 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.277 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2024-04-14T15:24:10.277 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:20.992566+0000", 2024-04-14T15:24:10.277 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037213399999999998 2024-04-14T15:24:10.278 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.278 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.278 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:24:10.278 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:20.992636+0000", 2024-04-14T15:24:10.278 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0041000000000002e-05 2024-04-14T15:24:10.278 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:24:10.278 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:24:10.278 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:24:10.278 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.278 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.278 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.14634.0:29 3.1 3:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e607)", 2024-04-14T15:24:10.278 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:23:23.162696+0000", 2024-04-14T15:24:10.278 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 47.089623359000001, 2024-04-14T15:24:10.278 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00049649399999999999, 2024-04-14T15:24:10.278 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:24:10.279 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:24:10.279 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-04-14T15:24:10.279 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:24:10.279 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.14634", 2024-04-14T15:24:10.279 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1646083328", 2024-04-14T15:24:10.279 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 29 2024-04-14T15:24:10.279 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.279 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:24:10.279 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.279 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:24:10.279 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:23.162696+0000", 2024-04-14T15:24:10.279 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:24:10.279 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.279 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.279 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:24:10.279 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:23.162696+0000", 2024-04-14T15:24:10.279 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:24:10.280 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.280 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.280 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:24:10.280 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:23.162700+0000", 2024-04-14T15:24:10.280 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5240000000000001e-06 2024-04-14T15:24:10.280 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.280 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.280 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:24:10.280 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:23.162714+0000", 2024-04-14T15:24:10.280 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.363e-05 2024-04-14T15:24:10.280 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.280 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.280 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:24:10.280 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:23.162733+0000", 2024-04-14T15:24:10.280 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9338e-05 2024-04-14T15:24:10.280 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.280 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.281 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:24:10.281 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:23.162743+0000", 2024-04-14T15:24:10.281 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0261999999999999e-05 2024-04-14T15:24:10.281 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.281 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.281 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:24:10.281 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:23.163073+0000", 2024-04-14T15:24:10.281 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032978100000000002 2024-04-14T15:24:10.281 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.281 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.281 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:24:10.281 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:23.163193+0000", 2024-04-14T15:24:10.281 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011995899999999999 2024-04-14T15:24:10.281 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:24:10.281 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:24:10.281 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:24:10.282 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.282 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.282 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.49576.0:256 326.13 326:cb57bcf4:test-rados-api-smithi090-93652-117::bam:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e668)", 2024-04-14T15:24:10.282 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:23:24.010409+0000", 2024-04-14T15:24:10.282 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 46.241910738999998, 2024-04-14T15:24:10.282 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0082264910000000007, 2024-04-14T15:24:10.282 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:24:10.282 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:24:10.282 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-14T15:24:10.282 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:24:10.282 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.49576", 2024-04-14T15:24:10.282 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2125385135", 2024-04-14T15:24:10.282 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 256 2024-04-14T15:24:10.282 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.283 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:24:10.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.283 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:24:10.283 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:24.010409+0000", 2024-04-14T15:24:10.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:24:10.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.283 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:24:10.283 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:24.010409+0000", 2024-04-14T15:24:10.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:24:10.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.283 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:24:10.283 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:24.010413+0000", 2024-04-14T15:24:10.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.551e-06 2024-04-14T15:24:10.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.284 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:24:10.284 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:24.010425+0000", 2024-04-14T15:24:10.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2258e-05 2024-04-14T15:24:10.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.284 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:24:10.284 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:24.010443+0000", 2024-04-14T15:24:10.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7734999999999999e-05 2024-04-14T15:24:10.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.284 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:24:10.284 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:24.010453+0000", 2024-04-14T15:24:10.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0448000000000001e-05 2024-04-14T15:24:10.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:24:10.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:24.010515+0000", 2024-04-14T15:24:10.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1741000000000004e-05 2024-04-14T15:24:10.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2024-04-14T15:24:10.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:24.010519+0000", 2024-04-14T15:24:10.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.578e-06 2024-04-14T15:24:10.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:24:10.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:24.012961+0000", 2024-04-14T15:24:10.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0024416099999999999 2024-04-14T15:24:10.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-14T15:24:10.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:24.013273+0000", 2024-04-14T15:24:10.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031194099999999999 2024-04-14T15:24:10.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.286 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:24:10.286 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:24.015629+0000", 2024-04-14T15:24:10.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023557719999999999 2024-04-14T15:24:10.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.286 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-14T15:24:10.286 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:24.015685+0000", 2024-04-14T15:24:10.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6339999999999999e-05 2024-04-14T15:24:10.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.286 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:24:10.286 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:24.016165+0000", 2024-04-14T15:24:10.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0004795 2024-04-14T15:24:10.287 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.287 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.287 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:24:10.287 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:24.016258+0000", 2024-04-14T15:24:10.287 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3095000000000002e-05 2024-04-14T15:24:10.287 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.287 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.287 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:24:10.287 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:24.018636+0000", 2024-04-14T15:24:10.287 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023779220000000002 2024-04-14T15:24:10.287 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:24:10.287 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:24:10.287 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:24:10.287 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.287 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.287 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24373.0:3044 5.3 5:c609908c:::notify.5:head [watch ping cookie 94738481570944 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e676)", 2024-04-14T15:24:10.288 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:23:38.227200+0000", 2024-04-14T15:24:10.288 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 32.025119920999998, 2024-04-14T15:24:10.288 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00056487300000000002, 2024-04-14T15:24:10.288 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:24:10.288 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:24:10.288 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-14T15:24:10.288 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:24:10.288 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24373", 2024-04-14T15:24:10.288 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2924083629", 2024-04-14T15:24:10.288 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3044 2024-04-14T15:24:10.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.288 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:24:10.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.288 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:24:10.288 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:38.227200+0000", 2024-04-14T15:24:10.288 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:24:10.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:24:10.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:38.227200+0000", 2024-04-14T15:24:10.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:24:10.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:24:10.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:38.227204+0000", 2024-04-14T15:24:10.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3030000000000001e-06 2024-04-14T15:24:10.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:24:10.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:38.227217+0000", 2024-04-14T15:24:10.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2666e-05 2024-04-14T15:24:10.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.290 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:24:10.290 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:38.227246+0000", 2024-04-14T15:24:10.290 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8921999999999999e-05 2024-04-14T15:24:10.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.290 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:24:10.290 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:38.227257+0000", 2024-04-14T15:24:10.290 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1599e-05 2024-04-14T15:24:10.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.290 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:24:10.290 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:38.227481+0000", 2024-04-14T15:24:10.290 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022409900000000001 2024-04-14T15:24:10.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.290 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:24:10.291 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:38.227559+0000", 2024-04-14T15:24:10.291 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7245999999999999e-05 2024-04-14T15:24:10.291 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.291 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:24:10.291 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:38.227765+0000", 2024-04-14T15:24:10.291 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020603799999999999 2024-04-14T15:24:10.291 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:24:10.291 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:24:10.291 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:24:10.291 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.291 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(323.0s2 678/647 ECSubWrite(tid=1247, reqid=osd.1.0:1247, at_version=678'5, trim_to=0'0, roll_forward_to=665'3))", 2024-04-14T15:24:10.291 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:23:40.014641+0000", 2024-04-14T15:24:10.291 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 30.237678729999999, 2024-04-14T15:24:10.291 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00067577999999999996, 2024-04-14T15:24:10.292 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:24:10.292 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:24:10.292 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-04-14T15:24:10.292 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:24:10.292 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.292 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:24:10.292 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:40.014641+0000", 2024-04-14T15:24:10.292 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:24:10.292 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.292 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.292 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:24:10.292 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:40.014641+0000", 2024-04-14T15:24:10.292 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:24:10.292 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.292 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.292 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:24:10.293 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:40.014641+0000", 2024-04-14T15:24:10.293 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1e-07 2024-04-14T15:24:10.293 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.293 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.293 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:24:10.293 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:40.014669+0000", 2024-04-14T15:24:10.293 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8129000000000001e-05 2024-04-14T15:24:10.293 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.293 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.293 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:24:10.293 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:40.014687+0000", 2024-04-14T15:24:10.293 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7492000000000001e-05 2024-04-14T15:24:10.293 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.293 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.293 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:24:10.293 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:40.014700+0000", 2024-04-14T15:24:10.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3113e-05 2024-04-14T15:24:10.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.294 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.294 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:24:10.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:40.014785+0000", 2024-04-14T15:24:10.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4670000000000001e-05 2024-04-14T15:24:10.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.294 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.294 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2024-04-14T15:24:10.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:40.014796+0000", 2024-04-14T15:24:10.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1087000000000001e-05 2024-04-14T15:24:10.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.294 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.294 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2024-04-14T15:24:10.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:40.015282+0000", 2024-04-14T15:24:10.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00048625200000000001 2024-04-14T15:24:10.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.295 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.295 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:24:10.295 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:40.015317+0000", 2024-04-14T15:24:10.295 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4727000000000001e-05 2024-04-14T15:24:10.295 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:24:10.295 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:24:10.295 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:24:10.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.295 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.295 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(323.0s2 678/647 ECSubWrite(tid=1248, reqid=osd.1.0:1247, at_version=0'0, trim_to=0'0, roll_forward_to=678'5))", 2024-04-14T15:24:10.295 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:23:40.016040+0000", 2024-04-14T15:24:10.295 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 30.236280139000002, 2024-04-14T15:24:10.295 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001820267, 2024-04-14T15:24:10.295 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:24:10.295 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:24:10.295 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-04-14T15:24:10.296 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:24:10.296 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.296 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:24:10.296 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:40.016040+0000", 2024-04-14T15:24:10.296 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:24:10.296 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.296 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.296 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:24:10.296 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:40.016040+0000", 2024-04-14T15:24:10.296 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:24:10.296 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.296 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.296 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:24:10.296 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:40.016041+0000", 2024-04-14T15:24:10.296 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.42e-06 2024-04-14T15:24:10.296 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.297 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.297 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:24:10.297 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:40.016053+0000", 2024-04-14T15:24:10.297 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1758e-05 2024-04-14T15:24:10.297 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.297 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.297 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:24:10.297 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:40.016064+0000", 2024-04-14T15:24:10.297 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0995000000000001e-05 2024-04-14T15:24:10.297 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.297 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.297 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:24:10.297 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:40.016072+0000", 2024-04-14T15:24:10.297 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5369999999999997e-06 2024-04-14T15:24:10.297 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.297 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.298 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:24:10.298 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:40.016151+0000", 2024-04-14T15:24:10.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8720000000000005e-05 2024-04-14T15:24:10.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.298 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.298 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2024-04-14T15:24:10.298 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:40.016160+0000", 2024-04-14T15:24:10.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7509999999999997e-06 2024-04-14T15:24:10.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.298 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.298 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2024-04-14T15:24:10.298 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:40.017831+0000", 2024-04-14T15:24:10.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001670957 2024-04-14T15:24:10.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.298 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.298 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:24:10.299 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:40.017860+0000", 2024-04-14T15:24:10.299 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9128999999999998e-05 2024-04-14T15:24:10.299 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:24:10.299 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:24:10.299 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:24:10.299 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.299 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.299 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(osd.7.0:1100 326.8 e678/663 326:157db765:test-rados-api-smithi090-93652-117::bar:2 v 678'4, mlcod=677'1)", 2024-04-14T15:24:10.299 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:23:40.022258+0000", 2024-04-14T15:24:10.299 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 30.230062196999999, 2024-04-14T15:24:10.299 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00083041699999999996, 2024-04-14T15:24:10.299 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:24:10.299 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:24:10.299 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:24:10.299 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:24:10.299 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.300 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:24:10.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:40.022258+0000", 2024-04-14T15:24:10.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:24:10.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.300 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.300 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:24:10.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:40.022258+0000", 2024-04-14T15:24:10.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:24:10.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.300 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.300 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:24:10.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:40.022258+0000", 2024-04-14T15:24:10.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3499999999999996e-07 2024-04-14T15:24:10.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.300 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.300 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:24:10.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:40.022265+0000", 2024-04-14T15:24:10.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1720000000000001e-06 2024-04-14T15:24:10.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.301 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.301 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:24:10.301 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:40.022272+0000", 2024-04-14T15:24:10.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.421e-06 2024-04-14T15:24:10.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.301 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.301 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:24:10.301 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:40.022279+0000", 2024-04-14T15:24:10.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4120000000000004e-06 2024-04-14T15:24:10.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.301 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.301 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:24:10.301 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:40.022323+0000", 2024-04-14T15:24:10.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3535999999999998e-05 2024-04-14T15:24:10.302 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.302 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:24:10.302 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:40.022343+0000", 2024-04-14T15:24:10.302 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0139000000000001e-05 2024-04-14T15:24:10.302 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.302 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:24:10.302 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:40.023024+0000", 2024-04-14T15:24:10.302 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000680878 2024-04-14T15:24:10.302 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.302 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:24:10.302 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:40.023088+0000", 2024-04-14T15:24:10.302 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4324000000000003e-05 2024-04-14T15:24:10.302 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:24:10.303 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:24:10.303 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:24:10.303 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.303 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.303 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.49576.0:301 326.13 326:cb57bcf4:test-rados-api-smithi090-93652-117::bam:head [cache-evict] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e678)", 2024-04-14T15:24:10.303 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:23:40.038352+0000", 2024-04-14T15:24:10.303 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 30.213967958000001, 2024-04-14T15:24:10.303 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00095479500000000004, 2024-04-14T15:24:10.303 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:24:10.303 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:24:10.303 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:24:10.303 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:24:10.303 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.49576", 2024-04-14T15:24:10.303 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2125385135", 2024-04-14T15:24:10.303 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 301 2024-04-14T15:24:10.303 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.303 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:24:10.304 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.304 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:24:10.304 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:40.038352+0000", 2024-04-14T15:24:10.304 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:24:10.304 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.304 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.304 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:24:10.304 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:40.038352+0000", 2024-04-14T15:24:10.304 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:24:10.304 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.304 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.304 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:24:10.304 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:40.038353+0000", 2024-04-14T15:24:10.304 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.083e-06 2024-04-14T15:24:10.304 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.304 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.305 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:24:10.305 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:40.038357+0000", 2024-04-14T15:24:10.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1749999999999998e-06 2024-04-14T15:24:10.305 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.305 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:24:10.305 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:40.038367+0000", 2024-04-14T15:24:10.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0216e-05 2024-04-14T15:24:10.305 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.305 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:24:10.305 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:40.038374+0000", 2024-04-14T15:24:10.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0639999999999996e-06 2024-04-14T15:24:10.305 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.305 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:24:10.305 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:40.038416+0000", 2024-04-14T15:24:10.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1730000000000002e-05 2024-04-14T15:24:10.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:24:10.306 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:40.038472+0000", 2024-04-14T15:24:10.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6360000000000002e-05 2024-04-14T15:24:10.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,4", 2024-04-14T15:24:10.306 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:40.038570+0000", 2024-04-14T15:24:10.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7643e-05 2024-04-14T15:24:10.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:24:10.306 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:40.038825+0000", 2024-04-14T15:24:10.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025445099999999999 2024-04-14T15:24:10.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:24:10.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:40.039114+0000", 2024-04-14T15:24:10.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028909999999999998 2024-04-14T15:24:10.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:24:10.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:40.039250+0000", 2024-04-14T15:24:10.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000136534 2024-04-14T15:24:10.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:24:10.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:40.039280+0000", 2024-04-14T15:24:10.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9561999999999999e-05 2024-04-14T15:24:10.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.308 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:24:10.308 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:40.039307+0000", 2024-04-14T15:24:10.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6877e-05 2024-04-14T15:24:10.308 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:24:10.308 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:24:10.308 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:24:10.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.308 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.49576.0:312 326.a 326:50000000::::head [pgnls start_epoch 680] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e680)", 2024-04-14T15:24:10.308 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:23:42.036315+0000", 2024-04-14T15:24:10.308 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 28.216004578, 2024-04-14T15:24:10.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0021978369999999998, 2024-04-14T15:24:10.308 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:24:10.308 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:24:10.308 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-14T15:24:10.309 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:24:10.309 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.49576", 2024-04-14T15:24:10.309 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2125385135", 2024-04-14T15:24:10.309 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 312 2024-04-14T15:24:10.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.309 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:24:10.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.309 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:24:10.309 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:42.036315+0000", 2024-04-14T15:24:10.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:24:10.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.309 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:24:10.309 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:42.036315+0000", 2024-04-14T15:24:10.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:24:10.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.310 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:24:10.310 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:42.036318+0000", 2024-04-14T15:24:10.310 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.027e-06 2024-04-14T15:24:10.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.310 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.310 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:24:10.310 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:42.036328+0000", 2024-04-14T15:24:10.310 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0189e-05 2024-04-14T15:24:10.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.310 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.310 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:24:10.310 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:42.036349+0000", 2024-04-14T15:24:10.310 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0625999999999999e-05 2024-04-14T15:24:10.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.310 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:24:10.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:42.036360+0000", 2024-04-14T15:24:10.311 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1274000000000001e-05 2024-04-14T15:24:10.311 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:24:10.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:42.036419+0000", 2024-04-14T15:24:10.311 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8783000000000003e-05 2024-04-14T15:24:10.311 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2024-04-14T15:24:10.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:42.036423+0000", 2024-04-14T15:24:10.311 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2590000000000002e-06 2024-04-14T15:24:10.311 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:24:10.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:42.038342+0000", 2024-04-14T15:24:10.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0019187550000000001 2024-04-14T15:24:10.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:24:10.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:42.038391+0000", 2024-04-14T15:24:10.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8896000000000001e-05 2024-04-14T15:24:10.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:24:10.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:42.038513+0000", 2024-04-14T15:24:10.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000122028 2024-04-14T15:24:10.312 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:24:10.312 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:24:10.312 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:24:10.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.313 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.313 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.49576.0:368 323.4s0 323:37a9f7e3:test-rados-api-smithi090-93652-119::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e683)", 2024-04-14T15:24:10.313 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:23:45.035373+0000", 2024-04-14T15:24:10.313 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 25.216946478000001, 2024-04-14T15:24:10.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.021610085000000001, 2024-04-14T15:24:10.313 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:24:10.313 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:24:10.313 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:24:10.313 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:24:10.313 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.49576", 2024-04-14T15:24:10.313 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2125385135", 2024-04-14T15:24:10.313 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 368 2024-04-14T15:24:10.313 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.313 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:24:10.313 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.313 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:24:10.313 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:45.035373+0000", 2024-04-14T15:24:10.314 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:24:10.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.314 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:24:10.314 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:45.035373+0000", 2024-04-14T15:24:10.314 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:24:10.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.314 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:24:10.314 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:45.035376+0000", 2024-04-14T15:24:10.314 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3870000000000002e-06 2024-04-14T15:24:10.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.314 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:24:10.314 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:45.035385+0000", 2024-04-14T15:24:10.314 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5219999999999996e-06 2024-04-14T15:24:10.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:24:10.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:45.035405+0000", 2024-04-14T15:24:10.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0136000000000002e-05 2024-04-14T15:24:10.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:24:10.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:45.035417+0000", 2024-04-14T15:24:10.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1364e-05 2024-04-14T15:24:10.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:24:10.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:45.036057+0000", 2024-04-14T15:24:10.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064019700000000001 2024-04-14T15:24:10.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.316 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2024-04-14T15:24:10.316 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:45.036065+0000", 2024-04-14T15:24:10.316 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7090000000000002e-06 2024-04-14T15:24:10.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.316 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:24:10.316 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:45.055149+0000", 2024-04-14T15:24:10.316 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019084719999999999 2024-04-14T15:24:10.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.316 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:24:10.316 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:45.055296+0000", 2024-04-14T15:24:10.316 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014702899999999999 2024-04-14T15:24:10.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.317 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:24:10.318 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2024-04-14T15:24:10.318 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:45.055793+0000", 2024-04-14T15:24:10.318 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00049636000000000005 2024-04-14T15:24:10.318 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.318 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.318 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2024-04-14T15:24:10.318 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:45.056330+0000", 2024-04-14T15:24:10.318 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000537603 2024-04-14T15:24:10.318 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.318 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.318 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:24:10.319 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:45.056913+0000", 2024-04-14T15:24:10.319 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00058263300000000004 2024-04-14T15:24:10.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.319 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:24:10.319 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:45.056983+0000", 2024-04-14T15:24:10.319 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0425000000000004e-05 2024-04-14T15:24:10.319 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:24:10.319 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:24:10.319 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:24:10.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.319 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.319 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.49576.0:370 323.4s0 323:37a9f7e3:test-rados-api-smithi090-93652-119::foo:head [writefull 0~5] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e684)", 2024-04-14T15:24:10.319 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:23:45.988238+0000", 2024-04-14T15:24:10.319 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 24.264081286, 2024-04-14T15:24:10.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003769475, 2024-04-14T15:24:10.320 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:24:10.320 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:24:10.320 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:24:10.320 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:24:10.320 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.49576", 2024-04-14T15:24:10.320 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2125385135", 2024-04-14T15:24:10.320 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 370 2024-04-14T15:24:10.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.320 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:24:10.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.320 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:24:10.320 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:45.988238+0000", 2024-04-14T15:24:10.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:24:10.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.321 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:24:10.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:45.988238+0000", 2024-04-14T15:24:10.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:24:10.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.321 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:24:10.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:45.988241+0000", 2024-04-14T15:24:10.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9859999999999999e-06 2024-04-14T15:24:10.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.321 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:24:10.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:45.988253+0000", 2024-04-14T15:24:10.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1664e-05 2024-04-14T15:24:10.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.322 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.322 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:24:10.322 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:45.988267+0000", 2024-04-14T15:24:10.322 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3720000000000001e-05 2024-04-14T15:24:10.322 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.322 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.322 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:24:10.322 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:45.988279+0000", 2024-04-14T15:24:10.322 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1800999999999999e-05 2024-04-14T15:24:10.322 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.322 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.322 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:24:10.322 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:45.988337+0000", 2024-04-14T15:24:10.323 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8847000000000001e-05 2024-04-14T15:24:10.323 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.323 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:24:10.323 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:45.988447+0000", 2024-04-14T15:24:10.323 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000109924 2024-04-14T15:24:10.323 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.323 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2024-04-14T15:24:10.323 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:45.989131+0000", 2024-04-14T15:24:10.323 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00068343699999999996 2024-04-14T15:24:10.323 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.323 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2024-04-14T15:24:10.323 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:45.990422+0000", 2024-04-14T15:24:10.323 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001290774 2024-04-14T15:24:10.324 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.324 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.324 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:24:10.324 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:45.991913+0000", 2024-04-14T15:24:10.324 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0014917229999999999 2024-04-14T15:24:10.324 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.324 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.324 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:24:10.324 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:45.992008+0000", 2024-04-14T15:24:10.324 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4598999999999993e-05 2024-04-14T15:24:10.324 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:24:10.324 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:24:10.324 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:24:10.324 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.324 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.325 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWriteReply(323.4s0 683/647 ECSubWriteReply(tid=1662, last_complete=684'3, committed=1, applied=1))", 2024-04-14T15:24:10.325 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:23:45.991741+0000", 2024-04-14T15:24:10.325 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 24.260578398, 2024-04-14T15:24:10.325 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00051570800000000003, 2024-04-14T15:24:10.325 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:24:10.325 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:24:10.325 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-04-14T15:24:10.325 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:24:10.325 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.325 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:24:10.325 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:45.991741+0000", 2024-04-14T15:24:10.325 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:24:10.325 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.325 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.325 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:24:10.325 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:45.991741+0000", 2024-04-14T15:24:10.326 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:24:10.326 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.326 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.326 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:24:10.326 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:45.991743+0000", 2024-04-14T15:24:10.326 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0760000000000001e-06 2024-04-14T15:24:10.326 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.326 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.326 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:24:10.326 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:45.991752+0000", 2024-04-14T15:24:10.326 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2029999999999992e-06 2024-04-14T15:24:10.326 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.326 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.326 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:24:10.327 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:45.991766+0000", 2024-04-14T15:24:10.327 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.437e-05 2024-04-14T15:24:10.327 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.327 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:24:10.327 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:45.991776+0000", 2024-04-14T15:24:10.327 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0190000000000001e-05 2024-04-14T15:24:10.327 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.327 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:24:10.327 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:45.991836+0000", 2024-04-14T15:24:10.327 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9932999999999997e-05 2024-04-14T15:24:10.327 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.327 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:24:10.327 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:45.992257+0000", 2024-04-14T15:24:10.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00042093600000000001 2024-04-14T15:24:10.328 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:24:10.328 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:24:10.328 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:24:10.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.328 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(osd.3.0:798 2.19 e684/527 2:9971178b:::zone_info.5575f7e5-79b5-44de-b9b5-51360b975a8b:head v 684'3, mlcod=0'0)", 2024-04-14T15:24:10.328 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:23:46.930548+0000", 2024-04-14T15:24:10.328 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 23.32177197, 2024-04-14T15:24:10.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00050247099999999999, 2024-04-14T15:24:10.328 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:24:10.328 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:24:10.328 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:24:10.328 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:24:10.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.328 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:24:10.328 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:46.930548+0000", 2024-04-14T15:24:10.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:24:10.329 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.329 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:24:10.329 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:46.930548+0000", 2024-04-14T15:24:10.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:24:10.329 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.329 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:24:10.329 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:46.930549+0000", 2024-04-14T15:24:10.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1200000000000001e-06 2024-04-14T15:24:10.329 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.329 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:24:10.330 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:46.930556+0000", 2024-04-14T15:24:10.330 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8499999999999996e-06 2024-04-14T15:24:10.330 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.330 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:24:10.330 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:46.930565+0000", 2024-04-14T15:24:10.330 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5300000000000002e-06 2024-04-14T15:24:10.330 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.330 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:24:10.330 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:46.930573+0000", 2024-04-14T15:24:10.330 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2620000000000001e-06 2024-04-14T15:24:10.330 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.331 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:24:10.331 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:46.930625+0000", 2024-04-14T15:24:10.331 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2055000000000002e-05 2024-04-14T15:24:10.331 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.331 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.331 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:24:10.331 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:46.930645+0000", 2024-04-14T15:24:10.331 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0392999999999999e-05 2024-04-14T15:24:10.331 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.331 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.331 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:24:10.331 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:46.931020+0000", 2024-04-14T15:24:10.331 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037533199999999998 2024-04-14T15:24:10.332 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.332 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.332 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:24:10.332 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:46.931050+0000", 2024-04-14T15:24:10.332 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9929000000000001e-05 2024-04-14T15:24:10.332 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:24:10.332 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:24:10.332 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:24:10.332 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.332 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.332 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24373.0:3095 5.3 5:c609908c:::notify.5:head [watch ping cookie 94738481570944 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e697)", 2024-04-14T15:24:10.332 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:24:03.232237+0000", 2024-04-14T15:24:10.332 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 7.0200827629999996, 2024-04-14T15:24:10.332 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00052712199999999999, 2024-04-14T15:24:10.332 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:24:10.333 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:24:10.333 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-14T15:24:10.333 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:24:10.333 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24373", 2024-04-14T15:24:10.333 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2924083629", 2024-04-14T15:24:10.333 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3095 2024-04-14T15:24:10.333 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.333 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:24:10.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:24:10.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:03.232237+0000", 2024-04-14T15:24:10.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:24:10.333 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.334 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:24:10.334 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:03.232237+0000", 2024-04-14T15:24:10.334 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:24:10.334 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.334 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.334 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:24:10.334 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:03.232240+0000", 2024-04-14T15:24:10.334 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2490000000000001e-06 2024-04-14T15:24:10.334 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.334 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.334 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:24:10.334 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:03.232249+0000", 2024-04-14T15:24:10.334 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5299999999999996e-06 2024-04-14T15:24:10.334 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.335 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.335 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:24:10.335 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:03.232271+0000", 2024-04-14T15:24:10.335 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1985e-05 2024-04-14T15:24:10.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.335 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.335 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:24:10.335 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:03.232281+0000", 2024-04-14T15:24:10.335 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0267e-05 2024-04-14T15:24:10.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.335 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.335 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:24:10.335 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:03.232583+0000", 2024-04-14T15:24:10.335 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000301575 2024-04-14T15:24:10.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.335 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.336 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:24:10.336 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:03.232674+0000", 2024-04-14T15:24:10.336 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1856999999999998e-05 2024-04-14T15:24:10.336 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.336 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.336 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:24:10.336 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:03.232764+0000", 2024-04-14T15:24:10.336 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9659000000000003e-05 2024-04-14T15:24:10.336 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:24:10.336 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:24:10.336 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:24:10.336 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.336 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.336 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.49576.0:651 329.0 329:00000000:test-rados-api-smithi090-93652-123:::head [pgnls start_epoch 704] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e704)", 2024-04-14T15:24:10.336 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:24:06.013824+0000", 2024-04-14T15:24:10.336 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.2384954209999997, 2024-04-14T15:24:10.336 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026777469999999999, 2024-04-14T15:24:10.337 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:24:10.337 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:24:10.337 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-14T15:24:10.337 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:24:10.337 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.49576", 2024-04-14T15:24:10.337 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2125385135", 2024-04-14T15:24:10.337 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 651 2024-04-14T15:24:10.337 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.337 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:24:10.337 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.337 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:24:10.337 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:06.013824+0000", 2024-04-14T15:24:10.337 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:24:10.337 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.337 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.337 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:24:10.338 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:06.013824+0000", 2024-04-14T15:24:10.338 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:24:10.338 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.338 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.338 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:24:10.338 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:06.013827+0000", 2024-04-14T15:24:10.338 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2869999999999999e-06 2024-04-14T15:24:10.338 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.338 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.338 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:24:10.338 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:06.013834+0000", 2024-04-14T15:24:10.338 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8650000000000001e-06 2024-04-14T15:24:10.338 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.338 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.338 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:24:10.339 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:06.013849+0000", 2024-04-14T15:24:10.339 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4591e-05 2024-04-14T15:24:10.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.339 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.339 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:24:10.339 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:06.013857+0000", 2024-04-14T15:24:10.339 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8280000000000003e-06 2024-04-14T15:24:10.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.339 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.339 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:24:10.339 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:06.013908+0000", 2024-04-14T15:24:10.339 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1561999999999999e-05 2024-04-14T15:24:10.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.339 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.339 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2024-04-14T15:24:10.339 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:06.013912+0000", 2024-04-14T15:24:10.340 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9790000000000004e-06 2024-04-14T15:24:10.340 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.340 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.340 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:24:10.340 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:06.016225+0000", 2024-04-14T15:24:10.340 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002312414 2024-04-14T15:24:10.340 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.340 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.340 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:24:10.340 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:06.016256+0000", 2024-04-14T15:24:10.340 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1353999999999998e-05 2024-04-14T15:24:10.340 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.340 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.340 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:24:10.340 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:06.016502+0000", 2024-04-14T15:24:10.340 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024586699999999998 2024-04-14T15:24:10.340 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:24:10.341 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:24:10.341 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:24:10.341 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.341 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.341 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.49576.0:725 329.0 329:00000000::::head [pgnls start_epoch 706] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e706)", 2024-04-14T15:24:10.341 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:24:08.050577+0000", 2024-04-14T15:24:10.341 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 2.2017426709999999, 2024-04-14T15:24:10.341 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026819220000000002, 2024-04-14T15:24:10.341 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:24:10.341 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:24:10.341 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-14T15:24:10.341 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:24:10.341 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.49576", 2024-04-14T15:24:10.341 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2125385135", 2024-04-14T15:24:10.341 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 725 2024-04-14T15:24:10.342 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.342 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:24:10.342 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.342 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:24:10.342 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:08.050577+0000", 2024-04-14T15:24:10.342 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:24:10.342 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.342 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.342 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:24:10.342 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:08.050577+0000", 2024-04-14T15:24:10.342 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:24:10.342 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.342 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.342 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:24:10.342 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:08.050580+0000", 2024-04-14T15:24:10.343 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0089999999999999e-06 2024-04-14T15:24:10.343 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.343 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.343 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:24:10.343 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:08.050593+0000", 2024-04-14T15:24:10.343 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2638e-05 2024-04-14T15:24:10.343 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.343 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.343 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:24:10.343 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:08.050612+0000", 2024-04-14T15:24:10.343 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9514e-05 2024-04-14T15:24:10.343 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.343 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.343 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:24:10.343 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:08.050623+0000", 2024-04-14T15:24:10.343 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0376e-05 2024-04-14T15:24:10.344 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.344 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.344 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:24:10.344 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:08.050698+0000", 2024-04-14T15:24:10.344 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5797999999999995e-05 2024-04-14T15:24:10.344 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.344 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.344 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2024-04-14T15:24:10.344 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:08.050705+0000", 2024-04-14T15:24:10.344 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2620000000000004e-06 2024-04-14T15:24:10.344 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.344 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.344 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:24:10.344 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:08.052974+0000", 2024-04-14T15:24:10.344 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022690760000000001 2024-04-14T15:24:10.344 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.344 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.345 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:24:10.345 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:08.053018+0000", 2024-04-14T15:24:10.345 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4328e-05 2024-04-14T15:24:10.345 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:24:10.345 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:24:10.345 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:24:10.345 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:08.053259+0000", 2024-04-14T15:24:10.345 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024092100000000001 2024-04-14T15:24:10.345 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:24:10.345 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:24:10.345 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:24:10.345 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:24:10.345 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:24:10.345 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-14T15:24:10.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:10.369 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:10.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:24:10.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:10 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:24:10.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:10 smithi092 ceph-mon[33966]: osdmap e707: 8 total, 8 up, 7 in 2024-04-14T15:24:10.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:10 smithi092 ceph-mon[33966]: pgmap v1044: 182 pgs: 182 active+clean; 582 KiB data, 672 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-14T15:24:10.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:10 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:24:10.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:10 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:24:10.470 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:24:10.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:10.494 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:10.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:24:10.595 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:24:10.622 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:10.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:10.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:24:10.724 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:24:10.750 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:10.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:10.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:24:10.852 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:24:10.875 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:10.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:10.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:24:10.907 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-14T15:24:10.907 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-14T15:24:10.908 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-04-14T15:24:10.908 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-04-14T15:24:10.908 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph pg dump --format=json 2024-04-14T15:24:10.977 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:24:11.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:11.001 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:11.001 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:24:11.071 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:11 smithi090 ceph-mon[30399]: osdmap e708: 8 total, 8 up, 7 in 2024-04-14T15:24:11.071 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:11 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:24:11.071 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:11 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:24:11.102 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:24:11.129 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:11.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:11.130 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:24:11.231 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:24:11.254 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:11.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:11.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:24:11.356 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:24:11.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:11 smithi092 ceph-mon[33966]: osdmap e708: 8 total, 8 up, 7 in 2024-04-14T15:24:11.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:11 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:24:11.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:11 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:24:11.383 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:11.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:11.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:24:11.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:11 smithi090 ceph-mon[41670]: osdmap e708: 8 total, 8 up, 7 in 2024-04-14T15:24:11.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:11 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:24:11.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:11 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:24:11.485 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:24:11.511 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:11.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:11.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:24:11.612 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:24:11.636 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:11.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:11.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:24:11.738 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:24:11.765 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:11.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:11.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:24:11.866 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:24:11.890 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:11.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:11.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:24:11.992 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:24:12.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:12.017 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:12.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:24:12.117 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:24:12.140 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:12.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:12.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:24:12.241 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-14T15:24:12.242 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:24:12.266 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:12.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:12.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:24:12.368 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:24:12.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:12 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:24:12.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:12 smithi092 ceph-mon[33966]: osdmap e709: 8 total, 8 up, 7 in 2024-04-14T15:24:12.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:12 smithi092 ceph-mon[33966]: pgmap v1047: 214 pgs: 32 unknown, 182 active+clean; 582 KiB data, 672 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-14T15:24:12.377 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:12 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:24:12.377 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:12 smithi090 ceph-mon[30399]: osdmap e709: 8 total, 8 up, 7 in 2024-04-14T15:24:12.377 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:12 smithi090 ceph-mon[30399]: pgmap v1047: 214 pgs: 32 unknown, 182 active+clean; 582 KiB data, 672 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-14T15:24:12.378 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:12 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:24:12.378 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:12 smithi090 ceph-mon[41670]: osdmap e709: 8 total, 8 up, 7 in 2024-04-14T15:24:12.378 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:12 smithi090 ceph-mon[41670]: pgmap v1047: 214 pgs: 32 unknown, 182 active+clean; 582 KiB data, 672 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-14T15:24:12.392 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:12.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:12.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:24:12.493 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:24:12.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:12.522 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:12.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:24:12.623 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:24:12.651 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:12.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:12.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:24:12.752 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:24:12.775 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:24:12.779 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:12.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:12.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:24:12.881 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:24:12.907 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:12.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:12.909 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:24:13.009 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:24:13.032 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:13.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:13.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:24:13.134 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:24:13.158 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:13.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:13.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:24:13.260 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:24:13.283 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:13.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:13.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:24:13.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:13 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:24:13.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:13 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:24:13.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:13 smithi092 ceph-mon[33966]: osdmap e710: 8 total, 8 up, 7 in 2024-04-14T15:24:13.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:13 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:24:13.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:13 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:24:13.385 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:24:13.413 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:13.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:13.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:24:13.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:13 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:24:13.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:13 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:24:13.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:13 smithi090 ceph-mon[41670]: osdmap e710: 8 total, 8 up, 7 in 2024-04-14T15:24:13.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:13 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:24:13.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:13 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:24:13.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:13 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:24:13.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:13 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:24:13.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:13 smithi090 ceph-mon[30399]: osdmap e710: 8 total, 8 up, 7 in 2024-04-14T15:24:13.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:13 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:24:13.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:13 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:24:13.515 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:24:13.539 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:13.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:13.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:24:13.640 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:24:13.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:13.668 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:13.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:24:13.769 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:24:13.795 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:13.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:13.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:24:13.897 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:24:13.924 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:13.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:13.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:24:14.026 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:24:14.050 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:14.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:14.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:24:14.127 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:24:13 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:24:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:24:14.127 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:14 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-124", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:24:14.127 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:14 smithi090 ceph-mon[41670]: osdmap e711: 8 total, 8 up, 7 in 2024-04-14T15:24:14.127 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:14 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-124"}]: dispatch 2024-04-14T15:24:14.127 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:14 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-124"}]: dispatch 2024-04-14T15:24:14.127 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:14 smithi090 ceph-mon[41670]: pgmap v1050: 214 pgs: 32 unknown, 182 active+clean; 582 KiB data, 672 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:24:14.127 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:14 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-124", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:24:14.128 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:14 smithi090 ceph-mon[30399]: osdmap e711: 8 total, 8 up, 7 in 2024-04-14T15:24:14.128 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:14 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-124"}]: dispatch 2024-04-14T15:24:14.128 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:14 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-124"}]: dispatch 2024-04-14T15:24:14.128 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:14 smithi090 ceph-mon[30399]: pgmap v1050: 214 pgs: 32 unknown, 182 active+clean; 582 KiB data, 672 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:24:14.151 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:24:14.178 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:14.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:14.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:24:14.280 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:24:14.304 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:14.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:14.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:24:14.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.232+0000 7fe487460700 1 -- 172.21.15.90:0/3583368960 >> v1:172.21.15.90:6789/0 conn(0x7fe48010b720 legacy=0x7fe48010db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:14.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.232+0000 7fe487460700 1 -- 172.21.15.90:0/3583368960 shutdown_connections 2024-04-14T15:24:14.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.232+0000 7fe487460700 1 -- 172.21.15.90:0/3583368960 >> 172.21.15.90:0/3583368960 conn(0x7fe4800fe3a0 msgr2=0x7fe4801007c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:24:14.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.232+0000 7fe487460700 1 -- 172.21.15.90:0/3583368960 shutdown_connections 2024-04-14T15:24:14.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.232+0000 7fe487460700 1 -- 172.21.15.90:0/3583368960 wait complete. 2024-04-14T15:24:14.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.233+0000 7fe487460700 1 librados: starting msgr at 2024-04-14T15:24:14.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.233+0000 7fe487460700 1 librados: starting objecter 2024-04-14T15:24:14.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.233+0000 7fe487460700 1 Processor -- start 2024-04-14T15:24:14.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.233+0000 7fe487460700 1 -- start start 2024-04-14T15:24:14.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.233+0000 7fe487460700 1 librados: setting wanted keys 2024-04-14T15:24:14.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.233+0000 7fe487460700 1 librados: calling monclient init 2024-04-14T15:24:14.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.233+0000 7fe487460700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe4801a9bb0 con 0x7fe48010f2f0 2024-04-14T15:24:14.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.233+0000 7fe487460700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe4801aadd0 con 0x7fe48010b720 2024-04-14T15:24:14.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.233+0000 7fe487460700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe4801ac010 con 0x7fe480108870 2024-04-14T15:24:14.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.233+0000 7fe4851fc700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7fe480108870 0x7fe48010eb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:47078/0 (socket says 172.21.15.90:47078) 2024-04-14T15:24:14.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.233+0000 7fe4859fd700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fe48010f2f0 0x7fe4801a82c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:45566/0 (socket says 172.21.15.90:45566) 2024-04-14T15:24:14.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.233+0000 7fe4851fc700 1 -- 172.21.15.90:0/639231086 learned_addr learned my addr 172.21.15.90:0/639231086 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:24:14.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.233+0000 7fe4767fc700 10 client.?.objecter ms_handle_connect 0x7fe48010f2f0 2024-04-14T15:24:14.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.233+0000 7fe4767fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:24:14.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.233+0000 7fe4767fc700 10 client.?.objecter ms_handle_connect 0x7fe480108870 2024-04-14T15:24:14.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.233+0000 7fe4767fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:24:14.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.233+0000 7fe4767fc700 1 -- 172.21.15.90:0/639231086 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 182188836 0 0) 0x7fe4801a9bb0 con 0x7fe48010f2f0 2024-04-14T15:24:14.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.233+0000 7fe4767fc700 1 -- 172.21.15.90:0/639231086 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe46c0035b0 con 0x7fe48010f2f0 2024-04-14T15:24:14.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.233+0000 7fe4767fc700 1 -- 172.21.15.90:0/639231086 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2129063581 0 0) 0x7fe4801ac010 con 0x7fe480108870 2024-04-14T15:24:14.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.233+0000 7fe4767fc700 1 -- 172.21.15.90:0/639231086 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe4801a9bb0 con 0x7fe480108870 2024-04-14T15:24:14.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.233+0000 7fe4767fc700 10 client.?.objecter ms_handle_connect 0x7fe48010b720 2024-04-14T15:24:14.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.233+0000 7fe4767fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:24:14.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.233+0000 7fe4767fc700 1 -- 172.21.15.90:0/639231086 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3771177951 0 0) 0x7fe46c0035b0 con 0x7fe48010f2f0 2024-04-14T15:24:14.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.234+0000 7fe4767fc700 1 -- 172.21.15.90:0/639231086 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe4801ac010 con 0x7fe48010f2f0 2024-04-14T15:24:14.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.234+0000 7fe4767fc700 1 -- 172.21.15.90:0/639231086 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 196222875 0 0) 0x7fe4801aadd0 con 0x7fe48010b720 2024-04-14T15:24:14.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.234+0000 7fe4767fc700 1 -- 172.21.15.90:0/639231086 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe46c0035b0 con 0x7fe48010b720 2024-04-14T15:24:14.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.234+0000 7fe4767fc700 1 -- 172.21.15.90:0/639231086 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fe478001bb0 con 0x7fe48010f2f0 2024-04-14T15:24:14.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.234+0000 7fe4767fc700 1 -- 172.21.15.90:0/639231086 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1172278306 0 0) 0x7fe4801a9bb0 con 0x7fe480108870 2024-04-14T15:24:14.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.234+0000 7fe4767fc700 1 -- 172.21.15.90:0/639231086 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe4801aadd0 con 0x7fe480108870 2024-04-14T15:24:14.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.234+0000 7fe4767fc700 1 -- 172.21.15.90:0/639231086 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fe47c0018f0 con 0x7fe480108870 2024-04-14T15:24:14.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.234+0000 7fe4767fc700 1 -- 172.21.15.90:0/639231086 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1136299070 0 0) 0x7fe4801ac010 con 0x7fe48010f2f0 2024-04-14T15:24:14.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.234+0000 7fe4767fc700 1 -- 172.21.15.90:0/639231086 >> v1:172.21.15.90:6790/0 conn(0x7fe480108870 legacy=0x7fe48010eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:14.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.234+0000 7fe4767fc700 1 -- 172.21.15.90:0/639231086 >> v1:172.21.15.92:6789/0 conn(0x7fe48010b720 legacy=0x7fe4801a4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:14.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.234+0000 7fe4767fc700 1 -- 172.21.15.90:0/639231086 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4801ad250 con 0x7fe48010f2f0 2024-04-14T15:24:14.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.234+0000 7fe487460700 1 -- 172.21.15.90:0/639231086 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe4801ac240 con 0x7fe48010f2f0 2024-04-14T15:24:14.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.234+0000 7fe487460700 10 client.37881.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:24:14.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.234+0000 7fe487460700 1 -- 172.21.15.90:0/639231086 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe4801ac700 con 0x7fe48010f2f0 2024-04-14T15:24:14.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.234+0000 7fe487460700 1 librados: init done 2024-04-14T15:24:14.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.234+0000 7fe4767fc700 1 -- 172.21.15.90:0/639231086 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fe478003ec0 con 0x7fe48010f2f0 2024-04-14T15:24:14.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.234+0000 7fe4767fc700 1 -- 172.21.15.90:0/639231086 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fe478005fa0 con 0x7fe48010f2f0 2024-04-14T15:24:14.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.235+0000 7fe487460700 1 -- 172.21.15.90:0/639231086 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe464005120 con 0x7fe48010f2f0 2024-04-14T15:24:14.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.235+0000 7fe4767fc700 1 -- 172.21.15.90:0/639231086 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fe478003a00 con 0x7fe48010f2f0 2024-04-14T15:24:14.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.236+0000 7fe4767fc700 1 -- 172.21.15.90:0/639231086 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(712..712 src has 467..712) ==== 8729+0+0 (unknown 1644923596 0 0) 0x7fe47808b080 con 0x7fe48010f2f0 2024-04-14T15:24:14.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.236+0000 7fe4767fc700 10 client.37881.objecter ms_dispatch 0x7fe480001550 osd_map(712..712 src has 467..712) 2024-04-14T15:24:14.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.236+0000 7fe4767fc700 3 client.37881.objecter handle_osd_map got epochs [712,712] > 0 2024-04-14T15:24:14.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.236+0000 7fe4767fc700 3 client.37881.objecter handle_osd_map decoding full epoch 712 2024-04-14T15:24:14.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.236+0000 7fe4767fc700 20 client.37881.objecter dump_active .. 0 homeless 2024-04-14T15:24:14.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.236+0000 7fe4767fc700 10 client.37881.objecter ms_handle_connect 0x7fe46c071220 2024-04-14T15:24:14.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.242+0000 7fe4767fc700 1 -- 172.21.15.90:0/639231086 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fe478057850 con 0x7fe48010f2f0 2024-04-14T15:24:14.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:14 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-124", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:24:14.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:14 smithi092 ceph-mon[33966]: osdmap e711: 8 total, 8 up, 7 in 2024-04-14T15:24:14.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:14 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-124"}]: dispatch 2024-04-14T15:24:14.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:14 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-124"}]: dispatch 2024-04-14T15:24:14.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:14 smithi092 ceph-mon[33966]: pgmap v1050: 214 pgs: 32 unknown, 182 active+clean; 582 KiB data, 672 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:24:14.406 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:24:14.433 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:14.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:14.434 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:24:14.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.459+0000 7fe487460700 1 -- 172.21.15.90:0/639231086 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7fe464005410 con 0x7fe48010f2f0 2024-04-14T15:24:14.535 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:24:14.559 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:14.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:14.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:24:14.661 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:24:14.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:14.689 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:14.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:24:14.789 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:24:14.840 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:14.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:14.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:24:14.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.845+0000 7fdc0bf3d700 1 -- 172.21.15.90:0/1095810642 >> v1:172.21.15.90:6789/0 conn(0x7fdc0410b8d0 legacy=0x7fdc0410dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:14.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.846+0000 7fdc0bf3d700 1 -- 172.21.15.90:0/1095810642 shutdown_connections 2024-04-14T15:24:14.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.846+0000 7fdc0bf3d700 1 -- 172.21.15.90:0/1095810642 >> 172.21.15.90:0/1095810642 conn(0x7fdc040fd290 msgr2=0x7fdc040ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:24:14.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.846+0000 7fdc0bf3d700 1 -- 172.21.15.90:0/1095810642 shutdown_connections 2024-04-14T15:24:14.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.846+0000 7fdc0bf3d700 1 -- 172.21.15.90:0/1095810642 wait complete. 2024-04-14T15:24:14.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.846+0000 7fdc0bf3d700 1 librados: starting msgr at 2024-04-14T15:24:14.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.846+0000 7fdc0bf3d700 1 librados: starting objecter 2024-04-14T15:24:14.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.846+0000 7fdc0bf3d700 1 Processor -- start 2024-04-14T15:24:14.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.846+0000 7fdc0bf3d700 1 -- start start 2024-04-14T15:24:14.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.846+0000 7fdc0bf3d700 1 librados: setting wanted keys 2024-04-14T15:24:14.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.846+0000 7fdc0bf3d700 1 librados: calling monclient init 2024-04-14T15:24:14.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.846+0000 7fdc0bf3d700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdc0410b520 con 0x7fdc04103ac0 2024-04-14T15:24:14.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.846+0000 7fdc0bf3d700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdc041a9010 con 0x7fdc0410b8d0 2024-04-14T15:24:14.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.846+0000 7fdc0bf3d700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdc041aa1f0 con 0x7fdc0410f440 2024-04-14T15:24:14.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.847+0000 7fdc09cd9700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fdc04103ac0 0x7fdc041083c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:45594/0 (socket says 172.21.15.90:45594) 2024-04-14T15:24:14.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.847+0000 7fdc09cd9700 1 -- 172.21.15.90:0/4179733851 learned_addr learned my addr 172.21.15.90:0/4179733851 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:24:14.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.847+0000 7fdbfaffd700 10 client.?.objecter ms_handle_connect 0x7fdc0410f440 2024-04-14T15:24:14.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.847+0000 7fdbfaffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:24:14.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.847+0000 7fdbfaffd700 10 client.?.objecter ms_handle_connect 0x7fdc04103ac0 2024-04-14T15:24:14.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.847+0000 7fdbfaffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:24:14.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.847+0000 7fdbfaffd700 10 client.?.objecter ms_handle_connect 0x7fdc0410b8d0 2024-04-14T15:24:14.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.847+0000 7fdbfaffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:24:14.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.847+0000 7fdbfaffd700 1 -- 172.21.15.90:0/4179733851 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3999798297 0 0) 0x7fdc041aa1f0 con 0x7fdc0410f440 2024-04-14T15:24:14.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.847+0000 7fdbfaffd700 1 -- 172.21.15.90:0/4179733851 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdbf00035b0 con 0x7fdc0410f440 2024-04-14T15:24:14.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.847+0000 7fdbfaffd700 1 -- 172.21.15.90:0/4179733851 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2890923705 0 0) 0x7fdc0410b520 con 0x7fdc04103ac0 2024-04-14T15:24:14.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.847+0000 7fdbfaffd700 1 -- 172.21.15.90:0/4179733851 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc041aa1f0 con 0x7fdc04103ac0 2024-04-14T15:24:14.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.847+0000 7fdbfaffd700 1 -- 172.21.15.90:0/4179733851 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2304972023 0 0) 0x7fdc041a9010 con 0x7fdc0410b8d0 2024-04-14T15:24:14.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.847+0000 7fdbfaffd700 1 -- 172.21.15.90:0/4179733851 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc0410b520 con 0x7fdc0410b8d0 2024-04-14T15:24:14.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.847+0000 7fdbfaffd700 1 -- 172.21.15.90:0/4179733851 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3479258733 0 0) 0x7fdbf00035b0 con 0x7fdc0410f440 2024-04-14T15:24:14.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.847+0000 7fdbfaffd700 1 -- 172.21.15.90:0/4179733851 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdc041a9010 con 0x7fdc0410f440 2024-04-14T15:24:14.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.847+0000 7fdbfaffd700 1 -- 172.21.15.90:0/4179733851 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fdbfc001c50 con 0x7fdc0410f440 2024-04-14T15:24:14.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.847+0000 7fdbfaffd700 1 -- 172.21.15.90:0/4179733851 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2331235614 0 0) 0x7fdc041aa1f0 con 0x7fdc04103ac0 2024-04-14T15:24:14.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.847+0000 7fdbfaffd700 1 -- 172.21.15.90:0/4179733851 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdbf00035b0 con 0x7fdc04103ac0 2024-04-14T15:24:14.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.847+0000 7fdbfaffd700 1 -- 172.21.15.90:0/4179733851 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fdc00001870 con 0x7fdc04103ac0 2024-04-14T15:24:14.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.847+0000 7fdbfaffd700 1 -- 172.21.15.90:0/4179733851 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1320157212 0 0) 0x7fdc0410b520 con 0x7fdc0410b8d0 2024-04-14T15:24:14.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.847+0000 7fdbfaffd700 1 -- 172.21.15.90:0/4179733851 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdc041aa1f0 con 0x7fdc0410b8d0 2024-04-14T15:24:14.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.847+0000 7fdbfaffd700 1 -- 172.21.15.90:0/4179733851 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fdbf40019d0 con 0x7fdc0410b8d0 2024-04-14T15:24:14.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.847+0000 7fdbfaffd700 1 -- 172.21.15.90:0/4179733851 <== mon.2 v1:172.21.15.90:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3993816286 0 0) 0x7fdc041a9010 con 0x7fdc0410f440 2024-04-14T15:24:14.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.847+0000 7fdbfaffd700 1 -- 172.21.15.90:0/4179733851 >> v1:172.21.15.92:6789/0 conn(0x7fdc0410b8d0 legacy=0x7fdc04108ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:14.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.847+0000 7fdbfaffd700 1 -- 172.21.15.90:0/4179733851 >> v1:172.21.15.90:6789/0 conn(0x7fdc04103ac0 legacy=0x7fdc041083c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:14.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.847+0000 7fdbfaffd700 1 -- 172.21.15.90:0/4179733851 --> v1:172.21.15.90:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc041ab3d0 con 0x7fdc0410f440 2024-04-14T15:24:14.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.847+0000 7fdc0bf3d700 1 -- 172.21.15.90:0/4179733851 --> v1:172.21.15.90:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdc041a8060 con 0x7fdc0410f440 2024-04-14T15:24:14.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.847+0000 7fdc0bf3d700 10 client.47837.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:24:14.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.848+0000 7fdc0bf3d700 1 -- 172.21.15.90:0/4179733851 --> v1:172.21.15.90:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fdc041a8520 con 0x7fdc0410f440 2024-04-14T15:24:14.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.848+0000 7fdc0bf3d700 1 librados: init done 2024-04-14T15:24:14.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.848+0000 7fdbfaffd700 1 -- 172.21.15.90:0/4179733851 <== mon.2 v1:172.21.15.90:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fdbfc003ec0 con 0x7fdc0410f440 2024-04-14T15:24:14.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.848+0000 7fdbfaffd700 1 -- 172.21.15.90:0/4179733851 <== mon.2 v1:172.21.15.90:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fdbfc004e10 con 0x7fdc0410f440 2024-04-14T15:24:14.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.848+0000 7fdc0bf3d700 1 -- 172.21.15.90:0/4179733851 --> v1:172.21.15.90:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fdbe8005100 con 0x7fdc0410f440 2024-04-14T15:24:14.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.849+0000 7fdbfaffd700 1 -- 172.21.15.90:0/4179733851 <== mon.2 v1:172.21.15.90:6790/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fdbfc003a20 con 0x7fdc0410f440 2024-04-14T15:24:14.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.849+0000 7fdbfaffd700 1 -- 172.21.15.90:0/4179733851 <== mon.2 v1:172.21.15.90:6790/0 8 ==== osd_map(712..712 src has 467..712) ==== 8729+0+0 (unknown 1644923596 0 0) 0x7fdbfc089c60 con 0x7fdc0410f440 2024-04-14T15:24:14.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.849+0000 7fdbfaffd700 10 client.47837.objecter ms_dispatch 0x7fdc04001550 osd_map(712..712 src has 467..712) 2024-04-14T15:24:14.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.849+0000 7fdbfaffd700 3 client.47837.objecter handle_osd_map got epochs [712,712] > 0 2024-04-14T15:24:14.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.849+0000 7fdbfaffd700 3 client.47837.objecter handle_osd_map decoding full epoch 712 2024-04-14T15:24:14.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.850+0000 7fdbfaffd700 20 client.47837.objecter dump_active .. 0 homeless 2024-04-14T15:24:14.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.850+0000 7fdbfaffd700 1 -- 172.21.15.90:0/4179733851 <== mon.2 v1:172.21.15.90:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v712) ==== 54+0+16 (unknown 1834271652 0 2420000192) 0x7fdbfc089fd0 con 0x7fdc0410f440 2024-04-14T15:24:14.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.850+0000 7fdbfaffd700 10 client.47837.objecter ms_handle_connect 0x7fdbf00712e0 2024-04-14T15:24:14.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.850+0000 7fdc0bf3d700 10 client.47837.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:24:14.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.850+0000 7fdc0bf3d700 15 client.47837.objecter _session_command_op_assign -1 1 2024-04-14T15:24:14.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.850+0000 7fdc0bf3d700 20 client.47837.objecter _get_session s=0x7fdbe80053e0 osd=0 3 2024-04-14T15:24:14.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.850+0000 7fdc0bf3d700 20 client.47837.objecter put_session s=0x7fdbe80053e0 osd=0 3 2024-04-14T15:24:14.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.850+0000 7fdc0bf3d700 20 client.47837.objecter _get_session s=0x7fdbe80053e0 osd=0 3 2024-04-14T15:24:14.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.850+0000 7fdc0bf3d700 15 client.47837.objecter _session_command_op_remove -1 1 2024-04-14T15:24:14.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.850+0000 7fdc0bf3d700 20 client.47837.objecter get_session s=0x7fdbe80053e0 osd=0 3 2024-04-14T15:24:14.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.850+0000 7fdc0bf3d700 15 client.47837.objecter _session_command_op_assign 0 1 2024-04-14T15:24:14.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.850+0000 7fdc0bf3d700 20 client.47837.objecter put_session s=0x7fdbe80053e0 osd=0 4 2024-04-14T15:24:14.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.850+0000 7fdc0bf3d700 10 client.47837.objecter _send_command 1 2024-04-14T15:24:14.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.850+0000 7fdc0bf3d700 1 -- 172.21.15.90:0/4179733851 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdbe8004b70 con 0x7fdbe8004730 2024-04-14T15:24:14.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.850+0000 7fdbfaffd700 10 client.47837.objecter ms_handle_connect 0x7fdbe8004730 2024-04-14T15:24:14.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.851+0000 7fdbfaffd700 1 -- 172.21.15.90:0/4179733851 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fdbe8004b70 con 0x7fdbe8004730 2024-04-14T15:24:14.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.851+0000 7fdbfaffd700 10 client.47837.objecter ms_dispatch 0x7fdc04001550 command_reply(tid 1: 0 ) 2024-04-14T15:24:14.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.851+0000 7fdbfaffd700 10 client.47837.objecter _finish_command 1 = system:0 2024-04-14T15:24:14.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.851+0000 7fdbfaffd700 20 client.47837.objecter put_session s=0x7fdbe80053e0 osd=0 4 2024-04-14T15:24:14.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.851+0000 7fdbfaffd700 15 client.47837.objecter _session_command_op_remove 0 1 2024-04-14T15:24:14.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.867+0000 7fdc0bf3d700 10 client.47837.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:24:14.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.867+0000 7fdc0bf3d700 15 client.47837.objecter _session_command_op_assign -1 2 2024-04-14T15:24:14.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.867+0000 7fdc0bf3d700 20 client.47837.objecter _get_session s=0x7fdbe80053e0 osd=0 3 2024-04-14T15:24:14.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.867+0000 7fdc0bf3d700 20 client.47837.objecter put_session s=0x7fdbe80053e0 osd=0 3 2024-04-14T15:24:14.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.867+0000 7fdc0bf3d700 20 client.47837.objecter _get_session s=0x7fdbe80053e0 osd=0 3 2024-04-14T15:24:14.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.867+0000 7fdc0bf3d700 15 client.47837.objecter _session_command_op_remove -1 2 2024-04-14T15:24:14.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.867+0000 7fdc0bf3d700 20 client.47837.objecter get_session s=0x7fdbe80053e0 osd=0 3 2024-04-14T15:24:14.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.867+0000 7fdc0bf3d700 15 client.47837.objecter _session_command_op_assign 0 2 2024-04-14T15:24:14.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.867+0000 7fdc0bf3d700 20 client.47837.objecter put_session s=0x7fdbe80053e0 osd=0 4 2024-04-14T15:24:14.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.867+0000 7fdc0bf3d700 10 client.47837.objecter _send_command 2 2024-04-14T15:24:14.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.867+0000 7fdc0bf3d700 1 -- 172.21.15.90:0/4179733851 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fdbe800be00 con 0x7fdbe8004730 2024-04-14T15:24:14.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.869+0000 7fdbfaffd700 1 -- 172.21.15.90:0/4179733851 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fdbe800be00 con 0x7fdbe8004730 2024-04-14T15:24:14.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.869+0000 7fdbfaffd700 10 client.47837.objecter ms_dispatch 0x7fdc04001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:24:14.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.869+0000 7fdbfaffd700 10 client.47837.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:24:14.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.869+0000 7fdbfaffd700 20 client.47837.objecter put_session s=0x7fdbe80053e0 osd=0 4 2024-04-14T15:24:14.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.869+0000 7fdbfaffd700 15 client.47837.objecter _session_command_op_remove 0 2 2024-04-14T15:24:14.870 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:24:14.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.869+0000 7fdc0bf3d700 10 client.47837.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:24:14.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.869+0000 7fdc0bf3d700 15 client.47837.objecter _session_command_op_assign -1 3 2024-04-14T15:24:14.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.869+0000 7fdc0bf3d700 20 client.47837.objecter _get_session s=0x7fdbe80040f0 osd=1 3 2024-04-14T15:24:14.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.869+0000 7fdc0bf3d700 20 client.47837.objecter put_session s=0x7fdbe80040f0 osd=1 3 2024-04-14T15:24:14.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.869+0000 7fdc0bf3d700 20 client.47837.objecter _get_session s=0x7fdbe80040f0 osd=1 3 2024-04-14T15:24:14.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.869+0000 7fdc0bf3d700 15 client.47837.objecter _session_command_op_remove -1 3 2024-04-14T15:24:14.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.869+0000 7fdc0bf3d700 20 client.47837.objecter get_session s=0x7fdbe80040f0 osd=1 3 2024-04-14T15:24:14.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.869+0000 7fdc0bf3d700 15 client.47837.objecter _session_command_op_assign 1 3 2024-04-14T15:24:14.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.869+0000 7fdc0bf3d700 20 client.47837.objecter put_session s=0x7fdbe80040f0 osd=1 4 2024-04-14T15:24:14.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.869+0000 7fdc0bf3d700 10 client.47837.objecter _send_command 3 2024-04-14T15:24:14.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.869+0000 7fdc0bf3d700 1 -- 172.21.15.90:0/4179733851 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fdbe8004430 con 0x7fdbe8002100 2024-04-14T15:24:14.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.869+0000 7fdbfaffd700 10 client.47837.objecter ms_handle_connect 0x7fdbe8002100 2024-04-14T15:24:14.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.870+0000 7fdbfaffd700 1 -- 172.21.15.90:0/4179733851 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fdbe8004430 con 0x7fdbe8002100 2024-04-14T15:24:14.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.870+0000 7fdbfaffd700 10 client.47837.objecter ms_dispatch 0x7fdc04001550 command_reply(tid 3: 0 ) 2024-04-14T15:24:14.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.870+0000 7fdbfaffd700 10 client.47837.objecter _finish_command 3 = system:0 2024-04-14T15:24:14.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.870+0000 7fdbfaffd700 20 client.47837.objecter put_session s=0x7fdbe80040f0 osd=1 4 2024-04-14T15:24:14.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.870+0000 7fdbfaffd700 15 client.47837.objecter _session_command_op_remove 1 3 2024-04-14T15:24:14.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.886+0000 7fdc0bf3d700 10 client.47837.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:24:14.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.886+0000 7fdc0bf3d700 15 client.47837.objecter _session_command_op_assign -1 4 2024-04-14T15:24:14.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.886+0000 7fdc0bf3d700 20 client.47837.objecter _get_session s=0x7fdbe80040f0 osd=1 3 2024-04-14T15:24:14.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.886+0000 7fdc0bf3d700 20 client.47837.objecter put_session s=0x7fdbe80040f0 osd=1 3 2024-04-14T15:24:14.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.886+0000 7fdc0bf3d700 20 client.47837.objecter _get_session s=0x7fdbe80040f0 osd=1 3 2024-04-14T15:24:14.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.886+0000 7fdc0bf3d700 15 client.47837.objecter _session_command_op_remove -1 4 2024-04-14T15:24:14.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.886+0000 7fdc0bf3d700 20 client.47837.objecter get_session s=0x7fdbe80040f0 osd=1 3 2024-04-14T15:24:14.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.886+0000 7fdc0bf3d700 15 client.47837.objecter _session_command_op_assign 1 4 2024-04-14T15:24:14.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.886+0000 7fdc0bf3d700 20 client.47837.objecter put_session s=0x7fdbe80040f0 osd=1 4 2024-04-14T15:24:14.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.886+0000 7fdc0bf3d700 10 client.47837.objecter _send_command 4 2024-04-14T15:24:14.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.886+0000 7fdc0bf3d700 1 -- 172.21.15.90:0/4179733851 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fdbe80016c0 con 0x7fdbe8002100 2024-04-14T15:24:14.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.888+0000 7fdbfaffd700 1 -- 172.21.15.90:0/4179733851 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fdbe80016c0 con 0x7fdbe8002100 2024-04-14T15:24:14.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.888+0000 7fdbfaffd700 10 client.47837.objecter ms_dispatch 0x7fdc04001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:24:14.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.888+0000 7fdbfaffd700 10 client.47837.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:24:14.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.888+0000 7fdbfaffd700 20 client.47837.objecter put_session s=0x7fdbe80040f0 osd=1 4 2024-04-14T15:24:14.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.888+0000 7fdbfaffd700 15 client.47837.objecter _session_command_op_remove 1 4 2024-04-14T15:24:14.889 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:24:14.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.888+0000 7fdc0bf3d700 10 client.47837.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:24:14.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.888+0000 7fdc0bf3d700 15 client.47837.objecter _session_command_op_assign -1 5 2024-04-14T15:24:14.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.888+0000 7fdc0bf3d700 20 client.47837.objecter _get_session s=0x7fdbe800c780 osd=2 3 2024-04-14T15:24:14.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.888+0000 7fdc0bf3d700 20 client.47837.objecter put_session s=0x7fdbe800c780 osd=2 3 2024-04-14T15:24:14.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.888+0000 7fdc0bf3d700 20 client.47837.objecter _get_session s=0x7fdbe800c780 osd=2 3 2024-04-14T15:24:14.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.888+0000 7fdc0bf3d700 15 client.47837.objecter _session_command_op_remove -1 5 2024-04-14T15:24:14.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.888+0000 7fdc0bf3d700 20 client.47837.objecter get_session s=0x7fdbe800c780 osd=2 3 2024-04-14T15:24:14.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.888+0000 7fdc0bf3d700 15 client.47837.objecter _session_command_op_assign 2 5 2024-04-14T15:24:14.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.888+0000 7fdc0bf3d700 20 client.47837.objecter put_session s=0x7fdbe800c780 osd=2 4 2024-04-14T15:24:14.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.888+0000 7fdc0bf3d700 10 client.47837.objecter _send_command 5 2024-04-14T15:24:14.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.888+0000 7fdc0bf3d700 1 -- 172.21.15.90:0/4179733851 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fdbe80154f0 con 0x7fdbe80119e0 2024-04-14T15:24:14.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.889+0000 7fdbfaffd700 10 client.47837.objecter ms_handle_connect 0x7fdbe80119e0 2024-04-14T15:24:14.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.889+0000 7fdbfaffd700 1 -- 172.21.15.90:0/4179733851 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fdbe80154f0 con 0x7fdbe80119e0 2024-04-14T15:24:14.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.889+0000 7fdbfaffd700 10 client.47837.objecter ms_dispatch 0x7fdc04001550 command_reply(tid 5: 0 ) 2024-04-14T15:24:14.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.889+0000 7fdbfaffd700 10 client.47837.objecter _finish_command 5 = system:0 2024-04-14T15:24:14.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.889+0000 7fdbfaffd700 20 client.47837.objecter put_session s=0x7fdbe800c780 osd=2 4 2024-04-14T15:24:14.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.889+0000 7fdbfaffd700 15 client.47837.objecter _session_command_op_remove 2 5 2024-04-14T15:24:14.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.905+0000 7fdc0bf3d700 10 client.47837.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:24:14.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.905+0000 7fdc0bf3d700 15 client.47837.objecter _session_command_op_assign -1 6 2024-04-14T15:24:14.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.905+0000 7fdc0bf3d700 20 client.47837.objecter _get_session s=0x7fdbe800c780 osd=2 3 2024-04-14T15:24:14.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.905+0000 7fdc0bf3d700 20 client.47837.objecter put_session s=0x7fdbe800c780 osd=2 3 2024-04-14T15:24:14.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.905+0000 7fdc0bf3d700 20 client.47837.objecter _get_session s=0x7fdbe800c780 osd=2 3 2024-04-14T15:24:14.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.905+0000 7fdc0bf3d700 15 client.47837.objecter _session_command_op_remove -1 6 2024-04-14T15:24:14.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.905+0000 7fdc0bf3d700 20 client.47837.objecter get_session s=0x7fdbe800c780 osd=2 3 2024-04-14T15:24:14.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.905+0000 7fdc0bf3d700 15 client.47837.objecter _session_command_op_assign 2 6 2024-04-14T15:24:14.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.905+0000 7fdc0bf3d700 20 client.47837.objecter put_session s=0x7fdbe800c780 osd=2 4 2024-04-14T15:24:14.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.905+0000 7fdc0bf3d700 10 client.47837.objecter _send_command 6 2024-04-14T15:24:14.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.905+0000 7fdc0bf3d700 1 -- 172.21.15.90:0/4179733851 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fdbe800c2c0 con 0x7fdbe80119e0 2024-04-14T15:24:14.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.907+0000 7fdbfaffd700 1 -- 172.21.15.90:0/4179733851 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fdbe800c2c0 con 0x7fdbe80119e0 2024-04-14T15:24:14.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.907+0000 7fdbfaffd700 10 client.47837.objecter ms_dispatch 0x7fdc04001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:24:14.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.907+0000 7fdbfaffd700 10 client.47837.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:24:14.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.907+0000 7fdbfaffd700 20 client.47837.objecter put_session s=0x7fdbe800c780 osd=2 4 2024-04-14T15:24:14.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.907+0000 7fdbfaffd700 15 client.47837.objecter _session_command_op_remove 2 6 2024-04-14T15:24:14.908 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:24:14.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.907+0000 7fdc0bf3d700 10 client.47837.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:24:14.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.907+0000 7fdc0bf3d700 15 client.47837.objecter _session_command_op_assign -1 7 2024-04-14T15:24:14.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.907+0000 7fdc0bf3d700 20 client.47837.objecter _get_session s=0x7fdbe800c4d0 osd=3 3 2024-04-14T15:24:14.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.907+0000 7fdc0bf3d700 20 client.47837.objecter put_session s=0x7fdbe800c4d0 osd=3 3 2024-04-14T15:24:14.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.907+0000 7fdc0bf3d700 20 client.47837.objecter _get_session s=0x7fdbe800c4d0 osd=3 3 2024-04-14T15:24:14.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.907+0000 7fdc0bf3d700 15 client.47837.objecter _session_command_op_remove -1 7 2024-04-14T15:24:14.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.907+0000 7fdc0bf3d700 20 client.47837.objecter get_session s=0x7fdbe800c4d0 osd=3 3 2024-04-14T15:24:14.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.907+0000 7fdc0bf3d700 15 client.47837.objecter _session_command_op_assign 3 7 2024-04-14T15:24:14.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.907+0000 7fdc0bf3d700 20 client.47837.objecter put_session s=0x7fdbe800c4d0 osd=3 4 2024-04-14T15:24:14.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.907+0000 7fdc0bf3d700 10 client.47837.objecter _send_command 7 2024-04-14T15:24:14.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.907+0000 7fdc0bf3d700 1 -- 172.21.15.90:0/4179733851 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fdbe80117d0 con 0x7fdbe8016820 2024-04-14T15:24:14.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.907+0000 7fdbfaffd700 10 client.47837.objecter ms_handle_connect 0x7fdbe8016820 2024-04-14T15:24:14.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.908+0000 7fdbfaffd700 1 -- 172.21.15.90:0/4179733851 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fdbe80117d0 con 0x7fdbe8016820 2024-04-14T15:24:14.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.908+0000 7fdbfaffd700 10 client.47837.objecter ms_dispatch 0x7fdc04001550 command_reply(tid 7: 0 ) 2024-04-14T15:24:14.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.908+0000 7fdbfaffd700 10 client.47837.objecter _finish_command 7 = system:0 2024-04-14T15:24:14.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.908+0000 7fdbfaffd700 20 client.47837.objecter put_session s=0x7fdbe800c4d0 osd=3 4 2024-04-14T15:24:14.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.908+0000 7fdbfaffd700 15 client.47837.objecter _session_command_op_remove 3 7 2024-04-14T15:24:14.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.925+0000 7fdc0bf3d700 10 client.47837.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:24:14.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.925+0000 7fdc0bf3d700 15 client.47837.objecter _session_command_op_assign -1 8 2024-04-14T15:24:14.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.925+0000 7fdc0bf3d700 20 client.47837.objecter _get_session s=0x7fdbe800c4d0 osd=3 3 2024-04-14T15:24:14.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.925+0000 7fdc0bf3d700 20 client.47837.objecter put_session s=0x7fdbe800c4d0 osd=3 3 2024-04-14T15:24:14.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.925+0000 7fdc0bf3d700 20 client.47837.objecter _get_session s=0x7fdbe800c4d0 osd=3 3 2024-04-14T15:24:14.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.925+0000 7fdc0bf3d700 15 client.47837.objecter _session_command_op_remove -1 8 2024-04-14T15:24:14.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.925+0000 7fdc0bf3d700 20 client.47837.objecter get_session s=0x7fdbe800c4d0 osd=3 3 2024-04-14T15:24:14.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.925+0000 7fdc0bf3d700 15 client.47837.objecter _session_command_op_assign 3 8 2024-04-14T15:24:14.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.925+0000 7fdc0bf3d700 20 client.47837.objecter put_session s=0x7fdbe800c4d0 osd=3 4 2024-04-14T15:24:14.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.925+0000 7fdc0bf3d700 10 client.47837.objecter _send_command 8 2024-04-14T15:24:14.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.925+0000 7fdc0bf3d700 1 -- 172.21.15.90:0/4179733851 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fdbe8017b80 con 0x7fdbe8016820 2024-04-14T15:24:14.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.926+0000 7fdbfaffd700 1 -- 172.21.15.90:0/4179733851 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fdbe8017b80 con 0x7fdbe8016820 2024-04-14T15:24:14.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.927+0000 7fdbfaffd700 10 client.47837.objecter ms_dispatch 0x7fdc04001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:24:14.927 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:24:14.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.927+0000 7fdbfaffd700 10 client.47837.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:24:14.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.927+0000 7fdbfaffd700 20 client.47837.objecter put_session s=0x7fdbe800c4d0 osd=3 4 2024-04-14T15:24:14.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.927+0000 7fdbfaffd700 15 client.47837.objecter _session_command_op_remove 3 8 2024-04-14T15:24:14.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.927+0000 7fdc0bf3d700 10 client.47837.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:24:14.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.927+0000 7fdc0bf3d700 15 client.47837.objecter _session_command_op_assign -1 9 2024-04-14T15:24:14.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.927+0000 7fdc0bf3d700 20 client.47837.objecter _get_session s=0x7fdbe8017bc0 osd=4 3 2024-04-14T15:24:14.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.927+0000 7fdc0bf3d700 20 client.47837.objecter put_session s=0x7fdbe8017bc0 osd=4 3 2024-04-14T15:24:14.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.927+0000 7fdc0bf3d700 20 client.47837.objecter _get_session s=0x7fdbe8017bc0 osd=4 3 2024-04-14T15:24:14.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.927+0000 7fdc0bf3d700 15 client.47837.objecter _session_command_op_remove -1 9 2024-04-14T15:24:14.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.927+0000 7fdc0bf3d700 20 client.47837.objecter get_session s=0x7fdbe8017bc0 osd=4 3 2024-04-14T15:24:14.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.927+0000 7fdc0bf3d700 15 client.47837.objecter _session_command_op_assign 4 9 2024-04-14T15:24:14.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.927+0000 7fdc0bf3d700 20 client.47837.objecter put_session s=0x7fdbe8017bc0 osd=4 4 2024-04-14T15:24:14.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.927+0000 7fdc0bf3d700 10 client.47837.objecter _send_command 9 2024-04-14T15:24:14.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.927+0000 7fdc0bf3d700 1 -- 172.21.15.90:0/4179733851 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fdbe8010f50 con 0x7fdbe801b540 2024-04-14T15:24:14.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.927+0000 7fdbfaffd700 10 client.47837.objecter ms_handle_connect 0x7fdbe801b540 2024-04-14T15:24:14.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.928+0000 7fdbfaffd700 1 -- 172.21.15.90:0/4179733851 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fdbe8010f50 con 0x7fdbe801b540 2024-04-14T15:24:14.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.928+0000 7fdbfaffd700 10 client.47837.objecter ms_dispatch 0x7fdc04001550 command_reply(tid 9: 0 ) 2024-04-14T15:24:14.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.928+0000 7fdbfaffd700 10 client.47837.objecter _finish_command 9 = system:0 2024-04-14T15:24:14.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.928+0000 7fdbfaffd700 20 client.47837.objecter put_session s=0x7fdbe8017bc0 osd=4 4 2024-04-14T15:24:14.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.928+0000 7fdbfaffd700 15 client.47837.objecter _session_command_op_remove 4 9 2024-04-14T15:24:14.942 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:24:14.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.944+0000 7fdc0bf3d700 10 client.47837.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:24:14.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.944+0000 7fdc0bf3d700 15 client.47837.objecter _session_command_op_assign -1 10 2024-04-14T15:24:14.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.944+0000 7fdc0bf3d700 20 client.47837.objecter _get_session s=0x7fdbe8017bc0 osd=4 3 2024-04-14T15:24:14.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.944+0000 7fdc0bf3d700 20 client.47837.objecter put_session s=0x7fdbe8017bc0 osd=4 3 2024-04-14T15:24:14.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.944+0000 7fdc0bf3d700 20 client.47837.objecter _get_session s=0x7fdbe8017bc0 osd=4 3 2024-04-14T15:24:14.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.944+0000 7fdc0bf3d700 15 client.47837.objecter _session_command_op_remove -1 10 2024-04-14T15:24:14.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.944+0000 7fdc0bf3d700 20 client.47837.objecter get_session s=0x7fdbe8017bc0 osd=4 3 2024-04-14T15:24:14.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.944+0000 7fdc0bf3d700 15 client.47837.objecter _session_command_op_assign 4 10 2024-04-14T15:24:14.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.944+0000 7fdc0bf3d700 20 client.47837.objecter put_session s=0x7fdbe8017bc0 osd=4 4 2024-04-14T15:24:14.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.944+0000 7fdc0bf3d700 10 client.47837.objecter _send_command 10 2024-04-14T15:24:14.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.944+0000 7fdc0bf3d700 1 -- 172.21.15.90:0/4179733851 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fdbe8010f50 con 0x7fdbe801b540 2024-04-14T15:24:14.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.946+0000 7fdbfaffd700 1 -- 172.21.15.90:0/4179733851 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fdbe8010f50 con 0x7fdbe801b540 2024-04-14T15:24:14.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.946+0000 7fdbfaffd700 10 client.47837.objecter ms_dispatch 0x7fdc04001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:24:14.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.946+0000 7fdbfaffd700 10 client.47837.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:24:14.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.946+0000 7fdbfaffd700 20 client.47837.objecter put_session s=0x7fdbe8017bc0 osd=4 4 2024-04-14T15:24:14.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.946+0000 7fdbfaffd700 15 client.47837.objecter _session_command_op_remove 4 10 2024-04-14T15:24:14.947 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:24:14.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.946+0000 7fdc0bf3d700 10 client.47837.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:24:14.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.946+0000 7fdc0bf3d700 15 client.47837.objecter _session_command_op_assign -1 11 2024-04-14T15:24:14.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.946+0000 7fdc0bf3d700 20 client.47837.objecter _get_session s=0x7fdbe8010f70 osd=5 3 2024-04-14T15:24:14.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.946+0000 7fdc0bf3d700 20 client.47837.objecter put_session s=0x7fdbe8010f70 osd=5 3 2024-04-14T15:24:14.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.946+0000 7fdc0bf3d700 20 client.47837.objecter _get_session s=0x7fdbe8010f70 osd=5 3 2024-04-14T15:24:14.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.946+0000 7fdc0bf3d700 15 client.47837.objecter _session_command_op_remove -1 11 2024-04-14T15:24:14.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.946+0000 7fdc0bf3d700 20 client.47837.objecter get_session s=0x7fdbe8010f70 osd=5 3 2024-04-14T15:24:14.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.946+0000 7fdc0bf3d700 15 client.47837.objecter _session_command_op_assign 5 11 2024-04-14T15:24:14.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.946+0000 7fdc0bf3d700 20 client.47837.objecter put_session s=0x7fdbe8010f70 osd=5 4 2024-04-14T15:24:14.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.946+0000 7fdc0bf3d700 10 client.47837.objecter _send_command 11 2024-04-14T15:24:14.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.946+0000 7fdc0bf3d700 1 -- 172.21.15.90:0/4179733851 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fdbe80232e0 con 0x7fdbe801f5c0 2024-04-14T15:24:14.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.947+0000 7fdbfaffd700 10 client.47837.objecter ms_handle_connect 0x7fdbe801f5c0 2024-04-14T15:24:14.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.948+0000 7fdbfaffd700 1 -- 172.21.15.90:0/4179733851 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fdbe80232e0 con 0x7fdbe801f5c0 2024-04-14T15:24:14.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.948+0000 7fdbfaffd700 10 client.47837.objecter ms_dispatch 0x7fdc04001550 command_reply(tid 11: 0 ) 2024-04-14T15:24:14.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.948+0000 7fdbfaffd700 10 client.47837.objecter _finish_command 11 = system:0 2024-04-14T15:24:14.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.948+0000 7fdbfaffd700 20 client.47837.objecter put_session s=0x7fdbe8010f70 osd=5 4 2024-04-14T15:24:14.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.948+0000 7fdbfaffd700 15 client.47837.objecter _session_command_op_remove 5 11 2024-04-14T15:24:14.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.964+0000 7fdc0bf3d700 10 client.47837.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:24:14.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.964+0000 7fdc0bf3d700 15 client.47837.objecter _session_command_op_assign -1 12 2024-04-14T15:24:14.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.964+0000 7fdc0bf3d700 20 client.47837.objecter _get_session s=0x7fdbe8010f70 osd=5 3 2024-04-14T15:24:14.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.964+0000 7fdc0bf3d700 20 client.47837.objecter put_session s=0x7fdbe8010f70 osd=5 3 2024-04-14T15:24:14.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.964+0000 7fdc0bf3d700 20 client.47837.objecter _get_session s=0x7fdbe8010f70 osd=5 3 2024-04-14T15:24:14.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.964+0000 7fdc0bf3d700 15 client.47837.objecter _session_command_op_remove -1 12 2024-04-14T15:24:14.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.964+0000 7fdc0bf3d700 20 client.47837.objecter get_session s=0x7fdbe8010f70 osd=5 3 2024-04-14T15:24:14.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.964+0000 7fdc0bf3d700 15 client.47837.objecter _session_command_op_assign 5 12 2024-04-14T15:24:14.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.964+0000 7fdc0bf3d700 20 client.47837.objecter put_session s=0x7fdbe8010f70 osd=5 4 2024-04-14T15:24:14.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.964+0000 7fdc0bf3d700 10 client.47837.objecter _send_command 12 2024-04-14T15:24:14.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.964+0000 7fdc0bf3d700 1 -- 172.21.15.90:0/4179733851 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fdbe80232a0 con 0x7fdbe801f5c0 2024-04-14T15:24:14.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:14.966 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:14.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:24:14.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.966+0000 7fdbfaffd700 1 -- 172.21.15.90:0/4179733851 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fdbe80232a0 con 0x7fdbe801f5c0 2024-04-14T15:24:14.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.966+0000 7fdbfaffd700 10 client.47837.objecter ms_dispatch 0x7fdc04001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:24:14.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.966+0000 7fdbfaffd700 10 client.47837.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:24:14.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.966+0000 7fdbfaffd700 20 client.47837.objecter put_session s=0x7fdbe8010f70 osd=5 4 2024-04-14T15:24:14.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.966+0000 7fdbfaffd700 15 client.47837.objecter _session_command_op_remove 5 12 2024-04-14T15:24:14.968 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:24:14.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.967+0000 7fdc0bf3d700 10 client.47837.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:24:14.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.967+0000 7fdc0bf3d700 15 client.47837.objecter _session_command_op_assign -1 13 2024-04-14T15:24:14.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.967+0000 7fdc0bf3d700 20 client.47837.objecter _get_session s=0x7fdbe80232e0 osd=6 3 2024-04-14T15:24:14.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.967+0000 7fdc0bf3d700 20 client.47837.objecter put_session s=0x7fdbe80232e0 osd=6 3 2024-04-14T15:24:14.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.967+0000 7fdc0bf3d700 20 client.47837.objecter _get_session s=0x7fdbe80232e0 osd=6 3 2024-04-14T15:24:14.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.967+0000 7fdc0bf3d700 15 client.47837.objecter _session_command_op_remove -1 13 2024-04-14T15:24:14.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.967+0000 7fdc0bf3d700 20 client.47837.objecter get_session s=0x7fdbe80232e0 osd=6 3 2024-04-14T15:24:14.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.967+0000 7fdc0bf3d700 15 client.47837.objecter _session_command_op_assign 6 13 2024-04-14T15:24:14.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.967+0000 7fdc0bf3d700 20 client.47837.objecter put_session s=0x7fdbe80232e0 osd=6 4 2024-04-14T15:24:14.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.967+0000 7fdc0bf3d700 10 client.47837.objecter _send_command 13 2024-04-14T15:24:14.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.967+0000 7fdc0bf3d700 1 -- 172.21.15.90:0/4179733851 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fdbe8024560 con 0x7fdbe80251e0 2024-04-14T15:24:14.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.967+0000 7fdbfaffd700 10 client.47837.objecter ms_handle_connect 0x7fdbe80251e0 2024-04-14T15:24:14.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.968+0000 7fdbfaffd700 1 -- 172.21.15.90:0/4179733851 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fdbe8024560 con 0x7fdbe80251e0 2024-04-14T15:24:14.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.968+0000 7fdbfaffd700 10 client.47837.objecter ms_dispatch 0x7fdc04001550 command_reply(tid 13: 0 ) 2024-04-14T15:24:14.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.968+0000 7fdbfaffd700 10 client.47837.objecter _finish_command 13 = system:0 2024-04-14T15:24:14.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.968+0000 7fdbfaffd700 20 client.47837.objecter put_session s=0x7fdbe80232e0 osd=6 4 2024-04-14T15:24:14.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.968+0000 7fdbfaffd700 15 client.47837.objecter _session_command_op_remove 6 13 2024-04-14T15:24:14.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.985+0000 7fdc0bf3d700 10 client.47837.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:24:14.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.985+0000 7fdc0bf3d700 15 client.47837.objecter _session_command_op_assign -1 14 2024-04-14T15:24:14.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.985+0000 7fdc0bf3d700 20 client.47837.objecter _get_session s=0x7fdbe80232e0 osd=6 3 2024-04-14T15:24:14.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.985+0000 7fdc0bf3d700 20 client.47837.objecter put_session s=0x7fdbe80232e0 osd=6 3 2024-04-14T15:24:14.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.985+0000 7fdc0bf3d700 20 client.47837.objecter _get_session s=0x7fdbe80232e0 osd=6 3 2024-04-14T15:24:14.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.985+0000 7fdc0bf3d700 15 client.47837.objecter _session_command_op_remove -1 14 2024-04-14T15:24:14.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.985+0000 7fdc0bf3d700 20 client.47837.objecter get_session s=0x7fdbe80232e0 osd=6 3 2024-04-14T15:24:14.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.985+0000 7fdc0bf3d700 15 client.47837.objecter _session_command_op_assign 6 14 2024-04-14T15:24:14.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.985+0000 7fdc0bf3d700 20 client.47837.objecter put_session s=0x7fdbe80232e0 osd=6 4 2024-04-14T15:24:14.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.985+0000 7fdc0bf3d700 10 client.47837.objecter _send_command 14 2024-04-14T15:24:14.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.985+0000 7fdc0bf3d700 1 -- 172.21.15.90:0/4179733851 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fdbe8024560 con 0x7fdbe80251e0 2024-04-14T15:24:14.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.987+0000 7fdbfaffd700 1 -- 172.21.15.90:0/4179733851 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fdbe8024560 con 0x7fdbe80251e0 2024-04-14T15:24:14.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.987+0000 7fdbfaffd700 10 client.47837.objecter ms_dispatch 0x7fdc04001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:24:14.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.987+0000 7fdbfaffd700 10 client.47837.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:24:14.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.987+0000 7fdbfaffd700 20 client.47837.objecter put_session s=0x7fdbe80232e0 osd=6 4 2024-04-14T15:24:14.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.987+0000 7fdbfaffd700 15 client.47837.objecter _session_command_op_remove 6 14 2024-04-14T15:24:14.988 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:24:14.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.987+0000 7fdc0bf3d700 10 client.47837.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:24:14.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.987+0000 7fdc0bf3d700 15 client.47837.objecter _session_command_op_assign -1 15 2024-04-14T15:24:14.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.987+0000 7fdc0bf3d700 20 client.47837.objecter _get_session s=0x7fdbe8024560 osd=7 3 2024-04-14T15:24:14.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.987+0000 7fdc0bf3d700 20 client.47837.objecter put_session s=0x7fdbe8024560 osd=7 3 2024-04-14T15:24:14.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.987+0000 7fdc0bf3d700 20 client.47837.objecter _get_session s=0x7fdbe8024560 osd=7 3 2024-04-14T15:24:14.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.987+0000 7fdc0bf3d700 15 client.47837.objecter _session_command_op_remove -1 15 2024-04-14T15:24:14.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.987+0000 7fdc0bf3d700 20 client.47837.objecter get_session s=0x7fdbe8024560 osd=7 3 2024-04-14T15:24:14.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.987+0000 7fdc0bf3d700 15 client.47837.objecter _session_command_op_assign 7 15 2024-04-14T15:24:14.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.987+0000 7fdc0bf3d700 20 client.47837.objecter put_session s=0x7fdbe8024560 osd=7 4 2024-04-14T15:24:14.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.987+0000 7fdc0bf3d700 10 client.47837.objecter _send_command 15 2024-04-14T15:24:14.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.987+0000 7fdc0bf3d700 1 -- 172.21.15.90:0/4179733851 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fdbe802bc00 con 0x7fdbe80247e0 2024-04-14T15:24:14.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.988+0000 7fdbfaffd700 10 client.47837.objecter ms_handle_connect 0x7fdbe80247e0 2024-04-14T15:24:14.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.989+0000 7fdbfaffd700 1 -- 172.21.15.90:0/4179733851 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fdbe802bc00 con 0x7fdbe80247e0 2024-04-14T15:24:14.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.989+0000 7fdbfaffd700 10 client.47837.objecter ms_dispatch 0x7fdc04001550 command_reply(tid 15: 0 ) 2024-04-14T15:24:14.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.989+0000 7fdbfaffd700 10 client.47837.objecter _finish_command 15 = system:0 2024-04-14T15:24:14.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.989+0000 7fdbfaffd700 20 client.47837.objecter put_session s=0x7fdbe8024560 osd=7 4 2024-04-14T15:24:14.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:14.989+0000 7fdbfaffd700 15 client.47837.objecter _session_command_op_remove 7 15 2024-04-14T15:24:15.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.005+0000 7fdc0bf3d700 10 client.47837.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:24:15.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.005+0000 7fdc0bf3d700 15 client.47837.objecter _session_command_op_assign -1 16 2024-04-14T15:24:15.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.005+0000 7fdc0bf3d700 20 client.47837.objecter _get_session s=0x7fdbe8024560 osd=7 3 2024-04-14T15:24:15.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.005+0000 7fdc0bf3d700 20 client.47837.objecter put_session s=0x7fdbe8024560 osd=7 3 2024-04-14T15:24:15.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.005+0000 7fdc0bf3d700 20 client.47837.objecter _get_session s=0x7fdbe8024560 osd=7 3 2024-04-14T15:24:15.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.005+0000 7fdc0bf3d700 15 client.47837.objecter _session_command_op_remove -1 16 2024-04-14T15:24:15.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.005+0000 7fdc0bf3d700 20 client.47837.objecter get_session s=0x7fdbe8024560 osd=7 3 2024-04-14T15:24:15.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.005+0000 7fdc0bf3d700 15 client.47837.objecter _session_command_op_assign 7 16 2024-04-14T15:24:15.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.005+0000 7fdc0bf3d700 20 client.47837.objecter put_session s=0x7fdbe8024560 osd=7 4 2024-04-14T15:24:15.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.005+0000 7fdc0bf3d700 10 client.47837.objecter _send_command 16 2024-04-14T15:24:15.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.005+0000 7fdc0bf3d700 1 -- 172.21.15.90:0/4179733851 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fdbe802bde0 con 0x7fdbe80247e0 2024-04-14T15:24:15.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.007+0000 7fdbfaffd700 1 -- 172.21.15.90:0/4179733851 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fdbe802bde0 con 0x7fdbe80247e0 2024-04-14T15:24:15.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.007+0000 7fdbfaffd700 10 client.47837.objecter ms_dispatch 0x7fdc04001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:24:15.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.007+0000 7fdbfaffd700 10 client.47837.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:24:15.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.007+0000 7fdbfaffd700 20 client.47837.objecter put_session s=0x7fdbe8024560 osd=7 4 2024-04-14T15:24:15.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.007+0000 7fdbfaffd700 15 client.47837.objecter _session_command_op_remove 7 16 2024-04-14T15:24:15.008 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:24:15.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.008+0000 7fdc0bf3d700 10 librados: watch_flush enter 2024-04-14T15:24:15.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.008+0000 7fdc0bf3d700 10 librados: watch_flush exit 2024-04-14T15:24:15.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.008+0000 7fdc0bf3d700 10 client.47837.objecter close_session for osd.0 2024-04-14T15:24:15.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.008+0000 7fdc0bf3d700 1 -- 172.21.15.90:0/4179733851 >> v1:172.21.15.90:6801/3761806885 conn(0x7fdbe8004730 legacy=0x7fdbe8002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:15.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.008+0000 7fdc0bf3d700 20 client.47837.objecter put_session s=0x7fdbe80053e0 osd=0 1 2024-04-14T15:24:15.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.008+0000 7fdc0bf3d700 10 client.47837.objecter close_session for osd.1 2024-04-14T15:24:15.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.008+0000 7fdc0bf3d700 1 -- 172.21.15.90:0/4179733851 >> v1:172.21.15.90:6805/4170361699 conn(0x7fdbe8002100 legacy=0x7fdbe800f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:15.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.008+0000 7fdc0bf3d700 20 client.47837.objecter put_session s=0x7fdbe80040f0 osd=1 1 2024-04-14T15:24:15.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.008+0000 7fdc0bf3d700 10 client.47837.objecter close_session for osd.2 2024-04-14T15:24:15.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.008+0000 7fdc0bf3d700 1 -- 172.21.15.90:0/4179733851 >> v1:172.21.15.90:6809/1686184251 conn(0x7fdbe80119e0 legacy=0x7fdbe8013dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:15.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.008+0000 7fdc0bf3d700 20 client.47837.objecter put_session s=0x7fdbe800c780 osd=2 1 2024-04-14T15:24:15.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.008+0000 7fdc0bf3d700 10 client.47837.objecter close_session for osd.3 2024-04-14T15:24:15.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.008+0000 7fdc0bf3d700 1 -- 172.21.15.90:0/4179733851 >> v1:172.21.15.90:6813/2270166774 conn(0x7fdbe8016820 legacy=0x7fdbe8016ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:15.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.008+0000 7fdc0bf3d700 20 client.47837.objecter put_session s=0x7fdbe800c4d0 osd=3 1 2024-04-14T15:24:15.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.008+0000 7fdc0bf3d700 10 client.47837.objecter close_session for osd.4 2024-04-14T15:24:15.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.008+0000 7fdc0bf3d700 1 -- 172.21.15.90:0/4179733851 >> v1:172.21.15.92:6800/3887549733 conn(0x7fdbe801b540 legacy=0x7fdbe801d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:15.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.008+0000 7fdc0bf3d700 20 client.47837.objecter put_session s=0x7fdbe8017bc0 osd=4 1 2024-04-14T15:24:15.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.008+0000 7fdc0bf3d700 10 client.47837.objecter close_session for osd.5 2024-04-14T15:24:15.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.008+0000 7fdc0bf3d700 1 -- 172.21.15.90:0/4179733851 >> v1:172.21.15.92:6804/3244561440 conn(0x7fdbe801f5c0 legacy=0x7fdbe80219f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:15.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.008+0000 7fdc0bf3d700 20 client.47837.objecter put_session s=0x7fdbe8010f70 osd=5 1 2024-04-14T15:24:15.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.008+0000 7fdc0bf3d700 10 client.47837.objecter close_session for osd.6 2024-04-14T15:24:15.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.008+0000 7fdc0bf3d700 1 -- 172.21.15.90:0/4179733851 >> v1:172.21.15.92:6808/636125963 conn(0x7fdbe80251e0 legacy=0x7fdbe8025640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:15.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.008+0000 7fdc0bf3d700 20 client.47837.objecter put_session s=0x7fdbe80232e0 osd=6 1 2024-04-14T15:24:15.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.008+0000 7fdc0bf3d700 10 client.47837.objecter close_session for osd.7 2024-04-14T15:24:15.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.008+0000 7fdc0bf3d700 1 -- 172.21.15.90:0/4179733851 >> v1:172.21.15.92:6812/1773706716 conn(0x7fdbe80247e0 legacy=0x7fdbe802a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:15.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.008+0000 7fdc0bf3d700 20 client.47837.objecter put_session s=0x7fdbe8024560 osd=7 1 2024-04-14T15:24:15.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.008+0000 7fdc0bf3d700 20 client.47837.objecter shutdown clearing up homeless session... 2024-04-14T15:24:15.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.008+0000 7fdc0bf3d700 10 client.47837.objecter successfully canceled tick 2024-04-14T15:24:15.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.008+0000 7fdc0a4da700 1 -- 172.21.15.90:0/4179733851 reap_dead start 2024-04-14T15:24:15.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.008+0000 7fdc0a4da700 1 -- 172.21.15.90:0/4179733851 reap_dead start 2024-04-14T15:24:15.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.008+0000 7fdc0bf3d700 1 -- 172.21.15.90:0/4179733851 >> v1:172.21.15.90:6800/420935427 conn(0x7fdbf00712e0 legacy=0x7fdbf00737a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:15.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.008+0000 7fdc0bf3d700 1 -- 172.21.15.90:0/4179733851 >> v1:172.21.15.90:6790/0 conn(0x7fdc0410f440 legacy=0x7fdc041a68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:15.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.008+0000 7fdc0a4da700 1 -- 172.21.15.90:0/4179733851 reap_dead start 2024-04-14T15:24:15.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.008+0000 7fdc0a4da700 1 -- 172.21.15.90:0/4179733851 reap_dead start 2024-04-14T15:24:15.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.008+0000 7fdc0a4da700 1 -- 172.21.15.90:0/4179733851 reap_dead start 2024-04-14T15:24:15.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.008+0000 7fdc0bf3d700 1 -- 172.21.15.90:0/4179733851 shutdown_connections 2024-04-14T15:24:15.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.008+0000 7fdc0bf3d700 1 -- 172.21.15.90:0/4179733851 >> 172.21.15.90:0/4179733851 conn(0x7fdc040fd290 msgr2=0x7fdc0410f940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:24:15.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.008+0000 7fdc0bf3d700 1 -- 172.21.15.90:0/4179733851 shutdown_connections 2024-04-14T15:24:15.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.008+0000 7fdc0bf3d700 1 -- 172.21.15.90:0/4179733851 wait complete. 2024-04-14T15:24:15.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.009+0000 7fdc0bf3d700 1 librados: shutdown 2024-04-14T15:24:15.012 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:24:15.012 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:24:15.012 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:24:15.012 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:24:15.012 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:24:15.013 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:24:15.013 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:24:15.013 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:24:15.067 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:24:15.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.087+0000 7fe4767fc700 1 -- 172.21.15.90:0/639231086 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v713) ==== 91+0+0 (unknown 434836466 0 0) 0x7fe47805b500 con 0x7fe48010f2f0 2024-04-14T15:24:15.093 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:15.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.090+0000 7fe487460700 10 librados: watch_flush enter 2024-04-14T15:24:15.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.090+0000 7fe487460700 10 librados: watch_flush exit 2024-04-14T15:24:15.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.090+0000 7fe487460700 20 client.37881.objecter shutdown clearing up homeless session... 2024-04-14T15:24:15.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.090+0000 7fe487460700 10 client.37881.objecter successfully canceled tick 2024-04-14T15:24:15.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.090+0000 7fe487460700 1 -- 172.21.15.90:0/639231086 >> v1:172.21.15.90:6800/420935427 conn(0x7fe46c071220 legacy=0x7fe46c0736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:15.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.090+0000 7fe487460700 1 -- 172.21.15.90:0/639231086 >> v1:172.21.15.90:6789/0 conn(0x7fe48010f2f0 legacy=0x7fe4801a82c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:15.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.090+0000 7fe487460700 1 -- 172.21.15.90:0/639231086 shutdown_connections 2024-04-14T15:24:15.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.090+0000 7fe487460700 1 -- 172.21.15.90:0/639231086 >> 172.21.15.90:0/639231086 conn(0x7fe4800fe3a0 msgr2=0x7fe480111fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:24:15.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.092+0000 7fe487460700 1 -- 172.21.15.90:0/639231086 shutdown_connections 2024-04-14T15:24:15.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.092+0000 7fe487460700 1 -- 172.21.15.90:0/639231086 wait complete. 2024-04-14T15:24:15.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:15.092+0000 7fe487460700 1 librados: shutdown 2024-04-14T15:24:15.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:15.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:24:15.096 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is unset 2024-04-14T15:24:15.196 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:24:15.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:15.221 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:15.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:24:15.321 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:24:15.348 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:15.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:15.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:24:15.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:15 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-124"}]': finished 2024-04-14T15:24:15.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:15 smithi092 ceph-mon[33966]: osdmap e712: 8 total, 8 up, 7 in 2024-04-14T15:24:15.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:15 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-124", "mode": "writeback"}]: dispatch 2024-04-14T15:24:15.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:15 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-124", "mode": "writeback"}]: dispatch 2024-04-14T15:24:15.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:15 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/639231086' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:24:15.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:15 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/4179733851' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:24:15.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:15 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-124"}]': finished 2024-04-14T15:24:15.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:15 smithi090 ceph-mon[41670]: osdmap e712: 8 total, 8 up, 7 in 2024-04-14T15:24:15.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:15 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-124", "mode": "writeback"}]: dispatch 2024-04-14T15:24:15.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:15 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-124", "mode": "writeback"}]: dispatch 2024-04-14T15:24:15.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:15 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/639231086' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:24:15.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:15 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/4179733851' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:24:15.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:15 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-124"}]': finished 2024-04-14T15:24:15.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:15 smithi090 ceph-mon[30399]: osdmap e712: 8 total, 8 up, 7 in 2024-04-14T15:24:15.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:15 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-124", "mode": "writeback"}]: dispatch 2024-04-14T15:24:15.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:15 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-124", "mode": "writeback"}]: dispatch 2024-04-14T15:24:15.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:15 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/639231086' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:24:15.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:15 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/4179733851' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:24:15.451 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:24:15.478 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:15.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:15.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:24:15.580 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:24:15.609 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:15.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:15.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:24:15.711 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:24:15.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:15.736 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:15.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:24:15.837 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:24:15.862 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:15.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:15.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:24:15.872 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:24:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:24:15.964 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:24:15.987 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:15.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:15.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:24:16.089 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:24:16.113 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:16.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:16.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:24:16.215 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:24:16.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.230+0000 7f29d535a700 1 -- 172.21.15.90:0/3411459483 >> v1:172.21.15.90:6789/0 conn(0x7f29d01012c0 legacy=0x7f29d01016a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:16.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.230+0000 7f29d535a700 1 -- 172.21.15.90:0/3411459483 shutdown_connections 2024-04-14T15:24:16.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.230+0000 7f29d535a700 1 -- 172.21.15.90:0/3411459483 >> 172.21.15.90:0/3411459483 conn(0x7f29d0076a70 msgr2=0x7f29d0076e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:24:16.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.230+0000 7f29d535a700 1 -- 172.21.15.90:0/3411459483 shutdown_connections 2024-04-14T15:24:16.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.230+0000 7f29d535a700 1 -- 172.21.15.90:0/3411459483 wait complete. 2024-04-14T15:24:16.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.231+0000 7f29d535a700 1 librados: starting msgr at 2024-04-14T15:24:16.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.231+0000 7f29d535a700 1 librados: starting objecter 2024-04-14T15:24:16.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.231+0000 7f29d535a700 1 Processor -- start 2024-04-14T15:24:16.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.231+0000 7f29d535a700 1 -- start start 2024-04-14T15:24:16.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.231+0000 7f29d535a700 1 librados: setting wanted keys 2024-04-14T15:24:16.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.231+0000 7f29d535a700 1 librados: calling monclient init 2024-04-14T15:24:16.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.231+0000 7f29d535a700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f29d01a9e10 con 0x7f29d0109e40 2024-04-14T15:24:16.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.231+0000 7f29d535a700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f29d01ab050 con 0x7f29d010d9e0 2024-04-14T15:24:16.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.231+0000 7f29d535a700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f29d01ac290 con 0x7f29d01012c0 2024-04-14T15:24:16.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.231+0000 7f29ce7fc700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f29d0109e40 0x7f29d0105970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:45608/0 (socket says 172.21.15.90:45608) 2024-04-14T15:24:16.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.231+0000 7f29ce7fc700 1 -- 172.21.15.90:0/3785313381 learned_addr learned my addr 172.21.15.90:0/3785313381 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:24:16.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.231+0000 7f29b7fff700 10 client.?.objecter ms_handle_connect 0x7f29d01012c0 2024-04-14T15:24:16.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.231+0000 7f29b7fff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:24:16.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.231+0000 7f29b7fff700 10 client.?.objecter ms_handle_connect 0x7f29d010d9e0 2024-04-14T15:24:16.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.231+0000 7f29b7fff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:24:16.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.231+0000 7f29b7fff700 10 client.?.objecter ms_handle_connect 0x7f29d0109e40 2024-04-14T15:24:16.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.231+0000 7f29b7fff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:24:16.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.232+0000 7f29b7fff700 1 -- 172.21.15.90:0/3785313381 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2646214193 0 0) 0x7f29d01ab050 con 0x7f29d010d9e0 2024-04-14T15:24:16.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.232+0000 7f29b7fff700 1 -- 172.21.15.90:0/3785313381 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29b80046c0 con 0x7f29d010d9e0 2024-04-14T15:24:16.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.232+0000 7f29b7fff700 1 -- 172.21.15.90:0/3785313381 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1434849825 0 0) 0x7f29d01ac290 con 0x7f29d01012c0 2024-04-14T15:24:16.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.232+0000 7f29b7fff700 1 -- 172.21.15.90:0/3785313381 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29d01ab050 con 0x7f29d01012c0 2024-04-14T15:24:16.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.232+0000 7f29b7fff700 1 -- 172.21.15.90:0/3785313381 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 895718026 0 0) 0x7f29d01a9e10 con 0x7f29d0109e40 2024-04-14T15:24:16.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.232+0000 7f29b7fff700 1 -- 172.21.15.90:0/3785313381 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29d01ac290 con 0x7f29d0109e40 2024-04-14T15:24:16.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.232+0000 7f29b7fff700 1 -- 172.21.15.90:0/3785313381 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4042530423 0 0) 0x7f29b80046c0 con 0x7f29d010d9e0 2024-04-14T15:24:16.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.232+0000 7f29b7fff700 1 -- 172.21.15.90:0/3785313381 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f29d01a9e10 con 0x7f29d010d9e0 2024-04-14T15:24:16.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.232+0000 7f29b7fff700 1 -- 172.21.15.90:0/3785313381 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f29c8001c30 con 0x7f29d010d9e0 2024-04-14T15:24:16.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.232+0000 7f29b7fff700 1 -- 172.21.15.90:0/3785313381 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 944137937 0 0) 0x7f29d01ab050 con 0x7f29d01012c0 2024-04-14T15:24:16.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.232+0000 7f29b7fff700 1 -- 172.21.15.90:0/3785313381 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f29b80046c0 con 0x7f29d01012c0 2024-04-14T15:24:16.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.232+0000 7f29b7fff700 1 -- 172.21.15.90:0/3785313381 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f29c00015d0 con 0x7f29d01012c0 2024-04-14T15:24:16.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.232+0000 7f29b7fff700 1 -- 172.21.15.90:0/3785313381 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3515458823 0 0) 0x7f29d01ac290 con 0x7f29d0109e40 2024-04-14T15:24:16.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.232+0000 7f29b7fff700 1 -- 172.21.15.90:0/3785313381 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f29d01ab050 con 0x7f29d0109e40 2024-04-14T15:24:16.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.232+0000 7f29b7fff700 1 -- 172.21.15.90:0/3785313381 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 801883882 0 0) 0x7f29d01a9e10 con 0x7f29d010d9e0 2024-04-14T15:24:16.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.232+0000 7f29b7fff700 1 -- 172.21.15.90:0/3785313381 >> v1:172.21.15.90:6790/0 conn(0x7f29d01012c0 legacy=0x7f29d0105260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:16.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.232+0000 7f29b7fff700 1 -- 172.21.15.90:0/3785313381 >> v1:172.21.15.90:6789/0 conn(0x7f29d0109e40 legacy=0x7f29d0105970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:16.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.232+0000 7f29b7fff700 1 -- 172.21.15.90:0/3785313381 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29d01ad4d0 con 0x7f29d010d9e0 2024-04-14T15:24:16.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.232+0000 7f29d535a700 1 -- 172.21.15.90:0/3785313381 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f29d01ab280 con 0x7f29d010d9e0 2024-04-14T15:24:16.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.232+0000 7f29d535a700 10 client.49723.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:24:16.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.232+0000 7f29d535a700 1 -- 172.21.15.90:0/3785313381 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f29d01ab740 con 0x7f29d010d9e0 2024-04-14T15:24:16.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.232+0000 7f29d535a700 1 librados: init done 2024-04-14T15:24:16.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.232+0000 7f29b7fff700 1 -- 172.21.15.90:0/3785313381 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f29c8003ec0 con 0x7f29d010d9e0 2024-04-14T15:24:16.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.232+0000 7f29b7fff700 1 -- 172.21.15.90:0/3785313381 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f29c8004e00 con 0x7f29d010d9e0 2024-04-14T15:24:16.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.233+0000 7f29d535a700 1 -- 172.21.15.90:0/3785313381 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29bc005150 con 0x7f29d010d9e0 2024-04-14T15:24:16.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.234+0000 7f29b7fff700 1 -- 172.21.15.90:0/3785313381 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f29c8003a20 con 0x7f29d010d9e0 2024-04-14T15:24:16.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.234+0000 7f29b7fff700 1 -- 172.21.15.90:0/3785313381 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(714..714 src has 467..714) ==== 8841+0+0 (unknown 3912123001 0 0) 0x7f29c8089da0 con 0x7f29d010d9e0 2024-04-14T15:24:16.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.234+0000 7f29b7fff700 10 client.49723.objecter ms_dispatch 0x7f29d0001550 osd_map(714..714 src has 467..714) 2024-04-14T15:24:16.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.234+0000 7f29b7fff700 3 client.49723.objecter handle_osd_map got epochs [714,714] > 0 2024-04-14T15:24:16.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.234+0000 7f29b7fff700 3 client.49723.objecter handle_osd_map decoding full epoch 714 2024-04-14T15:24:16.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.234+0000 7f29b7fff700 20 client.49723.objecter dump_active .. 0 homeless 2024-04-14T15:24:16.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.234+0000 7f29b7fff700 10 client.49723.objecter ms_handle_connect 0x7f29b8071290 2024-04-14T15:24:16.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.239+0000 7f29b7fff700 1 -- 172.21.15.90:0/3785313381 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f29c8056500 con 0x7f29d010d9e0 2024-04-14T15:24:16.239 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:16.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:16.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:24:16.341 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:24:16.366 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:16.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:16.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:24:16.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:16 smithi092 ceph-mon[33966]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:24:16.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:16 smithi092 ceph-mon[33966]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:24:16.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:16 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-124", "mode": "writeback"}]': finished 2024-04-14T15:24:16.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:16 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/639231086' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-14T15:24:16.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:16 smithi092 ceph-mon[33966]: osdmap e713: 8 total, 8 up, 7 in 2024-04-14T15:24:16.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:16 smithi092 ceph-mon[33966]: pgmap v1053: 214 pgs: 214 active+clean; 583 KiB data, 661 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-04-14T15:24:16.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:16 smithi092 ceph-mon[33966]: 323.2s0 scrub starts 2024-04-14T15:24:16.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:16 smithi092 ceph-mon[33966]: 323.2 scrub ok 2024-04-14T15:24:16.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:16 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:24:16.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:16 smithi092 ceph-mon[33966]: 4.13 scrub starts 2024-04-14T15:24:16.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:16 smithi092 ceph-mon[33966]: 4.13 scrub ok 2024-04-14T15:24:16.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:16 smithi092 ceph-mon[33966]: osdmap e714: 8 total, 8 up, 7 in 2024-04-14T15:24:16.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:16 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:24:16.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:16 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:24:16.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.436+0000 7f29d535a700 1 -- 172.21.15.90:0/3785313381 --> v1:172.21.15.90:6800/420935427 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f29bc000bc0 con 0x7f29b8071290 2024-04-14T15:24:16.437 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:16 smithi090 ceph-mon[30399]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:24:16.437 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:16 smithi090 ceph-mon[30399]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:24:16.437 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:16 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-124", "mode": "writeback"}]': finished 2024-04-14T15:24:16.437 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:16 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/639231086' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-14T15:24:16.437 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:16 smithi090 ceph-mon[30399]: osdmap e713: 8 total, 8 up, 7 in 2024-04-14T15:24:16.437 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:16 smithi090 ceph-mon[30399]: pgmap v1053: 214 pgs: 214 active+clean; 583 KiB data, 661 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-04-14T15:24:16.437 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:16 smithi090 ceph-mon[30399]: 323.2s0 scrub starts 2024-04-14T15:24:16.437 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:16 smithi090 ceph-mon[30399]: 323.2 scrub ok 2024-04-14T15:24:16.437 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:16 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:24:16.437 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:16 smithi090 ceph-mon[30399]: 4.13 scrub starts 2024-04-14T15:24:16.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:16 smithi090 ceph-mon[30399]: 4.13 scrub ok 2024-04-14T15:24:16.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:16 smithi090 ceph-mon[30399]: osdmap e714: 8 total, 8 up, 7 in 2024-04-14T15:24:16.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:16 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:24:16.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:16 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:24:16.438 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:16 smithi090 ceph-mon[41670]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:24:16.438 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:16 smithi090 ceph-mon[41670]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:24:16.438 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:16 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-124", "mode": "writeback"}]': finished 2024-04-14T15:24:16.438 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:16 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/639231086' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-14T15:24:16.438 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:16 smithi090 ceph-mon[41670]: osdmap e713: 8 total, 8 up, 7 in 2024-04-14T15:24:16.438 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:16 smithi090 ceph-mon[41670]: pgmap v1053: 214 pgs: 214 active+clean; 583 KiB data, 661 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-04-14T15:24:16.438 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:16 smithi090 ceph-mon[41670]: 323.2s0 scrub starts 2024-04-14T15:24:16.438 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:16 smithi090 ceph-mon[41670]: 323.2 scrub ok 2024-04-14T15:24:16.438 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:16 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:24:16.439 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:16 smithi090 ceph-mon[41670]: 4.13 scrub starts 2024-04-14T15:24:16.439 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:16 smithi090 ceph-mon[41670]: 4.13 scrub ok 2024-04-14T15:24:16.439 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:16 smithi090 ceph-mon[41670]: osdmap e714: 8 total, 8 up, 7 in 2024-04-14T15:24:16.439 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:16 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:24:16.439 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:16 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:24:16.444 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:24:16.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.452+0000 7f29b7fff700 1 -- 172.21.15.90:0/3785313381 <== mgr.14610 v1:172.21.15.90:6800/420935427 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+555022 (unknown 2965378022 0 3732685007) 0x7f29bc000bc0 con 0x7f29b8071290 2024-04-14T15:24:16.461 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:24:16.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.457+0000 7f29d535a700 10 librados: watch_flush enter 2024-04-14T15:24:16.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.457+0000 7f29d535a700 10 librados: watch_flush exit 2024-04-14T15:24:16.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.457+0000 7f29d535a700 20 client.49723.objecter shutdown clearing up homeless session... 2024-04-14T15:24:16.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.457+0000 7f29d535a700 10 client.49723.objecter successfully canceled tick 2024-04-14T15:24:16.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.457+0000 7f29d535a700 1 -- 172.21.15.90:0/3785313381 >> v1:172.21.15.90:6800/420935427 conn(0x7f29b8071290 legacy=0x7f29b8073750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:16.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.457+0000 7f29d535a700 1 -- 172.21.15.90:0/3785313381 >> v1:172.21.15.92:6789/0 conn(0x7f29d010d9e0 legacy=0x7f29d01a84d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:16.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.457+0000 7f29d535a700 1 -- 172.21.15.90:0/3785313381 shutdown_connections 2024-04-14T15:24:16.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.457+0000 7f29d535a700 1 -- 172.21.15.90:0/3785313381 >> 172.21.15.90:0/3785313381 conn(0x7f29d0076a70 msgr2=0x7f29d00fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:24:16.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.457+0000 7f29d535a700 1 -- 172.21.15.90:0/3785313381 shutdown_connections 2024-04-14T15:24:16.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.457+0000 7f29d535a700 1 -- 172.21.15.90:0/3785313381 wait complete. 2024-04-14T15:24:16.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:16.457+0000 7f29d535a700 1 librados: shutdown 2024-04-14T15:24:16.466 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-14T15:24:16.468 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:24:16.492 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:16.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:16.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:24:16.594 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:24:16.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:16.619 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:16.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:24:16.720 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:24:16.744 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:16.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:16.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:24:16.846 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:24:16.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:16.871 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:16.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:24:16.972 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:24:16.996 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:16.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:16.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:24:17.097 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:24:17.121 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:17.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:17.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:24:17.222 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:24:17.246 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:17.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:17.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:24:17.347 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:24:17.372 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:17.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:17.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:24:17.474 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:24:17.497 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:17.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:17.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:24:17.598 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:24:17.622 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:17.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:17.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:24:17.724 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:24:17.751 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:17.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:17.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:24:17.853 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:24:17.879 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:17.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:17.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:24:17.981 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:24:18.007 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:18.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:18.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:24:18.009 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:18 smithi090 ceph-mon[30399]: from='client.49723 v1:172.21.15.90:0/3785313381' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:24:18.009 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:18 smithi090 ceph-mon[30399]: 4.17 scrub starts 2024-04-14T15:24:18.009 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:18 smithi090 ceph-mon[30399]: 4.17 scrub ok 2024-04-14T15:24:18.009 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:18 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]': finished 2024-04-14T15:24:18.009 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:18 smithi090 ceph-mon[30399]: osdmap e715: 8 total, 8 up, 7 in 2024-04-14T15:24:18.009 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:18 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-124"}]: dispatch 2024-04-14T15:24:18.009 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:18 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-124"}]: dispatch 2024-04-14T15:24:18.109 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:24:18.137 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:18.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:18.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:24:18.238 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:24:18.265 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:18.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:18.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:24:18.367 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:24:18.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:18 smithi092 ceph-mon[33966]: from='client.49723 v1:172.21.15.90:0/3785313381' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:24:18.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:18 smithi092 ceph-mon[33966]: 4.17 scrub starts 2024-04-14T15:24:18.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:18 smithi092 ceph-mon[33966]: 4.17 scrub ok 2024-04-14T15:24:18.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:18 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]': finished 2024-04-14T15:24:18.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:18 smithi092 ceph-mon[33966]: osdmap e715: 8 total, 8 up, 7 in 2024-04-14T15:24:18.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:18 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-124"}]: dispatch 2024-04-14T15:24:18.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:18 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-124"}]: dispatch 2024-04-14T15:24:18.390 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:18.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:18.391 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:24:18.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:18 smithi090 ceph-mon[41670]: from='client.49723 v1:172.21.15.90:0/3785313381' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:24:18.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:18 smithi090 ceph-mon[41670]: 4.17 scrub starts 2024-04-14T15:24:18.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:18 smithi090 ceph-mon[41670]: 4.17 scrub ok 2024-04-14T15:24:18.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:18 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]': finished 2024-04-14T15:24:18.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:18 smithi090 ceph-mon[41670]: osdmap e715: 8 total, 8 up, 7 in 2024-04-14T15:24:18.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:18 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-124"}]: dispatch 2024-04-14T15:24:18.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:18 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-124"}]: dispatch 2024-04-14T15:24:18.492 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:24:18.505 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":1054,"stamp":"2024-04-14T15:24:15.991727+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596492,"num_objects":216,"num_object_clones":4,"num_object_copies":648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":216,"num_whiteouts":1,"num_read":2022,"num_read_kb":1766,"num_write":754,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":126,"num_bytes_recovered":921158,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":933,"ondisk_log_size":933,"up":642,"acting":642,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":618,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":676520,"kb_used_data":36212,"kb_used_omap":10,"kb_used_meta":640181,"kb_avail":655539544,"statfs":{"total":671965249536,"available":671272493056,"internally_reserved":0,"allocated":37081088,"data_stored":18521597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11156,"internal_metadata":655545452},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":47,"num_objects":8,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":5,"num_read_kb":5,"num_write":8,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.918744"},"pg_stats":[{"pgid":"330.1f","version":"0'0","reported_seq":16,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.321487+0000","last_change":"2024-04-14T15:24:11.083031+0000","last_active":"2024-04-14T15:24:13.321487+0000","last_peered":"2024-04-14T15:24:13.321487+0000","last_clean":"2024-04-14T15:24:13.321487+0000","last_became_active":"2024-04-14T15:24:11.082646+0000","last_became_peered":"2024-04-14T15:24:11.082646+0000","last_unstale":"2024-04-14T15:24:13.321487+0000","last_undegraded":"2024-04-14T15:24:13.321487+0000","last_fullsized":"2024-04-14T15:24:13.321487+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":708,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:10.051866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:10.051866+0000","last_clean_scrub_stamp":"2024-04-14T15:24:10.051866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:12.997451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"330.1e","version":"0'0","reported_seq":19,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.095113+0000","last_change":"2024-04-14T15:24:11.085437+0000","last_active":"2024-04-14T15:24:14.095113+0000","last_peered":"2024-04-14T15:24:14.095113+0000","last_clean":"2024-04-14T15:24:14.095113+0000","last_became_active":"2024-04-14T15:24:11.083679+0000","last_became_peered":"2024-04-14T15:24:11.083679+0000","last_unstale":"2024-04-14T15:24:14.095113+0000","last_undegraded":"2024-04-14T15:24:14.095113+0000","last_fullsized":"2024-04-14T15:24:14.095113+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":708,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:10.051866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:10.051866+0000","last_clean_scrub_stamp":"2024-04-14T15:24:10.051866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:17.094383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"330.1d","version":"0'0","reported_seq":16,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.320615+0000","last_change":"2024-04-14T15:24:11.083329+0000","last_active":"2024-04-14T15:24:13.320615+0000","last_peered":"2024-04-14T15:24:13.320615+0000","last_clean":"2024-04-14T15:24:13.320615+0000","last_became_active":"2024-04-14T15:24:11.082769+0000","last_became_peered":"2024-04-14T15:24:11.082769+0000","last_unstale":"2024-04-14T15:24:13.320615+0000","last_undegraded":"2024-04-14T15:24:13.320615+0000","last_fullsized":"2024-04-14T15:24:13.320615+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":708,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:10.051866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:10.051866+0000","last_clean_scrub_stamp":"2024-04-14T15:24:10.051866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:11.636880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"330.1c","version":"0'0","reported_seq":16,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.318720+0000","last_change":"2024-04-14T15:24:11.070787+0000","last_active":"2024-04-14T15:24:13.318720+0000","last_peered":"2024-04-14T15:24:13.318720+0000","last_clean":"2024-04-14T15:24:13.318720+0000","last_became_active":"2024-04-14T15:24:11.069810+0000","last_became_peered":"2024-04-14T15:24:11.069810+0000","last_unstale":"2024-04-14T15:24:13.318720+0000","last_undegraded":"2024-04-14T15:24:13.318720+0000","last_fullsized":"2024-04-14T15:24:13.318720+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":708,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:10.051866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:10.051866+0000","last_clean_scrub_stamp":"2024-04-14T15:24:10.051866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:10.955475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"330.1b","version":"0'0","reported_seq":16,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.317182+0000","last_change":"2024-04-14T15:24:11.083088+0000","last_active":"2024-04-14T15:24:13.317182+0000","last_peered":"2024-04-14T15:24:13.317182+0000","last_clean":"2024-04-14T15:24:13.317182+0000","last_became_active":"2024-04-14T15:24:11.082787+0000","last_became_peered":"2024-04-14T15:24:11.082787+0000","last_unstale":"2024-04-14T15:24:13.317182+0000","last_undegraded":"2024-04-14T15:24:13.317182+0000","last_fullsized":"2024-04-14T15:24:13.317182+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":708,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:10.051866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:10.051866+0000","last_clean_scrub_stamp":"2024-04-14T15:24:10.051866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:11.432753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"330.1a","version":"0'0","reported_seq":16,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.315852+0000","last_change":"2024-04-14T15:24:11.094584+0000","last_active":"2024-04-14T15:24:13.315852+0000","last_peered":"2024-04-14T15:24:13.315852+0000","last_clean":"2024-04-14T15:24:13.315852+0000","last_became_active":"2024-04-14T15:24:11.094034+0000","last_became_peered":"2024-04-14T15:24:11.094034+0000","last_unstale":"2024-04-14T15:24:13.315852+0000","last_undegraded":"2024-04-14T15:24:13.315852+0000","last_fullsized":"2024-04-14T15:24:13.315852+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":708,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:10.051866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:10.051866+0000","last_clean_scrub_stamp":"2024-04-14T15:24:10.051866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:21.174087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"330.19","version":"0'0","reported_seq":16,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.317732+0000","last_change":"2024-04-14T15:24:11.093026+0000","last_active":"2024-04-14T15:24:13.317732+0000","last_peered":"2024-04-14T15:24:13.317732+0000","last_clean":"2024-04-14T15:24:13.317732+0000","last_became_active":"2024-04-14T15:24:11.092346+0000","last_became_peered":"2024-04-14T15:24:11.092346+0000","last_unstale":"2024-04-14T15:24:13.317732+0000","last_undegraded":"2024-04-14T15:24:13.317732+0000","last_fullsized":"2024-04-14T15:24:13.317732+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":708,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:10.051866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:10.051866+0000","last_clean_scrub_stamp":"2024-04-14T15:24:10.051866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:36.721746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"330.18","version":"0'0","reported_seq":16,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.316988+0000","last_change":"2024-04-14T15:24:11.072771+0000","last_active":"2024-04-14T15:24:13.316988+0000","last_peered":"2024-04-14T15:24:13.316988+0000","last_clean":"2024-04-14T15:24:13.316988+0000","last_became_active":"2024-04-14T15:24:11.072288+0000","last_became_peered":"2024-04-14T15:24:11.072288+0000","last_unstale":"2024-04-14T15:24:13.316988+0000","last_undegraded":"2024-04-14T15:24:13.316988+0000","last_fullsized":"2024-04-14T15:24:13.316988+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":708,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:10.051866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:10.051866+0000","last_clean_scrub_stamp":"2024-04-14T15:24:10.051866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:25.590695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"330.17","version":"0'0","reported_seq":16,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.321493+0000","last_change":"2024-04-14T15:24:11.096596+0000","last_active":"2024-04-14T15:24:13.321493+0000","last_peered":"2024-04-14T15:24:13.321493+0000","last_clean":"2024-04-14T15:24:13.321493+0000","last_became_active":"2024-04-14T15:24:11.096202+0000","last_became_peered":"2024-04-14T15:24:11.096202+0000","last_unstale":"2024-04-14T15:24:13.321493+0000","last_undegraded":"2024-04-14T15:24:13.321493+0000","last_fullsized":"2024-04-14T15:24:13.321493+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":708,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:10.051866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:10.051866+0000","last_clean_scrub_stamp":"2024-04-14T15:24:10.051866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:16.015071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"330.16","version":"0'0","reported_seq":16,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.319528+0000","last_change":"2024-04-14T15:24:11.078006+0000","last_active":"2024-04-14T15:24:13.319528+0000","last_peered":"2024-04-14T15:24:13.319528+0000","last_clean":"2024-04-14T15:24:13.319528+0000","last_became_active":"2024-04-14T15:24:11.077682+0000","last_became_peered":"2024-04-14T15:24:11.077682+0000","last_unstale":"2024-04-14T15:24:13.319528+0000","last_undegraded":"2024-04-14T15:24:13.319528+0000","last_fullsized":"2024-04-14T15:24:13.319528+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":708,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:10.051866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:10.051866+0000","last_clean_scrub_stamp":"2024-04-14T15:24:10.051866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:35.858719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"330.15","version":"0'0","reported_seq":19,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.095028+0000","last_change":"2024-04-14T15:24:11.080203+0000","last_active":"2024-04-14T15:24:14.095028+0000","last_peered":"2024-04-14T15:24:14.095028+0000","last_clean":"2024-04-14T15:24:14.095028+0000","last_became_active":"2024-04-14T15:24:11.079931+0000","last_became_peered":"2024-04-14T15:24:11.079931+0000","last_unstale":"2024-04-14T15:24:14.095028+0000","last_undegraded":"2024-04-14T15:24:14.095028+0000","last_fullsized":"2024-04-14T15:24:14.095028+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":708,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:10.051866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:10.051866+0000","last_clean_scrub_stamp":"2024-04-14T15:24:10.051866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:13.534817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"330.14","version":"0'0","reported_seq":16,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.291134+0000","last_change":"2024-04-14T15:24:11.082269+0000","last_active":"2024-04-14T15:24:13.291134+0000","last_peered":"2024-04-14T15:24:13.291134+0000","last_clean":"2024-04-14T15:24:13.291134+0000","last_became_active":"2024-04-14T15:24:11.081472+0000","last_became_peered":"2024-04-14T15:24:11.081472+0000","last_unstale":"2024-04-14T15:24:13.291134+0000","last_undegraded":"2024-04-14T15:24:13.291134+0000","last_fullsized":"2024-04-14T15:24:13.291134+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":708,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:10.051866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:10.051866+0000","last_clean_scrub_stamp":"2024-04-14T15:24:10.051866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:32.660687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"330.13","version":"0'0","reported_seq":19,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.091201+0000","last_change":"2024-04-14T15:24:11.095181+0000","last_active":"2024-04-14T15:24:14.091201+0000","last_peered":"2024-04-14T15:24:14.091201+0000","last_clean":"2024-04-14T15:24:14.091201+0000","last_became_active":"2024-04-14T15:24:11.094884+0000","last_became_peered":"2024-04-14T15:24:11.094884+0000","last_unstale":"2024-04-14T15:24:14.091201+0000","last_undegraded":"2024-04-14T15:24:14.091201+0000","last_fullsized":"2024-04-14T15:24:14.091201+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":708,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:10.051866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:10.051866+0000","last_clean_scrub_stamp":"2024-04-14T15:24:10.051866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:25.038378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"330.12","version":"0'0","reported_seq":16,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.290857+0000","last_change":"2024-04-14T15:24:11.082701+0000","last_active":"2024-04-14T15:24:13.290857+0000","last_peered":"2024-04-14T15:24:13.290857+0000","last_clean":"2024-04-14T15:24:13.290857+0000","last_became_active":"2024-04-14T15:24:11.082429+0000","last_became_peered":"2024-04-14T15:24:11.082429+0000","last_unstale":"2024-04-14T15:24:13.290857+0000","last_undegraded":"2024-04-14T15:24:13.290857+0000","last_fullsized":"2024-04-14T15:24:13.290857+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":708,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:10.051866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:10.051866+0000","last_clean_scrub_stamp":"2024-04-14T15:24:10.051866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:29.744966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"330.11","version":"0'0","reported_seq":19,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.092688+0000","last_change":"2024-04-14T15:24:11.092447+0000","last_active":"2024-04-14T15:24:14.092688+0000","last_peered":"2024-04-14T15:24:14.092688+0000","last_clean":"2024-04-14T15:24:14.092688+0000","last_became_active":"2024-04-14T15:24:11.092108+0000","last_became_peered":"2024-04-14T15:24:11.092108+0000","last_unstale":"2024-04-14T15:24:14.092688+0000","last_undegraded":"2024-04-14T15:24:14.092688+0000","last_fullsized":"2024-04-14T15:24:14.092688+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":708,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:10.051866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:10.051866+0000","last_clean_scrub_stamp":"2024-04-14T15:24:10.051866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:22.440546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"330.10","version":"0'0","reported_seq":19,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.091077+0000","last_change":"2024-04-14T15:24:11.077816+0000","last_active":"2024-04-14T15:24:14.091077+0000","last_peered":"2024-04-14T15:24:14.091077+0000","last_clean":"2024-04-14T15:24:14.091077+0000","last_became_active":"2024-04-14T15:24:11.077522+0000","last_became_peered":"2024-04-14T15:24:11.077522+0000","last_unstale":"2024-04-14T15:24:14.091077+0000","last_undegraded":"2024-04-14T15:24:14.091077+0000","last_fullsized":"2024-04-14T15:24:14.091077+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":708,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:10.051866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:10.051866+0000","last_clean_scrub_stamp":"2024-04-14T15:24:10.051866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:13.341983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"330.7","version":"0'0","reported_seq":16,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.320465+0000","last_change":"2024-04-14T15:24:11.074505+0000","last_active":"2024-04-14T15:24:13.320465+0000","last_peered":"2024-04-14T15:24:13.320465+0000","last_clean":"2024-04-14T15:24:13.320465+0000","last_became_active":"2024-04-14T15:24:11.074135+0000","last_became_peered":"2024-04-14T15:24:11.074135+0000","last_unstale":"2024-04-14T15:24:13.320465+0000","last_undegraded":"2024-04-14T15:24:13.320465+0000","last_fullsized":"2024-04-14T15:24:13.320465+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":708,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:10.051866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:10.051866+0000","last_clean_scrub_stamp":"2024-04-14T15:24:10.051866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:31.350952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"330.6","version":"0'0","reported_seq":16,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.291618+0000","last_change":"2024-04-14T15:24:11.095352+0000","last_active":"2024-04-14T15:24:13.291618+0000","last_peered":"2024-04-14T15:24:13.291618+0000","last_clean":"2024-04-14T15:24:13.291618+0000","last_became_active":"2024-04-14T15:24:11.095074+0000","last_became_peered":"2024-04-14T15:24:11.095074+0000","last_unstale":"2024-04-14T15:24:13.291618+0000","last_undegraded":"2024-04-14T15:24:13.291618+0000","last_fullsized":"2024-04-14T15:24:13.291618+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":708,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:10.051866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:10.051866+0000","last_clean_scrub_stamp":"2024-04-14T15:24:10.051866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:27.086353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"330.5","version":"0'0","reported_seq":19,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.095051+0000","last_change":"2024-04-14T15:24:11.095164+0000","last_active":"2024-04-14T15:24:14.095051+0000","last_peered":"2024-04-14T15:24:14.095051+0000","last_clean":"2024-04-14T15:24:14.095051+0000","last_became_active":"2024-04-14T15:24:11.094769+0000","last_became_peered":"2024-04-14T15:24:11.094769+0000","last_unstale":"2024-04-14T15:24:14.095051+0000","last_undegraded":"2024-04-14T15:24:14.095051+0000","last_fullsized":"2024-04-14T15:24:14.095051+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":708,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:10.051866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:10.051866+0000","last_clean_scrub_stamp":"2024-04-14T15:24:10.051866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:23.862891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"330.4","version":"0'0","reported_seq":19,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.091727+0000","last_change":"2024-04-14T15:24:11.077733+0000","last_active":"2024-04-14T15:24:14.091727+0000","last_peered":"2024-04-14T15:24:14.091727+0000","last_clean":"2024-04-14T15:24:14.091727+0000","last_became_active":"2024-04-14T15:24:11.077361+0000","last_became_peered":"2024-04-14T15:24:11.077361+0000","last_unstale":"2024-04-14T15:24:14.091727+0000","last_undegraded":"2024-04-14T15:24:14.091727+0000","last_fullsized":"2024-04-14T15:24:14.091727+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":708,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:10.051866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:10.051866+0000","last_clean_scrub_stamp":"2024-04-14T15:24:10.051866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:30.119594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"330.3","version":"0'0","reported_seq":19,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.091779+0000","last_change":"2024-04-14T15:24:11.094798+0000","last_active":"2024-04-14T15:24:14.091779+0000","last_peered":"2024-04-14T15:24:14.091779+0000","last_clean":"2024-04-14T15:24:14.091779+0000","last_became_active":"2024-04-14T15:24:11.094469+0000","last_became_peered":"2024-04-14T15:24:11.094469+0000","last_unstale":"2024-04-14T15:24:14.091779+0000","last_undegraded":"2024-04-14T15:24:14.091779+0000","last_fullsized":"2024-04-14T15:24:14.091779+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":708,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:10.051866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:10.051866+0000","last_clean_scrub_stamp":"2024-04-14T15:24:10.051866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:19.829175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"330.2","version":"0'0","reported_seq":16,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.315922+0000","last_change":"2024-04-14T15:24:11.080667+0000","last_active":"2024-04-14T15:24:13.315922+0000","last_peered":"2024-04-14T15:24:13.315922+0000","last_clean":"2024-04-14T15:24:13.315922+0000","last_became_active":"2024-04-14T15:24:11.079765+0000","last_became_peered":"2024-04-14T15:24:11.079765+0000","last_unstale":"2024-04-14T15:24:13.315922+0000","last_undegraded":"2024-04-14T15:24:13.315922+0000","last_fullsized":"2024-04-14T15:24:13.315922+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":708,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:10.051866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:10.051866+0000","last_clean_scrub_stamp":"2024-04-14T15:24:10.051866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:19.972146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"330.1","version":"0'0","reported_seq":16,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.291773+0000","last_change":"2024-04-14T15:24:11.078611+0000","last_active":"2024-04-14T15:24:13.291773+0000","last_peered":"2024-04-14T15:24:13.291773+0000","last_clean":"2024-04-14T15:24:13.291773+0000","last_became_active":"2024-04-14T15:24:11.078342+0000","last_became_peered":"2024-04-14T15:24:11.078342+0000","last_unstale":"2024-04-14T15:24:13.291773+0000","last_undegraded":"2024-04-14T15:24:13.291773+0000","last_fullsized":"2024-04-14T15:24:13.291773+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":708,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:10.051866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:10.051866+0000","last_clean_scrub_stamp":"2024-04-14T15:24:10.051866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:37.640576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"330.0","version":"0'0","reported_seq":19,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.092898+0000","last_change":"2024-04-14T15:24:11.091981+0000","last_active":"2024-04-14T15:24:14.092898+0000","last_peered":"2024-04-14T15:24:14.092898+0000","last_clean":"2024-04-14T15:24:14.092898+0000","last_became_active":"2024-04-14T15:24:11.091664+0000","last_became_peered":"2024-04-14T15:24:11.091664+0000","last_unstale":"2024-04-14T15:24:14.092898+0000","last_undegraded":"2024-04-14T15:24:14.092898+0000","last_fullsized":"2024-04-14T15:24:14.092898+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":708,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:10.051866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:10.051866+0000","last_clean_scrub_stamp":"2024-04-14T15:24:10.051866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:26.518669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"330.e","version":"0'0","reported_seq":19,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.095825+0000","last_change":"2024-04-14T15:24:11.084582+0000","last_active":"2024-04-14T15:24:14.095825+0000","last_peered":"2024-04-14T15:24:14.095825+0000","last_clean":"2024-04-14T15:24:14.095825+0000","last_became_active":"2024-04-14T15:24:11.082497+0000","last_became_peered":"2024-04-14T15:24:11.082497+0000","last_unstale":"2024-04-14T15:24:14.095825+0000","last_undegraded":"2024-04-14T15:24:14.095825+0000","last_fullsized":"2024-04-14T15:24:14.095825+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":708,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:10.051866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:10.051866+0000","last_clean_scrub_stamp":"2024-04-14T15:24:10.051866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:24.293030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"323.7","version":"0'0","reported_seq":174,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.317051+0000","last_change":"2024-04-14T15:23:50.029854+0000","last_active":"2024-04-14T15:24:13.317051+0000","last_peered":"2024-04-14T15:24:13.317051+0000","last_clean":"2024-04-14T15:24:13.317051+0000","last_became_active":"2024-04-14T15:23:04.005793+0000","last_became_peered":"2024-04-14T15:23:04.005793+0000","last_unstale":"2024-04-14T15:24:13.317051+0000","last_undegraded":"2024-04-14T15:24:13.317051+0000","last_fullsized":"2024-04-14T15:24:13.317051+0000","mapping_epoch":647,"log_start":"0'0","ondisk_log_start":"0'0","created":647,"last_epoch_clean":648,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_clean_scrub_stamp":"2024-04-14T15:23:02.982695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093659699999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"330.f","version":"0'0","reported_seq":19,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.095760+0000","last_change":"2024-04-14T15:24:11.093939+0000","last_active":"2024-04-14T15:24:14.095760+0000","last_peered":"2024-04-14T15:24:14.095760+0000","last_clean":"2024-04-14T15:24:14.095760+0000","last_became_active":"2024-04-14T15:24:11.093609+0000","last_became_peered":"2024-04-14T15:24:11.093609+0000","last_unstale":"2024-04-14T15:24:14.095760+0000","last_undegraded":"2024-04-14T15:24:14.095760+0000","last_fullsized":"2024-04-14T15:24:14.095760+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":708,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:10.051866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:10.051866+0000","last_clean_scrub_stamp":"2024-04-14T15:24:10.051866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:36.283195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"323.6","version":"656'1","reported_seq":181,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.092244+0000","last_change":"2024-04-14T15:23:50.017981+0000","last_active":"2024-04-14T15:24:14.092244+0000","last_peered":"2024-04-14T15:24:14.092244+0000","last_clean":"2024-04-14T15:24:14.092244+0000","last_became_active":"2024-04-14T15:23:04.006098+0000","last_became_peered":"2024-04-14T15:23:04.006098+0000","last_unstale":"2024-04-14T15:24:14.092244+0000","last_undegraded":"2024-04-14T15:24:14.092244+0000","last_fullsized":"2024-04-14T15:24:14.092244+0000","mapping_epoch":647,"log_start":"0'0","ondisk_log_start":"0'0","created":647,"last_epoch_clean":648,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_clean_scrub_stamp":"2024-04-14T15:23:02.982695+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:30.591727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022213300000000001,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"330.c","version":"0'0","reported_seq":16,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.318881+0000","last_change":"2024-04-14T15:24:11.094617+0000","last_active":"2024-04-14T15:24:13.318881+0000","last_peered":"2024-04-14T15:24:13.318881+0000","last_clean":"2024-04-14T15:24:13.318881+0000","last_became_active":"2024-04-14T15:24:11.093975+0000","last_became_peered":"2024-04-14T15:24:11.093975+0000","last_unstale":"2024-04-14T15:24:13.318881+0000","last_undegraded":"2024-04-14T15:24:13.318881+0000","last_fullsized":"2024-04-14T15:24:13.318881+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":708,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:10.051866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:10.051866+0000","last_clean_scrub_stamp":"2024-04-14T15:24:10.051866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:31.812947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"323.5","version":"678'10","reported_seq":187,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.319162+0000","last_change":"2024-04-14T15:23:50.033380+0000","last_active":"2024-04-14T15:24:13.319162+0000","last_peered":"2024-04-14T15:24:13.319162+0000","last_clean":"2024-04-14T15:24:13.319162+0000","last_became_active":"2024-04-14T15:23:04.002370+0000","last_became_peered":"2024-04-14T15:23:04.002370+0000","last_unstale":"2024-04-14T15:24:13.319162+0000","last_undegraded":"2024-04-14T15:24:13.319162+0000","last_fullsized":"2024-04-14T15:24:13.319162+0000","mapping_epoch":647,"log_start":"0'0","ondisk_log_start":"0'0","created":647,"last_epoch_clean":648,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_clean_scrub_stamp":"2024-04-14T15:23:02.982695+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:20.542529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001357116,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"330.d","version":"0'0","reported_seq":16,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.318919+0000","last_change":"2024-04-14T15:24:11.094358+0000","last_active":"2024-04-14T15:24:13.318919+0000","last_peered":"2024-04-14T15:24:13.318919+0000","last_clean":"2024-04-14T15:24:13.318919+0000","last_became_active":"2024-04-14T15:24:11.094019+0000","last_became_peered":"2024-04-14T15:24:11.094019+0000","last_unstale":"2024-04-14T15:24:13.318919+0000","last_undegraded":"2024-04-14T15:24:13.318919+0000","last_fullsized":"2024-04-14T15:24:13.318919+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":708,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:10.051866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:10.051866+0000","last_clean_scrub_stamp":"2024-04-14T15:24:10.051866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:25.155998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"323.4","version":"710'8","reported_seq":182,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.291191+0000","last_change":"2024-04-14T15:23:50.024880+0000","last_active":"2024-04-14T15:24:13.291191+0000","last_peered":"2024-04-14T15:24:13.291191+0000","last_clean":"2024-04-14T15:24:13.291191+0000","last_became_active":"2024-04-14T15:23:03.997912+0000","last_became_peered":"2024-04-14T15:23:03.997912+0000","last_unstale":"2024-04-14T15:24:13.291191+0000","last_undegraded":"2024-04-14T15:24:13.291191+0000","last_fullsized":"2024-04-14T15:24:13.291191+0000","mapping_epoch":647,"log_start":"0'0","ondisk_log_start":"0'0","created":647,"last_epoch_clean":648,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_clean_scrub_stamp":"2024-04-14T15:23:02.982695+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:22.314690+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0047207309999999997,"stat_sum":{"num_bytes":18,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"330.a","version":"0'0","reported_seq":16,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.290891+0000","last_change":"2024-04-14T15:24:11.094593+0000","last_active":"2024-04-14T15:24:13.290891+0000","last_peered":"2024-04-14T15:24:13.290891+0000","last_clean":"2024-04-14T15:24:13.290891+0000","last_became_active":"2024-04-14T15:24:11.093749+0000","last_became_peered":"2024-04-14T15:24:11.093749+0000","last_unstale":"2024-04-14T15:24:13.290891+0000","last_undegraded":"2024-04-14T15:24:13.290891+0000","last_fullsized":"2024-04-14T15:24:13.290891+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":708,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:10.051866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:10.051866+0000","last_clean_scrub_stamp":"2024-04-14T15:24:10.051866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:16.294008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"323.3","version":"710'10","reported_seq":186,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.316028+0000","last_change":"2024-04-14T15:23:50.029109+0000","last_active":"2024-04-14T15:24:13.316028+0000","last_peered":"2024-04-14T15:24:13.316028+0000","last_clean":"2024-04-14T15:24:13.316028+0000","last_became_active":"2024-04-14T15:23:04.001213+0000","last_became_peered":"2024-04-14T15:23:04.001213+0000","last_unstale":"2024-04-14T15:24:13.316028+0000","last_undegraded":"2024-04-14T15:24:13.316028+0000","last_fullsized":"2024-04-14T15:24:13.316028+0000","mapping_epoch":647,"log_start":"0'0","ondisk_log_start":"0'0","created":647,"last_epoch_clean":648,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_clean_scrub_stamp":"2024-04-14T15:23:02.982695+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:31.678945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016731459999999999,"stat_sum":{"num_bytes":21,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":2,"num_read_kb":0,"num_write":6,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"330.b","version":"0'0","reported_seq":16,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.290973+0000","last_change":"2024-04-14T15:24:11.097436+0000","last_active":"2024-04-14T15:24:13.290973+0000","last_peered":"2024-04-14T15:24:13.290973+0000","last_clean":"2024-04-14T15:24:13.290973+0000","last_became_active":"2024-04-14T15:24:11.097007+0000","last_became_peered":"2024-04-14T15:24:11.097007+0000","last_unstale":"2024-04-14T15:24:13.290973+0000","last_undegraded":"2024-04-14T15:24:13.290973+0000","last_fullsized":"2024-04-14T15:24:13.290973+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":708,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:10.051866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:10.051866+0000","last_clean_scrub_stamp":"2024-04-14T15:24:10.051866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:15.304693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"323.2","version":"710'6","reported_seq":182,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.091054+0000","last_change":"2024-04-14T15:23:50.017364+0000","last_active":"2024-04-14T15:24:14.091054+0000","last_peered":"2024-04-14T15:24:14.091054+0000","last_clean":"2024-04-14T15:24:14.091054+0000","last_became_active":"2024-04-14T15:23:04.001557+0000","last_became_peered":"2024-04-14T15:23:04.001557+0000","last_unstale":"2024-04-14T15:24:14.091054+0000","last_undegraded":"2024-04-14T15:24:14.091054+0000","last_fullsized":"2024-04-14T15:24:14.091054+0000","mapping_epoch":647,"log_start":"0'0","ondisk_log_start":"0'0","created":647,"last_epoch_clean":648,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_clean_scrub_stamp":"2024-04-14T15:23:02.982695+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030027539999999998,"stat_sum":{"num_bytes":26,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"330.8","version":"0'0","reported_seq":16,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.316299+0000","last_change":"2024-04-14T15:24:11.092025+0000","last_active":"2024-04-14T15:24:13.316299+0000","last_peered":"2024-04-14T15:24:13.316299+0000","last_clean":"2024-04-14T15:24:13.316299+0000","last_became_active":"2024-04-14T15:24:11.091685+0000","last_became_peered":"2024-04-14T15:24:11.091685+0000","last_unstale":"2024-04-14T15:24:13.316299+0000","last_undegraded":"2024-04-14T15:24:13.316299+0000","last_fullsized":"2024-04-14T15:24:13.316299+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":708,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:10.051866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:10.051866+0000","last_clean_scrub_stamp":"2024-04-14T15:24:10.051866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:28.734464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"323.1","version":"696'3","reported_seq":181,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.318320+0000","last_change":"2024-04-14T15:23:50.012341+0000","last_active":"2024-04-14T15:24:13.318320+0000","last_peered":"2024-04-14T15:24:13.318320+0000","last_clean":"2024-04-14T15:24:13.318320+0000","last_became_active":"2024-04-14T15:23:03.998498+0000","last_became_peered":"2024-04-14T15:23:03.998498+0000","last_unstale":"2024-04-14T15:24:13.318320+0000","last_undegraded":"2024-04-14T15:24:13.318320+0000","last_fullsized":"2024-04-14T15:24:13.318320+0000","mapping_epoch":647,"log_start":"0'0","ondisk_log_start":"0'0","created":647,"last_epoch_clean":648,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_clean_scrub_stamp":"2024-04-14T15:23:02.982695+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:14.712668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017628399999999999,"stat_sum":{"num_bytes":12,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"330.9","version":"0'0","reported_seq":16,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.290725+0000","last_change":"2024-04-14T15:24:11.095424+0000","last_active":"2024-04-14T15:24:13.290725+0000","last_peered":"2024-04-14T15:24:13.290725+0000","last_clean":"2024-04-14T15:24:13.290725+0000","last_became_active":"2024-04-14T15:24:11.094902+0000","last_became_peered":"2024-04-14T15:24:11.094902+0000","last_unstale":"2024-04-14T15:24:13.290725+0000","last_undegraded":"2024-04-14T15:24:13.290725+0000","last_fullsized":"2024-04-14T15:24:13.290725+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":708,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:10.051866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:10.051866+0000","last_clean_scrub_stamp":"2024-04-14T15:24:10.051866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:34.035291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"323.0","version":"704'6","reported_seq":186,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.090725+0000","last_change":"2024-04-14T15:23:50.013063+0000","last_active":"2024-04-14T15:24:14.090725+0000","last_peered":"2024-04-14T15:24:14.090725+0000","last_clean":"2024-04-14T15:24:14.090725+0000","last_became_active":"2024-04-14T15:23:04.001020+0000","last_became_peered":"2024-04-14T15:23:04.001020+0000","last_unstale":"2024-04-14T15:24:14.090725+0000","last_undegraded":"2024-04-14T15:24:14.090725+0000","last_fullsized":"2024-04-14T15:24:14.090725+0000","mapping_epoch":647,"log_start":"0'0","ondisk_log_start":"0'0","created":647,"last_epoch_clean":648,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_clean_scrub_stamp":"2024-04-14T15:23:02.982695+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:15.215210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015329849999999999,"stat_sum":{"num_bytes":5,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":901,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.093085+0000","last_change":"2024-04-14T15:23:27.292731+0000","last_active":"2024-04-14T15:24:14.093085+0000","last_peered":"2024-04-14T15:24:14.093085+0000","last_clean":"2024-04-14T15:24:14.093085+0000","last_became_active":"2024-04-14T15:15:44.882941+0000","last_became_peered":"2024-04-14T15:15:44.882941+0000","last_unstale":"2024-04-14T15:24:14.093085+0000","last_undegraded":"2024-04-14T15:24:14.093085+0000","last_fullsized":"2024-04-14T15:24:14.093085+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:27.292590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:23:27.292590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:36.719319+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.9","version":"528'10","reported_seq":1533,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.318986+0000","last_change":"2024-04-14T15:20:52.998038+0000","last_active":"2024-04-14T15:24:13.318986+0000","last_peered":"2024-04-14T15:24:13.318986+0000","last_clean":"2024-04-14T15:24:13.318986+0000","last_became_active":"2024-04-14T15:20:52.984943+0000","last_became_peered":"2024-04-14T15:20:52.984943+0000","last_unstale":"2024-04-14T15:24:13.318986+0000","last_undegraded":"2024-04-14T15:24:13.318986+0000","last_fullsized":"2024-04-14T15:24:13.318986+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9","last_scrub_stamp":"2024-04-14T15:20:45.737773+0000","last_deep_scrub":"85'9","last_deep_scrub_stamp":"2024-04-14T15:10:51.950580+0000","last_clean_scrub_stamp":"2024-04-14T15:20:45.737773+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:13.524783+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"479'34","reported_seq":988,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.317374+0000","last_change":"2024-04-14T15:21:27.711513+0000","last_active":"2024-04-14T15:24:13.317374+0000","last_peered":"2024-04-14T15:24:13.317374+0000","last_clean":"2024-04-14T15:24:13.317374+0000","last_became_active":"2024-04-14T15:20:52.979204+0000","last_became_peered":"2024-04-14T15:20:52.979204+0000","last_unstale":"2024-04-14T15:24:13.317374+0000","last_undegraded":"2024-04-14T15:24:13.317374+0000","last_fullsized":"2024-04-14T15:24:13.317374+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'34","last_scrub_stamp":"2024-04-14T15:21:27.711357+0000","last_deep_scrub":"430'22","last_deep_scrub_stamp":"2024-04-14T15:18:39.704883+0000","last_clean_scrub_stamp":"2024-04-14T15:21:27.711357+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:13.524904+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":1354,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.095248+0000","last_change":"2024-04-14T15:23:24.788057+0000","last_active":"2024-04-14T15:24:14.095248+0000","last_peered":"2024-04-14T15:24:14.095248+0000","last_clean":"2024-04-14T15:24:14.095248+0000","last_became_active":"2024-04-14T15:09:43.021460+0000","last_became_peered":"2024-04-14T15:09:43.021460+0000","last_unstale":"2024-04-14T15:24:14.095248+0000","last_undegraded":"2024-04-14T15:24:14.095248+0000","last_fullsized":"2024-04-14T15:24:14.095248+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:24.787958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:23:24.787958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:50.309267+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":389,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.291213+0000","last_change":"2024-04-14T15:23:05.437375+0000","last_active":"2024-04-14T15:24:13.291213+0000","last_peered":"2024-04-14T15:24:13.291213+0000","last_clean":"2024-04-14T15:24:13.291213+0000","last_became_active":"2024-04-14T15:20:52.989666+0000","last_became_peered":"2024-04-14T15:20:52.989666+0000","last_unstale":"2024-04-14T15:24:13.291213+0000","last_undegraded":"2024-04-14T15:24:13.291213+0000","last_fullsized":"2024-04-14T15:24:13.291213+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:05.437268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:15:40.368219+0000","last_clean_scrub_stamp":"2024-04-14T15:23:05.437268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:30.217014+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":1250,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.320691+0000","last_change":"2024-04-14T15:21:24.713987+0000","last_active":"2024-04-14T15:24:13.320691+0000","last_peered":"2024-04-14T15:24:13.320691+0000","last_clean":"2024-04-14T15:24:13.320691+0000","last_became_active":"2024-04-14T15:20:52.991952+0000","last_became_peered":"2024-04-14T15:20:52.991952+0000","last_unstale":"2024-04-14T15:24:13.320691+0000","last_undegraded":"2024-04-14T15:24:13.320691+0000","last_fullsized":"2024-04-14T15:24:13.320691+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:24.713877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:21:24.713877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:13.524850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"479'15","reported_seq":1375,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.291247+0000","last_change":"2024-04-14T15:23:32.253957+0000","last_active":"2024-04-14T15:24:13.291247+0000","last_peered":"2024-04-14T15:24:13.291247+0000","last_clean":"2024-04-14T15:24:13.291247+0000","last_became_active":"2024-04-14T15:09:45.037179+0000","last_became_peered":"2024-04-14T15:09:45.037179+0000","last_unstale":"2024-04-14T15:24:13.291247+0000","last_undegraded":"2024-04-14T15:24:13.291247+0000","last_fullsized":"2024-04-14T15:24:13.291247+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'15","last_scrub_stamp":"2024-04-14T15:23:32.253815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:23:32.253815+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:37.113811+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":1351,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.092484+0000","last_change":"2024-04-14T15:22:17.414577+0000","last_active":"2024-04-14T15:24:14.092484+0000","last_peered":"2024-04-14T15:24:14.092484+0000","last_clean":"2024-04-14T15:24:14.092484+0000","last_became_active":"2024-04-14T15:09:43.020044+0000","last_became_peered":"2024-04-14T15:09:43.020044+0000","last_unstale":"2024-04-14T15:24:14.092484+0000","last_undegraded":"2024-04-14T15:24:14.092484+0000","last_fullsized":"2024-04-14T15:24:14.092484+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:17.414499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:22:17.414499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:16.232675+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1346,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.094224+0000","last_change":"2024-04-14T15:21:28.774876+0000","last_active":"2024-04-14T15:24:14.094224+0000","last_peered":"2024-04-14T15:24:14.094224+0000","last_clean":"2024-04-14T15:24:14.094224+0000","last_became_active":"2024-04-14T15:09:47.046153+0000","last_became_peered":"2024-04-14T15:09:47.046153+0000","last_unstale":"2024-04-14T15:24:14.094224+0000","last_undegraded":"2024-04-14T15:24:14.094224+0000","last_fullsized":"2024-04-14T15:24:14.094224+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:28.774798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:21:28.774798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:17.526271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1342,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.290444+0000","last_change":"2024-04-14T15:23:37.201923+0000","last_active":"2024-04-14T15:24:13.290444+0000","last_peered":"2024-04-14T15:24:13.290444+0000","last_clean":"2024-04-14T15:24:13.290444+0000","last_became_active":"2024-04-14T15:09:49.061495+0000","last_became_peered":"2024-04-14T15:09:49.061495+0000","last_unstale":"2024-04-14T15:24:13.290444+0000","last_undegraded":"2024-04-14T15:24:13.290444+0000","last_fullsized":"2024-04-14T15:24:13.290444+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:37.201795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:23:37.201795+0000","last_clean_scrub_stamp":"2024-04-14T15:23:37.201795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:49.569529+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":1250,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.320860+0000","last_change":"2024-04-14T15:21:20.684941+0000","last_active":"2024-04-14T15:24:13.320860+0000","last_peered":"2024-04-14T15:24:13.320860+0000","last_clean":"2024-04-14T15:24:13.320860+0000","last_became_active":"2024-04-14T15:20:52.999845+0000","last_became_peered":"2024-04-14T15:20:52.999845+0000","last_unstale":"2024-04-14T15:24:13.320860+0000","last_undegraded":"2024-04-14T15:24:13.320860+0000","last_fullsized":"2024-04-14T15:24:13.320860+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:20.684813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:21:20.684813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:13.524921+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"479'44","reported_seq":241,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.093882+0000","last_change":"2024-04-14T15:23:03.739862+0000","last_active":"2024-04-14T15:24:14.093882+0000","last_peered":"2024-04-14T15:24:14.093882+0000","last_clean":"2024-04-14T15:24:14.093882+0000","last_became_active":"2024-04-14T15:22:25.970899+0000","last_became_peered":"2024-04-14T15:22:25.970899+0000","last_unstale":"2024-04-14T15:24:14.093882+0000","last_undegraded":"2024-04-14T15:24:14.093882+0000","last_fullsized":"2024-04-14T15:24:14.093882+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'44","last_scrub_stamp":"2024-04-14T15:23:03.739762+0000","last_deep_scrub":"479'44","last_deep_scrub_stamp":"2024-04-14T15:19:58.856115+0000","last_clean_scrub_stamp":"2024-04-14T15:23:03.739762+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:21.683327+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":1362,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.094693+0000","last_change":"2024-04-14T15:23:44.419193+0000","last_active":"2024-04-14T15:24:14.094693+0000","last_peered":"2024-04-14T15:24:14.094693+0000","last_clean":"2024-04-14T15:24:14.094693+0000","last_became_active":"2024-04-14T15:09:43.020268+0000","last_became_peered":"2024-04-14T15:09:43.020268+0000","last_unstale":"2024-04-14T15:24:14.094693+0000","last_undegraded":"2024-04-14T15:24:14.094693+0000","last_fullsized":"2024-04-14T15:24:14.094693+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:44.419100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:34.342793+0000","last_clean_scrub_stamp":"2024-04-14T15:23:44.419100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:56.886676+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1338,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.094124+0000","last_change":"2024-04-14T15:21:23.738251+0000","last_active":"2024-04-14T15:24:14.094124+0000","last_peered":"2024-04-14T15:24:14.094124+0000","last_clean":"2024-04-14T15:24:14.094124+0000","last_became_active":"2024-04-14T15:09:47.045542+0000","last_became_peered":"2024-04-14T15:09:47.045542+0000","last_unstale":"2024-04-14T15:24:14.094124+0000","last_undegraded":"2024-04-14T15:24:14.094124+0000","last_fullsized":"2024-04-14T15:24:14.094124+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:23.738176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:21:23.738176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:17.526218+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1335,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.315959+0000","last_change":"2024-04-14T15:22:55.492856+0000","last_active":"2024-04-14T15:24:13.315959+0000","last_peered":"2024-04-14T15:24:13.315959+0000","last_clean":"2024-04-14T15:24:13.315959+0000","last_became_active":"2024-04-14T15:09:49.065397+0000","last_became_peered":"2024-04-14T15:09:49.065397+0000","last_unstale":"2024-04-14T15:24:13.315959+0000","last_undegraded":"2024-04-14T15:24:13.315959+0000","last_fullsized":"2024-04-14T15:24:13.315959+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:55.492753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:22:55.492753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:25.188387+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"91'130","reported_seq":1289,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.092497+0000","last_change":"2024-04-14T15:21:58.912480+0000","last_active":"2024-04-14T15:24:14.092497+0000","last_peered":"2024-04-14T15:24:14.092497+0000","last_clean":"2024-04-14T15:24:14.092497+0000","last_became_active":"2024-04-14T15:21:12.961419+0000","last_became_peered":"2024-04-14T15:21:12.961419+0000","last_unstale":"2024-04-14T15:24:14.092497+0000","last_undegraded":"2024-04-14T15:24:14.092497+0000","last_fullsized":"2024-04-14T15:24:14.092497+0000","mapping_epoch":276,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:21:58.912387+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:21:58.912387+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":70,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:17.526349+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":1251,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.094140+0000","last_change":"2024-04-14T15:22:14.964486+0000","last_active":"2024-04-14T15:24:14.094140+0000","last_peered":"2024-04-14T15:24:14.094140+0000","last_clean":"2024-04-14T15:24:14.094140+0000","last_became_active":"2024-04-14T15:12:56.474322+0000","last_became_peered":"2024-04-14T15:12:56.474322+0000","last_unstale":"2024-04-14T15:24:14.094140+0000","last_undegraded":"2024-04-14T15:24:14.094140+0000","last_fullsized":"2024-04-14T15:24:14.094140+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:14.964404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:22:14.964404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:17.526412+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"479'31","reported_seq":1440,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.320681+0000","last_change":"2024-04-14T15:22:02.037366+0000","last_active":"2024-04-14T15:24:13.320681+0000","last_peered":"2024-04-14T15:24:13.320681+0000","last_clean":"2024-04-14T15:24:13.320681+0000","last_became_active":"2024-04-14T15:13:12.933907+0000","last_became_peered":"2024-04-14T15:13:12.933907+0000","last_unstale":"2024-04-14T15:24:13.320681+0000","last_undegraded":"2024-04-14T15:24:13.320681+0000","last_fullsized":"2024-04-14T15:24:13.320681+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'31","last_scrub_stamp":"2024-04-14T15:22:02.037237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:22:02.037237+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":1347,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.319996+0000","last_change":"2024-04-14T15:23:26.611299+0000","last_active":"2024-04-14T15:24:13.319996+0000","last_peered":"2024-04-14T15:24:13.319996+0000","last_clean":"2024-04-14T15:24:13.319996+0000","last_became_active":"2024-04-14T15:09:43.022351+0000","last_became_peered":"2024-04-14T15:09:43.022351+0000","last_unstale":"2024-04-14T15:24:13.319996+0000","last_undegraded":"2024-04-14T15:24:13.319996+0000","last_fullsized":"2024-04-14T15:24:13.319996+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:26.611217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:23:26.611217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:40.741998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1343,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.092248+0000","last_change":"2024-04-14T15:23:25.344592+0000","last_active":"2024-04-14T15:24:14.092248+0000","last_peered":"2024-04-14T15:24:14.092248+0000","last_clean":"2024-04-14T15:24:14.092248+0000","last_became_active":"2024-04-14T15:09:47.042810+0000","last_became_peered":"2024-04-14T15:09:47.042810+0000","last_unstale":"2024-04-14T15:24:14.092248+0000","last_undegraded":"2024-04-14T15:24:14.092248+0000","last_fullsized":"2024-04-14T15:24:14.092248+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:25.344505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:23:25.344505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:27.884803+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1342,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.095691+0000","last_change":"2024-04-14T15:22:58.703767+0000","last_active":"2024-04-14T15:24:14.095691+0000","last_peered":"2024-04-14T15:24:14.095691+0000","last_clean":"2024-04-14T15:24:14.095691+0000","last_became_active":"2024-04-14T15:09:49.069594+0000","last_became_peered":"2024-04-14T15:09:49.069594+0000","last_unstale":"2024-04-14T15:24:14.095691+0000","last_undegraded":"2024-04-14T15:24:14.095691+0000","last_fullsized":"2024-04-14T15:24:14.095691+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:58.703688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:22:58.703688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:25.423495+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"91'130","reported_seq":1692,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.318603+0000","last_change":"2024-04-14T15:22:04.530538+0000","last_active":"2024-04-14T15:24:13.318603+0000","last_peered":"2024-04-14T15:24:13.318603+0000","last_clean":"2024-04-14T15:24:13.318603+0000","last_became_active":"2024-04-14T15:21:09.020300+0000","last_became_peered":"2024-04-14T15:21:09.020300+0000","last_unstale":"2024-04-14T15:24:13.318603+0000","last_undegraded":"2024-04-14T15:24:13.318603+0000","last_fullsized":"2024-04-14T15:24:13.318603+0000","mapping_epoch":189,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":544,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:22:04.530456+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:22:04.530456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:13.524941+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":387,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.319791+0000","last_change":"2024-04-14T15:23:41.487909+0000","last_active":"2024-04-14T15:24:13.319791+0000","last_peered":"2024-04-14T15:24:13.319791+0000","last_clean":"2024-04-14T15:24:13.319791+0000","last_became_active":"2024-04-14T15:20:52.980375+0000","last_became_peered":"2024-04-14T15:20:52.980375+0000","last_unstale":"2024-04-14T15:24:13.319791+0000","last_undegraded":"2024-04-14T15:24:13.319791+0000","last_fullsized":"2024-04-14T15:24:13.319791+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:41.487814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:23:41.487814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:56.105699+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"479'25","reported_seq":1388,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.319147+0000","last_change":"2024-04-14T15:20:53.030926+0000","last_active":"2024-04-14T15:24:13.319147+0000","last_peered":"2024-04-14T15:24:13.319147+0000","last_clean":"2024-04-14T15:24:13.319147+0000","last_became_active":"2024-04-14T15:20:52.987699+0000","last_became_peered":"2024-04-14T15:20:52.987699+0000","last_unstale":"2024-04-14T15:24:13.319147+0000","last_undegraded":"2024-04-14T15:24:13.319147+0000","last_fullsized":"2024-04-14T15:24:13.319147+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'25","last_scrub_stamp":"2024-04-14T15:20:03.571160+0000","last_deep_scrub":"479'25","last_deep_scrub_stamp":"2024-04-14T15:20:03.571160+0000","last_clean_scrub_stamp":"2024-04-14T15:20:03.571160+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":1351,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.091620+0000","last_change":"2024-04-14T15:21:31.700100+0000","last_active":"2024-04-14T15:24:14.091620+0000","last_peered":"2024-04-14T15:24:14.091620+0000","last_clean":"2024-04-14T15:24:14.091620+0000","last_became_active":"2024-04-14T15:09:43.018871+0000","last_became_peered":"2024-04-14T15:09:43.018871+0000","last_unstale":"2024-04-14T15:24:14.091620+0000","last_undegraded":"2024-04-14T15:24:14.091620+0000","last_fullsized":"2024-04-14T15:24:14.091620+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:31.700000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:21:31.700000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:16.232561+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"400'9","reported_seq":1530,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.095272+0000","last_change":"2024-04-14T15:23:30.695693+0000","last_active":"2024-04-14T15:24:14.095272+0000","last_peered":"2024-04-14T15:24:14.095272+0000","last_clean":"2024-04-14T15:24:14.095272+0000","last_became_active":"2024-04-14T15:09:47.044085+0000","last_became_peered":"2024-04-14T15:09:47.044085+0000","last_unstale":"2024-04-14T15:24:14.095272+0000","last_undegraded":"2024-04-14T15:24:14.095272+0000","last_fullsized":"2024-04-14T15:24:14.095272+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'9","last_scrub_stamp":"2024-04-14T15:23:30.695563+0000","last_deep_scrub":"400'9","last_deep_scrub_stamp":"2024-04-14T15:17:54.845881+0000","last_clean_scrub_stamp":"2024-04-14T15:23:30.695563+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:41.740172+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1327,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.315909+0000","last_change":"2024-04-14T15:19:18.588681+0000","last_active":"2024-04-14T15:24:13.315909+0000","last_peered":"2024-04-14T15:24:13.315909+0000","last_clean":"2024-04-14T15:24:13.315909+0000","last_became_active":"2024-04-14T15:09:49.065274+0000","last_became_peered":"2024-04-14T15:09:49.065274+0000","last_unstale":"2024-04-14T15:24:13.315909+0000","last_undegraded":"2024-04-14T15:24:13.315909+0000","last_fullsized":"2024-04-14T15:24:13.315909+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:18.588546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:18.588546+0000","last_clean_scrub_stamp":"2024-04-14T15:19:18.588546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"91'130","reported_seq":1304,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.094025+0000","last_change":"2024-04-14T15:22:06.865915+0000","last_active":"2024-04-14T15:24:14.094025+0000","last_peered":"2024-04-14T15:24:14.094025+0000","last_clean":"2024-04-14T15:24:14.094025+0000","last_became_active":"2024-04-14T15:21:04.971515+0000","last_became_peered":"2024-04-14T15:21:04.971515+0000","last_unstale":"2024-04-14T15:24:14.094025+0000","last_undegraded":"2024-04-14T15:24:14.094025+0000","last_fullsized":"2024-04-14T15:24:14.094025+0000","mapping_epoch":530,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":540,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:22:06.865825+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:22:06.865825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:17.526382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":1248,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.091244+0000","last_change":"2024-04-14T15:23:29.219186+0000","last_active":"2024-04-14T15:24:14.091244+0000","last_peered":"2024-04-14T15:24:14.091244+0000","last_clean":"2024-04-14T15:24:14.091244+0000","last_became_active":"2024-04-14T15:12:56.487615+0000","last_became_peered":"2024-04-14T15:12:56.487615+0000","last_unstale":"2024-04-14T15:24:14.091244+0000","last_undegraded":"2024-04-14T15:24:14.091244+0000","last_fullsized":"2024-04-14T15:24:14.091244+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:29.219111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:23:29.219111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:30.505271+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"479'34","reported_seq":1397,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.093223+0000","last_change":"2024-04-14T15:21:13.577045+0000","last_active":"2024-04-14T15:24:14.093223+0000","last_peered":"2024-04-14T15:24:14.093223+0000","last_clean":"2024-04-14T15:24:14.093223+0000","last_became_active":"2024-04-14T15:09:45.033331+0000","last_became_peered":"2024-04-14T15:09:45.033331+0000","last_unstale":"2024-04-14T15:24:14.093223+0000","last_undegraded":"2024-04-14T15:24:14.093223+0000","last_fullsized":"2024-04-14T15:24:14.093223+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'34","last_scrub_stamp":"2024-04-14T15:21:13.576916+0000","last_deep_scrub":"398'22","last_deep_scrub_stamp":"2024-04-14T15:17:52.883850+0000","last_clean_scrub_stamp":"2024-04-14T15:21:13.576916+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:16.232512+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":1351,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.091019+0000","last_change":"2024-04-14T15:22:55.384701+0000","last_active":"2024-04-14T15:24:14.091019+0000","last_peered":"2024-04-14T15:24:14.091019+0000","last_clean":"2024-04-14T15:24:14.091019+0000","last_became_active":"2024-04-14T15:09:43.020878+0000","last_became_peered":"2024-04-14T15:09:43.020878+0000","last_unstale":"2024-04-14T15:24:14.091019+0000","last_undegraded":"2024-04-14T15:24:14.091019+0000","last_fullsized":"2024-04-14T15:24:14.091019+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:55.384626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:22:55.384626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:21.049067+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"72'8","reported_seq":1514,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.320323+0000","last_change":"2024-04-14T15:19:19.596882+0000","last_active":"2024-04-14T15:24:13.320323+0000","last_peered":"2024-04-14T15:24:13.320323+0000","last_clean":"2024-04-14T15:24:13.320323+0000","last_became_active":"2024-04-14T15:09:47.050962+0000","last_became_peered":"2024-04-14T15:09:47.050962+0000","last_unstale":"2024-04-14T15:24:13.320323+0000","last_undegraded":"2024-04-14T15:24:13.320323+0000","last_fullsized":"2024-04-14T15:24:13.320323+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:19:19.596773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:19:19.596773+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":902,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.092163+0000","last_change":"2024-04-14T15:20:44.583925+0000","last_active":"2024-04-14T15:24:14.092163+0000","last_peered":"2024-04-14T15:24:14.092163+0000","last_clean":"2024-04-14T15:24:14.092163+0000","last_became_active":"2024-04-14T15:15:44.879699+0000","last_became_peered":"2024-04-14T15:15:44.879699+0000","last_unstale":"2024-04-14T15:24:14.092163+0000","last_undegraded":"2024-04-14T15:24:14.092163+0000","last_fullsized":"2024-04-14T15:24:14.092163+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:44.583849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:20:44.583849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:16.232464+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"233'8","reported_seq":666,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.315709+0000","last_change":"2024-04-14T15:23:43.908078+0000","last_active":"2024-04-14T15:24:13.315709+0000","last_peered":"2024-04-14T15:24:13.315709+0000","last_clean":"2024-04-14T15:24:13.315709+0000","last_became_active":"2024-04-14T15:20:52.995129+0000","last_became_peered":"2024-04-14T15:20:52.995129+0000","last_unstale":"2024-04-14T15:24:13.315709+0000","last_undegraded":"2024-04-14T15:24:13.315709+0000","last_fullsized":"2024-04-14T15:24:13.315709+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'8","last_scrub_stamp":"2024-04-14T15:23:43.907946+0000","last_deep_scrub":"233'8","last_deep_scrub_stamp":"2024-04-14T15:15:01.295800+0000","last_clean_scrub_stamp":"2024-04-14T15:23:43.907946+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:59.919832+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":5,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"479'24","reported_seq":1397,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.319660+0000","last_change":"2024-04-14T15:21:13.939265+0000","last_active":"2024-04-14T15:24:13.319660+0000","last_peered":"2024-04-14T15:24:13.319660+0000","last_clean":"2024-04-14T15:24:13.319660+0000","last_became_active":"2024-04-14T15:20:52.998853+0000","last_became_peered":"2024-04-14T15:20:52.998853+0000","last_unstale":"2024-04-14T15:24:13.319660+0000","last_undegraded":"2024-04-14T15:24:13.319660+0000","last_fullsized":"2024-04-14T15:24:13.319660+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'24","last_scrub_stamp":"2024-04-14T15:21:13.939121+0000","last_deep_scrub":"85'16","last_deep_scrub_stamp":"2024-04-14T15:10:47.456169+0000","last_clean_scrub_stamp":"2024-04-14T15:21:13.939121+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":1355,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.317610+0000","last_change":"2024-04-14T15:23:36.489554+0000","last_active":"2024-04-14T15:24:13.317610+0000","last_peered":"2024-04-14T15:24:13.317610+0000","last_clean":"2024-04-14T15:24:13.317610+0000","last_became_active":"2024-04-14T15:09:43.018601+0000","last_became_peered":"2024-04-14T15:09:43.018601+0000","last_unstale":"2024-04-14T15:24:13.317610+0000","last_undegraded":"2024-04-14T15:24:13.317610+0000","last_fullsized":"2024-04-14T15:24:13.317610+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:36.489475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:23:36.489475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:53.352519+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":903,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.319682+0000","last_change":"2024-04-14T15:20:52.974819+0000","last_active":"2024-04-14T15:24:13.319682+0000","last_peered":"2024-04-14T15:24:13.319682+0000","last_clean":"2024-04-14T15:24:13.319682+0000","last_became_active":"2024-04-14T15:20:52.970831+0000","last_became_peered":"2024-04-14T15:20:52.970831+0000","last_unstale":"2024-04-14T15:24:13.319682+0000","last_undegraded":"2024-04-14T15:24:13.319682+0000","last_fullsized":"2024-04-14T15:24:13.319682+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:59.596964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:19:59.596964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1013,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.316148+0000","last_change":"2024-04-14T15:21:17.980121+0000","last_active":"2024-04-14T15:24:13.316148+0000","last_peered":"2024-04-14T15:24:13.316148+0000","last_clean":"2024-04-14T15:24:13.316148+0000","last_became_active":"2024-04-14T15:20:52.999771+0000","last_became_peered":"2024-04-14T15:20:52.999771+0000","last_unstale":"2024-04-14T15:24:13.316148+0000","last_undegraded":"2024-04-14T15:24:13.316148+0000","last_fullsized":"2024-04-14T15:24:13.316148+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:17.980009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:57.214046+0000","last_clean_scrub_stamp":"2024-04-14T15:21:17.980009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:15.809978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"91'130","reported_seq":1479,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.316043+0000","last_change":"2024-04-14T15:23:29.044374+0000","last_active":"2024-04-14T15:24:13.316043+0000","last_peered":"2024-04-14T15:24:13.316043+0000","last_clean":"2024-04-14T15:24:13.316043+0000","last_became_active":"2024-04-14T15:23:29.044065+0000","last_became_peered":"2024-04-14T15:23:29.044065+0000","last_unstale":"2024-04-14T15:24:13.316043+0000","last_undegraded":"2024-04-14T15:24:13.316043+0000","last_fullsized":"2024-04-14T15:24:13.316043+0000","mapping_epoch":189,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":672,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:22:03.138556+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:22:03.138556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":60,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:15.810035+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":1244,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.320051+0000","last_change":"2024-04-14T15:20:37.765993+0000","last_active":"2024-04-14T15:24:13.320051+0000","last_peered":"2024-04-14T15:24:13.320051+0000","last_clean":"2024-04-14T15:24:13.320051+0000","last_became_active":"2024-04-14T15:12:56.427276+0000","last_became_peered":"2024-04-14T15:12:56.427276+0000","last_unstale":"2024-04-14T15:24:13.320051+0000","last_undegraded":"2024-04-14T15:24:13.320051+0000","last_fullsized":"2024-04-14T15:24:13.320051+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:37.765916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:20:37.765916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"255'10","reported_seq":1164,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.315704+0000","last_change":"2024-04-14T15:23:26.964698+0000","last_active":"2024-04-14T15:24:13.315704+0000","last_peered":"2024-04-14T15:24:13.315704+0000","last_clean":"2024-04-14T15:24:13.315704+0000","last_became_active":"2024-04-14T15:15:21.873747+0000","last_became_peered":"2024-04-14T15:15:21.873747+0000","last_unstale":"2024-04-14T15:24:13.315704+0000","last_undegraded":"2024-04-14T15:24:13.315704+0000","last_fullsized":"2024-04-14T15:24:13.315704+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'10","last_scrub_stamp":"2024-04-14T15:23:26.964616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:23:26.964616+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:29.334703+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"479'28","reported_seq":1407,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.317936+0000","last_change":"2024-04-14T15:22:09.093925+0000","last_active":"2024-04-14T15:24:13.317936+0000","last_peered":"2024-04-14T15:24:13.317936+0000","last_clean":"2024-04-14T15:24:13.317936+0000","last_became_active":"2024-04-14T15:20:52.989635+0000","last_became_peered":"2024-04-14T15:20:52.989635+0000","last_unstale":"2024-04-14T15:24:13.317936+0000","last_undegraded":"2024-04-14T15:24:13.317936+0000","last_fullsized":"2024-04-14T15:24:13.317936+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'28","last_scrub_stamp":"2024-04-14T15:22:09.093771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:22:09.093771+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":379,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.321687+0000","last_change":"2024-04-14T15:20:52.999728+0000","last_active":"2024-04-14T15:24:13.321687+0000","last_peered":"2024-04-14T15:24:13.321687+0000","last_clean":"2024-04-14T15:24:13.321687+0000","last_became_active":"2024-04-14T15:20:52.999379+0000","last_became_peered":"2024-04-14T15:20:52.999379+0000","last_unstale":"2024-04-14T15:24:13.321687+0000","last_undegraded":"2024-04-14T15:24:13.321687+0000","last_fullsized":"2024-04-14T15:24:13.321687+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:41.799129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:20:41.799129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:13.524763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1341,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.320353+0000","last_change":"2024-04-14T15:23:28.571478+0000","last_active":"2024-04-14T15:24:13.320353+0000","last_peered":"2024-04-14T15:24:13.320353+0000","last_clean":"2024-04-14T15:24:13.320353+0000","last_became_active":"2024-04-14T15:20:52.991388+0000","last_became_peered":"2024-04-14T15:20:52.991388+0000","last_unstale":"2024-04-14T15:24:13.320353+0000","last_undegraded":"2024-04-14T15:24:13.320353+0000","last_fullsized":"2024-04-14T15:24:13.320353+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:28.571374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:23:28.571374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:45.653854+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"91'130","reported_seq":1495,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.319110+0000","last_change":"2024-04-14T15:23:06.153451+0000","last_active":"2024-04-14T15:24:13.319110+0000","last_peered":"2024-04-14T15:24:13.319110+0000","last_clean":"2024-04-14T15:24:13.319110+0000","last_became_active":"2024-04-14T15:23:02.990846+0000","last_became_peered":"2024-04-14T15:23:02.990846+0000","last_unstale":"2024-04-14T15:24:13.319110+0000","last_undegraded":"2024-04-14T15:24:13.319110+0000","last_fullsized":"2024-04-14T15:24:13.319110+0000","mapping_epoch":530,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":647,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:23:06.153339+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:23:06.153339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:30.763302+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":1244,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.321616+0000","last_change":"2024-04-14T15:20:40.724743+0000","last_active":"2024-04-14T15:24:13.321616+0000","last_peered":"2024-04-14T15:24:13.321616+0000","last_clean":"2024-04-14T15:24:13.321616+0000","last_became_active":"2024-04-14T15:12:56.355254+0000","last_became_peered":"2024-04-14T15:12:56.355254+0000","last_unstale":"2024-04-14T15:24:13.321616+0000","last_undegraded":"2024-04-14T15:24:13.321616+0000","last_fullsized":"2024-04-14T15:24:13.321616+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:40.724658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:20:40.724658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1339,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.318107+0000","last_change":"2024-04-14T15:22:55.245731+0000","last_active":"2024-04-14T15:24:13.318107+0000","last_peered":"2024-04-14T15:24:13.318107+0000","last_clean":"2024-04-14T15:24:13.318107+0000","last_became_active":"2024-04-14T15:09:47.044610+0000","last_became_peered":"2024-04-14T15:09:47.044610+0000","last_unstale":"2024-04-14T15:24:13.318107+0000","last_undegraded":"2024-04-14T15:24:13.318107+0000","last_fullsized":"2024-04-14T15:24:13.318107+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:55.245639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:39.394655+0000","last_clean_scrub_stamp":"2024-04-14T15:22:55.245639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"479'30","reported_seq":1075,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.315696+0000","last_change":"2024-04-14T15:22:58.001736+0000","last_active":"2024-04-14T15:24:13.315696+0000","last_peered":"2024-04-14T15:24:13.315696+0000","last_clean":"2024-04-14T15:24:13.315696+0000","last_became_active":"2024-04-14T15:15:21.880565+0000","last_became_peered":"2024-04-14T15:15:21.880565+0000","last_unstale":"2024-04-14T15:24:13.315696+0000","last_undegraded":"2024-04-14T15:24:13.315696+0000","last_fullsized":"2024-04-14T15:24:13.315696+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'30","last_scrub_stamp":"2024-04-14T15:22:58.001609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:22:58.001609+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:16.166054+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'2","reported_seq":1355,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.092714+0000","last_change":"2024-04-14T15:21:25.663010+0000","last_active":"2024-04-14T15:24:14.092714+0000","last_peered":"2024-04-14T15:24:14.092714+0000","last_clean":"2024-04-14T15:24:14.092714+0000","last_became_active":"2024-04-14T15:20:52.987576+0000","last_became_peered":"2024-04-14T15:20:52.987576+0000","last_unstale":"2024-04-14T15:24:14.092714+0000","last_undegraded":"2024-04-14T15:24:14.092714+0000","last_fullsized":"2024-04-14T15:24:14.092714+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'2","last_scrub_stamp":"2024-04-14T15:21:25.662901+0000","last_deep_scrub":"91'2","last_deep_scrub_stamp":"2024-04-14T15:17:48.850069+0000","last_clean_scrub_stamp":"2024-04-14T15:21:25.662901+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:16.232530+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1335,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.315829+0000","last_change":"2024-04-14T15:23:33.834918+0000","last_active":"2024-04-14T15:24:13.315829+0000","last_peered":"2024-04-14T15:24:13.315829+0000","last_clean":"2024-04-14T15:24:13.315829+0000","last_became_active":"2024-04-14T15:09:49.065684+0000","last_became_peered":"2024-04-14T15:09:49.065684+0000","last_unstale":"2024-04-14T15:24:13.315829+0000","last_undegraded":"2024-04-14T15:24:13.315829+0000","last_fullsized":"2024-04-14T15:24:13.315829+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:33.834838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:23:33.834838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:37.110012+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"91'130","reported_seq":765,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.291796+0000","last_change":"2024-04-14T15:23:43.310976+0000","last_active":"2024-04-14T15:24:13.291796+0000","last_peered":"2024-04-14T15:24:13.291796+0000","last_clean":"2024-04-14T15:24:13.291796+0000","last_became_active":"2024-04-14T15:22:58.991118+0000","last_became_peered":"2024-04-14T15:22:58.991118+0000","last_unstale":"2024-04-14T15:24:13.291796+0000","last_undegraded":"2024-04-14T15:24:13.291796+0000","last_fullsized":"2024-04-14T15:24:13.291796+0000","mapping_epoch":530,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:23:43.310892+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:23:43.310892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:49.797838+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":1256,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.092100+0000","last_change":"2024-04-14T15:23:24.298794+0000","last_active":"2024-04-14T15:24:14.092100+0000","last_peered":"2024-04-14T15:24:14.092100+0000","last_clean":"2024-04-14T15:24:14.092100+0000","last_became_active":"2024-04-14T15:12:56.402023+0000","last_became_peered":"2024-04-14T15:12:56.402023+0000","last_unstale":"2024-04-14T15:24:14.092100+0000","last_undegraded":"2024-04-14T15:24:14.092100+0000","last_fullsized":"2024-04-14T15:24:14.092100+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:24.298717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:35.003968+0000","last_clean_scrub_stamp":"2024-04-14T15:23:24.298717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:35.516401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":385,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.291110+0000","last_change":"2024-04-14T15:23:00.401837+0000","last_active":"2024-04-14T15:24:13.291110+0000","last_peered":"2024-04-14T15:24:13.291110+0000","last_clean":"2024-04-14T15:24:13.291110+0000","last_became_active":"2024-04-14T15:20:52.987883+0000","last_became_peered":"2024-04-14T15:20:52.987883+0000","last_unstale":"2024-04-14T15:24:13.291110+0000","last_undegraded":"2024-04-14T15:24:13.291110+0000","last_fullsized":"2024-04-14T15:24:13.291110+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:00.401762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:23:00.401762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:16.324028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"479'37","reported_seq":1409,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.291366+0000","last_change":"2024-04-14T15:23:38.246620+0000","last_active":"2024-04-14T15:24:13.291366+0000","last_peered":"2024-04-14T15:24:13.291366+0000","last_clean":"2024-04-14T15:24:13.291366+0000","last_became_active":"2024-04-14T15:09:45.179182+0000","last_became_peered":"2024-04-14T15:09:45.179182+0000","last_unstale":"2024-04-14T15:24:13.291366+0000","last_undegraded":"2024-04-14T15:24:13.291366+0000","last_fullsized":"2024-04-14T15:24:13.291366+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'37","last_scrub_stamp":"2024-04-14T15:23:38.246517+0000","last_deep_scrub":"102'27","last_deep_scrub_stamp":"2024-04-14T15:17:46.924330+0000","last_clean_scrub_stamp":"2024-04-14T15:23:38.246517+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:41.818665+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1347,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.320639+0000","last_change":"2024-04-14T15:22:56.541983+0000","last_active":"2024-04-14T15:24:13.320639+0000","last_peered":"2024-04-14T15:24:13.320639+0000","last_clean":"2024-04-14T15:24:13.320639+0000","last_became_active":"2024-04-14T15:09:43.021006+0000","last_became_peered":"2024-04-14T15:09:43.021006+0000","last_unstale":"2024-04-14T15:24:13.320639+0000","last_undegraded":"2024-04-14T15:24:13.320639+0000","last_fullsized":"2024-04-14T15:24:13.320639+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:56.541899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:22:56.541899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1345,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.091110+0000","last_change":"2024-04-14T15:23:34.182007+0000","last_active":"2024-04-14T15:24:14.091110+0000","last_peered":"2024-04-14T15:24:14.091110+0000","last_clean":"2024-04-14T15:24:14.091110+0000","last_became_active":"2024-04-14T15:20:52.978765+0000","last_became_peered":"2024-04-14T15:20:52.978765+0000","last_unstale":"2024-04-14T15:24:14.091110+0000","last_undegraded":"2024-04-14T15:24:14.091110+0000","last_fullsized":"2024-04-14T15:24:14.091110+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:34.181914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:56.206649+0000","last_clean_scrub_stamp":"2024-04-14T15:23:34.181914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:48.624030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"91'130","reported_seq":1494,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.091388+0000","last_change":"2024-04-14T15:23:32.988316+0000","last_active":"2024-04-14T15:24:14.091388+0000","last_peered":"2024-04-14T15:24:14.091388+0000","last_clean":"2024-04-14T15:24:14.091388+0000","last_became_active":"2024-04-14T15:23:32.987931+0000","last_became_peered":"2024-04-14T15:23:32.987931+0000","last_unstale":"2024-04-14T15:24:14.091388+0000","last_undegraded":"2024-04-14T15:24:14.091388+0000","last_fullsized":"2024-04-14T15:24:14.091388+0000","mapping_epoch":530,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:22:57.365464+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:22:57.365464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:19.705355+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":962,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.315803+0000","last_change":"2024-04-14T15:23:30.968804+0000","last_active":"2024-04-14T15:24:13.315803+0000","last_peered":"2024-04-14T15:24:13.315803+0000","last_clean":"2024-04-14T15:24:13.315803+0000","last_became_active":"2024-04-14T15:20:52.999776+0000","last_became_peered":"2024-04-14T15:20:52.999776+0000","last_unstale":"2024-04-14T15:24:13.315803+0000","last_undegraded":"2024-04-14T15:24:13.315803+0000","last_fullsized":"2024-04-14T15:24:13.315803+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:30.968711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:23:30.968711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:34.065587+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":901,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.093477+0000","last_change":"2024-04-14T15:21:59.095980+0000","last_active":"2024-04-14T15:24:14.093477+0000","last_peered":"2024-04-14T15:24:14.093477+0000","last_clean":"2024-04-14T15:24:14.093477+0000","last_became_active":"2024-04-14T15:15:44.883694+0000","last_became_peered":"2024-04-14T15:15:44.883694+0000","last_unstale":"2024-04-14T15:24:14.093477+0000","last_undegraded":"2024-04-14T15:24:14.093477+0000","last_fullsized":"2024-04-14T15:24:14.093477+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:59.095899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:21:59.095899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:16.232593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"479'30","reported_seq":1392,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.291711+0000","last_change":"2024-04-14T15:23:30.295880+0000","last_active":"2024-04-14T15:24:13.291711+0000","last_peered":"2024-04-14T15:24:13.291711+0000","last_clean":"2024-04-14T15:24:13.291711+0000","last_became_active":"2024-04-14T15:09:45.037231+0000","last_became_peered":"2024-04-14T15:09:45.037231+0000","last_unstale":"2024-04-14T15:24:13.291711+0000","last_undegraded":"2024-04-14T15:24:13.291711+0000","last_fullsized":"2024-04-14T15:24:13.291711+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'30","last_scrub_stamp":"2024-04-14T15:23:30.295747+0000","last_deep_scrub":"182'18","last_deep_scrub_stamp":"2024-04-14T15:14:10.383906+0000","last_clean_scrub_stamp":"2024-04-14T15:23:30.295747+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:49.622040+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1348,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.317882+0000","last_change":"2024-04-14T15:20:42.694332+0000","last_active":"2024-04-14T15:24:13.317882+0000","last_peered":"2024-04-14T15:24:13.317882+0000","last_clean":"2024-04-14T15:24:13.317882+0000","last_became_active":"2024-04-14T15:09:43.025095+0000","last_became_peered":"2024-04-14T15:09:43.025095+0000","last_unstale":"2024-04-14T15:24:13.317882+0000","last_undegraded":"2024-04-14T15:24:13.317882+0000","last_fullsized":"2024-04-14T15:24:13.317882+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:42.694243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:42.694243+0000","last_clean_scrub_stamp":"2024-04-14T15:20:42.694243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:13.524746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1361,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.291159+0000","last_change":"2024-04-14T15:23:40.238604+0000","last_active":"2024-04-14T15:24:13.291159+0000","last_peered":"2024-04-14T15:24:13.291159+0000","last_clean":"2024-04-14T15:24:13.291159+0000","last_became_active":"2024-04-14T15:09:43.018509+0000","last_became_peered":"2024-04-14T15:09:43.018509+0000","last_unstale":"2024-04-14T15:24:13.291159+0000","last_undegraded":"2024-04-14T15:24:13.291159+0000","last_fullsized":"2024-04-14T15:24:13.291159+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:40.238527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:23:40.238527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:06.849826+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0018592680000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"355'2","reported_seq":1341,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.092265+0000","last_change":"2024-04-14T15:23:43.059738+0000","last_active":"2024-04-14T15:24:14.092265+0000","last_peered":"2024-04-14T15:24:14.092265+0000","last_clean":"2024-04-14T15:24:14.092265+0000","last_became_active":"2024-04-14T15:09:49.065243+0000","last_became_peered":"2024-04-14T15:09:49.065243+0000","last_unstale":"2024-04-14T15:24:14.092265+0000","last_undegraded":"2024-04-14T15:24:14.092265+0000","last_fullsized":"2024-04-14T15:24:14.092265+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'2","last_scrub_stamp":"2024-04-14T15:23:43.059603+0000","last_deep_scrub":"355'2","last_deep_scrub_stamp":"2024-04-14T15:20:04.842881+0000","last_clean_scrub_stamp":"2024-04-14T15:23:43.059603+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:47.245706+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":946,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.316071+0000","last_change":"2024-04-14T15:21:28.122721+0000","last_active":"2024-04-14T15:24:13.316071+0000","last_peered":"2024-04-14T15:24:13.316071+0000","last_clean":"2024-04-14T15:24:13.316071+0000","last_became_active":"2024-04-14T15:20:53.000058+0000","last_became_peered":"2024-04-14T15:20:53.000058+0000","last_unstale":"2024-04-14T15:24:13.316071+0000","last_undegraded":"2024-04-14T15:24:13.316071+0000","last_fullsized":"2024-04-14T15:24:13.316071+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:28.122647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:21:28.122647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:15.809913+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1339,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.321041+0000","last_change":"2024-04-14T15:23:05.574296+0000","last_active":"2024-04-14T15:24:13.321041+0000","last_peered":"2024-04-14T15:24:13.321041+0000","last_clean":"2024-04-14T15:24:13.321041+0000","last_became_active":"2024-04-14T15:09:47.047201+0000","last_became_peered":"2024-04-14T15:09:47.047201+0000","last_unstale":"2024-04-14T15:24:13.321041+0000","last_undegraded":"2024-04-14T15:24:13.321041+0000","last_fullsized":"2024-04-14T15:24:13.321041+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:05.574212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:23:05.574212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:21.008761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"528'37","reported_seq":1484,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.320620+0000","last_change":"2024-04-14T15:23:29.977939+0000","last_active":"2024-04-14T15:24:13.320620+0000","last_peered":"2024-04-14T15:24:13.320620+0000","last_clean":"2024-04-14T15:24:13.320620+0000","last_became_active":"2024-04-14T15:20:53.002282+0000","last_became_peered":"2024-04-14T15:20:53.002282+0000","last_unstale":"2024-04-14T15:24:13.320620+0000","last_undegraded":"2024-04-14T15:24:13.320620+0000","last_fullsized":"2024-04-14T15:24:13.320620+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"528'37","last_scrub_stamp":"2024-04-14T15:23:29.977844+0000","last_deep_scrub":"90'24","last_deep_scrub_stamp":"2024-04-14T15:11:25.120692+0000","last_clean_scrub_stamp":"2024-04-14T15:23:29.977844+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:32.485938+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"91'130","reported_seq":1692,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.317776+0000","last_change":"2024-04-14T15:22:19.518674+0000","last_active":"2024-04-14T15:24:13.317776+0000","last_peered":"2024-04-14T15:24:13.317776+0000","last_clean":"2024-04-14T15:24:13.317776+0000","last_became_active":"2024-04-14T15:22:14.980657+0000","last_became_peered":"2024-04-14T15:22:14.980657+0000","last_unstale":"2024-04-14T15:24:13.317776+0000","last_undegraded":"2024-04-14T15:24:13.317776+0000","last_fullsized":"2024-04-14T15:24:13.317776+0000","mapping_epoch":530,"log_start":"91'130","ondisk_log_start":"91'130","created":19,"last_epoch_clean":599,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:22:19.518583+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:22:19.518583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:13.524978+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":913,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.094158+0000","last_change":"2024-04-14T15:23:29.691160+0000","last_active":"2024-04-14T15:24:14.094158+0000","last_peered":"2024-04-14T15:24:14.094158+0000","last_clean":"2024-04-14T15:24:14.094158+0000","last_became_active":"2024-04-14T15:15:44.885315+0000","last_became_peered":"2024-04-14T15:15:44.885315+0000","last_unstale":"2024-04-14T15:24:14.094158+0000","last_undegraded":"2024-04-14T15:24:14.094158+0000","last_fullsized":"2024-04-14T15:24:14.094158+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:29.691081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:47.627305+0000","last_clean_scrub_stamp":"2024-04-14T15:23:29.691081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:36.414447+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"69'2","reported_seq":1048,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.316164+0000","last_change":"2024-04-14T15:23:44.863113+0000","last_active":"2024-04-14T15:24:13.316164+0000","last_peered":"2024-04-14T15:24:13.316164+0000","last_clean":"2024-04-14T15:24:13.316164+0000","last_became_active":"2024-04-14T15:20:53.002545+0000","last_became_peered":"2024-04-14T15:20:53.002545+0000","last_unstale":"2024-04-14T15:24:13.316164+0000","last_undegraded":"2024-04-14T15:24:13.316164+0000","last_fullsized":"2024-04-14T15:24:13.316164+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-04-14T15:23:44.862962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:23:44.862962+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:47.817158+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0058046169999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":902,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.317335+0000","last_change":"2024-04-14T15:23:33.934414+0000","last_active":"2024-04-14T15:24:13.317335+0000","last_peered":"2024-04-14T15:24:13.317335+0000","last_clean":"2024-04-14T15:24:13.317335+0000","last_became_active":"2024-04-14T15:15:44.887934+0000","last_became_peered":"2024-04-14T15:15:44.887934+0000","last_unstale":"2024-04-14T15:24:13.317335+0000","last_undegraded":"2024-04-14T15:24:13.317335+0000","last_fullsized":"2024-04-14T15:24:13.317335+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:33.934334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:23:33.934334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:56.348377+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"91'130","reported_seq":1692,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.318815+0000","last_change":"2024-04-14T15:22:17.518773+0000","last_active":"2024-04-14T15:24:13.318815+0000","last_peered":"2024-04-14T15:24:13.318815+0000","last_clean":"2024-04-14T15:24:13.318815+0000","last_became_active":"2024-04-14T15:22:06.969269+0000","last_became_peered":"2024-04-14T15:22:06.969269+0000","last_unstale":"2024-04-14T15:24:13.318815+0000","last_undegraded":"2024-04-14T15:24:13.318815+0000","last_fullsized":"2024-04-14T15:24:13.318815+0000","mapping_epoch":189,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:22:17.518666+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:22:17.518666+0000","last_clean_scrub_stamp":"2024-04-14T15:22:17.518666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:13.524959+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":954,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.315593+0000","last_change":"2024-04-14T15:23:03.091708+0000","last_active":"2024-04-14T15:24:13.315593+0000","last_peered":"2024-04-14T15:24:13.315593+0000","last_clean":"2024-04-14T15:24:13.315593+0000","last_became_active":"2024-04-14T15:20:52.986308+0000","last_became_peered":"2024-04-14T15:20:52.986308+0000","last_unstale":"2024-04-14T15:24:13.315593+0000","last_undegraded":"2024-04-14T15:24:13.315593+0000","last_fullsized":"2024-04-14T15:24:13.315593+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:03.091585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:23:03.091585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:17.869054+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1346,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.095664+0000","last_change":"2024-04-14T15:23:05.676756+0000","last_active":"2024-04-14T15:24:14.095664+0000","last_peered":"2024-04-14T15:24:14.095664+0000","last_clean":"2024-04-14T15:24:14.095664+0000","last_became_active":"2024-04-14T15:09:47.049457+0000","last_became_peered":"2024-04-14T15:09:47.049457+0000","last_unstale":"2024-04-14T15:24:14.095664+0000","last_undegraded":"2024-04-14T15:24:14.095664+0000","last_fullsized":"2024-04-14T15:24:14.095664+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:05.676677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:23:05.676677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:17.526473+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"479'22","reported_seq":1385,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.093385+0000","last_change":"2024-04-14T15:23:05.309013+0000","last_active":"2024-04-14T15:24:14.093385+0000","last_peered":"2024-04-14T15:24:14.093385+0000","last_clean":"2024-04-14T15:24:14.093385+0000","last_became_active":"2024-04-14T15:09:45.033883+0000","last_became_peered":"2024-04-14T15:09:45.033883+0000","last_unstale":"2024-04-14T15:24:14.093385+0000","last_undegraded":"2024-04-14T15:24:14.093385+0000","last_fullsized":"2024-04-14T15:24:14.093385+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'22","last_scrub_stamp":"2024-04-14T15:23:05.308902+0000","last_deep_scrub":"308'18","last_deep_scrub_stamp":"2024-04-14T15:16:16.054032+0000","last_clean_scrub_stamp":"2024-04-14T15:23:05.308902+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:32.912046+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1354,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.093721+0000","last_change":"2024-04-14T15:22:08.870838+0000","last_active":"2024-04-14T15:24:14.093721+0000","last_peered":"2024-04-14T15:24:14.093721+0000","last_clean":"2024-04-14T15:24:14.093721+0000","last_became_active":"2024-04-14T15:09:43.021188+0000","last_became_peered":"2024-04-14T15:09:43.021188+0000","last_unstale":"2024-04-14T15:24:14.093721+0000","last_undegraded":"2024-04-14T15:24:14.093721+0000","last_fullsized":"2024-04-14T15:24:14.093721+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:08.870757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:22:08.870757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:17.526287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"73'5","reported_seq":2235,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.091345+0000","last_change":"2024-04-14T15:22:09.278270+0000","last_active":"2024-04-14T15:24:14.091345+0000","last_peered":"2024-04-14T15:24:14.091345+0000","last_clean":"2024-04-14T15:24:14.091345+0000","last_became_active":"2024-04-14T15:20:52.975473+0000","last_became_peered":"2024-04-14T15:20:52.975473+0000","last_unstale":"2024-04-14T15:24:14.091345+0000","last_undegraded":"2024-04-14T15:24:14.091345+0000","last_fullsized":"2024-04-14T15:24:14.091345+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-04-14T15:22:09.278172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:22:09.278172+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:16.232644+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00056542099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":862,"num_read_kb":857,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1341,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.320644+0000","last_change":"2024-04-14T15:23:01.536017+0000","last_active":"2024-04-14T15:24:13.320644+0000","last_peered":"2024-04-14T15:24:13.320644+0000","last_clean":"2024-04-14T15:24:13.320644+0000","last_became_active":"2024-04-14T15:20:52.999464+0000","last_became_peered":"2024-04-14T15:20:52.999464+0000","last_unstale":"2024-04-14T15:24:13.320644+0000","last_undegraded":"2024-04-14T15:24:13.320644+0000","last_fullsized":"2024-04-14T15:24:13.320644+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:01.535941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:20.827589+0000","last_clean_scrub_stamp":"2024-04-14T15:23:01.535941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:27.975040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"91'130","reported_seq":1304,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.290850+0000","last_change":"2024-04-14T15:22:10.976286+0000","last_active":"2024-04-14T15:24:13.290850+0000","last_peered":"2024-04-14T15:24:13.290850+0000","last_clean":"2024-04-14T15:24:13.290850+0000","last_became_active":"2024-04-14T15:22:10.975960+0000","last_became_peered":"2024-04-14T15:22:10.975960+0000","last_unstale":"2024-04-14T15:24:13.290850+0000","last_undegraded":"2024-04-14T15:24:13.290850+0000","last_fullsized":"2024-04-14T15:24:13.290850+0000","mapping_epoch":276,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":595,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:21:58.509331+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:21:58.509331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":947,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.316173+0000","last_change":"2024-04-14T15:22:53.975763+0000","last_active":"2024-04-14T15:24:13.316173+0000","last_peered":"2024-04-14T15:24:13.316173+0000","last_clean":"2024-04-14T15:24:13.316173+0000","last_became_active":"2024-04-14T15:15:21.883717+0000","last_became_peered":"2024-04-14T15:15:21.883717+0000","last_unstale":"2024-04-14T15:24:13.316173+0000","last_undegraded":"2024-04-14T15:24:13.316173+0000","last_fullsized":"2024-04-14T15:24:13.316173+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:53.975656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:22:53.975656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:13.894894+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1338,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.291517+0000","last_change":"2024-04-14T15:22:10.378568+0000","last_active":"2024-04-14T15:24:13.291517+0000","last_peered":"2024-04-14T15:24:13.291517+0000","last_clean":"2024-04-14T15:24:13.291517+0000","last_became_active":"2024-04-14T15:09:47.040185+0000","last_became_peered":"2024-04-14T15:09:47.040185+0000","last_unstale":"2024-04-14T15:24:13.291517+0000","last_undegraded":"2024-04-14T15:24:13.291517+0000","last_fullsized":"2024-04-14T15:24:13.291517+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:10.378485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:53.731610+0000","last_clean_scrub_stamp":"2024-04-14T15:22:10.378485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"479'40","reported_seq":1588,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.091923+0000","last_change":"2024-04-14T15:23:31.198816+0000","last_active":"2024-04-14T15:24:14.091923+0000","last_peered":"2024-04-14T15:24:14.091923+0000","last_clean":"2024-04-14T15:24:14.091923+0000","last_became_active":"2024-04-14T15:09:45.034168+0000","last_became_peered":"2024-04-14T15:09:45.034168+0000","last_unstale":"2024-04-14T15:24:14.091923+0000","last_undegraded":"2024-04-14T15:24:14.091923+0000","last_fullsized":"2024-04-14T15:24:14.091923+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'40","last_scrub_stamp":"2024-04-14T15:23:31.198708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:23:31.198708+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:42.421703+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":916,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.091449+0000","last_change":"2024-04-14T15:23:30.217883+0000","last_active":"2024-04-14T15:24:14.091449+0000","last_peered":"2024-04-14T15:24:14.091449+0000","last_clean":"2024-04-14T15:24:14.091449+0000","last_became_active":"2024-04-14T15:20:52.979943+0000","last_became_peered":"2024-04-14T15:20:52.979943+0000","last_unstale":"2024-04-14T15:24:14.091449+0000","last_undegraded":"2024-04-14T15:24:14.091449+0000","last_fullsized":"2024-04-14T15:24:14.091449+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:30.217808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:24.066275+0000","last_clean_scrub_stamp":"2024-04-14T15:23:30.217808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:51.962682+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1000,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.317258+0000","last_change":"2024-04-14T15:21:21.103290+0000","last_active":"2024-04-14T15:24:13.317258+0000","last_peered":"2024-04-14T15:24:13.317258+0000","last_clean":"2024-04-14T15:24:13.317258+0000","last_became_active":"2024-04-14T15:15:21.874742+0000","last_became_peered":"2024-04-14T15:15:21.874742+0000","last_unstale":"2024-04-14T15:24:13.317258+0000","last_undegraded":"2024-04-14T15:24:13.317258+0000","last_fullsized":"2024-04-14T15:24:13.317258+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:21.103213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:44.084010+0000","last_clean_scrub_stamp":"2024-04-14T15:21:21.103213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:15.809946+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"91'130","reported_seq":761,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.317065+0000","last_change":"2024-04-14T15:22:18.995181+0000","last_active":"2024-04-14T15:24:13.317065+0000","last_peered":"2024-04-14T15:24:13.317065+0000","last_clean":"2024-04-14T15:24:13.317065+0000","last_became_active":"2024-04-14T15:22:18.993883+0000","last_became_peered":"2024-04-14T15:22:18.993883+0000","last_unstale":"2024-04-14T15:24:13.317065+0000","last_undegraded":"2024-04-14T15:24:13.317065+0000","last_fullsized":"2024-04-14T15:24:13.317065+0000","mapping_epoch":530,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:21:11.055159+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:21:11.055159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:15.809846+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":1257,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.095699+0000","last_change":"2024-04-14T15:22:51.765619+0000","last_active":"2024-04-14T15:24:14.095699+0000","last_peered":"2024-04-14T15:24:14.095699+0000","last_clean":"2024-04-14T15:24:14.095699+0000","last_became_active":"2024-04-14T15:20:52.989272+0000","last_became_peered":"2024-04-14T15:20:52.989272+0000","last_unstale":"2024-04-14T15:24:14.095699+0000","last_undegraded":"2024-04-14T15:24:14.095699+0000","last_fullsized":"2024-04-14T15:24:14.095699+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:51.765499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:22:51.765499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:17.526457+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"528'9","reported_seq":1539,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.291118+0000","last_change":"2024-04-14T15:22:53.253431+0000","last_active":"2024-04-14T15:24:13.291118+0000","last_peered":"2024-04-14T15:24:13.291118+0000","last_clean":"2024-04-14T15:24:13.291118+0000","last_became_active":"2024-04-14T15:20:53.000669+0000","last_became_peered":"2024-04-14T15:20:53.000669+0000","last_unstale":"2024-04-14T15:24:13.291118+0000","last_undegraded":"2024-04-14T15:24:13.291118+0000","last_fullsized":"2024-04-14T15:24:13.291118+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"528'9","last_scrub_stamp":"2024-04-14T15:22:53.253302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:22:53.253302+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:17.455584+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"479'20","reported_seq":354,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.091153+0000","last_change":"2024-04-14T15:22:13.333897+0000","last_active":"2024-04-14T15:24:14.091153+0000","last_peered":"2024-04-14T15:24:14.091153+0000","last_clean":"2024-04-14T15:24:14.091153+0000","last_became_active":"2024-04-14T15:21:16.034027+0000","last_became_peered":"2024-04-14T15:21:16.034027+0000","last_unstale":"2024-04-14T15:24:14.091153+0000","last_undegraded":"2024-04-14T15:24:14.091153+0000","last_fullsized":"2024-04-14T15:24:14.091153+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'20","last_scrub_stamp":"2024-04-14T15:22:13.333807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:22:13.333807+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:16.232610+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1343,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.091874+0000","last_change":"2024-04-14T15:21:28.684960+0000","last_active":"2024-04-14T15:24:14.091874+0000","last_peered":"2024-04-14T15:24:14.091874+0000","last_clean":"2024-04-14T15:24:14.091874+0000","last_became_active":"2024-04-14T15:09:43.022597+0000","last_became_peered":"2024-04-14T15:09:43.022597+0000","last_unstale":"2024-04-14T15:24:14.091874+0000","last_undegraded":"2024-04-14T15:24:14.091874+0000","last_fullsized":"2024-04-14T15:24:14.091874+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:28.684818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:21:28.684818+0000","last_clean_scrub_stamp":"2024-04-14T15:21:28.684818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:16.232546+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1335,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.290452+0000","last_change":"2024-04-14T15:22:13.358361+0000","last_active":"2024-04-14T15:24:13.290452+0000","last_peered":"2024-04-14T15:24:13.290452+0000","last_clean":"2024-04-14T15:24:13.290452+0000","last_became_active":"2024-04-14T15:09:49.066648+0000","last_became_peered":"2024-04-14T15:09:49.066648+0000","last_unstale":"2024-04-14T15:24:13.290452+0000","last_undegraded":"2024-04-14T15:24:13.290452+0000","last_fullsized":"2024-04-14T15:24:13.290452+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:13.358256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:22:13.358256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":1251,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.291466+0000","last_change":"2024-04-14T15:23:44.358966+0000","last_active":"2024-04-14T15:24:13.291466+0000","last_peered":"2024-04-14T15:24:13.291466+0000","last_clean":"2024-04-14T15:24:13.291466+0000","last_became_active":"2024-04-14T15:12:56.563074+0000","last_became_peered":"2024-04-14T15:12:56.563074+0000","last_unstale":"2024-04-14T15:24:13.291466+0000","last_undegraded":"2024-04-14T15:24:13.291466+0000","last_fullsized":"2024-04-14T15:24:13.291466+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:44.358887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:23:44.358887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:52.408476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1338,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.092702+0000","last_change":"2024-04-14T15:21:14.769138+0000","last_active":"2024-04-14T15:24:14.092702+0000","last_peered":"2024-04-14T15:24:14.092702+0000","last_clean":"2024-04-14T15:24:14.092702+0000","last_became_active":"2024-04-14T15:09:47.049005+0000","last_became_peered":"2024-04-14T15:09:47.049005+0000","last_unstale":"2024-04-14T15:24:14.092702+0000","last_undegraded":"2024-04-14T15:24:14.092702+0000","last_fullsized":"2024-04-14T15:24:14.092702+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:14.769006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:56.915544+0000","last_clean_scrub_stamp":"2024-04-14T15:21:14.769006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:17.526202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"479'23","reported_seq":1380,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.318055+0000","last_change":"2024-04-14T15:19:44.525830+0000","last_active":"2024-04-14T15:24:13.318055+0000","last_peered":"2024-04-14T15:24:13.318055+0000","last_clean":"2024-04-14T15:24:13.318055+0000","last_became_active":"2024-04-14T15:09:45.032892+0000","last_became_peered":"2024-04-14T15:09:45.032892+0000","last_unstale":"2024-04-14T15:24:13.318055+0000","last_undegraded":"2024-04-14T15:24:13.318055+0000","last_fullsized":"2024-04-14T15:24:13.318055+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'17","last_scrub_stamp":"2024-04-14T15:19:44.525712+0000","last_deep_scrub":"183'17","last_deep_scrub_stamp":"2024-04-14T15:14:11.577574+0000","last_clean_scrub_stamp":"2024-04-14T15:19:44.525712+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"677'6","reported_seq":1368,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.319539+0000","last_change":"2024-04-14T15:23:38.773659+0000","last_active":"2024-04-14T15:24:13.319539+0000","last_peered":"2024-04-14T15:24:13.319539+0000","last_clean":"2024-04-14T15:24:13.319539+0000","last_became_active":"2024-04-14T15:09:43.020490+0000","last_became_peered":"2024-04-14T15:09:43.020490+0000","last_unstale":"2024-04-14T15:24:13.319539+0000","last_undegraded":"2024-04-14T15:24:13.319539+0000","last_fullsized":"2024-04-14T15:24:13.319539+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"677'6","last_scrub_stamp":"2024-04-14T15:23:38.773539+0000","last_deep_scrub":"677'6","last_deep_scrub_stamp":"2024-04-14T15:23:38.773539+0000","last_clean_scrub_stamp":"2024-04-14T15:23:38.773539+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:04.801057+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":528,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1007,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.316622+0000","last_change":"2024-04-14T15:23:40.854331+0000","last_active":"2024-04-14T15:24:13.316622+0000","last_peered":"2024-04-14T15:24:13.316622+0000","last_clean":"2024-04-14T15:24:13.316622+0000","last_became_active":"2024-04-14T15:15:21.876410+0000","last_became_peered":"2024-04-14T15:15:21.876410+0000","last_unstale":"2024-04-14T15:24:13.316622+0000","last_undegraded":"2024-04-14T15:24:13.316622+0000","last_fullsized":"2024-04-14T15:24:13.316622+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:40.854253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:23:40.854253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:10.566240+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":1250,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.290819+0000","last_change":"2024-04-14T15:22:18.299434+0000","last_active":"2024-04-14T15:24:13.290819+0000","last_peered":"2024-04-14T15:24:13.290819+0000","last_clean":"2024-04-14T15:24:13.290819+0000","last_became_active":"2024-04-14T15:12:56.551177+0000","last_became_peered":"2024-04-14T15:12:56.551177+0000","last_unstale":"2024-04-14T15:24:13.290819+0000","last_undegraded":"2024-04-14T15:24:13.290819+0000","last_fullsized":"2024-04-14T15:24:13.290819+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:18.299330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:22:18.299330+0000","last_clean_scrub_stamp":"2024-04-14T15:22:18.299330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"479'19","reported_seq":1370,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.291776+0000","last_change":"2024-04-14T15:22:55.321370+0000","last_active":"2024-04-14T15:24:13.291776+0000","last_peered":"2024-04-14T15:24:13.291776+0000","last_clean":"2024-04-14T15:24:13.291776+0000","last_became_active":"2024-04-14T15:09:45.180375+0000","last_became_peered":"2024-04-14T15:09:45.180375+0000","last_unstale":"2024-04-14T15:24:13.291776+0000","last_undegraded":"2024-04-14T15:24:13.291776+0000","last_fullsized":"2024-04-14T15:24:13.291776+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'19","last_scrub_stamp":"2024-04-14T15:22:55.321247+0000","last_deep_scrub":"347'13","last_deep_scrub_stamp":"2024-04-14T15:16:55.694599+0000","last_clean_scrub_stamp":"2024-04-14T15:22:55.321247+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":1354,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.092726+0000","last_change":"2024-04-14T15:22:16.961934+0000","last_active":"2024-04-14T15:24:14.092726+0000","last_peered":"2024-04-14T15:24:14.092726+0000","last_clean":"2024-04-14T15:24:14.092726+0000","last_became_active":"2024-04-14T15:09:43.016788+0000","last_became_peered":"2024-04-14T15:09:43.016788+0000","last_unstale":"2024-04-14T15:24:14.092726+0000","last_undegraded":"2024-04-14T15:24:14.092726+0000","last_fullsized":"2024-04-14T15:24:14.092726+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:16.961846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:22:16.961846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:17.526427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1019,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.317300+0000","last_change":"2024-04-14T15:21:29.095634+0000","last_active":"2024-04-14T15:24:13.317300+0000","last_peered":"2024-04-14T15:24:13.317300+0000","last_clean":"2024-04-14T15:24:13.317300+0000","last_became_active":"2024-04-14T15:15:21.873580+0000","last_became_peered":"2024-04-14T15:15:21.873580+0000","last_unstale":"2024-04-14T15:24:13.317300+0000","last_undegraded":"2024-04-14T15:24:13.317300+0000","last_fullsized":"2024-04-14T15:24:13.317300+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:29.095557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:21:29.095557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:15.810001+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1342,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.094642+0000","last_change":"2024-04-14T15:22:12.923356+0000","last_active":"2024-04-14T15:24:14.094642+0000","last_peered":"2024-04-14T15:24:14.094642+0000","last_clean":"2024-04-14T15:24:14.094642+0000","last_became_active":"2024-04-14T15:09:49.064296+0000","last_became_peered":"2024-04-14T15:09:49.064296+0000","last_unstale":"2024-04-14T15:24:14.094642+0000","last_undegraded":"2024-04-14T15:24:14.094642+0000","last_fullsized":"2024-04-14T15:24:14.094642+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:12.923270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:22:12.923270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:17.526397+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":1256,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.092602+0000","last_change":"2024-04-14T15:23:32.648604+0000","last_active":"2024-04-14T15:24:14.092602+0000","last_peered":"2024-04-14T15:24:14.092602+0000","last_clean":"2024-04-14T15:24:14.092602+0000","last_became_active":"2024-04-14T15:20:52.977894+0000","last_became_peered":"2024-04-14T15:20:52.977894+0000","last_unstale":"2024-04-14T15:24:14.092602+0000","last_undegraded":"2024-04-14T15:24:14.092602+0000","last_fullsized":"2024-04-14T15:24:14.092602+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:32.648522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:23:32.648522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:53.725757+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.17","version":"479'12","reported_seq":1356,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.317280+0000","last_change":"2024-04-14T15:20:41.939842+0000","last_active":"2024-04-14T15:24:13.317280+0000","last_peered":"2024-04-14T15:24:13.317280+0000","last_clean":"2024-04-14T15:24:13.317280+0000","last_became_active":"2024-04-14T15:09:45.180037+0000","last_became_peered":"2024-04-14T15:09:45.180037+0000","last_unstale":"2024-04-14T15:24:13.317280+0000","last_undegraded":"2024-04-14T15:24:13.317280+0000","last_fullsized":"2024-04-14T15:24:13.317280+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'12","last_scrub_stamp":"2024-04-14T15:20:41.939761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:20:41.939761+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:15.809792+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":387,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.317965+0000","last_change":"2024-04-14T15:23:40.765638+0000","last_active":"2024-04-14T15:24:13.317965+0000","last_peered":"2024-04-14T15:24:13.317965+0000","last_clean":"2024-04-14T15:24:13.317965+0000","last_became_active":"2024-04-14T15:20:52.986974+0000","last_became_peered":"2024-04-14T15:20:52.986974+0000","last_unstale":"2024-04-14T15:24:13.317965+0000","last_undegraded":"2024-04-14T15:24:13.317965+0000","last_fullsized":"2024-04-14T15:24:13.317965+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:40.765537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:59.640565+0000","last_clean_scrub_stamp":"2024-04-14T15:23:40.765537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:06.708615+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":893,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.317111+0000","last_change":"2024-04-14T15:20:44.996397+0000","last_active":"2024-04-14T15:24:13.317111+0000","last_peered":"2024-04-14T15:24:13.317111+0000","last_clean":"2024-04-14T15:24:13.317111+0000","last_became_active":"2024-04-14T15:15:44.889786+0000","last_became_peered":"2024-04-14T15:15:44.889786+0000","last_unstale":"2024-04-14T15:24:13.317111+0000","last_undegraded":"2024-04-14T15:24:13.317111+0000","last_fullsized":"2024-04-14T15:24:13.317111+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:44.996260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:13.735024+0000","last_clean_scrub_stamp":"2024-04-14T15:20:44.996260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:15.809811+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1340,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.320976+0000","last_change":"2024-04-14T15:23:29.527743+0000","last_active":"2024-04-14T15:24:13.320976+0000","last_peered":"2024-04-14T15:24:13.320976+0000","last_clean":"2024-04-14T15:24:13.320976+0000","last_became_active":"2024-04-14T15:20:52.985136+0000","last_became_peered":"2024-04-14T15:20:52.985136+0000","last_unstale":"2024-04-14T15:24:13.320976+0000","last_undegraded":"2024-04-14T15:24:13.320976+0000","last_fullsized":"2024-04-14T15:24:13.320976+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:29.527656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:23:29.527656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:53.193549+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":1251,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.318698+0000","last_change":"2024-04-14T15:22:53.207244+0000","last_active":"2024-04-14T15:24:13.318698+0000","last_peered":"2024-04-14T15:24:13.318698+0000","last_clean":"2024-04-14T15:24:13.318698+0000","last_became_active":"2024-04-14T15:12:56.203577+0000","last_became_peered":"2024-04-14T15:12:56.203577+0000","last_unstale":"2024-04-14T15:24:13.318698+0000","last_undegraded":"2024-04-14T15:24:13.318698+0000","last_fullsized":"2024-04-14T15:24:13.318698+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:53.207140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:46.369231+0000","last_clean_scrub_stamp":"2024-04-14T15:22:53.207140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"479'20","reported_seq":1053,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.316262+0000","last_change":"2024-04-14T15:22:01.194496+0000","last_active":"2024-04-14T15:24:13.316262+0000","last_peered":"2024-04-14T15:24:13.316262+0000","last_clean":"2024-04-14T15:24:13.316262+0000","last_became_active":"2024-04-14T15:15:21.877667+0000","last_became_peered":"2024-04-14T15:15:21.877667+0000","last_unstale":"2024-04-14T15:24:13.316262+0000","last_undegraded":"2024-04-14T15:24:13.316262+0000","last_fullsized":"2024-04-14T15:24:13.316262+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'20","last_scrub_stamp":"2024-04-14T15:22:01.194418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:22:01.194418+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:15.810053+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":901,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.316337+0000","last_change":"2024-04-14T15:23:32.000531+0000","last_active":"2024-04-14T15:24:13.316337+0000","last_peered":"2024-04-14T15:24:13.316337+0000","last_clean":"2024-04-14T15:24:13.316337+0000","last_became_active":"2024-04-14T15:15:44.887452+0000","last_became_peered":"2024-04-14T15:15:44.887452+0000","last_unstale":"2024-04-14T15:24:13.316337+0000","last_undegraded":"2024-04-14T15:24:13.316337+0000","last_fullsized":"2024-04-14T15:24:13.316337+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:32.000401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:23:32.000401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:38.654484+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"280'9","reported_seq":1028,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.092389+0000","last_change":"2024-04-14T15:22:01.181123+0000","last_active":"2024-04-14T15:24:14.092389+0000","last_peered":"2024-04-14T15:24:14.092389+0000","last_clean":"2024-04-14T15:24:14.092389+0000","last_became_active":"2024-04-14T15:15:44.882916+0000","last_became_peered":"2024-04-14T15:15:44.882916+0000","last_unstale":"2024-04-14T15:24:14.092389+0000","last_undegraded":"2024-04-14T15:24:14.092389+0000","last_fullsized":"2024-04-14T15:24:14.092389+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"280'9","last_scrub_stamp":"2024-04-14T15:22:01.181035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:22:01.181035+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:16.232578+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1333,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.291652+0000","last_change":"2024-04-14T15:23:29.304889+0000","last_active":"2024-04-14T15:24:13.291652+0000","last_peered":"2024-04-14T15:24:13.291652+0000","last_clean":"2024-04-14T15:24:13.291652+0000","last_became_active":"2024-04-14T15:09:49.066068+0000","last_became_peered":"2024-04-14T15:09:49.066068+0000","last_unstale":"2024-04-14T15:24:13.291652+0000","last_undegraded":"2024-04-14T15:24:13.291652+0000","last_fullsized":"2024-04-14T15:24:13.291652+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:29.304812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:54.724681+0000","last_clean_scrub_stamp":"2024-04-14T15:23:29.304812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:39.703795+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":893,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.315996+0000","last_change":"2024-04-14T15:21:24.134615+0000","last_active":"2024-04-14T15:24:13.315996+0000","last_peered":"2024-04-14T15:24:13.315996+0000","last_clean":"2024-04-14T15:24:13.315996+0000","last_became_active":"2024-04-14T15:15:44.889106+0000","last_became_peered":"2024-04-14T15:15:44.889106+0000","last_unstale":"2024-04-14T15:24:13.315996+0000","last_undegraded":"2024-04-14T15:24:13.315996+0000","last_fullsized":"2024-04-14T15:24:13.315996+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:24.134540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:21:24.134540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:15.809881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"479'19","reported_seq":1209,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.320789+0000","last_change":"2024-04-14T15:23:46.548415+0000","last_active":"2024-04-14T15:24:13.320789+0000","last_peered":"2024-04-14T15:24:13.320789+0000","last_clean":"2024-04-14T15:24:13.320789+0000","last_became_active":"2024-04-14T15:22:25.971104+0000","last_became_peered":"2024-04-14T15:22:25.971104+0000","last_unstale":"2024-04-14T15:24:13.320789+0000","last_undegraded":"2024-04-14T15:24:13.320789+0000","last_fullsized":"2024-04-14T15:24:13.320789+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'19","last_scrub_stamp":"2024-04-14T15:23:46.548255+0000","last_deep_scrub":"183'13","last_deep_scrub_stamp":"2024-04-14T15:14:11.755252+0000","last_clean_scrub_stamp":"2024-04-14T15:23:46.548255+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:55.797274+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1354,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.291299+0000","last_change":"2024-04-14T15:23:28.265599+0000","last_active":"2024-04-14T15:24:13.291299+0000","last_peered":"2024-04-14T15:24:13.291299+0000","last_clean":"2024-04-14T15:24:13.291299+0000","last_became_active":"2024-04-14T15:09:43.019301+0000","last_became_peered":"2024-04-14T15:09:43.019301+0000","last_unstale":"2024-04-14T15:24:13.291299+0000","last_undegraded":"2024-04-14T15:24:13.291299+0000","last_fullsized":"2024-04-14T15:24:13.291299+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:28.265471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:23:28.265471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:28.508263+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"255'10","reported_seq":1165,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.316521+0000","last_change":"2024-04-14T15:23:04.116217+0000","last_active":"2024-04-14T15:24:13.316521+0000","last_peered":"2024-04-14T15:24:13.316521+0000","last_clean":"2024-04-14T15:24:13.316521+0000","last_became_active":"2024-04-14T15:15:21.884468+0000","last_became_peered":"2024-04-14T15:15:21.884468+0000","last_unstale":"2024-04-14T15:24:13.316521+0000","last_undegraded":"2024-04-14T15:24:13.316521+0000","last_fullsized":"2024-04-14T15:24:13.316521+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'10","last_scrub_stamp":"2024-04-14T15:23:04.116126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:23:04.116126+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:23.810752+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1326,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.316424+0000","last_change":"2024-04-14T15:19:15.642110+0000","last_active":"2024-04-14T15:24:13.316424+0000","last_peered":"2024-04-14T15:24:13.316424+0000","last_clean":"2024-04-14T15:24:13.316424+0000","last_became_active":"2024-04-14T15:09:49.065242+0000","last_became_peered":"2024-04-14T15:09:49.065242+0000","last_unstale":"2024-04-14T15:24:13.316424+0000","last_undegraded":"2024-04-14T15:24:13.316424+0000","last_fullsized":"2024-04-14T15:24:13.316424+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:15.642017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:19:15.642017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":1243,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.320962+0000","last_change":"2024-04-14T15:21:16.642973+0000","last_active":"2024-04-14T15:24:13.320962+0000","last_peered":"2024-04-14T15:24:13.320962+0000","last_clean":"2024-04-14T15:24:13.320962+0000","last_became_active":"2024-04-14T15:12:56.389204+0000","last_became_peered":"2024-04-14T15:12:56.389204+0000","last_unstale":"2024-04-14T15:24:13.320962+0000","last_undegraded":"2024-04-14T15:24:13.320962+0000","last_fullsized":"2024-04-14T15:24:13.320962+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:16.642880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:21:16.642880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:13.524801+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"479'19","reported_seq":805,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.094311+0000","last_change":"2024-04-14T15:21:25.737152+0000","last_active":"2024-04-14T15:24:14.094311+0000","last_peered":"2024-04-14T15:24:14.094311+0000","last_clean":"2024-04-14T15:24:14.094311+0000","last_became_active":"2024-04-14T15:20:52.990622+0000","last_became_peered":"2024-04-14T15:20:52.990622+0000","last_unstale":"2024-04-14T15:24:14.094311+0000","last_undegraded":"2024-04-14T15:24:14.094311+0000","last_fullsized":"2024-04-14T15:24:14.094311+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'19","last_scrub_stamp":"2024-04-14T15:21:25.737057+0000","last_deep_scrub":"344'13","last_deep_scrub_stamp":"2024-04-14T15:16:52.738163+0000","last_clean_scrub_stamp":"2024-04-14T15:21:25.737057+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:17.526237+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":385,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.316363+0000","last_change":"2024-04-14T15:22:55.001625+0000","last_active":"2024-04-14T15:24:13.316363+0000","last_peered":"2024-04-14T15:24:13.316363+0000","last_clean":"2024-04-14T15:24:13.316363+0000","last_became_active":"2024-04-14T15:20:52.996548+0000","last_became_peered":"2024-04-14T15:20:52.996548+0000","last_unstale":"2024-04-14T15:24:13.316363+0000","last_undegraded":"2024-04-14T15:24:13.316363+0000","last_fullsized":"2024-04-14T15:24:13.316363+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:55.001498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:22:55.001498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:14.779847+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1011,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.316584+0000","last_change":"2024-04-14T15:21:23.131643+0000","last_active":"2024-04-14T15:24:13.316584+0000","last_peered":"2024-04-14T15:24:13.316584+0000","last_clean":"2024-04-14T15:24:13.316584+0000","last_became_active":"2024-04-14T15:15:21.882643+0000","last_became_peered":"2024-04-14T15:15:21.882643+0000","last_unstale":"2024-04-14T15:24:13.316584+0000","last_undegraded":"2024-04-14T15:24:13.316584+0000","last_fullsized":"2024-04-14T15:24:13.316584+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:23.131512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:33.485411+0000","last_clean_scrub_stamp":"2024-04-14T15:21:23.131512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:15.809962+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1337,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.290553+0000","last_change":"2024-04-14T15:21:25.392328+0000","last_active":"2024-04-14T15:24:13.290553+0000","last_peered":"2024-04-14T15:24:13.290553+0000","last_clean":"2024-04-14T15:24:13.290553+0000","last_became_active":"2024-04-14T15:09:49.059711+0000","last_became_peered":"2024-04-14T15:09:49.059711+0000","last_unstale":"2024-04-14T15:24:13.290553+0000","last_undegraded":"2024-04-14T15:24:13.290553+0000","last_fullsized":"2024-04-14T15:24:13.290553+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:25.392246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:21:25.392246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":393,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.320621+0000","last_change":"2024-04-14T15:23:03.603742+0000","last_active":"2024-04-14T15:24:13.320621+0000","last_peered":"2024-04-14T15:24:13.320621+0000","last_clean":"2024-04-14T15:24:13.320621+0000","last_became_active":"2024-04-14T15:22:26.203388+0000","last_became_peered":"2024-04-14T15:22:26.203388+0000","last_unstale":"2024-04-14T15:24:13.320621+0000","last_undegraded":"2024-04-14T15:24:13.320621+0000","last_fullsized":"2024-04-14T15:24:13.320621+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:03.603663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:23:03.603663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:20.384580+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"479'24","reported_seq":1389,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.318931+0000","last_change":"2024-04-14T15:20:53.071526+0000","last_active":"2024-04-14T15:24:13.318931+0000","last_peered":"2024-04-14T15:24:13.318931+0000","last_clean":"2024-04-14T15:24:13.318931+0000","last_became_active":"2024-04-14T15:20:52.989300+0000","last_became_peered":"2024-04-14T15:20:52.989300+0000","last_unstale":"2024-04-14T15:24:13.318931+0000","last_undegraded":"2024-04-14T15:24:13.318931+0000","last_fullsized":"2024-04-14T15:24:13.318931+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"464'16","last_scrub_stamp":"2024-04-14T15:19:17.589999+0000","last_deep_scrub":"464'16","last_deep_scrub_stamp":"2024-04-14T15:19:17.589999+0000","last_clean_scrub_stamp":"2024-04-14T15:19:17.589999+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":1351,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.092454+0000","last_change":"2024-04-14T15:20:45.587983+0000","last_active":"2024-04-14T15:24:14.092454+0000","last_peered":"2024-04-14T15:24:14.092454+0000","last_clean":"2024-04-14T15:24:14.092454+0000","last_became_active":"2024-04-14T15:09:43.023583+0000","last_became_peered":"2024-04-14T15:09:43.023583+0000","last_unstale":"2024-04-14T15:24:14.092454+0000","last_undegraded":"2024-04-14T15:24:14.092454+0000","last_fullsized":"2024-04-14T15:24:14.092454+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:45.587898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:20:45.587898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:16.232494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1335,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.091716+0000","last_change":"2024-04-14T15:20:01.971662+0000","last_active":"2024-04-14T15:24:14.091716+0000","last_peered":"2024-04-14T15:24:14.091716+0000","last_clean":"2024-04-14T15:24:14.091716+0000","last_became_active":"2024-04-14T15:09:47.041683+0000","last_became_peered":"2024-04-14T15:09:47.041683+0000","last_unstale":"2024-04-14T15:24:14.091716+0000","last_undegraded":"2024-04-14T15:24:14.091716+0000","last_fullsized":"2024-04-14T15:24:14.091716+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:01.971564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:08.486217+0000","last_clean_scrub_stamp":"2024-04-14T15:20:01.971564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:16.232429+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1015,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.315597+0000","last_change":"2024-04-14T15:23:28.009795+0000","last_active":"2024-04-14T15:24:13.315597+0000","last_peered":"2024-04-14T15:24:13.315597+0000","last_clean":"2024-04-14T15:24:13.315597+0000","last_became_active":"2024-04-14T15:15:21.889567+0000","last_became_peered":"2024-04-14T15:15:21.889567+0000","last_unstale":"2024-04-14T15:24:13.315597+0000","last_undegraded":"2024-04-14T15:24:13.315597+0000","last_fullsized":"2024-04-14T15:24:13.315597+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:28.009713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:23:28.009713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:47.266664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":1251,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.095215+0000","last_change":"2024-04-14T15:22:19.933335+0000","last_active":"2024-04-14T15:24:14.095215+0000","last_peered":"2024-04-14T15:24:14.095215+0000","last_clean":"2024-04-14T15:24:14.095215+0000","last_became_active":"2024-04-14T15:12:56.387723+0000","last_became_peered":"2024-04-14T15:12:56.387723+0000","last_unstale":"2024-04-14T15:24:14.095215+0000","last_undegraded":"2024-04-14T15:24:14.095215+0000","last_fullsized":"2024-04-14T15:24:14.095215+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:19.933257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:03.685045+0000","last_clean_scrub_stamp":"2024-04-14T15:22:19.933257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:17.526333+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.10","version":"479'8","reported_seq":1363,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.315527+0000","last_change":"2024-04-14T15:21:20.054070+0000","last_active":"2024-04-14T15:24:13.315527+0000","last_peered":"2024-04-14T15:24:13.315527+0000","last_clean":"2024-04-14T15:24:13.315527+0000","last_became_active":"2024-04-14T15:20:53.001300+0000","last_became_peered":"2024-04-14T15:20:53.001300+0000","last_unstale":"2024-04-14T15:24:13.315527+0000","last_undegraded":"2024-04-14T15:24:13.315527+0000","last_fullsized":"2024-04-14T15:24:13.315527+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'8","last_scrub_stamp":"2024-04-14T15:21:20.053929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:21:20.053929+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:15.809864+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":390,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.095660+0000","last_change":"2024-04-14T15:23:43.407976+0000","last_active":"2024-04-14T15:24:14.095660+0000","last_peered":"2024-04-14T15:24:14.095660+0000","last_clean":"2024-04-14T15:24:14.095660+0000","last_became_active":"2024-04-14T15:20:52.979496+0000","last_became_peered":"2024-04-14T15:20:52.979496+0000","last_unstale":"2024-04-14T15:24:14.095660+0000","last_undegraded":"2024-04-14T15:24:14.095660+0000","last_fullsized":"2024-04-14T15:24:14.095660+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:43.407897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:08.712289+0000","last_clean_scrub_stamp":"2024-04-14T15:23:43.407897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:47.213727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":379,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.317865+0000","last_change":"2024-04-14T15:20:52.991440+0000","last_active":"2024-04-14T15:24:13.317865+0000","last_peered":"2024-04-14T15:24:13.317865+0000","last_clean":"2024-04-14T15:24:13.317865+0000","last_became_active":"2024-04-14T15:20:52.990347+0000","last_became_peered":"2024-04-14T15:20:52.990347+0000","last_unstale":"2024-04-14T15:24:13.317865+0000","last_undegraded":"2024-04-14T15:24:13.317865+0000","last_fullsized":"2024-04-14T15:24:13.317865+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:48.867704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:19:48.867704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1327,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.316109+0000","last_change":"2024-04-14T15:19:59.634301+0000","last_active":"2024-04-14T15:24:13.316109+0000","last_peered":"2024-04-14T15:24:13.316109+0000","last_clean":"2024-04-14T15:24:13.316109+0000","last_became_active":"2024-04-14T15:09:49.061359+0000","last_became_peered":"2024-04-14T15:09:49.061359+0000","last_unstale":"2024-04-14T15:24:13.316109+0000","last_undegraded":"2024-04-14T15:24:13.316109+0000","last_fullsized":"2024-04-14T15:24:13.316109+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:59.634212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:19:59.634212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:13.524690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":1250,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.320722+0000","last_change":"2024-04-14T15:21:28.711361+0000","last_active":"2024-04-14T15:24:13.320722+0000","last_peered":"2024-04-14T15:24:13.320722+0000","last_clean":"2024-04-14T15:24:13.320722+0000","last_became_active":"2024-04-14T15:20:52.999623+0000","last_became_peered":"2024-04-14T15:20:52.999623+0000","last_unstale":"2024-04-14T15:24:13.320722+0000","last_undegraded":"2024-04-14T15:24:13.320722+0000","last_fullsized":"2024-04-14T15:24:13.320722+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:28.711266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:15:43.680852+0000","last_clean_scrub_stamp":"2024-04-14T15:21:28.711266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:13.524868+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"479'24","reported_seq":1030,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.315967+0000","last_change":"2024-04-14T15:21:13.011149+0000","last_active":"2024-04-14T15:24:13.315967+0000","last_peered":"2024-04-14T15:24:13.315967+0000","last_clean":"2024-04-14T15:24:13.315967+0000","last_became_active":"2024-04-14T15:20:52.996917+0000","last_became_peered":"2024-04-14T15:20:52.996917+0000","last_unstale":"2024-04-14T15:24:13.315967+0000","last_undegraded":"2024-04-14T15:24:13.315967+0000","last_fullsized":"2024-04-14T15:24:13.315967+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'24","last_scrub_stamp":"2024-04-14T15:21:13.011012+0000","last_deep_scrub":"102'16","last_deep_scrub_stamp":"2024-04-14T15:12:35.999326+0000","last_clean_scrub_stamp":"2024-04-14T15:21:13.011012+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:15.809931+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":387,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.095240+0000","last_change":"2024-04-14T15:22:11.889589+0000","last_active":"2024-04-14T15:24:14.095240+0000","last_peered":"2024-04-14T15:24:14.095240+0000","last_clean":"2024-04-14T15:24:14.095240+0000","last_became_active":"2024-04-14T15:20:52.970959+0000","last_became_peered":"2024-04-14T15:20:52.970959+0000","last_unstale":"2024-04-14T15:24:14.095240+0000","last_undegraded":"2024-04-14T15:24:14.095240+0000","last_fullsized":"2024-04-14T15:24:14.095240+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:11.889509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:22:11.889509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:17.526318+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1337,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.317237+0000","last_change":"2024-04-14T15:20:52.984259+0000","last_active":"2024-04-14T15:24:13.317237+0000","last_peered":"2024-04-14T15:24:13.317237+0000","last_clean":"2024-04-14T15:24:13.317237+0000","last_became_active":"2024-04-14T15:20:52.983919+0000","last_became_peered":"2024-04-14T15:20:52.983919+0000","last_unstale":"2024-04-14T15:24:13.317237+0000","last_undegraded":"2024-04-14T15:24:13.317237+0000","last_fullsized":"2024-04-14T15:24:13.317237+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:54.634279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:19:54.634279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:13.524655+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1021,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.316854+0000","last_change":"2024-04-14T15:22:09.053629+0000","last_active":"2024-04-14T15:24:13.316854+0000","last_peered":"2024-04-14T15:24:13.316854+0000","last_clean":"2024-04-14T15:24:13.316854+0000","last_became_active":"2024-04-14T15:20:52.993918+0000","last_became_peered":"2024-04-14T15:20:52.993918+0000","last_unstale":"2024-04-14T15:24:13.316854+0000","last_undegraded":"2024-04-14T15:24:13.316854+0000","last_fullsized":"2024-04-14T15:24:13.316854+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:09.053546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:22:09.053546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:15.810017+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":1242,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.093343+0000","last_change":"2024-04-14T15:21:15.764125+0000","last_active":"2024-04-14T15:24:14.093343+0000","last_peered":"2024-04-14T15:24:14.093343+0000","last_clean":"2024-04-14T15:24:14.093343+0000","last_became_active":"2024-04-14T15:12:56.335230+0000","last_became_peered":"2024-04-14T15:12:56.335230+0000","last_unstale":"2024-04-14T15:24:14.093343+0000","last_undegraded":"2024-04-14T15:24:14.093343+0000","last_fullsized":"2024-04-14T15:24:14.093343+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:15.764017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:21:15.764017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:17.526255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"479'20","reported_seq":1373,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.292105+0000","last_change":"2024-04-14T15:22:58.400820+0000","last_active":"2024-04-14T15:24:13.292105+0000","last_peered":"2024-04-14T15:24:13.292105+0000","last_clean":"2024-04-14T15:24:13.292105+0000","last_became_active":"2024-04-14T15:09:45.181456+0000","last_became_peered":"2024-04-14T15:09:45.181456+0000","last_unstale":"2024-04-14T15:24:13.292105+0000","last_undegraded":"2024-04-14T15:24:13.292105+0000","last_fullsized":"2024-04-14T15:24:13.292105+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'20","last_scrub_stamp":"2024-04-14T15:22:58.400727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:22:58.400727+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:18.598942+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":897,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.317107+0000","last_change":"2024-04-14T15:19:57.575006+0000","last_active":"2024-04-14T15:24:13.317107+0000","last_peered":"2024-04-14T15:24:13.317107+0000","last_clean":"2024-04-14T15:24:13.317107+0000","last_became_active":"2024-04-14T15:15:44.878336+0000","last_became_peered":"2024-04-14T15:15:44.878336+0000","last_unstale":"2024-04-14T15:24:13.317107+0000","last_undegraded":"2024-04-14T15:24:13.317107+0000","last_fullsized":"2024-04-14T15:24:13.317107+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:57.574879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:19:57.574879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"528'9","reported_seq":452,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.320493+0000","last_change":"2024-04-14T15:20:53.001282+0000","last_active":"2024-04-14T15:24:13.320493+0000","last_peered":"2024-04-14T15:24:13.320493+0000","last_clean":"2024-04-14T15:24:13.320493+0000","last_became_active":"2024-04-14T15:20:52.982112+0000","last_became_peered":"2024-04-14T15:20:52.982112+0000","last_unstale":"2024-04-14T15:24:13.320493+0000","last_undegraded":"2024-04-14T15:24:13.320493+0000","last_fullsized":"2024-04-14T15:24:13.320493+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:20:45.809489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:20:45.809489+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"72'1","reported_seq":1343,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.316432+0000","last_change":"2024-04-14T15:23:32.511432+0000","last_active":"2024-04-14T15:24:13.316432+0000","last_peered":"2024-04-14T15:24:13.316432+0000","last_clean":"2024-04-14T15:24:13.316432+0000","last_became_active":"2024-04-14T15:09:49.062051+0000","last_became_peered":"2024-04-14T15:09:49.062051+0000","last_unstale":"2024-04-14T15:24:13.316432+0000","last_undegraded":"2024-04-14T15:24:13.316432+0000","last_fullsized":"2024-04-14T15:24:13.316432+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-04-14T15:23:32.511322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:23:32.511322+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:33.567939+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":1248,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.290686+0000","last_change":"2024-04-14T15:22:14.354882+0000","last_active":"2024-04-14T15:24:13.290686+0000","last_peered":"2024-04-14T15:24:13.290686+0000","last_clean":"2024-04-14T15:24:13.290686+0000","last_became_active":"2024-04-14T15:20:52.986657+0000","last_became_peered":"2024-04-14T15:20:52.986657+0000","last_unstale":"2024-04-14T15:24:13.290686+0000","last_undegraded":"2024-04-14T15:24:13.290686+0000","last_fullsized":"2024-04-14T15:24:13.290686+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:14.354795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:22:14.354795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"519'24","reported_seq":1402,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.095849+0000","last_change":"2024-04-14T15:23:40.504101+0000","last_active":"2024-04-14T15:24:14.095849+0000","last_peered":"2024-04-14T15:24:14.095849+0000","last_clean":"2024-04-14T15:24:14.095849+0000","last_became_active":"2024-04-14T15:20:52.987333+0000","last_became_peered":"2024-04-14T15:20:52.987333+0000","last_unstale":"2024-04-14T15:24:14.095849+0000","last_undegraded":"2024-04-14T15:24:14.095849+0000","last_fullsized":"2024-04-14T15:24:14.095849+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"519'24","last_scrub_stamp":"2024-04-14T15:23:40.504002+0000","last_deep_scrub":"519'24","last_deep_scrub_stamp":"2024-04-14T15:20:44.763417+0000","last_clean_scrub_stamp":"2024-04-14T15:23:40.504002+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:57.259947+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.19","version":"684'4","reported_seq":1048,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.315726+0000","last_change":"2024-04-14T15:23:46.931649+0000","last_active":"2024-04-14T15:24:13.315726+0000","last_peered":"2024-04-14T15:24:13.315726+0000","last_clean":"2024-04-14T15:24:13.315726+0000","last_became_active":"2024-04-14T15:20:52.995628+0000","last_became_peered":"2024-04-14T15:20:52.995628+0000","last_unstale":"2024-04-14T15:24:13.315726+0000","last_undegraded":"2024-04-14T15:24:13.315726+0000","last_fullsized":"2024-04-14T15:24:13.315726+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"684'4","last_scrub_stamp":"2024-04-14T15:23:46.931560+0000","last_deep_scrub":"684'4","last_deep_scrub_stamp":"2024-04-14T15:23:46.931560+0000","last_clean_scrub_stamp":"2024-04-14T15:23:46.931560+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:51.788030+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":957,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1339,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.292109+0000","last_change":"2024-04-14T15:22:20.256256+0000","last_active":"2024-04-14T15:24:13.292109+0000","last_peered":"2024-04-14T15:24:13.292109+0000","last_clean":"2024-04-14T15:24:13.292109+0000","last_became_active":"2024-04-14T15:09:47.045730+0000","last_became_peered":"2024-04-14T15:09:47.045730+0000","last_unstale":"2024-04-14T15:24:13.292109+0000","last_undegraded":"2024-04-14T15:24:13.292109+0000","last_fullsized":"2024-04-14T15:24:13.292109+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:20.256177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:22:20.256177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1330,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.092199+0000","last_change":"2024-04-14T15:20:38.518996+0000","last_active":"2024-04-14T15:24:14.092199+0000","last_peered":"2024-04-14T15:24:14.092199+0000","last_clean":"2024-04-14T15:24:14.092199+0000","last_became_active":"2024-04-14T15:09:49.057545+0000","last_became_peered":"2024-04-14T15:09:49.057545+0000","last_unstale":"2024-04-14T15:24:14.092199+0000","last_undegraded":"2024-04-14T15:24:14.092199+0000","last_fullsized":"2024-04-14T15:24:14.092199+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:38.518885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:21.247695+0000","last_clean_scrub_stamp":"2024-04-14T15:20:38.518885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:16.232480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":897,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.318594+0000","last_change":"2024-04-14T15:21:58.069606+0000","last_active":"2024-04-14T15:24:13.318594+0000","last_peered":"2024-04-14T15:24:13.318594+0000","last_clean":"2024-04-14T15:24:13.318594+0000","last_became_active":"2024-04-14T15:15:44.882604+0000","last_became_peered":"2024-04-14T15:15:44.882604+0000","last_unstale":"2024-04-14T15:24:13.318594+0000","last_undegraded":"2024-04-14T15:24:13.318594+0000","last_fullsized":"2024-04-14T15:24:13.318594+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:58.069509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:21:58.069509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"479'27","reported_seq":1384,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.094739+0000","last_change":"2024-04-14T15:21:20.766716+0000","last_active":"2024-04-14T15:24:14.094739+0000","last_peered":"2024-04-14T15:24:14.094739+0000","last_clean":"2024-04-14T15:24:14.094739+0000","last_became_active":"2024-04-14T15:09:45.032887+0000","last_became_peered":"2024-04-14T15:09:45.032887+0000","last_unstale":"2024-04-14T15:24:14.094739+0000","last_undegraded":"2024-04-14T15:24:14.094739+0000","last_fullsized":"2024-04-14T15:24:14.094739+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'27","last_scrub_stamp":"2024-04-14T15:21:20.766623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:21:20.766623+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:17.526184+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":379,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.318169+0000","last_change":"2024-04-14T15:20:52.990991+0000","last_active":"2024-04-14T15:24:13.318169+0000","last_peered":"2024-04-14T15:24:13.318169+0000","last_clean":"2024-04-14T15:24:13.318169+0000","last_became_active":"2024-04-14T15:20:52.990634+0000","last_became_peered":"2024-04-14T15:20:52.990634+0000","last_unstale":"2024-04-14T15:24:13.318169+0000","last_undegraded":"2024-04-14T15:24:13.318169+0000","last_fullsized":"2024-04-14T15:24:13.318169+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:42.754721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:42.754721+0000","last_clean_scrub_stamp":"2024-04-14T15:20:42.754721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1342,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.092365+0000","last_change":"2024-04-14T15:22:18.460314+0000","last_active":"2024-04-14T15:24:14.092365+0000","last_peered":"2024-04-14T15:24:14.092365+0000","last_clean":"2024-04-14T15:24:14.092365+0000","last_became_active":"2024-04-14T15:09:47.042733+0000","last_became_peered":"2024-04-14T15:09:47.042733+0000","last_unstale":"2024-04-14T15:24:14.092365+0000","last_undegraded":"2024-04-14T15:24:14.092365+0000","last_fullsized":"2024-04-14T15:24:14.092365+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:18.460230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:22:18.460230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:16.232627+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1342,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.320124+0000","last_change":"2024-04-14T15:22:08.071314+0000","last_active":"2024-04-14T15:24:13.320124+0000","last_peered":"2024-04-14T15:24:13.320124+0000","last_clean":"2024-04-14T15:24:13.320124+0000","last_became_active":"2024-04-14T15:20:52.998253+0000","last_became_peered":"2024-04-14T15:20:52.998253+0000","last_unstale":"2024-04-14T15:24:13.320124+0000","last_undegraded":"2024-04-14T15:24:13.320124+0000","last_fullsized":"2024-04-14T15:24:13.320124+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:08.071208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:54.453174+0000","last_clean_scrub_stamp":"2024-04-14T15:22:08.071208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":1243,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.318969+0000","last_change":"2024-04-14T15:21:17.675180+0000","last_active":"2024-04-14T15:24:13.318969+0000","last_peered":"2024-04-14T15:24:13.318969+0000","last_clean":"2024-04-14T15:24:13.318969+0000","last_became_active":"2024-04-14T15:12:56.392352+0000","last_became_peered":"2024-04-14T15:12:56.392352+0000","last_unstale":"2024-04-14T15:24:13.318969+0000","last_undegraded":"2024-04-14T15:24:13.318969+0000","last_fullsized":"2024-04-14T15:24:13.318969+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:17.675093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:21:17.675093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:13.524884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"479'18","reported_seq":1055,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.316886+0000","last_change":"2024-04-14T15:22:06.042903+0000","last_active":"2024-04-14T15:24:13.316886+0000","last_peered":"2024-04-14T15:24:13.316886+0000","last_clean":"2024-04-14T15:24:13.316886+0000","last_became_active":"2024-04-14T15:15:21.877838+0000","last_became_peered":"2024-04-14T15:15:21.877838+0000","last_unstale":"2024-04-14T15:24:13.316886+0000","last_undegraded":"2024-04-14T15:24:13.316886+0000","last_fullsized":"2024-04-14T15:24:13.316886+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'18","last_scrub_stamp":"2024-04-14T15:22:06.042808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:22:06.042808+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:15.810071+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":1033,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.316530+0000","last_change":"2024-04-14T15:22:52.950407+0000","last_active":"2024-04-14T15:24:13.316530+0000","last_peered":"2024-04-14T15:24:13.316530+0000","last_clean":"2024-04-14T15:24:13.316530+0000","last_became_active":"2024-04-14T15:20:53.001534+0000","last_became_peered":"2024-04-14T15:20:53.001534+0000","last_unstale":"2024-04-14T15:24:13.316530+0000","last_undegraded":"2024-04-14T15:24:13.316530+0000","last_fullsized":"2024-04-14T15:24:13.316530+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:52.950279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:22:52.950279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:21.260602+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1338,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.319220+0000","last_change":"2024-04-14T15:20:39.717630+0000","last_active":"2024-04-14T15:24:13.319220+0000","last_peered":"2024-04-14T15:24:13.319220+0000","last_clean":"2024-04-14T15:24:13.319220+0000","last_became_active":"2024-04-14T15:09:47.048045+0000","last_became_peered":"2024-04-14T15:09:47.048045+0000","last_unstale":"2024-04-14T15:24:13.319220+0000","last_undegraded":"2024-04-14T15:24:13.319220+0000","last_fullsized":"2024-04-14T15:24:13.319220+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:39.717522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:20:39.717522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1023,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.317239+0000","last_change":"2024-04-14T15:22:20.946609+0000","last_active":"2024-04-14T15:24:13.317239+0000","last_peered":"2024-04-14T15:24:13.317239+0000","last_clean":"2024-04-14T15:24:13.317239+0000","last_became_active":"2024-04-14T15:20:52.998775+0000","last_became_peered":"2024-04-14T15:20:52.998775+0000","last_unstale":"2024-04-14T15:24:13.317239+0000","last_undegraded":"2024-04-14T15:24:13.317239+0000","last_fullsized":"2024-04-14T15:24:13.317239+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:20.946524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:22:20.946524+0000","last_clean_scrub_stamp":"2024-04-14T15:22:20.946524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:15.810087+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":1250,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.095712+0000","last_change":"2024-04-14T15:22:18.954117+0000","last_active":"2024-04-14T15:24:14.095712+0000","last_peered":"2024-04-14T15:24:14.095712+0000","last_clean":"2024-04-14T15:24:14.095712+0000","last_became_active":"2024-04-14T15:12:56.375278+0000","last_became_peered":"2024-04-14T15:12:56.375278+0000","last_unstale":"2024-04-14T15:24:14.095712+0000","last_undegraded":"2024-04-14T15:24:14.095712+0000","last_fullsized":"2024-04-14T15:24:14.095712+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:18.954041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:01.840924+0000","last_clean_scrub_stamp":"2024-04-14T15:22:18.954041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:17.526441+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"479'19","reported_seq":1371,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.318106+0000","last_change":"2024-04-14T15:21:22.085022+0000","last_active":"2024-04-14T15:24:13.318106+0000","last_peered":"2024-04-14T15:24:13.318106+0000","last_clean":"2024-04-14T15:24:13.318106+0000","last_became_active":"2024-04-14T15:09:45.030093+0000","last_became_peered":"2024-04-14T15:09:45.030093+0000","last_unstale":"2024-04-14T15:24:13.318106+0000","last_undegraded":"2024-04-14T15:24:13.318106+0000","last_fullsized":"2024-04-14T15:24:13.318106+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'19","last_scrub_stamp":"2024-04-14T15:21:22.084927+0000","last_deep_scrub":"276'13","last_deep_scrub_stamp":"2024-04-14T15:15:43.969775+0000","last_clean_scrub_stamp":"2024-04-14T15:21:22.084927+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":1044,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.320307+0000","last_change":"2024-04-14T15:20:00.650164+0000","last_active":"2024-04-14T15:24:13.320307+0000","last_peered":"2024-04-14T15:24:13.320307+0000","last_clean":"2024-04-14T15:24:13.320307+0000","last_became_active":"2024-04-14T15:14:30.857727+0000","last_became_peered":"2024-04-14T15:14:30.857727+0000","last_unstale":"2024-04-14T15:24:13.320307+0000","last_undegraded":"2024-04-14T15:24:13.320307+0000","last_fullsized":"2024-04-14T15:24:13.320307+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:00.650062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:00.650062+0000","last_clean_scrub_stamp":"2024-04-14T15:20:00.650062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:13.524710+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":904,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.291718+0000","last_change":"2024-04-14T15:23:04.472665+0000","last_active":"2024-04-14T15:24:13.291718+0000","last_peered":"2024-04-14T15:24:13.291718+0000","last_clean":"2024-04-14T15:24:13.291718+0000","last_became_active":"2024-04-14T15:15:44.883121+0000","last_became_peered":"2024-04-14T15:15:44.883121+0000","last_unstale":"2024-04-14T15:24:13.291718+0000","last_undegraded":"2024-04-14T15:24:13.291718+0000","last_fullsized":"2024-04-14T15:24:13.291718+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:04.472586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:23:04.472586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1341,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.290444+0000","last_change":"2024-04-14T15:23:01.449252+0000","last_active":"2024-04-14T15:24:13.290444+0000","last_peered":"2024-04-14T15:24:13.290444+0000","last_clean":"2024-04-14T15:24:13.290444+0000","last_became_active":"2024-04-14T15:09:49.061756+0000","last_became_peered":"2024-04-14T15:09:49.061756+0000","last_unstale":"2024-04-14T15:24:13.290444+0000","last_undegraded":"2024-04-14T15:24:13.290444+0000","last_fullsized":"2024-04-14T15:24:13.290444+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:01.449169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:23:01.449169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:27.214881+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":379,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.318874+0000","last_change":"2024-04-14T15:20:52.991133+0000","last_active":"2024-04-14T15:24:13.318874+0000","last_peered":"2024-04-14T15:24:13.318874+0000","last_clean":"2024-04-14T15:24:13.318874+0000","last_became_active":"2024-04-14T15:20:52.990838+0000","last_became_peered":"2024-04-14T15:20:52.990838+0000","last_unstale":"2024-04-14T15:24:13.318874+0000","last_undegraded":"2024-04-14T15:24:13.318874+0000","last_fullsized":"2024-04-14T15:24:13.318874+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:12.723073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:19:12.723073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"479'9","reported_seq":1357,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.319102+0000","last_change":"2024-04-14T15:22:54.203852+0000","last_active":"2024-04-14T15:24:13.319102+0000","last_peered":"2024-04-14T15:24:13.319102+0000","last_clean":"2024-04-14T15:24:13.319102+0000","last_became_active":"2024-04-14T15:09:45.028667+0000","last_became_peered":"2024-04-14T15:09:45.028667+0000","last_unstale":"2024-04-14T15:24:13.319102+0000","last_undegraded":"2024-04-14T15:24:13.319102+0000","last_fullsized":"2024-04-14T15:24:13.319102+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'9","last_scrub_stamp":"2024-04-14T15:22:54.203720+0000","last_deep_scrub":"87'7","last_deep_scrub_stamp":"2024-04-14T15:11:10.510785+0000","last_clean_scrub_stamp":"2024-04-14T15:22:54.203720+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:17.348552+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":829,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.095178+0000","last_change":"2024-04-14T15:22:15.944094+0000","last_active":"2024-04-14T15:24:14.095178+0000","last_peered":"2024-04-14T15:24:14.095178+0000","last_clean":"2024-04-14T15:24:14.095178+0000","last_became_active":"2024-04-14T15:20:52.989307+0000","last_became_peered":"2024-04-14T15:20:52.989307+0000","last_unstale":"2024-04-14T15:24:14.095178+0000","last_undegraded":"2024-04-14T15:24:14.095178+0000","last_fullsized":"2024-04-14T15:24:14.095178+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:15.944005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:22:15.944005+0000","last_clean_scrub_stamp":"2024-04-14T15:22:15.944005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:17.526365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1338,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.319487+0000","last_change":"2024-04-14T15:22:05.564704+0000","last_active":"2024-04-14T15:24:13.319487+0000","last_peered":"2024-04-14T15:24:13.319487+0000","last_clean":"2024-04-14T15:24:13.319487+0000","last_became_active":"2024-04-14T15:09:47.046517+0000","last_became_peered":"2024-04-14T15:09:47.046517+0000","last_unstale":"2024-04-14T15:24:13.319487+0000","last_undegraded":"2024-04-14T15:24:13.319487+0000","last_fullsized":"2024-04-14T15:24:13.319487+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:05.564624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:22:05.564624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:13.524994+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":902,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.091146+0000","last_change":"2024-04-14T15:22:08.325886+0000","last_active":"2024-04-14T15:24:14.091146+0000","last_peered":"2024-04-14T15:24:14.091146+0000","last_clean":"2024-04-14T15:24:14.091146+0000","last_became_active":"2024-04-14T15:15:44.887227+0000","last_became_peered":"2024-04-14T15:15:44.887227+0000","last_unstale":"2024-04-14T15:24:14.091146+0000","last_undegraded":"2024-04-14T15:24:14.091146+0000","last_fullsized":"2024-04-14T15:24:14.091146+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:08.325808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:56.268086+0000","last_clean_scrub_stamp":"2024-04-14T15:22:08.325808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:16.232660+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":1249,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.321585+0000","last_change":"2024-04-14T15:21:21.668496+0000","last_active":"2024-04-14T15:24:13.321585+0000","last_peered":"2024-04-14T15:24:13.321585+0000","last_clean":"2024-04-14T15:24:13.321585+0000","last_became_active":"2024-04-14T15:20:52.984387+0000","last_became_peered":"2024-04-14T15:20:52.984387+0000","last_unstale":"2024-04-14T15:24:13.321585+0000","last_undegraded":"2024-04-14T15:24:13.321585+0000","last_fullsized":"2024-04-14T15:24:13.321585+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:21.668407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:34.558312+0000","last_clean_scrub_stamp":"2024-04-14T15:21:21.668407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:13.524817+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.18","version":"479'15","reported_seq":1383,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.316863+0000","last_change":"2024-04-14T15:22:57.315447+0000","last_active":"2024-04-14T15:24:13.316863+0000","last_peered":"2024-04-14T15:24:13.316863+0000","last_clean":"2024-04-14T15:24:13.316863+0000","last_became_active":"2024-04-14T15:20:52.985210+0000","last_became_peered":"2024-04-14T15:20:52.985210+0000","last_unstale":"2024-04-14T15:24:13.316863+0000","last_undegraded":"2024-04-14T15:24:13.316863+0000","last_fullsized":"2024-04-14T15:24:13.316863+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'15","last_scrub_stamp":"2024-04-14T15:22:57.315355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:22:57.315355+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:26.025559+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":1021,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.317146+0000","last_change":"2024-04-14T15:20:35.984146+0000","last_active":"2024-04-14T15:24:13.317146+0000","last_peered":"2024-04-14T15:24:13.317146+0000","last_clean":"2024-04-14T15:24:13.317146+0000","last_became_active":"2024-04-14T15:15:21.891926+0000","last_became_peered":"2024-04-14T15:15:21.891926+0000","last_unstale":"2024-04-14T15:24:13.317146+0000","last_undegraded":"2024-04-14T15:24:13.317146+0000","last_fullsized":"2024-04-14T15:24:13.317146+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:35.984075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:20:35.984075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:15.809827+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1334,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-14T15:24:14.091000+0000","last_change":"2024-04-14T15:19:56.903138+0000","last_active":"2024-04-14T15:24:14.091000+0000","last_peered":"2024-04-14T15:24:14.091000+0000","last_clean":"2024-04-14T15:24:14.091000+0000","last_became_active":"2024-04-14T15:09:47.041486+0000","last_became_peered":"2024-04-14T15:09:47.041486+0000","last_unstale":"2024-04-14T15:24:14.091000+0000","last_undegraded":"2024-04-14T15:24:14.091000+0000","last_fullsized":"2024-04-14T15:24:14.091000+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:56.903059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:19:56.903059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:16.232447+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1326,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.315846+0000","last_change":"2024-04-14T15:20:36.761316+0000","last_active":"2024-04-14T15:24:13.315846+0000","last_peered":"2024-04-14T15:24:13.315846+0000","last_clean":"2024-04-14T15:24:13.315846+0000","last_became_active":"2024-04-14T15:09:49.058904+0000","last_became_peered":"2024-04-14T15:09:49.058904+0000","last_unstale":"2024-04-14T15:24:13.315846+0000","last_undegraded":"2024-04-14T15:24:13.315846+0000","last_fullsized":"2024-04-14T15:24:13.315846+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:36.761246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:20:36.761246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":897,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.320386+0000","last_change":"2024-04-14T15:21:14.673688+0000","last_active":"2024-04-14T15:24:13.320386+0000","last_peered":"2024-04-14T15:24:13.320386+0000","last_clean":"2024-04-14T15:24:13.320386+0000","last_became_active":"2024-04-14T15:15:44.889605+0000","last_became_peered":"2024-04-14T15:15:44.889605+0000","last_unstale":"2024-04-14T15:24:13.320386+0000","last_undegraded":"2024-04-14T15:24:13.320386+0000","last_fullsized":"2024-04-14T15:24:13.320386+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:14.673591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:21:14.673591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:13.524834+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"479'34","reported_seq":1073,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.316051+0000","last_change":"2024-04-14T15:23:00.061412+0000","last_active":"2024-04-14T15:24:13.316051+0000","last_peered":"2024-04-14T15:24:13.316051+0000","last_clean":"2024-04-14T15:24:13.316051+0000","last_became_active":"2024-04-14T15:15:21.884310+0000","last_became_peered":"2024-04-14T15:15:21.884310+0000","last_unstale":"2024-04-14T15:24:13.316051+0000","last_undegraded":"2024-04-14T15:24:13.316051+0000","last_fullsized":"2024-04-14T15:24:13.316051+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'34","last_scrub_stamp":"2024-04-14T15:23:00.061294+0000","last_deep_scrub":"306'22","last_deep_scrub_stamp":"2024-04-14T15:16:13.898106+0000","last_clean_scrub_stamp":"2024-04-14T15:23:00.061294+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:15.810123+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":1346,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.291611+0000","last_change":"2024-04-14T15:22:09.335374+0000","last_active":"2024-04-14T15:24:13.291611+0000","last_peered":"2024-04-14T15:24:13.291611+0000","last_clean":"2024-04-14T15:24:13.291611+0000","last_became_active":"2024-04-14T15:09:43.017351+0000","last_became_peered":"2024-04-14T15:09:43.017351+0000","last_unstale":"2024-04-14T15:24:13.291611+0000","last_undegraded":"2024-04-14T15:24:13.291611+0000","last_fullsized":"2024-04-14T15:24:13.291611+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:09.335294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:22:09.335294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1336,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.317355+0000","last_change":"2024-04-14T15:20:52.976225+0000","last_active":"2024-04-14T15:24:13.317355+0000","last_peered":"2024-04-14T15:24:13.317355+0000","last_clean":"2024-04-14T15:24:13.317355+0000","last_became_active":"2024-04-14T15:20:52.974360+0000","last_became_peered":"2024-04-14T15:20:52.974360+0000","last_unstale":"2024-04-14T15:24:13.317355+0000","last_undegraded":"2024-04-14T15:24:13.317355+0000","last_fullsized":"2024-04-14T15:24:13.317355+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:49.631273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:08.512130+0000","last_clean_scrub_stamp":"2024-04-14T15:19:49.631273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1336,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-04-14T15:24:13.316033+0000","last_change":"2024-04-14T15:21:22.119138+0000","last_active":"2024-04-14T15:24:13.316033+0000","last_peered":"2024-04-14T15:24:13.316033+0000","last_clean":"2024-04-14T15:24:13.316033+0000","last_became_active":"2024-04-14T15:20:52.998938+0000","last_became_peered":"2024-04-14T15:20:52.998938+0000","last_unstale":"2024-04-14T15:24:13.316033+0000","last_undegraded":"2024-04-14T15:24:13.316033+0000","last_fullsized":"2024-04-14T15:24:13.316033+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:22.119061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:21:22.119061+0000","last_clean_scrub_stamp":"2024-04-14T15:21:22.119061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:15.809897+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":330,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":7},{"poolid":323,"num_pg":8,"stat_sum":{"num_bytes":95,"num_objects":17,"num_object_clones":4,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":1,"num_read":9,"num_read_kb":1,"num_write":25,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":184320,"data_stored":184320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":24,"acting":24,"num_store_stats":7},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":5,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1003,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":918048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2109440,"data_stored":2098784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":862,"num_read_kb":857,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1121,"num_read_kb":878,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":1694,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":785,"ondisk_log_size":785,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103348,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":94712,"kb_used_data":5228,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":93650440,"statfs":{"total":95995035648,"available":95898050560,"internally_reserved":0,"allocated":5353472,"data_stored":2719797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":91617738},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":210453397687,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575057,"num_pgs":88,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":116644,"kb_used_data":4944,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93628508,"statfs":{"total":95995035648,"available":95875592192,"internally_reserved":0,"allocated":5062656,"data_stored":2400613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738582,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":90236,"kb_used_data":4912,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":93654916,"statfs":{"total":95995035648,"available":95902633984,"internally_reserved":0,"allocated":5029888,"data_stored":2375709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":87357895},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411533,"num_pgs":92,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":90516,"kb_used_data":5188,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":93654636,"statfs":{"total":95995035648,"available":95902347264,"internally_reserved":0,"allocated":5312512,"data_stored":2671886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182599,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":68660,"kb_used_data":5536,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93676492,"statfs":{"total":95995035648,"available":95924727808,"internally_reserved":0,"allocated":5668864,"data_stored":2991573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986369,"num_pgs":87,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":116616,"kb_used_data":4920,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93628536,"statfs":{"total":95995035648,"available":95875620864,"internally_reserved":0,"allocated":5038080,"data_stored":2388046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659326,"num_pgs":91,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":99136,"kb_used_data":5484,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":93646016,"statfs":{"total":95995035648,"available":95893520384,"internally_reserved":0,"allocated":5615616,"data_stored":2973973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":20480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":330,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":330,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":330,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":330,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":330,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":330,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":330,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-14T15:24:18.513 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 1, currently 11 2024-04-14T15:24:18.520 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:18.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:18.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:24:18.621 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:24:18.645 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:18.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:18.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:24:18.746 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:24:18.771 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:18.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:18.772 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:24:18.872 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:24:18.899 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:18.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:18.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:24:19.001 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:24:19.030 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:19.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:19.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:24:19.101 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-14T15:24:19.102 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:19 smithi090 ceph-mon[41670]: pgmap v1056: 214 pgs: 214 active+clean; 583 KiB data, 661 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-04-14T15:24:19.102 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:19 smithi090 ceph-mon[41670]: 5.10 scrub starts 2024-04-14T15:24:19.102 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:19 smithi090 ceph-mon[41670]: 5.10 scrub ok 2024-04-14T15:24:19.102 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:19 smithi090 ceph-mon[41670]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:24:19.102 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:19 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-124"}]': finished 2024-04-14T15:24:19.102 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:19 smithi090 ceph-mon[41670]: osdmap e716: 8 total, 8 up, 7 in 2024-04-14T15:24:19.102 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:19 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:24:19.102 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:19 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:24:19.103 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:19 smithi090 ceph-mon[41670]: 5.2 scrub starts 2024-04-14T15:24:19.103 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:19 smithi090 ceph-mon[41670]: 5.2 scrub ok 2024-04-14T15:24:19.103 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:19 smithi090 ceph-mon[30399]: pgmap v1056: 214 pgs: 214 active+clean; 583 KiB data, 661 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-04-14T15:24:19.103 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:19 smithi090 ceph-mon[30399]: 5.10 scrub starts 2024-04-14T15:24:19.103 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:19 smithi090 ceph-mon[30399]: 5.10 scrub ok 2024-04-14T15:24:19.103 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:19 smithi090 ceph-mon[30399]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:24:19.103 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:19 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-124"}]': finished 2024-04-14T15:24:19.103 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:19 smithi090 ceph-mon[30399]: osdmap e716: 8 total, 8 up, 7 in 2024-04-14T15:24:19.103 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:19 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:24:19.103 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:19 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:24:19.104 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:19 smithi090 ceph-mon[30399]: 5.2 scrub starts 2024-04-14T15:24:19.104 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:19 smithi090 ceph-mon[30399]: 5.2 scrub ok 2024-04-14T15:24:19.132 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:24:19.158 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:19.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:19.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:24:19.259 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:24:19.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:19.289 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:19.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:24:19.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:19 smithi092 ceph-mon[33966]: pgmap v1056: 214 pgs: 214 active+clean; 583 KiB data, 661 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-04-14T15:24:19.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:19 smithi092 ceph-mon[33966]: 5.10 scrub starts 2024-04-14T15:24:19.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:19 smithi092 ceph-mon[33966]: 5.10 scrub ok 2024-04-14T15:24:19.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:19 smithi092 ceph-mon[33966]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:24:19.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:19 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-124"}]': finished 2024-04-14T15:24:19.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:19 smithi092 ceph-mon[33966]: osdmap e716: 8 total, 8 up, 7 in 2024-04-14T15:24:19.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:19 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:24:19.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:19 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:24:19.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:19 smithi092 ceph-mon[33966]: 5.2 scrub starts 2024-04-14T15:24:19.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:19 smithi092 ceph-mon[33966]: 5.2 scrub ok 2024-04-14T15:24:19.390 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:24:19.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:19.415 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:19.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:24:19.516 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:24:19.542 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:19.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:19.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:24:19.644 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:24:19.671 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:19.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:19.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:24:19.773 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:24:19.797 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:19.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:19.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:24:19.898 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:24:19.925 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:19.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:19.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:24:20.027 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:24:20.056 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:20.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:20.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:24:20.158 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:24:20.182 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:20.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:20.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:24:20.284 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:24:20.311 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:20.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:20.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:24:20.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:20 smithi092 ceph-mon[33966]: 6.12 scrub starts 2024-04-14T15:24:20.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:20 smithi092 ceph-mon[33966]: 6.12 scrub ok 2024-04-14T15:24:20.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:20 smithi092 ceph-mon[33966]: 5.16 scrub starts 2024-04-14T15:24:20.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:20 smithi092 ceph-mon[33966]: 5.16 scrub ok 2024-04-14T15:24:20.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:20 smithi092 ceph-mon[33966]: osdmap e717: 8 total, 8 up, 7 in 2024-04-14T15:24:20.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:20 smithi092 ceph-mon[33966]: 5.12 scrub starts 2024-04-14T15:24:20.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:20 smithi092 ceph-mon[33966]: 5.12 scrub ok 2024-04-14T15:24:20.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:20 smithi092 ceph-mon[33966]: 6.e scrub starts 2024-04-14T15:24:20.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:20 smithi092 ceph-mon[33966]: 6.e scrub ok 2024-04-14T15:24:20.411 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:20 smithi090 ceph-mon[30399]: 6.12 scrub starts 2024-04-14T15:24:20.411 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:20 smithi090 ceph-mon[30399]: 6.12 scrub ok 2024-04-14T15:24:20.411 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:20 smithi090 ceph-mon[30399]: 5.16 scrub starts 2024-04-14T15:24:20.411 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:20 smithi090 ceph-mon[30399]: 5.16 scrub ok 2024-04-14T15:24:20.412 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:20 smithi090 ceph-mon[30399]: osdmap e717: 8 total, 8 up, 7 in 2024-04-14T15:24:20.412 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:20 smithi090 ceph-mon[30399]: 5.12 scrub starts 2024-04-14T15:24:20.412 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:20 smithi090 ceph-mon[30399]: 5.12 scrub ok 2024-04-14T15:24:20.412 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:20 smithi090 ceph-mon[30399]: 6.e scrub starts 2024-04-14T15:24:20.412 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:20 smithi090 ceph-mon[30399]: 6.e scrub ok 2024-04-14T15:24:20.412 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:20 smithi090 ceph-mon[41670]: 6.12 scrub starts 2024-04-14T15:24:20.412 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:20 smithi090 ceph-mon[41670]: 6.12 scrub ok 2024-04-14T15:24:20.412 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:20 smithi090 ceph-mon[41670]: 5.16 scrub starts 2024-04-14T15:24:20.412 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:20 smithi090 ceph-mon[41670]: 5.16 scrub ok 2024-04-14T15:24:20.412 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:20 smithi090 ceph-mon[41670]: osdmap e717: 8 total, 8 up, 7 in 2024-04-14T15:24:20.412 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:20 smithi090 ceph-mon[41670]: 5.12 scrub starts 2024-04-14T15:24:20.412 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:20 smithi090 ceph-mon[41670]: 5.12 scrub ok 2024-04-14T15:24:20.413 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:20 smithi090 ceph-mon[41670]: 6.e scrub starts 2024-04-14T15:24:20.413 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:20 smithi090 ceph-mon[41670]: 6.e scrub ok 2024-04-14T15:24:20.413 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:24:20.436 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:20.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:20.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:24:20.537 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:24:20.561 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:20.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:20.562 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:24:20.663 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:24:20.686 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:20.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:20.687 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:24:20.788 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:24:20.811 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:20.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:20.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:24:20.913 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:24:20.941 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:20.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:20.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:24:21.038 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:24:21.042 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:24:21.067 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:21.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:21.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:24:21.169 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:24:21.194 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:21.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:21.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:24:21.296 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:21 smithi090 ceph-mon[30399]: pgmap v1059: 182 pgs: 182 active+clean; 582 KiB data, 679 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:24:21.296 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:21 smithi090 ceph-mon[30399]: osdmap e718: 8 total, 8 up, 7 in 2024-04-14T15:24:21.296 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:21 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:24:21.296 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:21 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:24:21.296 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:21 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:24:21.296 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:21 smithi090 ceph-mon[30399]: osdmap e719: 8 total, 8 up, 7 in 2024-04-14T15:24:21.296 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:21 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:24:21.296 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:21 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:24:21.296 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:21 smithi090 ceph-mon[41670]: pgmap v1059: 182 pgs: 182 active+clean; 582 KiB data, 679 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:24:21.297 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:21 smithi090 ceph-mon[41670]: osdmap e718: 8 total, 8 up, 7 in 2024-04-14T15:24:21.297 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:21 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:24:21.297 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:21 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:24:21.297 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:21 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:24:21.297 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:21 smithi090 ceph-mon[41670]: osdmap e719: 8 total, 8 up, 7 in 2024-04-14T15:24:21.297 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:21 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:24:21.297 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:21 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:24:21.297 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:24:21.322 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:21.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:21.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:24:21.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:21 smithi092 ceph-mon[33966]: pgmap v1059: 182 pgs: 182 active+clean; 582 KiB data, 679 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:24:21.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:21 smithi092 ceph-mon[33966]: osdmap e718: 8 total, 8 up, 7 in 2024-04-14T15:24:21.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:21 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:24:21.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:21 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:24:21.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:21 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:24:21.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:21 smithi092 ceph-mon[33966]: osdmap e719: 8 total, 8 up, 7 in 2024-04-14T15:24:21.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:21 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:24:21.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:21 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:24:21.423 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:24:21.447 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:21.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:21.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:24:21.541 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-14T15:24:21.549 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:24:21.573 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:21.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:21.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:24:21.675 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:24:21.699 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:21.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:21.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:24:21.801 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:24:21.824 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:21.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:21.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:24:21.926 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:24:21.949 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:21.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:21.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:24:22.051 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:24:22.076 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:22.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:22.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:24:22.178 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:24:22.183 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-14T15:24:22.183 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:24:22.207 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:22.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:22.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:24:22.308 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:24:22.332 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:22.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:22.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:24:22.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:22 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:24:22.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:22 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:24:22.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:22 smithi092 ceph-mon[33966]: pgmap v1062: 214 pgs: 32 unknown, 182 active+clean; 582 KiB data, 679 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:24:22.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:22 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-126", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:24:22.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:22 smithi092 ceph-mon[33966]: osdmap e720: 8 total, 8 up, 7 in 2024-04-14T15:24:22.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:22 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-126"}]: dispatch 2024-04-14T15:24:22.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:22 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-126"}]: dispatch 2024-04-14T15:24:22.434 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:24:22.440 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:24:22.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:22 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:24:22.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:22 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:24:22.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:22 smithi090 ceph-mon[30399]: pgmap v1062: 214 pgs: 32 unknown, 182 active+clean; 582 KiB data, 679 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:24:22.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:22 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-126", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:24:22.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:22 smithi090 ceph-mon[30399]: osdmap e720: 8 total, 8 up, 7 in 2024-04-14T15:24:22.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:22 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-126"}]: dispatch 2024-04-14T15:24:22.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:22 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-126"}]: dispatch 2024-04-14T15:24:22.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:22 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:24:22.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:22 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:24:22.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:22 smithi090 ceph-mon[41670]: pgmap v1062: 214 pgs: 32 unknown, 182 active+clean; 582 KiB data, 679 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:24:22.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:22 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-126", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:24:22.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:22 smithi090 ceph-mon[41670]: osdmap e720: 8 total, 8 up, 7 in 2024-04-14T15:24:22.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:22 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-126"}]: dispatch 2024-04-14T15:24:22.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:22 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-126"}]: dispatch 2024-04-14T15:24:22.460 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:22.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:22.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:24:22.562 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:24:22.586 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:22.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:22.587 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:24:22.687 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:24:22.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:22.712 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:22.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:24:22.813 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:24:22.836 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:22.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:22.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:24:22.936 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-14T15:24:22.938 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:24:22.961 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:22.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:22.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:24:23.062 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:24:23.087 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:23.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:23.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:24:23.189 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:24:23.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:23 smithi090 ceph-mon[30399]: 4.1c scrub starts 2024-04-14T15:24:23.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:23 smithi090 ceph-mon[30399]: 4.1c scrub ok 2024-04-14T15:24:23.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:23 smithi090 ceph-mon[30399]: 2.1e scrub starts 2024-04-14T15:24:23.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:23 smithi090 ceph-mon[30399]: 2.1e scrub ok 2024-04-14T15:24:23.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:23 smithi090 ceph-mon[30399]: 5.19 scrub starts 2024-04-14T15:24:23.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:23 smithi090 ceph-mon[30399]: 5.19 scrub ok 2024-04-14T15:24:23.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:23 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:24:23.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:23 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:24:23.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:23 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:24:23.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:23 smithi090 ceph-mon[30399]: 6.10 scrub starts 2024-04-14T15:24:23.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:23 smithi090 ceph-mon[30399]: 6.10 scrub ok 2024-04-14T15:24:23.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:23 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-126"}]': finished 2024-04-14T15:24:23.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:23 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-126", "mode": "writeback"}]: dispatch 2024-04-14T15:24:23.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:23 smithi090 ceph-mon[30399]: osdmap e721: 8 total, 8 up, 7 in 2024-04-14T15:24:23.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:23 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-126", "mode": "writeback"}]: dispatch 2024-04-14T15:24:23.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:23 smithi090 ceph-mon[41670]: 4.1c scrub starts 2024-04-14T15:24:23.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:23 smithi090 ceph-mon[41670]: 4.1c scrub ok 2024-04-14T15:24:23.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:23 smithi090 ceph-mon[41670]: 2.1e scrub starts 2024-04-14T15:24:23.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:23 smithi090 ceph-mon[41670]: 2.1e scrub ok 2024-04-14T15:24:23.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:23 smithi090 ceph-mon[41670]: 5.19 scrub starts 2024-04-14T15:24:23.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:23 smithi090 ceph-mon[41670]: 5.19 scrub ok 2024-04-14T15:24:23.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:23 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:24:23.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:23 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:24:23.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:23 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:24:23.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:23 smithi090 ceph-mon[41670]: 6.10 scrub starts 2024-04-14T15:24:23.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:23 smithi090 ceph-mon[41670]: 6.10 scrub ok 2024-04-14T15:24:23.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:23 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-126"}]': finished 2024-04-14T15:24:23.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:23 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-126", "mode": "writeback"}]: dispatch 2024-04-14T15:24:23.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:23 smithi090 ceph-mon[41670]: osdmap e721: 8 total, 8 up, 7 in 2024-04-14T15:24:23.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:23 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-126", "mode": "writeback"}]: dispatch 2024-04-14T15:24:23.212 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:23.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:23.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:24:23.314 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:24:23.337 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:23.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:23.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:24:23.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:23 smithi092 ceph-mon[33966]: 4.1c scrub starts 2024-04-14T15:24:23.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:23 smithi092 ceph-mon[33966]: 4.1c scrub ok 2024-04-14T15:24:23.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:23 smithi092 ceph-mon[33966]: 2.1e scrub starts 2024-04-14T15:24:23.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:23 smithi092 ceph-mon[33966]: 2.1e scrub ok 2024-04-14T15:24:23.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:23 smithi092 ceph-mon[33966]: 5.19 scrub starts 2024-04-14T15:24:23.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:23 smithi092 ceph-mon[33966]: 5.19 scrub ok 2024-04-14T15:24:23.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:23 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:24:23.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:23 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:24:23.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:23 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:24:23.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:23 smithi092 ceph-mon[33966]: 6.10 scrub starts 2024-04-14T15:24:23.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:23 smithi092 ceph-mon[33966]: 6.10 scrub ok 2024-04-14T15:24:23.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:23 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-126"}]': finished 2024-04-14T15:24:23.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:23 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-126", "mode": "writeback"}]: dispatch 2024-04-14T15:24:23.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:23 smithi092 ceph-mon[33966]: osdmap e721: 8 total, 8 up, 7 in 2024-04-14T15:24:23.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:23 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-126", "mode": "writeback"}]: dispatch 2024-04-14T15:24:23.440 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:24:23.464 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:23.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:23.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:24:23.514 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-14T15:24:23.514 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-14T15:24:23.514 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-04-14T15:24:23.514 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-04-14T15:24:23.515 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph pg dump --format=json 2024-04-14T15:24:23.566 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:24:23.590 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:23.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:23.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:24:23.692 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:24:23.716 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:23.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:23.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:24:23.817 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:24:23.841 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:23.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:23.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:24:23.942 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:24:23.966 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:23.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:23.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:24:23.972 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:24:23 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:24:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:24:24.068 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:24:24.091 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:24.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:24.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:24:24.193 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:24:24.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.198+0000 7f5bfa249700 1 -- 172.21.15.90:0/1084322487 >> v1:172.21.15.90:6789/0 conn(0x7f5bf410b6f0 legacy=0x7f5bf410dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:24.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.198+0000 7f5bfa249700 1 -- 172.21.15.90:0/1084322487 shutdown_connections 2024-04-14T15:24:24.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.198+0000 7f5bfa249700 1 -- 172.21.15.90:0/1084322487 >> 172.21.15.90:0/1084322487 conn(0x7f5bf4076aa0 msgr2=0x7f5bf4076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:24:24.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.199+0000 7f5bfa249700 1 -- 172.21.15.90:0/1084322487 shutdown_connections 2024-04-14T15:24:24.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.199+0000 7f5bfa249700 1 -- 172.21.15.90:0/1084322487 wait complete. 2024-04-14T15:24:24.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.199+0000 7f5bfa249700 1 librados: starting msgr at 2024-04-14T15:24:24.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.199+0000 7f5bfa249700 1 librados: starting objecter 2024-04-14T15:24:24.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.199+0000 7f5bfa249700 1 Processor -- start 2024-04-14T15:24:24.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.199+0000 7f5bfa249700 1 -- start start 2024-04-14T15:24:24.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.199+0000 7f5bfa249700 1 librados: setting wanted keys 2024-04-14T15:24:24.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.199+0000 7f5bfa249700 1 librados: calling monclient init 2024-04-14T15:24:24.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.200+0000 7f5bfa249700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5bf41a9de0 con 0x7f5bf410b6f0 2024-04-14T15:24:24.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.200+0000 7f5bfa249700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5bf41ab000 con 0x7f5bf410f2f0 2024-04-14T15:24:24.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.200+0000 7f5bfa249700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5bf41ac240 con 0x7f5bf4108980 2024-04-14T15:24:24.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.200+0000 7f5bf2ffd700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f5bf410b6f0 0x7f5bf41a4b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:39430/0 (socket says 172.21.15.90:39430) 2024-04-14T15:24:24.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.200+0000 7f5bf2ffd700 1 -- 172.21.15.90:0/708853304 learned_addr learned my addr 172.21.15.90:0/708853304 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:24:24.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.200+0000 7f5bf0ff9700 10 client.?.objecter ms_handle_connect 0x7f5bf4108980 2024-04-14T15:24:24.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.200+0000 7f5bf0ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:24:24.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.200+0000 7f5bf0ff9700 10 client.?.objecter ms_handle_connect 0x7f5bf410b6f0 2024-04-14T15:24:24.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.200+0000 7f5bf0ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:24:24.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.200+0000 7f5bf0ff9700 10 client.?.objecter ms_handle_connect 0x7f5bf410f2f0 2024-04-14T15:24:24.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.200+0000 7f5bf0ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:24:24.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.200+0000 7f5bf0ff9700 1 -- 172.21.15.90:0/708853304 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1783456296 0 0) 0x7f5bf41ab000 con 0x7f5bf410f2f0 2024-04-14T15:24:24.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.200+0000 7f5bf0ff9700 1 -- 172.21.15.90:0/708853304 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5bdc0035b0 con 0x7f5bf410f2f0 2024-04-14T15:24:24.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.200+0000 7f5bf0ff9700 1 -- 172.21.15.90:0/708853304 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 445254941 0 0) 0x7f5bf41a9de0 con 0x7f5bf410b6f0 2024-04-14T15:24:24.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.200+0000 7f5bf0ff9700 1 -- 172.21.15.90:0/708853304 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5bf41ab000 con 0x7f5bf410b6f0 2024-04-14T15:24:24.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.200+0000 7f5bf0ff9700 1 -- 172.21.15.90:0/708853304 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 362328718 0 0) 0x7f5bf41ac240 con 0x7f5bf4108980 2024-04-14T15:24:24.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.200+0000 7f5bf0ff9700 1 -- 172.21.15.90:0/708853304 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5bf41a9de0 con 0x7f5bf4108980 2024-04-14T15:24:24.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.200+0000 7f5bf0ff9700 1 -- 172.21.15.90:0/708853304 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 548905894 0 0) 0x7f5bdc0035b0 con 0x7f5bf410f2f0 2024-04-14T15:24:24.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.200+0000 7f5bf0ff9700 1 -- 172.21.15.90:0/708853304 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5bf41ac240 con 0x7f5bf410f2f0 2024-04-14T15:24:24.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.200+0000 7f5bf0ff9700 1 -- 172.21.15.90:0/708853304 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2544615429 0 0) 0x7f5bf41a9de0 con 0x7f5bf4108980 2024-04-14T15:24:24.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.200+0000 7f5bf0ff9700 1 -- 172.21.15.90:0/708853304 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5bdc0035b0 con 0x7f5bf4108980 2024-04-14T15:24:24.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.200+0000 7f5bf0ff9700 1 -- 172.21.15.90:0/708853304 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 387887178 0 0) 0x7f5bf41ab000 con 0x7f5bf410b6f0 2024-04-14T15:24:24.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.200+0000 7f5bf0ff9700 1 -- 172.21.15.90:0/708853304 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5bf41a9de0 con 0x7f5bf410b6f0 2024-04-14T15:24:24.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.200+0000 7f5bf0ff9700 1 -- 172.21.15.90:0/708853304 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f5bec001c30 con 0x7f5bf410f2f0 2024-04-14T15:24:24.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.200+0000 7f5bf0ff9700 1 -- 172.21.15.90:0/708853304 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f5be4001830 con 0x7f5bf4108980 2024-04-14T15:24:24.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.201+0000 7f5bf0ff9700 1 -- 172.21.15.90:0/708853304 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f5be8001430 con 0x7f5bf410b6f0 2024-04-14T15:24:24.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.201+0000 7f5bf0ff9700 1 -- 172.21.15.90:0/708853304 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 433650918 0 0) 0x7f5bf41ac240 con 0x7f5bf410f2f0 2024-04-14T15:24:24.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.201+0000 7f5bf0ff9700 1 -- 172.21.15.90:0/708853304 >> v1:172.21.15.90:6790/0 conn(0x7f5bf4108980 legacy=0x7f5bf4107d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:24.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.201+0000 7f5bf0ff9700 1 -- 172.21.15.90:0/708853304 >> v1:172.21.15.90:6789/0 conn(0x7f5bf410b6f0 legacy=0x7f5bf41a4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:24.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.201+0000 7f5bf0ff9700 1 -- 172.21.15.90:0/708853304 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bf41ad480 con 0x7f5bf410f2f0 2024-04-14T15:24:24.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.201+0000 7f5bfa249700 1 -- 172.21.15.90:0/708853304 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5bf41aa010 con 0x7f5bf410f2f0 2024-04-14T15:24:24.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.201+0000 7f5bfa249700 10 client.49729.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:24:24.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.201+0000 7f5bfa249700 1 -- 172.21.15.90:0/708853304 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f5bf41aa4d0 con 0x7f5bf410f2f0 2024-04-14T15:24:24.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.201+0000 7f5bfa249700 1 librados: init done 2024-04-14T15:24:24.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.201+0000 7f5bf0ff9700 1 -- 172.21.15.90:0/708853304 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f5bec003ec0 con 0x7f5bf410f2f0 2024-04-14T15:24:24.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.201+0000 7f5bf0ff9700 1 -- 172.21.15.90:0/708853304 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f5bec004df0 con 0x7f5bf410f2f0 2024-04-14T15:24:24.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.202+0000 7f5bfa249700 1 -- 172.21.15.90:0/708853304 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f5be0005100 con 0x7f5bf410f2f0 2024-04-14T15:24:24.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.202+0000 7f5bf0ff9700 1 -- 172.21.15.90:0/708853304 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f5bec01ba80 con 0x7f5bf410f2f0 2024-04-14T15:24:24.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.203+0000 7f5bf0ff9700 1 -- 172.21.15.90:0/708853304 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(722..722 src has 467..722) ==== 8757+0+0 (unknown 3244270124 0 0) 0x7f5bec089da0 con 0x7f5bf410f2f0 2024-04-14T15:24:24.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.203+0000 7f5bf0ff9700 10 client.49729.objecter ms_dispatch 0x7f5bf4001550 osd_map(722..722 src has 467..722) 2024-04-14T15:24:24.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.203+0000 7f5bf0ff9700 3 client.49729.objecter handle_osd_map got epochs [722,722] > 0 2024-04-14T15:24:24.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.203+0000 7f5bf0ff9700 3 client.49729.objecter handle_osd_map decoding full epoch 722 2024-04-14T15:24:24.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.203+0000 7f5bf0ff9700 20 client.49729.objecter dump_active .. 0 homeless 2024-04-14T15:24:24.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.203+0000 7f5bf0ff9700 1 -- 172.21.15.90:0/708853304 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v722) ==== 54+0+16 (unknown 2168025390 0 2420000192) 0x7f5bec08c1e0 con 0x7f5bf410f2f0 2024-04-14T15:24:24.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.203+0000 7f5bf0ff9700 10 client.49729.objecter ms_handle_connect 0x7f5bdc071300 2024-04-14T15:24:24.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.203+0000 7f5bfa249700 10 client.49729.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:24:24.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.203+0000 7f5bfa249700 15 client.49729.objecter _session_command_op_assign -1 1 2024-04-14T15:24:24.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.203+0000 7f5bfa249700 20 client.49729.objecter _get_session s=0x7f5be00053e0 osd=0 3 2024-04-14T15:24:24.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.203+0000 7f5bfa249700 20 client.49729.objecter put_session s=0x7f5be00053e0 osd=0 3 2024-04-14T15:24:24.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.203+0000 7f5bfa249700 20 client.49729.objecter _get_session s=0x7f5be00053e0 osd=0 3 2024-04-14T15:24:24.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.203+0000 7f5bfa249700 15 client.49729.objecter _session_command_op_remove -1 1 2024-04-14T15:24:24.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.203+0000 7f5bfa249700 20 client.49729.objecter get_session s=0x7f5be00053e0 osd=0 3 2024-04-14T15:24:24.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.203+0000 7f5bfa249700 15 client.49729.objecter _session_command_op_assign 0 1 2024-04-14T15:24:24.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.203+0000 7f5bfa249700 20 client.49729.objecter put_session s=0x7f5be00053e0 osd=0 4 2024-04-14T15:24:24.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.203+0000 7f5bfa249700 10 client.49729.objecter _send_command 1 2024-04-14T15:24:24.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.203+0000 7f5bfa249700 1 -- 172.21.15.90:0/708853304 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5be0004b50 con 0x7f5be0004730 2024-04-14T15:24:24.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.204+0000 7f5bf0ff9700 10 client.49729.objecter ms_handle_connect 0x7f5be0004730 2024-04-14T15:24:24.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.204+0000 7f5bf0ff9700 1 -- 172.21.15.90:0/708853304 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f5be0004b50 con 0x7f5be0004730 2024-04-14T15:24:24.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.204+0000 7f5bf0ff9700 10 client.49729.objecter ms_dispatch 0x7f5bf4001550 command_reply(tid 1: 0 ) 2024-04-14T15:24:24.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.204+0000 7f5bf0ff9700 10 client.49729.objecter _finish_command 1 = system:0 2024-04-14T15:24:24.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.204+0000 7f5bf0ff9700 20 client.49729.objecter put_session s=0x7f5be00053e0 osd=0 4 2024-04-14T15:24:24.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.204+0000 7f5bf0ff9700 15 client.49729.objecter _session_command_op_remove 0 1 2024-04-14T15:24:24.217 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:24.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:24.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:24:24.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.221+0000 7f5bfa249700 10 client.49729.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:24:24.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.221+0000 7f5bfa249700 15 client.49729.objecter _session_command_op_assign -1 2 2024-04-14T15:24:24.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.221+0000 7f5bfa249700 20 client.49729.objecter _get_session s=0x7f5be00053e0 osd=0 3 2024-04-14T15:24:24.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.221+0000 7f5bfa249700 20 client.49729.objecter put_session s=0x7f5be00053e0 osd=0 3 2024-04-14T15:24:24.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.221+0000 7f5bfa249700 20 client.49729.objecter _get_session s=0x7f5be00053e0 osd=0 3 2024-04-14T15:24:24.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.221+0000 7f5bfa249700 15 client.49729.objecter _session_command_op_remove -1 2 2024-04-14T15:24:24.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.221+0000 7f5bfa249700 20 client.49729.objecter get_session s=0x7f5be00053e0 osd=0 3 2024-04-14T15:24:24.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.221+0000 7f5bfa249700 15 client.49729.objecter _session_command_op_assign 0 2 2024-04-14T15:24:24.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.221+0000 7f5bfa249700 20 client.49729.objecter put_session s=0x7f5be00053e0 osd=0 4 2024-04-14T15:24:24.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.221+0000 7f5bfa249700 10 client.49729.objecter _send_command 2 2024-04-14T15:24:24.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.221+0000 7f5bfa249700 1 -- 172.21.15.90:0/708853304 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f5be0007b00 con 0x7f5be0004730 2024-04-14T15:24:24.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.223+0000 7f5bf0ff9700 1 -- 172.21.15.90:0/708853304 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f5be0007b00 con 0x7f5be0004730 2024-04-14T15:24:24.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.223+0000 7f5bf0ff9700 10 client.49729.objecter ms_dispatch 0x7f5bf4001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:24:24.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.223+0000 7f5bf0ff9700 10 client.49729.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:24:24.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.223+0000 7f5bf0ff9700 20 client.49729.objecter put_session s=0x7f5be00053e0 osd=0 4 2024-04-14T15:24:24.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.223+0000 7f5bf0ff9700 15 client.49729.objecter _session_command_op_remove 0 2 2024-04-14T15:24:24.223 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:24:24.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.223+0000 7f5bfa249700 10 client.49729.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:24:24.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.223+0000 7f5bfa249700 15 client.49729.objecter _session_command_op_assign -1 3 2024-04-14T15:24:24.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.223+0000 7f5bfa249700 20 client.49729.objecter _get_session s=0x7f5be0004130 osd=1 3 2024-04-14T15:24:24.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.223+0000 7f5bfa249700 20 client.49729.objecter put_session s=0x7f5be0004130 osd=1 3 2024-04-14T15:24:24.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.223+0000 7f5bfa249700 20 client.49729.objecter _get_session s=0x7f5be0004130 osd=1 3 2024-04-14T15:24:24.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.223+0000 7f5bfa249700 15 client.49729.objecter _session_command_op_remove -1 3 2024-04-14T15:24:24.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.223+0000 7f5bfa249700 20 client.49729.objecter get_session s=0x7f5be0004130 osd=1 3 2024-04-14T15:24:24.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.223+0000 7f5bfa249700 15 client.49729.objecter _session_command_op_assign 1 3 2024-04-14T15:24:24.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.223+0000 7f5bfa249700 20 client.49729.objecter put_session s=0x7f5be0004130 osd=1 4 2024-04-14T15:24:24.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.223+0000 7f5bfa249700 10 client.49729.objecter _send_command 3 2024-04-14T15:24:24.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.223+0000 7f5bfa249700 1 -- 172.21.15.90:0/708853304 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f5be0002130 con 0x7f5be00042a0 2024-04-14T15:24:24.225 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:24 smithi090 ceph-mon[41670]: 5.8 scrub starts 2024-04-14T15:24:24.225 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:24 smithi090 ceph-mon[41670]: 5.8 scrub ok 2024-04-14T15:24:24.225 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:24 smithi090 ceph-mon[41670]: pgmap v1065: 214 pgs: 32 unknown, 182 active+clean; 582 KiB data, 679 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:24:24.225 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:24 smithi090 ceph-mon[41670]: 1.1 scrub starts 2024-04-14T15:24:24.225 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:24 smithi090 ceph-mon[41670]: 1.1 scrub ok 2024-04-14T15:24:24.225 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:24 smithi090 ceph-mon[41670]: 6.17 scrub starts 2024-04-14T15:24:24.225 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:24 smithi090 ceph-mon[41670]: 6.17 scrub ok 2024-04-14T15:24:24.225 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:24 smithi090 ceph-mon[41670]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:24:24.225 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:24 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-126", "mode": "writeback"}]': finished 2024-04-14T15:24:24.225 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:24 smithi090 ceph-mon[41670]: osdmap e722: 8 total, 8 up, 7 in 2024-04-14T15:24:24.226 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:24 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:24:24.226 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:24 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:24:24.226 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:24 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:24:24.226 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:24 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:24:24.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:24 smithi090 ceph-mon[30399]: 5.8 scrub starts 2024-04-14T15:24:24.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:24 smithi090 ceph-mon[30399]: 5.8 scrub ok 2024-04-14T15:24:24.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:24 smithi090 ceph-mon[30399]: pgmap v1065: 214 pgs: 32 unknown, 182 active+clean; 582 KiB data, 679 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:24:24.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:24 smithi090 ceph-mon[30399]: 1.1 scrub starts 2024-04-14T15:24:24.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:24 smithi090 ceph-mon[30399]: 1.1 scrub ok 2024-04-14T15:24:24.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:24 smithi090 ceph-mon[30399]: 6.17 scrub starts 2024-04-14T15:24:24.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:24 smithi090 ceph-mon[30399]: 6.17 scrub ok 2024-04-14T15:24:24.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:24 smithi090 ceph-mon[30399]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:24:24.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:24 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-126", "mode": "writeback"}]': finished 2024-04-14T15:24:24.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:24 smithi090 ceph-mon[30399]: osdmap e722: 8 total, 8 up, 7 in 2024-04-14T15:24:24.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:24 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:24:24.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:24 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:24:24.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:24 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:24:24.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:24 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:24:24.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.223+0000 7f5bf0ff9700 10 client.49729.objecter ms_handle_connect 0x7f5be00042a0 2024-04-14T15:24:24.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.224+0000 7f5bf0ff9700 1 -- 172.21.15.90:0/708853304 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f5be0002130 con 0x7f5be00042a0 2024-04-14T15:24:24.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.224+0000 7f5bf0ff9700 10 client.49729.objecter ms_dispatch 0x7f5bf4001550 command_reply(tid 3: 0 ) 2024-04-14T15:24:24.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.224+0000 7f5bf0ff9700 10 client.49729.objecter _finish_command 3 = system:0 2024-04-14T15:24:24.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.224+0000 7f5bf0ff9700 20 client.49729.objecter put_session s=0x7f5be0004130 osd=1 4 2024-04-14T15:24:24.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.224+0000 7f5bf0ff9700 15 client.49729.objecter _session_command_op_remove 1 3 2024-04-14T15:24:24.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.241+0000 7f5bfa249700 10 client.49729.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:24:24.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.241+0000 7f5bfa249700 15 client.49729.objecter _session_command_op_assign -1 4 2024-04-14T15:24:24.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.241+0000 7f5bfa249700 20 client.49729.objecter _get_session s=0x7f5be0004130 osd=1 3 2024-04-14T15:24:24.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.241+0000 7f5bfa249700 20 client.49729.objecter put_session s=0x7f5be0004130 osd=1 3 2024-04-14T15:24:24.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.241+0000 7f5bfa249700 20 client.49729.objecter _get_session s=0x7f5be0004130 osd=1 3 2024-04-14T15:24:24.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.241+0000 7f5bfa249700 15 client.49729.objecter _session_command_op_remove -1 4 2024-04-14T15:24:24.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.241+0000 7f5bfa249700 20 client.49729.objecter get_session s=0x7f5be0004130 osd=1 3 2024-04-14T15:24:24.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.241+0000 7f5bfa249700 15 client.49729.objecter _session_command_op_assign 1 4 2024-04-14T15:24:24.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.241+0000 7f5bfa249700 20 client.49729.objecter put_session s=0x7f5be0004130 osd=1 4 2024-04-14T15:24:24.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.241+0000 7f5bfa249700 10 client.49729.objecter _send_command 4 2024-04-14T15:24:24.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.241+0000 7f5bfa249700 1 -- 172.21.15.90:0/708853304 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f5be0001600 con 0x7f5be00042a0 2024-04-14T15:24:24.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.242+0000 7f5bf0ff9700 1 -- 172.21.15.90:0/708853304 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f5be0001600 con 0x7f5be00042a0 2024-04-14T15:24:24.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.242+0000 7f5bf0ff9700 10 client.49729.objecter ms_dispatch 0x7f5bf4001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:24:24.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.242+0000 7f5bf0ff9700 10 client.49729.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:24:24.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.242+0000 7f5bf0ff9700 20 client.49729.objecter put_session s=0x7f5be0004130 osd=1 4 2024-04-14T15:24:24.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.242+0000 7f5bf0ff9700 15 client.49729.objecter _session_command_op_remove 1 4 2024-04-14T15:24:24.243 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:24:24.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.242+0000 7f5bfa249700 10 client.49729.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:24:24.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.242+0000 7f5bfa249700 15 client.49729.objecter _session_command_op_assign -1 5 2024-04-14T15:24:24.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.242+0000 7f5bfa249700 20 client.49729.objecter _get_session s=0x7f5be0001a00 osd=2 3 2024-04-14T15:24:24.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.242+0000 7f5bfa249700 20 client.49729.objecter put_session s=0x7f5be0001a00 osd=2 3 2024-04-14T15:24:24.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.242+0000 7f5bfa249700 20 client.49729.objecter _get_session s=0x7f5be0001a00 osd=2 3 2024-04-14T15:24:24.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.242+0000 7f5bfa249700 15 client.49729.objecter _session_command_op_remove -1 5 2024-04-14T15:24:24.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.242+0000 7f5bfa249700 20 client.49729.objecter get_session s=0x7f5be0001a00 osd=2 3 2024-04-14T15:24:24.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.242+0000 7f5bfa249700 15 client.49729.objecter _session_command_op_assign 2 5 2024-04-14T15:24:24.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.242+0000 7f5bfa249700 20 client.49729.objecter put_session s=0x7f5be0001a00 osd=2 4 2024-04-14T15:24:24.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.242+0000 7f5bfa249700 10 client.49729.objecter _send_command 5 2024-04-14T15:24:24.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.242+0000 7f5bfa249700 1 -- 172.21.15.90:0/708853304 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f5be0010fe0 con 0x7f5be0008990 2024-04-14T15:24:24.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.243+0000 7f5bf0ff9700 10 client.49729.objecter ms_handle_connect 0x7f5be0008990 2024-04-14T15:24:24.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.243+0000 7f5bf0ff9700 1 -- 172.21.15.90:0/708853304 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f5be0010fe0 con 0x7f5be0008990 2024-04-14T15:24:24.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.243+0000 7f5bf0ff9700 10 client.49729.objecter ms_dispatch 0x7f5bf4001550 command_reply(tid 5: 0 ) 2024-04-14T15:24:24.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.243+0000 7f5bf0ff9700 10 client.49729.objecter _finish_command 5 = system:0 2024-04-14T15:24:24.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.243+0000 7f5bf0ff9700 20 client.49729.objecter put_session s=0x7f5be0001a00 osd=2 4 2024-04-14T15:24:24.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.243+0000 7f5bf0ff9700 15 client.49729.objecter _session_command_op_remove 2 5 2024-04-14T15:24:24.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.260+0000 7f5bfa249700 10 client.49729.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:24:24.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.260+0000 7f5bfa249700 15 client.49729.objecter _session_command_op_assign -1 6 2024-04-14T15:24:24.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.260+0000 7f5bfa249700 20 client.49729.objecter _get_session s=0x7f5be0001a00 osd=2 3 2024-04-14T15:24:24.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.260+0000 7f5bfa249700 20 client.49729.objecter put_session s=0x7f5be0001a00 osd=2 3 2024-04-14T15:24:24.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.260+0000 7f5bfa249700 20 client.49729.objecter _get_session s=0x7f5be0001a00 osd=2 3 2024-04-14T15:24:24.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.260+0000 7f5bfa249700 15 client.49729.objecter _session_command_op_remove -1 6 2024-04-14T15:24:24.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.260+0000 7f5bfa249700 20 client.49729.objecter get_session s=0x7f5be0001a00 osd=2 3 2024-04-14T15:24:24.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.260+0000 7f5bfa249700 15 client.49729.objecter _session_command_op_assign 2 6 2024-04-14T15:24:24.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.260+0000 7f5bfa249700 20 client.49729.objecter put_session s=0x7f5be0001a00 osd=2 4 2024-04-14T15:24:24.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.260+0000 7f5bfa249700 10 client.49729.objecter _send_command 6 2024-04-14T15:24:24.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.260+0000 7f5bfa249700 1 -- 172.21.15.90:0/708853304 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f5be0007f60 con 0x7f5be0008990 2024-04-14T15:24:24.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.261+0000 7f5bf0ff9700 1 -- 172.21.15.90:0/708853304 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f5be0007f60 con 0x7f5be0008990 2024-04-14T15:24:24.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.261+0000 7f5bf0ff9700 10 client.49729.objecter ms_dispatch 0x7f5bf4001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:24:24.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.261+0000 7f5bf0ff9700 10 client.49729.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:24:24.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.261+0000 7f5bf0ff9700 20 client.49729.objecter put_session s=0x7f5be0001a00 osd=2 4 2024-04-14T15:24:24.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.261+0000 7f5bf0ff9700 15 client.49729.objecter _session_command_op_remove 2 6 2024-04-14T15:24:24.262 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:24:24.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.262+0000 7f5bfa249700 10 client.49729.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:24:24.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.262+0000 7f5bfa249700 15 client.49729.objecter _session_command_op_assign -1 7 2024-04-14T15:24:24.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.262+0000 7f5bfa249700 20 client.49729.objecter _get_session s=0x7f5be00081e0 osd=3 3 2024-04-14T15:24:24.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.262+0000 7f5bfa249700 20 client.49729.objecter put_session s=0x7f5be00081e0 osd=3 3 2024-04-14T15:24:24.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.262+0000 7f5bfa249700 20 client.49729.objecter _get_session s=0x7f5be00081e0 osd=3 3 2024-04-14T15:24:24.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.262+0000 7f5bfa249700 15 client.49729.objecter _session_command_op_remove -1 7 2024-04-14T15:24:24.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.262+0000 7f5bfa249700 20 client.49729.objecter get_session s=0x7f5be00081e0 osd=3 3 2024-04-14T15:24:24.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.262+0000 7f5bfa249700 15 client.49729.objecter _session_command_op_assign 3 7 2024-04-14T15:24:24.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.262+0000 7f5bfa249700 20 client.49729.objecter put_session s=0x7f5be00081e0 osd=3 4 2024-04-14T15:24:24.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.262+0000 7f5bfa249700 10 client.49729.objecter _send_command 7 2024-04-14T15:24:24.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.262+0000 7f5bfa249700 1 -- 172.21.15.90:0/708853304 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f5be000d490 con 0x7f5be00111c0 2024-04-14T15:24:24.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.262+0000 7f5bf0ff9700 10 client.49729.objecter ms_handle_connect 0x7f5be00111c0 2024-04-14T15:24:24.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.262+0000 7f5bf0ff9700 1 -- 172.21.15.90:0/708853304 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f5be000d490 con 0x7f5be00111c0 2024-04-14T15:24:24.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.262+0000 7f5bf0ff9700 10 client.49729.objecter ms_dispatch 0x7f5bf4001550 command_reply(tid 7: 0 ) 2024-04-14T15:24:24.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.262+0000 7f5bf0ff9700 10 client.49729.objecter _finish_command 7 = system:0 2024-04-14T15:24:24.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.262+0000 7f5bf0ff9700 20 client.49729.objecter put_session s=0x7f5be00081e0 osd=3 4 2024-04-14T15:24:24.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.262+0000 7f5bf0ff9700 15 client.49729.objecter _session_command_op_remove 3 7 2024-04-14T15:24:24.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.279+0000 7f5bfa249700 10 client.49729.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:24:24.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.279+0000 7f5bfa249700 15 client.49729.objecter _session_command_op_assign -1 8 2024-04-14T15:24:24.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.279+0000 7f5bfa249700 20 client.49729.objecter _get_session s=0x7f5be00081e0 osd=3 3 2024-04-14T15:24:24.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.279+0000 7f5bfa249700 20 client.49729.objecter put_session s=0x7f5be00081e0 osd=3 3 2024-04-14T15:24:24.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.279+0000 7f5bfa249700 20 client.49729.objecter _get_session s=0x7f5be00081e0 osd=3 3 2024-04-14T15:24:24.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.279+0000 7f5bfa249700 15 client.49729.objecter _session_command_op_remove -1 8 2024-04-14T15:24:24.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.279+0000 7f5bfa249700 20 client.49729.objecter get_session s=0x7f5be00081e0 osd=3 3 2024-04-14T15:24:24.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.279+0000 7f5bfa249700 15 client.49729.objecter _session_command_op_assign 3 8 2024-04-14T15:24:24.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.279+0000 7f5bfa249700 20 client.49729.objecter put_session s=0x7f5be00081e0 osd=3 4 2024-04-14T15:24:24.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.279+0000 7f5bfa249700 10 client.49729.objecter _send_command 8 2024-04-14T15:24:24.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.279+0000 7f5bfa249700 1 -- 172.21.15.90:0/708853304 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f5be0012550 con 0x7f5be00111c0 2024-04-14T15:24:24.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.281+0000 7f5bf0ff9700 1 -- 172.21.15.90:0/708853304 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f5be0012550 con 0x7f5be00111c0 2024-04-14T15:24:24.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.281+0000 7f5bf0ff9700 10 client.49729.objecter ms_dispatch 0x7f5bf4001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:24:24.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.281+0000 7f5bf0ff9700 10 client.49729.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:24:24.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.281+0000 7f5bf0ff9700 20 client.49729.objecter put_session s=0x7f5be00081e0 osd=3 4 2024-04-14T15:24:24.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.281+0000 7f5bf0ff9700 15 client.49729.objecter _session_command_op_remove 3 8 2024-04-14T15:24:24.282 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:24:24.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.281+0000 7f5bfa249700 10 client.49729.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:24:24.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.281+0000 7f5bfa249700 15 client.49729.objecter _session_command_op_assign -1 9 2024-04-14T15:24:24.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.281+0000 7f5bfa249700 20 client.49729.objecter _get_session s=0x7f5be0013980 osd=4 3 2024-04-14T15:24:24.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.281+0000 7f5bfa249700 20 client.49729.objecter put_session s=0x7f5be0013980 osd=4 3 2024-04-14T15:24:24.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.281+0000 7f5bfa249700 20 client.49729.objecter _get_session s=0x7f5be0013980 osd=4 3 2024-04-14T15:24:24.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.281+0000 7f5bfa249700 15 client.49729.objecter _session_command_op_remove -1 9 2024-04-14T15:24:24.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.281+0000 7f5bfa249700 20 client.49729.objecter get_session s=0x7f5be0013980 osd=4 3 2024-04-14T15:24:24.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.281+0000 7f5bfa249700 15 client.49729.objecter _session_command_op_assign 4 9 2024-04-14T15:24:24.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.281+0000 7f5bfa249700 20 client.49729.objecter put_session s=0x7f5be0013980 osd=4 4 2024-04-14T15:24:24.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.281+0000 7f5bfa249700 10 client.49729.objecter _send_command 9 2024-04-14T15:24:24.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.281+0000 7f5bfa249700 1 -- 172.21.15.90:0/708853304 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f5be001a7e0 con 0x7f5be000cac0 2024-04-14T15:24:24.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.282+0000 7f5bf0ff9700 10 client.49729.objecter ms_handle_connect 0x7f5be000cac0 2024-04-14T15:24:24.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.282+0000 7f5bf0ff9700 1 -- 172.21.15.90:0/708853304 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f5be001a7e0 con 0x7f5be000cac0 2024-04-14T15:24:24.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.282+0000 7f5bf0ff9700 10 client.49729.objecter ms_dispatch 0x7f5bf4001550 command_reply(tid 9: 0 ) 2024-04-14T15:24:24.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.282+0000 7f5bf0ff9700 10 client.49729.objecter _finish_command 9 = system:0 2024-04-14T15:24:24.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.282+0000 7f5bf0ff9700 20 client.49729.objecter put_session s=0x7f5be0013980 osd=4 4 2024-04-14T15:24:24.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.282+0000 7f5bf0ff9700 15 client.49729.objecter _session_command_op_remove 4 9 2024-04-14T15:24:24.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.298+0000 7f5bfa249700 10 client.49729.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:24:24.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.298+0000 7f5bfa249700 15 client.49729.objecter _session_command_op_assign -1 10 2024-04-14T15:24:24.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.298+0000 7f5bfa249700 20 client.49729.objecter _get_session s=0x7f5be0013980 osd=4 3 2024-04-14T15:24:24.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.298+0000 7f5bfa249700 20 client.49729.objecter put_session s=0x7f5be0013980 osd=4 3 2024-04-14T15:24:24.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.298+0000 7f5bfa249700 20 client.49729.objecter _get_session s=0x7f5be0013980 osd=4 3 2024-04-14T15:24:24.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.298+0000 7f5bfa249700 15 client.49729.objecter _session_command_op_remove -1 10 2024-04-14T15:24:24.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.298+0000 7f5bfa249700 20 client.49729.objecter get_session s=0x7f5be0013980 osd=4 3 2024-04-14T15:24:24.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.298+0000 7f5bfa249700 15 client.49729.objecter _session_command_op_assign 4 10 2024-04-14T15:24:24.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.298+0000 7f5bfa249700 20 client.49729.objecter put_session s=0x7f5be0013980 osd=4 4 2024-04-14T15:24:24.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.298+0000 7f5bfa249700 10 client.49729.objecter _send_command 10 2024-04-14T15:24:24.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.298+0000 7f5bfa249700 1 -- 172.21.15.90:0/708853304 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f5be0011fd0 con 0x7f5be000cac0 2024-04-14T15:24:24.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.300+0000 7f5bf0ff9700 1 -- 172.21.15.90:0/708853304 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f5be0011fd0 con 0x7f5be000cac0 2024-04-14T15:24:24.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.300+0000 7f5bf0ff9700 10 client.49729.objecter ms_dispatch 0x7f5bf4001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:24:24.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.300+0000 7f5bf0ff9700 10 client.49729.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:24:24.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.300+0000 7f5bf0ff9700 20 client.49729.objecter put_session s=0x7f5be0013980 osd=4 4 2024-04-14T15:24:24.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.300+0000 7f5bf0ff9700 15 client.49729.objecter _session_command_op_remove 4 10 2024-04-14T15:24:24.301 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:24:24.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.301+0000 7f5bfa249700 10 client.49729.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:24:24.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.301+0000 7f5bfa249700 15 client.49729.objecter _session_command_op_assign -1 11 2024-04-14T15:24:24.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.301+0000 7f5bfa249700 20 client.49729.objecter _get_session s=0x7f5be000c930 osd=5 3 2024-04-14T15:24:24.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.301+0000 7f5bfa249700 20 client.49729.objecter put_session s=0x7f5be000c930 osd=5 3 2024-04-14T15:24:24.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.301+0000 7f5bfa249700 20 client.49729.objecter _get_session s=0x7f5be000c930 osd=5 3 2024-04-14T15:24:24.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.301+0000 7f5bfa249700 15 client.49729.objecter _session_command_op_remove -1 11 2024-04-14T15:24:24.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.301+0000 7f5bfa249700 20 client.49729.objecter get_session s=0x7f5be000c930 osd=5 3 2024-04-14T15:24:24.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.301+0000 7f5bfa249700 15 client.49729.objecter _session_command_op_assign 5 11 2024-04-14T15:24:24.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.301+0000 7f5bfa249700 20 client.49729.objecter put_session s=0x7f5be000c930 osd=5 4 2024-04-14T15:24:24.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.301+0000 7f5bfa249700 10 client.49729.objecter _send_command 11 2024-04-14T15:24:24.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.301+0000 7f5bfa249700 1 -- 172.21.15.90:0/708853304 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f5be001d0c0 con 0x7f5be0012fd0 2024-04-14T15:24:24.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.301+0000 7f5bf0ff9700 10 client.49729.objecter ms_handle_connect 0x7f5be0012fd0 2024-04-14T15:24:24.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.302+0000 7f5bf0ff9700 1 -- 172.21.15.90:0/708853304 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f5be001d0c0 con 0x7f5be0012fd0 2024-04-14T15:24:24.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.302+0000 7f5bf0ff9700 10 client.49729.objecter ms_dispatch 0x7f5bf4001550 command_reply(tid 11: 0 ) 2024-04-14T15:24:24.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.302+0000 7f5bf0ff9700 10 client.49729.objecter _finish_command 11 = system:0 2024-04-14T15:24:24.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.302+0000 7f5bf0ff9700 20 client.49729.objecter put_session s=0x7f5be000c930 osd=5 4 2024-04-14T15:24:24.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.302+0000 7f5bf0ff9700 15 client.49729.objecter _session_command_op_remove 5 11 2024-04-14T15:24:24.318 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:24:24.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.319+0000 7f5bfa249700 10 client.49729.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:24:24.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.319+0000 7f5bfa249700 15 client.49729.objecter _session_command_op_assign -1 12 2024-04-14T15:24:24.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.319+0000 7f5bfa249700 20 client.49729.objecter _get_session s=0x7f5be000c930 osd=5 3 2024-04-14T15:24:24.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.319+0000 7f5bfa249700 20 client.49729.objecter put_session s=0x7f5be000c930 osd=5 3 2024-04-14T15:24:24.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.319+0000 7f5bfa249700 20 client.49729.objecter _get_session s=0x7f5be000c930 osd=5 3 2024-04-14T15:24:24.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.319+0000 7f5bfa249700 15 client.49729.objecter _session_command_op_remove -1 12 2024-04-14T15:24:24.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.319+0000 7f5bfa249700 20 client.49729.objecter get_session s=0x7f5be000c930 osd=5 3 2024-04-14T15:24:24.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.319+0000 7f5bfa249700 15 client.49729.objecter _session_command_op_assign 5 12 2024-04-14T15:24:24.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.319+0000 7f5bfa249700 20 client.49729.objecter put_session s=0x7f5be000c930 osd=5 4 2024-04-14T15:24:24.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.319+0000 7f5bfa249700 10 client.49729.objecter _send_command 12 2024-04-14T15:24:24.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.319+0000 7f5bfa249700 1 -- 172.21.15.90:0/708853304 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f5be0013570 con 0x7f5be0012fd0 2024-04-14T15:24:24.321 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:24:24.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.321+0000 7f5bf0ff9700 1 -- 172.21.15.90:0/708853304 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f5be0013570 con 0x7f5be0012fd0 2024-04-14T15:24:24.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.321+0000 7f5bf0ff9700 10 client.49729.objecter ms_dispatch 0x7f5bf4001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:24:24.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.321+0000 7f5bf0ff9700 10 client.49729.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:24:24.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.321+0000 7f5bf0ff9700 20 client.49729.objecter put_session s=0x7f5be000c930 osd=5 4 2024-04-14T15:24:24.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.321+0000 7f5bf0ff9700 15 client.49729.objecter _session_command_op_remove 5 12 2024-04-14T15:24:24.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.321+0000 7f5bfa249700 10 client.49729.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:24:24.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.321+0000 7f5bfa249700 15 client.49729.objecter _session_command_op_assign -1 13 2024-04-14T15:24:24.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.321+0000 7f5bfa249700 20 client.49729.objecter _get_session s=0x7f5be001d120 osd=6 3 2024-04-14T15:24:24.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.321+0000 7f5bfa249700 20 client.49729.objecter put_session s=0x7f5be001d120 osd=6 3 2024-04-14T15:24:24.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.321+0000 7f5bfa249700 20 client.49729.objecter _get_session s=0x7f5be001d120 osd=6 3 2024-04-14T15:24:24.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.321+0000 7f5bfa249700 15 client.49729.objecter _session_command_op_remove -1 13 2024-04-14T15:24:24.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.321+0000 7f5bfa249700 20 client.49729.objecter get_session s=0x7f5be001d120 osd=6 3 2024-04-14T15:24:24.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.321+0000 7f5bfa249700 15 client.49729.objecter _session_command_op_assign 6 13 2024-04-14T15:24:24.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.321+0000 7f5bfa249700 20 client.49729.objecter put_session s=0x7f5be001d120 osd=6 4 2024-04-14T15:24:24.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.321+0000 7f5bfa249700 10 client.49729.objecter _send_command 13 2024-04-14T15:24:24.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.321+0000 7f5bfa249700 1 -- 172.21.15.90:0/708853304 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f5be0023160 con 0x7f5be0013570 2024-04-14T15:24:24.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.322+0000 7f5bf0ff9700 10 client.49729.objecter ms_handle_connect 0x7f5be0013570 2024-04-14T15:24:24.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.322+0000 7f5bf0ff9700 1 -- 172.21.15.90:0/708853304 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f5be0023160 con 0x7f5be0013570 2024-04-14T15:24:24.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.322+0000 7f5bf0ff9700 10 client.49729.objecter ms_dispatch 0x7f5bf4001550 command_reply(tid 13: 0 ) 2024-04-14T15:24:24.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.322+0000 7f5bf0ff9700 10 client.49729.objecter _finish_command 13 = system:0 2024-04-14T15:24:24.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.322+0000 7f5bf0ff9700 20 client.49729.objecter put_session s=0x7f5be001d120 osd=6 4 2024-04-14T15:24:24.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.322+0000 7f5bf0ff9700 15 client.49729.objecter _session_command_op_remove 6 13 2024-04-14T15:24:24.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.340+0000 7f5bfa249700 10 client.49729.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:24:24.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.340+0000 7f5bfa249700 15 client.49729.objecter _session_command_op_assign -1 14 2024-04-14T15:24:24.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.340+0000 7f5bfa249700 20 client.49729.objecter _get_session s=0x7f5be001d120 osd=6 3 2024-04-14T15:24:24.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.340+0000 7f5bfa249700 20 client.49729.objecter put_session s=0x7f5be001d120 osd=6 3 2024-04-14T15:24:24.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.340+0000 7f5bfa249700 20 client.49729.objecter _get_session s=0x7f5be001d120 osd=6 3 2024-04-14T15:24:24.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.340+0000 7f5bfa249700 15 client.49729.objecter _session_command_op_remove -1 14 2024-04-14T15:24:24.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.340+0000 7f5bfa249700 20 client.49729.objecter get_session s=0x7f5be001d120 osd=6 3 2024-04-14T15:24:24.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.340+0000 7f5bfa249700 15 client.49729.objecter _session_command_op_assign 6 14 2024-04-14T15:24:24.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.340+0000 7f5bfa249700 20 client.49729.objecter put_session s=0x7f5be001d120 osd=6 4 2024-04-14T15:24:24.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.340+0000 7f5bfa249700 10 client.49729.objecter _send_command 14 2024-04-14T15:24:24.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.340+0000 7f5bfa249700 1 -- 172.21.15.90:0/708853304 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f5be0012a50 con 0x7f5be0013570 2024-04-14T15:24:24.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.342+0000 7f5bf0ff9700 1 -- 172.21.15.90:0/708853304 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f5be0012a50 con 0x7f5be0013570 2024-04-14T15:24:24.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.342+0000 7f5bf0ff9700 10 client.49729.objecter ms_dispatch 0x7f5bf4001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:24:24.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.342+0000 7f5bf0ff9700 10 client.49729.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:24:24.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.342+0000 7f5bf0ff9700 20 client.49729.objecter put_session s=0x7f5be001d120 osd=6 4 2024-04-14T15:24:24.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.342+0000 7f5bf0ff9700 15 client.49729.objecter _session_command_op_remove 6 14 2024-04-14T15:24:24.343 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:24:24.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.342+0000 7f5bfa249700 10 client.49729.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:24:24.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.342+0000 7f5bfa249700 15 client.49729.objecter _session_command_op_assign -1 15 2024-04-14T15:24:24.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.342+0000 7f5bfa249700 20 client.49729.objecter _get_session s=0x7f5be0012800 osd=7 3 2024-04-14T15:24:24.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.342+0000 7f5bfa249700 20 client.49729.objecter put_session s=0x7f5be0012800 osd=7 3 2024-04-14T15:24:24.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.342+0000 7f5bfa249700 20 client.49729.objecter _get_session s=0x7f5be0012800 osd=7 3 2024-04-14T15:24:24.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.342+0000 7f5bfa249700 15 client.49729.objecter _session_command_op_remove -1 15 2024-04-14T15:24:24.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.342+0000 7f5bfa249700 20 client.49729.objecter get_session s=0x7f5be0012800 osd=7 3 2024-04-14T15:24:24.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.342+0000 7f5bfa249700 15 client.49729.objecter _session_command_op_assign 7 15 2024-04-14T15:24:24.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.342+0000 7f5bfa249700 20 client.49729.objecter put_session s=0x7f5be0012800 osd=7 4 2024-04-14T15:24:24.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.342+0000 7f5bfa249700 10 client.49729.objecter _send_command 15 2024-04-14T15:24:24.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.342+0000 7f5bfa249700 1 -- 172.21.15.90:0/708853304 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f5be0012970 con 0x7f5be001afb0 2024-04-14T15:24:24.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.343+0000 7f5bf0ff9700 10 client.49729.objecter ms_handle_connect 0x7f5be001afb0 2024-04-14T15:24:24.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.343+0000 7f5bf0ff9700 1 -- 172.21.15.90:0/708853304 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f5be0012970 con 0x7f5be001afb0 2024-04-14T15:24:24.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.343+0000 7f5bf0ff9700 10 client.49729.objecter ms_dispatch 0x7f5bf4001550 command_reply(tid 15: 0 ) 2024-04-14T15:24:24.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.343+0000 7f5bf0ff9700 10 client.49729.objecter _finish_command 15 = system:0 2024-04-14T15:24:24.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.343+0000 7f5bf0ff9700 20 client.49729.objecter put_session s=0x7f5be0012800 osd=7 4 2024-04-14T15:24:24.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.343+0000 7f5bf0ff9700 15 client.49729.objecter _session_command_op_remove 7 15 2024-04-14T15:24:24.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:24.345 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:24.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:24:24.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.360+0000 7f5bfa249700 10 client.49729.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:24:24.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.360+0000 7f5bfa249700 15 client.49729.objecter _session_command_op_assign -1 16 2024-04-14T15:24:24.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.360+0000 7f5bfa249700 20 client.49729.objecter _get_session s=0x7f5be0012800 osd=7 3 2024-04-14T15:24:24.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.360+0000 7f5bfa249700 20 client.49729.objecter put_session s=0x7f5be0012800 osd=7 3 2024-04-14T15:24:24.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.360+0000 7f5bfa249700 20 client.49729.objecter _get_session s=0x7f5be0012800 osd=7 3 2024-04-14T15:24:24.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.360+0000 7f5bfa249700 15 client.49729.objecter _session_command_op_remove -1 16 2024-04-14T15:24:24.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.360+0000 7f5bfa249700 20 client.49729.objecter get_session s=0x7f5be0012800 osd=7 3 2024-04-14T15:24:24.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.360+0000 7f5bfa249700 15 client.49729.objecter _session_command_op_assign 7 16 2024-04-14T15:24:24.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.360+0000 7f5bfa249700 20 client.49729.objecter put_session s=0x7f5be0012800 osd=7 4 2024-04-14T15:24:24.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.360+0000 7f5bfa249700 10 client.49729.objecter _send_command 16 2024-04-14T15:24:24.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.360+0000 7f5bfa249700 1 -- 172.21.15.90:0/708853304 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f5be0012970 con 0x7f5be001afb0 2024-04-14T15:24:24.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.362+0000 7f5bf0ff9700 1 -- 172.21.15.90:0/708853304 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f5be0012970 con 0x7f5be001afb0 2024-04-14T15:24:24.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.362+0000 7f5bf0ff9700 10 client.49729.objecter ms_dispatch 0x7f5bf4001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:24:24.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.362+0000 7f5bf0ff9700 10 client.49729.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:24:24.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.362+0000 7f5bf0ff9700 20 client.49729.objecter put_session s=0x7f5be0012800 osd=7 4 2024-04-14T15:24:24.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.362+0000 7f5bf0ff9700 15 client.49729.objecter _session_command_op_remove 7 16 2024-04-14T15:24:24.363 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:24:24.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.362+0000 7f5bfa249700 10 librados: watch_flush enter 2024-04-14T15:24:24.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.362+0000 7f5bfa249700 10 librados: watch_flush exit 2024-04-14T15:24:24.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.362+0000 7f5bfa249700 10 client.49729.objecter close_session for osd.0 2024-04-14T15:24:24.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.362+0000 7f5bfa249700 1 -- 172.21.15.90:0/708853304 >> v1:172.21.15.90:6801/3761806885 conn(0x7f5be0004730 legacy=0x7f5be0002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:24.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.362+0000 7f5bfa249700 20 client.49729.objecter put_session s=0x7f5be00053e0 osd=0 1 2024-04-14T15:24:24.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.362+0000 7f5bfa249700 10 client.49729.objecter close_session for osd.1 2024-04-14T15:24:24.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.362+0000 7f5bfa249700 1 -- 172.21.15.90:0/708853304 >> v1:172.21.15.90:6805/4170361699 conn(0x7f5be00042a0 legacy=0x7f5be000ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:24.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.362+0000 7f5bfa249700 20 client.49729.objecter put_session s=0x7f5be0004130 osd=1 1 2024-04-14T15:24:24.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.362+0000 7f5bfa249700 10 client.49729.objecter close_session for osd.2 2024-04-14T15:24:24.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.362+0000 7f5bfa249700 1 -- 172.21.15.90:0/708853304 >> v1:172.21.15.90:6809/1686184251 conn(0x7f5be0008990 legacy=0x7f5be000f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:24.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.362+0000 7f5bfa249700 20 client.49729.objecter put_session s=0x7f5be0001a00 osd=2 1 2024-04-14T15:24:24.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.362+0000 7f5bfa249700 10 client.49729.objecter close_session for osd.3 2024-04-14T15:24:24.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.362+0000 7f5bfa249700 1 -- 172.21.15.90:0/708853304 >> v1:172.21.15.90:6813/2270166774 conn(0x7f5be00111c0 legacy=0x7f5be0011620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:24.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.362+0000 7f5bfa249700 20 client.49729.objecter put_session s=0x7f5be00081e0 osd=3 1 2024-04-14T15:24:24.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.362+0000 7f5bfa249700 10 client.49729.objecter close_session for osd.4 2024-04-14T15:24:24.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.362+0000 7f5bfa249700 1 -- 172.21.15.90:0/708853304 >> v1:172.21.15.92:6800/3887549733 conn(0x7f5be000cac0 legacy=0x7f5be00190c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:24.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.362+0000 7f5bfa249700 20 client.49729.objecter put_session s=0x7f5be0013980 osd=4 1 2024-04-14T15:24:24.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.362+0000 7f5bf3fff700 1 -- 172.21.15.90:0/708853304 reap_dead start 2024-04-14T15:24:24.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.362+0000 7f5bfa249700 10 client.49729.objecter close_session for osd.5 2024-04-14T15:24:24.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.362+0000 7f5bfa249700 1 -- 172.21.15.90:0/708853304 >> v1:172.21.15.92:6804/3244561440 conn(0x7f5be0012fd0 legacy=0x7f5be001b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:24.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.362+0000 7f5bf3fff700 1 -- 172.21.15.90:0/708853304 reap_dead start 2024-04-14T15:24:24.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.362+0000 7f5bf3fff700 1 -- 172.21.15.90:0/708853304 reap_dead start 2024-04-14T15:24:24.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.362+0000 7f5bfa249700 20 client.49729.objecter put_session s=0x7f5be000c930 osd=5 1 2024-04-14T15:24:24.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.362+0000 7f5bfa249700 10 client.49729.objecter close_session for osd.6 2024-04-14T15:24:24.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.362+0000 7f5bfa249700 1 -- 172.21.15.90:0/708853304 >> v1:172.21.15.92:6808/636125963 conn(0x7f5be0013570 legacy=0x7f5be0021870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:24.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.362+0000 7f5bfa249700 20 client.49729.objecter put_session s=0x7f5be001d120 osd=6 1 2024-04-14T15:24:24.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.362+0000 7f5bfa249700 10 client.49729.objecter close_session for osd.7 2024-04-14T15:24:24.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.362+0000 7f5bfa249700 1 -- 172.21.15.90:0/708853304 >> v1:172.21.15.92:6812/1773706716 conn(0x7f5be001afb0 legacy=0x7f5be0026b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:24.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.362+0000 7f5bfa249700 20 client.49729.objecter put_session s=0x7f5be0012800 osd=7 1 2024-04-14T15:24:24.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.362+0000 7f5bfa249700 20 client.49729.objecter shutdown clearing up homeless session... 2024-04-14T15:24:24.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.362+0000 7f5bfa249700 10 client.49729.objecter successfully canceled tick 2024-04-14T15:24:24.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.363+0000 7f5bfa249700 1 -- 172.21.15.90:0/708853304 >> v1:172.21.15.90:6800/420935427 conn(0x7f5bdc071300 legacy=0x7f5bdc0737c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:24.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.363+0000 7f5bfa249700 1 -- 172.21.15.90:0/708853304 >> v1:172.21.15.92:6789/0 conn(0x7f5bf410f2f0 legacy=0x7f5bf41a84a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:24.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.363+0000 7f5bf3fff700 1 -- 172.21.15.90:0/708853304 reap_dead start 2024-04-14T15:24:24.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.363+0000 7f5bfa249700 1 -- 172.21.15.90:0/708853304 shutdown_connections 2024-04-14T15:24:24.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.363+0000 7f5bfa249700 1 -- 172.21.15.90:0/708853304 >> 172.21.15.90:0/708853304 conn(0x7f5bf4076aa0 msgr2=0x7f5bf4109ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:24:24.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.363+0000 7f5bfa249700 1 -- 172.21.15.90:0/708853304 shutdown_connections 2024-04-14T15:24:24.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.363+0000 7f5bfa249700 1 -- 172.21.15.90:0/708853304 wait complete. 2024-04-14T15:24:24.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:24.363+0000 7f5bfa249700 1 librados: shutdown 2024-04-14T15:24:24.367 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:24:24.367 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:24:24.367 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:24:24.367 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:24:24.367 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:24:24.367 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:24:24.367 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:24:24.367 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:24:24.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:24 smithi092 ceph-mon[33966]: 5.8 scrub starts 2024-04-14T15:24:24.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:24 smithi092 ceph-mon[33966]: 5.8 scrub ok 2024-04-14T15:24:24.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:24 smithi092 ceph-mon[33966]: pgmap v1065: 214 pgs: 32 unknown, 182 active+clean; 582 KiB data, 679 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:24:24.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:24 smithi092 ceph-mon[33966]: 1.1 scrub starts 2024-04-14T15:24:24.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:24 smithi092 ceph-mon[33966]: 1.1 scrub ok 2024-04-14T15:24:24.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:24 smithi092 ceph-mon[33966]: 6.17 scrub starts 2024-04-14T15:24:24.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:24 smithi092 ceph-mon[33966]: 6.17 scrub ok 2024-04-14T15:24:24.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:24 smithi092 ceph-mon[33966]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:24:24.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:24 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-126", "mode": "writeback"}]': finished 2024-04-14T15:24:24.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:24 smithi092 ceph-mon[33966]: osdmap e722: 8 total, 8 up, 7 in 2024-04-14T15:24:24.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:24 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:24:24.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:24 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:24:24.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:24 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:24:24.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:24 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:24:24.446 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:24:24.469 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:24.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:24.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:24:24.571 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:24:24.594 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:24.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:24.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:24:24.695 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:24:24.719 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:24.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:24.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:24:24.821 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:24:24.844 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:24.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:24.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:24:24.946 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:24:24.970 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:24.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:24.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:24:25.072 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:24:25.095 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:25.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:25.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:24:25.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:25 smithi090 ceph-mon[41670]: 5.b scrub starts 2024-04-14T15:24:25.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:25 smithi090 ceph-mon[41670]: 5.b scrub ok 2024-04-14T15:24:25.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:25 smithi090 ceph-mon[41670]: 1.7 scrub starts 2024-04-14T15:24:25.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:25 smithi090 ceph-mon[41670]: 1.7 scrub ok 2024-04-14T15:24:25.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:25 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/708853304' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:24:25.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:25 smithi090 ceph-mon[41670]: 2.1f scrub starts 2024-04-14T15:24:25.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:25 smithi090 ceph-mon[41670]: 2.1f scrub ok 2024-04-14T15:24:25.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:25 smithi090 ceph-mon[41670]: 11.14 scrub starts 2024-04-14T15:24:25.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:25 smithi090 ceph-mon[41670]: 11.14 scrub ok 2024-04-14T15:24:25.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:25 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]': finished 2024-04-14T15:24:25.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:25 smithi090 ceph-mon[41670]: osdmap e723: 8 total, 8 up, 7 in 2024-04-14T15:24:25.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:25 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-126"}]: dispatch 2024-04-14T15:24:25.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:25 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-126"}]: dispatch 2024-04-14T15:24:25.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:25 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "331.1a", "id": [2, 1]}]: dispatch 2024-04-14T15:24:25.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:25 smithi090 ceph-mon[30399]: 5.b scrub starts 2024-04-14T15:24:25.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:25 smithi090 ceph-mon[30399]: 5.b scrub ok 2024-04-14T15:24:25.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:25 smithi090 ceph-mon[30399]: 1.7 scrub starts 2024-04-14T15:24:25.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:25 smithi090 ceph-mon[30399]: 1.7 scrub ok 2024-04-14T15:24:25.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:25 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/708853304' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:24:25.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:25 smithi090 ceph-mon[30399]: 2.1f scrub starts 2024-04-14T15:24:25.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:25 smithi090 ceph-mon[30399]: 2.1f scrub ok 2024-04-14T15:24:25.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:25 smithi090 ceph-mon[30399]: 11.14 scrub starts 2024-04-14T15:24:25.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:25 smithi090 ceph-mon[30399]: 11.14 scrub ok 2024-04-14T15:24:25.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:25 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]': finished 2024-04-14T15:24:25.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:25 smithi090 ceph-mon[30399]: osdmap e723: 8 total, 8 up, 7 in 2024-04-14T15:24:25.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:25 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-126"}]: dispatch 2024-04-14T15:24:25.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:25 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-126"}]: dispatch 2024-04-14T15:24:25.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:25 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "331.1a", "id": [2, 1]}]: dispatch 2024-04-14T15:24:25.197 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:24:25.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:25.221 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:25.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:24:25.322 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:24:25.347 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:25.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:25.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:24:25.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:25 smithi092 ceph-mon[33966]: 5.b scrub starts 2024-04-14T15:24:25.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:25 smithi092 ceph-mon[33966]: 5.b scrub ok 2024-04-14T15:24:25.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:25 smithi092 ceph-mon[33966]: 1.7 scrub starts 2024-04-14T15:24:25.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:25 smithi092 ceph-mon[33966]: 1.7 scrub ok 2024-04-14T15:24:25.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:25 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/708853304' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:24:25.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:25 smithi092 ceph-mon[33966]: 2.1f scrub starts 2024-04-14T15:24:25.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:25 smithi092 ceph-mon[33966]: 2.1f scrub ok 2024-04-14T15:24:25.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:25 smithi092 ceph-mon[33966]: 11.14 scrub starts 2024-04-14T15:24:25.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:25 smithi092 ceph-mon[33966]: 11.14 scrub ok 2024-04-14T15:24:25.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:25 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]': finished 2024-04-14T15:24:25.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:25 smithi092 ceph-mon[33966]: osdmap e723: 8 total, 8 up, 7 in 2024-04-14T15:24:25.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:25 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-126"}]: dispatch 2024-04-14T15:24:25.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:25 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-126"}]: dispatch 2024-04-14T15:24:25.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:25 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "331.1a", "id": [2, 1]}]: dispatch 2024-04-14T15:24:25.449 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:24:25.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:25.474 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:25.474 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:24:25.574 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:24:25.598 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:25.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:25.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:24:25.701 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:24:25.724 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:25.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:25.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:24:25.825 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:24:25.849 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:25.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:25.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:24:25.872 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:24:25 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:24:25.950 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:24:25.973 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:25.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:25.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:24:26.075 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:24:26.099 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:26.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:26.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:24:26.110 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:24:26.110 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:24:26.202 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:24:26.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:26.226 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:26.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:24:26.327 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:24:26.352 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:26.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:26.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:24:26.367 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:26 smithi090 ceph-mon[41670]: 4.12 scrub starts 2024-04-14T15:24:26.367 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:26 smithi090 ceph-mon[41670]: 4.12 scrub ok 2024-04-14T15:24:26.368 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:26 smithi090 ceph-mon[41670]: 4.10 scrub starts 2024-04-14T15:24:26.368 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:26 smithi090 ceph-mon[41670]: 4.10 scrub ok 2024-04-14T15:24:26.368 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:26 smithi090 ceph-mon[41670]: pgmap v1068: 214 pgs: 214 active+clean; 582 KiB data, 701 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-14T15:24:26.368 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:26 smithi090 ceph-mon[41670]: 6.0 scrub starts 2024-04-14T15:24:26.368 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:26 smithi090 ceph-mon[41670]: 6.0 scrub ok 2024-04-14T15:24:26.368 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:26 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:24:26.368 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:26 smithi090 ceph-mon[41670]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:24:26.368 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:26 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-126"}]': finished 2024-04-14T15:24:26.368 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:26 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "331.1a", "id": [2, 1]}]': finished 2024-04-14T15:24:26.368 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:26 smithi090 ceph-mon[41670]: osdmap e724: 8 total, 8 up, 7 in 2024-04-14T15:24:26.369 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:26 smithi090 ceph-mon[30399]: 4.12 scrub starts 2024-04-14T15:24:26.369 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:26 smithi090 ceph-mon[30399]: 4.12 scrub ok 2024-04-14T15:24:26.369 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:26 smithi090 ceph-mon[30399]: 4.10 scrub starts 2024-04-14T15:24:26.369 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:26 smithi090 ceph-mon[30399]: 4.10 scrub ok 2024-04-14T15:24:26.369 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:26 smithi090 ceph-mon[30399]: pgmap v1068: 214 pgs: 214 active+clean; 582 KiB data, 701 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-14T15:24:26.369 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:26 smithi090 ceph-mon[30399]: 6.0 scrub starts 2024-04-14T15:24:26.369 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:26 smithi090 ceph-mon[30399]: 6.0 scrub ok 2024-04-14T15:24:26.369 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:26 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:24:26.369 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:26 smithi090 ceph-mon[30399]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:24:26.369 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:26 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-126"}]': finished 2024-04-14T15:24:26.369 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:26 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "331.1a", "id": [2, 1]}]': finished 2024-04-14T15:24:26.369 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:26 smithi090 ceph-mon[30399]: osdmap e724: 8 total, 8 up, 7 in 2024-04-14T15:24:26.454 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:24:26.478 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:26.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:26.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:24:26.580 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:24:26.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:26.606 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:26.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:24:26.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:26 smithi092 ceph-mon[33966]: 4.12 scrub starts 2024-04-14T15:24:26.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:26 smithi092 ceph-mon[33966]: 4.12 scrub ok 2024-04-14T15:24:26.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:26 smithi092 ceph-mon[33966]: 4.10 scrub starts 2024-04-14T15:24:26.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:26 smithi092 ceph-mon[33966]: 4.10 scrub ok 2024-04-14T15:24:26.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:26 smithi092 ceph-mon[33966]: pgmap v1068: 214 pgs: 214 active+clean; 582 KiB data, 701 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-14T15:24:26.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:26 smithi092 ceph-mon[33966]: 6.0 scrub starts 2024-04-14T15:24:26.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:26 smithi092 ceph-mon[33966]: 6.0 scrub ok 2024-04-14T15:24:26.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:26 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:24:26.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:26 smithi092 ceph-mon[33966]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:24:26.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:26 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-126"}]': finished 2024-04-14T15:24:26.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:26 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "331.1a", "id": [2, 1]}]': finished 2024-04-14T15:24:26.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:26 smithi092 ceph-mon[33966]: osdmap e724: 8 total, 8 up, 7 in 2024-04-14T15:24:26.707 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:24:26.735 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:26.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:26.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:24:26.837 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:24:26.861 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:26.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:26.862 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:24:26.963 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:24:26.989 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:26.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:26.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:24:27.091 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:24:27.115 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:27.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:27.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:24:27.216 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:24:27.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:27.244 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:27.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:24:27.345 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:24:27.369 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:27.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:27.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:24:27.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:27 smithi090 ceph-mon[41670]: 5.5 scrub starts 2024-04-14T15:24:27.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:27 smithi090 ceph-mon[41670]: 5.5 scrub ok 2024-04-14T15:24:27.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:27 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2462277338' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:24:27.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:27 smithi090 ceph-mon[41670]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-14T15:24:27.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:27 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2462277338' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-14T15:24:27.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:27 smithi090 ceph-mon[41670]: osdmap e725: 8 total, 8 up, 7 in 2024-04-14T15:24:27.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:27 smithi090 ceph-mon[30399]: 5.5 scrub starts 2024-04-14T15:24:27.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:27 smithi090 ceph-mon[30399]: 5.5 scrub ok 2024-04-14T15:24:27.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:27 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2462277338' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:24:27.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:27 smithi090 ceph-mon[30399]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-14T15:24:27.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:27 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2462277338' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-14T15:24:27.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:27 smithi090 ceph-mon[30399]: osdmap e725: 8 total, 8 up, 7 in 2024-04-14T15:24:27.471 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:24:27.498 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:27.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:27.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:24:27.600 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:24:27.621 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-14T15:24:27.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:27 smithi092 ceph-mon[33966]: 5.5 scrub starts 2024-04-14T15:24:27.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:27 smithi092 ceph-mon[33966]: 5.5 scrub ok 2024-04-14T15:24:27.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:27 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2462277338' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:24:27.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:27 smithi092 ceph-mon[33966]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-14T15:24:27.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:27 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2462277338' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-14T15:24:27.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:27 smithi092 ceph-mon[33966]: osdmap e725: 8 total, 8 up, 7 in 2024-04-14T15:24:27.623 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:27.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:27.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:24:27.725 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:24:27.753 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:27.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:27.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:24:27.855 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:24:27.880 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:27.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:27.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:24:27.981 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:24:28.009 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:28.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:28.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:24:28.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:26.693+0000 7f71b5c3b700 1 -- 172.21.15.90:0/1782426337 >> v1:172.21.15.90:6789/0 conn(0x7f71b010b880 legacy=0x7f71b010dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:28.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:26.693+0000 7f71b5c3b700 1 -- 172.21.15.90:0/1782426337 shutdown_connections 2024-04-14T15:24:28.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:26.693+0000 7f71b5c3b700 1 -- 172.21.15.90:0/1782426337 >> 172.21.15.90:0/1782426337 conn(0x7f71b00fd2a0 msgr2=0x7f71b00ff6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:24:28.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:26.693+0000 7f71b5c3b700 1 -- 172.21.15.90:0/1782426337 shutdown_connections 2024-04-14T15:24:28.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:26.693+0000 7f71b5c3b700 1 -- 172.21.15.90:0/1782426337 wait complete. 2024-04-14T15:24:28.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:26.693+0000 7f71b5c3b700 1 librados: starting msgr at 2024-04-14T15:24:28.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:26.693+0000 7f71b5c3b700 1 librados: starting objecter 2024-04-14T15:24:28.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:26.693+0000 7f71b5c3b700 1 Processor -- start 2024-04-14T15:24:28.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:26.693+0000 7f71b5c3b700 1 -- start start 2024-04-14T15:24:28.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:26.693+0000 7f71b5c3b700 1 librados: setting wanted keys 2024-04-14T15:24:28.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:26.693+0000 7f71b5c3b700 1 librados: calling monclient init 2024-04-14T15:24:28.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:26.694+0000 7f71b5c3b700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f71b010b470 con 0x7f71b0103a70 2024-04-14T15:24:28.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:26.694+0000 7f71b5c3b700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f71b010b680 con 0x7f71b010f390 2024-04-14T15:24:28.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:26.694+0000 7f71b5c3b700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f71b01a9f60 con 0x7f71b010b880 2024-04-14T15:24:28.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:26.694+0000 7f71aeffd700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f71b010b880 0x7f71b01089d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:39208/0 (socket says 172.21.15.90:39208) 2024-04-14T15:24:28.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:26.694+0000 7f71aeffd700 1 -- 172.21.15.90:0/2462277338 learned_addr learned my addr 172.21.15.90:0/2462277338 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:24:28.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:26.694+0000 7f71af7fe700 1 --1- 172.21.15.90:0/2462277338 >> v1:172.21.15.90:6789/0 conn(0x7f71b0103a70 0x7f71b01082c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:39452/0 (socket says 172.21.15.90:39452) 2024-04-14T15:24:28.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:26.694+0000 7f71acff9700 10 client.?.objecter ms_handle_connect 0x7f71b0103a70 2024-04-14T15:24:28.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:26.694+0000 7f71acff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:24:28.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:26.694+0000 7f71acff9700 10 client.?.objecter ms_handle_connect 0x7f71b010b880 2024-04-14T15:24:28.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:26.694+0000 7f71acff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:24:28.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:26.694+0000 7f71acff9700 10 client.?.objecter ms_handle_connect 0x7f71b010f390 2024-04-14T15:24:28.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:26.694+0000 7f71acff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:24:28.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:26.694+0000 7f71acff9700 1 -- 172.21.15.90:0/2462277338 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2124099987 0 0) 0x7f71b010b470 con 0x7f71b0103a70 2024-04-14T15:24:28.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:26.694+0000 7f71acff9700 1 -- 172.21.15.90:0/2462277338 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71980035b0 con 0x7f71b0103a70 2024-04-14T15:24:28.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:26.694+0000 7f71acff9700 1 -- 172.21.15.90:0/2462277338 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2289984498 0 0) 0x7f71b01a9f60 con 0x7f71b010b880 2024-04-14T15:24:28.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:26.694+0000 7f71acff9700 1 -- 172.21.15.90:0/2462277338 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71b010b470 con 0x7f71b010b880 2024-04-14T15:24:28.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:26.694+0000 7f71acff9700 1 -- 172.21.15.90:0/2462277338 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 74415544 0 0) 0x7f71b010b680 con 0x7f71b010f390 2024-04-14T15:24:28.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:26.694+0000 7f71acff9700 1 -- 172.21.15.90:0/2462277338 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71b01a9f60 con 0x7f71b010f390 2024-04-14T15:24:28.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:26.694+0000 7f71acff9700 1 -- 172.21.15.90:0/2462277338 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3045765786 0 0) 0x7f71980035b0 con 0x7f71b0103a70 2024-04-14T15:24:28.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:26.694+0000 7f71acff9700 1 -- 172.21.15.90:0/2462277338 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f71b010b680 con 0x7f71b0103a70 2024-04-14T15:24:28.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:26.694+0000 7f71acff9700 1 -- 172.21.15.90:0/2462277338 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f71a00018b0 con 0x7f71b0103a70 2024-04-14T15:24:28.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:26.694+0000 7f71acff9700 1 -- 172.21.15.90:0/2462277338 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1589189290 0 0) 0x7f71b01a9f60 con 0x7f71b010f390 2024-04-14T15:24:28.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:26.694+0000 7f71acff9700 1 -- 172.21.15.90:0/2462277338 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f71980035b0 con 0x7f71b010f390 2024-04-14T15:24:28.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:26.694+0000 7f71acff9700 1 -- 172.21.15.90:0/2462277338 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f71a8001bd0 con 0x7f71b010f390 2024-04-14T15:24:28.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:26.694+0000 7f71acff9700 1 -- 172.21.15.90:0/2462277338 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1419301961 0 0) 0x7f71b010b470 con 0x7f71b010b880 2024-04-14T15:24:28.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:26.694+0000 7f71acff9700 1 -- 172.21.15.90:0/2462277338 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f71b01a9f60 con 0x7f71b010b880 2024-04-14T15:24:28.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:26.694+0000 7f71acff9700 1 -- 172.21.15.90:0/2462277338 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f71a4001430 con 0x7f71b010b880 2024-04-14T15:24:28.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:26.694+0000 7f71acff9700 1 -- 172.21.15.90:0/2462277338 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3001394946 0 0) 0x7f71b010b680 con 0x7f71b0103a70 2024-04-14T15:24:28.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:26.694+0000 7f71acff9700 1 -- 172.21.15.90:0/2462277338 >> v1:172.21.15.90:6790/0 conn(0x7f71b010b880 legacy=0x7f71b01089d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:28.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:26.694+0000 7f71acff9700 1 -- 172.21.15.90:0/2462277338 >> v1:172.21.15.92:6789/0 conn(0x7f71b010f390 legacy=0x7f71b01a67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:28.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:26.694+0000 7f71acff9700 1 -- 172.21.15.90:0/2462277338 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71b01ab1c0 con 0x7f71b0103a70 2024-04-14T15:24:28.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:26.695+0000 7f71b5c3b700 1 -- 172.21.15.90:0/2462277338 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f71b01aa190 con 0x7f71b0103a70 2024-04-14T15:24:28.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:26.695+0000 7f71b5c3b700 10 client.37905.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:24:28.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:26.695+0000 7f71b5c3b700 1 -- 172.21.15.90:0/2462277338 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f71b01aa650 con 0x7f71b0103a70 2024-04-14T15:24:28.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:26.695+0000 7f71b5c3b700 1 librados: init done 2024-04-14T15:24:28.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:26.695+0000 7f71acff9700 1 -- 172.21.15.90:0/2462277338 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f71a0003b40 con 0x7f71b0103a70 2024-04-14T15:24:28.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:26.695+0000 7f71acff9700 1 -- 172.21.15.90:0/2462277338 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f71a0005d10 con 0x7f71b0103a70 2024-04-14T15:24:28.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:26.696+0000 7f71b5c3b700 1 -- 172.21.15.90:0/2462277338 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f719c005120 con 0x7f71b0103a70 2024-04-14T15:24:28.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:26.696+0000 7f71acff9700 1 -- 172.21.15.90:0/2462277338 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f71a00036a0 con 0x7f71b0103a70 2024-04-14T15:24:28.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:26.697+0000 7f71acff9700 1 -- 172.21.15.90:0/2462277338 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(724..724 src has 467..724) ==== 8778+0+0 (unknown 1185239004 0 0) 0x7f71a008ab50 con 0x7f71b0103a70 2024-04-14T15:24:28.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:26.697+0000 7f71acff9700 10 client.37905.objecter ms_dispatch 0x7f71b0001550 osd_map(724..724 src has 467..724) 2024-04-14T15:24:28.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:26.697+0000 7f71acff9700 3 client.37905.objecter handle_osd_map got epochs [724,724] > 0 2024-04-14T15:24:28.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:26.697+0000 7f71acff9700 3 client.37905.objecter handle_osd_map decoding full epoch 724 2024-04-14T15:24:28.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:26.697+0000 7f71acff9700 20 client.37905.objecter dump_active .. 0 homeless 2024-04-14T15:24:28.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:26.697+0000 7f71acff9700 10 client.37905.objecter ms_handle_connect 0x7f7198071360 2024-04-14T15:24:28.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:26.701+0000 7f71acff9700 1 -- 172.21.15.90:0/2462277338 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f71a0057310 con 0x7f71b0103a70 2024-04-14T15:24:28.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:26.944+0000 7f71b5c3b700 1 -- 172.21.15.90:0/2462277338 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7f719c005410 con 0x7f71b0103a70 2024-04-14T15:24:28.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:26.999+0000 7f71acff9700 1 -- 172.21.15.90:0/2462277338 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v725) ==== 101+0+0 (unknown 2735919335 0 0) 0x7f71a005afc0 con 0x7f71b0103a70 2024-04-14T15:24:28.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:27.002+0000 7f71b5c3b700 10 librados: watch_flush enter 2024-04-14T15:24:28.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:27.002+0000 7f71b5c3b700 10 librados: watch_flush exit 2024-04-14T15:24:28.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:27.002+0000 7f71b5c3b700 20 client.37905.objecter shutdown clearing up homeless session... 2024-04-14T15:24:28.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:27.002+0000 7f71b5c3b700 10 client.37905.objecter successfully canceled tick 2024-04-14T15:24:28.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:27.003+0000 7f71b5c3b700 1 -- 172.21.15.90:0/2462277338 >> v1:172.21.15.90:6800/420935427 conn(0x7f7198071360 legacy=0x7f7198073820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:28.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:27.003+0000 7f71b5c3b700 1 -- 172.21.15.90:0/2462277338 >> v1:172.21.15.90:6789/0 conn(0x7f71b0103a70 legacy=0x7f71b01082c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:28.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:27.003+0000 7f71b5c3b700 1 -- 172.21.15.90:0/2462277338 shutdown_connections 2024-04-14T15:24:28.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:27.003+0000 7f71b5c3b700 1 -- 172.21.15.90:0/2462277338 >> 172.21.15.90:0/2462277338 conn(0x7f71b00fd2a0 msgr2=0x7f71b01117c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:24:28.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:27.004+0000 7f71b5c3b700 1 -- 172.21.15.90:0/2462277338 shutdown_connections 2024-04-14T15:24:28.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:27.004+0000 7f71b5c3b700 1 -- 172.21.15.90:0/2462277338 wait complete. 2024-04-14T15:24:28.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:27.005+0000 7f71b5c3b700 1 librados: shutdown 2024-04-14T15:24:28.038 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is unset 2024-04-14T15:24:28.111 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:24:28.138 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: require_osd_release = squid 2024-04-14T15:24:28.138 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringFlush (8993 ms) 2024-04-14T15:24:28.138 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapHasChunk 2024-04-14T15:24:28.139 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: require_osd_release = squid 2024-04-14T15:24:28.139 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapHasChunk (6038 ms) 2024-04-14T15:24:28.139 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollback 2024-04-14T15:24:28.139 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: require_osd_release = squid 2024-04-14T15:24:28.139 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollback (5023 ms) 2024-04-14T15:24:28.139 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollbackRefcount 2024-04-14T15:24:28.139 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: require_osd_release = squid 2024-04-14T15:24:28.139 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollbackRefcount (24941 ms) 2024-04-14T15:24:28.139 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictRollback 2024-04-14T15:24:28.139 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: require_osd_release = squid 2024-04-14T15:24:28.139 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictRollback (13998 ms) 2024-04-14T15:24:28.139 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PropagateBaseTierError 2024-04-14T15:24:28.139 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PropagateBaseTierError (12017 ms) 2024-04-14T15:24:28.139 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HelloWriteReturn 2024-04-14T15:24:28.139 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: 00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2024-04-14T15:24:28.139 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: 00000010 69 73 |is| 2024-04-14T15:24:28.140 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: 00000012 2024-04-14T15:24:28.140 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HelloWriteReturn (11996 ms) 2024-04-14T15:24:28.140 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier 2024-04-14T15:24:28.140 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: require_osd_release = squid 2024-04-14T15:24:28.140 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier (6003 ms) 2024-04-14T15:24:28.140 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP (553116 ms total) 2024-04-14T15:24:28.140 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: 2024-04-14T15:24:28.140 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP 2024-04-14T15:24:28.140 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.Dirty 2024-04-14T15:24:28.140 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.Dirty (1054 ms) 2024-04-14T15:24:28.140 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.FlushWriteRaces 2024-04-14T15:24:28.140 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.FlushWriteRaces (10952 ms) 2024-04-14T15:24:28.140 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.CallForcesPromote 2024-04-14T15:24:28.140 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.CallForcesPromote (18002 ms) 2024-04-14T15:24:28.140 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.HitSetNone 2024-04-14T15:24:28.141 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.HitSetNone (2 ms) 2024-04-14T15:24:28.141 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP (30010 ms total) 2024-04-14T15:24:28.141 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: 2024-04-14T15:24:28.141 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP 2024-04-14T15:24:28.141 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Overlay 2024-04-14T15:24:28.141 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Overlay (6994 ms) 2024-04-14T15:24:28.141 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Promote 2024-04-14T15:24:28.141 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Promote (8023 ms) 2024-04-14T15:24:28.141 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2024-04-14T15:24:28.141 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: waiting for scrub... 2024-04-14T15:24:28.141 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: done waiting 2024-04-14T15:24:28.141 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (25026 ms) 2024-04-14T15:24:28.141 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2024-04-14T15:24:28.142 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (9996 ms) 2024-04-14T15:24:28.142 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2024-04-14T15:24:28.142 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Whiteout (7965 ms) 2024-04-14T15:24:28.142 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Evict 2024-04-14T15:24:28.142 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Evict (8062 ms) 2024-04-14T15:24:28.142 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2024-04-14T15:24:28.142 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (9967 ms) 2024-04-14T15:24:28.142 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2024-04-14T15:24:28.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:28.143 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:28.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:24:28.244 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:24:28.271 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:28.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:28.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:24:28.373 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:24:28.379 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:28 smithi090 ceph-mon[41670]: 2.1c scrub starts 2024-04-14T15:24:28.379 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:28 smithi090 ceph-mon[41670]: 2.1c scrub ok 2024-04-14T15:24:28.380 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:28 smithi090 ceph-mon[41670]: 11.11 scrub starts 2024-04-14T15:24:28.380 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:28 smithi090 ceph-mon[41670]: 11.11 scrub ok 2024-04-14T15:24:28.380 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:28 smithi090 ceph-mon[41670]: 11.1a scrub starts 2024-04-14T15:24:28.380 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:28 smithi090 ceph-mon[41670]: 11.1a scrub ok 2024-04-14T15:24:28.380 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:28 smithi090 ceph-mon[41670]: pgmap v1071: 214 pgs: 214 active+clean; 582 KiB data, 701 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-14T15:24:28.380 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:28 smithi090 ceph-mon[41670]: 6.d scrub starts 2024-04-14T15:24:28.380 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:28 smithi090 ceph-mon[41670]: 6.d scrub ok 2024-04-14T15:24:28.380 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:28 smithi090 ceph-mon[41670]: 11.19 scrub starts 2024-04-14T15:24:28.380 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:28 smithi090 ceph-mon[41670]: 11.19 scrub ok 2024-04-14T15:24:28.380 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:28 smithi090 ceph-mon[41670]: 5.e scrub starts 2024-04-14T15:24:28.380 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:28 smithi090 ceph-mon[41670]: 5.e scrub ok 2024-04-14T15:24:28.380 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:28 smithi090 ceph-mon[30399]: 2.1c scrub starts 2024-04-14T15:24:28.381 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:28 smithi090 ceph-mon[30399]: 2.1c scrub ok 2024-04-14T15:24:28.381 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:28 smithi090 ceph-mon[30399]: 11.11 scrub starts 2024-04-14T15:24:28.381 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:28 smithi090 ceph-mon[30399]: 11.11 scrub ok 2024-04-14T15:24:28.381 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:28 smithi090 ceph-mon[30399]: 11.1a scrub starts 2024-04-14T15:24:28.381 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:28 smithi090 ceph-mon[30399]: 11.1a scrub ok 2024-04-14T15:24:28.381 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:28 smithi090 ceph-mon[30399]: pgmap v1071: 214 pgs: 214 active+clean; 582 KiB data, 701 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-14T15:24:28.381 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:28 smithi090 ceph-mon[30399]: 6.d scrub starts 2024-04-14T15:24:28.381 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:28 smithi090 ceph-mon[30399]: 6.d scrub ok 2024-04-14T15:24:28.381 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:28 smithi090 ceph-mon[30399]: 11.19 scrub starts 2024-04-14T15:24:28.381 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:28 smithi090 ceph-mon[30399]: 11.19 scrub ok 2024-04-14T15:24:28.381 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:28 smithi090 ceph-mon[30399]: 5.e scrub starts 2024-04-14T15:24:28.381 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:28 smithi090 ceph-mon[30399]: 5.e scrub ok 2024-04-14T15:24:28.400 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:28.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:28.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:24:28.502 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:24:28.529 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:28.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:28.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:24:28.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:28 smithi092 ceph-mon[33966]: 2.1c scrub starts 2024-04-14T15:24:28.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:28 smithi092 ceph-mon[33966]: 2.1c scrub ok 2024-04-14T15:24:28.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:28 smithi092 ceph-mon[33966]: 11.11 scrub starts 2024-04-14T15:24:28.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:28 smithi092 ceph-mon[33966]: 11.11 scrub ok 2024-04-14T15:24:28.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:28 smithi092 ceph-mon[33966]: 11.1a scrub starts 2024-04-14T15:24:28.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:28 smithi092 ceph-mon[33966]: 11.1a scrub ok 2024-04-14T15:24:28.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:28 smithi092 ceph-mon[33966]: pgmap v1071: 214 pgs: 214 active+clean; 582 KiB data, 701 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-14T15:24:28.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:28 smithi092 ceph-mon[33966]: 6.d scrub starts 2024-04-14T15:24:28.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:28 smithi092 ceph-mon[33966]: 6.d scrub ok 2024-04-14T15:24:28.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:28 smithi092 ceph-mon[33966]: 11.19 scrub starts 2024-04-14T15:24:28.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:28 smithi092 ceph-mon[33966]: 11.19 scrub ok 2024-04-14T15:24:28.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:28 smithi092 ceph-mon[33966]: 5.e scrub starts 2024-04-14T15:24:28.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:28 smithi092 ceph-mon[33966]: 5.e scrub ok 2024-04-14T15:24:28.631 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:24:28.658 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:28.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:28.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:24:28.760 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:24:28.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:28.788 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:28.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:24:28.888 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:24:28.916 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:28.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:28.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:24:29.018 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:24:29.042 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:29.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:29.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:24:29.144 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:24:29.171 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:29.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:29.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:24:29.274 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:24:29.301 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:29.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:29.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:24:29.388 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:29 smithi090 ceph-mon[30399]: 11.f scrub starts 2024-04-14T15:24:29.389 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:29 smithi090 ceph-mon[30399]: 5.a scrub starts 2024-04-14T15:24:29.389 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:29 smithi090 ceph-mon[30399]: 11.f scrub ok 2024-04-14T15:24:29.389 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:29 smithi090 ceph-mon[30399]: 5.a scrub ok 2024-04-14T15:24:29.389 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:29 smithi090 ceph-mon[30399]: 6.1b deep-scrub starts 2024-04-14T15:24:29.389 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:29 smithi090 ceph-mon[30399]: 6.1b deep-scrub ok 2024-04-14T15:24:29.389 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:29 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:24:29.389 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:29 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:24:29.389 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:29 smithi090 ceph-mon[30399]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:24:29.389 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:29 smithi090 ceph-mon[30399]: osdmap e726: 8 total, 8 up, 7 in 2024-04-14T15:24:29.389 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:29 smithi090 ceph-mon[30399]: 4.9 scrub starts 2024-04-14T15:24:29.389 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:29 smithi090 ceph-mon[30399]: 4.9 scrub ok 2024-04-14T15:24:29.389 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:29 smithi090 ceph-mon[41670]: 11.f scrub starts 2024-04-14T15:24:29.389 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:29 smithi090 ceph-mon[41670]: 5.a scrub starts 2024-04-14T15:24:29.390 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:29 smithi090 ceph-mon[41670]: 11.f scrub ok 2024-04-14T15:24:29.390 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:29 smithi090 ceph-mon[41670]: 5.a scrub ok 2024-04-14T15:24:29.390 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:29 smithi090 ceph-mon[41670]: 6.1b deep-scrub starts 2024-04-14T15:24:29.390 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:29 smithi090 ceph-mon[41670]: 6.1b deep-scrub ok 2024-04-14T15:24:29.390 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:29 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:24:29.390 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:29 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:24:29.390 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:29 smithi090 ceph-mon[41670]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:24:29.390 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:29 smithi090 ceph-mon[41670]: osdmap e726: 8 total, 8 up, 7 in 2024-04-14T15:24:29.390 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:29 smithi090 ceph-mon[41670]: 4.9 scrub starts 2024-04-14T15:24:29.390 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:29 smithi090 ceph-mon[41670]: 4.9 scrub ok 2024-04-14T15:24:29.403 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:24:29.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:29.427 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:29.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:24:29.528 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:24:29.551 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:29.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:29.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:24:29.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:29 smithi092 ceph-mon[33966]: 11.f scrub starts 2024-04-14T15:24:29.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:29 smithi092 ceph-mon[33966]: 5.a scrub starts 2024-04-14T15:24:29.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:29 smithi092 ceph-mon[33966]: 11.f scrub ok 2024-04-14T15:24:29.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:29 smithi092 ceph-mon[33966]: 5.a scrub ok 2024-04-14T15:24:29.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:29 smithi092 ceph-mon[33966]: 6.1b deep-scrub starts 2024-04-14T15:24:29.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:29 smithi092 ceph-mon[33966]: 6.1b deep-scrub ok 2024-04-14T15:24:29.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:29 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:24:29.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:29 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:24:29.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:29 smithi092 ceph-mon[33966]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:24:29.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:29 smithi092 ceph-mon[33966]: osdmap e726: 8 total, 8 up, 7 in 2024-04-14T15:24:29.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:29 smithi092 ceph-mon[33966]: 4.9 scrub starts 2024-04-14T15:24:29.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:29 smithi092 ceph-mon[33966]: 4.9 scrub ok 2024-04-14T15:24:29.653 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:24:29.677 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:29.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:29.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:24:29.779 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:24:29.802 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:29.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:29.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:24:29.903 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:24:29.931 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:29.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:29.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:24:30.032 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:24:30.056 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:30.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:30.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:24:30.158 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:24:30.172 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:24:30.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:30.190 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:30.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:24:30.213 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-14T15:24:30.213 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-14T15:24:30.214 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-14T15:24:30.214 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-14T15:24:30.214 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-14T15:24:30.291 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:24:30.315 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:30.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:30.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:24:30.417 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:24:30.427 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:30 smithi090 ceph-mon[30399]: 11.a scrub starts 2024-04-14T15:24:30.427 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:30 smithi090 ceph-mon[30399]: 11.a scrub ok 2024-04-14T15:24:30.427 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:30 smithi090 ceph-mon[30399]: osdmap e727: 8 total, 8 up, 7 in 2024-04-14T15:24:30.427 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:30 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:24:30.428 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:30 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:24:30.428 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:30 smithi090 ceph-mon[30399]: pgmap v1074: 214 pgs: 20 creating+peering, 12 unknown, 182 active+clean; 582 KiB data, 702 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-04-14T15:24:30.428 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:30 smithi090 ceph-mon[30399]: 11.10 deep-scrub starts 2024-04-14T15:24:30.428 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:30 smithi090 ceph-mon[30399]: 11.10 deep-scrub ok 2024-04-14T15:24:30.428 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:30 smithi090 ceph-mon[41670]: 11.a scrub starts 2024-04-14T15:24:30.428 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:30 smithi090 ceph-mon[41670]: 11.a scrub ok 2024-04-14T15:24:30.428 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:30 smithi090 ceph-mon[41670]: osdmap e727: 8 total, 8 up, 7 in 2024-04-14T15:24:30.428 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:30 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:24:30.428 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:30 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:24:30.428 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:30 smithi090 ceph-mon[41670]: pgmap v1074: 214 pgs: 20 creating+peering, 12 unknown, 182 active+clean; 582 KiB data, 702 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-04-14T15:24:30.428 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:30 smithi090 ceph-mon[41670]: 11.10 deep-scrub starts 2024-04-14T15:24:30.428 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:30 smithi090 ceph-mon[41670]: 11.10 deep-scrub ok 2024-04-14T15:24:30.443 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:30.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:30.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:24:30.545 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:24:30.568 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:30.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:30.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:24:30.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:30 smithi092 ceph-mon[33966]: 11.a scrub starts 2024-04-14T15:24:30.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:30 smithi092 ceph-mon[33966]: 11.a scrub ok 2024-04-14T15:24:30.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:30 smithi092 ceph-mon[33966]: osdmap e727: 8 total, 8 up, 7 in 2024-04-14T15:24:30.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:30 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:24:30.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:30 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:24:30.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:30 smithi092 ceph-mon[33966]: pgmap v1074: 214 pgs: 20 creating+peering, 12 unknown, 182 active+clean; 582 KiB data, 702 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-04-14T15:24:30.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:30 smithi092 ceph-mon[33966]: 11.10 deep-scrub starts 2024-04-14T15:24:30.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:30 smithi092 ceph-mon[33966]: 11.10 deep-scrub ok 2024-04-14T15:24:30.670 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:24:30.694 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:30.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:30.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:24:30.796 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:24:30.810 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd set noscrub 2024-04-14T15:24:30.823 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:30.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:30.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:24:30.924 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:24:30.951 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:30.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:30.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:24:31.052 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:24:31.076 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:31.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:31.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:24:31.178 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:24:31.202 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:31.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:31.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:24:31.303 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:24:31.329 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:31.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:31.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:24:31.400 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-14T15:24:31.424 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:31 smithi090 ceph-mon[41670]: 2.1 scrub starts 2024-04-14T15:24:31.424 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:31 smithi090 ceph-mon[41670]: 2.1 scrub ok 2024-04-14T15:24:31.424 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:31 smithi090 ceph-mon[41670]: 2.0 deep-scrub starts 2024-04-14T15:24:31.424 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:31 smithi090 ceph-mon[41670]: 2.0 deep-scrub ok 2024-04-14T15:24:31.424 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:31 smithi090 ceph-mon[41670]: 4.11 scrub starts 2024-04-14T15:24:31.424 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:31 smithi090 ceph-mon[41670]: 4.11 scrub ok 2024-04-14T15:24:31.424 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:31 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:24:31.424 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:31 smithi090 ceph-mon[41670]: osdmap e728: 8 total, 8 up, 7 in 2024-04-14T15:24:31.424 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:31 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:24:31.424 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:31 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:24:31.424 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:31 smithi090 ceph-mon[41670]: 6.1d deep-scrub starts 2024-04-14T15:24:31.425 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:31 smithi090 ceph-mon[41670]: 6.1d deep-scrub ok 2024-04-14T15:24:31.425 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:31 smithi090 ceph-mon[41670]: 5.1e scrub starts 2024-04-14T15:24:31.425 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:31 smithi090 ceph-mon[41670]: 5.1e scrub ok 2024-04-14T15:24:31.425 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:31 smithi090 ceph-mon[41670]: 5.11 scrub starts 2024-04-14T15:24:31.425 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:31 smithi090 ceph-mon[41670]: 5.11 scrub ok 2024-04-14T15:24:31.425 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:31 smithi090 ceph-mon[30399]: 2.1 scrub starts 2024-04-14T15:24:31.425 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:31 smithi090 ceph-mon[30399]: 2.1 scrub ok 2024-04-14T15:24:31.425 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:31 smithi090 ceph-mon[30399]: 2.0 deep-scrub starts 2024-04-14T15:24:31.425 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:31 smithi090 ceph-mon[30399]: 2.0 deep-scrub ok 2024-04-14T15:24:31.425 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:31 smithi090 ceph-mon[30399]: 4.11 scrub starts 2024-04-14T15:24:31.425 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:31 smithi090 ceph-mon[30399]: 4.11 scrub ok 2024-04-14T15:24:31.425 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:31 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:24:31.426 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:31 smithi090 ceph-mon[30399]: osdmap e728: 8 total, 8 up, 7 in 2024-04-14T15:24:31.426 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:31 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:24:31.426 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:31 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:24:31.426 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:31 smithi090 ceph-mon[30399]: 6.1d deep-scrub starts 2024-04-14T15:24:31.426 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:31 smithi090 ceph-mon[30399]: 6.1d deep-scrub ok 2024-04-14T15:24:31.426 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:31 smithi090 ceph-mon[30399]: 5.1e scrub starts 2024-04-14T15:24:31.426 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:31 smithi090 ceph-mon[30399]: 5.1e scrub ok 2024-04-14T15:24:31.426 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:31 smithi090 ceph-mon[30399]: 5.11 scrub starts 2024-04-14T15:24:31.426 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:31 smithi090 ceph-mon[30399]: 5.11 scrub ok 2024-04-14T15:24:31.431 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:24:31.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:31.458 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:31.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:24:31.559 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:24:31.583 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:31.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:31.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:24:31.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.601+0000 7f09fae9e700 1 -- 172.21.15.90:0/2214571360 >> v1:172.21.15.90:6789/0 conn(0x7f09f4107890 legacy=0x7f09f4107c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:31.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.601+0000 7f09fae9e700 1 -- 172.21.15.90:0/2214571360 shutdown_connections 2024-04-14T15:24:31.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.601+0000 7f09fae9e700 1 -- 172.21.15.90:0/2214571360 >> 172.21.15.90:0/2214571360 conn(0x7f09f40fd290 msgr2=0x7f09f40ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:24:31.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.601+0000 7f09fae9e700 1 -- 172.21.15.90:0/2214571360 shutdown_connections 2024-04-14T15:24:31.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.601+0000 7f09fae9e700 1 -- 172.21.15.90:0/2214571360 wait complete. 2024-04-14T15:24:31.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.601+0000 7f09fae9e700 1 librados: starting msgr at 2024-04-14T15:24:31.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.601+0000 7f09fae9e700 1 librados: starting objecter 2024-04-14T15:24:31.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.602+0000 7f09fae9e700 1 Processor -- start 2024-04-14T15:24:31.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.602+0000 7f09fae9e700 1 -- start start 2024-04-14T15:24:31.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.602+0000 7f09fae9e700 1 librados: setting wanted keys 2024-04-14T15:24:31.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.602+0000 7f09fae9e700 1 librados: calling monclient init 2024-04-14T15:24:31.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.602+0000 7f09fae9e700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f09f4110390 con 0x7f09f410e310 2024-04-14T15:24:31.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.602+0000 7f09fae9e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f09f41105a0 con 0x7f09f4107890 2024-04-14T15:24:31.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.602+0000 7f09fae9e700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f09f41107b0 con 0x7f09f410a740 2024-04-14T15:24:31.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.602+0000 7f09f3fff700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f09f410a740 0x7f09f410efa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:58232/0 (socket says 172.21.15.90:58232) 2024-04-14T15:24:31.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.602+0000 7f09f3fff700 1 -- 172.21.15.90:0/3666733127 learned_addr learned my addr 172.21.15.90:0/3666733127 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:24:31.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.602+0000 7f09f1ffb700 10 client.?.objecter ms_handle_connect 0x7f09f410e310 2024-04-14T15:24:31.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.602+0000 7f09f1ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:24:31.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.602+0000 7f09f1ffb700 10 client.?.objecter ms_handle_connect 0x7f09f4107890 2024-04-14T15:24:31.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.602+0000 7f09f1ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:24:31.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.602+0000 7f09f1ffb700 10 client.?.objecter ms_handle_connect 0x7f09f410a740 2024-04-14T15:24:31.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.602+0000 7f09f1ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:24:31.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.602+0000 7f09f1ffb700 1 -- 172.21.15.90:0/3666733127 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1383724274 0 0) 0x7f09f41105a0 con 0x7f09f4107890 2024-04-14T15:24:31.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.602+0000 7f09f1ffb700 1 -- 172.21.15.90:0/3666733127 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f09e40035b0 con 0x7f09f4107890 2024-04-14T15:24:31.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.602+0000 7f09f1ffb700 1 -- 172.21.15.90:0/3666733127 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1250499189 0 0) 0x7f09f4110390 con 0x7f09f410e310 2024-04-14T15:24:31.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.602+0000 7f09f1ffb700 1 -- 172.21.15.90:0/3666733127 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f09f41105a0 con 0x7f09f410e310 2024-04-14T15:24:31.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.602+0000 7f09f1ffb700 1 -- 172.21.15.90:0/3666733127 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3982677654 0 0) 0x7f09f41107b0 con 0x7f09f410a740 2024-04-14T15:24:31.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.602+0000 7f09f1ffb700 1 -- 172.21.15.90:0/3666733127 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f09f4110390 con 0x7f09f410a740 2024-04-14T15:24:31.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.603+0000 7f09f1ffb700 1 -- 172.21.15.90:0/3666733127 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1769408686 0 0) 0x7f09e40035b0 con 0x7f09f4107890 2024-04-14T15:24:31.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.603+0000 7f09f1ffb700 1 -- 172.21.15.90:0/3666733127 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f09f41107b0 con 0x7f09f4107890 2024-04-14T15:24:31.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.603+0000 7f09f1ffb700 1 -- 172.21.15.90:0/3666733127 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f09e8004640 con 0x7f09f4107890 2024-04-14T15:24:31.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.603+0000 7f09f1ffb700 1 -- 172.21.15.90:0/3666733127 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2739161319 0 0) 0x7f09f41105a0 con 0x7f09f410e310 2024-04-14T15:24:31.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.603+0000 7f09f1ffb700 1 -- 172.21.15.90:0/3666733127 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f09e40035b0 con 0x7f09f410e310 2024-04-14T15:24:31.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.603+0000 7f09f1ffb700 1 -- 172.21.15.90:0/3666733127 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f09ec001bb0 con 0x7f09f410e310 2024-04-14T15:24:31.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.603+0000 7f09f1ffb700 1 -- 172.21.15.90:0/3666733127 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2426700767 0 0) 0x7f09f4110390 con 0x7f09f410a740 2024-04-14T15:24:31.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.603+0000 7f09f1ffb700 1 -- 172.21.15.90:0/3666733127 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f09f41105a0 con 0x7f09f410a740 2024-04-14T15:24:31.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.603+0000 7f09f1ffb700 1 -- 172.21.15.90:0/3666733127 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f09e00018b0 con 0x7f09f410a740 2024-04-14T15:24:31.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.603+0000 7f09f1ffb700 1 -- 172.21.15.90:0/3666733127 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1886523198 0 0) 0x7f09f41107b0 con 0x7f09f4107890 2024-04-14T15:24:31.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.603+0000 7f09f1ffb700 1 -- 172.21.15.90:0/3666733127 >> v1:172.21.15.90:6790/0 conn(0x7f09f410a740 legacy=0x7f09f410efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:31.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.603+0000 7f09f1ffb700 1 -- 172.21.15.90:0/3666733127 >> v1:172.21.15.90:6789/0 conn(0x7f09f410e310 legacy=0x7f09f41a8bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:31.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.603+0000 7f09f1ffb700 1 -- 172.21.15.90:0/3666733127 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09f41ad380 con 0x7f09f4107890 2024-04-14T15:24:31.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.603+0000 7f09fae9e700 1 -- 172.21.15.90:0/3666733127 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f09f41aa370 con 0x7f09f4107890 2024-04-14T15:24:31.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.603+0000 7f09f1ffb700 1 -- 172.21.15.90:0/3666733127 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f09e8001b70 con 0x7f09f4107890 2024-04-14T15:24:31.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.603+0000 7f09f1ffb700 1 -- 172.21.15.90:0/3666733127 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f09e8004fe0 con 0x7f09f4107890 2024-04-14T15:24:31.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.603+0000 7f09fae9e700 10 client.49741.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:24:31.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.603+0000 7f09fae9e700 1 -- 172.21.15.90:0/3666733127 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f09f41aa830 con 0x7f09f4107890 2024-04-14T15:24:31.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.603+0000 7f09fae9e700 1 librados: init done 2024-04-14T15:24:31.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.604+0000 7f09fae9e700 1 -- 172.21.15.90:0/3666733127 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09f40693d0 con 0x7f09f4107890 2024-04-14T15:24:31.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.604+0000 7f09f1ffb700 1 -- 172.21.15.90:0/3666733127 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f09e8003740 con 0x7f09f4107890 2024-04-14T15:24:31.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.605+0000 7f09f1ffb700 1 -- 172.21.15.90:0/3666733127 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(729..729 src has 467..729) ==== 8757+0+0 (unknown 4160959315 0 0) 0x7f09e8089ba0 con 0x7f09f4107890 2024-04-14T15:24:31.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.605+0000 7f09f1ffb700 10 client.49741.objecter ms_dispatch 0x7f09f4001550 osd_map(729..729 src has 467..729) 2024-04-14T15:24:31.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.605+0000 7f09f1ffb700 3 client.49741.objecter handle_osd_map got epochs [729,729] > 0 2024-04-14T15:24:31.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.605+0000 7f09f1ffb700 3 client.49741.objecter handle_osd_map decoding full epoch 729 2024-04-14T15:24:31.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.605+0000 7f09f1ffb700 20 client.49741.objecter dump_active .. 0 homeless 2024-04-14T15:24:31.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.605+0000 7f09f1ffb700 10 client.49741.objecter ms_handle_connect 0x7f09e40714f0 2024-04-14T15:24:31.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.610+0000 7f09f1ffb700 1 -- 172.21.15.90:0/3666733127 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f09e8056380 con 0x7f09f4107890 2024-04-14T15:24:31.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:31 smithi092 ceph-mon[33966]: 2.1 scrub starts 2024-04-14T15:24:31.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:31 smithi092 ceph-mon[33966]: 2.1 scrub ok 2024-04-14T15:24:31.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:31 smithi092 ceph-mon[33966]: 2.0 deep-scrub starts 2024-04-14T15:24:31.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:31 smithi092 ceph-mon[33966]: 2.0 deep-scrub ok 2024-04-14T15:24:31.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:31 smithi092 ceph-mon[33966]: 4.11 scrub starts 2024-04-14T15:24:31.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:31 smithi092 ceph-mon[33966]: 4.11 scrub ok 2024-04-14T15:24:31.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:31 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:24:31.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:31 smithi092 ceph-mon[33966]: osdmap e728: 8 total, 8 up, 7 in 2024-04-14T15:24:31.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:31 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:24:31.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:31 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:24:31.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:31 smithi092 ceph-mon[33966]: 6.1d deep-scrub starts 2024-04-14T15:24:31.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:31 smithi092 ceph-mon[33966]: 6.1d deep-scrub ok 2024-04-14T15:24:31.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:31 smithi092 ceph-mon[33966]: 5.1e scrub starts 2024-04-14T15:24:31.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:31 smithi092 ceph-mon[33966]: 5.1e scrub ok 2024-04-14T15:24:31.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:31 smithi092 ceph-mon[33966]: 5.11 scrub starts 2024-04-14T15:24:31.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:31 smithi092 ceph-mon[33966]: 5.11 scrub ok 2024-04-14T15:24:31.685 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:24:31.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:31.709 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:31.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:24:31.810 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:24:31.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.822+0000 7f09fae9e700 1 -- 172.21.15.90:0/3666733127 --> v1:172.21.15.90:6800/420935427 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f09f4114130 con 0x7f09e40714f0 2024-04-14T15:24:31.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.836+0000 7f09f1ffb700 1 -- 172.21.15.90:0/3666733127 <== mgr.14610 v1:172.21.15.90:6800/420935427 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+548663 (unknown 2965378022 0 564686142) 0x7f09f4114130 con 0x7f09e40714f0 2024-04-14T15:24:31.837 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:31.843 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:24:31.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.841+0000 7f09fae9e700 10 librados: watch_flush enter 2024-04-14T15:24:31.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.841+0000 7f09fae9e700 10 librados: watch_flush exit 2024-04-14T15:24:31.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.841+0000 7f09fae9e700 20 client.49741.objecter shutdown clearing up homeless session... 2024-04-14T15:24:31.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.841+0000 7f09fae9e700 10 client.49741.objecter successfully canceled tick 2024-04-14T15:24:31.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.841+0000 7f09fae9e700 1 -- 172.21.15.90:0/3666733127 >> v1:172.21.15.90:6800/420935427 conn(0x7f09e40714f0 legacy=0x7f09e40739b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:31.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.841+0000 7f09fae9e700 1 -- 172.21.15.90:0/3666733127 >> v1:172.21.15.92:6789/0 conn(0x7f09f4107890 legacy=0x7f09f410e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:31.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.841+0000 7f09fae9e700 1 -- 172.21.15.90:0/3666733127 shutdown_connections 2024-04-14T15:24:31.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.841+0000 7f09fae9e700 1 -- 172.21.15.90:0/3666733127 >> 172.21.15.90:0/3666733127 conn(0x7f09f40fd290 msgr2=0x7f09f4111830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:24:31.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.841+0000 7f09fae9e700 1 -- 172.21.15.90:0/3666733127 shutdown_connections 2024-04-14T15:24:31.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.841+0000 7f09fae9e700 1 -- 172.21.15.90:0/3666733127 wait complete. 2024-04-14T15:24:31.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:31.842+0000 7f09fae9e700 1 librados: shutdown 2024-04-14T15:24:31.851 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-14T15:24:31.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:31.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:24:31.952 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:24:31.976 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:31.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:31.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:24:32.078 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:24:32.101 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:32.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:32.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:24:32.203 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:24:32.231 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:32.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:32.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:24:32.333 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:24:32.358 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:32.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:32.359 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:24:32.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:32 smithi090 ceph-mon[41670]: 2.17 deep-scrub starts 2024-04-14T15:24:32.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:32 smithi090 ceph-mon[41670]: 2.17 deep-scrub ok 2024-04-14T15:24:32.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:32 smithi090 ceph-mon[41670]: 6.6 deep-scrub starts 2024-04-14T15:24:32.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:32 smithi090 ceph-mon[41670]: 6.6 deep-scrub ok 2024-04-14T15:24:32.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:32 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-128", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:24:32.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:32 smithi090 ceph-mon[41670]: osdmap e729: 8 total, 8 up, 7 in 2024-04-14T15:24:32.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:32 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-128"}]: dispatch 2024-04-14T15:24:32.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:32 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-128"}]: dispatch 2024-04-14T15:24:32.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:32 smithi090 ceph-mon[41670]: pgmap v1077: 214 pgs: 20 creating+peering, 12 unknown, 182 active+clean; 582 KiB data, 702 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-04-14T15:24:32.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:32 smithi090 ceph-mon[41670]: 2.18 scrub starts 2024-04-14T15:24:32.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:32 smithi090 ceph-mon[41670]: 2.18 scrub ok 2024-04-14T15:24:32.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:32 smithi090 ceph-mon[41670]: from='client.49741 v1:172.21.15.90:0/3666733127' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:24:32.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:32 smithi090 ceph-mon[30399]: 2.17 deep-scrub starts 2024-04-14T15:24:32.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:32 smithi090 ceph-mon[30399]: 2.17 deep-scrub ok 2024-04-14T15:24:32.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:32 smithi090 ceph-mon[30399]: 6.6 deep-scrub starts 2024-04-14T15:24:32.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:32 smithi090 ceph-mon[30399]: 6.6 deep-scrub ok 2024-04-14T15:24:32.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:32 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-128", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:24:32.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:32 smithi090 ceph-mon[30399]: osdmap e729: 8 total, 8 up, 7 in 2024-04-14T15:24:32.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:32 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-128"}]: dispatch 2024-04-14T15:24:32.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:32 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-128"}]: dispatch 2024-04-14T15:24:32.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:32 smithi090 ceph-mon[30399]: pgmap v1077: 214 pgs: 20 creating+peering, 12 unknown, 182 active+clean; 582 KiB data, 702 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-04-14T15:24:32.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:32 smithi090 ceph-mon[30399]: 2.18 scrub starts 2024-04-14T15:24:32.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:32 smithi090 ceph-mon[30399]: 2.18 scrub ok 2024-04-14T15:24:32.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:32 smithi090 ceph-mon[30399]: from='client.49741 v1:172.21.15.90:0/3666733127' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:24:32.460 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:24:32.487 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:32.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:32.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:24:32.589 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:24:32.615 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:32.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:32.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:24:32.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:32 smithi092 ceph-mon[33966]: 2.17 deep-scrub starts 2024-04-14T15:24:32.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:32 smithi092 ceph-mon[33966]: 2.17 deep-scrub ok 2024-04-14T15:24:32.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:32 smithi092 ceph-mon[33966]: 6.6 deep-scrub starts 2024-04-14T15:24:32.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:32 smithi092 ceph-mon[33966]: 6.6 deep-scrub ok 2024-04-14T15:24:32.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:32 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-128", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:24:32.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:32 smithi092 ceph-mon[33966]: osdmap e729: 8 total, 8 up, 7 in 2024-04-14T15:24:32.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:32 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-128"}]: dispatch 2024-04-14T15:24:32.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:32 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-128"}]: dispatch 2024-04-14T15:24:32.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:32 smithi092 ceph-mon[33966]: pgmap v1077: 214 pgs: 20 creating+peering, 12 unknown, 182 active+clean; 582 KiB data, 702 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-04-14T15:24:32.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:32 smithi092 ceph-mon[33966]: 2.18 scrub starts 2024-04-14T15:24:32.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:32 smithi092 ceph-mon[33966]: 2.18 scrub ok 2024-04-14T15:24:32.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:32 smithi092 ceph-mon[33966]: from='client.49741 v1:172.21.15.90:0/3666733127' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:24:32.717 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:24:32.743 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:32.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:32.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:24:32.845 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:24:32.868 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:32.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:32.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:24:32.956 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":1077,"stamp":"2024-04-14T15:24:31.175219+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596468,"num_objects":212,"num_object_clones":0,"num_object_copies":636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":212,"num_whiteouts":0,"num_read":2040,"num_read_kb":1781,"num_write":755,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":126,"num_bytes_recovered":921158,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":942,"ondisk_log_size":942,"up":606,"acting":606,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":643,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":718428,"kb_used_data":37868,"kb_used_omap":10,"kb_used_meta":680437,"kb_avail":655497636,"statfs":{"total":671965249536,"available":671229579264,"internally_reserved":0,"allocated":38776832,"data_stored":19460367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11156,"internal_metadata":696767596},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":5,"num_write":-4,"num_write_kb":-1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":-1,"num_flush_kb":-1,"num_evict":0,"num_evict_kb":0,"num_promote":-1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001108"},"pg_stats":[{"pgid":"332.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:24:29.131816+0000","last_change":"2024-04-14T15:24:29.131816+0000","last_active":"2024-04-14T15:24:29.131816+0000","last_peered":"2024-04-14T15:24:29.131816+0000","last_clean":"2024-04-14T15:24:29.131816+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:24:29.131816+0000","last_undegraded":"2024-04-14T15:24:29.131816+0000","last_fullsized":"2024-04-14T15:24:29.131816+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:29.131816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:29.131816+0000","last_clean_scrub_stamp":"2024-04-14T15:24:29.131816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"332.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:24:29.131816+0000","last_change":"2024-04-14T15:24:29.131816+0000","last_active":"2024-04-14T15:24:29.131816+0000","last_peered":"2024-04-14T15:24:29.131816+0000","last_clean":"2024-04-14T15:24:29.131816+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:24:29.131816+0000","last_undegraded":"2024-04-14T15:24:29.131816+0000","last_fullsized":"2024-04-14T15:24:29.131816+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:29.131816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:29.131816+0000","last_clean_scrub_stamp":"2024-04-14T15:24:29.131816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"332.1d","version":"0'0","reported_seq":1,"reported_epoch":727,"state":"creating+peering","last_fresh":"2024-04-14T15:24:29.142975+0000","last_change":"2024-04-14T15:24:29.142975+0000","last_active":"2024-04-14T15:24:29.131816+0000","last_peered":"2024-04-14T15:24:29.131816+0000","last_clean":"2024-04-14T15:24:29.131816+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:24:29.142975+0000","last_undegraded":"2024-04-14T15:24:29.142975+0000","last_fullsized":"2024-04-14T15:24:29.142975+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":727,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:29.131816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:29.131816+0000","last_clean_scrub_stamp":"2024-04-14T15:24:29.131816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"332.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:24:29.131816+0000","last_change":"2024-04-14T15:24:29.131816+0000","last_active":"2024-04-14T15:24:29.131816+0000","last_peered":"2024-04-14T15:24:29.131816+0000","last_clean":"2024-04-14T15:24:29.131816+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:24:29.131816+0000","last_undegraded":"2024-04-14T15:24:29.131816+0000","last_fullsized":"2024-04-14T15:24:29.131816+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:29.131816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:29.131816+0000","last_clean_scrub_stamp":"2024-04-14T15:24:29.131816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"332.1b","version":"0'0","reported_seq":1,"reported_epoch":727,"state":"creating+peering","last_fresh":"2024-04-14T15:24:29.144856+0000","last_change":"2024-04-14T15:24:29.144856+0000","last_active":"2024-04-14T15:24:29.131816+0000","last_peered":"2024-04-14T15:24:29.131816+0000","last_clean":"2024-04-14T15:24:29.131816+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:24:29.144856+0000","last_undegraded":"2024-04-14T15:24:29.144856+0000","last_fullsized":"2024-04-14T15:24:29.144856+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":727,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:29.131816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:29.131816+0000","last_clean_scrub_stamp":"2024-04-14T15:24:29.131816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"332.1a","version":"0'0","reported_seq":1,"reported_epoch":727,"state":"creating+peering","last_fresh":"2024-04-14T15:24:29.147552+0000","last_change":"2024-04-14T15:24:29.147552+0000","last_active":"2024-04-14T15:24:29.131816+0000","last_peered":"2024-04-14T15:24:29.131816+0000","last_clean":"2024-04-14T15:24:29.131816+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:24:29.147552+0000","last_undegraded":"2024-04-14T15:24:29.147552+0000","last_fullsized":"2024-04-14T15:24:29.147552+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":727,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:29.131816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:29.131816+0000","last_clean_scrub_stamp":"2024-04-14T15:24:29.131816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"332.19","version":"0'0","reported_seq":1,"reported_epoch":727,"state":"creating+peering","last_fresh":"2024-04-14T15:24:29.144024+0000","last_change":"2024-04-14T15:24:29.144024+0000","last_active":"2024-04-14T15:24:29.131816+0000","last_peered":"2024-04-14T15:24:29.131816+0000","last_clean":"2024-04-14T15:24:29.131816+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:24:29.144024+0000","last_undegraded":"2024-04-14T15:24:29.144024+0000","last_fullsized":"2024-04-14T15:24:29.144024+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":727,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:29.131816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:29.131816+0000","last_clean_scrub_stamp":"2024-04-14T15:24:29.131816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"332.18","version":"0'0","reported_seq":1,"reported_epoch":727,"state":"creating+peering","last_fresh":"2024-04-14T15:24:29.147004+0000","last_change":"2024-04-14T15:24:29.147004+0000","last_active":"2024-04-14T15:24:29.131816+0000","last_peered":"2024-04-14T15:24:29.131816+0000","last_clean":"2024-04-14T15:24:29.131816+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:24:29.147004+0000","last_undegraded":"2024-04-14T15:24:29.147004+0000","last_fullsized":"2024-04-14T15:24:29.147004+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":727,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:29.131816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:29.131816+0000","last_clean_scrub_stamp":"2024-04-14T15:24:29.131816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"332.17","version":"0'0","reported_seq":1,"reported_epoch":727,"state":"creating+peering","last_fresh":"2024-04-14T15:24:29.144039+0000","last_change":"2024-04-14T15:24:29.144039+0000","last_active":"2024-04-14T15:24:29.131816+0000","last_peered":"2024-04-14T15:24:29.131816+0000","last_clean":"2024-04-14T15:24:29.131816+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:24:29.144039+0000","last_undegraded":"2024-04-14T15:24:29.144039+0000","last_fullsized":"2024-04-14T15:24:29.144039+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":727,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:29.131816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:29.131816+0000","last_clean_scrub_stamp":"2024-04-14T15:24:29.131816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"332.16","version":"0'0","reported_seq":1,"reported_epoch":727,"state":"creating+peering","last_fresh":"2024-04-14T15:24:29.143039+0000","last_change":"2024-04-14T15:24:29.143039+0000","last_active":"2024-04-14T15:24:29.131816+0000","last_peered":"2024-04-14T15:24:29.131816+0000","last_clean":"2024-04-14T15:24:29.131816+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:24:29.143039+0000","last_undegraded":"2024-04-14T15:24:29.143039+0000","last_fullsized":"2024-04-14T15:24:29.143039+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":727,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:29.131816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:29.131816+0000","last_clean_scrub_stamp":"2024-04-14T15:24:29.131816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"332.15","version":"0'0","reported_seq":1,"reported_epoch":727,"state":"creating+peering","last_fresh":"2024-04-14T15:24:29.142228+0000","last_change":"2024-04-14T15:24:29.142228+0000","last_active":"2024-04-14T15:24:29.131816+0000","last_peered":"2024-04-14T15:24:29.131816+0000","last_clean":"2024-04-14T15:24:29.131816+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:24:29.142228+0000","last_undegraded":"2024-04-14T15:24:29.142228+0000","last_fullsized":"2024-04-14T15:24:29.142228+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":727,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:29.131816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:29.131816+0000","last_clean_scrub_stamp":"2024-04-14T15:24:29.131816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"332.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:24:29.131816+0000","last_change":"2024-04-14T15:24:29.131816+0000","last_active":"2024-04-14T15:24:29.131816+0000","last_peered":"2024-04-14T15:24:29.131816+0000","last_clean":"2024-04-14T15:24:29.131816+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:24:29.131816+0000","last_undegraded":"2024-04-14T15:24:29.131816+0000","last_fullsized":"2024-04-14T15:24:29.131816+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:29.131816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:29.131816+0000","last_clean_scrub_stamp":"2024-04-14T15:24:29.131816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"332.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:24:29.131816+0000","last_change":"2024-04-14T15:24:29.131816+0000","last_active":"2024-04-14T15:24:29.131816+0000","last_peered":"2024-04-14T15:24:29.131816+0000","last_clean":"2024-04-14T15:24:29.131816+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:24:29.131816+0000","last_undegraded":"2024-04-14T15:24:29.131816+0000","last_fullsized":"2024-04-14T15:24:29.131816+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:29.131816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:29.131816+0000","last_clean_scrub_stamp":"2024-04-14T15:24:29.131816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"332.12","version":"0'0","reported_seq":1,"reported_epoch":727,"state":"creating+peering","last_fresh":"2024-04-14T15:24:29.141494+0000","last_change":"2024-04-14T15:24:29.141494+0000","last_active":"2024-04-14T15:24:29.131816+0000","last_peered":"2024-04-14T15:24:29.131816+0000","last_clean":"2024-04-14T15:24:29.131816+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:24:29.141494+0000","last_undegraded":"2024-04-14T15:24:29.141494+0000","last_fullsized":"2024-04-14T15:24:29.141494+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":727,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:29.131816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:29.131816+0000","last_clean_scrub_stamp":"2024-04-14T15:24:29.131816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"332.11","version":"0'0","reported_seq":1,"reported_epoch":727,"state":"creating+peering","last_fresh":"2024-04-14T15:24:29.147217+0000","last_change":"2024-04-14T15:24:29.147217+0000","last_active":"2024-04-14T15:24:29.131816+0000","last_peered":"2024-04-14T15:24:29.131816+0000","last_clean":"2024-04-14T15:24:29.131816+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:24:29.147217+0000","last_undegraded":"2024-04-14T15:24:29.147217+0000","last_fullsized":"2024-04-14T15:24:29.147217+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":727,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:29.131816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:29.131816+0000","last_clean_scrub_stamp":"2024-04-14T15:24:29.131816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"332.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:24:29.131816+0000","last_change":"2024-04-14T15:24:29.131816+0000","last_active":"2024-04-14T15:24:29.131816+0000","last_peered":"2024-04-14T15:24:29.131816+0000","last_clean":"2024-04-14T15:24:29.131816+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:24:29.131816+0000","last_undegraded":"2024-04-14T15:24:29.131816+0000","last_fullsized":"2024-04-14T15:24:29.131816+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:29.131816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:29.131816+0000","last_clean_scrub_stamp":"2024-04-14T15:24:29.131816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"332.7","version":"0'0","reported_seq":1,"reported_epoch":727,"state":"creating+peering","last_fresh":"2024-04-14T15:24:29.145967+0000","last_change":"2024-04-14T15:24:29.145967+0000","last_active":"2024-04-14T15:24:29.131816+0000","last_peered":"2024-04-14T15:24:29.131816+0000","last_clean":"2024-04-14T15:24:29.131816+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:24:29.145967+0000","last_undegraded":"2024-04-14T15:24:29.145967+0000","last_fullsized":"2024-04-14T15:24:29.145967+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":727,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:29.131816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:29.131816+0000","last_clean_scrub_stamp":"2024-04-14T15:24:29.131816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"332.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:24:29.131816+0000","last_change":"2024-04-14T15:24:29.131816+0000","last_active":"2024-04-14T15:24:29.131816+0000","last_peered":"2024-04-14T15:24:29.131816+0000","last_clean":"2024-04-14T15:24:29.131816+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:24:29.131816+0000","last_undegraded":"2024-04-14T15:24:29.131816+0000","last_fullsized":"2024-04-14T15:24:29.131816+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:29.131816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:29.131816+0000","last_clean_scrub_stamp":"2024-04-14T15:24:29.131816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"332.5","version":"0'0","reported_seq":1,"reported_epoch":727,"state":"creating+peering","last_fresh":"2024-04-14T15:24:29.149761+0000","last_change":"2024-04-14T15:24:29.149761+0000","last_active":"2024-04-14T15:24:29.131816+0000","last_peered":"2024-04-14T15:24:29.131816+0000","last_clean":"2024-04-14T15:24:29.131816+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:24:29.149761+0000","last_undegraded":"2024-04-14T15:24:29.149761+0000","last_fullsized":"2024-04-14T15:24:29.149761+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":727,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:29.131816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:29.131816+0000","last_clean_scrub_stamp":"2024-04-14T15:24:29.131816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"332.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:24:29.131816+0000","last_change":"2024-04-14T15:24:29.131816+0000","last_active":"2024-04-14T15:24:29.131816+0000","last_peered":"2024-04-14T15:24:29.131816+0000","last_clean":"2024-04-14T15:24:29.131816+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:24:29.131816+0000","last_undegraded":"2024-04-14T15:24:29.131816+0000","last_fullsized":"2024-04-14T15:24:29.131816+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:29.131816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:29.131816+0000","last_clean_scrub_stamp":"2024-04-14T15:24:29.131816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"332.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:24:29.131816+0000","last_change":"2024-04-14T15:24:29.131816+0000","last_active":"2024-04-14T15:24:29.131816+0000","last_peered":"2024-04-14T15:24:29.131816+0000","last_clean":"2024-04-14T15:24:29.131816+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:24:29.131816+0000","last_undegraded":"2024-04-14T15:24:29.131816+0000","last_fullsized":"2024-04-14T15:24:29.131816+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:29.131816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:29.131816+0000","last_clean_scrub_stamp":"2024-04-14T15:24:29.131816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"332.2","version":"0'0","reported_seq":1,"reported_epoch":727,"state":"creating+peering","last_fresh":"2024-04-14T15:24:29.141037+0000","last_change":"2024-04-14T15:24:29.141037+0000","last_active":"2024-04-14T15:24:29.131816+0000","last_peered":"2024-04-14T15:24:29.131816+0000","last_clean":"2024-04-14T15:24:29.131816+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:24:29.141037+0000","last_undegraded":"2024-04-14T15:24:29.141037+0000","last_fullsized":"2024-04-14T15:24:29.141037+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":727,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:29.131816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:29.131816+0000","last_clean_scrub_stamp":"2024-04-14T15:24:29.131816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"332.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:24:29.131816+0000","last_change":"2024-04-14T15:24:29.131816+0000","last_active":"2024-04-14T15:24:29.131816+0000","last_peered":"2024-04-14T15:24:29.131816+0000","last_clean":"2024-04-14T15:24:29.131816+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:24:29.131816+0000","last_undegraded":"2024-04-14T15:24:29.131816+0000","last_fullsized":"2024-04-14T15:24:29.131816+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:29.131816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:29.131816+0000","last_clean_scrub_stamp":"2024-04-14T15:24:29.131816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"332.0","version":"0'0","reported_seq":1,"reported_epoch":727,"state":"creating+peering","last_fresh":"2024-04-14T15:24:29.140455+0000","last_change":"2024-04-14T15:24:29.140455+0000","last_active":"2024-04-14T15:24:29.131816+0000","last_peered":"2024-04-14T15:24:29.131816+0000","last_clean":"2024-04-14T15:24:29.131816+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:24:29.140455+0000","last_undegraded":"2024-04-14T15:24:29.140455+0000","last_fullsized":"2024-04-14T15:24:29.140455+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":727,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:29.131816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:29.131816+0000","last_clean_scrub_stamp":"2024-04-14T15:24:29.131816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"332.8","version":"0'0","reported_seq":1,"reported_epoch":727,"state":"creating+peering","last_fresh":"2024-04-14T15:24:29.146445+0000","last_change":"2024-04-14T15:24:29.146445+0000","last_active":"2024-04-14T15:24:29.131816+0000","last_peered":"2024-04-14T15:24:29.131816+0000","last_clean":"2024-04-14T15:24:29.131816+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:24:29.146445+0000","last_undegraded":"2024-04-14T15:24:29.146445+0000","last_fullsized":"2024-04-14T15:24:29.146445+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":727,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:29.131816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:29.131816+0000","last_clean_scrub_stamp":"2024-04-14T15:24:29.131816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"323.7","version":"0'0","reported_seq":215,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.151409+0000","last_change":"2024-04-14T15:24:15.999335+0000","last_active":"2024-04-14T15:24:29.151409+0000","last_peered":"2024-04-14T15:24:29.151409+0000","last_clean":"2024-04-14T15:24:29.151409+0000","last_became_active":"2024-04-14T15:23:04.005793+0000","last_became_peered":"2024-04-14T15:23:04.005793+0000","last_unstale":"2024-04-14T15:24:29.151409+0000","last_undegraded":"2024-04-14T15:24:29.151409+0000","last_fullsized":"2024-04-14T15:24:29.151409+0000","mapping_epoch":647,"log_start":"0'0","ondisk_log_start":"0'0","created":647,"last_epoch_clean":648,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_clean_scrub_stamp":"2024-04-14T15:23:02.982695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002596845,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"332.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:24:29.131816+0000","last_change":"2024-04-14T15:24:29.131816+0000","last_active":"2024-04-14T15:24:29.131816+0000","last_peered":"2024-04-14T15:24:29.131816+0000","last_clean":"2024-04-14T15:24:29.131816+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:24:29.131816+0000","last_undegraded":"2024-04-14T15:24:29.131816+0000","last_fullsized":"2024-04-14T15:24:29.131816+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:29.131816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:29.131816+0000","last_clean_scrub_stamp":"2024-04-14T15:24:29.131816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"323.6","version":"656'1","reported_seq":222,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.142470+0000","last_change":"2024-04-14T15:24:16.002091+0000","last_active":"2024-04-14T15:24:29.142470+0000","last_peered":"2024-04-14T15:24:29.142470+0000","last_clean":"2024-04-14T15:24:29.142470+0000","last_became_active":"2024-04-14T15:23:04.006098+0000","last_became_peered":"2024-04-14T15:23:04.006098+0000","last_unstale":"2024-04-14T15:24:29.142470+0000","last_undegraded":"2024-04-14T15:24:29.142470+0000","last_fullsized":"2024-04-14T15:24:29.142470+0000","mapping_epoch":647,"log_start":"0'0","ondisk_log_start":"0'0","created":647,"last_epoch_clean":648,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_clean_scrub_stamp":"2024-04-14T15:23:02.982695+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:29.246742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0050417159999999999,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"332.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:24:29.131816+0000","last_change":"2024-04-14T15:24:29.131816+0000","last_active":"2024-04-14T15:24:29.131816+0000","last_peered":"2024-04-14T15:24:29.131816+0000","last_clean":"2024-04-14T15:24:29.131816+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:24:29.131816+0000","last_undegraded":"2024-04-14T15:24:29.131816+0000","last_fullsized":"2024-04-14T15:24:29.131816+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:29.131816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:29.131816+0000","last_clean_scrub_stamp":"2024-04-14T15:24:29.131816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"323.5","version":"678'10","reported_seq":227,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-14T15:24:28.136281+0000","last_change":"2024-04-14T15:24:16.005799+0000","last_active":"2024-04-14T15:24:28.136281+0000","last_peered":"2024-04-14T15:24:28.136281+0000","last_clean":"2024-04-14T15:24:28.136281+0000","last_became_active":"2024-04-14T15:23:04.002370+0000","last_became_peered":"2024-04-14T15:23:04.002370+0000","last_unstale":"2024-04-14T15:24:28.136281+0000","last_undegraded":"2024-04-14T15:24:28.136281+0000","last_fullsized":"2024-04-14T15:24:28.136281+0000","mapping_epoch":647,"log_start":"0'0","ondisk_log_start":"0'0","created":647,"last_epoch_clean":648,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_clean_scrub_stamp":"2024-04-14T15:23:02.982695+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016554479999999999,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"332.b","version":"0'0","reported_seq":1,"reported_epoch":727,"state":"creating+peering","last_fresh":"2024-04-14T15:24:29.146905+0000","last_change":"2024-04-14T15:24:29.146905+0000","last_active":"2024-04-14T15:24:29.131816+0000","last_peered":"2024-04-14T15:24:29.131816+0000","last_clean":"2024-04-14T15:24:29.131816+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:24:29.146905+0000","last_undegraded":"2024-04-14T15:24:29.146905+0000","last_fullsized":"2024-04-14T15:24:29.146905+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":727,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:29.131816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:29.131816+0000","last_clean_scrub_stamp":"2024-04-14T15:24:29.131816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"323.4","version":"714'10","reported_seq":230,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.148908+0000","last_change":"2024-04-14T15:24:16.012928+0000","last_active":"2024-04-14T15:24:29.148908+0000","last_peered":"2024-04-14T15:24:29.148908+0000","last_clean":"2024-04-14T15:24:29.148908+0000","last_became_active":"2024-04-14T15:23:03.997912+0000","last_became_peered":"2024-04-14T15:23:03.997912+0000","last_unstale":"2024-04-14T15:24:29.148908+0000","last_undegraded":"2024-04-14T15:24:29.148908+0000","last_fullsized":"2024-04-14T15:24:29.148908+0000","mapping_epoch":647,"log_start":"0'0","ondisk_log_start":"0'0","created":647,"last_epoch_clean":648,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_clean_scrub_stamp":"2024-04-14T15:23:02.982695+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0025287679999999998,"stat_sum":{"num_bytes":10,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"332.c","version":"0'0","reported_seq":1,"reported_epoch":727,"state":"creating+peering","last_fresh":"2024-04-14T15:24:29.146197+0000","last_change":"2024-04-14T15:24:29.146197+0000","last_active":"2024-04-14T15:24:29.131816+0000","last_peered":"2024-04-14T15:24:29.131816+0000","last_clean":"2024-04-14T15:24:29.131816+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:24:29.146197+0000","last_undegraded":"2024-04-14T15:24:29.146197+0000","last_fullsized":"2024-04-14T15:24:29.146197+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":727,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:29.131816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:29.131816+0000","last_clean_scrub_stamp":"2024-04-14T15:24:29.131816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"323.3","version":"722'13","reported_seq":230,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.145724+0000","last_change":"2024-04-14T15:24:18.000702+0000","last_active":"2024-04-14T15:24:29.145724+0000","last_peered":"2024-04-14T15:24:29.145724+0000","last_clean":"2024-04-14T15:24:29.145724+0000","last_became_active":"2024-04-14T15:23:04.001213+0000","last_became_peered":"2024-04-14T15:23:04.001213+0000","last_unstale":"2024-04-14T15:24:29.145724+0000","last_undegraded":"2024-04-14T15:24:29.145724+0000","last_fullsized":"2024-04-14T15:24:29.145724+0000","mapping_epoch":647,"log_start":"0'0","ondisk_log_start":"0'0","created":647,"last_epoch_clean":648,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_clean_scrub_stamp":"2024-04-14T15:23:02.982695+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.25140135000000002,"stat_sum":{"num_bytes":21,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"332.d","version":"0'0","reported_seq":1,"reported_epoch":727,"state":"creating+peering","last_fresh":"2024-04-14T15:24:29.144164+0000","last_change":"2024-04-14T15:24:29.144164+0000","last_active":"2024-04-14T15:24:29.131816+0000","last_peered":"2024-04-14T15:24:29.131816+0000","last_clean":"2024-04-14T15:24:29.131816+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:24:29.144164+0000","last_undegraded":"2024-04-14T15:24:29.144164+0000","last_fullsized":"2024-04-14T15:24:29.144164+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":727,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:29.131816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:29.131816+0000","last_clean_scrub_stamp":"2024-04-14T15:24:29.131816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"323.2","version":"714'10","reported_seq":239,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.140786+0000","last_change":"2024-04-14T15:24:18.004566+0000","last_active":"2024-04-14T15:24:29.140786+0000","last_peered":"2024-04-14T15:24:29.140786+0000","last_clean":"2024-04-14T15:24:29.140786+0000","last_became_active":"2024-04-14T15:23:04.001557+0000","last_became_peered":"2024-04-14T15:23:04.001557+0000","last_unstale":"2024-04-14T15:24:29.140786+0000","last_undegraded":"2024-04-14T15:24:29.140786+0000","last_fullsized":"2024-04-14T15:24:29.140786+0000","mapping_epoch":647,"log_start":"0'0","ondisk_log_start":"0'0","created":647,"last_epoch_clean":648,"parent":"0.0","parent_split_bits":0,"last_scrub":"710'6","last_scrub_stamp":"2024-04-14T15:24:15.331688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_clean_scrub_stamp":"2024-04-14T15:24:15.331688+0000","objects_scrubbed":4,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:15.783842+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":1.0003185059999999,"stat_sum":{"num_bytes":10,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"332.e","version":"0'0","reported_seq":1,"reported_epoch":727,"state":"creating+peering","last_fresh":"2024-04-14T15:24:29.151431+0000","last_change":"2024-04-14T15:24:29.151431+0000","last_active":"2024-04-14T15:24:29.131816+0000","last_peered":"2024-04-14T15:24:29.131816+0000","last_clean":"2024-04-14T15:24:29.131816+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:24:29.151431+0000","last_undegraded":"2024-04-14T15:24:29.151431+0000","last_fullsized":"2024-04-14T15:24:29.151431+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":727,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:29.131816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:29.131816+0000","last_clean_scrub_stamp":"2024-04-14T15:24:29.131816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"323.1","version":"696'3","reported_seq":222,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-14T15:24:28.775923+0000","last_change":"2024-04-14T15:24:16.006310+0000","last_active":"2024-04-14T15:24:28.775923+0000","last_peered":"2024-04-14T15:24:28.775923+0000","last_clean":"2024-04-14T15:24:28.775923+0000","last_became_active":"2024-04-14T15:23:03.998498+0000","last_became_peered":"2024-04-14T15:23:03.998498+0000","last_unstale":"2024-04-14T15:24:28.775923+0000","last_undegraded":"2024-04-14T15:24:28.775923+0000","last_fullsized":"2024-04-14T15:24:28.775923+0000","mapping_epoch":647,"log_start":"0'0","ondisk_log_start":"0'0","created":647,"last_epoch_clean":648,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_clean_scrub_stamp":"2024-04-14T15:23:02.982695+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:29.794331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060225600000000002,"stat_sum":{"num_bytes":12,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"332.f","version":"0'0","reported_seq":1,"reported_epoch":727,"state":"creating+peering","last_fresh":"2024-04-14T15:24:29.146610+0000","last_change":"2024-04-14T15:24:29.146610+0000","last_active":"2024-04-14T15:24:29.131816+0000","last_peered":"2024-04-14T15:24:29.131816+0000","last_clean":"2024-04-14T15:24:29.131816+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:24:29.146610+0000","last_undegraded":"2024-04-14T15:24:29.146610+0000","last_fullsized":"2024-04-14T15:24:29.146610+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":727,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:29.131816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:29.131816+0000","last_clean_scrub_stamp":"2024-04-14T15:24:29.131816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"323.0","version":"704'6","reported_seq":227,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.139952+0000","last_change":"2024-04-14T15:24:15.995558+0000","last_active":"2024-04-14T15:24:29.139952+0000","last_peered":"2024-04-14T15:24:29.139952+0000","last_clean":"2024-04-14T15:24:29.139952+0000","last_became_active":"2024-04-14T15:23:04.001020+0000","last_became_peered":"2024-04-14T15:23:04.001020+0000","last_unstale":"2024-04-14T15:24:29.139952+0000","last_undegraded":"2024-04-14T15:24:29.139952+0000","last_fullsized":"2024-04-14T15:24:29.139952+0000","mapping_epoch":647,"log_start":"0'0","ondisk_log_start":"0'0","created":647,"last_epoch_clean":648,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_clean_scrub_stamp":"2024-04-14T15:23:02.982695+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024036109999999999,"stat_sum":{"num_bytes":5,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":932,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.143195+0000","last_change":"2024-04-14T15:23:27.292731+0000","last_active":"2024-04-14T15:24:29.143195+0000","last_peered":"2024-04-14T15:24:29.143195+0000","last_clean":"2024-04-14T15:24:29.143195+0000","last_became_active":"2024-04-14T15:15:44.882941+0000","last_became_peered":"2024-04-14T15:15:44.882941+0000","last_unstale":"2024-04-14T15:24:29.143195+0000","last_undegraded":"2024-04-14T15:24:29.143195+0000","last_fullsized":"2024-04-14T15:24:29.143195+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:27.292590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:23:27.292590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:36.719319+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.9","version":"528'10","reported_seq":1565,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-14T15:24:28.776968+0000","last_change":"2024-04-14T15:20:52.998038+0000","last_active":"2024-04-14T15:24:28.776968+0000","last_peered":"2024-04-14T15:24:28.776968+0000","last_clean":"2024-04-14T15:24:28.776968+0000","last_became_active":"2024-04-14T15:20:52.984943+0000","last_became_peered":"2024-04-14T15:20:52.984943+0000","last_unstale":"2024-04-14T15:24:28.776968+0000","last_undegraded":"2024-04-14T15:24:28.776968+0000","last_fullsized":"2024-04-14T15:24:28.776968+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9","last_scrub_stamp":"2024-04-14T15:20:45.737773+0000","last_deep_scrub":"85'9","last_deep_scrub_stamp":"2024-04-14T15:10:51.950580+0000","last_clean_scrub_stamp":"2024-04-14T15:20:45.737773+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"479'34","reported_seq":1018,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-14T15:24:28.776059+0000","last_change":"2024-04-14T15:21:27.711513+0000","last_active":"2024-04-14T15:24:28.776059+0000","last_peered":"2024-04-14T15:24:28.776059+0000","last_clean":"2024-04-14T15:24:28.776059+0000","last_became_active":"2024-04-14T15:20:52.979204+0000","last_became_peered":"2024-04-14T15:20:52.979204+0000","last_unstale":"2024-04-14T15:24:28.776059+0000","last_undegraded":"2024-04-14T15:24:28.776059+0000","last_fullsized":"2024-04-14T15:24:28.776059+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'34","last_scrub_stamp":"2024-04-14T15:21:27.711357+0000","last_deep_scrub":"430'22","last_deep_scrub_stamp":"2024-04-14T15:18:39.704883+0000","last_clean_scrub_stamp":"2024-04-14T15:21:27.711357+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":1383,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.157412+0000","last_change":"2024-04-14T15:23:24.788057+0000","last_active":"2024-04-14T15:24:29.157412+0000","last_peered":"2024-04-14T15:24:29.157412+0000","last_clean":"2024-04-14T15:24:29.157412+0000","last_became_active":"2024-04-14T15:09:43.021460+0000","last_became_peered":"2024-04-14T15:09:43.021460+0000","last_unstale":"2024-04-14T15:24:29.157412+0000","last_undegraded":"2024-04-14T15:24:29.157412+0000","last_fullsized":"2024-04-14T15:24:29.157412+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:24.787958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:23:24.787958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:50.309267+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":420,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.147954+0000","last_change":"2024-04-14T15:23:05.437375+0000","last_active":"2024-04-14T15:24:29.147954+0000","last_peered":"2024-04-14T15:24:29.147954+0000","last_clean":"2024-04-14T15:24:29.147954+0000","last_became_active":"2024-04-14T15:20:52.989666+0000","last_became_peered":"2024-04-14T15:20:52.989666+0000","last_unstale":"2024-04-14T15:24:29.147954+0000","last_undegraded":"2024-04-14T15:24:29.147954+0000","last_fullsized":"2024-04-14T15:24:29.147954+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:05.437268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:15:40.368219+0000","last_clean_scrub_stamp":"2024-04-14T15:23:05.437268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:30.217014+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":1279,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-14T15:24:28.778466+0000","last_change":"2024-04-14T15:21:24.713987+0000","last_active":"2024-04-14T15:24:28.778466+0000","last_peered":"2024-04-14T15:24:28.778466+0000","last_clean":"2024-04-14T15:24:28.778466+0000","last_became_active":"2024-04-14T15:20:52.991952+0000","last_became_peered":"2024-04-14T15:20:52.991952+0000","last_unstale":"2024-04-14T15:24:28.778466+0000","last_undegraded":"2024-04-14T15:24:28.778466+0000","last_fullsized":"2024-04-14T15:24:28.778466+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:24.713877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:21:24.713877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"479'15","reported_seq":1406,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.148223+0000","last_change":"2024-04-14T15:23:32.253957+0000","last_active":"2024-04-14T15:24:29.148223+0000","last_peered":"2024-04-14T15:24:29.148223+0000","last_clean":"2024-04-14T15:24:29.148223+0000","last_became_active":"2024-04-14T15:09:45.037179+0000","last_became_peered":"2024-04-14T15:09:45.037179+0000","last_unstale":"2024-04-14T15:24:29.148223+0000","last_undegraded":"2024-04-14T15:24:29.148223+0000","last_fullsized":"2024-04-14T15:24:29.148223+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'15","last_scrub_stamp":"2024-04-14T15:23:32.253815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:23:32.253815+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:37.113811+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":1381,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.142894+0000","last_change":"2024-04-14T15:22:17.414577+0000","last_active":"2024-04-14T15:24:29.142894+0000","last_peered":"2024-04-14T15:24:29.142894+0000","last_clean":"2024-04-14T15:24:29.142894+0000","last_became_active":"2024-04-14T15:09:43.020044+0000","last_became_peered":"2024-04-14T15:09:43.020044+0000","last_unstale":"2024-04-14T15:24:29.142894+0000","last_undegraded":"2024-04-14T15:24:29.142894+0000","last_fullsized":"2024-04-14T15:24:29.142894+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:17.414499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:22:17.414499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1383,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.155044+0000","last_change":"2024-04-14T15:24:27.475600+0000","last_active":"2024-04-14T15:24:29.155044+0000","last_peered":"2024-04-14T15:24:29.155044+0000","last_clean":"2024-04-14T15:24:29.155044+0000","last_became_active":"2024-04-14T15:09:47.046153+0000","last_became_peered":"2024-04-14T15:09:47.046153+0000","last_unstale":"2024-04-14T15:24:29.155044+0000","last_undegraded":"2024-04-14T15:24:29.155044+0000","last_fullsized":"2024-04-14T15:24:29.155044+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:27.475490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:24:27.475490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:53.144228+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1373,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.146023+0000","last_change":"2024-04-14T15:23:37.201923+0000","last_active":"2024-04-14T15:24:29.146023+0000","last_peered":"2024-04-14T15:24:29.146023+0000","last_clean":"2024-04-14T15:24:29.146023+0000","last_became_active":"2024-04-14T15:09:49.061495+0000","last_became_peered":"2024-04-14T15:09:49.061495+0000","last_unstale":"2024-04-14T15:24:29.146023+0000","last_undegraded":"2024-04-14T15:24:29.146023+0000","last_fullsized":"2024-04-14T15:24:29.146023+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:37.201795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:23:37.201795+0000","last_clean_scrub_stamp":"2024-04-14T15:23:37.201795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:49.569529+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":1279,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-14T15:24:28.778573+0000","last_change":"2024-04-14T15:21:20.684941+0000","last_active":"2024-04-14T15:24:28.778573+0000","last_peered":"2024-04-14T15:24:28.778573+0000","last_clean":"2024-04-14T15:24:28.778573+0000","last_became_active":"2024-04-14T15:20:52.999845+0000","last_became_peered":"2024-04-14T15:20:52.999845+0000","last_unstale":"2024-04-14T15:24:28.778573+0000","last_undegraded":"2024-04-14T15:24:28.778573+0000","last_fullsized":"2024-04-14T15:24:28.778573+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:20.684813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:21:20.684813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"479'44","reported_seq":270,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.154507+0000","last_change":"2024-04-14T15:23:03.739862+0000","last_active":"2024-04-14T15:24:29.154507+0000","last_peered":"2024-04-14T15:24:29.154507+0000","last_clean":"2024-04-14T15:24:29.154507+0000","last_became_active":"2024-04-14T15:22:25.970899+0000","last_became_peered":"2024-04-14T15:22:25.970899+0000","last_unstale":"2024-04-14T15:24:29.154507+0000","last_undegraded":"2024-04-14T15:24:29.154507+0000","last_fullsized":"2024-04-14T15:24:29.154507+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'44","last_scrub_stamp":"2024-04-14T15:23:03.739762+0000","last_deep_scrub":"479'44","last_deep_scrub_stamp":"2024-04-14T15:19:58.856115+0000","last_clean_scrub_stamp":"2024-04-14T15:23:03.739762+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":1391,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.158570+0000","last_change":"2024-04-14T15:23:44.419193+0000","last_active":"2024-04-14T15:24:29.158570+0000","last_peered":"2024-04-14T15:24:29.158570+0000","last_clean":"2024-04-14T15:24:29.158570+0000","last_became_active":"2024-04-14T15:09:43.020268+0000","last_became_peered":"2024-04-14T15:09:43.020268+0000","last_unstale":"2024-04-14T15:24:29.158570+0000","last_undegraded":"2024-04-14T15:24:29.158570+0000","last_fullsized":"2024-04-14T15:24:29.158570+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:44.419100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:34.342793+0000","last_clean_scrub_stamp":"2024-04-14T15:23:44.419100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:56.886676+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1375,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.154123+0000","last_change":"2024-04-14T15:24:23.499160+0000","last_active":"2024-04-14T15:24:29.154123+0000","last_peered":"2024-04-14T15:24:29.154123+0000","last_clean":"2024-04-14T15:24:29.154123+0000","last_became_active":"2024-04-14T15:09:47.045542+0000","last_became_peered":"2024-04-14T15:09:47.045542+0000","last_unstale":"2024-04-14T15:24:29.154123+0000","last_undegraded":"2024-04-14T15:24:29.154123+0000","last_fullsized":"2024-04-14T15:24:29.154123+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:23.499070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:24:23.499070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:23.657015+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1364,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-14T15:24:28.776516+0000","last_change":"2024-04-14T15:22:55.492856+0000","last_active":"2024-04-14T15:24:28.776516+0000","last_peered":"2024-04-14T15:24:28.776516+0000","last_clean":"2024-04-14T15:24:28.776516+0000","last_became_active":"2024-04-14T15:09:49.065397+0000","last_became_peered":"2024-04-14T15:09:49.065397+0000","last_unstale":"2024-04-14T15:24:28.776516+0000","last_undegraded":"2024-04-14T15:24:28.776516+0000","last_fullsized":"2024-04-14T15:24:28.776516+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:55.492753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:22:55.492753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"91'130","reported_seq":1319,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.153411+0000","last_change":"2024-04-14T15:21:58.912480+0000","last_active":"2024-04-14T15:24:29.153411+0000","last_peered":"2024-04-14T15:24:29.153411+0000","last_clean":"2024-04-14T15:24:29.153411+0000","last_became_active":"2024-04-14T15:21:12.961419+0000","last_became_peered":"2024-04-14T15:21:12.961419+0000","last_unstale":"2024-04-14T15:24:29.153411+0000","last_undegraded":"2024-04-14T15:24:29.153411+0000","last_fullsized":"2024-04-14T15:24:29.153411+0000","mapping_epoch":276,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:21:58.912387+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:21:58.912387+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":70,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":1280,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.154910+0000","last_change":"2024-04-14T15:22:14.964486+0000","last_active":"2024-04-14T15:24:29.154910+0000","last_peered":"2024-04-14T15:24:29.154910+0000","last_clean":"2024-04-14T15:24:29.154910+0000","last_became_active":"2024-04-14T15:12:56.474322+0000","last_became_peered":"2024-04-14T15:12:56.474322+0000","last_unstale":"2024-04-14T15:24:29.154910+0000","last_undegraded":"2024-04-14T15:24:29.154910+0000","last_fullsized":"2024-04-14T15:24:29.154910+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:14.964404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:22:14.964404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"479'31","reported_seq":1469,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-14T15:24:28.138387+0000","last_change":"2024-04-14T15:22:02.037366+0000","last_active":"2024-04-14T15:24:28.138387+0000","last_peered":"2024-04-14T15:24:28.138387+0000","last_clean":"2024-04-14T15:24:28.138387+0000","last_became_active":"2024-04-14T15:13:12.933907+0000","last_became_peered":"2024-04-14T15:13:12.933907+0000","last_unstale":"2024-04-14T15:24:28.138387+0000","last_undegraded":"2024-04-14T15:24:28.138387+0000","last_fullsized":"2024-04-14T15:24:28.138387+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'31","last_scrub_stamp":"2024-04-14T15:22:02.037237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:22:02.037237+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":1376,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-14T15:24:28.778159+0000","last_change":"2024-04-14T15:23:26.611299+0000","last_active":"2024-04-14T15:24:28.778159+0000","last_peered":"2024-04-14T15:24:28.778159+0000","last_clean":"2024-04-14T15:24:28.778159+0000","last_became_active":"2024-04-14T15:09:43.022351+0000","last_became_peered":"2024-04-14T15:09:43.022351+0000","last_unstale":"2024-04-14T15:24:28.778159+0000","last_undegraded":"2024-04-14T15:24:28.778159+0000","last_fullsized":"2024-04-14T15:24:28.778159+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:26.611217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:23:26.611217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:40.741998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1373,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.142329+0000","last_change":"2024-04-14T15:23:25.344592+0000","last_active":"2024-04-14T15:24:29.142329+0000","last_peered":"2024-04-14T15:24:29.142329+0000","last_clean":"2024-04-14T15:24:29.142329+0000","last_became_active":"2024-04-14T15:09:47.042810+0000","last_became_peered":"2024-04-14T15:09:47.042810+0000","last_unstale":"2024-04-14T15:24:29.142329+0000","last_undegraded":"2024-04-14T15:24:29.142329+0000","last_fullsized":"2024-04-14T15:24:29.142329+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:25.344505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:23:25.344505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1371,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.151649+0000","last_change":"2024-04-14T15:22:58.703767+0000","last_active":"2024-04-14T15:24:29.151649+0000","last_peered":"2024-04-14T15:24:29.151649+0000","last_clean":"2024-04-14T15:24:29.151649+0000","last_became_active":"2024-04-14T15:09:49.069594+0000","last_became_peered":"2024-04-14T15:09:49.069594+0000","last_unstale":"2024-04-14T15:24:29.151649+0000","last_undegraded":"2024-04-14T15:24:29.151649+0000","last_fullsized":"2024-04-14T15:24:29.151649+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:58.703688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:22:58.703688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"91'130","reported_seq":1721,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-14T15:24:28.776151+0000","last_change":"2024-04-14T15:22:04.530538+0000","last_active":"2024-04-14T15:24:28.776151+0000","last_peered":"2024-04-14T15:24:28.776151+0000","last_clean":"2024-04-14T15:24:28.776151+0000","last_became_active":"2024-04-14T15:21:09.020300+0000","last_became_peered":"2024-04-14T15:21:09.020300+0000","last_unstale":"2024-04-14T15:24:28.776151+0000","last_undegraded":"2024-04-14T15:24:28.776151+0000","last_fullsized":"2024-04-14T15:24:28.776151+0000","mapping_epoch":189,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":544,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:22:04.530456+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:22:04.530456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":416,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-14T15:24:28.777617+0000","last_change":"2024-04-14T15:23:41.487909+0000","last_active":"2024-04-14T15:24:28.777617+0000","last_peered":"2024-04-14T15:24:28.777617+0000","last_clean":"2024-04-14T15:24:28.777617+0000","last_became_active":"2024-04-14T15:20:52.980375+0000","last_became_peered":"2024-04-14T15:20:52.980375+0000","last_unstale":"2024-04-14T15:24:28.777617+0000","last_undegraded":"2024-04-14T15:24:28.777617+0000","last_fullsized":"2024-04-14T15:24:28.777617+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:41.487814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:23:41.487814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:56.105699+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"479'25","reported_seq":1417,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-14T15:24:28.136935+0000","last_change":"2024-04-14T15:20:53.030926+0000","last_active":"2024-04-14T15:24:28.136935+0000","last_peered":"2024-04-14T15:24:28.136935+0000","last_clean":"2024-04-14T15:24:28.136935+0000","last_became_active":"2024-04-14T15:20:52.987699+0000","last_became_peered":"2024-04-14T15:20:52.987699+0000","last_unstale":"2024-04-14T15:24:28.136935+0000","last_undegraded":"2024-04-14T15:24:28.136935+0000","last_fullsized":"2024-04-14T15:24:28.136935+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'25","last_scrub_stamp":"2024-04-14T15:20:03.571160+0000","last_deep_scrub":"479'25","last_deep_scrub_stamp":"2024-04-14T15:20:03.571160+0000","last_clean_scrub_stamp":"2024-04-14T15:20:03.571160+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":1381,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.141279+0000","last_change":"2024-04-14T15:21:31.700100+0000","last_active":"2024-04-14T15:24:29.141279+0000","last_peered":"2024-04-14T15:24:29.141279+0000","last_clean":"2024-04-14T15:24:29.141279+0000","last_became_active":"2024-04-14T15:09:43.018871+0000","last_became_peered":"2024-04-14T15:09:43.018871+0000","last_unstale":"2024-04-14T15:24:29.141279+0000","last_undegraded":"2024-04-14T15:24:29.141279+0000","last_fullsized":"2024-04-14T15:24:29.141279+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:31.700000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:21:31.700000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"400'9","reported_seq":1562,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.156323+0000","last_change":"2024-04-14T15:23:30.695693+0000","last_active":"2024-04-14T15:24:29.156323+0000","last_peered":"2024-04-14T15:24:29.156323+0000","last_clean":"2024-04-14T15:24:29.156323+0000","last_became_active":"2024-04-14T15:09:47.044085+0000","last_became_peered":"2024-04-14T15:09:47.044085+0000","last_unstale":"2024-04-14T15:24:29.156323+0000","last_undegraded":"2024-04-14T15:24:29.156323+0000","last_fullsized":"2024-04-14T15:24:29.156323+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'9","last_scrub_stamp":"2024-04-14T15:23:30.695563+0000","last_deep_scrub":"400'9","last_deep_scrub_stamp":"2024-04-14T15:17:54.845881+0000","last_clean_scrub_stamp":"2024-04-14T15:23:30.695563+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:41.740172+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1364,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-14T15:24:28.137439+0000","last_change":"2024-04-14T15:24:19.552836+0000","last_active":"2024-04-14T15:24:28.137439+0000","last_peered":"2024-04-14T15:24:28.137439+0000","last_clean":"2024-04-14T15:24:28.137439+0000","last_became_active":"2024-04-14T15:09:49.065274+0000","last_became_peered":"2024-04-14T15:09:49.065274+0000","last_unstale":"2024-04-14T15:24:28.137439+0000","last_undegraded":"2024-04-14T15:24:28.137439+0000","last_fullsized":"2024-04-14T15:24:28.137439+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:19.552703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:18.588546+0000","last_clean_scrub_stamp":"2024-04-14T15:24:19.552703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:36.798103+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"91'130","reported_seq":1334,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.154697+0000","last_change":"2024-04-14T15:22:06.865915+0000","last_active":"2024-04-14T15:24:29.154697+0000","last_peered":"2024-04-14T15:24:29.154697+0000","last_clean":"2024-04-14T15:24:29.154697+0000","last_became_active":"2024-04-14T15:21:04.971515+0000","last_became_peered":"2024-04-14T15:21:04.971515+0000","last_unstale":"2024-04-14T15:24:29.154697+0000","last_undegraded":"2024-04-14T15:24:29.154697+0000","last_fullsized":"2024-04-14T15:24:29.154697+0000","mapping_epoch":530,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":540,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:22:06.865825+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:22:06.865825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":1278,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.140452+0000","last_change":"2024-04-14T15:23:29.219186+0000","last_active":"2024-04-14T15:24:29.140452+0000","last_peered":"2024-04-14T15:24:29.140452+0000","last_clean":"2024-04-14T15:24:29.140452+0000","last_became_active":"2024-04-14T15:12:56.487615+0000","last_became_peered":"2024-04-14T15:12:56.487615+0000","last_unstale":"2024-04-14T15:24:29.140452+0000","last_undegraded":"2024-04-14T15:24:29.140452+0000","last_fullsized":"2024-04-14T15:24:29.140452+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:29.219111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:23:29.219111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:30.505271+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"479'34","reported_seq":1427,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.143312+0000","last_change":"2024-04-14T15:21:13.577045+0000","last_active":"2024-04-14T15:24:29.143312+0000","last_peered":"2024-04-14T15:24:29.143312+0000","last_clean":"2024-04-14T15:24:29.143312+0000","last_became_active":"2024-04-14T15:09:45.033331+0000","last_became_peered":"2024-04-14T15:09:45.033331+0000","last_unstale":"2024-04-14T15:24:29.143312+0000","last_undegraded":"2024-04-14T15:24:29.143312+0000","last_fullsized":"2024-04-14T15:24:29.143312+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'34","last_scrub_stamp":"2024-04-14T15:21:13.576916+0000","last_deep_scrub":"398'22","last_deep_scrub_stamp":"2024-04-14T15:17:52.883850+0000","last_clean_scrub_stamp":"2024-04-14T15:21:13.576916+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":1381,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.140565+0000","last_change":"2024-04-14T15:22:55.384701+0000","last_active":"2024-04-14T15:24:29.140565+0000","last_peered":"2024-04-14T15:24:29.140565+0000","last_clean":"2024-04-14T15:24:29.140565+0000","last_became_active":"2024-04-14T15:09:43.020878+0000","last_became_peered":"2024-04-14T15:09:43.020878+0000","last_unstale":"2024-04-14T15:24:29.140565+0000","last_undegraded":"2024-04-14T15:24:29.140565+0000","last_fullsized":"2024-04-14T15:24:29.140565+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:55.384626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:22:55.384626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"72'8","reported_seq":1554,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-14T15:24:28.137368+0000","last_change":"2024-04-14T15:24:27.403836+0000","last_active":"2024-04-14T15:24:28.137368+0000","last_peered":"2024-04-14T15:24:28.137368+0000","last_clean":"2024-04-14T15:24:28.137368+0000","last_became_active":"2024-04-14T15:09:47.050962+0000","last_became_peered":"2024-04-14T15:09:47.050962+0000","last_unstale":"2024-04-14T15:24:28.137368+0000","last_undegraded":"2024-04-14T15:24:28.137368+0000","last_fullsized":"2024-04-14T15:24:28.137368+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:24:27.403735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:24:27.403735+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:30.439681+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":940,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.142566+0000","last_change":"2024-04-14T15:24:27.318413+0000","last_active":"2024-04-14T15:24:29.142566+0000","last_peered":"2024-04-14T15:24:29.142566+0000","last_clean":"2024-04-14T15:24:29.142566+0000","last_became_active":"2024-04-14T15:15:44.879699+0000","last_became_peered":"2024-04-14T15:15:44.879699+0000","last_unstale":"2024-04-14T15:24:29.142566+0000","last_undegraded":"2024-04-14T15:24:29.142566+0000","last_fullsized":"2024-04-14T15:24:29.142566+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:27.318330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:24:27.318330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:32.125472+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"233'8","reported_seq":696,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.144823+0000","last_change":"2024-04-14T15:23:43.908078+0000","last_active":"2024-04-14T15:24:29.144823+0000","last_peered":"2024-04-14T15:24:29.144823+0000","last_clean":"2024-04-14T15:24:29.144823+0000","last_became_active":"2024-04-14T15:20:52.995129+0000","last_became_peered":"2024-04-14T15:20:52.995129+0000","last_unstale":"2024-04-14T15:24:29.144823+0000","last_undegraded":"2024-04-14T15:24:29.144823+0000","last_fullsized":"2024-04-14T15:24:29.144823+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'8","last_scrub_stamp":"2024-04-14T15:23:43.907946+0000","last_deep_scrub":"233'8","last_deep_scrub_stamp":"2024-04-14T15:15:01.295800+0000","last_clean_scrub_stamp":"2024-04-14T15:23:43.907946+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:59.919832+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":5,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"479'24","reported_seq":1426,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-14T15:24:28.136086+0000","last_change":"2024-04-14T15:21:13.939265+0000","last_active":"2024-04-14T15:24:28.136086+0000","last_peered":"2024-04-14T15:24:28.136086+0000","last_clean":"2024-04-14T15:24:28.136086+0000","last_became_active":"2024-04-14T15:20:52.998853+0000","last_became_peered":"2024-04-14T15:20:52.998853+0000","last_unstale":"2024-04-14T15:24:28.136086+0000","last_undegraded":"2024-04-14T15:24:28.136086+0000","last_fullsized":"2024-04-14T15:24:28.136086+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'24","last_scrub_stamp":"2024-04-14T15:21:13.939121+0000","last_deep_scrub":"85'16","last_deep_scrub_stamp":"2024-04-14T15:10:47.456169+0000","last_clean_scrub_stamp":"2024-04-14T15:21:13.939121+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":1384,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-14T15:24:28.776278+0000","last_change":"2024-04-14T15:23:36.489554+0000","last_active":"2024-04-14T15:24:28.776278+0000","last_peered":"2024-04-14T15:24:28.776278+0000","last_clean":"2024-04-14T15:24:28.776278+0000","last_became_active":"2024-04-14T15:09:43.018601+0000","last_became_peered":"2024-04-14T15:09:43.018601+0000","last_unstale":"2024-04-14T15:24:28.776278+0000","last_undegraded":"2024-04-14T15:24:28.776278+0000","last_fullsized":"2024-04-14T15:24:28.776278+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:36.489475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:23:36.489475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:53.352519+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":932,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-14T15:24:28.138327+0000","last_change":"2024-04-14T15:20:52.974819+0000","last_active":"2024-04-14T15:24:28.138327+0000","last_peered":"2024-04-14T15:24:28.138327+0000","last_clean":"2024-04-14T15:24:28.138327+0000","last_became_active":"2024-04-14T15:20:52.970831+0000","last_became_peered":"2024-04-14T15:20:52.970831+0000","last_unstale":"2024-04-14T15:24:28.138327+0000","last_undegraded":"2024-04-14T15:24:28.138327+0000","last_fullsized":"2024-04-14T15:24:28.138327+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:59.596964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:19:59.596964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1044,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.149120+0000","last_change":"2024-04-14T15:21:17.980121+0000","last_active":"2024-04-14T15:24:29.149120+0000","last_peered":"2024-04-14T15:24:29.149120+0000","last_clean":"2024-04-14T15:24:29.149120+0000","last_became_active":"2024-04-14T15:20:52.999771+0000","last_became_peered":"2024-04-14T15:20:52.999771+0000","last_unstale":"2024-04-14T15:24:29.149120+0000","last_undegraded":"2024-04-14T15:24:29.149120+0000","last_fullsized":"2024-04-14T15:24:29.149120+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:17.980009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:57.214046+0000","last_clean_scrub_stamp":"2024-04-14T15:21:17.980009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"91'130","reported_seq":1509,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.148699+0000","last_change":"2024-04-14T15:23:29.044374+0000","last_active":"2024-04-14T15:24:29.148699+0000","last_peered":"2024-04-14T15:24:29.148699+0000","last_clean":"2024-04-14T15:24:29.148699+0000","last_became_active":"2024-04-14T15:23:29.044065+0000","last_became_peered":"2024-04-14T15:23:29.044065+0000","last_unstale":"2024-04-14T15:24:29.148699+0000","last_undegraded":"2024-04-14T15:24:29.148699+0000","last_fullsized":"2024-04-14T15:24:29.148699+0000","mapping_epoch":189,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":672,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:22:03.138556+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:22:03.138556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":60,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":1273,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-14T15:24:28.135934+0000","last_change":"2024-04-14T15:20:37.765993+0000","last_active":"2024-04-14T15:24:28.135934+0000","last_peered":"2024-04-14T15:24:28.135934+0000","last_clean":"2024-04-14T15:24:28.135934+0000","last_became_active":"2024-04-14T15:12:56.427276+0000","last_became_peered":"2024-04-14T15:12:56.427276+0000","last_unstale":"2024-04-14T15:24:28.135934+0000","last_undegraded":"2024-04-14T15:24:28.135934+0000","last_fullsized":"2024-04-14T15:24:28.135934+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:37.765916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:20:37.765916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"255'10","reported_seq":1197,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.144567+0000","last_change":"2024-04-14T15:23:26.964698+0000","last_active":"2024-04-14T15:24:29.144567+0000","last_peered":"2024-04-14T15:24:29.144567+0000","last_clean":"2024-04-14T15:24:29.144567+0000","last_became_active":"2024-04-14T15:15:21.873747+0000","last_became_peered":"2024-04-14T15:15:21.873747+0000","last_unstale":"2024-04-14T15:24:29.144567+0000","last_undegraded":"2024-04-14T15:24:29.144567+0000","last_fullsized":"2024-04-14T15:24:29.144567+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'10","last_scrub_stamp":"2024-04-14T15:23:26.964616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:23:26.964616+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:29.334703+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"479'28","reported_seq":1437,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-14T15:24:28.134763+0000","last_change":"2024-04-14T15:22:09.093925+0000","last_active":"2024-04-14T15:24:28.134763+0000","last_peered":"2024-04-14T15:24:28.134763+0000","last_clean":"2024-04-14T15:24:28.134763+0000","last_became_active":"2024-04-14T15:20:52.989635+0000","last_became_peered":"2024-04-14T15:20:52.989635+0000","last_unstale":"2024-04-14T15:24:28.134763+0000","last_undegraded":"2024-04-14T15:24:28.134763+0000","last_fullsized":"2024-04-14T15:24:28.134763+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'28","last_scrub_stamp":"2024-04-14T15:22:09.093771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:22:09.093771+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":408,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-14T15:24:28.779206+0000","last_change":"2024-04-14T15:20:52.999728+0000","last_active":"2024-04-14T15:24:28.779206+0000","last_peered":"2024-04-14T15:24:28.779206+0000","last_clean":"2024-04-14T15:24:28.779206+0000","last_became_active":"2024-04-14T15:20:52.999379+0000","last_became_peered":"2024-04-14T15:20:52.999379+0000","last_unstale":"2024-04-14T15:24:28.779206+0000","last_undegraded":"2024-04-14T15:24:28.779206+0000","last_fullsized":"2024-04-14T15:24:28.779206+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:41.799129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:20:41.799129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1371,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-14T15:24:28.778703+0000","last_change":"2024-04-14T15:23:28.571478+0000","last_active":"2024-04-14T15:24:28.778703+0000","last_peered":"2024-04-14T15:24:28.778703+0000","last_clean":"2024-04-14T15:24:28.778703+0000","last_became_active":"2024-04-14T15:20:52.991388+0000","last_became_peered":"2024-04-14T15:20:52.991388+0000","last_unstale":"2024-04-14T15:24:28.778703+0000","last_undegraded":"2024-04-14T15:24:28.778703+0000","last_fullsized":"2024-04-14T15:24:28.778703+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:28.571374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:23:28.571374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:45.653854+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"91'130","reported_seq":1524,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-14T15:24:28.136911+0000","last_change":"2024-04-14T15:23:06.153451+0000","last_active":"2024-04-14T15:24:28.136911+0000","last_peered":"2024-04-14T15:24:28.136911+0000","last_clean":"2024-04-14T15:24:28.136911+0000","last_became_active":"2024-04-14T15:23:02.990846+0000","last_became_peered":"2024-04-14T15:23:02.990846+0000","last_unstale":"2024-04-14T15:24:28.136911+0000","last_undegraded":"2024-04-14T15:24:28.136911+0000","last_fullsized":"2024-04-14T15:24:28.136911+0000","mapping_epoch":530,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":647,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:23:06.153339+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:23:06.153339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:30.763302+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":1281,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-14T15:24:28.778980+0000","last_change":"2024-04-14T15:24:27.475544+0000","last_active":"2024-04-14T15:24:28.778980+0000","last_peered":"2024-04-14T15:24:28.778980+0000","last_clean":"2024-04-14T15:24:28.778980+0000","last_became_active":"2024-04-14T15:12:56.355254+0000","last_became_peered":"2024-04-14T15:12:56.355254+0000","last_unstale":"2024-04-14T15:24:28.778980+0000","last_undegraded":"2024-04-14T15:24:28.778980+0000","last_fullsized":"2024-04-14T15:24:28.778980+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:27.475460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:24:27.475460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:35.815654+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1369,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-14T15:24:28.135376+0000","last_change":"2024-04-14T15:22:55.245731+0000","last_active":"2024-04-14T15:24:28.135376+0000","last_peered":"2024-04-14T15:24:28.135376+0000","last_clean":"2024-04-14T15:24:28.135376+0000","last_became_active":"2024-04-14T15:09:47.044610+0000","last_became_peered":"2024-04-14T15:09:47.044610+0000","last_unstale":"2024-04-14T15:24:28.135376+0000","last_undegraded":"2024-04-14T15:24:28.135376+0000","last_fullsized":"2024-04-14T15:24:28.135376+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:55.245639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:39.394655+0000","last_clean_scrub_stamp":"2024-04-14T15:22:55.245639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"479'30","reported_seq":1105,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.144491+0000","last_change":"2024-04-14T15:22:58.001736+0000","last_active":"2024-04-14T15:24:29.144491+0000","last_peered":"2024-04-14T15:24:29.144491+0000","last_clean":"2024-04-14T15:24:29.144491+0000","last_became_active":"2024-04-14T15:15:21.880565+0000","last_became_peered":"2024-04-14T15:15:21.880565+0000","last_unstale":"2024-04-14T15:24:29.144491+0000","last_undegraded":"2024-04-14T15:24:29.144491+0000","last_fullsized":"2024-04-14T15:24:29.144491+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'30","last_scrub_stamp":"2024-04-14T15:22:58.001609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:22:58.001609+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'2","reported_seq":1385,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.142630+0000","last_change":"2024-04-14T15:21:25.663010+0000","last_active":"2024-04-14T15:24:29.142630+0000","last_peered":"2024-04-14T15:24:29.142630+0000","last_clean":"2024-04-14T15:24:29.142630+0000","last_became_active":"2024-04-14T15:20:52.987576+0000","last_became_peered":"2024-04-14T15:20:52.987576+0000","last_unstale":"2024-04-14T15:24:29.142630+0000","last_undegraded":"2024-04-14T15:24:29.142630+0000","last_fullsized":"2024-04-14T15:24:29.142630+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'2","last_scrub_stamp":"2024-04-14T15:21:25.662901+0000","last_deep_scrub":"91'2","last_deep_scrub_stamp":"2024-04-14T15:17:48.850069+0000","last_clean_scrub_stamp":"2024-04-14T15:21:25.662901+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1365,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-14T15:24:28.136842+0000","last_change":"2024-04-14T15:23:33.834918+0000","last_active":"2024-04-14T15:24:28.136842+0000","last_peered":"2024-04-14T15:24:28.136842+0000","last_clean":"2024-04-14T15:24:28.136842+0000","last_became_active":"2024-04-14T15:09:49.065684+0000","last_became_peered":"2024-04-14T15:09:49.065684+0000","last_unstale":"2024-04-14T15:24:28.136842+0000","last_undegraded":"2024-04-14T15:24:28.136842+0000","last_fullsized":"2024-04-14T15:24:28.136842+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:33.834838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:23:33.834838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:37.110012+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"91'130","reported_seq":796,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.149935+0000","last_change":"2024-04-14T15:23:43.310976+0000","last_active":"2024-04-14T15:24:29.149935+0000","last_peered":"2024-04-14T15:24:29.149935+0000","last_clean":"2024-04-14T15:24:29.149935+0000","last_became_active":"2024-04-14T15:22:58.991118+0000","last_became_peered":"2024-04-14T15:22:58.991118+0000","last_unstale":"2024-04-14T15:24:29.149935+0000","last_undegraded":"2024-04-14T15:24:29.149935+0000","last_fullsized":"2024-04-14T15:24:29.149935+0000","mapping_epoch":530,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:23:43.310892+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:23:43.310892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:49.797838+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":1286,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.142160+0000","last_change":"2024-04-14T15:23:24.298794+0000","last_active":"2024-04-14T15:24:29.142160+0000","last_peered":"2024-04-14T15:24:29.142160+0000","last_clean":"2024-04-14T15:24:29.142160+0000","last_became_active":"2024-04-14T15:12:56.402023+0000","last_became_peered":"2024-04-14T15:12:56.402023+0000","last_unstale":"2024-04-14T15:24:29.142160+0000","last_undegraded":"2024-04-14T15:24:29.142160+0000","last_fullsized":"2024-04-14T15:24:29.142160+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:24.298717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:35.003968+0000","last_clean_scrub_stamp":"2024-04-14T15:23:24.298717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:35.516401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":424,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.147863+0000","last_change":"2024-04-14T15:24:18.368805+0000","last_active":"2024-04-14T15:24:29.147863+0000","last_peered":"2024-04-14T15:24:29.147863+0000","last_clean":"2024-04-14T15:24:29.147863+0000","last_became_active":"2024-04-14T15:20:52.987883+0000","last_became_peered":"2024-04-14T15:20:52.987883+0000","last_unstale":"2024-04-14T15:24:29.147863+0000","last_undegraded":"2024-04-14T15:24:29.147863+0000","last_fullsized":"2024-04-14T15:24:29.147863+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:18.368725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:24:18.368725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:42.909210+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"479'37","reported_seq":1441,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.148408+0000","last_change":"2024-04-14T15:23:38.246620+0000","last_active":"2024-04-14T15:24:29.148408+0000","last_peered":"2024-04-14T15:24:29.148408+0000","last_clean":"2024-04-14T15:24:29.148408+0000","last_became_active":"2024-04-14T15:09:45.179182+0000","last_became_peered":"2024-04-14T15:09:45.179182+0000","last_unstale":"2024-04-14T15:24:29.148408+0000","last_undegraded":"2024-04-14T15:24:29.148408+0000","last_fullsized":"2024-04-14T15:24:29.148408+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'37","last_scrub_stamp":"2024-04-14T15:23:38.246517+0000","last_deep_scrub":"102'27","last_deep_scrub_stamp":"2024-04-14T15:17:46.924330+0000","last_clean_scrub_stamp":"2024-04-14T15:23:38.246517+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:41.818665+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1377,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-14T15:24:28.778243+0000","last_change":"2024-04-14T15:22:56.541983+0000","last_active":"2024-04-14T15:24:28.778243+0000","last_peered":"2024-04-14T15:24:28.778243+0000","last_clean":"2024-04-14T15:24:28.778243+0000","last_became_active":"2024-04-14T15:09:43.021006+0000","last_became_peered":"2024-04-14T15:09:43.021006+0000","last_unstale":"2024-04-14T15:24:28.778243+0000","last_undegraded":"2024-04-14T15:24:28.778243+0000","last_fullsized":"2024-04-14T15:24:28.778243+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:56.541899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:22:56.541899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1376,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.140699+0000","last_change":"2024-04-14T15:23:34.182007+0000","last_active":"2024-04-14T15:24:29.140699+0000","last_peered":"2024-04-14T15:24:29.140699+0000","last_clean":"2024-04-14T15:24:29.140699+0000","last_became_active":"2024-04-14T15:20:52.978765+0000","last_became_peered":"2024-04-14T15:20:52.978765+0000","last_unstale":"2024-04-14T15:24:29.140699+0000","last_undegraded":"2024-04-14T15:24:29.140699+0000","last_fullsized":"2024-04-14T15:24:29.140699+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:34.181914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:56.206649+0000","last_clean_scrub_stamp":"2024-04-14T15:23:34.181914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:48.624030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"91'130","reported_seq":1524,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.141531+0000","last_change":"2024-04-14T15:23:32.988316+0000","last_active":"2024-04-14T15:24:29.141531+0000","last_peered":"2024-04-14T15:24:29.141531+0000","last_clean":"2024-04-14T15:24:29.141531+0000","last_became_active":"2024-04-14T15:23:32.987931+0000","last_became_peered":"2024-04-14T15:23:32.987931+0000","last_unstale":"2024-04-14T15:24:29.141531+0000","last_undegraded":"2024-04-14T15:24:29.141531+0000","last_fullsized":"2024-04-14T15:24:29.141531+0000","mapping_epoch":530,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:22:57.365464+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:22:57.365464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":992,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.145134+0000","last_change":"2024-04-14T15:23:30.968804+0000","last_active":"2024-04-14T15:24:29.145134+0000","last_peered":"2024-04-14T15:24:29.145134+0000","last_clean":"2024-04-14T15:24:29.145134+0000","last_became_active":"2024-04-14T15:20:52.999776+0000","last_became_peered":"2024-04-14T15:20:52.999776+0000","last_unstale":"2024-04-14T15:24:29.145134+0000","last_undegraded":"2024-04-14T15:24:29.145134+0000","last_fullsized":"2024-04-14T15:24:29.145134+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:30.968711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:23:30.968711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:34.065587+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":932,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.143477+0000","last_change":"2024-04-14T15:21:59.095980+0000","last_active":"2024-04-14T15:24:29.143477+0000","last_peered":"2024-04-14T15:24:29.143477+0000","last_clean":"2024-04-14T15:24:29.143477+0000","last_became_active":"2024-04-14T15:15:44.883694+0000","last_became_peered":"2024-04-14T15:15:44.883694+0000","last_unstale":"2024-04-14T15:24:29.143477+0000","last_undegraded":"2024-04-14T15:24:29.143477+0000","last_fullsized":"2024-04-14T15:24:29.143477+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:59.095899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:21:59.095899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"479'30","reported_seq":1423,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.150431+0000","last_change":"2024-04-14T15:23:30.295880+0000","last_active":"2024-04-14T15:24:29.150431+0000","last_peered":"2024-04-14T15:24:29.150431+0000","last_clean":"2024-04-14T15:24:29.150431+0000","last_became_active":"2024-04-14T15:09:45.037231+0000","last_became_peered":"2024-04-14T15:09:45.037231+0000","last_unstale":"2024-04-14T15:24:29.150431+0000","last_undegraded":"2024-04-14T15:24:29.150431+0000","last_fullsized":"2024-04-14T15:24:29.150431+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'30","last_scrub_stamp":"2024-04-14T15:23:30.295747+0000","last_deep_scrub":"182'18","last_deep_scrub_stamp":"2024-04-14T15:14:10.383906+0000","last_clean_scrub_stamp":"2024-04-14T15:23:30.295747+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:49.622040+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1377,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-14T15:24:28.776392+0000","last_change":"2024-04-14T15:20:42.694332+0000","last_active":"2024-04-14T15:24:28.776392+0000","last_peered":"2024-04-14T15:24:28.776392+0000","last_clean":"2024-04-14T15:24:28.776392+0000","last_became_active":"2024-04-14T15:09:43.025095+0000","last_became_peered":"2024-04-14T15:09:43.025095+0000","last_unstale":"2024-04-14T15:24:28.776392+0000","last_undegraded":"2024-04-14T15:24:28.776392+0000","last_fullsized":"2024-04-14T15:24:28.776392+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:42.694243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:42.694243+0000","last_clean_scrub_stamp":"2024-04-14T15:20:42.694243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1393,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.145787+0000","last_change":"2024-04-14T15:23:40.238604+0000","last_active":"2024-04-14T15:24:29.145787+0000","last_peered":"2024-04-14T15:24:29.145787+0000","last_clean":"2024-04-14T15:24:29.145787+0000","last_became_active":"2024-04-14T15:09:43.018509+0000","last_became_peered":"2024-04-14T15:09:43.018509+0000","last_unstale":"2024-04-14T15:24:29.145787+0000","last_undegraded":"2024-04-14T15:24:29.145787+0000","last_fullsized":"2024-04-14T15:24:29.145787+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:40.238527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:23:40.238527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:06.849826+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0018592680000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"355'2","reported_seq":1372,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.142372+0000","last_change":"2024-04-14T15:23:43.059738+0000","last_active":"2024-04-14T15:24:29.142372+0000","last_peered":"2024-04-14T15:24:29.142372+0000","last_clean":"2024-04-14T15:24:29.142372+0000","last_became_active":"2024-04-14T15:09:49.065243+0000","last_became_peered":"2024-04-14T15:09:49.065243+0000","last_unstale":"2024-04-14T15:24:29.142372+0000","last_undegraded":"2024-04-14T15:24:29.142372+0000","last_fullsized":"2024-04-14T15:24:29.142372+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'2","last_scrub_stamp":"2024-04-14T15:23:43.059603+0000","last_deep_scrub":"355'2","last_deep_scrub_stamp":"2024-04-14T15:20:04.842881+0000","last_clean_scrub_stamp":"2024-04-14T15:23:43.059603+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:47.245706+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":984,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.144784+0000","last_change":"2024-04-14T15:24:28.904419+0000","last_active":"2024-04-14T15:24:29.144784+0000","last_peered":"2024-04-14T15:24:29.144784+0000","last_clean":"2024-04-14T15:24:29.144784+0000","last_became_active":"2024-04-14T15:20:53.000058+0000","last_became_peered":"2024-04-14T15:20:53.000058+0000","last_unstale":"2024-04-14T15:24:29.144784+0000","last_undegraded":"2024-04-14T15:24:29.144784+0000","last_fullsized":"2024-04-14T15:24:29.144784+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:28.904336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:24:28.904336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:29.518613+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1369,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-14T15:24:28.778658+0000","last_change":"2024-04-14T15:23:05.574296+0000","last_active":"2024-04-14T15:24:28.778658+0000","last_peered":"2024-04-14T15:24:28.778658+0000","last_clean":"2024-04-14T15:24:28.778658+0000","last_became_active":"2024-04-14T15:09:47.047201+0000","last_became_peered":"2024-04-14T15:09:47.047201+0000","last_unstale":"2024-04-14T15:24:28.778658+0000","last_undegraded":"2024-04-14T15:24:28.778658+0000","last_fullsized":"2024-04-14T15:24:28.778658+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:05.574212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:23:05.574212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"528'37","reported_seq":1516,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-14T15:24:28.138319+0000","last_change":"2024-04-14T15:23:29.977939+0000","last_active":"2024-04-14T15:24:28.138319+0000","last_peered":"2024-04-14T15:24:28.138319+0000","last_clean":"2024-04-14T15:24:28.138319+0000","last_became_active":"2024-04-14T15:20:53.002282+0000","last_became_peered":"2024-04-14T15:20:53.002282+0000","last_unstale":"2024-04-14T15:24:28.138319+0000","last_undegraded":"2024-04-14T15:24:28.138319+0000","last_fullsized":"2024-04-14T15:24:28.138319+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"528'37","last_scrub_stamp":"2024-04-14T15:23:29.977844+0000","last_deep_scrub":"90'24","last_deep_scrub_stamp":"2024-04-14T15:11:25.120692+0000","last_clean_scrub_stamp":"2024-04-14T15:23:29.977844+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:32.485938+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"91'130","reported_seq":1721,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-14T15:24:28.776481+0000","last_change":"2024-04-14T15:22:19.518674+0000","last_active":"2024-04-14T15:24:28.776481+0000","last_peered":"2024-04-14T15:24:28.776481+0000","last_clean":"2024-04-14T15:24:28.776481+0000","last_became_active":"2024-04-14T15:22:14.980657+0000","last_became_peered":"2024-04-14T15:22:14.980657+0000","last_unstale":"2024-04-14T15:24:28.776481+0000","last_undegraded":"2024-04-14T15:24:28.776481+0000","last_fullsized":"2024-04-14T15:24:28.776481+0000","mapping_epoch":530,"log_start":"91'130","ondisk_log_start":"91'130","created":19,"last_epoch_clean":599,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:22:19.518583+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:22:19.518583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":942,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.154259+0000","last_change":"2024-04-14T15:23:29.691160+0000","last_active":"2024-04-14T15:24:29.154259+0000","last_peered":"2024-04-14T15:24:29.154259+0000","last_clean":"2024-04-14T15:24:29.154259+0000","last_became_active":"2024-04-14T15:15:44.885315+0000","last_became_peered":"2024-04-14T15:15:44.885315+0000","last_unstale":"2024-04-14T15:24:29.154259+0000","last_undegraded":"2024-04-14T15:24:29.154259+0000","last_fullsized":"2024-04-14T15:24:29.154259+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:29.691081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:47.627305+0000","last_clean_scrub_stamp":"2024-04-14T15:23:29.691081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:36.414447+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"69'2","reported_seq":1078,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.144868+0000","last_change":"2024-04-14T15:23:44.863113+0000","last_active":"2024-04-14T15:24:29.144868+0000","last_peered":"2024-04-14T15:24:29.144868+0000","last_clean":"2024-04-14T15:24:29.144868+0000","last_became_active":"2024-04-14T15:20:53.002545+0000","last_became_peered":"2024-04-14T15:20:53.002545+0000","last_unstale":"2024-04-14T15:24:29.144868+0000","last_undegraded":"2024-04-14T15:24:29.144868+0000","last_fullsized":"2024-04-14T15:24:29.144868+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-04-14T15:23:44.862962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:23:44.862962+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:47.817158+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0058046169999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":933,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.154174+0000","last_change":"2024-04-14T15:23:33.934414+0000","last_active":"2024-04-14T15:24:29.154174+0000","last_peered":"2024-04-14T15:24:29.154174+0000","last_clean":"2024-04-14T15:24:29.154174+0000","last_became_active":"2024-04-14T15:15:44.887934+0000","last_became_peered":"2024-04-14T15:15:44.887934+0000","last_unstale":"2024-04-14T15:24:29.154174+0000","last_undegraded":"2024-04-14T15:24:29.154174+0000","last_fullsized":"2024-04-14T15:24:29.154174+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:33.934334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:23:33.934334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:56.348377+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"91'130","reported_seq":1721,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-14T15:24:28.776770+0000","last_change":"2024-04-14T15:22:17.518773+0000","last_active":"2024-04-14T15:24:28.776770+0000","last_peered":"2024-04-14T15:24:28.776770+0000","last_clean":"2024-04-14T15:24:28.776770+0000","last_became_active":"2024-04-14T15:22:06.969269+0000","last_became_peered":"2024-04-14T15:22:06.969269+0000","last_unstale":"2024-04-14T15:24:28.776770+0000","last_undegraded":"2024-04-14T15:24:28.776770+0000","last_fullsized":"2024-04-14T15:24:28.776770+0000","mapping_epoch":189,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:22:17.518666+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:22:17.518666+0000","last_clean_scrub_stamp":"2024-04-14T15:22:17.518666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":984,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.144120+0000","last_change":"2024-04-14T15:23:03.091708+0000","last_active":"2024-04-14T15:24:29.144120+0000","last_peered":"2024-04-14T15:24:29.144120+0000","last_clean":"2024-04-14T15:24:29.144120+0000","last_became_active":"2024-04-14T15:20:52.986308+0000","last_became_peered":"2024-04-14T15:20:52.986308+0000","last_unstale":"2024-04-14T15:24:29.144120+0000","last_undegraded":"2024-04-14T15:24:29.144120+0000","last_fullsized":"2024-04-14T15:24:29.144120+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:03.091585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:23:03.091585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1375,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.157554+0000","last_change":"2024-04-14T15:23:05.676756+0000","last_active":"2024-04-14T15:24:29.157554+0000","last_peered":"2024-04-14T15:24:29.157554+0000","last_clean":"2024-04-14T15:24:29.157554+0000","last_became_active":"2024-04-14T15:09:47.049457+0000","last_became_peered":"2024-04-14T15:09:47.049457+0000","last_unstale":"2024-04-14T15:24:29.157554+0000","last_undegraded":"2024-04-14T15:24:29.157554+0000","last_fullsized":"2024-04-14T15:24:29.157554+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:05.676677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:23:05.676677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"479'22","reported_seq":1415,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.143365+0000","last_change":"2024-04-14T15:23:05.309013+0000","last_active":"2024-04-14T15:24:29.143365+0000","last_peered":"2024-04-14T15:24:29.143365+0000","last_clean":"2024-04-14T15:24:29.143365+0000","last_became_active":"2024-04-14T15:09:45.033883+0000","last_became_peered":"2024-04-14T15:09:45.033883+0000","last_unstale":"2024-04-14T15:24:29.143365+0000","last_undegraded":"2024-04-14T15:24:29.143365+0000","last_fullsized":"2024-04-14T15:24:29.143365+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'22","last_scrub_stamp":"2024-04-14T15:23:05.308902+0000","last_deep_scrub":"308'18","last_deep_scrub_stamp":"2024-04-14T15:16:16.054032+0000","last_clean_scrub_stamp":"2024-04-14T15:23:05.308902+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:32.912046+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1384,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.155465+0000","last_change":"2024-04-14T15:22:08.870838+0000","last_active":"2024-04-14T15:24:29.155465+0000","last_peered":"2024-04-14T15:24:29.155465+0000","last_clean":"2024-04-14T15:24:29.155465+0000","last_became_active":"2024-04-14T15:09:43.021188+0000","last_became_peered":"2024-04-14T15:09:43.021188+0000","last_unstale":"2024-04-14T15:24:29.155465+0000","last_undegraded":"2024-04-14T15:24:29.155465+0000","last_fullsized":"2024-04-14T15:24:29.155465+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:08.870757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:22:08.870757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"73'5","reported_seq":2280,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.140681+0000","last_change":"2024-04-14T15:22:09.278270+0000","last_active":"2024-04-14T15:24:29.140681+0000","last_peered":"2024-04-14T15:24:29.140681+0000","last_clean":"2024-04-14T15:24:29.140681+0000","last_became_active":"2024-04-14T15:20:52.975473+0000","last_became_peered":"2024-04-14T15:20:52.975473+0000","last_unstale":"2024-04-14T15:24:29.140681+0000","last_undegraded":"2024-04-14T15:24:29.140681+0000","last_fullsized":"2024-04-14T15:24:29.140681+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-04-14T15:22:09.278172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:22:09.278172+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00056542099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":877,"num_read_kb":872,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1371,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-14T15:24:28.778497+0000","last_change":"2024-04-14T15:23:01.536017+0000","last_active":"2024-04-14T15:24:28.778497+0000","last_peered":"2024-04-14T15:24:28.778497+0000","last_clean":"2024-04-14T15:24:28.778497+0000","last_became_active":"2024-04-14T15:20:52.999464+0000","last_became_peered":"2024-04-14T15:20:52.999464+0000","last_unstale":"2024-04-14T15:24:28.778497+0000","last_undegraded":"2024-04-14T15:24:28.778497+0000","last_fullsized":"2024-04-14T15:24:28.778497+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:01.535941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:20.827589+0000","last_clean_scrub_stamp":"2024-04-14T15:23:01.535941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"91'130","reported_seq":1343,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.145617+0000","last_change":"2024-04-14T15:24:23.372384+0000","last_active":"2024-04-14T15:24:29.145617+0000","last_peered":"2024-04-14T15:24:29.145617+0000","last_clean":"2024-04-14T15:24:29.145617+0000","last_became_active":"2024-04-14T15:22:10.975960+0000","last_became_peered":"2024-04-14T15:22:10.975960+0000","last_unstale":"2024-04-14T15:24:29.145617+0000","last_undegraded":"2024-04-14T15:24:29.145617+0000","last_fullsized":"2024-04-14T15:24:29.145617+0000","mapping_epoch":276,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":595,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:24:23.372311+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:24:23.372311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:38.263904+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":978,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.146155+0000","last_change":"2024-04-14T15:22:53.975763+0000","last_active":"2024-04-14T15:24:29.146155+0000","last_peered":"2024-04-14T15:24:29.146155+0000","last_clean":"2024-04-14T15:24:29.146155+0000","last_became_active":"2024-04-14T15:15:21.883717+0000","last_became_peered":"2024-04-14T15:15:21.883717+0000","last_unstale":"2024-04-14T15:24:29.146155+0000","last_undegraded":"2024-04-14T15:24:29.146155+0000","last_fullsized":"2024-04-14T15:24:29.146155+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:53.975656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:22:53.975656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1378,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.150741+0000","last_change":"2024-04-14T15:24:26.376324+0000","last_active":"2024-04-14T15:24:29.150741+0000","last_peered":"2024-04-14T15:24:29.150741+0000","last_clean":"2024-04-14T15:24:29.150741+0000","last_became_active":"2024-04-14T15:09:47.040185+0000","last_became_peered":"2024-04-14T15:09:47.040185+0000","last_unstale":"2024-04-14T15:24:29.150741+0000","last_undegraded":"2024-04-14T15:24:29.150741+0000","last_fullsized":"2024-04-14T15:24:29.150741+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:26.376179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:53.731610+0000","last_clean_scrub_stamp":"2024-04-14T15:24:26.376179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:37.687634+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"479'40","reported_seq":1622,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.142046+0000","last_change":"2024-04-14T15:23:31.198816+0000","last_active":"2024-04-14T15:24:29.142046+0000","last_peered":"2024-04-14T15:24:29.142046+0000","last_clean":"2024-04-14T15:24:29.142046+0000","last_became_active":"2024-04-14T15:09:45.034168+0000","last_became_peered":"2024-04-14T15:09:45.034168+0000","last_unstale":"2024-04-14T15:24:29.142046+0000","last_undegraded":"2024-04-14T15:24:29.142046+0000","last_fullsized":"2024-04-14T15:24:29.142046+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'40","last_scrub_stamp":"2024-04-14T15:23:31.198708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:23:31.198708+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:42.421703+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":946,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.141114+0000","last_change":"2024-04-14T15:23:30.217883+0000","last_active":"2024-04-14T15:24:29.141114+0000","last_peered":"2024-04-14T15:24:29.141114+0000","last_clean":"2024-04-14T15:24:29.141114+0000","last_became_active":"2024-04-14T15:20:52.979943+0000","last_became_peered":"2024-04-14T15:20:52.979943+0000","last_unstale":"2024-04-14T15:24:29.141114+0000","last_undegraded":"2024-04-14T15:24:29.141114+0000","last_fullsized":"2024-04-14T15:24:29.141114+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:30.217808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:24.066275+0000","last_clean_scrub_stamp":"2024-04-14T15:23:30.217808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:51.962682+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1030,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.153229+0000","last_change":"2024-04-14T15:21:21.103290+0000","last_active":"2024-04-14T15:24:29.153229+0000","last_peered":"2024-04-14T15:24:29.153229+0000","last_clean":"2024-04-14T15:24:29.153229+0000","last_became_active":"2024-04-14T15:15:21.874742+0000","last_became_peered":"2024-04-14T15:15:21.874742+0000","last_unstale":"2024-04-14T15:24:29.153229+0000","last_undegraded":"2024-04-14T15:24:29.153229+0000","last_fullsized":"2024-04-14T15:24:29.153229+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:21.103213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:44.084010+0000","last_clean_scrub_stamp":"2024-04-14T15:21:21.103213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"91'130","reported_seq":799,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.153496+0000","last_change":"2024-04-14T15:24:23.862421+0000","last_active":"2024-04-14T15:24:29.153496+0000","last_peered":"2024-04-14T15:24:29.153496+0000","last_clean":"2024-04-14T15:24:29.153496+0000","last_became_active":"2024-04-14T15:22:18.993883+0000","last_became_peered":"2024-04-14T15:22:18.993883+0000","last_unstale":"2024-04-14T15:24:29.153496+0000","last_undegraded":"2024-04-14T15:24:29.153496+0000","last_fullsized":"2024-04-14T15:24:29.153496+0000","mapping_epoch":530,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:24:23.862316+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:24:23.862316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:26.896043+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":1286,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.149059+0000","last_change":"2024-04-14T15:22:51.765619+0000","last_active":"2024-04-14T15:24:29.149059+0000","last_peered":"2024-04-14T15:24:29.149059+0000","last_clean":"2024-04-14T15:24:29.149059+0000","last_became_active":"2024-04-14T15:20:52.989272+0000","last_became_peered":"2024-04-14T15:20:52.989272+0000","last_unstale":"2024-04-14T15:24:29.149059+0000","last_undegraded":"2024-04-14T15:24:29.149059+0000","last_fullsized":"2024-04-14T15:24:29.149059+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:51.765499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:22:51.765499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"528'9","reported_seq":1574,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.148119+0000","last_change":"2024-04-14T15:22:53.253431+0000","last_active":"2024-04-14T15:24:29.148119+0000","last_peered":"2024-04-14T15:24:29.148119+0000","last_clean":"2024-04-14T15:24:29.148119+0000","last_became_active":"2024-04-14T15:20:53.000669+0000","last_became_peered":"2024-04-14T15:20:53.000669+0000","last_unstale":"2024-04-14T15:24:29.148119+0000","last_undegraded":"2024-04-14T15:24:29.148119+0000","last_fullsized":"2024-04-14T15:24:29.148119+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"528'9","last_scrub_stamp":"2024-04-14T15:22:53.253302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:22:53.253302+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"479'20","reported_seq":384,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.140918+0000","last_change":"2024-04-14T15:22:13.333897+0000","last_active":"2024-04-14T15:24:29.140918+0000","last_peered":"2024-04-14T15:24:29.140918+0000","last_clean":"2024-04-14T15:24:29.140918+0000","last_became_active":"2024-04-14T15:21:16.034027+0000","last_became_peered":"2024-04-14T15:21:16.034027+0000","last_unstale":"2024-04-14T15:24:29.140918+0000","last_undegraded":"2024-04-14T15:24:29.140918+0000","last_fullsized":"2024-04-14T15:24:29.140918+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'20","last_scrub_stamp":"2024-04-14T15:22:13.333807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:22:13.333807+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1374,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.142050+0000","last_change":"2024-04-14T15:21:28.684960+0000","last_active":"2024-04-14T15:24:29.142050+0000","last_peered":"2024-04-14T15:24:29.142050+0000","last_clean":"2024-04-14T15:24:29.142050+0000","last_became_active":"2024-04-14T15:09:43.022597+0000","last_became_peered":"2024-04-14T15:09:43.022597+0000","last_unstale":"2024-04-14T15:24:29.142050+0000","last_undegraded":"2024-04-14T15:24:29.142050+0000","last_fullsized":"2024-04-14T15:24:29.142050+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:28.684818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:21:28.684818+0000","last_clean_scrub_stamp":"2024-04-14T15:21:28.684818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1374,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.146494+0000","last_change":"2024-04-14T15:24:25.330276+0000","last_active":"2024-04-14T15:24:29.146494+0000","last_peered":"2024-04-14T15:24:29.146494+0000","last_clean":"2024-04-14T15:24:29.146494+0000","last_became_active":"2024-04-14T15:09:49.066648+0000","last_became_peered":"2024-04-14T15:09:49.066648+0000","last_unstale":"2024-04-14T15:24:29.146494+0000","last_undegraded":"2024-04-14T15:24:29.146494+0000","last_fullsized":"2024-04-14T15:24:29.146494+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:25.330148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:24:25.330148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:36.679635+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":1282,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.149871+0000","last_change":"2024-04-14T15:23:44.358966+0000","last_active":"2024-04-14T15:24:29.149871+0000","last_peered":"2024-04-14T15:24:29.149871+0000","last_clean":"2024-04-14T15:24:29.149871+0000","last_became_active":"2024-04-14T15:12:56.563074+0000","last_became_peered":"2024-04-14T15:12:56.563074+0000","last_unstale":"2024-04-14T15:24:29.149871+0000","last_undegraded":"2024-04-14T15:24:29.149871+0000","last_fullsized":"2024-04-14T15:24:29.149871+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:44.358887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:23:44.358887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:52.408476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1375,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.153459+0000","last_change":"2024-04-14T15:24:22.536720+0000","last_active":"2024-04-14T15:24:29.153459+0000","last_peered":"2024-04-14T15:24:29.153459+0000","last_clean":"2024-04-14T15:24:29.153459+0000","last_became_active":"2024-04-14T15:09:47.049005+0000","last_became_peered":"2024-04-14T15:09:47.049005+0000","last_unstale":"2024-04-14T15:24:29.153459+0000","last_undegraded":"2024-04-14T15:24:29.153459+0000","last_fullsized":"2024-04-14T15:24:29.153459+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:22.536562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:56.915544+0000","last_clean_scrub_stamp":"2024-04-14T15:24:22.536562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:25.774677+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"479'23","reported_seq":1417,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-14T15:24:28.394299+0000","last_change":"2024-04-14T15:24:28.394342+0000","last_active":"2024-04-14T15:24:28.394299+0000","last_peered":"2024-04-14T15:24:28.394299+0000","last_clean":"2024-04-14T15:24:28.394299+0000","last_became_active":"2024-04-14T15:09:45.032892+0000","last_became_peered":"2024-04-14T15:09:45.032892+0000","last_unstale":"2024-04-14T15:24:28.394299+0000","last_undegraded":"2024-04-14T15:24:28.394299+0000","last_fullsized":"2024-04-14T15:24:28.394299+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'23","last_scrub_stamp":"2024-04-14T15:24:28.394222+0000","last_deep_scrub":"183'17","last_deep_scrub_stamp":"2024-04-14T15:14:11.577574+0000","last_clean_scrub_stamp":"2024-04-14T15:24:28.394222+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:55.253909+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"677'6","reported_seq":1397,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-14T15:24:28.137882+0000","last_change":"2024-04-14T15:23:38.773659+0000","last_active":"2024-04-14T15:24:28.137882+0000","last_peered":"2024-04-14T15:24:28.137882+0000","last_clean":"2024-04-14T15:24:28.137882+0000","last_became_active":"2024-04-14T15:09:43.020490+0000","last_became_peered":"2024-04-14T15:09:43.020490+0000","last_unstale":"2024-04-14T15:24:28.137882+0000","last_undegraded":"2024-04-14T15:24:28.137882+0000","last_fullsized":"2024-04-14T15:24:28.137882+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"677'6","last_scrub_stamp":"2024-04-14T15:23:38.773539+0000","last_deep_scrub":"677'6","last_deep_scrub_stamp":"2024-04-14T15:23:38.773539+0000","last_clean_scrub_stamp":"2024-04-14T15:23:38.773539+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:04.801057+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":528,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1038,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.146542+0000","last_change":"2024-04-14T15:23:40.854331+0000","last_active":"2024-04-14T15:24:29.146542+0000","last_peered":"2024-04-14T15:24:29.146542+0000","last_clean":"2024-04-14T15:24:29.146542+0000","last_became_active":"2024-04-14T15:15:21.876410+0000","last_became_peered":"2024-04-14T15:15:21.876410+0000","last_unstale":"2024-04-14T15:24:29.146542+0000","last_undegraded":"2024-04-14T15:24:29.146542+0000","last_fullsized":"2024-04-14T15:24:29.146542+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:40.854253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:23:40.854253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:10.566240+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":1289,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.145603+0000","last_change":"2024-04-14T15:24:27.371452+0000","last_active":"2024-04-14T15:24:29.145603+0000","last_peered":"2024-04-14T15:24:29.145603+0000","last_clean":"2024-04-14T15:24:29.145603+0000","last_became_active":"2024-04-14T15:12:56.551177+0000","last_became_peered":"2024-04-14T15:12:56.551177+0000","last_unstale":"2024-04-14T15:24:29.145603+0000","last_undegraded":"2024-04-14T15:24:29.145603+0000","last_fullsized":"2024-04-14T15:24:29.145603+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:27.371370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:22:18.299330+0000","last_clean_scrub_stamp":"2024-04-14T15:24:27.371370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:34.049052+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"479'19","reported_seq":1402,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.150076+0000","last_change":"2024-04-14T15:22:55.321370+0000","last_active":"2024-04-14T15:24:29.150076+0000","last_peered":"2024-04-14T15:24:29.150076+0000","last_clean":"2024-04-14T15:24:29.150076+0000","last_became_active":"2024-04-14T15:09:45.180375+0000","last_became_peered":"2024-04-14T15:09:45.180375+0000","last_unstale":"2024-04-14T15:24:29.150076+0000","last_undegraded":"2024-04-14T15:24:29.150076+0000","last_fullsized":"2024-04-14T15:24:29.150076+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'19","last_scrub_stamp":"2024-04-14T15:22:55.321247+0000","last_deep_scrub":"347'13","last_deep_scrub_stamp":"2024-04-14T15:16:55.694599+0000","last_clean_scrub_stamp":"2024-04-14T15:22:55.321247+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":1383,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.153326+0000","last_change":"2024-04-14T15:22:16.961934+0000","last_active":"2024-04-14T15:24:29.153326+0000","last_peered":"2024-04-14T15:24:29.153326+0000","last_clean":"2024-04-14T15:24:29.153326+0000","last_became_active":"2024-04-14T15:09:43.016788+0000","last_became_peered":"2024-04-14T15:09:43.016788+0000","last_unstale":"2024-04-14T15:24:29.153326+0000","last_undegraded":"2024-04-14T15:24:29.153326+0000","last_fullsized":"2024-04-14T15:24:29.153326+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:16.961846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:22:16.961846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1049,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.154087+0000","last_change":"2024-04-14T15:21:29.095634+0000","last_active":"2024-04-14T15:24:29.154087+0000","last_peered":"2024-04-14T15:24:29.154087+0000","last_clean":"2024-04-14T15:24:29.154087+0000","last_became_active":"2024-04-14T15:15:21.873580+0000","last_became_peered":"2024-04-14T15:15:21.873580+0000","last_unstale":"2024-04-14T15:24:29.154087+0000","last_undegraded":"2024-04-14T15:24:29.154087+0000","last_fullsized":"2024-04-14T15:24:29.154087+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:29.095557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:21:29.095557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1371,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.147816+0000","last_change":"2024-04-14T15:22:12.923356+0000","last_active":"2024-04-14T15:24:29.147816+0000","last_peered":"2024-04-14T15:24:29.147816+0000","last_clean":"2024-04-14T15:24:29.147816+0000","last_became_active":"2024-04-14T15:09:49.064296+0000","last_became_peered":"2024-04-14T15:09:49.064296+0000","last_unstale":"2024-04-14T15:24:29.147816+0000","last_undegraded":"2024-04-14T15:24:29.147816+0000","last_fullsized":"2024-04-14T15:24:29.147816+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:12.923270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:22:12.923270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":1285,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.145908+0000","last_change":"2024-04-14T15:23:32.648604+0000","last_active":"2024-04-14T15:24:29.145908+0000","last_peered":"2024-04-14T15:24:29.145908+0000","last_clean":"2024-04-14T15:24:29.145908+0000","last_became_active":"2024-04-14T15:20:52.977894+0000","last_became_peered":"2024-04-14T15:20:52.977894+0000","last_unstale":"2024-04-14T15:24:29.145908+0000","last_undegraded":"2024-04-14T15:24:29.145908+0000","last_fullsized":"2024-04-14T15:24:29.145908+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:32.648522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:23:32.648522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:53.725757+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.17","version":"479'12","reported_seq":1395,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.153976+0000","last_change":"2024-04-14T15:24:16.798489+0000","last_active":"2024-04-14T15:24:29.153976+0000","last_peered":"2024-04-14T15:24:29.153976+0000","last_clean":"2024-04-14T15:24:29.153976+0000","last_became_active":"2024-04-14T15:09:45.180037+0000","last_became_peered":"2024-04-14T15:09:45.180037+0000","last_unstale":"2024-04-14T15:24:29.153976+0000","last_undegraded":"2024-04-14T15:24:29.153976+0000","last_fullsized":"2024-04-14T15:24:29.153976+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'12","last_scrub_stamp":"2024-04-14T15:24:16.798377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:24:16.798377+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:35.888607+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":416,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-14T15:24:28.134937+0000","last_change":"2024-04-14T15:23:40.765638+0000","last_active":"2024-04-14T15:24:28.134937+0000","last_peered":"2024-04-14T15:24:28.134937+0000","last_clean":"2024-04-14T15:24:28.134937+0000","last_became_active":"2024-04-14T15:20:52.986974+0000","last_became_peered":"2024-04-14T15:20:52.986974+0000","last_unstale":"2024-04-14T15:24:28.134937+0000","last_undegraded":"2024-04-14T15:24:28.134937+0000","last_fullsized":"2024-04-14T15:24:28.134937+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:40.765537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:59.640565+0000","last_clean_scrub_stamp":"2024-04-14T15:23:40.765537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:06.708615+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":931,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.152205+0000","last_change":"2024-04-14T15:24:18.710723+0000","last_active":"2024-04-14T15:24:29.152205+0000","last_peered":"2024-04-14T15:24:29.152205+0000","last_clean":"2024-04-14T15:24:29.152205+0000","last_became_active":"2024-04-14T15:15:44.889786+0000","last_became_peered":"2024-04-14T15:15:44.889786+0000","last_unstale":"2024-04-14T15:24:29.152205+0000","last_undegraded":"2024-04-14T15:24:29.152205+0000","last_fullsized":"2024-04-14T15:24:29.152205+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:18.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:13.735024+0000","last_clean_scrub_stamp":"2024-04-14T15:24:18.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:37.843986+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1370,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-14T15:24:28.778906+0000","last_change":"2024-04-14T15:23:29.527743+0000","last_active":"2024-04-14T15:24:28.778906+0000","last_peered":"2024-04-14T15:24:28.778906+0000","last_clean":"2024-04-14T15:24:28.778906+0000","last_became_active":"2024-04-14T15:20:52.985136+0000","last_became_peered":"2024-04-14T15:20:52.985136+0000","last_unstale":"2024-04-14T15:24:28.778906+0000","last_undegraded":"2024-04-14T15:24:28.778906+0000","last_fullsized":"2024-04-14T15:24:28.778906+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:29.527656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:23:29.527656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:53.193549+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":1280,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-14T15:24:28.135630+0000","last_change":"2024-04-14T15:22:53.207244+0000","last_active":"2024-04-14T15:24:28.135630+0000","last_peered":"2024-04-14T15:24:28.135630+0000","last_clean":"2024-04-14T15:24:28.135630+0000","last_became_active":"2024-04-14T15:12:56.203577+0000","last_became_peered":"2024-04-14T15:12:56.203577+0000","last_unstale":"2024-04-14T15:24:28.135630+0000","last_undegraded":"2024-04-14T15:24:28.135630+0000","last_fullsized":"2024-04-14T15:24:28.135630+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:53.207140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:46.369231+0000","last_clean_scrub_stamp":"2024-04-14T15:22:53.207140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"479'20","reported_seq":1083,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.149261+0000","last_change":"2024-04-14T15:22:01.194496+0000","last_active":"2024-04-14T15:24:29.149261+0000","last_peered":"2024-04-14T15:24:29.149261+0000","last_clean":"2024-04-14T15:24:29.149261+0000","last_became_active":"2024-04-14T15:15:21.877667+0000","last_became_peered":"2024-04-14T15:15:21.877667+0000","last_unstale":"2024-04-14T15:24:29.149261+0000","last_undegraded":"2024-04-14T15:24:29.149261+0000","last_fullsized":"2024-04-14T15:24:29.149261+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'20","last_scrub_stamp":"2024-04-14T15:22:01.194418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:22:01.194418+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":931,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.145147+0000","last_change":"2024-04-14T15:23:32.000531+0000","last_active":"2024-04-14T15:24:29.145147+0000","last_peered":"2024-04-14T15:24:29.145147+0000","last_clean":"2024-04-14T15:24:29.145147+0000","last_became_active":"2024-04-14T15:15:44.887452+0000","last_became_peered":"2024-04-14T15:15:44.887452+0000","last_unstale":"2024-04-14T15:24:29.145147+0000","last_undegraded":"2024-04-14T15:24:29.145147+0000","last_fullsized":"2024-04-14T15:24:29.145147+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:32.000401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:23:32.000401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:38.654484+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"280'9","reported_seq":1061,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.142677+0000","last_change":"2024-04-14T15:22:01.181123+0000","last_active":"2024-04-14T15:24:29.142677+0000","last_peered":"2024-04-14T15:24:29.142677+0000","last_clean":"2024-04-14T15:24:29.142677+0000","last_became_active":"2024-04-14T15:15:44.882916+0000","last_became_peered":"2024-04-14T15:15:44.882916+0000","last_unstale":"2024-04-14T15:24:29.142677+0000","last_undegraded":"2024-04-14T15:24:29.142677+0000","last_fullsized":"2024-04-14T15:24:29.142677+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"280'9","last_scrub_stamp":"2024-04-14T15:22:01.181035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:22:01.181035+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1365,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.149967+0000","last_change":"2024-04-14T15:23:29.304889+0000","last_active":"2024-04-14T15:24:29.149967+0000","last_peered":"2024-04-14T15:24:29.149967+0000","last_clean":"2024-04-14T15:24:29.149967+0000","last_became_active":"2024-04-14T15:09:49.066068+0000","last_became_peered":"2024-04-14T15:09:49.066068+0000","last_unstale":"2024-04-14T15:24:29.149967+0000","last_undegraded":"2024-04-14T15:24:29.149967+0000","last_fullsized":"2024-04-14T15:24:29.149967+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:29.304812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:54.724681+0000","last_clean_scrub_stamp":"2024-04-14T15:23:29.304812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:39.703795+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":931,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.144649+0000","last_change":"2024-04-14T15:24:26.878660+0000","last_active":"2024-04-14T15:24:29.144649+0000","last_peered":"2024-04-14T15:24:29.144649+0000","last_clean":"2024-04-14T15:24:29.144649+0000","last_became_active":"2024-04-14T15:15:44.889106+0000","last_became_peered":"2024-04-14T15:15:44.889106+0000","last_unstale":"2024-04-14T15:24:29.144649+0000","last_undegraded":"2024-04-14T15:24:29.144649+0000","last_fullsized":"2024-04-14T15:24:29.144649+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:26.878584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:24:26.878584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:35.591554+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"479'19","reported_seq":1238,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-14T15:24:28.778632+0000","last_change":"2024-04-14T15:23:46.548415+0000","last_active":"2024-04-14T15:24:28.778632+0000","last_peered":"2024-04-14T15:24:28.778632+0000","last_clean":"2024-04-14T15:24:28.778632+0000","last_became_active":"2024-04-14T15:22:25.971104+0000","last_became_peered":"2024-04-14T15:22:25.971104+0000","last_unstale":"2024-04-14T15:24:28.778632+0000","last_undegraded":"2024-04-14T15:24:28.778632+0000","last_fullsized":"2024-04-14T15:24:28.778632+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'19","last_scrub_stamp":"2024-04-14T15:23:46.548255+0000","last_deep_scrub":"183'13","last_deep_scrub_stamp":"2024-04-14T15:14:11.755252+0000","last_clean_scrub_stamp":"2024-04-14T15:23:46.548255+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:55.797274+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1385,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.148318+0000","last_change":"2024-04-14T15:23:28.265599+0000","last_active":"2024-04-14T15:24:29.148318+0000","last_peered":"2024-04-14T15:24:29.148318+0000","last_clean":"2024-04-14T15:24:29.148318+0000","last_became_active":"2024-04-14T15:09:43.019301+0000","last_became_peered":"2024-04-14T15:09:43.019301+0000","last_unstale":"2024-04-14T15:24:29.148318+0000","last_undegraded":"2024-04-14T15:24:29.148318+0000","last_fullsized":"2024-04-14T15:24:29.148318+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:28.265471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:23:28.265471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"255'10","reported_seq":1198,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.150279+0000","last_change":"2024-04-14T15:23:04.116217+0000","last_active":"2024-04-14T15:24:29.150279+0000","last_peered":"2024-04-14T15:24:29.150279+0000","last_clean":"2024-04-14T15:24:29.150279+0000","last_became_active":"2024-04-14T15:15:21.884468+0000","last_became_peered":"2024-04-14T15:15:21.884468+0000","last_unstale":"2024-04-14T15:24:29.150279+0000","last_undegraded":"2024-04-14T15:24:29.150279+0000","last_fullsized":"2024-04-14T15:24:29.150279+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'10","last_scrub_stamp":"2024-04-14T15:23:04.116126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:23:04.116126+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1364,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-14T15:24:28.138034+0000","last_change":"2024-04-14T15:24:23.473852+0000","last_active":"2024-04-14T15:24:28.138034+0000","last_peered":"2024-04-14T15:24:28.138034+0000","last_clean":"2024-04-14T15:24:28.138034+0000","last_became_active":"2024-04-14T15:09:49.065242+0000","last_became_peered":"2024-04-14T15:09:49.065242+0000","last_unstale":"2024-04-14T15:24:28.138034+0000","last_undegraded":"2024-04-14T15:24:28.138034+0000","last_fullsized":"2024-04-14T15:24:28.138034+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:23.473749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:24:23.473749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:46.923229+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":1272,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-14T15:24:28.779076+0000","last_change":"2024-04-14T15:21:16.642973+0000","last_active":"2024-04-14T15:24:28.779076+0000","last_peered":"2024-04-14T15:24:28.779076+0000","last_clean":"2024-04-14T15:24:28.779076+0000","last_became_active":"2024-04-14T15:12:56.389204+0000","last_became_peered":"2024-04-14T15:12:56.389204+0000","last_unstale":"2024-04-14T15:24:28.779076+0000","last_undegraded":"2024-04-14T15:24:28.779076+0000","last_fullsized":"2024-04-14T15:24:28.779076+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:16.642880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:21:16.642880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"479'19","reported_seq":842,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.148400+0000","last_change":"2024-04-14T15:24:24.516920+0000","last_active":"2024-04-14T15:24:29.148400+0000","last_peered":"2024-04-14T15:24:29.148400+0000","last_clean":"2024-04-14T15:24:29.148400+0000","last_became_active":"2024-04-14T15:20:52.990622+0000","last_became_peered":"2024-04-14T15:20:52.990622+0000","last_unstale":"2024-04-14T15:24:29.148400+0000","last_undegraded":"2024-04-14T15:24:29.148400+0000","last_fullsized":"2024-04-14T15:24:29.148400+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'19","last_scrub_stamp":"2024-04-14T15:24:24.516820+0000","last_deep_scrub":"344'13","last_deep_scrub_stamp":"2024-04-14T15:16:52.738163+0000","last_clean_scrub_stamp":"2024-04-14T15:24:24.516820+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:41.273155+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":415,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.149362+0000","last_change":"2024-04-14T15:22:55.001625+0000","last_active":"2024-04-14T15:24:29.149362+0000","last_peered":"2024-04-14T15:24:29.149362+0000","last_clean":"2024-04-14T15:24:29.149362+0000","last_became_active":"2024-04-14T15:20:52.996548+0000","last_became_peered":"2024-04-14T15:20:52.996548+0000","last_unstale":"2024-04-14T15:24:29.149362+0000","last_undegraded":"2024-04-14T15:24:29.149362+0000","last_fullsized":"2024-04-14T15:24:29.149362+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:55.001498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:22:55.001498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1041,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.146365+0000","last_change":"2024-04-14T15:21:23.131643+0000","last_active":"2024-04-14T15:24:29.146365+0000","last_peered":"2024-04-14T15:24:29.146365+0000","last_clean":"2024-04-14T15:24:29.146365+0000","last_became_active":"2024-04-14T15:15:21.882643+0000","last_became_peered":"2024-04-14T15:15:21.882643+0000","last_unstale":"2024-04-14T15:24:29.146365+0000","last_undegraded":"2024-04-14T15:24:29.146365+0000","last_fullsized":"2024-04-14T15:24:29.146365+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:23.131512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:33.485411+0000","last_clean_scrub_stamp":"2024-04-14T15:21:23.131512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1376,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.146382+0000","last_change":"2024-04-14T15:24:22.395640+0000","last_active":"2024-04-14T15:24:29.146382+0000","last_peered":"2024-04-14T15:24:29.146382+0000","last_clean":"2024-04-14T15:24:29.146382+0000","last_became_active":"2024-04-14T15:09:49.059711+0000","last_became_peered":"2024-04-14T15:09:49.059711+0000","last_unstale":"2024-04-14T15:24:29.146382+0000","last_undegraded":"2024-04-14T15:24:29.146382+0000","last_fullsized":"2024-04-14T15:24:29.146382+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:22.395562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:24:22.395562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:34.649415+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":422,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-14T15:24:28.778396+0000","last_change":"2024-04-14T15:23:03.603742+0000","last_active":"2024-04-14T15:24:28.778396+0000","last_peered":"2024-04-14T15:24:28.778396+0000","last_clean":"2024-04-14T15:24:28.778396+0000","last_became_active":"2024-04-14T15:22:26.203388+0000","last_became_peered":"2024-04-14T15:22:26.203388+0000","last_unstale":"2024-04-14T15:24:28.778396+0000","last_undegraded":"2024-04-14T15:24:28.778396+0000","last_fullsized":"2024-04-14T15:24:28.778396+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:03.603663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:23:03.603663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"479'24","reported_seq":1426,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-14T15:24:28.136098+0000","last_change":"2024-04-14T15:24:15.604196+0000","last_active":"2024-04-14T15:24:28.136098+0000","last_peered":"2024-04-14T15:24:28.136098+0000","last_clean":"2024-04-14T15:24:28.136098+0000","last_became_active":"2024-04-14T15:20:52.989300+0000","last_became_peered":"2024-04-14T15:20:52.989300+0000","last_unstale":"2024-04-14T15:24:28.136098+0000","last_undegraded":"2024-04-14T15:24:28.136098+0000","last_fullsized":"2024-04-14T15:24:28.136098+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'24","last_scrub_stamp":"2024-04-14T15:24:15.604065+0000","last_deep_scrub":"464'16","last_deep_scrub_stamp":"2024-04-14T15:19:17.589999+0000","last_clean_scrub_stamp":"2024-04-14T15:24:15.604065+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:34.969047+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":1381,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.142809+0000","last_change":"2024-04-14T15:20:45.587983+0000","last_active":"2024-04-14T15:24:29.142809+0000","last_peered":"2024-04-14T15:24:29.142809+0000","last_clean":"2024-04-14T15:24:29.142809+0000","last_became_active":"2024-04-14T15:09:43.023583+0000","last_became_peered":"2024-04-14T15:09:43.023583+0000","last_unstale":"2024-04-14T15:24:29.142809+0000","last_undegraded":"2024-04-14T15:24:29.142809+0000","last_fullsized":"2024-04-14T15:24:29.142809+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:45.587898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:20:45.587898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1373,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.141367+0000","last_change":"2024-04-14T15:24:19.367902+0000","last_active":"2024-04-14T15:24:29.141367+0000","last_peered":"2024-04-14T15:24:29.141367+0000","last_clean":"2024-04-14T15:24:29.141367+0000","last_became_active":"2024-04-14T15:09:47.041683+0000","last_became_peered":"2024-04-14T15:09:47.041683+0000","last_unstale":"2024-04-14T15:24:29.141367+0000","last_undegraded":"2024-04-14T15:24:29.141367+0000","last_fullsized":"2024-04-14T15:24:29.141367+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:19.367758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:08.486217+0000","last_clean_scrub_stamp":"2024-04-14T15:24:19.367758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:40.367460+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1045,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.144648+0000","last_change":"2024-04-14T15:23:28.009795+0000","last_active":"2024-04-14T15:24:29.144648+0000","last_peered":"2024-04-14T15:24:29.144648+0000","last_clean":"2024-04-14T15:24:29.144648+0000","last_became_active":"2024-04-14T15:15:21.889567+0000","last_became_peered":"2024-04-14T15:15:21.889567+0000","last_unstale":"2024-04-14T15:24:29.144648+0000","last_undegraded":"2024-04-14T15:24:29.144648+0000","last_fullsized":"2024-04-14T15:24:29.144648+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:28.009713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:23:28.009713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:47.266664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":1280,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.157414+0000","last_change":"2024-04-14T15:22:19.933335+0000","last_active":"2024-04-14T15:24:29.157414+0000","last_peered":"2024-04-14T15:24:29.157414+0000","last_clean":"2024-04-14T15:24:29.157414+0000","last_became_active":"2024-04-14T15:12:56.387723+0000","last_became_peered":"2024-04-14T15:12:56.387723+0000","last_unstale":"2024-04-14T15:24:29.157414+0000","last_undegraded":"2024-04-14T15:24:29.157414+0000","last_fullsized":"2024-04-14T15:24:29.157414+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:19.933257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:03.685045+0000","last_clean_scrub_stamp":"2024-04-14T15:22:19.933257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.10","version":"479'8","reported_seq":1401,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.143959+0000","last_change":"2024-04-14T15:24:24.848430+0000","last_active":"2024-04-14T15:24:29.143959+0000","last_peered":"2024-04-14T15:24:29.143959+0000","last_clean":"2024-04-14T15:24:29.143959+0000","last_became_active":"2024-04-14T15:20:53.001300+0000","last_became_peered":"2024-04-14T15:20:53.001300+0000","last_unstale":"2024-04-14T15:24:29.143959+0000","last_undegraded":"2024-04-14T15:24:29.143959+0000","last_fullsized":"2024-04-14T15:24:29.143959+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'8","last_scrub_stamp":"2024-04-14T15:24:24.848331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:24:24.848331+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:48.717429+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":419,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.150979+0000","last_change":"2024-04-14T15:23:43.407976+0000","last_active":"2024-04-14T15:24:29.150979+0000","last_peered":"2024-04-14T15:24:29.150979+0000","last_clean":"2024-04-14T15:24:29.150979+0000","last_became_active":"2024-04-14T15:20:52.979496+0000","last_became_peered":"2024-04-14T15:20:52.979496+0000","last_unstale":"2024-04-14T15:24:29.150979+0000","last_undegraded":"2024-04-14T15:24:29.150979+0000","last_fullsized":"2024-04-14T15:24:29.150979+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:43.407897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:08.712289+0000","last_clean_scrub_stamp":"2024-04-14T15:23:43.407897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:47.213727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":408,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-14T15:24:28.134647+0000","last_change":"2024-04-14T15:20:52.991440+0000","last_active":"2024-04-14T15:24:28.134647+0000","last_peered":"2024-04-14T15:24:28.134647+0000","last_clean":"2024-04-14T15:24:28.134647+0000","last_became_active":"2024-04-14T15:20:52.990347+0000","last_became_peered":"2024-04-14T15:20:52.990347+0000","last_unstale":"2024-04-14T15:24:28.134647+0000","last_undegraded":"2024-04-14T15:24:28.134647+0000","last_fullsized":"2024-04-14T15:24:28.134647+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:48.867704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:19:48.867704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1364,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-14T15:24:28.777260+0000","last_change":"2024-04-14T15:24:18.506706+0000","last_active":"2024-04-14T15:24:28.777260+0000","last_peered":"2024-04-14T15:24:28.777260+0000","last_clean":"2024-04-14T15:24:28.777260+0000","last_became_active":"2024-04-14T15:09:49.061359+0000","last_became_peered":"2024-04-14T15:09:49.061359+0000","last_unstale":"2024-04-14T15:24:28.777260+0000","last_undegraded":"2024-04-14T15:24:28.777260+0000","last_fullsized":"2024-04-14T15:24:28.777260+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:18.506553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:24:18.506553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:40.357723+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":1279,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-14T15:24:28.777727+0000","last_change":"2024-04-14T15:21:28.711361+0000","last_active":"2024-04-14T15:24:28.777727+0000","last_peered":"2024-04-14T15:24:28.777727+0000","last_clean":"2024-04-14T15:24:28.777727+0000","last_became_active":"2024-04-14T15:20:52.999623+0000","last_became_peered":"2024-04-14T15:20:52.999623+0000","last_unstale":"2024-04-14T15:24:28.777727+0000","last_undegraded":"2024-04-14T15:24:28.777727+0000","last_fullsized":"2024-04-14T15:24:28.777727+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:28.711266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:15:43.680852+0000","last_clean_scrub_stamp":"2024-04-14T15:21:28.711266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"479'24","reported_seq":1060,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.145448+0000","last_change":"2024-04-14T15:21:13.011149+0000","last_active":"2024-04-14T15:24:29.145448+0000","last_peered":"2024-04-14T15:24:29.145448+0000","last_clean":"2024-04-14T15:24:29.145448+0000","last_became_active":"2024-04-14T15:20:52.996917+0000","last_became_peered":"2024-04-14T15:20:52.996917+0000","last_unstale":"2024-04-14T15:24:29.145448+0000","last_undegraded":"2024-04-14T15:24:29.145448+0000","last_fullsized":"2024-04-14T15:24:29.145448+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'24","last_scrub_stamp":"2024-04-14T15:21:13.011012+0000","last_deep_scrub":"102'16","last_deep_scrub_stamp":"2024-04-14T15:12:35.999326+0000","last_clean_scrub_stamp":"2024-04-14T15:21:13.011012+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":416,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.151957+0000","last_change":"2024-04-14T15:22:11.889589+0000","last_active":"2024-04-14T15:24:29.151957+0000","last_peered":"2024-04-14T15:24:29.151957+0000","last_clean":"2024-04-14T15:24:29.151957+0000","last_became_active":"2024-04-14T15:20:52.970959+0000","last_became_peered":"2024-04-14T15:20:52.970959+0000","last_unstale":"2024-04-14T15:24:29.151957+0000","last_undegraded":"2024-04-14T15:24:29.151957+0000","last_fullsized":"2024-04-14T15:24:29.151957+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:11.889509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:22:11.889509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1374,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-14T15:24:28.775922+0000","last_change":"2024-04-14T15:24:17.557862+0000","last_active":"2024-04-14T15:24:28.775922+0000","last_peered":"2024-04-14T15:24:28.775922+0000","last_clean":"2024-04-14T15:24:28.775922+0000","last_became_active":"2024-04-14T15:20:52.983919+0000","last_became_peered":"2024-04-14T15:20:52.983919+0000","last_unstale":"2024-04-14T15:24:28.775922+0000","last_undegraded":"2024-04-14T15:24:28.775922+0000","last_fullsized":"2024-04-14T15:24:28.775922+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:17.557766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:24:17.557766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:36.450647+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1051,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.146635+0000","last_change":"2024-04-14T15:22:09.053629+0000","last_active":"2024-04-14T15:24:29.146635+0000","last_peered":"2024-04-14T15:24:29.146635+0000","last_clean":"2024-04-14T15:24:29.146635+0000","last_became_active":"2024-04-14T15:20:52.993918+0000","last_became_peered":"2024-04-14T15:20:52.993918+0000","last_unstale":"2024-04-14T15:24:29.146635+0000","last_undegraded":"2024-04-14T15:24:29.146635+0000","last_fullsized":"2024-04-14T15:24:29.146635+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:09.053546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:22:09.053546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":1279,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.155210+0000","last_change":"2024-04-14T15:24:26.504869+0000","last_active":"2024-04-14T15:24:29.155210+0000","last_peered":"2024-04-14T15:24:29.155210+0000","last_clean":"2024-04-14T15:24:29.155210+0000","last_became_active":"2024-04-14T15:12:56.335230+0000","last_became_peered":"2024-04-14T15:12:56.335230+0000","last_unstale":"2024-04-14T15:24:29.155210+0000","last_undegraded":"2024-04-14T15:24:29.155210+0000","last_fullsized":"2024-04-14T15:24:29.155210+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:26.504790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:24:26.504790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:44.382683+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"479'20","reported_seq":1404,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.150397+0000","last_change":"2024-04-14T15:22:58.400820+0000","last_active":"2024-04-14T15:24:29.150397+0000","last_peered":"2024-04-14T15:24:29.150397+0000","last_clean":"2024-04-14T15:24:29.150397+0000","last_became_active":"2024-04-14T15:09:45.181456+0000","last_became_peered":"2024-04-14T15:09:45.181456+0000","last_unstale":"2024-04-14T15:24:29.150397+0000","last_undegraded":"2024-04-14T15:24:29.150397+0000","last_fullsized":"2024-04-14T15:24:29.150397+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'20","last_scrub_stamp":"2024-04-14T15:22:58.400727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:22:58.400727+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":926,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-14T15:24:28.134922+0000","last_change":"2024-04-14T15:19:57.575006+0000","last_active":"2024-04-14T15:24:28.134922+0000","last_peered":"2024-04-14T15:24:28.134922+0000","last_clean":"2024-04-14T15:24:28.134922+0000","last_became_active":"2024-04-14T15:15:44.878336+0000","last_became_peered":"2024-04-14T15:15:44.878336+0000","last_unstale":"2024-04-14T15:24:28.134922+0000","last_undegraded":"2024-04-14T15:24:28.134922+0000","last_fullsized":"2024-04-14T15:24:28.134922+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:57.574879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:19:57.574879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"528'9","reported_seq":484,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-14T15:24:28.138276+0000","last_change":"2024-04-14T15:20:53.001282+0000","last_active":"2024-04-14T15:24:28.138276+0000","last_peered":"2024-04-14T15:24:28.138276+0000","last_clean":"2024-04-14T15:24:28.138276+0000","last_became_active":"2024-04-14T15:20:52.982112+0000","last_became_peered":"2024-04-14T15:20:52.982112+0000","last_unstale":"2024-04-14T15:24:28.138276+0000","last_undegraded":"2024-04-14T15:24:28.138276+0000","last_fullsized":"2024-04-14T15:24:28.138276+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:20:45.809489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:20:45.809489+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"72'1","reported_seq":1372,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-14T15:24:28.778728+0000","last_change":"2024-04-14T15:23:32.511432+0000","last_active":"2024-04-14T15:24:28.778728+0000","last_peered":"2024-04-14T15:24:28.778728+0000","last_clean":"2024-04-14T15:24:28.778728+0000","last_became_active":"2024-04-14T15:09:49.062051+0000","last_became_peered":"2024-04-14T15:09:49.062051+0000","last_unstale":"2024-04-14T15:24:28.778728+0000","last_undegraded":"2024-04-14T15:24:28.778728+0000","last_fullsized":"2024-04-14T15:24:28.778728+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-04-14T15:23:32.511322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:23:32.511322+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:33.567939+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":1279,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.146587+0000","last_change":"2024-04-14T15:22:14.354882+0000","last_active":"2024-04-14T15:24:29.146587+0000","last_peered":"2024-04-14T15:24:29.146587+0000","last_clean":"2024-04-14T15:24:29.146587+0000","last_became_active":"2024-04-14T15:20:52.986657+0000","last_became_peered":"2024-04-14T15:20:52.986657+0000","last_unstale":"2024-04-14T15:24:29.146587+0000","last_undegraded":"2024-04-14T15:24:29.146587+0000","last_fullsized":"2024-04-14T15:24:29.146587+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:14.354795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:22:14.354795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"519'24","reported_seq":1431,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.151327+0000","last_change":"2024-04-14T15:23:40.504101+0000","last_active":"2024-04-14T15:24:29.151327+0000","last_peered":"2024-04-14T15:24:29.151327+0000","last_clean":"2024-04-14T15:24:29.151327+0000","last_became_active":"2024-04-14T15:20:52.987333+0000","last_became_peered":"2024-04-14T15:20:52.987333+0000","last_unstale":"2024-04-14T15:24:29.151327+0000","last_undegraded":"2024-04-14T15:24:29.151327+0000","last_fullsized":"2024-04-14T15:24:29.151327+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"519'24","last_scrub_stamp":"2024-04-14T15:23:40.504002+0000","last_deep_scrub":"519'24","last_deep_scrub_stamp":"2024-04-14T15:20:44.763417+0000","last_clean_scrub_stamp":"2024-04-14T15:23:40.504002+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:57.259947+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.19","version":"684'4","reported_seq":1078,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.145021+0000","last_change":"2024-04-14T15:23:46.931649+0000","last_active":"2024-04-14T15:24:29.145021+0000","last_peered":"2024-04-14T15:24:29.145021+0000","last_clean":"2024-04-14T15:24:29.145021+0000","last_became_active":"2024-04-14T15:20:52.995628+0000","last_became_peered":"2024-04-14T15:20:52.995628+0000","last_unstale":"2024-04-14T15:24:29.145021+0000","last_undegraded":"2024-04-14T15:24:29.145021+0000","last_fullsized":"2024-04-14T15:24:29.145021+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"684'4","last_scrub_stamp":"2024-04-14T15:23:46.931560+0000","last_deep_scrub":"684'4","last_deep_scrub_stamp":"2024-04-14T15:23:46.931560+0000","last_clean_scrub_stamp":"2024-04-14T15:23:46.931560+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:51.788030+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":957,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1370,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.150508+0000","last_change":"2024-04-14T15:22:20.256256+0000","last_active":"2024-04-14T15:24:29.150508+0000","last_peered":"2024-04-14T15:24:29.150508+0000","last_clean":"2024-04-14T15:24:29.150508+0000","last_became_active":"2024-04-14T15:09:47.045730+0000","last_became_peered":"2024-04-14T15:09:47.045730+0000","last_unstale":"2024-04-14T15:24:29.150508+0000","last_undegraded":"2024-04-14T15:24:29.150508+0000","last_fullsized":"2024-04-14T15:24:29.150508+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:20.256177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:22:20.256177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1360,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.142569+0000","last_change":"2024-04-14T15:20:38.518996+0000","last_active":"2024-04-14T15:24:29.142569+0000","last_peered":"2024-04-14T15:24:29.142569+0000","last_clean":"2024-04-14T15:24:29.142569+0000","last_became_active":"2024-04-14T15:09:49.057545+0000","last_became_peered":"2024-04-14T15:09:49.057545+0000","last_unstale":"2024-04-14T15:24:29.142569+0000","last_undegraded":"2024-04-14T15:24:29.142569+0000","last_fullsized":"2024-04-14T15:24:29.142569+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:38.518885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:21.247695+0000","last_clean_scrub_stamp":"2024-04-14T15:20:38.518885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":926,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-14T15:24:28.135433+0000","last_change":"2024-04-14T15:21:58.069606+0000","last_active":"2024-04-14T15:24:28.135433+0000","last_peered":"2024-04-14T15:24:28.135433+0000","last_clean":"2024-04-14T15:24:28.135433+0000","last_became_active":"2024-04-14T15:15:44.882604+0000","last_became_peered":"2024-04-14T15:15:44.882604+0000","last_unstale":"2024-04-14T15:24:28.135433+0000","last_undegraded":"2024-04-14T15:24:28.135433+0000","last_fullsized":"2024-04-14T15:24:28.135433+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:58.069509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:21:58.069509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"479'27","reported_seq":1421,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.158570+0000","last_change":"2024-04-14T15:24:21.500258+0000","last_active":"2024-04-14T15:24:29.158570+0000","last_peered":"2024-04-14T15:24:29.158570+0000","last_clean":"2024-04-14T15:24:29.158570+0000","last_became_active":"2024-04-14T15:09:45.032887+0000","last_became_peered":"2024-04-14T15:09:45.032887+0000","last_unstale":"2024-04-14T15:24:29.158570+0000","last_undegraded":"2024-04-14T15:24:29.158570+0000","last_fullsized":"2024-04-14T15:24:29.158570+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'27","last_scrub_stamp":"2024-04-14T15:24:21.500156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:24:21.500156+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:30.530072+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":408,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-14T15:24:28.136381+0000","last_change":"2024-04-14T15:20:52.990991+0000","last_active":"2024-04-14T15:24:28.136381+0000","last_peered":"2024-04-14T15:24:28.136381+0000","last_clean":"2024-04-14T15:24:28.136381+0000","last_became_active":"2024-04-14T15:20:52.990634+0000","last_became_peered":"2024-04-14T15:20:52.990634+0000","last_unstale":"2024-04-14T15:24:28.136381+0000","last_undegraded":"2024-04-14T15:24:28.136381+0000","last_fullsized":"2024-04-14T15:24:28.136381+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:42.754721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:42.754721+0000","last_clean_scrub_stamp":"2024-04-14T15:20:42.754721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1372,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.142723+0000","last_change":"2024-04-14T15:22:18.460314+0000","last_active":"2024-04-14T15:24:29.142723+0000","last_peered":"2024-04-14T15:24:29.142723+0000","last_clean":"2024-04-14T15:24:29.142723+0000","last_became_active":"2024-04-14T15:09:47.042733+0000","last_became_peered":"2024-04-14T15:09:47.042733+0000","last_unstale":"2024-04-14T15:24:29.142723+0000","last_undegraded":"2024-04-14T15:24:29.142723+0000","last_fullsized":"2024-04-14T15:24:29.142723+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:18.460230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:22:18.460230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1371,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-14T15:24:28.136634+0000","last_change":"2024-04-14T15:22:08.071314+0000","last_active":"2024-04-14T15:24:28.136634+0000","last_peered":"2024-04-14T15:24:28.136634+0000","last_clean":"2024-04-14T15:24:28.136634+0000","last_became_active":"2024-04-14T15:20:52.998253+0000","last_became_peered":"2024-04-14T15:20:52.998253+0000","last_unstale":"2024-04-14T15:24:28.136634+0000","last_undegraded":"2024-04-14T15:24:28.136634+0000","last_fullsized":"2024-04-14T15:24:28.136634+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:08.071208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:54.453174+0000","last_clean_scrub_stamp":"2024-04-14T15:22:08.071208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":1272,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-14T15:24:28.776687+0000","last_change":"2024-04-14T15:21:17.675180+0000","last_active":"2024-04-14T15:24:28.776687+0000","last_peered":"2024-04-14T15:24:28.776687+0000","last_clean":"2024-04-14T15:24:28.776687+0000","last_became_active":"2024-04-14T15:12:56.392352+0000","last_became_peered":"2024-04-14T15:12:56.392352+0000","last_unstale":"2024-04-14T15:24:28.776687+0000","last_undegraded":"2024-04-14T15:24:28.776687+0000","last_fullsized":"2024-04-14T15:24:28.776687+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:17.675093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:21:17.675093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"479'18","reported_seq":1085,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.150212+0000","last_change":"2024-04-14T15:22:06.042903+0000","last_active":"2024-04-14T15:24:29.150212+0000","last_peered":"2024-04-14T15:24:29.150212+0000","last_clean":"2024-04-14T15:24:29.150212+0000","last_became_active":"2024-04-14T15:15:21.877838+0000","last_became_peered":"2024-04-14T15:15:21.877838+0000","last_unstale":"2024-04-14T15:24:29.150212+0000","last_undegraded":"2024-04-14T15:24:29.150212+0000","last_fullsized":"2024-04-14T15:24:29.150212+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'18","last_scrub_stamp":"2024-04-14T15:22:06.042808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:22:06.042808+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":1063,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.146455+0000","last_change":"2024-04-14T15:22:52.950407+0000","last_active":"2024-04-14T15:24:29.146455+0000","last_peered":"2024-04-14T15:24:29.146455+0000","last_clean":"2024-04-14T15:24:29.146455+0000","last_became_active":"2024-04-14T15:20:53.001534+0000","last_became_peered":"2024-04-14T15:20:53.001534+0000","last_unstale":"2024-04-14T15:24:29.146455+0000","last_undegraded":"2024-04-14T15:24:29.146455+0000","last_fullsized":"2024-04-14T15:24:29.146455+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:52.950279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:22:52.950279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1367,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-14T15:24:28.136900+0000","last_change":"2024-04-14T15:20:39.717630+0000","last_active":"2024-04-14T15:24:28.136900+0000","last_peered":"2024-04-14T15:24:28.136900+0000","last_clean":"2024-04-14T15:24:28.136900+0000","last_became_active":"2024-04-14T15:09:47.048045+0000","last_became_peered":"2024-04-14T15:09:47.048045+0000","last_unstale":"2024-04-14T15:24:28.136900+0000","last_undegraded":"2024-04-14T15:24:28.136900+0000","last_fullsized":"2024-04-14T15:24:28.136900+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:39.717522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:20:39.717522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1053,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.153794+0000","last_change":"2024-04-14T15:22:20.946609+0000","last_active":"2024-04-14T15:24:29.153794+0000","last_peered":"2024-04-14T15:24:29.153794+0000","last_clean":"2024-04-14T15:24:29.153794+0000","last_became_active":"2024-04-14T15:20:52.998775+0000","last_became_peered":"2024-04-14T15:20:52.998775+0000","last_unstale":"2024-04-14T15:24:29.153794+0000","last_undegraded":"2024-04-14T15:24:29.153794+0000","last_fullsized":"2024-04-14T15:24:29.153794+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:20.946524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:22:20.946524+0000","last_clean_scrub_stamp":"2024-04-14T15:22:20.946524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":1279,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.156671+0000","last_change":"2024-04-14T15:22:18.954117+0000","last_active":"2024-04-14T15:24:29.156671+0000","last_peered":"2024-04-14T15:24:29.156671+0000","last_clean":"2024-04-14T15:24:29.156671+0000","last_became_active":"2024-04-14T15:12:56.375278+0000","last_became_peered":"2024-04-14T15:12:56.375278+0000","last_unstale":"2024-04-14T15:24:29.156671+0000","last_undegraded":"2024-04-14T15:24:29.156671+0000","last_fullsized":"2024-04-14T15:24:29.156671+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:18.954041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:01.840924+0000","last_clean_scrub_stamp":"2024-04-14T15:22:18.954041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"479'19","reported_seq":1400,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-14T15:24:28.135547+0000","last_change":"2024-04-14T15:21:22.085022+0000","last_active":"2024-04-14T15:24:28.135547+0000","last_peered":"2024-04-14T15:24:28.135547+0000","last_clean":"2024-04-14T15:24:28.135547+0000","last_became_active":"2024-04-14T15:09:45.030093+0000","last_became_peered":"2024-04-14T15:09:45.030093+0000","last_unstale":"2024-04-14T15:24:28.135547+0000","last_undegraded":"2024-04-14T15:24:28.135547+0000","last_fullsized":"2024-04-14T15:24:28.135547+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'19","last_scrub_stamp":"2024-04-14T15:21:22.084927+0000","last_deep_scrub":"276'13","last_deep_scrub_stamp":"2024-04-14T15:15:43.969775+0000","last_clean_scrub_stamp":"2024-04-14T15:21:22.084927+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":1081,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-14T15:24:28.777480+0000","last_change":"2024-04-14T15:24:26.465423+0000","last_active":"2024-04-14T15:24:28.777480+0000","last_peered":"2024-04-14T15:24:28.777480+0000","last_clean":"2024-04-14T15:24:28.777480+0000","last_became_active":"2024-04-14T15:14:30.857727+0000","last_became_peered":"2024-04-14T15:14:30.857727+0000","last_unstale":"2024-04-14T15:24:28.777480+0000","last_undegraded":"2024-04-14T15:24:28.777480+0000","last_fullsized":"2024-04-14T15:24:28.777480+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:26.465343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:00.650062+0000","last_clean_scrub_stamp":"2024-04-14T15:24:26.465343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:42.640865+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":935,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.148504+0000","last_change":"2024-04-14T15:23:04.472665+0000","last_active":"2024-04-14T15:24:29.148504+0000","last_peered":"2024-04-14T15:24:29.148504+0000","last_clean":"2024-04-14T15:24:29.148504+0000","last_became_active":"2024-04-14T15:15:44.883121+0000","last_became_peered":"2024-04-14T15:15:44.883121+0000","last_unstale":"2024-04-14T15:24:29.148504+0000","last_undegraded":"2024-04-14T15:24:29.148504+0000","last_fullsized":"2024-04-14T15:24:29.148504+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:04.472586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:23:04.472586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1373,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.146978+0000","last_change":"2024-04-14T15:23:01.449252+0000","last_active":"2024-04-14T15:24:29.146978+0000","last_peered":"2024-04-14T15:24:29.146978+0000","last_clean":"2024-04-14T15:24:29.146978+0000","last_became_active":"2024-04-14T15:09:49.061756+0000","last_became_peered":"2024-04-14T15:09:49.061756+0000","last_unstale":"2024-04-14T15:24:29.146978+0000","last_undegraded":"2024-04-14T15:24:29.146978+0000","last_fullsized":"2024-04-14T15:24:29.146978+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:01.449169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:23:01.449169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":416,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-14T15:24:28.135823+0000","last_change":"2024-04-14T15:24:24.522250+0000","last_active":"2024-04-14T15:24:28.135823+0000","last_peered":"2024-04-14T15:24:28.135823+0000","last_clean":"2024-04-14T15:24:28.135823+0000","last_became_active":"2024-04-14T15:20:52.990838+0000","last_became_peered":"2024-04-14T15:20:52.990838+0000","last_unstale":"2024-04-14T15:24:28.135823+0000","last_undegraded":"2024-04-14T15:24:28.135823+0000","last_fullsized":"2024-04-14T15:24:28.135823+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:24.522164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:24:24.522164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:27.651945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"479'9","reported_seq":1386,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-14T15:24:28.136704+0000","last_change":"2024-04-14T15:22:54.203852+0000","last_active":"2024-04-14T15:24:28.136704+0000","last_peered":"2024-04-14T15:24:28.136704+0000","last_clean":"2024-04-14T15:24:28.136704+0000","last_became_active":"2024-04-14T15:09:45.028667+0000","last_became_peered":"2024-04-14T15:09:45.028667+0000","last_unstale":"2024-04-14T15:24:28.136704+0000","last_undegraded":"2024-04-14T15:24:28.136704+0000","last_fullsized":"2024-04-14T15:24:28.136704+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'9","last_scrub_stamp":"2024-04-14T15:22:54.203720+0000","last_deep_scrub":"87'7","last_deep_scrub_stamp":"2024-04-14T15:11:10.510785+0000","last_clean_scrub_stamp":"2024-04-14T15:22:54.203720+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":858,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.149234+0000","last_change":"2024-04-14T15:22:15.944094+0000","last_active":"2024-04-14T15:24:29.149234+0000","last_peered":"2024-04-14T15:24:29.149234+0000","last_clean":"2024-04-14T15:24:29.149234+0000","last_became_active":"2024-04-14T15:20:52.989307+0000","last_became_peered":"2024-04-14T15:20:52.989307+0000","last_unstale":"2024-04-14T15:24:29.149234+0000","last_undegraded":"2024-04-14T15:24:29.149234+0000","last_fullsized":"2024-04-14T15:24:29.149234+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:15.944005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:22:15.944005+0000","last_clean_scrub_stamp":"2024-04-14T15:22:15.944005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1367,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-14T15:24:28.777169+0000","last_change":"2024-04-14T15:22:05.564704+0000","last_active":"2024-04-14T15:24:28.777169+0000","last_peered":"2024-04-14T15:24:28.777169+0000","last_clean":"2024-04-14T15:24:28.777169+0000","last_became_active":"2024-04-14T15:09:47.046517+0000","last_became_peered":"2024-04-14T15:09:47.046517+0000","last_unstale":"2024-04-14T15:24:28.777169+0000","last_undegraded":"2024-04-14T15:24:28.777169+0000","last_fullsized":"2024-04-14T15:24:28.777169+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:05.564624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:22:05.564624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":932,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.140794+0000","last_change":"2024-04-14T15:22:08.325886+0000","last_active":"2024-04-14T15:24:29.140794+0000","last_peered":"2024-04-14T15:24:29.140794+0000","last_clean":"2024-04-14T15:24:29.140794+0000","last_became_active":"2024-04-14T15:15:44.887227+0000","last_became_peered":"2024-04-14T15:15:44.887227+0000","last_unstale":"2024-04-14T15:24:29.140794+0000","last_undegraded":"2024-04-14T15:24:29.140794+0000","last_fullsized":"2024-04-14T15:24:29.140794+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:08.325808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:56.268086+0000","last_clean_scrub_stamp":"2024-04-14T15:22:08.325808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":1278,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-14T15:24:28.778859+0000","last_change":"2024-04-14T15:21:21.668496+0000","last_active":"2024-04-14T15:24:28.778859+0000","last_peered":"2024-04-14T15:24:28.778859+0000","last_clean":"2024-04-14T15:24:28.778859+0000","last_became_active":"2024-04-14T15:20:52.984387+0000","last_became_peered":"2024-04-14T15:20:52.984387+0000","last_unstale":"2024-04-14T15:24:28.778859+0000","last_undegraded":"2024-04-14T15:24:28.778859+0000","last_fullsized":"2024-04-14T15:24:28.778859+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:21.668407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:34.558312+0000","last_clean_scrub_stamp":"2024-04-14T15:21:21.668407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.18","version":"479'15","reported_seq":1413,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-14T15:24:28.134720+0000","last_change":"2024-04-14T15:22:57.315447+0000","last_active":"2024-04-14T15:24:28.134720+0000","last_peered":"2024-04-14T15:24:28.134720+0000","last_clean":"2024-04-14T15:24:28.134720+0000","last_became_active":"2024-04-14T15:20:52.985210+0000","last_became_peered":"2024-04-14T15:20:52.985210+0000","last_unstale":"2024-04-14T15:24:28.134720+0000","last_undegraded":"2024-04-14T15:24:28.134720+0000","last_fullsized":"2024-04-14T15:24:28.134720+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'15","last_scrub_stamp":"2024-04-14T15:22:57.315355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:22:57.315355+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":1059,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.152298+0000","last_change":"2024-04-14T15:24:21.791334+0000","last_active":"2024-04-14T15:24:29.152298+0000","last_peered":"2024-04-14T15:24:29.152298+0000","last_clean":"2024-04-14T15:24:29.152298+0000","last_became_active":"2024-04-14T15:15:21.891926+0000","last_became_peered":"2024-04-14T15:15:21.891926+0000","last_unstale":"2024-04-14T15:24:29.152298+0000","last_undegraded":"2024-04-14T15:24:29.152298+0000","last_fullsized":"2024-04-14T15:24:29.152298+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:21.791233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:24:21.791233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:34.842188+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1373,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.140527+0000","last_change":"2024-04-14T15:24:22.302587+0000","last_active":"2024-04-14T15:24:29.140527+0000","last_peered":"2024-04-14T15:24:29.140527+0000","last_clean":"2024-04-14T15:24:29.140527+0000","last_became_active":"2024-04-14T15:09:47.041486+0000","last_became_peered":"2024-04-14T15:09:47.041486+0000","last_unstale":"2024-04-14T15:24:29.140527+0000","last_undegraded":"2024-04-14T15:24:29.140527+0000","last_fullsized":"2024-04-14T15:24:29.140527+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:22.302513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:24:22.302513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:40.486676+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1355,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-14T15:24:28.134970+0000","last_change":"2024-04-14T15:20:36.761316+0000","last_active":"2024-04-14T15:24:28.134970+0000","last_peered":"2024-04-14T15:24:28.134970+0000","last_clean":"2024-04-14T15:24:28.134970+0000","last_became_active":"2024-04-14T15:09:49.058904+0000","last_became_peered":"2024-04-14T15:09:49.058904+0000","last_unstale":"2024-04-14T15:24:28.134970+0000","last_undegraded":"2024-04-14T15:24:28.134970+0000","last_fullsized":"2024-04-14T15:24:28.134970+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:36.761246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:20:36.761246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":926,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-14T15:24:28.777610+0000","last_change":"2024-04-14T15:21:14.673688+0000","last_active":"2024-04-14T15:24:28.777610+0000","last_peered":"2024-04-14T15:24:28.777610+0000","last_clean":"2024-04-14T15:24:28.777610+0000","last_became_active":"2024-04-14T15:15:44.889605+0000","last_became_peered":"2024-04-14T15:15:44.889605+0000","last_unstale":"2024-04-14T15:24:28.777610+0000","last_undegraded":"2024-04-14T15:24:28.777610+0000","last_fullsized":"2024-04-14T15:24:28.777610+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:14.673591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:21:14.673591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"479'34","reported_seq":1103,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.145547+0000","last_change":"2024-04-14T15:23:00.061412+0000","last_active":"2024-04-14T15:24:29.145547+0000","last_peered":"2024-04-14T15:24:29.145547+0000","last_clean":"2024-04-14T15:24:29.145547+0000","last_became_active":"2024-04-14T15:15:21.884310+0000","last_became_peered":"2024-04-14T15:15:21.884310+0000","last_unstale":"2024-04-14T15:24:29.145547+0000","last_undegraded":"2024-04-14T15:24:29.145547+0000","last_fullsized":"2024-04-14T15:24:29.145547+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'34","last_scrub_stamp":"2024-04-14T15:23:00.061294+0000","last_deep_scrub":"306'22","last_deep_scrub_stamp":"2024-04-14T15:16:13.898106+0000","last_clean_scrub_stamp":"2024-04-14T15:23:00.061294+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":1386,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.150077+0000","last_change":"2024-04-14T15:24:24.353115+0000","last_active":"2024-04-14T15:24:29.150077+0000","last_peered":"2024-04-14T15:24:29.150077+0000","last_clean":"2024-04-14T15:24:29.150077+0000","last_became_active":"2024-04-14T15:09:43.017351+0000","last_became_peered":"2024-04-14T15:09:43.017351+0000","last_unstale":"2024-04-14T15:24:29.150077+0000","last_undegraded":"2024-04-14T15:24:29.150077+0000","last_fullsized":"2024-04-14T15:24:29.150077+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:24.352978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:24:24.352978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:43.866848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1366,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-14T15:24:28.135078+0000","last_change":"2024-04-14T15:20:52.976225+0000","last_active":"2024-04-14T15:24:28.135078+0000","last_peered":"2024-04-14T15:24:28.135078+0000","last_clean":"2024-04-14T15:24:28.135078+0000","last_became_active":"2024-04-14T15:20:52.974360+0000","last_became_peered":"2024-04-14T15:20:52.974360+0000","last_unstale":"2024-04-14T15:24:28.135078+0000","last_undegraded":"2024-04-14T15:24:28.135078+0000","last_fullsized":"2024-04-14T15:24:28.135078+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:19:49.631273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:08.512130+0000","last_clean_scrub_stamp":"2024-04-14T15:19:49.631273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1374,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-04-14T15:24:29.146055+0000","last_change":"2024-04-14T15:24:27.874590+0000","last_active":"2024-04-14T15:24:29.146055+0000","last_peered":"2024-04-14T15:24:29.146055+0000","last_clean":"2024-04-14T15:24:29.146055+0000","last_became_active":"2024-04-14T15:20:52.998938+0000","last_became_peered":"2024-04-14T15:20:52.998938+0000","last_unstale":"2024-04-14T15:24:29.146055+0000","last_undegraded":"2024-04-14T15:24:29.146055+0000","last_fullsized":"2024-04-14T15:24:29.146055+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:27.874498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:27.874498+0000","last_clean_scrub_stamp":"2024-04-14T15:24:27.874498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:29.705955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":332,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":60,"acting":60,"num_store_stats":4},{"poolid":323,"num_pg":8,"stat_sum":{"num_bytes":71,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":12,"num_read_kb":1,"num_write":26,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":147456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53,"ondisk_log_size":53,"up":24,"acting":24,"num_store_stats":7},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":5,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1003,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":918048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2109440,"data_stored":2098784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":877,"num_read_kb":872,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1121,"num_read_kb":878,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":1694,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":785,"ondisk_log_size":785,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103351,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":99104,"kb_used_data":5452,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":93646048,"statfs":{"total":95995035648,"available":95893553152,"internally_reserved":0,"allocated":5582848,"data_stored":2848544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":95877578},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":210453397690,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575060,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":116872,"kb_used_data":5184,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93628280,"statfs":{"total":95995035648,"available":95875358720,"internally_reserved":0,"allocated":5308416,"data_stored":2534288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738585,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":90504,"kb_used_data":5172,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":93654648,"statfs":{"total":95995035648,"available":95902359552,"internally_reserved":0,"allocated":5296128,"data_stored":2523117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":87357895},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411536,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":108776,"kb_used_data":5404,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93636376,"statfs":{"total":95995035648,"available":95883649024,"internally_reserved":0,"allocated":5533696,"data_stored":2792441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182602,"num_pgs":99,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":68900,"kb_used_data":5776,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93676252,"statfs":{"total":95995035648,"available":95924482048,"internally_reserved":0,"allocated":5914624,"data_stored":3125248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986372,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":116864,"kb_used_data":5168,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93628288,"statfs":{"total":95995035648,"available":95875366912,"internally_reserved":0,"allocated":5292032,"data_stored":2529913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659329,"num_pgs":88,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":117408,"kb_used_data":5712,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93627744,"statfs":{"total":95995035648,"available":95874809856,"internally_reserved":0,"allocated":5849088,"data_stored":3106816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":12288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":20480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":20480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":332,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":332,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":332,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":332,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-14T15:24:32.965 INFO:tasks.cephadm.ceph_manager.ceph:too many creating 2024-04-14T15:24:32.971 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:24:32.995 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:32.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:32.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:24:33.034 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:24:33.035 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:24:33.097 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:24:33.121 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:33.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:33.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:24:33.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:33 smithi090 ceph-mon[41670]: 11.1f scrub starts 2024-04-14T15:24:33.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:33 smithi090 ceph-mon[41670]: 11.1f scrub ok 2024-04-14T15:24:33.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:33 smithi090 ceph-mon[41670]: 5.13 scrub starts 2024-04-14T15:24:33.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:33 smithi090 ceph-mon[41670]: 5.13 scrub ok 2024-04-14T15:24:33.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:33 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-128"}]': finished 2024-04-14T15:24:33.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:33 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-128", "mode": "writeback"}]: dispatch 2024-04-14T15:24:33.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:33 smithi090 ceph-mon[41670]: osdmap e730: 8 total, 8 up, 7 in 2024-04-14T15:24:33.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:33 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-128", "mode": "writeback"}]: dispatch 2024-04-14T15:24:33.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:33 smithi090 ceph-mon[41670]: 5.18 deep-scrub starts 2024-04-14T15:24:33.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:33 smithi090 ceph-mon[41670]: 5.18 deep-scrub ok 2024-04-14T15:24:33.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:33 smithi090 ceph-mon[30399]: 11.1f scrub starts 2024-04-14T15:24:33.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:33 smithi090 ceph-mon[30399]: 11.1f scrub ok 2024-04-14T15:24:33.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:33 smithi090 ceph-mon[30399]: 5.13 scrub starts 2024-04-14T15:24:33.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:33 smithi090 ceph-mon[30399]: 5.13 scrub ok 2024-04-14T15:24:33.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:33 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-128"}]': finished 2024-04-14T15:24:33.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:33 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-128", "mode": "writeback"}]: dispatch 2024-04-14T15:24:33.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:33 smithi090 ceph-mon[30399]: osdmap e730: 8 total, 8 up, 7 in 2024-04-14T15:24:33.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:33 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-128", "mode": "writeback"}]: dispatch 2024-04-14T15:24:33.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:33 smithi090 ceph-mon[30399]: 5.18 deep-scrub starts 2024-04-14T15:24:33.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:33 smithi090 ceph-mon[30399]: 5.18 deep-scrub ok 2024-04-14T15:24:33.223 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:24:33.248 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:33.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:33.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:24:33.349 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:24:33.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:33 smithi092 ceph-mon[33966]: 11.1f scrub starts 2024-04-14T15:24:33.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:33 smithi092 ceph-mon[33966]: 11.1f scrub ok 2024-04-14T15:24:33.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:33 smithi092 ceph-mon[33966]: 5.13 scrub starts 2024-04-14T15:24:33.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:33 smithi092 ceph-mon[33966]: 5.13 scrub ok 2024-04-14T15:24:33.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:33 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-128"}]': finished 2024-04-14T15:24:33.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:33 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-128", "mode": "writeback"}]: dispatch 2024-04-14T15:24:33.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:33 smithi092 ceph-mon[33966]: osdmap e730: 8 total, 8 up, 7 in 2024-04-14T15:24:33.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:33 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-128", "mode": "writeback"}]: dispatch 2024-04-14T15:24:33.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:33 smithi092 ceph-mon[33966]: 5.18 deep-scrub starts 2024-04-14T15:24:33.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:33 smithi092 ceph-mon[33966]: 5.18 deep-scrub ok 2024-04-14T15:24:33.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:33.375 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:33.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:24:33.476 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:24:33.500 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:33.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:33.501 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:24:33.602 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:24:33.627 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:33.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:33.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:24:33.728 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:24:33.752 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:33.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:33.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:24:33.853 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:24:33.877 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:33.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:33.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:24:33.978 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:24:34.002 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:34.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:34.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:24:34.083 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:24:33 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:24:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:24:34.104 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:24:34.127 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:34.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:34.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:24:34.229 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:24:34.252 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:34.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:34.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:24:34.353 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:24:34.377 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:34.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:34.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:24:34.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:34 smithi090 ceph-mon[41670]: 2.7 scrub starts 2024-04-14T15:24:34.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:34 smithi090 ceph-mon[41670]: 2.7 scrub ok 2024-04-14T15:24:34.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:34 smithi090 ceph-mon[41670]: 6.c scrub starts 2024-04-14T15:24:34.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:34 smithi090 ceph-mon[41670]: 6.c scrub ok 2024-04-14T15:24:34.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:34 smithi090 ceph-mon[41670]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:24:34.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:34 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-128", "mode": "writeback"}]': finished 2024-04-14T15:24:34.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:34 smithi090 ceph-mon[41670]: osdmap e731: 8 total, 8 up, 7 in 2024-04-14T15:24:34.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:34 smithi090 ceph-mon[41670]: pgmap v1080: 214 pgs: 20 creating+peering, 12 unknown, 182 active+clean; 582 KiB data, 702 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:24:34.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:34 smithi090 ceph-mon[41670]: 2.15 scrub starts 2024-04-14T15:24:34.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:34 smithi090 ceph-mon[41670]: 2.15 scrub ok 2024-04-14T15:24:34.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:34 smithi090 ceph-mon[41670]: 4.16 scrub starts 2024-04-14T15:24:34.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:34 smithi090 ceph-mon[41670]: 4.16 scrub ok 2024-04-14T15:24:34.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:34 smithi090 ceph-mon[30399]: 2.7 scrub starts 2024-04-14T15:24:34.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:34 smithi090 ceph-mon[30399]: 2.7 scrub ok 2024-04-14T15:24:34.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:34 smithi090 ceph-mon[30399]: 6.c scrub starts 2024-04-14T15:24:34.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:34 smithi090 ceph-mon[30399]: 6.c scrub ok 2024-04-14T15:24:34.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:34 smithi090 ceph-mon[30399]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:24:34.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:34 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-128", "mode": "writeback"}]': finished 2024-04-14T15:24:34.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:34 smithi090 ceph-mon[30399]: osdmap e731: 8 total, 8 up, 7 in 2024-04-14T15:24:34.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:34 smithi090 ceph-mon[30399]: pgmap v1080: 214 pgs: 20 creating+peering, 12 unknown, 182 active+clean; 582 KiB data, 702 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:24:34.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:34 smithi090 ceph-mon[30399]: 2.15 scrub starts 2024-04-14T15:24:34.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:34 smithi090 ceph-mon[30399]: 2.15 scrub ok 2024-04-14T15:24:34.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:34 smithi090 ceph-mon[30399]: 4.16 scrub starts 2024-04-14T15:24:34.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:34 smithi090 ceph-mon[30399]: 4.16 scrub ok 2024-04-14T15:24:34.479 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:24:34.504 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:34.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:34.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:24:34.606 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:24:34.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:34 smithi092 ceph-mon[33966]: 2.7 scrub starts 2024-04-14T15:24:34.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:34 smithi092 ceph-mon[33966]: 2.7 scrub ok 2024-04-14T15:24:34.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:34 smithi092 ceph-mon[33966]: 6.c scrub starts 2024-04-14T15:24:34.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:34 smithi092 ceph-mon[33966]: 6.c scrub ok 2024-04-14T15:24:34.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:34 smithi092 ceph-mon[33966]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:24:34.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:34 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-128", "mode": "writeback"}]': finished 2024-04-14T15:24:34.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:34 smithi092 ceph-mon[33966]: osdmap e731: 8 total, 8 up, 7 in 2024-04-14T15:24:34.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:34 smithi092 ceph-mon[33966]: pgmap v1080: 214 pgs: 20 creating+peering, 12 unknown, 182 active+clean; 582 KiB data, 702 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:24:34.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:34 smithi092 ceph-mon[33966]: 2.15 scrub starts 2024-04-14T15:24:34.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:34 smithi092 ceph-mon[33966]: 2.15 scrub ok 2024-04-14T15:24:34.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:34 smithi092 ceph-mon[33966]: 4.16 scrub starts 2024-04-14T15:24:34.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:34 smithi092 ceph-mon[33966]: 4.16 scrub ok 2024-04-14T15:24:34.631 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:34.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:34.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:24:34.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.708+0000 7f99e5e5e700 1 -- 172.21.15.90:0/3609468308 >> v1:172.21.15.90:6789/0 conn(0x7f99e010f2f0 legacy=0x7f99e01117b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:34.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.708+0000 7f99e5e5e700 1 -- 172.21.15.90:0/3609468308 shutdown_connections 2024-04-14T15:24:34.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.708+0000 7f99e5e5e700 1 -- 172.21.15.90:0/3609468308 >> 172.21.15.90:0/3609468308 conn(0x7f99e00fe3a0 msgr2=0x7f99e01007c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:24:34.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.708+0000 7f99e5e5e700 1 -- 172.21.15.90:0/3609468308 shutdown_connections 2024-04-14T15:24:34.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.708+0000 7f99e5e5e700 1 -- 172.21.15.90:0/3609468308 wait complete. 2024-04-14T15:24:34.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.709+0000 7f99e5e5e700 1 librados: starting msgr at 2024-04-14T15:24:34.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.709+0000 7f99e5e5e700 1 librados: starting objecter 2024-04-14T15:24:34.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.709+0000 7f99e5e5e700 1 Processor -- start 2024-04-14T15:24:34.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.709+0000 7f99e5e5e700 1 -- start start 2024-04-14T15:24:34.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.709+0000 7f99e5e5e700 1 librados: setting wanted keys 2024-04-14T15:24:34.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.709+0000 7f99e5e5e700 1 librados: calling monclient init 2024-04-14T15:24:34.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.709+0000 7f99e5e5e700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f99e01a7ac0 con 0x7f99e010b720 2024-04-14T15:24:34.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.709+0000 7f99e5e5e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f99e01a8ce0 con 0x7f99e010f2f0 2024-04-14T15:24:34.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.709+0000 7f99e5e5e700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f99e01a9f20 con 0x7f99e0108870 2024-04-14T15:24:34.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.709+0000 7f99deffd700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f99e010b720 0x7f99e01a27e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:55174/0 (socket says 172.21.15.90:55174) 2024-04-14T15:24:34.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.709+0000 7f99deffd700 1 -- 172.21.15.90:0/564891731 learned_addr learned my addr 172.21.15.90:0/564891731 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:24:34.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.709+0000 7f99dcff9700 10 client.?.objecter ms_handle_connect 0x7f99e010b720 2024-04-14T15:24:34.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.709+0000 7f99dcff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:24:34.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.709+0000 7f99dcff9700 10 client.?.objecter ms_handle_connect 0x7f99e010f2f0 2024-04-14T15:24:34.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.709+0000 7f99dcff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:24:34.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.710+0000 7f99dcff9700 1 -- 172.21.15.90:0/564891731 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2731652779 0 0) 0x7f99e01a7ac0 con 0x7f99e010b720 2024-04-14T15:24:34.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.710+0000 7f99dcff9700 1 -- 172.21.15.90:0/564891731 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f99c8003aa0 con 0x7f99e010b720 2024-04-14T15:24:34.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.710+0000 7f99dcff9700 1 -- 172.21.15.90:0/564891731 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3570080981 0 0) 0x7f99e01a8ce0 con 0x7f99e010f2f0 2024-04-14T15:24:34.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.710+0000 7f99dcff9700 1 -- 172.21.15.90:0/564891731 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f99e01a7ac0 con 0x7f99e010f2f0 2024-04-14T15:24:34.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.710+0000 7f99dcff9700 10 client.?.objecter ms_handle_connect 0x7f99e0108870 2024-04-14T15:24:34.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.710+0000 7f99dcff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:24:34.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.710+0000 7f99dcff9700 1 -- 172.21.15.90:0/564891731 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2912039628 0 0) 0x7f99c8003aa0 con 0x7f99e010b720 2024-04-14T15:24:34.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.710+0000 7f99dcff9700 1 -- 172.21.15.90:0/564891731 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f99e01a8ce0 con 0x7f99e010b720 2024-04-14T15:24:34.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.710+0000 7f99dcff9700 1 -- 172.21.15.90:0/564891731 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f99d40018f0 con 0x7f99e010b720 2024-04-14T15:24:34.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.710+0000 7f99dcff9700 1 -- 172.21.15.90:0/564891731 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3794084601 0 0) 0x7f99e01a7ac0 con 0x7f99e010f2f0 2024-04-14T15:24:34.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.710+0000 7f99dcff9700 1 -- 172.21.15.90:0/564891731 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f99c8003aa0 con 0x7f99e010f2f0 2024-04-14T15:24:34.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.710+0000 7f99dcff9700 1 -- 172.21.15.90:0/564891731 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 152666537 0 0) 0x7f99e01a9f20 con 0x7f99e0108870 2024-04-14T15:24:34.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.710+0000 7f99dcff9700 1 -- 172.21.15.90:0/564891731 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f99e01a7ac0 con 0x7f99e0108870 2024-04-14T15:24:34.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.710+0000 7f99dcff9700 1 -- 172.21.15.90:0/564891731 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f99d8001c00 con 0x7f99e010f2f0 2024-04-14T15:24:34.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.710+0000 7f99dcff9700 1 -- 172.21.15.90:0/564891731 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2955249292 0 0) 0x7f99e01a8ce0 con 0x7f99e010b720 2024-04-14T15:24:34.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.710+0000 7f99dcff9700 1 -- 172.21.15.90:0/564891731 >> v1:172.21.15.90:6790/0 conn(0x7f99e0108870 legacy=0x7f99e010ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:34.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.710+0000 7f99dcff9700 1 -- 172.21.15.90:0/564891731 >> v1:172.21.15.92:6789/0 conn(0x7f99e010f2f0 legacy=0x7f99e01a61c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:34.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.710+0000 7f99dcff9700 1 -- 172.21.15.90:0/564891731 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99e01ab160 con 0x7f99e010b720 2024-04-14T15:24:34.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.710+0000 7f99e5e5e700 1 -- 172.21.15.90:0/564891731 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f99e01a8f10 con 0x7f99e010b720 2024-04-14T15:24:34.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.710+0000 7f99e5e5e700 10 client.37917.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:24:34.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.710+0000 7f99e5e5e700 1 -- 172.21.15.90:0/564891731 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f99e01a93d0 con 0x7f99e010b720 2024-04-14T15:24:34.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.710+0000 7f99dcff9700 1 -- 172.21.15.90:0/564891731 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f99d4003b40 con 0x7f99e010b720 2024-04-14T15:24:34.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.710+0000 7f99e5e5e700 1 librados: init done 2024-04-14T15:24:34.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.710+0000 7f99dcff9700 1 -- 172.21.15.90:0/564891731 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f99d4005d40 con 0x7f99e010b720 2024-04-14T15:24:34.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.711+0000 7f99e5e5e700 1 -- 172.21.15.90:0/564891731 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f99cc005130 con 0x7f99e010b720 2024-04-14T15:24:34.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.712+0000 7f99dcff9700 1 -- 172.21.15.90:0/564891731 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f99d4003680 con 0x7f99e010b720 2024-04-14T15:24:34.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.712+0000 7f99dcff9700 1 -- 172.21.15.90:0/564891731 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(731..731 src has 467..731) ==== 8757+0+0 (unknown 2840716743 0 0) 0x7f99d408ac70 con 0x7f99e010b720 2024-04-14T15:24:34.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.712+0000 7f99dcff9700 10 client.37917.objecter ms_dispatch 0x7f99e0001550 osd_map(731..731 src has 467..731) 2024-04-14T15:24:34.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.712+0000 7f99dcff9700 3 client.37917.objecter handle_osd_map got epochs [731,731] > 0 2024-04-14T15:24:34.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.712+0000 7f99dcff9700 3 client.37917.objecter handle_osd_map decoding full epoch 731 2024-04-14T15:24:34.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.712+0000 7f99dcff9700 20 client.37917.objecter dump_active .. 0 homeless 2024-04-14T15:24:34.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.712+0000 7f99dcff9700 1 -- 172.21.15.90:0/564891731 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v731) ==== 54+0+16 (unknown 3703141071 0 2420000192) 0x7f99d408b070 con 0x7f99e010b720 2024-04-14T15:24:34.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.712+0000 7f99dcff9700 10 client.37917.objecter ms_handle_connect 0x7f99c8071550 2024-04-14T15:24:34.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.713+0000 7f99e5e5e700 10 client.37917.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:24:34.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.713+0000 7f99e5e5e700 15 client.37917.objecter _session_command_op_assign -1 1 2024-04-14T15:24:34.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.713+0000 7f99e5e5e700 20 client.37917.objecter _get_session s=0x7f99cc005410 osd=0 3 2024-04-14T15:24:34.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.713+0000 7f99e5e5e700 20 client.37917.objecter put_session s=0x7f99cc005410 osd=0 3 2024-04-14T15:24:34.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.713+0000 7f99e5e5e700 20 client.37917.objecter _get_session s=0x7f99cc005410 osd=0 3 2024-04-14T15:24:34.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.713+0000 7f99e5e5e700 15 client.37917.objecter _session_command_op_remove -1 1 2024-04-14T15:24:34.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.713+0000 7f99e5e5e700 20 client.37917.objecter get_session s=0x7f99cc005410 osd=0 3 2024-04-14T15:24:34.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.713+0000 7f99e5e5e700 15 client.37917.objecter _session_command_op_assign 0 1 2024-04-14T15:24:34.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.713+0000 7f99e5e5e700 20 client.37917.objecter put_session s=0x7f99cc005410 osd=0 4 2024-04-14T15:24:34.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.713+0000 7f99e5e5e700 10 client.37917.objecter _send_command 1 2024-04-14T15:24:34.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.713+0000 7f99e5e5e700 1 -- 172.21.15.90:0/564891731 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f99cc004b80 con 0x7f99cc004760 2024-04-14T15:24:34.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.713+0000 7f99dcff9700 10 client.37917.objecter ms_handle_connect 0x7f99cc004760 2024-04-14T15:24:34.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.714+0000 7f99dcff9700 1 -- 172.21.15.90:0/564891731 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f99cc004b80 con 0x7f99cc004760 2024-04-14T15:24:34.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.714+0000 7f99dcff9700 10 client.37917.objecter ms_dispatch 0x7f99e0001550 command_reply(tid 1: 0 ) 2024-04-14T15:24:34.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.714+0000 7f99dcff9700 10 client.37917.objecter _finish_command 1 = system:0 2024-04-14T15:24:34.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.714+0000 7f99dcff9700 20 client.37917.objecter put_session s=0x7f99cc005410 osd=0 4 2024-04-14T15:24:34.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.714+0000 7f99dcff9700 15 client.37917.objecter _session_command_op_remove 0 1 2024-04-14T15:24:34.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.730+0000 7f99e5e5e700 10 client.37917.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:24:34.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.730+0000 7f99e5e5e700 15 client.37917.objecter _session_command_op_assign -1 2 2024-04-14T15:24:34.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.730+0000 7f99e5e5e700 20 client.37917.objecter _get_session s=0x7f99cc005410 osd=0 3 2024-04-14T15:24:34.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.730+0000 7f99e5e5e700 20 client.37917.objecter put_session s=0x7f99cc005410 osd=0 3 2024-04-14T15:24:34.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.730+0000 7f99e5e5e700 20 client.37917.objecter _get_session s=0x7f99cc005410 osd=0 3 2024-04-14T15:24:34.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.730+0000 7f99e5e5e700 15 client.37917.objecter _session_command_op_remove -1 2 2024-04-14T15:24:34.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.730+0000 7f99e5e5e700 20 client.37917.objecter get_session s=0x7f99cc005410 osd=0 3 2024-04-14T15:24:34.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.730+0000 7f99e5e5e700 15 client.37917.objecter _session_command_op_assign 0 2 2024-04-14T15:24:34.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.730+0000 7f99e5e5e700 20 client.37917.objecter put_session s=0x7f99cc005410 osd=0 4 2024-04-14T15:24:34.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.730+0000 7f99e5e5e700 10 client.37917.objecter _send_command 2 2024-04-14T15:24:34.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.730+0000 7f99e5e5e700 1 -- 172.21.15.90:0/564891731 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f99cc007a20 con 0x7f99cc004760 2024-04-14T15:24:34.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.732+0000 7f99dcff9700 1 -- 172.21.15.90:0/564891731 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f99cc007a20 con 0x7f99cc004760 2024-04-14T15:24:34.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.732+0000 7f99dcff9700 10 client.37917.objecter ms_dispatch 0x7f99e0001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:24:34.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.732+0000 7f99dcff9700 10 client.37917.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:24:34.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.732+0000 7f99dcff9700 20 client.37917.objecter put_session s=0x7f99cc005410 osd=0 4 2024-04-14T15:24:34.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.732+0000 7f99dcff9700 15 client.37917.objecter _session_command_op_remove 0 2 2024-04-14T15:24:34.733 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:24:34.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.732+0000 7f99e5e5e700 10 client.37917.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:24:34.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.732+0000 7f99e5e5e700 15 client.37917.objecter _session_command_op_assign -1 3 2024-04-14T15:24:34.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.732+0000 7f99e5e5e700 20 client.37917.objecter _get_session s=0x7f99cc004140 osd=1 3 2024-04-14T15:24:34.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.732+0000 7f99e5e5e700 20 client.37917.objecter put_session s=0x7f99cc004140 osd=1 3 2024-04-14T15:24:34.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.732+0000 7f99e5e5e700 20 client.37917.objecter _get_session s=0x7f99cc004140 osd=1 3 2024-04-14T15:24:34.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.732+0000 7f99e5e5e700 15 client.37917.objecter _session_command_op_remove -1 3 2024-04-14T15:24:34.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.732+0000 7f99e5e5e700 20 client.37917.objecter get_session s=0x7f99cc004140 osd=1 3 2024-04-14T15:24:34.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.732+0000 7f99e5e5e700 15 client.37917.objecter _session_command_op_assign 1 3 2024-04-14T15:24:34.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.732+0000 7f99e5e5e700 20 client.37917.objecter put_session s=0x7f99cc004140 osd=1 4 2024-04-14T15:24:34.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.732+0000 7f99e5e5e700 10 client.37917.objecter _send_command 3 2024-04-14T15:24:34.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.732+0000 7f99e5e5e700 1 -- 172.21.15.90:0/564891731 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f99cc002190 con 0x7f99cc0042b0 2024-04-14T15:24:34.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.732+0000 7f99dcff9700 10 client.37917.objecter ms_handle_connect 0x7f99cc0042b0 2024-04-14T15:24:34.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.733+0000 7f99dcff9700 1 -- 172.21.15.90:0/564891731 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f99cc002190 con 0x7f99cc0042b0 2024-04-14T15:24:34.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.733+0000 7f99dcff9700 10 client.37917.objecter ms_dispatch 0x7f99e0001550 command_reply(tid 3: 0 ) 2024-04-14T15:24:34.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.733+0000 7f99dcff9700 10 client.37917.objecter _finish_command 3 = system:0 2024-04-14T15:24:34.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.733+0000 7f99dcff9700 20 client.37917.objecter put_session s=0x7f99cc004140 osd=1 4 2024-04-14T15:24:34.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.733+0000 7f99dcff9700 15 client.37917.objecter _session_command_op_remove 1 3 2024-04-14T15:24:34.735 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:24:34.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.751+0000 7f99e5e5e700 10 client.37917.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:24:34.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.751+0000 7f99e5e5e700 15 client.37917.objecter _session_command_op_assign -1 4 2024-04-14T15:24:34.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.751+0000 7f99e5e5e700 20 client.37917.objecter _get_session s=0x7f99cc004140 osd=1 3 2024-04-14T15:24:34.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.751+0000 7f99e5e5e700 20 client.37917.objecter put_session s=0x7f99cc004140 osd=1 3 2024-04-14T15:24:34.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.751+0000 7f99e5e5e700 20 client.37917.objecter _get_session s=0x7f99cc004140 osd=1 3 2024-04-14T15:24:34.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.751+0000 7f99e5e5e700 15 client.37917.objecter _session_command_op_remove -1 4 2024-04-14T15:24:34.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.751+0000 7f99e5e5e700 20 client.37917.objecter get_session s=0x7f99cc004140 osd=1 3 2024-04-14T15:24:34.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.751+0000 7f99e5e5e700 15 client.37917.objecter _session_command_op_assign 1 4 2024-04-14T15:24:34.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.751+0000 7f99e5e5e700 20 client.37917.objecter put_session s=0x7f99cc004140 osd=1 4 2024-04-14T15:24:34.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.751+0000 7f99e5e5e700 10 client.37917.objecter _send_command 4 2024-04-14T15:24:34.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.751+0000 7f99e5e5e700 1 -- 172.21.15.90:0/564891731 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f99cc001630 con 0x7f99cc0042b0 2024-04-14T15:24:34.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.753+0000 7f99dcff9700 1 -- 172.21.15.90:0/564891731 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f99cc001630 con 0x7f99cc0042b0 2024-04-14T15:24:34.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.753+0000 7f99dcff9700 10 client.37917.objecter ms_dispatch 0x7f99e0001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:24:34.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.753+0000 7f99dcff9700 10 client.37917.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:24:34.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.753+0000 7f99dcff9700 20 client.37917.objecter put_session s=0x7f99cc004140 osd=1 4 2024-04-14T15:24:34.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.753+0000 7f99dcff9700 15 client.37917.objecter _session_command_op_remove 1 4 2024-04-14T15:24:34.754 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:24:34.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.753+0000 7f99e5e5e700 10 client.37917.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:24:34.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.753+0000 7f99e5e5e700 15 client.37917.objecter _session_command_op_assign -1 5 2024-04-14T15:24:34.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.753+0000 7f99e5e5e700 20 client.37917.objecter _get_session s=0x7f99cc001a30 osd=2 3 2024-04-14T15:24:34.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.753+0000 7f99e5e5e700 20 client.37917.objecter put_session s=0x7f99cc001a30 osd=2 3 2024-04-14T15:24:34.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.753+0000 7f99e5e5e700 20 client.37917.objecter _get_session s=0x7f99cc001a30 osd=2 3 2024-04-14T15:24:34.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.753+0000 7f99e5e5e700 15 client.37917.objecter _session_command_op_remove -1 5 2024-04-14T15:24:34.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.753+0000 7f99e5e5e700 20 client.37917.objecter get_session s=0x7f99cc001a30 osd=2 3 2024-04-14T15:24:34.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.753+0000 7f99e5e5e700 15 client.37917.objecter _session_command_op_assign 2 5 2024-04-14T15:24:34.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.753+0000 7f99e5e5e700 20 client.37917.objecter put_session s=0x7f99cc001a30 osd=2 4 2024-04-14T15:24:34.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.753+0000 7f99e5e5e700 10 client.37917.objecter _send_command 5 2024-04-14T15:24:34.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.753+0000 7f99e5e5e700 1 -- 172.21.15.90:0/564891731 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f99cc010f70 con 0x7f99cc0088b0 2024-04-14T15:24:34.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.753+0000 7f99dcff9700 10 client.37917.objecter ms_handle_connect 0x7f99cc0088b0 2024-04-14T15:24:34.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.754+0000 7f99dcff9700 1 -- 172.21.15.90:0/564891731 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f99cc010f70 con 0x7f99cc0088b0 2024-04-14T15:24:34.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.754+0000 7f99dcff9700 10 client.37917.objecter ms_dispatch 0x7f99e0001550 command_reply(tid 5: 0 ) 2024-04-14T15:24:34.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.754+0000 7f99dcff9700 10 client.37917.objecter _finish_command 5 = system:0 2024-04-14T15:24:34.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.754+0000 7f99dcff9700 20 client.37917.objecter put_session s=0x7f99cc001a30 osd=2 4 2024-04-14T15:24:34.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.754+0000 7f99dcff9700 15 client.37917.objecter _session_command_op_remove 2 5 2024-04-14T15:24:34.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:34.760 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:34.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:24:34.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.771+0000 7f99e5e5e700 10 client.37917.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:24:34.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.771+0000 7f99e5e5e700 15 client.37917.objecter _session_command_op_assign -1 6 2024-04-14T15:24:34.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.771+0000 7f99e5e5e700 20 client.37917.objecter _get_session s=0x7f99cc001a30 osd=2 3 2024-04-14T15:24:34.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.771+0000 7f99e5e5e700 20 client.37917.objecter put_session s=0x7f99cc001a30 osd=2 3 2024-04-14T15:24:34.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.771+0000 7f99e5e5e700 20 client.37917.objecter _get_session s=0x7f99cc001a30 osd=2 3 2024-04-14T15:24:34.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.771+0000 7f99e5e5e700 15 client.37917.objecter _session_command_op_remove -1 6 2024-04-14T15:24:34.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.771+0000 7f99e5e5e700 20 client.37917.objecter get_session s=0x7f99cc001a30 osd=2 3 2024-04-14T15:24:34.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.771+0000 7f99e5e5e700 15 client.37917.objecter _session_command_op_assign 2 6 2024-04-14T15:24:34.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.771+0000 7f99e5e5e700 20 client.37917.objecter put_session s=0x7f99cc001a30 osd=2 4 2024-04-14T15:24:34.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.771+0000 7f99e5e5e700 10 client.37917.objecter _send_command 6 2024-04-14T15:24:34.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.771+0000 7f99e5e5e700 1 -- 172.21.15.90:0/564891731 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f99cc007f40 con 0x7f99cc0088b0 2024-04-14T15:24:34.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.773+0000 7f99dcff9700 1 -- 172.21.15.90:0/564891731 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f99cc007f40 con 0x7f99cc0088b0 2024-04-14T15:24:34.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.773+0000 7f99dcff9700 10 client.37917.objecter ms_dispatch 0x7f99e0001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:24:34.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.773+0000 7f99dcff9700 10 client.37917.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:24:34.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.773+0000 7f99dcff9700 20 client.37917.objecter put_session s=0x7f99cc001a30 osd=2 4 2024-04-14T15:24:34.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.773+0000 7f99dcff9700 15 client.37917.objecter _session_command_op_remove 2 6 2024-04-14T15:24:34.775 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:24:34.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.773+0000 7f99e5e5e700 10 client.37917.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:24:34.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.773+0000 7f99e5e5e700 15 client.37917.objecter _session_command_op_assign -1 7 2024-04-14T15:24:34.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.773+0000 7f99e5e5e700 20 client.37917.objecter _get_session s=0x7f99cc0081c0 osd=3 3 2024-04-14T15:24:34.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.773+0000 7f99e5e5e700 20 client.37917.objecter put_session s=0x7f99cc0081c0 osd=3 3 2024-04-14T15:24:34.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.773+0000 7f99e5e5e700 20 client.37917.objecter _get_session s=0x7f99cc0081c0 osd=3 3 2024-04-14T15:24:34.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.773+0000 7f99e5e5e700 15 client.37917.objecter _session_command_op_remove -1 7 2024-04-14T15:24:34.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.773+0000 7f99e5e5e700 20 client.37917.objecter get_session s=0x7f99cc0081c0 osd=3 3 2024-04-14T15:24:34.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.773+0000 7f99e5e5e700 15 client.37917.objecter _session_command_op_assign 3 7 2024-04-14T15:24:34.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.773+0000 7f99e5e5e700 20 client.37917.objecter put_session s=0x7f99cc0081c0 osd=3 4 2024-04-14T15:24:34.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.773+0000 7f99e5e5e700 10 client.37917.objecter _send_command 7 2024-04-14T15:24:34.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.773+0000 7f99e5e5e700 1 -- 172.21.15.90:0/564891731 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f99cc00d320 con 0x7f99cc00cea0 2024-04-14T15:24:34.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.773+0000 7f99dcff9700 10 client.37917.objecter ms_handle_connect 0x7f99cc00cea0 2024-04-14T15:24:34.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.774+0000 7f99dcff9700 1 -- 172.21.15.90:0/564891731 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f99cc00d320 con 0x7f99cc00cea0 2024-04-14T15:24:34.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.774+0000 7f99dcff9700 10 client.37917.objecter ms_dispatch 0x7f99e0001550 command_reply(tid 7: 0 ) 2024-04-14T15:24:34.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.774+0000 7f99dcff9700 10 client.37917.objecter _finish_command 7 = system:0 2024-04-14T15:24:34.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.774+0000 7f99dcff9700 20 client.37917.objecter put_session s=0x7f99cc0081c0 osd=3 4 2024-04-14T15:24:34.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.774+0000 7f99dcff9700 15 client.37917.objecter _session_command_op_remove 3 7 2024-04-14T15:24:34.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.790+0000 7f99e5e5e700 10 client.37917.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:24:34.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.790+0000 7f99e5e5e700 15 client.37917.objecter _session_command_op_assign -1 8 2024-04-14T15:24:34.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.790+0000 7f99e5e5e700 20 client.37917.objecter _get_session s=0x7f99cc0081c0 osd=3 3 2024-04-14T15:24:34.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.790+0000 7f99e5e5e700 20 client.37917.objecter put_session s=0x7f99cc0081c0 osd=3 3 2024-04-14T15:24:34.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.790+0000 7f99e5e5e700 20 client.37917.objecter _get_session s=0x7f99cc0081c0 osd=3 3 2024-04-14T15:24:34.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.790+0000 7f99e5e5e700 15 client.37917.objecter _session_command_op_remove -1 8 2024-04-14T15:24:34.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.790+0000 7f99e5e5e700 20 client.37917.objecter get_session s=0x7f99cc0081c0 osd=3 3 2024-04-14T15:24:34.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.790+0000 7f99e5e5e700 15 client.37917.objecter _session_command_op_assign 3 8 2024-04-14T15:24:34.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.790+0000 7f99e5e5e700 20 client.37917.objecter put_session s=0x7f99cc0081c0 osd=3 4 2024-04-14T15:24:34.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.790+0000 7f99e5e5e700 10 client.37917.objecter _send_command 8 2024-04-14T15:24:34.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.790+0000 7f99e5e5e700 1 -- 172.21.15.90:0/564891731 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f99cc00c920 con 0x7f99cc00cea0 2024-04-14T15:24:34.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.792+0000 7f99dcff9700 1 -- 172.21.15.90:0/564891731 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f99cc00c920 con 0x7f99cc00cea0 2024-04-14T15:24:34.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.792+0000 7f99dcff9700 10 client.37917.objecter ms_dispatch 0x7f99e0001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:24:34.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.792+0000 7f99dcff9700 10 client.37917.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:24:34.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.792+0000 7f99dcff9700 20 client.37917.objecter put_session s=0x7f99cc0081c0 osd=3 4 2024-04-14T15:24:34.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.792+0000 7f99dcff9700 15 client.37917.objecter _session_command_op_remove 3 8 2024-04-14T15:24:34.793 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:24:34.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.792+0000 7f99e5e5e700 10 client.37917.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:24:34.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.792+0000 7f99e5e5e700 15 client.37917.objecter _session_command_op_assign -1 9 2024-04-14T15:24:34.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.792+0000 7f99e5e5e700 20 client.37917.objecter _get_session s=0x7f99cc00c6d0 osd=4 3 2024-04-14T15:24:34.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.792+0000 7f99e5e5e700 20 client.37917.objecter put_session s=0x7f99cc00c6d0 osd=4 3 2024-04-14T15:24:34.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.792+0000 7f99e5e5e700 20 client.37917.objecter _get_session s=0x7f99cc00c6d0 osd=4 3 2024-04-14T15:24:34.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.792+0000 7f99e5e5e700 15 client.37917.objecter _session_command_op_remove -1 9 2024-04-14T15:24:34.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.792+0000 7f99e5e5e700 20 client.37917.objecter get_session s=0x7f99cc00c6d0 osd=4 3 2024-04-14T15:24:34.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.792+0000 7f99e5e5e700 15 client.37917.objecter _session_command_op_assign 4 9 2024-04-14T15:24:34.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.792+0000 7f99e5e5e700 20 client.37917.objecter put_session s=0x7f99cc00c6d0 osd=4 4 2024-04-14T15:24:34.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.792+0000 7f99e5e5e700 10 client.37917.objecter _send_command 9 2024-04-14T15:24:34.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.792+0000 7f99e5e5e700 1 -- 172.21.15.90:0/564891731 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f99cc00c840 con 0x7f99cc011e00 2024-04-14T15:24:34.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.793+0000 7f99dcff9700 10 client.37917.objecter ms_handle_connect 0x7f99cc011e00 2024-04-14T15:24:34.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.793+0000 7f99dcff9700 1 -- 172.21.15.90:0/564891731 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f99cc00c840 con 0x7f99cc011e00 2024-04-14T15:24:34.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.793+0000 7f99dcff9700 10 client.37917.objecter ms_dispatch 0x7f99e0001550 command_reply(tid 9: 0 ) 2024-04-14T15:24:34.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.793+0000 7f99dcff9700 10 client.37917.objecter _finish_command 9 = system:0 2024-04-14T15:24:34.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.793+0000 7f99dcff9700 20 client.37917.objecter put_session s=0x7f99cc00c6d0 osd=4 4 2024-04-14T15:24:34.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.793+0000 7f99dcff9700 15 client.37917.objecter _session_command_op_remove 4 9 2024-04-14T15:24:34.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.810+0000 7f99e5e5e700 10 client.37917.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:24:34.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.810+0000 7f99e5e5e700 15 client.37917.objecter _session_command_op_assign -1 10 2024-04-14T15:24:34.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.810+0000 7f99e5e5e700 20 client.37917.objecter _get_session s=0x7f99cc00c6d0 osd=4 3 2024-04-14T15:24:34.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.810+0000 7f99e5e5e700 20 client.37917.objecter put_session s=0x7f99cc00c6d0 osd=4 3 2024-04-14T15:24:34.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.810+0000 7f99e5e5e700 20 client.37917.objecter _get_session s=0x7f99cc00c6d0 osd=4 3 2024-04-14T15:24:34.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.810+0000 7f99e5e5e700 15 client.37917.objecter _session_command_op_remove -1 10 2024-04-14T15:24:34.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.810+0000 7f99e5e5e700 20 client.37917.objecter get_session s=0x7f99cc00c6d0 osd=4 3 2024-04-14T15:24:34.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.810+0000 7f99e5e5e700 15 client.37917.objecter _session_command_op_assign 4 10 2024-04-14T15:24:34.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.810+0000 7f99e5e5e700 20 client.37917.objecter put_session s=0x7f99cc00c6d0 osd=4 4 2024-04-14T15:24:34.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.810+0000 7f99e5e5e700 10 client.37917.objecter _send_command 10 2024-04-14T15:24:34.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.810+0000 7f99e5e5e700 1 -- 172.21.15.90:0/564891731 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f99cc00c840 con 0x7f99cc011e00 2024-04-14T15:24:34.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.812+0000 7f99dcff9700 1 -- 172.21.15.90:0/564891731 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f99cc00c840 con 0x7f99cc011e00 2024-04-14T15:24:34.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.812+0000 7f99dcff9700 10 client.37917.objecter ms_dispatch 0x7f99e0001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:24:34.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.812+0000 7f99dcff9700 10 client.37917.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:24:34.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.812+0000 7f99dcff9700 20 client.37917.objecter put_session s=0x7f99cc00c6d0 osd=4 4 2024-04-14T15:24:34.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.812+0000 7f99dcff9700 15 client.37917.objecter _session_command_op_remove 4 10 2024-04-14T15:24:34.812 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:24:34.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.812+0000 7f99e5e5e700 10 client.37917.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:24:34.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.812+0000 7f99e5e5e700 15 client.37917.objecter _session_command_op_assign -1 11 2024-04-14T15:24:34.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.812+0000 7f99e5e5e700 20 client.37917.objecter _get_session s=0x7f99cc00c840 osd=5 3 2024-04-14T15:24:34.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.812+0000 7f99e5e5e700 20 client.37917.objecter put_session s=0x7f99cc00c840 osd=5 3 2024-04-14T15:24:34.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.812+0000 7f99e5e5e700 20 client.37917.objecter _get_session s=0x7f99cc00c840 osd=5 3 2024-04-14T15:24:34.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.812+0000 7f99e5e5e700 15 client.37917.objecter _session_command_op_remove -1 11 2024-04-14T15:24:34.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.812+0000 7f99e5e5e700 20 client.37917.objecter get_session s=0x7f99cc00c840 osd=5 3 2024-04-14T15:24:34.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.812+0000 7f99e5e5e700 15 client.37917.objecter _session_command_op_assign 5 11 2024-04-14T15:24:34.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.812+0000 7f99e5e5e700 20 client.37917.objecter put_session s=0x7f99cc00c840 osd=5 4 2024-04-14T15:24:34.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.812+0000 7f99e5e5e700 10 client.37917.objecter _send_command 11 2024-04-14T15:24:34.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.812+0000 7f99e5e5e700 1 -- 172.21.15.90:0/564891731 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f99cc01ea10 con 0x7f99cc01ae10 2024-04-14T15:24:34.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.812+0000 7f99dcff9700 10 client.37917.objecter ms_handle_connect 0x7f99cc01ae10 2024-04-14T15:24:34.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.813+0000 7f99dcff9700 1 -- 172.21.15.90:0/564891731 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f99cc01ea10 con 0x7f99cc01ae10 2024-04-14T15:24:34.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.813+0000 7f99dcff9700 10 client.37917.objecter ms_dispatch 0x7f99e0001550 command_reply(tid 11: 0 ) 2024-04-14T15:24:34.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.813+0000 7f99dcff9700 10 client.37917.objecter _finish_command 11 = system:0 2024-04-14T15:24:34.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.813+0000 7f99dcff9700 20 client.37917.objecter put_session s=0x7f99cc00c840 osd=5 4 2024-04-14T15:24:34.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.813+0000 7f99dcff9700 15 client.37917.objecter _session_command_op_remove 5 11 2024-04-14T15:24:34.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.830+0000 7f99e5e5e700 10 client.37917.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:24:34.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.830+0000 7f99e5e5e700 15 client.37917.objecter _session_command_op_assign -1 12 2024-04-14T15:24:34.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.830+0000 7f99e5e5e700 20 client.37917.objecter _get_session s=0x7f99cc00c840 osd=5 3 2024-04-14T15:24:34.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.830+0000 7f99e5e5e700 20 client.37917.objecter put_session s=0x7f99cc00c840 osd=5 3 2024-04-14T15:24:34.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.830+0000 7f99e5e5e700 20 client.37917.objecter _get_session s=0x7f99cc00c840 osd=5 3 2024-04-14T15:24:34.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.830+0000 7f99e5e5e700 15 client.37917.objecter _session_command_op_remove -1 12 2024-04-14T15:24:34.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.830+0000 7f99e5e5e700 20 client.37917.objecter get_session s=0x7f99cc00c840 osd=5 3 2024-04-14T15:24:34.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.830+0000 7f99e5e5e700 15 client.37917.objecter _session_command_op_assign 5 12 2024-04-14T15:24:34.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.830+0000 7f99e5e5e700 20 client.37917.objecter put_session s=0x7f99cc00c840 osd=5 4 2024-04-14T15:24:34.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.830+0000 7f99e5e5e700 10 client.37917.objecter _send_command 12 2024-04-14T15:24:34.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.830+0000 7f99e5e5e700 1 -- 172.21.15.90:0/564891731 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f99cc011370 con 0x7f99cc01ae10 2024-04-14T15:24:34.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.832+0000 7f99dcff9700 1 -- 172.21.15.90:0/564891731 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f99cc011370 con 0x7f99cc01ae10 2024-04-14T15:24:34.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.832+0000 7f99dcff9700 10 client.37917.objecter ms_dispatch 0x7f99e0001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:24:34.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.832+0000 7f99dcff9700 10 client.37917.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:24:34.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.832+0000 7f99dcff9700 20 client.37917.objecter put_session s=0x7f99cc00c840 osd=5 4 2024-04-14T15:24:34.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.832+0000 7f99dcff9700 15 client.37917.objecter _session_command_op_remove 5 12 2024-04-14T15:24:34.833 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:24:34.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.832+0000 7f99e5e5e700 10 client.37917.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:24:34.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.832+0000 7f99e5e5e700 15 client.37917.objecter _session_command_op_assign -1 13 2024-04-14T15:24:34.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.832+0000 7f99e5e5e700 20 client.37917.objecter _get_session s=0x7f99cc011120 osd=6 3 2024-04-14T15:24:34.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.832+0000 7f99e5e5e700 20 client.37917.objecter put_session s=0x7f99cc011120 osd=6 3 2024-04-14T15:24:34.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.832+0000 7f99e5e5e700 20 client.37917.objecter _get_session s=0x7f99cc011120 osd=6 3 2024-04-14T15:24:34.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.832+0000 7f99e5e5e700 15 client.37917.objecter _session_command_op_remove -1 13 2024-04-14T15:24:34.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.832+0000 7f99e5e5e700 20 client.37917.objecter get_session s=0x7f99cc011120 osd=6 3 2024-04-14T15:24:34.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.832+0000 7f99e5e5e700 15 client.37917.objecter _session_command_op_assign 6 13 2024-04-14T15:24:34.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.832+0000 7f99e5e5e700 20 client.37917.objecter put_session s=0x7f99cc011120 osd=6 4 2024-04-14T15:24:34.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.832+0000 7f99e5e5e700 10 client.37917.objecter _send_command 13 2024-04-14T15:24:34.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.832+0000 7f99e5e5e700 1 -- 172.21.15.90:0/564891731 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f99cc020130 con 0x7f99cc016d30 2024-04-14T15:24:34.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.833+0000 7f99dcff9700 10 client.37917.objecter ms_handle_connect 0x7f99cc016d30 2024-04-14T15:24:34.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.834+0000 7f99dcff9700 1 -- 172.21.15.90:0/564891731 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f99cc020130 con 0x7f99cc016d30 2024-04-14T15:24:34.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.834+0000 7f99dcff9700 10 client.37917.objecter ms_dispatch 0x7f99e0001550 command_reply(tid 13: 0 ) 2024-04-14T15:24:34.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.834+0000 7f99dcff9700 10 client.37917.objecter _finish_command 13 = system:0 2024-04-14T15:24:34.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.834+0000 7f99dcff9700 20 client.37917.objecter put_session s=0x7f99cc011120 osd=6 4 2024-04-14T15:24:34.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.834+0000 7f99dcff9700 15 client.37917.objecter _session_command_op_remove 6 13 2024-04-14T15:24:34.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.850+0000 7f99e5e5e700 10 client.37917.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:24:34.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.850+0000 7f99e5e5e700 15 client.37917.objecter _session_command_op_assign -1 14 2024-04-14T15:24:34.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.850+0000 7f99e5e5e700 20 client.37917.objecter _get_session s=0x7f99cc011120 osd=6 3 2024-04-14T15:24:34.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.850+0000 7f99e5e5e700 20 client.37917.objecter put_session s=0x7f99cc011120 osd=6 3 2024-04-14T15:24:34.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.850+0000 7f99e5e5e700 20 client.37917.objecter _get_session s=0x7f99cc011120 osd=6 3 2024-04-14T15:24:34.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.850+0000 7f99e5e5e700 15 client.37917.objecter _session_command_op_remove -1 14 2024-04-14T15:24:34.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.850+0000 7f99e5e5e700 20 client.37917.objecter get_session s=0x7f99cc011120 osd=6 3 2024-04-14T15:24:34.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.850+0000 7f99e5e5e700 15 client.37917.objecter _session_command_op_assign 6 14 2024-04-14T15:24:34.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.850+0000 7f99e5e5e700 20 client.37917.objecter put_session s=0x7f99cc011120 osd=6 4 2024-04-14T15:24:34.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.850+0000 7f99e5e5e700 10 client.37917.objecter _send_command 14 2024-04-14T15:24:34.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.850+0000 7f99e5e5e700 1 -- 172.21.15.90:0/564891731 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f99cc020130 con 0x7f99cc016d30 2024-04-14T15:24:34.852 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:24:34.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.852+0000 7f99dcff9700 1 -- 172.21.15.90:0/564891731 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f99cc020130 con 0x7f99cc016d30 2024-04-14T15:24:34.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.852+0000 7f99dcff9700 10 client.37917.objecter ms_dispatch 0x7f99e0001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:24:34.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.852+0000 7f99dcff9700 10 client.37917.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:24:34.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.852+0000 7f99dcff9700 20 client.37917.objecter put_session s=0x7f99cc011120 osd=6 4 2024-04-14T15:24:34.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.852+0000 7f99dcff9700 15 client.37917.objecter _session_command_op_remove 6 14 2024-04-14T15:24:34.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.852+0000 7f99e5e5e700 10 client.37917.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:24:34.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.852+0000 7f99e5e5e700 15 client.37917.objecter _session_command_op_assign -1 15 2024-04-14T15:24:34.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.852+0000 7f99e5e5e700 20 client.37917.objecter _get_session s=0x7f99cc020130 osd=7 3 2024-04-14T15:24:34.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.852+0000 7f99e5e5e700 20 client.37917.objecter put_session s=0x7f99cc020130 osd=7 3 2024-04-14T15:24:34.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.852+0000 7f99e5e5e700 20 client.37917.objecter _get_session s=0x7f99cc020130 osd=7 3 2024-04-14T15:24:34.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.852+0000 7f99e5e5e700 15 client.37917.objecter _session_command_op_remove -1 15 2024-04-14T15:24:34.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.852+0000 7f99e5e5e700 20 client.37917.objecter get_session s=0x7f99cc020130 osd=7 3 2024-04-14T15:24:34.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.852+0000 7f99e5e5e700 15 client.37917.objecter _session_command_op_assign 7 15 2024-04-14T15:24:34.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.852+0000 7f99e5e5e700 20 client.37917.objecter put_session s=0x7f99cc020130 osd=7 4 2024-04-14T15:24:34.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.852+0000 7f99e5e5e700 10 client.37917.objecter _send_command 15 2024-04-14T15:24:34.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.852+0000 7f99e5e5e700 1 -- 172.21.15.90:0/564891731 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f99cc0115f0 con 0x7f99cc027ee0 2024-04-14T15:24:34.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.853+0000 7f99dcff9700 10 client.37917.objecter ms_handle_connect 0x7f99cc027ee0 2024-04-14T15:24:34.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.854+0000 7f99dcff9700 1 -- 172.21.15.90:0/564891731 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f99cc0115f0 con 0x7f99cc027ee0 2024-04-14T15:24:34.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.854+0000 7f99dcff9700 10 client.37917.objecter ms_dispatch 0x7f99e0001550 command_reply(tid 15: 0 ) 2024-04-14T15:24:34.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.854+0000 7f99dcff9700 10 client.37917.objecter _finish_command 15 = system:0 2024-04-14T15:24:34.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.854+0000 7f99dcff9700 20 client.37917.objecter put_session s=0x7f99cc020130 osd=7 4 2024-04-14T15:24:34.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.854+0000 7f99dcff9700 15 client.37917.objecter _session_command_op_remove 7 15 2024-04-14T15:24:34.860 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:24:34.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.870+0000 7f99e5e5e700 10 client.37917.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:24:34.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.870+0000 7f99e5e5e700 15 client.37917.objecter _session_command_op_assign -1 16 2024-04-14T15:24:34.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.870+0000 7f99e5e5e700 20 client.37917.objecter _get_session s=0x7f99cc020130 osd=7 3 2024-04-14T15:24:34.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.870+0000 7f99e5e5e700 20 client.37917.objecter put_session s=0x7f99cc020130 osd=7 3 2024-04-14T15:24:34.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.870+0000 7f99e5e5e700 20 client.37917.objecter _get_session s=0x7f99cc020130 osd=7 3 2024-04-14T15:24:34.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.870+0000 7f99e5e5e700 15 client.37917.objecter _session_command_op_remove -1 16 2024-04-14T15:24:34.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.870+0000 7f99e5e5e700 20 client.37917.objecter get_session s=0x7f99cc020130 osd=7 3 2024-04-14T15:24:34.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.870+0000 7f99e5e5e700 15 client.37917.objecter _session_command_op_assign 7 16 2024-04-14T15:24:34.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.870+0000 7f99e5e5e700 20 client.37917.objecter put_session s=0x7f99cc020130 osd=7 4 2024-04-14T15:24:34.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.870+0000 7f99e5e5e700 10 client.37917.objecter _send_command 16 2024-04-14T15:24:34.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.870+0000 7f99e5e5e700 1 -- 172.21.15.90:0/564891731 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f99cc02b9f0 con 0x7f99cc027ee0 2024-04-14T15:24:34.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.872+0000 7f99dcff9700 1 -- 172.21.15.90:0/564891731 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f99cc02b9f0 con 0x7f99cc027ee0 2024-04-14T15:24:34.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.872+0000 7f99dcff9700 10 client.37917.objecter ms_dispatch 0x7f99e0001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:24:34.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.872+0000 7f99dcff9700 10 client.37917.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:24:34.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.872+0000 7f99dcff9700 20 client.37917.objecter put_session s=0x7f99cc020130 osd=7 4 2024-04-14T15:24:34.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.872+0000 7f99dcff9700 15 client.37917.objecter _session_command_op_remove 7 16 2024-04-14T15:24:34.873 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:24:34.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.873+0000 7f99e5e5e700 10 librados: watch_flush enter 2024-04-14T15:24:34.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.873+0000 7f99e5e5e700 10 librados: watch_flush exit 2024-04-14T15:24:34.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.873+0000 7f99e5e5e700 10 client.37917.objecter close_session for osd.0 2024-04-14T15:24:34.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.873+0000 7f99e5e5e700 1 -- 172.21.15.90:0/564891731 >> v1:172.21.15.90:6801/3761806885 conn(0x7f99cc004760 legacy=0x7f99cc002780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:34.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.873+0000 7f99e5e5e700 20 client.37917.objecter put_session s=0x7f99cc005410 osd=0 1 2024-04-14T15:24:34.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.873+0000 7f99e5e5e700 10 client.37917.objecter close_session for osd.1 2024-04-14T15:24:34.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.873+0000 7f99e5e5e700 1 -- 172.21.15.90:0/564891731 >> v1:172.21.15.90:6805/4170361699 conn(0x7f99cc0042b0 legacy=0x7f99cc00ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:34.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.873+0000 7f99e5e5e700 20 client.37917.objecter put_session s=0x7f99cc004140 osd=1 1 2024-04-14T15:24:34.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.873+0000 7f99e5e5e700 10 client.37917.objecter close_session for osd.2 2024-04-14T15:24:34.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.873+0000 7f99e5e5e700 1 -- 172.21.15.90:0/564891731 >> v1:172.21.15.90:6809/1686184251 conn(0x7f99cc0088b0 legacy=0x7f99cc00f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:34.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.873+0000 7f99e5e5e700 20 client.37917.objecter put_session s=0x7f99cc001a30 osd=2 1 2024-04-14T15:24:34.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.873+0000 7f99e5e5e700 10 client.37917.objecter close_session for osd.3 2024-04-14T15:24:34.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.873+0000 7f99e5e5e700 1 -- 172.21.15.90:0/564891731 >> v1:172.21.15.90:6813/2270166774 conn(0x7f99cc00cea0 legacy=0x7f99cc014930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:34.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.873+0000 7f99e5e5e700 20 client.37917.objecter put_session s=0x7f99cc0081c0 osd=3 1 2024-04-14T15:24:34.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.873+0000 7f99e5e5e700 10 client.37917.objecter close_session for osd.4 2024-04-14T15:24:34.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.873+0000 7f99e5e5e700 1 -- 172.21.15.90:0/564891731 >> v1:172.21.15.92:6800/3887549733 conn(0x7f99cc011e00 legacy=0x7f99cc0191e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:34.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.873+0000 7f99e5e5e700 20 client.37917.objecter put_session s=0x7f99cc00c6d0 osd=4 1 2024-04-14T15:24:34.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.873+0000 7f99e5e5e700 10 client.37917.objecter close_session for osd.5 2024-04-14T15:24:34.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.873+0000 7f99e5e5e700 1 -- 172.21.15.90:0/564891731 >> v1:172.21.15.92:6804/3244561440 conn(0x7f99cc01ae10 legacy=0x7f99cc01d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:34.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.873+0000 7f99dffff700 1 -- 172.21.15.90:0/564891731 reap_dead start 2024-04-14T15:24:34.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.873+0000 7f99e5e5e700 20 client.37917.objecter put_session s=0x7f99cc00c840 osd=5 1 2024-04-14T15:24:34.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.873+0000 7f99e5e5e700 10 client.37917.objecter close_session for osd.6 2024-04-14T15:24:34.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.873+0000 7f99e5e5e700 1 -- 172.21.15.90:0/564891731 >> v1:172.21.15.92:6808/636125963 conn(0x7f99cc016d30 legacy=0x7f99cc01ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:34.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.873+0000 7f99dffff700 1 -- 172.21.15.90:0/564891731 reap_dead start 2024-04-14T15:24:34.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.873+0000 7f99e5e5e700 20 client.37917.objecter put_session s=0x7f99cc011120 osd=6 1 2024-04-14T15:24:34.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.873+0000 7f99e5e5e700 10 client.37917.objecter close_session for osd.7 2024-04-14T15:24:34.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.873+0000 7f99e5e5e700 1 -- 172.21.15.90:0/564891731 >> v1:172.21.15.92:6812/1773706716 conn(0x7f99cc027ee0 legacy=0x7f99cc02a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:34.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.873+0000 7f99e5e5e700 20 client.37917.objecter put_session s=0x7f99cc020130 osd=7 1 2024-04-14T15:24:34.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.873+0000 7f99e5e5e700 20 client.37917.objecter shutdown clearing up homeless session... 2024-04-14T15:24:34.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.873+0000 7f99e5e5e700 10 client.37917.objecter successfully canceled tick 2024-04-14T15:24:34.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.873+0000 7f99dffff700 1 -- 172.21.15.90:0/564891731 reap_dead start 2024-04-14T15:24:34.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.873+0000 7f99dffff700 1 -- 172.21.15.90:0/564891731 reap_dead start 2024-04-14T15:24:34.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.873+0000 7f99e5e5e700 1 -- 172.21.15.90:0/564891731 >> v1:172.21.15.90:6800/420935427 conn(0x7f99c8071550 legacy=0x7f99c8073a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:34.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.873+0000 7f99e5e5e700 1 -- 172.21.15.90:0/564891731 >> v1:172.21.15.90:6789/0 conn(0x7f99e010b720 legacy=0x7f99e01a27e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:34.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.873+0000 7f99e5e5e700 1 -- 172.21.15.90:0/564891731 shutdown_connections 2024-04-14T15:24:34.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.873+0000 7f99e5e5e700 1 -- 172.21.15.90:0/564891731 >> 172.21.15.90:0/564891731 conn(0x7f99e00fe3a0 msgr2=0x7f99e00ffe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:24:34.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.873+0000 7f99e5e5e700 1 -- 172.21.15.90:0/564891731 shutdown_connections 2024-04-14T15:24:34.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.873+0000 7f99e5e5e700 1 -- 172.21.15.90:0/564891731 wait complete. 2024-04-14T15:24:34.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:34.873+0000 7f99e5e5e700 1 librados: shutdown 2024-04-14T15:24:34.877 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:24:34.877 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:24:34.877 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:24:34.877 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:24:34.877 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:24:34.877 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:24:34.877 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:24:34.877 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:24:34.886 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:34.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:34.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:24:34.988 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:24:35.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:35.013 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:35.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:24:35.114 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:24:35.137 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:35.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:35.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:24:35.239 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:24:35.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:35.349 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:35.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:24:35.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:35 smithi090 ceph-mon[30399]: 5.17 scrub starts 2024-04-14T15:24:35.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:35 smithi090 ceph-mon[30399]: 5.17 scrub ok 2024-04-14T15:24:35.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:35 smithi090 ceph-mon[30399]: 4.c scrub starts 2024-04-14T15:24:35.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:35 smithi090 ceph-mon[30399]: 4.c scrub ok 2024-04-14T15:24:35.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:35 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/564891731' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:24:35.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:35 smithi090 ceph-mon[41670]: 5.17 scrub starts 2024-04-14T15:24:35.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:35 smithi090 ceph-mon[41670]: 5.17 scrub ok 2024-04-14T15:24:35.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:35 smithi090 ceph-mon[41670]: 4.c scrub starts 2024-04-14T15:24:35.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:35 smithi090 ceph-mon[41670]: 4.c scrub ok 2024-04-14T15:24:35.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:35 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/564891731' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:24:35.450 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:24:35.478 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:35.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:35.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:24:35.580 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:24:35.606 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:35.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:35.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:24:35.622 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:24:35 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:24:35.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:35 smithi092 ceph-mon[33966]: 5.17 scrub starts 2024-04-14T15:24:35.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:35 smithi092 ceph-mon[33966]: 5.17 scrub ok 2024-04-14T15:24:35.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:35 smithi092 ceph-mon[33966]: 4.c scrub starts 2024-04-14T15:24:35.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:35 smithi092 ceph-mon[33966]: 4.c scrub ok 2024-04-14T15:24:35.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:35 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/564891731' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:24:35.708 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:24:35.733 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:35.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:35.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:24:35.836 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:24:35.860 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:35.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:35.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:24:35.962 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:24:35.987 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:35.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:35.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:24:36.089 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:24:36.113 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:36.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:36.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:24:36.215 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:24:36.241 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:36.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:36.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:24:36.343 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:24:36.370 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:36.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:36.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:24:36.472 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:24:36.496 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:36.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:36.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:24:36.541 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:36 smithi090 ceph-mon[41670]: 6.13 scrub starts 2024-04-14T15:24:36.541 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:36 smithi090 ceph-mon[41670]: 6.13 scrub ok 2024-04-14T15:24:36.542 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:36 smithi090 ceph-mon[41670]: pgmap v1081: 214 pgs: 214 active+clean; 582 KiB data, 706 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 407 B/s wr, 1 op/s 2024-04-14T15:24:36.542 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:36 smithi090 ceph-mon[41670]: 4.f deep-scrub starts 2024-04-14T15:24:36.542 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:36 smithi090 ceph-mon[41670]: 4.f deep-scrub ok 2024-04-14T15:24:36.542 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:36 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:24:36.542 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:36 smithi090 ceph-mon[41670]: 5.f scrub starts 2024-04-14T15:24:36.542 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:36 smithi090 ceph-mon[41670]: 5.f scrub ok 2024-04-14T15:24:36.542 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:36 smithi090 ceph-mon[41670]: 5.1b scrub starts 2024-04-14T15:24:36.542 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:36 smithi090 ceph-mon[41670]: 5.1b scrub ok 2024-04-14T15:24:36.542 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:36 smithi090 ceph-mon[41670]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:24:36.542 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:36 smithi090 ceph-mon[41670]: osdmap e732: 8 total, 8 up, 7 in 2024-04-14T15:24:36.542 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:36 smithi090 ceph-mon[30399]: 6.13 scrub starts 2024-04-14T15:24:36.543 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:36 smithi090 ceph-mon[30399]: 6.13 scrub ok 2024-04-14T15:24:36.543 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:36 smithi090 ceph-mon[30399]: pgmap v1081: 214 pgs: 214 active+clean; 582 KiB data, 706 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 407 B/s wr, 1 op/s 2024-04-14T15:24:36.543 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:36 smithi090 ceph-mon[30399]: 4.f deep-scrub starts 2024-04-14T15:24:36.543 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:36 smithi090 ceph-mon[30399]: 4.f deep-scrub ok 2024-04-14T15:24:36.543 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:36 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:24:36.543 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:36 smithi090 ceph-mon[30399]: 5.f scrub starts 2024-04-14T15:24:36.543 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:36 smithi090 ceph-mon[30399]: 5.f scrub ok 2024-04-14T15:24:36.543 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:36 smithi090 ceph-mon[30399]: 5.1b scrub starts 2024-04-14T15:24:36.543 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:36 smithi090 ceph-mon[30399]: 5.1b scrub ok 2024-04-14T15:24:36.543 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:36 smithi090 ceph-mon[30399]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:24:36.543 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:36 smithi090 ceph-mon[30399]: osdmap e732: 8 total, 8 up, 7 in 2024-04-14T15:24:36.598 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:24:36.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:36 smithi092 ceph-mon[33966]: 6.13 scrub starts 2024-04-14T15:24:36.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:36 smithi092 ceph-mon[33966]: 6.13 scrub ok 2024-04-14T15:24:36.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:36 smithi092 ceph-mon[33966]: pgmap v1081: 214 pgs: 214 active+clean; 582 KiB data, 706 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 407 B/s wr, 1 op/s 2024-04-14T15:24:36.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:36 smithi092 ceph-mon[33966]: 4.f deep-scrub starts 2024-04-14T15:24:36.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:36 smithi092 ceph-mon[33966]: 4.f deep-scrub ok 2024-04-14T15:24:36.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:36 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:24:36.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:36 smithi092 ceph-mon[33966]: 5.f scrub starts 2024-04-14T15:24:36.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:36 smithi092 ceph-mon[33966]: 5.f scrub ok 2024-04-14T15:24:36.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:36 smithi092 ceph-mon[33966]: 5.1b scrub starts 2024-04-14T15:24:36.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:36 smithi092 ceph-mon[33966]: 5.1b scrub ok 2024-04-14T15:24:36.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:36 smithi092 ceph-mon[33966]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:24:36.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:36 smithi092 ceph-mon[33966]: osdmap e732: 8 total, 8 up, 7 in 2024-04-14T15:24:36.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:36.627 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:36.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:24:36.728 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:24:36.755 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:36.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:36.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:24:36.857 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:24:36.882 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:36.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:36.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:24:36.984 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:24:37.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:37.014 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:37.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:24:37.114 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:24:37.140 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:37.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:37.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:24:37.241 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:24:37.266 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:37.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:37.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:24:37.367 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:24:37.392 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:37.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:37.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:24:37.494 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:24:37.505 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:37 smithi090 ceph-mon[30399]: 1.4 scrub starts 2024-04-14T15:24:37.505 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:37 smithi090 ceph-mon[30399]: 1.4 scrub ok 2024-04-14T15:24:37.505 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:37 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:24:37.506 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:37 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:24:37.506 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:37 smithi090 ceph-mon[30399]: osdmap e733: 8 total, 8 up, 7 in 2024-04-14T15:24:37.506 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:37 smithi090 ceph-mon[41670]: 1.4 scrub starts 2024-04-14T15:24:37.506 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:37 smithi090 ceph-mon[41670]: 1.4 scrub ok 2024-04-14T15:24:37.506 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:37 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:24:37.506 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:37 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:24:37.506 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:37 smithi090 ceph-mon[41670]: osdmap e733: 8 total, 8 up, 7 in 2024-04-14T15:24:37.520 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:37.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:37.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:24:37.621 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:24:37.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:37 smithi092 ceph-mon[33966]: 1.4 scrub starts 2024-04-14T15:24:37.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:37 smithi092 ceph-mon[33966]: 1.4 scrub ok 2024-04-14T15:24:37.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:37 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:24:37.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:37 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:24:37.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:37 smithi092 ceph-mon[33966]: osdmap e733: 8 total, 8 up, 7 in 2024-04-14T15:24:37.650 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:37.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:37.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:24:37.752 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:24:37.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:37.760+0000 7fef025c9700 1 -- 172.21.15.90:0/1984578529 >> v1:172.21.15.90:6789/0 conn(0x7feefc107760 legacy=0x7feefc107b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:37.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:37.760+0000 7fef025c9700 1 -- 172.21.15.90:0/1984578529 shutdown_connections 2024-04-14T15:24:37.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:37.760+0000 7fef025c9700 1 -- 172.21.15.90:0/1984578529 >> 172.21.15.90:0/1984578529 conn(0x7feefc0fd290 msgr2=0x7feefc0ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:24:37.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:37.760+0000 7fef025c9700 1 -- 172.21.15.90:0/1984578529 shutdown_connections 2024-04-14T15:24:37.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:37.761+0000 7fef025c9700 1 -- 172.21.15.90:0/1984578529 wait complete. 2024-04-14T15:24:37.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:37.761+0000 7fef025c9700 1 librados: starting msgr at 2024-04-14T15:24:37.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:37.761+0000 7fef025c9700 1 librados: starting objecter 2024-04-14T15:24:37.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:37.761+0000 7fef025c9700 1 Processor -- start 2024-04-14T15:24:37.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:37.761+0000 7fef025c9700 1 -- start start 2024-04-14T15:24:37.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:37.761+0000 7fef025c9700 1 librados: setting wanted keys 2024-04-14T15:24:37.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:37.761+0000 7fef025c9700 1 librados: calling monclient init 2024-04-14T15:24:37.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:37.762+0000 7fef025c9700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7feefc1a67e0 con 0x7feefc10e1e0 2024-04-14T15:24:37.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:37.762+0000 7fef025c9700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7feefc1a79c0 con 0x7feefc107760 2024-04-14T15:24:37.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:37.762+0000 7fef025c9700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7feefc1a8c00 con 0x7feefc10a610 2024-04-14T15:24:37.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:37.762+0000 7feefb7fe700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7feefc10a610 0x7feefc1a1690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:58270/0 (socket says 172.21.15.90:58270) 2024-04-14T15:24:37.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:37.762+0000 7feefb7fe700 1 -- 172.21.15.90:0/1201676296 learned_addr learned my addr 172.21.15.90:0/1201676296 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:24:37.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:37.762+0000 7feef97fa700 10 client.?.objecter ms_handle_connect 0x7feefc107760 2024-04-14T15:24:37.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:37.762+0000 7feef97fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:24:37.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:37.762+0000 7feef97fa700 10 client.?.objecter ms_handle_connect 0x7feefc10a610 2024-04-14T15:24:37.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:37.762+0000 7feef97fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:24:37.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:37.762+0000 7feef97fa700 10 client.?.objecter ms_handle_connect 0x7feefc10e1e0 2024-04-14T15:24:37.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:37.762+0000 7feef97fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:24:37.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:37.762+0000 7feef97fa700 1 -- 172.21.15.90:0/1201676296 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1879931567 0 0) 0x7feefc1a79c0 con 0x7feefc107760 2024-04-14T15:24:37.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:37.762+0000 7feef97fa700 1 -- 172.21.15.90:0/1201676296 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feee40031f0 con 0x7feefc107760 2024-04-14T15:24:37.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:37.762+0000 7feef97fa700 1 -- 172.21.15.90:0/1201676296 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2772114536 0 0) 0x7feefc1a8c00 con 0x7feefc10a610 2024-04-14T15:24:37.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:37.762+0000 7feef97fa700 1 -- 172.21.15.90:0/1201676296 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feefc1a79c0 con 0x7feefc10a610 2024-04-14T15:24:37.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:37.762+0000 7feef97fa700 1 -- 172.21.15.90:0/1201676296 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4067794078 0 0) 0x7feefc1a67e0 con 0x7feefc10e1e0 2024-04-14T15:24:37.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:37.762+0000 7feef97fa700 1 -- 172.21.15.90:0/1201676296 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feefc1a8c00 con 0x7feefc10e1e0 2024-04-14T15:24:37.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:37.763+0000 7feef97fa700 1 -- 172.21.15.90:0/1201676296 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1685268037 0 0) 0x7feee40031f0 con 0x7feefc107760 2024-04-14T15:24:37.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:37.763+0000 7feef97fa700 1 -- 172.21.15.90:0/1201676296 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feefc1a67e0 con 0x7feefc107760 2024-04-14T15:24:37.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:37.763+0000 7feef97fa700 1 -- 172.21.15.90:0/1201676296 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7feeec004540 con 0x7feefc107760 2024-04-14T15:24:37.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:37.763+0000 7feef97fa700 1 -- 172.21.15.90:0/1201676296 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3276843139 0 0) 0x7feefc1a79c0 con 0x7feefc10a610 2024-04-14T15:24:37.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:37.763+0000 7feef97fa700 1 -- 172.21.15.90:0/1201676296 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feee40031f0 con 0x7feefc10a610 2024-04-14T15:24:37.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:37.763+0000 7feef97fa700 1 -- 172.21.15.90:0/1201676296 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 327988560 0 0) 0x7feefc1a8c00 con 0x7feefc10e1e0 2024-04-14T15:24:37.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:37.763+0000 7feef97fa700 1 -- 172.21.15.90:0/1201676296 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feefc1a79c0 con 0x7feefc10e1e0 2024-04-14T15:24:37.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:37.763+0000 7feef97fa700 1 -- 172.21.15.90:0/1201676296 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7feef0001130 con 0x7feefc10a610 2024-04-14T15:24:37.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:37.763+0000 7feef97fa700 1 -- 172.21.15.90:0/1201676296 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7feef4001bb0 con 0x7feefc10e1e0 2024-04-14T15:24:37.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:37.763+0000 7feef97fa700 1 -- 172.21.15.90:0/1201676296 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1136265693 0 0) 0x7feefc1a67e0 con 0x7feefc107760 2024-04-14T15:24:37.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:37.763+0000 7feef97fa700 1 -- 172.21.15.90:0/1201676296 >> v1:172.21.15.90:6790/0 conn(0x7feefc10a610 legacy=0x7feefc1a1690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:37.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:37.763+0000 7feef97fa700 1 -- 172.21.15.90:0/1201676296 >> v1:172.21.15.90:6789/0 conn(0x7feefc10e1e0 legacy=0x7feefc1a4e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:37.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:37.763+0000 7feef97fa700 1 -- 172.21.15.90:0/1201676296 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feefc1a9e40 con 0x7feefc107760 2024-04-14T15:24:37.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:37.763+0000 7fef025c9700 1 -- 172.21.15.90:0/1201676296 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7feefc1a7bf0 con 0x7feefc107760 2024-04-14T15:24:37.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:37.763+0000 7fef025c9700 10 client.49753.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:24:37.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:37.763+0000 7fef025c9700 1 -- 172.21.15.90:0/1201676296 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7feefc1a80b0 con 0x7feefc107760 2024-04-14T15:24:37.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:37.763+0000 7fef025c9700 1 librados: init done 2024-04-14T15:24:37.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:37.763+0000 7feef97fa700 1 -- 172.21.15.90:0/1201676296 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7feeec001b70 con 0x7feefc107760 2024-04-14T15:24:37.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:37.763+0000 7feef97fa700 1 -- 172.21.15.90:0/1201676296 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7feeec004d40 con 0x7feefc107760 2024-04-14T15:24:37.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:37.764+0000 7fef025c9700 1 -- 172.21.15.90:0/1201676296 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feee8005150 con 0x7feefc107760 2024-04-14T15:24:37.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:37.765+0000 7feef97fa700 1 -- 172.21.15.90:0/1201676296 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7feeec003760 con 0x7feefc107760 2024-04-14T15:24:37.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:37.765+0000 7feef97fa700 1 -- 172.21.15.90:0/1201676296 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(733..733 src has 467..733) ==== 8757+0+0 (unknown 1520868987 0 0) 0x7feeec089d60 con 0x7feefc107760 2024-04-14T15:24:37.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:37.765+0000 7feef97fa700 10 client.49753.objecter ms_dispatch 0x7feefc001550 osd_map(733..733 src has 467..733) 2024-04-14T15:24:37.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:37.765+0000 7feef97fa700 3 client.49753.objecter handle_osd_map got epochs [733,733] > 0 2024-04-14T15:24:37.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:37.765+0000 7feef97fa700 3 client.49753.objecter handle_osd_map decoding full epoch 733 2024-04-14T15:24:37.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:37.766+0000 7feef97fa700 20 client.49753.objecter dump_active .. 0 homeless 2024-04-14T15:24:37.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:37.766+0000 7feef97fa700 10 client.49753.objecter ms_handle_connect 0x7feee4075740 2024-04-14T15:24:37.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:37.770+0000 7feef97fa700 1 -- 172.21.15.90:0/1201676296 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7feeec056520 con 0x7feefc107760 2024-04-14T15:24:37.777 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:37.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:37.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:24:37.836 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-14T15:24:37.879 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:24:37.902 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:37.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:37.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:24:37.966 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-14T15:24:37.966 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-14T15:24:37.966 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-04-14T15:24:37.966 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph pg dump --format=json 2024-04-14T15:24:37.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:37.990+0000 7fef025c9700 1 -- 172.21.15.90:0/1201676296 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7feee8005440 con 0x7feefc107760 2024-04-14T15:24:38.004 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:24:38.028 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:38.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:38.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:24:38.129 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:24:38.156 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:38.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:38.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:24:38.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:38.244+0000 7feef97fa700 1 -- 172.21.15.90:0/1201676296 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v734) ==== 87+0+0 (unknown 3584876365 0 0) 0x7feeec05a1d0 con 0x7feefc107760 2024-04-14T15:24:38.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:38.247+0000 7fef025c9700 10 librados: watch_flush enter 2024-04-14T15:24:38.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:38.247+0000 7fef025c9700 10 librados: watch_flush exit 2024-04-14T15:24:38.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:38.247+0000 7fef025c9700 20 client.49753.objecter shutdown clearing up homeless session... 2024-04-14T15:24:38.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:38.247+0000 7fef025c9700 10 client.49753.objecter successfully canceled tick 2024-04-14T15:24:38.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:38.247+0000 7fef025c9700 1 -- 172.21.15.90:0/1201676296 >> v1:172.21.15.90:6800/420935427 conn(0x7feee4075740 legacy=0x7feee4077c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:38.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:38.247+0000 7fef025c9700 1 -- 172.21.15.90:0/1201676296 >> v1:172.21.15.92:6789/0 conn(0x7feefc107760 legacy=0x7feefc10d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:38.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:38.248+0000 7fef025c9700 1 -- 172.21.15.90:0/1201676296 shutdown_connections 2024-04-14T15:24:38.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:38.248+0000 7fef025c9700 1 -- 172.21.15.90:0/1201676296 >> 172.21.15.90:0/1201676296 conn(0x7feefc0fd290 msgr2=0x7feefc100340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:24:38.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:38.248+0000 7fef025c9700 1 -- 172.21.15.90:0/1201676296 shutdown_connections 2024-04-14T15:24:38.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:38.248+0000 7fef025c9700 1 -- 172.21.15.90:0/1201676296 wait complete. 2024-04-14T15:24:38.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:38.248+0000 7fef025c9700 1 librados: shutdown 2024-04-14T15:24:38.250 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is set 2024-04-14T15:24:38.258 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:24:38.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:38.286 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:38.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:24:38.387 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:24:38.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:38.415 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:38.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:24:38.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:38 smithi090 ceph-mon[41670]: 1.8 scrub starts 2024-04-14T15:24:38.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:38 smithi090 ceph-mon[41670]: 1.8 scrub ok 2024-04-14T15:24:38.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:38 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:24:38.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:38 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:24:38.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:38 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:24:38.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:38 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:24:38.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:38 smithi090 ceph-mon[41670]: pgmap v1084: 214 pgs: 214 active+clean; 582 KiB data, 706 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 408 B/s wr, 1 op/s 2024-04-14T15:24:38.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:38 smithi090 ceph-mon[41670]: 11.e scrub starts 2024-04-14T15:24:38.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:38 smithi090 ceph-mon[41670]: 11.e scrub ok 2024-04-14T15:24:38.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:38 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1201676296' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:24:38.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:38 smithi090 ceph-mon[41670]: from='client.49753 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:24:38.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:38 smithi090 ceph-mon[30399]: 1.8 scrub starts 2024-04-14T15:24:38.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:38 smithi090 ceph-mon[30399]: 1.8 scrub ok 2024-04-14T15:24:38.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:38 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:24:38.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:38 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:24:38.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:38 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:24:38.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:38 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:24:38.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:38 smithi090 ceph-mon[30399]: pgmap v1084: 214 pgs: 214 active+clean; 582 KiB data, 706 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 408 B/s wr, 1 op/s 2024-04-14T15:24:38.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:38 smithi090 ceph-mon[30399]: 11.e scrub starts 2024-04-14T15:24:38.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:38 smithi090 ceph-mon[30399]: 11.e scrub ok 2024-04-14T15:24:38.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:38 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1201676296' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:24:38.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:38 smithi090 ceph-mon[30399]: from='client.49753 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:24:38.450 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:24:38.515 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:24:38.538 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:38.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:38.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:24:38.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:38 smithi092 ceph-mon[33966]: 1.8 scrub starts 2024-04-14T15:24:38.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:38 smithi092 ceph-mon[33966]: 1.8 scrub ok 2024-04-14T15:24:38.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:38 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:24:38.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:38 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:24:38.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:38 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:24:38.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:38 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:24:38.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:38 smithi092 ceph-mon[33966]: pgmap v1084: 214 pgs: 214 active+clean; 582 KiB data, 706 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 408 B/s wr, 1 op/s 2024-04-14T15:24:38.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:38 smithi092 ceph-mon[33966]: 11.e scrub starts 2024-04-14T15:24:38.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:38 smithi092 ceph-mon[33966]: 11.e scrub ok 2024-04-14T15:24:38.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:38 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1201676296' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:24:38.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:38 smithi092 ceph-mon[33966]: from='client.49753 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:24:38.640 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:24:38.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:38.664 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:38.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:24:38.765 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:24:38.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:38.789 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:38.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:24:38.890 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:24:38.913 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:38.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:38.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:24:39.015 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:24:39.032 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:24:39.039 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:39.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:39.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:24:39.061 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-14T15:24:39.061 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:24:39.141 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:24:39.169 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:39.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:39.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:24:39.271 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:24:39.296 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:39.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:39.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:24:39.398 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:24:39.423 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:39.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:39.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:24:39.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:39 smithi090 ceph-mon[41670]: 5.9 scrub starts 2024-04-14T15:24:39.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:39 smithi090 ceph-mon[41670]: 5.9 scrub ok 2024-04-14T15:24:39.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:39 smithi090 ceph-mon[41670]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:24:39.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:39 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:24:39.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:39 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:24:39.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:39 smithi090 ceph-mon[41670]: from='client.49753 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-14T15:24:39.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:39 smithi090 ceph-mon[41670]: osdmap e734: 8 total, 8 up, 7 in 2024-04-14T15:24:39.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:39 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:24:39.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:39 smithi090 ceph-mon[30399]: 5.9 scrub starts 2024-04-14T15:24:39.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:39 smithi090 ceph-mon[30399]: 5.9 scrub ok 2024-04-14T15:24:39.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:39 smithi090 ceph-mon[30399]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:24:39.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:39 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:24:39.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:39 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:24:39.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:39 smithi090 ceph-mon[30399]: from='client.49753 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-14T15:24:39.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:39 smithi090 ceph-mon[30399]: osdmap e734: 8 total, 8 up, 7 in 2024-04-14T15:24:39.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:39 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:24:39.525 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:24:39.549 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:39.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:39.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:24:39.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:39 smithi092 ceph-mon[33966]: 5.9 scrub starts 2024-04-14T15:24:39.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:39 smithi092 ceph-mon[33966]: 5.9 scrub ok 2024-04-14T15:24:39.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:39 smithi092 ceph-mon[33966]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:24:39.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:39 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:24:39.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:39 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:24:39.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:39 smithi092 ceph-mon[33966]: from='client.49753 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-14T15:24:39.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:39 smithi092 ceph-mon[33966]: osdmap e734: 8 total, 8 up, 7 in 2024-04-14T15:24:39.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:39 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:24:39.650 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:24:39.678 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:39.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:39.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:24:39.780 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:24:39.808 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:39.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:39.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:24:39.909 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:24:39.936 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:39.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:39.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:24:40.038 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:24:40.062 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:40.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:40.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:24:40.163 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:24:40.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:40.192 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:40.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:24:40.293 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:24:40.316 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:40.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:40.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:24:40.418 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:24:40.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:40.442 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:40.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:24:40.523 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:40 smithi090 ceph-mon[41670]: pgmap v1086: 214 pgs: 214 active+clean; 582 KiB data, 707 MiB used, 625 GiB / 626 GiB avail; 1.8 KiB/s rd, 341 B/s wr, 2 op/s 2024-04-14T15:24:40.523 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:40 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]': finished 2024-04-14T15:24:40.523 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:40 smithi090 ceph-mon[41670]: osdmap e735: 8 total, 8 up, 7 in 2024-04-14T15:24:40.523 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:40 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-128"}]: dispatch 2024-04-14T15:24:40.523 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:40 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-128"}]: dispatch 2024-04-14T15:24:40.524 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:40 smithi090 ceph-mon[30399]: pgmap v1086: 214 pgs: 214 active+clean; 582 KiB data, 707 MiB used, 625 GiB / 626 GiB avail; 1.8 KiB/s rd, 341 B/s wr, 2 op/s 2024-04-14T15:24:40.524 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:40 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]': finished 2024-04-14T15:24:40.524 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:40 smithi090 ceph-mon[30399]: osdmap e735: 8 total, 8 up, 7 in 2024-04-14T15:24:40.524 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:40 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-128"}]: dispatch 2024-04-14T15:24:40.524 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:40 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-128"}]: dispatch 2024-04-14T15:24:40.543 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:24:40.548 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-14T15:24:40.570 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:40.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:40.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:24:40.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:40 smithi092 ceph-mon[33966]: pgmap v1086: 214 pgs: 214 active+clean; 582 KiB data, 707 MiB used, 625 GiB / 626 GiB avail; 1.8 KiB/s rd, 341 B/s wr, 2 op/s 2024-04-14T15:24:40.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:40 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]': finished 2024-04-14T15:24:40.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:40 smithi092 ceph-mon[33966]: osdmap e735: 8 total, 8 up, 7 in 2024-04-14T15:24:40.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:40 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-128"}]: dispatch 2024-04-14T15:24:40.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:40 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-128"}]: dispatch 2024-04-14T15:24:40.673 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:24:40.697 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:40.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:40.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:24:40.799 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:24:40.827 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:40.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:40.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:24:40.929 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:24:40.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:40.954 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:40.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:24:41.055 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:24:41.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:41.081 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:41.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:24:41.182 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:24:41.207 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:41.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:41.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:24:41.309 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:24:41.338 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:41.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:41.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:24:41.440 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:24:41.464 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:41.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:41.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:24:41.565 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:24:41.589 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:41.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:41.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:24:41.617 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:41 smithi090 ceph-mon[41670]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:24:41.617 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:41 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-128"}]': finished 2024-04-14T15:24:41.617 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:41 smithi090 ceph-mon[41670]: osdmap e736: 8 total, 8 up, 7 in 2024-04-14T15:24:41.617 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:41 smithi090 ceph-mon[30399]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:24:41.617 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:41 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-128"}]': finished 2024-04-14T15:24:41.617 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:41 smithi090 ceph-mon[30399]: osdmap e736: 8 total, 8 up, 7 in 2024-04-14T15:24:41.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:41 smithi092 ceph-mon[33966]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:24:41.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:41 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-128"}]': finished 2024-04-14T15:24:41.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:41 smithi092 ceph-mon[33966]: osdmap e736: 8 total, 8 up, 7 in 2024-04-14T15:24:41.691 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:24:41.715 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:41.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:41.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:24:41.817 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:24:41.845 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:41.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:41.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:24:41.947 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:24:41.969 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-14T15:24:41.975 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:41.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:41.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:24:42.076 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:24:42.104 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:42.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:42.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:24:42.205 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:24:42.233 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:42.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:42.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:24:42.334 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:24:42.361 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:42.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:42.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:24:42.463 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:24:42.490 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:42.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:42.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:24:42.556 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:42 smithi090 ceph-mon[41670]: pgmap v1089: 214 pgs: 214 active+clean; 582 KiB data, 707 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-14T15:24:42.556 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:42 smithi090 ceph-mon[41670]: osdmap e737: 8 total, 8 up, 7 in 2024-04-14T15:24:42.556 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:42 smithi090 ceph-mon[41670]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:24:42.557 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:42 smithi090 ceph-mon[30399]: pgmap v1089: 214 pgs: 214 active+clean; 582 KiB data, 707 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-14T15:24:42.557 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:42 smithi090 ceph-mon[30399]: osdmap e737: 8 total, 8 up, 7 in 2024-04-14T15:24:42.557 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:42 smithi090 ceph-mon[30399]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:24:42.592 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:24:42.616 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:42.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:42.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:24:42.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:42 smithi092 ceph-mon[33966]: pgmap v1089: 214 pgs: 214 active+clean; 582 KiB data, 707 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-14T15:24:42.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:42 smithi092 ceph-mon[33966]: osdmap e737: 8 total, 8 up, 7 in 2024-04-14T15:24:42.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:42 smithi092 ceph-mon[33966]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:24:42.706 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:24:42.718 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:24:42.745 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:42.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:42.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:24:42.847 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:24:42.874 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:42.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:42.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:24:42.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.910+0000 7f09d9adc700 1 -- 172.21.15.90:0/1063918599 >> v1:172.21.15.90:6790/0 conn(0x7f09d4107890 legacy=0x7f09d4107c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:42.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.910+0000 7f09d9adc700 1 -- 172.21.15.90:0/1063918599 shutdown_connections 2024-04-14T15:24:42.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.910+0000 7f09d9adc700 1 -- 172.21.15.90:0/1063918599 >> 172.21.15.90:0/1063918599 conn(0x7f09d40fd290 msgr2=0x7f09d40ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:24:42.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.910+0000 7f09d9adc700 1 -- 172.21.15.90:0/1063918599 shutdown_connections 2024-04-14T15:24:42.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.910+0000 7f09d9adc700 1 -- 172.21.15.90:0/1063918599 wait complete. 2024-04-14T15:24:42.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.911+0000 7f09d9adc700 1 librados: starting msgr at 2024-04-14T15:24:42.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.911+0000 7f09d9adc700 1 librados: starting objecter 2024-04-14T15:24:42.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.911+0000 7f09d9adc700 1 Processor -- start 2024-04-14T15:24:42.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.911+0000 7f09d9adc700 1 -- start start 2024-04-14T15:24:42.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.911+0000 7f09d9adc700 1 librados: setting wanted keys 2024-04-14T15:24:42.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.911+0000 7f09d9adc700 1 librados: calling monclient init 2024-04-14T15:24:42.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.911+0000 7f09d9adc700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f09d41a9d60 con 0x7f09d4107890 2024-04-14T15:24:42.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.911+0000 7f09d9adc700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f09d41aaf40 con 0x7f09d410a740 2024-04-14T15:24:42.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.911+0000 7f09d9adc700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f09d41ac180 con 0x7f09d410e310 2024-04-14T15:24:42.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.911+0000 7f09d37fe700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f09d4107890 0x7f09d410db50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:51760/0 (socket says 172.21.15.90:51760) 2024-04-14T15:24:42.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.911+0000 7f09d37fe700 1 -- 172.21.15.90:0/1340141548 learned_addr learned my addr 172.21.15.90:0/1340141548 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:24:42.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.911+0000 7f09d17fa700 10 client.?.objecter ms_handle_connect 0x7f09d4107890 2024-04-14T15:24:42.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.911+0000 7f09d17fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:24:42.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.911+0000 7f09d17fa700 10 client.?.objecter ms_handle_connect 0x7f09d410e310 2024-04-14T15:24:42.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.911+0000 7f09d17fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:24:42.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.911+0000 7f09d17fa700 1 -- 172.21.15.90:0/1340141548 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3228272592 0 0) 0x7f09d41a9d60 con 0x7f09d4107890 2024-04-14T15:24:42.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.911+0000 7f09d17fa700 1 -- 172.21.15.90:0/1340141548 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f09c0003c50 con 0x7f09d4107890 2024-04-14T15:24:42.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.911+0000 7f09d17fa700 10 client.?.objecter ms_handle_connect 0x7f09d410a740 2024-04-14T15:24:42.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.911+0000 7f09d17fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:24:42.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.911+0000 7f09d17fa700 1 -- 172.21.15.90:0/1340141548 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3072191548 0 0) 0x7f09d41ac180 con 0x7f09d410e310 2024-04-14T15:24:42.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.912+0000 7f09d17fa700 1 -- 172.21.15.90:0/1340141548 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f09d41a9d60 con 0x7f09d410e310 2024-04-14T15:24:42.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.912+0000 7f09d17fa700 1 -- 172.21.15.90:0/1340141548 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2774269617 0 0) 0x7f09d41aaf40 con 0x7f09d410a740 2024-04-14T15:24:42.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.912+0000 7f09d17fa700 1 -- 172.21.15.90:0/1340141548 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f09d41ac180 con 0x7f09d410a740 2024-04-14T15:24:42.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.912+0000 7f09d17fa700 1 -- 172.21.15.90:0/1340141548 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1602179747 0 0) 0x7f09c0003c50 con 0x7f09d4107890 2024-04-14T15:24:42.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.912+0000 7f09d17fa700 1 -- 172.21.15.90:0/1340141548 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f09d41aaf40 con 0x7f09d4107890 2024-04-14T15:24:42.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.912+0000 7f09d17fa700 1 -- 172.21.15.90:0/1340141548 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f09c4001350 con 0x7f09d4107890 2024-04-14T15:24:42.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.912+0000 7f09d17fa700 1 -- 172.21.15.90:0/1340141548 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2471962816 0 0) 0x7f09d41a9d60 con 0x7f09d410e310 2024-04-14T15:24:42.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.912+0000 7f09d17fa700 1 -- 172.21.15.90:0/1340141548 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f09c0003c50 con 0x7f09d410e310 2024-04-14T15:24:42.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.912+0000 7f09d17fa700 1 -- 172.21.15.90:0/1340141548 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f09cc001b70 con 0x7f09d410e310 2024-04-14T15:24:42.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.912+0000 7f09d17fa700 1 -- 172.21.15.90:0/1340141548 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2747523184 0 0) 0x7f09d41ac180 con 0x7f09d410a740 2024-04-14T15:24:42.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.912+0000 7f09d17fa700 1 -- 172.21.15.90:0/1340141548 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f09d41a9d60 con 0x7f09d410a740 2024-04-14T15:24:42.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.912+0000 7f09d17fa700 1 -- 172.21.15.90:0/1340141548 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f09bc001a30 con 0x7f09d410a740 2024-04-14T15:24:42.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.912+0000 7f09d17fa700 1 -- 172.21.15.90:0/1340141548 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 152507605 0 0) 0x7f09d41aaf40 con 0x7f09d4107890 2024-04-14T15:24:42.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.912+0000 7f09d17fa700 1 -- 172.21.15.90:0/1340141548 >> v1:172.21.15.90:6790/0 conn(0x7f09d410e310 legacy=0x7f09d41a83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:42.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.912+0000 7f09d17fa700 1 -- 172.21.15.90:0/1340141548 >> v1:172.21.15.92:6789/0 conn(0x7f09d410a740 legacy=0x7f09d41a4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:42.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.912+0000 7f09d17fa700 1 -- 172.21.15.90:0/1340141548 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09d41ad3c0 con 0x7f09d4107890 2024-04-14T15:24:42.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.912+0000 7f09d9adc700 1 -- 172.21.15.90:0/1340141548 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f09d41ac3b0 con 0x7f09d4107890 2024-04-14T15:24:42.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.912+0000 7f09d9adc700 10 client.37929.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:24:42.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.912+0000 7f09d9adc700 1 -- 172.21.15.90:0/1340141548 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f09d41ac870 con 0x7f09d4107890 2024-04-14T15:24:42.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.912+0000 7f09d9adc700 1 librados: init done 2024-04-14T15:24:42.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.912+0000 7f09d17fa700 1 -- 172.21.15.90:0/1340141548 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f09c4004680 con 0x7f09d4107890 2024-04-14T15:24:42.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.912+0000 7f09d17fa700 1 -- 172.21.15.90:0/1340141548 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f09c40050a0 con 0x7f09d4107890 2024-04-14T15:24:42.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.913+0000 7f09d9adc700 1 -- 172.21.15.90:0/1340141548 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f09d40693d0 con 0x7f09d4107890 2024-04-14T15:24:42.939 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:24:42.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.914+0000 7f09d17fa700 1 -- 172.21.15.90:0/1340141548 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f09c40053e0 con 0x7f09d4107890 2024-04-14T15:24:42.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.914+0000 7f09d17fa700 1 -- 172.21.15.90:0/1340141548 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(738..738 src has 467..738) ==== 8749+0+0 (unknown 2822462751 0 0) 0x7f09c408aeb0 con 0x7f09d4107890 2024-04-14T15:24:42.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.914+0000 7f09d17fa700 10 client.37929.objecter ms_dispatch 0x7f09d4001550 osd_map(738..738 src has 467..738) 2024-04-14T15:24:42.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.914+0000 7f09d17fa700 3 client.37929.objecter handle_osd_map got epochs [738,738] > 0 2024-04-14T15:24:42.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.914+0000 7f09d17fa700 3 client.37929.objecter handle_osd_map decoding full epoch 738 2024-04-14T15:24:42.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.914+0000 7f09d17fa700 20 client.37929.objecter dump_active .. 0 homeless 2024-04-14T15:24:42.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.914+0000 7f09d17fa700 1 -- 172.21.15.90:0/1340141548 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v738) ==== 54+0+16 (unknown 2008594543 0 2420000192) 0x7f09c408d310 con 0x7f09d4107890 2024-04-14T15:24:42.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.914+0000 7f09d17fa700 10 client.37929.objecter ms_handle_connect 0x7f09c0071460 2024-04-14T15:24:42.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.914+0000 7f09d9adc700 10 client.37929.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:24:42.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.914+0000 7f09d9adc700 15 client.37929.objecter _session_command_op_assign -1 1 2024-04-14T15:24:42.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.914+0000 7f09d9adc700 20 client.37929.objecter _get_session s=0x7f09d41122e0 osd=0 3 2024-04-14T15:24:42.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.914+0000 7f09d9adc700 20 client.37929.objecter put_session s=0x7f09d41122e0 osd=0 3 2024-04-14T15:24:42.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.914+0000 7f09d9adc700 20 client.37929.objecter _get_session s=0x7f09d41122e0 osd=0 3 2024-04-14T15:24:42.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.914+0000 7f09d9adc700 15 client.37929.objecter _session_command_op_remove -1 1 2024-04-14T15:24:42.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.914+0000 7f09d9adc700 20 client.37929.objecter get_session s=0x7f09d41122e0 osd=0 3 2024-04-14T15:24:42.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.914+0000 7f09d9adc700 15 client.37929.objecter _session_command_op_assign 0 1 2024-04-14T15:24:42.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.914+0000 7f09d9adc700 20 client.37929.objecter put_session s=0x7f09d41122e0 osd=0 4 2024-04-14T15:24:42.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.914+0000 7f09d9adc700 10 client.37929.objecter _send_command 1 2024-04-14T15:24:42.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.915+0000 7f09d9adc700 1 -- 172.21.15.90:0/1340141548 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f09d41058c0 con 0x7f09d41ab110 2024-04-14T15:24:42.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.915+0000 7f09d17fa700 10 client.37929.objecter ms_handle_connect 0x7f09d41ab110 2024-04-14T15:24:42.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.915+0000 7f09d17fa700 1 -- 172.21.15.90:0/1340141548 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f09d41058c0 con 0x7f09d41ab110 2024-04-14T15:24:42.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.915+0000 7f09d17fa700 10 client.37929.objecter ms_dispatch 0x7f09d4001550 command_reply(tid 1: 0 ) 2024-04-14T15:24:42.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.915+0000 7f09d17fa700 10 client.37929.objecter _finish_command 1 = system:0 2024-04-14T15:24:42.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.915+0000 7f09d17fa700 20 client.37929.objecter put_session s=0x7f09d41122e0 osd=0 4 2024-04-14T15:24:42.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.915+0000 7f09d17fa700 15 client.37929.objecter _session_command_op_remove 0 1 2024-04-14T15:24:42.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.932+0000 7f09d9adc700 10 client.37929.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:24:42.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.932+0000 7f09d9adc700 15 client.37929.objecter _session_command_op_assign -1 2 2024-04-14T15:24:42.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.932+0000 7f09d9adc700 20 client.37929.objecter _get_session s=0x7f09d41122e0 osd=0 3 2024-04-14T15:24:42.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.932+0000 7f09d9adc700 20 client.37929.objecter put_session s=0x7f09d41122e0 osd=0 3 2024-04-14T15:24:42.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.932+0000 7f09d9adc700 20 client.37929.objecter _get_session s=0x7f09d41122e0 osd=0 3 2024-04-14T15:24:42.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.932+0000 7f09d9adc700 15 client.37929.objecter _session_command_op_remove -1 2 2024-04-14T15:24:42.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.932+0000 7f09d9adc700 20 client.37929.objecter get_session s=0x7f09d41122e0 osd=0 3 2024-04-14T15:24:42.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.932+0000 7f09d9adc700 15 client.37929.objecter _session_command_op_assign 0 2 2024-04-14T15:24:42.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.932+0000 7f09d9adc700 20 client.37929.objecter put_session s=0x7f09d41122e0 osd=0 4 2024-04-14T15:24:42.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.932+0000 7f09d9adc700 10 client.37929.objecter _send_command 2 2024-04-14T15:24:42.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.932+0000 7f09d9adc700 1 -- 172.21.15.90:0/1340141548 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f09d4102ec0 con 0x7f09d41ab110 2024-04-14T15:24:42.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.933+0000 7f09d17fa700 1 -- 172.21.15.90:0/1340141548 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f09d4102ec0 con 0x7f09d41ab110 2024-04-14T15:24:42.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.933+0000 7f09d17fa700 10 client.37929.objecter ms_dispatch 0x7f09d4001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:24:42.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.933+0000 7f09d17fa700 10 client.37929.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:24:42.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.933+0000 7f09d17fa700 20 client.37929.objecter put_session s=0x7f09d41122e0 osd=0 4 2024-04-14T15:24:42.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.933+0000 7f09d17fa700 15 client.37929.objecter _session_command_op_remove 0 2 2024-04-14T15:24:42.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.934+0000 7f09d9adc700 10 client.37929.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:24:42.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.934+0000 7f09d9adc700 15 client.37929.objecter _session_command_op_assign -1 3 2024-04-14T15:24:42.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.934+0000 7f09d9adc700 20 client.37929.objecter _get_session s=0x7f09d4103070 osd=1 3 2024-04-14T15:24:42.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.934+0000 7f09d9adc700 20 client.37929.objecter put_session s=0x7f09d4103070 osd=1 3 2024-04-14T15:24:42.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.934+0000 7f09d9adc700 20 client.37929.objecter _get_session s=0x7f09d4103070 osd=1 3 2024-04-14T15:24:42.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.934+0000 7f09d9adc700 15 client.37929.objecter _session_command_op_remove -1 3 2024-04-14T15:24:42.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.934+0000 7f09d9adc700 20 client.37929.objecter get_session s=0x7f09d4103070 osd=1 3 2024-04-14T15:24:42.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.934+0000 7f09d9adc700 15 client.37929.objecter _session_command_op_assign 1 3 2024-04-14T15:24:42.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.934+0000 7f09d9adc700 20 client.37929.objecter put_session s=0x7f09d4103070 osd=1 4 2024-04-14T15:24:42.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.934+0000 7f09d9adc700 10 client.37929.objecter _send_command 3 2024-04-14T15:24:42.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.934+0000 7f09d9adc700 1 -- 172.21.15.90:0/1340141548 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f09d41abe50 con 0x7f09d410ecd0 2024-04-14T15:24:42.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.934+0000 7f09d17fa700 10 client.37929.objecter ms_handle_connect 0x7f09d410ecd0 2024-04-14T15:24:42.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.935+0000 7f09d17fa700 1 -- 172.21.15.90:0/1340141548 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f09d41abe50 con 0x7f09d410ecd0 2024-04-14T15:24:42.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.935+0000 7f09d17fa700 10 client.37929.objecter ms_dispatch 0x7f09d4001550 command_reply(tid 3: 0 ) 2024-04-14T15:24:42.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.935+0000 7f09d17fa700 10 client.37929.objecter _finish_command 3 = system:0 2024-04-14T15:24:42.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.935+0000 7f09d17fa700 20 client.37929.objecter put_session s=0x7f09d4103070 osd=1 4 2024-04-14T15:24:42.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.935+0000 7f09d17fa700 15 client.37929.objecter _session_command_op_remove 1 3 2024-04-14T15:24:42.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.951+0000 7f09d9adc700 10 client.37929.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:24:42.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.951+0000 7f09d9adc700 15 client.37929.objecter _session_command_op_assign -1 4 2024-04-14T15:24:42.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.951+0000 7f09d9adc700 20 client.37929.objecter _get_session s=0x7f09d4103070 osd=1 3 2024-04-14T15:24:42.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.951+0000 7f09d9adc700 20 client.37929.objecter put_session s=0x7f09d4103070 osd=1 3 2024-04-14T15:24:42.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.951+0000 7f09d9adc700 20 client.37929.objecter _get_session s=0x7f09d4103070 osd=1 3 2024-04-14T15:24:42.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.951+0000 7f09d9adc700 15 client.37929.objecter _session_command_op_remove -1 4 2024-04-14T15:24:42.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.951+0000 7f09d9adc700 20 client.37929.objecter get_session s=0x7f09d4103070 osd=1 3 2024-04-14T15:24:42.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.951+0000 7f09d9adc700 15 client.37929.objecter _session_command_op_assign 1 4 2024-04-14T15:24:42.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.951+0000 7f09d9adc700 20 client.37929.objecter put_session s=0x7f09d4103070 osd=1 4 2024-04-14T15:24:42.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.951+0000 7f09d9adc700 10 client.37929.objecter _send_command 4 2024-04-14T15:24:42.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.951+0000 7f09d9adc700 1 -- 172.21.15.90:0/1340141548 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f09d402e080 con 0x7f09d410ecd0 2024-04-14T15:24:42.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.953+0000 7f09d17fa700 1 -- 172.21.15.90:0/1340141548 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f09d402e080 con 0x7f09d410ecd0 2024-04-14T15:24:42.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.953+0000 7f09d17fa700 10 client.37929.objecter ms_dispatch 0x7f09d4001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:24:42.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.953+0000 7f09d17fa700 10 client.37929.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:24:42.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.953+0000 7f09d17fa700 20 client.37929.objecter put_session s=0x7f09d4103070 osd=1 4 2024-04-14T15:24:42.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.953+0000 7f09d17fa700 15 client.37929.objecter _session_command_op_remove 1 4 2024-04-14T15:24:42.953 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:24:42.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.953+0000 7f09d9adc700 10 client.37929.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:24:42.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.953+0000 7f09d9adc700 15 client.37929.objecter _session_command_op_assign -1 5 2024-04-14T15:24:42.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.953+0000 7f09d9adc700 20 client.37929.objecter _get_session s=0x7f09d41b1670 osd=2 3 2024-04-14T15:24:42.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.953+0000 7f09d9adc700 20 client.37929.objecter put_session s=0x7f09d41b1670 osd=2 3 2024-04-14T15:24:42.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.953+0000 7f09d9adc700 20 client.37929.objecter _get_session s=0x7f09d41b1670 osd=2 3 2024-04-14T15:24:42.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.953+0000 7f09d9adc700 15 client.37929.objecter _session_command_op_remove -1 5 2024-04-14T15:24:42.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.953+0000 7f09d9adc700 20 client.37929.objecter get_session s=0x7f09d41b1670 osd=2 3 2024-04-14T15:24:42.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.953+0000 7f09d9adc700 15 client.37929.objecter _session_command_op_assign 2 5 2024-04-14T15:24:42.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.953+0000 7f09d9adc700 20 client.37929.objecter put_session s=0x7f09d41b1670 osd=2 4 2024-04-14T15:24:42.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.953+0000 7f09d9adc700 10 client.37929.objecter _send_command 5 2024-04-14T15:24:42.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.953+0000 7f09d9adc700 1 -- 172.21.15.90:0/1340141548 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f09d41b5800 con 0x7f09d41b1cf0 2024-04-14T15:24:42.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.953+0000 7f09d17fa700 10 client.37929.objecter ms_handle_connect 0x7f09d41b1cf0 2024-04-14T15:24:42.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.954+0000 7f09d17fa700 1 -- 172.21.15.90:0/1340141548 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f09d41b5800 con 0x7f09d41b1cf0 2024-04-14T15:24:42.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.954+0000 7f09d17fa700 10 client.37929.objecter ms_dispatch 0x7f09d4001550 command_reply(tid 5: 0 ) 2024-04-14T15:24:42.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.954+0000 7f09d17fa700 10 client.37929.objecter _finish_command 5 = system:0 2024-04-14T15:24:42.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.954+0000 7f09d17fa700 20 client.37929.objecter put_session s=0x7f09d41b1670 osd=2 4 2024-04-14T15:24:42.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.954+0000 7f09d17fa700 15 client.37929.objecter _session_command_op_remove 2 5 2024-04-14T15:24:42.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.970+0000 7f09d9adc700 10 client.37929.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:24:42.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.970+0000 7f09d9adc700 15 client.37929.objecter _session_command_op_assign -1 6 2024-04-14T15:24:42.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.970+0000 7f09d9adc700 20 client.37929.objecter _get_session s=0x7f09d41b1670 osd=2 3 2024-04-14T15:24:42.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.970+0000 7f09d9adc700 20 client.37929.objecter put_session s=0x7f09d41b1670 osd=2 3 2024-04-14T15:24:42.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.970+0000 7f09d9adc700 20 client.37929.objecter _get_session s=0x7f09d41b1670 osd=2 3 2024-04-14T15:24:42.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.970+0000 7f09d9adc700 15 client.37929.objecter _session_command_op_remove -1 6 2024-04-14T15:24:42.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.970+0000 7f09d9adc700 20 client.37929.objecter get_session s=0x7f09d41b1670 osd=2 3 2024-04-14T15:24:42.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.970+0000 7f09d9adc700 15 client.37929.objecter _session_command_op_assign 2 6 2024-04-14T15:24:42.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.970+0000 7f09d9adc700 20 client.37929.objecter put_session s=0x7f09d41b1670 osd=2 4 2024-04-14T15:24:42.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.970+0000 7f09d9adc700 10 client.37929.objecter _send_command 6 2024-04-14T15:24:42.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.970+0000 7f09d9adc700 1 -- 172.21.15.90:0/1340141548 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f09d41118e0 con 0x7f09d41b1cf0 2024-04-14T15:24:42.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.972+0000 7f09d17fa700 1 -- 172.21.15.90:0/1340141548 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f09d41118e0 con 0x7f09d41b1cf0 2024-04-14T15:24:42.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.972+0000 7f09d17fa700 10 client.37929.objecter ms_dispatch 0x7f09d4001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:24:42.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.972+0000 7f09d17fa700 10 client.37929.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:24:42.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.972+0000 7f09d17fa700 20 client.37929.objecter put_session s=0x7f09d41b1670 osd=2 4 2024-04-14T15:24:42.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.972+0000 7f09d17fa700 15 client.37929.objecter _session_command_op_remove 2 6 2024-04-14T15:24:42.972 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:24:42.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.972+0000 7f09d9adc700 10 client.37929.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:24:42.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.972+0000 7f09d9adc700 15 client.37929.objecter _session_command_op_assign -1 7 2024-04-14T15:24:42.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.972+0000 7f09d9adc700 20 client.37929.objecter _get_session s=0x7f09d4111ce0 osd=3 3 2024-04-14T15:24:42.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.972+0000 7f09d9adc700 20 client.37929.objecter put_session s=0x7f09d4111ce0 osd=3 3 2024-04-14T15:24:42.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.972+0000 7f09d9adc700 20 client.37929.objecter _get_session s=0x7f09d4111ce0 osd=3 3 2024-04-14T15:24:42.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.972+0000 7f09d9adc700 15 client.37929.objecter _session_command_op_remove -1 7 2024-04-14T15:24:42.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.972+0000 7f09d9adc700 20 client.37929.objecter get_session s=0x7f09d4111ce0 osd=3 3 2024-04-14T15:24:42.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.972+0000 7f09d9adc700 15 client.37929.objecter _session_command_op_assign 3 7 2024-04-14T15:24:42.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.972+0000 7f09d9adc700 20 client.37929.objecter put_session s=0x7f09d4111ce0 osd=3 4 2024-04-14T15:24:42.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.972+0000 7f09d9adc700 10 client.37929.objecter _send_command 7 2024-04-14T15:24:42.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.972+0000 7f09d9adc700 1 -- 172.21.15.90:0/1340141548 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f09d41ba3d0 con 0x7f09d4063bd0 2024-04-14T15:24:42.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.972+0000 7f09d17fa700 10 client.37929.objecter ms_handle_connect 0x7f09d4063bd0 2024-04-14T15:24:42.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.973+0000 7f09d17fa700 1 -- 172.21.15.90:0/1340141548 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f09d41ba3d0 con 0x7f09d4063bd0 2024-04-14T15:24:42.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.973+0000 7f09d17fa700 10 client.37929.objecter ms_dispatch 0x7f09d4001550 command_reply(tid 7: 0 ) 2024-04-14T15:24:42.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.973+0000 7f09d17fa700 10 client.37929.objecter _finish_command 7 = system:0 2024-04-14T15:24:42.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.973+0000 7f09d17fa700 20 client.37929.objecter put_session s=0x7f09d4111ce0 osd=3 4 2024-04-14T15:24:42.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.973+0000 7f09d17fa700 15 client.37929.objecter _session_command_op_remove 3 7 2024-04-14T15:24:42.976 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:24:42.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.990+0000 7f09d9adc700 10 client.37929.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:24:42.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.990+0000 7f09d9adc700 15 client.37929.objecter _session_command_op_assign -1 8 2024-04-14T15:24:42.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.990+0000 7f09d9adc700 20 client.37929.objecter _get_session s=0x7f09d4111ce0 osd=3 3 2024-04-14T15:24:42.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.990+0000 7f09d9adc700 20 client.37929.objecter put_session s=0x7f09d4111ce0 osd=3 3 2024-04-14T15:24:42.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.990+0000 7f09d9adc700 20 client.37929.objecter _get_session s=0x7f09d4111ce0 osd=3 3 2024-04-14T15:24:42.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.990+0000 7f09d9adc700 15 client.37929.objecter _session_command_op_remove -1 8 2024-04-14T15:24:42.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.990+0000 7f09d9adc700 20 client.37929.objecter get_session s=0x7f09d4111ce0 osd=3 3 2024-04-14T15:24:42.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.990+0000 7f09d9adc700 15 client.37929.objecter _session_command_op_assign 3 8 2024-04-14T15:24:42.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.990+0000 7f09d9adc700 20 client.37929.objecter put_session s=0x7f09d4111ce0 osd=3 4 2024-04-14T15:24:42.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.990+0000 7f09d9adc700 10 client.37929.objecter _send_command 8 2024-04-14T15:24:42.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.990+0000 7f09d9adc700 1 -- 172.21.15.90:0/1340141548 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f09d4063140 con 0x7f09d4063bd0 2024-04-14T15:24:42.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.992+0000 7f09d17fa700 1 -- 172.21.15.90:0/1340141548 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f09d4063140 con 0x7f09d4063bd0 2024-04-14T15:24:42.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.992+0000 7f09d17fa700 10 client.37929.objecter ms_dispatch 0x7f09d4001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:24:42.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.992+0000 7f09d17fa700 10 client.37929.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:24:42.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.992+0000 7f09d17fa700 20 client.37929.objecter put_session s=0x7f09d4111ce0 osd=3 4 2024-04-14T15:24:42.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.992+0000 7f09d17fa700 15 client.37929.objecter _session_command_op_remove 3 8 2024-04-14T15:24:42.992 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:24:42.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.992+0000 7f09d9adc700 10 client.37929.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:24:42.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.992+0000 7f09d9adc700 15 client.37929.objecter _session_command_op_assign -1 9 2024-04-14T15:24:42.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.992+0000 7f09d9adc700 20 client.37929.objecter _get_session s=0x7f09d4062ef0 osd=4 3 2024-04-14T15:24:42.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.992+0000 7f09d9adc700 20 client.37929.objecter put_session s=0x7f09d4062ef0 osd=4 3 2024-04-14T15:24:42.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.992+0000 7f09d9adc700 20 client.37929.objecter _get_session s=0x7f09d4062ef0 osd=4 3 2024-04-14T15:24:42.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.992+0000 7f09d9adc700 15 client.37929.objecter _session_command_op_remove -1 9 2024-04-14T15:24:42.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.992+0000 7f09d9adc700 20 client.37929.objecter get_session s=0x7f09d4062ef0 osd=4 3 2024-04-14T15:24:42.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.992+0000 7f09d9adc700 15 client.37929.objecter _session_command_op_assign 4 9 2024-04-14T15:24:42.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.992+0000 7f09d9adc700 20 client.37929.objecter put_session s=0x7f09d4062ef0 osd=4 4 2024-04-14T15:24:42.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.992+0000 7f09d9adc700 10 client.37929.objecter _send_command 9 2024-04-14T15:24:42.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.992+0000 7f09d9adc700 1 -- 172.21.15.90:0/1340141548 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f09d4063060 con 0x7f09d41b6690 2024-04-14T15:24:42.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.992+0000 7f09d17fa700 10 client.37929.objecter ms_handle_connect 0x7f09d41b6690 2024-04-14T15:24:42.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.993+0000 7f09d17fa700 1 -- 172.21.15.90:0/1340141548 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f09d4063060 con 0x7f09d41b6690 2024-04-14T15:24:42.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.993+0000 7f09d17fa700 10 client.37929.objecter ms_dispatch 0x7f09d4001550 command_reply(tid 9: 0 ) 2024-04-14T15:24:42.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.993+0000 7f09d17fa700 10 client.37929.objecter _finish_command 9 = system:0 2024-04-14T15:24:42.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.993+0000 7f09d17fa700 20 client.37929.objecter put_session s=0x7f09d4062ef0 osd=4 4 2024-04-14T15:24:42.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:42.993+0000 7f09d17fa700 15 client.37929.objecter _session_command_op_remove 4 9 2024-04-14T15:24:42.999 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:43.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:43.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:24:43.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.010+0000 7f09d9adc700 10 client.37929.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:24:43.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.010+0000 7f09d9adc700 15 client.37929.objecter _session_command_op_assign -1 10 2024-04-14T15:24:43.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.010+0000 7f09d9adc700 20 client.37929.objecter _get_session s=0x7f09d4062ef0 osd=4 3 2024-04-14T15:24:43.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.010+0000 7f09d9adc700 20 client.37929.objecter put_session s=0x7f09d4062ef0 osd=4 3 2024-04-14T15:24:43.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.010+0000 7f09d9adc700 20 client.37929.objecter _get_session s=0x7f09d4062ef0 osd=4 3 2024-04-14T15:24:43.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.010+0000 7f09d9adc700 15 client.37929.objecter _session_command_op_remove -1 10 2024-04-14T15:24:43.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.010+0000 7f09d9adc700 20 client.37929.objecter get_session s=0x7f09d4062ef0 osd=4 3 2024-04-14T15:24:43.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.010+0000 7f09d9adc700 15 client.37929.objecter _session_command_op_assign 4 10 2024-04-14T15:24:43.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.010+0000 7f09d9adc700 20 client.37929.objecter put_session s=0x7f09d4062ef0 osd=4 4 2024-04-14T15:24:43.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.010+0000 7f09d9adc700 10 client.37929.objecter _send_command 10 2024-04-14T15:24:43.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.010+0000 7f09d9adc700 1 -- 172.21.15.90:0/1340141548 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f09d41bef60 con 0x7f09d41b6690 2024-04-14T15:24:43.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.012+0000 7f09d17fa700 1 -- 172.21.15.90:0/1340141548 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f09d41bef60 con 0x7f09d41b6690 2024-04-14T15:24:43.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.012+0000 7f09d17fa700 10 client.37929.objecter ms_dispatch 0x7f09d4001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:24:43.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.012+0000 7f09d17fa700 10 client.37929.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:24:43.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.012+0000 7f09d17fa700 20 client.37929.objecter put_session s=0x7f09d4062ef0 osd=4 4 2024-04-14T15:24:43.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.012+0000 7f09d17fa700 15 client.37929.objecter _session_command_op_remove 4 10 2024-04-14T15:24:43.013 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:24:43.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.013+0000 7f09d9adc700 10 client.37929.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:24:43.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.013+0000 7f09d9adc700 15 client.37929.objecter _session_command_op_assign -1 11 2024-04-14T15:24:43.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.013+0000 7f09d9adc700 20 client.37929.objecter _get_session s=0x7f09d4000990 osd=5 3 2024-04-14T15:24:43.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.013+0000 7f09d9adc700 20 client.37929.objecter put_session s=0x7f09d4000990 osd=5 3 2024-04-14T15:24:43.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.013+0000 7f09d9adc700 20 client.37929.objecter _get_session s=0x7f09d4000990 osd=5 3 2024-04-14T15:24:43.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.013+0000 7f09d9adc700 15 client.37929.objecter _session_command_op_remove -1 11 2024-04-14T15:24:43.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.013+0000 7f09d9adc700 20 client.37929.objecter get_session s=0x7f09d4000990 osd=5 3 2024-04-14T15:24:43.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.013+0000 7f09d9adc700 15 client.37929.objecter _session_command_op_assign 5 11 2024-04-14T15:24:43.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.013+0000 7f09d9adc700 20 client.37929.objecter put_session s=0x7f09d4000990 osd=5 4 2024-04-14T15:24:43.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.013+0000 7f09d9adc700 10 client.37929.objecter _send_command 11 2024-04-14T15:24:43.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.013+0000 7f09d9adc700 1 -- 172.21.15.90:0/1340141548 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f09d41c2ff0 con 0x7f09d41bf470 2024-04-14T15:24:43.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.013+0000 7f09d17fa700 10 client.37929.objecter ms_handle_connect 0x7f09d41bf470 2024-04-14T15:24:43.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.014+0000 7f09d17fa700 1 -- 172.21.15.90:0/1340141548 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f09d41c2ff0 con 0x7f09d41bf470 2024-04-14T15:24:43.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.014+0000 7f09d17fa700 10 client.37929.objecter ms_dispatch 0x7f09d4001550 command_reply(tid 11: 0 ) 2024-04-14T15:24:43.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.014+0000 7f09d17fa700 10 client.37929.objecter _finish_command 11 = system:0 2024-04-14T15:24:43.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.014+0000 7f09d17fa700 20 client.37929.objecter put_session s=0x7f09d4000990 osd=5 4 2024-04-14T15:24:43.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.014+0000 7f09d17fa700 15 client.37929.objecter _session_command_op_remove 5 11 2024-04-14T15:24:43.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.031+0000 7f09d9adc700 10 client.37929.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:24:43.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.031+0000 7f09d9adc700 15 client.37929.objecter _session_command_op_assign -1 12 2024-04-14T15:24:43.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.031+0000 7f09d9adc700 20 client.37929.objecter _get_session s=0x7f09d4000990 osd=5 3 2024-04-14T15:24:43.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.031+0000 7f09d9adc700 20 client.37929.objecter put_session s=0x7f09d4000990 osd=5 3 2024-04-14T15:24:43.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.031+0000 7f09d9adc700 20 client.37929.objecter _get_session s=0x7f09d4000990 osd=5 3 2024-04-14T15:24:43.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.031+0000 7f09d9adc700 15 client.37929.objecter _session_command_op_remove -1 12 2024-04-14T15:24:43.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.031+0000 7f09d9adc700 20 client.37929.objecter get_session s=0x7f09d4000990 osd=5 3 2024-04-14T15:24:43.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.031+0000 7f09d9adc700 15 client.37929.objecter _session_command_op_assign 5 12 2024-04-14T15:24:43.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.031+0000 7f09d9adc700 20 client.37929.objecter put_session s=0x7f09d4000990 osd=5 4 2024-04-14T15:24:43.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.031+0000 7f09d9adc700 10 client.37929.objecter _send_command 12 2024-04-14T15:24:43.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.031+0000 7f09d9adc700 1 -- 172.21.15.90:0/1340141548 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f09d41b5c00 con 0x7f09d41bf470 2024-04-14T15:24:43.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.033+0000 7f09d17fa700 1 -- 172.21.15.90:0/1340141548 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f09d41b5c00 con 0x7f09d41bf470 2024-04-14T15:24:43.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.033+0000 7f09d17fa700 10 client.37929.objecter ms_dispatch 0x7f09d4001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:24:43.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.033+0000 7f09d17fa700 10 client.37929.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:24:43.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.033+0000 7f09d17fa700 20 client.37929.objecter put_session s=0x7f09d4000990 osd=5 4 2024-04-14T15:24:43.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.033+0000 7f09d17fa700 15 client.37929.objecter _session_command_op_remove 5 12 2024-04-14T15:24:43.034 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:24:43.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.033+0000 7f09d9adc700 10 client.37929.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:24:43.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.033+0000 7f09d9adc700 15 client.37929.objecter _session_command_op_assign -1 13 2024-04-14T15:24:43.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.033+0000 7f09d9adc700 20 client.37929.objecter _get_session s=0x7f09d4063410 osd=6 3 2024-04-14T15:24:43.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.033+0000 7f09d9adc700 20 client.37929.objecter put_session s=0x7f09d4063410 osd=6 3 2024-04-14T15:24:43.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.033+0000 7f09d9adc700 20 client.37929.objecter _get_session s=0x7f09d4063410 osd=6 3 2024-04-14T15:24:43.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.033+0000 7f09d9adc700 15 client.37929.objecter _session_command_op_remove -1 13 2024-04-14T15:24:43.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.033+0000 7f09d9adc700 20 client.37929.objecter get_session s=0x7f09d4063410 osd=6 3 2024-04-14T15:24:43.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.033+0000 7f09d9adc700 15 client.37929.objecter _session_command_op_assign 6 13 2024-04-14T15:24:43.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.033+0000 7f09d9adc700 20 client.37929.objecter put_session s=0x7f09d4063410 osd=6 4 2024-04-14T15:24:43.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.033+0000 7f09d9adc700 10 client.37929.objecter _send_command 13 2024-04-14T15:24:43.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.033+0000 7f09d9adc700 1 -- 172.21.15.90:0/1340141548 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f09d41c47b0 con 0x7f09d41b59b0 2024-04-14T15:24:43.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.033+0000 7f09d17fa700 10 client.37929.objecter ms_handle_connect 0x7f09d41b59b0 2024-04-14T15:24:43.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.034+0000 7f09d17fa700 1 -- 172.21.15.90:0/1340141548 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f09cc006500 con 0x7f09d41b59b0 2024-04-14T15:24:43.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.034+0000 7f09d17fa700 10 client.37929.objecter ms_dispatch 0x7f09d4001550 command_reply(tid 13: 0 ) 2024-04-14T15:24:43.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.034+0000 7f09d17fa700 10 client.37929.objecter _finish_command 13 = system:0 2024-04-14T15:24:43.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.034+0000 7f09d17fa700 20 client.37929.objecter put_session s=0x7f09d4063410 osd=6 4 2024-04-14T15:24:43.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.034+0000 7f09d17fa700 15 client.37929.objecter _session_command_op_remove 6 13 2024-04-14T15:24:43.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.052+0000 7f09d9adc700 10 client.37929.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:24:43.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.052+0000 7f09d9adc700 15 client.37929.objecter _session_command_op_assign -1 14 2024-04-14T15:24:43.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.052+0000 7f09d9adc700 20 client.37929.objecter _get_session s=0x7f09d4063410 osd=6 3 2024-04-14T15:24:43.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.052+0000 7f09d9adc700 20 client.37929.objecter put_session s=0x7f09d4063410 osd=6 3 2024-04-14T15:24:43.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.052+0000 7f09d9adc700 20 client.37929.objecter _get_session s=0x7f09d4063410 osd=6 3 2024-04-14T15:24:43.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.052+0000 7f09d9adc700 15 client.37929.objecter _session_command_op_remove -1 14 2024-04-14T15:24:43.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.052+0000 7f09d9adc700 20 client.37929.objecter get_session s=0x7f09d4063410 osd=6 3 2024-04-14T15:24:43.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.052+0000 7f09d9adc700 15 client.37929.objecter _session_command_op_assign 6 14 2024-04-14T15:24:43.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.052+0000 7f09d9adc700 20 client.37929.objecter put_session s=0x7f09d4063410 osd=6 4 2024-04-14T15:24:43.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.052+0000 7f09d9adc700 10 client.37929.objecter _send_command 14 2024-04-14T15:24:43.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.052+0000 7f09d9adc700 1 -- 172.21.15.90:0/1340141548 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f09d41bcad0 con 0x7f09d41b59b0 2024-04-14T15:24:43.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.054+0000 7f09d17fa700 1 -- 172.21.15.90:0/1340141548 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f09d41bcad0 con 0x7f09d41b59b0 2024-04-14T15:24:43.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.054+0000 7f09d17fa700 10 client.37929.objecter ms_dispatch 0x7f09d4001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:24:43.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.054+0000 7f09d17fa700 10 client.37929.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:24:43.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.054+0000 7f09d17fa700 20 client.37929.objecter put_session s=0x7f09d4063410 osd=6 4 2024-04-14T15:24:43.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.054+0000 7f09d17fa700 15 client.37929.objecter _session_command_op_remove 6 14 2024-04-14T15:24:43.055 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:24:43.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.054+0000 7f09d9adc700 10 client.37929.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:24:43.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.054+0000 7f09d9adc700 15 client.37929.objecter _session_command_op_assign -1 15 2024-04-14T15:24:43.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.054+0000 7f09d9adc700 20 client.37929.objecter _get_session s=0x7f09d41bcad0 osd=7 3 2024-04-14T15:24:43.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.054+0000 7f09d9adc700 20 client.37929.objecter put_session s=0x7f09d41bcad0 osd=7 3 2024-04-14T15:24:43.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.054+0000 7f09d9adc700 20 client.37929.objecter _get_session s=0x7f09d41bcad0 osd=7 3 2024-04-14T15:24:43.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.054+0000 7f09d9adc700 15 client.37929.objecter _session_command_op_remove -1 15 2024-04-14T15:24:43.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.054+0000 7f09d9adc700 20 client.37929.objecter get_session s=0x7f09d41bcad0 osd=7 3 2024-04-14T15:24:43.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.054+0000 7f09d9adc700 15 client.37929.objecter _session_command_op_assign 7 15 2024-04-14T15:24:43.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.054+0000 7f09d9adc700 20 client.37929.objecter put_session s=0x7f09d41bcad0 osd=7 4 2024-04-14T15:24:43.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.054+0000 7f09d9adc700 10 client.37929.objecter _send_command 15 2024-04-14T15:24:43.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.054+0000 7f09d9adc700 1 -- 172.21.15.90:0/1340141548 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f09d41cbbc0 con 0x7f09d41c80b0 2024-04-14T15:24:43.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.055+0000 7f09d17fa700 10 client.37929.objecter ms_handle_connect 0x7f09d41c80b0 2024-04-14T15:24:43.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.056+0000 7f09d17fa700 1 -- 172.21.15.90:0/1340141548 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f09d41cbbc0 con 0x7f09d41c80b0 2024-04-14T15:24:43.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.056+0000 7f09d17fa700 10 client.37929.objecter ms_dispatch 0x7f09d4001550 command_reply(tid 15: 0 ) 2024-04-14T15:24:43.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.056+0000 7f09d17fa700 10 client.37929.objecter _finish_command 15 = system:0 2024-04-14T15:24:43.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.056+0000 7f09d17fa700 20 client.37929.objecter put_session s=0x7f09d41bcad0 osd=7 4 2024-04-14T15:24:43.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.056+0000 7f09d17fa700 15 client.37929.objecter _session_command_op_remove 7 15 2024-04-14T15:24:43.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.074+0000 7f09d9adc700 10 client.37929.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:24:43.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.074+0000 7f09d9adc700 15 client.37929.objecter _session_command_op_assign -1 16 2024-04-14T15:24:43.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.074+0000 7f09d9adc700 20 client.37929.objecter _get_session s=0x7f09d41bcad0 osd=7 3 2024-04-14T15:24:43.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.074+0000 7f09d9adc700 20 client.37929.objecter put_session s=0x7f09d41bcad0 osd=7 3 2024-04-14T15:24:43.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.074+0000 7f09d9adc700 20 client.37929.objecter _get_session s=0x7f09d41bcad0 osd=7 3 2024-04-14T15:24:43.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.074+0000 7f09d9adc700 15 client.37929.objecter _session_command_op_remove -1 16 2024-04-14T15:24:43.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.074+0000 7f09d9adc700 20 client.37929.objecter get_session s=0x7f09d41bcad0 osd=7 3 2024-04-14T15:24:43.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.074+0000 7f09d9adc700 15 client.37929.objecter _session_command_op_assign 7 16 2024-04-14T15:24:43.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.074+0000 7f09d9adc700 20 client.37929.objecter put_session s=0x7f09d41bcad0 osd=7 4 2024-04-14T15:24:43.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.074+0000 7f09d9adc700 10 client.37929.objecter _send_command 16 2024-04-14T15:24:43.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.074+0000 7f09d9adc700 1 -- 172.21.15.90:0/1340141548 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f09d41b5e10 con 0x7f09d41c80b0 2024-04-14T15:24:43.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.076+0000 7f09d17fa700 1 -- 172.21.15.90:0/1340141548 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f09d41b5e10 con 0x7f09d41c80b0 2024-04-14T15:24:43.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.076+0000 7f09d17fa700 10 client.37929.objecter ms_dispatch 0x7f09d4001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:24:43.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.076+0000 7f09d17fa700 10 client.37929.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:24:43.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.076+0000 7f09d17fa700 20 client.37929.objecter put_session s=0x7f09d41bcad0 osd=7 4 2024-04-14T15:24:43.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.076+0000 7f09d17fa700 15 client.37929.objecter _session_command_op_remove 7 16 2024-04-14T15:24:43.077 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:24:43.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.076+0000 7f09d9adc700 10 librados: watch_flush enter 2024-04-14T15:24:43.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.076+0000 7f09d9adc700 10 librados: watch_flush exit 2024-04-14T15:24:43.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.076+0000 7f09d9adc700 10 client.37929.objecter close_session for osd.0 2024-04-14T15:24:43.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.076+0000 7f09d9adc700 1 -- 172.21.15.90:0/1340141548 >> v1:172.21.15.90:6801/3761806885 conn(0x7f09d41ab110 legacy=0x7f09d41ab4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:43.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.076+0000 7f09d9adc700 20 client.37929.objecter put_session s=0x7f09d41122e0 osd=0 1 2024-04-14T15:24:43.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.076+0000 7f09d9adc700 10 client.37929.objecter close_session for osd.1 2024-04-14T15:24:43.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.076+0000 7f09d9adc700 1 -- 172.21.15.90:0/1340141548 >> v1:172.21.15.90:6805/4170361699 conn(0x7f09d410ecd0 legacy=0x7f09d410f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:43.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.076+0000 7f09d9adc700 20 client.37929.objecter put_session s=0x7f09d4103070 osd=1 1 2024-04-14T15:24:43.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.076+0000 7f09d9adc700 10 client.37929.objecter close_session for osd.2 2024-04-14T15:24:43.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.076+0000 7f09d9adc700 1 -- 172.21.15.90:0/1340141548 >> v1:172.21.15.90:6809/1686184251 conn(0x7f09d41b1cf0 legacy=0x7f09d41b40e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:43.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.077+0000 7f09d9adc700 20 client.37929.objecter put_session s=0x7f09d41b1670 osd=2 1 2024-04-14T15:24:43.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.077+0000 7f09d9adc700 10 client.37929.objecter close_session for osd.3 2024-04-14T15:24:43.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.077+0000 7f09d9adc700 1 -- 172.21.15.90:0/1340141548 >> v1:172.21.15.90:6813/2270166774 conn(0x7f09d4063bd0 legacy=0x7f09d41b8b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:43.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.077+0000 7f09d9adc700 20 client.37929.objecter put_session s=0x7f09d4111ce0 osd=3 1 2024-04-14T15:24:43.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.077+0000 7f09d9adc700 10 client.37929.objecter close_session for osd.4 2024-04-14T15:24:43.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.077+0000 7f09d9adc700 1 -- 172.21.15.90:0/1340141548 >> v1:172.21.15.92:6800/3887549733 conn(0x7f09d41b6690 legacy=0x7f09d41ba5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:43.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.077+0000 7f09d9adc700 20 client.37929.objecter put_session s=0x7f09d4062ef0 osd=4 1 2024-04-14T15:24:43.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.077+0000 7f09d9adc700 10 client.37929.objecter close_session for osd.5 2024-04-14T15:24:43.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.077+0000 7f09d9adc700 1 -- 172.21.15.90:0/1340141548 >> v1:172.21.15.92:6804/3244561440 conn(0x7f09d41bf470 legacy=0x7f09d41c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:43.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.077+0000 7f09d9adc700 20 client.37929.objecter put_session s=0x7f09d4000990 osd=5 1 2024-04-14T15:24:43.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.077+0000 7f09d9adc700 10 client.37929.objecter close_session for osd.6 2024-04-14T15:24:43.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.077+0000 7f09d9adc700 1 -- 172.21.15.90:0/1340141548 >> v1:172.21.15.92:6808/636125963 conn(0x7f09d41b59b0 legacy=0x7f09d41c2ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:43.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.077+0000 7f09d3fff700 1 -- 172.21.15.90:0/1340141548 reap_dead start 2024-04-14T15:24:43.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.077+0000 7f09d3fff700 1 -- 172.21.15.90:0/1340141548 reap_dead start 2024-04-14T15:24:43.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.077+0000 7f09d3fff700 1 -- 172.21.15.90:0/1340141548 reap_dead start 2024-04-14T15:24:43.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.077+0000 7f09d9adc700 20 client.37929.objecter put_session s=0x7f09d4063410 osd=6 1 2024-04-14T15:24:43.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.077+0000 7f09d9adc700 10 client.37929.objecter close_session for osd.7 2024-04-14T15:24:43.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.077+0000 7f09d9adc700 1 -- 172.21.15.90:0/1340141548 >> v1:172.21.15.92:6812/1773706716 conn(0x7f09d41c80b0 legacy=0x7f09d41ca4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:43.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.077+0000 7f09d9adc700 20 client.37929.objecter put_session s=0x7f09d41bcad0 osd=7 1 2024-04-14T15:24:43.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.077+0000 7f09d9adc700 20 client.37929.objecter shutdown clearing up homeless session... 2024-04-14T15:24:43.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.077+0000 7f09d9adc700 10 client.37929.objecter successfully canceled tick 2024-04-14T15:24:43.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.077+0000 7f09d9adc700 1 -- 172.21.15.90:0/1340141548 >> v1:172.21.15.90:6800/420935427 conn(0x7f09c0071460 legacy=0x7f09c0073920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:43.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.077+0000 7f09d9adc700 1 -- 172.21.15.90:0/1340141548 >> v1:172.21.15.90:6789/0 conn(0x7f09d4107890 legacy=0x7f09d410db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:43.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.077+0000 7f09d9adc700 1 -- 172.21.15.90:0/1340141548 shutdown_connections 2024-04-14T15:24:43.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.077+0000 7f09d9adc700 1 -- 172.21.15.90:0/1340141548 >> 172.21.15.90:0/1340141548 conn(0x7f09d40fd290 msgr2=0x7f09d4110f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:24:43.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.077+0000 7f09d9adc700 1 -- 172.21.15.90:0/1340141548 shutdown_connections 2024-04-14T15:24:43.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.077+0000 7f09d9adc700 1 -- 172.21.15.90:0/1340141548 wait complete. 2024-04-14T15:24:43.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:43.077+0000 7f09d9adc700 1 librados: shutdown 2024-04-14T15:24:43.081 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:24:43.081 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:24:43.081 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:24:43.081 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:24:43.081 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:24:43.081 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:24:43.081 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:24:43.082 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:24:43.101 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:24:43.124 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:43.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:43.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:24:43.226 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:24:43.254 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:43.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:43.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:24:43.356 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:24:43.382 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:43.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:43.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:24:43.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:43 smithi090 ceph-mon[41670]: osdmap e738: 8 total, 8 up, 7 in 2024-04-14T15:24:43.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:43 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:24:43.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:43 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:24:43.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:43 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1340141548' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:24:43.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:43 smithi090 ceph-mon[30399]: osdmap e738: 8 total, 8 up, 7 in 2024-04-14T15:24:43.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:43 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:24:43.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:43 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:24:43.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:43 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1340141548' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:24:43.483 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:24:43.509 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:43.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:43.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:24:43.610 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:24:43.621 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:43 smithi092 ceph-mon[33966]: osdmap e738: 8 total, 8 up, 7 in 2024-04-14T15:24:43.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:43 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:24:43.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:43 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:24:43.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:43 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1340141548' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:24:43.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:43.637 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:43.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:24:43.738 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:24:43.762 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:43.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:43.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:24:43.864 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:24:43.888 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:43.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:43.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:24:43.989 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:24:44.015 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:44.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:44.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:24:44.117 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:24:44.145 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:44.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:44.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:24:44.204 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:24:43 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:24:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:24:44.247 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:24:44.273 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:44.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:44.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:24:44.374 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:24:44.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:44.401 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:44.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:24:44.502 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:24:44.528 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:44.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:44.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:24:44.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:44 smithi092 ceph-mon[33966]: pgmap v1092: 214 pgs: 32 unknown, 182 active+clean; 582 KiB data, 707 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:24:44.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:44 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:24:44.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:44 smithi092 ceph-mon[33966]: osdmap e739: 8 total, 8 up, 7 in 2024-04-14T15:24:44.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:44 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:24:44.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:44 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:24:44.630 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:24:44.655 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:44.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:44.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:24:44.685 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:44 smithi090 ceph-mon[30399]: pgmap v1092: 214 pgs: 32 unknown, 182 active+clean; 582 KiB data, 707 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:24:44.685 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:44 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:24:44.686 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:44 smithi090 ceph-mon[30399]: osdmap e739: 8 total, 8 up, 7 in 2024-04-14T15:24:44.686 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:44 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:24:44.686 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:44 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:24:44.686 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:44 smithi090 ceph-mon[41670]: pgmap v1092: 214 pgs: 32 unknown, 182 active+clean; 582 KiB data, 707 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:24:44.686 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:44 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:24:44.686 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:44 smithi090 ceph-mon[41670]: osdmap e739: 8 total, 8 up, 7 in 2024-04-14T15:24:44.686 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:44 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:24:44.686 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:44 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:24:44.756 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:24:44.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:44.782 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:44.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:24:44.883 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:24:44.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:44.907 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:44.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:24:44.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:44.972+0000 7f9c47e6f700 1 -- 172.21.15.90:0/3737022030 >> v1:172.21.15.90:6789/0 conn(0x7f9c4010b730 legacy=0x7f9c4010db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:44.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:44.972+0000 7f9c47e6f700 1 -- 172.21.15.90:0/3737022030 shutdown_connections 2024-04-14T15:24:44.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:44.972+0000 7f9c47e6f700 1 -- 172.21.15.90:0/3737022030 >> 172.21.15.90:0/3737022030 conn(0x7f9c400fe3d0 msgr2=0x7f9c401007f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:24:44.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:44.973+0000 7f9c47e6f700 1 -- 172.21.15.90:0/3737022030 shutdown_connections 2024-04-14T15:24:44.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:44.973+0000 7f9c47e6f700 1 -- 172.21.15.90:0/3737022030 wait complete. 2024-04-14T15:24:44.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:44.973+0000 7f9c47e6f700 1 librados: starting msgr at 2024-04-14T15:24:44.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:44.973+0000 7f9c47e6f700 1 librados: starting objecter 2024-04-14T15:24:44.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:44.973+0000 7f9c47e6f700 1 Processor -- start 2024-04-14T15:24:44.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:44.973+0000 7f9c47e6f700 1 -- start start 2024-04-14T15:24:44.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:44.973+0000 7f9c47e6f700 1 librados: setting wanted keys 2024-04-14T15:24:44.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:44.973+0000 7f9c47e6f700 1 librados: calling monclient init 2024-04-14T15:24:44.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:44.973+0000 7f9c47e6f700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f9c4010f050 con 0x7f9c4010b730 2024-04-14T15:24:44.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:44.973+0000 7f9c47e6f700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f9c401a8f10 con 0x7f9c4010f300 2024-04-14T15:24:44.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:44.973+0000 7f9c47e6f700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f9c401aa0f0 con 0x7f9c40108880 2024-04-14T15:24:44.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:44.973+0000 7f9c45c0b700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f9c40108880 0x7f9c4010d810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:50410/0 (socket says 172.21.15.90:50410) 2024-04-14T15:24:44.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:44.973+0000 7f9c45c0b700 1 -- 172.21.15.90:0/3637852971 learned_addr learned my addr 172.21.15.90:0/3637852971 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:24:44.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:44.973+0000 7f9c4540a700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f9c4010b730 0x7f9c4010df20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:51790/0 (socket says 172.21.15.90:51790) 2024-04-14T15:24:44.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:44.973+0000 7f9c36ffd700 10 client.?.objecter ms_handle_connect 0x7f9c40108880 2024-04-14T15:24:44.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:44.974+0000 7f9c36ffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:24:44.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:44.974+0000 7f9c36ffd700 10 client.?.objecter ms_handle_connect 0x7f9c4010b730 2024-04-14T15:24:44.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:44.974+0000 7f9c36ffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:24:44.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:44.974+0000 7f9c36ffd700 10 client.?.objecter ms_handle_connect 0x7f9c4010f300 2024-04-14T15:24:44.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:44.974+0000 7f9c36ffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:24:44.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:44.974+0000 7f9c36ffd700 1 -- 172.21.15.90:0/3637852971 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 319852456 0 0) 0x7f9c4010f050 con 0x7f9c4010b730 2024-04-14T15:24:44.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:44.974+0000 7f9c36ffd700 1 -- 172.21.15.90:0/3637852971 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9c2c0035b0 con 0x7f9c4010b730 2024-04-14T15:24:44.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:44.974+0000 7f9c36ffd700 1 -- 172.21.15.90:0/3637852971 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1562542704 0 0) 0x7f9c401aa0f0 con 0x7f9c40108880 2024-04-14T15:24:44.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:44.974+0000 7f9c36ffd700 1 -- 172.21.15.90:0/3637852971 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9c4010f050 con 0x7f9c40108880 2024-04-14T15:24:44.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:44.974+0000 7f9c36ffd700 1 -- 172.21.15.90:0/3637852971 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1044366416 0 0) 0x7f9c401a8f10 con 0x7f9c4010f300 2024-04-14T15:24:44.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:44.974+0000 7f9c36ffd700 1 -- 172.21.15.90:0/3637852971 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9c401aa0f0 con 0x7f9c4010f300 2024-04-14T15:24:44.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:44.974+0000 7f9c36ffd700 1 -- 172.21.15.90:0/3637852971 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3447260801 0 0) 0x7f9c4010f050 con 0x7f9c40108880 2024-04-14T15:24:44.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:44.974+0000 7f9c36ffd700 1 -- 172.21.15.90:0/3637852971 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9c401a8f10 con 0x7f9c40108880 2024-04-14T15:24:44.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:44.974+0000 7f9c36ffd700 1 -- 172.21.15.90:0/3637852971 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f9c3c001870 con 0x7f9c40108880 2024-04-14T15:24:44.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:44.974+0000 7f9c36ffd700 1 -- 172.21.15.90:0/3637852971 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4187810265 0 0) 0x7f9c2c0035b0 con 0x7f9c4010b730 2024-04-14T15:24:44.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:44.974+0000 7f9c36ffd700 1 -- 172.21.15.90:0/3637852971 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9c4010f050 con 0x7f9c4010b730 2024-04-14T15:24:44.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:44.974+0000 7f9c36ffd700 1 -- 172.21.15.90:0/3637852971 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f9c30001430 con 0x7f9c4010b730 2024-04-14T15:24:44.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:44.974+0000 7f9c36ffd700 1 -- 172.21.15.90:0/3637852971 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3449719939 0 0) 0x7f9c401aa0f0 con 0x7f9c4010f300 2024-04-14T15:24:44.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:44.974+0000 7f9c36ffd700 1 -- 172.21.15.90:0/3637852971 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9c2c0035b0 con 0x7f9c4010f300 2024-04-14T15:24:44.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:44.974+0000 7f9c36ffd700 1 -- 172.21.15.90:0/3637852971 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f9c38001c50 con 0x7f9c4010f300 2024-04-14T15:24:44.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:44.974+0000 7f9c36ffd700 1 -- 172.21.15.90:0/3637852971 <== mon.2 v1:172.21.15.90:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3165348528 0 0) 0x7f9c401a8f10 con 0x7f9c40108880 2024-04-14T15:24:44.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:44.974+0000 7f9c36ffd700 1 -- 172.21.15.90:0/3637852971 >> v1:172.21.15.92:6789/0 conn(0x7f9c4010f300 legacy=0x7f9c401a67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:44.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:44.974+0000 7f9c36ffd700 1 -- 172.21.15.90:0/3637852971 >> v1:172.21.15.90:6789/0 conn(0x7f9c4010b730 legacy=0x7f9c4010df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:44.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:44.974+0000 7f9c36ffd700 1 -- 172.21.15.90:0/3637852971 --> v1:172.21.15.90:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c401ab310 con 0x7f9c40108880 2024-04-14T15:24:44.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:44.974+0000 7f9c47e6f700 1 -- 172.21.15.90:0/3637852971 --> v1:172.21.15.90:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9c401a7f60 con 0x7f9c40108880 2024-04-14T15:24:44.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:44.974+0000 7f9c47e6f700 10 client.47885.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:24:44.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:44.974+0000 7f9c47e6f700 1 -- 172.21.15.90:0/3637852971 --> v1:172.21.15.90:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f9c401a8420 con 0x7f9c40108880 2024-04-14T15:24:44.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:44.974+0000 7f9c47e6f700 1 librados: init done 2024-04-14T15:24:44.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:44.974+0000 7f9c36ffd700 1 -- 172.21.15.90:0/3637852971 <== mon.2 v1:172.21.15.90:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f9c3c003b40 con 0x7f9c40108880 2024-04-14T15:24:44.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:44.974+0000 7f9c36ffd700 1 -- 172.21.15.90:0/3637852971 <== mon.2 v1:172.21.15.90:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f9c3c005c50 con 0x7f9c40108880 2024-04-14T15:24:44.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:44.975+0000 7f9c47e6f700 1 -- 172.21.15.90:0/3637852971 --> v1:172.21.15.90:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c24005150 con 0x7f9c40108880 2024-04-14T15:24:44.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:44.976+0000 7f9c36ffd700 1 -- 172.21.15.90:0/3637852971 <== mon.2 v1:172.21.15.90:6790/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f9c3c0036a0 con 0x7f9c40108880 2024-04-14T15:24:44.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:44.976+0000 7f9c36ffd700 1 -- 172.21.15.90:0/3637852971 <== mon.2 v1:172.21.15.90:6790/0 8 ==== osd_map(740..740 src has 467..740) ==== 8757+0+0 (unknown 3029746855 0 0) 0x7f9c3c08aae0 con 0x7f9c40108880 2024-04-14T15:24:44.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:44.976+0000 7f9c36ffd700 10 client.47885.objecter ms_dispatch 0x7f9c40001550 osd_map(740..740 src has 467..740) 2024-04-14T15:24:44.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:44.976+0000 7f9c36ffd700 3 client.47885.objecter handle_osd_map got epochs [740,740] > 0 2024-04-14T15:24:44.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:44.976+0000 7f9c36ffd700 3 client.47885.objecter handle_osd_map decoding full epoch 740 2024-04-14T15:24:44.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:44.976+0000 7f9c36ffd700 20 client.47885.objecter dump_active .. 0 homeless 2024-04-14T15:24:44.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:44.976+0000 7f9c36ffd700 10 client.47885.objecter ms_handle_connect 0x7f9c2c0713b0 2024-04-14T15:24:44.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:44.981+0000 7f9c36ffd700 1 -- 172.21.15.90:0/3637852971 <== mon.2 v1:172.21.15.90:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f9c3c0572a0 con 0x7f9c40108880 2024-04-14T15:24:45.008 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:24:45.031 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:45.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:45.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:24:45.133 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:24:45.161 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:45.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:45.162 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:24:45.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:45.194+0000 7f9c47e6f700 1 -- 172.21.15.90:0/3637852971 --> v1:172.21.15.90:6800/420935427 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9c24000bc0 con 0x7f9c2c0713b0 2024-04-14T15:24:45.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:45.209+0000 7f9c36ffd700 1 -- 172.21.15.90:0/3637852971 <== mgr.14610 v1:172.21.15.90:6800/420935427 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+555731 (unknown 2965378022 0 3596288498) 0x7f9c24000bc0 con 0x7f9c2c0713b0 2024-04-14T15:24:45.216 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:24:45.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:45.213+0000 7f9c47e6f700 10 librados: watch_flush enter 2024-04-14T15:24:45.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:45.213+0000 7f9c47e6f700 10 librados: watch_flush exit 2024-04-14T15:24:45.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:45.213+0000 7f9c47e6f700 20 client.47885.objecter shutdown clearing up homeless session... 2024-04-14T15:24:45.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:45.213+0000 7f9c47e6f700 10 client.47885.objecter successfully canceled tick 2024-04-14T15:24:45.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:45.213+0000 7f9c47e6f700 1 -- 172.21.15.90:0/3637852971 >> v1:172.21.15.90:6800/420935427 conn(0x7f9c2c0713b0 legacy=0x7f9c2c073870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:45.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:45.213+0000 7f9c47e6f700 1 -- 172.21.15.90:0/3637852971 >> v1:172.21.15.90:6790/0 conn(0x7f9c40108880 legacy=0x7f9c4010d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:45.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:45.213+0000 7f9c47e6f700 1 -- 172.21.15.90:0/3637852971 shutdown_connections 2024-04-14T15:24:45.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:45.213+0000 7f9c47e6f700 1 -- 172.21.15.90:0/3637852971 >> 172.21.15.90:0/3637852971 conn(0x7f9c400fe3d0 msgr2=0x7f9c40112860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:24:45.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:45.213+0000 7f9c47e6f700 1 -- 172.21.15.90:0/3637852971 shutdown_connections 2024-04-14T15:24:45.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:45.213+0000 7f9c47e6f700 1 -- 172.21.15.90:0/3637852971 wait complete. 2024-04-14T15:24:45.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:45.213+0000 7f9c47e6f700 1 librados: shutdown 2024-04-14T15:24:45.222 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-14T15:24:45.263 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:24:45.291 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:45.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:45.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:24:45.393 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:24:45.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:45.417 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:45.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:24:45.518 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:24:45.546 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:45.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:45.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:24:45.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:45 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-130", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:24:45.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:45 smithi092 ceph-mon[33966]: osdmap e740: 8 total, 8 up, 7 in 2024-04-14T15:24:45.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:45 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-130"}]: dispatch 2024-04-14T15:24:45.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:45 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-130"}]: dispatch 2024-04-14T15:24:45.622 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:24:45 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:24:45.650 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:24:45.651 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:45 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-130", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:24:45.651 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:45 smithi090 ceph-mon[41670]: osdmap e740: 8 total, 8 up, 7 in 2024-04-14T15:24:45.651 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:45 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-130"}]: dispatch 2024-04-14T15:24:45.651 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:45 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-130"}]: dispatch 2024-04-14T15:24:45.651 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:45 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-130", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:24:45.651 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:45 smithi090 ceph-mon[30399]: osdmap e740: 8 total, 8 up, 7 in 2024-04-14T15:24:45.651 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:45 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-130"}]: dispatch 2024-04-14T15:24:45.651 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:45 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-130"}]: dispatch 2024-04-14T15:24:45.678 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:45.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:45.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:24:45.780 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:24:45.803 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:45.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:45.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:24:45.905 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:24:45.928 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:45.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:45.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:24:46.030 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:24:46.058 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:46.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:46.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:24:46.161 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:24:46.186 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:46.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:46.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:24:46.287 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:24:46.288 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:24:46.305 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-14T15:24:46.305 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-14T15:24:46.305 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-14T15:24:46.305 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-14T15:24:46.314 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:46.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:46.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:24:46.416 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:24:46.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:46 smithi090 ceph-mon[41670]: pgmap v1095: 214 pgs: 214 active+clean; 582 KiB data, 707 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:24:46.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:46 smithi090 ceph-mon[41670]: from='client.47885 v1:172.21.15.90:0/3637852971' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:24:46.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:46 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-130"}]': finished 2024-04-14T15:24:46.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:46 smithi090 ceph-mon[41670]: osdmap e741: 8 total, 8 up, 7 in 2024-04-14T15:24:46.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:46 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-130", "mode": "writeback"}]: dispatch 2024-04-14T15:24:46.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:46 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-130", "mode": "writeback"}]: dispatch 2024-04-14T15:24:46.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:46 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:24:46.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:46 smithi090 ceph-mon[41670]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:24:46.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:46 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-130", "mode": "writeback"}]': finished 2024-04-14T15:24:46.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:46 smithi090 ceph-mon[41670]: osdmap e742: 8 total, 8 up, 7 in 2024-04-14T15:24:46.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:46 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:24:46.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:46 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:24:46.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:46 smithi090 ceph-mon[30399]: pgmap v1095: 214 pgs: 214 active+clean; 582 KiB data, 707 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:24:46.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:46 smithi090 ceph-mon[30399]: from='client.47885 v1:172.21.15.90:0/3637852971' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:24:46.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:46 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-130"}]': finished 2024-04-14T15:24:46.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:46 smithi090 ceph-mon[30399]: osdmap e741: 8 total, 8 up, 7 in 2024-04-14T15:24:46.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:46 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-130", "mode": "writeback"}]: dispatch 2024-04-14T15:24:46.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:46 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-130", "mode": "writeback"}]: dispatch 2024-04-14T15:24:46.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:46 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:24:46.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:46 smithi090 ceph-mon[30399]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:24:46.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:46 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-130", "mode": "writeback"}]': finished 2024-04-14T15:24:46.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:46 smithi090 ceph-mon[30399]: osdmap e742: 8 total, 8 up, 7 in 2024-04-14T15:24:46.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:46 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:24:46.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:46 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:24:46.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:46.444 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:46.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:24:46.545 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:24:46.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:46.571 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:46.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:24:46.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:46 smithi092 ceph-mon[33966]: pgmap v1095: 214 pgs: 214 active+clean; 582 KiB data, 707 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:24:46.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:46 smithi092 ceph-mon[33966]: from='client.47885 v1:172.21.15.90:0/3637852971' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:24:46.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:46 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-130"}]': finished 2024-04-14T15:24:46.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:46 smithi092 ceph-mon[33966]: osdmap e741: 8 total, 8 up, 7 in 2024-04-14T15:24:46.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:46 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-130", "mode": "writeback"}]: dispatch 2024-04-14T15:24:46.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:46 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-130", "mode": "writeback"}]: dispatch 2024-04-14T15:24:46.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:46 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:24:46.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:46 smithi092 ceph-mon[33966]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:24:46.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:46 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-130", "mode": "writeback"}]': finished 2024-04-14T15:24:46.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:46 smithi092 ceph-mon[33966]: osdmap e742: 8 total, 8 up, 7 in 2024-04-14T15:24:46.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:46 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:24:46.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:46 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:24:46.673 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:24:46.698 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:46.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:46.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:24:46.712 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-14T15:24:46.739 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":1095,"stamp":"2024-04-14T15:24:45.179190+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596468,"num_objects":212,"num_object_clones":0,"num_object_copies":636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":212,"num_whiteouts":0,"num_read":2055,"num_read_kb":1796,"num_write":755,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":126,"num_bytes_recovered":921158,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":943,"ondisk_log_size":943,"up":642,"acting":642,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":611,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":724116,"kb_used_data":39396,"kb_used_omap":10,"kb_used_meta":684597,"kb_avail":655491948,"statfs":{"total":671965249536,"available":671223754752,"internally_reserved":0,"allocated":40341504,"data_stored":20327119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11156,"internal_metadata":701027436},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":-4,"num_write_kb":-2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":-2,"num_flush_kb":-1,"num_evict":0,"num_evict_kb":0,"num_promote":-1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000900"},"pg_stats":[{"pgid":"333.1f","version":"0'0","reported_seq":10,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.508627+0000","last_change":"2024-04-14T15:24:43.508082+0000","last_active":"2024-04-14T15:24:43.508627+0000","last_peered":"2024-04-14T15:24:43.508627+0000","last_clean":"2024-04-14T15:24:43.508627+0000","last_became_active":"2024-04-14T15:24:43.507773+0000","last_became_peered":"2024-04-14T15:24:43.507773+0000","last_unstale":"2024-04-14T15:24:43.508627+0000","last_undegraded":"2024-04-14T15:24:43.508627+0000","last_fullsized":"2024-04-14T15:24:43.508627+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":738,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:42.261473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:42.261473+0000","last_clean_scrub_stamp":"2024-04-14T15:24:42.261473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:08.098907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"333.1e","version":"0'0","reported_seq":10,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.330003+0000","last_change":"2024-04-14T15:24:43.309220+0000","last_active":"2024-04-14T15:24:43.330003+0000","last_peered":"2024-04-14T15:24:43.330003+0000","last_clean":"2024-04-14T15:24:43.330003+0000","last_became_active":"2024-04-14T15:24:43.308983+0000","last_became_peered":"2024-04-14T15:24:43.308983+0000","last_unstale":"2024-04-14T15:24:43.330003+0000","last_undegraded":"2024-04-14T15:24:43.330003+0000","last_fullsized":"2024-04-14T15:24:43.330003+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":738,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:42.261473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:42.261473+0000","last_clean_scrub_stamp":"2024-04-14T15:24:42.261473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:08.420713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"333.1d","version":"0'0","reported_seq":10,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.286260+0000","last_change":"2024-04-14T15:24:43.285825+0000","last_active":"2024-04-14T15:24:43.286260+0000","last_peered":"2024-04-14T15:24:43.286260+0000","last_clean":"2024-04-14T15:24:43.286260+0000","last_became_active":"2024-04-14T15:24:43.285433+0000","last_became_peered":"2024-04-14T15:24:43.285433+0000","last_unstale":"2024-04-14T15:24:43.286260+0000","last_undegraded":"2024-04-14T15:24:43.286260+0000","last_fullsized":"2024-04-14T15:24:43.286260+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":738,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:42.261473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:42.261473+0000","last_clean_scrub_stamp":"2024-04-14T15:24:42.261473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:45.407575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"333.1c","version":"0'0","reported_seq":10,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.507572+0000","last_change":"2024-04-14T15:24:43.506525+0000","last_active":"2024-04-14T15:24:43.507572+0000","last_peered":"2024-04-14T15:24:43.507572+0000","last_clean":"2024-04-14T15:24:43.507572+0000","last_became_active":"2024-04-14T15:24:43.506224+0000","last_became_peered":"2024-04-14T15:24:43.506224+0000","last_unstale":"2024-04-14T15:24:43.507572+0000","last_undegraded":"2024-04-14T15:24:43.507572+0000","last_fullsized":"2024-04-14T15:24:43.507572+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":738,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:42.261473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:42.261473+0000","last_clean_scrub_stamp":"2024-04-14T15:24:42.261473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:48.543465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"333.1b","version":"0'0","reported_seq":10,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.298251+0000","last_change":"2024-04-14T15:24:43.293098+0000","last_active":"2024-04-14T15:24:43.298251+0000","last_peered":"2024-04-14T15:24:43.298251+0000","last_clean":"2024-04-14T15:24:43.298251+0000","last_became_active":"2024-04-14T15:24:43.292813+0000","last_became_peered":"2024-04-14T15:24:43.292813+0000","last_unstale":"2024-04-14T15:24:43.298251+0000","last_undegraded":"2024-04-14T15:24:43.298251+0000","last_fullsized":"2024-04-14T15:24:43.298251+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":738,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:42.261473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:42.261473+0000","last_clean_scrub_stamp":"2024-04-14T15:24:42.261473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:12.172227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"333.1a","version":"0'0","reported_seq":10,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.507719+0000","last_change":"2024-04-14T15:24:43.507397+0000","last_active":"2024-04-14T15:24:43.507719+0000","last_peered":"2024-04-14T15:24:43.507719+0000","last_clean":"2024-04-14T15:24:43.507719+0000","last_became_active":"2024-04-14T15:24:43.507025+0000","last_became_peered":"2024-04-14T15:24:43.507025+0000","last_unstale":"2024-04-14T15:24:43.507719+0000","last_undegraded":"2024-04-14T15:24:43.507719+0000","last_fullsized":"2024-04-14T15:24:43.507719+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":738,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:42.261473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:42.261473+0000","last_clean_scrub_stamp":"2024-04-14T15:24:42.261473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:53.323281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"333.19","version":"0'0","reported_seq":10,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.509597+0000","last_change":"2024-04-14T15:24:43.507567+0000","last_active":"2024-04-14T15:24:43.509597+0000","last_peered":"2024-04-14T15:24:43.509597+0000","last_clean":"2024-04-14T15:24:43.509597+0000","last_became_active":"2024-04-14T15:24:43.506909+0000","last_became_peered":"2024-04-14T15:24:43.506909+0000","last_unstale":"2024-04-14T15:24:43.509597+0000","last_undegraded":"2024-04-14T15:24:43.509597+0000","last_fullsized":"2024-04-14T15:24:43.509597+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":738,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:42.261473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:42.261473+0000","last_clean_scrub_stamp":"2024-04-14T15:24:42.261473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:04.168114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"333.18","version":"0'0","reported_seq":10,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.509584+0000","last_change":"2024-04-14T15:24:43.506556+0000","last_active":"2024-04-14T15:24:43.509584+0000","last_peered":"2024-04-14T15:24:43.509584+0000","last_clean":"2024-04-14T15:24:43.509584+0000","last_became_active":"2024-04-14T15:24:43.506207+0000","last_became_peered":"2024-04-14T15:24:43.506207+0000","last_unstale":"2024-04-14T15:24:43.509584+0000","last_undegraded":"2024-04-14T15:24:43.509584+0000","last_fullsized":"2024-04-14T15:24:43.509584+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":738,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:42.261473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:42.261473+0000","last_clean_scrub_stamp":"2024-04-14T15:24:42.261473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:10.635987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"333.17","version":"0'0","reported_seq":10,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.509425+0000","last_change":"2024-04-14T15:24:43.508284+0000","last_active":"2024-04-14T15:24:43.509425+0000","last_peered":"2024-04-14T15:24:43.509425+0000","last_clean":"2024-04-14T15:24:43.509425+0000","last_became_active":"2024-04-14T15:24:43.508012+0000","last_became_peered":"2024-04-14T15:24:43.508012+0000","last_unstale":"2024-04-14T15:24:43.509425+0000","last_undegraded":"2024-04-14T15:24:43.509425+0000","last_fullsized":"2024-04-14T15:24:43.509425+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":738,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:42.261473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:42.261473+0000","last_clean_scrub_stamp":"2024-04-14T15:24:42.261473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:52.973675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"333.16","version":"0'0","reported_seq":10,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.509643+0000","last_change":"2024-04-14T15:24:43.508929+0000","last_active":"2024-04-14T15:24:43.509643+0000","last_peered":"2024-04-14T15:24:43.509643+0000","last_clean":"2024-04-14T15:24:43.509643+0000","last_became_active":"2024-04-14T15:24:43.508347+0000","last_became_peered":"2024-04-14T15:24:43.508347+0000","last_unstale":"2024-04-14T15:24:43.509643+0000","last_undegraded":"2024-04-14T15:24:43.509643+0000","last_fullsized":"2024-04-14T15:24:43.509643+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":738,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:42.261473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:42.261473+0000","last_clean_scrub_stamp":"2024-04-14T15:24:42.261473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:50.346763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"333.15","version":"0'0","reported_seq":10,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.308449+0000","last_change":"2024-04-14T15:24:43.302727+0000","last_active":"2024-04-14T15:24:43.308449+0000","last_peered":"2024-04-14T15:24:43.308449+0000","last_clean":"2024-04-14T15:24:43.308449+0000","last_became_active":"2024-04-14T15:24:43.302428+0000","last_became_peered":"2024-04-14T15:24:43.302428+0000","last_unstale":"2024-04-14T15:24:43.308449+0000","last_undegraded":"2024-04-14T15:24:43.308449+0000","last_fullsized":"2024-04-14T15:24:43.308449+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":738,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:42.261473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:42.261473+0000","last_clean_scrub_stamp":"2024-04-14T15:24:42.261473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:58.759528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"333.14","version":"0'0","reported_seq":10,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.298315+0000","last_change":"2024-04-14T15:24:43.293770+0000","last_active":"2024-04-14T15:24:43.298315+0000","last_peered":"2024-04-14T15:24:43.298315+0000","last_clean":"2024-04-14T15:24:43.298315+0000","last_became_active":"2024-04-14T15:24:43.293466+0000","last_became_peered":"2024-04-14T15:24:43.293466+0000","last_unstale":"2024-04-14T15:24:43.298315+0000","last_undegraded":"2024-04-14T15:24:43.298315+0000","last_fullsized":"2024-04-14T15:24:43.298315+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":738,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:42.261473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:42.261473+0000","last_clean_scrub_stamp":"2024-04-14T15:24:42.261473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:52.188212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"333.13","version":"0'0","reported_seq":10,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.292579+0000","last_change":"2024-04-14T15:24:43.292241+0000","last_active":"2024-04-14T15:24:43.292579+0000","last_peered":"2024-04-14T15:24:43.292579+0000","last_clean":"2024-04-14T15:24:43.292579+0000","last_became_active":"2024-04-14T15:24:43.291891+0000","last_became_peered":"2024-04-14T15:24:43.291891+0000","last_unstale":"2024-04-14T15:24:43.292579+0000","last_undegraded":"2024-04-14T15:24:43.292579+0000","last_fullsized":"2024-04-14T15:24:43.292579+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":738,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:42.261473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:42.261473+0000","last_clean_scrub_stamp":"2024-04-14T15:24:42.261473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:10.771909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"333.12","version":"0'0","reported_seq":10,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.298762+0000","last_change":"2024-04-14T15:24:43.298441+0000","last_active":"2024-04-14T15:24:43.298762+0000","last_peered":"2024-04-14T15:24:43.298762+0000","last_clean":"2024-04-14T15:24:43.298762+0000","last_became_active":"2024-04-14T15:24:43.298079+0000","last_became_peered":"2024-04-14T15:24:43.298079+0000","last_unstale":"2024-04-14T15:24:43.298762+0000","last_undegraded":"2024-04-14T15:24:43.298762+0000","last_fullsized":"2024-04-14T15:24:43.298762+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":738,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:42.261473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:42.261473+0000","last_clean_scrub_stamp":"2024-04-14T15:24:42.261473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:03.971878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"333.11","version":"0'0","reported_seq":10,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.298203+0000","last_change":"2024-04-14T15:24:43.292486+0000","last_active":"2024-04-14T15:24:43.298203+0000","last_peered":"2024-04-14T15:24:43.298203+0000","last_clean":"2024-04-14T15:24:43.298203+0000","last_became_active":"2024-04-14T15:24:43.290667+0000","last_became_peered":"2024-04-14T15:24:43.290667+0000","last_unstale":"2024-04-14T15:24:43.298203+0000","last_undegraded":"2024-04-14T15:24:43.298203+0000","last_fullsized":"2024-04-14T15:24:43.298203+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":738,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:42.261473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:42.261473+0000","last_clean_scrub_stamp":"2024-04-14T15:24:42.261473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:46.755083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"333.10","version":"0'0","reported_seq":10,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.509605+0000","last_change":"2024-04-14T15:24:43.506792+0000","last_active":"2024-04-14T15:24:43.509605+0000","last_peered":"2024-04-14T15:24:43.509605+0000","last_clean":"2024-04-14T15:24:43.509605+0000","last_became_active":"2024-04-14T15:24:43.506513+0000","last_became_peered":"2024-04-14T15:24:43.506513+0000","last_unstale":"2024-04-14T15:24:43.509605+0000","last_undegraded":"2024-04-14T15:24:43.509605+0000","last_fullsized":"2024-04-14T15:24:43.509605+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":738,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:42.261473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:42.261473+0000","last_clean_scrub_stamp":"2024-04-14T15:24:42.261473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:08.370846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"333.7","version":"0'0","reported_seq":10,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.508032+0000","last_change":"2024-04-14T15:24:43.507364+0000","last_active":"2024-04-14T15:24:43.508032+0000","last_peered":"2024-04-14T15:24:43.508032+0000","last_clean":"2024-04-14T15:24:43.508032+0000","last_became_active":"2024-04-14T15:24:43.507062+0000","last_became_peered":"2024-04-14T15:24:43.507062+0000","last_unstale":"2024-04-14T15:24:43.508032+0000","last_undegraded":"2024-04-14T15:24:43.508032+0000","last_fullsized":"2024-04-14T15:24:43.508032+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":738,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:42.261473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:42.261473+0000","last_clean_scrub_stamp":"2024-04-14T15:24:42.261473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:54.802144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"333.6","version":"0'0","reported_seq":10,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.508479+0000","last_change":"2024-04-14T15:24:43.506462+0000","last_active":"2024-04-14T15:24:43.508479+0000","last_peered":"2024-04-14T15:24:43.508479+0000","last_clean":"2024-04-14T15:24:43.508479+0000","last_became_active":"2024-04-14T15:24:43.506167+0000","last_became_peered":"2024-04-14T15:24:43.506167+0000","last_unstale":"2024-04-14T15:24:43.508479+0000","last_undegraded":"2024-04-14T15:24:43.508479+0000","last_fullsized":"2024-04-14T15:24:43.508479+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":738,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:42.261473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:42.261473+0000","last_clean_scrub_stamp":"2024-04-14T15:24:42.261473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:44.778637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"333.5","version":"0'0","reported_seq":10,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.292834+0000","last_change":"2024-04-14T15:24:43.292480+0000","last_active":"2024-04-14T15:24:43.292834+0000","last_peered":"2024-04-14T15:24:43.292834+0000","last_clean":"2024-04-14T15:24:43.292834+0000","last_became_active":"2024-04-14T15:24:43.292051+0000","last_became_peered":"2024-04-14T15:24:43.292051+0000","last_unstale":"2024-04-14T15:24:43.292834+0000","last_undegraded":"2024-04-14T15:24:43.292834+0000","last_fullsized":"2024-04-14T15:24:43.292834+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":738,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:42.261473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:42.261473+0000","last_clean_scrub_stamp":"2024-04-14T15:24:42.261473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:48.097996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"333.4","version":"0'0","reported_seq":10,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.300399+0000","last_change":"2024-04-14T15:24:43.299878+0000","last_active":"2024-04-14T15:24:43.300399+0000","last_peered":"2024-04-14T15:24:43.300399+0000","last_clean":"2024-04-14T15:24:43.300399+0000","last_became_active":"2024-04-14T15:24:43.299580+0000","last_became_peered":"2024-04-14T15:24:43.299580+0000","last_unstale":"2024-04-14T15:24:43.300399+0000","last_undegraded":"2024-04-14T15:24:43.300399+0000","last_fullsized":"2024-04-14T15:24:43.300399+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":738,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:42.261473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:42.261473+0000","last_clean_scrub_stamp":"2024-04-14T15:24:42.261473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:44.915568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"333.3","version":"0'0","reported_seq":10,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.308739+0000","last_change":"2024-04-14T15:24:43.302952+0000","last_active":"2024-04-14T15:24:43.308739+0000","last_peered":"2024-04-14T15:24:43.308739+0000","last_clean":"2024-04-14T15:24:43.308739+0000","last_became_active":"2024-04-14T15:24:43.302373+0000","last_became_peered":"2024-04-14T15:24:43.302373+0000","last_unstale":"2024-04-14T15:24:43.308739+0000","last_undegraded":"2024-04-14T15:24:43.308739+0000","last_fullsized":"2024-04-14T15:24:43.308739+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":738,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:42.261473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:42.261473+0000","last_clean_scrub_stamp":"2024-04-14T15:24:42.261473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:00.319594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"333.2","version":"0'0","reported_seq":10,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.301745+0000","last_change":"2024-04-14T15:24:43.301379+0000","last_active":"2024-04-14T15:24:43.301745+0000","last_peered":"2024-04-14T15:24:43.301745+0000","last_clean":"2024-04-14T15:24:43.301745+0000","last_became_active":"2024-04-14T15:24:43.301143+0000","last_became_peered":"2024-04-14T15:24:43.301143+0000","last_unstale":"2024-04-14T15:24:43.301745+0000","last_undegraded":"2024-04-14T15:24:43.301745+0000","last_fullsized":"2024-04-14T15:24:43.301745+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":738,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:42.261473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:42.261473+0000","last_clean_scrub_stamp":"2024-04-14T15:24:42.261473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:01.340893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"333.1","version":"0'0","reported_seq":10,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.306522+0000","last_change":"2024-04-14T15:24:43.305563+0000","last_active":"2024-04-14T15:24:43.306522+0000","last_peered":"2024-04-14T15:24:43.306522+0000","last_clean":"2024-04-14T15:24:43.306522+0000","last_became_active":"2024-04-14T15:24:43.305280+0000","last_became_peered":"2024-04-14T15:24:43.305280+0000","last_unstale":"2024-04-14T15:24:43.306522+0000","last_undegraded":"2024-04-14T15:24:43.306522+0000","last_fullsized":"2024-04-14T15:24:43.306522+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":738,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:42.261473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:42.261473+0000","last_clean_scrub_stamp":"2024-04-14T15:24:42.261473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:05.553247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"333.0","version":"0'0","reported_seq":10,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.291478+0000","last_change":"2024-04-14T15:24:43.291164+0000","last_active":"2024-04-14T15:24:43.291478+0000","last_peered":"2024-04-14T15:24:43.291478+0000","last_clean":"2024-04-14T15:24:43.291478+0000","last_became_active":"2024-04-14T15:24:43.290818+0000","last_became_peered":"2024-04-14T15:24:43.290818+0000","last_unstale":"2024-04-14T15:24:43.291478+0000","last_undegraded":"2024-04-14T15:24:43.291478+0000","last_fullsized":"2024-04-14T15:24:43.291478+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":738,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:42.261473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:42.261473+0000","last_clean_scrub_stamp":"2024-04-14T15:24:42.261473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:42.540101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"333.9","version":"0'0","reported_seq":10,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.287483+0000","last_change":"2024-04-14T15:24:43.287048+0000","last_active":"2024-04-14T15:24:43.287483+0000","last_peered":"2024-04-14T15:24:43.287483+0000","last_clean":"2024-04-14T15:24:43.287483+0000","last_became_active":"2024-04-14T15:24:43.286712+0000","last_became_peered":"2024-04-14T15:24:43.286712+0000","last_unstale":"2024-04-14T15:24:43.287483+0000","last_undegraded":"2024-04-14T15:24:43.287483+0000","last_fullsized":"2024-04-14T15:24:43.287483+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":738,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:42.261473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:42.261473+0000","last_clean_scrub_stamp":"2024-04-14T15:24:42.261473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:51.207331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"323.7","version":"0'0","reported_seq":242,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.307311+0000","last_change":"2024-04-14T15:24:15.999335+0000","last_active":"2024-04-14T15:24:43.307311+0000","last_peered":"2024-04-14T15:24:43.307311+0000","last_clean":"2024-04-14T15:24:43.307311+0000","last_became_active":"2024-04-14T15:23:04.005793+0000","last_became_peered":"2024-04-14T15:23:04.005793+0000","last_unstale":"2024-04-14T15:24:43.307311+0000","last_undegraded":"2024-04-14T15:24:43.307311+0000","last_fullsized":"2024-04-14T15:24:43.307311+0000","mapping_epoch":647,"log_start":"0'0","ondisk_log_start":"0'0","created":647,"last_epoch_clean":648,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_clean_scrub_stamp":"2024-04-14T15:23:02.982695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:45.892384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002596845,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"333.8","version":"0'0","reported_seq":10,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.295416+0000","last_change":"2024-04-14T15:24:43.295044+0000","last_active":"2024-04-14T15:24:43.295416+0000","last_peered":"2024-04-14T15:24:43.295416+0000","last_clean":"2024-04-14T15:24:43.295416+0000","last_became_active":"2024-04-14T15:24:43.294742+0000","last_became_peered":"2024-04-14T15:24:43.294742+0000","last_unstale":"2024-04-14T15:24:43.295416+0000","last_undegraded":"2024-04-14T15:24:43.295416+0000","last_fullsized":"2024-04-14T15:24:43.295416+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":738,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:42.261473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:42.261473+0000","last_clean_scrub_stamp":"2024-04-14T15:24:42.261473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:03.785439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"323.6","version":"656'1","reported_seq":250,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.290373+0000","last_change":"2024-04-14T15:24:16.002091+0000","last_active":"2024-04-14T15:24:43.290373+0000","last_peered":"2024-04-14T15:24:43.290373+0000","last_clean":"2024-04-14T15:24:43.290373+0000","last_became_active":"2024-04-14T15:23:04.006098+0000","last_became_peered":"2024-04-14T15:23:04.006098+0000","last_unstale":"2024-04-14T15:24:43.290373+0000","last_undegraded":"2024-04-14T15:24:43.290373+0000","last_fullsized":"2024-04-14T15:24:43.290373+0000","mapping_epoch":647,"log_start":"0'0","ondisk_log_start":"0'0","created":647,"last_epoch_clean":648,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_clean_scrub_stamp":"2024-04-14T15:23:02.982695+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:45.375244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0050417159999999999,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"333.b","version":"0'0","reported_seq":10,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.507681+0000","last_change":"2024-04-14T15:24:43.506651+0000","last_active":"2024-04-14T15:24:43.507681+0000","last_peered":"2024-04-14T15:24:43.507681+0000","last_clean":"2024-04-14T15:24:43.507681+0000","last_became_active":"2024-04-14T15:24:43.506338+0000","last_became_peered":"2024-04-14T15:24:43.506338+0000","last_unstale":"2024-04-14T15:24:43.507681+0000","last_undegraded":"2024-04-14T15:24:43.507681+0000","last_fullsized":"2024-04-14T15:24:43.507681+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":738,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:42.261473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:42.261473+0000","last_clean_scrub_stamp":"2024-04-14T15:24:42.261473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:08.369010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"323.5","version":"733'11","reported_seq":258,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.286832+0000","last_change":"2024-04-14T15:24:16.005799+0000","last_active":"2024-04-14T15:24:43.286832+0000","last_peered":"2024-04-14T15:24:43.286832+0000","last_clean":"2024-04-14T15:24:43.286832+0000","last_became_active":"2024-04-14T15:23:04.002370+0000","last_became_peered":"2024-04-14T15:23:04.002370+0000","last_unstale":"2024-04-14T15:24:43.286832+0000","last_undegraded":"2024-04-14T15:24:43.286832+0000","last_fullsized":"2024-04-14T15:24:43.286832+0000","mapping_epoch":647,"log_start":"0'0","ondisk_log_start":"0'0","created":647,"last_epoch_clean":648,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_clean_scrub_stamp":"2024-04-14T15:23:02.982695+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016554479999999999,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"333.a","version":"0'0","reported_seq":10,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.507936+0000","last_change":"2024-04-14T15:24:43.506793+0000","last_active":"2024-04-14T15:24:43.507936+0000","last_peered":"2024-04-14T15:24:43.507936+0000","last_clean":"2024-04-14T15:24:43.507936+0000","last_became_active":"2024-04-14T15:24:43.506495+0000","last_became_peered":"2024-04-14T15:24:43.506495+0000","last_unstale":"2024-04-14T15:24:43.507936+0000","last_undegraded":"2024-04-14T15:24:43.507936+0000","last_fullsized":"2024-04-14T15:24:43.507936+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":738,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:42.261473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:42.261473+0000","last_clean_scrub_stamp":"2024-04-14T15:24:42.261473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:46.635505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"323.4","version":"714'10","reported_seq":257,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.288556+0000","last_change":"2024-04-14T15:24:16.012928+0000","last_active":"2024-04-14T15:24:43.288556+0000","last_peered":"2024-04-14T15:24:43.288556+0000","last_clean":"2024-04-14T15:24:43.288556+0000","last_became_active":"2024-04-14T15:23:03.997912+0000","last_became_peered":"2024-04-14T15:23:03.997912+0000","last_unstale":"2024-04-14T15:24:43.288556+0000","last_undegraded":"2024-04-14T15:24:43.288556+0000","last_fullsized":"2024-04-14T15:24:43.288556+0000","mapping_epoch":647,"log_start":"0'0","ondisk_log_start":"0'0","created":647,"last_epoch_clean":648,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_clean_scrub_stamp":"2024-04-14T15:23:02.982695+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0025287679999999998,"stat_sum":{"num_bytes":10,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"333.d","version":"0'0","reported_seq":10,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.304936+0000","last_change":"2024-04-14T15:24:43.304576+0000","last_active":"2024-04-14T15:24:43.304936+0000","last_peered":"2024-04-14T15:24:43.304936+0000","last_clean":"2024-04-14T15:24:43.304936+0000","last_became_active":"2024-04-14T15:24:43.304262+0000","last_became_peered":"2024-04-14T15:24:43.304262+0000","last_unstale":"2024-04-14T15:24:43.304936+0000","last_undegraded":"2024-04-14T15:24:43.304936+0000","last_fullsized":"2024-04-14T15:24:43.304936+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":738,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:42.261473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:42.261473+0000","last_clean_scrub_stamp":"2024-04-14T15:24:42.261473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:06.488896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"323.3","version":"722'13","reported_seq":257,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.306555+0000","last_change":"2024-04-14T15:24:18.000702+0000","last_active":"2024-04-14T15:24:43.306555+0000","last_peered":"2024-04-14T15:24:43.306555+0000","last_clean":"2024-04-14T15:24:43.306555+0000","last_became_active":"2024-04-14T15:23:04.001213+0000","last_became_peered":"2024-04-14T15:23:04.001213+0000","last_unstale":"2024-04-14T15:24:43.306555+0000","last_undegraded":"2024-04-14T15:24:43.306555+0000","last_fullsized":"2024-04-14T15:24:43.306555+0000","mapping_epoch":647,"log_start":"0'0","ondisk_log_start":"0'0","created":647,"last_epoch_clean":648,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_clean_scrub_stamp":"2024-04-14T15:23:02.982695+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:45.892420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.25140135000000002,"stat_sum":{"num_bytes":21,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"333.c","version":"0'0","reported_seq":10,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.508475+0000","last_change":"2024-04-14T15:24:43.507826+0000","last_active":"2024-04-14T15:24:43.508475+0000","last_peered":"2024-04-14T15:24:43.508475+0000","last_clean":"2024-04-14T15:24:43.508475+0000","last_became_active":"2024-04-14T15:24:43.506482+0000","last_became_peered":"2024-04-14T15:24:43.506482+0000","last_unstale":"2024-04-14T15:24:43.508475+0000","last_undegraded":"2024-04-14T15:24:43.508475+0000","last_fullsized":"2024-04-14T15:24:43.508475+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":738,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:42.261473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:42.261473+0000","last_clean_scrub_stamp":"2024-04-14T15:24:42.261473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:47.125574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"323.2","version":"714'10","reported_seq":267,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.283263+0000","last_change":"2024-04-14T15:24:18.004566+0000","last_active":"2024-04-14T15:24:43.283263+0000","last_peered":"2024-04-14T15:24:43.283263+0000","last_clean":"2024-04-14T15:24:43.283263+0000","last_became_active":"2024-04-14T15:23:04.001557+0000","last_became_peered":"2024-04-14T15:23:04.001557+0000","last_unstale":"2024-04-14T15:24:43.283263+0000","last_undegraded":"2024-04-14T15:24:43.283263+0000","last_fullsized":"2024-04-14T15:24:43.283263+0000","mapping_epoch":647,"log_start":"0'0","ondisk_log_start":"0'0","created":647,"last_epoch_clean":648,"parent":"0.0","parent_split_bits":0,"last_scrub":"710'6","last_scrub_stamp":"2024-04-14T15:24:15.331688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_clean_scrub_stamp":"2024-04-14T15:24:15.331688+0000","objects_scrubbed":4,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:35.207524+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":1.0003185059999999,"stat_sum":{"num_bytes":10,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"333.f","version":"0'0","reported_seq":10,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.297094+0000","last_change":"2024-04-14T15:24:43.296053+0000","last_active":"2024-04-14T15:24:43.297094+0000","last_peered":"2024-04-14T15:24:43.297094+0000","last_clean":"2024-04-14T15:24:43.297094+0000","last_became_active":"2024-04-14T15:24:43.295748+0000","last_became_peered":"2024-04-14T15:24:43.295748+0000","last_unstale":"2024-04-14T15:24:43.297094+0000","last_undegraded":"2024-04-14T15:24:43.297094+0000","last_fullsized":"2024-04-14T15:24:43.297094+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":738,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:42.261473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:42.261473+0000","last_clean_scrub_stamp":"2024-04-14T15:24:42.261473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:53.050423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"323.1","version":"696'3","reported_seq":251,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.278161+0000","last_change":"2024-04-14T15:24:16.006310+0000","last_active":"2024-04-14T15:24:43.278161+0000","last_peered":"2024-04-14T15:24:43.278161+0000","last_clean":"2024-04-14T15:24:43.278161+0000","last_became_active":"2024-04-14T15:23:03.998498+0000","last_became_peered":"2024-04-14T15:23:03.998498+0000","last_unstale":"2024-04-14T15:24:43.278161+0000","last_undegraded":"2024-04-14T15:24:43.278161+0000","last_fullsized":"2024-04-14T15:24:43.278161+0000","mapping_epoch":647,"log_start":"0'0","ondisk_log_start":"0'0","created":647,"last_epoch_clean":648,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_clean_scrub_stamp":"2024-04-14T15:23:02.982695+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:45.750675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060225600000000002,"stat_sum":{"num_bytes":12,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"333.e","version":"0'0","reported_seq":10,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.509625+0000","last_change":"2024-04-14T15:24:43.508493+0000","last_active":"2024-04-14T15:24:43.509625+0000","last_peered":"2024-04-14T15:24:43.509625+0000","last_clean":"2024-04-14T15:24:43.509625+0000","last_became_active":"2024-04-14T15:24:43.508140+0000","last_became_peered":"2024-04-14T15:24:43.508140+0000","last_unstale":"2024-04-14T15:24:43.509625+0000","last_undegraded":"2024-04-14T15:24:43.509625+0000","last_fullsized":"2024-04-14T15:24:43.509625+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":738,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:42.261473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:42.261473+0000","last_clean_scrub_stamp":"2024-04-14T15:24:42.261473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:51.167136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"323.0","version":"704'6","reported_seq":255,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.282572+0000","last_change":"2024-04-14T15:24:15.995558+0000","last_active":"2024-04-14T15:24:43.282572+0000","last_peered":"2024-04-14T15:24:43.282572+0000","last_clean":"2024-04-14T15:24:43.282572+0000","last_became_active":"2024-04-14T15:23:04.001020+0000","last_became_peered":"2024-04-14T15:23:04.001020+0000","last_unstale":"2024-04-14T15:24:43.282572+0000","last_undegraded":"2024-04-14T15:24:43.282572+0000","last_fullsized":"2024-04-14T15:24:43.282572+0000","mapping_epoch":647,"log_start":"0'0","ondisk_log_start":"0'0","created":647,"last_epoch_clean":648,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_clean_scrub_stamp":"2024-04-14T15:23:02.982695+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:45.375222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024036109999999999,"stat_sum":{"num_bytes":5,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":956,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.294452+0000","last_change":"2024-04-14T15:23:27.292731+0000","last_active":"2024-04-14T15:24:43.294452+0000","last_peered":"2024-04-14T15:24:43.294452+0000","last_clean":"2024-04-14T15:24:43.294452+0000","last_became_active":"2024-04-14T15:15:44.882941+0000","last_became_peered":"2024-04-14T15:15:44.882941+0000","last_unstale":"2024-04-14T15:24:43.294452+0000","last_undegraded":"2024-04-14T15:24:43.294452+0000","last_fullsized":"2024-04-14T15:24:43.294452+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:27.292590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:23:27.292590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:44.375918+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.9","version":"528'10","reported_seq":1601,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.278945+0000","last_change":"2024-04-14T15:24:37.737273+0000","last_active":"2024-04-14T15:24:43.278945+0000","last_peered":"2024-04-14T15:24:43.278945+0000","last_clean":"2024-04-14T15:24:43.278945+0000","last_became_active":"2024-04-14T15:20:52.984943+0000","last_became_peered":"2024-04-14T15:20:52.984943+0000","last_unstale":"2024-04-14T15:24:43.278945+0000","last_undegraded":"2024-04-14T15:24:43.278945+0000","last_fullsized":"2024-04-14T15:24:43.278945+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"528'10","last_scrub_stamp":"2024-04-14T15:24:37.737147+0000","last_deep_scrub":"85'9","last_deep_scrub_stamp":"2024-04-14T15:10:51.950580+0000","last_clean_scrub_stamp":"2024-04-14T15:24:37.737147+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:37.881476+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"479'34","reported_seq":1043,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.278139+0000","last_change":"2024-04-14T15:21:27.711513+0000","last_active":"2024-04-14T15:24:43.278139+0000","last_peered":"2024-04-14T15:24:43.278139+0000","last_clean":"2024-04-14T15:24:43.278139+0000","last_became_active":"2024-04-14T15:20:52.979204+0000","last_became_peered":"2024-04-14T15:20:52.979204+0000","last_unstale":"2024-04-14T15:24:43.278139+0000","last_undegraded":"2024-04-14T15:24:43.278139+0000","last_fullsized":"2024-04-14T15:24:43.278139+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'34","last_scrub_stamp":"2024-04-14T15:21:27.711357+0000","last_deep_scrub":"430'22","last_deep_scrub_stamp":"2024-04-14T15:18:39.704883+0000","last_clean_scrub_stamp":"2024-04-14T15:21:27.711357+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:44.703540+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":1407,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.299248+0000","last_change":"2024-04-14T15:23:24.788057+0000","last_active":"2024-04-14T15:24:43.299248+0000","last_peered":"2024-04-14T15:24:43.299248+0000","last_clean":"2024-04-14T15:24:43.299248+0000","last_became_active":"2024-04-14T15:09:43.021460+0000","last_became_peered":"2024-04-14T15:09:43.021460+0000","last_unstale":"2024-04-14T15:24:43.299248+0000","last_undegraded":"2024-04-14T15:24:43.299248+0000","last_fullsized":"2024-04-14T15:24:43.299248+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:24.787958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:23:24.787958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:50.309267+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":443,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.285925+0000","last_change":"2024-04-14T15:23:05.437375+0000","last_active":"2024-04-14T15:24:43.285925+0000","last_peered":"2024-04-14T15:24:43.285925+0000","last_clean":"2024-04-14T15:24:43.285925+0000","last_became_active":"2024-04-14T15:20:52.989666+0000","last_became_peered":"2024-04-14T15:20:52.989666+0000","last_unstale":"2024-04-14T15:24:43.285925+0000","last_undegraded":"2024-04-14T15:24:43.285925+0000","last_fullsized":"2024-04-14T15:24:43.285925+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:05.437268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:15:40.368219+0000","last_clean_scrub_stamp":"2024-04-14T15:23:05.437268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:44.501964+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":1304,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.280183+0000","last_change":"2024-04-14T15:21:24.713987+0000","last_active":"2024-04-14T15:24:43.280183+0000","last_peered":"2024-04-14T15:24:43.280183+0000","last_clean":"2024-04-14T15:24:43.280183+0000","last_became_active":"2024-04-14T15:20:52.991952+0000","last_became_peered":"2024-04-14T15:20:52.991952+0000","last_unstale":"2024-04-14T15:24:43.280183+0000","last_undegraded":"2024-04-14T15:24:43.280183+0000","last_fullsized":"2024-04-14T15:24:43.280183+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:24.713877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:21:24.713877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:44.703466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"479'15","reported_seq":1429,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.286083+0000","last_change":"2024-04-14T15:23:32.253957+0000","last_active":"2024-04-14T15:24:43.286083+0000","last_peered":"2024-04-14T15:24:43.286083+0000","last_clean":"2024-04-14T15:24:43.286083+0000","last_became_active":"2024-04-14T15:09:45.037179+0000","last_became_peered":"2024-04-14T15:09:45.037179+0000","last_unstale":"2024-04-14T15:24:43.286083+0000","last_undegraded":"2024-04-14T15:24:43.286083+0000","last_fullsized":"2024-04-14T15:24:43.286083+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'15","last_scrub_stamp":"2024-04-14T15:23:32.253815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:23:32.253815+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:44.502024+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":1405,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.291710+0000","last_change":"2024-04-14T15:22:17.414577+0000","last_active":"2024-04-14T15:24:43.291710+0000","last_peered":"2024-04-14T15:24:43.291710+0000","last_clean":"2024-04-14T15:24:43.291710+0000","last_became_active":"2024-04-14T15:09:43.020044+0000","last_became_peered":"2024-04-14T15:09:43.020044+0000","last_unstale":"2024-04-14T15:24:43.291710+0000","last_undegraded":"2024-04-14T15:24:43.291710+0000","last_fullsized":"2024-04-14T15:24:43.291710+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:17.414499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:22:17.414499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:44.375773+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1407,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.296399+0000","last_change":"2024-04-14T15:24:27.475600+0000","last_active":"2024-04-14T15:24:43.296399+0000","last_peered":"2024-04-14T15:24:43.296399+0000","last_clean":"2024-04-14T15:24:43.296399+0000","last_became_active":"2024-04-14T15:09:47.046153+0000","last_became_peered":"2024-04-14T15:09:47.046153+0000","last_unstale":"2024-04-14T15:24:43.296399+0000","last_undegraded":"2024-04-14T15:24:43.296399+0000","last_fullsized":"2024-04-14T15:24:43.296399+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:27.475490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:24:27.475490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:53.144228+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1396,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.284147+0000","last_change":"2024-04-14T15:23:37.201923+0000","last_active":"2024-04-14T15:24:43.284147+0000","last_peered":"2024-04-14T15:24:43.284147+0000","last_clean":"2024-04-14T15:24:43.284147+0000","last_became_active":"2024-04-14T15:09:49.061495+0000","last_became_peered":"2024-04-14T15:09:49.061495+0000","last_unstale":"2024-04-14T15:24:43.284147+0000","last_undegraded":"2024-04-14T15:24:43.284147+0000","last_fullsized":"2024-04-14T15:24:43.284147+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:37.201795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:23:37.201795+0000","last_clean_scrub_stamp":"2024-04-14T15:23:37.201795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:49.569529+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":1304,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.281158+0000","last_change":"2024-04-14T15:21:20.684941+0000","last_active":"2024-04-14T15:24:43.281158+0000","last_peered":"2024-04-14T15:24:43.281158+0000","last_clean":"2024-04-14T15:24:43.281158+0000","last_became_active":"2024-04-14T15:20:52.999845+0000","last_became_peered":"2024-04-14T15:20:52.999845+0000","last_unstale":"2024-04-14T15:24:43.281158+0000","last_undegraded":"2024-04-14T15:24:43.281158+0000","last_fullsized":"2024-04-14T15:24:43.281158+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:20.684813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:21:20.684813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:44.703564+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"479'44","reported_seq":295,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.295695+0000","last_change":"2024-04-14T15:23:03.739862+0000","last_active":"2024-04-14T15:24:43.295695+0000","last_peered":"2024-04-14T15:24:43.295695+0000","last_clean":"2024-04-14T15:24:43.295695+0000","last_became_active":"2024-04-14T15:22:25.970899+0000","last_became_peered":"2024-04-14T15:22:25.970899+0000","last_unstale":"2024-04-14T15:24:43.295695+0000","last_undegraded":"2024-04-14T15:24:43.295695+0000","last_fullsized":"2024-04-14T15:24:43.295695+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'44","last_scrub_stamp":"2024-04-14T15:23:03.739762+0000","last_deep_scrub":"479'44","last_deep_scrub_stamp":"2024-04-14T15:19:58.856115+0000","last_clean_scrub_stamp":"2024-04-14T15:23:03.739762+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:44.445919+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":1415,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.299101+0000","last_change":"2024-04-14T15:23:44.419193+0000","last_active":"2024-04-14T15:24:43.299101+0000","last_peered":"2024-04-14T15:24:43.299101+0000","last_clean":"2024-04-14T15:24:43.299101+0000","last_became_active":"2024-04-14T15:09:43.020268+0000","last_became_peered":"2024-04-14T15:09:43.020268+0000","last_unstale":"2024-04-14T15:24:43.299101+0000","last_undegraded":"2024-04-14T15:24:43.299101+0000","last_fullsized":"2024-04-14T15:24:43.299101+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:44.419100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:34.342793+0000","last_clean_scrub_stamp":"2024-04-14T15:23:44.419100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:56.886676+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1399,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.297458+0000","last_change":"2024-04-14T15:24:23.499160+0000","last_active":"2024-04-14T15:24:43.297458+0000","last_peered":"2024-04-14T15:24:43.297458+0000","last_clean":"2024-04-14T15:24:43.297458+0000","last_became_active":"2024-04-14T15:09:47.045542+0000","last_became_peered":"2024-04-14T15:09:47.045542+0000","last_unstale":"2024-04-14T15:24:43.297458+0000","last_undegraded":"2024-04-14T15:24:43.297458+0000","last_fullsized":"2024-04-14T15:24:43.297458+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:23.499070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:24:23.499070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:23.657015+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1389,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.278376+0000","last_change":"2024-04-14T15:22:55.492856+0000","last_active":"2024-04-14T15:24:43.278376+0000","last_peered":"2024-04-14T15:24:43.278376+0000","last_clean":"2024-04-14T15:24:43.278376+0000","last_became_active":"2024-04-14T15:09:49.065397+0000","last_became_peered":"2024-04-14T15:09:49.065397+0000","last_unstale":"2024-04-14T15:24:43.278376+0000","last_undegraded":"2024-04-14T15:24:43.278376+0000","last_fullsized":"2024-04-14T15:24:43.278376+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:55.492753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:22:55.492753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:44.703710+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"91'130","reported_seq":1351,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.295282+0000","last_change":"2024-04-14T15:24:36.459484+0000","last_active":"2024-04-14T15:24:43.295282+0000","last_peered":"2024-04-14T15:24:43.295282+0000","last_clean":"2024-04-14T15:24:43.295282+0000","last_became_active":"2024-04-14T15:21:12.961419+0000","last_became_peered":"2024-04-14T15:21:12.961419+0000","last_unstale":"2024-04-14T15:24:43.295282+0000","last_undegraded":"2024-04-14T15:24:43.295282+0000","last_fullsized":"2024-04-14T15:24:43.295282+0000","mapping_epoch":276,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:24:36.459402+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:24:36.459402+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":70,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:43.342393+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":1304,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.296234+0000","last_change":"2024-04-14T15:22:14.964486+0000","last_active":"2024-04-14T15:24:43.296234+0000","last_peered":"2024-04-14T15:24:43.296234+0000","last_clean":"2024-04-14T15:24:43.296234+0000","last_became_active":"2024-04-14T15:12:56.474322+0000","last_became_peered":"2024-04-14T15:12:56.474322+0000","last_unstale":"2024-04-14T15:24:43.296234+0000","last_undegraded":"2024-04-14T15:24:43.296234+0000","last_fullsized":"2024-04-14T15:24:43.296234+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:14.964404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:22:14.964404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:44.445835+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"479'31","reported_seq":1495,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.287530+0000","last_change":"2024-04-14T15:22:02.037366+0000","last_active":"2024-04-14T15:24:43.287530+0000","last_peered":"2024-04-14T15:24:43.287530+0000","last_clean":"2024-04-14T15:24:43.287530+0000","last_became_active":"2024-04-14T15:13:12.933907+0000","last_became_peered":"2024-04-14T15:13:12.933907+0000","last_unstale":"2024-04-14T15:24:43.287530+0000","last_undegraded":"2024-04-14T15:24:43.287530+0000","last_fullsized":"2024-04-14T15:24:43.287530+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'31","last_scrub_stamp":"2024-04-14T15:22:02.037237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:22:02.037237+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":1401,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.280293+0000","last_change":"2024-04-14T15:23:26.611299+0000","last_active":"2024-04-14T15:24:43.280293+0000","last_peered":"2024-04-14T15:24:43.280293+0000","last_clean":"2024-04-14T15:24:43.280293+0000","last_became_active":"2024-04-14T15:09:43.022351+0000","last_became_peered":"2024-04-14T15:09:43.022351+0000","last_unstale":"2024-04-14T15:24:43.280293+0000","last_undegraded":"2024-04-14T15:24:43.280293+0000","last_fullsized":"2024-04-14T15:24:43.280293+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:26.611217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:23:26.611217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:45.750697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1397,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.287365+0000","last_change":"2024-04-14T15:23:25.344592+0000","last_active":"2024-04-14T15:24:43.287365+0000","last_peered":"2024-04-14T15:24:43.287365+0000","last_clean":"2024-04-14T15:24:43.287365+0000","last_became_active":"2024-04-14T15:09:47.042810+0000","last_became_peered":"2024-04-14T15:09:47.042810+0000","last_unstale":"2024-04-14T15:24:43.287365+0000","last_undegraded":"2024-04-14T15:24:43.287365+0000","last_fullsized":"2024-04-14T15:24:43.287365+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:25.344505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:23:25.344505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:44.375856+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1395,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.297724+0000","last_change":"2024-04-14T15:22:58.703767+0000","last_active":"2024-04-14T15:24:43.297724+0000","last_peered":"2024-04-14T15:24:43.297724+0000","last_clean":"2024-04-14T15:24:43.297724+0000","last_became_active":"2024-04-14T15:09:49.069594+0000","last_became_peered":"2024-04-14T15:09:49.069594+0000","last_unstale":"2024-04-14T15:24:43.297724+0000","last_undegraded":"2024-04-14T15:24:43.297724+0000","last_fullsized":"2024-04-14T15:24:43.297724+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:58.703688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:22:58.703688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:44.445934+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"91'130","reported_seq":1746,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.278424+0000","last_change":"2024-04-14T15:22:04.530538+0000","last_active":"2024-04-14T15:24:43.278424+0000","last_peered":"2024-04-14T15:24:43.278424+0000","last_clean":"2024-04-14T15:24:43.278424+0000","last_became_active":"2024-04-14T15:21:09.020300+0000","last_became_peered":"2024-04-14T15:21:09.020300+0000","last_unstale":"2024-04-14T15:24:43.278424+0000","last_undegraded":"2024-04-14T15:24:43.278424+0000","last_fullsized":"2024-04-14T15:24:43.278424+0000","mapping_epoch":189,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":544,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:22:04.530456+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:22:04.530456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:44.703585+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":441,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.279264+0000","last_change":"2024-04-14T15:23:41.487909+0000","last_active":"2024-04-14T15:24:43.279264+0000","last_peered":"2024-04-14T15:24:43.279264+0000","last_clean":"2024-04-14T15:24:43.279264+0000","last_became_active":"2024-04-14T15:20:52.980375+0000","last_became_peered":"2024-04-14T15:20:52.980375+0000","last_unstale":"2024-04-14T15:24:43.279264+0000","last_undegraded":"2024-04-14T15:24:43.279264+0000","last_fullsized":"2024-04-14T15:24:43.279264+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:41.487814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:23:41.487814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:56.105699+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"479'25","reported_seq":1451,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.286545+0000","last_change":"2024-04-14T15:24:34.394922+0000","last_active":"2024-04-14T15:24:43.286545+0000","last_peered":"2024-04-14T15:24:43.286545+0000","last_clean":"2024-04-14T15:24:43.286545+0000","last_became_active":"2024-04-14T15:20:52.987699+0000","last_became_peered":"2024-04-14T15:20:52.987699+0000","last_unstale":"2024-04-14T15:24:43.286545+0000","last_undegraded":"2024-04-14T15:24:43.286545+0000","last_fullsized":"2024-04-14T15:24:43.286545+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'25","last_scrub_stamp":"2024-04-14T15:24:34.394825+0000","last_deep_scrub":"479'25","last_deep_scrub_stamp":"2024-04-14T15:20:03.571160+0000","last_clean_scrub_stamp":"2024-04-14T15:24:34.394825+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:35.077282+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":1405,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.283711+0000","last_change":"2024-04-14T15:21:31.700100+0000","last_active":"2024-04-14T15:24:43.283711+0000","last_peered":"2024-04-14T15:24:43.283711+0000","last_clean":"2024-04-14T15:24:43.283711+0000","last_became_active":"2024-04-14T15:09:43.018871+0000","last_became_peered":"2024-04-14T15:09:43.018871+0000","last_unstale":"2024-04-14T15:24:43.283711+0000","last_undegraded":"2024-04-14T15:24:43.283711+0000","last_fullsized":"2024-04-14T15:24:43.283711+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:31.700000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:21:31.700000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:44.375665+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"400'9","reported_seq":1589,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.296911+0000","last_change":"2024-04-14T15:23:30.695693+0000","last_active":"2024-04-14T15:24:43.296911+0000","last_peered":"2024-04-14T15:24:43.296911+0000","last_clean":"2024-04-14T15:24:43.296911+0000","last_became_active":"2024-04-14T15:09:47.044085+0000","last_became_peered":"2024-04-14T15:09:47.044085+0000","last_unstale":"2024-04-14T15:24:43.296911+0000","last_undegraded":"2024-04-14T15:24:43.296911+0000","last_fullsized":"2024-04-14T15:24:43.296911+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'9","last_scrub_stamp":"2024-04-14T15:23:30.695563+0000","last_deep_scrub":"400'9","last_deep_scrub_stamp":"2024-04-14T15:17:54.845881+0000","last_clean_scrub_stamp":"2024-04-14T15:23:30.695563+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1390,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.288094+0000","last_change":"2024-04-14T15:24:19.552836+0000","last_active":"2024-04-14T15:24:43.288094+0000","last_peered":"2024-04-14T15:24:43.288094+0000","last_clean":"2024-04-14T15:24:43.288094+0000","last_became_active":"2024-04-14T15:09:49.065274+0000","last_became_peered":"2024-04-14T15:09:49.065274+0000","last_unstale":"2024-04-14T15:24:43.288094+0000","last_undegraded":"2024-04-14T15:24:43.288094+0000","last_fullsized":"2024-04-14T15:24:43.288094+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:19.552703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:18.588546+0000","last_clean_scrub_stamp":"2024-04-14T15:24:19.552703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:36.798103+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"91'130","reported_seq":1358,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.296075+0000","last_change":"2024-04-14T15:22:06.865915+0000","last_active":"2024-04-14T15:24:43.296075+0000","last_peered":"2024-04-14T15:24:43.296075+0000","last_clean":"2024-04-14T15:24:43.296075+0000","last_became_active":"2024-04-14T15:21:04.971515+0000","last_became_peered":"2024-04-14T15:21:04.971515+0000","last_unstale":"2024-04-14T15:24:43.296075+0000","last_undegraded":"2024-04-14T15:24:43.296075+0000","last_fullsized":"2024-04-14T15:24:43.296075+0000","mapping_epoch":530,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":540,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:22:06.865825+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:22:06.865825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:44.445802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":1302,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.283203+0000","last_change":"2024-04-14T15:23:29.219186+0000","last_active":"2024-04-14T15:24:43.283203+0000","last_peered":"2024-04-14T15:24:43.283203+0000","last_clean":"2024-04-14T15:24:43.283203+0000","last_became_active":"2024-04-14T15:12:56.487615+0000","last_became_peered":"2024-04-14T15:12:56.487615+0000","last_unstale":"2024-04-14T15:24:43.283203+0000","last_undegraded":"2024-04-14T15:24:43.283203+0000","last_fullsized":"2024-04-14T15:24:43.283203+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:29.219111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:23:29.219111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:44.375871+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"479'34","reported_seq":1459,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.294595+0000","last_change":"2024-04-14T15:24:35.378498+0000","last_active":"2024-04-14T15:24:43.294595+0000","last_peered":"2024-04-14T15:24:43.294595+0000","last_clean":"2024-04-14T15:24:43.294595+0000","last_became_active":"2024-04-14T15:09:45.033331+0000","last_became_peered":"2024-04-14T15:09:45.033331+0000","last_unstale":"2024-04-14T15:24:43.294595+0000","last_undegraded":"2024-04-14T15:24:43.294595+0000","last_fullsized":"2024-04-14T15:24:43.294595+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'34","last_scrub_stamp":"2024-04-14T15:24:35.378379+0000","last_deep_scrub":"479'34","last_deep_scrub_stamp":"2024-04-14T15:24:35.378379+0000","last_clean_scrub_stamp":"2024-04-14T15:24:35.378379+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:53.212057+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":1405,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.282812+0000","last_change":"2024-04-14T15:22:55.384701+0000","last_active":"2024-04-14T15:24:43.282812+0000","last_peered":"2024-04-14T15:24:43.282812+0000","last_clean":"2024-04-14T15:24:43.282812+0000","last_became_active":"2024-04-14T15:09:43.020878+0000","last_became_peered":"2024-04-14T15:09:43.020878+0000","last_unstale":"2024-04-14T15:24:43.282812+0000","last_undegraded":"2024-04-14T15:24:43.282812+0000","last_fullsized":"2024-04-14T15:24:43.282812+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:55.384626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:22:55.384626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:44.375822+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"72'8","reported_seq":1583,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.288026+0000","last_change":"2024-04-14T15:24:27.403836+0000","last_active":"2024-04-14T15:24:43.288026+0000","last_peered":"2024-04-14T15:24:43.288026+0000","last_clean":"2024-04-14T15:24:43.288026+0000","last_became_active":"2024-04-14T15:09:47.050962+0000","last_became_peered":"2024-04-14T15:09:47.050962+0000","last_unstale":"2024-04-14T15:24:43.288026+0000","last_undegraded":"2024-04-14T15:24:43.288026+0000","last_fullsized":"2024-04-14T15:24:43.288026+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:24:27.403735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:24:27.403735+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:30.439681+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":965,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.291173+0000","last_change":"2024-04-14T15:24:27.318413+0000","last_active":"2024-04-14T15:24:43.291173+0000","last_peered":"2024-04-14T15:24:43.291173+0000","last_clean":"2024-04-14T15:24:43.291173+0000","last_became_active":"2024-04-14T15:15:44.879699+0000","last_became_peered":"2024-04-14T15:15:44.879699+0000","last_unstale":"2024-04-14T15:24:43.291173+0000","last_undegraded":"2024-04-14T15:24:43.291173+0000","last_fullsized":"2024-04-14T15:24:43.291173+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:27.318330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:24:27.318330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:32.125472+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"233'8","reported_seq":719,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.304212+0000","last_change":"2024-04-14T15:23:43.908078+0000","last_active":"2024-04-14T15:24:43.304212+0000","last_peered":"2024-04-14T15:24:43.304212+0000","last_clean":"2024-04-14T15:24:43.304212+0000","last_became_active":"2024-04-14T15:20:52.995129+0000","last_became_peered":"2024-04-14T15:20:52.995129+0000","last_unstale":"2024-04-14T15:24:43.304212+0000","last_undegraded":"2024-04-14T15:24:43.304212+0000","last_fullsized":"2024-04-14T15:24:43.304212+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'8","last_scrub_stamp":"2024-04-14T15:23:43.907946+0000","last_deep_scrub":"233'8","last_deep_scrub_stamp":"2024-04-14T15:15:01.295800+0000","last_clean_scrub_stamp":"2024-04-14T15:23:43.907946+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:59.919832+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":5,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"479'24","reported_seq":1452,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.287867+0000","last_change":"2024-04-14T15:21:13.939265+0000","last_active":"2024-04-14T15:24:43.287867+0000","last_peered":"2024-04-14T15:24:43.287867+0000","last_clean":"2024-04-14T15:24:43.287867+0000","last_became_active":"2024-04-14T15:20:52.998853+0000","last_became_peered":"2024-04-14T15:20:52.998853+0000","last_unstale":"2024-04-14T15:24:43.287867+0000","last_undegraded":"2024-04-14T15:24:43.287867+0000","last_fullsized":"2024-04-14T15:24:43.287867+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'24","last_scrub_stamp":"2024-04-14T15:21:13.939121+0000","last_deep_scrub":"85'16","last_deep_scrub_stamp":"2024-04-14T15:10:47.456169+0000","last_clean_scrub_stamp":"2024-04-14T15:21:13.939121+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":1409,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.278269+0000","last_change":"2024-04-14T15:23:36.489554+0000","last_active":"2024-04-14T15:24:43.278269+0000","last_peered":"2024-04-14T15:24:43.278269+0000","last_clean":"2024-04-14T15:24:43.278269+0000","last_became_active":"2024-04-14T15:09:43.018601+0000","last_became_peered":"2024-04-14T15:09:43.018601+0000","last_unstale":"2024-04-14T15:24:43.278269+0000","last_undegraded":"2024-04-14T15:24:43.278269+0000","last_fullsized":"2024-04-14T15:24:43.278269+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:36.489475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:23:36.489475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:53.352519+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":967,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.288302+0000","last_change":"2024-04-14T15:24:35.426298+0000","last_active":"2024-04-14T15:24:43.288302+0000","last_peered":"2024-04-14T15:24:43.288302+0000","last_clean":"2024-04-14T15:24:43.288302+0000","last_became_active":"2024-04-14T15:20:52.970831+0000","last_became_peered":"2024-04-14T15:20:52.970831+0000","last_unstale":"2024-04-14T15:24:43.288302+0000","last_undegraded":"2024-04-14T15:24:43.288302+0000","last_fullsized":"2024-04-14T15:24:43.288302+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:35.426204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:24:35.426204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:51.358261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1075,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.306757+0000","last_change":"2024-04-14T15:24:32.995639+0000","last_active":"2024-04-14T15:24:43.306757+0000","last_peered":"2024-04-14T15:24:43.306757+0000","last_clean":"2024-04-14T15:24:43.306757+0000","last_became_active":"2024-04-14T15:20:52.999771+0000","last_became_peered":"2024-04-14T15:20:52.999771+0000","last_unstale":"2024-04-14T15:24:43.306757+0000","last_undegraded":"2024-04-14T15:24:43.306757+0000","last_fullsized":"2024-04-14T15:24:43.306757+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:32.995512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:57.214046+0000","last_clean_scrub_stamp":"2024-04-14T15:24:32.995512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:37.796818+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"91'130","reported_seq":1541,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.306673+0000","last_change":"2024-04-14T15:24:35.949096+0000","last_active":"2024-04-14T15:24:43.306673+0000","last_peered":"2024-04-14T15:24:43.306673+0000","last_clean":"2024-04-14T15:24:43.306673+0000","last_became_active":"2024-04-14T15:23:29.044065+0000","last_became_peered":"2024-04-14T15:23:29.044065+0000","last_unstale":"2024-04-14T15:24:43.306673+0000","last_undegraded":"2024-04-14T15:24:43.306673+0000","last_fullsized":"2024-04-14T15:24:43.306673+0000","mapping_epoch":189,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":672,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:24:35.948962+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:24:35.948962+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":60,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:43.083410+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":1307,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.287767+0000","last_change":"2024-04-14T15:24:37.496873+0000","last_active":"2024-04-14T15:24:43.287767+0000","last_peered":"2024-04-14T15:24:43.287767+0000","last_clean":"2024-04-14T15:24:43.287767+0000","last_became_active":"2024-04-14T15:12:56.427276+0000","last_became_peered":"2024-04-14T15:12:56.427276+0000","last_unstale":"2024-04-14T15:24:43.287767+0000","last_undegraded":"2024-04-14T15:24:43.287767+0000","last_fullsized":"2024-04-14T15:24:43.287767+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:37.496778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:24:37.496778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:03.328438+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"255'10","reported_seq":1223,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.304273+0000","last_change":"2024-04-14T15:23:26.964698+0000","last_active":"2024-04-14T15:24:43.304273+0000","last_peered":"2024-04-14T15:24:43.304273+0000","last_clean":"2024-04-14T15:24:43.304273+0000","last_became_active":"2024-04-14T15:15:21.873747+0000","last_became_peered":"2024-04-14T15:15:21.873747+0000","last_unstale":"2024-04-14T15:24:43.304273+0000","last_undegraded":"2024-04-14T15:24:43.304273+0000","last_fullsized":"2024-04-14T15:24:43.304273+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'10","last_scrub_stamp":"2024-04-14T15:23:26.964616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:23:26.964616+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:45.892531+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"479'28","reported_seq":1463,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.285904+0000","last_change":"2024-04-14T15:22:09.093925+0000","last_active":"2024-04-14T15:24:43.285904+0000","last_peered":"2024-04-14T15:24:43.285904+0000","last_clean":"2024-04-14T15:24:43.285904+0000","last_became_active":"2024-04-14T15:20:52.989635+0000","last_became_peered":"2024-04-14T15:20:52.989635+0000","last_unstale":"2024-04-14T15:24:43.285904+0000","last_undegraded":"2024-04-14T15:24:43.285904+0000","last_fullsized":"2024-04-14T15:24:43.285904+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'28","last_scrub_stamp":"2024-04-14T15:22:09.093771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:22:09.093771+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":441,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.281163+0000","last_change":"2024-04-14T15:24:32.603456+0000","last_active":"2024-04-14T15:24:43.281163+0000","last_peered":"2024-04-14T15:24:43.281163+0000","last_clean":"2024-04-14T15:24:43.281163+0000","last_became_active":"2024-04-14T15:20:52.999379+0000","last_became_peered":"2024-04-14T15:20:52.999379+0000","last_unstale":"2024-04-14T15:24:43.281163+0000","last_undegraded":"2024-04-14T15:24:43.281163+0000","last_fullsized":"2024-04-14T15:24:43.281163+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:32.603327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:24:32.603327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:45.890218+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1396,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.279885+0000","last_change":"2024-04-14T15:23:28.571478+0000","last_active":"2024-04-14T15:24:43.279885+0000","last_peered":"2024-04-14T15:24:43.279885+0000","last_clean":"2024-04-14T15:24:43.279885+0000","last_became_active":"2024-04-14T15:20:52.991388+0000","last_became_peered":"2024-04-14T15:20:52.991388+0000","last_unstale":"2024-04-14T15:24:43.279885+0000","last_undegraded":"2024-04-14T15:24:43.279885+0000","last_fullsized":"2024-04-14T15:24:43.279885+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:28.571374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:23:28.571374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:45.653854+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"91'130","reported_seq":1551,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.286966+0000","last_change":"2024-04-14T15:23:06.153451+0000","last_active":"2024-04-14T15:24:43.286966+0000","last_peered":"2024-04-14T15:24:43.286966+0000","last_clean":"2024-04-14T15:24:43.286966+0000","last_became_active":"2024-04-14T15:23:02.990846+0000","last_became_peered":"2024-04-14T15:23:02.990846+0000","last_unstale":"2024-04-14T15:24:43.286966+0000","last_undegraded":"2024-04-14T15:24:43.286966+0000","last_fullsized":"2024-04-14T15:24:43.286966+0000","mapping_epoch":530,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":647,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:23:06.153339+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:23:06.153339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":1306,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.280932+0000","last_change":"2024-04-14T15:24:27.475544+0000","last_active":"2024-04-14T15:24:43.280932+0000","last_peered":"2024-04-14T15:24:43.280932+0000","last_clean":"2024-04-14T15:24:43.280932+0000","last_became_active":"2024-04-14T15:12:56.355254+0000","last_became_peered":"2024-04-14T15:12:56.355254+0000","last_unstale":"2024-04-14T15:24:43.280932+0000","last_undegraded":"2024-04-14T15:24:43.280932+0000","last_fullsized":"2024-04-14T15:24:43.280932+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:27.475460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:24:27.475460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:35.815654+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1395,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.286101+0000","last_change":"2024-04-14T15:22:55.245731+0000","last_active":"2024-04-14T15:24:43.286101+0000","last_peered":"2024-04-14T15:24:43.286101+0000","last_clean":"2024-04-14T15:24:43.286101+0000","last_became_active":"2024-04-14T15:09:47.044610+0000","last_became_peered":"2024-04-14T15:09:47.044610+0000","last_unstale":"2024-04-14T15:24:43.286101+0000","last_undegraded":"2024-04-14T15:24:43.286101+0000","last_fullsized":"2024-04-14T15:24:43.286101+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:55.245639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:39.394655+0000","last_clean_scrub_stamp":"2024-04-14T15:22:55.245639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"479'30","reported_seq":1128,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.304167+0000","last_change":"2024-04-14T15:22:58.001736+0000","last_active":"2024-04-14T15:24:43.304167+0000","last_peered":"2024-04-14T15:24:43.304167+0000","last_clean":"2024-04-14T15:24:43.304167+0000","last_became_active":"2024-04-14T15:15:21.880565+0000","last_became_peered":"2024-04-14T15:15:21.880565+0000","last_unstale":"2024-04-14T15:24:43.304167+0000","last_undegraded":"2024-04-14T15:24:43.304167+0000","last_fullsized":"2024-04-14T15:24:43.304167+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'30","last_scrub_stamp":"2024-04-14T15:22:58.001609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:22:58.001609+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:45.892483+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'2","reported_seq":1409,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.290688+0000","last_change":"2024-04-14T15:21:25.663010+0000","last_active":"2024-04-14T15:24:43.290688+0000","last_peered":"2024-04-14T15:24:43.290688+0000","last_clean":"2024-04-14T15:24:43.290688+0000","last_became_active":"2024-04-14T15:20:52.987576+0000","last_became_peered":"2024-04-14T15:20:52.987576+0000","last_unstale":"2024-04-14T15:24:43.290688+0000","last_undegraded":"2024-04-14T15:24:43.290688+0000","last_fullsized":"2024-04-14T15:24:43.290688+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'2","last_scrub_stamp":"2024-04-14T15:21:25.662901+0000","last_deep_scrub":"91'2","last_deep_scrub_stamp":"2024-04-14T15:17:48.850069+0000","last_clean_scrub_stamp":"2024-04-14T15:21:25.662901+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:44.375631+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1391,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.287466+0000","last_change":"2024-04-14T15:23:33.834918+0000","last_active":"2024-04-14T15:24:43.287466+0000","last_peered":"2024-04-14T15:24:43.287466+0000","last_clean":"2024-04-14T15:24:43.287466+0000","last_became_active":"2024-04-14T15:09:49.065684+0000","last_became_peered":"2024-04-14T15:09:49.065684+0000","last_unstale":"2024-04-14T15:24:43.287466+0000","last_undegraded":"2024-04-14T15:24:43.287466+0000","last_fullsized":"2024-04-14T15:24:43.287466+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:33.834838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:23:33.834838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"91'130","reported_seq":819,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.289976+0000","last_change":"2024-04-14T15:23:43.310976+0000","last_active":"2024-04-14T15:24:43.289976+0000","last_peered":"2024-04-14T15:24:43.289976+0000","last_clean":"2024-04-14T15:24:43.289976+0000","last_became_active":"2024-04-14T15:22:58.991118+0000","last_became_peered":"2024-04-14T15:22:58.991118+0000","last_unstale":"2024-04-14T15:24:43.289976+0000","last_undegraded":"2024-04-14T15:24:43.289976+0000","last_fullsized":"2024-04-14T15:24:43.289976+0000","mapping_epoch":530,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:23:43.310892+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:23:43.310892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:49.797838+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":1310,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.286546+0000","last_change":"2024-04-14T15:23:24.298794+0000","last_active":"2024-04-14T15:24:43.286546+0000","last_peered":"2024-04-14T15:24:43.286546+0000","last_clean":"2024-04-14T15:24:43.286546+0000","last_became_active":"2024-04-14T15:12:56.402023+0000","last_became_peered":"2024-04-14T15:12:56.402023+0000","last_unstale":"2024-04-14T15:24:43.286546+0000","last_undegraded":"2024-04-14T15:24:43.286546+0000","last_fullsized":"2024-04-14T15:24:43.286546+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:24.298717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:35.003968+0000","last_clean_scrub_stamp":"2024-04-14T15:23:24.298717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:44.375902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":447,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.285771+0000","last_change":"2024-04-14T15:24:18.368805+0000","last_active":"2024-04-14T15:24:43.285771+0000","last_peered":"2024-04-14T15:24:43.285771+0000","last_clean":"2024-04-14T15:24:43.285771+0000","last_became_active":"2024-04-14T15:20:52.987883+0000","last_became_peered":"2024-04-14T15:20:52.987883+0000","last_unstale":"2024-04-14T15:24:43.285771+0000","last_undegraded":"2024-04-14T15:24:43.285771+0000","last_fullsized":"2024-04-14T15:24:43.285771+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:18.368725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:24:18.368725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:42.909210+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"479'37","reported_seq":1464,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.286276+0000","last_change":"2024-04-14T15:23:38.246620+0000","last_active":"2024-04-14T15:24:43.286276+0000","last_peered":"2024-04-14T15:24:43.286276+0000","last_clean":"2024-04-14T15:24:43.286276+0000","last_became_active":"2024-04-14T15:09:45.179182+0000","last_became_peered":"2024-04-14T15:09:45.179182+0000","last_unstale":"2024-04-14T15:24:43.286276+0000","last_undegraded":"2024-04-14T15:24:43.286276+0000","last_fullsized":"2024-04-14T15:24:43.286276+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'37","last_scrub_stamp":"2024-04-14T15:23:38.246517+0000","last_deep_scrub":"102'27","last_deep_scrub_stamp":"2024-04-14T15:17:46.924330+0000","last_clean_scrub_stamp":"2024-04-14T15:23:38.246517+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1402,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.281041+0000","last_change":"2024-04-14T15:22:56.541983+0000","last_active":"2024-04-14T15:24:43.281041+0000","last_peered":"2024-04-14T15:24:43.281041+0000","last_clean":"2024-04-14T15:24:43.281041+0000","last_became_active":"2024-04-14T15:09:43.021006+0000","last_became_peered":"2024-04-14T15:09:43.021006+0000","last_unstale":"2024-04-14T15:24:43.281041+0000","last_undegraded":"2024-04-14T15:24:43.281041+0000","last_fullsized":"2024-04-14T15:24:43.281041+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:56.541899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:22:56.541899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:44.703659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1400,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.282962+0000","last_change":"2024-04-14T15:23:34.182007+0000","last_active":"2024-04-14T15:24:43.282962+0000","last_peered":"2024-04-14T15:24:43.282962+0000","last_clean":"2024-04-14T15:24:43.282962+0000","last_became_active":"2024-04-14T15:20:52.978765+0000","last_became_peered":"2024-04-14T15:20:52.978765+0000","last_unstale":"2024-04-14T15:24:43.282962+0000","last_undegraded":"2024-04-14T15:24:43.282962+0000","last_fullsized":"2024-04-14T15:24:43.282962+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:34.181914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:56.206649+0000","last_clean_scrub_stamp":"2024-04-14T15:23:34.181914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:48.624030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"91'130","reported_seq":1549,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.283553+0000","last_change":"2024-04-14T15:23:32.988316+0000","last_active":"2024-04-14T15:24:43.283553+0000","last_peered":"2024-04-14T15:24:43.283553+0000","last_clean":"2024-04-14T15:24:43.283553+0000","last_became_active":"2024-04-14T15:23:32.987931+0000","last_became_peered":"2024-04-14T15:23:32.987931+0000","last_unstale":"2024-04-14T15:24:43.283553+0000","last_undegraded":"2024-04-14T15:24:43.283553+0000","last_fullsized":"2024-04-14T15:24:43.283553+0000","mapping_epoch":530,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:22:57.365464+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:22:57.365464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:44.375808+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":1015,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.304422+0000","last_change":"2024-04-14T15:23:30.968804+0000","last_active":"2024-04-14T15:24:43.304422+0000","last_peered":"2024-04-14T15:24:43.304422+0000","last_clean":"2024-04-14T15:24:43.304422+0000","last_became_active":"2024-04-14T15:20:52.999776+0000","last_became_peered":"2024-04-14T15:20:52.999776+0000","last_unstale":"2024-04-14T15:24:43.304422+0000","last_undegraded":"2024-04-14T15:24:43.304422+0000","last_fullsized":"2024-04-14T15:24:43.304422+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:30.968711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:23:30.968711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:45.892547+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":956,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.294910+0000","last_change":"2024-04-14T15:21:59.095980+0000","last_active":"2024-04-14T15:24:43.294910+0000","last_peered":"2024-04-14T15:24:43.294910+0000","last_clean":"2024-04-14T15:24:43.294910+0000","last_became_active":"2024-04-14T15:15:44.883694+0000","last_became_peered":"2024-04-14T15:15:44.883694+0000","last_unstale":"2024-04-14T15:24:43.294910+0000","last_undegraded":"2024-04-14T15:24:43.294910+0000","last_fullsized":"2024-04-14T15:24:43.294910+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:59.095899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:21:59.095899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:44.375697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"479'30","reported_seq":1446,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.290359+0000","last_change":"2024-04-14T15:23:30.295880+0000","last_active":"2024-04-14T15:24:43.290359+0000","last_peered":"2024-04-14T15:24:43.290359+0000","last_clean":"2024-04-14T15:24:43.290359+0000","last_became_active":"2024-04-14T15:09:45.037231+0000","last_became_peered":"2024-04-14T15:09:45.037231+0000","last_unstale":"2024-04-14T15:24:43.290359+0000","last_undegraded":"2024-04-14T15:24:43.290359+0000","last_fullsized":"2024-04-14T15:24:43.290359+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'30","last_scrub_stamp":"2024-04-14T15:23:30.295747+0000","last_deep_scrub":"182'18","last_deep_scrub_stamp":"2024-04-14T15:14:10.383906+0000","last_clean_scrub_stamp":"2024-04-14T15:23:30.295747+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:49.622040+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1410,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.278374+0000","last_change":"2024-04-14T15:24:29.563849+0000","last_active":"2024-04-14T15:24:43.278374+0000","last_peered":"2024-04-14T15:24:43.278374+0000","last_clean":"2024-04-14T15:24:43.278374+0000","last_became_active":"2024-04-14T15:09:43.025095+0000","last_became_peered":"2024-04-14T15:09:43.025095+0000","last_unstale":"2024-04-14T15:24:43.278374+0000","last_undegraded":"2024-04-14T15:24:43.278374+0000","last_fullsized":"2024-04-14T15:24:43.278374+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:29.563759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:29.563759+0000","last_clean_scrub_stamp":"2024-04-14T15:24:29.563759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:39.694627+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1416,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.283765+0000","last_change":"2024-04-14T15:23:40.238604+0000","last_active":"2024-04-14T15:24:43.283765+0000","last_peered":"2024-04-14T15:24:43.283765+0000","last_clean":"2024-04-14T15:24:43.283765+0000","last_became_active":"2024-04-14T15:09:43.018509+0000","last_became_peered":"2024-04-14T15:09:43.018509+0000","last_unstale":"2024-04-14T15:24:43.283765+0000","last_undegraded":"2024-04-14T15:24:43.283765+0000","last_fullsized":"2024-04-14T15:24:43.283765+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:40.238527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:23:40.238527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:06.849826+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0018592680000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"355'2","reported_seq":1396,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.287458+0000","last_change":"2024-04-14T15:23:43.059738+0000","last_active":"2024-04-14T15:24:43.287458+0000","last_peered":"2024-04-14T15:24:43.287458+0000","last_clean":"2024-04-14T15:24:43.287458+0000","last_became_active":"2024-04-14T15:09:49.065243+0000","last_became_peered":"2024-04-14T15:09:49.065243+0000","last_unstale":"2024-04-14T15:24:43.287458+0000","last_undegraded":"2024-04-14T15:24:43.287458+0000","last_fullsized":"2024-04-14T15:24:43.287458+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'2","last_scrub_stamp":"2024-04-14T15:23:43.059603+0000","last_deep_scrub":"355'2","last_deep_scrub_stamp":"2024-04-14T15:20:04.842881+0000","last_clean_scrub_stamp":"2024-04-14T15:23:43.059603+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:47.245706+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":1007,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.304771+0000","last_change":"2024-04-14T15:24:28.904419+0000","last_active":"2024-04-14T15:24:43.304771+0000","last_peered":"2024-04-14T15:24:43.304771+0000","last_clean":"2024-04-14T15:24:43.304771+0000","last_became_active":"2024-04-14T15:20:53.000058+0000","last_became_peered":"2024-04-14T15:20:53.000058+0000","last_unstale":"2024-04-14T15:24:43.304771+0000","last_undegraded":"2024-04-14T15:24:43.304771+0000","last_fullsized":"2024-04-14T15:24:43.304771+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:28.904336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:24:28.904336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:29.518613+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1394,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.281454+0000","last_change":"2024-04-14T15:23:05.574296+0000","last_active":"2024-04-14T15:24:43.281454+0000","last_peered":"2024-04-14T15:24:43.281454+0000","last_clean":"2024-04-14T15:24:43.281454+0000","last_became_active":"2024-04-14T15:09:47.047201+0000","last_became_peered":"2024-04-14T15:09:47.047201+0000","last_unstale":"2024-04-14T15:24:43.281454+0000","last_undegraded":"2024-04-14T15:24:43.281454+0000","last_fullsized":"2024-04-14T15:24:43.281454+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:05.574212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:23:05.574212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:44.703693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"528'37","reported_seq":1545,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.287417+0000","last_change":"2024-04-14T15:23:29.977939+0000","last_active":"2024-04-14T15:24:43.287417+0000","last_peered":"2024-04-14T15:24:43.287417+0000","last_clean":"2024-04-14T15:24:43.287417+0000","last_became_active":"2024-04-14T15:20:53.002282+0000","last_became_peered":"2024-04-14T15:20:53.002282+0000","last_unstale":"2024-04-14T15:24:43.287417+0000","last_undegraded":"2024-04-14T15:24:43.287417+0000","last_fullsized":"2024-04-14T15:24:43.287417+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"528'37","last_scrub_stamp":"2024-04-14T15:23:29.977844+0000","last_deep_scrub":"90'24","last_deep_scrub_stamp":"2024-04-14T15:11:25.120692+0000","last_clean_scrub_stamp":"2024-04-14T15:23:29.977844+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"91'130","reported_seq":1746,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.278277+0000","last_change":"2024-04-14T15:22:19.518674+0000","last_active":"2024-04-14T15:24:43.278277+0000","last_peered":"2024-04-14T15:24:43.278277+0000","last_clean":"2024-04-14T15:24:43.278277+0000","last_became_active":"2024-04-14T15:22:14.980657+0000","last_became_peered":"2024-04-14T15:22:14.980657+0000","last_unstale":"2024-04-14T15:24:43.278277+0000","last_undegraded":"2024-04-14T15:24:43.278277+0000","last_fullsized":"2024-04-14T15:24:43.278277+0000","mapping_epoch":530,"log_start":"91'130","ondisk_log_start":"91'130","created":19,"last_epoch_clean":599,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:22:19.518583+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:22:19.518583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:44.703626+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":966,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.297545+0000","last_change":"2024-04-14T15:23:29.691160+0000","last_active":"2024-04-14T15:24:43.297545+0000","last_peered":"2024-04-14T15:24:43.297545+0000","last_clean":"2024-04-14T15:24:43.297545+0000","last_became_active":"2024-04-14T15:15:44.885315+0000","last_became_peered":"2024-04-14T15:15:44.885315+0000","last_unstale":"2024-04-14T15:24:43.297545+0000","last_undegraded":"2024-04-14T15:24:43.297545+0000","last_fullsized":"2024-04-14T15:24:43.297545+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:29.691081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:47.627305+0000","last_clean_scrub_stamp":"2024-04-14T15:23:29.691081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:44.445951+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"69'2","reported_seq":1101,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.304866+0000","last_change":"2024-04-14T15:23:44.863113+0000","last_active":"2024-04-14T15:24:43.304866+0000","last_peered":"2024-04-14T15:24:43.304866+0000","last_clean":"2024-04-14T15:24:43.304866+0000","last_became_active":"2024-04-14T15:20:53.002545+0000","last_became_peered":"2024-04-14T15:20:53.002545+0000","last_unstale":"2024-04-14T15:24:43.304866+0000","last_undegraded":"2024-04-14T15:24:43.304866+0000","last_fullsized":"2024-04-14T15:24:43.304866+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-04-14T15:23:44.862962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:23:44.862962+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:47.817158+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0058046169999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":956,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.307640+0000","last_change":"2024-04-14T15:23:33.934414+0000","last_active":"2024-04-14T15:24:43.307640+0000","last_peered":"2024-04-14T15:24:43.307640+0000","last_clean":"2024-04-14T15:24:43.307640+0000","last_became_active":"2024-04-14T15:15:44.887934+0000","last_became_peered":"2024-04-14T15:15:44.887934+0000","last_unstale":"2024-04-14T15:24:43.307640+0000","last_undegraded":"2024-04-14T15:24:43.307640+0000","last_fullsized":"2024-04-14T15:24:43.307640+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:33.934334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:23:33.934334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:56.348377+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"91'130","reported_seq":1747,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.279134+0000","last_change":"2024-04-14T15:22:17.518773+0000","last_active":"2024-04-14T15:24:43.279134+0000","last_peered":"2024-04-14T15:24:43.279134+0000","last_clean":"2024-04-14T15:24:43.279134+0000","last_became_active":"2024-04-14T15:22:06.969269+0000","last_became_peered":"2024-04-14T15:22:06.969269+0000","last_unstale":"2024-04-14T15:24:43.279134+0000","last_undegraded":"2024-04-14T15:24:43.279134+0000","last_fullsized":"2024-04-14T15:24:43.279134+0000","mapping_epoch":189,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:22:17.518666+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:22:17.518666+0000","last_clean_scrub_stamp":"2024-04-14T15:22:17.518666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:44.703607+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":1007,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.304002+0000","last_change":"2024-04-14T15:23:03.091708+0000","last_active":"2024-04-14T15:24:43.304002+0000","last_peered":"2024-04-14T15:24:43.304002+0000","last_clean":"2024-04-14T15:24:43.304002+0000","last_became_active":"2024-04-14T15:20:52.986308+0000","last_became_peered":"2024-04-14T15:20:52.986308+0000","last_unstale":"2024-04-14T15:24:43.304002+0000","last_undegraded":"2024-04-14T15:24:43.304002+0000","last_fullsized":"2024-04-14T15:24:43.304002+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:03.091585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:23:03.091585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:45.892436+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1399,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.299337+0000","last_change":"2024-04-14T15:23:05.676756+0000","last_active":"2024-04-14T15:24:43.299337+0000","last_peered":"2024-04-14T15:24:43.299337+0000","last_clean":"2024-04-14T15:24:43.299337+0000","last_became_active":"2024-04-14T15:09:47.049457+0000","last_became_peered":"2024-04-14T15:09:47.049457+0000","last_unstale":"2024-04-14T15:24:43.299337+0000","last_undegraded":"2024-04-14T15:24:43.299337+0000","last_fullsized":"2024-04-14T15:24:43.299337+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:05.676677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:23:05.676677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:44.445902+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"479'22","reported_seq":1439,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.294748+0000","last_change":"2024-04-14T15:23:05.309013+0000","last_active":"2024-04-14T15:24:43.294748+0000","last_peered":"2024-04-14T15:24:43.294748+0000","last_clean":"2024-04-14T15:24:43.294748+0000","last_became_active":"2024-04-14T15:09:45.033883+0000","last_became_peered":"2024-04-14T15:09:45.033883+0000","last_unstale":"2024-04-14T15:24:43.294748+0000","last_undegraded":"2024-04-14T15:24:43.294748+0000","last_fullsized":"2024-04-14T15:24:43.294748+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'22","last_scrub_stamp":"2024-04-14T15:23:05.308902+0000","last_deep_scrub":"308'18","last_deep_scrub_stamp":"2024-04-14T15:16:16.054032+0000","last_clean_scrub_stamp":"2024-04-14T15:23:05.308902+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:44.375887+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1416,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.296046+0000","last_change":"2024-04-14T15:24:29.474621+0000","last_active":"2024-04-14T15:24:43.296046+0000","last_peered":"2024-04-14T15:24:43.296046+0000","last_clean":"2024-04-14T15:24:43.296046+0000","last_became_active":"2024-04-14T15:09:43.021188+0000","last_became_peered":"2024-04-14T15:09:43.021188+0000","last_unstale":"2024-04-14T15:24:43.296046+0000","last_undegraded":"2024-04-14T15:24:43.296046+0000","last_fullsized":"2024-04-14T15:24:43.296046+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:29.474544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:24:29.474544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:59.120277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"73'5","reported_seq":2319,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.283480+0000","last_change":"2024-04-14T15:22:09.278270+0000","last_active":"2024-04-14T15:24:43.283480+0000","last_peered":"2024-04-14T15:24:43.283480+0000","last_clean":"2024-04-14T15:24:43.283480+0000","last_became_active":"2024-04-14T15:20:52.975473+0000","last_became_peered":"2024-04-14T15:20:52.975473+0000","last_unstale":"2024-04-14T15:24:43.283480+0000","last_undegraded":"2024-04-14T15:24:43.283480+0000","last_fullsized":"2024-04-14T15:24:43.283480+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-04-14T15:22:09.278172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:22:09.278172+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:44.375744+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00056542099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":892,"num_read_kb":887,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1396,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.281019+0000","last_change":"2024-04-14T15:23:01.536017+0000","last_active":"2024-04-14T15:24:43.281019+0000","last_peered":"2024-04-14T15:24:43.281019+0000","last_clean":"2024-04-14T15:24:43.281019+0000","last_became_active":"2024-04-14T15:20:52.999464+0000","last_became_peered":"2024-04-14T15:20:52.999464+0000","last_unstale":"2024-04-14T15:24:43.281019+0000","last_undegraded":"2024-04-14T15:24:43.281019+0000","last_fullsized":"2024-04-14T15:24:43.281019+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:01.535941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:20.827589+0000","last_clean_scrub_stamp":"2024-04-14T15:23:01.535941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:44.703725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"91'130","reported_seq":1366,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.283545+0000","last_change":"2024-04-14T15:24:23.372384+0000","last_active":"2024-04-14T15:24:43.283545+0000","last_peered":"2024-04-14T15:24:43.283545+0000","last_clean":"2024-04-14T15:24:43.283545+0000","last_became_active":"2024-04-14T15:22:10.975960+0000","last_became_peered":"2024-04-14T15:22:10.975960+0000","last_unstale":"2024-04-14T15:24:43.283545+0000","last_undegraded":"2024-04-14T15:24:43.283545+0000","last_fullsized":"2024-04-14T15:24:43.283545+0000","mapping_epoch":276,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":595,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:24:23.372311+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:24:23.372311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:38.263904+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":1001,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.306658+0000","last_change":"2024-04-14T15:22:53.975763+0000","last_active":"2024-04-14T15:24:43.306658+0000","last_peered":"2024-04-14T15:24:43.306658+0000","last_clean":"2024-04-14T15:24:43.306658+0000","last_became_active":"2024-04-14T15:15:21.883717+0000","last_became_peered":"2024-04-14T15:15:21.883717+0000","last_unstale":"2024-04-14T15:24:43.306658+0000","last_undegraded":"2024-04-14T15:24:43.306658+0000","last_fullsized":"2024-04-14T15:24:43.306658+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:53.975656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:22:53.975656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:45.892451+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1401,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.289800+0000","last_change":"2024-04-14T15:24:26.376324+0000","last_active":"2024-04-14T15:24:43.289800+0000","last_peered":"2024-04-14T15:24:43.289800+0000","last_clean":"2024-04-14T15:24:43.289800+0000","last_became_active":"2024-04-14T15:09:47.040185+0000","last_became_peered":"2024-04-14T15:09:47.040185+0000","last_unstale":"2024-04-14T15:24:43.289800+0000","last_undegraded":"2024-04-14T15:24:43.289800+0000","last_fullsized":"2024-04-14T15:24:43.289800+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:26.376179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:53.731610+0000","last_clean_scrub_stamp":"2024-04-14T15:24:26.376179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:37.687634+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"479'40","reported_seq":1649,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.286445+0000","last_change":"2024-04-14T15:23:31.198816+0000","last_active":"2024-04-14T15:24:43.286445+0000","last_peered":"2024-04-14T15:24:43.286445+0000","last_clean":"2024-04-14T15:24:43.286445+0000","last_became_active":"2024-04-14T15:09:45.034168+0000","last_became_peered":"2024-04-14T15:09:45.034168+0000","last_unstale":"2024-04-14T15:24:43.286445+0000","last_undegraded":"2024-04-14T15:24:43.286445+0000","last_fullsized":"2024-04-14T15:24:43.286445+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'40","last_scrub_stamp":"2024-04-14T15:23:31.198708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:23:31.198708+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":970,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.283551+0000","last_change":"2024-04-14T15:23:30.217883+0000","last_active":"2024-04-14T15:24:43.283551+0000","last_peered":"2024-04-14T15:24:43.283551+0000","last_clean":"2024-04-14T15:24:43.283551+0000","last_became_active":"2024-04-14T15:20:52.979943+0000","last_became_peered":"2024-04-14T15:20:52.979943+0000","last_unstale":"2024-04-14T15:24:43.283551+0000","last_undegraded":"2024-04-14T15:24:43.283551+0000","last_fullsized":"2024-04-14T15:24:43.283551+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:30.217808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:24.066275+0000","last_clean_scrub_stamp":"2024-04-14T15:23:30.217808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:51.962682+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1061,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.307549+0000","last_change":"2024-04-14T15:24:30.949646+0000","last_active":"2024-04-14T15:24:43.307549+0000","last_peered":"2024-04-14T15:24:43.307549+0000","last_clean":"2024-04-14T15:24:43.307549+0000","last_became_active":"2024-04-14T15:15:21.874742+0000","last_became_peered":"2024-04-14T15:15:21.874742+0000","last_unstale":"2024-04-14T15:24:43.307549+0000","last_undegraded":"2024-04-14T15:24:43.307549+0000","last_fullsized":"2024-04-14T15:24:43.307549+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:30.949569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:30.949569+0000","last_clean_scrub_stamp":"2024-04-14T15:24:30.949569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:00.672224+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"91'130","reported_seq":822,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.307401+0000","last_change":"2024-04-14T15:24:23.862421+0000","last_active":"2024-04-14T15:24:43.307401+0000","last_peered":"2024-04-14T15:24:43.307401+0000","last_clean":"2024-04-14T15:24:43.307401+0000","last_became_active":"2024-04-14T15:22:18.993883+0000","last_became_peered":"2024-04-14T15:22:18.993883+0000","last_unstale":"2024-04-14T15:24:43.307401+0000","last_undegraded":"2024-04-14T15:24:43.307401+0000","last_fullsized":"2024-04-14T15:24:43.307401+0000","mapping_epoch":530,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:24:23.862316+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:24:23.862316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:26.896043+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":1310,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.297148+0000","last_change":"2024-04-14T15:22:51.765619+0000","last_active":"2024-04-14T15:24:43.297148+0000","last_peered":"2024-04-14T15:24:43.297148+0000","last_clean":"2024-04-14T15:24:43.297148+0000","last_became_active":"2024-04-14T15:20:52.989272+0000","last_became_peered":"2024-04-14T15:20:52.989272+0000","last_unstale":"2024-04-14T15:24:43.297148+0000","last_undegraded":"2024-04-14T15:24:43.297148+0000","last_fullsized":"2024-04-14T15:24:43.297148+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:51.765499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:22:51.765499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:44.445885+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"528'9","reported_seq":1600,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.285604+0000","last_change":"2024-04-14T15:22:53.253431+0000","last_active":"2024-04-14T15:24:43.285604+0000","last_peered":"2024-04-14T15:24:43.285604+0000","last_clean":"2024-04-14T15:24:43.285604+0000","last_became_active":"2024-04-14T15:20:53.000669+0000","last_became_peered":"2024-04-14T15:20:53.000669+0000","last_unstale":"2024-04-14T15:24:43.285604+0000","last_undegraded":"2024-04-14T15:24:43.285604+0000","last_fullsized":"2024-04-14T15:24:43.285604+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"528'9","last_scrub_stamp":"2024-04-14T15:22:53.253302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:22:53.253302+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:44.501804+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"479'20","reported_seq":408,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.283382+0000","last_change":"2024-04-14T15:22:13.333897+0000","last_active":"2024-04-14T15:24:43.283382+0000","last_peered":"2024-04-14T15:24:43.283382+0000","last_clean":"2024-04-14T15:24:43.283382+0000","last_became_active":"2024-04-14T15:21:16.034027+0000","last_became_peered":"2024-04-14T15:21:16.034027+0000","last_unstale":"2024-04-14T15:24:43.283382+0000","last_undegraded":"2024-04-14T15:24:43.283382+0000","last_fullsized":"2024-04-14T15:24:43.283382+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'20","last_scrub_stamp":"2024-04-14T15:22:13.333807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:22:13.333807+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:44.375714+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1398,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.286354+0000","last_change":"2024-04-14T15:21:28.684960+0000","last_active":"2024-04-14T15:24:43.286354+0000","last_peered":"2024-04-14T15:24:43.286354+0000","last_clean":"2024-04-14T15:24:43.286354+0000","last_became_active":"2024-04-14T15:09:43.022597+0000","last_became_peered":"2024-04-14T15:09:43.022597+0000","last_unstale":"2024-04-14T15:24:43.286354+0000","last_undegraded":"2024-04-14T15:24:43.286354+0000","last_fullsized":"2024-04-14T15:24:43.286354+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:28.684818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:21:28.684818+0000","last_clean_scrub_stamp":"2024-04-14T15:21:28.684818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:44.375649+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1397,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.280224+0000","last_change":"2024-04-14T15:24:25.330276+0000","last_active":"2024-04-14T15:24:43.280224+0000","last_peered":"2024-04-14T15:24:43.280224+0000","last_clean":"2024-04-14T15:24:43.280224+0000","last_became_active":"2024-04-14T15:09:49.066648+0000","last_became_peered":"2024-04-14T15:09:49.066648+0000","last_unstale":"2024-04-14T15:24:43.280224+0000","last_undegraded":"2024-04-14T15:24:43.280224+0000","last_fullsized":"2024-04-14T15:24:43.280224+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:25.330148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:24:25.330148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:36.679635+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":1305,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.290175+0000","last_change":"2024-04-14T15:23:44.358966+0000","last_active":"2024-04-14T15:24:43.290175+0000","last_peered":"2024-04-14T15:24:43.290175+0000","last_clean":"2024-04-14T15:24:43.290175+0000","last_became_active":"2024-04-14T15:12:56.563074+0000","last_became_peered":"2024-04-14T15:12:56.563074+0000","last_unstale":"2024-04-14T15:24:43.290175+0000","last_undegraded":"2024-04-14T15:24:43.290175+0000","last_fullsized":"2024-04-14T15:24:43.290175+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:44.358887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:23:44.358887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:52.408476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1399,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.295632+0000","last_change":"2024-04-14T15:24:22.536720+0000","last_active":"2024-04-14T15:24:43.295632+0000","last_peered":"2024-04-14T15:24:43.295632+0000","last_clean":"2024-04-14T15:24:43.295632+0000","last_became_active":"2024-04-14T15:09:47.049005+0000","last_became_peered":"2024-04-14T15:09:47.049005+0000","last_unstale":"2024-04-14T15:24:43.295632+0000","last_undegraded":"2024-04-14T15:24:43.295632+0000","last_fullsized":"2024-04-14T15:24:43.295632+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:22.536562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:56.915544+0000","last_clean_scrub_stamp":"2024-04-14T15:24:22.536562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:25.774677+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"479'23","reported_seq":1443,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.286099+0000","last_change":"2024-04-14T15:24:28.394342+0000","last_active":"2024-04-14T15:24:43.286099+0000","last_peered":"2024-04-14T15:24:43.286099+0000","last_clean":"2024-04-14T15:24:43.286099+0000","last_became_active":"2024-04-14T15:09:45.032892+0000","last_became_peered":"2024-04-14T15:09:45.032892+0000","last_unstale":"2024-04-14T15:24:43.286099+0000","last_undegraded":"2024-04-14T15:24:43.286099+0000","last_fullsized":"2024-04-14T15:24:43.286099+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'23","last_scrub_stamp":"2024-04-14T15:24:28.394222+0000","last_deep_scrub":"183'17","last_deep_scrub_stamp":"2024-04-14T15:14:11.577574+0000","last_clean_scrub_stamp":"2024-04-14T15:24:28.394222+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:55.253909+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"677'6","reported_seq":1423,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.288113+0000","last_change":"2024-04-14T15:23:38.773659+0000","last_active":"2024-04-14T15:24:43.288113+0000","last_peered":"2024-04-14T15:24:43.288113+0000","last_clean":"2024-04-14T15:24:43.288113+0000","last_became_active":"2024-04-14T15:09:43.020490+0000","last_became_peered":"2024-04-14T15:09:43.020490+0000","last_unstale":"2024-04-14T15:24:43.288113+0000","last_undegraded":"2024-04-14T15:24:43.288113+0000","last_fullsized":"2024-04-14T15:24:43.288113+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"677'6","last_scrub_stamp":"2024-04-14T15:23:38.773539+0000","last_deep_scrub":"677'6","last_deep_scrub_stamp":"2024-04-14T15:23:38.773539+0000","last_clean_scrub_stamp":"2024-04-14T15:23:38.773539+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:04.801057+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":528,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1061,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.306857+0000","last_change":"2024-04-14T15:23:40.854331+0000","last_active":"2024-04-14T15:24:43.306857+0000","last_peered":"2024-04-14T15:24:43.306857+0000","last_clean":"2024-04-14T15:24:43.306857+0000","last_became_active":"2024-04-14T15:15:21.876410+0000","last_became_peered":"2024-04-14T15:15:21.876410+0000","last_unstale":"2024-04-14T15:24:43.306857+0000","last_undegraded":"2024-04-14T15:24:43.306857+0000","last_fullsized":"2024-04-14T15:24:43.306857+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:40.854253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:23:40.854253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:10.566240+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":1312,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.283638+0000","last_change":"2024-04-14T15:24:27.371452+0000","last_active":"2024-04-14T15:24:43.283638+0000","last_peered":"2024-04-14T15:24:43.283638+0000","last_clean":"2024-04-14T15:24:43.283638+0000","last_became_active":"2024-04-14T15:12:56.551177+0000","last_became_peered":"2024-04-14T15:12:56.551177+0000","last_unstale":"2024-04-14T15:24:43.283638+0000","last_undegraded":"2024-04-14T15:24:43.283638+0000","last_fullsized":"2024-04-14T15:24:43.283638+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:27.371370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:22:18.299330+0000","last_clean_scrub_stamp":"2024-04-14T15:24:27.371370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:34.049052+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"479'19","reported_seq":1433,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.290457+0000","last_change":"2024-04-14T15:24:33.412261+0000","last_active":"2024-04-14T15:24:43.290457+0000","last_peered":"2024-04-14T15:24:43.290457+0000","last_clean":"2024-04-14T15:24:43.290457+0000","last_became_active":"2024-04-14T15:09:45.180375+0000","last_became_peered":"2024-04-14T15:09:45.180375+0000","last_unstale":"2024-04-14T15:24:43.290457+0000","last_undegraded":"2024-04-14T15:24:43.290457+0000","last_fullsized":"2024-04-14T15:24:43.290457+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'19","last_scrub_stamp":"2024-04-14T15:24:33.412181+0000","last_deep_scrub":"347'13","last_deep_scrub_stamp":"2024-04-14T15:16:55.694599+0000","last_clean_scrub_stamp":"2024-04-14T15:24:33.412181+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:49.427144+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":1407,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.295545+0000","last_change":"2024-04-14T15:22:16.961934+0000","last_active":"2024-04-14T15:24:43.295545+0000","last_peered":"2024-04-14T15:24:43.295545+0000","last_clean":"2024-04-14T15:24:43.295545+0000","last_became_active":"2024-04-14T15:09:43.016788+0000","last_became_peered":"2024-04-14T15:09:43.016788+0000","last_unstale":"2024-04-14T15:24:43.295545+0000","last_undegraded":"2024-04-14T15:24:43.295545+0000","last_fullsized":"2024-04-14T15:24:43.295545+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:16.961846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:22:16.961846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:44.445851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1080,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.307635+0000","last_change":"2024-04-14T15:24:33.956401+0000","last_active":"2024-04-14T15:24:43.307635+0000","last_peered":"2024-04-14T15:24:43.307635+0000","last_clean":"2024-04-14T15:24:43.307635+0000","last_became_active":"2024-04-14T15:15:21.873580+0000","last_became_peered":"2024-04-14T15:15:21.873580+0000","last_unstale":"2024-04-14T15:24:43.307635+0000","last_undegraded":"2024-04-14T15:24:43.307635+0000","last_fullsized":"2024-04-14T15:24:43.307635+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:33.956328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:24:33.956328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:55.595287+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1395,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.298995+0000","last_change":"2024-04-14T15:22:12.923356+0000","last_active":"2024-04-14T15:24:43.298995+0000","last_peered":"2024-04-14T15:24:43.298995+0000","last_clean":"2024-04-14T15:24:43.298995+0000","last_became_active":"2024-04-14T15:09:49.064296+0000","last_became_peered":"2024-04-14T15:09:49.064296+0000","last_unstale":"2024-04-14T15:24:43.298995+0000","last_undegraded":"2024-04-14T15:24:43.298995+0000","last_fullsized":"2024-04-14T15:24:43.298995+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:12.923270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:22:12.923270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:44.445819+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":1309,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.295387+0000","last_change":"2024-04-14T15:23:32.648604+0000","last_active":"2024-04-14T15:24:43.295387+0000","last_peered":"2024-04-14T15:24:43.295387+0000","last_clean":"2024-04-14T15:24:43.295387+0000","last_became_active":"2024-04-14T15:20:52.977894+0000","last_became_peered":"2024-04-14T15:20:52.977894+0000","last_unstale":"2024-04-14T15:24:43.295387+0000","last_undegraded":"2024-04-14T15:24:43.295387+0000","last_fullsized":"2024-04-14T15:24:43.295387+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:32.648522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:23:32.648522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:53.725757+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.17","version":"479'12","reported_seq":1418,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.307537+0000","last_change":"2024-04-14T15:24:16.798489+0000","last_active":"2024-04-14T15:24:43.307537+0000","last_peered":"2024-04-14T15:24:43.307537+0000","last_clean":"2024-04-14T15:24:43.307537+0000","last_became_active":"2024-04-14T15:09:45.180037+0000","last_became_peered":"2024-04-14T15:09:45.180037+0000","last_unstale":"2024-04-14T15:24:43.307537+0000","last_undegraded":"2024-04-14T15:24:43.307537+0000","last_fullsized":"2024-04-14T15:24:43.307537+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'12","last_scrub_stamp":"2024-04-14T15:24:16.798377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:24:16.798377+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:35.888607+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":442,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.285938+0000","last_change":"2024-04-14T15:23:40.765638+0000","last_active":"2024-04-14T15:24:43.285938+0000","last_peered":"2024-04-14T15:24:43.285938+0000","last_clean":"2024-04-14T15:24:43.285938+0000","last_became_active":"2024-04-14T15:20:52.986974+0000","last_became_peered":"2024-04-14T15:20:52.986974+0000","last_unstale":"2024-04-14T15:24:43.285938+0000","last_undegraded":"2024-04-14T15:24:43.285938+0000","last_fullsized":"2024-04-14T15:24:43.285938+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:40.765537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:59.640565+0000","last_clean_scrub_stamp":"2024-04-14T15:23:40.765537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:06.708615+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":954,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.307442+0000","last_change":"2024-04-14T15:24:18.710723+0000","last_active":"2024-04-14T15:24:43.307442+0000","last_peered":"2024-04-14T15:24:43.307442+0000","last_clean":"2024-04-14T15:24:43.307442+0000","last_became_active":"2024-04-14T15:15:44.889786+0000","last_became_peered":"2024-04-14T15:15:44.889786+0000","last_unstale":"2024-04-14T15:24:43.307442+0000","last_undegraded":"2024-04-14T15:24:43.307442+0000","last_fullsized":"2024-04-14T15:24:43.307442+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:18.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:13.735024+0000","last_clean_scrub_stamp":"2024-04-14T15:24:18.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:37.843986+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1395,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.281350+0000","last_change":"2024-04-14T15:23:29.527743+0000","last_active":"2024-04-14T15:24:43.281350+0000","last_peered":"2024-04-14T15:24:43.281350+0000","last_clean":"2024-04-14T15:24:43.281350+0000","last_became_active":"2024-04-14T15:20:52.985136+0000","last_became_peered":"2024-04-14T15:20:52.985136+0000","last_unstale":"2024-04-14T15:24:43.281350+0000","last_undegraded":"2024-04-14T15:24:43.281350+0000","last_fullsized":"2024-04-14T15:24:43.281350+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:29.527656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:23:29.527656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:53.193549+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":1306,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.286929+0000","last_change":"2024-04-14T15:22:53.207244+0000","last_active":"2024-04-14T15:24:43.286929+0000","last_peered":"2024-04-14T15:24:43.286929+0000","last_clean":"2024-04-14T15:24:43.286929+0000","last_became_active":"2024-04-14T15:12:56.203577+0000","last_became_peered":"2024-04-14T15:12:56.203577+0000","last_unstale":"2024-04-14T15:24:43.286929+0000","last_undegraded":"2024-04-14T15:24:43.286929+0000","last_fullsized":"2024-04-14T15:24:43.286929+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:53.207140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:46.369231+0000","last_clean_scrub_stamp":"2024-04-14T15:22:53.207140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"479'20","reported_seq":1107,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.306830+0000","last_change":"2024-04-14T15:22:01.194496+0000","last_active":"2024-04-14T15:24:43.306830+0000","last_peered":"2024-04-14T15:24:43.306830+0000","last_clean":"2024-04-14T15:24:43.306830+0000","last_became_active":"2024-04-14T15:15:21.877667+0000","last_became_peered":"2024-04-14T15:15:21.877667+0000","last_unstale":"2024-04-14T15:24:43.306830+0000","last_undegraded":"2024-04-14T15:24:43.306830+0000","last_fullsized":"2024-04-14T15:24:43.306830+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'20","last_scrub_stamp":"2024-04-14T15:22:01.194418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:22:01.194418+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:45.892328+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":954,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.305331+0000","last_change":"2024-04-14T15:23:32.000531+0000","last_active":"2024-04-14T15:24:43.305331+0000","last_peered":"2024-04-14T15:24:43.305331+0000","last_clean":"2024-04-14T15:24:43.305331+0000","last_became_active":"2024-04-14T15:15:44.887452+0000","last_became_peered":"2024-04-14T15:15:44.887452+0000","last_unstale":"2024-04-14T15:24:43.305331+0000","last_undegraded":"2024-04-14T15:24:43.305331+0000","last_fullsized":"2024-04-14T15:24:43.305331+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:32.000401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:23:32.000401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:45.892562+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"280'9","reported_seq":1088,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.291388+0000","last_change":"2024-04-14T15:22:01.181123+0000","last_active":"2024-04-14T15:24:43.291388+0000","last_peered":"2024-04-14T15:24:43.291388+0000","last_clean":"2024-04-14T15:24:43.291388+0000","last_became_active":"2024-04-14T15:15:44.882916+0000","last_became_peered":"2024-04-14T15:15:44.882916+0000","last_unstale":"2024-04-14T15:24:43.291388+0000","last_undegraded":"2024-04-14T15:24:43.291388+0000","last_fullsized":"2024-04-14T15:24:43.291388+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"280'9","last_scrub_stamp":"2024-04-14T15:22:01.181035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:22:01.181035+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:44.375682+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1388,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.290265+0000","last_change":"2024-04-14T15:23:29.304889+0000","last_active":"2024-04-14T15:24:43.290265+0000","last_peered":"2024-04-14T15:24:43.290265+0000","last_clean":"2024-04-14T15:24:43.290265+0000","last_became_active":"2024-04-14T15:09:49.066068+0000","last_became_peered":"2024-04-14T15:09:49.066068+0000","last_unstale":"2024-04-14T15:24:43.290265+0000","last_undegraded":"2024-04-14T15:24:43.290265+0000","last_fullsized":"2024-04-14T15:24:43.290265+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:29.304812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:54.724681+0000","last_clean_scrub_stamp":"2024-04-14T15:23:29.304812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":954,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.304617+0000","last_change":"2024-04-14T15:24:26.878660+0000","last_active":"2024-04-14T15:24:43.304617+0000","last_peered":"2024-04-14T15:24:43.304617+0000","last_clean":"2024-04-14T15:24:43.304617+0000","last_became_active":"2024-04-14T15:15:44.889106+0000","last_became_peered":"2024-04-14T15:15:44.889106+0000","last_unstale":"2024-04-14T15:24:43.304617+0000","last_undegraded":"2024-04-14T15:24:43.304617+0000","last_fullsized":"2024-04-14T15:24:43.304617+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:26.878584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:24:26.878584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:35.591554+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"479'19","reported_seq":1263,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.281143+0000","last_change":"2024-04-14T15:23:46.548415+0000","last_active":"2024-04-14T15:24:43.281143+0000","last_peered":"2024-04-14T15:24:43.281143+0000","last_clean":"2024-04-14T15:24:43.281143+0000","last_became_active":"2024-04-14T15:22:25.971104+0000","last_became_peered":"2024-04-14T15:22:25.971104+0000","last_unstale":"2024-04-14T15:24:43.281143+0000","last_undegraded":"2024-04-14T15:24:43.281143+0000","last_fullsized":"2024-04-14T15:24:43.281143+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'19","last_scrub_stamp":"2024-04-14T15:23:46.548255+0000","last_deep_scrub":"183'13","last_deep_scrub_stamp":"2024-04-14T15:14:11.755252+0000","last_clean_scrub_stamp":"2024-04-14T15:23:46.548255+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:55.797274+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1408,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.286185+0000","last_change":"2024-04-14T15:23:28.265599+0000","last_active":"2024-04-14T15:24:43.286185+0000","last_peered":"2024-04-14T15:24:43.286185+0000","last_clean":"2024-04-14T15:24:43.286185+0000","last_became_active":"2024-04-14T15:09:43.019301+0000","last_became_peered":"2024-04-14T15:09:43.019301+0000","last_unstale":"2024-04-14T15:24:43.286185+0000","last_undegraded":"2024-04-14T15:24:43.286185+0000","last_fullsized":"2024-04-14T15:24:43.286185+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:28.265471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:23:28.265471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:44.501900+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"255'10","reported_seq":1225,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.307082+0000","last_change":"2024-04-14T15:23:04.116217+0000","last_active":"2024-04-14T15:24:43.307082+0000","last_peered":"2024-04-14T15:24:43.307082+0000","last_clean":"2024-04-14T15:24:43.307082+0000","last_became_active":"2024-04-14T15:15:21.884468+0000","last_became_peered":"2024-04-14T15:15:21.884468+0000","last_unstale":"2024-04-14T15:24:43.307082+0000","last_undegraded":"2024-04-14T15:24:43.307082+0000","last_fullsized":"2024-04-14T15:24:43.307082+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'10","last_scrub_stamp":"2024-04-14T15:23:04.116126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:23:04.116126+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:45.892515+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1390,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.288367+0000","last_change":"2024-04-14T15:24:23.473852+0000","last_active":"2024-04-14T15:24:43.288367+0000","last_peered":"2024-04-14T15:24:43.288367+0000","last_clean":"2024-04-14T15:24:43.288367+0000","last_became_active":"2024-04-14T15:09:49.065242+0000","last_became_peered":"2024-04-14T15:09:49.065242+0000","last_unstale":"2024-04-14T15:24:43.288367+0000","last_undegraded":"2024-04-14T15:24:43.288367+0000","last_fullsized":"2024-04-14T15:24:43.288367+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:23.473749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:24:23.473749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:46.923229+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":1297,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.281310+0000","last_change":"2024-04-14T15:21:16.642973+0000","last_active":"2024-04-14T15:24:43.281310+0000","last_peered":"2024-04-14T15:24:43.281310+0000","last_clean":"2024-04-14T15:24:43.281310+0000","last_became_active":"2024-04-14T15:12:56.389204+0000","last_became_peered":"2024-04-14T15:12:56.389204+0000","last_unstale":"2024-04-14T15:24:43.281310+0000","last_undegraded":"2024-04-14T15:24:43.281310+0000","last_fullsized":"2024-04-14T15:24:43.281310+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:16.642880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:21:16.642880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:44.703409+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"479'19","reported_seq":866,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.296490+0000","last_change":"2024-04-14T15:24:24.516920+0000","last_active":"2024-04-14T15:24:43.296490+0000","last_peered":"2024-04-14T15:24:43.296490+0000","last_clean":"2024-04-14T15:24:43.296490+0000","last_became_active":"2024-04-14T15:20:52.990622+0000","last_became_peered":"2024-04-14T15:20:52.990622+0000","last_unstale":"2024-04-14T15:24:43.296490+0000","last_undegraded":"2024-04-14T15:24:43.296490+0000","last_fullsized":"2024-04-14T15:24:43.296490+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'19","last_scrub_stamp":"2024-04-14T15:24:24.516820+0000","last_deep_scrub":"344'13","last_deep_scrub_stamp":"2024-04-14T15:16:52.738163+0000","last_clean_scrub_stamp":"2024-04-14T15:24:24.516820+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:41.273155+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":438,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.306859+0000","last_change":"2024-04-14T15:22:55.001625+0000","last_active":"2024-04-14T15:24:43.306859+0000","last_peered":"2024-04-14T15:24:43.306859+0000","last_clean":"2024-04-14T15:24:43.306859+0000","last_became_active":"2024-04-14T15:20:52.996548+0000","last_became_peered":"2024-04-14T15:20:52.996548+0000","last_unstale":"2024-04-14T15:24:43.306859+0000","last_undegraded":"2024-04-14T15:24:43.306859+0000","last_fullsized":"2024-04-14T15:24:43.306859+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:55.001498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:22:55.001498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:45.892465+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1073,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.306815+0000","last_change":"2024-04-14T15:24:31.955026+0000","last_active":"2024-04-14T15:24:43.306815+0000","last_peered":"2024-04-14T15:24:43.306815+0000","last_clean":"2024-04-14T15:24:43.306815+0000","last_became_active":"2024-04-14T15:15:21.882643+0000","last_became_peered":"2024-04-14T15:15:21.882643+0000","last_unstale":"2024-04-14T15:24:43.306815+0000","last_undegraded":"2024-04-14T15:24:43.306815+0000","last_fullsized":"2024-04-14T15:24:43.306815+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:31.954940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:33.485411+0000","last_clean_scrub_stamp":"2024-04-14T15:24:31.954940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:00.985315+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1399,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.280092+0000","last_change":"2024-04-14T15:24:22.395640+0000","last_active":"2024-04-14T15:24:43.280092+0000","last_peered":"2024-04-14T15:24:43.280092+0000","last_clean":"2024-04-14T15:24:43.280092+0000","last_became_active":"2024-04-14T15:09:49.059711+0000","last_became_peered":"2024-04-14T15:09:49.059711+0000","last_unstale":"2024-04-14T15:24:43.280092+0000","last_undegraded":"2024-04-14T15:24:43.280092+0000","last_fullsized":"2024-04-14T15:24:43.280092+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:22.395562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:24:22.395562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:34.649415+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":447,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.280081+0000","last_change":"2024-04-14T15:23:03.603742+0000","last_active":"2024-04-14T15:24:43.280081+0000","last_peered":"2024-04-14T15:24:43.280081+0000","last_clean":"2024-04-14T15:24:43.280081+0000","last_became_active":"2024-04-14T15:22:26.203388+0000","last_became_peered":"2024-04-14T15:22:26.203388+0000","last_unstale":"2024-04-14T15:24:43.280081+0000","last_undegraded":"2024-04-14T15:24:43.280081+0000","last_fullsized":"2024-04-14T15:24:43.280081+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:03.603663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:23:03.603663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:44.703676+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"479'24","reported_seq":1452,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.287098+0000","last_change":"2024-04-14T15:24:15.604196+0000","last_active":"2024-04-14T15:24:43.287098+0000","last_peered":"2024-04-14T15:24:43.287098+0000","last_clean":"2024-04-14T15:24:43.287098+0000","last_became_active":"2024-04-14T15:20:52.989300+0000","last_became_peered":"2024-04-14T15:20:52.989300+0000","last_unstale":"2024-04-14T15:24:43.287098+0000","last_undegraded":"2024-04-14T15:24:43.287098+0000","last_fullsized":"2024-04-14T15:24:43.287098+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'24","last_scrub_stamp":"2024-04-14T15:24:15.604065+0000","last_deep_scrub":"464'16","last_deep_scrub_stamp":"2024-04-14T15:19:17.589999+0000","last_clean_scrub_stamp":"2024-04-14T15:24:15.604065+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:34.969047+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":1413,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.291608+0000","last_change":"2024-04-14T15:24:33.369216+0000","last_active":"2024-04-14T15:24:43.291608+0000","last_peered":"2024-04-14T15:24:43.291608+0000","last_clean":"2024-04-14T15:24:43.291608+0000","last_became_active":"2024-04-14T15:09:43.023583+0000","last_became_peered":"2024-04-14T15:09:43.023583+0000","last_unstale":"2024-04-14T15:24:43.291608+0000","last_undegraded":"2024-04-14T15:24:43.291608+0000","last_fullsized":"2024-04-14T15:24:43.291608+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:33.369110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:24:33.369110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:48.406288+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1397,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.283795+0000","last_change":"2024-04-14T15:24:19.367902+0000","last_active":"2024-04-14T15:24:43.283795+0000","last_peered":"2024-04-14T15:24:43.283795+0000","last_clean":"2024-04-14T15:24:43.283795+0000","last_became_active":"2024-04-14T15:09:47.041683+0000","last_became_peered":"2024-04-14T15:09:47.041683+0000","last_unstale":"2024-04-14T15:24:43.283795+0000","last_undegraded":"2024-04-14T15:24:43.283795+0000","last_fullsized":"2024-04-14T15:24:43.283795+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:19.367758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:08.486217+0000","last_clean_scrub_stamp":"2024-04-14T15:24:19.367758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:40.367460+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1069,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.305035+0000","last_change":"2024-04-14T15:23:28.009795+0000","last_active":"2024-04-14T15:24:43.305035+0000","last_peered":"2024-04-14T15:24:43.305035+0000","last_clean":"2024-04-14T15:24:43.305035+0000","last_became_active":"2024-04-14T15:15:21.889567+0000","last_became_peered":"2024-04-14T15:15:21.889567+0000","last_unstale":"2024-04-14T15:24:43.305035+0000","last_undegraded":"2024-04-14T15:24:43.305035+0000","last_fullsized":"2024-04-14T15:24:43.305035+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:28.009713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:23:28.009713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:47.266664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":1312,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.297454+0000","last_change":"2024-04-14T15:24:31.508385+0000","last_active":"2024-04-14T15:24:43.297454+0000","last_peered":"2024-04-14T15:24:43.297454+0000","last_clean":"2024-04-14T15:24:43.297454+0000","last_became_active":"2024-04-14T15:12:56.387723+0000","last_became_peered":"2024-04-14T15:12:56.387723+0000","last_unstale":"2024-04-14T15:24:43.297454+0000","last_undegraded":"2024-04-14T15:24:43.297454+0000","last_fullsized":"2024-04-14T15:24:43.297454+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:31.508252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:03.685045+0000","last_clean_scrub_stamp":"2024-04-14T15:24:31.508252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:33.390202+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.10","version":"479'8","reported_seq":1424,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.303877+0000","last_change":"2024-04-14T15:24:24.848430+0000","last_active":"2024-04-14T15:24:43.303877+0000","last_peered":"2024-04-14T15:24:43.303877+0000","last_clean":"2024-04-14T15:24:43.303877+0000","last_became_active":"2024-04-14T15:20:53.001300+0000","last_became_peered":"2024-04-14T15:20:53.001300+0000","last_unstale":"2024-04-14T15:24:43.303877+0000","last_undegraded":"2024-04-14T15:24:43.303877+0000","last_fullsized":"2024-04-14T15:24:43.303877+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'8","last_scrub_stamp":"2024-04-14T15:24:24.848331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:24:24.848331+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:48.717429+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":443,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.299311+0000","last_change":"2024-04-14T15:23:43.407976+0000","last_active":"2024-04-14T15:24:43.299311+0000","last_peered":"2024-04-14T15:24:43.299311+0000","last_clean":"2024-04-14T15:24:43.299311+0000","last_became_active":"2024-04-14T15:20:52.979496+0000","last_became_peered":"2024-04-14T15:20:52.979496+0000","last_unstale":"2024-04-14T15:24:43.299311+0000","last_undegraded":"2024-04-14T15:24:43.299311+0000","last_fullsized":"2024-04-14T15:24:43.299311+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:43.407897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:08.712289+0000","last_clean_scrub_stamp":"2024-04-14T15:23:43.407897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:47.213727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":442,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.285782+0000","last_change":"2024-04-14T15:24:30.420853+0000","last_active":"2024-04-14T15:24:43.285782+0000","last_peered":"2024-04-14T15:24:43.285782+0000","last_clean":"2024-04-14T15:24:43.285782+0000","last_became_active":"2024-04-14T15:20:52.990347+0000","last_became_peered":"2024-04-14T15:20:52.990347+0000","last_unstale":"2024-04-14T15:24:43.285782+0000","last_undegraded":"2024-04-14T15:24:43.285782+0000","last_fullsized":"2024-04-14T15:24:43.285782+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:30.420762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:24:30.420762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:37.805454+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1389,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.279771+0000","last_change":"2024-04-14T15:24:18.506706+0000","last_active":"2024-04-14T15:24:43.279771+0000","last_peered":"2024-04-14T15:24:43.279771+0000","last_clean":"2024-04-14T15:24:43.279771+0000","last_became_active":"2024-04-14T15:09:49.061359+0000","last_became_peered":"2024-04-14T15:09:49.061359+0000","last_unstale":"2024-04-14T15:24:43.279771+0000","last_undegraded":"2024-04-14T15:24:43.279771+0000","last_fullsized":"2024-04-14T15:24:43.279771+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:18.506553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:24:18.506553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:40.357723+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":1304,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.280881+0000","last_change":"2024-04-14T15:21:28.711361+0000","last_active":"2024-04-14T15:24:43.280881+0000","last_peered":"2024-04-14T15:24:43.280881+0000","last_clean":"2024-04-14T15:24:43.280881+0000","last_became_active":"2024-04-14T15:20:52.999623+0000","last_became_peered":"2024-04-14T15:20:52.999623+0000","last_unstale":"2024-04-14T15:24:43.280881+0000","last_undegraded":"2024-04-14T15:24:43.280881+0000","last_fullsized":"2024-04-14T15:24:43.280881+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:28.711266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:15:43.680852+0000","last_clean_scrub_stamp":"2024-04-14T15:21:28.711266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:44.703486+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"479'24","reported_seq":1092,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.304773+0000","last_change":"2024-04-14T15:24:29.941597+0000","last_active":"2024-04-14T15:24:43.304773+0000","last_peered":"2024-04-14T15:24:43.304773+0000","last_clean":"2024-04-14T15:24:43.304773+0000","last_became_active":"2024-04-14T15:20:52.996917+0000","last_became_peered":"2024-04-14T15:20:52.996917+0000","last_unstale":"2024-04-14T15:24:43.304773+0000","last_undegraded":"2024-04-14T15:24:43.304773+0000","last_fullsized":"2024-04-14T15:24:43.304773+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'24","last_scrub_stamp":"2024-04-14T15:24:29.941474+0000","last_deep_scrub":"102'16","last_deep_scrub_stamp":"2024-04-14T15:12:35.999326+0000","last_clean_scrub_stamp":"2024-04-14T15:24:29.941474+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:42.781028+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":448,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.297544+0000","last_change":"2024-04-14T15:24:30.484775+0000","last_active":"2024-04-14T15:24:43.297544+0000","last_peered":"2024-04-14T15:24:43.297544+0000","last_clean":"2024-04-14T15:24:43.297544+0000","last_became_active":"2024-04-14T15:20:52.970959+0000","last_became_peered":"2024-04-14T15:20:52.970959+0000","last_unstale":"2024-04-14T15:24:43.297544+0000","last_undegraded":"2024-04-14T15:24:43.297544+0000","last_fullsized":"2024-04-14T15:24:43.297544+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:30.484673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:30.484673+0000","last_clean_scrub_stamp":"2024-04-14T15:24:30.484673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:55.297106+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1399,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.278042+0000","last_change":"2024-04-14T15:24:17.557862+0000","last_active":"2024-04-14T15:24:43.278042+0000","last_peered":"2024-04-14T15:24:43.278042+0000","last_clean":"2024-04-14T15:24:43.278042+0000","last_became_active":"2024-04-14T15:20:52.983919+0000","last_became_peered":"2024-04-14T15:20:52.983919+0000","last_unstale":"2024-04-14T15:24:43.278042+0000","last_undegraded":"2024-04-14T15:24:43.278042+0000","last_fullsized":"2024-04-14T15:24:43.278042+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:17.557766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:24:17.557766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:36.450647+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1083,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.306915+0000","last_change":"2024-04-14T15:24:34.995365+0000","last_active":"2024-04-14T15:24:43.306915+0000","last_peered":"2024-04-14T15:24:43.306915+0000","last_clean":"2024-04-14T15:24:43.306915+0000","last_became_active":"2024-04-14T15:20:52.993918+0000","last_became_peered":"2024-04-14T15:20:52.993918+0000","last_unstale":"2024-04-14T15:24:43.306915+0000","last_undegraded":"2024-04-14T15:24:43.306915+0000","last_fullsized":"2024-04-14T15:24:43.306915+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:34.995256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:24:34.995256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:38.220485+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":1304,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.295888+0000","last_change":"2024-04-14T15:24:26.504869+0000","last_active":"2024-04-14T15:24:43.295888+0000","last_peered":"2024-04-14T15:24:43.295888+0000","last_clean":"2024-04-14T15:24:43.295888+0000","last_became_active":"2024-04-14T15:12:56.335230+0000","last_became_peered":"2024-04-14T15:12:56.335230+0000","last_unstale":"2024-04-14T15:24:43.295888+0000","last_undegraded":"2024-04-14T15:24:43.295888+0000","last_fullsized":"2024-04-14T15:24:43.295888+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:26.504790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:24:26.504790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:44.382683+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"479'20","reported_seq":1427,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.290663+0000","last_change":"2024-04-14T15:22:58.400820+0000","last_active":"2024-04-14T15:24:43.290663+0000","last_peered":"2024-04-14T15:24:43.290663+0000","last_clean":"2024-04-14T15:24:43.290663+0000","last_became_active":"2024-04-14T15:09:45.181456+0000","last_became_peered":"2024-04-14T15:09:45.181456+0000","last_unstale":"2024-04-14T15:24:43.290663+0000","last_undegraded":"2024-04-14T15:24:43.290663+0000","last_fullsized":"2024-04-14T15:24:43.290663+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'20","last_scrub_stamp":"2024-04-14T15:22:58.400727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:22:58.400727+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:44.501839+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":960,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.285135+0000","last_change":"2024-04-14T15:24:31.421036+0000","last_active":"2024-04-14T15:24:43.285135+0000","last_peered":"2024-04-14T15:24:43.285135+0000","last_clean":"2024-04-14T15:24:43.285135+0000","last_became_active":"2024-04-14T15:15:44.878336+0000","last_became_peered":"2024-04-14T15:15:44.878336+0000","last_unstale":"2024-04-14T15:24:43.285135+0000","last_undegraded":"2024-04-14T15:24:43.285135+0000","last_fullsized":"2024-04-14T15:24:43.285135+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:31.420910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:24:31.420910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:43.864027+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"528'9","reported_seq":513,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.288451+0000","last_change":"2024-04-14T15:20:53.001282+0000","last_active":"2024-04-14T15:24:43.288451+0000","last_peered":"2024-04-14T15:24:43.288451+0000","last_clean":"2024-04-14T15:24:43.288451+0000","last_became_active":"2024-04-14T15:20:52.982112+0000","last_became_peered":"2024-04-14T15:20:52.982112+0000","last_unstale":"2024-04-14T15:24:43.288451+0000","last_undegraded":"2024-04-14T15:24:43.288451+0000","last_fullsized":"2024-04-14T15:24:43.288451+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:20:45.809489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:20:45.809489+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"72'1","reported_seq":1397,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.281580+0000","last_change":"2024-04-14T15:23:32.511432+0000","last_active":"2024-04-14T15:24:43.281580+0000","last_peered":"2024-04-14T15:24:43.281580+0000","last_clean":"2024-04-14T15:24:43.281580+0000","last_became_active":"2024-04-14T15:09:49.062051+0000","last_became_peered":"2024-04-14T15:09:49.062051+0000","last_unstale":"2024-04-14T15:24:43.281580+0000","last_undegraded":"2024-04-14T15:24:43.281580+0000","last_fullsized":"2024-04-14T15:24:43.281580+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-04-14T15:23:32.511322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:23:32.511322+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:44.703764+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":1311,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.280346+0000","last_change":"2024-04-14T15:24:29.418267+0000","last_active":"2024-04-14T15:24:43.280346+0000","last_peered":"2024-04-14T15:24:43.280346+0000","last_clean":"2024-04-14T15:24:43.280346+0000","last_became_active":"2024-04-14T15:20:52.986657+0000","last_became_peered":"2024-04-14T15:20:52.986657+0000","last_unstale":"2024-04-14T15:24:43.280346+0000","last_undegraded":"2024-04-14T15:24:43.280346+0000","last_fullsized":"2024-04-14T15:24:43.280346+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:29.418183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:29.418183+0000","last_clean_scrub_stamp":"2024-04-14T15:24:29.418183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:48.402168+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"519'24","reported_seq":1455,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.297882+0000","last_change":"2024-04-14T15:23:40.504101+0000","last_active":"2024-04-14T15:24:43.297882+0000","last_peered":"2024-04-14T15:24:43.297882+0000","last_clean":"2024-04-14T15:24:43.297882+0000","last_became_active":"2024-04-14T15:20:52.987333+0000","last_became_peered":"2024-04-14T15:20:52.987333+0000","last_unstale":"2024-04-14T15:24:43.297882+0000","last_undegraded":"2024-04-14T15:24:43.297882+0000","last_fullsized":"2024-04-14T15:24:43.297882+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"519'24","last_scrub_stamp":"2024-04-14T15:23:40.504002+0000","last_deep_scrub":"519'24","last_deep_scrub_stamp":"2024-04-14T15:20:44.763417+0000","last_clean_scrub_stamp":"2024-04-14T15:23:40.504002+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:57.259947+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.19","version":"684'4","reported_seq":1101,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.304324+0000","last_change":"2024-04-14T15:23:46.931649+0000","last_active":"2024-04-14T15:24:43.304324+0000","last_peered":"2024-04-14T15:24:43.304324+0000","last_clean":"2024-04-14T15:24:43.304324+0000","last_became_active":"2024-04-14T15:20:52.995628+0000","last_became_peered":"2024-04-14T15:20:52.995628+0000","last_unstale":"2024-04-14T15:24:43.304324+0000","last_undegraded":"2024-04-14T15:24:43.304324+0000","last_fullsized":"2024-04-14T15:24:43.304324+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"684'4","last_scrub_stamp":"2024-04-14T15:23:46.931560+0000","last_deep_scrub":"684'4","last_deep_scrub_stamp":"2024-04-14T15:23:46.931560+0000","last_clean_scrub_stamp":"2024-04-14T15:23:46.931560+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:51.788030+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":957,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1401,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.290757+0000","last_change":"2024-04-14T15:24:30.397659+0000","last_active":"2024-04-14T15:24:43.290757+0000","last_peered":"2024-04-14T15:24:43.290757+0000","last_clean":"2024-04-14T15:24:43.290757+0000","last_became_active":"2024-04-14T15:09:47.045730+0000","last_became_peered":"2024-04-14T15:09:47.045730+0000","last_unstale":"2024-04-14T15:24:43.290757+0000","last_undegraded":"2024-04-14T15:24:43.290757+0000","last_fullsized":"2024-04-14T15:24:43.290757+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:30.397585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:24:30.397585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:55.143985+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1392,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.291231+0000","last_change":"2024-04-14T15:24:30.343193+0000","last_active":"2024-04-14T15:24:43.291231+0000","last_peered":"2024-04-14T15:24:43.291231+0000","last_clean":"2024-04-14T15:24:43.291231+0000","last_became_active":"2024-04-14T15:09:49.057545+0000","last_became_peered":"2024-04-14T15:09:49.057545+0000","last_unstale":"2024-04-14T15:24:43.291231+0000","last_undegraded":"2024-04-14T15:24:43.291231+0000","last_fullsized":"2024-04-14T15:24:43.291231+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:30.343106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:30.343106+0000","last_clean_scrub_stamp":"2024-04-14T15:24:30.343106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:41.873021+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":952,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.286325+0000","last_change":"2024-04-14T15:21:58.069606+0000","last_active":"2024-04-14T15:24:43.286325+0000","last_peered":"2024-04-14T15:24:43.286325+0000","last_clean":"2024-04-14T15:24:43.286325+0000","last_became_active":"2024-04-14T15:15:44.882604+0000","last_became_peered":"2024-04-14T15:15:44.882604+0000","last_unstale":"2024-04-14T15:24:43.286325+0000","last_undegraded":"2024-04-14T15:24:43.286325+0000","last_fullsized":"2024-04-14T15:24:43.286325+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:58.069509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:21:58.069509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"479'27","reported_seq":1446,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.299219+0000","last_change":"2024-04-14T15:24:21.500258+0000","last_active":"2024-04-14T15:24:43.299219+0000","last_peered":"2024-04-14T15:24:43.299219+0000","last_clean":"2024-04-14T15:24:43.299219+0000","last_became_active":"2024-04-14T15:09:45.032887+0000","last_became_peered":"2024-04-14T15:09:45.032887+0000","last_unstale":"2024-04-14T15:24:43.299219+0000","last_undegraded":"2024-04-14T15:24:43.299219+0000","last_fullsized":"2024-04-14T15:24:43.299219+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'27","last_scrub_stamp":"2024-04-14T15:24:21.500156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:24:21.500156+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:30.530072+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":434,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.286687+0000","last_change":"2024-04-14T15:20:52.990991+0000","last_active":"2024-04-14T15:24:43.286687+0000","last_peered":"2024-04-14T15:24:43.286687+0000","last_clean":"2024-04-14T15:24:43.286687+0000","last_became_active":"2024-04-14T15:20:52.990634+0000","last_became_peered":"2024-04-14T15:20:52.990634+0000","last_unstale":"2024-04-14T15:24:43.286687+0000","last_undegraded":"2024-04-14T15:24:43.286687+0000","last_fullsized":"2024-04-14T15:24:43.286687+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:42.754721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:42.754721+0000","last_clean_scrub_stamp":"2024-04-14T15:20:42.754721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1396,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.291495+0000","last_change":"2024-04-14T15:22:18.460314+0000","last_active":"2024-04-14T15:24:43.291495+0000","last_peered":"2024-04-14T15:24:43.291495+0000","last_clean":"2024-04-14T15:24:43.291495+0000","last_became_active":"2024-04-14T15:09:47.042733+0000","last_became_peered":"2024-04-14T15:09:47.042733+0000","last_unstale":"2024-04-14T15:24:43.291495+0000","last_undegraded":"2024-04-14T15:24:43.291495+0000","last_fullsized":"2024-04-14T15:24:43.291495+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:18.460230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:22:18.460230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:44.375728+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1397,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.287877+0000","last_change":"2024-04-14T15:22:08.071314+0000","last_active":"2024-04-14T15:24:43.287877+0000","last_peered":"2024-04-14T15:24:43.287877+0000","last_clean":"2024-04-14T15:24:43.287877+0000","last_became_active":"2024-04-14T15:20:52.998253+0000","last_became_peered":"2024-04-14T15:20:52.998253+0000","last_unstale":"2024-04-14T15:24:43.287877+0000","last_undegraded":"2024-04-14T15:24:43.287877+0000","last_fullsized":"2024-04-14T15:24:43.287877+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:08.071208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:54.453174+0000","last_clean_scrub_stamp":"2024-04-14T15:22:08.071208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":1298,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.279297+0000","last_change":"2024-04-14T15:21:17.675180+0000","last_active":"2024-04-14T15:24:43.279297+0000","last_peered":"2024-04-14T15:24:43.279297+0000","last_clean":"2024-04-14T15:24:43.279297+0000","last_became_active":"2024-04-14T15:12:56.392352+0000","last_became_peered":"2024-04-14T15:12:56.392352+0000","last_unstale":"2024-04-14T15:24:43.279297+0000","last_undegraded":"2024-04-14T15:24:43.279297+0000","last_fullsized":"2024-04-14T15:24:43.279297+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:17.675093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:21:17.675093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:44.703520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"479'18","reported_seq":1108,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.307083+0000","last_change":"2024-04-14T15:22:06.042903+0000","last_active":"2024-04-14T15:24:43.307083+0000","last_peered":"2024-04-14T15:24:43.307083+0000","last_clean":"2024-04-14T15:24:43.307083+0000","last_became_active":"2024-04-14T15:15:21.877838+0000","last_became_peered":"2024-04-14T15:15:21.877838+0000","last_unstale":"2024-04-14T15:24:43.307083+0000","last_undegraded":"2024-04-14T15:24:43.307083+0000","last_fullsized":"2024-04-14T15:24:43.307083+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'18","last_scrub_stamp":"2024-04-14T15:22:06.042808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:22:06.042808+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:45.892350+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":1086,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.306799+0000","last_change":"2024-04-14T15:22:52.950407+0000","last_active":"2024-04-14T15:24:43.306799+0000","last_peered":"2024-04-14T15:24:43.306799+0000","last_clean":"2024-04-14T15:24:43.306799+0000","last_became_active":"2024-04-14T15:20:53.001534+0000","last_became_peered":"2024-04-14T15:20:53.001534+0000","last_unstale":"2024-04-14T15:24:43.306799+0000","last_undegraded":"2024-04-14T15:24:43.306799+0000","last_fullsized":"2024-04-14T15:24:43.306799+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:52.950279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:22:52.950279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:45.892497+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1393,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.286568+0000","last_change":"2024-04-14T15:20:39.717630+0000","last_active":"2024-04-14T15:24:43.286568+0000","last_peered":"2024-04-14T15:24:43.286568+0000","last_clean":"2024-04-14T15:24:43.286568+0000","last_became_active":"2024-04-14T15:09:47.048045+0000","last_became_peered":"2024-04-14T15:09:47.048045+0000","last_unstale":"2024-04-14T15:24:43.286568+0000","last_undegraded":"2024-04-14T15:24:43.286568+0000","last_fullsized":"2024-04-14T15:24:43.286568+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:39.717522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:20:39.717522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1076,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.307529+0000","last_change":"2024-04-14T15:22:20.946609+0000","last_active":"2024-04-14T15:24:43.307529+0000","last_peered":"2024-04-14T15:24:43.307529+0000","last_clean":"2024-04-14T15:24:43.307529+0000","last_became_active":"2024-04-14T15:20:52.998775+0000","last_became_peered":"2024-04-14T15:20:52.998775+0000","last_unstale":"2024-04-14T15:24:43.307529+0000","last_undegraded":"2024-04-14T15:24:43.307529+0000","last_fullsized":"2024-04-14T15:24:43.307529+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:20.946524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:22:20.946524+0000","last_clean_scrub_stamp":"2024-04-14T15:22:20.946524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:45.892367+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":1303,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.297239+0000","last_change":"2024-04-14T15:22:18.954117+0000","last_active":"2024-04-14T15:24:43.297239+0000","last_peered":"2024-04-14T15:24:43.297239+0000","last_clean":"2024-04-14T15:24:43.297239+0000","last_became_active":"2024-04-14T15:12:56.375278+0000","last_became_peered":"2024-04-14T15:12:56.375278+0000","last_unstale":"2024-04-14T15:24:43.297239+0000","last_undegraded":"2024-04-14T15:24:43.297239+0000","last_fullsized":"2024-04-14T15:24:43.297239+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:18.954041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:01.840924+0000","last_clean_scrub_stamp":"2024-04-14T15:22:18.954041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:44.445869+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"479'19","reported_seq":1427,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.286618+0000","last_change":"2024-04-14T15:21:22.085022+0000","last_active":"2024-04-14T15:24:43.286618+0000","last_peered":"2024-04-14T15:24:43.286618+0000","last_clean":"2024-04-14T15:24:43.286618+0000","last_became_active":"2024-04-14T15:09:45.030093+0000","last_became_peered":"2024-04-14T15:09:45.030093+0000","last_unstale":"2024-04-14T15:24:43.286618+0000","last_undegraded":"2024-04-14T15:24:43.286618+0000","last_fullsized":"2024-04-14T15:24:43.286618+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'19","last_scrub_stamp":"2024-04-14T15:21:22.084927+0000","last_deep_scrub":"276'13","last_deep_scrub_stamp":"2024-04-14T15:15:43.969775+0000","last_clean_scrub_stamp":"2024-04-14T15:21:22.084927+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":1107,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.279875+0000","last_change":"2024-04-14T15:24:26.465423+0000","last_active":"2024-04-14T15:24:43.279875+0000","last_peered":"2024-04-14T15:24:43.279875+0000","last_clean":"2024-04-14T15:24:43.279875+0000","last_became_active":"2024-04-14T15:14:30.857727+0000","last_became_peered":"2024-04-14T15:14:30.857727+0000","last_unstale":"2024-04-14T15:24:43.279875+0000","last_undegraded":"2024-04-14T15:24:43.279875+0000","last_fullsized":"2024-04-14T15:24:43.279875+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:26.465343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:00.650062+0000","last_clean_scrub_stamp":"2024-04-14T15:24:26.465343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:42.640865+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":966,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.286374+0000","last_change":"2024-04-14T15:24:35.503356+0000","last_active":"2024-04-14T15:24:43.286374+0000","last_peered":"2024-04-14T15:24:43.286374+0000","last_clean":"2024-04-14T15:24:43.286374+0000","last_became_active":"2024-04-14T15:15:44.883121+0000","last_became_peered":"2024-04-14T15:15:44.883121+0000","last_unstale":"2024-04-14T15:24:43.286374+0000","last_undegraded":"2024-04-14T15:24:43.286374+0000","last_fullsized":"2024-04-14T15:24:43.286374+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:35.503279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:24:35.503279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:44.271019+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1396,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.287834+0000","last_change":"2024-04-14T15:23:01.449252+0000","last_active":"2024-04-14T15:24:43.287834+0000","last_peered":"2024-04-14T15:24:43.287834+0000","last_clean":"2024-04-14T15:24:43.287834+0000","last_became_active":"2024-04-14T15:09:49.061756+0000","last_became_peered":"2024-04-14T15:09:49.061756+0000","last_unstale":"2024-04-14T15:24:43.287834+0000","last_undegraded":"2024-04-14T15:24:43.287834+0000","last_fullsized":"2024-04-14T15:24:43.287834+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:01.449169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:23:01.449169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:44.501871+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":442,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.286325+0000","last_change":"2024-04-14T15:24:24.522250+0000","last_active":"2024-04-14T15:24:43.286325+0000","last_peered":"2024-04-14T15:24:43.286325+0000","last_clean":"2024-04-14T15:24:43.286325+0000","last_became_active":"2024-04-14T15:20:52.990838+0000","last_became_peered":"2024-04-14T15:20:52.990838+0000","last_unstale":"2024-04-14T15:24:43.286325+0000","last_undegraded":"2024-04-14T15:24:43.286325+0000","last_fullsized":"2024-04-14T15:24:43.286325+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:24.522164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:24:24.522164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:27.651945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"479'9","reported_seq":1412,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.287376+0000","last_change":"2024-04-14T15:22:54.203852+0000","last_active":"2024-04-14T15:24:43.287376+0000","last_peered":"2024-04-14T15:24:43.287376+0000","last_clean":"2024-04-14T15:24:43.287376+0000","last_became_active":"2024-04-14T15:09:45.028667+0000","last_became_peered":"2024-04-14T15:09:45.028667+0000","last_unstale":"2024-04-14T15:24:43.287376+0000","last_undegraded":"2024-04-14T15:24:43.287376+0000","last_fullsized":"2024-04-14T15:24:43.287376+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'9","last_scrub_stamp":"2024-04-14T15:22:54.203720+0000","last_deep_scrub":"87'7","last_deep_scrub_stamp":"2024-04-14T15:11:10.510785+0000","last_clean_scrub_stamp":"2024-04-14T15:22:54.203720+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":882,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.296660+0000","last_change":"2024-04-14T15:22:15.944094+0000","last_active":"2024-04-14T15:24:43.296660+0000","last_peered":"2024-04-14T15:24:43.296660+0000","last_clean":"2024-04-14T15:24:43.296660+0000","last_became_active":"2024-04-14T15:20:52.989307+0000","last_became_peered":"2024-04-14T15:20:52.989307+0000","last_unstale":"2024-04-14T15:24:43.296660+0000","last_undegraded":"2024-04-14T15:24:43.296660+0000","last_fullsized":"2024-04-14T15:24:43.296660+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:15.944005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:22:15.944005+0000","last_clean_scrub_stamp":"2024-04-14T15:22:15.944005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:44.445781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1392,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.279604+0000","last_change":"2024-04-14T15:22:05.564704+0000","last_active":"2024-04-14T15:24:43.279604+0000","last_peered":"2024-04-14T15:24:43.279604+0000","last_clean":"2024-04-14T15:24:43.279604+0000","last_became_active":"2024-04-14T15:09:47.046517+0000","last_became_peered":"2024-04-14T15:09:47.046517+0000","last_unstale":"2024-04-14T15:24:43.279604+0000","last_undegraded":"2024-04-14T15:24:43.279604+0000","last_fullsized":"2024-04-14T15:24:43.279604+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:05.564624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:22:05.564624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:44.703643+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":956,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.283097+0000","last_change":"2024-04-14T15:22:08.325886+0000","last_active":"2024-04-14T15:24:43.283097+0000","last_peered":"2024-04-14T15:24:43.283097+0000","last_clean":"2024-04-14T15:24:43.283097+0000","last_became_active":"2024-04-14T15:15:44.887227+0000","last_became_peered":"2024-04-14T15:15:44.887227+0000","last_unstale":"2024-04-14T15:24:43.283097+0000","last_undegraded":"2024-04-14T15:24:43.283097+0000","last_fullsized":"2024-04-14T15:24:43.283097+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:08.325808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:56.268086+0000","last_clean_scrub_stamp":"2024-04-14T15:22:08.325808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:44.375759+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":1303,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.280835+0000","last_change":"2024-04-14T15:21:21.668496+0000","last_active":"2024-04-14T15:24:43.280835+0000","last_peered":"2024-04-14T15:24:43.280835+0000","last_clean":"2024-04-14T15:24:43.280835+0000","last_became_active":"2024-04-14T15:20:52.984387+0000","last_became_peered":"2024-04-14T15:20:52.984387+0000","last_unstale":"2024-04-14T15:24:43.280835+0000","last_undegraded":"2024-04-14T15:24:43.280835+0000","last_fullsized":"2024-04-14T15:24:43.280835+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:21.668407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:34.558312+0000","last_clean_scrub_stamp":"2024-04-14T15:21:21.668407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:44.703430+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.18","version":"479'15","reported_seq":1439,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.285024+0000","last_change":"2024-04-14T15:22:57.315447+0000","last_active":"2024-04-14T15:24:43.285024+0000","last_peered":"2024-04-14T15:24:43.285024+0000","last_clean":"2024-04-14T15:24:43.285024+0000","last_became_active":"2024-04-14T15:20:52.985210+0000","last_became_peered":"2024-04-14T15:20:52.985210+0000","last_unstale":"2024-04-14T15:24:43.285024+0000","last_undegraded":"2024-04-14T15:24:43.285024+0000","last_fullsized":"2024-04-14T15:24:43.285024+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'15","last_scrub_stamp":"2024-04-14T15:22:57.315355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:22:57.315355+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":1082,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.307467+0000","last_change":"2024-04-14T15:24:21.791334+0000","last_active":"2024-04-14T15:24:43.307467+0000","last_peered":"2024-04-14T15:24:43.307467+0000","last_clean":"2024-04-14T15:24:43.307467+0000","last_became_active":"2024-04-14T15:15:21.891926+0000","last_became_peered":"2024-04-14T15:15:21.891926+0000","last_unstale":"2024-04-14T15:24:43.307467+0000","last_undegraded":"2024-04-14T15:24:43.307467+0000","last_fullsized":"2024-04-14T15:24:43.307467+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:21.791233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:24:21.791233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:34.842188+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1397,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.282675+0000","last_change":"2024-04-14T15:24:22.302587+0000","last_active":"2024-04-14T15:24:43.282675+0000","last_peered":"2024-04-14T15:24:43.282675+0000","last_clean":"2024-04-14T15:24:43.282675+0000","last_became_active":"2024-04-14T15:09:47.041486+0000","last_became_peered":"2024-04-14T15:09:47.041486+0000","last_unstale":"2024-04-14T15:24:43.282675+0000","last_undegraded":"2024-04-14T15:24:43.282675+0000","last_fullsized":"2024-04-14T15:24:43.282675+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:22.302513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:24:22.302513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:40.486676+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1381,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.286433+0000","last_change":"2024-04-14T15:20:36.761316+0000","last_active":"2024-04-14T15:24:43.286433+0000","last_peered":"2024-04-14T15:24:43.286433+0000","last_clean":"2024-04-14T15:24:43.286433+0000","last_became_active":"2024-04-14T15:09:49.058904+0000","last_became_peered":"2024-04-14T15:09:49.058904+0000","last_unstale":"2024-04-14T15:24:43.286433+0000","last_undegraded":"2024-04-14T15:24:43.286433+0000","last_fullsized":"2024-04-14T15:24:43.286433+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:20:36.761246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:20:36.761246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":951,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.280692+0000","last_change":"2024-04-14T15:21:14.673688+0000","last_active":"2024-04-14T15:24:43.280692+0000","last_peered":"2024-04-14T15:24:43.280692+0000","last_clean":"2024-04-14T15:24:43.280692+0000","last_became_active":"2024-04-14T15:15:44.889605+0000","last_became_peered":"2024-04-14T15:15:44.889605+0000","last_unstale":"2024-04-14T15:24:43.280692+0000","last_undegraded":"2024-04-14T15:24:43.280692+0000","last_fullsized":"2024-04-14T15:24:43.280692+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:14.673591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:21:14.673591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:44.703448+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"479'34","reported_seq":1126,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.304826+0000","last_change":"2024-04-14T15:23:00.061412+0000","last_active":"2024-04-14T15:24:43.304826+0000","last_peered":"2024-04-14T15:24:43.304826+0000","last_clean":"2024-04-14T15:24:43.304826+0000","last_became_active":"2024-04-14T15:15:21.884310+0000","last_became_peered":"2024-04-14T15:15:21.884310+0000","last_unstale":"2024-04-14T15:24:43.304826+0000","last_undegraded":"2024-04-14T15:24:43.304826+0000","last_fullsized":"2024-04-14T15:24:43.304826+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'34","last_scrub_stamp":"2024-04-14T15:23:00.061294+0000","last_deep_scrub":"306'22","last_deep_scrub_stamp":"2024-04-14T15:16:13.898106+0000","last_clean_scrub_stamp":"2024-04-14T15:23:00.061294+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:24:45.892401+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":1409,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.291843+0000","last_change":"2024-04-14T15:24:24.353115+0000","last_active":"2024-04-14T15:24:43.291843+0000","last_peered":"2024-04-14T15:24:43.291843+0000","last_clean":"2024-04-14T15:24:43.291843+0000","last_became_active":"2024-04-14T15:09:43.017351+0000","last_became_peered":"2024-04-14T15:09:43.017351+0000","last_unstale":"2024-04-14T15:24:43.291843+0000","last_undegraded":"2024-04-14T15:24:43.291843+0000","last_fullsized":"2024-04-14T15:24:43.291843+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:24.352978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:24:24.352978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:43.866848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1400,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.285231+0000","last_change":"2024-04-14T15:24:32.403277+0000","last_active":"2024-04-14T15:24:43.285231+0000","last_peered":"2024-04-14T15:24:43.285231+0000","last_clean":"2024-04-14T15:24:43.285231+0000","last_became_active":"2024-04-14T15:20:52.974360+0000","last_became_peered":"2024-04-14T15:20:52.974360+0000","last_unstale":"2024-04-14T15:24:43.285231+0000","last_undegraded":"2024-04-14T15:24:43.285231+0000","last_fullsized":"2024-04-14T15:24:43.285231+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:32.403077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:32.403077+0000","last_clean_scrub_stamp":"2024-04-14T15:24:32.403077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:44.775030+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1397,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-14T15:24:43.306601+0000","last_change":"2024-04-14T15:24:27.874590+0000","last_active":"2024-04-14T15:24:43.306601+0000","last_peered":"2024-04-14T15:24:43.306601+0000","last_clean":"2024-04-14T15:24:43.306601+0000","last_became_active":"2024-04-14T15:20:52.998938+0000","last_became_peered":"2024-04-14T15:20:52.998938+0000","last_unstale":"2024-04-14T15:24:43.306601+0000","last_undegraded":"2024-04-14T15:24:43.306601+0000","last_fullsized":"2024-04-14T15:24:43.306601+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:27.874498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:27.874498+0000","last_clean_scrub_stamp":"2024-04-14T15:24:27.874498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:29.705955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":333,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":7},{"poolid":323,"num_pg":8,"stat_sum":{"num_bytes":71,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":12,"num_read_kb":1,"num_write":26,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":147456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":24,"acting":24,"num_store_stats":7},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":5,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1003,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":918048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2109440,"data_stored":2098784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":892,"num_read_kb":887,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1121,"num_read_kb":878,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":1694,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":785,"ondisk_log_size":785,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103354,"num_pgs":92,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":99328,"kb_used_data":5684,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":93645824,"statfs":{"total":95995035648,"available":95893323776,"internally_reserved":0,"allocated":5820416,"data_stored":2978858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":95877578},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":210453397693,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575063,"num_pgs":101,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":117088,"kb_used_data":5396,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93628064,"statfs":{"total":95995035648,"available":95875137536,"internally_reserved":0,"allocated":5525504,"data_stored":2655578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738588,"num_pgs":84,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":90716,"kb_used_data":5384,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":93654436,"statfs":{"total":95995035648,"available":95902142464,"internally_reserved":0,"allocated":5513216,"data_stored":2644407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":87357895},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411539,"num_pgs":89,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":109012,"kb_used_data":5636,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93636140,"statfs":{"total":95995035648,"available":95883407360,"internally_reserved":0,"allocated":5771264,"data_stored":2922755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182605,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":69112,"kb_used_data":5988,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93676040,"statfs":{"total":95995035648,"available":95924264960,"internally_reserved":0,"allocated":6131712,"data_stored":3246538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986375,"num_pgs":87,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":121236,"kb_used_data":5380,"kb_used_omap":1,"kb_used_meta":115838,"kb_avail":93623916,"statfs":{"total":95995035648,"available":95870889984,"internally_reserved":0,"allocated":5509120,"data_stored":2651203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":118618565},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659332,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":117624,"kb_used_data":5928,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93627528,"statfs":{"total":95995035648,"available":95874588672,"internally_reserved":0,"allocated":6070272,"data_stored":3227780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":12288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":20480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":20480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":333,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":333,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":333,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":333,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":333,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":333,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":333,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-14T15:24:46.748 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-04-14T15:24:46.748 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd pool set .mgr pg_num 21 2024-04-14T15:24:46.800 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:24:46.845 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:46.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:46.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:24:46.947 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:24:46.974 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:46.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:46.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:24:47.076 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:24:47.102 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:47.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:47.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:24:47.204 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:24:47.228 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:47.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:47.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:24:47.330 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:24:47.353 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:47.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:47.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:24:47.455 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:24:47.475 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-14T15:24:47.482 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:47.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:47.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:24:47.584 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:24:47.612 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:47.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:47.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:24:47.714 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:24:47.739 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:47.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:47.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:24:47.841 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:24:47.865 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:47.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:47.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:24:47.966 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:24:47.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:47.992 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:47.992 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:24:48.006 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:48 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]': finished 2024-04-14T15:24:48.093 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:24:48.121 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:48.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:48.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:24:48.223 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:24:48.251 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:48.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:48.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:24:48.324 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:48 smithi090 ceph-mon[30399]: osdmap e743: 8 total, 8 up, 7 in 2024-04-14T15:24:48.324 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:48 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-130"}]: dispatch 2024-04-14T15:24:48.325 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:48 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-130"}]: dispatch 2024-04-14T15:24:48.325 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:48 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]': finished 2024-04-14T15:24:48.325 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:48 smithi090 ceph-mon[41670]: osdmap e743: 8 total, 8 up, 7 in 2024-04-14T15:24:48.325 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:48 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-130"}]: dispatch 2024-04-14T15:24:48.325 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:48 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-130"}]: dispatch 2024-04-14T15:24:48.352 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:24:48.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:48 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]': finished 2024-04-14T15:24:48.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:48 smithi092 ceph-mon[33966]: osdmap e743: 8 total, 8 up, 7 in 2024-04-14T15:24:48.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:48 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-130"}]: dispatch 2024-04-14T15:24:48.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:48 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-130"}]: dispatch 2024-04-14T15:24:48.380 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:48.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:48.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:24:48.482 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:24:48.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:48.506 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:48.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:24:48.607 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:24:48.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:48.632 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:48.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:24:48.733 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:24:48.757 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:48.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:48.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:24:48.859 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:24:48.887 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:48.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:48.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:24:48.988 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:24:49.018 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:49.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:49.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:24:49.120 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:24:49.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:49.145 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:49.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:24:49.204 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:49 smithi090 ceph-mon[30399]: pgmap v1099: 214 pgs: 214 active+clean; 582 KiB data, 707 MiB used, 625 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-14T15:24:49.204 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:49 smithi090 ceph-mon[30399]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:24:49.204 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:49 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-130"}]': finished 2024-04-14T15:24:49.204 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:49 smithi090 ceph-mon[30399]: osdmap e744: 8 total, 8 up, 7 in 2024-04-14T15:24:49.204 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:49 smithi090 ceph-mon[41670]: pgmap v1099: 214 pgs: 214 active+clean; 582 KiB data, 707 MiB used, 625 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-14T15:24:49.205 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:49 smithi090 ceph-mon[41670]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:24:49.205 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:49 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-130"}]': finished 2024-04-14T15:24:49.205 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:49 smithi090 ceph-mon[41670]: osdmap e744: 8 total, 8 up, 7 in 2024-04-14T15:24:49.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:49.203+0000 7f659bc25700 1 -- 172.21.15.90:0/1924915888 >> v1:172.21.15.90:6789/0 conn(0x7f6594109df0 legacy=0x7f659410c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:49.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:49.203+0000 7f659bc25700 1 -- 172.21.15.90:0/1924915888 shutdown_connections 2024-04-14T15:24:49.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:49.203+0000 7f659bc25700 1 -- 172.21.15.90:0/1924915888 >> 172.21.15.90:0/1924915888 conn(0x7f6594076aa0 msgr2=0x7f6594076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:24:49.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:49.203+0000 7f659bc25700 1 -- 172.21.15.90:0/1924915888 shutdown_connections 2024-04-14T15:24:49.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:49.203+0000 7f659bc25700 1 -- 172.21.15.90:0/1924915888 wait complete. 2024-04-14T15:24:49.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:49.204+0000 7f659bc25700 1 librados: starting msgr at 2024-04-14T15:24:49.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:49.204+0000 7f659bc25700 1 librados: starting objecter 2024-04-14T15:24:49.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:49.204+0000 7f659bc25700 1 Processor -- start 2024-04-14T15:24:49.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:49.204+0000 7f659bc25700 1 -- start start 2024-04-14T15:24:49.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:49.204+0000 7f659bc25700 1 librados: setting wanted keys 2024-04-14T15:24:49.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:49.204+0000 7f659bc25700 1 librados: calling monclient init 2024-04-14T15:24:49.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:49.205+0000 7f659bc25700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f65941a5980 con 0x7f659410d990 2024-04-14T15:24:49.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:49.205+0000 7f659bc25700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f65941a6ba0 con 0x7f6594109df0 2024-04-14T15:24:49.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:49.205+0000 7f659bc25700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f65941a7de0 con 0x7f65941012e0 2024-04-14T15:24:49.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:49.205+0000 7f65999c1700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f65941012e0 0x7f6594103220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:50442/0 (socket says 172.21.15.90:50442) 2024-04-14T15:24:49.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:49.205+0000 7f65999c1700 1 -- 172.21.15.90:0/417739841 learned_addr learned my addr 172.21.15.90:0/417739841 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:24:49.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:49.205+0000 7f658affd700 10 client.?.objecter ms_handle_connect 0x7f659410d990 2024-04-14T15:24:49.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:49.205+0000 7f658affd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:24:49.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:49.205+0000 7f658affd700 10 client.?.objecter ms_handle_connect 0x7f65941012e0 2024-04-14T15:24:49.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:49.205+0000 7f658affd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:24:49.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:49.205+0000 7f658affd700 10 client.?.objecter ms_handle_connect 0x7f6594109df0 2024-04-14T15:24:49.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:49.205+0000 7f658affd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:24:49.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:49.205+0000 7f658affd700 1 -- 172.21.15.90:0/417739841 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 909876661 0 0) 0x7f65941a7de0 con 0x7f65941012e0 2024-04-14T15:24:49.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:49.205+0000 7f658affd700 1 -- 172.21.15.90:0/417739841 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f65800031f0 con 0x7f65941012e0 2024-04-14T15:24:49.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:49.205+0000 7f658affd700 1 -- 172.21.15.90:0/417739841 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 543434444 0 0) 0x7f65941a5980 con 0x7f659410d990 2024-04-14T15:24:49.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:49.205+0000 7f658affd700 1 -- 172.21.15.90:0/417739841 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f65941a7de0 con 0x7f659410d990 2024-04-14T15:24:49.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:49.205+0000 7f658affd700 1 -- 172.21.15.90:0/417739841 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2011037315 0 0) 0x7f65941a6ba0 con 0x7f6594109df0 2024-04-14T15:24:49.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:49.205+0000 7f658affd700 1 -- 172.21.15.90:0/417739841 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f65941a5980 con 0x7f6594109df0 2024-04-14T15:24:49.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:49.205+0000 7f658affd700 1 -- 172.21.15.90:0/417739841 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3046197234 0 0) 0x7f65800031f0 con 0x7f65941012e0 2024-04-14T15:24:49.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:49.205+0000 7f658affd700 1 -- 172.21.15.90:0/417739841 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f65941a6ba0 con 0x7f65941012e0 2024-04-14T15:24:49.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:49.205+0000 7f658affd700 1 -- 172.21.15.90:0/417739841 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f6590001930 con 0x7f65941012e0 2024-04-14T15:24:49.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:49.205+0000 7f658affd700 1 -- 172.21.15.90:0/417739841 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 590525612 0 0) 0x7f65941a5980 con 0x7f6594109df0 2024-04-14T15:24:49.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:49.205+0000 7f658affd700 1 -- 172.21.15.90:0/417739841 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f65800031f0 con 0x7f6594109df0 2024-04-14T15:24:49.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:49.205+0000 7f658affd700 1 -- 172.21.15.90:0/417739841 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2619058254 0 0) 0x7f65941a7de0 con 0x7f659410d990 2024-04-14T15:24:49.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:49.205+0000 7f658affd700 1 -- 172.21.15.90:0/417739841 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f65941a5980 con 0x7f659410d990 2024-04-14T15:24:49.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:49.205+0000 7f658affd700 1 -- 172.21.15.90:0/417739841 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f65840015d0 con 0x7f6594109df0 2024-04-14T15:24:49.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:49.205+0000 7f658affd700 1 -- 172.21.15.90:0/417739841 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f658c001bb0 con 0x7f659410d990 2024-04-14T15:24:49.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:49.205+0000 7f658affd700 1 -- 172.21.15.90:0/417739841 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2242462737 0 0) 0x7f65800031f0 con 0x7f6594109df0 2024-04-14T15:24:49.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:49.205+0000 7f658affd700 1 -- 172.21.15.90:0/417739841 >> v1:172.21.15.90:6790/0 conn(0x7f65941012e0 legacy=0x7f6594103220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:49.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:49.206+0000 7f658affd700 1 -- 172.21.15.90:0/417739841 >> v1:172.21.15.90:6789/0 conn(0x7f659410d990 legacy=0x7f65941a4040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:49.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:49.206+0000 7f658affd700 1 -- 172.21.15.90:0/417739841 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65941a9020 con 0x7f6594109df0 2024-04-14T15:24:49.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:49.206+0000 7f659bc25700 1 -- 172.21.15.90:0/417739841 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f65941a8010 con 0x7f6594109df0 2024-04-14T15:24:49.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:49.206+0000 7f659bc25700 10 client.49771.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:24:49.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:49.206+0000 7f659bc25700 1 -- 172.21.15.90:0/417739841 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f65941a84d0 con 0x7f6594109df0 2024-04-14T15:24:49.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:49.206+0000 7f659bc25700 1 librados: init done 2024-04-14T15:24:49.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:49.206+0000 7f658affd700 1 -- 172.21.15.90:0/417739841 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f6584001190 con 0x7f6594109df0 2024-04-14T15:24:49.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:49.206+0000 7f658affd700 1 -- 172.21.15.90:0/417739841 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f6584004ca0 con 0x7f6594109df0 2024-04-14T15:24:49.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:49.207+0000 7f659bc25700 1 -- 172.21.15.90:0/417739841 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6578005120 con 0x7f6594109df0 2024-04-14T15:24:49.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:49.207+0000 7f658affd700 1 -- 172.21.15.90:0/417739841 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f6584001b70 con 0x7f6594109df0 2024-04-14T15:24:49.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:49.208+0000 7f658affd700 1 -- 172.21.15.90:0/417739841 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(745..745 src has 467..745) ==== 8376+0+0 (unknown 335460840 0 0) 0x7f6584089870 con 0x7f6594109df0 2024-04-14T15:24:49.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:49.208+0000 7f658affd700 10 client.49771.objecter ms_dispatch 0x7f6594001550 osd_map(745..745 src has 467..745) 2024-04-14T15:24:49.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:49.208+0000 7f658affd700 3 client.49771.objecter handle_osd_map got epochs [745,745] > 0 2024-04-14T15:24:49.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:49.208+0000 7f658affd700 3 client.49771.objecter handle_osd_map decoding full epoch 745 2024-04-14T15:24:49.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:49.208+0000 7f658affd700 20 client.49771.objecter dump_active .. 0 homeless 2024-04-14T15:24:49.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:49.208+0000 7f658affd700 10 client.49771.objecter ms_handle_connect 0x7f6580073600 2024-04-14T15:24:49.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:49.212+0000 7f658affd700 1 -- 172.21.15.90:0/417739841 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f65840561d0 con 0x7f6594109df0 2024-04-14T15:24:49.247 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:24:49.275 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:49.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:49.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:24:49.286 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:24:49.286 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:24:49.294 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:24:49.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:49 smithi092 ceph-mon[33966]: pgmap v1099: 214 pgs: 214 active+clean; 582 KiB data, 707 MiB used, 625 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-14T15:24:49.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:49 smithi092 ceph-mon[33966]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:24:49.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:49 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-130"}]': finished 2024-04-14T15:24:49.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:49 smithi092 ceph-mon[33966]: osdmap e744: 8 total, 8 up, 7 in 2024-04-14T15:24:49.377 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:24:49.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:49.401 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:49.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:24:49.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:49.444+0000 7f659bc25700 1 -- 172.21.15.90:0/417739841 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7f6578005410 con 0x7f6594109df0 2024-04-14T15:24:49.502 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:24:49.530 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:49.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:49.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:24:49.632 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:24:49.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:49.656 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:49.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:24:49.757 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:24:49.785 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:49.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:49.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:24:49.887 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:24:49.911 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:49.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:49.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:24:50.013 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:24:50.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:50.025+0000 7f658affd700 1 -- 172.21.15.90:0/417739841 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v746) ==== 97+0+0 (unknown 2492100701 0 0) 0x7f6584059e80 con 0x7f6594109df0 2024-04-14T15:24:50.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:50.029+0000 7f659bc25700 10 librados: watch_flush enter 2024-04-14T15:24:50.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:50.030+0000 7f659bc25700 10 librados: watch_flush exit 2024-04-14T15:24:50.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:50.030+0000 7f659bc25700 20 client.49771.objecter shutdown clearing up homeless session... 2024-04-14T15:24:50.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:50.030+0000 7f659bc25700 10 client.49771.objecter successfully canceled tick 2024-04-14T15:24:50.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:50.031+0000 7f659bc25700 1 -- 172.21.15.90:0/417739841 >> v1:172.21.15.90:6800/420935427 conn(0x7f6580073600 legacy=0x7f6580075ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:50.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:50.031+0000 7f659bc25700 1 -- 172.21.15.90:0/417739841 >> v1:172.21.15.92:6789/0 conn(0x7f6594109df0 legacy=0x7f6594103930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:50.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:50.032+0000 7f659bc25700 1 -- 172.21.15.90:0/417739841 shutdown_connections 2024-04-14T15:24:50.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:50.032+0000 7f659bc25700 1 -- 172.21.15.90:0/417739841 >> 172.21.15.90:0/417739841 conn(0x7f6594076aa0 msgr2=0x7f6594110e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:24:50.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:50.034+0000 7f659bc25700 1 -- 172.21.15.90:0/417739841 shutdown_connections 2024-04-14T15:24:50.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:50.034+0000 7f659bc25700 1 -- 172.21.15.90:0/417739841 wait complete. 2024-04-14T15:24:50.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:50.034+0000 7f659bc25700 1 librados: shutdown 2024-04-14T15:24:50.035 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is set 2024-04-14T15:24:50.047 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:50.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:50.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:24:50.122 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:50 smithi090 ceph-mon[30399]: osdmap e745: 8 total, 8 up, 7 in 2024-04-14T15:24:50.122 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:50 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/417739841' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:24:50.122 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:50 smithi090 ceph-mon[30399]: from='client.49771 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:24:50.123 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:50 smithi090 ceph-mon[41670]: osdmap e745: 8 total, 8 up, 7 in 2024-04-14T15:24:50.123 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:50 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/417739841' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:24:50.123 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:50 smithi090 ceph-mon[41670]: from='client.49771 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:24:50.149 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:24:50.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:50.174 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:50.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:24:50.275 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:24:50.303 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:50.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:50.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:24:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:50 smithi092 ceph-mon[33966]: osdmap e745: 8 total, 8 up, 7 in 2024-04-14T15:24:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:50 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/417739841' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:24:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:50 smithi092 ceph-mon[33966]: from='client.49771 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:24:50.405 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:24:50.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:50.430 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:50.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:24:50.531 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:24:50.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:50.559 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:50.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:24:50.660 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:24:50.686 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:50.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:50.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:24:50.788 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:24:50.814 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:50.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:50.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:24:50.916 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:24:50.941 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:50.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:50.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:24:51.043 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:24:51.069 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:51.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:51.070 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:24:51.171 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:24:51.199 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:51.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:51.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:24:51.301 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:24:51.325 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:51.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:51.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:24:51.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:51 smithi092 ceph-mon[33966]: pgmap v1102: 182 pgs: 182 active+clean; 582 KiB data, 708 MiB used, 625 GiB / 626 GiB avail; 1.3 KiB/s rd, 263 B/s wr, 2 op/s 2024-04-14T15:24:51.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:51 smithi092 ceph-mon[33966]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:24:51.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:51 smithi092 ceph-mon[33966]: from='client.49771 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-14T15:24:51.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:51 smithi092 ceph-mon[33966]: osdmap e746: 8 total, 8 up, 7 in 2024-04-14T15:24:51.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:51 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:24:51.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:51 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:24:51.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:51 smithi092 ceph-mon[33966]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:24:51.376 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:51 smithi090 ceph-mon[41670]: pgmap v1102: 182 pgs: 182 active+clean; 582 KiB data, 708 MiB used, 625 GiB / 626 GiB avail; 1.3 KiB/s rd, 263 B/s wr, 2 op/s 2024-04-14T15:24:51.376 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:51 smithi090 ceph-mon[41670]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:24:51.376 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:51 smithi090 ceph-mon[41670]: from='client.49771 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-14T15:24:51.376 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:51 smithi090 ceph-mon[41670]: osdmap e746: 8 total, 8 up, 7 in 2024-04-14T15:24:51.376 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:51 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:24:51.376 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:51 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:24:51.376 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:51 smithi090 ceph-mon[41670]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:24:51.376 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:51 smithi090 ceph-mon[30399]: pgmap v1102: 182 pgs: 182 active+clean; 582 KiB data, 708 MiB used, 625 GiB / 626 GiB avail; 1.3 KiB/s rd, 263 B/s wr, 2 op/s 2024-04-14T15:24:51.376 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:51 smithi090 ceph-mon[30399]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:24:51.376 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:51 smithi090 ceph-mon[30399]: from='client.49771 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-14T15:24:51.376 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:51 smithi090 ceph-mon[30399]: osdmap e746: 8 total, 8 up, 7 in 2024-04-14T15:24:51.377 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:51 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:24:51.377 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:51 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:24:51.377 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:51 smithi090 ceph-mon[30399]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:24:51.426 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:24:51.453 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:51.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:51.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:24:51.555 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:24:51.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:51.583 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:51.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:24:51.684 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:24:51.711 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:51.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:51.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:24:51.813 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:24:51.840 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:51.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:51.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:24:51.942 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:24:51.969 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:51.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:51.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:24:52.071 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:24:52.095 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:52.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:52.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:24:52.197 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:24:52.222 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:52.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:52.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:24:52.286 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:52 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:24:52.286 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:52 smithi092 ceph-mon[33966]: osdmap e747: 8 total, 8 up, 7 in 2024-04-14T15:24:52.287 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:52 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:24:52.287 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:52 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:24:52.324 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:24:52.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:52.349 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:52.349 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:24:52.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:52 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:24:52.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:52 smithi090 ceph-mon[41670]: osdmap e747: 8 total, 8 up, 7 in 2024-04-14T15:24:52.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:52 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:24:52.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:52 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:24:52.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:52 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:24:52.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:52 smithi090 ceph-mon[30399]: osdmap e747: 8 total, 8 up, 7 in 2024-04-14T15:24:52.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:52 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:24:52.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:52 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:24:52.450 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:24:52.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:52.474 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:52.474 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:24:52.575 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:24:52.599 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:52.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:52.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:24:52.701 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:24:52.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:52.726 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:52.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:24:52.827 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:24:52.850 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:52.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:52.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:24:52.952 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:24:52.975 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:52.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:52.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:24:53.077 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:24:53.101 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:53.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:53.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:24:53.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:53 smithi090 ceph-mon[41670]: pgmap v1105: 214 pgs: 32 unknown, 182 active+clean; 582 KiB data, 708 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-14T15:24:53.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:53 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-132", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:24:53.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:53 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-132"}]: dispatch 2024-04-14T15:24:53.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:53 smithi090 ceph-mon[41670]: osdmap e748: 8 total, 8 up, 7 in 2024-04-14T15:24:53.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:53 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-132"}]: dispatch 2024-04-14T15:24:53.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:53 smithi090 ceph-mon[30399]: pgmap v1105: 214 pgs: 32 unknown, 182 active+clean; 582 KiB data, 708 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-14T15:24:53.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:53 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-132", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:24:53.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:53 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-132"}]: dispatch 2024-04-14T15:24:53.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:53 smithi090 ceph-mon[30399]: osdmap e748: 8 total, 8 up, 7 in 2024-04-14T15:24:53.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:53 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-132"}]: dispatch 2024-04-14T15:24:53.203 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:24:53.226 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:53.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:53.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:24:53.327 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:24:53.351 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:53.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:53.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:24:53.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:53 smithi092 ceph-mon[33966]: pgmap v1105: 214 pgs: 32 unknown, 182 active+clean; 582 KiB data, 708 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-14T15:24:53.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:53 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-132", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:24:53.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:53 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-132"}]: dispatch 2024-04-14T15:24:53.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:53 smithi092 ceph-mon[33966]: osdmap e748: 8 total, 8 up, 7 in 2024-04-14T15:24:53.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:53 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-132"}]: dispatch 2024-04-14T15:24:53.452 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:24:53.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:53.456+0000 7fc956d76700 1 -- 172.21.15.90:0/1409304814 >> v1:172.21.15.90:6789/0 conn(0x7fc95010b820 legacy=0x7fc95010dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:53.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:53.456+0000 7fc956d76700 1 -- 172.21.15.90:0/1409304814 shutdown_connections 2024-04-14T15:24:53.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:53.456+0000 7fc956d76700 1 -- 172.21.15.90:0/1409304814 >> 172.21.15.90:0/1409304814 conn(0x7fc950076aa0 msgr2=0x7fc950076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:24:53.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:53.456+0000 7fc956d76700 1 -- 172.21.15.90:0/1409304814 shutdown_connections 2024-04-14T15:24:53.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:53.456+0000 7fc956d76700 1 -- 172.21.15.90:0/1409304814 wait complete. 2024-04-14T15:24:53.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:53.457+0000 7fc956d76700 1 librados: starting msgr at 2024-04-14T15:24:53.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:53.457+0000 7fc956d76700 1 librados: starting objecter 2024-04-14T15:24:53.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:53.457+0000 7fc956d76700 1 Processor -- start 2024-04-14T15:24:53.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:53.457+0000 7fc956d76700 1 -- start start 2024-04-14T15:24:53.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:53.457+0000 7fc956d76700 1 librados: setting wanted keys 2024-04-14T15:24:53.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:53.457+0000 7fc956d76700 1 librados: calling monclient init 2024-04-14T15:24:53.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:53.457+0000 7fc956d76700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc950101c40 con 0x7fc95010f420 2024-04-14T15:24:53.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:53.457+0000 7fc956d76700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc950101e50 con 0x7fc95010b820 2024-04-14T15:24:53.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:53.457+0000 7fc956d76700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc9501aa020 con 0x7fc950108a20 2024-04-14T15:24:53.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:53.457+0000 7fc955313700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fc95010f420 0x7fc9501a68e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:52452/0 (socket says 172.21.15.90:52452) 2024-04-14T15:24:53.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:53.457+0000 7fc955313700 1 -- 172.21.15.90:0/1116368187 learned_addr learned my addr 172.21.15.90:0/1116368187 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:24:53.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:53.458+0000 7fc94dffb700 10 client.?.objecter ms_handle_connect 0x7fc95010f420 2024-04-14T15:24:53.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:53.458+0000 7fc94dffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:24:53.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:53.458+0000 7fc94dffb700 10 client.?.objecter ms_handle_connect 0x7fc95010b820 2024-04-14T15:24:53.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:53.458+0000 7fc94dffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:24:53.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:53.458+0000 7fc94dffb700 10 client.?.objecter ms_handle_connect 0x7fc950108a20 2024-04-14T15:24:53.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:53.458+0000 7fc94dffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:24:53.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:53.458+0000 7fc94dffb700 1 -- 172.21.15.90:0/1116368187 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1081520299 0 0) 0x7fc950101c40 con 0x7fc95010f420 2024-04-14T15:24:53.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:53.458+0000 7fc94dffb700 1 -- 172.21.15.90:0/1116368187 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9400035b0 con 0x7fc95010f420 2024-04-14T15:24:53.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:53.458+0000 7fc94dffb700 1 -- 172.21.15.90:0/1116368187 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3072945711 0 0) 0x7fc950101e50 con 0x7fc95010b820 2024-04-14T15:24:53.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:53.458+0000 7fc94dffb700 1 -- 172.21.15.90:0/1116368187 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc950101c40 con 0x7fc95010b820 2024-04-14T15:24:53.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:53.458+0000 7fc94dffb700 1 -- 172.21.15.90:0/1116368187 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2554854863 0 0) 0x7fc9501aa020 con 0x7fc950108a20 2024-04-14T15:24:53.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:53.458+0000 7fc94dffb700 1 -- 172.21.15.90:0/1116368187 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc950101e50 con 0x7fc950108a20 2024-04-14T15:24:53.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:53.458+0000 7fc94dffb700 1 -- 172.21.15.90:0/1116368187 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1230854009 0 0) 0x7fc9400035b0 con 0x7fc95010f420 2024-04-14T15:24:53.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:53.458+0000 7fc94dffb700 1 -- 172.21.15.90:0/1116368187 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc9501aa020 con 0x7fc95010f420 2024-04-14T15:24:53.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:53.458+0000 7fc94dffb700 1 -- 172.21.15.90:0/1116368187 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fc948001c30 con 0x7fc95010f420 2024-04-14T15:24:53.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:53.458+0000 7fc94dffb700 1 -- 172.21.15.90:0/1116368187 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 212950444 0 0) 0x7fc950101c40 con 0x7fc95010b820 2024-04-14T15:24:53.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:53.458+0000 7fc94dffb700 1 -- 172.21.15.90:0/1116368187 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc9400035b0 con 0x7fc95010b820 2024-04-14T15:24:53.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:53.458+0000 7fc94dffb700 1 -- 172.21.15.90:0/1116368187 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 662697089 0 0) 0x7fc950101e50 con 0x7fc950108a20 2024-04-14T15:24:53.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:53.458+0000 7fc94dffb700 1 -- 172.21.15.90:0/1116368187 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc950101c40 con 0x7fc950108a20 2024-04-14T15:24:53.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:53.458+0000 7fc94dffb700 1 -- 172.21.15.90:0/1116368187 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fc93c0015d0 con 0x7fc95010b820 2024-04-14T15:24:53.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:53.458+0000 7fc94dffb700 1 -- 172.21.15.90:0/1116368187 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fc9440017f0 con 0x7fc950108a20 2024-04-14T15:24:53.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:53.458+0000 7fc94dffb700 1 -- 172.21.15.90:0/1116368187 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3290879137 0 0) 0x7fc9501aa020 con 0x7fc95010f420 2024-04-14T15:24:53.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:53.458+0000 7fc94dffb700 1 -- 172.21.15.90:0/1116368187 >> v1:172.21.15.90:6790/0 conn(0x7fc950108a20 legacy=0x7fc9500fea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:53.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:53.458+0000 7fc94dffb700 1 -- 172.21.15.90:0/1116368187 >> v1:172.21.15.92:6789/0 conn(0x7fc95010b820 legacy=0x7fc9500ff1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:53.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:53.458+0000 7fc94dffb700 1 -- 172.21.15.90:0/1116368187 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9501ab260 con 0x7fc95010f420 2024-04-14T15:24:53.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:53.458+0000 7fc956d76700 1 -- 172.21.15.90:0/1116368187 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc9501aa250 con 0x7fc95010f420 2024-04-14T15:24:53.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:53.458+0000 7fc94dffb700 1 -- 172.21.15.90:0/1116368187 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fc948003ea0 con 0x7fc95010f420 2024-04-14T15:24:53.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:53.459+0000 7fc94dffb700 1 -- 172.21.15.90:0/1116368187 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fc948006070 con 0x7fc95010f420 2024-04-14T15:24:53.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:53.459+0000 7fc956d76700 10 client.37947.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:24:53.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:53.459+0000 7fc956d76700 1 -- 172.21.15.90:0/1116368187 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc9501aa710 con 0x7fc95010f420 2024-04-14T15:24:53.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:53.459+0000 7fc956d76700 1 librados: init done 2024-04-14T15:24:53.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:53.459+0000 7fc956d76700 1 -- 172.21.15.90:0/1116368187 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc95006aaf0 con 0x7fc95010f420 2024-04-14T15:24:53.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:53.460+0000 7fc94dffb700 1 -- 172.21.15.90:0/1116368187 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fc948003a00 con 0x7fc95010f420 2024-04-14T15:24:53.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:53.460+0000 7fc94dffb700 1 -- 172.21.15.90:0/1116368187 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(749..749 src has 467..749) ==== 8757+0+0 (unknown 3700949836 0 0) 0x7fc94808af80 con 0x7fc95010f420 2024-04-14T15:24:53.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:53.460+0000 7fc94dffb700 10 client.37947.objecter ms_dispatch 0x7fc950001550 osd_map(749..749 src has 467..749) 2024-04-14T15:24:53.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:53.460+0000 7fc94dffb700 3 client.37947.objecter handle_osd_map got epochs [749,749] > 0 2024-04-14T15:24:53.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:53.460+0000 7fc94dffb700 3 client.37947.objecter handle_osd_map decoding full epoch 749 2024-04-14T15:24:53.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:53.461+0000 7fc94dffb700 20 client.37947.objecter dump_active .. 0 homeless 2024-04-14T15:24:53.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:53.461+0000 7fc94dffb700 10 client.37947.objecter ms_handle_connect 0x7fc940071440 2024-04-14T15:24:53.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:53.465+0000 7fc94dffb700 1 -- 172.21.15.90:0/1116368187 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fc948057760 con 0x7fc95010f420 2024-04-14T15:24:53.477 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:53.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:53.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:24:53.578 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:24:53.602 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:53.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:53.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:24:53.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:53.677+0000 7fc956d76700 1 -- 172.21.15.90:0/1116368187 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} v 0) -- 0x7fc9500fffc0 con 0x7fc95010f420 2024-04-14T15:24:53.704 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:24:53.728 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:53.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:53.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:24:53.830 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:24:53.854 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:53.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:53.855 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:24:53.955 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:24:53.978 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:53.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:53.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:24:54.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:54.030+0000 7fc94dffb700 1 -- 172.21.15.90:0/1116368187 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]=0 set pool 1 pg_num to 21 v750) ==== 129+0+0 (unknown 3348896147 0 0) 0x7fc94805b410 con 0x7fc95010f420 2024-04-14T15:24:54.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:54.032+0000 7fc956d76700 10 librados: watch_flush enter 2024-04-14T15:24:54.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:54.032+0000 7fc956d76700 10 librados: watch_flush exit 2024-04-14T15:24:54.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:54.032+0000 7fc956d76700 20 client.37947.objecter shutdown clearing up homeless session... 2024-04-14T15:24:54.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:54.032+0000 7fc956d76700 10 client.37947.objecter successfully canceled tick 2024-04-14T15:24:54.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:54.033+0000 7fc956d76700 1 -- 172.21.15.90:0/1116368187 >> v1:172.21.15.90:6800/420935427 conn(0x7fc940071440 legacy=0x7fc940073900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:54.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:54.033+0000 7fc956d76700 1 -- 172.21.15.90:0/1116368187 >> v1:172.21.15.90:6789/0 conn(0x7fc95010f420 legacy=0x7fc9501a68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:54.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:54.033+0000 7fc956d76700 1 -- 172.21.15.90:0/1116368187 shutdown_connections 2024-04-14T15:24:54.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:54.033+0000 7fc956d76700 1 -- 172.21.15.90:0/1116368187 >> 172.21.15.90:0/1116368187 conn(0x7fc950076aa0 msgr2=0x7fc950111880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:24:54.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:54.033+0000 7fc956d76700 1 -- 172.21.15.90:0/1116368187 shutdown_connections 2024-04-14T15:24:54.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:54.033+0000 7fc956d76700 1 -- 172.21.15.90:0/1116368187 wait complete. 2024-04-14T15:24:54.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:54.034+0000 7fc956d76700 1 librados: shutdown 2024-04-14T15:24:54.037 INFO:teuthology.orchestra.run.smithi090.stderr:set pool 1 pg_num to 21 2024-04-14T15:24:54.055 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd unset noscrub 2024-04-14T15:24:54.080 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:24:54.122 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:54.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:54.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:24:54.224 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:24:54.248 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:54.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:54.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:24:54.350 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:24:54.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:54 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-132"}]': finished 2024-04-14T15:24:54.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:54 smithi092 ceph-mon[33966]: osdmap e749: 8 total, 8 up, 7 in 2024-04-14T15:24:54.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:54 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-132", "mode": "writeback"}]: dispatch 2024-04-14T15:24:54.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:54 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-132", "mode": "writeback"}]: dispatch 2024-04-14T15:24:54.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:54 smithi092 ceph-mon[33966]: pgmap v1108: 214 pgs: 32 unknown, 182 active+clean; 582 KiB data, 708 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:24:54.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:54 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1116368187' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-04-14T15:24:54.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:54 smithi092 ceph-mon[33966]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:24:54.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:54 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-132", "mode": "writeback"}]': finished 2024-04-14T15:24:54.374 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:54.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:54.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:24:54.375 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:24:53 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:24:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:24:54.376 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:54 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-132"}]': finished 2024-04-14T15:24:54.376 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:54 smithi090 ceph-mon[41670]: osdmap e749: 8 total, 8 up, 7 in 2024-04-14T15:24:54.376 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:54 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-132", "mode": "writeback"}]: dispatch 2024-04-14T15:24:54.376 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:54 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-132", "mode": "writeback"}]: dispatch 2024-04-14T15:24:54.376 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:54 smithi090 ceph-mon[41670]: pgmap v1108: 214 pgs: 32 unknown, 182 active+clean; 582 KiB data, 708 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:24:54.376 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:54 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1116368187' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-04-14T15:24:54.376 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:54 smithi090 ceph-mon[41670]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:24:54.376 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:54 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-132", "mode": "writeback"}]': finished 2024-04-14T15:24:54.376 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:54 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-132"}]': finished 2024-04-14T15:24:54.377 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:54 smithi090 ceph-mon[30399]: osdmap e749: 8 total, 8 up, 7 in 2024-04-14T15:24:54.377 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:54 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-132", "mode": "writeback"}]: dispatch 2024-04-14T15:24:54.377 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:54 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-132", "mode": "writeback"}]: dispatch 2024-04-14T15:24:54.377 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:54 smithi090 ceph-mon[30399]: pgmap v1108: 214 pgs: 32 unknown, 182 active+clean; 582 KiB data, 708 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:24:54.377 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:54 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1116368187' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-04-14T15:24:54.377 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:54 smithi090 ceph-mon[30399]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:24:54.377 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:54 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-132", "mode": "writeback"}]': finished 2024-04-14T15:24:54.475 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:24:54.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:54.499 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:54.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:24:54.600 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:24:54.624 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:54.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:54.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:24:54.726 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:24:54.748 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:54.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:54.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:24:54.851 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:24:54.876 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:54.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:54.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:24:54.977 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:24:55.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:55.002 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:55.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:24:55.103 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:24:55.127 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:55.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:55.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:24:55.229 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:24:55.252 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:55.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:55.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:24:55.315 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:55 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1116368187' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-04-14T15:24:55.315 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:55 smithi090 ceph-mon[41670]: osdmap e750: 8 total, 8 up, 7 in 2024-04-14T15:24:55.316 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:55 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:24:55.316 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:55 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1116368187' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-04-14T15:24:55.316 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:55 smithi090 ceph-mon[30399]: osdmap e750: 8 total, 8 up, 7 in 2024-04-14T15:24:55.316 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:55 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:24:55.354 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:24:55.371 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:55 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1116368187' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-04-14T15:24:55.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:55 smithi092 ceph-mon[33966]: osdmap e750: 8 total, 8 up, 7 in 2024-04-14T15:24:55.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:55 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:24:55.378 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:55.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:55.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:24:55.479 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:24:55.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:55.504 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:55.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:24:55.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.528+0000 7fb5254ce700 1 -- 172.21.15.90:0/3077067265 >> v1:172.21.15.90:6789/0 conn(0x7fb52010e230 legacy=0x7fb5201106f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:55.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.528+0000 7fb5254ce700 1 -- 172.21.15.90:0/3077067265 shutdown_connections 2024-04-14T15:24:55.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.528+0000 7fb5254ce700 1 -- 172.21.15.90:0/3077067265 >> 172.21.15.90:0/3077067265 conn(0x7fb5200fd2a0 msgr2=0x7fb5200ff6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:24:55.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.528+0000 7fb5254ce700 1 -- 172.21.15.90:0/3077067265 shutdown_connections 2024-04-14T15:24:55.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.528+0000 7fb5254ce700 1 -- 172.21.15.90:0/3077067265 wait complete. 2024-04-14T15:24:55.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.529+0000 7fb5254ce700 1 librados: starting msgr at 2024-04-14T15:24:55.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.529+0000 7fb5254ce700 1 librados: starting objecter 2024-04-14T15:24:55.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.529+0000 7fb5254ce700 1 Processor -- start 2024-04-14T15:24:55.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.529+0000 7fb5254ce700 1 -- start start 2024-04-14T15:24:55.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.529+0000 7fb5254ce700 1 librados: setting wanted keys 2024-04-14T15:24:55.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.529+0000 7fb5254ce700 1 librados: calling monclient init 2024-04-14T15:24:55.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.529+0000 7fb5254ce700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb5200ff250 con 0x7fb52010a660 2024-04-14T15:24:55.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.529+0000 7fb5254ce700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb5200ff460 con 0x7fb5201077b0 2024-04-14T15:24:55.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.529+0000 7fb5254ce700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb5200ff670 con 0x7fb52010e230 2024-04-14T15:24:55.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.529+0000 7fb51e7fc700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fb52010a660 0x7fb5200fde60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:52476/0 (socket says 172.21.15.90:52476) 2024-04-14T15:24:55.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.529+0000 7fb51e7fc700 1 -- 172.21.15.90:0/1951774438 learned_addr learned my addr 172.21.15.90:0/1951774438 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:24:55.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.529+0000 7fb517fff700 10 client.?.objecter ms_handle_connect 0x7fb52010a660 2024-04-14T15:24:55.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.529+0000 7fb517fff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:24:55.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.529+0000 7fb517fff700 10 client.?.objecter ms_handle_connect 0x7fb52010e230 2024-04-14T15:24:55.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.529+0000 7fb517fff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:24:55.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.529+0000 7fb517fff700 10 client.?.objecter ms_handle_connect 0x7fb5201077b0 2024-04-14T15:24:55.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.529+0000 7fb517fff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:24:55.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.529+0000 7fb517fff700 1 -- 172.21.15.90:0/1951774438 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 704916178 0 0) 0x7fb5200ff460 con 0x7fb5201077b0 2024-04-14T15:24:55.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.529+0000 7fb517fff700 1 -- 172.21.15.90:0/1951774438 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb50c0031f0 con 0x7fb5201077b0 2024-04-14T15:24:55.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.529+0000 7fb517fff700 1 -- 172.21.15.90:0/1951774438 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3242861599 0 0) 0x7fb5200ff250 con 0x7fb52010a660 2024-04-14T15:24:55.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.530+0000 7fb517fff700 1 -- 172.21.15.90:0/1951774438 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb5200ff460 con 0x7fb52010a660 2024-04-14T15:24:55.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.530+0000 7fb517fff700 1 -- 172.21.15.90:0/1951774438 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3189423920 0 0) 0x7fb5200ff670 con 0x7fb52010e230 2024-04-14T15:24:55.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.530+0000 7fb517fff700 1 -- 172.21.15.90:0/1951774438 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb5200ff250 con 0x7fb52010e230 2024-04-14T15:24:55.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.530+0000 7fb517fff700 1 -- 172.21.15.90:0/1951774438 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1003383909 0 0) 0x7fb50c0031f0 con 0x7fb5201077b0 2024-04-14T15:24:55.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.530+0000 7fb517fff700 1 -- 172.21.15.90:0/1951774438 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb5200ff670 con 0x7fb5201077b0 2024-04-14T15:24:55.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.530+0000 7fb517fff700 1 -- 172.21.15.90:0/1951774438 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fb518001bb0 con 0x7fb5201077b0 2024-04-14T15:24:55.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.530+0000 7fb517fff700 1 -- 172.21.15.90:0/1951774438 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2217932609 0 0) 0x7fb5200ff460 con 0x7fb52010a660 2024-04-14T15:24:55.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.530+0000 7fb517fff700 1 -- 172.21.15.90:0/1951774438 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb50c0031f0 con 0x7fb52010a660 2024-04-14T15:24:55.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.530+0000 7fb517fff700 1 -- 172.21.15.90:0/1951774438 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3452534697 0 0) 0x7fb5200ff250 con 0x7fb52010e230 2024-04-14T15:24:55.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.530+0000 7fb517fff700 1 -- 172.21.15.90:0/1951774438 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb5200ff460 con 0x7fb52010e230 2024-04-14T15:24:55.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.530+0000 7fb517fff700 1 -- 172.21.15.90:0/1951774438 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fb508001890 con 0x7fb52010a660 2024-04-14T15:24:55.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.530+0000 7fb517fff700 1 -- 172.21.15.90:0/1951774438 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fb510001800 con 0x7fb52010e230 2024-04-14T15:24:55.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.530+0000 7fb517fff700 1 -- 172.21.15.90:0/1951774438 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2190346923 0 0) 0x7fb5200ff670 con 0x7fb5201077b0 2024-04-14T15:24:55.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.530+0000 7fb517fff700 1 -- 172.21.15.90:0/1951774438 >> v1:172.21.15.90:6790/0 conn(0x7fb52010e230 legacy=0x7fb5201a8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:55.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.530+0000 7fb517fff700 1 -- 172.21.15.90:0/1951774438 >> v1:172.21.15.90:6789/0 conn(0x7fb52010a660 legacy=0x7fb5200fde60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:55.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.530+0000 7fb517fff700 1 -- 172.21.15.90:0/1951774438 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5201ad2a0 con 0x7fb5201077b0 2024-04-14T15:24:55.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.530+0000 7fb5254ce700 1 -- 172.21.15.90:0/1951774438 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb5201aa290 con 0x7fb5201077b0 2024-04-14T15:24:55.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.530+0000 7fb5254ce700 10 client.49783.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:24:55.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.530+0000 7fb5254ce700 1 -- 172.21.15.90:0/1951774438 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb5201aa750 con 0x7fb5201077b0 2024-04-14T15:24:55.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.530+0000 7fb5254ce700 1 librados: init done 2024-04-14T15:24:55.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.530+0000 7fb517fff700 1 -- 172.21.15.90:0/1951774438 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fb518003ea0 con 0x7fb5201077b0 2024-04-14T15:24:55.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.530+0000 7fb517fff700 1 -- 172.21.15.90:0/1951774438 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fb518004e20 con 0x7fb5201077b0 2024-04-14T15:24:55.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.531+0000 7fb5254ce700 1 -- 172.21.15.90:0/1951774438 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fb500005100 con 0x7fb5201077b0 2024-04-14T15:24:55.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.531+0000 7fb517fff700 1 -- 172.21.15.90:0/1951774438 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fb518003a00 con 0x7fb5201077b0 2024-04-14T15:24:55.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.532+0000 7fb517fff700 1 -- 172.21.15.90:0/1951774438 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(751..751 src has 467..751) ==== 8757+0+0 (unknown 2378880472 0 0) 0x7fb518089ff0 con 0x7fb5201077b0 2024-04-14T15:24:55.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.532+0000 7fb517fff700 10 client.49783.objecter ms_dispatch 0x7fb520001550 osd_map(751..751 src has 467..751) 2024-04-14T15:24:55.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.532+0000 7fb517fff700 3 client.49783.objecter handle_osd_map got epochs [751,751] > 0 2024-04-14T15:24:55.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.532+0000 7fb517fff700 3 client.49783.objecter handle_osd_map decoding full epoch 751 2024-04-14T15:24:55.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.532+0000 7fb517fff700 20 client.49783.objecter dump_active .. 0 homeless 2024-04-14T15:24:55.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.532+0000 7fb517fff700 1 -- 172.21.15.90:0/1951774438 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v751) ==== 54+0+16 (unknown 26108202 0 2420000192) 0x7fb51808a400 con 0x7fb5201077b0 2024-04-14T15:24:55.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.532+0000 7fb517fff700 10 client.49783.objecter ms_handle_connect 0x7fb50c071440 2024-04-14T15:24:55.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.532+0000 7fb5254ce700 10 client.49783.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:24:55.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.532+0000 7fb5254ce700 15 client.49783.objecter _session_command_op_assign -1 1 2024-04-14T15:24:55.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.532+0000 7fb5254ce700 20 client.49783.objecter _get_session s=0x7fb5000053e0 osd=0 3 2024-04-14T15:24:55.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.532+0000 7fb5254ce700 20 client.49783.objecter put_session s=0x7fb5000053e0 osd=0 3 2024-04-14T15:24:55.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.532+0000 7fb5254ce700 20 client.49783.objecter _get_session s=0x7fb5000053e0 osd=0 3 2024-04-14T15:24:55.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.532+0000 7fb5254ce700 15 client.49783.objecter _session_command_op_remove -1 1 2024-04-14T15:24:55.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.532+0000 7fb5254ce700 20 client.49783.objecter get_session s=0x7fb5000053e0 osd=0 3 2024-04-14T15:24:55.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.532+0000 7fb5254ce700 15 client.49783.objecter _session_command_op_assign 0 1 2024-04-14T15:24:55.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.532+0000 7fb5254ce700 20 client.49783.objecter put_session s=0x7fb5000053e0 osd=0 4 2024-04-14T15:24:55.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.532+0000 7fb5254ce700 10 client.49783.objecter _send_command 1 2024-04-14T15:24:55.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.532+0000 7fb5254ce700 1 -- 172.21.15.90:0/1951774438 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb500004b50 con 0x7fb500004730 2024-04-14T15:24:55.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.533+0000 7fb517fff700 10 client.49783.objecter ms_handle_connect 0x7fb500004730 2024-04-14T15:24:55.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.533+0000 7fb517fff700 1 -- 172.21.15.90:0/1951774438 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fb500004b50 con 0x7fb500004730 2024-04-14T15:24:55.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.533+0000 7fb517fff700 10 client.49783.objecter ms_dispatch 0x7fb520001550 command_reply(tid 1: 0 ) 2024-04-14T15:24:55.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.533+0000 7fb517fff700 10 client.49783.objecter _finish_command 1 = system:0 2024-04-14T15:24:55.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.533+0000 7fb517fff700 20 client.49783.objecter put_session s=0x7fb5000053e0 osd=0 4 2024-04-14T15:24:55.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.533+0000 7fb517fff700 15 client.49783.objecter _session_command_op_remove 0 1 2024-04-14T15:24:55.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.550+0000 7fb5254ce700 10 client.49783.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:24:55.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.550+0000 7fb5254ce700 15 client.49783.objecter _session_command_op_assign -1 2 2024-04-14T15:24:55.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.550+0000 7fb5254ce700 20 client.49783.objecter _get_session s=0x7fb5000053e0 osd=0 3 2024-04-14T15:24:55.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.550+0000 7fb5254ce700 20 client.49783.objecter put_session s=0x7fb5000053e0 osd=0 3 2024-04-14T15:24:55.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.550+0000 7fb5254ce700 20 client.49783.objecter _get_session s=0x7fb5000053e0 osd=0 3 2024-04-14T15:24:55.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.550+0000 7fb5254ce700 15 client.49783.objecter _session_command_op_remove -1 2 2024-04-14T15:24:55.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.550+0000 7fb5254ce700 20 client.49783.objecter get_session s=0x7fb5000053e0 osd=0 3 2024-04-14T15:24:55.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.550+0000 7fb5254ce700 15 client.49783.objecter _session_command_op_assign 0 2 2024-04-14T15:24:55.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.550+0000 7fb5254ce700 20 client.49783.objecter put_session s=0x7fb5000053e0 osd=0 4 2024-04-14T15:24:55.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.550+0000 7fb5254ce700 10 client.49783.objecter _send_command 2 2024-04-14T15:24:55.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.550+0000 7fb5254ce700 1 -- 172.21.15.90:0/1951774438 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fb5000079f0 con 0x7fb500004730 2024-04-14T15:24:55.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.552+0000 7fb517fff700 1 -- 172.21.15.90:0/1951774438 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fb5000079f0 con 0x7fb500004730 2024-04-14T15:24:55.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.552+0000 7fb517fff700 10 client.49783.objecter ms_dispatch 0x7fb520001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:24:55.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.552+0000 7fb517fff700 10 client.49783.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:24:55.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.552+0000 7fb517fff700 20 client.49783.objecter put_session s=0x7fb5000053e0 osd=0 4 2024-04-14T15:24:55.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.552+0000 7fb517fff700 15 client.49783.objecter _session_command_op_remove 0 2 2024-04-14T15:24:55.552 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:24:55.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.552+0000 7fb5254ce700 10 client.49783.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:24:55.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.552+0000 7fb5254ce700 15 client.49783.objecter _session_command_op_assign -1 3 2024-04-14T15:24:55.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.552+0000 7fb5254ce700 20 client.49783.objecter _get_session s=0x7fb500004110 osd=1 3 2024-04-14T15:24:55.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.552+0000 7fb5254ce700 20 client.49783.objecter put_session s=0x7fb500004110 osd=1 3 2024-04-14T15:24:55.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.552+0000 7fb5254ce700 20 client.49783.objecter _get_session s=0x7fb500004110 osd=1 3 2024-04-14T15:24:55.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.552+0000 7fb5254ce700 15 client.49783.objecter _session_command_op_remove -1 3 2024-04-14T15:24:55.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.552+0000 7fb5254ce700 20 client.49783.objecter get_session s=0x7fb500004110 osd=1 3 2024-04-14T15:24:55.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.552+0000 7fb5254ce700 15 client.49783.objecter _session_command_op_assign 1 3 2024-04-14T15:24:55.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.552+0000 7fb5254ce700 20 client.49783.objecter put_session s=0x7fb500004110 osd=1 4 2024-04-14T15:24:55.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.552+0000 7fb5254ce700 10 client.49783.objecter _send_command 3 2024-04-14T15:24:55.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.552+0000 7fb5254ce700 1 -- 172.21.15.90:0/1951774438 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fb500002180 con 0x7fb500004280 2024-04-14T15:24:55.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.552+0000 7fb517fff700 10 client.49783.objecter ms_handle_connect 0x7fb500004280 2024-04-14T15:24:55.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.553+0000 7fb517fff700 1 -- 172.21.15.90:0/1951774438 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fb500002180 con 0x7fb500004280 2024-04-14T15:24:55.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.553+0000 7fb517fff700 10 client.49783.objecter ms_dispatch 0x7fb520001550 command_reply(tid 3: 0 ) 2024-04-14T15:24:55.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.553+0000 7fb517fff700 10 client.49783.objecter _finish_command 3 = system:0 2024-04-14T15:24:55.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.553+0000 7fb517fff700 20 client.49783.objecter put_session s=0x7fb500004110 osd=1 4 2024-04-14T15:24:55.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.553+0000 7fb517fff700 15 client.49783.objecter _session_command_op_remove 1 3 2024-04-14T15:24:55.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.569+0000 7fb5254ce700 10 client.49783.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:24:55.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.569+0000 7fb5254ce700 15 client.49783.objecter _session_command_op_assign -1 4 2024-04-14T15:24:55.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.569+0000 7fb5254ce700 20 client.49783.objecter _get_session s=0x7fb500004110 osd=1 3 2024-04-14T15:24:55.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.569+0000 7fb5254ce700 20 client.49783.objecter put_session s=0x7fb500004110 osd=1 3 2024-04-14T15:24:55.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.569+0000 7fb5254ce700 20 client.49783.objecter _get_session s=0x7fb500004110 osd=1 3 2024-04-14T15:24:55.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.569+0000 7fb5254ce700 15 client.49783.objecter _session_command_op_remove -1 4 2024-04-14T15:24:55.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.569+0000 7fb5254ce700 20 client.49783.objecter get_session s=0x7fb500004110 osd=1 3 2024-04-14T15:24:55.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.569+0000 7fb5254ce700 15 client.49783.objecter _session_command_op_assign 1 4 2024-04-14T15:24:55.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.569+0000 7fb5254ce700 20 client.49783.objecter put_session s=0x7fb500004110 osd=1 4 2024-04-14T15:24:55.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.569+0000 7fb5254ce700 10 client.49783.objecter _send_command 4 2024-04-14T15:24:55.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.569+0000 7fb5254ce700 1 -- 172.21.15.90:0/1951774438 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fb500001600 con 0x7fb500004280 2024-04-14T15:24:55.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.571+0000 7fb517fff700 1 -- 172.21.15.90:0/1951774438 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fb500001600 con 0x7fb500004280 2024-04-14T15:24:55.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.571+0000 7fb517fff700 10 client.49783.objecter ms_dispatch 0x7fb520001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:24:55.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.571+0000 7fb517fff700 10 client.49783.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:24:55.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.571+0000 7fb517fff700 20 client.49783.objecter put_session s=0x7fb500004110 osd=1 4 2024-04-14T15:24:55.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.571+0000 7fb517fff700 15 client.49783.objecter _session_command_op_remove 1 4 2024-04-14T15:24:55.572 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:24:55.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.571+0000 7fb5254ce700 10 client.49783.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:24:55.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.571+0000 7fb5254ce700 15 client.49783.objecter _session_command_op_assign -1 5 2024-04-14T15:24:55.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.571+0000 7fb5254ce700 20 client.49783.objecter _get_session s=0x7fb500001a00 osd=2 3 2024-04-14T15:24:55.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.571+0000 7fb5254ce700 20 client.49783.objecter put_session s=0x7fb500001a00 osd=2 3 2024-04-14T15:24:55.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.571+0000 7fb5254ce700 20 client.49783.objecter _get_session s=0x7fb500001a00 osd=2 3 2024-04-14T15:24:55.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.571+0000 7fb5254ce700 15 client.49783.objecter _session_command_op_remove -1 5 2024-04-14T15:24:55.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.571+0000 7fb5254ce700 20 client.49783.objecter get_session s=0x7fb500001a00 osd=2 3 2024-04-14T15:24:55.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.571+0000 7fb5254ce700 15 client.49783.objecter _session_command_op_assign 2 5 2024-04-14T15:24:55.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.571+0000 7fb5254ce700 20 client.49783.objecter put_session s=0x7fb500001a00 osd=2 4 2024-04-14T15:24:55.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.571+0000 7fb5254ce700 10 client.49783.objecter _send_command 5 2024-04-14T15:24:55.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.571+0000 7fb5254ce700 1 -- 172.21.15.90:0/1951774438 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fb500010f20 con 0x7fb500008880 2024-04-14T15:24:55.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.571+0000 7fb517fff700 10 client.49783.objecter ms_handle_connect 0x7fb500008880 2024-04-14T15:24:55.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.572+0000 7fb517fff700 1 -- 172.21.15.90:0/1951774438 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fb500010f20 con 0x7fb500008880 2024-04-14T15:24:55.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.572+0000 7fb517fff700 10 client.49783.objecter ms_dispatch 0x7fb520001550 command_reply(tid 5: 0 ) 2024-04-14T15:24:55.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.572+0000 7fb517fff700 10 client.49783.objecter _finish_command 5 = system:0 2024-04-14T15:24:55.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.572+0000 7fb517fff700 20 client.49783.objecter put_session s=0x7fb500001a00 osd=2 4 2024-04-14T15:24:55.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.572+0000 7fb517fff700 15 client.49783.objecter _session_command_op_remove 2 5 2024-04-14T15:24:55.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.596+0000 7fb5254ce700 10 client.49783.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:24:55.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.596+0000 7fb5254ce700 15 client.49783.objecter _session_command_op_assign -1 6 2024-04-14T15:24:55.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.596+0000 7fb5254ce700 20 client.49783.objecter _get_session s=0x7fb500001a00 osd=2 3 2024-04-14T15:24:55.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.596+0000 7fb5254ce700 20 client.49783.objecter put_session s=0x7fb500001a00 osd=2 3 2024-04-14T15:24:55.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.596+0000 7fb5254ce700 20 client.49783.objecter _get_session s=0x7fb500001a00 osd=2 3 2024-04-14T15:24:55.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.596+0000 7fb5254ce700 15 client.49783.objecter _session_command_op_remove -1 6 2024-04-14T15:24:55.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.596+0000 7fb5254ce700 20 client.49783.objecter get_session s=0x7fb500001a00 osd=2 3 2024-04-14T15:24:55.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.596+0000 7fb5254ce700 15 client.49783.objecter _session_command_op_assign 2 6 2024-04-14T15:24:55.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.596+0000 7fb5254ce700 20 client.49783.objecter put_session s=0x7fb500001a00 osd=2 4 2024-04-14T15:24:55.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.596+0000 7fb5254ce700 10 client.49783.objecter _send_command 6 2024-04-14T15:24:55.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.596+0000 7fb5254ce700 1 -- 172.21.15.90:0/1951774438 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fb500007f10 con 0x7fb500008880 2024-04-14T15:24:55.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.597+0000 7fb517fff700 1 -- 172.21.15.90:0/1951774438 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fb500007f10 con 0x7fb500008880 2024-04-14T15:24:55.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.598+0000 7fb517fff700 10 client.49783.objecter ms_dispatch 0x7fb520001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:24:55.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.598+0000 7fb517fff700 10 client.49783.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:24:55.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.598+0000 7fb517fff700 20 client.49783.objecter put_session s=0x7fb500001a00 osd=2 4 2024-04-14T15:24:55.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.598+0000 7fb517fff700 15 client.49783.objecter _session_command_op_remove 2 6 2024-04-14T15:24:55.599 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:24:55.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.598+0000 7fb5254ce700 10 client.49783.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:24:55.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.598+0000 7fb5254ce700 15 client.49783.objecter _session_command_op_assign -1 7 2024-04-14T15:24:55.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.598+0000 7fb5254ce700 20 client.49783.objecter _get_session s=0x7fb5000081a0 osd=3 3 2024-04-14T15:24:55.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.598+0000 7fb5254ce700 20 client.49783.objecter put_session s=0x7fb5000081a0 osd=3 3 2024-04-14T15:24:55.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.598+0000 7fb5254ce700 20 client.49783.objecter _get_session s=0x7fb5000081a0 osd=3 3 2024-04-14T15:24:55.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.598+0000 7fb5254ce700 15 client.49783.objecter _session_command_op_remove -1 7 2024-04-14T15:24:55.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.598+0000 7fb5254ce700 20 client.49783.objecter get_session s=0x7fb5000081a0 osd=3 3 2024-04-14T15:24:55.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.598+0000 7fb5254ce700 15 client.49783.objecter _session_command_op_assign 3 7 2024-04-14T15:24:55.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.598+0000 7fb5254ce700 20 client.49783.objecter put_session s=0x7fb5000081a0 osd=3 4 2024-04-14T15:24:55.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.598+0000 7fb5254ce700 10 client.49783.objecter _send_command 7 2024-04-14T15:24:55.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.598+0000 7fb5254ce700 1 -- 172.21.15.90:0/1951774438 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fb500011c70 con 0x7fb500011100 2024-04-14T15:24:55.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.598+0000 7fb517fff700 10 client.49783.objecter ms_handle_connect 0x7fb500011100 2024-04-14T15:24:55.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.599+0000 7fb517fff700 1 -- 172.21.15.90:0/1951774438 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fb500011c70 con 0x7fb500011100 2024-04-14T15:24:55.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.599+0000 7fb517fff700 10 client.49783.objecter ms_dispatch 0x7fb520001550 command_reply(tid 7: 0 ) 2024-04-14T15:24:55.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.599+0000 7fb517fff700 10 client.49783.objecter _finish_command 7 = system:0 2024-04-14T15:24:55.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.599+0000 7fb517fff700 20 client.49783.objecter put_session s=0x7fb5000081a0 osd=3 4 2024-04-14T15:24:55.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.599+0000 7fb517fff700 15 client.49783.objecter _session_command_op_remove 3 7 2024-04-14T15:24:55.605 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:24:55.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.617+0000 7fb5254ce700 10 client.49783.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:24:55.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.617+0000 7fb5254ce700 15 client.49783.objecter _session_command_op_assign -1 8 2024-04-14T15:24:55.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.617+0000 7fb5254ce700 20 client.49783.objecter _get_session s=0x7fb5000081a0 osd=3 3 2024-04-14T15:24:55.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.617+0000 7fb5254ce700 20 client.49783.objecter put_session s=0x7fb5000081a0 osd=3 3 2024-04-14T15:24:55.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.617+0000 7fb5254ce700 20 client.49783.objecter _get_session s=0x7fb5000081a0 osd=3 3 2024-04-14T15:24:55.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.617+0000 7fb5254ce700 15 client.49783.objecter _session_command_op_remove -1 8 2024-04-14T15:24:55.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.617+0000 7fb5254ce700 20 client.49783.objecter get_session s=0x7fb5000081a0 osd=3 3 2024-04-14T15:24:55.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.617+0000 7fb5254ce700 15 client.49783.objecter _session_command_op_assign 3 8 2024-04-14T15:24:55.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.617+0000 7fb5254ce700 20 client.49783.objecter put_session s=0x7fb5000081a0 osd=3 4 2024-04-14T15:24:55.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.617+0000 7fb5254ce700 10 client.49783.objecter _send_command 8 2024-04-14T15:24:55.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.617+0000 7fb5254ce700 1 -- 172.21.15.90:0/1951774438 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fb500011c70 con 0x7fb500011100 2024-04-14T15:24:55.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.618+0000 7fb517fff700 1 -- 172.21.15.90:0/1951774438 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fb500011c70 con 0x7fb500011100 2024-04-14T15:24:55.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.618+0000 7fb517fff700 10 client.49783.objecter ms_dispatch 0x7fb520001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:24:55.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.619+0000 7fb517fff700 10 client.49783.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:24:55.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.619+0000 7fb517fff700 20 client.49783.objecter put_session s=0x7fb5000081a0 osd=3 4 2024-04-14T15:24:55.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.619+0000 7fb517fff700 15 client.49783.objecter _session_command_op_remove 3 8 2024-04-14T15:24:55.619 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:24:55.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.619+0000 7fb5254ce700 10 client.49783.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:24:55.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.619+0000 7fb5254ce700 15 client.49783.objecter _session_command_op_assign -1 9 2024-04-14T15:24:55.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.619+0000 7fb5254ce700 20 client.49783.objecter _get_session s=0x7fb500013880 osd=4 3 2024-04-14T15:24:55.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.619+0000 7fb5254ce700 20 client.49783.objecter put_session s=0x7fb500013880 osd=4 3 2024-04-14T15:24:55.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.619+0000 7fb5254ce700 20 client.49783.objecter _get_session s=0x7fb500013880 osd=4 3 2024-04-14T15:24:55.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.619+0000 7fb5254ce700 15 client.49783.objecter _session_command_op_remove -1 9 2024-04-14T15:24:55.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.619+0000 7fb5254ce700 20 client.49783.objecter get_session s=0x7fb500013880 osd=4 3 2024-04-14T15:24:55.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.619+0000 7fb5254ce700 15 client.49783.objecter _session_command_op_assign 4 9 2024-04-14T15:24:55.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.619+0000 7fb5254ce700 20 client.49783.objecter put_session s=0x7fb500013880 osd=4 4 2024-04-14T15:24:55.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.619+0000 7fb5254ce700 10 client.49783.objecter _send_command 9 2024-04-14T15:24:55.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.619+0000 7fb5254ce700 1 -- 172.21.15.90:0/1951774438 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fb50001a5f0 con 0x7fb500016ae0 2024-04-14T15:24:55.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.619+0000 7fb517fff700 10 client.49783.objecter ms_handle_connect 0x7fb500016ae0 2024-04-14T15:24:55.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.620+0000 7fb517fff700 1 -- 172.21.15.90:0/1951774438 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fb50001a5f0 con 0x7fb500016ae0 2024-04-14T15:24:55.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.620+0000 7fb517fff700 10 client.49783.objecter ms_dispatch 0x7fb520001550 command_reply(tid 9: 0 ) 2024-04-14T15:24:55.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.620+0000 7fb517fff700 10 client.49783.objecter _finish_command 9 = system:0 2024-04-14T15:24:55.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.620+0000 7fb517fff700 20 client.49783.objecter put_session s=0x7fb500013880 osd=4 4 2024-04-14T15:24:55.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.620+0000 7fb517fff700 15 client.49783.objecter _session_command_op_remove 4 9 2024-04-14T15:24:55.629 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:55.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:55.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:24:55.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.638+0000 7fb5254ce700 10 client.49783.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:24:55.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.638+0000 7fb5254ce700 15 client.49783.objecter _session_command_op_assign -1 10 2024-04-14T15:24:55.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.638+0000 7fb5254ce700 20 client.49783.objecter _get_session s=0x7fb500013880 osd=4 3 2024-04-14T15:24:55.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.638+0000 7fb5254ce700 20 client.49783.objecter put_session s=0x7fb500013880 osd=4 3 2024-04-14T15:24:55.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.638+0000 7fb5254ce700 20 client.49783.objecter _get_session s=0x7fb500013880 osd=4 3 2024-04-14T15:24:55.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.638+0000 7fb5254ce700 15 client.49783.objecter _session_command_op_remove -1 10 2024-04-14T15:24:55.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.638+0000 7fb5254ce700 20 client.49783.objecter get_session s=0x7fb500013880 osd=4 3 2024-04-14T15:24:55.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.638+0000 7fb5254ce700 15 client.49783.objecter _session_command_op_assign 4 10 2024-04-14T15:24:55.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.638+0000 7fb5254ce700 20 client.49783.objecter put_session s=0x7fb500013880 osd=4 4 2024-04-14T15:24:55.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.638+0000 7fb5254ce700 10 client.49783.objecter _send_command 10 2024-04-14T15:24:55.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.638+0000 7fb5254ce700 1 -- 172.21.15.90:0/1951774438 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fb50000c700 con 0x7fb500016ae0 2024-04-14T15:24:55.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.640+0000 7fb517fff700 1 -- 172.21.15.90:0/1951774438 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fb50000c700 con 0x7fb500016ae0 2024-04-14T15:24:55.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.640+0000 7fb517fff700 10 client.49783.objecter ms_dispatch 0x7fb520001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:24:55.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.640+0000 7fb517fff700 10 client.49783.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:24:55.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.640+0000 7fb517fff700 20 client.49783.objecter put_session s=0x7fb500013880 osd=4 4 2024-04-14T15:24:55.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.640+0000 7fb517fff700 15 client.49783.objecter _session_command_op_remove 4 10 2024-04-14T15:24:55.641 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:24:55.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.640+0000 7fb5254ce700 10 client.49783.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:24:55.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.640+0000 7fb5254ce700 15 client.49783.objecter _session_command_op_assign -1 11 2024-04-14T15:24:55.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.640+0000 7fb5254ce700 20 client.49783.objecter _get_session s=0x7fb50000c700 osd=5 3 2024-04-14T15:24:55.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.640+0000 7fb5254ce700 20 client.49783.objecter put_session s=0x7fb50000c700 osd=5 3 2024-04-14T15:24:55.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.640+0000 7fb5254ce700 20 client.49783.objecter _get_session s=0x7fb50000c700 osd=5 3 2024-04-14T15:24:55.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.640+0000 7fb5254ce700 15 client.49783.objecter _session_command_op_remove -1 11 2024-04-14T15:24:55.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.640+0000 7fb5254ce700 20 client.49783.objecter get_session s=0x7fb50000c700 osd=5 3 2024-04-14T15:24:55.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.640+0000 7fb5254ce700 15 client.49783.objecter _session_command_op_assign 5 11 2024-04-14T15:24:55.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.640+0000 7fb5254ce700 20 client.49783.objecter put_session s=0x7fb50000c700 osd=5 4 2024-04-14T15:24:55.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.640+0000 7fb5254ce700 10 client.49783.objecter _send_command 11 2024-04-14T15:24:55.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.640+0000 7fb5254ce700 1 -- 172.21.15.90:0/1951774438 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fb50001bbb0 con 0x7fb50001b770 2024-04-14T15:24:55.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.641+0000 7fb517fff700 10 client.49783.objecter ms_handle_connect 0x7fb50001b770 2024-04-14T15:24:55.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.642+0000 7fb517fff700 1 -- 172.21.15.90:0/1951774438 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fb50001bbb0 con 0x7fb50001b770 2024-04-14T15:24:55.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.642+0000 7fb517fff700 10 client.49783.objecter ms_dispatch 0x7fb520001550 command_reply(tid 11: 0 ) 2024-04-14T15:24:55.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.642+0000 7fb517fff700 10 client.49783.objecter _finish_command 11 = system:0 2024-04-14T15:24:55.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.642+0000 7fb517fff700 20 client.49783.objecter put_session s=0x7fb50000c700 osd=5 4 2024-04-14T15:24:55.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.642+0000 7fb517fff700 15 client.49783.objecter _session_command_op_remove 5 11 2024-04-14T15:24:55.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.658+0000 7fb5254ce700 10 client.49783.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:24:55.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.658+0000 7fb5254ce700 15 client.49783.objecter _session_command_op_assign -1 12 2024-04-14T15:24:55.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.658+0000 7fb5254ce700 20 client.49783.objecter _get_session s=0x7fb50000c700 osd=5 3 2024-04-14T15:24:55.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.658+0000 7fb5254ce700 20 client.49783.objecter put_session s=0x7fb50000c700 osd=5 3 2024-04-14T15:24:55.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.658+0000 7fb5254ce700 20 client.49783.objecter _get_session s=0x7fb50000c700 osd=5 3 2024-04-14T15:24:55.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.658+0000 7fb5254ce700 15 client.49783.objecter _session_command_op_remove -1 12 2024-04-14T15:24:55.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.658+0000 7fb5254ce700 20 client.49783.objecter get_session s=0x7fb50000c700 osd=5 3 2024-04-14T15:24:55.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.658+0000 7fb5254ce700 15 client.49783.objecter _session_command_op_assign 5 12 2024-04-14T15:24:55.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.658+0000 7fb5254ce700 20 client.49783.objecter put_session s=0x7fb50000c700 osd=5 4 2024-04-14T15:24:55.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.658+0000 7fb5254ce700 10 client.49783.objecter _send_command 12 2024-04-14T15:24:55.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.658+0000 7fb5254ce700 1 -- 172.21.15.90:0/1951774438 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fb50001bd90 con 0x7fb50001b770 2024-04-14T15:24:55.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.660+0000 7fb517fff700 1 -- 172.21.15.90:0/1951774438 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fb50001bd90 con 0x7fb50001b770 2024-04-14T15:24:55.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.660+0000 7fb517fff700 10 client.49783.objecter ms_dispatch 0x7fb520001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:24:55.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.660+0000 7fb517fff700 10 client.49783.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:24:55.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.660+0000 7fb517fff700 20 client.49783.objecter put_session s=0x7fb50000c700 osd=5 4 2024-04-14T15:24:55.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.660+0000 7fb517fff700 15 client.49783.objecter _session_command_op_remove 5 12 2024-04-14T15:24:55.661 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:24:55.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.660+0000 7fb5254ce700 10 client.49783.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:24:55.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.660+0000 7fb5254ce700 15 client.49783.objecter _session_command_op_assign -1 13 2024-04-14T15:24:55.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.660+0000 7fb5254ce700 20 client.49783.objecter _get_session s=0x7fb50001bbb0 osd=6 3 2024-04-14T15:24:55.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.660+0000 7fb5254ce700 20 client.49783.objecter put_session s=0x7fb50001bbb0 osd=6 3 2024-04-14T15:24:55.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.660+0000 7fb5254ce700 20 client.49783.objecter _get_session s=0x7fb50001bbb0 osd=6 3 2024-04-14T15:24:55.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.660+0000 7fb5254ce700 15 client.49783.objecter _session_command_op_remove -1 13 2024-04-14T15:24:55.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.660+0000 7fb5254ce700 20 client.49783.objecter get_session s=0x7fb50001bbb0 osd=6 3 2024-04-14T15:24:55.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.660+0000 7fb5254ce700 15 client.49783.objecter _session_command_op_assign 6 13 2024-04-14T15:24:55.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.660+0000 7fb5254ce700 20 client.49783.objecter put_session s=0x7fb50001bbb0 osd=6 4 2024-04-14T15:24:55.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.660+0000 7fb5254ce700 10 client.49783.objecter _send_command 13 2024-04-14T15:24:55.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.660+0000 7fb5254ce700 1 -- 172.21.15.90:0/1951774438 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fb500022fb0 con 0x7fb50001f4a0 2024-04-14T15:24:55.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.660+0000 7fb517fff700 10 client.49783.objecter ms_handle_connect 0x7fb50001f4a0 2024-04-14T15:24:55.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.661+0000 7fb517fff700 1 -- 172.21.15.90:0/1951774438 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fb500022fb0 con 0x7fb50001f4a0 2024-04-14T15:24:55.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.661+0000 7fb517fff700 10 client.49783.objecter ms_dispatch 0x7fb520001550 command_reply(tid 13: 0 ) 2024-04-14T15:24:55.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.661+0000 7fb517fff700 10 client.49783.objecter _finish_command 13 = system:0 2024-04-14T15:24:55.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.661+0000 7fb517fff700 20 client.49783.objecter put_session s=0x7fb50001bbb0 osd=6 4 2024-04-14T15:24:55.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.661+0000 7fb517fff700 15 client.49783.objecter _session_command_op_remove 6 13 2024-04-14T15:24:55.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.678+0000 7fb5254ce700 10 client.49783.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:24:55.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.678+0000 7fb5254ce700 15 client.49783.objecter _session_command_op_assign -1 14 2024-04-14T15:24:55.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.678+0000 7fb5254ce700 20 client.49783.objecter _get_session s=0x7fb50001bbb0 osd=6 3 2024-04-14T15:24:55.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.678+0000 7fb5254ce700 20 client.49783.objecter put_session s=0x7fb50001bbb0 osd=6 3 2024-04-14T15:24:55.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.678+0000 7fb5254ce700 20 client.49783.objecter _get_session s=0x7fb50001bbb0 osd=6 3 2024-04-14T15:24:55.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.678+0000 7fb5254ce700 15 client.49783.objecter _session_command_op_remove -1 14 2024-04-14T15:24:55.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.678+0000 7fb5254ce700 20 client.49783.objecter get_session s=0x7fb50001bbb0 osd=6 3 2024-04-14T15:24:55.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.678+0000 7fb5254ce700 15 client.49783.objecter _session_command_op_assign 6 14 2024-04-14T15:24:55.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.678+0000 7fb5254ce700 20 client.49783.objecter put_session s=0x7fb50001bbb0 osd=6 4 2024-04-14T15:24:55.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.678+0000 7fb5254ce700 10 client.49783.objecter _send_command 14 2024-04-14T15:24:55.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.678+0000 7fb5254ce700 1 -- 172.21.15.90:0/1951774438 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fb500023190 con 0x7fb50001f4a0 2024-04-14T15:24:55.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.679+0000 7fb517fff700 1 -- 172.21.15.90:0/1951774438 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fb500023190 con 0x7fb50001f4a0 2024-04-14T15:24:55.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.680+0000 7fb517fff700 10 client.49783.objecter ms_dispatch 0x7fb520001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:24:55.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.680+0000 7fb517fff700 10 client.49783.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:24:55.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.680+0000 7fb517fff700 20 client.49783.objecter put_session s=0x7fb50001bbb0 osd=6 4 2024-04-14T15:24:55.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.680+0000 7fb517fff700 15 client.49783.objecter _session_command_op_remove 6 14 2024-04-14T15:24:55.681 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:24:55.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.680+0000 7fb5254ce700 10 client.49783.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:24:55.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.680+0000 7fb5254ce700 15 client.49783.objecter _session_command_op_assign -1 15 2024-04-14T15:24:55.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.680+0000 7fb5254ce700 20 client.49783.objecter _get_session s=0x7fb500023190 osd=7 3 2024-04-14T15:24:55.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.680+0000 7fb5254ce700 20 client.49783.objecter put_session s=0x7fb500023190 osd=7 3 2024-04-14T15:24:55.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.680+0000 7fb5254ce700 20 client.49783.objecter _get_session s=0x7fb500023190 osd=7 3 2024-04-14T15:24:55.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.680+0000 7fb5254ce700 15 client.49783.objecter _session_command_op_remove -1 15 2024-04-14T15:24:55.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.680+0000 7fb5254ce700 20 client.49783.objecter get_session s=0x7fb500023190 osd=7 3 2024-04-14T15:24:55.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.680+0000 7fb5254ce700 15 client.49783.objecter _session_command_op_assign 7 15 2024-04-14T15:24:55.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.680+0000 7fb5254ce700 20 client.49783.objecter put_session s=0x7fb500023190 osd=7 4 2024-04-14T15:24:55.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.680+0000 7fb5254ce700 10 client.49783.objecter _send_command 15 2024-04-14T15:24:55.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.680+0000 7fb5254ce700 1 -- 172.21.15.90:0/1951774438 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fb500024490 con 0x7fb5000238b0 2024-04-14T15:24:55.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.680+0000 7fb517fff700 10 client.49783.objecter ms_handle_connect 0x7fb5000238b0 2024-04-14T15:24:55.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.681+0000 7fb517fff700 1 -- 172.21.15.90:0/1951774438 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fb500024490 con 0x7fb5000238b0 2024-04-14T15:24:55.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.681+0000 7fb517fff700 10 client.49783.objecter ms_dispatch 0x7fb520001550 command_reply(tid 15: 0 ) 2024-04-14T15:24:55.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.681+0000 7fb517fff700 10 client.49783.objecter _finish_command 15 = system:0 2024-04-14T15:24:55.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.681+0000 7fb517fff700 20 client.49783.objecter put_session s=0x7fb500023190 osd=7 4 2024-04-14T15:24:55.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.681+0000 7fb517fff700 15 client.49783.objecter _session_command_op_remove 7 15 2024-04-14T15:24:55.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.697+0000 7fb5254ce700 10 client.49783.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:24:55.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.697+0000 7fb5254ce700 15 client.49783.objecter _session_command_op_assign -1 16 2024-04-14T15:24:55.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.697+0000 7fb5254ce700 20 client.49783.objecter _get_session s=0x7fb500023190 osd=7 3 2024-04-14T15:24:55.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.697+0000 7fb5254ce700 20 client.49783.objecter put_session s=0x7fb500023190 osd=7 3 2024-04-14T15:24:55.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.697+0000 7fb5254ce700 20 client.49783.objecter _get_session s=0x7fb500023190 osd=7 3 2024-04-14T15:24:55.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.697+0000 7fb5254ce700 15 client.49783.objecter _session_command_op_remove -1 16 2024-04-14T15:24:55.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.697+0000 7fb5254ce700 20 client.49783.objecter get_session s=0x7fb500023190 osd=7 3 2024-04-14T15:24:55.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.697+0000 7fb5254ce700 15 client.49783.objecter _session_command_op_assign 7 16 2024-04-14T15:24:55.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.697+0000 7fb5254ce700 20 client.49783.objecter put_session s=0x7fb500023190 osd=7 4 2024-04-14T15:24:55.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.697+0000 7fb5254ce700 10 client.49783.objecter _send_command 16 2024-04-14T15:24:55.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.697+0000 7fb5254ce700 1 -- 172.21.15.90:0/1951774438 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fb500024490 con 0x7fb5000238b0 2024-04-14T15:24:55.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.699+0000 7fb517fff700 1 -- 172.21.15.90:0/1951774438 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fb500024490 con 0x7fb5000238b0 2024-04-14T15:24:55.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.699+0000 7fb517fff700 10 client.49783.objecter ms_dispatch 0x7fb520001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:24:55.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.699+0000 7fb517fff700 10 client.49783.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:24:55.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.699+0000 7fb517fff700 20 client.49783.objecter put_session s=0x7fb500023190 osd=7 4 2024-04-14T15:24:55.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.699+0000 7fb517fff700 15 client.49783.objecter _session_command_op_remove 7 16 2024-04-14T15:24:55.700 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:24:55.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.699+0000 7fb5254ce700 10 librados: watch_flush enter 2024-04-14T15:24:55.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.699+0000 7fb5254ce700 10 librados: watch_flush exit 2024-04-14T15:24:55.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.699+0000 7fb5254ce700 10 client.49783.objecter close_session for osd.0 2024-04-14T15:24:55.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.699+0000 7fb5254ce700 1 -- 172.21.15.90:0/1951774438 >> v1:172.21.15.90:6801/3761806885 conn(0x7fb500004730 legacy=0x7fb500002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:55.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.699+0000 7fb5254ce700 20 client.49783.objecter put_session s=0x7fb5000053e0 osd=0 1 2024-04-14T15:24:55.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.699+0000 7fb5254ce700 10 client.49783.objecter close_session for osd.1 2024-04-14T15:24:55.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.699+0000 7fb5254ce700 1 -- 172.21.15.90:0/1951774438 >> v1:172.21.15.90:6805/4170361699 conn(0x7fb500004280 legacy=0x7fb50000ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:55.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.699+0000 7fb5254ce700 20 client.49783.objecter put_session s=0x7fb500004110 osd=1 1 2024-04-14T15:24:55.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.699+0000 7fb5254ce700 10 client.49783.objecter close_session for osd.2 2024-04-14T15:24:55.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.699+0000 7fb5254ce700 1 -- 172.21.15.90:0/1951774438 >> v1:172.21.15.90:6809/1686184251 conn(0x7fb500008880 legacy=0x7fb50000f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:55.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.699+0000 7fb5254ce700 20 client.49783.objecter put_session s=0x7fb500001a00 osd=2 1 2024-04-14T15:24:55.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.699+0000 7fb5254ce700 10 client.49783.objecter close_session for osd.3 2024-04-14T15:24:55.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.699+0000 7fb5254ce700 1 -- 172.21.15.90:0/1951774438 >> v1:172.21.15.90:6813/2270166774 conn(0x7fb500011100 legacy=0x7fb500011560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:55.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.699+0000 7fb5254ce700 20 client.49783.objecter put_session s=0x7fb5000081a0 osd=3 1 2024-04-14T15:24:55.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.699+0000 7fb5254ce700 10 client.49783.objecter close_session for osd.4 2024-04-14T15:24:55.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.699+0000 7fb5254ce700 1 -- 172.21.15.90:0/1951774438 >> v1:172.21.15.92:6800/3887549733 conn(0x7fb500016ae0 legacy=0x7fb500018ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:55.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.699+0000 7fb5254ce700 20 client.49783.objecter put_session s=0x7fb500013880 osd=4 1 2024-04-14T15:24:55.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.699+0000 7fb5254ce700 10 client.49783.objecter close_session for osd.5 2024-04-14T15:24:55.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.699+0000 7fb5254ce700 1 -- 172.21.15.90:0/1951774438 >> v1:172.21.15.92:6804/3244561440 conn(0x7fb50001b770 legacy=0x7fb500012ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:55.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.699+0000 7fb51f7fe700 1 -- 172.21.15.90:0/1951774438 reap_dead start 2024-04-14T15:24:55.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.699+0000 7fb5254ce700 20 client.49783.objecter put_session s=0x7fb50000c700 osd=5 1 2024-04-14T15:24:55.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.699+0000 7fb5254ce700 10 client.49783.objecter close_session for osd.6 2024-04-14T15:24:55.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.699+0000 7fb5254ce700 1 -- 172.21.15.90:0/1951774438 >> v1:172.21.15.92:6808/636125963 conn(0x7fb50001f4a0 legacy=0x7fb500021890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:55.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.699+0000 7fb51f7fe700 1 -- 172.21.15.90:0/1951774438 reap_dead start 2024-04-14T15:24:55.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.699+0000 7fb5254ce700 20 client.49783.objecter put_session s=0x7fb50001bbb0 osd=6 1 2024-04-14T15:24:55.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.699+0000 7fb5254ce700 10 client.49783.objecter close_session for osd.7 2024-04-14T15:24:55.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.699+0000 7fb5254ce700 1 -- 172.21.15.90:0/1951774438 >> v1:172.21.15.92:6812/1773706716 conn(0x7fb5000238b0 legacy=0x7fb500023d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:55.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.699+0000 7fb5254ce700 20 client.49783.objecter put_session s=0x7fb500023190 osd=7 1 2024-04-14T15:24:55.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.699+0000 7fb5254ce700 20 client.49783.objecter shutdown clearing up homeless session... 2024-04-14T15:24:55.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.699+0000 7fb5254ce700 10 client.49783.objecter successfully canceled tick 2024-04-14T15:24:55.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.699+0000 7fb51f7fe700 1 -- 172.21.15.90:0/1951774438 reap_dead start 2024-04-14T15:24:55.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.699+0000 7fb51f7fe700 1 -- 172.21.15.90:0/1951774438 reap_dead start 2024-04-14T15:24:55.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.699+0000 7fb5254ce700 1 -- 172.21.15.90:0/1951774438 >> v1:172.21.15.90:6800/420935427 conn(0x7fb50c071440 legacy=0x7fb50c073900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:55.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.700+0000 7fb5254ce700 1 -- 172.21.15.90:0/1951774438 >> v1:172.21.15.92:6789/0 conn(0x7fb5201077b0 legacy=0x7fb5200fd750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:24:55.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.700+0000 7fb5254ce700 1 -- 172.21.15.90:0/1951774438 shutdown_connections 2024-04-14T15:24:55.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.700+0000 7fb5254ce700 1 -- 172.21.15.90:0/1951774438 >> 172.21.15.90:0/1951774438 conn(0x7fb5200fd2a0 msgr2=0x7fb5201003b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:24:55.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.700+0000 7fb5254ce700 1 -- 172.21.15.90:0/1951774438 shutdown_connections 2024-04-14T15:24:55.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.700+0000 7fb5254ce700 1 -- 172.21.15.90:0/1951774438 wait complete. 2024-04-14T15:24:55.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:24:55.700+0000 7fb5254ce700 1 librados: shutdown 2024-04-14T15:24:55.703 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:24:55.703 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:24:55.703 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:24:55.703 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:24:55.703 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:24:55.703 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:24:55.703 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:24:55.704 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:24:55.731 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:24:55.800 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:55.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:55.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:24:55.872 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:24:55 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:24:55.902 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:24:55.927 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:55.927 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-14T15:24:55.927 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:24:55.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:55.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:24:56.029 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:24:56.053 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:56.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:56.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:24:56.155 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:24:56.179 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:56.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:56.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:24:56.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:56 smithi090 ceph-mon[30399]: osdmap e751: 8 total, 8 up, 7 in 2024-04-14T15:24:56.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:56 smithi090 ceph-mon[30399]: pgmap v1111: 214 pgs: 214 active+clean; 582 KiB data, 708 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-14T15:24:56.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:56 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-04-14T15:24:56.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:56 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:24:56.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:56 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1951774438' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:24:56.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:56 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-04-14T15:24:56.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:56 smithi090 ceph-mon[30399]: osdmap e752: 8 total, 8 up, 7 in 2024-04-14T15:24:56.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:56 smithi090 ceph-mon[41670]: osdmap e751: 8 total, 8 up, 7 in 2024-04-14T15:24:56.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:56 smithi090 ceph-mon[41670]: pgmap v1111: 214 pgs: 214 active+clean; 582 KiB data, 708 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-14T15:24:56.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:56 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-04-14T15:24:56.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:56 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:24:56.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:56 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1951774438' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:24:56.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:56 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-04-14T15:24:56.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:56 smithi090 ceph-mon[41670]: osdmap e752: 8 total, 8 up, 7 in 2024-04-14T15:24:56.281 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:24:56.305 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:56.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:56.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:24:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:56 smithi092 ceph-mon[33966]: osdmap e751: 8 total, 8 up, 7 in 2024-04-14T15:24:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:56 smithi092 ceph-mon[33966]: pgmap v1111: 214 pgs: 214 active+clean; 582 KiB data, 708 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-14T15:24:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:56 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-04-14T15:24:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:56 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:24:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:56 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1951774438' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:24:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:56 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-04-14T15:24:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:56 smithi092 ceph-mon[33966]: osdmap e752: 8 total, 8 up, 7 in 2024-04-14T15:24:56.407 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:24:56.431 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:56.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:56.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:24:56.532 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:24:56.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:56.557 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:56.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:24:56.658 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:24:56.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:56.683 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:56.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:24:56.704 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-14T15:24:56.784 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:24:56.808 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:56.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:56.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:24:56.910 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:24:56.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:56.934 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:56.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:24:57.035 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:24:57.058 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:57.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:57.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:24:57.160 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:24:57.183 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:57.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:57.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:24:57.285 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:24:57.308 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:57.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:57.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:24:57.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:57 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:24:57.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:57 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:24:57.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:57 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]': finished 2024-04-14T15:24:57.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:57 smithi092 ceph-mon[33966]: osdmap e753: 8 total, 8 up, 7 in 2024-04-14T15:24:57.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:57 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-132"}]: dispatch 2024-04-14T15:24:57.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:57 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-132"}]: dispatch 2024-04-14T15:24:57.410 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:24:57.433 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:57.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:57.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:24:57.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:57 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:24:57.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:57 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:24:57.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:57 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]': finished 2024-04-14T15:24:57.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:57 smithi090 ceph-mon[41670]: osdmap e753: 8 total, 8 up, 7 in 2024-04-14T15:24:57.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:57 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-132"}]: dispatch 2024-04-14T15:24:57.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:57 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-132"}]: dispatch 2024-04-14T15:24:57.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:57 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:24:57.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:57 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:24:57.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:57 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]': finished 2024-04-14T15:24:57.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:57 smithi090 ceph-mon[30399]: osdmap e753: 8 total, 8 up, 7 in 2024-04-14T15:24:57.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:57 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-132"}]: dispatch 2024-04-14T15:24:57.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:57 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-132"}]: dispatch 2024-04-14T15:24:57.535 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:24:57.558 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:57.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:57.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:24:57.575 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:24:57.661 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:24:57.684 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:57.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:57.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:24:57.785 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:24:57.807 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:24:57.809 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:57.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:57.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:24:57.911 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:24:57.935 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:57.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:57.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:24:58.037 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:24:58.060 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:58.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:58.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:24:58.162 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:24:58.185 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:58.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:58.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:24:58.287 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:24:58.310 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:58.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:58.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:24:58.340 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:58 smithi090 ceph-mon[41670]: pgmap v1114: 224 pgs: 10 unknown, 214 active+clean; 582 KiB data, 708 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-14T15:24:58.340 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:58 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-04-14T15:24:58.340 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:58 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-132"}]': finished 2024-04-14T15:24:58.340 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:58 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-04-14T15:24:58.340 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:24:58 smithi090 ceph-mon[41670]: osdmap e754: 8 total, 8 up, 7 in 2024-04-14T15:24:58.340 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:58 smithi090 ceph-mon[30399]: pgmap v1114: 224 pgs: 10 unknown, 214 active+clean; 582 KiB data, 708 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-14T15:24:58.340 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:58 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-04-14T15:24:58.340 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:58 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-132"}]': finished 2024-04-14T15:24:58.340 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:58 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-04-14T15:24:58.341 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:24:58 smithi090 ceph-mon[30399]: osdmap e754: 8 total, 8 up, 7 in 2024-04-14T15:24:58.371 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:58 smithi092 ceph-mon[33966]: pgmap v1114: 224 pgs: 10 unknown, 214 active+clean; 582 KiB data, 708 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-14T15:24:58.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:58 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-04-14T15:24:58.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:58 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-132"}]': finished 2024-04-14T15:24:58.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:58 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-04-14T15:24:58.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:24:58 smithi092 ceph-mon[33966]: osdmap e754: 8 total, 8 up, 7 in 2024-04-14T15:24:58.412 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:24:58.436 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:58.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:58.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:24:58.537 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:24:58.562 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:58.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:58.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:24:58.664 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:24:58.688 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:58.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:58.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:24:58.790 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:24:58.815 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:58.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:58.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:24:58.917 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:24:58.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:58.942 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:58.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:24:59.043 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:24:59.066 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:59.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:59.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:24:59.167 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:24:59.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:59.191 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:59.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:24:59.292 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:24:59.315 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:24:59.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:59.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:24:59.416 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:24:59.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:59.443 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:59.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:24:59.544 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-14T15:24:59.544 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:24:59.570 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:59.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:59.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:24:59.672 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:24:59.697 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:59.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:59.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:24:59.799 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:24:59.811 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-14T15:24:59.811 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd reweight 0 1 2024-04-14T15:24:59.824 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:59.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:59.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:24:59.926 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:24:59.951 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:24:59.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:24:59.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:25:00.052 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:25:00.075 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:00.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:00.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:25:00.100 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:00 smithi090 ceph-mon[41670]: osdmap e755: 8 total, 8 up, 7 in 2024-04-14T15:25:00.100 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:00 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:25:00.100 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:00 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:25:00.101 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:00 smithi090 ceph-mon[30399]: osdmap e755: 8 total, 8 up, 7 in 2024-04-14T15:25:00.101 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:00 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:25:00.101 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:00 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:25:00.177 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:00.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:00.202 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:00.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:00.302 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:00.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:00.327 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:00.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:00.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:00 smithi092 ceph-mon[33966]: osdmap e755: 8 total, 8 up, 7 in 2024-04-14T15:25:00.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:00 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:25:00.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:00 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:25:00.428 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:25:00.456 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:00.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:00.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:25:00.558 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:25:00.585 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:00.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:00.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:25:00.686 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:25:00.710 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:00.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:00.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:25:00.812 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:25:00.835 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:00.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:00.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:25:00.864 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:25:00.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:00.887+0000 7f8f4d7a3700 1 -- 172.21.15.90:0/158393049 >> v1:172.21.15.90:6790/0 conn(0x7f8f48103a40 legacy=0x7f8f48103e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:00.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:00.887+0000 7f8f4d7a3700 1 -- 172.21.15.90:0/158393049 shutdown_connections 2024-04-14T15:25:00.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:00.887+0000 7f8f4d7a3700 1 -- 172.21.15.90:0/158393049 >> 172.21.15.90:0/158393049 conn(0x7f8f480fd290 msgr2=0x7f8f480ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:25:00.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:00.887+0000 7f8f4d7a3700 1 -- 172.21.15.90:0/158393049 shutdown_connections 2024-04-14T15:25:00.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:00.887+0000 7f8f4d7a3700 1 -- 172.21.15.90:0/158393049 wait complete. 2024-04-14T15:25:00.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:00.888+0000 7f8f4d7a3700 1 librados: starting msgr at 2024-04-14T15:25:00.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:00.888+0000 7f8f4d7a3700 1 librados: starting objecter 2024-04-14T15:25:00.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:00.888+0000 7f8f4d7a3700 1 Processor -- start 2024-04-14T15:25:00.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:00.888+0000 7f8f4d7a3700 1 -- start start 2024-04-14T15:25:00.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:00.888+0000 7f8f4d7a3700 1 librados: setting wanted keys 2024-04-14T15:25:00.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:00.888+0000 7f8f4d7a3700 1 librados: calling monclient init 2024-04-14T15:25:00.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:00.888+0000 7f8f4d7a3700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f8f481a7900 con 0x7f8f48103a40 2024-04-14T15:25:00.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:00.888+0000 7f8f467fc700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f8f4810b930 0x7f8f481a2910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:57108/0 (socket says 172.21.15.90:57108) 2024-04-14T15:25:00.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:00.888+0000 7f8f467fc700 1 -- 172.21.15.90:0/96791544 learned_addr learned my addr 172.21.15.90:0/96791544 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:25:00.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:00.888+0000 7f8f4d7a3700 1 -- 172.21.15.90:0/96791544 --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f8f481a8b20 con 0x7f8f4810f4d0 2024-04-14T15:25:00.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:00.888+0000 7f8f4d7a3700 1 -- 172.21.15.90:0/96791544 --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f8f481a9d60 con 0x7f8f4810b930 2024-04-14T15:25:00.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:00.889+0000 7f8f2ffff700 10 client.?.objecter ms_handle_connect 0x7f8f4810b930 2024-04-14T15:25:00.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:00.889+0000 7f8f2ffff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:25:00.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:00.889+0000 7f8f2ffff700 10 client.?.objecter ms_handle_connect 0x7f8f48103a40 2024-04-14T15:25:00.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:00.889+0000 7f8f2ffff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:25:00.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:00.889+0000 7f8f2ffff700 10 client.?.objecter ms_handle_connect 0x7f8f4810f4d0 2024-04-14T15:25:00.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:00.889+0000 7f8f2ffff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:25:00.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:00.889+0000 7f8f2ffff700 1 -- 172.21.15.90:0/96791544 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2001526116 0 0) 0x7f8f481a9d60 con 0x7f8f4810b930 2024-04-14T15:25:00.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:00.889+0000 7f8f2ffff700 1 -- 172.21.15.90:0/96791544 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8f30003aa0 con 0x7f8f4810b930 2024-04-14T15:25:00.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:00.889+0000 7f8f2ffff700 1 -- 172.21.15.90:0/96791544 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 369377140 0 0) 0x7f8f481a8b20 con 0x7f8f4810f4d0 2024-04-14T15:25:00.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:00.889+0000 7f8f2ffff700 1 -- 172.21.15.90:0/96791544 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8f481a9d60 con 0x7f8f4810f4d0 2024-04-14T15:25:00.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:00.889+0000 7f8f2ffff700 1 -- 172.21.15.90:0/96791544 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3651725475 0 0) 0x7f8f481a7900 con 0x7f8f48103a40 2024-04-14T15:25:00.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:00.889+0000 7f8f2ffff700 1 -- 172.21.15.90:0/96791544 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8f481a8b20 con 0x7f8f48103a40 2024-04-14T15:25:00.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:00.889+0000 7f8f2ffff700 1 -- 172.21.15.90:0/96791544 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3785717336 0 0) 0x7f8f481a9d60 con 0x7f8f4810f4d0 2024-04-14T15:25:00.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:00.889+0000 7f8f2ffff700 1 -- 172.21.15.90:0/96791544 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8f481a7900 con 0x7f8f4810f4d0 2024-04-14T15:25:00.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:00.889+0000 7f8f2ffff700 1 -- 172.21.15.90:0/96791544 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f8f40001c10 con 0x7f8f4810f4d0 2024-04-14T15:25:00.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:00.889+0000 7f8f2ffff700 1 -- 172.21.15.90:0/96791544 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 128633664 0 0) 0x7f8f481a8b20 con 0x7f8f48103a40 2024-04-14T15:25:00.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:00.889+0000 7f8f2ffff700 1 -- 172.21.15.90:0/96791544 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8f481a9d60 con 0x7f8f48103a40 2024-04-14T15:25:00.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:00.889+0000 7f8f2ffff700 1 -- 172.21.15.90:0/96791544 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2452159947 0 0) 0x7f8f30003aa0 con 0x7f8f4810b930 2024-04-14T15:25:00.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:00.889+0000 7f8f2ffff700 1 -- 172.21.15.90:0/96791544 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8f481a8b20 con 0x7f8f4810b930 2024-04-14T15:25:00.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:00.889+0000 7f8f2ffff700 1 -- 172.21.15.90:0/96791544 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f8f38004410 con 0x7f8f48103a40 2024-04-14T15:25:00.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:00.889+0000 7f8f2ffff700 1 -- 172.21.15.90:0/96791544 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f8f3c001940 con 0x7f8f4810b930 2024-04-14T15:25:00.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:00.889+0000 7f8f2ffff700 1 -- 172.21.15.90:0/96791544 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1003040113 0 0) 0x7f8f481a7900 con 0x7f8f4810f4d0 2024-04-14T15:25:00.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:00.889+0000 7f8f2ffff700 1 -- 172.21.15.90:0/96791544 >> v1:172.21.15.90:6790/0 conn(0x7f8f4810b930 legacy=0x7f8f481a2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:00.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:00.889+0000 7f8f2ffff700 1 -- 172.21.15.90:0/96791544 >> v1:172.21.15.90:6789/0 conn(0x7f8f48103a40 legacy=0x7f8f4810af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:00.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:00.889+0000 7f8f2ffff700 1 -- 172.21.15.90:0/96791544 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f481aafa0 con 0x7f8f4810f4d0 2024-04-14T15:25:00.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:00.890+0000 7f8f4d7a3700 1 -- 172.21.15.90:0/96791544 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8f481a7b30 con 0x7f8f4810f4d0 2024-04-14T15:25:00.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:00.890+0000 7f8f4d7a3700 10 client.49789.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:25:00.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:00.890+0000 7f8f4d7a3700 1 -- 172.21.15.90:0/96791544 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f8f481a7ff0 con 0x7f8f4810f4d0 2024-04-14T15:25:00.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:00.890+0000 7f8f2ffff700 1 -- 172.21.15.90:0/96791544 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f8f40003ea0 con 0x7f8f4810f4d0 2024-04-14T15:25:00.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:00.890+0000 7f8f4d7a3700 1 librados: init done 2024-04-14T15:25:00.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:00.890+0000 7f8f2ffff700 1 -- 172.21.15.90:0/96791544 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f8f40004cb0 con 0x7f8f4810f4d0 2024-04-14T15:25:00.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:00.891+0000 7f8f4d7a3700 1 -- 172.21.15.90:0/96791544 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f34005120 con 0x7f8f4810f4d0 2024-04-14T15:25:00.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:00.891+0000 7f8f2ffff700 1 -- 172.21.15.90:0/96791544 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f8f40003a00 con 0x7f8f4810f4d0 2024-04-14T15:25:00.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:00.892+0000 7f8f2ffff700 1 -- 172.21.15.90:0/96791544 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(756..756 src has 467..756) ==== 9143+0+0 (unknown 4155006609 0 0) 0x7f8f40089c70 con 0x7f8f4810f4d0 2024-04-14T15:25:00.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:00.892+0000 7f8f2ffff700 10 client.49789.objecter ms_dispatch 0x7f8f48001550 osd_map(756..756 src has 467..756) 2024-04-14T15:25:00.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:00.892+0000 7f8f2ffff700 3 client.49789.objecter handle_osd_map got epochs [756,756] > 0 2024-04-14T15:25:00.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:00.892+0000 7f8f2ffff700 3 client.49789.objecter handle_osd_map decoding full epoch 756 2024-04-14T15:25:00.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:00.892+0000 7f8f2ffff700 20 client.49789.objecter dump_active .. 0 homeless 2024-04-14T15:25:00.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:00.892+0000 7f8f2ffff700 10 client.49789.objecter ms_handle_connect 0x7f8f30071500 2024-04-14T15:25:00.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:00.897+0000 7f8f2ffff700 1 -- 172.21.15.90:0/96791544 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f8f400562b0 con 0x7f8f4810f4d0 2024-04-14T15:25:00.937 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:25:00.965 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:00.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:00.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:25:01.067 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:01.090 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:01.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:01.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:01.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:01.135+0000 7f8f4d7a3700 1 -- 172.21.15.90:0/96791544 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7f8f34005410 con 0x7f8f4810f4d0 2024-04-14T15:25:01.140 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:01 smithi090 ceph-mon[30399]: pgmap v1117: 224 pgs: 1 active+clean+snaptrim, 10 remapped+peering, 213 active+clean; 622 KiB data, 693 MiB used, 625 GiB / 626 GiB avail; 8.0 KiB/s rd, 42 KiB/s wr, 10 op/s; 29 KiB/s, 0 objects/s recovering 2024-04-14T15:25:01.141 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:01 smithi090 ceph-mon[30399]: Health check failed: Reduced data availability: 2 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-04-14T15:25:01.141 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:01 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]': finished 2024-04-14T15:25:01.141 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:01 smithi090 ceph-mon[30399]: 1.13 starting backfill to osd.3 from (0'0,0'0] MAX to 91'130 2024-04-14T15:25:01.141 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:01 smithi090 ceph-mon[30399]: 1.d starting backfill to osd.1 from (0'0,0'0] MAX to 91'130 2024-04-14T15:25:01.141 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:01 smithi090 ceph-mon[30399]: 1.11 starting backfill to osd.3 from (0'0,0'0] MAX to 91'130 2024-04-14T15:25:01.141 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:01 smithi090 ceph-mon[30399]: 1.d starting backfill to osd.7 from (0'0,0'0] MAX to 91'130 2024-04-14T15:25:01.141 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:01 smithi090 ceph-mon[30399]: osdmap e756: 8 total, 8 up, 7 in 2024-04-14T15:25:01.141 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:01 smithi090 ceph-mon[30399]: 1.12 starting backfill to osd.1 from (0'0,0'0] MAX to 91'130 2024-04-14T15:25:01.141 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:01 smithi090 ceph-mon[30399]: 1.12 starting backfill to osd.5 from (0'0,0'0] MAX to 91'130 2024-04-14T15:25:01.141 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:01 smithi090 ceph-mon[30399]: 1.10 starting backfill to osd.4 from (0'0,0'0] MAX to 91'130 2024-04-14T15:25:01.141 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:01 smithi090 ceph-mon[30399]: 1.10 starting backfill to osd.5 from (0'0,0'0] MAX to 91'130 2024-04-14T15:25:01.141 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:01 smithi090 ceph-mon[30399]: 1.b starting backfill to osd.0 from (0'0,0'0] MAX to 91'130 2024-04-14T15:25:01.141 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:01 smithi090 ceph-mon[30399]: 1.c starting backfill to osd.0 from (0'0,0'0] MAX to 91'130 2024-04-14T15:25:01.142 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:01 smithi090 ceph-mon[30399]: 1.14 starting backfill to osd.2 from (0'0,0'0] MAX to 91'130 2024-04-14T15:25:01.142 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:01 smithi090 ceph-mon[30399]: 1.f starting backfill to osd.4 from (0'0,0'0] MAX to 91'130 2024-04-14T15:25:01.142 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:01 smithi090 ceph-mon[30399]: 1.14 starting backfill to osd.4 from (0'0,0'0] MAX to 91'130 2024-04-14T15:25:01.142 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:01 smithi090 ceph-mon[30399]: 1.c starting backfill to osd.4 from (0'0,0'0] MAX to 91'130 2024-04-14T15:25:01.142 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:01 smithi090 ceph-mon[30399]: 1.e starting backfill to osd.7 from (0'0,0'0] MAX to 91'130 2024-04-14T15:25:01.142 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:01 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-132"}]: dispatch 2024-04-14T15:25:01.142 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:01 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-132"}]: dispatch 2024-04-14T15:25:01.142 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:01 smithi090 ceph-mon[30399]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:25:01.142 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:01 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-132"}]': finished 2024-04-14T15:25:01.142 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:01 smithi090 ceph-mon[30399]: osdmap e757: 8 total, 8 up, 7 in 2024-04-14T15:25:01.142 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:01 smithi090 ceph-mon[41670]: pgmap v1117: 224 pgs: 1 active+clean+snaptrim, 10 remapped+peering, 213 active+clean; 622 KiB data, 693 MiB used, 625 GiB / 626 GiB avail; 8.0 KiB/s rd, 42 KiB/s wr, 10 op/s; 29 KiB/s, 0 objects/s recovering 2024-04-14T15:25:01.143 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:01 smithi090 ceph-mon[41670]: Health check failed: Reduced data availability: 2 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-04-14T15:25:01.143 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:01 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]': finished 2024-04-14T15:25:01.143 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:01 smithi090 ceph-mon[41670]: 1.13 starting backfill to osd.3 from (0'0,0'0] MAX to 91'130 2024-04-14T15:25:01.143 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:01 smithi090 ceph-mon[41670]: 1.d starting backfill to osd.1 from (0'0,0'0] MAX to 91'130 2024-04-14T15:25:01.143 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:01 smithi090 ceph-mon[41670]: 1.11 starting backfill to osd.3 from (0'0,0'0] MAX to 91'130 2024-04-14T15:25:01.143 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:01 smithi090 ceph-mon[41670]: 1.d starting backfill to osd.7 from (0'0,0'0] MAX to 91'130 2024-04-14T15:25:01.143 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:01 smithi090 ceph-mon[41670]: osdmap e756: 8 total, 8 up, 7 in 2024-04-14T15:25:01.143 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:01 smithi090 ceph-mon[41670]: 1.12 starting backfill to osd.1 from (0'0,0'0] MAX to 91'130 2024-04-14T15:25:01.143 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:01 smithi090 ceph-mon[41670]: 1.12 starting backfill to osd.5 from (0'0,0'0] MAX to 91'130 2024-04-14T15:25:01.143 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:01 smithi090 ceph-mon[41670]: 1.10 starting backfill to osd.4 from (0'0,0'0] MAX to 91'130 2024-04-14T15:25:01.143 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:01 smithi090 ceph-mon[41670]: 1.10 starting backfill to osd.5 from (0'0,0'0] MAX to 91'130 2024-04-14T15:25:01.143 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:01 smithi090 ceph-mon[41670]: 1.b starting backfill to osd.0 from (0'0,0'0] MAX to 91'130 2024-04-14T15:25:01.143 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:01 smithi090 ceph-mon[41670]: 1.c starting backfill to osd.0 from (0'0,0'0] MAX to 91'130 2024-04-14T15:25:01.143 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:01 smithi090 ceph-mon[41670]: 1.14 starting backfill to osd.2 from (0'0,0'0] MAX to 91'130 2024-04-14T15:25:01.143 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:01 smithi090 ceph-mon[41670]: 1.f starting backfill to osd.4 from (0'0,0'0] MAX to 91'130 2024-04-14T15:25:01.144 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:01 smithi090 ceph-mon[41670]: 1.14 starting backfill to osd.4 from (0'0,0'0] MAX to 91'130 2024-04-14T15:25:01.144 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:01 smithi090 ceph-mon[41670]: 1.c starting backfill to osd.4 from (0'0,0'0] MAX to 91'130 2024-04-14T15:25:01.144 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:01 smithi090 ceph-mon[41670]: 1.e starting backfill to osd.7 from (0'0,0'0] MAX to 91'130 2024-04-14T15:25:01.144 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:01 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-132"}]: dispatch 2024-04-14T15:25:01.144 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:01 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-132"}]: dispatch 2024-04-14T15:25:01.144 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:01 smithi090 ceph-mon[41670]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:25:01.144 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:01 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-132"}]': finished 2024-04-14T15:25:01.144 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:01 smithi090 ceph-mon[41670]: osdmap e757: 8 total, 8 up, 7 in 2024-04-14T15:25:01.193 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:25:01.216 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:01.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:01.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:25:01.318 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:25:01.343 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:01.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:01.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:25:01.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:01 smithi092 ceph-mon[33966]: pgmap v1117: 224 pgs: 1 active+clean+snaptrim, 10 remapped+peering, 213 active+clean; 622 KiB data, 693 MiB used, 625 GiB / 626 GiB avail; 8.0 KiB/s rd, 42 KiB/s wr, 10 op/s; 29 KiB/s, 0 objects/s recovering 2024-04-14T15:25:01.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:01 smithi092 ceph-mon[33966]: Health check failed: Reduced data availability: 2 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-04-14T15:25:01.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:01 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]': finished 2024-04-14T15:25:01.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:01 smithi092 ceph-mon[33966]: 1.13 starting backfill to osd.3 from (0'0,0'0] MAX to 91'130 2024-04-14T15:25:01.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:01 smithi092 ceph-mon[33966]: 1.d starting backfill to osd.1 from (0'0,0'0] MAX to 91'130 2024-04-14T15:25:01.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:01 smithi092 ceph-mon[33966]: 1.11 starting backfill to osd.3 from (0'0,0'0] MAX to 91'130 2024-04-14T15:25:01.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:01 smithi092 ceph-mon[33966]: 1.d starting backfill to osd.7 from (0'0,0'0] MAX to 91'130 2024-04-14T15:25:01.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:01 smithi092 ceph-mon[33966]: osdmap e756: 8 total, 8 up, 7 in 2024-04-14T15:25:01.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:01 smithi092 ceph-mon[33966]: 1.12 starting backfill to osd.1 from (0'0,0'0] MAX to 91'130 2024-04-14T15:25:01.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:01 smithi092 ceph-mon[33966]: 1.12 starting backfill to osd.5 from (0'0,0'0] MAX to 91'130 2024-04-14T15:25:01.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:01 smithi092 ceph-mon[33966]: 1.10 starting backfill to osd.4 from (0'0,0'0] MAX to 91'130 2024-04-14T15:25:01.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:01 smithi092 ceph-mon[33966]: 1.10 starting backfill to osd.5 from (0'0,0'0] MAX to 91'130 2024-04-14T15:25:01.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:01 smithi092 ceph-mon[33966]: 1.b starting backfill to osd.0 from (0'0,0'0] MAX to 91'130 2024-04-14T15:25:01.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:01 smithi092 ceph-mon[33966]: 1.c starting backfill to osd.0 from (0'0,0'0] MAX to 91'130 2024-04-14T15:25:01.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:01 smithi092 ceph-mon[33966]: 1.14 starting backfill to osd.2 from (0'0,0'0] MAX to 91'130 2024-04-14T15:25:01.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:01 smithi092 ceph-mon[33966]: 1.f starting backfill to osd.4 from (0'0,0'0] MAX to 91'130 2024-04-14T15:25:01.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:01 smithi092 ceph-mon[33966]: 1.14 starting backfill to osd.4 from (0'0,0'0] MAX to 91'130 2024-04-14T15:25:01.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:01 smithi092 ceph-mon[33966]: 1.c starting backfill to osd.4 from (0'0,0'0] MAX to 91'130 2024-04-14T15:25:01.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:01 smithi092 ceph-mon[33966]: 1.e starting backfill to osd.7 from (0'0,0'0] MAX to 91'130 2024-04-14T15:25:01.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:01 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-132"}]: dispatch 2024-04-14T15:25:01.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:01 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-132"}]: dispatch 2024-04-14T15:25:01.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:01 smithi092 ceph-mon[33966]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:25:01.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:01 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-132"}]': finished 2024-04-14T15:25:01.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:01 smithi092 ceph-mon[33966]: osdmap e757: 8 total, 8 up, 7 in 2024-04-14T15:25:01.445 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:25:01.469 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:01.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:01.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:25:01.557 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:25:01.571 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:25:01.579 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-14T15:25:01.579 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:25:01.596 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:01.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:01.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:25:01.697 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:01.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:01.722 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:01.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:01.823 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:25:01.847 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:01.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:01.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:25:01.948 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:25:01.971 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:01.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:01.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:25:02.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:02.011+0000 7f8f2ffff700 1 -- 172.21.15.90:0/96791544 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v758) ==== 91+0+0 (unknown 36584173 0 0) 0x7f8f40059f60 con 0x7f8f4810f4d0 2024-04-14T15:25:02.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:02.016+0000 7f8f4d7a3700 10 librados: watch_flush enter 2024-04-14T15:25:02.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:02.016+0000 7f8f4d7a3700 10 librados: watch_flush exit 2024-04-14T15:25:02.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:02.016+0000 7f8f4d7a3700 20 client.49789.objecter shutdown clearing up homeless session... 2024-04-14T15:25:02.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:02.016+0000 7f8f4d7a3700 10 client.49789.objecter successfully canceled tick 2024-04-14T15:25:02.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:02.016+0000 7f8f4d7a3700 1 -- 172.21.15.90:0/96791544 >> v1:172.21.15.90:6800/420935427 conn(0x7f8f30071500 legacy=0x7f8f300739c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:02.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:02.016+0000 7f8f4d7a3700 1 -- 172.21.15.90:0/96791544 >> v1:172.21.15.92:6789/0 conn(0x7f8f4810f4d0 legacy=0x7f8f481a6040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:02.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:02.016+0000 7f8f4d7a3700 1 -- 172.21.15.90:0/96791544 shutdown_connections 2024-04-14T15:25:02.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:02.016+0000 7f8f4d7a3700 1 -- 172.21.15.90:0/96791544 >> 172.21.15.90:0/96791544 conn(0x7f8f480fd290 msgr2=0x7f8f48111950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:25:02.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:02.017+0000 7f8f4d7a3700 1 -- 172.21.15.90:0/96791544 shutdown_connections 2024-04-14T15:25:02.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:02.017+0000 7f8f4d7a3700 1 -- 172.21.15.90:0/96791544 wait complete. 2024-04-14T15:25:02.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:02.017+0000 7f8f4d7a3700 1 librados: shutdown 2024-04-14T15:25:02.019 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is unset 2024-04-14T15:25:02.073 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:25:02.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:02.097 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:02.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:25:02.108 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:02 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/96791544' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:25:02.108 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:02 smithi090 ceph-mon[41670]: from='client.49789 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:25:02.108 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:02 smithi090 ceph-mon[41670]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:25:02.108 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:02 smithi090 ceph-mon[41670]: from='client.49789 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-14T15:25:02.108 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:02 smithi090 ceph-mon[41670]: osdmap e758: 8 total, 8 up, 7 in 2024-04-14T15:25:02.108 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:02 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/96791544' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:25:02.108 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:02 smithi090 ceph-mon[30399]: from='client.49789 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:25:02.109 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:02 smithi090 ceph-mon[30399]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:25:02.109 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:02 smithi090 ceph-mon[30399]: from='client.49789 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-14T15:25:02.109 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:02 smithi090 ceph-mon[30399]: osdmap e758: 8 total, 8 up, 7 in 2024-04-14T15:25:02.198 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:25:02.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:02.223 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:02.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:25:02.324 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:25:02.352 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:02.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:02.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:25:02.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:02 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/96791544' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:25:02.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:02 smithi092 ceph-mon[33966]: from='client.49789 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:25:02.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:02 smithi092 ceph-mon[33966]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:25:02.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:02 smithi092 ceph-mon[33966]: from='client.49789 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-14T15:25:02.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:02 smithi092 ceph-mon[33966]: osdmap e758: 8 total, 8 up, 7 in 2024-04-14T15:25:02.453 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:25:02.481 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:02.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:02.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:25:02.493 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-14T15:25:02.583 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:25:02.611 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:02.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:02.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:25:02.713 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:25:02.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:02.737 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:02.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:25:02.838 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:25:02.866 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:02.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:02.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:25:02.968 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:25:02.993 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:02.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:02.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:25:03.095 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:25:03.119 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:03.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:03.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:25:03.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:03 smithi090 ceph-mon[30399]: pgmap v1120: 224 pgs: 1 active+clean+snaptrim, 10 remapped+peering, 213 active+clean; 622 KiB data, 693 MiB used, 625 GiB / 626 GiB avail; 8.0 KiB/s rd, 42 KiB/s wr, 10 op/s; 29 KiB/s, 0 objects/s recovering 2024-04-14T15:25:03.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:03 smithi090 ceph-mon[30399]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:25:03.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:03 smithi090 ceph-mon[30399]: 1.e scrub starts 2024-04-14T15:25:03.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:03 smithi090 ceph-mon[30399]: 1.e scrub ok 2024-04-14T15:25:03.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:03 smithi090 ceph-mon[30399]: 1.b scrub starts 2024-04-14T15:25:03.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:03 smithi090 ceph-mon[30399]: 1.b scrub ok 2024-04-14T15:25:03.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:03 smithi090 ceph-mon[30399]: osdmap e759: 8 total, 8 up, 7 in 2024-04-14T15:25:03.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:03 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:25:03.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:03 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:25:03.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:03 smithi090 ceph-mon[41670]: pgmap v1120: 224 pgs: 1 active+clean+snaptrim, 10 remapped+peering, 213 active+clean; 622 KiB data, 693 MiB used, 625 GiB / 626 GiB avail; 8.0 KiB/s rd, 42 KiB/s wr, 10 op/s; 29 KiB/s, 0 objects/s recovering 2024-04-14T15:25:03.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:03 smithi090 ceph-mon[41670]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:25:03.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:03 smithi090 ceph-mon[41670]: 1.e scrub starts 2024-04-14T15:25:03.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:03 smithi090 ceph-mon[41670]: 1.e scrub ok 2024-04-14T15:25:03.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:03 smithi090 ceph-mon[41670]: 1.b scrub starts 2024-04-14T15:25:03.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:03 smithi090 ceph-mon[41670]: 1.b scrub ok 2024-04-14T15:25:03.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:03 smithi090 ceph-mon[41670]: osdmap e759: 8 total, 8 up, 7 in 2024-04-14T15:25:03.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:03 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:25:03.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:03 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:25:03.221 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:25:03.245 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:03.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:03.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:25:03.347 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:25:03.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:03.372 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:03.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:25:03.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:03 smithi092 ceph-mon[33966]: pgmap v1120: 224 pgs: 1 active+clean+snaptrim, 10 remapped+peering, 213 active+clean; 622 KiB data, 693 MiB used, 625 GiB / 626 GiB avail; 8.0 KiB/s rd, 42 KiB/s wr, 10 op/s; 29 KiB/s, 0 objects/s recovering 2024-04-14T15:25:03.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:03 smithi092 ceph-mon[33966]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:25:03.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:03 smithi092 ceph-mon[33966]: 1.e scrub starts 2024-04-14T15:25:03.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:03 smithi092 ceph-mon[33966]: 1.e scrub ok 2024-04-14T15:25:03.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:03 smithi092 ceph-mon[33966]: 1.b scrub starts 2024-04-14T15:25:03.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:03 smithi092 ceph-mon[33966]: 1.b scrub ok 2024-04-14T15:25:03.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:03 smithi092 ceph-mon[33966]: osdmap e759: 8 total, 8 up, 7 in 2024-04-14T15:25:03.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:03 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:25:03.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:03 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:25:03.472 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:25:03.496 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:03.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:03.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:25:03.598 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:25:03.621 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:03.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:03.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:25:03.723 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:25:03.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:03.747 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:03.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:25:03.848 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:25:03.871 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:03.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:03.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:25:03.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:03.919+0000 7f6794030700 1 -- 172.21.15.90:0/459454950 >> v1:172.21.15.90:6789/0 conn(0x7f678c10e310 legacy=0x7f678c1107d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:03.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:03.919+0000 7f6794030700 1 -- 172.21.15.90:0/459454950 shutdown_connections 2024-04-14T15:25:03.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:03.919+0000 7f6794030700 1 -- 172.21.15.90:0/459454950 >> 172.21.15.90:0/459454950 conn(0x7f678c0fd290 msgr2=0x7f678c0ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:25:03.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:03.919+0000 7f6794030700 1 -- 172.21.15.90:0/459454950 shutdown_connections 2024-04-14T15:25:03.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:03.919+0000 7f6794030700 1 -- 172.21.15.90:0/459454950 wait complete. 2024-04-14T15:25:03.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:03.919+0000 7f6794030700 1 librados: starting msgr at 2024-04-14T15:25:03.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:03.919+0000 7f6794030700 1 librados: starting objecter 2024-04-14T15:25:03.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:03.919+0000 7f6794030700 1 Processor -- start 2024-04-14T15:25:03.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:03.920+0000 7f6794030700 1 -- start start 2024-04-14T15:25:03.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:03.920+0000 7f6794030700 1 librados: setting wanted keys 2024-04-14T15:25:03.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:03.920+0000 7f6794030700 1 librados: calling monclient init 2024-04-14T15:25:03.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:03.920+0000 7f6794030700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f678c10e0c0 con 0x7f678c10e310 2024-04-14T15:25:03.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:03.920+0000 7f6794030700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f678c1a7f20 con 0x7f678c10a740 2024-04-14T15:25:03.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:03.920+0000 7f6794030700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f678c1a9140 con 0x7f678c107890 2024-04-14T15:25:03.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:03.920+0000 7f6791dcc700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f678c107890 0x7f678c10b070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:57122/0 (socket says 172.21.15.90:57122) 2024-04-14T15:25:03.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:03.920+0000 7f6791dcc700 1 -- 172.21.15.90:0/687516363 learned_addr learned my addr 172.21.15.90:0/687516363 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:25:03.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:03.920+0000 7f6782ffd700 10 client.?.objecter ms_handle_connect 0x7f678c107890 2024-04-14T15:25:03.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:03.920+0000 7f6782ffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:25:03.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:03.920+0000 7f6782ffd700 10 client.?.objecter ms_handle_connect 0x7f678c10e310 2024-04-14T15:25:03.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:03.920+0000 7f6782ffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:25:03.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:03.920+0000 7f6782ffd700 10 client.?.objecter ms_handle_connect 0x7f678c10a740 2024-04-14T15:25:03.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:03.920+0000 7f6782ffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:25:03.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:03.920+0000 7f6782ffd700 1 -- 172.21.15.90:0/687516363 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4193816762 0 0) 0x7f678c1a9140 con 0x7f678c107890 2024-04-14T15:25:03.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:03.920+0000 7f6782ffd700 1 -- 172.21.15.90:0/687516363 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f67780035b0 con 0x7f678c107890 2024-04-14T15:25:03.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:03.920+0000 7f6782ffd700 1 -- 172.21.15.90:0/687516363 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 74193197 0 0) 0x7f678c10e0c0 con 0x7f678c10e310 2024-04-14T15:25:03.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:03.920+0000 7f6782ffd700 1 -- 172.21.15.90:0/687516363 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f678c1a9140 con 0x7f678c10e310 2024-04-14T15:25:03.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:03.920+0000 7f6782ffd700 1 -- 172.21.15.90:0/687516363 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 887886088 0 0) 0x7f678c1a7f20 con 0x7f678c10a740 2024-04-14T15:25:03.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:03.920+0000 7f6782ffd700 1 -- 172.21.15.90:0/687516363 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f678c10e0c0 con 0x7f678c10a740 2024-04-14T15:25:03.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:03.920+0000 7f6782ffd700 1 -- 172.21.15.90:0/687516363 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2946097981 0 0) 0x7f67780035b0 con 0x7f678c107890 2024-04-14T15:25:03.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:03.920+0000 7f6782ffd700 1 -- 172.21.15.90:0/687516363 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f678c1a7f20 con 0x7f678c107890 2024-04-14T15:25:03.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:03.920+0000 7f6782ffd700 1 -- 172.21.15.90:0/687516363 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f67880018b0 con 0x7f678c107890 2024-04-14T15:25:03.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:03.920+0000 7f6782ffd700 1 -- 172.21.15.90:0/687516363 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1317602825 0 0) 0x7f678c10e0c0 con 0x7f678c10a740 2024-04-14T15:25:03.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:03.920+0000 7f6782ffd700 1 -- 172.21.15.90:0/687516363 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f67780035b0 con 0x7f678c10a740 2024-04-14T15:25:03.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:03.921+0000 7f6782ffd700 1 -- 172.21.15.90:0/687516363 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f677c0017f0 con 0x7f678c10a740 2024-04-14T15:25:03.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:03.921+0000 7f6782ffd700 1 -- 172.21.15.90:0/687516363 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 595834344 0 0) 0x7f678c1a9140 con 0x7f678c10e310 2024-04-14T15:25:03.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:03.921+0000 7f6782ffd700 1 -- 172.21.15.90:0/687516363 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f678c10e0c0 con 0x7f678c10e310 2024-04-14T15:25:03.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:03.921+0000 7f6782ffd700 1 -- 172.21.15.90:0/687516363 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f6784001950 con 0x7f678c10e310 2024-04-14T15:25:03.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:03.921+0000 7f6782ffd700 1 -- 172.21.15.90:0/687516363 <== mon.2 v1:172.21.15.90:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2833403209 0 0) 0x7f678c1a7f20 con 0x7f678c107890 2024-04-14T15:25:03.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:03.921+0000 7f6782ffd700 1 -- 172.21.15.90:0/687516363 >> v1:172.21.15.92:6789/0 conn(0x7f678c10a740 legacy=0x7f678c10b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:03.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:03.921+0000 7f6782ffd700 1 -- 172.21.15.90:0/687516363 >> v1:172.21.15.90:6789/0 conn(0x7f678c10e310 legacy=0x7f678c1a57f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:03.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:03.921+0000 7f6782ffd700 1 -- 172.21.15.90:0/687516363 --> v1:172.21.15.90:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f678c1aa360 con 0x7f678c107890 2024-04-14T15:25:03.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:03.921+0000 7f6794030700 1 -- 172.21.15.90:0/687516363 --> v1:172.21.15.90:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f678c1a9370 con 0x7f678c107890 2024-04-14T15:25:03.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:03.921+0000 7f6794030700 10 client.47915.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:25:03.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:03.921+0000 7f6794030700 1 -- 172.21.15.90:0/687516363 --> v1:172.21.15.90:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f678c1a9830 con 0x7f678c107890 2024-04-14T15:25:03.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:03.921+0000 7f6794030700 1 librados: init done 2024-04-14T15:25:03.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:03.921+0000 7f6782ffd700 1 -- 172.21.15.90:0/687516363 <== mon.2 v1:172.21.15.90:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f6788003b20 con 0x7f678c107890 2024-04-14T15:25:03.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:03.921+0000 7f6782ffd700 1 -- 172.21.15.90:0/687516363 <== mon.2 v1:172.21.15.90:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f6788005ce0 con 0x7f678c107890 2024-04-14T15:25:03.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:03.922+0000 7f6794030700 1 -- 172.21.15.90:0/687516363 --> v1:172.21.15.90:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6770005120 con 0x7f678c107890 2024-04-14T15:25:03.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:03.922+0000 7f6782ffd700 1 -- 172.21.15.90:0/687516363 <== mon.2 v1:172.21.15.90:6790/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f6788003680 con 0x7f678c107890 2024-04-14T15:25:03.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:03.923+0000 7f6782ffd700 1 -- 172.21.15.90:0/687516363 <== mon.2 v1:172.21.15.90:6790/0 8 ==== osd_map(759..759 src has 467..759) ==== 8909+0+0 (unknown 1235498003 0 0) 0x7f678808ace0 con 0x7f678c107890 2024-04-14T15:25:03.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:03.923+0000 7f6782ffd700 10 client.47915.objecter ms_dispatch 0x7f678c001550 osd_map(759..759 src has 467..759) 2024-04-14T15:25:03.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:03.923+0000 7f6782ffd700 3 client.47915.objecter handle_osd_map got epochs [759,759] > 0 2024-04-14T15:25:03.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:03.923+0000 7f6782ffd700 3 client.47915.objecter handle_osd_map decoding full epoch 759 2024-04-14T15:25:03.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:03.923+0000 7f6782ffd700 20 client.47915.objecter dump_active .. 0 homeless 2024-04-14T15:25:03.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:03.923+0000 7f6782ffd700 10 client.47915.objecter ms_handle_connect 0x7f67780734e0 2024-04-14T15:25:03.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:03.928+0000 7f6782ffd700 1 -- 172.21.15.90:0/687516363 <== mon.2 v1:172.21.15.90:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f6788057400 con 0x7f678c107890 2024-04-14T15:25:03.973 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:25:03.996 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:03.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:03.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:25:04.097 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:25:04.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:04.122 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:04.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:25:04.167 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:25:03 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:25:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:25:04.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:04.167+0000 7f6794030700 1 -- 172.21.15.90:0/687516363 --> v1:172.21.15.90:6790/0 -- mon_command({"prefix": "osd reweight", "id": 0, "weight": 1.0} v 0) -- 0x7f6770005410 con 0x7f678c107890 2024-04-14T15:25:04.168 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:04 smithi090 ceph-mon[30399]: 1.14 scrub starts 2024-04-14T15:25:04.168 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:04 smithi090 ceph-mon[30399]: 1.14 scrub ok 2024-04-14T15:25:04.168 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:04 smithi090 ceph-mon[30399]: 1.11 scrub starts 2024-04-14T15:25:04.168 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:04 smithi090 ceph-mon[30399]: 1.11 scrub ok 2024-04-14T15:25:04.168 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:04 smithi090 ceph-mon[30399]: 1.d scrub starts 2024-04-14T15:25:04.168 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:04 smithi090 ceph-mon[30399]: 1.d scrub ok 2024-04-14T15:25:04.168 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:04 smithi090 ceph-mon[30399]: pgmap v1123: 224 pgs: 32 unknown, 1 active+clean+snaptrim, 10 remapped+peering, 181 active+clean; 622 KiB data, 693 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:25:04.168 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:04 smithi090 ceph-mon[30399]: 1.f scrub starts 2024-04-14T15:25:04.169 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:04 smithi090 ceph-mon[30399]: 1.f scrub ok 2024-04-14T15:25:04.169 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:04 smithi090 ceph-mon[30399]: 1.3 scrub starts 2024-04-14T15:25:04.169 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:04 smithi090 ceph-mon[30399]: 1.3 scrub ok 2024-04-14T15:25:04.169 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:04 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:25:04.169 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:04 smithi090 ceph-mon[30399]: osdmap e760: 8 total, 8 up, 7 in 2024-04-14T15:25:04.169 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:04 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:25:04.169 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:04 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:25:04.169 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:04 smithi090 ceph-mon[41670]: 1.14 scrub starts 2024-04-14T15:25:04.169 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:04 smithi090 ceph-mon[41670]: 1.14 scrub ok 2024-04-14T15:25:04.169 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:04 smithi090 ceph-mon[41670]: 1.11 scrub starts 2024-04-14T15:25:04.169 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:04 smithi090 ceph-mon[41670]: 1.11 scrub ok 2024-04-14T15:25:04.169 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:04 smithi090 ceph-mon[41670]: 1.d scrub starts 2024-04-14T15:25:04.170 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:04 smithi090 ceph-mon[41670]: 1.d scrub ok 2024-04-14T15:25:04.170 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:04 smithi090 ceph-mon[41670]: pgmap v1123: 224 pgs: 32 unknown, 1 active+clean+snaptrim, 10 remapped+peering, 181 active+clean; 622 KiB data, 693 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:25:04.170 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:04 smithi090 ceph-mon[41670]: 1.f scrub starts 2024-04-14T15:25:04.170 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:04 smithi090 ceph-mon[41670]: 1.f scrub ok 2024-04-14T15:25:04.170 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:04 smithi090 ceph-mon[41670]: 1.3 scrub starts 2024-04-14T15:25:04.170 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:04 smithi090 ceph-mon[41670]: 1.3 scrub ok 2024-04-14T15:25:04.170 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:04 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:25:04.170 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:04 smithi090 ceph-mon[41670]: osdmap e760: 8 total, 8 up, 7 in 2024-04-14T15:25:04.170 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:04 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:25:04.170 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:04 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:25:04.223 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:25:04.246 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:04.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:04.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:25:04.348 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:25:04.371 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:04.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:04 smithi092 ceph-mon[33966]: 1.14 scrub starts 2024-04-14T15:25:04.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:04 smithi092 ceph-mon[33966]: 1.14 scrub ok 2024-04-14T15:25:04.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:04 smithi092 ceph-mon[33966]: 1.11 scrub starts 2024-04-14T15:25:04.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:04 smithi092 ceph-mon[33966]: 1.11 scrub ok 2024-04-14T15:25:04.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:04 smithi092 ceph-mon[33966]: 1.d scrub starts 2024-04-14T15:25:04.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:04 smithi092 ceph-mon[33966]: 1.d scrub ok 2024-04-14T15:25:04.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:04 smithi092 ceph-mon[33966]: pgmap v1123: 224 pgs: 32 unknown, 1 active+clean+snaptrim, 10 remapped+peering, 181 active+clean; 622 KiB data, 693 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:25:04.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:04 smithi092 ceph-mon[33966]: 1.f scrub starts 2024-04-14T15:25:04.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:04 smithi092 ceph-mon[33966]: 1.f scrub ok 2024-04-14T15:25:04.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:04 smithi092 ceph-mon[33966]: 1.3 scrub starts 2024-04-14T15:25:04.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:04 smithi092 ceph-mon[33966]: 1.3 scrub ok 2024-04-14T15:25:04.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:04 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:25:04.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:04 smithi092 ceph-mon[33966]: osdmap e760: 8 total, 8 up, 7 in 2024-04-14T15:25:04.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:04 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:25:04.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:04 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:25:04.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:04.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:25:04.474 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:04.498 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:04.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:04.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:04.600 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:25:04.623 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:04.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:04.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:25:04.725 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:25:04.748 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:04.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:04.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:25:04.850 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:25:04.873 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:04.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:04.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:25:04.975 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:25:05.001 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:05.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:05.002 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:25:05.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:05.013+0000 7f6782ffd700 1 -- 172.21.15.90:0/687516363 <== mon.2 v1:172.21.15.90:6790/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 0, "weight": 1.0}]=0 reweighted osd.0 to 1 (10000) v761) ==== 113+0+0 (unknown 1800797072 0 0) 0x7f678805b0b0 con 0x7f678c107890 2024-04-14T15:25:05.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:05.016+0000 7f6794030700 10 librados: watch_flush enter 2024-04-14T15:25:05.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:05.016+0000 7f6794030700 10 librados: watch_flush exit 2024-04-14T15:25:05.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:05.016+0000 7f6794030700 20 client.47915.objecter shutdown clearing up homeless session... 2024-04-14T15:25:05.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:05.016+0000 7f6794030700 10 client.47915.objecter successfully canceled tick 2024-04-14T15:25:05.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:05.016+0000 7f6794030700 1 -- 172.21.15.90:0/687516363 >> v1:172.21.15.90:6800/420935427 conn(0x7f67780734e0 legacy=0x7f67780759a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:05.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:05.016+0000 7f6794030700 1 -- 172.21.15.90:0/687516363 >> v1:172.21.15.90:6790/0 conn(0x7f678c107890 legacy=0x7f678c10b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:05.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:05.017+0000 7f6794030700 1 -- 172.21.15.90:0/687516363 shutdown_connections 2024-04-14T15:25:05.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:05.017+0000 7f6794030700 1 -- 172.21.15.90:0/687516363 >> 172.21.15.90:0/687516363 conn(0x7f678c0fd290 msgr2=0x7f678c111880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:25:05.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:05.017+0000 7f6794030700 1 -- 172.21.15.90:0/687516363 shutdown_connections 2024-04-14T15:25:05.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:05.017+0000 7f6794030700 1 -- 172.21.15.90:0/687516363 wait complete. 2024-04-14T15:25:05.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:05.017+0000 7f6794030700 1 librados: shutdown 2024-04-14T15:25:05.018 INFO:teuthology.orchestra.run.smithi090.stderr:reweighted osd.0 to 1 (10000) 2024-04-14T15:25:05.103 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:25:05.126 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:05.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:05.127 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:25:05.228 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:25:05.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:05.252 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:05.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:25:05.353 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:05.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:05 smithi092 ceph-mon[33966]: 1.1 scrub starts 2024-04-14T15:25:05.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:05 smithi092 ceph-mon[33966]: 1.1 scrub ok 2024-04-14T15:25:05.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:05 smithi092 ceph-mon[33966]: 4.14 scrub starts 2024-04-14T15:25:05.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:05 smithi092 ceph-mon[33966]: 4.14 scrub ok 2024-04-14T15:25:05.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:05 smithi092 ceph-mon[33966]: 1.2 scrub starts 2024-04-14T15:25:05.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:05 smithi092 ceph-mon[33966]: 1.2 scrub ok 2024-04-14T15:25:05.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:05 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/687516363' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-14T15:25:05.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:05 smithi092 ceph-mon[33966]: from='client.47915 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-14T15:25:05.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:05 smithi092 ceph-mon[33966]: 1.5 scrub starts 2024-04-14T15:25:05.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:05 smithi092 ceph-mon[33966]: 1.5 scrub ok 2024-04-14T15:25:05.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:05 smithi092 ceph-mon[33966]: 323.0s0 scrub starts 2024-04-14T15:25:05.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:05 smithi092 ceph-mon[33966]: 323.0 scrub ok 2024-04-14T15:25:05.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:05 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-134", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:25:05.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:05 smithi092 ceph-mon[33966]: from='client.47915 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-14T15:25:05.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:05 smithi092 ceph-mon[33966]: osdmap e761: 8 total, 8 up, 7 in 2024-04-14T15:25:05.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:05 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-134"}]: dispatch 2024-04-14T15:25:05.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:05 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-134"}]: dispatch 2024-04-14T15:25:05.376 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:05 smithi090 ceph-mon[41670]: 1.1 scrub starts 2024-04-14T15:25:05.376 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:05 smithi090 ceph-mon[41670]: 1.1 scrub ok 2024-04-14T15:25:05.376 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:05 smithi090 ceph-mon[41670]: 4.14 scrub starts 2024-04-14T15:25:05.376 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:05 smithi090 ceph-mon[41670]: 4.14 scrub ok 2024-04-14T15:25:05.376 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:05 smithi090 ceph-mon[41670]: 1.2 scrub starts 2024-04-14T15:25:05.376 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:05 smithi090 ceph-mon[41670]: 1.2 scrub ok 2024-04-14T15:25:05.376 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:05 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/687516363' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-14T15:25:05.377 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:05 smithi090 ceph-mon[41670]: from='client.47915 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-14T15:25:05.377 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:05 smithi090 ceph-mon[41670]: 1.5 scrub starts 2024-04-14T15:25:05.377 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:05 smithi090 ceph-mon[41670]: 1.5 scrub ok 2024-04-14T15:25:05.377 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:05 smithi090 ceph-mon[41670]: 323.0s0 scrub starts 2024-04-14T15:25:05.377 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:05 smithi090 ceph-mon[41670]: 323.0 scrub ok 2024-04-14T15:25:05.377 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:05 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-134", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:25:05.377 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:05 smithi090 ceph-mon[41670]: from='client.47915 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-14T15:25:05.377 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:05 smithi090 ceph-mon[41670]: osdmap e761: 8 total, 8 up, 7 in 2024-04-14T15:25:05.377 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:05 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-134"}]: dispatch 2024-04-14T15:25:05.377 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:05 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-134"}]: dispatch 2024-04-14T15:25:05.378 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:05 smithi090 ceph-mon[30399]: 1.1 scrub starts 2024-04-14T15:25:05.378 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:05 smithi090 ceph-mon[30399]: 1.1 scrub ok 2024-04-14T15:25:05.378 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:05 smithi090 ceph-mon[30399]: 4.14 scrub starts 2024-04-14T15:25:05.378 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:05 smithi090 ceph-mon[30399]: 4.14 scrub ok 2024-04-14T15:25:05.378 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:05 smithi090 ceph-mon[30399]: 1.2 scrub starts 2024-04-14T15:25:05.378 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:05 smithi090 ceph-mon[30399]: 1.2 scrub ok 2024-04-14T15:25:05.378 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:05 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/687516363' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-14T15:25:05.378 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:05 smithi090 ceph-mon[30399]: from='client.47915 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-14T15:25:05.378 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:05 smithi090 ceph-mon[30399]: 1.5 scrub starts 2024-04-14T15:25:05.378 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:05 smithi090 ceph-mon[30399]: 1.5 scrub ok 2024-04-14T15:25:05.379 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:05 smithi090 ceph-mon[30399]: 323.0s0 scrub starts 2024-04-14T15:25:05.379 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:05 smithi090 ceph-mon[30399]: 323.0 scrub ok 2024-04-14T15:25:05.379 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:05 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-134", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:25:05.379 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:05 smithi090 ceph-mon[30399]: from='client.47915 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-14T15:25:05.379 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:05 smithi090 ceph-mon[30399]: osdmap e761: 8 total, 8 up, 7 in 2024-04-14T15:25:05.379 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:05 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-134"}]: dispatch 2024-04-14T15:25:05.379 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:05 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-134"}]: dispatch 2024-04-14T15:25:05.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:05.380 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:05.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:05.481 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:25:05.505 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:05.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:05.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:25:05.606 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:25:05.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:05.632 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:05.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:25:05.733 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:25:05.757 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:05.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:05.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:25:05.852 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:25:05.859 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:25:05.872 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:25:05 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:25:05.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:05.883 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:05.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:25:05.984 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:25:06.010 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:06.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:06.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:25:06.112 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:25:06.135 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:06.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:06.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:25:06.237 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:25:06.262 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:06.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:06.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:25:06.363 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:25:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:06 smithi092 ceph-mon[33966]: 1.4 scrub starts 2024-04-14T15:25:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:06 smithi092 ceph-mon[33966]: 1.4 scrub ok 2024-04-14T15:25:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:06 smithi092 ceph-mon[33966]: 1.6 scrub starts 2024-04-14T15:25:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:06 smithi092 ceph-mon[33966]: 1.6 scrub ok 2024-04-14T15:25:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:06 smithi092 ceph-mon[33966]: 1.0 scrub starts 2024-04-14T15:25:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:06 smithi092 ceph-mon[33966]: 1.0 scrub ok 2024-04-14T15:25:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:06 smithi092 ceph-mon[33966]: pgmap v1126: 224 pgs: 4 creating+activating, 4 active+remapped+backfill_toofull, 216 active+clean; 489 KiB data, 715 MiB used, 625 GiB / 626 GiB avail; 2/642 objects misplaced (0.312%) 2024-04-14T15:25:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:06 smithi092 ceph-mon[33966]: 5.1f scrub starts 2024-04-14T15:25:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:06 smithi092 ceph-mon[33966]: 5.1f scrub ok 2024-04-14T15:25:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:06 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:25:06.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:06 smithi092 ceph-mon[33966]: 2.6 scrub starts 2024-04-14T15:25:06.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:06 smithi092 ceph-mon[33966]: 2.6 scrub ok 2024-04-14T15:25:06.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:06 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-134"}]': finished 2024-04-14T15:25:06.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:06 smithi092 ceph-mon[33966]: osdmap e762: 8 total, 8 up, 7 in 2024-04-14T15:25:06.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:06 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-134", "mode": "writeback"}]: dispatch 2024-04-14T15:25:06.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:06 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-134", "mode": "writeback"}]: dispatch 2024-04-14T15:25:06.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:06.388 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:06.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:25:06.403 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:06 smithi090 ceph-mon[41670]: 1.4 scrub starts 2024-04-14T15:25:06.403 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:06 smithi090 ceph-mon[41670]: 1.4 scrub ok 2024-04-14T15:25:06.403 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:06 smithi090 ceph-mon[41670]: 1.6 scrub starts 2024-04-14T15:25:06.403 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:06 smithi090 ceph-mon[41670]: 1.6 scrub ok 2024-04-14T15:25:06.403 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:06 smithi090 ceph-mon[41670]: 1.0 scrub starts 2024-04-14T15:25:06.403 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:06 smithi090 ceph-mon[41670]: 1.0 scrub ok 2024-04-14T15:25:06.403 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:06 smithi090 ceph-mon[41670]: pgmap v1126: 224 pgs: 4 creating+activating, 4 active+remapped+backfill_toofull, 216 active+clean; 489 KiB data, 715 MiB used, 625 GiB / 626 GiB avail; 2/642 objects misplaced (0.312%) 2024-04-14T15:25:06.403 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:06 smithi090 ceph-mon[41670]: 5.1f scrub starts 2024-04-14T15:25:06.403 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:06 smithi090 ceph-mon[41670]: 5.1f scrub ok 2024-04-14T15:25:06.404 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:06 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:25:06.404 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:06 smithi090 ceph-mon[41670]: 2.6 scrub starts 2024-04-14T15:25:06.404 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:06 smithi090 ceph-mon[41670]: 2.6 scrub ok 2024-04-14T15:25:06.404 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:06 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-134"}]': finished 2024-04-14T15:25:06.404 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:06 smithi090 ceph-mon[41670]: osdmap e762: 8 total, 8 up, 7 in 2024-04-14T15:25:06.404 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:06 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-134", "mode": "writeback"}]: dispatch 2024-04-14T15:25:06.404 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:06 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-134", "mode": "writeback"}]: dispatch 2024-04-14T15:25:06.404 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:06 smithi090 ceph-mon[30399]: 1.4 scrub starts 2024-04-14T15:25:06.404 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:06 smithi090 ceph-mon[30399]: 1.4 scrub ok 2024-04-14T15:25:06.404 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:06 smithi090 ceph-mon[30399]: 1.6 scrub starts 2024-04-14T15:25:06.404 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:06 smithi090 ceph-mon[30399]: 1.6 scrub ok 2024-04-14T15:25:06.404 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:06 smithi090 ceph-mon[30399]: 1.0 scrub starts 2024-04-14T15:25:06.404 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:06 smithi090 ceph-mon[30399]: 1.0 scrub ok 2024-04-14T15:25:06.405 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:06 smithi090 ceph-mon[30399]: pgmap v1126: 224 pgs: 4 creating+activating, 4 active+remapped+backfill_toofull, 216 active+clean; 489 KiB data, 715 MiB used, 625 GiB / 626 GiB avail; 2/642 objects misplaced (0.312%) 2024-04-14T15:25:06.405 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:06 smithi090 ceph-mon[30399]: 5.1f scrub starts 2024-04-14T15:25:06.405 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:06 smithi090 ceph-mon[30399]: 5.1f scrub ok 2024-04-14T15:25:06.405 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:06 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:25:06.405 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:06 smithi090 ceph-mon[30399]: 2.6 scrub starts 2024-04-14T15:25:06.405 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:06 smithi090 ceph-mon[30399]: 2.6 scrub ok 2024-04-14T15:25:06.405 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:06 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-134"}]': finished 2024-04-14T15:25:06.405 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:06 smithi090 ceph-mon[30399]: osdmap e762: 8 total, 8 up, 7 in 2024-04-14T15:25:06.405 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:06 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-134", "mode": "writeback"}]: dispatch 2024-04-14T15:25:06.405 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:06 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-134", "mode": "writeback"}]: dispatch 2024-04-14T15:25:06.434 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd reweight 1 1 2024-04-14T15:25:06.489 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:25:06.512 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:06.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:06.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:25:06.536 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-14T15:25:06.614 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:25:06.638 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:06.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:06.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:25:06.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.724+0000 7f1566dd3700 1 -- 172.21.15.90:0/3549036636 >> v1:172.21.15.92:6789/0 conn(0x7f1560103a40 legacy=0x7f1560103e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:06.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.725+0000 7f1566dd3700 1 -- 172.21.15.90:0/3549036636 shutdown_connections 2024-04-14T15:25:06.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.725+0000 7f1566dd3700 1 -- 172.21.15.90:0/3549036636 >> 172.21.15.90:0/3549036636 conn(0x7f15600fd290 msgr2=0x7f15600ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:25:06.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.725+0000 7f1566dd3700 1 -- 172.21.15.90:0/3549036636 shutdown_connections 2024-04-14T15:25:06.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.725+0000 7f1566dd3700 1 -- 172.21.15.90:0/3549036636 wait complete. 2024-04-14T15:25:06.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.725+0000 7f1566dd3700 1 librados: starting msgr at 2024-04-14T15:25:06.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.725+0000 7f1566dd3700 1 librados: starting objecter 2024-04-14T15:25:06.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.725+0000 7f1566dd3700 1 Processor -- start 2024-04-14T15:25:06.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.725+0000 7f1566dd3700 1 -- start start 2024-04-14T15:25:06.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.725+0000 7f1566dd3700 1 librados: setting wanted keys 2024-04-14T15:25:06.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.725+0000 7f1566dd3700 1 librados: calling monclient init 2024-04-14T15:25:06.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.726+0000 7f1566dd3700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f15601a9c50 con 0x7f1560103a40 2024-04-14T15:25:06.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.726+0000 7f1566dd3700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f15601aae70 con 0x7f156010b930 2024-04-14T15:25:06.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.726+0000 7f1566dd3700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f15601ac0b0 con 0x7f156010f4d0 2024-04-14T15:25:06.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.726+0000 7f1564b6f700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f1560103a40 0x7f156010b050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:39210/0 (socket says 172.21.15.90:39210) 2024-04-14T15:25:06.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.726+0000 7f1564b6f700 1 -- 172.21.15.90:0/1202463926 learned_addr learned my addr 172.21.15.90:0/1202463926 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:25:06.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.726+0000 7f155ffff700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f156010b930 0x7f15601a4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.90:42098/0 (socket says 172.21.15.90:42098) 2024-04-14T15:25:06.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.726+0000 7f155dffb700 10 client.?.objecter ms_handle_connect 0x7f1560103a40 2024-04-14T15:25:06.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.726+0000 7f155dffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:25:06.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.726+0000 7f155dffb700 10 client.?.objecter ms_handle_connect 0x7f156010b930 2024-04-14T15:25:06.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.726+0000 7f155dffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:25:06.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.726+0000 7f155dffb700 10 client.?.objecter ms_handle_connect 0x7f156010f4d0 2024-04-14T15:25:06.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.726+0000 7f155dffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:25:06.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.726+0000 7f155dffb700 1 -- 172.21.15.90:0/1202463926 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 816655298 0 0) 0x7f15601a9c50 con 0x7f1560103a40 2024-04-14T15:25:06.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.726+0000 7f155dffb700 1 -- 172.21.15.90:0/1202463926 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1550003aa0 con 0x7f1560103a40 2024-04-14T15:25:06.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.726+0000 7f155dffb700 1 -- 172.21.15.90:0/1202463926 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 175179488 0 0) 0x7f15601aae70 con 0x7f156010b930 2024-04-14T15:25:06.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.726+0000 7f155dffb700 1 -- 172.21.15.90:0/1202463926 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f15601a9c50 con 0x7f156010b930 2024-04-14T15:25:06.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.726+0000 7f155dffb700 1 -- 172.21.15.90:0/1202463926 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3759931678 0 0) 0x7f15601ac0b0 con 0x7f156010f4d0 2024-04-14T15:25:06.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.726+0000 7f155dffb700 1 -- 172.21.15.90:0/1202463926 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f15601aae70 con 0x7f156010f4d0 2024-04-14T15:25:06.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.726+0000 7f155dffb700 1 -- 172.21.15.90:0/1202463926 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3312769704 0 0) 0x7f15601a9c50 con 0x7f156010b930 2024-04-14T15:25:06.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.726+0000 7f155dffb700 1 -- 172.21.15.90:0/1202463926 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f15601ac0b0 con 0x7f156010b930 2024-04-14T15:25:06.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.727+0000 7f155dffb700 1 -- 172.21.15.90:0/1202463926 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f154c001870 con 0x7f156010b930 2024-04-14T15:25:06.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.727+0000 7f155dffb700 1 -- 172.21.15.90:0/1202463926 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3947839920 0 0) 0x7f1550003aa0 con 0x7f1560103a40 2024-04-14T15:25:06.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.727+0000 7f155dffb700 1 -- 172.21.15.90:0/1202463926 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f15601a9c50 con 0x7f1560103a40 2024-04-14T15:25:06.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.727+0000 7f155dffb700 1 -- 172.21.15.90:0/1202463926 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f15540017d0 con 0x7f1560103a40 2024-04-14T15:25:06.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.727+0000 7f155dffb700 1 -- 172.21.15.90:0/1202463926 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1712467421 0 0) 0x7f15601ac0b0 con 0x7f156010b930 2024-04-14T15:25:06.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.727+0000 7f155dffb700 1 -- 172.21.15.90:0/1202463926 >> v1:172.21.15.90:6790/0 conn(0x7f156010f4d0 legacy=0x7f15601a8430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:06.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.727+0000 7f155dffb700 1 -- 172.21.15.90:0/1202463926 >> v1:172.21.15.90:6789/0 conn(0x7f1560103a40 legacy=0x7f156010b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:06.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.727+0000 7f155dffb700 1 -- 172.21.15.90:0/1202463926 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15601ad2f0 con 0x7f156010b930 2024-04-14T15:25:06.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.727+0000 7f1566dd3700 1 -- 172.21.15.90:0/1202463926 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f15601ab0a0 con 0x7f156010b930 2024-04-14T15:25:06.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.727+0000 7f155dffb700 1 -- 172.21.15.90:0/1202463926 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f154c001cc0 con 0x7f156010b930 2024-04-14T15:25:06.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.727+0000 7f155dffb700 1 -- 172.21.15.90:0/1202463926 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f154c005d40 con 0x7f156010b930 2024-04-14T15:25:06.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.727+0000 7f1566dd3700 10 client.49801.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:25:06.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.727+0000 7f1566dd3700 1 -- 172.21.15.90:0/1202463926 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f15601ab560 con 0x7f156010b930 2024-04-14T15:25:06.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.727+0000 7f1566dd3700 1 librados: init done 2024-04-14T15:25:06.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.728+0000 7f1566dd3700 1 -- 172.21.15.90:0/1202463926 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f156006b5d0 con 0x7f156010b930 2024-04-14T15:25:06.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.729+0000 7f155dffb700 1 -- 172.21.15.90:0/1202463926 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f154c01ca40 con 0x7f156010b930 2024-04-14T15:25:06.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.729+0000 7f155dffb700 1 -- 172.21.15.90:0/1202463926 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(762..762 src has 467..762) ==== 8917+0+0 (unknown 2914923142 0 0) 0x7f154c08afe0 con 0x7f156010b930 2024-04-14T15:25:06.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.729+0000 7f155dffb700 10 client.49801.objecter ms_dispatch 0x7f1560001550 osd_map(762..762 src has 467..762) 2024-04-14T15:25:06.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.730+0000 7f155dffb700 3 client.49801.objecter handle_osd_map got epochs [762,762] > 0 2024-04-14T15:25:06.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.730+0000 7f155dffb700 3 client.49801.objecter handle_osd_map decoding full epoch 762 2024-04-14T15:25:06.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.730+0000 7f155dffb700 20 client.49801.objecter dump_active .. 0 homeless 2024-04-14T15:25:06.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.730+0000 7f155dffb700 1 -- 172.21.15.90:0/1202463926 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v762) ==== 54+0+16 (unknown 2383308727 0 2420000192) 0x7f154c08d400 con 0x7f156010b930 2024-04-14T15:25:06.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.730+0000 7f155dffb700 10 client.49801.objecter ms_handle_connect 0x7f1550079b50 2024-04-14T15:25:06.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.730+0000 7f1566dd3700 10 client.49801.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:25:06.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.730+0000 7f1566dd3700 15 client.49801.objecter _session_command_op_assign -1 1 2024-04-14T15:25:06.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.730+0000 7f1566dd3700 20 client.49801.objecter _get_session s=0x7f1560113490 osd=0 3 2024-04-14T15:25:06.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.730+0000 7f1566dd3700 20 client.49801.objecter put_session s=0x7f1560113490 osd=0 3 2024-04-14T15:25:06.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.730+0000 7f1566dd3700 20 client.49801.objecter _get_session s=0x7f1560113490 osd=0 3 2024-04-14T15:25:06.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.730+0000 7f1566dd3700 15 client.49801.objecter _session_command_op_remove -1 1 2024-04-14T15:25:06.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.730+0000 7f1566dd3700 20 client.49801.objecter get_session s=0x7f1560113490 osd=0 3 2024-04-14T15:25:06.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.730+0000 7f1566dd3700 15 client.49801.objecter _session_command_op_assign 0 1 2024-04-14T15:25:06.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.730+0000 7f1566dd3700 20 client.49801.objecter put_session s=0x7f1560113490 osd=0 4 2024-04-14T15:25:06.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.730+0000 7f1566dd3700 10 client.49801.objecter _send_command 1 2024-04-14T15:25:06.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.730+0000 7f1566dd3700 1 -- 172.21.15.90:0/1202463926 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f15600693d0 con 0x7f15601ac280 2024-04-14T15:25:06.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.730+0000 7f155dffb700 10 client.49801.objecter ms_handle_connect 0x7f15601ac280 2024-04-14T15:25:06.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.731+0000 7f155dffb700 1 -- 172.21.15.90:0/1202463926 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f15600693d0 con 0x7f15601ac280 2024-04-14T15:25:06.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.731+0000 7f155dffb700 10 client.49801.objecter ms_dispatch 0x7f1560001550 command_reply(tid 1: 0 ) 2024-04-14T15:25:06.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.731+0000 7f155dffb700 10 client.49801.objecter _finish_command 1 = system:0 2024-04-14T15:25:06.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.732+0000 7f155dffb700 20 client.49801.objecter put_session s=0x7f1560113490 osd=0 4 2024-04-14T15:25:06.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.732+0000 7f155dffb700 15 client.49801.objecter _session_command_op_remove 0 1 2024-04-14T15:25:06.739 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:25:06.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.750+0000 7f1566dd3700 10 client.49801.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:25:06.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.750+0000 7f1566dd3700 15 client.49801.objecter _session_command_op_assign -1 2 2024-04-14T15:25:06.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.750+0000 7f1566dd3700 20 client.49801.objecter _get_session s=0x7f1560113490 osd=0 3 2024-04-14T15:25:06.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.750+0000 7f1566dd3700 20 client.49801.objecter put_session s=0x7f1560113490 osd=0 3 2024-04-14T15:25:06.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.750+0000 7f1566dd3700 20 client.49801.objecter _get_session s=0x7f1560113490 osd=0 3 2024-04-14T15:25:06.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.750+0000 7f1566dd3700 15 client.49801.objecter _session_command_op_remove -1 2 2024-04-14T15:25:06.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.750+0000 7f1566dd3700 20 client.49801.objecter get_session s=0x7f1560113490 osd=0 3 2024-04-14T15:25:06.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.750+0000 7f1566dd3700 15 client.49801.objecter _session_command_op_assign 0 2 2024-04-14T15:25:06.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.750+0000 7f1566dd3700 20 client.49801.objecter put_session s=0x7f1560113490 osd=0 4 2024-04-14T15:25:06.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.750+0000 7f1566dd3700 10 client.49801.objecter _send_command 2 2024-04-14T15:25:06.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.750+0000 7f1566dd3700 1 -- 172.21.15.90:0/1202463926 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f1560102bd0 con 0x7f15601ac280 2024-04-14T15:25:06.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.751+0000 7f155dffb700 1 -- 172.21.15.90:0/1202463926 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1560102bd0 con 0x7f15601ac280 2024-04-14T15:25:06.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.751+0000 7f155dffb700 10 client.49801.objecter ms_dispatch 0x7f1560001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:25:06.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.751+0000 7f155dffb700 10 client.49801.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:25:06.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.751+0000 7f155dffb700 20 client.49801.objecter put_session s=0x7f1560113490 osd=0 4 2024-04-14T15:25:06.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.751+0000 7f155dffb700 15 client.49801.objecter _session_command_op_remove 0 2 2024-04-14T15:25:06.752 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:25:06.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.752+0000 7f1566dd3700 10 client.49801.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:25:06.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.752+0000 7f1566dd3700 15 client.49801.objecter _session_command_op_assign -1 3 2024-04-14T15:25:06.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.752+0000 7f1566dd3700 20 client.49801.objecter _get_session s=0x7f15601acfc0 osd=1 3 2024-04-14T15:25:06.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.752+0000 7f1566dd3700 20 client.49801.objecter put_session s=0x7f15601acfc0 osd=1 3 2024-04-14T15:25:06.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.752+0000 7f1566dd3700 20 client.49801.objecter _get_session s=0x7f15601acfc0 osd=1 3 2024-04-14T15:25:06.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.752+0000 7f1566dd3700 15 client.49801.objecter _session_command_op_remove -1 3 2024-04-14T15:25:06.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.752+0000 7f1566dd3700 20 client.49801.objecter get_session s=0x7f15601acfc0 osd=1 3 2024-04-14T15:25:06.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.752+0000 7f1566dd3700 15 client.49801.objecter _session_command_op_assign 1 3 2024-04-14T15:25:06.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.752+0000 7f1566dd3700 20 client.49801.objecter put_session s=0x7f15601acfc0 osd=1 4 2024-04-14T15:25:06.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.752+0000 7f1566dd3700 10 client.49801.objecter _send_command 3 2024-04-14T15:25:06.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.752+0000 7f1566dd3700 1 -- 172.21.15.90:0/1202463926 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f156019f8e0 con 0x7f156010c220 2024-04-14T15:25:06.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.752+0000 7f155dffb700 10 client.49801.objecter ms_handle_connect 0x7f156010c220 2024-04-14T15:25:06.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.753+0000 7f155dffb700 1 -- 172.21.15.90:0/1202463926 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f156019f8e0 con 0x7f156010c220 2024-04-14T15:25:06.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.753+0000 7f155dffb700 10 client.49801.objecter ms_dispatch 0x7f1560001550 command_reply(tid 3: 0 ) 2024-04-14T15:25:06.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.753+0000 7f155dffb700 10 client.49801.objecter _finish_command 3 = system:0 2024-04-14T15:25:06.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.753+0000 7f155dffb700 20 client.49801.objecter put_session s=0x7f15601acfc0 osd=1 4 2024-04-14T15:25:06.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.753+0000 7f155dffb700 15 client.49801.objecter _session_command_op_remove 1 3 2024-04-14T15:25:06.763 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:06.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:06.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:25:06.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.769+0000 7f1566dd3700 10 client.49801.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:25:06.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.769+0000 7f1566dd3700 15 client.49801.objecter _session_command_op_assign -1 4 2024-04-14T15:25:06.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.769+0000 7f1566dd3700 20 client.49801.objecter _get_session s=0x7f15601acfc0 osd=1 3 2024-04-14T15:25:06.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.769+0000 7f1566dd3700 20 client.49801.objecter put_session s=0x7f15601acfc0 osd=1 3 2024-04-14T15:25:06.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.769+0000 7f1566dd3700 20 client.49801.objecter _get_session s=0x7f15601acfc0 osd=1 3 2024-04-14T15:25:06.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.769+0000 7f1566dd3700 15 client.49801.objecter _session_command_op_remove -1 4 2024-04-14T15:25:06.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.769+0000 7f1566dd3700 20 client.49801.objecter get_session s=0x7f15601acfc0 osd=1 3 2024-04-14T15:25:06.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.769+0000 7f1566dd3700 15 client.49801.objecter _session_command_op_assign 1 4 2024-04-14T15:25:06.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.769+0000 7f1566dd3700 20 client.49801.objecter put_session s=0x7f15601acfc0 osd=1 4 2024-04-14T15:25:06.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.769+0000 7f1566dd3700 10 client.49801.objecter _send_command 4 2024-04-14T15:25:06.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.769+0000 7f1566dd3700 1 -- 172.21.15.90:0/1202463926 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f156019fac0 con 0x7f156010c220 2024-04-14T15:25:06.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.771+0000 7f155dffb700 1 -- 172.21.15.90:0/1202463926 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f156019fac0 con 0x7f156010c220 2024-04-14T15:25:06.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.771+0000 7f155dffb700 10 client.49801.objecter ms_dispatch 0x7f1560001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:25:06.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.771+0000 7f155dffb700 10 client.49801.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:25:06.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.771+0000 7f155dffb700 20 client.49801.objecter put_session s=0x7f15601acfc0 osd=1 4 2024-04-14T15:25:06.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.771+0000 7f155dffb700 15 client.49801.objecter _session_command_op_remove 1 4 2024-04-14T15:25:06.772 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:25:06.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.771+0000 7f1566dd3700 10 client.49801.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:25:06.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.771+0000 7f1566dd3700 15 client.49801.objecter _session_command_op_assign -1 5 2024-04-14T15:25:06.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.771+0000 7f1566dd3700 20 client.49801.objecter _get_session s=0x7f156002e2d0 osd=2 3 2024-04-14T15:25:06.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.771+0000 7f1566dd3700 20 client.49801.objecter put_session s=0x7f156002e2d0 osd=2 3 2024-04-14T15:25:06.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.771+0000 7f1566dd3700 20 client.49801.objecter _get_session s=0x7f156002e2d0 osd=2 3 2024-04-14T15:25:06.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.771+0000 7f1566dd3700 15 client.49801.objecter _session_command_op_remove -1 5 2024-04-14T15:25:06.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.771+0000 7f1566dd3700 20 client.49801.objecter get_session s=0x7f156002e2d0 osd=2 3 2024-04-14T15:25:06.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.771+0000 7f1566dd3700 15 client.49801.objecter _session_command_op_assign 2 5 2024-04-14T15:25:06.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.771+0000 7f1566dd3700 20 client.49801.objecter put_session s=0x7f156002e2d0 osd=2 4 2024-04-14T15:25:06.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.771+0000 7f1566dd3700 10 client.49801.objecter _send_command 5 2024-04-14T15:25:06.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.771+0000 7f1566dd3700 1 -- 172.21.15.90:0/1202463926 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f15601b15a0 con 0x7f15601ada90 2024-04-14T15:25:06.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.772+0000 7f155dffb700 10 client.49801.objecter ms_handle_connect 0x7f15601ada90 2024-04-14T15:25:06.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.772+0000 7f155dffb700 1 -- 172.21.15.90:0/1202463926 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f15601b15a0 con 0x7f15601ada90 2024-04-14T15:25:06.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.772+0000 7f155dffb700 10 client.49801.objecter ms_dispatch 0x7f1560001550 command_reply(tid 5: 0 ) 2024-04-14T15:25:06.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.772+0000 7f155dffb700 10 client.49801.objecter _finish_command 5 = system:0 2024-04-14T15:25:06.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.772+0000 7f155dffb700 20 client.49801.objecter put_session s=0x7f156002e2d0 osd=2 4 2024-04-14T15:25:06.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.772+0000 7f155dffb700 15 client.49801.objecter _session_command_op_remove 2 5 2024-04-14T15:25:06.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.789+0000 7f1566dd3700 10 client.49801.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:25:06.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.789+0000 7f1566dd3700 15 client.49801.objecter _session_command_op_assign -1 6 2024-04-14T15:25:06.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.789+0000 7f1566dd3700 20 client.49801.objecter _get_session s=0x7f156002e2d0 osd=2 3 2024-04-14T15:25:06.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.789+0000 7f1566dd3700 20 client.49801.objecter put_session s=0x7f156002e2d0 osd=2 3 2024-04-14T15:25:06.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.789+0000 7f1566dd3700 20 client.49801.objecter _get_session s=0x7f156002e2d0 osd=2 3 2024-04-14T15:25:06.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.789+0000 7f1566dd3700 15 client.49801.objecter _session_command_op_remove -1 6 2024-04-14T15:25:06.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.789+0000 7f1566dd3700 20 client.49801.objecter get_session s=0x7f156002e2d0 osd=2 3 2024-04-14T15:25:06.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.789+0000 7f1566dd3700 15 client.49801.objecter _session_command_op_assign 2 6 2024-04-14T15:25:06.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.789+0000 7f1566dd3700 20 client.49801.objecter put_session s=0x7f156002e2d0 osd=2 4 2024-04-14T15:25:06.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.789+0000 7f1566dd3700 10 client.49801.objecter _send_command 6 2024-04-14T15:25:06.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.789+0000 7f1566dd3700 1 -- 172.21.15.90:0/1202463926 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f15600ff440 con 0x7f15601ada90 2024-04-14T15:25:06.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.790+0000 7f155dffb700 1 -- 172.21.15.90:0/1202463926 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f15600ff440 con 0x7f15601ada90 2024-04-14T15:25:06.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.790+0000 7f155dffb700 10 client.49801.objecter ms_dispatch 0x7f1560001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:25:06.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.790+0000 7f155dffb700 10 client.49801.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:25:06.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.790+0000 7f155dffb700 20 client.49801.objecter put_session s=0x7f156002e2d0 osd=2 4 2024-04-14T15:25:06.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.790+0000 7f155dffb700 15 client.49801.objecter _session_command_op_remove 2 6 2024-04-14T15:25:06.791 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:25:06.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.790+0000 7f1566dd3700 10 client.49801.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:25:06.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.790+0000 7f1566dd3700 15 client.49801.objecter _session_command_op_assign -1 7 2024-04-14T15:25:06.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.790+0000 7f1566dd3700 20 client.49801.objecter _get_session s=0x7f156002e440 osd=3 3 2024-04-14T15:25:06.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.790+0000 7f1566dd3700 20 client.49801.objecter put_session s=0x7f156002e440 osd=3 3 2024-04-14T15:25:06.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.790+0000 7f1566dd3700 20 client.49801.objecter _get_session s=0x7f156002e440 osd=3 3 2024-04-14T15:25:06.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.790+0000 7f1566dd3700 15 client.49801.objecter _session_command_op_remove -1 7 2024-04-14T15:25:06.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.790+0000 7f1566dd3700 20 client.49801.objecter get_session s=0x7f156002e440 osd=3 3 2024-04-14T15:25:06.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.790+0000 7f1566dd3700 15 client.49801.objecter _session_command_op_assign 3 7 2024-04-14T15:25:06.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.790+0000 7f1566dd3700 20 client.49801.objecter put_session s=0x7f156002e440 osd=3 4 2024-04-14T15:25:06.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.790+0000 7f1566dd3700 10 client.49801.objecter _send_command 7 2024-04-14T15:25:06.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.790+0000 7f1566dd3700 1 -- 172.21.15.90:0/1202463926 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f15601b6000 con 0x7f1560063bd0 2024-04-14T15:25:06.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.791+0000 7f155dffb700 10 client.49801.objecter ms_handle_connect 0x7f1560063bd0 2024-04-14T15:25:06.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.791+0000 7f155dffb700 1 -- 172.21.15.90:0/1202463926 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f15601b6000 con 0x7f1560063bd0 2024-04-14T15:25:06.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.791+0000 7f155dffb700 10 client.49801.objecter ms_dispatch 0x7f1560001550 command_reply(tid 7: 0 ) 2024-04-14T15:25:06.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.791+0000 7f155dffb700 10 client.49801.objecter _finish_command 7 = system:0 2024-04-14T15:25:06.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.791+0000 7f155dffb700 20 client.49801.objecter put_session s=0x7f156002e440 osd=3 4 2024-04-14T15:25:06.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.791+0000 7f155dffb700 15 client.49801.objecter _session_command_op_remove 3 7 2024-04-14T15:25:06.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.808+0000 7f1566dd3700 10 client.49801.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:25:06.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.808+0000 7f1566dd3700 15 client.49801.objecter _session_command_op_assign -1 8 2024-04-14T15:25:06.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.808+0000 7f1566dd3700 20 client.49801.objecter _get_session s=0x7f156002e440 osd=3 3 2024-04-14T15:25:06.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.808+0000 7f1566dd3700 20 client.49801.objecter put_session s=0x7f156002e440 osd=3 3 2024-04-14T15:25:06.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.808+0000 7f1566dd3700 20 client.49801.objecter _get_session s=0x7f156002e440 osd=3 3 2024-04-14T15:25:06.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.808+0000 7f1566dd3700 15 client.49801.objecter _session_command_op_remove -1 8 2024-04-14T15:25:06.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.808+0000 7f1566dd3700 20 client.49801.objecter get_session s=0x7f156002e440 osd=3 3 2024-04-14T15:25:06.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.808+0000 7f1566dd3700 15 client.49801.objecter _session_command_op_assign 3 8 2024-04-14T15:25:06.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.808+0000 7f1566dd3700 20 client.49801.objecter put_session s=0x7f156002e440 osd=3 4 2024-04-14T15:25:06.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.808+0000 7f1566dd3700 10 client.49801.objecter _send_command 8 2024-04-14T15:25:06.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.808+0000 7f1566dd3700 1 -- 172.21.15.90:0/1202463926 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f1560063200 con 0x7f1560063bd0 2024-04-14T15:25:06.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.809+0000 7f155dffb700 1 -- 172.21.15.90:0/1202463926 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1560063200 con 0x7f1560063bd0 2024-04-14T15:25:06.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.809+0000 7f155dffb700 10 client.49801.objecter ms_dispatch 0x7f1560001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:25:06.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.809+0000 7f155dffb700 10 client.49801.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:25:06.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.809+0000 7f155dffb700 20 client.49801.objecter put_session s=0x7f156002e440 osd=3 4 2024-04-14T15:25:06.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.809+0000 7f155dffb700 15 client.49801.objecter _session_command_op_remove 3 8 2024-04-14T15:25:06.810 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:25:06.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.810+0000 7f1566dd3700 10 client.49801.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:25:06.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.810+0000 7f1566dd3700 15 client.49801.objecter _session_command_op_assign -1 9 2024-04-14T15:25:06.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.810+0000 7f1566dd3700 20 client.49801.objecter _get_session s=0x7f1560062fb0 osd=4 3 2024-04-14T15:25:06.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.810+0000 7f1566dd3700 20 client.49801.objecter put_session s=0x7f1560062fb0 osd=4 3 2024-04-14T15:25:06.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.810+0000 7f1566dd3700 20 client.49801.objecter _get_session s=0x7f1560062fb0 osd=4 3 2024-04-14T15:25:06.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.810+0000 7f1566dd3700 15 client.49801.objecter _session_command_op_remove -1 9 2024-04-14T15:25:06.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.810+0000 7f1566dd3700 20 client.49801.objecter get_session s=0x7f1560062fb0 osd=4 3 2024-04-14T15:25:06.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.810+0000 7f1566dd3700 15 client.49801.objecter _session_command_op_assign 4 9 2024-04-14T15:25:06.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.810+0000 7f1566dd3700 20 client.49801.objecter put_session s=0x7f1560062fb0 osd=4 4 2024-04-14T15:25:06.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.810+0000 7f1566dd3700 10 client.49801.objecter _send_command 9 2024-04-14T15:25:06.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.810+0000 7f1566dd3700 1 -- 172.21.15.90:0/1202463926 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f15600631e0 con 0x7f15601b2430 2024-04-14T15:25:06.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.810+0000 7f155dffb700 10 client.49801.objecter ms_handle_connect 0x7f15601b2430 2024-04-14T15:25:06.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.811+0000 7f155dffb700 1 -- 172.21.15.90:0/1202463926 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f15600631e0 con 0x7f15601b2430 2024-04-14T15:25:06.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.811+0000 7f155dffb700 10 client.49801.objecter ms_dispatch 0x7f1560001550 command_reply(tid 9: 0 ) 2024-04-14T15:25:06.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.811+0000 7f155dffb700 10 client.49801.objecter _finish_command 9 = system:0 2024-04-14T15:25:06.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.811+0000 7f155dffb700 20 client.49801.objecter put_session s=0x7f1560062fb0 osd=4 4 2024-04-14T15:25:06.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.811+0000 7f155dffb700 15 client.49801.objecter _session_command_op_remove 4 9 2024-04-14T15:25:06.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.827+0000 7f1566dd3700 10 client.49801.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:25:06.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.827+0000 7f1566dd3700 15 client.49801.objecter _session_command_op_assign -1 10 2024-04-14T15:25:06.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.827+0000 7f1566dd3700 20 client.49801.objecter _get_session s=0x7f1560062fb0 osd=4 3 2024-04-14T15:25:06.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.827+0000 7f1566dd3700 20 client.49801.objecter put_session s=0x7f1560062fb0 osd=4 3 2024-04-14T15:25:06.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.827+0000 7f1566dd3700 20 client.49801.objecter _get_session s=0x7f1560062fb0 osd=4 3 2024-04-14T15:25:06.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.827+0000 7f1566dd3700 15 client.49801.objecter _session_command_op_remove -1 10 2024-04-14T15:25:06.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.827+0000 7f1566dd3700 20 client.49801.objecter get_session s=0x7f1560062fb0 osd=4 3 2024-04-14T15:25:06.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.827+0000 7f1566dd3700 15 client.49801.objecter _session_command_op_assign 4 10 2024-04-14T15:25:06.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.827+0000 7f1566dd3700 20 client.49801.objecter put_session s=0x7f1560062fb0 osd=4 4 2024-04-14T15:25:06.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.827+0000 7f1566dd3700 10 client.49801.objecter _send_command 10 2024-04-14T15:25:06.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.827+0000 7f1566dd3700 1 -- 172.21.15.90:0/1202463926 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f15601bc330 con 0x7f15601b2430 2024-04-14T15:25:06.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.829+0000 7f155dffb700 1 -- 172.21.15.90:0/1202463926 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f15601bc330 con 0x7f15601b2430 2024-04-14T15:25:06.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.829+0000 7f155dffb700 10 client.49801.objecter ms_dispatch 0x7f1560001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:25:06.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.829+0000 7f155dffb700 10 client.49801.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:25:06.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.829+0000 7f155dffb700 20 client.49801.objecter put_session s=0x7f1560062fb0 osd=4 4 2024-04-14T15:25:06.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.829+0000 7f155dffb700 15 client.49801.objecter _session_command_op_remove 4 10 2024-04-14T15:25:06.830 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:25:06.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.830+0000 7f1566dd3700 10 client.49801.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:25:06.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.830+0000 7f1566dd3700 15 client.49801.objecter _session_command_op_assign -1 11 2024-04-14T15:25:06.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.830+0000 7f1566dd3700 20 client.49801.objecter _get_session s=0x7f1560063240 osd=5 3 2024-04-14T15:25:06.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.830+0000 7f1566dd3700 20 client.49801.objecter put_session s=0x7f1560063240 osd=5 3 2024-04-14T15:25:06.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.830+0000 7f1566dd3700 20 client.49801.objecter _get_session s=0x7f1560063240 osd=5 3 2024-04-14T15:25:06.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.830+0000 7f1566dd3700 15 client.49801.objecter _session_command_op_remove -1 11 2024-04-14T15:25:06.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.830+0000 7f1566dd3700 20 client.49801.objecter get_session s=0x7f1560063240 osd=5 3 2024-04-14T15:25:06.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.830+0000 7f1566dd3700 15 client.49801.objecter _session_command_op_assign 5 11 2024-04-14T15:25:06.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.830+0000 7f1566dd3700 20 client.49801.objecter put_session s=0x7f1560063240 osd=5 4 2024-04-14T15:25:06.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.830+0000 7f1566dd3700 10 client.49801.objecter _send_command 11 2024-04-14T15:25:06.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.830+0000 7f1566dd3700 1 -- 172.21.15.90:0/1202463926 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f15601bceb0 con 0x7f15601bca50 2024-04-14T15:25:06.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.830+0000 7f155dffb700 10 client.49801.objecter ms_handle_connect 0x7f15601bca50 2024-04-14T15:25:06.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.831+0000 7f155dffb700 1 -- 172.21.15.90:0/1202463926 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f15601bceb0 con 0x7f15601bca50 2024-04-14T15:25:06.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.831+0000 7f155dffb700 10 client.49801.objecter ms_dispatch 0x7f1560001550 command_reply(tid 11: 0 ) 2024-04-14T15:25:06.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.831+0000 7f155dffb700 10 client.49801.objecter _finish_command 11 = system:0 2024-04-14T15:25:06.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.831+0000 7f155dffb700 20 client.49801.objecter put_session s=0x7f1560063240 osd=5 4 2024-04-14T15:25:06.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.831+0000 7f155dffb700 15 client.49801.objecter _session_command_op_remove 5 11 2024-04-14T15:25:06.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.847+0000 7f1566dd3700 10 client.49801.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:25:06.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.847+0000 7f1566dd3700 15 client.49801.objecter _session_command_op_assign -1 12 2024-04-14T15:25:06.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.847+0000 7f1566dd3700 20 client.49801.objecter _get_session s=0x7f1560063240 osd=5 3 2024-04-14T15:25:06.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.847+0000 7f1566dd3700 20 client.49801.objecter put_session s=0x7f1560063240 osd=5 3 2024-04-14T15:25:06.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.847+0000 7f1566dd3700 20 client.49801.objecter _get_session s=0x7f1560063240 osd=5 3 2024-04-14T15:25:06.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.847+0000 7f1566dd3700 15 client.49801.objecter _session_command_op_remove -1 12 2024-04-14T15:25:06.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.847+0000 7f1566dd3700 20 client.49801.objecter get_session s=0x7f1560063240 osd=5 3 2024-04-14T15:25:06.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.847+0000 7f1566dd3700 15 client.49801.objecter _session_command_op_assign 5 12 2024-04-14T15:25:06.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.847+0000 7f1566dd3700 20 client.49801.objecter put_session s=0x7f1560063240 osd=5 4 2024-04-14T15:25:06.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.847+0000 7f1566dd3700 10 client.49801.objecter _send_command 12 2024-04-14T15:25:06.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.847+0000 7f1566dd3700 1 -- 172.21.15.90:0/1202463926 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f15601bd0e0 con 0x7f15601bca50 2024-04-14T15:25:06.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.849+0000 7f155dffb700 1 -- 172.21.15.90:0/1202463926 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f15601bd0e0 con 0x7f15601bca50 2024-04-14T15:25:06.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.849+0000 7f155dffb700 10 client.49801.objecter ms_dispatch 0x7f1560001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:25:06.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.849+0000 7f155dffb700 10 client.49801.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:25:06.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.849+0000 7f155dffb700 20 client.49801.objecter put_session s=0x7f1560063240 osd=5 4 2024-04-14T15:25:06.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.849+0000 7f155dffb700 15 client.49801.objecter _session_command_op_remove 5 12 2024-04-14T15:25:06.850 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:25:06.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.850+0000 7f1566dd3700 10 client.49801.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:25:06.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.850+0000 7f1566dd3700 15 client.49801.objecter _session_command_op_assign -1 13 2024-04-14T15:25:06.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.850+0000 7f1566dd3700 20 client.49801.objecter _get_session s=0x7f15601bceb0 osd=6 3 2024-04-14T15:25:06.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.850+0000 7f1566dd3700 20 client.49801.objecter put_session s=0x7f15601bceb0 osd=6 3 2024-04-14T15:25:06.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.850+0000 7f1566dd3700 20 client.49801.objecter _get_session s=0x7f15601bceb0 osd=6 3 2024-04-14T15:25:06.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.850+0000 7f1566dd3700 15 client.49801.objecter _session_command_op_remove -1 13 2024-04-14T15:25:06.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.850+0000 7f1566dd3700 20 client.49801.objecter get_session s=0x7f15601bceb0 osd=6 3 2024-04-14T15:25:06.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.850+0000 7f1566dd3700 15 client.49801.objecter _session_command_op_assign 6 13 2024-04-14T15:25:06.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.850+0000 7f1566dd3700 20 client.49801.objecter put_session s=0x7f15601bceb0 osd=6 4 2024-04-14T15:25:06.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.850+0000 7f1566dd3700 10 client.49801.objecter _send_command 13 2024-04-14T15:25:06.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.850+0000 7f1566dd3700 1 -- 172.21.15.90:0/1202463926 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f15601bd2d0 con 0x7f15601bfa20 2024-04-14T15:25:06.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.850+0000 7f155dffb700 10 client.49801.objecter ms_handle_connect 0x7f15601bfa20 2024-04-14T15:25:06.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.851+0000 7f155dffb700 1 -- 172.21.15.90:0/1202463926 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f15601bd2d0 con 0x7f15601bfa20 2024-04-14T15:25:06.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.851+0000 7f155dffb700 10 client.49801.objecter ms_dispatch 0x7f1560001550 command_reply(tid 13: 0 ) 2024-04-14T15:25:06.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.851+0000 7f155dffb700 10 client.49801.objecter _finish_command 13 = system:0 2024-04-14T15:25:06.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.851+0000 7f155dffb700 20 client.49801.objecter put_session s=0x7f15601bceb0 osd=6 4 2024-04-14T15:25:06.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.851+0000 7f155dffb700 15 client.49801.objecter _session_command_op_remove 6 13 2024-04-14T15:25:06.865 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:25:06.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.871+0000 7f1566dd3700 10 client.49801.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:25:06.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.871+0000 7f1566dd3700 15 client.49801.objecter _session_command_op_assign -1 14 2024-04-14T15:25:06.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.871+0000 7f1566dd3700 20 client.49801.objecter _get_session s=0x7f15601bceb0 osd=6 3 2024-04-14T15:25:06.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.871+0000 7f1566dd3700 20 client.49801.objecter put_session s=0x7f15601bceb0 osd=6 3 2024-04-14T15:25:06.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.871+0000 7f1566dd3700 20 client.49801.objecter _get_session s=0x7f15601bceb0 osd=6 3 2024-04-14T15:25:06.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.871+0000 7f1566dd3700 15 client.49801.objecter _session_command_op_remove -1 14 2024-04-14T15:25:06.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.871+0000 7f1566dd3700 20 client.49801.objecter get_session s=0x7f15601bceb0 osd=6 3 2024-04-14T15:25:06.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.871+0000 7f1566dd3700 15 client.49801.objecter _session_command_op_assign 6 14 2024-04-14T15:25:06.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.871+0000 7f1566dd3700 20 client.49801.objecter put_session s=0x7f15601bceb0 osd=6 4 2024-04-14T15:25:06.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.871+0000 7f1566dd3700 10 client.49801.objecter _send_command 14 2024-04-14T15:25:06.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.871+0000 7f1566dd3700 1 -- 172.21.15.90:0/1202463926 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f15601c3530 con 0x7f15601bfa20 2024-04-14T15:25:06.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.873+0000 7f155dffb700 1 -- 172.21.15.90:0/1202463926 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f15601c3530 con 0x7f15601bfa20 2024-04-14T15:25:06.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.873+0000 7f155dffb700 10 client.49801.objecter ms_dispatch 0x7f1560001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:25:06.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.873+0000 7f155dffb700 10 client.49801.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:25:06.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.873+0000 7f155dffb700 20 client.49801.objecter put_session s=0x7f15601bceb0 osd=6 4 2024-04-14T15:25:06.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.873+0000 7f155dffb700 15 client.49801.objecter _session_command_op_remove 6 14 2024-04-14T15:25:06.874 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:25:06.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.873+0000 7f1566dd3700 10 client.49801.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:25:06.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.873+0000 7f1566dd3700 15 client.49801.objecter _session_command_op_assign -1 15 2024-04-14T15:25:06.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.873+0000 7f1566dd3700 20 client.49801.objecter _get_session s=0x7f15601c3530 osd=7 3 2024-04-14T15:25:06.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.873+0000 7f1566dd3700 20 client.49801.objecter put_session s=0x7f15601c3530 osd=7 3 2024-04-14T15:25:06.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.873+0000 7f1566dd3700 20 client.49801.objecter _get_session s=0x7f15601c3530 osd=7 3 2024-04-14T15:25:06.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.873+0000 7f1566dd3700 15 client.49801.objecter _session_command_op_remove -1 15 2024-04-14T15:25:06.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.873+0000 7f1566dd3700 20 client.49801.objecter get_session s=0x7f15601c3530 osd=7 3 2024-04-14T15:25:06.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.873+0000 7f1566dd3700 15 client.49801.objecter _session_command_op_assign 7 15 2024-04-14T15:25:06.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.873+0000 7f1566dd3700 20 client.49801.objecter put_session s=0x7f15601c3530 osd=7 4 2024-04-14T15:25:06.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.873+0000 7f1566dd3700 10 client.49801.objecter _send_command 15 2024-04-14T15:25:06.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.873+0000 7f1566dd3700 1 -- 172.21.15.90:0/1202463926 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f15601c4860 con 0x7f15601c3c50 2024-04-14T15:25:06.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.873+0000 7f155dffb700 10 client.49801.objecter ms_handle_connect 0x7f15601c3c50 2024-04-14T15:25:06.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.874+0000 7f155dffb700 1 -- 172.21.15.90:0/1202463926 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f15601c4860 con 0x7f15601c3c50 2024-04-14T15:25:06.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.874+0000 7f155dffb700 10 client.49801.objecter ms_dispatch 0x7f1560001550 command_reply(tid 15: 0 ) 2024-04-14T15:25:06.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.874+0000 7f155dffb700 10 client.49801.objecter _finish_command 15 = system:0 2024-04-14T15:25:06.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.874+0000 7f155dffb700 20 client.49801.objecter put_session s=0x7f15601c3530 osd=7 4 2024-04-14T15:25:06.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.874+0000 7f155dffb700 15 client.49801.objecter _session_command_op_remove 7 15 2024-04-14T15:25:06.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:06.891 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:06.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:25:06.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.893+0000 7f1566dd3700 10 client.49801.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:25:06.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.893+0000 7f1566dd3700 15 client.49801.objecter _session_command_op_assign -1 16 2024-04-14T15:25:06.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.893+0000 7f1566dd3700 20 client.49801.objecter _get_session s=0x7f15601c3530 osd=7 3 2024-04-14T15:25:06.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.893+0000 7f1566dd3700 20 client.49801.objecter put_session s=0x7f15601c3530 osd=7 3 2024-04-14T15:25:06.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.893+0000 7f1566dd3700 20 client.49801.objecter _get_session s=0x7f15601c3530 osd=7 3 2024-04-14T15:25:06.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.893+0000 7f1566dd3700 15 client.49801.objecter _session_command_op_remove -1 16 2024-04-14T15:25:06.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.893+0000 7f1566dd3700 20 client.49801.objecter get_session s=0x7f15601c3530 osd=7 3 2024-04-14T15:25:06.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.893+0000 7f1566dd3700 15 client.49801.objecter _session_command_op_assign 7 16 2024-04-14T15:25:06.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.893+0000 7f1566dd3700 20 client.49801.objecter put_session s=0x7f15601c3530 osd=7 4 2024-04-14T15:25:06.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.893+0000 7f1566dd3700 10 client.49801.objecter _send_command 16 2024-04-14T15:25:06.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.893+0000 7f1566dd3700 1 -- 172.21.15.90:0/1202463926 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f15601c4860 con 0x7f15601c3c50 2024-04-14T15:25:06.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.895+0000 7f155dffb700 1 -- 172.21.15.90:0/1202463926 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f15601c4860 con 0x7f15601c3c50 2024-04-14T15:25:06.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.895+0000 7f155dffb700 10 client.49801.objecter ms_dispatch 0x7f1560001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:25:06.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.895+0000 7f155dffb700 10 client.49801.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:25:06.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.895+0000 7f155dffb700 20 client.49801.objecter put_session s=0x7f15601c3530 osd=7 4 2024-04-14T15:25:06.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.895+0000 7f155dffb700 15 client.49801.objecter _session_command_op_remove 7 16 2024-04-14T15:25:06.896 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:25:06.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.896+0000 7f1566dd3700 10 librados: watch_flush enter 2024-04-14T15:25:06.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.896+0000 7f1566dd3700 10 librados: watch_flush exit 2024-04-14T15:25:06.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.896+0000 7f1566dd3700 10 client.49801.objecter close_session for osd.0 2024-04-14T15:25:06.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.896+0000 7f1566dd3700 1 -- 172.21.15.90:0/1202463926 >> v1:172.21.15.90:6801/3761806885 conn(0x7f15601ac280 legacy=0x7f15601ac660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:06.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.896+0000 7f1566dd3700 20 client.49801.objecter put_session s=0x7f1560113490 osd=0 1 2024-04-14T15:25:06.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.896+0000 7f1566dd3700 10 client.49801.objecter close_session for osd.1 2024-04-14T15:25:06.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.896+0000 7f1566dd3700 1 -- 172.21.15.90:0/1202463926 >> v1:172.21.15.90:6805/4170361699 conn(0x7f156010c220 legacy=0x7f156010c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:06.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.896+0000 7f1566dd3700 20 client.49801.objecter put_session s=0x7f15601acfc0 osd=1 1 2024-04-14T15:25:06.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.896+0000 7f1566dd3700 10 client.49801.objecter close_session for osd.2 2024-04-14T15:25:06.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.896+0000 7f1566dd3700 1 -- 172.21.15.90:0/1202463926 >> v1:172.21.15.90:6809/1686184251 conn(0x7f15601ada90 legacy=0x7f15601afe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:06.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.896+0000 7f1566dd3700 20 client.49801.objecter put_session s=0x7f156002e2d0 osd=2 1 2024-04-14T15:25:06.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.896+0000 7f1566dd3700 10 client.49801.objecter close_session for osd.3 2024-04-14T15:25:06.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.896+0000 7f1566dd3700 1 -- 172.21.15.90:0/1202463926 >> v1:172.21.15.90:6813/2270166774 conn(0x7f1560063bd0 legacy=0x7f15601b48e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:06.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.896+0000 7f1566dd3700 20 client.49801.objecter put_session s=0x7f156002e440 osd=3 1 2024-04-14T15:25:06.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.896+0000 7f1566dd3700 10 client.49801.objecter close_session for osd.4 2024-04-14T15:25:06.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.896+0000 7f1566dd3700 1 -- 172.21.15.90:0/1202463926 >> v1:172.21.15.92:6800/3887549733 conn(0x7f15601b2430 legacy=0x7f15601b7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:06.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.896+0000 7f1566dd3700 20 client.49801.objecter put_session s=0x7f1560062fb0 osd=4 1 2024-04-14T15:25:06.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.896+0000 7f1566dd3700 10 client.49801.objecter close_session for osd.5 2024-04-14T15:25:06.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.896+0000 7f1566dd3700 1 -- 172.21.15.90:0/1202463926 >> v1:172.21.15.92:6804/3244561440 conn(0x7f15601bca50 legacy=0x7f15601bab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:06.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.896+0000 7f1566dd3700 20 client.49801.objecter put_session s=0x7f1560063240 osd=5 1 2024-04-14T15:25:06.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.896+0000 7f1566dd3700 10 client.49801.objecter close_session for osd.6 2024-04-14T15:25:06.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.896+0000 7f1566dd3700 1 -- 172.21.15.90:0/1202463926 >> v1:172.21.15.92:6808/636125963 conn(0x7f15601bfa20 legacy=0x7f15601c1e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:06.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.896+0000 7f1566dd3700 20 client.49801.objecter put_session s=0x7f15601bceb0 osd=6 1 2024-04-14T15:25:06.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.896+0000 7f1565370700 1 -- 172.21.15.90:0/1202463926 reap_dead start 2024-04-14T15:25:06.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.896+0000 7f1566dd3700 10 client.49801.objecter close_session for osd.7 2024-04-14T15:25:06.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.896+0000 7f1566dd3700 1 -- 172.21.15.90:0/1202463926 >> v1:172.21.15.92:6812/1773706716 conn(0x7f15601c3c50 legacy=0x7f15601c40b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:06.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.896+0000 7f1565370700 1 -- 172.21.15.90:0/1202463926 reap_dead start 2024-04-14T15:25:06.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.896+0000 7f1565370700 1 -- 172.21.15.90:0/1202463926 reap_dead start 2024-04-14T15:25:06.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.896+0000 7f1565370700 1 -- 172.21.15.90:0/1202463926 reap_dead start 2024-04-14T15:25:06.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.896+0000 7f1566dd3700 20 client.49801.objecter put_session s=0x7f15601c3530 osd=7 1 2024-04-14T15:25:06.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.896+0000 7f1566dd3700 20 client.49801.objecter shutdown clearing up homeless session... 2024-04-14T15:25:06.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.896+0000 7f1566dd3700 10 client.49801.objecter successfully canceled tick 2024-04-14T15:25:06.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.896+0000 7f1566dd3700 1 -- 172.21.15.90:0/1202463926 >> v1:172.21.15.90:6800/420935427 conn(0x7f1550079b50 legacy=0x7f155007c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:06.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.896+0000 7f1566dd3700 1 -- 172.21.15.90:0/1202463926 >> v1:172.21.15.92:6789/0 conn(0x7f156010b930 legacy=0x7f15601a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:06.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.896+0000 7f1566dd3700 1 -- 172.21.15.90:0/1202463926 shutdown_connections 2024-04-14T15:25:06.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.896+0000 7f1566dd3700 1 -- 172.21.15.90:0/1202463926 >> 172.21.15.90:0/1202463926 conn(0x7f15600fd290 msgr2=0x7f1560111950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:25:06.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.896+0000 7f1566dd3700 1 -- 172.21.15.90:0/1202463926 shutdown_connections 2024-04-14T15:25:06.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.896+0000 7f1566dd3700 1 -- 172.21.15.90:0/1202463926 wait complete. 2024-04-14T15:25:06.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:06.896+0000 7f1566dd3700 1 librados: shutdown 2024-04-14T15:25:06.900 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:25:06.900 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:25:06.900 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:25:06.900 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:25:06.901 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:25:06.901 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:25:06.901 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:25:06.901 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:25:06.991 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:25:07.018 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:07.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:07.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:25:07.123 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:25:07.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:07.147 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:07.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:25:07.248 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:07.272 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:07.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:07.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:07.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:07 smithi092 ceph-mon[33966]: 1.7 scrub starts 2024-04-14T15:25:07.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:07 smithi092 ceph-mon[33966]: 1.7 scrub ok 2024-04-14T15:25:07.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:07 smithi092 ceph-mon[33966]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-04-14T15:25:07.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:07 smithi092 ceph-mon[33966]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 3 pgs peering) 2024-04-14T15:25:07.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:07 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1202463926' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:25:07.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:07 smithi092 ceph-mon[33966]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:25:07.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:07 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-134", "mode": "writeback"}]': finished 2024-04-14T15:25:07.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:07 smithi092 ceph-mon[33966]: osdmap e763: 8 total, 8 up, 7 in 2024-04-14T15:25:07.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:07 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:25:07.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:07 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:25:07.374 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:25:07.397 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:07.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:07.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:25:07.436 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:07 smithi090 ceph-mon[41670]: 1.7 scrub starts 2024-04-14T15:25:07.436 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:07 smithi090 ceph-mon[41670]: 1.7 scrub ok 2024-04-14T15:25:07.436 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:07 smithi090 ceph-mon[41670]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-04-14T15:25:07.436 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:07 smithi090 ceph-mon[41670]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 3 pgs peering) 2024-04-14T15:25:07.436 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:07 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1202463926' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:25:07.436 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:07 smithi090 ceph-mon[41670]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:25:07.437 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:07 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-134", "mode": "writeback"}]': finished 2024-04-14T15:25:07.437 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:07 smithi090 ceph-mon[41670]: osdmap e763: 8 total, 8 up, 7 in 2024-04-14T15:25:07.437 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:07 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:25:07.437 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:07 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:25:07.437 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:07 smithi090 ceph-mon[30399]: 1.7 scrub starts 2024-04-14T15:25:07.437 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:07 smithi090 ceph-mon[30399]: 1.7 scrub ok 2024-04-14T15:25:07.437 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:07 smithi090 ceph-mon[30399]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-04-14T15:25:07.437 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:07 smithi090 ceph-mon[30399]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 3 pgs peering) 2024-04-14T15:25:07.437 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:07 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1202463926' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:25:07.437 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:07 smithi090 ceph-mon[30399]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:25:07.437 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:07 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-134", "mode": "writeback"}]': finished 2024-04-14T15:25:07.437 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:07 smithi090 ceph-mon[30399]: osdmap e763: 8 total, 8 up, 7 in 2024-04-14T15:25:07.437 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:07 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:25:07.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:07 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:25:07.499 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:25:07.522 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:07.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:07.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:25:07.624 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:25:07.647 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:07.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:07.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:25:07.748 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:25:07.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:07.773 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:07.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:25:07.874 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:25:07.898 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:07.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:07.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:25:08.000 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:25:08.024 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:08.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:08.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:25:08.126 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:25:08.150 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:08.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:08.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:25:08.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:08 smithi090 ceph-mon[41670]: 1.a scrub starts 2024-04-14T15:25:08.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:08 smithi090 ceph-mon[41670]: 1.a scrub ok 2024-04-14T15:25:08.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:08 smithi090 ceph-mon[41670]: pgmap v1129: 224 pgs: 4 creating+activating, 4 active+remapped+backfill_toofull, 216 active+clean; 489 KiB data, 715 MiB used, 625 GiB / 626 GiB avail; 2/642 objects misplaced (0.312%) 2024-04-14T15:25:08.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:08 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]': finished 2024-04-14T15:25:08.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:08 smithi090 ceph-mon[41670]: osdmap e764: 8 total, 8 up, 7 in 2024-04-14T15:25:08.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:08 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-134"}]: dispatch 2024-04-14T15:25:08.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:08 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-134"}]: dispatch 2024-04-14T15:25:08.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:08 smithi090 ceph-mon[30399]: 1.a scrub starts 2024-04-14T15:25:08.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:08 smithi090 ceph-mon[30399]: 1.a scrub ok 2024-04-14T15:25:08.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:08 smithi090 ceph-mon[30399]: pgmap v1129: 224 pgs: 4 creating+activating, 4 active+remapped+backfill_toofull, 216 active+clean; 489 KiB data, 715 MiB used, 625 GiB / 626 GiB avail; 2/642 objects misplaced (0.312%) 2024-04-14T15:25:08.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:08 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]': finished 2024-04-14T15:25:08.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:08 smithi090 ceph-mon[30399]: osdmap e764: 8 total, 8 up, 7 in 2024-04-14T15:25:08.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:08 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-134"}]: dispatch 2024-04-14T15:25:08.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:08 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-134"}]: dispatch 2024-04-14T15:25:08.252 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:25:08.275 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:08.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:08.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:25:08.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:08 smithi092 ceph-mon[33966]: 1.a scrub starts 2024-04-14T15:25:08.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:08 smithi092 ceph-mon[33966]: 1.a scrub ok 2024-04-14T15:25:08.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:08 smithi092 ceph-mon[33966]: pgmap v1129: 224 pgs: 4 creating+activating, 4 active+remapped+backfill_toofull, 216 active+clean; 489 KiB data, 715 MiB used, 625 GiB / 626 GiB avail; 2/642 objects misplaced (0.312%) 2024-04-14T15:25:08.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:08 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]': finished 2024-04-14T15:25:08.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:08 smithi092 ceph-mon[33966]: osdmap e764: 8 total, 8 up, 7 in 2024-04-14T15:25:08.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:08 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-134"}]: dispatch 2024-04-14T15:25:08.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:08 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-134"}]: dispatch 2024-04-14T15:25:08.376 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:08.401 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:08.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:08.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:08.503 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:08.527 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:08.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:08.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:08.629 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:25:08.652 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:08.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:08.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:25:08.754 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:25:08.777 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:08.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:08.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:25:08.879 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:25:08.904 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:08.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:08.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:25:09.005 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:25:09.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:09.031 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:09.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:25:09.076 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:25:09.076 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:25:09.132 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:25:09.156 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:09.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:09.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:25:09.258 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:25:09.281 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:09.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:09.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:25:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:09 smithi092 ceph-mon[33966]: 2.1d scrub starts 2024-04-14T15:25:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:09 smithi092 ceph-mon[33966]: 2.1d scrub ok 2024-04-14T15:25:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:09 smithi092 ceph-mon[33966]: 4.1d scrub starts 2024-04-14T15:25:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:09 smithi092 ceph-mon[33966]: 4.1d scrub ok 2024-04-14T15:25:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:09 smithi092 ceph-mon[33966]: 6.1a scrub starts 2024-04-14T15:25:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:09 smithi092 ceph-mon[33966]: 6.1a scrub ok 2024-04-14T15:25:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:09 smithi092 ceph-mon[33966]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:25:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:09 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-134"}]': finished 2024-04-14T15:25:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:09 smithi092 ceph-mon[33966]: osdmap e765: 8 total, 8 up, 7 in 2024-04-14T15:25:09.383 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:25:09.385 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:09 smithi090 ceph-mon[41670]: 2.1d scrub starts 2024-04-14T15:25:09.385 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:09 smithi090 ceph-mon[41670]: 2.1d scrub ok 2024-04-14T15:25:09.385 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:09 smithi090 ceph-mon[41670]: 4.1d scrub starts 2024-04-14T15:25:09.386 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:09 smithi090 ceph-mon[41670]: 4.1d scrub ok 2024-04-14T15:25:09.386 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:09 smithi090 ceph-mon[41670]: 6.1a scrub starts 2024-04-14T15:25:09.386 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:09 smithi090 ceph-mon[41670]: 6.1a scrub ok 2024-04-14T15:25:09.386 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:09 smithi090 ceph-mon[41670]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:25:09.386 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:09 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-134"}]': finished 2024-04-14T15:25:09.386 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:09 smithi090 ceph-mon[41670]: osdmap e765: 8 total, 8 up, 7 in 2024-04-14T15:25:09.386 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:09 smithi090 ceph-mon[30399]: 2.1d scrub starts 2024-04-14T15:25:09.386 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:09 smithi090 ceph-mon[30399]: 2.1d scrub ok 2024-04-14T15:25:09.386 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:09 smithi090 ceph-mon[30399]: 4.1d scrub starts 2024-04-14T15:25:09.386 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:09 smithi090 ceph-mon[30399]: 4.1d scrub ok 2024-04-14T15:25:09.387 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:09 smithi090 ceph-mon[30399]: 6.1a scrub starts 2024-04-14T15:25:09.387 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:09 smithi090 ceph-mon[30399]: 6.1a scrub ok 2024-04-14T15:25:09.387 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:09 smithi090 ceph-mon[30399]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:25:09.387 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:09 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-134"}]': finished 2024-04-14T15:25:09.387 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:09 smithi090 ceph-mon[30399]: osdmap e765: 8 total, 8 up, 7 in 2024-04-14T15:25:09.406 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:09.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:09.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:25:09.508 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:25:09.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:09.532 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:09.532 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:25:09.633 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:25:09.657 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:09.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:09.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:25:09.759 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:25:09.783 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:09.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:09.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:25:09.885 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:09.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:09.910 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:09.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:10.011 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:25:10.036 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:10.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:10.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:25:10.138 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:25:10.162 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:10.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:10.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:25:10.247 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:10 smithi090 ceph-mon[41670]: 6.14 scrub starts 2024-04-14T15:25:10.247 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:10 smithi090 ceph-mon[41670]: 6.14 scrub ok 2024-04-14T15:25:10.248 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:10 smithi090 ceph-mon[41670]: 11.1d scrub starts 2024-04-14T15:25:10.248 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:10 smithi090 ceph-mon[41670]: 11.1d scrub ok 2024-04-14T15:25:10.248 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:10 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:25:10.248 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:10 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:25:10.248 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:10 smithi090 ceph-mon[41670]: pgmap v1132: 224 pgs: 4 active+remapped+backfill_toofull, 220 active+clean; 382 KiB data, 733 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 3 op/s; 2/645 objects misplaced (0.310%) 2024-04-14T15:25:10.248 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:10 smithi090 ceph-mon[41670]: 2.4 scrub starts 2024-04-14T15:25:10.248 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:10 smithi090 ceph-mon[41670]: 2.4 scrub ok 2024-04-14T15:25:10.248 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:10 smithi090 ceph-mon[41670]: osdmap e766: 8 total, 8 up, 7 in 2024-04-14T15:25:10.248 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:10 smithi090 ceph-mon[30399]: 6.14 scrub starts 2024-04-14T15:25:10.248 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:10 smithi090 ceph-mon[30399]: 6.14 scrub ok 2024-04-14T15:25:10.248 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:10 smithi090 ceph-mon[30399]: 11.1d scrub starts 2024-04-14T15:25:10.248 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:10 smithi090 ceph-mon[30399]: 11.1d scrub ok 2024-04-14T15:25:10.248 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:10 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:25:10.249 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:10 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:25:10.249 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:10 smithi090 ceph-mon[30399]: pgmap v1132: 224 pgs: 4 active+remapped+backfill_toofull, 220 active+clean; 382 KiB data, 733 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 3 op/s; 2/645 objects misplaced (0.310%) 2024-04-14T15:25:10.249 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:10 smithi090 ceph-mon[30399]: 2.4 scrub starts 2024-04-14T15:25:10.249 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:10 smithi090 ceph-mon[30399]: 2.4 scrub ok 2024-04-14T15:25:10.249 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:10 smithi090 ceph-mon[30399]: osdmap e766: 8 total, 8 up, 7 in 2024-04-14T15:25:10.264 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:25:10.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:10.291 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:10.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:25:10.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:10 smithi092 ceph-mon[33966]: 6.14 scrub starts 2024-04-14T15:25:10.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:10 smithi092 ceph-mon[33966]: 6.14 scrub ok 2024-04-14T15:25:10.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:10 smithi092 ceph-mon[33966]: 11.1d scrub starts 2024-04-14T15:25:10.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:10 smithi092 ceph-mon[33966]: 11.1d scrub ok 2024-04-14T15:25:10.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:10 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:25:10.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:10 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:25:10.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:10 smithi092 ceph-mon[33966]: pgmap v1132: 224 pgs: 4 active+remapped+backfill_toofull, 220 active+clean; 382 KiB data, 733 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 3 op/s; 2/645 objects misplaced (0.310%) 2024-04-14T15:25:10.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:10 smithi092 ceph-mon[33966]: 2.4 scrub starts 2024-04-14T15:25:10.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:10 smithi092 ceph-mon[33966]: 2.4 scrub ok 2024-04-14T15:25:10.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:10 smithi092 ceph-mon[33966]: osdmap e766: 8 total, 8 up, 7 in 2024-04-14T15:25:10.392 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:25:10.416 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:10.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:10.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:25:10.423 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-14T15:25:10.518 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:10.541 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:10.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:10.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:10.643 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:25:10.667 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:10.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:10.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:25:10.769 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:25:10.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:10.793 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:10.794 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:25:10.895 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:25:10.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:10.919 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:10.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:25:11.019 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:25:11.043 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:11.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:11.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:25:11.145 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:25:11.220 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:11.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:11.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:25:11.322 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:25:11.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:11.347 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:11.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:25:11.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:11 smithi092 ceph-mon[33966]: 6.1f scrub starts 2024-04-14T15:25:11.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:11 smithi092 ceph-mon[33966]: 6.1f scrub ok 2024-04-14T15:25:11.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:11 smithi092 ceph-mon[33966]: 11.17 scrub starts 2024-04-14T15:25:11.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:11 smithi092 ceph-mon[33966]: 11.17 scrub ok 2024-04-14T15:25:11.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:11 smithi092 ceph-mon[33966]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:25:11.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:11 smithi092 ceph-mon[33966]: osdmap e767: 8 total, 8 up, 7 in 2024-04-14T15:25:11.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:11 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:25:11.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:11 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:25:11.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:11 smithi090 ceph-mon[41670]: 6.1f scrub starts 2024-04-14T15:25:11.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:11 smithi090 ceph-mon[41670]: 6.1f scrub ok 2024-04-14T15:25:11.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:11 smithi090 ceph-mon[41670]: 11.17 scrub starts 2024-04-14T15:25:11.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:11 smithi090 ceph-mon[41670]: 11.17 scrub ok 2024-04-14T15:25:11.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:11 smithi090 ceph-mon[41670]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:25:11.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:11 smithi090 ceph-mon[41670]: osdmap e767: 8 total, 8 up, 7 in 2024-04-14T15:25:11.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:11 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:25:11.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:11 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:25:11.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:11 smithi090 ceph-mon[30399]: 6.1f scrub starts 2024-04-14T15:25:11.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:11 smithi090 ceph-mon[30399]: 6.1f scrub ok 2024-04-14T15:25:11.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:11 smithi090 ceph-mon[30399]: 11.17 scrub starts 2024-04-14T15:25:11.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:11 smithi090 ceph-mon[30399]: 11.17 scrub ok 2024-04-14T15:25:11.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:11 smithi090 ceph-mon[30399]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:25:11.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:11 smithi090 ceph-mon[30399]: osdmap e767: 8 total, 8 up, 7 in 2024-04-14T15:25:11.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:11 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:25:11.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:11 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:25:11.448 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:25:11.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:11.472 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:11.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:25:11.573 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:25:11.598 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:11.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:11.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:25:11.611 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-14T15:25:11.611 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-14T15:25:11.611 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-14T15:25:11.611 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-14T15:25:11.700 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:11.724 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:11.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:11.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:11.826 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:25:11.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:11.851 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:11.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:25:11.952 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:25:11.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:11.976 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:11.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:25:12.077 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:25:12.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:12.102 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:12.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:25:12.202 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:25:12.226 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:12.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:12.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:25:12.330 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:25:12.332 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:12 smithi090 ceph-mon[41670]: 4.19 scrub starts 2024-04-14T15:25:12.333 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:12 smithi090 ceph-mon[41670]: 4.19 scrub ok 2024-04-14T15:25:12.333 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:12 smithi090 ceph-mon[41670]: pgmap v1135: 224 pgs: 32 unknown, 4 active+remapped+backfill_toofull, 188 active+clean; 382 KiB data, 733 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s; 2/645 objects misplaced (0.310%) 2024-04-14T15:25:12.333 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:12 smithi090 ceph-mon[41670]: 2.a scrub starts 2024-04-14T15:25:12.333 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:12 smithi090 ceph-mon[41670]: 2.a scrub ok 2024-04-14T15:25:12.333 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:12 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:25:12.333 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:12 smithi090 ceph-mon[41670]: osdmap e768: 8 total, 8 up, 7 in 2024-04-14T15:25:12.333 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:12 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:25:12.333 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:12 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:25:12.333 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:12 smithi090 ceph-mon[30399]: 4.19 scrub starts 2024-04-14T15:25:12.333 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:12 smithi090 ceph-mon[30399]: 4.19 scrub ok 2024-04-14T15:25:12.334 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:12 smithi090 ceph-mon[30399]: pgmap v1135: 224 pgs: 32 unknown, 4 active+remapped+backfill_toofull, 188 active+clean; 382 KiB data, 733 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s; 2/645 objects misplaced (0.310%) 2024-04-14T15:25:12.334 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:12 smithi090 ceph-mon[30399]: 2.a scrub starts 2024-04-14T15:25:12.334 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:12 smithi090 ceph-mon[30399]: 2.a scrub ok 2024-04-14T15:25:12.334 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:12 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:25:12.334 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:12 smithi090 ceph-mon[30399]: osdmap e768: 8 total, 8 up, 7 in 2024-04-14T15:25:12.334 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:12 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:25:12.334 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:12 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:25:12.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:12.356 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:12.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:25:12.457 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:12.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:12.481 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:12.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:12.581 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:25:12.607 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:12.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:12.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:25:12.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:12 smithi092 ceph-mon[33966]: 4.19 scrub starts 2024-04-14T15:25:12.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:12 smithi092 ceph-mon[33966]: 4.19 scrub ok 2024-04-14T15:25:12.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:12 smithi092 ceph-mon[33966]: pgmap v1135: 224 pgs: 32 unknown, 4 active+remapped+backfill_toofull, 188 active+clean; 382 KiB data, 733 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s; 2/645 objects misplaced (0.310%) 2024-04-14T15:25:12.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:12 smithi092 ceph-mon[33966]: 2.a scrub starts 2024-04-14T15:25:12.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:12 smithi092 ceph-mon[33966]: 2.a scrub ok 2024-04-14T15:25:12.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:12 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:25:12.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:12 smithi092 ceph-mon[33966]: osdmap e768: 8 total, 8 up, 7 in 2024-04-14T15:25:12.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:12 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:25:12.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:12 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:25:12.709 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:25:12.738 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:12.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:12.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:25:12.840 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:12.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:12.864 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:12.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:12.964 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:25:12.988 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:12.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:12.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:25:13.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.006+0000 7f62e924c700 1 -- 172.21.15.90:0/2759108518 >> v1:172.21.15.90:6789/0 conn(0x7f62e410e260 legacy=0x7f62e4110700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:13.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.006+0000 7f62e924c700 1 -- 172.21.15.90:0/2759108518 shutdown_connections 2024-04-14T15:25:13.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.006+0000 7f62e924c700 1 -- 172.21.15.90:0/2759108518 >> 172.21.15.90:0/2759108518 conn(0x7f62e4076a70 msgr2=0x7f62e4076e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:25:13.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.006+0000 7f62e924c700 1 -- 172.21.15.90:0/2759108518 shutdown_connections 2024-04-14T15:25:13.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.006+0000 7f62e924c700 1 -- 172.21.15.90:0/2759108518 wait complete. 2024-04-14T15:25:13.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.006+0000 7f62e924c700 1 librados: starting msgr at 2024-04-14T15:25:13.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.006+0000 7f62e924c700 1 librados: starting objecter 2024-04-14T15:25:13.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.007+0000 7f62e924c700 1 Processor -- start 2024-04-14T15:25:13.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.007+0000 7f62e924c700 1 -- start start 2024-04-14T15:25:13.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.007+0000 7f62e924c700 1 librados: setting wanted keys 2024-04-14T15:25:13.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.007+0000 7f62e924c700 1 librados: calling monclient init 2024-04-14T15:25:13.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.007+0000 7f62e924c700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f62e4110450 con 0x7f62e41023d0 2024-04-14T15:25:13.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.007+0000 7f62e924c700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f62e4110660 con 0x7f62e410a6c0 2024-04-14T15:25:13.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.007+0000 7f62e924c700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f62e41ac260 con 0x7f62e410e260 2024-04-14T15:25:13.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.007+0000 7f62e2d9d700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f62e41023d0 0x7f62e410e780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:57696/0 (socket says 172.21.15.90:57696) 2024-04-14T15:25:13.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.007+0000 7f62e2d9d700 1 -- 172.21.15.90:0/3585456003 learned_addr learned my addr 172.21.15.90:0/3585456003 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:25:13.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.007+0000 7f62e0d99700 10 client.?.objecter ms_handle_connect 0x7f62e41023d0 2024-04-14T15:25:13.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.007+0000 7f62e0d99700 10 client.?.objecter resend_mon_ops 2024-04-14T15:25:13.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.007+0000 7f62e0d99700 10 client.?.objecter ms_handle_connect 0x7f62e410a6c0 2024-04-14T15:25:13.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.007+0000 7f62e0d99700 10 client.?.objecter resend_mon_ops 2024-04-14T15:25:13.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.007+0000 7f62e0d99700 10 client.?.objecter ms_handle_connect 0x7f62e410e260 2024-04-14T15:25:13.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.007+0000 7f62e0d99700 10 client.?.objecter resend_mon_ops 2024-04-14T15:25:13.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.007+0000 7f62e0d99700 1 -- 172.21.15.90:0/3585456003 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2862314127 0 0) 0x7f62e4110660 con 0x7f62e410a6c0 2024-04-14T15:25:13.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.007+0000 7f62e0d99700 1 -- 172.21.15.90:0/3585456003 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f62d00035b0 con 0x7f62e410a6c0 2024-04-14T15:25:13.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.007+0000 7f62e0d99700 1 -- 172.21.15.90:0/3585456003 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 141005252 0 0) 0x7f62e41ac260 con 0x7f62e410e260 2024-04-14T15:25:13.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.007+0000 7f62e0d99700 1 -- 172.21.15.90:0/3585456003 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f62e4110660 con 0x7f62e410e260 2024-04-14T15:25:13.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.007+0000 7f62e0d99700 1 -- 172.21.15.90:0/3585456003 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1542963341 0 0) 0x7f62e4110450 con 0x7f62e41023d0 2024-04-14T15:25:13.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.007+0000 7f62e0d99700 1 -- 172.21.15.90:0/3585456003 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f62e41ac260 con 0x7f62e41023d0 2024-04-14T15:25:13.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.007+0000 7f62e0d99700 1 -- 172.21.15.90:0/3585456003 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1110792813 0 0) 0x7f62d00035b0 con 0x7f62e410a6c0 2024-04-14T15:25:13.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.008+0000 7f62e0d99700 1 -- 172.21.15.90:0/3585456003 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f62e4110450 con 0x7f62e410a6c0 2024-04-14T15:25:13.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.008+0000 7f62e0d99700 1 -- 172.21.15.90:0/3585456003 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f62cc0018b0 con 0x7f62e410a6c0 2024-04-14T15:25:13.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.008+0000 7f62e0d99700 1 -- 172.21.15.90:0/3585456003 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1419409344 0 0) 0x7f62e4110660 con 0x7f62e410e260 2024-04-14T15:25:13.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.008+0000 7f62e0d99700 1 -- 172.21.15.90:0/3585456003 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f62d00035b0 con 0x7f62e410e260 2024-04-14T15:25:13.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.008+0000 7f62e0d99700 1 -- 172.21.15.90:0/3585456003 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f62dc0017b0 con 0x7f62e410e260 2024-04-14T15:25:13.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.008+0000 7f62e0d99700 1 -- 172.21.15.90:0/3585456003 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2713871016 0 0) 0x7f62e4110450 con 0x7f62e410a6c0 2024-04-14T15:25:13.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.008+0000 7f62e0d99700 1 -- 172.21.15.90:0/3585456003 >> v1:172.21.15.90:6790/0 conn(0x7f62e410e260 legacy=0x7f62e41a8b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:13.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.008+0000 7f62e0d99700 1 -- 172.21.15.90:0/3585456003 >> v1:172.21.15.90:6789/0 conn(0x7f62e41023d0 legacy=0x7f62e410e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:13.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.008+0000 7f62e0d99700 1 -- 172.21.15.90:0/3585456003 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62e41ad480 con 0x7f62e410a6c0 2024-04-14T15:25:13.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.008+0000 7f62e924c700 1 -- 172.21.15.90:0/3585456003 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f62e41ac490 con 0x7f62e410a6c0 2024-04-14T15:25:13.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.008+0000 7f62e924c700 10 client.49807.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:25:13.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.008+0000 7f62e924c700 1 -- 172.21.15.90:0/3585456003 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f62e41ac950 con 0x7f62e410a6c0 2024-04-14T15:25:13.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.008+0000 7f62e924c700 1 librados: init done 2024-04-14T15:25:13.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.008+0000 7f62e0d99700 1 -- 172.21.15.90:0/3585456003 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f62cc003b40 con 0x7f62e410a6c0 2024-04-14T15:25:13.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.008+0000 7f62e0d99700 1 -- 172.21.15.90:0/3585456003 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f62cc0049a0 con 0x7f62e410a6c0 2024-04-14T15:25:13.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.009+0000 7f62e924c700 1 -- 172.21.15.90:0/3585456003 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62c4005120 con 0x7f62e410a6c0 2024-04-14T15:25:13.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.010+0000 7f62e0d99700 1 -- 172.21.15.90:0/3585456003 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f62cc005c70 con 0x7f62e410a6c0 2024-04-14T15:25:13.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.010+0000 7f62e0d99700 1 -- 172.21.15.90:0/3585456003 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(768..768 src has 467..768) ==== 8894+0+0 (unknown 980044391 0 0) 0x7f62cc089af0 con 0x7f62e410a6c0 2024-04-14T15:25:13.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.010+0000 7f62e0d99700 10 client.49807.objecter ms_dispatch 0x7f62e4001550 osd_map(768..768 src has 467..768) 2024-04-14T15:25:13.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.010+0000 7f62e0d99700 3 client.49807.objecter handle_osd_map got epochs [768,768] > 0 2024-04-14T15:25:13.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.010+0000 7f62e0d99700 3 client.49807.objecter handle_osd_map decoding full epoch 768 2024-04-14T15:25:13.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.011+0000 7f62e0d99700 20 client.49807.objecter dump_active .. 0 homeless 2024-04-14T15:25:13.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.011+0000 7f62e0d99700 10 client.49807.objecter ms_handle_connect 0x7f62d0071290 2024-04-14T15:25:13.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.022+0000 7f62e0d99700 1 -- 172.21.15.90:0/3585456003 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f62cc056220 con 0x7f62e410a6c0 2024-04-14T15:25:13.089 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:25:13.114 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:13.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:13.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:25:13.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:13 smithi090 ceph-mon[30399]: 11.2 scrub starts 2024-04-14T15:25:13.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:13 smithi090 ceph-mon[30399]: 11.2 scrub ok 2024-04-14T15:25:13.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:13 smithi090 ceph-mon[30399]: 323.7s1 scrub starts 2024-04-14T15:25:13.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:13 smithi090 ceph-mon[30399]: 323.7 scrub ok 2024-04-14T15:25:13.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:13 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-136", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:25:13.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:13 smithi090 ceph-mon[30399]: osdmap e769: 8 total, 8 up, 7 in 2024-04-14T15:25:13.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:13 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-136"}]: dispatch 2024-04-14T15:25:13.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:13 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-136"}]: dispatch 2024-04-14T15:25:13.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:13 smithi090 ceph-mon[41670]: 11.2 scrub starts 2024-04-14T15:25:13.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:13 smithi090 ceph-mon[41670]: 11.2 scrub ok 2024-04-14T15:25:13.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:13 smithi090 ceph-mon[41670]: 323.7s1 scrub starts 2024-04-14T15:25:13.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:13 smithi090 ceph-mon[41670]: 323.7 scrub ok 2024-04-14T15:25:13.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:13 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-136", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:25:13.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:13 smithi090 ceph-mon[41670]: osdmap e769: 8 total, 8 up, 7 in 2024-04-14T15:25:13.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:13 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-136"}]: dispatch 2024-04-14T15:25:13.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:13 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-136"}]: dispatch 2024-04-14T15:25:13.216 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:25:13.241 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:13.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:13.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:25:13.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.247+0000 7f62e924c700 1 -- 172.21.15.90:0/3585456003 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd reweight", "id": 1, "weight": 1.0} v 0) -- 0x7f62c4005410 con 0x7f62e410a6c0 2024-04-14T15:25:13.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.221+0000 7f422d989700 1 -- 172.21.15.90:0/77123731 >> v1:172.21.15.90:6789/0 conn(0x7f422810b7c0 legacy=0x7f422810dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:13.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.221+0000 7f422d989700 1 -- 172.21.15.90:0/77123731 shutdown_connections 2024-04-14T15:25:13.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.221+0000 7f422d989700 1 -- 172.21.15.90:0/77123731 >> 172.21.15.90:0/77123731 conn(0x7f4228076a70 msgr2=0x7f4228076e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:25:13.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.221+0000 7f422d989700 1 -- 172.21.15.90:0/77123731 shutdown_connections 2024-04-14T15:25:13.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.221+0000 7f422d989700 1 -- 172.21.15.90:0/77123731 wait complete. 2024-04-14T15:25:13.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.222+0000 7f422d989700 1 librados: starting msgr at 2024-04-14T15:25:13.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.222+0000 7f422d989700 1 librados: starting objecter 2024-04-14T15:25:13.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.222+0000 7f422d989700 1 Processor -- start 2024-04-14T15:25:13.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.222+0000 7f422d989700 1 -- start start 2024-04-14T15:25:13.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.222+0000 7f422d989700 1 librados: setting wanted keys 2024-04-14T15:25:13.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.222+0000 7f422d989700 1 librados: calling monclient init 2024-04-14T15:25:13.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.222+0000 7f422d989700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f42281a78e0 con 0x7f422810f360 2024-04-14T15:25:13.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.222+0000 7f422d989700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f42281a8b00 con 0x7f42281012c0 2024-04-14T15:25:13.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.222+0000 7f422d989700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f42281a9d40 con 0x7f422810b7c0 2024-04-14T15:25:13.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.222+0000 7f42267fc700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f422810b7c0 0x7f42281a27f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:56932/0 (socket says 172.21.15.90:56932) 2024-04-14T15:25:13.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.222+0000 7f42267fc700 1 -- 172.21.15.90:0/3840278200 learned_addr learned my addr 172.21.15.90:0/3840278200 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:25:13.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.223+0000 7f422c987700 10 client.?.objecter ms_handle_connect 0x7f422810f360 2024-04-14T15:25:13.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.223+0000 7f422c987700 10 client.?.objecter resend_mon_ops 2024-04-14T15:25:13.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.223+0000 7f422c987700 10 client.?.objecter ms_handle_connect 0x7f422810b7c0 2024-04-14T15:25:13.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.223+0000 7f422c987700 10 client.?.objecter resend_mon_ops 2024-04-14T15:25:13.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.223+0000 7f422c987700 10 client.?.objecter ms_handle_connect 0x7f42281012c0 2024-04-14T15:25:13.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.223+0000 7f422c987700 10 client.?.objecter resend_mon_ops 2024-04-14T15:25:13.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.223+0000 7f422c987700 1 -- 172.21.15.90:0/3840278200 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1234294496 0 0) 0x7f42281a78e0 con 0x7f422810f360 2024-04-14T15:25:13.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.223+0000 7f422c987700 1 -- 172.21.15.90:0/3840278200 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42100035b0 con 0x7f422810f360 2024-04-14T15:25:13.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.223+0000 7f422c987700 1 -- 172.21.15.90:0/3840278200 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3531901924 0 0) 0x7f42281a9d40 con 0x7f422810b7c0 2024-04-14T15:25:13.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.223+0000 7f422c987700 1 -- 172.21.15.90:0/3840278200 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42281a78e0 con 0x7f422810b7c0 2024-04-14T15:25:13.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.223+0000 7f422c987700 1 -- 172.21.15.90:0/3840278200 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 539633225 0 0) 0x7f42281a8b00 con 0x7f42281012c0 2024-04-14T15:25:13.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.223+0000 7f422c987700 1 -- 172.21.15.90:0/3840278200 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42281a9d40 con 0x7f42281012c0 2024-04-14T15:25:13.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.223+0000 7f422c987700 1 -- 172.21.15.90:0/3840278200 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3920475400 0 0) 0x7f42281a9d40 con 0x7f42281012c0 2024-04-14T15:25:13.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.223+0000 7f422c987700 1 -- 172.21.15.90:0/3840278200 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f42281a8b00 con 0x7f42281012c0 2024-04-14T15:25:13.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.223+0000 7f422c987700 1 -- 172.21.15.90:0/3840278200 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3810305672 0 0) 0x7f42281a78e0 con 0x7f422810b7c0 2024-04-14T15:25:13.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.223+0000 7f422c987700 1 -- 172.21.15.90:0/3840278200 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f42281a9d40 con 0x7f422810b7c0 2024-04-14T15:25:13.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.223+0000 7f422c987700 1 -- 172.21.15.90:0/3840278200 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3610214225 0 0) 0x7f42100035b0 con 0x7f422810f360 2024-04-14T15:25:13.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.223+0000 7f422c987700 1 -- 172.21.15.90:0/3840278200 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f42281a78e0 con 0x7f422810f360 2024-04-14T15:25:13.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.223+0000 7f422c987700 1 -- 172.21.15.90:0/3840278200 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f4218001130 con 0x7f42281012c0 2024-04-14T15:25:13.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.223+0000 7f422c987700 1 -- 172.21.15.90:0/3840278200 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f421c0043f0 con 0x7f422810b7c0 2024-04-14T15:25:13.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.223+0000 7f422c987700 1 -- 172.21.15.90:0/3840278200 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f4220001bb0 con 0x7f422810f360 2024-04-14T15:25:13.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.223+0000 7f422c987700 1 -- 172.21.15.90:0/3840278200 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 463686255 0 0) 0x7f42281a8b00 con 0x7f42281012c0 2024-04-14T15:25:13.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.223+0000 7f422c987700 1 -- 172.21.15.90:0/3840278200 >> v1:172.21.15.90:6790/0 conn(0x7f422810b7c0 legacy=0x7f42281a27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:13.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.223+0000 7f422c987700 1 -- 172.21.15.90:0/3840278200 >> v1:172.21.15.90:6789/0 conn(0x7f422810f360 legacy=0x7f42281a5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:13.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.223+0000 7f422c987700 1 -- 172.21.15.90:0/3840278200 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42281aaf80 con 0x7f42281012c0 2024-04-14T15:25:13.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.223+0000 7f422d989700 1 -- 172.21.15.90:0/3840278200 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f42281a9f70 con 0x7f42281012c0 2024-04-14T15:25:13.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.223+0000 7f422d989700 10 client.49813.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:25:13.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.223+0000 7f422d989700 1 -- 172.21.15.90:0/3840278200 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f42281aa430 con 0x7f42281012c0 2024-04-14T15:25:13.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.223+0000 7f422d989700 1 librados: init done 2024-04-14T15:25:13.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.223+0000 7f422c987700 1 -- 172.21.15.90:0/3840278200 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f4218003b20 con 0x7f42281012c0 2024-04-14T15:25:13.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.223+0000 7f422c987700 1 -- 172.21.15.90:0/3840278200 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f42180049d0 con 0x7f42281012c0 2024-04-14T15:25:13.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.224+0000 7f422d989700 1 -- 172.21.15.90:0/3840278200 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4214005120 con 0x7f42281012c0 2024-04-14T15:25:13.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.225+0000 7f422c987700 1 -- 172.21.15.90:0/3840278200 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f421801b3e0 con 0x7f42281012c0 2024-04-14T15:25:13.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.225+0000 7f422c987700 1 -- 172.21.15.90:0/3840278200 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(769..769 src has 467..769) ==== 8889+0+0 (unknown 2857716337 0 0) 0x7f4218089930 con 0x7f42281012c0 2024-04-14T15:25:13.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.225+0000 7f422c987700 10 client.49813.objecter ms_dispatch 0x7f4228001550 osd_map(769..769 src has 467..769) 2024-04-14T15:25:13.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.225+0000 7f422c987700 3 client.49813.objecter handle_osd_map got epochs [769,769] > 0 2024-04-14T15:25:13.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.225+0000 7f422c987700 3 client.49813.objecter handle_osd_map decoding full epoch 769 2024-04-14T15:25:13.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.226+0000 7f422c987700 20 client.49813.objecter dump_active .. 0 homeless 2024-04-14T15:25:13.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.226+0000 7f422c987700 10 client.49813.objecter ms_handle_connect 0x7f4210071420 2024-04-14T15:25:13.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.231+0000 7f422c987700 1 -- 172.21.15.90:0/3840278200 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f4218056060 con 0x7f42281012c0 2024-04-14T15:25:13.343 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:13.366 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:13.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:13.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:13.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:13.445+0000 7f422d989700 1 -- 172.21.15.90:0/3840278200 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7f4214005410 con 0x7f42281012c0 2024-04-14T15:25:13.468 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:25:13.491 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:13.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:13.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:25:13.593 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:25:13.616 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:13.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:13.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:25:13.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:13 smithi092 ceph-mon[33966]: 11.2 scrub starts 2024-04-14T15:25:13.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:13 smithi092 ceph-mon[33966]: 11.2 scrub ok 2024-04-14T15:25:13.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:13 smithi092 ceph-mon[33966]: 323.7s1 scrub starts 2024-04-14T15:25:13.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:13 smithi092 ceph-mon[33966]: 323.7 scrub ok 2024-04-14T15:25:13.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:13 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-136", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:25:13.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:13 smithi092 ceph-mon[33966]: osdmap e769: 8 total, 8 up, 7 in 2024-04-14T15:25:13.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:13 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-136"}]: dispatch 2024-04-14T15:25:13.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:13 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-136"}]: dispatch 2024-04-14T15:25:13.718 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:25:13.742 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:13.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:13.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:25:13.744 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:25:13.770 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-14T15:25:13.844 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:25:13.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:13.870 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:13.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:25:13.971 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:25:13.994 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:13.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:13.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:25:14.013 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:25:13 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:25:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:25:14.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:14.018+0000 7f422c987700 1 -- 172.21.15.90:0/3840278200 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v770) ==== 101+0+0 (unknown 2471089574 0 0) 0x7f4218059d10 con 0x7f42281012c0 2024-04-14T15:25:14.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:14.022+0000 7f422d989700 10 librados: watch_flush enter 2024-04-14T15:25:14.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:14.022+0000 7f422d989700 10 librados: watch_flush exit 2024-04-14T15:25:14.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:14.022+0000 7f422d989700 20 client.49813.objecter shutdown clearing up homeless session... 2024-04-14T15:25:14.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:14.022+0000 7f422d989700 10 client.49813.objecter successfully canceled tick 2024-04-14T15:25:14.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:14.022+0000 7f422d989700 1 -- 172.21.15.90:0/3840278200 >> v1:172.21.15.90:6800/420935427 conn(0x7f4210071420 legacy=0x7f42100738e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:14.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:14.022+0000 7f422d989700 1 -- 172.21.15.90:0/3840278200 >> v1:172.21.15.92:6789/0 conn(0x7f42281012c0 legacy=0x7f4228107ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:14.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:14.019+0000 7f62e0d99700 1 -- 172.21.15.90:0/3585456003 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 1, "weight": 1.0}]=0 reweighted osd.1 to 1 (10000) v770) ==== 113+0+0 (unknown 3983388758 0 0) 0x7f62cc059ed0 con 0x7f62e410a6c0 2024-04-14T15:25:14.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:14.026+0000 7f62e924c700 10 librados: watch_flush enter 2024-04-14T15:25:14.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:14.027+0000 7f62e924c700 10 librados: watch_flush exit 2024-04-14T15:25:14.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:14.027+0000 7f62e924c700 20 client.49807.objecter shutdown clearing up homeless session... 2024-04-14T15:25:14.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:14.027+0000 7f62e924c700 10 client.49807.objecter successfully canceled tick 2024-04-14T15:25:14.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:14.027+0000 7f62e924c700 1 -- 172.21.15.90:0/3585456003 >> v1:172.21.15.90:6800/420935427 conn(0x7f62d0071290 legacy=0x7f62d0073750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:14.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:14.027+0000 7f62e924c700 1 -- 172.21.15.90:0/3585456003 >> v1:172.21.15.92:6789/0 conn(0x7f62e410a6c0 legacy=0x7f62e410ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:14.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:14.027+0000 7f62e924c700 1 -- 172.21.15.90:0/3585456003 shutdown_connections 2024-04-14T15:25:14.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:14.027+0000 7f62e924c700 1 -- 172.21.15.90:0/3585456003 >> 172.21.15.90:0/3585456003 conn(0x7f62e4076a70 msgr2=0x7f62e410cab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:25:14.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:14.027+0000 7f62e924c700 1 -- 172.21.15.90:0/3585456003 shutdown_connections 2024-04-14T15:25:14.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:14.027+0000 7f62e924c700 1 -- 172.21.15.90:0/3585456003 wait complete. 2024-04-14T15:25:14.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:14.026+0000 7f422d989700 1 -- 172.21.15.90:0/3840278200 shutdown_connections 2024-04-14T15:25:14.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:14.026+0000 7f422d989700 1 -- 172.21.15.90:0/3840278200 >> 172.21.15.90:0/3840278200 conn(0x7f4228076a70 msgr2=0x7f42281129b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:25:14.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:14.027+0000 7f62e924c700 1 librados: shutdown 2024-04-14T15:25:14.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:14.027+0000 7f422d989700 1 -- 172.21.15.90:0/3840278200 shutdown_connections 2024-04-14T15:25:14.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:14.028+0000 7f422d989700 1 -- 172.21.15.90:0/3840278200 wait complete. 2024-04-14T15:25:14.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:14.028+0000 7f422d989700 1 librados: shutdown 2024-04-14T15:25:14.030 INFO:teuthology.orchestra.run.smithi090.stderr:reweighted osd.1 to 1 (10000) 2024-04-14T15:25:14.030 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is unset 2024-04-14T15:25:14.095 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:25:14.120 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:14.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:14.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:25:14.221 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:14.245 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:14.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:14.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:14.347 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:14.360 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:14 smithi090 ceph-mon[30399]: pgmap v1138: 224 pgs: 32 unknown, 4 active+remapped+backfill_toofull, 188 active+clean; 382 KiB data, 733 MiB used, 625 GiB / 626 GiB avail; 2/645 objects misplaced (0.310%) 2024-04-14T15:25:14.360 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:14 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3585456003' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-14T15:25:14.360 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:14 smithi090 ceph-mon[30399]: from='client.49807 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-14T15:25:14.360 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:14 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3840278200' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:25:14.360 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:14 smithi090 ceph-mon[30399]: from='client.49813 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:25:14.360 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:14 smithi090 ceph-mon[30399]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-14T15:25:14.360 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:14 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-136"}]': finished 2024-04-14T15:25:14.360 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:14 smithi090 ceph-mon[30399]: from='client.49807 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-14T15:25:14.360 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:14 smithi090 ceph-mon[30399]: from='client.49813 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-14T15:25:14.360 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:14 smithi090 ceph-mon[30399]: osdmap e770: 8 total, 8 up, 7 in 2024-04-14T15:25:14.360 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:14 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-136", "mode": "writeback"}]: dispatch 2024-04-14T15:25:14.361 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:14 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-136", "mode": "writeback"}]: dispatch 2024-04-14T15:25:14.361 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:14 smithi090 ceph-mon[41670]: pgmap v1138: 224 pgs: 32 unknown, 4 active+remapped+backfill_toofull, 188 active+clean; 382 KiB data, 733 MiB used, 625 GiB / 626 GiB avail; 2/645 objects misplaced (0.310%) 2024-04-14T15:25:14.361 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:14 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3585456003' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-14T15:25:14.361 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:14 smithi090 ceph-mon[41670]: from='client.49807 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-14T15:25:14.361 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:14 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3840278200' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:25:14.361 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:14 smithi090 ceph-mon[41670]: from='client.49813 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:25:14.361 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:14 smithi090 ceph-mon[41670]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-14T15:25:14.361 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:14 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-136"}]': finished 2024-04-14T15:25:14.361 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:14 smithi090 ceph-mon[41670]: from='client.49807 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-14T15:25:14.361 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:14 smithi090 ceph-mon[41670]: from='client.49813 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-14T15:25:14.361 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:14 smithi090 ceph-mon[41670]: osdmap e770: 8 total, 8 up, 7 in 2024-04-14T15:25:14.362 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:14 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-136", "mode": "writeback"}]: dispatch 2024-04-14T15:25:14.362 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:14 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-136", "mode": "writeback"}]: dispatch 2024-04-14T15:25:14.372 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:14.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:14.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:14.473 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:14.496 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:14.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:14.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:14.598 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:25:14.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:14 smithi092 ceph-mon[33966]: pgmap v1138: 224 pgs: 32 unknown, 4 active+remapped+backfill_toofull, 188 active+clean; 382 KiB data, 733 MiB used, 625 GiB / 626 GiB avail; 2/645 objects misplaced (0.310%) 2024-04-14T15:25:14.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:14 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3585456003' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-14T15:25:14.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:14 smithi092 ceph-mon[33966]: from='client.49807 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-14T15:25:14.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:14 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3840278200' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:25:14.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:14 smithi092 ceph-mon[33966]: from='client.49813 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:25:14.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:14 smithi092 ceph-mon[33966]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-14T15:25:14.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:14 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-136"}]': finished 2024-04-14T15:25:14.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:14 smithi092 ceph-mon[33966]: from='client.49807 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-14T15:25:14.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:14 smithi092 ceph-mon[33966]: from='client.49813 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-14T15:25:14.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:14 smithi092 ceph-mon[33966]: osdmap e770: 8 total, 8 up, 7 in 2024-04-14T15:25:14.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:14 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-136", "mode": "writeback"}]: dispatch 2024-04-14T15:25:14.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:14 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-136", "mode": "writeback"}]: dispatch 2024-04-14T15:25:14.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:14.624 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:14.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:25:14.725 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:14.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:14.749 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:14.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:14.849 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:25:14.877 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:14.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:14.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:25:14.978 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:25:15.006 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:15.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:15.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:25:15.108 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:25:15.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:15.134 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:15.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:25:15.207 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd reweight 2 1 2024-04-14T15:25:15.234 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:25:15.260 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:15.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:15.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:25:15.361 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:15.385 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:15.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:15.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:15.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:15 smithi090 ceph-mon[41670]: 5.15 scrub starts 2024-04-14T15:25:15.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:15 smithi090 ceph-mon[41670]: 5.15 scrub ok 2024-04-14T15:25:15.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:15 smithi090 ceph-mon[41670]: 5.1c scrub starts 2024-04-14T15:25:15.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:15 smithi090 ceph-mon[41670]: 5.1c scrub ok 2024-04-14T15:25:15.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:15 smithi090 ceph-mon[41670]: 5.7 scrub starts 2024-04-14T15:25:15.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:15 smithi090 ceph-mon[41670]: 5.7 scrub ok 2024-04-14T15:25:15.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:15 smithi090 ceph-mon[41670]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:25:15.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:15 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-136", "mode": "writeback"}]': finished 2024-04-14T15:25:15.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:15 smithi090 ceph-mon[41670]: osdmap e771: 8 total, 8 up, 7 in 2024-04-14T15:25:15.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:15 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:25:15.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:15 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:25:15.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:15 smithi090 ceph-mon[30399]: 5.15 scrub starts 2024-04-14T15:25:15.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:15 smithi090 ceph-mon[30399]: 5.15 scrub ok 2024-04-14T15:25:15.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:15 smithi090 ceph-mon[30399]: 5.1c scrub starts 2024-04-14T15:25:15.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:15 smithi090 ceph-mon[30399]: 5.1c scrub ok 2024-04-14T15:25:15.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:15 smithi090 ceph-mon[30399]: 5.7 scrub starts 2024-04-14T15:25:15.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:15 smithi090 ceph-mon[30399]: 5.7 scrub ok 2024-04-14T15:25:15.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:15 smithi090 ceph-mon[30399]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:25:15.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:15 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-136", "mode": "writeback"}]': finished 2024-04-14T15:25:15.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:15 smithi090 ceph-mon[30399]: osdmap e771: 8 total, 8 up, 7 in 2024-04-14T15:25:15.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:15 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:25:15.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:15 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:25:15.461 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:15 smithi092 ceph-mon[33966]: 5.15 scrub starts 2024-04-14T15:25:15.462 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:15 smithi092 ceph-mon[33966]: 5.15 scrub ok 2024-04-14T15:25:15.462 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:15 smithi092 ceph-mon[33966]: 5.1c scrub starts 2024-04-14T15:25:15.462 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:15 smithi092 ceph-mon[33966]: 5.1c scrub ok 2024-04-14T15:25:15.462 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:15 smithi092 ceph-mon[33966]: 5.7 scrub starts 2024-04-14T15:25:15.462 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:15 smithi092 ceph-mon[33966]: 5.7 scrub ok 2024-04-14T15:25:15.462 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:15 smithi092 ceph-mon[33966]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:25:15.462 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:15 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-136", "mode": "writeback"}]': finished 2024-04-14T15:25:15.462 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:15 smithi092 ceph-mon[33966]: osdmap e771: 8 total, 8 up, 7 in 2024-04-14T15:25:15.462 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:15 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:25:15.462 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:15 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:25:15.488 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:25:15.513 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:15.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:15.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:25:15.615 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:25:15.641 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:15.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:15.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:25:15.743 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:25:15.768 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:15.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:15.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:25:15.870 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:15.872 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:25:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:25:15.895 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:15.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:15.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:15.997 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:16.024 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:16.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:16.025 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:16.126 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:25:16.151 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:16.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:16.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:25:16.254 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:25:16.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:16.283 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:16.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:25:16.375 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:16 smithi090 ceph-mon[41670]: pgmap v1141: 224 pgs: 4 active+remapped+backfill_toofull, 220 active+clean; 382 KiB data, 734 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/645 objects misplaced (0.310%) 2024-04-14T15:25:16.375 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:16 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:25:16.375 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:16 smithi090 ceph-mon[41670]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:25:16.375 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:16 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]': finished 2024-04-14T15:25:16.375 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:16 smithi090 ceph-mon[41670]: osdmap e772: 8 total, 8 up, 7 in 2024-04-14T15:25:16.375 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:16 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-136"}]: dispatch 2024-04-14T15:25:16.375 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:16 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-136"}]: dispatch 2024-04-14T15:25:16.375 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:16 smithi090 ceph-mon[30399]: pgmap v1141: 224 pgs: 4 active+remapped+backfill_toofull, 220 active+clean; 382 KiB data, 734 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/645 objects misplaced (0.310%) 2024-04-14T15:25:16.376 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:16 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:25:16.376 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:16 smithi090 ceph-mon[30399]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:25:16.376 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:16 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]': finished 2024-04-14T15:25:16.376 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:16 smithi090 ceph-mon[30399]: osdmap e772: 8 total, 8 up, 7 in 2024-04-14T15:25:16.376 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:16 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-136"}]: dispatch 2024-04-14T15:25:16.376 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:16 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-136"}]: dispatch 2024-04-14T15:25:16.383 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:25:16.409 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:16.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:16.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:25:16.415 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:25:16.511 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:25:16.535 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:16.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:16.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:25:16.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:16 smithi092 ceph-mon[33966]: pgmap v1141: 224 pgs: 4 active+remapped+backfill_toofull, 220 active+clean; 382 KiB data, 734 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/645 objects misplaced (0.310%) 2024-04-14T15:25:16.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:16 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:25:16.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:16 smithi092 ceph-mon[33966]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:25:16.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:16 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]': finished 2024-04-14T15:25:16.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:16 smithi092 ceph-mon[33966]: osdmap e772: 8 total, 8 up, 7 in 2024-04-14T15:25:16.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:16 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-136"}]: dispatch 2024-04-14T15:25:16.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:16 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-136"}]: dispatch 2024-04-14T15:25:16.637 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:25:16.665 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:16.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:16.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:25:16.767 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:25:16.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:16.791 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:16.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:25:16.892 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:25:16.916 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:16.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:16.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:25:17.018 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:25:17.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:17.047 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:17.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:25:17.148 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:25:17.174 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:17.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:17.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:25:17.276 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:25:17.302 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:17.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:17.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:25:17.404 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:25:17.425 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:17 smithi090 ceph-mon[41670]: 2.1a scrub starts 2024-04-14T15:25:17.425 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:17 smithi090 ceph-mon[41670]: 2.1a scrub ok 2024-04-14T15:25:17.425 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:17 smithi090 ceph-mon[41670]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:25:17.425 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:17 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-136"}]': finished 2024-04-14T15:25:17.425 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:17 smithi090 ceph-mon[41670]: osdmap e773: 8 total, 8 up, 7 in 2024-04-14T15:25:17.425 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:17 smithi090 ceph-mon[30399]: 2.1a scrub starts 2024-04-14T15:25:17.426 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:17 smithi090 ceph-mon[30399]: 2.1a scrub ok 2024-04-14T15:25:17.426 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:17 smithi090 ceph-mon[30399]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:25:17.426 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:17 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-136"}]': finished 2024-04-14T15:25:17.426 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:17 smithi090 ceph-mon[30399]: osdmap e773: 8 total, 8 up, 7 in 2024-04-14T15:25:17.427 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:17.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:17.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:25:17.529 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:25:17.557 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:17.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:17.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:25:17.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:17 smithi092 ceph-mon[33966]: 2.1a scrub starts 2024-04-14T15:25:17.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:17 smithi092 ceph-mon[33966]: 2.1a scrub ok 2024-04-14T15:25:17.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:17 smithi092 ceph-mon[33966]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:25:17.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:17 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-136"}]': finished 2024-04-14T15:25:17.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:17 smithi092 ceph-mon[33966]: osdmap e773: 8 total, 8 up, 7 in 2024-04-14T15:25:17.659 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:25:17.687 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:17.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:17.688 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:25:17.788 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:25:17.815 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:17.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:17.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:25:17.917 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:25:17.945 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:17.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:17.946 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:25:18.001 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd set noscrub 2024-04-14T15:25:18.047 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:25:18.073 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:18.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:18.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:25:18.174 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:18.201 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:18.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:18.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:18.302 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:25:18.329 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:18.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:18.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:25:18.361 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:18 smithi090 ceph-mon[41670]: 2.10 scrub starts 2024-04-14T15:25:18.361 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:18 smithi090 ceph-mon[41670]: 2.10 scrub ok 2024-04-14T15:25:18.361 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:18 smithi090 ceph-mon[41670]: 11.16 scrub starts 2024-04-14T15:25:18.361 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:18 smithi090 ceph-mon[41670]: 11.16 scrub ok 2024-04-14T15:25:18.361 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:18 smithi090 ceph-mon[41670]: 5.3 deep-scrub starts 2024-04-14T15:25:18.361 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:18 smithi090 ceph-mon[41670]: 5.3 deep-scrub ok 2024-04-14T15:25:18.362 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:18 smithi090 ceph-mon[41670]: pgmap v1144: 224 pgs: 4 active+remapped+backfill_toofull, 220 active+clean; 382 KiB data, 734 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/645 objects misplaced (0.310%) 2024-04-14T15:25:18.362 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:18 smithi090 ceph-mon[41670]: osdmap e774: 8 total, 8 up, 7 in 2024-04-14T15:25:18.362 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:18 smithi090 ceph-mon[30399]: 2.10 scrub starts 2024-04-14T15:25:18.362 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:18 smithi090 ceph-mon[30399]: 2.10 scrub ok 2024-04-14T15:25:18.362 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:18 smithi090 ceph-mon[30399]: 11.16 scrub starts 2024-04-14T15:25:18.362 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:18 smithi090 ceph-mon[30399]: 11.16 scrub ok 2024-04-14T15:25:18.362 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:18 smithi090 ceph-mon[30399]: 5.3 deep-scrub starts 2024-04-14T15:25:18.362 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:18 smithi090 ceph-mon[30399]: 5.3 deep-scrub ok 2024-04-14T15:25:18.362 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:18 smithi090 ceph-mon[30399]: pgmap v1144: 224 pgs: 4 active+remapped+backfill_toofull, 220 active+clean; 382 KiB data, 734 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/645 objects misplaced (0.310%) 2024-04-14T15:25:18.362 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:18 smithi090 ceph-mon[30399]: osdmap e774: 8 total, 8 up, 7 in 2024-04-14T15:25:18.430 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:25:18.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:18.457 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:18.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:25:18.558 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:25:18.583 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:18.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:18.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:25:18.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:18 smithi092 ceph-mon[33966]: 2.10 scrub starts 2024-04-14T15:25:18.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:18 smithi092 ceph-mon[33966]: 2.10 scrub ok 2024-04-14T15:25:18.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:18 smithi092 ceph-mon[33966]: 11.16 scrub starts 2024-04-14T15:25:18.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:18 smithi092 ceph-mon[33966]: 11.16 scrub ok 2024-04-14T15:25:18.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:18 smithi092 ceph-mon[33966]: 5.3 deep-scrub starts 2024-04-14T15:25:18.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:18 smithi092 ceph-mon[33966]: 5.3 deep-scrub ok 2024-04-14T15:25:18.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:18 smithi092 ceph-mon[33966]: pgmap v1144: 224 pgs: 4 active+remapped+backfill_toofull, 220 active+clean; 382 KiB data, 734 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/645 objects misplaced (0.310%) 2024-04-14T15:25:18.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:18 smithi092 ceph-mon[33966]: osdmap e774: 8 total, 8 up, 7 in 2024-04-14T15:25:18.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.380+0000 7f27d7b51700 1 -- 172.21.15.90:0/261056667 >> v1:172.21.15.90:6789/0 conn(0x7f27d010a740 legacy=0x7f27d010cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:18.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.380+0000 7f27d7b51700 1 -- 172.21.15.90:0/261056667 shutdown_connections 2024-04-14T15:25:18.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.380+0000 7f27d7b51700 1 -- 172.21.15.90:0/261056667 >> 172.21.15.90:0/261056667 conn(0x7f27d00fd290 msgr2=0x7f27d00ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:25:18.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.380+0000 7f27d7b51700 1 -- 172.21.15.90:0/261056667 shutdown_connections 2024-04-14T15:25:18.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.380+0000 7f27d7b51700 1 -- 172.21.15.90:0/261056667 wait complete. 2024-04-14T15:25:18.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.380+0000 7f27d7b51700 1 librados: starting msgr at 2024-04-14T15:25:18.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.380+0000 7f27d7b51700 1 librados: starting objecter 2024-04-14T15:25:18.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.380+0000 7f27d7b51700 1 Processor -- start 2024-04-14T15:25:18.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.380+0000 7f27d7b51700 1 -- start start 2024-04-14T15:25:18.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.380+0000 7f27d7b51700 1 librados: setting wanted keys 2024-04-14T15:25:18.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.380+0000 7f27d7b51700 1 librados: calling monclient init 2024-04-14T15:25:18.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.381+0000 7f27d7b51700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f27d010df50 con 0x7f27d0107890 2024-04-14T15:25:18.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.381+0000 7f27d7b51700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f27d01ab320 con 0x7f27d010e310 2024-04-14T15:25:18.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.381+0000 7f27d7b51700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f27d01ac500 con 0x7f27d010a740 2024-04-14T15:25:18.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.381+0000 7f27d58ed700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f27d0107890 0x7f27d006acf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:57738/0 (socket says 172.21.15.90:57738) 2024-04-14T15:25:18.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.381+0000 7f27d58ed700 1 -- 172.21.15.90:0/3153549364 learned_addr learned my addr 172.21.15.90:0/3153549364 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:25:18.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.381+0000 7f27d50ec700 1 --1- 172.21.15.90:0/3153549364 >> v1:172.21.15.90:6790/0 conn(0x7f27d010a740 0x7f27d006b400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:56942/0 (socket says 172.21.15.90:56942) 2024-04-14T15:25:18.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.381+0000 7f27c6ffd700 10 client.?.objecter ms_handle_connect 0x7f27d0107890 2024-04-14T15:25:18.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.381+0000 7f27c6ffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:25:18.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.381+0000 7f27c6ffd700 10 client.?.objecter ms_handle_connect 0x7f27d010a740 2024-04-14T15:25:18.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.381+0000 7f27c6ffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:25:18.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.381+0000 7f27c6ffd700 10 client.?.objecter ms_handle_connect 0x7f27d010e310 2024-04-14T15:25:18.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.381+0000 7f27c6ffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:25:18.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.381+0000 7f27c6ffd700 1 -- 172.21.15.90:0/3153549364 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 144417616 0 0) 0x7f27d010df50 con 0x7f27d0107890 2024-04-14T15:25:18.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.381+0000 7f27c6ffd700 1 -- 172.21.15.90:0/3153549364 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f27bc0035b0 con 0x7f27d0107890 2024-04-14T15:25:18.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.381+0000 7f27c6ffd700 1 -- 172.21.15.90:0/3153549364 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1213350001 0 0) 0x7f27d01ac500 con 0x7f27d010a740 2024-04-14T15:25:18.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.381+0000 7f27c6ffd700 1 -- 172.21.15.90:0/3153549364 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f27d010df50 con 0x7f27d010a740 2024-04-14T15:25:18.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.381+0000 7f27c6ffd700 1 -- 172.21.15.90:0/3153549364 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2673372733 0 0) 0x7f27d01ab320 con 0x7f27d010e310 2024-04-14T15:25:18.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.381+0000 7f27c6ffd700 1 -- 172.21.15.90:0/3153549364 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f27d01ac500 con 0x7f27d010e310 2024-04-14T15:25:18.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.381+0000 7f27c6ffd700 1 -- 172.21.15.90:0/3153549364 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1368048781 0 0) 0x7f27d010df50 con 0x7f27d010a740 2024-04-14T15:25:18.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.381+0000 7f27c6ffd700 1 -- 172.21.15.90:0/3153549364 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f27d01ab320 con 0x7f27d010a740 2024-04-14T15:25:18.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.381+0000 7f27c6ffd700 1 -- 172.21.15.90:0/3153549364 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f27c00015d0 con 0x7f27d010a740 2024-04-14T15:25:18.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.381+0000 7f27c6ffd700 1 -- 172.21.15.90:0/3153549364 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2998364328 0 0) 0x7f27d01ac500 con 0x7f27d010e310 2024-04-14T15:25:18.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.381+0000 7f27c6ffd700 1 -- 172.21.15.90:0/3153549364 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f27d010df50 con 0x7f27d010e310 2024-04-14T15:25:18.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.381+0000 7f27c6ffd700 1 -- 172.21.15.90:0/3153549364 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2261788508 0 0) 0x7f27bc0035b0 con 0x7f27d0107890 2024-04-14T15:25:18.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.381+0000 7f27c6ffd700 1 -- 172.21.15.90:0/3153549364 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f27d01ac500 con 0x7f27d0107890 2024-04-14T15:25:18.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.381+0000 7f27c6ffd700 1 -- 172.21.15.90:0/3153549364 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f27c8001a60 con 0x7f27d010e310 2024-04-14T15:25:18.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.381+0000 7f27c6ffd700 1 -- 172.21.15.90:0/3153549364 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f27cc001890 con 0x7f27d0107890 2024-04-14T15:25:18.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.382+0000 7f27c6ffd700 1 -- 172.21.15.90:0/3153549364 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1054211420 0 0) 0x7f27d01ac500 con 0x7f27d0107890 2024-04-14T15:25:18.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.382+0000 7f27c6ffd700 1 -- 172.21.15.90:0/3153549364 >> v1:172.21.15.90:6790/0 conn(0x7f27d010a740 legacy=0x7f27d006b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:18.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.382+0000 7f27c6ffd700 1 -- 172.21.15.90:0/3153549364 >> v1:172.21.15.92:6789/0 conn(0x7f27d010e310 legacy=0x7f27d01a8bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:18.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.382+0000 7f27c6ffd700 1 -- 172.21.15.90:0/3153549364 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27d01ad6e0 con 0x7f27d0107890 2024-04-14T15:25:18.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.382+0000 7f27d7b51700 1 -- 172.21.15.90:0/3153549364 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f27d01ac730 con 0x7f27d0107890 2024-04-14T15:25:18.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.382+0000 7f27d7b51700 10 client.37989.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:25:18.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.382+0000 7f27d7b51700 1 -- 172.21.15.90:0/3153549364 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f27d01acbf0 con 0x7f27d0107890 2024-04-14T15:25:18.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.382+0000 7f27d7b51700 1 librados: init done 2024-04-14T15:25:18.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.382+0000 7f27c6ffd700 1 -- 172.21.15.90:0/3153549364 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f27cc003b20 con 0x7f27d0107890 2024-04-14T15:25:18.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.382+0000 7f27c6ffd700 1 -- 172.21.15.90:0/3153549364 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f27cc005c00 con 0x7f27d0107890 2024-04-14T15:25:18.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.382+0000 7f27d7b51700 1 -- 172.21.15.90:0/3153549364 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f27b4005100 con 0x7f27d0107890 2024-04-14T15:25:18.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.383+0000 7f27c6ffd700 1 -- 172.21.15.90:0/3153549364 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f27cc003680 con 0x7f27d0107890 2024-04-14T15:25:18.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.384+0000 7f27c6ffd700 1 -- 172.21.15.90:0/3153549364 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(774..774 src has 467..774) ==== 8508+0+0 (unknown 3343603281 0 0) 0x7f27cc08aa70 con 0x7f27d0107890 2024-04-14T15:25:18.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.384+0000 7f27c6ffd700 10 client.37989.objecter ms_dispatch 0x7f27d0001550 osd_map(774..774 src has 467..774) 2024-04-14T15:25:18.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.384+0000 7f27c6ffd700 3 client.37989.objecter handle_osd_map got epochs [774,774] > 0 2024-04-14T15:25:18.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.384+0000 7f27c6ffd700 3 client.37989.objecter handle_osd_map decoding full epoch 774 2024-04-14T15:25:18.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.384+0000 7f27c6ffd700 20 client.37989.objecter dump_active .. 0 homeless 2024-04-14T15:25:18.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.384+0000 7f27c6ffd700 1 -- 172.21.15.90:0/3153549364 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v774) ==== 54+0+16 (unknown 426413159 0 2420000192) 0x7f27cc08b060 con 0x7f27d0107890 2024-04-14T15:25:18.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.384+0000 7f27c6ffd700 10 client.37989.objecter ms_handle_connect 0x7f27bc0713a0 2024-04-14T15:25:18.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.384+0000 7f27d7b51700 10 client.37989.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:25:18.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.384+0000 7f27d7b51700 15 client.37989.objecter _session_command_op_assign -1 1 2024-04-14T15:25:18.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.384+0000 7f27d7b51700 20 client.37989.objecter _get_session s=0x7f27b40053e0 osd=0 3 2024-04-14T15:25:18.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.384+0000 7f27d7b51700 20 client.37989.objecter put_session s=0x7f27b40053e0 osd=0 3 2024-04-14T15:25:18.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.384+0000 7f27d7b51700 20 client.37989.objecter _get_session s=0x7f27b40053e0 osd=0 3 2024-04-14T15:25:18.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.384+0000 7f27d7b51700 15 client.37989.objecter _session_command_op_remove -1 1 2024-04-14T15:25:18.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.384+0000 7f27d7b51700 20 client.37989.objecter get_session s=0x7f27b40053e0 osd=0 3 2024-04-14T15:25:18.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.384+0000 7f27d7b51700 15 client.37989.objecter _session_command_op_assign 0 1 2024-04-14T15:25:18.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.384+0000 7f27d7b51700 20 client.37989.objecter put_session s=0x7f27b40053e0 osd=0 4 2024-04-14T15:25:18.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.384+0000 7f27d7b51700 10 client.37989.objecter _send_command 1 2024-04-14T15:25:18.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.384+0000 7f27d7b51700 1 -- 172.21.15.90:0/3153549364 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f27b4004b50 con 0x7f27b4004730 2024-04-14T15:25:18.650 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:25:18.650 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:25:18.650 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:25:18.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.385+0000 7f27c6ffd700 10 client.37989.objecter ms_handle_connect 0x7f27b4004730 2024-04-14T15:25:18.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.385+0000 7f27c6ffd700 1 -- 172.21.15.90:0/3153549364 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f27b4004b50 con 0x7f27b4004730 2024-04-14T15:25:18.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.385+0000 7f27c6ffd700 10 client.37989.objecter ms_dispatch 0x7f27d0001550 command_reply(tid 1: 0 ) 2024-04-14T15:25:18.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.385+0000 7f27c6ffd700 10 client.37989.objecter _finish_command 1 = system:0 2024-04-14T15:25:18.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.385+0000 7f27c6ffd700 20 client.37989.objecter put_session s=0x7f27b40053e0 osd=0 4 2024-04-14T15:25:18.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.385+0000 7f27c6ffd700 15 client.37989.objecter _session_command_op_remove 0 1 2024-04-14T15:25:18.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.402+0000 7f27d7b51700 10 client.37989.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:25:18.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.402+0000 7f27d7b51700 15 client.37989.objecter _session_command_op_assign -1 2 2024-04-14T15:25:18.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.402+0000 7f27d7b51700 20 client.37989.objecter _get_session s=0x7f27b40053e0 osd=0 3 2024-04-14T15:25:18.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.402+0000 7f27d7b51700 20 client.37989.objecter put_session s=0x7f27b40053e0 osd=0 3 2024-04-14T15:25:18.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.402+0000 7f27d7b51700 20 client.37989.objecter _get_session s=0x7f27b40053e0 osd=0 3 2024-04-14T15:25:18.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.402+0000 7f27d7b51700 15 client.37989.objecter _session_command_op_remove -1 2 2024-04-14T15:25:18.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.402+0000 7f27d7b51700 20 client.37989.objecter get_session s=0x7f27b40053e0 osd=0 3 2024-04-14T15:25:18.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.402+0000 7f27d7b51700 15 client.37989.objecter _session_command_op_assign 0 2 2024-04-14T15:25:18.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.402+0000 7f27d7b51700 20 client.37989.objecter put_session s=0x7f27b40053e0 osd=0 4 2024-04-14T15:25:18.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.402+0000 7f27d7b51700 10 client.37989.objecter _send_command 2 2024-04-14T15:25:18.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.402+0000 7f27d7b51700 1 -- 172.21.15.90:0/3153549364 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f27b40079f0 con 0x7f27b4004730 2024-04-14T15:25:18.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.404+0000 7f27c6ffd700 1 -- 172.21.15.90:0/3153549364 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f27b40079f0 con 0x7f27b4004730 2024-04-14T15:25:18.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.404+0000 7f27c6ffd700 10 client.37989.objecter ms_dispatch 0x7f27d0001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:25:18.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.404+0000 7f27c6ffd700 10 client.37989.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:25:18.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.404+0000 7f27c6ffd700 20 client.37989.objecter put_session s=0x7f27b40053e0 osd=0 4 2024-04-14T15:25:18.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.404+0000 7f27c6ffd700 15 client.37989.objecter _session_command_op_remove 0 2 2024-04-14T15:25:18.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.404+0000 7f27d7b51700 10 client.37989.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:25:18.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.404+0000 7f27d7b51700 15 client.37989.objecter _session_command_op_assign -1 3 2024-04-14T15:25:18.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.404+0000 7f27d7b51700 20 client.37989.objecter _get_session s=0x7f27b4004110 osd=1 3 2024-04-14T15:25:18.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.404+0000 7f27d7b51700 20 client.37989.objecter put_session s=0x7f27b4004110 osd=1 3 2024-04-14T15:25:18.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.404+0000 7f27d7b51700 20 client.37989.objecter _get_session s=0x7f27b4004110 osd=1 3 2024-04-14T15:25:18.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.404+0000 7f27d7b51700 15 client.37989.objecter _session_command_op_remove -1 3 2024-04-14T15:25:18.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.404+0000 7f27d7b51700 20 client.37989.objecter get_session s=0x7f27b4004110 osd=1 3 2024-04-14T15:25:18.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.404+0000 7f27d7b51700 15 client.37989.objecter _session_command_op_assign 1 3 2024-04-14T15:25:18.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.404+0000 7f27d7b51700 20 client.37989.objecter put_session s=0x7f27b4004110 osd=1 4 2024-04-14T15:25:18.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.404+0000 7f27d7b51700 10 client.37989.objecter _send_command 3 2024-04-14T15:25:18.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.404+0000 7f27d7b51700 1 -- 172.21.15.90:0/3153549364 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f27b4002180 con 0x7f27b4004280 2024-04-14T15:25:18.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.404+0000 7f27c6ffd700 10 client.37989.objecter ms_handle_connect 0x7f27b4004280 2024-04-14T15:25:18.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.405+0000 7f27c6ffd700 1 -- 172.21.15.90:0/3153549364 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f27b4002180 con 0x7f27b4004280 2024-04-14T15:25:18.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.405+0000 7f27c6ffd700 10 client.37989.objecter ms_dispatch 0x7f27d0001550 command_reply(tid 3: 0 ) 2024-04-14T15:25:18.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.405+0000 7f27c6ffd700 10 client.37989.objecter _finish_command 3 = system:0 2024-04-14T15:25:18.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.405+0000 7f27c6ffd700 20 client.37989.objecter put_session s=0x7f27b4004110 osd=1 4 2024-04-14T15:25:18.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.405+0000 7f27c6ffd700 15 client.37989.objecter _session_command_op_remove 1 3 2024-04-14T15:25:18.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.440+0000 7f27d7b51700 10 client.37989.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:25:18.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.440+0000 7f27d7b51700 15 client.37989.objecter _session_command_op_assign -1 4 2024-04-14T15:25:18.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.440+0000 7f27d7b51700 20 client.37989.objecter _get_session s=0x7f27b4004110 osd=1 3 2024-04-14T15:25:18.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.440+0000 7f27d7b51700 20 client.37989.objecter put_session s=0x7f27b4004110 osd=1 3 2024-04-14T15:25:18.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.440+0000 7f27d7b51700 20 client.37989.objecter _get_session s=0x7f27b4004110 osd=1 3 2024-04-14T15:25:18.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.440+0000 7f27d7b51700 15 client.37989.objecter _session_command_op_remove -1 4 2024-04-14T15:25:18.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.440+0000 7f27d7b51700 20 client.37989.objecter get_session s=0x7f27b4004110 osd=1 3 2024-04-14T15:25:18.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.440+0000 7f27d7b51700 15 client.37989.objecter _session_command_op_assign 1 4 2024-04-14T15:25:18.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.440+0000 7f27d7b51700 20 client.37989.objecter put_session s=0x7f27b4004110 osd=1 4 2024-04-14T15:25:18.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.440+0000 7f27d7b51700 10 client.37989.objecter _send_command 4 2024-04-14T15:25:18.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.440+0000 7f27d7b51700 1 -- 172.21.15.90:0/3153549364 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f27b4001600 con 0x7f27b4004280 2024-04-14T15:25:18.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.442+0000 7f27c6ffd700 1 -- 172.21.15.90:0/3153549364 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f27b4001600 con 0x7f27b4004280 2024-04-14T15:25:18.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.442+0000 7f27c6ffd700 10 client.37989.objecter ms_dispatch 0x7f27d0001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:25:18.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.442+0000 7f27c6ffd700 10 client.37989.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:25:18.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.442+0000 7f27c6ffd700 20 client.37989.objecter put_session s=0x7f27b4004110 osd=1 4 2024-04-14T15:25:18.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.442+0000 7f27c6ffd700 15 client.37989.objecter _session_command_op_remove 1 4 2024-04-14T15:25:18.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.442+0000 7f27d7b51700 10 client.37989.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:25:18.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.442+0000 7f27d7b51700 15 client.37989.objecter _session_command_op_assign -1 5 2024-04-14T15:25:18.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.442+0000 7f27d7b51700 20 client.37989.objecter _get_session s=0x7f27b4001a00 osd=2 3 2024-04-14T15:25:18.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.442+0000 7f27d7b51700 20 client.37989.objecter put_session s=0x7f27b4001a00 osd=2 3 2024-04-14T15:25:18.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.442+0000 7f27d7b51700 20 client.37989.objecter _get_session s=0x7f27b4001a00 osd=2 3 2024-04-14T15:25:18.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.442+0000 7f27d7b51700 15 client.37989.objecter _session_command_op_remove -1 5 2024-04-14T15:25:18.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.442+0000 7f27d7b51700 20 client.37989.objecter get_session s=0x7f27b4001a00 osd=2 3 2024-04-14T15:25:18.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.442+0000 7f27d7b51700 15 client.37989.objecter _session_command_op_assign 2 5 2024-04-14T15:25:18.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.442+0000 7f27d7b51700 20 client.37989.objecter put_session s=0x7f27b4001a00 osd=2 4 2024-04-14T15:25:18.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.442+0000 7f27d7b51700 10 client.37989.objecter _send_command 5 2024-04-14T15:25:18.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.442+0000 7f27d7b51700 1 -- 172.21.15.90:0/3153549364 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f27b4010f20 con 0x7f27b4008880 2024-04-14T15:25:18.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.442+0000 7f27c6ffd700 10 client.37989.objecter ms_handle_connect 0x7f27b4008880 2024-04-14T15:25:18.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.443+0000 7f27c6ffd700 1 -- 172.21.15.90:0/3153549364 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f27b4010f20 con 0x7f27b4008880 2024-04-14T15:25:18.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.443+0000 7f27c6ffd700 10 client.37989.objecter ms_dispatch 0x7f27d0001550 command_reply(tid 5: 0 ) 2024-04-14T15:25:18.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.443+0000 7f27c6ffd700 10 client.37989.objecter _finish_command 5 = system:0 2024-04-14T15:25:18.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.443+0000 7f27c6ffd700 20 client.37989.objecter put_session s=0x7f27b4001a00 osd=2 4 2024-04-14T15:25:18.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.443+0000 7f27c6ffd700 15 client.37989.objecter _session_command_op_remove 2 5 2024-04-14T15:25:18.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.459+0000 7f27d7b51700 10 client.37989.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:25:18.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.459+0000 7f27d7b51700 15 client.37989.objecter _session_command_op_assign -1 6 2024-04-14T15:25:18.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.459+0000 7f27d7b51700 20 client.37989.objecter _get_session s=0x7f27b4001a00 osd=2 3 2024-04-14T15:25:18.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.459+0000 7f27d7b51700 20 client.37989.objecter put_session s=0x7f27b4001a00 osd=2 3 2024-04-14T15:25:18.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.459+0000 7f27d7b51700 20 client.37989.objecter _get_session s=0x7f27b4001a00 osd=2 3 2024-04-14T15:25:18.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.459+0000 7f27d7b51700 15 client.37989.objecter _session_command_op_remove -1 6 2024-04-14T15:25:18.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.459+0000 7f27d7b51700 20 client.37989.objecter get_session s=0x7f27b4001a00 osd=2 3 2024-04-14T15:25:18.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.459+0000 7f27d7b51700 15 client.37989.objecter _session_command_op_assign 2 6 2024-04-14T15:25:18.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.459+0000 7f27d7b51700 20 client.37989.objecter put_session s=0x7f27b4001a00 osd=2 4 2024-04-14T15:25:18.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.459+0000 7f27d7b51700 10 client.37989.objecter _send_command 6 2024-04-14T15:25:18.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.459+0000 7f27d7b51700 1 -- 172.21.15.90:0/3153549364 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f27b4007f10 con 0x7f27b4008880 2024-04-14T15:25:18.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.461+0000 7f27c6ffd700 1 -- 172.21.15.90:0/3153549364 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f27b4007f10 con 0x7f27b4008880 2024-04-14T15:25:18.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.461+0000 7f27c6ffd700 10 client.37989.objecter ms_dispatch 0x7f27d0001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:25:18.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.461+0000 7f27c6ffd700 10 client.37989.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:25:18.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.461+0000 7f27c6ffd700 20 client.37989.objecter put_session s=0x7f27b4001a00 osd=2 4 2024-04-14T15:25:18.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.461+0000 7f27c6ffd700 15 client.37989.objecter _session_command_op_remove 2 6 2024-04-14T15:25:18.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.461+0000 7f27d7b51700 10 client.37989.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:25:18.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.461+0000 7f27d7b51700 15 client.37989.objecter _session_command_op_assign -1 7 2024-04-14T15:25:18.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.461+0000 7f27d7b51700 20 client.37989.objecter _get_session s=0x7f27b40081a0 osd=3 3 2024-04-14T15:25:18.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.461+0000 7f27d7b51700 20 client.37989.objecter put_session s=0x7f27b40081a0 osd=3 3 2024-04-14T15:25:18.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.461+0000 7f27d7b51700 20 client.37989.objecter _get_session s=0x7f27b40081a0 osd=3 3 2024-04-14T15:25:18.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.461+0000 7f27d7b51700 15 client.37989.objecter _session_command_op_remove -1 7 2024-04-14T15:25:18.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.461+0000 7f27d7b51700 20 client.37989.objecter get_session s=0x7f27b40081a0 osd=3 3 2024-04-14T15:25:18.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.461+0000 7f27d7b51700 15 client.37989.objecter _session_command_op_assign 3 7 2024-04-14T15:25:18.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.461+0000 7f27d7b51700 20 client.37989.objecter put_session s=0x7f27b40081a0 osd=3 4 2024-04-14T15:25:18.657 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:25:18.657 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:25:18.658 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:25:18.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.461+0000 7f27d7b51700 10 client.37989.objecter _send_command 7 2024-04-14T15:25:18.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.461+0000 7f27d7b51700 1 -- 172.21.15.90:0/3153549364 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f27b4011c70 con 0x7f27b4011100 2024-04-14T15:25:18.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.462+0000 7f27c6ffd700 10 client.37989.objecter ms_handle_connect 0x7f27b4011100 2024-04-14T15:25:18.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.462+0000 7f27c6ffd700 1 -- 172.21.15.90:0/3153549364 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f27b4011c70 con 0x7f27b4011100 2024-04-14T15:25:18.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.462+0000 7f27c6ffd700 10 client.37989.objecter ms_dispatch 0x7f27d0001550 command_reply(tid 7: 0 ) 2024-04-14T15:25:18.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.462+0000 7f27c6ffd700 10 client.37989.objecter _finish_command 7 = system:0 2024-04-14T15:25:18.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.462+0000 7f27c6ffd700 20 client.37989.objecter put_session s=0x7f27b40081a0 osd=3 4 2024-04-14T15:25:18.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.462+0000 7f27c6ffd700 15 client.37989.objecter _session_command_op_remove 3 7 2024-04-14T15:25:18.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.487+0000 7f27d7b51700 10 client.37989.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:25:18.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.487+0000 7f27d7b51700 15 client.37989.objecter _session_command_op_assign -1 8 2024-04-14T15:25:18.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.487+0000 7f27d7b51700 20 client.37989.objecter _get_session s=0x7f27b40081a0 osd=3 3 2024-04-14T15:25:18.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.487+0000 7f27d7b51700 20 client.37989.objecter put_session s=0x7f27b40081a0 osd=3 3 2024-04-14T15:25:18.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.487+0000 7f27d7b51700 20 client.37989.objecter _get_session s=0x7f27b40081a0 osd=3 3 2024-04-14T15:25:18.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.487+0000 7f27d7b51700 15 client.37989.objecter _session_command_op_remove -1 8 2024-04-14T15:25:18.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.487+0000 7f27d7b51700 20 client.37989.objecter get_session s=0x7f27b40081a0 osd=3 3 2024-04-14T15:25:18.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.487+0000 7f27d7b51700 15 client.37989.objecter _session_command_op_assign 3 8 2024-04-14T15:25:18.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.487+0000 7f27d7b51700 20 client.37989.objecter put_session s=0x7f27b40081a0 osd=3 4 2024-04-14T15:25:18.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.487+0000 7f27d7b51700 10 client.37989.objecter _send_command 8 2024-04-14T15:25:18.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.487+0000 7f27d7b51700 1 -- 172.21.15.90:0/3153549364 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f27b4012620 con 0x7f27b4011100 2024-04-14T15:25:18.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.490+0000 7f27c6ffd700 1 -- 172.21.15.90:0/3153549364 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f27b4012620 con 0x7f27b4011100 2024-04-14T15:25:18.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.490+0000 7f27c6ffd700 10 client.37989.objecter ms_dispatch 0x7f27d0001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:25:18.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.490+0000 7f27c6ffd700 10 client.37989.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:25:18.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.490+0000 7f27c6ffd700 20 client.37989.objecter put_session s=0x7f27b40081a0 osd=3 4 2024-04-14T15:25:18.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.490+0000 7f27c6ffd700 15 client.37989.objecter _session_command_op_remove 3 8 2024-04-14T15:25:18.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.490+0000 7f27d7b51700 10 client.37989.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:25:18.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.490+0000 7f27d7b51700 15 client.37989.objecter _session_command_op_assign -1 9 2024-04-14T15:25:18.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.490+0000 7f27d7b51700 20 client.37989.objecter _get_session s=0x7f27b40137a0 osd=4 3 2024-04-14T15:25:18.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.490+0000 7f27d7b51700 20 client.37989.objecter put_session s=0x7f27b40137a0 osd=4 3 2024-04-14T15:25:18.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.490+0000 7f27d7b51700 20 client.37989.objecter _get_session s=0x7f27b40137a0 osd=4 3 2024-04-14T15:25:18.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.490+0000 7f27d7b51700 15 client.37989.objecter _session_command_op_remove -1 9 2024-04-14T15:25:18.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.490+0000 7f27d7b51700 20 client.37989.objecter get_session s=0x7f27b40137a0 osd=4 3 2024-04-14T15:25:18.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.490+0000 7f27d7b51700 15 client.37989.objecter _session_command_op_assign 4 9 2024-04-14T15:25:18.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.490+0000 7f27d7b51700 20 client.37989.objecter put_session s=0x7f27b40137a0 osd=4 4 2024-04-14T15:25:18.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.490+0000 7f27d7b51700 10 client.37989.objecter _send_command 9 2024-04-14T15:25:18.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.490+0000 7f27d7b51700 1 -- 172.21.15.90:0/3153549364 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f27b401a720 con 0x7f27b4016ae0 2024-04-14T15:25:18.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.491+0000 7f27c6ffd700 10 client.37989.objecter ms_handle_connect 0x7f27b4016ae0 2024-04-14T15:25:18.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.492+0000 7f27c6ffd700 1 -- 172.21.15.90:0/3153549364 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f27b401a720 con 0x7f27b4016ae0 2024-04-14T15:25:18.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.492+0000 7f27c6ffd700 10 client.37989.objecter ms_dispatch 0x7f27d0001550 command_reply(tid 9: 0 ) 2024-04-14T15:25:18.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.492+0000 7f27c6ffd700 10 client.37989.objecter _finish_command 9 = system:0 2024-04-14T15:25:18.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.492+0000 7f27c6ffd700 20 client.37989.objecter put_session s=0x7f27b40137a0 osd=4 4 2024-04-14T15:25:18.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.492+0000 7f27c6ffd700 15 client.37989.objecter _session_command_op_remove 4 9 2024-04-14T15:25:18.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.508+0000 7f27d7b51700 10 client.37989.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:25:18.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.508+0000 7f27d7b51700 15 client.37989.objecter _session_command_op_assign -1 10 2024-04-14T15:25:18.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.508+0000 7f27d7b51700 20 client.37989.objecter _get_session s=0x7f27b40137a0 osd=4 3 2024-04-14T15:25:18.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.508+0000 7f27d7b51700 20 client.37989.objecter put_session s=0x7f27b40137a0 osd=4 3 2024-04-14T15:25:18.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.508+0000 7f27d7b51700 20 client.37989.objecter _get_session s=0x7f27b40137a0 osd=4 3 2024-04-14T15:25:18.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.508+0000 7f27d7b51700 15 client.37989.objecter _session_command_op_remove -1 10 2024-04-14T15:25:18.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.508+0000 7f27d7b51700 20 client.37989.objecter get_session s=0x7f27b40137a0 osd=4 3 2024-04-14T15:25:18.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.508+0000 7f27d7b51700 15 client.37989.objecter _session_command_op_assign 4 10 2024-04-14T15:25:18.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.508+0000 7f27d7b51700 20 client.37989.objecter put_session s=0x7f27b40137a0 osd=4 4 2024-04-14T15:25:18.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.508+0000 7f27d7b51700 10 client.37989.objecter _send_command 10 2024-04-14T15:25:18.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.508+0000 7f27d7b51700 1 -- 172.21.15.90:0/3153549364 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f27b401a6e0 con 0x7f27b4016ae0 2024-04-14T15:25:18.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.510+0000 7f27c6ffd700 1 -- 172.21.15.90:0/3153549364 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f27b401a6e0 con 0x7f27b4016ae0 2024-04-14T15:25:18.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.510+0000 7f27c6ffd700 10 client.37989.objecter ms_dispatch 0x7f27d0001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:25:18.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.510+0000 7f27c6ffd700 10 client.37989.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:25:18.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.510+0000 7f27c6ffd700 20 client.37989.objecter put_session s=0x7f27b40137a0 osd=4 4 2024-04-14T15:25:18.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.510+0000 7f27c6ffd700 15 client.37989.objecter _session_command_op_remove 4 10 2024-04-14T15:25:18.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.510+0000 7f27d7b51700 10 client.37989.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:25:18.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.510+0000 7f27d7b51700 15 client.37989.objecter _session_command_op_assign -1 11 2024-04-14T15:25:18.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.510+0000 7f27d7b51700 20 client.37989.objecter _get_session s=0x7f27b400c760 osd=5 3 2024-04-14T15:25:18.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.510+0000 7f27d7b51700 20 client.37989.objecter put_session s=0x7f27b400c760 osd=5 3 2024-04-14T15:25:18.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.510+0000 7f27d7b51700 20 client.37989.objecter _get_session s=0x7f27b400c760 osd=5 3 2024-04-14T15:25:18.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.510+0000 7f27d7b51700 15 client.37989.objecter _session_command_op_remove -1 11 2024-04-14T15:25:18.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.510+0000 7f27d7b51700 20 client.37989.objecter get_session s=0x7f27b400c760 osd=5 3 2024-04-14T15:25:18.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.510+0000 7f27d7b51700 15 client.37989.objecter _session_command_op_assign 5 11 2024-04-14T15:25:18.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.510+0000 7f27d7b51700 20 client.37989.objecter put_session s=0x7f27b400c760 osd=5 4 2024-04-14T15:25:18.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.510+0000 7f27d7b51700 10 client.37989.objecter _send_command 11 2024-04-14T15:25:18.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.510+0000 7f27d7b51700 1 -- 172.21.15.90:0/3153549364 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f27b401be90 con 0x7f27b401b310 2024-04-14T15:25:18.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.511+0000 7f27c6ffd700 10 client.37989.objecter ms_handle_connect 0x7f27b401b310 2024-04-14T15:25:18.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.512+0000 7f27c6ffd700 1 -- 172.21.15.90:0/3153549364 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f27b401be90 con 0x7f27b401b310 2024-04-14T15:25:18.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.512+0000 7f27c6ffd700 10 client.37989.objecter ms_dispatch 0x7f27d0001550 command_reply(tid 11: 0 ) 2024-04-14T15:25:18.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.512+0000 7f27c6ffd700 10 client.37989.objecter _finish_command 11 = system:0 2024-04-14T15:25:18.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.512+0000 7f27c6ffd700 20 client.37989.objecter put_session s=0x7f27b400c760 osd=5 4 2024-04-14T15:25:18.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.512+0000 7f27c6ffd700 15 client.37989.objecter _session_command_op_remove 5 11 2024-04-14T15:25:18.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.528+0000 7f27d7b51700 10 client.37989.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:25:18.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.528+0000 7f27d7b51700 15 client.37989.objecter _session_command_op_assign -1 12 2024-04-14T15:25:18.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.528+0000 7f27d7b51700 20 client.37989.objecter _get_session s=0x7f27b400c760 osd=5 3 2024-04-14T15:25:18.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.528+0000 7f27d7b51700 20 client.37989.objecter put_session s=0x7f27b400c760 osd=5 3 2024-04-14T15:25:18.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.528+0000 7f27d7b51700 20 client.37989.objecter _get_session s=0x7f27b400c760 osd=5 3 2024-04-14T15:25:18.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.528+0000 7f27d7b51700 15 client.37989.objecter _session_command_op_remove -1 12 2024-04-14T15:25:18.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.528+0000 7f27d7b51700 20 client.37989.objecter get_session s=0x7f27b400c760 osd=5 3 2024-04-14T15:25:18.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.528+0000 7f27d7b51700 15 client.37989.objecter _session_command_op_assign 5 12 2024-04-14T15:25:18.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.528+0000 7f27d7b51700 20 client.37989.objecter put_session s=0x7f27b400c760 osd=5 4 2024-04-14T15:25:18.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.528+0000 7f27d7b51700 10 client.37989.objecter _send_command 12 2024-04-14T15:25:18.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.528+0000 7f27d7b51700 1 -- 172.21.15.90:0/3153549364 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f27b401b0c0 con 0x7f27b401b310 2024-04-14T15:25:18.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.530+0000 7f27c6ffd700 1 -- 172.21.15.90:0/3153549364 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f27b401b0c0 con 0x7f27b401b310 2024-04-14T15:25:18.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.530+0000 7f27c6ffd700 10 client.37989.objecter ms_dispatch 0x7f27d0001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:25:18.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.530+0000 7f27c6ffd700 10 client.37989.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:25:18.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.530+0000 7f27c6ffd700 20 client.37989.objecter put_session s=0x7f27b400c760 osd=5 4 2024-04-14T15:25:18.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.530+0000 7f27c6ffd700 15 client.37989.objecter _session_command_op_remove 5 12 2024-04-14T15:25:18.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.531+0000 7f27d7b51700 10 client.37989.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:25:18.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.531+0000 7f27d7b51700 15 client.37989.objecter _session_command_op_assign -1 13 2024-04-14T15:25:18.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.531+0000 7f27d7b51700 20 client.37989.objecter _get_session s=0x7f27b401b0c0 osd=6 3 2024-04-14T15:25:18.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.531+0000 7f27d7b51700 20 client.37989.objecter put_session s=0x7f27b401b0c0 osd=6 3 2024-04-14T15:25:18.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.531+0000 7f27d7b51700 20 client.37989.objecter _get_session s=0x7f27b401b0c0 osd=6 3 2024-04-14T15:25:18.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.531+0000 7f27d7b51700 15 client.37989.objecter _session_command_op_remove -1 13 2024-04-14T15:25:18.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.531+0000 7f27d7b51700 20 client.37989.objecter get_session s=0x7f27b401b0c0 osd=6 3 2024-04-14T15:25:18.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.531+0000 7f27d7b51700 15 client.37989.objecter _session_command_op_assign 6 13 2024-04-14T15:25:18.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.531+0000 7f27d7b51700 20 client.37989.objecter put_session s=0x7f27b401b0c0 osd=6 4 2024-04-14T15:25:18.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.531+0000 7f27d7b51700 10 client.37989.objecter _send_command 13 2024-04-14T15:25:18.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.531+0000 7f27d7b51700 1 -- 172.21.15.90:0/3153549364 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f27b4023180 con 0x7f27b401f600 2024-04-14T15:25:18.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.531+0000 7f27c6ffd700 10 client.37989.objecter ms_handle_connect 0x7f27b401f600 2024-04-14T15:25:18.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.532+0000 7f27c6ffd700 1 -- 172.21.15.90:0/3153549364 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f27b4023180 con 0x7f27b401f600 2024-04-14T15:25:18.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.532+0000 7f27c6ffd700 10 client.37989.objecter ms_dispatch 0x7f27d0001550 command_reply(tid 13: 0 ) 2024-04-14T15:25:18.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.532+0000 7f27c6ffd700 10 client.37989.objecter _finish_command 13 = system:0 2024-04-14T15:25:18.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.532+0000 7f27c6ffd700 20 client.37989.objecter put_session s=0x7f27b401b0c0 osd=6 4 2024-04-14T15:25:18.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.532+0000 7f27c6ffd700 15 client.37989.objecter _session_command_op_remove 6 13 2024-04-14T15:25:18.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.549+0000 7f27d7b51700 10 client.37989.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:25:18.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.549+0000 7f27d7b51700 15 client.37989.objecter _session_command_op_assign -1 14 2024-04-14T15:25:18.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.549+0000 7f27d7b51700 20 client.37989.objecter _get_session s=0x7f27b401b0c0 osd=6 3 2024-04-14T15:25:18.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.549+0000 7f27d7b51700 20 client.37989.objecter put_session s=0x7f27b401b0c0 osd=6 3 2024-04-14T15:25:18.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.549+0000 7f27d7b51700 20 client.37989.objecter _get_session s=0x7f27b401b0c0 osd=6 3 2024-04-14T15:25:18.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.549+0000 7f27d7b51700 15 client.37989.objecter _session_command_op_remove -1 14 2024-04-14T15:25:18.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.549+0000 7f27d7b51700 20 client.37989.objecter get_session s=0x7f27b401b0c0 osd=6 3 2024-04-14T15:25:18.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.549+0000 7f27d7b51700 15 client.37989.objecter _session_command_op_assign 6 14 2024-04-14T15:25:18.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.549+0000 7f27d7b51700 20 client.37989.objecter put_session s=0x7f27b401b0c0 osd=6 4 2024-04-14T15:25:18.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.549+0000 7f27d7b51700 10 client.37989.objecter _send_command 14 2024-04-14T15:25:18.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.549+0000 7f27d7b51700 1 -- 172.21.15.90:0/3153549364 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f27b4024b30 con 0x7f27b401f600 2024-04-14T15:25:18.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.551+0000 7f27c6ffd700 1 -- 172.21.15.90:0/3153549364 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f27b4024b30 con 0x7f27b401f600 2024-04-14T15:25:18.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.551+0000 7f27c6ffd700 10 client.37989.objecter ms_dispatch 0x7f27d0001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:25:18.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.551+0000 7f27c6ffd700 10 client.37989.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:25:18.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.551+0000 7f27c6ffd700 20 client.37989.objecter put_session s=0x7f27b401b0c0 osd=6 4 2024-04-14T15:25:18.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.551+0000 7f27c6ffd700 15 client.37989.objecter _session_command_op_remove 6 14 2024-04-14T15:25:18.666 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:25:18.666 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:25:18.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.551+0000 7f27d7b51700 10 client.37989.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:25:18.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.551+0000 7f27d7b51700 15 client.37989.objecter _session_command_op_assign -1 15 2024-04-14T15:25:18.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.551+0000 7f27d7b51700 20 client.37989.objecter _get_session s=0x7f27b4024b30 osd=7 3 2024-04-14T15:25:18.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.551+0000 7f27d7b51700 20 client.37989.objecter put_session s=0x7f27b4024b30 osd=7 3 2024-04-14T15:25:18.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.551+0000 7f27d7b51700 20 client.37989.objecter _get_session s=0x7f27b4024b30 osd=7 3 2024-04-14T15:25:18.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.551+0000 7f27d7b51700 15 client.37989.objecter _session_command_op_remove -1 15 2024-04-14T15:25:18.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.551+0000 7f27d7b51700 20 client.37989.objecter get_session s=0x7f27b4024b30 osd=7 3 2024-04-14T15:25:18.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.551+0000 7f27d7b51700 15 client.37989.objecter _session_command_op_assign 7 15 2024-04-14T15:25:18.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.551+0000 7f27d7b51700 20 client.37989.objecter put_session s=0x7f27b4024b30 osd=7 4 2024-04-14T15:25:18.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.551+0000 7f27d7b51700 10 client.37989.objecter _send_command 15 2024-04-14T15:25:18.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.551+0000 7f27d7b51700 1 -- 172.21.15.90:0/3153549364 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f27b4025620 con 0x7f27b40251b0 2024-04-14T15:25:18.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.552+0000 7f27c6ffd700 10 client.37989.objecter ms_handle_connect 0x7f27b40251b0 2024-04-14T15:25:18.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.553+0000 7f27c6ffd700 1 -- 172.21.15.90:0/3153549364 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f27b4025620 con 0x7f27b40251b0 2024-04-14T15:25:18.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.553+0000 7f27c6ffd700 10 client.37989.objecter ms_dispatch 0x7f27d0001550 command_reply(tid 15: 0 ) 2024-04-14T15:25:18.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.553+0000 7f27c6ffd700 10 client.37989.objecter _finish_command 15 = system:0 2024-04-14T15:25:18.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.553+0000 7f27c6ffd700 20 client.37989.objecter put_session s=0x7f27b4024b30 osd=7 4 2024-04-14T15:25:18.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.553+0000 7f27c6ffd700 15 client.37989.objecter _session_command_op_remove 7 15 2024-04-14T15:25:18.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.569+0000 7f27d7b51700 10 client.37989.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:25:18.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.569+0000 7f27d7b51700 15 client.37989.objecter _session_command_op_assign -1 16 2024-04-14T15:25:18.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.569+0000 7f27d7b51700 20 client.37989.objecter _get_session s=0x7f27b4024b30 osd=7 3 2024-04-14T15:25:18.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.569+0000 7f27d7b51700 20 client.37989.objecter put_session s=0x7f27b4024b30 osd=7 3 2024-04-14T15:25:18.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.569+0000 7f27d7b51700 20 client.37989.objecter _get_session s=0x7f27b4024b30 osd=7 3 2024-04-14T15:25:18.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.569+0000 7f27d7b51700 15 client.37989.objecter _session_command_op_remove -1 16 2024-04-14T15:25:18.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.569+0000 7f27d7b51700 20 client.37989.objecter get_session s=0x7f27b4024b30 osd=7 3 2024-04-14T15:25:18.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.569+0000 7f27d7b51700 15 client.37989.objecter _session_command_op_assign 7 16 2024-04-14T15:25:18.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.569+0000 7f27d7b51700 20 client.37989.objecter put_session s=0x7f27b4024b30 osd=7 4 2024-04-14T15:25:18.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.569+0000 7f27d7b51700 10 client.37989.objecter _send_command 16 2024-04-14T15:25:18.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.569+0000 7f27d7b51700 1 -- 172.21.15.90:0/3153549364 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f27b4025620 con 0x7f27b40251b0 2024-04-14T15:25:18.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.571+0000 7f27c6ffd700 1 -- 172.21.15.90:0/3153549364 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f27b4025620 con 0x7f27b40251b0 2024-04-14T15:25:18.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.571+0000 7f27c6ffd700 10 client.37989.objecter ms_dispatch 0x7f27d0001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:25:18.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.571+0000 7f27c6ffd700 10 client.37989.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:25:18.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.571+0000 7f27c6ffd700 20 client.37989.objecter put_session s=0x7f27b4024b30 osd=7 4 2024-04-14T15:25:18.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.571+0000 7f27c6ffd700 15 client.37989.objecter _session_command_op_remove 7 16 2024-04-14T15:25:18.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.572+0000 7f27d7b51700 10 librados: watch_flush enter 2024-04-14T15:25:18.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.572+0000 7f27d7b51700 10 librados: watch_flush exit 2024-04-14T15:25:18.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.572+0000 7f27d7b51700 10 client.37989.objecter close_session for osd.0 2024-04-14T15:25:18.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.572+0000 7f27d7b51700 1 -- 172.21.15.90:0/3153549364 >> v1:172.21.15.90:6801/3761806885 conn(0x7f27b4004730 legacy=0x7f27b4002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:18.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.572+0000 7f27d7b51700 20 client.37989.objecter put_session s=0x7f27b40053e0 osd=0 1 2024-04-14T15:25:18.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.572+0000 7f27d7b51700 10 client.37989.objecter close_session for osd.1 2024-04-14T15:25:18.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.572+0000 7f27d7b51700 1 -- 172.21.15.90:0/3153549364 >> v1:172.21.15.90:6805/4170361699 conn(0x7f27b4004280 legacy=0x7f27b400ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:18.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.572+0000 7f27d7b51700 20 client.37989.objecter put_session s=0x7f27b4004110 osd=1 1 2024-04-14T15:25:18.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.572+0000 7f27d7b51700 10 client.37989.objecter close_session for osd.2 2024-04-14T15:25:18.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.572+0000 7f27d7b51700 1 -- 172.21.15.90:0/3153549364 >> v1:172.21.15.90:6809/1686184251 conn(0x7f27b4008880 legacy=0x7f27b400f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:18.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.572+0000 7f27d7b51700 20 client.37989.objecter put_session s=0x7f27b4001a00 osd=2 1 2024-04-14T15:25:18.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.572+0000 7f27d7b51700 10 client.37989.objecter close_session for osd.3 2024-04-14T15:25:18.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.572+0000 7f27d7b51700 1 -- 172.21.15.90:0/3153549364 >> v1:172.21.15.90:6813/2270166774 conn(0x7f27b4011100 legacy=0x7f27b4011560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:18.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.572+0000 7f27d7b51700 20 client.37989.objecter put_session s=0x7f27b40081a0 osd=3 1 2024-04-14T15:25:18.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.572+0000 7f27d7b51700 10 client.37989.objecter close_session for osd.4 2024-04-14T15:25:18.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.572+0000 7f27d7b51700 1 -- 172.21.15.90:0/3153549364 >> v1:172.21.15.92:6800/3887549733 conn(0x7f27b4016ae0 legacy=0x7f27b4018ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:18.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.572+0000 7f27d7b51700 20 client.37989.objecter put_session s=0x7f27b40137a0 osd=4 1 2024-04-14T15:25:18.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.572+0000 7f27d7b51700 10 client.37989.objecter close_session for osd.5 2024-04-14T15:25:18.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.572+0000 7f27d7b51700 1 -- 172.21.15.90:0/3153549364 >> v1:172.21.15.92:6804/3244561440 conn(0x7f27b401b310 legacy=0x7f27b401b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:18.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.572+0000 7f27d7b51700 20 client.37989.objecter put_session s=0x7f27b400c760 osd=5 1 2024-04-14T15:25:18.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.572+0000 7f27d7b51700 10 client.37989.objecter close_session for osd.6 2024-04-14T15:25:18.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.572+0000 7f27d7b51700 1 -- 172.21.15.90:0/3153549364 >> v1:172.21.15.92:6808/636125963 conn(0x7f27b401f600 legacy=0x7f27b40219f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:18.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.572+0000 7f27d60ee700 1 -- 172.21.15.90:0/3153549364 reap_dead start 2024-04-14T15:25:18.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.572+0000 7f27d7b51700 20 client.37989.objecter put_session s=0x7f27b401b0c0 osd=6 1 2024-04-14T15:25:18.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.572+0000 7f27d7b51700 10 client.37989.objecter close_session for osd.7 2024-04-14T15:25:18.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.572+0000 7f27d7b51700 1 -- 172.21.15.90:0/3153549364 >> v1:172.21.15.92:6812/1773706716 conn(0x7f27b40251b0 legacy=0x7f27b4023360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:18.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.572+0000 7f27d60ee700 1 -- 172.21.15.90:0/3153549364 reap_dead start 2024-04-14T15:25:18.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.572+0000 7f27d60ee700 1 -- 172.21.15.90:0/3153549364 reap_dead start 2024-04-14T15:25:18.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.572+0000 7f27d60ee700 1 -- 172.21.15.90:0/3153549364 reap_dead start 2024-04-14T15:25:18.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.572+0000 7f27d7b51700 20 client.37989.objecter put_session s=0x7f27b4024b30 osd=7 1 2024-04-14T15:25:18.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.572+0000 7f27d7b51700 20 client.37989.objecter shutdown clearing up homeless session... 2024-04-14T15:25:18.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.572+0000 7f27d7b51700 10 client.37989.objecter successfully canceled tick 2024-04-14T15:25:18.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.572+0000 7f27d7b51700 1 -- 172.21.15.90:0/3153549364 >> v1:172.21.15.90:6800/420935427 conn(0x7f27bc0713a0 legacy=0x7f27bc073860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:18.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.572+0000 7f27d7b51700 1 -- 172.21.15.90:0/3153549364 >> v1:172.21.15.90:6789/0 conn(0x7f27d0107890 legacy=0x7f27d006acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:18.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.572+0000 7f27d7b51700 1 -- 172.21.15.90:0/3153549364 shutdown_connections 2024-04-14T15:25:18.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.572+0000 7f27d7b51700 1 -- 172.21.15.90:0/3153549364 >> 172.21.15.90:0/3153549364 conn(0x7f27d00fd290 msgr2=0x7f27d01118f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:25:18.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.572+0000 7f27d7b51700 1 -- 172.21.15.90:0/3153549364 shutdown_connections 2024-04-14T15:25:18.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.572+0000 7f27d7b51700 1 -- 172.21.15.90:0/3153549364 wait complete. 2024-04-14T15:25:18.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:18.572+0000 7f27d7b51700 1 librados: shutdown 2024-04-14T15:25:18.672 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:25:18.672 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:25:18.672 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:25:18.672 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:25:18.672 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:25:18.672 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:25:18.673 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:25:18.673 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:25:18.685 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:25:18.708 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:18.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:18.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:25:18.809 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:25:18.838 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:18.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:18.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:25:18.940 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:25:18.966 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:18.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:18.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:25:19.068 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:25:19.093 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:19.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:19.094 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:25:19.163 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:25:19.195 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:25:19.218 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:19.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:19.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:25:19.319 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:25:19.343 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:19.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:19.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:25:19.392 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-14T15:25:19.392 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:25:19.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:19 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3153549364' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:25:19.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:19 smithi090 ceph-mon[41670]: osdmap e775: 8 total, 8 up, 7 in 2024-04-14T15:25:19.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:19 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:25:19.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:19 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:25:19.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:19 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3153549364' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:25:19.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:19 smithi090 ceph-mon[30399]: osdmap e775: 8 total, 8 up, 7 in 2024-04-14T15:25:19.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:19 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:25:19.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:19 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:25:19.444 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:25:19.472 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:19.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:19.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:25:19.573 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:25:19.601 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:19.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:19.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:25:19.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:19 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3153549364' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:25:19.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:19 smithi092 ceph-mon[33966]: osdmap e775: 8 total, 8 up, 7 in 2024-04-14T15:25:19.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:19 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:25:19.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:19 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:25:19.703 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:25:19.726 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:19.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:19.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:25:19.828 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:25:19.851 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:19.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:19.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:25:19.953 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:25:19.981 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:19.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:19.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:25:20.083 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:25:20.108 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:20.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:20.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:25:20.210 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:25:20.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:20.237 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:20.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:25:20.283 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-14T15:25:20.338 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:20.361 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:20.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:20.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:20.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:20 smithi090 ceph-mon[30399]: 11.8 scrub starts 2024-04-14T15:25:20.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:20 smithi090 ceph-mon[30399]: 11.8 scrub ok 2024-04-14T15:25:20.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:20 smithi090 ceph-mon[30399]: pgmap v1147: 224 pgs: 32 creating+peering, 4 active+remapped+backfill_toofull, 188 active+clean; 4.2 MiB data, 744 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s; 2/648 objects misplaced (0.309%) 2024-04-14T15:25:20.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:20 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:25:20.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:20 smithi090 ceph-mon[30399]: osdmap e776: 8 total, 8 up, 7 in 2024-04-14T15:25:20.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:20 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:25:20.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:20 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:25:20.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:20 smithi090 ceph-mon[41670]: 11.8 scrub starts 2024-04-14T15:25:20.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:20 smithi090 ceph-mon[41670]: 11.8 scrub ok 2024-04-14T15:25:20.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:20 smithi090 ceph-mon[41670]: pgmap v1147: 224 pgs: 32 creating+peering, 4 active+remapped+backfill_toofull, 188 active+clean; 4.2 MiB data, 744 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s; 2/648 objects misplaced (0.309%) 2024-04-14T15:25:20.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:20 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:25:20.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:20 smithi090 ceph-mon[41670]: osdmap e776: 8 total, 8 up, 7 in 2024-04-14T15:25:20.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:20 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:25:20.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:20 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:25:20.463 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:25:20.486 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:20.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:20.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:25:20.588 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:25:20.614 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:20.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:20.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:25:20.621 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:20 smithi092 ceph-mon[33966]: 11.8 scrub starts 2024-04-14T15:25:20.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:20 smithi092 ceph-mon[33966]: 11.8 scrub ok 2024-04-14T15:25:20.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:20 smithi092 ceph-mon[33966]: pgmap v1147: 224 pgs: 32 creating+peering, 4 active+remapped+backfill_toofull, 188 active+clean; 4.2 MiB data, 744 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s; 2/648 objects misplaced (0.309%) 2024-04-14T15:25:20.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:20 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:25:20.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:20 smithi092 ceph-mon[33966]: osdmap e776: 8 total, 8 up, 7 in 2024-04-14T15:25:20.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:20 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:25:20.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:20 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:25:20.716 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:25:20.744 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:20.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:20.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:25:20.846 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:25:20.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:20.873 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:20.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:25:20.974 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:25:20.997 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:20.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:20.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:25:21.099 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:25:21.122 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:21.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:21.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:25:21.226 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:21.250 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:21.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:21.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:21.352 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:21.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:21.376 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:21.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:21.415 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:21 smithi090 ceph-mon[30399]: 4.2 scrub starts 2024-04-14T15:25:21.415 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:21 smithi090 ceph-mon[30399]: 4.2 scrub ok 2024-04-14T15:25:21.415 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:21 smithi090 ceph-mon[30399]: 11.15 deep-scrub starts 2024-04-14T15:25:21.415 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:21 smithi090 ceph-mon[30399]: 11.15 deep-scrub ok 2024-04-14T15:25:21.416 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:21 smithi090 ceph-mon[30399]: 323.3s0 scrub starts 2024-04-14T15:25:21.416 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:21 smithi090 ceph-mon[30399]: 323.3 scrub ok 2024-04-14T15:25:21.416 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:21 smithi090 ceph-mon[30399]: 4.e scrub starts 2024-04-14T15:25:21.416 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:21 smithi090 ceph-mon[30399]: 4.e scrub ok 2024-04-14T15:25:21.416 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:21 smithi090 ceph-mon[30399]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:25:21.416 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:21 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-138", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:25:21.416 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:21 smithi090 ceph-mon[30399]: osdmap e777: 8 total, 8 up, 7 in 2024-04-14T15:25:21.416 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:21 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-138","var": "hit_set_count","val": "2"}]: dispatch 2024-04-14T15:25:21.416 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:21 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-138","var": "hit_set_count","val": "2"}]: dispatch 2024-04-14T15:25:21.416 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:21 smithi090 ceph-mon[41670]: 4.2 scrub starts 2024-04-14T15:25:21.417 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:21 smithi090 ceph-mon[41670]: 4.2 scrub ok 2024-04-14T15:25:21.417 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:21 smithi090 ceph-mon[41670]: 11.15 deep-scrub starts 2024-04-14T15:25:21.417 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:21 smithi090 ceph-mon[41670]: 11.15 deep-scrub ok 2024-04-14T15:25:21.417 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:21 smithi090 ceph-mon[41670]: 323.3s0 scrub starts 2024-04-14T15:25:21.417 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:21 smithi090 ceph-mon[41670]: 323.3 scrub ok 2024-04-14T15:25:21.417 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:21 smithi090 ceph-mon[41670]: 4.e scrub starts 2024-04-14T15:25:21.417 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:21 smithi090 ceph-mon[41670]: 4.e scrub ok 2024-04-14T15:25:21.417 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:21 smithi090 ceph-mon[41670]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:25:21.417 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:21 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-138", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:25:21.417 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:21 smithi090 ceph-mon[41670]: osdmap e777: 8 total, 8 up, 7 in 2024-04-14T15:25:21.417 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:21 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-138","var": "hit_set_count","val": "2"}]: dispatch 2024-04-14T15:25:21.417 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:21 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-138","var": "hit_set_count","val": "2"}]: dispatch 2024-04-14T15:25:21.477 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:25:21.505 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:21.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:21.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:25:21.607 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:21.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:21 smithi092 ceph-mon[33966]: 4.2 scrub starts 2024-04-14T15:25:21.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:21 smithi092 ceph-mon[33966]: 4.2 scrub ok 2024-04-14T15:25:21.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:21 smithi092 ceph-mon[33966]: 11.15 deep-scrub starts 2024-04-14T15:25:21.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:21 smithi092 ceph-mon[33966]: 11.15 deep-scrub ok 2024-04-14T15:25:21.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:21 smithi092 ceph-mon[33966]: 323.3s0 scrub starts 2024-04-14T15:25:21.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:21 smithi092 ceph-mon[33966]: 323.3 scrub ok 2024-04-14T15:25:21.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:21 smithi092 ceph-mon[33966]: 4.e scrub starts 2024-04-14T15:25:21.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:21 smithi092 ceph-mon[33966]: 4.e scrub ok 2024-04-14T15:25:21.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:21 smithi092 ceph-mon[33966]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:25:21.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:21 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-138", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:25:21.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:21 smithi092 ceph-mon[33966]: osdmap e777: 8 total, 8 up, 7 in 2024-04-14T15:25:21.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:21 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-138","var": "hit_set_count","val": "2"}]: dispatch 2024-04-14T15:25:21.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:21 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-138","var": "hit_set_count","val": "2"}]: dispatch 2024-04-14T15:25:21.631 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:21.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:21.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:21.733 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:21.756 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:21.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:21.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:21.858 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:25:21.886 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:21.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:21.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:25:21.988 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:25:22.012 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:22.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:22.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:25:22.114 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:25:22.141 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:22.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:22.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:25:22.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:22 smithi090 ceph-mon[30399]: 11.4 scrub starts 2024-04-14T15:25:22.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:22 smithi090 ceph-mon[30399]: 11.4 scrub ok 2024-04-14T15:25:22.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:22 smithi090 ceph-mon[30399]: 4.1e deep-scrub starts 2024-04-14T15:25:22.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:22 smithi090 ceph-mon[30399]: 4.1e deep-scrub ok 2024-04-14T15:25:22.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:22 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:25:22.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:22 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:25:22.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:22 smithi090 ceph-mon[30399]: pgmap v1150: 224 pgs: 32 creating+peering, 4 active+remapped+backfill_toofull, 188 active+clean; 4.2 MiB data, 744 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s; 2/648 objects misplaced (0.309%) 2024-04-14T15:25:22.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:22 smithi090 ceph-mon[30399]: 4.1a scrub starts 2024-04-14T15:25:22.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:22 smithi090 ceph-mon[30399]: 4.1a scrub ok 2024-04-14T15:25:22.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:22 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-138","var": "hit_set_count","val": "2"}]': finished 2024-04-14T15:25:22.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:22 smithi090 ceph-mon[30399]: osdmap e778: 8 total, 8 up, 7 in 2024-04-14T15:25:22.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:22 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-138","var": "hit_set_period","val": "600"}]: dispatch 2024-04-14T15:25:22.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:22 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-138","var": "hit_set_period","val": "600"}]: dispatch 2024-04-14T15:25:22.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:22 smithi090 ceph-mon[41670]: 11.4 scrub starts 2024-04-14T15:25:22.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:22 smithi090 ceph-mon[41670]: 11.4 scrub ok 2024-04-14T15:25:22.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:22 smithi090 ceph-mon[41670]: 4.1e deep-scrub starts 2024-04-14T15:25:22.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:22 smithi090 ceph-mon[41670]: 4.1e deep-scrub ok 2024-04-14T15:25:22.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:22 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:25:22.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:22 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:25:22.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:22 smithi090 ceph-mon[41670]: pgmap v1150: 224 pgs: 32 creating+peering, 4 active+remapped+backfill_toofull, 188 active+clean; 4.2 MiB data, 744 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s; 2/648 objects misplaced (0.309%) 2024-04-14T15:25:22.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:22 smithi090 ceph-mon[41670]: 4.1a scrub starts 2024-04-14T15:25:22.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:22 smithi090 ceph-mon[41670]: 4.1a scrub ok 2024-04-14T15:25:22.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:22 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-138","var": "hit_set_count","val": "2"}]': finished 2024-04-14T15:25:22.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:22 smithi090 ceph-mon[41670]: osdmap e778: 8 total, 8 up, 7 in 2024-04-14T15:25:22.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:22 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-138","var": "hit_set_period","val": "600"}]: dispatch 2024-04-14T15:25:22.194 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:22 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-138","var": "hit_set_period","val": "600"}]: dispatch 2024-04-14T15:25:22.242 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:25:22.266 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:22.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:22.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:25:22.368 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:25:22.395 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:22.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:22.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:25:22.497 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:25:22.520 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:22.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:22.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:25:22.522 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:22 smithi092 ceph-mon[33966]: 11.4 scrub starts 2024-04-14T15:25:22.523 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:22 smithi092 ceph-mon[33966]: 11.4 scrub ok 2024-04-14T15:25:22.523 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:22 smithi092 ceph-mon[33966]: 4.1e deep-scrub starts 2024-04-14T15:25:22.523 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:22 smithi092 ceph-mon[33966]: 4.1e deep-scrub ok 2024-04-14T15:25:22.523 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:22 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:25:22.523 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:22 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:25:22.523 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:22 smithi092 ceph-mon[33966]: pgmap v1150: 224 pgs: 32 creating+peering, 4 active+remapped+backfill_toofull, 188 active+clean; 4.2 MiB data, 744 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s; 2/648 objects misplaced (0.309%) 2024-04-14T15:25:22.523 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:22 smithi092 ceph-mon[33966]: 4.1a scrub starts 2024-04-14T15:25:22.523 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:22 smithi092 ceph-mon[33966]: 4.1a scrub ok 2024-04-14T15:25:22.523 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:22 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-138","var": "hit_set_count","val": "2"}]': finished 2024-04-14T15:25:22.523 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:22 smithi092 ceph-mon[33966]: osdmap e778: 8 total, 8 up, 7 in 2024-04-14T15:25:22.523 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:22 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-138","var": "hit_set_period","val": "600"}]: dispatch 2024-04-14T15:25:22.523 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:22 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-138","var": "hit_set_period","val": "600"}]: dispatch 2024-04-14T15:25:22.609 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:25:22.623 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:22.688 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:22.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:22.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:22.790 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:25:22.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:22.814 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:22.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:25:22.915 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:25:22.939 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:22.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:22.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:25:23.041 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:23.066 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:23.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:23.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:23.168 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:25:23.191 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:23.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:23.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:25:23.279 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:23 smithi090 ceph-mon[30399]: 11.d scrub starts 2024-04-14T15:25:23.279 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:23 smithi090 ceph-mon[30399]: 11.d scrub ok 2024-04-14T15:25:23.279 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:23 smithi090 ceph-mon[30399]: 11.b scrub starts 2024-04-14T15:25:23.279 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:23 smithi090 ceph-mon[30399]: 11.b scrub ok 2024-04-14T15:25:23.279 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:23 smithi090 ceph-mon[30399]: 6.18 scrub starts 2024-04-14T15:25:23.279 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:23 smithi090 ceph-mon[30399]: 6.18 scrub ok 2024-04-14T15:25:23.279 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:23 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:25:23.279 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:23 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-138","var": "hit_set_period","val": "600"}]': finished 2024-04-14T15:25:23.279 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:23 smithi090 ceph-mon[30399]: osdmap e779: 8 total, 8 up, 7 in 2024-04-14T15:25:23.280 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:23 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-14T15:25:23.280 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:23 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-14T15:25:23.280 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:23 smithi090 ceph-mon[41670]: 11.d scrub starts 2024-04-14T15:25:23.280 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:23 smithi090 ceph-mon[41670]: 11.d scrub ok 2024-04-14T15:25:23.280 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:23 smithi090 ceph-mon[41670]: 11.b scrub starts 2024-04-14T15:25:23.280 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:23 smithi090 ceph-mon[41670]: 11.b scrub ok 2024-04-14T15:25:23.280 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:23 smithi090 ceph-mon[41670]: 6.18 scrub starts 2024-04-14T15:25:23.280 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:23 smithi090 ceph-mon[41670]: 6.18 scrub ok 2024-04-14T15:25:23.280 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:23 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:25:23.280 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:23 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-138","var": "hit_set_period","val": "600"}]': finished 2024-04-14T15:25:23.280 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:23 smithi090 ceph-mon[41670]: osdmap e779: 8 total, 8 up, 7 in 2024-04-14T15:25:23.280 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:23 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-14T15:25:23.281 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:23 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-14T15:25:23.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:23.283+0000 7fd9f4226700 1 -- 172.21.15.90:0/3296695801 >> v1:172.21.15.90:6789/0 conn(0x7fd9ec10e360 legacy=0x7fd9ec110820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:23.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:23.284+0000 7fd9f4226700 1 -- 172.21.15.90:0/3296695801 shutdown_connections 2024-04-14T15:25:23.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:23.284+0000 7fd9f4226700 1 -- 172.21.15.90:0/3296695801 >> 172.21.15.90:0/3296695801 conn(0x7fd9ec0fd2c0 msgr2=0x7fd9ec0ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:25:23.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:23.284+0000 7fd9f4226700 1 -- 172.21.15.90:0/3296695801 shutdown_connections 2024-04-14T15:25:23.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:23.284+0000 7fd9f4226700 1 -- 172.21.15.90:0/3296695801 wait complete. 2024-04-14T15:25:23.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:23.284+0000 7fd9f4226700 1 librados: starting msgr at 2024-04-14T15:25:23.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:23.284+0000 7fd9f4226700 1 librados: starting objecter 2024-04-14T15:25:23.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:23.284+0000 7fd9f4226700 1 Processor -- start 2024-04-14T15:25:23.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:23.284+0000 7fd9f4226700 1 -- start start 2024-04-14T15:25:23.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:23.284+0000 7fd9f4226700 1 librados: setting wanted keys 2024-04-14T15:25:23.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:23.284+0000 7fd9f4226700 1 librados: calling monclient init 2024-04-14T15:25:23.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:23.284+0000 7fd9f4226700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd9ec10e100 con 0x7fd9ec1078e0 2024-04-14T15:25:23.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:23.284+0000 7fd9f4226700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd9ec1a8f80 con 0x7fd9ec10e360 2024-04-14T15:25:23.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:23.284+0000 7fd9f4226700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd9ec1aa1a0 con 0x7fd9ec10a790 2024-04-14T15:25:23.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:23.285+0000 7fd9f1fc2700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fd9ec1078e0 0x7fd9ec10bf20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:33324/0 (socket says 172.21.15.90:33324) 2024-04-14T15:25:23.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:23.285+0000 7fd9f1fc2700 1 -- 172.21.15.90:0/773064505 learned_addr learned my addr 172.21.15.90:0/773064505 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:25:23.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:23.285+0000 7fd9e2ffd700 10 client.?.objecter ms_handle_connect 0x7fd9ec1078e0 2024-04-14T15:25:23.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:23.285+0000 7fd9e2ffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:25:23.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:23.285+0000 7fd9e2ffd700 10 client.?.objecter ms_handle_connect 0x7fd9ec10a790 2024-04-14T15:25:23.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:23.285+0000 7fd9e2ffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:25:23.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:23.285+0000 7fd9e2ffd700 10 client.?.objecter ms_handle_connect 0x7fd9ec10e360 2024-04-14T15:25:23.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:23.285+0000 7fd9e2ffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:25:23.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:23.285+0000 7fd9e2ffd700 1 -- 172.21.15.90:0/773064505 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2810309314 0 0) 0x7fd9ec1aa1a0 con 0x7fd9ec10a790 2024-04-14T15:25:23.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:23.285+0000 7fd9e2ffd700 1 -- 172.21.15.90:0/773064505 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9d80031f0 con 0x7fd9ec10a790 2024-04-14T15:25:23.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:23.285+0000 7fd9e2ffd700 1 -- 172.21.15.90:0/773064505 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 371075588 0 0) 0x7fd9ec10e100 con 0x7fd9ec1078e0 2024-04-14T15:25:23.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:23.285+0000 7fd9e2ffd700 1 -- 172.21.15.90:0/773064505 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9ec1aa1a0 con 0x7fd9ec1078e0 2024-04-14T15:25:23.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:23.285+0000 7fd9e2ffd700 1 -- 172.21.15.90:0/773064505 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3425874952 0 0) 0x7fd9ec1a8f80 con 0x7fd9ec10e360 2024-04-14T15:25:23.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:23.285+0000 7fd9e2ffd700 1 -- 172.21.15.90:0/773064505 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9ec10e100 con 0x7fd9ec10e360 2024-04-14T15:25:23.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:23.285+0000 7fd9e2ffd700 1 -- 172.21.15.90:0/773064505 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1546516883 0 0) 0x7fd9ec1aa1a0 con 0x7fd9ec1078e0 2024-04-14T15:25:23.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:23.285+0000 7fd9e2ffd700 1 -- 172.21.15.90:0/773064505 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd9ec1a8f80 con 0x7fd9ec1078e0 2024-04-14T15:25:23.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:23.285+0000 7fd9e2ffd700 1 -- 172.21.15.90:0/773064505 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fd9e8001870 con 0x7fd9ec1078e0 2024-04-14T15:25:23.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:23.285+0000 7fd9e2ffd700 1 -- 172.21.15.90:0/773064505 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 684398206 0 0) 0x7fd9d80031f0 con 0x7fd9ec10a790 2024-04-14T15:25:23.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:23.285+0000 7fd9e2ffd700 1 -- 172.21.15.90:0/773064505 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd9ec1aa1a0 con 0x7fd9ec10a790 2024-04-14T15:25:23.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:23.285+0000 7fd9e2ffd700 1 -- 172.21.15.90:0/773064505 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fd9dc0018b0 con 0x7fd9ec10a790 2024-04-14T15:25:23.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:23.285+0000 7fd9e2ffd700 1 -- 172.21.15.90:0/773064505 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3783906772 0 0) 0x7fd9ec10e100 con 0x7fd9ec10e360 2024-04-14T15:25:23.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:23.285+0000 7fd9e2ffd700 1 -- 172.21.15.90:0/773064505 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd9d80031f0 con 0x7fd9ec10e360 2024-04-14T15:25:23.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:23.285+0000 7fd9e2ffd700 1 -- 172.21.15.90:0/773064505 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fd9e4001950 con 0x7fd9ec10e360 2024-04-14T15:25:23.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:23.285+0000 7fd9e2ffd700 1 -- 172.21.15.90:0/773064505 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 411602861 0 0) 0x7fd9ec1a8f80 con 0x7fd9ec1078e0 2024-04-14T15:25:23.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:23.285+0000 7fd9e2ffd700 1 -- 172.21.15.90:0/773064505 >> v1:172.21.15.90:6790/0 conn(0x7fd9ec10a790 legacy=0x7fd9ec10c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:23.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:23.285+0000 7fd9e2ffd700 1 -- 172.21.15.90:0/773064505 >> v1:172.21.15.92:6789/0 conn(0x7fd9ec10e360 legacy=0x7fd9ec1a6850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:23.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:23.285+0000 7fd9e2ffd700 1 -- 172.21.15.90:0/773064505 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9ec1ab3e0 con 0x7fd9ec1078e0 2024-04-14T15:25:23.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:23.285+0000 7fd9f4226700 1 -- 172.21.15.90:0/773064505 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd9ec1aa3d0 con 0x7fd9ec1078e0 2024-04-14T15:25:23.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:23.286+0000 7fd9f4226700 10 client.37995.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:25:23.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:23.286+0000 7fd9f4226700 1 -- 172.21.15.90:0/773064505 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd9ec1aa890 con 0x7fd9ec1078e0 2024-04-14T15:25:23.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:23.286+0000 7fd9f4226700 1 librados: init done 2024-04-14T15:25:23.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:23.286+0000 7fd9e2ffd700 1 -- 172.21.15.90:0/773064505 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fd9e8003b40 con 0x7fd9ec1078e0 2024-04-14T15:25:23.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:23.286+0000 7fd9e2ffd700 1 -- 172.21.15.90:0/773064505 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fd9e8005cc0 con 0x7fd9ec1078e0 2024-04-14T15:25:23.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:23.287+0000 7fd9f4226700 1 -- 172.21.15.90:0/773064505 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9d0005120 con 0x7fd9ec1078e0 2024-04-14T15:25:23.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:23.287+0000 7fd9e2ffd700 1 -- 172.21.15.90:0/773064505 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fd9e80036a0 con 0x7fd9ec1078e0 2024-04-14T15:25:23.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:23.288+0000 7fd9e2ffd700 1 -- 172.21.15.90:0/773064505 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(779..779 src has 467..779) ==== 8889+0+0 (unknown 3127388570 0 0) 0x7fd9e808ac90 con 0x7fd9ec1078e0 2024-04-14T15:25:23.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:23.288+0000 7fd9e2ffd700 10 client.37995.objecter ms_dispatch 0x7fd9ec001550 osd_map(779..779 src has 467..779) 2024-04-14T15:25:23.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:23.288+0000 7fd9e2ffd700 3 client.37995.objecter handle_osd_map got epochs [779,779] > 0 2024-04-14T15:25:23.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:23.288+0000 7fd9e2ffd700 3 client.37995.objecter handle_osd_map decoding full epoch 779 2024-04-14T15:25:23.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:23.288+0000 7fd9e2ffd700 20 client.37995.objecter dump_active .. 0 homeless 2024-04-14T15:25:23.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:23.288+0000 7fd9e2ffd700 10 client.37995.objecter ms_handle_connect 0x7fd9d80713e0 2024-04-14T15:25:23.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:23.292+0000 7fd9e2ffd700 1 -- 172.21.15.90:0/773064505 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fd9e80573e0 con 0x7fd9ec1078e0 2024-04-14T15:25:23.293 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:23.317 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:23.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:23.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:23.420 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:25:23.445 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:23.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:23.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:25:23.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:23.525+0000 7fd9f4226700 1 -- 172.21.15.90:0/773064505 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd reweight", "id": 2, "weight": 1.0} v 0) -- 0x7fd9d0005410 con 0x7fd9ec1078e0 2024-04-14T15:25:23.547 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:25:23.570 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:23.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:23.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:25:23.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:23 smithi092 ceph-mon[33966]: 11.d scrub starts 2024-04-14T15:25:23.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:23 smithi092 ceph-mon[33966]: 11.d scrub ok 2024-04-14T15:25:23.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:23 smithi092 ceph-mon[33966]: 11.b scrub starts 2024-04-14T15:25:23.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:23 smithi092 ceph-mon[33966]: 11.b scrub ok 2024-04-14T15:25:23.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:23 smithi092 ceph-mon[33966]: 6.18 scrub starts 2024-04-14T15:25:23.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:23 smithi092 ceph-mon[33966]: 6.18 scrub ok 2024-04-14T15:25:23.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:23 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:25:23.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:23 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-138","var": "hit_set_period","val": "600"}]': finished 2024-04-14T15:25:23.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:23 smithi092 ceph-mon[33966]: osdmap e779: 8 total, 8 up, 7 in 2024-04-14T15:25:23.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:23 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-14T15:25:23.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:23 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-14T15:25:23.671 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:25:23.695 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:23.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:23.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:25:23.796 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:25:23.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:23.820 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:23.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:25:23.921 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:23.945 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:23.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:23.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:24.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:24.026+0000 7fd9e2ffd700 1 -- 172.21.15.90:0/773064505 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 2, "weight": 1.0}]=0 reweighted osd.2 to 1 (10000) v780) ==== 113+0+0 (unknown 1713476474 0 0) 0x7fd9e805b090 con 0x7fd9ec1078e0 2024-04-14T15:25:24.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:24.030+0000 7fd9f4226700 10 librados: watch_flush enter 2024-04-14T15:25:24.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:24.030+0000 7fd9f4226700 10 librados: watch_flush exit 2024-04-14T15:25:24.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:24.030+0000 7fd9f4226700 20 client.37995.objecter shutdown clearing up homeless session... 2024-04-14T15:25:24.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:24.030+0000 7fd9f4226700 10 client.37995.objecter successfully canceled tick 2024-04-14T15:25:24.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:24.031+0000 7fd9f4226700 1 -- 172.21.15.90:0/773064505 >> v1:172.21.15.90:6800/420935427 conn(0x7fd9d80713e0 legacy=0x7fd9d80738a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:24.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:24.031+0000 7fd9f4226700 1 -- 172.21.15.90:0/773064505 >> v1:172.21.15.90:6789/0 conn(0x7fd9ec1078e0 legacy=0x7fd9ec10bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:24.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:24.031+0000 7fd9f4226700 1 -- 172.21.15.90:0/773064505 shutdown_connections 2024-04-14T15:25:24.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:24.031+0000 7fd9f4226700 1 -- 172.21.15.90:0/773064505 >> 172.21.15.90:0/773064505 conn(0x7fd9ec0fd2c0 msgr2=0x7fd9ec101870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:25:24.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:24.031+0000 7fd9f4226700 1 -- 172.21.15.90:0/773064505 shutdown_connections 2024-04-14T15:25:24.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:24.032+0000 7fd9f4226700 1 -- 172.21.15.90:0/773064505 wait complete. 2024-04-14T15:25:24.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:24.032+0000 7fd9f4226700 1 librados: shutdown 2024-04-14T15:25:24.033 INFO:teuthology.orchestra.run.smithi090.stderr:reweighted osd.2 to 1 (10000) 2024-04-14T15:25:24.047 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:25:24.071 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:24.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:24.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:25:24.172 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:25:24.196 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:24.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:24.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:25:24.298 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:25:24.321 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:24.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:24.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:25:24.423 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:25:24.441 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:25:23 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:25:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:25:24.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:24 smithi090 ceph-mon[41670]: 5.1d deep-scrub starts 2024-04-14T15:25:24.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:24 smithi090 ceph-mon[41670]: 5.1d deep-scrub ok 2024-04-14T15:25:24.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:24 smithi090 ceph-mon[41670]: 5.6 scrub starts 2024-04-14T15:25:24.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:24 smithi090 ceph-mon[41670]: 5.6 scrub ok 2024-04-14T15:25:24.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:24 smithi090 ceph-mon[41670]: 2.14 scrub starts 2024-04-14T15:25:24.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:24 smithi090 ceph-mon[41670]: 2.14 scrub ok 2024-04-14T15:25:24.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:24 smithi090 ceph-mon[41670]: 11.1e scrub starts 2024-04-14T15:25:24.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:24 smithi090 ceph-mon[41670]: 11.1e scrub ok 2024-04-14T15:25:24.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:24 smithi090 ceph-mon[41670]: pgmap v1153: 224 pgs: 32 creating+peering, 4 active+remapped+backfill_toofull, 188 active+clean; 4.2 MiB data, 744 MiB used, 625 GiB / 626 GiB avail; 2/648 objects misplaced (0.309%) 2024-04-14T15:25:24.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:24 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:25:24.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:24 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:25:24.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:24 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:25:24.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:24 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/773064505' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-14T15:25:24.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:24 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:25:24.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:24 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:25:24.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:24 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-138","var": "hit_set_type","val": "explicit_object"}]': finished 2024-04-14T15:25:24.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:24 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/773064505' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-14T15:25:24.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:24 smithi090 ceph-mon[41670]: osdmap e780: 8 total, 8 up, 7 in 2024-04-14T15:25:24.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:24 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:25:24.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:24 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:25:24.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:24 smithi090 ceph-mon[30399]: 5.1d deep-scrub starts 2024-04-14T15:25:24.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:24 smithi090 ceph-mon[30399]: 5.1d deep-scrub ok 2024-04-14T15:25:24.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:24 smithi090 ceph-mon[30399]: 5.6 scrub starts 2024-04-14T15:25:24.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:24 smithi090 ceph-mon[30399]: 5.6 scrub ok 2024-04-14T15:25:24.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:24 smithi090 ceph-mon[30399]: 2.14 scrub starts 2024-04-14T15:25:24.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:24 smithi090 ceph-mon[30399]: 2.14 scrub ok 2024-04-14T15:25:24.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:24 smithi090 ceph-mon[30399]: 11.1e scrub starts 2024-04-14T15:25:24.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:24 smithi090 ceph-mon[30399]: 11.1e scrub ok 2024-04-14T15:25:24.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:24 smithi090 ceph-mon[30399]: pgmap v1153: 224 pgs: 32 creating+peering, 4 active+remapped+backfill_toofull, 188 active+clean; 4.2 MiB data, 744 MiB used, 625 GiB / 626 GiB avail; 2/648 objects misplaced (0.309%) 2024-04-14T15:25:24.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:24 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:25:24.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:24 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:25:24.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:24 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:25:24.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:24 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/773064505' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-14T15:25:24.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:24 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:25:24.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:24 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:25:24.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:24 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-138","var": "hit_set_type","val": "explicit_object"}]': finished 2024-04-14T15:25:24.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:24 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/773064505' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-14T15:25:24.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:24 smithi090 ceph-mon[30399]: osdmap e780: 8 total, 8 up, 7 in 2024-04-14T15:25:24.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:24 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:25:24.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:24 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:25:24.447 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:24.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:24.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:25:24.549 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:25:24.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:24.574 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:24.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:25:24.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:24 smithi092 ceph-mon[33966]: 5.1d deep-scrub starts 2024-04-14T15:25:24.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:24 smithi092 ceph-mon[33966]: 5.1d deep-scrub ok 2024-04-14T15:25:24.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:24 smithi092 ceph-mon[33966]: 5.6 scrub starts 2024-04-14T15:25:24.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:24 smithi092 ceph-mon[33966]: 5.6 scrub ok 2024-04-14T15:25:24.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:24 smithi092 ceph-mon[33966]: 2.14 scrub starts 2024-04-14T15:25:24.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:24 smithi092 ceph-mon[33966]: 2.14 scrub ok 2024-04-14T15:25:24.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:24 smithi092 ceph-mon[33966]: 11.1e scrub starts 2024-04-14T15:25:24.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:24 smithi092 ceph-mon[33966]: 11.1e scrub ok 2024-04-14T15:25:24.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:24 smithi092 ceph-mon[33966]: pgmap v1153: 224 pgs: 32 creating+peering, 4 active+remapped+backfill_toofull, 188 active+clean; 4.2 MiB data, 744 MiB used, 625 GiB / 626 GiB avail; 2/648 objects misplaced (0.309%) 2024-04-14T15:25:24.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:24 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:25:24.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:24 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:25:24.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:24 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:25:24.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:24 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/773064505' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-14T15:25:24.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:24 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:25:24.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:24 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:25:24.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:24 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-138","var": "hit_set_type","val": "explicit_object"}]': finished 2024-04-14T15:25:24.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:24 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/773064505' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-14T15:25:24.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:24 smithi092 ceph-mon[33966]: osdmap e780: 8 total, 8 up, 7 in 2024-04-14T15:25:24.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:24 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:25:24.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:24 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:25:24.625 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-14T15:25:24.676 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:25:24.700 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:24.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:24.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:25:24.801 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:25:24.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:24.826 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:24.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:25:24.927 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:25:24.953 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:24.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:24.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:25:24.964 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd reweight 3 1 2024-04-14T15:25:25.032 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:25:25.054 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:25:25.078 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:25.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:25.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:25:25.179 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:25:25.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:25 smithi090 ceph-mon[41670]: 4.d deep-scrub starts 2024-04-14T15:25:25.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:25 smithi090 ceph-mon[41670]: 4.d deep-scrub ok 2024-04-14T15:25:25.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:25 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:25:25.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:25 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:25:25.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:25 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-138"}]: dispatch 2024-04-14T15:25:25.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:25 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-138"}]: dispatch 2024-04-14T15:25:25.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:25 smithi090 ceph-mon[30399]: 4.d deep-scrub starts 2024-04-14T15:25:25.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:25 smithi090 ceph-mon[30399]: 4.d deep-scrub ok 2024-04-14T15:25:25.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:25 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:25:25.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:25 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:25:25.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:25 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-138"}]: dispatch 2024-04-14T15:25:25.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:25 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-138"}]: dispatch 2024-04-14T15:25:25.204 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:25.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:25.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:25:25.306 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:25:25.330 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:25.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:25.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:25:25.431 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:25:25.455 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:25.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:25.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:25:25.472 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:25 smithi092 ceph-mon[33966]: 4.d deep-scrub starts 2024-04-14T15:25:25.472 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:25 smithi092 ceph-mon[33966]: 4.d deep-scrub ok 2024-04-14T15:25:25.472 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:25 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:25:25.473 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:25 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:25:25.473 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:25 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-138"}]: dispatch 2024-04-14T15:25:25.473 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:25 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-138"}]: dispatch 2024-04-14T15:25:25.556 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:25.580 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:25.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:25.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:25.682 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:25:25.705 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:25.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:25.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:25:25.807 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:25.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:25.831 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:25.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:25.872 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:25:25 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:25:25.931 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:25:25.956 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:25.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:25.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:25:26.058 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:25:26.089 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:26.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:26.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:25:26.191 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:25:26.217 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:26.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:26.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:25:26.319 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:25:26.345 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:26.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:26.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:25:26.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:26 smithi090 ceph-mon[41670]: 11.12 scrub starts 2024-04-14T15:25:26.473 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:26 smithi090 ceph-mon[41670]: 11.12 scrub ok 2024-04-14T15:25:26.473 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:26 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-138"}]': finished 2024-04-14T15:25:26.473 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:26 smithi090 ceph-mon[41670]: osdmap e781: 8 total, 8 up, 7 in 2024-04-14T15:25:26.473 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:26 smithi090 ceph-mon[41670]: pgmap v1156: 224 pgs: 4 active+remapped+backfill_toofull, 220 active+clean; 4.2 MiB data, 749 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/648 objects misplaced (0.309%) 2024-04-14T15:25:26.473 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:26 smithi090 ceph-mon[41670]: 11.13 scrub starts 2024-04-14T15:25:26.473 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:26 smithi090 ceph-mon[41670]: 11.13 scrub ok 2024-04-14T15:25:26.473 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:26 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:25:26.474 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:26 smithi090 ceph-mon[41670]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:25:26.474 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:26 smithi090 ceph-mon[41670]: osdmap e782: 8 total, 8 up, 7 in 2024-04-14T15:25:26.474 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:26 smithi090 ceph-mon[30399]: 11.12 scrub starts 2024-04-14T15:25:26.475 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:26 smithi090 ceph-mon[30399]: 11.12 scrub ok 2024-04-14T15:25:26.475 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:26 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-138"}]': finished 2024-04-14T15:25:26.475 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:26 smithi090 ceph-mon[30399]: osdmap e781: 8 total, 8 up, 7 in 2024-04-14T15:25:26.475 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:26 smithi090 ceph-mon[30399]: pgmap v1156: 224 pgs: 4 active+remapped+backfill_toofull, 220 active+clean; 4.2 MiB data, 749 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/648 objects misplaced (0.309%) 2024-04-14T15:25:26.475 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:26 smithi090 ceph-mon[30399]: 11.13 scrub starts 2024-04-14T15:25:26.475 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:26 smithi090 ceph-mon[30399]: 11.13 scrub ok 2024-04-14T15:25:26.475 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:26 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:25:26.475 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:26 smithi090 ceph-mon[30399]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:25:26.475 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:26 smithi090 ceph-mon[30399]: osdmap e782: 8 total, 8 up, 7 in 2024-04-14T15:25:26.476 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:25:26.476 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:25:26.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:26.504 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:26.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:25:26.605 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:26.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:26 smithi092 ceph-mon[33966]: 11.12 scrub starts 2024-04-14T15:25:26.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:26 smithi092 ceph-mon[33966]: 11.12 scrub ok 2024-04-14T15:25:26.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:26 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-138"}]': finished 2024-04-14T15:25:26.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:26 smithi092 ceph-mon[33966]: osdmap e781: 8 total, 8 up, 7 in 2024-04-14T15:25:26.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:26 smithi092 ceph-mon[33966]: pgmap v1156: 224 pgs: 4 active+remapped+backfill_toofull, 220 active+clean; 4.2 MiB data, 749 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/648 objects misplaced (0.309%) 2024-04-14T15:25:26.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:26 smithi092 ceph-mon[33966]: 11.13 scrub starts 2024-04-14T15:25:26.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:26 smithi092 ceph-mon[33966]: 11.13 scrub ok 2024-04-14T15:25:26.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:26 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:25:26.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:26 smithi092 ceph-mon[33966]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:25:26.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:26 smithi092 ceph-mon[33966]: osdmap e782: 8 total, 8 up, 7 in 2024-04-14T15:25:26.629 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:26.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:26.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:26.726 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:25:26.731 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:25:26.758 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:26.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:26.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:25:26.860 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:25:26.884 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:26.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:26.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:25:26.986 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:27.010 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:27.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:27.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:27.112 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:27.135 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:27.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:27.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:27.237 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:25:27.265 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:27.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:27.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:25:27.367 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:25:27.391 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:27.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:27.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:25:27.493 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:25:27.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:27.521 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:27.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:25:27.622 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:27.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:27.648 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:27.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:27.749 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:27.773 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:27.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:27.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:27.875 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:25:27.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:27.905 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:27.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:25:28.006 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:25:28.037 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:28.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:28.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:25:28.131 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:28 smithi090 ceph-mon[41670]: 4.a deep-scrub starts 2024-04-14T15:25:28.131 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:28 smithi090 ceph-mon[41670]: 4.a deep-scrub ok 2024-04-14T15:25:28.131 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:28 smithi090 ceph-mon[41670]: 3.0 scrub starts 2024-04-14T15:25:28.131 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:28 smithi090 ceph-mon[41670]: 3.0 scrub ok 2024-04-14T15:25:28.131 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:28 smithi090 ceph-mon[41670]: 4.8 scrub starts 2024-04-14T15:25:28.131 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:28 smithi090 ceph-mon[41670]: 4.8 scrub ok 2024-04-14T15:25:28.132 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:28 smithi090 ceph-mon[41670]: osdmap e783: 8 total, 8 up, 7 in 2024-04-14T15:25:28.132 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:28 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:25:28.132 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:28 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:25:28.132 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:28 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:25:28.132 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:28 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:25:28.132 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:28 smithi090 ceph-mon[30399]: 4.a deep-scrub starts 2024-04-14T15:25:28.132 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:28 smithi090 ceph-mon[30399]: 4.a deep-scrub ok 2024-04-14T15:25:28.132 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:28 smithi090 ceph-mon[30399]: 3.0 scrub starts 2024-04-14T15:25:28.132 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:28 smithi090 ceph-mon[30399]: 3.0 scrub ok 2024-04-14T15:25:28.132 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:28 smithi090 ceph-mon[30399]: 4.8 scrub starts 2024-04-14T15:25:28.132 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:28 smithi090 ceph-mon[30399]: 4.8 scrub ok 2024-04-14T15:25:28.132 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:28 smithi090 ceph-mon[30399]: osdmap e783: 8 total, 8 up, 7 in 2024-04-14T15:25:28.133 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:28 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:25:28.133 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:28 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:25:28.133 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:28 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:25:28.133 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:28 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:25:28.139 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:25:28.166 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:28.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:28.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:25:28.268 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:28.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:28.295 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:28.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:28.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:28 smithi092 ceph-mon[33966]: 4.a deep-scrub starts 2024-04-14T15:25:28.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:28 smithi092 ceph-mon[33966]: 4.a deep-scrub ok 2024-04-14T15:25:28.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:28 smithi092 ceph-mon[33966]: 3.0 scrub starts 2024-04-14T15:25:28.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:28 smithi092 ceph-mon[33966]: 3.0 scrub ok 2024-04-14T15:25:28.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:28 smithi092 ceph-mon[33966]: 4.8 scrub starts 2024-04-14T15:25:28.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:28 smithi092 ceph-mon[33966]: 4.8 scrub ok 2024-04-14T15:25:28.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:28 smithi092 ceph-mon[33966]: osdmap e783: 8 total, 8 up, 7 in 2024-04-14T15:25:28.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:28 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:25:28.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:28 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:25:28.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:28 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:25:28.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:28 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:25:28.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:28.389+0000 7f85c83e5700 1 -- 172.21.15.90:0/509145943 >> v1:172.21.15.90:6789/0 conn(0x7f85c0108980 legacy=0x7f85c0108d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:28.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:28.389+0000 7f85c83e5700 1 -- 172.21.15.90:0/509145943 shutdown_connections 2024-04-14T15:25:28.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:28.389+0000 7f85c83e5700 1 -- 172.21.15.90:0/509145943 >> 172.21.15.90:0/509145943 conn(0x7f85c0076aa0 msgr2=0x7f85c0076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:25:28.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:28.389+0000 7f85c83e5700 1 -- 172.21.15.90:0/509145943 shutdown_connections 2024-04-14T15:25:28.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:28.389+0000 7f85c83e5700 1 -- 172.21.15.90:0/509145943 wait complete. 2024-04-14T15:25:28.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:28.389+0000 7f85c83e5700 1 librados: starting msgr at 2024-04-14T15:25:28.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:28.389+0000 7f85c83e5700 1 librados: starting objecter 2024-04-14T15:25:28.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:28.390+0000 7f85c83e5700 1 Processor -- start 2024-04-14T15:25:28.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:28.390+0000 7f85c83e5700 1 -- start start 2024-04-14T15:25:28.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:28.390+0000 7f85c83e5700 1 librados: setting wanted keys 2024-04-14T15:25:28.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:28.390+0000 7f85c83e5700 1 librados: calling monclient init 2024-04-14T15:25:28.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:28.390+0000 7f85c83e5700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f85c0108120 con 0x7f85c010f2f0 2024-04-14T15:25:28.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:28.390+0000 7f85c83e5700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f85c01a8f00 con 0x7f85c0108980 2024-04-14T15:25:28.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:28.390+0000 7f85c83e5700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f85c01aa0e0 con 0x7f85c010b6f0 2024-04-14T15:25:28.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:28.390+0000 7f85c5980700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f85c010b6f0 0x7f85c0105680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:59270/0 (socket says 172.21.15.90:59270) 2024-04-14T15:25:28.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:28.390+0000 7f85c5980700 1 -- 172.21.15.90:0/2132774417 learned_addr learned my addr 172.21.15.90:0/2132774417 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:25:28.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:28.390+0000 7f85b77fe700 10 client.?.objecter ms_handle_connect 0x7f85c010b6f0 2024-04-14T15:25:28.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:28.390+0000 7f85b77fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:25:28.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:28.390+0000 7f85b77fe700 10 client.?.objecter ms_handle_connect 0x7f85c010f2f0 2024-04-14T15:25:28.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:28.390+0000 7f85b77fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:25:28.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:28.390+0000 7f85b77fe700 10 client.?.objecter ms_handle_connect 0x7f85c0108980 2024-04-14T15:25:28.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:28.390+0000 7f85b77fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:25:28.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:28.390+0000 7f85b77fe700 1 -- 172.21.15.90:0/2132774417 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2411614249 0 0) 0x7f85c01aa0e0 con 0x7f85c010b6f0 2024-04-14T15:25:28.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:28.390+0000 7f85b77fe700 1 -- 172.21.15.90:0/2132774417 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f85ac0035b0 con 0x7f85c010b6f0 2024-04-14T15:25:28.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:28.390+0000 7f85b77fe700 1 -- 172.21.15.90:0/2132774417 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4238024463 0 0) 0x7f85c0108120 con 0x7f85c010f2f0 2024-04-14T15:25:28.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:28.390+0000 7f85b77fe700 1 -- 172.21.15.90:0/2132774417 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f85c01aa0e0 con 0x7f85c010f2f0 2024-04-14T15:25:28.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:28.390+0000 7f85b77fe700 1 -- 172.21.15.90:0/2132774417 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2653805523 0 0) 0x7f85c01a8f00 con 0x7f85c0108980 2024-04-14T15:25:28.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:28.390+0000 7f85b77fe700 1 -- 172.21.15.90:0/2132774417 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f85c0108120 con 0x7f85c0108980 2024-04-14T15:25:28.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:28.390+0000 7f85b77fe700 1 -- 172.21.15.90:0/2132774417 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3623992980 0 0) 0x7f85ac0035b0 con 0x7f85c010b6f0 2024-04-14T15:25:28.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:28.390+0000 7f85b77fe700 1 -- 172.21.15.90:0/2132774417 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f85c01a8f00 con 0x7f85c010b6f0 2024-04-14T15:25:28.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:28.391+0000 7f85b77fe700 1 -- 172.21.15.90:0/2132774417 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f85b0001930 con 0x7f85c010b6f0 2024-04-14T15:25:28.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:28.391+0000 7f85b77fe700 1 -- 172.21.15.90:0/2132774417 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4186962686 0 0) 0x7f85c01aa0e0 con 0x7f85c010f2f0 2024-04-14T15:25:28.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:28.391+0000 7f85b77fe700 1 -- 172.21.15.90:0/2132774417 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f85ac0035b0 con 0x7f85c010f2f0 2024-04-14T15:25:28.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:28.391+0000 7f85b77fe700 1 -- 172.21.15.90:0/2132774417 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3710254449 0 0) 0x7f85c0108120 con 0x7f85c0108980 2024-04-14T15:25:28.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:28.391+0000 7f85b77fe700 1 -- 172.21.15.90:0/2132774417 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f85c01aa0e0 con 0x7f85c0108980 2024-04-14T15:25:28.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:28.391+0000 7f85b77fe700 1 -- 172.21.15.90:0/2132774417 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f85bc001bb0 con 0x7f85c010f2f0 2024-04-14T15:25:28.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:28.391+0000 7f85b77fe700 1 -- 172.21.15.90:0/2132774417 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f85b8001430 con 0x7f85c0108980 2024-04-14T15:25:28.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:28.391+0000 7f85b77fe700 1 -- 172.21.15.90:0/2132774417 <== mon.2 v1:172.21.15.90:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 557135904 0 0) 0x7f85c01a8f00 con 0x7f85c010b6f0 2024-04-14T15:25:28.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:28.391+0000 7f85b77fe700 1 -- 172.21.15.90:0/2132774417 >> v1:172.21.15.92:6789/0 conn(0x7f85c0108980 legacy=0x7f85c0104f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:28.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:28.391+0000 7f85b77fe700 1 -- 172.21.15.90:0/2132774417 >> v1:172.21.15.90:6789/0 conn(0x7f85c010f2f0 legacy=0x7f85c01a67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:28.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:28.391+0000 7f85b77fe700 1 -- 172.21.15.90:0/2132774417 --> v1:172.21.15.90:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85c01ab2c0 con 0x7f85c010b6f0 2024-04-14T15:25:28.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:28.391+0000 7f85c83e5700 1 -- 172.21.15.90:0/2132774417 --> v1:172.21.15.90:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f85c01a7f50 con 0x7f85c010b6f0 2024-04-14T15:25:28.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:28.391+0000 7f85c83e5700 10 client.47951.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:25:28.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:28.391+0000 7f85c83e5700 1 -- 172.21.15.90:0/2132774417 --> v1:172.21.15.90:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f85c01a8410 con 0x7f85c010b6f0 2024-04-14T15:25:28.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:28.391+0000 7f85c83e5700 1 librados: init done 2024-04-14T15:25:28.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:28.391+0000 7f85b77fe700 1 -- 172.21.15.90:0/2132774417 <== mon.2 v1:172.21.15.90:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f85b0001d20 con 0x7f85c010b6f0 2024-04-14T15:25:28.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:28.391+0000 7f85b77fe700 1 -- 172.21.15.90:0/2132774417 <== mon.2 v1:172.21.15.90:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f85b0005d80 con 0x7f85c010b6f0 2024-04-14T15:25:28.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:28.392+0000 7f85c83e5700 1 -- 172.21.15.90:0/2132774417 --> v1:172.21.15.90:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85a4005120 con 0x7f85c010b6f0 2024-04-14T15:25:28.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:28.392+0000 7f85b77fe700 1 -- 172.21.15.90:0/2132774417 <== mon.2 v1:172.21.15.90:6790/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f85b0003950 con 0x7f85c010b6f0 2024-04-14T15:25:28.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:28.393+0000 7f85b77fe700 1 -- 172.21.15.90:0/2132774417 <== mon.2 v1:172.21.15.90:6790/0 8 ==== osd_map(784..784 src has 467..784) ==== 8894+0+0 (unknown 3754653666 0 0) 0x7f85b008aca0 con 0x7f85c010b6f0 2024-04-14T15:25:28.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:28.393+0000 7f85b77fe700 10 client.47951.objecter ms_dispatch 0x7f85c0001550 osd_map(784..784 src has 467..784) 2024-04-14T15:25:28.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:28.393+0000 7f85b77fe700 3 client.47951.objecter handle_osd_map got epochs [784,784] > 0 2024-04-14T15:25:28.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:28.393+0000 7f85b77fe700 3 client.47951.objecter handle_osd_map decoding full epoch 784 2024-04-14T15:25:28.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:28.393+0000 7f85b77fe700 20 client.47951.objecter dump_active .. 0 homeless 2024-04-14T15:25:28.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:28.393+0000 7f85b77fe700 10 client.47951.objecter ms_handle_connect 0x7f85ac0714b0 2024-04-14T15:25:28.397 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:25:28.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:28.398+0000 7f85b77fe700 1 -- 172.21.15.90:0/2132774417 <== mon.2 v1:172.21.15.90:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f85b00573f0 con 0x7f85c010b6f0 2024-04-14T15:25:28.420 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:28.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:28.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:25:28.522 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:28.545 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:28.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:28.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:28.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:28.616+0000 7f85c83e5700 1 -- 172.21.15.90:0/2132774417 --> v1:172.21.15.90:6790/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7f85a4005410 con 0x7f85c010b6f0 2024-04-14T15:25:28.648 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:25:28.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:28.672 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:28.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:25:28.773 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:25:28.800 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:28.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:28.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:25:28.902 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:25:28.926 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:28.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:28.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:25:29.028 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:25:29.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:29.031+0000 7f85b77fe700 1 -- 172.21.15.90:0/2132774417 <== mon.2 v1:172.21.15.90:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v785) ==== 87+0+0 (unknown 3302241001 0 0) 0x7f85b005b0a0 con 0x7f85c010b6f0 2024-04-14T15:25:29.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:29.034+0000 7f85c83e5700 10 librados: watch_flush enter 2024-04-14T15:25:29.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:29.034+0000 7f85c83e5700 10 librados: watch_flush exit 2024-04-14T15:25:29.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:29.034+0000 7f85c83e5700 20 client.47951.objecter shutdown clearing up homeless session... 2024-04-14T15:25:29.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:29.034+0000 7f85c83e5700 10 client.47951.objecter successfully canceled tick 2024-04-14T15:25:29.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:29.034+0000 7f85c83e5700 1 -- 172.21.15.90:0/2132774417 >> v1:172.21.15.90:6800/420935427 conn(0x7f85ac0714b0 legacy=0x7f85ac073970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:29.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:29.034+0000 7f85c83e5700 1 -- 172.21.15.90:0/2132774417 >> v1:172.21.15.90:6790/0 conn(0x7f85c010b6f0 legacy=0x7f85c0105680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:29.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:29.034+0000 7f85c83e5700 1 -- 172.21.15.90:0/2132774417 shutdown_connections 2024-04-14T15:25:29.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:29.034+0000 7f85c83e5700 1 -- 172.21.15.90:0/2132774417 >> 172.21.15.90:0/2132774417 conn(0x7f85c0076aa0 msgr2=0x7f85c0111790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:25:29.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:29.034+0000 7f85c83e5700 1 -- 172.21.15.90:0/2132774417 shutdown_connections 2024-04-14T15:25:29.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:29.034+0000 7f85c83e5700 1 -- 172.21.15.90:0/2132774417 wait complete. 2024-04-14T15:25:29.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:29.034+0000 7f85c83e5700 1 librados: shutdown 2024-04-14T15:25:29.035 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is set 2024-04-14T15:25:29.053 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:29.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:29.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:25:29.155 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:25:29.181 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:29.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:29.182 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:25:29.199 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:29 smithi090 ceph-mon[30399]: pgmap v1159: 224 pgs: 32 unknown, 4 active+remapped+backfill_toofull, 188 active+clean; 4.2 MiB data, 749 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/648 objects misplaced (0.309%) 2024-04-14T15:25:29.199 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:29 smithi090 ceph-mon[30399]: 6.f scrub starts 2024-04-14T15:25:29.199 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:29 smithi090 ceph-mon[30399]: 6.f scrub ok 2024-04-14T15:25:29.199 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:29 smithi090 ceph-mon[30399]: 4.0 deep-scrub starts 2024-04-14T15:25:29.199 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:29 smithi090 ceph-mon[30399]: 4.0 deep-scrub ok 2024-04-14T15:25:29.199 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:29 smithi090 ceph-mon[30399]: 11.5 deep-scrub starts 2024-04-14T15:25:29.199 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:29 smithi090 ceph-mon[30399]: 11.5 deep-scrub ok 2024-04-14T15:25:29.199 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:29 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:25:29.199 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:29 smithi090 ceph-mon[30399]: osdmap e784: 8 total, 8 up, 7 in 2024-04-14T15:25:29.199 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:29 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:25:29.199 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:29 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:25:29.200 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:29 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2132774417' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:25:29.200 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:29 smithi090 ceph-mon[30399]: from='client.47951 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:25:29.200 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:29 smithi090 ceph-mon[41670]: pgmap v1159: 224 pgs: 32 unknown, 4 active+remapped+backfill_toofull, 188 active+clean; 4.2 MiB data, 749 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/648 objects misplaced (0.309%) 2024-04-14T15:25:29.200 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:29 smithi090 ceph-mon[41670]: 6.f scrub starts 2024-04-14T15:25:29.200 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:29 smithi090 ceph-mon[41670]: 6.f scrub ok 2024-04-14T15:25:29.200 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:29 smithi090 ceph-mon[41670]: 4.0 deep-scrub starts 2024-04-14T15:25:29.200 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:29 smithi090 ceph-mon[41670]: 4.0 deep-scrub ok 2024-04-14T15:25:29.200 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:29 smithi090 ceph-mon[41670]: 11.5 deep-scrub starts 2024-04-14T15:25:29.200 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:29 smithi090 ceph-mon[41670]: 11.5 deep-scrub ok 2024-04-14T15:25:29.200 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:29 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:25:29.200 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:29 smithi090 ceph-mon[41670]: osdmap e784: 8 total, 8 up, 7 in 2024-04-14T15:25:29.200 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:29 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:25:29.200 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:29 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:25:29.200 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:29 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2132774417' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:25:29.201 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:29 smithi090 ceph-mon[41670]: from='client.47951 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:25:29.283 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:25:29.311 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:29.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:29.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:25:29.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:29 smithi092 ceph-mon[33966]: pgmap v1159: 224 pgs: 32 unknown, 4 active+remapped+backfill_toofull, 188 active+clean; 4.2 MiB data, 749 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/648 objects misplaced (0.309%) 2024-04-14T15:25:29.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:29 smithi092 ceph-mon[33966]: 6.f scrub starts 2024-04-14T15:25:29.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:29 smithi092 ceph-mon[33966]: 6.f scrub ok 2024-04-14T15:25:29.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:29 smithi092 ceph-mon[33966]: 4.0 deep-scrub starts 2024-04-14T15:25:29.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:29 smithi092 ceph-mon[33966]: 4.0 deep-scrub ok 2024-04-14T15:25:29.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:29 smithi092 ceph-mon[33966]: 11.5 deep-scrub starts 2024-04-14T15:25:29.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:29 smithi092 ceph-mon[33966]: 11.5 deep-scrub ok 2024-04-14T15:25:29.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:29 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:25:29.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:29 smithi092 ceph-mon[33966]: osdmap e784: 8 total, 8 up, 7 in 2024-04-14T15:25:29.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:29 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:25:29.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:29 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:25:29.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:29 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2132774417' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:25:29.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:29 smithi092 ceph-mon[33966]: from='client.47951 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:25:29.414 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:25:29.438 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:29.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:29.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:25:29.540 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:25:29.568 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:29.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:29.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:25:29.670 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:25:29.694 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:29.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:29.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:25:29.796 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:29.821 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:29.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:29.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:29.923 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:25:29.948 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:29.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:29.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:25:30.050 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:25:30.066 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:30 smithi090 ceph-mon[41670]: 6.19 scrub starts 2024-04-14T15:25:30.066 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:30 smithi090 ceph-mon[41670]: 6.19 scrub ok 2024-04-14T15:25:30.066 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:30 smithi090 ceph-mon[41670]: 323.4s0 scrub starts 2024-04-14T15:25:30.066 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:30 smithi090 ceph-mon[41670]: 323.4 scrub ok 2024-04-14T15:25:30.066 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:30 smithi090 ceph-mon[41670]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:25:30.066 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:30 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-140", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:25:30.066 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:30 smithi090 ceph-mon[41670]: from='client.47951 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-14T15:25:30.066 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:30 smithi090 ceph-mon[41670]: osdmap e785: 8 total, 8 up, 7 in 2024-04-14T15:25:30.067 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:30 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-140","var": "hit_set_count","val": "3"}]: dispatch 2024-04-14T15:25:30.067 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:30 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-140","var": "hit_set_count","val": "3"}]: dispatch 2024-04-14T15:25:30.067 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:30 smithi090 ceph-mon[30399]: 6.19 scrub starts 2024-04-14T15:25:30.067 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:30 smithi090 ceph-mon[30399]: 6.19 scrub ok 2024-04-14T15:25:30.067 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:30 smithi090 ceph-mon[30399]: 323.4s0 scrub starts 2024-04-14T15:25:30.067 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:30 smithi090 ceph-mon[30399]: 323.4 scrub ok 2024-04-14T15:25:30.067 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:30 smithi090 ceph-mon[30399]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:25:30.067 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:30 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-140", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:25:30.067 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:30 smithi090 ceph-mon[30399]: from='client.47951 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-14T15:25:30.068 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:30 smithi090 ceph-mon[30399]: osdmap e785: 8 total, 8 up, 7 in 2024-04-14T15:25:30.068 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:30 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-140","var": "hit_set_count","val": "3"}]: dispatch 2024-04-14T15:25:30.068 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:30 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-140","var": "hit_set_count","val": "3"}]: dispatch 2024-04-14T15:25:30.078 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:30.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:30.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:25:30.180 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:25:30.204 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:30.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:30.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:25:30.306 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:25:30.334 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:30.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:30.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:25:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:30 smithi092 ceph-mon[33966]: 6.19 scrub starts 2024-04-14T15:25:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:30 smithi092 ceph-mon[33966]: 6.19 scrub ok 2024-04-14T15:25:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:30 smithi092 ceph-mon[33966]: 323.4s0 scrub starts 2024-04-14T15:25:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:30 smithi092 ceph-mon[33966]: 323.4 scrub ok 2024-04-14T15:25:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:30 smithi092 ceph-mon[33966]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:25:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:30 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-140", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:25:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:30 smithi092 ceph-mon[33966]: from='client.47951 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-14T15:25:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:30 smithi092 ceph-mon[33966]: osdmap e785: 8 total, 8 up, 7 in 2024-04-14T15:25:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:30 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-140","var": "hit_set_count","val": "3"}]: dispatch 2024-04-14T15:25:30.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:30 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-140","var": "hit_set_count","val": "3"}]: dispatch 2024-04-14T15:25:30.436 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:25:30.460 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:30.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:30.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:25:30.562 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:25:30.586 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:30.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:30.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:25:30.688 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:25:30.712 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:30.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:30.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:25:30.814 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:30.838 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:30.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:30.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:30.939 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-14T15:25:30.940 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:25:30.940 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:25:30.968 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:30.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:30.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:25:31.070 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:25:31.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:31.094 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:31.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:25:31.124 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:31 smithi090 ceph-mon[30399]: pgmap v1162: 224 pgs: 4 active+remapped+backfill_toofull, 220 active+clean; 4.2 MiB data, 749 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/648 objects misplaced (0.309%) 2024-04-14T15:25:31.124 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:31 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-140","var": "hit_set_count","val": "3"}]': finished 2024-04-14T15:25:31.124 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:31 smithi090 ceph-mon[30399]: osdmap e786: 8 total, 8 up, 7 in 2024-04-14T15:25:31.124 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:31 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-140","var": "hit_set_period","val": "3"}]: dispatch 2024-04-14T15:25:31.124 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:31 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-140","var": "hit_set_period","val": "3"}]: dispatch 2024-04-14T15:25:31.124 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:31 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-140","var": "hit_set_period","val": "3"}]': finished 2024-04-14T15:25:31.124 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:31 smithi090 ceph-mon[30399]: osdmap e787: 8 total, 8 up, 7 in 2024-04-14T15:25:31.124 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:31 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-14T15:25:31.124 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:31 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-14T15:25:31.125 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:31 smithi090 ceph-mon[41670]: pgmap v1162: 224 pgs: 4 active+remapped+backfill_toofull, 220 active+clean; 4.2 MiB data, 749 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/648 objects misplaced (0.309%) 2024-04-14T15:25:31.125 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:31 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-140","var": "hit_set_count","val": "3"}]': finished 2024-04-14T15:25:31.125 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:31 smithi090 ceph-mon[41670]: osdmap e786: 8 total, 8 up, 7 in 2024-04-14T15:25:31.125 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:31 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-140","var": "hit_set_period","val": "3"}]: dispatch 2024-04-14T15:25:31.125 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:31 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-140","var": "hit_set_period","val": "3"}]: dispatch 2024-04-14T15:25:31.125 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:31 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-140","var": "hit_set_period","val": "3"}]': finished 2024-04-14T15:25:31.125 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:31 smithi090 ceph-mon[41670]: osdmap e787: 8 total, 8 up, 7 in 2024-04-14T15:25:31.125 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:31 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-14T15:25:31.125 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:31 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-14T15:25:31.196 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:25:31.219 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:31.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:31.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:25:31.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:31.144+0000 7f3f6a17b700 1 -- 172.21.15.90:0/2962756761 >> v1:172.21.15.90:6789/0 conn(0x7f3f64108880 legacy=0x7f3f64108c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:31.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:31.144+0000 7f3f6a17b700 1 -- 172.21.15.90:0/2962756761 shutdown_connections 2024-04-14T15:25:31.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:31.144+0000 7f3f6a17b700 1 -- 172.21.15.90:0/2962756761 >> 172.21.15.90:0/2962756761 conn(0x7f3f640fe3d0 msgr2=0x7f3f641007f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:25:31.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:31.144+0000 7f3f6a17b700 1 -- 172.21.15.90:0/2962756761 shutdown_connections 2024-04-14T15:25:31.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:31.144+0000 7f3f6a17b700 1 -- 172.21.15.90:0/2962756761 wait complete. 2024-04-14T15:25:31.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:31.145+0000 7f3f6a17b700 1 librados: starting msgr at 2024-04-14T15:25:31.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:31.145+0000 7f3f6a17b700 1 librados: starting objecter 2024-04-14T15:25:31.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:31.145+0000 7f3f6a17b700 1 Processor -- start 2024-04-14T15:25:31.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:31.145+0000 7f3f6a17b700 1 -- start start 2024-04-14T15:25:31.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:31.145+0000 7f3f6a17b700 1 librados: setting wanted keys 2024-04-14T15:25:31.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:31.145+0000 7f3f6a17b700 1 librados: calling monclient init 2024-04-14T15:25:31.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:31.145+0000 7f3f6a17b700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3f6410ef60 con 0x7f3f6410b730 2024-04-14T15:25:31.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:31.145+0000 7f3f6a17b700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3f641a8f10 con 0x7f3f6410f300 2024-04-14T15:25:31.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:31.145+0000 7f3f6a17b700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3f641aa0f0 con 0x7f3f64108880 2024-04-14T15:25:31.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:31.146+0000 7f3f62ffd700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f3f6410b730 0x7f3f6410c510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:48558/0 (socket says 172.21.15.90:48558) 2024-04-14T15:25:31.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:31.146+0000 7f3f62ffd700 1 -- 172.21.15.90:0/3011724587 learned_addr learned my addr 172.21.15.90:0/3011724587 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:25:31.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:31.146+0000 7f3f60ff9700 10 client.?.objecter ms_handle_connect 0x7f3f64108880 2024-04-14T15:25:31.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:31.146+0000 7f3f60ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:25:31.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:31.146+0000 7f3f60ff9700 10 client.?.objecter ms_handle_connect 0x7f3f6410b730 2024-04-14T15:25:31.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:31.146+0000 7f3f60ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:25:31.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:31.146+0000 7f3f60ff9700 10 client.?.objecter ms_handle_connect 0x7f3f6410f300 2024-04-14T15:25:31.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:31.146+0000 7f3f60ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:25:31.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:31.146+0000 7f3f60ff9700 1 -- 172.21.15.90:0/3011724587 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2386932442 0 0) 0x7f3f641aa0f0 con 0x7f3f64108880 2024-04-14T15:25:31.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:31.146+0000 7f3f60ff9700 1 -- 172.21.15.90:0/3011724587 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f4c0035b0 con 0x7f3f64108880 2024-04-14T15:25:31.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:31.146+0000 7f3f60ff9700 1 -- 172.21.15.90:0/3011724587 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3881075297 0 0) 0x7f3f6410ef60 con 0x7f3f6410b730 2024-04-14T15:25:31.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:31.146+0000 7f3f60ff9700 1 -- 172.21.15.90:0/3011724587 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f641aa0f0 con 0x7f3f6410b730 2024-04-14T15:25:31.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:31.146+0000 7f3f60ff9700 1 -- 172.21.15.90:0/3011724587 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 911867019 0 0) 0x7f3f641a8f10 con 0x7f3f6410f300 2024-04-14T15:25:31.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:31.146+0000 7f3f60ff9700 1 -- 172.21.15.90:0/3011724587 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f6410ef60 con 0x7f3f6410f300 2024-04-14T15:25:31.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:31.146+0000 7f3f60ff9700 1 -- 172.21.15.90:0/3011724587 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1887241573 0 0) 0x7f3f641aa0f0 con 0x7f3f6410b730 2024-04-14T15:25:31.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:31.146+0000 7f3f60ff9700 1 -- 172.21.15.90:0/3011724587 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3f641a8f10 con 0x7f3f6410b730 2024-04-14T15:25:31.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:31.146+0000 7f3f60ff9700 1 -- 172.21.15.90:0/3011724587 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f3f58001890 con 0x7f3f6410b730 2024-04-14T15:25:31.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:31.146+0000 7f3f60ff9700 1 -- 172.21.15.90:0/3011724587 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2973601620 0 0) 0x7f3f4c0035b0 con 0x7f3f64108880 2024-04-14T15:25:31.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:31.146+0000 7f3f60ff9700 1 -- 172.21.15.90:0/3011724587 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3f641aa0f0 con 0x7f3f64108880 2024-04-14T15:25:31.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:31.146+0000 7f3f60ff9700 1 -- 172.21.15.90:0/3011724587 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f3f54001430 con 0x7f3f64108880 2024-04-14T15:25:31.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:31.146+0000 7f3f60ff9700 1 -- 172.21.15.90:0/3011724587 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1670758686 0 0) 0x7f3f6410ef60 con 0x7f3f6410f300 2024-04-14T15:25:31.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:31.146+0000 7f3f60ff9700 1 -- 172.21.15.90:0/3011724587 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3f4c0035b0 con 0x7f3f6410f300 2024-04-14T15:25:31.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:31.146+0000 7f3f60ff9700 1 -- 172.21.15.90:0/3011724587 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f3f5c001af0 con 0x7f3f6410f300 2024-04-14T15:25:31.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:31.146+0000 7f3f60ff9700 1 -- 172.21.15.90:0/3011724587 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3049423374 0 0) 0x7f3f641a8f10 con 0x7f3f6410b730 2024-04-14T15:25:31.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:31.146+0000 7f3f60ff9700 1 -- 172.21.15.90:0/3011724587 >> v1:172.21.15.90:6790/0 conn(0x7f3f64108880 legacy=0x7f3f6410be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:31.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:31.146+0000 7f3f60ff9700 1 -- 172.21.15.90:0/3011724587 >> v1:172.21.15.92:6789/0 conn(0x7f3f6410f300 legacy=0x7f3f641a67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:31.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:31.146+0000 7f3f60ff9700 1 -- 172.21.15.90:0/3011724587 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f641ab2d0 con 0x7f3f6410b730 2024-04-14T15:25:31.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:31.146+0000 7f3f6a17b700 1 -- 172.21.15.90:0/3011724587 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3f641a7f60 con 0x7f3f6410b730 2024-04-14T15:25:31.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:31.146+0000 7f3f6a17b700 10 client.38007.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:25:31.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:31.146+0000 7f3f6a17b700 1 -- 172.21.15.90:0/3011724587 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f3f641a8420 con 0x7f3f6410b730 2024-04-14T15:25:31.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:31.146+0000 7f3f6a17b700 1 librados: init done 2024-04-14T15:25:31.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:31.146+0000 7f3f60ff9700 1 -- 172.21.15.90:0/3011724587 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f3f58003b20 con 0x7f3f6410b730 2024-04-14T15:25:31.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:31.147+0000 7f3f60ff9700 1 -- 172.21.15.90:0/3011724587 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f3f58005c70 con 0x7f3f6410b730 2024-04-14T15:25:31.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:31.147+0000 7f3f6a17b700 1 -- 172.21.15.90:0/3011724587 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f50005150 con 0x7f3f6410b730 2024-04-14T15:25:31.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:31.148+0000 7f3f60ff9700 1 -- 172.21.15.90:0/3011724587 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f3f58003680 con 0x7f3f6410b730 2024-04-14T15:25:31.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:31.148+0000 7f3f60ff9700 1 -- 172.21.15.90:0/3011724587 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(787..787 src has 467..787) ==== 8790+0+0 (unknown 3954229861 0 0) 0x7f3f5808ad70 con 0x7f3f6410b730 2024-04-14T15:25:31.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:31.148+0000 7f3f60ff9700 10 client.38007.objecter ms_dispatch 0x7f3f64001550 osd_map(787..787 src has 467..787) 2024-04-14T15:25:31.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:31.148+0000 7f3f60ff9700 3 client.38007.objecter handle_osd_map got epochs [787,787] > 0 2024-04-14T15:25:31.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:31.148+0000 7f3f60ff9700 3 client.38007.objecter handle_osd_map decoding full epoch 787 2024-04-14T15:25:31.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:31.149+0000 7f3f60ff9700 20 client.38007.objecter dump_active .. 0 homeless 2024-04-14T15:25:31.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:31.149+0000 7f3f60ff9700 10 client.38007.objecter ms_handle_connect 0x7f3f4c0713a0 2024-04-14T15:25:31.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:31.153+0000 7f3f60ff9700 1 -- 172.21.15.90:0/3011724587 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f3f58057530 con 0x7f3f6410b730 2024-04-14T15:25:31.320 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:31.343 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:31.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:31.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:31.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:31 smithi092 ceph-mon[33966]: pgmap v1162: 224 pgs: 4 active+remapped+backfill_toofull, 220 active+clean; 4.2 MiB data, 749 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/648 objects misplaced (0.309%) 2024-04-14T15:25:31.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:31 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-140","var": "hit_set_count","val": "3"}]': finished 2024-04-14T15:25:31.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:31 smithi092 ceph-mon[33966]: osdmap e786: 8 total, 8 up, 7 in 2024-04-14T15:25:31.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:31 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-140","var": "hit_set_period","val": "3"}]: dispatch 2024-04-14T15:25:31.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:31 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-140","var": "hit_set_period","val": "3"}]: dispatch 2024-04-14T15:25:31.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:31 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-140","var": "hit_set_period","val": "3"}]': finished 2024-04-14T15:25:31.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:31 smithi092 ceph-mon[33966]: osdmap e787: 8 total, 8 up, 7 in 2024-04-14T15:25:31.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:31 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-14T15:25:31.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:31 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-14T15:25:31.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:31.372+0000 7f3f6a17b700 1 -- 172.21.15.90:0/3011724587 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd reweight", "id": 3, "weight": 1.0} v 0) -- 0x7f3f50005440 con 0x7f3f6410b730 2024-04-14T15:25:31.445 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:25:31.473 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:31.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:31.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:25:31.575 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:25:31.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:31.607 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:31.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:25:31.709 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:25:31.736 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:31.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:31.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:25:31.827 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-14T15:25:31.838 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:25:31.866 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:31.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:31.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:25:31.969 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:25:31.996 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:31.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:31.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:25:32.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:32.016+0000 7f3f60ff9700 1 -- 172.21.15.90:0/3011724587 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 3, "weight": 1.0}]=0 reweighted osd.3 to 1 (10000) v788) ==== 113+0+0 (unknown 2166071273 0 0) 0x7f3f5805b1e0 con 0x7f3f6410b730 2024-04-14T15:25:32.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:32.022+0000 7f3f6a17b700 10 librados: watch_flush enter 2024-04-14T15:25:32.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:32.023+0000 7f3f6a17b700 10 librados: watch_flush exit 2024-04-14T15:25:32.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:32.023+0000 7f3f6a17b700 20 client.38007.objecter shutdown clearing up homeless session... 2024-04-14T15:25:32.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:32.023+0000 7f3f6a17b700 10 client.38007.objecter successfully canceled tick 2024-04-14T15:25:32.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:32.023+0000 7f3f6a17b700 1 -- 172.21.15.90:0/3011724587 >> v1:172.21.15.90:6800/420935427 conn(0x7f3f4c0713a0 legacy=0x7f3f4c073860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:32.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:32.023+0000 7f3f6a17b700 1 -- 172.21.15.90:0/3011724587 >> v1:172.21.15.90:6789/0 conn(0x7f3f6410b730 legacy=0x7f3f6410c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:32.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:32.023+0000 7f3f6a17b700 1 -- 172.21.15.90:0/3011724587 shutdown_connections 2024-04-14T15:25:32.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:32.023+0000 7f3f6a17b700 1 -- 172.21.15.90:0/3011724587 >> 172.21.15.90:0/3011724587 conn(0x7f3f640fe3d0 msgr2=0x7f3f64112820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:25:32.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:32.024+0000 7f3f6a17b700 1 -- 172.21.15.90:0/3011724587 shutdown_connections 2024-04-14T15:25:32.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:32.025+0000 7f3f6a17b700 1 -- 172.21.15.90:0/3011724587 wait complete. 2024-04-14T15:25:32.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:32.025+0000 7f3f6a17b700 1 librados: shutdown 2024-04-14T15:25:32.026 INFO:teuthology.orchestra.run.smithi090.stderr:reweighted osd.3 to 1 (10000) 2024-04-14T15:25:32.098 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:25:32.123 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:32.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:32.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:25:32.225 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:25:32.250 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:32.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:32.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:25:32.351 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:32.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:32 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3011724587' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-14T15:25:32.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:32 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-140","var": "hit_set_type","val": "bloom"}]': finished 2024-04-14T15:25:32.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:32 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3011724587' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-14T15:25:32.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:32 smithi092 ceph-mon[33966]: osdmap e788: 8 total, 8 up, 7 in 2024-04-14T15:25:32.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:32 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-04-14T15:25:32.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:32 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-04-14T15:25:32.376 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:32.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:32.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:32.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:32 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3011724587' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-14T15:25:32.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:32 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-140","var": "hit_set_type","val": "bloom"}]': finished 2024-04-14T15:25:32.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:32 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3011724587' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-14T15:25:32.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:32 smithi090 ceph-mon[41670]: osdmap e788: 8 total, 8 up, 7 in 2024-04-14T15:25:32.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:32 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-04-14T15:25:32.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:32 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-04-14T15:25:32.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:32 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3011724587' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-14T15:25:32.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:32 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-140","var": "hit_set_type","val": "bloom"}]': finished 2024-04-14T15:25:32.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:32 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3011724587' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-14T15:25:32.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:32 smithi090 ceph-mon[30399]: osdmap e788: 8 total, 8 up, 7 in 2024-04-14T15:25:32.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:32 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-04-14T15:25:32.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:32 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-04-14T15:25:32.478 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:25:32.503 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:32.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:32.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:25:32.605 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:25:32.629 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:32.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:32.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:25:32.731 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:25:32.756 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:32.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:32.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:25:32.830 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-14T15:25:32.858 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:32.881 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:32.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:32.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:32.982 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:25:33.007 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:33.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:33.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:25:33.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.045+0000 7fed75aa0700 1 -- 172.21.15.90:0/3946680489 >> v1:172.21.15.90:6789/0 conn(0x7fed70107760 legacy=0x7fed70107b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:33.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.045+0000 7fed75aa0700 1 -- 172.21.15.90:0/3946680489 shutdown_connections 2024-04-14T15:25:33.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.045+0000 7fed75aa0700 1 -- 172.21.15.90:0/3946680489 >> 172.21.15.90:0/3946680489 conn(0x7fed700fd290 msgr2=0x7fed700ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:25:33.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.045+0000 7fed75aa0700 1 -- 172.21.15.90:0/3946680489 shutdown_connections 2024-04-14T15:25:33.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.045+0000 7fed75aa0700 1 -- 172.21.15.90:0/3946680489 wait complete. 2024-04-14T15:25:33.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.046+0000 7fed75aa0700 1 librados: starting msgr at 2024-04-14T15:25:33.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.046+0000 7fed75aa0700 1 librados: starting objecter 2024-04-14T15:25:33.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.046+0000 7fed75aa0700 1 Processor -- start 2024-04-14T15:25:33.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.046+0000 7fed75aa0700 1 -- start start 2024-04-14T15:25:33.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.046+0000 7fed75aa0700 1 librados: setting wanted keys 2024-04-14T15:25:33.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.046+0000 7fed75aa0700 1 librados: calling monclient init 2024-04-14T15:25:33.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.046+0000 7fed75aa0700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fed701a7df0 con 0x7fed7010e1e0 2024-04-14T15:25:33.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.046+0000 7fed75aa0700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fed701a8fd0 con 0x7fed70107760 2024-04-14T15:25:33.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.046+0000 7fed75aa0700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fed701aa1b0 con 0x7fed7010a610 2024-04-14T15:25:33.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.046+0000 7fed6ffff700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fed7010e1e0 0x7fed701a66d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:48572/0 (socket says 172.21.15.90:48572) 2024-04-14T15:25:33.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.046+0000 7fed6ffff700 1 -- 172.21.15.90:0/434777481 learned_addr learned my addr 172.21.15.90:0/434777481 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:25:33.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.046+0000 7fed6effd700 1 --1- 172.21.15.90:0/434777481 >> v1:172.21.15.90:6790/0 conn(0x7fed7010a610 0x7fed7006ab70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:35950/0 (socket says 172.21.15.90:35950) 2024-04-14T15:25:33.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.047+0000 7fed6cff9700 10 client.?.objecter ms_handle_connect 0x7fed7010e1e0 2024-04-14T15:25:33.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.047+0000 7fed6cff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:25:33.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.047+0000 7fed6cff9700 10 client.?.objecter ms_handle_connect 0x7fed70107760 2024-04-14T15:25:33.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.047+0000 7fed6cff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:25:33.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.047+0000 7fed6cff9700 10 client.?.objecter ms_handle_connect 0x7fed7010a610 2024-04-14T15:25:33.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.047+0000 7fed6cff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:25:33.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.047+0000 7fed6cff9700 1 -- 172.21.15.90:0/434777481 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2980555397 0 0) 0x7fed701a8fd0 con 0x7fed70107760 2024-04-14T15:25:33.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.047+0000 7fed6cff9700 1 -- 172.21.15.90:0/434777481 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fed580035b0 con 0x7fed70107760 2024-04-14T15:25:33.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.047+0000 7fed6cff9700 1 -- 172.21.15.90:0/434777481 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3488827718 0 0) 0x7fed701a7df0 con 0x7fed7010e1e0 2024-04-14T15:25:33.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.047+0000 7fed6cff9700 1 -- 172.21.15.90:0/434777481 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fed701a8fd0 con 0x7fed7010e1e0 2024-04-14T15:25:33.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.047+0000 7fed6cff9700 1 -- 172.21.15.90:0/434777481 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1005035831 0 0) 0x7fed701aa1b0 con 0x7fed7010a610 2024-04-14T15:25:33.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.047+0000 7fed6cff9700 1 -- 172.21.15.90:0/434777481 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fed701a7df0 con 0x7fed7010a610 2024-04-14T15:25:33.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.047+0000 7fed6cff9700 1 -- 172.21.15.90:0/434777481 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3684602912 0 0) 0x7fed580035b0 con 0x7fed70107760 2024-04-14T15:25:33.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.047+0000 7fed6cff9700 1 -- 172.21.15.90:0/434777481 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fed701aa1b0 con 0x7fed70107760 2024-04-14T15:25:33.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.047+0000 7fed6cff9700 1 -- 172.21.15.90:0/434777481 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fed600015d0 con 0x7fed70107760 2024-04-14T15:25:33.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.047+0000 7fed6cff9700 1 -- 172.21.15.90:0/434777481 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3371241220 0 0) 0x7fed701a8fd0 con 0x7fed7010e1e0 2024-04-14T15:25:33.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.047+0000 7fed6cff9700 1 -- 172.21.15.90:0/434777481 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fed580035b0 con 0x7fed7010e1e0 2024-04-14T15:25:33.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.047+0000 7fed6cff9700 1 -- 172.21.15.90:0/434777481 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fed68001bb0 con 0x7fed7010e1e0 2024-04-14T15:25:33.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.047+0000 7fed6cff9700 1 -- 172.21.15.90:0/434777481 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 154379975 0 0) 0x7fed701aa1b0 con 0x7fed70107760 2024-04-14T15:25:33.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.047+0000 7fed6cff9700 1 -- 172.21.15.90:0/434777481 >> v1:172.21.15.90:6790/0 conn(0x7fed7010a610 legacy=0x7fed7006ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:33.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.047+0000 7fed6cff9700 1 -- 172.21.15.90:0/434777481 >> v1:172.21.15.90:6789/0 conn(0x7fed7010e1e0 legacy=0x7fed701a66d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:33.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.047+0000 7fed6cff9700 1 -- 172.21.15.90:0/434777481 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed701ab390 con 0x7fed70107760 2024-04-14T15:25:33.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.047+0000 7fed75aa0700 1 -- 172.21.15.90:0/434777481 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fed701a9200 con 0x7fed70107760 2024-04-14T15:25:33.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.047+0000 7fed75aa0700 10 client.49843.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:25:33.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.047+0000 7fed75aa0700 1 -- 172.21.15.90:0/434777481 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fed701a96c0 con 0x7fed70107760 2024-04-14T15:25:33.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.047+0000 7fed75aa0700 1 librados: init done 2024-04-14T15:25:33.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.047+0000 7fed6cff9700 1 -- 172.21.15.90:0/434777481 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fed60003760 con 0x7fed70107760 2024-04-14T15:25:33.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.047+0000 7fed6cff9700 1 -- 172.21.15.90:0/434777481 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fed60004d60 con 0x7fed70107760 2024-04-14T15:25:33.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.048+0000 7fed75aa0700 1 -- 172.21.15.90:0/434777481 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fed5c005130 con 0x7fed70107760 2024-04-14T15:25:33.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.049+0000 7fed6cff9700 1 -- 172.21.15.90:0/434777481 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fed6001b990 con 0x7fed70107760 2024-04-14T15:25:33.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.049+0000 7fed6cff9700 1 -- 172.21.15.90:0/434777481 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(789..789 src has 467..789) ==== 8816+0+0 (unknown 760811526 0 0) 0x7fed60089ba0 con 0x7fed70107760 2024-04-14T15:25:33.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.049+0000 7fed6cff9700 10 client.49843.objecter ms_dispatch 0x7fed70001550 osd_map(789..789 src has 467..789) 2024-04-14T15:25:33.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.049+0000 7fed6cff9700 3 client.49843.objecter handle_osd_map got epochs [789,789] > 0 2024-04-14T15:25:33.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.049+0000 7fed6cff9700 3 client.49843.objecter handle_osd_map decoding full epoch 789 2024-04-14T15:25:33.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.049+0000 7fed6cff9700 20 client.49843.objecter dump_active .. 0 homeless 2024-04-14T15:25:33.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.049+0000 7fed6cff9700 1 -- 172.21.15.90:0/434777481 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v789) ==== 54+0+16 (unknown 2827435276 0 2420000192) 0x7fed60089f10 con 0x7fed70107760 2024-04-14T15:25:33.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.049+0000 7fed6cff9700 10 client.49843.objecter ms_handle_connect 0x7fed58071080 2024-04-14T15:25:33.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.050+0000 7fed75aa0700 10 client.49843.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:25:33.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.050+0000 7fed75aa0700 15 client.49843.objecter _session_command_op_assign -1 1 2024-04-14T15:25:33.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.050+0000 7fed75aa0700 20 client.49843.objecter _get_session s=0x7fed5c005410 osd=0 3 2024-04-14T15:25:33.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.050+0000 7fed75aa0700 20 client.49843.objecter put_session s=0x7fed5c005410 osd=0 3 2024-04-14T15:25:33.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.050+0000 7fed75aa0700 20 client.49843.objecter _get_session s=0x7fed5c005410 osd=0 3 2024-04-14T15:25:33.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.050+0000 7fed75aa0700 15 client.49843.objecter _session_command_op_remove -1 1 2024-04-14T15:25:33.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.050+0000 7fed75aa0700 20 client.49843.objecter get_session s=0x7fed5c005410 osd=0 3 2024-04-14T15:25:33.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.050+0000 7fed75aa0700 15 client.49843.objecter _session_command_op_assign 0 1 2024-04-14T15:25:33.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.050+0000 7fed75aa0700 20 client.49843.objecter put_session s=0x7fed5c005410 osd=0 4 2024-04-14T15:25:33.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.050+0000 7fed75aa0700 10 client.49843.objecter _send_command 1 2024-04-14T15:25:33.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.050+0000 7fed75aa0700 1 -- 172.21.15.90:0/434777481 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fed5c004b80 con 0x7fed5c004760 2024-04-14T15:25:33.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.050+0000 7fed6cff9700 10 client.49843.objecter ms_handle_connect 0x7fed5c004760 2024-04-14T15:25:33.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.051+0000 7fed6cff9700 1 -- 172.21.15.90:0/434777481 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fed5c004b80 con 0x7fed5c004760 2024-04-14T15:25:33.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.051+0000 7fed6cff9700 10 client.49843.objecter ms_dispatch 0x7fed70001550 command_reply(tid 1: 0 ) 2024-04-14T15:25:33.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.051+0000 7fed6cff9700 10 client.49843.objecter _finish_command 1 = system:0 2024-04-14T15:25:33.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.051+0000 7fed6cff9700 20 client.49843.objecter put_session s=0x7fed5c005410 osd=0 4 2024-04-14T15:25:33.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.051+0000 7fed6cff9700 15 client.49843.objecter _session_command_op_remove 0 1 2024-04-14T15:25:33.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.067+0000 7fed75aa0700 10 client.49843.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:25:33.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.067+0000 7fed75aa0700 15 client.49843.objecter _session_command_op_assign -1 2 2024-04-14T15:25:33.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.067+0000 7fed75aa0700 20 client.49843.objecter _get_session s=0x7fed5c005410 osd=0 3 2024-04-14T15:25:33.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.067+0000 7fed75aa0700 20 client.49843.objecter put_session s=0x7fed5c005410 osd=0 3 2024-04-14T15:25:33.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.067+0000 7fed75aa0700 20 client.49843.objecter _get_session s=0x7fed5c005410 osd=0 3 2024-04-14T15:25:33.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.067+0000 7fed75aa0700 15 client.49843.objecter _session_command_op_remove -1 2 2024-04-14T15:25:33.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.067+0000 7fed75aa0700 20 client.49843.objecter get_session s=0x7fed5c005410 osd=0 3 2024-04-14T15:25:33.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.067+0000 7fed75aa0700 15 client.49843.objecter _session_command_op_assign 0 2 2024-04-14T15:25:33.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.067+0000 7fed75aa0700 20 client.49843.objecter put_session s=0x7fed5c005410 osd=0 4 2024-04-14T15:25:33.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.067+0000 7fed75aa0700 10 client.49843.objecter _send_command 2 2024-04-14T15:25:33.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.067+0000 7fed75aa0700 1 -- 172.21.15.90:0/434777481 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fed5c007b30 con 0x7fed5c004760 2024-04-14T15:25:33.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.069+0000 7fed6cff9700 1 -- 172.21.15.90:0/434777481 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fed5c007b30 con 0x7fed5c004760 2024-04-14T15:25:33.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.069+0000 7fed6cff9700 10 client.49843.objecter ms_dispatch 0x7fed70001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:25:33.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.069+0000 7fed6cff9700 10 client.49843.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:25:33.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.069+0000 7fed6cff9700 20 client.49843.objecter put_session s=0x7fed5c005410 osd=0 4 2024-04-14T15:25:33.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.069+0000 7fed6cff9700 15 client.49843.objecter _session_command_op_remove 0 2 2024-04-14T15:25:33.070 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:25:33.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.069+0000 7fed75aa0700 10 client.49843.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:25:33.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.069+0000 7fed75aa0700 15 client.49843.objecter _session_command_op_assign -1 3 2024-04-14T15:25:33.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.069+0000 7fed75aa0700 20 client.49843.objecter _get_session s=0x7fed5c004160 osd=1 3 2024-04-14T15:25:33.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.069+0000 7fed75aa0700 20 client.49843.objecter put_session s=0x7fed5c004160 osd=1 3 2024-04-14T15:25:33.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.069+0000 7fed75aa0700 20 client.49843.objecter _get_session s=0x7fed5c004160 osd=1 3 2024-04-14T15:25:33.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.069+0000 7fed75aa0700 15 client.49843.objecter _session_command_op_remove -1 3 2024-04-14T15:25:33.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.069+0000 7fed75aa0700 20 client.49843.objecter get_session s=0x7fed5c004160 osd=1 3 2024-04-14T15:25:33.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.069+0000 7fed75aa0700 15 client.49843.objecter _session_command_op_assign 1 3 2024-04-14T15:25:33.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.069+0000 7fed75aa0700 20 client.49843.objecter put_session s=0x7fed5c004160 osd=1 4 2024-04-14T15:25:33.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.069+0000 7fed75aa0700 10 client.49843.objecter _send_command 3 2024-04-14T15:25:33.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.069+0000 7fed75aa0700 1 -- 172.21.15.90:0/434777481 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fed5c002190 con 0x7fed5c0042d0 2024-04-14T15:25:33.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.069+0000 7fed6cff9700 10 client.49843.objecter ms_handle_connect 0x7fed5c0042d0 2024-04-14T15:25:33.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.070+0000 7fed6cff9700 1 -- 172.21.15.90:0/434777481 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fed5c002190 con 0x7fed5c0042d0 2024-04-14T15:25:33.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.070+0000 7fed6cff9700 10 client.49843.objecter ms_dispatch 0x7fed70001550 command_reply(tid 3: 0 ) 2024-04-14T15:25:33.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.070+0000 7fed6cff9700 10 client.49843.objecter _finish_command 3 = system:0 2024-04-14T15:25:33.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.070+0000 7fed6cff9700 20 client.49843.objecter put_session s=0x7fed5c004160 osd=1 4 2024-04-14T15:25:33.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.070+0000 7fed6cff9700 15 client.49843.objecter _session_command_op_remove 1 3 2024-04-14T15:25:33.082 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:25:33.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.086+0000 7fed75aa0700 10 client.49843.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:25:33.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.086+0000 7fed75aa0700 15 client.49843.objecter _session_command_op_assign -1 4 2024-04-14T15:25:33.089 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:33 smithi090 ceph-mon[41670]: pgmap v1165: 224 pgs: 4 active+remapped+backfill_toofull, 220 active+clean; 4.2 MiB data, 749 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/648 objects misplaced (0.309%) 2024-04-14T15:25:33.089 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:33 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-140","var": "hit_set_fpp","val": ".01"}]': finished 2024-04-14T15:25:33.089 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:33 smithi090 ceph-mon[41670]: osdmap e789: 8 total, 8 up, 7 in 2024-04-14T15:25:33.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.086+0000 7fed75aa0700 20 client.49843.objecter _get_session s=0x7fed5c004160 osd=1 3 2024-04-14T15:25:33.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.086+0000 7fed75aa0700 20 client.49843.objecter put_session s=0x7fed5c004160 osd=1 3 2024-04-14T15:25:33.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.086+0000 7fed75aa0700 20 client.49843.objecter _get_session s=0x7fed5c004160 osd=1 3 2024-04-14T15:25:33.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.086+0000 7fed75aa0700 15 client.49843.objecter _session_command_op_remove -1 4 2024-04-14T15:25:33.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.086+0000 7fed75aa0700 20 client.49843.objecter get_session s=0x7fed5c004160 osd=1 3 2024-04-14T15:25:33.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.086+0000 7fed75aa0700 15 client.49843.objecter _session_command_op_assign 1 4 2024-04-14T15:25:33.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.086+0000 7fed75aa0700 20 client.49843.objecter put_session s=0x7fed5c004160 osd=1 4 2024-04-14T15:25:33.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.086+0000 7fed75aa0700 10 client.49843.objecter _send_command 4 2024-04-14T15:25:33.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.086+0000 7fed75aa0700 1 -- 172.21.15.90:0/434777481 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fed5c001630 con 0x7fed5c0042d0 2024-04-14T15:25:33.091 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:33 smithi090 ceph-mon[30399]: pgmap v1165: 224 pgs: 4 active+remapped+backfill_toofull, 220 active+clean; 4.2 MiB data, 749 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/648 objects misplaced (0.309%) 2024-04-14T15:25:33.091 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:33 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-140","var": "hit_set_fpp","val": ".01"}]': finished 2024-04-14T15:25:33.091 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:33 smithi090 ceph-mon[30399]: osdmap e789: 8 total, 8 up, 7 in 2024-04-14T15:25:33.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.089+0000 7fed6cff9700 1 -- 172.21.15.90:0/434777481 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fed5c001630 con 0x7fed5c0042d0 2024-04-14T15:25:33.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.089+0000 7fed6cff9700 10 client.49843.objecter ms_dispatch 0x7fed70001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:25:33.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.089+0000 7fed6cff9700 10 client.49843.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:25:33.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.089+0000 7fed6cff9700 20 client.49843.objecter put_session s=0x7fed5c004160 osd=1 4 2024-04-14T15:25:33.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.089+0000 7fed6cff9700 15 client.49843.objecter _session_command_op_remove 1 4 2024-04-14T15:25:33.092 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:25:33.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.089+0000 7fed75aa0700 10 client.49843.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:25:33.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.089+0000 7fed75aa0700 15 client.49843.objecter _session_command_op_assign -1 5 2024-04-14T15:25:33.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.089+0000 7fed75aa0700 20 client.49843.objecter _get_session s=0x7fed5c001a30 osd=2 3 2024-04-14T15:25:33.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.089+0000 7fed75aa0700 20 client.49843.objecter put_session s=0x7fed5c001a30 osd=2 3 2024-04-14T15:25:33.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.089+0000 7fed75aa0700 20 client.49843.objecter _get_session s=0x7fed5c001a30 osd=2 3 2024-04-14T15:25:33.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.089+0000 7fed75aa0700 15 client.49843.objecter _session_command_op_remove -1 5 2024-04-14T15:25:33.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.089+0000 7fed75aa0700 20 client.49843.objecter get_session s=0x7fed5c001a30 osd=2 3 2024-04-14T15:25:33.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.089+0000 7fed75aa0700 15 client.49843.objecter _session_command_op_assign 2 5 2024-04-14T15:25:33.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.089+0000 7fed75aa0700 20 client.49843.objecter put_session s=0x7fed5c001a30 osd=2 4 2024-04-14T15:25:33.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.089+0000 7fed75aa0700 10 client.49843.objecter _send_command 5 2024-04-14T15:25:33.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.089+0000 7fed75aa0700 1 -- 172.21.15.90:0/434777481 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fed5c010fe0 con 0x7fed5c0089c0 2024-04-14T15:25:33.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.090+0000 7fed6cff9700 10 client.49843.objecter ms_handle_connect 0x7fed5c0089c0 2024-04-14T15:25:33.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.091+0000 7fed6cff9700 1 -- 172.21.15.90:0/434777481 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fed5c010fe0 con 0x7fed5c0089c0 2024-04-14T15:25:33.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.091+0000 7fed6cff9700 10 client.49843.objecter ms_dispatch 0x7fed70001550 command_reply(tid 5: 0 ) 2024-04-14T15:25:33.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.091+0000 7fed6cff9700 10 client.49843.objecter _finish_command 5 = system:0 2024-04-14T15:25:33.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.091+0000 7fed6cff9700 20 client.49843.objecter put_session s=0x7fed5c001a30 osd=2 4 2024-04-14T15:25:33.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.091+0000 7fed6cff9700 15 client.49843.objecter _session_command_op_remove 2 5 2024-04-14T15:25:33.109 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:25:33.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.109+0000 7fed75aa0700 10 client.49843.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:25:33.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.109+0000 7fed75aa0700 15 client.49843.objecter _session_command_op_assign -1 6 2024-04-14T15:25:33.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.109+0000 7fed75aa0700 20 client.49843.objecter _get_session s=0x7fed5c001a30 osd=2 3 2024-04-14T15:25:33.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.109+0000 7fed75aa0700 20 client.49843.objecter put_session s=0x7fed5c001a30 osd=2 3 2024-04-14T15:25:33.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.109+0000 7fed75aa0700 20 client.49843.objecter _get_session s=0x7fed5c001a30 osd=2 3 2024-04-14T15:25:33.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.109+0000 7fed75aa0700 15 client.49843.objecter _session_command_op_remove -1 6 2024-04-14T15:25:33.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.109+0000 7fed75aa0700 20 client.49843.objecter get_session s=0x7fed5c001a30 osd=2 3 2024-04-14T15:25:33.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.109+0000 7fed75aa0700 15 client.49843.objecter _session_command_op_assign 2 6 2024-04-14T15:25:33.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.109+0000 7fed75aa0700 20 client.49843.objecter put_session s=0x7fed5c001a30 osd=2 4 2024-04-14T15:25:33.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.109+0000 7fed75aa0700 10 client.49843.objecter _send_command 6 2024-04-14T15:25:33.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.109+0000 7fed75aa0700 1 -- 172.21.15.90:0/434777481 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fed5c008050 con 0x7fed5c0089c0 2024-04-14T15:25:33.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.111+0000 7fed6cff9700 1 -- 172.21.15.90:0/434777481 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fed5c008050 con 0x7fed5c0089c0 2024-04-14T15:25:33.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.111+0000 7fed6cff9700 10 client.49843.objecter ms_dispatch 0x7fed70001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:25:33.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.111+0000 7fed6cff9700 10 client.49843.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:25:33.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.111+0000 7fed6cff9700 20 client.49843.objecter put_session s=0x7fed5c001a30 osd=2 4 2024-04-14T15:25:33.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.111+0000 7fed6cff9700 15 client.49843.objecter _session_command_op_remove 2 6 2024-04-14T15:25:33.114 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:25:33.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.112+0000 7fed75aa0700 10 client.49843.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:25:33.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.112+0000 7fed75aa0700 15 client.49843.objecter _session_command_op_assign -1 7 2024-04-14T15:25:33.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.112+0000 7fed75aa0700 20 client.49843.objecter _get_session s=0x7fed5c008320 osd=3 3 2024-04-14T15:25:33.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.112+0000 7fed75aa0700 20 client.49843.objecter put_session s=0x7fed5c008320 osd=3 3 2024-04-14T15:25:33.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.112+0000 7fed75aa0700 20 client.49843.objecter _get_session s=0x7fed5c008320 osd=3 3 2024-04-14T15:25:33.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.112+0000 7fed75aa0700 15 client.49843.objecter _session_command_op_remove -1 7 2024-04-14T15:25:33.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.112+0000 7fed75aa0700 20 client.49843.objecter get_session s=0x7fed5c008320 osd=3 3 2024-04-14T15:25:33.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.112+0000 7fed75aa0700 15 client.49843.objecter _session_command_op_assign 3 7 2024-04-14T15:25:33.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.112+0000 7fed75aa0700 20 client.49843.objecter put_session s=0x7fed5c008320 osd=3 4 2024-04-14T15:25:33.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.112+0000 7fed75aa0700 10 client.49843.objecter _send_command 7 2024-04-14T15:25:33.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.112+0000 7fed75aa0700 1 -- 172.21.15.90:0/434777481 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fed5c011d10 con 0x7fed5c0111c0 2024-04-14T15:25:33.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.112+0000 7fed6cff9700 10 client.49843.objecter ms_handle_connect 0x7fed5c0111c0 2024-04-14T15:25:33.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.113+0000 7fed6cff9700 1 -- 172.21.15.90:0/434777481 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fed5c011d10 con 0x7fed5c0111c0 2024-04-14T15:25:33.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.113+0000 7fed6cff9700 10 client.49843.objecter ms_dispatch 0x7fed70001550 command_reply(tid 7: 0 ) 2024-04-14T15:25:33.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.113+0000 7fed6cff9700 10 client.49843.objecter _finish_command 7 = system:0 2024-04-14T15:25:33.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.113+0000 7fed6cff9700 20 client.49843.objecter put_session s=0x7fed5c008320 osd=3 4 2024-04-14T15:25:33.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.113+0000 7fed6cff9700 15 client.49843.objecter _session_command_op_remove 3 7 2024-04-14T15:25:33.121 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:33 smithi092 ceph-mon[33966]: pgmap v1165: 224 pgs: 4 active+remapped+backfill_toofull, 220 active+clean; 4.2 MiB data, 749 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/648 objects misplaced (0.309%) 2024-04-14T15:25:33.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:33 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-140","var": "hit_set_fpp","val": ".01"}]': finished 2024-04-14T15:25:33.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:33 smithi092 ceph-mon[33966]: osdmap e789: 8 total, 8 up, 7 in 2024-04-14T15:25:33.132 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:33.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:33.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:25:33.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.141+0000 7fed75aa0700 10 client.49843.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:25:33.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.141+0000 7fed75aa0700 15 client.49843.objecter _session_command_op_assign -1 8 2024-04-14T15:25:33.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.141+0000 7fed75aa0700 20 client.49843.objecter _get_session s=0x7fed5c008320 osd=3 3 2024-04-14T15:25:33.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.141+0000 7fed75aa0700 20 client.49843.objecter put_session s=0x7fed5c008320 osd=3 3 2024-04-14T15:25:33.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.141+0000 7fed75aa0700 20 client.49843.objecter _get_session s=0x7fed5c008320 osd=3 3 2024-04-14T15:25:33.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.141+0000 7fed75aa0700 15 client.49843.objecter _session_command_op_remove -1 8 2024-04-14T15:25:33.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.141+0000 7fed75aa0700 20 client.49843.objecter get_session s=0x7fed5c008320 osd=3 3 2024-04-14T15:25:33.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.141+0000 7fed75aa0700 15 client.49843.objecter _session_command_op_assign 3 8 2024-04-14T15:25:33.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.141+0000 7fed75aa0700 20 client.49843.objecter put_session s=0x7fed5c008320 osd=3 4 2024-04-14T15:25:33.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.141+0000 7fed75aa0700 10 client.49843.objecter _send_command 8 2024-04-14T15:25:33.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.141+0000 7fed75aa0700 1 -- 172.21.15.90:0/434777481 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fed5c0126c0 con 0x7fed5c0111c0 2024-04-14T15:25:33.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.142+0000 7fed6cff9700 1 -- 172.21.15.90:0/434777481 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fed5c0126c0 con 0x7fed5c0111c0 2024-04-14T15:25:33.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.142+0000 7fed6cff9700 10 client.49843.objecter ms_dispatch 0x7fed70001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:25:33.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.142+0000 7fed6cff9700 10 client.49843.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:25:33.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.142+0000 7fed6cff9700 20 client.49843.objecter put_session s=0x7fed5c008320 osd=3 4 2024-04-14T15:25:33.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.142+0000 7fed6cff9700 15 client.49843.objecter _session_command_op_remove 3 8 2024-04-14T15:25:33.144 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:25:33.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.143+0000 7fed75aa0700 10 client.49843.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:25:33.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.143+0000 7fed75aa0700 15 client.49843.objecter _session_command_op_assign -1 9 2024-04-14T15:25:33.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.143+0000 7fed75aa0700 20 client.49843.objecter _get_session s=0x7fed5c013840 osd=4 3 2024-04-14T15:25:33.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.143+0000 7fed75aa0700 20 client.49843.objecter put_session s=0x7fed5c013840 osd=4 3 2024-04-14T15:25:33.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.143+0000 7fed75aa0700 20 client.49843.objecter _get_session s=0x7fed5c013840 osd=4 3 2024-04-14T15:25:33.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.143+0000 7fed75aa0700 15 client.49843.objecter _session_command_op_remove -1 9 2024-04-14T15:25:33.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.143+0000 7fed75aa0700 20 client.49843.objecter get_session s=0x7fed5c013840 osd=4 3 2024-04-14T15:25:33.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.143+0000 7fed75aa0700 15 client.49843.objecter _session_command_op_assign 4 9 2024-04-14T15:25:33.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.143+0000 7fed75aa0700 20 client.49843.objecter put_session s=0x7fed5c013840 osd=4 4 2024-04-14T15:25:33.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.143+0000 7fed75aa0700 10 client.49843.objecter _send_command 9 2024-04-14T15:25:33.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.143+0000 7fed75aa0700 1 -- 172.21.15.90:0/434777481 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fed5c01eac0 con 0x7fed5c01afb0 2024-04-14T15:25:33.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.143+0000 7fed6cff9700 10 client.49843.objecter ms_handle_connect 0x7fed5c01afb0 2024-04-14T15:25:33.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.144+0000 7fed6cff9700 1 -- 172.21.15.90:0/434777481 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fed5c01eac0 con 0x7fed5c01afb0 2024-04-14T15:25:33.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.144+0000 7fed6cff9700 10 client.49843.objecter ms_dispatch 0x7fed70001550 command_reply(tid 9: 0 ) 2024-04-14T15:25:33.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.144+0000 7fed6cff9700 10 client.49843.objecter _finish_command 9 = system:0 2024-04-14T15:25:33.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.144+0000 7fed6cff9700 20 client.49843.objecter put_session s=0x7fed5c013840 osd=4 4 2024-04-14T15:25:33.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.144+0000 7fed6cff9700 15 client.49843.objecter _session_command_op_remove 4 9 2024-04-14T15:25:33.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.160+0000 7fed75aa0700 10 client.49843.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:25:33.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.160+0000 7fed75aa0700 15 client.49843.objecter _session_command_op_assign -1 10 2024-04-14T15:25:33.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.160+0000 7fed75aa0700 20 client.49843.objecter _get_session s=0x7fed5c013840 osd=4 3 2024-04-14T15:25:33.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.160+0000 7fed75aa0700 20 client.49843.objecter put_session s=0x7fed5c013840 osd=4 3 2024-04-14T15:25:33.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.160+0000 7fed75aa0700 20 client.49843.objecter _get_session s=0x7fed5c013840 osd=4 3 2024-04-14T15:25:33.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.160+0000 7fed75aa0700 15 client.49843.objecter _session_command_op_remove -1 10 2024-04-14T15:25:33.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.160+0000 7fed75aa0700 20 client.49843.objecter get_session s=0x7fed5c013840 osd=4 3 2024-04-14T15:25:33.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.160+0000 7fed75aa0700 15 client.49843.objecter _session_command_op_assign 4 10 2024-04-14T15:25:33.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.160+0000 7fed75aa0700 20 client.49843.objecter put_session s=0x7fed5c013840 osd=4 4 2024-04-14T15:25:33.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.160+0000 7fed75aa0700 10 client.49843.objecter _send_command 10 2024-04-14T15:25:33.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.160+0000 7fed75aa0700 1 -- 172.21.15.90:0/434777481 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fed5c00c840 con 0x7fed5c01afb0 2024-04-14T15:25:33.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.162+0000 7fed6cff9700 1 -- 172.21.15.90:0/434777481 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fed5c00c840 con 0x7fed5c01afb0 2024-04-14T15:25:33.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.162+0000 7fed6cff9700 10 client.49843.objecter ms_dispatch 0x7fed70001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:25:33.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.162+0000 7fed6cff9700 10 client.49843.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:25:33.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.162+0000 7fed6cff9700 20 client.49843.objecter put_session s=0x7fed5c013840 osd=4 4 2024-04-14T15:25:33.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.162+0000 7fed6cff9700 15 client.49843.objecter _session_command_op_remove 4 10 2024-04-14T15:25:33.163 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:25:33.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.162+0000 7fed75aa0700 10 client.49843.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:25:33.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.162+0000 7fed75aa0700 15 client.49843.objecter _session_command_op_assign -1 11 2024-04-14T15:25:33.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.162+0000 7fed75aa0700 20 client.49843.objecter _get_session s=0x7fed5c00c840 osd=5 3 2024-04-14T15:25:33.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.162+0000 7fed75aa0700 20 client.49843.objecter put_session s=0x7fed5c00c840 osd=5 3 2024-04-14T15:25:33.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.162+0000 7fed75aa0700 20 client.49843.objecter _get_session s=0x7fed5c00c840 osd=5 3 2024-04-14T15:25:33.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.162+0000 7fed75aa0700 15 client.49843.objecter _session_command_op_remove -1 11 2024-04-14T15:25:33.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.162+0000 7fed75aa0700 20 client.49843.objecter get_session s=0x7fed5c00c840 osd=5 3 2024-04-14T15:25:33.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.162+0000 7fed75aa0700 15 client.49843.objecter _session_command_op_assign 5 11 2024-04-14T15:25:33.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.162+0000 7fed75aa0700 20 client.49843.objecter put_session s=0x7fed5c00c840 osd=5 4 2024-04-14T15:25:33.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.162+0000 7fed75aa0700 10 client.49843.objecter _send_command 11 2024-04-14T15:25:33.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.162+0000 7fed75aa0700 1 -- 172.21.15.90:0/434777481 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fed5c021390 con 0x7fed5c012e80 2024-04-14T15:25:33.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.162+0000 7fed6cff9700 10 client.49843.objecter ms_handle_connect 0x7fed5c012e80 2024-04-14T15:25:33.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.163+0000 7fed6cff9700 1 -- 172.21.15.90:0/434777481 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fed5c021390 con 0x7fed5c012e80 2024-04-14T15:25:33.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.163+0000 7fed6cff9700 10 client.49843.objecter ms_dispatch 0x7fed70001550 command_reply(tid 11: 0 ) 2024-04-14T15:25:33.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.163+0000 7fed6cff9700 10 client.49843.objecter _finish_command 11 = system:0 2024-04-14T15:25:33.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.163+0000 7fed6cff9700 20 client.49843.objecter put_session s=0x7fed5c00c840 osd=5 4 2024-04-14T15:25:33.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.163+0000 7fed6cff9700 15 client.49843.objecter _session_command_op_remove 5 11 2024-04-14T15:25:33.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.180+0000 7fed75aa0700 10 client.49843.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:25:33.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.180+0000 7fed75aa0700 15 client.49843.objecter _session_command_op_assign -1 12 2024-04-14T15:25:33.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.180+0000 7fed75aa0700 20 client.49843.objecter _get_session s=0x7fed5c00c840 osd=5 3 2024-04-14T15:25:33.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.180+0000 7fed75aa0700 20 client.49843.objecter put_session s=0x7fed5c00c840 osd=5 3 2024-04-14T15:25:33.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.180+0000 7fed75aa0700 20 client.49843.objecter _get_session s=0x7fed5c00c840 osd=5 3 2024-04-14T15:25:33.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.180+0000 7fed75aa0700 15 client.49843.objecter _session_command_op_remove -1 12 2024-04-14T15:25:33.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.180+0000 7fed75aa0700 20 client.49843.objecter get_session s=0x7fed5c00c840 osd=5 3 2024-04-14T15:25:33.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.180+0000 7fed75aa0700 15 client.49843.objecter _session_command_op_assign 5 12 2024-04-14T15:25:33.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.180+0000 7fed75aa0700 20 client.49843.objecter put_session s=0x7fed5c00c840 osd=5 4 2024-04-14T15:25:33.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.180+0000 7fed75aa0700 10 client.49843.objecter _send_command 12 2024-04-14T15:25:33.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.180+0000 7fed75aa0700 1 -- 172.21.15.90:0/434777481 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fed5c013670 con 0x7fed5c012e80 2024-04-14T15:25:33.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.181+0000 7fed6cff9700 1 -- 172.21.15.90:0/434777481 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fed5c013670 con 0x7fed5c012e80 2024-04-14T15:25:33.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.181+0000 7fed6cff9700 10 client.49843.objecter ms_dispatch 0x7fed70001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:25:33.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.181+0000 7fed6cff9700 10 client.49843.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:25:33.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.181+0000 7fed6cff9700 20 client.49843.objecter put_session s=0x7fed5c00c840 osd=5 4 2024-04-14T15:25:33.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.181+0000 7fed6cff9700 15 client.49843.objecter _session_command_op_remove 5 12 2024-04-14T15:25:33.183 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:25:33.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.181+0000 7fed75aa0700 10 client.49843.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:25:33.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.181+0000 7fed75aa0700 15 client.49843.objecter _session_command_op_assign -1 13 2024-04-14T15:25:33.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.181+0000 7fed75aa0700 20 client.49843.objecter _get_session s=0x7fed5c013670 osd=6 3 2024-04-14T15:25:33.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.181+0000 7fed75aa0700 20 client.49843.objecter put_session s=0x7fed5c013670 osd=6 3 2024-04-14T15:25:33.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.181+0000 7fed75aa0700 20 client.49843.objecter _get_session s=0x7fed5c013670 osd=6 3 2024-04-14T15:25:33.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.181+0000 7fed75aa0700 15 client.49843.objecter _session_command_op_remove -1 13 2024-04-14T15:25:33.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.181+0000 7fed75aa0700 20 client.49843.objecter get_session s=0x7fed5c013670 osd=6 3 2024-04-14T15:25:33.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.181+0000 7fed75aa0700 15 client.49843.objecter _session_command_op_assign 6 13 2024-04-14T15:25:33.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.181+0000 7fed75aa0700 20 client.49843.objecter put_session s=0x7fed5c013670 osd=6 4 2024-04-14T15:25:33.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.181+0000 7fed75aa0700 10 client.49843.objecter _send_command 13 2024-04-14T15:25:33.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.181+0000 7fed75aa0700 1 -- 172.21.15.90:0/434777481 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fed5c0127b0 con 0x7fed5c023b00 2024-04-14T15:25:33.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.182+0000 7fed6cff9700 10 client.49843.objecter ms_handle_connect 0x7fed5c023b00 2024-04-14T15:25:33.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.182+0000 7fed6cff9700 1 -- 172.21.15.90:0/434777481 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fed68012ab0 con 0x7fed5c023b00 2024-04-14T15:25:33.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.182+0000 7fed6cff9700 10 client.49843.objecter ms_dispatch 0x7fed70001550 command_reply(tid 13: 0 ) 2024-04-14T15:25:33.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.182+0000 7fed6cff9700 10 client.49843.objecter _finish_command 13 = system:0 2024-04-14T15:25:33.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.182+0000 7fed6cff9700 20 client.49843.objecter put_session s=0x7fed5c013670 osd=6 4 2024-04-14T15:25:33.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.182+0000 7fed6cff9700 15 client.49843.objecter _session_command_op_remove 6 13 2024-04-14T15:25:33.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.201+0000 7fed75aa0700 10 client.49843.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:25:33.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.201+0000 7fed75aa0700 15 client.49843.objecter _session_command_op_assign -1 14 2024-04-14T15:25:33.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.201+0000 7fed75aa0700 20 client.49843.objecter _get_session s=0x7fed5c013670 osd=6 3 2024-04-14T15:25:33.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.201+0000 7fed75aa0700 20 client.49843.objecter put_session s=0x7fed5c013670 osd=6 3 2024-04-14T15:25:33.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.201+0000 7fed75aa0700 20 client.49843.objecter _get_session s=0x7fed5c013670 osd=6 3 2024-04-14T15:25:33.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.201+0000 7fed75aa0700 15 client.49843.objecter _session_command_op_remove -1 14 2024-04-14T15:25:33.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.201+0000 7fed75aa0700 20 client.49843.objecter get_session s=0x7fed5c013670 osd=6 3 2024-04-14T15:25:33.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.201+0000 7fed75aa0700 15 client.49843.objecter _session_command_op_assign 6 14 2024-04-14T15:25:33.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.201+0000 7fed75aa0700 20 client.49843.objecter put_session s=0x7fed5c013670 osd=6 4 2024-04-14T15:25:33.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.201+0000 7fed75aa0700 10 client.49843.objecter _send_command 14 2024-04-14T15:25:33.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.201+0000 7fed75aa0700 1 -- 172.21.15.90:0/434777481 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fed5c012140 con 0x7fed5c023b00 2024-04-14T15:25:33.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.204+0000 7fed6cff9700 1 -- 172.21.15.90:0/434777481 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fed5c012140 con 0x7fed5c023b00 2024-04-14T15:25:33.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.204+0000 7fed6cff9700 10 client.49843.objecter ms_dispatch 0x7fed70001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:25:33.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.204+0000 7fed6cff9700 10 client.49843.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:25:33.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.204+0000 7fed6cff9700 20 client.49843.objecter put_session s=0x7fed5c013670 osd=6 4 2024-04-14T15:25:33.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.204+0000 7fed6cff9700 15 client.49843.objecter _session_command_op_remove 6 14 2024-04-14T15:25:33.204 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:25:33.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.204+0000 7fed75aa0700 10 client.49843.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:25:33.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.204+0000 7fed75aa0700 15 client.49843.objecter _session_command_op_assign -1 15 2024-04-14T15:25:33.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.204+0000 7fed75aa0700 20 client.49843.objecter _get_session s=0x7fed5c011ef0 osd=7 3 2024-04-14T15:25:33.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.204+0000 7fed75aa0700 20 client.49843.objecter put_session s=0x7fed5c011ef0 osd=7 3 2024-04-14T15:25:33.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.204+0000 7fed75aa0700 20 client.49843.objecter _get_session s=0x7fed5c011ef0 osd=7 3 2024-04-14T15:25:33.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.204+0000 7fed75aa0700 15 client.49843.objecter _session_command_op_remove -1 15 2024-04-14T15:25:33.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.204+0000 7fed75aa0700 20 client.49843.objecter get_session s=0x7fed5c011ef0 osd=7 3 2024-04-14T15:25:33.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.204+0000 7fed75aa0700 15 client.49843.objecter _session_command_op_assign 7 15 2024-04-14T15:25:33.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.204+0000 7fed75aa0700 20 client.49843.objecter put_session s=0x7fed5c011ef0 osd=7 4 2024-04-14T15:25:33.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.204+0000 7fed75aa0700 10 client.49843.objecter _send_command 15 2024-04-14T15:25:33.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.204+0000 7fed75aa0700 1 -- 172.21.15.90:0/434777481 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fed5c012060 con 0x7fed5c01f7a0 2024-04-14T15:25:33.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.204+0000 7fed6cff9700 10 client.49843.objecter ms_handle_connect 0x7fed5c01f7a0 2024-04-14T15:25:33.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.204+0000 7fed6cff9700 1 -- 172.21.15.90:0/434777481 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fed5c012060 con 0x7fed5c01f7a0 2024-04-14T15:25:33.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.204+0000 7fed6cff9700 10 client.49843.objecter ms_dispatch 0x7fed70001550 command_reply(tid 15: 0 ) 2024-04-14T15:25:33.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.204+0000 7fed6cff9700 10 client.49843.objecter _finish_command 15 = system:0 2024-04-14T15:25:33.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.204+0000 7fed6cff9700 20 client.49843.objecter put_session s=0x7fed5c011ef0 osd=7 4 2024-04-14T15:25:33.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.204+0000 7fed6cff9700 15 client.49843.objecter _session_command_op_remove 7 15 2024-04-14T15:25:33.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.220+0000 7fed75aa0700 10 client.49843.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:25:33.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.220+0000 7fed75aa0700 15 client.49843.objecter _session_command_op_assign -1 16 2024-04-14T15:25:33.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.220+0000 7fed75aa0700 20 client.49843.objecter _get_session s=0x7fed5c011ef0 osd=7 3 2024-04-14T15:25:33.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.220+0000 7fed75aa0700 20 client.49843.objecter put_session s=0x7fed5c011ef0 osd=7 3 2024-04-14T15:25:33.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.220+0000 7fed75aa0700 20 client.49843.objecter _get_session s=0x7fed5c011ef0 osd=7 3 2024-04-14T15:25:33.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.220+0000 7fed75aa0700 15 client.49843.objecter _session_command_op_remove -1 16 2024-04-14T15:25:33.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.220+0000 7fed75aa0700 20 client.49843.objecter get_session s=0x7fed5c011ef0 osd=7 3 2024-04-14T15:25:33.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.220+0000 7fed75aa0700 15 client.49843.objecter _session_command_op_assign 7 16 2024-04-14T15:25:33.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.220+0000 7fed75aa0700 20 client.49843.objecter put_session s=0x7fed5c011ef0 osd=7 4 2024-04-14T15:25:33.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.220+0000 7fed75aa0700 10 client.49843.objecter _send_command 16 2024-04-14T15:25:33.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.220+0000 7fed75aa0700 1 -- 172.21.15.90:0/434777481 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fed5c012060 con 0x7fed5c01f7a0 2024-04-14T15:25:33.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.222+0000 7fed6cff9700 1 -- 172.21.15.90:0/434777481 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fed5c012060 con 0x7fed5c01f7a0 2024-04-14T15:25:33.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.222+0000 7fed6cff9700 10 client.49843.objecter ms_dispatch 0x7fed70001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:25:33.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.222+0000 7fed6cff9700 10 client.49843.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:25:33.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.222+0000 7fed6cff9700 20 client.49843.objecter put_session s=0x7fed5c011ef0 osd=7 4 2024-04-14T15:25:33.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.222+0000 7fed6cff9700 15 client.49843.objecter _session_command_op_remove 7 16 2024-04-14T15:25:33.224 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:25:33.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.223+0000 7fed75aa0700 10 librados: watch_flush enter 2024-04-14T15:25:33.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.223+0000 7fed75aa0700 10 librados: watch_flush exit 2024-04-14T15:25:33.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.223+0000 7fed75aa0700 10 client.49843.objecter close_session for osd.0 2024-04-14T15:25:33.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.223+0000 7fed75aa0700 1 -- 172.21.15.90:0/434777481 >> v1:172.21.15.90:6801/3761806885 conn(0x7fed5c004760 legacy=0x7fed5c002780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:33.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.223+0000 7fed75aa0700 20 client.49843.objecter put_session s=0x7fed5c005410 osd=0 1 2024-04-14T15:25:33.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.223+0000 7fed75aa0700 10 client.49843.objecter close_session for osd.1 2024-04-14T15:25:33.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.223+0000 7fed75aa0700 1 -- 172.21.15.90:0/434777481 >> v1:172.21.15.90:6805/4170361699 conn(0x7fed5c0042d0 legacy=0x7fed5c00ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:33.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.223+0000 7fed75aa0700 20 client.49843.objecter put_session s=0x7fed5c004160 osd=1 1 2024-04-14T15:25:33.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.223+0000 7fed75aa0700 10 client.49843.objecter close_session for osd.2 2024-04-14T15:25:33.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.223+0000 7fed75aa0700 1 -- 172.21.15.90:0/434777481 >> v1:172.21.15.90:6809/1686184251 conn(0x7fed5c0089c0 legacy=0x7fed5c00f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:33.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.223+0000 7fed75aa0700 20 client.49843.objecter put_session s=0x7fed5c001a30 osd=2 1 2024-04-14T15:25:33.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.223+0000 7fed75aa0700 10 client.49843.objecter close_session for osd.3 2024-04-14T15:25:33.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.223+0000 7fed75aa0700 1 -- 172.21.15.90:0/434777481 >> v1:172.21.15.90:6813/2270166774 conn(0x7fed5c0111c0 legacy=0x7fed5c011600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:33.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.223+0000 7fed75aa0700 20 client.49843.objecter put_session s=0x7fed5c008320 osd=3 1 2024-04-14T15:25:33.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.223+0000 7fed75aa0700 10 client.49843.objecter close_session for osd.4 2024-04-14T15:25:33.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.223+0000 7fed75aa0700 1 -- 172.21.15.90:0/434777481 >> v1:172.21.15.92:6800/3887549733 conn(0x7fed5c01afb0 legacy=0x7fed5c01d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:33.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.223+0000 7fed75aa0700 20 client.49843.objecter put_session s=0x7fed5c013840 osd=4 1 2024-04-14T15:25:33.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.223+0000 7fed75aa0700 10 client.49843.objecter close_session for osd.5 2024-04-14T15:25:33.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.223+0000 7fed75aa0700 1 -- 172.21.15.90:0/434777481 >> v1:172.21.15.92:6804/3244561440 conn(0x7fed5c012e80 legacy=0x7fed5c01fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:33.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.223+0000 7fed75aa0700 20 client.49843.objecter put_session s=0x7fed5c00c840 osd=5 1 2024-04-14T15:25:33.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.223+0000 7fed75aa0700 10 client.49843.objecter close_session for osd.6 2024-04-14T15:25:33.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.223+0000 7fed75aa0700 1 -- 172.21.15.90:0/434777481 >> v1:172.21.15.92:6808/636125963 conn(0x7fed5c023b00 legacy=0x7fed5c025ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:33.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.223+0000 7fed75aa0700 20 client.49843.objecter put_session s=0x7fed5c013670 osd=6 1 2024-04-14T15:25:33.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.223+0000 7fed75aa0700 10 client.49843.objecter close_session for osd.7 2024-04-14T15:25:33.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.223+0000 7fed75aa0700 1 -- 172.21.15.90:0/434777481 >> v1:172.21.15.92:6812/1773706716 conn(0x7fed5c01f7a0 legacy=0x7fed5c027640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:33.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.223+0000 7fed75aa0700 20 client.49843.objecter put_session s=0x7fed5c011ef0 osd=7 1 2024-04-14T15:25:33.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.223+0000 7fed75aa0700 20 client.49843.objecter shutdown clearing up homeless session... 2024-04-14T15:25:33.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.223+0000 7fed75aa0700 10 client.49843.objecter successfully canceled tick 2024-04-14T15:25:33.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.223+0000 7fed6ffff700 1 -- 172.21.15.90:0/434777481 reap_dead start 2024-04-14T15:25:33.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.223+0000 7fed6ffff700 1 -- 172.21.15.90:0/434777481 reap_dead start 2024-04-14T15:25:33.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.223+0000 7fed75aa0700 1 -- 172.21.15.90:0/434777481 >> v1:172.21.15.90:6800/420935427 conn(0x7fed58071080 legacy=0x7fed58073540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:33.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.223+0000 7fed75aa0700 1 -- 172.21.15.90:0/434777481 >> v1:172.21.15.92:6789/0 conn(0x7fed70107760 legacy=0x7fed7006a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:33.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.223+0000 7fed6ffff700 1 -- 172.21.15.90:0/434777481 reap_dead start 2024-04-14T15:25:33.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.223+0000 7fed6ffff700 1 -- 172.21.15.90:0/434777481 reap_dead start 2024-04-14T15:25:33.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.223+0000 7fed6ffff700 1 -- 172.21.15.90:0/434777481 reap_dead start 2024-04-14T15:25:33.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.223+0000 7fed75aa0700 1 -- 172.21.15.90:0/434777481 shutdown_connections 2024-04-14T15:25:33.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.223+0000 7fed75aa0700 1 -- 172.21.15.90:0/434777481 >> 172.21.15.90:0/434777481 conn(0x7fed700fd290 msgr2=0x7fed701117b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:25:33.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.223+0000 7fed75aa0700 1 -- 172.21.15.90:0/434777481 shutdown_connections 2024-04-14T15:25:33.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.223+0000 7fed75aa0700 1 -- 172.21.15.90:0/434777481 wait complete. 2024-04-14T15:25:33.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:33.223+0000 7fed75aa0700 1 librados: shutdown 2024-04-14T15:25:33.229 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:25:33.229 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:25:33.229 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:25:33.229 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:25:33.229 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:25:33.229 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:25:33.229 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:25:33.230 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:25:33.234 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:25:33.257 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:33.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:33.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:25:33.359 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:25:33.383 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:33.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:33.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:25:33.484 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:25:33.509 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:33.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:33.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:25:33.611 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:25:33.634 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:33.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:33.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:25:33.736 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:25:33.738 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd reweight 4 1 2024-04-14T15:25:33.801 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:33.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:33.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:25:33.903 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:25:33.926 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:33.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:33.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:25:34.027 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:25:34.037 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:25:33 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:25:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:25:34.052 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:34.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:34.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:25:34.153 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:25:34.176 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:34.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:34.177 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:25:34.278 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:25:34.303 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:34.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:34.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:25:34.351 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:34 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:25:34.352 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:34 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:25:34.352 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:34 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/434777481' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:25:34.405 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:34.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:34.430 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:34.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:34.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:34 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:25:34.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:34 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:25:34.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:34 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/434777481' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:25:34.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:34 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:25:34.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:34 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:25:34.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:34 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/434777481' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:25:34.483 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:25:34.532 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:25:34.555 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:34.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:34.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:25:34.657 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:25:34.682 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:34.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:34.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:25:34.783 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:25:34.807 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:34.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:34.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:25:34.909 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:25:34.923 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-04-14T15:25:34.923 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-04-14T15:25:34.923 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 0 2024-04-14T15:25:34.923 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-04-14T15:25:34.934 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:34.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:34.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:25:35.035 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:25:35.059 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:35.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:35.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:25:35.161 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:25:35.185 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:35.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:35.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:25:35.287 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:25:35.310 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:35.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:35.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:25:35.360 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:35 smithi090 ceph-mon[41670]: pgmap v1168: 224 pgs: 4 active+remapped+backfill_toofull, 220 active+clean; 4.2 MiB data, 749 MiB used, 625 GiB / 626 GiB avail; 2/648 objects misplaced (0.309%) 2024-04-14T15:25:35.361 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:35 smithi090 ceph-mon[30399]: pgmap v1168: 224 pgs: 4 active+remapped+backfill_toofull, 220 active+clean; 4.2 MiB data, 749 MiB used, 625 GiB / 626 GiB avail; 2/648 objects misplaced (0.309%) 2024-04-14T15:25:35.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:35 smithi092 ceph-mon[33966]: pgmap v1168: 224 pgs: 4 active+remapped+backfill_toofull, 220 active+clean; 4.2 MiB data, 749 MiB used, 625 GiB / 626 GiB avail; 2/648 objects misplaced (0.309%) 2024-04-14T15:25:35.412 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:25:35.435 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:35.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:35.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:25:35.537 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:25:35.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:35.562 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:35.562 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:25:35.663 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:25:35.686 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:35.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:35.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:25:35.717 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:25:35 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:25:35.742 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-14T15:25:35.788 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:25:35.814 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:35.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:35.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:25:35.915 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:35.938 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:35.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:35.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:35.978 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:25:36.032 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:25:36.040 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:25:36.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:36.065 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:36.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:25:36.166 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:36.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:36.192 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:36.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:36.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:36.275+0000 7f7996870700 1 -- 172.21.15.90:0/2497695226 >> v1:172.21.15.90:6789/0 conn(0x7f7990101830 legacy=0x7f7990101c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:36.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:36.276+0000 7f7996870700 1 -- 172.21.15.90:0/2497695226 shutdown_connections 2024-04-14T15:25:36.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:36.276+0000 7f7996870700 1 -- 172.21.15.90:0/2497695226 >> 172.21.15.90:0/2497695226 conn(0x7f79900fd290 msgr2=0x7f79900ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:25:36.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:36.276+0000 7f7996870700 1 -- 172.21.15.90:0/2497695226 shutdown_connections 2024-04-14T15:25:36.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:36.276+0000 7f7996870700 1 -- 172.21.15.90:0/2497695226 wait complete. 2024-04-14T15:25:36.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:36.276+0000 7f7996870700 1 librados: starting msgr at 2024-04-14T15:25:36.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:36.276+0000 7f7996870700 1 librados: starting objecter 2024-04-14T15:25:36.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:36.276+0000 7f7996870700 1 Processor -- start 2024-04-14T15:25:36.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:36.276+0000 7f7996870700 1 -- start start 2024-04-14T15:25:36.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:36.276+0000 7f7996870700 1 librados: setting wanted keys 2024-04-14T15:25:36.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:36.276+0000 7f7996870700 1 librados: calling monclient init 2024-04-14T15:25:36.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:36.277+0000 7f7996870700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7990115740 con 0x7f7990107630 2024-04-14T15:25:36.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:36.277+0000 7f7996870700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7990115950 con 0x7f7990101830 2024-04-14T15:25:36.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:36.277+0000 7f7996870700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f79901b0550 con 0x7f7990115b40 2024-04-14T15:25:36.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:36.277+0000 7f7994e0d700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f7990115b40 0x7f79901acdd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:35966/0 (socket says 172.21.15.90:35966) 2024-04-14T15:25:36.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:36.277+0000 7f798f7fe700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f7990107630 0x7f7990112cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:48588/0 (socket says 172.21.15.90:48588) 2024-04-14T15:25:36.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:36.277+0000 7f7994e0d700 1 -- 172.21.15.90:0/3645330050 learned_addr learned my addr 172.21.15.90:0/3645330050 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:25:36.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:36.277+0000 7f798d7fa700 10 client.?.objecter ms_handle_connect 0x7f7990107630 2024-04-14T15:25:36.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:36.277+0000 7f798d7fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:25:36.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:36.277+0000 7f798d7fa700 10 client.?.objecter ms_handle_connect 0x7f7990101830 2024-04-14T15:25:36.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:36.277+0000 7f798d7fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:25:36.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:36.277+0000 7f798d7fa700 10 client.?.objecter ms_handle_connect 0x7f7990115b40 2024-04-14T15:25:36.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:36.277+0000 7f798d7fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:25:36.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:36.277+0000 7f798d7fa700 1 -- 172.21.15.90:0/3645330050 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3362541727 0 0) 0x7f7990115740 con 0x7f7990107630 2024-04-14T15:25:36.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:36.277+0000 7f798d7fa700 1 -- 172.21.15.90:0/3645330050 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f79780035b0 con 0x7f7990107630 2024-04-14T15:25:36.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:36.277+0000 7f798d7fa700 1 -- 172.21.15.90:0/3645330050 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2335274292 0 0) 0x7f7990115950 con 0x7f7990101830 2024-04-14T15:25:36.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:36.277+0000 7f798d7fa700 1 -- 172.21.15.90:0/3645330050 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7990115740 con 0x7f7990101830 2024-04-14T15:25:36.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:36.277+0000 7f798d7fa700 1 -- 172.21.15.90:0/3645330050 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 583944088 0 0) 0x7f79901b0550 con 0x7f7990115b40 2024-04-14T15:25:36.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:36.277+0000 7f798d7fa700 1 -- 172.21.15.90:0/3645330050 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7990115950 con 0x7f7990115b40 2024-04-14T15:25:36.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:36.277+0000 7f798d7fa700 1 -- 172.21.15.90:0/3645330050 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3057225142 0 0) 0x7f7990115740 con 0x7f7990101830 2024-04-14T15:25:36.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:36.277+0000 7f798d7fa700 1 -- 172.21.15.90:0/3645330050 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f79901b0550 con 0x7f7990101830 2024-04-14T15:25:36.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:36.277+0000 7f798d7fa700 1 -- 172.21.15.90:0/3645330050 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f79800015d0 con 0x7f7990101830 2024-04-14T15:25:36.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:36.277+0000 7f798d7fa700 1 -- 172.21.15.90:0/3645330050 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2227475998 0 0) 0x7f79780035b0 con 0x7f7990107630 2024-04-14T15:25:36.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:36.277+0000 7f798d7fa700 1 -- 172.21.15.90:0/3645330050 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7990115740 con 0x7f7990107630 2024-04-14T15:25:36.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:36.277+0000 7f798d7fa700 1 -- 172.21.15.90:0/3645330050 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f7984001830 con 0x7f7990107630 2024-04-14T15:25:36.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:36.277+0000 7f798d7fa700 1 -- 172.21.15.90:0/3645330050 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3851106341 0 0) 0x7f7990115950 con 0x7f7990115b40 2024-04-14T15:25:36.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:36.277+0000 7f798d7fa700 1 -- 172.21.15.90:0/3645330050 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f79780035b0 con 0x7f7990115b40 2024-04-14T15:25:36.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:36.278+0000 7f798d7fa700 1 -- 172.21.15.90:0/3645330050 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f7988001c70 con 0x7f7990115b40 2024-04-14T15:25:36.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:36.278+0000 7f798d7fa700 1 -- 172.21.15.90:0/3645330050 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2858225391 0 0) 0x7f79901b0550 con 0x7f7990101830 2024-04-14T15:25:36.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:36.278+0000 7f798d7fa700 1 -- 172.21.15.90:0/3645330050 >> v1:172.21.15.90:6790/0 conn(0x7f7990115b40 legacy=0x7f79901acdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:36.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:36.278+0000 7f798d7fa700 1 -- 172.21.15.90:0/3645330050 >> v1:172.21.15.90:6789/0 conn(0x7f7990107630 legacy=0x7f7990112cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:36.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:36.278+0000 7f798d7fa700 1 -- 172.21.15.90:0/3645330050 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79901b1790 con 0x7f7990101830 2024-04-14T15:25:36.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:36.278+0000 7f7996870700 1 -- 172.21.15.90:0/3645330050 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f79901b0780 con 0x7f7990101830 2024-04-14T15:25:36.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:36.278+0000 7f7996870700 10 client.49849.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:25:36.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:36.278+0000 7f7996870700 1 -- 172.21.15.90:0/3645330050 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f79901b0c40 con 0x7f7990101830 2024-04-14T15:25:36.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:36.278+0000 7f7996870700 1 librados: init done 2024-04-14T15:25:36.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:36.278+0000 7f798d7fa700 1 -- 172.21.15.90:0/3645330050 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f7980003740 con 0x7f7990101830 2024-04-14T15:25:36.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:36.278+0000 7f798d7fa700 1 -- 172.21.15.90:0/3645330050 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f7980004d60 con 0x7f7990101830 2024-04-14T15:25:36.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:36.279+0000 7f7996870700 1 -- 172.21.15.90:0/3645330050 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f797c005120 con 0x7f7990101830 2024-04-14T15:25:36.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:36.279+0000 7f798d7fa700 1 -- 172.21.15.90:0/3645330050 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f7980001b70 con 0x7f7990101830 2024-04-14T15:25:36.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:36.280+0000 7f798d7fa700 1 -- 172.21.15.90:0/3645330050 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(790..790 src has 467..790) ==== 8816+0+0 (unknown 82271722 0 0) 0x7f7980089a50 con 0x7f7990101830 2024-04-14T15:25:36.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:36.280+0000 7f798d7fa700 10 client.49849.objecter ms_dispatch 0x7f7990001550 osd_map(790..790 src has 467..790) 2024-04-14T15:25:36.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:36.280+0000 7f798d7fa700 3 client.49849.objecter handle_osd_map got epochs [790,790] > 0 2024-04-14T15:25:36.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:36.280+0000 7f798d7fa700 3 client.49849.objecter handle_osd_map decoding full epoch 790 2024-04-14T15:25:36.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:36.280+0000 7f798d7fa700 20 client.49849.objecter dump_active .. 0 homeless 2024-04-14T15:25:36.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:36.280+0000 7f798d7fa700 10 client.49849.objecter ms_handle_connect 0x7f7978071380 2024-04-14T15:25:36.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:36.285+0000 7f798d7fa700 1 -- 172.21.15.90:0/3645330050 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f79800561d0 con 0x7f7990101830 2024-04-14T15:25:36.293 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:25:36.316 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:36.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:36.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:25:36.418 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:25:36.440 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:36.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:36.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:25:36.479 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-14T15:25:36.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:36.516+0000 7f7996870700 1 -- 172.21.15.90:0/3645330050 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7f797c005410 con 0x7f7990101830 2024-04-14T15:25:36.543 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:25:36.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:36.567 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:36.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:25:36.668 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:25:36.691 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:36.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:36.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:25:36.793 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:25:36.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:36.818 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:36.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:25:36.919 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:25:36.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:36.943 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:36.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:25:37.044 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:25:37.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:37.047+0000 7f798d7fa700 1 -- 172.21.15.90:0/3645330050 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v791) ==== 97+0+0 (unknown 291207896 0 0) 0x7f7980059e80 con 0x7f7990101830 2024-04-14T15:25:37.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:37.049+0000 7f7996870700 10 librados: watch_flush enter 2024-04-14T15:25:37.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:37.049+0000 7f7996870700 10 librados: watch_flush exit 2024-04-14T15:25:37.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:37.049+0000 7f7996870700 20 client.49849.objecter shutdown clearing up homeless session... 2024-04-14T15:25:37.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:37.049+0000 7f7996870700 10 client.49849.objecter successfully canceled tick 2024-04-14T15:25:37.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:37.050+0000 7f7996870700 1 -- 172.21.15.90:0/3645330050 >> v1:172.21.15.90:6800/420935427 conn(0x7f7978071380 legacy=0x7f7978073840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:37.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:37.050+0000 7f7996870700 1 -- 172.21.15.90:0/3645330050 >> v1:172.21.15.92:6789/0 conn(0x7f7990101830 legacy=0x7f79901125e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:37.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:37.050+0000 7f7996870700 1 -- 172.21.15.90:0/3645330050 shutdown_connections 2024-04-14T15:25:37.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:37.050+0000 7f7996870700 1 -- 172.21.15.90:0/3645330050 >> 172.21.15.90:0/3645330050 conn(0x7f79900fd290 msgr2=0x7f79900fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:25:37.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:37.050+0000 7f7996870700 1 -- 172.21.15.90:0/3645330050 shutdown_connections 2024-04-14T15:25:37.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:37.050+0000 7f7996870700 1 -- 172.21.15.90:0/3645330050 wait complete. 2024-04-14T15:25:37.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:37.050+0000 7f7996870700 1 librados: shutdown 2024-04-14T15:25:37.051 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is set 2024-04-14T15:25:37.070 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:37.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:37.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:25:37.172 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:25:37.195 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:37.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:37.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:25:37.297 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:37.317 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:37 smithi092 ceph-mon[33966]: pgmap v1169: 224 pgs: 1 active+remapped+backfill_toofull, 223 active+clean; 4.2 MiB data, 750 MiB used, 625 GiB / 626 GiB avail; 991 B/s rd, 3.1 KiB/s wr, 1 op/s; 62 KiB/s, 0 objects/s recovering 2024-04-14T15:25:37.317 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:37 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:25:37.317 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:37 smithi092 ceph-mon[33966]: osdmap e790: 8 total, 8 up, 7 in 2024-04-14T15:25:37.317 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:37 smithi092 ceph-mon[33966]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-04-14T15:25:37.317 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:37 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3645330050' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:25:37.317 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:37 smithi092 ceph-mon[33966]: from='client.49849 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:25:37.321 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:37.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:37.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:37.423 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:25:37.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:37 smithi090 ceph-mon[41670]: pgmap v1169: 224 pgs: 1 active+remapped+backfill_toofull, 223 active+clean; 4.2 MiB data, 750 MiB used, 625 GiB / 626 GiB avail; 991 B/s rd, 3.1 KiB/s wr, 1 op/s; 62 KiB/s, 0 objects/s recovering 2024-04-14T15:25:37.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:37 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:25:37.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:37 smithi090 ceph-mon[41670]: osdmap e790: 8 total, 8 up, 7 in 2024-04-14T15:25:37.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:37 smithi090 ceph-mon[41670]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-04-14T15:25:37.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:37 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3645330050' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:25:37.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:37 smithi090 ceph-mon[41670]: from='client.49849 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:25:37.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:37 smithi090 ceph-mon[30399]: pgmap v1169: 224 pgs: 1 active+remapped+backfill_toofull, 223 active+clean; 4.2 MiB data, 750 MiB used, 625 GiB / 626 GiB avail; 991 B/s rd, 3.1 KiB/s wr, 1 op/s; 62 KiB/s, 0 objects/s recovering 2024-04-14T15:25:37.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:37 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:25:37.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:37 smithi090 ceph-mon[30399]: osdmap e790: 8 total, 8 up, 7 in 2024-04-14T15:25:37.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:37 smithi090 ceph-mon[30399]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-04-14T15:25:37.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:37 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3645330050' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:25:37.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:37 smithi090 ceph-mon[30399]: from='client.49849 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:25:37.446 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:37.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:37.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:25:37.548 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:25:37.572 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:37.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:37.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:25:37.674 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:25:37.698 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:37.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:37.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:25:37.799 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:25:37.824 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:37.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:37.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:25:37.870 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-04-14T15:25:37.871 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-04-14T15:25:37.871 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-04-14T15:25:37.871 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-04-14T15:25:37.871 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-04-14T15:25:37.925 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:25:37.949 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:37.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:37.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:25:38.051 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:38.075 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:38.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:38.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:38.108 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:38 smithi090 ceph-mon[41670]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:25:38.108 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:38 smithi090 ceph-mon[41670]: from='client.49849 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-14T15:25:38.108 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:38 smithi090 ceph-mon[41670]: osdmap e791: 8 total, 8 up, 7 in 2024-04-14T15:25:38.108 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:38 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:25:38.108 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:38 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:25:38.108 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:38 smithi090 ceph-mon[30399]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:25:38.109 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:38 smithi090 ceph-mon[30399]: from='client.49849 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-14T15:25:38.109 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:38 smithi090 ceph-mon[30399]: osdmap e791: 8 total, 8 up, 7 in 2024-04-14T15:25:38.109 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:38 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:25:38.109 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:38 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:25:38.145 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:25:38.177 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:25:38.201 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:38.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:38.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:25:38.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:38 smithi092 ceph-mon[33966]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:25:38.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:38 smithi092 ceph-mon[33966]: from='client.49849 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-14T15:25:38.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:38 smithi092 ceph-mon[33966]: osdmap e791: 8 total, 8 up, 7 in 2024-04-14T15:25:38.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:38 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:25:38.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:38 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:25:38.302 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:25:38.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:38.326 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:38.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:25:38.427 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:25:38.450 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:38.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:38.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:25:38.552 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:25:38.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:38.578 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:38.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:25:38.651 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-14T15:25:38.679 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:25:38.703 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:38.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:38.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:25:38.806 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:25:38.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:38.812+0000 7fa0f840a700 1 -- 172.21.15.90:0/369121125 >> v1:172.21.15.90:6789/0 conn(0x7fa0f010b750 legacy=0x7fa0f010db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:38.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:38.812+0000 7fa0f840a700 1 -- 172.21.15.90:0/369121125 shutdown_connections 2024-04-14T15:25:38.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:38.812+0000 7fa0f840a700 1 -- 172.21.15.90:0/369121125 >> 172.21.15.90:0/369121125 conn(0x7fa0f00fe390 msgr2=0x7fa0f01007b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:25:38.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:38.812+0000 7fa0f840a700 1 -- 172.21.15.90:0/369121125 shutdown_connections 2024-04-14T15:25:38.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:38.812+0000 7fa0f840a700 1 -- 172.21.15.90:0/369121125 wait complete. 2024-04-14T15:25:38.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:38.812+0000 7fa0f840a700 1 librados: starting msgr at 2024-04-14T15:25:38.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:38.812+0000 7fa0f840a700 1 librados: starting objecter 2024-04-14T15:25:38.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:38.813+0000 7fa0f840a700 1 Processor -- start 2024-04-14T15:25:38.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:38.813+0000 7fa0f840a700 1 -- start start 2024-04-14T15:25:38.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:38.813+0000 7fa0f840a700 1 librados: setting wanted keys 2024-04-14T15:25:38.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:38.813+0000 7fa0f840a700 1 librados: calling monclient init 2024-04-14T15:25:38.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:38.813+0000 7fa0f840a700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa0f010efa0 con 0x7fa0f01088a0 2024-04-14T15:25:38.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:38.813+0000 7fa0f840a700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa0f01a8f30 con 0x7fa0f010f320 2024-04-14T15:25:38.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:38.813+0000 7fa0f840a700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa0f01aa110 con 0x7fa0f010b750 2024-04-14T15:25:38.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:38.813+0000 7fa0f61a6700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fa0f01088a0 0x7fa0f010bdf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:48596/0 (socket says 172.21.15.90:48596) 2024-04-14T15:25:38.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:38.813+0000 7fa0f61a6700 1 -- 172.21.15.90:0/3783327058 learned_addr learned my addr 172.21.15.90:0/3783327058 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:25:38.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:38.813+0000 7fa0f59a5700 1 --1- 172.21.15.90:0/3783327058 >> v1:172.21.15.90:6790/0 conn(0x7fa0f010b750 0x7fa0f010c500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:35974/0 (socket says 172.21.15.90:35974) 2024-04-14T15:25:38.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:38.813+0000 7fa0e77fe700 10 client.?.objecter ms_handle_connect 0x7fa0f010b750 2024-04-14T15:25:38.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:38.813+0000 7fa0e77fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:25:38.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:38.813+0000 7fa0e77fe700 10 client.?.objecter ms_handle_connect 0x7fa0f01088a0 2024-04-14T15:25:38.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:38.813+0000 7fa0e77fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:25:38.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:38.813+0000 7fa0e77fe700 10 client.?.objecter ms_handle_connect 0x7fa0f010f320 2024-04-14T15:25:38.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:38.813+0000 7fa0e77fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:25:38.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:38.813+0000 7fa0e77fe700 1 -- 172.21.15.90:0/3783327058 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 721715294 0 0) 0x7fa0f01aa110 con 0x7fa0f010b750 2024-04-14T15:25:38.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:38.813+0000 7fa0e77fe700 1 -- 172.21.15.90:0/3783327058 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa0dc0035b0 con 0x7fa0f010b750 2024-04-14T15:25:38.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:38.813+0000 7fa0e77fe700 1 -- 172.21.15.90:0/3783327058 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 713541891 0 0) 0x7fa0f010efa0 con 0x7fa0f01088a0 2024-04-14T15:25:38.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:38.813+0000 7fa0e77fe700 1 -- 172.21.15.90:0/3783327058 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa0f01aa110 con 0x7fa0f01088a0 2024-04-14T15:25:38.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:38.813+0000 7fa0e77fe700 1 -- 172.21.15.90:0/3783327058 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 522186507 0 0) 0x7fa0f01a8f30 con 0x7fa0f010f320 2024-04-14T15:25:38.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:38.813+0000 7fa0e77fe700 1 -- 172.21.15.90:0/3783327058 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa0f010efa0 con 0x7fa0f010f320 2024-04-14T15:25:38.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:38.813+0000 7fa0e77fe700 1 -- 172.21.15.90:0/3783327058 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 413508966 0 0) 0x7fa0f01aa110 con 0x7fa0f01088a0 2024-04-14T15:25:38.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:38.814+0000 7fa0e77fe700 1 -- 172.21.15.90:0/3783327058 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa0f01a8f30 con 0x7fa0f01088a0 2024-04-14T15:25:38.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:38.814+0000 7fa0e77fe700 1 -- 172.21.15.90:0/3783327058 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3001446163 0 0) 0x7fa0dc0035b0 con 0x7fa0f010b750 2024-04-14T15:25:38.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:38.814+0000 7fa0e77fe700 1 -- 172.21.15.90:0/3783327058 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa0f01aa110 con 0x7fa0f010b750 2024-04-14T15:25:38.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:38.814+0000 7fa0e77fe700 1 -- 172.21.15.90:0/3783327058 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fa0e00018f0 con 0x7fa0f01088a0 2024-04-14T15:25:38.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:38.814+0000 7fa0e77fe700 1 -- 172.21.15.90:0/3783327058 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fa0ec001430 con 0x7fa0f010b750 2024-04-14T15:25:38.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:38.814+0000 7fa0e77fe700 1 -- 172.21.15.90:0/3783327058 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 252071284 0 0) 0x7fa0f010efa0 con 0x7fa0f010f320 2024-04-14T15:25:38.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:38.814+0000 7fa0e77fe700 1 -- 172.21.15.90:0/3783327058 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa0dc0035b0 con 0x7fa0f010f320 2024-04-14T15:25:38.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:38.814+0000 7fa0e77fe700 1 -- 172.21.15.90:0/3783327058 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fa0e8001bb0 con 0x7fa0f010f320 2024-04-14T15:25:38.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:38.814+0000 7fa0e77fe700 1 -- 172.21.15.90:0/3783327058 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 831741162 0 0) 0x7fa0f01a8f30 con 0x7fa0f01088a0 2024-04-14T15:25:38.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:38.814+0000 7fa0e77fe700 1 -- 172.21.15.90:0/3783327058 >> v1:172.21.15.90:6790/0 conn(0x7fa0f010b750 legacy=0x7fa0f010c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:38.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:38.814+0000 7fa0e77fe700 1 -- 172.21.15.90:0/3783327058 >> v1:172.21.15.92:6789/0 conn(0x7fa0f010f320 legacy=0x7fa0f01a6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:38.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:38.814+0000 7fa0e77fe700 1 -- 172.21.15.90:0/3783327058 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0f01ab2f0 con 0x7fa0f01088a0 2024-04-14T15:25:38.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:38.814+0000 7fa0f840a700 1 -- 172.21.15.90:0/3783327058 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa0f01a7f80 con 0x7fa0f01088a0 2024-04-14T15:25:38.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:38.814+0000 7fa0f840a700 10 client.38025.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:25:38.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:38.814+0000 7fa0f840a700 1 -- 172.21.15.90:0/3783327058 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa0f01a8440 con 0x7fa0f01088a0 2024-04-14T15:25:38.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:38.814+0000 7fa0f840a700 1 librados: init done 2024-04-14T15:25:38.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:38.814+0000 7fa0e77fe700 1 -- 172.21.15.90:0/3783327058 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fa0e0001d00 con 0x7fa0f01088a0 2024-04-14T15:25:38.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:38.814+0000 7fa0e77fe700 1 -- 172.21.15.90:0/3783327058 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fa0e0005d80 con 0x7fa0f01088a0 2024-04-14T15:25:38.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:38.815+0000 7fa0f840a700 1 -- 172.21.15.90:0/3783327058 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0d4005150 con 0x7fa0f01088a0 2024-04-14T15:25:38.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:38.815+0000 7fa0e77fe700 1 -- 172.21.15.90:0/3783327058 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fa0e0003950 con 0x7fa0f01088a0 2024-04-14T15:25:38.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:38.816+0000 7fa0e77fe700 1 -- 172.21.15.90:0/3783327058 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(792..792 src has 467..792) ==== 8816+0+0 (unknown 2469782290 0 0) 0x7fa0e008ad40 con 0x7fa0f01088a0 2024-04-14T15:25:38.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:38.816+0000 7fa0e77fe700 10 client.38025.objecter ms_dispatch 0x7fa0f0001550 osd_map(792..792 src has 467..792) 2024-04-14T15:25:38.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:38.816+0000 7fa0e77fe700 3 client.38025.objecter handle_osd_map got epochs [792,792] > 0 2024-04-14T15:25:38.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:38.816+0000 7fa0e77fe700 3 client.38025.objecter handle_osd_map decoding full epoch 792 2024-04-14T15:25:38.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:38.816+0000 7fa0e77fe700 20 client.38025.objecter dump_active .. 0 homeless 2024-04-14T15:25:38.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:38.816+0000 7fa0e77fe700 10 client.38025.objecter ms_handle_connect 0x7fa0dc071430 2024-04-14T15:25:38.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:38.820+0000 7fa0e77fe700 1 -- 172.21.15.90:0/3783327058 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fa0e00574e0 con 0x7fa0f01088a0 2024-04-14T15:25:38.831 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:38.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:38.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:25:38.889 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:25:38.934 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:25:38.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:38.958 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:38.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:25:39.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:39.040+0000 7fa0f840a700 1 -- 172.21.15.90:0/3783327058 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd reweight", "id": 4, "weight": 1.0} v 0) -- 0x7fa0d4005440 con 0x7fa0f01088a0 2024-04-14T15:25:39.060 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:39.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:39.084 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:39.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:39.186 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:25:39.210 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:39.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:39.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:25:39.268 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:39 smithi090 ceph-mon[30399]: pgmap v1172: 224 pgs: 1 active+remapped+backfill_toofull, 223 active+clean; 4.2 MiB data, 750 MiB used, 625 GiB / 626 GiB avail; 989 B/s rd, 3.1 KiB/s wr, 1 op/s; 62 KiB/s, 0 objects/s recovering 2024-04-14T15:25:39.268 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:39 smithi090 ceph-mon[30399]: osdmap e792: 8 total, 8 up, 7 in 2024-04-14T15:25:39.268 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:39 smithi090 ceph-mon[41670]: pgmap v1172: 224 pgs: 1 active+remapped+backfill_toofull, 223 active+clean; 4.2 MiB data, 750 MiB used, 625 GiB / 626 GiB avail; 989 B/s rd, 3.1 KiB/s wr, 1 op/s; 62 KiB/s, 0 objects/s recovering 2024-04-14T15:25:39.269 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:39 smithi090 ceph-mon[41670]: osdmap e792: 8 total, 8 up, 7 in 2024-04-14T15:25:39.311 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:39 smithi092 ceph-mon[33966]: pgmap v1172: 224 pgs: 1 active+remapped+backfill_toofull, 223 active+clean; 4.2 MiB data, 750 MiB used, 625 GiB / 626 GiB avail; 989 B/s rd, 3.1 KiB/s wr, 1 op/s; 62 KiB/s, 0 objects/s recovering 2024-04-14T15:25:39.311 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:39 smithi092 ceph-mon[33966]: osdmap e792: 8 total, 8 up, 7 in 2024-04-14T15:25:39.313 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:39.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:39.338 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:39.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:39.439 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:25:39.463 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:39.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:39.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:25:39.565 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:25:39.588 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:39.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:39.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:25:39.690 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:25:39.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:39.714 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:39.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:25:39.815 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:25:39.839 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:39.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:39.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:25:39.940 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:25:39.964 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:39.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:39.965 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:25:40.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:40.023+0000 7fa0e77fe700 1 -- 172.21.15.90:0/3783327058 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 4, "weight": 1.0}]=0 reweighted osd.4 to 1 (10000) v793) ==== 113+0+0 (unknown 3325870311 0 0) 0x7fa0e005b190 con 0x7fa0f01088a0 2024-04-14T15:25:40.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:40.027+0000 7fa0f840a700 10 librados: watch_flush enter 2024-04-14T15:25:40.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:40.027+0000 7fa0f840a700 10 librados: watch_flush exit 2024-04-14T15:25:40.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:40.027+0000 7fa0f840a700 20 client.38025.objecter shutdown clearing up homeless session... 2024-04-14T15:25:40.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:40.027+0000 7fa0f840a700 10 client.38025.objecter successfully canceled tick 2024-04-14T15:25:40.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:40.027+0000 7fa0f840a700 1 -- 172.21.15.90:0/3783327058 >> v1:172.21.15.90:6800/420935427 conn(0x7fa0dc071430 legacy=0x7fa0dc0738f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:40.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:40.027+0000 7fa0f840a700 1 -- 172.21.15.90:0/3783327058 >> v1:172.21.15.90:6789/0 conn(0x7fa0f01088a0 legacy=0x7fa0f010bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:40.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:40.028+0000 7fa0f840a700 1 -- 172.21.15.90:0/3783327058 shutdown_connections 2024-04-14T15:25:40.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:40.028+0000 7fa0f840a700 1 -- 172.21.15.90:0/3783327058 >> 172.21.15.90:0/3783327058 conn(0x7fa0f00fe390 msgr2=0x7fa0f01128b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:25:40.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:40.030+0000 7fa0f840a700 1 -- 172.21.15.90:0/3783327058 shutdown_connections 2024-04-14T15:25:40.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:40.030+0000 7fa0f840a700 1 -- 172.21.15.90:0/3783327058 wait complete. 2024-04-14T15:25:40.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:40.030+0000 7fa0f840a700 1 librados: shutdown 2024-04-14T15:25:40.031 INFO:teuthology.orchestra.run.smithi090.stderr:reweighted osd.4 to 1 (10000) 2024-04-14T15:25:40.065 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:40.089 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:40.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:40.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:40.141 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd unset noscrub 2024-04-14T15:25:40.190 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:25:40.214 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:40.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:40.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:25:40.316 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:25:40.327 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:40 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3783327058' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-14T15:25:40.327 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:40 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:25:40.327 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:40 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:25:40.328 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:40 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3783327058' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-14T15:25:40.328 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:40 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:25:40.328 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:40 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:25:40.341 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:40.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:40.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:25:40.351 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:40 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3783327058' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-14T15:25:40.351 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:40 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:25:40.351 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:40 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:25:40.443 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:25:40.467 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:40.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:40.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:25:40.569 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:25:40.588 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-04-14T15:25:40.588 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-04-14T15:25:40.588 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-04-14T15:25:40.588 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-04-14T15:25:40.588 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.588 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.36864.0:1 259.7s0 259:e732be06:::oid_WaitForSafePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e274)", 2024-04-14T15:25:40.588 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:15:41.861974+0000", 2024-04-14T15:25:40.588 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 598.72427782299997, 2024-04-14T15:25:40.588 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.009675004, 2024-04-14T15:25:40.588 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:25:40.588 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:25:40.589 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:25:40.589 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:25:40.589 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.36864", 2024-04-14T15:25:40.589 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/3688286685", 2024-04-14T15:25:40.589 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 1 2024-04-14T15:25:40.589 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.589 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:25:40.589 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.589 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:25:40.589 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:41.861974+0000", 2024-04-14T15:25:40.589 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:25:40.589 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.589 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.589 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:25:40.590 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:41.861974+0000", 2024-04-14T15:25:40.590 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:25:40.590 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.590 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.590 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:25:40.590 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:41.861976+0000", 2024-04-14T15:25:40.590 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5090000000000001e-06 2024-04-14T15:25:40.590 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.590 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.590 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:25:40.590 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:41.861982+0000", 2024-04-14T15:25:40.590 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6659999999999996e-06 2024-04-14T15:25:40.590 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.591 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.591 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:25:40.591 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:41.861990+0000", 2024-04-14T15:25:40.591 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2789999999999998e-06 2024-04-14T15:25:40.591 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.591 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.591 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:25:40.591 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:41.861999+0000", 2024-04-14T15:25:40.591 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0040000000000005e-06 2024-04-14T15:25:40.591 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.591 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.591 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:25:40.591 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:41.862045+0000", 2024-04-14T15:25:40.591 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5679000000000001e-05 2024-04-14T15:25:40.591 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.592 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.592 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:25:40.592 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:41.862048+0000", 2024-04-14T15:25:40.592 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0639999999999998e-06 2024-04-14T15:25:40.592 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.592 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.592 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:25:40.592 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:42.869700+0000", 2024-04-14T15:25:40.592 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0076524060000001 2024-04-14T15:25:40.592 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.592 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.592 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:25:40.592 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:42.869825+0000", 2024-04-14T15:25:40.592 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012442599999999999 2024-04-14T15:25:40.592 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.593 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.593 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_started", 2024-04-14T15:25:40.593 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:42.870232+0000", 2024-04-14T15:25:40.593 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00040733700000000002 2024-04-14T15:25:40.593 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.593 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.593 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_committed", 2024-04-14T15:25:40.593 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:42.870790+0000", 2024-04-14T15:25:40.593 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00055832199999999998 2024-04-14T15:25:40.593 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.593 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.593 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:25:40.593 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:42.871585+0000", 2024-04-14T15:25:40.593 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00079456100000000005 2024-04-14T15:25:40.594 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.594 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.594 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:25:40.594 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:15:42.871649+0000", 2024-04-14T15:25:40.594 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4751000000000004e-05 2024-04-14T15:25:40.594 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:25:40.594 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:25:40.594 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:25:40.594 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.594 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.594 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.37038.0:1 277.5s1 277:ab97ba35:::nonexistent:head [read 0~128] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e326)", 2024-04-14T15:25:40.594 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:16:33.929634+0000", 2024-04-14T15:25:40.594 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 546.65661846700004, 2024-04-14T15:25:40.594 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.473047293, 2024-04-14T15:25:40.594 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:25:40.595 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:25:40.595 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-04-14T15:25:40.595 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:25:40.595 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.37038", 2024-04-14T15:25:40.595 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/1251688097", 2024-04-14T15:25:40.595 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 1 2024-04-14T15:25:40.595 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.595 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:25:40.595 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.595 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:25:40.595 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:16:33.929634+0000", 2024-04-14T15:25:40.595 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:25:40.595 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.595 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.596 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:25:40.596 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:16:33.929634+0000", 2024-04-14T15:25:40.596 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:25:40.596 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.596 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.596 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:25:40.596 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:16:33.929636+0000", 2024-04-14T15:25:40.596 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.996e-06 2024-04-14T15:25:40.596 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.596 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.596 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:25:40.596 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:16:33.929642+0000", 2024-04-14T15:25:40.596 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3300000000000004e-06 2024-04-14T15:25:40.596 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.597 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.597 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:25:40.597 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:16:33.929653+0000", 2024-04-14T15:25:40.597 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1212e-05 2024-04-14T15:25:40.597 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.597 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.597 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:25:40.597 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:16:33.929664+0000", 2024-04-14T15:25:40.597 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.024e-05 2024-04-14T15:25:40.597 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.597 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.597 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:25:40.597 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:16:33.929724+0000", 2024-04-14T15:25:40.597 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0699999999999998e-05 2024-04-14T15:25:40.597 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.597 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.598 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-04-14T15:25:40.598 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:16:33.929728+0000", 2024-04-14T15:25:40.598 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.7759999999999999e-06 2024-04-14T15:25:40.598 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.598 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.598 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:25:40.598 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:16:35.402540+0000", 2024-04-14T15:25:40.598 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4728114589999999 2024-04-14T15:25:40.598 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.598 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.598 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:25:40.598 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:16:35.402681+0000", 2024-04-14T15:25:40.598 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014158000000000001 2024-04-14T15:25:40.598 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:25:40.599 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:25:40.599 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:25:40.599 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.599 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.599 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.49576.0:14 324.1c 324:38000000:%01:::head [pgnls start_epoch 651] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e651)", 2024-04-14T15:25:40.599 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:23:06.989565+0000", 2024-04-14T15:25:40.599 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 153.59668721599999, 2024-04-14T15:25:40.599 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00239339, 2024-04-14T15:25:40.599 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:25:40.599 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:25:40.599 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-04-14T15:25:40.599 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:25:40.599 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.49576", 2024-04-14T15:25:40.599 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/2125385135", 2024-04-14T15:25:40.599 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 14 2024-04-14T15:25:40.600 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.600 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:25:40.600 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.600 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:25:40.600 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:06.989565+0000", 2024-04-14T15:25:40.600 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:25:40.600 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.600 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.600 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:25:40.600 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:06.989565+0000", 2024-04-14T15:25:40.600 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:25:40.600 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.600 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.600 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:25:40.601 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:06.989567+0000", 2024-04-14T15:25:40.601 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9139999999999998e-06 2024-04-14T15:25:40.601 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.601 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.601 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:25:40.601 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:06.989574+0000", 2024-04-14T15:25:40.601 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.7850000000000003e-06 2024-04-14T15:25:40.601 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.601 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.601 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:25:40.602 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:06.989583+0000", 2024-04-14T15:25:40.602 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.1740000000000006e-06 2024-04-14T15:25:40.602 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.602 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.602 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:25:40.602 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:06.989590+0000", 2024-04-14T15:25:40.602 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0829999999999998e-06 2024-04-14T15:25:40.602 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.602 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.603 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:25:40.603 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:06.989642+0000", 2024-04-14T15:25:40.603 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.1972000000000003e-05 2024-04-14T15:25:40.603 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.603 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.603 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op must wait for map", 2024-04-14T15:25:40.603 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:06.989646+0000", 2024-04-14T15:25:40.603 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3229999999999999e-06 2024-04-14T15:25:40.603 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.603 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.603 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:25:40.603 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:06.991816+0000", 2024-04-14T15:25:40.603 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002169327 2024-04-14T15:25:40.603 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.603 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.604 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:25:40.604 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:06.991853+0000", 2024-04-14T15:25:40.604 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6981999999999999e-05 2024-04-14T15:25:40.604 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.604 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.604 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:25:40.604 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:06.991958+0000", 2024-04-14T15:25:40.604 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010582999999999999 2024-04-14T15:25:40.604 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:25:40.604 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:25:40.604 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:25:40.604 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.604 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.604 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.49576.0:48 324.1c 324:38000000::::head [pgnls start_epoch 653] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e653)", 2024-04-14T15:25:40.604 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:23:08.988838+0000", 2024-04-14T15:25:40.605 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 151.59741441599999, 2024-04-14T15:25:40.605 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0024608820000000002, 2024-04-14T15:25:40.605 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:25:40.605 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:25:40.605 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-04-14T15:25:40.605 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:25:40.605 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.49576", 2024-04-14T15:25:40.605 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/2125385135", 2024-04-14T15:25:40.605 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 48 2024-04-14T15:25:40.605 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.605 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:25:40.605 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.605 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:25:40.605 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:08.988838+0000", 2024-04-14T15:25:40.606 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:25:40.606 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.606 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.606 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:25:40.606 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:08.988838+0000", 2024-04-14T15:25:40.606 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:25:40.606 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.606 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.606 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:25:40.606 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:08.988839+0000", 2024-04-14T15:25:40.606 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.249e-06 2024-04-14T15:25:40.606 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.606 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.606 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:25:40.607 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:08.988844+0000", 2024-04-14T15:25:40.607 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.1730000000000003e-06 2024-04-14T15:25:40.607 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.607 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.607 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:25:40.607 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:08.988853+0000", 2024-04-14T15:25:40.607 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.4400000000000005e-06 2024-04-14T15:25:40.607 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.607 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.607 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:25:40.607 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:08.988862+0000", 2024-04-14T15:25:40.607 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8950000000000004e-06 2024-04-14T15:25:40.607 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.607 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.608 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:25:40.608 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:08.988918+0000", 2024-04-14T15:25:40.608 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6589000000000003e-05 2024-04-14T15:25:40.608 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.608 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.608 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op must wait for map", 2024-04-14T15:25:40.608 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:08.988925+0000", 2024-04-14T15:25:40.608 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2790000000000004e-06 2024-04-14T15:25:40.608 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.608 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.608 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:25:40.608 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:08.991134+0000", 2024-04-14T15:25:40.608 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002209061 2024-04-14T15:25:40.608 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.609 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.609 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:25:40.609 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:08.991169+0000", 2024-04-14T15:25:40.609 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5559000000000003e-05 2024-04-14T15:25:40.609 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.609 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.609 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:25:40.609 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:08.991299+0000", 2024-04-14T15:25:40.609 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012963699999999999 2024-04-14T15:25:40.609 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:25:40.609 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:25:40.609 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:25:40.609 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.609 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.610 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.49576.0:246 323.5s0 323:a73618ef:test-rados-api-smithi090-93652-117::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e664)", 2024-04-14T15:25:40.610 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:23:20.019048+0000", 2024-04-14T15:25:40.610 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 140.567204283, 2024-04-14T15:25:40.610 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.012425383, 2024-04-14T15:25:40.610 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:25:40.610 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:25:40.610 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:25:40.610 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:25:40.610 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.49576", 2024-04-14T15:25:40.610 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/2125385135", 2024-04-14T15:25:40.610 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 246 2024-04-14T15:25:40.610 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.610 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:25:40.610 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.611 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:25:40.611 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:20.019048+0000", 2024-04-14T15:25:40.611 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:25:40.611 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.611 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.611 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:25:40.611 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:20.019048+0000", 2024-04-14T15:25:40.611 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:25:40.611 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.611 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.611 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:25:40.611 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:20.019050+0000", 2024-04-14T15:25:40.611 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6390000000000001e-06 2024-04-14T15:25:40.611 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.612 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.612 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:25:40.612 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:20.019057+0000", 2024-04-14T15:25:40.612 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1509999999999998e-06 2024-04-14T15:25:40.612 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.612 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.612 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:25:40.612 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:20.019067+0000", 2024-04-14T15:25:40.612 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0448999999999999e-05 2024-04-14T15:25:40.612 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.612 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.612 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:25:40.612 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:20.019077+0000", 2024-04-14T15:25:40.612 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.7380000000000006e-06 2024-04-14T15:25:40.612 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.613 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.613 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:25:40.613 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:20.019199+0000", 2024-04-14T15:25:40.613 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012173899999999999 2024-04-14T15:25:40.613 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.613 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.613 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op must wait for map", 2024-04-14T15:25:40.613 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:20.019206+0000", 2024-04-14T15:25:40.613 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9399999999999996e-06 2024-04-14T15:25:40.613 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.613 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.613 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:25:40.613 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:20.020444+0000", 2024-04-14T15:25:40.613 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0012387710000000001 2024-04-14T15:25:40.614 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.614 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.614 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:25:40.614 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:20.020571+0000", 2024-04-14T15:25:40.614 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012692500000000001 2024-04-14T15:25:40.614 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.614 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.614 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_started", 2024-04-14T15:25:40.614 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:20.021028+0000", 2024-04-14T15:25:40.614 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00045696099999999998 2024-04-14T15:25:40.614 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.614 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.614 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_committed", 2024-04-14T15:25:40.614 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:20.021465+0000", 2024-04-14T15:25:40.615 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00043707700000000002 2024-04-14T15:25:40.615 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.615 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.615 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:25:40.615 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:20.031392+0000", 2024-04-14T15:25:40.615 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0099261850000000006 2024-04-14T15:25:40.615 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.615 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.615 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:25:40.615 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:20.031473+0000", 2024-04-14T15:25:40.615 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1808e-05 2024-04-14T15:25:40.615 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:25:40.615 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:25:40.615 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:25:40.615 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.616 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.616 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.49576.0:251 323.5s0 323:a73618ef:test-rados-api-smithi090-93652-117::foo:head [writefull 0~5] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e665)", 2024-04-14T15:25:40.616 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:23:20.980644+0000", 2024-04-14T15:25:40.616 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 139.60560784699999, 2024-04-14T15:25:40.616 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0027989730000000002, 2024-04-14T15:25:40.616 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:25:40.616 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:25:40.616 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:25:40.616 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:25:40.616 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.49576", 2024-04-14T15:25:40.616 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/2125385135", 2024-04-14T15:25:40.616 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 251 2024-04-14T15:25:40.616 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.616 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:25:40.616 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.617 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:25:40.617 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:20.980644+0000", 2024-04-14T15:25:40.617 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:25:40.617 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.617 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.617 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:25:40.617 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:20.980644+0000", 2024-04-14T15:25:40.617 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:25:40.617 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.617 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.617 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:25:40.617 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:20.980646+0000", 2024-04-14T15:25:40.617 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0080000000000001e-06 2024-04-14T15:25:40.617 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.617 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.618 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:25:40.618 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:20.980655+0000", 2024-04-14T15:25:40.618 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3699999999999995e-06 2024-04-14T15:25:40.618 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.618 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.618 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:25:40.618 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:20.980664+0000", 2024-04-14T15:25:40.618 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.064e-06 2024-04-14T15:25:40.618 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.618 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.618 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:25:40.618 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:20.980674+0000", 2024-04-14T15:25:40.618 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0395e-05 2024-04-14T15:25:40.618 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.619 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.619 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:25:40.619 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:20.980709+0000", 2024-04-14T15:25:40.619 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4619000000000002e-05 2024-04-14T15:25:40.619 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.619 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.619 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:25:40.619 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:20.980772+0000", 2024-04-14T15:25:40.619 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2700999999999995e-05 2024-04-14T15:25:40.619 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.619 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.619 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_started", 2024-04-14T15:25:40.619 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:20.981163+0000", 2024-04-14T15:25:40.619 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00039100099999999998 2024-04-14T15:25:40.619 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.620 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.620 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_committed", 2024-04-14T15:25:40.620 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:20.981679+0000", 2024-04-14T15:25:40.620 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00051639100000000005 2024-04-14T15:25:40.620 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.620 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.620 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:25:40.620 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:20.983357+0000", 2024-04-14T15:25:40.620 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0016779570000000001 2024-04-14T15:25:40.620 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.620 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.620 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:25:40.620 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:23:20.983443+0000", 2024-04-14T15:25:40.620 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6466999999999997e-05 2024-04-14T15:25:40.620 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:25:40.621 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:25:40.621 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:25:40.621 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.621 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.621 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.49576.0:647 329.3 329:c5c461f0:test-rados-api-smithi090-93652-123::foo:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e704)", 2024-04-14T15:25:40.621 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:24:06.000092+0000", 2024-04-14T15:25:40.621 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 94.586160766000006, 2024-04-14T15:25:40.621 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.010687972, 2024-04-14T15:25:40.621 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:25:40.621 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:25:40.621 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-04-14T15:25:40.621 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:25:40.621 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.49576", 2024-04-14T15:25:40.621 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/2125385135", 2024-04-14T15:25:40.622 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 647 2024-04-14T15:25:40.622 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.622 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:25:40.622 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.622 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:25:40.622 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:24:06.000092+0000", 2024-04-14T15:25:40.622 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:25:40.622 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.622 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.622 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:25:40.622 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:24:06.000092+0000", 2024-04-14T15:25:40.622 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:25:40.622 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.622 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.623 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:25:40.623 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:24:06.000095+0000", 2024-04-14T15:25:40.623 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4139999999999998e-06 2024-04-14T15:25:40.623 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.623 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.623 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:25:40.623 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:24:06.000106+0000", 2024-04-14T15:25:40.623 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1415e-05 2024-04-14T15:25:40.623 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.623 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.623 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:25:40.623 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:24:06.000124+0000", 2024-04-14T15:25:40.623 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8014000000000001e-05 2024-04-14T15:25:40.623 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.624 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.624 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:25:40.624 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:24:06.000153+0000", 2024-04-14T15:25:40.624 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8155000000000001e-05 2024-04-14T15:25:40.624 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.624 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.624 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:25:40.624 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:24:06.000240+0000", 2024-04-14T15:25:40.624 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.7280999999999993e-05 2024-04-14T15:25:40.624 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.624 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.624 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op must wait for map", 2024-04-14T15:25:40.624 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:24:06.000247+0000", 2024-04-14T15:25:40.624 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2479999999999999e-06 2024-04-14T15:25:40.624 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.625 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.625 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:25:40.625 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:24:06.002506+0000", 2024-04-14T15:25:40.625 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0022589659999999998 2024-04-14T15:25:40.625 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.625 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.625 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for blocked object", 2024-04-14T15:25:40.625 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:24:06.002747+0000", 2024-04-14T15:25:40.625 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00024048599999999999 2024-04-14T15:25:40.625 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.625 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.625 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:25:40.625 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:24:06.008170+0000", 2024-04-14T15:25:40.625 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0054234950000000004 2024-04-14T15:25:40.625 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.626 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.626 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for rw locks", 2024-04-14T15:25:40.626 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:24:06.008274+0000", 2024-04-14T15:25:40.626 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000104289 2024-04-14T15:25:40.626 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.626 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.626 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:25:40.626 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:24:06.008638+0000", 2024-04-14T15:25:40.626 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00036393999999999999 2024-04-14T15:25:40.626 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.626 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.626 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:25:40.626 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:24:06.008698+0000", 2024-04-14T15:25:40.626 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9781999999999998e-05 2024-04-14T15:25:40.627 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.627 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.627 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:25:40.627 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:24:06.010780+0000", 2024-04-14T15:25:40.627 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0020814869999999999 2024-04-14T15:25:40.627 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:25:40.627 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:25:40.627 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:25:40.627 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.627 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.627 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.49576.0:686 329.3 329:c5c461f0:test-rados-api-smithi090-93652-123::foo:head [cache-flush] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e704)", 2024-04-14T15:25:40.627 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:24:06.029573+0000", 2024-04-14T15:25:40.627 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 94.556678989999995, 2024-04-14T15:25:40.627 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0023573130000000002, 2024-04-14T15:25:40.627 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:25:40.628 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:25:40.628 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-04-14T15:25:40.628 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:25:40.628 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.49576", 2024-04-14T15:25:40.628 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/2125385135", 2024-04-14T15:25:40.628 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 686 2024-04-14T15:25:40.628 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.628 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:25:40.628 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.628 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:25:40.628 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:24:06.029573+0000", 2024-04-14T15:25:40.628 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:25:40.628 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:40.629 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:40.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:25:40.630 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.631 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:25:40.631 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:24:06.029573+0000", 2024-04-14T15:25:40.631 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:25:40.631 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.631 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.631 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:25:40.631 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:24:06.029574+0000", 2024-04-14T15:25:40.631 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.9100000000000002e-07 2024-04-14T15:25:40.631 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.631 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.631 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:25:40.631 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:24:06.029578+0000", 2024-04-14T15:25:40.631 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.7299999999999999e-06 2024-04-14T15:25:40.631 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.632 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.632 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:25:40.632 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:24:06.029586+0000", 2024-04-14T15:25:40.632 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1580000000000005e-06 2024-04-14T15:25:40.632 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.632 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.632 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:25:40.632 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:24:06.029594+0000", 2024-04-14T15:25:40.632 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.9189999999999999e-06 2024-04-14T15:25:40.632 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.632 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.632 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:25:40.632 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:24:06.029627+0000", 2024-04-14T15:25:40.632 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2648000000000003e-05 2024-04-14T15:25:40.633 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.633 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.633 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:25:40.633 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:24:06.029690+0000", 2024-04-14T15:25:40.633 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3268999999999993e-05 2024-04-14T15:25:40.633 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.633 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.633 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:25:40.633 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:24:06.031815+0000", 2024-04-14T15:25:40.633 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0021251759999999999 2024-04-14T15:25:40.633 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.633 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.633 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:25:40.633 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:24:06.031871+0000", 2024-04-14T15:25:40.634 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5906000000000003e-05 2024-04-14T15:25:40.634 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.634 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.634 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:25:40.634 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:24:06.031931+0000", 2024-04-14T15:25:40.634 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9616e-05 2024-04-14T15:25:40.634 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:25:40.634 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:25:40.634 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:25:40.634 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.634 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.634 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.49576.0:1295 333.0 333:00000000:test-rados-api-smithi090-93652-131:::head [pgnls start_epoch 742] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e742)", 2024-04-14T15:25:40.634 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:24:46.009957+0000", 2024-04-14T15:25:40.634 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 54.576295402, 2024-04-14T15:25:40.635 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0023940110000000001, 2024-04-14T15:25:40.635 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:25:40.635 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:25:40.635 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-04-14T15:25:40.635 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:25:40.635 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.49576", 2024-04-14T15:25:40.635 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/2125385135", 2024-04-14T15:25:40.635 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 1295 2024-04-14T15:25:40.635 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.635 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:25:40.635 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.635 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:25:40.635 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:24:46.009957+0000", 2024-04-14T15:25:40.635 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:25:40.635 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.636 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.636 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:25:40.636 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:24:46.009957+0000", 2024-04-14T15:25:40.636 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:25:40.636 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.636 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.636 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:25:40.636 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:24:46.009960+0000", 2024-04-14T15:25:40.636 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9660000000000001e-06 2024-04-14T15:25:40.636 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.636 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.636 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:25:40.636 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:24:46.009972+0000", 2024-04-14T15:25:40.636 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1759e-05 2024-04-14T15:25:40.636 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.637 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.637 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:25:40.637 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:24:46.009990+0000", 2024-04-14T15:25:40.637 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8780999999999999e-05 2024-04-14T15:25:40.637 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.637 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.637 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:25:40.637 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:24:46.010018+0000", 2024-04-14T15:25:40.637 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.7968999999999999e-05 2024-04-14T15:25:40.637 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.637 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.637 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:25:40.637 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:24:46.010115+0000", 2024-04-14T15:25:40.637 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.6497999999999997e-05 2024-04-14T15:25:40.638 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.638 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.638 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op must wait for map", 2024-04-14T15:25:40.638 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:24:46.010121+0000", 2024-04-14T15:25:40.638 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0449999999999997e-06 2024-04-14T15:25:40.638 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.638 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.638 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:25:40.638 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:24:46.011993+0000", 2024-04-14T15:25:40.638 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001871972 2024-04-14T15:25:40.638 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.638 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.638 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:25:40.638 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:24:46.012032+0000", 2024-04-14T15:25:40.638 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9388999999999998e-05 2024-04-14T15:25:40.639 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.639 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.639 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:25:40.639 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:24:46.012351+0000", 2024-04-14T15:25:40.639 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00031863200000000001 2024-04-14T15:25:40.639 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:25:40.639 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:25:40.639 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:25:40.639 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.639 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.639 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.49576.0:1458 333.0 333:00000000::::head [pgnls start_epoch 744] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e744)", 2024-04-14T15:25:40.639 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:24:48.011260+0000", 2024-04-14T15:25:40.639 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 52.574991986000001, 2024-04-14T15:25:40.639 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0024709670000000001, 2024-04-14T15:25:40.639 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:25:40.640 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:25:40.640 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-04-14T15:25:40.640 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:25:40.640 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.49576", 2024-04-14T15:25:40.640 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/2125385135", 2024-04-14T15:25:40.640 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 1458 2024-04-14T15:25:40.640 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.640 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:25:40.640 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.640 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:25:40.640 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:24:48.011260+0000", 2024-04-14T15:25:40.640 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:25:40.640 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.640 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.640 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:25:40.641 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:24:48.011260+0000", 2024-04-14T15:25:40.641 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:25:40.641 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.641 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.641 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:25:40.641 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:24:48.011263+0000", 2024-04-14T15:25:40.641 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.932e-06 2024-04-14T15:25:40.641 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.641 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.641 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:25:40.641 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:24:48.011273+0000", 2024-04-14T15:25:40.641 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.9729999999999992e-06 2024-04-14T15:25:40.641 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.641 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.642 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:25:40.642 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:24:48.011289+0000", 2024-04-14T15:25:40.642 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.558e-05 2024-04-14T15:25:40.642 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.642 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.642 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:25:40.642 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:24:48.011302+0000", 2024-04-14T15:25:40.642 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3628999999999999e-05 2024-04-14T15:25:40.642 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.642 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.642 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:25:40.642 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:24:48.011372+0000", 2024-04-14T15:25:40.642 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9203999999999997e-05 2024-04-14T15:25:40.642 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.643 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.643 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op must wait for map", 2024-04-14T15:25:40.643 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:24:48.011378+0000", 2024-04-14T15:25:40.643 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.6630000000000004e-06 2024-04-14T15:25:40.643 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.643 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.643 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:25:40.643 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:24:48.013375+0000", 2024-04-14T15:25:40.643 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0019965310000000002 2024-04-14T15:25:40.643 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.643 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.643 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:25:40.643 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:24:48.013429+0000", 2024-04-14T15:25:40.643 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3758000000000003e-05 2024-04-14T15:25:40.643 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.644 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.644 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:25:40.644 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:24:48.013731+0000", 2024-04-14T15:25:40.644 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00030269699999999999 2024-04-14T15:25:40.644 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:25:40.644 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:25:40.644 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:25:40.644 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.644 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.644 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.49576.0:1682 335.3 335:c3dc6263:test-rados-api-smithi090-93652-135::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e763)", 2024-04-14T15:25:40.644 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:25:07.012989+0000", 2024-04-14T15:25:40.644 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 33.573262876000001, 2024-04-14T15:25:40.644 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0040546169999999999, 2024-04-14T15:25:40.644 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:25:40.645 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:25:40.645 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:25:40.645 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:25:40.645 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.49576", 2024-04-14T15:25:40.645 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/2125385135", 2024-04-14T15:25:40.645 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 1682 2024-04-14T15:25:40.645 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.645 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:25:40.645 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.645 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:25:40.645 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.012989+0000", 2024-04-14T15:25:40.645 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:25:40.645 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.645 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.646 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:25:40.646 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.012989+0000", 2024-04-14T15:25:40.646 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:25:40.646 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.646 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.646 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:25:40.646 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.012992+0000", 2024-04-14T15:25:40.646 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.092e-06 2024-04-14T15:25:40.646 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.646 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.646 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:25:40.646 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.013001+0000", 2024-04-14T15:25:40.646 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.1050000000000001e-06 2024-04-14T15:25:40.646 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.647 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.647 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:25:40.647 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.013013+0000", 2024-04-14T15:25:40.647 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2282e-05 2024-04-14T15:25:40.647 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.647 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.647 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:25:40.647 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.013024+0000", 2024-04-14T15:25:40.647 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.06e-05 2024-04-14T15:25:40.647 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.647 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.647 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:25:40.647 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.013067+0000", 2024-04-14T15:25:40.647 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3322000000000003e-05 2024-04-14T15:25:40.648 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.648 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.648 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for blocked object", 2024-04-14T15:25:40.648 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.013226+0000", 2024-04-14T15:25:40.648 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000158756 2024-04-14T15:25:40.648 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.648 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.648 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:25:40.648 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.015632+0000", 2024-04-14T15:25:40.648 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0024059910000000001 2024-04-14T15:25:40.648 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.648 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.648 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:25:40.648 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.015730+0000", 2024-04-14T15:25:40.648 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8014e-05 2024-04-14T15:25:40.649 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.649 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.649 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,5", 2024-04-14T15:25:40.649 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.015868+0000", 2024-04-14T15:25:40.649 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013850599999999999 2024-04-14T15:25:40.649 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.649 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.649 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-04-14T15:25:40.649 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.016225+0000", 2024-04-14T15:25:40.649 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000357392 2024-04-14T15:25:40.649 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.649 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.649 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:25:40.649 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.016444+0000", 2024-04-14T15:25:40.650 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000218495 2024-04-14T15:25:40.650 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.650 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.650 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:25:40.650 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.016898+0000", 2024-04-14T15:25:40.650 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00045367999999999998 2024-04-14T15:25:40.650 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.650 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.650 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-04-14T15:25:40.650 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.017003+0000", 2024-04-14T15:25:40.650 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000105368 2024-04-14T15:25:40.650 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.650 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.650 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:25:40.650 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.017044+0000", 2024-04-14T15:25:40.651 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1013999999999997e-05 2024-04-14T15:25:40.651 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:25:40.651 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:25:40.651 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:25:40.651 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.651 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.651 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.49576.0:1683 335.3 335:c3dc6263:test-rados-api-smithi090-93652-135::foo:head [cache-flush] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e763)", 2024-04-14T15:25:40.651 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:25:07.017208+0000", 2024-04-14T15:25:40.651 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 33.569044544999997, 2024-04-14T15:25:40.651 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0026517979999999999, 2024-04-14T15:25:40.651 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:25:40.651 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:25:40.651 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-04-14T15:25:40.651 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:25:40.652 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.49576", 2024-04-14T15:25:40.652 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/2125385135", 2024-04-14T15:25:40.652 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 1683 2024-04-14T15:25:40.652 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.652 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:25:40.652 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.652 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:25:40.652 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.017208+0000", 2024-04-14T15:25:40.652 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:25:40.652 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.652 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.652 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:25:40.652 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.017208+0000", 2024-04-14T15:25:40.652 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:25:40.653 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.653 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.653 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:25:40.653 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.017209+0000", 2024-04-14T15:25:40.653 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2419999999999999e-06 2024-04-14T15:25:40.653 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.653 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.653 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:25:40.653 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.017214+0000", 2024-04-14T15:25:40.653 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2490000000000001e-06 2024-04-14T15:25:40.653 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.653 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.653 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:25:40.653 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.017229+0000", 2024-04-14T15:25:40.654 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4513e-05 2024-04-14T15:25:40.654 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.654 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.654 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:25:40.654 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.017237+0000", 2024-04-14T15:25:40.654 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1580000000000005e-06 2024-04-14T15:25:40.654 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.654 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.654 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:25:40.654 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.017312+0000", 2024-04-14T15:25:40.654 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.4777999999999995e-05 2024-04-14T15:25:40.654 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.654 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.654 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:25:40.654 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.017399+0000", 2024-04-14T15:25:40.655 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.7571000000000006e-05 2024-04-14T15:25:40.655 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.655 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.655 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:25:40.655 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.019745+0000", 2024-04-14T15:25:40.655 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0023457159999999999 2024-04-14T15:25:40.655 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.655 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.655 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:25:40.655 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.019807+0000", 2024-04-14T15:25:40.655 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1983999999999995e-05 2024-04-14T15:25:40.655 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.655 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.655 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:25:40.656 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.019860+0000", 2024-04-14T15:25:40.656 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2587000000000003e-05 2024-04-14T15:25:40.656 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:25:40.656 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:25:40.656 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:25:40.656 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.656 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.656 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.49576.0:1684 335.3 335:c3dc6263:test-rados-api-smithi090-93652-135::foo:head [cache-flush] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e763)", 2024-04-14T15:25:40.656 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:25:07.017255+0000", 2024-04-14T15:25:40.656 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 33.568997052999997, 2024-04-14T15:25:40.656 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0027135290000000001, 2024-04-14T15:25:40.656 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:25:40.656 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:25:40.656 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-04-14T15:25:40.656 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:25:40.657 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.49576", 2024-04-14T15:25:40.657 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/2125385135", 2024-04-14T15:25:40.657 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 1684 2024-04-14T15:25:40.657 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.657 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:25:40.657 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.657 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:25:40.657 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.017255+0000", 2024-04-14T15:25:40.657 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:25:40.657 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.657 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.657 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:25:40.657 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.017255+0000", 2024-04-14T15:25:40.657 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:25:40.658 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.658 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.658 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:25:40.658 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.017256+0000", 2024-04-14T15:25:40.658 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5099999999999999e-07 2024-04-14T15:25:40.658 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.658 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.658 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:25:40.658 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.017259+0000", 2024-04-14T15:25:40.658 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.3419999999999999e-06 2024-04-14T15:25:40.658 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.658 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.658 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:25:40.659 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.017266+0000", 2024-04-14T15:25:40.659 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4670000000000001e-06 2024-04-14T15:25:40.659 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.659 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.659 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:25:40.659 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.017272+0000", 2024-04-14T15:25:40.659 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9100000000000002e-06 2024-04-14T15:25:40.659 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.659 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.659 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:25:40.659 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.017608+0000", 2024-04-14T15:25:40.659 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00033668899999999999 2024-04-14T15:25:40.659 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.659 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.660 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for blocked object", 2024-04-14T15:25:40.660 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.017652+0000", 2024-04-14T15:25:40.660 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3606999999999998e-05 2024-04-14T15:25:40.660 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.660 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.660 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:25:40.660 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.019881+0000", 2024-04-14T15:25:40.660 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0022289390000000001 2024-04-14T15:25:40.660 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.660 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.660 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:25:40.660 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.019928+0000", 2024-04-14T15:25:40.660 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6941999999999999e-05 2024-04-14T15:25:40.660 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.660 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.661 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:25:40.661 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.019969+0000", 2024-04-14T15:25:40.661 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0881999999999999e-05 2024-04-14T15:25:40.661 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:25:40.661 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:25:40.661 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:25:40.661 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.661 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.661 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.49576.0:1686 335.3 335:c3dc6263:test-rados-api-smithi090-93652-135::foo:head [cache-flush] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e763)", 2024-04-14T15:25:40.661 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:25:07.021047+0000", 2024-04-14T15:25:40.661 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 33.565205028000001, 2024-04-14T15:25:40.661 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0021143109999999998, 2024-04-14T15:25:40.661 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:25:40.661 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:25:40.662 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-04-14T15:25:40.662 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:25:40.662 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.49576", 2024-04-14T15:25:40.662 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/2125385135", 2024-04-14T15:25:40.662 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 1686 2024-04-14T15:25:40.662 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.662 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:25:40.662 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.662 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:25:40.662 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.021047+0000", 2024-04-14T15:25:40.662 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:25:40.662 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.662 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.662 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:25:40.663 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.021047+0000", 2024-04-14T15:25:40.663 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:25:40.663 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.663 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.663 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:25:40.663 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.021048+0000", 2024-04-14T15:25:40.663 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4199999999999995e-07 2024-04-14T15:25:40.663 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.663 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.663 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:25:40.663 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.021051+0000", 2024-04-14T15:25:40.663 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.632e-06 2024-04-14T15:25:40.663 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.663 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.663 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:25:40.664 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.021057+0000", 2024-04-14T15:25:40.664 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5849999999999996e-06 2024-04-14T15:25:40.664 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.664 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.664 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:25:40.664 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.021062+0000", 2024-04-14T15:25:40.664 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.391e-06 2024-04-14T15:25:40.664 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.664 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.664 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:25:40.664 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.021099+0000", 2024-04-14T15:25:40.664 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.7616e-05 2024-04-14T15:25:40.664 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.664 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.665 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:25:40.665 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.021181+0000", 2024-04-14T15:25:40.665 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1621999999999999e-05 2024-04-14T15:25:40.665 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.665 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.665 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:25:40.665 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.023026+0000", 2024-04-14T15:25:40.665 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001845361 2024-04-14T15:25:40.665 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.665 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.665 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:25:40.665 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.023103+0000", 2024-04-14T15:25:40.665 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.6457000000000003e-05 2024-04-14T15:25:40.665 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.666 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.666 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:25:40.666 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.023162+0000", 2024-04-14T15:25:40.666 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9005000000000002e-05 2024-04-14T15:25:40.666 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:25:40.666 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:25:40.666 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:25:40.666 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.666 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.666 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.49576.0:1687 335.3 335:c3dc6263:test-rados-api-smithi090-93652-135::foo:head [cache-try-flush] snapc 0=[] ondisk+read+skiprwlocks+ignore_overlay+known_if_redirected+supports_pool_eio e763)", 2024-04-14T15:25:40.666 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:25:07.021073+0000", 2024-04-14T15:25:40.666 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 33.565179010000001, 2024-04-14T15:25:40.666 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0022501309999999998, 2024-04-14T15:25:40.666 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:25:40.667 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:25:40.667 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-04-14T15:25:40.667 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:25:40.667 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.49576", 2024-04-14T15:25:40.667 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/2125385135", 2024-04-14T15:25:40.667 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 1687 2024-04-14T15:25:40.667 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.667 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:25:40.667 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.667 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:25:40.667 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.021073+0000", 2024-04-14T15:25:40.667 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:25:40.667 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.667 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.668 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:25:40.668 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.021073+0000", 2024-04-14T15:25:40.668 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:25:40.668 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.668 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.668 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:25:40.668 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.021074+0000", 2024-04-14T15:25:40.668 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.7800000000000002e-07 2024-04-14T15:25:40.668 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.668 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.668 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:25:40.668 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.021076+0000", 2024-04-14T15:25:40.668 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9520000000000001e-06 2024-04-14T15:25:40.668 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.668 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.669 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:25:40.669 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.021079+0000", 2024-04-14T15:25:40.669 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5130000000000001e-06 2024-04-14T15:25:40.669 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.669 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.669 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:25:40.669 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.021083+0000", 2024-04-14T15:25:40.669 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4750000000000002e-06 2024-04-14T15:25:40.669 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.669 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.669 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:25:40.669 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.021259+0000", 2024-04-14T15:25:40.669 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000176546 2024-04-14T15:25:40.669 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.669 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.670 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for blocked object", 2024-04-14T15:25:40.670 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.021287+0000", 2024-04-14T15:25:40.670 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8102999999999998e-05 2024-04-14T15:25:40.670 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.670 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.670 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:25:40.670 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.023204+0000", 2024-04-14T15:25:40.670 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0019169370000000001 2024-04-14T15:25:40.670 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.670 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.670 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:25:40.670 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.023277+0000", 2024-04-14T15:25:40.670 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2613000000000004e-05 2024-04-14T15:25:40.670 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.671 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.671 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:25:40.671 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.023323+0000", 2024-04-14T15:25:40.671 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6613999999999998e-05 2024-04-14T15:25:40.671 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:25:40.671 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:25:40.671 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:25:40.671 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.671 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.671 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.49576.0:1690 335.3 335:c3dc6263:test-rados-api-smithi090-93652-135::foo:head [cache-flush] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e763)", 2024-04-14T15:25:40.671 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:25:07.024348+0000", 2024-04-14T15:25:40.671 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 33.561904638999998, 2024-04-14T15:25:40.671 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0030786250000000002, 2024-04-14T15:25:40.671 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:25:40.672 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:25:40.672 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-04-14T15:25:40.672 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:25:40.672 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.49576", 2024-04-14T15:25:40.672 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/2125385135", 2024-04-14T15:25:40.672 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 1690 2024-04-14T15:25:40.672 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.672 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:25:40.672 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.672 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:25:40.672 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.024348+0000", 2024-04-14T15:25:40.672 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:25:40.672 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.672 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.672 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:25:40.673 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.024348+0000", 2024-04-14T15:25:40.673 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:25:40.673 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.673 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.673 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:25:40.673 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.024348+0000", 2024-04-14T15:25:40.673 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9099999999999999e-07 2024-04-14T15:25:40.673 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.673 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.673 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:25:40.673 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.024350+0000", 2024-04-14T15:25:40.673 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.091e-06 2024-04-14T15:25:40.673 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.673 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.674 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:25:40.674 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.024354+0000", 2024-04-14T15:25:40.674 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8639999999999998e-06 2024-04-14T15:25:40.674 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.674 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.674 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:25:40.674 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.024358+0000", 2024-04-14T15:25:40.674 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8750000000000002e-06 2024-04-14T15:25:40.674 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.674 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.674 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:25:40.674 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.024497+0000", 2024-04-14T15:25:40.674 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000139274 2024-04-14T15:25:40.674 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.675 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.675 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:25:40.675 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.024555+0000", 2024-04-14T15:25:40.675 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.8211000000000002e-05 2024-04-14T15:25:40.675 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.675 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.675 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:25:40.675 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.027254+0000", 2024-04-14T15:25:40.675 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00269864 2024-04-14T15:25:40.675 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.675 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.675 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:25:40.675 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.027347+0000", 2024-04-14T15:25:40.675 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.2492999999999997e-05 2024-04-14T15:25:40.675 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.676 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.676 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:25:40.676 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.027426+0000", 2024-04-14T15:25:40.676 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.9785999999999998e-05 2024-04-14T15:25:40.676 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:25:40.676 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:25:40.676 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:25:40.676 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.676 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.676 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.49576.0:1692 335.3 335:c3dc6263:test-rados-api-smithi090-93652-135::foo:head [cache-try-flush] snapc 0=[] ondisk+read+skiprwlocks+ignore_overlay+known_if_redirected+supports_pool_eio e763)", 2024-04-14T15:25:40.676 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:25:07.028441+0000", 2024-04-14T15:25:40.676 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 33.557811032000004, 2024-04-14T15:25:40.676 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0019580579999999999, 2024-04-14T15:25:40.676 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:25:40.677 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:25:40.677 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-04-14T15:25:40.677 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:25:40.677 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.49576", 2024-04-14T15:25:40.677 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/2125385135", 2024-04-14T15:25:40.677 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 1692 2024-04-14T15:25:40.677 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.677 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:25:40.677 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.677 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:25:40.677 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.028441+0000", 2024-04-14T15:25:40.677 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:25:40.678 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.678 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.678 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:25:40.678 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.028441+0000", 2024-04-14T15:25:40.678 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:25:40.678 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.678 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.678 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:25:40.678 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.028442+0000", 2024-04-14T15:25:40.678 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.4799999999999997e-07 2024-04-14T15:25:40.678 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.678 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.678 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:25:40.678 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.028445+0000", 2024-04-14T15:25:40.679 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0149999999999999e-06 2024-04-14T15:25:40.679 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.679 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.679 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:25:40.679 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.028452+0000", 2024-04-14T15:25:40.679 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1080000000000004e-06 2024-04-14T15:25:40.679 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.679 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.679 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:25:40.679 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.028457+0000", 2024-04-14T15:25:40.679 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6260000000000003e-06 2024-04-14T15:25:40.679 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.679 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.679 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:25:40.679 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.028488+0000", 2024-04-14T15:25:40.680 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1130999999999997e-05 2024-04-14T15:25:40.680 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.680 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.680 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:25:40.680 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.028549+0000", 2024-04-14T15:25:40.680 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0890999999999997e-05 2024-04-14T15:25:40.680 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.680 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.680 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:25:40.680 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.030286+0000", 2024-04-14T15:25:40.680 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001737239 2024-04-14T15:25:40.680 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.680 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.680 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:25:40.681 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.030342+0000", 2024-04-14T15:25:40.681 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6378999999999997e-05 2024-04-14T15:25:40.681 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.681 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.681 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:25:40.681 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.030399+0000", 2024-04-14T15:25:40.681 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6920999999999999e-05 2024-04-14T15:25:40.681 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:25:40.681 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:25:40.681 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:25:40.681 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.681 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.681 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.49576.0:1693 335.3 335:c3dc6263:test-rados-api-smithi090-93652-135::foo:head [cache-try-flush] snapc 0=[] ondisk+read+skiprwlocks+ignore_overlay+known_if_redirected+supports_pool_eio e763)", 2024-04-14T15:25:40.681 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:25:07.028466+0000", 2024-04-14T15:25:40.681 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 33.557786401999998, 2024-04-14T15:25:40.682 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0021235730000000001, 2024-04-14T15:25:40.682 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:25:40.682 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:25:40.682 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-04-14T15:25:40.682 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:25:40.682 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.49576", 2024-04-14T15:25:40.682 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/2125385135", 2024-04-14T15:25:40.682 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 1693 2024-04-14T15:25:40.682 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.682 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:25:40.682 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.682 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:25:40.682 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.028466+0000", 2024-04-14T15:25:40.682 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:25:40.683 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.683 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.683 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:25:40.683 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.028466+0000", 2024-04-14T15:25:40.683 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:25:40.683 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.683 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.683 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:25:40.683 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.028466+0000", 2024-04-14T15:25:40.683 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.39e-07 2024-04-14T15:25:40.683 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.683 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.683 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:25:40.683 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.028468+0000", 2024-04-14T15:25:40.684 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7600000000000001e-06 2024-04-14T15:25:40.684 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.684 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.684 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:25:40.684 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.028471+0000", 2024-04-14T15:25:40.684 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1669999999999999e-06 2024-04-14T15:25:40.684 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.684 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.684 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:25:40.684 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.028474+0000", 2024-04-14T15:25:40.684 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2449999999999998e-06 2024-04-14T15:25:40.684 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.684 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.684 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:25:40.685 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.028609+0000", 2024-04-14T15:25:40.685 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000134905 2024-04-14T15:25:40.685 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.685 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.685 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:25:40.685 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.028667+0000", 2024-04-14T15:25:40.685 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7664999999999998e-05 2024-04-14T15:25:40.685 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.685 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.685 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:25:40.685 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.030473+0000", 2024-04-14T15:25:40.685 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0018063650000000001 2024-04-14T15:25:40.685 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.685 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.686 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:25:40.686 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.030542+0000", 2024-04-14T15:25:40.686 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8907999999999998e-05 2024-04-14T15:25:40.686 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.686 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.686 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:25:40.686 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:07.030589+0000", 2024-04-14T15:25:40.686 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7219000000000003e-05 2024-04-14T15:25:40.686 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:25:40.686 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:25:40.686 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:25:40.686 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.686 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.686 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.49576.0:1729 335.10 335:08000000::::head [pgnls start_epoch 765] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e765)", 2024-04-14T15:25:40.687 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:25:09.016602+0000", 2024-04-14T15:25:40.687 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 31.569650483, 2024-04-14T15:25:40.687 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0023321959999999999, 2024-04-14T15:25:40.687 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:25:40.687 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:25:40.687 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-04-14T15:25:40.687 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-04-14T15:25:40.687 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.49576", 2024-04-14T15:25:40.687 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.90:0/2125385135", 2024-04-14T15:25:40.687 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 1729 2024-04-14T15:25:40.687 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.687 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:25:40.687 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.687 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:25:40.688 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:09.016602+0000", 2024-04-14T15:25:40.688 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:25:40.688 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.688 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.688 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:25:40.688 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:09.016602+0000", 2024-04-14T15:25:40.688 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:25:40.688 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.688 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.688 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:25:40.688 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:09.016605+0000", 2024-04-14T15:25:40.688 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4489999999999999e-06 2024-04-14T15:25:40.688 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.688 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.688 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:25:40.689 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:09.016619+0000", 2024-04-14T15:25:40.689 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3896999999999999e-05 2024-04-14T15:25:40.689 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.689 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.689 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:25:40.689 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:09.016642+0000", 2024-04-14T15:25:40.689 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2512000000000001e-05 2024-04-14T15:25:40.689 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.689 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.689 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:25:40.689 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:09.016655+0000", 2024-04-14T15:25:40.689 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3349e-05 2024-04-14T15:25:40.689 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.689 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.690 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:25:40.690 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:09.016730+0000", 2024-04-14T15:25:40.690 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5177000000000002e-05 2024-04-14T15:25:40.690 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.690 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.690 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op must wait for map", 2024-04-14T15:25:40.690 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:09.016738+0000", 2024-04-14T15:25:40.690 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5639999999999999e-06 2024-04-14T15:25:40.690 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.690 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.690 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:25:40.690 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:09.018767+0000", 2024-04-14T15:25:40.690 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0020289230000000002 2024-04-14T15:25:40.690 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.690 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.691 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-04-14T15:25:40.691 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:09.018801+0000", 2024-04-14T15:25:40.691 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4684000000000002e-05 2024-04-14T15:25:40.691 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.691 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.691 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:25:40.691 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:09.018934+0000", 2024-04-14T15:25:40.691 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013264099999999999 2024-04-14T15:25:40.691 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:25:40.691 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:25:40.691 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:25:40.691 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.691 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.692 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDECSubOpWrite(323.0s1 771/647 ECSubWrite(tid=1291, reqid=osd.2.18:91, at_version=771'7, trim_to=0'0, roll_forward_to=704'6))", 2024-04-14T15:25:40.692 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-04-14T15:25:15.054061+0000", 2024-04-14T15:25:40.692 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 25.532191045000001, 2024-04-14T15:25:40.692 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0045814970000000003, 2024-04-14T15:25:40.692 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-04-14T15:25:40.692 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-04-14T15:25:40.692 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-04-14T15:25:40.692 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-04-14T15:25:40.692 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.692 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-04-14T15:25:40.692 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:15.054061+0000", 2024-04-14T15:25:40.692 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:25:40.692 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.692 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.692 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-04-14T15:25:40.692 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:15.054061+0000", 2024-04-14T15:25:40.693 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-04-14T15:25:40.693 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.693 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.693 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-04-14T15:25:40.693 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:15.054062+0000", 2024-04-14T15:25:40.693 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8099999999999998e-07 2024-04-14T15:25:40.693 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.693 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.693 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-04-14T15:25:40.693 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:15.056228+0000", 2024-04-14T15:25:40.693 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002166856 2024-04-14T15:25:40.693 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.693 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.693 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-04-14T15:25:40.694 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:15.056241+0000", 2024-04-14T15:25:40.694 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2465e-05 2024-04-14T15:25:40.694 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.694 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.694 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-04-14T15:25:40.694 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:15.056250+0000", 2024-04-14T15:25:40.694 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.9339999999999995e-06 2024-04-14T15:25:40.694 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.694 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.694 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-04-14T15:25:40.694 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:15.056348+0000", 2024-04-14T15:25:40.694 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.7838999999999996e-05 2024-04-14T15:25:40.694 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.694 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.695 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_started", 2024-04-14T15:25:40.695 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:15.056358+0000", 2024-04-14T15:25:40.695 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.7850000000000003e-06 2024-04-14T15:25:40.695 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.695 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.695 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_committed", 2024-04-14T15:25:40.695 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:15.058611+0000", 2024-04-14T15:25:40.695 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0022539320000000002 2024-04-14T15:25:40.695 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-14T15:25:40.695 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-14T15:25:40.695 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-04-14T15:25:40.695 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-04-14T15:25:15.058643+0000", 2024-04-14T15:25:40.695 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1304999999999999e-05 2024-04-14T15:25:40.695 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:25:40.695 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:25:40.695 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:25:40.696 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-14T15:25:40.696 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-14T15:25:40.696 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-04-14T15:25:40.730 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:40.754 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:40.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:40.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:40.856 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:25:40.858 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd reweight 5 1 2024-04-14T15:25:40.922 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:40.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:40.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:25:41.024 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:25:41.053 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:41.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:41.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:25:41.127 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:41 smithi090 ceph-mon[41670]: pgmap v1174: 224 pgs: 1 active+remapped+backfill_toofull, 223 active+clean; 4.2 MiB data, 750 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 9.7 KiB/s wr, 3 op/s; 53 KiB/s, 0 objects/s recovering 2024-04-14T15:25:41.128 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:41 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3783327058' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-14T15:25:41.128 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:41 smithi090 ceph-mon[41670]: osdmap e793: 8 total, 8 up, 7 in 2024-04-14T15:25:41.128 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:41 smithi090 ceph-mon[41670]: osdmap e794: 8 total, 8 up, 7 in 2024-04-14T15:25:41.128 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:41 smithi090 ceph-mon[30399]: pgmap v1174: 224 pgs: 1 active+remapped+backfill_toofull, 223 active+clean; 4.2 MiB data, 750 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 9.7 KiB/s wr, 3 op/s; 53 KiB/s, 0 objects/s recovering 2024-04-14T15:25:41.128 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:41 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3783327058' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-14T15:25:41.128 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:41 smithi090 ceph-mon[30399]: osdmap e793: 8 total, 8 up, 7 in 2024-04-14T15:25:41.128 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:41 smithi090 ceph-mon[30399]: osdmap e794: 8 total, 8 up, 7 in 2024-04-14T15:25:41.155 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:25:41.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:41.182 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:41.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:25:41.283 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:25:41.306 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:41.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:41.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:25:41.348 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:25:41.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.365+0000 7fe605c97700 1 -- 172.21.15.90:0/1523483410 >> v1:172.21.15.92:6789/0 conn(0x7fe60010b720 legacy=0x7fe60010db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:41.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.365+0000 7fe605c97700 1 -- 172.21.15.90:0/1523483410 shutdown_connections 2024-04-14T15:25:41.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.365+0000 7fe605c97700 1 -- 172.21.15.90:0/1523483410 >> 172.21.15.90:0/1523483410 conn(0x7fe6000fe3a0 msgr2=0x7fe6001007c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:25:41.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.365+0000 7fe605c97700 1 -- 172.21.15.90:0/1523483410 shutdown_connections 2024-04-14T15:25:41.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.365+0000 7fe605c97700 1 -- 172.21.15.90:0/1523483410 wait complete. 2024-04-14T15:25:41.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.366+0000 7fe605c97700 1 librados: starting msgr at 2024-04-14T15:25:41.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.366+0000 7fe605c97700 1 librados: starting objecter 2024-04-14T15:25:41.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.366+0000 7fe605c97700 1 Processor -- start 2024-04-14T15:25:41.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.366+0000 7fe605c97700 1 -- start start 2024-04-14T15:25:41.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.366+0000 7fe605c97700 1 librados: setting wanted keys 2024-04-14T15:25:41.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.366+0000 7fe605c97700 1 librados: calling monclient init 2024-04-14T15:25:41.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.367+0000 7fe605c97700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe60010f020 con 0x7fe60010b720 2024-04-14T15:25:41.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.367+0000 7fe5fffff700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fe60010f2f0 0x7fe6001a67d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.90:36130/0 (socket says 172.21.15.90:36130) 2024-04-14T15:25:41.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.367+0000 7fe5fffff700 1 -- 172.21.15.90:0/1251125376 learned_addr learned my addr 172.21.15.90:0/1251125376 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:25:41.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.367+0000 7fe5fcff9700 10 client.?.objecter ms_handle_connect 0x7fe60010f2f0 2024-04-14T15:25:41.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.367+0000 7fe5fcff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:25:41.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.367+0000 7fe5fcff9700 10 client.?.objecter ms_handle_connect 0x7fe60010b720 2024-04-14T15:25:41.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.367+0000 7fe5fcff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:25:41.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.367+0000 7fe5fcff9700 10 client.?.objecter ms_handle_connect 0x7fe600108870 2024-04-14T15:25:41.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.367+0000 7fe5fcff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:25:41.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.367+0000 7fe605c97700 1 -- 172.21.15.90:0/1251125376 --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe6001a8f00 con 0x7fe60010f2f0 2024-04-14T15:25:41.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.367+0000 7fe605c97700 1 -- 172.21.15.90:0/1251125376 --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe6001a7ef0 con 0x7fe600108870 2024-04-14T15:25:41.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.367+0000 7fe5fcff9700 1 -- 172.21.15.90:0/1251125376 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3280166252 0 0) 0x7fe6001a8f00 con 0x7fe60010f2f0 2024-04-14T15:25:41.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.367+0000 7fe5fcff9700 1 -- 172.21.15.90:0/1251125376 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe5e80035b0 con 0x7fe60010f2f0 2024-04-14T15:25:41.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.367+0000 7fe5fcff9700 1 -- 172.21.15.90:0/1251125376 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4112998661 0 0) 0x7fe60010f020 con 0x7fe60010b720 2024-04-14T15:25:41.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.367+0000 7fe5fcff9700 1 -- 172.21.15.90:0/1251125376 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe6001a8f00 con 0x7fe60010b720 2024-04-14T15:25:41.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.367+0000 7fe5fcff9700 1 -- 172.21.15.90:0/1251125376 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 604062272 0 0) 0x7fe6001a7ef0 con 0x7fe600108870 2024-04-14T15:25:41.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.367+0000 7fe5fcff9700 1 -- 172.21.15.90:0/1251125376 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe60010f020 con 0x7fe600108870 2024-04-14T15:25:41.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.367+0000 7fe5fcff9700 1 -- 172.21.15.90:0/1251125376 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2536017607 0 0) 0x7fe5e80035b0 con 0x7fe60010f2f0 2024-04-14T15:25:41.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.367+0000 7fe5fcff9700 1 -- 172.21.15.90:0/1251125376 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe6001a7ef0 con 0x7fe60010f2f0 2024-04-14T15:25:41.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.368+0000 7fe5fcff9700 1 -- 172.21.15.90:0/1251125376 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fe5f8001b60 con 0x7fe60010f2f0 2024-04-14T15:25:41.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.368+0000 7fe5fcff9700 1 -- 172.21.15.90:0/1251125376 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2505550082 0 0) 0x7fe6001a8f00 con 0x7fe60010b720 2024-04-14T15:25:41.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.368+0000 7fe5fcff9700 1 -- 172.21.15.90:0/1251125376 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe5e80035b0 con 0x7fe60010b720 2024-04-14T15:25:41.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.368+0000 7fe5fcff9700 1 -- 172.21.15.90:0/1251125376 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1756588814 0 0) 0x7fe6001a7ef0 con 0x7fe60010f2f0 2024-04-14T15:25:41.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.368+0000 7fe5fcff9700 1 -- 172.21.15.90:0/1251125376 >> v1:172.21.15.90:6790/0 conn(0x7fe600108870 legacy=0x7fe60010d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:41.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.368+0000 7fe5fcff9700 1 -- 172.21.15.90:0/1251125376 >> v1:172.21.15.90:6789/0 conn(0x7fe60010b720 legacy=0x7fe60010dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:41.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.368+0000 7fe5fcff9700 1 -- 172.21.15.90:0/1251125376 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6001a80c0 con 0x7fe60010f2f0 2024-04-14T15:25:41.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.368+0000 7fe605c97700 1 -- 172.21.15.90:0/1251125376 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe6001a83b0 con 0x7fe60010f2f0 2024-04-14T15:25:41.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.368+0000 7fe605c97700 10 client.49861.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:25:41.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.368+0000 7fe605c97700 1 -- 172.21.15.90:0/1251125376 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe6001a8870 con 0x7fe60010f2f0 2024-04-14T15:25:41.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.368+0000 7fe605c97700 1 librados: init done 2024-04-14T15:25:41.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.368+0000 7fe5fcff9700 1 -- 172.21.15.90:0/1251125376 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fe5f8003ec0 con 0x7fe60010f2f0 2024-04-14T15:25:41.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.368+0000 7fe5fcff9700 1 -- 172.21.15.90:0/1251125376 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fe5f8005ff0 con 0x7fe60010f2f0 2024-04-14T15:25:41.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.369+0000 7fe605c97700 1 -- 172.21.15.90:0/1251125376 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fe5ec005100 con 0x7fe60010f2f0 2024-04-14T15:25:41.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.370+0000 7fe5fcff9700 1 -- 172.21.15.90:0/1251125376 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fe5f801cba0 con 0x7fe60010f2f0 2024-04-14T15:25:41.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.370+0000 7fe5fcff9700 1 -- 172.21.15.90:0/1251125376 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(794..794 src has 467..794) ==== 8816+0+0 (unknown 2719902298 0 0) 0x7fe5f808b070 con 0x7fe60010f2f0 2024-04-14T15:25:41.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.370+0000 7fe5fcff9700 10 client.49861.objecter ms_dispatch 0x7fe600001550 osd_map(794..794 src has 467..794) 2024-04-14T15:25:41.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.370+0000 7fe5fcff9700 3 client.49861.objecter handle_osd_map got epochs [794,794] > 0 2024-04-14T15:25:41.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.370+0000 7fe5fcff9700 3 client.49861.objecter handle_osd_map decoding full epoch 794 2024-04-14T15:25:41.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.370+0000 7fe5fcff9700 20 client.49861.objecter dump_active .. 0 homeless 2024-04-14T15:25:41.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.370+0000 7fe5fcff9700 1 -- 172.21.15.90:0/1251125376 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v794) ==== 54+0+16 (unknown 3412756763 0 2420000192) 0x7fe5f808d4a0 con 0x7fe60010f2f0 2024-04-14T15:25:41.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.371+0000 7fe5fcff9700 10 client.49861.objecter ms_handle_connect 0x7fe5e80712c0 2024-04-14T15:25:41.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.371+0000 7fe605c97700 10 client.49861.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:25:41.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.371+0000 7fe605c97700 15 client.49861.objecter _session_command_op_assign -1 1 2024-04-14T15:25:41.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.371+0000 7fe605c97700 20 client.49861.objecter _get_session s=0x7fe5ec0053e0 osd=0 3 2024-04-14T15:25:41.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.371+0000 7fe605c97700 20 client.49861.objecter put_session s=0x7fe5ec0053e0 osd=0 3 2024-04-14T15:25:41.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.371+0000 7fe605c97700 20 client.49861.objecter _get_session s=0x7fe5ec0053e0 osd=0 3 2024-04-14T15:25:41.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.371+0000 7fe605c97700 15 client.49861.objecter _session_command_op_remove -1 1 2024-04-14T15:25:41.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.371+0000 7fe605c97700 20 client.49861.objecter get_session s=0x7fe5ec0053e0 osd=0 3 2024-04-14T15:25:41.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.371+0000 7fe605c97700 15 client.49861.objecter _session_command_op_assign 0 1 2024-04-14T15:25:41.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.371+0000 7fe605c97700 20 client.49861.objecter put_session s=0x7fe5ec0053e0 osd=0 4 2024-04-14T15:25:41.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.371+0000 7fe605c97700 10 client.49861.objecter _send_command 1 2024-04-14T15:25:41.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.371+0000 7fe605c97700 1 -- 172.21.15.90:0/1251125376 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe5ec004b50 con 0x7fe5ec004730 2024-04-14T15:25:41.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.372+0000 7fe5fcff9700 10 client.49861.objecter ms_handle_connect 0x7fe5ec004730 2024-04-14T15:25:41.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:41 smithi092 ceph-mon[33966]: pgmap v1174: 224 pgs: 1 active+remapped+backfill_toofull, 223 active+clean; 4.2 MiB data, 750 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 9.7 KiB/s wr, 3 op/s; 53 KiB/s, 0 objects/s recovering 2024-04-14T15:25:41.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:41 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3783327058' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-14T15:25:41.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:41 smithi092 ceph-mon[33966]: osdmap e793: 8 total, 8 up, 7 in 2024-04-14T15:25:41.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:41 smithi092 ceph-mon[33966]: osdmap e794: 8 total, 8 up, 7 in 2024-04-14T15:25:41.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.373+0000 7fe5fcff9700 1 -- 172.21.15.90:0/1251125376 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fe5ec004b50 con 0x7fe5ec004730 2024-04-14T15:25:41.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.373+0000 7fe5fcff9700 10 client.49861.objecter ms_dispatch 0x7fe600001550 command_reply(tid 1: 0 ) 2024-04-14T15:25:41.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.373+0000 7fe5fcff9700 10 client.49861.objecter _finish_command 1 = system:0 2024-04-14T15:25:41.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.373+0000 7fe5fcff9700 20 client.49861.objecter put_session s=0x7fe5ec0053e0 osd=0 4 2024-04-14T15:25:41.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.373+0000 7fe5fcff9700 15 client.49861.objecter _session_command_op_remove 0 1 2024-04-14T15:25:41.390 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:25:41.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.393+0000 7fe605c97700 10 client.49861.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:25:41.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.393+0000 7fe605c97700 15 client.49861.objecter _session_command_op_assign -1 2 2024-04-14T15:25:41.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.393+0000 7fe605c97700 20 client.49861.objecter _get_session s=0x7fe5ec0053e0 osd=0 3 2024-04-14T15:25:41.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.393+0000 7fe605c97700 20 client.49861.objecter put_session s=0x7fe5ec0053e0 osd=0 3 2024-04-14T15:25:41.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.393+0000 7fe605c97700 20 client.49861.objecter _get_session s=0x7fe5ec0053e0 osd=0 3 2024-04-14T15:25:41.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.393+0000 7fe605c97700 15 client.49861.objecter _session_command_op_remove -1 2 2024-04-14T15:25:41.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.393+0000 7fe605c97700 20 client.49861.objecter get_session s=0x7fe5ec0053e0 osd=0 3 2024-04-14T15:25:41.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.393+0000 7fe605c97700 15 client.49861.objecter _session_command_op_assign 0 2 2024-04-14T15:25:41.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.393+0000 7fe605c97700 20 client.49861.objecter put_session s=0x7fe5ec0053e0 osd=0 4 2024-04-14T15:25:41.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.393+0000 7fe605c97700 10 client.49861.objecter _send_command 2 2024-04-14T15:25:41.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.393+0000 7fe605c97700 1 -- 172.21.15.90:0/1251125376 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe5ec007b00 con 0x7fe5ec004730 2024-04-14T15:25:41.396 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:25:41.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.395+0000 7fe5fcff9700 1 -- 172.21.15.90:0/1251125376 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fe5ec007b00 con 0x7fe5ec004730 2024-04-14T15:25:41.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.396+0000 7fe5fcff9700 10 client.49861.objecter ms_dispatch 0x7fe600001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:25:41.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.396+0000 7fe5fcff9700 10 client.49861.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:25:41.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.396+0000 7fe5fcff9700 20 client.49861.objecter put_session s=0x7fe5ec0053e0 osd=0 4 2024-04-14T15:25:41.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.396+0000 7fe5fcff9700 15 client.49861.objecter _session_command_op_remove 0 2 2024-04-14T15:25:41.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.396+0000 7fe605c97700 10 client.49861.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:25:41.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.396+0000 7fe605c97700 15 client.49861.objecter _session_command_op_assign -1 3 2024-04-14T15:25:41.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.396+0000 7fe605c97700 20 client.49861.objecter _get_session s=0x7fe5ec004f90 osd=1 3 2024-04-14T15:25:41.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.396+0000 7fe605c97700 20 client.49861.objecter put_session s=0x7fe5ec004f90 osd=1 3 2024-04-14T15:25:41.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.396+0000 7fe605c97700 20 client.49861.objecter _get_session s=0x7fe5ec004f90 osd=1 3 2024-04-14T15:25:41.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.396+0000 7fe605c97700 15 client.49861.objecter _session_command_op_remove -1 3 2024-04-14T15:25:41.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.396+0000 7fe605c97700 20 client.49861.objecter get_session s=0x7fe5ec004f90 osd=1 3 2024-04-14T15:25:41.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.396+0000 7fe605c97700 15 client.49861.objecter _session_command_op_assign 1 3 2024-04-14T15:25:41.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.396+0000 7fe605c97700 20 client.49861.objecter put_session s=0x7fe5ec004f90 osd=1 4 2024-04-14T15:25:41.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.396+0000 7fe605c97700 10 client.49861.objecter _send_command 3 2024-04-14T15:25:41.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.396+0000 7fe605c97700 1 -- 172.21.15.90:0/1251125376 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fe5ec002030 con 0x7fe5ec001b80 2024-04-14T15:25:41.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.396+0000 7fe5fcff9700 10 client.49861.objecter ms_handle_connect 0x7fe5ec001b80 2024-04-14T15:25:41.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.397+0000 7fe5fcff9700 1 -- 172.21.15.90:0/1251125376 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fe5ec002030 con 0x7fe5ec001b80 2024-04-14T15:25:41.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.397+0000 7fe5fcff9700 10 client.49861.objecter ms_dispatch 0x7fe600001550 command_reply(tid 3: 0 ) 2024-04-14T15:25:41.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.397+0000 7fe5fcff9700 10 client.49861.objecter _finish_command 3 = system:0 2024-04-14T15:25:41.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.397+0000 7fe5fcff9700 20 client.49861.objecter put_session s=0x7fe5ec004f90 osd=1 4 2024-04-14T15:25:41.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.397+0000 7fe5fcff9700 15 client.49861.objecter _session_command_op_remove 1 3 2024-04-14T15:25:41.408 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:41.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.415+0000 7fe605c97700 10 client.49861.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:25:41.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.415+0000 7fe605c97700 15 client.49861.objecter _session_command_op_assign -1 4 2024-04-14T15:25:41.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.415+0000 7fe605c97700 20 client.49861.objecter _get_session s=0x7fe5ec004f90 osd=1 3 2024-04-14T15:25:41.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.415+0000 7fe605c97700 20 client.49861.objecter put_session s=0x7fe5ec004f90 osd=1 3 2024-04-14T15:25:41.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.415+0000 7fe605c97700 20 client.49861.objecter _get_session s=0x7fe5ec004f90 osd=1 3 2024-04-14T15:25:41.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.415+0000 7fe605c97700 15 client.49861.objecter _session_command_op_remove -1 4 2024-04-14T15:25:41.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.415+0000 7fe605c97700 20 client.49861.objecter get_session s=0x7fe5ec004f90 osd=1 3 2024-04-14T15:25:41.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.415+0000 7fe605c97700 15 client.49861.objecter _session_command_op_assign 1 4 2024-04-14T15:25:41.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.415+0000 7fe605c97700 20 client.49861.objecter put_session s=0x7fe5ec004f90 osd=1 4 2024-04-14T15:25:41.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.415+0000 7fe605c97700 10 client.49861.objecter _send_command 4 2024-04-14T15:25:41.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.415+0000 7fe605c97700 1 -- 172.21.15.90:0/1251125376 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe5ec002370 con 0x7fe5ec001b80 2024-04-14T15:25:41.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.417+0000 7fe5fcff9700 1 -- 172.21.15.90:0/1251125376 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fe5ec002370 con 0x7fe5ec001b80 2024-04-14T15:25:41.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.417+0000 7fe5fcff9700 10 client.49861.objecter ms_dispatch 0x7fe600001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:25:41.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.417+0000 7fe5fcff9700 10 client.49861.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:25:41.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.417+0000 7fe5fcff9700 20 client.49861.objecter put_session s=0x7fe5ec004f90 osd=1 4 2024-04-14T15:25:41.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.417+0000 7fe5fcff9700 15 client.49861.objecter _session_command_op_remove 1 4 2024-04-14T15:25:41.418 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:25:41.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.417+0000 7fe605c97700 10 client.49861.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:25:41.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.417+0000 7fe605c97700 15 client.49861.objecter _session_command_op_assign -1 5 2024-04-14T15:25:41.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.417+0000 7fe605c97700 20 client.49861.objecter _get_session s=0x7fe5ec001900 osd=2 3 2024-04-14T15:25:41.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.417+0000 7fe605c97700 20 client.49861.objecter put_session s=0x7fe5ec001900 osd=2 3 2024-04-14T15:25:41.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.417+0000 7fe605c97700 20 client.49861.objecter _get_session s=0x7fe5ec001900 osd=2 3 2024-04-14T15:25:41.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.417+0000 7fe605c97700 15 client.49861.objecter _session_command_op_remove -1 5 2024-04-14T15:25:41.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.417+0000 7fe605c97700 20 client.49861.objecter get_session s=0x7fe5ec001900 osd=2 3 2024-04-14T15:25:41.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.417+0000 7fe605c97700 15 client.49861.objecter _session_command_op_assign 2 5 2024-04-14T15:25:41.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.417+0000 7fe605c97700 20 client.49861.objecter put_session s=0x7fe5ec001900 osd=2 4 2024-04-14T15:25:41.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.417+0000 7fe605c97700 10 client.49861.objecter _send_command 5 2024-04-14T15:25:41.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.417+0000 7fe605c97700 1 -- 172.21.15.90:0/1251125376 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fe5ec0153d0 con 0x7fe5ec008990 2024-04-14T15:25:41.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.417+0000 7fe5fcff9700 10 client.49861.objecter ms_handle_connect 0x7fe5ec008990 2024-04-14T15:25:41.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.418+0000 7fe5fcff9700 1 -- 172.21.15.90:0/1251125376 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fe5ec0153d0 con 0x7fe5ec008990 2024-04-14T15:25:41.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.418+0000 7fe5fcff9700 10 client.49861.objecter ms_dispatch 0x7fe600001550 command_reply(tid 5: 0 ) 2024-04-14T15:25:41.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.418+0000 7fe5fcff9700 10 client.49861.objecter _finish_command 5 = system:0 2024-04-14T15:25:41.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.418+0000 7fe5fcff9700 20 client.49861.objecter put_session s=0x7fe5ec001900 osd=2 4 2024-04-14T15:25:41.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.418+0000 7fe5fcff9700 15 client.49861.objecter _session_command_op_remove 2 5 2024-04-14T15:25:41.432 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:41.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:41.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:41.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.435+0000 7fe605c97700 10 client.49861.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:25:41.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.435+0000 7fe605c97700 15 client.49861.objecter _session_command_op_assign -1 6 2024-04-14T15:25:41.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.435+0000 7fe605c97700 20 client.49861.objecter _get_session s=0x7fe5ec001900 osd=2 3 2024-04-14T15:25:41.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.435+0000 7fe605c97700 20 client.49861.objecter put_session s=0x7fe5ec001900 osd=2 3 2024-04-14T15:25:41.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.435+0000 7fe605c97700 20 client.49861.objecter _get_session s=0x7fe5ec001900 osd=2 3 2024-04-14T15:25:41.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.435+0000 7fe605c97700 15 client.49861.objecter _session_command_op_remove -1 6 2024-04-14T15:25:41.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.435+0000 7fe605c97700 20 client.49861.objecter get_session s=0x7fe5ec001900 osd=2 3 2024-04-14T15:25:41.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.435+0000 7fe605c97700 15 client.49861.objecter _session_command_op_assign 2 6 2024-04-14T15:25:41.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.435+0000 7fe605c97700 20 client.49861.objecter put_session s=0x7fe5ec001900 osd=2 4 2024-04-14T15:25:41.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.435+0000 7fe605c97700 10 client.49861.objecter _send_command 6 2024-04-14T15:25:41.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.435+0000 7fe605c97700 1 -- 172.21.15.90:0/1251125376 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe5ec007fc0 con 0x7fe5ec008990 2024-04-14T15:25:41.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.436+0000 7fe5fcff9700 1 -- 172.21.15.90:0/1251125376 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fe5ec007fc0 con 0x7fe5ec008990 2024-04-14T15:25:41.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.436+0000 7fe5fcff9700 10 client.49861.objecter ms_dispatch 0x7fe600001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:25:41.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.436+0000 7fe5fcff9700 10 client.49861.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:25:41.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.436+0000 7fe5fcff9700 20 client.49861.objecter put_session s=0x7fe5ec001900 osd=2 4 2024-04-14T15:25:41.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.436+0000 7fe5fcff9700 15 client.49861.objecter _session_command_op_remove 2 6 2024-04-14T15:25:41.437 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:25:41.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.437+0000 7fe605c97700 10 client.49861.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:25:41.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.437+0000 7fe605c97700 15 client.49861.objecter _session_command_op_assign -1 7 2024-04-14T15:25:41.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.437+0000 7fe605c97700 20 client.49861.objecter _get_session s=0x7fe5ec0081d0 osd=3 3 2024-04-14T15:25:41.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.437+0000 7fe605c97700 20 client.49861.objecter put_session s=0x7fe5ec0081d0 osd=3 3 2024-04-14T15:25:41.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.437+0000 7fe605c97700 20 client.49861.objecter _get_session s=0x7fe5ec0081d0 osd=3 3 2024-04-14T15:25:41.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.437+0000 7fe605c97700 15 client.49861.objecter _session_command_op_remove -1 7 2024-04-14T15:25:41.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.437+0000 7fe605c97700 20 client.49861.objecter get_session s=0x7fe5ec0081d0 osd=3 3 2024-04-14T15:25:41.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.437+0000 7fe605c97700 15 client.49861.objecter _session_command_op_assign 3 7 2024-04-14T15:25:41.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.437+0000 7fe605c97700 20 client.49861.objecter put_session s=0x7fe5ec0081d0 osd=3 4 2024-04-14T15:25:41.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.437+0000 7fe605c97700 10 client.49861.objecter _send_command 7 2024-04-14T15:25:41.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.437+0000 7fe605c97700 1 -- 172.21.15.90:0/1251125376 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fe5ec016110 con 0x7fe5ec015580 2024-04-14T15:25:41.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.437+0000 7fe5fcff9700 10 client.49861.objecter ms_handle_connect 0x7fe5ec015580 2024-04-14T15:25:41.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.438+0000 7fe5fcff9700 1 -- 172.21.15.90:0/1251125376 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fe5ec016110 con 0x7fe5ec015580 2024-04-14T15:25:41.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.438+0000 7fe5fcff9700 10 client.49861.objecter ms_dispatch 0x7fe600001550 command_reply(tid 7: 0 ) 2024-04-14T15:25:41.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.438+0000 7fe5fcff9700 10 client.49861.objecter _finish_command 7 = system:0 2024-04-14T15:25:41.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.438+0000 7fe5fcff9700 20 client.49861.objecter put_session s=0x7fe5ec0081d0 osd=3 4 2024-04-14T15:25:41.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.438+0000 7fe5fcff9700 15 client.49861.objecter _session_command_op_remove 3 7 2024-04-14T15:25:41.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.454+0000 7fe605c97700 10 client.49861.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:25:41.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.454+0000 7fe605c97700 15 client.49861.objecter _session_command_op_assign -1 8 2024-04-14T15:25:41.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.454+0000 7fe605c97700 20 client.49861.objecter _get_session s=0x7fe5ec0081d0 osd=3 3 2024-04-14T15:25:41.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.454+0000 7fe605c97700 20 client.49861.objecter put_session s=0x7fe5ec0081d0 osd=3 3 2024-04-14T15:25:41.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.454+0000 7fe605c97700 20 client.49861.objecter _get_session s=0x7fe5ec0081d0 osd=3 3 2024-04-14T15:25:41.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.454+0000 7fe605c97700 15 client.49861.objecter _session_command_op_remove -1 8 2024-04-14T15:25:41.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.454+0000 7fe605c97700 20 client.49861.objecter get_session s=0x7fe5ec0081d0 osd=3 3 2024-04-14T15:25:41.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.454+0000 7fe605c97700 15 client.49861.objecter _session_command_op_assign 3 8 2024-04-14T15:25:41.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.454+0000 7fe605c97700 20 client.49861.objecter put_session s=0x7fe5ec0081d0 osd=3 4 2024-04-14T15:25:41.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.454+0000 7fe605c97700 10 client.49861.objecter _send_command 8 2024-04-14T15:25:41.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.454+0000 7fe605c97700 1 -- 172.21.15.90:0/1251125376 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe5ec016a90 con 0x7fe5ec015580 2024-04-14T15:25:41.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.456+0000 7fe5fcff9700 1 -- 172.21.15.90:0/1251125376 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fe5ec016a90 con 0x7fe5ec015580 2024-04-14T15:25:41.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.456+0000 7fe5fcff9700 10 client.49861.objecter ms_dispatch 0x7fe600001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:25:41.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.456+0000 7fe5fcff9700 10 client.49861.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:25:41.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.456+0000 7fe5fcff9700 20 client.49861.objecter put_session s=0x7fe5ec0081d0 osd=3 4 2024-04-14T15:25:41.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.456+0000 7fe5fcff9700 15 client.49861.objecter _session_command_op_remove 3 8 2024-04-14T15:25:41.457 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:25:41.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.456+0000 7fe605c97700 10 client.49861.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:25:41.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.456+0000 7fe605c97700 15 client.49861.objecter _session_command_op_assign -1 9 2024-04-14T15:25:41.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.456+0000 7fe605c97700 20 client.49861.objecter _get_session s=0x7fe5ec016a90 osd=4 3 2024-04-14T15:25:41.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.456+0000 7fe605c97700 20 client.49861.objecter put_session s=0x7fe5ec016a90 osd=4 3 2024-04-14T15:25:41.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.456+0000 7fe605c97700 20 client.49861.objecter _get_session s=0x7fe5ec016a90 osd=4 3 2024-04-14T15:25:41.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.456+0000 7fe605c97700 15 client.49861.objecter _session_command_op_remove -1 9 2024-04-14T15:25:41.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.456+0000 7fe605c97700 20 client.49861.objecter get_session s=0x7fe5ec016a90 osd=4 3 2024-04-14T15:25:41.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.456+0000 7fe605c97700 15 client.49861.objecter _session_command_op_assign 4 9 2024-04-14T15:25:41.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.456+0000 7fe605c97700 20 client.49861.objecter put_session s=0x7fe5ec016a90 osd=4 4 2024-04-14T15:25:41.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.456+0000 7fe605c97700 10 client.49861.objecter _send_command 9 2024-04-14T15:25:41.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.456+0000 7fe605c97700 1 -- 172.21.15.90:0/1251125376 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fe5ec01eb70 con 0x7fe5ec01af70 2024-04-14T15:25:41.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.457+0000 7fe5fcff9700 10 client.49861.objecter ms_handle_connect 0x7fe5ec01af70 2024-04-14T15:25:41.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.457+0000 7fe5fcff9700 1 -- 172.21.15.90:0/1251125376 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fe5ec01eb70 con 0x7fe5ec01af70 2024-04-14T15:25:41.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.457+0000 7fe5fcff9700 10 client.49861.objecter ms_dispatch 0x7fe600001550 command_reply(tid 9: 0 ) 2024-04-14T15:25:41.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.457+0000 7fe5fcff9700 10 client.49861.objecter _finish_command 9 = system:0 2024-04-14T15:25:41.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.457+0000 7fe5fcff9700 20 client.49861.objecter put_session s=0x7fe5ec016a90 osd=4 4 2024-04-14T15:25:41.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.457+0000 7fe5fcff9700 15 client.49861.objecter _session_command_op_remove 4 9 2024-04-14T15:25:41.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.474+0000 7fe605c97700 10 client.49861.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:25:41.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.474+0000 7fe605c97700 15 client.49861.objecter _session_command_op_assign -1 10 2024-04-14T15:25:41.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.474+0000 7fe605c97700 20 client.49861.objecter _get_session s=0x7fe5ec016a90 osd=4 3 2024-04-14T15:25:41.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.474+0000 7fe605c97700 20 client.49861.objecter put_session s=0x7fe5ec016a90 osd=4 3 2024-04-14T15:25:41.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.474+0000 7fe605c97700 20 client.49861.objecter _get_session s=0x7fe5ec016a90 osd=4 3 2024-04-14T15:25:41.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.474+0000 7fe605c97700 15 client.49861.objecter _session_command_op_remove -1 10 2024-04-14T15:25:41.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.474+0000 7fe605c97700 20 client.49861.objecter get_session s=0x7fe5ec016a90 osd=4 3 2024-04-14T15:25:41.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.474+0000 7fe605c97700 15 client.49861.objecter _session_command_op_assign 4 10 2024-04-14T15:25:41.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.474+0000 7fe605c97700 20 client.49861.objecter put_session s=0x7fe5ec016a90 osd=4 4 2024-04-14T15:25:41.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.474+0000 7fe605c97700 10 client.49861.objecter _send_command 10 2024-04-14T15:25:41.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.474+0000 7fe605c97700 1 -- 172.21.15.90:0/1251125376 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe5ec020520 con 0x7fe5ec01af70 2024-04-14T15:25:41.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.476+0000 7fe5fcff9700 1 -- 172.21.15.90:0/1251125376 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fe5ec020520 con 0x7fe5ec01af70 2024-04-14T15:25:41.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.476+0000 7fe5fcff9700 10 client.49861.objecter ms_dispatch 0x7fe600001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:25:41.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.476+0000 7fe5fcff9700 10 client.49861.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:25:41.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.476+0000 7fe5fcff9700 20 client.49861.objecter put_session s=0x7fe5ec016a90 osd=4 4 2024-04-14T15:25:41.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.476+0000 7fe5fcff9700 15 client.49861.objecter _session_command_op_remove 4 10 2024-04-14T15:25:41.476 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:25:41.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.476+0000 7fe605c97700 10 client.49861.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:25:41.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.476+0000 7fe605c97700 15 client.49861.objecter _session_command_op_assign -1 11 2024-04-14T15:25:41.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.476+0000 7fe605c97700 20 client.49861.objecter _get_session s=0x7fe5ec020520 osd=5 3 2024-04-14T15:25:41.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.476+0000 7fe605c97700 20 client.49861.objecter put_session s=0x7fe5ec020520 osd=5 3 2024-04-14T15:25:41.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.476+0000 7fe605c97700 20 client.49861.objecter _get_session s=0x7fe5ec020520 osd=5 3 2024-04-14T15:25:41.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.476+0000 7fe605c97700 15 client.49861.objecter _session_command_op_remove -1 11 2024-04-14T15:25:41.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.476+0000 7fe605c97700 20 client.49861.objecter get_session s=0x7fe5ec020520 osd=5 3 2024-04-14T15:25:41.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.476+0000 7fe605c97700 15 client.49861.objecter _session_command_op_assign 5 11 2024-04-14T15:25:41.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.476+0000 7fe605c97700 20 client.49861.objecter put_session s=0x7fe5ec020520 osd=5 4 2024-04-14T15:25:41.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.476+0000 7fe605c97700 10 client.49861.objecter _send_command 11 2024-04-14T15:25:41.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.476+0000 7fe605c97700 1 -- 172.21.15.90:0/1251125376 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fe5ec021100 con 0x7fe5ec020c40 2024-04-14T15:25:41.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.476+0000 7fe5fcff9700 10 client.49861.objecter ms_handle_connect 0x7fe5ec020c40 2024-04-14T15:25:41.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.477+0000 7fe5fcff9700 1 -- 172.21.15.90:0/1251125376 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fe5ec021100 con 0x7fe5ec020c40 2024-04-14T15:25:41.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.477+0000 7fe5fcff9700 10 client.49861.objecter ms_dispatch 0x7fe600001550 command_reply(tid 11: 0 ) 2024-04-14T15:25:41.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.477+0000 7fe5fcff9700 10 client.49861.objecter _finish_command 11 = system:0 2024-04-14T15:25:41.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.477+0000 7fe5fcff9700 20 client.49861.objecter put_session s=0x7fe5ec020520 osd=5 4 2024-04-14T15:25:41.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.477+0000 7fe5fcff9700 15 client.49861.objecter _session_command_op_remove 5 11 2024-04-14T15:25:41.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.494+0000 7fe605c97700 10 client.49861.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:25:41.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.494+0000 7fe605c97700 15 client.49861.objecter _session_command_op_assign -1 12 2024-04-14T15:25:41.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.494+0000 7fe605c97700 20 client.49861.objecter _get_session s=0x7fe5ec020520 osd=5 3 2024-04-14T15:25:41.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.494+0000 7fe605c97700 20 client.49861.objecter put_session s=0x7fe5ec020520 osd=5 3 2024-04-14T15:25:41.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.494+0000 7fe605c97700 20 client.49861.objecter _get_session s=0x7fe5ec020520 osd=5 3 2024-04-14T15:25:41.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.494+0000 7fe605c97700 15 client.49861.objecter _session_command_op_remove -1 12 2024-04-14T15:25:41.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.494+0000 7fe605c97700 20 client.49861.objecter get_session s=0x7fe5ec020520 osd=5 3 2024-04-14T15:25:41.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.494+0000 7fe605c97700 15 client.49861.objecter _session_command_op_assign 5 12 2024-04-14T15:25:41.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.494+0000 7fe605c97700 20 client.49861.objecter put_session s=0x7fe5ec020520 osd=5 4 2024-04-14T15:25:41.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.494+0000 7fe605c97700 10 client.49861.objecter _send_command 12 2024-04-14T15:25:41.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.494+0000 7fe605c97700 1 -- 172.21.15.90:0/1251125376 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe5ec021360 con 0x7fe5ec020c40 2024-04-14T15:25:41.496 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:25:41.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.496+0000 7fe5fcff9700 1 -- 172.21.15.90:0/1251125376 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fe5ec021360 con 0x7fe5ec020c40 2024-04-14T15:25:41.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.496+0000 7fe5fcff9700 10 client.49861.objecter ms_dispatch 0x7fe600001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:25:41.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.496+0000 7fe5fcff9700 10 client.49861.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:25:41.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.496+0000 7fe5fcff9700 20 client.49861.objecter put_session s=0x7fe5ec020520 osd=5 4 2024-04-14T15:25:41.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.496+0000 7fe5fcff9700 15 client.49861.objecter _session_command_op_remove 5 12 2024-04-14T15:25:41.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.496+0000 7fe605c97700 10 client.49861.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:25:41.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.496+0000 7fe605c97700 15 client.49861.objecter _session_command_op_assign -1 13 2024-04-14T15:25:41.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.496+0000 7fe605c97700 20 client.49861.objecter _get_session s=0x7fe5ec021100 osd=6 3 2024-04-14T15:25:41.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.496+0000 7fe605c97700 20 client.49861.objecter put_session s=0x7fe5ec021100 osd=6 3 2024-04-14T15:25:41.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.496+0000 7fe605c97700 20 client.49861.objecter _get_session s=0x7fe5ec021100 osd=6 3 2024-04-14T15:25:41.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.496+0000 7fe605c97700 15 client.49861.objecter _session_command_op_remove -1 13 2024-04-14T15:25:41.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.496+0000 7fe605c97700 20 client.49861.objecter get_session s=0x7fe5ec021100 osd=6 3 2024-04-14T15:25:41.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.496+0000 7fe605c97700 15 client.49861.objecter _session_command_op_assign 6 13 2024-04-14T15:25:41.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.496+0000 7fe605c97700 20 client.49861.objecter put_session s=0x7fe5ec021100 osd=6 4 2024-04-14T15:25:41.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.496+0000 7fe605c97700 10 client.49861.objecter _send_command 13 2024-04-14T15:25:41.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.496+0000 7fe605c97700 1 -- 172.21.15.90:0/1251125376 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fe5ec027740 con 0x7fe5ec023c30 2024-04-14T15:25:41.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.497+0000 7fe5fcff9700 10 client.49861.objecter ms_handle_connect 0x7fe5ec023c30 2024-04-14T15:25:41.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.498+0000 7fe5fcff9700 1 -- 172.21.15.90:0/1251125376 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fe5ec027740 con 0x7fe5ec023c30 2024-04-14T15:25:41.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.498+0000 7fe5fcff9700 10 client.49861.objecter ms_dispatch 0x7fe600001550 command_reply(tid 13: 0 ) 2024-04-14T15:25:41.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.498+0000 7fe5fcff9700 10 client.49861.objecter _finish_command 13 = system:0 2024-04-14T15:25:41.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.498+0000 7fe5fcff9700 20 client.49861.objecter put_session s=0x7fe5ec021100 osd=6 4 2024-04-14T15:25:41.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.498+0000 7fe5fcff9700 15 client.49861.objecter _session_command_op_remove 6 13 2024-04-14T15:25:41.500 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-14T15:25:41.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.514+0000 7fe605c97700 10 client.49861.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:25:41.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.514+0000 7fe605c97700 15 client.49861.objecter _session_command_op_assign -1 14 2024-04-14T15:25:41.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.514+0000 7fe605c97700 20 client.49861.objecter _get_session s=0x7fe5ec021100 osd=6 3 2024-04-14T15:25:41.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.514+0000 7fe605c97700 20 client.49861.objecter put_session s=0x7fe5ec021100 osd=6 3 2024-04-14T15:25:41.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.514+0000 7fe605c97700 20 client.49861.objecter _get_session s=0x7fe5ec021100 osd=6 3 2024-04-14T15:25:41.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.514+0000 7fe605c97700 15 client.49861.objecter _session_command_op_remove -1 14 2024-04-14T15:25:41.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.514+0000 7fe605c97700 20 client.49861.objecter get_session s=0x7fe5ec021100 osd=6 3 2024-04-14T15:25:41.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.514+0000 7fe605c97700 15 client.49861.objecter _session_command_op_assign 6 14 2024-04-14T15:25:41.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.514+0000 7fe605c97700 20 client.49861.objecter put_session s=0x7fe5ec021100 osd=6 4 2024-04-14T15:25:41.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.514+0000 7fe605c97700 10 client.49861.objecter _send_command 14 2024-04-14T15:25:41.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.514+0000 7fe605c97700 1 -- 172.21.15.90:0/1251125376 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe5ec027740 con 0x7fe5ec023c30 2024-04-14T15:25:41.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.516+0000 7fe5fcff9700 1 -- 172.21.15.90:0/1251125376 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fe5ec027740 con 0x7fe5ec023c30 2024-04-14T15:25:41.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.516+0000 7fe5fcff9700 10 client.49861.objecter ms_dispatch 0x7fe600001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:25:41.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.516+0000 7fe5fcff9700 10 client.49861.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:25:41.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.516+0000 7fe5fcff9700 20 client.49861.objecter put_session s=0x7fe5ec021100 osd=6 4 2024-04-14T15:25:41.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.516+0000 7fe5fcff9700 15 client.49861.objecter _session_command_op_remove 6 14 2024-04-14T15:25:41.516 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:25:41.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.516+0000 7fe605c97700 10 client.49861.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:25:41.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.516+0000 7fe605c97700 15 client.49861.objecter _session_command_op_assign -1 15 2024-04-14T15:25:41.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.516+0000 7fe605c97700 20 client.49861.objecter _get_session s=0x7fe5ec027740 osd=7 3 2024-04-14T15:25:41.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.516+0000 7fe605c97700 20 client.49861.objecter put_session s=0x7fe5ec027740 osd=7 3 2024-04-14T15:25:41.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.516+0000 7fe605c97700 20 client.49861.objecter _get_session s=0x7fe5ec027740 osd=7 3 2024-04-14T15:25:41.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.516+0000 7fe605c97700 15 client.49861.objecter _session_command_op_remove -1 15 2024-04-14T15:25:41.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.516+0000 7fe605c97700 20 client.49861.objecter get_session s=0x7fe5ec027740 osd=7 3 2024-04-14T15:25:41.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.516+0000 7fe605c97700 15 client.49861.objecter _session_command_op_assign 7 15 2024-04-14T15:25:41.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.516+0000 7fe605c97700 20 client.49861.objecter put_session s=0x7fe5ec027740 osd=7 4 2024-04-14T15:25:41.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.516+0000 7fe605c97700 10 client.49861.objecter _send_command 15 2024-04-14T15:25:41.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.516+0000 7fe605c97700 1 -- 172.21.15.90:0/1251125376 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fe5ec02ba40 con 0x7fe5ec027dc0 2024-04-14T15:25:41.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.516+0000 7fe5fcff9700 10 client.49861.objecter ms_handle_connect 0x7fe5ec027dc0 2024-04-14T15:25:41.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.517+0000 7fe5fcff9700 1 -- 172.21.15.90:0/1251125376 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fe5ec02ba40 con 0x7fe5ec027dc0 2024-04-14T15:25:41.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.517+0000 7fe5fcff9700 10 client.49861.objecter ms_dispatch 0x7fe600001550 command_reply(tid 15: 0 ) 2024-04-14T15:25:41.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.517+0000 7fe5fcff9700 10 client.49861.objecter _finish_command 15 = system:0 2024-04-14T15:25:41.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.517+0000 7fe5fcff9700 20 client.49861.objecter put_session s=0x7fe5ec027740 osd=7 4 2024-04-14T15:25:41.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.517+0000 7fe5fcff9700 15 client.49861.objecter _session_command_op_remove 7 15 2024-04-14T15:25:41.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.533+0000 7fe605c97700 10 client.49861.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:25:41.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.533+0000 7fe605c97700 15 client.49861.objecter _session_command_op_assign -1 16 2024-04-14T15:25:41.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.533+0000 7fe605c97700 20 client.49861.objecter _get_session s=0x7fe5ec027740 osd=7 3 2024-04-14T15:25:41.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.533+0000 7fe605c97700 20 client.49861.objecter put_session s=0x7fe5ec027740 osd=7 3 2024-04-14T15:25:41.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.533+0000 7fe605c97700 20 client.49861.objecter _get_session s=0x7fe5ec027740 osd=7 3 2024-04-14T15:25:41.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.533+0000 7fe605c97700 15 client.49861.objecter _session_command_op_remove -1 16 2024-04-14T15:25:41.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.533+0000 7fe605c97700 20 client.49861.objecter get_session s=0x7fe5ec027740 osd=7 3 2024-04-14T15:25:41.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.533+0000 7fe605c97700 15 client.49861.objecter _session_command_op_assign 7 16 2024-04-14T15:25:41.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.533+0000 7fe605c97700 20 client.49861.objecter put_session s=0x7fe5ec027740 osd=7 4 2024-04-14T15:25:41.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.533+0000 7fe605c97700 10 client.49861.objecter _send_command 16 2024-04-14T15:25:41.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.533+0000 7fe605c97700 1 -- 172.21.15.90:0/1251125376 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe5ec02d3f0 con 0x7fe5ec027dc0 2024-04-14T15:25:41.535 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:25:41.535 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:25:41.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.535+0000 7fe5fcff9700 1 -- 172.21.15.90:0/1251125376 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fe5ec02d3f0 con 0x7fe5ec027dc0 2024-04-14T15:25:41.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.535+0000 7fe5fcff9700 10 client.49861.objecter ms_dispatch 0x7fe600001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:25:41.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.535+0000 7fe5fcff9700 10 client.49861.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:25:41.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.535+0000 7fe5fcff9700 20 client.49861.objecter put_session s=0x7fe5ec027740 osd=7 4 2024-04-14T15:25:41.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.535+0000 7fe5fcff9700 15 client.49861.objecter _session_command_op_remove 7 16 2024-04-14T15:25:41.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.535+0000 7fe605c97700 10 librados: watch_flush enter 2024-04-14T15:25:41.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.535+0000 7fe605c97700 10 librados: watch_flush exit 2024-04-14T15:25:41.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.535+0000 7fe605c97700 10 client.49861.objecter close_session for osd.0 2024-04-14T15:25:41.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.535+0000 7fe605c97700 1 -- 172.21.15.90:0/1251125376 >> v1:172.21.15.90:6801/3761806885 conn(0x7fe5ec004730 legacy=0x7fe5ec002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:41.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.535+0000 7fe605c97700 20 client.49861.objecter put_session s=0x7fe5ec0053e0 osd=0 1 2024-04-14T15:25:41.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.535+0000 7fe605c97700 10 client.49861.objecter close_session for osd.1 2024-04-14T15:25:41.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.535+0000 7fe605c97700 1 -- 172.21.15.90:0/1251125376 >> v1:172.21.15.90:6805/4170361699 conn(0x7fe5ec001b80 legacy=0x7fe5ec008e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:41.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.535+0000 7fe605c97700 20 client.49861.objecter put_session s=0x7fe5ec004f90 osd=1 1 2024-04-14T15:25:41.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.535+0000 7fe605c97700 10 client.49861.objecter close_session for osd.2 2024-04-14T15:25:41.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.535+0000 7fe605c97700 1 -- 172.21.15.90:0/1251125376 >> v1:172.21.15.90:6809/1686184251 conn(0x7fe5ec008990 legacy=0x7fe5ec013be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:41.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.535+0000 7fe605c97700 20 client.49861.objecter put_session s=0x7fe5ec001900 osd=2 1 2024-04-14T15:25:41.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.535+0000 7fe605c97700 10 client.49861.objecter close_session for osd.3 2024-04-14T15:25:41.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.535+0000 7fe605c97700 1 -- 172.21.15.90:0/1251125376 >> v1:172.21.15.90:6813/2270166774 conn(0x7fe5ec015580 legacy=0x7fe5ec015a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:41.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.535+0000 7fe605c97700 20 client.49861.objecter put_session s=0x7fe5ec0081d0 osd=3 1 2024-04-14T15:25:41.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.535+0000 7fe5fffff700 1 -- 172.21.15.90:0/1251125376 reap_dead start 2024-04-14T15:25:41.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.535+0000 7fe605c97700 10 client.49861.objecter close_session for osd.4 2024-04-14T15:25:41.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.535+0000 7fe605c97700 1 -- 172.21.15.90:0/1251125376 >> v1:172.21.15.92:6800/3887549733 conn(0x7fe5ec01af70 legacy=0x7fe5ec01d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:41.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.535+0000 7fe605c97700 20 client.49861.objecter put_session s=0x7fe5ec016a90 osd=4 1 2024-04-14T15:25:41.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.535+0000 7fe605c97700 10 client.49861.objecter close_session for osd.5 2024-04-14T15:25:41.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.535+0000 7fe5fffff700 1 -- 172.21.15.90:0/1251125376 reap_dead start 2024-04-14T15:25:41.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.535+0000 7fe605c97700 1 -- 172.21.15.90:0/1251125376 >> v1:172.21.15.92:6804/3244561440 conn(0x7fe5ec020c40 legacy=0x7fe5ec01ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:41.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.535+0000 7fe605c97700 20 client.49861.objecter put_session s=0x7fe5ec020520 osd=5 1 2024-04-14T15:25:41.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.535+0000 7fe605c97700 10 client.49861.objecter close_session for osd.6 2024-04-14T15:25:41.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.535+0000 7fe605c97700 1 -- 172.21.15.90:0/1251125376 >> v1:172.21.15.92:6808/636125963 conn(0x7fe5ec023c30 legacy=0x7fe5ec026020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:41.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.535+0000 7fe605c97700 20 client.49861.objecter put_session s=0x7fe5ec021100 osd=6 1 2024-04-14T15:25:41.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.535+0000 7fe605c97700 10 client.49861.objecter close_session for osd.7 2024-04-14T15:25:41.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.535+0000 7fe605c97700 1 -- 172.21.15.90:0/1251125376 >> v1:172.21.15.92:6812/1773706716 conn(0x7fe5ec027dc0 legacy=0x7fe5ec02a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:41.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.535+0000 7fe605c97700 20 client.49861.objecter put_session s=0x7fe5ec027740 osd=7 1 2024-04-14T15:25:41.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.535+0000 7fe605c97700 20 client.49861.objecter shutdown clearing up homeless session... 2024-04-14T15:25:41.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.535+0000 7fe605c97700 10 client.49861.objecter successfully canceled tick 2024-04-14T15:25:41.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.535+0000 7fe605c97700 1 -- 172.21.15.90:0/1251125376 >> v1:172.21.15.90:6800/420935427 conn(0x7fe5e80712c0 legacy=0x7fe5e8073780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:41.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.535+0000 7fe605c97700 1 -- 172.21.15.90:0/1251125376 >> v1:172.21.15.92:6789/0 conn(0x7fe60010f2f0 legacy=0x7fe6001a67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:41.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.535+0000 7fe5fffff700 1 -- 172.21.15.90:0/1251125376 reap_dead start 2024-04-14T15:25:41.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.535+0000 7fe605c97700 1 -- 172.21.15.90:0/1251125376 shutdown_connections 2024-04-14T15:25:41.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.535+0000 7fe605c97700 1 -- 172.21.15.90:0/1251125376 >> 172.21.15.90:0/1251125376 conn(0x7fe6000fe3a0 msgr2=0x7fe6001128a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:25:41.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.536+0000 7fe605c97700 1 -- 172.21.15.90:0/1251125376 shutdown_connections 2024-04-14T15:25:41.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.536+0000 7fe605c97700 1 -- 172.21.15.90:0/1251125376 wait complete. 2024-04-14T15:25:41.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:41.536+0000 7fe605c97700 1 librados: shutdown 2024-04-14T15:25:41.539 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:25:41.540 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:25:41.540 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:25:41.540 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:25:41.540 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:25:41.540 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:25:41.540 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:25:41.540 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:25:41.565 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:41.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:41.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:25:41.667 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:25:41.693 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:41.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:41.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:25:41.740 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:25:41.795 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:25:41.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:41.819 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:41.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:25:41.921 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:25:41.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:41.946 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:41.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:25:42.047 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:25:42.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:42.073 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:42.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:25:42.074 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:42 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1251125376' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:25:42.074 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:42 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1251125376' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:25:42.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:42 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1251125376' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:25:42.175 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:25:42.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:42.200 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:42.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:25:42.302 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:25:42.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:42.327 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:42.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:25:42.429 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:25:42.452 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:42.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:42.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:25:42.554 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:25:42.578 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:42.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:42.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:25:42.680 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:25:42.703 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:42.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:42.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:25:42.805 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:25:42.828 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:42.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:42.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:25:42.930 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:25:42.955 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:42.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:42.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:25:43.056 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:43.081 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:43.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:43.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:43.182 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:25:43.208 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:43.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:43.209 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:25:43.219 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:43 smithi092 ceph-mon[33966]: pgmap v1177: 224 pgs: 1 active+remapped+backfill_toofull, 223 active+clean; 4.2 MiB data, 750 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 10 KiB/s wr, 2 op/s 2024-04-14T15:25:43.294 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:43 smithi090 ceph-mon[41670]: pgmap v1177: 224 pgs: 1 active+remapped+backfill_toofull, 223 active+clean; 4.2 MiB data, 750 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 10 KiB/s wr, 2 op/s 2024-04-14T15:25:43.294 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:43 smithi090 ceph-mon[30399]: pgmap v1177: 224 pgs: 1 active+remapped+backfill_toofull, 223 active+clean; 4.2 MiB data, 750 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 10 KiB/s wr, 2 op/s 2024-04-14T15:25:43.310 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:25:43.335 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:43.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:43.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:25:43.436 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:25:43.460 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:43.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:43.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:25:43.468 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-04-14T15:25:43.468 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:25:43.562 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:25:43.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:43.586 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:43.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:25:43.687 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:25:43.712 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:43.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:43.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:25:43.814 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:25:43.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:43.838 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:43.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:25:43.939 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:25:43.965 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:43.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:43.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:25:44.046 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:25:43 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:25:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:25:44.057 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-14T15:25:44.066 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:25:44.091 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:44.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:44.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:25:44.193 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:44.218 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:44.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:44.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:44.295 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:25:44.312 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:44 smithi090 ceph-mon[41670]: pgmap v1178: 224 pgs: 1 active+remapped+backfill_toofull, 223 active+clean; 4.2 MiB data, 750 MiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 7.0 KiB/s wr, 1 op/s 2024-04-14T15:25:44.312 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:44 smithi090 ceph-mon[30399]: pgmap v1178: 224 pgs: 1 active+remapped+backfill_toofull, 223 active+clean; 4.2 MiB data, 750 MiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 7.0 KiB/s wr, 1 op/s 2024-04-14T15:25:44.320 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:25:44.344 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:44.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:44.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:25:44.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:44 smithi092 ceph-mon[33966]: pgmap v1178: 224 pgs: 1 active+remapped+backfill_toofull, 223 active+clean; 4.2 MiB data, 750 MiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 7.0 KiB/s wr, 1 op/s 2024-04-14T15:25:44.446 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:25:44.469 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:44.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:44.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:25:44.571 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:25:44.596 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:44.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:44.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:25:44.698 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:25:44.721 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:44.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:44.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:25:44.823 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:25:44.848 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:44.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:44.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:25:44.950 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:25:44.974 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:44.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:44.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:25:45.075 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:25:45.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:45.099 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:45.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:25:45.200 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:25:45.224 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:45.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:45.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:25:45.325 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:25:45.350 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:45.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:45.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:25:45.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:45 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:25:45.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:45 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:25:45.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:45 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-140"}]: dispatch 2024-04-14T15:25:45.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:45 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-140"}]: dispatch 2024-04-14T15:25:45.427 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:45 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:25:45.428 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:45 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:25:45.428 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:45 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-140"}]: dispatch 2024-04-14T15:25:45.428 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:45 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-140"}]: dispatch 2024-04-14T15:25:45.428 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:45 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:25:45.428 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:45 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:25:45.428 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:45 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-140"}]: dispatch 2024-04-14T15:25:45.428 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:45 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-140"}]: dispatch 2024-04-14T15:25:45.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.427+0000 7f3d455ce700 1 -- 172.21.15.90:0/2041153222 >> v1:172.21.15.90:6789/0 conn(0x7f3d40107890 legacy=0x7f3d40107c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:45.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.427+0000 7f3d455ce700 1 -- 172.21.15.90:0/2041153222 shutdown_connections 2024-04-14T15:25:45.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.427+0000 7f3d455ce700 1 -- 172.21.15.90:0/2041153222 >> 172.21.15.90:0/2041153222 conn(0x7f3d400fd290 msgr2=0x7f3d400ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:25:45.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.427+0000 7f3d455ce700 1 -- 172.21.15.90:0/2041153222 shutdown_connections 2024-04-14T15:25:45.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.427+0000 7f3d455ce700 1 -- 172.21.15.90:0/2041153222 wait complete. 2024-04-14T15:25:45.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.427+0000 7f3d455ce700 1 librados: starting msgr at 2024-04-14T15:25:45.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.427+0000 7f3d455ce700 1 librados: starting objecter 2024-04-14T15:25:45.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.428+0000 7f3d455ce700 1 Processor -- start 2024-04-14T15:25:45.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.428+0000 7f3d455ce700 1 -- start start 2024-04-14T15:25:45.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.428+0000 7f3d455ce700 1 librados: setting wanted keys 2024-04-14T15:25:45.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.428+0000 7f3d455ce700 1 librados: calling monclient init 2024-04-14T15:25:45.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.428+0000 7f3d455ce700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3d401a7f20 con 0x7f3d4010a740 2024-04-14T15:25:45.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.428+0000 7f3d455ce700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3d401a9100 con 0x7f3d4010e310 2024-04-14T15:25:45.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.428+0000 7f3d455ce700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3d401aa2e0 con 0x7f3d40107890 2024-04-14T15:25:45.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.428+0000 7f3d3e7fc700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f3d4010a740 0x7f3d4010c680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:34820/0 (socket says 172.21.15.90:34820) 2024-04-14T15:25:45.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.428+0000 7f3d3e7fc700 1 -- 172.21.15.90:0/912913031 learned_addr learned my addr 172.21.15.90:0/912913031 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:25:45.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.428+0000 7f3d27fff700 10 client.?.objecter ms_handle_connect 0x7f3d4010a740 2024-04-14T15:25:45.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.428+0000 7f3d27fff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:25:45.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.428+0000 7f3d27fff700 10 client.?.objecter ms_handle_connect 0x7f3d40107890 2024-04-14T15:25:45.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.428+0000 7f3d27fff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:25:45.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.428+0000 7f3d27fff700 1 -- 172.21.15.90:0/912913031 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1095403533 0 0) 0x7f3d401a7f20 con 0x7f3d4010a740 2024-04-14T15:25:45.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.428+0000 7f3d27fff700 1 -- 172.21.15.90:0/912913031 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d280035b0 con 0x7f3d4010a740 2024-04-14T15:25:45.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.428+0000 7f3d27fff700 1 -- 172.21.15.90:0/912913031 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2030854081 0 0) 0x7f3d401aa2e0 con 0x7f3d40107890 2024-04-14T15:25:45.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.429+0000 7f3d27fff700 1 -- 172.21.15.90:0/912913031 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d401a7f20 con 0x7f3d40107890 2024-04-14T15:25:45.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.429+0000 7f3d27fff700 10 client.?.objecter ms_handle_connect 0x7f3d4010e310 2024-04-14T15:25:45.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.429+0000 7f3d27fff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:25:45.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.429+0000 7f3d27fff700 1 -- 172.21.15.90:0/912913031 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3495098453 0 0) 0x7f3d401a9100 con 0x7f3d4010e310 2024-04-14T15:25:45.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.429+0000 7f3d27fff700 1 -- 172.21.15.90:0/912913031 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d401aa2e0 con 0x7f3d4010e310 2024-04-14T15:25:45.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.429+0000 7f3d27fff700 1 -- 172.21.15.90:0/912913031 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3194442020 0 0) 0x7f3d280035b0 con 0x7f3d4010a740 2024-04-14T15:25:45.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.429+0000 7f3d27fff700 1 -- 172.21.15.90:0/912913031 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3d401a9100 con 0x7f3d4010a740 2024-04-14T15:25:45.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.429+0000 7f3d27fff700 1 -- 172.21.15.90:0/912913031 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f3d30001930 con 0x7f3d4010a740 2024-04-14T15:25:45.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.429+0000 7f3d27fff700 1 -- 172.21.15.90:0/912913031 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 269205959 0 0) 0x7f3d401aa2e0 con 0x7f3d4010e310 2024-04-14T15:25:45.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.429+0000 7f3d27fff700 1 -- 172.21.15.90:0/912913031 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3d280035b0 con 0x7f3d4010e310 2024-04-14T15:25:45.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.429+0000 7f3d27fff700 1 -- 172.21.15.90:0/912913031 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f3d38001c30 con 0x7f3d4010e310 2024-04-14T15:25:45.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.429+0000 7f3d27fff700 1 -- 172.21.15.90:0/912913031 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1311163380 0 0) 0x7f3d401a7f20 con 0x7f3d40107890 2024-04-14T15:25:45.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.429+0000 7f3d27fff700 1 -- 172.21.15.90:0/912913031 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3d401aa2e0 con 0x7f3d40107890 2024-04-14T15:25:45.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.429+0000 7f3d27fff700 1 -- 172.21.15.90:0/912913031 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f3d34001830 con 0x7f3d40107890 2024-04-14T15:25:45.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.429+0000 7f3d27fff700 1 -- 172.21.15.90:0/912913031 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1303833949 0 0) 0x7f3d401a9100 con 0x7f3d4010a740 2024-04-14T15:25:45.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.429+0000 7f3d27fff700 1 -- 172.21.15.90:0/912913031 >> v1:172.21.15.90:6790/0 conn(0x7f3d40107890 legacy=0x7f3d4010bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:45.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.429+0000 7f3d27fff700 1 -- 172.21.15.90:0/912913031 >> v1:172.21.15.92:6789/0 conn(0x7f3d4010e310 legacy=0x7f3d401a6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:45.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.429+0000 7f3d27fff700 1 -- 172.21.15.90:0/912913031 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d401ab4c0 con 0x7f3d4010a740 2024-04-14T15:25:45.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.429+0000 7f3d455ce700 1 -- 172.21.15.90:0/912913031 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3d401a8150 con 0x7f3d4010a740 2024-04-14T15:25:45.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.429+0000 7f3d455ce700 10 client.38043.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:25:45.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.429+0000 7f3d455ce700 1 -- 172.21.15.90:0/912913031 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f3d401a8610 con 0x7f3d4010a740 2024-04-14T15:25:45.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.429+0000 7f3d455ce700 1 librados: init done 2024-04-14T15:25:45.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.429+0000 7f3d27fff700 1 -- 172.21.15.90:0/912913031 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f3d30001c50 con 0x7f3d4010a740 2024-04-14T15:25:45.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.429+0000 7f3d27fff700 1 -- 172.21.15.90:0/912913031 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f3d30005cc0 con 0x7f3d4010a740 2024-04-14T15:25:45.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.430+0000 7f3d455ce700 1 -- 172.21.15.90:0/912913031 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d2c005120 con 0x7f3d4010a740 2024-04-14T15:25:45.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.431+0000 7f3d27fff700 1 -- 172.21.15.90:0/912913031 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f3d30003950 con 0x7f3d4010a740 2024-04-14T15:25:45.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.431+0000 7f3d27fff700 1 -- 172.21.15.90:0/912913031 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(795..795 src has 467..795) ==== 8782+0+0 (unknown 1427666450 0 0) 0x7f3d3008ac20 con 0x7f3d4010a740 2024-04-14T15:25:45.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.431+0000 7f3d27fff700 10 client.38043.objecter ms_dispatch 0x7f3d40001550 osd_map(795..795 src has 467..795) 2024-04-14T15:25:45.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.431+0000 7f3d27fff700 3 client.38043.objecter handle_osd_map got epochs [795,795] > 0 2024-04-14T15:25:45.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.431+0000 7f3d27fff700 3 client.38043.objecter handle_osd_map decoding full epoch 795 2024-04-14T15:25:45.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.432+0000 7f3d27fff700 20 client.38043.objecter dump_active .. 0 homeless 2024-04-14T15:25:45.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.432+0000 7f3d27fff700 10 client.38043.objecter ms_handle_connect 0x7f3d280714a0 2024-04-14T15:25:45.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.436+0000 7f3d27fff700 1 -- 172.21.15.90:0/912913031 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f3d300573e0 con 0x7f3d4010a740 2024-04-14T15:25:45.452 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:25:45.475 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:45.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:45.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:25:45.577 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:25:45.601 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:45.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:45.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:25:45.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.650+0000 7f3d455ce700 1 -- 172.21.15.90:0/912913031 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7f3d2c005410 con 0x7f3d4010a740 2024-04-14T15:25:45.703 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:25:45.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.337+0000 7f6d9f9ff700 1 -- 172.21.15.90:0/3602233868 >> v1:172.21.15.90:6790/0 conn(0x7f6d9810f2f0 legacy=0x7f6d981117b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:45.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.337+0000 7f6d9f9ff700 1 -- 172.21.15.90:0/3602233868 shutdown_connections 2024-04-14T15:25:45.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.337+0000 7f6d9f9ff700 1 -- 172.21.15.90:0/3602233868 >> 172.21.15.90:0/3602233868 conn(0x7f6d98076aa0 msgr2=0x7f6d98076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:25:45.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.337+0000 7f6d9f9ff700 1 -- 172.21.15.90:0/3602233868 shutdown_connections 2024-04-14T15:25:45.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.337+0000 7f6d9f9ff700 1 -- 172.21.15.90:0/3602233868 wait complete. 2024-04-14T15:25:45.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.337+0000 7f6d9f9ff700 1 librados: starting msgr at 2024-04-14T15:25:45.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.337+0000 7f6d9f9ff700 1 librados: starting objecter 2024-04-14T15:25:45.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.338+0000 7f6d9f9ff700 1 Processor -- start 2024-04-14T15:25:45.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.338+0000 7f6d9f9ff700 1 -- start start 2024-04-14T15:25:45.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.338+0000 7f6d9f9ff700 1 librados: setting wanted keys 2024-04-14T15:25:45.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.338+0000 7f6d9f9ff700 1 librados: calling monclient init 2024-04-14T15:25:45.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.338+0000 7f6d9f9ff700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6d981a9a70 con 0x7f6d98108980 2024-04-14T15:25:45.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.338+0000 7f6d9f9ff700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6d981aac90 con 0x7f6d9810f2f0 2024-04-14T15:25:45.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.338+0000 7f6d9f9ff700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6d981abed0 con 0x7f6d9810b6f0 2024-04-14T15:25:45.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.338+0000 7f6d9d79b700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f6d98108980 0x7f6d98107be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:34798/0 (socket says 172.21.15.90:34798) 2024-04-14T15:25:45.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.338+0000 7f6d9d79b700 1 -- 172.21.15.90:0/922514973 learned_addr learned my addr 172.21.15.90:0/922514973 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:25:45.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.338+0000 7f6d8e7fc700 10 client.?.objecter ms_handle_connect 0x7f6d98108980 2024-04-14T15:25:45.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.338+0000 7f6d8e7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:25:45.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.338+0000 7f6d8e7fc700 10 client.?.objecter ms_handle_connect 0x7f6d9810f2f0 2024-04-14T15:25:45.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.338+0000 7f6d8e7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:25:45.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.338+0000 7f6d8e7fc700 1 -- 172.21.15.90:0/922514973 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4196659472 0 0) 0x7f6d981a9a70 con 0x7f6d98108980 2024-04-14T15:25:45.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.338+0000 7f6d8e7fc700 1 -- 172.21.15.90:0/922514973 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6d84003aa0 con 0x7f6d98108980 2024-04-14T15:25:45.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.338+0000 7f6d8e7fc700 1 -- 172.21.15.90:0/922514973 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1749614303 0 0) 0x7f6d981aac90 con 0x7f6d9810f2f0 2024-04-14T15:25:45.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.338+0000 7f6d8e7fc700 1 -- 172.21.15.90:0/922514973 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6d981a9a70 con 0x7f6d9810f2f0 2024-04-14T15:25:45.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.338+0000 7f6d8e7fc700 10 client.?.objecter ms_handle_connect 0x7f6d9810b6f0 2024-04-14T15:25:45.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.338+0000 7f6d8e7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:25:45.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.339+0000 7f6d8e7fc700 1 -- 172.21.15.90:0/922514973 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1307310540 0 0) 0x7f6d981a9a70 con 0x7f6d9810f2f0 2024-04-14T15:25:45.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.339+0000 7f6d8e7fc700 1 -- 172.21.15.90:0/922514973 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6d981aac90 con 0x7f6d9810f2f0 2024-04-14T15:25:45.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.339+0000 7f6d8e7fc700 1 -- 172.21.15.90:0/922514973 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f6d900048d0 con 0x7f6d9810f2f0 2024-04-14T15:25:45.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.339+0000 7f6d8e7fc700 1 -- 172.21.15.90:0/922514973 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2985892259 0 0) 0x7f6d84003aa0 con 0x7f6d98108980 2024-04-14T15:25:45.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.339+0000 7f6d8e7fc700 1 -- 172.21.15.90:0/922514973 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6d981a9a70 con 0x7f6d98108980 2024-04-14T15:25:45.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.339+0000 7f6d8e7fc700 1 -- 172.21.15.90:0/922514973 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4222895571 0 0) 0x7f6d981abed0 con 0x7f6d9810b6f0 2024-04-14T15:25:45.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.339+0000 7f6d8e7fc700 1 -- 172.21.15.90:0/922514973 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6d84003aa0 con 0x7f6d9810b6f0 2024-04-14T15:25:45.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.339+0000 7f6d8e7fc700 1 -- 172.21.15.90:0/922514973 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f6d94001940 con 0x7f6d98108980 2024-04-14T15:25:45.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.339+0000 7f6d8e7fc700 1 -- 172.21.15.90:0/922514973 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4039690083 0 0) 0x7f6d981aac90 con 0x7f6d9810f2f0 2024-04-14T15:25:45.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.339+0000 7f6d8e7fc700 1 -- 172.21.15.90:0/922514973 >> v1:172.21.15.90:6790/0 conn(0x7f6d9810b6f0 legacy=0x7f6d981a4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:45.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.339+0000 7f6d8e7fc700 1 -- 172.21.15.90:0/922514973 >> v1:172.21.15.90:6789/0 conn(0x7f6d98108980 legacy=0x7f6d98107be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:45.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.339+0000 7f6d8e7fc700 1 -- 172.21.15.90:0/922514973 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d981ad110 con 0x7f6d9810f2f0 2024-04-14T15:25:45.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.339+0000 7f6d9f9ff700 1 -- 172.21.15.90:0/922514973 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6d981aaec0 con 0x7f6d9810f2f0 2024-04-14T15:25:45.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.339+0000 7f6d9f9ff700 10 client.49867.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:25:45.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.339+0000 7f6d9f9ff700 1 -- 172.21.15.90:0/922514973 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f6d981ab380 con 0x7f6d9810f2f0 2024-04-14T15:25:45.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.339+0000 7f6d9f9ff700 1 librados: init done 2024-04-14T15:25:45.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.339+0000 7f6d8e7fc700 1 -- 172.21.15.90:0/922514973 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f6d90001e30 con 0x7f6d9810f2f0 2024-04-14T15:25:45.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.339+0000 7f6d8e7fc700 1 -- 172.21.15.90:0/922514973 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f6d90005020 con 0x7f6d9810f2f0 2024-04-14T15:25:45.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.340+0000 7f6d9f9ff700 1 -- 172.21.15.90:0/922514973 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d7c005120 con 0x7f6d9810f2f0 2024-04-14T15:25:45.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.340+0000 7f6d8e7fc700 1 -- 172.21.15.90:0/922514973 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f6d90006270 con 0x7f6d9810f2f0 2024-04-14T15:25:45.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.341+0000 7f6d8e7fc700 1 -- 172.21.15.90:0/922514973 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(795..795 src has 467..795) ==== 8782+0+0 (unknown 1427666450 0 0) 0x7f6d9008a050 con 0x7f6d9810f2f0 2024-04-14T15:25:45.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.341+0000 7f6d8e7fc700 10 client.49867.objecter ms_dispatch 0x7f6d98001550 osd_map(795..795 src has 467..795) 2024-04-14T15:25:45.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.341+0000 7f6d8e7fc700 3 client.49867.objecter handle_osd_map got epochs [795,795] > 0 2024-04-14T15:25:45.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.341+0000 7f6d8e7fc700 3 client.49867.objecter handle_osd_map decoding full epoch 795 2024-04-14T15:25:45.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.341+0000 7f6d8e7fc700 20 client.49867.objecter dump_active .. 0 homeless 2024-04-14T15:25:45.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.341+0000 7f6d8e7fc700 10 client.49867.objecter ms_handle_connect 0x7f6d840713b0 2024-04-14T15:25:45.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.345+0000 7f6d8e7fc700 1 -- 172.21.15.90:0/922514973 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f6d900567f0 con 0x7f6d9810f2f0 2024-04-14T15:25:45.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:45.563+0000 7f6d9f9ff700 1 -- 172.21.15.90:0/922514973 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd reweight", "id": 5, "weight": 1.0} v 0) -- 0x7f6d7c005410 con 0x7f6d9810f2f0 2024-04-14T15:25:45.726 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:45.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:45.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:25:45.740 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:25:45 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:25:45.829 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:25:45.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:45.853 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:45.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:25:45.954 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:25:45.978 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:45.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:45.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:25:46.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:46.017+0000 7f3d27fff700 1 -- 172.21.15.90:0/912913031 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v796) ==== 91+0+0 (unknown 1653485850 0 0) 0x7f3d3005b090 con 0x7f3d4010a740 2024-04-14T15:25:46.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:46.019+0000 7f3d455ce700 10 librados: watch_flush enter 2024-04-14T15:25:46.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:46.019+0000 7f3d455ce700 10 librados: watch_flush exit 2024-04-14T15:25:46.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:46.019+0000 7f3d455ce700 20 client.38043.objecter shutdown clearing up homeless session... 2024-04-14T15:25:46.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:46.019+0000 7f3d455ce700 10 client.38043.objecter successfully canceled tick 2024-04-14T15:25:46.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:46.019+0000 7f3d455ce700 1 -- 172.21.15.90:0/912913031 >> v1:172.21.15.90:6800/420935427 conn(0x7f3d280714a0 legacy=0x7f3d28073960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:46.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:46.019+0000 7f3d455ce700 1 -- 172.21.15.90:0/912913031 >> v1:172.21.15.90:6789/0 conn(0x7f3d4010a740 legacy=0x7f3d4010c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:46.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:46.019+0000 7f3d455ce700 1 -- 172.21.15.90:0/912913031 shutdown_connections 2024-04-14T15:25:46.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:46.019+0000 7f3d455ce700 1 -- 172.21.15.90:0/912913031 >> 172.21.15.90:0/912913031 conn(0x7f3d400fd290 msgr2=0x7f3d400fffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:25:46.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:46.019+0000 7f3d455ce700 1 -- 172.21.15.90:0/912913031 shutdown_connections 2024-04-14T15:25:46.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:46.019+0000 7f3d455ce700 1 -- 172.21.15.90:0/912913031 wait complete. 2024-04-14T15:25:46.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:46.019+0000 7f3d455ce700 1 librados: shutdown 2024-04-14T15:25:46.021 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is unset 2024-04-14T15:25:46.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:46.021+0000 7f6d8e7fc700 1 -- 172.21.15.90:0/922514973 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 5, "weight": 1.0}]=0 reweighted osd.5 to 1 (10000) v796) ==== 113+0+0 (unknown 1407482935 0 0) 0x7f6d9005a4a0 con 0x7f6d9810f2f0 2024-04-14T15:25:46.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:46.025+0000 7f6d9f9ff700 10 librados: watch_flush enter 2024-04-14T15:25:46.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:46.025+0000 7f6d9f9ff700 10 librados: watch_flush exit 2024-04-14T15:25:46.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:46.025+0000 7f6d9f9ff700 20 client.49867.objecter shutdown clearing up homeless session... 2024-04-14T15:25:46.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:46.025+0000 7f6d9f9ff700 10 client.49867.objecter successfully canceled tick 2024-04-14T15:25:46.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:46.025+0000 7f6d9f9ff700 1 -- 172.21.15.90:0/922514973 >> v1:172.21.15.90:6800/420935427 conn(0x7f6d840713b0 legacy=0x7f6d84073870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:46.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:46.025+0000 7f6d9f9ff700 1 -- 172.21.15.90:0/922514973 >> v1:172.21.15.92:6789/0 conn(0x7f6d9810f2f0 legacy=0x7f6d981a82c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:46.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:46.025+0000 7f6d9f9ff700 1 -- 172.21.15.90:0/922514973 shutdown_connections 2024-04-14T15:25:46.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:46.025+0000 7f6d9f9ff700 1 -- 172.21.15.90:0/922514973 >> 172.21.15.90:0/922514973 conn(0x7f6d98076aa0 msgr2=0x7f6d98109620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:25:46.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:46.026+0000 7f6d9f9ff700 1 -- 172.21.15.90:0/922514973 shutdown_connections 2024-04-14T15:25:46.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:46.028+0000 7f6d9f9ff700 1 -- 172.21.15.90:0/922514973 wait complete. 2024-04-14T15:25:46.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:46.028+0000 7f6d9f9ff700 1 librados: shutdown 2024-04-14T15:25:46.032 INFO:teuthology.orchestra.run.smithi090.stderr:reweighted osd.5 to 1 (10000) 2024-04-14T15:25:46.079 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:25:46.103 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:46.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:46.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:25:46.201 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-14T15:25:46.204 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:25:46.228 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:46.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:46.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:25:46.268 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-04-14T15:25:46.268 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:25:46.329 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:25:46.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:46.354 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:46.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:25:46.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:46 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-140"}]': finished 2024-04-14T15:25:46.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:46 smithi092 ceph-mon[33966]: osdmap e795: 8 total, 8 up, 7 in 2024-04-14T15:25:46.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:46 smithi092 ceph-mon[33966]: pgmap v1180: 224 pgs: 1 active+remapped+backfill_toofull, 223 active+clean; 4.3 MiB data, 751 MiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 6.8 KiB/s wr, 1 op/s 2024-04-14T15:25:46.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:46 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:25:46.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:46 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/922514973' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-14T15:25:46.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:46 smithi092 ceph-mon[33966]: from='client.49867 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-14T15:25:46.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:46 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/912913031' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:25:46.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:46 smithi092 ceph-mon[33966]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:25:46.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:46 smithi092 ceph-mon[33966]: from='client.49867 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-14T15:25:46.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:46 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/912913031' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-14T15:25:46.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:46 smithi092 ceph-mon[33966]: osdmap e796: 8 total, 8 up, 7 in 2024-04-14T15:25:46.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:46 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-140"}]': finished 2024-04-14T15:25:46.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:46 smithi090 ceph-mon[41670]: osdmap e795: 8 total, 8 up, 7 in 2024-04-14T15:25:46.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:46 smithi090 ceph-mon[41670]: pgmap v1180: 224 pgs: 1 active+remapped+backfill_toofull, 223 active+clean; 4.3 MiB data, 751 MiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 6.8 KiB/s wr, 1 op/s 2024-04-14T15:25:46.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:46 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:25:46.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:46 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/922514973' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-14T15:25:46.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:46 smithi090 ceph-mon[41670]: from='client.49867 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-14T15:25:46.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:46 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/912913031' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:25:46.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:46 smithi090 ceph-mon[41670]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:25:46.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:46 smithi090 ceph-mon[41670]: from='client.49867 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-14T15:25:46.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:46 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/912913031' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-14T15:25:46.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:46 smithi090 ceph-mon[41670]: osdmap e796: 8 total, 8 up, 7 in 2024-04-14T15:25:46.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:46 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-140"}]': finished 2024-04-14T15:25:46.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:46 smithi090 ceph-mon[30399]: osdmap e795: 8 total, 8 up, 7 in 2024-04-14T15:25:46.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:46 smithi090 ceph-mon[30399]: pgmap v1180: 224 pgs: 1 active+remapped+backfill_toofull, 223 active+clean; 4.3 MiB data, 751 MiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 6.8 KiB/s wr, 1 op/s 2024-04-14T15:25:46.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:46 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:25:46.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:46 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/922514973' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-14T15:25:46.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:46 smithi090 ceph-mon[30399]: from='client.49867 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-14T15:25:46.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:46 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/912913031' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:25:46.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:46 smithi090 ceph-mon[30399]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:25:46.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:46 smithi090 ceph-mon[30399]: from='client.49867 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-14T15:25:46.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:46 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/912913031' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-14T15:25:46.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:46 smithi090 ceph-mon[30399]: osdmap e796: 8 total, 8 up, 7 in 2024-04-14T15:25:46.456 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:25:46.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:46.482 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:46.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:25:46.583 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:25:46.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:46.607 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:46.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:25:46.708 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:25:46.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:46.732 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:46.732 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:25:46.833 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:25:46.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:46.857 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:46.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:25:46.928 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-14T15:25:46.958 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:25:46.982 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:46.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:46.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:25:47.084 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:25:47.107 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:47.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:47.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:25:47.208 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:47.215 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:25:47.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:47.232 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:47.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:47.333 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:47.336 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:47 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:25:47.336 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:47 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:25:47.337 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:47 smithi092 ceph-mon[33966]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:25:47.337 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:47 smithi092 ceph-mon[33966]: 323.5s0 scrub starts 2024-04-14T15:25:47.337 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:47 smithi092 ceph-mon[33966]: 323.5 scrub ok 2024-04-14T15:25:47.337 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:47 smithi092 ceph-mon[33966]: osdmap e797: 8 total, 8 up, 7 in 2024-04-14T15:25:47.337 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:47 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:25:47.337 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:47 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:25:47.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:47.357 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:47.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:47.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:47 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:25:47.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:47 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:25:47.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:47 smithi090 ceph-mon[41670]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:25:47.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:47 smithi090 ceph-mon[41670]: 323.5s0 scrub starts 2024-04-14T15:25:47.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:47 smithi090 ceph-mon[41670]: 323.5 scrub ok 2024-04-14T15:25:47.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:47 smithi090 ceph-mon[41670]: osdmap e797: 8 total, 8 up, 7 in 2024-04-14T15:25:47.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:47 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:25:47.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:47 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:25:47.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:47 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:25:47.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:47 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:25:47.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:47 smithi090 ceph-mon[30399]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:25:47.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:47 smithi090 ceph-mon[30399]: 323.5s0 scrub starts 2024-04-14T15:25:47.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:47 smithi090 ceph-mon[30399]: 323.5 scrub ok 2024-04-14T15:25:47.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:47 smithi090 ceph-mon[30399]: osdmap e797: 8 total, 8 up, 7 in 2024-04-14T15:25:47.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:47 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:25:47.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:47 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:25:47.458 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:25:47.483 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:47.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:47.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:25:47.512 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd reweight 7 1 2024-04-14T15:25:47.563 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:25:47.584 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:25:47.607 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:47.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:47.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:25:47.709 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:25:47.733 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:47.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:47.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:25:47.835 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:25:47.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:47.860 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:47.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:25:47.961 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:25:47.984 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:47.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:47.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:25:48.086 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:25:48.109 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:48.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:48.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:25:48.210 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:48.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:48.234 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:48.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:48.335 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:25:48.359 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:48.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:48.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:25:48.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:48 smithi092 ceph-mon[33966]: 2.1b scrub starts 2024-04-14T15:25:48.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:48 smithi092 ceph-mon[33966]: 2.1b scrub ok 2024-04-14T15:25:48.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:48 smithi092 ceph-mon[33966]: 11.11 scrub starts 2024-04-14T15:25:48.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:48 smithi092 ceph-mon[33966]: 11.11 scrub ok 2024-04-14T15:25:48.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:48 smithi092 ceph-mon[33966]: 2.7 scrub starts 2024-04-14T15:25:48.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:48 smithi092 ceph-mon[33966]: 2.7 scrub ok 2024-04-14T15:25:48.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:48 smithi092 ceph-mon[33966]: 2.d scrub starts 2024-04-14T15:25:48.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:48 smithi092 ceph-mon[33966]: 2.d scrub ok 2024-04-14T15:25:48.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:48 smithi092 ceph-mon[33966]: pgmap v1183: 224 pgs: 32 unknown, 1 active+remapped+backfill_toofull, 191 active+clean; 4.2 MiB data, 751 MiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:25:48.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:48 smithi092 ceph-mon[33966]: 4.1 scrub starts 2024-04-14T15:25:48.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:48 smithi092 ceph-mon[33966]: 4.1 scrub ok 2024-04-14T15:25:48.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:48 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:25:48.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:48 smithi092 ceph-mon[33966]: osdmap e798: 8 total, 8 up, 7 in 2024-04-14T15:25:48.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:48 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:25:48.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:48 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:25:48.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:48 smithi090 ceph-mon[41670]: 2.1b scrub starts 2024-04-14T15:25:48.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:48 smithi090 ceph-mon[41670]: 2.1b scrub ok 2024-04-14T15:25:48.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:48 smithi090 ceph-mon[41670]: 11.11 scrub starts 2024-04-14T15:25:48.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:48 smithi090 ceph-mon[41670]: 11.11 scrub ok 2024-04-14T15:25:48.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:48 smithi090 ceph-mon[41670]: 2.7 scrub starts 2024-04-14T15:25:48.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:48 smithi090 ceph-mon[41670]: 2.7 scrub ok 2024-04-14T15:25:48.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:48 smithi090 ceph-mon[41670]: 2.d scrub starts 2024-04-14T15:25:48.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:48 smithi090 ceph-mon[41670]: 2.d scrub ok 2024-04-14T15:25:48.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:48 smithi090 ceph-mon[41670]: pgmap v1183: 224 pgs: 32 unknown, 1 active+remapped+backfill_toofull, 191 active+clean; 4.2 MiB data, 751 MiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:25:48.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:48 smithi090 ceph-mon[41670]: 4.1 scrub starts 2024-04-14T15:25:48.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:48 smithi090 ceph-mon[41670]: 4.1 scrub ok 2024-04-14T15:25:48.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:48 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:25:48.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:48 smithi090 ceph-mon[41670]: osdmap e798: 8 total, 8 up, 7 in 2024-04-14T15:25:48.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:48 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:25:48.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:48 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:25:48.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:48 smithi090 ceph-mon[30399]: 2.1b scrub starts 2024-04-14T15:25:48.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:48 smithi090 ceph-mon[30399]: 2.1b scrub ok 2024-04-14T15:25:48.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:48 smithi090 ceph-mon[30399]: 11.11 scrub starts 2024-04-14T15:25:48.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:48 smithi090 ceph-mon[30399]: 11.11 scrub ok 2024-04-14T15:25:48.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:48 smithi090 ceph-mon[30399]: 2.7 scrub starts 2024-04-14T15:25:48.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:48 smithi090 ceph-mon[30399]: 2.7 scrub ok 2024-04-14T15:25:48.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:48 smithi090 ceph-mon[30399]: 2.d scrub starts 2024-04-14T15:25:48.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:48 smithi090 ceph-mon[30399]: 2.d scrub ok 2024-04-14T15:25:48.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:48 smithi090 ceph-mon[30399]: pgmap v1183: 224 pgs: 32 unknown, 1 active+remapped+backfill_toofull, 191 active+clean; 4.2 MiB data, 751 MiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:25:48.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:48 smithi090 ceph-mon[30399]: 4.1 scrub starts 2024-04-14T15:25:48.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:48 smithi090 ceph-mon[30399]: 4.1 scrub ok 2024-04-14T15:25:48.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:48 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:25:48.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:48 smithi090 ceph-mon[30399]: osdmap e798: 8 total, 8 up, 7 in 2024-04-14T15:25:48.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:48 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:25:48.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:48 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:25:48.461 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:48.485 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:48.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:48.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:48.586 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:25:48.610 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:48.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:48.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:25:48.712 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:25:48.735 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:48.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:48.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:25:48.837 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:25:48.861 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:48.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:48.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:25:48.962 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:25:48.973 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-04-14T15:25:48.973 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:25:48.988 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:48.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:48.989 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:25:48.999 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-14T15:25:49.090 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:25:49.116 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:49.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:49.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:25:49.217 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:25:49.240 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:49.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:49.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:25:49.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.317+0000 7f9736ff8700 1 -- 172.21.15.90:0/3599234885 >> v1:172.21.15.90:6789/0 conn(0x7f97301012c0 legacy=0x7f97301016a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:49.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.317+0000 7f9736ff8700 1 -- 172.21.15.90:0/3599234885 shutdown_connections 2024-04-14T15:25:49.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.317+0000 7f9736ff8700 1 -- 172.21.15.90:0/3599234885 >> 172.21.15.90:0/3599234885 conn(0x7f9730076a70 msgr2=0x7f9730076e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:25:49.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.317+0000 7f9736ff8700 1 -- 172.21.15.90:0/3599234885 shutdown_connections 2024-04-14T15:25:49.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.317+0000 7f9736ff8700 1 -- 172.21.15.90:0/3599234885 wait complete. 2024-04-14T15:25:49.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.318+0000 7f9736ff8700 1 librados: starting msgr at 2024-04-14T15:25:49.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.318+0000 7f9736ff8700 1 librados: starting objecter 2024-04-14T15:25:49.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.318+0000 7f9736ff8700 1 Processor -- start 2024-04-14T15:25:49.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.318+0000 7f9736ff8700 1 -- start start 2024-04-14T15:25:49.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.318+0000 7f9736ff8700 1 librados: setting wanted keys 2024-04-14T15:25:49.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.318+0000 7f9736ff8700 1 librados: calling monclient init 2024-04-14T15:25:49.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.318+0000 7f9736ff8700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f97301a9c00 con 0x7f973010f360 2024-04-14T15:25:49.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.318+0000 7f9736ff8700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f97301aae20 con 0x7f973010b7c0 2024-04-14T15:25:49.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.318+0000 7f9736ff8700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f97301ac060 con 0x7f97301012c0 2024-04-14T15:25:49.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.318+0000 7f9735595700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f973010f360 0x7f97301a8310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:34834/0 (socket says 172.21.15.90:34834) 2024-04-14T15:25:49.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.318+0000 7f9734d94700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f97301012c0 0x7f9730107dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:46538/0 (socket says 172.21.15.90:46538) 2024-04-14T15:25:49.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.318+0000 7f9735595700 1 -- 172.21.15.90:0/3488307638 learned_addr learned my addr 172.21.15.90:0/3488307638 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:25:49.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.318+0000 7f972dffb700 10 client.?.objecter ms_handle_connect 0x7f973010f360 2024-04-14T15:25:49.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.318+0000 7f972dffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:25:49.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.318+0000 7f972dffb700 10 client.?.objecter ms_handle_connect 0x7f97301012c0 2024-04-14T15:25:49.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.318+0000 7f972dffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:25:49.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.318+0000 7f972dffb700 1 -- 172.21.15.90:0/3488307638 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2643656765 0 0) 0x7f97301a9c00 con 0x7f973010f360 2024-04-14T15:25:49.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.318+0000 7f972dffb700 1 -- 172.21.15.90:0/3488307638 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f97200035b0 con 0x7f973010f360 2024-04-14T15:25:49.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.318+0000 7f972dffb700 10 client.?.objecter ms_handle_connect 0x7f973010b7c0 2024-04-14T15:25:49.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.318+0000 7f972dffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:25:49.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.318+0000 7f972dffb700 1 -- 172.21.15.90:0/3488307638 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3977681357 0 0) 0x7f97301ac060 con 0x7f97301012c0 2024-04-14T15:25:49.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.318+0000 7f972dffb700 1 -- 172.21.15.90:0/3488307638 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f97301a9c00 con 0x7f97301012c0 2024-04-14T15:25:49.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.319+0000 7f972dffb700 1 -- 172.21.15.90:0/3488307638 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4005755311 0 0) 0x7f97301aae20 con 0x7f973010b7c0 2024-04-14T15:25:49.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.319+0000 7f972dffb700 1 -- 172.21.15.90:0/3488307638 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f97301ac060 con 0x7f973010b7c0 2024-04-14T15:25:49.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.319+0000 7f972dffb700 1 -- 172.21.15.90:0/3488307638 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3012898097 0 0) 0x7f97301a9c00 con 0x7f97301012c0 2024-04-14T15:25:49.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.319+0000 7f972dffb700 1 -- 172.21.15.90:0/3488307638 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f97301aae20 con 0x7f97301012c0 2024-04-14T15:25:49.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.319+0000 7f972dffb700 1 -- 172.21.15.90:0/3488307638 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1900539007 0 0) 0x7f97200035b0 con 0x7f973010f360 2024-04-14T15:25:49.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.319+0000 7f972dffb700 1 -- 172.21.15.90:0/3488307638 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f97301a9c00 con 0x7f973010f360 2024-04-14T15:25:49.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.319+0000 7f972dffb700 1 -- 172.21.15.90:0/3488307638 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f9724004680 con 0x7f97301012c0 2024-04-14T15:25:49.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.319+0000 7f972dffb700 1 -- 172.21.15.90:0/3488307638 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f9728001650 con 0x7f973010f360 2024-04-14T15:25:49.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.319+0000 7f972dffb700 1 -- 172.21.15.90:0/3488307638 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3078039650 0 0) 0x7f97301ac060 con 0x7f973010b7c0 2024-04-14T15:25:49.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.319+0000 7f972dffb700 1 -- 172.21.15.90:0/3488307638 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f97200035b0 con 0x7f973010b7c0 2024-04-14T15:25:49.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.319+0000 7f972dffb700 1 -- 172.21.15.90:0/3488307638 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f971c0017f0 con 0x7f973010b7c0 2024-04-14T15:25:49.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.319+0000 7f972dffb700 1 -- 172.21.15.90:0/3488307638 <== mon.2 v1:172.21.15.90:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3238394249 0 0) 0x7f97301aae20 con 0x7f97301012c0 2024-04-14T15:25:49.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.319+0000 7f972dffb700 1 -- 172.21.15.90:0/3488307638 >> v1:172.21.15.92:6789/0 conn(0x7f973010b7c0 legacy=0x7f97301a4be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:49.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.319+0000 7f972dffb700 1 -- 172.21.15.90:0/3488307638 >> v1:172.21.15.90:6789/0 conn(0x7f973010f360 legacy=0x7f97301a8310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:49.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.319+0000 7f972dffb700 1 -- 172.21.15.90:0/3488307638 --> v1:172.21.15.90:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97301ad2a0 con 0x7f97301012c0 2024-04-14T15:25:49.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.319+0000 7f9736ff8700 1 -- 172.21.15.90:0/3488307638 --> v1:172.21.15.90:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f97301ac290 con 0x7f97301012c0 2024-04-14T15:25:49.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.319+0000 7f972dffb700 1 -- 172.21.15.90:0/3488307638 <== mon.2 v1:172.21.15.90:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f9724001af0 con 0x7f97301012c0 2024-04-14T15:25:49.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.319+0000 7f972dffb700 1 -- 172.21.15.90:0/3488307638 <== mon.2 v1:172.21.15.90:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f9724005050 con 0x7f97301012c0 2024-04-14T15:25:49.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.319+0000 7f9736ff8700 10 client.47999.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:25:49.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.319+0000 7f9736ff8700 1 -- 172.21.15.90:0/3488307638 --> v1:172.21.15.90:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f97301ac750 con 0x7f97301012c0 2024-04-14T15:25:49.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.319+0000 7f9736ff8700 1 librados: init done 2024-04-14T15:25:49.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.320+0000 7f9736ff8700 1 -- 172.21.15.90:0/3488307638 --> v1:172.21.15.90:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f973006ab20 con 0x7f97301012c0 2024-04-14T15:25:49.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.320+0000 7f972dffb700 1 -- 172.21.15.90:0/3488307638 <== mon.2 v1:172.21.15.90:6790/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f9724005320 con 0x7f97301012c0 2024-04-14T15:25:49.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.321+0000 7f972dffb700 1 -- 172.21.15.90:0/3488307638 <== mon.2 v1:172.21.15.90:6790/0 8 ==== osd_map(799..799 src has 467..799) ==== 8790+0+0 (unknown 3320550733 0 0) 0x7f972408ae30 con 0x7f97301012c0 2024-04-14T15:25:49.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.321+0000 7f972dffb700 10 client.47999.objecter ms_dispatch 0x7f9730001550 osd_map(799..799 src has 467..799) 2024-04-14T15:25:49.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.321+0000 7f972dffb700 3 client.47999.objecter handle_osd_map got epochs [799,799] > 0 2024-04-14T15:25:49.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.321+0000 7f972dffb700 3 client.47999.objecter handle_osd_map decoding full epoch 799 2024-04-14T15:25:49.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.321+0000 7f972dffb700 20 client.47999.objecter dump_active .. 0 homeless 2024-04-14T15:25:49.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.321+0000 7f972dffb700 1 -- 172.21.15.90:0/3488307638 <== mon.2 v1:172.21.15.90:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v799) ==== 54+0+16 (unknown 3929138454 0 2420000192) 0x7f972408d290 con 0x7f97301012c0 2024-04-14T15:25:49.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.321+0000 7f972dffb700 10 client.47999.objecter ms_handle_connect 0x7f9720071360 2024-04-14T15:25:49.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.321+0000 7f9736ff8700 10 client.47999.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:25:49.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.321+0000 7f9736ff8700 15 client.47999.objecter _session_command_op_assign -1 1 2024-04-14T15:25:49.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.321+0000 7f9736ff8700 20 client.47999.objecter _get_session s=0x7f9730113310 osd=0 3 2024-04-14T15:25:49.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.321+0000 7f9736ff8700 20 client.47999.objecter put_session s=0x7f9730113310 osd=0 3 2024-04-14T15:25:49.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.321+0000 7f9736ff8700 20 client.47999.objecter _get_session s=0x7f9730113310 osd=0 3 2024-04-14T15:25:49.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.321+0000 7f9736ff8700 15 client.47999.objecter _session_command_op_remove -1 1 2024-04-14T15:25:49.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.321+0000 7f9736ff8700 20 client.47999.objecter get_session s=0x7f9730113310 osd=0 3 2024-04-14T15:25:49.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.321+0000 7f9736ff8700 15 client.47999.objecter _session_command_op_assign 0 1 2024-04-14T15:25:49.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.321+0000 7f9736ff8700 20 client.47999.objecter put_session s=0x7f9730113310 osd=0 4 2024-04-14T15:25:49.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.321+0000 7f9736ff8700 10 client.47999.objecter _send_command 1 2024-04-14T15:25:49.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.321+0000 7f9736ff8700 1 -- 172.21.15.90:0/3488307638 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9730103770 con 0x7f97301aaff0 2024-04-14T15:25:49.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.322+0000 7f972dffb700 10 client.47999.objecter ms_handle_connect 0x7f97301aaff0 2024-04-14T15:25:49.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.322+0000 7f972dffb700 1 -- 172.21.15.90:0/3488307638 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f9730103770 con 0x7f97301aaff0 2024-04-14T15:25:49.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.322+0000 7f972dffb700 10 client.47999.objecter ms_dispatch 0x7f9730001550 command_reply(tid 1: 0 ) 2024-04-14T15:25:49.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.322+0000 7f972dffb700 10 client.47999.objecter _finish_command 1 = system:0 2024-04-14T15:25:49.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.322+0000 7f972dffb700 20 client.47999.objecter put_session s=0x7f9730113310 osd=0 4 2024-04-14T15:25:49.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.322+0000 7f972dffb700 15 client.47999.objecter _session_command_op_remove 0 1 2024-04-14T15:25:49.332 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:25:49.342 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:25:49.343 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:49 smithi090 ceph-mon[41670]: 5.14 scrub starts 2024-04-14T15:25:49.343 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:49 smithi090 ceph-mon[41670]: 5.14 scrub ok 2024-04-14T15:25:49.343 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:49 smithi090 ceph-mon[41670]: 2.9 scrub starts 2024-04-14T15:25:49.343 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:49 smithi090 ceph-mon[41670]: 2.9 scrub ok 2024-04-14T15:25:49.343 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:49 smithi090 ceph-mon[41670]: 2.13 scrub starts 2024-04-14T15:25:49.343 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:49 smithi090 ceph-mon[41670]: 2.13 scrub ok 2024-04-14T15:25:49.344 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:49 smithi090 ceph-mon[41670]: 6.1e scrub starts 2024-04-14T15:25:49.344 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:49 smithi090 ceph-mon[41670]: 6.1e scrub ok 2024-04-14T15:25:49.344 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:49 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-142", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:25:49.344 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:49 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-142"}]: dispatch 2024-04-14T15:25:49.344 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:49 smithi090 ceph-mon[41670]: osdmap e799: 8 total, 8 up, 7 in 2024-04-14T15:25:49.344 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:49 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-142"}]: dispatch 2024-04-14T15:25:49.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.342+0000 7f9736ff8700 10 client.47999.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:25:49.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.342+0000 7f9736ff8700 15 client.47999.objecter _session_command_op_assign -1 2 2024-04-14T15:25:49.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.342+0000 7f9736ff8700 20 client.47999.objecter _get_session s=0x7f9730113310 osd=0 3 2024-04-14T15:25:49.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.342+0000 7f9736ff8700 20 client.47999.objecter put_session s=0x7f9730113310 osd=0 3 2024-04-14T15:25:49.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.342+0000 7f9736ff8700 20 client.47999.objecter _get_session s=0x7f9730113310 osd=0 3 2024-04-14T15:25:49.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.342+0000 7f9736ff8700 15 client.47999.objecter _session_command_op_remove -1 2 2024-04-14T15:25:49.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.342+0000 7f9736ff8700 20 client.47999.objecter get_session s=0x7f9730113310 osd=0 3 2024-04-14T15:25:49.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.342+0000 7f9736ff8700 15 client.47999.objecter _session_command_op_assign 0 2 2024-04-14T15:25:49.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.342+0000 7f9736ff8700 20 client.47999.objecter put_session s=0x7f9730113310 osd=0 4 2024-04-14T15:25:49.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.342+0000 7f9736ff8700 10 client.47999.objecter _send_command 2 2024-04-14T15:25:49.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.342+0000 7f9736ff8700 1 -- 172.21.15.90:0/3488307638 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f97300693d0 con 0x7f97301aaff0 2024-04-14T15:25:49.345 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:49 smithi090 ceph-mon[30399]: 5.14 scrub starts 2024-04-14T15:25:49.345 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:49 smithi090 ceph-mon[30399]: 5.14 scrub ok 2024-04-14T15:25:49.345 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:49 smithi090 ceph-mon[30399]: 2.9 scrub starts 2024-04-14T15:25:49.345 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:49 smithi090 ceph-mon[30399]: 2.9 scrub ok 2024-04-14T15:25:49.345 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:49 smithi090 ceph-mon[30399]: 2.13 scrub starts 2024-04-14T15:25:49.346 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:49 smithi090 ceph-mon[30399]: 2.13 scrub ok 2024-04-14T15:25:49.346 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:49 smithi090 ceph-mon[30399]: 6.1e scrub starts 2024-04-14T15:25:49.346 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:49 smithi090 ceph-mon[30399]: 6.1e scrub ok 2024-04-14T15:25:49.346 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:49 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-142", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:25:49.346 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:49 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-142"}]: dispatch 2024-04-14T15:25:49.346 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:49 smithi090 ceph-mon[30399]: osdmap e799: 8 total, 8 up, 7 in 2024-04-14T15:25:49.346 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:49 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-142"}]: dispatch 2024-04-14T15:25:49.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.344+0000 7f972dffb700 1 -- 172.21.15.90:0/3488307638 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f97300693d0 con 0x7f97301aaff0 2024-04-14T15:25:49.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.344+0000 7f972dffb700 10 client.47999.objecter ms_dispatch 0x7f9730001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:25:49.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.344+0000 7f972dffb700 10 client.47999.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:25:49.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.344+0000 7f972dffb700 20 client.47999.objecter put_session s=0x7f9730113310 osd=0 4 2024-04-14T15:25:49.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.344+0000 7f972dffb700 15 client.47999.objecter _session_command_op_remove 0 2 2024-04-14T15:25:49.347 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:25:49.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.345+0000 7f9736ff8700 10 client.47999.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:25:49.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.345+0000 7f9736ff8700 15 client.47999.objecter _session_command_op_assign -1 3 2024-04-14T15:25:49.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.345+0000 7f9736ff8700 20 client.47999.objecter _get_session s=0x7f97301abd30 osd=1 3 2024-04-14T15:25:49.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.345+0000 7f9736ff8700 20 client.47999.objecter put_session s=0x7f97301abd30 osd=1 3 2024-04-14T15:25:49.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.345+0000 7f9736ff8700 20 client.47999.objecter _get_session s=0x7f97301abd30 osd=1 3 2024-04-14T15:25:49.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.345+0000 7f9736ff8700 15 client.47999.objecter _session_command_op_remove -1 3 2024-04-14T15:25:49.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.345+0000 7f9736ff8700 20 client.47999.objecter get_session s=0x7f97301abd30 osd=1 3 2024-04-14T15:25:49.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.345+0000 7f9736ff8700 15 client.47999.objecter _session_command_op_assign 1 3 2024-04-14T15:25:49.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.345+0000 7f9736ff8700 20 client.47999.objecter put_session s=0x7f97301abd30 osd=1 4 2024-04-14T15:25:49.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.345+0000 7f9736ff8700 10 client.47999.objecter _send_command 3 2024-04-14T15:25:49.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.345+0000 7f9736ff8700 1 -- 172.21.15.90:0/3488307638 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f973002de30 con 0x7f97300fe550 2024-04-14T15:25:49.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.346+0000 7f972dffb700 10 client.47999.objecter ms_handle_connect 0x7f97300fe550 2024-04-14T15:25:49.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.346+0000 7f972dffb700 1 -- 172.21.15.90:0/3488307638 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f973002de30 con 0x7f97300fe550 2024-04-14T15:25:49.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.346+0000 7f972dffb700 10 client.47999.objecter ms_dispatch 0x7f9730001550 command_reply(tid 3: 0 ) 2024-04-14T15:25:49.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.346+0000 7f972dffb700 10 client.47999.objecter _finish_command 3 = system:0 2024-04-14T15:25:49.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.346+0000 7f972dffb700 20 client.47999.objecter put_session s=0x7f97301abd30 osd=1 4 2024-04-14T15:25:49.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.346+0000 7f972dffb700 15 client.47999.objecter _session_command_op_remove 1 3 2024-04-14T15:25:49.366 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:49.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:49.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:25:49.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.371+0000 7f9736ff8700 10 client.47999.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:25:49.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.371+0000 7f9736ff8700 15 client.47999.objecter _session_command_op_assign -1 4 2024-04-14T15:25:49.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.371+0000 7f9736ff8700 20 client.47999.objecter _get_session s=0x7f97301abd30 osd=1 3 2024-04-14T15:25:49.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.371+0000 7f9736ff8700 20 client.47999.objecter put_session s=0x7f97301abd30 osd=1 3 2024-04-14T15:25:49.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.371+0000 7f9736ff8700 20 client.47999.objecter _get_session s=0x7f97301abd30 osd=1 3 2024-04-14T15:25:49.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.371+0000 7f9736ff8700 15 client.47999.objecter _session_command_op_remove -1 4 2024-04-14T15:25:49.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.371+0000 7f9736ff8700 20 client.47999.objecter get_session s=0x7f97301abd30 osd=1 3 2024-04-14T15:25:49.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.371+0000 7f9736ff8700 15 client.47999.objecter _session_command_op_assign 1 4 2024-04-14T15:25:49.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.371+0000 7f9736ff8700 20 client.47999.objecter put_session s=0x7f97301abd30 osd=1 4 2024-04-14T15:25:49.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.371+0000 7f9736ff8700 10 client.47999.objecter _send_command 4 2024-04-14T15:25:49.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.371+0000 7f9736ff8700 1 -- 172.21.15.90:0/3488307638 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f973002e230 con 0x7f97300fe550 2024-04-14T15:25:49.374 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:25:49.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.373+0000 7f972dffb700 1 -- 172.21.15.90:0/3488307638 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f973002e230 con 0x7f97300fe550 2024-04-14T15:25:49.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.373+0000 7f972dffb700 10 client.47999.objecter ms_dispatch 0x7f9730001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:25:49.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.373+0000 7f972dffb700 10 client.47999.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:25:49.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.373+0000 7f972dffb700 20 client.47999.objecter put_session s=0x7f97301abd30 osd=1 4 2024-04-14T15:25:49.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.373+0000 7f972dffb700 15 client.47999.objecter _session_command_op_remove 1 4 2024-04-14T15:25:49.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.374+0000 7f9736ff8700 10 client.47999.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:25:49.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.374+0000 7f9736ff8700 15 client.47999.objecter _session_command_op_assign -1 5 2024-04-14T15:25:49.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.374+0000 7f9736ff8700 20 client.47999.objecter _get_session s=0x7f97301abea0 osd=2 3 2024-04-14T15:25:49.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.374+0000 7f9736ff8700 20 client.47999.objecter put_session s=0x7f97301abea0 osd=2 3 2024-04-14T15:25:49.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.374+0000 7f9736ff8700 20 client.47999.objecter _get_session s=0x7f97301abea0 osd=2 3 2024-04-14T15:25:49.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.374+0000 7f9736ff8700 15 client.47999.objecter _session_command_op_remove -1 5 2024-04-14T15:25:49.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.374+0000 7f9736ff8700 20 client.47999.objecter get_session s=0x7f97301abea0 osd=2 3 2024-04-14T15:25:49.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.374+0000 7f9736ff8700 15 client.47999.objecter _session_command_op_assign 2 5 2024-04-14T15:25:49.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.374+0000 7f9736ff8700 20 client.47999.objecter put_session s=0x7f97301abea0 osd=2 4 2024-04-14T15:25:49.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.374+0000 7f9736ff8700 10 client.47999.objecter _send_command 5 2024-04-14T15:25:49.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.374+0000 7f9736ff8700 1 -- 172.21.15.90:0/3488307638 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f973002e3e0 con 0x7f97300636c0 2024-04-14T15:25:49.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.374+0000 7f972dffb700 10 client.47999.objecter ms_handle_connect 0x7f97300636c0 2024-04-14T15:25:49.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.375+0000 7f972dffb700 1 -- 172.21.15.90:0/3488307638 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f973002e3e0 con 0x7f97300636c0 2024-04-14T15:25:49.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.375+0000 7f972dffb700 10 client.47999.objecter ms_dispatch 0x7f9730001550 command_reply(tid 5: 0 ) 2024-04-14T15:25:49.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.375+0000 7f972dffb700 10 client.47999.objecter _finish_command 5 = system:0 2024-04-14T15:25:49.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.375+0000 7f972dffb700 20 client.47999.objecter put_session s=0x7f97301abea0 osd=2 4 2024-04-14T15:25:49.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.375+0000 7f972dffb700 15 client.47999.objecter _session_command_op_remove 2 5 2024-04-14T15:25:49.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.392+0000 7f9736ff8700 10 client.47999.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:25:49.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.392+0000 7f9736ff8700 15 client.47999.objecter _session_command_op_assign -1 6 2024-04-14T15:25:49.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.392+0000 7f9736ff8700 20 client.47999.objecter _get_session s=0x7f97301abea0 osd=2 3 2024-04-14T15:25:49.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.392+0000 7f9736ff8700 20 client.47999.objecter put_session s=0x7f97301abea0 osd=2 3 2024-04-14T15:25:49.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.392+0000 7f9736ff8700 20 client.47999.objecter _get_session s=0x7f97301abea0 osd=2 3 2024-04-14T15:25:49.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.392+0000 7f9736ff8700 15 client.47999.objecter _session_command_op_remove -1 6 2024-04-14T15:25:49.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.392+0000 7f9736ff8700 20 client.47999.objecter get_session s=0x7f97301abea0 osd=2 3 2024-04-14T15:25:49.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.392+0000 7f9736ff8700 15 client.47999.objecter _session_command_op_assign 2 6 2024-04-14T15:25:49.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.392+0000 7f9736ff8700 20 client.47999.objecter put_session s=0x7f97301abea0 osd=2 4 2024-04-14T15:25:49.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.392+0000 7f9736ff8700 10 client.47999.objecter _send_command 6 2024-04-14T15:25:49.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.392+0000 7f9736ff8700 1 -- 172.21.15.90:0/3488307638 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f9730063de0 con 0x7f97300636c0 2024-04-14T15:25:49.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.393+0000 7f972dffb700 1 -- 172.21.15.90:0/3488307638 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f9730063de0 con 0x7f97300636c0 2024-04-14T15:25:49.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.393+0000 7f972dffb700 10 client.47999.objecter ms_dispatch 0x7f9730001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:25:49.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.393+0000 7f972dffb700 10 client.47999.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:25:49.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.393+0000 7f972dffb700 20 client.47999.objecter put_session s=0x7f97301abea0 osd=2 4 2024-04-14T15:25:49.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.393+0000 7f972dffb700 15 client.47999.objecter _session_command_op_remove 2 6 2024-04-14T15:25:49.394 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:25:49.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.394+0000 7f9736ff8700 10 client.47999.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:25:49.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.394+0000 7f9736ff8700 15 client.47999.objecter _session_command_op_assign -1 7 2024-04-14T15:25:49.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.394+0000 7f9736ff8700 20 client.47999.objecter _get_session s=0x7f9730063140 osd=3 3 2024-04-14T15:25:49.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.394+0000 7f9736ff8700 20 client.47999.objecter put_session s=0x7f9730063140 osd=3 3 2024-04-14T15:25:49.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.394+0000 7f9736ff8700 20 client.47999.objecter _get_session s=0x7f9730063140 osd=3 3 2024-04-14T15:25:49.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.394+0000 7f9736ff8700 15 client.47999.objecter _session_command_op_remove -1 7 2024-04-14T15:25:49.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.394+0000 7f9736ff8700 20 client.47999.objecter get_session s=0x7f9730063140 osd=3 3 2024-04-14T15:25:49.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.394+0000 7f9736ff8700 15 client.47999.objecter _session_command_op_assign 3 7 2024-04-14T15:25:49.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.394+0000 7f9736ff8700 20 client.47999.objecter put_session s=0x7f9730063140 osd=3 4 2024-04-14T15:25:49.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.394+0000 7f9736ff8700 10 client.47999.objecter _send_command 7 2024-04-14T15:25:49.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.394+0000 7f9736ff8700 1 -- 172.21.15.90:0/3488307638 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f97301b2420 con 0x7f97300632b0 2024-04-14T15:25:49.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.394+0000 7f972dffb700 10 client.47999.objecter ms_handle_connect 0x7f97300632b0 2024-04-14T15:25:49.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.395+0000 7f972dffb700 1 -- 172.21.15.90:0/3488307638 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f97301b2420 con 0x7f97300632b0 2024-04-14T15:25:49.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.395+0000 7f972dffb700 10 client.47999.objecter ms_dispatch 0x7f9730001550 command_reply(tid 7: 0 ) 2024-04-14T15:25:49.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.395+0000 7f972dffb700 10 client.47999.objecter _finish_command 7 = system:0 2024-04-14T15:25:49.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.395+0000 7f972dffb700 20 client.47999.objecter put_session s=0x7f9730063140 osd=3 4 2024-04-14T15:25:49.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.395+0000 7f972dffb700 15 client.47999.objecter _session_command_op_remove 3 7 2024-04-14T15:25:49.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.412+0000 7f9736ff8700 10 client.47999.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:25:49.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.412+0000 7f9736ff8700 15 client.47999.objecter _session_command_op_assign -1 8 2024-04-14T15:25:49.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.412+0000 7f9736ff8700 20 client.47999.objecter _get_session s=0x7f9730063140 osd=3 3 2024-04-14T15:25:49.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.412+0000 7f9736ff8700 20 client.47999.objecter put_session s=0x7f9730063140 osd=3 3 2024-04-14T15:25:49.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.412+0000 7f9736ff8700 20 client.47999.objecter _get_session s=0x7f9730063140 osd=3 3 2024-04-14T15:25:49.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.412+0000 7f9736ff8700 15 client.47999.objecter _session_command_op_remove -1 8 2024-04-14T15:25:49.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.412+0000 7f9736ff8700 20 client.47999.objecter get_session s=0x7f9730063140 osd=3 3 2024-04-14T15:25:49.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.412+0000 7f9736ff8700 15 client.47999.objecter _session_command_op_assign 3 8 2024-04-14T15:25:49.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.412+0000 7f9736ff8700 20 client.47999.objecter put_session s=0x7f9730063140 osd=3 4 2024-04-14T15:25:49.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.412+0000 7f9736ff8700 10 client.47999.objecter _send_command 8 2024-04-14T15:25:49.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.412+0000 7f9736ff8700 1 -- 172.21.15.90:0/3488307638 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f97301b1860 con 0x7f97300632b0 2024-04-14T15:25:49.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.414+0000 7f972dffb700 1 -- 172.21.15.90:0/3488307638 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f97301b1860 con 0x7f97300632b0 2024-04-14T15:25:49.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.414+0000 7f972dffb700 10 client.47999.objecter ms_dispatch 0x7f9730001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:25:49.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.414+0000 7f972dffb700 10 client.47999.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:25:49.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.414+0000 7f972dffb700 20 client.47999.objecter put_session s=0x7f9730063140 osd=3 4 2024-04-14T15:25:49.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.414+0000 7f972dffb700 15 client.47999.objecter _session_command_op_remove 3 8 2024-04-14T15:25:49.415 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:25:49.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.414+0000 7f9736ff8700 10 client.47999.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:25:49.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.414+0000 7f9736ff8700 15 client.47999.objecter _session_command_op_assign -1 9 2024-04-14T15:25:49.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.414+0000 7f9736ff8700 20 client.47999.objecter _get_session s=0x7f97301b1610 osd=4 3 2024-04-14T15:25:49.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.414+0000 7f9736ff8700 20 client.47999.objecter put_session s=0x7f97301b1610 osd=4 3 2024-04-14T15:25:49.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.414+0000 7f9736ff8700 20 client.47999.objecter _get_session s=0x7f97301b1610 osd=4 3 2024-04-14T15:25:49.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.414+0000 7f9736ff8700 15 client.47999.objecter _session_command_op_remove -1 9 2024-04-14T15:25:49.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.414+0000 7f9736ff8700 20 client.47999.objecter get_session s=0x7f97301b1610 osd=4 3 2024-04-14T15:25:49.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.414+0000 7f9736ff8700 15 client.47999.objecter _session_command_op_assign 4 9 2024-04-14T15:25:49.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.414+0000 7f9736ff8700 20 client.47999.objecter put_session s=0x7f97301b1610 osd=4 4 2024-04-14T15:25:49.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.414+0000 7f9736ff8700 10 client.47999.objecter _send_command 9 2024-04-14T15:25:49.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.414+0000 7f9736ff8700 1 -- 172.21.15.90:0/3488307638 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f97301b18b0 con 0x7f97301b6ae0 2024-04-14T15:25:49.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.415+0000 7f972dffb700 10 client.47999.objecter ms_handle_connect 0x7f97301b6ae0 2024-04-14T15:25:49.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.415+0000 7f972dffb700 1 -- 172.21.15.90:0/3488307638 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f97240575f0 con 0x7f97301b6ae0 2024-04-14T15:25:49.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.415+0000 7f972dffb700 10 client.47999.objecter ms_dispatch 0x7f9730001550 command_reply(tid 9: 0 ) 2024-04-14T15:25:49.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.415+0000 7f972dffb700 10 client.47999.objecter _finish_command 9 = system:0 2024-04-14T15:25:49.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.415+0000 7f972dffb700 20 client.47999.objecter put_session s=0x7f97301b1610 osd=4 4 2024-04-14T15:25:49.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.415+0000 7f972dffb700 15 client.47999.objecter _session_command_op_remove 4 9 2024-04-14T15:25:49.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.432+0000 7f9736ff8700 10 client.47999.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:25:49.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.432+0000 7f9736ff8700 15 client.47999.objecter _session_command_op_assign -1 10 2024-04-14T15:25:49.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.432+0000 7f9736ff8700 20 client.47999.objecter _get_session s=0x7f97301b1610 osd=4 3 2024-04-14T15:25:49.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.432+0000 7f9736ff8700 20 client.47999.objecter put_session s=0x7f97301b1610 osd=4 3 2024-04-14T15:25:49.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.432+0000 7f9736ff8700 20 client.47999.objecter _get_session s=0x7f97301b1610 osd=4 3 2024-04-14T15:25:49.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.432+0000 7f9736ff8700 15 client.47999.objecter _session_command_op_remove -1 10 2024-04-14T15:25:49.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.432+0000 7f9736ff8700 20 client.47999.objecter get_session s=0x7f97301b1610 osd=4 3 2024-04-14T15:25:49.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.432+0000 7f9736ff8700 15 client.47999.objecter _session_command_op_assign 4 10 2024-04-14T15:25:49.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.432+0000 7f9736ff8700 20 client.47999.objecter put_session s=0x7f97301b1610 osd=4 4 2024-04-14T15:25:49.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.432+0000 7f9736ff8700 10 client.47999.objecter _send_command 10 2024-04-14T15:25:49.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.432+0000 7f9736ff8700 1 -- 172.21.15.90:0/3488307638 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f97301b60b0 con 0x7f97301b6ae0 2024-04-14T15:25:49.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.434+0000 7f972dffb700 1 -- 172.21.15.90:0/3488307638 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f97301b60b0 con 0x7f97301b6ae0 2024-04-14T15:25:49.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.434+0000 7f972dffb700 10 client.47999.objecter ms_dispatch 0x7f9730001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:25:49.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.434+0000 7f972dffb700 10 client.47999.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:25:49.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.434+0000 7f972dffb700 20 client.47999.objecter put_session s=0x7f97301b1610 osd=4 4 2024-04-14T15:25:49.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.434+0000 7f972dffb700 15 client.47999.objecter _session_command_op_remove 4 10 2024-04-14T15:25:49.434 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:25:49.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.434+0000 7f9736ff8700 10 client.47999.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:25:49.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.434+0000 7f9736ff8700 15 client.47999.objecter _session_command_op_assign -1 11 2024-04-14T15:25:49.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.434+0000 7f9736ff8700 20 client.47999.objecter _get_session s=0x7f97301b5e60 osd=5 3 2024-04-14T15:25:49.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.434+0000 7f9736ff8700 20 client.47999.objecter put_session s=0x7f97301b5e60 osd=5 3 2024-04-14T15:25:49.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.434+0000 7f9736ff8700 20 client.47999.objecter _get_session s=0x7f97301b5e60 osd=5 3 2024-04-14T15:25:49.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.434+0000 7f9736ff8700 15 client.47999.objecter _session_command_op_remove -1 11 2024-04-14T15:25:49.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.434+0000 7f9736ff8700 20 client.47999.objecter get_session s=0x7f97301b5e60 osd=5 3 2024-04-14T15:25:49.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.434+0000 7f9736ff8700 15 client.47999.objecter _session_command_op_assign 5 11 2024-04-14T15:25:49.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.434+0000 7f9736ff8700 20 client.47999.objecter put_session s=0x7f97301b5e60 osd=5 4 2024-04-14T15:25:49.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.434+0000 7f9736ff8700 10 client.47999.objecter _send_command 11 2024-04-14T15:25:49.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.434+0000 7f9736ff8700 1 -- 172.21.15.90:0/3488307638 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f97301b6090 con 0x7f97301bb390 2024-04-14T15:25:49.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.434+0000 7f972dffb700 10 client.47999.objecter ms_handle_connect 0x7f97301bb390 2024-04-14T15:25:49.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.435+0000 7f972dffb700 1 -- 172.21.15.90:0/3488307638 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f97301b6090 con 0x7f97301bb390 2024-04-14T15:25:49.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.435+0000 7f972dffb700 10 client.47999.objecter ms_dispatch 0x7f9730001550 command_reply(tid 11: 0 ) 2024-04-14T15:25:49.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.435+0000 7f972dffb700 10 client.47999.objecter _finish_command 11 = system:0 2024-04-14T15:25:49.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.435+0000 7f972dffb700 20 client.47999.objecter put_session s=0x7f97301b5e60 osd=5 4 2024-04-14T15:25:49.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.435+0000 7f972dffb700 15 client.47999.objecter _session_command_op_remove 5 11 2024-04-14T15:25:49.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.452+0000 7f9736ff8700 10 client.47999.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:25:49.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.452+0000 7f9736ff8700 15 client.47999.objecter _session_command_op_assign -1 12 2024-04-14T15:25:49.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.452+0000 7f9736ff8700 20 client.47999.objecter _get_session s=0x7f97301b5e60 osd=5 3 2024-04-14T15:25:49.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.452+0000 7f9736ff8700 20 client.47999.objecter put_session s=0x7f97301b5e60 osd=5 3 2024-04-14T15:25:49.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.452+0000 7f9736ff8700 20 client.47999.objecter _get_session s=0x7f97301b5e60 osd=5 3 2024-04-14T15:25:49.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.452+0000 7f9736ff8700 15 client.47999.objecter _session_command_op_remove -1 12 2024-04-14T15:25:49.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.452+0000 7f9736ff8700 20 client.47999.objecter get_session s=0x7f97301b5e60 osd=5 3 2024-04-14T15:25:49.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.452+0000 7f9736ff8700 15 client.47999.objecter _session_command_op_assign 5 12 2024-04-14T15:25:49.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.452+0000 7f9736ff8700 20 client.47999.objecter put_session s=0x7f97301b5e60 osd=5 4 2024-04-14T15:25:49.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.452+0000 7f9736ff8700 10 client.47999.objecter _send_command 12 2024-04-14T15:25:49.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.452+0000 7f9736ff8700 1 -- 172.21.15.90:0/3488307638 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f97301b6090 con 0x7f97301bb390 2024-04-14T15:25:49.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.453+0000 7f972dffb700 1 -- 172.21.15.90:0/3488307638 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f97301b6090 con 0x7f97301bb390 2024-04-14T15:25:49.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.453+0000 7f972dffb700 10 client.47999.objecter ms_dispatch 0x7f9730001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:25:49.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.454+0000 7f972dffb700 10 client.47999.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:25:49.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.454+0000 7f972dffb700 20 client.47999.objecter put_session s=0x7f97301b5e60 osd=5 4 2024-04-14T15:25:49.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.454+0000 7f972dffb700 15 client.47999.objecter _session_command_op_remove 5 12 2024-04-14T15:25:49.455 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:25:49.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.454+0000 7f9736ff8700 10 client.47999.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:25:49.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.454+0000 7f9736ff8700 15 client.47999.objecter _session_command_op_assign -1 13 2024-04-14T15:25:49.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.454+0000 7f9736ff8700 20 client.47999.objecter _get_session s=0x7f97301b60d0 osd=6 3 2024-04-14T15:25:49.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.454+0000 7f9736ff8700 20 client.47999.objecter put_session s=0x7f97301b60d0 osd=6 3 2024-04-14T15:25:49.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.454+0000 7f9736ff8700 20 client.47999.objecter _get_session s=0x7f97301b60d0 osd=6 3 2024-04-14T15:25:49.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.454+0000 7f9736ff8700 15 client.47999.objecter _session_command_op_remove -1 13 2024-04-14T15:25:49.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.454+0000 7f9736ff8700 20 client.47999.objecter get_session s=0x7f97301b60d0 osd=6 3 2024-04-14T15:25:49.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.454+0000 7f9736ff8700 15 client.47999.objecter _session_command_op_assign 6 13 2024-04-14T15:25:49.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.454+0000 7f9736ff8700 20 client.47999.objecter put_session s=0x7f97301b60d0 osd=6 4 2024-04-14T15:25:49.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.454+0000 7f9736ff8700 10 client.47999.objecter _send_command 13 2024-04-14T15:25:49.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.454+0000 7f9736ff8700 1 -- 172.21.15.90:0/3488307638 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f97301c4980 con 0x7f97301c5660 2024-04-14T15:25:49.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.454+0000 7f972dffb700 10 client.47999.objecter ms_handle_connect 0x7f97301c5660 2024-04-14T15:25:49.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.455+0000 7f972dffb700 1 -- 172.21.15.90:0/3488307638 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f97301c4980 con 0x7f97301c5660 2024-04-14T15:25:49.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.455+0000 7f972dffb700 10 client.47999.objecter ms_dispatch 0x7f9730001550 command_reply(tid 13: 0 ) 2024-04-14T15:25:49.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.455+0000 7f972dffb700 10 client.47999.objecter _finish_command 13 = system:0 2024-04-14T15:25:49.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.455+0000 7f972dffb700 20 client.47999.objecter put_session s=0x7f97301b60d0 osd=6 4 2024-04-14T15:25:49.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.455+0000 7f972dffb700 15 client.47999.objecter _session_command_op_remove 6 13 2024-04-14T15:25:49.468 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:49.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.471+0000 7f9736ff8700 10 client.47999.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:25:49.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.471+0000 7f9736ff8700 15 client.47999.objecter _session_command_op_assign -1 14 2024-04-14T15:25:49.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.471+0000 7f9736ff8700 20 client.47999.objecter _get_session s=0x7f97301b60d0 osd=6 3 2024-04-14T15:25:49.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.471+0000 7f9736ff8700 20 client.47999.objecter put_session s=0x7f97301b60d0 osd=6 3 2024-04-14T15:25:49.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.471+0000 7f9736ff8700 20 client.47999.objecter _get_session s=0x7f97301b60d0 osd=6 3 2024-04-14T15:25:49.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.471+0000 7f9736ff8700 15 client.47999.objecter _session_command_op_remove -1 14 2024-04-14T15:25:49.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.471+0000 7f9736ff8700 20 client.47999.objecter get_session s=0x7f97301b60d0 osd=6 3 2024-04-14T15:25:49.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.471+0000 7f9736ff8700 15 client.47999.objecter _session_command_op_assign 6 14 2024-04-14T15:25:49.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.471+0000 7f9736ff8700 20 client.47999.objecter put_session s=0x7f97301b60d0 osd=6 4 2024-04-14T15:25:49.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.471+0000 7f9736ff8700 10 client.47999.objecter _send_command 14 2024-04-14T15:25:49.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.471+0000 7f9736ff8700 1 -- 172.21.15.90:0/3488307638 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f97301c4980 con 0x7f97301c5660 2024-04-14T15:25:49.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.473+0000 7f972dffb700 1 -- 172.21.15.90:0/3488307638 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f97301c4980 con 0x7f97301c5660 2024-04-14T15:25:49.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.473+0000 7f972dffb700 10 client.47999.objecter ms_dispatch 0x7f9730001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:25:49.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.473+0000 7f972dffb700 10 client.47999.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:25:49.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.473+0000 7f972dffb700 20 client.47999.objecter put_session s=0x7f97301b60d0 osd=6 4 2024-04-14T15:25:49.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.473+0000 7f972dffb700 15 client.47999.objecter _session_command_op_remove 6 14 2024-04-14T15:25:49.474 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:25:49.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.473+0000 7f9736ff8700 10 client.47999.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:25:49.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.473+0000 7f9736ff8700 15 client.47999.objecter _session_command_op_assign -1 15 2024-04-14T15:25:49.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.473+0000 7f9736ff8700 20 client.47999.objecter _get_session s=0x7f97301c4980 osd=7 3 2024-04-14T15:25:49.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.473+0000 7f9736ff8700 20 client.47999.objecter put_session s=0x7f97301c4980 osd=7 3 2024-04-14T15:25:49.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.473+0000 7f9736ff8700 20 client.47999.objecter _get_session s=0x7f97301c4980 osd=7 3 2024-04-14T15:25:49.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.473+0000 7f9736ff8700 15 client.47999.objecter _session_command_op_remove -1 15 2024-04-14T15:25:49.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.473+0000 7f9736ff8700 20 client.47999.objecter get_session s=0x7f97301c4980 osd=7 3 2024-04-14T15:25:49.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.473+0000 7f9736ff8700 15 client.47999.objecter _session_command_op_assign 7 15 2024-04-14T15:25:49.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.473+0000 7f9736ff8700 20 client.47999.objecter put_session s=0x7f97301c4980 osd=7 4 2024-04-14T15:25:49.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.473+0000 7f9736ff8700 10 client.47999.objecter _send_command 15 2024-04-14T15:25:49.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.473+0000 7f9736ff8700 1 -- 172.21.15.90:0/3488307638 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f97301cc0f0 con 0x7f97301c4cb0 2024-04-14T15:25:49.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.474+0000 7f972dffb700 10 client.47999.objecter ms_handle_connect 0x7f97301c4cb0 2024-04-14T15:25:49.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.474+0000 7f972dffb700 1 -- 172.21.15.90:0/3488307638 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f97301cc0f0 con 0x7f97301c4cb0 2024-04-14T15:25:49.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.474+0000 7f972dffb700 10 client.47999.objecter ms_dispatch 0x7f9730001550 command_reply(tid 15: 0 ) 2024-04-14T15:25:49.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.474+0000 7f972dffb700 10 client.47999.objecter _finish_command 15 = system:0 2024-04-14T15:25:49.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.474+0000 7f972dffb700 20 client.47999.objecter put_session s=0x7f97301c4980 osd=7 4 2024-04-14T15:25:49.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.474+0000 7f972dffb700 15 client.47999.objecter _session_command_op_remove 7 15 2024-04-14T15:25:49.491 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:49.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:49.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:49.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.491+0000 7f9736ff8700 10 client.47999.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:25:49.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.491+0000 7f9736ff8700 15 client.47999.objecter _session_command_op_assign -1 16 2024-04-14T15:25:49.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.491+0000 7f9736ff8700 20 client.47999.objecter _get_session s=0x7f97301c4980 osd=7 3 2024-04-14T15:25:49.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.491+0000 7f9736ff8700 20 client.47999.objecter put_session s=0x7f97301c4980 osd=7 3 2024-04-14T15:25:49.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.491+0000 7f9736ff8700 20 client.47999.objecter _get_session s=0x7f97301c4980 osd=7 3 2024-04-14T15:25:49.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.492+0000 7f9736ff8700 15 client.47999.objecter _session_command_op_remove -1 16 2024-04-14T15:25:49.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.492+0000 7f9736ff8700 20 client.47999.objecter get_session s=0x7f97301c4980 osd=7 3 2024-04-14T15:25:49.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.492+0000 7f9736ff8700 15 client.47999.objecter _session_command_op_assign 7 16 2024-04-14T15:25:49.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.492+0000 7f9736ff8700 20 client.47999.objecter put_session s=0x7f97301c4980 osd=7 4 2024-04-14T15:25:49.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.492+0000 7f9736ff8700 10 client.47999.objecter _send_command 16 2024-04-14T15:25:49.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.492+0000 7f9736ff8700 1 -- 172.21.15.90:0/3488307638 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f97301c0760 con 0x7f97301c4cb0 2024-04-14T15:25:49.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.494+0000 7f972dffb700 1 -- 172.21.15.90:0/3488307638 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f97301c0760 con 0x7f97301c4cb0 2024-04-14T15:25:49.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.494+0000 7f972dffb700 10 client.47999.objecter ms_dispatch 0x7f9730001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:25:49.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.494+0000 7f972dffb700 10 client.47999.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:25:49.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.494+0000 7f972dffb700 20 client.47999.objecter put_session s=0x7f97301c4980 osd=7 4 2024-04-14T15:25:49.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.494+0000 7f972dffb700 15 client.47999.objecter _session_command_op_remove 7 16 2024-04-14T15:25:49.494 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:25:49.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.494+0000 7f9736ff8700 10 librados: watch_flush enter 2024-04-14T15:25:49.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.494+0000 7f9736ff8700 10 librados: watch_flush exit 2024-04-14T15:25:49.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.494+0000 7f9736ff8700 10 client.47999.objecter close_session for osd.0 2024-04-14T15:25:49.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.494+0000 7f9736ff8700 1 -- 172.21.15.90:0/3488307638 >> v1:172.21.15.90:6801/3761806885 conn(0x7f97301aaff0 legacy=0x7f97301ab3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:49.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.494+0000 7f9736ff8700 20 client.47999.objecter put_session s=0x7f9730113310 osd=0 1 2024-04-14T15:25:49.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.494+0000 7f9736ff8700 10 client.47999.objecter close_session for osd.1 2024-04-14T15:25:49.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.494+0000 7f9736ff8700 1 -- 172.21.15.90:0/3488307638 >> v1:172.21.15.90:6805/4170361699 conn(0x7f97300fe550 legacy=0x7f97300fe970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:49.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.494+0000 7f9736ff8700 20 client.47999.objecter put_session s=0x7f97301abd30 osd=1 1 2024-04-14T15:25:49.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.494+0000 7f9736ff8700 10 client.47999.objecter close_session for osd.2 2024-04-14T15:25:49.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.494+0000 7f9736ff8700 1 -- 172.21.15.90:0/3488307638 >> v1:172.21.15.90:6809/1686184251 conn(0x7f97300636c0 legacy=0x7f97301b46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:49.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.494+0000 7f9736ff8700 20 client.47999.objecter put_session s=0x7f97301abea0 osd=2 1 2024-04-14T15:25:49.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.494+0000 7f9736ff8700 10 client.47999.objecter close_session for osd.3 2024-04-14T15:25:49.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.494+0000 7f9736ff8700 1 -- 172.21.15.90:0/3488307638 >> v1:172.21.15.90:6813/2270166774 conn(0x7f97300632b0 legacy=0x7f97301b8f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:49.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.494+0000 7f9736ff8700 20 client.47999.objecter put_session s=0x7f9730063140 osd=3 1 2024-04-14T15:25:49.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.494+0000 7f9736ff8700 10 client.47999.objecter close_session for osd.4 2024-04-14T15:25:49.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.494+0000 7f9736ff8700 1 -- 172.21.15.90:0/3488307638 >> v1:172.21.15.92:6800/3887549733 conn(0x7f97301b6ae0 legacy=0x7f97301bd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:49.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.494+0000 7f9736ff8700 20 client.47999.objecter put_session s=0x7f97301b1610 osd=4 1 2024-04-14T15:25:49.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.494+0000 7f9736ff8700 10 client.47999.objecter close_session for osd.5 2024-04-14T15:25:49.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.494+0000 7f9735595700 1 -- 172.21.15.90:0/3488307638 reap_dead start 2024-04-14T15:25:49.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.494+0000 7f9736ff8700 1 -- 172.21.15.90:0/3488307638 >> v1:172.21.15.92:6804/3244561440 conn(0x7f97301bb390 legacy=0x7f97301bef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:49.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.494+0000 7f9735595700 1 -- 172.21.15.90:0/3488307638 reap_dead start 2024-04-14T15:25:49.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.494+0000 7f9735595700 1 -- 172.21.15.90:0/3488307638 reap_dead start 2024-04-14T15:25:49.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.494+0000 7f9736ff8700 20 client.47999.objecter put_session s=0x7f97301b5e60 osd=5 1 2024-04-14T15:25:49.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.494+0000 7f9736ff8700 10 client.47999.objecter close_session for osd.6 2024-04-14T15:25:49.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.494+0000 7f9736ff8700 1 -- 172.21.15.90:0/3488307638 >> v1:172.21.15.92:6808/636125963 conn(0x7f97301c5660 legacy=0x7f97301c5a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:49.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.494+0000 7f9736ff8700 20 client.47999.objecter put_session s=0x7f97301b60d0 osd=6 1 2024-04-14T15:25:49.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.494+0000 7f9736ff8700 10 client.47999.objecter close_session for osd.7 2024-04-14T15:25:49.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.494+0000 7f9736ff8700 1 -- 172.21.15.90:0/3488307638 >> v1:172.21.15.92:6812/1773706716 conn(0x7f97301c4cb0 legacy=0x7f97301ca860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:49.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.494+0000 7f9736ff8700 20 client.47999.objecter put_session s=0x7f97301c4980 osd=7 1 2024-04-14T15:25:49.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.494+0000 7f9736ff8700 20 client.47999.objecter shutdown clearing up homeless session... 2024-04-14T15:25:49.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.494+0000 7f9736ff8700 10 client.47999.objecter successfully canceled tick 2024-04-14T15:25:49.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.494+0000 7f9736ff8700 1 -- 172.21.15.90:0/3488307638 >> v1:172.21.15.90:6800/420935427 conn(0x7f9720071360 legacy=0x7f9720073820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:49.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.494+0000 7f9736ff8700 1 -- 172.21.15.90:0/3488307638 >> v1:172.21.15.90:6790/0 conn(0x7f97301012c0 legacy=0x7f9730107dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:49.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.494+0000 7f9735595700 1 -- 172.21.15.90:0/3488307638 reap_dead start 2024-04-14T15:25:49.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.494+0000 7f9736ff8700 1 -- 172.21.15.90:0/3488307638 shutdown_connections 2024-04-14T15:25:49.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.494+0000 7f9736ff8700 1 -- 172.21.15.90:0/3488307638 >> 172.21.15.90:0/3488307638 conn(0x7f9730076a70 msgr2=0x7f9730112b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:25:49.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.495+0000 7f9736ff8700 1 -- 172.21.15.90:0/3488307638 shutdown_connections 2024-04-14T15:25:49.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.495+0000 7f9736ff8700 1 -- 172.21.15.90:0/3488307638 wait complete. 2024-04-14T15:25:49.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:49.495+0000 7f9736ff8700 1 librados: shutdown 2024-04-14T15:25:49.499 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:25:49.499 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:25:49.499 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:25:49.499 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:25:49.499 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:25:49.499 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:25:49.499 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:25:49.499 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:25:49.593 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:49.617 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:49.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:49.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:49.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:49 smithi092 ceph-mon[33966]: 5.14 scrub starts 2024-04-14T15:25:49.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:49 smithi092 ceph-mon[33966]: 5.14 scrub ok 2024-04-14T15:25:49.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:49 smithi092 ceph-mon[33966]: 2.9 scrub starts 2024-04-14T15:25:49.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:49 smithi092 ceph-mon[33966]: 2.9 scrub ok 2024-04-14T15:25:49.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:49 smithi092 ceph-mon[33966]: 2.13 scrub starts 2024-04-14T15:25:49.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:49 smithi092 ceph-mon[33966]: 2.13 scrub ok 2024-04-14T15:25:49.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:49 smithi092 ceph-mon[33966]: 6.1e scrub starts 2024-04-14T15:25:49.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:49 smithi092 ceph-mon[33966]: 6.1e scrub ok 2024-04-14T15:25:49.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:49 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-142", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:25:49.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:49 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-142"}]: dispatch 2024-04-14T15:25:49.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:49 smithi092 ceph-mon[33966]: osdmap e799: 8 total, 8 up, 7 in 2024-04-14T15:25:49.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:49 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-142"}]: dispatch 2024-04-14T15:25:49.677 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-14T15:25:49.718 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:25:49.742 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:49.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:49.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:25:49.844 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:25:49.867 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:49.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:49.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:25:49.914 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:25:49.968 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:25:49.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:49.993 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:49.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:25:50.094 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:25:50.118 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:50.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:50.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:25:50.220 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:25:50.243 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:50.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:50.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:25:50.344 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:25:50.368 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:50.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:50.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:25:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:50 smithi092 ceph-mon[33966]: 2.3 scrub starts 2024-04-14T15:25:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:50 smithi092 ceph-mon[33966]: 5.0 scrub starts 2024-04-14T15:25:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:50 smithi092 ceph-mon[33966]: 2.3 scrub ok 2024-04-14T15:25:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:50 smithi092 ceph-mon[33966]: 5.0 scrub ok 2024-04-14T15:25:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:50 smithi092 ceph-mon[33966]: 1.10 scrub starts 2024-04-14T15:25:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:50 smithi092 ceph-mon[33966]: 1.10 scrub ok 2024-04-14T15:25:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:50 smithi092 ceph-mon[33966]: 1.13 scrub starts 2024-04-14T15:25:50.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:50 smithi092 ceph-mon[33966]: 1.13 scrub ok 2024-04-14T15:25:50.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:50 smithi092 ceph-mon[33966]: 6.a scrub starts 2024-04-14T15:25:50.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:50 smithi092 ceph-mon[33966]: 6.a scrub ok 2024-04-14T15:25:50.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:50 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:25:50.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:50 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:25:50.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:50 smithi092 ceph-mon[33966]: pgmap v1186: 224 pgs: 1 active+remapped+backfill_toofull, 223 active+clean; 4.2 MiB data, 755 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-14T15:25:50.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:50 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3488307638' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:25:50.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:50 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-142"}]': finished 2024-04-14T15:25:50.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:50 smithi092 ceph-mon[33966]: osdmap e800: 8 total, 8 up, 7 in 2024-04-14T15:25:50.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:50 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-142", "mode": "writeback"}]: dispatch 2024-04-14T15:25:50.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:50 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-142", "mode": "writeback"}]: dispatch 2024-04-14T15:25:50.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:50 smithi090 ceph-mon[30399]: 2.3 scrub starts 2024-04-14T15:25:50.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:50 smithi090 ceph-mon[30399]: 5.0 scrub starts 2024-04-14T15:25:50.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:50 smithi090 ceph-mon[30399]: 2.3 scrub ok 2024-04-14T15:25:50.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:50 smithi090 ceph-mon[30399]: 5.0 scrub ok 2024-04-14T15:25:50.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:50 smithi090 ceph-mon[30399]: 1.10 scrub starts 2024-04-14T15:25:50.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:50 smithi090 ceph-mon[30399]: 1.10 scrub ok 2024-04-14T15:25:50.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:50 smithi090 ceph-mon[30399]: 1.13 scrub starts 2024-04-14T15:25:50.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:50 smithi090 ceph-mon[30399]: 1.13 scrub ok 2024-04-14T15:25:50.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:50 smithi090 ceph-mon[30399]: 6.a scrub starts 2024-04-14T15:25:50.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:50 smithi090 ceph-mon[30399]: 6.a scrub ok 2024-04-14T15:25:50.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:50 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:25:50.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:50 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:25:50.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:50 smithi090 ceph-mon[30399]: pgmap v1186: 224 pgs: 1 active+remapped+backfill_toofull, 223 active+clean; 4.2 MiB data, 755 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-14T15:25:50.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:50 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3488307638' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:25:50.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:50 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-142"}]': finished 2024-04-14T15:25:50.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:50 smithi090 ceph-mon[30399]: osdmap e800: 8 total, 8 up, 7 in 2024-04-14T15:25:50.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:50 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-142", "mode": "writeback"}]: dispatch 2024-04-14T15:25:50.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:50 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-142", "mode": "writeback"}]: dispatch 2024-04-14T15:25:50.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:50 smithi090 ceph-mon[41670]: 2.3 scrub starts 2024-04-14T15:25:50.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:50 smithi090 ceph-mon[41670]: 5.0 scrub starts 2024-04-14T15:25:50.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:50 smithi090 ceph-mon[41670]: 2.3 scrub ok 2024-04-14T15:25:50.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:50 smithi090 ceph-mon[41670]: 5.0 scrub ok 2024-04-14T15:25:50.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:50 smithi090 ceph-mon[41670]: 1.10 scrub starts 2024-04-14T15:25:50.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:50 smithi090 ceph-mon[41670]: 1.10 scrub ok 2024-04-14T15:25:50.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:50 smithi090 ceph-mon[41670]: 1.13 scrub starts 2024-04-14T15:25:50.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:50 smithi090 ceph-mon[41670]: 1.13 scrub ok 2024-04-14T15:25:50.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:50 smithi090 ceph-mon[41670]: 6.a scrub starts 2024-04-14T15:25:50.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:50 smithi090 ceph-mon[41670]: 6.a scrub ok 2024-04-14T15:25:50.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:50 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:25:50.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:50 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:25:50.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:50 smithi090 ceph-mon[41670]: pgmap v1186: 224 pgs: 1 active+remapped+backfill_toofull, 223 active+clean; 4.2 MiB data, 755 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-14T15:25:50.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:50 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3488307638' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:25:50.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:50 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-142"}]': finished 2024-04-14T15:25:50.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:50 smithi090 ceph-mon[41670]: osdmap e800: 8 total, 8 up, 7 in 2024-04-14T15:25:50.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:50 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-142", "mode": "writeback"}]: dispatch 2024-04-14T15:25:50.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:50 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-142", "mode": "writeback"}]: dispatch 2024-04-14T15:25:50.470 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:25:50.494 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:50.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:50.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:25:50.595 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:50.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:50.619 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:50.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:50.720 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:50.744 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:50.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:50.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:50.846 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:25:50.869 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:50.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:50.870 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:25:50.971 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:50.994 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:50.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:50.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:51.096 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:25:51.120 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:51.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:51.121 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:25:51.222 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:25:51.245 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:51.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:51.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:25:51.347 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:51.371 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:51.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:51.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:51.407 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:51 smithi090 ceph-mon[30399]: 5.d scrub starts 2024-04-14T15:25:51.407 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:51 smithi090 ceph-mon[30399]: 5.d scrub ok 2024-04-14T15:25:51.407 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:51 smithi090 ceph-mon[30399]: 323.6s0 scrub starts 2024-04-14T15:25:51.408 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:51 smithi090 ceph-mon[30399]: 323.6 scrub ok 2024-04-14T15:25:51.408 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:51 smithi090 ceph-mon[30399]: 4.b scrub starts 2024-04-14T15:25:51.408 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:51 smithi090 ceph-mon[30399]: 4.b scrub ok 2024-04-14T15:25:51.408 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:51 smithi090 ceph-mon[30399]: 11.1b scrub starts 2024-04-14T15:25:51.408 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:51 smithi090 ceph-mon[30399]: 11.1b scrub ok 2024-04-14T15:25:51.408 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:51 smithi090 ceph-mon[30399]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:25:51.408 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:51 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-142", "mode": "writeback"}]': finished 2024-04-14T15:25:51.408 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:51 smithi090 ceph-mon[30399]: osdmap e801: 8 total, 8 up, 7 in 2024-04-14T15:25:51.408 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:51 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-142","var": "hit_set_count","val": "2"}]: dispatch 2024-04-14T15:25:51.408 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:51 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-142","var": "hit_set_count","val": "2"}]: dispatch 2024-04-14T15:25:51.408 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:51 smithi090 ceph-mon[41670]: 5.d scrub starts 2024-04-14T15:25:51.408 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:51 smithi090 ceph-mon[41670]: 5.d scrub ok 2024-04-14T15:25:51.408 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:51 smithi090 ceph-mon[41670]: 323.6s0 scrub starts 2024-04-14T15:25:51.408 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:51 smithi090 ceph-mon[41670]: 323.6 scrub ok 2024-04-14T15:25:51.409 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:51 smithi090 ceph-mon[41670]: 4.b scrub starts 2024-04-14T15:25:51.409 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:51 smithi090 ceph-mon[41670]: 4.b scrub ok 2024-04-14T15:25:51.409 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:51 smithi090 ceph-mon[41670]: 11.1b scrub starts 2024-04-14T15:25:51.409 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:51 smithi090 ceph-mon[41670]: 11.1b scrub ok 2024-04-14T15:25:51.409 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:51 smithi090 ceph-mon[41670]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:25:51.409 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:51 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-142", "mode": "writeback"}]': finished 2024-04-14T15:25:51.409 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:51 smithi090 ceph-mon[41670]: osdmap e801: 8 total, 8 up, 7 in 2024-04-14T15:25:51.409 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:51 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-142","var": "hit_set_count","val": "2"}]: dispatch 2024-04-14T15:25:51.409 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:51 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-142","var": "hit_set_count","val": "2"}]: dispatch 2024-04-14T15:25:51.452 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:51 smithi092 ceph-mon[33966]: 5.d scrub starts 2024-04-14T15:25:51.452 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:51 smithi092 ceph-mon[33966]: 5.d scrub ok 2024-04-14T15:25:51.452 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:51 smithi092 ceph-mon[33966]: 323.6s0 scrub starts 2024-04-14T15:25:51.452 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:51 smithi092 ceph-mon[33966]: 323.6 scrub ok 2024-04-14T15:25:51.452 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:51 smithi092 ceph-mon[33966]: 4.b scrub starts 2024-04-14T15:25:51.452 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:51 smithi092 ceph-mon[33966]: 4.b scrub ok 2024-04-14T15:25:51.452 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:51 smithi092 ceph-mon[33966]: 11.1b scrub starts 2024-04-14T15:25:51.452 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:51 smithi092 ceph-mon[33966]: 11.1b scrub ok 2024-04-14T15:25:51.452 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:51 smithi092 ceph-mon[33966]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:25:51.452 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:51 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-142", "mode": "writeback"}]': finished 2024-04-14T15:25:51.452 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:51 smithi092 ceph-mon[33966]: osdmap e801: 8 total, 8 up, 7 in 2024-04-14T15:25:51.453 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:51 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-142","var": "hit_set_count","val": "2"}]: dispatch 2024-04-14T15:25:51.453 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:51 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-142","var": "hit_set_count","val": "2"}]: dispatch 2024-04-14T15:25:51.473 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:25:51.497 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:51.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:51.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:25:51.579 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-04-14T15:25:51.579 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-04-14T15:25:51.579 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 0 2024-04-14T15:25:51.579 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-04-14T15:25:51.599 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:25:51.624 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:25:51.624 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:51.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:51.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:25:51.726 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:25:51.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:51.751 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:51.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:25:51.852 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:25:51.876 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:51.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:51.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:25:51.978 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:25:52.001 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:52.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:52.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:25:52.103 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:25:52.127 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:52.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:52.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:25:52.181 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-14T15:25:52.228 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:25:52.251 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:52.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:52.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:25:52.353 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:52.374 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:52 smithi090 ceph-mon[41670]: 2.16 scrub starts 2024-04-14T15:25:52.374 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:52 smithi090 ceph-mon[41670]: 2.16 scrub ok 2024-04-14T15:25:52.374 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:52 smithi090 ceph-mon[41670]: 1.c scrub starts 2024-04-14T15:25:52.374 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:52 smithi090 ceph-mon[41670]: 1.c scrub ok 2024-04-14T15:25:52.374 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:52 smithi090 ceph-mon[41670]: 5.c scrub starts 2024-04-14T15:25:52.374 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:52 smithi090 ceph-mon[41670]: 5.c scrub ok 2024-04-14T15:25:52.374 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:52 smithi090 ceph-mon[41670]: pgmap v1189: 224 pgs: 1 active+remapped+backfill_toofull, 223 active+clean; 4.2 MiB data, 755 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-04-14T15:25:52.375 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:52 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-142","var": "hit_set_count","val": "2"}]': finished 2024-04-14T15:25:52.375 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:52 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-142","var": "hit_set_period","val": "600"}]: dispatch 2024-04-14T15:25:52.375 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:52 smithi090 ceph-mon[41670]: osdmap e802: 8 total, 8 up, 7 in 2024-04-14T15:25:52.375 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:52 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-142","var": "hit_set_period","val": "600"}]: dispatch 2024-04-14T15:25:52.375 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:52 smithi090 ceph-mon[30399]: 2.16 scrub starts 2024-04-14T15:25:52.375 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:52 smithi090 ceph-mon[30399]: 2.16 scrub ok 2024-04-14T15:25:52.375 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:52 smithi090 ceph-mon[30399]: 1.c scrub starts 2024-04-14T15:25:52.375 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:52 smithi090 ceph-mon[30399]: 1.c scrub ok 2024-04-14T15:25:52.375 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:52 smithi090 ceph-mon[30399]: 5.c scrub starts 2024-04-14T15:25:52.375 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:52 smithi090 ceph-mon[30399]: 5.c scrub ok 2024-04-14T15:25:52.375 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:52 smithi090 ceph-mon[30399]: pgmap v1189: 224 pgs: 1 active+remapped+backfill_toofull, 223 active+clean; 4.2 MiB data, 755 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-04-14T15:25:52.375 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:52 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-142","var": "hit_set_count","val": "2"}]': finished 2024-04-14T15:25:52.375 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:52 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-142","var": "hit_set_period","val": "600"}]: dispatch 2024-04-14T15:25:52.376 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:52 smithi090 ceph-mon[30399]: osdmap e802: 8 total, 8 up, 7 in 2024-04-14T15:25:52.376 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:52 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-142","var": "hit_set_period","val": "600"}]: dispatch 2024-04-14T15:25:52.377 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:52.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:52.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:52.423 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:25:52.480 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:25:52.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:52.503 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:52.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:25:52.544 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:52 smithi092 ceph-mon[33966]: 2.16 scrub starts 2024-04-14T15:25:52.544 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:52 smithi092 ceph-mon[33966]: 2.16 scrub ok 2024-04-14T15:25:52.544 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:52 smithi092 ceph-mon[33966]: 1.c scrub starts 2024-04-14T15:25:52.545 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:52 smithi092 ceph-mon[33966]: 1.c scrub ok 2024-04-14T15:25:52.545 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:52 smithi092 ceph-mon[33966]: 5.c scrub starts 2024-04-14T15:25:52.545 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:52 smithi092 ceph-mon[33966]: 5.c scrub ok 2024-04-14T15:25:52.545 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:52 smithi092 ceph-mon[33966]: pgmap v1189: 224 pgs: 1 active+remapped+backfill_toofull, 223 active+clean; 4.2 MiB data, 755 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-04-14T15:25:52.545 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:52 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-142","var": "hit_set_count","val": "2"}]': finished 2024-04-14T15:25:52.545 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:52 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-142","var": "hit_set_period","val": "600"}]: dispatch 2024-04-14T15:25:52.545 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:52 smithi092 ceph-mon[33966]: osdmap e802: 8 total, 8 up, 7 in 2024-04-14T15:25:52.545 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:52 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-142","var": "hit_set_period","val": "600"}]: dispatch 2024-04-14T15:25:52.587 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-14T15:25:52.604 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:25:52.628 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:52.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:52.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:25:52.729 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:25:52.753 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:52.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:52.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:25:52.854 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:52.878 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:52.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:52.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:52.979 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:25:53.003 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:53.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:53.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:25:53.105 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:25:53.128 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:53.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:53.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:25:53.230 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:25:53.254 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:53.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:53.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:25:53.355 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:25:53.379 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:53.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:53.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:25:53.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:53 smithi090 ceph-mon[30399]: 2.e scrub starts 2024-04-14T15:25:53.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:53 smithi090 ceph-mon[30399]: 2.e scrub ok 2024-04-14T15:25:53.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:53 smithi090 ceph-mon[30399]: 1.9 scrub starts 2024-04-14T15:25:53.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:53 smithi090 ceph-mon[30399]: 1.9 scrub ok 2024-04-14T15:25:53.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:53 smithi090 ceph-mon[30399]: 5.1 scrub starts 2024-04-14T15:25:53.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:53 smithi090 ceph-mon[30399]: 5.1 scrub ok 2024-04-14T15:25:53.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:53 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-142","var": "hit_set_period","val": "600"}]': finished 2024-04-14T15:25:53.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:53 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-14T15:25:53.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:53 smithi090 ceph-mon[30399]: osdmap e803: 8 total, 8 up, 7 in 2024-04-14T15:25:53.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:53 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-14T15:25:53.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:53 smithi090 ceph-mon[41670]: 2.e scrub starts 2024-04-14T15:25:53.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:53 smithi090 ceph-mon[41670]: 2.e scrub ok 2024-04-14T15:25:53.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:53 smithi090 ceph-mon[41670]: 1.9 scrub starts 2024-04-14T15:25:53.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:53 smithi090 ceph-mon[41670]: 1.9 scrub ok 2024-04-14T15:25:53.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:53 smithi090 ceph-mon[41670]: 5.1 scrub starts 2024-04-14T15:25:53.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:53 smithi090 ceph-mon[41670]: 5.1 scrub ok 2024-04-14T15:25:53.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:53 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-142","var": "hit_set_period","val": "600"}]': finished 2024-04-14T15:25:53.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:53 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-14T15:25:53.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:53 smithi090 ceph-mon[41670]: osdmap e803: 8 total, 8 up, 7 in 2024-04-14T15:25:53.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:53 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-14T15:25:53.443 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:53 smithi092 ceph-mon[33966]: 2.e scrub starts 2024-04-14T15:25:53.443 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:53 smithi092 ceph-mon[33966]: 2.e scrub ok 2024-04-14T15:25:53.443 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:53 smithi092 ceph-mon[33966]: 1.9 scrub starts 2024-04-14T15:25:53.443 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:53 smithi092 ceph-mon[33966]: 1.9 scrub ok 2024-04-14T15:25:53.443 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:53 smithi092 ceph-mon[33966]: 5.1 scrub starts 2024-04-14T15:25:53.443 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:53 smithi092 ceph-mon[33966]: 5.1 scrub ok 2024-04-14T15:25:53.444 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:53 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-142","var": "hit_set_period","val": "600"}]': finished 2024-04-14T15:25:53.444 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:53 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-14T15:25:53.444 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:53 smithi092 ceph-mon[33966]: osdmap e803: 8 total, 8 up, 7 in 2024-04-14T15:25:53.444 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:53 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-14T15:25:53.481 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:25:53.505 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:53.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:53.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:25:53.607 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:25:53.630 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:53.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:53.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:25:53.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:53.715+0000 7f48277da700 1 -- 172.21.15.90:0/2793974792 >> v1:172.21.15.90:6789/0 conn(0x7f48201077a0 legacy=0x7f4820107b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:53.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:53.715+0000 7f48277da700 1 -- 172.21.15.90:0/2793974792 shutdown_connections 2024-04-14T15:25:53.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:53.715+0000 7f48277da700 1 -- 172.21.15.90:0/2793974792 >> 172.21.15.90:0/2793974792 conn(0x7f48200fd2b0 msgr2=0x7f48200ff6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:25:53.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:53.715+0000 7f48277da700 1 -- 172.21.15.90:0/2793974792 shutdown_connections 2024-04-14T15:25:53.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:53.715+0000 7f48277da700 1 -- 172.21.15.90:0/2793974792 wait complete. 2024-04-14T15:25:53.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:53.716+0000 7f48277da700 1 librados: starting msgr at 2024-04-14T15:25:53.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:53.716+0000 7f48277da700 1 librados: starting objecter 2024-04-14T15:25:53.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:53.716+0000 7f48277da700 1 Processor -- start 2024-04-14T15:25:53.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:53.716+0000 7f48277da700 1 -- start start 2024-04-14T15:25:53.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:53.716+0000 7f48277da700 1 librados: setting wanted keys 2024-04-14T15:25:53.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:53.716+0000 7f48277da700 1 librados: calling monclient init 2024-04-14T15:25:53.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:53.716+0000 7f48277da700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f482010df80 con 0x7f48201077a0 2024-04-14T15:25:53.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:53.716+0000 7f48277da700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f48201a8e40 con 0x7f482010a650 2024-04-14T15:25:53.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:53.716+0000 7f48277da700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f48201aa020 con 0x7f482010e220 2024-04-14T15:25:53.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:53.716+0000 7f4825576700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f48201077a0 0x7f482006aeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:56216/0 (socket says 172.21.15.90:56216) 2024-04-14T15:25:53.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:53.716+0000 7f4825576700 1 -- 172.21.15.90:0/3440918204 learned_addr learned my addr 172.21.15.90:0/3440918204 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:25:53.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:53.716+0000 7f48167fc700 10 client.?.objecter ms_handle_connect 0x7f48201077a0 2024-04-14T15:25:53.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:53.716+0000 7f48167fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:25:53.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:53.716+0000 7f48167fc700 10 client.?.objecter ms_handle_connect 0x7f482010a650 2024-04-14T15:25:53.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:53.716+0000 7f48167fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:25:53.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:53.716+0000 7f48167fc700 10 client.?.objecter ms_handle_connect 0x7f482010e220 2024-04-14T15:25:53.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:53.716+0000 7f48167fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:25:53.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:53.717+0000 7f48167fc700 1 -- 172.21.15.90:0/3440918204 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2737384424 0 0) 0x7f48201a8e40 con 0x7f482010a650 2024-04-14T15:25:53.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:53.717+0000 7f48167fc700 1 -- 172.21.15.90:0/3440918204 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f480c0035b0 con 0x7f482010a650 2024-04-14T15:25:53.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:53.717+0000 7f48167fc700 1 -- 172.21.15.90:0/3440918204 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3199962576 0 0) 0x7f482010df80 con 0x7f48201077a0 2024-04-14T15:25:53.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:53.717+0000 7f48167fc700 1 -- 172.21.15.90:0/3440918204 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f48201a8e40 con 0x7f48201077a0 2024-04-14T15:25:53.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:53.717+0000 7f48167fc700 1 -- 172.21.15.90:0/3440918204 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1788447929 0 0) 0x7f48201aa020 con 0x7f482010e220 2024-04-14T15:25:53.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:53.717+0000 7f48167fc700 1 -- 172.21.15.90:0/3440918204 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f482010df80 con 0x7f482010e220 2024-04-14T15:25:53.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:53.717+0000 7f48167fc700 1 -- 172.21.15.90:0/3440918204 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1146597428 0 0) 0x7f48201a8e40 con 0x7f48201077a0 2024-04-14T15:25:53.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:53.717+0000 7f48167fc700 1 -- 172.21.15.90:0/3440918204 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f48201aa020 con 0x7f48201077a0 2024-04-14T15:25:53.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:53.717+0000 7f48167fc700 1 -- 172.21.15.90:0/3440918204 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f481c0017b0 con 0x7f48201077a0 2024-04-14T15:25:53.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:53.717+0000 7f48167fc700 1 -- 172.21.15.90:0/3440918204 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 946548389 0 0) 0x7f480c0035b0 con 0x7f482010a650 2024-04-14T15:25:53.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:53.717+0000 7f48167fc700 1 -- 172.21.15.90:0/3440918204 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f48201a8e40 con 0x7f482010a650 2024-04-14T15:25:53.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:53.717+0000 7f48167fc700 1 -- 172.21.15.90:0/3440918204 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 318790189 0 0) 0x7f482010df80 con 0x7f482010e220 2024-04-14T15:25:53.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:53.717+0000 7f48167fc700 1 -- 172.21.15.90:0/3440918204 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f480c0035b0 con 0x7f482010e220 2024-04-14T15:25:53.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:53.717+0000 7f48167fc700 1 -- 172.21.15.90:0/3440918204 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f48100018d0 con 0x7f482010a650 2024-04-14T15:25:53.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:53.717+0000 7f48167fc700 1 -- 172.21.15.90:0/3440918204 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f4818001770 con 0x7f482010e220 2024-04-14T15:25:53.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:53.717+0000 7f48167fc700 1 -- 172.21.15.90:0/3440918204 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2312311358 0 0) 0x7f48201aa020 con 0x7f48201077a0 2024-04-14T15:25:53.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:53.717+0000 7f48167fc700 1 -- 172.21.15.90:0/3440918204 >> v1:172.21.15.90:6790/0 conn(0x7f482010e220 legacy=0x7f48201a6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:53.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:53.717+0000 7f48167fc700 1 -- 172.21.15.90:0/3440918204 >> v1:172.21.15.92:6789/0 conn(0x7f482010a650 legacy=0x7f482006b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:53.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:53.717+0000 7f48167fc700 1 -- 172.21.15.90:0/3440918204 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48201ab240 con 0x7f48201077a0 2024-04-14T15:25:53.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:53.717+0000 7f48277da700 1 -- 172.21.15.90:0/3440918204 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f48201a7e90 con 0x7f48201077a0 2024-04-14T15:25:53.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:53.717+0000 7f48277da700 10 client.38055.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:25:53.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:53.717+0000 7f48277da700 1 -- 172.21.15.90:0/3440918204 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f48201a8350 con 0x7f48201077a0 2024-04-14T15:25:53.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:53.717+0000 7f48277da700 1 librados: init done 2024-04-14T15:25:53.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:53.717+0000 7f48167fc700 1 -- 172.21.15.90:0/3440918204 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f481c001520 con 0x7f48201077a0 2024-04-14T15:25:53.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:53.717+0000 7f48167fc700 1 -- 172.21.15.90:0/3440918204 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f481c0050b0 con 0x7f48201077a0 2024-04-14T15:25:53.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:53.719+0000 7f48277da700 1 -- 172.21.15.90:0/3440918204 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4804005150 con 0x7f48201077a0 2024-04-14T15:25:53.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:53.719+0000 7f48167fc700 1 -- 172.21.15.90:0/3440918204 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f481c001eb0 con 0x7f48201077a0 2024-04-14T15:25:53.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:53.719+0000 7f48167fc700 1 -- 172.21.15.90:0/3440918204 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(803..803 src has 467..803) ==== 8790+0+0 (unknown 1390255813 0 0) 0x7f481c08af90 con 0x7f48201077a0 2024-04-14T15:25:53.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:53.719+0000 7f48167fc700 10 client.38055.objecter ms_dispatch 0x7f4820001550 osd_map(803..803 src has 467..803) 2024-04-14T15:25:53.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:53.719+0000 7f48167fc700 3 client.38055.objecter handle_osd_map got epochs [803,803] > 0 2024-04-14T15:25:53.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:53.719+0000 7f48167fc700 3 client.38055.objecter handle_osd_map decoding full epoch 803 2024-04-14T15:25:53.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:53.720+0000 7f48167fc700 20 client.38055.objecter dump_active .. 0 homeless 2024-04-14T15:25:53.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:53.720+0000 7f48167fc700 10 client.38055.objecter ms_handle_connect 0x7f480c0713f0 2024-04-14T15:25:53.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:53.724+0000 7f48167fc700 1 -- 172.21.15.90:0/3440918204 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f481c057750 con 0x7f48201077a0 2024-04-14T15:25:53.732 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:25:53.757 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:53.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:53.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:25:53.859 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:53.882 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:53.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:53.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:53.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:53.950+0000 7f48277da700 1 -- 172.21.15.90:0/3440918204 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd reweight", "id": 7, "weight": 1.0} v 0) -- 0x7f4804005440 con 0x7f48201077a0 2024-04-14T15:25:53.983 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:25:54.009 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:54.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:54.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:25:54.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.029+0000 7f48167fc700 1 -- 172.21.15.90:0/3440918204 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 7, "weight": 1.0}]=0 reweighted osd.7 to 1 (10000) v804) ==== 113+0+0 (unknown 467460217 0 0) 0x7f481c05b400 con 0x7f48201077a0 2024-04-14T15:25:54.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.031+0000 7f48277da700 10 librados: watch_flush enter 2024-04-14T15:25:54.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.031+0000 7f48277da700 10 librados: watch_flush exit 2024-04-14T15:25:54.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.031+0000 7f48277da700 20 client.38055.objecter shutdown clearing up homeless session... 2024-04-14T15:25:54.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.031+0000 7f48277da700 10 client.38055.objecter successfully canceled tick 2024-04-14T15:25:54.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.031+0000 7f48277da700 1 -- 172.21.15.90:0/3440918204 >> v1:172.21.15.90:6800/420935427 conn(0x7f480c0713f0 legacy=0x7f480c0738b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:54.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.031+0000 7f48277da700 1 -- 172.21.15.90:0/3440918204 >> v1:172.21.15.90:6789/0 conn(0x7f48201077a0 legacy=0x7f482006aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:54.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.031+0000 7f48277da700 1 -- 172.21.15.90:0/3440918204 shutdown_connections 2024-04-14T15:25:54.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.031+0000 7f48277da700 1 -- 172.21.15.90:0/3440918204 >> 172.21.15.90:0/3440918204 conn(0x7f48200fd2b0 msgr2=0x7f4820111740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:25:54.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.031+0000 7f48277da700 1 -- 172.21.15.90:0/3440918204 shutdown_connections 2024-04-14T15:25:54.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.031+0000 7f48277da700 1 -- 172.21.15.90:0/3440918204 wait complete. 2024-04-14T15:25:54.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.031+0000 7f48277da700 1 librados: shutdown 2024-04-14T15:25:54.032 INFO:teuthology.orchestra.run.smithi090.stderr:reweighted osd.7 to 1 (10000) 2024-04-14T15:25:54.111 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:25:54.135 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:54.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:54.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:25:54.153 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:25:53 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:25:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:25:54.237 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:25:54.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:54.261 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:54.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:25:54.266 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-04-14T15:25:54.266 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-04-14T15:25:54.266 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-04-14T15:25:54.266 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-04-14T15:25:54.266 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-04-14T15:25:54.275 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:54 smithi092 ceph-mon[33966]: 11.18 scrub starts 2024-04-14T15:25:54.275 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:54 smithi092 ceph-mon[33966]: 11.18 scrub ok 2024-04-14T15:25:54.275 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:54 smithi092 ceph-mon[33966]: 5.1a scrub starts 2024-04-14T15:25:54.275 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:54 smithi092 ceph-mon[33966]: 5.1a scrub ok 2024-04-14T15:25:54.275 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:54 smithi092 ceph-mon[33966]: pgmap v1192: 224 pgs: 1 active+remapped+backfill_toofull, 223 active+clean; 4.2 MiB data, 755 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:25:54.276 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:54 smithi092 ceph-mon[33966]: 4.1b scrub starts 2024-04-14T15:25:54.276 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:54 smithi092 ceph-mon[33966]: 4.1b scrub ok 2024-04-14T15:25:54.276 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:54 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3440918204' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-14T15:25:54.276 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:54 smithi092 ceph-mon[33966]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:25:54.276 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:54 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-142","var": "hit_set_type","val": "bloom"}]': finished 2024-04-14T15:25:54.276 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:54 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3440918204' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-14T15:25:54.276 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:54 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-14T15:25:54.276 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:54 smithi092 ceph-mon[33966]: osdmap e804: 8 total, 8 up, 7 in 2024-04-14T15:25:54.276 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:54 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-14T15:25:54.363 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:25:54.386 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:54.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:54.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:25:54.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:54 smithi090 ceph-mon[30399]: 11.18 scrub starts 2024-04-14T15:25:54.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:54 smithi090 ceph-mon[30399]: 11.18 scrub ok 2024-04-14T15:25:54.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:54 smithi090 ceph-mon[30399]: 5.1a scrub starts 2024-04-14T15:25:54.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:54 smithi090 ceph-mon[30399]: 5.1a scrub ok 2024-04-14T15:25:54.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:54 smithi090 ceph-mon[30399]: pgmap v1192: 224 pgs: 1 active+remapped+backfill_toofull, 223 active+clean; 4.2 MiB data, 755 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:25:54.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:54 smithi090 ceph-mon[30399]: 4.1b scrub starts 2024-04-14T15:25:54.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:54 smithi090 ceph-mon[30399]: 4.1b scrub ok 2024-04-14T15:25:54.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:54 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3440918204' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-14T15:25:54.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:54 smithi090 ceph-mon[30399]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:25:54.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:54 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-142","var": "hit_set_type","val": "bloom"}]': finished 2024-04-14T15:25:54.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:54 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3440918204' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-14T15:25:54.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:54 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-14T15:25:54.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:54 smithi090 ceph-mon[30399]: osdmap e804: 8 total, 8 up, 7 in 2024-04-14T15:25:54.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:54 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-14T15:25:54.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:54 smithi090 ceph-mon[41670]: 11.18 scrub starts 2024-04-14T15:25:54.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:54 smithi090 ceph-mon[41670]: 11.18 scrub ok 2024-04-14T15:25:54.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:54 smithi090 ceph-mon[41670]: 5.1a scrub starts 2024-04-14T15:25:54.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:54 smithi090 ceph-mon[41670]: 5.1a scrub ok 2024-04-14T15:25:54.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:54 smithi090 ceph-mon[41670]: pgmap v1192: 224 pgs: 1 active+remapped+backfill_toofull, 223 active+clean; 4.2 MiB data, 755 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:25:54.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:54 smithi090 ceph-mon[41670]: 4.1b scrub starts 2024-04-14T15:25:54.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:54 smithi090 ceph-mon[41670]: 4.1b scrub ok 2024-04-14T15:25:54.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:54 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3440918204' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-14T15:25:54.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:54 smithi090 ceph-mon[41670]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:25:54.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:54 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-142","var": "hit_set_type","val": "bloom"}]': finished 2024-04-14T15:25:54.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:54 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3440918204' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-14T15:25:54.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:54 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-14T15:25:54.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:54 smithi090 ceph-mon[41670]: osdmap e804: 8 total, 8 up, 7 in 2024-04-14T15:25:54.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:54 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-14T15:25:54.488 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:25:54.511 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:54.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:54.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:25:54.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.594+0000 7f1c1bd6b700 1 -- 172.21.15.90:0/1530937026 >> v1:172.21.15.90:6789/0 conn(0x7f1c141012c0 legacy=0x7f1c141016a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:54.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.595+0000 7f1c1bd6b700 1 -- 172.21.15.90:0/1530937026 shutdown_connections 2024-04-14T15:25:54.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.595+0000 7f1c1bd6b700 1 -- 172.21.15.90:0/1530937026 >> 172.21.15.90:0/1530937026 conn(0x7f1c14076a70 msgr2=0x7f1c14076e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:25:54.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.595+0000 7f1c1bd6b700 1 -- 172.21.15.90:0/1530937026 shutdown_connections 2024-04-14T15:25:54.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.595+0000 7f1c1bd6b700 1 -- 172.21.15.90:0/1530937026 wait complete. 2024-04-14T15:25:54.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.595+0000 7f1c1bd6b700 1 librados: starting msgr at 2024-04-14T15:25:54.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.595+0000 7f1c1bd6b700 1 librados: starting objecter 2024-04-14T15:25:54.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.595+0000 7f1c1bd6b700 1 Processor -- start 2024-04-14T15:25:54.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.595+0000 7f1c1bd6b700 1 -- start start 2024-04-14T15:25:54.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.595+0000 7f1c1bd6b700 1 librados: setting wanted keys 2024-04-14T15:25:54.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.595+0000 7f1c1bd6b700 1 librados: calling monclient init 2024-04-14T15:25:54.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.596+0000 7f1c1bd6b700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1c14108200 con 0x7f1c141012c0 2024-04-14T15:25:54.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.596+0000 7f1c1bd6b700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1c141a8f50 con 0x7f1c1410f360 2024-04-14T15:25:54.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.596+0000 7f1c1bd6b700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1c141aa130 con 0x7f1c1410b7c0 2024-04-14T15:25:54.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.596+0000 7f1c19306700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f1c1410b7c0 0x7f1c14105760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:41298/0 (socket says 172.21.15.90:41298) 2024-04-14T15:25:54.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.596+0000 7f1c19306700 1 -- 172.21.15.90:0/504621603 learned_addr learned my addr 172.21.15.90:0/504621603 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:25:54.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.596+0000 7f1c0affd700 10 client.?.objecter ms_handle_connect 0x7f1c1410b7c0 2024-04-14T15:25:54.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.596+0000 7f1c0affd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:25:54.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.596+0000 7f1c0affd700 10 client.?.objecter ms_handle_connect 0x7f1c1410f360 2024-04-14T15:25:54.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.596+0000 7f1c0affd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:25:54.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.596+0000 7f1c0affd700 10 client.?.objecter ms_handle_connect 0x7f1c141012c0 2024-04-14T15:25:54.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.596+0000 7f1c0affd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:25:54.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.596+0000 7f1c0affd700 1 -- 172.21.15.90:0/504621603 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3302238203 0 0) 0x7f1c141aa130 con 0x7f1c1410b7c0 2024-04-14T15:25:54.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.596+0000 7f1c0affd700 1 -- 172.21.15.90:0/504621603 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1c000035b0 con 0x7f1c1410b7c0 2024-04-14T15:25:54.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.596+0000 7f1c0affd700 1 -- 172.21.15.90:0/504621603 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2675675412 0 0) 0x7f1c141a8f50 con 0x7f1c1410f360 2024-04-14T15:25:54.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.596+0000 7f1c0affd700 1 -- 172.21.15.90:0/504621603 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1c141aa130 con 0x7f1c1410f360 2024-04-14T15:25:54.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.596+0000 7f1c0affd700 1 -- 172.21.15.90:0/504621603 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2533775676 0 0) 0x7f1c14108200 con 0x7f1c141012c0 2024-04-14T15:25:54.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.596+0000 7f1c0affd700 1 -- 172.21.15.90:0/504621603 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1c141a8f50 con 0x7f1c141012c0 2024-04-14T15:25:54.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.596+0000 7f1c0affd700 1 -- 172.21.15.90:0/504621603 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2012677196 0 0) 0x7f1c000035b0 con 0x7f1c1410b7c0 2024-04-14T15:25:54.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.596+0000 7f1c0affd700 1 -- 172.21.15.90:0/504621603 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1c14108200 con 0x7f1c1410b7c0 2024-04-14T15:25:54.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.597+0000 7f1c0affd700 1 -- 172.21.15.90:0/504621603 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f1c04001930 con 0x7f1c1410b7c0 2024-04-14T15:25:54.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.597+0000 7f1c0affd700 1 -- 172.21.15.90:0/504621603 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1597834230 0 0) 0x7f1c141aa130 con 0x7f1c1410f360 2024-04-14T15:25:54.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.597+0000 7f1c0affd700 1 -- 172.21.15.90:0/504621603 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1c000035b0 con 0x7f1c1410f360 2024-04-14T15:25:54.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.597+0000 7f1c0affd700 1 -- 172.21.15.90:0/504621603 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3481340370 0 0) 0x7f1c141a8f50 con 0x7f1c141012c0 2024-04-14T15:25:54.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.597+0000 7f1c0affd700 1 -- 172.21.15.90:0/504621603 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1c141aa130 con 0x7f1c141012c0 2024-04-14T15:25:54.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.597+0000 7f1c0affd700 1 -- 172.21.15.90:0/504621603 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f1c0c001bb0 con 0x7f1c1410f360 2024-04-14T15:25:54.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.597+0000 7f1c0affd700 1 -- 172.21.15.90:0/504621603 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f1c10001430 con 0x7f1c141012c0 2024-04-14T15:25:54.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.597+0000 7f1c0affd700 1 -- 172.21.15.90:0/504621603 <== mon.2 v1:172.21.15.90:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 256709292 0 0) 0x7f1c14108200 con 0x7f1c1410b7c0 2024-04-14T15:25:54.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.597+0000 7f1c0affd700 1 -- 172.21.15.90:0/504621603 >> v1:172.21.15.92:6789/0 conn(0x7f1c1410f360 legacy=0x7f1c141a6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:54.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.597+0000 7f1c0affd700 1 -- 172.21.15.90:0/504621603 >> v1:172.21.15.90:6789/0 conn(0x7f1c141012c0 legacy=0x7f1c14105050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:54.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.597+0000 7f1c0affd700 1 -- 172.21.15.90:0/504621603 --> v1:172.21.15.90:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c141ab310 con 0x7f1c1410b7c0 2024-04-14T15:25:54.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.597+0000 7f1c1bd6b700 1 -- 172.21.15.90:0/504621603 --> v1:172.21.15.90:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1c141a9180 con 0x7f1c1410b7c0 2024-04-14T15:25:54.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.597+0000 7f1c1bd6b700 10 client.48011.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:25:54.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.597+0000 7f1c1bd6b700 1 -- 172.21.15.90:0/504621603 --> v1:172.21.15.90:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f1c141a9640 con 0x7f1c1410b7c0 2024-04-14T15:25:54.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.597+0000 7f1c1bd6b700 1 librados: init done 2024-04-14T15:25:54.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.597+0000 7f1c0affd700 1 -- 172.21.15.90:0/504621603 <== mon.2 v1:172.21.15.90:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f1c04001d20 con 0x7f1c1410b7c0 2024-04-14T15:25:54.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.597+0000 7f1c0affd700 1 -- 172.21.15.90:0/504621603 <== mon.2 v1:172.21.15.90:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f1c04005d80 con 0x7f1c1410b7c0 2024-04-14T15:25:54.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.598+0000 7f1c1bd6b700 1 -- 172.21.15.90:0/504621603 --> v1:172.21.15.90:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bf8005120 con 0x7f1c1410b7c0 2024-04-14T15:25:54.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.599+0000 7f1c0affd700 1 -- 172.21.15.90:0/504621603 <== mon.2 v1:172.21.15.90:6790/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f1c0401ca10 con 0x7f1c1410b7c0 2024-04-14T15:25:54.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.600+0000 7f1c0affd700 1 -- 172.21.15.90:0/504621603 <== mon.2 v1:172.21.15.90:6790/0 8 ==== osd_map(804..804 src has 467..804) ==== 8816+0+0 (unknown 2784028544 0 0) 0x7f1c0408ad10 con 0x7f1c1410b7c0 2024-04-14T15:25:54.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.600+0000 7f1c0affd700 10 client.48011.objecter ms_dispatch 0x7f1c14001550 osd_map(804..804 src has 467..804) 2024-04-14T15:25:54.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.600+0000 7f1c0affd700 3 client.48011.objecter handle_osd_map got epochs [804,804] > 0 2024-04-14T15:25:54.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.600+0000 7f1c0affd700 3 client.48011.objecter handle_osd_map decoding full epoch 804 2024-04-14T15:25:54.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.600+0000 7f1c0affd700 20 client.48011.objecter dump_active .. 0 homeless 2024-04-14T15:25:54.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.603+0000 7f1c0affd700 10 client.48011.objecter ms_handle_connect 0x7f1c000715a0 2024-04-14T15:25:54.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.609+0000 7f1c0affd700 1 -- 172.21.15.90:0/504621603 <== mon.2 v1:172.21.15.90:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f1c04057490 con 0x7f1c1410b7c0 2024-04-14T15:25:54.613 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:25:54.638 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:54.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:54.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:25:54.739 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:25:54.763 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:54.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:54.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:25:54.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:54.824+0000 7f1c1bd6b700 1 -- 172.21.15.90:0/504621603 --> v1:172.21.15.90:6790/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7f1bf8005410 con 0x7f1c1410b7c0 2024-04-14T15:25:54.865 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:25:54.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:54.889 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:54.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:25:54.923 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-14T15:25:54.990 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:25:55.029 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:55.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:55.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:25:55.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:55.035+0000 7f1c0affd700 1 -- 172.21.15.90:0/504621603 <== mon.2 v1:172.21.15.90:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v805) ==== 101+0+0 (unknown 2604866479 0 0) 0x7f1c0405b140 con 0x7f1c1410b7c0 2024-04-14T15:25:55.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:55.038+0000 7f1c1bd6b700 10 librados: watch_flush enter 2024-04-14T15:25:55.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:55.038+0000 7f1c1bd6b700 10 librados: watch_flush exit 2024-04-14T15:25:55.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:55.038+0000 7f1c1bd6b700 20 client.48011.objecter shutdown clearing up homeless session... 2024-04-14T15:25:55.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:55.038+0000 7f1c1bd6b700 10 client.48011.objecter successfully canceled tick 2024-04-14T15:25:55.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:55.039+0000 7f1c1bd6b700 1 -- 172.21.15.90:0/504621603 >> v1:172.21.15.90:6800/420935427 conn(0x7f1c000715a0 legacy=0x7f1c00073a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:55.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:55.039+0000 7f1c1bd6b700 1 -- 172.21.15.90:0/504621603 >> v1:172.21.15.90:6790/0 conn(0x7f1c1410b7c0 legacy=0x7f1c14105760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:55.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:55.039+0000 7f1c1bd6b700 1 -- 172.21.15.90:0/504621603 shutdown_connections 2024-04-14T15:25:55.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:55.039+0000 7f1c1bd6b700 1 -- 172.21.15.90:0/504621603 >> 172.21.15.90:0/504621603 conn(0x7f1c14076a70 msgr2=0x7f1c14111800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:25:55.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:55.040+0000 7f1c1bd6b700 1 -- 172.21.15.90:0/504621603 shutdown_connections 2024-04-14T15:25:55.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:55.040+0000 7f1c1bd6b700 1 -- 172.21.15.90:0/504621603 wait complete. 2024-04-14T15:25:55.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:55.040+0000 7f1c1bd6b700 1 librados: shutdown 2024-04-14T15:25:55.041 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is unset 2024-04-14T15:25:55.131 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:25:55.156 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:55.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:55.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:25:55.210 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:25:55.258 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:25:55.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:55.282 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:55.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:25:55.292 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:55 smithi090 ceph-mon[41670]: 2.c scrub starts 2024-04-14T15:25:55.292 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:55 smithi090 ceph-mon[41670]: 2.c scrub ok 2024-04-14T15:25:55.292 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:55 smithi090 ceph-mon[41670]: 11.9 scrub starts 2024-04-14T15:25:55.292 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:55 smithi090 ceph-mon[41670]: 11.9 scrub ok 2024-04-14T15:25:55.292 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:55 smithi090 ceph-mon[41670]: 11.0 scrub starts 2024-04-14T15:25:55.292 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:55 smithi090 ceph-mon[41670]: 11.0 scrub ok 2024-04-14T15:25:55.292 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:55 smithi090 ceph-mon[41670]: 6.16 scrub starts 2024-04-14T15:25:55.292 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:55 smithi090 ceph-mon[41670]: 6.16 scrub ok 2024-04-14T15:25:55.292 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:55 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:25:55.292 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:55 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/504621603' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:25:55.293 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:55 smithi090 ceph-mon[41670]: from='client.48011 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:25:55.293 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:55 smithi090 ceph-mon[41670]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-14T15:25:55.293 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:55 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-142","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-04-14T15:25:55.293 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:55 smithi090 ceph-mon[41670]: from='client.48011 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-14T15:25:55.293 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:55 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-04-14T15:25:55.293 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:55 smithi090 ceph-mon[41670]: osdmap e805: 8 total, 8 up, 7 in 2024-04-14T15:25:55.293 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:55 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-04-14T15:25:55.293 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:55 smithi090 ceph-mon[30399]: 2.c scrub starts 2024-04-14T15:25:55.293 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:55 smithi090 ceph-mon[30399]: 2.c scrub ok 2024-04-14T15:25:55.293 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:55 smithi090 ceph-mon[30399]: 11.9 scrub starts 2024-04-14T15:25:55.294 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:55 smithi090 ceph-mon[30399]: 11.9 scrub ok 2024-04-14T15:25:55.294 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:55 smithi090 ceph-mon[30399]: 11.0 scrub starts 2024-04-14T15:25:55.294 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:55 smithi090 ceph-mon[30399]: 11.0 scrub ok 2024-04-14T15:25:55.294 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:55 smithi090 ceph-mon[30399]: 6.16 scrub starts 2024-04-14T15:25:55.294 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:55 smithi090 ceph-mon[30399]: 6.16 scrub ok 2024-04-14T15:25:55.294 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:55 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:25:55.294 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:55 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/504621603' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:25:55.294 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:55 smithi090 ceph-mon[30399]: from='client.48011 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:25:55.294 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:55 smithi090 ceph-mon[30399]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-14T15:25:55.294 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:55 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-142","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-04-14T15:25:55.294 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:55 smithi090 ceph-mon[30399]: from='client.48011 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-14T15:25:55.294 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:55 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-04-14T15:25:55.295 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:55 smithi090 ceph-mon[30399]: osdmap e805: 8 total, 8 up, 7 in 2024-04-14T15:25:55.295 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:55 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-04-14T15:25:55.319 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-04-14T15:25:55.320 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph pg dump --format=json 2024-04-14T15:25:55.383 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:25:55.406 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:55.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:55.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:25:55.500 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:55 smithi092 ceph-mon[33966]: 2.c scrub starts 2024-04-14T15:25:55.500 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:55 smithi092 ceph-mon[33966]: 2.c scrub ok 2024-04-14T15:25:55.500 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:55 smithi092 ceph-mon[33966]: 11.9 scrub starts 2024-04-14T15:25:55.500 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:55 smithi092 ceph-mon[33966]: 11.9 scrub ok 2024-04-14T15:25:55.500 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:55 smithi092 ceph-mon[33966]: 11.0 scrub starts 2024-04-14T15:25:55.500 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:55 smithi092 ceph-mon[33966]: 11.0 scrub ok 2024-04-14T15:25:55.500 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:55 smithi092 ceph-mon[33966]: 6.16 scrub starts 2024-04-14T15:25:55.501 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:55 smithi092 ceph-mon[33966]: 6.16 scrub ok 2024-04-14T15:25:55.501 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:55 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:25:55.501 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:55 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/504621603' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:25:55.501 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:55 smithi092 ceph-mon[33966]: from='client.48011 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:25:55.501 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:55 smithi092 ceph-mon[33966]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-14T15:25:55.501 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:55 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-142","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-04-14T15:25:55.501 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:55 smithi092 ceph-mon[33966]: from='client.48011 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-14T15:25:55.501 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:55 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-04-14T15:25:55.501 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:55 smithi092 ceph-mon[33966]: osdmap e805: 8 total, 8 up, 7 in 2024-04-14T15:25:55.501 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:55 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-04-14T15:25:55.508 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:55.532 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:55.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:55.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:55.634 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:25:55.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:55.658 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:55.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:25:55.759 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:25:55.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:55.784 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:55.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:25:55.792 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:25:55 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:25:55.885 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:25:55.910 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:55.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:55.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:25:56.011 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:25:56.036 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:56.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:56.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:25:56.138 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:25:56.154 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:25:56.162 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:56.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:56.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:25:56.264 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:25:56.289 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:56.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:56.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:25:56.391 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:56.415 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:56.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:56.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:56.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:56 smithi090 ceph-mon[30399]: 323.1s0 scrub starts 2024-04-14T15:25:56.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:56 smithi090 ceph-mon[30399]: 323.1 scrub ok 2024-04-14T15:25:56.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:56 smithi090 ceph-mon[30399]: pgmap v1195: 224 pgs: 1 active+remapped+backfill_toofull, 223 active+clean; 4.4 MiB data, 756 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:25:56.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:56 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:25:56.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:56 smithi090 ceph-mon[30399]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:25:56.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:56 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-142","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-04-14T15:25:56.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:56 smithi090 ceph-mon[30399]: osdmap e806: 8 total, 8 up, 7 in 2024-04-14T15:25:56.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:56 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-04-14T15:25:56.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:56 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-04-14T15:25:56.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:56 smithi090 ceph-mon[41670]: 323.1s0 scrub starts 2024-04-14T15:25:56.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:56 smithi090 ceph-mon[41670]: 323.1 scrub ok 2024-04-14T15:25:56.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:56 smithi090 ceph-mon[41670]: pgmap v1195: 224 pgs: 1 active+remapped+backfill_toofull, 223 active+clean; 4.4 MiB data, 756 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:25:56.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:56 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:25:56.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:56 smithi090 ceph-mon[41670]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:25:56.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:56 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-142","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-04-14T15:25:56.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:56 smithi090 ceph-mon[41670]: osdmap e806: 8 total, 8 up, 7 in 2024-04-14T15:25:56.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:56 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-04-14T15:25:56.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:56 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-04-14T15:25:56.517 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:25:56.541 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:56.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:56.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:25:56.596 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:56 smithi092 ceph-mon[33966]: 323.1s0 scrub starts 2024-04-14T15:25:56.597 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:56 smithi092 ceph-mon[33966]: 323.1 scrub ok 2024-04-14T15:25:56.597 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:56 smithi092 ceph-mon[33966]: pgmap v1195: 224 pgs: 1 active+remapped+backfill_toofull, 223 active+clean; 4.4 MiB data, 756 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:25:56.597 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:56 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:25:56.597 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:56 smithi092 ceph-mon[33966]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:25:56.597 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:56 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-142","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-04-14T15:25:56.597 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:56 smithi092 ceph-mon[33966]: osdmap e806: 8 total, 8 up, 7 in 2024-04-14T15:25:56.597 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:56 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-04-14T15:25:56.597 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:56 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-04-14T15:25:56.642 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:25:56.666 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:56.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:56.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:25:56.768 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:25:56.792 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:56.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:56.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:25:56.894 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:25:56.918 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:56.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:56.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:25:57.020 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:25:57.045 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:57.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:57.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:25:57.147 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:25:57.172 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:57.235 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-04-14T15:25:57.235 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-04-14T15:25:57.235 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-04-14T15:25:57.235 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [] 2024-04-14T15:25:57.235 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-04-14T15:25:57.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:57.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:25:57.337 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:25:57.361 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:57.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:57.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:25:57.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:57 smithi090 ceph-mon[41670]: 4.1f scrub starts 2024-04-14T15:25:57.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:57 smithi090 ceph-mon[41670]: 4.1f scrub ok 2024-04-14T15:25:57.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:57 smithi090 ceph-mon[41670]: 2.5 scrub starts 2024-04-14T15:25:57.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:57 smithi090 ceph-mon[41670]: 2.5 scrub ok 2024-04-14T15:25:57.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:57 smithi090 ceph-mon[41670]: 4.7 scrub starts 2024-04-14T15:25:57.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:57 smithi090 ceph-mon[41670]: 4.7 scrub ok 2024-04-14T15:25:57.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:57 smithi090 ceph-mon[41670]: 4.18 deep-scrub starts 2024-04-14T15:25:57.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:57 smithi090 ceph-mon[41670]: 4.18 deep-scrub ok 2024-04-14T15:25:57.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:57 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-142","var": "hit_set_search_last_n","val": "1"}]': finished 2024-04-14T15:25:57.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:57 smithi090 ceph-mon[41670]: osdmap e807: 8 total, 8 up, 7 in 2024-04-14T15:25:57.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:57 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:25:57.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:57 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:25:57.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:57 smithi090 ceph-mon[30399]: 4.1f scrub starts 2024-04-14T15:25:57.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:57 smithi090 ceph-mon[30399]: 4.1f scrub ok 2024-04-14T15:25:57.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:57 smithi090 ceph-mon[30399]: 2.5 scrub starts 2024-04-14T15:25:57.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:57 smithi090 ceph-mon[30399]: 2.5 scrub ok 2024-04-14T15:25:57.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:57 smithi090 ceph-mon[30399]: 4.7 scrub starts 2024-04-14T15:25:57.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:57 smithi090 ceph-mon[30399]: 4.7 scrub ok 2024-04-14T15:25:57.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:57 smithi090 ceph-mon[30399]: 4.18 deep-scrub starts 2024-04-14T15:25:57.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:57 smithi090 ceph-mon[30399]: 4.18 deep-scrub ok 2024-04-14T15:25:57.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:57 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-142","var": "hit_set_search_last_n","val": "1"}]': finished 2024-04-14T15:25:57.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:57 smithi090 ceph-mon[30399]: osdmap e807: 8 total, 8 up, 7 in 2024-04-14T15:25:57.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:57 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:25:57.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:57 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:25:57.462 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:25:57.485 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:57.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:57.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:25:57.488 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:57 smithi092 ceph-mon[33966]: 4.1f scrub starts 2024-04-14T15:25:57.488 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:57 smithi092 ceph-mon[33966]: 4.1f scrub ok 2024-04-14T15:25:57.488 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:57 smithi092 ceph-mon[33966]: 2.5 scrub starts 2024-04-14T15:25:57.488 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:57 smithi092 ceph-mon[33966]: 2.5 scrub ok 2024-04-14T15:25:57.488 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:57 smithi092 ceph-mon[33966]: 4.7 scrub starts 2024-04-14T15:25:57.488 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:57 smithi092 ceph-mon[33966]: 4.7 scrub ok 2024-04-14T15:25:57.488 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:57 smithi092 ceph-mon[33966]: 4.18 deep-scrub starts 2024-04-14T15:25:57.488 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:57 smithi092 ceph-mon[33966]: 4.18 deep-scrub ok 2024-04-14T15:25:57.488 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:57 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-142","var": "hit_set_search_last_n","val": "1"}]': finished 2024-04-14T15:25:57.488 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:57 smithi092 ceph-mon[33966]: osdmap e807: 8 total, 8 up, 7 in 2024-04-14T15:25:57.488 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:57 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:25:57.488 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:57 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:25:57.587 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:25:57.611 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:57.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:57.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:25:57.712 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:25:57.736 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:57.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:57.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:25:57.790 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-14T15:25:57.838 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:57.863 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:57.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:57.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:57.871 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:25:57.965 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:25:57.990 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:57.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:57.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:25:58.032 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:25:58.092 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:25:58.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:58.116 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:58.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:25:58.155 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd set noscrub 2024-04-14T15:25:58.217 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:58.241 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:58.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:58.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:58.325 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:58 smithi090 ceph-mon[41670]: 5.b deep-scrub starts 2024-04-14T15:25:58.325 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:58 smithi090 ceph-mon[41670]: 5.b deep-scrub ok 2024-04-14T15:25:58.325 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:58 smithi090 ceph-mon[41670]: 11.1c scrub starts 2024-04-14T15:25:58.325 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:58 smithi090 ceph-mon[41670]: 11.1c scrub ok 2024-04-14T15:25:58.325 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:58 smithi090 ceph-mon[41670]: pgmap v1198: 224 pgs: 1 active+remapped+backfill_toofull, 223 active+clean; 4.4 MiB data, 756 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:25:58.325 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:58 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]': finished 2024-04-14T15:25:58.325 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:58 smithi090 ceph-mon[41670]: osdmap e808: 8 total, 8 up, 7 in 2024-04-14T15:25:58.326 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:58 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-142"}]: dispatch 2024-04-14T15:25:58.326 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:58 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-142"}]: dispatch 2024-04-14T15:25:58.326 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:58 smithi090 ceph-mon[30399]: 5.b deep-scrub starts 2024-04-14T15:25:58.326 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:58 smithi090 ceph-mon[30399]: 5.b deep-scrub ok 2024-04-14T15:25:58.326 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:58 smithi090 ceph-mon[30399]: 11.1c scrub starts 2024-04-14T15:25:58.326 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:58 smithi090 ceph-mon[30399]: 11.1c scrub ok 2024-04-14T15:25:58.326 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:58 smithi090 ceph-mon[30399]: pgmap v1198: 224 pgs: 1 active+remapped+backfill_toofull, 223 active+clean; 4.4 MiB data, 756 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:25:58.326 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:58 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]': finished 2024-04-14T15:25:58.326 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:58 smithi090 ceph-mon[30399]: osdmap e808: 8 total, 8 up, 7 in 2024-04-14T15:25:58.326 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:58 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-142"}]: dispatch 2024-04-14T15:25:58.327 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:58 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-142"}]: dispatch 2024-04-14T15:25:58.343 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:58.366 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:58.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:58.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:58.449 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:58 smithi092 ceph-mon[33966]: 5.b deep-scrub starts 2024-04-14T15:25:58.450 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:58 smithi092 ceph-mon[33966]: 5.b deep-scrub ok 2024-04-14T15:25:58.450 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:58 smithi092 ceph-mon[33966]: 11.1c scrub starts 2024-04-14T15:25:58.450 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:58 smithi092 ceph-mon[33966]: 11.1c scrub ok 2024-04-14T15:25:58.450 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:58 smithi092 ceph-mon[33966]: pgmap v1198: 224 pgs: 1 active+remapped+backfill_toofull, 223 active+clean; 4.4 MiB data, 756 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:25:58.450 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:58 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]': finished 2024-04-14T15:25:58.450 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:58 smithi092 ceph-mon[33966]: osdmap e808: 8 total, 8 up, 7 in 2024-04-14T15:25:58.450 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:58 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-142"}]: dispatch 2024-04-14T15:25:58.450 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:58 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-142"}]: dispatch 2024-04-14T15:25:58.467 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:25:58.492 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:58.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:58.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:25:58.594 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:25:58.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:58.618 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:58.618 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:25:58.719 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:25:58.743 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:58.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:58.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:25:58.845 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:25:58.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:58.869 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:58.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:25:58.970 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:25:58.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:58.995 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:58.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:25:59.096 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:25:59.120 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:59.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:59.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:25:59.222 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:25:59.228 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:59 smithi092 ceph-mon[33966]: 5.8 scrub starts 2024-04-14T15:25:59.228 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:59 smithi092 ceph-mon[33966]: 5.8 scrub ok 2024-04-14T15:25:59.228 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:59 smithi092 ceph-mon[33966]: 2.12 scrub starts 2024-04-14T15:25:59.229 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:59 smithi092 ceph-mon[33966]: 2.12 scrub ok 2024-04-14T15:25:59.229 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:59 smithi092 ceph-mon[33966]: 4.3 scrub starts 2024-04-14T15:25:59.229 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:59 smithi092 ceph-mon[33966]: 4.3 scrub ok 2024-04-14T15:25:59.229 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:59 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-142"}]': finished 2024-04-14T15:25:59.229 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:59 smithi092 ceph-mon[33966]: osdmap e809: 8 total, 8 up, 7 in 2024-04-14T15:25:59.229 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:59 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:25:59.229 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:59 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:25:59.229 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:59 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-142"}]: dispatch 2024-04-14T15:25:59.229 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:25:59 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-142"}]: dispatch 2024-04-14T15:25:59.247 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:59.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:59.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:25:59.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.306+0000 7fd48b74c700 1 -- 172.21.15.90:0/3368335271 >> v1:172.21.15.90:6790/0 conn(0x7fd48410f3c0 legacy=0x7fd4841117d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:59.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.306+0000 7fd48b74c700 1 -- 172.21.15.90:0/3368335271 shutdown_connections 2024-04-14T15:25:59.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.306+0000 7fd48b74c700 1 -- 172.21.15.90:0/3368335271 >> 172.21.15.90:0/3368335271 conn(0x7fd4840fd290 msgr2=0x7fd4840ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:25:59.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.306+0000 7fd48b74c700 1 -- 172.21.15.90:0/3368335271 shutdown_connections 2024-04-14T15:25:59.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.306+0000 7fd48b74c700 1 -- 172.21.15.90:0/3368335271 wait complete. 2024-04-14T15:25:59.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.307+0000 7fd48b74c700 1 librados: starting msgr at 2024-04-14T15:25:59.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.307+0000 7fd48b74c700 1 librados: starting objecter 2024-04-14T15:25:59.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.307+0000 7fd48b74c700 1 Processor -- start 2024-04-14T15:25:59.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.307+0000 7fd48b74c700 1 -- start start 2024-04-14T15:25:59.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.307+0000 7fd48b74c700 1 librados: setting wanted keys 2024-04-14T15:25:59.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.307+0000 7fd48b74c700 1 librados: calling monclient init 2024-04-14T15:25:59.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.307+0000 7fd48b74c700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd48410b2d0 con 0x7fd484103aa0 2024-04-14T15:25:59.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.307+0000 7fd48b74c700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd48410b4e0 con 0x7fd48410f3c0 2024-04-14T15:25:59.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.307+0000 7fd48b74c700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd4841ac2c0 con 0x7fd48410b8b0 2024-04-14T15:25:59.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.307+0000 7fd4894e8700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fd484103aa0 0x7fd484101700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:56236/0 (socket says 172.21.15.90:56236) 2024-04-14T15:25:59.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.307+0000 7fd4894e8700 1 -- 172.21.15.90:0/3088153630 learned_addr learned my addr 172.21.15.90:0/3088153630 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:25:59.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.308+0000 7fd47a7fc700 10 client.?.objecter ms_handle_connect 0x7fd484103aa0 2024-04-14T15:25:59.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.308+0000 7fd47a7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:25:59.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.308+0000 7fd47a7fc700 10 client.?.objecter ms_handle_connect 0x7fd48410b8b0 2024-04-14T15:25:59.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.308+0000 7fd47a7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:25:59.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.308+0000 7fd47a7fc700 10 client.?.objecter ms_handle_connect 0x7fd48410f3c0 2024-04-14T15:25:59.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.308+0000 7fd47a7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:25:59.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.308+0000 7fd47a7fc700 1 -- 172.21.15.90:0/3088153630 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 359423177 0 0) 0x7fd48410b2d0 con 0x7fd484103aa0 2024-04-14T15:25:59.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.308+0000 7fd47a7fc700 1 -- 172.21.15.90:0/3088153630 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd4700031f0 con 0x7fd484103aa0 2024-04-14T15:25:59.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.308+0000 7fd47a7fc700 1 -- 172.21.15.90:0/3088153630 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 50321724 0 0) 0x7fd4841ac2c0 con 0x7fd48410b8b0 2024-04-14T15:25:59.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.308+0000 7fd47a7fc700 1 -- 172.21.15.90:0/3088153630 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd48410b2d0 con 0x7fd48410b8b0 2024-04-14T15:25:59.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.308+0000 7fd47a7fc700 1 -- 172.21.15.90:0/3088153630 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 915072755 0 0) 0x7fd48410b4e0 con 0x7fd48410f3c0 2024-04-14T15:25:59.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.308+0000 7fd47a7fc700 1 -- 172.21.15.90:0/3088153630 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd4841ac2c0 con 0x7fd48410f3c0 2024-04-14T15:25:59.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.308+0000 7fd47a7fc700 1 -- 172.21.15.90:0/3088153630 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3207308819 0 0) 0x7fd4700031f0 con 0x7fd484103aa0 2024-04-14T15:25:59.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.308+0000 7fd47a7fc700 1 -- 172.21.15.90:0/3088153630 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd48410b4e0 con 0x7fd484103aa0 2024-04-14T15:25:59.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.308+0000 7fd47a7fc700 1 -- 172.21.15.90:0/3088153630 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fd47c001cb0 con 0x7fd484103aa0 2024-04-14T15:25:59.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.308+0000 7fd47a7fc700 1 -- 172.21.15.90:0/3088153630 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1177853400 0 0) 0x7fd48410b2d0 con 0x7fd48410b8b0 2024-04-14T15:25:59.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.308+0000 7fd47a7fc700 1 -- 172.21.15.90:0/3088153630 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd4700031f0 con 0x7fd48410b8b0 2024-04-14T15:25:59.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.308+0000 7fd47a7fc700 1 -- 172.21.15.90:0/3088153630 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fd474001a30 con 0x7fd48410b8b0 2024-04-14T15:25:59.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.308+0000 7fd47a7fc700 1 -- 172.21.15.90:0/3088153630 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2942575967 0 0) 0x7fd4841ac2c0 con 0x7fd48410f3c0 2024-04-14T15:25:59.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.308+0000 7fd47a7fc700 1 -- 172.21.15.90:0/3088153630 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd48410b2d0 con 0x7fd48410f3c0 2024-04-14T15:25:59.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.308+0000 7fd47a7fc700 1 -- 172.21.15.90:0/3088153630 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fd480001430 con 0x7fd48410f3c0 2024-04-14T15:25:59.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.308+0000 7fd47a7fc700 1 -- 172.21.15.90:0/3088153630 <== mon.2 v1:172.21.15.90:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3544281965 0 0) 0x7fd4700031f0 con 0x7fd48410b8b0 2024-04-14T15:25:59.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.308+0000 7fd47a7fc700 1 -- 172.21.15.90:0/3088153630 >> v1:172.21.15.92:6789/0 conn(0x7fd48410f3c0 legacy=0x7fd4841a8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:59.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.308+0000 7fd47a7fc700 1 -- 172.21.15.90:0/3088153630 >> v1:172.21.15.90:6789/0 conn(0x7fd484103aa0 legacy=0x7fd484101700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:59.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.308+0000 7fd47a7fc700 1 -- 172.21.15.90:0/3088153630 --> v1:172.21.15.90:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4841ad4a0 con 0x7fd48410b8b0 2024-04-14T15:25:59.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.308+0000 7fd48b74c700 1 -- 172.21.15.90:0/3088153630 --> v1:172.21.15.90:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd4841ac490 con 0x7fd48410b8b0 2024-04-14T15:25:59.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.308+0000 7fd48b74c700 10 client.48017.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:25:59.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.308+0000 7fd48b74c700 1 -- 172.21.15.90:0/3088153630 --> v1:172.21.15.90:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fd4841ac950 con 0x7fd48410b8b0 2024-04-14T15:25:59.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.308+0000 7fd48b74c700 1 librados: init done 2024-04-14T15:25:59.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.308+0000 7fd47a7fc700 1 -- 172.21.15.90:0/3088153630 <== mon.2 v1:172.21.15.90:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fd474001bd0 con 0x7fd48410b8b0 2024-04-14T15:25:59.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.308+0000 7fd47a7fc700 1 -- 172.21.15.90:0/3088153630 <== mon.2 v1:172.21.15.90:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fd474004980 con 0x7fd48410b8b0 2024-04-14T15:25:59.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.309+0000 7fd48b74c700 1 -- 172.21.15.90:0/3088153630 --> v1:172.21.15.90:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fd468005100 con 0x7fd48410b8b0 2024-04-14T15:25:59.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.310+0000 7fd47a7fc700 1 -- 172.21.15.90:0/3088153630 <== mon.2 v1:172.21.15.90:6790/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fd474003930 con 0x7fd48410b8b0 2024-04-14T15:25:59.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.310+0000 7fd47a7fc700 1 -- 172.21.15.90:0/3088153630 <== mon.2 v1:172.21.15.90:6790/0 8 ==== osd_map(809..809 src has 467..809) ==== 8782+0+0 (unknown 2830735564 0 0) 0x7fd474089870 con 0x7fd48410b8b0 2024-04-14T15:25:59.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.310+0000 7fd47a7fc700 10 client.48017.objecter ms_dispatch 0x7fd484001550 osd_map(809..809 src has 467..809) 2024-04-14T15:25:59.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.310+0000 7fd47a7fc700 3 client.48017.objecter handle_osd_map got epochs [809,809] > 0 2024-04-14T15:25:59.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.310+0000 7fd47a7fc700 3 client.48017.objecter handle_osd_map decoding full epoch 809 2024-04-14T15:25:59.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.310+0000 7fd47a7fc700 20 client.48017.objecter dump_active .. 0 homeless 2024-04-14T15:25:59.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.310+0000 7fd47a7fc700 1 -- 172.21.15.90:0/3088153630 <== mon.2 v1:172.21.15.90:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v809) ==== 54+0+16 (unknown 584948641 0 2420000192) 0x7fd474089cb0 con 0x7fd48410b8b0 2024-04-14T15:25:59.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.310+0000 7fd47a7fc700 10 client.48017.objecter ms_handle_connect 0x7fd470071340 2024-04-14T15:25:59.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.311+0000 7fd48b74c700 10 client.48017.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:25:59.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.311+0000 7fd48b74c700 15 client.48017.objecter _session_command_op_assign -1 1 2024-04-14T15:25:59.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.311+0000 7fd48b74c700 20 client.48017.objecter _get_session s=0x7fd4680053e0 osd=0 3 2024-04-14T15:25:59.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.311+0000 7fd48b74c700 20 client.48017.objecter put_session s=0x7fd4680053e0 osd=0 3 2024-04-14T15:25:59.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.311+0000 7fd48b74c700 20 client.48017.objecter _get_session s=0x7fd4680053e0 osd=0 3 2024-04-14T15:25:59.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.311+0000 7fd48b74c700 15 client.48017.objecter _session_command_op_remove -1 1 2024-04-14T15:25:59.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.311+0000 7fd48b74c700 20 client.48017.objecter get_session s=0x7fd4680053e0 osd=0 3 2024-04-14T15:25:59.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.311+0000 7fd48b74c700 15 client.48017.objecter _session_command_op_assign 0 1 2024-04-14T15:25:59.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.311+0000 7fd48b74c700 20 client.48017.objecter put_session s=0x7fd4680053e0 osd=0 4 2024-04-14T15:25:59.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.311+0000 7fd48b74c700 10 client.48017.objecter _send_command 1 2024-04-14T15:25:59.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.311+0000 7fd48b74c700 1 -- 172.21.15.90:0/3088153630 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd468004b50 con 0x7fd468004730 2024-04-14T15:25:59.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.311+0000 7fd47a7fc700 10 client.48017.objecter ms_handle_connect 0x7fd468004730 2024-04-14T15:25:59.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.312+0000 7fd47a7fc700 1 -- 172.21.15.90:0/3088153630 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd468004b50 con 0x7fd468004730 2024-04-14T15:25:59.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.312+0000 7fd47a7fc700 10 client.48017.objecter ms_dispatch 0x7fd484001550 command_reply(tid 1: 0 ) 2024-04-14T15:25:59.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.312+0000 7fd47a7fc700 10 client.48017.objecter _finish_command 1 = system:0 2024-04-14T15:25:59.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.312+0000 7fd47a7fc700 20 client.48017.objecter put_session s=0x7fd4680053e0 osd=0 4 2024-04-14T15:25:59.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.312+0000 7fd47a7fc700 15 client.48017.objecter _session_command_op_remove 0 1 2024-04-14T15:25:59.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.328+0000 7fd48b74c700 10 client.48017.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:25:59.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.328+0000 7fd48b74c700 15 client.48017.objecter _session_command_op_assign -1 2 2024-04-14T15:25:59.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.328+0000 7fd48b74c700 20 client.48017.objecter _get_session s=0x7fd4680053e0 osd=0 3 2024-04-14T15:25:59.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.328+0000 7fd48b74c700 20 client.48017.objecter put_session s=0x7fd4680053e0 osd=0 3 2024-04-14T15:25:59.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.328+0000 7fd48b74c700 20 client.48017.objecter _get_session s=0x7fd4680053e0 osd=0 3 2024-04-14T15:25:59.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.328+0000 7fd48b74c700 15 client.48017.objecter _session_command_op_remove -1 2 2024-04-14T15:25:59.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.328+0000 7fd48b74c700 20 client.48017.objecter get_session s=0x7fd4680053e0 osd=0 3 2024-04-14T15:25:59.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.328+0000 7fd48b74c700 15 client.48017.objecter _session_command_op_assign 0 2 2024-04-14T15:25:59.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.328+0000 7fd48b74c700 20 client.48017.objecter put_session s=0x7fd4680053e0 osd=0 4 2024-04-14T15:25:59.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.328+0000 7fd48b74c700 10 client.48017.objecter _send_command 2 2024-04-14T15:25:59.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.328+0000 7fd48b74c700 1 -- 172.21.15.90:0/3088153630 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd468007b00 con 0x7fd468004730 2024-04-14T15:25:59.331 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:25:59.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.330+0000 7fd47a7fc700 1 -- 172.21.15.90:0/3088153630 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd468007b00 con 0x7fd468004730 2024-04-14T15:25:59.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.330+0000 7fd47a7fc700 10 client.48017.objecter ms_dispatch 0x7fd484001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:25:59.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.330+0000 7fd47a7fc700 10 client.48017.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:25:59.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.330+0000 7fd47a7fc700 20 client.48017.objecter put_session s=0x7fd4680053e0 osd=0 4 2024-04-14T15:25:59.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.330+0000 7fd47a7fc700 15 client.48017.objecter _session_command_op_remove 0 2 2024-04-14T15:25:59.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.330+0000 7fd48b74c700 10 client.48017.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:25:59.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.330+0000 7fd48b74c700 15 client.48017.objecter _session_command_op_assign -1 3 2024-04-14T15:25:59.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.330+0000 7fd48b74c700 20 client.48017.objecter _get_session s=0x7fd468004130 osd=1 3 2024-04-14T15:25:59.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.330+0000 7fd48b74c700 20 client.48017.objecter put_session s=0x7fd468004130 osd=1 3 2024-04-14T15:25:59.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.330+0000 7fd48b74c700 20 client.48017.objecter _get_session s=0x7fd468004130 osd=1 3 2024-04-14T15:25:59.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.330+0000 7fd48b74c700 15 client.48017.objecter _session_command_op_remove -1 3 2024-04-14T15:25:59.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.330+0000 7fd48b74c700 20 client.48017.objecter get_session s=0x7fd468004130 osd=1 3 2024-04-14T15:25:59.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.330+0000 7fd48b74c700 15 client.48017.objecter _session_command_op_assign 1 3 2024-04-14T15:25:59.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.330+0000 7fd48b74c700 20 client.48017.objecter put_session s=0x7fd468004130 osd=1 4 2024-04-14T15:25:59.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.330+0000 7fd48b74c700 10 client.48017.objecter _send_command 3 2024-04-14T15:25:59.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.330+0000 7fd48b74c700 1 -- 172.21.15.90:0/3088153630 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fd468002130 con 0x7fd4680042a0 2024-04-14T15:25:59.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.331+0000 7fd47a7fc700 10 client.48017.objecter ms_handle_connect 0x7fd4680042a0 2024-04-14T15:25:59.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.331+0000 7fd47a7fc700 1 -- 172.21.15.90:0/3088153630 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd468002130 con 0x7fd4680042a0 2024-04-14T15:25:59.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.331+0000 7fd47a7fc700 10 client.48017.objecter ms_dispatch 0x7fd484001550 command_reply(tid 3: 0 ) 2024-04-14T15:25:59.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.331+0000 7fd47a7fc700 10 client.48017.objecter _finish_command 3 = system:0 2024-04-14T15:25:59.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.331+0000 7fd47a7fc700 20 client.48017.objecter put_session s=0x7fd468004130 osd=1 4 2024-04-14T15:25:59.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.331+0000 7fd47a7fc700 15 client.48017.objecter _session_command_op_remove 1 3 2024-04-14T15:25:59.349 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:25:59.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.353+0000 7fd48b74c700 10 client.48017.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:25:59.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.353+0000 7fd48b74c700 15 client.48017.objecter _session_command_op_assign -1 4 2024-04-14T15:25:59.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.353+0000 7fd48b74c700 20 client.48017.objecter _get_session s=0x7fd468004130 osd=1 3 2024-04-14T15:25:59.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.353+0000 7fd48b74c700 20 client.48017.objecter put_session s=0x7fd468004130 osd=1 3 2024-04-14T15:25:59.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.353+0000 7fd48b74c700 20 client.48017.objecter _get_session s=0x7fd468004130 osd=1 3 2024-04-14T15:25:59.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.353+0000 7fd48b74c700 15 client.48017.objecter _session_command_op_remove -1 4 2024-04-14T15:25:59.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.353+0000 7fd48b74c700 20 client.48017.objecter get_session s=0x7fd468004130 osd=1 3 2024-04-14T15:25:59.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.353+0000 7fd48b74c700 15 client.48017.objecter _session_command_op_assign 1 4 2024-04-14T15:25:59.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.353+0000 7fd48b74c700 20 client.48017.objecter put_session s=0x7fd468004130 osd=1 4 2024-04-14T15:25:59.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.353+0000 7fd48b74c700 10 client.48017.objecter _send_command 4 2024-04-14T15:25:59.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.353+0000 7fd48b74c700 1 -- 172.21.15.90:0/3088153630 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd468001600 con 0x7fd4680042a0 2024-04-14T15:25:59.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.355+0000 7fd47a7fc700 1 -- 172.21.15.90:0/3088153630 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd468001600 con 0x7fd4680042a0 2024-04-14T15:25:59.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.355+0000 7fd47a7fc700 10 client.48017.objecter ms_dispatch 0x7fd484001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:25:59.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.355+0000 7fd47a7fc700 10 client.48017.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:25:59.356 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:25:59.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.355+0000 7fd47a7fc700 20 client.48017.objecter put_session s=0x7fd468004130 osd=1 4 2024-04-14T15:25:59.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.355+0000 7fd47a7fc700 15 client.48017.objecter _session_command_op_remove 1 4 2024-04-14T15:25:59.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.355+0000 7fd48b74c700 10 client.48017.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:25:59.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.355+0000 7fd48b74c700 15 client.48017.objecter _session_command_op_assign -1 5 2024-04-14T15:25:59.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.355+0000 7fd48b74c700 20 client.48017.objecter _get_session s=0x7fd468001a00 osd=2 3 2024-04-14T15:25:59.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.355+0000 7fd48b74c700 20 client.48017.objecter put_session s=0x7fd468001a00 osd=2 3 2024-04-14T15:25:59.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.355+0000 7fd48b74c700 20 client.48017.objecter _get_session s=0x7fd468001a00 osd=2 3 2024-04-14T15:25:59.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.355+0000 7fd48b74c700 15 client.48017.objecter _session_command_op_remove -1 5 2024-04-14T15:25:59.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.355+0000 7fd48b74c700 20 client.48017.objecter get_session s=0x7fd468001a00 osd=2 3 2024-04-14T15:25:59.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.355+0000 7fd48b74c700 15 client.48017.objecter _session_command_op_assign 2 5 2024-04-14T15:25:59.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.355+0000 7fd48b74c700 20 client.48017.objecter put_session s=0x7fd468001a00 osd=2 4 2024-04-14T15:25:59.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.355+0000 7fd48b74c700 10 client.48017.objecter _send_command 5 2024-04-14T15:25:59.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.355+0000 7fd48b74c700 1 -- 172.21.15.90:0/3088153630 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fd468010fe0 con 0x7fd468008990 2024-04-14T15:25:59.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.355+0000 7fd47a7fc700 10 client.48017.objecter ms_handle_connect 0x7fd468008990 2024-04-14T15:25:59.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.356+0000 7fd47a7fc700 1 -- 172.21.15.90:0/3088153630 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd468010fe0 con 0x7fd468008990 2024-04-14T15:25:59.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.356+0000 7fd47a7fc700 10 client.48017.objecter ms_dispatch 0x7fd484001550 command_reply(tid 5: 0 ) 2024-04-14T15:25:59.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.356+0000 7fd47a7fc700 10 client.48017.objecter _finish_command 5 = system:0 2024-04-14T15:25:59.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.356+0000 7fd47a7fc700 20 client.48017.objecter put_session s=0x7fd468001a00 osd=2 4 2024-04-14T15:25:59.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.356+0000 7fd47a7fc700 15 client.48017.objecter _session_command_op_remove 2 5 2024-04-14T15:25:59.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.373+0000 7fd48b74c700 10 client.48017.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:25:59.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.373+0000 7fd48b74c700 15 client.48017.objecter _session_command_op_assign -1 6 2024-04-14T15:25:59.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.373+0000 7fd48b74c700 20 client.48017.objecter _get_session s=0x7fd468001a00 osd=2 3 2024-04-14T15:25:59.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.373+0000 7fd48b74c700 20 client.48017.objecter put_session s=0x7fd468001a00 osd=2 3 2024-04-14T15:25:59.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.373+0000 7fd48b74c700 20 client.48017.objecter _get_session s=0x7fd468001a00 osd=2 3 2024-04-14T15:25:59.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.373+0000 7fd48b74c700 15 client.48017.objecter _session_command_op_remove -1 6 2024-04-14T15:25:59.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.373+0000 7fd48b74c700 20 client.48017.objecter get_session s=0x7fd468001a00 osd=2 3 2024-04-14T15:25:59.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.373+0000 7fd48b74c700 15 client.48017.objecter _session_command_op_assign 2 6 2024-04-14T15:25:59.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.373+0000 7fd48b74c700 20 client.48017.objecter put_session s=0x7fd468001a00 osd=2 4 2024-04-14T15:25:59.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.373+0000 7fd48b74c700 10 client.48017.objecter _send_command 6 2024-04-14T15:25:59.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.373+0000 7fd48b74c700 1 -- 172.21.15.90:0/3088153630 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd468007fa0 con 0x7fd468008990 2024-04-14T15:25:59.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:59.375 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:59.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:25:59.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.375+0000 7fd47a7fc700 1 -- 172.21.15.90:0/3088153630 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd468007fa0 con 0x7fd468008990 2024-04-14T15:25:59.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.375+0000 7fd47a7fc700 10 client.48017.objecter ms_dispatch 0x7fd484001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:25:59.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.375+0000 7fd47a7fc700 10 client.48017.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:25:59.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.375+0000 7fd47a7fc700 20 client.48017.objecter put_session s=0x7fd468001a00 osd=2 4 2024-04-14T15:25:59.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.375+0000 7fd47a7fc700 15 client.48017.objecter _session_command_op_remove 2 6 2024-04-14T15:25:59.376 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:25:59.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.375+0000 7fd48b74c700 10 client.48017.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:25:59.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.375+0000 7fd48b74c700 15 client.48017.objecter _session_command_op_assign -1 7 2024-04-14T15:25:59.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.375+0000 7fd48b74c700 20 client.48017.objecter _get_session s=0x7fd468008200 osd=3 3 2024-04-14T15:25:59.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.375+0000 7fd48b74c700 20 client.48017.objecter put_session s=0x7fd468008200 osd=3 3 2024-04-14T15:25:59.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.375+0000 7fd48b74c700 20 client.48017.objecter _get_session s=0x7fd468008200 osd=3 3 2024-04-14T15:25:59.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.375+0000 7fd48b74c700 15 client.48017.objecter _session_command_op_remove -1 7 2024-04-14T15:25:59.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.375+0000 7fd48b74c700 20 client.48017.objecter get_session s=0x7fd468008200 osd=3 3 2024-04-14T15:25:59.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.375+0000 7fd48b74c700 15 client.48017.objecter _session_command_op_assign 3 7 2024-04-14T15:25:59.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.375+0000 7fd48b74c700 20 client.48017.objecter put_session s=0x7fd468008200 osd=3 4 2024-04-14T15:25:59.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.375+0000 7fd48b74c700 10 client.48017.objecter _send_command 7 2024-04-14T15:25:59.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.375+0000 7fd48b74c700 1 -- 172.21.15.90:0/3088153630 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fd46800d500 con 0x7fd468011190 2024-04-14T15:25:59.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.375+0000 7fd47a7fc700 10 client.48017.objecter ms_handle_connect 0x7fd468011190 2024-04-14T15:25:59.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.376+0000 7fd47a7fc700 1 -- 172.21.15.90:0/3088153630 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd46800d500 con 0x7fd468011190 2024-04-14T15:25:59.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.376+0000 7fd47a7fc700 10 client.48017.objecter ms_dispatch 0x7fd484001550 command_reply(tid 7: 0 ) 2024-04-14T15:25:59.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.376+0000 7fd47a7fc700 10 client.48017.objecter _finish_command 7 = system:0 2024-04-14T15:25:59.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.376+0000 7fd47a7fc700 20 client.48017.objecter put_session s=0x7fd468008200 osd=3 4 2024-04-14T15:25:59.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.376+0000 7fd47a7fc700 15 client.48017.objecter _session_command_op_remove 3 7 2024-04-14T15:25:59.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.393+0000 7fd48b74c700 10 client.48017.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:25:59.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.393+0000 7fd48b74c700 15 client.48017.objecter _session_command_op_assign -1 8 2024-04-14T15:25:59.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.393+0000 7fd48b74c700 20 client.48017.objecter _get_session s=0x7fd468008200 osd=3 3 2024-04-14T15:25:59.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.393+0000 7fd48b74c700 20 client.48017.objecter put_session s=0x7fd468008200 osd=3 3 2024-04-14T15:25:59.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.393+0000 7fd48b74c700 20 client.48017.objecter _get_session s=0x7fd468008200 osd=3 3 2024-04-14T15:25:59.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.393+0000 7fd48b74c700 15 client.48017.objecter _session_command_op_remove -1 8 2024-04-14T15:25:59.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.393+0000 7fd48b74c700 20 client.48017.objecter get_session s=0x7fd468008200 osd=3 3 2024-04-14T15:25:59.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.393+0000 7fd48b74c700 15 client.48017.objecter _session_command_op_assign 3 8 2024-04-14T15:25:59.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.393+0000 7fd48b74c700 20 client.48017.objecter put_session s=0x7fd468008200 osd=3 4 2024-04-14T15:25:59.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.393+0000 7fd48b74c700 10 client.48017.objecter _send_command 8 2024-04-14T15:25:59.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.393+0000 7fd48b74c700 1 -- 172.21.15.90:0/3088153630 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd468012570 con 0x7fd468011190 2024-04-14T15:25:59.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.394+0000 7fd47a7fc700 1 -- 172.21.15.90:0/3088153630 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd468012570 con 0x7fd468011190 2024-04-14T15:25:59.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.394+0000 7fd47a7fc700 10 client.48017.objecter ms_dispatch 0x7fd484001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:25:59.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.394+0000 7fd47a7fc700 10 client.48017.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:25:59.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.394+0000 7fd47a7fc700 20 client.48017.objecter put_session s=0x7fd468008200 osd=3 4 2024-04-14T15:25:59.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.394+0000 7fd47a7fc700 15 client.48017.objecter _session_command_op_remove 3 8 2024-04-14T15:25:59.395 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:25:59.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.395+0000 7fd48b74c700 10 client.48017.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:25:59.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.395+0000 7fd48b74c700 15 client.48017.objecter _session_command_op_assign -1 9 2024-04-14T15:25:59.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.395+0000 7fd48b74c700 20 client.48017.objecter _get_session s=0x7fd468013950 osd=4 3 2024-04-14T15:25:59.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.395+0000 7fd48b74c700 20 client.48017.objecter put_session s=0x7fd468013950 osd=4 3 2024-04-14T15:25:59.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.395+0000 7fd48b74c700 20 client.48017.objecter _get_session s=0x7fd468013950 osd=4 3 2024-04-14T15:25:59.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.395+0000 7fd48b74c700 15 client.48017.objecter _session_command_op_remove -1 9 2024-04-14T15:25:59.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.395+0000 7fd48b74c700 20 client.48017.objecter get_session s=0x7fd468013950 osd=4 3 2024-04-14T15:25:59.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.395+0000 7fd48b74c700 15 client.48017.objecter _session_command_op_assign 4 9 2024-04-14T15:25:59.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.395+0000 7fd48b74c700 20 client.48017.objecter put_session s=0x7fd468013950 osd=4 4 2024-04-14T15:25:59.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.395+0000 7fd48b74c700 10 client.48017.objecter _send_command 9 2024-04-14T15:25:59.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.395+0000 7fd48b74c700 1 -- 172.21.15.90:0/3088153630 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fd46801a7e0 con 0x7fd46800cb20 2024-04-14T15:25:59.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.395+0000 7fd47a7fc700 10 client.48017.objecter ms_handle_connect 0x7fd46800cb20 2024-04-14T15:25:59.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.396+0000 7fd47a7fc700 1 -- 172.21.15.90:0/3088153630 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd46801a7e0 con 0x7fd46800cb20 2024-04-14T15:25:59.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.396+0000 7fd47a7fc700 10 client.48017.objecter ms_dispatch 0x7fd484001550 command_reply(tid 9: 0 ) 2024-04-14T15:25:59.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.396+0000 7fd47a7fc700 10 client.48017.objecter _finish_command 9 = system:0 2024-04-14T15:25:59.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.396+0000 7fd47a7fc700 20 client.48017.objecter put_session s=0x7fd468013950 osd=4 4 2024-04-14T15:25:59.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.396+0000 7fd47a7fc700 15 client.48017.objecter _session_command_op_remove 4 9 2024-04-14T15:25:59.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.412+0000 7fd48b74c700 10 client.48017.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:25:59.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.412+0000 7fd48b74c700 15 client.48017.objecter _session_command_op_assign -1 10 2024-04-14T15:25:59.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.412+0000 7fd48b74c700 20 client.48017.objecter _get_session s=0x7fd468013950 osd=4 3 2024-04-14T15:25:59.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.412+0000 7fd48b74c700 20 client.48017.objecter put_session s=0x7fd468013950 osd=4 3 2024-04-14T15:25:59.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.412+0000 7fd48b74c700 20 client.48017.objecter _get_session s=0x7fd468013950 osd=4 3 2024-04-14T15:25:59.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.412+0000 7fd48b74c700 15 client.48017.objecter _session_command_op_remove -1 10 2024-04-14T15:25:59.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.412+0000 7fd48b74c700 20 client.48017.objecter get_session s=0x7fd468013950 osd=4 3 2024-04-14T15:25:59.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.412+0000 7fd48b74c700 15 client.48017.objecter _session_command_op_assign 4 10 2024-04-14T15:25:59.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.412+0000 7fd48b74c700 20 client.48017.objecter put_session s=0x7fd468013950 osd=4 4 2024-04-14T15:25:59.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.412+0000 7fd48b74c700 10 client.48017.objecter _send_command 10 2024-04-14T15:25:59.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.412+0000 7fd48b74c700 1 -- 172.21.15.90:0/3088153630 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd468011ff0 con 0x7fd46800cb20 2024-04-14T15:25:59.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.414+0000 7fd47a7fc700 1 -- 172.21.15.90:0/3088153630 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd468011ff0 con 0x7fd46800cb20 2024-04-14T15:25:59.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.414+0000 7fd47a7fc700 10 client.48017.objecter ms_dispatch 0x7fd484001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:25:59.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.414+0000 7fd47a7fc700 10 client.48017.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:25:59.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.414+0000 7fd47a7fc700 20 client.48017.objecter put_session s=0x7fd468013950 osd=4 4 2024-04-14T15:25:59.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.414+0000 7fd47a7fc700 15 client.48017.objecter _session_command_op_remove 4 10 2024-04-14T15:25:59.415 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:25:59.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.414+0000 7fd48b74c700 10 client.48017.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:25:59.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.414+0000 7fd48b74c700 15 client.48017.objecter _session_command_op_assign -1 11 2024-04-14T15:25:59.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.414+0000 7fd48b74c700 20 client.48017.objecter _get_session s=0x7fd468011da0 osd=5 3 2024-04-14T15:25:59.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.414+0000 7fd48b74c700 20 client.48017.objecter put_session s=0x7fd468011da0 osd=5 3 2024-04-14T15:25:59.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.414+0000 7fd48b74c700 20 client.48017.objecter _get_session s=0x7fd468011da0 osd=5 3 2024-04-14T15:25:59.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.414+0000 7fd48b74c700 15 client.48017.objecter _session_command_op_remove -1 11 2024-04-14T15:25:59.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.414+0000 7fd48b74c700 20 client.48017.objecter get_session s=0x7fd468011da0 osd=5 3 2024-04-14T15:25:59.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.414+0000 7fd48b74c700 15 client.48017.objecter _session_command_op_assign 5 11 2024-04-14T15:25:59.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.414+0000 7fd48b74c700 20 client.48017.objecter put_session s=0x7fd468011da0 osd=5 4 2024-04-14T15:25:59.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.414+0000 7fd48b74c700 10 client.48017.objecter _send_command 11 2024-04-14T15:25:59.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.414+0000 7fd48b74c700 1 -- 172.21.15.90:0/3088153630 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fd46801d080 con 0x7fd4680134b0 2024-04-14T15:25:59.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.414+0000 7fd47a7fc700 10 client.48017.objecter ms_handle_connect 0x7fd4680134b0 2024-04-14T15:25:59.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.415+0000 7fd47a7fc700 1 -- 172.21.15.90:0/3088153630 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd46801d080 con 0x7fd4680134b0 2024-04-14T15:25:59.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.415+0000 7fd47a7fc700 10 client.48017.objecter ms_dispatch 0x7fd484001550 command_reply(tid 11: 0 ) 2024-04-14T15:25:59.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.415+0000 7fd47a7fc700 10 client.48017.objecter _finish_command 11 = system:0 2024-04-14T15:25:59.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.415+0000 7fd47a7fc700 20 client.48017.objecter put_session s=0x7fd468011da0 osd=5 4 2024-04-14T15:25:59.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.415+0000 7fd47a7fc700 15 client.48017.objecter _session_command_op_remove 5 11 2024-04-14T15:25:59.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.431+0000 7fd48b74c700 10 client.48017.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:25:59.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.431+0000 7fd48b74c700 15 client.48017.objecter _session_command_op_assign -1 12 2024-04-14T15:25:59.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.431+0000 7fd48b74c700 20 client.48017.objecter _get_session s=0x7fd468011da0 osd=5 3 2024-04-14T15:25:59.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.431+0000 7fd48b74c700 20 client.48017.objecter put_session s=0x7fd468011da0 osd=5 3 2024-04-14T15:25:59.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.431+0000 7fd48b74c700 20 client.48017.objecter _get_session s=0x7fd468011da0 osd=5 3 2024-04-14T15:25:59.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.431+0000 7fd48b74c700 15 client.48017.objecter _session_command_op_remove -1 12 2024-04-14T15:25:59.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.431+0000 7fd48b74c700 20 client.48017.objecter get_session s=0x7fd468011da0 osd=5 3 2024-04-14T15:25:59.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.431+0000 7fd48b74c700 15 client.48017.objecter _session_command_op_assign 5 12 2024-04-14T15:25:59.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.431+0000 7fd48b74c700 20 client.48017.objecter put_session s=0x7fd468011da0 osd=5 4 2024-04-14T15:25:59.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.431+0000 7fd48b74c700 10 client.48017.objecter _send_command 12 2024-04-14T15:25:59.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.431+0000 7fd48b74c700 1 -- 172.21.15.90:0/3088153630 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd4680008d0 con 0x7fd4680134b0 2024-04-14T15:25:59.432 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:59 smithi090 ceph-mon[30399]: 5.8 scrub starts 2024-04-14T15:25:59.433 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:59 smithi090 ceph-mon[30399]: 5.8 scrub ok 2024-04-14T15:25:59.433 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:59 smithi090 ceph-mon[30399]: 2.12 scrub starts 2024-04-14T15:25:59.433 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:59 smithi090 ceph-mon[30399]: 2.12 scrub ok 2024-04-14T15:25:59.433 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:59 smithi090 ceph-mon[30399]: 4.3 scrub starts 2024-04-14T15:25:59.433 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:59 smithi090 ceph-mon[30399]: 4.3 scrub ok 2024-04-14T15:25:59.433 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:59 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-142"}]': finished 2024-04-14T15:25:59.433 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:59 smithi090 ceph-mon[30399]: osdmap e809: 8 total, 8 up, 7 in 2024-04-14T15:25:59.433 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:59 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:25:59.433 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:59 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:25:59.433 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:59 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-142"}]: dispatch 2024-04-14T15:25:59.433 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:25:59 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-142"}]: dispatch 2024-04-14T15:25:59.434 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:59 smithi090 ceph-mon[41670]: 5.8 scrub starts 2024-04-14T15:25:59.434 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:59 smithi090 ceph-mon[41670]: 5.8 scrub ok 2024-04-14T15:25:59.434 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:59 smithi090 ceph-mon[41670]: 2.12 scrub starts 2024-04-14T15:25:59.434 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:59 smithi090 ceph-mon[41670]: 2.12 scrub ok 2024-04-14T15:25:59.434 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:59 smithi090 ceph-mon[41670]: 4.3 scrub starts 2024-04-14T15:25:59.434 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:59 smithi090 ceph-mon[41670]: 4.3 scrub ok 2024-04-14T15:25:59.434 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:59 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-142"}]': finished 2024-04-14T15:25:59.434 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:59 smithi090 ceph-mon[41670]: osdmap e809: 8 total, 8 up, 7 in 2024-04-14T15:25:59.434 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:59 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:25:59.434 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:59 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:25:59.434 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:59 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-142"}]: dispatch 2024-04-14T15:25:59.435 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:25:59 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-142"}]: dispatch 2024-04-14T15:25:59.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.433+0000 7fd47a7fc700 1 -- 172.21.15.90:0/3088153630 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd4680008d0 con 0x7fd4680134b0 2024-04-14T15:25:59.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.433+0000 7fd47a7fc700 10 client.48017.objecter ms_dispatch 0x7fd484001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:25:59.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.433+0000 7fd47a7fc700 10 client.48017.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:25:59.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.433+0000 7fd47a7fc700 20 client.48017.objecter put_session s=0x7fd468011da0 osd=5 4 2024-04-14T15:25:59.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.433+0000 7fd47a7fc700 15 client.48017.objecter _session_command_op_remove 5 12 2024-04-14T15:25:59.435 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:25:59.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.433+0000 7fd48b74c700 10 client.48017.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:25:59.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.433+0000 7fd48b74c700 15 client.48017.objecter _session_command_op_assign -1 13 2024-04-14T15:25:59.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.433+0000 7fd48b74c700 20 client.48017.objecter _get_session s=0x7fd46801d140 osd=6 3 2024-04-14T15:25:59.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.433+0000 7fd48b74c700 20 client.48017.objecter put_session s=0x7fd46801d140 osd=6 3 2024-04-14T15:25:59.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.433+0000 7fd48b74c700 20 client.48017.objecter _get_session s=0x7fd46801d140 osd=6 3 2024-04-14T15:25:59.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.433+0000 7fd48b74c700 15 client.48017.objecter _session_command_op_remove -1 13 2024-04-14T15:25:59.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.433+0000 7fd48b74c700 20 client.48017.objecter get_session s=0x7fd46801d140 osd=6 3 2024-04-14T15:25:59.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.433+0000 7fd48b74c700 15 client.48017.objecter _session_command_op_assign 6 13 2024-04-14T15:25:59.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.433+0000 7fd48b74c700 20 client.48017.objecter put_session s=0x7fd46801d140 osd=6 4 2024-04-14T15:25:59.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.433+0000 7fd48b74c700 10 client.48017.objecter _send_command 13 2024-04-14T15:25:59.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.433+0000 7fd48b74c700 1 -- 172.21.15.90:0/3088153630 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fd468023330 con 0x7fd46801f820 2024-04-14T15:25:59.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.434+0000 7fd47a7fc700 10 client.48017.objecter ms_handle_connect 0x7fd46801f820 2024-04-14T15:25:59.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.434+0000 7fd47a7fc700 1 -- 172.21.15.90:0/3088153630 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd468023330 con 0x7fd46801f820 2024-04-14T15:25:59.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.434+0000 7fd47a7fc700 10 client.48017.objecter ms_dispatch 0x7fd484001550 command_reply(tid 13: 0 ) 2024-04-14T15:25:59.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.434+0000 7fd47a7fc700 10 client.48017.objecter _finish_command 13 = system:0 2024-04-14T15:25:59.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.434+0000 7fd47a7fc700 20 client.48017.objecter put_session s=0x7fd46801d140 osd=6 4 2024-04-14T15:25:59.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.434+0000 7fd47a7fc700 15 client.48017.objecter _session_command_op_remove 6 13 2024-04-14T15:25:59.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.451+0000 7fd48b74c700 10 client.48017.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:25:59.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.451+0000 7fd48b74c700 15 client.48017.objecter _session_command_op_assign -1 14 2024-04-14T15:25:59.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.451+0000 7fd48b74c700 20 client.48017.objecter _get_session s=0x7fd46801d140 osd=6 3 2024-04-14T15:25:59.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.451+0000 7fd48b74c700 20 client.48017.objecter put_session s=0x7fd46801d140 osd=6 3 2024-04-14T15:25:59.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.451+0000 7fd48b74c700 20 client.48017.objecter _get_session s=0x7fd46801d140 osd=6 3 2024-04-14T15:25:59.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.451+0000 7fd48b74c700 15 client.48017.objecter _session_command_op_remove -1 14 2024-04-14T15:25:59.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.451+0000 7fd48b74c700 20 client.48017.objecter get_session s=0x7fd46801d140 osd=6 3 2024-04-14T15:25:59.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.451+0000 7fd48b74c700 15 client.48017.objecter _session_command_op_assign 6 14 2024-04-14T15:25:59.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.451+0000 7fd48b74c700 20 client.48017.objecter put_session s=0x7fd46801d140 osd=6 4 2024-04-14T15:25:59.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.451+0000 7fd48b74c700 10 client.48017.objecter _send_command 14 2024-04-14T15:25:59.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.451+0000 7fd48b74c700 1 -- 172.21.15.90:0/3088153630 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd468012a20 con 0x7fd46801f820 2024-04-14T15:25:59.453 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:25:59.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.453+0000 7fd47a7fc700 1 -- 172.21.15.90:0/3088153630 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd468012a20 con 0x7fd46801f820 2024-04-14T15:25:59.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.453+0000 7fd47a7fc700 10 client.48017.objecter ms_dispatch 0x7fd484001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:25:59.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.453+0000 7fd47a7fc700 10 client.48017.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:25:59.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.453+0000 7fd47a7fc700 20 client.48017.objecter put_session s=0x7fd46801d140 osd=6 4 2024-04-14T15:25:59.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.453+0000 7fd47a7fc700 15 client.48017.objecter _session_command_op_remove 6 14 2024-04-14T15:25:59.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.453+0000 7fd48b74c700 10 client.48017.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:25:59.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.453+0000 7fd48b74c700 15 client.48017.objecter _session_command_op_assign -1 15 2024-04-14T15:25:59.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.453+0000 7fd48b74c700 20 client.48017.objecter _get_session s=0x7fd468012a20 osd=7 3 2024-04-14T15:25:59.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.453+0000 7fd48b74c700 20 client.48017.objecter put_session s=0x7fd468012a20 osd=7 3 2024-04-14T15:25:59.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.453+0000 7fd48b74c700 20 client.48017.objecter _get_session s=0x7fd468012a20 osd=7 3 2024-04-14T15:25:59.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.453+0000 7fd48b74c700 15 client.48017.objecter _session_command_op_remove -1 15 2024-04-14T15:25:59.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.453+0000 7fd48b74c700 20 client.48017.objecter get_session s=0x7fd468012a20 osd=7 3 2024-04-14T15:25:59.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.453+0000 7fd48b74c700 15 client.48017.objecter _session_command_op_assign 7 15 2024-04-14T15:25:59.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.453+0000 7fd48b74c700 20 client.48017.objecter put_session s=0x7fd468012a20 osd=7 4 2024-04-14T15:25:59.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.453+0000 7fd48b74c700 10 client.48017.objecter _send_command 15 2024-04-14T15:25:59.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.453+0000 7fd48b74c700 1 -- 172.21.15.90:0/3088153630 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fd46801b570 con 0x7fd468012b90 2024-04-14T15:25:59.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.453+0000 7fd47a7fc700 10 client.48017.objecter ms_handle_connect 0x7fd468012b90 2024-04-14T15:25:59.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.454+0000 7fd47a7fc700 1 -- 172.21.15.90:0/3088153630 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd46801b570 con 0x7fd468012b90 2024-04-14T15:25:59.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.454+0000 7fd47a7fc700 10 client.48017.objecter ms_dispatch 0x7fd484001550 command_reply(tid 15: 0 ) 2024-04-14T15:25:59.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.454+0000 7fd47a7fc700 10 client.48017.objecter _finish_command 15 = system:0 2024-04-14T15:25:59.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.454+0000 7fd47a7fc700 20 client.48017.objecter put_session s=0x7fd468012a20 osd=7 4 2024-04-14T15:25:59.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.454+0000 7fd47a7fc700 15 client.48017.objecter _session_command_op_remove 7 15 2024-04-14T15:25:59.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.470+0000 7fd48b74c700 10 client.48017.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:25:59.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.470+0000 7fd48b74c700 15 client.48017.objecter _session_command_op_assign -1 16 2024-04-14T15:25:59.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.470+0000 7fd48b74c700 20 client.48017.objecter _get_session s=0x7fd468012a20 osd=7 3 2024-04-14T15:25:59.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.470+0000 7fd48b74c700 20 client.48017.objecter put_session s=0x7fd468012a20 osd=7 3 2024-04-14T15:25:59.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.470+0000 7fd48b74c700 20 client.48017.objecter _get_session s=0x7fd468012a20 osd=7 3 2024-04-14T15:25:59.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.470+0000 7fd48b74c700 15 client.48017.objecter _session_command_op_remove -1 16 2024-04-14T15:25:59.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.470+0000 7fd48b74c700 20 client.48017.objecter get_session s=0x7fd468012a20 osd=7 3 2024-04-14T15:25:59.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.470+0000 7fd48b74c700 15 client.48017.objecter _session_command_op_assign 7 16 2024-04-14T15:25:59.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.470+0000 7fd48b74c700 20 client.48017.objecter put_session s=0x7fd468012a20 osd=7 4 2024-04-14T15:25:59.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.470+0000 7fd48b74c700 10 client.48017.objecter _send_command 16 2024-04-14T15:25:59.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.470+0000 7fd48b74c700 1 -- 172.21.15.90:0/3088153630 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd46801b530 con 0x7fd468012b90 2024-04-14T15:25:59.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.472+0000 7fd47a7fc700 1 -- 172.21.15.90:0/3088153630 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd46801b530 con 0x7fd468012b90 2024-04-14T15:25:59.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.472+0000 7fd47a7fc700 10 client.48017.objecter ms_dispatch 0x7fd484001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:25:59.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.472+0000 7fd47a7fc700 10 client.48017.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:25:59.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.472+0000 7fd47a7fc700 20 client.48017.objecter put_session s=0x7fd468012a20 osd=7 4 2024-04-14T15:25:59.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.472+0000 7fd47a7fc700 15 client.48017.objecter _session_command_op_remove 7 16 2024-04-14T15:25:59.473 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:25:59.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.472+0000 7fd48b74c700 10 librados: watch_flush enter 2024-04-14T15:25:59.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.472+0000 7fd48b74c700 10 librados: watch_flush exit 2024-04-14T15:25:59.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.472+0000 7fd48b74c700 10 client.48017.objecter close_session for osd.0 2024-04-14T15:25:59.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.472+0000 7fd48b74c700 1 -- 172.21.15.90:0/3088153630 >> v1:172.21.15.90:6801/3761806885 conn(0x7fd468004730 legacy=0x7fd468002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:59.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.472+0000 7fd48b74c700 20 client.48017.objecter put_session s=0x7fd4680053e0 osd=0 1 2024-04-14T15:25:59.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.472+0000 7fd48b74c700 10 client.48017.objecter close_session for osd.1 2024-04-14T15:25:59.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.472+0000 7fd48b74c700 1 -- 172.21.15.90:0/3088153630 >> v1:172.21.15.90:6805/4170361699 conn(0x7fd4680042a0 legacy=0x7fd46800ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:59.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.472+0000 7fd48b74c700 20 client.48017.objecter put_session s=0x7fd468004130 osd=1 1 2024-04-14T15:25:59.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.472+0000 7fd48b74c700 10 client.48017.objecter close_session for osd.2 2024-04-14T15:25:59.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.472+0000 7fd48b74c700 1 -- 172.21.15.90:0/3088153630 >> v1:172.21.15.90:6809/1686184251 conn(0x7fd468008990 legacy=0x7fd46800f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:59.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.472+0000 7fd48b74c700 20 client.48017.objecter put_session s=0x7fd468001a00 osd=2 1 2024-04-14T15:25:59.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.472+0000 7fd48b74c700 10 client.48017.objecter close_session for osd.3 2024-04-14T15:25:59.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.472+0000 7fd489ce9700 1 -- 172.21.15.90:0/3088153630 reap_dead start 2024-04-14T15:25:59.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.472+0000 7fd48b74c700 1 -- 172.21.15.90:0/3088153630 >> v1:172.21.15.90:6813/2270166774 conn(0x7fd468011190 legacy=0x7fd4680115f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:59.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.472+0000 7fd48b74c700 20 client.48017.objecter put_session s=0x7fd468008200 osd=3 1 2024-04-14T15:25:59.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.472+0000 7fd48b74c700 10 client.48017.objecter close_session for osd.4 2024-04-14T15:25:59.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.472+0000 7fd48b74c700 1 -- 172.21.15.90:0/3088153630 >> v1:172.21.15.92:6800/3887549733 conn(0x7fd46800cb20 legacy=0x7fd4680190c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:59.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.472+0000 7fd48b74c700 20 client.48017.objecter put_session s=0x7fd468013950 osd=4 1 2024-04-14T15:25:59.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.472+0000 7fd48b74c700 10 client.48017.objecter close_session for osd.5 2024-04-14T15:25:59.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.472+0000 7fd48b74c700 1 -- 172.21.15.90:0/3088153630 >> v1:172.21.15.92:6804/3244561440 conn(0x7fd4680134b0 legacy=0x7fd46801b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:59.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.472+0000 7fd48b74c700 20 client.48017.objecter put_session s=0x7fd468011da0 osd=5 1 2024-04-14T15:25:59.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.472+0000 7fd48b74c700 10 client.48017.objecter close_session for osd.6 2024-04-14T15:25:59.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.472+0000 7fd48b74c700 1 -- 172.21.15.90:0/3088153630 >> v1:172.21.15.92:6808/636125963 conn(0x7fd46801f820 legacy=0x7fd468021c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:59.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.472+0000 7fd48b74c700 20 client.48017.objecter put_session s=0x7fd46801d140 osd=6 1 2024-04-14T15:25:59.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.472+0000 7fd48b74c700 10 client.48017.objecter close_session for osd.7 2024-04-14T15:25:59.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.472+0000 7fd48b74c700 1 -- 172.21.15.90:0/3088153630 >> v1:172.21.15.92:6812/1773706716 conn(0x7fd468012b90 legacy=0x7fd4680264c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:59.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.472+0000 7fd48b74c700 20 client.48017.objecter put_session s=0x7fd468012a20 osd=7 1 2024-04-14T15:25:59.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.472+0000 7fd48b74c700 20 client.48017.objecter shutdown clearing up homeless session... 2024-04-14T15:25:59.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.472+0000 7fd48b74c700 10 client.48017.objecter successfully canceled tick 2024-04-14T15:25:59.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.472+0000 7fd489ce9700 1 -- 172.21.15.90:0/3088153630 reap_dead start 2024-04-14T15:25:59.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.472+0000 7fd48b74c700 1 -- 172.21.15.90:0/3088153630 >> v1:172.21.15.90:6800/420935427 conn(0x7fd470071340 legacy=0x7fd470073800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:59.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.472+0000 7fd48b74c700 1 -- 172.21.15.90:0/3088153630 >> v1:172.21.15.90:6790/0 conn(0x7fd48410b8b0 legacy=0x7fd484101e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:25:59.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.472+0000 7fd48b74c700 1 -- 172.21.15.90:0/3088153630 shutdown_connections 2024-04-14T15:25:59.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.472+0000 7fd48b74c700 1 -- 172.21.15.90:0/3088153630 >> 172.21.15.90:0/3088153630 conn(0x7fd4840fd290 msgr2=0x7fd4840ff680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:25:59.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.473+0000 7fd48b74c700 1 -- 172.21.15.90:0/3088153630 shutdown_connections 2024-04-14T15:25:59.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.473+0000 7fd48b74c700 1 -- 172.21.15.90:0/3088153630 wait complete. 2024-04-14T15:25:59.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:25:59.473+0000 7fd48b74c700 1 librados: shutdown 2024-04-14T15:25:59.477 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:25:59.477 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:25:59.477 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:25:59.477 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:25:59.477 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:25:59.477 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:25:59.477 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:25:59.477 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:25:59.477 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:25:59.502 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:59.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:59.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:25:59.604 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:25:59.628 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:59.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:59.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:25:59.730 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:25:59.753 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:25:59.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:59.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:25:59.844 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-04-14T15:25:59.844 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:25:59.855 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:25:59.880 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:25:59.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:25:59.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:25:59.982 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:26:00.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:00.007 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:00.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:26:00.108 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:26:00.131 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:00.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:00.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:26:00.233 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:26:00.256 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:00.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:00.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:26:00.358 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:26:00.383 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:00.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:00.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:26:00.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:00 smithi090 ceph-mon[41670]: 6.11 scrub starts 2024-04-14T15:26:00.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:00 smithi090 ceph-mon[41670]: 6.11 scrub ok 2024-04-14T15:26:00.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:00 smithi090 ceph-mon[41670]: pgmap v1201: 224 pgs: 1 active+remapped+backfill_toofull, 223 active+clean; 4.4 MiB data, 760 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-14T15:26:00.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:00 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3088153630' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:26:00.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:00 smithi090 ceph-mon[41670]: osdmap e810: 8 total, 8 up, 7 in 2024-04-14T15:26:00.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:00 smithi090 ceph-mon[30399]: 6.11 scrub starts 2024-04-14T15:26:00.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:00 smithi090 ceph-mon[30399]: 6.11 scrub ok 2024-04-14T15:26:00.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:00 smithi090 ceph-mon[30399]: pgmap v1201: 224 pgs: 1 active+remapped+backfill_toofull, 223 active+clean; 4.4 MiB data, 760 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-14T15:26:00.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:00 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3088153630' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:26:00.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:00 smithi090 ceph-mon[30399]: osdmap e810: 8 total, 8 up, 7 in 2024-04-14T15:26:00.485 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:26:00.509 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:00.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:00.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:26:00.511 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:00 smithi092 ceph-mon[33966]: 6.11 scrub starts 2024-04-14T15:26:00.511 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:00 smithi092 ceph-mon[33966]: 6.11 scrub ok 2024-04-14T15:26:00.511 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:00 smithi092 ceph-mon[33966]: pgmap v1201: 224 pgs: 1 active+remapped+backfill_toofull, 223 active+clean; 4.4 MiB data, 760 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-14T15:26:00.512 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:00 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3088153630' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:26:00.512 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:00 smithi092 ceph-mon[33966]: osdmap e810: 8 total, 8 up, 7 in 2024-04-14T15:26:00.538 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-14T15:26:00.611 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:26:00.635 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:00.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:00.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:26:00.736 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:26:00.760 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:00.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:00.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:26:00.816 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:26:00.862 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:26:00.886 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:00.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:00.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:26:00.988 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:26:01.011 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:01.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:01.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:26:01.113 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:26:01.123 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:26:01.137 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:01.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:01.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:26:01.239 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:26:01.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:01.264 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:01.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:26:01.299 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:01 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:26:01.299 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:01 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:26:01.299 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:01 smithi092 ceph-mon[33966]: 4.5 scrub starts 2024-04-14T15:26:01.299 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:01 smithi092 ceph-mon[33966]: 4.5 scrub ok 2024-04-14T15:26:01.299 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:01 smithi092 ceph-mon[33966]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:26:01.299 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:01 smithi092 ceph-mon[33966]: osdmap e811: 8 total, 8 up, 7 in 2024-04-14T15:26:01.299 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:01 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:26:01.299 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:01 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:26:01.340 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:01 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:26:01.341 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:01 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:26:01.341 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:01 smithi090 ceph-mon[41670]: 4.5 scrub starts 2024-04-14T15:26:01.341 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:01 smithi090 ceph-mon[41670]: 4.5 scrub ok 2024-04-14T15:26:01.341 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:01 smithi090 ceph-mon[41670]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:26:01.341 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:01 smithi090 ceph-mon[41670]: osdmap e811: 8 total, 8 up, 7 in 2024-04-14T15:26:01.341 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:01 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:26:01.341 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:01 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:26:01.341 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:01 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:26:01.341 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:01 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:26:01.341 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:01 smithi090 ceph-mon[30399]: 4.5 scrub starts 2024-04-14T15:26:01.341 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:01 smithi090 ceph-mon[30399]: 4.5 scrub ok 2024-04-14T15:26:01.341 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:01 smithi090 ceph-mon[30399]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:26:01.342 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:01 smithi090 ceph-mon[30399]: osdmap e811: 8 total, 8 up, 7 in 2024-04-14T15:26:01.342 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:01 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:26:01.342 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:01 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:26:01.365 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:26:01.389 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:01.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:01.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:26:01.490 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:26:01.514 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:01.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:01.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:26:01.616 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:26:01.640 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:01.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:01.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:26:01.742 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:26:01.765 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:01.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:01.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:26:01.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:01.808+0000 7f41a78fb700 1 -- 172.21.15.90:0/3963783269 >> v1:172.21.15.90:6789/0 conn(0x7f41a010b830 legacy=0x7f41a010dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:01.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:01.808+0000 7f41a78fb700 1 -- 172.21.15.90:0/3963783269 shutdown_connections 2024-04-14T15:26:01.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:01.808+0000 7f41a78fb700 1 -- 172.21.15.90:0/3963783269 >> 172.21.15.90:0/3963783269 conn(0x7f41a00fd290 msgr2=0x7f41a00ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:26:01.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:01.808+0000 7f41a78fb700 1 -- 172.21.15.90:0/3963783269 shutdown_connections 2024-04-14T15:26:01.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:01.808+0000 7f41a78fb700 1 -- 172.21.15.90:0/3963783269 wait complete. 2024-04-14T15:26:01.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:01.808+0000 7f41a78fb700 1 librados: starting msgr at 2024-04-14T15:26:01.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:01.808+0000 7f41a78fb700 1 librados: starting objecter 2024-04-14T15:26:01.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:01.808+0000 7f41a78fb700 1 Processor -- start 2024-04-14T15:26:01.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:01.809+0000 7f41a78fb700 1 -- start start 2024-04-14T15:26:01.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:01.809+0000 7f41a78fb700 1 librados: setting wanted keys 2024-04-14T15:26:01.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:01.809+0000 7f41a78fb700 1 librados: calling monclient init 2024-04-14T15:26:01.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:01.809+0000 7f41a78fb700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f41a010ebe0 con 0x7f41a010b830 2024-04-14T15:26:01.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:01.809+0000 7f41a78fb700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f41a01aad80 con 0x7f41a010f340 2024-04-14T15:26:01.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:01.809+0000 7f41a78fb700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f41a01abfc0 con 0x7f41a0103a20 2024-04-14T15:26:01.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:01.809+0000 7f41a5697700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f41a0103a20 0x7f41a010dfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:36582/0 (socket says 172.21.15.90:36582) 2024-04-14T15:26:01.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:01.809+0000 7f41a4e96700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f41a010b830 0x7f41a01a4f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:47046/0 (socket says 172.21.15.90:47046) 2024-04-14T15:26:01.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:01.809+0000 7f41a5697700 1 -- 172.21.15.90:0/703097973 learned_addr learned my addr 172.21.15.90:0/703097973 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:26:01.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:01.809+0000 7f419e7fc700 10 client.?.objecter ms_handle_connect 0x7f41a010b830 2024-04-14T15:26:01.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:01.809+0000 7f419e7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:26:01.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:01.809+0000 7f419e7fc700 10 client.?.objecter ms_handle_connect 0x7f41a0103a20 2024-04-14T15:26:01.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:01.809+0000 7f419e7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:26:01.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:01.809+0000 7f419e7fc700 1 -- 172.21.15.90:0/703097973 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3002055048 0 0) 0x7f41a010ebe0 con 0x7f41a010b830 2024-04-14T15:26:01.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:01.809+0000 7f419e7fc700 1 -- 172.21.15.90:0/703097973 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f41880031f0 con 0x7f41a010b830 2024-04-14T15:26:01.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:01.809+0000 7f419e7fc700 10 client.?.objecter ms_handle_connect 0x7f41a010f340 2024-04-14T15:26:01.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:01.809+0000 7f419e7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:26:01.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:01.809+0000 7f419e7fc700 1 -- 172.21.15.90:0/703097973 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4144406855 0 0) 0x7f41a01abfc0 con 0x7f41a0103a20 2024-04-14T15:26:01.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:01.809+0000 7f419e7fc700 1 -- 172.21.15.90:0/703097973 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f41a010ebe0 con 0x7f41a0103a20 2024-04-14T15:26:01.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:01.809+0000 7f419e7fc700 1 -- 172.21.15.90:0/703097973 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4183869020 0 0) 0x7f41a01aad80 con 0x7f41a010f340 2024-04-14T15:26:01.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:01.809+0000 7f419e7fc700 1 -- 172.21.15.90:0/703097973 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f41a01abfc0 con 0x7f41a010f340 2024-04-14T15:26:01.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:01.809+0000 7f419e7fc700 1 -- 172.21.15.90:0/703097973 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3057387744 0 0) 0x7f41880031f0 con 0x7f41a010b830 2024-04-14T15:26:01.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:01.809+0000 7f419e7fc700 1 -- 172.21.15.90:0/703097973 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f41a01aad80 con 0x7f41a010b830 2024-04-14T15:26:01.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:01.809+0000 7f419e7fc700 1 -- 172.21.15.90:0/703097973 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f4194004590 con 0x7f41a010b830 2024-04-14T15:26:01.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:01.810+0000 7f419e7fc700 1 -- 172.21.15.90:0/703097973 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1916958133 0 0) 0x7f41a01abfc0 con 0x7f41a010f340 2024-04-14T15:26:01.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:01.810+0000 7f419e7fc700 1 -- 172.21.15.90:0/703097973 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f41880031f0 con 0x7f41a010f340 2024-04-14T15:26:01.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:01.810+0000 7f419e7fc700 1 -- 172.21.15.90:0/703097973 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 168617613 0 0) 0x7f41a010ebe0 con 0x7f41a0103a20 2024-04-14T15:26:01.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:01.810+0000 7f419e7fc700 1 -- 172.21.15.90:0/703097973 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f41a01abfc0 con 0x7f41a0103a20 2024-04-14T15:26:01.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:01.810+0000 7f419e7fc700 1 -- 172.21.15.90:0/703097973 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f41900018b0 con 0x7f41a010f340 2024-04-14T15:26:01.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:01.810+0000 7f419e7fc700 1 -- 172.21.15.90:0/703097973 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f4198001df0 con 0x7f41a0103a20 2024-04-14T15:26:01.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:01.810+0000 7f419e7fc700 1 -- 172.21.15.90:0/703097973 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2720647756 0 0) 0x7f41a01aad80 con 0x7f41a010b830 2024-04-14T15:26:01.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:01.810+0000 7f419e7fc700 1 -- 172.21.15.90:0/703097973 >> v1:172.21.15.90:6790/0 conn(0x7f41a0103a20 legacy=0x7f41a010dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:01.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:01.810+0000 7f419e7fc700 1 -- 172.21.15.90:0/703097973 >> v1:172.21.15.92:6789/0 conn(0x7f41a010f340 legacy=0x7f41a01a8650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:01.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:01.810+0000 7f419e7fc700 1 -- 172.21.15.90:0/703097973 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41a01ad200 con 0x7f41a010b830 2024-04-14T15:26:01.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:01.810+0000 7f41a78fb700 1 -- 172.21.15.90:0/703097973 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f41a01ac1f0 con 0x7f41a010b830 2024-04-14T15:26:01.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:01.810+0000 7f41a78fb700 10 client.38073.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:26:01.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:01.810+0000 7f41a78fb700 1 -- 172.21.15.90:0/703097973 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f41a01ac6b0 con 0x7f41a010b830 2024-04-14T15:26:01.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:01.810+0000 7f41a78fb700 1 librados: init done 2024-04-14T15:26:01.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:01.810+0000 7f419e7fc700 1 -- 172.21.15.90:0/703097973 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f4194001af0 con 0x7f41a010b830 2024-04-14T15:26:01.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:01.810+0000 7f419e7fc700 1 -- 172.21.15.90:0/703097973 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f4194004d40 con 0x7f41a010b830 2024-04-14T15:26:01.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:01.811+0000 7f41a78fb700 1 -- 172.21.15.90:0/703097973 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f418c005120 con 0x7f41a010b830 2024-04-14T15:26:01.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:01.811+0000 7f419e7fc700 1 -- 172.21.15.90:0/703097973 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f4194003780 con 0x7f41a010b830 2024-04-14T15:26:01.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:01.812+0000 7f419e7fc700 1 -- 172.21.15.90:0/703097973 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(811..811 src has 467..811) ==== 8749+0+0 (unknown 3845801372 0 0) 0x7f419408ac20 con 0x7f41a010b830 2024-04-14T15:26:01.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:01.812+0000 7f419e7fc700 10 client.38073.objecter ms_dispatch 0x7f41a0001550 osd_map(811..811 src has 467..811) 2024-04-14T15:26:01.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:01.812+0000 7f419e7fc700 3 client.38073.objecter handle_osd_map got epochs [811,811] > 0 2024-04-14T15:26:01.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:01.812+0000 7f419e7fc700 3 client.38073.objecter handle_osd_map decoding full epoch 811 2024-04-14T15:26:01.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:01.812+0000 7f419e7fc700 20 client.38073.objecter dump_active .. 0 homeless 2024-04-14T15:26:01.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:01.812+0000 7f419e7fc700 10 client.38073.objecter ms_handle_connect 0x7f4188071360 2024-04-14T15:26:01.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:01.817+0000 7f419e7fc700 1 -- 172.21.15.90:0/703097973 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f4194057400 con 0x7f41a010b830 2024-04-14T15:26:01.867 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:26:01.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:01.891 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:01.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:26:01.992 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:26:02.015 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:02.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:02.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:26:02.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:02.021+0000 7f41a78fb700 1 -- 172.21.15.90:0/703097973 --> v1:172.21.15.90:6800/420935427 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f418c000b90 con 0x7f4188071360 2024-04-14T15:26:02.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:02.049+0000 7f419e7fc700 1 -- 172.21.15.90:0/703097973 <== mgr.14610 v1:172.21.15.90:6800/420935427 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+572329 (unknown 2965378022 0 1909278602) 0x7f418c000b90 con 0x7f4188071360 2024-04-14T15:26:02.058 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:26:02.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:02.056+0000 7f41a78fb700 10 librados: watch_flush enter 2024-04-14T15:26:02.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:02.056+0000 7f41a78fb700 10 librados: watch_flush exit 2024-04-14T15:26:02.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:02.056+0000 7f41a78fb700 20 client.38073.objecter shutdown clearing up homeless session... 2024-04-14T15:26:02.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:02.056+0000 7f41a78fb700 10 client.38073.objecter successfully canceled tick 2024-04-14T15:26:02.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:02.056+0000 7f41a78fb700 1 -- 172.21.15.90:0/703097973 >> v1:172.21.15.90:6800/420935427 conn(0x7f4188071360 legacy=0x7f4188073820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:02.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:02.056+0000 7f41a78fb700 1 -- 172.21.15.90:0/703097973 >> v1:172.21.15.90:6789/0 conn(0x7f41a010b830 legacy=0x7f41a01a4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:02.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:02.056+0000 7f41a78fb700 1 -- 172.21.15.90:0/703097973 shutdown_connections 2024-04-14T15:26:02.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:02.056+0000 7f41a78fb700 1 -- 172.21.15.90:0/703097973 >> 172.21.15.90:0/703097973 conn(0x7f41a00fd290 msgr2=0x7f41a01117d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:26:02.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:02.056+0000 7f41a78fb700 1 -- 172.21.15.90:0/703097973 shutdown_connections 2024-04-14T15:26:02.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:02.056+0000 7f41a78fb700 1 -- 172.21.15.90:0/703097973 wait complete. 2024-04-14T15:26:02.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:02.056+0000 7f41a78fb700 1 librados: shutdown 2024-04-14T15:26:02.064 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-14T15:26:02.117 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:26:02.141 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:02.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:02.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:26:02.243 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:26:02.266 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:02.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:02.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:26:02.347 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:02 smithi092 ceph-mon[33966]: 4.1c scrub starts 2024-04-14T15:26:02.347 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:02 smithi092 ceph-mon[33966]: 4.1c scrub ok 2024-04-14T15:26:02.348 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:02 smithi092 ceph-mon[33966]: 5.4 scrub starts 2024-04-14T15:26:02.348 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:02 smithi092 ceph-mon[33966]: 5.4 scrub ok 2024-04-14T15:26:02.349 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:02 smithi092 ceph-mon[33966]: 11.c scrub starts 2024-04-14T15:26:02.349 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:02 smithi092 ceph-mon[33966]: 11.c scrub ok 2024-04-14T15:26:02.349 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:02 smithi092 ceph-mon[33966]: pgmap v1204: 224 pgs: 32 unknown, 1 active+remapped+backfill_toofull, 191 active+clean; 4.4 MiB data, 760 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-14T15:26:02.349 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:02 smithi092 ceph-mon[33966]: from='client.38073 v1:172.21.15.90:0/703097973' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:26:02.349 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:02 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:26:02.350 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:02 smithi092 ceph-mon[33966]: osdmap e812: 8 total, 8 up, 7 in 2024-04-14T15:26:02.350 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:02 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:26:02.350 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:02 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:26:02.368 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:26:02.392 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:02.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:02.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:26:02.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:02 smithi090 ceph-mon[30399]: 4.1c scrub starts 2024-04-14T15:26:02.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:02 smithi090 ceph-mon[30399]: 4.1c scrub ok 2024-04-14T15:26:02.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:02 smithi090 ceph-mon[30399]: 5.4 scrub starts 2024-04-14T15:26:02.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:02 smithi090 ceph-mon[30399]: 5.4 scrub ok 2024-04-14T15:26:02.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:02 smithi090 ceph-mon[30399]: 11.c scrub starts 2024-04-14T15:26:02.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:02 smithi090 ceph-mon[30399]: 11.c scrub ok 2024-04-14T15:26:02.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:02 smithi090 ceph-mon[30399]: pgmap v1204: 224 pgs: 32 unknown, 1 active+remapped+backfill_toofull, 191 active+clean; 4.4 MiB data, 760 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-14T15:26:02.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:02 smithi090 ceph-mon[30399]: from='client.38073 v1:172.21.15.90:0/703097973' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:26:02.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:02 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:26:02.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:02 smithi090 ceph-mon[30399]: osdmap e812: 8 total, 8 up, 7 in 2024-04-14T15:26:02.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:02 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:26:02.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:02 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:26:02.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:02 smithi090 ceph-mon[41670]: 4.1c scrub starts 2024-04-14T15:26:02.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:02 smithi090 ceph-mon[41670]: 4.1c scrub ok 2024-04-14T15:26:02.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:02 smithi090 ceph-mon[41670]: 5.4 scrub starts 2024-04-14T15:26:02.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:02 smithi090 ceph-mon[41670]: 5.4 scrub ok 2024-04-14T15:26:02.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:02 smithi090 ceph-mon[41670]: 11.c scrub starts 2024-04-14T15:26:02.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:02 smithi090 ceph-mon[41670]: 11.c scrub ok 2024-04-14T15:26:02.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:02 smithi090 ceph-mon[41670]: pgmap v1204: 224 pgs: 32 unknown, 1 active+remapped+backfill_toofull, 191 active+clean; 4.4 MiB data, 760 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-14T15:26:02.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:02 smithi090 ceph-mon[41670]: from='client.38073 v1:172.21.15.90:0/703097973' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:26:02.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:02 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:26:02.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:02 smithi090 ceph-mon[41670]: osdmap e812: 8 total, 8 up, 7 in 2024-04-14T15:26:02.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:02 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:26:02.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:02 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:26:02.494 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:26:02.518 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:02.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:02.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:26:02.599 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-04-14T15:26:02.599 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:26:02.619 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:26:02.643 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:02.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:02.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:26:02.745 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:26:02.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:02.770 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:02.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:26:02.871 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:26:02.895 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:02.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:02.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:26:02.997 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:26:03.013 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":1204,"stamp":"2024-04-14T15:26:01.201413+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4610064,"num_objects":236,"num_object_clones":1,"num_object_copies":708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":236,"num_whiteouts":0,"num_read":2142,"num_read_kb":1883,"num_write":790,"num_write_kb":4066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":124,"num_bytes_recovered":306051,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1005,"ondisk_log_size":1005,"up":576,"acting":576,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":674,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":778460,"kb_used_data":53488,"kb_used_omap":10,"kb_used_meta":724853,"kb_avail":655437604,"statfs":{"total":671965249536,"available":671168106496,"internally_reserved":0,"allocated":54771712,"data_stored":31246843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11156,"internal_metadata":742249580},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":7,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001578"},"pg_stats":[{"pgid":"340.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:26:01.020810+0000","last_change":"2024-04-14T15:26:01.020810+0000","last_active":"2024-04-14T15:26:01.020810+0000","last_peered":"2024-04-14T15:26:01.020810+0000","last_clean":"2024-04-14T15:26:01.020810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:26:01.020810+0000","last_undegraded":"2024-04-14T15:26:01.020810+0000","last_fullsized":"2024-04-14T15:26:01.020810+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.020810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"340.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:26:01.020810+0000","last_change":"2024-04-14T15:26:01.020810+0000","last_active":"2024-04-14T15:26:01.020810+0000","last_peered":"2024-04-14T15:26:01.020810+0000","last_clean":"2024-04-14T15:26:01.020810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:26:01.020810+0000","last_undegraded":"2024-04-14T15:26:01.020810+0000","last_fullsized":"2024-04-14T15:26:01.020810+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.020810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"340.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:26:01.020810+0000","last_change":"2024-04-14T15:26:01.020810+0000","last_active":"2024-04-14T15:26:01.020810+0000","last_peered":"2024-04-14T15:26:01.020810+0000","last_clean":"2024-04-14T15:26:01.020810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:26:01.020810+0000","last_undegraded":"2024-04-14T15:26:01.020810+0000","last_fullsized":"2024-04-14T15:26:01.020810+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.020810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"340.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:26:01.020810+0000","last_change":"2024-04-14T15:26:01.020810+0000","last_active":"2024-04-14T15:26:01.020810+0000","last_peered":"2024-04-14T15:26:01.020810+0000","last_clean":"2024-04-14T15:26:01.020810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:26:01.020810+0000","last_undegraded":"2024-04-14T15:26:01.020810+0000","last_fullsized":"2024-04-14T15:26:01.020810+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.020810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"340.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:26:01.020810+0000","last_change":"2024-04-14T15:26:01.020810+0000","last_active":"2024-04-14T15:26:01.020810+0000","last_peered":"2024-04-14T15:26:01.020810+0000","last_clean":"2024-04-14T15:26:01.020810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:26:01.020810+0000","last_undegraded":"2024-04-14T15:26:01.020810+0000","last_fullsized":"2024-04-14T15:26:01.020810+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.020810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"340.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:26:01.020810+0000","last_change":"2024-04-14T15:26:01.020810+0000","last_active":"2024-04-14T15:26:01.020810+0000","last_peered":"2024-04-14T15:26:01.020810+0000","last_clean":"2024-04-14T15:26:01.020810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:26:01.020810+0000","last_undegraded":"2024-04-14T15:26:01.020810+0000","last_fullsized":"2024-04-14T15:26:01.020810+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.020810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"340.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:26:01.020810+0000","last_change":"2024-04-14T15:26:01.020810+0000","last_active":"2024-04-14T15:26:01.020810+0000","last_peered":"2024-04-14T15:26:01.020810+0000","last_clean":"2024-04-14T15:26:01.020810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:26:01.020810+0000","last_undegraded":"2024-04-14T15:26:01.020810+0000","last_fullsized":"2024-04-14T15:26:01.020810+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.020810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"340.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:26:01.020810+0000","last_change":"2024-04-14T15:26:01.020810+0000","last_active":"2024-04-14T15:26:01.020810+0000","last_peered":"2024-04-14T15:26:01.020810+0000","last_clean":"2024-04-14T15:26:01.020810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:26:01.020810+0000","last_undegraded":"2024-04-14T15:26:01.020810+0000","last_fullsized":"2024-04-14T15:26:01.020810+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.020810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"340.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:26:01.020810+0000","last_change":"2024-04-14T15:26:01.020810+0000","last_active":"2024-04-14T15:26:01.020810+0000","last_peered":"2024-04-14T15:26:01.020810+0000","last_clean":"2024-04-14T15:26:01.020810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:26:01.020810+0000","last_undegraded":"2024-04-14T15:26:01.020810+0000","last_fullsized":"2024-04-14T15:26:01.020810+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.020810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"340.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:26:01.020810+0000","last_change":"2024-04-14T15:26:01.020810+0000","last_active":"2024-04-14T15:26:01.020810+0000","last_peered":"2024-04-14T15:26:01.020810+0000","last_clean":"2024-04-14T15:26:01.020810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:26:01.020810+0000","last_undegraded":"2024-04-14T15:26:01.020810+0000","last_fullsized":"2024-04-14T15:26:01.020810+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.020810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"340.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:26:01.020810+0000","last_change":"2024-04-14T15:26:01.020810+0000","last_active":"2024-04-14T15:26:01.020810+0000","last_peered":"2024-04-14T15:26:01.020810+0000","last_clean":"2024-04-14T15:26:01.020810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:26:01.020810+0000","last_undegraded":"2024-04-14T15:26:01.020810+0000","last_fullsized":"2024-04-14T15:26:01.020810+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.020810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"340.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:26:01.020810+0000","last_change":"2024-04-14T15:26:01.020810+0000","last_active":"2024-04-14T15:26:01.020810+0000","last_peered":"2024-04-14T15:26:01.020810+0000","last_clean":"2024-04-14T15:26:01.020810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:26:01.020810+0000","last_undegraded":"2024-04-14T15:26:01.020810+0000","last_fullsized":"2024-04-14T15:26:01.020810+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.020810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"340.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:26:01.020810+0000","last_change":"2024-04-14T15:26:01.020810+0000","last_active":"2024-04-14T15:26:01.020810+0000","last_peered":"2024-04-14T15:26:01.020810+0000","last_clean":"2024-04-14T15:26:01.020810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:26:01.020810+0000","last_undegraded":"2024-04-14T15:26:01.020810+0000","last_fullsized":"2024-04-14T15:26:01.020810+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.020810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"340.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:26:01.020810+0000","last_change":"2024-04-14T15:26:01.020810+0000","last_active":"2024-04-14T15:26:01.020810+0000","last_peered":"2024-04-14T15:26:01.020810+0000","last_clean":"2024-04-14T15:26:01.020810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:26:01.020810+0000","last_undegraded":"2024-04-14T15:26:01.020810+0000","last_fullsized":"2024-04-14T15:26:01.020810+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.020810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"340.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:26:01.020810+0000","last_change":"2024-04-14T15:26:01.020810+0000","last_active":"2024-04-14T15:26:01.020810+0000","last_peered":"2024-04-14T15:26:01.020810+0000","last_clean":"2024-04-14T15:26:01.020810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:26:01.020810+0000","last_undegraded":"2024-04-14T15:26:01.020810+0000","last_fullsized":"2024-04-14T15:26:01.020810+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.020810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"340.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:26:01.020810+0000","last_change":"2024-04-14T15:26:01.020810+0000","last_active":"2024-04-14T15:26:01.020810+0000","last_peered":"2024-04-14T15:26:01.020810+0000","last_clean":"2024-04-14T15:26:01.020810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:26:01.020810+0000","last_undegraded":"2024-04-14T15:26:01.020810+0000","last_fullsized":"2024-04-14T15:26:01.020810+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.020810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"340.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:26:01.020810+0000","last_change":"2024-04-14T15:26:01.020810+0000","last_active":"2024-04-14T15:26:01.020810+0000","last_peered":"2024-04-14T15:26:01.020810+0000","last_clean":"2024-04-14T15:26:01.020810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:26:01.020810+0000","last_undegraded":"2024-04-14T15:26:01.020810+0000","last_fullsized":"2024-04-14T15:26:01.020810+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.020810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"340.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:26:01.020810+0000","last_change":"2024-04-14T15:26:01.020810+0000","last_active":"2024-04-14T15:26:01.020810+0000","last_peered":"2024-04-14T15:26:01.020810+0000","last_clean":"2024-04-14T15:26:01.020810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:26:01.020810+0000","last_undegraded":"2024-04-14T15:26:01.020810+0000","last_fullsized":"2024-04-14T15:26:01.020810+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.020810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"340.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:26:01.020810+0000","last_change":"2024-04-14T15:26:01.020810+0000","last_active":"2024-04-14T15:26:01.020810+0000","last_peered":"2024-04-14T15:26:01.020810+0000","last_clean":"2024-04-14T15:26:01.020810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:26:01.020810+0000","last_undegraded":"2024-04-14T15:26:01.020810+0000","last_fullsized":"2024-04-14T15:26:01.020810+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.020810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"340.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:26:01.020810+0000","last_change":"2024-04-14T15:26:01.020810+0000","last_active":"2024-04-14T15:26:01.020810+0000","last_peered":"2024-04-14T15:26:01.020810+0000","last_clean":"2024-04-14T15:26:01.020810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:26:01.020810+0000","last_undegraded":"2024-04-14T15:26:01.020810+0000","last_fullsized":"2024-04-14T15:26:01.020810+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.020810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"340.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:26:01.020810+0000","last_change":"2024-04-14T15:26:01.020810+0000","last_active":"2024-04-14T15:26:01.020810+0000","last_peered":"2024-04-14T15:26:01.020810+0000","last_clean":"2024-04-14T15:26:01.020810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:26:01.020810+0000","last_undegraded":"2024-04-14T15:26:01.020810+0000","last_fullsized":"2024-04-14T15:26:01.020810+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.020810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"340.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:26:01.020810+0000","last_change":"2024-04-14T15:26:01.020810+0000","last_active":"2024-04-14T15:26:01.020810+0000","last_peered":"2024-04-14T15:26:01.020810+0000","last_clean":"2024-04-14T15:26:01.020810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:26:01.020810+0000","last_undegraded":"2024-04-14T15:26:01.020810+0000","last_fullsized":"2024-04-14T15:26:01.020810+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.020810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"340.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:26:01.020810+0000","last_change":"2024-04-14T15:26:01.020810+0000","last_active":"2024-04-14T15:26:01.020810+0000","last_peered":"2024-04-14T15:26:01.020810+0000","last_clean":"2024-04-14T15:26:01.020810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:26:01.020810+0000","last_undegraded":"2024-04-14T15:26:01.020810+0000","last_fullsized":"2024-04-14T15:26:01.020810+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.020810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"340.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:26:01.020810+0000","last_change":"2024-04-14T15:26:01.020810+0000","last_active":"2024-04-14T15:26:01.020810+0000","last_peered":"2024-04-14T15:26:01.020810+0000","last_clean":"2024-04-14T15:26:01.020810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:26:01.020810+0000","last_undegraded":"2024-04-14T15:26:01.020810+0000","last_fullsized":"2024-04-14T15:26:01.020810+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.020810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"340.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:26:01.020810+0000","last_change":"2024-04-14T15:26:01.020810+0000","last_active":"2024-04-14T15:26:01.020810+0000","last_peered":"2024-04-14T15:26:01.020810+0000","last_clean":"2024-04-14T15:26:01.020810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:26:01.020810+0000","last_undegraded":"2024-04-14T15:26:01.020810+0000","last_fullsized":"2024-04-14T15:26:01.020810+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.020810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"323.7","version":"798'3","reported_seq":427,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.046632+0000","last_change":"2024-04-14T15:25:11.799519+0000","last_active":"2024-04-14T15:25:59.046632+0000","last_peered":"2024-04-14T15:25:59.046632+0000","last_clean":"2024-04-14T15:25:59.046632+0000","last_became_active":"2024-04-14T15:23:04.005793+0000","last_became_peered":"2024-04-14T15:23:04.005793+0000","last_unstale":"2024-04-14T15:25:59.046632+0000","last_undegraded":"2024-04-14T15:25:59.046632+0000","last_fullsized":"2024-04-14T15:25:59.046632+0000","mapping_epoch":647,"log_start":"0'0","ondisk_log_start":"0'0","created":647,"last_epoch_clean":648,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:11.799436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_clean_scrub_stamp":"2024-04-14T15:25:11.799436+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:38.480870+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00079989799999999995,"stat_sum":{"num_bytes":24,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"340.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:26:01.020810+0000","last_change":"2024-04-14T15:26:01.020810+0000","last_active":"2024-04-14T15:26:01.020810+0000","last_peered":"2024-04-14T15:26:01.020810+0000","last_clean":"2024-04-14T15:26:01.020810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:26:01.020810+0000","last_undegraded":"2024-04-14T15:26:01.020810+0000","last_fullsized":"2024-04-14T15:26:01.020810+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.020810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"323.6","version":"798'4","reported_seq":439,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.030981+0000","last_change":"2024-04-14T15:25:49.959163+0000","last_active":"2024-04-14T15:25:59.030981+0000","last_peered":"2024-04-14T15:25:59.030981+0000","last_clean":"2024-04-14T15:25:59.030981+0000","last_became_active":"2024-04-14T15:23:04.006098+0000","last_became_peered":"2024-04-14T15:23:04.006098+0000","last_unstale":"2024-04-14T15:25:59.030981+0000","last_undegraded":"2024-04-14T15:25:59.030981+0000","last_fullsized":"2024-04-14T15:25:59.030981+0000","mapping_epoch":647,"log_start":"0'0","ondisk_log_start":"0'0","created":647,"last_epoch_clean":648,"parent":"0.0","parent_split_bits":0,"last_scrub":"798'4","last_scrub_stamp":"2024-04-14T15:25:49.959068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_clean_scrub_stamp":"2024-04-14T15:25:49.959068+0000","objects_scrubbed":4,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:04.142971+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0040397810000000001,"stat_sum":{"num_bytes":32,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"340.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:26:01.020810+0000","last_change":"2024-04-14T15:26:01.020810+0000","last_active":"2024-04-14T15:26:01.020810+0000","last_peered":"2024-04-14T15:26:01.020810+0000","last_clean":"2024-04-14T15:26:01.020810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:26:01.020810+0000","last_undegraded":"2024-04-14T15:26:01.020810+0000","last_fullsized":"2024-04-14T15:26:01.020810+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.020810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"323.5","version":"798'17","reported_seq":444,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.027899+0000","last_change":"2024-04-14T15:25:46.249717+0000","last_active":"2024-04-14T15:25:59.027899+0000","last_peered":"2024-04-14T15:25:59.027899+0000","last_clean":"2024-04-14T15:25:59.027899+0000","last_became_active":"2024-04-14T15:23:04.002370+0000","last_became_peered":"2024-04-14T15:23:04.002370+0000","last_unstale":"2024-04-14T15:25:59.027899+0000","last_undegraded":"2024-04-14T15:25:59.027899+0000","last_fullsized":"2024-04-14T15:25:59.027899+0000","mapping_epoch":647,"log_start":"0'0","ondisk_log_start":"0'0","created":647,"last_epoch_clean":648,"parent":"0.0","parent_split_bits":0,"last_scrub":"733'11","last_scrub_stamp":"2024-04-14T15:25:46.249607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_clean_scrub_stamp":"2024-04-14T15:25:46.249607+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:10.606236+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.0032579309999999999,"stat_sum":{"num_bytes":53,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"340.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:26:01.020810+0000","last_change":"2024-04-14T15:26:01.020810+0000","last_active":"2024-04-14T15:26:01.020810+0000","last_peered":"2024-04-14T15:26:01.020810+0000","last_clean":"2024-04-14T15:26:01.020810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:26:01.020810+0000","last_undegraded":"2024-04-14T15:26:01.020810+0000","last_fullsized":"2024-04-14T15:26:01.020810+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.020810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"323.4","version":"798'15","reported_seq":446,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.033971+0000","last_change":"2024-04-14T15:25:29.009921+0000","last_active":"2024-04-14T15:25:59.033971+0000","last_peered":"2024-04-14T15:25:59.033971+0000","last_clean":"2024-04-14T15:25:59.033971+0000","last_became_active":"2024-04-14T15:23:03.997912+0000","last_became_peered":"2024-04-14T15:23:03.997912+0000","last_unstale":"2024-04-14T15:25:59.033971+0000","last_undegraded":"2024-04-14T15:25:59.033971+0000","last_fullsized":"2024-04-14T15:25:59.033971+0000","mapping_epoch":647,"log_start":"0'0","ondisk_log_start":"0'0","created":647,"last_epoch_clean":648,"parent":"0.0","parent_split_bits":0,"last_scrub":"742'12","last_scrub_stamp":"2024-04-14T15:25:29.009822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_clean_scrub_stamp":"2024-04-14T15:25:29.009822+0000","objects_scrubbed":2,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:56.136530+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001097995,"stat_sum":{"num_bytes":34,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":9,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"340.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:26:01.020810+0000","last_change":"2024-04-14T15:26:01.020810+0000","last_active":"2024-04-14T15:26:01.020810+0000","last_peered":"2024-04-14T15:26:01.020810+0000","last_clean":"2024-04-14T15:26:01.020810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:26:01.020810+0000","last_undegraded":"2024-04-14T15:26:01.020810+0000","last_fullsized":"2024-04-14T15:26:01.020810+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.020810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"323.3","version":"798'19","reported_seq":445,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.046286+0000","last_change":"2024-04-14T15:25:19.807060+0000","last_active":"2024-04-14T15:25:59.046286+0000","last_peered":"2024-04-14T15:25:59.046286+0000","last_clean":"2024-04-14T15:25:59.046286+0000","last_became_active":"2024-04-14T15:23:04.001213+0000","last_became_peered":"2024-04-14T15:23:04.001213+0000","last_unstale":"2024-04-14T15:25:59.046286+0000","last_undegraded":"2024-04-14T15:25:59.046286+0000","last_fullsized":"2024-04-14T15:25:59.046286+0000","mapping_epoch":647,"log_start":"0'0","ondisk_log_start":"0'0","created":647,"last_epoch_clean":648,"parent":"0.0","parent_split_bits":0,"last_scrub":"763'18","last_scrub_stamp":"2024-04-14T15:25:19.806953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_clean_scrub_stamp":"2024-04-14T15:25:19.806953+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:26.152851+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00085465599999999997,"stat_sum":{"num_bytes":37,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":13,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"340.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:26:01.020810+0000","last_change":"2024-04-14T15:26:01.020810+0000","last_active":"2024-04-14T15:26:01.020810+0000","last_peered":"2024-04-14T15:26:01.020810+0000","last_clean":"2024-04-14T15:26:01.020810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:26:01.020810+0000","last_undegraded":"2024-04-14T15:26:01.020810+0000","last_fullsized":"2024-04-14T15:26:01.020810+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.020810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"323.2","version":"798'19","reported_seq":451,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.028582+0000","last_change":"2024-04-14T15:25:01.027310+0000","last_active":"2024-04-14T15:25:59.028582+0000","last_peered":"2024-04-14T15:25:59.028582+0000","last_clean":"2024-04-14T15:25:59.028582+0000","last_became_active":"2024-04-14T15:23:04.001557+0000","last_became_peered":"2024-04-14T15:23:04.001557+0000","last_unstale":"2024-04-14T15:25:59.028582+0000","last_undegraded":"2024-04-14T15:25:59.028582+0000","last_fullsized":"2024-04-14T15:25:59.028582+0000","mapping_epoch":647,"log_start":"0'0","ondisk_log_start":"0'0","created":647,"last_epoch_clean":648,"parent":"0.0","parent_split_bits":0,"last_scrub":"710'6","last_scrub_stamp":"2024-04-14T15:24:15.331688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_clean_scrub_stamp":"2024-04-14T15:24:15.331688+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.013115428,"stat_sum":{"num_bytes":20,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"340.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:26:01.020810+0000","last_change":"2024-04-14T15:26:01.020810+0000","last_active":"2024-04-14T15:26:01.020810+0000","last_peered":"2024-04-14T15:26:01.020810+0000","last_clean":"2024-04-14T15:26:01.020810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:26:01.020810+0000","last_undegraded":"2024-04-14T15:26:01.020810+0000","last_fullsized":"2024-04-14T15:26:01.020810+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.020810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"323.1","version":"798'5","reported_seq":435,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.037379+0000","last_change":"2024-04-14T15:25:54.933798+0000","last_active":"2024-04-14T15:25:59.037379+0000","last_peered":"2024-04-14T15:25:59.037379+0000","last_clean":"2024-04-14T15:25:59.037379+0000","last_became_active":"2024-04-14T15:23:03.998498+0000","last_became_peered":"2024-04-14T15:23:03.998498+0000","last_unstale":"2024-04-14T15:25:59.037379+0000","last_undegraded":"2024-04-14T15:25:59.037379+0000","last_fullsized":"2024-04-14T15:25:59.037379+0000","mapping_epoch":647,"log_start":"0'0","ondisk_log_start":"0'0","created":647,"last_epoch_clean":648,"parent":"0.0","parent_split_bits":0,"last_scrub":"798'5","last_scrub_stamp":"2024-04-14T15:25:54.933693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_clean_scrub_stamp":"2024-04-14T15:25:54.933693+0000","objects_scrubbed":4,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:01.716886+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001861021,"stat_sum":{"num_bytes":28,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"340.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:26:01.020810+0000","last_change":"2024-04-14T15:26:01.020810+0000","last_active":"2024-04-14T15:26:01.020810+0000","last_peered":"2024-04-14T15:26:01.020810+0000","last_clean":"2024-04-14T15:26:01.020810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:26:01.020810+0000","last_undegraded":"2024-04-14T15:26:01.020810+0000","last_fullsized":"2024-04-14T15:26:01.020810+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.020810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"323.0","version":"798'8","reported_seq":439,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.027446+0000","last_change":"2024-04-14T15:25:04.467370+0000","last_active":"2024-04-14T15:25:59.027446+0000","last_peered":"2024-04-14T15:25:59.027446+0000","last_clean":"2024-04-14T15:25:59.027446+0000","last_became_active":"2024-04-14T15:23:04.001020+0000","last_became_peered":"2024-04-14T15:23:04.001020+0000","last_unstale":"2024-04-14T15:25:59.027446+0000","last_undegraded":"2024-04-14T15:25:59.027446+0000","last_fullsized":"2024-04-14T15:25:59.027446+0000","mapping_epoch":647,"log_start":"0'0","ondisk_log_start":"0'0","created":647,"last_epoch_clean":648,"parent":"0.0","parent_split_bits":0,"last_scrub":"704'6","last_scrub_stamp":"2024-04-14T15:25:04.467266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_clean_scrub_stamp":"2024-04-14T15:25:04.467266+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:16.302185+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0014140909999999999,"stat_sum":{"num_bytes":4000021,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":3910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"91'130","reported_seq":452,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.038359+0000","last_change":"2024-04-14T15:25:02.924588+0000","last_active":"2024-04-14T15:25:59.038359+0000","last_peered":"2024-04-14T15:25:59.038359+0000","last_clean":"2024-04-14T15:25:59.038359+0000","last_became_active":"2024-04-14T15:25:02.025875+0000","last_became_peered":"2024-04-14T15:25:02.025875+0000","last_unstale":"2024-04-14T15:25:59.038359+0000","last_undegraded":"2024-04-14T15:25:59.038359+0000","last_fullsized":"2024-04-14T15:25:59.038359+0000","mapping_epoch":757,"log_start":"91'130","ondisk_log_start":"91'130","created":752,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:25:02.924460+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:25:02.924460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:10.071683+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":1096,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.031322+0000","last_change":"2024-04-14T15:23:27.292731+0000","last_active":"2024-04-14T15:25:59.031322+0000","last_peered":"2024-04-14T15:25:59.031322+0000","last_clean":"2024-04-14T15:25:59.031322+0000","last_became_active":"2024-04-14T15:15:44.882941+0000","last_became_peered":"2024-04-14T15:15:44.882941+0000","last_unstale":"2024-04-14T15:25:59.031322+0000","last_undegraded":"2024-04-14T15:25:59.031322+0000","last_fullsized":"2024-04-14T15:25:59.031322+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:27.292590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:23:27.292590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.9","version":"528'10","reported_seq":1757,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.037824+0000","last_change":"2024-04-14T15:24:37.737273+0000","last_active":"2024-04-14T15:25:59.037824+0000","last_peered":"2024-04-14T15:25:59.037824+0000","last_clean":"2024-04-14T15:25:59.037824+0000","last_became_active":"2024-04-14T15:20:52.984943+0000","last_became_peered":"2024-04-14T15:20:52.984943+0000","last_unstale":"2024-04-14T15:25:59.037824+0000","last_undegraded":"2024-04-14T15:25:59.037824+0000","last_fullsized":"2024-04-14T15:25:59.037824+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"528'10","last_scrub_stamp":"2024-04-14T15:24:37.737147+0000","last_deep_scrub":"85'9","last_deep_scrub_stamp":"2024-04-14T15:10:51.950580+0000","last_clean_scrub_stamp":"2024-04-14T15:24:37.737147+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"479'34","reported_seq":1191,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.037384+0000","last_change":"2024-04-14T15:25:26.852970+0000","last_active":"2024-04-14T15:25:59.037384+0000","last_peered":"2024-04-14T15:25:59.037384+0000","last_clean":"2024-04-14T15:25:59.037384+0000","last_became_active":"2024-04-14T15:20:52.979204+0000","last_became_peered":"2024-04-14T15:20:52.979204+0000","last_unstale":"2024-04-14T15:25:59.037384+0000","last_undegraded":"2024-04-14T15:25:59.037384+0000","last_fullsized":"2024-04-14T15:25:59.037384+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'34","last_scrub_stamp":"2024-04-14T15:25:26.852851+0000","last_deep_scrub":"430'22","last_deep_scrub_stamp":"2024-04-14T15:18:39.704883+0000","last_clean_scrub_stamp":"2024-04-14T15:25:26.852851+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:34.262841+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":1555,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.030764+0000","last_change":"2024-04-14T15:25:51.674598+0000","last_active":"2024-04-14T15:25:59.030764+0000","last_peered":"2024-04-14T15:25:59.030764+0000","last_clean":"2024-04-14T15:25:59.030764+0000","last_became_active":"2024-04-14T15:09:43.021460+0000","last_became_peered":"2024-04-14T15:09:43.021460+0000","last_unstale":"2024-04-14T15:25:59.030764+0000","last_undegraded":"2024-04-14T15:25:59.030764+0000","last_fullsized":"2024-04-14T15:25:59.030764+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:51.674520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:25:51.674520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:08.097729+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":592,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.033584+0000","last_change":"2024-04-14T15:25:49.122361+0000","last_active":"2024-04-14T15:25:59.033584+0000","last_peered":"2024-04-14T15:25:59.033584+0000","last_clean":"2024-04-14T15:25:59.033584+0000","last_became_active":"2024-04-14T15:20:52.989666+0000","last_became_peered":"2024-04-14T15:20:52.989666+0000","last_unstale":"2024-04-14T15:25:59.033584+0000","last_undegraded":"2024-04-14T15:25:59.033584+0000","last_fullsized":"2024-04-14T15:25:59.033584+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:49.122278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:15:40.368219+0000","last_clean_scrub_stamp":"2024-04-14T15:25:49.122278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:18.115547+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"91'130","reported_seq":455,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.028046+0000","last_change":"2024-04-14T15:25:02.372541+0000","last_active":"2024-04-14T15:25:59.028046+0000","last_peered":"2024-04-14T15:25:59.028046+0000","last_clean":"2024-04-14T15:25:59.028046+0000","last_became_active":"2024-04-14T15:25:02.026456+0000","last_became_peered":"2024-04-14T15:25:02.026456+0000","last_unstale":"2024-04-14T15:25:59.028046+0000","last_undegraded":"2024-04-14T15:25:59.028046+0000","last_fullsized":"2024-04-14T15:25:59.028046+0000","mapping_epoch":757,"log_start":"91'130","ondisk_log_start":"91'130","created":752,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:25:02.372440+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:25:02.372440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:17.006246+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":1453,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.038491+0000","last_change":"2024-04-14T15:25:20.823771+0000","last_active":"2024-04-14T15:25:59.038491+0000","last_peered":"2024-04-14T15:25:59.038491+0000","last_clean":"2024-04-14T15:25:59.038491+0000","last_became_active":"2024-04-14T15:20:52.991952+0000","last_became_peered":"2024-04-14T15:20:52.991952+0000","last_unstale":"2024-04-14T15:25:59.038491+0000","last_undegraded":"2024-04-14T15:25:59.038491+0000","last_fullsized":"2024-04-14T15:25:59.038491+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:20.823680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:25:20.823680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:22.548626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"479'15","reported_seq":1578,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.033293+0000","last_change":"2024-04-14T15:25:50.132021+0000","last_active":"2024-04-14T15:25:59.033293+0000","last_peered":"2024-04-14T15:25:59.033293+0000","last_clean":"2024-04-14T15:25:59.033293+0000","last_became_active":"2024-04-14T15:09:45.037179+0000","last_became_peered":"2024-04-14T15:09:45.037179+0000","last_unstale":"2024-04-14T15:25:59.033293+0000","last_undegraded":"2024-04-14T15:25:59.033293+0000","last_fullsized":"2024-04-14T15:25:59.033293+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'15","last_scrub_stamp":"2024-04-14T15:25:50.131933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:25:50.131933+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:51.346947+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":1554,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.031620+0000","last_change":"2024-04-14T15:25:47.000217+0000","last_active":"2024-04-14T15:25:59.031620+0000","last_peered":"2024-04-14T15:25:59.031620+0000","last_clean":"2024-04-14T15:25:59.031620+0000","last_became_active":"2024-04-14T15:09:43.020044+0000","last_became_peered":"2024-04-14T15:09:43.020044+0000","last_unstale":"2024-04-14T15:25:59.031620+0000","last_undegraded":"2024-04-14T15:25:59.031620+0000","last_fullsized":"2024-04-14T15:25:59.031620+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:47.000098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:25:47.000098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:52.332018+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1547,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.028943+0000","last_change":"2024-04-14T15:24:27.475600+0000","last_active":"2024-04-14T15:25:59.028943+0000","last_peered":"2024-04-14T15:25:59.028943+0000","last_clean":"2024-04-14T15:25:59.028943+0000","last_became_active":"2024-04-14T15:09:47.046153+0000","last_became_peered":"2024-04-14T15:09:47.046153+0000","last_unstale":"2024-04-14T15:25:59.028943+0000","last_undegraded":"2024-04-14T15:25:59.028943+0000","last_fullsized":"2024-04-14T15:25:59.028943+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:27.475490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:24:27.475490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1537,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.033786+0000","last_change":"2024-04-14T15:23:37.201923+0000","last_active":"2024-04-14T15:25:59.033786+0000","last_peered":"2024-04-14T15:25:59.033786+0000","last_clean":"2024-04-14T15:25:59.033786+0000","last_became_active":"2024-04-14T15:09:49.061495+0000","last_became_peered":"2024-04-14T15:09:49.061495+0000","last_unstale":"2024-04-14T15:25:59.033786+0000","last_undegraded":"2024-04-14T15:25:59.033786+0000","last_fullsized":"2024-04-14T15:25:59.033786+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:37.201795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:23:37.201795+0000","last_clean_scrub_stamp":"2024-04-14T15:23:37.201795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"91'130","reported_seq":989,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.028167+0000","last_change":"2024-04-14T15:25:03.337059+0000","last_active":"2024-04-14T15:25:59.028167+0000","last_peered":"2024-04-14T15:25:59.028167+0000","last_clean":"2024-04-14T15:25:59.028167+0000","last_became_active":"2024-04-14T15:25:02.029756+0000","last_became_peered":"2024-04-14T15:25:02.029756+0000","last_unstale":"2024-04-14T15:25:59.028167+0000","last_undegraded":"2024-04-14T15:25:59.028167+0000","last_fullsized":"2024-04-14T15:25:59.028167+0000","mapping_epoch":757,"log_start":"91'130","ondisk_log_start":"91'130","created":752,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:25:03.336977+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:25:03.336977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:25.945613+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":1453,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.038832+0000","last_change":"2024-04-14T15:25:27.868432+0000","last_active":"2024-04-14T15:25:59.038832+0000","last_peered":"2024-04-14T15:25:59.038832+0000","last_clean":"2024-04-14T15:25:59.038832+0000","last_became_active":"2024-04-14T15:20:52.999845+0000","last_became_peered":"2024-04-14T15:20:52.999845+0000","last_unstale":"2024-04-14T15:25:59.038832+0000","last_undegraded":"2024-04-14T15:25:59.038832+0000","last_fullsized":"2024-04-14T15:25:59.038832+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:27.868337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:25:27.868337+0000","last_clean_scrub_stamp":"2024-04-14T15:25:27.868337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:34.757816+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"479'44","reported_seq":442,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.028264+0000","last_change":"2024-04-14T15:25:26.767683+0000","last_active":"2024-04-14T15:25:59.028264+0000","last_peered":"2024-04-14T15:25:59.028264+0000","last_clean":"2024-04-14T15:25:59.028264+0000","last_became_active":"2024-04-14T15:22:25.970899+0000","last_became_peered":"2024-04-14T15:22:25.970899+0000","last_unstale":"2024-04-14T15:25:59.028264+0000","last_undegraded":"2024-04-14T15:25:59.028264+0000","last_fullsized":"2024-04-14T15:25:59.028264+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'44","last_scrub_stamp":"2024-04-14T15:25:26.767559+0000","last_deep_scrub":"479'44","last_deep_scrub_stamp":"2024-04-14T15:25:26.767559+0000","last_clean_scrub_stamp":"2024-04-14T15:25:26.767559+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:36.496759+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":1563,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.029542+0000","last_change":"2024-04-14T15:25:53.717840+0000","last_active":"2024-04-14T15:25:59.029542+0000","last_peered":"2024-04-14T15:25:59.029542+0000","last_clean":"2024-04-14T15:25:59.029542+0000","last_became_active":"2024-04-14T15:09:43.020268+0000","last_became_peered":"2024-04-14T15:09:43.020268+0000","last_unstale":"2024-04-14T15:25:59.029542+0000","last_undegraded":"2024-04-14T15:25:59.029542+0000","last_fullsized":"2024-04-14T15:25:59.029542+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:53.717687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:34.342793+0000","last_clean_scrub_stamp":"2024-04-14T15:25:53.717687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:01.063625+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1547,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.028663+0000","last_change":"2024-04-14T15:25:56.701857+0000","last_active":"2024-04-14T15:25:59.028663+0000","last_peered":"2024-04-14T15:25:59.028663+0000","last_clean":"2024-04-14T15:25:59.028663+0000","last_became_active":"2024-04-14T15:09:47.045542+0000","last_became_peered":"2024-04-14T15:09:47.045542+0000","last_unstale":"2024-04-14T15:25:59.028663+0000","last_undegraded":"2024-04-14T15:25:59.028663+0000","last_fullsized":"2024-04-14T15:25:59.028663+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:56.701776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:25:56.701776+0000","last_clean_scrub_stamp":"2024-04-14T15:25:56.701776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:18.809662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1530,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.037646+0000","last_change":"2024-04-14T15:22:55.492856+0000","last_active":"2024-04-14T15:25:59.037646+0000","last_peered":"2024-04-14T15:25:59.037646+0000","last_clean":"2024-04-14T15:25:59.037646+0000","last_became_active":"2024-04-14T15:09:49.065397+0000","last_became_peered":"2024-04-14T15:09:49.065397+0000","last_unstale":"2024-04-14T15:25:59.037646+0000","last_undegraded":"2024-04-14T15:25:59.037646+0000","last_fullsized":"2024-04-14T15:25:59.037646+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:22:55.492753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:22:55.492753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"91'130","reported_seq":1493,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.027460+0000","last_change":"2024-04-14T15:24:36.459484+0000","last_active":"2024-04-14T15:25:59.027460+0000","last_peered":"2024-04-14T15:25:59.027460+0000","last_clean":"2024-04-14T15:25:59.027460+0000","last_became_active":"2024-04-14T15:21:12.961419+0000","last_became_peered":"2024-04-14T15:21:12.961419+0000","last_unstale":"2024-04-14T15:25:59.027460+0000","last_undegraded":"2024-04-14T15:25:59.027460+0000","last_fullsized":"2024-04-14T15:25:59.027460+0000","mapping_epoch":276,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:24:36.459402+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:24:36.459402+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":70,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":1452,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.028670+0000","last_change":"2024-04-14T15:25:11.638473+0000","last_active":"2024-04-14T15:25:59.028670+0000","last_peered":"2024-04-14T15:25:59.028670+0000","last_clean":"2024-04-14T15:25:59.028670+0000","last_became_active":"2024-04-14T15:12:56.474322+0000","last_became_peered":"2024-04-14T15:12:56.474322+0000","last_unstale":"2024-04-14T15:25:59.028670+0000","last_undegraded":"2024-04-14T15:25:59.028670+0000","last_fullsized":"2024-04-14T15:25:59.028670+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:11.638381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:25:11.638381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:26.353663+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"780'39","reported_seq":1651,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.028501+0000","last_change":"2024-04-14T15:25:24.296864+0000","last_active":"2024-04-14T15:25:59.028501+0000","last_peered":"2024-04-14T15:25:59.028501+0000","last_clean":"2024-04-14T15:25:59.028501+0000","last_became_active":"2024-04-14T15:13:12.933907+0000","last_became_peered":"2024-04-14T15:13:12.933907+0000","last_unstale":"2024-04-14T15:25:59.028501+0000","last_undegraded":"2024-04-14T15:25:59.028501+0000","last_fullsized":"2024-04-14T15:25:59.028501+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"780'39","last_scrub_stamp":"2024-04-14T15:25:24.296779+0000","last_deep_scrub":"780'39","last_deep_scrub_stamp":"2024-04-14T15:25:24.296779+0000","last_clean_scrub_stamp":"2024-04-14T15:25:24.296779+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:37.386005+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":1542,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.037795+0000","last_change":"2024-04-14T15:23:26.611299+0000","last_active":"2024-04-14T15:25:59.037795+0000","last_peered":"2024-04-14T15:25:59.037795+0000","last_clean":"2024-04-14T15:25:59.037795+0000","last_became_active":"2024-04-14T15:09:43.022351+0000","last_became_peered":"2024-04-14T15:09:43.022351+0000","last_unstale":"2024-04-14T15:25:59.037795+0000","last_undegraded":"2024-04-14T15:25:59.037795+0000","last_fullsized":"2024-04-14T15:25:59.037795+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:26.611217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:23:26.611217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1546,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.030709+0000","last_change":"2024-04-14T15:25:50.984808+0000","last_active":"2024-04-14T15:25:59.030709+0000","last_peered":"2024-04-14T15:25:59.030709+0000","last_clean":"2024-04-14T15:25:59.030709+0000","last_became_active":"2024-04-14T15:09:47.042810+0000","last_became_peered":"2024-04-14T15:09:47.042810+0000","last_unstale":"2024-04-14T15:25:59.030709+0000","last_undegraded":"2024-04-14T15:25:59.030709+0000","last_fullsized":"2024-04-14T15:25:59.030709+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:50.984722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:25:50.984722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:14.657168+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1543,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.030339+0000","last_change":"2024-04-14T15:25:27.741050+0000","last_active":"2024-04-14T15:25:59.030339+0000","last_peered":"2024-04-14T15:25:59.030339+0000","last_clean":"2024-04-14T15:25:59.030339+0000","last_became_active":"2024-04-14T15:09:49.069594+0000","last_became_peered":"2024-04-14T15:09:49.069594+0000","last_unstale":"2024-04-14T15:25:59.030339+0000","last_undegraded":"2024-04-14T15:25:59.030339+0000","last_fullsized":"2024-04-14T15:25:59.030339+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:27.740959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:25:27.740959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:50.237809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"91'130","reported_seq":1898,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.037401+0000","last_change":"2024-04-14T15:25:51.968976+0000","last_active":"2024-04-14T15:25:59.037401+0000","last_peered":"2024-04-14T15:25:59.037401+0000","last_clean":"2024-04-14T15:25:59.037401+0000","last_became_active":"2024-04-14T15:21:09.020300+0000","last_became_peered":"2024-04-14T15:21:09.020300+0000","last_unstale":"2024-04-14T15:25:59.037401+0000","last_undegraded":"2024-04-14T15:25:59.037401+0000","last_fullsized":"2024-04-14T15:25:59.037401+0000","mapping_epoch":189,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":544,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:25:51.968864+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:25:51.968864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:57.531574+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":582,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.037611+0000","last_change":"2024-04-14T15:23:41.487909+0000","last_active":"2024-04-14T15:25:59.037611+0000","last_peered":"2024-04-14T15:25:59.037611+0000","last_clean":"2024-04-14T15:25:59.037611+0000","last_became_active":"2024-04-14T15:20:52.980375+0000","last_became_peered":"2024-04-14T15:20:52.980375+0000","last_unstale":"2024-04-14T15:25:59.037611+0000","last_undegraded":"2024-04-14T15:25:59.037611+0000","last_fullsized":"2024-04-14T15:25:59.037611+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:41.487814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:23:41.487814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"479'25","reported_seq":1591,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.027687+0000","last_change":"2024-04-14T15:24:34.394922+0000","last_active":"2024-04-14T15:25:59.027687+0000","last_peered":"2024-04-14T15:25:59.027687+0000","last_clean":"2024-04-14T15:25:59.027687+0000","last_became_active":"2024-04-14T15:20:52.987699+0000","last_became_peered":"2024-04-14T15:20:52.987699+0000","last_unstale":"2024-04-14T15:25:59.027687+0000","last_undegraded":"2024-04-14T15:25:59.027687+0000","last_fullsized":"2024-04-14T15:25:59.027687+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'25","last_scrub_stamp":"2024-04-14T15:24:34.394825+0000","last_deep_scrub":"479'25","last_deep_scrub_stamp":"2024-04-14T15:20:03.571160+0000","last_clean_scrub_stamp":"2024-04-14T15:24:34.394825+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":1554,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.029222+0000","last_change":"2024-04-14T15:25:11.596445+0000","last_active":"2024-04-14T15:25:59.029222+0000","last_peered":"2024-04-14T15:25:59.029222+0000","last_clean":"2024-04-14T15:25:59.029222+0000","last_became_active":"2024-04-14T15:09:43.018871+0000","last_became_peered":"2024-04-14T15:09:43.018871+0000","last_unstale":"2024-04-14T15:25:59.029222+0000","last_undegraded":"2024-04-14T15:25:59.029222+0000","last_fullsized":"2024-04-14T15:25:59.029222+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:11.596346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:25:11.596346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:26.607882+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"400'9","reported_seq":1752,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.029886+0000","last_change":"2024-04-14T15:25:49.676706+0000","last_active":"2024-04-14T15:25:59.029886+0000","last_peered":"2024-04-14T15:25:59.029886+0000","last_clean":"2024-04-14T15:25:59.029886+0000","last_became_active":"2024-04-14T15:09:47.044085+0000","last_became_peered":"2024-04-14T15:09:47.044085+0000","last_unstale":"2024-04-14T15:25:59.029886+0000","last_undegraded":"2024-04-14T15:25:59.029886+0000","last_fullsized":"2024-04-14T15:25:59.029886+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'9","last_scrub_stamp":"2024-04-14T15:25:49.676621+0000","last_deep_scrub":"400'9","last_deep_scrub_stamp":"2024-04-14T15:17:54.845881+0000","last_clean_scrub_stamp":"2024-04-14T15:25:49.676621+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:06.813931+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1530,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.028605+0000","last_change":"2024-04-14T15:24:19.552836+0000","last_active":"2024-04-14T15:25:59.028605+0000","last_peered":"2024-04-14T15:25:59.028605+0000","last_clean":"2024-04-14T15:25:59.028605+0000","last_became_active":"2024-04-14T15:09:49.065274+0000","last_became_peered":"2024-04-14T15:09:49.065274+0000","last_unstale":"2024-04-14T15:25:59.028605+0000","last_undegraded":"2024-04-14T15:25:59.028605+0000","last_fullsized":"2024-04-14T15:25:59.028605+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:19.552703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:18.588546+0000","last_clean_scrub_stamp":"2024-04-14T15:24:19.552703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"91'130","reported_seq":1508,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.028509+0000","last_change":"2024-04-14T15:25:06.600246+0000","last_active":"2024-04-14T15:25:59.028509+0000","last_peered":"2024-04-14T15:25:59.028509+0000","last_clean":"2024-04-14T15:25:59.028509+0000","last_became_active":"2024-04-14T15:21:04.971515+0000","last_became_peered":"2024-04-14T15:21:04.971515+0000","last_unstale":"2024-04-14T15:25:59.028509+0000","last_undegraded":"2024-04-14T15:25:59.028509+0000","last_fullsized":"2024-04-14T15:25:59.028509+0000","mapping_epoch":530,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":540,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:25:06.600156+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:25:06.600156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:09.247668+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":1451,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.029057+0000","last_change":"2024-04-14T15:25:54.058911+0000","last_active":"2024-04-14T15:25:59.029057+0000","last_peered":"2024-04-14T15:25:59.029057+0000","last_clean":"2024-04-14T15:25:59.029057+0000","last_became_active":"2024-04-14T15:12:56.487615+0000","last_became_peered":"2024-04-14T15:12:56.487615+0000","last_unstale":"2024-04-14T15:25:59.029057+0000","last_undegraded":"2024-04-14T15:25:59.029057+0000","last_fullsized":"2024-04-14T15:25:59.029057+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:54.058770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:25:54.058770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:22.907144+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"479'34","reported_seq":1600,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.031386+0000","last_change":"2024-04-14T15:24:35.378498+0000","last_active":"2024-04-14T15:25:59.031386+0000","last_peered":"2024-04-14T15:25:59.031386+0000","last_clean":"2024-04-14T15:25:59.031386+0000","last_became_active":"2024-04-14T15:09:45.033331+0000","last_became_peered":"2024-04-14T15:09:45.033331+0000","last_unstale":"2024-04-14T15:25:59.031386+0000","last_undegraded":"2024-04-14T15:25:59.031386+0000","last_fullsized":"2024-04-14T15:25:59.031386+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'34","last_scrub_stamp":"2024-04-14T15:24:35.378379+0000","last_deep_scrub":"479'34","last_deep_scrub_stamp":"2024-04-14T15:24:35.378379+0000","last_clean_scrub_stamp":"2024-04-14T15:24:35.378379+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":1554,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.027926+0000","last_change":"2024-04-14T15:25:47.989868+0000","last_active":"2024-04-14T15:25:59.027926+0000","last_peered":"2024-04-14T15:25:59.027926+0000","last_clean":"2024-04-14T15:25:59.027926+0000","last_became_active":"2024-04-14T15:09:43.020878+0000","last_became_peered":"2024-04-14T15:09:43.020878+0000","last_unstale":"2024-04-14T15:25:59.027926+0000","last_undegraded":"2024-04-14T15:25:59.027926+0000","last_fullsized":"2024-04-14T15:25:59.027926+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:47.989790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:25:47.989790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:53.293672+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"72'8","reported_seq":1738,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.028485+0000","last_change":"2024-04-14T15:24:27.403836+0000","last_active":"2024-04-14T15:25:59.028485+0000","last_peered":"2024-04-14T15:25:59.028485+0000","last_clean":"2024-04-14T15:25:59.028485+0000","last_became_active":"2024-04-14T15:09:47.050962+0000","last_became_peered":"2024-04-14T15:09:47.050962+0000","last_unstale":"2024-04-14T15:25:59.028485+0000","last_undegraded":"2024-04-14T15:25:59.028485+0000","last_fullsized":"2024-04-14T15:25:59.028485+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:24:27.403735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:24:27.403735+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1105,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.030700+0000","last_change":"2024-04-14T15:24:27.318413+0000","last_active":"2024-04-14T15:25:59.030700+0000","last_peered":"2024-04-14T15:25:59.030700+0000","last_clean":"2024-04-14T15:25:59.030700+0000","last_became_active":"2024-04-14T15:15:44.879699+0000","last_became_peered":"2024-04-14T15:15:44.879699+0000","last_unstale":"2024-04-14T15:25:59.030700+0000","last_undegraded":"2024-04-14T15:25:59.030700+0000","last_fullsized":"2024-04-14T15:25:59.030700+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:27.318330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:24:27.318330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"91'130","reported_seq":1719,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.029880+0000","last_change":"2024-04-14T15:25:02.472703+0000","last_active":"2024-04-14T15:25:59.029880+0000","last_peered":"2024-04-14T15:25:59.029880+0000","last_clean":"2024-04-14T15:25:59.029880+0000","last_became_active":"2024-04-14T15:25:02.018082+0000","last_became_peered":"2024-04-14T15:25:02.018082+0000","last_unstale":"2024-04-14T15:25:59.029880+0000","last_undegraded":"2024-04-14T15:25:59.029880+0000","last_fullsized":"2024-04-14T15:25:59.029880+0000","mapping_epoch":757,"log_start":"91'130","ondisk_log_start":"91'130","created":752,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:25:02.472613+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:25:02.472613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:09.501028+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"233'8","reported_seq":860,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.045586+0000","last_change":"2024-04-14T15:23:43.908078+0000","last_active":"2024-04-14T15:25:59.045586+0000","last_peered":"2024-04-14T15:25:59.045586+0000","last_clean":"2024-04-14T15:25:59.045586+0000","last_became_active":"2024-04-14T15:20:52.995129+0000","last_became_peered":"2024-04-14T15:20:52.995129+0000","last_unstale":"2024-04-14T15:25:59.045586+0000","last_undegraded":"2024-04-14T15:25:59.045586+0000","last_fullsized":"2024-04-14T15:25:59.045586+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'8","last_scrub_stamp":"2024-04-14T15:23:43.907946+0000","last_deep_scrub":"233'8","last_deep_scrub_stamp":"2024-04-14T15:15:01.295800+0000","last_clean_scrub_stamp":"2024-04-14T15:23:43.907946+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":5,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"479'24","reported_seq":1600,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.028352+0000","last_change":"2024-04-14T15:25:20.326455+0000","last_active":"2024-04-14T15:25:59.028352+0000","last_peered":"2024-04-14T15:25:59.028352+0000","last_clean":"2024-04-14T15:25:59.028352+0000","last_became_active":"2024-04-14T15:20:52.998853+0000","last_became_peered":"2024-04-14T15:20:52.998853+0000","last_unstale":"2024-04-14T15:25:59.028352+0000","last_undegraded":"2024-04-14T15:25:59.028352+0000","last_fullsized":"2024-04-14T15:25:59.028352+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'24","last_scrub_stamp":"2024-04-14T15:25:20.326301+0000","last_deep_scrub":"85'16","last_deep_scrub_stamp":"2024-04-14T15:10:47.456169+0000","last_clean_scrub_stamp":"2024-04-14T15:25:20.326301+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:26.745364+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":1550,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.037510+0000","last_change":"2024-04-14T15:23:36.489554+0000","last_active":"2024-04-14T15:25:59.037510+0000","last_peered":"2024-04-14T15:25:59.037510+0000","last_clean":"2024-04-14T15:25:59.037510+0000","last_became_active":"2024-04-14T15:09:43.018601+0000","last_became_peered":"2024-04-14T15:09:43.018601+0000","last_unstale":"2024-04-14T15:25:59.037510+0000","last_undegraded":"2024-04-14T15:25:59.037510+0000","last_fullsized":"2024-04-14T15:25:59.037510+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:36.489475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:23:36.489475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1106,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.028485+0000","last_change":"2024-04-14T15:24:35.426298+0000","last_active":"2024-04-14T15:25:59.028485+0000","last_peered":"2024-04-14T15:25:59.028485+0000","last_clean":"2024-04-14T15:25:59.028485+0000","last_became_active":"2024-04-14T15:20:52.970831+0000","last_became_peered":"2024-04-14T15:20:52.970831+0000","last_unstale":"2024-04-14T15:25:59.028485+0000","last_undegraded":"2024-04-14T15:25:59.028485+0000","last_fullsized":"2024-04-14T15:25:59.028485+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:35.426204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:24:35.426204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1215,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.046001+0000","last_change":"2024-04-14T15:24:32.995639+0000","last_active":"2024-04-14T15:25:59.046001+0000","last_peered":"2024-04-14T15:25:59.046001+0000","last_clean":"2024-04-14T15:25:59.046001+0000","last_became_active":"2024-04-14T15:20:52.999771+0000","last_became_peered":"2024-04-14T15:20:52.999771+0000","last_unstale":"2024-04-14T15:25:59.046001+0000","last_undegraded":"2024-04-14T15:25:59.046001+0000","last_fullsized":"2024-04-14T15:25:59.046001+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:32.995512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:57.214046+0000","last_clean_scrub_stamp":"2024-04-14T15:24:32.995512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"91'130","reported_seq":1697,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.045841+0000","last_change":"2024-04-14T15:25:04.733894+0000","last_active":"2024-04-14T15:25:59.045841+0000","last_peered":"2024-04-14T15:25:59.045841+0000","last_clean":"2024-04-14T15:25:59.045841+0000","last_became_active":"2024-04-14T15:24:57.023208+0000","last_became_peered":"2024-04-14T15:24:57.023208+0000","last_unstale":"2024-04-14T15:25:59.045841+0000","last_undegraded":"2024-04-14T15:25:59.045841+0000","last_fullsized":"2024-04-14T15:25:59.045841+0000","mapping_epoch":189,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":753,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:25:04.733792+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:25:04.733792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":60,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:15.643111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":1446,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.028283+0000","last_change":"2024-04-14T15:24:37.496873+0000","last_active":"2024-04-14T15:25:59.028283+0000","last_peered":"2024-04-14T15:25:59.028283+0000","last_clean":"2024-04-14T15:25:59.028283+0000","last_became_active":"2024-04-14T15:12:56.427276+0000","last_became_peered":"2024-04-14T15:12:56.427276+0000","last_unstale":"2024-04-14T15:25:59.028283+0000","last_undegraded":"2024-04-14T15:25:59.028283+0000","last_fullsized":"2024-04-14T15:25:59.028283+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:37.496778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:24:37.496778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:03.328438+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"255'10","reported_seq":1387,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.045500+0000","last_change":"2024-04-14T15:25:48.705124+0000","last_active":"2024-04-14T15:25:59.045500+0000","last_peered":"2024-04-14T15:25:59.045500+0000","last_clean":"2024-04-14T15:25:59.045500+0000","last_became_active":"2024-04-14T15:15:21.873747+0000","last_became_peered":"2024-04-14T15:15:21.873747+0000","last_unstale":"2024-04-14T15:25:59.045500+0000","last_undegraded":"2024-04-14T15:25:59.045500+0000","last_fullsized":"2024-04-14T15:25:59.045500+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'10","last_scrub_stamp":"2024-04-14T15:25:48.705041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:25:48.705041+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:50.660926+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"479'28","reported_seq":1610,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.027171+0000","last_change":"2024-04-14T15:25:47.281406+0000","last_active":"2024-04-14T15:25:59.027171+0000","last_peered":"2024-04-14T15:25:59.027171+0000","last_clean":"2024-04-14T15:25:59.027171+0000","last_became_active":"2024-04-14T15:20:52.989635+0000","last_became_peered":"2024-04-14T15:20:52.989635+0000","last_unstale":"2024-04-14T15:25:59.027171+0000","last_undegraded":"2024-04-14T15:25:59.027171+0000","last_fullsized":"2024-04-14T15:25:59.027171+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'28","last_scrub_stamp":"2024-04-14T15:25:47.281304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:25:47.281304+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:02.110971+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":590,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.039570+0000","last_change":"2024-04-14T15:25:46.849300+0000","last_active":"2024-04-14T15:25:59.039570+0000","last_peered":"2024-04-14T15:25:59.039570+0000","last_clean":"2024-04-14T15:25:59.039570+0000","last_became_active":"2024-04-14T15:20:52.999379+0000","last_became_peered":"2024-04-14T15:20:52.999379+0000","last_unstale":"2024-04-14T15:25:59.039570+0000","last_undegraded":"2024-04-14T15:25:59.039570+0000","last_fullsized":"2024-04-14T15:25:59.039570+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:46.849220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:25:46.849220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:58.079083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1536,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.038246+0000","last_change":"2024-04-14T15:23:28.571478+0000","last_active":"2024-04-14T15:25:59.038246+0000","last_peered":"2024-04-14T15:25:59.038246+0000","last_clean":"2024-04-14T15:25:59.038246+0000","last_became_active":"2024-04-14T15:20:52.991388+0000","last_became_peered":"2024-04-14T15:20:52.991388+0000","last_unstale":"2024-04-14T15:25:59.038246+0000","last_undegraded":"2024-04-14T15:25:59.038246+0000","last_fullsized":"2024-04-14T15:25:59.038246+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:28.571374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:23:28.571374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"91'130","reported_seq":1706,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.027897+0000","last_change":"2024-04-14T15:25:04.328620+0000","last_active":"2024-04-14T15:25:59.027897+0000","last_peered":"2024-04-14T15:25:59.027897+0000","last_clean":"2024-04-14T15:25:59.027897+0000","last_became_active":"2024-04-14T15:24:57.015409+0000","last_became_peered":"2024-04-14T15:24:57.015409+0000","last_unstale":"2024-04-14T15:25:59.027897+0000","last_undegraded":"2024-04-14T15:25:59.027897+0000","last_fullsized":"2024-04-14T15:25:59.027897+0000","mapping_epoch":530,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":753,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:25:04.328526+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:25:04.328526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:29.270285+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":1446,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.038772+0000","last_change":"2024-04-14T15:24:27.475544+0000","last_active":"2024-04-14T15:25:59.038772+0000","last_peered":"2024-04-14T15:25:59.038772+0000","last_clean":"2024-04-14T15:25:59.038772+0000","last_became_active":"2024-04-14T15:12:56.355254+0000","last_became_peered":"2024-04-14T15:12:56.355254+0000","last_unstale":"2024-04-14T15:25:59.038772+0000","last_undegraded":"2024-04-14T15:25:59.038772+0000","last_fullsized":"2024-04-14T15:25:59.038772+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:27.475460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:24:27.475460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1542,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.027392+0000","last_change":"2024-04-14T15:25:52.349307+0000","last_active":"2024-04-14T15:25:59.027392+0000","last_peered":"2024-04-14T15:25:59.027392+0000","last_clean":"2024-04-14T15:25:59.027392+0000","last_became_active":"2024-04-14T15:09:47.044610+0000","last_became_peered":"2024-04-14T15:09:47.044610+0000","last_unstale":"2024-04-14T15:25:59.027392+0000","last_undegraded":"2024-04-14T15:25:59.027392+0000","last_fullsized":"2024-04-14T15:25:59.027392+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:52.349217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:39.394655+0000","last_clean_scrub_stamp":"2024-04-14T15:25:52.349217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:02.647785+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"783'38","reported_seq":1285,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.045430+0000","last_change":"2024-04-14T15:25:27.804694+0000","last_active":"2024-04-14T15:25:59.045430+0000","last_peered":"2024-04-14T15:25:59.045430+0000","last_clean":"2024-04-14T15:25:59.045430+0000","last_became_active":"2024-04-14T15:15:21.880565+0000","last_became_peered":"2024-04-14T15:15:21.880565+0000","last_unstale":"2024-04-14T15:25:59.045430+0000","last_undegraded":"2024-04-14T15:25:59.045430+0000","last_fullsized":"2024-04-14T15:25:59.045430+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"783'38","last_scrub_stamp":"2024-04-14T15:25:27.804604+0000","last_deep_scrub":"783'38","last_deep_scrub_stamp":"2024-04-14T15:25:27.804604+0000","last_clean_scrub_stamp":"2024-04-14T15:25:27.804604+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:49.748235+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'2","reported_seq":1558,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.031117+0000","last_change":"2024-04-14T15:25:05.498299+0000","last_active":"2024-04-14T15:25:59.031117+0000","last_peered":"2024-04-14T15:25:59.031117+0000","last_clean":"2024-04-14T15:25:59.031117+0000","last_became_active":"2024-04-14T15:20:52.987576+0000","last_became_peered":"2024-04-14T15:20:52.987576+0000","last_unstale":"2024-04-14T15:25:59.031117+0000","last_undegraded":"2024-04-14T15:25:59.031117+0000","last_fullsized":"2024-04-14T15:25:59.031117+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'2","last_scrub_stamp":"2024-04-14T15:25:05.498215+0000","last_deep_scrub":"91'2","last_deep_scrub_stamp":"2024-04-14T15:17:48.850069+0000","last_clean_scrub_stamp":"2024-04-14T15:25:05.498215+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:29.023294+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1530,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.028308+0000","last_change":"2024-04-14T15:23:33.834918+0000","last_active":"2024-04-14T15:25:59.028308+0000","last_peered":"2024-04-14T15:25:59.028308+0000","last_clean":"2024-04-14T15:25:59.028308+0000","last_became_active":"2024-04-14T15:09:49.065684+0000","last_became_peered":"2024-04-14T15:09:49.065684+0000","last_unstale":"2024-04-14T15:25:59.028308+0000","last_undegraded":"2024-04-14T15:25:59.028308+0000","last_fullsized":"2024-04-14T15:25:59.028308+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:33.834838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:23:33.834838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"91'130","reported_seq":976,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.034192+0000","last_change":"2024-04-14T15:25:04.746467+0000","last_active":"2024-04-14T15:25:59.034192+0000","last_peered":"2024-04-14T15:25:59.034192+0000","last_clean":"2024-04-14T15:25:59.034192+0000","last_became_active":"2024-04-14T15:24:57.025580+0000","last_became_peered":"2024-04-14T15:24:57.025580+0000","last_unstale":"2024-04-14T15:25:59.034192+0000","last_undegraded":"2024-04-14T15:25:59.034192+0000","last_fullsized":"2024-04-14T15:25:59.034192+0000","mapping_epoch":530,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":753,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:25:04.746393+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:25:04.746393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:33.227095+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":1450,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.030616+0000","last_change":"2024-04-14T15:23:24.298794+0000","last_active":"2024-04-14T15:25:59.030616+0000","last_peered":"2024-04-14T15:25:59.030616+0000","last_clean":"2024-04-14T15:25:59.030616+0000","last_became_active":"2024-04-14T15:12:56.402023+0000","last_became_peered":"2024-04-14T15:12:56.402023+0000","last_unstale":"2024-04-14T15:25:59.030616+0000","last_undegraded":"2024-04-14T15:25:59.030616+0000","last_fullsized":"2024-04-14T15:25:59.030616+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:24.298717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:35.003968+0000","last_clean_scrub_stamp":"2024-04-14T15:23:24.298717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":588,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.033444+0000","last_change":"2024-04-14T15:24:18.368805+0000","last_active":"2024-04-14T15:25:59.033444+0000","last_peered":"2024-04-14T15:25:59.033444+0000","last_clean":"2024-04-14T15:25:59.033444+0000","last_became_active":"2024-04-14T15:20:52.987883+0000","last_became_peered":"2024-04-14T15:20:52.987883+0000","last_unstale":"2024-04-14T15:25:59.033444+0000","last_undegraded":"2024-04-14T15:25:59.033444+0000","last_fullsized":"2024-04-14T15:25:59.033444+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:18.368725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:24:18.368725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"479'37","reported_seq":1612,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.033459+0000","last_change":"2024-04-14T15:25:58.107928+0000","last_active":"2024-04-14T15:25:59.033459+0000","last_peered":"2024-04-14T15:25:59.033459+0000","last_clean":"2024-04-14T15:25:59.033459+0000","last_became_active":"2024-04-14T15:09:45.179182+0000","last_became_peered":"2024-04-14T15:09:45.179182+0000","last_unstale":"2024-04-14T15:25:59.033459+0000","last_undegraded":"2024-04-14T15:25:59.033459+0000","last_fullsized":"2024-04-14T15:25:59.033459+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'37","last_scrub_stamp":"2024-04-14T15:25:58.107835+0000","last_deep_scrub":"102'27","last_deep_scrub_stamp":"2024-04-14T15:17:46.924330+0000","last_clean_scrub_stamp":"2024-04-14T15:25:58.107835+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:03.487475+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1550,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.038638+0000","last_change":"2024-04-14T15:25:55.977430+0000","last_active":"2024-04-14T15:25:59.038638+0000","last_peered":"2024-04-14T15:25:59.038638+0000","last_clean":"2024-04-14T15:25:59.038638+0000","last_became_active":"2024-04-14T15:09:43.021006+0000","last_became_peered":"2024-04-14T15:09:43.021006+0000","last_unstale":"2024-04-14T15:25:59.038638+0000","last_undegraded":"2024-04-14T15:25:59.038638+0000","last_fullsized":"2024-04-14T15:25:59.038638+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:55.977352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:25:55.977352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:56.535906+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1540,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.028044+0000","last_change":"2024-04-14T15:23:34.182007+0000","last_active":"2024-04-14T15:25:59.028044+0000","last_peered":"2024-04-14T15:25:59.028044+0000","last_clean":"2024-04-14T15:25:59.028044+0000","last_became_active":"2024-04-14T15:20:52.978765+0000","last_became_peered":"2024-04-14T15:20:52.978765+0000","last_unstale":"2024-04-14T15:25:59.028044+0000","last_undegraded":"2024-04-14T15:25:59.028044+0000","last_fullsized":"2024-04-14T15:25:59.028044+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:34.181914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:56.206649+0000","last_clean_scrub_stamp":"2024-04-14T15:23:34.181914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"91'130","reported_seq":1705,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.030186+0000","last_change":"2024-04-14T15:25:03.514185+0000","last_active":"2024-04-14T15:25:59.030186+0000","last_peered":"2024-04-14T15:25:59.030186+0000","last_clean":"2024-04-14T15:25:59.030186+0000","last_became_active":"2024-04-14T15:24:57.012410+0000","last_became_peered":"2024-04-14T15:24:57.012410+0000","last_unstale":"2024-04-14T15:25:59.030186+0000","last_undegraded":"2024-04-14T15:25:59.030186+0000","last_fullsized":"2024-04-14T15:25:59.030186+0000","mapping_epoch":530,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":753,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:25:03.514098+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:25:03.514098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:33.492762+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":1163,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.045680+0000","last_change":"2024-04-14T15:25:53.791322+0000","last_active":"2024-04-14T15:25:59.045680+0000","last_peered":"2024-04-14T15:25:59.045680+0000","last_clean":"2024-04-14T15:25:59.045680+0000","last_became_active":"2024-04-14T15:20:52.999776+0000","last_became_peered":"2024-04-14T15:20:52.999776+0000","last_unstale":"2024-04-14T15:25:59.045680+0000","last_undegraded":"2024-04-14T15:25:59.045680+0000","last_fullsized":"2024-04-14T15:25:59.045680+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:53.791183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:25:53.791183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:19.819673+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1104,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.031531+0000","last_change":"2024-04-14T15:25:14.592763+0000","last_active":"2024-04-14T15:25:59.031531+0000","last_peered":"2024-04-14T15:25:59.031531+0000","last_clean":"2024-04-14T15:25:59.031531+0000","last_became_active":"2024-04-14T15:15:44.883694+0000","last_became_peered":"2024-04-14T15:15:44.883694+0000","last_unstale":"2024-04-14T15:25:59.031531+0000","last_undegraded":"2024-04-14T15:25:59.031531+0000","last_fullsized":"2024-04-14T15:25:59.031531+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:14.592662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:25:14.592662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:40.007813+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"479'30","reported_seq":1587,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.034427+0000","last_change":"2024-04-14T15:23:30.295880+0000","last_active":"2024-04-14T15:25:59.034427+0000","last_peered":"2024-04-14T15:25:59.034427+0000","last_clean":"2024-04-14T15:25:59.034427+0000","last_became_active":"2024-04-14T15:09:45.037231+0000","last_became_peered":"2024-04-14T15:09:45.037231+0000","last_unstale":"2024-04-14T15:25:59.034427+0000","last_undegraded":"2024-04-14T15:25:59.034427+0000","last_fullsized":"2024-04-14T15:25:59.034427+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'30","last_scrub_stamp":"2024-04-14T15:23:30.295747+0000","last_deep_scrub":"182'18","last_deep_scrub_stamp":"2024-04-14T15:14:10.383906+0000","last_clean_scrub_stamp":"2024-04-14T15:23:30.295747+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1551,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.037632+0000","last_change":"2024-04-14T15:24:29.563849+0000","last_active":"2024-04-14T15:25:59.037632+0000","last_peered":"2024-04-14T15:25:59.037632+0000","last_clean":"2024-04-14T15:25:59.037632+0000","last_became_active":"2024-04-14T15:09:43.025095+0000","last_became_peered":"2024-04-14T15:09:43.025095+0000","last_unstale":"2024-04-14T15:25:59.037632+0000","last_undegraded":"2024-04-14T15:25:59.037632+0000","last_fullsized":"2024-04-14T15:25:59.037632+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:29.563759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:29.563759+0000","last_clean_scrub_stamp":"2024-04-14T15:24:29.563759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1556,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.033180+0000","last_change":"2024-04-14T15:23:40.238604+0000","last_active":"2024-04-14T15:25:59.033180+0000","last_peered":"2024-04-14T15:25:59.033180+0000","last_clean":"2024-04-14T15:25:59.033180+0000","last_became_active":"2024-04-14T15:09:43.018509+0000","last_became_peered":"2024-04-14T15:09:43.018509+0000","last_unstale":"2024-04-14T15:25:59.033180+0000","last_undegraded":"2024-04-14T15:25:59.033180+0000","last_fullsized":"2024-04-14T15:25:59.033180+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:40.238527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:23:40.238527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0018592680000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"355'2","reported_seq":1536,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.030780+0000","last_change":"2024-04-14T15:23:43.059738+0000","last_active":"2024-04-14T15:25:59.030780+0000","last_peered":"2024-04-14T15:25:59.030780+0000","last_clean":"2024-04-14T15:25:59.030780+0000","last_became_active":"2024-04-14T15:09:49.065243+0000","last_became_peered":"2024-04-14T15:09:49.065243+0000","last_unstale":"2024-04-14T15:25:59.030780+0000","last_undegraded":"2024-04-14T15:25:59.030780+0000","last_fullsized":"2024-04-14T15:25:59.030780+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'2","last_scrub_stamp":"2024-04-14T15:23:43.059603+0000","last_deep_scrub":"355'2","last_deep_scrub_stamp":"2024-04-14T15:20:04.842881+0000","last_clean_scrub_stamp":"2024-04-14T15:23:43.059603+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":1147,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.045553+0000","last_change":"2024-04-14T15:24:28.904419+0000","last_active":"2024-04-14T15:25:59.045553+0000","last_peered":"2024-04-14T15:25:59.045553+0000","last_clean":"2024-04-14T15:25:59.045553+0000","last_became_active":"2024-04-14T15:20:53.000058+0000","last_became_peered":"2024-04-14T15:20:53.000058+0000","last_unstale":"2024-04-14T15:25:59.045553+0000","last_undegraded":"2024-04-14T15:25:59.045553+0000","last_fullsized":"2024-04-14T15:25:59.045553+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:28.904336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:24:28.904336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1534,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.038706+0000","last_change":"2024-04-14T15:23:05.574296+0000","last_active":"2024-04-14T15:25:59.038706+0000","last_peered":"2024-04-14T15:25:59.038706+0000","last_clean":"2024-04-14T15:25:59.038706+0000","last_became_active":"2024-04-14T15:09:47.047201+0000","last_became_peered":"2024-04-14T15:09:47.047201+0000","last_unstale":"2024-04-14T15:25:59.038706+0000","last_undegraded":"2024-04-14T15:25:59.038706+0000","last_fullsized":"2024-04-14T15:25:59.038706+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:05.574212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:23:05.574212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"528'37","reported_seq":1700,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.028238+0000","last_change":"2024-04-14T15:23:29.977939+0000","last_active":"2024-04-14T15:25:59.028238+0000","last_peered":"2024-04-14T15:25:59.028238+0000","last_clean":"2024-04-14T15:25:59.028238+0000","last_became_active":"2024-04-14T15:20:53.002282+0000","last_became_peered":"2024-04-14T15:20:53.002282+0000","last_unstale":"2024-04-14T15:25:59.028238+0000","last_undegraded":"2024-04-14T15:25:59.028238+0000","last_fullsized":"2024-04-14T15:25:59.028238+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"528'37","last_scrub_stamp":"2024-04-14T15:23:29.977844+0000","last_deep_scrub":"90'24","last_deep_scrub_stamp":"2024-04-14T15:11:25.120692+0000","last_clean_scrub_stamp":"2024-04-14T15:23:29.977844+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"91'130","reported_seq":1903,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.037521+0000","last_change":"2024-04-14T15:25:04.953500+0000","last_active":"2024-04-14T15:25:59.037521+0000","last_peered":"2024-04-14T15:25:59.037521+0000","last_clean":"2024-04-14T15:25:59.037521+0000","last_became_active":"2024-04-14T15:24:57.020227+0000","last_became_peered":"2024-04-14T15:24:57.020227+0000","last_unstale":"2024-04-14T15:25:59.037521+0000","last_undegraded":"2024-04-14T15:25:59.037521+0000","last_fullsized":"2024-04-14T15:25:59.037521+0000","mapping_epoch":530,"log_start":"91'130","ondisk_log_start":"91'130","created":19,"last_epoch_clean":753,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:25:04.953418+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:25:04.953418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:12.268060+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1113,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.028766+0000","last_change":"2024-04-14T15:25:48.704691+0000","last_active":"2024-04-14T15:25:59.028766+0000","last_peered":"2024-04-14T15:25:59.028766+0000","last_clean":"2024-04-14T15:25:59.028766+0000","last_became_active":"2024-04-14T15:15:44.885315+0000","last_became_peered":"2024-04-14T15:15:44.885315+0000","last_unstale":"2024-04-14T15:25:59.028766+0000","last_undegraded":"2024-04-14T15:25:59.028766+0000","last_fullsized":"2024-04-14T15:25:59.028766+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:48.704559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:47.627305+0000","last_clean_scrub_stamp":"2024-04-14T15:25:48.704559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:50.327058+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"69'2","reported_seq":1242,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.045609+0000","last_change":"2024-04-14T15:23:44.863113+0000","last_active":"2024-04-14T15:25:59.045609+0000","last_peered":"2024-04-14T15:25:59.045609+0000","last_clean":"2024-04-14T15:25:59.045609+0000","last_became_active":"2024-04-14T15:20:53.002545+0000","last_became_peered":"2024-04-14T15:20:53.002545+0000","last_unstale":"2024-04-14T15:25:59.045609+0000","last_undegraded":"2024-04-14T15:25:59.045609+0000","last_fullsized":"2024-04-14T15:25:59.045609+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-04-14T15:23:44.862962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:23:44.862962+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0058046169999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1096,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.047607+0000","last_change":"2024-04-14T15:23:33.934414+0000","last_active":"2024-04-14T15:25:59.047607+0000","last_peered":"2024-04-14T15:25:59.047607+0000","last_clean":"2024-04-14T15:25:59.047607+0000","last_became_active":"2024-04-14T15:15:44.887934+0000","last_became_peered":"2024-04-14T15:15:44.887934+0000","last_unstale":"2024-04-14T15:25:59.047607+0000","last_undegraded":"2024-04-14T15:25:59.047607+0000","last_fullsized":"2024-04-14T15:25:59.047607+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:33.934334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:23:33.934334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"91'130","reported_seq":1903,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.037584+0000","last_change":"2024-04-14T15:25:03.960485+0000","last_active":"2024-04-14T15:25:59.037584+0000","last_peered":"2024-04-14T15:25:59.037584+0000","last_clean":"2024-04-14T15:25:59.037584+0000","last_became_active":"2024-04-14T15:24:57.011910+0000","last_became_peered":"2024-04-14T15:24:57.011910+0000","last_unstale":"2024-04-14T15:25:59.037584+0000","last_undegraded":"2024-04-14T15:25:59.037584+0000","last_fullsized":"2024-04-14T15:25:59.037584+0000","mapping_epoch":189,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":753,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:25:03.960401+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:22:17.518666+0000","last_clean_scrub_stamp":"2024-04-14T15:25:03.960401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:08.114054+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":1155,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.045318+0000","last_change":"2024-04-14T15:25:18.772143+0000","last_active":"2024-04-14T15:25:59.045318+0000","last_peered":"2024-04-14T15:25:59.045318+0000","last_clean":"2024-04-14T15:25:59.045318+0000","last_became_active":"2024-04-14T15:20:52.986308+0000","last_became_peered":"2024-04-14T15:20:52.986308+0000","last_unstale":"2024-04-14T15:25:59.045318+0000","last_undegraded":"2024-04-14T15:25:59.045318+0000","last_fullsized":"2024-04-14T15:25:59.045318+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:18.772063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:25:18.772063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:44.850845+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1547,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.030626+0000","last_change":"2024-04-14T15:25:22.763238+0000","last_active":"2024-04-14T15:25:59.030626+0000","last_peered":"2024-04-14T15:25:59.030626+0000","last_clean":"2024-04-14T15:25:59.030626+0000","last_became_active":"2024-04-14T15:09:47.049457+0000","last_became_peered":"2024-04-14T15:09:47.049457+0000","last_unstale":"2024-04-14T15:25:59.030626+0000","last_undegraded":"2024-04-14T15:25:59.030626+0000","last_fullsized":"2024-04-14T15:25:59.030626+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:22.763147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:25:22.763147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:42.033176+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"479'22","reported_seq":1588,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.031465+0000","last_change":"2024-04-14T15:25:56.109765+0000","last_active":"2024-04-14T15:25:59.031465+0000","last_peered":"2024-04-14T15:25:59.031465+0000","last_clean":"2024-04-14T15:25:59.031465+0000","last_became_active":"2024-04-14T15:09:45.033883+0000","last_became_peered":"2024-04-14T15:09:45.033883+0000","last_unstale":"2024-04-14T15:25:59.031465+0000","last_undegraded":"2024-04-14T15:25:59.031465+0000","last_fullsized":"2024-04-14T15:25:59.031465+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'22","last_scrub_stamp":"2024-04-14T15:25:56.109673+0000","last_deep_scrub":"308'18","last_deep_scrub_stamp":"2024-04-14T15:16:16.054032+0000","last_clean_scrub_stamp":"2024-04-14T15:25:56.109673+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:12.349554+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1555,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.028730+0000","last_change":"2024-04-14T15:24:29.474621+0000","last_active":"2024-04-14T15:25:59.028730+0000","last_peered":"2024-04-14T15:25:59.028730+0000","last_clean":"2024-04-14T15:25:59.028730+0000","last_became_active":"2024-04-14T15:09:43.021188+0000","last_became_peered":"2024-04-14T15:09:43.021188+0000","last_unstale":"2024-04-14T15:25:59.028730+0000","last_undegraded":"2024-04-14T15:25:59.028730+0000","last_fullsized":"2024-04-14T15:25:59.028730+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:29.474544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:24:29.474544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:25:59.120277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"73'5","reported_seq":2543,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.029340+0000","last_change":"2024-04-14T15:25:26.770287+0000","last_active":"2024-04-14T15:25:59.029340+0000","last_peered":"2024-04-14T15:25:59.029340+0000","last_clean":"2024-04-14T15:25:59.029340+0000","last_became_active":"2024-04-14T15:20:52.975473+0000","last_became_peered":"2024-04-14T15:20:52.975473+0000","last_unstale":"2024-04-14T15:25:59.029340+0000","last_undegraded":"2024-04-14T15:25:59.029340+0000","last_fullsized":"2024-04-14T15:25:59.029340+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-04-14T15:25:26.770199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:25:26.770199+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:30.974644+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00056542099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":967,"num_read_kb":962,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1536,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.038685+0000","last_change":"2024-04-14T15:23:01.536017+0000","last_active":"2024-04-14T15:25:59.038685+0000","last_peered":"2024-04-14T15:25:59.038685+0000","last_clean":"2024-04-14T15:25:59.038685+0000","last_became_active":"2024-04-14T15:20:52.999464+0000","last_became_peered":"2024-04-14T15:20:52.999464+0000","last_unstale":"2024-04-14T15:25:59.038685+0000","last_undegraded":"2024-04-14T15:25:59.038685+0000","last_fullsized":"2024-04-14T15:25:59.038685+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:01.535941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:20.827589+0000","last_clean_scrub_stamp":"2024-04-14T15:23:01.535941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"91'130","reported_seq":1522,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.032979+0000","last_change":"2024-04-14T15:25:03.722817+0000","last_active":"2024-04-14T15:25:59.032979+0000","last_peered":"2024-04-14T15:25:59.032979+0000","last_clean":"2024-04-14T15:25:59.032979+0000","last_became_active":"2024-04-14T15:24:57.020139+0000","last_became_peered":"2024-04-14T15:24:57.020139+0000","last_unstale":"2024-04-14T15:25:59.032979+0000","last_undegraded":"2024-04-14T15:25:59.032979+0000","last_fullsized":"2024-04-14T15:25:59.032979+0000","mapping_epoch":276,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":753,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:25:03.722693+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:25:03.722693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:20.430755+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":1149,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.046397+0000","last_change":"2024-04-14T15:25:21.855268+0000","last_active":"2024-04-14T15:25:59.046397+0000","last_peered":"2024-04-14T15:25:59.046397+0000","last_clean":"2024-04-14T15:25:59.046397+0000","last_became_active":"2024-04-14T15:15:21.883717+0000","last_became_peered":"2024-04-14T15:15:21.883717+0000","last_unstale":"2024-04-14T15:25:59.046397+0000","last_undegraded":"2024-04-14T15:25:59.046397+0000","last_fullsized":"2024-04-14T15:25:59.046397+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:21.855183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:25:21.855183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:36.412681+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1541,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.034372+0000","last_change":"2024-04-14T15:24:26.376324+0000","last_active":"2024-04-14T15:25:59.034372+0000","last_peered":"2024-04-14T15:25:59.034372+0000","last_clean":"2024-04-14T15:25:59.034372+0000","last_became_active":"2024-04-14T15:09:47.040185+0000","last_became_peered":"2024-04-14T15:09:47.040185+0000","last_unstale":"2024-04-14T15:25:59.034372+0000","last_undegraded":"2024-04-14T15:25:59.034372+0000","last_fullsized":"2024-04-14T15:25:59.034372+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:26.376179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:53.731610+0000","last_clean_scrub_stamp":"2024-04-14T15:24:26.376179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"479'40","reported_seq":1804,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.030468+0000","last_change":"2024-04-14T15:23:31.198816+0000","last_active":"2024-04-14T15:25:59.030468+0000","last_peered":"2024-04-14T15:25:59.030468+0000","last_clean":"2024-04-14T15:25:59.030468+0000","last_became_active":"2024-04-14T15:09:45.034168+0000","last_became_peered":"2024-04-14T15:09:45.034168+0000","last_unstale":"2024-04-14T15:25:59.030468+0000","last_undegraded":"2024-04-14T15:25:59.030468+0000","last_fullsized":"2024-04-14T15:25:59.030468+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'40","last_scrub_stamp":"2024-04-14T15:23:31.198708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:23:31.198708+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1110,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.029022+0000","last_change":"2024-04-14T15:23:30.217883+0000","last_active":"2024-04-14T15:25:59.029022+0000","last_peered":"2024-04-14T15:25:59.029022+0000","last_clean":"2024-04-14T15:25:59.029022+0000","last_became_active":"2024-04-14T15:20:52.979943+0000","last_became_peered":"2024-04-14T15:20:52.979943+0000","last_unstale":"2024-04-14T15:25:59.029022+0000","last_undegraded":"2024-04-14T15:25:59.029022+0000","last_fullsized":"2024-04-14T15:25:59.029022+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:30.217808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:24.066275+0000","last_clean_scrub_stamp":"2024-04-14T15:23:30.217808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1201,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.046822+0000","last_change":"2024-04-14T15:24:30.949646+0000","last_active":"2024-04-14T15:25:59.046822+0000","last_peered":"2024-04-14T15:25:59.046822+0000","last_clean":"2024-04-14T15:25:59.046822+0000","last_became_active":"2024-04-14T15:15:21.874742+0000","last_became_peered":"2024-04-14T15:15:21.874742+0000","last_unstale":"2024-04-14T15:25:59.046822+0000","last_undegraded":"2024-04-14T15:25:59.046822+0000","last_fullsized":"2024-04-14T15:25:59.046822+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:30.949569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:30.949569+0000","last_clean_scrub_stamp":"2024-04-14T15:24:30.949569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:00.672224+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"91'130","reported_seq":979,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.046774+0000","last_change":"2024-04-14T15:25:05.749046+0000","last_active":"2024-04-14T15:25:59.046774+0000","last_peered":"2024-04-14T15:25:59.046774+0000","last_clean":"2024-04-14T15:25:59.046774+0000","last_became_active":"2024-04-14T15:24:57.023069+0000","last_became_peered":"2024-04-14T15:24:57.023069+0000","last_unstale":"2024-04-14T15:25:59.046774+0000","last_undegraded":"2024-04-14T15:25:59.046774+0000","last_fullsized":"2024-04-14T15:25:59.046774+0000","mapping_epoch":530,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":753,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:25:05.748903+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:25:05.748903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:22.990045+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":1457,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.029950+0000","last_change":"2024-04-14T15:25:21.747973+0000","last_active":"2024-04-14T15:25:59.029950+0000","last_peered":"2024-04-14T15:25:59.029950+0000","last_clean":"2024-04-14T15:25:59.029950+0000","last_became_active":"2024-04-14T15:20:52.989272+0000","last_became_peered":"2024-04-14T15:20:52.989272+0000","last_unstale":"2024-04-14T15:25:59.029950+0000","last_undegraded":"2024-04-14T15:25:59.029950+0000","last_fullsized":"2024-04-14T15:25:59.029950+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:21.747886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:25:21.747886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:46.136245+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"773'10","reported_seq":1764,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.033240+0000","last_change":"2024-04-14T15:25:17.080605+0000","last_active":"2024-04-14T15:25:59.033240+0000","last_peered":"2024-04-14T15:25:59.033240+0000","last_clean":"2024-04-14T15:25:59.033240+0000","last_became_active":"2024-04-14T15:20:53.000669+0000","last_became_peered":"2024-04-14T15:20:53.000669+0000","last_unstale":"2024-04-14T15:25:59.033240+0000","last_undegraded":"2024-04-14T15:25:59.033240+0000","last_fullsized":"2024-04-14T15:25:59.033240+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"773'10","last_scrub_stamp":"2024-04-14T15:25:17.080518+0000","last_deep_scrub":"773'10","last_deep_scrub_stamp":"2024-04-14T15:25:17.080518+0000","last_clean_scrub_stamp":"2024-04-14T15:25:17.080518+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:20.960802+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"479'20","reported_seq":557,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.028707+0000","last_change":"2024-04-14T15:25:19.646343+0000","last_active":"2024-04-14T15:25:59.028707+0000","last_peered":"2024-04-14T15:25:59.028707+0000","last_clean":"2024-04-14T15:25:59.028707+0000","last_became_active":"2024-04-14T15:21:16.034027+0000","last_became_peered":"2024-04-14T15:21:16.034027+0000","last_unstale":"2024-04-14T15:25:59.028707+0000","last_undegraded":"2024-04-14T15:25:59.028707+0000","last_fullsized":"2024-04-14T15:25:59.028707+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'20","last_scrub_stamp":"2024-04-14T15:25:19.646254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:25:19.646254+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:28.326862+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1546,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.030459+0000","last_change":"2024-04-14T15:25:09.533597+0000","last_active":"2024-04-14T15:25:59.030459+0000","last_peered":"2024-04-14T15:25:59.030459+0000","last_clean":"2024-04-14T15:25:59.030459+0000","last_became_active":"2024-04-14T15:09:43.022597+0000","last_became_peered":"2024-04-14T15:09:43.022597+0000","last_unstale":"2024-04-14T15:25:59.030459+0000","last_undegraded":"2024-04-14T15:25:59.030459+0000","last_fullsized":"2024-04-14T15:25:59.030459+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:09.533502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:21:28.684818+0000","last_clean_scrub_stamp":"2024-04-14T15:25:09.533502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:14.810582+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1538,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.033268+0000","last_change":"2024-04-14T15:24:25.330276+0000","last_active":"2024-04-14T15:25:59.033268+0000","last_peered":"2024-04-14T15:25:59.033268+0000","last_clean":"2024-04-14T15:25:59.033268+0000","last_became_active":"2024-04-14T15:09:49.066648+0000","last_became_peered":"2024-04-14T15:09:49.066648+0000","last_unstale":"2024-04-14T15:25:59.033268+0000","last_undegraded":"2024-04-14T15:25:59.033268+0000","last_fullsized":"2024-04-14T15:25:59.033268+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:25.330148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:24:25.330148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.c","version":"91'130","reported_seq":1719,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.045910+0000","last_change":"2024-04-14T15:25:50.734599+0000","last_active":"2024-04-14T15:25:59.045910+0000","last_peered":"2024-04-14T15:25:59.045910+0000","last_clean":"2024-04-14T15:25:59.045910+0000","last_became_active":"2024-04-14T15:25:32.025973+0000","last_became_peered":"2024-04-14T15:25:32.025973+0000","last_unstale":"2024-04-14T15:25:59.045910+0000","last_undegraded":"2024-04-14T15:25:59.045910+0000","last_fullsized":"2024-04-14T15:25:59.045910+0000","mapping_epoch":787,"log_start":"91'130","ondisk_log_start":"91'130","created":752,"last_epoch_clean":788,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:25:50.734496+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:25:50.734496+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":60,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:57.399176+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":1446,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.034363+0000","last_change":"2024-04-14T15:23:44.358966+0000","last_active":"2024-04-14T15:25:59.034363+0000","last_peered":"2024-04-14T15:25:59.034363+0000","last_clean":"2024-04-14T15:25:59.034363+0000","last_became_active":"2024-04-14T15:12:56.563074+0000","last_became_peered":"2024-04-14T15:12:56.563074+0000","last_unstale":"2024-04-14T15:25:59.034363+0000","last_undegraded":"2024-04-14T15:25:59.034363+0000","last_fullsized":"2024-04-14T15:25:59.034363+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:44.358887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:23:44.358887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1547,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.027638+0000","last_change":"2024-04-14T15:25:57.664021+0000","last_active":"2024-04-14T15:25:59.027638+0000","last_peered":"2024-04-14T15:25:59.027638+0000","last_clean":"2024-04-14T15:25:59.027638+0000","last_became_active":"2024-04-14T15:09:47.049005+0000","last_became_peered":"2024-04-14T15:09:47.049005+0000","last_unstale":"2024-04-14T15:25:59.027638+0000","last_undegraded":"2024-04-14T15:25:59.027638+0000","last_fullsized":"2024-04-14T15:25:59.027638+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:57.663924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:56.915544+0000","last_clean_scrub_stamp":"2024-04-14T15:25:57.663924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:04.748057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"479'23","reported_seq":1583,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.027315+0000","last_change":"2024-04-14T15:24:28.394342+0000","last_active":"2024-04-14T15:25:59.027315+0000","last_peered":"2024-04-14T15:25:59.027315+0000","last_clean":"2024-04-14T15:25:59.027315+0000","last_became_active":"2024-04-14T15:09:45.032892+0000","last_became_peered":"2024-04-14T15:09:45.032892+0000","last_unstale":"2024-04-14T15:25:59.027315+0000","last_undegraded":"2024-04-14T15:25:59.027315+0000","last_fullsized":"2024-04-14T15:25:59.027315+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'23","last_scrub_stamp":"2024-04-14T15:24:28.394222+0000","last_deep_scrub":"183'17","last_deep_scrub_stamp":"2024-04-14T15:14:11.577574+0000","last_clean_scrub_stamp":"2024-04-14T15:24:28.394222+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"677'6","reported_seq":1563,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.028424+0000","last_change":"2024-04-14T15:23:38.773659+0000","last_active":"2024-04-14T15:25:59.028424+0000","last_peered":"2024-04-14T15:25:59.028424+0000","last_clean":"2024-04-14T15:25:59.028424+0000","last_became_active":"2024-04-14T15:09:43.020490+0000","last_became_peered":"2024-04-14T15:09:43.020490+0000","last_unstale":"2024-04-14T15:25:59.028424+0000","last_undegraded":"2024-04-14T15:25:59.028424+0000","last_fullsized":"2024-04-14T15:25:59.028424+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"677'6","last_scrub_stamp":"2024-04-14T15:23:38.773539+0000","last_deep_scrub":"677'6","last_deep_scrub_stamp":"2024-04-14T15:23:38.773539+0000","last_clean_scrub_stamp":"2024-04-14T15:23:38.773539+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":528,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1201,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.046585+0000","last_change":"2024-04-14T15:23:40.854331+0000","last_active":"2024-04-14T15:25:59.046585+0000","last_peered":"2024-04-14T15:25:59.046585+0000","last_clean":"2024-04-14T15:25:59.046585+0000","last_became_active":"2024-04-14T15:15:21.876410+0000","last_became_peered":"2024-04-14T15:15:21.876410+0000","last_unstale":"2024-04-14T15:25:59.046585+0000","last_undegraded":"2024-04-14T15:25:59.046585+0000","last_fullsized":"2024-04-14T15:25:59.046585+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:40.854253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:23:40.854253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"91'130","reported_seq":1723,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.029749+0000","last_change":"2024-04-14T15:25:48.950790+0000","last_active":"2024-04-14T15:25:59.029749+0000","last_peered":"2024-04-14T15:25:59.029749+0000","last_clean":"2024-04-14T15:25:59.029749+0000","last_became_active":"2024-04-14T15:25:32.024118+0000","last_became_peered":"2024-04-14T15:25:32.024118+0000","last_unstale":"2024-04-14T15:25:59.029749+0000","last_undegraded":"2024-04-14T15:25:59.029749+0000","last_fullsized":"2024-04-14T15:25:59.029749+0000","mapping_epoch":787,"log_start":"91'130","ondisk_log_start":"91'130","created":752,"last_epoch_clean":788,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:25:48.950683+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:25:48.950683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:54.252465+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":1453,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.033796+0000","last_change":"2024-04-14T15:24:27.371452+0000","last_active":"2024-04-14T15:25:59.033796+0000","last_peered":"2024-04-14T15:25:59.033796+0000","last_clean":"2024-04-14T15:25:59.033796+0000","last_became_active":"2024-04-14T15:12:56.551177+0000","last_became_peered":"2024-04-14T15:12:56.551177+0000","last_unstale":"2024-04-14T15:25:59.033796+0000","last_undegraded":"2024-04-14T15:25:59.033796+0000","last_fullsized":"2024-04-14T15:25:59.033796+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:27.371370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:22:18.299330+0000","last_clean_scrub_stamp":"2024-04-14T15:24:27.371370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"479'19","reported_seq":1573,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.034435+0000","last_change":"2024-04-14T15:24:33.412261+0000","last_active":"2024-04-14T15:25:59.034435+0000","last_peered":"2024-04-14T15:25:59.034435+0000","last_clean":"2024-04-14T15:25:59.034435+0000","last_became_active":"2024-04-14T15:09:45.180375+0000","last_became_peered":"2024-04-14T15:09:45.180375+0000","last_unstale":"2024-04-14T15:25:59.034435+0000","last_undegraded":"2024-04-14T15:25:59.034435+0000","last_fullsized":"2024-04-14T15:25:59.034435+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'19","last_scrub_stamp":"2024-04-14T15:24:33.412181+0000","last_deep_scrub":"347'13","last_deep_scrub_stamp":"2024-04-14T15:16:55.694599+0000","last_clean_scrub_stamp":"2024-04-14T15:24:33.412181+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":1554,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.027634+0000","last_change":"2024-04-14T15:25:16.586933+0000","last_active":"2024-04-14T15:25:59.027634+0000","last_peered":"2024-04-14T15:25:59.027634+0000","last_clean":"2024-04-14T15:25:59.027634+0000","last_became_active":"2024-04-14T15:09:43.016788+0000","last_became_peered":"2024-04-14T15:09:43.016788+0000","last_unstale":"2024-04-14T15:25:59.027634+0000","last_undegraded":"2024-04-14T15:25:59.027634+0000","last_fullsized":"2024-04-14T15:25:59.027634+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:16.586848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:25:16.586848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:43.100059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1221,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.047589+0000","last_change":"2024-04-14T15:24:33.956401+0000","last_active":"2024-04-14T15:25:59.047589+0000","last_peered":"2024-04-14T15:25:59.047589+0000","last_clean":"2024-04-14T15:25:59.047589+0000","last_became_active":"2024-04-14T15:15:21.873580+0000","last_became_peered":"2024-04-14T15:15:21.873580+0000","last_unstale":"2024-04-14T15:25:59.047589+0000","last_undegraded":"2024-04-14T15:25:59.047589+0000","last_fullsized":"2024-04-14T15:25:59.047589+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:33.956328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:24:33.956328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1542,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.029457+0000","last_change":"2024-04-14T15:25:08.595634+0000","last_active":"2024-04-14T15:25:59.029457+0000","last_peered":"2024-04-14T15:25:59.029457+0000","last_clean":"2024-04-14T15:25:59.029457+0000","last_became_active":"2024-04-14T15:09:49.064296+0000","last_became_peered":"2024-04-14T15:09:49.064296+0000","last_unstale":"2024-04-14T15:25:59.029457+0000","last_undegraded":"2024-04-14T15:25:59.029457+0000","last_fullsized":"2024-04-14T15:25:59.029457+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:08.595556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:25:08.595556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:25.224030+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.12","version":"91'130","reported_seq":440,"reported_epoch":809,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-14T15:25:59.033072+0000","last_change":"2024-04-14T15:25:30.031191+0000","last_active":"2024-04-14T15:25:59.033072+0000","last_peered":"2024-04-14T15:25:59.033072+0000","last_clean":"2024-04-14T15:11:28.874295+0000","last_became_active":"2024-04-14T15:25:00.028240+0000","last_became_peered":"2024-04-14T15:25:00.028240+0000","last_unstale":"2024-04-14T15:25:59.033072+0000","last_undegraded":"2024-04-14T15:25:59.033072+0000","last_fullsized":"2024-04-14T15:25:59.033072+0000","mapping_epoch":755,"log_start":"91'130","ondisk_log_start":"91'130","created":752,"last_epoch_clean":753,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:22:17.518666+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:22:17.518666+0000","last_clean_scrub_stamp":"2024-04-14T15:22:17.518666+0000","objects_scrubbed":2,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":7,"num_write":4,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":40253,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[0,7,4],"avail_no_missing":["0","1","4","5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":1457,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.027555+0000","last_change":"2024-04-14T15:25:52.667570+0000","last_active":"2024-04-14T15:25:59.027555+0000","last_peered":"2024-04-14T15:25:59.027555+0000","last_clean":"2024-04-14T15:25:59.027555+0000","last_became_active":"2024-04-14T15:20:52.977894+0000","last_became_peered":"2024-04-14T15:20:52.977894+0000","last_unstale":"2024-04-14T15:25:59.027555+0000","last_undegraded":"2024-04-14T15:25:59.027555+0000","last_fullsized":"2024-04-14T15:25:59.027555+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:52.667467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:25:52.667467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:19.080509+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.17","version":"479'12","reported_seq":1558,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.047510+0000","last_change":"2024-04-14T15:24:16.798489+0000","last_active":"2024-04-14T15:25:59.047510+0000","last_peered":"2024-04-14T15:25:59.047510+0000","last_clean":"2024-04-14T15:25:59.047510+0000","last_became_active":"2024-04-14T15:09:45.180037+0000","last_became_peered":"2024-04-14T15:09:45.180037+0000","last_unstale":"2024-04-14T15:25:59.047510+0000","last_undegraded":"2024-04-14T15:25:59.047510+0000","last_fullsized":"2024-04-14T15:25:59.047510+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'12","last_scrub_stamp":"2024-04-14T15:24:16.798377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:24:16.798377+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":582,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.027214+0000","last_change":"2024-04-14T15:23:40.765638+0000","last_active":"2024-04-14T15:25:59.027214+0000","last_peered":"2024-04-14T15:25:59.027214+0000","last_clean":"2024-04-14T15:25:59.027214+0000","last_became_active":"2024-04-14T15:20:52.986974+0000","last_became_peered":"2024-04-14T15:20:52.986974+0000","last_unstale":"2024-04-14T15:25:59.027214+0000","last_undegraded":"2024-04-14T15:25:59.027214+0000","last_fullsized":"2024-04-14T15:25:59.027214+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:40.765537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:59.640565+0000","last_clean_scrub_stamp":"2024-04-14T15:23:40.765537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1095,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.046736+0000","last_change":"2024-04-14T15:24:18.710723+0000","last_active":"2024-04-14T15:25:59.046736+0000","last_peered":"2024-04-14T15:25:59.046736+0000","last_clean":"2024-04-14T15:25:59.046736+0000","last_became_active":"2024-04-14T15:15:44.889786+0000","last_became_peered":"2024-04-14T15:15:44.889786+0000","last_unstale":"2024-04-14T15:25:59.046736+0000","last_undegraded":"2024-04-14T15:25:59.046736+0000","last_fullsized":"2024-04-14T15:25:59.046736+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:18.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:13.735024+0000","last_clean_scrub_stamp":"2024-04-14T15:24:18.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1535,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.039579+0000","last_change":"2024-04-14T15:23:29.527743+0000","last_active":"2024-04-14T15:25:59.039579+0000","last_peered":"2024-04-14T15:25:59.039579+0000","last_clean":"2024-04-14T15:25:59.039579+0000","last_became_active":"2024-04-14T15:20:52.985136+0000","last_became_peered":"2024-04-14T15:20:52.985136+0000","last_unstale":"2024-04-14T15:25:59.039579+0000","last_undegraded":"2024-04-14T15:25:59.039579+0000","last_fullsized":"2024-04-14T15:25:59.039579+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:29.527656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:23:29.527656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.11","version":"91'130","reported_seq":1536,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.032825+0000","last_change":"2024-04-14T15:25:02.713875+0000","last_active":"2024-04-14T15:25:59.032825+0000","last_peered":"2024-04-14T15:25:59.032825+0000","last_clean":"2024-04-14T15:25:59.032825+0000","last_became_active":"2024-04-14T15:25:02.030974+0000","last_became_peered":"2024-04-14T15:25:02.030974+0000","last_unstale":"2024-04-14T15:25:59.032825+0000","last_undegraded":"2024-04-14T15:25:59.032825+0000","last_fullsized":"2024-04-14T15:25:59.032825+0000","mapping_epoch":757,"log_start":"91'130","ondisk_log_start":"91'130","created":752,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:25:02.713791+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:25:02.713791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:32.110890+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":1454,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.027686+0000","last_change":"2024-04-14T15:25:50.325506+0000","last_active":"2024-04-14T15:25:59.027686+0000","last_peered":"2024-04-14T15:25:59.027686+0000","last_clean":"2024-04-14T15:25:59.027686+0000","last_became_active":"2024-04-14T15:12:56.203577+0000","last_became_peered":"2024-04-14T15:12:56.203577+0000","last_unstale":"2024-04-14T15:25:59.027686+0000","last_undegraded":"2024-04-14T15:25:59.027686+0000","last_fullsized":"2024-04-14T15:25:59.027686+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:50.325429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:46.369231+0000","last_clean_scrub_stamp":"2024-04-14T15:25:50.325429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:50.471569+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"479'20","reported_seq":1255,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.046039+0000","last_change":"2024-04-14T15:25:03.783837+0000","last_active":"2024-04-14T15:25:59.046039+0000","last_peered":"2024-04-14T15:25:59.046039+0000","last_clean":"2024-04-14T15:25:59.046039+0000","last_became_active":"2024-04-14T15:15:21.877667+0000","last_became_peered":"2024-04-14T15:15:21.877667+0000","last_unstale":"2024-04-14T15:25:59.046039+0000","last_undegraded":"2024-04-14T15:25:59.046039+0000","last_fullsized":"2024-04-14T15:25:59.046039+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'20","last_scrub_stamp":"2024-04-14T15:25:03.783721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:25:03.783721+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:07.946207+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":1103,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.045737+0000","last_change":"2024-04-14T15:25:57.772181+0000","last_active":"2024-04-14T15:25:59.045737+0000","last_peered":"2024-04-14T15:25:59.045737+0000","last_clean":"2024-04-14T15:25:59.045737+0000","last_became_active":"2024-04-14T15:15:44.887452+0000","last_became_peered":"2024-04-14T15:15:44.887452+0000","last_unstale":"2024-04-14T15:25:59.045737+0000","last_undegraded":"2024-04-14T15:25:59.045737+0000","last_fullsized":"2024-04-14T15:25:59.045737+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:57.772100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:25:57.772100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:01.749540+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"280'9","reported_seq":1252,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.030829+0000","last_change":"2024-04-14T15:25:13.639711+0000","last_active":"2024-04-14T15:25:59.030829+0000","last_peered":"2024-04-14T15:25:59.030829+0000","last_clean":"2024-04-14T15:25:59.030829+0000","last_became_active":"2024-04-14T15:15:44.882916+0000","last_became_peered":"2024-04-14T15:15:44.882916+0000","last_unstale":"2024-04-14T15:25:59.030829+0000","last_undegraded":"2024-04-14T15:25:59.030829+0000","last_fullsized":"2024-04-14T15:25:59.030829+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"280'9","last_scrub_stamp":"2024-04-14T15:25:13.639579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:25:13.639579+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:26.089142+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1536,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.034325+0000","last_change":"2024-04-14T15:25:54.151674+0000","last_active":"2024-04-14T15:25:59.034325+0000","last_peered":"2024-04-14T15:25:59.034325+0000","last_clean":"2024-04-14T15:25:59.034325+0000","last_became_active":"2024-04-14T15:09:49.066068+0000","last_became_peered":"2024-04-14T15:09:49.066068+0000","last_unstale":"2024-04-14T15:25:59.034325+0000","last_undegraded":"2024-04-14T15:25:59.034325+0000","last_fullsized":"2024-04-14T15:25:59.034325+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:54.151593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:54.724681+0000","last_clean_scrub_stamp":"2024-04-14T15:25:54.151593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:55.084660+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"91'130","reported_seq":1921,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.037270+0000","last_change":"2024-04-14T15:25:48.929066+0000","last_active":"2024-04-14T15:25:59.037270+0000","last_peered":"2024-04-14T15:25:59.037270+0000","last_clean":"2024-04-14T15:25:59.037270+0000","last_became_active":"2024-04-14T15:25:32.027404+0000","last_became_peered":"2024-04-14T15:25:32.027404+0000","last_unstale":"2024-04-14T15:25:59.037270+0000","last_undegraded":"2024-04-14T15:25:59.037270+0000","last_fullsized":"2024-04-14T15:25:59.037270+0000","mapping_epoch":787,"log_start":"91'130","ondisk_log_start":"91'130","created":752,"last_epoch_clean":788,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:25:48.928929+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:25:48.928929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:01.358105+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":1095,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.045462+0000","last_change":"2024-04-14T15:24:26.878660+0000","last_active":"2024-04-14T15:25:59.045462+0000","last_peered":"2024-04-14T15:25:59.045462+0000","last_clean":"2024-04-14T15:25:59.045462+0000","last_became_active":"2024-04-14T15:15:44.889106+0000","last_became_peered":"2024-04-14T15:15:44.889106+0000","last_unstale":"2024-04-14T15:25:59.045462+0000","last_undegraded":"2024-04-14T15:25:59.045462+0000","last_fullsized":"2024-04-14T15:25:59.045462+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:26.878584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:24:26.878584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"479'19","reported_seq":1404,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.038800+0000","last_change":"2024-04-14T15:23:46.548415+0000","last_active":"2024-04-14T15:25:59.038800+0000","last_peered":"2024-04-14T15:25:59.038800+0000","last_clean":"2024-04-14T15:25:59.038800+0000","last_became_active":"2024-04-14T15:22:25.971104+0000","last_became_peered":"2024-04-14T15:22:25.971104+0000","last_unstale":"2024-04-14T15:25:59.038800+0000","last_undegraded":"2024-04-14T15:25:59.038800+0000","last_fullsized":"2024-04-14T15:25:59.038800+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'19","last_scrub_stamp":"2024-04-14T15:23:46.548255+0000","last_deep_scrub":"183'13","last_deep_scrub_stamp":"2024-04-14T15:14:11.755252+0000","last_clean_scrub_stamp":"2024-04-14T15:23:46.548255+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1556,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.033350+0000","last_change":"2024-04-14T15:25:48.143955+0000","last_active":"2024-04-14T15:25:59.033350+0000","last_peered":"2024-04-14T15:25:59.033350+0000","last_clean":"2024-04-14T15:25:59.033350+0000","last_became_active":"2024-04-14T15:09:43.019301+0000","last_became_peered":"2024-04-14T15:09:43.019301+0000","last_unstale":"2024-04-14T15:25:59.033350+0000","last_undegraded":"2024-04-14T15:25:59.033350+0000","last_fullsized":"2024-04-14T15:25:59.033350+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:48.143875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:25:48.143875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:13.335549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"255'10","reported_seq":1388,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.046301+0000","last_change":"2024-04-14T15:25:47.681818+0000","last_active":"2024-04-14T15:25:59.046301+0000","last_peered":"2024-04-14T15:25:59.046301+0000","last_clean":"2024-04-14T15:25:59.046301+0000","last_became_active":"2024-04-14T15:15:21.884468+0000","last_became_peered":"2024-04-14T15:15:21.884468+0000","last_unstale":"2024-04-14T15:25:59.046301+0000","last_undegraded":"2024-04-14T15:25:59.046301+0000","last_fullsized":"2024-04-14T15:25:59.046301+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'10","last_scrub_stamp":"2024-04-14T15:25:47.681710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:25:47.681710+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:48.424176+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1529,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.028529+0000","last_change":"2024-04-14T15:24:23.473852+0000","last_active":"2024-04-14T15:25:59.028529+0000","last_peered":"2024-04-14T15:25:59.028529+0000","last_clean":"2024-04-14T15:25:59.028529+0000","last_became_active":"2024-04-14T15:09:49.065242+0000","last_became_peered":"2024-04-14T15:09:49.065242+0000","last_unstale":"2024-04-14T15:25:59.028529+0000","last_undegraded":"2024-04-14T15:25:59.028529+0000","last_fullsized":"2024-04-14T15:25:59.028529+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:23.473749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:24:23.473749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":1446,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.039615+0000","last_change":"2024-04-14T15:25:08.869842+0000","last_active":"2024-04-14T15:25:59.039615+0000","last_peered":"2024-04-14T15:25:59.039615+0000","last_clean":"2024-04-14T15:25:59.039615+0000","last_became_active":"2024-04-14T15:12:56.389204+0000","last_became_peered":"2024-04-14T15:12:56.389204+0000","last_unstale":"2024-04-14T15:25:59.039615+0000","last_undegraded":"2024-04-14T15:25:59.039615+0000","last_fullsized":"2024-04-14T15:25:59.039615+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:08.869762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:25:08.869762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:26.152506+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"479'19","reported_seq":1006,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.029013+0000","last_change":"2024-04-14T15:24:24.516920+0000","last_active":"2024-04-14T15:25:59.029013+0000","last_peered":"2024-04-14T15:25:59.029013+0000","last_clean":"2024-04-14T15:25:59.029013+0000","last_became_active":"2024-04-14T15:20:52.990622+0000","last_became_peered":"2024-04-14T15:20:52.990622+0000","last_unstale":"2024-04-14T15:25:59.029013+0000","last_undegraded":"2024-04-14T15:25:59.029013+0000","last_fullsized":"2024-04-14T15:25:59.029013+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'19","last_scrub_stamp":"2024-04-14T15:24:24.516820+0000","last_deep_scrub":"344'13","last_deep_scrub_stamp":"2024-04-14T15:16:52.738163+0000","last_clean_scrub_stamp":"2024-04-14T15:24:24.516820+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":587,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.046091+0000","last_change":"2024-04-14T15:25:22.828103+0000","last_active":"2024-04-14T15:25:59.046091+0000","last_peered":"2024-04-14T15:25:59.046091+0000","last_clean":"2024-04-14T15:25:59.046091+0000","last_became_active":"2024-04-14T15:20:52.996548+0000","last_became_peered":"2024-04-14T15:20:52.996548+0000","last_unstale":"2024-04-14T15:25:59.046091+0000","last_undegraded":"2024-04-14T15:25:59.046091+0000","last_fullsized":"2024-04-14T15:25:59.046091+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:22.828025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:25:22.828025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:30.255807+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1213,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.046493+0000","last_change":"2024-04-14T15:24:31.955026+0000","last_active":"2024-04-14T15:25:59.046493+0000","last_peered":"2024-04-14T15:25:59.046493+0000","last_clean":"2024-04-14T15:25:59.046493+0000","last_became_active":"2024-04-14T15:15:21.882643+0000","last_became_peered":"2024-04-14T15:15:21.882643+0000","last_unstale":"2024-04-14T15:25:59.046493+0000","last_undegraded":"2024-04-14T15:25:59.046493+0000","last_fullsized":"2024-04-14T15:25:59.046493+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:31.954940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:33.485411+0000","last_clean_scrub_stamp":"2024-04-14T15:24:31.954940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:00.985315+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1539,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.033156+0000","last_change":"2024-04-14T15:24:22.395640+0000","last_active":"2024-04-14T15:25:59.033156+0000","last_peered":"2024-04-14T15:25:59.033156+0000","last_clean":"2024-04-14T15:25:59.033156+0000","last_became_active":"2024-04-14T15:09:49.059711+0000","last_became_peered":"2024-04-14T15:09:49.059711+0000","last_unstale":"2024-04-14T15:25:59.033156+0000","last_undegraded":"2024-04-14T15:25:59.033156+0000","last_fullsized":"2024-04-14T15:25:59.033156+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:22.395562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:24:22.395562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":596,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.038486+0000","last_change":"2024-04-14T15:25:57.008522+0000","last_active":"2024-04-14T15:25:59.038486+0000","last_peered":"2024-04-14T15:25:59.038486+0000","last_clean":"2024-04-14T15:25:59.038486+0000","last_became_active":"2024-04-14T15:22:26.203388+0000","last_became_peered":"2024-04-14T15:22:26.203388+0000","last_unstale":"2024-04-14T15:25:59.038486+0000","last_undegraded":"2024-04-14T15:25:59.038486+0000","last_fullsized":"2024-04-14T15:25:59.038486+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:57.008388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:25:57.008388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:18.120983+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"479'24","reported_seq":1591,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.027791+0000","last_change":"2024-04-14T15:24:15.604196+0000","last_active":"2024-04-14T15:25:59.027791+0000","last_peered":"2024-04-14T15:25:59.027791+0000","last_clean":"2024-04-14T15:25:59.027791+0000","last_became_active":"2024-04-14T15:20:52.989300+0000","last_became_peered":"2024-04-14T15:20:52.989300+0000","last_unstale":"2024-04-14T15:25:59.027791+0000","last_undegraded":"2024-04-14T15:25:59.027791+0000","last_fullsized":"2024-04-14T15:25:59.027791+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'24","last_scrub_stamp":"2024-04-14T15:24:15.604065+0000","last_deep_scrub":"464'16","last_deep_scrub_stamp":"2024-04-14T15:19:17.589999+0000","last_clean_scrub_stamp":"2024-04-14T15:24:15.604065+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":1553,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.031017+0000","last_change":"2024-04-14T15:24:33.369216+0000","last_active":"2024-04-14T15:25:59.031017+0000","last_peered":"2024-04-14T15:25:59.031017+0000","last_clean":"2024-04-14T15:25:59.031017+0000","last_became_active":"2024-04-14T15:09:43.023583+0000","last_became_peered":"2024-04-14T15:09:43.023583+0000","last_unstale":"2024-04-14T15:25:59.031017+0000","last_undegraded":"2024-04-14T15:25:59.031017+0000","last_fullsized":"2024-04-14T15:25:59.031017+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:33.369110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:24:33.369110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1537,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.029316+0000","last_change":"2024-04-14T15:24:19.367902+0000","last_active":"2024-04-14T15:25:59.029316+0000","last_peered":"2024-04-14T15:25:59.029316+0000","last_clean":"2024-04-14T15:25:59.029316+0000","last_became_active":"2024-04-14T15:09:47.041683+0000","last_became_peered":"2024-04-14T15:09:47.041683+0000","last_unstale":"2024-04-14T15:25:59.029316+0000","last_undegraded":"2024-04-14T15:25:59.029316+0000","last_fullsized":"2024-04-14T15:25:59.029316+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:19.367758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:08.486217+0000","last_clean_scrub_stamp":"2024-04-14T15:24:19.367758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1217,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.045465+0000","last_change":"2024-04-14T15:25:58.808326+0000","last_active":"2024-04-14T15:25:59.045465+0000","last_peered":"2024-04-14T15:25:59.045465+0000","last_clean":"2024-04-14T15:25:59.045465+0000","last_became_active":"2024-04-14T15:15:21.889567+0000","last_became_peered":"2024-04-14T15:15:21.889567+0000","last_unstale":"2024-04-14T15:25:59.045465+0000","last_undegraded":"2024-04-14T15:25:59.045465+0000","last_fullsized":"2024-04-14T15:25:59.045465+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:58.808247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:25:58.808247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:06.061451+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":1452,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.030178+0000","last_change":"2024-04-14T15:24:31.508385+0000","last_active":"2024-04-14T15:25:59.030178+0000","last_peered":"2024-04-14T15:25:59.030178+0000","last_clean":"2024-04-14T15:25:59.030178+0000","last_became_active":"2024-04-14T15:12:56.387723+0000","last_became_peered":"2024-04-14T15:12:56.387723+0000","last_unstale":"2024-04-14T15:25:59.030178+0000","last_undegraded":"2024-04-14T15:25:59.030178+0000","last_fullsized":"2024-04-14T15:25:59.030178+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:31.508252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:03.685045+0000","last_clean_scrub_stamp":"2024-04-14T15:24:31.508252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.10","version":"479'8","reported_seq":1564,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.045284+0000","last_change":"2024-04-14T15:24:24.848430+0000","last_active":"2024-04-14T15:25:59.045284+0000","last_peered":"2024-04-14T15:25:59.045284+0000","last_clean":"2024-04-14T15:25:59.045284+0000","last_became_active":"2024-04-14T15:20:53.001300+0000","last_became_peered":"2024-04-14T15:20:53.001300+0000","last_unstale":"2024-04-14T15:25:59.045284+0000","last_undegraded":"2024-04-14T15:25:59.045284+0000","last_fullsized":"2024-04-14T15:25:59.045284+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'8","last_scrub_stamp":"2024-04-14T15:24:24.848331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:24:24.848331+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":591,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.030535+0000","last_change":"2024-04-14T15:25:50.711121+0000","last_active":"2024-04-14T15:25:59.030535+0000","last_peered":"2024-04-14T15:25:59.030535+0000","last_clean":"2024-04-14T15:25:59.030535+0000","last_became_active":"2024-04-14T15:20:52.979496+0000","last_became_peered":"2024-04-14T15:20:52.979496+0000","last_unstale":"2024-04-14T15:25:59.030535+0000","last_undegraded":"2024-04-14T15:25:59.030535+0000","last_fullsized":"2024-04-14T15:25:59.030535+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:50.711037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:08.712289+0000","last_clean_scrub_stamp":"2024-04-14T15:25:50.711037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:20.652725+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":582,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.027114+0000","last_change":"2024-04-14T15:24:30.420853+0000","last_active":"2024-04-14T15:25:59.027114+0000","last_peered":"2024-04-14T15:25:59.027114+0000","last_clean":"2024-04-14T15:25:59.027114+0000","last_became_active":"2024-04-14T15:20:52.990347+0000","last_became_peered":"2024-04-14T15:20:52.990347+0000","last_unstale":"2024-04-14T15:25:59.027114+0000","last_undegraded":"2024-04-14T15:25:59.027114+0000","last_fullsized":"2024-04-14T15:25:59.027114+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:30.420762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:24:30.420762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1529,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.037914+0000","last_change":"2024-04-14T15:24:18.506706+0000","last_active":"2024-04-14T15:25:59.037914+0000","last_peered":"2024-04-14T15:25:59.037914+0000","last_clean":"2024-04-14T15:25:59.037914+0000","last_became_active":"2024-04-14T15:09:49.061359+0000","last_became_peered":"2024-04-14T15:09:49.061359+0000","last_unstale":"2024-04-14T15:25:59.037914+0000","last_undegraded":"2024-04-14T15:25:59.037914+0000","last_fullsized":"2024-04-14T15:25:59.037914+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:18.506553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:24:18.506553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"91'130","reported_seq":1709,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.028463+0000","last_change":"2024-04-14T15:25:02.619287+0000","last_active":"2024-04-14T15:25:59.028463+0000","last_peered":"2024-04-14T15:25:59.028463+0000","last_clean":"2024-04-14T15:25:59.028463+0000","last_became_active":"2024-04-14T15:25:02.033333+0000","last_became_peered":"2024-04-14T15:25:02.033333+0000","last_unstale":"2024-04-14T15:25:59.028463+0000","last_undegraded":"2024-04-14T15:25:59.028463+0000","last_fullsized":"2024-04-14T15:25:59.028463+0000","mapping_epoch":757,"log_start":"91'130","ondisk_log_start":"91'130","created":752,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:25:02.619210+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:25:02.619210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":60,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:19.530117+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":1453,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.038584+0000","last_change":"2024-04-14T15:25:22.844265+0000","last_active":"2024-04-14T15:25:59.038584+0000","last_peered":"2024-04-14T15:25:59.038584+0000","last_clean":"2024-04-14T15:25:59.038584+0000","last_became_active":"2024-04-14T15:20:52.999623+0000","last_became_peered":"2024-04-14T15:20:52.999623+0000","last_unstale":"2024-04-14T15:25:59.038584+0000","last_undegraded":"2024-04-14T15:25:59.038584+0000","last_fullsized":"2024-04-14T15:25:59.038584+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:22.844179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:15:43.680852+0000","last_clean_scrub_stamp":"2024-04-14T15:25:22.844179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:44.664180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"479'24","reported_seq":1232,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.045939+0000","last_change":"2024-04-14T15:24:29.941597+0000","last_active":"2024-04-14T15:25:59.045939+0000","last_peered":"2024-04-14T15:25:59.045939+0000","last_clean":"2024-04-14T15:25:59.045939+0000","last_became_active":"2024-04-14T15:20:52.996917+0000","last_became_peered":"2024-04-14T15:20:52.996917+0000","last_unstale":"2024-04-14T15:25:59.045939+0000","last_undegraded":"2024-04-14T15:25:59.045939+0000","last_fullsized":"2024-04-14T15:25:59.045939+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'24","last_scrub_stamp":"2024-04-14T15:24:29.941474+0000","last_deep_scrub":"102'16","last_deep_scrub_stamp":"2024-04-14T15:12:35.999326+0000","last_clean_scrub_stamp":"2024-04-14T15:24:29.941474+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":588,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.030260+0000","last_change":"2024-04-14T15:24:30.484775+0000","last_active":"2024-04-14T15:25:59.030260+0000","last_peered":"2024-04-14T15:25:59.030260+0000","last_clean":"2024-04-14T15:25:59.030260+0000","last_became_active":"2024-04-14T15:20:52.970959+0000","last_became_peered":"2024-04-14T15:20:52.970959+0000","last_unstale":"2024-04-14T15:25:59.030260+0000","last_undegraded":"2024-04-14T15:25:59.030260+0000","last_fullsized":"2024-04-14T15:25:59.030260+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:30.484673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:30.484673+0000","last_clean_scrub_stamp":"2024-04-14T15:24:30.484673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1539,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.037346+0000","last_change":"2024-04-14T15:24:17.557862+0000","last_active":"2024-04-14T15:25:59.037346+0000","last_peered":"2024-04-14T15:25:59.037346+0000","last_clean":"2024-04-14T15:25:59.037346+0000","last_became_active":"2024-04-14T15:20:52.983919+0000","last_became_peered":"2024-04-14T15:20:52.983919+0000","last_unstale":"2024-04-14T15:25:59.037346+0000","last_undegraded":"2024-04-14T15:25:59.037346+0000","last_fullsized":"2024-04-14T15:25:59.037346+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:17.557766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:24:17.557766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1223,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.046606+0000","last_change":"2024-04-14T15:24:34.995365+0000","last_active":"2024-04-14T15:25:59.046606+0000","last_peered":"2024-04-14T15:25:59.046606+0000","last_clean":"2024-04-14T15:25:59.046606+0000","last_became_active":"2024-04-14T15:20:52.993918+0000","last_became_peered":"2024-04-14T15:20:52.993918+0000","last_unstale":"2024-04-14T15:25:59.046606+0000","last_undegraded":"2024-04-14T15:25:59.046606+0000","last_fullsized":"2024-04-14T15:25:59.046606+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:34.995256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:24:34.995256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":1451,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.028537+0000","last_change":"2024-04-14T15:25:46.718417+0000","last_active":"2024-04-14T15:25:59.028537+0000","last_peered":"2024-04-14T15:25:59.028537+0000","last_clean":"2024-04-14T15:25:59.028537+0000","last_became_active":"2024-04-14T15:12:56.335230+0000","last_became_peered":"2024-04-14T15:12:56.335230+0000","last_unstale":"2024-04-14T15:25:59.028537+0000","last_undegraded":"2024-04-14T15:25:59.028537+0000","last_fullsized":"2024-04-14T15:25:59.028537+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:46.718339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:25:46.718339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:52.632654+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"776'25","reported_seq":1581,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.034606+0000","last_change":"2024-04-14T15:25:21.063734+0000","last_active":"2024-04-14T15:25:59.034606+0000","last_peered":"2024-04-14T15:25:59.034606+0000","last_clean":"2024-04-14T15:25:59.034606+0000","last_became_active":"2024-04-14T15:09:45.181456+0000","last_became_peered":"2024-04-14T15:09:45.181456+0000","last_unstale":"2024-04-14T15:25:59.034606+0000","last_undegraded":"2024-04-14T15:25:59.034606+0000","last_fullsized":"2024-04-14T15:25:59.034606+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"776'25","last_scrub_stamp":"2024-04-14T15:25:21.063579+0000","last_deep_scrub":"776'25","last_deep_scrub_stamp":"2024-04-14T15:25:21.063579+0000","last_clean_scrub_stamp":"2024-04-14T15:25:21.063579+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:28.000096+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":1100,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.027099+0000","last_change":"2024-04-14T15:24:31.421036+0000","last_active":"2024-04-14T15:25:59.027099+0000","last_peered":"2024-04-14T15:25:59.027099+0000","last_clean":"2024-04-14T15:25:59.027099+0000","last_became_active":"2024-04-14T15:15:44.878336+0000","last_became_peered":"2024-04-14T15:15:44.878336+0000","last_unstale":"2024-04-14T15:25:59.027099+0000","last_undegraded":"2024-04-14T15:25:59.027099+0000","last_fullsized":"2024-04-14T15:25:59.027099+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:31.420910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:24:31.420910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"528'9","reported_seq":676,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.028616+0000","last_change":"2024-04-14T15:25:05.354926+0000","last_active":"2024-04-14T15:25:59.028616+0000","last_peered":"2024-04-14T15:25:59.028616+0000","last_clean":"2024-04-14T15:25:59.028616+0000","last_became_active":"2024-04-14T15:20:52.982112+0000","last_became_peered":"2024-04-14T15:20:52.982112+0000","last_unstale":"2024-04-14T15:25:59.028616+0000","last_undegraded":"2024-04-14T15:25:59.028616+0000","last_fullsized":"2024-04-14T15:25:59.028616+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"528'9","last_scrub_stamp":"2024-04-14T15:25:05.354849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:25:05.354849+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:22.493402+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"72'1","reported_seq":1538,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.039632+0000","last_change":"2024-04-14T15:23:32.511432+0000","last_active":"2024-04-14T15:25:59.039632+0000","last_peered":"2024-04-14T15:25:59.039632+0000","last_clean":"2024-04-14T15:25:59.039632+0000","last_became_active":"2024-04-14T15:09:49.062051+0000","last_became_peered":"2024-04-14T15:09:49.062051+0000","last_unstale":"2024-04-14T15:25:59.039632+0000","last_undegraded":"2024-04-14T15:25:59.039632+0000","last_fullsized":"2024-04-14T15:25:59.039632+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-04-14T15:23:32.511322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:23:32.511322+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":1451,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.033363+0000","last_change":"2024-04-14T15:24:29.418267+0000","last_active":"2024-04-14T15:25:59.033363+0000","last_peered":"2024-04-14T15:25:59.033363+0000","last_clean":"2024-04-14T15:25:59.033363+0000","last_became_active":"2024-04-14T15:20:52.986657+0000","last_became_peered":"2024-04-14T15:20:52.986657+0000","last_unstale":"2024-04-14T15:25:59.033363+0000","last_undegraded":"2024-04-14T15:25:59.033363+0000","last_fullsized":"2024-04-14T15:25:59.033363+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:29.418183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:29.418183+0000","last_clean_scrub_stamp":"2024-04-14T15:24:29.418183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"519'24","reported_seq":1603,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.031042+0000","last_change":"2024-04-14T15:25:55.662949+0000","last_active":"2024-04-14T15:25:59.031042+0000","last_peered":"2024-04-14T15:25:59.031042+0000","last_clean":"2024-04-14T15:25:59.031042+0000","last_became_active":"2024-04-14T15:20:52.987333+0000","last_became_peered":"2024-04-14T15:20:52.987333+0000","last_unstale":"2024-04-14T15:25:59.031042+0000","last_undegraded":"2024-04-14T15:25:59.031042+0000","last_fullsized":"2024-04-14T15:25:59.031042+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"519'24","last_scrub_stamp":"2024-04-14T15:25:55.662871+0000","last_deep_scrub":"519'24","last_deep_scrub_stamp":"2024-04-14T15:20:44.763417+0000","last_clean_scrub_stamp":"2024-04-14T15:25:55.662871+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:14.030284+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.19","version":"684'4","reported_seq":1242,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.045598+0000","last_change":"2024-04-14T15:23:46.931649+0000","last_active":"2024-04-14T15:25:59.045598+0000","last_peered":"2024-04-14T15:25:59.045598+0000","last_clean":"2024-04-14T15:25:59.045598+0000","last_became_active":"2024-04-14T15:20:52.995628+0000","last_became_peered":"2024-04-14T15:20:52.995628+0000","last_unstale":"2024-04-14T15:25:59.045598+0000","last_undegraded":"2024-04-14T15:25:59.045598+0000","last_fullsized":"2024-04-14T15:25:59.045598+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"684'4","last_scrub_stamp":"2024-04-14T15:23:46.931560+0000","last_deep_scrub":"684'4","last_deep_scrub_stamp":"2024-04-14T15:23:46.931560+0000","last_clean_scrub_stamp":"2024-04-14T15:23:46.931560+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":957,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1542,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.034610+0000","last_change":"2024-04-14T15:24:30.397659+0000","last_active":"2024-04-14T15:25:59.034610+0000","last_peered":"2024-04-14T15:25:59.034610+0000","last_clean":"2024-04-14T15:25:59.034610+0000","last_became_active":"2024-04-14T15:09:47.045730+0000","last_became_peered":"2024-04-14T15:09:47.045730+0000","last_unstale":"2024-04-14T15:25:59.034610+0000","last_undegraded":"2024-04-14T15:25:59.034610+0000","last_fullsized":"2024-04-14T15:25:59.034610+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:30.397585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:24:30.397585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1533,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.030705+0000","last_change":"2024-04-14T15:24:30.343193+0000","last_active":"2024-04-14T15:25:59.030705+0000","last_peered":"2024-04-14T15:25:59.030705+0000","last_clean":"2024-04-14T15:25:59.030705+0000","last_became_active":"2024-04-14T15:09:49.057545+0000","last_became_peered":"2024-04-14T15:09:49.057545+0000","last_unstale":"2024-04-14T15:25:59.030705+0000","last_undegraded":"2024-04-14T15:25:59.030705+0000","last_fullsized":"2024-04-14T15:25:59.030705+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:30.343106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:30.343106+0000","last_clean_scrub_stamp":"2024-04-14T15:24:30.343106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":1100,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.027630+0000","last_change":"2024-04-14T15:25:25.320811+0000","last_active":"2024-04-14T15:25:59.027630+0000","last_peered":"2024-04-14T15:25:59.027630+0000","last_clean":"2024-04-14T15:25:59.027630+0000","last_became_active":"2024-04-14T15:15:44.882604+0000","last_became_peered":"2024-04-14T15:15:44.882604+0000","last_unstale":"2024-04-14T15:25:59.027630+0000","last_undegraded":"2024-04-14T15:25:59.027630+0000","last_fullsized":"2024-04-14T15:25:59.027630+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:25.320719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:25:25.320719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:54.397193+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"479'27","reported_seq":1585,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.029631+0000","last_change":"2024-04-14T15:24:21.500258+0000","last_active":"2024-04-14T15:25:59.029631+0000","last_peered":"2024-04-14T15:25:59.029631+0000","last_clean":"2024-04-14T15:25:59.029631+0000","last_became_active":"2024-04-14T15:09:45.032887+0000","last_became_peered":"2024-04-14T15:09:45.032887+0000","last_unstale":"2024-04-14T15:25:59.029631+0000","last_undegraded":"2024-04-14T15:25:59.029631+0000","last_fullsized":"2024-04-14T15:25:59.029631+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'27","last_scrub_stamp":"2024-04-14T15:24:21.500156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:24:21.500156+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":582,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.028078+0000","last_change":"2024-04-14T15:25:16.387612+0000","last_active":"2024-04-14T15:25:59.028078+0000","last_peered":"2024-04-14T15:25:59.028078+0000","last_clean":"2024-04-14T15:25:59.028078+0000","last_became_active":"2024-04-14T15:20:52.990634+0000","last_became_peered":"2024-04-14T15:20:52.990634+0000","last_unstale":"2024-04-14T15:25:59.028078+0000","last_undegraded":"2024-04-14T15:25:59.028078+0000","last_fullsized":"2024-04-14T15:25:59.028078+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:16.387432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:42.754721+0000","last_clean_scrub_stamp":"2024-04-14T15:25:16.387432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:27.097015+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1545,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.030931+0000","last_change":"2024-04-14T15:25:22.689804+0000","last_active":"2024-04-14T15:25:59.030931+0000","last_peered":"2024-04-14T15:25:59.030931+0000","last_clean":"2024-04-14T15:25:59.030931+0000","last_became_active":"2024-04-14T15:09:47.042733+0000","last_became_peered":"2024-04-14T15:09:47.042733+0000","last_unstale":"2024-04-14T15:25:59.030931+0000","last_undegraded":"2024-04-14T15:25:59.030931+0000","last_fullsized":"2024-04-14T15:25:59.030931+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:22.689718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:25:22.689718+0000","last_clean_scrub_stamp":"2024-04-14T15:25:22.689718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:46.132955+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1545,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.028377+0000","last_change":"2024-04-14T15:25:48.252911+0000","last_active":"2024-04-14T15:25:59.028377+0000","last_peered":"2024-04-14T15:25:59.028377+0000","last_clean":"2024-04-14T15:25:59.028377+0000","last_became_active":"2024-04-14T15:20:52.998253+0000","last_became_peered":"2024-04-14T15:20:52.998253+0000","last_unstale":"2024-04-14T15:25:59.028377+0000","last_undegraded":"2024-04-14T15:25:59.028377+0000","last_fullsized":"2024-04-14T15:25:59.028377+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:48.252835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:54.453174+0000","last_clean_scrub_stamp":"2024-04-14T15:25:48.252835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:16.009083+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":1446,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.037627+0000","last_change":"2024-04-14T15:25:24.895474+0000","last_active":"2024-04-14T15:25:59.037627+0000","last_peered":"2024-04-14T15:25:59.037627+0000","last_clean":"2024-04-14T15:25:59.037627+0000","last_became_active":"2024-04-14T15:12:56.392352+0000","last_became_peered":"2024-04-14T15:12:56.392352+0000","last_unstale":"2024-04-14T15:25:59.037627+0000","last_undegraded":"2024-04-14T15:25:59.037627+0000","last_fullsized":"2024-04-14T15:25:59.037627+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:24.895381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:25:24.895381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:30.449450+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"479'18","reported_seq":1257,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.046349+0000","last_change":"2024-04-14T15:25:07.782722+0000","last_active":"2024-04-14T15:25:59.046349+0000","last_peered":"2024-04-14T15:25:59.046349+0000","last_clean":"2024-04-14T15:25:59.046349+0000","last_became_active":"2024-04-14T15:15:21.877838+0000","last_became_peered":"2024-04-14T15:15:21.877838+0000","last_unstale":"2024-04-14T15:25:59.046349+0000","last_undegraded":"2024-04-14T15:25:59.046349+0000","last_fullsized":"2024-04-14T15:25:59.046349+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'18","last_scrub_stamp":"2024-04-14T15:25:07.782608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:25:07.782608+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:18.348588+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":1235,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.046538+0000","last_change":"2024-04-14T15:25:46.654660+0000","last_active":"2024-04-14T15:25:59.046538+0000","last_peered":"2024-04-14T15:25:59.046538+0000","last_clean":"2024-04-14T15:25:59.046538+0000","last_became_active":"2024-04-14T15:20:53.001534+0000","last_became_peered":"2024-04-14T15:20:53.001534+0000","last_unstale":"2024-04-14T15:25:59.046538+0000","last_undegraded":"2024-04-14T15:25:59.046538+0000","last_fullsized":"2024-04-14T15:25:59.046538+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:46.654578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:25:46.654578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:10.485147+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1541,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.027729+0000","last_change":"2024-04-14T15:25:14.382875+0000","last_active":"2024-04-14T15:25:59.027729+0000","last_peered":"2024-04-14T15:25:59.027729+0000","last_clean":"2024-04-14T15:25:59.027729+0000","last_became_active":"2024-04-14T15:09:47.048045+0000","last_became_peered":"2024-04-14T15:09:47.048045+0000","last_unstale":"2024-04-14T15:25:59.027729+0000","last_undegraded":"2024-04-14T15:25:59.027729+0000","last_fullsized":"2024-04-14T15:25:59.027729+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:14.382783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:25:14.382783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:25.959641+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1225,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.046872+0000","last_change":"2024-04-14T15:25:09.737733+0000","last_active":"2024-04-14T15:25:59.046872+0000","last_peered":"2024-04-14T15:25:59.046872+0000","last_clean":"2024-04-14T15:25:59.046872+0000","last_became_active":"2024-04-14T15:20:52.998775+0000","last_became_peered":"2024-04-14T15:20:52.998775+0000","last_unstale":"2024-04-14T15:25:59.046872+0000","last_undegraded":"2024-04-14T15:25:59.046872+0000","last_fullsized":"2024-04-14T15:25:59.046872+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:09.737658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:22:20.946524+0000","last_clean_scrub_stamp":"2024-04-14T15:25:09.737658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:24.666319+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":1451,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.029999+0000","last_change":"2024-04-14T15:25:19.668735+0000","last_active":"2024-04-14T15:25:59.029999+0000","last_peered":"2024-04-14T15:25:59.029999+0000","last_clean":"2024-04-14T15:25:59.029999+0000","last_became_active":"2024-04-14T15:12:56.375278+0000","last_became_peered":"2024-04-14T15:12:56.375278+0000","last_unstale":"2024-04-14T15:25:59.029999+0000","last_undegraded":"2024-04-14T15:25:59.029999+0000","last_fullsized":"2024-04-14T15:25:59.029999+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:19.668662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:25:19.668662+0000","last_clean_scrub_stamp":"2024-04-14T15:25:19.668662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:28.031137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"479'19","reported_seq":1574,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.027501+0000","last_change":"2024-04-14T15:25:21.286947+0000","last_active":"2024-04-14T15:25:59.027501+0000","last_peered":"2024-04-14T15:25:59.027501+0000","last_clean":"2024-04-14T15:25:59.027501+0000","last_became_active":"2024-04-14T15:09:45.030093+0000","last_became_peered":"2024-04-14T15:09:45.030093+0000","last_unstale":"2024-04-14T15:25:59.027501+0000","last_undegraded":"2024-04-14T15:25:59.027501+0000","last_fullsized":"2024-04-14T15:25:59.027501+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'19","last_scrub_stamp":"2024-04-14T15:25:21.286844+0000","last_deep_scrub":"276'13","last_deep_scrub_stamp":"2024-04-14T15:15:43.969775+0000","last_clean_scrub_stamp":"2024-04-14T15:25:21.286844+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:31.501208+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":1247,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.038362+0000","last_change":"2024-04-14T15:24:26.465423+0000","last_active":"2024-04-14T15:25:59.038362+0000","last_peered":"2024-04-14T15:25:59.038362+0000","last_clean":"2024-04-14T15:25:59.038362+0000","last_became_active":"2024-04-14T15:14:30.857727+0000","last_became_peered":"2024-04-14T15:14:30.857727+0000","last_unstale":"2024-04-14T15:25:59.038362+0000","last_undegraded":"2024-04-14T15:25:59.038362+0000","last_fullsized":"2024-04-14T15:25:59.038362+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:26.465343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:00.650062+0000","last_clean_scrub_stamp":"2024-04-14T15:24:26.465343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1107,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.033621+0000","last_change":"2024-04-14T15:24:35.503356+0000","last_active":"2024-04-14T15:25:59.033621+0000","last_peered":"2024-04-14T15:25:59.033621+0000","last_clean":"2024-04-14T15:25:59.033621+0000","last_became_active":"2024-04-14T15:15:44.883121+0000","last_became_peered":"2024-04-14T15:15:44.883121+0000","last_unstale":"2024-04-14T15:25:59.033621+0000","last_undegraded":"2024-04-14T15:25:59.033621+0000","last_fullsized":"2024-04-14T15:25:59.033621+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:35.503279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:24:35.503279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1544,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.033707+0000","last_change":"2024-04-14T15:25:22.008490+0000","last_active":"2024-04-14T15:25:59.033707+0000","last_peered":"2024-04-14T15:25:59.033707+0000","last_clean":"2024-04-14T15:25:59.033707+0000","last_became_active":"2024-04-14T15:09:49.061756+0000","last_became_peered":"2024-04-14T15:09:49.061756+0000","last_unstale":"2024-04-14T15:25:59.033707+0000","last_undegraded":"2024-04-14T15:25:59.033707+0000","last_fullsized":"2024-04-14T15:25:59.033707+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:22.008411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:25:22.008411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:43.085533+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":582,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.027575+0000","last_change":"2024-04-14T15:24:24.522250+0000","last_active":"2024-04-14T15:25:59.027575+0000","last_peered":"2024-04-14T15:25:59.027575+0000","last_clean":"2024-04-14T15:25:59.027575+0000","last_became_active":"2024-04-14T15:20:52.990838+0000","last_became_peered":"2024-04-14T15:20:52.990838+0000","last_unstale":"2024-04-14T15:25:59.027575+0000","last_undegraded":"2024-04-14T15:25:59.027575+0000","last_fullsized":"2024-04-14T15:25:59.027575+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:24.522164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:24:24.522164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"479'9","reported_seq":1560,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.027909+0000","last_change":"2024-04-14T15:25:53.355715+0000","last_active":"2024-04-14T15:25:59.027909+0000","last_peered":"2024-04-14T15:25:59.027909+0000","last_clean":"2024-04-14T15:25:59.027909+0000","last_became_active":"2024-04-14T15:09:45.028667+0000","last_became_peered":"2024-04-14T15:09:45.028667+0000","last_unstale":"2024-04-14T15:25:59.027909+0000","last_undegraded":"2024-04-14T15:25:59.027909+0000","last_fullsized":"2024-04-14T15:25:59.027909+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'9","last_scrub_stamp":"2024-04-14T15:25:53.355630+0000","last_deep_scrub":"87'7","last_deep_scrub_stamp":"2024-04-14T15:11:10.510785+0000","last_clean_scrub_stamp":"2024-04-14T15:25:53.355630+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:55.830404+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":1029,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.029590+0000","last_change":"2024-04-14T15:25:07.585542+0000","last_active":"2024-04-14T15:25:59.029590+0000","last_peered":"2024-04-14T15:25:59.029590+0000","last_clean":"2024-04-14T15:25:59.029590+0000","last_became_active":"2024-04-14T15:20:52.989307+0000","last_became_peered":"2024-04-14T15:20:52.989307+0000","last_unstale":"2024-04-14T15:25:59.029590+0000","last_undegraded":"2024-04-14T15:25:59.029590+0000","last_fullsized":"2024-04-14T15:25:59.029590+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:07.585464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:22:15.944005+0000","last_clean_scrub_stamp":"2024-04-14T15:25:07.585464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:35.619340+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1541,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.037843+0000","last_change":"2024-04-14T15:25:52.924634+0000","last_active":"2024-04-14T15:25:59.037843+0000","last_peered":"2024-04-14T15:25:59.037843+0000","last_clean":"2024-04-14T15:25:59.037843+0000","last_became_active":"2024-04-14T15:09:47.046517+0000","last_became_peered":"2024-04-14T15:09:47.046517+0000","last_unstale":"2024-04-14T15:25:59.037843+0000","last_undegraded":"2024-04-14T15:25:59.037843+0000","last_fullsized":"2024-04-14T15:25:59.037843+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:52.924559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:25:52.924559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:02.028028+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1105,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.028147+0000","last_change":"2024-04-14T15:25:28.774275+0000","last_active":"2024-04-14T15:25:59.028147+0000","last_peered":"2024-04-14T15:25:59.028147+0000","last_clean":"2024-04-14T15:25:59.028147+0000","last_became_active":"2024-04-14T15:15:44.887227+0000","last_became_peered":"2024-04-14T15:15:44.887227+0000","last_unstale":"2024-04-14T15:25:59.028147+0000","last_undegraded":"2024-04-14T15:25:59.028147+0000","last_fullsized":"2024-04-14T15:25:59.028147+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:28.774199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:56.268086+0000","last_clean_scrub_stamp":"2024-04-14T15:25:28.774199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:30.928823+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":1452,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.038714+0000","last_change":"2024-04-14T15:25:09.858241+0000","last_active":"2024-04-14T15:25:59.038714+0000","last_peered":"2024-04-14T15:25:59.038714+0000","last_clean":"2024-04-14T15:25:59.038714+0000","last_became_active":"2024-04-14T15:20:52.984387+0000","last_became_peered":"2024-04-14T15:20:52.984387+0000","last_unstale":"2024-04-14T15:25:59.038714+0000","last_undegraded":"2024-04-14T15:25:59.038714+0000","last_fullsized":"2024-04-14T15:25:59.038714+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:09.858143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:34.558312+0000","last_clean_scrub_stamp":"2024-04-14T15:25:09.858143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:09.861223+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.18","version":"806'19","reported_seq":1590,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.026982+0000","last_change":"2024-04-14T15:25:56.405925+0000","last_active":"2024-04-14T15:25:59.026982+0000","last_peered":"2024-04-14T15:25:59.026982+0000","last_clean":"2024-04-14T15:25:59.026982+0000","last_became_active":"2024-04-14T15:20:52.985210+0000","last_became_peered":"2024-04-14T15:20:52.985210+0000","last_unstale":"2024-04-14T15:25:59.026982+0000","last_undegraded":"2024-04-14T15:25:59.026982+0000","last_fullsized":"2024-04-14T15:25:59.026982+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"806'19","last_scrub_stamp":"2024-04-14T15:25:56.405833+0000","last_deep_scrub":"806'19","last_deep_scrub_stamp":"2024-04-14T15:25:56.405833+0000","last_clean_scrub_stamp":"2024-04-14T15:25:56.405833+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:23.474418+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":1223,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.046788+0000","last_change":"2024-04-14T15:24:21.791334+0000","last_active":"2024-04-14T15:25:59.046788+0000","last_peered":"2024-04-14T15:25:59.046788+0000","last_clean":"2024-04-14T15:25:59.046788+0000","last_became_active":"2024-04-14T15:15:21.891926+0000","last_became_peered":"2024-04-14T15:15:21.891926+0000","last_unstale":"2024-04-14T15:25:59.046788+0000","last_undegraded":"2024-04-14T15:25:59.046788+0000","last_fullsized":"2024-04-14T15:25:59.046788+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:21.791233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:24:21.791233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1537,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.027808+0000","last_change":"2024-04-14T15:24:22.302587+0000","last_active":"2024-04-14T15:25:59.027808+0000","last_peered":"2024-04-14T15:25:59.027808+0000","last_clean":"2024-04-14T15:25:59.027808+0000","last_became_active":"2024-04-14T15:09:47.041486+0000","last_became_peered":"2024-04-14T15:09:47.041486+0000","last_unstale":"2024-04-14T15:25:59.027808+0000","last_undegraded":"2024-04-14T15:25:59.027808+0000","last_fullsized":"2024-04-14T15:25:59.027808+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:22.302513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:24:22.302513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1529,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.027418+0000","last_change":"2024-04-14T15:25:08.411730+0000","last_active":"2024-04-14T15:25:59.027418+0000","last_peered":"2024-04-14T15:25:59.027418+0000","last_clean":"2024-04-14T15:25:59.027418+0000","last_became_active":"2024-04-14T15:09:49.058904+0000","last_became_peered":"2024-04-14T15:09:49.058904+0000","last_unstale":"2024-04-14T15:25:59.027418+0000","last_undegraded":"2024-04-14T15:25:59.027418+0000","last_fullsized":"2024-04-14T15:25:59.027418+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:08.411650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:25:08.411650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:11.025399+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":1100,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.038462+0000","last_change":"2024-04-14T15:25:16.796730+0000","last_active":"2024-04-14T15:25:59.038462+0000","last_peered":"2024-04-14T15:25:59.038462+0000","last_clean":"2024-04-14T15:25:59.038462+0000","last_became_active":"2024-04-14T15:15:44.889605+0000","last_became_peered":"2024-04-14T15:15:44.889605+0000","last_unstale":"2024-04-14T15:25:59.038462+0000","last_undegraded":"2024-04-14T15:25:59.038462+0000","last_fullsized":"2024-04-14T15:25:59.038462+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:16.796595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:25:16.796595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:25.693275+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"479'34","reported_seq":1275,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.045895+0000","last_change":"2024-04-14T15:25:10.762245+0000","last_active":"2024-04-14T15:25:59.045895+0000","last_peered":"2024-04-14T15:25:59.045895+0000","last_clean":"2024-04-14T15:25:59.045895+0000","last_became_active":"2024-04-14T15:15:21.884310+0000","last_became_peered":"2024-04-14T15:15:21.884310+0000","last_unstale":"2024-04-14T15:25:59.045895+0000","last_undegraded":"2024-04-14T15:25:59.045895+0000","last_fullsized":"2024-04-14T15:25:59.045895+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'34","last_scrub_stamp":"2024-04-14T15:25:10.762092+0000","last_deep_scrub":"306'22","last_deep_scrub_stamp":"2024-04-14T15:16:13.898106+0000","last_clean_scrub_stamp":"2024-04-14T15:25:10.762092+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:27.589475+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":1549,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.034482+0000","last_change":"2024-04-14T15:24:24.353115+0000","last_active":"2024-04-14T15:25:59.034482+0000","last_peered":"2024-04-14T15:25:59.034482+0000","last_clean":"2024-04-14T15:25:59.034482+0000","last_became_active":"2024-04-14T15:09:43.017351+0000","last_became_peered":"2024-04-14T15:09:43.017351+0000","last_unstale":"2024-04-14T15:25:59.034482+0000","last_undegraded":"2024-04-14T15:25:59.034482+0000","last_fullsized":"2024-04-14T15:25:59.034482+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:24.352978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:24:24.352978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1539,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.027168+0000","last_change":"2024-04-14T15:24:32.403277+0000","last_active":"2024-04-14T15:25:59.027168+0000","last_peered":"2024-04-14T15:25:59.027168+0000","last_clean":"2024-04-14T15:25:59.027168+0000","last_became_active":"2024-04-14T15:20:52.974360+0000","last_became_peered":"2024-04-14T15:20:52.974360+0000","last_unstale":"2024-04-14T15:25:59.027168+0000","last_undegraded":"2024-04-14T15:25:59.027168+0000","last_fullsized":"2024-04-14T15:25:59.027168+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:32.403077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:32.403077+0000","last_clean_scrub_stamp":"2024-04-14T15:24:32.403077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1538,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-14T15:25:59.046289+0000","last_change":"2024-04-14T15:24:27.874590+0000","last_active":"2024-04-14T15:25:59.046289+0000","last_peered":"2024-04-14T15:25:59.046289+0000","last_clean":"2024-04-14T15:25:59.046289+0000","last_became_active":"2024-04-14T15:20:52.998938+0000","last_became_peered":"2024-04-14T15:20:52.998938+0000","last_unstale":"2024-04-14T15:25:59.046289+0000","last_undegraded":"2024-04-14T15:25:59.046289+0000","last_fullsized":"2024-04-14T15:25:59.046289+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:27.874498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:27.874498+0000","last_clean_scrub_stamp":"2024-04-14T15:24:27.874498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":340,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":323,"num_pg":8,"stat_sum":{"num_bytes":4000249,"num_objects":37,"num_object_clones":1,"num_object_copies":111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":19,"num_read_kb":6,"num_write":57,"num_write_kb":3959,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6438912,"data_stored":6438912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90,"ondisk_log_size":90,"up":24,"acting":24,"num_store_stats":7},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":5,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1003,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":603786,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":7,"num_write":4,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":302941,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2109440,"data_stored":2098784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":63,"acting":63,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":967,"num_read_kb":962,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1121,"num_read_kb":878,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":1694,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":810,"ondisk_log_size":810,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103369,"num_pgs":102,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":100180,"kb_used_data":6536,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":93644972,"statfs":{"total":95995035648,"available":95892451328,"internally_reserved":0,"allocated":6692864,"data_stored":3353312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":95877578},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":210453397708,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575078,"num_pgs":101,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":124396,"kb_used_data":8544,"kb_used_omap":1,"kb_used_meta":115838,"kb_avail":93620756,"statfs":{"total":95995035648,"available":95867654144,"internally_reserved":0,"allocated":8749056,"data_stored":5374434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":118618565},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738603,"num_pgs":91,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":116360,"kb_used_data":8820,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93628792,"statfs":{"total":95995035648,"available":95875883008,"internally_reserved":0,"allocated":9031680,"data_stored":5658175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":110098887},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411554,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":114344,"kb_used_data":6808,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93630808,"statfs":{"total":95995035648,"available":95877947392,"internally_reserved":0,"allocated":6971392,"data_stored":3624889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182620,"num_pgs":103,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":92504,"kb_used_data":7172,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":93652648,"statfs":{"total":95995035648,"available":95900311552,"internally_reserved":0,"allocated":7344128,"data_stored":3960960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":87357895},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986390,"num_pgs":91,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":108036,"kb_used_data":8824,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93637116,"statfs":{"total":95995035648,"available":95884406784,"internally_reserved":0,"allocated":9035776,"data_stored":5675777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659347,"num_pgs":90,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":122640,"kb_used_data":6784,"kb_used_omap":1,"kb_used_meta":115838,"kb_avail":93622512,"statfs":{"total":95995035648,"available":95869452288,"internally_reserved":0,"allocated":6946816,"data_stored":3599296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":118618565},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2031616,"data_stored":2031616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2076672,"data_stored":2076672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":73728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":81920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2060288,"data_stored":2060288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":49152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":65536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-14T15:26:03.021 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph pg dump --format=json 2024-04-14T15:26:03.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:03.023 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:03.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:26:03.112 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-14T15:26:03.124 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:26:03.148 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:03.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:03.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:26:03.249 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:26:03.260 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:03 smithi092 ceph-mon[33966]: 11.1f scrub starts 2024-04-14T15:26:03.260 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:03 smithi092 ceph-mon[33966]: 11.1f scrub ok 2024-04-14T15:26:03.260 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:03 smithi092 ceph-mon[33966]: 6.8 scrub starts 2024-04-14T15:26:03.260 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:03 smithi092 ceph-mon[33966]: 6.8 scrub ok 2024-04-14T15:26:03.260 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:03 smithi092 ceph-mon[33966]: 1.12 scrub starts 2024-04-14T15:26:03.261 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:03 smithi092 ceph-mon[33966]: 1.12 scrub ok 2024-04-14T15:26:03.261 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:03 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-144", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:26:03.261 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:03 smithi092 ceph-mon[33966]: osdmap e813: 8 total, 8 up, 7 in 2024-04-14T15:26:03.261 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:03 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-144"}]: dispatch 2024-04-14T15:26:03.261 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:03 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-144"}]: dispatch 2024-04-14T15:26:03.274 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:03.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:03.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:26:03.288 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-14T15:26:03.376 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:26:03.400 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:03.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:03.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:26:03.440 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:03 smithi090 ceph-mon[30399]: 11.1f scrub starts 2024-04-14T15:26:03.440 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:03 smithi090 ceph-mon[30399]: 11.1f scrub ok 2024-04-14T15:26:03.440 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:03 smithi090 ceph-mon[30399]: 6.8 scrub starts 2024-04-14T15:26:03.440 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:03 smithi090 ceph-mon[30399]: 6.8 scrub ok 2024-04-14T15:26:03.440 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:03 smithi090 ceph-mon[30399]: 1.12 scrub starts 2024-04-14T15:26:03.440 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:03 smithi090 ceph-mon[30399]: 1.12 scrub ok 2024-04-14T15:26:03.440 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:03 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-144", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:26:03.440 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:03 smithi090 ceph-mon[30399]: osdmap e813: 8 total, 8 up, 7 in 2024-04-14T15:26:03.440 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:03 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-144"}]: dispatch 2024-04-14T15:26:03.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:03 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-144"}]: dispatch 2024-04-14T15:26:03.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:03 smithi090 ceph-mon[41670]: 11.1f scrub starts 2024-04-14T15:26:03.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:03 smithi090 ceph-mon[41670]: 11.1f scrub ok 2024-04-14T15:26:03.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:03 smithi090 ceph-mon[41670]: 6.8 scrub starts 2024-04-14T15:26:03.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:03 smithi090 ceph-mon[41670]: 6.8 scrub ok 2024-04-14T15:26:03.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:03 smithi090 ceph-mon[41670]: 1.12 scrub starts 2024-04-14T15:26:03.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:03 smithi090 ceph-mon[41670]: 1.12 scrub ok 2024-04-14T15:26:03.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:03 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-144", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:26:03.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:03 smithi090 ceph-mon[41670]: osdmap e813: 8 total, 8 up, 7 in 2024-04-14T15:26:03.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:03 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-144"}]: dispatch 2024-04-14T15:26:03.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:03 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-144"}]: dispatch 2024-04-14T15:26:03.502 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:26:03.525 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:26:03.529 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:03.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:03.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:26:03.572 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:26:03.631 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:26:03.655 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:03.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:03.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:26:03.757 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:26:03.780 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:03.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:03.781 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:26:03.882 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:26:03.907 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:03.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:03.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:26:03.975 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:26:03.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:03.975+0000 7f1fc028b700 1 -- 172.21.15.90:0/194669146 >> v1:172.21.15.90:6789/0 conn(0x7f1fb810b8b0 legacy=0x7f1fb810dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:03.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:03.975+0000 7f1fc028b700 1 -- 172.21.15.90:0/194669146 shutdown_connections 2024-04-14T15:26:03.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:03.975+0000 7f1fc028b700 1 -- 172.21.15.90:0/194669146 >> 172.21.15.90:0/194669146 conn(0x7f1fb80fd290 msgr2=0x7f1fb80ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:26:03.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:03.975+0000 7f1fc028b700 1 -- 172.21.15.90:0/194669146 shutdown_connections 2024-04-14T15:26:03.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:03.975+0000 7f1fc028b700 1 -- 172.21.15.90:0/194669146 wait complete. 2024-04-14T15:26:03.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:03.975+0000 7f1fc028b700 1 librados: starting msgr at 2024-04-14T15:26:03.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:03.975+0000 7f1fc028b700 1 librados: starting objecter 2024-04-14T15:26:03.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:03.975+0000 7f1fc028b700 1 Processor -- start 2024-04-14T15:26:03.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:03.975+0000 7f1fc028b700 1 -- start start 2024-04-14T15:26:03.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:03.975+0000 7f1fc028b700 1 librados: setting wanted keys 2024-04-14T15:26:03.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:03.975+0000 7f1fc028b700 1 librados: calling monclient init 2024-04-14T15:26:03.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:03.976+0000 7f1fc028b700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1fb810b500 con 0x7f1fb8103aa0 2024-04-14T15:26:03.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:03.976+0000 7f1fc028b700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1fb81a8ec0 con 0x7f1fb810b8b0 2024-04-14T15:26:03.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:03.976+0000 7f1fc028b700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1fb81aa0a0 con 0x7f1fb810f3c0 2024-04-14T15:26:03.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:03.976+0000 7f1fbe027700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f1fb8103aa0 0x7f1fb8108350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:47060/0 (socket says 172.21.15.90:47060) 2024-04-14T15:26:03.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:03.976+0000 7f1fbe027700 1 -- 172.21.15.90:0/2590482593 learned_addr learned my addr 172.21.15.90:0/2590482593 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:26:03.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:03.976+0000 7f1faf7fe700 10 client.?.objecter ms_handle_connect 0x7f1fb8103aa0 2024-04-14T15:26:03.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:03.976+0000 7f1faf7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:26:03.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:03.976+0000 7f1faf7fe700 10 client.?.objecter ms_handle_connect 0x7f1fb810f3c0 2024-04-14T15:26:03.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:03.976+0000 7f1faf7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:26:03.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:03.976+0000 7f1faf7fe700 1 -- 172.21.15.90:0/2590482593 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 655195624 0 0) 0x7f1fb810b500 con 0x7f1fb8103aa0 2024-04-14T15:26:03.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:03.976+0000 7f1faf7fe700 1 -- 172.21.15.90:0/2590482593 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1fa40031f0 con 0x7f1fb8103aa0 2024-04-14T15:26:03.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:03.976+0000 7f1faf7fe700 1 -- 172.21.15.90:0/2590482593 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3206774053 0 0) 0x7f1fb81aa0a0 con 0x7f1fb810f3c0 2024-04-14T15:26:03.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:03.976+0000 7f1faf7fe700 1 -- 172.21.15.90:0/2590482593 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1fb810b500 con 0x7f1fb810f3c0 2024-04-14T15:26:03.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:03.976+0000 7f1faf7fe700 1 -- 172.21.15.90:0/2590482593 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1313133990 0 0) 0x7f1fa40031f0 con 0x7f1fb8103aa0 2024-04-14T15:26:03.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:03.976+0000 7f1faf7fe700 1 -- 172.21.15.90:0/2590482593 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1fb81aa0a0 con 0x7f1fb8103aa0 2024-04-14T15:26:03.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:03.976+0000 7f1faf7fe700 10 client.?.objecter ms_handle_connect 0x7f1fb810b8b0 2024-04-14T15:26:03.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:03.976+0000 7f1faf7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:26:03.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:03.976+0000 7f1faf7fe700 1 -- 172.21.15.90:0/2590482593 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f1fb40018b0 con 0x7f1fb8103aa0 2024-04-14T15:26:03.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:03.976+0000 7f1faf7fe700 1 -- 172.21.15.90:0/2590482593 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2115571906 0 0) 0x7f1fb810b500 con 0x7f1fb810f3c0 2024-04-14T15:26:03.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:03.976+0000 7f1faf7fe700 1 -- 172.21.15.90:0/2590482593 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1fa40031f0 con 0x7f1fb810f3c0 2024-04-14T15:26:03.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:03.976+0000 7f1faf7fe700 1 -- 172.21.15.90:0/2590482593 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f1fb0001bb0 con 0x7f1fb810f3c0 2024-04-14T15:26:03.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:03.976+0000 7f1faf7fe700 1 -- 172.21.15.90:0/2590482593 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3128934327 0 0) 0x7f1fb81a8ec0 con 0x7f1fb810b8b0 2024-04-14T15:26:03.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:03.977+0000 7f1faf7fe700 1 -- 172.21.15.90:0/2590482593 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1fb810b500 con 0x7f1fb810b8b0 2024-04-14T15:26:03.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:03.977+0000 7f1faf7fe700 1 -- 172.21.15.90:0/2590482593 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4278354299 0 0) 0x7f1fb81aa0a0 con 0x7f1fb8103aa0 2024-04-14T15:26:03.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:03.977+0000 7f1faf7fe700 1 -- 172.21.15.90:0/2590482593 >> v1:172.21.15.90:6790/0 conn(0x7f1fb810f3c0 legacy=0x7f1fb81a6790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:03.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:03.977+0000 7f1faf7fe700 1 -- 172.21.15.90:0/2590482593 >> v1:172.21.15.92:6789/0 conn(0x7f1fb810b8b0 legacy=0x7f1fb8108a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:03.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:03.977+0000 7f1faf7fe700 1 -- 172.21.15.90:0/2590482593 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fb81ab280 con 0x7f1fb8103aa0 2024-04-14T15:26:03.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:03.977+0000 7f1fc028b700 1 -- 172.21.15.90:0/2590482593 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1fb81aa2d0 con 0x7f1fb8103aa0 2024-04-14T15:26:03.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:03.977+0000 7f1fc028b700 10 client.38079.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:26:03.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:03.977+0000 7f1fc028b700 1 -- 172.21.15.90:0/2590482593 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f1fb81aa790 con 0x7f1fb8103aa0 2024-04-14T15:26:03.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:03.977+0000 7f1fc028b700 1 librados: init done 2024-04-14T15:26:03.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:03.977+0000 7f1faf7fe700 1 -- 172.21.15.90:0/2590482593 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f1fb4001ca0 con 0x7f1fb8103aa0 2024-04-14T15:26:03.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:03.977+0000 7f1faf7fe700 1 -- 172.21.15.90:0/2590482593 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f1fb4005d70 con 0x7f1fb8103aa0 2024-04-14T15:26:03.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:03.978+0000 7f1fc028b700 1 -- 172.21.15.90:0/2590482593 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f9c005120 con 0x7f1fb8103aa0 2024-04-14T15:26:03.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:03.979+0000 7f1faf7fe700 1 -- 172.21.15.90:0/2590482593 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f1fb401c920 con 0x7f1fb8103aa0 2024-04-14T15:26:03.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:03.979+0000 7f1faf7fe700 1 -- 172.21.15.90:0/2590482593 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(813..813 src has 467..813) ==== 8757+0+0 (unknown 4097620442 0 0) 0x7f1fb408ad70 con 0x7f1fb8103aa0 2024-04-14T15:26:03.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:03.979+0000 7f1faf7fe700 10 client.38079.objecter ms_dispatch 0x7f1fb8001550 osd_map(813..813 src has 467..813) 2024-04-14T15:26:03.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:03.979+0000 7f1faf7fe700 3 client.38079.objecter handle_osd_map got epochs [813,813] > 0 2024-04-14T15:26:03.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:03.979+0000 7f1faf7fe700 3 client.38079.objecter handle_osd_map decoding full epoch 813 2024-04-14T15:26:03.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:03.980+0000 7f1faf7fe700 20 client.38079.objecter dump_active .. 0 homeless 2024-04-14T15:26:03.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:03.980+0000 7f1faf7fe700 10 client.38079.objecter ms_handle_connect 0x7f1fa40711a0 2024-04-14T15:26:03.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:03.984+0000 7f1faf7fe700 1 -- 172.21.15.90:0/2590482593 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f1fb4057550 con 0x7f1fb8103aa0 2024-04-14T15:26:04.009 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:26:04.036 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:04.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:04.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:26:04.079 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:26:03 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:26:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:26:04.138 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:26:04.163 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:04.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:04.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:26:04.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:04.211+0000 7f1fc028b700 1 -- 172.21.15.90:0/2590482593 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7f1f9c005410 con 0x7f1fb8103aa0 2024-04-14T15:26:04.264 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:26:04.289 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:04.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:04.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:26:04.390 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:26:04.414 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:04.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:04.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:26:04.516 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:26:04.540 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:04.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:04.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:26:04.641 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:26:04.665 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:04.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:04.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:26:04.767 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:26:04.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:04.792 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:04.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:26:04.894 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:26:04.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:04.917 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:04.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:26:05.018 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:26:05.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:05.026+0000 7f1faf7fe700 1 -- 172.21.15.90:0/2590482593 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v815) ==== 87+0+0 (unknown 3339318722 0 0) 0x7f1fb405b200 con 0x7f1fb8103aa0 2024-04-14T15:26:05.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:05.029+0000 7f1fc028b700 10 librados: watch_flush enter 2024-04-14T15:26:05.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:05.029+0000 7f1fc028b700 10 librados: watch_flush exit 2024-04-14T15:26:05.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:05.029+0000 7f1fc028b700 20 client.38079.objecter shutdown clearing up homeless session... 2024-04-14T15:26:05.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:05.029+0000 7f1fc028b700 10 client.38079.objecter successfully canceled tick 2024-04-14T15:26:05.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:05.029+0000 7f1fc028b700 1 -- 172.21.15.90:0/2590482593 >> v1:172.21.15.90:6800/420935427 conn(0x7f1fa40711a0 legacy=0x7f1fa4073660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:05.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:05.029+0000 7f1fc028b700 1 -- 172.21.15.90:0/2590482593 >> v1:172.21.15.90:6789/0 conn(0x7f1fb8103aa0 legacy=0x7f1fb8108350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:05.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:05.029+0000 7f1fc028b700 1 -- 172.21.15.90:0/2590482593 shutdown_connections 2024-04-14T15:26:05.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:05.029+0000 7f1fc028b700 1 -- 172.21.15.90:0/2590482593 >> 172.21.15.90:0/2590482593 conn(0x7f1fb80fd290 msgr2=0x7f1fb80ff680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:26:05.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:05.029+0000 7f1fc028b700 1 -- 172.21.15.90:0/2590482593 shutdown_connections 2024-04-14T15:26:05.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:05.029+0000 7f1fc028b700 1 -- 172.21.15.90:0/2590482593 wait complete. 2024-04-14T15:26:05.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:05.029+0000 7f1fc028b700 1 librados: shutdown 2024-04-14T15:26:05.031 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is set 2024-04-14T15:26:05.050 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:05.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:05.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:26:05.152 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:26:05.175 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:05.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:05.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:26:05.277 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:26:05.289 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:05 smithi092 ceph-mon[33966]: pgmap v1207: 224 pgs: 32 unknown, 1 active+remapped+backfill_toofull, 191 active+clean; 4.4 MiB data, 760 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:26:05.289 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:05 smithi092 ceph-mon[33966]: 3.2 deep-scrub starts 2024-04-14T15:26:05.289 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:05 smithi092 ceph-mon[33966]: 3.2 deep-scrub ok 2024-04-14T15:26:05.289 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:05 smithi092 ceph-mon[33966]: 6.5 scrub starts 2024-04-14T15:26:05.289 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:05 smithi092 ceph-mon[33966]: 6.5 scrub ok 2024-04-14T15:26:05.289 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:05 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-144"}]': finished 2024-04-14T15:26:05.289 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:05 smithi092 ceph-mon[33966]: osdmap e814: 8 total, 8 up, 7 in 2024-04-14T15:26:05.289 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:05 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-144", "mode": "readproxy"}]: dispatch 2024-04-14T15:26:05.289 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:05 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-144", "mode": "readproxy"}]: dispatch 2024-04-14T15:26:05.289 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:05 smithi092 ceph-mon[33966]: 11.7 scrub starts 2024-04-14T15:26:05.289 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:05 smithi092 ceph-mon[33966]: 11.7 scrub ok 2024-04-14T15:26:05.290 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:05 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2590482593' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:26:05.301 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:05.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:05.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:26:05.315 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-04-14T15:26:05.315 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:26:05.403 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:26:05.426 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:05.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:05.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:26:05.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:05 smithi090 ceph-mon[30399]: pgmap v1207: 224 pgs: 32 unknown, 1 active+remapped+backfill_toofull, 191 active+clean; 4.4 MiB data, 760 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:26:05.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:05 smithi090 ceph-mon[30399]: 3.2 deep-scrub starts 2024-04-14T15:26:05.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:05 smithi090 ceph-mon[30399]: 3.2 deep-scrub ok 2024-04-14T15:26:05.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:05 smithi090 ceph-mon[30399]: 6.5 scrub starts 2024-04-14T15:26:05.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:05 smithi090 ceph-mon[30399]: 6.5 scrub ok 2024-04-14T15:26:05.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:05 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-144"}]': finished 2024-04-14T15:26:05.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:05 smithi090 ceph-mon[30399]: osdmap e814: 8 total, 8 up, 7 in 2024-04-14T15:26:05.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:05 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-144", "mode": "readproxy"}]: dispatch 2024-04-14T15:26:05.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:05 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-144", "mode": "readproxy"}]: dispatch 2024-04-14T15:26:05.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:05 smithi090 ceph-mon[30399]: 11.7 scrub starts 2024-04-14T15:26:05.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:05 smithi090 ceph-mon[30399]: 11.7 scrub ok 2024-04-14T15:26:05.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:05 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2590482593' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:26:05.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:05 smithi090 ceph-mon[41670]: pgmap v1207: 224 pgs: 32 unknown, 1 active+remapped+backfill_toofull, 191 active+clean; 4.4 MiB data, 760 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:26:05.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:05 smithi090 ceph-mon[41670]: 3.2 deep-scrub starts 2024-04-14T15:26:05.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:05 smithi090 ceph-mon[41670]: 3.2 deep-scrub ok 2024-04-14T15:26:05.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:05 smithi090 ceph-mon[41670]: 6.5 scrub starts 2024-04-14T15:26:05.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:05 smithi090 ceph-mon[41670]: 6.5 scrub ok 2024-04-14T15:26:05.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:05 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-144"}]': finished 2024-04-14T15:26:05.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:05 smithi090 ceph-mon[41670]: osdmap e814: 8 total, 8 up, 7 in 2024-04-14T15:26:05.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:05 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-144", "mode": "readproxy"}]: dispatch 2024-04-14T15:26:05.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:05 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-144", "mode": "readproxy"}]: dispatch 2024-04-14T15:26:05.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:05 smithi090 ceph-mon[41670]: 11.7 scrub starts 2024-04-14T15:26:05.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:05 smithi090 ceph-mon[41670]: 11.7 scrub ok 2024-04-14T15:26:05.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:05 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2590482593' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:26:05.528 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:26:05.552 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:05.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:05.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:26:05.622 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:26:05 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:26:05.654 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:26:05.677 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:05.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:05.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:26:05.779 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:26:05.802 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:05.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:05.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:26:05.903 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:26:05.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:05.930 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:05.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:26:05.952 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-14T15:26:06.031 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:26:06.054 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:06.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:06.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:26:06.156 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:26:06.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:06.180 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:06.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:26:06.281 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:26:06.305 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:06.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:06.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:26:06.348 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:06 smithi090 ceph-mon[41670]: 1.8 scrub starts 2024-04-14T15:26:06.348 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:06 smithi090 ceph-mon[41670]: 1.8 scrub ok 2024-04-14T15:26:06.348 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:06 smithi090 ceph-mon[41670]: 2.19 scrub starts 2024-04-14T15:26:06.348 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:06 smithi090 ceph-mon[41670]: 2.19 scrub ok 2024-04-14T15:26:06.348 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:06 smithi090 ceph-mon[41670]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:26:06.348 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:06 smithi090 ceph-mon[41670]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:26:06.349 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:06 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-144", "mode": "readproxy"}]': finished 2024-04-14T15:26:06.349 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:06 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2590482593' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-14T15:26:06.349 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:06 smithi090 ceph-mon[41670]: osdmap e815: 8 total, 8 up, 7 in 2024-04-14T15:26:06.349 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:06 smithi090 ceph-mon[41670]: 6.9 scrub starts 2024-04-14T15:26:06.349 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:06 smithi090 ceph-mon[41670]: 6.9 scrub ok 2024-04-14T15:26:06.349 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:06 smithi090 ceph-mon[41670]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:26:06.349 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:06 smithi090 ceph-mon[41670]: osdmap e816: 8 total, 8 up, 7 in 2024-04-14T15:26:06.349 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:06 smithi090 ceph-mon[30399]: 1.8 scrub starts 2024-04-14T15:26:06.349 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:06 smithi090 ceph-mon[30399]: 1.8 scrub ok 2024-04-14T15:26:06.349 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:06 smithi090 ceph-mon[30399]: 2.19 scrub starts 2024-04-14T15:26:06.349 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:06 smithi090 ceph-mon[30399]: 2.19 scrub ok 2024-04-14T15:26:06.349 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:06 smithi090 ceph-mon[30399]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:26:06.350 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:06 smithi090 ceph-mon[30399]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:26:06.350 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:06 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-144", "mode": "readproxy"}]': finished 2024-04-14T15:26:06.350 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:06 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2590482593' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-14T15:26:06.350 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:06 smithi090 ceph-mon[30399]: osdmap e815: 8 total, 8 up, 7 in 2024-04-14T15:26:06.350 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:06 smithi090 ceph-mon[30399]: 6.9 scrub starts 2024-04-14T15:26:06.350 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:06 smithi090 ceph-mon[30399]: 6.9 scrub ok 2024-04-14T15:26:06.350 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:06 smithi090 ceph-mon[30399]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:26:06.350 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:06 smithi090 ceph-mon[30399]: osdmap e816: 8 total, 8 up, 7 in 2024-04-14T15:26:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:06 smithi092 ceph-mon[33966]: 1.8 scrub starts 2024-04-14T15:26:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:06 smithi092 ceph-mon[33966]: 1.8 scrub ok 2024-04-14T15:26:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:06 smithi092 ceph-mon[33966]: 2.19 scrub starts 2024-04-14T15:26:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:06 smithi092 ceph-mon[33966]: 2.19 scrub ok 2024-04-14T15:26:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:06 smithi092 ceph-mon[33966]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:26:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:06 smithi092 ceph-mon[33966]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:26:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:06 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-144", "mode": "readproxy"}]': finished 2024-04-14T15:26:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:06 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2590482593' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-14T15:26:06.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:06 smithi092 ceph-mon[33966]: osdmap e815: 8 total, 8 up, 7 in 2024-04-14T15:26:06.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:06 smithi092 ceph-mon[33966]: 6.9 scrub starts 2024-04-14T15:26:06.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:06 smithi092 ceph-mon[33966]: 6.9 scrub ok 2024-04-14T15:26:06.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:06 smithi092 ceph-mon[33966]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:26:06.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:06 smithi092 ceph-mon[33966]: osdmap e816: 8 total, 8 up, 7 in 2024-04-14T15:26:06.407 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:26:06.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:06.432 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:06.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:26:06.533 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:26:06.557 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:06.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:06.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:26:06.659 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:26:06.682 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:06.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:06.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:26:06.783 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:26:06.807 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:06.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:06.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:26:06.909 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:26:06.932 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:06.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:06.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:26:07.034 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:26:07.058 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:07.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:07.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:26:07.143 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:07 smithi090 ceph-mon[41670]: pgmap v1210: 224 pgs: 224 active+clean; 4.4 MiB data, 761 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:26:07.144 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:07 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:26:07.144 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:07 smithi090 ceph-mon[41670]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-04-14T15:26:07.144 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:07 smithi090 ceph-mon[30399]: pgmap v1210: 224 pgs: 224 active+clean; 4.4 MiB data, 761 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:26:07.144 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:07 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:26:07.144 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:07 smithi090 ceph-mon[30399]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-04-14T15:26:07.160 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:26:07.185 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:07.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:07.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:26:07.287 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:26:07.311 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:07.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:07.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:26:07.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:07 smithi092 ceph-mon[33966]: pgmap v1210: 224 pgs: 224 active+clean; 4.4 MiB data, 761 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:26:07.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:07 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:26:07.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:07 smithi092 ceph-mon[33966]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-04-14T15:26:07.413 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:26:07.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:07.440 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:07.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:26:07.541 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:26:07.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:07.567 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:07.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:26:07.668 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:26:07.691 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:07.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:07.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:26:07.793 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:26:07.817 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:07.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:07.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:26:07.919 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:26:07.946 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:07.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:07.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:26:08.048 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:26:08.073 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:08.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:08.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:26:08.175 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:26:08.201 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:08.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:08.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:26:08.268 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:08 smithi090 ceph-mon[41670]: pgmap v1212: 224 pgs: 224 active+clean; 4.4 MiB data, 761 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:26:08.268 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:08 smithi090 ceph-mon[30399]: pgmap v1212: 224 pgs: 224 active+clean; 4.4 MiB data, 761 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:26:08.304 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:26:08.330 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:08.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:08.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:26:08.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:08 smithi092 ceph-mon[33966]: pgmap v1212: 224 pgs: 224 active+clean; 4.4 MiB data, 761 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:26:08.432 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:26:08.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:08.458 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:08.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:26:08.559 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:26:08.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:08.586 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:08.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:26:08.687 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:26:08.712 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:08.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:08.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:26:08.813 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:26:08.838 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:08.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:08.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:26:08.940 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:26:08.963 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:08.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:08.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:26:09.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.000+0000 7f996b216700 1 -- 172.21.15.90:0/110942307 >> v1:172.21.15.90:6789/0 conn(0x7f996410e1e0 legacy=0x7f99641106a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:09.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.000+0000 7f996b216700 1 -- 172.21.15.90:0/110942307 shutdown_connections 2024-04-14T15:26:09.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.000+0000 7f996b216700 1 -- 172.21.15.90:0/110942307 >> 172.21.15.90:0/110942307 conn(0x7f99640fd290 msgr2=0x7f99640ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:26:09.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.000+0000 7f996b216700 1 -- 172.21.15.90:0/110942307 shutdown_connections 2024-04-14T15:26:09.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.000+0000 7f996b216700 1 -- 172.21.15.90:0/110942307 wait complete. 2024-04-14T15:26:09.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.000+0000 7f996b216700 1 librados: starting msgr at 2024-04-14T15:26:09.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.000+0000 7f996b216700 1 librados: starting objecter 2024-04-14T15:26:09.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.000+0000 7f996b216700 1 Processor -- start 2024-04-14T15:26:09.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.000+0000 7f996b216700 1 -- start start 2024-04-14T15:26:09.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.000+0000 7f996b216700 1 librados: setting wanted keys 2024-04-14T15:26:09.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.000+0000 7f996b216700 1 librados: calling monclient init 2024-04-14T15:26:09.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.001+0000 7f996b216700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f996410df50 con 0x7f996410e1e0 2024-04-14T15:26:09.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.001+0000 7f996b216700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f99641a7df0 con 0x7f9964107760 2024-04-14T15:26:09.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.001+0000 7f996b216700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f99641a8fd0 con 0x7f996410a610 2024-04-14T15:26:09.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.001+0000 7f9968fb2700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f9964107760 0x7f996410c7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.90:38164/0 (socket says 172.21.15.90:38164) 2024-04-14T15:26:09.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.001+0000 7f9963fff700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f996410a610 0x7f996410cef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:36634/0 (socket says 172.21.15.90:36634) 2024-04-14T15:26:09.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.001+0000 7f9968fb2700 1 -- 172.21.15.90:0/149769965 learned_addr learned my addr 172.21.15.90:0/149769965 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:26:09.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.001+0000 7f99697b3700 1 --1- 172.21.15.90:0/149769965 >> v1:172.21.15.90:6789/0 conn(0x7f996410e1e0 0x7f99641a56c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:47102/0 (socket says 172.21.15.90:47102) 2024-04-14T15:26:09.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.001+0000 7f9961ffb700 10 client.?.objecter ms_handle_connect 0x7f996410e1e0 2024-04-14T15:26:09.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.001+0000 7f9961ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:26:09.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.001+0000 7f9961ffb700 10 client.?.objecter ms_handle_connect 0x7f996410a610 2024-04-14T15:26:09.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.001+0000 7f9961ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:26:09.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.001+0000 7f9961ffb700 10 client.?.objecter ms_handle_connect 0x7f9964107760 2024-04-14T15:26:09.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.001+0000 7f9961ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:26:09.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.001+0000 7f9961ffb700 1 -- 172.21.15.90:0/149769965 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3280948658 0 0) 0x7f996410df50 con 0x7f996410e1e0 2024-04-14T15:26:09.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.001+0000 7f9961ffb700 1 -- 172.21.15.90:0/149769965 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f99540035b0 con 0x7f996410e1e0 2024-04-14T15:26:09.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.001+0000 7f9961ffb700 1 -- 172.21.15.90:0/149769965 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3554866377 0 0) 0x7f99641a8fd0 con 0x7f996410a610 2024-04-14T15:26:09.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.001+0000 7f9961ffb700 1 -- 172.21.15.90:0/149769965 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f996410df50 con 0x7f996410a610 2024-04-14T15:26:09.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.001+0000 7f9961ffb700 1 -- 172.21.15.90:0/149769965 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2091442782 0 0) 0x7f99641a7df0 con 0x7f9964107760 2024-04-14T15:26:09.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.001+0000 7f9961ffb700 1 -- 172.21.15.90:0/149769965 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f99641a8fd0 con 0x7f9964107760 2024-04-14T15:26:09.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.001+0000 7f9961ffb700 1 -- 172.21.15.90:0/149769965 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 528330396 0 0) 0x7f99540035b0 con 0x7f996410e1e0 2024-04-14T15:26:09.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.001+0000 7f9961ffb700 1 -- 172.21.15.90:0/149769965 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f99641a7df0 con 0x7f996410e1e0 2024-04-14T15:26:09.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.001+0000 7f9961ffb700 1 -- 172.21.15.90:0/149769965 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f995c0017b0 con 0x7f996410e1e0 2024-04-14T15:26:09.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.001+0000 7f9961ffb700 1 -- 172.21.15.90:0/149769965 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1518824960 0 0) 0x7f99641a8fd0 con 0x7f9964107760 2024-04-14T15:26:09.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.001+0000 7f9961ffb700 1 -- 172.21.15.90:0/149769965 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f99540035b0 con 0x7f9964107760 2024-04-14T15:26:09.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.001+0000 7f9961ffb700 1 -- 172.21.15.90:0/149769965 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f9958001850 con 0x7f9964107760 2024-04-14T15:26:09.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.001+0000 7f9961ffb700 1 -- 172.21.15.90:0/149769965 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1844137679 0 0) 0x7f996410df50 con 0x7f996410a610 2024-04-14T15:26:09.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.002+0000 7f9961ffb700 1 -- 172.21.15.90:0/149769965 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f99641a8fd0 con 0x7f996410a610 2024-04-14T15:26:09.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.002+0000 7f9961ffb700 1 -- 172.21.15.90:0/149769965 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f99500017b0 con 0x7f996410a610 2024-04-14T15:26:09.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.002+0000 7f9961ffb700 1 -- 172.21.15.90:0/149769965 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3560333928 0 0) 0x7f99641a7df0 con 0x7f996410e1e0 2024-04-14T15:26:09.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.002+0000 7f9961ffb700 1 -- 172.21.15.90:0/149769965 >> v1:172.21.15.90:6790/0 conn(0x7f996410a610 legacy=0x7f996410cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:09.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.002+0000 7f9961ffb700 1 -- 172.21.15.90:0/149769965 >> v1:172.21.15.92:6789/0 conn(0x7f9964107760 legacy=0x7f996410c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:09.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.002+0000 7f9961ffb700 1 -- 172.21.15.90:0/149769965 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99641aa210 con 0x7f996410e1e0 2024-04-14T15:26:09.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.002+0000 7f996b216700 1 -- 172.21.15.90:0/149769965 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f99641a6e40 con 0x7f996410e1e0 2024-04-14T15:26:09.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.002+0000 7f9961ffb700 1 -- 172.21.15.90:0/149769965 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f995c001eb0 con 0x7f996410e1e0 2024-04-14T15:26:09.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.002+0000 7f9961ffb700 1 -- 172.21.15.90:0/149769965 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f995c005430 con 0x7f996410e1e0 2024-04-14T15:26:09.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.002+0000 7f996b216700 10 client.38091.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:26:09.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.002+0000 7f996b216700 1 -- 172.21.15.90:0/149769965 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f99641a7300 con 0x7f996410e1e0 2024-04-14T15:26:09.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.002+0000 7f996b216700 1 librados: init done 2024-04-14T15:26:09.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.003+0000 7f996b216700 1 -- 172.21.15.90:0/149769965 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9964102ea0 con 0x7f996410e1e0 2024-04-14T15:26:09.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.003+0000 7f9961ffb700 1 -- 172.21.15.90:0/149769965 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f995c005670 con 0x7f996410e1e0 2024-04-14T15:26:09.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.004+0000 7f9961ffb700 1 -- 172.21.15.90:0/149769965 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(816..816 src has 467..816) ==== 8757+0+0 (unknown 3218672219 0 0) 0x7f995c08b0d0 con 0x7f996410e1e0 2024-04-14T15:26:09.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.004+0000 7f9961ffb700 10 client.38091.objecter ms_dispatch 0x7f9964001550 osd_map(816..816 src has 467..816) 2024-04-14T15:26:09.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.004+0000 7f9961ffb700 3 client.38091.objecter handle_osd_map got epochs [816,816] > 0 2024-04-14T15:26:09.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.004+0000 7f9961ffb700 3 client.38091.objecter handle_osd_map decoding full epoch 816 2024-04-14T15:26:09.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.004+0000 7f9961ffb700 20 client.38091.objecter dump_active .. 0 homeless 2024-04-14T15:26:09.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.004+0000 7f9961ffb700 10 client.38091.objecter ms_handle_connect 0x7f99540713a0 2024-04-14T15:26:09.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.008+0000 7f9961ffb700 1 -- 172.21.15.90:0/149769965 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f995c0578b0 con 0x7f996410e1e0 2024-04-14T15:26:09.066 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:26:09.090 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:09.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:09.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:26:09.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.123+0000 7ff61eea3700 1 -- 172.21.15.90:0/8116846 >> v1:172.21.15.90:6789/0 conn(0x7ff61810a610 legacy=0x7ff61810ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:09.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.123+0000 7ff61eea3700 1 -- 172.21.15.90:0/8116846 shutdown_connections 2024-04-14T15:26:09.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.123+0000 7ff61eea3700 1 -- 172.21.15.90:0/8116846 >> 172.21.15.90:0/8116846 conn(0x7ff6180fd290 msgr2=0x7ff6180ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:26:09.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.123+0000 7ff61eea3700 1 -- 172.21.15.90:0/8116846 shutdown_connections 2024-04-14T15:26:09.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.123+0000 7ff61eea3700 1 -- 172.21.15.90:0/8116846 wait complete. 2024-04-14T15:26:09.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.124+0000 7ff61eea3700 1 librados: starting msgr at 2024-04-14T15:26:09.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.124+0000 7ff61eea3700 1 librados: starting objecter 2024-04-14T15:26:09.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.124+0000 7ff61eea3700 1 Processor -- start 2024-04-14T15:26:09.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.124+0000 7ff61eea3700 1 -- start start 2024-04-14T15:26:09.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.124+0000 7ff61eea3700 1 librados: setting wanted keys 2024-04-14T15:26:09.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.124+0000 7ff61eea3700 1 librados: calling monclient init 2024-04-14T15:26:09.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.124+0000 7ff61eea3700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff6181a9b80 con 0x7ff61810a610 2024-04-14T15:26:09.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.124+0000 7ff61eea3700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff6181aada0 con 0x7ff618107760 2024-04-14T15:26:09.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.124+0000 7ff61eea3700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff6181abfe0 con 0x7ff61810e1e0 2024-04-14T15:26:09.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.124+0000 7ff61d440700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7ff61810e1e0 0x7ff6181a8240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:36614/0 (socket says 172.21.15.90:36614) 2024-04-14T15:26:09.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.124+0000 7ff617fff700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7ff61810a610 0x7ff6181a4a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:47086/0 (socket says 172.21.15.90:47086) 2024-04-14T15:26:09.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.124+0000 7ff61d440700 1 -- 172.21.15.90:0/3919189616 learned_addr learned my addr 172.21.15.90:0/3919189616 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:26:09.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.124+0000 7ff615ffb700 10 client.?.objecter ms_handle_connect 0x7ff61810e1e0 2024-04-14T15:26:09.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.124+0000 7ff615ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:26:09.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.124+0000 7ff615ffb700 10 client.?.objecter ms_handle_connect 0x7ff61810a610 2024-04-14T15:26:09.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.124+0000 7ff615ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:26:09.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.125+0000 7ff615ffb700 10 client.?.objecter ms_handle_connect 0x7ff618107760 2024-04-14T15:26:09.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.125+0000 7ff615ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:26:09.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.125+0000 7ff615ffb700 1 -- 172.21.15.90:0/3919189616 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 435059010 0 0) 0x7ff6181a9b80 con 0x7ff61810a610 2024-04-14T15:26:09.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.125+0000 7ff615ffb700 1 -- 172.21.15.90:0/3919189616 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff6000031f0 con 0x7ff61810a610 2024-04-14T15:26:09.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.125+0000 7ff615ffb700 1 -- 172.21.15.90:0/3919189616 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 439329830 0 0) 0x7ff6181abfe0 con 0x7ff61810e1e0 2024-04-14T15:26:09.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.125+0000 7ff615ffb700 1 -- 172.21.15.90:0/3919189616 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff6181a9b80 con 0x7ff61810e1e0 2024-04-14T15:26:09.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.125+0000 7ff615ffb700 1 -- 172.21.15.90:0/3919189616 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3158349637 0 0) 0x7ff6181aada0 con 0x7ff618107760 2024-04-14T15:26:09.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.125+0000 7ff615ffb700 1 -- 172.21.15.90:0/3919189616 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff6181abfe0 con 0x7ff618107760 2024-04-14T15:26:09.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.125+0000 7ff615ffb700 1 -- 172.21.15.90:0/3919189616 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 829280149 0 0) 0x7ff6000031f0 con 0x7ff61810a610 2024-04-14T15:26:09.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.125+0000 7ff615ffb700 1 -- 172.21.15.90:0/3919189616 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6181aada0 con 0x7ff61810a610 2024-04-14T15:26:09.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.125+0000 7ff615ffb700 1 -- 172.21.15.90:0/3919189616 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7ff60c0044b0 con 0x7ff61810a610 2024-04-14T15:26:09.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.125+0000 7ff615ffb700 1 -- 172.21.15.90:0/3919189616 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 218368433 0 0) 0x7ff6181a9b80 con 0x7ff61810e1e0 2024-04-14T15:26:09.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.125+0000 7ff615ffb700 1 -- 172.21.15.90:0/3919189616 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6000031f0 con 0x7ff61810e1e0 2024-04-14T15:26:09.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.125+0000 7ff615ffb700 1 -- 172.21.15.90:0/3919189616 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7ff6080018f0 con 0x7ff61810e1e0 2024-04-14T15:26:09.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.125+0000 7ff615ffb700 1 -- 172.21.15.90:0/3919189616 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3454753228 0 0) 0x7ff6181abfe0 con 0x7ff618107760 2024-04-14T15:26:09.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.125+0000 7ff615ffb700 1 -- 172.21.15.90:0/3919189616 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6181a9b80 con 0x7ff618107760 2024-04-14T15:26:09.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.125+0000 7ff615ffb700 1 -- 172.21.15.90:0/3919189616 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7ff610001650 con 0x7ff618107760 2024-04-14T15:26:09.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.125+0000 7ff615ffb700 1 -- 172.21.15.90:0/3919189616 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3409502780 0 0) 0x7ff6181aada0 con 0x7ff61810a610 2024-04-14T15:26:09.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.125+0000 7ff615ffb700 1 -- 172.21.15.90:0/3919189616 >> v1:172.21.15.90:6790/0 conn(0x7ff61810e1e0 legacy=0x7ff6181a8240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:09.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.125+0000 7ff615ffb700 1 -- 172.21.15.90:0/3919189616 >> v1:172.21.15.92:6789/0 conn(0x7ff618107760 legacy=0x7ff618110130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:09.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.125+0000 7ff615ffb700 1 -- 172.21.15.90:0/3919189616 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6181ad220 con 0x7ff61810a610 2024-04-14T15:26:09.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.125+0000 7ff61eea3700 1 -- 172.21.15.90:0/3919189616 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff6181ac210 con 0x7ff61810a610 2024-04-14T15:26:09.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.125+0000 7ff615ffb700 1 -- 172.21.15.90:0/3919189616 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7ff60c001af0 con 0x7ff61810a610 2024-04-14T15:26:09.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.125+0000 7ff615ffb700 1 -- 172.21.15.90:0/3919189616 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7ff60c005000 con 0x7ff61810a610 2024-04-14T15:26:09.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.125+0000 7ff61eea3700 10 client.38085.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:26:09.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.125+0000 7ff61eea3700 1 -- 172.21.15.90:0/3919189616 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff6181ac6d0 con 0x7ff61810a610 2024-04-14T15:26:09.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.125+0000 7ff61eea3700 1 librados: init done 2024-04-14T15:26:09.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.126+0000 7ff61eea3700 1 -- 172.21.15.90:0/3919189616 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7ff618102ea0 con 0x7ff61810a610 2024-04-14T15:26:09.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.127+0000 7ff615ffb700 1 -- 172.21.15.90:0/3919189616 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7ff60c005000 con 0x7ff61810a610 2024-04-14T15:26:09.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.127+0000 7ff615ffb700 1 -- 172.21.15.90:0/3919189616 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(816..816 src has 467..816) ==== 8757+0+0 (unknown 3218672219 0 0) 0x7ff60c08ac70 con 0x7ff61810a610 2024-04-14T15:26:09.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.127+0000 7ff615ffb700 10 client.38085.objecter ms_dispatch 0x7ff618001550 osd_map(816..816 src has 467..816) 2024-04-14T15:26:09.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.127+0000 7ff615ffb700 3 client.38085.objecter handle_osd_map got epochs [816,816] > 0 2024-04-14T15:26:09.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.127+0000 7ff615ffb700 3 client.38085.objecter handle_osd_map decoding full epoch 816 2024-04-14T15:26:09.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.127+0000 7ff615ffb700 20 client.38085.objecter dump_active .. 0 homeless 2024-04-14T15:26:09.161 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:26:09.162 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:26:09.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.127+0000 7ff615ffb700 1 -- 172.21.15.90:0/3919189616 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v816) ==== 54+0+16 (unknown 3515282128 0 2420000192) 0x7ff60c08d050 con 0x7ff61810a610 2024-04-14T15:26:09.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.127+0000 7ff615ffb700 10 client.38085.objecter ms_handle_connect 0x7ff600071290 2024-04-14T15:26:09.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.128+0000 7ff61eea3700 10 client.38085.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:26:09.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.128+0000 7ff61eea3700 15 client.38085.objecter _session_command_op_assign -1 1 2024-04-14T15:26:09.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.128+0000 7ff61eea3700 20 client.38085.objecter _get_session s=0x7ff6181121b0 osd=0 3 2024-04-14T15:26:09.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.128+0000 7ff61eea3700 20 client.38085.objecter put_session s=0x7ff6181121b0 osd=0 3 2024-04-14T15:26:09.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.128+0000 7ff61eea3700 20 client.38085.objecter _get_session s=0x7ff6181121b0 osd=0 3 2024-04-14T15:26:09.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.128+0000 7ff61eea3700 15 client.38085.objecter _session_command_op_remove -1 1 2024-04-14T15:26:09.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.128+0000 7ff61eea3700 20 client.38085.objecter get_session s=0x7ff6181121b0 osd=0 3 2024-04-14T15:26:09.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.128+0000 7ff61eea3700 15 client.38085.objecter _session_command_op_assign 0 1 2024-04-14T15:26:09.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.128+0000 7ff61eea3700 20 client.38085.objecter put_session s=0x7ff6181121b0 osd=0 4 2024-04-14T15:26:09.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.128+0000 7ff61eea3700 10 client.38085.objecter _send_command 1 2024-04-14T15:26:09.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.128+0000 7ff61eea3700 1 -- 172.21.15.90:0/3919189616 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff61810fbf0 con 0x7ff6181aaf70 2024-04-14T15:26:09.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.128+0000 7ff615ffb700 10 client.38085.objecter ms_handle_connect 0x7ff6181aaf70 2024-04-14T15:26:09.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.128+0000 7ff615ffb700 1 -- 172.21.15.90:0/3919189616 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7ff61810fbf0 con 0x7ff6181aaf70 2024-04-14T15:26:09.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.128+0000 7ff615ffb700 10 client.38085.objecter ms_dispatch 0x7ff618001550 command_reply(tid 1: 0 ) 2024-04-14T15:26:09.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.128+0000 7ff615ffb700 10 client.38085.objecter _finish_command 1 = system:0 2024-04-14T15:26:09.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.128+0000 7ff615ffb700 20 client.38085.objecter put_session s=0x7ff6181121b0 osd=0 4 2024-04-14T15:26:09.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.128+0000 7ff615ffb700 15 client.38085.objecter _session_command_op_remove 0 1 2024-04-14T15:26:09.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.145+0000 7ff61eea3700 10 client.38085.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:26:09.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.145+0000 7ff61eea3700 15 client.38085.objecter _session_command_op_assign -1 2 2024-04-14T15:26:09.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.145+0000 7ff61eea3700 20 client.38085.objecter _get_session s=0x7ff6181121b0 osd=0 3 2024-04-14T15:26:09.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.145+0000 7ff61eea3700 20 client.38085.objecter put_session s=0x7ff6181121b0 osd=0 3 2024-04-14T15:26:09.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.145+0000 7ff61eea3700 20 client.38085.objecter _get_session s=0x7ff6181121b0 osd=0 3 2024-04-14T15:26:09.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.145+0000 7ff61eea3700 15 client.38085.objecter _session_command_op_remove -1 2 2024-04-14T15:26:09.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.145+0000 7ff61eea3700 20 client.38085.objecter get_session s=0x7ff6181121b0 osd=0 3 2024-04-14T15:26:09.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.145+0000 7ff61eea3700 15 client.38085.objecter _session_command_op_assign 0 2 2024-04-14T15:26:09.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.145+0000 7ff61eea3700 20 client.38085.objecter put_session s=0x7ff6181121b0 osd=0 4 2024-04-14T15:26:09.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.145+0000 7ff61eea3700 10 client.38085.objecter _send_command 2 2024-04-14T15:26:09.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.145+0000 7ff61eea3700 1 -- 172.21.15.90:0/3919189616 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff6180693d0 con 0x7ff6181aaf70 2024-04-14T15:26:09.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.147+0000 7ff615ffb700 1 -- 172.21.15.90:0/3919189616 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff6180693d0 con 0x7ff6181aaf70 2024-04-14T15:26:09.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.147+0000 7ff615ffb700 10 client.38085.objecter ms_dispatch 0x7ff618001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:26:09.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.147+0000 7ff615ffb700 10 client.38085.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:26:09.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.147+0000 7ff615ffb700 20 client.38085.objecter put_session s=0x7ff6181121b0 osd=0 4 2024-04-14T15:26:09.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.147+0000 7ff615ffb700 15 client.38085.objecter _session_command_op_remove 0 2 2024-04-14T15:26:09.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.147+0000 7ff61eea3700 10 client.38085.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:26:09.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.147+0000 7ff61eea3700 15 client.38085.objecter _session_command_op_assign -1 3 2024-04-14T15:26:09.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.147+0000 7ff61eea3700 20 client.38085.objecter _get_session s=0x7ff6181abcb0 osd=1 3 2024-04-14T15:26:09.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.147+0000 7ff61eea3700 20 client.38085.objecter put_session s=0x7ff6181abcb0 osd=1 3 2024-04-14T15:26:09.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.147+0000 7ff61eea3700 20 client.38085.objecter _get_session s=0x7ff6181abcb0 osd=1 3 2024-04-14T15:26:09.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.147+0000 7ff61eea3700 15 client.38085.objecter _session_command_op_remove -1 3 2024-04-14T15:26:09.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.147+0000 7ff61eea3700 20 client.38085.objecter get_session s=0x7ff6181abcb0 osd=1 3 2024-04-14T15:26:09.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.147+0000 7ff61eea3700 15 client.38085.objecter _session_command_op_assign 1 3 2024-04-14T15:26:09.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.147+0000 7ff61eea3700 20 client.38085.objecter put_session s=0x7ff6181abcb0 osd=1 4 2024-04-14T15:26:09.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.147+0000 7ff61eea3700 10 client.38085.objecter _send_command 3 2024-04-14T15:26:09.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.147+0000 7ff61eea3700 1 -- 172.21.15.90:0/3919189616 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7ff61802de30 con 0x7ff6180fdb80 2024-04-14T15:26:09.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.148+0000 7ff615ffb700 10 client.38085.objecter ms_handle_connect 0x7ff6180fdb80 2024-04-14T15:26:09.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.148+0000 7ff615ffb700 1 -- 172.21.15.90:0/3919189616 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7ff61802de30 con 0x7ff6180fdb80 2024-04-14T15:26:09.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.148+0000 7ff615ffb700 10 client.38085.objecter ms_dispatch 0x7ff618001550 command_reply(tid 3: 0 ) 2024-04-14T15:26:09.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.148+0000 7ff615ffb700 10 client.38085.objecter _finish_command 3 = system:0 2024-04-14T15:26:09.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.148+0000 7ff615ffb700 20 client.38085.objecter put_session s=0x7ff6181abcb0 osd=1 4 2024-04-14T15:26:09.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.148+0000 7ff615ffb700 15 client.38085.objecter _session_command_op_remove 1 3 2024-04-14T15:26:09.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.165+0000 7ff61eea3700 10 client.38085.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:26:09.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.165+0000 7ff61eea3700 15 client.38085.objecter _session_command_op_assign -1 4 2024-04-14T15:26:09.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.165+0000 7ff61eea3700 20 client.38085.objecter _get_session s=0x7ff6181abcb0 osd=1 3 2024-04-14T15:26:09.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.165+0000 7ff61eea3700 20 client.38085.objecter put_session s=0x7ff6181abcb0 osd=1 3 2024-04-14T15:26:09.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.165+0000 7ff61eea3700 20 client.38085.objecter _get_session s=0x7ff6181abcb0 osd=1 3 2024-04-14T15:26:09.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.165+0000 7ff61eea3700 15 client.38085.objecter _session_command_op_remove -1 4 2024-04-14T15:26:09.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.165+0000 7ff61eea3700 20 client.38085.objecter get_session s=0x7ff6181abcb0 osd=1 3 2024-04-14T15:26:09.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.165+0000 7ff61eea3700 15 client.38085.objecter _session_command_op_assign 1 4 2024-04-14T15:26:09.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.165+0000 7ff61eea3700 20 client.38085.objecter put_session s=0x7ff6181abcb0 osd=1 4 2024-04-14T15:26:09.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.165+0000 7ff61eea3700 10 client.38085.objecter _send_command 4 2024-04-14T15:26:09.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.165+0000 7ff61eea3700 1 -- 172.21.15.90:0/3919189616 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff61802e230 con 0x7ff6180fdb80 2024-04-14T15:26:09.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.166+0000 7ff615ffb700 1 -- 172.21.15.90:0/3919189616 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff61802e230 con 0x7ff6180fdb80 2024-04-14T15:26:09.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.166+0000 7ff615ffb700 10 client.38085.objecter ms_dispatch 0x7ff618001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:26:09.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.166+0000 7ff615ffb700 10 client.38085.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:26:09.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.166+0000 7ff615ffb700 20 client.38085.objecter put_session s=0x7ff6181abcb0 osd=1 4 2024-04-14T15:26:09.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.166+0000 7ff615ffb700 15 client.38085.objecter _session_command_op_remove 1 4 2024-04-14T15:26:09.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.166+0000 7ff61eea3700 10 client.38085.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:26:09.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.166+0000 7ff61eea3700 15 client.38085.objecter _session_command_op_assign -1 5 2024-04-14T15:26:09.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.166+0000 7ff61eea3700 20 client.38085.objecter _get_session s=0x7ff61802e3e0 osd=2 3 2024-04-14T15:26:09.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.166+0000 7ff61eea3700 20 client.38085.objecter put_session s=0x7ff61802e3e0 osd=2 3 2024-04-14T15:26:09.167 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:26:09.167 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:26:09.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.167+0000 7ff61eea3700 20 client.38085.objecter _get_session s=0x7ff61802e3e0 osd=2 3 2024-04-14T15:26:09.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.167+0000 7ff61eea3700 15 client.38085.objecter _session_command_op_remove -1 5 2024-04-14T15:26:09.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.167+0000 7ff61eea3700 20 client.38085.objecter get_session s=0x7ff61802e3e0 osd=2 3 2024-04-14T15:26:09.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.167+0000 7ff61eea3700 15 client.38085.objecter _session_command_op_assign 2 5 2024-04-14T15:26:09.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.167+0000 7ff61eea3700 20 client.38085.objecter put_session s=0x7ff61802e3e0 osd=2 4 2024-04-14T15:26:09.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.167+0000 7ff61eea3700 10 client.38085.objecter _send_command 5 2024-04-14T15:26:09.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.167+0000 7ff61eea3700 1 -- 172.21.15.90:0/3919189616 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7ff618063b70 con 0x7ff6180636c0 2024-04-14T15:26:09.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.167+0000 7ff615ffb700 10 client.38085.objecter ms_handle_connect 0x7ff6180636c0 2024-04-14T15:26:09.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.167+0000 7ff615ffb700 1 -- 172.21.15.90:0/3919189616 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7ff618063b70 con 0x7ff6180636c0 2024-04-14T15:26:09.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.167+0000 7ff615ffb700 10 client.38085.objecter ms_dispatch 0x7ff618001550 command_reply(tid 5: 0 ) 2024-04-14T15:26:09.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.167+0000 7ff615ffb700 10 client.38085.objecter _finish_command 5 = system:0 2024-04-14T15:26:09.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.167+0000 7ff615ffb700 20 client.38085.objecter put_session s=0x7ff61802e3e0 osd=2 4 2024-04-14T15:26:09.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.167+0000 7ff615ffb700 15 client.38085.objecter _session_command_op_remove 2 5 2024-04-14T15:26:09.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.184+0000 7ff61eea3700 10 client.38085.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:26:09.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.184+0000 7ff61eea3700 15 client.38085.objecter _session_command_op_assign -1 6 2024-04-14T15:26:09.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.184+0000 7ff61eea3700 20 client.38085.objecter _get_session s=0x7ff61802e3e0 osd=2 3 2024-04-14T15:26:09.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.184+0000 7ff61eea3700 20 client.38085.objecter put_session s=0x7ff61802e3e0 osd=2 3 2024-04-14T15:26:09.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.184+0000 7ff61eea3700 20 client.38085.objecter _get_session s=0x7ff61802e3e0 osd=2 3 2024-04-14T15:26:09.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.184+0000 7ff61eea3700 15 client.38085.objecter _session_command_op_remove -1 6 2024-04-14T15:26:09.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.184+0000 7ff61eea3700 20 client.38085.objecter get_session s=0x7ff61802e3e0 osd=2 3 2024-04-14T15:26:09.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.184+0000 7ff61eea3700 15 client.38085.objecter _session_command_op_assign 2 6 2024-04-14T15:26:09.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.184+0000 7ff61eea3700 20 client.38085.objecter put_session s=0x7ff61802e3e0 osd=2 4 2024-04-14T15:26:09.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.184+0000 7ff61eea3700 10 client.38085.objecter _send_command 6 2024-04-14T15:26:09.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.184+0000 7ff61eea3700 1 -- 172.21.15.90:0/3919189616 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff6181b5d80 con 0x7ff6180636c0 2024-04-14T15:26:09.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.186+0000 7ff615ffb700 1 -- 172.21.15.90:0/3919189616 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff6181b5d80 con 0x7ff6180636c0 2024-04-14T15:26:09.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.186+0000 7ff615ffb700 10 client.38085.objecter ms_dispatch 0x7ff618001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:26:09.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.186+0000 7ff615ffb700 10 client.38085.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:26:09.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.186+0000 7ff615ffb700 20 client.38085.objecter put_session s=0x7ff61802e3e0 osd=2 4 2024-04-14T15:26:09.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.186+0000 7ff615ffb700 15 client.38085.objecter _session_command_op_remove 2 6 2024-04-14T15:26:09.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.186+0000 7ff61eea3700 10 client.38085.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:26:09.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.186+0000 7ff61eea3700 15 client.38085.objecter _session_command_op_assign -1 7 2024-04-14T15:26:09.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.186+0000 7ff61eea3700 20 client.38085.objecter _get_session s=0x7ff618063140 osd=3 3 2024-04-14T15:26:09.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.186+0000 7ff61eea3700 20 client.38085.objecter put_session s=0x7ff618063140 osd=3 3 2024-04-14T15:26:09.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.186+0000 7ff61eea3700 20 client.38085.objecter _get_session s=0x7ff618063140 osd=3 3 2024-04-14T15:26:09.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.186+0000 7ff61eea3700 15 client.38085.objecter _session_command_op_remove -1 7 2024-04-14T15:26:09.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.186+0000 7ff61eea3700 20 client.38085.objecter get_session s=0x7ff618063140 osd=3 3 2024-04-14T15:26:09.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.186+0000 7ff61eea3700 15 client.38085.objecter _session_command_op_assign 3 7 2024-04-14T15:26:09.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.186+0000 7ff61eea3700 20 client.38085.objecter put_session s=0x7ff618063140 osd=3 4 2024-04-14T15:26:09.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.186+0000 7ff61eea3700 10 client.38085.objecter _send_command 7 2024-04-14T15:26:09.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.186+0000 7ff61eea3700 1 -- 172.21.15.90:0/3919189616 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7ff6181b2220 con 0x7ff6180632b0 2024-04-14T15:26:09.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.186+0000 7ff615ffb700 10 client.38085.objecter ms_handle_connect 0x7ff6180632b0 2024-04-14T15:26:09.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.187+0000 7ff615ffb700 1 -- 172.21.15.90:0/3919189616 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7ff6181b2220 con 0x7ff6180632b0 2024-04-14T15:26:09.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.187+0000 7ff615ffb700 10 client.38085.objecter ms_dispatch 0x7ff618001550 command_reply(tid 7: 0 ) 2024-04-14T15:26:09.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.187+0000 7ff615ffb700 10 client.38085.objecter _finish_command 7 = system:0 2024-04-14T15:26:09.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.187+0000 7ff615ffb700 20 client.38085.objecter put_session s=0x7ff618063140 osd=3 4 2024-04-14T15:26:09.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.187+0000 7ff615ffb700 15 client.38085.objecter _session_command_op_remove 3 7 2024-04-14T15:26:09.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.205+0000 7ff61eea3700 10 client.38085.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:26:09.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.205+0000 7ff61eea3700 15 client.38085.objecter _session_command_op_assign -1 8 2024-04-14T15:26:09.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.205+0000 7ff61eea3700 20 client.38085.objecter _get_session s=0x7ff618063140 osd=3 3 2024-04-14T15:26:09.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.205+0000 7ff61eea3700 20 client.38085.objecter put_session s=0x7ff618063140 osd=3 3 2024-04-14T15:26:09.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.205+0000 7ff61eea3700 20 client.38085.objecter _get_session s=0x7ff618063140 osd=3 3 2024-04-14T15:26:09.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.205+0000 7ff61eea3700 15 client.38085.objecter _session_command_op_remove -1 8 2024-04-14T15:26:09.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.205+0000 7ff61eea3700 20 client.38085.objecter get_session s=0x7ff618063140 osd=3 3 2024-04-14T15:26:09.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.205+0000 7ff61eea3700 15 client.38085.objecter _session_command_op_assign 3 8 2024-04-14T15:26:09.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.205+0000 7ff61eea3700 20 client.38085.objecter put_session s=0x7ff618063140 osd=3 4 2024-04-14T15:26:09.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.205+0000 7ff61eea3700 10 client.38085.objecter _send_command 8 2024-04-14T15:26:09.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.205+0000 7ff61eea3700 1 -- 172.21.15.90:0/3919189616 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff6181b24a0 con 0x7ff6180632b0 2024-04-14T15:26:09.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.206+0000 7ff615ffb700 1 -- 172.21.15.90:0/3919189616 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff6181b24a0 con 0x7ff6180632b0 2024-04-14T15:26:09.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.206+0000 7ff615ffb700 10 client.38085.objecter ms_dispatch 0x7ff618001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:26:09.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.206+0000 7ff615ffb700 10 client.38085.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:26:09.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.206+0000 7ff615ffb700 20 client.38085.objecter put_session s=0x7ff618063140 osd=3 4 2024-04-14T15:26:09.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.206+0000 7ff615ffb700 15 client.38085.objecter _session_command_op_remove 3 8 2024-04-14T15:26:09.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.207+0000 7ff61eea3700 10 client.38085.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:26:09.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.207+0000 7ff61eea3700 15 client.38085.objecter _session_command_op_assign -1 9 2024-04-14T15:26:09.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.207+0000 7ff61eea3700 20 client.38085.objecter _get_session s=0x7ff6181b24a0 osd=4 3 2024-04-14T15:26:09.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.207+0000 7ff61eea3700 20 client.38085.objecter put_session s=0x7ff6181b24a0 osd=4 3 2024-04-14T15:26:09.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.207+0000 7ff61eea3700 20 client.38085.objecter _get_session s=0x7ff6181b24a0 osd=4 3 2024-04-14T15:26:09.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.207+0000 7ff61eea3700 15 client.38085.objecter _session_command_op_remove -1 9 2024-04-14T15:26:09.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.207+0000 7ff61eea3700 20 client.38085.objecter get_session s=0x7ff6181b24a0 osd=4 3 2024-04-14T15:26:09.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.207+0000 7ff61eea3700 15 client.38085.objecter _session_command_op_assign 4 9 2024-04-14T15:26:09.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.207+0000 7ff61eea3700 20 client.38085.objecter put_session s=0x7ff6181b24a0 osd=4 4 2024-04-14T15:26:09.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.207+0000 7ff61eea3700 10 client.38085.objecter _send_command 9 2024-04-14T15:26:09.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.207+0000 7ff61eea3700 1 -- 172.21.15.90:0/3919189616 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7ff6181b6c70 con 0x7ff6181b6700 2024-04-14T15:26:09.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.207+0000 7ff615ffb700 10 client.38085.objecter ms_handle_connect 0x7ff6181b6700 2024-04-14T15:26:09.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.208+0000 7ff615ffb700 1 -- 172.21.15.90:0/3919189616 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7ff6181b6c70 con 0x7ff6181b6700 2024-04-14T15:26:09.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.208+0000 7ff615ffb700 10 client.38085.objecter ms_dispatch 0x7ff618001550 command_reply(tid 9: 0 ) 2024-04-14T15:26:09.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.208+0000 7ff615ffb700 10 client.38085.objecter _finish_command 9 = system:0 2024-04-14T15:26:09.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.208+0000 7ff615ffb700 20 client.38085.objecter put_session s=0x7ff6181b24a0 osd=4 4 2024-04-14T15:26:09.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.208+0000 7ff615ffb700 15 client.38085.objecter _session_command_op_remove 4 9 2024-04-14T15:26:09.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.224+0000 7ff61eea3700 10 client.38085.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:26:09.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.224+0000 7ff61eea3700 15 client.38085.objecter _session_command_op_assign -1 10 2024-04-14T15:26:09.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.224+0000 7ff61eea3700 20 client.38085.objecter _get_session s=0x7ff6181b24a0 osd=4 3 2024-04-14T15:26:09.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.224+0000 7ff61eea3700 20 client.38085.objecter put_session s=0x7ff6181b24a0 osd=4 3 2024-04-14T15:26:09.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.224+0000 7ff61eea3700 20 client.38085.objecter _get_session s=0x7ff6181b24a0 osd=4 3 2024-04-14T15:26:09.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.224+0000 7ff61eea3700 15 client.38085.objecter _session_command_op_remove -1 10 2024-04-14T15:26:09.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.224+0000 7ff61eea3700 20 client.38085.objecter get_session s=0x7ff6181b24a0 osd=4 3 2024-04-14T15:26:09.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.224+0000 7ff61eea3700 15 client.38085.objecter _session_command_op_assign 4 10 2024-04-14T15:26:09.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.224+0000 7ff61eea3700 20 client.38085.objecter put_session s=0x7ff6181b24a0 osd=4 4 2024-04-14T15:26:09.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.224+0000 7ff61eea3700 10 client.38085.objecter _send_command 10 2024-04-14T15:26:09.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.224+0000 7ff61eea3700 1 -- 172.21.15.90:0/3919189616 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff6181b6ea0 con 0x7ff6181b6700 2024-04-14T15:26:09.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.226+0000 7ff615ffb700 1 -- 172.21.15.90:0/3919189616 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff6181b6ea0 con 0x7ff6181b6700 2024-04-14T15:26:09.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.226+0000 7ff615ffb700 10 client.38085.objecter ms_dispatch 0x7ff618001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:26:09.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.226+0000 7ff615ffb700 10 client.38085.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:26:09.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.226+0000 7ff615ffb700 20 client.38085.objecter put_session s=0x7ff6181b24a0 osd=4 4 2024-04-14T15:26:09.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.226+0000 7ff615ffb700 15 client.38085.objecter _session_command_op_remove 4 10 2024-04-14T15:26:09.174 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:26:09.175 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:26:09.175 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:26:09.175 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:26:09.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.227+0000 7ff61eea3700 10 client.38085.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:26:09.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.227+0000 7ff61eea3700 15 client.38085.objecter _session_command_op_assign -1 11 2024-04-14T15:26:09.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.227+0000 7ff61eea3700 20 client.38085.objecter _get_session s=0x7ff6181b6ea0 osd=5 3 2024-04-14T15:26:09.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.227+0000 7ff61eea3700 20 client.38085.objecter put_session s=0x7ff6181b6ea0 osd=5 3 2024-04-14T15:26:09.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.227+0000 7ff61eea3700 20 client.38085.objecter _get_session s=0x7ff6181b6ea0 osd=5 3 2024-04-14T15:26:09.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.227+0000 7ff61eea3700 15 client.38085.objecter _session_command_op_remove -1 11 2024-04-14T15:26:09.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.227+0000 7ff61eea3700 20 client.38085.objecter get_session s=0x7ff6181b6ea0 osd=5 3 2024-04-14T15:26:09.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.227+0000 7ff61eea3700 15 client.38085.objecter _session_command_op_assign 5 11 2024-04-14T15:26:09.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.227+0000 7ff61eea3700 20 client.38085.objecter put_session s=0x7ff6181b6ea0 osd=5 4 2024-04-14T15:26:09.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.227+0000 7ff61eea3700 10 client.38085.objecter _send_command 11 2024-04-14T15:26:09.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.227+0000 7ff61eea3700 1 -- 172.21.15.90:0/3919189616 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7ff6181c31f0 con 0x7ff6181bf5a0 2024-04-14T15:26:09.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.227+0000 7ff615ffb700 10 client.38085.objecter ms_handle_connect 0x7ff6181bf5a0 2024-04-14T15:26:09.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.228+0000 7ff615ffb700 1 -- 172.21.15.90:0/3919189616 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7ff6181c31f0 con 0x7ff6181bf5a0 2024-04-14T15:26:09.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.228+0000 7ff615ffb700 10 client.38085.objecter ms_dispatch 0x7ff618001550 command_reply(tid 11: 0 ) 2024-04-14T15:26:09.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.228+0000 7ff615ffb700 10 client.38085.objecter _finish_command 11 = system:0 2024-04-14T15:26:09.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.228+0000 7ff615ffb700 20 client.38085.objecter put_session s=0x7ff6181b6ea0 osd=5 4 2024-04-14T15:26:09.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.228+0000 7ff615ffb700 15 client.38085.objecter _session_command_op_remove 5 11 2024-04-14T15:26:09.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.244+0000 7ff61eea3700 10 client.38085.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:26:09.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.244+0000 7ff61eea3700 15 client.38085.objecter _session_command_op_assign -1 12 2024-04-14T15:26:09.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.244+0000 7ff61eea3700 20 client.38085.objecter _get_session s=0x7ff6181b6ea0 osd=5 3 2024-04-14T15:26:09.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.244+0000 7ff61eea3700 20 client.38085.objecter put_session s=0x7ff6181b6ea0 osd=5 3 2024-04-14T15:26:09.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.244+0000 7ff61eea3700 20 client.38085.objecter _get_session s=0x7ff6181b6ea0 osd=5 3 2024-04-14T15:26:09.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.244+0000 7ff61eea3700 15 client.38085.objecter _session_command_op_remove -1 12 2024-04-14T15:26:09.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.244+0000 7ff61eea3700 20 client.38085.objecter get_session s=0x7ff6181b6ea0 osd=5 3 2024-04-14T15:26:09.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.244+0000 7ff61eea3700 15 client.38085.objecter _session_command_op_assign 5 12 2024-04-14T15:26:09.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.244+0000 7ff61eea3700 20 client.38085.objecter put_session s=0x7ff6181b6ea0 osd=5 4 2024-04-14T15:26:09.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.244+0000 7ff61eea3700 10 client.38085.objecter _send_command 12 2024-04-14T15:26:09.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.244+0000 7ff61eea3700 1 -- 172.21.15.90:0/3919189616 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff6181c4ba0 con 0x7ff6181bf5a0 2024-04-14T15:26:09.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.247+0000 7ff615ffb700 1 -- 172.21.15.90:0/3919189616 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff6181c4ba0 con 0x7ff6181bf5a0 2024-04-14T15:26:09.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.247+0000 7ff615ffb700 10 client.38085.objecter ms_dispatch 0x7ff618001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:26:09.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.247+0000 7ff615ffb700 10 client.38085.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:26:09.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.247+0000 7ff615ffb700 20 client.38085.objecter put_session s=0x7ff6181b6ea0 osd=5 4 2024-04-14T15:26:09.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.247+0000 7ff615ffb700 15 client.38085.objecter _session_command_op_remove 5 12 2024-04-14T15:26:09.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.247+0000 7ff61eea3700 10 client.38085.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:26:09.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.247+0000 7ff61eea3700 15 client.38085.objecter _session_command_op_assign -1 13 2024-04-14T15:26:09.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.247+0000 7ff61eea3700 20 client.38085.objecter _get_session s=0x7ff6181c4ba0 osd=6 3 2024-04-14T15:26:09.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.247+0000 7ff61eea3700 20 client.38085.objecter put_session s=0x7ff6181c4ba0 osd=6 3 2024-04-14T15:26:09.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.247+0000 7ff61eea3700 20 client.38085.objecter _get_session s=0x7ff6181c4ba0 osd=6 3 2024-04-14T15:26:09.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.247+0000 7ff61eea3700 15 client.38085.objecter _session_command_op_remove -1 13 2024-04-14T15:26:09.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.247+0000 7ff61eea3700 20 client.38085.objecter get_session s=0x7ff6181c4ba0 osd=6 3 2024-04-14T15:26:09.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.247+0000 7ff61eea3700 15 client.38085.objecter _session_command_op_assign 6 13 2024-04-14T15:26:09.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.247+0000 7ff61eea3700 20 client.38085.objecter put_session s=0x7ff6181c4ba0 osd=6 4 2024-04-14T15:26:09.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.247+0000 7ff61eea3700 10 client.38085.objecter _send_command 13 2024-04-14T15:26:09.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.247+0000 7ff61eea3700 1 -- 172.21.15.90:0/3919189616 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7ff6181c49b0 con 0x7ff6181c52c0 2024-04-14T15:26:09.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.247+0000 7ff615ffb700 10 client.38085.objecter ms_handle_connect 0x7ff6181c52c0 2024-04-14T15:26:09.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.248+0000 7ff615ffb700 1 -- 172.21.15.90:0/3919189616 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7ff6080063d0 con 0x7ff6181c52c0 2024-04-14T15:26:09.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.248+0000 7ff615ffb700 10 client.38085.objecter ms_dispatch 0x7ff618001550 command_reply(tid 13: 0 ) 2024-04-14T15:26:09.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.248+0000 7ff615ffb700 10 client.38085.objecter _finish_command 13 = system:0 2024-04-14T15:26:09.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.248+0000 7ff615ffb700 20 client.38085.objecter put_session s=0x7ff6181c4ba0 osd=6 4 2024-04-14T15:26:09.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.248+0000 7ff615ffb700 15 client.38085.objecter _session_command_op_remove 6 13 2024-04-14T15:26:09.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.265+0000 7ff61eea3700 10 client.38085.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:26:09.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.265+0000 7ff61eea3700 15 client.38085.objecter _session_command_op_assign -1 14 2024-04-14T15:26:09.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.265+0000 7ff61eea3700 20 client.38085.objecter _get_session s=0x7ff6181c4ba0 osd=6 3 2024-04-14T15:26:09.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.265+0000 7ff61eea3700 20 client.38085.objecter put_session s=0x7ff6181c4ba0 osd=6 3 2024-04-14T15:26:09.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.265+0000 7ff61eea3700 20 client.38085.objecter _get_session s=0x7ff6181c4ba0 osd=6 3 2024-04-14T15:26:09.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.265+0000 7ff61eea3700 15 client.38085.objecter _session_command_op_remove -1 14 2024-04-14T15:26:09.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.265+0000 7ff61eea3700 20 client.38085.objecter get_session s=0x7ff6181c4ba0 osd=6 3 2024-04-14T15:26:09.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.265+0000 7ff61eea3700 15 client.38085.objecter _session_command_op_assign 6 14 2024-04-14T15:26:09.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.265+0000 7ff61eea3700 20 client.38085.objecter put_session s=0x7ff6181c4ba0 osd=6 4 2024-04-14T15:26:09.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.265+0000 7ff61eea3700 10 client.38085.objecter _send_command 14 2024-04-14T15:26:09.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.265+0000 7ff61eea3700 1 -- 172.21.15.90:0/3919189616 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff6181c5890 con 0x7ff6181c52c0 2024-04-14T15:26:09.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.267+0000 7ff615ffb700 1 -- 172.21.15.90:0/3919189616 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff6181c5890 con 0x7ff6181c52c0 2024-04-14T15:26:09.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.267+0000 7ff615ffb700 10 client.38085.objecter ms_dispatch 0x7ff618001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:26:09.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.267+0000 7ff615ffb700 10 client.38085.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:26:09.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.267+0000 7ff615ffb700 20 client.38085.objecter put_session s=0x7ff6181c4ba0 osd=6 4 2024-04-14T15:26:09.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.267+0000 7ff615ffb700 15 client.38085.objecter _session_command_op_remove 6 14 2024-04-14T15:26:09.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.267+0000 7ff61eea3700 10 client.38085.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:26:09.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.267+0000 7ff61eea3700 15 client.38085.objecter _session_command_op_assign -1 15 2024-04-14T15:26:09.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.268+0000 7ff61eea3700 20 client.38085.objecter _get_session s=0x7ff6181c5890 osd=7 3 2024-04-14T15:26:09.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.268+0000 7ff61eea3700 20 client.38085.objecter put_session s=0x7ff6181c5890 osd=7 3 2024-04-14T15:26:09.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.268+0000 7ff61eea3700 20 client.38085.objecter _get_session s=0x7ff6181c5890 osd=7 3 2024-04-14T15:26:09.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.268+0000 7ff61eea3700 15 client.38085.objecter _session_command_op_remove -1 15 2024-04-14T15:26:09.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.268+0000 7ff61eea3700 20 client.38085.objecter get_session s=0x7ff6181c5890 osd=7 3 2024-04-14T15:26:09.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.268+0000 7ff61eea3700 15 client.38085.objecter _session_command_op_assign 7 15 2024-04-14T15:26:09.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.268+0000 7ff61eea3700 20 client.38085.objecter put_session s=0x7ff6181c5890 osd=7 4 2024-04-14T15:26:09.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.268+0000 7ff61eea3700 10 client.38085.objecter _send_command 15 2024-04-14T15:26:09.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.268+0000 7ff61eea3700 1 -- 172.21.15.90:0/3919189616 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7ff6181cbda0 con 0x7ff6181c8290 2024-04-14T15:26:09.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.268+0000 7ff615ffb700 10 client.38085.objecter ms_handle_connect 0x7ff6181c8290 2024-04-14T15:26:09.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.269+0000 7ff615ffb700 1 -- 172.21.15.90:0/3919189616 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7ff6181cbda0 con 0x7ff6181c8290 2024-04-14T15:26:09.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.269+0000 7ff615ffb700 10 client.38085.objecter ms_dispatch 0x7ff618001550 command_reply(tid 15: 0 ) 2024-04-14T15:26:09.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.269+0000 7ff615ffb700 10 client.38085.objecter _finish_command 15 = system:0 2024-04-14T15:26:09.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.269+0000 7ff615ffb700 20 client.38085.objecter put_session s=0x7ff6181c5890 osd=7 4 2024-04-14T15:26:09.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.269+0000 7ff615ffb700 15 client.38085.objecter _session_command_op_remove 7 15 2024-04-14T15:26:09.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.287+0000 7ff61eea3700 10 client.38085.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:26:09.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.287+0000 7ff61eea3700 15 client.38085.objecter _session_command_op_assign -1 16 2024-04-14T15:26:09.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.287+0000 7ff61eea3700 20 client.38085.objecter _get_session s=0x7ff6181c5890 osd=7 3 2024-04-14T15:26:09.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.287+0000 7ff61eea3700 20 client.38085.objecter put_session s=0x7ff6181c5890 osd=7 3 2024-04-14T15:26:09.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.287+0000 7ff61eea3700 20 client.38085.objecter _get_session s=0x7ff6181c5890 osd=7 3 2024-04-14T15:26:09.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.287+0000 7ff61eea3700 15 client.38085.objecter _session_command_op_remove -1 16 2024-04-14T15:26:09.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.287+0000 7ff61eea3700 20 client.38085.objecter get_session s=0x7ff6181c5890 osd=7 3 2024-04-14T15:26:09.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.287+0000 7ff61eea3700 15 client.38085.objecter _session_command_op_assign 7 16 2024-04-14T15:26:09.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.287+0000 7ff61eea3700 20 client.38085.objecter put_session s=0x7ff6181c5890 osd=7 4 2024-04-14T15:26:09.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.287+0000 7ff61eea3700 10 client.38085.objecter _send_command 16 2024-04-14T15:26:09.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.287+0000 7ff61eea3700 1 -- 172.21.15.90:0/3919189616 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff6181b1530 con 0x7ff6181c8290 2024-04-14T15:26:09.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.289+0000 7ff615ffb700 1 -- 172.21.15.90:0/3919189616 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff6181b1530 con 0x7ff6181c8290 2024-04-14T15:26:09.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.289+0000 7ff615ffb700 10 client.38085.objecter ms_dispatch 0x7ff618001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:26:09.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.289+0000 7ff615ffb700 10 client.38085.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:26:09.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.289+0000 7ff615ffb700 20 client.38085.objecter put_session s=0x7ff6181c5890 osd=7 4 2024-04-14T15:26:09.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.289+0000 7ff615ffb700 15 client.38085.objecter _session_command_op_remove 7 16 2024-04-14T15:26:09.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.290+0000 7ff61eea3700 10 librados: watch_flush enter 2024-04-14T15:26:09.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.290+0000 7ff61eea3700 10 librados: watch_flush exit 2024-04-14T15:26:09.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.290+0000 7ff61eea3700 10 client.38085.objecter close_session for osd.0 2024-04-14T15:26:09.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.290+0000 7ff61eea3700 1 -- 172.21.15.90:0/3919189616 >> v1:172.21.15.90:6801/3761806885 conn(0x7ff6181aaf70 legacy=0x7ff6181ab350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:09.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.290+0000 7ff61eea3700 20 client.38085.objecter put_session s=0x7ff6181121b0 osd=0 1 2024-04-14T15:26:09.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.290+0000 7ff61eea3700 10 client.38085.objecter close_session for osd.1 2024-04-14T15:26:09.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.290+0000 7ff61eea3700 1 -- 172.21.15.90:0/3919189616 >> v1:172.21.15.90:6805/4170361699 conn(0x7ff6180fdb80 legacy=0x7ff6180fdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:09.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.290+0000 7ff61eea3700 20 client.38085.objecter put_session s=0x7ff6181abcb0 osd=1 1 2024-04-14T15:26:09.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.290+0000 7ff61eea3700 10 client.38085.objecter close_session for osd.2 2024-04-14T15:26:09.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.290+0000 7ff61eea3700 1 -- 172.21.15.90:0/3919189616 >> v1:172.21.15.90:6809/1686184251 conn(0x7ff6180636c0 legacy=0x7ff6181b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:09.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.290+0000 7ff61eea3700 20 client.38085.objecter put_session s=0x7ff61802e3e0 osd=2 1 2024-04-14T15:26:09.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.290+0000 7ff61eea3700 10 client.38085.objecter close_session for osd.3 2024-04-14T15:26:09.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.290+0000 7ff61eea3700 1 -- 172.21.15.90:0/3919189616 >> v1:172.21.15.90:6813/2270166774 conn(0x7ff6180632b0 legacy=0x7ff6181b90c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:09.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.290+0000 7ff61eea3700 20 client.38085.objecter put_session s=0x7ff618063140 osd=3 1 2024-04-14T15:26:09.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.290+0000 7ff61eea3700 10 client.38085.objecter close_session for osd.4 2024-04-14T15:26:09.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.290+0000 7ff61eea3700 1 -- 172.21.15.90:0/3919189616 >> v1:172.21.15.92:6800/3887549733 conn(0x7ff6181b6700 legacy=0x7ff6181bd970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:09.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.290+0000 7ff61eea3700 20 client.38085.objecter put_session s=0x7ff6181b24a0 osd=4 1 2024-04-14T15:26:09.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.290+0000 7ff61d440700 1 -- 172.21.15.90:0/3919189616 reap_dead start 2024-04-14T15:26:09.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.290+0000 7ff61eea3700 10 client.38085.objecter close_session for osd.5 2024-04-14T15:26:09.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.290+0000 7ff61eea3700 1 -- 172.21.15.90:0/3919189616 >> v1:172.21.15.92:6804/3244561440 conn(0x7ff6181bf5a0 legacy=0x7ff6181c1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:09.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.290+0000 7ff61d440700 1 -- 172.21.15.90:0/3919189616 reap_dead start 2024-04-14T15:26:09.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.290+0000 7ff61eea3700 20 client.38085.objecter put_session s=0x7ff6181b6ea0 osd=5 1 2024-04-14T15:26:09.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.290+0000 7ff61eea3700 10 client.38085.objecter close_session for osd.6 2024-04-14T15:26:09.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.290+0000 7ff61eea3700 1 -- 172.21.15.90:0/3919189616 >> v1:172.21.15.92:6808/636125963 conn(0x7ff6181c52c0 legacy=0x7ff6181c31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:09.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.290+0000 7ff61eea3700 20 client.38085.objecter put_session s=0x7ff6181c4ba0 osd=6 1 2024-04-14T15:26:09.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.290+0000 7ff61eea3700 10 client.38085.objecter close_session for osd.7 2024-04-14T15:26:09.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.290+0000 7ff61eea3700 1 -- 172.21.15.90:0/3919189616 >> v1:172.21.15.92:6812/1773706716 conn(0x7ff6181c8290 legacy=0x7ff6181ca680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:09.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.290+0000 7ff61eea3700 20 client.38085.objecter put_session s=0x7ff6181c5890 osd=7 1 2024-04-14T15:26:09.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.290+0000 7ff61eea3700 20 client.38085.objecter shutdown clearing up homeless session... 2024-04-14T15:26:09.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.290+0000 7ff61eea3700 10 client.38085.objecter successfully canceled tick 2024-04-14T15:26:09.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.290+0000 7ff61d440700 1 -- 172.21.15.90:0/3919189616 reap_dead start 2024-04-14T15:26:09.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.290+0000 7ff61eea3700 1 -- 172.21.15.90:0/3919189616 >> v1:172.21.15.90:6800/420935427 conn(0x7ff600071290 legacy=0x7ff600073750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:09.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.290+0000 7ff61eea3700 1 -- 172.21.15.90:0/3919189616 >> v1:172.21.15.90:6789/0 conn(0x7ff61810a610 legacy=0x7ff6181a4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:09.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.290+0000 7ff61eea3700 1 -- 172.21.15.90:0/3919189616 shutdown_connections 2024-04-14T15:26:09.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.290+0000 7ff61eea3700 1 -- 172.21.15.90:0/3919189616 >> 172.21.15.90:0/3919189616 conn(0x7ff6180fd290 msgr2=0x7ff61806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:26:09.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.290+0000 7ff61eea3700 1 -- 172.21.15.90:0/3919189616 shutdown_connections 2024-04-14T15:26:09.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.290+0000 7ff61eea3700 1 -- 172.21.15.90:0/3919189616 wait complete. 2024-04-14T15:26:09.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:08.290+0000 7ff61eea3700 1 librados: shutdown 2024-04-14T15:26:09.185 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:26:09.185 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:26:09.185 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:26:09.185 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:26:09.185 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:26:09.185 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:26:09.185 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:26:09.185 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:26:09.192 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:26:09.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.214+0000 7f996b216700 1 -- 172.21.15.90:0/149769965 --> v1:172.21.15.90:6800/420935427 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f996410c2a0 con 0x7f99540713a0 2024-04-14T15:26:09.215 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:09.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:09.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:26:09.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.229+0000 7f9961ffb700 1 -- 172.21.15.90:0/149769965 <== mgr.14610 v1:172.21.15.90:6800/420935427 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+578398 (unknown 2965378022 0 3437249231) 0x7f996410c2a0 con 0x7f99540713a0 2024-04-14T15:26:09.236 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:26:09.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.234+0000 7f996b216700 10 librados: watch_flush enter 2024-04-14T15:26:09.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.235+0000 7f996b216700 10 librados: watch_flush exit 2024-04-14T15:26:09.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.235+0000 7f996b216700 20 client.38091.objecter shutdown clearing up homeless session... 2024-04-14T15:26:09.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.235+0000 7f996b216700 10 client.38091.objecter successfully canceled tick 2024-04-14T15:26:09.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.235+0000 7f996b216700 1 -- 172.21.15.90:0/149769965 >> v1:172.21.15.90:6800/420935427 conn(0x7f99540713a0 legacy=0x7f9954073860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:09.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.235+0000 7f996b216700 1 -- 172.21.15.90:0/149769965 >> v1:172.21.15.90:6789/0 conn(0x7f996410e1e0 legacy=0x7f99641a56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:09.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.235+0000 7f996b216700 1 -- 172.21.15.90:0/149769965 shutdown_connections 2024-04-14T15:26:09.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.235+0000 7f996b216700 1 -- 172.21.15.90:0/149769965 >> 172.21.15.90:0/149769965 conn(0x7f99640fd290 msgr2=0x7f9964111780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:26:09.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.235+0000 7f996b216700 1 -- 172.21.15.90:0/149769965 shutdown_connections 2024-04-14T15:26:09.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.235+0000 7f996b216700 1 -- 172.21.15.90:0/149769965 wait complete. 2024-04-14T15:26:09.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:09.235+0000 7f996b216700 1 librados: shutdown 2024-04-14T15:26:09.242 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-14T15:26:09.316 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:26:09.340 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:09.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:09.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:26:09.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:09 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3919189616' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:26:09.376 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:09 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3919189616' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:26:09.376 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:09 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3919189616' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:26:09.442 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:26:09.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:09.468 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:09.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:26:09.569 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:26:09.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:09.594 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:09.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:26:09.695 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:26:09.718 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:09.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:09.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:26:09.748 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-14T15:26:09.821 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:26:09.845 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:09.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:09.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:26:09.947 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:26:09.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:09.971 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:09.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:26:10.072 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:26:10.096 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:10.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:10.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:26:10.197 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:26:10.221 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:10.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:10.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:26:10.323 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:26:10.345 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:10 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:26:10.345 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:10 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:26:10.345 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:10 smithi090 ceph-mon[41670]: pgmap v1213: 224 pgs: 224 active+clean; 4.4 MiB data, 745 MiB used, 625 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-04-14T15:26:10.345 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:10 smithi090 ceph-mon[41670]: from='client.38091 v1:172.21.15.90:0/149769965' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:26:10.345 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:10 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:26:10.345 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:10 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:26:10.345 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:10 smithi090 ceph-mon[30399]: pgmap v1213: 224 pgs: 224 active+clean; 4.4 MiB data, 745 MiB used, 625 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-04-14T15:26:10.345 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:10 smithi090 ceph-mon[30399]: from='client.38091 v1:172.21.15.90:0/149769965' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:26:10.346 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:10.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:10.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:26:10.448 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:26:10.471 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:10.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:10.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:26:10.573 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:26:10.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:10.597 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:10.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:26:10.621 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:10 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:26:10.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:10 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:26:10.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:10 smithi092 ceph-mon[33966]: pgmap v1213: 224 pgs: 224 active+clean; 4.4 MiB data, 745 MiB used, 625 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-04-14T15:26:10.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:10 smithi092 ceph-mon[33966]: from='client.38091 v1:172.21.15.90:0/149769965' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:26:10.698 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:26:10.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:10.723 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:10.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:26:10.824 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:26:10.847 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:10.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:10.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:26:10.934 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":1213,"stamp":"2024-04-14T15:26:09.203037+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4596654,"num_objects":237,"num_object_clones":1,"num_object_copies":711,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":237,"num_whiteouts":0,"num_read":2148,"num_read_kb":1887,"num_write":787,"num_write_kb":4012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":124,"num_bytes_recovered":265798,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1007,"ondisk_log_size":1007,"up":672,"acting":672,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":672,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":762720,"kb_used_data":54388,"kb_used_omap":10,"kb_used_meta":708213,"kb_avail":655453344,"statfs":{"total":671965249536,"available":671184224256,"internally_reserved":0,"allocated":55693312,"data_stored":31764857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11156,"internal_metadata":725210220},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13410,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":-3,"num_write_kb":-54,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":-40253,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001224"},"pg_stats":[{"pgid":"340.1f","version":"0'0","reported_seq":21,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.033450+0000","last_change":"2024-04-14T15:26:02.045092+0000","last_active":"2024-04-14T15:26:06.033450+0000","last_peered":"2024-04-14T15:26:06.033450+0000","last_clean":"2024-04-14T15:26:06.033450+0000","last_became_active":"2024-04-14T15:26:02.044815+0000","last_became_peered":"2024-04-14T15:26:02.044815+0000","last_unstale":"2024-04-14T15:26:06.033450+0000","last_undegraded":"2024-04-14T15:26:06.033450+0000","last_fullsized":"2024-04-14T15:26:06.033450+0000","mapping_epoch":811,"log_start":"0'0","ondisk_log_start":"0'0","created":811,"last_epoch_clean":812,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.020810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:12.316844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"340.1e","version":"0'0","reported_seq":21,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.033433+0000","last_change":"2024-04-14T15:26:02.048272+0000","last_active":"2024-04-14T15:26:06.033433+0000","last_peered":"2024-04-14T15:26:06.033433+0000","last_clean":"2024-04-14T15:26:06.033433+0000","last_became_active":"2024-04-14T15:26:02.047876+0000","last_became_peered":"2024-04-14T15:26:02.047876+0000","last_unstale":"2024-04-14T15:26:06.033433+0000","last_undegraded":"2024-04-14T15:26:06.033433+0000","last_fullsized":"2024-04-14T15:26:06.033433+0000","mapping_epoch":811,"log_start":"0'0","ondisk_log_start":"0'0","created":811,"last_epoch_clean":812,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.020810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:16.223738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"340.1d","version":"0'0","reported_seq":21,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.063697+0000","last_change":"2024-04-14T15:26:02.053657+0000","last_active":"2024-04-14T15:26:06.063697+0000","last_peered":"2024-04-14T15:26:06.063697+0000","last_clean":"2024-04-14T15:26:06.063697+0000","last_became_active":"2024-04-14T15:26:02.050522+0000","last_became_peered":"2024-04-14T15:26:02.050522+0000","last_unstale":"2024-04-14T15:26:06.063697+0000","last_undegraded":"2024-04-14T15:26:06.063697+0000","last_fullsized":"2024-04-14T15:26:06.063697+0000","mapping_epoch":811,"log_start":"0'0","ondisk_log_start":"0'0","created":811,"last_epoch_clean":812,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.020810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:22.453387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"340.1c","version":"0'0","reported_seq":21,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.026627+0000","last_change":"2024-04-14T15:26:02.037847+0000","last_active":"2024-04-14T15:26:06.026627+0000","last_peered":"2024-04-14T15:26:06.026627+0000","last_clean":"2024-04-14T15:26:06.026627+0000","last_became_active":"2024-04-14T15:26:02.037102+0000","last_became_peered":"2024-04-14T15:26:02.037102+0000","last_unstale":"2024-04-14T15:26:06.026627+0000","last_undegraded":"2024-04-14T15:26:06.026627+0000","last_fullsized":"2024-04-14T15:26:06.026627+0000","mapping_epoch":811,"log_start":"0'0","ondisk_log_start":"0'0","created":811,"last_epoch_clean":812,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.020810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:29.054087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"340.1b","version":"0'0","reported_seq":21,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.029070+0000","last_change":"2024-04-14T15:26:02.052764+0000","last_active":"2024-04-14T15:26:06.029070+0000","last_peered":"2024-04-14T15:26:06.029070+0000","last_clean":"2024-04-14T15:26:06.029070+0000","last_became_active":"2024-04-14T15:26:02.051917+0000","last_became_peered":"2024-04-14T15:26:02.051917+0000","last_unstale":"2024-04-14T15:26:06.029070+0000","last_undegraded":"2024-04-14T15:26:06.029070+0000","last_fullsized":"2024-04-14T15:26:06.029070+0000","mapping_epoch":811,"log_start":"0'0","ondisk_log_start":"0'0","created":811,"last_epoch_clean":812,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.020810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:05.898149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"340.1a","version":"0'0","reported_seq":21,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.059812+0000","last_change":"2024-04-14T15:26:02.045201+0000","last_active":"2024-04-14T15:26:06.059812+0000","last_peered":"2024-04-14T15:26:06.059812+0000","last_clean":"2024-04-14T15:26:06.059812+0000","last_became_active":"2024-04-14T15:26:02.044610+0000","last_became_peered":"2024-04-14T15:26:02.044610+0000","last_unstale":"2024-04-14T15:26:06.059812+0000","last_undegraded":"2024-04-14T15:26:06.059812+0000","last_fullsized":"2024-04-14T15:26:06.059812+0000","mapping_epoch":811,"log_start":"0'0","ondisk_log_start":"0'0","created":811,"last_epoch_clean":812,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.020810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:27.294408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"340.19","version":"0'0","reported_seq":21,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.067594+0000","last_change":"2024-04-14T15:26:02.041489+0000","last_active":"2024-04-14T15:26:06.067594+0000","last_peered":"2024-04-14T15:26:06.067594+0000","last_clean":"2024-04-14T15:26:06.067594+0000","last_became_active":"2024-04-14T15:26:02.041183+0000","last_became_peered":"2024-04-14T15:26:02.041183+0000","last_unstale":"2024-04-14T15:26:06.067594+0000","last_undegraded":"2024-04-14T15:26:06.067594+0000","last_fullsized":"2024-04-14T15:26:06.067594+0000","mapping_epoch":811,"log_start":"0'0","ondisk_log_start":"0'0","created":811,"last_epoch_clean":812,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.020810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:25.611268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"340.18","version":"0'0","reported_seq":21,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.059629+0000","last_change":"2024-04-14T15:26:02.045365+0000","last_active":"2024-04-14T15:26:06.059629+0000","last_peered":"2024-04-14T15:26:06.059629+0000","last_clean":"2024-04-14T15:26:06.059629+0000","last_became_active":"2024-04-14T15:26:02.044876+0000","last_became_peered":"2024-04-14T15:26:02.044876+0000","last_unstale":"2024-04-14T15:26:06.059629+0000","last_undegraded":"2024-04-14T15:26:06.059629+0000","last_fullsized":"2024-04-14T15:26:06.059629+0000","mapping_epoch":811,"log_start":"0'0","ondisk_log_start":"0'0","created":811,"last_epoch_clean":812,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.020810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:09.199600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"340.f","version":"0'0","reported_seq":21,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.064234+0000","last_change":"2024-04-14T15:26:02.049556+0000","last_active":"2024-04-14T15:26:06.064234+0000","last_peered":"2024-04-14T15:26:06.064234+0000","last_clean":"2024-04-14T15:26:06.064234+0000","last_became_active":"2024-04-14T15:26:02.049270+0000","last_became_peered":"2024-04-14T15:26:02.049270+0000","last_unstale":"2024-04-14T15:26:06.064234+0000","last_undegraded":"2024-04-14T15:26:06.064234+0000","last_fullsized":"2024-04-14T15:26:06.064234+0000","mapping_epoch":811,"log_start":"0'0","ondisk_log_start":"0'0","created":811,"last_epoch_clean":812,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.020810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:10.874507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"340.e","version":"0'0","reported_seq":21,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.032008+0000","last_change":"2024-04-14T15:26:02.047752+0000","last_active":"2024-04-14T15:26:06.032008+0000","last_peered":"2024-04-14T15:26:06.032008+0000","last_clean":"2024-04-14T15:26:06.032008+0000","last_became_active":"2024-04-14T15:26:02.047466+0000","last_became_peered":"2024-04-14T15:26:02.047466+0000","last_unstale":"2024-04-14T15:26:06.032008+0000","last_undegraded":"2024-04-14T15:26:06.032008+0000","last_fullsized":"2024-04-14T15:26:06.032008+0000","mapping_epoch":811,"log_start":"0'0","ondisk_log_start":"0'0","created":811,"last_epoch_clean":812,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.020810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:15.648532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"340.d","version":"0'0","reported_seq":21,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.068131+0000","last_change":"2024-04-14T15:26:02.037469+0000","last_active":"2024-04-14T15:26:06.068131+0000","last_peered":"2024-04-14T15:26:06.068131+0000","last_clean":"2024-04-14T15:26:06.068131+0000","last_became_active":"2024-04-14T15:26:02.037158+0000","last_became_peered":"2024-04-14T15:26:02.037158+0000","last_unstale":"2024-04-14T15:26:06.068131+0000","last_undegraded":"2024-04-14T15:26:06.068131+0000","last_fullsized":"2024-04-14T15:26:06.068131+0000","mapping_epoch":811,"log_start":"0'0","ondisk_log_start":"0'0","created":811,"last_epoch_clean":812,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.020810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:24.048802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"340.c","version":"0'0","reported_seq":21,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.026764+0000","last_change":"2024-04-14T15:26:02.039526+0000","last_active":"2024-04-14T15:26:06.026764+0000","last_peered":"2024-04-14T15:26:06.026764+0000","last_clean":"2024-04-14T15:26:06.026764+0000","last_became_active":"2024-04-14T15:26:02.039226+0000","last_became_peered":"2024-04-14T15:26:02.039226+0000","last_unstale":"2024-04-14T15:26:06.026764+0000","last_undegraded":"2024-04-14T15:26:06.026764+0000","last_fullsized":"2024-04-14T15:26:06.026764+0000","mapping_epoch":811,"log_start":"0'0","ondisk_log_start":"0'0","created":811,"last_epoch_clean":812,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.020810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:12.904947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"340.b","version":"0'0","reported_seq":21,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.029341+0000","last_change":"2024-04-14T15:26:02.046628+0000","last_active":"2024-04-14T15:26:06.029341+0000","last_peered":"2024-04-14T15:26:06.029341+0000","last_clean":"2024-04-14T15:26:06.029341+0000","last_became_active":"2024-04-14T15:26:02.046231+0000","last_became_peered":"2024-04-14T15:26:02.046231+0000","last_unstale":"2024-04-14T15:26:06.029341+0000","last_undegraded":"2024-04-14T15:26:06.029341+0000","last_fullsized":"2024-04-14T15:26:06.029341+0000","mapping_epoch":811,"log_start":"0'0","ondisk_log_start":"0'0","created":811,"last_epoch_clean":812,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.020810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:02.548341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"340.a","version":"0'0","reported_seq":21,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.059974+0000","last_change":"2024-04-14T15:26:02.043396+0000","last_active":"2024-04-14T15:26:06.059974+0000","last_peered":"2024-04-14T15:26:06.059974+0000","last_clean":"2024-04-14T15:26:06.059974+0000","last_became_active":"2024-04-14T15:26:02.043121+0000","last_became_peered":"2024-04-14T15:26:02.043121+0000","last_unstale":"2024-04-14T15:26:06.059974+0000","last_undegraded":"2024-04-14T15:26:06.059974+0000","last_fullsized":"2024-04-14T15:26:06.059974+0000","mapping_epoch":811,"log_start":"0'0","ondisk_log_start":"0'0","created":811,"last_epoch_clean":812,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.020810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:15.270991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"340.9","version":"0'0","reported_seq":21,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.028415+0000","last_change":"2024-04-14T15:26:02.043604+0000","last_active":"2024-04-14T15:26:06.028415+0000","last_peered":"2024-04-14T15:26:06.028415+0000","last_clean":"2024-04-14T15:26:06.028415+0000","last_became_active":"2024-04-14T15:26:02.043334+0000","last_became_peered":"2024-04-14T15:26:02.043334+0000","last_unstale":"2024-04-14T15:26:06.028415+0000","last_undegraded":"2024-04-14T15:26:06.028415+0000","last_fullsized":"2024-04-14T15:26:06.028415+0000","mapping_epoch":811,"log_start":"0'0","ondisk_log_start":"0'0","created":811,"last_epoch_clean":812,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.020810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:06.253974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"340.8","version":"0'0","reported_seq":21,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.059653+0000","last_change":"2024-04-14T15:26:02.052793+0000","last_active":"2024-04-14T15:26:06.059653+0000","last_peered":"2024-04-14T15:26:06.059653+0000","last_clean":"2024-04-14T15:26:06.059653+0000","last_became_active":"2024-04-14T15:26:02.052393+0000","last_became_peered":"2024-04-14T15:26:02.052393+0000","last_unstale":"2024-04-14T15:26:06.059653+0000","last_undegraded":"2024-04-14T15:26:06.059653+0000","last_fullsized":"2024-04-14T15:26:06.059653+0000","mapping_epoch":811,"log_start":"0'0","ondisk_log_start":"0'0","created":811,"last_epoch_clean":812,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.020810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:01.432455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"340.7","version":"0'0","reported_seq":21,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.061154+0000","last_change":"2024-04-14T15:26:02.045953+0000","last_active":"2024-04-14T15:26:06.061154+0000","last_peered":"2024-04-14T15:26:06.061154+0000","last_clean":"2024-04-14T15:26:06.061154+0000","last_became_active":"2024-04-14T15:26:02.044024+0000","last_became_peered":"2024-04-14T15:26:02.044024+0000","last_unstale":"2024-04-14T15:26:06.061154+0000","last_undegraded":"2024-04-14T15:26:06.061154+0000","last_fullsized":"2024-04-14T15:26:06.061154+0000","mapping_epoch":811,"log_start":"0'0","ondisk_log_start":"0'0","created":811,"last_epoch_clean":812,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.020810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:14.272039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"340.6","version":"0'0","reported_seq":21,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.068514+0000","last_change":"2024-04-14T15:26:02.038886+0000","last_active":"2024-04-14T15:26:06.068514+0000","last_peered":"2024-04-14T15:26:06.068514+0000","last_clean":"2024-04-14T15:26:06.068514+0000","last_became_active":"2024-04-14T15:26:02.038576+0000","last_became_peered":"2024-04-14T15:26:02.038576+0000","last_unstale":"2024-04-14T15:26:06.068514+0000","last_undegraded":"2024-04-14T15:26:06.068514+0000","last_fullsized":"2024-04-14T15:26:06.068514+0000","mapping_epoch":811,"log_start":"0'0","ondisk_log_start":"0'0","created":811,"last_epoch_clean":812,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.020810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:12.075437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"340.5","version":"0'0","reported_seq":21,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.027085+0000","last_change":"2024-04-14T15:26:02.039769+0000","last_active":"2024-04-14T15:26:06.027085+0000","last_peered":"2024-04-14T15:26:06.027085+0000","last_clean":"2024-04-14T15:26:06.027085+0000","last_became_active":"2024-04-14T15:26:02.039234+0000","last_became_peered":"2024-04-14T15:26:02.039234+0000","last_unstale":"2024-04-14T15:26:06.027085+0000","last_undegraded":"2024-04-14T15:26:06.027085+0000","last_fullsized":"2024-04-14T15:26:06.027085+0000","mapping_epoch":811,"log_start":"0'0","ondisk_log_start":"0'0","created":811,"last_epoch_clean":812,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.020810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:22.966082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"340.4","version":"0'0","reported_seq":21,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.063405+0000","last_change":"2024-04-14T15:26:02.042804+0000","last_active":"2024-04-14T15:26:06.063405+0000","last_peered":"2024-04-14T15:26:06.063405+0000","last_clean":"2024-04-14T15:26:06.063405+0000","last_became_active":"2024-04-14T15:26:02.042231+0000","last_became_peered":"2024-04-14T15:26:02.042231+0000","last_unstale":"2024-04-14T15:26:06.063405+0000","last_undegraded":"2024-04-14T15:26:06.063405+0000","last_fullsized":"2024-04-14T15:26:06.063405+0000","mapping_epoch":811,"log_start":"0'0","ondisk_log_start":"0'0","created":811,"last_epoch_clean":812,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.020810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:25.524450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"340.3","version":"0'0","reported_seq":21,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.030301+0000","last_change":"2024-04-14T15:26:02.041623+0000","last_active":"2024-04-14T15:26:06.030301+0000","last_peered":"2024-04-14T15:26:06.030301+0000","last_clean":"2024-04-14T15:26:06.030301+0000","last_became_active":"2024-04-14T15:26:02.041321+0000","last_became_peered":"2024-04-14T15:26:02.041321+0000","last_unstale":"2024-04-14T15:26:06.030301+0000","last_undegraded":"2024-04-14T15:26:06.030301+0000","last_fullsized":"2024-04-14T15:26:06.030301+0000","mapping_epoch":811,"log_start":"0'0","ondisk_log_start":"0'0","created":811,"last_epoch_clean":812,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.020810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:07.258067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"340.2","version":"0'0","reported_seq":21,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.062966+0000","last_change":"2024-04-14T15:26:02.053784+0000","last_active":"2024-04-14T15:26:06.062966+0000","last_peered":"2024-04-14T15:26:06.062966+0000","last_clean":"2024-04-14T15:26:06.062966+0000","last_became_active":"2024-04-14T15:26:02.052375+0000","last_became_peered":"2024-04-14T15:26:02.052375+0000","last_unstale":"2024-04-14T15:26:06.062966+0000","last_undegraded":"2024-04-14T15:26:06.062966+0000","last_fullsized":"2024-04-14T15:26:06.062966+0000","mapping_epoch":811,"log_start":"0'0","ondisk_log_start":"0'0","created":811,"last_epoch_clean":812,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.020810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:01.341551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"340.1","version":"0'0","reported_seq":21,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.029914+0000","last_change":"2024-04-14T15:26:02.048350+0000","last_active":"2024-04-14T15:26:06.029914+0000","last_peered":"2024-04-14T15:26:06.029914+0000","last_clean":"2024-04-14T15:26:06.029914+0000","last_became_active":"2024-04-14T15:26:02.047842+0000","last_became_peered":"2024-04-14T15:26:02.047842+0000","last_unstale":"2024-04-14T15:26:06.029914+0000","last_undegraded":"2024-04-14T15:26:06.029914+0000","last_fullsized":"2024-04-14T15:26:06.029914+0000","mapping_epoch":811,"log_start":"0'0","ondisk_log_start":"0'0","created":811,"last_epoch_clean":812,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.020810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:08.458544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"340.0","version":"0'0","reported_seq":21,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.067514+0000","last_change":"2024-04-14T15:26:02.044869+0000","last_active":"2024-04-14T15:26:06.067514+0000","last_peered":"2024-04-14T15:26:06.067514+0000","last_clean":"2024-04-14T15:26:06.067514+0000","last_became_active":"2024-04-14T15:26:02.044508+0000","last_became_peered":"2024-04-14T15:26:02.044508+0000","last_unstale":"2024-04-14T15:26:06.067514+0000","last_undegraded":"2024-04-14T15:26:06.067514+0000","last_fullsized":"2024-04-14T15:26:06.067514+0000","mapping_epoch":811,"log_start":"0'0","ondisk_log_start":"0'0","created":811,"last_epoch_clean":812,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.020810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:12.107587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"340.10","version":"0'0","reported_seq":21,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.028342+0000","last_change":"2024-04-14T15:26:02.037338+0000","last_active":"2024-04-14T15:26:06.028342+0000","last_peered":"2024-04-14T15:26:06.028342+0000","last_clean":"2024-04-14T15:26:06.028342+0000","last_became_active":"2024-04-14T15:26:02.037050+0000","last_became_peered":"2024-04-14T15:26:02.037050+0000","last_unstale":"2024-04-14T15:26:06.028342+0000","last_undegraded":"2024-04-14T15:26:06.028342+0000","last_fullsized":"2024-04-14T15:26:06.028342+0000","mapping_epoch":811,"log_start":"0'0","ondisk_log_start":"0'0","created":811,"last_epoch_clean":812,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.020810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:16.795066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"323.7","version":"798'3","reported_seq":443,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.064241+0000","last_change":"2024-04-14T15:25:11.799519+0000","last_active":"2024-04-14T15:26:06.064241+0000","last_peered":"2024-04-14T15:26:06.064241+0000","last_clean":"2024-04-14T15:26:06.064241+0000","last_became_active":"2024-04-14T15:23:04.005793+0000","last_became_peered":"2024-04-14T15:23:04.005793+0000","last_unstale":"2024-04-14T15:26:06.064241+0000","last_undegraded":"2024-04-14T15:26:06.064241+0000","last_fullsized":"2024-04-14T15:26:06.064241+0000","mapping_epoch":647,"log_start":"0'0","ondisk_log_start":"0'0","created":647,"last_epoch_clean":648,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:11.799436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_clean_scrub_stamp":"2024-04-14T15:25:11.799436+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:41.514646+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00079989799999999995,"stat_sum":{"num_bytes":24,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"340.11","version":"0'0","reported_seq":21,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.029824+0000","last_change":"2024-04-14T15:26:02.052418+0000","last_active":"2024-04-14T15:26:06.029824+0000","last_peered":"2024-04-14T15:26:06.029824+0000","last_clean":"2024-04-14T15:26:06.029824+0000","last_became_active":"2024-04-14T15:26:02.051667+0000","last_became_peered":"2024-04-14T15:26:02.051667+0000","last_unstale":"2024-04-14T15:26:06.029824+0000","last_undegraded":"2024-04-14T15:26:06.029824+0000","last_fullsized":"2024-04-14T15:26:06.029824+0000","mapping_epoch":811,"log_start":"0'0","ondisk_log_start":"0'0","created":811,"last_epoch_clean":812,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.020810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:12.773798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"323.6","version":"798'4","reported_seq":455,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.032140+0000","last_change":"2024-04-14T15:25:49.959163+0000","last_active":"2024-04-14T15:26:06.032140+0000","last_peered":"2024-04-14T15:26:06.032140+0000","last_clean":"2024-04-14T15:26:06.032140+0000","last_became_active":"2024-04-14T15:23:04.006098+0000","last_became_peered":"2024-04-14T15:23:04.006098+0000","last_unstale":"2024-04-14T15:26:06.032140+0000","last_undegraded":"2024-04-14T15:26:06.032140+0000","last_fullsized":"2024-04-14T15:26:06.032140+0000","mapping_epoch":647,"log_start":"0'0","ondisk_log_start":"0'0","created":647,"last_epoch_clean":648,"parent":"0.0","parent_split_bits":0,"last_scrub":"798'4","last_scrub_stamp":"2024-04-14T15:25:49.959068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_clean_scrub_stamp":"2024-04-14T15:25:49.959068+0000","objects_scrubbed":4,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:00.272091+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0040397810000000001,"stat_sum":{"num_bytes":32,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"340.12","version":"0'0","reported_seq":21,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.029047+0000","last_change":"2024-04-14T15:26:02.047364+0000","last_active":"2024-04-14T15:26:06.029047+0000","last_peered":"2024-04-14T15:26:06.029047+0000","last_clean":"2024-04-14T15:26:06.029047+0000","last_became_active":"2024-04-14T15:26:02.047012+0000","last_became_peered":"2024-04-14T15:26:02.047012+0000","last_unstale":"2024-04-14T15:26:06.029047+0000","last_undegraded":"2024-04-14T15:26:06.029047+0000","last_fullsized":"2024-04-14T15:26:06.029047+0000","mapping_epoch":811,"log_start":"0'0","ondisk_log_start":"0'0","created":811,"last_epoch_clean":812,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.020810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:24.460594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"323.5","version":"812'18","reported_seq":462,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.027155+0000","last_change":"2024-04-14T15:25:46.249717+0000","last_active":"2024-04-14T15:26:06.027155+0000","last_peered":"2024-04-14T15:26:06.027155+0000","last_clean":"2024-04-14T15:26:06.027155+0000","last_became_active":"2024-04-14T15:23:04.002370+0000","last_became_peered":"2024-04-14T15:23:04.002370+0000","last_unstale":"2024-04-14T15:26:06.027155+0000","last_undegraded":"2024-04-14T15:26:06.027155+0000","last_fullsized":"2024-04-14T15:26:06.027155+0000","mapping_epoch":647,"log_start":"0'0","ondisk_log_start":"0'0","created":647,"last_epoch_clean":648,"parent":"0.0","parent_split_bits":0,"last_scrub":"733'11","last_scrub_stamp":"2024-04-14T15:25:46.249607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_clean_scrub_stamp":"2024-04-14T15:25:46.249607+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:53.495234+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.0032579309999999999,"stat_sum":{"num_bytes":61,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":11,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"340.13","version":"0'0","reported_seq":21,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.059970+0000","last_change":"2024-04-14T15:26:02.043758+0000","last_active":"2024-04-14T15:26:06.059970+0000","last_peered":"2024-04-14T15:26:06.059970+0000","last_clean":"2024-04-14T15:26:06.059970+0000","last_became_active":"2024-04-14T15:26:02.042979+0000","last_became_peered":"2024-04-14T15:26:02.042979+0000","last_unstale":"2024-04-14T15:26:06.059970+0000","last_undegraded":"2024-04-14T15:26:06.059970+0000","last_fullsized":"2024-04-14T15:26:06.059970+0000","mapping_epoch":811,"log_start":"0'0","ondisk_log_start":"0'0","created":811,"last_epoch_clean":812,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.020810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:16.808704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"323.4","version":"798'15","reported_seq":462,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.031164+0000","last_change":"2024-04-14T15:25:29.009921+0000","last_active":"2024-04-14T15:26:06.031164+0000","last_peered":"2024-04-14T15:26:06.031164+0000","last_clean":"2024-04-14T15:26:06.031164+0000","last_became_active":"2024-04-14T15:23:03.997912+0000","last_became_peered":"2024-04-14T15:23:03.997912+0000","last_unstale":"2024-04-14T15:26:06.031164+0000","last_undegraded":"2024-04-14T15:26:06.031164+0000","last_fullsized":"2024-04-14T15:26:06.031164+0000","mapping_epoch":647,"log_start":"0'0","ondisk_log_start":"0'0","created":647,"last_epoch_clean":648,"parent":"0.0","parent_split_bits":0,"last_scrub":"742'12","last_scrub_stamp":"2024-04-14T15:25:29.009822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_clean_scrub_stamp":"2024-04-14T15:25:29.009822+0000","objects_scrubbed":2,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:34.000141+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001097995,"stat_sum":{"num_bytes":34,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":9,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"340.14","version":"0'0","reported_seq":21,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.026758+0000","last_change":"2024-04-14T15:26:02.033715+0000","last_active":"2024-04-14T15:26:06.026758+0000","last_peered":"2024-04-14T15:26:06.026758+0000","last_clean":"2024-04-14T15:26:06.026758+0000","last_became_active":"2024-04-14T15:26:02.033157+0000","last_became_peered":"2024-04-14T15:26:02.033157+0000","last_unstale":"2024-04-14T15:26:06.026758+0000","last_undegraded":"2024-04-14T15:26:06.026758+0000","last_fullsized":"2024-04-14T15:26:06.026758+0000","mapping_epoch":811,"log_start":"0'0","ondisk_log_start":"0'0","created":811,"last_epoch_clean":812,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.020810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:10.709028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"323.3","version":"798'19","reported_seq":461,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.063295+0000","last_change":"2024-04-14T15:25:19.807060+0000","last_active":"2024-04-14T15:26:06.063295+0000","last_peered":"2024-04-14T15:26:06.063295+0000","last_clean":"2024-04-14T15:26:06.063295+0000","last_became_active":"2024-04-14T15:23:04.001213+0000","last_became_peered":"2024-04-14T15:23:04.001213+0000","last_unstale":"2024-04-14T15:26:06.063295+0000","last_undegraded":"2024-04-14T15:26:06.063295+0000","last_fullsized":"2024-04-14T15:26:06.063295+0000","mapping_epoch":647,"log_start":"0'0","ondisk_log_start":"0'0","created":647,"last_epoch_clean":648,"parent":"0.0","parent_split_bits":0,"last_scrub":"763'18","last_scrub_stamp":"2024-04-14T15:25:19.806953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_clean_scrub_stamp":"2024-04-14T15:25:19.806953+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:29.471384+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00085465599999999997,"stat_sum":{"num_bytes":37,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":13,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"340.15","version":"0'0","reported_seq":21,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.060593+0000","last_change":"2024-04-14T15:26:02.046119+0000","last_active":"2024-04-14T15:26:06.060593+0000","last_peered":"2024-04-14T15:26:06.060593+0000","last_clean":"2024-04-14T15:26:06.060593+0000","last_became_active":"2024-04-14T15:26:02.045758+0000","last_became_peered":"2024-04-14T15:26:02.045758+0000","last_unstale":"2024-04-14T15:26:06.060593+0000","last_undegraded":"2024-04-14T15:26:06.060593+0000","last_fullsized":"2024-04-14T15:26:06.060593+0000","mapping_epoch":811,"log_start":"0'0","ondisk_log_start":"0'0","created":811,"last_epoch_clean":812,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.020810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:07.263125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"323.2","version":"798'19","reported_seq":467,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.030088+0000","last_change":"2024-04-14T15:25:01.027310+0000","last_active":"2024-04-14T15:26:06.030088+0000","last_peered":"2024-04-14T15:26:06.030088+0000","last_clean":"2024-04-14T15:26:06.030088+0000","last_became_active":"2024-04-14T15:23:04.001557+0000","last_became_peered":"2024-04-14T15:23:04.001557+0000","last_unstale":"2024-04-14T15:26:06.030088+0000","last_undegraded":"2024-04-14T15:26:06.030088+0000","last_fullsized":"2024-04-14T15:26:06.030088+0000","mapping_epoch":647,"log_start":"0'0","ondisk_log_start":"0'0","created":647,"last_epoch_clean":648,"parent":"0.0","parent_split_bits":0,"last_scrub":"710'6","last_scrub_stamp":"2024-04-14T15:24:15.331688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_clean_scrub_stamp":"2024-04-14T15:24:15.331688+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.013115428,"stat_sum":{"num_bytes":20,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"340.16","version":"0'0","reported_seq":21,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.063888+0000","last_change":"2024-04-14T15:26:02.053588+0000","last_active":"2024-04-14T15:26:06.063888+0000","last_peered":"2024-04-14T15:26:06.063888+0000","last_clean":"2024-04-14T15:26:06.063888+0000","last_became_active":"2024-04-14T15:26:02.052709+0000","last_became_peered":"2024-04-14T15:26:02.052709+0000","last_unstale":"2024-04-14T15:26:06.063888+0000","last_undegraded":"2024-04-14T15:26:06.063888+0000","last_fullsized":"2024-04-14T15:26:06.063888+0000","mapping_epoch":811,"log_start":"0'0","ondisk_log_start":"0'0","created":811,"last_epoch_clean":812,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.020810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:25.111516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"323.1","version":"798'5","reported_seq":451,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.067668+0000","last_change":"2024-04-14T15:25:54.933798+0000","last_active":"2024-04-14T15:26:06.067668+0000","last_peered":"2024-04-14T15:26:06.067668+0000","last_clean":"2024-04-14T15:26:06.067668+0000","last_became_active":"2024-04-14T15:23:03.998498+0000","last_became_peered":"2024-04-14T15:23:03.998498+0000","last_unstale":"2024-04-14T15:26:06.067668+0000","last_undegraded":"2024-04-14T15:26:06.067668+0000","last_fullsized":"2024-04-14T15:26:06.067668+0000","mapping_epoch":647,"log_start":"0'0","ondisk_log_start":"0'0","created":647,"last_epoch_clean":648,"parent":"0.0","parent_split_bits":0,"last_scrub":"798'5","last_scrub_stamp":"2024-04-14T15:25:54.933693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_clean_scrub_stamp":"2024-04-14T15:25:54.933693+0000","objects_scrubbed":4,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:56.442818+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001861021,"stat_sum":{"num_bytes":28,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"340.17","version":"0'0","reported_seq":21,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.033583+0000","last_change":"2024-04-14T15:26:02.046864+0000","last_active":"2024-04-14T15:26:06.033583+0000","last_peered":"2024-04-14T15:26:06.033583+0000","last_clean":"2024-04-14T15:26:06.033583+0000","last_became_active":"2024-04-14T15:26:02.046568+0000","last_became_peered":"2024-04-14T15:26:02.046568+0000","last_unstale":"2024-04-14T15:26:06.033583+0000","last_undegraded":"2024-04-14T15:26:06.033583+0000","last_fullsized":"2024-04-14T15:26:06.033583+0000","mapping_epoch":811,"log_start":"0'0","ondisk_log_start":"0'0","created":811,"last_epoch_clean":812,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:01.020810+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.020810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:04.763937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"323.0","version":"798'8","reported_seq":455,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.028457+0000","last_change":"2024-04-14T15:25:04.467370+0000","last_active":"2024-04-14T15:26:06.028457+0000","last_peered":"2024-04-14T15:26:06.028457+0000","last_clean":"2024-04-14T15:26:06.028457+0000","last_became_active":"2024-04-14T15:23:04.001020+0000","last_became_peered":"2024-04-14T15:23:04.001020+0000","last_unstale":"2024-04-14T15:26:06.028457+0000","last_undegraded":"2024-04-14T15:26:06.028457+0000","last_fullsized":"2024-04-14T15:26:06.028457+0000","mapping_epoch":647,"log_start":"0'0","ondisk_log_start":"0'0","created":647,"last_epoch_clean":648,"parent":"0.0","parent_split_bits":0,"last_scrub":"704'6","last_scrub_stamp":"2024-04-14T15:25:04.467266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_clean_scrub_stamp":"2024-04-14T15:25:04.467266+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:30.942972+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0014140909999999999,"stat_sum":{"num_bytes":4000021,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":3910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"91'130","reported_seq":466,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.067442+0000","last_change":"2024-04-14T15:25:02.924588+0000","last_active":"2024-04-14T15:26:06.067442+0000","last_peered":"2024-04-14T15:26:06.067442+0000","last_clean":"2024-04-14T15:26:06.067442+0000","last_became_active":"2024-04-14T15:25:02.025875+0000","last_became_peered":"2024-04-14T15:25:02.025875+0000","last_unstale":"2024-04-14T15:26:06.067442+0000","last_undegraded":"2024-04-14T15:26:06.067442+0000","last_fullsized":"2024-04-14T15:26:06.067442+0000","mapping_epoch":757,"log_start":"91'130","ondisk_log_start":"91'130","created":752,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:25:02.924460+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:25:02.924460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:10.071683+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":1118,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.032874+0000","last_change":"2024-04-14T15:26:04.159299+0000","last_active":"2024-04-14T15:26:06.032874+0000","last_peered":"2024-04-14T15:26:06.032874+0000","last_clean":"2024-04-14T15:26:06.032874+0000","last_became_active":"2024-04-14T15:15:44.882941+0000","last_became_peered":"2024-04-14T15:15:44.882941+0000","last_unstale":"2024-04-14T15:26:06.032874+0000","last_undegraded":"2024-04-14T15:26:06.032874+0000","last_fullsized":"2024-04-14T15:26:06.032874+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:04.159220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:26:04.159220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:27.728793+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.9","version":"528'10","reported_seq":1773,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.068382+0000","last_change":"2024-04-14T15:24:37.737273+0000","last_active":"2024-04-14T15:26:06.068382+0000","last_peered":"2024-04-14T15:26:06.068382+0000","last_clean":"2024-04-14T15:26:06.068382+0000","last_became_active":"2024-04-14T15:20:52.984943+0000","last_became_peered":"2024-04-14T15:20:52.984943+0000","last_unstale":"2024-04-14T15:26:06.068382+0000","last_undegraded":"2024-04-14T15:26:06.068382+0000","last_fullsized":"2024-04-14T15:26:06.068382+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"528'10","last_scrub_stamp":"2024-04-14T15:24:37.737147+0000","last_deep_scrub":"85'9","last_deep_scrub_stamp":"2024-04-14T15:10:51.950580+0000","last_clean_scrub_stamp":"2024-04-14T15:24:37.737147+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"479'34","reported_seq":1205,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.067720+0000","last_change":"2024-04-14T15:25:26.852970+0000","last_active":"2024-04-14T15:26:06.067720+0000","last_peered":"2024-04-14T15:26:06.067720+0000","last_clean":"2024-04-14T15:26:06.067720+0000","last_became_active":"2024-04-14T15:20:52.979204+0000","last_became_peered":"2024-04-14T15:20:52.979204+0000","last_unstale":"2024-04-14T15:26:06.067720+0000","last_undegraded":"2024-04-14T15:26:06.067720+0000","last_fullsized":"2024-04-14T15:26:06.067720+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'34","last_scrub_stamp":"2024-04-14T15:25:26.852851+0000","last_deep_scrub":"430'22","last_deep_scrub_stamp":"2024-04-14T15:18:39.704883+0000","last_clean_scrub_stamp":"2024-04-14T15:25:26.852851+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:34.262841+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":1569,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.061171+0000","last_change":"2024-04-14T15:25:51.674598+0000","last_active":"2024-04-14T15:26:06.061171+0000","last_peered":"2024-04-14T15:26:06.061171+0000","last_clean":"2024-04-14T15:26:06.061171+0000","last_became_active":"2024-04-14T15:09:43.021460+0000","last_became_peered":"2024-04-14T15:09:43.021460+0000","last_unstale":"2024-04-14T15:26:06.061171+0000","last_undegraded":"2024-04-14T15:26:06.061171+0000","last_fullsized":"2024-04-14T15:26:06.061171+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:51.674520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:25:51.674520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:08.097729+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":606,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.029706+0000","last_change":"2024-04-14T15:25:49.122361+0000","last_active":"2024-04-14T15:26:06.029706+0000","last_peered":"2024-04-14T15:26:06.029706+0000","last_clean":"2024-04-14T15:26:06.029706+0000","last_became_active":"2024-04-14T15:20:52.989666+0000","last_became_peered":"2024-04-14T15:20:52.989666+0000","last_unstale":"2024-04-14T15:26:06.029706+0000","last_undegraded":"2024-04-14T15:26:06.029706+0000","last_fullsized":"2024-04-14T15:26:06.029706+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:49.122278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:15:40.368219+0000","last_clean_scrub_stamp":"2024-04-14T15:25:49.122278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:18.115547+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"91'130","reported_seq":469,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.027333+0000","last_change":"2024-04-14T15:25:02.372541+0000","last_active":"2024-04-14T15:26:06.027333+0000","last_peered":"2024-04-14T15:26:06.027333+0000","last_clean":"2024-04-14T15:26:06.027333+0000","last_became_active":"2024-04-14T15:25:02.026456+0000","last_became_peered":"2024-04-14T15:25:02.026456+0000","last_unstale":"2024-04-14T15:26:06.027333+0000","last_undegraded":"2024-04-14T15:26:06.027333+0000","last_fullsized":"2024-04-14T15:26:06.027333+0000","mapping_epoch":757,"log_start":"91'130","ondisk_log_start":"91'130","created":752,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:25:02.372440+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:25:02.372440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:17.006246+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":1467,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.068360+0000","last_change":"2024-04-14T15:25:20.823771+0000","last_active":"2024-04-14T15:26:06.068360+0000","last_peered":"2024-04-14T15:26:06.068360+0000","last_clean":"2024-04-14T15:26:06.068360+0000","last_became_active":"2024-04-14T15:20:52.991952+0000","last_became_peered":"2024-04-14T15:20:52.991952+0000","last_unstale":"2024-04-14T15:26:06.068360+0000","last_undegraded":"2024-04-14T15:26:06.068360+0000","last_fullsized":"2024-04-14T15:26:06.068360+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:20.823680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:25:20.823680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:22.548626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"479'15","reported_seq":1592,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.029288+0000","last_change":"2024-04-14T15:25:50.132021+0000","last_active":"2024-04-14T15:26:06.029288+0000","last_peered":"2024-04-14T15:26:06.029288+0000","last_clean":"2024-04-14T15:26:06.029288+0000","last_became_active":"2024-04-14T15:09:45.037179+0000","last_became_peered":"2024-04-14T15:09:45.037179+0000","last_unstale":"2024-04-14T15:26:06.029288+0000","last_undegraded":"2024-04-14T15:26:06.029288+0000","last_fullsized":"2024-04-14T15:26:06.029288+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'15","last_scrub_stamp":"2024-04-14T15:25:50.131933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:25:50.131933+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:51.346947+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":1568,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.032346+0000","last_change":"2024-04-14T15:25:47.000217+0000","last_active":"2024-04-14T15:26:06.032346+0000","last_peered":"2024-04-14T15:26:06.032346+0000","last_clean":"2024-04-14T15:26:06.032346+0000","last_became_active":"2024-04-14T15:09:43.020044+0000","last_became_peered":"2024-04-14T15:09:43.020044+0000","last_unstale":"2024-04-14T15:26:06.032346+0000","last_undegraded":"2024-04-14T15:26:06.032346+0000","last_fullsized":"2024-04-14T15:26:06.032346+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:47.000098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:25:47.000098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:52.332018+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1561,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.060484+0000","last_change":"2024-04-14T15:24:27.475600+0000","last_active":"2024-04-14T15:26:06.060484+0000","last_peered":"2024-04-14T15:26:06.060484+0000","last_clean":"2024-04-14T15:26:06.060484+0000","last_became_active":"2024-04-14T15:09:47.046153+0000","last_became_peered":"2024-04-14T15:09:47.046153+0000","last_unstale":"2024-04-14T15:26:06.060484+0000","last_undegraded":"2024-04-14T15:26:06.060484+0000","last_fullsized":"2024-04-14T15:26:06.060484+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:27.475490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:24:27.475490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:10.771965+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1559,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.030868+0000","last_change":"2024-04-14T15:26:05.045908+0000","last_active":"2024-04-14T15:26:06.030868+0000","last_peered":"2024-04-14T15:26:06.030868+0000","last_clean":"2024-04-14T15:26:06.030868+0000","last_became_active":"2024-04-14T15:09:49.061495+0000","last_became_peered":"2024-04-14T15:09:49.061495+0000","last_unstale":"2024-04-14T15:26:06.030868+0000","last_undegraded":"2024-04-14T15:26:06.030868+0000","last_fullsized":"2024-04-14T15:26:06.030868+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:05.045782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:23:37.201795+0000","last_clean_scrub_stamp":"2024-04-14T15:26:05.045782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:30.855824+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"91'130","reported_seq":1003,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.027460+0000","last_change":"2024-04-14T15:25:03.337059+0000","last_active":"2024-04-14T15:26:06.027460+0000","last_peered":"2024-04-14T15:26:06.027460+0000","last_clean":"2024-04-14T15:26:06.027460+0000","last_became_active":"2024-04-14T15:25:02.029756+0000","last_became_peered":"2024-04-14T15:25:02.029756+0000","last_unstale":"2024-04-14T15:26:06.027460+0000","last_undegraded":"2024-04-14T15:26:06.027460+0000","last_fullsized":"2024-04-14T15:26:06.027460+0000","mapping_epoch":757,"log_start":"91'130","ondisk_log_start":"91'130","created":752,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:25:03.336977+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:25:03.336977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:25.945613+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":1467,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.068737+0000","last_change":"2024-04-14T15:25:27.868432+0000","last_active":"2024-04-14T15:26:06.068737+0000","last_peered":"2024-04-14T15:26:06.068737+0000","last_clean":"2024-04-14T15:26:06.068737+0000","last_became_active":"2024-04-14T15:20:52.999845+0000","last_became_peered":"2024-04-14T15:20:52.999845+0000","last_unstale":"2024-04-14T15:26:06.068737+0000","last_undegraded":"2024-04-14T15:26:06.068737+0000","last_fullsized":"2024-04-14T15:26:06.068737+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:27.868337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:25:27.868337+0000","last_clean_scrub_stamp":"2024-04-14T15:25:27.868337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:34.757816+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"479'44","reported_seq":456,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.060069+0000","last_change":"2024-04-14T15:25:26.767683+0000","last_active":"2024-04-14T15:26:06.060069+0000","last_peered":"2024-04-14T15:26:06.060069+0000","last_clean":"2024-04-14T15:26:06.060069+0000","last_became_active":"2024-04-14T15:22:25.970899+0000","last_became_peered":"2024-04-14T15:22:25.970899+0000","last_unstale":"2024-04-14T15:26:06.060069+0000","last_undegraded":"2024-04-14T15:26:06.060069+0000","last_fullsized":"2024-04-14T15:26:06.060069+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'44","last_scrub_stamp":"2024-04-14T15:25:26.767559+0000","last_deep_scrub":"479'44","last_deep_scrub_stamp":"2024-04-14T15:25:26.767559+0000","last_clean_scrub_stamp":"2024-04-14T15:25:26.767559+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:36.496759+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":1577,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.060704+0000","last_change":"2024-04-14T15:25:53.717840+0000","last_active":"2024-04-14T15:26:06.060704+0000","last_peered":"2024-04-14T15:26:06.060704+0000","last_clean":"2024-04-14T15:26:06.060704+0000","last_became_active":"2024-04-14T15:09:43.020268+0000","last_became_peered":"2024-04-14T15:09:43.020268+0000","last_unstale":"2024-04-14T15:26:06.060704+0000","last_undegraded":"2024-04-14T15:26:06.060704+0000","last_fullsized":"2024-04-14T15:26:06.060704+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:53.717687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:34.342793+0000","last_clean_scrub_stamp":"2024-04-14T15:25:53.717687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:01.063625+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1561,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.060152+0000","last_change":"2024-04-14T15:25:56.701857+0000","last_active":"2024-04-14T15:26:06.060152+0000","last_peered":"2024-04-14T15:26:06.060152+0000","last_clean":"2024-04-14T15:26:06.060152+0000","last_became_active":"2024-04-14T15:09:47.045542+0000","last_became_peered":"2024-04-14T15:09:47.045542+0000","last_unstale":"2024-04-14T15:26:06.060152+0000","last_undegraded":"2024-04-14T15:26:06.060152+0000","last_fullsized":"2024-04-14T15:26:06.060152+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:56.701776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:25:56.701776+0000","last_clean_scrub_stamp":"2024-04-14T15:25:56.701776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:18.809662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1552,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.068397+0000","last_change":"2024-04-14T15:26:01.950975+0000","last_active":"2024-04-14T15:26:06.068397+0000","last_peered":"2024-04-14T15:26:06.068397+0000","last_clean":"2024-04-14T15:26:06.068397+0000","last_became_active":"2024-04-14T15:09:49.065397+0000","last_became_peered":"2024-04-14T15:09:49.065397+0000","last_unstale":"2024-04-14T15:26:06.068397+0000","last_undegraded":"2024-04-14T15:26:06.068397+0000","last_fullsized":"2024-04-14T15:26:06.068397+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.950897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.950897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:17.096261+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"91'130","reported_seq":1515,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.059770+0000","last_change":"2024-04-14T15:26:04.744080+0000","last_active":"2024-04-14T15:26:06.059770+0000","last_peered":"2024-04-14T15:26:06.059770+0000","last_clean":"2024-04-14T15:26:06.059770+0000","last_became_active":"2024-04-14T15:21:12.961419+0000","last_became_peered":"2024-04-14T15:21:12.961419+0000","last_unstale":"2024-04-14T15:26:06.059770+0000","last_undegraded":"2024-04-14T15:26:06.059770+0000","last_fullsized":"2024-04-14T15:26:06.059770+0000","mapping_epoch":276,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:26:04.743991+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:26:04.743991+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":70,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:13.195386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":1466,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.060457+0000","last_change":"2024-04-14T15:25:11.638473+0000","last_active":"2024-04-14T15:26:06.060457+0000","last_peered":"2024-04-14T15:26:06.060457+0000","last_clean":"2024-04-14T15:26:06.060457+0000","last_became_active":"2024-04-14T15:12:56.474322+0000","last_became_peered":"2024-04-14T15:12:56.474322+0000","last_unstale":"2024-04-14T15:26:06.060457+0000","last_undegraded":"2024-04-14T15:26:06.060457+0000","last_fullsized":"2024-04-14T15:26:06.060457+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:11.638381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:25:11.638381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:26.353663+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"780'39","reported_seq":1665,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.027540+0000","last_change":"2024-04-14T15:25:24.296864+0000","last_active":"2024-04-14T15:26:06.027540+0000","last_peered":"2024-04-14T15:26:06.027540+0000","last_clean":"2024-04-14T15:26:06.027540+0000","last_became_active":"2024-04-14T15:13:12.933907+0000","last_became_peered":"2024-04-14T15:13:12.933907+0000","last_unstale":"2024-04-14T15:26:06.027540+0000","last_undegraded":"2024-04-14T15:26:06.027540+0000","last_fullsized":"2024-04-14T15:26:06.027540+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"780'39","last_scrub_stamp":"2024-04-14T15:25:24.296779+0000","last_deep_scrub":"780'39","last_deep_scrub_stamp":"2024-04-14T15:25:24.296779+0000","last_clean_scrub_stamp":"2024-04-14T15:25:24.296779+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:37.386005+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":1556,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.067955+0000","last_change":"2024-04-14T15:23:26.611299+0000","last_active":"2024-04-14T15:26:06.067955+0000","last_peered":"2024-04-14T15:26:06.067955+0000","last_clean":"2024-04-14T15:26:06.067955+0000","last_became_active":"2024-04-14T15:09:43.022351+0000","last_became_peered":"2024-04-14T15:09:43.022351+0000","last_unstale":"2024-04-14T15:26:06.067955+0000","last_undegraded":"2024-04-14T15:26:06.067955+0000","last_fullsized":"2024-04-14T15:26:06.067955+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:26.611217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:23:26.611217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1560,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.031985+0000","last_change":"2024-04-14T15:25:50.984808+0000","last_active":"2024-04-14T15:26:06.031985+0000","last_peered":"2024-04-14T15:26:06.031985+0000","last_clean":"2024-04-14T15:26:06.031985+0000","last_became_active":"2024-04-14T15:09:47.042810+0000","last_became_peered":"2024-04-14T15:09:47.042810+0000","last_unstale":"2024-04-14T15:26:06.031985+0000","last_undegraded":"2024-04-14T15:26:06.031985+0000","last_fullsized":"2024-04-14T15:26:06.031985+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:50.984722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:25:50.984722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:14.657168+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1557,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.061522+0000","last_change":"2024-04-14T15:25:27.741050+0000","last_active":"2024-04-14T15:26:06.061522+0000","last_peered":"2024-04-14T15:26:06.061522+0000","last_clean":"2024-04-14T15:26:06.061522+0000","last_became_active":"2024-04-14T15:09:49.069594+0000","last_became_peered":"2024-04-14T15:09:49.069594+0000","last_unstale":"2024-04-14T15:26:06.061522+0000","last_undegraded":"2024-04-14T15:26:06.061522+0000","last_fullsized":"2024-04-14T15:26:06.061522+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:27.740959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:25:27.740959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:50.237809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"91'130","reported_seq":1912,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.067748+0000","last_change":"2024-04-14T15:25:51.968976+0000","last_active":"2024-04-14T15:26:06.067748+0000","last_peered":"2024-04-14T15:26:06.067748+0000","last_clean":"2024-04-14T15:26:06.067748+0000","last_became_active":"2024-04-14T15:21:09.020300+0000","last_became_peered":"2024-04-14T15:21:09.020300+0000","last_unstale":"2024-04-14T15:26:06.067748+0000","last_undegraded":"2024-04-14T15:26:06.067748+0000","last_fullsized":"2024-04-14T15:26:06.067748+0000","mapping_epoch":189,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":544,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:25:51.968864+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:25:51.968864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:57.531574+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":596,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.067867+0000","last_change":"2024-04-14T15:23:41.487909+0000","last_active":"2024-04-14T15:26:06.067867+0000","last_peered":"2024-04-14T15:26:06.067867+0000","last_clean":"2024-04-14T15:26:06.067867+0000","last_became_active":"2024-04-14T15:20:52.980375+0000","last_became_peered":"2024-04-14T15:20:52.980375+0000","last_unstale":"2024-04-14T15:26:06.067867+0000","last_undegraded":"2024-04-14T15:26:06.067867+0000","last_fullsized":"2024-04-14T15:26:06.067867+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:41.487814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:23:41.487814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"479'25","reported_seq":1605,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.027100+0000","last_change":"2024-04-14T15:24:34.394922+0000","last_active":"2024-04-14T15:26:06.027100+0000","last_peered":"2024-04-14T15:26:06.027100+0000","last_clean":"2024-04-14T15:26:06.027100+0000","last_became_active":"2024-04-14T15:20:52.987699+0000","last_became_peered":"2024-04-14T15:20:52.987699+0000","last_unstale":"2024-04-14T15:26:06.027100+0000","last_undegraded":"2024-04-14T15:26:06.027100+0000","last_fullsized":"2024-04-14T15:26:06.027100+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'25","last_scrub_stamp":"2024-04-14T15:24:34.394825+0000","last_deep_scrub":"479'25","last_deep_scrub_stamp":"2024-04-14T15:20:03.571160+0000","last_clean_scrub_stamp":"2024-04-14T15:24:34.394825+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":1568,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.030545+0000","last_change":"2024-04-14T15:25:11.596445+0000","last_active":"2024-04-14T15:26:06.030545+0000","last_peered":"2024-04-14T15:26:06.030545+0000","last_clean":"2024-04-14T15:26:06.030545+0000","last_became_active":"2024-04-14T15:09:43.018871+0000","last_became_peered":"2024-04-14T15:09:43.018871+0000","last_unstale":"2024-04-14T15:26:06.030545+0000","last_undegraded":"2024-04-14T15:26:06.030545+0000","last_fullsized":"2024-04-14T15:26:06.030545+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:11.596346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:25:11.596346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:26.607882+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"400'9","reported_seq":1768,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.061004+0000","last_change":"2024-04-14T15:25:49.676706+0000","last_active":"2024-04-14T15:26:06.061004+0000","last_peered":"2024-04-14T15:26:06.061004+0000","last_clean":"2024-04-14T15:26:06.061004+0000","last_became_active":"2024-04-14T15:09:47.044085+0000","last_became_peered":"2024-04-14T15:09:47.044085+0000","last_unstale":"2024-04-14T15:26:06.061004+0000","last_undegraded":"2024-04-14T15:26:06.061004+0000","last_fullsized":"2024-04-14T15:26:06.061004+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'9","last_scrub_stamp":"2024-04-14T15:25:49.676621+0000","last_deep_scrub":"400'9","last_deep_scrub_stamp":"2024-04-14T15:17:54.845881+0000","last_clean_scrub_stamp":"2024-04-14T15:25:49.676621+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:06.813931+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1544,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.027631+0000","last_change":"2024-04-14T15:24:19.552836+0000","last_active":"2024-04-14T15:26:06.027631+0000","last_peered":"2024-04-14T15:26:06.027631+0000","last_clean":"2024-04-14T15:26:06.027631+0000","last_became_active":"2024-04-14T15:09:49.065274+0000","last_became_peered":"2024-04-14T15:09:49.065274+0000","last_unstale":"2024-04-14T15:26:06.027631+0000","last_undegraded":"2024-04-14T15:26:06.027631+0000","last_fullsized":"2024-04-14T15:26:06.027631+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:19.552703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:18.588546+0000","last_clean_scrub_stamp":"2024-04-14T15:24:19.552703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"91'130","reported_seq":1522,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.060382+0000","last_change":"2024-04-14T15:25:06.600246+0000","last_active":"2024-04-14T15:26:06.060382+0000","last_peered":"2024-04-14T15:26:06.060382+0000","last_clean":"2024-04-14T15:26:06.060382+0000","last_became_active":"2024-04-14T15:21:04.971515+0000","last_became_peered":"2024-04-14T15:21:04.971515+0000","last_unstale":"2024-04-14T15:26:06.060382+0000","last_undegraded":"2024-04-14T15:26:06.060382+0000","last_fullsized":"2024-04-14T15:26:06.060382+0000","mapping_epoch":530,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":540,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:25:06.600156+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:25:06.600156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:09.247668+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":1465,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.028818+0000","last_change":"2024-04-14T15:25:54.058911+0000","last_active":"2024-04-14T15:26:06.028818+0000","last_peered":"2024-04-14T15:26:06.028818+0000","last_clean":"2024-04-14T15:26:06.028818+0000","last_became_active":"2024-04-14T15:12:56.487615+0000","last_became_peered":"2024-04-14T15:12:56.487615+0000","last_unstale":"2024-04-14T15:26:06.028818+0000","last_undegraded":"2024-04-14T15:26:06.028818+0000","last_fullsized":"2024-04-14T15:26:06.028818+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:54.058770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:25:54.058770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:22.907144+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"479'34","reported_seq":1614,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.032951+0000","last_change":"2024-04-14T15:24:35.378498+0000","last_active":"2024-04-14T15:26:06.032951+0000","last_peered":"2024-04-14T15:26:06.032951+0000","last_clean":"2024-04-14T15:26:06.032951+0000","last_became_active":"2024-04-14T15:09:45.033331+0000","last_became_peered":"2024-04-14T15:09:45.033331+0000","last_unstale":"2024-04-14T15:26:06.032951+0000","last_undegraded":"2024-04-14T15:26:06.032951+0000","last_fullsized":"2024-04-14T15:26:06.032951+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'34","last_scrub_stamp":"2024-04-14T15:24:35.378379+0000","last_deep_scrub":"479'34","last_deep_scrub_stamp":"2024-04-14T15:24:35.378379+0000","last_clean_scrub_stamp":"2024-04-14T15:24:35.378379+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":1568,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.029649+0000","last_change":"2024-04-14T15:25:47.989868+0000","last_active":"2024-04-14T15:26:06.029649+0000","last_peered":"2024-04-14T15:26:06.029649+0000","last_clean":"2024-04-14T15:26:06.029649+0000","last_became_active":"2024-04-14T15:09:43.020878+0000","last_became_peered":"2024-04-14T15:09:43.020878+0000","last_unstale":"2024-04-14T15:26:06.029649+0000","last_undegraded":"2024-04-14T15:26:06.029649+0000","last_fullsized":"2024-04-14T15:26:06.029649+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:47.989790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:25:47.989790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:53.293672+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"72'8","reported_seq":1754,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.027653+0000","last_change":"2024-04-14T15:24:27.403836+0000","last_active":"2024-04-14T15:26:06.027653+0000","last_peered":"2024-04-14T15:26:06.027653+0000","last_clean":"2024-04-14T15:26:06.027653+0000","last_became_active":"2024-04-14T15:09:47.050962+0000","last_became_peered":"2024-04-14T15:09:47.050962+0000","last_unstale":"2024-04-14T15:26:06.027653+0000","last_undegraded":"2024-04-14T15:26:06.027653+0000","last_fullsized":"2024-04-14T15:26:06.027653+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:24:27.403735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:24:27.403735+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1119,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.031824+0000","last_change":"2024-04-14T15:24:27.318413+0000","last_active":"2024-04-14T15:26:06.031824+0000","last_peered":"2024-04-14T15:26:06.031824+0000","last_clean":"2024-04-14T15:26:06.031824+0000","last_became_active":"2024-04-14T15:15:44.879699+0000","last_became_peered":"2024-04-14T15:15:44.879699+0000","last_unstale":"2024-04-14T15:26:06.031824+0000","last_undegraded":"2024-04-14T15:26:06.031824+0000","last_fullsized":"2024-04-14T15:26:06.031824+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:27.318330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:24:27.318330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"91'130","reported_seq":1733,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.030493+0000","last_change":"2024-04-14T15:25:02.472703+0000","last_active":"2024-04-14T15:26:06.030493+0000","last_peered":"2024-04-14T15:26:06.030493+0000","last_clean":"2024-04-14T15:26:06.030493+0000","last_became_active":"2024-04-14T15:25:02.018082+0000","last_became_peered":"2024-04-14T15:25:02.018082+0000","last_unstale":"2024-04-14T15:26:06.030493+0000","last_undegraded":"2024-04-14T15:26:06.030493+0000","last_fullsized":"2024-04-14T15:26:06.030493+0000","mapping_epoch":757,"log_start":"91'130","ondisk_log_start":"91'130","created":752,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:25:02.472613+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:25:02.472613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:09.501028+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"233'8","reported_seq":874,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.062815+0000","last_change":"2024-04-14T15:23:43.908078+0000","last_active":"2024-04-14T15:26:06.062815+0000","last_peered":"2024-04-14T15:26:06.062815+0000","last_clean":"2024-04-14T15:26:06.062815+0000","last_became_active":"2024-04-14T15:20:52.995129+0000","last_became_peered":"2024-04-14T15:20:52.995129+0000","last_unstale":"2024-04-14T15:26:06.062815+0000","last_undegraded":"2024-04-14T15:26:06.062815+0000","last_fullsized":"2024-04-14T15:26:06.062815+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'8","last_scrub_stamp":"2024-04-14T15:23:43.907946+0000","last_deep_scrub":"233'8","last_deep_scrub_stamp":"2024-04-14T15:15:01.295800+0000","last_clean_scrub_stamp":"2024-04-14T15:23:43.907946+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":5,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"479'24","reported_seq":1614,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.027447+0000","last_change":"2024-04-14T15:25:20.326455+0000","last_active":"2024-04-14T15:26:06.027447+0000","last_peered":"2024-04-14T15:26:06.027447+0000","last_clean":"2024-04-14T15:26:06.027447+0000","last_became_active":"2024-04-14T15:20:52.998853+0000","last_became_peered":"2024-04-14T15:20:52.998853+0000","last_unstale":"2024-04-14T15:26:06.027447+0000","last_undegraded":"2024-04-14T15:26:06.027447+0000","last_fullsized":"2024-04-14T15:26:06.027447+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'24","last_scrub_stamp":"2024-04-14T15:25:20.326301+0000","last_deep_scrub":"85'16","last_deep_scrub_stamp":"2024-04-14T15:10:47.456169+0000","last_clean_scrub_stamp":"2024-04-14T15:25:20.326301+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:26.745364+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":1564,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.067830+0000","last_change":"2024-04-14T15:23:36.489554+0000","last_active":"2024-04-14T15:26:06.067830+0000","last_peered":"2024-04-14T15:26:06.067830+0000","last_clean":"2024-04-14T15:26:06.067830+0000","last_became_active":"2024-04-14T15:09:43.018601+0000","last_became_peered":"2024-04-14T15:09:43.018601+0000","last_unstale":"2024-04-14T15:26:06.067830+0000","last_undegraded":"2024-04-14T15:26:06.067830+0000","last_fullsized":"2024-04-14T15:26:06.067830+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:36.489475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:23:36.489475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1120,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.027775+0000","last_change":"2024-04-14T15:24:35.426298+0000","last_active":"2024-04-14T15:26:06.027775+0000","last_peered":"2024-04-14T15:26:06.027775+0000","last_clean":"2024-04-14T15:26:06.027775+0000","last_became_active":"2024-04-14T15:20:52.970831+0000","last_became_peered":"2024-04-14T15:20:52.970831+0000","last_unstale":"2024-04-14T15:26:06.027775+0000","last_undegraded":"2024-04-14T15:26:06.027775+0000","last_fullsized":"2024-04-14T15:26:06.027775+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:35.426204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:24:35.426204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1229,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.063642+0000","last_change":"2024-04-14T15:24:32.995639+0000","last_active":"2024-04-14T15:26:06.063642+0000","last_peered":"2024-04-14T15:26:06.063642+0000","last_clean":"2024-04-14T15:26:06.063642+0000","last_became_active":"2024-04-14T15:20:52.999771+0000","last_became_peered":"2024-04-14T15:20:52.999771+0000","last_unstale":"2024-04-14T15:26:06.063642+0000","last_undegraded":"2024-04-14T15:26:06.063642+0000","last_fullsized":"2024-04-14T15:26:06.063642+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:32.995512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:57.214046+0000","last_clean_scrub_stamp":"2024-04-14T15:24:32.995512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"91'130","reported_seq":1711,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.063572+0000","last_change":"2024-04-14T15:25:04.733894+0000","last_active":"2024-04-14T15:26:06.063572+0000","last_peered":"2024-04-14T15:26:06.063572+0000","last_clean":"2024-04-14T15:26:06.063572+0000","last_became_active":"2024-04-14T15:24:57.023208+0000","last_became_peered":"2024-04-14T15:24:57.023208+0000","last_unstale":"2024-04-14T15:26:06.063572+0000","last_undegraded":"2024-04-14T15:26:06.063572+0000","last_fullsized":"2024-04-14T15:26:06.063572+0000","mapping_epoch":189,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":753,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:25:04.733792+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:25:04.733792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":60,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:15.643111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":1460,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.027400+0000","last_change":"2024-04-14T15:24:37.496873+0000","last_active":"2024-04-14T15:26:06.027400+0000","last_peered":"2024-04-14T15:26:06.027400+0000","last_clean":"2024-04-14T15:26:06.027400+0000","last_became_active":"2024-04-14T15:12:56.427276+0000","last_became_peered":"2024-04-14T15:12:56.427276+0000","last_unstale":"2024-04-14T15:26:06.027400+0000","last_undegraded":"2024-04-14T15:26:06.027400+0000","last_fullsized":"2024-04-14T15:26:06.027400+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:37.496778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:24:37.496778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"255'10","reported_seq":1403,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.062859+0000","last_change":"2024-04-14T15:25:48.705124+0000","last_active":"2024-04-14T15:26:06.062859+0000","last_peered":"2024-04-14T15:26:06.062859+0000","last_clean":"2024-04-14T15:26:06.062859+0000","last_became_active":"2024-04-14T15:15:21.873747+0000","last_became_peered":"2024-04-14T15:15:21.873747+0000","last_unstale":"2024-04-14T15:26:06.062859+0000","last_undegraded":"2024-04-14T15:26:06.062859+0000","last_fullsized":"2024-04-14T15:26:06.062859+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'10","last_scrub_stamp":"2024-04-14T15:25:48.705041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:25:48.705041+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:50.660926+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"479'28","reported_seq":1624,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.026235+0000","last_change":"2024-04-14T15:25:47.281406+0000","last_active":"2024-04-14T15:26:06.026235+0000","last_peered":"2024-04-14T15:26:06.026235+0000","last_clean":"2024-04-14T15:26:06.026235+0000","last_became_active":"2024-04-14T15:20:52.989635+0000","last_became_peered":"2024-04-14T15:20:52.989635+0000","last_unstale":"2024-04-14T15:26:06.026235+0000","last_undegraded":"2024-04-14T15:26:06.026235+0000","last_fullsized":"2024-04-14T15:26:06.026235+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'28","last_scrub_stamp":"2024-04-14T15:25:47.281304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:25:47.281304+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:02.110971+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":604,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.068924+0000","last_change":"2024-04-14T15:25:46.849300+0000","last_active":"2024-04-14T15:26:06.068924+0000","last_peered":"2024-04-14T15:26:06.068924+0000","last_clean":"2024-04-14T15:26:06.068924+0000","last_became_active":"2024-04-14T15:20:52.999379+0000","last_became_peered":"2024-04-14T15:20:52.999379+0000","last_unstale":"2024-04-14T15:26:06.068924+0000","last_undegraded":"2024-04-14T15:26:06.068924+0000","last_fullsized":"2024-04-14T15:26:06.068924+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:46.849220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:25:46.849220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:58.079083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1550,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.068254+0000","last_change":"2024-04-14T15:23:28.571478+0000","last_active":"2024-04-14T15:26:06.068254+0000","last_peered":"2024-04-14T15:26:06.068254+0000","last_clean":"2024-04-14T15:26:06.068254+0000","last_became_active":"2024-04-14T15:20:52.991388+0000","last_became_peered":"2024-04-14T15:20:52.991388+0000","last_unstale":"2024-04-14T15:26:06.068254+0000","last_undegraded":"2024-04-14T15:26:06.068254+0000","last_fullsized":"2024-04-14T15:26:06.068254+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:28.571374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:23:28.571374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"91'130","reported_seq":1720,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.027206+0000","last_change":"2024-04-14T15:25:04.328620+0000","last_active":"2024-04-14T15:26:06.027206+0000","last_peered":"2024-04-14T15:26:06.027206+0000","last_clean":"2024-04-14T15:26:06.027206+0000","last_became_active":"2024-04-14T15:24:57.015409+0000","last_became_peered":"2024-04-14T15:24:57.015409+0000","last_unstale":"2024-04-14T15:26:06.027206+0000","last_undegraded":"2024-04-14T15:26:06.027206+0000","last_fullsized":"2024-04-14T15:26:06.027206+0000","mapping_epoch":530,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":753,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:25:04.328526+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:25:04.328526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:29.270285+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":1460,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.068759+0000","last_change":"2024-04-14T15:24:27.475544+0000","last_active":"2024-04-14T15:26:06.068759+0000","last_peered":"2024-04-14T15:26:06.068759+0000","last_clean":"2024-04-14T15:26:06.068759+0000","last_became_active":"2024-04-14T15:12:56.355254+0000","last_became_peered":"2024-04-14T15:12:56.355254+0000","last_unstale":"2024-04-14T15:26:06.068759+0000","last_undegraded":"2024-04-14T15:26:06.068759+0000","last_fullsized":"2024-04-14T15:26:06.068759+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:27.475460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:24:27.475460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1556,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.026487+0000","last_change":"2024-04-14T15:25:52.349307+0000","last_active":"2024-04-14T15:26:06.026487+0000","last_peered":"2024-04-14T15:26:06.026487+0000","last_clean":"2024-04-14T15:26:06.026487+0000","last_became_active":"2024-04-14T15:09:47.044610+0000","last_became_peered":"2024-04-14T15:09:47.044610+0000","last_unstale":"2024-04-14T15:26:06.026487+0000","last_undegraded":"2024-04-14T15:26:06.026487+0000","last_fullsized":"2024-04-14T15:26:06.026487+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:52.349217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:39.394655+0000","last_clean_scrub_stamp":"2024-04-14T15:25:52.349217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:02.647785+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"783'38","reported_seq":1299,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.062743+0000","last_change":"2024-04-14T15:25:27.804694+0000","last_active":"2024-04-14T15:26:06.062743+0000","last_peered":"2024-04-14T15:26:06.062743+0000","last_clean":"2024-04-14T15:26:06.062743+0000","last_became_active":"2024-04-14T15:15:21.880565+0000","last_became_peered":"2024-04-14T15:15:21.880565+0000","last_unstale":"2024-04-14T15:26:06.062743+0000","last_undegraded":"2024-04-14T15:26:06.062743+0000","last_fullsized":"2024-04-14T15:26:06.062743+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"783'38","last_scrub_stamp":"2024-04-14T15:25:27.804604+0000","last_deep_scrub":"783'38","last_deep_scrub_stamp":"2024-04-14T15:25:27.804604+0000","last_clean_scrub_stamp":"2024-04-14T15:25:27.804604+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:49.748235+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'2","reported_seq":1572,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.032328+0000","last_change":"2024-04-14T15:25:05.498299+0000","last_active":"2024-04-14T15:26:06.032328+0000","last_peered":"2024-04-14T15:26:06.032328+0000","last_clean":"2024-04-14T15:26:06.032328+0000","last_became_active":"2024-04-14T15:20:52.987576+0000","last_became_peered":"2024-04-14T15:20:52.987576+0000","last_unstale":"2024-04-14T15:26:06.032328+0000","last_undegraded":"2024-04-14T15:26:06.032328+0000","last_fullsized":"2024-04-14T15:26:06.032328+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'2","last_scrub_stamp":"2024-04-14T15:25:05.498215+0000","last_deep_scrub":"91'2","last_deep_scrub_stamp":"2024-04-14T15:17:48.850069+0000","last_clean_scrub_stamp":"2024-04-14T15:25:05.498215+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:29.023294+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1544,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.026872+0000","last_change":"2024-04-14T15:23:33.834918+0000","last_active":"2024-04-14T15:26:06.026872+0000","last_peered":"2024-04-14T15:26:06.026872+0000","last_clean":"2024-04-14T15:26:06.026872+0000","last_became_active":"2024-04-14T15:09:49.065684+0000","last_became_peered":"2024-04-14T15:09:49.065684+0000","last_unstale":"2024-04-14T15:26:06.026872+0000","last_undegraded":"2024-04-14T15:26:06.026872+0000","last_fullsized":"2024-04-14T15:26:06.026872+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:33.834838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:23:33.834838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"91'130","reported_seq":990,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.033535+0000","last_change":"2024-04-14T15:25:04.746467+0000","last_active":"2024-04-14T15:26:06.033535+0000","last_peered":"2024-04-14T15:26:06.033535+0000","last_clean":"2024-04-14T15:26:06.033535+0000","last_became_active":"2024-04-14T15:24:57.025580+0000","last_became_peered":"2024-04-14T15:24:57.025580+0000","last_unstale":"2024-04-14T15:26:06.033535+0000","last_undegraded":"2024-04-14T15:26:06.033535+0000","last_fullsized":"2024-04-14T15:26:06.033535+0000","mapping_epoch":530,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":753,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:25:04.746393+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:25:04.746393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:33.227095+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":1472,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.031881+0000","last_change":"2024-04-14T15:26:01.161438+0000","last_active":"2024-04-14T15:26:06.031881+0000","last_peered":"2024-04-14T15:26:06.031881+0000","last_clean":"2024-04-14T15:26:06.031881+0000","last_became_active":"2024-04-14T15:12:56.402023+0000","last_became_peered":"2024-04-14T15:12:56.402023+0000","last_unstale":"2024-04-14T15:26:06.031881+0000","last_undegraded":"2024-04-14T15:26:06.031881+0000","last_fullsized":"2024-04-14T15:26:06.031881+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.161362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:35.003968+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.161362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:23.708036+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":602,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.029516+0000","last_change":"2024-04-14T15:24:18.368805+0000","last_active":"2024-04-14T15:26:06.029516+0000","last_peered":"2024-04-14T15:26:06.029516+0000","last_clean":"2024-04-14T15:26:06.029516+0000","last_became_active":"2024-04-14T15:20:52.987883+0000","last_became_peered":"2024-04-14T15:20:52.987883+0000","last_unstale":"2024-04-14T15:26:06.029516+0000","last_undegraded":"2024-04-14T15:26:06.029516+0000","last_fullsized":"2024-04-14T15:26:06.029516+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:18.368725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:24:18.368725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"479'37","reported_seq":1626,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.031342+0000","last_change":"2024-04-14T15:25:58.107928+0000","last_active":"2024-04-14T15:26:06.031342+0000","last_peered":"2024-04-14T15:26:06.031342+0000","last_clean":"2024-04-14T15:26:06.031342+0000","last_became_active":"2024-04-14T15:09:45.179182+0000","last_became_peered":"2024-04-14T15:09:45.179182+0000","last_unstale":"2024-04-14T15:26:06.031342+0000","last_undegraded":"2024-04-14T15:26:06.031342+0000","last_fullsized":"2024-04-14T15:26:06.031342+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'37","last_scrub_stamp":"2024-04-14T15:25:58.107835+0000","last_deep_scrub":"102'27","last_deep_scrub_stamp":"2024-04-14T15:17:46.924330+0000","last_clean_scrub_stamp":"2024-04-14T15:25:58.107835+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:03.487475+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1564,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.068577+0000","last_change":"2024-04-14T15:25:55.977430+0000","last_active":"2024-04-14T15:26:06.068577+0000","last_peered":"2024-04-14T15:26:06.068577+0000","last_clean":"2024-04-14T15:26:06.068577+0000","last_became_active":"2024-04-14T15:09:43.021006+0000","last_became_peered":"2024-04-14T15:09:43.021006+0000","last_unstale":"2024-04-14T15:26:06.068577+0000","last_undegraded":"2024-04-14T15:26:06.068577+0000","last_fullsized":"2024-04-14T15:26:06.068577+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:55.977352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:25:55.977352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:56.535906+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1554,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.029679+0000","last_change":"2024-04-14T15:23:34.182007+0000","last_active":"2024-04-14T15:26:06.029679+0000","last_peered":"2024-04-14T15:26:06.029679+0000","last_clean":"2024-04-14T15:26:06.029679+0000","last_became_active":"2024-04-14T15:20:52.978765+0000","last_became_peered":"2024-04-14T15:20:52.978765+0000","last_unstale":"2024-04-14T15:26:06.029679+0000","last_undegraded":"2024-04-14T15:26:06.029679+0000","last_fullsized":"2024-04-14T15:26:06.029679+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:34.181914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:56.206649+0000","last_clean_scrub_stamp":"2024-04-14T15:23:34.181914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"91'130","reported_seq":1719,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.030713+0000","last_change":"2024-04-14T15:25:03.514185+0000","last_active":"2024-04-14T15:26:06.030713+0000","last_peered":"2024-04-14T15:26:06.030713+0000","last_clean":"2024-04-14T15:26:06.030713+0000","last_became_active":"2024-04-14T15:24:57.012410+0000","last_became_peered":"2024-04-14T15:24:57.012410+0000","last_unstale":"2024-04-14T15:26:06.030713+0000","last_undegraded":"2024-04-14T15:26:06.030713+0000","last_fullsized":"2024-04-14T15:26:06.030713+0000","mapping_epoch":530,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":753,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:25:03.514098+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:25:03.514098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:33.492762+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":1177,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.062924+0000","last_change":"2024-04-14T15:25:53.791322+0000","last_active":"2024-04-14T15:26:06.062924+0000","last_peered":"2024-04-14T15:26:06.062924+0000","last_clean":"2024-04-14T15:26:06.062924+0000","last_became_active":"2024-04-14T15:20:52.999776+0000","last_became_peered":"2024-04-14T15:20:52.999776+0000","last_unstale":"2024-04-14T15:26:06.062924+0000","last_undegraded":"2024-04-14T15:26:06.062924+0000","last_fullsized":"2024-04-14T15:26:06.062924+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:53.791183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:25:53.791183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:19.819673+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1118,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.033244+0000","last_change":"2024-04-14T15:25:14.592763+0000","last_active":"2024-04-14T15:26:06.033244+0000","last_peered":"2024-04-14T15:26:06.033244+0000","last_clean":"2024-04-14T15:26:06.033244+0000","last_became_active":"2024-04-14T15:15:44.883694+0000","last_became_peered":"2024-04-14T15:15:44.883694+0000","last_unstale":"2024-04-14T15:26:06.033244+0000","last_undegraded":"2024-04-14T15:26:06.033244+0000","last_fullsized":"2024-04-14T15:26:06.033244+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:14.592662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:25:14.592662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:40.007813+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"479'30","reported_seq":1601,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.033653+0000","last_change":"2024-04-14T15:23:30.295880+0000","last_active":"2024-04-14T15:26:06.033653+0000","last_peered":"2024-04-14T15:26:06.033653+0000","last_clean":"2024-04-14T15:26:06.033653+0000","last_became_active":"2024-04-14T15:09:45.037231+0000","last_became_peered":"2024-04-14T15:09:45.037231+0000","last_unstale":"2024-04-14T15:26:06.033653+0000","last_undegraded":"2024-04-14T15:26:06.033653+0000","last_fullsized":"2024-04-14T15:26:06.033653+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'30","last_scrub_stamp":"2024-04-14T15:23:30.295747+0000","last_deep_scrub":"182'18","last_deep_scrub_stamp":"2024-04-14T15:14:10.383906+0000","last_clean_scrub_stamp":"2024-04-14T15:23:30.295747+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1565,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.067917+0000","last_change":"2024-04-14T15:24:29.563849+0000","last_active":"2024-04-14T15:26:06.067917+0000","last_peered":"2024-04-14T15:26:06.067917+0000","last_clean":"2024-04-14T15:26:06.067917+0000","last_became_active":"2024-04-14T15:09:43.025095+0000","last_became_peered":"2024-04-14T15:09:43.025095+0000","last_unstale":"2024-04-14T15:26:06.067917+0000","last_undegraded":"2024-04-14T15:26:06.067917+0000","last_fullsized":"2024-04-14T15:26:06.067917+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:29.563759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:29.563759+0000","last_clean_scrub_stamp":"2024-04-14T15:24:29.563759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1570,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.030505+0000","last_change":"2024-04-14T15:23:40.238604+0000","last_active":"2024-04-14T15:26:06.030505+0000","last_peered":"2024-04-14T15:26:06.030505+0000","last_clean":"2024-04-14T15:26:06.030505+0000","last_became_active":"2024-04-14T15:09:43.018509+0000","last_became_peered":"2024-04-14T15:09:43.018509+0000","last_unstale":"2024-04-14T15:26:06.030505+0000","last_undegraded":"2024-04-14T15:26:06.030505+0000","last_fullsized":"2024-04-14T15:26:06.030505+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:40.238527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:23:40.238527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0018592680000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"355'2","reported_seq":1550,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.032088+0000","last_change":"2024-04-14T15:23:43.059738+0000","last_active":"2024-04-14T15:26:06.032088+0000","last_peered":"2024-04-14T15:26:06.032088+0000","last_clean":"2024-04-14T15:26:06.032088+0000","last_became_active":"2024-04-14T15:09:49.065243+0000","last_became_peered":"2024-04-14T15:09:49.065243+0000","last_unstale":"2024-04-14T15:26:06.032088+0000","last_undegraded":"2024-04-14T15:26:06.032088+0000","last_fullsized":"2024-04-14T15:26:06.032088+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'2","last_scrub_stamp":"2024-04-14T15:23:43.059603+0000","last_deep_scrub":"355'2","last_deep_scrub_stamp":"2024-04-14T15:20:04.842881+0000","last_clean_scrub_stamp":"2024-04-14T15:23:43.059603+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":1161,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.063164+0000","last_change":"2024-04-14T15:24:28.904419+0000","last_active":"2024-04-14T15:26:06.063164+0000","last_peered":"2024-04-14T15:26:06.063164+0000","last_clean":"2024-04-14T15:26:06.063164+0000","last_became_active":"2024-04-14T15:20:53.000058+0000","last_became_peered":"2024-04-14T15:20:53.000058+0000","last_unstale":"2024-04-14T15:26:06.063164+0000","last_undegraded":"2024-04-14T15:26:06.063164+0000","last_fullsized":"2024-04-14T15:26:06.063164+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:28.904336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:24:28.904336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1556,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.068482+0000","last_change":"2024-04-14T15:26:00.992934+0000","last_active":"2024-04-14T15:26:06.068482+0000","last_peered":"2024-04-14T15:26:06.068482+0000","last_clean":"2024-04-14T15:26:06.068482+0000","last_became_active":"2024-04-14T15:09:47.047201+0000","last_became_peered":"2024-04-14T15:09:47.047201+0000","last_unstale":"2024-04-14T15:26:06.068482+0000","last_undegraded":"2024-04-14T15:26:06.068482+0000","last_fullsized":"2024-04-14T15:26:06.068482+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:00.992853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:26:00.992853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:05.677476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"528'37","reported_seq":1724,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.027551+0000","last_change":"2024-04-14T15:26:00.375445+0000","last_active":"2024-04-14T15:26:06.027551+0000","last_peered":"2024-04-14T15:26:06.027551+0000","last_clean":"2024-04-14T15:26:06.027551+0000","last_became_active":"2024-04-14T15:20:53.002282+0000","last_became_peered":"2024-04-14T15:20:53.002282+0000","last_unstale":"2024-04-14T15:26:06.027551+0000","last_undegraded":"2024-04-14T15:26:06.027551+0000","last_fullsized":"2024-04-14T15:26:06.027551+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"528'37","last_scrub_stamp":"2024-04-14T15:26:00.375331+0000","last_deep_scrub":"90'24","last_deep_scrub_stamp":"2024-04-14T15:11:25.120692+0000","last_clean_scrub_stamp":"2024-04-14T15:26:00.375331+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:08.881036+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"91'130","reported_seq":1917,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.067883+0000","last_change":"2024-04-14T15:25:04.953500+0000","last_active":"2024-04-14T15:26:06.067883+0000","last_peered":"2024-04-14T15:26:06.067883+0000","last_clean":"2024-04-14T15:26:06.067883+0000","last_became_active":"2024-04-14T15:24:57.020227+0000","last_became_peered":"2024-04-14T15:24:57.020227+0000","last_unstale":"2024-04-14T15:26:06.067883+0000","last_undegraded":"2024-04-14T15:26:06.067883+0000","last_fullsized":"2024-04-14T15:26:06.067883+0000","mapping_epoch":530,"log_start":"91'130","ondisk_log_start":"91'130","created":19,"last_epoch_clean":753,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:25:04.953418+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:25:04.953418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:12.268060+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1128,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.060195+0000","last_change":"2024-04-14T15:25:48.704691+0000","last_active":"2024-04-14T15:26:06.060195+0000","last_peered":"2024-04-14T15:26:06.060195+0000","last_clean":"2024-04-14T15:26:06.060195+0000","last_became_active":"2024-04-14T15:15:44.885315+0000","last_became_peered":"2024-04-14T15:15:44.885315+0000","last_unstale":"2024-04-14T15:26:06.060195+0000","last_undegraded":"2024-04-14T15:26:06.060195+0000","last_fullsized":"2024-04-14T15:26:06.060195+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:48.704559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:47.627305+0000","last_clean_scrub_stamp":"2024-04-14T15:25:48.704559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:50.327058+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"813'3","reported_seq":1265,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.063174+0000","last_change":"2024-04-14T15:26:03.783407+0000","last_active":"2024-04-14T15:26:06.063174+0000","last_peered":"2024-04-14T15:26:06.063174+0000","last_clean":"2024-04-14T15:26:06.063174+0000","last_became_active":"2024-04-14T15:20:53.002545+0000","last_became_peered":"2024-04-14T15:20:53.002545+0000","last_unstale":"2024-04-14T15:26:06.063174+0000","last_undegraded":"2024-04-14T15:26:06.063174+0000","last_fullsized":"2024-04-14T15:26:06.063174+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"813'3","last_scrub_stamp":"2024-04-14T15:26:03.783312+0000","last_deep_scrub":"813'3","last_deep_scrub_stamp":"2024-04-14T15:26:03.783312+0000","last_clean_scrub_stamp":"2024-04-14T15:26:03.783312+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:28.963379+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0058046169999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1110,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.064642+0000","last_change":"2024-04-14T15:23:33.934414+0000","last_active":"2024-04-14T15:26:06.064642+0000","last_peered":"2024-04-14T15:26:06.064642+0000","last_clean":"2024-04-14T15:26:06.064642+0000","last_became_active":"2024-04-14T15:15:44.887934+0000","last_became_peered":"2024-04-14T15:15:44.887934+0000","last_unstale":"2024-04-14T15:26:06.064642+0000","last_undegraded":"2024-04-14T15:26:06.064642+0000","last_fullsized":"2024-04-14T15:26:06.064642+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:33.934334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:23:33.934334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"91'130","reported_seq":1917,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.068004+0000","last_change":"2024-04-14T15:25:03.960485+0000","last_active":"2024-04-14T15:26:06.068004+0000","last_peered":"2024-04-14T15:26:06.068004+0000","last_clean":"2024-04-14T15:26:06.068004+0000","last_became_active":"2024-04-14T15:24:57.011910+0000","last_became_peered":"2024-04-14T15:24:57.011910+0000","last_unstale":"2024-04-14T15:26:06.068004+0000","last_undegraded":"2024-04-14T15:26:06.068004+0000","last_fullsized":"2024-04-14T15:26:06.068004+0000","mapping_epoch":189,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":753,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:25:03.960401+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:22:17.518666+0000","last_clean_scrub_stamp":"2024-04-14T15:25:03.960401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:08.114054+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":1169,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.062629+0000","last_change":"2024-04-14T15:25:18.772143+0000","last_active":"2024-04-14T15:26:06.062629+0000","last_peered":"2024-04-14T15:26:06.062629+0000","last_clean":"2024-04-14T15:26:06.062629+0000","last_became_active":"2024-04-14T15:20:52.986308+0000","last_became_peered":"2024-04-14T15:20:52.986308+0000","last_unstale":"2024-04-14T15:26:06.062629+0000","last_undegraded":"2024-04-14T15:26:06.062629+0000","last_fullsized":"2024-04-14T15:26:06.062629+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:18.772063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:25:18.772063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:44.850845+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1561,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.061265+0000","last_change":"2024-04-14T15:25:22.763238+0000","last_active":"2024-04-14T15:26:06.061265+0000","last_peered":"2024-04-14T15:26:06.061265+0000","last_clean":"2024-04-14T15:26:06.061265+0000","last_became_active":"2024-04-14T15:09:47.049457+0000","last_became_peered":"2024-04-14T15:09:47.049457+0000","last_unstale":"2024-04-14T15:26:06.061265+0000","last_undegraded":"2024-04-14T15:26:06.061265+0000","last_fullsized":"2024-04-14T15:26:06.061265+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:22.763147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:25:22.763147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:42.033176+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"479'22","reported_seq":1602,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.033065+0000","last_change":"2024-04-14T15:25:56.109765+0000","last_active":"2024-04-14T15:26:06.033065+0000","last_peered":"2024-04-14T15:26:06.033065+0000","last_clean":"2024-04-14T15:26:06.033065+0000","last_became_active":"2024-04-14T15:09:45.033883+0000","last_became_peered":"2024-04-14T15:09:45.033883+0000","last_unstale":"2024-04-14T15:26:06.033065+0000","last_undegraded":"2024-04-14T15:26:06.033065+0000","last_fullsized":"2024-04-14T15:26:06.033065+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'22","last_scrub_stamp":"2024-04-14T15:25:56.109673+0000","last_deep_scrub":"308'18","last_deep_scrub_stamp":"2024-04-14T15:16:16.054032+0000","last_clean_scrub_stamp":"2024-04-14T15:25:56.109673+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:12.349554+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1569,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.060148+0000","last_change":"2024-04-14T15:24:29.474621+0000","last_active":"2024-04-14T15:26:06.060148+0000","last_peered":"2024-04-14T15:26:06.060148+0000","last_clean":"2024-04-14T15:26:06.060148+0000","last_became_active":"2024-04-14T15:09:43.021188+0000","last_became_peered":"2024-04-14T15:09:43.021188+0000","last_unstale":"2024-04-14T15:26:06.060148+0000","last_undegraded":"2024-04-14T15:26:06.060148+0000","last_fullsized":"2024-04-14T15:26:06.060148+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:29.474544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:24:29.474544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:10.772028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"73'5","reported_seq":2567,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:07.482814+0000","last_change":"2024-04-14T15:25:26.770287+0000","last_active":"2024-04-14T15:26:07.482814+0000","last_peered":"2024-04-14T15:26:07.482814+0000","last_clean":"2024-04-14T15:26:07.482814+0000","last_became_active":"2024-04-14T15:20:52.975473+0000","last_became_peered":"2024-04-14T15:20:52.975473+0000","last_unstale":"2024-04-14T15:26:07.482814+0000","last_undegraded":"2024-04-14T15:26:07.482814+0000","last_fullsized":"2024-04-14T15:26:07.482814+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-04-14T15:25:26.770199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:25:26.770199+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:30.974644+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00056542099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":977,"num_read_kb":972,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1558,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.068611+0000","last_change":"2024-04-14T15:26:03.955505+0000","last_active":"2024-04-14T15:26:06.068611+0000","last_peered":"2024-04-14T15:26:06.068611+0000","last_clean":"2024-04-14T15:26:06.068611+0000","last_became_active":"2024-04-14T15:20:52.999464+0000","last_became_peered":"2024-04-14T15:20:52.999464+0000","last_unstale":"2024-04-14T15:26:06.068611+0000","last_undegraded":"2024-04-14T15:26:06.068611+0000","last_fullsized":"2024-04-14T15:26:06.068611+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:03.955419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:20.827589+0000","last_clean_scrub_stamp":"2024-04-14T15:26:03.955419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:19.378832+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"91'130","reported_seq":1537,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.030305+0000","last_change":"2024-04-14T15:25:03.722817+0000","last_active":"2024-04-14T15:26:06.030305+0000","last_peered":"2024-04-14T15:26:06.030305+0000","last_clean":"2024-04-14T15:26:06.030305+0000","last_became_active":"2024-04-14T15:24:57.020139+0000","last_became_peered":"2024-04-14T15:24:57.020139+0000","last_unstale":"2024-04-14T15:26:06.030305+0000","last_undegraded":"2024-04-14T15:26:06.030305+0000","last_fullsized":"2024-04-14T15:26:06.030305+0000","mapping_epoch":276,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":753,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:25:03.722693+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:25:03.722693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:20.430755+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":1163,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.063452+0000","last_change":"2024-04-14T15:25:21.855268+0000","last_active":"2024-04-14T15:26:06.063452+0000","last_peered":"2024-04-14T15:26:06.063452+0000","last_clean":"2024-04-14T15:26:06.063452+0000","last_became_active":"2024-04-14T15:15:21.883717+0000","last_became_peered":"2024-04-14T15:15:21.883717+0000","last_unstale":"2024-04-14T15:26:06.063452+0000","last_undegraded":"2024-04-14T15:26:06.063452+0000","last_fullsized":"2024-04-14T15:26:06.063452+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:21.855183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:25:21.855183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:36.412681+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1555,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.033236+0000","last_change":"2024-04-14T15:24:26.376324+0000","last_active":"2024-04-14T15:26:06.033236+0000","last_peered":"2024-04-14T15:26:06.033236+0000","last_clean":"2024-04-14T15:26:06.033236+0000","last_became_active":"2024-04-14T15:09:47.040185+0000","last_became_peered":"2024-04-14T15:09:47.040185+0000","last_unstale":"2024-04-14T15:26:06.033236+0000","last_undegraded":"2024-04-14T15:26:06.033236+0000","last_fullsized":"2024-04-14T15:26:06.033236+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:26.376179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:53.731610+0000","last_clean_scrub_stamp":"2024-04-14T15:24:26.376179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"479'40","reported_seq":1820,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.031862+0000","last_change":"2024-04-14T15:23:31.198816+0000","last_active":"2024-04-14T15:26:06.031862+0000","last_peered":"2024-04-14T15:26:06.031862+0000","last_clean":"2024-04-14T15:26:06.031862+0000","last_became_active":"2024-04-14T15:09:45.034168+0000","last_became_peered":"2024-04-14T15:09:45.034168+0000","last_unstale":"2024-04-14T15:26:06.031862+0000","last_undegraded":"2024-04-14T15:26:06.031862+0000","last_fullsized":"2024-04-14T15:26:06.031862+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'40","last_scrub_stamp":"2024-04-14T15:23:31.198708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:23:31.198708+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1125,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.030480+0000","last_change":"2024-04-14T15:23:30.217883+0000","last_active":"2024-04-14T15:26:06.030480+0000","last_peered":"2024-04-14T15:26:06.030480+0000","last_clean":"2024-04-14T15:26:06.030480+0000","last_became_active":"2024-04-14T15:20:52.979943+0000","last_became_peered":"2024-04-14T15:20:52.979943+0000","last_unstale":"2024-04-14T15:26:06.030480+0000","last_undegraded":"2024-04-14T15:26:06.030480+0000","last_fullsized":"2024-04-14T15:26:06.030480+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:30.217808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:24.066275+0000","last_clean_scrub_stamp":"2024-04-14T15:23:30.217808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1215,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.064379+0000","last_change":"2024-04-14T15:24:30.949646+0000","last_active":"2024-04-14T15:26:06.064379+0000","last_peered":"2024-04-14T15:26:06.064379+0000","last_clean":"2024-04-14T15:26:06.064379+0000","last_became_active":"2024-04-14T15:15:21.874742+0000","last_became_peered":"2024-04-14T15:15:21.874742+0000","last_unstale":"2024-04-14T15:26:06.064379+0000","last_undegraded":"2024-04-14T15:26:06.064379+0000","last_fullsized":"2024-04-14T15:26:06.064379+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:30.949569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:30.949569+0000","last_clean_scrub_stamp":"2024-04-14T15:24:30.949569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"91'130","reported_seq":993,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.064361+0000","last_change":"2024-04-14T15:25:05.749046+0000","last_active":"2024-04-14T15:26:06.064361+0000","last_peered":"2024-04-14T15:26:06.064361+0000","last_clean":"2024-04-14T15:26:06.064361+0000","last_became_active":"2024-04-14T15:24:57.023069+0000","last_became_peered":"2024-04-14T15:24:57.023069+0000","last_unstale":"2024-04-14T15:26:06.064361+0000","last_undegraded":"2024-04-14T15:26:06.064361+0000","last_fullsized":"2024-04-14T15:26:06.064361+0000","mapping_epoch":530,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":753,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:25:05.748903+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:25:05.748903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:22.990045+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":1471,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.061059+0000","last_change":"2024-04-14T15:25:21.747973+0000","last_active":"2024-04-14T15:26:06.061059+0000","last_peered":"2024-04-14T15:26:06.061059+0000","last_clean":"2024-04-14T15:26:06.061059+0000","last_became_active":"2024-04-14T15:20:52.989272+0000","last_became_peered":"2024-04-14T15:20:52.989272+0000","last_unstale":"2024-04-14T15:26:06.061059+0000","last_undegraded":"2024-04-14T15:26:06.061059+0000","last_fullsized":"2024-04-14T15:26:06.061059+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:21.747886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:25:21.747886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:46.136245+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"773'10","reported_seq":1780,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.029260+0000","last_change":"2024-04-14T15:25:17.080605+0000","last_active":"2024-04-14T15:26:06.029260+0000","last_peered":"2024-04-14T15:26:06.029260+0000","last_clean":"2024-04-14T15:26:06.029260+0000","last_became_active":"2024-04-14T15:20:53.000669+0000","last_became_peered":"2024-04-14T15:20:53.000669+0000","last_unstale":"2024-04-14T15:26:06.029260+0000","last_undegraded":"2024-04-14T15:26:06.029260+0000","last_fullsized":"2024-04-14T15:26:06.029260+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"773'10","last_scrub_stamp":"2024-04-14T15:25:17.080518+0000","last_deep_scrub":"773'10","last_deep_scrub_stamp":"2024-04-14T15:25:17.080518+0000","last_clean_scrub_stamp":"2024-04-14T15:25:17.080518+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:20.960802+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"479'20","reported_seq":571,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.030195+0000","last_change":"2024-04-14T15:25:19.646343+0000","last_active":"2024-04-14T15:26:06.030195+0000","last_peered":"2024-04-14T15:26:06.030195+0000","last_clean":"2024-04-14T15:26:06.030195+0000","last_became_active":"2024-04-14T15:21:16.034027+0000","last_became_peered":"2024-04-14T15:21:16.034027+0000","last_unstale":"2024-04-14T15:26:06.030195+0000","last_undegraded":"2024-04-14T15:26:06.030195+0000","last_fullsized":"2024-04-14T15:26:06.030195+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'20","last_scrub_stamp":"2024-04-14T15:25:19.646254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:25:19.646254+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:28.326862+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1560,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.031788+0000","last_change":"2024-04-14T15:25:09.533597+0000","last_active":"2024-04-14T15:26:06.031788+0000","last_peered":"2024-04-14T15:26:06.031788+0000","last_clean":"2024-04-14T15:26:06.031788+0000","last_became_active":"2024-04-14T15:09:43.022597+0000","last_became_peered":"2024-04-14T15:09:43.022597+0000","last_unstale":"2024-04-14T15:26:06.031788+0000","last_undegraded":"2024-04-14T15:26:06.031788+0000","last_fullsized":"2024-04-14T15:26:06.031788+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:09.533502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:21:28.684818+0000","last_clean_scrub_stamp":"2024-04-14T15:25:09.533502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:14.810582+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1552,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.030981+0000","last_change":"2024-04-14T15:24:25.330276+0000","last_active":"2024-04-14T15:26:06.030981+0000","last_peered":"2024-04-14T15:26:06.030981+0000","last_clean":"2024-04-14T15:26:06.030981+0000","last_became_active":"2024-04-14T15:09:49.066648+0000","last_became_peered":"2024-04-14T15:09:49.066648+0000","last_unstale":"2024-04-14T15:26:06.030981+0000","last_undegraded":"2024-04-14T15:26:06.030981+0000","last_fullsized":"2024-04-14T15:26:06.030981+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:25.330148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:24:25.330148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.c","version":"91'130","reported_seq":1733,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.063478+0000","last_change":"2024-04-14T15:25:50.734599+0000","last_active":"2024-04-14T15:26:06.063478+0000","last_peered":"2024-04-14T15:26:06.063478+0000","last_clean":"2024-04-14T15:26:06.063478+0000","last_became_active":"2024-04-14T15:25:32.025973+0000","last_became_peered":"2024-04-14T15:25:32.025973+0000","last_unstale":"2024-04-14T15:26:06.063478+0000","last_undegraded":"2024-04-14T15:26:06.063478+0000","last_fullsized":"2024-04-14T15:26:06.063478+0000","mapping_epoch":787,"log_start":"91'130","ondisk_log_start":"91'130","created":752,"last_epoch_clean":788,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:25:50.734496+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:25:50.734496+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":60,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:57.399176+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":1460,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.033542+0000","last_change":"2024-04-14T15:23:44.358966+0000","last_active":"2024-04-14T15:26:06.033542+0000","last_peered":"2024-04-14T15:26:06.033542+0000","last_clean":"2024-04-14T15:26:06.033542+0000","last_became_active":"2024-04-14T15:12:56.563074+0000","last_became_peered":"2024-04-14T15:12:56.563074+0000","last_unstale":"2024-04-14T15:26:06.033542+0000","last_undegraded":"2024-04-14T15:26:06.033542+0000","last_fullsized":"2024-04-14T15:26:06.033542+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:44.358887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:23:44.358887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1561,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.059930+0000","last_change":"2024-04-14T15:25:57.664021+0000","last_active":"2024-04-14T15:26:06.059930+0000","last_peered":"2024-04-14T15:26:06.059930+0000","last_clean":"2024-04-14T15:26:06.059930+0000","last_became_active":"2024-04-14T15:09:47.049005+0000","last_became_peered":"2024-04-14T15:09:47.049005+0000","last_unstale":"2024-04-14T15:26:06.059930+0000","last_undegraded":"2024-04-14T15:26:06.059930+0000","last_fullsized":"2024-04-14T15:26:06.059930+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:57.663924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:56.915544+0000","last_clean_scrub_stamp":"2024-04-14T15:25:57.663924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:04.748057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"479'23","reported_seq":1597,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.026425+0000","last_change":"2024-04-14T15:24:28.394342+0000","last_active":"2024-04-14T15:26:06.026425+0000","last_peered":"2024-04-14T15:26:06.026425+0000","last_clean":"2024-04-14T15:26:06.026425+0000","last_became_active":"2024-04-14T15:09:45.032892+0000","last_became_peered":"2024-04-14T15:09:45.032892+0000","last_unstale":"2024-04-14T15:26:06.026425+0000","last_undegraded":"2024-04-14T15:26:06.026425+0000","last_fullsized":"2024-04-14T15:26:06.026425+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'23","last_scrub_stamp":"2024-04-14T15:24:28.394222+0000","last_deep_scrub":"183'17","last_deep_scrub_stamp":"2024-04-14T15:14:11.577574+0000","last_clean_scrub_stamp":"2024-04-14T15:24:28.394222+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"677'6","reported_seq":1577,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.027713+0000","last_change":"2024-04-14T15:23:38.773659+0000","last_active":"2024-04-14T15:26:06.027713+0000","last_peered":"2024-04-14T15:26:06.027713+0000","last_clean":"2024-04-14T15:26:06.027713+0000","last_became_active":"2024-04-14T15:09:43.020490+0000","last_became_peered":"2024-04-14T15:09:43.020490+0000","last_unstale":"2024-04-14T15:26:06.027713+0000","last_undegraded":"2024-04-14T15:26:06.027713+0000","last_fullsized":"2024-04-14T15:26:06.027713+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"677'6","last_scrub_stamp":"2024-04-14T15:23:38.773539+0000","last_deep_scrub":"677'6","last_deep_scrub_stamp":"2024-04-14T15:23:38.773539+0000","last_clean_scrub_stamp":"2024-04-14T15:23:38.773539+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":528,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1215,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.063694+0000","last_change":"2024-04-14T15:23:40.854331+0000","last_active":"2024-04-14T15:26:06.063694+0000","last_peered":"2024-04-14T15:26:06.063694+0000","last_clean":"2024-04-14T15:26:06.063694+0000","last_became_active":"2024-04-14T15:15:21.876410+0000","last_became_peered":"2024-04-14T15:15:21.876410+0000","last_unstale":"2024-04-14T15:26:06.063694+0000","last_undegraded":"2024-04-14T15:26:06.063694+0000","last_fullsized":"2024-04-14T15:26:06.063694+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:40.854253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:23:40.854253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"91'130","reported_seq":1737,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.030392+0000","last_change":"2024-04-14T15:25:48.950790+0000","last_active":"2024-04-14T15:26:06.030392+0000","last_peered":"2024-04-14T15:26:06.030392+0000","last_clean":"2024-04-14T15:26:06.030392+0000","last_became_active":"2024-04-14T15:25:32.024118+0000","last_became_peered":"2024-04-14T15:25:32.024118+0000","last_unstale":"2024-04-14T15:26:06.030392+0000","last_undegraded":"2024-04-14T15:26:06.030392+0000","last_fullsized":"2024-04-14T15:26:06.030392+0000","mapping_epoch":787,"log_start":"91'130","ondisk_log_start":"91'130","created":752,"last_epoch_clean":788,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:25:48.950683+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:25:48.950683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:54.252465+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":1467,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.030358+0000","last_change":"2024-04-14T15:24:27.371452+0000","last_active":"2024-04-14T15:26:06.030358+0000","last_peered":"2024-04-14T15:26:06.030358+0000","last_clean":"2024-04-14T15:26:06.030358+0000","last_became_active":"2024-04-14T15:12:56.551177+0000","last_became_peered":"2024-04-14T15:12:56.551177+0000","last_unstale":"2024-04-14T15:26:06.030358+0000","last_undegraded":"2024-04-14T15:26:06.030358+0000","last_fullsized":"2024-04-14T15:26:06.030358+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:27.371370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:22:18.299330+0000","last_clean_scrub_stamp":"2024-04-14T15:24:27.371370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"479'19","reported_seq":1587,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.033748+0000","last_change":"2024-04-14T15:24:33.412261+0000","last_active":"2024-04-14T15:26:06.033748+0000","last_peered":"2024-04-14T15:26:06.033748+0000","last_clean":"2024-04-14T15:26:06.033748+0000","last_became_active":"2024-04-14T15:09:45.180375+0000","last_became_peered":"2024-04-14T15:09:45.180375+0000","last_unstale":"2024-04-14T15:26:06.033748+0000","last_undegraded":"2024-04-14T15:26:06.033748+0000","last_fullsized":"2024-04-14T15:26:06.033748+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'19","last_scrub_stamp":"2024-04-14T15:24:33.412181+0000","last_deep_scrub":"347'13","last_deep_scrub_stamp":"2024-04-14T15:16:55.694599+0000","last_clean_scrub_stamp":"2024-04-14T15:24:33.412181+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":1569,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.059920+0000","last_change":"2024-04-14T15:25:16.586933+0000","last_active":"2024-04-14T15:26:06.059920+0000","last_peered":"2024-04-14T15:26:06.059920+0000","last_clean":"2024-04-14T15:26:06.059920+0000","last_became_active":"2024-04-14T15:09:43.016788+0000","last_became_peered":"2024-04-14T15:09:43.016788+0000","last_unstale":"2024-04-14T15:26:06.059920+0000","last_undegraded":"2024-04-14T15:26:06.059920+0000","last_fullsized":"2024-04-14T15:26:06.059920+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:16.586848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:25:16.586848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:43.100059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1235,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.064563+0000","last_change":"2024-04-14T15:24:33.956401+0000","last_active":"2024-04-14T15:26:06.064563+0000","last_peered":"2024-04-14T15:26:06.064563+0000","last_clean":"2024-04-14T15:26:06.064563+0000","last_became_active":"2024-04-14T15:15:21.873580+0000","last_became_peered":"2024-04-14T15:15:21.873580+0000","last_unstale":"2024-04-14T15:26:06.064563+0000","last_undegraded":"2024-04-14T15:26:06.064563+0000","last_fullsized":"2024-04-14T15:26:06.064563+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:33.956328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:24:33.956328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1556,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.060615+0000","last_change":"2024-04-14T15:25:08.595634+0000","last_active":"2024-04-14T15:26:06.060615+0000","last_peered":"2024-04-14T15:26:06.060615+0000","last_clean":"2024-04-14T15:26:06.060615+0000","last_became_active":"2024-04-14T15:09:49.064296+0000","last_became_peered":"2024-04-14T15:09:49.064296+0000","last_unstale":"2024-04-14T15:26:06.060615+0000","last_undegraded":"2024-04-14T15:26:06.060615+0000","last_fullsized":"2024-04-14T15:26:06.060615+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:08.595556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:25:08.595556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:25.224030+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.12","version":"91'130","reported_seq":472,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.029509+0000","last_change":"2024-04-14T15:26:02.145813+0000","last_active":"2024-04-14T15:26:06.029509+0000","last_peered":"2024-04-14T15:26:06.029509+0000","last_clean":"2024-04-14T15:26:06.029509+0000","last_became_active":"2024-04-14T15:26:02.038061+0000","last_became_peered":"2024-04-14T15:26:02.038061+0000","last_unstale":"2024-04-14T15:26:06.029509+0000","last_undegraded":"2024-04-14T15:26:06.029509+0000","last_fullsized":"2024-04-14T15:26:06.029509+0000","mapping_epoch":811,"log_start":"91'130","ondisk_log_start":"91'130","created":752,"last_epoch_clean":812,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:26:02.145733+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:22:17.518666+0000","last_clean_scrub_stamp":"2024-04-14T15:26:02.145733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:13.803728+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":1471,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.059805+0000","last_change":"2024-04-14T15:25:52.667570+0000","last_active":"2024-04-14T15:26:06.059805+0000","last_peered":"2024-04-14T15:26:06.059805+0000","last_clean":"2024-04-14T15:26:06.059805+0000","last_became_active":"2024-04-14T15:20:52.977894+0000","last_became_peered":"2024-04-14T15:20:52.977894+0000","last_unstale":"2024-04-14T15:26:06.059805+0000","last_undegraded":"2024-04-14T15:26:06.059805+0000","last_fullsized":"2024-04-14T15:26:06.059805+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:52.667467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:25:52.667467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:19.080509+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.17","version":"479'12","reported_seq":1572,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.064521+0000","last_change":"2024-04-14T15:24:16.798489+0000","last_active":"2024-04-14T15:26:06.064521+0000","last_peered":"2024-04-14T15:26:06.064521+0000","last_clean":"2024-04-14T15:26:06.064521+0000","last_became_active":"2024-04-14T15:09:45.180037+0000","last_became_peered":"2024-04-14T15:09:45.180037+0000","last_unstale":"2024-04-14T15:26:06.064521+0000","last_undegraded":"2024-04-14T15:26:06.064521+0000","last_fullsized":"2024-04-14T15:26:06.064521+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'12","last_scrub_stamp":"2024-04-14T15:24:16.798377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:24:16.798377+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":596,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.026383+0000","last_change":"2024-04-14T15:23:40.765638+0000","last_active":"2024-04-14T15:26:06.026383+0000","last_peered":"2024-04-14T15:26:06.026383+0000","last_clean":"2024-04-14T15:26:06.026383+0000","last_became_active":"2024-04-14T15:20:52.986974+0000","last_became_peered":"2024-04-14T15:20:52.986974+0000","last_unstale":"2024-04-14T15:26:06.026383+0000","last_undegraded":"2024-04-14T15:26:06.026383+0000","last_fullsized":"2024-04-14T15:26:06.026383+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:40.765537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:59.640565+0000","last_clean_scrub_stamp":"2024-04-14T15:23:40.765537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1109,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.064271+0000","last_change":"2024-04-14T15:24:18.710723+0000","last_active":"2024-04-14T15:26:06.064271+0000","last_peered":"2024-04-14T15:26:06.064271+0000","last_clean":"2024-04-14T15:26:06.064271+0000","last_became_active":"2024-04-14T15:15:44.889786+0000","last_became_peered":"2024-04-14T15:15:44.889786+0000","last_unstale":"2024-04-14T15:26:06.064271+0000","last_undegraded":"2024-04-14T15:26:06.064271+0000","last_fullsized":"2024-04-14T15:26:06.064271+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:18.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:13.735024+0000","last_clean_scrub_stamp":"2024-04-14T15:24:18.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1549,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.068782+0000","last_change":"2024-04-14T15:23:29.527743+0000","last_active":"2024-04-14T15:26:06.068782+0000","last_peered":"2024-04-14T15:26:06.068782+0000","last_clean":"2024-04-14T15:26:06.068782+0000","last_became_active":"2024-04-14T15:20:52.985136+0000","last_became_peered":"2024-04-14T15:20:52.985136+0000","last_unstale":"2024-04-14T15:26:06.068782+0000","last_undegraded":"2024-04-14T15:26:06.068782+0000","last_fullsized":"2024-04-14T15:26:06.068782+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:29.527656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:23:29.527656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.11","version":"91'130","reported_seq":1550,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.030029+0000","last_change":"2024-04-14T15:25:02.713875+0000","last_active":"2024-04-14T15:26:06.030029+0000","last_peered":"2024-04-14T15:26:06.030029+0000","last_clean":"2024-04-14T15:26:06.030029+0000","last_became_active":"2024-04-14T15:25:02.030974+0000","last_became_peered":"2024-04-14T15:25:02.030974+0000","last_unstale":"2024-04-14T15:26:06.030029+0000","last_undegraded":"2024-04-14T15:26:06.030029+0000","last_fullsized":"2024-04-14T15:26:06.030029+0000","mapping_epoch":757,"log_start":"91'130","ondisk_log_start":"91'130","created":752,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:25:02.713791+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:25:02.713791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:32.110890+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":1468,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.026708+0000","last_change":"2024-04-14T15:25:50.325506+0000","last_active":"2024-04-14T15:26:06.026708+0000","last_peered":"2024-04-14T15:26:06.026708+0000","last_clean":"2024-04-14T15:26:06.026708+0000","last_became_active":"2024-04-14T15:12:56.203577+0000","last_became_peered":"2024-04-14T15:12:56.203577+0000","last_unstale":"2024-04-14T15:26:06.026708+0000","last_undegraded":"2024-04-14T15:26:06.026708+0000","last_fullsized":"2024-04-14T15:26:06.026708+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:50.325429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:46.369231+0000","last_clean_scrub_stamp":"2024-04-14T15:25:50.325429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:50.471569+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"479'20","reported_seq":1269,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.063650+0000","last_change":"2024-04-14T15:25:03.783837+0000","last_active":"2024-04-14T15:26:06.063650+0000","last_peered":"2024-04-14T15:26:06.063650+0000","last_clean":"2024-04-14T15:26:06.063650+0000","last_became_active":"2024-04-14T15:15:21.877667+0000","last_became_peered":"2024-04-14T15:15:21.877667+0000","last_unstale":"2024-04-14T15:26:06.063650+0000","last_undegraded":"2024-04-14T15:26:06.063650+0000","last_fullsized":"2024-04-14T15:26:06.063650+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'20","last_scrub_stamp":"2024-04-14T15:25:03.783721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:25:03.783721+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:07.946207+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":1117,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.063300+0000","last_change":"2024-04-14T15:25:57.772181+0000","last_active":"2024-04-14T15:26:06.063300+0000","last_peered":"2024-04-14T15:26:06.063300+0000","last_clean":"2024-04-14T15:26:06.063300+0000","last_became_active":"2024-04-14T15:15:44.887452+0000","last_became_peered":"2024-04-14T15:15:44.887452+0000","last_unstale":"2024-04-14T15:26:06.063300+0000","last_undegraded":"2024-04-14T15:26:06.063300+0000","last_fullsized":"2024-04-14T15:26:06.063300+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:57.772100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:25:57.772100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:01.749540+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"280'9","reported_seq":1268,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.032097+0000","last_change":"2024-04-14T15:25:13.639711+0000","last_active":"2024-04-14T15:26:06.032097+0000","last_peered":"2024-04-14T15:26:06.032097+0000","last_clean":"2024-04-14T15:26:06.032097+0000","last_became_active":"2024-04-14T15:15:44.882916+0000","last_became_peered":"2024-04-14T15:15:44.882916+0000","last_unstale":"2024-04-14T15:26:06.032097+0000","last_undegraded":"2024-04-14T15:26:06.032097+0000","last_fullsized":"2024-04-14T15:26:06.032097+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"280'9","last_scrub_stamp":"2024-04-14T15:25:13.639579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:25:13.639579+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:26.089142+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1550,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.033648+0000","last_change":"2024-04-14T15:25:54.151674+0000","last_active":"2024-04-14T15:26:06.033648+0000","last_peered":"2024-04-14T15:26:06.033648+0000","last_clean":"2024-04-14T15:26:06.033648+0000","last_became_active":"2024-04-14T15:09:49.066068+0000","last_became_peered":"2024-04-14T15:09:49.066068+0000","last_unstale":"2024-04-14T15:26:06.033648+0000","last_undegraded":"2024-04-14T15:26:06.033648+0000","last_fullsized":"2024-04-14T15:26:06.033648+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:54.151593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:54.724681+0000","last_clean_scrub_stamp":"2024-04-14T15:25:54.151593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:55.084660+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"91'130","reported_seq":1935,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.067555+0000","last_change":"2024-04-14T15:25:48.929066+0000","last_active":"2024-04-14T15:26:06.067555+0000","last_peered":"2024-04-14T15:26:06.067555+0000","last_clean":"2024-04-14T15:26:06.067555+0000","last_became_active":"2024-04-14T15:25:32.027404+0000","last_became_peered":"2024-04-14T15:25:32.027404+0000","last_unstale":"2024-04-14T15:26:06.067555+0000","last_undegraded":"2024-04-14T15:26:06.067555+0000","last_fullsized":"2024-04-14T15:26:06.067555+0000","mapping_epoch":787,"log_start":"91'130","ondisk_log_start":"91'130","created":752,"last_epoch_clean":788,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:25:48.928929+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:25:48.928929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:01.358105+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":1109,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.063064+0000","last_change":"2024-04-14T15:24:26.878660+0000","last_active":"2024-04-14T15:26:06.063064+0000","last_peered":"2024-04-14T15:26:06.063064+0000","last_clean":"2024-04-14T15:26:06.063064+0000","last_became_active":"2024-04-14T15:15:44.889106+0000","last_became_peered":"2024-04-14T15:15:44.889106+0000","last_unstale":"2024-04-14T15:26:06.063064+0000","last_undegraded":"2024-04-14T15:26:06.063064+0000","last_fullsized":"2024-04-14T15:26:06.063064+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:26.878584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:24:26.878584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"479'19","reported_seq":1418,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.068669+0000","last_change":"2024-04-14T15:23:46.548415+0000","last_active":"2024-04-14T15:26:06.068669+0000","last_peered":"2024-04-14T15:26:06.068669+0000","last_clean":"2024-04-14T15:26:06.068669+0000","last_became_active":"2024-04-14T15:22:25.971104+0000","last_became_peered":"2024-04-14T15:22:25.971104+0000","last_unstale":"2024-04-14T15:26:06.068669+0000","last_undegraded":"2024-04-14T15:26:06.068669+0000","last_fullsized":"2024-04-14T15:26:06.068669+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'19","last_scrub_stamp":"2024-04-14T15:23:46.548255+0000","last_deep_scrub":"183'13","last_deep_scrub_stamp":"2024-04-14T15:14:11.755252+0000","last_clean_scrub_stamp":"2024-04-14T15:23:46.548255+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1571,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.029426+0000","last_change":"2024-04-14T15:25:48.143955+0000","last_active":"2024-04-14T15:26:06.029426+0000","last_peered":"2024-04-14T15:26:06.029426+0000","last_clean":"2024-04-14T15:26:06.029426+0000","last_became_active":"2024-04-14T15:09:43.019301+0000","last_became_peered":"2024-04-14T15:09:43.019301+0000","last_unstale":"2024-04-14T15:26:06.029426+0000","last_undegraded":"2024-04-14T15:26:06.029426+0000","last_fullsized":"2024-04-14T15:26:06.029426+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:48.143875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:25:48.143875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:13.335549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"255'10","reported_seq":1404,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.063913+0000","last_change":"2024-04-14T15:25:47.681818+0000","last_active":"2024-04-14T15:26:06.063913+0000","last_peered":"2024-04-14T15:26:06.063913+0000","last_clean":"2024-04-14T15:26:06.063913+0000","last_became_active":"2024-04-14T15:15:21.884468+0000","last_became_peered":"2024-04-14T15:15:21.884468+0000","last_unstale":"2024-04-14T15:26:06.063913+0000","last_undegraded":"2024-04-14T15:26:06.063913+0000","last_fullsized":"2024-04-14T15:26:06.063913+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'10","last_scrub_stamp":"2024-04-14T15:25:47.681710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:25:47.681710+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:48.424176+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1543,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.027823+0000","last_change":"2024-04-14T15:24:23.473852+0000","last_active":"2024-04-14T15:26:06.027823+0000","last_peered":"2024-04-14T15:26:06.027823+0000","last_clean":"2024-04-14T15:26:06.027823+0000","last_became_active":"2024-04-14T15:09:49.065242+0000","last_became_peered":"2024-04-14T15:09:49.065242+0000","last_unstale":"2024-04-14T15:26:06.027823+0000","last_undegraded":"2024-04-14T15:26:06.027823+0000","last_fullsized":"2024-04-14T15:26:06.027823+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:23.473749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:24:23.473749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":1460,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.068839+0000","last_change":"2024-04-14T15:25:08.869842+0000","last_active":"2024-04-14T15:26:06.068839+0000","last_peered":"2024-04-14T15:26:06.068839+0000","last_clean":"2024-04-14T15:26:06.068839+0000","last_became_active":"2024-04-14T15:12:56.389204+0000","last_became_peered":"2024-04-14T15:12:56.389204+0000","last_unstale":"2024-04-14T15:26:06.068839+0000","last_undegraded":"2024-04-14T15:26:06.068839+0000","last_fullsized":"2024-04-14T15:26:06.068839+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:08.869762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:25:08.869762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:26.152506+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"479'19","reported_seq":1020,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.060560+0000","last_change":"2024-04-14T15:24:24.516920+0000","last_active":"2024-04-14T15:26:06.060560+0000","last_peered":"2024-04-14T15:26:06.060560+0000","last_clean":"2024-04-14T15:26:06.060560+0000","last_became_active":"2024-04-14T15:20:52.990622+0000","last_became_peered":"2024-04-14T15:20:52.990622+0000","last_unstale":"2024-04-14T15:26:06.060560+0000","last_undegraded":"2024-04-14T15:26:06.060560+0000","last_fullsized":"2024-04-14T15:26:06.060560+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'19","last_scrub_stamp":"2024-04-14T15:24:24.516820+0000","last_deep_scrub":"344'13","last_deep_scrub_stamp":"2024-04-14T15:16:52.738163+0000","last_clean_scrub_stamp":"2024-04-14T15:24:24.516820+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:10.771946+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":601,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.063749+0000","last_change":"2024-04-14T15:25:22.828103+0000","last_active":"2024-04-14T15:26:06.063749+0000","last_peered":"2024-04-14T15:26:06.063749+0000","last_clean":"2024-04-14T15:26:06.063749+0000","last_became_active":"2024-04-14T15:20:52.996548+0000","last_became_peered":"2024-04-14T15:20:52.996548+0000","last_unstale":"2024-04-14T15:26:06.063749+0000","last_undegraded":"2024-04-14T15:26:06.063749+0000","last_fullsized":"2024-04-14T15:26:06.063749+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:22.828025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:25:22.828025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:30.255807+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1227,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.063582+0000","last_change":"2024-04-14T15:24:31.955026+0000","last_active":"2024-04-14T15:26:06.063582+0000","last_peered":"2024-04-14T15:26:06.063582+0000","last_clean":"2024-04-14T15:26:06.063582+0000","last_became_active":"2024-04-14T15:15:21.882643+0000","last_became_peered":"2024-04-14T15:15:21.882643+0000","last_unstale":"2024-04-14T15:26:06.063582+0000","last_undegraded":"2024-04-14T15:26:06.063582+0000","last_fullsized":"2024-04-14T15:26:06.063582+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:31.954940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:33.485411+0000","last_clean_scrub_stamp":"2024-04-14T15:24:31.954940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1554,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.031071+0000","last_change":"2024-04-14T15:24:22.395640+0000","last_active":"2024-04-14T15:26:06.031071+0000","last_peered":"2024-04-14T15:26:06.031071+0000","last_clean":"2024-04-14T15:26:06.031071+0000","last_became_active":"2024-04-14T15:09:49.059711+0000","last_became_peered":"2024-04-14T15:09:49.059711+0000","last_unstale":"2024-04-14T15:26:06.031071+0000","last_undegraded":"2024-04-14T15:26:06.031071+0000","last_fullsized":"2024-04-14T15:26:06.031071+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:22.395562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:24:22.395562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":610,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.068316+0000","last_change":"2024-04-14T15:25:57.008522+0000","last_active":"2024-04-14T15:26:06.068316+0000","last_peered":"2024-04-14T15:26:06.068316+0000","last_clean":"2024-04-14T15:26:06.068316+0000","last_became_active":"2024-04-14T15:22:26.203388+0000","last_became_peered":"2024-04-14T15:22:26.203388+0000","last_unstale":"2024-04-14T15:26:06.068316+0000","last_undegraded":"2024-04-14T15:26:06.068316+0000","last_fullsized":"2024-04-14T15:26:06.068316+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:57.008388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:25:57.008388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:18.120983+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"479'24","reported_seq":1606,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.026827+0000","last_change":"2024-04-14T15:24:15.604196+0000","last_active":"2024-04-14T15:26:06.026827+0000","last_peered":"2024-04-14T15:26:06.026827+0000","last_clean":"2024-04-14T15:26:06.026827+0000","last_became_active":"2024-04-14T15:20:52.989300+0000","last_became_peered":"2024-04-14T15:20:52.989300+0000","last_unstale":"2024-04-14T15:26:06.026827+0000","last_undegraded":"2024-04-14T15:26:06.026827+0000","last_fullsized":"2024-04-14T15:26:06.026827+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'24","last_scrub_stamp":"2024-04-14T15:24:15.604065+0000","last_deep_scrub":"464'16","last_deep_scrub_stamp":"2024-04-14T15:19:17.589999+0000","last_clean_scrub_stamp":"2024-04-14T15:24:15.604065+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":1567,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.032245+0000","last_change":"2024-04-14T15:24:33.369216+0000","last_active":"2024-04-14T15:26:06.032245+0000","last_peered":"2024-04-14T15:26:06.032245+0000","last_clean":"2024-04-14T15:26:06.032245+0000","last_became_active":"2024-04-14T15:09:43.023583+0000","last_became_peered":"2024-04-14T15:09:43.023583+0000","last_unstale":"2024-04-14T15:26:06.032245+0000","last_undegraded":"2024-04-14T15:26:06.032245+0000","last_fullsized":"2024-04-14T15:26:06.032245+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:33.369110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:24:33.369110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1552,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.030587+0000","last_change":"2024-04-14T15:24:19.367902+0000","last_active":"2024-04-14T15:26:06.030587+0000","last_peered":"2024-04-14T15:26:06.030587+0000","last_clean":"2024-04-14T15:26:06.030587+0000","last_became_active":"2024-04-14T15:09:47.041683+0000","last_became_peered":"2024-04-14T15:09:47.041683+0000","last_unstale":"2024-04-14T15:26:06.030587+0000","last_undegraded":"2024-04-14T15:26:06.030587+0000","last_fullsized":"2024-04-14T15:26:06.030587+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:19.367758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:08.486217+0000","last_clean_scrub_stamp":"2024-04-14T15:24:19.367758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1231,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.062740+0000","last_change":"2024-04-14T15:25:58.808326+0000","last_active":"2024-04-14T15:26:06.062740+0000","last_peered":"2024-04-14T15:26:06.062740+0000","last_clean":"2024-04-14T15:26:06.062740+0000","last_became_active":"2024-04-14T15:15:21.889567+0000","last_became_peered":"2024-04-14T15:15:21.889567+0000","last_unstale":"2024-04-14T15:26:06.062740+0000","last_undegraded":"2024-04-14T15:26:06.062740+0000","last_fullsized":"2024-04-14T15:26:06.062740+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:58.808247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:25:58.808247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:06.061451+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":1474,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.061422+0000","last_change":"2024-04-14T15:26:01.742722+0000","last_active":"2024-04-14T15:26:06.061422+0000","last_peered":"2024-04-14T15:26:06.061422+0000","last_clean":"2024-04-14T15:26:06.061422+0000","last_became_active":"2024-04-14T15:12:56.387723+0000","last_became_peered":"2024-04-14T15:12:56.387723+0000","last_unstale":"2024-04-14T15:26:06.061422+0000","last_undegraded":"2024-04-14T15:26:06.061422+0000","last_fullsized":"2024-04-14T15:26:06.061422+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.742633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:03.685045+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.742633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:21.588419+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.10","version":"479'8","reported_seq":1579,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.062660+0000","last_change":"2024-04-14T15:24:24.848430+0000","last_active":"2024-04-14T15:26:06.062660+0000","last_peered":"2024-04-14T15:26:06.062660+0000","last_clean":"2024-04-14T15:26:06.062660+0000","last_became_active":"2024-04-14T15:20:53.001300+0000","last_became_peered":"2024-04-14T15:20:53.001300+0000","last_unstale":"2024-04-14T15:26:06.062660+0000","last_undegraded":"2024-04-14T15:26:06.062660+0000","last_fullsized":"2024-04-14T15:26:06.062660+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'8","last_scrub_stamp":"2024-04-14T15:24:24.848331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:24:24.848331+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:11.746413+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":605,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.061242+0000","last_change":"2024-04-14T15:25:50.711121+0000","last_active":"2024-04-14T15:26:06.061242+0000","last_peered":"2024-04-14T15:26:06.061242+0000","last_clean":"2024-04-14T15:26:06.061242+0000","last_became_active":"2024-04-14T15:20:52.979496+0000","last_became_peered":"2024-04-14T15:20:52.979496+0000","last_unstale":"2024-04-14T15:26:06.061242+0000","last_undegraded":"2024-04-14T15:26:06.061242+0000","last_fullsized":"2024-04-14T15:26:06.061242+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:50.711037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:08.712289+0000","last_clean_scrub_stamp":"2024-04-14T15:25:50.711037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:20.652725+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":596,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.026233+0000","last_change":"2024-04-14T15:24:30.420853+0000","last_active":"2024-04-14T15:26:06.026233+0000","last_peered":"2024-04-14T15:26:06.026233+0000","last_clean":"2024-04-14T15:26:06.026233+0000","last_became_active":"2024-04-14T15:20:52.990347+0000","last_became_peered":"2024-04-14T15:20:52.990347+0000","last_unstale":"2024-04-14T15:26:06.026233+0000","last_undegraded":"2024-04-14T15:26:06.026233+0000","last_fullsized":"2024-04-14T15:26:06.026233+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:30.420762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:24:30.420762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1544,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.068430+0000","last_change":"2024-04-14T15:24:18.506706+0000","last_active":"2024-04-14T15:26:06.068430+0000","last_peered":"2024-04-14T15:26:06.068430+0000","last_clean":"2024-04-14T15:26:06.068430+0000","last_became_active":"2024-04-14T15:09:49.061359+0000","last_became_peered":"2024-04-14T15:09:49.061359+0000","last_unstale":"2024-04-14T15:26:06.068430+0000","last_undegraded":"2024-04-14T15:26:06.068430+0000","last_fullsized":"2024-04-14T15:26:06.068430+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:18.506553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:24:18.506553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"91'130","reported_seq":1723,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.059581+0000","last_change":"2024-04-14T15:25:02.619287+0000","last_active":"2024-04-14T15:26:06.059581+0000","last_peered":"2024-04-14T15:26:06.059581+0000","last_clean":"2024-04-14T15:26:06.059581+0000","last_became_active":"2024-04-14T15:25:02.033333+0000","last_became_peered":"2024-04-14T15:25:02.033333+0000","last_unstale":"2024-04-14T15:26:06.059581+0000","last_undegraded":"2024-04-14T15:26:06.059581+0000","last_fullsized":"2024-04-14T15:26:06.059581+0000","mapping_epoch":757,"log_start":"91'130","ondisk_log_start":"91'130","created":752,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:25:02.619210+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:25:02.619210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":60,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:19.530117+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":1467,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.068661+0000","last_change":"2024-04-14T15:25:22.844265+0000","last_active":"2024-04-14T15:26:06.068661+0000","last_peered":"2024-04-14T15:26:06.068661+0000","last_clean":"2024-04-14T15:26:06.068661+0000","last_became_active":"2024-04-14T15:20:52.999623+0000","last_became_peered":"2024-04-14T15:20:52.999623+0000","last_unstale":"2024-04-14T15:26:06.068661+0000","last_undegraded":"2024-04-14T15:26:06.068661+0000","last_fullsized":"2024-04-14T15:26:06.068661+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:22.844179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:15:43.680852+0000","last_clean_scrub_stamp":"2024-04-14T15:25:22.844179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:44.664180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"479'24","reported_seq":1246,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.063106+0000","last_change":"2024-04-14T15:24:29.941597+0000","last_active":"2024-04-14T15:26:06.063106+0000","last_peered":"2024-04-14T15:26:06.063106+0000","last_clean":"2024-04-14T15:26:06.063106+0000","last_became_active":"2024-04-14T15:20:52.996917+0000","last_became_peered":"2024-04-14T15:20:52.996917+0000","last_unstale":"2024-04-14T15:26:06.063106+0000","last_undegraded":"2024-04-14T15:26:06.063106+0000","last_fullsized":"2024-04-14T15:26:06.063106+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'24","last_scrub_stamp":"2024-04-14T15:24:29.941474+0000","last_deep_scrub":"102'16","last_deep_scrub_stamp":"2024-04-14T15:12:35.999326+0000","last_clean_scrub_stamp":"2024-04-14T15:24:29.941474+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":602,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.061441+0000","last_change":"2024-04-14T15:24:30.484775+0000","last_active":"2024-04-14T15:26:06.061441+0000","last_peered":"2024-04-14T15:26:06.061441+0000","last_clean":"2024-04-14T15:26:06.061441+0000","last_became_active":"2024-04-14T15:20:52.970959+0000","last_became_peered":"2024-04-14T15:20:52.970959+0000","last_unstale":"2024-04-14T15:26:06.061441+0000","last_undegraded":"2024-04-14T15:26:06.061441+0000","last_fullsized":"2024-04-14T15:26:06.061441+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:30.484673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:30.484673+0000","last_clean_scrub_stamp":"2024-04-14T15:24:30.484673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:10.771980+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1554,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.067753+0000","last_change":"2024-04-14T15:24:17.557862+0000","last_active":"2024-04-14T15:26:06.067753+0000","last_peered":"2024-04-14T15:26:06.067753+0000","last_clean":"2024-04-14T15:26:06.067753+0000","last_became_active":"2024-04-14T15:20:52.983919+0000","last_became_peered":"2024-04-14T15:20:52.983919+0000","last_unstale":"2024-04-14T15:26:06.067753+0000","last_undegraded":"2024-04-14T15:26:06.067753+0000","last_fullsized":"2024-04-14T15:26:06.067753+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:17.557766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:24:17.557766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1237,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.063766+0000","last_change":"2024-04-14T15:24:34.995365+0000","last_active":"2024-04-14T15:26:06.063766+0000","last_peered":"2024-04-14T15:26:06.063766+0000","last_clean":"2024-04-14T15:26:06.063766+0000","last_became_active":"2024-04-14T15:20:52.993918+0000","last_became_peered":"2024-04-14T15:20:52.993918+0000","last_unstale":"2024-04-14T15:26:06.063766+0000","last_undegraded":"2024-04-14T15:26:06.063766+0000","last_fullsized":"2024-04-14T15:26:06.063766+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:34.995256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:24:34.995256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":1465,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.059974+0000","last_change":"2024-04-14T15:25:46.718417+0000","last_active":"2024-04-14T15:26:06.059974+0000","last_peered":"2024-04-14T15:26:06.059974+0000","last_clean":"2024-04-14T15:26:06.059974+0000","last_became_active":"2024-04-14T15:12:56.335230+0000","last_became_peered":"2024-04-14T15:12:56.335230+0000","last_unstale":"2024-04-14T15:26:06.059974+0000","last_undegraded":"2024-04-14T15:26:06.059974+0000","last_fullsized":"2024-04-14T15:26:06.059974+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:46.718339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:25:46.718339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:52.632654+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"776'25","reported_seq":1595,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.033884+0000","last_change":"2024-04-14T15:25:21.063734+0000","last_active":"2024-04-14T15:26:06.033884+0000","last_peered":"2024-04-14T15:26:06.033884+0000","last_clean":"2024-04-14T15:26:06.033884+0000","last_became_active":"2024-04-14T15:09:45.181456+0000","last_became_peered":"2024-04-14T15:09:45.181456+0000","last_unstale":"2024-04-14T15:26:06.033884+0000","last_undegraded":"2024-04-14T15:26:06.033884+0000","last_fullsized":"2024-04-14T15:26:06.033884+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"776'25","last_scrub_stamp":"2024-04-14T15:25:21.063579+0000","last_deep_scrub":"776'25","last_deep_scrub_stamp":"2024-04-14T15:25:21.063579+0000","last_clean_scrub_stamp":"2024-04-14T15:25:21.063579+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:28.000096+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":1114,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.026365+0000","last_change":"2024-04-14T15:24:31.421036+0000","last_active":"2024-04-14T15:26:06.026365+0000","last_peered":"2024-04-14T15:26:06.026365+0000","last_clean":"2024-04-14T15:26:06.026365+0000","last_became_active":"2024-04-14T15:15:44.878336+0000","last_became_peered":"2024-04-14T15:15:44.878336+0000","last_unstale":"2024-04-14T15:26:06.026365+0000","last_undegraded":"2024-04-14T15:26:06.026365+0000","last_fullsized":"2024-04-14T15:26:06.026365+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:31.420910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:24:31.420910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"528'9","reported_seq":692,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.027904+0000","last_change":"2024-04-14T15:25:05.354926+0000","last_active":"2024-04-14T15:26:06.027904+0000","last_peered":"2024-04-14T15:26:06.027904+0000","last_clean":"2024-04-14T15:26:06.027904+0000","last_became_active":"2024-04-14T15:20:52.982112+0000","last_became_peered":"2024-04-14T15:20:52.982112+0000","last_unstale":"2024-04-14T15:26:06.027904+0000","last_undegraded":"2024-04-14T15:26:06.027904+0000","last_fullsized":"2024-04-14T15:26:06.027904+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"528'9","last_scrub_stamp":"2024-04-14T15:25:05.354849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:25:05.354849+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:22.493402+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"72'1","reported_seq":1552,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.068584+0000","last_change":"2024-04-14T15:23:32.511432+0000","last_active":"2024-04-14T15:26:06.068584+0000","last_peered":"2024-04-14T15:26:06.068584+0000","last_clean":"2024-04-14T15:26:06.068584+0000","last_became_active":"2024-04-14T15:09:49.062051+0000","last_became_peered":"2024-04-14T15:09:49.062051+0000","last_unstale":"2024-04-14T15:26:06.068584+0000","last_undegraded":"2024-04-14T15:26:06.068584+0000","last_fullsized":"2024-04-14T15:26:06.068584+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-04-14T15:23:32.511322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:23:32.511322+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":1465,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.031086+0000","last_change":"2024-04-14T15:24:29.418267+0000","last_active":"2024-04-14T15:26:06.031086+0000","last_peered":"2024-04-14T15:26:06.031086+0000","last_clean":"2024-04-14T15:26:06.031086+0000","last_became_active":"2024-04-14T15:20:52.986657+0000","last_became_peered":"2024-04-14T15:20:52.986657+0000","last_unstale":"2024-04-14T15:26:06.031086+0000","last_undegraded":"2024-04-14T15:26:06.031086+0000","last_fullsized":"2024-04-14T15:26:06.031086+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:29.418183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:29.418183+0000","last_clean_scrub_stamp":"2024-04-14T15:24:29.418183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"519'24","reported_seq":1617,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.061608+0000","last_change":"2024-04-14T15:25:55.662949+0000","last_active":"2024-04-14T15:26:06.061608+0000","last_peered":"2024-04-14T15:26:06.061608+0000","last_clean":"2024-04-14T15:26:06.061608+0000","last_became_active":"2024-04-14T15:20:52.987333+0000","last_became_peered":"2024-04-14T15:20:52.987333+0000","last_unstale":"2024-04-14T15:26:06.061608+0000","last_undegraded":"2024-04-14T15:26:06.061608+0000","last_fullsized":"2024-04-14T15:26:06.061608+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"519'24","last_scrub_stamp":"2024-04-14T15:25:55.662871+0000","last_deep_scrub":"519'24","last_deep_scrub_stamp":"2024-04-14T15:20:44.763417+0000","last_clean_scrub_stamp":"2024-04-14T15:25:55.662871+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:14.030284+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.19","version":"684'4","reported_seq":1264,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.062838+0000","last_change":"2024-04-14T15:26:04.764095+0000","last_active":"2024-04-14T15:26:06.062838+0000","last_peered":"2024-04-14T15:26:06.062838+0000","last_clean":"2024-04-14T15:26:06.062838+0000","last_became_active":"2024-04-14T15:20:52.995628+0000","last_became_peered":"2024-04-14T15:20:52.995628+0000","last_unstale":"2024-04-14T15:26:06.062838+0000","last_undegraded":"2024-04-14T15:26:06.062838+0000","last_fullsized":"2024-04-14T15:26:06.062838+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"684'4","last_scrub_stamp":"2024-04-14T15:26:04.764014+0000","last_deep_scrub":"684'4","last_deep_scrub_stamp":"2024-04-14T15:23:46.931560+0000","last_clean_scrub_stamp":"2024-04-14T15:26:04.764014+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:16.001689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":957,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1556,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.033900+0000","last_change":"2024-04-14T15:24:30.397659+0000","last_active":"2024-04-14T15:26:06.033900+0000","last_peered":"2024-04-14T15:26:06.033900+0000","last_clean":"2024-04-14T15:26:06.033900+0000","last_became_active":"2024-04-14T15:09:47.045730+0000","last_became_peered":"2024-04-14T15:09:47.045730+0000","last_unstale":"2024-04-14T15:26:06.033900+0000","last_undegraded":"2024-04-14T15:26:06.033900+0000","last_fullsized":"2024-04-14T15:26:06.033900+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:30.397585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:24:30.397585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1547,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.031941+0000","last_change":"2024-04-14T15:24:30.343193+0000","last_active":"2024-04-14T15:26:06.031941+0000","last_peered":"2024-04-14T15:26:06.031941+0000","last_clean":"2024-04-14T15:26:06.031941+0000","last_became_active":"2024-04-14T15:09:49.057545+0000","last_became_peered":"2024-04-14T15:09:49.057545+0000","last_unstale":"2024-04-14T15:26:06.031941+0000","last_undegraded":"2024-04-14T15:26:06.031941+0000","last_fullsized":"2024-04-14T15:26:06.031941+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:30.343106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:30.343106+0000","last_clean_scrub_stamp":"2024-04-14T15:24:30.343106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":1114,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.026648+0000","last_change":"2024-04-14T15:25:25.320811+0000","last_active":"2024-04-14T15:26:06.026648+0000","last_peered":"2024-04-14T15:26:06.026648+0000","last_clean":"2024-04-14T15:26:06.026648+0000","last_became_active":"2024-04-14T15:15:44.882604+0000","last_became_peered":"2024-04-14T15:15:44.882604+0000","last_unstale":"2024-04-14T15:26:06.026648+0000","last_undegraded":"2024-04-14T15:26:06.026648+0000","last_fullsized":"2024-04-14T15:26:06.026648+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:25.320719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:25:25.320719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:54.397193+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"479'27","reported_seq":1607,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.060713+0000","last_change":"2024-04-14T15:26:00.726711+0000","last_active":"2024-04-14T15:26:06.060713+0000","last_peered":"2024-04-14T15:26:06.060713+0000","last_clean":"2024-04-14T15:26:06.060713+0000","last_became_active":"2024-04-14T15:09:45.032887+0000","last_became_peered":"2024-04-14T15:09:45.032887+0000","last_unstale":"2024-04-14T15:26:06.060713+0000","last_undegraded":"2024-04-14T15:26:06.060713+0000","last_fullsized":"2024-04-14T15:26:06.060713+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'27","last_scrub_stamp":"2024-04-14T15:26:00.726554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:26:00.726554+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:22.857124+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":596,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.026681+0000","last_change":"2024-04-14T15:25:16.387612+0000","last_active":"2024-04-14T15:26:06.026681+0000","last_peered":"2024-04-14T15:26:06.026681+0000","last_clean":"2024-04-14T15:26:06.026681+0000","last_became_active":"2024-04-14T15:20:52.990634+0000","last_became_peered":"2024-04-14T15:20:52.990634+0000","last_unstale":"2024-04-14T15:26:06.026681+0000","last_undegraded":"2024-04-14T15:26:06.026681+0000","last_fullsized":"2024-04-14T15:26:06.026681+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:16.387432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:42.754721+0000","last_clean_scrub_stamp":"2024-04-14T15:25:16.387432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:27.097015+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1559,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.032152+0000","last_change":"2024-04-14T15:25:22.689804+0000","last_active":"2024-04-14T15:26:06.032152+0000","last_peered":"2024-04-14T15:26:06.032152+0000","last_clean":"2024-04-14T15:26:06.032152+0000","last_became_active":"2024-04-14T15:09:47.042733+0000","last_became_peered":"2024-04-14T15:09:47.042733+0000","last_unstale":"2024-04-14T15:26:06.032152+0000","last_undegraded":"2024-04-14T15:26:06.032152+0000","last_fullsized":"2024-04-14T15:26:06.032152+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:22.689718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:25:22.689718+0000","last_clean_scrub_stamp":"2024-04-14T15:25:22.689718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:46.132955+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1559,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.027507+0000","last_change":"2024-04-14T15:25:48.252911+0000","last_active":"2024-04-14T15:26:06.027507+0000","last_peered":"2024-04-14T15:26:06.027507+0000","last_clean":"2024-04-14T15:26:06.027507+0000","last_became_active":"2024-04-14T15:20:52.998253+0000","last_became_peered":"2024-04-14T15:20:52.998253+0000","last_unstale":"2024-04-14T15:26:06.027507+0000","last_undegraded":"2024-04-14T15:26:06.027507+0000","last_fullsized":"2024-04-14T15:26:06.027507+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:48.252835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:54.453174+0000","last_clean_scrub_stamp":"2024-04-14T15:25:48.252835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:16.009083+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":1460,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.067974+0000","last_change":"2024-04-14T15:25:24.895474+0000","last_active":"2024-04-14T15:26:06.067974+0000","last_peered":"2024-04-14T15:26:06.067974+0000","last_clean":"2024-04-14T15:26:06.067974+0000","last_became_active":"2024-04-14T15:12:56.392352+0000","last_became_peered":"2024-04-14T15:12:56.392352+0000","last_unstale":"2024-04-14T15:26:06.067974+0000","last_undegraded":"2024-04-14T15:26:06.067974+0000","last_fullsized":"2024-04-14T15:26:06.067974+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:24.895381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:25:24.895381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:30.449450+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"479'18","reported_seq":1271,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.063863+0000","last_change":"2024-04-14T15:25:07.782722+0000","last_active":"2024-04-14T15:26:06.063863+0000","last_peered":"2024-04-14T15:26:06.063863+0000","last_clean":"2024-04-14T15:26:06.063863+0000","last_became_active":"2024-04-14T15:15:21.877838+0000","last_became_peered":"2024-04-14T15:15:21.877838+0000","last_unstale":"2024-04-14T15:26:06.063863+0000","last_undegraded":"2024-04-14T15:26:06.063863+0000","last_fullsized":"2024-04-14T15:26:06.063863+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'18","last_scrub_stamp":"2024-04-14T15:25:07.782608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:25:07.782608+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:18.348588+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":1249,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.063649+0000","last_change":"2024-04-14T15:25:46.654660+0000","last_active":"2024-04-14T15:26:06.063649+0000","last_peered":"2024-04-14T15:26:06.063649+0000","last_clean":"2024-04-14T15:26:06.063649+0000","last_became_active":"2024-04-14T15:20:53.001534+0000","last_became_peered":"2024-04-14T15:20:53.001534+0000","last_unstale":"2024-04-14T15:26:06.063649+0000","last_undegraded":"2024-04-14T15:26:06.063649+0000","last_fullsized":"2024-04-14T15:26:06.063649+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:46.654578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:25:46.654578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:10.485147+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1555,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.027102+0000","last_change":"2024-04-14T15:25:14.382875+0000","last_active":"2024-04-14T15:26:06.027102+0000","last_peered":"2024-04-14T15:26:06.027102+0000","last_clean":"2024-04-14T15:26:06.027102+0000","last_became_active":"2024-04-14T15:09:47.048045+0000","last_became_peered":"2024-04-14T15:09:47.048045+0000","last_unstale":"2024-04-14T15:26:06.027102+0000","last_undegraded":"2024-04-14T15:26:06.027102+0000","last_fullsized":"2024-04-14T15:26:06.027102+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:14.382783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:25:14.382783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:25.959641+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1239,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.064473+0000","last_change":"2024-04-14T15:25:09.737733+0000","last_active":"2024-04-14T15:26:06.064473+0000","last_peered":"2024-04-14T15:26:06.064473+0000","last_clean":"2024-04-14T15:26:06.064473+0000","last_became_active":"2024-04-14T15:20:52.998775+0000","last_became_peered":"2024-04-14T15:20:52.998775+0000","last_unstale":"2024-04-14T15:26:06.064473+0000","last_undegraded":"2024-04-14T15:26:06.064473+0000","last_fullsized":"2024-04-14T15:26:06.064473+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:09.737658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:22:20.946524+0000","last_clean_scrub_stamp":"2024-04-14T15:25:09.737658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:24.666319+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":1465,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.061067+0000","last_change":"2024-04-14T15:25:19.668735+0000","last_active":"2024-04-14T15:26:06.061067+0000","last_peered":"2024-04-14T15:26:06.061067+0000","last_clean":"2024-04-14T15:26:06.061067+0000","last_became_active":"2024-04-14T15:12:56.375278+0000","last_became_peered":"2024-04-14T15:12:56.375278+0000","last_unstale":"2024-04-14T15:26:06.061067+0000","last_undegraded":"2024-04-14T15:26:06.061067+0000","last_fullsized":"2024-04-14T15:26:06.061067+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:19.668662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:25:19.668662+0000","last_clean_scrub_stamp":"2024-04-14T15:25:19.668662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:28.031137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"479'19","reported_seq":1588,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.026594+0000","last_change":"2024-04-14T15:25:21.286947+0000","last_active":"2024-04-14T15:26:06.026594+0000","last_peered":"2024-04-14T15:26:06.026594+0000","last_clean":"2024-04-14T15:26:06.026594+0000","last_became_active":"2024-04-14T15:09:45.030093+0000","last_became_peered":"2024-04-14T15:09:45.030093+0000","last_unstale":"2024-04-14T15:26:06.026594+0000","last_undegraded":"2024-04-14T15:26:06.026594+0000","last_fullsized":"2024-04-14T15:26:06.026594+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'19","last_scrub_stamp":"2024-04-14T15:25:21.286844+0000","last_deep_scrub":"276'13","last_deep_scrub_stamp":"2024-04-14T15:15:43.969775+0000","last_clean_scrub_stamp":"2024-04-14T15:25:21.286844+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:31.501208+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":1261,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.068182+0000","last_change":"2024-04-14T15:24:26.465423+0000","last_active":"2024-04-14T15:26:06.068182+0000","last_peered":"2024-04-14T15:26:06.068182+0000","last_clean":"2024-04-14T15:26:06.068182+0000","last_became_active":"2024-04-14T15:14:30.857727+0000","last_became_peered":"2024-04-14T15:14:30.857727+0000","last_unstale":"2024-04-14T15:26:06.068182+0000","last_undegraded":"2024-04-14T15:26:06.068182+0000","last_fullsized":"2024-04-14T15:26:06.068182+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:26.465343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:00.650062+0000","last_clean_scrub_stamp":"2024-04-14T15:24:26.465343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1121,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.031519+0000","last_change":"2024-04-14T15:24:35.503356+0000","last_active":"2024-04-14T15:26:06.031519+0000","last_peered":"2024-04-14T15:26:06.031519+0000","last_clean":"2024-04-14T15:26:06.031519+0000","last_became_active":"2024-04-14T15:15:44.883121+0000","last_became_peered":"2024-04-14T15:15:44.883121+0000","last_unstale":"2024-04-14T15:26:06.031519+0000","last_undegraded":"2024-04-14T15:26:06.031519+0000","last_fullsized":"2024-04-14T15:26:06.031519+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:35.503279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:24:35.503279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1558,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.031858+0000","last_change":"2024-04-14T15:25:22.008490+0000","last_active":"2024-04-14T15:26:06.031858+0000","last_peered":"2024-04-14T15:26:06.031858+0000","last_clean":"2024-04-14T15:26:06.031858+0000","last_became_active":"2024-04-14T15:09:49.061756+0000","last_became_peered":"2024-04-14T15:09:49.061756+0000","last_unstale":"2024-04-14T15:26:06.031858+0000","last_undegraded":"2024-04-14T15:26:06.031858+0000","last_fullsized":"2024-04-14T15:26:06.031858+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:22.008411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:25:22.008411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:43.085533+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":596,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.026994+0000","last_change":"2024-04-14T15:24:24.522250+0000","last_active":"2024-04-14T15:26:06.026994+0000","last_peered":"2024-04-14T15:26:06.026994+0000","last_clean":"2024-04-14T15:26:06.026994+0000","last_became_active":"2024-04-14T15:20:52.990838+0000","last_became_peered":"2024-04-14T15:20:52.990838+0000","last_unstale":"2024-04-14T15:26:06.026994+0000","last_undegraded":"2024-04-14T15:26:06.026994+0000","last_fullsized":"2024-04-14T15:26:06.026994+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:24.522164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:24:24.522164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"479'9","reported_seq":1574,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.026879+0000","last_change":"2024-04-14T15:25:53.355715+0000","last_active":"2024-04-14T15:26:06.026879+0000","last_peered":"2024-04-14T15:26:06.026879+0000","last_clean":"2024-04-14T15:26:06.026879+0000","last_became_active":"2024-04-14T15:09:45.028667+0000","last_became_peered":"2024-04-14T15:09:45.028667+0000","last_unstale":"2024-04-14T15:26:06.026879+0000","last_undegraded":"2024-04-14T15:26:06.026879+0000","last_fullsized":"2024-04-14T15:26:06.026879+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'9","last_scrub_stamp":"2024-04-14T15:25:53.355630+0000","last_deep_scrub":"87'7","last_deep_scrub_stamp":"2024-04-14T15:11:10.510785+0000","last_clean_scrub_stamp":"2024-04-14T15:25:53.355630+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:55.830404+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":1043,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.060793+0000","last_change":"2024-04-14T15:25:07.585542+0000","last_active":"2024-04-14T15:26:06.060793+0000","last_peered":"2024-04-14T15:26:06.060793+0000","last_clean":"2024-04-14T15:26:06.060793+0000","last_became_active":"2024-04-14T15:20:52.989307+0000","last_became_peered":"2024-04-14T15:20:52.989307+0000","last_unstale":"2024-04-14T15:26:06.060793+0000","last_undegraded":"2024-04-14T15:26:06.060793+0000","last_fullsized":"2024-04-14T15:26:06.060793+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:07.585464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:22:15.944005+0000","last_clean_scrub_stamp":"2024-04-14T15:25:07.585464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:35.619340+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1555,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.068266+0000","last_change":"2024-04-14T15:25:52.924634+0000","last_active":"2024-04-14T15:26:06.068266+0000","last_peered":"2024-04-14T15:26:06.068266+0000","last_clean":"2024-04-14T15:26:06.068266+0000","last_became_active":"2024-04-14T15:09:47.046517+0000","last_became_peered":"2024-04-14T15:09:47.046517+0000","last_unstale":"2024-04-14T15:26:06.068266+0000","last_undegraded":"2024-04-14T15:26:06.068266+0000","last_fullsized":"2024-04-14T15:26:06.068266+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:52.924559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:25:52.924559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:02.028028+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1119,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.029747+0000","last_change":"2024-04-14T15:25:28.774275+0000","last_active":"2024-04-14T15:26:06.029747+0000","last_peered":"2024-04-14T15:26:06.029747+0000","last_clean":"2024-04-14T15:26:06.029747+0000","last_became_active":"2024-04-14T15:15:44.887227+0000","last_became_peered":"2024-04-14T15:15:44.887227+0000","last_unstale":"2024-04-14T15:26:06.029747+0000","last_undegraded":"2024-04-14T15:26:06.029747+0000","last_fullsized":"2024-04-14T15:26:06.029747+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:28.774199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:56.268086+0000","last_clean_scrub_stamp":"2024-04-14T15:25:28.774199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:30.928823+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":1466,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.068760+0000","last_change":"2024-04-14T15:25:09.858241+0000","last_active":"2024-04-14T15:26:06.068760+0000","last_peered":"2024-04-14T15:26:06.068760+0000","last_clean":"2024-04-14T15:26:06.068760+0000","last_became_active":"2024-04-14T15:20:52.984387+0000","last_became_peered":"2024-04-14T15:20:52.984387+0000","last_unstale":"2024-04-14T15:26:06.068760+0000","last_undegraded":"2024-04-14T15:26:06.068760+0000","last_fullsized":"2024-04-14T15:26:06.068760+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:09.858143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:34.558312+0000","last_clean_scrub_stamp":"2024-04-14T15:25:09.858143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:09.861223+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.18","version":"806'19","reported_seq":1604,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.026236+0000","last_change":"2024-04-14T15:25:56.405925+0000","last_active":"2024-04-14T15:26:06.026236+0000","last_peered":"2024-04-14T15:26:06.026236+0000","last_clean":"2024-04-14T15:26:06.026236+0000","last_became_active":"2024-04-14T15:20:52.985210+0000","last_became_peered":"2024-04-14T15:20:52.985210+0000","last_unstale":"2024-04-14T15:26:06.026236+0000","last_undegraded":"2024-04-14T15:26:06.026236+0000","last_fullsized":"2024-04-14T15:26:06.026236+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"806'19","last_scrub_stamp":"2024-04-14T15:25:56.405833+0000","last_deep_scrub":"806'19","last_deep_scrub_stamp":"2024-04-14T15:25:56.405833+0000","last_clean_scrub_stamp":"2024-04-14T15:25:56.405833+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:23.474418+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":1237,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.064295+0000","last_change":"2024-04-14T15:24:21.791334+0000","last_active":"2024-04-14T15:26:06.064295+0000","last_peered":"2024-04-14T15:26:06.064295+0000","last_clean":"2024-04-14T15:26:06.064295+0000","last_became_active":"2024-04-14T15:15:21.891926+0000","last_became_peered":"2024-04-14T15:15:21.891926+0000","last_unstale":"2024-04-14T15:26:06.064295+0000","last_undegraded":"2024-04-14T15:26:06.064295+0000","last_fullsized":"2024-04-14T15:26:06.064295+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:21.791233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:24:21.791233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1551,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.029627+0000","last_change":"2024-04-14T15:24:22.302587+0000","last_active":"2024-04-14T15:26:06.029627+0000","last_peered":"2024-04-14T15:26:06.029627+0000","last_clean":"2024-04-14T15:26:06.029627+0000","last_became_active":"2024-04-14T15:09:47.041486+0000","last_became_peered":"2024-04-14T15:09:47.041486+0000","last_unstale":"2024-04-14T15:26:06.029627+0000","last_undegraded":"2024-04-14T15:26:06.029627+0000","last_fullsized":"2024-04-14T15:26:06.029627+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:22.302513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:24:22.302513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1543,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.026576+0000","last_change":"2024-04-14T15:25:08.411730+0000","last_active":"2024-04-14T15:26:06.026576+0000","last_peered":"2024-04-14T15:26:06.026576+0000","last_clean":"2024-04-14T15:26:06.026576+0000","last_became_active":"2024-04-14T15:09:49.058904+0000","last_became_peered":"2024-04-14T15:09:49.058904+0000","last_unstale":"2024-04-14T15:26:06.026576+0000","last_undegraded":"2024-04-14T15:26:06.026576+0000","last_fullsized":"2024-04-14T15:26:06.026576+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:08.411650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:25:08.411650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:11.025399+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":1114,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.068554+0000","last_change":"2024-04-14T15:25:16.796730+0000","last_active":"2024-04-14T15:26:06.068554+0000","last_peered":"2024-04-14T15:26:06.068554+0000","last_clean":"2024-04-14T15:26:06.068554+0000","last_became_active":"2024-04-14T15:15:44.889605+0000","last_became_peered":"2024-04-14T15:15:44.889605+0000","last_unstale":"2024-04-14T15:26:06.068554+0000","last_undegraded":"2024-04-14T15:26:06.068554+0000","last_fullsized":"2024-04-14T15:26:06.068554+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:16.796595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:25:16.796595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:25.693275+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"479'34","reported_seq":1289,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.063115+0000","last_change":"2024-04-14T15:25:10.762245+0000","last_active":"2024-04-14T15:26:06.063115+0000","last_peered":"2024-04-14T15:26:06.063115+0000","last_clean":"2024-04-14T15:26:06.063115+0000","last_became_active":"2024-04-14T15:15:21.884310+0000","last_became_peered":"2024-04-14T15:15:21.884310+0000","last_unstale":"2024-04-14T15:26:06.063115+0000","last_undegraded":"2024-04-14T15:26:06.063115+0000","last_fullsized":"2024-04-14T15:26:06.063115+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'34","last_scrub_stamp":"2024-04-14T15:25:10.762092+0000","last_deep_scrub":"306'22","last_deep_scrub_stamp":"2024-04-14T15:16:13.898106+0000","last_clean_scrub_stamp":"2024-04-14T15:25:10.762092+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:27.589475+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":1563,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.033823+0000","last_change":"2024-04-14T15:24:24.353115+0000","last_active":"2024-04-14T15:26:06.033823+0000","last_peered":"2024-04-14T15:26:06.033823+0000","last_clean":"2024-04-14T15:26:06.033823+0000","last_became_active":"2024-04-14T15:09:43.017351+0000","last_became_peered":"2024-04-14T15:09:43.017351+0000","last_unstale":"2024-04-14T15:26:06.033823+0000","last_undegraded":"2024-04-14T15:26:06.033823+0000","last_fullsized":"2024-04-14T15:26:06.033823+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:24.352978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:24:24.352978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1553,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.026450+0000","last_change":"2024-04-14T15:24:32.403277+0000","last_active":"2024-04-14T15:26:06.026450+0000","last_peered":"2024-04-14T15:26:06.026450+0000","last_clean":"2024-04-14T15:26:06.026450+0000","last_became_active":"2024-04-14T15:20:52.974360+0000","last_became_peered":"2024-04-14T15:20:52.974360+0000","last_unstale":"2024-04-14T15:26:06.026450+0000","last_undegraded":"2024-04-14T15:26:06.026450+0000","last_fullsized":"2024-04-14T15:26:06.026450+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:32.403077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:32.403077+0000","last_clean_scrub_stamp":"2024-04-14T15:24:32.403077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1552,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-04-14T15:26:06.063402+0000","last_change":"2024-04-14T15:24:27.874590+0000","last_active":"2024-04-14T15:26:06.063402+0000","last_peered":"2024-04-14T15:26:06.063402+0000","last_clean":"2024-04-14T15:26:06.063402+0000","last_became_active":"2024-04-14T15:20:52.998938+0000","last_became_peered":"2024-04-14T15:20:52.998938+0000","last_unstale":"2024-04-14T15:26:06.063402+0000","last_undegraded":"2024-04-14T15:26:06.063402+0000","last_fullsized":"2024-04-14T15:26:06.063402+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:27.874498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:27.874498+0000","last_clean_scrub_stamp":"2024-04-14T15:24:27.874498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":340,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":7},{"poolid":323,"num_pg":8,"stat_sum":{"num_bytes":4000257,"num_objects":38,"num_object_clones":1,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":20,"num_read_kb":7,"num_write":58,"num_write_kb":3960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6451200,"data_stored":6451200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":24,"acting":24,"num_store_stats":7},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":5,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1003,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2109440,"data_stored":2098784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":63,"acting":63,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":977,"num_read_kb":972,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1121,"num_read_kb":878,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":1694,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":810,"ondisk_log_size":810,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103371,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":100324,"kb_used_data":6672,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":93644828,"statfs":{"total":95995035648,"available":95892303872,"internally_reserved":0,"allocated":6832128,"data_stored":3432422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":95877578},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":210453397710,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575080,"num_pgs":103,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":124516,"kb_used_data":8660,"kb_used_omap":1,"kb_used_meta":115838,"kb_avail":93620636,"statfs":{"total":95995035648,"available":95867531264,"internally_reserved":0,"allocated":8867840,"data_stored":5439766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":118618565},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738605,"num_pgs":88,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":116468,"kb_used_data":8936,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93628684,"statfs":{"total":95995035648,"available":95875772416,"internally_reserved":0,"allocated":9150464,"data_stored":5723507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":110098887},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411556,"num_pgs":92,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":114472,"kb_used_data":6940,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93630680,"statfs":{"total":95995035648,"available":95877816320,"internally_reserved":0,"allocated":7106560,"data_stored":3699903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182622,"num_pgs":101,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":92636,"kb_used_data":7308,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":93652516,"statfs":{"total":95995035648,"available":95900176384,"internally_reserved":0,"allocated":7483392,"data_stored":4040070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":87357895},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986392,"num_pgs":97,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":108176,"kb_used_data":8956,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93636976,"statfs":{"total":95995035648,"available":95884263424,"internally_reserved":0,"allocated":9170944,"data_stored":5754879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659349,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":106128,"kb_used_data":6916,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93639024,"statfs":{"total":95995035648,"available":95886360576,"internally_reserved":0,"allocated":7081984,"data_stored":3674310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2031616,"data_stored":2031616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2076672,"data_stored":2076672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":73728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":86016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2064384,"data_stored":2064384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":49152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":69632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":340,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":340,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":340,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":340,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":340,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":340,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":340,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-14T15:26:10.942 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-04-14T15:26:10.943 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-14T15:26:10.943 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-04-14T15:26:10.943 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-14T15:26:10.943 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-14T15:26:10.943 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-04-14T15:26:10.949 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:26:10.972 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:10.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:10.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:26:11.074 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:26:11.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:11.097 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:11.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:26:11.185 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:26:11.198 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:26:11.223 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:11.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:11.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:26:11.325 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:26:11.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:11.349 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:11.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:26:11.450 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:26:11.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:11.476 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:11.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:26:11.577 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:26:11.602 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:11.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:11.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:26:11.627 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:26:11.704 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:26:11.728 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:11.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:11.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:26:11.829 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:26:11.853 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:11.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:11.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:26:11.955 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:26:11.980 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:11.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:11.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:26:12.082 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:26:12.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:12.108 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:12.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:26:12.209 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:26:12.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:12.234 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:12.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:26:12.335 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:26:12.349 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:12 smithi092 ceph-mon[33966]: pgmap v1214: 224 pgs: 224 active+clean; 4.4 MiB data, 745 MiB used, 625 GiB / 626 GiB avail; 570 B/s rd, 0 op/s 2024-04-14T15:26:12.359 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:12.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:12.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:26:12.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:12 smithi090 ceph-mon[41670]: pgmap v1214: 224 pgs: 224 active+clean; 4.4 MiB data, 745 MiB used, 625 GiB / 626 GiB avail; 570 B/s rd, 0 op/s 2024-04-14T15:26:12.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:12 smithi090 ceph-mon[30399]: pgmap v1214: 224 pgs: 224 active+clean; 4.4 MiB data, 745 MiB used, 625 GiB / 626 GiB avail; 570 B/s rd, 0 op/s 2024-04-14T15:26:12.461 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:26:12.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:12.486 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:12.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:26:12.587 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:26:12.611 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:12.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:12.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:26:12.645 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:26:12.714 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:26:12.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:12.739 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:12.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:26:12.840 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:26:12.865 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:12.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:12.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:26:12.888 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-04-14T15:26:12.888 INFO:teuthology.orchestra.run.smithi092.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-14T15:26:12.889 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-04-14T15:26:12.967 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:26:12.991 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:12.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:12.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:26:13.093 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:26:13.118 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:13.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:13.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:26:13.220 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:26:13.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:13.245 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:13.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:26:13.346 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:26:13.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:13.372 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:13.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:26:13.473 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:26:13.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:13.500 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:13.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:26:13.601 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:26:13.621 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-14T15:26:13.625 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:13.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:13.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:26:13.727 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:26:13.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:13.752 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:13.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:26:13.853 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:26:13.876 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:13.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:13.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:26:13.978 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:26:14.002 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:14.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:14.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:26:14.104 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:26:14.127 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:14.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:14.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:26:14.191 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:26:13 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:26:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:26:14.229 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:26:14.252 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:14.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:14.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:26:14.354 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:26:14.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:14.379 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:14.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:26:14.479 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:26:14.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:14.504 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:14.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:26:14.510 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:14 smithi090 ceph-mon[41670]: pgmap v1215: 224 pgs: 224 active+clean; 4.4 MiB data, 745 MiB used, 625 GiB / 626 GiB avail; 751 B/s rd, 0 op/s 2024-04-14T15:26:14.510 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:14 smithi090 ceph-mon[30399]: pgmap v1215: 224 pgs: 224 active+clean; 4.4 MiB data, 745 MiB used, 625 GiB / 626 GiB avail; 751 B/s rd, 0 op/s 2024-04-14T15:26:14.605 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:26:14.621 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:14 smithi092 ceph-mon[33966]: pgmap v1215: 224 pgs: 224 active+clean; 4.4 MiB data, 745 MiB used, 625 GiB / 626 GiB avail; 751 B/s rd, 0 op/s 2024-04-14T15:26:14.628 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:14.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:14.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:26:14.730 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:26:14.753 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:14.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:14.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:26:14.760 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-14T15:26:14.760 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-14T15:26:14.760 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-14T15:26:14.760 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-14T15:26:14.856 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:26:14.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:14.880 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:14.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:26:14.981 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:26:15.004 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:15.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:15.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:26:15.106 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:26:15.130 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:15.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:15.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:26:15.232 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:26:15.256 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:15.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:15.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:26:15.358 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:26:15.382 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:15.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:15.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:26:15.484 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:26:15.507 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:15.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:15.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:26:15.522 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:15 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:26:15.522 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:15 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:26:15.522 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:26:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=sqlstore.transactions t=2024-04-14T15:26:15.249212563Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-04-14T15:26:15.608 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:26:15.632 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:15.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:15.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:26:15.691 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:15 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:26:15.691 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:15 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:26:15.691 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:15 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:26:15.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:15 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:26:15.733 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:26:15.761 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:15.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:15.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:26:15.804 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:26:15.863 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:26:15.872 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:26:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:26:15.887 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:15.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:15.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:26:15.989 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:26:16.015 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:16.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:16.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:26:16.117 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:26:16.141 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:16.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:16.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:26:16.243 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:26:16.267 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:16.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:16.268 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:26:16.368 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:26:16.392 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:16.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:16.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:26:16.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:16 smithi090 ceph-mon[41670]: pgmap v1216: 224 pgs: 224 active+clean; 4.4 MiB data, 745 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T15:26:16.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:16 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]': finished 2024-04-14T15:26:16.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:16 smithi090 ceph-mon[41670]: osdmap e817: 8 total, 8 up, 7 in 2024-04-14T15:26:16.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:16 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-144"}]: dispatch 2024-04-14T15:26:16.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:16 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-144"}]: dispatch 2024-04-14T15:26:16.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:16 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:26:16.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:16 smithi090 ceph-mon[41670]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:26:16.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:16 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-144"}]': finished 2024-04-14T15:26:16.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:16 smithi090 ceph-mon[41670]: osdmap e818: 8 total, 8 up, 7 in 2024-04-14T15:26:16.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:16 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:26:16.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:16 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:26:16.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:16 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-144"}]: dispatch 2024-04-14T15:26:16.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:16 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-144"}]: dispatch 2024-04-14T15:26:16.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:16 smithi090 ceph-mon[30399]: pgmap v1216: 224 pgs: 224 active+clean; 4.4 MiB data, 745 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T15:26:16.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:16 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]': finished 2024-04-14T15:26:16.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:16 smithi090 ceph-mon[30399]: osdmap e817: 8 total, 8 up, 7 in 2024-04-14T15:26:16.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:16 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-144"}]: dispatch 2024-04-14T15:26:16.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:16 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-144"}]: dispatch 2024-04-14T15:26:16.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:16 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:26:16.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:16 smithi090 ceph-mon[30399]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:26:16.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:16 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-144"}]': finished 2024-04-14T15:26:16.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:16 smithi090 ceph-mon[30399]: osdmap e818: 8 total, 8 up, 7 in 2024-04-14T15:26:16.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:16 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:26:16.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:16 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:26:16.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:16 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-144"}]: dispatch 2024-04-14T15:26:16.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:16 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-144"}]: dispatch 2024-04-14T15:26:16.494 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:26:16.517 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:16.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:16.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:26:16.619 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:26:16.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:16 smithi092 ceph-mon[33966]: pgmap v1216: 224 pgs: 224 active+clean; 4.4 MiB data, 745 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T15:26:16.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:16 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]': finished 2024-04-14T15:26:16.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:16 smithi092 ceph-mon[33966]: osdmap e817: 8 total, 8 up, 7 in 2024-04-14T15:26:16.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:16 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-144"}]: dispatch 2024-04-14T15:26:16.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:16 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-144"}]: dispatch 2024-04-14T15:26:16.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:16 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:26:16.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:16 smithi092 ceph-mon[33966]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:26:16.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:16 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-144"}]': finished 2024-04-14T15:26:16.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:16 smithi092 ceph-mon[33966]: osdmap e818: 8 total, 8 up, 7 in 2024-04-14T15:26:16.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:16 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:26:16.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:16 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:26:16.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:16 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-144"}]: dispatch 2024-04-14T15:26:16.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:16 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-144"}]: dispatch 2024-04-14T15:26:16.642 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:16.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:16.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:26:16.744 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:26:16.768 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:16.769 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-14T15:26:16.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:16.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:26:16.871 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:26:16.898 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:16.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:16.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:26:17.001 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:26:17.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:17.017+0000 7fd12ad3b700 1 -- 172.21.15.90:0/1484884298 >> v1:172.21.15.90:6789/0 conn(0x7fd12410f4d0 legacy=0x7fd124111990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:17.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:17.017+0000 7fd12ad3b700 1 -- 172.21.15.90:0/1484884298 shutdown_connections 2024-04-14T15:26:17.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:17.017+0000 7fd12ad3b700 1 -- 172.21.15.90:0/1484884298 >> 172.21.15.90:0/1484884298 conn(0x7fd1240fd290 msgr2=0x7fd1240ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:26:17.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:17.017+0000 7fd12ad3b700 1 -- 172.21.15.90:0/1484884298 shutdown_connections 2024-04-14T15:26:17.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:17.017+0000 7fd12ad3b700 1 -- 172.21.15.90:0/1484884298 wait complete. 2024-04-14T15:26:17.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:17.018+0000 7fd12ad3b700 1 librados: starting msgr at 2024-04-14T15:26:17.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:17.018+0000 7fd12ad3b700 1 librados: starting objecter 2024-04-14T15:26:17.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:17.018+0000 7fd12ad3b700 1 Processor -- start 2024-04-14T15:26:17.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:17.018+0000 7fd12ad3b700 1 -- start start 2024-04-14T15:26:17.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:17.018+0000 7fd12ad3b700 1 librados: setting wanted keys 2024-04-14T15:26:17.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:17.018+0000 7fd12ad3b700 1 librados: calling monclient init 2024-04-14T15:26:17.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:17.018+0000 7fd12ad3b700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd12410b5a0 con 0x7fd12410b930 2024-04-14T15:26:17.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:17.018+0000 7fd12ad3b700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd1241a9070 con 0x7fd124103a40 2024-04-14T15:26:17.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:17.018+0000 7fd12ad3b700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd1241aa250 con 0x7fd12410f4d0 2024-04-14T15:26:17.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:17.018+0000 7fd1292d8700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7fd12410f4d0 0x7fd1241a6940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:40662/0 (socket says 172.21.15.90:40662) 2024-04-14T15:26:17.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:17.018+0000 7fd1292d8700 1 -- 172.21.15.90:0/1410962076 learned_addr learned my addr 172.21.15.90:0/1410962076 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:26:17.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:17.018+0000 7fd123fff700 1 --1- 172.21.15.90:0/1410962076 >> v1:172.21.15.90:6789/0 conn(0x7fd12410b930 0x7fd124108b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:45384/0 (socket says 172.21.15.90:45384) 2024-04-14T15:26:17.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:17.019+0000 7fd121ffb700 10 client.?.objecter ms_handle_connect 0x7fd12410b930 2024-04-14T15:26:17.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:17.019+0000 7fd121ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:26:17.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:17.019+0000 7fd121ffb700 10 client.?.objecter ms_handle_connect 0x7fd12410f4d0 2024-04-14T15:26:17.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:17.019+0000 7fd121ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:26:17.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:17.019+0000 7fd121ffb700 10 client.?.objecter ms_handle_connect 0x7fd124103a40 2024-04-14T15:26:17.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:17.019+0000 7fd121ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:26:17.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:17.019+0000 7fd121ffb700 1 -- 172.21.15.90:0/1410962076 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1166524641 0 0) 0x7fd12410b5a0 con 0x7fd12410b930 2024-04-14T15:26:17.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:17.019+0000 7fd121ffb700 1 -- 172.21.15.90:0/1410962076 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1140035b0 con 0x7fd12410b930 2024-04-14T15:26:17.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:17.019+0000 7fd121ffb700 1 -- 172.21.15.90:0/1410962076 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1492331979 0 0) 0x7fd1241aa250 con 0x7fd12410f4d0 2024-04-14T15:26:17.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:17.019+0000 7fd121ffb700 1 -- 172.21.15.90:0/1410962076 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd12410b5a0 con 0x7fd12410f4d0 2024-04-14T15:26:17.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:17.019+0000 7fd121ffb700 1 -- 172.21.15.90:0/1410962076 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 455853586 0 0) 0x7fd1241a9070 con 0x7fd124103a40 2024-04-14T15:26:17.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:17.019+0000 7fd121ffb700 1 -- 172.21.15.90:0/1410962076 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1241aa250 con 0x7fd124103a40 2024-04-14T15:26:17.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:17.019+0000 7fd121ffb700 1 -- 172.21.15.90:0/1410962076 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 192718241 0 0) 0x7fd1140035b0 con 0x7fd12410b930 2024-04-14T15:26:17.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:17.019+0000 7fd121ffb700 1 -- 172.21.15.90:0/1410962076 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd1241a9070 con 0x7fd12410b930 2024-04-14T15:26:17.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:17.019+0000 7fd121ffb700 1 -- 172.21.15.90:0/1410962076 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fd1100018f0 con 0x7fd12410b930 2024-04-14T15:26:17.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:17.019+0000 7fd121ffb700 1 -- 172.21.15.90:0/1410962076 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1471732791 0 0) 0x7fd1241aa250 con 0x7fd124103a40 2024-04-14T15:26:17.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:17.019+0000 7fd121ffb700 1 -- 172.21.15.90:0/1410962076 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd1140035b0 con 0x7fd124103a40 2024-04-14T15:26:17.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:17.019+0000 7fd121ffb700 1 -- 172.21.15.90:0/1410962076 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fd1180017f0 con 0x7fd124103a40 2024-04-14T15:26:17.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:17.019+0000 7fd121ffb700 1 -- 172.21.15.90:0/1410962076 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2555825074 0 0) 0x7fd12410b5a0 con 0x7fd12410f4d0 2024-04-14T15:26:17.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:17.019+0000 7fd121ffb700 1 -- 172.21.15.90:0/1410962076 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd1241aa250 con 0x7fd12410f4d0 2024-04-14T15:26:17.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:17.019+0000 7fd121ffb700 1 -- 172.21.15.90:0/1410962076 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1989429238 0 0) 0x7fd1241a9070 con 0x7fd12410b930 2024-04-14T15:26:17.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:17.019+0000 7fd121ffb700 1 -- 172.21.15.90:0/1410962076 >> v1:172.21.15.90:6790/0 conn(0x7fd12410f4d0 legacy=0x7fd1241a6940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:17.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:17.019+0000 7fd121ffb700 1 -- 172.21.15.90:0/1410962076 >> v1:172.21.15.92:6789/0 conn(0x7fd124103a40 legacy=0x7fd1241083f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:17.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:17.019+0000 7fd121ffb700 1 -- 172.21.15.90:0/1410962076 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1241ab430 con 0x7fd12410b930 2024-04-14T15:26:17.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:17.019+0000 7fd12ad3b700 1 -- 172.21.15.90:0/1410962076 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd1241aa480 con 0x7fd12410b930 2024-04-14T15:26:17.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:17.020+0000 7fd12ad3b700 10 client.38097.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:26:17.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:17.020+0000 7fd121ffb700 1 -- 172.21.15.90:0/1410962076 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fd110001ce0 con 0x7fd12410b930 2024-04-14T15:26:17.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:17.020+0000 7fd12ad3b700 1 -- 172.21.15.90:0/1410962076 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd1241aa940 con 0x7fd12410b930 2024-04-14T15:26:17.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:17.020+0000 7fd12ad3b700 1 librados: init done 2024-04-14T15:26:17.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:17.020+0000 7fd121ffb700 1 -- 172.21.15.90:0/1410962076 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fd110005d00 con 0x7fd12410b930 2024-04-14T15:26:17.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:17.021+0000 7fd12ad3b700 1 -- 172.21.15.90:0/1410962076 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd108005120 con 0x7fd12410b930 2024-04-14T15:26:17.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:17.021+0000 7fd121ffb700 1 -- 172.21.15.90:0/1410962076 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fd110005f90 con 0x7fd12410b930 2024-04-14T15:26:17.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:17.022+0000 7fd121ffb700 1 -- 172.21.15.90:0/1410962076 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(818..818 src has 467..818) ==== 8749+0+0 (unknown 673906117 0 0) 0x7fd11008aba0 con 0x7fd12410b930 2024-04-14T15:26:17.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:17.022+0000 7fd121ffb700 10 client.38097.objecter ms_dispatch 0x7fd124001550 osd_map(818..818 src has 467..818) 2024-04-14T15:26:17.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:17.022+0000 7fd121ffb700 3 client.38097.objecter handle_osd_map got epochs [818,818] > 0 2024-04-14T15:26:17.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:17.022+0000 7fd121ffb700 3 client.38097.objecter handle_osd_map decoding full epoch 818 2024-04-14T15:26:17.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:17.022+0000 7fd121ffb700 20 client.38097.objecter dump_active .. 0 homeless 2024-04-14T15:26:17.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:17.022+0000 7fd121ffb700 10 client.38097.objecter ms_handle_connect 0x7fd114079c60 2024-04-14T15:26:17.026 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:17.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:17.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:26:17.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:17.027+0000 7fd121ffb700 1 -- 172.21.15.90:0/1410962076 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fd110057380 con 0x7fd12410b930 2024-04-14T15:26:17.128 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:26:17.155 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:17.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:17.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:26:17.257 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:26:17.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:17.258+0000 7fd12ad3b700 1 -- 172.21.15.90:0/1410962076 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7fd108005410 con 0x7fd12410b930 2024-04-14T15:26:17.281 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:17.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:17.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:26:17.382 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:26:17.407 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:17.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:17.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:26:17.508 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:26:17.536 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:17.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:17.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:26:17.637 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:26:17.661 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:17.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:17.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:26:17.718 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:26:17.763 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:26:17.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:17.788 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:17.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:26:17.889 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:26:17.916 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:17.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:17.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:26:18.018 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:26:18.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:18.041+0000 7fd121ffb700 1 -- 172.21.15.90:0/1410962076 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v820) ==== 97+0+0 (unknown 827126933 0 0) 0x7fd11005b030 con 0x7fd12410b930 2024-04-14T15:26:18.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:18.044+0000 7fd12ad3b700 10 librados: watch_flush enter 2024-04-14T15:26:18.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:18.045+0000 7fd12ad3b700 10 librados: watch_flush exit 2024-04-14T15:26:18.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:18.045+0000 7fd12ad3b700 20 client.38097.objecter shutdown clearing up homeless session... 2024-04-14T15:26:18.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:18.045+0000 7fd12ad3b700 10 client.38097.objecter successfully canceled tick 2024-04-14T15:26:18.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:18.045+0000 7fd12ad3b700 1 -- 172.21.15.90:0/1410962076 >> v1:172.21.15.90:6800/420935427 conn(0x7fd114079c60 legacy=0x7fd11407c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:18.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:18.045+0000 7fd12ad3b700 1 -- 172.21.15.90:0/1410962076 >> v1:172.21.15.90:6789/0 conn(0x7fd12410b930 legacy=0x7fd124108b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:18.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:18.045+0000 7fd12ad3b700 1 -- 172.21.15.90:0/1410962076 shutdown_connections 2024-04-14T15:26:18.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:18.045+0000 7fd12ad3b700 1 -- 172.21.15.90:0/1410962076 >> 172.21.15.90:0/1410962076 conn(0x7fd1240fd290 msgr2=0x7fd124111950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:26:18.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:18.047+0000 7fd12ad3b700 1 -- 172.21.15.90:0/1410962076 shutdown_connections 2024-04-14T15:26:18.048 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:18.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:18.048+0000 7fd12ad3b700 1 -- 172.21.15.90:0/1410962076 wait complete. 2024-04-14T15:26:18.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:18.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:26:18.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:18.050+0000 7fd12ad3b700 1 librados: shutdown 2024-04-14T15:26:18.055 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is set 2024-04-14T15:26:18.128 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:18 smithi090 ceph-mon[41670]: osdmap e819: 8 total, 8 up, 7 in 2024-04-14T15:26:18.129 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:18 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1410962076' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:26:18.129 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:18 smithi090 ceph-mon[30399]: osdmap e819: 8 total, 8 up, 7 in 2024-04-14T15:26:18.129 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:18 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1410962076' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:26:18.150 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:26:18.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:18.175 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:18.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:26:18.276 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:26:18.304 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:18.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:18.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:26:18.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:18 smithi092 ceph-mon[33966]: osdmap e819: 8 total, 8 up, 7 in 2024-04-14T15:26:18.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:18 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1410962076' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:26:18.406 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:26:18.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:18.434 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:18.434 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:26:18.518 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-14T15:26:18.518 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-14T15:26:18.518 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-04-14T15:26:18.518 INFO:tasks.thrashosds.thrasher:Setting osd 5 primary_affinity to 1.000000 2024-04-14T15:26:18.519 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd primary-affinity 5 1 2024-04-14T15:26:18.535 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:26:18.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:18.587 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:18.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:26:18.688 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:26:18.712 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:18.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:18.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:26:18.813 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:26:18.841 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:18.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:18.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:26:18.943 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:26:18.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:18.973 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:18.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:26:19.074 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:26:19.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:19.103 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:19.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:26:19.189 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:19 smithi090 ceph-mon[41670]: pgmap v1220: 192 pgs: 192 active+clean; 4.4 MiB data, 745 MiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:26:19.189 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:19 smithi090 ceph-mon[41670]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:26:19.190 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:19 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1410962076' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-14T15:26:19.190 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:19 smithi090 ceph-mon[41670]: osdmap e820: 8 total, 8 up, 7 in 2024-04-14T15:26:19.190 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:19 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:26:19.190 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:19 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:26:19.190 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:19 smithi090 ceph-mon[30399]: pgmap v1220: 192 pgs: 192 active+clean; 4.4 MiB data, 745 MiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:26:19.190 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:19 smithi090 ceph-mon[30399]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:26:19.190 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:19 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1410962076' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-14T15:26:19.190 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:19 smithi090 ceph-mon[30399]: osdmap e820: 8 total, 8 up, 7 in 2024-04-14T15:26:19.190 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:19 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:26:19.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:19 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:26:19.203 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:26:19.228 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:19.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:19.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:26:19.329 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:26:19.354 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:19.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:19.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:26:19.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:19 smithi092 ceph-mon[33966]: pgmap v1220: 192 pgs: 192 active+clean; 4.4 MiB data, 745 MiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:26:19.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:19 smithi092 ceph-mon[33966]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:26:19.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:19 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1410962076' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-14T15:26:19.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:19 smithi092 ceph-mon[33966]: osdmap e820: 8 total, 8 up, 7 in 2024-04-14T15:26:19.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:19 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:26:19.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:19 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:26:19.456 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:26:19.480 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:19.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:19.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:26:19.583 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:26:19.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:19.610 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:19.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:26:19.711 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:26:19.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:19.736 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:19.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:26:19.837 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:26:19.862 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:19.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:19.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:26:19.967 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:26:19.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:19.994 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:19.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:26:20.095 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:26:20.118 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:20.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:20.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:26:20.220 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:26:20.247 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:20.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:20.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:26:20.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.265+0000 7f09ae067700 1 -- 172.21.15.90:0/3919092219 >> v1:172.21.15.90:6789/0 conn(0x7f09a8074ee0 legacy=0x7f09a80752c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:20.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.265+0000 7f09ae067700 1 -- 172.21.15.90:0/3919092219 shutdown_connections 2024-04-14T15:26:20.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.265+0000 7f09ae067700 1 -- 172.21.15.90:0/3919092219 >> 172.21.15.90:0/3919092219 conn(0x7f09a80fe3d0 msgr2=0x7f09a81007f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:26:20.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.265+0000 7f09ae067700 1 -- 172.21.15.90:0/3919092219 shutdown_connections 2024-04-14T15:26:20.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.265+0000 7f09ae067700 1 -- 172.21.15.90:0/3919092219 wait complete. 2024-04-14T15:26:20.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.265+0000 7f09ae067700 1 librados: starting msgr at 2024-04-14T15:26:20.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.265+0000 7f09ae067700 1 librados: starting objecter 2024-04-14T15:26:20.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.265+0000 7f09ae067700 1 Processor -- start 2024-04-14T15:26:20.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.266+0000 7f09ae067700 1 -- start start 2024-04-14T15:26:20.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.266+0000 7f09ae067700 1 librados: setting wanted keys 2024-04-14T15:26:20.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.266+0000 7f09ae067700 1 librados: calling monclient init 2024-04-14T15:26:20.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.266+0000 7f09ae067700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f09a810d7c0 con 0x7f09a80759d0 2024-04-14T15:26:20.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.266+0000 7f09ae067700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f09a810d9d0 con 0x7f09a8074ee0 2024-04-14T15:26:20.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.266+0000 7f09ae067700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f09a81ac180 con 0x7f09a8111580 2024-04-14T15:26:20.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.266+0000 7f09a7fff700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f09a8111580 0x7f09a81a8a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:40674/0 (socket says 172.21.15.90:40674) 2024-04-14T15:26:20.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.266+0000 7f09a7fff700 1 -- 172.21.15.90:0/3923713408 learned_addr learned my addr 172.21.15.90:0/3923713408 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:26:20.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.266+0000 7f09a4ff9700 10 client.?.objecter ms_handle_connect 0x7f09a80759d0 2024-04-14T15:26:20.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.266+0000 7f09a4ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:26:20.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.266+0000 7f09a4ff9700 10 client.?.objecter ms_handle_connect 0x7f09a8111580 2024-04-14T15:26:20.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.266+0000 7f09a4ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:26:20.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.266+0000 7f09a4ff9700 10 client.?.objecter ms_handle_connect 0x7f09a8074ee0 2024-04-14T15:26:20.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.266+0000 7f09a4ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:26:20.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.266+0000 7f09a4ff9700 1 -- 172.21.15.90:0/3923713408 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1254575720 0 0) 0x7f09a810d7c0 con 0x7f09a80759d0 2024-04-14T15:26:20.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.266+0000 7f09a4ff9700 1 -- 172.21.15.90:0/3923713408 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f09940035b0 con 0x7f09a80759d0 2024-04-14T15:26:20.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.266+0000 7f09a4ff9700 1 -- 172.21.15.90:0/3923713408 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 408263145 0 0) 0x7f09a810d9d0 con 0x7f09a8074ee0 2024-04-14T15:26:20.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.266+0000 7f09a4ff9700 1 -- 172.21.15.90:0/3923713408 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f09a810d7c0 con 0x7f09a8074ee0 2024-04-14T15:26:20.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.266+0000 7f09a4ff9700 1 -- 172.21.15.90:0/3923713408 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1845435079 0 0) 0x7f09a81ac180 con 0x7f09a8111580 2024-04-14T15:26:20.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.266+0000 7f09a4ff9700 1 -- 172.21.15.90:0/3923713408 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f09a810d9d0 con 0x7f09a8111580 2024-04-14T15:26:20.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.266+0000 7f09a4ff9700 1 -- 172.21.15.90:0/3923713408 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2244514078 0 0) 0x7f09a810d7c0 con 0x7f09a8074ee0 2024-04-14T15:26:20.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.266+0000 7f09a4ff9700 1 -- 172.21.15.90:0/3923713408 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f09a81ac180 con 0x7f09a8074ee0 2024-04-14T15:26:20.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.266+0000 7f09a4ff9700 1 -- 172.21.15.90:0/3923713408 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f09a0001950 con 0x7f09a8074ee0 2024-04-14T15:26:20.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.267+0000 7f09a4ff9700 1 -- 172.21.15.90:0/3923713408 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3415098550 0 0) 0x7f09940035b0 con 0x7f09a80759d0 2024-04-14T15:26:20.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.267+0000 7f09a4ff9700 1 -- 172.21.15.90:0/3923713408 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f09a810d7c0 con 0x7f09a80759d0 2024-04-14T15:26:20.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.267+0000 7f09a4ff9700 1 -- 172.21.15.90:0/3923713408 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f0990001830 con 0x7f09a80759d0 2024-04-14T15:26:20.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.267+0000 7f09a4ff9700 1 -- 172.21.15.90:0/3923713408 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 632149971 0 0) 0x7f09a810d9d0 con 0x7f09a8111580 2024-04-14T15:26:20.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.267+0000 7f09a4ff9700 1 -- 172.21.15.90:0/3923713408 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f09940035b0 con 0x7f09a8111580 2024-04-14T15:26:20.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.267+0000 7f09a4ff9700 1 -- 172.21.15.90:0/3923713408 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f0998001870 con 0x7f09a8111580 2024-04-14T15:26:20.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.267+0000 7f09a4ff9700 1 -- 172.21.15.90:0/3923713408 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4261518230 0 0) 0x7f09a81ac180 con 0x7f09a8074ee0 2024-04-14T15:26:20.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.267+0000 7f09a4ff9700 1 -- 172.21.15.90:0/3923713408 >> v1:172.21.15.90:6790/0 conn(0x7f09a8111580 legacy=0x7f09a81a8a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:20.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.267+0000 7f09a4ff9700 1 -- 172.21.15.90:0/3923713408 >> v1:172.21.15.90:6789/0 conn(0x7f09a80759d0 legacy=0x7f09a810ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:20.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.267+0000 7f09a4ff9700 1 -- 172.21.15.90:0/3923713408 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09a81ad3c0 con 0x7f09a8074ee0 2024-04-14T15:26:20.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.267+0000 7f09ae067700 1 -- 172.21.15.90:0/3923713408 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f09a81ac3b0 con 0x7f09a8074ee0 2024-04-14T15:26:20.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.267+0000 7f09ae067700 10 client.49933.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:26:20.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.267+0000 7f09ae067700 1 -- 172.21.15.90:0/3923713408 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f09a81ac870 con 0x7f09a8074ee0 2024-04-14T15:26:20.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.267+0000 7f09ae067700 1 librados: init done 2024-04-14T15:26:20.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.267+0000 7f09a4ff9700 1 -- 172.21.15.90:0/3923713408 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f09a0001ef0 con 0x7f09a8074ee0 2024-04-14T15:26:20.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.267+0000 7f09a4ff9700 1 -- 172.21.15.90:0/3923713408 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f09a0005020 con 0x7f09a8074ee0 2024-04-14T15:26:20.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.268+0000 7f09ae067700 1 -- 172.21.15.90:0/3923713408 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f0988005100 con 0x7f09a8074ee0 2024-04-14T15:26:20.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.268+0000 7f09a4ff9700 1 -- 172.21.15.90:0/3923713408 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f09a0004280 con 0x7f09a8074ee0 2024-04-14T15:26:20.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.269+0000 7f09a4ff9700 1 -- 172.21.15.90:0/3923713408 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(822..822 src has 467..822) ==== 8757+0+0 (unknown 1278112231 0 0) 0x7f09a0089c40 con 0x7f09a8074ee0 2024-04-14T15:26:20.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.269+0000 7f09a4ff9700 10 client.49933.objecter ms_dispatch 0x7f09a8001550 osd_map(822..822 src has 467..822) 2024-04-14T15:26:20.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.269+0000 7f09a4ff9700 3 client.49933.objecter handle_osd_map got epochs [822,822] > 0 2024-04-14T15:26:20.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.269+0000 7f09a4ff9700 3 client.49933.objecter handle_osd_map decoding full epoch 822 2024-04-14T15:26:20.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.269+0000 7f09a4ff9700 20 client.49933.objecter dump_active .. 0 homeless 2024-04-14T15:26:20.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.269+0000 7f09a4ff9700 1 -- 172.21.15.90:0/3923713408 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v822) ==== 54+0+16 (unknown 4025173286 0 2420000192) 0x7f09a008a060 con 0x7f09a8074ee0 2024-04-14T15:26:20.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.269+0000 7f09a4ff9700 10 client.49933.objecter ms_handle_connect 0x7f0994071460 2024-04-14T15:26:20.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.269+0000 7f09ae067700 10 client.49933.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:26:20.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.269+0000 7f09ae067700 15 client.49933.objecter _session_command_op_assign -1 1 2024-04-14T15:26:20.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.269+0000 7f09ae067700 20 client.49933.objecter _get_session s=0x7f09880053e0 osd=0 3 2024-04-14T15:26:20.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.269+0000 7f09ae067700 20 client.49933.objecter put_session s=0x7f09880053e0 osd=0 3 2024-04-14T15:26:20.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.269+0000 7f09ae067700 20 client.49933.objecter _get_session s=0x7f09880053e0 osd=0 3 2024-04-14T15:26:20.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.269+0000 7f09ae067700 15 client.49933.objecter _session_command_op_remove -1 1 2024-04-14T15:26:20.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.269+0000 7f09ae067700 20 client.49933.objecter get_session s=0x7f09880053e0 osd=0 3 2024-04-14T15:26:20.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.269+0000 7f09ae067700 15 client.49933.objecter _session_command_op_assign 0 1 2024-04-14T15:26:20.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.269+0000 7f09ae067700 20 client.49933.objecter put_session s=0x7f09880053e0 osd=0 4 2024-04-14T15:26:20.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.269+0000 7f09ae067700 10 client.49933.objecter _send_command 1 2024-04-14T15:26:20.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.269+0000 7f09ae067700 1 -- 172.21.15.90:0/3923713408 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0988004b50 con 0x7f0988004730 2024-04-14T15:26:20.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.270+0000 7f09a4ff9700 10 client.49933.objecter ms_handle_connect 0x7f0988004730 2024-04-14T15:26:20.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.270+0000 7f09a4ff9700 1 -- 172.21.15.90:0/3923713408 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f0988004b50 con 0x7f0988004730 2024-04-14T15:26:20.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.270+0000 7f09a4ff9700 10 client.49933.objecter ms_dispatch 0x7f09a8001550 command_reply(tid 1: 0 ) 2024-04-14T15:26:20.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.270+0000 7f09a4ff9700 10 client.49933.objecter _finish_command 1 = system:0 2024-04-14T15:26:20.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.270+0000 7f09a4ff9700 20 client.49933.objecter put_session s=0x7f09880053e0 osd=0 4 2024-04-14T15:26:20.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.270+0000 7f09a4ff9700 15 client.49933.objecter _session_command_op_remove 0 1 2024-04-14T15:26:20.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.287+0000 7f09ae067700 10 client.49933.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:26:20.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.287+0000 7f09ae067700 15 client.49933.objecter _session_command_op_assign -1 2 2024-04-14T15:26:20.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.287+0000 7f09ae067700 20 client.49933.objecter _get_session s=0x7f09880053e0 osd=0 3 2024-04-14T15:26:20.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.287+0000 7f09ae067700 20 client.49933.objecter put_session s=0x7f09880053e0 osd=0 3 2024-04-14T15:26:20.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.287+0000 7f09ae067700 20 client.49933.objecter _get_session s=0x7f09880053e0 osd=0 3 2024-04-14T15:26:20.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.287+0000 7f09ae067700 15 client.49933.objecter _session_command_op_remove -1 2 2024-04-14T15:26:20.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.287+0000 7f09ae067700 20 client.49933.objecter get_session s=0x7f09880053e0 osd=0 3 2024-04-14T15:26:20.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.287+0000 7f09ae067700 15 client.49933.objecter _session_command_op_assign 0 2 2024-04-14T15:26:20.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.287+0000 7f09ae067700 20 client.49933.objecter put_session s=0x7f09880053e0 osd=0 4 2024-04-14T15:26:20.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.287+0000 7f09ae067700 10 client.49933.objecter _send_command 2 2024-04-14T15:26:20.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.287+0000 7f09ae067700 1 -- 172.21.15.90:0/3923713408 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f09880079f0 con 0x7f0988004730 2024-04-14T15:26:20.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.289+0000 7f09a4ff9700 1 -- 172.21.15.90:0/3923713408 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f09880079f0 con 0x7f0988004730 2024-04-14T15:26:20.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.289+0000 7f09a4ff9700 10 client.49933.objecter ms_dispatch 0x7f09a8001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:26:20.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.289+0000 7f09a4ff9700 10 client.49933.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:26:20.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.289+0000 7f09a4ff9700 20 client.49933.objecter put_session s=0x7f09880053e0 osd=0 4 2024-04-14T15:26:20.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.289+0000 7f09a4ff9700 15 client.49933.objecter _session_command_op_remove 0 2 2024-04-14T15:26:20.292 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:26:20.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.289+0000 7f09ae067700 10 client.49933.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:26:20.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.289+0000 7f09ae067700 15 client.49933.objecter _session_command_op_assign -1 3 2024-04-14T15:26:20.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.289+0000 7f09ae067700 20 client.49933.objecter _get_session s=0x7f0988004110 osd=1 3 2024-04-14T15:26:20.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.289+0000 7f09ae067700 20 client.49933.objecter put_session s=0x7f0988004110 osd=1 3 2024-04-14T15:26:20.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.289+0000 7f09ae067700 20 client.49933.objecter _get_session s=0x7f0988004110 osd=1 3 2024-04-14T15:26:20.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.289+0000 7f09ae067700 15 client.49933.objecter _session_command_op_remove -1 3 2024-04-14T15:26:20.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.289+0000 7f09ae067700 20 client.49933.objecter get_session s=0x7f0988004110 osd=1 3 2024-04-14T15:26:20.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.289+0000 7f09ae067700 15 client.49933.objecter _session_command_op_assign 1 3 2024-04-14T15:26:20.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.289+0000 7f09ae067700 20 client.49933.objecter put_session s=0x7f0988004110 osd=1 4 2024-04-14T15:26:20.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.289+0000 7f09ae067700 10 client.49933.objecter _send_command 3 2024-04-14T15:26:20.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.289+0000 7f09ae067700 1 -- 172.21.15.90:0/3923713408 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f0988002180 con 0x7f0988004280 2024-04-14T15:26:20.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.289+0000 7f09a4ff9700 10 client.49933.objecter ms_handle_connect 0x7f0988004280 2024-04-14T15:26:20.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.290+0000 7f09a4ff9700 1 -- 172.21.15.90:0/3923713408 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f0988002180 con 0x7f0988004280 2024-04-14T15:26:20.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.290+0000 7f09a4ff9700 10 client.49933.objecter ms_dispatch 0x7f09a8001550 command_reply(tid 3: 0 ) 2024-04-14T15:26:20.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.290+0000 7f09a4ff9700 10 client.49933.objecter _finish_command 3 = system:0 2024-04-14T15:26:20.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.290+0000 7f09a4ff9700 20 client.49933.objecter put_session s=0x7f0988004110 osd=1 4 2024-04-14T15:26:20.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.290+0000 7f09a4ff9700 15 client.49933.objecter _session_command_op_remove 1 3 2024-04-14T15:26:20.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.307+0000 7f09ae067700 10 client.49933.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:26:20.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.307+0000 7f09ae067700 15 client.49933.objecter _session_command_op_assign -1 4 2024-04-14T15:26:20.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.307+0000 7f09ae067700 20 client.49933.objecter _get_session s=0x7f0988004110 osd=1 3 2024-04-14T15:26:20.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.307+0000 7f09ae067700 20 client.49933.objecter put_session s=0x7f0988004110 osd=1 3 2024-04-14T15:26:20.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.307+0000 7f09ae067700 20 client.49933.objecter _get_session s=0x7f0988004110 osd=1 3 2024-04-14T15:26:20.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.307+0000 7f09ae067700 15 client.49933.objecter _session_command_op_remove -1 4 2024-04-14T15:26:20.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.307+0000 7f09ae067700 20 client.49933.objecter get_session s=0x7f0988004110 osd=1 3 2024-04-14T15:26:20.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.307+0000 7f09ae067700 15 client.49933.objecter _session_command_op_assign 1 4 2024-04-14T15:26:20.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.307+0000 7f09ae067700 20 client.49933.objecter put_session s=0x7f0988004110 osd=1 4 2024-04-14T15:26:20.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.307+0000 7f09ae067700 10 client.49933.objecter _send_command 4 2024-04-14T15:26:20.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.307+0000 7f09ae067700 1 -- 172.21.15.90:0/3923713408 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f0988001600 con 0x7f0988004280 2024-04-14T15:26:20.309 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:26:20.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.308+0000 7f09a4ff9700 1 -- 172.21.15.90:0/3923713408 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f0988001600 con 0x7f0988004280 2024-04-14T15:26:20.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.308+0000 7f09a4ff9700 10 client.49933.objecter ms_dispatch 0x7f09a8001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:26:20.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.308+0000 7f09a4ff9700 10 client.49933.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:26:20.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.308+0000 7f09a4ff9700 20 client.49933.objecter put_session s=0x7f0988004110 osd=1 4 2024-04-14T15:26:20.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.308+0000 7f09a4ff9700 15 client.49933.objecter _session_command_op_remove 1 4 2024-04-14T15:26:20.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.308+0000 7f09ae067700 10 client.49933.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:26:20.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.308+0000 7f09ae067700 15 client.49933.objecter _session_command_op_assign -1 5 2024-04-14T15:26:20.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.308+0000 7f09ae067700 20 client.49933.objecter _get_session s=0x7f0988001a00 osd=2 3 2024-04-14T15:26:20.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.308+0000 7f09ae067700 20 client.49933.objecter put_session s=0x7f0988001a00 osd=2 3 2024-04-14T15:26:20.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.308+0000 7f09ae067700 20 client.49933.objecter _get_session s=0x7f0988001a00 osd=2 3 2024-04-14T15:26:20.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.308+0000 7f09ae067700 15 client.49933.objecter _session_command_op_remove -1 5 2024-04-14T15:26:20.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.308+0000 7f09ae067700 20 client.49933.objecter get_session s=0x7f0988001a00 osd=2 3 2024-04-14T15:26:20.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.308+0000 7f09ae067700 15 client.49933.objecter _session_command_op_assign 2 5 2024-04-14T15:26:20.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.308+0000 7f09ae067700 20 client.49933.objecter put_session s=0x7f0988001a00 osd=2 4 2024-04-14T15:26:20.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.308+0000 7f09ae067700 10 client.49933.objecter _send_command 5 2024-04-14T15:26:20.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.308+0000 7f09ae067700 1 -- 172.21.15.90:0/3923713408 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f0988010f20 con 0x7f0988008880 2024-04-14T15:26:20.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.309+0000 7f09a4ff9700 10 client.49933.objecter ms_handle_connect 0x7f0988008880 2024-04-14T15:26:20.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.309+0000 7f09a4ff9700 1 -- 172.21.15.90:0/3923713408 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f0988010f20 con 0x7f0988008880 2024-04-14T15:26:20.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.309+0000 7f09a4ff9700 10 client.49933.objecter ms_dispatch 0x7f09a8001550 command_reply(tid 5: 0 ) 2024-04-14T15:26:20.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.309+0000 7f09a4ff9700 10 client.49933.objecter _finish_command 5 = system:0 2024-04-14T15:26:20.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.309+0000 7f09a4ff9700 20 client.49933.objecter put_session s=0x7f0988001a00 osd=2 4 2024-04-14T15:26:20.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.309+0000 7f09a4ff9700 15 client.49933.objecter _session_command_op_remove 2 5 2024-04-14T15:26:20.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.326+0000 7f09ae067700 10 client.49933.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:26:20.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.326+0000 7f09ae067700 15 client.49933.objecter _session_command_op_assign -1 6 2024-04-14T15:26:20.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.326+0000 7f09ae067700 20 client.49933.objecter _get_session s=0x7f0988001a00 osd=2 3 2024-04-14T15:26:20.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.326+0000 7f09ae067700 20 client.49933.objecter put_session s=0x7f0988001a00 osd=2 3 2024-04-14T15:26:20.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.326+0000 7f09ae067700 20 client.49933.objecter _get_session s=0x7f0988001a00 osd=2 3 2024-04-14T15:26:20.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.326+0000 7f09ae067700 15 client.49933.objecter _session_command_op_remove -1 6 2024-04-14T15:26:20.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.326+0000 7f09ae067700 20 client.49933.objecter get_session s=0x7f0988001a00 osd=2 3 2024-04-14T15:26:20.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.326+0000 7f09ae067700 15 client.49933.objecter _session_command_op_assign 2 6 2024-04-14T15:26:20.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.326+0000 7f09ae067700 20 client.49933.objecter put_session s=0x7f0988001a00 osd=2 4 2024-04-14T15:26:20.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.326+0000 7f09ae067700 10 client.49933.objecter _send_command 6 2024-04-14T15:26:20.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.326+0000 7f09ae067700 1 -- 172.21.15.90:0/3923713408 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f0988007f10 con 0x7f0988008880 2024-04-14T15:26:20.327 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:20 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-146","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:26:20.327 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:20 smithi090 ceph-mon[30399]: osdmap e821: 8 total, 8 up, 7 in 2024-04-14T15:26:20.327 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:20 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:26:20.327 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:20 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:26:20.327 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:20 smithi090 ceph-mon[30399]: pgmap v1223: 224 pgs: 10 creating+peering, 214 active+clean; 4.4 MiB data, 753 MiB used, 625 GiB / 626 GiB avail; 1.3 KiB/s rd, 1.0 KiB/s wr, 2 op/s 2024-04-14T15:26:20.328 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:20 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-146","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:26:20.328 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:20 smithi090 ceph-mon[41670]: osdmap e821: 8 total, 8 up, 7 in 2024-04-14T15:26:20.328 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:20 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:26:20.328 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:20 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:26:20.328 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:20 smithi090 ceph-mon[41670]: pgmap v1223: 224 pgs: 10 creating+peering, 214 active+clean; 4.4 MiB data, 753 MiB used, 625 GiB / 626 GiB avail; 1.3 KiB/s rd, 1.0 KiB/s wr, 2 op/s 2024-04-14T15:26:20.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.327+0000 7f09a4ff9700 1 -- 172.21.15.90:0/3923713408 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f0988007f10 con 0x7f0988008880 2024-04-14T15:26:20.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.327+0000 7f09a4ff9700 10 client.49933.objecter ms_dispatch 0x7f09a8001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:26:20.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.327+0000 7f09a4ff9700 10 client.49933.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:26:20.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.327+0000 7f09a4ff9700 20 client.49933.objecter put_session s=0x7f0988001a00 osd=2 4 2024-04-14T15:26:20.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.327+0000 7f09a4ff9700 15 client.49933.objecter _session_command_op_remove 2 6 2024-04-14T15:26:20.329 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:26:20.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.327+0000 7f09ae067700 10 client.49933.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:26:20.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.327+0000 7f09ae067700 15 client.49933.objecter _session_command_op_assign -1 7 2024-04-14T15:26:20.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.328+0000 7f09ae067700 20 client.49933.objecter _get_session s=0x7f09880081a0 osd=3 3 2024-04-14T15:26:20.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.328+0000 7f09ae067700 20 client.49933.objecter put_session s=0x7f09880081a0 osd=3 3 2024-04-14T15:26:20.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.328+0000 7f09ae067700 20 client.49933.objecter _get_session s=0x7f09880081a0 osd=3 3 2024-04-14T15:26:20.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.328+0000 7f09ae067700 15 client.49933.objecter _session_command_op_remove -1 7 2024-04-14T15:26:20.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.328+0000 7f09ae067700 20 client.49933.objecter get_session s=0x7f09880081a0 osd=3 3 2024-04-14T15:26:20.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.328+0000 7f09ae067700 15 client.49933.objecter _session_command_op_assign 3 7 2024-04-14T15:26:20.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.328+0000 7f09ae067700 20 client.49933.objecter put_session s=0x7f09880081a0 osd=3 4 2024-04-14T15:26:20.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.328+0000 7f09ae067700 10 client.49933.objecter _send_command 7 2024-04-14T15:26:20.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.328+0000 7f09ae067700 1 -- 172.21.15.90:0/3923713408 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f0988011ca0 con 0x7f0988011100 2024-04-14T15:26:20.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.328+0000 7f09a4ff9700 10 client.49933.objecter ms_handle_connect 0x7f0988011100 2024-04-14T15:26:20.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.328+0000 7f09a4ff9700 1 -- 172.21.15.90:0/3923713408 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f0988011ca0 con 0x7f0988011100 2024-04-14T15:26:20.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.328+0000 7f09a4ff9700 10 client.49933.objecter ms_dispatch 0x7f09a8001550 command_reply(tid 7: 0 ) 2024-04-14T15:26:20.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.328+0000 7f09a4ff9700 10 client.49933.objecter _finish_command 7 = system:0 2024-04-14T15:26:20.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.328+0000 7f09a4ff9700 20 client.49933.objecter put_session s=0x7f09880081a0 osd=3 4 2024-04-14T15:26:20.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.328+0000 7f09a4ff9700 15 client.49933.objecter _session_command_op_remove 3 7 2024-04-14T15:26:20.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.345+0000 7f09ae067700 10 client.49933.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:26:20.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.345+0000 7f09ae067700 15 client.49933.objecter _session_command_op_assign -1 8 2024-04-14T15:26:20.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.345+0000 7f09ae067700 20 client.49933.objecter _get_session s=0x7f09880081a0 osd=3 3 2024-04-14T15:26:20.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.345+0000 7f09ae067700 20 client.49933.objecter put_session s=0x7f09880081a0 osd=3 3 2024-04-14T15:26:20.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.345+0000 7f09ae067700 20 client.49933.objecter _get_session s=0x7f09880081a0 osd=3 3 2024-04-14T15:26:20.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.345+0000 7f09ae067700 15 client.49933.objecter _session_command_op_remove -1 8 2024-04-14T15:26:20.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.345+0000 7f09ae067700 20 client.49933.objecter get_session s=0x7f09880081a0 osd=3 3 2024-04-14T15:26:20.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.345+0000 7f09ae067700 15 client.49933.objecter _session_command_op_assign 3 8 2024-04-14T15:26:20.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.345+0000 7f09ae067700 20 client.49933.objecter put_session s=0x7f09880081a0 osd=3 4 2024-04-14T15:26:20.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.345+0000 7f09ae067700 10 client.49933.objecter _send_command 8 2024-04-14T15:26:20.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.345+0000 7f09ae067700 1 -- 172.21.15.90:0/3923713408 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f0988012650 con 0x7f0988011100 2024-04-14T15:26:20.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.347+0000 7f09a4ff9700 1 -- 172.21.15.90:0/3923713408 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f0988012650 con 0x7f0988011100 2024-04-14T15:26:20.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.347+0000 7f09a4ff9700 10 client.49933.objecter ms_dispatch 0x7f09a8001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:26:20.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.347+0000 7f09a4ff9700 10 client.49933.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:26:20.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.347+0000 7f09a4ff9700 20 client.49933.objecter put_session s=0x7f09880081a0 osd=3 4 2024-04-14T15:26:20.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.347+0000 7f09a4ff9700 15 client.49933.objecter _session_command_op_remove 3 8 2024-04-14T15:26:20.348 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:26:20.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.347+0000 7f09ae067700 10 client.49933.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:26:20.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.347+0000 7f09ae067700 15 client.49933.objecter _session_command_op_assign -1 9 2024-04-14T15:26:20.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.347+0000 7f09ae067700 20 client.49933.objecter _get_session s=0x7f0988012650 osd=4 3 2024-04-14T15:26:20.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.347+0000 7f09ae067700 20 client.49933.objecter put_session s=0x7f0988012650 osd=4 3 2024-04-14T15:26:20.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.347+0000 7f09ae067700 20 client.49933.objecter _get_session s=0x7f0988012650 osd=4 3 2024-04-14T15:26:20.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.347+0000 7f09ae067700 15 client.49933.objecter _session_command_op_remove -1 9 2024-04-14T15:26:20.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.347+0000 7f09ae067700 20 client.49933.objecter get_session s=0x7f0988012650 osd=4 3 2024-04-14T15:26:20.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.347+0000 7f09ae067700 15 client.49933.objecter _session_command_op_assign 4 9 2024-04-14T15:26:20.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.347+0000 7f09ae067700 20 client.49933.objecter put_session s=0x7f0988012650 osd=4 4 2024-04-14T15:26:20.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.347+0000 7f09ae067700 10 client.49933.objecter _send_command 9 2024-04-14T15:26:20.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.347+0000 7f09ae067700 1 -- 172.21.15.90:0/3923713408 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f098801a6f0 con 0x7f0988016ae0 2024-04-14T15:26:20.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.347+0000 7f09a4ff9700 10 client.49933.objecter ms_handle_connect 0x7f0988016ae0 2024-04-14T15:26:20.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.348+0000 7f09a4ff9700 1 -- 172.21.15.90:0/3923713408 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f098801a6f0 con 0x7f0988016ae0 2024-04-14T15:26:20.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.348+0000 7f09a4ff9700 10 client.49933.objecter ms_dispatch 0x7f09a8001550 command_reply(tid 9: 0 ) 2024-04-14T15:26:20.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.348+0000 7f09a4ff9700 10 client.49933.objecter _finish_command 9 = system:0 2024-04-14T15:26:20.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.348+0000 7f09a4ff9700 20 client.49933.objecter put_session s=0x7f0988012650 osd=4 4 2024-04-14T15:26:20.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.348+0000 7f09a4ff9700 15 client.49933.objecter _session_command_op_remove 4 9 2024-04-14T15:26:20.351 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:26:20.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.365+0000 7f09ae067700 10 client.49933.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:26:20.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.365+0000 7f09ae067700 15 client.49933.objecter _session_command_op_assign -1 10 2024-04-14T15:26:20.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.365+0000 7f09ae067700 20 client.49933.objecter _get_session s=0x7f0988012650 osd=4 3 2024-04-14T15:26:20.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.365+0000 7f09ae067700 20 client.49933.objecter put_session s=0x7f0988012650 osd=4 3 2024-04-14T15:26:20.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.365+0000 7f09ae067700 20 client.49933.objecter _get_session s=0x7f0988012650 osd=4 3 2024-04-14T15:26:20.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.365+0000 7f09ae067700 15 client.49933.objecter _session_command_op_remove -1 10 2024-04-14T15:26:20.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.365+0000 7f09ae067700 20 client.49933.objecter get_session s=0x7f0988012650 osd=4 3 2024-04-14T15:26:20.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.365+0000 7f09ae067700 15 client.49933.objecter _session_command_op_assign 4 10 2024-04-14T15:26:20.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.365+0000 7f09ae067700 20 client.49933.objecter put_session s=0x7f0988012650 osd=4 4 2024-04-14T15:26:20.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.365+0000 7f09ae067700 10 client.49933.objecter _send_command 10 2024-04-14T15:26:20.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.365+0000 7f09ae067700 1 -- 172.21.15.90:0/3923713408 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f09880120d0 con 0x7f0988016ae0 2024-04-14T15:26:20.367 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:26:20.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.367+0000 7f09a4ff9700 1 -- 172.21.15.90:0/3923713408 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f09880120d0 con 0x7f0988016ae0 2024-04-14T15:26:20.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.367+0000 7f09a4ff9700 10 client.49933.objecter ms_dispatch 0x7f09a8001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:26:20.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.367+0000 7f09a4ff9700 10 client.49933.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:26:20.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.367+0000 7f09a4ff9700 20 client.49933.objecter put_session s=0x7f0988012650 osd=4 4 2024-04-14T15:26:20.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.367+0000 7f09a4ff9700 15 client.49933.objecter _session_command_op_remove 4 10 2024-04-14T15:26:20.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.367+0000 7f09ae067700 10 client.49933.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:26:20.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.367+0000 7f09ae067700 15 client.49933.objecter _session_command_op_assign -1 11 2024-04-14T15:26:20.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.367+0000 7f09ae067700 20 client.49933.objecter _get_session s=0x7f098800c760 osd=5 3 2024-04-14T15:26:20.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.367+0000 7f09ae067700 20 client.49933.objecter put_session s=0x7f098800c760 osd=5 3 2024-04-14T15:26:20.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.367+0000 7f09ae067700 20 client.49933.objecter _get_session s=0x7f098800c760 osd=5 3 2024-04-14T15:26:20.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.367+0000 7f09ae067700 15 client.49933.objecter _session_command_op_remove -1 11 2024-04-14T15:26:20.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.367+0000 7f09ae067700 20 client.49933.objecter get_session s=0x7f098800c760 osd=5 3 2024-04-14T15:26:20.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.367+0000 7f09ae067700 15 client.49933.objecter _session_command_op_assign 5 11 2024-04-14T15:26:20.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.367+0000 7f09ae067700 20 client.49933.objecter put_session s=0x7f098800c760 osd=5 4 2024-04-14T15:26:20.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.367+0000 7f09ae067700 10 client.49933.objecter _send_command 11 2024-04-14T15:26:20.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.367+0000 7f09ae067700 1 -- 172.21.15.90:0/3923713408 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f098801cd80 con 0x7f0988011e80 2024-04-14T15:26:20.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.367+0000 7f09a4ff9700 10 client.49933.objecter ms_handle_connect 0x7f0988011e80 2024-04-14T15:26:20.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.368+0000 7f09a4ff9700 1 -- 172.21.15.90:0/3923713408 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f098801cd80 con 0x7f0988011e80 2024-04-14T15:26:20.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.368+0000 7f09a4ff9700 10 client.49933.objecter ms_dispatch 0x7f09a8001550 command_reply(tid 11: 0 ) 2024-04-14T15:26:20.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.368+0000 7f09a4ff9700 10 client.49933.objecter _finish_command 11 = system:0 2024-04-14T15:26:20.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.368+0000 7f09a4ff9700 20 client.49933.objecter put_session s=0x7f098800c760 osd=5 4 2024-04-14T15:26:20.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.368+0000 7f09a4ff9700 15 client.49933.objecter _session_command_op_remove 5 11 2024-04-14T15:26:20.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:20 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-146","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:26:20.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:20 smithi092 ceph-mon[33966]: osdmap e821: 8 total, 8 up, 7 in 2024-04-14T15:26:20.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:20 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:26:20.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:20 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:26:20.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:20 smithi092 ceph-mon[33966]: pgmap v1223: 224 pgs: 10 creating+peering, 214 active+clean; 4.4 MiB data, 753 MiB used, 625 GiB / 626 GiB avail; 1.3 KiB/s rd, 1.0 KiB/s wr, 2 op/s 2024-04-14T15:26:20.377 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:20.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:20.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:26:20.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.384+0000 7f09ae067700 10 client.49933.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:26:20.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.384+0000 7f09ae067700 15 client.49933.objecter _session_command_op_assign -1 12 2024-04-14T15:26:20.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.384+0000 7f09ae067700 20 client.49933.objecter _get_session s=0x7f098800c760 osd=5 3 2024-04-14T15:26:20.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.384+0000 7f09ae067700 20 client.49933.objecter put_session s=0x7f098800c760 osd=5 3 2024-04-14T15:26:20.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.384+0000 7f09ae067700 20 client.49933.objecter _get_session s=0x7f098800c760 osd=5 3 2024-04-14T15:26:20.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.384+0000 7f09ae067700 15 client.49933.objecter _session_command_op_remove -1 12 2024-04-14T15:26:20.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.384+0000 7f09ae067700 20 client.49933.objecter get_session s=0x7f098800c760 osd=5 3 2024-04-14T15:26:20.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.384+0000 7f09ae067700 15 client.49933.objecter _session_command_op_assign 5 12 2024-04-14T15:26:20.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.384+0000 7f09ae067700 20 client.49933.objecter put_session s=0x7f098800c760 osd=5 4 2024-04-14T15:26:20.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.384+0000 7f09ae067700 10 client.49933.objecter _send_command 12 2024-04-14T15:26:20.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.384+0000 7f09ae067700 1 -- 172.21.15.90:0/3923713408 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f098801cd80 con 0x7f0988011e80 2024-04-14T15:26:20.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.386+0000 7f09a4ff9700 1 -- 172.21.15.90:0/3923713408 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f098801cd80 con 0x7f0988011e80 2024-04-14T15:26:20.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.386+0000 7f09a4ff9700 10 client.49933.objecter ms_dispatch 0x7f09a8001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:26:20.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.386+0000 7f09a4ff9700 10 client.49933.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:26:20.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.386+0000 7f09a4ff9700 20 client.49933.objecter put_session s=0x7f098800c760 osd=5 4 2024-04-14T15:26:20.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.386+0000 7f09a4ff9700 15 client.49933.objecter _session_command_op_remove 5 12 2024-04-14T15:26:20.387 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:26:20.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.387+0000 7f09ae067700 10 client.49933.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:26:20.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.387+0000 7f09ae067700 15 client.49933.objecter _session_command_op_assign -1 13 2024-04-14T15:26:20.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.387+0000 7f09ae067700 20 client.49933.objecter _get_session s=0x7f098801cd80 osd=6 3 2024-04-14T15:26:20.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.387+0000 7f09ae067700 20 client.49933.objecter put_session s=0x7f098801cd80 osd=6 3 2024-04-14T15:26:20.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.387+0000 7f09ae067700 20 client.49933.objecter _get_session s=0x7f098801cd80 osd=6 3 2024-04-14T15:26:20.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.387+0000 7f09ae067700 15 client.49933.objecter _session_command_op_remove -1 13 2024-04-14T15:26:20.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.387+0000 7f09ae067700 20 client.49933.objecter get_session s=0x7f098801cd80 osd=6 3 2024-04-14T15:26:20.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.387+0000 7f09ae067700 15 client.49933.objecter _session_command_op_assign 6 13 2024-04-14T15:26:20.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.387+0000 7f09ae067700 20 client.49933.objecter put_session s=0x7f098801cd80 osd=6 4 2024-04-14T15:26:20.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.387+0000 7f09ae067700 10 client.49933.objecter _send_command 13 2024-04-14T15:26:20.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.387+0000 7f09ae067700 1 -- 172.21.15.90:0/3923713408 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f098801cfc0 con 0x7f098801f7b0 2024-04-14T15:26:20.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.387+0000 7f09a4ff9700 10 client.49933.objecter ms_handle_connect 0x7f098801f7b0 2024-04-14T15:26:20.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.388+0000 7f09a4ff9700 1 -- 172.21.15.90:0/3923713408 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f098801cfc0 con 0x7f098801f7b0 2024-04-14T15:26:20.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.388+0000 7f09a4ff9700 10 client.49933.objecter ms_dispatch 0x7f09a8001550 command_reply(tid 13: 0 ) 2024-04-14T15:26:20.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.388+0000 7f09a4ff9700 10 client.49933.objecter _finish_command 13 = system:0 2024-04-14T15:26:20.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.388+0000 7f09a4ff9700 20 client.49933.objecter put_session s=0x7f098801cd80 osd=6 4 2024-04-14T15:26:20.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.388+0000 7f09a4ff9700 15 client.49933.objecter _session_command_op_remove 6 13 2024-04-14T15:26:20.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.404+0000 7f09ae067700 10 client.49933.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:26:20.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.404+0000 7f09ae067700 15 client.49933.objecter _session_command_op_assign -1 14 2024-04-14T15:26:20.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.404+0000 7f09ae067700 20 client.49933.objecter _get_session s=0x7f098801cd80 osd=6 3 2024-04-14T15:26:20.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.404+0000 7f09ae067700 20 client.49933.objecter put_session s=0x7f098801cd80 osd=6 3 2024-04-14T15:26:20.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.404+0000 7f09ae067700 20 client.49933.objecter _get_session s=0x7f098801cd80 osd=6 3 2024-04-14T15:26:20.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.404+0000 7f09ae067700 15 client.49933.objecter _session_command_op_remove -1 14 2024-04-14T15:26:20.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.404+0000 7f09ae067700 20 client.49933.objecter get_session s=0x7f098801cd80 osd=6 3 2024-04-14T15:26:20.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.404+0000 7f09ae067700 15 client.49933.objecter _session_command_op_assign 6 14 2024-04-14T15:26:20.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.404+0000 7f09ae067700 20 client.49933.objecter put_session s=0x7f098801cd80 osd=6 4 2024-04-14T15:26:20.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.404+0000 7f09ae067700 10 client.49933.objecter _send_command 14 2024-04-14T15:26:20.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.404+0000 7f09ae067700 1 -- 172.21.15.90:0/3923713408 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f09880232c0 con 0x7f098801f7b0 2024-04-14T15:26:20.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.407+0000 7f09a4ff9700 1 -- 172.21.15.90:0/3923713408 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f09880232c0 con 0x7f098801f7b0 2024-04-14T15:26:20.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.407+0000 7f09a4ff9700 10 client.49933.objecter ms_dispatch 0x7f09a8001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:26:20.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.407+0000 7f09a4ff9700 10 client.49933.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:26:20.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.407+0000 7f09a4ff9700 20 client.49933.objecter put_session s=0x7f098801cd80 osd=6 4 2024-04-14T15:26:20.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.407+0000 7f09a4ff9700 15 client.49933.objecter _session_command_op_remove 6 14 2024-04-14T15:26:20.408 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:26:20.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.407+0000 7f09ae067700 10 client.49933.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:26:20.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.407+0000 7f09ae067700 15 client.49933.objecter _session_command_op_assign -1 15 2024-04-14T15:26:20.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.407+0000 7f09ae067700 20 client.49933.objecter _get_session s=0x7f09880232c0 osd=7 3 2024-04-14T15:26:20.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.407+0000 7f09ae067700 20 client.49933.objecter put_session s=0x7f09880232c0 osd=7 3 2024-04-14T15:26:20.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.407+0000 7f09ae067700 20 client.49933.objecter _get_session s=0x7f09880232c0 osd=7 3 2024-04-14T15:26:20.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.407+0000 7f09ae067700 15 client.49933.objecter _session_command_op_remove -1 15 2024-04-14T15:26:20.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.407+0000 7f09ae067700 20 client.49933.objecter get_session s=0x7f09880232c0 osd=7 3 2024-04-14T15:26:20.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.407+0000 7f09ae067700 15 client.49933.objecter _session_command_op_assign 7 15 2024-04-14T15:26:20.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.407+0000 7f09ae067700 20 client.49933.objecter put_session s=0x7f09880232c0 osd=7 4 2024-04-14T15:26:20.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.407+0000 7f09ae067700 10 client.49933.objecter _send_command 15 2024-04-14T15:26:20.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.407+0000 7f09ae067700 1 -- 172.21.15.90:0/3923713408 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f0988027520 con 0x7f0988023940 2024-04-14T15:26:20.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.407+0000 7f09a4ff9700 10 client.49933.objecter ms_handle_connect 0x7f0988023940 2024-04-14T15:26:20.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.408+0000 7f09a4ff9700 1 -- 172.21.15.90:0/3923713408 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f0988027520 con 0x7f0988023940 2024-04-14T15:26:20.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.408+0000 7f09a4ff9700 10 client.49933.objecter ms_dispatch 0x7f09a8001550 command_reply(tid 15: 0 ) 2024-04-14T15:26:20.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.408+0000 7f09a4ff9700 10 client.49933.objecter _finish_command 15 = system:0 2024-04-14T15:26:20.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.408+0000 7f09a4ff9700 20 client.49933.objecter put_session s=0x7f09880232c0 osd=7 4 2024-04-14T15:26:20.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.408+0000 7f09a4ff9700 15 client.49933.objecter _session_command_op_remove 7 15 2024-04-14T15:26:20.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.424+0000 7f09ae067700 10 client.49933.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:26:20.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.424+0000 7f09ae067700 15 client.49933.objecter _session_command_op_assign -1 16 2024-04-14T15:26:20.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.424+0000 7f09ae067700 20 client.49933.objecter _get_session s=0x7f09880232c0 osd=7 3 2024-04-14T15:26:20.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.424+0000 7f09ae067700 20 client.49933.objecter put_session s=0x7f09880232c0 osd=7 3 2024-04-14T15:26:20.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.424+0000 7f09ae067700 20 client.49933.objecter _get_session s=0x7f09880232c0 osd=7 3 2024-04-14T15:26:20.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.424+0000 7f09ae067700 15 client.49933.objecter _session_command_op_remove -1 16 2024-04-14T15:26:20.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.424+0000 7f09ae067700 20 client.49933.objecter get_session s=0x7f09880232c0 osd=7 3 2024-04-14T15:26:20.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.424+0000 7f09ae067700 15 client.49933.objecter _session_command_op_assign 7 16 2024-04-14T15:26:20.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.424+0000 7f09ae067700 20 client.49933.objecter put_session s=0x7f09880232c0 osd=7 4 2024-04-14T15:26:20.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.424+0000 7f09ae067700 10 client.49933.objecter _send_command 16 2024-04-14T15:26:20.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.424+0000 7f09ae067700 1 -- 172.21.15.90:0/3923713408 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f098801c1b0 con 0x7f0988023940 2024-04-14T15:26:20.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.426+0000 7f09a4ff9700 1 -- 172.21.15.90:0/3923713408 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f098801c1b0 con 0x7f0988023940 2024-04-14T15:26:20.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.426+0000 7f09a4ff9700 10 client.49933.objecter ms_dispatch 0x7f09a8001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:26:20.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.426+0000 7f09a4ff9700 10 client.49933.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:26:20.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.426+0000 7f09a4ff9700 20 client.49933.objecter put_session s=0x7f09880232c0 osd=7 4 2024-04-14T15:26:20.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.426+0000 7f09a4ff9700 15 client.49933.objecter _session_command_op_remove 7 16 2024-04-14T15:26:20.427 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:26:20.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.426+0000 7f09ae067700 10 librados: watch_flush enter 2024-04-14T15:26:20.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.426+0000 7f09ae067700 10 librados: watch_flush exit 2024-04-14T15:26:20.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.426+0000 7f09ae067700 10 client.49933.objecter close_session for osd.0 2024-04-14T15:26:20.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.426+0000 7f09ae067700 1 -- 172.21.15.90:0/3923713408 >> v1:172.21.15.90:6801/3761806885 conn(0x7f0988004730 legacy=0x7f0988002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:20.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.426+0000 7f09ae067700 20 client.49933.objecter put_session s=0x7f09880053e0 osd=0 1 2024-04-14T15:26:20.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.426+0000 7f09ae067700 10 client.49933.objecter close_session for osd.1 2024-04-14T15:26:20.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.426+0000 7f09ae067700 1 -- 172.21.15.90:0/3923713408 >> v1:172.21.15.90:6805/4170361699 conn(0x7f0988004280 legacy=0x7f098800ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:20.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.426+0000 7f09ae067700 20 client.49933.objecter put_session s=0x7f0988004110 osd=1 1 2024-04-14T15:26:20.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.426+0000 7f09ae067700 10 client.49933.objecter close_session for osd.2 2024-04-14T15:26:20.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.426+0000 7f09ae067700 1 -- 172.21.15.90:0/3923713408 >> v1:172.21.15.90:6809/1686184251 conn(0x7f0988008880 legacy=0x7f098800f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:20.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.427+0000 7f09ae067700 20 client.49933.objecter put_session s=0x7f0988001a00 osd=2 1 2024-04-14T15:26:20.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.427+0000 7f09ae067700 10 client.49933.objecter close_session for osd.3 2024-04-14T15:26:20.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.427+0000 7f09ae067700 1 -- 172.21.15.90:0/3923713408 >> v1:172.21.15.90:6813/2270166774 conn(0x7f0988011100 legacy=0x7f0988011590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:20.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.427+0000 7f09ae067700 20 client.49933.objecter put_session s=0x7f09880081a0 osd=3 1 2024-04-14T15:26:20.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.427+0000 7f09ae067700 10 client.49933.objecter close_session for osd.4 2024-04-14T15:26:20.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.427+0000 7f09ae067700 1 -- 172.21.15.90:0/3923713408 >> v1:172.21.15.92:6800/3887549733 conn(0x7f0988016ae0 legacy=0x7f0988018ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:20.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.427+0000 7f09ae067700 20 client.49933.objecter put_session s=0x7f0988012650 osd=4 1 2024-04-14T15:26:20.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.427+0000 7f09ae067700 10 client.49933.objecter close_session for osd.5 2024-04-14T15:26:20.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.427+0000 7f09ae067700 1 -- 172.21.15.90:0/3923713408 >> v1:172.21.15.92:6804/3244561440 conn(0x7f0988011e80 legacy=0x7f098801a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:20.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.427+0000 7f09a7fff700 1 -- 172.21.15.90:0/3923713408 reap_dead start 2024-04-14T15:26:20.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.427+0000 7f09ae067700 20 client.49933.objecter put_session s=0x7f098800c760 osd=5 1 2024-04-14T15:26:20.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.427+0000 7f09ae067700 10 client.49933.objecter close_session for osd.6 2024-04-14T15:26:20.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.427+0000 7f09ae067700 1 -- 172.21.15.90:0/3923713408 >> v1:172.21.15.92:6808/636125963 conn(0x7f098801f7b0 legacy=0x7f0988021ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:20.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.427+0000 7f09ae067700 20 client.49933.objecter put_session s=0x7f098801cd80 osd=6 1 2024-04-14T15:26:20.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.427+0000 7f09ae067700 10 client.49933.objecter close_session for osd.7 2024-04-14T15:26:20.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.427+0000 7f09a7fff700 1 -- 172.21.15.90:0/3923713408 reap_dead start 2024-04-14T15:26:20.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.427+0000 7f09ae067700 1 -- 172.21.15.90:0/3923713408 >> v1:172.21.15.92:6812/1773706716 conn(0x7f0988023940 legacy=0x7f0988025d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:20.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.427+0000 7f09ae067700 20 client.49933.objecter put_session s=0x7f09880232c0 osd=7 1 2024-04-14T15:26:20.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.427+0000 7f09ae067700 20 client.49933.objecter shutdown clearing up homeless session... 2024-04-14T15:26:20.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.427+0000 7f09ae067700 10 client.49933.objecter successfully canceled tick 2024-04-14T15:26:20.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.427+0000 7f09a7fff700 1 -- 172.21.15.90:0/3923713408 reap_dead start 2024-04-14T15:26:20.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.427+0000 7f09a7fff700 1 -- 172.21.15.90:0/3923713408 reap_dead start 2024-04-14T15:26:20.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.427+0000 7f09ae067700 1 -- 172.21.15.90:0/3923713408 >> v1:172.21.15.90:6800/420935427 conn(0x7f0994071460 legacy=0x7f0994073920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:20.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.427+0000 7f09ae067700 1 -- 172.21.15.90:0/3923713408 >> v1:172.21.15.92:6789/0 conn(0x7f09a8074ee0 legacy=0x7f09a810a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:20.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.427+0000 7f09ae067700 1 -- 172.21.15.90:0/3923713408 shutdown_connections 2024-04-14T15:26:20.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.427+0000 7f09ae067700 1 -- 172.21.15.90:0/3923713408 >> 172.21.15.90:0/3923713408 conn(0x7f09a80fe3d0 msgr2=0x7f09a8110c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:26:20.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.427+0000 7f09ae067700 1 -- 172.21.15.90:0/3923713408 shutdown_connections 2024-04-14T15:26:20.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.427+0000 7f09ae067700 1 -- 172.21.15.90:0/3923713408 wait complete. 2024-04-14T15:26:20.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:20.427+0000 7f09ae067700 1 librados: shutdown 2024-04-14T15:26:20.432 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:26:20.432 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:26:20.432 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:26:20.432 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:26:20.432 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:26:20.432 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:26:20.432 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:26:20.432 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:26:20.480 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:26:20.504 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:20.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:20.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:26:20.606 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:26:20.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:20.634 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:20.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:26:20.736 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:26:20.762 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:20.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:20.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:26:20.863 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:26:20.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:20.888 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:20.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:26:20.989 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:26:21.014 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:21.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:21.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:26:21.116 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:26:21.140 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:21.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:21.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:26:21.241 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:26:21.268 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:21.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:21.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:26:21.307 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:21 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-146", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:26:21.307 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:21 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-146"}]: dispatch 2024-04-14T15:26:21.307 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:21 smithi090 ceph-mon[41670]: osdmap e822: 8 total, 8 up, 7 in 2024-04-14T15:26:21.307 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:21 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-146"}]: dispatch 2024-04-14T15:26:21.307 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:21 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3923713408' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:26:21.307 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:21 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-146", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:26:21.307 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:21 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-146"}]: dispatch 2024-04-14T15:26:21.308 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:21 smithi090 ceph-mon[30399]: osdmap e822: 8 total, 8 up, 7 in 2024-04-14T15:26:21.308 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:21 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-146"}]: dispatch 2024-04-14T15:26:21.308 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:21 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3923713408' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:26:21.329 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-14T15:26:21.330 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:26:21.369 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:26:21.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:21 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-146", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:26:21.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:21 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-146"}]: dispatch 2024-04-14T15:26:21.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:21 smithi092 ceph-mon[33966]: osdmap e822: 8 total, 8 up, 7 in 2024-04-14T15:26:21.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:21 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-146"}]: dispatch 2024-04-14T15:26:21.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:21 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3923713408' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:26:21.395 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:21.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:21.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:26:21.486 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd unset noscrub 2024-04-14T15:26:21.496 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:26:21.521 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:21.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:21.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:26:21.623 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:26:21.649 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:21.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:21.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:26:21.750 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:26:21.774 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:21.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:21.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:26:21.876 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:26:21.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:21.904 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:21.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:26:22.005 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:26:22.030 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:22.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:22.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:26:22.053 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:26:22.132 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:26:22.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:22.156 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:22.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:26:22.257 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:26:22.280 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:22.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:22.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:26:22.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:22 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-146"}]': finished 2024-04-14T15:26:22.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:22 smithi092 ceph-mon[33966]: osdmap e823: 8 total, 8 up, 7 in 2024-04-14T15:26:22.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:22 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-146", "mode": "writeback"}]: dispatch 2024-04-14T15:26:22.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:22 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-146", "mode": "writeback"}]: dispatch 2024-04-14T15:26:22.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:22 smithi092 ceph-mon[33966]: pgmap v1226: 224 pgs: 10 creating+peering, 214 active+clean; 4.4 MiB data, 753 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-04-14T15:26:22.382 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:26:22.406 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:22.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:22.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:26:22.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:22 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-146"}]': finished 2024-04-14T15:26:22.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:22 smithi090 ceph-mon[41670]: osdmap e823: 8 total, 8 up, 7 in 2024-04-14T15:26:22.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:22 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-146", "mode": "writeback"}]: dispatch 2024-04-14T15:26:22.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:22 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-146", "mode": "writeback"}]: dispatch 2024-04-14T15:26:22.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:22 smithi090 ceph-mon[41670]: pgmap v1226: 224 pgs: 10 creating+peering, 214 active+clean; 4.4 MiB data, 753 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-04-14T15:26:22.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:22 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi090-93652-111", "overlaypool": "test-rados-api-smithi090-93652-146"}]': finished 2024-04-14T15:26:22.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:22 smithi090 ceph-mon[30399]: osdmap e823: 8 total, 8 up, 7 in 2024-04-14T15:26:22.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:22 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-146", "mode": "writeback"}]: dispatch 2024-04-14T15:26:22.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:22 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-146", "mode": "writeback"}]: dispatch 2024-04-14T15:26:22.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:22 smithi090 ceph-mon[30399]: pgmap v1226: 224 pgs: 10 creating+peering, 214 active+clean; 4.4 MiB data, 753 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-04-14T15:26:22.507 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:26:22.519 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:26:22.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:22.533 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:22.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:26:22.537 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-14T15:26:22.634 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:26:22.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:22.660 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:22.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:26:22.761 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:26:22.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:22.785 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:22.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:26:22.886 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:26:22.910 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:22.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:22.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:26:23.012 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:26:23.035 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:23.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:23.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:26:23.137 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:26:23.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:23.161 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:23.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:26:23.262 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:26:23.285 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:23.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:23.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:26:23.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:23 smithi092 ceph-mon[33966]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:26:23.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:23 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-146", "mode": "writeback"}]': finished 2024-04-14T15:26:23.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:23 smithi092 ceph-mon[33966]: osdmap e824: 8 total, 8 up, 7 in 2024-04-14T15:26:23.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:23 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-146","var": "hit_set_count","val": "2"}]: dispatch 2024-04-14T15:26:23.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:23 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-146","var": "hit_set_count","val": "2"}]: dispatch 2024-04-14T15:26:23.387 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:26:23.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:23.411 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:23.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:26:23.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:23 smithi090 ceph-mon[41670]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:26:23.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:23 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-146", "mode": "writeback"}]': finished 2024-04-14T15:26:23.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:23 smithi090 ceph-mon[41670]: osdmap e824: 8 total, 8 up, 7 in 2024-04-14T15:26:23.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:23 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-146","var": "hit_set_count","val": "2"}]: dispatch 2024-04-14T15:26:23.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:23 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-146","var": "hit_set_count","val": "2"}]: dispatch 2024-04-14T15:26:23.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:23 smithi090 ceph-mon[30399]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-14T15:26:23.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:23 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi090-93652-146", "mode": "writeback"}]': finished 2024-04-14T15:26:23.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:23 smithi090 ceph-mon[30399]: osdmap e824: 8 total, 8 up, 7 in 2024-04-14T15:26:23.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:23 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-146","var": "hit_set_count","val": "2"}]: dispatch 2024-04-14T15:26:23.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:23 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-146","var": "hit_set_count","val": "2"}]: dispatch 2024-04-14T15:26:23.512 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:26:23.536 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:23.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:23.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:26:23.638 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:26:23.661 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:23.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:23.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:26:23.763 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:26:23.785 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:23.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:23.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:26:23.823 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-14T15:26:23.887 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:26:23.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:23.913 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:23.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:26:24.014 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:26:24.042 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:24.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:24.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:26:24.101 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:26:23 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:26:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:26:24.145 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:26:24.169 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:24.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:24.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:26:24.271 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:26:24.294 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:24.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:24.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:26:24.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:24 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-146","var": "hit_set_count","val": "2"}]': finished 2024-04-14T15:26:24.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:24 smithi092 ceph-mon[33966]: osdmap e825: 8 total, 8 up, 7 in 2024-04-14T15:26:24.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:24 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-146","var": "hit_set_period","val": "600"}]: dispatch 2024-04-14T15:26:24.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:24 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-146","var": "hit_set_period","val": "600"}]: dispatch 2024-04-14T15:26:24.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:24 smithi092 ceph-mon[33966]: pgmap v1229: 224 pgs: 10 creating+peering, 214 active+clean; 4.4 MiB data, 753 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:26:24.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:24 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:26:24.396 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:26:24.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:24.420 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:24.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:26:24.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:24 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-146","var": "hit_set_count","val": "2"}]': finished 2024-04-14T15:26:24.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:24 smithi090 ceph-mon[41670]: osdmap e825: 8 total, 8 up, 7 in 2024-04-14T15:26:24.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:24 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-146","var": "hit_set_period","val": "600"}]: dispatch 2024-04-14T15:26:24.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:24 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-146","var": "hit_set_period","val": "600"}]: dispatch 2024-04-14T15:26:24.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:24 smithi090 ceph-mon[41670]: pgmap v1229: 224 pgs: 10 creating+peering, 214 active+clean; 4.4 MiB data, 753 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:26:24.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:24 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:26:24.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:24 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-146","var": "hit_set_count","val": "2"}]': finished 2024-04-14T15:26:24.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:24 smithi090 ceph-mon[30399]: osdmap e825: 8 total, 8 up, 7 in 2024-04-14T15:26:24.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:24 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-146","var": "hit_set_period","val": "600"}]: dispatch 2024-04-14T15:26:24.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:24 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-146","var": "hit_set_period","val": "600"}]: dispatch 2024-04-14T15:26:24.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:24 smithi090 ceph-mon[30399]: pgmap v1229: 224 pgs: 10 creating+peering, 214 active+clean; 4.4 MiB data, 753 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:26:24.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:24 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:26:24.521 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:26:24.547 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:24.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:24.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:26:24.589 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:26:24.649 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:26:24.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:24.673 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:24.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:26:24.774 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:26:24.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:24.798 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:24.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:26:24.899 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:26:24.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:24.923 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:24.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:26:25.024 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:26:25.049 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:25.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:25.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:26:25.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:25.055+0000 7fc0b5457700 1 -- 172.21.15.90:0/1638088847 >> v1:172.21.15.90:6789/0 conn(0x7fc0b010a660 legacy=0x7fc0b010ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:25.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:25.055+0000 7fc0b5457700 1 -- 172.21.15.90:0/1638088847 shutdown_connections 2024-04-14T15:26:25.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:25.055+0000 7fc0b5457700 1 -- 172.21.15.90:0/1638088847 >> 172.21.15.90:0/1638088847 conn(0x7fc0b00fd2c0 msgr2=0x7fc0b00ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:26:25.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:25.055+0000 7fc0b5457700 1 -- 172.21.15.90:0/1638088847 shutdown_connections 2024-04-14T15:26:25.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:25.055+0000 7fc0b5457700 1 -- 172.21.15.90:0/1638088847 wait complete. 2024-04-14T15:26:25.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:25.056+0000 7fc0b5457700 1 librados: starting msgr at 2024-04-14T15:26:25.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:25.056+0000 7fc0b5457700 1 librados: starting objecter 2024-04-14T15:26:25.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:25.056+0000 7fc0b5457700 1 Processor -- start 2024-04-14T15:26:25.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:25.056+0000 7fc0b5457700 1 -- start start 2024-04-14T15:26:25.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:25.056+0000 7fc0b5457700 1 librados: setting wanted keys 2024-04-14T15:26:25.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:25.056+0000 7fc0b5457700 1 librados: calling monclient init 2024-04-14T15:26:25.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:25.056+0000 7fc0b5457700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc0b01a9bd0 con 0x7fc0b010e230 2024-04-14T15:26:25.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:25.056+0000 7fc0b5457700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc0b01aadf0 con 0x7fc0b01077b0 2024-04-14T15:26:25.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:25.056+0000 7fc0b5457700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc0b01ac030 con 0x7fc0b010a660 2024-04-14T15:26:25.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:25.056+0000 7fc0af7fe700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fc0b010e230 0x7fc0b01a8290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:46966/0 (socket says 172.21.15.90:46966) 2024-04-14T15:26:25.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:25.056+0000 7fc0af7fe700 1 -- 172.21.15.90:0/999594090 learned_addr learned my addr 172.21.15.90:0/999594090 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:26:25.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:25.056+0000 7fc0aeffd700 1 --1- 172.21.15.90:0/999594090 >> v1:172.21.15.92:6789/0 conn(0x7fc0b01077b0 0x7fc0b00ff190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.90:34144/0 (socket says 172.21.15.90:34144) 2024-04-14T15:26:25.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:25.056+0000 7fc097fff700 10 client.?.objecter ms_handle_connect 0x7fc0b010a660 2024-04-14T15:26:25.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:25.056+0000 7fc097fff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:26:25.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:25.056+0000 7fc097fff700 10 client.?.objecter ms_handle_connect 0x7fc0b01077b0 2024-04-14T15:26:25.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:25.056+0000 7fc097fff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:26:25.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:25.056+0000 7fc097fff700 10 client.?.objecter ms_handle_connect 0x7fc0b010e230 2024-04-14T15:26:25.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:25.056+0000 7fc097fff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:26:25.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:25.057+0000 7fc097fff700 1 -- 172.21.15.90:0/999594090 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4110818460 0 0) 0x7fc0b01aadf0 con 0x7fc0b01077b0 2024-04-14T15:26:25.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:25.057+0000 7fc097fff700 1 -- 172.21.15.90:0/999594090 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc0980031f0 con 0x7fc0b01077b0 2024-04-14T15:26:25.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:25.057+0000 7fc097fff700 1 -- 172.21.15.90:0/999594090 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1189876909 0 0) 0x7fc0b01ac030 con 0x7fc0b010a660 2024-04-14T15:26:25.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:25.057+0000 7fc097fff700 1 -- 172.21.15.90:0/999594090 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc0b01aadf0 con 0x7fc0b010a660 2024-04-14T15:26:25.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:25.057+0000 7fc097fff700 1 -- 172.21.15.90:0/999594090 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1758528584 0 0) 0x7fc0b01a9bd0 con 0x7fc0b010e230 2024-04-14T15:26:25.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:25.057+0000 7fc097fff700 1 -- 172.21.15.90:0/999594090 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc0b01ac030 con 0x7fc0b010e230 2024-04-14T15:26:25.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:25.057+0000 7fc097fff700 1 -- 172.21.15.90:0/999594090 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1110450932 0 0) 0x7fc0980031f0 con 0x7fc0b01077b0 2024-04-14T15:26:25.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:25.057+0000 7fc097fff700 1 -- 172.21.15.90:0/999594090 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc0b01a9bd0 con 0x7fc0b01077b0 2024-04-14T15:26:25.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:25.057+0000 7fc097fff700 1 -- 172.21.15.90:0/999594090 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fc0a00018b0 con 0x7fc0b01077b0 2024-04-14T15:26:25.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:25.057+0000 7fc097fff700 1 -- 172.21.15.90:0/999594090 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3679590507 0 0) 0x7fc0b01aadf0 con 0x7fc0b010a660 2024-04-14T15:26:25.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:25.057+0000 7fc097fff700 1 -- 172.21.15.90:0/999594090 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc0980031f0 con 0x7fc0b010a660 2024-04-14T15:26:25.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:25.057+0000 7fc097fff700 1 -- 172.21.15.90:0/999594090 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3512876405 0 0) 0x7fc0b01ac030 con 0x7fc0b010e230 2024-04-14T15:26:25.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:25.057+0000 7fc097fff700 1 -- 172.21.15.90:0/999594090 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc0b01aadf0 con 0x7fc0b010e230 2024-04-14T15:26:25.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:25.057+0000 7fc097fff700 1 -- 172.21.15.90:0/999594090 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fc0a4001410 con 0x7fc0b010a660 2024-04-14T15:26:25.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:25.057+0000 7fc097fff700 1 -- 172.21.15.90:0/999594090 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fc0a8001bd0 con 0x7fc0b010e230 2024-04-14T15:26:25.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:25.057+0000 7fc097fff700 1 -- 172.21.15.90:0/999594090 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3645453338 0 0) 0x7fc0b01a9bd0 con 0x7fc0b01077b0 2024-04-14T15:26:25.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:25.057+0000 7fc097fff700 1 -- 172.21.15.90:0/999594090 >> v1:172.21.15.90:6790/0 conn(0x7fc0b010a660 legacy=0x7fc0b01a4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:25.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:25.057+0000 7fc097fff700 1 -- 172.21.15.90:0/999594090 >> v1:172.21.15.90:6789/0 conn(0x7fc0b010e230 legacy=0x7fc0b01a8290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:25.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:25.057+0000 7fc097fff700 1 -- 172.21.15.90:0/999594090 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0b01ad270 con 0x7fc0b01077b0 2024-04-14T15:26:25.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:25.057+0000 7fc0b5457700 1 -- 172.21.15.90:0/999594090 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc0b01ab020 con 0x7fc0b01077b0 2024-04-14T15:26:25.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:25.057+0000 7fc0b5457700 10 client.49939.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:26:25.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:25.057+0000 7fc0b5457700 1 -- 172.21.15.90:0/999594090 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc0b01ab4e0 con 0x7fc0b01077b0 2024-04-14T15:26:25.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:25.057+0000 7fc0b5457700 1 librados: init done 2024-04-14T15:26:25.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:25.058+0000 7fc097fff700 1 -- 172.21.15.90:0/999594090 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fc0a0003b20 con 0x7fc0b01077b0 2024-04-14T15:26:25.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:25.058+0000 7fc097fff700 1 -- 172.21.15.90:0/999594090 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fc0a0005ca0 con 0x7fc0b01077b0 2024-04-14T15:26:25.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:25.059+0000 7fc0b5457700 1 -- 172.21.15.90:0/999594090 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc09c005120 con 0x7fc0b01077b0 2024-04-14T15:26:25.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:25.059+0000 7fc097fff700 1 -- 172.21.15.90:0/999594090 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fc0a001c6b0 con 0x7fc0b01077b0 2024-04-14T15:26:25.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:25.059+0000 7fc097fff700 1 -- 172.21.15.90:0/999594090 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(826..826 src has 467..826) ==== 8757+0+0 (unknown 2922180718 0 0) 0x7fc0a008ac10 con 0x7fc0b01077b0 2024-04-14T15:26:25.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:25.060+0000 7fc097fff700 10 client.49939.objecter ms_dispatch 0x7fc0b0001550 osd_map(826..826 src has 467..826) 2024-04-14T15:26:25.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:25.060+0000 7fc097fff700 3 client.49939.objecter handle_osd_map got epochs [826,826] > 0 2024-04-14T15:26:25.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:25.060+0000 7fc097fff700 3 client.49939.objecter handle_osd_map decoding full epoch 826 2024-04-14T15:26:25.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:25.060+0000 7fc097fff700 20 client.49939.objecter dump_active .. 0 homeless 2024-04-14T15:26:25.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:25.060+0000 7fc097fff700 10 client.49939.objecter ms_handle_connect 0x7fc0980716e0 2024-04-14T15:26:25.063 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:26:25.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:25.064+0000 7fc097fff700 1 -- 172.21.15.90:0/999594090 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fc0a00573d0 con 0x7fc0b01077b0 2024-04-14T15:26:25.151 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:26:25.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:25.175 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:25.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:26:25.201 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:25 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-146","var": "hit_set_period","val": "600"}]': finished 2024-04-14T15:26:25.201 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:25 smithi090 ceph-mon[41670]: osdmap e826: 8 total, 8 up, 7 in 2024-04-14T15:26:25.202 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:25 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-14T15:26:25.202 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:25 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:26:25.202 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:25 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-14T15:26:25.202 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:25 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:26:25.202 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:25 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:26:25.202 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:25 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:26:25.202 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:25 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:26:25.202 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:25 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:26:25.202 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:25 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:26:25.202 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:25 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:26:25.203 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:25 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-146","var": "hit_set_period","val": "600"}]': finished 2024-04-14T15:26:25.203 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:25 smithi090 ceph-mon[30399]: osdmap e826: 8 total, 8 up, 7 in 2024-04-14T15:26:25.203 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:25 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-14T15:26:25.203 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:25 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:26:25.203 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:25 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-14T15:26:25.203 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:25 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:26:25.203 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:25 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:26:25.203 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:25 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:26:25.203 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:25 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:26:25.203 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:25 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:26:25.203 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:25 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:26:25.203 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:25 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:26:25.276 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:26:25.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:25.291+0000 7fc0b5457700 1 -- 172.21.15.90:0/999594090 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} v 0) -- 0x7fc09c005c20 con 0x7fc0b01077b0 2024-04-14T15:26:25.300 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:25.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:25.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:26:25.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:25 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-146","var": "hit_set_period","val": "600"}]': finished 2024-04-14T15:26:25.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:25 smithi092 ceph-mon[33966]: osdmap e826: 8 total, 8 up, 7 in 2024-04-14T15:26:25.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:25 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-14T15:26:25.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:25 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:26:25.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:25 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-14T15:26:25.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:25 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:26:25.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:25 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:26:25.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:25 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:26:25.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:25 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:26:25.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:25 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:26:25.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:25 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:26:25.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:25 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:26:25.402 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:26:25.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:25.427 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:25.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:26:25.528 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:26:25.552 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:25.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:25.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:26:25.654 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:26:25.678 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:25.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:25.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:26:25.780 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:26:25.803 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:25.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:25.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:26:25.872 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:26:25 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:26:25.905 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:26:25.928 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:25.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:25.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:26:26.029 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:26:26.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.036+0000 7fc097fff700 1 -- 172.21.15.90:0/999594090 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]=0 set osd.5 primary-affinity to 1 (8655362) v828) ==== 133+0+0 (unknown 2847791201 0 0) 0x7fc0a005b080 con 0x7fc0b01077b0 2024-04-14T15:26:26.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.039+0000 7fc0b5457700 10 librados: watch_flush enter 2024-04-14T15:26:26.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.039+0000 7fc0b5457700 10 librados: watch_flush exit 2024-04-14T15:26:26.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.039+0000 7fc0b5457700 20 client.49939.objecter shutdown clearing up homeless session... 2024-04-14T15:26:26.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.039+0000 7fc0b5457700 10 client.49939.objecter successfully canceled tick 2024-04-14T15:26:26.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.040+0000 7fc0b5457700 1 -- 172.21.15.90:0/999594090 >> v1:172.21.15.90:6800/420935427 conn(0x7fc0980716e0 legacy=0x7fc098073ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:26.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.040+0000 7fc0b5457700 1 -- 172.21.15.90:0/999594090 >> v1:172.21.15.92:6789/0 conn(0x7fc0b01077b0 legacy=0x7fc0b00ff190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:26.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.040+0000 7fc0b5457700 1 -- 172.21.15.90:0/999594090 shutdown_connections 2024-04-14T15:26:26.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.040+0000 7fc0b5457700 1 -- 172.21.15.90:0/999594090 >> 172.21.15.90:0/999594090 conn(0x7fc0b00fd2c0 msgr2=0x7fc0b0100720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:26:26.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.040+0000 7fc0b5457700 1 -- 172.21.15.90:0/999594090 shutdown_connections 2024-04-14T15:26:26.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.040+0000 7fc0b5457700 1 -- 172.21.15.90:0/999594090 wait complete. 2024-04-14T15:26:26.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.040+0000 7fc0b5457700 1 librados: shutdown 2024-04-14T15:26:26.041 INFO:teuthology.orchestra.run.smithi090.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-04-14T15:26:26.061 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:26.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:26.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:26:26.163 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:26:26.189 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:26.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:26.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:26:26.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:26 smithi090 ceph-mon[30399]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:26:26.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:26 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-146","var": "hit_set_type","val": "bloom"}]': finished 2024-04-14T15:26:26.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:26 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-14T15:26:26.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:26 smithi090 ceph-mon[30399]: osdmap e827: 8 total, 8 up, 7 in 2024-04-14T15:26:26.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:26 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-14T15:26:26.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:26 smithi090 ceph-mon[30399]: pgmap v1232: 224 pgs: 224 active+clean; 4.4 MiB data, 738 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-04-14T15:26:26.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:26 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/999594090' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-04-14T15:26:26.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:26 smithi090 ceph-mon[30399]: from='client.49939 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-04-14T15:26:26.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:26 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:26:26.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:26 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-146","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-04-14T15:26:26.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:26 smithi090 ceph-mon[30399]: from='client.49939 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-04-14T15:26:26.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:26 smithi090 ceph-mon[30399]: osdmap e828: 8 total, 8 up, 7 in 2024-04-14T15:26:26.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:26 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-146","var": "target_max_objects","val": "1"}]: dispatch 2024-04-14T15:26:26.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:26 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-146","var": "target_max_objects","val": "1"}]: dispatch 2024-04-14T15:26:26.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:26 smithi090 ceph-mon[41670]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:26:26.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:26 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-146","var": "hit_set_type","val": "bloom"}]': finished 2024-04-14T15:26:26.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:26 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-14T15:26:26.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:26 smithi090 ceph-mon[41670]: osdmap e827: 8 total, 8 up, 7 in 2024-04-14T15:26:26.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:26 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-14T15:26:26.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:26 smithi090 ceph-mon[41670]: pgmap v1232: 224 pgs: 224 active+clean; 4.4 MiB data, 738 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-04-14T15:26:26.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:26 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/999594090' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-04-14T15:26:26.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:26 smithi090 ceph-mon[41670]: from='client.49939 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-04-14T15:26:26.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:26 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:26:26.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:26 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-146","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-04-14T15:26:26.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:26 smithi090 ceph-mon[41670]: from='client.49939 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-04-14T15:26:26.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:26 smithi090 ceph-mon[41670]: osdmap e828: 8 total, 8 up, 7 in 2024-04-14T15:26:26.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:26 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-146","var": "target_max_objects","val": "1"}]: dispatch 2024-04-14T15:26:26.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:26 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-146","var": "target_max_objects","val": "1"}]: dispatch 2024-04-14T15:26:26.291 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:26:26.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:26.317 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:26.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:26:26.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:26 smithi092 ceph-mon[33966]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-14T15:26:26.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:26 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-146","var": "hit_set_type","val": "bloom"}]': finished 2024-04-14T15:26:26.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:26 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-14T15:26:26.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:26 smithi092 ceph-mon[33966]: osdmap e827: 8 total, 8 up, 7 in 2024-04-14T15:26:26.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:26 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-14T15:26:26.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:26 smithi092 ceph-mon[33966]: pgmap v1232: 224 pgs: 224 active+clean; 4.4 MiB data, 738 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-04-14T15:26:26.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:26 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/999594090' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-04-14T15:26:26.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:26 smithi092 ceph-mon[33966]: from='client.49939 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-04-14T15:26:26.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:26 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:26:26.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:26 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-146","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-04-14T15:26:26.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:26 smithi092 ceph-mon[33966]: from='client.49939 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-04-14T15:26:26.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:26 smithi092 ceph-mon[33966]: osdmap e828: 8 total, 8 up, 7 in 2024-04-14T15:26:26.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:26 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-146","var": "target_max_objects","val": "1"}]: dispatch 2024-04-14T15:26:26.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:26 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-146","var": "target_max_objects","val": "1"}]: dispatch 2024-04-14T15:26:26.418 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:26:26.442 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:26.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:26.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:26:26.544 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:26:26.567 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:26.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:26.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:26:26.669 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:26:26.694 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:26.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:26.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:26:26.795 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:26:26.818 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:26.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:26.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:26:26.920 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:26:26.943 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:26.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:26.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:26:26.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.959+0000 7fac5f154700 1 -- 172.21.15.90:0/303249728 >> v1:172.21.15.90:6790/0 conn(0x7fac58107780 legacy=0x7fac58107b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:26.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.959+0000 7fac5f154700 1 -- 172.21.15.90:0/303249728 shutdown_connections 2024-04-14T15:26:26.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.959+0000 7fac5f154700 1 -- 172.21.15.90:0/303249728 >> 172.21.15.90:0/303249728 conn(0x7fac580fd290 msgr2=0x7fac580ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:26:26.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.959+0000 7fac5f154700 1 -- 172.21.15.90:0/303249728 shutdown_connections 2024-04-14T15:26:26.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.959+0000 7fac5f154700 1 -- 172.21.15.90:0/303249728 wait complete. 2024-04-14T15:26:26.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.959+0000 7fac5f154700 1 librados: starting msgr at 2024-04-14T15:26:26.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.959+0000 7fac5f154700 1 librados: starting objecter 2024-04-14T15:26:26.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.960+0000 7fac5f154700 1 Processor -- start 2024-04-14T15:26:26.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.960+0000 7fac5f154700 1 -- start start 2024-04-14T15:26:26.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.960+0000 7fac5f154700 1 librados: setting wanted keys 2024-04-14T15:26:26.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.960+0000 7fac5f154700 1 librados: calling monclient init 2024-04-14T15:26:26.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.960+0000 7fac5f154700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fac5806bc30 con 0x7fac5810a630 2024-04-14T15:26:26.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.960+0000 7fac5f154700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fac581a8e20 con 0x7fac58107780 2024-04-14T15:26:26.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.960+0000 7fac5f154700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fac581aa000 con 0x7fac5810e200 2024-04-14T15:26:26.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.960+0000 7fac57fff700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fac5810a630 0x7fac5810d3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:46994/0 (socket says 172.21.15.90:46994) 2024-04-14T15:26:26.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.960+0000 7fac5cef0700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fac58107780 0x7fac5810cc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.90:34172/0 (socket says 172.21.15.90:34172) 2024-04-14T15:26:26.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.960+0000 7fac57fff700 1 -- 172.21.15.90:0/1927075071 learned_addr learned my addr 172.21.15.90:0/1927075071 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:26:26.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.960+0000 7fac55ffb700 10 client.?.objecter ms_handle_connect 0x7fac5810a630 2024-04-14T15:26:26.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.960+0000 7fac55ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:26:26.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.960+0000 7fac55ffb700 10 client.?.objecter ms_handle_connect 0x7fac5810e200 2024-04-14T15:26:26.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.960+0000 7fac55ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:26:26.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.960+0000 7fac55ffb700 10 client.?.objecter ms_handle_connect 0x7fac58107780 2024-04-14T15:26:26.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.960+0000 7fac55ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:26:26.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.960+0000 7fac55ffb700 1 -- 172.21.15.90:0/1927075071 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3884410399 0 0) 0x7fac5806bc30 con 0x7fac5810a630 2024-04-14T15:26:26.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.960+0000 7fac55ffb700 1 -- 172.21.15.90:0/1927075071 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fac480031f0 con 0x7fac5810a630 2024-04-14T15:26:26.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.960+0000 7fac55ffb700 1 -- 172.21.15.90:0/1927075071 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3925459578 0 0) 0x7fac581a8e20 con 0x7fac58107780 2024-04-14T15:26:26.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.960+0000 7fac55ffb700 1 -- 172.21.15.90:0/1927075071 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fac5806bc30 con 0x7fac58107780 2024-04-14T15:26:26.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.960+0000 7fac55ffb700 1 -- 172.21.15.90:0/1927075071 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3264135096 0 0) 0x7fac581aa000 con 0x7fac5810e200 2024-04-14T15:26:26.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.960+0000 7fac55ffb700 1 -- 172.21.15.90:0/1927075071 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fac581a8e20 con 0x7fac5810e200 2024-04-14T15:26:26.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.961+0000 7fac55ffb700 1 -- 172.21.15.90:0/1927075071 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4106615163 0 0) 0x7fac5806bc30 con 0x7fac58107780 2024-04-14T15:26:26.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.961+0000 7fac55ffb700 1 -- 172.21.15.90:0/1927075071 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fac581aa000 con 0x7fac58107780 2024-04-14T15:26:26.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.961+0000 7fac55ffb700 1 -- 172.21.15.90:0/1927075071 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3286460674 0 0) 0x7fac581a8e20 con 0x7fac5810e200 2024-04-14T15:26:26.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.961+0000 7fac55ffb700 1 -- 172.21.15.90:0/1927075071 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fac5806bc30 con 0x7fac5810e200 2024-04-14T15:26:26.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.961+0000 7fac55ffb700 1 -- 172.21.15.90:0/1927075071 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fac4c0015d0 con 0x7fac58107780 2024-04-14T15:26:26.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.961+0000 7fac55ffb700 1 -- 172.21.15.90:0/1927075071 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1494183339 0 0) 0x7fac480031f0 con 0x7fac5810a630 2024-04-14T15:26:26.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.961+0000 7fac55ffb700 1 -- 172.21.15.90:0/1927075071 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fac581a8e20 con 0x7fac5810a630 2024-04-14T15:26:26.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.961+0000 7fac55ffb700 1 -- 172.21.15.90:0/1927075071 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fac500013f0 con 0x7fac5810e200 2024-04-14T15:26:26.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.961+0000 7fac55ffb700 1 -- 172.21.15.90:0/1927075071 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fac44001870 con 0x7fac5810a630 2024-04-14T15:26:26.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.961+0000 7fac55ffb700 1 -- 172.21.15.90:0/1927075071 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1986986399 0 0) 0x7fac581aa000 con 0x7fac58107780 2024-04-14T15:26:26.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.961+0000 7fac55ffb700 1 -- 172.21.15.90:0/1927075071 >> v1:172.21.15.90:6790/0 conn(0x7fac5810e200 legacy=0x7fac581a66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:26.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.961+0000 7fac55ffb700 1 -- 172.21.15.90:0/1927075071 >> v1:172.21.15.90:6789/0 conn(0x7fac5810a630 legacy=0x7fac5810d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:26.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.961+0000 7fac55ffb700 1 -- 172.21.15.90:0/1927075071 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac581ab1e0 con 0x7fac58107780 2024-04-14T15:26:26.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.961+0000 7fac5f154700 1 -- 172.21.15.90:0/1927075071 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fac581a9050 con 0x7fac58107780 2024-04-14T15:26:26.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.961+0000 7fac55ffb700 1 -- 172.21.15.90:0/1927075071 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fac4c003b30 con 0x7fac58107780 2024-04-14T15:26:26.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.961+0000 7fac55ffb700 1 -- 172.21.15.90:0/1927075071 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fac4c005000 con 0x7fac58107780 2024-04-14T15:26:26.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.961+0000 7fac5f154700 10 client.49945.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:26:26.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.961+0000 7fac5f154700 1 -- 172.21.15.90:0/1927075071 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fac581a9510 con 0x7fac58107780 2024-04-14T15:26:26.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.961+0000 7fac5f154700 1 librados: init done 2024-04-14T15:26:26.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.962+0000 7fac5f154700 1 -- 172.21.15.90:0/1927075071 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac58102ec0 con 0x7fac58107780 2024-04-14T15:26:26.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.962+0000 7fac55ffb700 1 -- 172.21.15.90:0/1927075071 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fac4c001880 con 0x7fac58107780 2024-04-14T15:26:26.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.963+0000 7fac55ffb700 1 -- 172.21.15.90:0/1927075071 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(828..828 src has 467..828) ==== 8783+0+0 (unknown 2753859795 0 0) 0x7fac4c08ac70 con 0x7fac58107780 2024-04-14T15:26:26.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.963+0000 7fac55ffb700 10 client.49945.objecter ms_dispatch 0x7fac58001550 osd_map(828..828 src has 467..828) 2024-04-14T15:26:26.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.963+0000 7fac55ffb700 3 client.49945.objecter handle_osd_map got epochs [828,828] > 0 2024-04-14T15:26:26.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.963+0000 7fac55ffb700 3 client.49945.objecter handle_osd_map decoding full epoch 828 2024-04-14T15:26:26.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.963+0000 7fac55ffb700 20 client.49945.objecter dump_active .. 0 homeless 2024-04-14T15:26:26.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.963+0000 7fac55ffb700 10 client.49945.objecter ms_handle_connect 0x7fac48071460 2024-04-14T15:26:26.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:26.968+0000 7fac55ffb700 1 -- 172.21.15.90:0/1927075071 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fac4c057410 con 0x7fac58107780 2024-04-14T15:26:27.045 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:26:27.088 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:27.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:27.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:26:27.190 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:26:27.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:27.201+0000 7fac5f154700 1 -- 172.21.15.90:0/1927075071 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7fac580693d0 con 0x7fac58107780 2024-04-14T15:26:27.216 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:27.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:27.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:26:27.318 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:26:27.344 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:27.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:27.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:26:27.446 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:26:27.472 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:27.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:27.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:26:27.574 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:26:27.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:27.601 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:27.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:26:27.702 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:26:27.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:27.728 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:27.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:26:27.829 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:26:27.856 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:27.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:27.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:26:27.958 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:26:28.042 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:28.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:28.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:26:28.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:28.052+0000 7fac55ffb700 1 -- 172.21.15.90:0/1927075071 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v830) ==== 91+0+0 (unknown 636738092 0 0) 0x7fac4c05b0c0 con 0x7fac58107780 2024-04-14T15:26:28.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:28.055+0000 7fac5f154700 10 librados: watch_flush enter 2024-04-14T15:26:28.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:28.055+0000 7fac5f154700 10 librados: watch_flush exit 2024-04-14T15:26:28.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:28.055+0000 7fac5f154700 20 client.49945.objecter shutdown clearing up homeless session... 2024-04-14T15:26:28.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:28.055+0000 7fac5f154700 10 client.49945.objecter successfully canceled tick 2024-04-14T15:26:28.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:28.055+0000 7fac5f154700 1 -- 172.21.15.90:0/1927075071 >> v1:172.21.15.90:6800/420935427 conn(0x7fac48071460 legacy=0x7fac48073920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:28.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:28.055+0000 7fac5f154700 1 -- 172.21.15.90:0/1927075071 >> v1:172.21.15.92:6789/0 conn(0x7fac58107780 legacy=0x7fac5810cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:28.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:28.055+0000 7fac5f154700 1 -- 172.21.15.90:0/1927075071 shutdown_connections 2024-04-14T15:26:28.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:28.055+0000 7fac5f154700 1 -- 172.21.15.90:0/1927075071 >> 172.21.15.90:0/1927075071 conn(0x7fac580fd290 msgr2=0x7fac580fed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:26:28.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:28.055+0000 7fac5f154700 1 -- 172.21.15.90:0/1927075071 shutdown_connections 2024-04-14T15:26:28.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:28.056+0000 7fac5f154700 1 -- 172.21.15.90:0/1927075071 wait complete. 2024-04-14T15:26:28.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:28.056+0000 7fac5f154700 1 librados: shutdown 2024-04-14T15:26:28.057 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is unset 2024-04-14T15:26:28.144 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:26:28.167 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:28.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:28.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:26:28.174 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:28 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-146","var": "target_max_objects","val": "1"}]': finished 2024-04-14T15:26:28.174 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:28 smithi090 ceph-mon[41670]: osdmap e829: 8 total, 8 up, 7 in 2024-04-14T15:26:28.174 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:28 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1927075071' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:26:28.174 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:28 smithi090 ceph-mon[41670]: from='client.49945 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:26:28.174 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:28 smithi090 ceph-mon[41670]: pgmap v1235: 224 pgs: 224 active+clean; 4.4 MiB data, 738 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-04-14T15:26:28.174 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:28 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-146","var": "target_max_objects","val": "1"}]': finished 2024-04-14T15:26:28.174 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:28 smithi090 ceph-mon[30399]: osdmap e829: 8 total, 8 up, 7 in 2024-04-14T15:26:28.175 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:28 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1927075071' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:26:28.175 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:28 smithi090 ceph-mon[30399]: from='client.49945 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:26:28.175 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:28 smithi090 ceph-mon[30399]: pgmap v1235: 224 pgs: 224 active+clean; 4.4 MiB data, 738 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-04-14T15:26:28.269 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:26:28.297 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:28.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:28.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:26:28.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:28 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-146","var": "target_max_objects","val": "1"}]': finished 2024-04-14T15:26:28.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:28 smithi092 ceph-mon[33966]: osdmap e829: 8 total, 8 up, 7 in 2024-04-14T15:26:28.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:28 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1927075071' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:26:28.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:28 smithi092 ceph-mon[33966]: from='client.49945 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:26:28.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:28 smithi092 ceph-mon[33966]: pgmap v1235: 224 pgs: 224 active+clean; 4.4 MiB data, 738 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-04-14T15:26:28.399 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:26:28.423 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:28.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:28.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:26:28.525 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:26:28.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:28.549 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:28.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:26:28.650 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:26:28.680 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:28.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:28.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:26:28.782 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:26:28.807 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:28.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:28.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:26:28.909 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:26:28.936 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:28.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:28.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:26:29.038 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:26:29.062 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:29.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:29.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:26:29.164 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:26:29.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:29.188 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:29.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:26:29.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:29 smithi090 ceph-mon[41670]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-04-14T15:26:29.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:29 smithi090 ceph-mon[41670]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:26:29.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:29 smithi090 ceph-mon[41670]: from='client.49945 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-14T15:26:29.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:29 smithi090 ceph-mon[41670]: osdmap e830: 8 total, 8 up, 7 in 2024-04-14T15:26:29.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:29 smithi090 ceph-mon[30399]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-04-14T15:26:29.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:29 smithi090 ceph-mon[30399]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:26:29.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:29 smithi090 ceph-mon[30399]: from='client.49945 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-14T15:26:29.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:29 smithi090 ceph-mon[30399]: osdmap e830: 8 total, 8 up, 7 in 2024-04-14T15:26:29.290 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:26:29.318 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:29.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:29.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:26:29.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:29 smithi092 ceph-mon[33966]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-04-14T15:26:29.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:29 smithi092 ceph-mon[33966]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:26:29.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:29 smithi092 ceph-mon[33966]: from='client.49945 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-14T15:26:29.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:29 smithi092 ceph-mon[33966]: osdmap e830: 8 total, 8 up, 7 in 2024-04-14T15:26:29.420 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:26:29.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:29.445 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:29.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:26:29.546 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:26:29.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:29.574 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:29.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:26:29.675 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:26:29.702 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:29.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:29.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:26:29.804 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:26:29.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:29.829 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:29.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:26:29.930 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:26:29.958 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:29.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:29.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:26:30.059 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:26:30.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:30.087 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:30.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:26:30.188 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:26:30.214 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:30.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:30.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:26:30.284 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-14T15:26:30.284 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:26:30.297 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:30 smithi090 ceph-mon[41670]: 11.2 scrub starts 2024-04-14T15:26:30.297 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:30 smithi090 ceph-mon[41670]: 11.2 scrub ok 2024-04-14T15:26:30.297 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:30 smithi090 ceph-mon[41670]: 4.1e scrub starts 2024-04-14T15:26:30.297 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:30 smithi090 ceph-mon[41670]: 4.1e scrub ok 2024-04-14T15:26:30.297 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:30 smithi090 ceph-mon[41670]: pgmap v1237: 224 pgs: 224 active+clean; 4.4 MiB data, 738 MiB used, 625 GiB / 626 GiB avail; 3.4 KiB/s rd, 20 KiB/s wr, 4 op/s 2024-04-14T15:26:30.297 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:30 smithi090 ceph-mon[41670]: 2.a scrub starts 2024-04-14T15:26:30.298 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:30 smithi090 ceph-mon[41670]: 2.a scrub ok 2024-04-14T15:26:30.298 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:30 smithi090 ceph-mon[30399]: 11.2 scrub starts 2024-04-14T15:26:30.298 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:30 smithi090 ceph-mon[30399]: 11.2 scrub ok 2024-04-14T15:26:30.298 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:30 smithi090 ceph-mon[30399]: 4.1e scrub starts 2024-04-14T15:26:30.298 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:30 smithi090 ceph-mon[30399]: 4.1e scrub ok 2024-04-14T15:26:30.298 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:30 smithi090 ceph-mon[30399]: pgmap v1237: 224 pgs: 224 active+clean; 4.4 MiB data, 738 MiB used, 625 GiB / 626 GiB avail; 3.4 KiB/s rd, 20 KiB/s wr, 4 op/s 2024-04-14T15:26:30.298 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:30 smithi090 ceph-mon[30399]: 2.a scrub starts 2024-04-14T15:26:30.298 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:30 smithi090 ceph-mon[30399]: 2.a scrub ok 2024-04-14T15:26:30.316 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:26:30.339 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:30.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:30.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:26:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:30 smithi092 ceph-mon[33966]: 11.2 scrub starts 2024-04-14T15:26:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:30 smithi092 ceph-mon[33966]: 11.2 scrub ok 2024-04-14T15:26:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:30 smithi092 ceph-mon[33966]: 4.1e scrub starts 2024-04-14T15:26:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:30 smithi092 ceph-mon[33966]: 4.1e scrub ok 2024-04-14T15:26:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:30 smithi092 ceph-mon[33966]: pgmap v1237: 224 pgs: 224 active+clean; 4.4 MiB data, 738 MiB used, 625 GiB / 626 GiB avail; 3.4 KiB/s rd, 20 KiB/s wr, 4 op/s 2024-04-14T15:26:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:30 smithi092 ceph-mon[33966]: 2.a scrub starts 2024-04-14T15:26:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:30 smithi092 ceph-mon[33966]: 2.a scrub ok 2024-04-14T15:26:30.441 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:26:30.468 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:30.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:30.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:26:30.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.255+0000 7f630ebd7700 1 -- 172.21.15.90:0/685483223 >> v1:172.21.15.90:6789/0 conn(0x7f630810e200 legacy=0x7f63081106c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:30.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.255+0000 7f630ebd7700 1 -- 172.21.15.90:0/685483223 shutdown_connections 2024-04-14T15:26:30.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.255+0000 7f630ebd7700 1 -- 172.21.15.90:0/685483223 >> 172.21.15.90:0/685483223 conn(0x7f63080fd2b0 msgr2=0x7f63080ff6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:26:30.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.255+0000 7f630ebd7700 1 -- 172.21.15.90:0/685483223 shutdown_connections 2024-04-14T15:26:30.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.255+0000 7f630ebd7700 1 -- 172.21.15.90:0/685483223 wait complete. 2024-04-14T15:26:30.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.255+0000 7f630ebd7700 1 librados: starting msgr at 2024-04-14T15:26:30.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.255+0000 7f630ebd7700 1 librados: starting objecter 2024-04-14T15:26:30.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.255+0000 7f630ebd7700 1 Processor -- start 2024-04-14T15:26:30.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.255+0000 7f630ebd7700 1 -- start start 2024-04-14T15:26:30.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.255+0000 7f630ebd7700 1 librados: setting wanted keys 2024-04-14T15:26:30.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.255+0000 7f630ebd7700 1 librados: calling monclient init 2024-04-14T15:26:30.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.256+0000 7f630ebd7700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6308110540 con 0x7f6308107780 2024-04-14T15:26:30.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.256+0000 7f630ebd7700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f63081ab210 con 0x7f630810e200 2024-04-14T15:26:30.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.256+0000 7f630ebd7700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f63081ac3f0 con 0x7f630810a630 2024-04-14T15:26:30.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.256+0000 7f6307fff700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f630810a630 0x7f630810ee50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:52262/0 (socket says 172.21.15.90:52262) 2024-04-14T15:26:30.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.256+0000 7f630c973700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f6308107780 0x7f630810e740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:47012/0 (socket says 172.21.15.90:47012) 2024-04-14T15:26:30.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.256+0000 7f6307fff700 1 -- 172.21.15.90:0/2891079402 learned_addr learned my addr 172.21.15.90:0/2891079402 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:26:30.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.256+0000 7f6305ffb700 10 client.?.objecter ms_handle_connect 0x7f6308107780 2024-04-14T15:26:30.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.256+0000 7f6305ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:26:30.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.256+0000 7f6305ffb700 10 client.?.objecter ms_handle_connect 0x7f630810a630 2024-04-14T15:26:30.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.256+0000 7f6305ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:26:30.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.256+0000 7f6305ffb700 10 client.?.objecter ms_handle_connect 0x7f630810e200 2024-04-14T15:26:30.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.256+0000 7f6305ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:26:30.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.256+0000 7f6305ffb700 1 -- 172.21.15.90:0/2891079402 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1095700200 0 0) 0x7f6308110540 con 0x7f6308107780 2024-04-14T15:26:30.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.256+0000 7f6305ffb700 1 -- 172.21.15.90:0/2891079402 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f62f00035b0 con 0x7f6308107780 2024-04-14T15:26:30.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.256+0000 7f6305ffb700 1 -- 172.21.15.90:0/2891079402 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 726182901 0 0) 0x7f63081ac3f0 con 0x7f630810a630 2024-04-14T15:26:30.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.256+0000 7f6305ffb700 1 -- 172.21.15.90:0/2891079402 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6308110540 con 0x7f630810a630 2024-04-14T15:26:30.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.256+0000 7f6305ffb700 1 -- 172.21.15.90:0/2891079402 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4674868 0 0) 0x7f63081ab210 con 0x7f630810e200 2024-04-14T15:26:30.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.256+0000 7f6305ffb700 1 -- 172.21.15.90:0/2891079402 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f63081ac3f0 con 0x7f630810e200 2024-04-14T15:26:30.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.256+0000 7f6305ffb700 1 -- 172.21.15.90:0/2891079402 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 883127554 0 0) 0x7f6308110540 con 0x7f630810a630 2024-04-14T15:26:30.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.256+0000 7f6305ffb700 1 -- 172.21.15.90:0/2891079402 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f63081ab210 con 0x7f630810a630 2024-04-14T15:26:30.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.256+0000 7f6305ffb700 1 -- 172.21.15.90:0/2891079402 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f62fc0018b0 con 0x7f630810a630 2024-04-14T15:26:30.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.256+0000 7f6305ffb700 1 -- 172.21.15.90:0/2891079402 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2048275146 0 0) 0x7f62f00035b0 con 0x7f6308107780 2024-04-14T15:26:30.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.256+0000 7f6305ffb700 1 -- 172.21.15.90:0/2891079402 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6308110540 con 0x7f6308107780 2024-04-14T15:26:30.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.256+0000 7f6305ffb700 1 -- 172.21.15.90:0/2891079402 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f6300001bd0 con 0x7f6308107780 2024-04-14T15:26:30.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.256+0000 7f6305ffb700 1 -- 172.21.15.90:0/2891079402 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1134942076 0 0) 0x7f63081ac3f0 con 0x7f630810e200 2024-04-14T15:26:30.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.256+0000 7f6305ffb700 1 -- 172.21.15.90:0/2891079402 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f62f00035b0 con 0x7f630810e200 2024-04-14T15:26:30.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.256+0000 7f6305ffb700 1 -- 172.21.15.90:0/2891079402 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f62f8001430 con 0x7f630810e200 2024-04-14T15:26:30.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.256+0000 7f6305ffb700 1 -- 172.21.15.90:0/2891079402 <== mon.2 v1:172.21.15.90:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1840951398 0 0) 0x7f63081ab210 con 0x7f630810a630 2024-04-14T15:26:30.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.256+0000 7f6305ffb700 1 -- 172.21.15.90:0/2891079402 >> v1:172.21.15.92:6789/0 conn(0x7f630810e200 legacy=0x7f63081a8ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:30.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.257+0000 7f6305ffb700 1 -- 172.21.15.90:0/2891079402 >> v1:172.21.15.90:6789/0 conn(0x7f6308107780 legacy=0x7f630810e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:30.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.257+0000 7f6305ffb700 1 -- 172.21.15.90:0/2891079402 --> v1:172.21.15.90:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63081ad5d0 con 0x7f630810a630 2024-04-14T15:26:30.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.257+0000 7f630ebd7700 1 -- 172.21.15.90:0/2891079402 --> v1:172.21.15.90:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f63081ac620 con 0x7f630810a630 2024-04-14T15:26:30.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.257+0000 7f630ebd7700 10 client.48071.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:26:30.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.257+0000 7f6305ffb700 1 -- 172.21.15.90:0/2891079402 <== mon.2 v1:172.21.15.90:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f62fc001cc0 con 0x7f630810a630 2024-04-14T15:26:30.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.257+0000 7f6305ffb700 1 -- 172.21.15.90:0/2891079402 <== mon.2 v1:172.21.15.90:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f62fc005d20 con 0x7f630810a630 2024-04-14T15:26:30.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.257+0000 7f630ebd7700 1 -- 172.21.15.90:0/2891079402 --> v1:172.21.15.90:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f63081acae0 con 0x7f630810a630 2024-04-14T15:26:30.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.257+0000 7f630ebd7700 1 librados: init done 2024-04-14T15:26:30.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.257+0000 7f630ebd7700 1 -- 172.21.15.90:0/2891079402 --> v1:172.21.15.90:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f6308102ec0 con 0x7f630810a630 2024-04-14T15:26:30.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.258+0000 7f6305ffb700 1 -- 172.21.15.90:0/2891079402 <== mon.2 v1:172.21.15.90:6790/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f62fc003950 con 0x7f630810a630 2024-04-14T15:26:30.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.259+0000 7f6305ffb700 1 -- 172.21.15.90:0/2891079402 <== mon.2 v1:172.21.15.90:6790/0 8 ==== osd_map(830..830 src has 467..830) ==== 8783+0+0 (unknown 192911901 0 0) 0x7f62fc08acb0 con 0x7f630810a630 2024-04-14T15:26:30.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.259+0000 7f6305ffb700 10 client.48071.objecter ms_dispatch 0x7f6308001550 osd_map(830..830 src has 467..830) 2024-04-14T15:26:30.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.259+0000 7f6305ffb700 3 client.48071.objecter handle_osd_map got epochs [830,830] > 0 2024-04-14T15:26:30.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.259+0000 7f6305ffb700 3 client.48071.objecter handle_osd_map decoding full epoch 830 2024-04-14T15:26:30.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.259+0000 7f6305ffb700 20 client.48071.objecter dump_active .. 0 homeless 2024-04-14T15:26:30.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.259+0000 7f6305ffb700 1 -- 172.21.15.90:0/2891079402 <== mon.2 v1:172.21.15.90:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v830) ==== 54+0+16 (unknown 3095851118 0 2420000192) 0x7f62fc08d060 con 0x7f630810a630 2024-04-14T15:26:30.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.259+0000 7f6305ffb700 10 client.48071.objecter ms_handle_connect 0x7f62f0071300 2024-04-14T15:26:30.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.259+0000 7f630ebd7700 10 client.48071.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:26:30.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.259+0000 7f630ebd7700 15 client.48071.objecter _session_command_op_assign -1 1 2024-04-14T15:26:30.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.259+0000 7f630ebd7700 20 client.48071.objecter _get_session s=0x7f63081121d0 osd=0 3 2024-04-14T15:26:30.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.259+0000 7f630ebd7700 20 client.48071.objecter put_session s=0x7f63081121d0 osd=0 3 2024-04-14T15:26:30.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.259+0000 7f630ebd7700 20 client.48071.objecter _get_session s=0x7f63081121d0 osd=0 3 2024-04-14T15:26:30.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.259+0000 7f630ebd7700 15 client.48071.objecter _session_command_op_remove -1 1 2024-04-14T15:26:30.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.259+0000 7f630ebd7700 20 client.48071.objecter get_session s=0x7f63081121d0 osd=0 3 2024-04-14T15:26:30.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.259+0000 7f630ebd7700 15 client.48071.objecter _session_command_op_assign 0 1 2024-04-14T15:26:30.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.259+0000 7f630ebd7700 20 client.48071.objecter put_session s=0x7f63081121d0 osd=0 4 2024-04-14T15:26:30.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.259+0000 7f630ebd7700 10 client.48071.objecter _send_command 1 2024-04-14T15:26:30.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.259+0000 7f630ebd7700 1 -- 172.21.15.90:0/2891079402 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f630810fc70 con 0x7f63081ab3e0 2024-04-14T15:26:30.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.259+0000 7f6305ffb700 10 client.48071.objecter ms_handle_connect 0x7f63081ab3e0 2024-04-14T15:26:30.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.260+0000 7f6305ffb700 1 -- 172.21.15.90:0/2891079402 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f630810fc70 con 0x7f63081ab3e0 2024-04-14T15:26:30.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.260+0000 7f6305ffb700 10 client.48071.objecter ms_dispatch 0x7f6308001550 command_reply(tid 1: 0 ) 2024-04-14T15:26:30.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.260+0000 7f6305ffb700 10 client.48071.objecter _finish_command 1 = system:0 2024-04-14T15:26:30.488 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:26:30.488 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:26:30.488 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:26:30.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.260+0000 7f6305ffb700 20 client.48071.objecter put_session s=0x7f63081121d0 osd=0 4 2024-04-14T15:26:30.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.260+0000 7f6305ffb700 15 client.48071.objecter _session_command_op_remove 0 1 2024-04-14T15:26:30.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.277+0000 7f630ebd7700 10 client.48071.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:26:30.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.277+0000 7f630ebd7700 15 client.48071.objecter _session_command_op_assign -1 2 2024-04-14T15:26:30.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.277+0000 7f630ebd7700 20 client.48071.objecter _get_session s=0x7f63081121d0 osd=0 3 2024-04-14T15:26:30.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.277+0000 7f630ebd7700 20 client.48071.objecter put_session s=0x7f63081121d0 osd=0 3 2024-04-14T15:26:30.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.277+0000 7f630ebd7700 20 client.48071.objecter _get_session s=0x7f63081121d0 osd=0 3 2024-04-14T15:26:30.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.277+0000 7f630ebd7700 15 client.48071.objecter _session_command_op_remove -1 2 2024-04-14T15:26:30.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.277+0000 7f630ebd7700 20 client.48071.objecter get_session s=0x7f63081121d0 osd=0 3 2024-04-14T15:26:30.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.277+0000 7f630ebd7700 15 client.48071.objecter _session_command_op_assign 0 2 2024-04-14T15:26:30.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.277+0000 7f630ebd7700 20 client.48071.objecter put_session s=0x7f63081121d0 osd=0 4 2024-04-14T15:26:30.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.277+0000 7f630ebd7700 10 client.48071.objecter _send_command 2 2024-04-14T15:26:30.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.277+0000 7f630ebd7700 1 -- 172.21.15.90:0/2891079402 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f63080693d0 con 0x7f63081ab3e0 2024-04-14T15:26:30.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.278+0000 7f6305ffb700 1 -- 172.21.15.90:0/2891079402 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f63080693d0 con 0x7f63081ab3e0 2024-04-14T15:26:30.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.278+0000 7f6305ffb700 10 client.48071.objecter ms_dispatch 0x7f6308001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:26:30.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.278+0000 7f6305ffb700 10 client.48071.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:26:30.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.278+0000 7f6305ffb700 20 client.48071.objecter put_session s=0x7f63081121d0 osd=0 4 2024-04-14T15:26:30.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.278+0000 7f6305ffb700 15 client.48071.objecter _session_command_op_remove 0 2 2024-04-14T15:26:30.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.279+0000 7f630ebd7700 10 client.48071.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:26:30.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.279+0000 7f630ebd7700 15 client.48071.objecter _session_command_op_assign -1 3 2024-04-14T15:26:30.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.279+0000 7f630ebd7700 20 client.48071.objecter _get_session s=0x7f63081ac120 osd=1 3 2024-04-14T15:26:30.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.279+0000 7f630ebd7700 20 client.48071.objecter put_session s=0x7f63081ac120 osd=1 3 2024-04-14T15:26:30.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.279+0000 7f630ebd7700 20 client.48071.objecter _get_session s=0x7f63081ac120 osd=1 3 2024-04-14T15:26:30.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.279+0000 7f630ebd7700 15 client.48071.objecter _session_command_op_remove -1 3 2024-04-14T15:26:30.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.279+0000 7f630ebd7700 20 client.48071.objecter get_session s=0x7f63081ac120 osd=1 3 2024-04-14T15:26:30.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.279+0000 7f630ebd7700 15 client.48071.objecter _session_command_op_assign 1 3 2024-04-14T15:26:30.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.279+0000 7f630ebd7700 20 client.48071.objecter put_session s=0x7f63081ac120 osd=1 4 2024-04-14T15:26:30.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.279+0000 7f630ebd7700 10 client.48071.objecter _send_command 3 2024-04-14T15:26:30.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.279+0000 7f630ebd7700 1 -- 172.21.15.90:0/2891079402 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f630802de30 con 0x7f6308108b10 2024-04-14T15:26:30.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.279+0000 7f6305ffb700 10 client.48071.objecter ms_handle_connect 0x7f6308108b10 2024-04-14T15:26:30.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.280+0000 7f6305ffb700 1 -- 172.21.15.90:0/2891079402 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f630802de30 con 0x7f6308108b10 2024-04-14T15:26:30.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.280+0000 7f6305ffb700 10 client.48071.objecter ms_dispatch 0x7f6308001550 command_reply(tid 3: 0 ) 2024-04-14T15:26:30.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.280+0000 7f6305ffb700 10 client.48071.objecter _finish_command 3 = system:0 2024-04-14T15:26:30.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.280+0000 7f6305ffb700 20 client.48071.objecter put_session s=0x7f63081ac120 osd=1 4 2024-04-14T15:26:30.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.280+0000 7f6305ffb700 15 client.48071.objecter _session_command_op_remove 1 3 2024-04-14T15:26:30.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.296+0000 7f630ebd7700 10 client.48071.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:26:30.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.296+0000 7f630ebd7700 15 client.48071.objecter _session_command_op_assign -1 4 2024-04-14T15:26:30.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.296+0000 7f630ebd7700 20 client.48071.objecter _get_session s=0x7f63081ac120 osd=1 3 2024-04-14T15:26:30.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.296+0000 7f630ebd7700 20 client.48071.objecter put_session s=0x7f63081ac120 osd=1 3 2024-04-14T15:26:30.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.296+0000 7f630ebd7700 20 client.48071.objecter _get_session s=0x7f63081ac120 osd=1 3 2024-04-14T15:26:30.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.296+0000 7f630ebd7700 15 client.48071.objecter _session_command_op_remove -1 4 2024-04-14T15:26:30.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.296+0000 7f630ebd7700 20 client.48071.objecter get_session s=0x7f63081ac120 osd=1 3 2024-04-14T15:26:30.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.296+0000 7f630ebd7700 15 client.48071.objecter _session_command_op_assign 1 4 2024-04-14T15:26:30.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.296+0000 7f630ebd7700 20 client.48071.objecter put_session s=0x7f63081ac120 osd=1 4 2024-04-14T15:26:30.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.296+0000 7f630ebd7700 10 client.48071.objecter _send_command 4 2024-04-14T15:26:30.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.296+0000 7f630ebd7700 1 -- 172.21.15.90:0/2891079402 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f630802e260 con 0x7f6308108b10 2024-04-14T15:26:30.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.298+0000 7f6305ffb700 1 -- 172.21.15.90:0/2891079402 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f630802e260 con 0x7f6308108b10 2024-04-14T15:26:30.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.298+0000 7f6305ffb700 10 client.48071.objecter ms_dispatch 0x7f6308001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:26:30.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.298+0000 7f6305ffb700 10 client.48071.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:26:30.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.298+0000 7f6305ffb700 20 client.48071.objecter put_session s=0x7f63081ac120 osd=1 4 2024-04-14T15:26:30.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.298+0000 7f6305ffb700 15 client.48071.objecter _session_command_op_remove 1 4 2024-04-14T15:26:30.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.298+0000 7f630ebd7700 10 client.48071.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:26:30.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.298+0000 7f630ebd7700 15 client.48071.objecter _session_command_op_assign -1 5 2024-04-14T15:26:30.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.298+0000 7f630ebd7700 20 client.48071.objecter _get_session s=0x7f630802e410 osd=2 3 2024-04-14T15:26:30.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.298+0000 7f630ebd7700 20 client.48071.objecter put_session s=0x7f630802e410 osd=2 3 2024-04-14T15:26:30.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.298+0000 7f630ebd7700 20 client.48071.objecter _get_session s=0x7f630802e410 osd=2 3 2024-04-14T15:26:30.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.298+0000 7f630ebd7700 15 client.48071.objecter _session_command_op_remove -1 5 2024-04-14T15:26:30.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.298+0000 7f630ebd7700 20 client.48071.objecter get_session s=0x7f630802e410 osd=2 3 2024-04-14T15:26:30.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.298+0000 7f630ebd7700 15 client.48071.objecter _session_command_op_assign 2 5 2024-04-14T15:26:30.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.298+0000 7f630ebd7700 20 client.48071.objecter put_session s=0x7f630802e410 osd=2 4 2024-04-14T15:26:30.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.298+0000 7f630ebd7700 10 client.48071.objecter _send_command 5 2024-04-14T15:26:30.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.298+0000 7f630ebd7700 1 -- 172.21.15.90:0/2891079402 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f6308063b40 con 0x7f63080636c0 2024-04-14T15:26:30.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.299+0000 7f6305ffb700 10 client.48071.objecter ms_handle_connect 0x7f63080636c0 2024-04-14T15:26:30.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.299+0000 7f6305ffb700 1 -- 172.21.15.90:0/2891079402 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6308063b40 con 0x7f63080636c0 2024-04-14T15:26:30.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.299+0000 7f6305ffb700 10 client.48071.objecter ms_dispatch 0x7f6308001550 command_reply(tid 5: 0 ) 2024-04-14T15:26:30.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.299+0000 7f6305ffb700 10 client.48071.objecter _finish_command 5 = system:0 2024-04-14T15:26:30.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.299+0000 7f6305ffb700 20 client.48071.objecter put_session s=0x7f630802e410 osd=2 4 2024-04-14T15:26:30.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.299+0000 7f6305ffb700 15 client.48071.objecter _session_command_op_remove 2 5 2024-04-14T15:26:30.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.315+0000 7f630ebd7700 10 client.48071.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:26:30.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.315+0000 7f630ebd7700 15 client.48071.objecter _session_command_op_assign -1 6 2024-04-14T15:26:30.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.315+0000 7f630ebd7700 20 client.48071.objecter _get_session s=0x7f630802e410 osd=2 3 2024-04-14T15:26:30.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.315+0000 7f630ebd7700 20 client.48071.objecter put_session s=0x7f630802e410 osd=2 3 2024-04-14T15:26:30.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.315+0000 7f630ebd7700 20 client.48071.objecter _get_session s=0x7f630802e410 osd=2 3 2024-04-14T15:26:30.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.315+0000 7f630ebd7700 15 client.48071.objecter _session_command_op_remove -1 6 2024-04-14T15:26:30.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.315+0000 7f630ebd7700 20 client.48071.objecter get_session s=0x7f630802e410 osd=2 3 2024-04-14T15:26:30.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.315+0000 7f630ebd7700 15 client.48071.objecter _session_command_op_assign 2 6 2024-04-14T15:26:30.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.315+0000 7f630ebd7700 20 client.48071.objecter put_session s=0x7f630802e410 osd=2 4 2024-04-14T15:26:30.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.315+0000 7f630ebd7700 10 client.48071.objecter _send_command 6 2024-04-14T15:26:30.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.315+0000 7f630ebd7700 1 -- 172.21.15.90:0/2891079402 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f63081b60d0 con 0x7f63080636c0 2024-04-14T15:26:30.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.317+0000 7f6305ffb700 1 -- 172.21.15.90:0/2891079402 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f63081b60d0 con 0x7f63080636c0 2024-04-14T15:26:30.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.317+0000 7f6305ffb700 10 client.48071.objecter ms_dispatch 0x7f6308001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:26:30.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.317+0000 7f6305ffb700 10 client.48071.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:26:30.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.317+0000 7f6305ffb700 20 client.48071.objecter put_session s=0x7f630802e410 osd=2 4 2024-04-14T15:26:30.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.317+0000 7f6305ffb700 15 client.48071.objecter _session_command_op_remove 2 6 2024-04-14T15:26:30.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.317+0000 7f630ebd7700 10 client.48071.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:26:30.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.317+0000 7f630ebd7700 15 client.48071.objecter _session_command_op_assign -1 7 2024-04-14T15:26:30.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.317+0000 7f630ebd7700 20 client.48071.objecter _get_session s=0x7f6308063140 osd=3 3 2024-04-14T15:26:30.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.317+0000 7f630ebd7700 20 client.48071.objecter put_session s=0x7f6308063140 osd=3 3 2024-04-14T15:26:30.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.317+0000 7f630ebd7700 20 client.48071.objecter _get_session s=0x7f6308063140 osd=3 3 2024-04-14T15:26:30.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.317+0000 7f630ebd7700 15 client.48071.objecter _session_command_op_remove -1 7 2024-04-14T15:26:30.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.317+0000 7f630ebd7700 20 client.48071.objecter get_session s=0x7f6308063140 osd=3 3 2024-04-14T15:26:30.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.317+0000 7f630ebd7700 15 client.48071.objecter _session_command_op_assign 3 7 2024-04-14T15:26:30.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.317+0000 7f630ebd7700 20 client.48071.objecter put_session s=0x7f6308063140 osd=3 4 2024-04-14T15:26:30.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.317+0000 7f630ebd7700 10 client.48071.objecter _send_command 7 2024-04-14T15:26:30.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.317+0000 7f630ebd7700 1 -- 172.21.15.90:0/2891079402 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f63081b2500 con 0x7f63080632b0 2024-04-14T15:26:30.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.318+0000 7f6305ffb700 10 client.48071.objecter ms_handle_connect 0x7f63080632b0 2024-04-14T15:26:30.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.318+0000 7f6305ffb700 1 -- 172.21.15.90:0/2891079402 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f63081b2500 con 0x7f63080632b0 2024-04-14T15:26:30.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.318+0000 7f6305ffb700 10 client.48071.objecter ms_dispatch 0x7f6308001550 command_reply(tid 7: 0 ) 2024-04-14T15:26:30.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.318+0000 7f6305ffb700 10 client.48071.objecter _finish_command 7 = system:0 2024-04-14T15:26:30.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.318+0000 7f6305ffb700 20 client.48071.objecter put_session s=0x7f6308063140 osd=3 4 2024-04-14T15:26:30.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.318+0000 7f6305ffb700 15 client.48071.objecter _session_command_op_remove 3 7 2024-04-14T15:26:30.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.336+0000 7f630ebd7700 10 client.48071.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:26:30.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.336+0000 7f630ebd7700 15 client.48071.objecter _session_command_op_assign -1 8 2024-04-14T15:26:30.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.336+0000 7f630ebd7700 20 client.48071.objecter _get_session s=0x7f6308063140 osd=3 3 2024-04-14T15:26:30.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.336+0000 7f630ebd7700 20 client.48071.objecter put_session s=0x7f6308063140 osd=3 3 2024-04-14T15:26:30.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.336+0000 7f630ebd7700 20 client.48071.objecter _get_session s=0x7f6308063140 osd=3 3 2024-04-14T15:26:30.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.336+0000 7f630ebd7700 15 client.48071.objecter _session_command_op_remove -1 8 2024-04-14T15:26:30.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.336+0000 7f630ebd7700 20 client.48071.objecter get_session s=0x7f6308063140 osd=3 3 2024-04-14T15:26:30.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.336+0000 7f630ebd7700 15 client.48071.objecter _session_command_op_assign 3 8 2024-04-14T15:26:30.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.336+0000 7f630ebd7700 20 client.48071.objecter put_session s=0x7f6308063140 osd=3 4 2024-04-14T15:26:30.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.336+0000 7f630ebd7700 10 client.48071.objecter _send_command 8 2024-04-14T15:26:30.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.336+0000 7f630ebd7700 1 -- 172.21.15.90:0/2891079402 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f63081b27a0 con 0x7f63080632b0 2024-04-14T15:26:30.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.338+0000 7f6305ffb700 1 -- 172.21.15.90:0/2891079402 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f63081b27a0 con 0x7f63080632b0 2024-04-14T15:26:30.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.338+0000 7f6305ffb700 10 client.48071.objecter ms_dispatch 0x7f6308001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:26:30.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.338+0000 7f6305ffb700 10 client.48071.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:26:30.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.338+0000 7f6305ffb700 20 client.48071.objecter put_session s=0x7f6308063140 osd=3 4 2024-04-14T15:26:30.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.338+0000 7f6305ffb700 15 client.48071.objecter _session_command_op_remove 3 8 2024-04-14T15:26:30.499 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:26:30.500 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:26:30.500 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:26:30.500 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:26:30.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.338+0000 7f630ebd7700 10 client.48071.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:26:30.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.338+0000 7f630ebd7700 15 client.48071.objecter _session_command_op_assign -1 9 2024-04-14T15:26:30.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.338+0000 7f630ebd7700 20 client.48071.objecter _get_session s=0x7f63081b27a0 osd=4 3 2024-04-14T15:26:30.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.338+0000 7f630ebd7700 20 client.48071.objecter put_session s=0x7f63081b27a0 osd=4 3 2024-04-14T15:26:30.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.338+0000 7f630ebd7700 20 client.48071.objecter _get_session s=0x7f63081b27a0 osd=4 3 2024-04-14T15:26:30.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.338+0000 7f630ebd7700 15 client.48071.objecter _session_command_op_remove -1 9 2024-04-14T15:26:30.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.338+0000 7f630ebd7700 20 client.48071.objecter get_session s=0x7f63081b27a0 osd=4 3 2024-04-14T15:26:30.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.338+0000 7f630ebd7700 15 client.48071.objecter _session_command_op_assign 4 9 2024-04-14T15:26:30.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.338+0000 7f630ebd7700 20 client.48071.objecter put_session s=0x7f63081b27a0 osd=4 4 2024-04-14T15:26:30.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.338+0000 7f630ebd7700 10 client.48071.objecter _send_command 9 2024-04-14T15:26:30.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.338+0000 7f630ebd7700 1 -- 172.21.15.90:0/2891079402 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f63081b7060 con 0x7f63081b6a50 2024-04-14T15:26:30.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.338+0000 7f6305ffb700 10 client.48071.objecter ms_handle_connect 0x7f63081b6a50 2024-04-14T15:26:30.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.339+0000 7f6305ffb700 1 -- 172.21.15.90:0/2891079402 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f63081b7060 con 0x7f63081b6a50 2024-04-14T15:26:30.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.339+0000 7f6305ffb700 10 client.48071.objecter ms_dispatch 0x7f6308001550 command_reply(tid 9: 0 ) 2024-04-14T15:26:30.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.339+0000 7f6305ffb700 10 client.48071.objecter _finish_command 9 = system:0 2024-04-14T15:26:30.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.339+0000 7f6305ffb700 20 client.48071.objecter put_session s=0x7f63081b27a0 osd=4 4 2024-04-14T15:26:30.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.339+0000 7f6305ffb700 15 client.48071.objecter _session_command_op_remove 4 9 2024-04-14T15:26:30.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.357+0000 7f630ebd7700 10 client.48071.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:26:30.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.357+0000 7f630ebd7700 15 client.48071.objecter _session_command_op_assign -1 10 2024-04-14T15:26:30.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.357+0000 7f630ebd7700 20 client.48071.objecter _get_session s=0x7f63081b27a0 osd=4 3 2024-04-14T15:26:30.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.357+0000 7f630ebd7700 20 client.48071.objecter put_session s=0x7f63081b27a0 osd=4 3 2024-04-14T15:26:30.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.357+0000 7f630ebd7700 20 client.48071.objecter _get_session s=0x7f63081b27a0 osd=4 3 2024-04-14T15:26:30.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.357+0000 7f630ebd7700 15 client.48071.objecter _session_command_op_remove -1 10 2024-04-14T15:26:30.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.357+0000 7f630ebd7700 20 client.48071.objecter get_session s=0x7f63081b27a0 osd=4 3 2024-04-14T15:26:30.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.357+0000 7f630ebd7700 15 client.48071.objecter _session_command_op_assign 4 10 2024-04-14T15:26:30.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.357+0000 7f630ebd7700 20 client.48071.objecter put_session s=0x7f63081b27a0 osd=4 4 2024-04-14T15:26:30.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.357+0000 7f630ebd7700 10 client.48071.objecter _send_command 10 2024-04-14T15:26:30.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.357+0000 7f630ebd7700 1 -- 172.21.15.90:0/2891079402 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f63081b7020 con 0x7f63081b6a50 2024-04-14T15:26:30.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.359+0000 7f6305ffb700 1 -- 172.21.15.90:0/2891079402 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f63081b7020 con 0x7f63081b6a50 2024-04-14T15:26:30.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.359+0000 7f6305ffb700 10 client.48071.objecter ms_dispatch 0x7f6308001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:26:30.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.359+0000 7f6305ffb700 10 client.48071.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:26:30.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.359+0000 7f6305ffb700 20 client.48071.objecter put_session s=0x7f63081b27a0 osd=4 4 2024-04-14T15:26:30.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.359+0000 7f6305ffb700 15 client.48071.objecter _session_command_op_remove 4 10 2024-04-14T15:26:30.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.359+0000 7f630ebd7700 10 client.48071.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:26:30.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.359+0000 7f630ebd7700 15 client.48071.objecter _session_command_op_assign -1 11 2024-04-14T15:26:30.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.359+0000 7f630ebd7700 20 client.48071.objecter _get_session s=0x7f63081b7020 osd=5 3 2024-04-14T15:26:30.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.359+0000 7f630ebd7700 20 client.48071.objecter put_session s=0x7f63081b7020 osd=5 3 2024-04-14T15:26:30.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.359+0000 7f630ebd7700 20 client.48071.objecter _get_session s=0x7f63081b7020 osd=5 3 2024-04-14T15:26:30.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.359+0000 7f630ebd7700 15 client.48071.objecter _session_command_op_remove -1 11 2024-04-14T15:26:30.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.359+0000 7f630ebd7700 20 client.48071.objecter get_session s=0x7f63081b7020 osd=5 3 2024-04-14T15:26:30.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.359+0000 7f630ebd7700 15 client.48071.objecter _session_command_op_assign 5 11 2024-04-14T15:26:30.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.359+0000 7f630ebd7700 20 client.48071.objecter put_session s=0x7f63081b7020 osd=5 4 2024-04-14T15:26:30.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.359+0000 7f630ebd7700 10 client.48071.objecter _send_command 11 2024-04-14T15:26:30.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.359+0000 7f630ebd7700 1 -- 172.21.15.90:0/2891079402 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f63081c3540 con 0x7f63081bf8f0 2024-04-14T15:26:30.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.359+0000 7f6305ffb700 10 client.48071.objecter ms_handle_connect 0x7f63081bf8f0 2024-04-14T15:26:30.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.360+0000 7f6305ffb700 1 -- 172.21.15.90:0/2891079402 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f63081c3540 con 0x7f63081bf8f0 2024-04-14T15:26:30.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.360+0000 7f6305ffb700 10 client.48071.objecter ms_dispatch 0x7f6308001550 command_reply(tid 11: 0 ) 2024-04-14T15:26:30.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.360+0000 7f6305ffb700 10 client.48071.objecter _finish_command 11 = system:0 2024-04-14T15:26:30.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.360+0000 7f6305ffb700 20 client.48071.objecter put_session s=0x7f63081b7020 osd=5 4 2024-04-14T15:26:30.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.360+0000 7f6305ffb700 15 client.48071.objecter _session_command_op_remove 5 11 2024-04-14T15:26:30.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.377+0000 7f630ebd7700 10 client.48071.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:26:30.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.377+0000 7f630ebd7700 15 client.48071.objecter _session_command_op_assign -1 12 2024-04-14T15:26:30.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.377+0000 7f630ebd7700 20 client.48071.objecter _get_session s=0x7f63081b7020 osd=5 3 2024-04-14T15:26:30.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.377+0000 7f630ebd7700 20 client.48071.objecter put_session s=0x7f63081b7020 osd=5 3 2024-04-14T15:26:30.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.377+0000 7f630ebd7700 20 client.48071.objecter _get_session s=0x7f63081b7020 osd=5 3 2024-04-14T15:26:30.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.377+0000 7f630ebd7700 15 client.48071.objecter _session_command_op_remove -1 12 2024-04-14T15:26:30.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.377+0000 7f630ebd7700 20 client.48071.objecter get_session s=0x7f63081b7020 osd=5 3 2024-04-14T15:26:30.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.377+0000 7f630ebd7700 15 client.48071.objecter _session_command_op_assign 5 12 2024-04-14T15:26:30.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.377+0000 7f630ebd7700 20 client.48071.objecter put_session s=0x7f63081b7020 osd=5 4 2024-04-14T15:26:30.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.377+0000 7f630ebd7700 10 client.48071.objecter _send_command 12 2024-04-14T15:26:30.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.377+0000 7f630ebd7700 1 -- 172.21.15.90:0/2891079402 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f63081b1880 con 0x7f63081bf8f0 2024-04-14T15:26:30.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.379+0000 7f6305ffb700 1 -- 172.21.15.90:0/2891079402 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f63081b1880 con 0x7f63081bf8f0 2024-04-14T15:26:30.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.379+0000 7f6305ffb700 10 client.48071.objecter ms_dispatch 0x7f6308001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:26:30.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.379+0000 7f6305ffb700 10 client.48071.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:26:30.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.379+0000 7f6305ffb700 20 client.48071.objecter put_session s=0x7f63081b7020 osd=5 4 2024-04-14T15:26:30.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.379+0000 7f6305ffb700 15 client.48071.objecter _session_command_op_remove 5 12 2024-04-14T15:26:30.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.379+0000 7f630ebd7700 10 client.48071.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:26:30.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.379+0000 7f630ebd7700 15 client.48071.objecter _session_command_op_assign -1 13 2024-04-14T15:26:30.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.379+0000 7f630ebd7700 20 client.48071.objecter _get_session s=0x7f63081b1880 osd=6 3 2024-04-14T15:26:30.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.379+0000 7f630ebd7700 20 client.48071.objecter put_session s=0x7f63081b1880 osd=6 3 2024-04-14T15:26:30.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.379+0000 7f630ebd7700 20 client.48071.objecter _get_session s=0x7f63081b1880 osd=6 3 2024-04-14T15:26:30.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.379+0000 7f630ebd7700 15 client.48071.objecter _session_command_op_remove -1 13 2024-04-14T15:26:30.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.379+0000 7f630ebd7700 20 client.48071.objecter get_session s=0x7f63081b1880 osd=6 3 2024-04-14T15:26:30.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.379+0000 7f630ebd7700 15 client.48071.objecter _session_command_op_assign 6 13 2024-04-14T15:26:30.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.379+0000 7f630ebd7700 20 client.48071.objecter put_session s=0x7f63081b1880 osd=6 4 2024-04-14T15:26:30.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.379+0000 7f630ebd7700 10 client.48071.objecter _send_command 13 2024-04-14T15:26:30.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.379+0000 7f630ebd7700 1 -- 172.21.15.90:0/2891079402 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f63081c4cd0 con 0x7f63081bb300 2024-04-14T15:26:30.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.379+0000 7f6305ffb700 10 client.48071.objecter ms_handle_connect 0x7f63081bb300 2024-04-14T15:26:30.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.380+0000 7f6305ffb700 1 -- 172.21.15.90:0/2891079402 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f63081c4cd0 con 0x7f63081bb300 2024-04-14T15:26:30.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.380+0000 7f6305ffb700 10 client.48071.objecter ms_dispatch 0x7f6308001550 command_reply(tid 13: 0 ) 2024-04-14T15:26:30.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.380+0000 7f6305ffb700 10 client.48071.objecter _finish_command 13 = system:0 2024-04-14T15:26:30.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.380+0000 7f6305ffb700 20 client.48071.objecter put_session s=0x7f63081b1880 osd=6 4 2024-04-14T15:26:30.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.380+0000 7f6305ffb700 15 client.48071.objecter _session_command_op_remove 6 13 2024-04-14T15:26:30.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.397+0000 7f630ebd7700 10 client.48071.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:26:30.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.397+0000 7f630ebd7700 15 client.48071.objecter _session_command_op_assign -1 14 2024-04-14T15:26:30.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.397+0000 7f630ebd7700 20 client.48071.objecter _get_session s=0x7f63081b1880 osd=6 3 2024-04-14T15:26:30.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.397+0000 7f630ebd7700 20 client.48071.objecter put_session s=0x7f63081b1880 osd=6 3 2024-04-14T15:26:30.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.397+0000 7f630ebd7700 20 client.48071.objecter _get_session s=0x7f63081b1880 osd=6 3 2024-04-14T15:26:30.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.397+0000 7f630ebd7700 15 client.48071.objecter _session_command_op_remove -1 14 2024-04-14T15:26:30.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.397+0000 7f630ebd7700 20 client.48071.objecter get_session s=0x7f63081b1880 osd=6 3 2024-04-14T15:26:30.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.397+0000 7f630ebd7700 15 client.48071.objecter _session_command_op_assign 6 14 2024-04-14T15:26:30.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.397+0000 7f630ebd7700 20 client.48071.objecter put_session s=0x7f63081b1880 osd=6 4 2024-04-14T15:26:30.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.397+0000 7f630ebd7700 10 client.48071.objecter _send_command 14 2024-04-14T15:26:30.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.397+0000 7f630ebd7700 1 -- 172.21.15.90:0/2891079402 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f63081bbad0 con 0x7f63081bb300 2024-04-14T15:26:30.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.399+0000 7f6305ffb700 1 -- 172.21.15.90:0/2891079402 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f63081bbad0 con 0x7f63081bb300 2024-04-14T15:26:30.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.399+0000 7f6305ffb700 10 client.48071.objecter ms_dispatch 0x7f6308001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:26:30.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.399+0000 7f6305ffb700 10 client.48071.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:26:30.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.399+0000 7f6305ffb700 20 client.48071.objecter put_session s=0x7f63081b1880 osd=6 4 2024-04-14T15:26:30.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.399+0000 7f6305ffb700 15 client.48071.objecter _session_command_op_remove 6 14 2024-04-14T15:26:30.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.399+0000 7f630ebd7700 10 client.48071.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:26:30.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.399+0000 7f630ebd7700 15 client.48071.objecter _session_command_op_assign -1 15 2024-04-14T15:26:30.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.399+0000 7f630ebd7700 20 client.48071.objecter _get_session s=0x7f63081b63a0 osd=7 3 2024-04-14T15:26:30.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.399+0000 7f630ebd7700 20 client.48071.objecter put_session s=0x7f63081b63a0 osd=7 3 2024-04-14T15:26:30.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.399+0000 7f630ebd7700 20 client.48071.objecter _get_session s=0x7f63081b63a0 osd=7 3 2024-04-14T15:26:30.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.399+0000 7f630ebd7700 15 client.48071.objecter _session_command_op_remove -1 15 2024-04-14T15:26:30.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.399+0000 7f630ebd7700 20 client.48071.objecter get_session s=0x7f63081b63a0 osd=7 3 2024-04-14T15:26:30.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.399+0000 7f630ebd7700 15 client.48071.objecter _session_command_op_assign 7 15 2024-04-14T15:26:30.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.399+0000 7f630ebd7700 20 client.48071.objecter put_session s=0x7f63081b63a0 osd=7 4 2024-04-14T15:26:30.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.399+0000 7f630ebd7700 10 client.48071.objecter _send_command 15 2024-04-14T15:26:30.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.399+0000 7f630ebd7700 1 -- 172.21.15.90:0/2891079402 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f63081c5e90 con 0x7f63081c85e0 2024-04-14T15:26:30.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.399+0000 7f6305ffb700 10 client.48071.objecter ms_handle_connect 0x7f63081c85e0 2024-04-14T15:26:30.508 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:26:30.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.400+0000 7f6305ffb700 1 -- 172.21.15.90:0/2891079402 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f63081c5e90 con 0x7f63081c85e0 2024-04-14T15:26:30.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.400+0000 7f6305ffb700 10 client.48071.objecter ms_dispatch 0x7f6308001550 command_reply(tid 15: 0 ) 2024-04-14T15:26:30.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.400+0000 7f6305ffb700 10 client.48071.objecter _finish_command 15 = system:0 2024-04-14T15:26:30.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.400+0000 7f6305ffb700 20 client.48071.objecter put_session s=0x7f63081b63a0 osd=7 4 2024-04-14T15:26:30.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.400+0000 7f6305ffb700 15 client.48071.objecter _session_command_op_remove 7 15 2024-04-14T15:26:30.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.416+0000 7f630ebd7700 10 client.48071.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:26:30.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.417+0000 7f630ebd7700 15 client.48071.objecter _session_command_op_assign -1 16 2024-04-14T15:26:30.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.417+0000 7f630ebd7700 20 client.48071.objecter _get_session s=0x7f63081b63a0 osd=7 3 2024-04-14T15:26:30.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.417+0000 7f630ebd7700 20 client.48071.objecter put_session s=0x7f63081b63a0 osd=7 3 2024-04-14T15:26:30.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.417+0000 7f630ebd7700 20 client.48071.objecter _get_session s=0x7f63081b63a0 osd=7 3 2024-04-14T15:26:30.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.417+0000 7f630ebd7700 15 client.48071.objecter _session_command_op_remove -1 16 2024-04-14T15:26:30.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.417+0000 7f630ebd7700 20 client.48071.objecter get_session s=0x7f63081b63a0 osd=7 3 2024-04-14T15:26:30.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.417+0000 7f630ebd7700 15 client.48071.objecter _session_command_op_assign 7 16 2024-04-14T15:26:30.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.417+0000 7f630ebd7700 20 client.48071.objecter put_session s=0x7f63081b63a0 osd=7 4 2024-04-14T15:26:30.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.417+0000 7f630ebd7700 10 client.48071.objecter _send_command 16 2024-04-14T15:26:30.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.417+0000 7f630ebd7700 1 -- 172.21.15.90:0/2891079402 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f63081c5e90 con 0x7f63081c85e0 2024-04-14T15:26:30.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.419+0000 7f6305ffb700 1 -- 172.21.15.90:0/2891079402 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f63081c5e90 con 0x7f63081c85e0 2024-04-14T15:26:30.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.419+0000 7f6305ffb700 10 client.48071.objecter ms_dispatch 0x7f6308001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:26:30.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.419+0000 7f6305ffb700 10 client.48071.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:26:30.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.419+0000 7f6305ffb700 20 client.48071.objecter put_session s=0x7f63081b63a0 osd=7 4 2024-04-14T15:26:30.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.419+0000 7f6305ffb700 15 client.48071.objecter _session_command_op_remove 7 16 2024-04-14T15:26:30.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.419+0000 7f630ebd7700 10 librados: watch_flush enter 2024-04-14T15:26:30.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.419+0000 7f630ebd7700 10 librados: watch_flush exit 2024-04-14T15:26:30.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.419+0000 7f630ebd7700 10 client.48071.objecter close_session for osd.0 2024-04-14T15:26:30.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.419+0000 7f630ebd7700 1 -- 172.21.15.90:0/2891079402 >> v1:172.21.15.90:6801/3761806885 conn(0x7f63081ab3e0 legacy=0x7f63081ab7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:30.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.419+0000 7f630ebd7700 20 client.48071.objecter put_session s=0x7f63081121d0 osd=0 1 2024-04-14T15:26:30.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.419+0000 7f630ebd7700 10 client.48071.objecter close_session for osd.1 2024-04-14T15:26:30.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.419+0000 7f630ebd7700 1 -- 172.21.15.90:0/2891079402 >> v1:172.21.15.90:6805/4170361699 conn(0x7f6308108b10 legacy=0x7f6308108f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:30.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.419+0000 7f630ebd7700 20 client.48071.objecter put_session s=0x7f63081ac120 osd=1 1 2024-04-14T15:26:30.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.419+0000 7f630ebd7700 10 client.48071.objecter close_session for osd.2 2024-04-14T15:26:30.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.419+0000 7f630ebd7700 1 -- 172.21.15.90:0/2891079402 >> v1:172.21.15.90:6809/1686184251 conn(0x7f63080636c0 legacy=0x7f63081b49b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:30.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.419+0000 7f630ebd7700 20 client.48071.objecter put_session s=0x7f630802e410 osd=2 1 2024-04-14T15:26:30.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.419+0000 7f630ebd7700 10 client.48071.objecter close_session for osd.3 2024-04-14T15:26:30.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.419+0000 7f630ebd7700 1 -- 172.21.15.90:0/2891079402 >> v1:172.21.15.90:6813/2270166774 conn(0x7f63080632b0 legacy=0x7f63081b9410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:30.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.419+0000 7f630ebd7700 20 client.48071.objecter put_session s=0x7f6308063140 osd=3 1 2024-04-14T15:26:30.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.419+0000 7f630ebd7700 10 client.48071.objecter close_session for osd.4 2024-04-14T15:26:30.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.419+0000 7f630ebd7700 1 -- 172.21.15.90:0/2891079402 >> v1:172.21.15.92:6800/3887549733 conn(0x7f63081b6a50 legacy=0x7f63081bdcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:30.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.419+0000 7f630ebd7700 20 client.48071.objecter put_session s=0x7f63081b27a0 osd=4 1 2024-04-14T15:26:30.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.419+0000 7f630ebd7700 10 client.48071.objecter close_session for osd.5 2024-04-14T15:26:30.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.419+0000 7f630ebd7700 1 -- 172.21.15.90:0/2891079402 >> v1:172.21.15.92:6804/3244561440 conn(0x7f63081bf8f0 legacy=0x7f63081c1ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:30.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.419+0000 7f630ebd7700 20 client.48071.objecter put_session s=0x7f63081b7020 osd=5 1 2024-04-14T15:26:30.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.419+0000 7f630ebd7700 10 client.48071.objecter close_session for osd.6 2024-04-14T15:26:30.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.419+0000 7f630ebd7700 1 -- 172.21.15.90:0/2891079402 >> v1:172.21.15.92:6808/636125963 conn(0x7f63081bb300 legacy=0x7f63081c3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:30.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.419+0000 7f630ebd7700 20 client.48071.objecter put_session s=0x7f63081b1880 osd=6 1 2024-04-14T15:26:30.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.419+0000 7f630ebd7700 10 client.48071.objecter close_session for osd.7 2024-04-14T15:26:30.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.419+0000 7f630ebd7700 1 -- 172.21.15.90:0/2891079402 >> v1:172.21.15.92:6812/1773706716 conn(0x7f63081c85e0 legacy=0x7f63081ca9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:30.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.419+0000 7f630ebd7700 20 client.48071.objecter put_session s=0x7f63081b63a0 osd=7 1 2024-04-14T15:26:30.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.419+0000 7f630ebd7700 20 client.48071.objecter shutdown clearing up homeless session... 2024-04-14T15:26:30.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.419+0000 7f630ebd7700 10 client.48071.objecter successfully canceled tick 2024-04-14T15:26:30.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.419+0000 7f630d174700 1 -- 172.21.15.90:0/2891079402 reap_dead start 2024-04-14T15:26:30.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.419+0000 7f630d174700 1 -- 172.21.15.90:0/2891079402 reap_dead start 2024-04-14T15:26:30.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.419+0000 7f630d174700 1 -- 172.21.15.90:0/2891079402 reap_dead start 2024-04-14T15:26:30.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.419+0000 7f630d174700 1 -- 172.21.15.90:0/2891079402 reap_dead start 2024-04-14T15:26:30.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.419+0000 7f630d174700 1 -- 172.21.15.90:0/2891079402 reap_dead start 2024-04-14T15:26:30.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.419+0000 7f630ebd7700 1 -- 172.21.15.90:0/2891079402 >> v1:172.21.15.90:6800/420935427 conn(0x7f62f0071300 legacy=0x7f62f00737c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:30.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.419+0000 7f630ebd7700 1 -- 172.21.15.90:0/2891079402 >> v1:172.21.15.90:6790/0 conn(0x7f630810a630 legacy=0x7f630810ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:30.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.419+0000 7f630ebd7700 1 -- 172.21.15.90:0/2891079402 shutdown_connections 2024-04-14T15:26:30.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.419+0000 7f630ebd7700 1 -- 172.21.15.90:0/2891079402 >> 172.21.15.90:0/2891079402 conn(0x7f63080fd2b0 msgr2=0x7f630806af30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:26:30.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.419+0000 7f630ebd7700 1 -- 172.21.15.90:0/2891079402 shutdown_connections 2024-04-14T15:26:30.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.419+0000 7f630ebd7700 1 -- 172.21.15.90:0/2891079402 wait complete. 2024-04-14T15:26:30.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:30.419+0000 7f630ebd7700 1 librados: shutdown 2024-04-14T15:26:30.513 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:26:30.513 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:26:30.513 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:26:30.513 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:26:30.513 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:26:30.513 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:26:30.513 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:26:30.513 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:26:30.570 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:26:30.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:30.594 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:30.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:26:30.697 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:26:30.721 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:30.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:30.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:26:30.823 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:26:30.846 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:30.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:30.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:26:30.933 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-14T15:26:30.948 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:26:30.976 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:30.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:30.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:26:31.078 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:26:31.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:31.106 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:31.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:26:31.207 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:26:31.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:31.231 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:31.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:26:31.332 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:26:31.359 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:31.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:31.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:26:31.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:31 smithi092 ceph-mon[33966]: 11.15 scrub starts 2024-04-14T15:26:31.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:31 smithi092 ceph-mon[33966]: 11.15 scrub ok 2024-04-14T15:26:31.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:31 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2891079402' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:26:31.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:31 smithi090 ceph-mon[41670]: 11.15 scrub starts 2024-04-14T15:26:31.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:31 smithi090 ceph-mon[41670]: 11.15 scrub ok 2024-04-14T15:26:31.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:31 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2891079402' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:26:31.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:31 smithi090 ceph-mon[30399]: 11.15 scrub starts 2024-04-14T15:26:31.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:31 smithi090 ceph-mon[30399]: 11.15 scrub ok 2024-04-14T15:26:31.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:31 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2891079402' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:26:31.461 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:26:31.487 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:31.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:31.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:26:31.589 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:26:31.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:31.617 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:31.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:26:31.718 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:26:31.745 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:31.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:31.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:26:31.846 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:26:31.870 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:31.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:31.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:26:31.971 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:26:31.995 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:31.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:31.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:26:32.097 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:26:32.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:32.127 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:32.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:26:32.229 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:26:32.253 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:32.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:32.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:26:32.355 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:26:32.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:32 smithi092 ceph-mon[33966]: 2.14 scrub starts 2024-04-14T15:26:32.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:32 smithi092 ceph-mon[33966]: 2.14 scrub ok 2024-04-14T15:26:32.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:32 smithi092 ceph-mon[33966]: 4.6 scrub starts 2024-04-14T15:26:32.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:32 smithi092 ceph-mon[33966]: 4.6 scrub ok 2024-04-14T15:26:32.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:32 smithi092 ceph-mon[33966]: pgmap v1238: 224 pgs: 224 active+clean; 4.4 MiB data, 738 MiB used, 625 GiB / 626 GiB avail; 2.3 KiB/s rd, 14 KiB/s wr, 2 op/s 2024-04-14T15:26:32.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:32 smithi092 ceph-mon[33966]: 4.4 scrub starts 2024-04-14T15:26:32.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:32 smithi092 ceph-mon[33966]: 4.4 scrub ok 2024-04-14T15:26:32.378 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:32.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:32.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:26:32.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:32 smithi090 ceph-mon[30399]: 2.14 scrub starts 2024-04-14T15:26:32.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:32 smithi090 ceph-mon[30399]: 2.14 scrub ok 2024-04-14T15:26:32.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:32 smithi090 ceph-mon[30399]: 4.6 scrub starts 2024-04-14T15:26:32.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:32 smithi090 ceph-mon[30399]: 4.6 scrub ok 2024-04-14T15:26:32.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:32 smithi090 ceph-mon[30399]: pgmap v1238: 224 pgs: 224 active+clean; 4.4 MiB data, 738 MiB used, 625 GiB / 626 GiB avail; 2.3 KiB/s rd, 14 KiB/s wr, 2 op/s 2024-04-14T15:26:32.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:32 smithi090 ceph-mon[30399]: 4.4 scrub starts 2024-04-14T15:26:32.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:32 smithi090 ceph-mon[30399]: 4.4 scrub ok 2024-04-14T15:26:32.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:32 smithi090 ceph-mon[41670]: 2.14 scrub starts 2024-04-14T15:26:32.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:32 smithi090 ceph-mon[41670]: 2.14 scrub ok 2024-04-14T15:26:32.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:32 smithi090 ceph-mon[41670]: 4.6 scrub starts 2024-04-14T15:26:32.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:32 smithi090 ceph-mon[41670]: 4.6 scrub ok 2024-04-14T15:26:32.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:32 smithi090 ceph-mon[41670]: pgmap v1238: 224 pgs: 224 active+clean; 4.4 MiB data, 738 MiB used, 625 GiB / 626 GiB avail; 2.3 KiB/s rd, 14 KiB/s wr, 2 op/s 2024-04-14T15:26:32.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:32 smithi090 ceph-mon[41670]: 4.4 scrub starts 2024-04-14T15:26:32.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:32 smithi090 ceph-mon[41670]: 4.4 scrub ok 2024-04-14T15:26:32.480 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:26:32.508 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:32.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:32.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:26:32.610 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:26:32.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:32.634 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:32.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:26:32.688 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-14T15:26:32.735 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:26:32.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:32.763 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:32.764 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:26:32.817 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:26:32.864 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:26:32.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:32.889 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:32.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:26:32.990 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:26:33.015 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:33.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:33.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:26:33.117 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:26:33.140 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:33.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:33.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:26:33.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:33 smithi090 ceph-mon[41670]: 11.6 scrub starts 2024-04-14T15:26:33.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:33 smithi090 ceph-mon[41670]: 11.6 scrub ok 2024-04-14T15:26:33.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:33 smithi090 ceph-mon[41670]: 1.10 scrub starts 2024-04-14T15:26:33.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:33 smithi090 ceph-mon[41670]: 1.10 scrub ok 2024-04-14T15:26:33.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:33 smithi090 ceph-mon[30399]: 11.6 scrub starts 2024-04-14T15:26:33.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:33 smithi090 ceph-mon[30399]: 11.6 scrub ok 2024-04-14T15:26:33.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:33 smithi090 ceph-mon[30399]: 1.10 scrub starts 2024-04-14T15:26:33.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:33 smithi090 ceph-mon[30399]: 1.10 scrub ok 2024-04-14T15:26:33.242 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:26:33.266 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:33.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:33.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:26:33.367 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:26:33.371 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:33 smithi092 ceph-mon[33966]: 11.6 scrub starts 2024-04-14T15:26:33.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:33 smithi092 ceph-mon[33966]: 11.6 scrub ok 2024-04-14T15:26:33.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:33 smithi092 ceph-mon[33966]: 1.10 scrub starts 2024-04-14T15:26:33.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:33 smithi092 ceph-mon[33966]: 1.10 scrub ok 2024-04-14T15:26:33.396 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:33.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:33.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:26:33.490 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-14T15:26:33.497 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:26:33.555 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:26:33.558 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:33.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:33.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:26:33.660 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:26:33.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:33.685 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:33.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:26:33.786 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:26:33.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:33.814 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:33.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:26:33.915 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:26:33.940 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:33.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:33.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:26:34.041 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:26:34.069 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:34.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:34.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:26:34.079 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:26:33 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:26:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:26:34.079 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:34 smithi090 ceph-mon[30399]: 5.a scrub starts 2024-04-14T15:26:34.171 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:26:34.198 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:34.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:34.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:26:34.300 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:26:34.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:34.325 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:34.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:26:34.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:34 smithi092 ceph-mon[33966]: 5.a scrub starts 2024-04-14T15:26:34.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:34 smithi092 ceph-mon[33966]: 5.a scrub ok 2024-04-14T15:26:34.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:34 smithi092 ceph-mon[33966]: 11.1 scrub starts 2024-04-14T15:26:34.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:34 smithi092 ceph-mon[33966]: 11.1 scrub ok 2024-04-14T15:26:34.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:34 smithi092 ceph-mon[33966]: 3.1 scrub starts 2024-04-14T15:26:34.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:34 smithi092 ceph-mon[33966]: 3.1 scrub ok 2024-04-14T15:26:34.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:34 smithi092 ceph-mon[33966]: 6.2 scrub starts 2024-04-14T15:26:34.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:34 smithi092 ceph-mon[33966]: pgmap v1239: 224 pgs: 224 active+clean; 4.4 MiB data, 738 MiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 12 KiB/s wr, 2 op/s 2024-04-14T15:26:34.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:34 smithi092 ceph-mon[33966]: 6.2 scrub ok 2024-04-14T15:26:34.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:34 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:26:34.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:34 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:26:34.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:34 smithi092 ceph-mon[33966]: 1.12 scrub starts 2024-04-14T15:26:34.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:34 smithi092 ceph-mon[33966]: 1.12 scrub ok 2024-04-14T15:26:34.426 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:26:34.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:34 smithi090 ceph-mon[30399]: 5.a scrub ok 2024-04-14T15:26:34.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:34 smithi090 ceph-mon[30399]: 11.1 scrub starts 2024-04-14T15:26:34.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:34 smithi090 ceph-mon[30399]: 11.1 scrub ok 2024-04-14T15:26:34.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:34 smithi090 ceph-mon[30399]: 3.1 scrub starts 2024-04-14T15:26:34.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:34 smithi090 ceph-mon[30399]: 3.1 scrub ok 2024-04-14T15:26:34.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:34 smithi090 ceph-mon[30399]: 6.2 scrub starts 2024-04-14T15:26:34.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:34 smithi090 ceph-mon[30399]: pgmap v1239: 224 pgs: 224 active+clean; 4.4 MiB data, 738 MiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 12 KiB/s wr, 2 op/s 2024-04-14T15:26:34.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:34 smithi090 ceph-mon[30399]: 6.2 scrub ok 2024-04-14T15:26:34.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:34 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:26:34.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:34 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:26:34.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:34 smithi090 ceph-mon[30399]: 1.12 scrub starts 2024-04-14T15:26:34.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:34 smithi090 ceph-mon[30399]: 1.12 scrub ok 2024-04-14T15:26:34.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:34 smithi090 ceph-mon[41670]: 5.a scrub starts 2024-04-14T15:26:34.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:34 smithi090 ceph-mon[41670]: 5.a scrub ok 2024-04-14T15:26:34.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:34 smithi090 ceph-mon[41670]: 11.1 scrub starts 2024-04-14T15:26:34.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:34 smithi090 ceph-mon[41670]: 11.1 scrub ok 2024-04-14T15:26:34.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:34 smithi090 ceph-mon[41670]: 3.1 scrub starts 2024-04-14T15:26:34.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:34 smithi090 ceph-mon[41670]: 3.1 scrub ok 2024-04-14T15:26:34.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:34 smithi090 ceph-mon[41670]: 6.2 scrub starts 2024-04-14T15:26:34.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:34 smithi090 ceph-mon[41670]: pgmap v1239: 224 pgs: 224 active+clean; 4.4 MiB data, 738 MiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 12 KiB/s wr, 2 op/s 2024-04-14T15:26:34.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:34 smithi090 ceph-mon[41670]: 6.2 scrub ok 2024-04-14T15:26:34.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:34 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:26:34.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:34 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:26:34.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:34 smithi090 ceph-mon[41670]: 1.12 scrub starts 2024-04-14T15:26:34.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:34 smithi090 ceph-mon[41670]: 1.12 scrub ok 2024-04-14T15:26:34.454 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:34.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:34.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:26:34.556 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:26:34.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:34.584 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:34.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:26:34.685 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:26:34.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:34.714 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:34.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:26:34.815 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:26:34.842 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:34.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:34.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:26:34.944 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:26:34.969 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:34.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:34.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:26:35.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:34.883+0000 7f18bd43d700 1 -- 172.21.15.90:0/666130102 >> v1:172.21.15.90:6789/0 conn(0x7f18b8107760 legacy=0x7f18b8107b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:35.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:34.883+0000 7f18bd43d700 1 -- 172.21.15.90:0/666130102 shutdown_connections 2024-04-14T15:26:35.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:34.883+0000 7f18bd43d700 1 -- 172.21.15.90:0/666130102 >> 172.21.15.90:0/666130102 conn(0x7f18b80fd290 msgr2=0x7f18b80ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:26:35.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:34.883+0000 7f18bd43d700 1 -- 172.21.15.90:0/666130102 shutdown_connections 2024-04-14T15:26:35.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:34.883+0000 7f18bd43d700 1 -- 172.21.15.90:0/666130102 wait complete. 2024-04-14T15:26:35.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:34.883+0000 7f18bd43d700 1 librados: starting msgr at 2024-04-14T15:26:35.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:34.883+0000 7f18bd43d700 1 librados: starting objecter 2024-04-14T15:26:35.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:34.883+0000 7f18bd43d700 1 Processor -- start 2024-04-14T15:26:35.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:34.884+0000 7f18bd43d700 1 -- start start 2024-04-14T15:26:35.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:34.884+0000 7f18bd43d700 1 librados: setting wanted keys 2024-04-14T15:26:35.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:34.884+0000 7f18bd43d700 1 librados: calling monclient init 2024-04-14T15:26:35.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:34.884+0000 7f18bd43d700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f18b810de90 con 0x7f18b810e1e0 2024-04-14T15:26:35.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:34.884+0000 7f18bd43d700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f18b81a7df0 con 0x7f18b810a610 2024-04-14T15:26:35.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:34.884+0000 7f18bd43d700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f18b81a8fd0 con 0x7f18b8107760 2024-04-14T15:26:35.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:34.884+0000 7f18b6ffd700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f18b8107760 0x7f18b810ace0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:39124/0 (socket says 172.21.15.90:39124) 2024-04-14T15:26:35.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:34.884+0000 7f18b6ffd700 1 -- 172.21.15.90:0/1266832884 learned_addr learned my addr 172.21.15.90:0/1266832884 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:26:35.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:34.884+0000 7f189ffff700 10 client.?.objecter ms_handle_connect 0x7f18b810e1e0 2024-04-14T15:26:35.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:34.884+0000 7f189ffff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:26:35.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:34.884+0000 7f189ffff700 10 client.?.objecter ms_handle_connect 0x7f18b8107760 2024-04-14T15:26:35.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:34.884+0000 7f189ffff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:26:35.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:34.884+0000 7f189ffff700 10 client.?.objecter ms_handle_connect 0x7f18b810a610 2024-04-14T15:26:35.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:34.884+0000 7f189ffff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:26:35.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:34.884+0000 7f189ffff700 1 -- 172.21.15.90:0/1266832884 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2695080517 0 0) 0x7f18b81a8fd0 con 0x7f18b8107760 2024-04-14T15:26:35.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:34.884+0000 7f189ffff700 1 -- 172.21.15.90:0/1266832884 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f18a00035b0 con 0x7f18b8107760 2024-04-14T15:26:35.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:34.884+0000 7f189ffff700 1 -- 172.21.15.90:0/1266832884 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1674530669 0 0) 0x7f18b810de90 con 0x7f18b810e1e0 2024-04-14T15:26:35.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:34.884+0000 7f189ffff700 1 -- 172.21.15.90:0/1266832884 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f18b81a8fd0 con 0x7f18b810e1e0 2024-04-14T15:26:35.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:34.884+0000 7f189ffff700 1 -- 172.21.15.90:0/1266832884 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3365021589 0 0) 0x7f18b81a7df0 con 0x7f18b810a610 2024-04-14T15:26:35.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:34.885+0000 7f189ffff700 1 -- 172.21.15.90:0/1266832884 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f18b810de90 con 0x7f18b810a610 2024-04-14T15:26:35.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:34.885+0000 7f189ffff700 1 -- 172.21.15.90:0/1266832884 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3198374728 0 0) 0x7f18b81a8fd0 con 0x7f18b810e1e0 2024-04-14T15:26:35.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:34.885+0000 7f189ffff700 1 -- 172.21.15.90:0/1266832884 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f18b81a7df0 con 0x7f18b810e1e0 2024-04-14T15:26:35.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:34.885+0000 7f189ffff700 1 -- 172.21.15.90:0/1266832884 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f18a8001b70 con 0x7f18b810e1e0 2024-04-14T15:26:35.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:34.885+0000 7f189ffff700 1 -- 172.21.15.90:0/1266832884 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4016645282 0 0) 0x7f18a00035b0 con 0x7f18b8107760 2024-04-14T15:26:35.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:34.885+0000 7f189ffff700 1 -- 172.21.15.90:0/1266832884 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f18b81a8fd0 con 0x7f18b8107760 2024-04-14T15:26:35.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:34.885+0000 7f189ffff700 1 -- 172.21.15.90:0/1266832884 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f18b0001430 con 0x7f18b8107760 2024-04-14T15:26:35.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:34.885+0000 7f189ffff700 1 -- 172.21.15.90:0/1266832884 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1635466421 0 0) 0x7f18b810de90 con 0x7f18b810a610 2024-04-14T15:26:35.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:34.885+0000 7f189ffff700 1 -- 172.21.15.90:0/1266832884 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f18a00035b0 con 0x7f18b810a610 2024-04-14T15:26:35.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:34.885+0000 7f189ffff700 1 -- 172.21.15.90:0/1266832884 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f18ac0018d0 con 0x7f18b810a610 2024-04-14T15:26:35.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:34.885+0000 7f189ffff700 1 -- 172.21.15.90:0/1266832884 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2673713372 0 0) 0x7f18b81a7df0 con 0x7f18b810e1e0 2024-04-14T15:26:35.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:34.885+0000 7f189ffff700 1 -- 172.21.15.90:0/1266832884 >> v1:172.21.15.90:6790/0 conn(0x7f18b8107760 legacy=0x7f18b810ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:35.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:34.885+0000 7f189ffff700 1 -- 172.21.15.90:0/1266832884 >> v1:172.21.15.92:6789/0 conn(0x7f18b810a610 legacy=0x7f18b810b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:35.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:34.885+0000 7f189ffff700 1 -- 172.21.15.90:0/1266832884 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18b81aa1b0 con 0x7f18b810e1e0 2024-04-14T15:26:35.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:34.885+0000 7f18bd43d700 1 -- 172.21.15.90:0/1266832884 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f18b81a9200 con 0x7f18b810e1e0 2024-04-14T15:26:35.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:34.885+0000 7f18bd43d700 10 client.38127.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:26:35.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:34.885+0000 7f18bd43d700 1 -- 172.21.15.90:0/1266832884 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f18b81a96c0 con 0x7f18b810e1e0 2024-04-14T15:26:35.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:34.885+0000 7f18bd43d700 1 librados: init done 2024-04-14T15:26:35.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:34.885+0000 7f189ffff700 1 -- 172.21.15.90:0/1266832884 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f18a8003ea0 con 0x7f18b810e1e0 2024-04-14T15:26:35.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:34.885+0000 7f189ffff700 1 -- 172.21.15.90:0/1266832884 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f18a8004dc0 con 0x7f18b810e1e0 2024-04-14T15:26:35.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:34.886+0000 7f18bd43d700 1 -- 172.21.15.90:0/1266832884 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18a4005120 con 0x7f18b810e1e0 2024-04-14T15:26:35.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:34.886+0000 7f189ffff700 1 -- 172.21.15.90:0/1266832884 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f18a8003a00 con 0x7f18b810e1e0 2024-04-14T15:26:35.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:34.887+0000 7f189ffff700 1 -- 172.21.15.90:0/1266832884 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(831..831 src has 467..831) ==== 8783+0+0 (unknown 1557274490 0 0) 0x7f18a8089c80 con 0x7f18b810e1e0 2024-04-14T15:26:35.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:34.887+0000 7f189ffff700 10 client.38127.objecter ms_dispatch 0x7f18b8001550 osd_map(831..831 src has 467..831) 2024-04-14T15:26:35.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:34.887+0000 7f189ffff700 3 client.38127.objecter handle_osd_map got epochs [831,831] > 0 2024-04-14T15:26:35.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:34.887+0000 7f189ffff700 3 client.38127.objecter handle_osd_map decoding full epoch 831 2024-04-14T15:26:35.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:34.887+0000 7f189ffff700 20 client.38127.objecter dump_active .. 0 homeless 2024-04-14T15:26:35.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:34.887+0000 7f189ffff700 10 client.38127.objecter ms_handle_connect 0x7f18a0073540 2024-04-14T15:26:35.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:34.892+0000 7f189ffff700 1 -- 172.21.15.90:0/1266832884 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f18a8056440 con 0x7f18b810e1e0 2024-04-14T15:26:35.072 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:26:35.097 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:35.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:35.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:26:35.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:35.112+0000 7f18bd43d700 1 -- 172.21.15.90:0/1266832884 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7f18a4005410 con 0x7f18b810e1e0 2024-04-14T15:26:35.199 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:26:35.221 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:26:35.227 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:35.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:35.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:26:35.329 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:26:35.330 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:35 smithi090 ceph-mon[30399]: 6.b scrub starts 2024-04-14T15:26:35.330 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:35 smithi090 ceph-mon[30399]: 6.b scrub ok 2024-04-14T15:26:35.330 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:35 smithi090 ceph-mon[30399]: osdmap e831: 8 total, 8 up, 7 in 2024-04-14T15:26:35.330 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:35 smithi090 ceph-mon[30399]: 2.f scrub starts 2024-04-14T15:26:35.330 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:35 smithi090 ceph-mon[30399]: 2.f scrub ok 2024-04-14T15:26:35.330 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:35 smithi090 ceph-mon[30399]: 1.8 scrub starts 2024-04-14T15:26:35.331 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:35 smithi090 ceph-mon[30399]: 1.8 scrub ok 2024-04-14T15:26:35.331 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:35 smithi090 ceph-mon[41670]: 6.b scrub starts 2024-04-14T15:26:35.331 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:35 smithi090 ceph-mon[41670]: 6.b scrub ok 2024-04-14T15:26:35.331 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:35 smithi090 ceph-mon[41670]: osdmap e831: 8 total, 8 up, 7 in 2024-04-14T15:26:35.331 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:35 smithi090 ceph-mon[41670]: 2.f scrub starts 2024-04-14T15:26:35.331 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:35 smithi090 ceph-mon[41670]: 2.f scrub ok 2024-04-14T15:26:35.331 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:35 smithi090 ceph-mon[41670]: 1.8 scrub starts 2024-04-14T15:26:35.331 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:35 smithi090 ceph-mon[41670]: 1.8 scrub ok 2024-04-14T15:26:35.356 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:35.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:35.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:26:35.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:35 smithi092 ceph-mon[33966]: 6.b scrub starts 2024-04-14T15:26:35.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:35 smithi092 ceph-mon[33966]: 6.b scrub ok 2024-04-14T15:26:35.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:35 smithi092 ceph-mon[33966]: osdmap e831: 8 total, 8 up, 7 in 2024-04-14T15:26:35.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:35 smithi092 ceph-mon[33966]: 2.f scrub starts 2024-04-14T15:26:35.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:35 smithi092 ceph-mon[33966]: 2.f scrub ok 2024-04-14T15:26:35.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:35 smithi092 ceph-mon[33966]: 1.8 scrub starts 2024-04-14T15:26:35.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:35 smithi092 ceph-mon[33966]: 1.8 scrub ok 2024-04-14T15:26:35.458 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:26:35.485 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:35.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:35.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:26:35.587 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:26:35.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:35.615 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:35.615 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:26:35.716 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:26:35.743 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:35.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:35.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:26:35.845 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:26:35.871 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:26:35 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:26:35.872 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:35.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:35.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:26:35.974 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:26:36.001 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:36.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:36.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:26:36.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:36.032+0000 7f189ffff700 1 -- 172.21.15.90:0/1266832884 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v832) ==== 101+0+0 (unknown 2589433373 0 0) 0x7f18a805a0f0 con 0x7f18b810e1e0 2024-04-14T15:26:36.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:36.034+0000 7f18bd43d700 10 librados: watch_flush enter 2024-04-14T15:26:36.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:36.034+0000 7f18bd43d700 10 librados: watch_flush exit 2024-04-14T15:26:36.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:36.034+0000 7f18bd43d700 20 client.38127.objecter shutdown clearing up homeless session... 2024-04-14T15:26:36.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:36.034+0000 7f18bd43d700 10 client.38127.objecter successfully canceled tick 2024-04-14T15:26:36.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:36.034+0000 7f18bd43d700 1 -- 172.21.15.90:0/1266832884 >> v1:172.21.15.90:6800/420935427 conn(0x7f18a0073540 legacy=0x7f18a0075a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:36.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:36.034+0000 7f18bd43d700 1 -- 172.21.15.90:0/1266832884 >> v1:172.21.15.90:6789/0 conn(0x7f18b810e1e0 legacy=0x7f18b81a56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:36.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:36.034+0000 7f18bd43d700 1 -- 172.21.15.90:0/1266832884 shutdown_connections 2024-04-14T15:26:36.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:36.034+0000 7f18bd43d700 1 -- 172.21.15.90:0/1266832884 >> 172.21.15.90:0/1266832884 conn(0x7f18b80fd290 msgr2=0x7f18b8111700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:26:36.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:36.035+0000 7f18bd43d700 1 -- 172.21.15.90:0/1266832884 shutdown_connections 2024-04-14T15:26:36.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:36.035+0000 7f18bd43d700 1 -- 172.21.15.90:0/1266832884 wait complete. 2024-04-14T15:26:36.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:36.035+0000 7f18bd43d700 1 librados: shutdown 2024-04-14T15:26:36.036 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is unset 2024-04-14T15:26:36.103 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:26:36.128 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:36.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:36.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:26:36.230 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:26:36.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:36.258 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:36.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:26:36.327 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:36 smithi090 ceph-mon[41670]: 11.a scrub starts 2024-04-14T15:26:36.328 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:36 smithi090 ceph-mon[41670]: 11.a scrub ok 2024-04-14T15:26:36.328 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:36 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1266832884' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:26:36.328 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:36 smithi090 ceph-mon[41670]: pgmap v1241: 224 pgs: 224 active+clean; 4.4 MiB data, 738 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 3.5 KiB/s wr, 1 op/s 2024-04-14T15:26:36.328 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:36 smithi090 ceph-mon[41670]: 6.4 scrub starts 2024-04-14T15:26:36.328 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:36 smithi090 ceph-mon[41670]: 6.4 scrub ok 2024-04-14T15:26:36.328 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:36 smithi090 ceph-mon[41670]: 1.1 scrub starts 2024-04-14T15:26:36.328 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:36 smithi090 ceph-mon[41670]: 1.1 scrub ok 2024-04-14T15:26:36.328 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:36 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:26:36.328 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:36 smithi090 ceph-mon[41670]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-14T15:26:36.328 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:36 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1266832884' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-14T15:26:36.328 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:36 smithi090 ceph-mon[41670]: osdmap e832: 8 total, 8 up, 7 in 2024-04-14T15:26:36.328 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:36 smithi090 ceph-mon[30399]: 11.a scrub starts 2024-04-14T15:26:36.329 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:36 smithi090 ceph-mon[30399]: 11.a scrub ok 2024-04-14T15:26:36.329 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:36 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1266832884' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:26:36.329 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:36 smithi090 ceph-mon[30399]: pgmap v1241: 224 pgs: 224 active+clean; 4.4 MiB data, 738 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 3.5 KiB/s wr, 1 op/s 2024-04-14T15:26:36.329 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:36 smithi090 ceph-mon[30399]: 6.4 scrub starts 2024-04-14T15:26:36.329 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:36 smithi090 ceph-mon[30399]: 6.4 scrub ok 2024-04-14T15:26:36.329 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:36 smithi090 ceph-mon[30399]: 1.1 scrub starts 2024-04-14T15:26:36.329 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:36 smithi090 ceph-mon[30399]: 1.1 scrub ok 2024-04-14T15:26:36.329 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:36 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:26:36.329 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:36 smithi090 ceph-mon[30399]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-14T15:26:36.329 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:36 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1266832884' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-14T15:26:36.329 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:36 smithi090 ceph-mon[30399]: osdmap e832: 8 total, 8 up, 7 in 2024-04-14T15:26:36.359 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:26:36.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:36 smithi092 ceph-mon[33966]: 11.a scrub starts 2024-04-14T15:26:36.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:36 smithi092 ceph-mon[33966]: 11.a scrub ok 2024-04-14T15:26:36.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:36 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1266832884' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:26:36.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:36 smithi092 ceph-mon[33966]: pgmap v1241: 224 pgs: 224 active+clean; 4.4 MiB data, 738 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 3.5 KiB/s wr, 1 op/s 2024-04-14T15:26:36.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:36 smithi092 ceph-mon[33966]: 6.4 scrub starts 2024-04-14T15:26:36.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:36 smithi092 ceph-mon[33966]: 6.4 scrub ok 2024-04-14T15:26:36.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:36 smithi092 ceph-mon[33966]: 1.1 scrub starts 2024-04-14T15:26:36.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:36 smithi092 ceph-mon[33966]: 1.1 scrub ok 2024-04-14T15:26:36.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:36 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:26:36.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:36 smithi092 ceph-mon[33966]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-14T15:26:36.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:36 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1266832884' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-14T15:26:36.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:36 smithi092 ceph-mon[33966]: osdmap e832: 8 total, 8 up, 7 in 2024-04-14T15:26:36.385 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:36.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:36.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:26:36.487 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:26:36.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:36.515 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:36.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:26:36.582 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-14T15:26:36.582 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-14T15:26:36.582 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-14T15:26:36.582 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-14T15:26:36.616 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:26:36.640 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:36.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:36.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:26:36.742 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:26:36.769 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:36.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:36.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:26:36.871 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:26:36.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:36.897 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:36.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:26:36.998 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:26:37.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:37.023 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:37.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:26:37.124 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:26:37.149 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:37.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:37.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:26:37.175 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-14T15:26:37.176 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-14T15:26:37.176 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-04-14T15:26:37.176 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-14T15:26:37.176 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-14T15:26:37.176 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-04-14T15:26:37.251 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:26:37.275 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:37.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:37.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:26:37.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:37 smithi092 ceph-mon[33966]: 2.1 scrub starts 2024-04-14T15:26:37.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:37 smithi092 ceph-mon[33966]: 2.1 scrub ok 2024-04-14T15:26:37.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:37 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:26:37.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:37 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:26:37.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:37 smithi092 ceph-mon[33966]: 1.a scrub starts 2024-04-14T15:26:37.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:37 smithi092 ceph-mon[33966]: 1.a scrub ok 2024-04-14T15:26:37.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:37 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:26:37.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:37 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:26:37.377 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:26:37.387 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:37 smithi090 ceph-mon[41670]: 2.1 scrub starts 2024-04-14T15:26:37.387 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:37 smithi090 ceph-mon[41670]: 2.1 scrub ok 2024-04-14T15:26:37.387 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:37 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:26:37.387 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:37 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:26:37.387 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:37 smithi090 ceph-mon[41670]: 1.a scrub starts 2024-04-14T15:26:37.387 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:37 smithi090 ceph-mon[41670]: 1.a scrub ok 2024-04-14T15:26:37.387 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:37 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:26:37.387 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:37 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:26:37.387 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:37 smithi090 ceph-mon[30399]: 2.1 scrub starts 2024-04-14T15:26:37.388 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:37 smithi090 ceph-mon[30399]: 2.1 scrub ok 2024-04-14T15:26:37.388 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:37 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:26:37.388 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:37 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:26:37.388 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:37 smithi090 ceph-mon[30399]: 1.a scrub starts 2024-04-14T15:26:37.388 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:37 smithi090 ceph-mon[30399]: 1.a scrub ok 2024-04-14T15:26:37.388 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:37 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:26:37.388 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:37 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:26:37.401 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:37.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:37.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:26:37.426 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:26:37.504 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:26:37.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:37.529 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:37.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:26:37.630 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:26:37.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:37.655 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:37.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:26:37.756 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:26:37.780 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:37.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:37.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:26:37.882 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:26:37.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:37.909 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:37.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:26:38.011 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:26:38.035 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:38.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:38.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:26:38.137 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:26:38.163 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:38.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:38.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:26:38.265 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:26:38.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:38.290 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:38.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:26:38.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:38 smithi092 ceph-mon[33966]: 1.14 scrub starts 2024-04-14T15:26:38.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:38 smithi092 ceph-mon[33966]: 1.14 scrub ok 2024-04-14T15:26:38.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:38 smithi092 ceph-mon[33966]: 11.19 scrub starts 2024-04-14T15:26:38.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:38 smithi092 ceph-mon[33966]: 11.19 scrub ok 2024-04-14T15:26:38.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:38 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]': finished 2024-04-14T15:26:38.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:38 smithi092 ceph-mon[33966]: osdmap e833: 8 total, 8 up, 7 in 2024-04-14T15:26:38.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:38 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-146"}]: dispatch 2024-04-14T15:26:38.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:38 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-146"}]: dispatch 2024-04-14T15:26:38.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:38 smithi092 ceph-mon[33966]: pgmap v1244: 224 pgs: 224 active+clean; 4.4 MiB data, 738 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:26:38.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:38 smithi092 ceph-mon[33966]: 1.f scrub starts 2024-04-14T15:26:38.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:38 smithi092 ceph-mon[33966]: 1.f scrub ok 2024-04-14T15:26:38.391 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:26:38.415 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:38.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:38.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:26:38.434 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:38 smithi090 ceph-mon[41670]: 1.14 scrub starts 2024-04-14T15:26:38.434 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:38 smithi090 ceph-mon[41670]: 1.14 scrub ok 2024-04-14T15:26:38.434 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:38 smithi090 ceph-mon[41670]: 11.19 scrub starts 2024-04-14T15:26:38.434 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:38 smithi090 ceph-mon[41670]: 11.19 scrub ok 2024-04-14T15:26:38.434 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:38 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]': finished 2024-04-14T15:26:38.434 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:38 smithi090 ceph-mon[41670]: osdmap e833: 8 total, 8 up, 7 in 2024-04-14T15:26:38.434 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:38 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-146"}]: dispatch 2024-04-14T15:26:38.434 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:38 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-146"}]: dispatch 2024-04-14T15:26:38.434 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:38 smithi090 ceph-mon[41670]: pgmap v1244: 224 pgs: 224 active+clean; 4.4 MiB data, 738 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:26:38.435 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:38 smithi090 ceph-mon[41670]: 1.f scrub starts 2024-04-14T15:26:38.435 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:38 smithi090 ceph-mon[41670]: 1.f scrub ok 2024-04-14T15:26:38.435 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:38 smithi090 ceph-mon[30399]: 1.14 scrub starts 2024-04-14T15:26:38.435 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:38 smithi090 ceph-mon[30399]: 1.14 scrub ok 2024-04-14T15:26:38.435 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:38 smithi090 ceph-mon[30399]: 11.19 scrub starts 2024-04-14T15:26:38.435 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:38 smithi090 ceph-mon[30399]: 11.19 scrub ok 2024-04-14T15:26:38.435 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:38 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]': finished 2024-04-14T15:26:38.435 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:38 smithi090 ceph-mon[30399]: osdmap e833: 8 total, 8 up, 7 in 2024-04-14T15:26:38.435 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:38 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-146"}]: dispatch 2024-04-14T15:26:38.435 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:38 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-146"}]: dispatch 2024-04-14T15:26:38.435 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:38 smithi090 ceph-mon[30399]: pgmap v1244: 224 pgs: 224 active+clean; 4.4 MiB data, 738 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:26:38.435 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:38 smithi090 ceph-mon[30399]: 1.f scrub starts 2024-04-14T15:26:38.436 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:38 smithi090 ceph-mon[30399]: 1.f scrub ok 2024-04-14T15:26:38.517 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:26:38.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:38.542 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:38.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:26:38.644 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:26:38.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:38.669 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:38.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:26:38.770 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:26:38.793 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:38.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:38.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:26:38.895 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:26:38.919 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:38.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:38.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:26:39.021 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:26:39.047 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:39.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:39.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:26:39.128 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-14T15:26:39.143 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:39 smithi092 ceph-mon[33966]: 6.1b scrub starts 2024-04-14T15:26:39.143 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:39 smithi092 ceph-mon[33966]: 6.1b scrub ok 2024-04-14T15:26:39.143 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:39 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-146"}]': finished 2024-04-14T15:26:39.143 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:39 smithi092 ceph-mon[33966]: osdmap e834: 8 total, 8 up, 7 in 2024-04-14T15:26:39.143 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:39 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:26:39.143 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:39 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:26:39.143 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:39 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-146"}]: dispatch 2024-04-14T15:26:39.143 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:39 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-146"}]: dispatch 2024-04-14T15:26:39.149 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:26:39.172 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:39.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:39.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:26:39.273 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-04-14T15:26:39.273 INFO:teuthology.orchestra.run.smithi092.stdout: "success": "" 2024-04-14T15:26:39.273 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-04-14T15:26:39.273 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:26:39.297 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:39.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:39.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:26:39.350 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:39 smithi090 ceph-mon[30399]: 6.1b scrub starts 2024-04-14T15:26:39.350 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:39 smithi090 ceph-mon[30399]: 6.1b scrub ok 2024-04-14T15:26:39.350 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:39 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-146"}]': finished 2024-04-14T15:26:39.351 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:39 smithi090 ceph-mon[30399]: osdmap e834: 8 total, 8 up, 7 in 2024-04-14T15:26:39.351 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:39 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:26:39.351 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:39 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:26:39.351 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:39 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-146"}]: dispatch 2024-04-14T15:26:39.351 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:39 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-146"}]: dispatch 2024-04-14T15:26:39.351 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:39 smithi090 ceph-mon[41670]: 6.1b scrub starts 2024-04-14T15:26:39.352 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:39 smithi090 ceph-mon[41670]: 6.1b scrub ok 2024-04-14T15:26:39.352 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:39 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-146"}]': finished 2024-04-14T15:26:39.352 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:39 smithi090 ceph-mon[41670]: osdmap e834: 8 total, 8 up, 7 in 2024-04-14T15:26:39.352 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:39 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:26:39.352 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:39 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:26:39.352 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:39 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-146"}]: dispatch 2024-04-14T15:26:39.353 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:39 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-146"}]: dispatch 2024-04-14T15:26:39.399 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:26:39.414 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd set noscrub 2024-04-14T15:26:39.424 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:39.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:39.425 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:26:39.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.349+0000 7f47296ee700 1 -- 172.21.15.90:0/4257701254 >> v1:172.21.15.90:6789/0 conn(0x7f4724107760 legacy=0x7f4724107b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:39.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.349+0000 7f47296ee700 1 -- 172.21.15.90:0/4257701254 shutdown_connections 2024-04-14T15:26:39.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.349+0000 7f47296ee700 1 -- 172.21.15.90:0/4257701254 >> 172.21.15.90:0/4257701254 conn(0x7f47240fd290 msgr2=0x7f47240ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:26:39.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.349+0000 7f47296ee700 1 -- 172.21.15.90:0/4257701254 shutdown_connections 2024-04-14T15:26:39.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.349+0000 7f47296ee700 1 -- 172.21.15.90:0/4257701254 wait complete. 2024-04-14T15:26:39.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.350+0000 7f47296ee700 1 librados: starting msgr at 2024-04-14T15:26:39.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.350+0000 7f47296ee700 1 librados: starting objecter 2024-04-14T15:26:39.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.350+0000 7f47296ee700 1 Processor -- start 2024-04-14T15:26:39.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.350+0000 7f47296ee700 1 -- start start 2024-04-14T15:26:39.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.350+0000 7f47296ee700 1 librados: setting wanted keys 2024-04-14T15:26:39.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.350+0000 7f47296ee700 1 librados: calling monclient init 2024-04-14T15:26:39.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.350+0000 7f47296ee700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f472410de80 con 0x7f472410a610 2024-04-14T15:26:39.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.350+0000 7f47296ee700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f47241b0690 con 0x7f4724107760 2024-04-14T15:26:39.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.350+0000 7f47296ee700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f47241b1870 con 0x7f472410e1e0 2024-04-14T15:26:39.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.350+0000 7f47227fc700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f472410a610 0x7f4724074530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:55298/0 (socket says 172.21.15.90:55298) 2024-04-14T15:26:39.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.350+0000 7f47227fc700 1 -- 172.21.15.90:0/1180320903 learned_addr learned my addr 172.21.15.90:0/1180320903 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:26:39.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.350+0000 7f470bfff700 10 client.?.objecter ms_handle_connect 0x7f472410a610 2024-04-14T15:26:39.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.350+0000 7f470bfff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:26:39.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.351+0000 7f470bfff700 10 client.?.objecter ms_handle_connect 0x7f472410e1e0 2024-04-14T15:26:39.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.351+0000 7f470bfff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:26:39.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.351+0000 7f470bfff700 10 client.?.objecter ms_handle_connect 0x7f4724107760 2024-04-14T15:26:39.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.351+0000 7f470bfff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:26:39.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.351+0000 7f470bfff700 1 -- 172.21.15.90:0/1180320903 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1411915276 0 0) 0x7f47241b0690 con 0x7f4724107760 2024-04-14T15:26:39.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.351+0000 7f470bfff700 1 -- 172.21.15.90:0/1180320903 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f470c0035b0 con 0x7f4724107760 2024-04-14T15:26:39.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.351+0000 7f470bfff700 1 -- 172.21.15.90:0/1180320903 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1422632677 0 0) 0x7f47241b1870 con 0x7f472410e1e0 2024-04-14T15:26:39.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.351+0000 7f470bfff700 1 -- 172.21.15.90:0/1180320903 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f47241b0690 con 0x7f472410e1e0 2024-04-14T15:26:39.449 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:26:39.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.351+0000 7f470bfff700 1 -- 172.21.15.90:0/1180320903 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1250394337 0 0) 0x7f472410de80 con 0x7f472410a610 2024-04-14T15:26:39.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.351+0000 7f470bfff700 1 -- 172.21.15.90:0/1180320903 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f47241b1870 con 0x7f472410a610 2024-04-14T15:26:39.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.351+0000 7f470bfff700 1 -- 172.21.15.90:0/1180320903 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3797158865 0 0) 0x7f47241b0690 con 0x7f472410e1e0 2024-04-14T15:26:39.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.351+0000 7f470bfff700 1 -- 172.21.15.90:0/1180320903 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f472410de80 con 0x7f472410e1e0 2024-04-14T15:26:39.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.351+0000 7f470bfff700 1 -- 172.21.15.90:0/1180320903 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2712752599 0 0) 0x7f470c0035b0 con 0x7f4724107760 2024-04-14T15:26:39.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.351+0000 7f470bfff700 1 -- 172.21.15.90:0/1180320903 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f47241b0690 con 0x7f4724107760 2024-04-14T15:26:39.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.351+0000 7f470bfff700 1 -- 172.21.15.90:0/1180320903 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 944300017 0 0) 0x7f47241b1870 con 0x7f472410a610 2024-04-14T15:26:39.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.351+0000 7f470bfff700 1 -- 172.21.15.90:0/1180320903 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f470c0035b0 con 0x7f472410a610 2024-04-14T15:26:39.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.351+0000 7f470bfff700 1 -- 172.21.15.90:0/1180320903 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f471c001af0 con 0x7f472410e1e0 2024-04-14T15:26:39.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.351+0000 7f470bfff700 1 -- 172.21.15.90:0/1180320903 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f4714001020 con 0x7f4724107760 2024-04-14T15:26:39.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.351+0000 7f470bfff700 1 -- 172.21.15.90:0/1180320903 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f4718001810 con 0x7f472410a610 2024-04-14T15:26:39.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.351+0000 7f470bfff700 1 -- 172.21.15.90:0/1180320903 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 646476551 0 0) 0x7f47241b0690 con 0x7f4724107760 2024-04-14T15:26:39.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.351+0000 7f470bfff700 1 -- 172.21.15.90:0/1180320903 >> v1:172.21.15.90:6790/0 conn(0x7f472410e1e0 legacy=0x7f472410cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:39.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.351+0000 7f470bfff700 1 -- 172.21.15.90:0/1180320903 >> v1:172.21.15.90:6789/0 conn(0x7f472410a610 legacy=0x7f4724074530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:39.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.351+0000 7f470bfff700 1 -- 172.21.15.90:0/1180320903 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47241b2a50 con 0x7f4724107760 2024-04-14T15:26:39.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.351+0000 7f47296ee700 1 -- 172.21.15.90:0/1180320903 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f47241b1aa0 con 0x7f4724107760 2024-04-14T15:26:39.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.351+0000 7f47296ee700 10 client.49957.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:26:39.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.351+0000 7f47296ee700 1 -- 172.21.15.90:0/1180320903 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f47241b1f60 con 0x7f4724107760 2024-04-14T15:26:39.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.351+0000 7f47296ee700 1 librados: init done 2024-04-14T15:26:39.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.351+0000 7f470bfff700 1 -- 172.21.15.90:0/1180320903 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f4714003900 con 0x7f4724107760 2024-04-14T15:26:39.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.351+0000 7f470bfff700 1 -- 172.21.15.90:0/1180320903 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f4714004e30 con 0x7f4724107760 2024-04-14T15:26:39.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.352+0000 7f47296ee700 1 -- 172.21.15.90:0/1180320903 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f4710005130 con 0x7f4724107760 2024-04-14T15:26:39.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.353+0000 7f470bfff700 1 -- 172.21.15.90:0/1180320903 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f4714001880 con 0x7f4724107760 2024-04-14T15:26:39.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.353+0000 7f470bfff700 1 -- 172.21.15.90:0/1180320903 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(835..835 src has 467..835) ==== 8376+0+0 (unknown 2776417171 0 0) 0x7f4714089810 con 0x7f4724107760 2024-04-14T15:26:39.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.353+0000 7f470bfff700 10 client.49957.objecter ms_dispatch 0x7f4724001550 osd_map(835..835 src has 467..835) 2024-04-14T15:26:39.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.353+0000 7f470bfff700 3 client.49957.objecter handle_osd_map got epochs [835,835] > 0 2024-04-14T15:26:39.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.353+0000 7f470bfff700 3 client.49957.objecter handle_osd_map decoding full epoch 835 2024-04-14T15:26:39.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.353+0000 7f470bfff700 20 client.49957.objecter dump_active .. 0 homeless 2024-04-14T15:26:39.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.353+0000 7f470bfff700 1 -- 172.21.15.90:0/1180320903 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v835) ==== 54+0+16 (unknown 2290395080 0 2420000192) 0x7f4714089c10 con 0x7f4724107760 2024-04-14T15:26:39.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.353+0000 7f470bfff700 10 client.49957.objecter ms_handle_connect 0x7f470c0713a0 2024-04-14T15:26:39.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.354+0000 7f47296ee700 10 client.49957.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:26:39.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.354+0000 7f47296ee700 15 client.49957.objecter _session_command_op_assign -1 1 2024-04-14T15:26:39.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.354+0000 7f47296ee700 20 client.49957.objecter _get_session s=0x7f4710005410 osd=0 3 2024-04-14T15:26:39.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.354+0000 7f47296ee700 20 client.49957.objecter put_session s=0x7f4710005410 osd=0 3 2024-04-14T15:26:39.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.354+0000 7f47296ee700 20 client.49957.objecter _get_session s=0x7f4710005410 osd=0 3 2024-04-14T15:26:39.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.354+0000 7f47296ee700 15 client.49957.objecter _session_command_op_remove -1 1 2024-04-14T15:26:39.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.354+0000 7f47296ee700 20 client.49957.objecter get_session s=0x7f4710005410 osd=0 3 2024-04-14T15:26:39.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.354+0000 7f47296ee700 15 client.49957.objecter _session_command_op_assign 0 1 2024-04-14T15:26:39.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.354+0000 7f47296ee700 20 client.49957.objecter put_session s=0x7f4710005410 osd=0 4 2024-04-14T15:26:39.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.354+0000 7f47296ee700 10 client.49957.objecter _send_command 1 2024-04-14T15:26:39.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.354+0000 7f47296ee700 1 -- 172.21.15.90:0/1180320903 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4710004b80 con 0x7f4710004760 2024-04-14T15:26:39.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.354+0000 7f470bfff700 10 client.49957.objecter ms_handle_connect 0x7f4710004760 2024-04-14T15:26:39.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.355+0000 7f470bfff700 1 -- 172.21.15.90:0/1180320903 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f4710004b80 con 0x7f4710004760 2024-04-14T15:26:39.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.355+0000 7f470bfff700 10 client.49957.objecter ms_dispatch 0x7f4724001550 command_reply(tid 1: 0 ) 2024-04-14T15:26:39.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.355+0000 7f470bfff700 10 client.49957.objecter _finish_command 1 = system:0 2024-04-14T15:26:39.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.355+0000 7f470bfff700 20 client.49957.objecter put_session s=0x7f4710005410 osd=0 4 2024-04-14T15:26:39.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.355+0000 7f470bfff700 15 client.49957.objecter _session_command_op_remove 0 1 2024-04-14T15:26:39.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.371+0000 7f47296ee700 10 client.49957.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:26:39.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.371+0000 7f47296ee700 15 client.49957.objecter _session_command_op_assign -1 2 2024-04-14T15:26:39.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.371+0000 7f47296ee700 20 client.49957.objecter _get_session s=0x7f4710005410 osd=0 3 2024-04-14T15:26:39.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.371+0000 7f47296ee700 20 client.49957.objecter put_session s=0x7f4710005410 osd=0 3 2024-04-14T15:26:39.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.371+0000 7f47296ee700 20 client.49957.objecter _get_session s=0x7f4710005410 osd=0 3 2024-04-14T15:26:39.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.371+0000 7f47296ee700 15 client.49957.objecter _session_command_op_remove -1 2 2024-04-14T15:26:39.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.371+0000 7f47296ee700 20 client.49957.objecter get_session s=0x7f4710005410 osd=0 3 2024-04-14T15:26:39.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.371+0000 7f47296ee700 15 client.49957.objecter _session_command_op_assign 0 2 2024-04-14T15:26:39.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.371+0000 7f47296ee700 20 client.49957.objecter put_session s=0x7f4710005410 osd=0 4 2024-04-14T15:26:39.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.371+0000 7f47296ee700 10 client.49957.objecter _send_command 2 2024-04-14T15:26:39.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.371+0000 7f47296ee700 1 -- 172.21.15.90:0/1180320903 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4710007a20 con 0x7f4710004760 2024-04-14T15:26:39.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.373+0000 7f470bfff700 1 -- 172.21.15.90:0/1180320903 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f4710007a20 con 0x7f4710004760 2024-04-14T15:26:39.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.373+0000 7f470bfff700 10 client.49957.objecter ms_dispatch 0x7f4724001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:26:39.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.373+0000 7f470bfff700 10 client.49957.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:26:39.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.373+0000 7f470bfff700 20 client.49957.objecter put_session s=0x7f4710005410 osd=0 4 2024-04-14T15:26:39.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.373+0000 7f470bfff700 15 client.49957.objecter _session_command_op_remove 0 2 2024-04-14T15:26:39.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.373+0000 7f47296ee700 10 client.49957.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:26:39.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.373+0000 7f47296ee700 15 client.49957.objecter _session_command_op_assign -1 3 2024-04-14T15:26:39.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.373+0000 7f47296ee700 20 client.49957.objecter _get_session s=0x7f4710004140 osd=1 3 2024-04-14T15:26:39.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.373+0000 7f47296ee700 20 client.49957.objecter put_session s=0x7f4710004140 osd=1 3 2024-04-14T15:26:39.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.373+0000 7f47296ee700 20 client.49957.objecter _get_session s=0x7f4710004140 osd=1 3 2024-04-14T15:26:39.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.373+0000 7f47296ee700 15 client.49957.objecter _session_command_op_remove -1 3 2024-04-14T15:26:39.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.373+0000 7f47296ee700 20 client.49957.objecter get_session s=0x7f4710004140 osd=1 3 2024-04-14T15:26:39.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.373+0000 7f47296ee700 15 client.49957.objecter _session_command_op_assign 1 3 2024-04-14T15:26:39.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.373+0000 7f47296ee700 20 client.49957.objecter put_session s=0x7f4710004140 osd=1 4 2024-04-14T15:26:39.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.373+0000 7f47296ee700 10 client.49957.objecter _send_command 3 2024-04-14T15:26:39.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.373+0000 7f47296ee700 1 -- 172.21.15.90:0/1180320903 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f47100021b0 con 0x7f47100042b0 2024-04-14T15:26:39.455 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:26:39.455 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:26:39.455 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:26:39.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.373+0000 7f470bfff700 10 client.49957.objecter ms_handle_connect 0x7f47100042b0 2024-04-14T15:26:39.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.374+0000 7f470bfff700 1 -- 172.21.15.90:0/1180320903 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f47100021b0 con 0x7f47100042b0 2024-04-14T15:26:39.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.374+0000 7f470bfff700 10 client.49957.objecter ms_dispatch 0x7f4724001550 command_reply(tid 3: 0 ) 2024-04-14T15:26:39.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.374+0000 7f470bfff700 10 client.49957.objecter _finish_command 3 = system:0 2024-04-14T15:26:39.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.374+0000 7f470bfff700 20 client.49957.objecter put_session s=0x7f4710004140 osd=1 4 2024-04-14T15:26:39.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.374+0000 7f470bfff700 15 client.49957.objecter _session_command_op_remove 1 3 2024-04-14T15:26:39.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.391+0000 7f47296ee700 10 client.49957.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:26:39.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.391+0000 7f47296ee700 15 client.49957.objecter _session_command_op_assign -1 4 2024-04-14T15:26:39.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.391+0000 7f47296ee700 20 client.49957.objecter _get_session s=0x7f4710004140 osd=1 3 2024-04-14T15:26:39.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.391+0000 7f47296ee700 20 client.49957.objecter put_session s=0x7f4710004140 osd=1 3 2024-04-14T15:26:39.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.391+0000 7f47296ee700 20 client.49957.objecter _get_session s=0x7f4710004140 osd=1 3 2024-04-14T15:26:39.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.391+0000 7f47296ee700 15 client.49957.objecter _session_command_op_remove -1 4 2024-04-14T15:26:39.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.391+0000 7f47296ee700 20 client.49957.objecter get_session s=0x7f4710004140 osd=1 3 2024-04-14T15:26:39.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.391+0000 7f47296ee700 15 client.49957.objecter _session_command_op_assign 1 4 2024-04-14T15:26:39.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.391+0000 7f47296ee700 20 client.49957.objecter put_session s=0x7f4710004140 osd=1 4 2024-04-14T15:26:39.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.391+0000 7f47296ee700 10 client.49957.objecter _send_command 4 2024-04-14T15:26:39.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.391+0000 7f47296ee700 1 -- 172.21.15.90:0/1180320903 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4710001630 con 0x7f47100042b0 2024-04-14T15:26:39.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.392+0000 7f470bfff700 1 -- 172.21.15.90:0/1180320903 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f4710001630 con 0x7f47100042b0 2024-04-14T15:26:39.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.392+0000 7f470bfff700 10 client.49957.objecter ms_dispatch 0x7f4724001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:26:39.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.392+0000 7f470bfff700 10 client.49957.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:26:39.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.392+0000 7f470bfff700 20 client.49957.objecter put_session s=0x7f4710004140 osd=1 4 2024-04-14T15:26:39.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.392+0000 7f470bfff700 15 client.49957.objecter _session_command_op_remove 1 4 2024-04-14T15:26:39.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.392+0000 7f47296ee700 10 client.49957.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:26:39.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.392+0000 7f47296ee700 15 client.49957.objecter _session_command_op_assign -1 5 2024-04-14T15:26:39.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.392+0000 7f47296ee700 20 client.49957.objecter _get_session s=0x7f4710011060 osd=2 3 2024-04-14T15:26:39.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.392+0000 7f47296ee700 20 client.49957.objecter put_session s=0x7f4710011060 osd=2 3 2024-04-14T15:26:39.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.392+0000 7f47296ee700 20 client.49957.objecter _get_session s=0x7f4710011060 osd=2 3 2024-04-14T15:26:39.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.392+0000 7f47296ee700 15 client.49957.objecter _session_command_op_remove -1 5 2024-04-14T15:26:39.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.392+0000 7f47296ee700 20 client.49957.objecter get_session s=0x7f4710011060 osd=2 3 2024-04-14T15:26:39.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.392+0000 7f47296ee700 15 client.49957.objecter _session_command_op_assign 2 5 2024-04-14T15:26:39.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.392+0000 7f47296ee700 20 client.49957.objecter put_session s=0x7f4710011060 osd=2 4 2024-04-14T15:26:39.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.392+0000 7f47296ee700 10 client.49957.objecter _send_command 5 2024-04-14T15:26:39.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.392+0000 7f47296ee700 1 -- 172.21.15.90:0/1180320903 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f4710015390 con 0x7f47100116e0 2024-04-14T15:26:39.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.393+0000 7f470bfff700 10 client.49957.objecter ms_handle_connect 0x7f47100116e0 2024-04-14T15:26:39.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.393+0000 7f470bfff700 1 -- 172.21.15.90:0/1180320903 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f4710015390 con 0x7f47100116e0 2024-04-14T15:26:39.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.393+0000 7f470bfff700 10 client.49957.objecter ms_dispatch 0x7f4724001550 command_reply(tid 5: 0 ) 2024-04-14T15:26:39.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.393+0000 7f470bfff700 10 client.49957.objecter _finish_command 5 = system:0 2024-04-14T15:26:39.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.393+0000 7f470bfff700 20 client.49957.objecter put_session s=0x7f4710011060 osd=2 4 2024-04-14T15:26:39.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.393+0000 7f470bfff700 15 client.49957.objecter _session_command_op_remove 2 5 2024-04-14T15:26:39.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.411+0000 7f47296ee700 10 client.49957.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:26:39.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.411+0000 7f47296ee700 15 client.49957.objecter _session_command_op_assign -1 6 2024-04-14T15:26:39.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.411+0000 7f47296ee700 20 client.49957.objecter _get_session s=0x7f4710011060 osd=2 3 2024-04-14T15:26:39.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.411+0000 7f47296ee700 20 client.49957.objecter put_session s=0x7f4710011060 osd=2 3 2024-04-14T15:26:39.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.411+0000 7f47296ee700 20 client.49957.objecter _get_session s=0x7f4710011060 osd=2 3 2024-04-14T15:26:39.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.411+0000 7f47296ee700 15 client.49957.objecter _session_command_op_remove -1 6 2024-04-14T15:26:39.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.411+0000 7f47296ee700 20 client.49957.objecter get_session s=0x7f4710011060 osd=2 3 2024-04-14T15:26:39.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.411+0000 7f47296ee700 15 client.49957.objecter _session_command_op_assign 2 6 2024-04-14T15:26:39.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.411+0000 7f47296ee700 20 client.49957.objecter put_session s=0x7f4710011060 osd=2 4 2024-04-14T15:26:39.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.411+0000 7f47296ee700 10 client.49957.objecter _send_command 6 2024-04-14T15:26:39.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.411+0000 7f47296ee700 1 -- 172.21.15.90:0/1180320903 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4710009770 con 0x7f47100116e0 2024-04-14T15:26:39.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.412+0000 7f470bfff700 1 -- 172.21.15.90:0/1180320903 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f4710009770 con 0x7f47100116e0 2024-04-14T15:26:39.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.412+0000 7f470bfff700 10 client.49957.objecter ms_dispatch 0x7f4724001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:26:39.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.412+0000 7f470bfff700 10 client.49957.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:26:39.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.412+0000 7f470bfff700 20 client.49957.objecter put_session s=0x7f4710011060 osd=2 4 2024-04-14T15:26:39.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.412+0000 7f470bfff700 15 client.49957.objecter _session_command_op_remove 2 6 2024-04-14T15:26:39.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.412+0000 7f47296ee700 10 client.49957.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:26:39.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.412+0000 7f47296ee700 15 client.49957.objecter _session_command_op_assign -1 7 2024-04-14T15:26:39.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.412+0000 7f47296ee700 20 client.49957.objecter _get_session s=0x7f4710009a00 osd=3 3 2024-04-14T15:26:39.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.412+0000 7f47296ee700 20 client.49957.objecter put_session s=0x7f4710009a00 osd=3 3 2024-04-14T15:26:39.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.412+0000 7f47296ee700 20 client.49957.objecter _get_session s=0x7f4710009a00 osd=3 3 2024-04-14T15:26:39.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.412+0000 7f47296ee700 15 client.49957.objecter _session_command_op_remove -1 7 2024-04-14T15:26:39.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.412+0000 7f47296ee700 20 client.49957.objecter get_session s=0x7f4710009a00 osd=3 3 2024-04-14T15:26:39.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.412+0000 7f47296ee700 15 client.49957.objecter _session_command_op_assign 3 7 2024-04-14T15:26:39.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.412+0000 7f47296ee700 20 client.49957.objecter put_session s=0x7f4710009a00 osd=3 4 2024-04-14T15:26:39.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.412+0000 7f47296ee700 10 client.49957.objecter _send_command 7 2024-04-14T15:26:39.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.412+0000 7f47296ee700 1 -- 172.21.15.90:0/1180320903 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f47100160e0 con 0x7f4710015570 2024-04-14T15:26:39.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.413+0000 7f470bfff700 10 client.49957.objecter ms_handle_connect 0x7f4710015570 2024-04-14T15:26:39.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.413+0000 7f470bfff700 1 -- 172.21.15.90:0/1180320903 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f47100160e0 con 0x7f4710015570 2024-04-14T15:26:39.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.413+0000 7f470bfff700 10 client.49957.objecter ms_dispatch 0x7f4724001550 command_reply(tid 7: 0 ) 2024-04-14T15:26:39.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.413+0000 7f470bfff700 10 client.49957.objecter _finish_command 7 = system:0 2024-04-14T15:26:39.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.413+0000 7f470bfff700 20 client.49957.objecter put_session s=0x7f4710009a00 osd=3 4 2024-04-14T15:26:39.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.413+0000 7f470bfff700 15 client.49957.objecter _session_command_op_remove 3 7 2024-04-14T15:26:39.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.432+0000 7f47296ee700 10 client.49957.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:26:39.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.432+0000 7f47296ee700 15 client.49957.objecter _session_command_op_assign -1 8 2024-04-14T15:26:39.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.432+0000 7f47296ee700 20 client.49957.objecter _get_session s=0x7f4710009a00 osd=3 3 2024-04-14T15:26:39.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.432+0000 7f47296ee700 20 client.49957.objecter put_session s=0x7f4710009a00 osd=3 3 2024-04-14T15:26:39.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.432+0000 7f47296ee700 20 client.49957.objecter _get_session s=0x7f4710009a00 osd=3 3 2024-04-14T15:26:39.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.432+0000 7f47296ee700 15 client.49957.objecter _session_command_op_remove -1 8 2024-04-14T15:26:39.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.432+0000 7f47296ee700 20 client.49957.objecter get_session s=0x7f4710009a00 osd=3 3 2024-04-14T15:26:39.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.432+0000 7f47296ee700 15 client.49957.objecter _session_command_op_assign 3 8 2024-04-14T15:26:39.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.432+0000 7f47296ee700 20 client.49957.objecter put_session s=0x7f4710009a00 osd=3 4 2024-04-14T15:26:39.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.432+0000 7f47296ee700 10 client.49957.objecter _send_command 8 2024-04-14T15:26:39.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.432+0000 7f47296ee700 1 -- 172.21.15.90:0/1180320903 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4710017c10 con 0x7f4710015570 2024-04-14T15:26:39.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.434+0000 7f470bfff700 1 -- 172.21.15.90:0/1180320903 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f4710017c10 con 0x7f4710015570 2024-04-14T15:26:39.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.434+0000 7f470bfff700 10 client.49957.objecter ms_dispatch 0x7f4724001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:26:39.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.434+0000 7f470bfff700 10 client.49957.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:26:39.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.434+0000 7f470bfff700 20 client.49957.objecter put_session s=0x7f4710009a00 osd=3 4 2024-04-14T15:26:39.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.434+0000 7f470bfff700 15 client.49957.objecter _session_command_op_remove 3 8 2024-04-14T15:26:39.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.434+0000 7f47296ee700 10 client.49957.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:26:39.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.434+0000 7f47296ee700 15 client.49957.objecter _session_command_op_assign -1 9 2024-04-14T15:26:39.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.434+0000 7f47296ee700 20 client.49957.objecter _get_session s=0x7f4710017c10 osd=4 3 2024-04-14T15:26:39.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.434+0000 7f47296ee700 20 client.49957.objecter put_session s=0x7f4710017c10 osd=4 3 2024-04-14T15:26:39.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.434+0000 7f47296ee700 20 client.49957.objecter _get_session s=0x7f4710017c10 osd=4 3 2024-04-14T15:26:39.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.434+0000 7f47296ee700 15 client.49957.objecter _session_command_op_remove -1 9 2024-04-14T15:26:39.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.434+0000 7f47296ee700 20 client.49957.objecter get_session s=0x7f4710017c10 osd=4 3 2024-04-14T15:26:39.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.434+0000 7f47296ee700 15 client.49957.objecter _session_command_op_assign 4 9 2024-04-14T15:26:39.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.434+0000 7f47296ee700 20 client.49957.objecter put_session s=0x7f4710017c10 osd=4 4 2024-04-14T15:26:39.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.434+0000 7f47296ee700 10 client.49957.objecter _send_command 9 2024-04-14T15:26:39.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.434+0000 7f47296ee700 1 -- 172.21.15.90:0/1180320903 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f471001eb50 con 0x7f471001af50 2024-04-14T15:26:39.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.434+0000 7f470bfff700 10 client.49957.objecter ms_handle_connect 0x7f471001af50 2024-04-14T15:26:39.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.435+0000 7f470bfff700 1 -- 172.21.15.90:0/1180320903 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f471001eb50 con 0x7f471001af50 2024-04-14T15:26:39.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.435+0000 7f470bfff700 10 client.49957.objecter ms_dispatch 0x7f4724001550 command_reply(tid 9: 0 ) 2024-04-14T15:26:39.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.435+0000 7f470bfff700 10 client.49957.objecter _finish_command 9 = system:0 2024-04-14T15:26:39.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.435+0000 7f470bfff700 20 client.49957.objecter put_session s=0x7f4710017c10 osd=4 4 2024-04-14T15:26:39.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.435+0000 7f470bfff700 15 client.49957.objecter _session_command_op_remove 4 9 2024-04-14T15:26:39.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.452+0000 7f47296ee700 10 client.49957.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:26:39.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.452+0000 7f47296ee700 15 client.49957.objecter _session_command_op_assign -1 10 2024-04-14T15:26:39.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.452+0000 7f47296ee700 20 client.49957.objecter _get_session s=0x7f4710017c10 osd=4 3 2024-04-14T15:26:39.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.452+0000 7f47296ee700 20 client.49957.objecter put_session s=0x7f4710017c10 osd=4 3 2024-04-14T15:26:39.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.452+0000 7f47296ee700 20 client.49957.objecter _get_session s=0x7f4710017c10 osd=4 3 2024-04-14T15:26:39.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.452+0000 7f47296ee700 15 client.49957.objecter _session_command_op_remove -1 10 2024-04-14T15:26:39.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.452+0000 7f47296ee700 20 client.49957.objecter get_session s=0x7f4710017c10 osd=4 3 2024-04-14T15:26:39.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.452+0000 7f47296ee700 15 client.49957.objecter _session_command_op_assign 4 10 2024-04-14T15:26:39.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.452+0000 7f47296ee700 20 client.49957.objecter put_session s=0x7f4710017c10 osd=4 4 2024-04-14T15:26:39.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.452+0000 7f47296ee700 10 client.49957.objecter _send_command 10 2024-04-14T15:26:39.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.452+0000 7f47296ee700 1 -- 172.21.15.90:0/1180320903 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4710020f20 con 0x7f471001af50 2024-04-14T15:26:39.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.454+0000 7f470bfff700 1 -- 172.21.15.90:0/1180320903 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f4710020f20 con 0x7f471001af50 2024-04-14T15:26:39.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.454+0000 7f470bfff700 10 client.49957.objecter ms_dispatch 0x7f4724001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:26:39.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.454+0000 7f470bfff700 10 client.49957.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:26:39.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.454+0000 7f470bfff700 20 client.49957.objecter put_session s=0x7f4710017c10 osd=4 4 2024-04-14T15:26:39.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.454+0000 7f470bfff700 15 client.49957.objecter _session_command_op_remove 4 10 2024-04-14T15:26:39.464 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:26:39.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.454+0000 7f47296ee700 10 client.49957.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:26:39.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.454+0000 7f47296ee700 15 client.49957.objecter _session_command_op_assign -1 11 2024-04-14T15:26:39.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.454+0000 7f47296ee700 20 client.49957.objecter _get_session s=0x7f4710007f40 osd=5 3 2024-04-14T15:26:39.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.454+0000 7f47296ee700 20 client.49957.objecter put_session s=0x7f4710007f40 osd=5 3 2024-04-14T15:26:39.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.454+0000 7f47296ee700 20 client.49957.objecter _get_session s=0x7f4710007f40 osd=5 3 2024-04-14T15:26:39.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.454+0000 7f47296ee700 15 client.49957.objecter _session_command_op_remove -1 11 2024-04-14T15:26:39.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.454+0000 7f47296ee700 20 client.49957.objecter get_session s=0x7f4710007f40 osd=5 3 2024-04-14T15:26:39.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.454+0000 7f47296ee700 15 client.49957.objecter _session_command_op_assign 5 11 2024-04-14T15:26:39.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.454+0000 7f47296ee700 20 client.49957.objecter put_session s=0x7f4710007f40 osd=5 4 2024-04-14T15:26:39.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.454+0000 7f47296ee700 10 client.49957.objecter _send_command 11 2024-04-14T15:26:39.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.454+0000 7f47296ee700 1 -- 172.21.15.90:0/1180320903 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f4710020270 con 0x7f4710020cd0 2024-04-14T15:26:39.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.455+0000 7f470bfff700 10 client.49957.objecter ms_handle_connect 0x7f4710020cd0 2024-04-14T15:26:39.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.455+0000 7f470bfff700 1 -- 172.21.15.90:0/1180320903 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f4710020270 con 0x7f4710020cd0 2024-04-14T15:26:39.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.455+0000 7f470bfff700 10 client.49957.objecter ms_dispatch 0x7f4724001550 command_reply(tid 11: 0 ) 2024-04-14T15:26:39.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.455+0000 7f470bfff700 10 client.49957.objecter _finish_command 11 = system:0 2024-04-14T15:26:39.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.455+0000 7f470bfff700 20 client.49957.objecter put_session s=0x7f4710007f40 osd=5 4 2024-04-14T15:26:39.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.455+0000 7f470bfff700 15 client.49957.objecter _session_command_op_remove 5 11 2024-04-14T15:26:39.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.475+0000 7f47296ee700 10 client.49957.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:26:39.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.475+0000 7f47296ee700 15 client.49957.objecter _session_command_op_assign -1 12 2024-04-14T15:26:39.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.475+0000 7f47296ee700 20 client.49957.objecter _get_session s=0x7f4710007f40 osd=5 3 2024-04-14T15:26:39.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.475+0000 7f47296ee700 20 client.49957.objecter put_session s=0x7f4710007f40 osd=5 3 2024-04-14T15:26:39.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.475+0000 7f47296ee700 20 client.49957.objecter _get_session s=0x7f4710007f40 osd=5 3 2024-04-14T15:26:39.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.475+0000 7f47296ee700 15 client.49957.objecter _session_command_op_remove -1 12 2024-04-14T15:26:39.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.475+0000 7f47296ee700 20 client.49957.objecter get_session s=0x7f4710007f40 osd=5 3 2024-04-14T15:26:39.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.475+0000 7f47296ee700 15 client.49957.objecter _session_command_op_assign 5 12 2024-04-14T15:26:39.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.475+0000 7f47296ee700 20 client.49957.objecter put_session s=0x7f4710007f40 osd=5 4 2024-04-14T15:26:39.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.475+0000 7f47296ee700 10 client.49957.objecter _send_command 12 2024-04-14T15:26:39.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.475+0000 7f47296ee700 1 -- 172.21.15.90:0/1180320903 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4710020270 con 0x7f4710020cd0 2024-04-14T15:26:39.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.477+0000 7f470bfff700 1 -- 172.21.15.90:0/1180320903 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f4710020270 con 0x7f4710020cd0 2024-04-14T15:26:39.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.477+0000 7f470bfff700 10 client.49957.objecter ms_dispatch 0x7f4724001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:26:39.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.477+0000 7f470bfff700 10 client.49957.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:26:39.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.477+0000 7f470bfff700 20 client.49957.objecter put_session s=0x7f4710007f40 osd=5 4 2024-04-14T15:26:39.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.477+0000 7f470bfff700 15 client.49957.objecter _session_command_op_remove 5 12 2024-04-14T15:26:39.479 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:26:39.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.478+0000 7f47296ee700 10 client.49957.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:26:39.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.478+0000 7f47296ee700 15 client.49957.objecter _session_command_op_assign -1 13 2024-04-14T15:26:39.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.478+0000 7f47296ee700 20 client.49957.objecter _get_session s=0x7f4710020270 osd=6 3 2024-04-14T15:26:39.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.478+0000 7f47296ee700 20 client.49957.objecter put_session s=0x7f4710020270 osd=6 3 2024-04-14T15:26:39.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.478+0000 7f47296ee700 20 client.49957.objecter _get_session s=0x7f4710020270 osd=6 3 2024-04-14T15:26:39.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.478+0000 7f47296ee700 15 client.49957.objecter _session_command_op_remove -1 13 2024-04-14T15:26:39.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.478+0000 7f47296ee700 20 client.49957.objecter get_session s=0x7f4710020270 osd=6 3 2024-04-14T15:26:39.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.478+0000 7f47296ee700 15 client.49957.objecter _session_command_op_assign 6 13 2024-04-14T15:26:39.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.478+0000 7f47296ee700 20 client.49957.objecter put_session s=0x7f4710020270 osd=6 4 2024-04-14T15:26:39.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.478+0000 7f47296ee700 10 client.49957.objecter _send_command 13 2024-04-14T15:26:39.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.478+0000 7f47296ee700 1 -- 172.21.15.90:0/1180320903 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f4710021420 con 0x7f4710023c10 2024-04-14T15:26:39.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.478+0000 7f470bfff700 10 client.49957.objecter ms_handle_connect 0x7f4710023c10 2024-04-14T15:26:39.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.479+0000 7f470bfff700 1 -- 172.21.15.90:0/1180320903 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f4710021420 con 0x7f4710023c10 2024-04-14T15:26:39.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.479+0000 7f470bfff700 10 client.49957.objecter ms_dispatch 0x7f4724001550 command_reply(tid 13: 0 ) 2024-04-14T15:26:39.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.479+0000 7f470bfff700 10 client.49957.objecter _finish_command 13 = system:0 2024-04-14T15:26:39.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.479+0000 7f470bfff700 20 client.49957.objecter put_session s=0x7f4710020270 osd=6 4 2024-04-14T15:26:39.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.479+0000 7f470bfff700 15 client.49957.objecter _session_command_op_remove 6 13 2024-04-14T15:26:39.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.496+0000 7f47296ee700 10 client.49957.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:26:39.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.496+0000 7f47296ee700 15 client.49957.objecter _session_command_op_assign -1 14 2024-04-14T15:26:39.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.496+0000 7f47296ee700 20 client.49957.objecter _get_session s=0x7f4710020270 osd=6 3 2024-04-14T15:26:39.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.496+0000 7f47296ee700 20 client.49957.objecter put_session s=0x7f4710020270 osd=6 3 2024-04-14T15:26:39.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.496+0000 7f47296ee700 20 client.49957.objecter _get_session s=0x7f4710020270 osd=6 3 2024-04-14T15:26:39.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.496+0000 7f47296ee700 15 client.49957.objecter _session_command_op_remove -1 14 2024-04-14T15:26:39.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.496+0000 7f47296ee700 20 client.49957.objecter get_session s=0x7f4710020270 osd=6 3 2024-04-14T15:26:39.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.496+0000 7f47296ee700 15 client.49957.objecter _session_command_op_assign 6 14 2024-04-14T15:26:39.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.496+0000 7f47296ee700 20 client.49957.objecter put_session s=0x7f4710020270 osd=6 4 2024-04-14T15:26:39.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.496+0000 7f47296ee700 10 client.49957.objecter _send_command 14 2024-04-14T15:26:39.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.496+0000 7f47296ee700 1 -- 172.21.15.90:0/1180320903 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4710027720 con 0x7f4710023c10 2024-04-14T15:26:39.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.497+0000 7f470bfff700 1 -- 172.21.15.90:0/1180320903 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f4710027720 con 0x7f4710023c10 2024-04-14T15:26:39.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.497+0000 7f470bfff700 10 client.49957.objecter ms_dispatch 0x7f4724001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:26:39.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.497+0000 7f470bfff700 10 client.49957.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:26:39.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.497+0000 7f470bfff700 20 client.49957.objecter put_session s=0x7f4710020270 osd=6 4 2024-04-14T15:26:39.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.497+0000 7f470bfff700 15 client.49957.objecter _session_command_op_remove 6 14 2024-04-14T15:26:39.498 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:26:39.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.498+0000 7f47296ee700 10 client.49957.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:26:39.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.498+0000 7f47296ee700 15 client.49957.objecter _session_command_op_assign -1 15 2024-04-14T15:26:39.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.498+0000 7f47296ee700 20 client.49957.objecter _get_session s=0x7f4710027720 osd=7 3 2024-04-14T15:26:39.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.498+0000 7f47296ee700 20 client.49957.objecter put_session s=0x7f4710027720 osd=7 3 2024-04-14T15:26:39.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.498+0000 7f47296ee700 20 client.49957.objecter _get_session s=0x7f4710027720 osd=7 3 2024-04-14T15:26:39.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.498+0000 7f47296ee700 15 client.49957.objecter _session_command_op_remove -1 15 2024-04-14T15:26:39.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.498+0000 7f47296ee700 20 client.49957.objecter get_session s=0x7f4710027720 osd=7 3 2024-04-14T15:26:39.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.498+0000 7f47296ee700 15 client.49957.objecter _session_command_op_assign 7 15 2024-04-14T15:26:39.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.498+0000 7f47296ee700 20 client.49957.objecter put_session s=0x7f4710027720 osd=7 4 2024-04-14T15:26:39.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.498+0000 7f47296ee700 10 client.49957.objecter _send_command 15 2024-04-14T15:26:39.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.498+0000 7f47296ee700 1 -- 172.21.15.90:0/1180320903 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f471002b950 con 0x7f4710027da0 2024-04-14T15:26:39.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.499+0000 7f470bfff700 10 client.49957.objecter ms_handle_connect 0x7f4710027da0 2024-04-14T15:26:39.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.499+0000 7f470bfff700 1 -- 172.21.15.90:0/1180320903 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f471002b950 con 0x7f4710027da0 2024-04-14T15:26:39.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.499+0000 7f470bfff700 10 client.49957.objecter ms_dispatch 0x7f4724001550 command_reply(tid 15: 0 ) 2024-04-14T15:26:39.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.499+0000 7f470bfff700 10 client.49957.objecter _finish_command 15 = system:0 2024-04-14T15:26:39.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.499+0000 7f470bfff700 20 client.49957.objecter put_session s=0x7f4710027720 osd=7 4 2024-04-14T15:26:39.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.499+0000 7f470bfff700 15 client.49957.objecter _session_command_op_remove 7 15 2024-04-14T15:26:39.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.517+0000 7f47296ee700 10 client.49957.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:26:39.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.517+0000 7f47296ee700 15 client.49957.objecter _session_command_op_assign -1 16 2024-04-14T15:26:39.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.517+0000 7f47296ee700 20 client.49957.objecter _get_session s=0x7f4710027720 osd=7 3 2024-04-14T15:26:39.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.517+0000 7f47296ee700 20 client.49957.objecter put_session s=0x7f4710027720 osd=7 3 2024-04-14T15:26:39.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.517+0000 7f47296ee700 20 client.49957.objecter _get_session s=0x7f4710027720 osd=7 3 2024-04-14T15:26:39.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.517+0000 7f47296ee700 15 client.49957.objecter _session_command_op_remove -1 16 2024-04-14T15:26:39.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.517+0000 7f47296ee700 20 client.49957.objecter get_session s=0x7f4710027720 osd=7 3 2024-04-14T15:26:39.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.517+0000 7f47296ee700 15 client.49957.objecter _session_command_op_assign 7 16 2024-04-14T15:26:39.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.517+0000 7f47296ee700 20 client.49957.objecter put_session s=0x7f4710027720 osd=7 4 2024-04-14T15:26:39.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.517+0000 7f47296ee700 10 client.49957.objecter _send_command 16 2024-04-14T15:26:39.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.517+0000 7f47296ee700 1 -- 172.21.15.90:0/1180320903 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f47100206a0 con 0x7f4710027da0 2024-04-14T15:26:39.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.519+0000 7f470bfff700 1 -- 172.21.15.90:0/1180320903 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f47100206a0 con 0x7f4710027da0 2024-04-14T15:26:39.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.519+0000 7f470bfff700 10 client.49957.objecter ms_dispatch 0x7f4724001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:26:39.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.519+0000 7f470bfff700 10 client.49957.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:26:39.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.519+0000 7f470bfff700 20 client.49957.objecter put_session s=0x7f4710027720 osd=7 4 2024-04-14T15:26:39.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.519+0000 7f470bfff700 15 client.49957.objecter _session_command_op_remove 7 16 2024-04-14T15:26:39.520 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:26:39.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.519+0000 7f47296ee700 10 librados: watch_flush enter 2024-04-14T15:26:39.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.519+0000 7f47296ee700 10 librados: watch_flush exit 2024-04-14T15:26:39.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.519+0000 7f47296ee700 10 client.49957.objecter close_session for osd.0 2024-04-14T15:26:39.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.519+0000 7f47296ee700 1 -- 172.21.15.90:0/1180320903 >> v1:172.21.15.90:6801/3761806885 conn(0x7f4710004760 legacy=0x7f4710002780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:39.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.519+0000 7f47296ee700 20 client.49957.objecter put_session s=0x7f4710005410 osd=0 1 2024-04-14T15:26:39.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.519+0000 7f47296ee700 10 client.49957.objecter close_session for osd.1 2024-04-14T15:26:39.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.519+0000 7f47296ee700 1 -- 172.21.15.90:0/1180320903 >> v1:172.21.15.90:6805/4170361699 conn(0x7f47100042b0 legacy=0x7f4710008d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:39.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.519+0000 7f47296ee700 20 client.49957.objecter put_session s=0x7f4710004140 osd=1 1 2024-04-14T15:26:39.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.519+0000 7f47296ee700 10 client.49957.objecter close_session for osd.2 2024-04-14T15:26:39.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.519+0000 7f47296ee700 1 -- 172.21.15.90:0/1180320903 >> v1:172.21.15.90:6809/1686184251 conn(0x7f47100116e0 legacy=0x7f4710013ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:39.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.520+0000 7f47237fe700 1 -- 172.21.15.90:0/1180320903 reap_dead start 2024-04-14T15:26:39.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.520+0000 7f47296ee700 20 client.49957.objecter put_session s=0x7f4710011060 osd=2 1 2024-04-14T15:26:39.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.520+0000 7f47296ee700 10 client.49957.objecter close_session for osd.3 2024-04-14T15:26:39.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.520+0000 7f47296ee700 1 -- 172.21.15.90:0/1180320903 >> v1:172.21.15.90:6813/2270166774 conn(0x7f4710015570 legacy=0x7f47100159d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:39.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.520+0000 7f47296ee700 20 client.49957.objecter put_session s=0x7f4710009a00 osd=3 1 2024-04-14T15:26:39.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.520+0000 7f47296ee700 10 client.49957.objecter close_session for osd.4 2024-04-14T15:26:39.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.520+0000 7f47296ee700 1 -- 172.21.15.90:0/1180320903 >> v1:172.21.15.92:6800/3887549733 conn(0x7f471001af50 legacy=0x7f471001d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:39.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.520+0000 7f47296ee700 20 client.49957.objecter put_session s=0x7f4710017c10 osd=4 1 2024-04-14T15:26:39.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.520+0000 7f47296ee700 10 client.49957.objecter close_session for osd.5 2024-04-14T15:26:39.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.520+0000 7f47296ee700 1 -- 172.21.15.90:0/1180320903 >> v1:172.21.15.92:6804/3244561440 conn(0x7f4710020cd0 legacy=0x7f471001eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:39.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.520+0000 7f47296ee700 20 client.49957.objecter put_session s=0x7f4710007f40 osd=5 1 2024-04-14T15:26:39.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.520+0000 7f47296ee700 10 client.49957.objecter close_session for osd.6 2024-04-14T15:26:39.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.520+0000 7f47296ee700 1 -- 172.21.15.90:0/1180320903 >> v1:172.21.15.92:6808/636125963 conn(0x7f4710023c10 legacy=0x7f4710026000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:39.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.520+0000 7f47296ee700 20 client.49957.objecter put_session s=0x7f4710020270 osd=6 1 2024-04-14T15:26:39.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.520+0000 7f47296ee700 10 client.49957.objecter close_session for osd.7 2024-04-14T15:26:39.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.520+0000 7f47296ee700 1 -- 172.21.15.90:0/1180320903 >> v1:172.21.15.92:6812/1773706716 conn(0x7f4710027da0 legacy=0x7f471002a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:39.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.520+0000 7f47296ee700 20 client.49957.objecter put_session s=0x7f4710027720 osd=7 1 2024-04-14T15:26:39.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.520+0000 7f47296ee700 20 client.49957.objecter shutdown clearing up homeless session... 2024-04-14T15:26:39.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.520+0000 7f47296ee700 10 client.49957.objecter successfully canceled tick 2024-04-14T15:26:39.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.520+0000 7f47237fe700 1 -- 172.21.15.90:0/1180320903 reap_dead start 2024-04-14T15:26:39.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.520+0000 7f47296ee700 1 -- 172.21.15.90:0/1180320903 >> v1:172.21.15.90:6800/420935427 conn(0x7f470c0713a0 legacy=0x7f470c073860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:39.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.520+0000 7f47296ee700 1 -- 172.21.15.90:0/1180320903 >> v1:172.21.15.92:6789/0 conn(0x7f4724107760 legacy=0x7f472410c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:39.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.520+0000 7f47296ee700 1 -- 172.21.15.90:0/1180320903 shutdown_connections 2024-04-14T15:26:39.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.520+0000 7f47296ee700 1 -- 172.21.15.90:0/1180320903 >> 172.21.15.90:0/1180320903 conn(0x7f47240fd290 msgr2=0x7f47241117b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:26:39.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.520+0000 7f47296ee700 1 -- 172.21.15.90:0/1180320903 shutdown_connections 2024-04-14T15:26:39.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.520+0000 7f47296ee700 1 -- 172.21.15.90:0/1180320903 wait complete. 2024-04-14T15:26:39.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:39.520+0000 7f47296ee700 1 librados: shutdown 2024-04-14T15:26:39.524 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:26:39.524 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:26:39.524 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:26:39.524 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:26:39.525 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:26:39.525 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:26:39.525 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:26:39.525 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:26:39.526 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:26:39.553 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:39.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:39.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:26:39.655 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:26:39.679 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:39.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:39.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:26:39.781 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:26:39.805 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:39.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:39.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:26:39.908 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:26:39.933 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:39.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:39.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:26:40.035 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:26:40.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:40.059 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:40.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:26:40.160 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:26:40.183 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:40.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:40.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:26:40.185 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:40 smithi090 ceph-mon[41670]: osdmap e835: 8 total, 8 up, 7 in 2024-04-14T15:26:40.185 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:40 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:26:40.185 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:40 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:26:40.185 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:40 smithi090 ceph-mon[41670]: pgmap v1247: 192 pgs: 192 active+clean; 4.4 MiB data, 739 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:26:40.185 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:40 smithi090 ceph-mon[41670]: 2.11 scrub starts 2024-04-14T15:26:40.185 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:40 smithi090 ceph-mon[41670]: 2.11 scrub ok 2024-04-14T15:26:40.186 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:40 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1180320903' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:26:40.186 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:40 smithi090 ceph-mon[41670]: 6.14 deep-scrub starts 2024-04-14T15:26:40.186 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:40 smithi090 ceph-mon[41670]: 6.14 deep-scrub ok 2024-04-14T15:26:40.186 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:40 smithi090 ceph-mon[41670]: 6.7 scrub starts 2024-04-14T15:26:40.186 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:40 smithi090 ceph-mon[41670]: 6.7 scrub ok 2024-04-14T15:26:40.186 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:40 smithi090 ceph-mon[30399]: osdmap e835: 8 total, 8 up, 7 in 2024-04-14T15:26:40.186 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:40 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:26:40.186 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:40 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:26:40.186 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:40 smithi090 ceph-mon[30399]: pgmap v1247: 192 pgs: 192 active+clean; 4.4 MiB data, 739 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:26:40.186 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:40 smithi090 ceph-mon[30399]: 2.11 scrub starts 2024-04-14T15:26:40.186 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:40 smithi090 ceph-mon[30399]: 2.11 scrub ok 2024-04-14T15:26:40.186 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:40 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1180320903' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:26:40.186 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:40 smithi090 ceph-mon[30399]: 6.14 deep-scrub starts 2024-04-14T15:26:40.187 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:40 smithi090 ceph-mon[30399]: 6.14 deep-scrub ok 2024-04-14T15:26:40.187 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:40 smithi090 ceph-mon[30399]: 6.7 scrub starts 2024-04-14T15:26:40.187 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:40 smithi090 ceph-mon[30399]: 6.7 scrub ok 2024-04-14T15:26:40.191 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:26:40.285 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:26:40.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:40.309 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:40.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:26:40.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:40 smithi092 ceph-mon[33966]: osdmap e835: 8 total, 8 up, 7 in 2024-04-14T15:26:40.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:40 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:26:40.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:40 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:26:40.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:40 smithi092 ceph-mon[33966]: pgmap v1247: 192 pgs: 192 active+clean; 4.4 MiB data, 739 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:26:40.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:40 smithi092 ceph-mon[33966]: 2.11 scrub starts 2024-04-14T15:26:40.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:40 smithi092 ceph-mon[33966]: 2.11 scrub ok 2024-04-14T15:26:40.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:40 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1180320903' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:26:40.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:40 smithi092 ceph-mon[33966]: 6.14 deep-scrub starts 2024-04-14T15:26:40.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:40 smithi092 ceph-mon[33966]: 6.14 deep-scrub ok 2024-04-14T15:26:40.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:40 smithi092 ceph-mon[33966]: 6.7 scrub starts 2024-04-14T15:26:40.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:40 smithi092 ceph-mon[33966]: 6.7 scrub ok 2024-04-14T15:26:40.410 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:26:40.434 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:40.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:40.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:26:40.536 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:26:40.559 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:40.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:40.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:26:40.661 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:26:40.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:40.685 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:40.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:26:40.786 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:26:40.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:40.813 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:40.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:26:40.914 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:26:40.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:40.939 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:40.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:26:41.040 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:26:41.064 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:41.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:41.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:26:41.111 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:26:41.167 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:26:41.190 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:41.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:41.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:26:41.292 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:26:41.316 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:41.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:41.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:26:41.418 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:26:41.441 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:41.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:41 smithi090 ceph-mon[41670]: 6.10 scrub starts 2024-04-14T15:26:41.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:41 smithi090 ceph-mon[41670]: 6.10 scrub ok 2024-04-14T15:26:41.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:41 smithi090 ceph-mon[41670]: 2.1e scrub starts 2024-04-14T15:26:41.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:41 smithi090 ceph-mon[41670]: 2.1e scrub ok 2024-04-14T15:26:41.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:41 smithi090 ceph-mon[41670]: osdmap e836: 8 total, 8 up, 7 in 2024-04-14T15:26:41.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:41 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:26:41.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:41 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:26:41.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:41 smithi090 ceph-mon[41670]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-04-14T15:26:41.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:41 smithi090 ceph-mon[41670]: 6.1 scrub starts 2024-04-14T15:26:41.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:41 smithi090 ceph-mon[41670]: 6.1 scrub ok 2024-04-14T15:26:41.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:41 smithi090 ceph-mon[30399]: 6.10 scrub starts 2024-04-14T15:26:41.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:41 smithi090 ceph-mon[30399]: 6.10 scrub ok 2024-04-14T15:26:41.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:41 smithi090 ceph-mon[30399]: 2.1e scrub starts 2024-04-14T15:26:41.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:41 smithi090 ceph-mon[30399]: 2.1e scrub ok 2024-04-14T15:26:41.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:41 smithi090 ceph-mon[30399]: osdmap e836: 8 total, 8 up, 7 in 2024-04-14T15:26:41.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:41 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:26:41.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:41 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:26:41.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:41 smithi090 ceph-mon[30399]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-04-14T15:26:41.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:41 smithi090 ceph-mon[30399]: 6.1 scrub starts 2024-04-14T15:26:41.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:41 smithi090 ceph-mon[30399]: 6.1 scrub ok 2024-04-14T15:26:41.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:41.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:26:41.545 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:26:41.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:41.569 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:41.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:26:41.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:41 smithi092 ceph-mon[33966]: 6.10 scrub starts 2024-04-14T15:26:41.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:41 smithi092 ceph-mon[33966]: 6.10 scrub ok 2024-04-14T15:26:41.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:41 smithi092 ceph-mon[33966]: 2.1e scrub starts 2024-04-14T15:26:41.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:41 smithi092 ceph-mon[33966]: 2.1e scrub ok 2024-04-14T15:26:41.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:41 smithi092 ceph-mon[33966]: osdmap e836: 8 total, 8 up, 7 in 2024-04-14T15:26:41.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:41 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:26:41.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:41 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:26:41.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:41 smithi092 ceph-mon[33966]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-04-14T15:26:41.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:41 smithi092 ceph-mon[33966]: 6.1 scrub starts 2024-04-14T15:26:41.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:41 smithi092 ceph-mon[33966]: 6.1 scrub ok 2024-04-14T15:26:41.670 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:26:41.694 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:41.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:41.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:26:41.795 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:26:41.819 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:41.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:41.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:26:41.921 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:26:41.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:41.946 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:41.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:26:42.047 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:26:42.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:42.075 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:42.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:26:42.176 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:26:42.203 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:42.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:42.204 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:26:42.219 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-14T15:26:42.304 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:26:42.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:42.329 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:42.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:26:42.430 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:26:42.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:42 smithi090 ceph-mon[41670]: 4.17 scrub starts 2024-04-14T15:26:42.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:42 smithi090 ceph-mon[41670]: 4.17 scrub ok 2024-04-14T15:26:42.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:42 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-148","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:26:42.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:42 smithi090 ceph-mon[41670]: osdmap e837: 8 total, 8 up, 7 in 2024-04-14T15:26:42.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:42 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:26:42.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:42 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:26:42.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:42 smithi090 ceph-mon[41670]: pgmap v1250: 224 pgs: 32 unknown, 192 active+clean; 4.4 MiB data, 739 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:26:42.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:42 smithi090 ceph-mon[41670]: 2.2 scrub starts 2024-04-14T15:26:42.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:42 smithi090 ceph-mon[41670]: 2.2 scrub ok 2024-04-14T15:26:42.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:42 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-148", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:26:42.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:42 smithi090 ceph-mon[41670]: osdmap e838: 8 total, 8 up, 7 in 2024-04-14T15:26:42.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:42 smithi090 ceph-mon[30399]: 4.17 scrub starts 2024-04-14T15:26:42.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:42 smithi090 ceph-mon[30399]: 4.17 scrub ok 2024-04-14T15:26:42.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:42 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-148","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:26:42.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:42 smithi090 ceph-mon[30399]: osdmap e837: 8 total, 8 up, 7 in 2024-04-14T15:26:42.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:42 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:26:42.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:42 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:26:42.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:42 smithi090 ceph-mon[30399]: pgmap v1250: 224 pgs: 32 unknown, 192 active+clean; 4.4 MiB data, 739 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:26:42.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:42 smithi090 ceph-mon[30399]: 2.2 scrub starts 2024-04-14T15:26:42.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:42 smithi090 ceph-mon[30399]: 2.2 scrub ok 2024-04-14T15:26:42.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:42 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-148", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:26:42.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:42 smithi090 ceph-mon[30399]: osdmap e838: 8 total, 8 up, 7 in 2024-04-14T15:26:42.458 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:42.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:42.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:26:42.560 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:26:42.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:42.588 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:42.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:26:42.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:42 smithi092 ceph-mon[33966]: 4.17 scrub starts 2024-04-14T15:26:42.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:42 smithi092 ceph-mon[33966]: 4.17 scrub ok 2024-04-14T15:26:42.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:42 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-148","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:26:42.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:42 smithi092 ceph-mon[33966]: osdmap e837: 8 total, 8 up, 7 in 2024-04-14T15:26:42.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:42 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:26:42.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:42 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-14T15:26:42.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:42 smithi092 ceph-mon[33966]: pgmap v1250: 224 pgs: 32 unknown, 192 active+clean; 4.4 MiB data, 739 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:26:42.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:42 smithi092 ceph-mon[33966]: 2.2 scrub starts 2024-04-14T15:26:42.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:42 smithi092 ceph-mon[33966]: 2.2 scrub ok 2024-04-14T15:26:42.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:42 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-148", "force_nonempty": "--force-nonempty" }]': finished 2024-04-14T15:26:42.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:42 smithi092 ceph-mon[33966]: osdmap e838: 8 total, 8 up, 7 in 2024-04-14T15:26:42.690 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:26:42.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:42.715 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:42.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:26:42.816 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:26:42.844 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:42.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:42.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:26:42.872 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-14T15:26:42.873 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:26:42.946 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:26:42.974 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:42.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:42.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:26:43.076 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:26:43.100 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:43.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:43.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:26:43.202 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:26:43.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:43.226 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:43.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:26:43.328 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:26:43.353 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:43.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:43.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:26:43.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:43 smithi090 ceph-mon[41670]: 2.1d scrub starts 2024-04-14T15:26:43.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:43 smithi090 ceph-mon[41670]: 2.1d scrub ok 2024-04-14T15:26:43.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:43 smithi090 ceph-mon[41670]: 6.0 scrub starts 2024-04-14T15:26:43.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:43 smithi090 ceph-mon[41670]: 6.0 scrub ok 2024-04-14T15:26:43.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:43 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-148"}]: dispatch 2024-04-14T15:26:43.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:43 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-148"}]: dispatch 2024-04-14T15:26:43.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:43 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-148"}]': finished 2024-04-14T15:26:43.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:43 smithi090 ceph-mon[41670]: osdmap e839: 8 total, 8 up, 7 in 2024-04-14T15:26:43.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:43 smithi090 ceph-mon[30399]: 2.1d scrub starts 2024-04-14T15:26:43.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:43 smithi090 ceph-mon[30399]: 2.1d scrub ok 2024-04-14T15:26:43.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:43 smithi090 ceph-mon[30399]: 6.0 scrub starts 2024-04-14T15:26:43.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:43 smithi090 ceph-mon[30399]: 6.0 scrub ok 2024-04-14T15:26:43.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:43 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-148"}]: dispatch 2024-04-14T15:26:43.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:43 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-148"}]: dispatch 2024-04-14T15:26:43.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:43 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-148"}]': finished 2024-04-14T15:26:43.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:43 smithi090 ceph-mon[30399]: osdmap e839: 8 total, 8 up, 7 in 2024-04-14T15:26:43.454 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:26:43.479 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:43.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:43.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:26:43.580 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:26:43.605 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:43.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:43.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:26:43.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:43 smithi092 ceph-mon[33966]: 2.1d scrub starts 2024-04-14T15:26:43.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:43 smithi092 ceph-mon[33966]: 2.1d scrub ok 2024-04-14T15:26:43.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:43 smithi092 ceph-mon[33966]: 6.0 scrub starts 2024-04-14T15:26:43.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:43 smithi092 ceph-mon[33966]: 6.0 scrub ok 2024-04-14T15:26:43.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:43 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-148"}]: dispatch 2024-04-14T15:26:43.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:43 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-148"}]: dispatch 2024-04-14T15:26:43.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:43 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-148"}]': finished 2024-04-14T15:26:43.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:43 smithi092 ceph-mon[33966]: osdmap e839: 8 total, 8 up, 7 in 2024-04-14T15:26:43.632 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-14T15:26:43.708 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:26:43.731 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:43.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:43.732 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:26:43.833 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:26:43.856 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:43.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:43.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:26:43.958 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:26:43.982 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:43.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:43.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:26:44.084 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:26:44.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:44.110 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:44.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:26:44.211 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:26:44.219 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:26:43 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:26:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:26:44.220 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:44 smithi090 ceph-mon[30399]: 5.5 scrub starts 2024-04-14T15:26:44.220 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:44 smithi090 ceph-mon[30399]: 5.5 scrub ok 2024-04-14T15:26:44.220 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:44 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:26:44.220 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:44 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:26:44.220 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:44 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-148"}]: dispatch 2024-04-14T15:26:44.220 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:44 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-148"}]: dispatch 2024-04-14T15:26:44.220 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:44 smithi090 ceph-mon[30399]: pgmap v1253: 224 pgs: 32 unknown, 192 active+clean; 4.4 MiB data, 739 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:26:44.220 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:44 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:26:44.220 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:44 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:26:44.220 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:44 smithi090 ceph-mon[41670]: 5.5 scrub starts 2024-04-14T15:26:44.221 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:44 smithi090 ceph-mon[41670]: 5.5 scrub ok 2024-04-14T15:26:44.221 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:44 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:26:44.221 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:44 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:26:44.221 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:44 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-148"}]: dispatch 2024-04-14T15:26:44.221 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:44 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-148"}]: dispatch 2024-04-14T15:26:44.221 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:44 smithi090 ceph-mon[41670]: pgmap v1253: 224 pgs: 32 unknown, 192 active+clean; 4.4 MiB data, 739 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:26:44.221 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:44 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:26:44.221 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:44 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:26:44.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:44.237 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:44.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:26:44.338 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:26:44.366 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:44.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:44.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:26:44.373 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:26:44.468 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:26:44.494 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:44.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:44.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:26:44.596 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:26:44.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:44.621 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:44.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:26:44.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:44 smithi092 ceph-mon[33966]: 5.5 scrub starts 2024-04-14T15:26:44.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:44 smithi092 ceph-mon[33966]: 5.5 scrub ok 2024-04-14T15:26:44.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:44 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:26:44.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:44 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:26:44.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:44 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-148"}]: dispatch 2024-04-14T15:26:44.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:44 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-148"}]: dispatch 2024-04-14T15:26:44.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:44 smithi092 ceph-mon[33966]: pgmap v1253: 224 pgs: 32 unknown, 192 active+clean; 4.4 MiB data, 739 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:26:44.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:44 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:26:44.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:44 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:26:44.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:44.625+0000 7fb5c834f700 1 -- 172.21.15.90:0/822824151 >> v1:172.21.15.90:6790/0 conn(0x7fb5c0107760 legacy=0x7fb5c0107b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:44.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:44.625+0000 7fb5c834f700 1 -- 172.21.15.90:0/822824151 shutdown_connections 2024-04-14T15:26:44.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:44.625+0000 7fb5c834f700 1 -- 172.21.15.90:0/822824151 >> 172.21.15.90:0/822824151 conn(0x7fb5c00fd290 msgr2=0x7fb5c00ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:26:44.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:44.625+0000 7fb5c834f700 1 -- 172.21.15.90:0/822824151 shutdown_connections 2024-04-14T15:26:44.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:44.625+0000 7fb5c834f700 1 -- 172.21.15.90:0/822824151 wait complete. 2024-04-14T15:26:44.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:44.626+0000 7fb5c834f700 1 librados: starting msgr at 2024-04-14T15:26:44.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:44.626+0000 7fb5c834f700 1 librados: starting objecter 2024-04-14T15:26:44.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:44.626+0000 7fb5c834f700 1 Processor -- start 2024-04-14T15:26:44.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:44.626+0000 7fb5c834f700 1 -- start start 2024-04-14T15:26:44.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:44.626+0000 7fb5c834f700 1 librados: setting wanted keys 2024-04-14T15:26:44.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:44.626+0000 7fb5c834f700 1 librados: calling monclient init 2024-04-14T15:26:44.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:44.626+0000 7fb5c834f700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb5c010de20 con 0x7fb5c010e1e0 2024-04-14T15:26:44.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:44.626+0000 7fb5c834f700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb5c01a8e00 con 0x7fb5c0107760 2024-04-14T15:26:44.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:44.626+0000 7fb5c834f700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb5c01a9fe0 con 0x7fb5c010a610 2024-04-14T15:26:44.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:44.626+0000 7fb5c58ea700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7fb5c010a610 0x7fb5c006ab70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:42080/0 (socket says 172.21.15.90:42080) 2024-04-14T15:26:44.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:44.626+0000 7fb5c58ea700 1 -- 172.21.15.90:0/4183691275 learned_addr learned my addr 172.21.15.90:0/4183691275 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:26:44.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:44.626+0000 7fb5b77fe700 10 client.?.objecter ms_handle_connect 0x7fb5c010e1e0 2024-04-14T15:26:44.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:44.626+0000 7fb5b77fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:26:44.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:44.626+0000 7fb5b77fe700 10 client.?.objecter ms_handle_connect 0x7fb5c010a610 2024-04-14T15:26:44.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:44.626+0000 7fb5b77fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:26:44.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:44.626+0000 7fb5b77fe700 10 client.?.objecter ms_handle_connect 0x7fb5c0107760 2024-04-14T15:26:44.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:44.626+0000 7fb5b77fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:26:44.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:44.626+0000 7fb5b77fe700 1 -- 172.21.15.90:0/4183691275 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3597620409 0 0) 0x7fb5c010de20 con 0x7fb5c010e1e0 2024-04-14T15:26:44.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:44.626+0000 7fb5b77fe700 1 -- 172.21.15.90:0/4183691275 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb5ac0031f0 con 0x7fb5c010e1e0 2024-04-14T15:26:44.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:44.626+0000 7fb5b77fe700 1 -- 172.21.15.90:0/4183691275 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 187013857 0 0) 0x7fb5c01a9fe0 con 0x7fb5c010a610 2024-04-14T15:26:44.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:44.626+0000 7fb5b77fe700 1 -- 172.21.15.90:0/4183691275 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb5c010de20 con 0x7fb5c010a610 2024-04-14T15:26:44.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:44.627+0000 7fb5b77fe700 1 -- 172.21.15.90:0/4183691275 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 763891620 0 0) 0x7fb5c01a8e00 con 0x7fb5c0107760 2024-04-14T15:26:44.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:44.627+0000 7fb5b77fe700 1 -- 172.21.15.90:0/4183691275 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb5c01a9fe0 con 0x7fb5c0107760 2024-04-14T15:26:44.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:44.627+0000 7fb5b77fe700 1 -- 172.21.15.90:0/4183691275 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 628817623 0 0) 0x7fb5ac0031f0 con 0x7fb5c010e1e0 2024-04-14T15:26:44.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:44.627+0000 7fb5b77fe700 1 -- 172.21.15.90:0/4183691275 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb5c01a8e00 con 0x7fb5c010e1e0 2024-04-14T15:26:44.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:44.627+0000 7fb5b77fe700 1 -- 172.21.15.90:0/4183691275 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fb5b80016f0 con 0x7fb5c010e1e0 2024-04-14T15:26:44.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:44.627+0000 7fb5b77fe700 1 -- 172.21.15.90:0/4183691275 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4126893196 0 0) 0x7fb5c010de20 con 0x7fb5c010a610 2024-04-14T15:26:44.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:44.627+0000 7fb5b77fe700 1 -- 172.21.15.90:0/4183691275 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb5ac0031f0 con 0x7fb5c010a610 2024-04-14T15:26:44.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:44.627+0000 7fb5b77fe700 1 -- 172.21.15.90:0/4183691275 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fb5b0001830 con 0x7fb5c010a610 2024-04-14T15:26:44.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:44.627+0000 7fb5b77fe700 1 -- 172.21.15.90:0/4183691275 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1624273384 0 0) 0x7fb5c01a9fe0 con 0x7fb5c0107760 2024-04-14T15:26:44.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:44.627+0000 7fb5b77fe700 1 -- 172.21.15.90:0/4183691275 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb5c010de20 con 0x7fb5c0107760 2024-04-14T15:26:44.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:44.627+0000 7fb5b77fe700 1 -- 172.21.15.90:0/4183691275 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 967387777 0 0) 0x7fb5c01a8e00 con 0x7fb5c010e1e0 2024-04-14T15:26:44.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:44.627+0000 7fb5b77fe700 1 -- 172.21.15.90:0/4183691275 >> v1:172.21.15.90:6790/0 conn(0x7fb5c010a610 legacy=0x7fb5c006ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:44.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:44.627+0000 7fb5b77fe700 1 -- 172.21.15.90:0/4183691275 >> v1:172.21.15.92:6789/0 conn(0x7fb5c0107760 legacy=0x7fb5c006a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:44.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:44.627+0000 7fb5b77fe700 1 -- 172.21.15.90:0/4183691275 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5c01ab1c0 con 0x7fb5c010e1e0 2024-04-14T15:26:44.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:44.627+0000 7fb5c834f700 1 -- 172.21.15.90:0/4183691275 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb5c01aa210 con 0x7fb5c010e1e0 2024-04-14T15:26:44.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:44.627+0000 7fb5c834f700 10 client.38139.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:26:44.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:44.627+0000 7fb5c834f700 1 -- 172.21.15.90:0/4183691275 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb5c01aa6d0 con 0x7fb5c010e1e0 2024-04-14T15:26:44.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:44.627+0000 7fb5c834f700 1 librados: init done 2024-04-14T15:26:44.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:44.627+0000 7fb5b77fe700 1 -- 172.21.15.90:0/4183691275 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fb5b80019a0 con 0x7fb5c010e1e0 2024-04-14T15:26:44.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:44.627+0000 7fb5b77fe700 1 -- 172.21.15.90:0/4183691275 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fb5b8004e50 con 0x7fb5c010e1e0 2024-04-14T15:26:44.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:44.628+0000 7fb5c834f700 1 -- 172.21.15.90:0/4183691275 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5a4005150 con 0x7fb5c010e1e0 2024-04-14T15:26:44.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:44.629+0000 7fb5b77fe700 1 -- 172.21.15.90:0/4183691275 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fb5b8003cb0 con 0x7fb5c010e1e0 2024-04-14T15:26:44.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:44.629+0000 7fb5b77fe700 1 -- 172.21.15.90:0/4183691275 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(840..840 src has 467..840) ==== 8376+0+0 (unknown 1876555886 0 0) 0x7fb5b8089a60 con 0x7fb5c010e1e0 2024-04-14T15:26:44.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:44.630+0000 7fb5b77fe700 10 client.38139.objecter ms_dispatch 0x7fb5c0001550 osd_map(840..840 src has 467..840) 2024-04-14T15:26:44.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:44.630+0000 7fb5b77fe700 3 client.38139.objecter handle_osd_map got epochs [840,840] > 0 2024-04-14T15:26:44.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:44.630+0000 7fb5b77fe700 3 client.38139.objecter handle_osd_map decoding full epoch 840 2024-04-14T15:26:44.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:44.630+0000 7fb5b77fe700 20 client.38139.objecter dump_active .. 0 homeless 2024-04-14T15:26:44.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:44.630+0000 7fb5b77fe700 10 client.38139.objecter ms_handle_connect 0x7fb5ac0713d0 2024-04-14T15:26:44.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:44.634+0000 7fb5b77fe700 1 -- 172.21.15.90:0/4183691275 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fb5b80563a0 con 0x7fb5c010e1e0 2024-04-14T15:26:44.722 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:26:44.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:44.747 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:44.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:26:44.848 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:26:44.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:44.856+0000 7fb5c834f700 1 -- 172.21.15.90:0/4183691275 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7fb5a4005440 con 0x7fb5c010e1e0 2024-04-14T15:26:44.874 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:44.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:44.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:26:44.974 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-14T15:26:44.975 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-14T15:26:44.975 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-04-14T15:26:44.975 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-04-14T15:26:44.975 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph pg dump --format=json 2024-04-14T15:26:44.976 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:26:45.000 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:45.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:45.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:26:45.102 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:26:45.130 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:45.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:45.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:26:45.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:45.156+0000 7fb5b77fe700 1 -- 172.21.15.90:0/4183691275 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v841) ==== 87+0+0 (unknown 2127429618 0 0) 0x7fb5b805a050 con 0x7fb5c010e1e0 2024-04-14T15:26:45.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:45.158+0000 7fb5c834f700 10 librados: watch_flush enter 2024-04-14T15:26:45.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:45.158+0000 7fb5c834f700 10 librados: watch_flush exit 2024-04-14T15:26:45.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:45.158+0000 7fb5c834f700 20 client.38139.objecter shutdown clearing up homeless session... 2024-04-14T15:26:45.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:45.158+0000 7fb5c834f700 10 client.38139.objecter successfully canceled tick 2024-04-14T15:26:45.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:45.159+0000 7fb5c834f700 1 -- 172.21.15.90:0/4183691275 >> v1:172.21.15.90:6800/420935427 conn(0x7fb5ac0713d0 legacy=0x7fb5ac073890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:45.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:45.159+0000 7fb5c834f700 1 -- 172.21.15.90:0/4183691275 >> v1:172.21.15.90:6789/0 conn(0x7fb5c010e1e0 legacy=0x7fb5c01a66d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:45.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:45.159+0000 7fb5c834f700 1 -- 172.21.15.90:0/4183691275 shutdown_connections 2024-04-14T15:26:45.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:45.159+0000 7fb5c834f700 1 -- 172.21.15.90:0/4183691275 >> 172.21.15.90:0/4183691275 conn(0x7fb5c00fd290 msgr2=0x7fb5c0111490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:26:45.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:45.159+0000 7fb5c834f700 1 -- 172.21.15.90:0/4183691275 shutdown_connections 2024-04-14T15:26:45.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:45.159+0000 7fb5c834f700 1 -- 172.21.15.90:0/4183691275 wait complete. 2024-04-14T15:26:45.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:45.159+0000 7fb5c834f700 1 librados: shutdown 2024-04-14T15:26:45.161 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is set 2024-04-14T15:26:45.232 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:26:45.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:45.257 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:45.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:26:45.358 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:26:45.386 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:45.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:45.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:26:45.408 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:45 smithi090 ceph-mon[41670]: 5.2 deep-scrub starts 2024-04-14T15:26:45.408 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:45 smithi090 ceph-mon[41670]: 5.2 deep-scrub ok 2024-04-14T15:26:45.408 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:45 smithi090 ceph-mon[41670]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:26:45.408 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:45 smithi090 ceph-mon[41670]: osdmap e840: 8 total, 8 up, 7 in 2024-04-14T15:26:45.408 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:45 smithi090 ceph-mon[41670]: 11.14 scrub starts 2024-04-14T15:26:45.408 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:45 smithi090 ceph-mon[41670]: 11.14 scrub ok 2024-04-14T15:26:45.408 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:45 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/4183691275' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:26:45.408 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:45 smithi090 ceph-mon[30399]: 5.2 deep-scrub starts 2024-04-14T15:26:45.408 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:45 smithi090 ceph-mon[30399]: 5.2 deep-scrub ok 2024-04-14T15:26:45.409 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:45 smithi090 ceph-mon[30399]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:26:45.409 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:45 smithi090 ceph-mon[30399]: osdmap e840: 8 total, 8 up, 7 in 2024-04-14T15:26:45.409 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:45 smithi090 ceph-mon[30399]: 11.14 scrub starts 2024-04-14T15:26:45.409 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:45 smithi090 ceph-mon[30399]: 11.14 scrub ok 2024-04-14T15:26:45.409 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:45 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/4183691275' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:26:45.487 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:26:45.514 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:45.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:45.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:26:45.555 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:45 smithi092 ceph-mon[33966]: 5.2 deep-scrub starts 2024-04-14T15:26:45.555 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:45 smithi092 ceph-mon[33966]: 5.2 deep-scrub ok 2024-04-14T15:26:45.555 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:45 smithi092 ceph-mon[33966]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:26:45.555 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:45 smithi092 ceph-mon[33966]: osdmap e840: 8 total, 8 up, 7 in 2024-04-14T15:26:45.555 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:45 smithi092 ceph-mon[33966]: 11.14 scrub starts 2024-04-14T15:26:45.555 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:45 smithi092 ceph-mon[33966]: 11.14 scrub ok 2024-04-14T15:26:45.555 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:45 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/4183691275' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-14T15:26:45.616 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:26:45.640 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:45.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:45.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:26:45.663 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:26:45.742 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:26:45.770 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:45.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:45.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:26:45.872 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:26:45.873 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:26:45 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:26:45.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:45.899 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:45.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:26:46.000 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:26:46.024 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:46.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:46.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:26:46.126 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:26:46.152 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:46.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:46.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:26:46.254 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:26:46.278 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:46.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:46.280 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:26:46.380 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:26:46.404 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:46.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:46.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:26:46.435 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:46 smithi090 ceph-mon[41670]: 6.1c scrub starts 2024-04-14T15:26:46.435 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:46 smithi090 ceph-mon[41670]: 6.1c scrub ok 2024-04-14T15:26:46.436 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:46 smithi090 ceph-mon[41670]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:26:46.436 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:46 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/4183691275' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-14T15:26:46.436 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:46 smithi090 ceph-mon[41670]: osdmap e841: 8 total, 8 up, 7 in 2024-04-14T15:26:46.436 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:46 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:26:46.436 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:46 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:26:46.436 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:46 smithi090 ceph-mon[41670]: pgmap v1256: 224 pgs: 32 unknown, 192 active+clean; 4.4 MiB data, 723 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-04-14T15:26:46.436 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:46 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:26:46.436 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:46 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-150","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:26:46.436 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:46 smithi090 ceph-mon[41670]: osdmap e842: 8 total, 8 up, 7 in 2024-04-14T15:26:46.436 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:46 smithi090 ceph-mon[30399]: 6.1c scrub starts 2024-04-14T15:26:46.436 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:46 smithi090 ceph-mon[30399]: 6.1c scrub ok 2024-04-14T15:26:46.437 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:46 smithi090 ceph-mon[30399]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:26:46.437 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:46 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/4183691275' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-14T15:26:46.437 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:46 smithi090 ceph-mon[30399]: osdmap e841: 8 total, 8 up, 7 in 2024-04-14T15:26:46.437 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:46 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:26:46.437 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:46 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:26:46.437 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:46 smithi090 ceph-mon[30399]: pgmap v1256: 224 pgs: 32 unknown, 192 active+clean; 4.4 MiB data, 723 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-04-14T15:26:46.437 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:46 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:26:46.437 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:46 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-150","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:26:46.437 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:46 smithi090 ceph-mon[30399]: osdmap e842: 8 total, 8 up, 7 in 2024-04-14T15:26:46.507 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:26:46.530 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:46.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:46.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:26:46.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:46 smithi092 ceph-mon[33966]: 6.1c scrub starts 2024-04-14T15:26:46.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:46 smithi092 ceph-mon[33966]: 6.1c scrub ok 2024-04-14T15:26:46.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:46 smithi092 ceph-mon[33966]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:26:46.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:46 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/4183691275' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-14T15:26:46.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:46 smithi092 ceph-mon[33966]: osdmap e841: 8 total, 8 up, 7 in 2024-04-14T15:26:46.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:46 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:26:46.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:46 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:26:46.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:46 smithi092 ceph-mon[33966]: pgmap v1256: 224 pgs: 32 unknown, 192 active+clean; 4.4 MiB data, 723 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-04-14T15:26:46.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:46 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:26:46.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:46 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-150","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:26:46.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:46 smithi092 ceph-mon[33966]: osdmap e842: 8 total, 8 up, 7 in 2024-04-14T15:26:46.633 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:26:46.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:46.660 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:46.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:26:46.761 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:26:46.784 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:46.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:46.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:26:46.886 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:26:46.914 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:46.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:46.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:26:47.015 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:26:47.046 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:47.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:47.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:26:47.148 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:26:47.173 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:47.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:47.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:26:47.275 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:26:47.298 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:47.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:47.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:26:47.400 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:26:47.425 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:47.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:47.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:26:47.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:47 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:26:47.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:47 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:26:47.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:47 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-150"}]: dispatch 2024-04-14T15:26:47.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:47 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-150"}]: dispatch 2024-04-14T15:26:47.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:47 smithi090 ceph-mon[41670]: osdmap e843: 8 total, 8 up, 7 in 2024-04-14T15:26:47.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:47 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:26:47.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:47 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:26:47.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:47 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-150"}]: dispatch 2024-04-14T15:26:47.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:47 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-150"}]: dispatch 2024-04-14T15:26:47.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:47 smithi090 ceph-mon[30399]: osdmap e843: 8 total, 8 up, 7 in 2024-04-14T15:26:47.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.474+0000 7f6e4f992700 1 -- 172.21.15.90:0/3590966939 >> v1:172.21.15.90:6790/0 conn(0x7f6e48107760 legacy=0x7f6e48107b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:47.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.474+0000 7f6e4f992700 1 -- 172.21.15.90:0/3590966939 shutdown_connections 2024-04-14T15:26:47.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.474+0000 7f6e4f992700 1 -- 172.21.15.90:0/3590966939 >> 172.21.15.90:0/3590966939 conn(0x7f6e480fd290 msgr2=0x7f6e480ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:26:47.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.474+0000 7f6e4f992700 1 -- 172.21.15.90:0/3590966939 shutdown_connections 2024-04-14T15:26:47.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.474+0000 7f6e4f992700 1 -- 172.21.15.90:0/3590966939 wait complete. 2024-04-14T15:26:47.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.475+0000 7f6e4f992700 1 librados: starting msgr at 2024-04-14T15:26:47.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.475+0000 7f6e4f992700 1 librados: starting objecter 2024-04-14T15:26:47.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.475+0000 7f6e4f992700 1 Processor -- start 2024-04-14T15:26:47.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.475+0000 7f6e4f992700 1 -- start start 2024-04-14T15:26:47.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.475+0000 7f6e4f992700 1 librados: setting wanted keys 2024-04-14T15:26:47.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.475+0000 7f6e4f992700 1 librados: calling monclient init 2024-04-14T15:26:47.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.475+0000 7f6e4f992700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6e481a9b20 con 0x7f6e48107760 2024-04-14T15:26:47.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.475+0000 7f6e4f992700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6e481aad40 con 0x7f6e4810a610 2024-04-14T15:26:47.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.475+0000 7f6e4f992700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6e481abf80 con 0x7f6e4810e1e0 2024-04-14T15:26:47.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.475+0000 7f6e4d72e700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f6e48107760 0x7f6e4810d810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:49918/0 (socket says 172.21.15.90:49918) 2024-04-14T15:26:47.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.475+0000 7f6e4d72e700 1 -- 172.21.15.90:0/1290052274 learned_addr learned my addr 172.21.15.90:0/1290052274 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:26:47.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.475+0000 7f6e3e7fc700 10 client.?.objecter ms_handle_connect 0x7f6e48107760 2024-04-14T15:26:47.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.475+0000 7f6e3e7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:26:47.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.475+0000 7f6e3e7fc700 10 client.?.objecter ms_handle_connect 0x7f6e4810e1e0 2024-04-14T15:26:47.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.475+0000 7f6e3e7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:26:47.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.475+0000 7f6e3e7fc700 10 client.?.objecter ms_handle_connect 0x7f6e4810a610 2024-04-14T15:26:47.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.475+0000 7f6e3e7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:26:47.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.475+0000 7f6e3e7fc700 1 -- 172.21.15.90:0/1290052274 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2495310087 0 0) 0x7f6e481abf80 con 0x7f6e4810e1e0 2024-04-14T15:26:47.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.476+0000 7f6e3e7fc700 1 -- 172.21.15.90:0/1290052274 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6e34003aa0 con 0x7f6e4810e1e0 2024-04-14T15:26:47.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.476+0000 7f6e3e7fc700 1 -- 172.21.15.90:0/1290052274 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3492679055 0 0) 0x7f6e481a9b20 con 0x7f6e48107760 2024-04-14T15:26:47.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.476+0000 7f6e3e7fc700 1 -- 172.21.15.90:0/1290052274 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6e481abf80 con 0x7f6e48107760 2024-04-14T15:26:47.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.476+0000 7f6e3e7fc700 1 -- 172.21.15.90:0/1290052274 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3903428595 0 0) 0x7f6e481aad40 con 0x7f6e4810a610 2024-04-14T15:26:47.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.476+0000 7f6e3e7fc700 1 -- 172.21.15.90:0/1290052274 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6e481a9b20 con 0x7f6e4810a610 2024-04-14T15:26:47.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.476+0000 7f6e3e7fc700 1 -- 172.21.15.90:0/1290052274 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4035353396 0 0) 0x7f6e34003aa0 con 0x7f6e4810e1e0 2024-04-14T15:26:47.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.476+0000 7f6e3e7fc700 1 -- 172.21.15.90:0/1290052274 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6e481aad40 con 0x7f6e4810e1e0 2024-04-14T15:26:47.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.476+0000 7f6e3e7fc700 1 -- 172.21.15.90:0/1290052274 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f6e40001c10 con 0x7f6e4810e1e0 2024-04-14T15:26:47.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.476+0000 7f6e3e7fc700 1 -- 172.21.15.90:0/1290052274 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 865222469 0 0) 0x7f6e481abf80 con 0x7f6e48107760 2024-04-14T15:26:47.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.476+0000 7f6e3e7fc700 1 -- 172.21.15.90:0/1290052274 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6e34003aa0 con 0x7f6e48107760 2024-04-14T15:26:47.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.476+0000 7f6e3e7fc700 1 -- 172.21.15.90:0/1290052274 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f6e44001a00 con 0x7f6e48107760 2024-04-14T15:26:47.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.476+0000 7f6e3e7fc700 1 -- 172.21.15.90:0/1290052274 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 266173271 0 0) 0x7f6e481a9b20 con 0x7f6e4810a610 2024-04-14T15:26:47.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.476+0000 7f6e3e7fc700 1 -- 172.21.15.90:0/1290052274 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6e481abf80 con 0x7f6e4810a610 2024-04-14T15:26:47.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.476+0000 7f6e3e7fc700 1 -- 172.21.15.90:0/1290052274 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f6e38001830 con 0x7f6e4810a610 2024-04-14T15:26:47.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.476+0000 7f6e3e7fc700 1 -- 172.21.15.90:0/1290052274 <== mon.2 v1:172.21.15.90:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1046340729 0 0) 0x7f6e481aad40 con 0x7f6e4810e1e0 2024-04-14T15:26:47.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.476+0000 7f6e3e7fc700 1 -- 172.21.15.90:0/1290052274 >> v1:172.21.15.92:6789/0 conn(0x7f6e4810a610 legacy=0x7f6e481a4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:47.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.476+0000 7f6e3e7fc700 1 -- 172.21.15.90:0/1290052274 >> v1:172.21.15.90:6789/0 conn(0x7f6e48107760 legacy=0x7f6e4810d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:47.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.476+0000 7f6e3e7fc700 1 -- 172.21.15.90:0/1290052274 --> v1:172.21.15.90:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e481ad1c0 con 0x7f6e4810e1e0 2024-04-14T15:26:47.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.476+0000 7f6e4f992700 1 -- 172.21.15.90:0/1290052274 --> v1:172.21.15.90:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6e481ac1b0 con 0x7f6e4810e1e0 2024-04-14T15:26:47.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.476+0000 7f6e4f992700 10 client.48095.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:26:47.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.476+0000 7f6e4f992700 1 -- 172.21.15.90:0/1290052274 --> v1:172.21.15.90:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f6e481ac670 con 0x7f6e4810e1e0 2024-04-14T15:26:47.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.476+0000 7f6e4f992700 1 librados: init done 2024-04-14T15:26:47.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.476+0000 7f6e3e7fc700 1 -- 172.21.15.90:0/1290052274 <== mon.2 v1:172.21.15.90:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f6e40003ea0 con 0x7f6e4810e1e0 2024-04-14T15:26:47.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.476+0000 7f6e3e7fc700 1 -- 172.21.15.90:0/1290052274 <== mon.2 v1:172.21.15.90:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f6e40004d50 con 0x7f6e4810e1e0 2024-04-14T15:26:47.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.477+0000 7f6e4f992700 1 -- 172.21.15.90:0/1290052274 --> v1:172.21.15.90:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f6e2c005100 con 0x7f6e4810e1e0 2024-04-14T15:26:47.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.478+0000 7f6e3e7fc700 1 -- 172.21.15.90:0/1290052274 <== mon.2 v1:172.21.15.90:6790/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f6e40003a00 con 0x7f6e4810e1e0 2024-04-14T15:26:47.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.478+0000 7f6e3e7fc700 1 -- 172.21.15.90:0/1290052274 <== mon.2 v1:172.21.15.90:6790/0 8 ==== osd_map(843..843 src has 467..843) ==== 8376+0+0 (unknown 4061831063 0 0) 0x7f6e40089a20 con 0x7f6e4810e1e0 2024-04-14T15:26:47.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.478+0000 7f6e3e7fc700 10 client.48095.objecter ms_dispatch 0x7f6e48001550 osd_map(843..843 src has 467..843) 2024-04-14T15:26:47.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.478+0000 7f6e3e7fc700 3 client.48095.objecter handle_osd_map got epochs [843,843] > 0 2024-04-14T15:26:47.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.478+0000 7f6e3e7fc700 3 client.48095.objecter handle_osd_map decoding full epoch 843 2024-04-14T15:26:47.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.478+0000 7f6e3e7fc700 20 client.48095.objecter dump_active .. 0 homeless 2024-04-14T15:26:47.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.478+0000 7f6e3e7fc700 1 -- 172.21.15.90:0/1290052274 <== mon.2 v1:172.21.15.90:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v843) ==== 54+0+16 (unknown 3756593792 0 2420000192) 0x7f6e4008a060 con 0x7f6e4810e1e0 2024-04-14T15:26:47.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.478+0000 7f6e3e7fc700 10 client.48095.objecter ms_handle_connect 0x7f6e34071350 2024-04-14T15:26:47.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.479+0000 7f6e4f992700 10 client.48095.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:26:47.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.479+0000 7f6e4f992700 15 client.48095.objecter _session_command_op_assign -1 1 2024-04-14T15:26:47.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.479+0000 7f6e4f992700 20 client.48095.objecter _get_session s=0x7f6e2c0053e0 osd=0 3 2024-04-14T15:26:47.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.479+0000 7f6e4f992700 20 client.48095.objecter put_session s=0x7f6e2c0053e0 osd=0 3 2024-04-14T15:26:47.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.479+0000 7f6e4f992700 20 client.48095.objecter _get_session s=0x7f6e2c0053e0 osd=0 3 2024-04-14T15:26:47.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.479+0000 7f6e4f992700 15 client.48095.objecter _session_command_op_remove -1 1 2024-04-14T15:26:47.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.479+0000 7f6e4f992700 20 client.48095.objecter get_session s=0x7f6e2c0053e0 osd=0 3 2024-04-14T15:26:47.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.479+0000 7f6e4f992700 15 client.48095.objecter _session_command_op_assign 0 1 2024-04-14T15:26:47.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.479+0000 7f6e4f992700 20 client.48095.objecter put_session s=0x7f6e2c0053e0 osd=0 4 2024-04-14T15:26:47.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.479+0000 7f6e4f992700 10 client.48095.objecter _send_command 1 2024-04-14T15:26:47.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.479+0000 7f6e4f992700 1 -- 172.21.15.90:0/1290052274 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6e2c004b50 con 0x7f6e2c004730 2024-04-14T15:26:47.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.479+0000 7f6e3e7fc700 10 client.48095.objecter ms_handle_connect 0x7f6e2c004730 2024-04-14T15:26:47.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.480+0000 7f6e3e7fc700 1 -- 172.21.15.90:0/1290052274 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6e2c004b50 con 0x7f6e2c004730 2024-04-14T15:26:47.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.480+0000 7f6e3e7fc700 10 client.48095.objecter ms_dispatch 0x7f6e48001550 command_reply(tid 1: 0 ) 2024-04-14T15:26:47.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.480+0000 7f6e3e7fc700 10 client.48095.objecter _finish_command 1 = system:0 2024-04-14T15:26:47.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.480+0000 7f6e3e7fc700 20 client.48095.objecter put_session s=0x7f6e2c0053e0 osd=0 4 2024-04-14T15:26:47.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.480+0000 7f6e3e7fc700 15 client.48095.objecter _session_command_op_remove 0 1 2024-04-14T15:26:47.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.496+0000 7f6e4f992700 10 client.48095.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:26:47.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.496+0000 7f6e4f992700 15 client.48095.objecter _session_command_op_assign -1 2 2024-04-14T15:26:47.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.496+0000 7f6e4f992700 20 client.48095.objecter _get_session s=0x7f6e2c0053e0 osd=0 3 2024-04-14T15:26:47.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.496+0000 7f6e4f992700 20 client.48095.objecter put_session s=0x7f6e2c0053e0 osd=0 3 2024-04-14T15:26:47.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.496+0000 7f6e4f992700 20 client.48095.objecter _get_session s=0x7f6e2c0053e0 osd=0 3 2024-04-14T15:26:47.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.496+0000 7f6e4f992700 15 client.48095.objecter _session_command_op_remove -1 2 2024-04-14T15:26:47.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.496+0000 7f6e4f992700 20 client.48095.objecter get_session s=0x7f6e2c0053e0 osd=0 3 2024-04-14T15:26:47.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.496+0000 7f6e4f992700 15 client.48095.objecter _session_command_op_assign 0 2 2024-04-14T15:26:47.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.496+0000 7f6e4f992700 20 client.48095.objecter put_session s=0x7f6e2c0053e0 osd=0 4 2024-04-14T15:26:47.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.496+0000 7f6e4f992700 10 client.48095.objecter _send_command 2 2024-04-14T15:26:47.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.496+0000 7f6e4f992700 1 -- 172.21.15.90:0/1290052274 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6e2c0079f0 con 0x7f6e2c004730 2024-04-14T15:26:47.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.498+0000 7f6e3e7fc700 1 -- 172.21.15.90:0/1290052274 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6e2c0079f0 con 0x7f6e2c004730 2024-04-14T15:26:47.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.498+0000 7f6e3e7fc700 10 client.48095.objecter ms_dispatch 0x7f6e48001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:26:47.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.498+0000 7f6e3e7fc700 10 client.48095.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:26:47.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.498+0000 7f6e3e7fc700 20 client.48095.objecter put_session s=0x7f6e2c0053e0 osd=0 4 2024-04-14T15:26:47.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.498+0000 7f6e3e7fc700 15 client.48095.objecter _session_command_op_remove 0 2 2024-04-14T15:26:47.499 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:26:47.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.498+0000 7f6e4f992700 10 client.48095.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:26:47.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.498+0000 7f6e4f992700 15 client.48095.objecter _session_command_op_assign -1 3 2024-04-14T15:26:47.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.498+0000 7f6e4f992700 20 client.48095.objecter _get_session s=0x7f6e2c004110 osd=1 3 2024-04-14T15:26:47.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.498+0000 7f6e4f992700 20 client.48095.objecter put_session s=0x7f6e2c004110 osd=1 3 2024-04-14T15:26:47.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.498+0000 7f6e4f992700 20 client.48095.objecter _get_session s=0x7f6e2c004110 osd=1 3 2024-04-14T15:26:47.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.498+0000 7f6e4f992700 15 client.48095.objecter _session_command_op_remove -1 3 2024-04-14T15:26:47.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.498+0000 7f6e4f992700 20 client.48095.objecter get_session s=0x7f6e2c004110 osd=1 3 2024-04-14T15:26:47.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.498+0000 7f6e4f992700 15 client.48095.objecter _session_command_op_assign 1 3 2024-04-14T15:26:47.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.498+0000 7f6e4f992700 20 client.48095.objecter put_session s=0x7f6e2c004110 osd=1 4 2024-04-14T15:26:47.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.498+0000 7f6e4f992700 10 client.48095.objecter _send_command 3 2024-04-14T15:26:47.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.498+0000 7f6e4f992700 1 -- 172.21.15.90:0/1290052274 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f6e2c002130 con 0x7f6e2c004280 2024-04-14T15:26:47.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.499+0000 7f6e3e7fc700 10 client.48095.objecter ms_handle_connect 0x7f6e2c004280 2024-04-14T15:26:47.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.499+0000 7f6e3e7fc700 1 -- 172.21.15.90:0/1290052274 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6e2c002130 con 0x7f6e2c004280 2024-04-14T15:26:47.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.499+0000 7f6e3e7fc700 10 client.48095.objecter ms_dispatch 0x7f6e48001550 command_reply(tid 3: 0 ) 2024-04-14T15:26:47.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.499+0000 7f6e3e7fc700 10 client.48095.objecter _finish_command 3 = system:0 2024-04-14T15:26:47.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.499+0000 7f6e3e7fc700 20 client.48095.objecter put_session s=0x7f6e2c004110 osd=1 4 2024-04-14T15:26:47.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.499+0000 7f6e3e7fc700 15 client.48095.objecter _session_command_op_remove 1 3 2024-04-14T15:26:47.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.515+0000 7f6e4f992700 10 client.48095.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:26:47.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.515+0000 7f6e4f992700 15 client.48095.objecter _session_command_op_assign -1 4 2024-04-14T15:26:47.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.515+0000 7f6e4f992700 20 client.48095.objecter _get_session s=0x7f6e2c004110 osd=1 3 2024-04-14T15:26:47.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.515+0000 7f6e4f992700 20 client.48095.objecter put_session s=0x7f6e2c004110 osd=1 3 2024-04-14T15:26:47.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.515+0000 7f6e4f992700 20 client.48095.objecter _get_session s=0x7f6e2c004110 osd=1 3 2024-04-14T15:26:47.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.515+0000 7f6e4f992700 15 client.48095.objecter _session_command_op_remove -1 4 2024-04-14T15:26:47.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.515+0000 7f6e4f992700 20 client.48095.objecter get_session s=0x7f6e2c004110 osd=1 3 2024-04-14T15:26:47.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.515+0000 7f6e4f992700 15 client.48095.objecter _session_command_op_assign 1 4 2024-04-14T15:26:47.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.515+0000 7f6e4f992700 20 client.48095.objecter put_session s=0x7f6e2c004110 osd=1 4 2024-04-14T15:26:47.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.515+0000 7f6e4f992700 10 client.48095.objecter _send_command 4 2024-04-14T15:26:47.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.515+0000 7f6e4f992700 1 -- 172.21.15.90:0/1290052274 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6e2c001600 con 0x7f6e2c004280 2024-04-14T15:26:47.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.517+0000 7f6e3e7fc700 1 -- 172.21.15.90:0/1290052274 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6e2c001600 con 0x7f6e2c004280 2024-04-14T15:26:47.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.517+0000 7f6e3e7fc700 10 client.48095.objecter ms_dispatch 0x7f6e48001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:26:47.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.517+0000 7f6e3e7fc700 10 client.48095.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:26:47.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.517+0000 7f6e3e7fc700 20 client.48095.objecter put_session s=0x7f6e2c004110 osd=1 4 2024-04-14T15:26:47.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.517+0000 7f6e3e7fc700 15 client.48095.objecter _session_command_op_remove 1 4 2024-04-14T15:26:47.518 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:26:47.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.517+0000 7f6e4f992700 10 client.48095.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:26:47.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.517+0000 7f6e4f992700 15 client.48095.objecter _session_command_op_assign -1 5 2024-04-14T15:26:47.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.517+0000 7f6e4f992700 20 client.48095.objecter _get_session s=0x7f6e2c001a00 osd=2 3 2024-04-14T15:26:47.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.517+0000 7f6e4f992700 20 client.48095.objecter put_session s=0x7f6e2c001a00 osd=2 3 2024-04-14T15:26:47.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.517+0000 7f6e4f992700 20 client.48095.objecter _get_session s=0x7f6e2c001a00 osd=2 3 2024-04-14T15:26:47.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.517+0000 7f6e4f992700 15 client.48095.objecter _session_command_op_remove -1 5 2024-04-14T15:26:47.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.517+0000 7f6e4f992700 20 client.48095.objecter get_session s=0x7f6e2c001a00 osd=2 3 2024-04-14T15:26:47.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.517+0000 7f6e4f992700 15 client.48095.objecter _session_command_op_assign 2 5 2024-04-14T15:26:47.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.517+0000 7f6e4f992700 20 client.48095.objecter put_session s=0x7f6e2c001a00 osd=2 4 2024-04-14T15:26:47.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.517+0000 7f6e4f992700 10 client.48095.objecter _send_command 5 2024-04-14T15:26:47.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.517+0000 7f6e4f992700 1 -- 172.21.15.90:0/1290052274 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f6e2c010f40 con 0x7f6e2c0088b0 2024-04-14T15:26:47.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.517+0000 7f6e3e7fc700 10 client.48095.objecter ms_handle_connect 0x7f6e2c0088b0 2024-04-14T15:26:47.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.518+0000 7f6e3e7fc700 1 -- 172.21.15.90:0/1290052274 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6e2c010f40 con 0x7f6e2c0088b0 2024-04-14T15:26:47.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.518+0000 7f6e3e7fc700 10 client.48095.objecter ms_dispatch 0x7f6e48001550 command_reply(tid 5: 0 ) 2024-04-14T15:26:47.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.518+0000 7f6e3e7fc700 10 client.48095.objecter _finish_command 5 = system:0 2024-04-14T15:26:47.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.518+0000 7f6e3e7fc700 20 client.48095.objecter put_session s=0x7f6e2c001a00 osd=2 4 2024-04-14T15:26:47.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.518+0000 7f6e3e7fc700 15 client.48095.objecter _session_command_op_remove 2 5 2024-04-14T15:26:47.527 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:26:47.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.534+0000 7f6e4f992700 10 client.48095.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:26:47.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.534+0000 7f6e4f992700 15 client.48095.objecter _session_command_op_assign -1 6 2024-04-14T15:26:47.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.534+0000 7f6e4f992700 20 client.48095.objecter _get_session s=0x7f6e2c001a00 osd=2 3 2024-04-14T15:26:47.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.534+0000 7f6e4f992700 20 client.48095.objecter put_session s=0x7f6e2c001a00 osd=2 3 2024-04-14T15:26:47.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.534+0000 7f6e4f992700 20 client.48095.objecter _get_session s=0x7f6e2c001a00 osd=2 3 2024-04-14T15:26:47.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.534+0000 7f6e4f992700 15 client.48095.objecter _session_command_op_remove -1 6 2024-04-14T15:26:47.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.534+0000 7f6e4f992700 20 client.48095.objecter get_session s=0x7f6e2c001a00 osd=2 3 2024-04-14T15:26:47.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.534+0000 7f6e4f992700 15 client.48095.objecter _session_command_op_assign 2 6 2024-04-14T15:26:47.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.534+0000 7f6e4f992700 20 client.48095.objecter put_session s=0x7f6e2c001a00 osd=2 4 2024-04-14T15:26:47.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.534+0000 7f6e4f992700 10 client.48095.objecter _send_command 6 2024-04-14T15:26:47.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.534+0000 7f6e4f992700 1 -- 172.21.15.90:0/1290052274 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6e2c007f10 con 0x7f6e2c0088b0 2024-04-14T15:26:47.536 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:26:47.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.536+0000 7f6e3e7fc700 1 -- 172.21.15.90:0/1290052274 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6e2c007f10 con 0x7f6e2c0088b0 2024-04-14T15:26:47.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.536+0000 7f6e3e7fc700 10 client.48095.objecter ms_dispatch 0x7f6e48001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:26:47.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.536+0000 7f6e3e7fc700 10 client.48095.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:26:47.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.536+0000 7f6e3e7fc700 20 client.48095.objecter put_session s=0x7f6e2c001a00 osd=2 4 2024-04-14T15:26:47.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.536+0000 7f6e3e7fc700 15 client.48095.objecter _session_command_op_remove 2 6 2024-04-14T15:26:47.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.536+0000 7f6e4f992700 10 client.48095.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:26:47.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.536+0000 7f6e4f992700 15 client.48095.objecter _session_command_op_assign -1 7 2024-04-14T15:26:47.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.536+0000 7f6e4f992700 20 client.48095.objecter _get_session s=0x7f6e2c008120 osd=3 3 2024-04-14T15:26:47.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.536+0000 7f6e4f992700 20 client.48095.objecter put_session s=0x7f6e2c008120 osd=3 3 2024-04-14T15:26:47.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.536+0000 7f6e4f992700 20 client.48095.objecter _get_session s=0x7f6e2c008120 osd=3 3 2024-04-14T15:26:47.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.536+0000 7f6e4f992700 15 client.48095.objecter _session_command_op_remove -1 7 2024-04-14T15:26:47.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.536+0000 7f6e4f992700 20 client.48095.objecter get_session s=0x7f6e2c008120 osd=3 3 2024-04-14T15:26:47.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.536+0000 7f6e4f992700 15 client.48095.objecter _session_command_op_assign 3 7 2024-04-14T15:26:47.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.536+0000 7f6e4f992700 20 client.48095.objecter put_session s=0x7f6e2c008120 osd=3 4 2024-04-14T15:26:47.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.536+0000 7f6e4f992700 10 client.48095.objecter _send_command 7 2024-04-14T15:26:47.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.536+0000 7f6e4f992700 1 -- 172.21.15.90:0/1290052274 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f6e2c00d2d0 con 0x7f6e2c00ce70 2024-04-14T15:26:47.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.536+0000 7f6e3e7fc700 10 client.48095.objecter ms_handle_connect 0x7f6e2c00ce70 2024-04-14T15:26:47.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.537+0000 7f6e3e7fc700 1 -- 172.21.15.90:0/1290052274 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6e2c00d2d0 con 0x7f6e2c00ce70 2024-04-14T15:26:47.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.537+0000 7f6e3e7fc700 10 client.48095.objecter ms_dispatch 0x7f6e48001550 command_reply(tid 7: 0 ) 2024-04-14T15:26:47.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.537+0000 7f6e3e7fc700 10 client.48095.objecter _finish_command 7 = system:0 2024-04-14T15:26:47.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.537+0000 7f6e3e7fc700 20 client.48095.objecter put_session s=0x7f6e2c008120 osd=3 4 2024-04-14T15:26:47.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.537+0000 7f6e3e7fc700 15 client.48095.objecter _session_command_op_remove 3 7 2024-04-14T15:26:47.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.553+0000 7f6e4f992700 10 client.48095.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:26:47.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.553+0000 7f6e4f992700 15 client.48095.objecter _session_command_op_assign -1 8 2024-04-14T15:26:47.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.553+0000 7f6e4f992700 20 client.48095.objecter _get_session s=0x7f6e2c008120 osd=3 3 2024-04-14T15:26:47.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.553+0000 7f6e4f992700 20 client.48095.objecter put_session s=0x7f6e2c008120 osd=3 3 2024-04-14T15:26:47.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.553+0000 7f6e4f992700 20 client.48095.objecter _get_session s=0x7f6e2c008120 osd=3 3 2024-04-14T15:26:47.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.553+0000 7f6e4f992700 15 client.48095.objecter _session_command_op_remove -1 8 2024-04-14T15:26:47.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.553+0000 7f6e4f992700 20 client.48095.objecter get_session s=0x7f6e2c008120 osd=3 3 2024-04-14T15:26:47.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.553+0000 7f6e4f992700 15 client.48095.objecter _session_command_op_assign 3 8 2024-04-14T15:26:47.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.553+0000 7f6e4f992700 20 client.48095.objecter put_session s=0x7f6e2c008120 osd=3 4 2024-04-14T15:26:47.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.553+0000 7f6e4f992700 10 client.48095.objecter _send_command 8 2024-04-14T15:26:47.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.553+0000 7f6e4f992700 1 -- 172.21.15.90:0/1290052274 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6e2c00c8f0 con 0x7f6e2c00ce70 2024-04-14T15:26:47.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:47.556 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:47.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:26:47.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.555+0000 7f6e3e7fc700 1 -- 172.21.15.90:0/1290052274 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6e2c00c8f0 con 0x7f6e2c00ce70 2024-04-14T15:26:47.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.555+0000 7f6e3e7fc700 10 client.48095.objecter ms_dispatch 0x7f6e48001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:26:47.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.555+0000 7f6e3e7fc700 10 client.48095.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:26:47.557 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:26:47.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.555+0000 7f6e3e7fc700 20 client.48095.objecter put_session s=0x7f6e2c008120 osd=3 4 2024-04-14T15:26:47.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.555+0000 7f6e3e7fc700 15 client.48095.objecter _session_command_op_remove 3 8 2024-04-14T15:26:47.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.555+0000 7f6e4f992700 10 client.48095.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:26:47.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.555+0000 7f6e4f992700 15 client.48095.objecter _session_command_op_assign -1 9 2024-04-14T15:26:47.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.555+0000 7f6e4f992700 20 client.48095.objecter _get_session s=0x7f6e2c00c6a0 osd=4 3 2024-04-14T15:26:47.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.555+0000 7f6e4f992700 20 client.48095.objecter put_session s=0x7f6e2c00c6a0 osd=4 3 2024-04-14T15:26:47.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.555+0000 7f6e4f992700 20 client.48095.objecter _get_session s=0x7f6e2c00c6a0 osd=4 3 2024-04-14T15:26:47.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.555+0000 7f6e4f992700 15 client.48095.objecter _session_command_op_remove -1 9 2024-04-14T15:26:47.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.555+0000 7f6e4f992700 20 client.48095.objecter get_session s=0x7f6e2c00c6a0 osd=4 3 2024-04-14T15:26:47.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.555+0000 7f6e4f992700 15 client.48095.objecter _session_command_op_assign 4 9 2024-04-14T15:26:47.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.555+0000 7f6e4f992700 20 client.48095.objecter put_session s=0x7f6e2c00c6a0 osd=4 4 2024-04-14T15:26:47.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.555+0000 7f6e4f992700 10 client.48095.objecter _send_command 9 2024-04-14T15:26:47.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.555+0000 7f6e4f992700 1 -- 172.21.15.90:0/1290052274 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f6e2c00c8d0 con 0x7f6e2c011dd0 2024-04-14T15:26:47.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.556+0000 7f6e3e7fc700 10 client.48095.objecter ms_handle_connect 0x7f6e2c011dd0 2024-04-14T15:26:47.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.557+0000 7f6e3e7fc700 1 -- 172.21.15.90:0/1290052274 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6e2c00c8d0 con 0x7f6e2c011dd0 2024-04-14T15:26:47.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.557+0000 7f6e3e7fc700 10 client.48095.objecter ms_dispatch 0x7f6e48001550 command_reply(tid 9: 0 ) 2024-04-14T15:26:47.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.557+0000 7f6e3e7fc700 10 client.48095.objecter _finish_command 9 = system:0 2024-04-14T15:26:47.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.557+0000 7f6e3e7fc700 20 client.48095.objecter put_session s=0x7f6e2c00c6a0 osd=4 4 2024-04-14T15:26:47.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.557+0000 7f6e3e7fc700 15 client.48095.objecter _session_command_op_remove 4 9 2024-04-14T15:26:47.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.572+0000 7f6e4f992700 10 client.48095.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:26:47.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.572+0000 7f6e4f992700 15 client.48095.objecter _session_command_op_assign -1 10 2024-04-14T15:26:47.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.572+0000 7f6e4f992700 20 client.48095.objecter _get_session s=0x7f6e2c00c6a0 osd=4 3 2024-04-14T15:26:47.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.572+0000 7f6e4f992700 20 client.48095.objecter put_session s=0x7f6e2c00c6a0 osd=4 3 2024-04-14T15:26:47.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.572+0000 7f6e4f992700 20 client.48095.objecter _get_session s=0x7f6e2c00c6a0 osd=4 3 2024-04-14T15:26:47.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.572+0000 7f6e4f992700 15 client.48095.objecter _session_command_op_remove -1 10 2024-04-14T15:26:47.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.572+0000 7f6e4f992700 20 client.48095.objecter get_session s=0x7f6e2c00c6a0 osd=4 3 2024-04-14T15:26:47.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.572+0000 7f6e4f992700 15 client.48095.objecter _session_command_op_assign 4 10 2024-04-14T15:26:47.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.572+0000 7f6e4f992700 20 client.48095.objecter put_session s=0x7f6e2c00c6a0 osd=4 4 2024-04-14T15:26:47.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.572+0000 7f6e4f992700 10 client.48095.objecter _send_command 10 2024-04-14T15:26:47.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.572+0000 7f6e4f992700 1 -- 172.21.15.90:0/1290052274 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6e2c00c8d0 con 0x7f6e2c011dd0 2024-04-14T15:26:47.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.574+0000 7f6e3e7fc700 1 -- 172.21.15.90:0/1290052274 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6e2c00c8d0 con 0x7f6e2c011dd0 2024-04-14T15:26:47.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.574+0000 7f6e3e7fc700 10 client.48095.objecter ms_dispatch 0x7f6e48001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:26:47.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.574+0000 7f6e3e7fc700 10 client.48095.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:26:47.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.574+0000 7f6e3e7fc700 20 client.48095.objecter put_session s=0x7f6e2c00c6a0 osd=4 4 2024-04-14T15:26:47.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.574+0000 7f6e3e7fc700 15 client.48095.objecter _session_command_op_remove 4 10 2024-04-14T15:26:47.575 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:26:47.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.575+0000 7f6e4f992700 10 client.48095.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:26:47.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.575+0000 7f6e4f992700 15 client.48095.objecter _session_command_op_assign -1 11 2024-04-14T15:26:47.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.575+0000 7f6e4f992700 20 client.48095.objecter _get_session s=0x7f6e2c00c8d0 osd=5 3 2024-04-14T15:26:47.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.575+0000 7f6e4f992700 20 client.48095.objecter put_session s=0x7f6e2c00c8d0 osd=5 3 2024-04-14T15:26:47.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.575+0000 7f6e4f992700 20 client.48095.objecter _get_session s=0x7f6e2c00c8d0 osd=5 3 2024-04-14T15:26:47.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.575+0000 7f6e4f992700 15 client.48095.objecter _session_command_op_remove -1 11 2024-04-14T15:26:47.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.575+0000 7f6e4f992700 20 client.48095.objecter get_session s=0x7f6e2c00c8d0 osd=5 3 2024-04-14T15:26:47.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.575+0000 7f6e4f992700 15 client.48095.objecter _session_command_op_assign 5 11 2024-04-14T15:26:47.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.575+0000 7f6e4f992700 20 client.48095.objecter put_session s=0x7f6e2c00c8d0 osd=5 4 2024-04-14T15:26:47.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.575+0000 7f6e4f992700 10 client.48095.objecter _send_command 11 2024-04-14T15:26:47.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.575+0000 7f6e4f992700 1 -- 172.21.15.90:0/1290052274 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f6e2c01ba00 con 0x7f6e2c01c630 2024-04-14T15:26:47.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.575+0000 7f6e3e7fc700 10 client.48095.objecter ms_handle_connect 0x7f6e2c01c630 2024-04-14T15:26:47.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.576+0000 7f6e3e7fc700 1 -- 172.21.15.90:0/1290052274 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6e2c01ba00 con 0x7f6e2c01c630 2024-04-14T15:26:47.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.576+0000 7f6e3e7fc700 10 client.48095.objecter ms_dispatch 0x7f6e48001550 command_reply(tid 11: 0 ) 2024-04-14T15:26:47.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.576+0000 7f6e3e7fc700 10 client.48095.objecter _finish_command 11 = system:0 2024-04-14T15:26:47.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.576+0000 7f6e3e7fc700 20 client.48095.objecter put_session s=0x7f6e2c00c8d0 osd=5 4 2024-04-14T15:26:47.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.576+0000 7f6e3e7fc700 15 client.48095.objecter _session_command_op_remove 5 11 2024-04-14T15:26:47.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.592+0000 7f6e4f992700 10 client.48095.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:26:47.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.592+0000 7f6e4f992700 15 client.48095.objecter _session_command_op_assign -1 12 2024-04-14T15:26:47.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.592+0000 7f6e4f992700 20 client.48095.objecter _get_session s=0x7f6e2c00c8d0 osd=5 3 2024-04-14T15:26:47.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.592+0000 7f6e4f992700 20 client.48095.objecter put_session s=0x7f6e2c00c8d0 osd=5 3 2024-04-14T15:26:47.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.592+0000 7f6e4f992700 20 client.48095.objecter _get_session s=0x7f6e2c00c8d0 osd=5 3 2024-04-14T15:26:47.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.592+0000 7f6e4f992700 15 client.48095.objecter _session_command_op_remove -1 12 2024-04-14T15:26:47.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.592+0000 7f6e4f992700 20 client.48095.objecter get_session s=0x7f6e2c00c8d0 osd=5 3 2024-04-14T15:26:47.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.592+0000 7f6e4f992700 15 client.48095.objecter _session_command_op_assign 5 12 2024-04-14T15:26:47.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.592+0000 7f6e4f992700 20 client.48095.objecter put_session s=0x7f6e2c00c8d0 osd=5 4 2024-04-14T15:26:47.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.592+0000 7f6e4f992700 10 client.48095.objecter _send_command 12 2024-04-14T15:26:47.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.592+0000 7f6e4f992700 1 -- 172.21.15.90:0/1290052274 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6e2c01ba00 con 0x7f6e2c01c630 2024-04-14T15:26:47.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.594+0000 7f6e3e7fc700 1 -- 172.21.15.90:0/1290052274 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6e2c01ba00 con 0x7f6e2c01c630 2024-04-14T15:26:47.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.594+0000 7f6e3e7fc700 10 client.48095.objecter ms_dispatch 0x7f6e48001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:26:47.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.594+0000 7f6e3e7fc700 10 client.48095.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:26:47.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.594+0000 7f6e3e7fc700 20 client.48095.objecter put_session s=0x7f6e2c00c8d0 osd=5 4 2024-04-14T15:26:47.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.594+0000 7f6e3e7fc700 15 client.48095.objecter _session_command_op_remove 5 12 2024-04-14T15:26:47.595 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:26:47.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.594+0000 7f6e4f992700 10 client.48095.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:26:47.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.594+0000 7f6e4f992700 15 client.48095.objecter _session_command_op_assign -1 13 2024-04-14T15:26:47.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.594+0000 7f6e4f992700 20 client.48095.objecter _get_session s=0x7f6e2c01ba00 osd=6 3 2024-04-14T15:26:47.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.594+0000 7f6e4f992700 20 client.48095.objecter put_session s=0x7f6e2c01ba00 osd=6 3 2024-04-14T15:26:47.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.594+0000 7f6e4f992700 20 client.48095.objecter _get_session s=0x7f6e2c01ba00 osd=6 3 2024-04-14T15:26:47.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.594+0000 7f6e4f992700 15 client.48095.objecter _session_command_op_remove -1 13 2024-04-14T15:26:47.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.594+0000 7f6e4f992700 20 client.48095.objecter get_session s=0x7f6e2c01ba00 osd=6 3 2024-04-14T15:26:47.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.594+0000 7f6e4f992700 15 client.48095.objecter _session_command_op_assign 6 13 2024-04-14T15:26:47.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.594+0000 7f6e4f992700 20 client.48095.objecter put_session s=0x7f6e2c01ba00 osd=6 4 2024-04-14T15:26:47.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.594+0000 7f6e4f992700 10 client.48095.objecter _send_command 13 2024-04-14T15:26:47.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.594+0000 7f6e4f992700 1 -- 172.21.15.90:0/1290052274 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f6e2c023020 con 0x7f6e2c01bcd0 2024-04-14T15:26:47.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.595+0000 7f6e3e7fc700 10 client.48095.objecter ms_handle_connect 0x7f6e2c01bcd0 2024-04-14T15:26:47.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.596+0000 7f6e3e7fc700 1 -- 172.21.15.90:0/1290052274 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6e2c023020 con 0x7f6e2c01bcd0 2024-04-14T15:26:47.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.596+0000 7f6e3e7fc700 10 client.48095.objecter ms_dispatch 0x7f6e48001550 command_reply(tid 13: 0 ) 2024-04-14T15:26:47.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.596+0000 7f6e3e7fc700 10 client.48095.objecter _finish_command 13 = system:0 2024-04-14T15:26:47.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.596+0000 7f6e3e7fc700 20 client.48095.objecter put_session s=0x7f6e2c01ba00 osd=6 4 2024-04-14T15:26:47.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.596+0000 7f6e3e7fc700 15 client.48095.objecter _session_command_op_remove 6 13 2024-04-14T15:26:47.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.612+0000 7f6e4f992700 10 client.48095.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:26:47.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.612+0000 7f6e4f992700 15 client.48095.objecter _session_command_op_assign -1 14 2024-04-14T15:26:47.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.612+0000 7f6e4f992700 20 client.48095.objecter _get_session s=0x7f6e2c01ba00 osd=6 3 2024-04-14T15:26:47.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.612+0000 7f6e4f992700 20 client.48095.objecter put_session s=0x7f6e2c01ba00 osd=6 3 2024-04-14T15:26:47.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.612+0000 7f6e4f992700 20 client.48095.objecter _get_session s=0x7f6e2c01ba00 osd=6 3 2024-04-14T15:26:47.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.612+0000 7f6e4f992700 15 client.48095.objecter _session_command_op_remove -1 14 2024-04-14T15:26:47.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.612+0000 7f6e4f992700 20 client.48095.objecter get_session s=0x7f6e2c01ba00 osd=6 3 2024-04-14T15:26:47.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.612+0000 7f6e4f992700 15 client.48095.objecter _session_command_op_assign 6 14 2024-04-14T15:26:47.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.612+0000 7f6e4f992700 20 client.48095.objecter put_session s=0x7f6e2c01ba00 osd=6 4 2024-04-14T15:26:47.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.612+0000 7f6e4f992700 10 client.48095.objecter _send_command 14 2024-04-14T15:26:47.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.612+0000 7f6e4f992700 1 -- 172.21.15.90:0/1290052274 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6e2c011340 con 0x7f6e2c01bcd0 2024-04-14T15:26:47.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.614+0000 7f6e3e7fc700 1 -- 172.21.15.90:0/1290052274 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6e2c011340 con 0x7f6e2c01bcd0 2024-04-14T15:26:47.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.614+0000 7f6e3e7fc700 10 client.48095.objecter ms_dispatch 0x7f6e48001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:26:47.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.614+0000 7f6e3e7fc700 10 client.48095.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:26:47.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.614+0000 7f6e3e7fc700 20 client.48095.objecter put_session s=0x7f6e2c01ba00 osd=6 4 2024-04-14T15:26:47.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.614+0000 7f6e3e7fc700 15 client.48095.objecter _session_command_op_remove 6 14 2024-04-14T15:26:47.615 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:26:47.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.614+0000 7f6e4f992700 10 client.48095.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:26:47.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.614+0000 7f6e4f992700 15 client.48095.objecter _session_command_op_assign -1 15 2024-04-14T15:26:47.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.614+0000 7f6e4f992700 20 client.48095.objecter _get_session s=0x7f6e2c00cc50 osd=7 3 2024-04-14T15:26:47.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.614+0000 7f6e4f992700 20 client.48095.objecter put_session s=0x7f6e2c00cc50 osd=7 3 2024-04-14T15:26:47.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.614+0000 7f6e4f992700 20 client.48095.objecter _get_session s=0x7f6e2c00cc50 osd=7 3 2024-04-14T15:26:47.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.614+0000 7f6e4f992700 15 client.48095.objecter _session_command_op_remove -1 15 2024-04-14T15:26:47.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.614+0000 7f6e4f992700 20 client.48095.objecter get_session s=0x7f6e2c00cc50 osd=7 3 2024-04-14T15:26:47.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.614+0000 7f6e4f992700 15 client.48095.objecter _session_command_op_assign 7 15 2024-04-14T15:26:47.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.614+0000 7f6e4f992700 20 client.48095.objecter put_session s=0x7f6e2c00cc50 osd=7 4 2024-04-14T15:26:47.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.614+0000 7f6e4f992700 10 client.48095.objecter _send_command 15 2024-04-14T15:26:47.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.614+0000 7f6e4f992700 1 -- 172.21.15.90:0/1290052274 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f6e2c0247e0 con 0x7f6e2c0110f0 2024-04-14T15:26:47.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.615+0000 7f6e3e7fc700 10 client.48095.objecter ms_handle_connect 0x7f6e2c0110f0 2024-04-14T15:26:47.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.616+0000 7f6e3e7fc700 1 -- 172.21.15.90:0/1290052274 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6e44003e70 con 0x7f6e2c0110f0 2024-04-14T15:26:47.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.616+0000 7f6e3e7fc700 10 client.48095.objecter ms_dispatch 0x7f6e48001550 command_reply(tid 15: 0 ) 2024-04-14T15:26:47.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.616+0000 7f6e3e7fc700 10 client.48095.objecter _finish_command 15 = system:0 2024-04-14T15:26:47.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.616+0000 7f6e3e7fc700 20 client.48095.objecter put_session s=0x7f6e2c00cc50 osd=7 4 2024-04-14T15:26:47.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.616+0000 7f6e3e7fc700 15 client.48095.objecter _session_command_op_remove 7 15 2024-04-14T15:26:47.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:47 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:26:47.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:47 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:26:47.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:47 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-150"}]: dispatch 2024-04-14T15:26:47.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:47 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-150"}]: dispatch 2024-04-14T15:26:47.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:47 smithi092 ceph-mon[33966]: osdmap e843: 8 total, 8 up, 7 in 2024-04-14T15:26:47.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.632+0000 7f6e4f992700 10 client.48095.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-14T15:26:47.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.632+0000 7f6e4f992700 15 client.48095.objecter _session_command_op_assign -1 16 2024-04-14T15:26:47.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.632+0000 7f6e4f992700 20 client.48095.objecter _get_session s=0x7f6e2c00cc50 osd=7 3 2024-04-14T15:26:47.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.632+0000 7f6e4f992700 20 client.48095.objecter put_session s=0x7f6e2c00cc50 osd=7 3 2024-04-14T15:26:47.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.632+0000 7f6e4f992700 20 client.48095.objecter _get_session s=0x7f6e2c00cc50 osd=7 3 2024-04-14T15:26:47.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.632+0000 7f6e4f992700 15 client.48095.objecter _session_command_op_remove -1 16 2024-04-14T15:26:47.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.632+0000 7f6e4f992700 20 client.48095.objecter get_session s=0x7f6e2c00cc50 osd=7 3 2024-04-14T15:26:47.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.632+0000 7f6e4f992700 15 client.48095.objecter _session_command_op_assign 7 16 2024-04-14T15:26:47.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.632+0000 7f6e4f992700 20 client.48095.objecter put_session s=0x7f6e2c00cc50 osd=7 4 2024-04-14T15:26:47.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.632+0000 7f6e4f992700 10 client.48095.objecter _send_command 16 2024-04-14T15:26:47.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.632+0000 7f6e4f992700 1 -- 172.21.15.90:0/1290052274 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6e2c016dd0 con 0x7f6e2c0110f0 2024-04-14T15:26:47.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.634+0000 7f6e3e7fc700 1 -- 172.21.15.90:0/1290052274 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6e2c016dd0 con 0x7f6e2c0110f0 2024-04-14T15:26:47.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.634+0000 7f6e3e7fc700 10 client.48095.objecter ms_dispatch 0x7f6e48001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:26:47.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.634+0000 7f6e3e7fc700 10 client.48095.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:26:47.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.634+0000 7f6e3e7fc700 20 client.48095.objecter put_session s=0x7f6e2c00cc50 osd=7 4 2024-04-14T15:26:47.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.634+0000 7f6e3e7fc700 15 client.48095.objecter _session_command_op_remove 7 16 2024-04-14T15:26:47.635 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:26:47.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.634+0000 7f6e4f992700 10 librados: watch_flush enter 2024-04-14T15:26:47.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.634+0000 7f6e4f992700 10 librados: watch_flush exit 2024-04-14T15:26:47.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.634+0000 7f6e4f992700 10 client.48095.objecter close_session for osd.0 2024-04-14T15:26:47.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.634+0000 7f6e4f992700 1 -- 172.21.15.90:0/1290052274 >> v1:172.21.15.90:6801/3761806885 conn(0x7f6e2c004730 legacy=0x7f6e2c002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:47.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.634+0000 7f6e4f992700 20 client.48095.objecter put_session s=0x7f6e2c0053e0 osd=0 1 2024-04-14T15:26:47.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.634+0000 7f6e4f992700 10 client.48095.objecter close_session for osd.1 2024-04-14T15:26:47.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.634+0000 7f6e4f992700 1 -- 172.21.15.90:0/1290052274 >> v1:172.21.15.90:6805/4170361699 conn(0x7f6e2c004280 legacy=0x7f6e2c00ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:47.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.634+0000 7f6e4f992700 20 client.48095.objecter put_session s=0x7f6e2c004110 osd=1 1 2024-04-14T15:26:47.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.634+0000 7f6e4f992700 10 client.48095.objecter close_session for osd.2 2024-04-14T15:26:47.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.634+0000 7f6e4f992700 1 -- 172.21.15.90:0/1290052274 >> v1:172.21.15.90:6809/1686184251 conn(0x7f6e2c0088b0 legacy=0x7f6e2c00f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:47.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.634+0000 7f6e4f992700 20 client.48095.objecter put_session s=0x7f6e2c001a00 osd=2 1 2024-04-14T15:26:47.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.634+0000 7f6e4f992700 10 client.48095.objecter close_session for osd.3 2024-04-14T15:26:47.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.634+0000 7f6e4f992700 1 -- 172.21.15.90:0/1290052274 >> v1:172.21.15.90:6813/2270166774 conn(0x7f6e2c00ce70 legacy=0x7f6e2c014900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:47.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.634+0000 7f6e4f992700 20 client.48095.objecter put_session s=0x7f6e2c008120 osd=3 1 2024-04-14T15:26:47.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.634+0000 7f6e4f992700 10 client.48095.objecter close_session for osd.4 2024-04-14T15:26:47.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.634+0000 7f6e4f992700 1 -- 172.21.15.90:0/1290052274 >> v1:172.21.15.92:6800/3887549733 conn(0x7f6e2c011dd0 legacy=0x7f6e2c0191b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:47.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.634+0000 7f6e4f992700 20 client.48095.objecter put_session s=0x7f6e2c00c6a0 osd=4 1 2024-04-14T15:26:47.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.634+0000 7f6e4f992700 10 client.48095.objecter close_session for osd.5 2024-04-14T15:26:47.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.634+0000 7f6e4f992700 1 -- 172.21.15.90:0/1290052274 >> v1:172.21.15.92:6804/3244561440 conn(0x7f6e2c01c630 legacy=0x7f6e2c01ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:47.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.635+0000 7f6e4f992700 20 client.48095.objecter put_session s=0x7f6e2c00c8d0 osd=5 1 2024-04-14T15:26:47.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.635+0000 7f6e4f992700 10 client.48095.objecter close_session for osd.6 2024-04-14T15:26:47.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.635+0000 7f6e4f992700 1 -- 172.21.15.90:0/1290052274 >> v1:172.21.15.92:6808/636125963 conn(0x7f6e2c01bcd0 legacy=0x7f6e2c021770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:47.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.635+0000 7f6e4f992700 20 client.48095.objecter put_session s=0x7f6e2c01ba00 osd=6 1 2024-04-14T15:26:47.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.635+0000 7f6e4f992700 10 client.48095.objecter close_session for osd.7 2024-04-14T15:26:47.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.635+0000 7f6e4f992700 1 -- 172.21.15.90:0/1290052274 >> v1:172.21.15.92:6812/1773706716 conn(0x7f6e2c0110f0 legacy=0x7f6e2c023020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:47.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.635+0000 7f6e4f992700 20 client.48095.objecter put_session s=0x7f6e2c00cc50 osd=7 1 2024-04-14T15:26:47.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.635+0000 7f6e4f992700 20 client.48095.objecter shutdown clearing up homeless session... 2024-04-14T15:26:47.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.635+0000 7f6e4f992700 10 client.48095.objecter successfully canceled tick 2024-04-14T15:26:47.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.635+0000 7f6e4df2f700 1 -- 172.21.15.90:0/1290052274 reap_dead start 2024-04-14T15:26:47.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.635+0000 7f6e4df2f700 1 -- 172.21.15.90:0/1290052274 reap_dead start 2024-04-14T15:26:47.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.635+0000 7f6e4df2f700 1 -- 172.21.15.90:0/1290052274 reap_dead start 2024-04-14T15:26:47.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.635+0000 7f6e4df2f700 1 -- 172.21.15.90:0/1290052274 reap_dead start 2024-04-14T15:26:47.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.635+0000 7f6e4df2f700 1 -- 172.21.15.90:0/1290052274 reap_dead start 2024-04-14T15:26:47.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.635+0000 7f6e4f992700 1 -- 172.21.15.90:0/1290052274 >> v1:172.21.15.90:6800/420935427 conn(0x7f6e34071350 legacy=0x7f6e34073810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:47.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.635+0000 7f6e4f992700 1 -- 172.21.15.90:0/1290052274 >> v1:172.21.15.90:6790/0 conn(0x7f6e4810e1e0 legacy=0x7f6e481a81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:47.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.635+0000 7f6e4f992700 1 -- 172.21.15.90:0/1290052274 shutdown_connections 2024-04-14T15:26:47.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.635+0000 7f6e4f992700 1 -- 172.21.15.90:0/1290052274 >> 172.21.15.90:0/1290052274 conn(0x7f6e480fd290 msgr2=0x7f6e480ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:26:47.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.635+0000 7f6e4f992700 1 -- 172.21.15.90:0/1290052274 shutdown_connections 2024-04-14T15:26:47.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.635+0000 7f6e4f992700 1 -- 172.21.15.90:0/1290052274 wait complete. 2024-04-14T15:26:47.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:47.635+0000 7f6e4f992700 1 librados: shutdown 2024-04-14T15:26:47.639 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:26:47.639 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:26:47.639 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:26:47.639 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:26:47.639 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:26:47.639 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:26:47.640 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:26:47.640 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:26:47.657 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:26:47.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:47.682 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:47.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:26:47.783 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:26:47.811 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:47.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:47.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:26:47.913 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:26:47.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:47.939 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:47.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:26:48.040 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:26:48.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:48.070 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:48.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:26:48.171 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:26:48.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:48.198 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:48.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:26:48.235 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:48 smithi090 ceph-mon[41670]: pgmap v1259: 192 pgs: 192 active+clean; 4.4 MiB data, 723 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-04-14T15:26:48.235 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:48 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1290052274' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:26:48.235 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:48 smithi090 ceph-mon[41670]: osdmap e844: 8 total, 8 up, 7 in 2024-04-14T15:26:48.235 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:48 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:26:48.235 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:48 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:26:48.236 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:48 smithi090 ceph-mon[30399]: pgmap v1259: 192 pgs: 192 active+clean; 4.4 MiB data, 723 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-04-14T15:26:48.236 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:48 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1290052274' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:26:48.236 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:48 smithi090 ceph-mon[30399]: osdmap e844: 8 total, 8 up, 7 in 2024-04-14T15:26:48.236 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:48 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:26:48.236 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:48 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:26:48.298 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:26:48.324 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:48.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:48.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:26:48.426 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:26:48.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:48.452 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:48.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:26:48.553 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:26:48.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:48.580 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:48.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:26:48.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:48 smithi092 ceph-mon[33966]: pgmap v1259: 192 pgs: 192 active+clean; 4.4 MiB data, 723 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-04-14T15:26:48.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:48 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1290052274' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:26:48.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:48 smithi092 ceph-mon[33966]: osdmap e844: 8 total, 8 up, 7 in 2024-04-14T15:26:48.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:48 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:26:48.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:48 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:26:48.681 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:26:48.707 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:48.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:48.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:26:48.809 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:26:48.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:48.833 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:48.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:26:48.852 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:26:48.935 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:26:48.959 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:48.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:48.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:26:49.061 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:26:49.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:49.088 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:49.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:26:49.189 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:26:49.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:49.213 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:49.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:26:49.315 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:26:49.340 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:49.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:49.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:26:49.442 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:26:49.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:49.469 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:49.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:26:49.570 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:26:49.595 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:49.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:49.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:26:49.698 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:26:49.723 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:49.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:49.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:26:49.825 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:26:49.851 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:49.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:49.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:26:49.952 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:26:49.977 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:49.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:49.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:26:50.004 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-14T15:26:50.079 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:26:50.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:50.103 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:50.104 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:26:50.205 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:26:50.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:50.234 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:50.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:26:50.337 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:26:50.339 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:50 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-152","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:26:50.339 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:50 smithi090 ceph-mon[41670]: osdmap e845: 8 total, 8 up, 7 in 2024-04-14T15:26:50.339 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:50 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:26:50.339 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:50 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:26:50.339 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:50 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-152"}]: dispatch 2024-04-14T15:26:50.340 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:50 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-152"}]: dispatch 2024-04-14T15:26:50.340 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:50 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-152","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:26:50.340 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:50 smithi090 ceph-mon[30399]: osdmap e845: 8 total, 8 up, 7 in 2024-04-14T15:26:50.340 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:50 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:26:50.340 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:50 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:26:50.340 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:50 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-152"}]: dispatch 2024-04-14T15:26:50.340 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:50 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-152"}]: dispatch 2024-04-14T15:26:50.365 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:50.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:50.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:26:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:50 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-152","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:26:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:50 smithi092 ceph-mon[33966]: osdmap e845: 8 total, 8 up, 7 in 2024-04-14T15:26:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:50 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:26:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:50 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:26:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:50 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-152"}]: dispatch 2024-04-14T15:26:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:50 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-152"}]: dispatch 2024-04-14T15:26:50.467 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:26:50.494 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:50.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:50.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:26:50.596 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:26:50.621 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:50.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:50.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:26:50.723 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:26:50.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:50.751 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:50.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:26:50.852 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:26:50.879 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:50.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:50.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:26:50.981 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:26:51.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:51.006 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:51.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:26:51.107 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6 2024-04-14T15:26:51.134 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:51.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:51.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.6' 2024-04-14T15:26:51.236 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:26:51.238 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-14T15:26:51.238 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-14T15:26:51.239 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-14T15:26:51.239 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-14T15:26:51.239 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.239 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24437.0:657 3.0 3:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e526)", 2024-04-14T15:26:51.239 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:20:52.848110+0000", 2024-04-14T15:26:51.239 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 358.38910538099998, 2024-04-14T15:26:51.239 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12803794499999999, 2024-04-14T15:26:51.239 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:26:51.239 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:26:51.239 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-14T15:26:51.239 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:26:51.239 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24437", 2024-04-14T15:26:51.239 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.92:0/2783975543", 2024-04-14T15:26:51.239 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 657 2024-04-14T15:26:51.240 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.240 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:26:51.240 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.240 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:26:51.240 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:20:52.848110+0000", 2024-04-14T15:26:51.240 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:26:51.240 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.240 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.240 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:26:51.240 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:20:52.848110+0000", 2024-04-14T15:26:51.240 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:26:51.240 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.240 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.240 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:26:51.241 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:20:52.848112+0000", 2024-04-14T15:26:51.241 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.554e-06 2024-04-14T15:26:51.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.241 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.241 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:26:51.241 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:20:52.848121+0000", 2024-04-14T15:26:51.241 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9209999999999998e-06 2024-04-14T15:26:51.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.241 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.241 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:26:51.241 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:20:52.848135+0000", 2024-04-14T15:26:51.241 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3215e-05 2024-04-14T15:26:51.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.242 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.242 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:26:51.242 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:20:52.848145+0000", 2024-04-14T15:26:51.242 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0767999999999999e-05 2024-04-14T15:26:51.242 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.242 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.242 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:26:51.242 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:20:52.848259+0000", 2024-04-14T15:26:51.242 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113909 2024-04-14T15:26:51.242 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.242 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.242 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:26:51.242 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:20:52.848271+0000", 2024-04-14T15:26:51.242 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1606999999999999e-05 2024-04-14T15:26:51.242 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.243 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:26:51.243 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:20:52.975869+0000", 2024-04-14T15:26:51.243 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12759867499999999 2024-04-14T15:26:51.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.243 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:26:51.243 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:20:52.976008+0000", 2024-04-14T15:26:51.243 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013836600000000001 2024-04-14T15:26:51.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.243 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:26:51.244 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:20:52.976148+0000", 2024-04-14T15:26:51.244 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013993 2024-04-14T15:26:51.244 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:26:51.244 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:26:51.244 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:26:51.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.244 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(323.1s2 649/647 ECSubWrite(tid=1092, reqid=client.49576.0:2, at_version=0'0, trim_to=0'0, roll_forward_to=649'1))", 2024-04-14T15:26:51.244 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:23:05.011882+0000", 2024-04-14T15:26:51.244 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 226.225333033, 2024-04-14T15:26:51.244 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001734013, 2024-04-14T15:26:51.244 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:26:51.244 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:26:51.244 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-04-14T15:26:51.245 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:26:51.245 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.245 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:26:51.245 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:05.011882+0000", 2024-04-14T15:26:51.245 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:26:51.245 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.245 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.245 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:26:51.245 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:05.011882+0000", 2024-04-14T15:26:51.245 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:26:51.245 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.245 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.245 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:26:51.246 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:05.011883+0000", 2024-04-14T15:26:51.246 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.82e-07 2024-04-14T15:26:51.246 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.246 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.246 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:26:51.246 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:05.011891+0000", 2024-04-14T15:26:51.246 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4339999999999992e-06 2024-04-14T15:26:51.246 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.246 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.246 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:26:51.246 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:05.011898+0000", 2024-04-14T15:26:51.246 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2049999999999996e-06 2024-04-14T15:26:51.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.247 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:26:51.247 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:05.011908+0000", 2024-04-14T15:26:51.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5599999999999999e-06 2024-04-14T15:26:51.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.247 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:26:51.247 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:05.013288+0000", 2024-04-14T15:26:51.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0013800799999999999 2024-04-14T15:26:51.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.247 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2024-04-14T15:26:51.247 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:05.013298+0000", 2024-04-14T15:26:51.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6360000000000006e-06 2024-04-14T15:26:51.248 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.248 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.248 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2024-04-14T15:26:51.248 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:05.013592+0000", 2024-04-14T15:26:51.248 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029467700000000002 2024-04-14T15:26:51.248 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.248 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.248 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:26:51.248 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:05.013616+0000", 2024-04-14T15:26:51.248 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3839e-05 2024-04-14T15:26:51.248 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:26:51.248 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:26:51.248 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:26:51.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.249 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.249 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.49576.0:247 323.0s0 323:157db765:test-rados-api-smithi090-93652-117::bar:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e664)", 2024-04-14T15:26:51.249 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:23:20.034001+0000", 2024-04-14T15:26:51.249 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 211.20321424100001, 2024-04-14T15:26:51.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0033240309999999999, 2024-04-14T15:26:51.249 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:26:51.249 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:26:51.249 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:26:51.249 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:26:51.249 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.49576", 2024-04-14T15:26:51.249 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2125385135", 2024-04-14T15:26:51.249 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 247 2024-04-14T15:26:51.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.249 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:26:51.250 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.250 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:26:51.250 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:20.034001+0000", 2024-04-14T15:26:51.250 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:26:51.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.250 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.250 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:26:51.250 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:20.034001+0000", 2024-04-14T15:26:51.250 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:26:51.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.250 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.250 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:26:51.250 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:20.034003+0000", 2024-04-14T15:26:51.250 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.058e-06 2024-04-14T15:26:51.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.251 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:26:51.251 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:20.034011+0000", 2024-04-14T15:26:51.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3410000000000003e-06 2024-04-14T15:26:51.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.251 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:26:51.251 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:20.034028+0000", 2024-04-14T15:26:51.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6659000000000001e-05 2024-04-14T15:26:51.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.251 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:26:51.251 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:20.034039+0000", 2024-04-14T15:26:51.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0728e-05 2024-04-14T15:26:51.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.252 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:26:51.252 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:20.034096+0000", 2024-04-14T15:26:51.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7104000000000001e-05 2024-04-14T15:26:51.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.252 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:26:51.252 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:20.034220+0000", 2024-04-14T15:26:51.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000123993 2024-04-14T15:26:51.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.252 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2024-04-14T15:26:51.252 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:20.034619+0000", 2024-04-14T15:26:51.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000399338 2024-04-14T15:26:51.253 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.253 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.253 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2024-04-14T15:26:51.253 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:20.036470+0000", 2024-04-14T15:26:51.253 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0018509010000000001 2024-04-14T15:26:51.253 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.253 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.253 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:26:51.253 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:20.037259+0000", 2024-04-14T15:26:51.253 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00078930000000000005 2024-04-14T15:26:51.253 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.253 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.253 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:26:51.253 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:20.037325+0000", 2024-04-14T15:26:51.253 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5609000000000002e-05 2024-04-14T15:26:51.254 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:26:51.254 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:26:51.254 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:26:51.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.254 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.254 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.49576.0:252 323.0s0 323:157db765:test-rados-api-smithi090-93652-117::bar:head [writefull 0~5] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e665)", 2024-04-14T15:26:51.254 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:23:20.984297+0000", 2024-04-14T15:26:51.254 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 210.25291784300001, 2024-04-14T15:26:51.254 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0067635760000000003, 2024-04-14T15:26:51.254 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:26:51.254 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:26:51.254 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:26:51.254 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:26:51.254 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.49576", 2024-04-14T15:26:51.255 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2125385135", 2024-04-14T15:26:51.255 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 252 2024-04-14T15:26:51.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.255 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:26:51.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.255 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:26:51.255 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:20.984297+0000", 2024-04-14T15:26:51.255 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:26:51.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.255 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:26:51.255 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:20.984297+0000", 2024-04-14T15:26:51.255 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:26:51.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.256 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:26:51.256 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:20.984300+0000", 2024-04-14T15:26:51.256 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4660000000000002e-06 2024-04-14T15:26:51.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.256 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:26:51.256 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:20.984311+0000", 2024-04-14T15:26:51.256 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1038e-05 2024-04-14T15:26:51.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.256 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:26:51.256 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:20.984324+0000", 2024-04-14T15:26:51.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.349e-05 2024-04-14T15:26:51.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:26:51.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:20.984336+0000", 2024-04-14T15:26:51.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1238e-05 2024-04-14T15:26:51.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:26:51.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:20.984397+0000", 2024-04-14T15:26:51.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1156999999999997e-05 2024-04-14T15:26:51.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:26:51.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:20.984497+0000", 2024-04-14T15:26:51.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9884999999999997e-05 2024-04-14T15:26:51.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2024-04-14T15:26:51.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:20.985891+0000", 2024-04-14T15:26:51.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0013939180000000001 2024-04-14T15:26:51.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2024-04-14T15:26:51.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:20.986925+0000", 2024-04-14T15:26:51.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00103399 2024-04-14T15:26:51.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.259 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:26:51.259 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:20.990972+0000", 2024-04-14T15:26:51.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040472119999999997 2024-04-14T15:26:51.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.259 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:26:51.259 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:23:20.991061+0000", 2024-04-14T15:26:51.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9182000000000001e-05 2024-04-14T15:26:51.259 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:26:51.259 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:26:51.259 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:26:51.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.259 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.49576.0:723 329.8 329:15b4ca79:test-rados-api-smithi090-93652-123::bar:head [cache-flush] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e704)", 2024-04-14T15:26:51.260 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:24:06.044872+0000", 2024-04-14T15:26:51.260 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 165.19234326399999, 2024-04-14T15:26:51.260 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023406759999999999, 2024-04-14T15:26:51.260 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:26:51.260 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:26:51.260 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-14T15:26:51.260 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:26:51.260 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.49576", 2024-04-14T15:26:51.260 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2125385135", 2024-04-14T15:26:51.260 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 723 2024-04-14T15:26:51.260 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.260 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:26:51.260 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.260 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:26:51.260 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:06.044872+0000", 2024-04-14T15:26:51.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:26:51.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.261 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.261 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:26:51.261 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:06.044872+0000", 2024-04-14T15:26:51.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:26:51.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.261 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.261 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:26:51.261 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:06.044873+0000", 2024-04-14T15:26:51.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.139e-06 2024-04-14T15:26:51.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.261 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.261 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:26:51.261 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:06.044877+0000", 2024-04-14T15:26:51.262 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0849999999999999e-06 2024-04-14T15:26:51.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.262 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:26:51.262 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:06.044886+0000", 2024-04-14T15:26:51.262 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7749999999999998e-06 2024-04-14T15:26:51.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.262 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:26:51.262 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:06.044891+0000", 2024-04-14T15:26:51.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2939999999999996e-06 2024-04-14T15:26:51.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:26:51.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:06.044941+0000", 2024-04-14T15:26:51.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0074999999999997e-05 2024-04-14T15:26:51.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:26:51.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:06.045061+0000", 2024-04-14T15:26:51.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000119908 2024-04-14T15:26:51.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.264 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:26:51.264 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:06.047064+0000", 2024-04-14T15:26:51.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020029330000000001 2024-04-14T15:26:51.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.264 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:26:51.264 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:06.047154+0000", 2024-04-14T15:26:51.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.009e-05 2024-04-14T15:26:51.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.264 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:26:51.265 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:06.047213+0000", 2024-04-14T15:26:51.265 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8377e-05 2024-04-14T15:26:51.265 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:26:51.265 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:26:51.265 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:26:51.265 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.265 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.265 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.49576.0:791 323.2s0 323:4244f219:test-rados-api-smithi090-93652-125::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e709)", 2024-04-14T15:26:51.265 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:24:11.074941+0000", 2024-04-14T15:26:51.265 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 160.16227430699999, 2024-04-14T15:26:51.265 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010550907999999999, 2024-04-14T15:26:51.265 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:26:51.266 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:26:51.266 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:26:51.266 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:26:51.266 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.49576", 2024-04-14T15:26:51.266 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2125385135", 2024-04-14T15:26:51.266 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 791 2024-04-14T15:26:51.266 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.266 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:26:51.266 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.266 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:26:51.266 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:11.074941+0000", 2024-04-14T15:26:51.266 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:26:51.266 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.267 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.267 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:26:51.267 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:11.074941+0000", 2024-04-14T15:26:51.267 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:26:51.267 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.267 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:51.269 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:26:51.270 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:11.074943+0000", 2024-04-14T15:26:51.270 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0329999999999998e-06 2024-04-14T15:26:51.270 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.270 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.270 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:26:51.270 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:11.074952+0000", 2024-04-14T15:26:51.270 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6079999999999996e-06 2024-04-14T15:26:51.270 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.270 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.270 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:26:51.270 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:11.074962+0000", 2024-04-14T15:26:51.270 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.022e-05 2024-04-14T15:26:51.271 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.271 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.271 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:26:51.271 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:11.074972+0000", 2024-04-14T15:26:51.271 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0451e-05 2024-04-14T15:26:51.271 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.271 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.271 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:26:51.271 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:11.075033+0000", 2024-04-14T15:26:51.271 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0467000000000002e-05 2024-04-14T15:26:51.271 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.271 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.271 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:26:51.272 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:11.075161+0000", 2024-04-14T15:26:51.272 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012795999999999999 2024-04-14T15:26:51.272 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.272 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.272 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2024-04-14T15:26:51.272 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:11.075669+0000", 2024-04-14T15:26:51.272 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00050791300000000003 2024-04-14T15:26:51.272 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.272 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.272 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2024-04-14T15:26:51.272 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:11.076962+0000", 2024-04-14T15:26:51.272 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001293857 2024-04-14T15:26:51.272 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.272 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.272 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:26:51.273 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:11.085414+0000", 2024-04-14T15:26:51.273 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0084517389999999998 2024-04-14T15:26:51.273 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.273 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.273 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:26:51.273 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:11.085492+0000", 2024-04-14T15:26:51.273 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7659999999999998e-05 2024-04-14T15:26:51.273 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:26:51.273 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:26:51.273 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:26:51.273 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.273 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.273 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWriteReply(323.2s0 709/647 ECSubWriteReply(tid=1255, last_complete=709'1, committed=1, applied=1))", 2024-04-14T15:26:51.273 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:24:11.082203+0000", 2024-04-14T15:26:51.274 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 160.15501186500001, 2024-04-14T15:26:51.274 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0030812050000000001, 2024-04-14T15:26:51.274 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:26:51.274 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:26:51.274 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-04-14T15:26:51.274 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:26:51.274 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.274 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:26:51.274 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:11.082203+0000", 2024-04-14T15:26:51.274 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:26:51.274 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.274 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.275 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:26:51.275 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:11.082203+0000", 2024-04-14T15:26:51.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:26:51.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.275 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.275 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:26:51.275 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:11.082205+0000", 2024-04-14T15:26:51.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.161e-06 2024-04-14T15:26:51.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.275 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.275 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:26:51.275 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:11.082210+0000", 2024-04-14T15:26:51.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8660000000000002e-06 2024-04-14T15:26:51.276 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.276 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:26:51.276 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:11.082225+0000", 2024-04-14T15:26:51.276 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4416000000000001e-05 2024-04-14T15:26:51.276 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.276 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:26:51.276 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:11.082234+0000", 2024-04-14T15:26:51.276 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6989999999999998e-06 2024-04-14T15:26:51.276 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.276 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:26:51.276 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:11.085264+0000", 2024-04-14T15:26:51.276 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0030292420000000001 2024-04-14T15:26:51.277 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.277 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.277 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:26:51.277 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:11.085285+0000", 2024-04-14T15:26:51.277 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0820999999999999e-05 2024-04-14T15:26:51.277 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:26:51.277 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:26:51.277 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:26:51.277 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.277 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.277 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWriteReply(323.2s0 709/647 ECSubWriteReply(tid=1255, last_complete=709'1, committed=1, applied=1))", 2024-04-14T15:26:51.277 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:24:11.082281+0000", 2024-04-14T15:26:51.277 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 160.15493453600001, 2024-04-14T15:26:51.277 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0034115299999999999, 2024-04-14T15:26:51.277 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:26:51.278 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:26:51.278 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-04-14T15:26:51.278 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:26:51.278 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.278 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:26:51.278 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:11.082281+0000", 2024-04-14T15:26:51.278 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:26:51.278 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.278 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.278 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:26:51.278 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:11.082281+0000", 2024-04-14T15:26:51.278 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:26:51.278 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.279 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.279 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:26:51.279 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:11.082281+0000", 2024-04-14T15:26:51.279 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5799999999999999e-07 2024-04-14T15:26:51.279 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.279 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.279 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:26:51.279 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:11.082285+0000", 2024-04-14T15:26:51.279 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5020000000000001e-06 2024-04-14T15:26:51.279 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.279 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.279 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:26:51.279 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:11.082291+0000", 2024-04-14T15:26:51.279 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9370000000000001e-06 2024-04-14T15:26:51.280 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.280 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.280 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:26:51.280 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:11.082298+0000", 2024-04-14T15:26:51.280 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9530000000000002e-06 2024-04-14T15:26:51.280 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.280 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.280 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:26:51.280 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:11.085343+0000", 2024-04-14T15:26:51.280 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0030452589999999998 2024-04-14T15:26:51.280 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.280 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.280 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:26:51.280 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:11.085692+0000", 2024-04-14T15:26:51.280 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034932099999999998 2024-04-14T15:26:51.281 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:26:51.281 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:26:51.281 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:26:51.281 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.281 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.281 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.49576.0:796 323.2s0 323:4244f219:test-rados-api-smithi090-93652-125::foo:head [writefull 0~5] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e710)", 2024-04-14T15:26:51.281 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:24:12.077502+0000", 2024-04-14T15:26:51.281 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 159.15971295099999, 2024-04-14T15:26:51.281 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002468736, 2024-04-14T15:26:51.281 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:26:51.281 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:26:51.281 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:26:51.281 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:26:51.281 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.49576", 2024-04-14T15:26:51.282 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2125385135", 2024-04-14T15:26:51.282 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 796 2024-04-14T15:26:51.282 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.282 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:26:51.282 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.282 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:26:51.282 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:12.077502+0000", 2024-04-14T15:26:51.282 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:26:51.282 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.282 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.282 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:26:51.282 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:12.077502+0000", 2024-04-14T15:26:51.282 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:26:51.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.283 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:26:51.283 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:12.077505+0000", 2024-04-14T15:26:51.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1860000000000001e-06 2024-04-14T15:26:51.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.283 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:26:51.283 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:12.077518+0000", 2024-04-14T15:26:51.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3025e-05 2024-04-14T15:26:51.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.283 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:26:51.283 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:12.078182+0000", 2024-04-14T15:26:51.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00066370000000000003 2024-04-14T15:26:51.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.284 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:26:51.284 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:12.078195+0000", 2024-04-14T15:26:51.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2452e-05 2024-04-14T15:26:51.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.284 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:26:51.284 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:12.078260+0000", 2024-04-14T15:26:51.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5434999999999999e-05 2024-04-14T15:26:51.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.284 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:26:51.284 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:12.078361+0000", 2024-04-14T15:26:51.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010121200000000001 2024-04-14T15:26:51.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2024-04-14T15:26:51.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:12.078769+0000", 2024-04-14T15:26:51.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00040754799999999998 2024-04-14T15:26:51.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2024-04-14T15:26:51.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:12.079315+0000", 2024-04-14T15:26:51.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000546603 2024-04-14T15:26:51.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:26:51.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:12.079897+0000", 2024-04-14T15:26:51.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00058128300000000004 2024-04-14T15:26:51.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.286 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:26:51.286 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:12.079971+0000", 2024-04-14T15:26:51.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4292e-05 2024-04-14T15:26:51.286 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:26:51.286 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:26:51.286 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:26:51.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.286 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.49576.0:1046 331.1a 331:58000000::::head [pgnls start_epoch 724] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e724)", 2024-04-14T15:26:51.286 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:24:26.012276+0000", 2024-04-14T15:26:51.287 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 145.224939312, 2024-04-14T15:26:51.287 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4607686069999999, 2024-04-14T15:26:51.287 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:26:51.287 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:26:51.287 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-14T15:26:51.287 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:26:51.287 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.49576", 2024-04-14T15:26:51.287 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2125385135", 2024-04-14T15:26:51.287 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1046 2024-04-14T15:26:51.287 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.287 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:26:51.287 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.287 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:26:51.288 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:26.012276+0000", 2024-04-14T15:26:51.288 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:26:51.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.288 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:26:51.288 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:26.012276+0000", 2024-04-14T15:26:51.288 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:26:51.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.288 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:26:51.288 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:26.012279+0000", 2024-04-14T15:26:51.288 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6910000000000002e-06 2024-04-14T15:26:51.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:26:51.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:26.012290+0000", 2024-04-14T15:26:51.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1319e-05 2024-04-14T15:26:51.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:26:51.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:26.012306+0000", 2024-04-14T15:26:51.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6427e-05 2024-04-14T15:26:51.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:26:51.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:26.012317+0000", 2024-04-14T15:26:51.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0475000000000001e-05 2024-04-14T15:26:51.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.290 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:26:51.290 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:26.012372+0000", 2024-04-14T15:26:51.290 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5126e-05 2024-04-14T15:26:51.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.290 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-14T15:26:51.290 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:26.012379+0000", 2024-04-14T15:26:51.290 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1999999999999997e-06 2024-04-14T15:26:51.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.290 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:26:51.290 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:27.472904+0000", 2024-04-14T15:26:51.291 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4605253300000001 2024-04-14T15:26:51.291 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.291 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:26:51.291 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:27.472942+0000", 2024-04-14T15:26:51.291 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7914999999999999e-05 2024-04-14T15:26:51.291 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.291 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:26:51.291 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:27.473044+0000", 2024-04-14T15:26:51.291 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000102124 2024-04-14T15:26:51.291 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:26:51.291 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:26:51.291 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:26:51.292 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.292 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.292 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.49576.0:1294 333.14 333:2df51650:test-rados-api-smithi090-93652-131::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e742)", 2024-04-14T15:26:51.292 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:24:46.004826+0000", 2024-04-14T15:26:51.292 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 125.232389588, 2024-04-14T15:26:51.292 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0047317269999999998, 2024-04-14T15:26:51.292 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:26:51.292 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:26:51.292 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:26:51.292 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:26:51.292 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.49576", 2024-04-14T15:26:51.292 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2125385135", 2024-04-14T15:26:51.292 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1294 2024-04-14T15:26:51.292 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.292 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:26:51.293 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.293 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:26:51.293 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:46.004826+0000", 2024-04-14T15:26:51.293 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:26:51.293 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.293 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.293 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:26:51.293 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:46.004826+0000", 2024-04-14T15:26:51.293 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:26:51.293 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.293 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.293 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:26:51.293 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:46.004829+0000", 2024-04-14T15:26:51.293 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3189999999999999e-06 2024-04-14T15:26:51.293 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.294 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.294 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:26:51.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:46.004842+0000", 2024-04-14T15:26:51.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.288e-05 2024-04-14T15:26:51.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.294 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.294 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:26:51.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:46.005053+0000", 2024-04-14T15:26:51.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021073700000000001 2024-04-14T15:26:51.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.294 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.294 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:26:51.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:46.005066+0000", 2024-04-14T15:26:51.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3617e-05 2024-04-14T15:26:51.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.295 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.295 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:26:51.295 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:46.005122+0000", 2024-04-14T15:26:51.295 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5674999999999998e-05 2024-04-14T15:26:51.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.295 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.295 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-14T15:26:51.295 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:46.005358+0000", 2024-04-14T15:26:51.295 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000236195 2024-04-14T15:26:51.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.295 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.295 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:26:51.295 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:46.008544+0000", 2024-04-14T15:26:51.296 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031859789999999998 2024-04-14T15:26:51.296 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.296 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.296 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:26:51.296 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:46.008618+0000", 2024-04-14T15:26:51.296 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3909e-05 2024-04-14T15:26:51.296 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.296 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.296 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,2", 2024-04-14T15:26:51.296 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:46.008703+0000", 2024-04-14T15:26:51.296 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5544000000000006e-05 2024-04-14T15:26:51.296 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.296 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.296 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:26:51.296 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:46.009121+0000", 2024-04-14T15:26:51.297 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000417193 2024-04-14T15:26:51.297 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.297 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.297 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:26:51.297 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:46.009427+0000", 2024-04-14T15:26:51.297 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030586600000000002 2024-04-14T15:26:51.297 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.297 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.297 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:26:51.297 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:46.009467+0000", 2024-04-14T15:26:51.297 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0697e-05 2024-04-14T15:26:51.297 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.297 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.297 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:26:51.298 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:46.009521+0000", 2024-04-14T15:26:51.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3931000000000003e-05 2024-04-14T15:26:51.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.298 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.298 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:26:51.298 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:46.009557+0000", 2024-04-14T15:26:51.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6185e-05 2024-04-14T15:26:51.298 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:26:51.298 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:26:51.298 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:26:51.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.298 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.298 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.49576.0:1339 333.14 333:2df51650:test-rados-api-smithi090-93652-131::foo:head [cache-flush] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e742)", 2024-04-14T15:26:51.299 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:24:46.032867+0000", 2024-04-14T15:26:51.299 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 125.20434851, 2024-04-14T15:26:51.299 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031071580000000001, 2024-04-14T15:26:51.299 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:26:51.299 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:26:51.299 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-14T15:26:51.299 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:26:51.299 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.49576", 2024-04-14T15:26:51.299 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2125385135", 2024-04-14T15:26:51.299 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1339 2024-04-14T15:26:51.299 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.299 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:26:51.299 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.299 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:26:51.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:46.032867+0000", 2024-04-14T15:26:51.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:26:51.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.300 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.300 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:26:51.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:46.032867+0000", 2024-04-14T15:26:51.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:26:51.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.300 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.300 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:26:51.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:46.032867+0000", 2024-04-14T15:26:51.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7800000000000001e-07 2024-04-14T15:26:51.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.301 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.301 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:26:51.301 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:46.032872+0000", 2024-04-14T15:26:51.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.048e-06 2024-04-14T15:26:51.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.301 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.301 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:26:51.301 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:46.032879+0000", 2024-04-14T15:26:51.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6140000000000003e-06 2024-04-14T15:26:51.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.301 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.301 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:26:51.301 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:46.032884+0000", 2024-04-14T15:26:51.302 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1050000000000003e-06 2024-04-14T15:26:51.302 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.302 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:26:51.302 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:46.032922+0000", 2024-04-14T15:26:51.302 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7511000000000001e-05 2024-04-14T15:26:51.302 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.302 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:26:51.302 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:46.032984+0000", 2024-04-14T15:26:51.302 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2255999999999995e-05 2024-04-14T15:26:51.302 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.303 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:26:51.303 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:46.035796+0000", 2024-04-14T15:26:51.303 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028121919999999998 2024-04-14T15:26:51.303 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.303 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.303 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:26:51.303 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:46.035885+0000", 2024-04-14T15:26:51.303 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8807000000000005e-05 2024-04-14T15:26:51.303 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.303 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.303 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:26:51.303 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:46.035974+0000", 2024-04-14T15:26:51.303 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8846999999999998e-05 2024-04-14T15:26:51.303 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:26:51.304 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:26:51.304 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:26:51.304 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.304 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.304 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.49576.0:1349 333.14 333:2df51650:test-rados-api-smithi090-93652-131::foo:head [cache-evict] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e742)", 2024-04-14T15:26:51.304 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:24:46.040859+0000", 2024-04-14T15:26:51.304 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 125.196355951, 2024-04-14T15:26:51.304 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0018395029999999999, 2024-04-14T15:26:51.304 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:26:51.304 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:26:51.304 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:26:51.304 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:26:51.304 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.49576", 2024-04-14T15:26:51.304 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2125385135", 2024-04-14T15:26:51.304 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1349 2024-04-14T15:26:51.305 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.305 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:26:51.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.305 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:26:51.305 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:46.040859+0000", 2024-04-14T15:26:51.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:26:51.305 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.305 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:26:51.305 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:46.040859+0000", 2024-04-14T15:26:51.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:26:51.305 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.305 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:26:51.305 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:46.040862+0000", 2024-04-14T15:26:51.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3290000000000001e-06 2024-04-14T15:26:51.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:26:51.306 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:46.040869+0000", 2024-04-14T15:26:51.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.362e-06 2024-04-14T15:26:51.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:26:51.306 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:46.040887+0000", 2024-04-14T15:26:51.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8355999999999998e-05 2024-04-14T15:26:51.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:26:51.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:46.040898+0000", 2024-04-14T15:26:51.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0251000000000001e-05 2024-04-14T15:26:51.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:26:51.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:46.040962+0000", 2024-04-14T15:26:51.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4236e-05 2024-04-14T15:26:51.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:26:51.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:46.041092+0000", 2024-04-14T15:26:51.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013054900000000001 2024-04-14T15:26:51.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,2", 2024-04-14T15:26:51.308 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:46.041211+0000", 2024-04-14T15:26:51.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000118935 2024-04-14T15:26:51.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.308 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-14T15:26:51.308 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:46.041829+0000", 2024-04-14T15:26:51.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00061737999999999995 2024-04-14T15:26:51.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.308 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:26:51.308 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:46.042092+0000", 2024-04-14T15:26:51.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026319000000000001 2024-04-14T15:26:51.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.308 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-14T15:26:51.309 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:46.042585+0000", 2024-04-14T15:26:51.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00049359299999999996 2024-04-14T15:26:51.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.309 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:26:51.309 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:46.042647+0000", 2024-04-14T15:26:51.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1674000000000006e-05 2024-04-14T15:26:51.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.309 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:26:51.309 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:46.042699+0000", 2024-04-14T15:26:51.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1647999999999997e-05 2024-04-14T15:26:51.309 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:26:51.309 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:26:51.309 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:26:51.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.310 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.310 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.49576.0:1382 333.14 333:2df51650:test-rados-api-smithi090-93652-131::foo:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e742)", 2024-04-14T15:26:51.310 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:24:46.052822+0000", 2024-04-14T15:26:51.310 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 125.18439290800001, 2024-04-14T15:26:51.310 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032534109999999999, 2024-04-14T15:26:51.310 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:26:51.310 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:26:51.310 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-14T15:26:51.310 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:26:51.310 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.49576", 2024-04-14T15:26:51.310 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2125385135", 2024-04-14T15:26:51.310 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1382 2024-04-14T15:26:51.311 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.311 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:26:51.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:26:51.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:46.052822+0000", 2024-04-14T15:26:51.311 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:26:51.311 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:26:51.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:46.052822+0000", 2024-04-14T15:26:51.311 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:26:51.311 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:26:51.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:46.052824+0000", 2024-04-14T15:26:51.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.26e-06 2024-04-14T15:26:51.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:26:51.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:46.052829+0000", 2024-04-14T15:26:51.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5480000000000001e-06 2024-04-14T15:26:51.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:26:51.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:46.052839+0000", 2024-04-14T15:26:51.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6709999999999994e-06 2024-04-14T15:26:51.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.313 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:26:51.313 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:46.052845+0000", 2024-04-14T15:26:51.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1489999999999999e-06 2024-04-14T15:26:51.313 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.313 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.313 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:26:51.313 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:46.052892+0000", 2024-04-14T15:26:51.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7277999999999997e-05 2024-04-14T15:26:51.313 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.313 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.313 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-14T15:26:51.313 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:46.053048+0000", 2024-04-14T15:26:51.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015562900000000001 2024-04-14T15:26:51.313 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.314 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:26:51.314 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:46.055026+0000", 2024-04-14T15:26:51.314 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0019782509999999999 2024-04-14T15:26:51.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.314 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-14T15:26:51.314 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:46.055105+0000", 2024-04-14T15:26:51.314 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9055999999999997e-05 2024-04-14T15:26:51.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.314 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:26:51.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:46.055700+0000", 2024-04-14T15:26:51.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00059512300000000003 2024-04-14T15:26:51.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:26:51.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:46.055758+0000", 2024-04-14T15:26:51.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8084000000000002e-05 2024-04-14T15:26:51.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:26:51.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:46.056076+0000", 2024-04-14T15:26:51.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031736200000000002 2024-04-14T15:26:51.315 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:26:51.315 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:26:51.315 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:26:51.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.316 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.49576.0:1384 333.14 333:2df51650:test-rados-api-smithi090-93652-131::foo:head [cache-flush] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e742)", 2024-04-14T15:26:51.316 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:24:46.057380+0000", 2024-04-14T15:26:51.316 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 125.17983544499999, 2024-04-14T15:26:51.316 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020964949999999999, 2024-04-14T15:26:51.316 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:26:51.316 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:26:51.316 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-14T15:26:51.316 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:26:51.316 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.49576", 2024-04-14T15:26:51.316 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2125385135", 2024-04-14T15:26:51.316 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1384 2024-04-14T15:26:51.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.317 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:26:51.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.317 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:26:51.317 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:46.057380+0000", 2024-04-14T15:26:51.317 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:26:51.317 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.317 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:26:51.317 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:46.057380+0000", 2024-04-14T15:26:51.317 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:26:51.317 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.317 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:26:51.318 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:46.057381+0000", 2024-04-14T15:26:51.318 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.207e-06 2024-04-14T15:26:51.318 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.318 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.318 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:26:51.318 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:46.057386+0000", 2024-04-14T15:26:51.318 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7650000000000001e-06 2024-04-14T15:26:51.318 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.318 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.318 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:26:51.318 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:46.057398+0000", 2024-04-14T15:26:51.318 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2704000000000001e-05 2024-04-14T15:26:51.318 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.319 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:26:51.319 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:46.057408+0000", 2024-04-14T15:26:51.319 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7489999999999993e-06 2024-04-14T15:26:51.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.319 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:26:51.319 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:46.057446+0000", 2024-04-14T15:26:51.319 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7670999999999999e-05 2024-04-14T15:26:51.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.319 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:26:51.319 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:46.057500+0000", 2024-04-14T15:26:51.319 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4344e-05 2024-04-14T15:26:51.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.320 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:26:51.320 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:46.059324+0000", 2024-04-14T15:26:51.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00182402 2024-04-14T15:26:51.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.320 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:26:51.320 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:46.059407+0000", 2024-04-14T15:26:51.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3121000000000006e-05 2024-04-14T15:26:51.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.320 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:26:51.320 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:24:46.059476+0000", 2024-04-14T15:26:51.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8913999999999998e-05 2024-04-14T15:26:51.321 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:26:51.321 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:26:51.321 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:26:51.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.321 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(osd.2.18:91 323.0s0 323:0e8501ae:test-rados-api-smithi090-93652-137::foo:head [copy-from ver 2] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e771)", 2024-04-14T15:26:51.321 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:25:15.043096+0000", 2024-04-14T15:26:51.321 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 96.194119493000002, 2024-04-14T15:26:51.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015722537000000002, 2024-04-14T15:26:51.321 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:26:51.321 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:26:51.321 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:26:51.321 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:26:51.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.322 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:26:51.322 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:25:15.043096+0000", 2024-04-14T15:26:51.322 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:26:51.322 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.322 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.322 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:26:51.322 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:25:15.043096+0000", 2024-04-14T15:26:51.322 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:26:51.322 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.322 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:26:51.323 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:25:15.043098+0000", 2024-04-14T15:26:51.323 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6089999999999999e-06 2024-04-14T15:26:51.323 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.323 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:26:51.323 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:25:15.043107+0000", 2024-04-14T15:26:51.323 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5629999999999992e-06 2024-04-14T15:26:51.323 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.323 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:26:51.323 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:25:15.043122+0000", 2024-04-14T15:26:51.324 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4914e-05 2024-04-14T15:26:51.324 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.324 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.324 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:26:51.324 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:25:15.043130+0000", 2024-04-14T15:26:51.324 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8499999999999994e-06 2024-04-14T15:26:51.324 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.324 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.324 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:26:51.324 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:25:15.043182+0000", 2024-04-14T15:26:51.324 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2034999999999999e-05 2024-04-14T15:26:51.324 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.324 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.324 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:26:51.324 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:25:15.043276+0000", 2024-04-14T15:26:51.325 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4087999999999998e-05 2024-04-14T15:26:51.325 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.325 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.325 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2024-04-14T15:26:51.325 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:25:15.053406+0000", 2024-04-14T15:26:51.325 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010129829 2024-04-14T15:26:51.325 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.325 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.325 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2024-04-14T15:26:51.325 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:25:15.057284+0000", 2024-04-14T15:26:51.325 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0038783189999999999 2024-04-14T15:26:51.325 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.325 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.325 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:26:51.325 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:25:15.058759+0000", 2024-04-14T15:26:51.325 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0014753489999999999 2024-04-14T15:26:51.326 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.326 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.326 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:26:51.326 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:25:15.058818+0000", 2024-04-14T15:26:51.326 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8981000000000003e-05 2024-04-14T15:26:51.326 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:26:51.326 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:26:51.326 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:26:51.327 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.327 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(osd.4.31:100 323.6s0 323:7ff8a9a1:test-rados-api-smithi090-93652-143::foo0:head [read 0~1] snapc 0=[] ondisk+read+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e807)", 2024-04-14T15:26:51.327 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:25:57.027643+0000", 2024-04-14T15:26:51.327 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 54.209572541999997, 2024-04-14T15:26:51.327 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003966458, 2024-04-14T15:26:51.327 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:26:51.327 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:26:51.327 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-14T15:26:51.327 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:26:51.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.328 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:26:51.328 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:25:57.027643+0000", 2024-04-14T15:26:51.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:26:51.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.328 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:26:51.328 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:25:57.027643+0000", 2024-04-14T15:26:51.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:26:51.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.328 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:26:51.328 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:25:57.027645+0000", 2024-04-14T15:26:51.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7640000000000001e-06 2024-04-14T15:26:51.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.329 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:26:51.329 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:25:57.027654+0000", 2024-04-14T15:26:51.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6449999999999994e-06 2024-04-14T15:26:51.329 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.329 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:26:51.329 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:25:57.027668+0000", 2024-04-14T15:26:51.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4166e-05 2024-04-14T15:26:51.329 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.329 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:26:51.329 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:25:57.027680+0000", 2024-04-14T15:26:51.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1280999999999999e-05 2024-04-14T15:26:51.329 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.329 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:26:51.330 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:25:57.027745+0000", 2024-04-14T15:26:51.330 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5415999999999998e-05 2024-04-14T15:26:51.330 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.330 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2024-04-14T15:26:51.330 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:25:57.027779+0000", 2024-04-14T15:26:51.330 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3630000000000002e-05 2024-04-14T15:26:51.330 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.330 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:26:51.330 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:25:57.030361+0000", 2024-04-14T15:26:51.330 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002582558 2024-04-14T15:26:51.330 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.331 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:26:51.331 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:25:57.030461+0000", 2024-04-14T15:26:51.331 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010015500000000001 2024-04-14T15:26:51.331 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.331 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.331 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:26:51.331 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:25:57.031609+0000", 2024-04-14T15:26:51.331 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001147843 2024-04-14T15:26:51.331 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:26:51.331 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:26:51.331 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:26:51.331 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.331 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.331 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.49576.0:2732 323.0s0 323:0956867a:test-rados-api-smithi090-93652-147::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e821)", 2024-04-14T15:26:51.332 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:26:19.066762+0000", 2024-04-14T15:26:51.332 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 32.170453530000003, 2024-04-14T15:26:51.332 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0036835349999999999, 2024-04-14T15:26:51.332 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:26:51.332 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:26:51.332 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:26:51.332 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:26:51.332 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.49576", 2024-04-14T15:26:51.332 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2125385135", 2024-04-14T15:26:51.332 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2732 2024-04-14T15:26:51.332 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.332 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:26:51.332 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.332 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:26:51.332 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:26:19.066762+0000", 2024-04-14T15:26:51.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:26:51.333 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:26:51.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:26:19.066762+0000", 2024-04-14T15:26:51.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:26:51.333 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:26:51.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:26:19.066763+0000", 2024-04-14T15:26:51.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5850000000000001e-06 2024-04-14T15:26:51.333 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:26:51.334 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:26:19.066770+0000", 2024-04-14T15:26:51.334 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2069999999999998e-06 2024-04-14T15:26:51.334 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.334 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.334 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:26:51.334 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:26:19.066780+0000", 2024-04-14T15:26:51.334 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3889999999999994e-06 2024-04-14T15:26:51.334 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.334 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.334 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:26:51.334 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:26:19.066789+0000", 2024-04-14T15:26:51.334 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6099999999999995e-06 2024-04-14T15:26:51.334 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.334 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.335 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:26:51.335 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:26:19.067282+0000", 2024-04-14T15:26:51.335 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00049275199999999995 2024-04-14T15:26:51.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.335 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.335 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:26:51.335 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:26:19.067408+0000", 2024-04-14T15:26:51.335 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012559999999999999 2024-04-14T15:26:51.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.335 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.335 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2024-04-14T15:26:51.335 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:26:19.067862+0000", 2024-04-14T15:26:51.335 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00045391600000000001 2024-04-14T15:26:51.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.336 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.336 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2024-04-14T15:26:51.336 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:26:19.069400+0000", 2024-04-14T15:26:51.336 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001538003 2024-04-14T15:26:51.336 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.336 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.336 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:26:51.336 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:26:19.070366+0000", 2024-04-14T15:26:51.336 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00096605400000000004 2024-04-14T15:26:51.336 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.336 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.336 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:26:51.336 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:26:19.070445+0000", 2024-04-14T15:26:51.336 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9419000000000007e-05 2024-04-14T15:26:51.337 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:26:51.337 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:26:51.337 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:26:51.337 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.337 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.337 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWriteReply(323.0s0 821/647 ECSubWriteReply(tid=1307, last_complete=821'9, committed=1, applied=1))", 2024-04-14T15:26:51.337 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:26:19.071793+0000", 2024-04-14T15:26:51.337 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 32.165421705999997, 2024-04-14T15:26:51.337 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0017081170000000001, 2024-04-14T15:26:51.337 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:26:51.337 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:26:51.337 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-04-14T15:26:51.337 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:26:51.337 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.337 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:26:51.338 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:26:19.071793+0000", 2024-04-14T15:26:51.338 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:26:51.338 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.338 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.338 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:26:51.338 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:26:19.071793+0000", 2024-04-14T15:26:51.338 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:26:51.338 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.338 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.338 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:26:51.338 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:26:19.071794+0000", 2024-04-14T15:26:51.338 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5300000000000004e-07 2024-04-14T15:26:51.338 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.339 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.339 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:26:51.339 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:26:19.071797+0000", 2024-04-14T15:26:51.339 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3620000000000002e-06 2024-04-14T15:26:51.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.339 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.339 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:26:51.339 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:26:19.071805+0000", 2024-04-14T15:26:51.339 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0600000000000008e-06 2024-04-14T15:26:51.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.339 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.340 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:26:51.340 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:26:19.071813+0000", 2024-04-14T15:26:51.340 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2849999999999997e-06 2024-04-14T15:26:51.340 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.340 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.340 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:26:51.340 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:26:19.073485+0000", 2024-04-14T15:26:51.340 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001672621 2024-04-14T15:26:51.340 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.340 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.340 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:26:51.340 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:26:19.073502+0000", 2024-04-14T15:26:51.340 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6235999999999998e-05 2024-04-14T15:26:51.340 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:26:51.340 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:26:51.341 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:26:51.341 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.341 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.341 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.49576.0:2734 323.6s0 323:7a53c057:test-rados-api-smithi090-93652-147::baz:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e821)", 2024-04-14T15:26:51.341 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-14T15:26:19.078093+0000", 2024-04-14T15:26:51.341 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 32.159121896999999, 2024-04-14T15:26:51.341 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0019367589999999999, 2024-04-14T15:26:51.341 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-04-14T15:26:51.341 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-14T15:26:51.341 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-14T15:26:51.341 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-14T15:26:51.341 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.49576", 2024-04-14T15:26:51.341 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2125385135", 2024-04-14T15:26:51.341 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2734 2024-04-14T15:26:51.342 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.342 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-14T15:26:51.342 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.342 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-14T15:26:51.342 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:26:19.078093+0000", 2024-04-14T15:26:51.342 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:26:51.342 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.342 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.342 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-14T15:26:51.342 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:26:19.078093+0000", 2024-04-14T15:26:51.342 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-14T15:26:51.342 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.342 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.342 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-14T15:26:51.342 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:26:19.078095+0000", 2024-04-14T15:26:51.343 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1500000000000002e-06 2024-04-14T15:26:51.343 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.343 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.343 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-14T15:26:51.343 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:26:19.078104+0000", 2024-04-14T15:26:51.343 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2740000000000007e-06 2024-04-14T15:26:51.343 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.343 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.343 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-14T15:26:51.343 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:26:19.078116+0000", 2024-04-14T15:26:51.343 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2683999999999999e-05 2024-04-14T15:26:51.343 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.343 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.343 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-14T15:26:51.344 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:26:19.078125+0000", 2024-04-14T15:26:51.344 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2490000000000001e-06 2024-04-14T15:26:51.344 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.344 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.344 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-14T15:26:51.344 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:26:19.078172+0000", 2024-04-14T15:26:51.344 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7366e-05 2024-04-14T15:26:51.344 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.344 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.344 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-14T15:26:51.344 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:26:19.078323+0000", 2024-04-14T15:26:51.344 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015080599999999999 2024-04-14T15:26:51.344 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.344 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.345 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2024-04-14T15:26:51.345 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:26:19.078765+0000", 2024-04-14T15:26:51.345 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00044213199999999997 2024-04-14T15:26:51.345 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.345 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.345 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2024-04-14T15:26:51.345 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:26:19.079488+0000", 2024-04-14T15:26:51.345 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00072344100000000001 2024-04-14T15:26:51.345 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.345 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.345 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-14T15:26:51.345 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:26:19.079959+0000", 2024-04-14T15:26:51.345 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00047049 2024-04-14T15:26:51.345 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-14T15:26:51.346 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-14T15:26:51.346 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-14T15:26:51.346 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-14T15:26:19.080030+0000", 2024-04-14T15:26:51.346 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1167000000000005e-05 2024-04-14T15:26:51.346 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:26:51.346 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:26:51.346 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:26:51.346 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-14T15:26:51.346 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-14T15:26:51.346 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-14T15:26:51.346 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:51.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:26:51.347 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-14T15:26:51.348 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:51 smithi090 ceph-mon[41670]: pgmap v1262: 224 pgs: 1 creating+activating, 6 creating+peering, 217 active+clean; 4.4 MiB data, 724 MiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 1023 B/s wr, 4 op/s 2024-04-14T15:26:51.348 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:51 smithi090 ceph-mon[41670]: osdmap e846: 8 total, 8 up, 7 in 2024-04-14T15:26:51.348 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:51 smithi090 ceph-mon[41670]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:26:51.348 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:51 smithi090 ceph-mon[30399]: pgmap v1262: 224 pgs: 1 creating+activating, 6 creating+peering, 217 active+clean; 4.4 MiB data, 724 MiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 1023 B/s wr, 4 op/s 2024-04-14T15:26:51.348 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:51 smithi090 ceph-mon[30399]: osdmap e846: 8 total, 8 up, 7 in 2024-04-14T15:26:51.348 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:51 smithi090 ceph-mon[30399]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:26:51.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:51 smithi092 ceph-mon[33966]: pgmap v1262: 224 pgs: 1 creating+activating, 6 creating+peering, 217 active+clean; 4.4 MiB data, 724 MiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 1023 B/s wr, 4 op/s 2024-04-14T15:26:51.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:51 smithi092 ceph-mon[33966]: osdmap e846: 8 total, 8 up, 7 in 2024-04-14T15:26:51.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:51 smithi092 ceph-mon[33966]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:26:51.447 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:26:51.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:51.473 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:51.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:26:51.574 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:26:51.597 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:51.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:51.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:26:51.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.536+0000 7f467000e700 1 -- 172.21.15.90:0/3102378857 >> v1:172.21.15.90:6789/0 conn(0x7f46681088b0 legacy=0x7f4668108c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:51.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.536+0000 7f467000e700 1 -- 172.21.15.90:0/3102378857 shutdown_connections 2024-04-14T15:26:51.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.536+0000 7f467000e700 1 -- 172.21.15.90:0/3102378857 >> 172.21.15.90:0/3102378857 conn(0x7f46680fe3c0 msgr2=0x7f46681007e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:26:51.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.536+0000 7f467000e700 1 -- 172.21.15.90:0/3102378857 shutdown_connections 2024-04-14T15:26:51.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.536+0000 7f467000e700 1 -- 172.21.15.90:0/3102378857 wait complete. 2024-04-14T15:26:51.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.537+0000 7f467000e700 1 librados: starting msgr at 2024-04-14T15:26:51.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.537+0000 7f467000e700 1 librados: starting objecter 2024-04-14T15:26:51.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.537+0000 7f467000e700 1 Processor -- start 2024-04-14T15:26:51.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.537+0000 7f467000e700 1 -- start start 2024-04-14T15:26:51.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.537+0000 7f467000e700 1 librados: setting wanted keys 2024-04-14T15:26:51.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.537+0000 7f467000e700 1 librados: calling monclient init 2024-04-14T15:26:51.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.537+0000 7f467000e700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f466810ef20 con 0x7f466810b760 2024-04-14T15:26:51.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.537+0000 7f467000e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f466810f130 con 0x7f466810f330 2024-04-14T15:26:51.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.537+0000 7f467000e700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f46681a9f90 con 0x7f46681088b0 2024-04-14T15:26:51.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.538+0000 7f466d5a9700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f466810b760 0x7f466810c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:60318/0 (socket says 172.21.15.90:60318) 2024-04-14T15:26:51.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.538+0000 7f466d5a9700 1 -- 172.21.15.90:0/3800404685 learned_addr learned my addr 172.21.15.90:0/3800404685 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:26:51.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.538+0000 7f465effd700 10 client.?.objecter ms_handle_connect 0x7f466810b760 2024-04-14T15:26:51.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.538+0000 7f465effd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:26:51.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.538+0000 7f465effd700 10 client.?.objecter ms_handle_connect 0x7f46681088b0 2024-04-14T15:26:51.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.538+0000 7f465effd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:26:51.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.538+0000 7f465effd700 1 -- 172.21.15.90:0/3800404685 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3016577979 0 0) 0x7f466810ef20 con 0x7f466810b760 2024-04-14T15:26:51.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.538+0000 7f465effd700 1 -- 172.21.15.90:0/3800404685 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f46540031f0 con 0x7f466810b760 2024-04-14T15:26:51.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.538+0000 7f465effd700 10 client.?.objecter ms_handle_connect 0x7f466810f330 2024-04-14T15:26:51.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.538+0000 7f465effd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:26:51.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.538+0000 7f465effd700 1 -- 172.21.15.90:0/3800404685 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3831610036 0 0) 0x7f46681a9f90 con 0x7f46681088b0 2024-04-14T15:26:51.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.538+0000 7f465effd700 1 -- 172.21.15.90:0/3800404685 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f466810ef20 con 0x7f46681088b0 2024-04-14T15:26:51.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.538+0000 7f465effd700 1 -- 172.21.15.90:0/3800404685 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1931501972 0 0) 0x7f466810f130 con 0x7f466810f330 2024-04-14T15:26:51.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.538+0000 7f465effd700 1 -- 172.21.15.90:0/3800404685 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f46681a9f90 con 0x7f466810f330 2024-04-14T15:26:51.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.538+0000 7f465effd700 1 -- 172.21.15.90:0/3800404685 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2486381919 0 0) 0x7f46540031f0 con 0x7f466810b760 2024-04-14T15:26:51.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.538+0000 7f465effd700 1 -- 172.21.15.90:0/3800404685 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f466810f130 con 0x7f466810b760 2024-04-14T15:26:51.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.538+0000 7f465effd700 1 -- 172.21.15.90:0/3800404685 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4007083220 0 0) 0x7f466810ef20 con 0x7f46681088b0 2024-04-14T15:26:51.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.538+0000 7f465effd700 1 -- 172.21.15.90:0/3800404685 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f46540031f0 con 0x7f46681088b0 2024-04-14T15:26:51.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.538+0000 7f465effd700 1 -- 172.21.15.90:0/3800404685 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f46640018d0 con 0x7f466810b760 2024-04-14T15:26:51.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.538+0000 7f465effd700 1 -- 172.21.15.90:0/3800404685 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f4658001020 con 0x7f46681088b0 2024-04-14T15:26:51.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.538+0000 7f465effd700 1 -- 172.21.15.90:0/3800404685 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3686950934 0 0) 0x7f46681a9f90 con 0x7f466810f330 2024-04-14T15:26:51.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.538+0000 7f465effd700 1 -- 172.21.15.90:0/3800404685 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f466810ef20 con 0x7f466810f330 2024-04-14T15:26:51.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.538+0000 7f465effd700 1 -- 172.21.15.90:0/3800404685 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f4660001af0 con 0x7f466810f330 2024-04-14T15:26:51.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.538+0000 7f465effd700 1 -- 172.21.15.90:0/3800404685 <== mon.2 v1:172.21.15.90:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4208701055 0 0) 0x7f46540031f0 con 0x7f46681088b0 2024-04-14T15:26:51.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.538+0000 7f465effd700 1 -- 172.21.15.90:0/3800404685 >> v1:172.21.15.92:6789/0 conn(0x7f466810f330 legacy=0x7f46681a6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:51.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.538+0000 7f465effd700 1 -- 172.21.15.90:0/3800404685 >> v1:172.21.15.90:6789/0 conn(0x7f466810b760 legacy=0x7f466810c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:51.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.538+0000 7f465effd700 1 -- 172.21.15.90:0/3800404685 --> v1:172.21.15.90:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46681ab1f0 con 0x7f46681088b0 2024-04-14T15:26:51.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.538+0000 7f467000e700 1 -- 172.21.15.90:0/3800404685 --> v1:172.21.15.90:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f46681aa1c0 con 0x7f46681088b0 2024-04-14T15:26:51.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.539+0000 7f467000e700 10 client.48101.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:26:51.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.539+0000 7f467000e700 1 -- 172.21.15.90:0/3800404685 --> v1:172.21.15.90:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f46681aa680 con 0x7f46681088b0 2024-04-14T15:26:51.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.539+0000 7f467000e700 1 librados: init done 2024-04-14T15:26:51.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.539+0000 7f465effd700 1 -- 172.21.15.90:0/3800404685 <== mon.2 v1:172.21.15.90:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f4658001d20 con 0x7f46681088b0 2024-04-14T15:26:51.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.539+0000 7f465effd700 1 -- 172.21.15.90:0/3800404685 <== mon.2 v1:172.21.15.90:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f4658004b70 con 0x7f46681088b0 2024-04-14T15:26:51.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.539+0000 7f467000e700 1 -- 172.21.15.90:0/3800404685 --> v1:172.21.15.90:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f464c005150 con 0x7f46681088b0 2024-04-14T15:26:51.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.541+0000 7f465effd700 1 -- 172.21.15.90:0/3800404685 <== mon.2 v1:172.21.15.90:6790/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f4658001880 con 0x7f46681088b0 2024-04-14T15:26:51.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.541+0000 7f465effd700 1 -- 172.21.15.90:0/3800404685 <== mon.2 v1:172.21.15.90:6790/0 8 ==== osd_map(847..847 src has 467..847) ==== 8749+0+0 (unknown 3746316267 0 0) 0x7f4658089820 con 0x7f46681088b0 2024-04-14T15:26:51.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.541+0000 7f465effd700 10 client.48101.objecter ms_dispatch 0x7f4668001550 osd_map(847..847 src has 467..847) 2024-04-14T15:26:51.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.541+0000 7f465effd700 3 client.48101.objecter handle_osd_map got epochs [847,847] > 0 2024-04-14T15:26:51.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.541+0000 7f465effd700 3 client.48101.objecter handle_osd_map decoding full epoch 847 2024-04-14T15:26:51.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.541+0000 7f465effd700 20 client.48101.objecter dump_active .. 0 homeless 2024-04-14T15:26:51.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.541+0000 7f465effd700 10 client.48101.objecter ms_handle_connect 0x7f4654071390 2024-04-14T15:26:51.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.545+0000 7f465effd700 1 -- 172.21.15.90:0/3800404685 <== mon.2 v1:172.21.15.90:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f4658055fe0 con 0x7f46681088b0 2024-04-14T15:26:51.699 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:26:51.722 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:51.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:51.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:26:51.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.745+0000 7f467000e700 1 -- 172.21.15.90:0/3800404685 --> v1:172.21.15.90:6800/420935427 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f464c000bc0 con 0x7f4654071390 2024-04-14T15:26:51.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.760+0000 7f465effd700 1 -- 172.21.15.90:0/3800404685 <== mgr.14610 v1:172.21.15.90:6800/420935427 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+575386 (unknown 2965378022 0 155267815) 0x7f464c000bc0 con 0x7f4654071390 2024-04-14T15:26:51.771 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:26:51.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.765+0000 7f467000e700 10 librados: watch_flush enter 2024-04-14T15:26:51.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.765+0000 7f467000e700 10 librados: watch_flush exit 2024-04-14T15:26:51.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.765+0000 7f467000e700 20 client.48101.objecter shutdown clearing up homeless session... 2024-04-14T15:26:51.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.765+0000 7f467000e700 10 client.48101.objecter successfully canceled tick 2024-04-14T15:26:51.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.765+0000 7f467000e700 1 -- 172.21.15.90:0/3800404685 >> v1:172.21.15.90:6800/420935427 conn(0x7f4654071390 legacy=0x7f4654073850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:51.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.765+0000 7f467000e700 1 -- 172.21.15.90:0/3800404685 >> v1:172.21.15.90:6790/0 conn(0x7f46681088b0 legacy=0x7f466810be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:51.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.765+0000 7f467000e700 1 -- 172.21.15.90:0/3800404685 shutdown_connections 2024-04-14T15:26:51.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.765+0000 7f467000e700 1 -- 172.21.15.90:0/3800404685 >> 172.21.15.90:0/3800404685 conn(0x7f46680fe3c0 msgr2=0x7f4668110f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:26:51.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.765+0000 7f467000e700 1 -- 172.21.15.90:0/3800404685 shutdown_connections 2024-04-14T15:26:51.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.765+0000 7f467000e700 1 -- 172.21.15.90:0/3800404685 wait complete. 2024-04-14T15:26:51.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:51.765+0000 7f467000e700 1 librados: shutdown 2024-04-14T15:26:51.778 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-14T15:26:51.824 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:26:51.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:51.853 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:51.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:26:51.954 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:26:51.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:51.979 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:51.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:26:52.080 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:26:52.107 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:52.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:52.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:26:52.209 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:26:52.235 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:52.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:52.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:26:52.337 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:26:52.360 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:52.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:52.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:26:52.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:52 smithi092 ceph-mon[33966]: osdmap e847: 8 total, 8 up, 7 in 2024-04-14T15:26:52.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:52 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:26:52.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:52 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:26:52.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:52 smithi092 ceph-mon[33966]: pgmap v1265: 224 pgs: 32 unknown, 192 active+clean; 4.4 MiB data, 724 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 3 op/s 2024-04-14T15:26:52.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:52 smithi092 ceph-mon[33966]: from='client.48101 v1:172.21.15.90:0/3800404685' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:26:52.426 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:52 smithi090 ceph-mon[41670]: osdmap e847: 8 total, 8 up, 7 in 2024-04-14T15:26:52.426 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:52 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:26:52.426 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:52 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:26:52.426 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:52 smithi090 ceph-mon[41670]: pgmap v1265: 224 pgs: 32 unknown, 192 active+clean; 4.4 MiB data, 724 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 3 op/s 2024-04-14T15:26:52.426 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:52 smithi090 ceph-mon[41670]: from='client.48101 v1:172.21.15.90:0/3800404685' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:26:52.426 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:52 smithi090 ceph-mon[30399]: osdmap e847: 8 total, 8 up, 7 in 2024-04-14T15:26:52.427 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:52 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:26:52.427 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:52 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:26:52.427 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:52 smithi090 ceph-mon[30399]: pgmap v1265: 224 pgs: 32 unknown, 192 active+clean; 4.4 MiB data, 724 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 3 op/s 2024-04-14T15:26:52.427 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:52 smithi090 ceph-mon[30399]: from='client.48101 v1:172.21.15.90:0/3800404685' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:26:52.462 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:26:52.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:52.487 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:52.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:26:52.588 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:26:52.611 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:52.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:52.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:26:52.712 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:26:52.736 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:52.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:52.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:26:52.838 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:26:52.861 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:52.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:52.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:26:52.963 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:26:52.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:52.987 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:52.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:26:53.060 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:53 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-154","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:26:53.061 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:53 smithi090 ceph-mon[30399]: osdmap e848: 8 total, 8 up, 7 in 2024-04-14T15:26:53.088 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:26:53.101 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-14T15:26:53.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:53.113 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:53.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:26:53.214 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:26:53.237 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:53.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:53.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:26:53.327 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:26:53.339 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:26:53.363 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:53.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:53.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:26:53.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:53 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-154","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:26:53.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:53 smithi092 ceph-mon[33966]: osdmap e848: 8 total, 8 up, 7 in 2024-04-14T15:26:53.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:53 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-111","var": "dedup_tier","val": "test-rados-api-smithi090-93652-154"}]: dispatch 2024-04-14T15:26:53.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:53 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-111","var": "dedup_tier","val": "test-rados-api-smithi090-93652-154"}]: dispatch 2024-04-14T15:26:53.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:53 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:26:53.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:53 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:26:53.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:53 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-154"}]: dispatch 2024-04-14T15:26:53.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:53 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-154"}]: dispatch 2024-04-14T15:26:53.434 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:53 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi090-93652-154","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-14T15:26:53.434 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:53 smithi090 ceph-mon[41670]: osdmap e848: 8 total, 8 up, 7 in 2024-04-14T15:26:53.434 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:53 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-111","var": "dedup_tier","val": "test-rados-api-smithi090-93652-154"}]: dispatch 2024-04-14T15:26:53.434 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:53 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-111","var": "dedup_tier","val": "test-rados-api-smithi090-93652-154"}]: dispatch 2024-04-14T15:26:53.435 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:53 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:26:53.435 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:53 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:26:53.435 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:53 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-154"}]: dispatch 2024-04-14T15:26:53.435 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:53 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-154"}]: dispatch 2024-04-14T15:26:53.435 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:53 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-111","var": "dedup_tier","val": "test-rados-api-smithi090-93652-154"}]: dispatch 2024-04-14T15:26:53.435 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:53 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi090-93652-111","var": "dedup_tier","val": "test-rados-api-smithi090-93652-154"}]: dispatch 2024-04-14T15:26:53.435 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:53 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:26:53.435 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:53 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:26:53.435 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:53 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-154"}]: dispatch 2024-04-14T15:26:53.435 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:53 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi090-93652-111", "tierpool": "test-rados-api-smithi090-93652-154"}]: dispatch 2024-04-14T15:26:53.465 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:26:53.489 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:53.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:53.490 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:26:53.591 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:26:53.615 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:53.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:53.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:26:53.717 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:26:53.741 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:53.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:53.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:26:53.754 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":1265,"stamp":"2024-04-14T15:26:51.214977+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4596696,"num_objects":243,"num_object_clones":1,"num_object_copies":729,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":243,"num_whiteouts":0,"num_read":2194,"num_read_kb":1927,"num_write":797,"num_write_kb":4020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":2670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1018,"ondisk_log_size":1018,"up":576,"acting":576,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":626,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":741068,"kb_used_data":57684,"kb_used_omap":10,"kb_used_meta":683253,"kb_avail":655474996,"statfs":{"total":671965249536,"available":671206395904,"internally_reserved":0,"allocated":59068416,"data_stored":33644971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11156,"internal_metadata":699651180},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":10,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":7,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001287"},"pg_stats":[{"pgid":"345.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:26:51.044917+0000","last_change":"2024-04-14T15:26:51.044917+0000","last_active":"2024-04-14T15:26:51.044917+0000","last_peered":"2024-04-14T15:26:51.044917+0000","last_clean":"2024-04-14T15:26:51.044917+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:26:51.044917+0000","last_undegraded":"2024-04-14T15:26:51.044917+0000","last_fullsized":"2024-04-14T15:26:51.044917+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:51.044917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:51.044917+0000","last_clean_scrub_stamp":"2024-04-14T15:26:51.044917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"345.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:26:51.044917+0000","last_change":"2024-04-14T15:26:51.044917+0000","last_active":"2024-04-14T15:26:51.044917+0000","last_peered":"2024-04-14T15:26:51.044917+0000","last_clean":"2024-04-14T15:26:51.044917+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:26:51.044917+0000","last_undegraded":"2024-04-14T15:26:51.044917+0000","last_fullsized":"2024-04-14T15:26:51.044917+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:51.044917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:51.044917+0000","last_clean_scrub_stamp":"2024-04-14T15:26:51.044917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"345.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:26:51.044917+0000","last_change":"2024-04-14T15:26:51.044917+0000","last_active":"2024-04-14T15:26:51.044917+0000","last_peered":"2024-04-14T15:26:51.044917+0000","last_clean":"2024-04-14T15:26:51.044917+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:26:51.044917+0000","last_undegraded":"2024-04-14T15:26:51.044917+0000","last_fullsized":"2024-04-14T15:26:51.044917+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:51.044917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:51.044917+0000","last_clean_scrub_stamp":"2024-04-14T15:26:51.044917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"345.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:26:51.044917+0000","last_change":"2024-04-14T15:26:51.044917+0000","last_active":"2024-04-14T15:26:51.044917+0000","last_peered":"2024-04-14T15:26:51.044917+0000","last_clean":"2024-04-14T15:26:51.044917+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:26:51.044917+0000","last_undegraded":"2024-04-14T15:26:51.044917+0000","last_fullsized":"2024-04-14T15:26:51.044917+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:51.044917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:51.044917+0000","last_clean_scrub_stamp":"2024-04-14T15:26:51.044917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"345.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:26:51.044917+0000","last_change":"2024-04-14T15:26:51.044917+0000","last_active":"2024-04-14T15:26:51.044917+0000","last_peered":"2024-04-14T15:26:51.044917+0000","last_clean":"2024-04-14T15:26:51.044917+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:26:51.044917+0000","last_undegraded":"2024-04-14T15:26:51.044917+0000","last_fullsized":"2024-04-14T15:26:51.044917+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:51.044917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:51.044917+0000","last_clean_scrub_stamp":"2024-04-14T15:26:51.044917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"345.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:26:51.044917+0000","last_change":"2024-04-14T15:26:51.044917+0000","last_active":"2024-04-14T15:26:51.044917+0000","last_peered":"2024-04-14T15:26:51.044917+0000","last_clean":"2024-04-14T15:26:51.044917+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:26:51.044917+0000","last_undegraded":"2024-04-14T15:26:51.044917+0000","last_fullsized":"2024-04-14T15:26:51.044917+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:51.044917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:51.044917+0000","last_clean_scrub_stamp":"2024-04-14T15:26:51.044917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"345.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:26:51.044917+0000","last_change":"2024-04-14T15:26:51.044917+0000","last_active":"2024-04-14T15:26:51.044917+0000","last_peered":"2024-04-14T15:26:51.044917+0000","last_clean":"2024-04-14T15:26:51.044917+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:26:51.044917+0000","last_undegraded":"2024-04-14T15:26:51.044917+0000","last_fullsized":"2024-04-14T15:26:51.044917+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:51.044917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:51.044917+0000","last_clean_scrub_stamp":"2024-04-14T15:26:51.044917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"345.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:26:51.044917+0000","last_change":"2024-04-14T15:26:51.044917+0000","last_active":"2024-04-14T15:26:51.044917+0000","last_peered":"2024-04-14T15:26:51.044917+0000","last_clean":"2024-04-14T15:26:51.044917+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:26:51.044917+0000","last_undegraded":"2024-04-14T15:26:51.044917+0000","last_fullsized":"2024-04-14T15:26:51.044917+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:51.044917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:51.044917+0000","last_clean_scrub_stamp":"2024-04-14T15:26:51.044917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"345.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:26:51.044917+0000","last_change":"2024-04-14T15:26:51.044917+0000","last_active":"2024-04-14T15:26:51.044917+0000","last_peered":"2024-04-14T15:26:51.044917+0000","last_clean":"2024-04-14T15:26:51.044917+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:26:51.044917+0000","last_undegraded":"2024-04-14T15:26:51.044917+0000","last_fullsized":"2024-04-14T15:26:51.044917+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:51.044917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:51.044917+0000","last_clean_scrub_stamp":"2024-04-14T15:26:51.044917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"345.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:26:51.044917+0000","last_change":"2024-04-14T15:26:51.044917+0000","last_active":"2024-04-14T15:26:51.044917+0000","last_peered":"2024-04-14T15:26:51.044917+0000","last_clean":"2024-04-14T15:26:51.044917+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:26:51.044917+0000","last_undegraded":"2024-04-14T15:26:51.044917+0000","last_fullsized":"2024-04-14T15:26:51.044917+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:51.044917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:51.044917+0000","last_clean_scrub_stamp":"2024-04-14T15:26:51.044917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"345.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:26:51.044917+0000","last_change":"2024-04-14T15:26:51.044917+0000","last_active":"2024-04-14T15:26:51.044917+0000","last_peered":"2024-04-14T15:26:51.044917+0000","last_clean":"2024-04-14T15:26:51.044917+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:26:51.044917+0000","last_undegraded":"2024-04-14T15:26:51.044917+0000","last_fullsized":"2024-04-14T15:26:51.044917+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:51.044917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:51.044917+0000","last_clean_scrub_stamp":"2024-04-14T15:26:51.044917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"345.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:26:51.044917+0000","last_change":"2024-04-14T15:26:51.044917+0000","last_active":"2024-04-14T15:26:51.044917+0000","last_peered":"2024-04-14T15:26:51.044917+0000","last_clean":"2024-04-14T15:26:51.044917+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:26:51.044917+0000","last_undegraded":"2024-04-14T15:26:51.044917+0000","last_fullsized":"2024-04-14T15:26:51.044917+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:51.044917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:51.044917+0000","last_clean_scrub_stamp":"2024-04-14T15:26:51.044917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"345.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:26:51.044917+0000","last_change":"2024-04-14T15:26:51.044917+0000","last_active":"2024-04-14T15:26:51.044917+0000","last_peered":"2024-04-14T15:26:51.044917+0000","last_clean":"2024-04-14T15:26:51.044917+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:26:51.044917+0000","last_undegraded":"2024-04-14T15:26:51.044917+0000","last_fullsized":"2024-04-14T15:26:51.044917+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:51.044917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:51.044917+0000","last_clean_scrub_stamp":"2024-04-14T15:26:51.044917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"345.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:26:51.044917+0000","last_change":"2024-04-14T15:26:51.044917+0000","last_active":"2024-04-14T15:26:51.044917+0000","last_peered":"2024-04-14T15:26:51.044917+0000","last_clean":"2024-04-14T15:26:51.044917+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:26:51.044917+0000","last_undegraded":"2024-04-14T15:26:51.044917+0000","last_fullsized":"2024-04-14T15:26:51.044917+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:51.044917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:51.044917+0000","last_clean_scrub_stamp":"2024-04-14T15:26:51.044917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"345.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:26:51.044917+0000","last_change":"2024-04-14T15:26:51.044917+0000","last_active":"2024-04-14T15:26:51.044917+0000","last_peered":"2024-04-14T15:26:51.044917+0000","last_clean":"2024-04-14T15:26:51.044917+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:26:51.044917+0000","last_undegraded":"2024-04-14T15:26:51.044917+0000","last_fullsized":"2024-04-14T15:26:51.044917+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:51.044917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:51.044917+0000","last_clean_scrub_stamp":"2024-04-14T15:26:51.044917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"345.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:26:51.044917+0000","last_change":"2024-04-14T15:26:51.044917+0000","last_active":"2024-04-14T15:26:51.044917+0000","last_peered":"2024-04-14T15:26:51.044917+0000","last_clean":"2024-04-14T15:26:51.044917+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:26:51.044917+0000","last_undegraded":"2024-04-14T15:26:51.044917+0000","last_fullsized":"2024-04-14T15:26:51.044917+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:51.044917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:51.044917+0000","last_clean_scrub_stamp":"2024-04-14T15:26:51.044917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"345.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:26:51.044917+0000","last_change":"2024-04-14T15:26:51.044917+0000","last_active":"2024-04-14T15:26:51.044917+0000","last_peered":"2024-04-14T15:26:51.044917+0000","last_clean":"2024-04-14T15:26:51.044917+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:26:51.044917+0000","last_undegraded":"2024-04-14T15:26:51.044917+0000","last_fullsized":"2024-04-14T15:26:51.044917+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:51.044917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:51.044917+0000","last_clean_scrub_stamp":"2024-04-14T15:26:51.044917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"345.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:26:51.044917+0000","last_change":"2024-04-14T15:26:51.044917+0000","last_active":"2024-04-14T15:26:51.044917+0000","last_peered":"2024-04-14T15:26:51.044917+0000","last_clean":"2024-04-14T15:26:51.044917+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:26:51.044917+0000","last_undegraded":"2024-04-14T15:26:51.044917+0000","last_fullsized":"2024-04-14T15:26:51.044917+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:51.044917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:51.044917+0000","last_clean_scrub_stamp":"2024-04-14T15:26:51.044917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"345.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:26:51.044917+0000","last_change":"2024-04-14T15:26:51.044917+0000","last_active":"2024-04-14T15:26:51.044917+0000","last_peered":"2024-04-14T15:26:51.044917+0000","last_clean":"2024-04-14T15:26:51.044917+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:26:51.044917+0000","last_undegraded":"2024-04-14T15:26:51.044917+0000","last_fullsized":"2024-04-14T15:26:51.044917+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:51.044917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:51.044917+0000","last_clean_scrub_stamp":"2024-04-14T15:26:51.044917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"345.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:26:51.044917+0000","last_change":"2024-04-14T15:26:51.044917+0000","last_active":"2024-04-14T15:26:51.044917+0000","last_peered":"2024-04-14T15:26:51.044917+0000","last_clean":"2024-04-14T15:26:51.044917+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:26:51.044917+0000","last_undegraded":"2024-04-14T15:26:51.044917+0000","last_fullsized":"2024-04-14T15:26:51.044917+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:51.044917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:51.044917+0000","last_clean_scrub_stamp":"2024-04-14T15:26:51.044917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"345.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:26:51.044917+0000","last_change":"2024-04-14T15:26:51.044917+0000","last_active":"2024-04-14T15:26:51.044917+0000","last_peered":"2024-04-14T15:26:51.044917+0000","last_clean":"2024-04-14T15:26:51.044917+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:26:51.044917+0000","last_undegraded":"2024-04-14T15:26:51.044917+0000","last_fullsized":"2024-04-14T15:26:51.044917+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:51.044917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:51.044917+0000","last_clean_scrub_stamp":"2024-04-14T15:26:51.044917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"345.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:26:51.044917+0000","last_change":"2024-04-14T15:26:51.044917+0000","last_active":"2024-04-14T15:26:51.044917+0000","last_peered":"2024-04-14T15:26:51.044917+0000","last_clean":"2024-04-14T15:26:51.044917+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:26:51.044917+0000","last_undegraded":"2024-04-14T15:26:51.044917+0000","last_fullsized":"2024-04-14T15:26:51.044917+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:51.044917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:51.044917+0000","last_clean_scrub_stamp":"2024-04-14T15:26:51.044917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"345.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:26:51.044917+0000","last_change":"2024-04-14T15:26:51.044917+0000","last_active":"2024-04-14T15:26:51.044917+0000","last_peered":"2024-04-14T15:26:51.044917+0000","last_clean":"2024-04-14T15:26:51.044917+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:26:51.044917+0000","last_undegraded":"2024-04-14T15:26:51.044917+0000","last_fullsized":"2024-04-14T15:26:51.044917+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:51.044917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:51.044917+0000","last_clean_scrub_stamp":"2024-04-14T15:26:51.044917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"345.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:26:51.044917+0000","last_change":"2024-04-14T15:26:51.044917+0000","last_active":"2024-04-14T15:26:51.044917+0000","last_peered":"2024-04-14T15:26:51.044917+0000","last_clean":"2024-04-14T15:26:51.044917+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:26:51.044917+0000","last_undegraded":"2024-04-14T15:26:51.044917+0000","last_fullsized":"2024-04-14T15:26:51.044917+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:51.044917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:51.044917+0000","last_clean_scrub_stamp":"2024-04-14T15:26:51.044917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"345.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:26:51.044917+0000","last_change":"2024-04-14T15:26:51.044917+0000","last_active":"2024-04-14T15:26:51.044917+0000","last_peered":"2024-04-14T15:26:51.044917+0000","last_clean":"2024-04-14T15:26:51.044917+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:26:51.044917+0000","last_undegraded":"2024-04-14T15:26:51.044917+0000","last_fullsized":"2024-04-14T15:26:51.044917+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:51.044917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:51.044917+0000","last_clean_scrub_stamp":"2024-04-14T15:26:51.044917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"323.7","version":"798'3","reported_seq":446,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-04-14T15:26:48.047717+0000","last_change":"2024-04-14T15:26:27.071902+0000","last_active":"2024-04-14T15:26:48.047717+0000","last_peered":"2024-04-14T15:26:48.047717+0000","last_clean":"2024-04-14T15:26:48.047717+0000","last_became_active":"2024-04-14T15:26:27.070297+0000","last_became_peered":"2024-04-14T15:26:27.070297+0000","last_unstale":"2024-04-14T15:26:48.047717+0000","last_undegraded":"2024-04-14T15:26:48.047717+0000","last_fullsized":"2024-04-14T15:26:48.047717+0000","mapping_epoch":828,"log_start":"0'0","ondisk_log_start":"0'0","created":647,"last_epoch_clean":829,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:11.799436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_clean_scrub_stamp":"2024-04-14T15:25:11.799436+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:51.529253+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00079989799999999995,"stat_sum":{"num_bytes":24,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"345.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:26:51.044917+0000","last_change":"2024-04-14T15:26:51.044917+0000","last_active":"2024-04-14T15:26:51.044917+0000","last_peered":"2024-04-14T15:26:51.044917+0000","last_clean":"2024-04-14T15:26:51.044917+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:26:51.044917+0000","last_undegraded":"2024-04-14T15:26:51.044917+0000","last_fullsized":"2024-04-14T15:26:51.044917+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:51.044917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:51.044917+0000","last_clean_scrub_stamp":"2024-04-14T15:26:51.044917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"323.6","version":"821'5","reported_seq":524,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.047360+0000","last_change":"2024-04-14T15:25:49.959163+0000","last_active":"2024-04-14T15:26:49.047360+0000","last_peered":"2024-04-14T15:26:49.047360+0000","last_clean":"2024-04-14T15:26:49.047360+0000","last_became_active":"2024-04-14T15:23:04.006098+0000","last_became_peered":"2024-04-14T15:23:04.006098+0000","last_unstale":"2024-04-14T15:26:49.047360+0000","last_undegraded":"2024-04-14T15:26:49.047360+0000","last_fullsized":"2024-04-14T15:26:49.047360+0000","mapping_epoch":647,"log_start":"0'0","ondisk_log_start":"0'0","created":647,"last_epoch_clean":648,"parent":"0.0","parent_split_bits":0,"last_scrub":"798'4","last_scrub_stamp":"2024-04-14T15:25:49.959068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_clean_scrub_stamp":"2024-04-14T15:25:49.959068+0000","objects_scrubbed":4,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:58.558311+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0040397810000000001,"stat_sum":{"num_bytes":40,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"345.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:26:51.044917+0000","last_change":"2024-04-14T15:26:51.044917+0000","last_active":"2024-04-14T15:26:51.044917+0000","last_peered":"2024-04-14T15:26:51.044917+0000","last_clean":"2024-04-14T15:26:51.044917+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:26:51.044917+0000","last_undegraded":"2024-04-14T15:26:51.044917+0000","last_fullsized":"2024-04-14T15:26:51.044917+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:51.044917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:51.044917+0000","last_clean_scrub_stamp":"2024-04-14T15:26:51.044917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"323.5","version":"812'18","reported_seq":524,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.051226+0000","last_change":"2024-04-14T15:25:46.249717+0000","last_active":"2024-04-14T15:26:49.051226+0000","last_peered":"2024-04-14T15:26:49.051226+0000","last_clean":"2024-04-14T15:26:49.051226+0000","last_became_active":"2024-04-14T15:23:04.002370+0000","last_became_peered":"2024-04-14T15:23:04.002370+0000","last_unstale":"2024-04-14T15:26:49.051226+0000","last_undegraded":"2024-04-14T15:26:49.051226+0000","last_fullsized":"2024-04-14T15:26:49.051226+0000","mapping_epoch":647,"log_start":"0'0","ondisk_log_start":"0'0","created":647,"last_epoch_clean":648,"parent":"0.0","parent_split_bits":0,"last_scrub":"733'11","last_scrub_stamp":"2024-04-14T15:25:46.249607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_clean_scrub_stamp":"2024-04-14T15:25:46.249607+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:11.515900+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.0032579309999999999,"stat_sum":{"num_bytes":61,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":11,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"345.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:26:51.044917+0000","last_change":"2024-04-14T15:26:51.044917+0000","last_active":"2024-04-14T15:26:51.044917+0000","last_peered":"2024-04-14T15:26:51.044917+0000","last_clean":"2024-04-14T15:26:51.044917+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:26:51.044917+0000","last_undegraded":"2024-04-14T15:26:51.044917+0000","last_fullsized":"2024-04-14T15:26:51.044917+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:51.044917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:51.044917+0000","last_clean_scrub_stamp":"2024-04-14T15:26:51.044917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"323.4","version":"842'20","reported_seq":535,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.059087+0000","last_change":"2024-04-14T15:25:29.009921+0000","last_active":"2024-04-14T15:26:49.059087+0000","last_peered":"2024-04-14T15:26:49.059087+0000","last_clean":"2024-04-14T15:26:49.059087+0000","last_became_active":"2024-04-14T15:23:03.997912+0000","last_became_peered":"2024-04-14T15:23:03.997912+0000","last_unstale":"2024-04-14T15:26:49.059087+0000","last_undegraded":"2024-04-14T15:26:49.059087+0000","last_fullsized":"2024-04-14T15:26:49.059087+0000","mapping_epoch":647,"log_start":"0'0","ondisk_log_start":"0'0","created":647,"last_epoch_clean":648,"parent":"0.0","parent_split_bits":0,"last_scrub":"742'12","last_scrub_stamp":"2024-04-14T15:25:29.009822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_clean_scrub_stamp":"2024-04-14T15:25:29.009822+0000","objects_scrubbed":2,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:52.918531+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001097995,"stat_sum":{"num_bytes":50,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":8,"num_read_kb":2,"num_write":14,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"345.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:26:51.044917+0000","last_change":"2024-04-14T15:26:51.044917+0000","last_active":"2024-04-14T15:26:51.044917+0000","last_peered":"2024-04-14T15:26:51.044917+0000","last_clean":"2024-04-14T15:26:51.044917+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:26:51.044917+0000","last_undegraded":"2024-04-14T15:26:51.044917+0000","last_fullsized":"2024-04-14T15:26:51.044917+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:51.044917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:51.044917+0000","last_clean_scrub_stamp":"2024-04-14T15:26:51.044917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"323.3","version":"798'19","reported_seq":523,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.050941+0000","last_change":"2024-04-14T15:25:19.807060+0000","last_active":"2024-04-14T15:26:49.050941+0000","last_peered":"2024-04-14T15:26:49.050941+0000","last_clean":"2024-04-14T15:26:49.050941+0000","last_became_active":"2024-04-14T15:23:04.001213+0000","last_became_peered":"2024-04-14T15:23:04.001213+0000","last_unstale":"2024-04-14T15:26:49.050941+0000","last_undegraded":"2024-04-14T15:26:49.050941+0000","last_fullsized":"2024-04-14T15:26:49.050941+0000","mapping_epoch":647,"log_start":"0'0","ondisk_log_start":"0'0","created":647,"last_epoch_clean":648,"parent":"0.0","parent_split_bits":0,"last_scrub":"763'18","last_scrub_stamp":"2024-04-14T15:25:19.806953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_clean_scrub_stamp":"2024-04-14T15:25:19.806953+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:52.880683+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00085465599999999997,"stat_sum":{"num_bytes":37,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":13,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"345.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:26:51.044917+0000","last_change":"2024-04-14T15:26:51.044917+0000","last_active":"2024-04-14T15:26:51.044917+0000","last_peered":"2024-04-14T15:26:51.044917+0000","last_clean":"2024-04-14T15:26:51.044917+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:26:51.044917+0000","last_undegraded":"2024-04-14T15:26:51.044917+0000","last_fullsized":"2024-04-14T15:26:51.044917+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:51.044917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:51.044917+0000","last_clean_scrub_stamp":"2024-04-14T15:26:51.044917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"323.2","version":"838'21","reported_seq":535,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.044678+0000","last_change":"2024-04-14T15:25:01.027310+0000","last_active":"2024-04-14T15:26:49.044678+0000","last_peered":"2024-04-14T15:26:49.044678+0000","last_clean":"2024-04-14T15:26:49.044678+0000","last_became_active":"2024-04-14T15:23:04.001557+0000","last_became_peered":"2024-04-14T15:23:04.001557+0000","last_unstale":"2024-04-14T15:26:49.044678+0000","last_undegraded":"2024-04-14T15:26:49.044678+0000","last_fullsized":"2024-04-14T15:26:49.044678+0000","mapping_epoch":647,"log_start":"0'0","ondisk_log_start":"0'0","created":647,"last_epoch_clean":648,"parent":"0.0","parent_split_bits":0,"last_scrub":"710'6","last_scrub_stamp":"2024-04-14T15:24:15.331688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_clean_scrub_stamp":"2024-04-14T15:24:15.331688+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:51.082266+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.013115428,"stat_sum":{"num_bytes":20,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"345.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:26:51.044917+0000","last_change":"2024-04-14T15:26:51.044917+0000","last_active":"2024-04-14T15:26:51.044917+0000","last_peered":"2024-04-14T15:26:51.044917+0000","last_clean":"2024-04-14T15:26:51.044917+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:26:51.044917+0000","last_undegraded":"2024-04-14T15:26:51.044917+0000","last_fullsized":"2024-04-14T15:26:51.044917+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:51.044917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:51.044917+0000","last_clean_scrub_stamp":"2024-04-14T15:26:51.044917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"323.1","version":"798'5","reported_seq":515,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.042134+0000","last_change":"2024-04-14T15:25:54.933798+0000","last_active":"2024-04-14T15:26:49.042134+0000","last_peered":"2024-04-14T15:26:49.042134+0000","last_clean":"2024-04-14T15:26:49.042134+0000","last_became_active":"2024-04-14T15:23:03.998498+0000","last_became_peered":"2024-04-14T15:23:03.998498+0000","last_unstale":"2024-04-14T15:26:49.042134+0000","last_undegraded":"2024-04-14T15:26:49.042134+0000","last_fullsized":"2024-04-14T15:26:49.042134+0000","mapping_epoch":647,"log_start":"0'0","ondisk_log_start":"0'0","created":647,"last_epoch_clean":648,"parent":"0.0","parent_split_bits":0,"last_scrub":"798'5","last_scrub_stamp":"2024-04-14T15:25:54.933693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_clean_scrub_stamp":"2024-04-14T15:25:54.933693+0000","objects_scrubbed":4,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:16.797030+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001861021,"stat_sum":{"num_bytes":28,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"345.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-14T15:26:51.044917+0000","last_change":"2024-04-14T15:26:51.044917+0000","last_active":"2024-04-14T15:26:51.044917+0000","last_peered":"2024-04-14T15:26:51.044917+0000","last_clean":"2024-04-14T15:26:51.044917+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-14T15:26:51.044917+0000","last_undegraded":"2024-04-14T15:26:51.044917+0000","last_fullsized":"2024-04-14T15:26:51.044917+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:51.044917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:51.044917+0000","last_clean_scrub_stamp":"2024-04-14T15:26:51.044917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"323.0","version":"845'10","reported_seq":527,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.072743+0000","last_change":"2024-04-14T15:25:04.467370+0000","last_active":"2024-04-14T15:26:49.072743+0000","last_peered":"2024-04-14T15:26:49.072743+0000","last_clean":"2024-04-14T15:26:49.072743+0000","last_became_active":"2024-04-14T15:23:04.001020+0000","last_became_peered":"2024-04-14T15:23:04.001020+0000","last_unstale":"2024-04-14T15:26:49.072743+0000","last_undegraded":"2024-04-14T15:26:49.072743+0000","last_fullsized":"2024-04-14T15:26:49.072743+0000","mapping_epoch":647,"log_start":"0'0","ondisk_log_start":"0'0","created":647,"last_epoch_clean":648,"parent":"0.0","parent_split_bits":0,"last_scrub":"704'6","last_scrub_stamp":"2024-04-14T15:25:04.467266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:23:02.982695+0000","last_clean_scrub_stamp":"2024-04-14T15:25:04.467266+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:51.082403+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0014140909999999999,"stat_sum":{"num_bytes":4000039,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":7,"num_write_kb":3912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"91'130","reported_seq":523,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.043534+0000","last_change":"2024-04-14T15:25:02.924588+0000","last_active":"2024-04-14T15:26:49.043534+0000","last_peered":"2024-04-14T15:26:49.043534+0000","last_clean":"2024-04-14T15:26:49.043534+0000","last_became_active":"2024-04-14T15:25:02.025875+0000","last_became_peered":"2024-04-14T15:25:02.025875+0000","last_unstale":"2024-04-14T15:26:49.043534+0000","last_undegraded":"2024-04-14T15:26:49.043534+0000","last_fullsized":"2024-04-14T15:26:49.043534+0000","mapping_epoch":757,"log_start":"91'130","ondisk_log_start":"91'130","created":752,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:25:02.924460+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:25:02.924460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:51.998645+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":395,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-04-14T15:26:48.047982+0000","last_change":"2024-04-14T15:26:27.059369+0000","last_active":"2024-04-14T15:26:48.047982+0000","last_peered":"2024-04-14T15:26:48.047982+0000","last_clean":"2024-04-14T15:26:48.047982+0000","last_became_active":"2024-04-14T15:26:27.058780+0000","last_became_peered":"2024-04-14T15:26:27.058780+0000","last_unstale":"2024-04-14T15:26:48.047982+0000","last_undegraded":"2024-04-14T15:26:48.047982+0000","last_fullsized":"2024-04-14T15:26:48.047982+0000","mapping_epoch":828,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":829,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:04.159220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:26:04.159220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:32.859605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"528'10","reported_seq":1838,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.043070+0000","last_change":"2024-04-14T15:24:37.737273+0000","last_active":"2024-04-14T15:26:49.043070+0000","last_peered":"2024-04-14T15:26:49.043070+0000","last_clean":"2024-04-14T15:26:49.043070+0000","last_became_active":"2024-04-14T15:20:52.984943+0000","last_became_peered":"2024-04-14T15:20:52.984943+0000","last_unstale":"2024-04-14T15:26:49.043070+0000","last_undegraded":"2024-04-14T15:26:49.043070+0000","last_fullsized":"2024-04-14T15:26:49.043070+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"528'10","last_scrub_stamp":"2024-04-14T15:24:37.737147+0000","last_deep_scrub":"85'9","last_deep_scrub_stamp":"2024-04-14T15:10:51.950580+0000","last_clean_scrub_stamp":"2024-04-14T15:24:37.737147+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:51.998532+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"479'34","reported_seq":532,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-04-14T15:26:48.044265+0000","last_change":"2024-04-14T15:26:27.059556+0000","last_active":"2024-04-14T15:26:48.044265+0000","last_peered":"2024-04-14T15:26:48.044265+0000","last_clean":"2024-04-14T15:26:48.044265+0000","last_became_active":"2024-04-14T15:26:27.058780+0000","last_became_peered":"2024-04-14T15:26:27.058780+0000","last_unstale":"2024-04-14T15:26:48.044265+0000","last_undegraded":"2024-04-14T15:26:48.044265+0000","last_fullsized":"2024-04-14T15:26:48.044265+0000","mapping_epoch":828,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":829,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'34","last_scrub_stamp":"2024-04-14T15:25:26.852851+0000","last_deep_scrub":"430'22","last_deep_scrub_stamp":"2024-04-14T15:18:39.704883+0000","last_clean_scrub_stamp":"2024-04-14T15:25:26.852851+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:51.529329+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":1627,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.044832+0000","last_change":"2024-04-14T15:25:51.674598+0000","last_active":"2024-04-14T15:26:49.044832+0000","last_peered":"2024-04-14T15:26:49.044832+0000","last_clean":"2024-04-14T15:26:49.044832+0000","last_became_active":"2024-04-14T15:09:43.021460+0000","last_became_peered":"2024-04-14T15:09:43.021460+0000","last_unstale":"2024-04-14T15:26:49.044832+0000","last_undegraded":"2024-04-14T15:26:49.044832+0000","last_fullsized":"2024-04-14T15:26:49.044832+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:51.674520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:25:51.674520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:08.097729+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":486,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-04-14T15:26:48.045802+0000","last_change":"2024-04-14T15:26:27.070494+0000","last_active":"2024-04-14T15:26:48.045802+0000","last_peered":"2024-04-14T15:26:48.045802+0000","last_clean":"2024-04-14T15:26:48.045802+0000","last_became_active":"2024-04-14T15:26:27.068039+0000","last_became_peered":"2024-04-14T15:26:27.068039+0000","last_unstale":"2024-04-14T15:26:48.045802+0000","last_undegraded":"2024-04-14T15:26:48.045802+0000","last_fullsized":"2024-04-14T15:26:48.045802+0000","mapping_epoch":828,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":829,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:49.122278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:15:40.368219+0000","last_clean_scrub_stamp":"2024-04-14T15:25:49.122278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:53.822924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"91'130","reported_seq":525,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.050830+0000","last_change":"2024-04-14T15:25:02.372541+0000","last_active":"2024-04-14T15:26:49.050830+0000","last_peered":"2024-04-14T15:26:49.050830+0000","last_clean":"2024-04-14T15:26:49.050830+0000","last_became_active":"2024-04-14T15:25:02.026456+0000","last_became_peered":"2024-04-14T15:25:02.026456+0000","last_unstale":"2024-04-14T15:26:49.050830+0000","last_undegraded":"2024-04-14T15:26:49.050830+0000","last_fullsized":"2024-04-14T15:26:49.050830+0000","mapping_epoch":757,"log_start":"91'130","ondisk_log_start":"91'130","created":752,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:25:02.372440+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:25:02.372440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:50.168016+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":1524,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.043655+0000","last_change":"2024-04-14T15:25:20.823771+0000","last_active":"2024-04-14T15:26:49.043655+0000","last_peered":"2024-04-14T15:26:49.043655+0000","last_clean":"2024-04-14T15:26:49.043655+0000","last_became_active":"2024-04-14T15:20:52.991952+0000","last_became_peered":"2024-04-14T15:20:52.991952+0000","last_unstale":"2024-04-14T15:26:49.043655+0000","last_undegraded":"2024-04-14T15:26:49.043655+0000","last_fullsized":"2024-04-14T15:26:49.043655+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:20.823680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:25:20.823680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:51.998680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"479'15","reported_seq":1648,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.053235+0000","last_change":"2024-04-14T15:25:50.132021+0000","last_active":"2024-04-14T15:26:49.053235+0000","last_peered":"2024-04-14T15:26:49.053235+0000","last_clean":"2024-04-14T15:26:49.053235+0000","last_became_active":"2024-04-14T15:09:45.037179+0000","last_became_peered":"2024-04-14T15:09:45.037179+0000","last_unstale":"2024-04-14T15:26:49.053235+0000","last_undegraded":"2024-04-14T15:26:49.053235+0000","last_fullsized":"2024-04-14T15:26:49.053235+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'15","last_scrub_stamp":"2024-04-14T15:25:50.131933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:25:50.131933+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:51.346947+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":1626,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.048616+0000","last_change":"2024-04-14T15:25:47.000217+0000","last_active":"2024-04-14T15:26:49.048616+0000","last_peered":"2024-04-14T15:26:49.048616+0000","last_clean":"2024-04-14T15:26:49.048616+0000","last_became_active":"2024-04-14T15:09:43.020044+0000","last_became_peered":"2024-04-14T15:09:43.020044+0000","last_unstale":"2024-04-14T15:26:49.048616+0000","last_undegraded":"2024-04-14T15:26:49.048616+0000","last_fullsized":"2024-04-14T15:26:49.048616+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:47.000098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:25:47.000098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:52.332018+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1627,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.046749+0000","last_change":"2024-04-14T15:26:32.506633+0000","last_active":"2024-04-14T15:26:49.046749+0000","last_peered":"2024-04-14T15:26:49.046749+0000","last_clean":"2024-04-14T15:26:49.046749+0000","last_became_active":"2024-04-14T15:09:47.046153+0000","last_became_peered":"2024-04-14T15:09:47.046153+0000","last_unstale":"2024-04-14T15:26:49.046749+0000","last_undegraded":"2024-04-14T15:26:49.046749+0000","last_fullsized":"2024-04-14T15:26:49.046749+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:32.506557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:26:32.506557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:52.401034+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1615,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.056451+0000","last_change":"2024-04-14T15:26:05.045908+0000","last_active":"2024-04-14T15:26:49.056451+0000","last_peered":"2024-04-14T15:26:49.056451+0000","last_clean":"2024-04-14T15:26:49.056451+0000","last_became_active":"2024-04-14T15:09:49.061495+0000","last_became_peered":"2024-04-14T15:09:49.061495+0000","last_unstale":"2024-04-14T15:26:49.056451+0000","last_undegraded":"2024-04-14T15:26:49.056451+0000","last_fullsized":"2024-04-14T15:26:49.056451+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:05.045782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:23:37.201795+0000","last_clean_scrub_stamp":"2024-04-14T15:26:05.045782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:30.855824+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"91'130","reported_seq":372,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-04-14T15:26:48.047538+0000","last_change":"2024-04-14T15:26:37.508075+0000","last_active":"2024-04-14T15:26:48.047538+0000","last_peered":"2024-04-14T15:26:48.047538+0000","last_clean":"2024-04-14T15:26:48.047538+0000","last_became_active":"2024-04-14T15:26:27.070732+0000","last_became_peered":"2024-04-14T15:26:27.070732+0000","last_unstale":"2024-04-14T15:26:48.047538+0000","last_undegraded":"2024-04-14T15:26:48.047538+0000","last_fullsized":"2024-04-14T15:26:48.047538+0000","mapping_epoch":828,"log_start":"91'130","ondisk_log_start":"91'130","created":752,"last_epoch_clean":829,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:26:37.507574+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:26:37.507574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:38.364719+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":1524,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.044141+0000","last_change":"2024-04-14T15:25:27.868432+0000","last_active":"2024-04-14T15:26:49.044141+0000","last_peered":"2024-04-14T15:26:49.044141+0000","last_clean":"2024-04-14T15:26:49.044141+0000","last_became_active":"2024-04-14T15:20:52.999845+0000","last_became_peered":"2024-04-14T15:20:52.999845+0000","last_unstale":"2024-04-14T15:26:49.044141+0000","last_undegraded":"2024-04-14T15:26:49.044141+0000","last_fullsized":"2024-04-14T15:26:49.044141+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:27.868337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:25:27.868337+0000","last_clean_scrub_stamp":"2024-04-14T15:25:27.868337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:51.998730+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"479'44","reported_seq":515,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.046255+0000","last_change":"2024-04-14T15:25:26.767683+0000","last_active":"2024-04-14T15:26:49.046255+0000","last_peered":"2024-04-14T15:26:49.046255+0000","last_clean":"2024-04-14T15:26:49.046255+0000","last_became_active":"2024-04-14T15:22:25.970899+0000","last_became_peered":"2024-04-14T15:22:25.970899+0000","last_unstale":"2024-04-14T15:26:49.046255+0000","last_undegraded":"2024-04-14T15:26:49.046255+0000","last_fullsized":"2024-04-14T15:26:49.046255+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'44","last_scrub_stamp":"2024-04-14T15:25:26.767559+0000","last_deep_scrub":"479'44","last_deep_scrub_stamp":"2024-04-14T15:25:26.767559+0000","last_clean_scrub_stamp":"2024-04-14T15:25:26.767559+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":1635,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.043596+0000","last_change":"2024-04-14T15:25:53.717840+0000","last_active":"2024-04-14T15:26:49.043596+0000","last_peered":"2024-04-14T15:26:49.043596+0000","last_clean":"2024-04-14T15:26:49.043596+0000","last_became_active":"2024-04-14T15:09:43.020268+0000","last_became_peered":"2024-04-14T15:09:43.020268+0000","last_unstale":"2024-04-14T15:26:49.043596+0000","last_undegraded":"2024-04-14T15:26:49.043596+0000","last_fullsized":"2024-04-14T15:26:49.043596+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:53.717687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:34.342793+0000","last_clean_scrub_stamp":"2024-04-14T15:25:53.717687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:01.063625+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1619,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.043480+0000","last_change":"2024-04-14T15:25:56.701857+0000","last_active":"2024-04-14T15:26:49.043480+0000","last_peered":"2024-04-14T15:26:49.043480+0000","last_clean":"2024-04-14T15:26:49.043480+0000","last_became_active":"2024-04-14T15:09:47.045542+0000","last_became_peered":"2024-04-14T15:09:47.045542+0000","last_unstale":"2024-04-14T15:26:49.043480+0000","last_undegraded":"2024-04-14T15:26:49.043480+0000","last_fullsized":"2024-04-14T15:26:49.043480+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:56.701776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:25:56.701776+0000","last_clean_scrub_stamp":"2024-04-14T15:25:56.701776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:18.809662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1609,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.042395+0000","last_change":"2024-04-14T15:26:01.950975+0000","last_active":"2024-04-14T15:26:49.042395+0000","last_peered":"2024-04-14T15:26:49.042395+0000","last_clean":"2024-04-14T15:26:49.042395+0000","last_became_active":"2024-04-14T15:09:49.065397+0000","last_became_peered":"2024-04-14T15:09:49.065397+0000","last_unstale":"2024-04-14T15:26:49.042395+0000","last_undegraded":"2024-04-14T15:26:49.042395+0000","last_fullsized":"2024-04-14T15:26:49.042395+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.950897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.950897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:17.096261+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"91'130","reported_seq":556,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-04-14T15:26:48.044098+0000","last_change":"2024-04-14T15:26:34.522057+0000","last_active":"2024-04-14T15:26:48.044098+0000","last_peered":"2024-04-14T15:26:48.044098+0000","last_clean":"2024-04-14T15:26:48.044098+0000","last_became_active":"2024-04-14T15:26:27.064317+0000","last_became_peered":"2024-04-14T15:26:27.064317+0000","last_unstale":"2024-04-14T15:26:48.044098+0000","last_undegraded":"2024-04-14T15:26:48.044098+0000","last_fullsized":"2024-04-14T15:26:48.044098+0000","mapping_epoch":828,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":829,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:26:34.521889+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:26:34.521889+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":70,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:49.683912+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":1532,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.046656+0000","last_change":"2024-04-14T15:26:28.499287+0000","last_active":"2024-04-14T15:26:49.046656+0000","last_peered":"2024-04-14T15:26:49.046656+0000","last_clean":"2024-04-14T15:26:49.046656+0000","last_became_active":"2024-04-14T15:12:56.474322+0000","last_became_peered":"2024-04-14T15:12:56.474322+0000","last_unstale":"2024-04-14T15:26:49.046656+0000","last_undegraded":"2024-04-14T15:26:49.046656+0000","last_fullsized":"2024-04-14T15:26:49.046656+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:28.499205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:26:28.499205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:29.725895+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"780'39","reported_seq":1721,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.052010+0000","last_change":"2024-04-14T15:25:24.296864+0000","last_active":"2024-04-14T15:26:49.052010+0000","last_peered":"2024-04-14T15:26:49.052010+0000","last_clean":"2024-04-14T15:26:49.052010+0000","last_became_active":"2024-04-14T15:13:12.933907+0000","last_became_peered":"2024-04-14T15:13:12.933907+0000","last_unstale":"2024-04-14T15:26:49.052010+0000","last_undegraded":"2024-04-14T15:26:49.052010+0000","last_fullsized":"2024-04-14T15:26:49.052010+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"780'39","last_scrub_stamp":"2024-04-14T15:25:24.296779+0000","last_deep_scrub":"780'39","last_deep_scrub_stamp":"2024-04-14T15:25:24.296779+0000","last_clean_scrub_stamp":"2024-04-14T15:25:24.296779+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:50.168242+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":1613,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.043102+0000","last_change":"2024-04-14T15:23:26.611299+0000","last_active":"2024-04-14T15:26:49.043102+0000","last_peered":"2024-04-14T15:26:49.043102+0000","last_clean":"2024-04-14T15:26:49.043102+0000","last_became_active":"2024-04-14T15:09:43.022351+0000","last_became_peered":"2024-04-14T15:09:43.022351+0000","last_unstale":"2024-04-14T15:26:49.043102+0000","last_undegraded":"2024-04-14T15:26:49.043102+0000","last_fullsized":"2024-04-14T15:26:49.043102+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:26.611217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:23:26.611217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:51.998370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1618,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.046979+0000","last_change":"2024-04-14T15:25:50.984808+0000","last_active":"2024-04-14T15:26:49.046979+0000","last_peered":"2024-04-14T15:26:49.046979+0000","last_clean":"2024-04-14T15:26:49.046979+0000","last_became_active":"2024-04-14T15:09:47.042810+0000","last_became_peered":"2024-04-14T15:09:47.042810+0000","last_unstale":"2024-04-14T15:26:49.046979+0000","last_undegraded":"2024-04-14T15:26:49.046979+0000","last_fullsized":"2024-04-14T15:26:49.046979+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:50.984722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:25:50.984722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:14.657168+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1615,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.043863+0000","last_change":"2024-04-14T15:25:27.741050+0000","last_active":"2024-04-14T15:26:49.043863+0000","last_peered":"2024-04-14T15:26:49.043863+0000","last_clean":"2024-04-14T15:26:49.043863+0000","last_became_active":"2024-04-14T15:09:49.069594+0000","last_became_peered":"2024-04-14T15:09:49.069594+0000","last_unstale":"2024-04-14T15:26:49.043863+0000","last_undegraded":"2024-04-14T15:26:49.043863+0000","last_fullsized":"2024-04-14T15:26:49.043863+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:27.740959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:25:27.740959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:50.237809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"91'130","reported_seq":1969,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.042433+0000","last_change":"2024-04-14T15:25:51.968976+0000","last_active":"2024-04-14T15:26:49.042433+0000","last_peered":"2024-04-14T15:26:49.042433+0000","last_clean":"2024-04-14T15:26:49.042433+0000","last_became_active":"2024-04-14T15:21:09.020300+0000","last_became_peered":"2024-04-14T15:21:09.020300+0000","last_unstale":"2024-04-14T15:26:49.042433+0000","last_undegraded":"2024-04-14T15:26:49.042433+0000","last_fullsized":"2024-04-14T15:26:49.042433+0000","mapping_epoch":189,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":544,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:25:51.968864+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:25:51.968864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:57.531574+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":653,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.042888+0000","last_change":"2024-04-14T15:23:41.487909+0000","last_active":"2024-04-14T15:26:49.042888+0000","last_peered":"2024-04-14T15:26:49.042888+0000","last_clean":"2024-04-14T15:26:49.042888+0000","last_became_active":"2024-04-14T15:20:52.980375+0000","last_became_peered":"2024-04-14T15:20:52.980375+0000","last_unstale":"2024-04-14T15:26:49.042888+0000","last_undegraded":"2024-04-14T15:26:49.042888+0000","last_fullsized":"2024-04-14T15:26:49.042888+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:41.487814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:23:41.487814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:51.998452+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"479'25","reported_seq":1661,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.050929+0000","last_change":"2024-04-14T15:24:34.394922+0000","last_active":"2024-04-14T15:26:49.050929+0000","last_peered":"2024-04-14T15:26:49.050929+0000","last_clean":"2024-04-14T15:26:49.050929+0000","last_became_active":"2024-04-14T15:20:52.987699+0000","last_became_peered":"2024-04-14T15:20:52.987699+0000","last_unstale":"2024-04-14T15:26:49.050929+0000","last_undegraded":"2024-04-14T15:26:49.050929+0000","last_fullsized":"2024-04-14T15:26:49.050929+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'25","last_scrub_stamp":"2024-04-14T15:24:34.394825+0000","last_deep_scrub":"479'25","last_deep_scrub_stamp":"2024-04-14T15:20:03.571160+0000","last_clean_scrub_stamp":"2024-04-14T15:24:34.394825+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:50.167776+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":1634,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.045261+0000","last_change":"2024-04-14T15:26:29.289052+0000","last_active":"2024-04-14T15:26:49.045261+0000","last_peered":"2024-04-14T15:26:49.045261+0000","last_clean":"2024-04-14T15:26:49.045261+0000","last_became_active":"2024-04-14T15:09:43.018871+0000","last_became_peered":"2024-04-14T15:09:43.018871+0000","last_unstale":"2024-04-14T15:26:49.045261+0000","last_undegraded":"2024-04-14T15:26:49.045261+0000","last_fullsized":"2024-04-14T15:26:49.045261+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:29.288963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:26:29.288963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:47.960749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"400'9","reported_seq":1834,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.042895+0000","last_change":"2024-04-14T15:25:49.676706+0000","last_active":"2024-04-14T15:26:49.042895+0000","last_peered":"2024-04-14T15:26:49.042895+0000","last_clean":"2024-04-14T15:26:49.042895+0000","last_became_active":"2024-04-14T15:09:47.044085+0000","last_became_peered":"2024-04-14T15:09:47.044085+0000","last_unstale":"2024-04-14T15:26:49.042895+0000","last_undegraded":"2024-04-14T15:26:49.042895+0000","last_fullsized":"2024-04-14T15:26:49.042895+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'9","last_scrub_stamp":"2024-04-14T15:25:49.676621+0000","last_deep_scrub":"400'9","last_deep_scrub_stamp":"2024-04-14T15:17:54.845881+0000","last_clean_scrub_stamp":"2024-04-14T15:25:49.676621+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:06.813931+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1600,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.051325+0000","last_change":"2024-04-14T15:24:19.552836+0000","last_active":"2024-04-14T15:26:49.051325+0000","last_peered":"2024-04-14T15:26:49.051325+0000","last_clean":"2024-04-14T15:26:49.051325+0000","last_became_active":"2024-04-14T15:09:49.065274+0000","last_became_peered":"2024-04-14T15:09:49.065274+0000","last_unstale":"2024-04-14T15:26:49.051325+0000","last_undegraded":"2024-04-14T15:26:49.051325+0000","last_fullsized":"2024-04-14T15:26:49.051325+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:19.552703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:18.588546+0000","last_clean_scrub_stamp":"2024-04-14T15:24:19.552703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:50.167804+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"91'130","reported_seq":556,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-04-14T15:26:48.044694+0000","last_change":"2024-04-14T15:26:36.486385+0000","last_active":"2024-04-14T15:26:48.044694+0000","last_peered":"2024-04-14T15:26:48.044694+0000","last_clean":"2024-04-14T15:26:48.044694+0000","last_became_active":"2024-04-14T15:26:27.067601+0000","last_became_peered":"2024-04-14T15:26:27.067601+0000","last_unstale":"2024-04-14T15:26:48.044694+0000","last_undegraded":"2024-04-14T15:26:48.044694+0000","last_fullsized":"2024-04-14T15:26:48.044694+0000","mapping_epoch":828,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":829,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:26:36.486259+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:26:36.486259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:44.330925+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":1523,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.043369+0000","last_change":"2024-04-14T15:25:54.058911+0000","last_active":"2024-04-14T15:26:49.043369+0000","last_peered":"2024-04-14T15:26:49.043369+0000","last_clean":"2024-04-14T15:26:49.043369+0000","last_became_active":"2024-04-14T15:12:56.487615+0000","last_became_peered":"2024-04-14T15:12:56.487615+0000","last_unstale":"2024-04-14T15:26:49.043369+0000","last_undegraded":"2024-04-14T15:26:49.043369+0000","last_fullsized":"2024-04-14T15:26:49.043369+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:54.058770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:25:54.058770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:22.907144+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"479'34","reported_seq":1672,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.048950+0000","last_change":"2024-04-14T15:24:35.378498+0000","last_active":"2024-04-14T15:26:49.048950+0000","last_peered":"2024-04-14T15:26:49.048950+0000","last_clean":"2024-04-14T15:26:49.048950+0000","last_became_active":"2024-04-14T15:09:45.033331+0000","last_became_peered":"2024-04-14T15:09:45.033331+0000","last_unstale":"2024-04-14T15:26:49.048950+0000","last_undegraded":"2024-04-14T15:26:49.048950+0000","last_fullsized":"2024-04-14T15:26:49.048950+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'34","last_scrub_stamp":"2024-04-14T15:24:35.378379+0000","last_deep_scrub":"479'34","last_deep_scrub_stamp":"2024-04-14T15:24:35.378379+0000","last_clean_scrub_stamp":"2024-04-14T15:24:35.378379+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:51.082349+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":1626,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.043677+0000","last_change":"2024-04-14T15:25:47.989868+0000","last_active":"2024-04-14T15:26:49.043677+0000","last_peered":"2024-04-14T15:26:49.043677+0000","last_clean":"2024-04-14T15:26:49.043677+0000","last_became_active":"2024-04-14T15:09:43.020878+0000","last_became_peered":"2024-04-14T15:09:43.020878+0000","last_unstale":"2024-04-14T15:26:49.043677+0000","last_undegraded":"2024-04-14T15:26:49.043677+0000","last_fullsized":"2024-04-14T15:26:49.043677+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:47.989790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:25:47.989790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:53.293672+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"72'8","reported_seq":1818,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.051297+0000","last_change":"2024-04-14T15:24:27.403836+0000","last_active":"2024-04-14T15:26:49.051297+0000","last_peered":"2024-04-14T15:26:49.051297+0000","last_clean":"2024-04-14T15:26:49.051297+0000","last_became_active":"2024-04-14T15:09:47.050962+0000","last_became_peered":"2024-04-14T15:09:47.050962+0000","last_unstale":"2024-04-14T15:26:49.051297+0000","last_undegraded":"2024-04-14T15:26:49.051297+0000","last_fullsized":"2024-04-14T15:26:49.051297+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:24:27.403735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:24:27.403735+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:50.167706+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":478,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-04-14T15:26:48.046595+0000","last_change":"2024-04-14T15:26:27.065961+0000","last_active":"2024-04-14T15:26:48.046595+0000","last_peered":"2024-04-14T15:26:48.046595+0000","last_clean":"2024-04-14T15:26:48.046595+0000","last_became_active":"2024-04-14T15:26:27.065649+0000","last_became_peered":"2024-04-14T15:26:27.065649+0000","last_unstale":"2024-04-14T15:26:48.046595+0000","last_undegraded":"2024-04-14T15:26:48.046595+0000","last_fullsized":"2024-04-14T15:26:48.046595+0000","mapping_epoch":828,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":829,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:27.318330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:24:27.318330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:51.529124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"91'130","reported_seq":1791,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.045030+0000","last_change":"2024-04-14T15:25:02.472703+0000","last_active":"2024-04-14T15:26:49.045030+0000","last_peered":"2024-04-14T15:26:49.045030+0000","last_clean":"2024-04-14T15:26:49.045030+0000","last_became_active":"2024-04-14T15:25:02.018082+0000","last_became_peered":"2024-04-14T15:25:02.018082+0000","last_unstale":"2024-04-14T15:26:49.045030+0000","last_undegraded":"2024-04-14T15:26:49.045030+0000","last_fullsized":"2024-04-14T15:26:49.045030+0000","mapping_epoch":757,"log_start":"91'130","ondisk_log_start":"91'130","created":752,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:25:02.472613+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:25:02.472613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:51.082368+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"233'8","reported_seq":938,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.050683+0000","last_change":"2024-04-14T15:26:32.864494+0000","last_active":"2024-04-14T15:26:49.050683+0000","last_peered":"2024-04-14T15:26:49.050683+0000","last_clean":"2024-04-14T15:26:49.050683+0000","last_became_active":"2024-04-14T15:20:52.995129+0000","last_became_peered":"2024-04-14T15:20:52.995129+0000","last_unstale":"2024-04-14T15:26:49.050683+0000","last_undegraded":"2024-04-14T15:26:49.050683+0000","last_fullsized":"2024-04-14T15:26:49.050683+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'8","last_scrub_stamp":"2024-04-14T15:26:32.864393+0000","last_deep_scrub":"233'8","last_deep_scrub_stamp":"2024-04-14T15:15:01.295800+0000","last_clean_scrub_stamp":"2024-04-14T15:26:32.864393+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:53.845756+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":5,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"479'24","reported_seq":1670,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.051072+0000","last_change":"2024-04-14T15:25:20.326455+0000","last_active":"2024-04-14T15:26:49.051072+0000","last_peered":"2024-04-14T15:26:49.051072+0000","last_clean":"2024-04-14T15:26:49.051072+0000","last_became_active":"2024-04-14T15:20:52.998853+0000","last_became_peered":"2024-04-14T15:20:52.998853+0000","last_unstale":"2024-04-14T15:26:49.051072+0000","last_undegraded":"2024-04-14T15:26:49.051072+0000","last_fullsized":"2024-04-14T15:26:49.051072+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'24","last_scrub_stamp":"2024-04-14T15:25:20.326301+0000","last_deep_scrub":"85'16","last_deep_scrub_stamp":"2024-04-14T15:10:47.456169+0000","last_clean_scrub_stamp":"2024-04-14T15:25:20.326301+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:50.168110+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":1621,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.042156+0000","last_change":"2024-04-14T15:23:36.489554+0000","last_active":"2024-04-14T15:26:49.042156+0000","last_peered":"2024-04-14T15:26:49.042156+0000","last_clean":"2024-04-14T15:26:49.042156+0000","last_became_active":"2024-04-14T15:09:43.018601+0000","last_became_peered":"2024-04-14T15:09:43.018601+0000","last_unstale":"2024-04-14T15:26:49.042156+0000","last_undegraded":"2024-04-14T15:26:49.042156+0000","last_fullsized":"2024-04-14T15:26:49.042156+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:36.489475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:23:36.489475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:51.998431+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":569,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-04-14T15:26:48.048217+0000","last_change":"2024-04-14T15:26:27.065115+0000","last_active":"2024-04-14T15:26:48.048217+0000","last_peered":"2024-04-14T15:26:48.048217+0000","last_clean":"2024-04-14T15:26:48.048217+0000","last_became_active":"2024-04-14T15:26:27.063970+0000","last_became_peered":"2024-04-14T15:26:27.063970+0000","last_unstale":"2024-04-14T15:26:48.048217+0000","last_undegraded":"2024-04-14T15:26:48.048217+0000","last_fullsized":"2024-04-14T15:26:48.048217+0000","mapping_epoch":828,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":829,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:35.426204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:24:35.426204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:51.529232+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1286,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.051835+0000","last_change":"2024-04-14T15:24:32.995639+0000","last_active":"2024-04-14T15:26:49.051835+0000","last_peered":"2024-04-14T15:26:49.051835+0000","last_clean":"2024-04-14T15:26:49.051835+0000","last_became_active":"2024-04-14T15:20:52.999771+0000","last_became_peered":"2024-04-14T15:20:52.999771+0000","last_unstale":"2024-04-14T15:26:49.051835+0000","last_undegraded":"2024-04-14T15:26:49.051835+0000","last_fullsized":"2024-04-14T15:26:49.051835+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:32.995512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:57.214046+0000","last_clean_scrub_stamp":"2024-04-14T15:24:32.995512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:52.880257+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"91'130","reported_seq":1768,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.051737+0000","last_change":"2024-04-14T15:25:04.733894+0000","last_active":"2024-04-14T15:26:49.051737+0000","last_peered":"2024-04-14T15:26:49.051737+0000","last_clean":"2024-04-14T15:26:49.051737+0000","last_became_active":"2024-04-14T15:24:57.023208+0000","last_became_peered":"2024-04-14T15:24:57.023208+0000","last_unstale":"2024-04-14T15:26:49.051737+0000","last_undegraded":"2024-04-14T15:26:49.051737+0000","last_fullsized":"2024-04-14T15:26:49.051737+0000","mapping_epoch":189,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":753,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:25:04.733792+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:25:04.733792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":60,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:52.880523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":1517,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.050991+0000","last_change":"2024-04-14T15:24:37.496873+0000","last_active":"2024-04-14T15:26:49.050991+0000","last_peered":"2024-04-14T15:26:49.050991+0000","last_clean":"2024-04-14T15:26:49.050991+0000","last_became_active":"2024-04-14T15:12:56.427276+0000","last_became_peered":"2024-04-14T15:12:56.427276+0000","last_unstale":"2024-04-14T15:26:49.050991+0000","last_undegraded":"2024-04-14T15:26:49.050991+0000","last_fullsized":"2024-04-14T15:26:49.050991+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:37.496778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:24:37.496778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:50.167952+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"255'10","reported_seq":1467,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.050726+0000","last_change":"2024-04-14T15:25:48.705124+0000","last_active":"2024-04-14T15:26:49.050726+0000","last_peered":"2024-04-14T15:26:49.050726+0000","last_clean":"2024-04-14T15:26:49.050726+0000","last_became_active":"2024-04-14T15:15:21.873747+0000","last_became_peered":"2024-04-14T15:15:21.873747+0000","last_unstale":"2024-04-14T15:26:49.050726+0000","last_undegraded":"2024-04-14T15:26:49.050726+0000","last_fullsized":"2024-04-14T15:26:49.050726+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'10","last_scrub_stamp":"2024-04-14T15:25:48.705041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:25:48.705041+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:50.660926+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"479'28","reported_seq":1681,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.048996+0000","last_change":"2024-04-14T15:25:47.281406+0000","last_active":"2024-04-14T15:26:49.048996+0000","last_peered":"2024-04-14T15:26:49.048996+0000","last_clean":"2024-04-14T15:26:49.048996+0000","last_became_active":"2024-04-14T15:20:52.989635+0000","last_became_peered":"2024-04-14T15:20:52.989635+0000","last_unstale":"2024-04-14T15:26:49.048996+0000","last_undegraded":"2024-04-14T15:26:49.048996+0000","last_fullsized":"2024-04-14T15:26:49.048996+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'28","last_scrub_stamp":"2024-04-14T15:25:47.281304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:25:47.281304+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:02.110971+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":661,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.045408+0000","last_change":"2024-04-14T15:25:46.849300+0000","last_active":"2024-04-14T15:26:49.045408+0000","last_peered":"2024-04-14T15:26:49.045408+0000","last_clean":"2024-04-14T15:26:49.045408+0000","last_became_active":"2024-04-14T15:20:52.999379+0000","last_became_peered":"2024-04-14T15:20:52.999379+0000","last_unstale":"2024-04-14T15:26:49.045408+0000","last_undegraded":"2024-04-14T15:26:49.045408+0000","last_fullsized":"2024-04-14T15:26:49.045408+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:46.849220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:25:46.849220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:58.079083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1608,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.043404+0000","last_change":"2024-04-14T15:23:28.571478+0000","last_active":"2024-04-14T15:26:49.043404+0000","last_peered":"2024-04-14T15:26:49.043404+0000","last_clean":"2024-04-14T15:26:49.043404+0000","last_became_active":"2024-04-14T15:20:52.991388+0000","last_became_peered":"2024-04-14T15:20:52.991388+0000","last_unstale":"2024-04-14T15:26:49.043404+0000","last_undegraded":"2024-04-14T15:26:49.043404+0000","last_fullsized":"2024-04-14T15:26:49.043404+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:28.571374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:23:28.571374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:51.998392+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"91'130","reported_seq":1777,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.051341+0000","last_change":"2024-04-14T15:25:04.328620+0000","last_active":"2024-04-14T15:26:49.051341+0000","last_peered":"2024-04-14T15:26:49.051341+0000","last_clean":"2024-04-14T15:26:49.051341+0000","last_became_active":"2024-04-14T15:24:57.015409+0000","last_became_peered":"2024-04-14T15:24:57.015409+0000","last_unstale":"2024-04-14T15:26:49.051341+0000","last_undegraded":"2024-04-14T15:26:49.051341+0000","last_fullsized":"2024-04-14T15:26:49.051341+0000","mapping_epoch":530,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":753,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:25:04.328526+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:25:04.328526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:50.168170+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":1518,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.045160+0000","last_change":"2024-04-14T15:24:27.475544+0000","last_active":"2024-04-14T15:26:49.045160+0000","last_peered":"2024-04-14T15:26:49.045160+0000","last_clean":"2024-04-14T15:26:49.045160+0000","last_became_active":"2024-04-14T15:12:56.355254+0000","last_became_peered":"2024-04-14T15:12:56.355254+0000","last_unstale":"2024-04-14T15:26:49.045160+0000","last_undegraded":"2024-04-14T15:26:49.045160+0000","last_fullsized":"2024-04-14T15:26:49.045160+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:27.475460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:24:27.475460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:51.998470+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1613,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.049198+0000","last_change":"2024-04-14T15:25:52.349307+0000","last_active":"2024-04-14T15:26:49.049198+0000","last_peered":"2024-04-14T15:26:49.049198+0000","last_clean":"2024-04-14T15:26:49.049198+0000","last_became_active":"2024-04-14T15:09:47.044610+0000","last_became_peered":"2024-04-14T15:09:47.044610+0000","last_unstale":"2024-04-14T15:26:49.049198+0000","last_undegraded":"2024-04-14T15:26:49.049198+0000","last_fullsized":"2024-04-14T15:26:49.049198+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:52.349217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:39.394655+0000","last_clean_scrub_stamp":"2024-04-14T15:25:52.349217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:02.647785+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"783'38","reported_seq":1355,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.050627+0000","last_change":"2024-04-14T15:25:27.804694+0000","last_active":"2024-04-14T15:26:49.050627+0000","last_peered":"2024-04-14T15:26:49.050627+0000","last_clean":"2024-04-14T15:26:49.050627+0000","last_became_active":"2024-04-14T15:15:21.880565+0000","last_became_peered":"2024-04-14T15:15:21.880565+0000","last_unstale":"2024-04-14T15:26:49.050627+0000","last_undegraded":"2024-04-14T15:26:49.050627+0000","last_fullsized":"2024-04-14T15:26:49.050627+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"783'38","last_scrub_stamp":"2024-04-14T15:25:27.804604+0000","last_deep_scrub":"783'38","last_deep_scrub_stamp":"2024-04-14T15:25:27.804604+0000","last_clean_scrub_stamp":"2024-04-14T15:25:27.804604+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:49.748235+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'2","reported_seq":1630,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.047542+0000","last_change":"2024-04-14T15:25:05.498299+0000","last_active":"2024-04-14T15:26:49.047542+0000","last_peered":"2024-04-14T15:26:49.047542+0000","last_clean":"2024-04-14T15:26:49.047542+0000","last_became_active":"2024-04-14T15:20:52.987576+0000","last_became_peered":"2024-04-14T15:20:52.987576+0000","last_unstale":"2024-04-14T15:26:49.047542+0000","last_undegraded":"2024-04-14T15:26:49.047542+0000","last_fullsized":"2024-04-14T15:26:49.047542+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'2","last_scrub_stamp":"2024-04-14T15:25:05.498215+0000","last_deep_scrub":"91'2","last_deep_scrub_stamp":"2024-04-14T15:17:48.850069+0000","last_clean_scrub_stamp":"2024-04-14T15:25:05.498215+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:51.082454+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1609,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.050452+0000","last_change":"2024-04-14T15:26:33.210864+0000","last_active":"2024-04-14T15:26:49.050452+0000","last_peered":"2024-04-14T15:26:49.050452+0000","last_clean":"2024-04-14T15:26:49.050452+0000","last_became_active":"2024-04-14T15:09:49.065684+0000","last_became_peered":"2024-04-14T15:09:49.065684+0000","last_unstale":"2024-04-14T15:26:49.050452+0000","last_undegraded":"2024-04-14T15:26:49.050452+0000","last_fullsized":"2024-04-14T15:26:49.050452+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:33.210769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:26:33.210769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:58.593801+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"91'130","reported_seq":1046,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.064101+0000","last_change":"2024-04-14T15:25:04.746467+0000","last_active":"2024-04-14T15:26:49.064101+0000","last_peered":"2024-04-14T15:26:49.064101+0000","last_clean":"2024-04-14T15:26:49.064101+0000","last_became_active":"2024-04-14T15:24:57.025580+0000","last_became_peered":"2024-04-14T15:24:57.025580+0000","last_unstale":"2024-04-14T15:26:49.064101+0000","last_undegraded":"2024-04-14T15:26:49.064101+0000","last_fullsized":"2024-04-14T15:26:49.064101+0000","mapping_epoch":530,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":753,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:25:04.746393+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:25:04.746393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:52.918513+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":1531,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.046836+0000","last_change":"2024-04-14T15:26:01.161438+0000","last_active":"2024-04-14T15:26:49.046836+0000","last_peered":"2024-04-14T15:26:49.046836+0000","last_clean":"2024-04-14T15:26:49.046836+0000","last_became_active":"2024-04-14T15:12:56.402023+0000","last_became_peered":"2024-04-14T15:12:56.402023+0000","last_unstale":"2024-04-14T15:26:49.046836+0000","last_undegraded":"2024-04-14T15:26:49.046836+0000","last_fullsized":"2024-04-14T15:26:49.046836+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.161362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:35.003968+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.161362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:23.708036+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":666,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.053058+0000","last_change":"2024-04-14T15:26:43.898914+0000","last_active":"2024-04-14T15:26:49.053058+0000","last_peered":"2024-04-14T15:26:49.053058+0000","last_clean":"2024-04-14T15:26:49.053058+0000","last_became_active":"2024-04-14T15:20:52.987883+0000","last_became_peered":"2024-04-14T15:20:52.987883+0000","last_unstale":"2024-04-14T15:26:49.053058+0000","last_undegraded":"2024-04-14T15:26:49.053058+0000","last_fullsized":"2024-04-14T15:26:49.053058+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:43.898835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:43.898835+0000","last_clean_scrub_stamp":"2024-04-14T15:26:43.898835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:28:08.893669+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"479'37","reported_seq":1683,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.055629+0000","last_change":"2024-04-14T15:25:58.107928+0000","last_active":"2024-04-14T15:26:49.055629+0000","last_peered":"2024-04-14T15:26:49.055629+0000","last_clean":"2024-04-14T15:26:49.055629+0000","last_became_active":"2024-04-14T15:09:45.179182+0000","last_became_peered":"2024-04-14T15:09:45.179182+0000","last_unstale":"2024-04-14T15:26:49.055629+0000","last_undegraded":"2024-04-14T15:26:49.055629+0000","last_fullsized":"2024-04-14T15:26:49.055629+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'37","last_scrub_stamp":"2024-04-14T15:25:58.107835+0000","last_deep_scrub":"102'27","last_deep_scrub_stamp":"2024-04-14T15:17:46.924330+0000","last_clean_scrub_stamp":"2024-04-14T15:25:58.107835+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:03.487475+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1622,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.043894+0000","last_change":"2024-04-14T15:25:55.977430+0000","last_active":"2024-04-14T15:26:49.043894+0000","last_peered":"2024-04-14T15:26:49.043894+0000","last_clean":"2024-04-14T15:26:49.043894+0000","last_became_active":"2024-04-14T15:09:43.021006+0000","last_became_peered":"2024-04-14T15:09:43.021006+0000","last_unstale":"2024-04-14T15:26:49.043894+0000","last_undegraded":"2024-04-14T15:26:49.043894+0000","last_fullsized":"2024-04-14T15:26:49.043894+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:55.977352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:25:55.977352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:56.535906+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1621,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.043850+0000","last_change":"2024-04-14T15:26:40.250025+0000","last_active":"2024-04-14T15:26:49.043850+0000","last_peered":"2024-04-14T15:26:49.043850+0000","last_clean":"2024-04-14T15:26:49.043850+0000","last_became_active":"2024-04-14T15:20:52.978765+0000","last_became_peered":"2024-04-14T15:20:52.978765+0000","last_unstale":"2024-04-14T15:26:49.043850+0000","last_undegraded":"2024-04-14T15:26:49.043850+0000","last_fullsized":"2024-04-14T15:26:49.043850+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:40.249923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:56.206649+0000","last_clean_scrub_stamp":"2024-04-14T15:26:40.249923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:28:09.119962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"91'130","reported_seq":1778,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.045366+0000","last_change":"2024-04-14T15:25:03.514185+0000","last_active":"2024-04-14T15:26:49.045366+0000","last_peered":"2024-04-14T15:26:49.045366+0000","last_clean":"2024-04-14T15:26:49.045366+0000","last_became_active":"2024-04-14T15:24:57.012410+0000","last_became_peered":"2024-04-14T15:24:57.012410+0000","last_unstale":"2024-04-14T15:26:49.045366+0000","last_undegraded":"2024-04-14T15:26:49.045366+0000","last_fullsized":"2024-04-14T15:26:49.045366+0000","mapping_epoch":530,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":753,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:25:03.514098+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:25:03.514098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:51.082489+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":1234,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.051034+0000","last_change":"2024-04-14T15:25:53.791322+0000","last_active":"2024-04-14T15:26:49.051034+0000","last_peered":"2024-04-14T15:26:49.051034+0000","last_clean":"2024-04-14T15:26:49.051034+0000","last_became_active":"2024-04-14T15:20:52.999776+0000","last_became_peered":"2024-04-14T15:20:52.999776+0000","last_unstale":"2024-04-14T15:26:49.051034+0000","last_undegraded":"2024-04-14T15:26:49.051034+0000","last_fullsized":"2024-04-14T15:26:49.051034+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:53.791183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:25:53.791183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:19.819673+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":490,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-04-14T15:26:48.048321+0000","last_change":"2024-04-14T15:26:27.071513+0000","last_active":"2024-04-14T15:26:48.048321+0000","last_peered":"2024-04-14T15:26:48.048321+0000","last_clean":"2024-04-14T15:26:48.048321+0000","last_became_active":"2024-04-14T15:26:27.069614+0000","last_became_peered":"2024-04-14T15:26:27.069614+0000","last_unstale":"2024-04-14T15:26:48.048321+0000","last_undegraded":"2024-04-14T15:26:48.048321+0000","last_fullsized":"2024-04-14T15:26:48.048321+0000","mapping_epoch":828,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":829,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:14.592662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:25:14.592662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:51.529308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"479'30","reported_seq":1665,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.064298+0000","last_change":"2024-04-14T15:26:31.018347+0000","last_active":"2024-04-14T15:26:49.064298+0000","last_peered":"2024-04-14T15:26:49.064298+0000","last_clean":"2024-04-14T15:26:49.064298+0000","last_became_active":"2024-04-14T15:09:45.037231+0000","last_became_peered":"2024-04-14T15:09:45.037231+0000","last_unstale":"2024-04-14T15:26:49.064298+0000","last_undegraded":"2024-04-14T15:26:49.064298+0000","last_fullsized":"2024-04-14T15:26:49.064298+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'30","last_scrub_stamp":"2024-04-14T15:26:31.018262+0000","last_deep_scrub":"182'18","last_deep_scrub_stamp":"2024-04-14T15:14:10.383906+0000","last_clean_scrub_stamp":"2024-04-14T15:26:31.018262+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:59.836282+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1622,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.042307+0000","last_change":"2024-04-14T15:24:29.563849+0000","last_active":"2024-04-14T15:26:49.042307+0000","last_peered":"2024-04-14T15:26:49.042307+0000","last_clean":"2024-04-14T15:26:49.042307+0000","last_became_active":"2024-04-14T15:09:43.025095+0000","last_became_peered":"2024-04-14T15:09:43.025095+0000","last_unstale":"2024-04-14T15:26:49.042307+0000","last_undegraded":"2024-04-14T15:26:49.042307+0000","last_fullsized":"2024-04-14T15:26:49.042307+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:29.563759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:29.563759+0000","last_clean_scrub_stamp":"2024-04-14T15:24:29.563759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:51.998550+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1635,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.056100+0000","last_change":"2024-04-14T15:26:32.972348+0000","last_active":"2024-04-14T15:26:49.056100+0000","last_peered":"2024-04-14T15:26:49.056100+0000","last_clean":"2024-04-14T15:26:49.056100+0000","last_became_active":"2024-04-14T15:09:43.018509+0000","last_became_peered":"2024-04-14T15:09:43.018509+0000","last_unstale":"2024-04-14T15:26:49.056100+0000","last_undegraded":"2024-04-14T15:26:49.056100+0000","last_fullsized":"2024-04-14T15:26:49.056100+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:32.972269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:26:32.972269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:38.493476+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0018592680000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"355'2","reported_seq":1617,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.046999+0000","last_change":"2024-04-14T15:26:35.340549+0000","last_active":"2024-04-14T15:26:49.046999+0000","last_peered":"2024-04-14T15:26:49.046999+0000","last_clean":"2024-04-14T15:26:49.046999+0000","last_became_active":"2024-04-14T15:09:49.065243+0000","last_became_peered":"2024-04-14T15:09:49.065243+0000","last_unstale":"2024-04-14T15:26:49.046999+0000","last_undegraded":"2024-04-14T15:26:49.046999+0000","last_fullsized":"2024-04-14T15:26:49.046999+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'2","last_scrub_stamp":"2024-04-14T15:26:35.340465+0000","last_deep_scrub":"355'2","last_deep_scrub_stamp":"2024-04-14T15:20:04.842881+0000","last_clean_scrub_stamp":"2024-04-14T15:26:35.340465+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:28:05.155467+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":1226,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.057680+0000","last_change":"2024-04-14T15:26:34.870663+0000","last_active":"2024-04-14T15:26:49.057680+0000","last_peered":"2024-04-14T15:26:49.057680+0000","last_clean":"2024-04-14T15:26:49.057680+0000","last_became_active":"2024-04-14T15:20:53.000058+0000","last_became_peered":"2024-04-14T15:20:53.000058+0000","last_unstale":"2024-04-14T15:26:49.057680+0000","last_undegraded":"2024-04-14T15:26:49.057680+0000","last_fullsized":"2024-04-14T15:26:49.057680+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:34.870583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:26:34.870583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:53.145119+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1614,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.043861+0000","last_change":"2024-04-14T15:26:00.992934+0000","last_active":"2024-04-14T15:26:49.043861+0000","last_peered":"2024-04-14T15:26:49.043861+0000","last_clean":"2024-04-14T15:26:49.043861+0000","last_became_active":"2024-04-14T15:09:47.047201+0000","last_became_peered":"2024-04-14T15:09:47.047201+0000","last_unstale":"2024-04-14T15:26:49.043861+0000","last_undegraded":"2024-04-14T15:26:49.043861+0000","last_fullsized":"2024-04-14T15:26:49.043861+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:00.992853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:26:00.992853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:05.677476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"528'37","reported_seq":1788,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.051875+0000","last_change":"2024-04-14T15:26:00.375445+0000","last_active":"2024-04-14T15:26:49.051875+0000","last_peered":"2024-04-14T15:26:49.051875+0000","last_clean":"2024-04-14T15:26:49.051875+0000","last_became_active":"2024-04-14T15:20:53.002282+0000","last_became_peered":"2024-04-14T15:20:53.002282+0000","last_unstale":"2024-04-14T15:26:49.051875+0000","last_undegraded":"2024-04-14T15:26:49.051875+0000","last_fullsized":"2024-04-14T15:26:49.051875+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"528'37","last_scrub_stamp":"2024-04-14T15:26:00.375331+0000","last_deep_scrub":"90'24","last_deep_scrub_stamp":"2024-04-14T15:11:25.120692+0000","last_clean_scrub_stamp":"2024-04-14T15:26:00.375331+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:08.881036+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"91'130","reported_seq":1974,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.042209+0000","last_change":"2024-04-14T15:25:04.953500+0000","last_active":"2024-04-14T15:26:49.042209+0000","last_peered":"2024-04-14T15:26:49.042209+0000","last_clean":"2024-04-14T15:26:49.042209+0000","last_became_active":"2024-04-14T15:24:57.020227+0000","last_became_peered":"2024-04-14T15:24:57.020227+0000","last_unstale":"2024-04-14T15:26:49.042209+0000","last_undegraded":"2024-04-14T15:26:49.042209+0000","last_fullsized":"2024-04-14T15:26:49.042209+0000","mapping_epoch":530,"log_start":"91'130","ondisk_log_start":"91'130","created":19,"last_epoch_clean":753,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:25:04.953418+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:25:04.953418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:51.998664+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":498,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-04-14T15:26:48.047679+0000","last_change":"2024-04-14T15:26:27.064479+0000","last_active":"2024-04-14T15:26:48.047679+0000","last_peered":"2024-04-14T15:26:48.047679+0000","last_clean":"2024-04-14T15:26:48.047679+0000","last_became_active":"2024-04-14T15:26:27.063604+0000","last_became_peered":"2024-04-14T15:26:27.063604+0000","last_unstale":"2024-04-14T15:26:48.047679+0000","last_undegraded":"2024-04-14T15:26:48.047679+0000","last_fullsized":"2024-04-14T15:26:48.047679+0000","mapping_epoch":828,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":829,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:48.704559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:47.627305+0000","last_clean_scrub_stamp":"2024-04-14T15:25:48.704559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:49.200443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"813'3","reported_seq":1321,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.057778+0000","last_change":"2024-04-14T15:26:03.783407+0000","last_active":"2024-04-14T15:26:49.057778+0000","last_peered":"2024-04-14T15:26:49.057778+0000","last_clean":"2024-04-14T15:26:49.057778+0000","last_became_active":"2024-04-14T15:20:53.002545+0000","last_became_peered":"2024-04-14T15:20:53.002545+0000","last_unstale":"2024-04-14T15:26:49.057778+0000","last_undegraded":"2024-04-14T15:26:49.057778+0000","last_fullsized":"2024-04-14T15:26:49.057778+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"813'3","last_scrub_stamp":"2024-04-14T15:26:03.783312+0000","last_deep_scrub":"813'3","last_deep_scrub_stamp":"2024-04-14T15:26:03.783312+0000","last_clean_scrub_stamp":"2024-04-14T15:26:03.783312+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:28.963379+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0058046169999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":486,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-04-14T15:26:48.048438+0000","last_change":"2024-04-14T15:26:39.546337+0000","last_active":"2024-04-14T15:26:48.048438+0000","last_peered":"2024-04-14T15:26:48.048438+0000","last_clean":"2024-04-14T15:26:48.048438+0000","last_became_active":"2024-04-14T15:26:27.070066+0000","last_became_peered":"2024-04-14T15:26:27.070066+0000","last_unstale":"2024-04-14T15:26:48.048438+0000","last_undegraded":"2024-04-14T15:26:48.048438+0000","last_fullsized":"2024-04-14T15:26:48.048438+0000","mapping_epoch":828,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":829,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:39.546232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:26:39.546232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:49.444447+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"91'130","reported_seq":1975,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.042518+0000","last_change":"2024-04-14T15:25:03.960485+0000","last_active":"2024-04-14T15:26:49.042518+0000","last_peered":"2024-04-14T15:26:49.042518+0000","last_clean":"2024-04-14T15:26:49.042518+0000","last_became_active":"2024-04-14T15:24:57.011910+0000","last_became_peered":"2024-04-14T15:24:57.011910+0000","last_unstale":"2024-04-14T15:26:49.042518+0000","last_undegraded":"2024-04-14T15:26:49.042518+0000","last_fullsized":"2024-04-14T15:26:49.042518+0000","mapping_epoch":189,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":753,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:25:03.960401+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:22:17.518666+0000","last_clean_scrub_stamp":"2024-04-14T15:25:03.960401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:51.998607+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":1226,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.050464+0000","last_change":"2024-04-14T15:25:18.772143+0000","last_active":"2024-04-14T15:26:49.050464+0000","last_peered":"2024-04-14T15:26:49.050464+0000","last_clean":"2024-04-14T15:26:49.050464+0000","last_became_active":"2024-04-14T15:20:52.986308+0000","last_became_peered":"2024-04-14T15:20:52.986308+0000","last_unstale":"2024-04-14T15:26:49.050464+0000","last_undegraded":"2024-04-14T15:26:49.050464+0000","last_fullsized":"2024-04-14T15:26:49.050464+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:18.772063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:25:18.772063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:52.880741+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1619,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.044954+0000","last_change":"2024-04-14T15:25:22.763238+0000","last_active":"2024-04-14T15:26:49.044954+0000","last_peered":"2024-04-14T15:26:49.044954+0000","last_clean":"2024-04-14T15:26:49.044954+0000","last_became_active":"2024-04-14T15:09:47.049457+0000","last_became_peered":"2024-04-14T15:09:47.049457+0000","last_unstale":"2024-04-14T15:26:49.044954+0000","last_undegraded":"2024-04-14T15:26:49.044954+0000","last_fullsized":"2024-04-14T15:26:49.044954+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:22.763147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:25:22.763147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"479'22","reported_seq":1660,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.049024+0000","last_change":"2024-04-14T15:25:56.109765+0000","last_active":"2024-04-14T15:26:49.049024+0000","last_peered":"2024-04-14T15:26:49.049024+0000","last_clean":"2024-04-14T15:26:49.049024+0000","last_became_active":"2024-04-14T15:09:45.033883+0000","last_became_peered":"2024-04-14T15:09:45.033883+0000","last_unstale":"2024-04-14T15:26:49.049024+0000","last_undegraded":"2024-04-14T15:26:49.049024+0000","last_fullsized":"2024-04-14T15:26:49.049024+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'22","last_scrub_stamp":"2024-04-14T15:25:56.109673+0000","last_deep_scrub":"308'18","last_deep_scrub_stamp":"2024-04-14T15:16:16.054032+0000","last_clean_scrub_stamp":"2024-04-14T15:25:56.109673+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:12.349554+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1636,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.052920+0000","last_change":"2024-04-14T15:26:35.429963+0000","last_active":"2024-04-14T15:26:49.052920+0000","last_peered":"2024-04-14T15:26:49.052920+0000","last_clean":"2024-04-14T15:26:49.052920+0000","last_became_active":"2024-04-14T15:09:43.021188+0000","last_became_peered":"2024-04-14T15:09:43.021188+0000","last_unstale":"2024-04-14T15:26:49.052920+0000","last_undegraded":"2024-04-14T15:26:49.052920+0000","last_fullsized":"2024-04-14T15:26:49.052920+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:35.429886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:26:35.429886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:28:01.367644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"73'5","reported_seq":2665,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.043743+0000","last_change":"2024-04-14T15:25:26.770287+0000","last_active":"2024-04-14T15:26:49.043743+0000","last_peered":"2024-04-14T15:26:49.043743+0000","last_clean":"2024-04-14T15:26:49.043743+0000","last_became_active":"2024-04-14T15:20:52.975473+0000","last_became_peered":"2024-04-14T15:20:52.975473+0000","last_unstale":"2024-04-14T15:26:49.043743+0000","last_undegraded":"2024-04-14T15:26:49.043743+0000","last_fullsized":"2024-04-14T15:26:49.043743+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-04-14T15:25:26.770199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:25:26.770199+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:51.082471+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00056542099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1017,"num_read_kb":1012,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1616,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.044031+0000","last_change":"2024-04-14T15:26:03.955505+0000","last_active":"2024-04-14T15:26:49.044031+0000","last_peered":"2024-04-14T15:26:49.044031+0000","last_clean":"2024-04-14T15:26:49.044031+0000","last_became_active":"2024-04-14T15:20:52.999464+0000","last_became_peered":"2024-04-14T15:20:52.999464+0000","last_unstale":"2024-04-14T15:26:49.044031+0000","last_undegraded":"2024-04-14T15:26:49.044031+0000","last_fullsized":"2024-04-14T15:26:49.044031+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:03.955419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:20.827589+0000","last_clean_scrub_stamp":"2024-04-14T15:26:03.955419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:19.378832+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"91'130","reported_seq":556,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-04-14T15:26:48.044154+0000","last_change":"2024-04-14T15:26:35.527281+0000","last_active":"2024-04-14T15:26:48.044154+0000","last_peered":"2024-04-14T15:26:48.044154+0000","last_clean":"2024-04-14T15:26:48.044154+0000","last_became_active":"2024-04-14T15:26:27.065655+0000","last_became_peered":"2024-04-14T15:26:27.065655+0000","last_unstale":"2024-04-14T15:26:48.044154+0000","last_undegraded":"2024-04-14T15:26:48.044154+0000","last_fullsized":"2024-04-14T15:26:48.044154+0000","mapping_epoch":828,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":829,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:26:35.527172+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:26:35.527172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:41.339593+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":1220,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.051196+0000","last_change":"2024-04-14T15:25:21.855268+0000","last_active":"2024-04-14T15:26:49.051196+0000","last_peered":"2024-04-14T15:26:49.051196+0000","last_clean":"2024-04-14T15:26:49.051196+0000","last_became_active":"2024-04-14T15:15:21.883717+0000","last_became_peered":"2024-04-14T15:15:21.883717+0000","last_unstale":"2024-04-14T15:26:49.051196+0000","last_undegraded":"2024-04-14T15:26:49.051196+0000","last_fullsized":"2024-04-14T15:26:49.051196+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:21.855183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:25:21.855183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:52.880712+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1620,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.063649+0000","last_change":"2024-04-14T15:26:42.913182+0000","last_active":"2024-04-14T15:26:49.063649+0000","last_peered":"2024-04-14T15:26:49.063649+0000","last_clean":"2024-04-14T15:26:49.063649+0000","last_became_active":"2024-04-14T15:09:47.040185+0000","last_became_peered":"2024-04-14T15:09:47.040185+0000","last_unstale":"2024-04-14T15:26:49.063649+0000","last_undegraded":"2024-04-14T15:26:49.063649+0000","last_fullsized":"2024-04-14T15:26:49.063649+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:42.913086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:53.731610+0000","last_clean_scrub_stamp":"2024-04-14T15:26:42.913086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:49.125054+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"479'40","reported_seq":1895,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.046735+0000","last_change":"2024-04-14T15:26:31.285594+0000","last_active":"2024-04-14T15:26:49.046735+0000","last_peered":"2024-04-14T15:26:49.046735+0000","last_clean":"2024-04-14T15:26:49.046735+0000","last_became_active":"2024-04-14T15:09:45.034168+0000","last_became_peered":"2024-04-14T15:09:45.034168+0000","last_unstale":"2024-04-14T15:26:49.046735+0000","last_undegraded":"2024-04-14T15:26:49.046735+0000","last_fullsized":"2024-04-14T15:26:49.046735+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'40","last_scrub_stamp":"2024-04-14T15:26:31.285457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:26:31.285457+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:36.977601+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":579,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-04-14T15:26:48.046136+0000","last_change":"2024-04-14T15:26:41.486404+0000","last_active":"2024-04-14T15:26:48.046136+0000","last_peered":"2024-04-14T15:26:48.046136+0000","last_clean":"2024-04-14T15:26:48.046136+0000","last_became_active":"2024-04-14T15:26:27.066084+0000","last_became_peered":"2024-04-14T15:26:27.066084+0000","last_unstale":"2024-04-14T15:26:48.046136+0000","last_undegraded":"2024-04-14T15:26:48.046136+0000","last_fullsized":"2024-04-14T15:26:48.046136+0000","mapping_epoch":828,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":829,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:41.486324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:24.066275+0000","last_clean_scrub_stamp":"2024-04-14T15:26:41.486324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:28:07.155674+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1272,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.051581+0000","last_change":"2024-04-14T15:24:30.949646+0000","last_active":"2024-04-14T15:26:49.051581+0000","last_peered":"2024-04-14T15:26:49.051581+0000","last_clean":"2024-04-14T15:26:49.051581+0000","last_became_active":"2024-04-14T15:15:21.874742+0000","last_became_peered":"2024-04-14T15:15:21.874742+0000","last_unstale":"2024-04-14T15:26:49.051581+0000","last_undegraded":"2024-04-14T15:26:49.051581+0000","last_fullsized":"2024-04-14T15:26:49.051581+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:30.949569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:30.949569+0000","last_clean_scrub_stamp":"2024-04-14T15:24:30.949569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:52.880429+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"91'130","reported_seq":1049,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.051518+0000","last_change":"2024-04-14T15:25:05.749046+0000","last_active":"2024-04-14T15:26:49.051518+0000","last_peered":"2024-04-14T15:26:49.051518+0000","last_clean":"2024-04-14T15:26:49.051518+0000","last_became_active":"2024-04-14T15:24:57.023069+0000","last_became_peered":"2024-04-14T15:24:57.023069+0000","last_unstale":"2024-04-14T15:26:49.051518+0000","last_undegraded":"2024-04-14T15:26:49.051518+0000","last_fullsized":"2024-04-14T15:26:49.051518+0000","mapping_epoch":530,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":753,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:25:05.748903+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:25:05.748903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:52.880589+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":1530,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.043037+0000","last_change":"2024-04-14T15:25:21.747973+0000","last_active":"2024-04-14T15:26:49.043037+0000","last_peered":"2024-04-14T15:26:49.043037+0000","last_clean":"2024-04-14T15:26:49.043037+0000","last_became_active":"2024-04-14T15:20:52.989272+0000","last_became_peered":"2024-04-14T15:20:52.989272+0000","last_unstale":"2024-04-14T15:26:49.043037+0000","last_undegraded":"2024-04-14T15:26:49.043037+0000","last_fullsized":"2024-04-14T15:26:49.043037+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:21.747886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:25:21.747886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"773'10","reported_seq":1845,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.053127+0000","last_change":"2024-04-14T15:25:17.080605+0000","last_active":"2024-04-14T15:26:49.053127+0000","last_peered":"2024-04-14T15:26:49.053127+0000","last_clean":"2024-04-14T15:26:49.053127+0000","last_became_active":"2024-04-14T15:20:53.000669+0000","last_became_peered":"2024-04-14T15:20:53.000669+0000","last_unstale":"2024-04-14T15:26:49.053127+0000","last_undegraded":"2024-04-14T15:26:49.053127+0000","last_fullsized":"2024-04-14T15:26:49.053127+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"773'10","last_scrub_stamp":"2024-04-14T15:25:17.080518+0000","last_deep_scrub":"773'10","last_deep_scrub_stamp":"2024-04-14T15:25:17.080518+0000","last_clean_scrub_stamp":"2024-04-14T15:25:17.080518+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:52.918476+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"479'20","reported_seq":629,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.044784+0000","last_change":"2024-04-14T15:25:19.646343+0000","last_active":"2024-04-14T15:26:49.044784+0000","last_peered":"2024-04-14T15:26:49.044784+0000","last_clean":"2024-04-14T15:26:49.044784+0000","last_became_active":"2024-04-14T15:21:16.034027+0000","last_became_peered":"2024-04-14T15:21:16.034027+0000","last_unstale":"2024-04-14T15:26:49.044784+0000","last_undegraded":"2024-04-14T15:26:49.044784+0000","last_fullsized":"2024-04-14T15:26:49.044784+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'20","last_scrub_stamp":"2024-04-14T15:25:19.646254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:25:19.646254+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:51.082438+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1619,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.046639+0000","last_change":"2024-04-14T15:25:09.533597+0000","last_active":"2024-04-14T15:26:49.046639+0000","last_peered":"2024-04-14T15:26:49.046639+0000","last_clean":"2024-04-14T15:26:49.046639+0000","last_became_active":"2024-04-14T15:09:43.022597+0000","last_became_peered":"2024-04-14T15:09:43.022597+0000","last_unstale":"2024-04-14T15:26:49.046639+0000","last_undegraded":"2024-04-14T15:26:49.046639+0000","last_fullsized":"2024-04-14T15:26:49.046639+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:09.533502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:21:28.684818+0000","last_clean_scrub_stamp":"2024-04-14T15:25:09.533502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:51.082420+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1616,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.052811+0000","last_change":"2024-04-14T15:26:41.906777+0000","last_active":"2024-04-14T15:26:49.052811+0000","last_peered":"2024-04-14T15:26:49.052811+0000","last_clean":"2024-04-14T15:26:49.052811+0000","last_became_active":"2024-04-14T15:09:49.066648+0000","last_became_peered":"2024-04-14T15:09:49.066648+0000","last_unstale":"2024-04-14T15:26:49.052811+0000","last_undegraded":"2024-04-14T15:26:49.052811+0000","last_fullsized":"2024-04-14T15:26:49.052811+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:41.906662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:26:41.906662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:59.810338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.c","version":"91'130","reported_seq":1789,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.051639+0000","last_change":"2024-04-14T15:25:50.734599+0000","last_active":"2024-04-14T15:26:49.051639+0000","last_peered":"2024-04-14T15:26:49.051639+0000","last_clean":"2024-04-14T15:26:49.051639+0000","last_became_active":"2024-04-14T15:25:32.025973+0000","last_became_peered":"2024-04-14T15:25:32.025973+0000","last_unstale":"2024-04-14T15:26:49.051639+0000","last_undegraded":"2024-04-14T15:26:49.051639+0000","last_fullsized":"2024-04-14T15:26:49.051639+0000","mapping_epoch":787,"log_start":"91'130","ondisk_log_start":"91'130","created":752,"last_epoch_clean":788,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:25:50.734496+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:25:50.734496+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":60,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:57.399176+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":1524,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.064162+0000","last_change":"2024-04-14T15:26:32.010370+0000","last_active":"2024-04-14T15:26:49.064162+0000","last_peered":"2024-04-14T15:26:49.064162+0000","last_clean":"2024-04-14T15:26:49.064162+0000","last_became_active":"2024-04-14T15:12:56.563074+0000","last_became_peered":"2024-04-14T15:12:56.563074+0000","last_unstale":"2024-04-14T15:26:49.064162+0000","last_undegraded":"2024-04-14T15:26:49.064162+0000","last_fullsized":"2024-04-14T15:26:49.064162+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:32.010285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:26:32.010285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:38.584332+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1619,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.045736+0000","last_change":"2024-04-14T15:25:57.664021+0000","last_active":"2024-04-14T15:26:49.045736+0000","last_peered":"2024-04-14T15:26:49.045736+0000","last_clean":"2024-04-14T15:26:49.045736+0000","last_became_active":"2024-04-14T15:09:47.049005+0000","last_became_peered":"2024-04-14T15:09:47.049005+0000","last_unstale":"2024-04-14T15:26:49.045736+0000","last_undegraded":"2024-04-14T15:26:49.045736+0000","last_fullsized":"2024-04-14T15:26:49.045736+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:57.663924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:56.915544+0000","last_clean_scrub_stamp":"2024-04-14T15:25:57.663924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:04.748057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"479'23","reported_seq":1653,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.049168+0000","last_change":"2024-04-14T15:24:28.394342+0000","last_active":"2024-04-14T15:26:49.049168+0000","last_peered":"2024-04-14T15:26:49.049168+0000","last_clean":"2024-04-14T15:26:49.049168+0000","last_became_active":"2024-04-14T15:09:45.032892+0000","last_became_peered":"2024-04-14T15:09:45.032892+0000","last_unstale":"2024-04-14T15:26:49.049168+0000","last_undegraded":"2024-04-14T15:26:49.049168+0000","last_fullsized":"2024-04-14T15:26:49.049168+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'23","last_scrub_stamp":"2024-04-14T15:24:28.394222+0000","last_deep_scrub":"183'17","last_deep_scrub_stamp":"2024-04-14T15:14:11.577574+0000","last_clean_scrub_stamp":"2024-04-14T15:24:28.394222+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:50.167926+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"677'6","reported_seq":1641,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.051457+0000","last_change":"2024-04-14T15:26:34.259298+0000","last_active":"2024-04-14T15:26:49.051457+0000","last_peered":"2024-04-14T15:26:49.051457+0000","last_clean":"2024-04-14T15:26:49.051457+0000","last_became_active":"2024-04-14T15:09:43.020490+0000","last_became_peered":"2024-04-14T15:09:43.020490+0000","last_unstale":"2024-04-14T15:26:49.051457+0000","last_undegraded":"2024-04-14T15:26:49.051457+0000","last_fullsized":"2024-04-14T15:26:49.051457+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"677'6","last_scrub_stamp":"2024-04-14T15:26:34.259182+0000","last_deep_scrub":"677'6","last_deep_scrub_stamp":"2024-04-14T15:23:38.773539+0000","last_clean_scrub_stamp":"2024-04-14T15:26:34.259182+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:41.327575+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":528,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1280,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.051606+0000","last_change":"2024-04-14T15:26:33.856187+0000","last_active":"2024-04-14T15:26:49.051606+0000","last_peered":"2024-04-14T15:26:49.051606+0000","last_clean":"2024-04-14T15:26:49.051606+0000","last_became_active":"2024-04-14T15:15:21.876410+0000","last_became_peered":"2024-04-14T15:15:21.876410+0000","last_unstale":"2024-04-14T15:26:49.051606+0000","last_undegraded":"2024-04-14T15:26:49.051606+0000","last_fullsized":"2024-04-14T15:26:49.051606+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:33.856104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:26:33.856104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:45.581528+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"91'130","reported_seq":1795,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.044872+0000","last_change":"2024-04-14T15:25:48.950790+0000","last_active":"2024-04-14T15:26:49.044872+0000","last_peered":"2024-04-14T15:26:49.044872+0000","last_clean":"2024-04-14T15:26:49.044872+0000","last_became_active":"2024-04-14T15:25:32.024118+0000","last_became_peered":"2024-04-14T15:25:32.024118+0000","last_unstale":"2024-04-14T15:26:49.044872+0000","last_undegraded":"2024-04-14T15:26:49.044872+0000","last_fullsized":"2024-04-14T15:26:49.044872+0000","mapping_epoch":787,"log_start":"91'130","ondisk_log_start":"91'130","created":752,"last_epoch_clean":788,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:25:48.950683+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:25:48.950683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:54.252465+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":1531,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.056042+0000","last_change":"2024-04-14T15:26:36.962660+0000","last_active":"2024-04-14T15:26:49.056042+0000","last_peered":"2024-04-14T15:26:49.056042+0000","last_clean":"2024-04-14T15:26:49.056042+0000","last_became_active":"2024-04-14T15:12:56.551177+0000","last_became_peered":"2024-04-14T15:12:56.551177+0000","last_unstale":"2024-04-14T15:26:49.056042+0000","last_undegraded":"2024-04-14T15:26:49.056042+0000","last_fullsized":"2024-04-14T15:26:49.056042+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:36.962558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:22:18.299330+0000","last_clean_scrub_stamp":"2024-04-14T15:26:36.962558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:46.139249+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"479'19","reported_seq":1644,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.064354+0000","last_change":"2024-04-14T15:24:33.412261+0000","last_active":"2024-04-14T15:26:49.064354+0000","last_peered":"2024-04-14T15:26:49.064354+0000","last_clean":"2024-04-14T15:26:49.064354+0000","last_became_active":"2024-04-14T15:09:45.180375+0000","last_became_peered":"2024-04-14T15:09:45.180375+0000","last_unstale":"2024-04-14T15:26:49.064354+0000","last_undegraded":"2024-04-14T15:26:49.064354+0000","last_fullsized":"2024-04-14T15:26:49.064354+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'19","last_scrub_stamp":"2024-04-14T15:24:33.412181+0000","last_deep_scrub":"347'13","last_deep_scrub_stamp":"2024-04-14T15:16:55.694599+0000","last_clean_scrub_stamp":"2024-04-14T15:24:33.412181+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:52.918441+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":1627,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.045651+0000","last_change":"2024-04-14T15:25:16.586933+0000","last_active":"2024-04-14T15:26:49.045651+0000","last_peered":"2024-04-14T15:26:49.045651+0000","last_clean":"2024-04-14T15:26:49.045651+0000","last_became_active":"2024-04-14T15:09:43.016788+0000","last_became_peered":"2024-04-14T15:09:43.016788+0000","last_unstale":"2024-04-14T15:26:49.045651+0000","last_undegraded":"2024-04-14T15:26:49.045651+0000","last_fullsized":"2024-04-14T15:26:49.045651+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:16.586848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:25:16.586848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1291,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.052002+0000","last_change":"2024-04-14T15:24:33.956401+0000","last_active":"2024-04-14T15:26:49.052002+0000","last_peered":"2024-04-14T15:26:49.052002+0000","last_clean":"2024-04-14T15:26:49.052002+0000","last_became_active":"2024-04-14T15:15:21.873580+0000","last_became_peered":"2024-04-14T15:15:21.873580+0000","last_unstale":"2024-04-14T15:26:49.052002+0000","last_undegraded":"2024-04-14T15:26:49.052002+0000","last_fullsized":"2024-04-14T15:26:49.052002+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:33.956328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:24:33.956328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:52.880402+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1623,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.043381+0000","last_change":"2024-04-14T15:26:39.410125+0000","last_active":"2024-04-14T15:26:49.043381+0000","last_peered":"2024-04-14T15:26:49.043381+0000","last_clean":"2024-04-14T15:26:49.043381+0000","last_became_active":"2024-04-14T15:09:49.064296+0000","last_became_peered":"2024-04-14T15:09:49.064296+0000","last_unstale":"2024-04-14T15:26:49.043381+0000","last_undegraded":"2024-04-14T15:26:49.043381+0000","last_fullsized":"2024-04-14T15:26:49.043381+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:39.410037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:39.410037+0000","last_clean_scrub_stamp":"2024-04-14T15:26:39.410037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:52.656103+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.12","version":"91'130","reported_seq":494,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-04-14T15:26:48.043879+0000","last_change":"2024-04-14T15:26:33.541319+0000","last_active":"2024-04-14T15:26:48.043879+0000","last_peered":"2024-04-14T15:26:48.043879+0000","last_clean":"2024-04-14T15:26:48.043879+0000","last_became_active":"2024-04-14T15:26:27.062496+0000","last_became_peered":"2024-04-14T15:26:27.062496+0000","last_unstale":"2024-04-14T15:26:48.043879+0000","last_undegraded":"2024-04-14T15:26:48.043879+0000","last_fullsized":"2024-04-14T15:26:48.043879+0000","mapping_epoch":828,"log_start":"91'130","ondisk_log_start":"91'130","created":752,"last_epoch_clean":829,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:26:33.541164+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:22:17.518666+0000","last_clean_scrub_stamp":"2024-04-14T15:26:33.541164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:52.763107+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":1529,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.045499+0000","last_change":"2024-04-14T15:25:52.667570+0000","last_active":"2024-04-14T15:26:49.045499+0000","last_peered":"2024-04-14T15:26:49.045499+0000","last_clean":"2024-04-14T15:26:49.045499+0000","last_became_active":"2024-04-14T15:20:52.977894+0000","last_became_peered":"2024-04-14T15:20:52.977894+0000","last_unstale":"2024-04-14T15:26:49.045499+0000","last_undegraded":"2024-04-14T15:26:49.045499+0000","last_fullsized":"2024-04-14T15:26:49.045499+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:52.667467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:25:52.667467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:19.080509+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.17","version":"479'12","reported_seq":1637,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.051971+0000","last_change":"2024-04-14T15:26:40.886595+0000","last_active":"2024-04-14T15:26:49.051971+0000","last_peered":"2024-04-14T15:26:49.051971+0000","last_clean":"2024-04-14T15:26:49.051971+0000","last_became_active":"2024-04-14T15:09:45.180037+0000","last_became_peered":"2024-04-14T15:09:45.180037+0000","last_unstale":"2024-04-14T15:26:49.051971+0000","last_undegraded":"2024-04-14T15:26:49.051971+0000","last_fullsized":"2024-04-14T15:26:49.051971+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'12","last_scrub_stamp":"2024-04-14T15:26:40.886512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:26:40.886512+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:28:00.157343+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":660,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.049104+0000","last_change":"2024-04-14T15:26:39.264107+0000","last_active":"2024-04-14T15:26:49.049104+0000","last_peered":"2024-04-14T15:26:49.049104+0000","last_clean":"2024-04-14T15:26:49.049104+0000","last_became_active":"2024-04-14T15:20:52.986974+0000","last_became_peered":"2024-04-14T15:20:52.986974+0000","last_unstale":"2024-04-14T15:26:49.049104+0000","last_undegraded":"2024-04-14T15:26:49.049104+0000","last_fullsized":"2024-04-14T15:26:49.049104+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:39.263995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:59.640565+0000","last_clean_scrub_stamp":"2024-04-14T15:26:39.263995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:28:01.730397+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":490,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-04-14T15:26:48.047555+0000","last_change":"2024-04-14T15:26:27.068405+0000","last_active":"2024-04-14T15:26:48.047555+0000","last_peered":"2024-04-14T15:26:48.047555+0000","last_clean":"2024-04-14T15:26:48.047555+0000","last_became_active":"2024-04-14T15:26:27.067715+0000","last_became_peered":"2024-04-14T15:26:27.067715+0000","last_unstale":"2024-04-14T15:26:48.047555+0000","last_undegraded":"2024-04-14T15:26:48.047555+0000","last_fullsized":"2024-04-14T15:26:48.047555+0000","mapping_epoch":828,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":829,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:18.710641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:13.735024+0000","last_clean_scrub_stamp":"2024-04-14T15:24:18.710641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:51.529106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1607,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.044184+0000","last_change":"2024-04-14T15:23:29.527743+0000","last_active":"2024-04-14T15:26:49.044184+0000","last_peered":"2024-04-14T15:26:49.044184+0000","last_clean":"2024-04-14T15:26:49.044184+0000","last_became_active":"2024-04-14T15:20:52.985136+0000","last_became_peered":"2024-04-14T15:20:52.985136+0000","last_unstale":"2024-04-14T15:26:49.044184+0000","last_undegraded":"2024-04-14T15:26:49.044184+0000","last_fullsized":"2024-04-14T15:26:49.044184+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:23:29.527656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:23:29.527656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:51.998413+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.11","version":"91'130","reported_seq":1606,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.053741+0000","last_change":"2024-04-14T15:25:02.713875+0000","last_active":"2024-04-14T15:26:49.053741+0000","last_peered":"2024-04-14T15:26:49.053741+0000","last_clean":"2024-04-14T15:26:49.053741+0000","last_became_active":"2024-04-14T15:25:02.030974+0000","last_became_peered":"2024-04-14T15:25:02.030974+0000","last_unstale":"2024-04-14T15:26:49.053741+0000","last_undegraded":"2024-04-14T15:26:49.053741+0000","last_fullsized":"2024-04-14T15:26:49.053741+0000","mapping_epoch":757,"log_start":"91'130","ondisk_log_start":"91'130","created":752,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:25:02.713791+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:25:02.713791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:52.918495+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":1524,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.049658+0000","last_change":"2024-04-14T15:25:50.325506+0000","last_active":"2024-04-14T15:26:49.049658+0000","last_peered":"2024-04-14T15:26:49.049658+0000","last_clean":"2024-04-14T15:26:49.049658+0000","last_became_active":"2024-04-14T15:12:56.203577+0000","last_became_peered":"2024-04-14T15:12:56.203577+0000","last_unstale":"2024-04-14T15:26:49.049658+0000","last_undegraded":"2024-04-14T15:26:49.049658+0000","last_fullsized":"2024-04-14T15:26:49.049658+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:50.325429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:46.369231+0000","last_clean_scrub_stamp":"2024-04-14T15:25:50.325429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:50.471569+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"479'20","reported_seq":1325,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.051870+0000","last_change":"2024-04-14T15:25:03.783837+0000","last_active":"2024-04-14T15:26:49.051870+0000","last_peered":"2024-04-14T15:26:49.051870+0000","last_clean":"2024-04-14T15:26:49.051870+0000","last_became_active":"2024-04-14T15:15:21.877667+0000","last_became_peered":"2024-04-14T15:15:21.877667+0000","last_unstale":"2024-04-14T15:26:49.051870+0000","last_undegraded":"2024-04-14T15:26:49.051870+0000","last_fullsized":"2024-04-14T15:26:49.051870+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'20","last_scrub_stamp":"2024-04-14T15:25:03.783721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:25:03.783721+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:52.880490+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":490,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-04-14T15:26:48.045923+0000","last_change":"2024-04-14T15:26:27.066432+0000","last_active":"2024-04-14T15:26:48.045923+0000","last_peered":"2024-04-14T15:26:48.045923+0000","last_clean":"2024-04-14T15:26:48.045923+0000","last_became_active":"2024-04-14T15:26:27.065734+0000","last_became_peered":"2024-04-14T15:26:27.065734+0000","last_unstale":"2024-04-14T15:26:48.045923+0000","last_undegraded":"2024-04-14T15:26:48.045923+0000","last_fullsized":"2024-04-14T15:26:48.045923+0000","mapping_epoch":828,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":829,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:57.772100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:25:57.772100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:57.968242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"829'10","reported_seq":576,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-04-14T15:26:48.046471+0000","last_change":"2024-04-14T15:26:27.066295+0000","last_active":"2024-04-14T15:26:48.046471+0000","last_peered":"2024-04-14T15:26:48.046471+0000","last_clean":"2024-04-14T15:26:48.046471+0000","last_became_active":"2024-04-14T15:26:27.065792+0000","last_became_peered":"2024-04-14T15:26:27.065792+0000","last_unstale":"2024-04-14T15:26:48.046471+0000","last_undegraded":"2024-04-14T15:26:48.046471+0000","last_fullsized":"2024-04-14T15:26:48.046471+0000","mapping_epoch":828,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":829,"parent":"0.0","parent_split_bits":0,"last_scrub":"280'9","last_scrub_stamp":"2024-04-14T15:25:13.639579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:25:13.639579+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:51.529290+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1607,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.064265+0000","last_change":"2024-04-14T15:25:54.151674+0000","last_active":"2024-04-14T15:26:49.064265+0000","last_peered":"2024-04-14T15:26:49.064265+0000","last_clean":"2024-04-14T15:26:49.064265+0000","last_became_active":"2024-04-14T15:09:49.066068+0000","last_became_peered":"2024-04-14T15:09:49.066068+0000","last_unstale":"2024-04-14T15:26:49.064265+0000","last_undegraded":"2024-04-14T15:26:49.064265+0000","last_fullsized":"2024-04-14T15:26:49.064265+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:54.151593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:54.724681+0000","last_clean_scrub_stamp":"2024-04-14T15:25:54.151593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:55.084660+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"91'130","reported_seq":1905,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-04-14T15:26:48.043876+0000","last_change":"2024-04-14T15:26:32.536312+0000","last_active":"2024-04-14T15:26:48.043876+0000","last_peered":"2024-04-14T15:26:48.043876+0000","last_clean":"2024-04-14T15:26:48.043876+0000","last_became_active":"2024-04-14T15:26:27.059151+0000","last_became_peered":"2024-04-14T15:26:27.059151+0000","last_unstale":"2024-04-14T15:26:48.043876+0000","last_undegraded":"2024-04-14T15:26:48.043876+0000","last_fullsized":"2024-04-14T15:26:48.043876+0000","mapping_epoch":828,"log_start":"91'130","ondisk_log_start":"91'130","created":752,"last_epoch_clean":829,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:26:32.536209+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:26:32.536209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:56.815886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":395,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-04-14T15:26:48.046031+0000","last_change":"2024-04-14T15:26:27.070131+0000","last_active":"2024-04-14T15:26:48.046031+0000","last_peered":"2024-04-14T15:26:48.046031+0000","last_clean":"2024-04-14T15:26:48.046031+0000","last_became_active":"2024-04-14T15:26:27.068033+0000","last_became_peered":"2024-04-14T15:26:27.068033+0000","last_unstale":"2024-04-14T15:26:48.046031+0000","last_undegraded":"2024-04-14T15:26:48.046031+0000","last_fullsized":"2024-04-14T15:26:48.046031+0000","mapping_epoch":828,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":829,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:26.878584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:24:26.878584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:51.529215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"479'19","reported_seq":751,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-04-14T15:26:48.046244+0000","last_change":"2024-04-14T15:26:27.063823+0000","last_active":"2024-04-14T15:26:48.046244+0000","last_peered":"2024-04-14T15:26:48.046244+0000","last_clean":"2024-04-14T15:26:48.046244+0000","last_became_active":"2024-04-14T15:26:27.062451+0000","last_became_peered":"2024-04-14T15:26:27.062451+0000","last_unstale":"2024-04-14T15:26:48.046244+0000","last_undegraded":"2024-04-14T15:26:48.046244+0000","last_fullsized":"2024-04-14T15:26:48.046244+0000","mapping_epoch":828,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":829,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'19","last_scrub_stamp":"2024-04-14T15:23:46.548255+0000","last_deep_scrub":"183'13","last_deep_scrub_stamp":"2024-04-14T15:14:11.755252+0000","last_clean_scrub_stamp":"2024-04-14T15:23:46.548255+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:51.529066+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1627,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.055533+0000","last_change":"2024-04-14T15:25:48.143955+0000","last_active":"2024-04-14T15:26:49.055533+0000","last_peered":"2024-04-14T15:26:49.055533+0000","last_clean":"2024-04-14T15:26:49.055533+0000","last_became_active":"2024-04-14T15:09:43.019301+0000","last_became_peered":"2024-04-14T15:09:43.019301+0000","last_unstale":"2024-04-14T15:26:49.055533+0000","last_undegraded":"2024-04-14T15:26:49.055533+0000","last_fullsized":"2024-04-14T15:26:49.055533+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:48.143875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:25:48.143875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:13.335549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"255'10","reported_seq":1468,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.052091+0000","last_change":"2024-04-14T15:25:47.681818+0000","last_active":"2024-04-14T15:26:49.052091+0000","last_peered":"2024-04-14T15:26:49.052091+0000","last_clean":"2024-04-14T15:26:49.052091+0000","last_became_active":"2024-04-14T15:15:21.884468+0000","last_became_peered":"2024-04-14T15:15:21.884468+0000","last_unstale":"2024-04-14T15:26:49.052091+0000","last_undegraded":"2024-04-14T15:26:49.052091+0000","last_fullsized":"2024-04-14T15:26:49.052091+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'10","last_scrub_stamp":"2024-04-14T15:25:47.681710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:25:47.681710+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1600,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.051711+0000","last_change":"2024-04-14T15:24:23.473852+0000","last_active":"2024-04-14T15:26:49.051711+0000","last_peered":"2024-04-14T15:26:49.051711+0000","last_clean":"2024-04-14T15:26:49.051711+0000","last_became_active":"2024-04-14T15:09:49.065242+0000","last_became_peered":"2024-04-14T15:09:49.065242+0000","last_unstale":"2024-04-14T15:26:49.051711+0000","last_undegraded":"2024-04-14T15:26:49.051711+0000","last_fullsized":"2024-04-14T15:26:49.051711+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:23.473749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:24:23.473749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:50.167894+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":1517,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.044316+0000","last_change":"2024-04-14T15:25:08.869842+0000","last_active":"2024-04-14T15:26:49.044316+0000","last_peered":"2024-04-14T15:26:49.044316+0000","last_clean":"2024-04-14T15:26:49.044316+0000","last_became_active":"2024-04-14T15:12:56.389204+0000","last_became_peered":"2024-04-14T15:12:56.389204+0000","last_unstale":"2024-04-14T15:26:49.044316+0000","last_undegraded":"2024-04-14T15:26:49.044316+0000","last_fullsized":"2024-04-14T15:26:49.044316+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:08.869762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:25:08.869762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:51.998696+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"479'19","reported_seq":450,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-04-14T15:26:48.044575+0000","last_change":"2024-04-14T15:26:27.070635+0000","last_active":"2024-04-14T15:26:48.044575+0000","last_peered":"2024-04-14T15:26:48.044575+0000","last_clean":"2024-04-14T15:26:48.044575+0000","last_became_active":"2024-04-14T15:26:27.069296+0000","last_became_peered":"2024-04-14T15:26:27.069296+0000","last_unstale":"2024-04-14T15:26:48.044575+0000","last_undegraded":"2024-04-14T15:26:48.044575+0000","last_fullsized":"2024-04-14T15:26:48.044575+0000","mapping_epoch":828,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":829,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'19","last_scrub_stamp":"2024-04-14T15:24:24.516820+0000","last_deep_scrub":"344'13","last_deep_scrub_stamp":"2024-04-14T15:16:52.738163+0000","last_clean_scrub_stamp":"2024-04-14T15:24:24.516820+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:51.529089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":665,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.051942+0000","last_change":"2024-04-14T15:26:30.781405+0000","last_active":"2024-04-14T15:26:49.051942+0000","last_peered":"2024-04-14T15:26:49.051942+0000","last_clean":"2024-04-14T15:26:49.051942+0000","last_became_active":"2024-04-14T15:20:52.996548+0000","last_became_peered":"2024-04-14T15:20:52.996548+0000","last_unstale":"2024-04-14T15:26:49.051942+0000","last_undegraded":"2024-04-14T15:26:49.051942+0000","last_fullsized":"2024-04-14T15:26:49.051942+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:30.781301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:26:30.781301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:39.896617+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1283,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.051401+0000","last_change":"2024-04-14T15:24:31.955026+0000","last_active":"2024-04-14T15:26:49.051401+0000","last_peered":"2024-04-14T15:26:49.051401+0000","last_clean":"2024-04-14T15:26:49.051401+0000","last_became_active":"2024-04-14T15:15:21.882643+0000","last_became_peered":"2024-04-14T15:15:21.882643+0000","last_unstale":"2024-04-14T15:26:49.051401+0000","last_undegraded":"2024-04-14T15:26:49.051401+0000","last_fullsized":"2024-04-14T15:26:49.051401+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:31.954940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:33.485411+0000","last_clean_scrub_stamp":"2024-04-14T15:24:31.954940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:52.880456+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1618,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.052716+0000","last_change":"2024-04-14T15:26:39.900049+0000","last_active":"2024-04-14T15:26:49.052716+0000","last_peered":"2024-04-14T15:26:49.052716+0000","last_clean":"2024-04-14T15:26:49.052716+0000","last_became_active":"2024-04-14T15:09:49.059711+0000","last_became_peered":"2024-04-14T15:09:49.059711+0000","last_unstale":"2024-04-14T15:26:49.052716+0000","last_undegraded":"2024-04-14T15:26:49.052716+0000","last_fullsized":"2024-04-14T15:26:49.052716+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:39.899957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:26:39.899957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:43.913677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":667,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.043695+0000","last_change":"2024-04-14T15:25:57.008522+0000","last_active":"2024-04-14T15:26:49.043695+0000","last_peered":"2024-04-14T15:26:49.043695+0000","last_clean":"2024-04-14T15:26:49.043695+0000","last_became_active":"2024-04-14T15:22:26.203388+0000","last_became_peered":"2024-04-14T15:22:26.203388+0000","last_unstale":"2024-04-14T15:26:49.043695+0000","last_undegraded":"2024-04-14T15:26:49.043695+0000","last_fullsized":"2024-04-14T15:26:49.043695+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:57.008388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:25:57.008388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:18.120983+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"479'24","reported_seq":1662,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.049716+0000","last_change":"2024-04-14T15:24:15.604196+0000","last_active":"2024-04-14T15:26:49.049716+0000","last_peered":"2024-04-14T15:26:49.049716+0000","last_clean":"2024-04-14T15:26:49.049716+0000","last_became_active":"2024-04-14T15:20:52.989300+0000","last_became_peered":"2024-04-14T15:20:52.989300+0000","last_unstale":"2024-04-14T15:26:49.049716+0000","last_undegraded":"2024-04-14T15:26:49.049716+0000","last_fullsized":"2024-04-14T15:26:49.049716+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'24","last_scrub_stamp":"2024-04-14T15:24:15.604065+0000","last_deep_scrub":"464'16","last_deep_scrub_stamp":"2024-04-14T15:19:17.589999+0000","last_clean_scrub_stamp":"2024-04-14T15:24:15.604065+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:50.167744+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":1626,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.048397+0000","last_change":"2024-04-14T15:24:33.369216+0000","last_active":"2024-04-14T15:26:49.048397+0000","last_peered":"2024-04-14T15:26:49.048397+0000","last_clean":"2024-04-14T15:26:49.048397+0000","last_became_active":"2024-04-14T15:09:43.023583+0000","last_became_peered":"2024-04-14T15:09:43.023583+0000","last_unstale":"2024-04-14T15:26:49.048397+0000","last_undegraded":"2024-04-14T15:26:49.048397+0000","last_fullsized":"2024-04-14T15:26:49.048397+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:33.369110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:24:33.369110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:51.082331+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1610,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.045160+0000","last_change":"2024-04-14T15:24:19.367902+0000","last_active":"2024-04-14T15:26:49.045160+0000","last_peered":"2024-04-14T15:26:49.045160+0000","last_clean":"2024-04-14T15:26:49.045160+0000","last_became_active":"2024-04-14T15:09:47.041683+0000","last_became_peered":"2024-04-14T15:09:47.041683+0000","last_unstale":"2024-04-14T15:26:49.045160+0000","last_undegraded":"2024-04-14T15:26:49.045160+0000","last_fullsized":"2024-04-14T15:26:49.045160+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:19.367758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:08.486217+0000","last_clean_scrub_stamp":"2024-04-14T15:24:19.367758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:51.082285+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1287,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.050578+0000","last_change":"2024-04-14T15:25:58.808326+0000","last_active":"2024-04-14T15:26:49.050578+0000","last_peered":"2024-04-14T15:26:49.050578+0000","last_clean":"2024-04-14T15:26:49.050578+0000","last_became_active":"2024-04-14T15:15:21.889567+0000","last_became_peered":"2024-04-14T15:15:21.889567+0000","last_unstale":"2024-04-14T15:26:49.050578+0000","last_undegraded":"2024-04-14T15:26:49.050578+0000","last_fullsized":"2024-04-14T15:26:49.050578+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:58.808247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:25:58.808247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:06.061451+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":1532,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.043348+0000","last_change":"2024-04-14T15:26:01.742722+0000","last_active":"2024-04-14T15:26:49.043348+0000","last_peered":"2024-04-14T15:26:49.043348+0000","last_clean":"2024-04-14T15:26:49.043348+0000","last_became_active":"2024-04-14T15:12:56.387723+0000","last_became_peered":"2024-04-14T15:12:56.387723+0000","last_unstale":"2024-04-14T15:26:49.043348+0000","last_undegraded":"2024-04-14T15:26:49.043348+0000","last_fullsized":"2024-04-14T15:26:49.043348+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:01.742633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:03.685045+0000","last_clean_scrub_stamp":"2024-04-14T15:26:01.742633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:21.588419+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.10","version":"479'8","reported_seq":1635,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.050324+0000","last_change":"2024-04-14T15:24:24.848430+0000","last_active":"2024-04-14T15:26:49.050324+0000","last_peered":"2024-04-14T15:26:49.050324+0000","last_clean":"2024-04-14T15:26:49.050324+0000","last_became_active":"2024-04-14T15:20:53.001300+0000","last_became_peered":"2024-04-14T15:20:53.001300+0000","last_unstale":"2024-04-14T15:26:49.050324+0000","last_undegraded":"2024-04-14T15:26:49.050324+0000","last_fullsized":"2024-04-14T15:26:49.050324+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'8","last_scrub_stamp":"2024-04-14T15:24:24.848331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:24:24.848331+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:52.880373+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":498,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-04-14T15:26:48.047365+0000","last_change":"2024-04-14T15:26:27.062501+0000","last_active":"2024-04-14T15:26:48.047365+0000","last_peered":"2024-04-14T15:26:48.047365+0000","last_clean":"2024-04-14T15:26:48.047365+0000","last_became_active":"2024-04-14T15:26:27.062051+0000","last_became_peered":"2024-04-14T15:26:27.062051+0000","last_unstale":"2024-04-14T15:26:48.047365+0000","last_undegraded":"2024-04-14T15:26:48.047365+0000","last_fullsized":"2024-04-14T15:26:48.047365+0000","mapping_epoch":828,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":829,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:50.711037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:08.712289+0000","last_clean_scrub_stamp":"2024-04-14T15:25:50.711037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:52.533019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":652,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.048835+0000","last_change":"2024-04-14T15:24:30.420853+0000","last_active":"2024-04-14T15:26:49.048835+0000","last_peered":"2024-04-14T15:26:49.048835+0000","last_clean":"2024-04-14T15:26:49.048835+0000","last_became_active":"2024-04-14T15:20:52.990347+0000","last_became_peered":"2024-04-14T15:20:52.990347+0000","last_unstale":"2024-04-14T15:26:49.048835+0000","last_undegraded":"2024-04-14T15:26:49.048835+0000","last_fullsized":"2024-04-14T15:26:49.048835+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:30.420762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:24:30.420762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:50.167835+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1601,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.043009+0000","last_change":"2024-04-14T15:24:18.506706+0000","last_active":"2024-04-14T15:26:49.043009+0000","last_peered":"2024-04-14T15:26:49.043009+0000","last_clean":"2024-04-14T15:26:49.043009+0000","last_became_active":"2024-04-14T15:09:49.061359+0000","last_became_peered":"2024-04-14T15:09:49.061359+0000","last_unstale":"2024-04-14T15:26:49.043009+0000","last_undegraded":"2024-04-14T15:26:49.043009+0000","last_fullsized":"2024-04-14T15:26:49.043009+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:18.506553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:24:18.506553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:51.998569+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"91'130","reported_seq":1789,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.042606+0000","last_change":"2024-04-14T15:26:36.428446+0000","last_active":"2024-04-14T15:26:49.042606+0000","last_peered":"2024-04-14T15:26:49.042606+0000","last_clean":"2024-04-14T15:26:49.042606+0000","last_became_active":"2024-04-14T15:25:02.033333+0000","last_became_peered":"2024-04-14T15:25:02.033333+0000","last_unstale":"2024-04-14T15:26:49.042606+0000","last_undegraded":"2024-04-14T15:26:49.042606+0000","last_fullsized":"2024-04-14T15:26:49.042606+0000","mapping_epoch":757,"log_start":"91'130","ondisk_log_start":"91'130","created":752,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:26:36.428359+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:26:36.428359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":60,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:28:01.339752+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":1524,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.044310+0000","last_change":"2024-04-14T15:25:22.844265+0000","last_active":"2024-04-14T15:26:49.044310+0000","last_peered":"2024-04-14T15:26:49.044310+0000","last_clean":"2024-04-14T15:26:49.044310+0000","last_became_active":"2024-04-14T15:20:52.999623+0000","last_became_peered":"2024-04-14T15:20:52.999623+0000","last_unstale":"2024-04-14T15:26:49.044310+0000","last_undegraded":"2024-04-14T15:26:49.044310+0000","last_fullsized":"2024-04-14T15:26:49.044310+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:22.844179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:15:43.680852+0000","last_clean_scrub_stamp":"2024-04-14T15:25:22.844179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:51.998746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"479'24","reported_seq":1302,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.051459+0000","last_change":"2024-04-14T15:24:29.941597+0000","last_active":"2024-04-14T15:26:49.051459+0000","last_peered":"2024-04-14T15:26:49.051459+0000","last_clean":"2024-04-14T15:26:49.051459+0000","last_became_active":"2024-04-14T15:20:52.996917+0000","last_became_peered":"2024-04-14T15:20:52.996917+0000","last_unstale":"2024-04-14T15:26:49.051459+0000","last_undegraded":"2024-04-14T15:26:49.051459+0000","last_fullsized":"2024-04-14T15:26:49.051459+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'24","last_scrub_stamp":"2024-04-14T15:24:29.941474+0000","last_deep_scrub":"102'16","last_deep_scrub_stamp":"2024-04-14T15:12:35.999326+0000","last_clean_scrub_stamp":"2024-04-14T15:24:29.941474+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:52.880340+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":44,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-04-14T15:26:48.048160+0000","last_change":"2024-04-14T15:26:27.064506+0000","last_active":"2024-04-14T15:26:48.048160+0000","last_peered":"2024-04-14T15:26:48.048160+0000","last_clean":"2024-04-14T15:26:48.048160+0000","last_became_active":"2024-04-14T15:26:27.063836+0000","last_became_peered":"2024-04-14T15:26:27.063836+0000","last_unstale":"2024-04-14T15:26:48.048160+0000","last_undegraded":"2024-04-14T15:26:48.048160+0000","last_fullsized":"2024-04-14T15:26:48.048160+0000","mapping_epoch":828,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":829,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:30.484673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:30.484673+0000","last_clean_scrub_stamp":"2024-04-14T15:24:30.484673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:51.529171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1611,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.042043+0000","last_change":"2024-04-14T15:24:17.557862+0000","last_active":"2024-04-14T15:26:49.042043+0000","last_peered":"2024-04-14T15:26:49.042043+0000","last_clean":"2024-04-14T15:26:49.042043+0000","last_became_active":"2024-04-14T15:20:52.983919+0000","last_became_peered":"2024-04-14T15:20:52.983919+0000","last_unstale":"2024-04-14T15:26:49.042043+0000","last_undegraded":"2024-04-14T15:26:49.042043+0000","last_fullsized":"2024-04-14T15:26:49.042043+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:17.557766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:24:17.557766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:51.998488+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1293,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.051698+0000","last_change":"2024-04-14T15:24:34.995365+0000","last_active":"2024-04-14T15:26:49.051698+0000","last_peered":"2024-04-14T15:26:49.051698+0000","last_clean":"2024-04-14T15:26:49.051698+0000","last_became_active":"2024-04-14T15:20:52.993918+0000","last_became_peered":"2024-04-14T15:20:52.993918+0000","last_unstale":"2024-04-14T15:26:49.051698+0000","last_undegraded":"2024-04-14T15:26:49.051698+0000","last_fullsized":"2024-04-14T15:26:49.051698+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:34.995256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:24:34.995256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:52.880307+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":1523,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.052710+0000","last_change":"2024-04-14T15:25:46.718417+0000","last_active":"2024-04-14T15:26:49.052710+0000","last_peered":"2024-04-14T15:26:49.052710+0000","last_clean":"2024-04-14T15:26:49.052710+0000","last_became_active":"2024-04-14T15:12:56.335230+0000","last_became_peered":"2024-04-14T15:12:56.335230+0000","last_unstale":"2024-04-14T15:26:49.052710+0000","last_undegraded":"2024-04-14T15:26:49.052710+0000","last_fullsized":"2024-04-14T15:26:49.052710+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:46.718339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:25:46.718339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:52.632654+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"776'25","reported_seq":1659,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.064460+0000","last_change":"2024-04-14T15:26:28.939001+0000","last_active":"2024-04-14T15:26:49.064460+0000","last_peered":"2024-04-14T15:26:49.064460+0000","last_clean":"2024-04-14T15:26:49.064460+0000","last_became_active":"2024-04-14T15:09:45.181456+0000","last_became_peered":"2024-04-14T15:09:45.181456+0000","last_unstale":"2024-04-14T15:26:49.064460+0000","last_undegraded":"2024-04-14T15:26:49.064460+0000","last_fullsized":"2024-04-14T15:26:49.064460+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"776'25","last_scrub_stamp":"2024-04-14T15:26:28.938864+0000","last_deep_scrub":"776'25","last_deep_scrub_stamp":"2024-04-14T15:25:21.063579+0000","last_clean_scrub_stamp":"2024-04-14T15:26:28.938864+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:54.484491+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":498,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-04-14T15:26:48.044360+0000","last_change":"2024-04-14T15:26:27.059939+0000","last_active":"2024-04-14T15:26:48.044360+0000","last_peered":"2024-04-14T15:26:48.044360+0000","last_clean":"2024-04-14T15:26:48.044360+0000","last_became_active":"2024-04-14T15:26:27.058655+0000","last_became_peered":"2024-04-14T15:26:27.058655+0000","last_unstale":"2024-04-14T15:26:48.044360+0000","last_undegraded":"2024-04-14T15:26:48.044360+0000","last_fullsized":"2024-04-14T15:26:48.044360+0000","mapping_epoch":828,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":829,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:31.420910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:24:31.420910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:51.529139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"528'9","reported_seq":756,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.051816+0000","last_change":"2024-04-14T15:25:05.354926+0000","last_active":"2024-04-14T15:26:49.051816+0000","last_peered":"2024-04-14T15:26:49.051816+0000","last_clean":"2024-04-14T15:26:49.051816+0000","last_became_active":"2024-04-14T15:20:52.982112+0000","last_became_peered":"2024-04-14T15:20:52.982112+0000","last_unstale":"2024-04-14T15:26:49.051816+0000","last_undegraded":"2024-04-14T15:26:49.051816+0000","last_fullsized":"2024-04-14T15:26:49.051816+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"528'9","last_scrub_stamp":"2024-04-14T15:25:05.354849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:25:05.354849+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:50.168050+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"72'1","reported_seq":1617,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.044123+0000","last_change":"2024-04-14T15:26:45.067993+0000","last_active":"2024-04-14T15:26:49.044123+0000","last_peered":"2024-04-14T15:26:49.044123+0000","last_clean":"2024-04-14T15:26:49.044123+0000","last_became_active":"2024-04-14T15:09:49.062051+0000","last_became_peered":"2024-04-14T15:09:49.062051+0000","last_unstale":"2024-04-14T15:26:49.044123+0000","last_undegraded":"2024-04-14T15:26:49.044123+0000","last_fullsized":"2024-04-14T15:26:49.044123+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-04-14T15:26:45.067818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:26:45.067818+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:57.026494+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":1521,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.052905+0000","last_change":"2024-04-14T15:24:29.418267+0000","last_active":"2024-04-14T15:26:49.052905+0000","last_peered":"2024-04-14T15:26:49.052905+0000","last_clean":"2024-04-14T15:26:49.052905+0000","last_became_active":"2024-04-14T15:20:52.986657+0000","last_became_peered":"2024-04-14T15:20:52.986657+0000","last_unstale":"2024-04-14T15:26:49.052905+0000","last_undegraded":"2024-04-14T15:26:49.052905+0000","last_fullsized":"2024-04-14T15:26:49.052905+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:29.418183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:29.418183+0000","last_clean_scrub_stamp":"2024-04-14T15:24:29.418183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:52.918423+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"519'24","reported_seq":1045,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-04-14T15:26:48.048294+0000","last_change":"2024-04-14T15:26:27.070797+0000","last_active":"2024-04-14T15:26:48.048294+0000","last_peered":"2024-04-14T15:26:48.048294+0000","last_clean":"2024-04-14T15:26:48.048294+0000","last_became_active":"2024-04-14T15:26:27.067489+0000","last_became_peered":"2024-04-14T15:26:27.067489+0000","last_unstale":"2024-04-14T15:26:48.048294+0000","last_undegraded":"2024-04-14T15:26:48.048294+0000","last_fullsized":"2024-04-14T15:26:48.048294+0000","mapping_epoch":828,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":829,"parent":"0.0","parent_split_bits":0,"last_scrub":"519'24","last_scrub_stamp":"2024-04-14T15:25:55.662871+0000","last_deep_scrub":"519'24","last_deep_scrub_stamp":"2024-04-14T15:20:44.763417+0000","last_clean_scrub_stamp":"2024-04-14T15:25:55.662871+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:14.925371+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"684'4","reported_seq":1320,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.050791+0000","last_change":"2024-04-14T15:26:04.764095+0000","last_active":"2024-04-14T15:26:49.050791+0000","last_peered":"2024-04-14T15:26:49.050791+0000","last_clean":"2024-04-14T15:26:49.050791+0000","last_became_active":"2024-04-14T15:20:52.995628+0000","last_became_peered":"2024-04-14T15:20:52.995628+0000","last_unstale":"2024-04-14T15:26:49.050791+0000","last_undegraded":"2024-04-14T15:26:49.050791+0000","last_fullsized":"2024-04-14T15:26:49.050791+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"684'4","last_scrub_stamp":"2024-04-14T15:26:04.764014+0000","last_deep_scrub":"684'4","last_deep_scrub_stamp":"2024-04-14T15:23:46.931560+0000","last_clean_scrub_stamp":"2024-04-14T15:26:04.764014+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:16.001689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":957,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1612,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.064580+0000","last_change":"2024-04-14T15:24:30.397659+0000","last_active":"2024-04-14T15:26:49.064580+0000","last_peered":"2024-04-14T15:26:49.064580+0000","last_clean":"2024-04-14T15:26:49.064580+0000","last_became_active":"2024-04-14T15:09:47.045730+0000","last_became_peered":"2024-04-14T15:09:47.045730+0000","last_unstale":"2024-04-14T15:26:49.064580+0000","last_undegraded":"2024-04-14T15:26:49.064580+0000","last_fullsized":"2024-04-14T15:26:49.064580+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:30.397585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:24:30.397585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:52.918459+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1605,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.048108+0000","last_change":"2024-04-14T15:24:30.343193+0000","last_active":"2024-04-14T15:26:49.048108+0000","last_peered":"2024-04-14T15:26:49.048108+0000","last_clean":"2024-04-14T15:26:49.048108+0000","last_became_active":"2024-04-14T15:09:49.057545+0000","last_became_peered":"2024-04-14T15:09:49.057545+0000","last_unstale":"2024-04-14T15:26:49.048108+0000","last_undegraded":"2024-04-14T15:26:49.048108+0000","last_fullsized":"2024-04-14T15:26:49.048108+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:30.343106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:30.343106+0000","last_clean_scrub_stamp":"2024-04-14T15:24:30.343106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:51.082316+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":395,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-04-14T15:26:48.047773+0000","last_change":"2024-04-14T15:26:27.060128+0000","last_active":"2024-04-14T15:26:48.047773+0000","last_peered":"2024-04-14T15:26:48.047773+0000","last_clean":"2024-04-14T15:26:48.047773+0000","last_became_active":"2024-04-14T15:26:27.059738+0000","last_became_peered":"2024-04-14T15:26:27.059738+0000","last_unstale":"2024-04-14T15:26:48.047773+0000","last_undegraded":"2024-04-14T15:26:48.047773+0000","last_fullsized":"2024-04-14T15:26:48.047773+0000","mapping_epoch":828,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":829,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:25.320719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:25:25.320719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"479'27","reported_seq":1665,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.045995+0000","last_change":"2024-04-14T15:26:00.726711+0000","last_active":"2024-04-14T15:26:49.045995+0000","last_peered":"2024-04-14T15:26:49.045995+0000","last_clean":"2024-04-14T15:26:49.045995+0000","last_became_active":"2024-04-14T15:09:45.032887+0000","last_became_peered":"2024-04-14T15:09:45.032887+0000","last_unstale":"2024-04-14T15:26:49.045995+0000","last_undegraded":"2024-04-14T15:26:49.045995+0000","last_fullsized":"2024-04-14T15:26:49.045995+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'27","last_scrub_stamp":"2024-04-14T15:26:00.726554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:26:00.726554+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:22.857124+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":652,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.050174+0000","last_change":"2024-04-14T15:25:16.387612+0000","last_active":"2024-04-14T15:26:49.050174+0000","last_peered":"2024-04-14T15:26:49.050174+0000","last_clean":"2024-04-14T15:26:49.050174+0000","last_became_active":"2024-04-14T15:20:52.990634+0000","last_became_peered":"2024-04-14T15:20:52.990634+0000","last_unstale":"2024-04-14T15:26:49.050174+0000","last_undegraded":"2024-04-14T15:26:49.050174+0000","last_fullsized":"2024-04-14T15:26:49.050174+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:16.387432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:42.754721+0000","last_clean_scrub_stamp":"2024-04-14T15:25:16.387432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:50.168138+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1617,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.048307+0000","last_change":"2024-04-14T15:25:22.689804+0000","last_active":"2024-04-14T15:26:49.048307+0000","last_peered":"2024-04-14T15:26:49.048307+0000","last_clean":"2024-04-14T15:26:49.048307+0000","last_became_active":"2024-04-14T15:09:47.042733+0000","last_became_peered":"2024-04-14T15:09:47.042733+0000","last_unstale":"2024-04-14T15:26:49.048307+0000","last_undegraded":"2024-04-14T15:26:49.048307+0000","last_fullsized":"2024-04-14T15:26:49.048307+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:22.689718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:25:22.689718+0000","last_clean_scrub_stamp":"2024-04-14T15:25:22.689718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1615,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.051185+0000","last_change":"2024-04-14T15:25:48.252911+0000","last_active":"2024-04-14T15:26:49.051185+0000","last_peered":"2024-04-14T15:26:49.051185+0000","last_clean":"2024-04-14T15:26:49.051185+0000","last_became_active":"2024-04-14T15:20:52.998253+0000","last_became_peered":"2024-04-14T15:20:52.998253+0000","last_unstale":"2024-04-14T15:26:49.051185+0000","last_undegraded":"2024-04-14T15:26:49.051185+0000","last_fullsized":"2024-04-14T15:26:49.051185+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:48.252835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:54.453174+0000","last_clean_scrub_stamp":"2024-04-14T15:25:48.252835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:16.009083+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":1517,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.042589+0000","last_change":"2024-04-14T15:25:24.895474+0000","last_active":"2024-04-14T15:26:49.042589+0000","last_peered":"2024-04-14T15:26:49.042589+0000","last_clean":"2024-04-14T15:26:49.042589+0000","last_became_active":"2024-04-14T15:12:56.392352+0000","last_became_peered":"2024-04-14T15:12:56.392352+0000","last_unstale":"2024-04-14T15:26:49.042589+0000","last_undegraded":"2024-04-14T15:26:49.042589+0000","last_fullsized":"2024-04-14T15:26:49.042589+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:24.895381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:25:24.895381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:51.998712+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"479'18","reported_seq":1327,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.051035+0000","last_change":"2024-04-14T15:25:07.782722+0000","last_active":"2024-04-14T15:26:49.051035+0000","last_peered":"2024-04-14T15:26:49.051035+0000","last_clean":"2024-04-14T15:26:49.051035+0000","last_became_active":"2024-04-14T15:15:21.877838+0000","last_became_peered":"2024-04-14T15:15:21.877838+0000","last_unstale":"2024-04-14T15:26:49.051035+0000","last_undegraded":"2024-04-14T15:26:49.051035+0000","last_fullsized":"2024-04-14T15:26:49.051035+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'18","last_scrub_stamp":"2024-04-14T15:25:07.782608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:25:07.782608+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:52.880556+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":1305,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.051515+0000","last_change":"2024-04-14T15:25:46.654660+0000","last_active":"2024-04-14T15:26:49.051515+0000","last_peered":"2024-04-14T15:26:49.051515+0000","last_clean":"2024-04-14T15:26:49.051515+0000","last_became_active":"2024-04-14T15:20:53.001534+0000","last_became_peered":"2024-04-14T15:20:53.001534+0000","last_unstale":"2024-04-14T15:26:49.051515+0000","last_undegraded":"2024-04-14T15:26:49.051515+0000","last_fullsized":"2024-04-14T15:26:49.051515+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:46.654578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:25:46.654578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:10.485147+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1611,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.051026+0000","last_change":"2024-04-14T15:25:14.382875+0000","last_active":"2024-04-14T15:26:49.051026+0000","last_peered":"2024-04-14T15:26:49.051026+0000","last_clean":"2024-04-14T15:26:49.051026+0000","last_became_active":"2024-04-14T15:09:47.048045+0000","last_became_peered":"2024-04-14T15:09:47.048045+0000","last_unstale":"2024-04-14T15:26:49.051026+0000","last_undegraded":"2024-04-14T15:26:49.051026+0000","last_fullsized":"2024-04-14T15:26:49.051026+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:14.382783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:25:14.382783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:50.168077+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1295,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.051833+0000","last_change":"2024-04-14T15:25:09.737733+0000","last_active":"2024-04-14T15:26:49.051833+0000","last_peered":"2024-04-14T15:26:49.051833+0000","last_clean":"2024-04-14T15:26:49.051833+0000","last_became_active":"2024-04-14T15:20:52.998775+0000","last_became_peered":"2024-04-14T15:20:52.998775+0000","last_unstale":"2024-04-14T15:26:49.051833+0000","last_undegraded":"2024-04-14T15:26:49.051833+0000","last_fullsized":"2024-04-14T15:26:49.051833+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:09.737658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:22:20.946524+0000","last_clean_scrub_stamp":"2024-04-14T15:25:09.737658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:52.880617+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":1531,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.043114+0000","last_change":"2024-04-14T15:26:29.547041+0000","last_active":"2024-04-14T15:26:49.043114+0000","last_peered":"2024-04-14T15:26:49.043114+0000","last_clean":"2024-04-14T15:26:49.043114+0000","last_became_active":"2024-04-14T15:12:56.375278+0000","last_became_peered":"2024-04-14T15:12:56.375278+0000","last_unstale":"2024-04-14T15:26:49.043114+0000","last_undegraded":"2024-04-14T15:26:49.043114+0000","last_fullsized":"2024-04-14T15:26:49.043114+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:29.546956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:25:19.668662+0000","last_clean_scrub_stamp":"2024-04-14T15:26:29.546956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:44.008546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"479'19","reported_seq":1644,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.050075+0000","last_change":"2024-04-14T15:25:21.286947+0000","last_active":"2024-04-14T15:26:49.050075+0000","last_peered":"2024-04-14T15:26:49.050075+0000","last_clean":"2024-04-14T15:26:49.050075+0000","last_became_active":"2024-04-14T15:09:45.030093+0000","last_became_peered":"2024-04-14T15:09:45.030093+0000","last_unstale":"2024-04-14T15:26:49.050075+0000","last_undegraded":"2024-04-14T15:26:49.050075+0000","last_fullsized":"2024-04-14T15:26:49.050075+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'19","last_scrub_stamp":"2024-04-14T15:25:21.286844+0000","last_deep_scrub":"276'13","last_deep_scrub_stamp":"2024-04-14T15:15:43.969775+0000","last_clean_scrub_stamp":"2024-04-14T15:25:21.286844+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:50.168209+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":1318,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.043499+0000","last_change":"2024-04-14T15:24:26.465423+0000","last_active":"2024-04-14T15:26:49.043499+0000","last_peered":"2024-04-14T15:26:49.043499+0000","last_clean":"2024-04-14T15:26:49.043499+0000","last_became_active":"2024-04-14T15:14:30.857727+0000","last_became_peered":"2024-04-14T15:14:30.857727+0000","last_unstale":"2024-04-14T15:26:49.043499+0000","last_undegraded":"2024-04-14T15:26:49.043499+0000","last_fullsized":"2024-04-14T15:26:49.043499+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:26.465343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:20:00.650062+0000","last_clean_scrub_stamp":"2024-04-14T15:24:26.465343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:51.998585+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":490,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-04-14T15:26:48.045551+0000","last_change":"2024-04-14T15:26:27.069886+0000","last_active":"2024-04-14T15:26:48.045551+0000","last_peered":"2024-04-14T15:26:48.045551+0000","last_clean":"2024-04-14T15:26:48.045551+0000","last_became_active":"2024-04-14T15:26:27.068043+0000","last_became_peered":"2024-04-14T15:26:27.068043+0000","last_unstale":"2024-04-14T15:26:48.045551+0000","last_undegraded":"2024-04-14T15:26:48.045551+0000","last_fullsized":"2024-04-14T15:26:48.045551+0000","mapping_epoch":828,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":829,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:35.503279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:24:35.503279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:51.529155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1615,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.053384+0000","last_change":"2024-04-14T15:25:22.008490+0000","last_active":"2024-04-14T15:26:49.053384+0000","last_peered":"2024-04-14T15:26:49.053384+0000","last_clean":"2024-04-14T15:26:49.053384+0000","last_became_active":"2024-04-14T15:09:49.061756+0000","last_became_peered":"2024-04-14T15:09:49.061756+0000","last_unstale":"2024-04-14T15:26:49.053384+0000","last_undegraded":"2024-04-14T15:26:49.053384+0000","last_fullsized":"2024-04-14T15:26:49.053384+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:22.008411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:25:22.008411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:52.918546+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":660,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.050670+0000","last_change":"2024-04-14T15:26:44.215656+0000","last_active":"2024-04-14T15:26:49.050670+0000","last_peered":"2024-04-14T15:26:49.050670+0000","last_clean":"2024-04-14T15:26:49.050670+0000","last_became_active":"2024-04-14T15:20:52.990838+0000","last_became_peered":"2024-04-14T15:20:52.990838+0000","last_unstale":"2024-04-14T15:26:49.050670+0000","last_undegraded":"2024-04-14T15:26:49.050670+0000","last_fullsized":"2024-04-14T15:26:49.050670+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:44.215516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:26:44.215516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:28:05.880587+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"479'9","reported_seq":1630,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.049907+0000","last_change":"2024-04-14T15:25:53.355715+0000","last_active":"2024-04-14T15:26:49.049907+0000","last_peered":"2024-04-14T15:26:49.049907+0000","last_clean":"2024-04-14T15:26:49.049907+0000","last_became_active":"2024-04-14T15:09:45.028667+0000","last_became_peered":"2024-04-14T15:09:45.028667+0000","last_unstale":"2024-04-14T15:26:49.049907+0000","last_undegraded":"2024-04-14T15:26:49.049907+0000","last_fullsized":"2024-04-14T15:26:49.049907+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'9","last_scrub_stamp":"2024-04-14T15:25:53.355630+0000","last_deep_scrub":"87'7","last_deep_scrub_stamp":"2024-04-14T15:11:10.510785+0000","last_clean_scrub_stamp":"2024-04-14T15:25:53.355630+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:55.830404+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":1110,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.042716+0000","last_change":"2024-04-14T15:26:41.432283+0000","last_active":"2024-04-14T15:26:49.042716+0000","last_peered":"2024-04-14T15:26:49.042716+0000","last_clean":"2024-04-14T15:26:49.042716+0000","last_became_active":"2024-04-14T15:20:52.989307+0000","last_became_peered":"2024-04-14T15:20:52.989307+0000","last_unstale":"2024-04-14T15:26:49.042716+0000","last_undegraded":"2024-04-14T15:26:49.042716+0000","last_fullsized":"2024-04-14T15:26:49.042716+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:41.432206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:22:15.944005+0000","last_clean_scrub_stamp":"2024-04-14T15:26:41.432206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:28:02.583122+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1612,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.042938+0000","last_change":"2024-04-14T15:25:52.924634+0000","last_active":"2024-04-14T15:26:49.042938+0000","last_peered":"2024-04-14T15:26:49.042938+0000","last_clean":"2024-04-14T15:26:49.042938+0000","last_became_active":"2024-04-14T15:09:47.046517+0000","last_became_peered":"2024-04-14T15:09:47.046517+0000","last_unstale":"2024-04-14T15:26:49.042938+0000","last_undegraded":"2024-04-14T15:26:49.042938+0000","last_fullsized":"2024-04-14T15:26:49.042938+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:52.924559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:25:52.924559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:02.028028+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":486,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-04-14T15:26:48.044377+0000","last_change":"2024-04-14T15:26:27.065886+0000","last_active":"2024-04-14T15:26:48.044377+0000","last_peered":"2024-04-14T15:26:48.044377+0000","last_clean":"2024-04-14T15:26:48.044377+0000","last_became_active":"2024-04-14T15:26:27.065443+0000","last_became_peered":"2024-04-14T15:26:27.065443+0000","last_unstale":"2024-04-14T15:26:48.044377+0000","last_undegraded":"2024-04-14T15:26:48.044377+0000","last_fullsized":"2024-04-14T15:26:48.044377+0000","mapping_epoch":828,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":829,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:28.774199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:56.268086+0000","last_clean_scrub_stamp":"2024-04-14T15:25:28.774199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:52.524760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":1523,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.045088+0000","last_change":"2024-04-14T15:25:09.858241+0000","last_active":"2024-04-14T15:26:49.045088+0000","last_peered":"2024-04-14T15:26:49.045088+0000","last_clean":"2024-04-14T15:26:49.045088+0000","last_became_active":"2024-04-14T15:20:52.984387+0000","last_became_peered":"2024-04-14T15:20:52.984387+0000","last_unstale":"2024-04-14T15:26:49.045088+0000","last_undegraded":"2024-04-14T15:26:49.045088+0000","last_fullsized":"2024-04-14T15:26:49.045088+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:09.858143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:34.558312+0000","last_clean_scrub_stamp":"2024-04-14T15:25:09.858143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:51.998625+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.18","version":"806'19","reported_seq":1661,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.048478+0000","last_change":"2024-04-14T15:25:56.405925+0000","last_active":"2024-04-14T15:26:49.048478+0000","last_peered":"2024-04-14T15:26:49.048478+0000","last_clean":"2024-04-14T15:26:49.048478+0000","last_became_active":"2024-04-14T15:20:52.985210+0000","last_became_peered":"2024-04-14T15:20:52.985210+0000","last_unstale":"2024-04-14T15:26:49.048478+0000","last_undegraded":"2024-04-14T15:26:49.048478+0000","last_fullsized":"2024-04-14T15:26:49.048478+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"806'19","last_scrub_stamp":"2024-04-14T15:25:56.405833+0000","last_deep_scrub":"806'19","last_deep_scrub_stamp":"2024-04-14T15:25:56.405833+0000","last_clean_scrub_stamp":"2024-04-14T15:25:56.405833+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:23.474418+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":1301,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.051477+0000","last_change":"2024-04-14T15:26:39.906886+0000","last_active":"2024-04-14T15:26:49.051477+0000","last_peered":"2024-04-14T15:26:49.051477+0000","last_clean":"2024-04-14T15:26:49.051477+0000","last_became_active":"2024-04-14T15:15:21.891926+0000","last_became_peered":"2024-04-14T15:15:21.891926+0000","last_unstale":"2024-04-14T15:26:49.051477+0000","last_undegraded":"2024-04-14T15:26:49.051477+0000","last_fullsized":"2024-04-14T15:26:49.051477+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:39.906816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:26:39.906816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:53.024639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1610,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.043523+0000","last_change":"2024-04-14T15:24:22.302587+0000","last_active":"2024-04-14T15:26:49.043523+0000","last_peered":"2024-04-14T15:26:49.043523+0000","last_clean":"2024-04-14T15:26:49.043523+0000","last_became_active":"2024-04-14T15:09:47.041486+0000","last_became_peered":"2024-04-14T15:09:47.041486+0000","last_unstale":"2024-04-14T15:26:49.043523+0000","last_undegraded":"2024-04-14T15:26:49.043523+0000","last_fullsized":"2024-04-14T15:26:49.043523+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:22.302513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:24:22.302513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:51.082301+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1599,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.049969+0000","last_change":"2024-04-14T15:25:08.411730+0000","last_active":"2024-04-14T15:26:49.049969+0000","last_peered":"2024-04-14T15:26:49.049969+0000","last_clean":"2024-04-14T15:26:49.049969+0000","last_became_active":"2024-04-14T15:09:49.058904+0000","last_became_peered":"2024-04-14T15:09:49.058904+0000","last_unstale":"2024-04-14T15:26:49.049969+0000","last_undegraded":"2024-04-14T15:26:49.049969+0000","last_fullsized":"2024-04-14T15:26:49.049969+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:08.411650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:25:08.411650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:50.167980+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":395,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-04-14T15:26:48.047682+0000","last_change":"2024-04-14T15:26:27.069363+0000","last_active":"2024-04-14T15:26:48.047682+0000","last_peered":"2024-04-14T15:26:48.047682+0000","last_clean":"2024-04-14T15:26:48.047682+0000","last_became_active":"2024-04-14T15:26:27.068240+0000","last_became_peered":"2024-04-14T15:26:27.068240+0000","last_unstale":"2024-04-14T15:26:48.047682+0000","last_undegraded":"2024-04-14T15:26:48.047682+0000","last_fullsized":"2024-04-14T15:26:48.047682+0000","mapping_epoch":828,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":829,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:25:16.796595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:25:16.796595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:51.529271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"479'34","reported_seq":1345,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.051572+0000","last_change":"2024-04-14T15:25:10.762245+0000","last_active":"2024-04-14T15:26:49.051572+0000","last_peered":"2024-04-14T15:26:49.051572+0000","last_clean":"2024-04-14T15:26:49.051572+0000","last_became_active":"2024-04-14T15:15:21.884310+0000","last_became_peered":"2024-04-14T15:15:21.884310+0000","last_unstale":"2024-04-14T15:26:49.051572+0000","last_undegraded":"2024-04-14T15:26:49.051572+0000","last_fullsized":"2024-04-14T15:26:49.051572+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'34","last_scrub_stamp":"2024-04-14T15:25:10.762092+0000","last_deep_scrub":"306'22","last_deep_scrub_stamp":"2024-04-14T15:16:13.898106+0000","last_clean_scrub_stamp":"2024-04-14T15:25:10.762092+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:52.880648+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":1620,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.064148+0000","last_change":"2024-04-14T15:24:24.353115+0000","last_active":"2024-04-14T15:26:49.064148+0000","last_peered":"2024-04-14T15:26:49.064148+0000","last_clean":"2024-04-14T15:26:49.064148+0000","last_became_active":"2024-04-14T15:09:43.017351+0000","last_became_peered":"2024-04-14T15:09:43.017351+0000","last_unstale":"2024-04-14T15:26:49.064148+0000","last_undegraded":"2024-04-14T15:26:49.064148+0000","last_fullsized":"2024-04-14T15:26:49.064148+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:24.352978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:24:24.352978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:52.918405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1610,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.048768+0000","last_change":"2024-04-14T15:24:32.403277+0000","last_active":"2024-04-14T15:26:49.048768+0000","last_peered":"2024-04-14T15:26:49.048768+0000","last_clean":"2024-04-14T15:26:49.048768+0000","last_became_active":"2024-04-14T15:20:52.974360+0000","last_became_peered":"2024-04-14T15:20:52.974360+0000","last_unstale":"2024-04-14T15:26:49.048768+0000","last_undegraded":"2024-04-14T15:26:49.048768+0000","last_fullsized":"2024-04-14T15:26:49.048768+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:24:32.403077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:32.403077+0000","last_clean_scrub_stamp":"2024-04-14T15:24:32.403077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:26:50.167866+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1616,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-04-14T15:26:49.051108+0000","last_change":"2024-04-14T15:26:37.898194+0000","last_active":"2024-04-14T15:26:49.051108+0000","last_peered":"2024-04-14T15:26:49.051108+0000","last_clean":"2024-04-14T15:26:49.051108+0000","last_became_active":"2024-04-14T15:20:52.998938+0000","last_became_peered":"2024-04-14T15:20:52.998938+0000","last_unstale":"2024-04-14T15:26:49.051108+0000","last_undegraded":"2024-04-14T15:26:49.051108+0000","last_fullsized":"2024-04-14T15:26:49.051108+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:26:37.898114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:27.874498+0000","last_clean_scrub_stamp":"2024-04-14T15:26:37.898114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:27:56.119031+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":345,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":323,"num_pg":8,"stat_sum":{"num_bytes":4000299,"num_objects":44,"num_object_clones":1,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":28,"num_read_kb":9,"num_write":68,"num_write_kb":3968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6508544,"data_stored":6508544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101,"ondisk_log_size":101,"up":24,"acting":24,"num_store_stats":7},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":5,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1003,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2109440,"data_stored":2098784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":63,"acting":63,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1017,"num_read_kb":1012,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1119,"num_read_kb":876,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":1254,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":810,"ondisk_log_size":810,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103379,"num_pgs":87,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":88316,"kb_used_data":7144,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93656836,"statfs":{"total":95995035648,"available":95904600064,"internally_reserved":0,"allocated":7315456,"data_stored":3702180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":83098058},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":210453397718,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575088,"num_pgs":101,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":108348,"kb_used_data":9132,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93636804,"statfs":{"total":95995035648,"available":95884087296,"internally_reserved":0,"allocated":9351168,"data_stored":5709524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738613,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":116944,"kb_used_data":9412,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93628208,"statfs":{"total":95995035648,"available":95875284992,"internally_reserved":0,"allocated":9637888,"data_stored":5997361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":110098887},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411564,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":114960,"kb_used_data":7416,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93630192,"statfs":{"total":95995035648,"available":95877316608,"internally_reserved":0,"allocated":7593984,"data_stored":3973757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182630,"num_pgs":91,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":93100,"kb_used_data":7772,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":93652052,"statfs":{"total":95995035648,"available":95899701248,"internally_reserved":0,"allocated":7958528,"data_stored":4301636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":87357895},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986400,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":112804,"kb_used_data":9424,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93632348,"statfs":{"total":95995035648,"available":95879524352,"internally_reserved":0,"allocated":9650176,"data_stored":6020541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659357,"num_pgs":99,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":106596,"kb_used_data":7384,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93638556,"statfs":{"total":95995035648,"available":95885881344,"internally_reserved":0,"allocated":7561216,"data_stored":3939972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2048000,"data_stored":2048000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2088960,"data_stored":2088960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":86016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":86016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2068480,"data_stored":2068480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":53248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":77824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-14T15:26:53.764 INFO:tasks.cephadm.ceph_manager.ceph:decrease pool size by 10 2024-04-14T15:26:53.764 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd pool set .mgr pg_num 11 2024-04-14T15:26:53.844 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:26:53.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:53.870 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:53.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:26:53.971 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:26:53.995 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:53.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:53.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:26:54.009 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:26:53 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:26:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:26:54.097 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:26:54.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:54.122 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:54.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:26:54.223 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:26:54.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:54.247 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:54.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:26:54.282 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:54 smithi090 ceph-mon[30399]: osdmap e849: 8 total, 8 up, 7 in 2024-04-14T15:26:54.282 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:54 smithi090 ceph-mon[30399]: pgmap v1268: 192 pgs: 192 active+clean; 4.4 MiB data, 724 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:26:54.282 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:54 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:26:54.282 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:54 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:26:54.282 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:54 smithi090 ceph-mon[41670]: osdmap e849: 8 total, 8 up, 7 in 2024-04-14T15:26:54.283 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:54 smithi090 ceph-mon[41670]: pgmap v1268: 192 pgs: 192 active+clean; 4.4 MiB data, 724 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:26:54.283 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:54 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:26:54.283 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:54 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:26:54.348 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:26:54.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:54 smithi092 ceph-mon[33966]: osdmap e849: 8 total, 8 up, 7 in 2024-04-14T15:26:54.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:54 smithi092 ceph-mon[33966]: pgmap v1268: 192 pgs: 192 active+clean; 4.4 MiB data, 724 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:26:54.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:54 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:26:54.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:54 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:26:54.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:54.373 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:54.373 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:26:54.474 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:26:54.499 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:54.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:54.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:26:54.502 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:26:54.548 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:26:54.601 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:26:54.624 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:54.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:54.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:26:54.726 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:26:54.751 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:54.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:54.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:26:54.852 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:26:54.876 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:54.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:54.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:26:54.978 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:26:54.988 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:26:55.004 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:55.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:55.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:26:55.106 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:26:55.129 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:55.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:55.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:26:55.223 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:55 smithi090 ceph-mon[41670]: osdmap e850: 8 total, 8 up, 7 in 2024-04-14T15:26:55.223 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:55 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:26:55.223 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:55 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:26:55.223 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:55 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:26:55.223 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:55 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:26:55.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:55 smithi090 ceph-mon[30399]: osdmap e850: 8 total, 8 up, 7 in 2024-04-14T15:26:55.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:55 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:26:55.224 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:55 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:26:55.224 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:55 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:26:55.224 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:55 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:26:55.231 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:26:55.254 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:55.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:55.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:26:55.356 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:26:55.371 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:55 smithi092 ceph-mon[33966]: osdmap e850: 8 total, 8 up, 7 in 2024-04-14T15:26:55.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:55 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:26:55.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:55 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:26:55.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:55 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:26:55.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:55 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:26:55.379 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:55.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:55.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:26:55.481 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:26:55.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:55.506 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:55.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:26:55.607 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0 2024-04-14T15:26:55.632 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:55.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:55.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.0' 2024-04-14T15:26:55.734 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4 2024-04-14T15:26:55.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:55.759 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:55.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.4' 2024-04-14T15:26:55.860 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:26:55.874 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:26:55 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:26:55.885 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:55.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:55.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:26:55.987 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:26:56.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:56.012 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:56.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:26:56.046 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlush (9117 ms) 2024-04-14T15:26:56.047 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2024-04-14T15:26:56.047 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (13136 ms) 2024-04-14T15:26:56.047 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Flush 2024-04-14T15:26:56.047 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Flush (7742 ms) 2024-04-14T15:26:56.047 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2024-04-14T15:26:56.047 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (12996 ms) 2024-04-14T15:26:56.048 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2024-04-14T15:26:56.048 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (8002 ms) 2024-04-14T15:26:56.048 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2024-04-14T15:26:56.048 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (8004 ms) 2024-04-14T15:26:56.048 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2024-04-14T15:26:56.048 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: hmm, no HitSet yet 2024-04-14T15:26:56.049 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: ok, hit_set contains 337:602f83fe:::foo:head 2024-04-14T15:26:56.049 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (7999 ms) 2024-04-14T15:26:56.049 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2024-04-14T15:26:56.049 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: got ls 1713108334,0 2024-04-14T15:26:56.049 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: first is 1713108334 2024-04-14T15:26:56.049 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: got ls 1713108334,0 2024-04-14T15:26:56.050 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: got ls 1713108334,0 2024-04-14T15:26:56.050 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: got ls 1713108334,1713108336,1713108337,0 2024-04-14T15:26:56.050 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: got ls 1713108334,1713108336,1713108337,0 2024-04-14T15:26:56.050 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: got ls 1713108334,1713108336,1713108337,0 2024-04-14T15:26:56.050 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: got ls 1713108334,1713108336,1713108337,1713108339,1713108340,0 2024-04-14T15:26:56.050 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: got ls 1713108334,1713108336,1713108337,1713108339,1713108340,0 2024-04-14T15:26:56.051 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: got ls 1713108334,1713108336,1713108337,1713108339,1713108340,0 2024-04-14T15:26:56.051 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: got ls 1713108334,1713108336,1713108337,1713108339,1713108340,1713108342,1713108343,0 2024-04-14T15:26:56.051 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: got ls 1713108334,1713108336,1713108337,1713108339,1713108340,1713108342,1713108343,0 2024-04-14T15:26:56.051 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: got ls 1713108334,1713108336,1713108337,1713108339,1713108340,1713108342,1713108343,0 2024-04-14T15:26:56.051 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: got ls 1713108337,1713108339,1713108340,1713108342,1713108343,1713108345,1713108346,0 2024-04-14T15:26:56.051 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: first now 1713108337, trimmed 2024-04-14T15:26:56.052 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (20008 ms) 2024-04-14T15:26:56.052 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2024-04-14T15:26:56.052 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: foo0 2024-04-14T15:26:56.052 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-04-14T15:26:56.052 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (14005 ms) 2024-04-14T15:26:56.052 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2024-04-14T15:26:56.053 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (17015 ms) 2024-04-14T15:26:56.053 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2024-04-14T15:26:56.053 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (22064 ms) 2024-04-14T15:26:56.053 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead 2024-04-14T15:26:56.053 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (5040 ms) 2024-04-14T15:26:56.053 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead 2024-04-14T15:26:56.054 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (2889 ms) 2024-04-14T15:26:56.054 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead 2024-04-14T15:26:56.054 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (3015 ms) 2024-04-14T15:26:56.054 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier 2024-04-14T15:26:56.055 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (3014 ms) 2024-04-14T15:26:56.055 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (230080 ms total) 2024-04-14T15:26:56.055 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: 2024-04-14T15:26:56.055 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [----------] Global test environment tear-down 2024-04-14T15:26:56.055 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [==========] 77 tests from 4 test suites ran. (841779 ms total) 2024-04-14T15:26:56.056 INFO:tasks.workunit.client.0.smithi090.stdout: api_tier_pp: [ PASSED ] 77 tests. 2024-04-14T15:26:56.065 INFO:tasks.workunit.client.0.smithi090.stderr:+ for t in "${!pids[@]}" 2024-04-14T15:26:56.065 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=93319 2024-04-14T15:26:56.065 INFO:tasks.workunit.client.0.smithi090.stderr:+ wait 93319 2024-04-14T15:26:56.065 INFO:tasks.workunit.client.0.smithi090.stderr:+ for t in "${!pids[@]}" 2024-04-14T15:26:56.065 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=93555 2024-04-14T15:26:56.065 INFO:tasks.workunit.client.0.smithi090.stderr:+ wait 93555 2024-04-14T15:26:56.065 INFO:tasks.workunit.client.0.smithi090.stderr:+ for t in "${!pids[@]}" 2024-04-14T15:26:56.065 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=94043 2024-04-14T15:26:56.065 INFO:tasks.workunit.client.0.smithi090.stderr:+ wait 94043 2024-04-14T15:26:56.065 INFO:tasks.workunit.client.0.smithi090.stderr:+ for t in "${!pids[@]}" 2024-04-14T15:26:56.065 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=93772 2024-04-14T15:26:56.065 INFO:tasks.workunit.client.0.smithi090.stderr:+ wait 93772 2024-04-14T15:26:56.066 INFO:tasks.workunit.client.0.smithi090.stderr:+ for t in "${!pids[@]}" 2024-04-14T15:26:56.066 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=93665 2024-04-14T15:26:56.066 INFO:tasks.workunit.client.0.smithi090.stderr:+ wait 93665 2024-04-14T15:26:56.066 INFO:tasks.workunit.client.0.smithi090.stderr:+ for t in "${!pids[@]}" 2024-04-14T15:26:56.066 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=93399 2024-04-14T15:26:56.066 INFO:tasks.workunit.client.0.smithi090.stderr:+ wait 93399 2024-04-14T15:26:56.066 INFO:tasks.workunit.client.0.smithi090.stderr:+ for t in "${!pids[@]}" 2024-04-14T15:26:56.066 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=93737 2024-04-14T15:26:56.066 INFO:tasks.workunit.client.0.smithi090.stderr:+ wait 93737 2024-04-14T15:26:56.066 INFO:tasks.workunit.client.0.smithi090.stderr:+ for t in "${!pids[@]}" 2024-04-14T15:26:56.066 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=94305 2024-04-14T15:26:56.066 INFO:tasks.workunit.client.0.smithi090.stderr:+ wait 94305 2024-04-14T15:26:56.066 INFO:tasks.workunit.client.0.smithi090.stderr:+ for t in "${!pids[@]}" 2024-04-14T15:26:56.067 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=94378 2024-04-14T15:26:56.067 INFO:tasks.workunit.client.0.smithi090.stderr:+ wait 94378 2024-04-14T15:26:56.067 INFO:tasks.workunit.client.0.smithi090.stderr:+ for t in "${!pids[@]}" 2024-04-14T15:26:56.067 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=93419 2024-04-14T15:26:56.067 INFO:tasks.workunit.client.0.smithi090.stderr:+ wait 93419 2024-04-14T15:26:56.067 INFO:tasks.workunit.client.0.smithi090.stderr:+ for t in "${!pids[@]}" 2024-04-14T15:26:56.067 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=94102 2024-04-14T15:26:56.067 INFO:tasks.workunit.client.0.smithi090.stderr:+ wait 94102 2024-04-14T15:26:56.067 INFO:tasks.workunit.client.0.smithi090.stderr:+ for t in "${!pids[@]}" 2024-04-14T15:26:56.067 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=93939 2024-04-14T15:26:56.067 INFO:tasks.workunit.client.0.smithi090.stderr:+ wait 93939 2024-04-14T15:26:56.067 INFO:tasks.workunit.client.0.smithi090.stderr:+ for t in "${!pids[@]}" 2024-04-14T15:26:56.067 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=93292 2024-04-14T15:26:56.067 INFO:tasks.workunit.client.0.smithi090.stderr:+ wait 93292 2024-04-14T15:26:56.067 INFO:tasks.workunit.client.0.smithi090.stderr:+ for t in "${!pids[@]}" 2024-04-14T15:26:56.068 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=93355 2024-04-14T15:26:56.068 INFO:tasks.workunit.client.0.smithi090.stderr:+ wait 93355 2024-04-14T15:26:56.068 INFO:tasks.workunit.client.0.smithi090.stderr:+ for t in "${!pids[@]}" 2024-04-14T15:26:56.068 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=93960 2024-04-14T15:26:56.068 INFO:tasks.workunit.client.0.smithi090.stderr:+ wait 93960 2024-04-14T15:26:56.068 INFO:tasks.workunit.client.0.smithi090.stderr:+ for t in "${!pids[@]}" 2024-04-14T15:26:56.068 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=93283 2024-04-14T15:26:56.068 INFO:tasks.workunit.client.0.smithi090.stderr:+ wait 93283 2024-04-14T15:26:56.068 INFO:tasks.workunit.client.0.smithi090.stderr:+ for t in "${!pids[@]}" 2024-04-14T15:26:56.068 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=93304 2024-04-14T15:26:56.068 INFO:tasks.workunit.client.0.smithi090.stderr:+ wait 93304 2024-04-14T15:26:56.069 INFO:tasks.workunit.client.0.smithi090.stderr:+ for t in "${!pids[@]}" 2024-04-14T15:26:56.069 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=94468 2024-04-14T15:26:56.070 INFO:tasks.workunit.client.0.smithi090.stderr:+ wait 94468 2024-04-14T15:26:56.070 INFO:tasks.workunit.client.0.smithi090.stderr:+ for t in "${!pids[@]}" 2024-04-14T15:26:56.070 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=93590 2024-04-14T15:26:56.070 INFO:tasks.workunit.client.0.smithi090.stderr:+ wait 93590 2024-04-14T15:26:56.070 INFO:tasks.workunit.client.0.smithi090.stderr:+ for t in "${!pids[@]}" 2024-04-14T15:26:56.071 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=94615 2024-04-14T15:26:56.071 INFO:tasks.workunit.client.0.smithi090.stderr:+ wait 94615 2024-04-14T15:26:56.071 INFO:tasks.workunit.client.0.smithi090.stderr:+ for t in "${!pids[@]}" 2024-04-14T15:26:56.071 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=93703 2024-04-14T15:26:56.071 INFO:tasks.workunit.client.0.smithi090.stderr:+ wait 93703 2024-04-14T15:26:56.071 INFO:tasks.workunit.client.0.smithi090.stderr:+ for t in "${!pids[@]}" 2024-04-14T15:26:56.072 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=94252 2024-04-14T15:26:56.072 INFO:tasks.workunit.client.0.smithi090.stderr:+ wait 94252 2024-04-14T15:26:56.072 INFO:tasks.workunit.client.0.smithi090.stderr:+ for t in "${!pids[@]}" 2024-04-14T15:26:56.072 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=94701 2024-04-14T15:26:56.072 INFO:tasks.workunit.client.0.smithi090.stderr:+ wait 94701 2024-04-14T15:26:56.072 INFO:tasks.workunit.client.0.smithi090.stderr:+ for t in "${!pids[@]}" 2024-04-14T15:26:56.072 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=93615 2024-04-14T15:26:56.072 INFO:tasks.workunit.client.0.smithi090.stderr:+ wait 93615 2024-04-14T15:26:56.073 INFO:tasks.workunit.client.0.smithi090.stderr:+ for t in "${!pids[@]}" 2024-04-14T15:26:56.073 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=94006 2024-04-14T15:26:56.073 INFO:tasks.workunit.client.0.smithi090.stderr:+ wait 94006 2024-04-14T15:26:56.073 INFO:tasks.workunit.client.0.smithi090.stderr:+ for t in "${!pids[@]}" 2024-04-14T15:26:56.073 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=94456 2024-04-14T15:26:56.073 INFO:tasks.workunit.client.0.smithi090.stderr:+ wait 94456 2024-04-14T15:26:56.073 INFO:tasks.workunit.client.0.smithi090.stderr:+ for t in "${!pids[@]}" 2024-04-14T15:26:56.073 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=94168 2024-04-14T15:26:56.073 INFO:tasks.workunit.client.0.smithi090.stderr:+ wait 94168 2024-04-14T15:26:56.073 INFO:tasks.workunit.client.0.smithi090.stderr:+ for t in "${!pids[@]}" 2024-04-14T15:26:56.074 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=94527 2024-04-14T15:26:56.074 INFO:tasks.workunit.client.0.smithi090.stderr:+ wait 94527 2024-04-14T15:26:56.074 INFO:tasks.workunit.client.0.smithi090.stderr:+ for t in "${!pids[@]}" 2024-04-14T15:26:56.074 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=93910 2024-04-14T15:26:56.075 INFO:tasks.workunit.client.0.smithi090.stderr:+ wait 93910 2024-04-14T15:26:56.075 INFO:tasks.workunit.client.0.smithi090.stderr:+ for t in "${!pids[@]}" 2024-04-14T15:26:56.075 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=93499 2024-04-14T15:26:56.075 INFO:tasks.workunit.client.0.smithi090.stderr:+ wait 93499 2024-04-14T15:26:56.075 INFO:tasks.workunit.client.0.smithi090.stderr:+ for t in "${!pids[@]}" 2024-04-14T15:26:56.075 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=94680 2024-04-14T15:26:56.075 INFO:tasks.workunit.client.0.smithi090.stderr:+ wait 94680 2024-04-14T15:26:56.077 INFO:tasks.workunit.client.0.smithi090.stderr:+ for t in "${!pids[@]}" 2024-04-14T15:26:56.077 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=93788 2024-04-14T15:26:56.078 INFO:tasks.workunit.client.0.smithi090.stderr:+ wait 93788 2024-04-14T15:26:56.078 INFO:tasks.workunit.client.0.smithi090.stderr:+ for t in "${!pids[@]}" 2024-04-14T15:26:56.078 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=93276 2024-04-14T15:26:56.078 INFO:tasks.workunit.client.0.smithi090.stderr:+ wait 93276 2024-04-14T15:26:56.079 INFO:tasks.workunit.client.0.smithi090.stderr:+ for t in "${!pids[@]}" 2024-04-14T15:26:56.079 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=93336 2024-04-14T15:26:56.079 INFO:tasks.workunit.client.0.smithi090.stderr:+ wait 93336 2024-04-14T15:26:56.079 INFO:tasks.workunit.client.0.smithi090.stderr:+ for t in "${!pids[@]}" 2024-04-14T15:26:56.079 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=93753 2024-04-14T15:26:56.079 INFO:tasks.workunit.client.0.smithi090.stderr:+ wait 93753 2024-04-14T15:26:56.080 INFO:tasks.workunit.client.0.smithi090.stderr:+ for t in "${!pids[@]}" 2024-04-14T15:26:56.080 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=93377 2024-04-14T15:26:56.080 INFO:tasks.workunit.client.0.smithi090.stderr:+ wait 93377 2024-04-14T15:26:56.080 INFO:tasks.workunit.client.0.smithi090.stderr:+ for t in "${!pids[@]}" 2024-04-14T15:26:56.080 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=93813 2024-04-14T15:26:56.080 INFO:tasks.workunit.client.0.smithi090.stderr:+ wait 93813 2024-04-14T15:26:56.080 INFO:tasks.workunit.client.0.smithi090.stderr:+ for t in "${!pids[@]}" 2024-04-14T15:26:56.081 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=93841 2024-04-14T15:26:56.081 INFO:tasks.workunit.client.0.smithi090.stderr:+ wait 93841 2024-04-14T15:26:56.081 INFO:tasks.workunit.client.0.smithi090.stderr:+ for t in "${!pids[@]}" 2024-04-14T15:26:56.081 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=94418 2024-04-14T15:26:56.081 INFO:tasks.workunit.client.0.smithi090.stderr:+ wait 94418 2024-04-14T15:26:56.081 INFO:tasks.workunit.client.0.smithi090.stderr:+ for t in "${!pids[@]}" 2024-04-14T15:26:56.081 INFO:tasks.workunit.client.0.smithi090.stderr:+ pid=94756 2024-04-14T15:26:56.081 INFO:tasks.workunit.client.0.smithi090.stderr:+ wait 94756 2024-04-14T15:26:56.081 INFO:tasks.workunit.client.0.smithi090.stderr:+ exit 0 2024-04-14T15:26:56.082 INFO:tasks.workunit.client.0.smithi090.stderr:+ cleanup 2024-04-14T15:26:56.082 INFO:tasks.workunit.client.0.smithi090.stderr:+ pkill -P 93270 2024-04-14T15:26:56.089 INFO:tasks.workunit.client.0.smithi090.stderr:+ true 2024-04-14T15:26:56.090 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-14T15:26:56.090 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-14T15:26:56.114 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1 2024-04-14T15:26:56.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:56.155 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:56.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.1' 2024-04-14T15:26:56.159 INFO:tasks.workunit:Stopping ['rados/test.sh'] on client.0... 2024-04-14T15:26:56.159 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-14T15:26:56.257 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5 2024-04-14T15:26:56.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:56.281 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:56.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.5' 2024-04-14T15:26:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:56 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi090-93652-111"}]': finished 2024-04-14T15:26:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:56 smithi092 ceph-mon[33966]: osdmap e851: 8 total, 8 up, 7 in 2024-04-14T15:26:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:56 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:26:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:56 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:26:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:56 smithi092 ceph-mon[33966]: pgmap v1271: 184 pgs: 184 active+clean; 582 KiB data, 728 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:26:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:56 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:26:56.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:56 smithi092 ceph-mon[33966]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi090-93652-111"}]': finished 2024-04-14T15:26:56.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:56 smithi092 ceph-mon[33966]: osdmap e852: 8 total, 8 up, 7 in 2024-04-14T15:26:56.382 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2 2024-04-14T15:26:56.406 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:56.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:56.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.2' 2024-04-14T15:26:56.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:56 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi090-93652-111"}]': finished 2024-04-14T15:26:56.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:56 smithi090 ceph-mon[41670]: osdmap e851: 8 total, 8 up, 7 in 2024-04-14T15:26:56.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:56 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:26:56.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:56 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:26:56.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:56 smithi090 ceph-mon[41670]: pgmap v1271: 184 pgs: 184 active+clean; 582 KiB data, 728 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:26:56.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:56 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:26:56.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:56 smithi090 ceph-mon[41670]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi090-93652-111"}]': finished 2024-04-14T15:26:56.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:56 smithi090 ceph-mon[41670]: osdmap e852: 8 total, 8 up, 7 in 2024-04-14T15:26:56.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:56 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi090-93652-111"}]': finished 2024-04-14T15:26:56.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:56 smithi090 ceph-mon[30399]: osdmap e851: 8 total, 8 up, 7 in 2024-04-14T15:26:56.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:56 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2125385135' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:26:56.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:56 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi090-93652-111"}]: dispatch 2024-04-14T15:26:56.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:56 smithi090 ceph-mon[30399]: pgmap v1271: 184 pgs: 184 active+clean; 582 KiB data, 728 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:26:56.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:56 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:26:56.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:56 smithi090 ceph-mon[30399]: from='client.49576 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi090-93652-111"}]': finished 2024-04-14T15:26:56.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:56 smithi090 ceph-mon[30399]: osdmap e852: 8 total, 8 up, 7 in 2024-04-14T15:26:56.509 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3 2024-04-14T15:26:56.516 DEBUG:teuthology.parallel:result is None 2024-04-14T15:26:56.517 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-04-14T15:26:56.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:56.533 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-04-14T15:26:56.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.3' 2024-04-14T15:26:56.558 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-04-14T15:26:56.559 DEBUG:teuthology.orchestra.run.smithi090:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-04-14T15:26:56.634 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7 2024-04-14T15:26:56.653 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-04-14T15:26:56.653 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-04-14T15:26:56.664 INFO:tasks.thrashosds:joining thrashosds 2024-04-14T15:26:56.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:26:56.665 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-04-14T15:26:56.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd.7' 2024-04-14T15:26:57.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:57 smithi092 ceph-mon[33966]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:26:57.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:57 smithi090 ceph-mon[30399]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:26:57.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:57 smithi090 ceph-mon[41670]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-14T15:26:58.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.157+0000 7f7037570700 1 -- 172.21.15.90:0/3255869788 >> v1:172.21.15.90:6789/0 conn(0x7f703010a620 legacy=0x7f703010ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:58.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.157+0000 7f7037570700 1 -- 172.21.15.90:0/3255869788 shutdown_connections 2024-04-14T15:26:58.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.157+0000 7f7037570700 1 -- 172.21.15.90:0/3255869788 >> 172.21.15.90:0/3255869788 conn(0x7f70300fd280 msgr2=0x7f70300ff6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:26:58.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.158+0000 7f7037570700 1 -- 172.21.15.90:0/3255869788 shutdown_connections 2024-04-14T15:26:58.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.158+0000 7f7037570700 1 -- 172.21.15.90:0/3255869788 wait complete. 2024-04-14T15:26:58.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.158+0000 7f7037570700 1 librados: starting msgr at 2024-04-14T15:26:58.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.158+0000 7f7037570700 1 librados: starting objecter 2024-04-14T15:26:58.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.158+0000 7f7037570700 1 Processor -- start 2024-04-14T15:26:58.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.158+0000 7f7037570700 1 -- start start 2024-04-14T15:26:58.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.158+0000 7f7037570700 1 librados: setting wanted keys 2024-04-14T15:26:58.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.158+0000 7f7037570700 1 librados: calling monclient init 2024-04-14T15:26:58.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.158+0000 7f7037570700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f70300ff3c0 con 0x7f7030107770 2024-04-14T15:26:58.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.158+0000 7f7037570700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f70300ff5d0 con 0x7f703010e1f0 2024-04-14T15:26:58.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.158+0000 7f7037570700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f70301ac210 con 0x7f703010a620 2024-04-14T15:26:58.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.158+0000 7f703530c700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f7030107770 0x7f70300fd730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:60332/0 (socket says 172.21.15.90:60332) 2024-04-14T15:26:58.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.158+0000 7f703530c700 1 -- 172.21.15.90:0/3453883900 learned_addr learned my addr 172.21.15.90:0/3453883900 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:26:58.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.158+0000 7f7034b0b700 1 --1- 172.21.15.90:0/3453883900 >> v1:172.21.15.90:6790/0 conn(0x7f703010a620 0x7f70300fde40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:56776/0 (socket says 172.21.15.90:56776) 2024-04-14T15:26:58.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.159+0000 7f702e7fc700 10 client.?.objecter ms_handle_connect 0x7f7030107770 2024-04-14T15:26:58.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.159+0000 7f702e7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:26:58.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.159+0000 7f702e7fc700 10 client.?.objecter ms_handle_connect 0x7f703010a620 2024-04-14T15:26:58.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.159+0000 7f702e7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:26:58.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.159+0000 7f702e7fc700 10 client.?.objecter ms_handle_connect 0x7f703010e1f0 2024-04-14T15:26:58.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.159+0000 7f702e7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:26:58.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.159+0000 7f702e7fc700 1 -- 172.21.15.90:0/3453883900 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4119991779 0 0) 0x7f70301ac210 con 0x7f703010a620 2024-04-14T15:26:58.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.159+0000 7f702e7fc700 1 -- 172.21.15.90:0/3453883900 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f70180035b0 con 0x7f703010a620 2024-04-14T15:26:58.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.159+0000 7f702e7fc700 1 -- 172.21.15.90:0/3453883900 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 851763199 0 0) 0x7f70300ff3c0 con 0x7f7030107770 2024-04-14T15:26:58.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.159+0000 7f702e7fc700 1 -- 172.21.15.90:0/3453883900 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f70301ac210 con 0x7f7030107770 2024-04-14T15:26:58.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.159+0000 7f702e7fc700 1 -- 172.21.15.90:0/3453883900 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3951620613 0 0) 0x7f70300ff5d0 con 0x7f703010e1f0 2024-04-14T15:26:58.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.159+0000 7f702e7fc700 1 -- 172.21.15.90:0/3453883900 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f70300ff3c0 con 0x7f703010e1f0 2024-04-14T15:26:58.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.159+0000 7f702e7fc700 1 -- 172.21.15.90:0/3453883900 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1333391170 0 0) 0x7f70180035b0 con 0x7f703010a620 2024-04-14T15:26:58.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.159+0000 7f702e7fc700 1 -- 172.21.15.90:0/3453883900 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f70300ff5d0 con 0x7f703010a620 2024-04-14T15:26:58.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.159+0000 7f702e7fc700 1 -- 172.21.15.90:0/3453883900 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f7024004500 con 0x7f703010a620 2024-04-14T15:26:58.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.159+0000 7f702e7fc700 1 -- 172.21.15.90:0/3453883900 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 613190827 0 0) 0x7f70301ac210 con 0x7f7030107770 2024-04-14T15:26:58.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.159+0000 7f702e7fc700 1 -- 172.21.15.90:0/3453883900 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f70180035b0 con 0x7f7030107770 2024-04-14T15:26:58.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.159+0000 7f702e7fc700 1 -- 172.21.15.90:0/3453883900 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f7028001c10 con 0x7f7030107770 2024-04-14T15:26:58.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.159+0000 7f702e7fc700 1 -- 172.21.15.90:0/3453883900 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2927500805 0 0) 0x7f70300ff3c0 con 0x7f703010e1f0 2024-04-14T15:26:58.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.159+0000 7f702e7fc700 1 -- 172.21.15.90:0/3453883900 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f70301ac210 con 0x7f703010e1f0 2024-04-14T15:26:58.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.159+0000 7f702e7fc700 1 -- 172.21.15.90:0/3453883900 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f7020001770 con 0x7f703010e1f0 2024-04-14T15:26:58.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.159+0000 7f702e7fc700 1 -- 172.21.15.90:0/3453883900 <== mon.2 v1:172.21.15.90:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2515649719 0 0) 0x7f70300ff5d0 con 0x7f703010a620 2024-04-14T15:26:58.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.159+0000 7f702e7fc700 1 -- 172.21.15.90:0/3453883900 >> v1:172.21.15.92:6789/0 conn(0x7f703010e1f0 legacy=0x7f70301a8ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:58.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.159+0000 7f702e7fc700 1 -- 172.21.15.90:0/3453883900 >> v1:172.21.15.90:6789/0 conn(0x7f7030107770 legacy=0x7f70300fd730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:58.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.159+0000 7f702e7fc700 1 -- 172.21.15.90:0/3453883900 --> v1:172.21.15.90:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70301ad3f0 con 0x7f703010a620 2024-04-14T15:26:58.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.159+0000 7f7037570700 1 -- 172.21.15.90:0/3453883900 --> v1:172.21.15.90:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f70301ac440 con 0x7f703010a620 2024-04-14T15:26:58.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.159+0000 7f7037570700 10 client.48107.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:26:58.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.159+0000 7f7037570700 1 -- 172.21.15.90:0/3453883900 --> v1:172.21.15.90:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f70301ac900 con 0x7f703010a620 2024-04-14T15:26:58.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.159+0000 7f7037570700 1 librados: init done 2024-04-14T15:26:58.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.160+0000 7f702e7fc700 1 -- 172.21.15.90:0/3453883900 <== mon.2 v1:172.21.15.90:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f7024001b30 con 0x7f703010a620 2024-04-14T15:26:58.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.160+0000 7f702e7fc700 1 -- 172.21.15.90:0/3453883900 <== mon.2 v1:172.21.15.90:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f7024004cb0 con 0x7f703010a620 2024-04-14T15:26:58.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.160+0000 7f7037570700 1 -- 172.21.15.90:0/3453883900 --> v1:172.21.15.90:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f701c005130 con 0x7f703010a620 2024-04-14T15:26:58.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.161+0000 7f702e7fc700 1 -- 172.21.15.90:0/3453883900 <== mon.2 v1:172.21.15.90:6790/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f70240038b0 con 0x7f703010a620 2024-04-14T15:26:58.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.162+0000 7f702e7fc700 1 -- 172.21.15.90:0/3453883900 <== mon.2 v1:172.21.15.90:6790/0 8 ==== osd_map(853..853 src has 467..853) ==== 7517+0+0 (unknown 833938570 0 0) 0x7f702408a7c0 con 0x7f703010a620 2024-04-14T15:26:58.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.162+0000 7f702e7fc700 10 client.48107.objecter ms_dispatch 0x7f7030001550 osd_map(853..853 src has 467..853) 2024-04-14T15:26:58.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.162+0000 7f702e7fc700 3 client.48107.objecter handle_osd_map got epochs [853,853] > 0 2024-04-14T15:26:58.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.162+0000 7f702e7fc700 3 client.48107.objecter handle_osd_map decoding full epoch 853 2024-04-14T15:26:58.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.162+0000 7f702e7fc700 20 client.48107.objecter dump_active .. 0 homeless 2024-04-14T15:26:58.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.162+0000 7f702e7fc700 1 -- 172.21.15.90:0/3453883900 <== mon.2 v1:172.21.15.90:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v853) ==== 54+0+16 (unknown 405961390 0 2420000192) 0x7f702408ac20 con 0x7f703010a620 2024-04-14T15:26:58.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.162+0000 7f702e7fc700 10 client.48107.objecter ms_handle_connect 0x7f7018071350 2024-04-14T15:26:58.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.162+0000 7f7037570700 10 client.48107.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:26:58.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.162+0000 7f7037570700 15 client.48107.objecter _session_command_op_assign -1 1 2024-04-14T15:26:58.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.162+0000 7f7037570700 20 client.48107.objecter _get_session s=0x7f701c005410 osd=0 3 2024-04-14T15:26:58.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.162+0000 7f7037570700 20 client.48107.objecter put_session s=0x7f701c005410 osd=0 3 2024-04-14T15:26:58.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.162+0000 7f7037570700 20 client.48107.objecter _get_session s=0x7f701c005410 osd=0 3 2024-04-14T15:26:58.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.162+0000 7f7037570700 15 client.48107.objecter _session_command_op_remove -1 1 2024-04-14T15:26:58.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.162+0000 7f7037570700 20 client.48107.objecter get_session s=0x7f701c005410 osd=0 3 2024-04-14T15:26:58.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.162+0000 7f7037570700 15 client.48107.objecter _session_command_op_assign 0 1 2024-04-14T15:26:58.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.162+0000 7f7037570700 20 client.48107.objecter put_session s=0x7f701c005410 osd=0 4 2024-04-14T15:26:58.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.162+0000 7f7037570700 10 client.48107.objecter _send_command 1 2024-04-14T15:26:58.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.162+0000 7f7037570700 1 -- 172.21.15.90:0/3453883900 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f701c004b80 con 0x7f701c004760 2024-04-14T15:26:58.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.162+0000 7f702e7fc700 10 client.48107.objecter ms_handle_connect 0x7f701c004760 2024-04-14T15:26:58.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.163+0000 7f702e7fc700 1 -- 172.21.15.90:0/3453883900 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f701c004b80 con 0x7f701c004760 2024-04-14T15:26:58.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.163+0000 7f702e7fc700 10 client.48107.objecter ms_dispatch 0x7f7030001550 command_reply(tid 1: 0 ) 2024-04-14T15:26:58.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.163+0000 7f702e7fc700 10 client.48107.objecter _finish_command 1 = system:0 2024-04-14T15:26:58.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.163+0000 7f702e7fc700 20 client.48107.objecter put_session s=0x7f701c005410 osd=0 4 2024-04-14T15:26:58.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.163+0000 7f702e7fc700 15 client.48107.objecter _session_command_op_remove 0 1 2024-04-14T15:26:58.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.180+0000 7f7037570700 10 client.48107.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:26:58.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.180+0000 7f7037570700 15 client.48107.objecter _session_command_op_assign -1 2 2024-04-14T15:26:58.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.180+0000 7f7037570700 20 client.48107.objecter _get_session s=0x7f701c005410 osd=0 3 2024-04-14T15:26:58.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.180+0000 7f7037570700 20 client.48107.objecter put_session s=0x7f701c005410 osd=0 3 2024-04-14T15:26:58.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.180+0000 7f7037570700 20 client.48107.objecter _get_session s=0x7f701c005410 osd=0 3 2024-04-14T15:26:58.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.180+0000 7f7037570700 15 client.48107.objecter _session_command_op_remove -1 2 2024-04-14T15:26:58.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.180+0000 7f7037570700 20 client.48107.objecter get_session s=0x7f701c005410 osd=0 3 2024-04-14T15:26:58.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.180+0000 7f7037570700 15 client.48107.objecter _session_command_op_assign 0 2 2024-04-14T15:26:58.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.180+0000 7f7037570700 20 client.48107.objecter put_session s=0x7f701c005410 osd=0 4 2024-04-14T15:26:58.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.180+0000 7f7037570700 10 client.48107.objecter _send_command 2 2024-04-14T15:26:58.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.180+0000 7f7037570700 1 -- 172.21.15.90:0/3453883900 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f701c007a20 con 0x7f701c004760 2024-04-14T15:26:58.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.181+0000 7f702e7fc700 1 -- 172.21.15.90:0/3453883900 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f701c007a20 con 0x7f701c004760 2024-04-14T15:26:58.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.181+0000 7f702e7fc700 10 client.48107.objecter ms_dispatch 0x7f7030001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:26:58.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.181+0000 7f702e7fc700 10 client.48107.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:26:58.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.181+0000 7f702e7fc700 20 client.48107.objecter put_session s=0x7f701c005410 osd=0 4 2024-04-14T15:26:58.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.181+0000 7f702e7fc700 15 client.48107.objecter _session_command_op_remove 0 2 2024-04-14T15:26:58.182 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-14T15:26:58.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.182+0000 7f7037570700 10 client.48107.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:26:58.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.182+0000 7f7037570700 15 client.48107.objecter _session_command_op_assign -1 3 2024-04-14T15:26:58.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.182+0000 7f7037570700 20 client.48107.objecter _get_session s=0x7f701c004140 osd=1 3 2024-04-14T15:26:58.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.182+0000 7f7037570700 20 client.48107.objecter put_session s=0x7f701c004140 osd=1 3 2024-04-14T15:26:58.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.182+0000 7f7037570700 20 client.48107.objecter _get_session s=0x7f701c004140 osd=1 3 2024-04-14T15:26:58.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.182+0000 7f7037570700 15 client.48107.objecter _session_command_op_remove -1 3 2024-04-14T15:26:58.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.182+0000 7f7037570700 20 client.48107.objecter get_session s=0x7f701c004140 osd=1 3 2024-04-14T15:26:58.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.182+0000 7f7037570700 15 client.48107.objecter _session_command_op_assign 1 3 2024-04-14T15:26:58.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.182+0000 7f7037570700 20 client.48107.objecter put_session s=0x7f701c004140 osd=1 4 2024-04-14T15:26:58.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.182+0000 7f7037570700 10 client.48107.objecter _send_command 3 2024-04-14T15:26:58.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.182+0000 7f7037570700 1 -- 172.21.15.90:0/3453883900 --> v1:172.21.15.90:6805/4170361699 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f701c0021b0 con 0x7f701c0042b0 2024-04-14T15:26:58.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.182+0000 7f702e7fc700 10 client.48107.objecter ms_handle_connect 0x7f701c0042b0 2024-04-14T15:26:58.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.183+0000 7f702e7fc700 1 -- 172.21.15.90:0/3453883900 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f701c0021b0 con 0x7f701c0042b0 2024-04-14T15:26:58.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.183+0000 7f702e7fc700 10 client.48107.objecter ms_dispatch 0x7f7030001550 command_reply(tid 3: 0 ) 2024-04-14T15:26:58.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.183+0000 7f702e7fc700 10 client.48107.objecter _finish_command 3 = system:0 2024-04-14T15:26:58.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.183+0000 7f702e7fc700 20 client.48107.objecter put_session s=0x7f701c004140 osd=1 4 2024-04-14T15:26:58.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.183+0000 7f702e7fc700 15 client.48107.objecter _session_command_op_remove 1 3 2024-04-14T15:26:58.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.199+0000 7f7037570700 10 client.48107.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:26:58.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.199+0000 7f7037570700 15 client.48107.objecter _session_command_op_assign -1 4 2024-04-14T15:26:58.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.199+0000 7f7037570700 20 client.48107.objecter _get_session s=0x7f701c004140 osd=1 3 2024-04-14T15:26:58.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.199+0000 7f7037570700 20 client.48107.objecter put_session s=0x7f701c004140 osd=1 3 2024-04-14T15:26:58.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.199+0000 7f7037570700 20 client.48107.objecter _get_session s=0x7f701c004140 osd=1 3 2024-04-14T15:26:58.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.199+0000 7f7037570700 15 client.48107.objecter _session_command_op_remove -1 4 2024-04-14T15:26:58.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.199+0000 7f7037570700 20 client.48107.objecter get_session s=0x7f701c004140 osd=1 3 2024-04-14T15:26:58.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.199+0000 7f7037570700 15 client.48107.objecter _session_command_op_assign 1 4 2024-04-14T15:26:58.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.199+0000 7f7037570700 20 client.48107.objecter put_session s=0x7f701c004140 osd=1 4 2024-04-14T15:26:58.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.199+0000 7f7037570700 10 client.48107.objecter _send_command 4 2024-04-14T15:26:58.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.199+0000 7f7037570700 1 -- 172.21.15.90:0/3453883900 --> v1:172.21.15.90:6805/4170361699 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f701c001630 con 0x7f701c0042b0 2024-04-14T15:26:58.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.201+0000 7f702e7fc700 1 -- 172.21.15.90:0/3453883900 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f701c001630 con 0x7f701c0042b0 2024-04-14T15:26:58.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.201+0000 7f702e7fc700 10 client.48107.objecter ms_dispatch 0x7f7030001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:26:58.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.201+0000 7f702e7fc700 10 client.48107.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:26:58.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.201+0000 7f702e7fc700 20 client.48107.objecter put_session s=0x7f701c004140 osd=1 4 2024-04-14T15:26:58.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.201+0000 7f702e7fc700 15 client.48107.objecter _session_command_op_remove 1 4 2024-04-14T15:26:58.202 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-14T15:26:58.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.201+0000 7f7037570700 10 client.48107.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:26:58.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.201+0000 7f7037570700 15 client.48107.objecter _session_command_op_assign -1 5 2024-04-14T15:26:58.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.201+0000 7f7037570700 20 client.48107.objecter _get_session s=0x7f701c0083a0 osd=2 3 2024-04-14T15:26:58.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.201+0000 7f7037570700 20 client.48107.objecter put_session s=0x7f701c0083a0 osd=2 3 2024-04-14T15:26:58.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.201+0000 7f7037570700 20 client.48107.objecter _get_session s=0x7f701c0083a0 osd=2 3 2024-04-14T15:26:58.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.201+0000 7f7037570700 15 client.48107.objecter _session_command_op_remove -1 5 2024-04-14T15:26:58.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.201+0000 7f7037570700 20 client.48107.objecter get_session s=0x7f701c0083a0 osd=2 3 2024-04-14T15:26:58.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.201+0000 7f7037570700 15 client.48107.objecter _session_command_op_assign 2 5 2024-04-14T15:26:58.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.201+0000 7f7037570700 20 client.48107.objecter put_session s=0x7f701c0083a0 osd=2 4 2024-04-14T15:26:58.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.201+0000 7f7037570700 10 client.48107.objecter _send_command 5 2024-04-14T15:26:58.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.201+0000 7f7037570700 1 -- 172.21.15.90:0/3453883900 --> v1:172.21.15.90:6809/1686184251 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f701c011110 con 0x7f701c00d600 2024-04-14T15:26:58.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.201+0000 7f702e7fc700 10 client.48107.objecter ms_handle_connect 0x7f701c00d600 2024-04-14T15:26:58.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.202+0000 7f702e7fc700 1 -- 172.21.15.90:0/3453883900 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f701c011110 con 0x7f701c00d600 2024-04-14T15:26:58.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.202+0000 7f702e7fc700 10 client.48107.objecter ms_dispatch 0x7f7030001550 command_reply(tid 5: 0 ) 2024-04-14T15:26:58.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.202+0000 7f702e7fc700 10 client.48107.objecter _finish_command 5 = system:0 2024-04-14T15:26:58.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.202+0000 7f702e7fc700 20 client.48107.objecter put_session s=0x7f701c0083a0 osd=2 4 2024-04-14T15:26:58.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.202+0000 7f702e7fc700 15 client.48107.objecter _session_command_op_remove 2 5 2024-04-14T15:26:58.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.219+0000 7f7037570700 10 client.48107.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:26:58.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.219+0000 7f7037570700 15 client.48107.objecter _session_command_op_assign -1 6 2024-04-14T15:26:58.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.219+0000 7f7037570700 20 client.48107.objecter _get_session s=0x7f701c0083a0 osd=2 3 2024-04-14T15:26:58.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.219+0000 7f7037570700 20 client.48107.objecter put_session s=0x7f701c0083a0 osd=2 3 2024-04-14T15:26:58.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.219+0000 7f7037570700 20 client.48107.objecter _get_session s=0x7f701c0083a0 osd=2 3 2024-04-14T15:26:58.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.219+0000 7f7037570700 15 client.48107.objecter _session_command_op_remove -1 6 2024-04-14T15:26:58.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.219+0000 7f7037570700 20 client.48107.objecter get_session s=0x7f701c0083a0 osd=2 3 2024-04-14T15:26:58.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.219+0000 7f7037570700 15 client.48107.objecter _session_command_op_assign 2 6 2024-04-14T15:26:58.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.219+0000 7f7037570700 20 client.48107.objecter put_session s=0x7f701c0083a0 osd=2 4 2024-04-14T15:26:58.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.219+0000 7f7037570700 10 client.48107.objecter _send_command 6 2024-04-14T15:26:58.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.219+0000 7f7037570700 1 -- 172.21.15.90:0/3453883900 --> v1:172.21.15.90:6809/1686184251 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f701c007ee0 con 0x7f701c00d600 2024-04-14T15:26:58.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.220+0000 7f702e7fc700 1 -- 172.21.15.90:0/3453883900 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f701c007ee0 con 0x7f701c00d600 2024-04-14T15:26:58.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.220+0000 7f702e7fc700 10 client.48107.objecter ms_dispatch 0x7f7030001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:26:58.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.220+0000 7f702e7fc700 10 client.48107.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:26:58.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.220+0000 7f702e7fc700 20 client.48107.objecter put_session s=0x7f701c0083a0 osd=2 4 2024-04-14T15:26:58.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.220+0000 7f702e7fc700 15 client.48107.objecter _session_command_op_remove 2 6 2024-04-14T15:26:58.221 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-14T15:26:58.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.221+0000 7f7037570700 10 client.48107.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:26:58.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.221+0000 7f7037570700 15 client.48107.objecter _session_command_op_assign -1 7 2024-04-14T15:26:58.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.221+0000 7f7037570700 20 client.48107.objecter _get_session s=0x7f701c00cc50 osd=3 3 2024-04-14T15:26:58.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.221+0000 7f7037570700 20 client.48107.objecter put_session s=0x7f701c00cc50 osd=3 3 2024-04-14T15:26:58.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.221+0000 7f7037570700 20 client.48107.objecter _get_session s=0x7f701c00cc50 osd=3 3 2024-04-14T15:26:58.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.221+0000 7f7037570700 15 client.48107.objecter _session_command_op_remove -1 7 2024-04-14T15:26:58.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.221+0000 7f7037570700 20 client.48107.objecter get_session s=0x7f701c00cc50 osd=3 3 2024-04-14T15:26:58.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.221+0000 7f7037570700 15 client.48107.objecter _session_command_op_assign 3 7 2024-04-14T15:26:58.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.221+0000 7f7037570700 20 client.48107.objecter put_session s=0x7f701c00cc50 osd=3 4 2024-04-14T15:26:58.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.221+0000 7f7037570700 10 client.48107.objecter _send_command 7 2024-04-14T15:26:58.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.221+0000 7f7037570700 1 -- 172.21.15.90:0/3453883900 --> v1:172.21.15.90:6813/2270166774 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f701c015f50 con 0x7f701c012440 2024-04-14T15:26:58.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.221+0000 7f702e7fc700 10 client.48107.objecter ms_handle_connect 0x7f701c012440 2024-04-14T15:26:58.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.221+0000 7f702e7fc700 1 -- 172.21.15.90:0/3453883900 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f701c015f50 con 0x7f701c012440 2024-04-14T15:26:58.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.221+0000 7f702e7fc700 10 client.48107.objecter ms_dispatch 0x7f7030001550 command_reply(tid 7: 0 ) 2024-04-14T15:26:58.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.221+0000 7f702e7fc700 10 client.48107.objecter _finish_command 7 = system:0 2024-04-14T15:26:58.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.221+0000 7f702e7fc700 20 client.48107.objecter put_session s=0x7f701c00cc50 osd=3 4 2024-04-14T15:26:58.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.221+0000 7f702e7fc700 15 client.48107.objecter _session_command_op_remove 3 7 2024-04-14T15:26:58.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.239+0000 7f7037570700 10 client.48107.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:26:58.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.239+0000 7f7037570700 15 client.48107.objecter _session_command_op_assign -1 8 2024-04-14T15:26:58.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.239+0000 7f7037570700 20 client.48107.objecter _get_session s=0x7f701c00cc50 osd=3 3 2024-04-14T15:26:58.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.239+0000 7f7037570700 20 client.48107.objecter put_session s=0x7f701c00cc50 osd=3 3 2024-04-14T15:26:58.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.239+0000 7f7037570700 20 client.48107.objecter _get_session s=0x7f701c00cc50 osd=3 3 2024-04-14T15:26:58.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.239+0000 7f7037570700 15 client.48107.objecter _session_command_op_remove -1 8 2024-04-14T15:26:58.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.239+0000 7f7037570700 20 client.48107.objecter get_session s=0x7f701c00cc50 osd=3 3 2024-04-14T15:26:58.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.239+0000 7f7037570700 15 client.48107.objecter _session_command_op_assign 3 8 2024-04-14T15:26:58.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.239+0000 7f7037570700 20 client.48107.objecter put_session s=0x7f701c00cc50 osd=3 4 2024-04-14T15:26:58.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.239+0000 7f7037570700 10 client.48107.objecter _send_command 8 2024-04-14T15:26:58.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.239+0000 7f7037570700 1 -- 172.21.15.90:0/3453883900 --> v1:172.21.15.90:6813/2270166774 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f701c00c730 con 0x7f701c012440 2024-04-14T15:26:58.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.240+0000 7f702e7fc700 1 -- 172.21.15.90:0/3453883900 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f701c00c730 con 0x7f701c012440 2024-04-14T15:26:58.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.240+0000 7f702e7fc700 10 client.48107.objecter ms_dispatch 0x7f7030001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:26:58.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.240+0000 7f702e7fc700 10 client.48107.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:26:58.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.240+0000 7f702e7fc700 20 client.48107.objecter put_session s=0x7f701c00cc50 osd=3 4 2024-04-14T15:26:58.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.240+0000 7f702e7fc700 15 client.48107.objecter _session_command_op_remove 3 8 2024-04-14T15:26:58.241 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-14T15:26:58.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.240+0000 7f7037570700 10 client.48107.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:26:58.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.240+0000 7f7037570700 15 client.48107.objecter _session_command_op_assign -1 9 2024-04-14T15:26:58.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.240+0000 7f7037570700 20 client.48107.objecter _get_session s=0x7f701c00c980 osd=4 3 2024-04-14T15:26:58.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.240+0000 7f7037570700 20 client.48107.objecter put_session s=0x7f701c00c980 osd=4 3 2024-04-14T15:26:58.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.240+0000 7f7037570700 20 client.48107.objecter _get_session s=0x7f701c00c980 osd=4 3 2024-04-14T15:26:58.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.240+0000 7f7037570700 15 client.48107.objecter _session_command_op_remove -1 9 2024-04-14T15:26:58.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.240+0000 7f7037570700 20 client.48107.objecter get_session s=0x7f701c00c980 osd=4 3 2024-04-14T15:26:58.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.240+0000 7f7037570700 15 client.48107.objecter _session_command_op_assign 4 9 2024-04-14T15:26:58.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.240+0000 7f7037570700 20 client.48107.objecter put_session s=0x7f701c00c980 osd=4 4 2024-04-14T15:26:58.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.240+0000 7f7037570700 10 client.48107.objecter _send_command 9 2024-04-14T15:26:58.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.240+0000 7f7037570700 1 -- 172.21.15.90:0/3453883900 --> v1:172.21.15.92:6800/3887549733 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f701c01a830 con 0x7f701c011fa0 2024-04-14T15:26:58.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.241+0000 7f702e7fc700 10 client.48107.objecter ms_handle_connect 0x7f701c011fa0 2024-04-14T15:26:58.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.242+0000 7f702e7fc700 1 -- 172.21.15.90:0/3453883900 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f701c01a830 con 0x7f701c011fa0 2024-04-14T15:26:58.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.242+0000 7f702e7fc700 10 client.48107.objecter ms_dispatch 0x7f7030001550 command_reply(tid 9: 0 ) 2024-04-14T15:26:58.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.242+0000 7f702e7fc700 10 client.48107.objecter _finish_command 9 = system:0 2024-04-14T15:26:58.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.242+0000 7f702e7fc700 20 client.48107.objecter put_session s=0x7f701c00c980 osd=4 4 2024-04-14T15:26:58.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.242+0000 7f702e7fc700 15 client.48107.objecter _session_command_op_remove 4 9 2024-04-14T15:26:58.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.258+0000 7f7037570700 10 client.48107.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:26:58.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.258+0000 7f7037570700 15 client.48107.objecter _session_command_op_assign -1 10 2024-04-14T15:26:58.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.258+0000 7f7037570700 20 client.48107.objecter _get_session s=0x7f701c00c980 osd=4 3 2024-04-14T15:26:58.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.258+0000 7f7037570700 20 client.48107.objecter put_session s=0x7f701c00c980 osd=4 3 2024-04-14T15:26:58.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.258+0000 7f7037570700 20 client.48107.objecter _get_session s=0x7f701c00c980 osd=4 3 2024-04-14T15:26:58.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.258+0000 7f7037570700 15 client.48107.objecter _session_command_op_remove -1 10 2024-04-14T15:26:58.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.258+0000 7f7037570700 20 client.48107.objecter get_session s=0x7f701c00c980 osd=4 3 2024-04-14T15:26:58.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.258+0000 7f7037570700 15 client.48107.objecter _session_command_op_assign 4 10 2024-04-14T15:26:58.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.258+0000 7f7037570700 20 client.48107.objecter put_session s=0x7f701c00c980 osd=4 4 2024-04-14T15:26:58.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.258+0000 7f7037570700 10 client.48107.objecter _send_command 10 2024-04-14T15:26:58.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.258+0000 7f7037570700 1 -- 172.21.15.90:0/3453883900 --> v1:172.21.15.92:6800/3887549733 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f701c01c1e0 con 0x7f701c011fa0 2024-04-14T15:26:58.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.261+0000 7f702e7fc700 1 -- 172.21.15.90:0/3453883900 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f701c01c1e0 con 0x7f701c011fa0 2024-04-14T15:26:58.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.261+0000 7f702e7fc700 10 client.48107.objecter ms_dispatch 0x7f7030001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:26:58.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.261+0000 7f702e7fc700 10 client.48107.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:26:58.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.261+0000 7f702e7fc700 20 client.48107.objecter put_session s=0x7f701c00c980 osd=4 4 2024-04-14T15:26:58.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.261+0000 7f702e7fc700 15 client.48107.objecter _session_command_op_remove 4 10 2024-04-14T15:26:58.262 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-14T15:26:58.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.261+0000 7f7037570700 10 client.48107.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:26:58.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.261+0000 7f7037570700 15 client.48107.objecter _session_command_op_assign -1 11 2024-04-14T15:26:58.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.261+0000 7f7037570700 20 client.48107.objecter _get_session s=0x7f701c01c1e0 osd=5 3 2024-04-14T15:26:58.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.261+0000 7f7037570700 20 client.48107.objecter put_session s=0x7f701c01c1e0 osd=5 3 2024-04-14T15:26:58.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.261+0000 7f7037570700 20 client.48107.objecter _get_session s=0x7f701c01c1e0 osd=5 3 2024-04-14T15:26:58.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.261+0000 7f7037570700 15 client.48107.objecter _session_command_op_remove -1 11 2024-04-14T15:26:58.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.261+0000 7f7037570700 20 client.48107.objecter get_session s=0x7f701c01c1e0 osd=5 3 2024-04-14T15:26:58.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.261+0000 7f7037570700 15 client.48107.objecter _session_command_op_assign 5 11 2024-04-14T15:26:58.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.261+0000 7f7037570700 20 client.48107.objecter put_session s=0x7f701c01c1e0 osd=5 4 2024-04-14T15:26:58.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.261+0000 7f7037570700 10 client.48107.objecter _send_command 11 2024-04-14T15:26:58.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.261+0000 7f7037570700 1 -- 172.21.15.90:0/3453883900 --> v1:172.21.15.92:6804/3244561440 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f701c01ce00 con 0x7f701c01c900 2024-04-14T15:26:58.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.261+0000 7f702e7fc700 10 client.48107.objecter ms_handle_connect 0x7f701c01c900 2024-04-14T15:26:58.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.262+0000 7f702e7fc700 1 -- 172.21.15.90:0/3453883900 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f701c01ce00 con 0x7f701c01c900 2024-04-14T15:26:58.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.262+0000 7f702e7fc700 10 client.48107.objecter ms_dispatch 0x7f7030001550 command_reply(tid 11: 0 ) 2024-04-14T15:26:58.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.262+0000 7f702e7fc700 10 client.48107.objecter _finish_command 11 = system:0 2024-04-14T15:26:58.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.262+0000 7f702e7fc700 20 client.48107.objecter put_session s=0x7f701c01c1e0 osd=5 4 2024-04-14T15:26:58.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.262+0000 7f702e7fc700 15 client.48107.objecter _session_command_op_remove 5 11 2024-04-14T15:26:58.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.279+0000 7f7037570700 10 client.48107.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:26:58.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.279+0000 7f7037570700 15 client.48107.objecter _session_command_op_assign -1 12 2024-04-14T15:26:58.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.279+0000 7f7037570700 20 client.48107.objecter _get_session s=0x7f701c01c1e0 osd=5 3 2024-04-14T15:26:58.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.279+0000 7f7037570700 20 client.48107.objecter put_session s=0x7f701c01c1e0 osd=5 3 2024-04-14T15:26:58.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.279+0000 7f7037570700 20 client.48107.objecter _get_session s=0x7f701c01c1e0 osd=5 3 2024-04-14T15:26:58.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.279+0000 7f7037570700 15 client.48107.objecter _session_command_op_remove -1 12 2024-04-14T15:26:58.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.279+0000 7f7037570700 20 client.48107.objecter get_session s=0x7f701c01c1e0 osd=5 3 2024-04-14T15:26:58.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.279+0000 7f7037570700 15 client.48107.objecter _session_command_op_assign 5 12 2024-04-14T15:26:58.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.279+0000 7f7037570700 20 client.48107.objecter put_session s=0x7f701c01c1e0 osd=5 4 2024-04-14T15:26:58.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.279+0000 7f7037570700 10 client.48107.objecter _send_command 12 2024-04-14T15:26:58.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.279+0000 7f7037570700 1 -- 172.21.15.90:0/3453883900 --> v1:172.21.15.92:6804/3244561440 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f701c01d060 con 0x7f701c01c900 2024-04-14T15:26:58.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.281+0000 7f702e7fc700 1 -- 172.21.15.90:0/3453883900 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f701c01d060 con 0x7f701c01c900 2024-04-14T15:26:58.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.281+0000 7f702e7fc700 10 client.48107.objecter ms_dispatch 0x7f7030001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:26:58.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.281+0000 7f702e7fc700 10 client.48107.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:26:58.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.281+0000 7f702e7fc700 20 client.48107.objecter put_session s=0x7f701c01c1e0 osd=5 4 2024-04-14T15:26:58.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.281+0000 7f702e7fc700 15 client.48107.objecter _session_command_op_remove 5 12 2024-04-14T15:26:58.282 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-14T15:26:58.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.281+0000 7f7037570700 10 client.48107.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:26:58.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.281+0000 7f7037570700 15 client.48107.objecter _session_command_op_assign -1 13 2024-04-14T15:26:58.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.281+0000 7f7037570700 20 client.48107.objecter _get_session s=0x7f701c01ce00 osd=6 3 2024-04-14T15:26:58.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.281+0000 7f7037570700 20 client.48107.objecter put_session s=0x7f701c01ce00 osd=6 3 2024-04-14T15:26:58.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.281+0000 7f7037570700 20 client.48107.objecter _get_session s=0x7f701c01ce00 osd=6 3 2024-04-14T15:26:58.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.281+0000 7f7037570700 15 client.48107.objecter _session_command_op_remove -1 13 2024-04-14T15:26:58.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.281+0000 7f7037570700 20 client.48107.objecter get_session s=0x7f701c01ce00 osd=6 3 2024-04-14T15:26:58.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.281+0000 7f7037570700 15 client.48107.objecter _session_command_op_assign 6 13 2024-04-14T15:26:58.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.281+0000 7f7037570700 20 client.48107.objecter put_session s=0x7f701c01ce00 osd=6 4 2024-04-14T15:26:58.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.281+0000 7f7037570700 10 client.48107.objecter _send_command 13 2024-04-14T15:26:58.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.281+0000 7f7037570700 1 -- 172.21.15.90:0/3453883900 --> v1:172.21.15.92:6808/636125963 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f701c023400 con 0x7f701c01f8f0 2024-04-14T15:26:58.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.282+0000 7f702e7fc700 10 client.48107.objecter ms_handle_connect 0x7f701c01f8f0 2024-04-14T15:26:58.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.283+0000 7f702e7fc700 1 -- 172.21.15.90:0/3453883900 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f701c023400 con 0x7f701c01f8f0 2024-04-14T15:26:58.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.283+0000 7f702e7fc700 10 client.48107.objecter ms_dispatch 0x7f7030001550 command_reply(tid 13: 0 ) 2024-04-14T15:26:58.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.283+0000 7f702e7fc700 10 client.48107.objecter _finish_command 13 = system:0 2024-04-14T15:26:58.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.283+0000 7f702e7fc700 20 client.48107.objecter put_session s=0x7f701c01ce00 osd=6 4 2024-04-14T15:26:58.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.283+0000 7f702e7fc700 15 client.48107.objecter _session_command_op_remove 6 13 2024-04-14T15:26:58.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.300+0000 7f7037570700 10 client.48107.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:26:58.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.300+0000 7f7037570700 15 client.48107.objecter _session_command_op_assign -1 14 2024-04-14T15:26:58.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.300+0000 7f7037570700 20 client.48107.objecter _get_session s=0x7f701c01ce00 osd=6 3 2024-04-14T15:26:58.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.300+0000 7f7037570700 20 client.48107.objecter put_session s=0x7f701c01ce00 osd=6 3 2024-04-14T15:26:58.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.300+0000 7f7037570700 20 client.48107.objecter _get_session s=0x7f701c01ce00 osd=6 3 2024-04-14T15:26:58.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.300+0000 7f7037570700 15 client.48107.objecter _session_command_op_remove -1 14 2024-04-14T15:26:58.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.300+0000 7f7037570700 20 client.48107.objecter get_session s=0x7f701c01ce00 osd=6 3 2024-04-14T15:26:58.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.300+0000 7f7037570700 15 client.48107.objecter _session_command_op_assign 6 14 2024-04-14T15:26:58.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.300+0000 7f7037570700 20 client.48107.objecter put_session s=0x7f701c01ce00 osd=6 4 2024-04-14T15:26:58.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.300+0000 7f7037570700 10 client.48107.objecter _send_command 14 2024-04-14T15:26:58.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.300+0000 7f7037570700 1 -- 172.21.15.90:0/3453883900 --> v1:172.21.15.92:6808/636125963 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f701c00c790 con 0x7f701c01f8f0 2024-04-14T15:26:58.301 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:58 smithi090 ceph-mon[30399]: pgmap v1273: 184 pgs: 184 active+clean; 582 KiB data, 728 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:26:58.301 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:58 smithi090 ceph-mon[41670]: pgmap v1273: 184 pgs: 184 active+clean; 582 KiB data, 728 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:26:58.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.302+0000 7f702e7fc700 1 -- 172.21.15.90:0/3453883900 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f701c00c790 con 0x7f701c01f8f0 2024-04-14T15:26:58.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.302+0000 7f702e7fc700 10 client.48107.objecter ms_dispatch 0x7f7030001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:26:58.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.302+0000 7f702e7fc700 10 client.48107.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:26:58.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.302+0000 7f702e7fc700 20 client.48107.objecter put_session s=0x7f701c01ce00 osd=6 4 2024-04-14T15:26:58.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.302+0000 7f702e7fc700 15 client.48107.objecter _session_command_op_remove 6 14 2024-04-14T15:26:58.303 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2024-04-14T15:26:58.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.302+0000 7f7037570700 10 client.48107.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:26:58.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.302+0000 7f7037570700 15 client.48107.objecter _session_command_op_assign -1 15 2024-04-14T15:26:58.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.302+0000 7f7037570700 20 client.48107.objecter _get_session s=0x7f701c023400 osd=7 3 2024-04-14T15:26:58.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.302+0000 7f7037570700 20 client.48107.objecter put_session s=0x7f701c023400 osd=7 3 2024-04-14T15:26:58.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.302+0000 7f7037570700 20 client.48107.objecter _get_session s=0x7f701c023400 osd=7 3 2024-04-14T15:26:58.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.302+0000 7f7037570700 15 client.48107.objecter _session_command_op_remove -1 15 2024-04-14T15:26:58.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.302+0000 7f7037570700 20 client.48107.objecter get_session s=0x7f701c023400 osd=7 3 2024-04-14T15:26:58.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.302+0000 7f7037570700 15 client.48107.objecter _session_command_op_assign 7 15 2024-04-14T15:26:58.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.302+0000 7f7037570700 20 client.48107.objecter put_session s=0x7f701c023400 osd=7 4 2024-04-14T15:26:58.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.302+0000 7f7037570700 10 client.48107.objecter _send_command 15 2024-04-14T15:26:58.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.302+0000 7f7037570700 1 -- 172.21.15.90:0/3453883900 --> v1:172.21.15.92:6812/1773706716 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f701c027720 con 0x7f701c023a80 2024-04-14T15:26:58.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.303+0000 7f702e7fc700 10 client.48107.objecter ms_handle_connect 0x7f701c023a80 2024-04-14T15:26:58.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.303+0000 7f702e7fc700 1 -- 172.21.15.90:0/3453883900 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f701c027720 con 0x7f701c023a80 2024-04-14T15:26:58.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.303+0000 7f702e7fc700 10 client.48107.objecter ms_dispatch 0x7f7030001550 command_reply(tid 15: 0 ) 2024-04-14T15:26:58.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.303+0000 7f702e7fc700 10 client.48107.objecter _finish_command 15 = system:0 2024-04-14T15:26:58.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.303+0000 7f702e7fc700 20 client.48107.objecter put_session s=0x7f701c023400 osd=7 4 2024-04-14T15:26:58.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.303+0000 7f702e7fc700 15 client.48107.objecter _session_command_op_remove 7 15 2024-04-14T15:26:58.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.320+0000 7f7037570700 10 client.48107.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-14T15:26:58.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.320+0000 7f7037570700 15 client.48107.objecter _session_command_op_assign -1 16 2024-04-14T15:26:58.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.320+0000 7f7037570700 20 client.48107.objecter _get_session s=0x7f701c023400 osd=7 3 2024-04-14T15:26:58.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.320+0000 7f7037570700 20 client.48107.objecter put_session s=0x7f701c023400 osd=7 3 2024-04-14T15:26:58.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.320+0000 7f7037570700 20 client.48107.objecter _get_session s=0x7f701c023400 osd=7 3 2024-04-14T15:26:58.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.320+0000 7f7037570700 15 client.48107.objecter _session_command_op_remove -1 16 2024-04-14T15:26:58.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.320+0000 7f7037570700 20 client.48107.objecter get_session s=0x7f701c023400 osd=7 3 2024-04-14T15:26:58.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.320+0000 7f7037570700 15 client.48107.objecter _session_command_op_assign 7 16 2024-04-14T15:26:58.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.320+0000 7f7037570700 20 client.48107.objecter put_session s=0x7f701c023400 osd=7 4 2024-04-14T15:26:58.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.320+0000 7f7037570700 10 client.48107.objecter _send_command 16 2024-04-14T15:26:58.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.320+0000 7f7037570700 1 -- 172.21.15.90:0/3453883900 --> v1:172.21.15.92:6812/1773706716 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f701c0290d0 con 0x7f701c023a80 2024-04-14T15:26:58.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.322+0000 7f702e7fc700 1 -- 172.21.15.90:0/3453883900 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f701c0290d0 con 0x7f701c023a80 2024-04-14T15:26:58.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.322+0000 7f702e7fc700 10 client.48107.objecter ms_dispatch 0x7f7030001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-14T15:26:58.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.322+0000 7f702e7fc700 10 client.48107.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-14T15:26:58.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.322+0000 7f702e7fc700 20 client.48107.objecter put_session s=0x7f701c023400 osd=7 4 2024-04-14T15:26:58.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.322+0000 7f702e7fc700 15 client.48107.objecter _session_command_op_remove 7 16 2024-04-14T15:26:58.323 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2024-04-14T15:26:58.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.322+0000 7f7037570700 10 librados: watch_flush enter 2024-04-14T15:26:58.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.322+0000 7f7037570700 10 librados: watch_flush exit 2024-04-14T15:26:58.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.322+0000 7f7037570700 10 client.48107.objecter close_session for osd.0 2024-04-14T15:26:58.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.322+0000 7f7037570700 1 -- 172.21.15.90:0/3453883900 >> v1:172.21.15.90:6801/3761806885 conn(0x7f701c004760 legacy=0x7f701c002780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:58.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.322+0000 7f7037570700 20 client.48107.objecter put_session s=0x7f701c005410 osd=0 1 2024-04-14T15:26:58.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.322+0000 7f7037570700 10 client.48107.objecter close_session for osd.1 2024-04-14T15:26:58.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.322+0000 7f7037570700 1 -- 172.21.15.90:0/3453883900 >> v1:172.21.15.90:6805/4170361699 conn(0x7f701c0042b0 legacy=0x7f701c00ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:58.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.322+0000 7f7037570700 20 client.48107.objecter put_session s=0x7f701c004140 osd=1 1 2024-04-14T15:26:58.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.322+0000 7f7037570700 10 client.48107.objecter close_session for osd.2 2024-04-14T15:26:58.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.322+0000 7f7037570700 1 -- 172.21.15.90:0/3453883900 >> v1:172.21.15.90:6809/1686184251 conn(0x7f701c00d600 legacy=0x7f701c00f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:58.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.322+0000 7f7037570700 20 client.48107.objecter put_session s=0x7f701c0083a0 osd=2 1 2024-04-14T15:26:58.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.322+0000 7f7037570700 10 client.48107.objecter close_session for osd.3 2024-04-14T15:26:58.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.322+0000 7f7037570700 1 -- 172.21.15.90:0/3453883900 >> v1:172.21.15.90:6813/2270166774 conn(0x7f701c012440 legacy=0x7f701c014830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:58.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.322+0000 7f7037570700 20 client.48107.objecter put_session s=0x7f701c00cc50 osd=3 1 2024-04-14T15:26:58.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.322+0000 7f7037570700 10 client.48107.objecter close_session for osd.4 2024-04-14T15:26:58.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.322+0000 7f7037570700 1 -- 172.21.15.90:0/3453883900 >> v1:172.21.15.92:6800/3887549733 conn(0x7f701c011fa0 legacy=0x7f701c0190e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:58.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.322+0000 7f7035b0d700 1 -- 172.21.15.90:0/3453883900 reap_dead start 2024-04-14T15:26:58.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.323+0000 7f7037570700 20 client.48107.objecter put_session s=0x7f701c00c980 osd=4 1 2024-04-14T15:26:58.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.323+0000 7f7037570700 10 client.48107.objecter close_session for osd.5 2024-04-14T15:26:58.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.323+0000 7f7037570700 1 -- 172.21.15.90:0/3453883900 >> v1:172.21.15.92:6804/3244561440 conn(0x7f701c01c900 legacy=0x7f701c01aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:58.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.323+0000 7f7035b0d700 1 -- 172.21.15.90:0/3453883900 reap_dead start 2024-04-14T15:26:58.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.323+0000 7f7037570700 20 client.48107.objecter put_session s=0x7f701c01c1e0 osd=5 1 2024-04-14T15:26:58.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.323+0000 7f7037570700 10 client.48107.objecter close_session for osd.6 2024-04-14T15:26:58.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.323+0000 7f7037570700 1 -- 172.21.15.90:0/3453883900 >> v1:172.21.15.92:6808/636125963 conn(0x7f701c01f8f0 legacy=0x7f701c021ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:58.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.323+0000 7f7037570700 20 client.48107.objecter put_session s=0x7f701c01ce00 osd=6 1 2024-04-14T15:26:58.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.323+0000 7f7037570700 10 client.48107.objecter close_session for osd.7 2024-04-14T15:26:58.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.323+0000 7f7037570700 1 -- 172.21.15.90:0/3453883900 >> v1:172.21.15.92:6812/1773706716 conn(0x7f701c023a80 legacy=0x7f701c025e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:58.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.323+0000 7f7037570700 20 client.48107.objecter put_session s=0x7f701c023400 osd=7 1 2024-04-14T15:26:58.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.323+0000 7f7037570700 20 client.48107.objecter shutdown clearing up homeless session... 2024-04-14T15:26:58.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.323+0000 7f7037570700 10 client.48107.objecter successfully canceled tick 2024-04-14T15:26:58.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.323+0000 7f7035b0d700 1 -- 172.21.15.90:0/3453883900 reap_dead start 2024-04-14T15:26:58.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.323+0000 7f7037570700 1 -- 172.21.15.90:0/3453883900 >> v1:172.21.15.90:6800/420935427 conn(0x7f7018071350 legacy=0x7f7018073810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:58.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.323+0000 7f7037570700 1 -- 172.21.15.90:0/3453883900 >> v1:172.21.15.90:6790/0 conn(0x7f703010a620 legacy=0x7f70300fde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:58.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.323+0000 7f7037570700 1 -- 172.21.15.90:0/3453883900 shutdown_connections 2024-04-14T15:26:58.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.323+0000 7f7037570700 1 -- 172.21.15.90:0/3453883900 >> 172.21.15.90:0/3453883900 conn(0x7f70300fd280 msgr2=0x7f7030111040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:26:58.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.323+0000 7f7037570700 1 -- 172.21.15.90:0/3453883900 shutdown_connections 2024-04-14T15:26:58.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.323+0000 7f7037570700 1 -- 172.21.15.90:0/3453883900 wait complete. 2024-04-14T15:26:58.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.323+0000 7f7037570700 1 librados: shutdown 2024-04-14T15:26:58.327 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-14T15:26:58.327 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-14T15:26:58.327 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-14T15:26:58.327 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-14T15:26:58.327 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-14T15:26:58.327 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-14T15:26:58.327 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-14T15:26:58.327 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-14T15:26:58.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:58 smithi092 ceph-mon[33966]: pgmap v1273: 184 pgs: 184 active+clean; 582 KiB data, 728 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:26:59.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:59 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:26:59.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:59 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:26:59.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:59 smithi092 ceph-mon[33966]: osdmap e853: 8 total, 8 up, 7 in 2024-04-14T15:26:59.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:26:59 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3453883900' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:26:59.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.892+0000 7f3c45a34700 1 -- 172.21.15.90:0/3513661037 >> v1:172.21.15.90:6790/0 conn(0x7f3c40103a40 legacy=0x7f3c40103e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:59.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.892+0000 7f3c45a34700 1 -- 172.21.15.90:0/3513661037 shutdown_connections 2024-04-14T15:26:59.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.892+0000 7f3c45a34700 1 -- 172.21.15.90:0/3513661037 >> 172.21.15.90:0/3513661037 conn(0x7f3c400fd270 msgr2=0x7f3c400ff690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:26:59.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.892+0000 7f3c45a34700 1 -- 172.21.15.90:0/3513661037 shutdown_connections 2024-04-14T15:26:59.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.892+0000 7f3c45a34700 1 -- 172.21.15.90:0/3513661037 wait complete. 2024-04-14T15:26:59.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.892+0000 7f3c45a34700 1 librados: starting msgr at 2024-04-14T15:26:59.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.892+0000 7f3c45a34700 1 librados: starting objecter 2024-04-14T15:26:59.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.892+0000 7f3c45a34700 1 Processor -- start 2024-04-14T15:26:59.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.892+0000 7f3c45a34700 1 -- start start 2024-04-14T15:26:59.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.892+0000 7f3c45a34700 1 librados: setting wanted keys 2024-04-14T15:26:59.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.892+0000 7f3c45a34700 1 librados: calling monclient init 2024-04-14T15:26:59.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.893+0000 7f3c45a34700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3c401a9e40 con 0x7f3c4010b850 2024-04-14T15:26:59.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.893+0000 7f3c45a34700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3c401ab020 con 0x7f3c4010f360 2024-04-14T15:26:59.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.893+0000 7f3c45a34700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3c401ac200 con 0x7f3c40103a40 2024-04-14T15:26:59.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.893+0000 7f3c3e7fc700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f3c4010b850 0x7f3c401a4ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:60354/0 (socket says 172.21.15.90:60354) 2024-04-14T15:26:59.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.893+0000 7f3c3effd700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f3c40103a40 0x7f3c401a01a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:56794/0 (socket says 172.21.15.90:56794) 2024-04-14T15:26:59.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.893+0000 7f3c3e7fc700 1 -- 172.21.15.90:0/1919333145 learned_addr learned my addr 172.21.15.90:0/1919333145 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:26:59.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.893+0000 7f3c44a32700 10 client.?.objecter ms_handle_connect 0x7f3c4010b850 2024-04-14T15:26:59.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.893+0000 7f3c44a32700 10 client.?.objecter resend_mon_ops 2024-04-14T15:26:59.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.893+0000 7f3c44a32700 10 client.?.objecter ms_handle_connect 0x7f3c40103a40 2024-04-14T15:26:59.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.893+0000 7f3c44a32700 10 client.?.objecter resend_mon_ops 2024-04-14T15:26:59.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.893+0000 7f3c44a32700 10 client.?.objecter ms_handle_connect 0x7f3c4010f360 2024-04-14T15:26:59.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.893+0000 7f3c44a32700 10 client.?.objecter resend_mon_ops 2024-04-14T15:26:59.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.893+0000 7f3c44a32700 1 -- 172.21.15.90:0/1919333145 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 902824315 0 0) 0x7f3c401a9e40 con 0x7f3c4010b850 2024-04-14T15:26:59.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.893+0000 7f3c44a32700 1 -- 172.21.15.90:0/1919333145 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3c28003aa0 con 0x7f3c4010b850 2024-04-14T15:26:59.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.893+0000 7f3c44a32700 1 -- 172.21.15.90:0/1919333145 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3144396872 0 0) 0x7f3c401ac200 con 0x7f3c40103a40 2024-04-14T15:26:59.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.893+0000 7f3c44a32700 1 -- 172.21.15.90:0/1919333145 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3c401a9e40 con 0x7f3c40103a40 2024-04-14T15:26:59.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.893+0000 7f3c44a32700 1 -- 172.21.15.90:0/1919333145 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2518663899 0 0) 0x7f3c401ab020 con 0x7f3c4010f360 2024-04-14T15:26:59.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.893+0000 7f3c44a32700 1 -- 172.21.15.90:0/1919333145 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3c401ac200 con 0x7f3c4010f360 2024-04-14T15:26:59.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.893+0000 7f3c44a32700 1 -- 172.21.15.90:0/1919333145 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4095906695 0 0) 0x7f3c28003aa0 con 0x7f3c4010b850 2024-04-14T15:26:59.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.893+0000 7f3c44a32700 1 -- 172.21.15.90:0/1919333145 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3c401ab020 con 0x7f3c4010b850 2024-04-14T15:26:59.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.893+0000 7f3c44a32700 1 -- 172.21.15.90:0/1919333145 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f3c340013d0 con 0x7f3c4010b850 2024-04-14T15:26:59.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.893+0000 7f3c44a32700 1 -- 172.21.15.90:0/1919333145 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2159920980 0 0) 0x7f3c401a9e40 con 0x7f3c40103a40 2024-04-14T15:26:59.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.893+0000 7f3c44a32700 1 -- 172.21.15.90:0/1919333145 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3c28003aa0 con 0x7f3c40103a40 2024-04-14T15:26:59.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.893+0000 7f3c44a32700 1 -- 172.21.15.90:0/1919333145 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f3c300038b0 con 0x7f3c40103a40 2024-04-14T15:26:59.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.893+0000 7f3c44a32700 1 -- 172.21.15.90:0/1919333145 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1801571599 0 0) 0x7f3c401ab020 con 0x7f3c4010b850 2024-04-14T15:26:59.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.893+0000 7f3c44a32700 1 -- 172.21.15.90:0/1919333145 >> v1:172.21.15.90:6790/0 conn(0x7f3c40103a40 legacy=0x7f3c401a01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:59.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.893+0000 7f3c44a32700 1 -- 172.21.15.90:0/1919333145 >> v1:172.21.15.92:6789/0 conn(0x7f3c4010f360 legacy=0x7f3c401a8720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:26:59.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.893+0000 7f3c44a32700 1 -- 172.21.15.90:0/1919333145 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c401ad3e0 con 0x7f3c4010b850 2024-04-14T15:26:59.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.893+0000 7f3c45a34700 1 -- 172.21.15.90:0/1919333145 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3c401ac430 con 0x7f3c4010b850 2024-04-14T15:26:59.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.893+0000 7f3c45a34700 10 client.38163.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:26:59.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.893+0000 7f3c45a34700 1 -- 172.21.15.90:0/1919333145 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f3c401ac8f0 con 0x7f3c4010b850 2024-04-14T15:26:59.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.893+0000 7f3c45a34700 1 librados: init done 2024-04-14T15:26:59.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.894+0000 7f3c44a32700 1 -- 172.21.15.90:0/1919333145 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f3c34001c70 con 0x7f3c4010b850 2024-04-14T15:26:59.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.894+0000 7f3c44a32700 1 -- 172.21.15.90:0/1919333145 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f3c34005ce0 con 0x7f3c4010b850 2024-04-14T15:26:59.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.895+0000 7f3c45a34700 1 -- 172.21.15.90:0/1919333145 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c2c005120 con 0x7f3c4010b850 2024-04-14T15:26:59.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.895+0000 7f3c44a32700 1 -- 172.21.15.90:0/1919333145 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f3c3401c6f0 con 0x7f3c4010b850 2024-04-14T15:26:59.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.896+0000 7f3c44a32700 1 -- 172.21.15.90:0/1919333145 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(853..853 src has 467..853) ==== 7517+0+0 (unknown 833938570 0 0) 0x7f3c3408a520 con 0x7f3c4010b850 2024-04-14T15:26:59.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.896+0000 7f3c44a32700 10 client.38163.objecter ms_dispatch 0x7f3c40001550 osd_map(853..853 src has 467..853) 2024-04-14T15:26:59.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.896+0000 7f3c44a32700 3 client.38163.objecter handle_osd_map got epochs [853,853] > 0 2024-04-14T15:26:59.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.896+0000 7f3c44a32700 3 client.38163.objecter handle_osd_map decoding full epoch 853 2024-04-14T15:26:59.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.896+0000 7f3c44a32700 20 client.38163.objecter dump_active .. 0 homeless 2024-04-14T15:26:59.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.896+0000 7f3c44a32700 10 client.38163.objecter ms_handle_connect 0x7f3c28071060 2024-04-14T15:26:59.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:58.900+0000 7f3c44a32700 1 -- 172.21.15.90:0/1919333145 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f3c34057300 con 0x7f3c4010b850 2024-04-14T15:26:59.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:59.116+0000 7f3c45a34700 1 -- 172.21.15.90:0/1919333145 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7f3c2c005410 con 0x7f3c4010b850 2024-04-14T15:26:59.420 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:59 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:26:59.420 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:59 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:26:59.420 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:59 smithi090 ceph-mon[41670]: osdmap e853: 8 total, 8 up, 7 in 2024-04-14T15:26:59.421 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:26:59 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3453883900' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:26:59.421 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:59 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:26:59.421 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:59 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:26:59.421 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:59 smithi090 ceph-mon[30399]: osdmap e853: 8 total, 8 up, 7 in 2024-04-14T15:26:59.421 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:26:59 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3453883900' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-14T15:27:00.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:00.111+0000 7f3c44a32700 1 -- 172.21.15.90:0/1919333145 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v854) ==== 97+0+0 (unknown 1446635267 0 0) 0x7f3c3405afb0 con 0x7f3c4010b850 2024-04-14T15:27:00.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:00.114+0000 7f3c45a34700 10 librados: watch_flush enter 2024-04-14T15:27:00.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:00.114+0000 7f3c45a34700 10 librados: watch_flush exit 2024-04-14T15:27:00.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:00.114+0000 7f3c45a34700 20 client.38163.objecter shutdown clearing up homeless session... 2024-04-14T15:27:00.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:00.114+0000 7f3c45a34700 10 client.38163.objecter successfully canceled tick 2024-04-14T15:27:00.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:00.114+0000 7f3c45a34700 1 -- 172.21.15.90:0/1919333145 >> v1:172.21.15.90:6800/420935427 conn(0x7f3c28071060 legacy=0x7f3c28073520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:00.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:00.114+0000 7f3c45a34700 1 -- 172.21.15.90:0/1919333145 >> v1:172.21.15.90:6789/0 conn(0x7f3c4010b850 legacy=0x7f3c401a4ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:00.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:00.114+0000 7f3c45a34700 1 -- 172.21.15.90:0/1919333145 shutdown_connections 2024-04-14T15:27:00.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:00.114+0000 7f3c45a34700 1 -- 172.21.15.90:0/1919333145 >> 172.21.15.90:0/1919333145 conn(0x7f3c400fd270 msgr2=0x7f3c401117f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:27:00.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:00.114+0000 7f3c45a34700 1 -- 172.21.15.90:0/1919333145 shutdown_connections 2024-04-14T15:27:00.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:00.114+0000 7f3c45a34700 1 -- 172.21.15.90:0/1919333145 wait complete. 2024-04-14T15:27:00.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:00.115+0000 7f3c45a34700 1 librados: shutdown 2024-04-14T15:27:00.116 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is set 2024-04-14T15:27:00.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:00 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1919333145' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:27:00.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:00 smithi092 ceph-mon[33966]: pgmap v1275: 184 pgs: 184 active+clean; 582 KiB data, 722 MiB used, 625 GiB / 626 GiB avail; 1.9 KiB/s rd, 1 op/s 2024-04-14T15:27:00.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:59.894+0000 7f3babf60700 1 -- 172.21.15.90:0/3965244778 >> v1:172.21.15.90:6789/0 conn(0x7f3ba410f420 legacy=0x7f3ba41118e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:00.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:59.894+0000 7f3babf60700 1 -- 172.21.15.90:0/3965244778 shutdown_connections 2024-04-14T15:27:00.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:59.894+0000 7f3babf60700 1 -- 172.21.15.90:0/3965244778 >> 172.21.15.90:0/3965244778 conn(0x7f3ba4076aa0 msgr2=0x7f3ba4076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:27:00.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:59.894+0000 7f3babf60700 1 -- 172.21.15.90:0/3965244778 shutdown_connections 2024-04-14T15:27:00.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:59.894+0000 7f3babf60700 1 -- 172.21.15.90:0/3965244778 wait complete. 2024-04-14T15:27:00.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:59.895+0000 7f3babf60700 1 librados: starting msgr at 2024-04-14T15:27:00.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:59.895+0000 7f3babf60700 1 librados: starting objecter 2024-04-14T15:27:00.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:59.895+0000 7f3babf60700 1 Processor -- start 2024-04-14T15:27:00.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:59.895+0000 7f3babf60700 1 -- start start 2024-04-14T15:27:00.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:59.895+0000 7f3babf60700 1 librados: setting wanted keys 2024-04-14T15:27:00.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:59.895+0000 7f3babf60700 1 librados: calling monclient init 2024-04-14T15:27:00.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:59.895+0000 7f3babf60700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3ba4101cc0 con 0x7f3ba4108a20 2024-04-14T15:27:00.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:59.895+0000 7f3babf60700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3ba41a9010 con 0x7f3ba410f420 2024-04-14T15:27:00.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:59.895+0000 7f3babf60700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3ba41aa1f0 con 0x7f3ba410b820 2024-04-14T15:27:00.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:59.895+0000 7f3ba9cfc700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f3ba4108a20 0x7f3ba40feb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:60360/0 (socket says 172.21.15.90:60360) 2024-04-14T15:27:00.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:59.895+0000 7f3ba9cfc700 1 -- 172.21.15.90:0/4095297040 learned_addr learned my addr 172.21.15.90:0/4095297040 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:27:00.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:59.895+0000 7f3b9affd700 10 client.?.objecter ms_handle_connect 0x7f3ba4108a20 2024-04-14T15:27:00.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:59.895+0000 7f3b9affd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:27:00.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:59.895+0000 7f3b9affd700 10 client.?.objecter ms_handle_connect 0x7f3ba410b820 2024-04-14T15:27:00.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:59.895+0000 7f3b9affd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:27:00.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:59.896+0000 7f3b9affd700 10 client.?.objecter ms_handle_connect 0x7f3ba410f420 2024-04-14T15:27:00.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:59.896+0000 7f3b9affd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:27:00.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:59.896+0000 7f3b9affd700 1 -- 172.21.15.90:0/4095297040 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3266851000 0 0) 0x7f3ba4101cc0 con 0x7f3ba4108a20 2024-04-14T15:27:00.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:59.896+0000 7f3b9affd700 1 -- 172.21.15.90:0/4095297040 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3b900035b0 con 0x7f3ba4108a20 2024-04-14T15:27:00.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:59.896+0000 7f3b9affd700 1 -- 172.21.15.90:0/4095297040 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 457855961 0 0) 0x7f3ba41aa1f0 con 0x7f3ba410b820 2024-04-14T15:27:00.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:59.896+0000 7f3b9affd700 1 -- 172.21.15.90:0/4095297040 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3ba4101cc0 con 0x7f3ba410b820 2024-04-14T15:27:00.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:59.896+0000 7f3b9affd700 1 -- 172.21.15.90:0/4095297040 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 68089103 0 0) 0x7f3ba41a9010 con 0x7f3ba410f420 2024-04-14T15:27:00.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:59.896+0000 7f3b9affd700 1 -- 172.21.15.90:0/4095297040 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3ba41aa1f0 con 0x7f3ba410f420 2024-04-14T15:27:00.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:59.896+0000 7f3b9affd700 1 -- 172.21.15.90:0/4095297040 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1022006260 0 0) 0x7f3b900035b0 con 0x7f3ba4108a20 2024-04-14T15:27:00.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:59.896+0000 7f3b9affd700 1 -- 172.21.15.90:0/4095297040 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3ba41a9010 con 0x7f3ba4108a20 2024-04-14T15:27:00.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:59.896+0000 7f3b9affd700 1 -- 172.21.15.90:0/4095297040 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f3ba0001930 con 0x7f3ba4108a20 2024-04-14T15:27:00.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:59.896+0000 7f3b9affd700 1 -- 172.21.15.90:0/4095297040 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 898982928 0 0) 0x7f3ba4101cc0 con 0x7f3ba410b820 2024-04-14T15:27:00.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:59.896+0000 7f3b9affd700 1 -- 172.21.15.90:0/4095297040 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3b900035b0 con 0x7f3ba410b820 2024-04-14T15:27:00.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:59.896+0000 7f3b9affd700 1 -- 172.21.15.90:0/4095297040 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f3b94001850 con 0x7f3ba410b820 2024-04-14T15:27:00.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:59.896+0000 7f3b9affd700 1 -- 172.21.15.90:0/4095297040 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1822443738 0 0) 0x7f3ba41aa1f0 con 0x7f3ba410f420 2024-04-14T15:27:00.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:59.896+0000 7f3b9affd700 1 -- 172.21.15.90:0/4095297040 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3ba4101cc0 con 0x7f3ba410f420 2024-04-14T15:27:00.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:59.896+0000 7f3b9affd700 1 -- 172.21.15.90:0/4095297040 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f3b9c0017b0 con 0x7f3ba410f420 2024-04-14T15:27:00.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:59.896+0000 7f3b9affd700 1 -- 172.21.15.90:0/4095297040 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2385604890 0 0) 0x7f3ba41a9010 con 0x7f3ba4108a20 2024-04-14T15:27:00.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:59.896+0000 7f3b9affd700 1 -- 172.21.15.90:0/4095297040 >> v1:172.21.15.90:6790/0 conn(0x7f3ba410b820 legacy=0x7f3ba40ff220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:00.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:59.896+0000 7f3b9affd700 1 -- 172.21.15.90:0/4095297040 >> v1:172.21.15.92:6789/0 conn(0x7f3ba410f420 legacy=0x7f3ba41a68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:00.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:59.896+0000 7f3b9affd700 1 -- 172.21.15.90:0/4095297040 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ba41ab3d0 con 0x7f3ba4108a20 2024-04-14T15:27:00.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:59.896+0000 7f3babf60700 1 -- 172.21.15.90:0/4095297040 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3ba41a8060 con 0x7f3ba4108a20 2024-04-14T15:27:00.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:59.896+0000 7f3babf60700 10 client.38169.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:27:00.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:59.896+0000 7f3babf60700 1 -- 172.21.15.90:0/4095297040 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f3ba41a8520 con 0x7f3ba4108a20 2024-04-14T15:27:00.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:59.896+0000 7f3babf60700 1 librados: init done 2024-04-14T15:27:00.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:59.896+0000 7f3b9affd700 1 -- 172.21.15.90:0/4095297040 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f3ba0001d00 con 0x7f3ba4108a20 2024-04-14T15:27:00.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:59.896+0000 7f3b9affd700 1 -- 172.21.15.90:0/4095297040 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f3ba0005df0 con 0x7f3ba4108a20 2024-04-14T15:27:00.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:59.897+0000 7f3babf60700 1 -- 172.21.15.90:0/4095297040 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b88005120 con 0x7f3ba4108a20 2024-04-14T15:27:00.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:59.898+0000 7f3b9affd700 1 -- 172.21.15.90:0/4095297040 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f3ba0003950 con 0x7f3ba4108a20 2024-04-14T15:27:00.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:59.898+0000 7f3b9affd700 1 -- 172.21.15.90:0/4095297040 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(853..853 src has 467..853) ==== 7517+0+0 (unknown 833938570 0 0) 0x7f3ba008a7a0 con 0x7f3ba4108a20 2024-04-14T15:27:00.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:59.898+0000 7f3b9affd700 10 client.38169.objecter ms_dispatch 0x7f3ba4001550 osd_map(853..853 src has 467..853) 2024-04-14T15:27:00.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:59.898+0000 7f3b9affd700 3 client.38169.objecter handle_osd_map got epochs [853,853] > 0 2024-04-14T15:27:00.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:59.898+0000 7f3b9affd700 3 client.38169.objecter handle_osd_map decoding full epoch 853 2024-04-14T15:27:00.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:59.898+0000 7f3b9affd700 20 client.38169.objecter dump_active .. 0 homeless 2024-04-14T15:27:00.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:59.898+0000 7f3b9affd700 10 client.38169.objecter ms_handle_connect 0x7f3b900713d0 2024-04-14T15:27:00.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:26:59.903+0000 7f3b9affd700 1 -- 172.21.15.90:0/4095297040 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f3ba00574a0 con 0x7f3ba4108a20 2024-04-14T15:27:00.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:00.104+0000 7f3babf60700 1 -- 172.21.15.90:0/4095297040 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} v 0) -- 0x7f3b88005410 con 0x7f3ba4108a20 2024-04-14T15:27:00.435 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:00 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1919333145' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:27:00.436 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:00 smithi090 ceph-mon[41670]: pgmap v1275: 184 pgs: 184 active+clean; 582 KiB data, 722 MiB used, 625 GiB / 626 GiB avail; 1.9 KiB/s rd, 1 op/s 2024-04-14T15:27:00.436 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:00 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1919333145' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:27:00.436 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:00 smithi090 ceph-mon[30399]: pgmap v1275: 184 pgs: 184 active+clean; 582 KiB data, 722 MiB used, 625 GiB / 626 GiB avail; 1.9 KiB/s rd, 1 op/s 2024-04-14T15:27:00.862 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-14T15:27:00.862 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:27:01.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:01.113+0000 7f3b9affd700 1 -- 172.21.15.90:0/4095297040 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]=0 set pool 1 pg_num to 11 v855) ==== 129+0+0 (unknown 1828860844 0 0) 0x7f3ba005b150 con 0x7f3ba4108a20 2024-04-14T15:27:01.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:01.115+0000 7f3babf60700 10 librados: watch_flush enter 2024-04-14T15:27:01.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:01.115+0000 7f3babf60700 10 librados: watch_flush exit 2024-04-14T15:27:01.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:01.115+0000 7f3babf60700 20 client.38169.objecter shutdown clearing up homeless session... 2024-04-14T15:27:01.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:01.115+0000 7f3babf60700 10 client.38169.objecter successfully canceled tick 2024-04-14T15:27:01.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:01.115+0000 7f3babf60700 1 -- 172.21.15.90:0/4095297040 >> v1:172.21.15.90:6800/420935427 conn(0x7f3b900713d0 legacy=0x7f3b90073890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:01.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:01.115+0000 7f3babf60700 1 -- 172.21.15.90:0/4095297040 >> v1:172.21.15.90:6789/0 conn(0x7f3ba4108a20 legacy=0x7f3ba40feb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:01.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:01.115+0000 7f3babf60700 1 -- 172.21.15.90:0/4095297040 shutdown_connections 2024-04-14T15:27:01.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:01.115+0000 7f3babf60700 1 -- 172.21.15.90:0/4095297040 >> 172.21.15.90:0/4095297040 conn(0x7f3ba4076aa0 msgr2=0x7f3ba41118a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:27:01.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:01.115+0000 7f3babf60700 1 -- 172.21.15.90:0/4095297040 shutdown_connections 2024-04-14T15:27:01.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:01.115+0000 7f3babf60700 1 -- 172.21.15.90:0/4095297040 wait complete. 2024-04-14T15:27:01.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:01.115+0000 7f3babf60700 1 librados: shutdown 2024-04-14T15:27:01.117 INFO:teuthology.orchestra.run.smithi090.stderr:set pool 1 pg_num to 11 2024-04-14T15:27:01.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:01 smithi092 ceph-mon[33966]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:27:01.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:01 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/4095297040' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-04-14T15:27:01.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:01 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1919333145' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-14T15:27:01.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:01 smithi092 ceph-mon[33966]: osdmap e854: 8 total, 8 up, 7 in 2024-04-14T15:27:01.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:01 smithi090 ceph-mon[41670]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:27:01.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:01 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/4095297040' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-04-14T15:27:01.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:01 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1919333145' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-14T15:27:01.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:01 smithi090 ceph-mon[41670]: osdmap e854: 8 total, 8 up, 7 in 2024-04-14T15:27:01.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:01 smithi090 ceph-mon[30399]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:27:01.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:01 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/4095297040' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-04-14T15:27:01.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:01 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1919333145' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-14T15:27:01.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:01 smithi090 ceph-mon[30399]: osdmap e854: 8 total, 8 up, 7 in 2024-04-14T15:27:02.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:02 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/4095297040' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-04-14T15:27:02.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:02 smithi090 ceph-mon[41670]: osdmap e855: 8 total, 8 up, 7 in 2024-04-14T15:27:02.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:02 smithi090 ceph-mon[41670]: pgmap v1278: 184 pgs: 184 active+clean; 582 KiB data, 722 MiB used, 625 GiB / 626 GiB avail; 988 B/s rd, 0 op/s 2024-04-14T15:27:02.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:02 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-04-14T15:27:02.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:02 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/4095297040' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-04-14T15:27:02.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:02 smithi090 ceph-mon[30399]: osdmap e855: 8 total, 8 up, 7 in 2024-04-14T15:27:02.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:02 smithi090 ceph-mon[30399]: pgmap v1278: 184 pgs: 184 active+clean; 582 KiB data, 722 MiB used, 625 GiB / 626 GiB avail; 988 B/s rd, 0 op/s 2024-04-14T15:27:02.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:02 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-04-14T15:27:02.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:02 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/4095297040' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-04-14T15:27:02.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:02 smithi092 ceph-mon[33966]: osdmap e855: 8 total, 8 up, 7 in 2024-04-14T15:27:02.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:02 smithi092 ceph-mon[33966]: pgmap v1278: 184 pgs: 184 active+clean; 582 KiB data, 722 MiB used, 625 GiB / 626 GiB avail; 988 B/s rd, 0 op/s 2024-04-14T15:27:02.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:02 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-04-14T15:27:03.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:03 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-04-14T15:27:03.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:03 smithi092 ceph-mon[33966]: osdmap e856: 8 total, 8 up, 7 in 2024-04-14T15:27:03.398 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:03 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-04-14T15:27:03.398 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:03 smithi090 ceph-mon[30399]: osdmap e856: 8 total, 8 up, 7 in 2024-04-14T15:27:03.398 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:03 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-04-14T15:27:03.398 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:03 smithi090 ceph-mon[41670]: osdmap e856: 8 total, 8 up, 7 in 2024-04-14T15:27:03.429 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-14T15:27:03.677 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd unset noscrub 2024-04-14T15:27:03.678 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:27:04.052 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:27:03 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:27:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:27:04.382 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:27:04.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:04 smithi090 ceph-mon[41670]: osdmap e857: 8 total, 8 up, 7 in 2024-04-14T15:27:04.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:04 smithi090 ceph-mon[41670]: pgmap v1281: 184 pgs: 184 active+clean; 582 KiB data, 722 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:27:04.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:04 smithi090 ceph-mon[30399]: osdmap e857: 8 total, 8 up, 7 in 2024-04-14T15:27:04.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:04 smithi090 ceph-mon[30399]: pgmap v1281: 184 pgs: 184 active+clean; 582 KiB data, 722 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:27:04.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:04 smithi092 ceph-mon[33966]: osdmap e857: 8 total, 8 up, 7 in 2024-04-14T15:27:04.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:04 smithi092 ceph-mon[33966]: pgmap v1281: 184 pgs: 184 active+clean; 582 KiB data, 722 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:27:05.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:05 smithi090 ceph-mon[30399]: 1.d starting backfill to osd.2 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:05.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:05 smithi090 ceph-mon[30399]: 1.d starting backfill to osd.5 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:05.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:05 smithi090 ceph-mon[30399]: 1.13 starting backfill to osd.5 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:05.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:05 smithi090 ceph-mon[30399]: 1.e starting backfill to osd.0 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:05.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:05 smithi090 ceph-mon[30399]: 1.c starting backfill to osd.5 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:05.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:05 smithi090 ceph-mon[30399]: 1.c starting backfill to osd.7 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:05.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:05 smithi090 ceph-mon[30399]: 1.f starting backfill to osd.7 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:05.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:05 smithi090 ceph-mon[30399]: 1.10 starting backfill to osd.0 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:05.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:05 smithi090 ceph-mon[30399]: 1.10 starting backfill to osd.1 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:05.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:05 smithi090 ceph-mon[30399]: 1.12 starting backfill to osd.0 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:05.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:05 smithi090 ceph-mon[30399]: 1.12 starting backfill to osd.4 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:05.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:05 smithi090 ceph-mon[30399]: 1.b starting backfill to osd.4 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:05.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:05 smithi090 ceph-mon[30399]: osdmap e858: 8 total, 8 up, 7 in 2024-04-14T15:27:05.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:05 smithi090 ceph-mon[30399]: 1.11 starting backfill to osd.2 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:05.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:05 smithi090 ceph-mon[30399]: 1.14 starting backfill to osd.3 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:05.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:05 smithi090 ceph-mon[30399]: 1.14 starting backfill to osd.5 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:05.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:05 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:27:05.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:05 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:27:05.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:05 smithi090 ceph-mon[41670]: 1.d starting backfill to osd.2 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:05.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:05 smithi090 ceph-mon[41670]: 1.d starting backfill to osd.5 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:05.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:05 smithi090 ceph-mon[41670]: 1.13 starting backfill to osd.5 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:05.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:05 smithi090 ceph-mon[41670]: 1.e starting backfill to osd.0 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:05.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:05 smithi090 ceph-mon[41670]: 1.c starting backfill to osd.5 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:05.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:05 smithi090 ceph-mon[41670]: 1.c starting backfill to osd.7 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:05.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:05 smithi090 ceph-mon[41670]: 1.f starting backfill to osd.7 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:05.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:05 smithi090 ceph-mon[41670]: 1.10 starting backfill to osd.0 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:05.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:05 smithi090 ceph-mon[41670]: 1.10 starting backfill to osd.1 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:05.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:05 smithi090 ceph-mon[41670]: 1.12 starting backfill to osd.0 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:05.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:05 smithi090 ceph-mon[41670]: 1.12 starting backfill to osd.4 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:05.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:05 smithi090 ceph-mon[41670]: 1.b starting backfill to osd.4 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:05.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:05 smithi090 ceph-mon[41670]: osdmap e858: 8 total, 8 up, 7 in 2024-04-14T15:27:05.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:05 smithi090 ceph-mon[41670]: 1.11 starting backfill to osd.2 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:05.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:05 smithi090 ceph-mon[41670]: 1.14 starting backfill to osd.3 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:05.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:05 smithi090 ceph-mon[41670]: 1.14 starting backfill to osd.5 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:05.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:05 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:27:05.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:05 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:27:05.566 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:05 smithi092 ceph-mon[33966]: 1.d starting backfill to osd.2 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:05.566 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:05 smithi092 ceph-mon[33966]: 1.d starting backfill to osd.5 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:05.567 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:05 smithi092 ceph-mon[33966]: 1.13 starting backfill to osd.5 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:05.567 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:05 smithi092 ceph-mon[33966]: 1.e starting backfill to osd.0 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:05.567 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:05 smithi092 ceph-mon[33966]: 1.c starting backfill to osd.5 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:05.567 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:05 smithi092 ceph-mon[33966]: 1.c starting backfill to osd.7 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:05.567 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:05 smithi092 ceph-mon[33966]: 1.f starting backfill to osd.7 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:05.567 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:05 smithi092 ceph-mon[33966]: 1.10 starting backfill to osd.0 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:05.567 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:05 smithi092 ceph-mon[33966]: 1.10 starting backfill to osd.1 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:05.567 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:05 smithi092 ceph-mon[33966]: 1.12 starting backfill to osd.0 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:05.567 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:05 smithi092 ceph-mon[33966]: 1.12 starting backfill to osd.4 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:05.567 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:05 smithi092 ceph-mon[33966]: 1.b starting backfill to osd.4 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:05.567 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:05 smithi092 ceph-mon[33966]: osdmap e858: 8 total, 8 up, 7 in 2024-04-14T15:27:05.567 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:05 smithi092 ceph-mon[33966]: 1.11 starting backfill to osd.2 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:05.567 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:05 smithi092 ceph-mon[33966]: 1.14 starting backfill to osd.3 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:05.567 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:05 smithi092 ceph-mon[33966]: 1.14 starting backfill to osd.5 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:05.568 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:05 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:27:05.568 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:05 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:27:05.872 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:27:05 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:27:06.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:06 smithi090 ceph-mon[30399]: osdmap e859: 8 total, 8 up, 7 in 2024-04-14T15:27:06.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:06 smithi090 ceph-mon[30399]: pgmap v1284: 184 pgs: 4 active+clean+remapped, 2 active+remapped+backfill_toofull, 4 remapped+peering, 174 active+clean; 596 KiB data, 723 MiB used, 625 GiB / 626 GiB avail; 3.0 KiB/s rd, 14 KiB/s wr, 3 op/s; 90 KiB/s, 0 objects/s recovering 2024-04-14T15:27:06.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:06 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:27:06.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:06 smithi090 ceph-mon[30399]: osdmap e860: 8 total, 8 up, 7 in 2024-04-14T15:27:06.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:06 smithi090 ceph-mon[41670]: osdmap e859: 8 total, 8 up, 7 in 2024-04-14T15:27:06.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:06 smithi090 ceph-mon[41670]: pgmap v1284: 184 pgs: 4 active+clean+remapped, 2 active+remapped+backfill_toofull, 4 remapped+peering, 174 active+clean; 596 KiB data, 723 MiB used, 625 GiB / 626 GiB avail; 3.0 KiB/s rd, 14 KiB/s wr, 3 op/s; 90 KiB/s, 0 objects/s recovering 2024-04-14T15:27:06.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:06 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:27:06.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:06 smithi090 ceph-mon[41670]: osdmap e860: 8 total, 8 up, 7 in 2024-04-14T15:27:06.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:06 smithi092 ceph-mon[33966]: osdmap e859: 8 total, 8 up, 7 in 2024-04-14T15:27:06.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:06 smithi092 ceph-mon[33966]: pgmap v1284: 184 pgs: 4 active+clean+remapped, 2 active+remapped+backfill_toofull, 4 remapped+peering, 174 active+clean; 596 KiB data, 723 MiB used, 625 GiB / 626 GiB avail; 3.0 KiB/s rd, 14 KiB/s wr, 3 op/s; 90 KiB/s, 0 objects/s recovering 2024-04-14T15:27:06.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:06 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:27:06.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:06 smithi092 ceph-mon[33966]: osdmap e860: 8 total, 8 up, 7 in 2024-04-14T15:27:06.928 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-14T15:27:06.928 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:27:07.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:07.139+0000 7fcc0da67700 1 -- 172.21.15.90:0/1309580354 >> v1:172.21.15.90:6789/0 conn(0x7fcc08108880 legacy=0x7fcc08108c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:07.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:07.139+0000 7fcc0da67700 1 -- 172.21.15.90:0/1309580354 shutdown_connections 2024-04-14T15:27:07.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:07.139+0000 7fcc0da67700 1 -- 172.21.15.90:0/1309580354 >> 172.21.15.90:0/1309580354 conn(0x7fcc080fe3d0 msgr2=0x7fcc081007f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:27:07.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:07.139+0000 7fcc0da67700 1 -- 172.21.15.90:0/1309580354 shutdown_connections 2024-04-14T15:27:07.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:07.139+0000 7fcc0da67700 1 -- 172.21.15.90:0/1309580354 wait complete. 2024-04-14T15:27:07.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:07.140+0000 7fcc0da67700 1 librados: starting msgr at 2024-04-14T15:27:07.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:07.140+0000 7fcc0da67700 1 librados: starting objecter 2024-04-14T15:27:07.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:07.140+0000 7fcc0da67700 1 Processor -- start 2024-04-14T15:27:07.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:07.140+0000 7fcc0da67700 1 -- start start 2024-04-14T15:27:07.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:07.140+0000 7fcc0da67700 1 librados: setting wanted keys 2024-04-14T15:27:07.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:07.140+0000 7fcc0da67700 1 librados: calling monclient init 2024-04-14T15:27:07.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:07.140+0000 7fcc0da67700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fcc0810ef60 con 0x7fcc0810b730 2024-04-14T15:27:07.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:07.140+0000 7fcc0da67700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fcc081a8f10 con 0x7fcc08108880 2024-04-14T15:27:07.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:07.140+0000 7fcc0da67700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fcc081aa0f0 con 0x7fcc0810f300 2024-04-14T15:27:07.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:07.141+0000 7fcc07fff700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7fcc0810f300 0x7fcc081a67e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:55624/0 (socket says 172.21.15.90:55624) 2024-04-14T15:27:07.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:07.141+0000 7fcc06ffd700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fcc0810b730 0x7fcc0810c510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:44840/0 (socket says 172.21.15.90:44840) 2024-04-14T15:27:07.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:07.141+0000 7fcc07fff700 1 -- 172.21.15.90:0/4274299571 learned_addr learned my addr 172.21.15.90:0/4274299571 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:27:07.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:07.141+0000 7fcc04ff9700 10 client.?.objecter ms_handle_connect 0x7fcc0810b730 2024-04-14T15:27:07.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:07.141+0000 7fcc04ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:27:07.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:07.141+0000 7fcc04ff9700 10 client.?.objecter ms_handle_connect 0x7fcc0810f300 2024-04-14T15:27:07.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:07.141+0000 7fcc04ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:27:07.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:07.141+0000 7fcc04ff9700 1 -- 172.21.15.90:0/4274299571 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 381469723 0 0) 0x7fcc0810ef60 con 0x7fcc0810b730 2024-04-14T15:27:07.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:07.141+0000 7fcc04ff9700 1 -- 172.21.15.90:0/4274299571 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcbf40035b0 con 0x7fcc0810b730 2024-04-14T15:27:07.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:07.141+0000 7fcc04ff9700 1 -- 172.21.15.90:0/4274299571 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2368681762 0 0) 0x7fcc081aa0f0 con 0x7fcc0810f300 2024-04-14T15:27:07.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:07.141+0000 7fcc04ff9700 1 -- 172.21.15.90:0/4274299571 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc0810ef60 con 0x7fcc0810f300 2024-04-14T15:27:07.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:07.141+0000 7fcc04ff9700 10 client.?.objecter ms_handle_connect 0x7fcc08108880 2024-04-14T15:27:07.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:07.141+0000 7fcc04ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:27:07.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:07.141+0000 7fcc04ff9700 1 -- 172.21.15.90:0/4274299571 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3554223391 0 0) 0x7fcc081a8f10 con 0x7fcc08108880 2024-04-14T15:27:07.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:07.141+0000 7fcc04ff9700 1 -- 172.21.15.90:0/4274299571 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc081aa0f0 con 0x7fcc08108880 2024-04-14T15:27:07.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:07.141+0000 7fcc04ff9700 1 -- 172.21.15.90:0/4274299571 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2606601207 0 0) 0x7fcbf40035b0 con 0x7fcc0810b730 2024-04-14T15:27:07.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:07.141+0000 7fcc04ff9700 1 -- 172.21.15.90:0/4274299571 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcc081a8f10 con 0x7fcc0810b730 2024-04-14T15:27:07.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:07.141+0000 7fcc04ff9700 1 -- 172.21.15.90:0/4274299571 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fcbf00018d0 con 0x7fcc0810b730 2024-04-14T15:27:07.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:07.141+0000 7fcc04ff9700 1 -- 172.21.15.90:0/4274299571 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 114655922 0 0) 0x7fcc0810ef60 con 0x7fcc0810f300 2024-04-14T15:27:07.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:07.141+0000 7fcc04ff9700 1 -- 172.21.15.90:0/4274299571 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcbf40035b0 con 0x7fcc0810f300 2024-04-14T15:27:07.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:07.141+0000 7fcc04ff9700 1 -- 172.21.15.90:0/4274299571 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fcbf8001870 con 0x7fcc0810f300 2024-04-14T15:27:07.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:07.141+0000 7fcc04ff9700 1 -- 172.21.15.90:0/4274299571 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2974359039 0 0) 0x7fcc081a8f10 con 0x7fcc0810b730 2024-04-14T15:27:07.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:07.141+0000 7fcc04ff9700 1 -- 172.21.15.90:0/4274299571 >> v1:172.21.15.90:6790/0 conn(0x7fcc0810f300 legacy=0x7fcc081a67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:07.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:07.141+0000 7fcc04ff9700 1 -- 172.21.15.90:0/4274299571 >> v1:172.21.15.92:6789/0 conn(0x7fcc08108880 legacy=0x7fcc0810be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:07.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:07.141+0000 7fcc04ff9700 1 -- 172.21.15.90:0/4274299571 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc081ab2d0 con 0x7fcc0810b730 2024-04-14T15:27:07.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:07.141+0000 7fcc0da67700 1 -- 172.21.15.90:0/4274299571 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fcc081aa320 con 0x7fcc0810b730 2024-04-14T15:27:07.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:07.141+0000 7fcc0da67700 10 client.38175.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:27:07.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:07.141+0000 7fcc0da67700 1 -- 172.21.15.90:0/4274299571 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fcc081aa7e0 con 0x7fcc0810b730 2024-04-14T15:27:07.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:07.141+0000 7fcc0da67700 1 librados: init done 2024-04-14T15:27:07.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:07.142+0000 7fcc04ff9700 1 -- 172.21.15.90:0/4274299571 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fcbf0003b40 con 0x7fcc0810b730 2024-04-14T15:27:07.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:07.142+0000 7fcc04ff9700 1 -- 172.21.15.90:0/4274299571 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fcbf0005c80 con 0x7fcc0810b730 2024-04-14T15:27:07.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:07.142+0000 7fcc0da67700 1 -- 172.21.15.90:0/4274299571 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbe8005120 con 0x7fcc0810b730 2024-04-14T15:27:07.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:07.143+0000 7fcc04ff9700 1 -- 172.21.15.90:0/4274299571 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fcbf00036a0 con 0x7fcc0810b730 2024-04-14T15:27:07.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:07.144+0000 7fcc04ff9700 1 -- 172.21.15.90:0/4274299571 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(860..860 src has 467..860) ==== 7616+0+0 (unknown 3925367153 0 0) 0x7fcbf008a780 con 0x7fcc0810b730 2024-04-14T15:27:07.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:07.144+0000 7fcc04ff9700 10 client.38175.objecter ms_dispatch 0x7fcc08001550 osd_map(860..860 src has 467..860) 2024-04-14T15:27:07.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:07.144+0000 7fcc04ff9700 3 client.38175.objecter handle_osd_map got epochs [860,860] > 0 2024-04-14T15:27:07.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:07.144+0000 7fcc04ff9700 3 client.38175.objecter handle_osd_map decoding full epoch 860 2024-04-14T15:27:07.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:07.144+0000 7fcc04ff9700 20 client.38175.objecter dump_active .. 0 homeless 2024-04-14T15:27:07.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:07.144+0000 7fcc04ff9700 10 client.38175.objecter ms_handle_connect 0x7fcbf40710a0 2024-04-14T15:27:07.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:07.148+0000 7fcc04ff9700 1 -- 172.21.15.90:0/4274299571 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fcbf00573c0 con 0x7fcc0810b730 2024-04-14T15:27:07.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:07.359+0000 7fcc0da67700 1 -- 172.21.15.90:0/4274299571 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7fcbe8005410 con 0x7fcc0810b730 2024-04-14T15:27:07.360 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:07 smithi090 ceph-mon[30399]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-04-14T15:27:07.360 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:07 smithi090 ceph-mon[30399]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-04-14T15:27:07.360 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:07 smithi090 ceph-mon[41670]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-04-14T15:27:07.360 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:07 smithi090 ceph-mon[41670]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-04-14T15:27:07.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:07 smithi092 ceph-mon[33966]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-04-14T15:27:07.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:07 smithi092 ceph-mon[33966]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-04-14T15:27:07.875 INFO:tasks.thrashosds.thrasher:inning osd 2024-04-14T15:27:07.875 INFO:tasks.thrashosds.thrasher:Adding osd 6 2024-04-14T15:27:07.875 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd in 6 2024-04-14T15:27:07.911 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-14T15:27:08.139 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:27:08.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:08.155+0000 7fcc04ff9700 1 -- 172.21.15.90:0/4274299571 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v861) ==== 91+0+0 (unknown 1321290299 0 0) 0x7fcbf005b070 con 0x7fcc0810b730 2024-04-14T15:27:08.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:08.158+0000 7fcc0da67700 10 librados: watch_flush enter 2024-04-14T15:27:08.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:08.159+0000 7fcc0da67700 10 librados: watch_flush exit 2024-04-14T15:27:08.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:08.159+0000 7fcc0da67700 20 client.38175.objecter shutdown clearing up homeless session... 2024-04-14T15:27:08.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:08.159+0000 7fcc0da67700 10 client.38175.objecter successfully canceled tick 2024-04-14T15:27:08.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:08.159+0000 7fcc0da67700 1 -- 172.21.15.90:0/4274299571 >> v1:172.21.15.90:6800/420935427 conn(0x7fcbf40710a0 legacy=0x7fcbf4073560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:08.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:08.159+0000 7fcc0da67700 1 -- 172.21.15.90:0/4274299571 >> v1:172.21.15.90:6789/0 conn(0x7fcc0810b730 legacy=0x7fcc0810c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:08.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:08.160+0000 7fcc0da67700 1 -- 172.21.15.90:0/4274299571 shutdown_connections 2024-04-14T15:27:08.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:08.160+0000 7fcc0da67700 1 -- 172.21.15.90:0/4274299571 >> 172.21.15.90:0/4274299571 conn(0x7fcc080fe3d0 msgr2=0x7fcc08112820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:27:08.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:08.161+0000 7fcc0da67700 1 -- 172.21.15.90:0/4274299571 shutdown_connections 2024-04-14T15:27:08.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:08.161+0000 7fcc0da67700 1 -- 172.21.15.90:0/4274299571 wait complete. 2024-04-14T15:27:08.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:08.161+0000 7fcc0da67700 1 librados: shutdown 2024-04-14T15:27:08.163 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is unset 2024-04-14T15:27:08.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:08 smithi090 ceph-mon[30399]: pgmap v1286: 184 pgs: 4 active+clean+remapped, 2 active+remapped+backfill_toofull, 4 remapped+peering, 174 active+clean; 596 KiB data, 723 MiB used, 625 GiB / 626 GiB avail; 2.9 KiB/s rd, 13 KiB/s wr, 3 op/s; 88 KiB/s, 0 objects/s recovering 2024-04-14T15:27:08.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:08 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/4274299571' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:27:08.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:08 smithi090 ceph-mon[41670]: pgmap v1286: 184 pgs: 4 active+clean+remapped, 2 active+remapped+backfill_toofull, 4 remapped+peering, 174 active+clean; 596 KiB data, 723 MiB used, 625 GiB / 626 GiB avail; 2.9 KiB/s rd, 13 KiB/s wr, 3 op/s; 88 KiB/s, 0 objects/s recovering 2024-04-14T15:27:08.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:08 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/4274299571' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:27:08.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:08 smithi092 ceph-mon[33966]: pgmap v1286: 184 pgs: 4 active+clean+remapped, 2 active+remapped+backfill_toofull, 4 remapped+peering, 174 active+clean; 596 KiB data, 723 MiB used, 625 GiB / 626 GiB avail; 2.9 KiB/s rd, 13 KiB/s wr, 3 op/s; 88 KiB/s, 0 objects/s recovering 2024-04-14T15:27:08.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:08 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/4274299571' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-14T15:27:08.992 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-14T15:27:09.215 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:09 smithi090 ceph-mon[30399]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:27:09.216 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:09 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/4274299571' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-14T15:27:09.216 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:09 smithi090 ceph-mon[30399]: osdmap e861: 8 total, 8 up, 7 in 2024-04-14T15:27:09.216 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:09 smithi090 ceph-mon[41670]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:27:09.216 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:09 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/4274299571' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-14T15:27:09.216 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:09 smithi090 ceph-mon[41670]: osdmap e861: 8 total, 8 up, 7 in 2024-04-14T15:27:09.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:09 smithi092 ceph-mon[33966]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:27:09.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:09 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/4274299571' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-14T15:27:09.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:09 smithi092 ceph-mon[33966]: osdmap e861: 8 total, 8 up, 7 in 2024-04-14T15:27:09.682 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:27:09.716 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:27:10.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:10 smithi090 ceph-mon[41670]: 5.12 scrub starts 2024-04-14T15:27:10.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:10 smithi090 ceph-mon[41670]: 5.12 scrub ok 2024-04-14T15:27:10.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:10 smithi090 ceph-mon[41670]: pgmap v1288: 184 pgs: 3 active+remapped+backfill_toofull, 181 active+clean; 596 KiB data, 723 MiB used, 625 GiB / 626 GiB avail; 3.3 KiB/s rd, 11 KiB/s wr, 3 op/s; 71 KiB/s, 0 objects/s recovering 2024-04-14T15:27:10.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:10 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-04-14T15:27:10.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:10 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:27:10.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:10 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:27:10.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:10 smithi090 ceph-mon[30399]: 5.12 scrub starts 2024-04-14T15:27:10.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:10 smithi090 ceph-mon[30399]: 5.12 scrub ok 2024-04-14T15:27:10.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:10 smithi090 ceph-mon[30399]: pgmap v1288: 184 pgs: 3 active+remapped+backfill_toofull, 181 active+clean; 596 KiB data, 723 MiB used, 625 GiB / 626 GiB avail; 3.3 KiB/s rd, 11 KiB/s wr, 3 op/s; 71 KiB/s, 0 objects/s recovering 2024-04-14T15:27:10.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:10 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-04-14T15:27:10.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:10 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:27:10.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:10 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:27:10.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:10 smithi092 ceph-mon[33966]: 5.12 scrub starts 2024-04-14T15:27:10.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:10 smithi092 ceph-mon[33966]: 5.12 scrub ok 2024-04-14T15:27:10.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:10 smithi092 ceph-mon[33966]: pgmap v1288: 184 pgs: 3 active+remapped+backfill_toofull, 181 active+clean; 596 KiB data, 723 MiB used, 625 GiB / 626 GiB avail; 3.3 KiB/s rd, 11 KiB/s wr, 3 op/s; 71 KiB/s, 0 objects/s recovering 2024-04-14T15:27:10.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:10 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-04-14T15:27:10.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:10 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:27:10.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:10 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:27:11.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:11 smithi090 ceph-mon[30399]: 4.a scrub starts 2024-04-14T15:27:11.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:11 smithi090 ceph-mon[30399]: 4.a scrub ok 2024-04-14T15:27:11.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:11 smithi090 ceph-mon[30399]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-04-14T15:27:11.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:11 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]': finished 2024-04-14T15:27:11.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:11 smithi090 ceph-mon[30399]: osdmap e862: 8 total, 8 up, 7 in 2024-04-14T15:27:11.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:11 smithi090 ceph-mon[30399]: 1.d scrub starts 2024-04-14T15:27:11.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:11 smithi090 ceph-mon[30399]: 1.d scrub ok 2024-04-14T15:27:11.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:11 smithi090 ceph-mon[30399]: 1.11 scrub starts 2024-04-14T15:27:11.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:11 smithi090 ceph-mon[30399]: 1.11 scrub ok 2024-04-14T15:27:11.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:11 smithi090 ceph-mon[41670]: 4.a scrub starts 2024-04-14T15:27:11.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:11 smithi090 ceph-mon[41670]: 4.a scrub ok 2024-04-14T15:27:11.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:11 smithi090 ceph-mon[41670]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-04-14T15:27:11.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:11 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]': finished 2024-04-14T15:27:11.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:11 smithi090 ceph-mon[41670]: osdmap e862: 8 total, 8 up, 7 in 2024-04-14T15:27:11.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:11 smithi090 ceph-mon[41670]: 1.d scrub starts 2024-04-14T15:27:11.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:11 smithi090 ceph-mon[41670]: 1.d scrub ok 2024-04-14T15:27:11.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:11 smithi090 ceph-mon[41670]: 1.11 scrub starts 2024-04-14T15:27:11.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:11 smithi090 ceph-mon[41670]: 1.11 scrub ok 2024-04-14T15:27:11.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:11 smithi092 ceph-mon[33966]: 4.a scrub starts 2024-04-14T15:27:11.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:11 smithi092 ceph-mon[33966]: 4.a scrub ok 2024-04-14T15:27:11.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:11 smithi092 ceph-mon[33966]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-04-14T15:27:11.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:11 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]': finished 2024-04-14T15:27:11.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:11 smithi092 ceph-mon[33966]: osdmap e862: 8 total, 8 up, 7 in 2024-04-14T15:27:11.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:11 smithi092 ceph-mon[33966]: 1.d scrub starts 2024-04-14T15:27:11.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:11 smithi092 ceph-mon[33966]: 1.d scrub ok 2024-04-14T15:27:11.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:11 smithi092 ceph-mon[33966]: 1.11 scrub starts 2024-04-14T15:27:11.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:11 smithi092 ceph-mon[33966]: 1.11 scrub ok 2024-04-14T15:27:12.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:11.999+0000 7f5af97cd700 1 -- 172.21.15.90:0/2111869429 >> v1:172.21.15.90:6789/0 conn(0x7f5af410b720 legacy=0x7f5af410db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:12.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:11.999+0000 7f5af97cd700 1 -- 172.21.15.90:0/2111869429 shutdown_connections 2024-04-14T15:27:12.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:11.999+0000 7f5af97cd700 1 -- 172.21.15.90:0/2111869429 >> 172.21.15.90:0/2111869429 conn(0x7f5af40fe3c0 msgr2=0x7f5af41007e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:27:12.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:11.999+0000 7f5af97cd700 1 -- 172.21.15.90:0/2111869429 shutdown_connections 2024-04-14T15:27:12.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:11.999+0000 7f5af97cd700 1 -- 172.21.15.90:0/2111869429 wait complete. 2024-04-14T15:27:12.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:12.000+0000 7f5af97cd700 1 librados: starting msgr at 2024-04-14T15:27:12.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:12.000+0000 7f5af97cd700 1 librados: starting objecter 2024-04-14T15:27:12.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:12.000+0000 7f5af97cd700 1 Processor -- start 2024-04-14T15:27:12.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:12.000+0000 7f5af97cd700 1 -- start start 2024-04-14T15:27:12.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:12.000+0000 7f5af97cd700 1 librados: setting wanted keys 2024-04-14T15:27:12.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:12.000+0000 7f5af97cd700 1 librados: calling monclient init 2024-04-14T15:27:12.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:12.000+0000 7f5af97cd700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5af41a9ba0 con 0x7f5af410f2f0 2024-04-14T15:27:12.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:12.000+0000 7f5af97cd700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5af41aadc0 con 0x7f5af410b720 2024-04-14T15:27:12.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:12.000+0000 7f5af97cd700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5af41ac000 con 0x7f5af4108870 2024-04-14T15:27:12.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:12.001+0000 7f5af2ffd700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f5af4108870 0x7f5af4076620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:56936/0 (socket says 172.21.15.90:56936) 2024-04-14T15:27:12.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:12.001+0000 7f5af2ffd700 1 -- 172.21.15.90:0/1669309110 learned_addr learned my addr 172.21.15.90:0/1669309110 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:27:12.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:12.001+0000 7f5adbfff700 10 client.?.objecter ms_handle_connect 0x7f5af410f2f0 2024-04-14T15:27:12.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:12.001+0000 7f5adbfff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:27:12.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:12.001+0000 7f5adbfff700 10 client.?.objecter ms_handle_connect 0x7f5af4108870 2024-04-14T15:27:12.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:12.001+0000 7f5adbfff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:27:12.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:12.001+0000 7f5adbfff700 10 client.?.objecter ms_handle_connect 0x7f5af410b720 2024-04-14T15:27:12.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:12.001+0000 7f5adbfff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:27:12.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:12.001+0000 7f5adbfff700 1 -- 172.21.15.90:0/1669309110 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 682117135 0 0) 0x7f5af41a9ba0 con 0x7f5af410f2f0 2024-04-14T15:27:12.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:12.001+0000 7f5adbfff700 1 -- 172.21.15.90:0/1669309110 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5adc0035b0 con 0x7f5af410f2f0 2024-04-14T15:27:12.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:12.001+0000 7f5adbfff700 1 -- 172.21.15.90:0/1669309110 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 714630088 0 0) 0x7f5af41ac000 con 0x7f5af4108870 2024-04-14T15:27:12.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:12.001+0000 7f5adbfff700 1 -- 172.21.15.90:0/1669309110 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5af41a9ba0 con 0x7f5af4108870 2024-04-14T15:27:12.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:12.001+0000 7f5adbfff700 1 -- 172.21.15.90:0/1669309110 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1016461049 0 0) 0x7f5af41aadc0 con 0x7f5af410b720 2024-04-14T15:27:12.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:12.001+0000 7f5adbfff700 1 -- 172.21.15.90:0/1669309110 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5af41ac000 con 0x7f5af410b720 2024-04-14T15:27:12.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:12.001+0000 7f5adbfff700 1 -- 172.21.15.90:0/1669309110 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2112964310 0 0) 0x7f5adc0035b0 con 0x7f5af410f2f0 2024-04-14T15:27:12.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:12.001+0000 7f5adbfff700 1 -- 172.21.15.90:0/1669309110 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5af41aadc0 con 0x7f5af410f2f0 2024-04-14T15:27:12.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:12.001+0000 7f5adbfff700 1 -- 172.21.15.90:0/1669309110 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f5aec001c30 con 0x7f5af410f2f0 2024-04-14T15:27:12.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:12.001+0000 7f5adbfff700 1 -- 172.21.15.90:0/1669309110 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2169271013 0 0) 0x7f5af41a9ba0 con 0x7f5af4108870 2024-04-14T15:27:12.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:12.001+0000 7f5adbfff700 1 -- 172.21.15.90:0/1669309110 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5adc0035b0 con 0x7f5af4108870 2024-04-14T15:27:12.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:12.001+0000 7f5adbfff700 1 -- 172.21.15.90:0/1669309110 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f5ae40018b0 con 0x7f5af4108870 2024-04-14T15:27:12.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:12.001+0000 7f5adbfff700 1 -- 172.21.15.90:0/1669309110 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2985757312 0 0) 0x7f5af41aadc0 con 0x7f5af410f2f0 2024-04-14T15:27:12.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:12.001+0000 7f5adbfff700 1 -- 172.21.15.90:0/1669309110 >> v1:172.21.15.90:6790/0 conn(0x7f5af4108870 legacy=0x7f5af4076620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:12.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:12.001+0000 7f5adbfff700 1 -- 172.21.15.90:0/1669309110 >> v1:172.21.15.92:6789/0 conn(0x7f5af410b720 legacy=0x7f5af41a4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:12.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:12.001+0000 7f5adbfff700 1 -- 172.21.15.90:0/1669309110 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5af41ad240 con 0x7f5af410f2f0 2024-04-14T15:27:12.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:12.001+0000 7f5af97cd700 1 -- 172.21.15.90:0/1669309110 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5af41ac230 con 0x7f5af410f2f0 2024-04-14T15:27:12.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:12.001+0000 7f5af97cd700 10 client.38181.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:27:12.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:12.001+0000 7f5af97cd700 1 -- 172.21.15.90:0/1669309110 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f5af41ac6f0 con 0x7f5af410f2f0 2024-04-14T15:27:12.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:12.001+0000 7f5af97cd700 1 librados: init done 2024-04-14T15:27:12.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:12.002+0000 7f5adbfff700 1 -- 172.21.15.90:0/1669309110 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f5aec003ec0 con 0x7f5af410f2f0 2024-04-14T15:27:12.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:12.002+0000 7f5adbfff700 1 -- 172.21.15.90:0/1669309110 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f5aec004de0 con 0x7f5af410f2f0 2024-04-14T15:27:12.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:12.002+0000 7f5af97cd700 1 -- 172.21.15.90:0/1669309110 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ae0005120 con 0x7f5af410f2f0 2024-04-14T15:27:12.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:12.003+0000 7f5adbfff700 1 -- 172.21.15.90:0/1669309110 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f5aec003a20 con 0x7f5af410f2f0 2024-04-14T15:27:12.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:12.003+0000 7f5adbfff700 1 -- 172.21.15.90:0/1669309110 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(863..863 src has 467..863) ==== 7616+0+0 (unknown 3423360331 0 0) 0x7f5aec089930 con 0x7f5af410f2f0 2024-04-14T15:27:12.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:12.003+0000 7f5adbfff700 10 client.38181.objecter ms_dispatch 0x7f5af4001550 osd_map(863..863 src has 467..863) 2024-04-14T15:27:12.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:12.003+0000 7f5adbfff700 3 client.38181.objecter handle_osd_map got epochs [863,863] > 0 2024-04-14T15:27:12.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:12.003+0000 7f5adbfff700 3 client.38181.objecter handle_osd_map decoding full epoch 863 2024-04-14T15:27:12.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:12.003+0000 7f5adbfff700 20 client.38181.objecter dump_active .. 0 homeless 2024-04-14T15:27:12.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:12.004+0000 7f5adbfff700 10 client.38181.objecter ms_handle_connect 0x7f5adc071280 2024-04-14T15:27:12.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:12.008+0000 7f5adbfff700 1 -- 172.21.15.90:0/1669309110 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f5aec056570 con 0x7f5af410f2f0 2024-04-14T15:27:12.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:12.242+0000 7f5af97cd700 1 -- 172.21.15.90:0/1669309110 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd in", "ids": ["6"]} v 0) -- 0x7f5ae0005410 con 0x7f5af410f2f0 2024-04-14T15:27:12.393 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:12 smithi090 ceph-mon[41670]: 2.b scrub starts 2024-04-14T15:27:12.393 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:12 smithi090 ceph-mon[41670]: 2.b scrub ok 2024-04-14T15:27:12.394 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:12 smithi090 ceph-mon[41670]: 1.f scrub starts 2024-04-14T15:27:12.394 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:12 smithi090 ceph-mon[41670]: 1.f scrub ok 2024-04-14T15:27:12.394 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:12 smithi090 ceph-mon[41670]: osdmap e863: 8 total, 8 up, 7 in 2024-04-14T15:27:12.394 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:12 smithi090 ceph-mon[41670]: 1.5 scrub starts 2024-04-14T15:27:12.394 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:12 smithi090 ceph-mon[41670]: 1.5 scrub ok 2024-04-14T15:27:12.394 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:12 smithi090 ceph-mon[41670]: pgmap v1291: 184 pgs: 3 active+remapped+backfill_toofull, 181 active+clean; 596 KiB data, 723 MiB used, 625 GiB / 626 GiB avail; 989 B/s rd, 0 op/s 2024-04-14T15:27:12.394 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:12 smithi090 ceph-mon[30399]: 2.b scrub starts 2024-04-14T15:27:12.394 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:12 smithi090 ceph-mon[30399]: 2.b scrub ok 2024-04-14T15:27:12.394 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:12 smithi090 ceph-mon[30399]: 1.f scrub starts 2024-04-14T15:27:12.394 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:12 smithi090 ceph-mon[30399]: 1.f scrub ok 2024-04-14T15:27:12.394 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:12 smithi090 ceph-mon[30399]: osdmap e863: 8 total, 8 up, 7 in 2024-04-14T15:27:12.395 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:12 smithi090 ceph-mon[30399]: 1.5 scrub starts 2024-04-14T15:27:12.395 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:12 smithi090 ceph-mon[30399]: 1.5 scrub ok 2024-04-14T15:27:12.395 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:12 smithi090 ceph-mon[30399]: pgmap v1291: 184 pgs: 3 active+remapped+backfill_toofull, 181 active+clean; 596 KiB data, 723 MiB used, 625 GiB / 626 GiB avail; 989 B/s rd, 0 op/s 2024-04-14T15:27:12.395 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:12 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mon-a[30376]: 2024-04-14T15:27:12.243+0000 7f0166ba8700 -1 mon.a@0(leader).osd e864 definitely_dead 0 2024-04-14T15:27:12.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:12 smithi092 ceph-mon[33966]: 2.b scrub starts 2024-04-14T15:27:12.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:12 smithi092 ceph-mon[33966]: 2.b scrub ok 2024-04-14T15:27:12.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:12 smithi092 ceph-mon[33966]: 1.f scrub starts 2024-04-14T15:27:12.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:12 smithi092 ceph-mon[33966]: 1.f scrub ok 2024-04-14T15:27:12.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:12 smithi092 ceph-mon[33966]: osdmap e863: 8 total, 8 up, 7 in 2024-04-14T15:27:12.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:12 smithi092 ceph-mon[33966]: 1.5 scrub starts 2024-04-14T15:27:12.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:12 smithi092 ceph-mon[33966]: 1.5 scrub ok 2024-04-14T15:27:12.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:12 smithi092 ceph-mon[33966]: pgmap v1291: 184 pgs: 3 active+remapped+backfill_toofull, 181 active+clean; 596 KiB data, 723 MiB used, 625 GiB / 626 GiB avail; 989 B/s rd, 0 op/s 2024-04-14T15:27:13.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:13.166+0000 7f5adbfff700 1 -- 172.21.15.90:0/1669309110 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd in", "ids": ["6"]}]=0 marked in osd.6. v865) ==== 85+0+0 (unknown 3163030661 0 0) 0x7f5aec05a220 con 0x7f5af410f2f0 2024-04-14T15:27:13.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:13.169+0000 7f5af97cd700 10 librados: watch_flush enter 2024-04-14T15:27:13.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:13.171+0000 7f5af97cd700 10 librados: watch_flush exit 2024-04-14T15:27:13.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:13.171+0000 7f5af97cd700 20 client.38181.objecter shutdown clearing up homeless session... 2024-04-14T15:27:13.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:13.171+0000 7f5af97cd700 10 client.38181.objecter successfully canceled tick 2024-04-14T15:27:13.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:13.171+0000 7f5af97cd700 1 -- 172.21.15.90:0/1669309110 >> v1:172.21.15.90:6800/420935427 conn(0x7f5adc071280 legacy=0x7f5adc073740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:13.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:13.171+0000 7f5af97cd700 1 -- 172.21.15.90:0/1669309110 >> v1:172.21.15.90:6789/0 conn(0x7f5af410f2f0 legacy=0x7f5af41a82c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:13.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:13.173+0000 7f5af97cd700 1 -- 172.21.15.90:0/1669309110 shutdown_connections 2024-04-14T15:27:13.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:13.173+0000 7f5af97cd700 1 -- 172.21.15.90:0/1669309110 >> 172.21.15.90:0/1669309110 conn(0x7f5af40fe3c0 msgr2=0x7f5af4101820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:27:13.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:13.173+0000 7f5af97cd700 1 -- 172.21.15.90:0/1669309110 shutdown_connections 2024-04-14T15:27:13.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:13.174+0000 7f5af97cd700 1 -- 172.21.15.90:0/1669309110 wait complete. 2024-04-14T15:27:13.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:13.174+0000 7f5af97cd700 1 librados: shutdown 2024-04-14T15:27:13.176 INFO:teuthology.orchestra.run.smithi090.stderr:marked in osd.6. 2024-04-14T15:27:13.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:13 smithi090 ceph-mon[41670]: osdmap e864: 8 total, 8 up, 7 in 2024-04-14T15:27:13.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:13 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1669309110' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-04-14T15:27:13.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:13 smithi090 ceph-mon[30399]: osdmap e864: 8 total, 8 up, 7 in 2024-04-14T15:27:13.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:13 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1669309110' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-04-14T15:27:13.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:13 smithi092 ceph-mon[33966]: osdmap e864: 8 total, 8 up, 7 in 2024-04-14T15:27:13.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:13 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1669309110' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-04-14T15:27:13.903 INFO:tasks.thrashosds.thrasher:Added osd 6 2024-04-14T15:27:13.904 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-04-14T15:27:13.904 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd pool set .mgr pgp_num 11 2024-04-14T15:27:14.142 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:27:13 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:27:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:27:14.178 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-14T15:27:14.178 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:27:14.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:14 smithi090 ceph-mon[41670]: 5.6 scrub starts 2024-04-14T15:27:14.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:14 smithi090 ceph-mon[41670]: 5.6 scrub ok 2024-04-14T15:27:14.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:14 smithi090 ceph-mon[41670]: 6.3 scrub starts 2024-04-14T15:27:14.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:14 smithi090 ceph-mon[41670]: 6.3 scrub ok 2024-04-14T15:27:14.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:14 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1669309110' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-04-14T15:27:14.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:14 smithi090 ceph-mon[41670]: osdmap e865: 8 total, 8 up, 8 in 2024-04-14T15:27:14.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:14 smithi090 ceph-mon[41670]: pgmap v1294: 183 pgs: 3 active+remapped+backfill_toofull, 180 active+clean; 596 KiB data, 723 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:27:14.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:14 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-04-14T15:27:14.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:14 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]': finished 2024-04-14T15:27:14.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:14 smithi090 ceph-mon[41670]: osdmap e866: 8 total, 8 up, 8 in 2024-04-14T15:27:14.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:14 smithi090 ceph-mon[30399]: 5.6 scrub starts 2024-04-14T15:27:14.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:14 smithi090 ceph-mon[30399]: 5.6 scrub ok 2024-04-14T15:27:14.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:14 smithi090 ceph-mon[30399]: 6.3 scrub starts 2024-04-14T15:27:14.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:14 smithi090 ceph-mon[30399]: 6.3 scrub ok 2024-04-14T15:27:14.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:14 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1669309110' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-04-14T15:27:14.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:14 smithi090 ceph-mon[30399]: osdmap e865: 8 total, 8 up, 8 in 2024-04-14T15:27:14.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:14 smithi090 ceph-mon[30399]: pgmap v1294: 183 pgs: 3 active+remapped+backfill_toofull, 180 active+clean; 596 KiB data, 723 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:27:14.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:14 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-04-14T15:27:14.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:14 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]': finished 2024-04-14T15:27:14.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:14 smithi090 ceph-mon[30399]: osdmap e866: 8 total, 8 up, 8 in 2024-04-14T15:27:14.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:14 smithi092 ceph-mon[33966]: 5.6 scrub starts 2024-04-14T15:27:14.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:14 smithi092 ceph-mon[33966]: 5.6 scrub ok 2024-04-14T15:27:14.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:14 smithi092 ceph-mon[33966]: 6.3 scrub starts 2024-04-14T15:27:14.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:14 smithi092 ceph-mon[33966]: 6.3 scrub ok 2024-04-14T15:27:14.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:14 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1669309110' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-04-14T15:27:14.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:14 smithi092 ceph-mon[33966]: osdmap e865: 8 total, 8 up, 8 in 2024-04-14T15:27:14.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:14 smithi092 ceph-mon[33966]: pgmap v1294: 183 pgs: 3 active+remapped+backfill_toofull, 180 active+clean; 596 KiB data, 723 MiB used, 625 GiB / 626 GiB avail 2024-04-14T15:27:14.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:14 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-04-14T15:27:14.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:14 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]': finished 2024-04-14T15:27:14.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:14 smithi092 ceph-mon[33966]: osdmap e866: 8 total, 8 up, 8 in 2024-04-14T15:27:14.889 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:27:14.922 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-14T15:27:15.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:15 smithi090 ceph-mon[30399]: 5.19 scrub starts 2024-04-14T15:27:15.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:15 smithi090 ceph-mon[30399]: 5.19 scrub ok 2024-04-14T15:27:15.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:15 smithi090 ceph-mon[30399]: 2.8 scrub starts 2024-04-14T15:27:15.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:15 smithi090 ceph-mon[30399]: 2.8 scrub ok 2024-04-14T15:27:15.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:15 smithi090 ceph-mon[30399]: 5.e scrub starts 2024-04-14T15:27:15.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:15 smithi090 ceph-mon[30399]: 5.e scrub ok 2024-04-14T15:27:15.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:15 smithi090 ceph-mon[30399]: 1.b continuing backfill to osd.4 from (91'130,91'130] MIN to 91'130 2024-04-14T15:27:15.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:15 smithi090 ceph-mon[30399]: 1.b starting backfill to osd.6 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:15.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:15 smithi090 ceph-mon[30399]: 1.e continuing backfill to osd.0 from (91'130,91'130] MIN to 91'130 2024-04-14T15:27:15.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:15 smithi090 ceph-mon[30399]: 1.e starting backfill to osd.6 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:15.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:15 smithi090 ceph-mon[30399]: osdmap e867: 8 total, 8 up, 8 in 2024-04-14T15:27:15.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:15 smithi090 ceph-mon[41670]: 5.19 scrub starts 2024-04-14T15:27:15.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:15 smithi090 ceph-mon[41670]: 5.19 scrub ok 2024-04-14T15:27:15.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:15 smithi090 ceph-mon[41670]: 2.8 scrub starts 2024-04-14T15:27:15.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:15 smithi090 ceph-mon[41670]: 2.8 scrub ok 2024-04-14T15:27:15.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:15 smithi090 ceph-mon[41670]: 5.e scrub starts 2024-04-14T15:27:15.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:15 smithi090 ceph-mon[41670]: 5.e scrub ok 2024-04-14T15:27:15.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:15 smithi090 ceph-mon[41670]: 1.b continuing backfill to osd.4 from (91'130,91'130] MIN to 91'130 2024-04-14T15:27:15.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:15 smithi090 ceph-mon[41670]: 1.b starting backfill to osd.6 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:15.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:15 smithi090 ceph-mon[41670]: 1.e continuing backfill to osd.0 from (91'130,91'130] MIN to 91'130 2024-04-14T15:27:15.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:15 smithi090 ceph-mon[41670]: 1.e starting backfill to osd.6 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:15.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:15 smithi090 ceph-mon[41670]: osdmap e867: 8 total, 8 up, 8 in 2024-04-14T15:27:15.578 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:15 smithi092 ceph-mon[33966]: 5.19 scrub starts 2024-04-14T15:27:15.578 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:15 smithi092 ceph-mon[33966]: 5.19 scrub ok 2024-04-14T15:27:15.578 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:15 smithi092 ceph-mon[33966]: 2.8 scrub starts 2024-04-14T15:27:15.578 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:15 smithi092 ceph-mon[33966]: 2.8 scrub ok 2024-04-14T15:27:15.578 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:15 smithi092 ceph-mon[33966]: 5.e scrub starts 2024-04-14T15:27:15.578 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:15 smithi092 ceph-mon[33966]: 5.e scrub ok 2024-04-14T15:27:15.578 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:15 smithi092 ceph-mon[33966]: 1.b continuing backfill to osd.4 from (91'130,91'130] MIN to 91'130 2024-04-14T15:27:15.578 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:15 smithi092 ceph-mon[33966]: 1.b starting backfill to osd.6 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:15.578 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:15 smithi092 ceph-mon[33966]: 1.e continuing backfill to osd.0 from (91'130,91'130] MIN to 91'130 2024-04-14T15:27:15.579 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:15 smithi092 ceph-mon[33966]: 1.e starting backfill to osd.6 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:15.579 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:15 smithi092 ceph-mon[33966]: osdmap e867: 8 total, 8 up, 8 in 2024-04-14T15:27:15.872 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:27:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:27:16.169 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:27:16.518 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:16 smithi090 ceph-mon[41670]: 2.10 scrub starts 2024-04-14T15:27:16.518 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:16 smithi090 ceph-mon[41670]: 2.10 scrub ok 2024-04-14T15:27:16.518 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:16 smithi090 ceph-mon[41670]: 2.1f scrub starts 2024-04-14T15:27:16.518 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:16 smithi090 ceph-mon[41670]: 2.1f scrub ok 2024-04-14T15:27:16.518 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:16 smithi090 ceph-mon[41670]: 6.15 scrub starts 2024-04-14T15:27:16.518 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:16 smithi090 ceph-mon[41670]: 6.15 scrub ok 2024-04-14T15:27:16.518 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:16 smithi090 ceph-mon[41670]: 1.e scrub starts 2024-04-14T15:27:16.518 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:16 smithi090 ceph-mon[41670]: 1.e scrub ok 2024-04-14T15:27:16.518 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:16 smithi090 ceph-mon[41670]: 1.13 starting backfill to osd.6 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:16.519 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:16 smithi090 ceph-mon[41670]: 1.3 starting backfill to osd.6 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:16.519 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:16 smithi090 ceph-mon[41670]: 1.0 starting backfill to osd.6 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:16.519 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:16 smithi090 ceph-mon[41670]: 1.10 starting backfill to osd.6 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:16.519 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:16 smithi090 ceph-mon[41670]: 1.d starting backfill to osd.6 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:16.519 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:16 smithi090 ceph-mon[41670]: 1.a starting backfill to osd.6 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:16.519 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:16 smithi090 ceph-mon[41670]: 1.5 starting backfill to osd.6 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:16.519 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:16 smithi090 ceph-mon[41670]: 1.f starting backfill to osd.6 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:16.519 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:16 smithi090 ceph-mon[41670]: 1.7 starting backfill to osd.6 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:16.519 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:16 smithi090 ceph-mon[41670]: 1.6 starting backfill to osd.6 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:16.519 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:16 smithi090 ceph-mon[41670]: pgmap v1297: 183 pgs: 15 activating, 113 active+clean, 1 active+remapped+backfill_toofull, 46 peering, 2 unknown, 3 remapped+peering, 1 activating+degraded, 2 activating+remapped; 596 KiB data, 818 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s; 11/597 objects degraded (1.843%) 2024-04-14T15:27:16.519 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:16 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:27:16.519 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:16 smithi090 ceph-mon[41670]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-04-14T15:27:16.519 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:16 smithi090 ceph-mon[41670]: osdmap e868: 8 total, 8 up, 8 in 2024-04-14T15:27:16.520 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:16 smithi090 ceph-mon[30399]: 2.10 scrub starts 2024-04-14T15:27:16.520 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:16 smithi090 ceph-mon[30399]: 2.10 scrub ok 2024-04-14T15:27:16.520 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:16 smithi090 ceph-mon[30399]: 2.1f scrub starts 2024-04-14T15:27:16.520 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:16 smithi090 ceph-mon[30399]: 2.1f scrub ok 2024-04-14T15:27:16.520 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:16 smithi090 ceph-mon[30399]: 6.15 scrub starts 2024-04-14T15:27:16.520 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:16 smithi090 ceph-mon[30399]: 6.15 scrub ok 2024-04-14T15:27:16.520 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:16 smithi090 ceph-mon[30399]: 1.e scrub starts 2024-04-14T15:27:16.520 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:16 smithi090 ceph-mon[30399]: 1.e scrub ok 2024-04-14T15:27:16.520 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:16 smithi090 ceph-mon[30399]: 1.13 starting backfill to osd.6 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:16.520 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:16 smithi090 ceph-mon[30399]: 1.3 starting backfill to osd.6 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:16.520 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:16 smithi090 ceph-mon[30399]: 1.0 starting backfill to osd.6 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:16.520 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:16 smithi090 ceph-mon[30399]: 1.10 starting backfill to osd.6 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:16.520 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:16 smithi090 ceph-mon[30399]: 1.d starting backfill to osd.6 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:16.521 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:16 smithi090 ceph-mon[30399]: 1.a starting backfill to osd.6 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:16.521 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:16 smithi090 ceph-mon[30399]: 1.5 starting backfill to osd.6 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:16.521 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:16 smithi090 ceph-mon[30399]: 1.f starting backfill to osd.6 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:16.521 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:16 smithi090 ceph-mon[30399]: 1.7 starting backfill to osd.6 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:16.521 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:16 smithi090 ceph-mon[30399]: 1.6 starting backfill to osd.6 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:16.521 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:16 smithi090 ceph-mon[30399]: pgmap v1297: 183 pgs: 15 activating, 113 active+clean, 1 active+remapped+backfill_toofull, 46 peering, 2 unknown, 3 remapped+peering, 1 activating+degraded, 2 activating+remapped; 596 KiB data, 818 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s; 11/597 objects degraded (1.843%) 2024-04-14T15:27:16.521 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:16 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:27:16.521 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:16 smithi090 ceph-mon[30399]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-04-14T15:27:16.521 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:16 smithi090 ceph-mon[30399]: osdmap e868: 8 total, 8 up, 8 in 2024-04-14T15:27:16.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:16 smithi092 ceph-mon[33966]: 2.10 scrub starts 2024-04-14T15:27:16.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:16 smithi092 ceph-mon[33966]: 2.10 scrub ok 2024-04-14T15:27:16.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:16 smithi092 ceph-mon[33966]: 2.1f scrub starts 2024-04-14T15:27:16.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:16 smithi092 ceph-mon[33966]: 2.1f scrub ok 2024-04-14T15:27:16.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:16 smithi092 ceph-mon[33966]: 6.15 scrub starts 2024-04-14T15:27:16.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:16 smithi092 ceph-mon[33966]: 6.15 scrub ok 2024-04-14T15:27:16.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:16 smithi092 ceph-mon[33966]: 1.e scrub starts 2024-04-14T15:27:16.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:16 smithi092 ceph-mon[33966]: 1.e scrub ok 2024-04-14T15:27:16.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:16 smithi092 ceph-mon[33966]: 1.13 starting backfill to osd.6 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:16.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:16 smithi092 ceph-mon[33966]: 1.3 starting backfill to osd.6 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:16.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:16 smithi092 ceph-mon[33966]: 1.0 starting backfill to osd.6 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:16.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:16 smithi092 ceph-mon[33966]: 1.10 starting backfill to osd.6 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:16.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:16 smithi092 ceph-mon[33966]: 1.d starting backfill to osd.6 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:16.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:16 smithi092 ceph-mon[33966]: 1.a starting backfill to osd.6 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:16.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:16 smithi092 ceph-mon[33966]: 1.5 starting backfill to osd.6 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:16.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:16 smithi092 ceph-mon[33966]: 1.f starting backfill to osd.6 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:16.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:16 smithi092 ceph-mon[33966]: 1.7 starting backfill to osd.6 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:16.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:16 smithi092 ceph-mon[33966]: 1.6 starting backfill to osd.6 from (0'0,0'0] MAX to 91'130 2024-04-14T15:27:16.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:16 smithi092 ceph-mon[33966]: pgmap v1297: 183 pgs: 15 activating, 113 active+clean, 1 active+remapped+backfill_toofull, 46 peering, 2 unknown, 3 remapped+peering, 1 activating+degraded, 2 activating+remapped; 596 KiB data, 818 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s; 11/597 objects degraded (1.843%) 2024-04-14T15:27:16.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:16 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:27:16.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:16 smithi092 ceph-mon[33966]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-04-14T15:27:16.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:16 smithi092 ceph-mon[33966]: osdmap e868: 8 total, 8 up, 8 in 2024-04-14T15:27:17.404 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:17 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:27:17.404 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:17 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:27:17.404 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:17 smithi090 ceph-mon[41670]: 11.3 scrub starts 2024-04-14T15:27:17.404 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:17 smithi090 ceph-mon[41670]: 11.3 scrub ok 2024-04-14T15:27:17.404 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:17 smithi090 ceph-mon[41670]: 6.1d scrub starts 2024-04-14T15:27:17.404 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:17 smithi090 ceph-mon[41670]: 6.1d scrub ok 2024-04-14T15:27:17.404 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:17 smithi090 ceph-mon[41670]: 4.11 scrub starts 2024-04-14T15:27:17.404 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:17 smithi090 ceph-mon[41670]: 4.11 scrub ok 2024-04-14T15:27:17.404 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:17 smithi090 ceph-mon[41670]: Health check failed: Reduced data availability: 11 pgs inactive, 12 pgs peering (PG_AVAILABILITY) 2024-04-14T15:27:17.404 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:17 smithi090 ceph-mon[41670]: Health check failed: Degraded data redundancy: 11/597 objects degraded (1.843%), 1 pg degraded (PG_DEGRADED) 2024-04-14T15:27:17.404 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:17 smithi090 ceph-mon[41670]: osdmap e869: 8 total, 8 up, 8 in 2024-04-14T15:27:17.405 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:17 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:27:17.405 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:17 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:27:17.405 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:17 smithi090 ceph-mon[30399]: 11.3 scrub starts 2024-04-14T15:27:17.405 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:17 smithi090 ceph-mon[30399]: 11.3 scrub ok 2024-04-14T15:27:17.405 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:17 smithi090 ceph-mon[30399]: 6.1d scrub starts 2024-04-14T15:27:17.405 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:17 smithi090 ceph-mon[30399]: 6.1d scrub ok 2024-04-14T15:27:17.405 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:17 smithi090 ceph-mon[30399]: 4.11 scrub starts 2024-04-14T15:27:17.405 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:17 smithi090 ceph-mon[30399]: 4.11 scrub ok 2024-04-14T15:27:17.405 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:17 smithi090 ceph-mon[30399]: Health check failed: Reduced data availability: 11 pgs inactive, 12 pgs peering (PG_AVAILABILITY) 2024-04-14T15:27:17.405 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:17 smithi090 ceph-mon[30399]: Health check failed: Degraded data redundancy: 11/597 objects degraded (1.843%), 1 pg degraded (PG_DEGRADED) 2024-04-14T15:27:17.405 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:17 smithi090 ceph-mon[30399]: osdmap e869: 8 total, 8 up, 8 in 2024-04-14T15:27:17.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:17 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:27:17.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:17 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:27:17.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:17 smithi092 ceph-mon[33966]: 11.3 scrub starts 2024-04-14T15:27:17.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:17 smithi092 ceph-mon[33966]: 11.3 scrub ok 2024-04-14T15:27:17.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:17 smithi092 ceph-mon[33966]: 6.1d scrub starts 2024-04-14T15:27:17.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:17 smithi092 ceph-mon[33966]: 6.1d scrub ok 2024-04-14T15:27:17.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:17 smithi092 ceph-mon[33966]: 4.11 scrub starts 2024-04-14T15:27:17.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:17 smithi092 ceph-mon[33966]: 4.11 scrub ok 2024-04-14T15:27:17.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:17 smithi092 ceph-mon[33966]: Health check failed: Reduced data availability: 11 pgs inactive, 12 pgs peering (PG_AVAILABILITY) 2024-04-14T15:27:17.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:17 smithi092 ceph-mon[33966]: Health check failed: Degraded data redundancy: 11/597 objects degraded (1.843%), 1 pg degraded (PG_DEGRADED) 2024-04-14T15:27:17.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:17 smithi092 ceph-mon[33966]: osdmap e869: 8 total, 8 up, 8 in 2024-04-14T15:27:18.485 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:18 smithi090 ceph-mon[41670]: 6.f scrub starts 2024-04-14T15:27:18.486 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:18 smithi090 ceph-mon[41670]: 6.f scrub ok 2024-04-14T15:27:18.486 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:18 smithi090 ceph-mon[41670]: 1.e scrub starts 2024-04-14T15:27:18.486 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:18 smithi090 ceph-mon[41670]: 1.e scrub ok 2024-04-14T15:27:18.486 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:18 smithi090 ceph-mon[41670]: 11.f scrub starts 2024-04-14T15:27:18.486 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:18 smithi090 ceph-mon[41670]: 11.f scrub ok 2024-04-14T15:27:18.486 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:18 smithi090 ceph-mon[41670]: pgmap v1300: 182 pgs: 15 activating, 113 active+clean, 1 active+remapped+backfill_toofull, 46 peering, 2 unknown, 2 remapped+peering, 1 activating+degraded, 2 activating+remapped; 596 KiB data, 818 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s; 11/597 objects degraded (1.843%) 2024-04-14T15:27:18.486 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:18 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2059844821' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:27:18.486 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:18 smithi090 ceph-mon[30399]: 6.f scrub starts 2024-04-14T15:27:18.487 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:18 smithi090 ceph-mon[30399]: 6.f scrub ok 2024-04-14T15:27:18.487 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:18 smithi090 ceph-mon[30399]: 1.e scrub starts 2024-04-14T15:27:18.487 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:18 smithi090 ceph-mon[30399]: 1.e scrub ok 2024-04-14T15:27:18.487 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:18 smithi090 ceph-mon[30399]: 11.f scrub starts 2024-04-14T15:27:18.487 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:18 smithi090 ceph-mon[30399]: 11.f scrub ok 2024-04-14T15:27:18.487 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:18 smithi090 ceph-mon[30399]: pgmap v1300: 182 pgs: 15 activating, 113 active+clean, 1 active+remapped+backfill_toofull, 46 peering, 2 unknown, 2 remapped+peering, 1 activating+degraded, 2 activating+remapped; 596 KiB data, 818 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s; 11/597 objects degraded (1.843%) 2024-04-14T15:27:18.487 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:18 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2059844821' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:27:18.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:18 smithi092 ceph-mon[33966]: 6.f scrub starts 2024-04-14T15:27:18.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:18 smithi092 ceph-mon[33966]: 6.f scrub ok 2024-04-14T15:27:18.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:18 smithi092 ceph-mon[33966]: 1.e scrub starts 2024-04-14T15:27:18.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:18 smithi092 ceph-mon[33966]: 1.e scrub ok 2024-04-14T15:27:18.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:18 smithi092 ceph-mon[33966]: 11.f scrub starts 2024-04-14T15:27:18.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:18 smithi092 ceph-mon[33966]: 11.f scrub ok 2024-04-14T15:27:18.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:18 smithi092 ceph-mon[33966]: pgmap v1300: 182 pgs: 15 activating, 113 active+clean, 1 active+remapped+backfill_toofull, 46 peering, 2 unknown, 2 remapped+peering, 1 activating+degraded, 2 activating+remapped; 596 KiB data, 818 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s; 11/597 objects degraded (1.843%) 2024-04-14T15:27:18.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:18 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2059844821' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-14T15:27:18.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:17.406+0000 7f7d5ffe7700 1 -- 172.21.15.90:0/248705291 >> v1:172.21.15.90:6789/0 conn(0x7f7d5810a740 legacy=0x7f7d5810cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:18.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:17.406+0000 7f7d5ffe7700 1 -- 172.21.15.90:0/248705291 shutdown_connections 2024-04-14T15:27:18.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:17.406+0000 7f7d5ffe7700 1 -- 172.21.15.90:0/248705291 >> 172.21.15.90:0/248705291 conn(0x7f7d580fd290 msgr2=0x7f7d580ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:27:18.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:17.407+0000 7f7d5ffe7700 1 -- 172.21.15.90:0/248705291 shutdown_connections 2024-04-14T15:27:18.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:17.407+0000 7f7d5ffe7700 1 -- 172.21.15.90:0/248705291 wait complete. 2024-04-14T15:27:18.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:17.407+0000 7f7d5ffe7700 1 librados: starting msgr at 2024-04-14T15:27:18.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:17.407+0000 7f7d5ffe7700 1 librados: starting objecter 2024-04-14T15:27:18.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:17.407+0000 7f7d5ffe7700 1 Processor -- start 2024-04-14T15:27:18.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:17.407+0000 7f7d5ffe7700 1 -- start start 2024-04-14T15:27:18.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:17.407+0000 7f7d5ffe7700 1 librados: setting wanted keys 2024-04-14T15:27:18.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:17.407+0000 7f7d5ffe7700 1 librados: calling monclient init 2024-04-14T15:27:18.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:17.407+0000 7f7d5ffe7700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7d5810dfc0 con 0x7f7d5810e310 2024-04-14T15:27:18.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:17.407+0000 7f7d5ffe7700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7d581a8f30 con 0x7f7d58107890 2024-04-14T15:27:18.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:17.407+0000 7f7d5ffe7700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7d581aa110 con 0x7f7d5810a740 2024-04-14T15:27:18.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:17.407+0000 7f7d5e584700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f7d5810e310 0x7f7d581a6800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:41028/0 (socket says 172.21.15.90:41028) 2024-04-14T15:27:18.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:17.407+0000 7f7d5e584700 1 -- 172.21.15.90:0/2059844821 learned_addr learned my addr 172.21.15.90:0/2059844821 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:27:18.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:17.407+0000 7f7d5d582700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f7d5810a740 0x7f7d5810c540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:56948/0 (socket says 172.21.15.90:56948) 2024-04-14T15:27:18.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:17.407+0000 7f7d4effd700 10 client.?.objecter ms_handle_connect 0x7f7d5810a740 2024-04-14T15:27:18.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:17.407+0000 7f7d4effd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:27:18.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:17.407+0000 7f7d4effd700 10 client.?.objecter ms_handle_connect 0x7f7d5810e310 2024-04-14T15:27:18.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:17.407+0000 7f7d4effd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:27:18.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:17.408+0000 7f7d4effd700 10 client.?.objecter ms_handle_connect 0x7f7d58107890 2024-04-14T15:27:18.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:17.408+0000 7f7d4effd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:27:18.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:17.408+0000 7f7d4effd700 1 -- 172.21.15.90:0/2059844821 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 837405806 0 0) 0x7f7d581aa110 con 0x7f7d5810a740 2024-04-14T15:27:18.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:17.408+0000 7f7d4effd700 1 -- 172.21.15.90:0/2059844821 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7d440031f0 con 0x7f7d5810a740 2024-04-14T15:27:18.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:17.408+0000 7f7d4effd700 1 -- 172.21.15.90:0/2059844821 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2795226006 0 0) 0x7f7d5810dfc0 con 0x7f7d5810e310 2024-04-14T15:27:18.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:17.408+0000 7f7d4effd700 1 -- 172.21.15.90:0/2059844821 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7d581aa110 con 0x7f7d5810e310 2024-04-14T15:27:18.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:17.408+0000 7f7d4effd700 1 -- 172.21.15.90:0/2059844821 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3581548081 0 0) 0x7f7d581a8f30 con 0x7f7d58107890 2024-04-14T15:27:18.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:17.408+0000 7f7d4effd700 1 -- 172.21.15.90:0/2059844821 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7d5810dfc0 con 0x7f7d58107890 2024-04-14T15:27:18.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:17.408+0000 7f7d4effd700 1 -- 172.21.15.90:0/2059844821 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2501757302 0 0) 0x7f7d581aa110 con 0x7f7d5810e310 2024-04-14T15:27:18.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:17.408+0000 7f7d4effd700 1 -- 172.21.15.90:0/2059844821 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7d581a8f30 con 0x7f7d5810e310 2024-04-14T15:27:18.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:17.408+0000 7f7d4effd700 1 -- 172.21.15.90:0/2059844821 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f7d50001c80 con 0x7f7d5810e310 2024-04-14T15:27:18.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:17.408+0000 7f7d4effd700 1 -- 172.21.15.90:0/2059844821 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1684310696 0 0) 0x7f7d440031f0 con 0x7f7d5810a740 2024-04-14T15:27:18.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:17.408+0000 7f7d4effd700 1 -- 172.21.15.90:0/2059844821 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7d581aa110 con 0x7f7d5810a740 2024-04-14T15:27:18.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:17.408+0000 7f7d4effd700 1 -- 172.21.15.90:0/2059844821 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f7d48004470 con 0x7f7d5810a740 2024-04-14T15:27:18.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:17.408+0000 7f7d4effd700 1 -- 172.21.15.90:0/2059844821 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2789850184 0 0) 0x7f7d5810dfc0 con 0x7f7d58107890 2024-04-14T15:27:18.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:17.408+0000 7f7d4effd700 1 -- 172.21.15.90:0/2059844821 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7d440031f0 con 0x7f7d58107890 2024-04-14T15:27:18.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:17.408+0000 7f7d4effd700 1 -- 172.21.15.90:0/2059844821 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f7d54001890 con 0x7f7d58107890 2024-04-14T15:27:18.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:17.408+0000 7f7d4effd700 1 -- 172.21.15.90:0/2059844821 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3131754608 0 0) 0x7f7d581a8f30 con 0x7f7d5810e310 2024-04-14T15:27:18.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:17.408+0000 7f7d4effd700 1 -- 172.21.15.90:0/2059844821 >> v1:172.21.15.90:6790/0 conn(0x7f7d5810a740 legacy=0x7f7d5810c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:18.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:17.408+0000 7f7d4effd700 1 -- 172.21.15.90:0/2059844821 >> v1:172.21.15.92:6789/0 conn(0x7f7d58107890 legacy=0x7f7d5810be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:18.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:17.408+0000 7f7d4effd700 1 -- 172.21.15.90:0/2059844821 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d581ab2f0 con 0x7f7d5810e310 2024-04-14T15:27:18.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:17.408+0000 7f7d5ffe7700 1 -- 172.21.15.90:0/2059844821 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7d581aa340 con 0x7f7d5810e310 2024-04-14T15:27:18.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:17.408+0000 7f7d5ffe7700 10 client.38187.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:27:18.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:17.408+0000 7f7d5ffe7700 1 -- 172.21.15.90:0/2059844821 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f7d581aa800 con 0x7f7d5810e310 2024-04-14T15:27:18.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:17.408+0000 7f7d5ffe7700 1 librados: init done 2024-04-14T15:27:18.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:17.408+0000 7f7d4effd700 1 -- 172.21.15.90:0/2059844821 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f7d50003ec0 con 0x7f7d5810e310 2024-04-14T15:27:18.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:17.408+0000 7f7d4effd700 1 -- 172.21.15.90:0/2059844821 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f7d50005fa0 con 0x7f7d5810e310 2024-04-14T15:27:18.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:17.409+0000 7f7d5ffe7700 1 -- 172.21.15.90:0/2059844821 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d3c005120 con 0x7f7d5810e310 2024-04-14T15:27:18.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:17.410+0000 7f7d4effd700 1 -- 172.21.15.90:0/2059844821 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f7d50003a20 con 0x7f7d5810e310 2024-04-14T15:27:18.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:17.410+0000 7f7d4effd700 1 -- 172.21.15.90:0/2059844821 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(869..869 src has 467..869) ==== 7649+0+0 (unknown 1730563604 0 0) 0x7f7d5008ab00 con 0x7f7d5810e310 2024-04-14T15:27:18.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:17.410+0000 7f7d4effd700 10 client.38187.objecter ms_dispatch 0x7f7d58001550 osd_map(869..869 src has 467..869) 2024-04-14T15:27:18.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:17.410+0000 7f7d4effd700 3 client.38187.objecter handle_osd_map got epochs [869,869] > 0 2024-04-14T15:27:18.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:17.410+0000 7f7d4effd700 3 client.38187.objecter handle_osd_map decoding full epoch 869 2024-04-14T15:27:18.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:17.410+0000 7f7d4effd700 20 client.38187.objecter dump_active .. 0 homeless 2024-04-14T15:27:18.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:17.410+0000 7f7d4effd700 10 client.38187.objecter ms_handle_connect 0x7f7d44071380 2024-04-14T15:27:18.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:17.415+0000 7f7d4effd700 1 -- 172.21.15.90:0/2059844821 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f7d50057770 con 0x7f7d5810e310 2024-04-14T15:27:18.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:17.624+0000 7f7d5ffe7700 1 -- 172.21.15.90:0/2059844821 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7f7d3c005410 con 0x7f7d5810e310 2024-04-14T15:27:18.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:18.209+0000 7f7d4effd700 1 -- 172.21.15.90:0/2059844821 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v870) ==== 101+0+0 (unknown 2940864550 0 0) 0x7f7d5005b420 con 0x7f7d5810e310 2024-04-14T15:27:18.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:18.211+0000 7f7d5ffe7700 10 librados: watch_flush enter 2024-04-14T15:27:18.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:18.211+0000 7f7d5ffe7700 10 librados: watch_flush exit 2024-04-14T15:27:18.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:18.211+0000 7f7d5ffe7700 20 client.38187.objecter shutdown clearing up homeless session... 2024-04-14T15:27:18.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:18.211+0000 7f7d5ffe7700 10 client.38187.objecter successfully canceled tick 2024-04-14T15:27:18.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:18.211+0000 7f7d5ffe7700 1 -- 172.21.15.90:0/2059844821 >> v1:172.21.15.90:6800/420935427 conn(0x7f7d44071380 legacy=0x7f7d44073840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:18.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:18.211+0000 7f7d5ffe7700 1 -- 172.21.15.90:0/2059844821 >> v1:172.21.15.90:6789/0 conn(0x7f7d5810e310 legacy=0x7f7d581a6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:18.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:18.212+0000 7f7d5ffe7700 1 -- 172.21.15.90:0/2059844821 shutdown_connections 2024-04-14T15:27:18.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:18.212+0000 7f7d5ffe7700 1 -- 172.21.15.90:0/2059844821 >> 172.21.15.90:0/2059844821 conn(0x7f7d580fd290 msgr2=0x7f7d5806b230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:27:18.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:18.212+0000 7f7d5ffe7700 1 -- 172.21.15.90:0/2059844821 shutdown_connections 2024-04-14T15:27:18.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:18.212+0000 7f7d5ffe7700 1 -- 172.21.15.90:0/2059844821 wait complete. 2024-04-14T15:27:18.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:18.212+0000 7f7d5ffe7700 1 librados: shutdown 2024-04-14T15:27:18.848 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is unset 2024-04-14T15:27:19.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:19 smithi090 ceph-mon[41670]: 2.0 scrub starts 2024-04-14T15:27:19.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:19 smithi090 ceph-mon[41670]: 2.0 scrub ok 2024-04-14T15:27:19.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:19 smithi090 ceph-mon[41670]: 4.10 scrub starts 2024-04-14T15:27:19.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:19 smithi090 ceph-mon[41670]: 4.10 scrub ok 2024-04-14T15:27:19.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:19 smithi090 ceph-mon[41670]: 4.13 scrub starts 2024-04-14T15:27:19.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:19 smithi090 ceph-mon[41670]: 4.13 scrub ok 2024-04-14T15:27:19.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:19 smithi090 ceph-mon[41670]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-14T15:27:19.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:19 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2059844821' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-14T15:27:19.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:19 smithi090 ceph-mon[41670]: osdmap e870: 8 total, 8 up, 8 in 2024-04-14T15:27:19.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:19 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:27:19.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:19 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:27:19.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:19 smithi090 ceph-mon[30399]: 2.0 scrub starts 2024-04-14T15:27:19.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:19 smithi090 ceph-mon[30399]: 2.0 scrub ok 2024-04-14T15:27:19.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:19 smithi090 ceph-mon[30399]: 4.10 scrub starts 2024-04-14T15:27:19.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:19 smithi090 ceph-mon[30399]: 4.10 scrub ok 2024-04-14T15:27:19.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:19 smithi090 ceph-mon[30399]: 4.13 scrub starts 2024-04-14T15:27:19.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:19 smithi090 ceph-mon[30399]: 4.13 scrub ok 2024-04-14T15:27:19.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:19 smithi090 ceph-mon[30399]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-14T15:27:19.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:19 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2059844821' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-14T15:27:19.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:19 smithi090 ceph-mon[30399]: osdmap e870: 8 total, 8 up, 8 in 2024-04-14T15:27:19.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:19 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:27:19.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:19 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:27:19.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:19 smithi092 ceph-mon[33966]: 2.0 scrub starts 2024-04-14T15:27:19.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:19 smithi092 ceph-mon[33966]: 2.0 scrub ok 2024-04-14T15:27:19.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:19 smithi092 ceph-mon[33966]: 4.10 scrub starts 2024-04-14T15:27:19.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:19 smithi092 ceph-mon[33966]: 4.10 scrub ok 2024-04-14T15:27:19.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:19 smithi092 ceph-mon[33966]: 4.13 scrub starts 2024-04-14T15:27:19.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:19 smithi092 ceph-mon[33966]: 4.13 scrub ok 2024-04-14T15:27:19.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:19 smithi092 ceph-mon[33966]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-14T15:27:19.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:19 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2059844821' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-14T15:27:19.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:19 smithi092 ceph-mon[33966]: osdmap e870: 8 total, 8 up, 8 in 2024-04-14T15:27:19.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:19 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:27:19.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:19 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:27:19.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:19.683+0000 7fd4c1d2a700 1 -- 172.21.15.90:0/4184011025 >> v1:172.21.15.90:6789/0 conn(0x7fd4bc10b740 legacy=0x7fd4bc10db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:19.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:19.683+0000 7fd4c1d2a700 1 -- 172.21.15.90:0/4184011025 shutdown_connections 2024-04-14T15:27:19.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:19.683+0000 7fd4c1d2a700 1 -- 172.21.15.90:0/4184011025 >> 172.21.15.90:0/4184011025 conn(0x7fd4bc0fe3a0 msgr2=0x7fd4bc1007c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:27:19.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:19.683+0000 7fd4c1d2a700 1 -- 172.21.15.90:0/4184011025 shutdown_connections 2024-04-14T15:27:19.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:19.683+0000 7fd4c1d2a700 1 -- 172.21.15.90:0/4184011025 wait complete. 2024-04-14T15:27:19.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:19.683+0000 7fd4c1d2a700 1 librados: starting msgr at 2024-04-14T15:27:19.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:19.683+0000 7fd4c1d2a700 1 librados: starting objecter 2024-04-14T15:27:19.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:19.683+0000 7fd4c1d2a700 1 Processor -- start 2024-04-14T15:27:19.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:19.684+0000 7fd4c1d2a700 1 -- start start 2024-04-14T15:27:19.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:19.684+0000 7fd4c1d2a700 1 librados: setting wanted keys 2024-04-14T15:27:19.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:19.684+0000 7fd4c1d2a700 1 librados: calling monclient init 2024-04-14T15:27:19.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:19.684+0000 7fd4c1d2a700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd4bc10efc0 con 0x7fd4bc10f310 2024-04-14T15:27:19.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:19.684+0000 7fd4c1d2a700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd4bc1a8f20 con 0x7fd4bc10b740 2024-04-14T15:27:19.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:19.684+0000 7fd4c1d2a700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd4bc1aa100 con 0x7fd4bc108890 2024-04-14T15:27:19.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:19.684+0000 7fd4bb7fe700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7fd4bc108890 0x7fd4bc10be10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:56976/0 (socket says 172.21.15.90:56976) 2024-04-14T15:27:19.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:19.684+0000 7fd4bb7fe700 1 -- 172.21.15.90:0/598324151 learned_addr learned my addr 172.21.15.90:0/598324151 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:27:19.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:19.684+0000 7fd4bbfff700 1 --1- 172.21.15.90:0/598324151 >> v1:172.21.15.90:6789/0 conn(0x7fd4bc10f310 0x7fd4bc1a67f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:41042/0 (socket says 172.21.15.90:41042) 2024-04-14T15:27:19.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:19.684+0000 7fd4b8ff9700 10 client.?.objecter ms_handle_connect 0x7fd4bc108890 2024-04-14T15:27:19.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:19.684+0000 7fd4b8ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:27:19.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:19.684+0000 7fd4b8ff9700 10 client.?.objecter ms_handle_connect 0x7fd4bc10f310 2024-04-14T15:27:19.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:19.684+0000 7fd4b8ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:27:19.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:19.684+0000 7fd4b8ff9700 10 client.?.objecter ms_handle_connect 0x7fd4bc10b740 2024-04-14T15:27:19.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:19.684+0000 7fd4b8ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:27:19.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:19.684+0000 7fd4b8ff9700 1 -- 172.21.15.90:0/598324151 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1981124376 0 0) 0x7fd4bc10efc0 con 0x7fd4bc10f310 2024-04-14T15:27:19.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:19.684+0000 7fd4b8ff9700 1 -- 172.21.15.90:0/598324151 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd4a40035b0 con 0x7fd4bc10f310 2024-04-14T15:27:19.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:19.684+0000 7fd4b8ff9700 1 -- 172.21.15.90:0/598324151 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2635996324 0 0) 0x7fd4bc1aa100 con 0x7fd4bc108890 2024-04-14T15:27:19.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:19.684+0000 7fd4b8ff9700 1 -- 172.21.15.90:0/598324151 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd4bc10efc0 con 0x7fd4bc108890 2024-04-14T15:27:19.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:19.684+0000 7fd4b8ff9700 1 -- 172.21.15.90:0/598324151 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3482593943 0 0) 0x7fd4bc1a8f20 con 0x7fd4bc10b740 2024-04-14T15:27:19.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:19.684+0000 7fd4b8ff9700 1 -- 172.21.15.90:0/598324151 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd4bc1aa100 con 0x7fd4bc10b740 2024-04-14T15:27:19.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:19.684+0000 7fd4b8ff9700 1 -- 172.21.15.90:0/598324151 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1874116467 0 0) 0x7fd4a40035b0 con 0x7fd4bc10f310 2024-04-14T15:27:19.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:19.684+0000 7fd4b8ff9700 1 -- 172.21.15.90:0/598324151 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd4bc1a8f20 con 0x7fd4bc10f310 2024-04-14T15:27:19.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:19.684+0000 7fd4b8ff9700 1 -- 172.21.15.90:0/598324151 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fd4b4001c50 con 0x7fd4bc10f310 2024-04-14T15:27:19.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:19.684+0000 7fd4b8ff9700 1 -- 172.21.15.90:0/598324151 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4000751635 0 0) 0x7fd4bc10efc0 con 0x7fd4bc108890 2024-04-14T15:27:19.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:19.685+0000 7fd4b8ff9700 1 -- 172.21.15.90:0/598324151 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd4a40035b0 con 0x7fd4bc108890 2024-04-14T15:27:19.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:19.685+0000 7fd4b8ff9700 1 -- 172.21.15.90:0/598324151 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fd4ac001870 con 0x7fd4bc108890 2024-04-14T15:27:19.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:19.685+0000 7fd4b8ff9700 1 -- 172.21.15.90:0/598324151 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3093413598 0 0) 0x7fd4bc1aa100 con 0x7fd4bc10b740 2024-04-14T15:27:19.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:19.685+0000 7fd4b8ff9700 1 -- 172.21.15.90:0/598324151 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd4bc10efc0 con 0x7fd4bc10b740 2024-04-14T15:27:19.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:19.685+0000 7fd4b8ff9700 1 -- 172.21.15.90:0/598324151 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fd4b0001430 con 0x7fd4bc10b740 2024-04-14T15:27:19.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:19.685+0000 7fd4b8ff9700 1 -- 172.21.15.90:0/598324151 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3784268998 0 0) 0x7fd4bc1a8f20 con 0x7fd4bc10f310 2024-04-14T15:27:19.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:19.685+0000 7fd4b8ff9700 1 -- 172.21.15.90:0/598324151 >> v1:172.21.15.90:6790/0 conn(0x7fd4bc108890 legacy=0x7fd4bc10be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:19.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:19.685+0000 7fd4b8ff9700 1 -- 172.21.15.90:0/598324151 >> v1:172.21.15.92:6789/0 conn(0x7fd4bc10b740 legacy=0x7fd4bc10c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:19.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:19.685+0000 7fd4b8ff9700 1 -- 172.21.15.90:0/598324151 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4bc1ab2e0 con 0x7fd4bc10f310 2024-04-14T15:27:19.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:19.685+0000 7fd4c1d2a700 1 -- 172.21.15.90:0/598324151 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd4bc1a7f70 con 0x7fd4bc10f310 2024-04-14T15:27:19.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:19.685+0000 7fd4c1d2a700 10 client.38193.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:27:19.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:19.685+0000 7fd4c1d2a700 1 -- 172.21.15.90:0/598324151 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd4bc1a8430 con 0x7fd4bc10f310 2024-04-14T15:27:19.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:19.685+0000 7fd4c1d2a700 1 librados: init done 2024-04-14T15:27:19.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:19.685+0000 7fd4b8ff9700 1 -- 172.21.15.90:0/598324151 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fd4b4003ec0 con 0x7fd4bc10f310 2024-04-14T15:27:19.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:19.685+0000 7fd4b8ff9700 1 -- 172.21.15.90:0/598324151 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fd4b4005ec0 con 0x7fd4bc10f310 2024-04-14T15:27:19.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:19.686+0000 7fd4c1d2a700 1 -- 172.21.15.90:0/598324151 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4a8005120 con 0x7fd4bc10f310 2024-04-14T15:27:19.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:19.686+0000 7fd4b8ff9700 1 -- 172.21.15.90:0/598324151 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fd4b4003a20 con 0x7fd4bc10f310 2024-04-14T15:27:19.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:19.687+0000 7fd4b8ff9700 1 -- 172.21.15.90:0/598324151 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(871..871 src has 467..871) ==== 7649+0+0 (unknown 1313409 0 0) 0x7fd4b408a980 con 0x7fd4bc10f310 2024-04-14T15:27:19.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:19.687+0000 7fd4b8ff9700 10 client.38193.objecter ms_dispatch 0x7fd4bc001550 osd_map(871..871 src has 467..871) 2024-04-14T15:27:19.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:19.687+0000 7fd4b8ff9700 3 client.38193.objecter handle_osd_map got epochs [871,871] > 0 2024-04-14T15:27:19.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:19.687+0000 7fd4b8ff9700 3 client.38193.objecter handle_osd_map decoding full epoch 871 2024-04-14T15:27:19.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:19.687+0000 7fd4b8ff9700 20 client.38193.objecter dump_active .. 0 homeless 2024-04-14T15:27:19.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:19.687+0000 7fd4b8ff9700 10 client.38193.objecter ms_handle_connect 0x7fd4a4071400 2024-04-14T15:27:19.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:19.691+0000 7fd4b8ff9700 1 -- 172.21.15.90:0/598324151 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fd4b40575d0 con 0x7fd4bc10f310 2024-04-14T15:27:19.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:19.896+0000 7fd4c1d2a700 1 -- 172.21.15.90:0/598324151 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} v 0) -- 0x7fd4a8005410 con 0x7fd4bc10f310 2024-04-14T15:27:19.934 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-14T15:27:19.935 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:27:20.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:20.224+0000 7fd4b8ff9700 1 -- 172.21.15.90:0/598324151 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]=0 set pool 1 pgp_num to 11 v872) ==== 131+0+0 (unknown 3188791977 0 0) 0x7fd4b405b280 con 0x7fd4bc10f310 2024-04-14T15:27:20.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:20.227+0000 7fd4c1d2a700 10 librados: watch_flush enter 2024-04-14T15:27:20.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:20.228+0000 7fd4c1d2a700 10 librados: watch_flush exit 2024-04-14T15:27:20.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:20.228+0000 7fd4c1d2a700 20 client.38193.objecter shutdown clearing up homeless session... 2024-04-14T15:27:20.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:20.228+0000 7fd4c1d2a700 10 client.38193.objecter successfully canceled tick 2024-04-14T15:27:20.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:20.229+0000 7fd4c1d2a700 1 -- 172.21.15.90:0/598324151 >> v1:172.21.15.90:6800/420935427 conn(0x7fd4a4071400 legacy=0x7fd4a40738c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:20.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:20.229+0000 7fd4c1d2a700 1 -- 172.21.15.90:0/598324151 >> v1:172.21.15.90:6789/0 conn(0x7fd4bc10f310 legacy=0x7fd4bc1a67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:20.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:20.229+0000 7fd4c1d2a700 1 -- 172.21.15.90:0/598324151 shutdown_connections 2024-04-14T15:27:20.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:20.229+0000 7fd4c1d2a700 1 -- 172.21.15.90:0/598324151 >> 172.21.15.90:0/598324151 conn(0x7fd4bc0fe3a0 msgr2=0x7fd4bc1127c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:27:20.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:20.229+0000 7fd4c1d2a700 1 -- 172.21.15.90:0/598324151 shutdown_connections 2024-04-14T15:27:20.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:20.229+0000 7fd4c1d2a700 1 -- 172.21.15.90:0/598324151 wait complete. 2024-04-14T15:27:20.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:20.230+0000 7fd4c1d2a700 1 librados: shutdown 2024-04-14T15:27:20.230 INFO:teuthology.orchestra.run.smithi090.stderr:set pool 1 pgp_num to 11 2024-04-14T15:27:20.614 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:20 smithi090 ceph-mon[41670]: 6.e scrub starts 2024-04-14T15:27:20.614 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:20 smithi090 ceph-mon[41670]: 6.e scrub ok 2024-04-14T15:27:20.614 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:20 smithi090 ceph-mon[41670]: pgmap v1303: 182 pgs: 178 active+clean, 4 active+remapped+backfill_toofull; 582 KiB data, 818 MiB used, 714 GiB / 715 GiB avail; 0 B/s, 18 objects/s recovering 2024-04-14T15:27:20.614 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:20 smithi090 ceph-mon[41670]: osdmap e871: 8 total, 8 up, 8 in 2024-04-14T15:27:20.615 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:20 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/598324151' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-04-14T15:27:20.615 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:20 smithi090 ceph-mon[30399]: 6.e scrub starts 2024-04-14T15:27:20.615 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:20 smithi090 ceph-mon[30399]: 6.e scrub ok 2024-04-14T15:27:20.615 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:20 smithi090 ceph-mon[30399]: pgmap v1303: 182 pgs: 178 active+clean, 4 active+remapped+backfill_toofull; 582 KiB data, 818 MiB used, 714 GiB / 715 GiB avail; 0 B/s, 18 objects/s recovering 2024-04-14T15:27:20.615 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:20 smithi090 ceph-mon[30399]: osdmap e871: 8 total, 8 up, 8 in 2024-04-14T15:27:20.616 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:20 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/598324151' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-04-14T15:27:20.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:20 smithi092 ceph-mon[33966]: 6.e scrub starts 2024-04-14T15:27:20.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:20 smithi092 ceph-mon[33966]: 6.e scrub ok 2024-04-14T15:27:20.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:20 smithi092 ceph-mon[33966]: pgmap v1303: 182 pgs: 178 active+clean, 4 active+remapped+backfill_toofull; 582 KiB data, 818 MiB used, 714 GiB / 715 GiB avail; 0 B/s, 18 objects/s recovering 2024-04-14T15:27:20.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:20 smithi092 ceph-mon[33966]: osdmap e871: 8 total, 8 up, 8 in 2024-04-14T15:27:20.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:20 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/598324151' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-04-14T15:27:20.645 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-14T15:27:21.259 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:21 smithi090 ceph-mon[41670]: 2.17 scrub starts 2024-04-14T15:27:21.259 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:21 smithi090 ceph-mon[41670]: 2.17 scrub ok 2024-04-14T15:27:21.259 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:21 smithi090 ceph-mon[41670]: 4.15 scrub starts 2024-04-14T15:27:21.260 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:21 smithi090 ceph-mon[41670]: 4.15 scrub ok 2024-04-14T15:27:21.260 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:21 smithi090 ceph-mon[41670]: 5.17 scrub starts 2024-04-14T15:27:21.260 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:21 smithi090 ceph-mon[41670]: 5.17 scrub ok 2024-04-14T15:27:21.260 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:21 smithi090 ceph-mon[41670]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 11 pgs inactive, 12 pgs peering) 2024-04-14T15:27:21.260 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:21 smithi090 ceph-mon[41670]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 11/597 objects degraded (1.843%), 1 pg degraded) 2024-04-14T15:27:21.260 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:21 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/598324151' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-04-14T15:27:21.260 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:21 smithi090 ceph-mon[41670]: osdmap e872: 8 total, 8 up, 8 in 2024-04-14T15:27:21.260 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:21 smithi090 ceph-mon[41670]: 11.d scrub starts 2024-04-14T15:27:21.260 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:21 smithi090 ceph-mon[41670]: 11.d scrub ok 2024-04-14T15:27:21.260 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:21 smithi090 ceph-mon[41670]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-04-14T15:27:21.261 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:21 smithi090 ceph-mon[30399]: 2.17 scrub starts 2024-04-14T15:27:21.261 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:21 smithi090 ceph-mon[30399]: 2.17 scrub ok 2024-04-14T15:27:21.261 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:21 smithi090 ceph-mon[30399]: 4.15 scrub starts 2024-04-14T15:27:21.261 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:21 smithi090 ceph-mon[30399]: 4.15 scrub ok 2024-04-14T15:27:21.261 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:21 smithi090 ceph-mon[30399]: 5.17 scrub starts 2024-04-14T15:27:21.261 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:21 smithi090 ceph-mon[30399]: 5.17 scrub ok 2024-04-14T15:27:21.261 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:21 smithi090 ceph-mon[30399]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 11 pgs inactive, 12 pgs peering) 2024-04-14T15:27:21.261 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:21 smithi090 ceph-mon[30399]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 11/597 objects degraded (1.843%), 1 pg degraded) 2024-04-14T15:27:21.261 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:21 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/598324151' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-04-14T15:27:21.261 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:21 smithi090 ceph-mon[30399]: osdmap e872: 8 total, 8 up, 8 in 2024-04-14T15:27:21.261 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:21 smithi090 ceph-mon[30399]: 11.d scrub starts 2024-04-14T15:27:21.262 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:21 smithi090 ceph-mon[30399]: 11.d scrub ok 2024-04-14T15:27:21.262 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:21 smithi090 ceph-mon[30399]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-04-14T15:27:21.322 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-04-14T15:27:21.353 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:27:21.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:21 smithi092 ceph-mon[33966]: 2.17 scrub starts 2024-04-14T15:27:21.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:21 smithi092 ceph-mon[33966]: 2.17 scrub ok 2024-04-14T15:27:21.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:21 smithi092 ceph-mon[33966]: 4.15 scrub starts 2024-04-14T15:27:21.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:21 smithi092 ceph-mon[33966]: 4.15 scrub ok 2024-04-14T15:27:21.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:21 smithi092 ceph-mon[33966]: 5.17 scrub starts 2024-04-14T15:27:21.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:21 smithi092 ceph-mon[33966]: 5.17 scrub ok 2024-04-14T15:27:21.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:21 smithi092 ceph-mon[33966]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 11 pgs inactive, 12 pgs peering) 2024-04-14T15:27:21.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:21 smithi092 ceph-mon[33966]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 11/597 objects degraded (1.843%), 1 pg degraded) 2024-04-14T15:27:21.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:21 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/598324151' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-04-14T15:27:21.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:21 smithi092 ceph-mon[33966]: osdmap e872: 8 total, 8 up, 8 in 2024-04-14T15:27:21.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:21 smithi092 ceph-mon[33966]: 11.d scrub starts 2024-04-14T15:27:21.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:21 smithi092 ceph-mon[33966]: 11.d scrub ok 2024-04-14T15:27:21.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:21 smithi092 ceph-mon[33966]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-04-14T15:27:22.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:22 smithi090 ceph-mon[30399]: 5.11 scrub starts 2024-04-14T15:27:22.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:22 smithi090 ceph-mon[30399]: 5.11 scrub ok 2024-04-14T15:27:22.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:22 smithi090 ceph-mon[30399]: 5.16 scrub starts 2024-04-14T15:27:22.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:22 smithi090 ceph-mon[30399]: 5.16 scrub ok 2024-04-14T15:27:22.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:22 smithi090 ceph-mon[30399]: 6.c scrub starts 2024-04-14T15:27:22.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:22 smithi090 ceph-mon[30399]: 6.c scrub ok 2024-04-14T15:27:22.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:22 smithi090 ceph-mon[30399]: 6.17 scrub starts 2024-04-14T15:27:22.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:22 smithi090 ceph-mon[30399]: 6.17 scrub ok 2024-04-14T15:27:22.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:22 smithi090 ceph-mon[30399]: pgmap v1305: 182 pgs: 178 active+clean, 4 active+remapped+backfill_toofull; 582 KiB data, 818 MiB used, 714 GiB / 715 GiB avail; 0 B/s, 17 objects/s recovering 2024-04-14T15:27:22.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:22 smithi090 ceph-mon[41670]: 5.11 scrub starts 2024-04-14T15:27:22.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:22 smithi090 ceph-mon[41670]: 5.11 scrub ok 2024-04-14T15:27:22.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:22 smithi090 ceph-mon[41670]: 5.16 scrub starts 2024-04-14T15:27:22.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:22 smithi090 ceph-mon[41670]: 5.16 scrub ok 2024-04-14T15:27:22.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:22 smithi090 ceph-mon[41670]: 6.c scrub starts 2024-04-14T15:27:22.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:22 smithi090 ceph-mon[41670]: 6.c scrub ok 2024-04-14T15:27:22.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:22 smithi090 ceph-mon[41670]: 6.17 scrub starts 2024-04-14T15:27:22.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:22 smithi090 ceph-mon[41670]: 6.17 scrub ok 2024-04-14T15:27:22.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:22 smithi090 ceph-mon[41670]: pgmap v1305: 182 pgs: 178 active+clean, 4 active+remapped+backfill_toofull; 582 KiB data, 818 MiB used, 714 GiB / 715 GiB avail; 0 B/s, 17 objects/s recovering 2024-04-14T15:27:22.523 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:22 smithi092 ceph-mon[33966]: 5.11 scrub starts 2024-04-14T15:27:22.523 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:22 smithi092 ceph-mon[33966]: 5.11 scrub ok 2024-04-14T15:27:22.524 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:22 smithi092 ceph-mon[33966]: 5.16 scrub starts 2024-04-14T15:27:22.524 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:22 smithi092 ceph-mon[33966]: 5.16 scrub ok 2024-04-14T15:27:22.524 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:22 smithi092 ceph-mon[33966]: 6.c scrub starts 2024-04-14T15:27:22.524 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:22 smithi092 ceph-mon[33966]: 6.c scrub ok 2024-04-14T15:27:22.524 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:22 smithi092 ceph-mon[33966]: 6.17 scrub starts 2024-04-14T15:27:22.524 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:22 smithi092 ceph-mon[33966]: 6.17 scrub ok 2024-04-14T15:27:22.524 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:22 smithi092 ceph-mon[33966]: pgmap v1305: 182 pgs: 178 active+clean, 4 active+remapped+backfill_toofull; 582 KiB data, 818 MiB used, 714 GiB / 715 GiB avail; 0 B/s, 17 objects/s recovering 2024-04-14T15:27:22.648 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:27:23.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:23 smithi092 ceph-mon[33966]: 4.16 scrub starts 2024-04-14T15:27:23.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:23 smithi092 ceph-mon[33966]: 4.16 scrub ok 2024-04-14T15:27:23.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:23 smithi092 ceph-mon[33966]: 2.15 scrub starts 2024-04-14T15:27:23.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:23 smithi092 ceph-mon[33966]: 2.15 scrub ok 2024-04-14T15:27:23.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:23 smithi092 ceph-mon[33966]: 5.18 scrub starts 2024-04-14T15:27:23.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:23 smithi092 ceph-mon[33966]: 5.18 scrub ok 2024-04-14T15:27:23.671 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:23 smithi090 ceph-mon[41670]: 4.16 scrub starts 2024-04-14T15:27:23.671 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:23 smithi090 ceph-mon[41670]: 4.16 scrub ok 2024-04-14T15:27:23.671 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:23 smithi090 ceph-mon[41670]: 2.15 scrub starts 2024-04-14T15:27:23.671 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:23 smithi090 ceph-mon[41670]: 2.15 scrub ok 2024-04-14T15:27:23.671 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:23 smithi090 ceph-mon[41670]: 5.18 scrub starts 2024-04-14T15:27:23.671 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:23 smithi090 ceph-mon[41670]: 5.18 scrub ok 2024-04-14T15:27:23.672 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:23 smithi090 ceph-mon[30399]: 4.16 scrub starts 2024-04-14T15:27:23.672 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:23 smithi090 ceph-mon[30399]: 4.16 scrub ok 2024-04-14T15:27:23.672 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:23 smithi090 ceph-mon[30399]: 2.15 scrub starts 2024-04-14T15:27:23.672 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:23 smithi090 ceph-mon[30399]: 2.15 scrub ok 2024-04-14T15:27:23.672 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:23 smithi090 ceph-mon[30399]: 5.18 scrub starts 2024-04-14T15:27:23.672 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:23 smithi090 ceph-mon[30399]: 5.18 scrub ok 2024-04-14T15:27:23.796 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-14T15:27:23.797 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:27:24.236 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:27:23 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:27:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:27:24.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:24 smithi092 ceph-mon[33966]: 1.7 deep-scrub starts 2024-04-14T15:27:24.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:24 smithi092 ceph-mon[33966]: 1.7 deep-scrub ok 2024-04-14T15:27:24.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:24 smithi092 ceph-mon[33966]: 5.1e deep-scrub starts 2024-04-14T15:27:24.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:24 smithi092 ceph-mon[33966]: 5.1e deep-scrub ok 2024-04-14T15:27:24.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:24 smithi092 ceph-mon[33966]: 6.6 scrub starts 2024-04-14T15:27:24.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:24 smithi092 ceph-mon[33966]: 6.6 scrub ok 2024-04-14T15:27:24.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:24 smithi092 ceph-mon[33966]: pgmap v1306: 182 pgs: 178 active+clean, 4 active+remapped+backfill_toofull; 582 KiB data, 818 MiB used, 714 GiB / 715 GiB avail; 0 B/s, 12 objects/s recovering 2024-04-14T15:27:24.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:24 smithi092 ceph-mon[33966]: 6.d deep-scrub starts 2024-04-14T15:27:24.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:24 smithi092 ceph-mon[33966]: 6.d deep-scrub ok 2024-04-14T15:27:24.691 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:24 smithi090 ceph-mon[30399]: 1.7 deep-scrub starts 2024-04-14T15:27:24.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:24 smithi090 ceph-mon[30399]: 1.7 deep-scrub ok 2024-04-14T15:27:24.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:24 smithi090 ceph-mon[30399]: 5.1e deep-scrub starts 2024-04-14T15:27:24.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:24 smithi090 ceph-mon[30399]: 5.1e deep-scrub ok 2024-04-14T15:27:24.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:24 smithi090 ceph-mon[30399]: 6.6 scrub starts 2024-04-14T15:27:24.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:24 smithi090 ceph-mon[30399]: 6.6 scrub ok 2024-04-14T15:27:24.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:24 smithi090 ceph-mon[30399]: pgmap v1306: 182 pgs: 178 active+clean, 4 active+remapped+backfill_toofull; 582 KiB data, 818 MiB used, 714 GiB / 715 GiB avail; 0 B/s, 12 objects/s recovering 2024-04-14T15:27:24.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:24 smithi090 ceph-mon[30399]: 6.d deep-scrub starts 2024-04-14T15:27:24.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:24 smithi090 ceph-mon[30399]: 6.d deep-scrub ok 2024-04-14T15:27:24.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:24 smithi090 ceph-mon[41670]: 1.7 deep-scrub starts 2024-04-14T15:27:24.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:24 smithi090 ceph-mon[41670]: 1.7 deep-scrub ok 2024-04-14T15:27:24.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:24 smithi090 ceph-mon[41670]: 5.1e deep-scrub starts 2024-04-14T15:27:24.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:24 smithi090 ceph-mon[41670]: 5.1e deep-scrub ok 2024-04-14T15:27:24.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:24 smithi090 ceph-mon[41670]: 6.6 scrub starts 2024-04-14T15:27:24.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:24 smithi090 ceph-mon[41670]: 6.6 scrub ok 2024-04-14T15:27:24.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:24 smithi090 ceph-mon[41670]: pgmap v1306: 182 pgs: 178 active+clean, 4 active+remapped+backfill_toofull; 582 KiB data, 818 MiB used, 714 GiB / 715 GiB avail; 0 B/s, 12 objects/s recovering 2024-04-14T15:27:24.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:24 smithi090 ceph-mon[41670]: 6.d deep-scrub starts 2024-04-14T15:27:24.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:24 smithi090 ceph-mon[41670]: 6.d deep-scrub ok 2024-04-14T15:27:25.059 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-14T15:27:25.290 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:25 smithi090 ceph-mon[41670]: 4.f scrub starts 2024-04-14T15:27:25.291 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:25 smithi090 ceph-mon[41670]: 4.f scrub ok 2024-04-14T15:27:25.291 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:25 smithi090 ceph-mon[41670]: 5.13 scrub starts 2024-04-14T15:27:25.291 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:25 smithi090 ceph-mon[41670]: 5.13 scrub ok 2024-04-14T15:27:25.291 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:25 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:27:25.291 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:25 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:27:25.291 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:25 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:27:25.291 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:25 smithi090 ceph-mon[30399]: 4.f scrub starts 2024-04-14T15:27:25.291 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:25 smithi090 ceph-mon[30399]: 4.f scrub ok 2024-04-14T15:27:25.291 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:25 smithi090 ceph-mon[30399]: 5.13 scrub starts 2024-04-14T15:27:25.292 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:25 smithi090 ceph-mon[30399]: 5.13 scrub ok 2024-04-14T15:27:25.292 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:25 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:27:25.292 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:25 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:27:25.292 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:25 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:27:25.587 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:25 smithi092 ceph-mon[33966]: 4.f scrub starts 2024-04-14T15:27:25.587 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:25 smithi092 ceph-mon[33966]: 4.f scrub ok 2024-04-14T15:27:25.587 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:25 smithi092 ceph-mon[33966]: 5.13 scrub starts 2024-04-14T15:27:25.588 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:25 smithi092 ceph-mon[33966]: 5.13 scrub ok 2024-04-14T15:27:25.588 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:25 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:27:25.588 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:25 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:27:25.588 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:25 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:27:25.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.646+0000 7f1114e5a700 1 -- 172.21.15.90:0/1344910660 >> v1:172.21.15.90:6790/0 conn(0x7f111010f350 legacy=0x7f1110111810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:25.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.646+0000 7f1114e5a700 1 -- 172.21.15.90:0/1344910660 shutdown_connections 2024-04-14T15:27:25.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.646+0000 7f1114e5a700 1 -- 172.21.15.90:0/1344910660 >> 172.21.15.90:0/1344910660 conn(0x7f11100fe3c0 msgr2=0x7f11101007e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:27:25.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.646+0000 7f1114e5a700 1 -- 172.21.15.90:0/1344910660 shutdown_connections 2024-04-14T15:27:25.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.646+0000 7f1114e5a700 1 -- 172.21.15.90:0/1344910660 wait complete. 2024-04-14T15:27:25.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.647+0000 7f1114e5a700 1 librados: starting msgr at 2024-04-14T15:27:25.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.647+0000 7f1114e5a700 1 librados: starting objecter 2024-04-14T15:27:25.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.647+0000 7f1114e5a700 1 Processor -- start 2024-04-14T15:27:25.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.647+0000 7f1114e5a700 1 -- start start 2024-04-14T15:27:25.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.647+0000 7f1114e5a700 1 librados: setting wanted keys 2024-04-14T15:27:25.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.647+0000 7f1114e5a700 1 librados: calling monclient init 2024-04-14T15:27:25.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.648+0000 7f1114e5a700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f11101a7a70 con 0x7f111010f350 2024-04-14T15:27:25.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.648+0000 7f1114e5a700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f11101a8c90 con 0x7f111010b780 2024-04-14T15:27:25.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.648+0000 7f1114e5a700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f11101a9ed0 con 0x7f11101088d0 2024-04-14T15:27:25.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.648+0000 7f110ed9d700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f111010f350 0x7f11101a6130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:44020/0 (socket says 172.21.15.90:44020) 2024-04-14T15:27:25.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.648+0000 7f110ed9d700 1 -- 172.21.15.90:0/2220364075 learned_addr learned my addr 172.21.15.90:0/2220364075 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:27:25.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.648+0000 7f10ff7fe700 10 client.?.objecter ms_handle_connect 0x7f111010b780 2024-04-14T15:27:25.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.648+0000 7f10ff7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:27:25.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.648+0000 7f10ff7fe700 10 client.?.objecter ms_handle_connect 0x7f111010f350 2024-04-14T15:27:25.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.648+0000 7f10ff7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:27:25.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.648+0000 7f10ff7fe700 10 client.?.objecter ms_handle_connect 0x7f11101088d0 2024-04-14T15:27:25.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.648+0000 7f10ff7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:27:25.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.648+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2211334974 0 0) 0x7f11101a8c90 con 0x7f111010b780 2024-04-14T15:27:25.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.648+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f10f80033a0 con 0x7f111010b780 2024-04-14T15:27:25.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.648+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4085734089 0 0) 0x7f11101a7a70 con 0x7f111010f350 2024-04-14T15:27:25.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.648+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f11101a8c90 con 0x7f111010f350 2024-04-14T15:27:25.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.648+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2164467289 0 0) 0x7f11101a9ed0 con 0x7f11101088d0 2024-04-14T15:27:25.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.648+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f11101a7a70 con 0x7f11101088d0 2024-04-14T15:27:25.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.648+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1228190811 0 0) 0x7f10f80033a0 con 0x7f111010b780 2024-04-14T15:27:25.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.648+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f11101a9ed0 con 0x7f111010b780 2024-04-14T15:27:25.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.648+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f11040018b0 con 0x7f111010b780 2024-04-14T15:27:25.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.649+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 375187623 0 0) 0x7f11101a7a70 con 0x7f11101088d0 2024-04-14T15:27:25.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.649+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f10f80033a0 con 0x7f11101088d0 2024-04-14T15:27:25.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.649+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f1100001a30 con 0x7f11101088d0 2024-04-14T15:27:25.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.649+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1624319237 0 0) 0x7f11101a9ed0 con 0x7f111010b780 2024-04-14T15:27:25.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.649+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 >> v1:172.21.15.90:6790/0 conn(0x7f11101088d0 legacy=0x7f111010ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:25.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.649+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 >> v1:172.21.15.90:6789/0 conn(0x7f111010f350 legacy=0x7f11101a6130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:25.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.649+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11101ab110 con 0x7f111010b780 2024-04-14T15:27:25.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.649+0000 7f1114e5a700 1 -- 172.21.15.90:0/2220364075 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f11101a7ca0 con 0x7f111010b780 2024-04-14T15:27:25.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.649+0000 7f1114e5a700 10 client.50023.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:27:25.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.649+0000 7f1114e5a700 1 -- 172.21.15.90:0/2220364075 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f11101a8160 con 0x7f111010b780 2024-04-14T15:27:25.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.649+0000 7f1114e5a700 1 librados: init done 2024-04-14T15:27:25.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.649+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f1104003b40 con 0x7f111010b780 2024-04-14T15:27:25.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.649+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f1104004aa0 con 0x7f111010b780 2024-04-14T15:27:25.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.650+0000 7f1114e5a700 1 -- 172.21.15.90:0/2220364075 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f10f0005140 con 0x7f111010b780 2024-04-14T15:27:25.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.650+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f110401b4b0 con 0x7f111010b780 2024-04-14T15:27:25.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.651+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(872..872 src has 467..872) ==== 7649+0+0 (unknown 2164832937 0 0) 0x7f1104089540 con 0x7f111010b780 2024-04-14T15:27:25.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.651+0000 7f10ff7fe700 10 client.50023.objecter ms_dispatch 0x7f1110001550 osd_map(872..872 src has 467..872) 2024-04-14T15:27:25.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.651+0000 7f10ff7fe700 3 client.50023.objecter handle_osd_map got epochs [872,872] > 0 2024-04-14T15:27:25.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.651+0000 7f10ff7fe700 3 client.50023.objecter handle_osd_map decoding full epoch 872 2024-04-14T15:27:25.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.651+0000 7f10ff7fe700 20 client.50023.objecter dump_active .. 0 homeless 2024-04-14T15:27:25.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.651+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1124 (unknown 409899127 0 1008968738) 0x7f110408b970 con 0x7f111010b780 2024-04-14T15:27:25.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.651+0000 7f10ff7fe700 10 client.50023.objecter ms_handle_connect 0x7f10f8071140 2024-04-14T15:27:25.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.651+0000 7f1114e5a700 1 -- 172.21.15.90:0/2220364075 --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 3) -- 0x7f10f0004dc0 con 0x7f10f00027a0 2024-04-14T15:27:25.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.652+0000 7f10ff7fe700 10 client.50023.objecter ms_handle_connect 0x7f10f00027a0 2024-04-14T15:27:25.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.652+0000 7f10ff7fe700 10 client.50023.objecter resend_mon_ops 2024-04-14T15:27:25.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.652+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1362513766 0 0) 0x7f10f0004dc0 con 0x7f10f00027a0 2024-04-14T15:27:25.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.652+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f11101a9ed0 con 0x7f10f00027a0 2024-04-14T15:27:25.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.652+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2987037819 0 0) 0x7f11101a9ed0 con 0x7f10f00027a0 2024-04-14T15:27:25.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.652+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f10f0004dc0 con 0x7f10f00027a0 2024-04-14T15:27:25.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.652+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f1108001c90 con 0x7f10f00027a0 2024-04-14T15:27:25.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.652+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1209694716 0 0) 0x7f10f0004dc0 con 0x7f10f00027a0 2024-04-14T15:27:25.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.652+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 --> v1:172.21.15.90:6789/0 -- command(tid 2: {"prefix": "get_command_descriptions"}) -- 0x7f10f000a740 con 0x7f10f00027a0 2024-04-14T15:27:25.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.653+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 <== mon.0 v1:172.21.15.90:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f1108003a00 con 0x7f10f00027a0 2024-04-14T15:27:25.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.653+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 >> v1:172.21.15.90:6789/0 conn(0x7f10f00027a0 legacy=0x7f10f00056f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:25.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.653+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 >> v1:172.21.15.90:6789/0 conn(0x7f10f00027a0 legacy=0x7f10f00056f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-14T15:27:25.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.660+0000 7f1114e5a700 1 -- 172.21.15.90:0/2220364075 --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 3) -- 0x7f10f000af40 con 0x7f10f000aae0 2024-04-14T15:27:25.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.660+0000 7f10ff7fe700 10 client.50023.objecter ms_handle_connect 0x7f10f000aae0 2024-04-14T15:27:25.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.660+0000 7f10ff7fe700 10 client.50023.objecter resend_mon_ops 2024-04-14T15:27:25.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.660+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2325291711 0 0) 0x7f10f000af40 con 0x7f10f000aae0 2024-04-14T15:27:25.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.660+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f10f0004dc0 con 0x7f10f000aae0 2024-04-14T15:27:25.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.660+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1147959643 0 0) 0x7f10f0004dc0 con 0x7f10f000aae0 2024-04-14T15:27:25.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.660+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f10f000af40 con 0x7f10f000aae0 2024-04-14T15:27:25.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.660+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f1108003d50 con 0x7f10f000aae0 2024-04-14T15:27:25.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.661+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4203242656 0 0) 0x7f10f000af40 con 0x7f10f000aae0 2024-04-14T15:27:25.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.661+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 --> v1:172.21.15.90:6789/0 -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=600"]}) -- 0x7f10f000c1a0 con 0x7f10f000aae0 2024-04-14T15:27:25.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.662+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 <== mon.0 v1:172.21.15.90:6789/0 5 ==== command_reply(tid 3: 0 mon_osd_down_out_interval = '' (not observed, change may require restart) ) ==== 82+0+3 (unknown 285646418 0 2423995997) 0x7f11080040a0 con 0x7f10f000aae0 2024-04-14T15:27:25.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.662+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 >> v1:172.21.15.90:6789/0 conn(0x7f10f000aae0 legacy=0x7f10f0008e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:25.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.662+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 >> v1:172.21.15.90:6789/0 conn(0x7f10f000aae0 legacy=0x7f10f0008e20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-14T15:27:25.663 INFO:teuthology.orchestra.run.smithi090.stdout:mon.a: {} 2024-04-14T15:27:25.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.662+0000 7f1114e5a700 1 -- 172.21.15.90:0/2220364075 --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 3) -- 0x7f10f000f3d0 con 0x7f10f0001ec0 2024-04-14T15:27:25.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.663+0000 7f10ff7fe700 10 client.50023.objecter ms_handle_connect 0x7f10f0001ec0 2024-04-14T15:27:25.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.663+0000 7f10ff7fe700 10 client.50023.objecter resend_mon_ops 2024-04-14T15:27:25.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.663+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2617155098 0 0) 0x7f10f000f3d0 con 0x7f10f0001ec0 2024-04-14T15:27:25.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.663+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f10f000af40 con 0x7f10f0001ec0 2024-04-14T15:27:25.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.663+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 799008035 0 0) 0x7f10f000af40 con 0x7f10f0001ec0 2024-04-14T15:27:25.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.663+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f10f000f3d0 con 0x7f10f0001ec0 2024-04-14T15:27:25.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.663+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f1108004de0 con 0x7f10f0001ec0 2024-04-14T15:27:25.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.663+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2677053232 0 0) 0x7f10f000f3d0 con 0x7f10f0001ec0 2024-04-14T15:27:25.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.663+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 --> v1:172.21.15.92:6789/0 -- command(tid 4: {"prefix": "get_command_descriptions"}) -- 0x7f10f000bff0 con 0x7f10f0001ec0 2024-04-14T15:27:25.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.664+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 <== mon.1 v1:172.21.15.92:6789/0 5 ==== command_reply(tid 4: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f1108004550 con 0x7f10f0001ec0 2024-04-14T15:27:25.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.664+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 >> v1:172.21.15.92:6789/0 conn(0x7f10f0001ec0 legacy=0x7f10f000b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:25.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.664+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 >> v1:172.21.15.92:6789/0 conn(0x7f10f0001ec0 legacy=0x7f10f000b8e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-14T15:27:25.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.664+0000 7f110ed9d700 1 -- 172.21.15.90:0/2220364075 reap_dead start 2024-04-14T15:27:25.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.671+0000 7f1114e5a700 1 -- 172.21.15.90:0/2220364075 --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 3) -- 0x7f10f0013250 con 0x7f10f000f720 2024-04-14T15:27:25.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.672+0000 7f10ff7fe700 10 client.50023.objecter ms_handle_connect 0x7f10f000f720 2024-04-14T15:27:25.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.672+0000 7f10ff7fe700 10 client.50023.objecter resend_mon_ops 2024-04-14T15:27:25.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.672+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4017335087 0 0) 0x7f10f0013250 con 0x7f10f000f720 2024-04-14T15:27:25.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.672+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f10f000f3d0 con 0x7f10f000f720 2024-04-14T15:27:25.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.672+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1988598604 0 0) 0x7f10f000f3d0 con 0x7f10f000f720 2024-04-14T15:27:25.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.672+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f10f0013250 con 0x7f10f000f720 2024-04-14T15:27:25.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.672+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f1108005350 con 0x7f10f000f720 2024-04-14T15:27:25.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.672+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2789744617 0 0) 0x7f10f0013250 con 0x7f10f000f720 2024-04-14T15:27:25.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.672+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 --> v1:172.21.15.92:6789/0 -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=600"]}) -- 0x7f10f0014510 con 0x7f10f000f720 2024-04-14T15:27:25.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.674+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 <== mon.1 v1:172.21.15.92:6789/0 5 ==== command_reply(tid 5: 0 mon_osd_down_out_interval = '' (not observed, change may require restart) ) ==== 82+0+3 (unknown 285646418 0 2423995997) 0x7f1108005580 con 0x7f10f000f720 2024-04-14T15:27:25.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.674+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 >> v1:172.21.15.92:6789/0 conn(0x7f10f000f720 legacy=0x7f10f0011b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:25.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.674+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 >> v1:172.21.15.92:6789/0 conn(0x7f10f000f720 legacy=0x7f10f0011b30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-14T15:27:25.674 INFO:teuthology.orchestra.run.smithi090.stdout:mon.b: {} 2024-04-14T15:27:25.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.674+0000 7f1114e5a700 1 -- 172.21.15.90:0/2220364075 --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 3) -- 0x7f10f0014210 con 0x7f10f0013580 2024-04-14T15:27:25.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.674+0000 7f10ff7fe700 10 client.50023.objecter ms_handle_connect 0x7f10f0013580 2024-04-14T15:27:25.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.674+0000 7f10ff7fe700 10 client.50023.objecter resend_mon_ops 2024-04-14T15:27:25.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.675+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1744582945 0 0) 0x7f10f0014210 con 0x7f10f0013580 2024-04-14T15:27:25.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.675+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f10f0013250 con 0x7f10f0013580 2024-04-14T15:27:25.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.675+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2900737407 0 0) 0x7f10f0013250 con 0x7f10f0013580 2024-04-14T15:27:25.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.675+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f10f0014210 con 0x7f10f0013580 2024-04-14T15:27:25.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.675+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f1108005bc0 con 0x7f10f0013580 2024-04-14T15:27:25.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.675+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 <== mon.2 v1:172.21.15.90:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 109541955 0 0) 0x7f10f0014210 con 0x7f10f0013580 2024-04-14T15:27:25.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.675+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 --> v1:172.21.15.90:6790/0 -- command(tid 6: {"prefix": "get_command_descriptions"}) -- 0x7f10f001b100 con 0x7f10f0013580 2024-04-14T15:27:25.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.675+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 <== mon.2 v1:172.21.15.90:6790/0 5 ==== command_reply(tid 6: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f1108009480 con 0x7f10f0013580 2024-04-14T15:27:25.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.676+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 >> v1:172.21.15.90:6790/0 conn(0x7f10f0013580 legacy=0x7f10f00139f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:25.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.676+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 >> v1:172.21.15.90:6790/0 conn(0x7f10f0013580 legacy=0x7f10f00139f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-14T15:27:25.682 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:27:25.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.683+0000 7f1114e5a700 1 -- 172.21.15.90:0/2220364075 --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 3) -- 0x7f10f0018e60 con 0x7f10f001b2e0 2024-04-14T15:27:25.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.683+0000 7f10ff7fe700 10 client.50023.objecter ms_handle_connect 0x7f10f001b2e0 2024-04-14T15:27:25.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.684+0000 7f10ff7fe700 10 client.50023.objecter resend_mon_ops 2024-04-14T15:27:25.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.684+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2382001165 0 0) 0x7f10f0018e60 con 0x7f10f001b2e0 2024-04-14T15:27:25.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.684+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f10f0013250 con 0x7f10f001b2e0 2024-04-14T15:27:25.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.684+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 990290147 0 0) 0x7f10f0013250 con 0x7f10f001b2e0 2024-04-14T15:27:25.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.684+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f10f0018e60 con 0x7f10f001b2e0 2024-04-14T15:27:25.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.684+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f1108007270 con 0x7f10f001b2e0 2024-04-14T15:27:25.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.684+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 <== mon.2 v1:172.21.15.90:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3799189943 0 0) 0x7f10f0018e60 con 0x7f10f001b2e0 2024-04-14T15:27:25.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.684+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 --> v1:172.21.15.90:6790/0 -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=600"]}) -- 0x7f10f001a0d0 con 0x7f10f001b2e0 2024-04-14T15:27:25.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.686+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 <== mon.2 v1:172.21.15.90:6790/0 5 ==== command_reply(tid 7: 0 mon_osd_down_out_interval = '' (not observed, change may require restart) ) ==== 82+0+3 (unknown 285646418 0 2423995997) 0x7f1108007510 con 0x7f10f001b2e0 2024-04-14T15:27:25.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.686+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 >> v1:172.21.15.90:6790/0 conn(0x7f10f001b2e0 legacy=0x7f10f0017740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:25.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.686+0000 7f10ff7fe700 1 -- 172.21.15.90:0/2220364075 >> v1:172.21.15.90:6790/0 conn(0x7f10f001b2e0 legacy=0x7f10f0017740 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-14T15:27:25.686 INFO:teuthology.orchestra.run.smithi090.stdout:mon.c: {} 2024-04-14T15:27:25.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.686+0000 7f1114e5a700 10 librados: watch_flush enter 2024-04-14T15:27:25.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.686+0000 7f1114e5a700 10 librados: watch_flush exit 2024-04-14T15:27:25.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.686+0000 7f1114e5a700 20 client.50023.objecter shutdown clearing up homeless session... 2024-04-14T15:27:25.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.686+0000 7f1114e5a700 10 client.50023.objecter successfully canceled tick 2024-04-14T15:27:25.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.686+0000 7f1114e5a700 1 -- 172.21.15.90:0/2220364075 >> v1:172.21.15.90:6800/420935427 conn(0x7f10f8071140 legacy=0x7f10f8073600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:25.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.686+0000 7f1114e5a700 1 -- 172.21.15.90:0/2220364075 >> v1:172.21.15.92:6789/0 conn(0x7f111010b780 legacy=0x7f11101a2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:25.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.686+0000 7f110ed9d700 1 -- 172.21.15.90:0/2220364075 reap_dead start 2024-04-14T15:27:25.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.686+0000 7f1114e5a700 1 -- 172.21.15.90:0/2220364075 shutdown_connections 2024-04-14T15:27:25.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.686+0000 7f1114e5a700 1 -- 172.21.15.90:0/2220364075 >> 172.21.15.90:0/2220364075 conn(0x7f11100fe3c0 msgr2=0x7f11101028d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:27:25.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.686+0000 7f1114e5a700 1 -- 172.21.15.90:0/2220364075 shutdown_connections 2024-04-14T15:27:25.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.686+0000 7f1114e5a700 1 -- 172.21.15.90:0/2220364075 wait complete. 2024-04-14T15:27:25.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:25.687+0000 7f1114e5a700 1 librados: shutdown 2024-04-14T15:27:25.688 INFO:teuthology.orchestra.run.smithi090.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-14T15:27:25.688 INFO:teuthology.orchestra.run.smithi090.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-14T15:27:25.689 INFO:teuthology.orchestra.run.smithi090.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-14T15:27:25.872 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:27:25 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:27:26.575 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:26 smithi090 ceph-mon[30399]: 6.13 deep-scrub starts 2024-04-14T15:27:26.575 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:26 smithi090 ceph-mon[30399]: 6.13 deep-scrub ok 2024-04-14T15:27:26.575 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:26 smithi090 ceph-mon[30399]: 4.9 scrub starts 2024-04-14T15:27:26.575 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:26 smithi090 ceph-mon[30399]: 4.9 scrub ok 2024-04-14T15:27:26.575 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:26 smithi090 ceph-mon[30399]: pgmap v1307: 182 pgs: 178 active+clean, 4 active+remapped+backfill_toofull; 582 KiB data, 819 MiB used, 714 GiB / 715 GiB avail; 438 B/s rd, 0 op/s; 0 B/s, 10 objects/s recovering 2024-04-14T15:27:26.575 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:26 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:27:26.576 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:26 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:27:26.576 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:26 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:27:26.576 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:26 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:27:26.576 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:26 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2220364075' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-14T15:27:26.576 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:26 smithi090 ceph-mon[41670]: 6.13 deep-scrub starts 2024-04-14T15:27:26.576 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:26 smithi090 ceph-mon[41670]: 6.13 deep-scrub ok 2024-04-14T15:27:26.576 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:26 smithi090 ceph-mon[41670]: 4.9 scrub starts 2024-04-14T15:27:26.576 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:26 smithi090 ceph-mon[41670]: 4.9 scrub ok 2024-04-14T15:27:26.576 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:26 smithi090 ceph-mon[41670]: pgmap v1307: 182 pgs: 178 active+clean, 4 active+remapped+backfill_toofull; 582 KiB data, 819 MiB used, 714 GiB / 715 GiB avail; 438 B/s rd, 0 op/s; 0 B/s, 10 objects/s recovering 2024-04-14T15:27:26.576 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:26 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:27:26.576 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:26 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:27:26.577 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:26 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:27:26.577 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:26 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:27:26.577 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:26 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2220364075' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-14T15:27:26.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:26 smithi092 ceph-mon[33966]: 6.13 deep-scrub starts 2024-04-14T15:27:26.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:26 smithi092 ceph-mon[33966]: 6.13 deep-scrub ok 2024-04-14T15:27:26.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:26 smithi092 ceph-mon[33966]: 4.9 scrub starts 2024-04-14T15:27:26.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:26 smithi092 ceph-mon[33966]: 4.9 scrub ok 2024-04-14T15:27:26.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:26 smithi092 ceph-mon[33966]: pgmap v1307: 182 pgs: 178 active+clean, 4 active+remapped+backfill_toofull; 582 KiB data, 819 MiB used, 714 GiB / 715 GiB avail; 438 B/s rd, 0 op/s; 0 B/s, 10 objects/s recovering 2024-04-14T15:27:26.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:26 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:27:26.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:26 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:27:26.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:26 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:27:26.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:26 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:27:26.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:26 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2220364075' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-14T15:27:27.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:27 smithi092 ceph-mon[33966]: 4.c scrub starts 2024-04-14T15:27:27.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:27 smithi092 ceph-mon[33966]: 4.c scrub ok 2024-04-14T15:27:27.691 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:27 smithi090 ceph-mon[30399]: 4.c scrub starts 2024-04-14T15:27:27.691 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:27 smithi090 ceph-mon[30399]: 4.c scrub ok 2024-04-14T15:27:27.691 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:27 smithi090 ceph-mon[41670]: 4.c scrub starts 2024-04-14T15:27:27.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:27 smithi090 ceph-mon[41670]: 4.c scrub ok 2024-04-14T15:27:28.433 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd reweight 0 1 2024-04-14T15:27:28.594 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:28 smithi090 ceph-mon[41670]: 5.1f scrub starts 2024-04-14T15:27:28.594 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:28 smithi090 ceph-mon[41670]: 5.1f scrub ok 2024-04-14T15:27:28.594 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:28 smithi090 ceph-mon[41670]: pgmap v1308: 182 pgs: 178 active+clean, 4 active+remapped+backfill_toofull; 582 KiB data, 819 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-14T15:27:28.595 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:28 smithi090 ceph-mon[30399]: 5.1f scrub starts 2024-04-14T15:27:28.595 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:28 smithi090 ceph-mon[30399]: 5.1f scrub ok 2024-04-14T15:27:28.595 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:28 smithi090 ceph-mon[30399]: pgmap v1308: 182 pgs: 178 active+clean, 4 active+remapped+backfill_toofull; 582 KiB data, 819 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-14T15:27:28.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:28 smithi092 ceph-mon[33966]: 5.1f scrub starts 2024-04-14T15:27:28.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:28 smithi092 ceph-mon[33966]: 5.1f scrub ok 2024-04-14T15:27:28.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:28 smithi092 ceph-mon[33966]: pgmap v1308: 182 pgs: 178 active+clean, 4 active+remapped+backfill_toofull; 582 KiB data, 819 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-14T15:27:29.205 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-14T15:27:29.205 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:27:29.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:29 smithi092 ceph-mon[33966]: 2.1a deep-scrub starts 2024-04-14T15:27:29.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:29 smithi092 ceph-mon[33966]: 2.1a deep-scrub ok 2024-04-14T15:27:29.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:29 smithi092 ceph-mon[33966]: 4.12 scrub starts 2024-04-14T15:27:29.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:29 smithi092 ceph-mon[33966]: 4.12 scrub ok 2024-04-14T15:27:29.691 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:29 smithi090 ceph-mon[30399]: 2.1a deep-scrub starts 2024-04-14T15:27:29.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:29 smithi090 ceph-mon[30399]: 2.1a deep-scrub ok 2024-04-14T15:27:29.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:29 smithi090 ceph-mon[30399]: 4.12 scrub starts 2024-04-14T15:27:29.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:29 smithi090 ceph-mon[30399]: 4.12 scrub ok 2024-04-14T15:27:29.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:29 smithi090 ceph-mon[41670]: 2.1a deep-scrub starts 2024-04-14T15:27:29.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:29 smithi090 ceph-mon[41670]: 2.1a deep-scrub ok 2024-04-14T15:27:29.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:29 smithi090 ceph-mon[41670]: 4.12 scrub starts 2024-04-14T15:27:29.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:29 smithi090 ceph-mon[41670]: 4.12 scrub ok 2024-04-14T15:27:30.016 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-14T15:27:30.021 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:27:30.275 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:27:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:30 smithi092 ceph-mon[33966]: 1.4 scrub starts 2024-04-14T15:27:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:30 smithi092 ceph-mon[33966]: 1.4 scrub ok 2024-04-14T15:27:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:30 smithi092 ceph-mon[33966]: 1.3 scrub starts 2024-04-14T15:27:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:30 smithi092 ceph-mon[33966]: 1.3 scrub ok 2024-04-14T15:27:30.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:30 smithi092 ceph-mon[33966]: pgmap v1309: 182 pgs: 178 active+clean, 4 active+remapped+backfill_toofull; 582 KiB data, 819 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:27:30.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:30 smithi092 ceph-mon[33966]: 11.11 scrub starts 2024-04-14T15:27:30.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:30 smithi092 ceph-mon[33966]: 11.11 scrub ok 2024-04-14T15:27:30.564 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:30 smithi090 ceph-mon[41670]: 1.4 scrub starts 2024-04-14T15:27:30.564 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:30 smithi090 ceph-mon[41670]: 1.4 scrub ok 2024-04-14T15:27:30.564 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:30 smithi090 ceph-mon[41670]: 1.3 scrub starts 2024-04-14T15:27:30.564 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:30 smithi090 ceph-mon[41670]: 1.3 scrub ok 2024-04-14T15:27:30.564 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:30 smithi090 ceph-mon[41670]: pgmap v1309: 182 pgs: 178 active+clean, 4 active+remapped+backfill_toofull; 582 KiB data, 819 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:27:30.564 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:30 smithi090 ceph-mon[41670]: 11.11 scrub starts 2024-04-14T15:27:30.565 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:30 smithi090 ceph-mon[41670]: 11.11 scrub ok 2024-04-14T15:27:30.565 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:30 smithi090 ceph-mon[30399]: 1.4 scrub starts 2024-04-14T15:27:30.565 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:30 smithi090 ceph-mon[30399]: 1.4 scrub ok 2024-04-14T15:27:30.565 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:30 smithi090 ceph-mon[30399]: 1.3 scrub starts 2024-04-14T15:27:30.565 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:30 smithi090 ceph-mon[30399]: 1.3 scrub ok 2024-04-14T15:27:30.565 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:30 smithi090 ceph-mon[30399]: pgmap v1309: 182 pgs: 178 active+clean, 4 active+remapped+backfill_toofull; 582 KiB data, 819 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:27:30.565 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:30 smithi090 ceph-mon[30399]: 11.11 scrub starts 2024-04-14T15:27:30.565 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:30 smithi090 ceph-mon[30399]: 11.11 scrub ok 2024-04-14T15:27:31.483 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:31 smithi092 ceph-mon[33966]: 2.7 scrub starts 2024-04-14T15:27:31.484 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:31 smithi092 ceph-mon[33966]: 2.7 scrub ok 2024-04-14T15:27:31.484 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:31 smithi092 ceph-mon[33966]: 2.c scrub starts 2024-04-14T15:27:31.484 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:31 smithi092 ceph-mon[33966]: 2.c scrub ok 2024-04-14T15:27:31.484 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:31 smithi092 ceph-mon[33966]: 2.18 deep-scrub starts 2024-04-14T15:27:31.484 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:31 smithi092 ceph-mon[33966]: 2.18 deep-scrub ok 2024-04-14T15:27:31.691 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:31 smithi090 ceph-mon[30399]: 2.7 scrub starts 2024-04-14T15:27:31.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:31 smithi090 ceph-mon[30399]: 2.7 scrub ok 2024-04-14T15:27:31.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:31 smithi090 ceph-mon[30399]: 2.c scrub starts 2024-04-14T15:27:31.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:31 smithi090 ceph-mon[30399]: 2.c scrub ok 2024-04-14T15:27:31.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:31 smithi090 ceph-mon[30399]: 2.18 deep-scrub starts 2024-04-14T15:27:31.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:31 smithi090 ceph-mon[30399]: 2.18 deep-scrub ok 2024-04-14T15:27:31.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:31 smithi090 ceph-mon[41670]: 2.7 scrub starts 2024-04-14T15:27:31.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:31 smithi090 ceph-mon[41670]: 2.7 scrub ok 2024-04-14T15:27:31.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:31 smithi090 ceph-mon[41670]: 2.c scrub starts 2024-04-14T15:27:31.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:31 smithi090 ceph-mon[41670]: 2.c scrub ok 2024-04-14T15:27:31.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:31 smithi090 ceph-mon[41670]: 2.18 deep-scrub starts 2024-04-14T15:27:31.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:31 smithi090 ceph-mon[41670]: 2.18 deep-scrub ok 2024-04-14T15:27:32.028 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-04-14T15:27:32.029 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:27:32.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:32 smithi092 ceph-mon[33966]: 4.1f scrub starts 2024-04-14T15:27:32.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:32 smithi092 ceph-mon[33966]: 4.1f scrub ok 2024-04-14T15:27:32.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:32 smithi092 ceph-mon[33966]: 4.14 scrub starts 2024-04-14T15:27:32.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:32 smithi092 ceph-mon[33966]: 4.14 scrub ok 2024-04-14T15:27:32.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:32 smithi092 ceph-mon[33966]: pgmap v1310: 182 pgs: 178 active+clean, 4 active+remapped+backfill_toofull; 582 KiB data, 819 MiB used, 714 GiB / 715 GiB avail; 930 B/s rd, 0 op/s 2024-04-14T15:27:32.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:32 smithi092 ceph-mon[33966]: 5.8 scrub starts 2024-04-14T15:27:32.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:32 smithi092 ceph-mon[33966]: 5.8 scrub ok 2024-04-14T15:27:32.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:32.435+0000 7f8117ab1700 1 -- 172.21.15.90:0/2970083694 >> v1:172.21.15.90:6789/0 conn(0x7f81101088b0 legacy=0x7f8110108c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:32.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:32.435+0000 7f8117ab1700 1 -- 172.21.15.90:0/2970083694 shutdown_connections 2024-04-14T15:27:32.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:32.435+0000 7f8117ab1700 1 -- 172.21.15.90:0/2970083694 >> 172.21.15.90:0/2970083694 conn(0x7f81100fe3c0 msgr2=0x7f81101007e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:27:32.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:32.436+0000 7f8117ab1700 1 -- 172.21.15.90:0/2970083694 shutdown_connections 2024-04-14T15:27:32.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:32.436+0000 7f8117ab1700 1 -- 172.21.15.90:0/2970083694 wait complete. 2024-04-14T15:27:32.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:32.436+0000 7f8117ab1700 1 librados: starting msgr at 2024-04-14T15:27:32.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:32.436+0000 7f8117ab1700 1 librados: starting objecter 2024-04-14T15:27:32.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:32.436+0000 7f8117ab1700 1 Processor -- start 2024-04-14T15:27:32.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:32.436+0000 7f8117ab1700 1 -- start start 2024-04-14T15:27:32.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:32.436+0000 7f8117ab1700 1 librados: setting wanted keys 2024-04-14T15:27:32.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:32.436+0000 7f8117ab1700 1 librados: calling monclient init 2024-04-14T15:27:32.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:32.436+0000 7f8117ab1700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f81101a9c20 con 0x7f811010f330 2024-04-14T15:27:32.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:32.436+0000 7f8117ab1700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f81101aae60 con 0x7f811010b760 2024-04-14T15:27:32.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:32.436+0000 7f8117ab1700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f81101ac0a0 con 0x7f81101088b0 2024-04-14T15:27:32.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:32.436+0000 7f811584d700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f81101088b0 0x7f8110076650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:58826/0 (socket says 172.21.15.90:58826) 2024-04-14T15:27:32.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:32.436+0000 7f811584d700 1 -- 172.21.15.90:0/466013927 learned_addr learned my addr 172.21.15.90:0/466013927 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:27:32.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:32.436+0000 7f811604e700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f811010f330 0x7f81101a8320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:47102/0 (socket says 172.21.15.90:47102) 2024-04-14T15:27:32.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:32.437+0000 7f8106ffd700 10 client.?.objecter ms_handle_connect 0x7f81101088b0 2024-04-14T15:27:32.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:32.437+0000 7f8106ffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:27:32.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:32.437+0000 7f8106ffd700 10 client.?.objecter ms_handle_connect 0x7f811010b760 2024-04-14T15:27:32.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:32.437+0000 7f8106ffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:27:32.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:32.437+0000 7f8106ffd700 10 client.?.objecter ms_handle_connect 0x7f811010f330 2024-04-14T15:27:32.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:32.437+0000 7f8106ffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:27:32.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:32.437+0000 7f8106ffd700 1 -- 172.21.15.90:0/466013927 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1123644547 0 0) 0x7f81101aae60 con 0x7f811010b760 2024-04-14T15:27:32.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:32.437+0000 7f8106ffd700 1 -- 172.21.15.90:0/466013927 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f80fc0031f0 con 0x7f811010b760 2024-04-14T15:27:32.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:32.437+0000 7f8106ffd700 1 -- 172.21.15.90:0/466013927 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1256624671 0 0) 0x7f81101ac0a0 con 0x7f81101088b0 2024-04-14T15:27:32.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:32.437+0000 7f8106ffd700 1 -- 172.21.15.90:0/466013927 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f81101aae60 con 0x7f81101088b0 2024-04-14T15:27:32.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:32.437+0000 7f8106ffd700 1 -- 172.21.15.90:0/466013927 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1786377972 0 0) 0x7f81101a9c20 con 0x7f811010f330 2024-04-14T15:27:32.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:32.437+0000 7f8106ffd700 1 -- 172.21.15.90:0/466013927 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f81101ac0a0 con 0x7f811010f330 2024-04-14T15:27:32.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:32.437+0000 7f8106ffd700 1 -- 172.21.15.90:0/466013927 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2934198385 0 0) 0x7f80fc0031f0 con 0x7f811010b760 2024-04-14T15:27:32.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:32.437+0000 7f8106ffd700 1 -- 172.21.15.90:0/466013927 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f81101a9c20 con 0x7f811010b760 2024-04-14T15:27:32.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:32.437+0000 7f8106ffd700 1 -- 172.21.15.90:0/466013927 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f8100001890 con 0x7f811010b760 2024-04-14T15:27:32.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:32.437+0000 7f8106ffd700 1 -- 172.21.15.90:0/466013927 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 212324931 0 0) 0x7f81101aae60 con 0x7f81101088b0 2024-04-14T15:27:32.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:32.437+0000 7f8106ffd700 1 -- 172.21.15.90:0/466013927 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f80fc0031f0 con 0x7f81101088b0 2024-04-14T15:27:32.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:32.437+0000 7f8106ffd700 1 -- 172.21.15.90:0/466013927 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1777736591 0 0) 0x7f81101ac0a0 con 0x7f811010f330 2024-04-14T15:27:32.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:32.437+0000 7f8106ffd700 1 -- 172.21.15.90:0/466013927 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f81101aae60 con 0x7f811010f330 2024-04-14T15:27:32.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:32.437+0000 7f8106ffd700 1 -- 172.21.15.90:0/466013927 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f810c004500 con 0x7f81101088b0 2024-04-14T15:27:32.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:32.437+0000 7f8106ffd700 1 -- 172.21.15.90:0/466013927 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f8108001af0 con 0x7f811010f330 2024-04-14T15:27:32.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:32.437+0000 7f8106ffd700 1 -- 172.21.15.90:0/466013927 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2210724532 0 0) 0x7f81101a9c20 con 0x7f811010b760 2024-04-14T15:27:32.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:32.437+0000 7f8106ffd700 1 -- 172.21.15.90:0/466013927 >> v1:172.21.15.90:6790/0 conn(0x7f81101088b0 legacy=0x7f8110076650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:32.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:32.437+0000 7f8106ffd700 1 -- 172.21.15.90:0/466013927 >> v1:172.21.15.90:6789/0 conn(0x7f811010f330 legacy=0x7f81101a8320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:32.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:32.437+0000 7f8106ffd700 1 -- 172.21.15.90:0/466013927 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81101ad2e0 con 0x7f811010b760 2024-04-14T15:27:32.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:32.437+0000 7f8117ab1700 1 -- 172.21.15.90:0/466013927 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f81101ac2d0 con 0x7f811010b760 2024-04-14T15:27:32.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:32.437+0000 7f8117ab1700 10 client.50035.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:27:32.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:32.437+0000 7f8117ab1700 1 -- 172.21.15.90:0/466013927 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f81101ac790 con 0x7f811010b760 2024-04-14T15:27:32.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:32.437+0000 7f8117ab1700 1 librados: init done 2024-04-14T15:27:32.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:32.437+0000 7f8106ffd700 1 -- 172.21.15.90:0/466013927 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f8100001c40 con 0x7f811010b760 2024-04-14T15:27:32.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:32.437+0000 7f8106ffd700 1 -- 172.21.15.90:0/466013927 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f8100004b40 con 0x7f811010b760 2024-04-14T15:27:32.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:32.438+0000 7f8117ab1700 1 -- 172.21.15.90:0/466013927 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80f4005150 con 0x7f811010b760 2024-04-14T15:27:32.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:32.439+0000 7f8106ffd700 1 -- 172.21.15.90:0/466013927 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f8100003930 con 0x7f811010b760 2024-04-14T15:27:32.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:32.439+0000 7f8106ffd700 1 -- 172.21.15.90:0/466013927 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(872..872 src has 467..872) ==== 7649+0+0 (unknown 2164832937 0 0) 0x7f8100089830 con 0x7f811010b760 2024-04-14T15:27:32.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:32.439+0000 7f8106ffd700 10 client.50035.objecter ms_dispatch 0x7f8110001550 osd_map(872..872 src has 467..872) 2024-04-14T15:27:32.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:32.439+0000 7f8106ffd700 3 client.50035.objecter handle_osd_map got epochs [872,872] > 0 2024-04-14T15:27:32.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:32.439+0000 7f8106ffd700 3 client.50035.objecter handle_osd_map decoding full epoch 872 2024-04-14T15:27:32.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:32.439+0000 7f8106ffd700 20 client.50035.objecter dump_active .. 0 homeless 2024-04-14T15:27:32.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:32.439+0000 7f8106ffd700 10 client.50035.objecter ms_handle_connect 0x7f80fc0713b0 2024-04-14T15:27:32.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:32.444+0000 7f8106ffd700 1 -- 172.21.15.90:0/466013927 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f8100056440 con 0x7f811010b760 2024-04-14T15:27:32.511 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:32 smithi090 ceph-mon[41670]: 4.1f scrub starts 2024-04-14T15:27:32.511 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:32 smithi090 ceph-mon[41670]: 4.1f scrub ok 2024-04-14T15:27:32.511 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:32 smithi090 ceph-mon[41670]: 4.14 scrub starts 2024-04-14T15:27:32.512 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:32 smithi090 ceph-mon[41670]: 4.14 scrub ok 2024-04-14T15:27:32.512 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:32 smithi090 ceph-mon[41670]: pgmap v1310: 182 pgs: 178 active+clean, 4 active+remapped+backfill_toofull; 582 KiB data, 819 MiB used, 714 GiB / 715 GiB avail; 930 B/s rd, 0 op/s 2024-04-14T15:27:32.512 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:32 smithi090 ceph-mon[41670]: 5.8 scrub starts 2024-04-14T15:27:32.512 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:32 smithi090 ceph-mon[41670]: 5.8 scrub ok 2024-04-14T15:27:32.512 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:32 smithi090 ceph-mon[30399]: 4.1f scrub starts 2024-04-14T15:27:32.512 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:32 smithi090 ceph-mon[30399]: 4.1f scrub ok 2024-04-14T15:27:32.512 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:32 smithi090 ceph-mon[30399]: 4.14 scrub starts 2024-04-14T15:27:32.512 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:32 smithi090 ceph-mon[30399]: 4.14 scrub ok 2024-04-14T15:27:32.512 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:32 smithi090 ceph-mon[30399]: pgmap v1310: 182 pgs: 178 active+clean, 4 active+remapped+backfill_toofull; 582 KiB data, 819 MiB used, 714 GiB / 715 GiB avail; 930 B/s rd, 0 op/s 2024-04-14T15:27:32.512 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:32 smithi090 ceph-mon[30399]: 5.8 scrub starts 2024-04-14T15:27:32.512 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:32 smithi090 ceph-mon[30399]: 5.8 scrub ok 2024-04-14T15:27:32.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:32.658+0000 7f8117ab1700 1 -- 172.21.15.90:0/466013927 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd reweight", "id": 0, "weight": 1.0} v 0) -- 0x7f80f4005440 con 0x7f811010b760 2024-04-14T15:27:32.708 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-14T15:27:32.946 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:27:33.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:33.279+0000 7f8106ffd700 1 -- 172.21.15.90:0/466013927 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 0, "weight": 1.0}]=0 reweighted osd.0 to 1 (10000) v873) ==== 113+0+0 (unknown 1447950459 0 0) 0x7f810005a0f0 con 0x7f811010b760 2024-04-14T15:27:33.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:33.281+0000 7f8117ab1700 10 librados: watch_flush enter 2024-04-14T15:27:33.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:33.281+0000 7f8117ab1700 10 librados: watch_flush exit 2024-04-14T15:27:33.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:33.281+0000 7f8117ab1700 20 client.50035.objecter shutdown clearing up homeless session... 2024-04-14T15:27:33.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:33.281+0000 7f8117ab1700 10 client.50035.objecter successfully canceled tick 2024-04-14T15:27:33.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:33.281+0000 7f8117ab1700 1 -- 172.21.15.90:0/466013927 >> v1:172.21.15.90:6800/420935427 conn(0x7f80fc0713b0 legacy=0x7f80fc073870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:33.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:33.281+0000 7f8117ab1700 1 -- 172.21.15.90:0/466013927 >> v1:172.21.15.92:6789/0 conn(0x7f811010b760 legacy=0x7f81101a4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:33.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:33.282+0000 7f8117ab1700 1 -- 172.21.15.90:0/466013927 shutdown_connections 2024-04-14T15:27:33.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:33.282+0000 7f8117ab1700 1 -- 172.21.15.90:0/466013927 >> 172.21.15.90:0/466013927 conn(0x7f81100fe3c0 msgr2=0x7f81101126f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:27:33.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:33.282+0000 7f8117ab1700 1 -- 172.21.15.90:0/466013927 shutdown_connections 2024-04-14T15:27:33.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:33.282+0000 7f8117ab1700 1 -- 172.21.15.90:0/466013927 wait complete. 2024-04-14T15:27:33.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:33.282+0000 7f8117ab1700 1 librados: shutdown 2024-04-14T15:27:33.283 INFO:teuthology.orchestra.run.smithi090.stderr:reweighted osd.0 to 1 (10000) 2024-04-14T15:27:33.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:33 smithi092 ceph-mon[33966]: 4.5 scrub starts 2024-04-14T15:27:33.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:33 smithi092 ceph-mon[33966]: 4.5 scrub ok 2024-04-14T15:27:33.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:33 smithi092 ceph-mon[33966]: 6.12 deep-scrub starts 2024-04-14T15:27:33.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:33 smithi092 ceph-mon[33966]: 6.12 deep-scrub ok 2024-04-14T15:27:33.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:33 smithi092 ceph-mon[33966]: 2.4 scrub starts 2024-04-14T15:27:33.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:33 smithi092 ceph-mon[33966]: 2.4 scrub ok 2024-04-14T15:27:33.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:33 smithi092 ceph-mon[33966]: 11.e scrub starts 2024-04-14T15:27:33.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:33 smithi092 ceph-mon[33966]: 11.e scrub ok 2024-04-14T15:27:33.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:33 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/466013927' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-14T15:27:33.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:33 smithi092 ceph-mon[33966]: from='client.50035 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-14T15:27:33.691 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:33 smithi090 ceph-mon[30399]: 4.5 scrub starts 2024-04-14T15:27:33.691 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:33 smithi090 ceph-mon[30399]: 4.5 scrub ok 2024-04-14T15:27:33.691 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:33 smithi090 ceph-mon[30399]: 6.12 deep-scrub starts 2024-04-14T15:27:33.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:33 smithi090 ceph-mon[30399]: 6.12 deep-scrub ok 2024-04-14T15:27:33.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:33 smithi090 ceph-mon[30399]: 2.4 scrub starts 2024-04-14T15:27:33.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:33 smithi090 ceph-mon[30399]: 2.4 scrub ok 2024-04-14T15:27:33.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:33 smithi090 ceph-mon[30399]: 11.e scrub starts 2024-04-14T15:27:33.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:33 smithi090 ceph-mon[30399]: 11.e scrub ok 2024-04-14T15:27:33.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:33 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/466013927' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-14T15:27:33.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:33 smithi090 ceph-mon[30399]: from='client.50035 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-14T15:27:33.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:33 smithi090 ceph-mon[41670]: 4.5 scrub starts 2024-04-14T15:27:33.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:33 smithi090 ceph-mon[41670]: 4.5 scrub ok 2024-04-14T15:27:33.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:33 smithi090 ceph-mon[41670]: 6.12 deep-scrub starts 2024-04-14T15:27:33.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:33 smithi090 ceph-mon[41670]: 6.12 deep-scrub ok 2024-04-14T15:27:33.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:33 smithi090 ceph-mon[41670]: 2.4 scrub starts 2024-04-14T15:27:33.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:33 smithi090 ceph-mon[41670]: 2.4 scrub ok 2024-04-14T15:27:33.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:33 smithi090 ceph-mon[41670]: 11.e scrub starts 2024-04-14T15:27:33.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:33 smithi090 ceph-mon[41670]: 11.e scrub ok 2024-04-14T15:27:33.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:33 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/466013927' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-14T15:27:33.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:33 smithi090 ceph-mon[41670]: from='client.50035 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-14T15:27:34.138 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:27:33 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:27:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:27:34.172 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd primary-affinity 0 1 2024-04-14T15:27:34.431 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:34 smithi092 ceph-mon[33966]: 11.10 scrub starts 2024-04-14T15:27:34.478 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:34 smithi092 ceph-mon[33966]: 11.10 scrub ok 2024-04-14T15:27:34.478 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:34 smithi092 ceph-mon[33966]: 2.6 deep-scrub starts 2024-04-14T15:27:34.478 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:34 smithi092 ceph-mon[33966]: 2.6 deep-scrub ok 2024-04-14T15:27:34.478 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:34 smithi092 ceph-mon[33966]: pgmap v1311: 182 pgs: 178 active+clean, 4 active+remapped+backfill_toofull; 582 KiB data, 819 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:27:34.478 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:34 smithi092 ceph-mon[33966]: from='client.50035 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-14T15:27:34.478 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:34 smithi092 ceph-mon[33966]: osdmap e873: 8 total, 8 up, 8 in 2024-04-14T15:27:34.479 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:34 smithi090 ceph-mon[41670]: 11.10 scrub starts 2024-04-14T15:27:34.479 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:34 smithi090 ceph-mon[41670]: 11.10 scrub ok 2024-04-14T15:27:34.479 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:34 smithi090 ceph-mon[41670]: 2.6 deep-scrub starts 2024-04-14T15:27:34.479 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:34 smithi090 ceph-mon[41670]: 2.6 deep-scrub ok 2024-04-14T15:27:34.479 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:34 smithi090 ceph-mon[41670]: pgmap v1311: 182 pgs: 178 active+clean, 4 active+remapped+backfill_toofull; 582 KiB data, 819 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:27:34.479 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:34 smithi090 ceph-mon[41670]: from='client.50035 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-14T15:27:34.479 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:34 smithi090 ceph-mon[41670]: osdmap e873: 8 total, 8 up, 8 in 2024-04-14T15:27:34.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:34 smithi090 ceph-mon[30399]: 11.10 scrub starts 2024-04-14T15:27:34.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:34 smithi090 ceph-mon[30399]: 11.10 scrub ok 2024-04-14T15:27:34.480 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:34 smithi090 ceph-mon[30399]: 2.6 deep-scrub starts 2024-04-14T15:27:34.480 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:34 smithi090 ceph-mon[30399]: 2.6 deep-scrub ok 2024-04-14T15:27:34.480 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:34 smithi090 ceph-mon[30399]: pgmap v1311: 182 pgs: 178 active+clean, 4 active+remapped+backfill_toofull; 582 KiB data, 819 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:27:34.480 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:34 smithi090 ceph-mon[30399]: from='client.50035 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-14T15:27:34.480 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:34 smithi090 ceph-mon[30399]: osdmap e873: 8 total, 8 up, 8 in 2024-04-14T15:27:34.480 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:27:34.660 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-04-14T15:27:34.660 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:27:35.514 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:35 smithi092 ceph-mon[33966]: 5.9 scrub starts 2024-04-14T15:27:35.514 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:35 smithi092 ceph-mon[33966]: 5.9 scrub ok 2024-04-14T15:27:35.514 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:35 smithi092 ceph-mon[33966]: 4.2 scrub starts 2024-04-14T15:27:35.514 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:35 smithi092 ceph-mon[33966]: 4.2 scrub ok 2024-04-14T15:27:35.514 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:35 smithi092 ceph-mon[33966]: 6.1a scrub starts 2024-04-14T15:27:35.514 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:35 smithi092 ceph-mon[33966]: 6.1a scrub ok 2024-04-14T15:27:35.515 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:35 smithi092 ceph-mon[33966]: osdmap e874: 8 total, 8 up, 8 in 2024-04-14T15:27:35.544 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-14T15:27:35.689 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:35 smithi090 ceph-mon[41670]: 5.9 scrub starts 2024-04-14T15:27:35.689 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:35 smithi090 ceph-mon[41670]: 5.9 scrub ok 2024-04-14T15:27:35.689 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:35 smithi090 ceph-mon[41670]: 4.2 scrub starts 2024-04-14T15:27:35.689 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:35 smithi090 ceph-mon[41670]: 4.2 scrub ok 2024-04-14T15:27:35.689 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:35 smithi090 ceph-mon[41670]: 6.1a scrub starts 2024-04-14T15:27:35.689 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:35 smithi090 ceph-mon[41670]: 6.1a scrub ok 2024-04-14T15:27:35.689 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:35 smithi090 ceph-mon[41670]: osdmap e874: 8 total, 8 up, 8 in 2024-04-14T15:27:35.690 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:35 smithi090 ceph-mon[30399]: 5.9 scrub starts 2024-04-14T15:27:35.690 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:35 smithi090 ceph-mon[30399]: 5.9 scrub ok 2024-04-14T15:27:35.690 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:35 smithi090 ceph-mon[30399]: 4.2 scrub starts 2024-04-14T15:27:35.690 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:35 smithi090 ceph-mon[30399]: 4.2 scrub ok 2024-04-14T15:27:35.690 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:35 smithi090 ceph-mon[30399]: 6.1a scrub starts 2024-04-14T15:27:35.690 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:35 smithi090 ceph-mon[30399]: 6.1a scrub ok 2024-04-14T15:27:35.690 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:35 smithi090 ceph-mon[30399]: osdmap e874: 8 total, 8 up, 8 in 2024-04-14T15:27:35.786 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:27:35.873 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:27:35 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:27:36.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:36.168+0000 7fd3ab709700 1 -- 172.21.15.90:0/3780142033 >> v1:172.21.15.90:6789/0 conn(0x7fd3a410f2f0 legacy=0x7fd3a41117b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:36.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:36.168+0000 7fd3ab709700 1 -- 172.21.15.90:0/3780142033 shutdown_connections 2024-04-14T15:27:36.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:36.168+0000 7fd3ab709700 1 -- 172.21.15.90:0/3780142033 >> 172.21.15.90:0/3780142033 conn(0x7fd3a40fe3a0 msgr2=0x7fd3a41007c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:27:36.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:36.169+0000 7fd3ab709700 1 -- 172.21.15.90:0/3780142033 shutdown_connections 2024-04-14T15:27:36.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:36.169+0000 7fd3ab709700 1 -- 172.21.15.90:0/3780142033 wait complete. 2024-04-14T15:27:36.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:36.169+0000 7fd3ab709700 1 librados: starting msgr at 2024-04-14T15:27:36.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:36.169+0000 7fd3ab709700 1 librados: starting objecter 2024-04-14T15:27:36.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:36.169+0000 7fd3ab709700 1 Processor -- start 2024-04-14T15:27:36.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:36.169+0000 7fd3ab709700 1 -- start start 2024-04-14T15:27:36.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:36.169+0000 7fd3ab709700 1 librados: setting wanted keys 2024-04-14T15:27:36.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:36.169+0000 7fd3ab709700 1 librados: calling monclient init 2024-04-14T15:27:36.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:36.169+0000 7fd3ab709700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd3a41a9c00 con 0x7fd3a410f2f0 2024-04-14T15:27:36.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:36.169+0000 7fd3ab709700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd3a41aae20 con 0x7fd3a410b720 2024-04-14T15:27:36.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:36.169+0000 7fd3ab709700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd3a41ac060 con 0x7fd3a4108870 2024-04-14T15:27:36.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:36.169+0000 7fd3a94a5700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7fd3a4108870 0x7fd3a410eb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:58840/0 (socket says 172.21.15.90:58840) 2024-04-14T15:27:36.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:36.169+0000 7fd3a94a5700 1 -- 172.21.15.90:0/1119133101 learned_addr learned my addr 172.21.15.90:0/1119133101 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:27:36.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:36.170+0000 7fd39a7fc700 10 client.?.objecter ms_handle_connect 0x7fd3a4108870 2024-04-14T15:27:36.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:36.170+0000 7fd39a7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:27:36.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:36.170+0000 7fd39a7fc700 10 client.?.objecter ms_handle_connect 0x7fd3a410f2f0 2024-04-14T15:27:36.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:36.170+0000 7fd39a7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:27:36.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:36.170+0000 7fd39a7fc700 10 client.?.objecter ms_handle_connect 0x7fd3a410b720 2024-04-14T15:27:36.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:36.170+0000 7fd39a7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:27:36.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:36.170+0000 7fd39a7fc700 1 -- 172.21.15.90:0/1119133101 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3338262122 0 0) 0x7fd3a41a9c00 con 0x7fd3a410f2f0 2024-04-14T15:27:36.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:36.170+0000 7fd39a7fc700 1 -- 172.21.15.90:0/1119133101 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd3900035b0 con 0x7fd3a410f2f0 2024-04-14T15:27:36.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:36.170+0000 7fd39a7fc700 1 -- 172.21.15.90:0/1119133101 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3832117563 0 0) 0x7fd3a41ac060 con 0x7fd3a4108870 2024-04-14T15:27:36.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:36.170+0000 7fd39a7fc700 1 -- 172.21.15.90:0/1119133101 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd3a41a9c00 con 0x7fd3a4108870 2024-04-14T15:27:36.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:36.170+0000 7fd39a7fc700 1 -- 172.21.15.90:0/1119133101 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1862133812 0 0) 0x7fd3a41aae20 con 0x7fd3a410b720 2024-04-14T15:27:36.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:36.170+0000 7fd39a7fc700 1 -- 172.21.15.90:0/1119133101 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd3a41ac060 con 0x7fd3a410b720 2024-04-14T15:27:36.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:36.170+0000 7fd39a7fc700 1 -- 172.21.15.90:0/1119133101 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3400175424 0 0) 0x7fd3900035b0 con 0x7fd3a410f2f0 2024-04-14T15:27:36.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:36.170+0000 7fd39a7fc700 1 -- 172.21.15.90:0/1119133101 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd3a41aae20 con 0x7fd3a410f2f0 2024-04-14T15:27:36.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:36.170+0000 7fd39a7fc700 1 -- 172.21.15.90:0/1119133101 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fd39c003bb0 con 0x7fd3a410f2f0 2024-04-14T15:27:36.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:36.170+0000 7fd39a7fc700 1 -- 172.21.15.90:0/1119133101 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1091850681 0 0) 0x7fd3a41ac060 con 0x7fd3a410b720 2024-04-14T15:27:36.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:36.170+0000 7fd39a7fc700 1 -- 172.21.15.90:0/1119133101 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd3900035b0 con 0x7fd3a410b720 2024-04-14T15:27:36.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:36.170+0000 7fd39a7fc700 1 -- 172.21.15.90:0/1119133101 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fd3940018d0 con 0x7fd3a410b720 2024-04-14T15:27:36.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:36.170+0000 7fd39a7fc700 1 -- 172.21.15.90:0/1119133101 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1860354245 0 0) 0x7fd3a41a9c00 con 0x7fd3a4108870 2024-04-14T15:27:36.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:36.170+0000 7fd39a7fc700 1 -- 172.21.15.90:0/1119133101 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd3a41ac060 con 0x7fd3a4108870 2024-04-14T15:27:36.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:36.170+0000 7fd39a7fc700 1 -- 172.21.15.90:0/1119133101 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fd3a0003fd0 con 0x7fd3a4108870 2024-04-14T15:27:36.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:36.170+0000 7fd39a7fc700 1 -- 172.21.15.90:0/1119133101 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3509573134 0 0) 0x7fd3900035b0 con 0x7fd3a410b720 2024-04-14T15:27:36.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:36.170+0000 7fd39a7fc700 1 -- 172.21.15.90:0/1119133101 >> v1:172.21.15.90:6790/0 conn(0x7fd3a4108870 legacy=0x7fd3a410eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:36.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:36.170+0000 7fd39a7fc700 1 -- 172.21.15.90:0/1119133101 >> v1:172.21.15.90:6789/0 conn(0x7fd3a410f2f0 legacy=0x7fd3a41a82c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:36.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:36.170+0000 7fd39a7fc700 1 -- 172.21.15.90:0/1119133101 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3a41ad2a0 con 0x7fd3a410b720 2024-04-14T15:27:36.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:36.170+0000 7fd3ab709700 1 -- 172.21.15.90:0/1119133101 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd3a41a9e30 con 0x7fd3a410b720 2024-04-14T15:27:36.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:36.170+0000 7fd3ab709700 10 client.50038.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:27:36.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:36.170+0000 7fd3ab709700 1 -- 172.21.15.90:0/1119133101 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd3a41aa2f0 con 0x7fd3a410b720 2024-04-14T15:27:36.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:36.170+0000 7fd3ab709700 1 librados: init done 2024-04-14T15:27:36.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:36.170+0000 7fd39a7fc700 1 -- 172.21.15.90:0/1119133101 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fd394003b40 con 0x7fd3a410b720 2024-04-14T15:27:36.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:36.170+0000 7fd39a7fc700 1 -- 172.21.15.90:0/1119133101 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fd394004a70 con 0x7fd3a410b720 2024-04-14T15:27:36.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:36.171+0000 7fd3ab709700 1 -- 172.21.15.90:0/1119133101 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd388005120 con 0x7fd3a410b720 2024-04-14T15:27:36.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:36.172+0000 7fd39a7fc700 1 -- 172.21.15.90:0/1119133101 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fd3940036a0 con 0x7fd3a410b720 2024-04-14T15:27:36.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:36.172+0000 7fd39a7fc700 1 -- 172.21.15.90:0/1119133101 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(876..876 src has 467..876) ==== 7616+0+0 (unknown 3177215795 0 0) 0x7fd394089560 con 0x7fd3a410b720 2024-04-14T15:27:36.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:36.172+0000 7fd39a7fc700 10 client.50038.objecter ms_dispatch 0x7fd3a4001550 osd_map(876..876 src has 467..876) 2024-04-14T15:27:36.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:36.172+0000 7fd39a7fc700 3 client.50038.objecter handle_osd_map got epochs [876,876] > 0 2024-04-14T15:27:36.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:36.172+0000 7fd39a7fc700 3 client.50038.objecter handle_osd_map decoding full epoch 876 2024-04-14T15:27:36.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:36.172+0000 7fd39a7fc700 20 client.50038.objecter dump_active .. 0 homeless 2024-04-14T15:27:36.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:36.172+0000 7fd39a7fc700 10 client.50038.objecter ms_handle_connect 0x7fd3900713b0 2024-04-14T15:27:36.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:36.177+0000 7fd39a7fc700 1 -- 172.21.15.90:0/1119133101 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fd3940561a0 con 0x7fd3a410b720 2024-04-14T15:27:36.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:36.394+0000 7fd3ab709700 1 -- 172.21.15.90:0/1119133101 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} v 0) -- 0x7fd388005c20 con 0x7fd3a410b720 2024-04-14T15:27:36.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:36 smithi092 ceph-mon[33966]: 11.1c scrub starts 2024-04-14T15:27:36.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:36 smithi092 ceph-mon[33966]: 11.1c scrub ok 2024-04-14T15:27:36.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:36 smithi092 ceph-mon[33966]: 5.3 scrub starts 2024-04-14T15:27:36.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:36 smithi092 ceph-mon[33966]: 5.3 scrub ok 2024-04-14T15:27:36.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:36 smithi092 ceph-mon[33966]: pgmap v1314: 182 pgs: 178 active+clean, 4 active+remapped+backfill_toofull; 582 KiB data, 819 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:27:36.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:36 smithi092 ceph-mon[33966]: 5.1c scrub starts 2024-04-14T15:27:36.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:36 smithi092 ceph-mon[33966]: 5.1c scrub ok 2024-04-14T15:27:36.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:36 smithi092 ceph-mon[33966]: osdmap e875: 8 total, 8 up, 8 in 2024-04-14T15:27:36.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:36 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:27:36.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:36 smithi092 ceph-mon[33966]: osdmap e876: 8 total, 8 up, 8 in 2024-04-14T15:27:36.691 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:36 smithi090 ceph-mon[30399]: 11.1c scrub starts 2024-04-14T15:27:36.691 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:36 smithi090 ceph-mon[30399]: 11.1c scrub ok 2024-04-14T15:27:36.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:36 smithi090 ceph-mon[30399]: 5.3 scrub starts 2024-04-14T15:27:36.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:36 smithi090 ceph-mon[30399]: 5.3 scrub ok 2024-04-14T15:27:36.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:36 smithi090 ceph-mon[30399]: pgmap v1314: 182 pgs: 178 active+clean, 4 active+remapped+backfill_toofull; 582 KiB data, 819 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:27:36.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:36 smithi090 ceph-mon[30399]: 5.1c scrub starts 2024-04-14T15:27:36.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:36 smithi090 ceph-mon[30399]: 5.1c scrub ok 2024-04-14T15:27:36.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:36 smithi090 ceph-mon[30399]: osdmap e875: 8 total, 8 up, 8 in 2024-04-14T15:27:36.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:36 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:27:36.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:36 smithi090 ceph-mon[30399]: osdmap e876: 8 total, 8 up, 8 in 2024-04-14T15:27:36.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:36 smithi090 ceph-mon[41670]: 11.1c scrub starts 2024-04-14T15:27:36.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:36 smithi090 ceph-mon[41670]: 11.1c scrub ok 2024-04-14T15:27:36.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:36 smithi090 ceph-mon[41670]: 5.3 scrub starts 2024-04-14T15:27:36.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:36 smithi090 ceph-mon[41670]: 5.3 scrub ok 2024-04-14T15:27:36.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:36 smithi090 ceph-mon[41670]: pgmap v1314: 182 pgs: 178 active+clean, 4 active+remapped+backfill_toofull; 582 KiB data, 819 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:27:36.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:36 smithi090 ceph-mon[41670]: 5.1c scrub starts 2024-04-14T15:27:36.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:36 smithi090 ceph-mon[41670]: 5.1c scrub ok 2024-04-14T15:27:36.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:36 smithi090 ceph-mon[41670]: osdmap e875: 8 total, 8 up, 8 in 2024-04-14T15:27:36.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:36 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:27:36.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:36 smithi090 ceph-mon[41670]: osdmap e876: 8 total, 8 up, 8 in 2024-04-14T15:27:37.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:37.303+0000 7fd39a7fc700 1 -- 172.21.15.90:0/1119133101 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]=0 set osd.0 primary-affinity to 1 (8655362) v877) ==== 133+0+0 (unknown 1649515931 0 0) 0x7fd394059e50 con 0x7fd3a410b720 2024-04-14T15:27:37.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:37.307+0000 7fd3ab709700 10 librados: watch_flush enter 2024-04-14T15:27:37.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:37.307+0000 7fd3ab709700 10 librados: watch_flush exit 2024-04-14T15:27:37.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:37.307+0000 7fd3ab709700 20 client.50038.objecter shutdown clearing up homeless session... 2024-04-14T15:27:37.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:37.307+0000 7fd3ab709700 10 client.50038.objecter successfully canceled tick 2024-04-14T15:27:37.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:37.307+0000 7fd3ab709700 1 -- 172.21.15.90:0/1119133101 >> v1:172.21.15.90:6800/420935427 conn(0x7fd3900713b0 legacy=0x7fd390073870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:37.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:37.307+0000 7fd3ab709700 1 -- 172.21.15.90:0/1119133101 >> v1:172.21.15.92:6789/0 conn(0x7fd3a410b720 legacy=0x7fd3a41a4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:37.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:37.308+0000 7fd3ab709700 1 -- 172.21.15.90:0/1119133101 shutdown_connections 2024-04-14T15:27:37.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:37.308+0000 7fd3ab709700 1 -- 172.21.15.90:0/1119133101 >> 172.21.15.90:0/1119133101 conn(0x7fd3a40fe3a0 msgr2=0x7fd3a4111f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:27:37.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:37.308+0000 7fd3ab709700 1 -- 172.21.15.90:0/1119133101 shutdown_connections 2024-04-14T15:27:37.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:37.309+0000 7fd3ab709700 1 -- 172.21.15.90:0/1119133101 wait complete. 2024-04-14T15:27:37.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:37.309+0000 7fd3ab709700 1 librados: shutdown 2024-04-14T15:27:37.310 INFO:teuthology.orchestra.run.smithi090.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-04-14T15:27:37.533 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-04-14T15:27:37.533 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:27:37.533 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:37 smithi092 ceph-mon[33966]: 1.12 deep-scrub starts 2024-04-14T15:27:37.533 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:37 smithi092 ceph-mon[33966]: 1.12 deep-scrub ok 2024-04-14T15:27:37.534 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:37 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1119133101' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-04-14T15:27:37.534 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:37 smithi092 ceph-mon[33966]: from='client.50038 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-04-14T15:27:37.613 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:37 smithi090 ceph-mon[41670]: 1.12 deep-scrub starts 2024-04-14T15:27:37.613 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:37 smithi090 ceph-mon[41670]: 1.12 deep-scrub ok 2024-04-14T15:27:37.613 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:37 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1119133101' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-04-14T15:27:37.613 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:37 smithi090 ceph-mon[41670]: from='client.50038 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-04-14T15:27:37.614 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:37 smithi090 ceph-mon[30399]: 1.12 deep-scrub starts 2024-04-14T15:27:37.614 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:37 smithi090 ceph-mon[30399]: 1.12 deep-scrub ok 2024-04-14T15:27:37.614 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:37 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1119133101' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-04-14T15:27:37.614 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:37 smithi090 ceph-mon[30399]: from='client.50038 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-04-14T15:27:38.074 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd reweight 1 1 2024-04-14T15:27:38.204 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-14T15:27:38.328 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:27:38.444 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:27:38.565 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:38 smithi092 ceph-mon[33966]: pgmap v1317: 182 pgs: 178 active+clean, 4 active+remapped+backfill_toofull; 582 KiB data, 819 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:27:38.566 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:38 smithi092 ceph-mon[33966]: from='client.50038 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-04-14T15:27:38.566 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:38 smithi092 ceph-mon[33966]: osdmap e877: 8 total, 8 up, 8 in 2024-04-14T15:27:38.566 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:38 smithi092 ceph-mon[33966]: 5.d scrub starts 2024-04-14T15:27:38.566 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:38 smithi092 ceph-mon[33966]: 5.d scrub ok 2024-04-14T15:27:38.574 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:38 smithi090 ceph-mon[41670]: pgmap v1317: 182 pgs: 178 active+clean, 4 active+remapped+backfill_toofull; 582 KiB data, 819 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:27:38.574 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:38 smithi090 ceph-mon[41670]: from='client.50038 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-04-14T15:27:38.574 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:38 smithi090 ceph-mon[41670]: osdmap e877: 8 total, 8 up, 8 in 2024-04-14T15:27:38.575 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:38 smithi090 ceph-mon[41670]: 5.d scrub starts 2024-04-14T15:27:38.575 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:38 smithi090 ceph-mon[41670]: 5.d scrub ok 2024-04-14T15:27:38.575 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:38 smithi090 ceph-mon[30399]: pgmap v1317: 182 pgs: 178 active+clean, 4 active+remapped+backfill_toofull; 582 KiB data, 819 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:27:38.575 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:38 smithi090 ceph-mon[30399]: from='client.50038 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-04-14T15:27:38.575 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:38 smithi090 ceph-mon[30399]: osdmap e877: 8 total, 8 up, 8 in 2024-04-14T15:27:38.575 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:38 smithi090 ceph-mon[30399]: 5.d scrub starts 2024-04-14T15:27:38.575 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:38 smithi090 ceph-mon[30399]: 5.d scrub ok 2024-04-14T15:27:39.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:39 smithi092 ceph-mon[33966]: 6.18 scrub starts 2024-04-14T15:27:39.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:39 smithi092 ceph-mon[33966]: 6.18 scrub ok 2024-04-14T15:27:39.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:39 smithi092 ceph-mon[33966]: 4.1a scrub starts 2024-04-14T15:27:39.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:39 smithi092 ceph-mon[33966]: 4.1a scrub ok 2024-04-14T15:27:39.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:39 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-04-14T15:27:39.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:39 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:27:39.691 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:39 smithi090 ceph-mon[30399]: 6.18 scrub starts 2024-04-14T15:27:39.691 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:39 smithi090 ceph-mon[30399]: 6.18 scrub ok 2024-04-14T15:27:39.691 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:39 smithi090 ceph-mon[30399]: 4.1a scrub starts 2024-04-14T15:27:39.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:39 smithi090 ceph-mon[30399]: 4.1a scrub ok 2024-04-14T15:27:39.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:39 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-04-14T15:27:39.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:39 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:27:39.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:39 smithi090 ceph-mon[41670]: 6.18 scrub starts 2024-04-14T15:27:39.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:39 smithi090 ceph-mon[41670]: 6.18 scrub ok 2024-04-14T15:27:39.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:39 smithi090 ceph-mon[41670]: 4.1a scrub starts 2024-04-14T15:27:39.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:39 smithi090 ceph-mon[41670]: 4.1a scrub ok 2024-04-14T15:27:39.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:39 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-04-14T15:27:39.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:39 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:27:40.201 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-04-14T15:27:40.201 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:27:40.545 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:40 smithi090 ceph-mon[41670]: 4.1d scrub starts 2024-04-14T15:27:40.546 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:40 smithi090 ceph-mon[41670]: 4.1d scrub ok 2024-04-14T15:27:40.546 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:40 smithi090 ceph-mon[41670]: 3.0 deep-scrub starts 2024-04-14T15:27:40.546 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:40 smithi090 ceph-mon[41670]: 3.0 deep-scrub ok 2024-04-14T15:27:40.546 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:40 smithi090 ceph-mon[41670]: 4.b scrub starts 2024-04-14T15:27:40.546 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:40 smithi090 ceph-mon[41670]: 4.b scrub ok 2024-04-14T15:27:40.547 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:40 smithi090 ceph-mon[41670]: pgmap v1319: 182 pgs: 179 active+clean, 3 active+remapped+backfill_toofull; 582 KiB data, 819 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-14T15:27:40.547 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:40 smithi090 ceph-mon[41670]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-04-14T15:27:40.547 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:40 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]': finished 2024-04-14T15:27:40.547 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:40 smithi090 ceph-mon[41670]: osdmap e878: 8 total, 8 up, 8 in 2024-04-14T15:27:40.547 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:40 smithi090 ceph-mon[41670]: 5.f deep-scrub starts 2024-04-14T15:27:40.548 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:40 smithi090 ceph-mon[41670]: 5.f deep-scrub ok 2024-04-14T15:27:40.548 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:40 smithi090 ceph-mon[30399]: 4.1d scrub starts 2024-04-14T15:27:40.548 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:40 smithi090 ceph-mon[30399]: 4.1d scrub ok 2024-04-14T15:27:40.548 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:40 smithi090 ceph-mon[30399]: 3.0 deep-scrub starts 2024-04-14T15:27:40.549 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:40 smithi090 ceph-mon[30399]: 3.0 deep-scrub ok 2024-04-14T15:27:40.549 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:40 smithi090 ceph-mon[30399]: 4.b scrub starts 2024-04-14T15:27:40.549 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:40 smithi090 ceph-mon[30399]: 4.b scrub ok 2024-04-14T15:27:40.549 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:40 smithi090 ceph-mon[30399]: pgmap v1319: 182 pgs: 179 active+clean, 3 active+remapped+backfill_toofull; 582 KiB data, 819 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-14T15:27:40.549 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:40 smithi090 ceph-mon[30399]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-04-14T15:27:40.549 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:40 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]': finished 2024-04-14T15:27:40.550 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:40 smithi090 ceph-mon[30399]: osdmap e878: 8 total, 8 up, 8 in 2024-04-14T15:27:40.550 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:40 smithi090 ceph-mon[30399]: 5.f deep-scrub starts 2024-04-14T15:27:40.550 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:40 smithi090 ceph-mon[30399]: 5.f deep-scrub ok 2024-04-14T15:27:40.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:40.547+0000 7fd1af03a700 1 -- 172.21.15.90:0/3625991183 >> v1:172.21.15.90:6789/0 conn(0x7fd1a8111da0 legacy=0x7fd1a8114190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:40.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:40.547+0000 7fd1af03a700 1 -- 172.21.15.90:0/3625991183 shutdown_connections 2024-04-14T15:27:40.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:40.547+0000 7fd1af03a700 1 -- 172.21.15.90:0/3625991183 >> 172.21.15.90:0/3625991183 conn(0x7fd1a80785b0 msgr2=0x7fd1a807a9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:27:40.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:40.547+0000 7fd1af03a700 1 -- 172.21.15.90:0/3625991183 shutdown_connections 2024-04-14T15:27:40.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:40.547+0000 7fd1af03a700 1 -- 172.21.15.90:0/3625991183 wait complete. 2024-04-14T15:27:40.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:40.547+0000 7fd1af03a700 1 librados: starting msgr at 2024-04-14T15:27:40.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:40.547+0000 7fd1af03a700 1 librados: starting objecter 2024-04-14T15:27:40.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:40.548+0000 7fd1af03a700 1 Processor -- start 2024-04-14T15:27:40.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:40.548+0000 7fd1af03a700 1 -- start start 2024-04-14T15:27:40.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:40.548+0000 7fd1af03a700 1 librados: setting wanted keys 2024-04-14T15:27:40.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:40.548+0000 7fd1af03a700 1 librados: calling monclient init 2024-04-14T15:27:40.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:40.548+0000 7fd1af03a700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd1a810e7e0 con 0x7fd1a8111da0 2024-04-14T15:27:40.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:40.548+0000 7fd1af03a700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd1a810e9f0 con 0x7fd1a8115940 2024-04-14T15:27:40.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:40.548+0000 7fd1af03a700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd1a81b0560 con 0x7fd1a8109110 2024-04-14T15:27:40.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:40.548+0000 7fd1a7fff700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fd1a8111da0 0x7fd1a810bb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:41462/0 (socket says 172.21.15.90:41462) 2024-04-14T15:27:40.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:40.548+0000 7fd1a7fff700 1 -- 172.21.15.90:0/257957803 learned_addr learned my addr 172.21.15.90:0/257957803 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:27:40.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:40.548+0000 7fd1a5ffb700 10 client.?.objecter ms_handle_connect 0x7fd1a8111da0 2024-04-14T15:27:40.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:40.548+0000 7fd1a5ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:27:40.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:40.548+0000 7fd1a5ffb700 10 client.?.objecter ms_handle_connect 0x7fd1a8109110 2024-04-14T15:27:40.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:40.548+0000 7fd1a5ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:27:40.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:40.548+0000 7fd1a5ffb700 10 client.?.objecter ms_handle_connect 0x7fd1a8115940 2024-04-14T15:27:40.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:40.548+0000 7fd1a5ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:27:40.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:40.548+0000 7fd1a5ffb700 1 -- 172.21.15.90:0/257957803 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2327370940 0 0) 0x7fd1a810e7e0 con 0x7fd1a8111da0 2024-04-14T15:27:40.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:40.548+0000 7fd1a5ffb700 1 -- 172.21.15.90:0/257957803 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1980035b0 con 0x7fd1a8111da0 2024-04-14T15:27:40.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:40.548+0000 7fd1a5ffb700 1 -- 172.21.15.90:0/257957803 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 101608066 0 0) 0x7fd1a810e9f0 con 0x7fd1a8115940 2024-04-14T15:27:40.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:40.548+0000 7fd1a5ffb700 1 -- 172.21.15.90:0/257957803 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1a810e7e0 con 0x7fd1a8115940 2024-04-14T15:27:40.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:40.548+0000 7fd1a5ffb700 1 -- 172.21.15.90:0/257957803 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2946300663 0 0) 0x7fd1a81b0560 con 0x7fd1a8109110 2024-04-14T15:27:40.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:40.548+0000 7fd1a5ffb700 1 -- 172.21.15.90:0/257957803 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1a810e9f0 con 0x7fd1a8109110 2024-04-14T15:27:40.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:40.549+0000 7fd1a5ffb700 1 -- 172.21.15.90:0/257957803 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3212064645 0 0) 0x7fd1980035b0 con 0x7fd1a8111da0 2024-04-14T15:27:40.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:40.549+0000 7fd1a5ffb700 1 -- 172.21.15.90:0/257957803 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd1a81b0560 con 0x7fd1a8111da0 2024-04-14T15:27:40.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:40.549+0000 7fd1a5ffb700 1 -- 172.21.15.90:0/257957803 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fd1940045e0 con 0x7fd1a8111da0 2024-04-14T15:27:40.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:40.549+0000 7fd1a5ffb700 1 -- 172.21.15.90:0/257957803 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 505232266 0 0) 0x7fd1a810e9f0 con 0x7fd1a8109110 2024-04-14T15:27:40.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:40.549+0000 7fd1a5ffb700 1 -- 172.21.15.90:0/257957803 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd1980035b0 con 0x7fd1a8109110 2024-04-14T15:27:40.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:40.549+0000 7fd1a5ffb700 1 -- 172.21.15.90:0/257957803 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fd19c0017b0 con 0x7fd1a8109110 2024-04-14T15:27:40.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:40.549+0000 7fd1a5ffb700 1 -- 172.21.15.90:0/257957803 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 974766995 0 0) 0x7fd1a81b0560 con 0x7fd1a8111da0 2024-04-14T15:27:40.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:40.549+0000 7fd1a5ffb700 1 -- 172.21.15.90:0/257957803 >> v1:172.21.15.90:6790/0 conn(0x7fd1a8109110 legacy=0x7fd1a810b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:40.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:40.549+0000 7fd1a5ffb700 1 -- 172.21.15.90:0/257957803 >> v1:172.21.15.92:6789/0 conn(0x7fd1a8115940 legacy=0x7fd1a81acde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:40.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:40.549+0000 7fd1a5ffb700 1 -- 172.21.15.90:0/257957803 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1a81b1780 con 0x7fd1a8111da0 2024-04-14T15:27:40.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:40.549+0000 7fd1af03a700 1 -- 172.21.15.90:0/257957803 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd1a81b0790 con 0x7fd1a8111da0 2024-04-14T15:27:40.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:40.549+0000 7fd1a5ffb700 1 -- 172.21.15.90:0/257957803 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fd194001ae0 con 0x7fd1a8111da0 2024-04-14T15:27:40.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:40.549+0000 7fd1a5ffb700 1 -- 172.21.15.90:0/257957803 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fd1940050c0 con 0x7fd1a8111da0 2024-04-14T15:27:40.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:40.549+0000 7fd1af03a700 10 client.38223.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:27:40.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:40.549+0000 7fd1af03a700 1 -- 172.21.15.90:0/257957803 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd1a81b0c50 con 0x7fd1a8111da0 2024-04-14T15:27:40.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:40.549+0000 7fd1af03a700 1 librados: init done 2024-04-14T15:27:40.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:40.550+0000 7fd1af03a700 1 -- 172.21.15.90:0/257957803 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1a810a5f0 con 0x7fd1a8111da0 2024-04-14T15:27:40.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:40.550+0000 7fd1a5ffb700 1 -- 172.21.15.90:0/257957803 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fd194005420 con 0x7fd1a8111da0 2024-04-14T15:27:40.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:40.551+0000 7fd1a5ffb700 1 -- 172.21.15.90:0/257957803 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(879..879 src has 467..879) ==== 7616+0+0 (unknown 2559867657 0 0) 0x7fd19408ad00 con 0x7fd1a8111da0 2024-04-14T15:27:40.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:40.551+0000 7fd1a5ffb700 10 client.38223.objecter ms_dispatch 0x7fd1a8001550 osd_map(879..879 src has 467..879) 2024-04-14T15:27:40.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:40.551+0000 7fd1a5ffb700 3 client.38223.objecter handle_osd_map got epochs [879,879] > 0 2024-04-14T15:27:40.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:40.551+0000 7fd1a5ffb700 3 client.38223.objecter handle_osd_map decoding full epoch 879 2024-04-14T15:27:40.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:40.551+0000 7fd1a5ffb700 20 client.38223.objecter dump_active .. 0 homeless 2024-04-14T15:27:40.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:40.551+0000 7fd1a5ffb700 10 client.38223.objecter ms_handle_connect 0x7fd1980712b0 2024-04-14T15:27:40.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:40.556+0000 7fd1a5ffb700 1 -- 172.21.15.90:0/257957803 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fd194057940 con 0x7fd1a8111da0 2024-04-14T15:27:40.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:40 smithi092 ceph-mon[33966]: 4.1d scrub starts 2024-04-14T15:27:40.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:40 smithi092 ceph-mon[33966]: 4.1d scrub ok 2024-04-14T15:27:40.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:40 smithi092 ceph-mon[33966]: 3.0 deep-scrub starts 2024-04-14T15:27:40.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:40 smithi092 ceph-mon[33966]: 3.0 deep-scrub ok 2024-04-14T15:27:40.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:40 smithi092 ceph-mon[33966]: 4.b scrub starts 2024-04-14T15:27:40.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:40 smithi092 ceph-mon[33966]: 4.b scrub ok 2024-04-14T15:27:40.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:40 smithi092 ceph-mon[33966]: pgmap v1319: 182 pgs: 179 active+clean, 3 active+remapped+backfill_toofull; 582 KiB data, 819 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-14T15:27:40.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:40 smithi092 ceph-mon[33966]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-04-14T15:27:40.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:40 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]': finished 2024-04-14T15:27:40.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:40 smithi092 ceph-mon[33966]: osdmap e878: 8 total, 8 up, 8 in 2024-04-14T15:27:40.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:40 smithi092 ceph-mon[33966]: 5.f deep-scrub starts 2024-04-14T15:27:40.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:40 smithi092 ceph-mon[33966]: 5.f deep-scrub ok 2024-04-14T15:27:40.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:40.776+0000 7fd1af03a700 1 -- 172.21.15.90:0/257957803 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd reweight", "id": 1, "weight": 1.0} v 0) -- 0x7fd1a810cb60 con 0x7fd1a8111da0 2024-04-14T15:27:40.863 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-14T15:27:41.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:41.047+0000 7fd1a5ffb700 1 -- 172.21.15.90:0/257957803 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 1, "weight": 1.0}]=0 reweighted osd.1 to 1 (10000) v880) ==== 113+0+0 (unknown 2176738373 0 0) 0x7fd19405b5f0 con 0x7fd1a8111da0 2024-04-14T15:27:41.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:41.050+0000 7fd1af03a700 10 librados: watch_flush enter 2024-04-14T15:27:41.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:41.050+0000 7fd1af03a700 10 librados: watch_flush exit 2024-04-14T15:27:41.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:41.050+0000 7fd1af03a700 20 client.38223.objecter shutdown clearing up homeless session... 2024-04-14T15:27:41.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:41.050+0000 7fd1af03a700 10 client.38223.objecter successfully canceled tick 2024-04-14T15:27:41.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:41.050+0000 7fd1af03a700 1 -- 172.21.15.90:0/257957803 >> v1:172.21.15.90:6800/420935427 conn(0x7fd1980712b0 legacy=0x7fd198073770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:41.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:41.050+0000 7fd1af03a700 1 -- 172.21.15.90:0/257957803 >> v1:172.21.15.90:6789/0 conn(0x7fd1a8111da0 legacy=0x7fd1a810bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:41.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:41.051+0000 7fd1af03a700 1 -- 172.21.15.90:0/257957803 shutdown_connections 2024-04-14T15:27:41.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:41.051+0000 7fd1af03a700 1 -- 172.21.15.90:0/257957803 >> 172.21.15.90:0/257957803 conn(0x7fd1a80785b0 msgr2=0x7fd1a8118e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:27:41.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:41.051+0000 7fd1af03a700 1 -- 172.21.15.90:0/257957803 shutdown_connections 2024-04-14T15:27:41.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:41.051+0000 7fd1af03a700 1 -- 172.21.15.90:0/257957803 wait complete. 2024-04-14T15:27:41.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:41.051+0000 7fd1af03a700 1 librados: shutdown 2024-04-14T15:27:41.054 INFO:teuthology.orchestra.run.smithi090.stderr:reweighted osd.1 to 1 (10000) 2024-04-14T15:27:41.111 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:27:41.545 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:41 smithi092 ceph-mon[33966]: 4.d scrub starts 2024-04-14T15:27:41.546 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:41 smithi092 ceph-mon[33966]: 4.d scrub ok 2024-04-14T15:27:41.546 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:41 smithi092 ceph-mon[33966]: 5.1d scrub starts 2024-04-14T15:27:41.546 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:41 smithi092 ceph-mon[33966]: 5.1d scrub ok 2024-04-14T15:27:41.546 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:41 smithi092 ceph-mon[33966]: osdmap e879: 8 total, 8 up, 8 in 2024-04-14T15:27:41.546 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:41 smithi092 ceph-mon[33966]: 2.e scrub starts 2024-04-14T15:27:41.546 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:41 smithi092 ceph-mon[33966]: 2.e scrub ok 2024-04-14T15:27:41.546 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:41 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/257957803' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-14T15:27:41.546 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:41 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/257957803' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-14T15:27:41.546 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:41 smithi092 ceph-mon[33966]: osdmap e880: 8 total, 8 up, 8 in 2024-04-14T15:27:41.546 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:41 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-04-14T15:27:41.634 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:41 smithi090 ceph-mon[41670]: 4.d scrub starts 2024-04-14T15:27:41.634 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:41 smithi090 ceph-mon[41670]: 4.d scrub ok 2024-04-14T15:27:41.634 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:41 smithi090 ceph-mon[41670]: 5.1d scrub starts 2024-04-14T15:27:41.634 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:41 smithi090 ceph-mon[41670]: 5.1d scrub ok 2024-04-14T15:27:41.634 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:41 smithi090 ceph-mon[41670]: osdmap e879: 8 total, 8 up, 8 in 2024-04-14T15:27:41.635 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:41 smithi090 ceph-mon[41670]: 2.e scrub starts 2024-04-14T15:27:41.635 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:41 smithi090 ceph-mon[41670]: 2.e scrub ok 2024-04-14T15:27:41.635 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:41 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/257957803' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-14T15:27:41.635 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:41 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/257957803' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-14T15:27:41.635 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:41 smithi090 ceph-mon[41670]: osdmap e880: 8 total, 8 up, 8 in 2024-04-14T15:27:41.635 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:41 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-04-14T15:27:41.635 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:41 smithi090 ceph-mon[30399]: 4.d scrub starts 2024-04-14T15:27:41.635 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:41 smithi090 ceph-mon[30399]: 4.d scrub ok 2024-04-14T15:27:41.635 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:41 smithi090 ceph-mon[30399]: 5.1d scrub starts 2024-04-14T15:27:41.635 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:41 smithi090 ceph-mon[30399]: 5.1d scrub ok 2024-04-14T15:27:41.635 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:41 smithi090 ceph-mon[30399]: osdmap e879: 8 total, 8 up, 8 in 2024-04-14T15:27:41.635 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:41 smithi090 ceph-mon[30399]: 2.e scrub starts 2024-04-14T15:27:41.636 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:41 smithi090 ceph-mon[30399]: 2.e scrub ok 2024-04-14T15:27:41.636 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:41 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/257957803' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-14T15:27:41.636 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:41 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/257957803' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-14T15:27:41.636 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:41 smithi090 ceph-mon[30399]: osdmap e880: 8 total, 8 up, 8 in 2024-04-14T15:27:41.636 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:41 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-04-14T15:27:42.163 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd primary-affinity 1 1 2024-04-14T15:27:42.409 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:27:42.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:42 smithi090 ceph-mon[41670]: 6.16 scrub starts 2024-04-14T15:27:42.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:42 smithi090 ceph-mon[41670]: 6.16 scrub ok 2024-04-14T15:27:42.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:42 smithi090 ceph-mon[41670]: pgmap v1323: 181 pgs: 178 active+clean, 3 active+remapped+backfill_toofull; 582 KiB data, 819 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:27:42.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:42 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:27:42.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:42 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:27:42.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:42 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]': finished 2024-04-14T15:27:42.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:42 smithi090 ceph-mon[41670]: osdmap e881: 8 total, 8 up, 8 in 2024-04-14T15:27:42.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:42 smithi090 ceph-mon[30399]: 6.16 scrub starts 2024-04-14T15:27:42.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:42 smithi090 ceph-mon[30399]: 6.16 scrub ok 2024-04-14T15:27:42.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:42 smithi090 ceph-mon[30399]: pgmap v1323: 181 pgs: 178 active+clean, 3 active+remapped+backfill_toofull; 582 KiB data, 819 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:27:42.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:42 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:27:42.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:42 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:27:42.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:42 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]': finished 2024-04-14T15:27:42.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:42 smithi090 ceph-mon[30399]: osdmap e881: 8 total, 8 up, 8 in 2024-04-14T15:27:42.587 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:42 smithi092 ceph-mon[33966]: 6.16 scrub starts 2024-04-14T15:27:42.587 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:42 smithi092 ceph-mon[33966]: 6.16 scrub ok 2024-04-14T15:27:42.587 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:42 smithi092 ceph-mon[33966]: pgmap v1323: 181 pgs: 178 active+clean, 3 active+remapped+backfill_toofull; 582 KiB data, 819 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:27:42.587 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:42 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:27:42.587 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:42 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:27:42.587 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:42 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]': finished 2024-04-14T15:27:42.587 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:42 smithi092 ceph-mon[33966]: osdmap e881: 8 total, 8 up, 8 in 2024-04-14T15:27:42.854 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-04-14T15:27:42.854 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:27:43.447 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:43 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:27:43.448 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:43 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:27:43.448 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:43 smithi092 ceph-mon[33966]: 4.e scrub starts 2024-04-14T15:27:43.448 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:43 smithi092 ceph-mon[33966]: 4.e scrub ok 2024-04-14T15:27:43.448 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:43 smithi092 ceph-mon[33966]: 5.1b scrub starts 2024-04-14T15:27:43.448 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:43 smithi092 ceph-mon[33966]: 5.1b scrub ok 2024-04-14T15:27:43.448 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:43 smithi092 ceph-mon[33966]: osdmap e882: 8 total, 8 up, 8 in 2024-04-14T15:27:43.475 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-14T15:27:43.691 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:43 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:27:43.691 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:43 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:27:43.691 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:43 smithi090 ceph-mon[41670]: 4.e scrub starts 2024-04-14T15:27:43.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:43 smithi090 ceph-mon[41670]: 4.e scrub ok 2024-04-14T15:27:43.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:43 smithi090 ceph-mon[41670]: 5.1b scrub starts 2024-04-14T15:27:43.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:43 smithi090 ceph-mon[41670]: 5.1b scrub ok 2024-04-14T15:27:43.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:43 smithi090 ceph-mon[41670]: osdmap e882: 8 total, 8 up, 8 in 2024-04-14T15:27:43.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:43 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:27:43.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:43 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:27:43.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:43 smithi090 ceph-mon[30399]: 4.e scrub starts 2024-04-14T15:27:43.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:43 smithi090 ceph-mon[30399]: 4.e scrub ok 2024-04-14T15:27:43.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:43 smithi090 ceph-mon[30399]: 5.1b scrub starts 2024-04-14T15:27:43.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:43 smithi090 ceph-mon[30399]: 5.1b scrub ok 2024-04-14T15:27:43.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:43 smithi090 ceph-mon[30399]: osdmap e882: 8 total, 8 up, 8 in 2024-04-14T15:27:43.713 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:27:44.266 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:27:43 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:27:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:27:44.534 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:44 smithi090 ceph-mon[41670]: 4.3 scrub starts 2024-04-14T15:27:44.534 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:44 smithi090 ceph-mon[41670]: 4.3 scrub ok 2024-04-14T15:27:44.534 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:44 smithi090 ceph-mon[41670]: pgmap v1326: 181 pgs: 178 active+clean, 3 active+remapped+backfill_toofull; 582 KiB data, 819 MiB used, 714 GiB / 715 GiB avail 2024-04-14T15:27:44.534 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:44 smithi090 ceph-mon[41670]: 11.1a scrub starts 2024-04-14T15:27:44.534 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:44 smithi090 ceph-mon[41670]: 11.1a scrub ok 2024-04-14T15:27:44.535 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:44 smithi090 ceph-mon[41670]: osdmap e883: 8 total, 8 up, 8 in 2024-04-14T15:27:44.535 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:44 smithi090 ceph-mon[30399]: 4.3 scrub starts 2024-04-14T15:27:44.535 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:44 smithi090 ceph-mon[30399]: 4.3 scrub ok 2024-04-14T15:27:44.535 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:44 smithi090 ceph-mon[30399]: pgmap v1326: 181 pgs: 178 active+clean, 3 active+remapped+backfill_toofull; 582 KiB data, 819 MiB used, 714 GiB / 715 GiB avail 2024-04-14T15:27:44.535 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:44 smithi090 ceph-mon[30399]: 11.1a scrub starts 2024-04-14T15:27:44.536 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:44 smithi090 ceph-mon[30399]: 11.1a scrub ok 2024-04-14T15:27:44.536 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:44 smithi090 ceph-mon[30399]: osdmap e883: 8 total, 8 up, 8 in 2024-04-14T15:27:44.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:44 smithi092 ceph-mon[33966]: 4.3 scrub starts 2024-04-14T15:27:44.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:44 smithi092 ceph-mon[33966]: 4.3 scrub ok 2024-04-14T15:27:44.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:44 smithi092 ceph-mon[33966]: pgmap v1326: 181 pgs: 178 active+clean, 3 active+remapped+backfill_toofull; 582 KiB data, 819 MiB used, 714 GiB / 715 GiB avail 2024-04-14T15:27:44.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:44 smithi092 ceph-mon[33966]: 11.1a scrub starts 2024-04-14T15:27:44.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:44 smithi092 ceph-mon[33966]: 11.1a scrub ok 2024-04-14T15:27:44.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:44 smithi092 ceph-mon[33966]: osdmap e883: 8 total, 8 up, 8 in 2024-04-14T15:27:44.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:44.729+0000 7f21c5313700 1 -- 172.21.15.90:0/2001532750 >> v1:172.21.15.90:6789/0 conn(0x7f21c0075970 legacy=0x7f21c010fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:44.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:44.729+0000 7f21c5313700 1 -- 172.21.15.90:0/2001532750 shutdown_connections 2024-04-14T15:27:44.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:44.729+0000 7f21c5313700 1 -- 172.21.15.90:0/2001532750 >> 172.21.15.90:0/2001532750 conn(0x7f21c00fe3d0 msgr2=0x7f21c01007f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:27:44.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:44.729+0000 7f21c5313700 1 -- 172.21.15.90:0/2001532750 shutdown_connections 2024-04-14T15:27:44.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:44.729+0000 7f21c5313700 1 -- 172.21.15.90:0/2001532750 wait complete. 2024-04-14T15:27:44.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:44.730+0000 7f21c5313700 1 librados: starting msgr at 2024-04-14T15:27:44.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:44.730+0000 7f21c5313700 1 librados: starting objecter 2024-04-14T15:27:44.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:44.730+0000 7f21c5313700 1 Processor -- start 2024-04-14T15:27:44.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:44.730+0000 7f21c5313700 1 -- start start 2024-04-14T15:27:44.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:44.730+0000 7f21c5313700 1 librados: setting wanted keys 2024-04-14T15:27:44.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:44.730+0000 7f21c5313700 1 librados: calling monclient init 2024-04-14T15:27:44.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:44.730+0000 7f21c5313700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f21c010ffb0 con 0x7f21c0075970 2024-04-14T15:27:44.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:44.730+0000 7f21c5313700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f21c01ab0e0 con 0x7f21c0074e80 2024-04-14T15:27:44.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:44.730+0000 7f21c5313700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f21c01ac2c0 con 0x7f21c0111530 2024-04-14T15:27:44.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:44.730+0000 7f21be7fc700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f21c0075970 0x7f21c010d560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:41478/0 (socket says 172.21.15.90:41478) 2024-04-14T15:27:44.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:44.730+0000 7f21be7fc700 1 -- 172.21.15.90:0/1593320067 learned_addr learned my addr 172.21.15.90:0/1593320067 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:27:44.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:44.730+0000 7f21a7fff700 10 client.?.objecter ms_handle_connect 0x7f21c0111530 2024-04-14T15:27:44.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:44.730+0000 7f21a7fff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:27:44.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:44.730+0000 7f21a7fff700 10 client.?.objecter ms_handle_connect 0x7f21c0075970 2024-04-14T15:27:44.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:44.730+0000 7f21a7fff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:27:44.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:44.730+0000 7f21a7fff700 10 client.?.objecter ms_handle_connect 0x7f21c0074e80 2024-04-14T15:27:44.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:44.730+0000 7f21a7fff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:27:44.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:44.731+0000 7f21a7fff700 1 -- 172.21.15.90:0/1593320067 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 980712931 0 0) 0x7f21c01ac2c0 con 0x7f21c0111530 2024-04-14T15:27:44.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:44.731+0000 7f21a7fff700 1 -- 172.21.15.90:0/1593320067 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f21a80035b0 con 0x7f21c0111530 2024-04-14T15:27:44.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:44.731+0000 7f21a7fff700 1 -- 172.21.15.90:0/1593320067 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 497264700 0 0) 0x7f21c010ffb0 con 0x7f21c0075970 2024-04-14T15:27:44.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:44.731+0000 7f21a7fff700 1 -- 172.21.15.90:0/1593320067 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f21c01ac2c0 con 0x7f21c0075970 2024-04-14T15:27:44.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:44.731+0000 7f21a7fff700 1 -- 172.21.15.90:0/1593320067 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3932979566 0 0) 0x7f21c01ab0e0 con 0x7f21c0074e80 2024-04-14T15:27:44.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:44.731+0000 7f21a7fff700 1 -- 172.21.15.90:0/1593320067 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f21c010ffb0 con 0x7f21c0074e80 2024-04-14T15:27:44.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:44.731+0000 7f21a7fff700 1 -- 172.21.15.90:0/1593320067 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3237008251 0 0) 0x7f21a80035b0 con 0x7f21c0111530 2024-04-14T15:27:44.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:44.731+0000 7f21a7fff700 1 -- 172.21.15.90:0/1593320067 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f21c01ab0e0 con 0x7f21c0111530 2024-04-14T15:27:44.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:44.731+0000 7f21a7fff700 1 -- 172.21.15.90:0/1593320067 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f21b8001b70 con 0x7f21c0111530 2024-04-14T15:27:44.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:44.731+0000 7f21a7fff700 1 -- 172.21.15.90:0/1593320067 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1058596955 0 0) 0x7f21c010ffb0 con 0x7f21c0074e80 2024-04-14T15:27:44.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:44.731+0000 7f21a7fff700 1 -- 172.21.15.90:0/1593320067 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f21a80035b0 con 0x7f21c0074e80 2024-04-14T15:27:44.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:44.731+0000 7f21a7fff700 1 -- 172.21.15.90:0/1593320067 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1475878554 0 0) 0x7f21c01ac2c0 con 0x7f21c0075970 2024-04-14T15:27:44.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:44.731+0000 7f21a7fff700 1 -- 172.21.15.90:0/1593320067 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f21c010ffb0 con 0x7f21c0075970 2024-04-14T15:27:44.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:44.731+0000 7f21a7fff700 1 -- 172.21.15.90:0/1593320067 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f21b0001850 con 0x7f21c0074e80 2024-04-14T15:27:44.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:44.731+0000 7f21a7fff700 1 -- 172.21.15.90:0/1593320067 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f21b4001430 con 0x7f21c0075970 2024-04-14T15:27:44.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:44.731+0000 7f21a7fff700 1 -- 172.21.15.90:0/1593320067 <== mon.2 v1:172.21.15.90:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1752524930 0 0) 0x7f21c01ab0e0 con 0x7f21c0111530 2024-04-14T15:27:44.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:44.731+0000 7f21a7fff700 1 -- 172.21.15.90:0/1593320067 >> v1:172.21.15.92:6789/0 conn(0x7f21c0074e80 legacy=0x7f21c010ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:44.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:44.731+0000 7f21a7fff700 1 -- 172.21.15.90:0/1593320067 >> v1:172.21.15.90:6789/0 conn(0x7f21c0075970 legacy=0x7f21c010d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:44.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:44.731+0000 7f21a7fff700 1 -- 172.21.15.90:0/1593320067 --> v1:172.21.15.90:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21c01ad4a0 con 0x7f21c0111530 2024-04-14T15:27:44.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:44.731+0000 7f21c5313700 1 -- 172.21.15.90:0/1593320067 --> v1:172.21.15.90:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f21c01aa130 con 0x7f21c0111530 2024-04-14T15:27:44.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:44.731+0000 7f21c5313700 10 client.48179.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:27:44.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:44.731+0000 7f21c5313700 1 -- 172.21.15.90:0/1593320067 --> v1:172.21.15.90:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f21c01aa5f0 con 0x7f21c0111530 2024-04-14T15:27:44.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:44.731+0000 7f21c5313700 1 librados: init done 2024-04-14T15:27:44.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:44.731+0000 7f21a7fff700 1 -- 172.21.15.90:0/1593320067 <== mon.2 v1:172.21.15.90:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f21b8003ea0 con 0x7f21c0111530 2024-04-14T15:27:44.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:44.731+0000 7f21a7fff700 1 -- 172.21.15.90:0/1593320067 <== mon.2 v1:172.21.15.90:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f21b8004e50 con 0x7f21c0111530 2024-04-14T15:27:44.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:44.732+0000 7f21c5313700 1 -- 172.21.15.90:0/1593320067 --> v1:172.21.15.90:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21ac005120 con 0x7f21c0111530 2024-04-14T15:27:44.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:44.733+0000 7f21a7fff700 1 -- 172.21.15.90:0/1593320067 <== mon.2 v1:172.21.15.90:6790/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f21b8003a00 con 0x7f21c0111530 2024-04-14T15:27:44.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:44.733+0000 7f21a7fff700 1 -- 172.21.15.90:0/1593320067 <== mon.2 v1:172.21.15.90:6790/0 8 ==== osd_map(883..883 src has 467..883) ==== 7616+0+0 (unknown 1020023263 0 0) 0x7f21b8089960 con 0x7f21c0111530 2024-04-14T15:27:44.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:44.733+0000 7f21a7fff700 10 client.48179.objecter ms_dispatch 0x7f21c0001550 osd_map(883..883 src has 467..883) 2024-04-14T15:27:44.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:44.733+0000 7f21a7fff700 3 client.48179.objecter handle_osd_map got epochs [883,883] > 0 2024-04-14T15:27:44.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:44.733+0000 7f21a7fff700 3 client.48179.objecter handle_osd_map decoding full epoch 883 2024-04-14T15:27:44.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:44.733+0000 7f21a7fff700 20 client.48179.objecter dump_active .. 0 homeless 2024-04-14T15:27:44.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:44.733+0000 7f21a7fff700 10 client.48179.objecter ms_handle_connect 0x7f21a8071300 2024-04-14T15:27:44.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:44.738+0000 7f21a7fff700 1 -- 172.21.15.90:0/1593320067 <== mon.2 v1:172.21.15.90:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f21b80565a0 con 0x7f21c0111530 2024-04-14T15:27:44.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:44.948+0000 7f21c5313700 1 -- 172.21.15.90:0/1593320067 --> v1:172.21.15.90:6790/0 -- mon_command({"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} v 0) -- 0x7f21ac005c20 con 0x7f21c0111530 2024-04-14T15:27:45.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:45.074+0000 7f21a7fff700 1 -- 172.21.15.90:0/1593320067 <== mon.2 v1:172.21.15.90:6790/0 10 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]=0 set osd.1 primary-affinity to 1 (8655362) v884) ==== 133+0+0 (unknown 828630536 0 0) 0x7f21b805a250 con 0x7f21c0111530 2024-04-14T15:27:45.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:45.078+0000 7f21c5313700 10 librados: watch_flush enter 2024-04-14T15:27:45.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:45.078+0000 7f21c5313700 10 librados: watch_flush exit 2024-04-14T15:27:45.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:45.078+0000 7f21c5313700 20 client.48179.objecter shutdown clearing up homeless session... 2024-04-14T15:27:45.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:45.078+0000 7f21c5313700 10 client.48179.objecter successfully canceled tick 2024-04-14T15:27:45.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:45.078+0000 7f21c5313700 1 -- 172.21.15.90:0/1593320067 >> v1:172.21.15.90:6800/420935427 conn(0x7f21a8071300 legacy=0x7f21a80737c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:45.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:45.078+0000 7f21c5313700 1 -- 172.21.15.90:0/1593320067 >> v1:172.21.15.90:6790/0 conn(0x7f21c0111530 legacy=0x7f21c01a89b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:45.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:45.078+0000 7f21c5313700 1 -- 172.21.15.90:0/1593320067 shutdown_connections 2024-04-14T15:27:45.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:45.078+0000 7f21c5313700 1 -- 172.21.15.90:0/1593320067 >> 172.21.15.90:0/1593320067 conn(0x7f21c00fe3d0 msgr2=0x7f21c0110c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:27:45.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:45.078+0000 7f21c5313700 1 -- 172.21.15.90:0/1593320067 shutdown_connections 2024-04-14T15:27:45.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:45.078+0000 7f21c5313700 1 -- 172.21.15.90:0/1593320067 wait complete. 2024-04-14T15:27:45.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:45.078+0000 7f21c5313700 1 librados: shutdown 2024-04-14T15:27:45.080 INFO:teuthology.orchestra.run.smithi090.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-04-14T15:27:45.486 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-04-14T15:27:45.486 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:27:45.487 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:45 smithi092 ceph-mon[33966]: 6.1f deep-scrub starts 2024-04-14T15:27:45.487 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:45 smithi092 ceph-mon[33966]: 6.1f deep-scrub ok 2024-04-14T15:27:45.487 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:45 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1593320067' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-04-14T15:27:45.487 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:45 smithi092 ceph-mon[33966]: from='client.48179 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-04-14T15:27:45.487 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:45 smithi092 ceph-mon[33966]: from='client.48179 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-04-14T15:27:45.487 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:45 smithi092 ceph-mon[33966]: osdmap e884: 8 total, 8 up, 8 in 2024-04-14T15:27:45.487 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:45 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-04-14T15:27:45.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:45 smithi090 ceph-mon[30399]: 6.1f deep-scrub starts 2024-04-14T15:27:45.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:45 smithi090 ceph-mon[30399]: 6.1f deep-scrub ok 2024-04-14T15:27:45.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:45 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1593320067' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-04-14T15:27:45.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:45 smithi090 ceph-mon[30399]: from='client.48179 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-04-14T15:27:45.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:45 smithi090 ceph-mon[30399]: from='client.48179 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-04-14T15:27:45.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:45 smithi090 ceph-mon[30399]: osdmap e884: 8 total, 8 up, 8 in 2024-04-14T15:27:45.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:45 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-04-14T15:27:45.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:45 smithi090 ceph-mon[41670]: 6.1f deep-scrub starts 2024-04-14T15:27:45.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:45 smithi090 ceph-mon[41670]: 6.1f deep-scrub ok 2024-04-14T15:27:45.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:45 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1593320067' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-04-14T15:27:45.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:45 smithi090 ceph-mon[41670]: from='client.48179 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-04-14T15:27:45.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:45 smithi090 ceph-mon[41670]: from='client.48179 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-04-14T15:27:45.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:45 smithi090 ceph-mon[41670]: osdmap e884: 8 total, 8 up, 8 in 2024-04-14T15:27:45.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:45 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-04-14T15:27:45.795 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:27:45 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:27:46.032 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd reweight 2 1 2024-04-14T15:27:46.208 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-14T15:27:46.285 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:27:46.451 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:27:46.572 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:46 smithi092 ceph-mon[33966]: 5.10 deep-scrub starts 2024-04-14T15:27:46.572 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:46 smithi092 ceph-mon[33966]: 5.10 deep-scrub ok 2024-04-14T15:27:46.572 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:46 smithi092 ceph-mon[33966]: 11.8 scrub starts 2024-04-14T15:27:46.572 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:46 smithi092 ceph-mon[33966]: 11.8 scrub ok 2024-04-14T15:27:46.572 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:46 smithi092 ceph-mon[33966]: pgmap v1329: 180 pgs: 1 clean+premerge+peered, 176 active+clean, 3 active+remapped+backfill_toofull; 582 KiB data, 828 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:27:46.572 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:46 smithi092 ceph-mon[33966]: 11.18 deep-scrub starts 2024-04-14T15:27:46.573 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:46 smithi092 ceph-mon[33966]: 11.18 deep-scrub ok 2024-04-14T15:27:46.573 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:46 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:27:46.573 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:46 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]': finished 2024-04-14T15:27:46.573 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:46 smithi092 ceph-mon[33966]: osdmap e885: 8 total, 8 up, 8 in 2024-04-14T15:27:46.683 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:46 smithi090 ceph-mon[41670]: 5.10 deep-scrub starts 2024-04-14T15:27:46.683 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:46 smithi090 ceph-mon[41670]: 5.10 deep-scrub ok 2024-04-14T15:27:46.684 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:46 smithi090 ceph-mon[41670]: 11.8 scrub starts 2024-04-14T15:27:46.684 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:46 smithi090 ceph-mon[41670]: 11.8 scrub ok 2024-04-14T15:27:46.684 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:46 smithi090 ceph-mon[41670]: pgmap v1329: 180 pgs: 1 clean+premerge+peered, 176 active+clean, 3 active+remapped+backfill_toofull; 582 KiB data, 828 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:27:46.684 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:46 smithi090 ceph-mon[41670]: 11.18 deep-scrub starts 2024-04-14T15:27:46.684 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:46 smithi090 ceph-mon[41670]: 11.18 deep-scrub ok 2024-04-14T15:27:46.684 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:46 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:27:46.684 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:46 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]': finished 2024-04-14T15:27:46.684 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:46 smithi090 ceph-mon[41670]: osdmap e885: 8 total, 8 up, 8 in 2024-04-14T15:27:46.684 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:46 smithi090 ceph-mon[30399]: 5.10 deep-scrub starts 2024-04-14T15:27:46.684 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:46 smithi090 ceph-mon[30399]: 5.10 deep-scrub ok 2024-04-14T15:27:46.684 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:46 smithi090 ceph-mon[30399]: 11.8 scrub starts 2024-04-14T15:27:46.685 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:46 smithi090 ceph-mon[30399]: 11.8 scrub ok 2024-04-14T15:27:46.685 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:46 smithi090 ceph-mon[30399]: pgmap v1329: 180 pgs: 1 clean+premerge+peered, 176 active+clean, 3 active+remapped+backfill_toofull; 582 KiB data, 828 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:27:46.685 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:46 smithi090 ceph-mon[30399]: 11.18 deep-scrub starts 2024-04-14T15:27:46.685 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:46 smithi090 ceph-mon[30399]: 11.18 deep-scrub ok 2024-04-14T15:27:46.685 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:46 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:27:46.685 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:46 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]': finished 2024-04-14T15:27:46.685 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:46 smithi090 ceph-mon[30399]: osdmap e885: 8 total, 8 up, 8 in 2024-04-14T15:27:47.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:47 smithi092 ceph-mon[33966]: 11.16 scrub starts 2024-04-14T15:27:47.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:47 smithi092 ceph-mon[33966]: 11.16 scrub ok 2024-04-14T15:27:47.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:47 smithi092 ceph-mon[33966]: 4.19 scrub starts 2024-04-14T15:27:47.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:47 smithi092 ceph-mon[33966]: 4.19 scrub ok 2024-04-14T15:27:47.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:47 smithi092 ceph-mon[33966]: 2.13 scrub starts 2024-04-14T15:27:47.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:47 smithi092 ceph-mon[33966]: 2.13 scrub ok 2024-04-14T15:27:47.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:47 smithi092 ceph-mon[33966]: 5.b scrub starts 2024-04-14T15:27:47.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:47 smithi092 ceph-mon[33966]: 5.b scrub ok 2024-04-14T15:27:47.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:47 smithi092 ceph-mon[33966]: osdmap e886: 8 total, 8 up, 8 in 2024-04-14T15:27:47.631 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:47 smithi090 ceph-mon[30399]: 11.16 scrub starts 2024-04-14T15:27:47.631 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:47 smithi090 ceph-mon[30399]: 11.16 scrub ok 2024-04-14T15:27:47.631 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:47 smithi090 ceph-mon[30399]: 4.19 scrub starts 2024-04-14T15:27:47.631 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:47 smithi090 ceph-mon[30399]: 4.19 scrub ok 2024-04-14T15:27:47.631 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:47 smithi090 ceph-mon[30399]: 2.13 scrub starts 2024-04-14T15:27:47.631 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:47 smithi090 ceph-mon[30399]: 2.13 scrub ok 2024-04-14T15:27:47.631 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:47 smithi090 ceph-mon[30399]: 5.b scrub starts 2024-04-14T15:27:47.631 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:47 smithi090 ceph-mon[30399]: 5.b scrub ok 2024-04-14T15:27:47.632 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:47 smithi090 ceph-mon[30399]: osdmap e886: 8 total, 8 up, 8 in 2024-04-14T15:27:47.632 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:47 smithi090 ceph-mon[41670]: 11.16 scrub starts 2024-04-14T15:27:47.632 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:47 smithi090 ceph-mon[41670]: 11.16 scrub ok 2024-04-14T15:27:47.632 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:47 smithi090 ceph-mon[41670]: 4.19 scrub starts 2024-04-14T15:27:47.632 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:47 smithi090 ceph-mon[41670]: 4.19 scrub ok 2024-04-14T15:27:47.632 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:47 smithi090 ceph-mon[41670]: 2.13 scrub starts 2024-04-14T15:27:47.632 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:47 smithi090 ceph-mon[41670]: 2.13 scrub ok 2024-04-14T15:27:47.632 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:47 smithi090 ceph-mon[41670]: 5.b scrub starts 2024-04-14T15:27:47.632 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:47 smithi090 ceph-mon[41670]: 5.b scrub ok 2024-04-14T15:27:47.632 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:47 smithi090 ceph-mon[41670]: osdmap e886: 8 total, 8 up, 8 in 2024-04-14T15:27:48.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:48.108+0000 7fedd65de700 1 -- 172.21.15.90:0/2714250100 >> v1:172.21.15.90:6789/0 conn(0x7fedd01023f0 legacy=0x7fedd01027d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:48.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:48.108+0000 7fedd65de700 1 -- 172.21.15.90:0/2714250100 shutdown_connections 2024-04-14T15:27:48.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:48.108+0000 7fedd65de700 1 -- 172.21.15.90:0/2714250100 >> 172.21.15.90:0/2714250100 conn(0x7fedd0076aa0 msgr2=0x7fedd0076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:27:48.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:48.109+0000 7fedd65de700 1 -- 172.21.15.90:0/2714250100 shutdown_connections 2024-04-14T15:27:48.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:48.109+0000 7fedd65de700 1 -- 172.21.15.90:0/2714250100 wait complete. 2024-04-14T15:27:48.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:48.109+0000 7fedd65de700 1 librados: starting msgr at 2024-04-14T15:27:48.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:48.109+0000 7fedd65de700 1 librados: starting objecter 2024-04-14T15:27:48.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:48.109+0000 7fedd65de700 1 Processor -- start 2024-04-14T15:27:48.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:48.109+0000 7fedd65de700 1 -- start start 2024-04-14T15:27:48.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:48.109+0000 7fedd65de700 1 librados: setting wanted keys 2024-04-14T15:27:48.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:48.109+0000 7fedd65de700 1 librados: calling monclient init 2024-04-14T15:27:48.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:48.110+0000 7fedd65de700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fedd01070b0 con 0x7fedd010e250 2024-04-14T15:27:48.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:48.110+0000 7fedd65de700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fedd01072c0 con 0x7fedd01023f0 2024-04-14T15:27:48.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:48.110+0000 7fedd65de700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fedd01ac290 con 0x7fedd010a6b0 2024-04-14T15:27:48.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:48.110+0000 7fedcf7fe700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7fedd010a6b0 0x7fedd01045a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:55690/0 (socket says 172.21.15.90:55690) 2024-04-14T15:27:48.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:48.110+0000 7fedcf7fe700 1 -- 172.21.15.90:0/2183438135 learned_addr learned my addr 172.21.15.90:0/2183438135 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:27:48.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:48.110+0000 7fedcd7fa700 10 client.?.objecter ms_handle_connect 0x7fedd010e250 2024-04-14T15:27:48.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:48.110+0000 7fedcd7fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:27:48.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:48.110+0000 7fedcd7fa700 10 client.?.objecter ms_handle_connect 0x7fedd010a6b0 2024-04-14T15:27:48.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:48.110+0000 7fedcd7fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:27:48.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:48.110+0000 7fedcd7fa700 10 client.?.objecter ms_handle_connect 0x7fedd01023f0 2024-04-14T15:27:48.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:48.110+0000 7fedcd7fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:27:48.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:48.110+0000 7fedcd7fa700 1 -- 172.21.15.90:0/2183438135 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 165190290 0 0) 0x7fedd01070b0 con 0x7fedd010e250 2024-04-14T15:27:48.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:48.110+0000 7fedcd7fa700 1 -- 172.21.15.90:0/2183438135 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fedb80035b0 con 0x7fedd010e250 2024-04-14T15:27:48.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:48.110+0000 7fedcd7fa700 1 -- 172.21.15.90:0/2183438135 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1953992782 0 0) 0x7fedd01072c0 con 0x7fedd01023f0 2024-04-14T15:27:48.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:48.110+0000 7fedcd7fa700 1 -- 172.21.15.90:0/2183438135 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fedd01070b0 con 0x7fedd01023f0 2024-04-14T15:27:48.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:48.110+0000 7fedcd7fa700 1 -- 172.21.15.90:0/2183438135 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3874846200 0 0) 0x7fedd01ac290 con 0x7fedd010a6b0 2024-04-14T15:27:48.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:48.110+0000 7fedcd7fa700 1 -- 172.21.15.90:0/2183438135 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fedd01072c0 con 0x7fedd010a6b0 2024-04-14T15:27:48.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:48.110+0000 7fedcd7fa700 1 -- 172.21.15.90:0/2183438135 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4091289399 0 0) 0x7fedd01070b0 con 0x7fedd01023f0 2024-04-14T15:27:48.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:48.110+0000 7fedcd7fa700 1 -- 172.21.15.90:0/2183438135 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fedd01ac290 con 0x7fedd01023f0 2024-04-14T15:27:48.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:48.110+0000 7fedcd7fa700 1 -- 172.21.15.90:0/2183438135 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3541095702 0 0) 0x7fedb80035b0 con 0x7fedd010e250 2024-04-14T15:27:48.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:48.110+0000 7fedcd7fa700 1 -- 172.21.15.90:0/2183438135 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fedd01070b0 con 0x7fedd010e250 2024-04-14T15:27:48.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:48.110+0000 7fedcd7fa700 1 -- 172.21.15.90:0/2183438135 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fedc0001430 con 0x7fedd01023f0 2024-04-14T15:27:48.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:48.110+0000 7fedcd7fa700 1 -- 172.21.15.90:0/2183438135 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1343675991 0 0) 0x7fedd01072c0 con 0x7fedd010a6b0 2024-04-14T15:27:48.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:48.110+0000 7fedcd7fa700 1 -- 172.21.15.90:0/2183438135 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fedb80035b0 con 0x7fedd010a6b0 2024-04-14T15:27:48.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:48.110+0000 7fedcd7fa700 1 -- 172.21.15.90:0/2183438135 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fedc8001c30 con 0x7fedd010e250 2024-04-14T15:27:48.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:48.110+0000 7fedcd7fa700 1 -- 172.21.15.90:0/2183438135 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fedc4001870 con 0x7fedd010a6b0 2024-04-14T15:27:48.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:48.110+0000 7fedcd7fa700 1 -- 172.21.15.90:0/2183438135 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3287588969 0 0) 0x7fedd01ac290 con 0x7fedd01023f0 2024-04-14T15:27:48.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:48.110+0000 7fedcd7fa700 1 -- 172.21.15.90:0/2183438135 >> v1:172.21.15.90:6790/0 conn(0x7fedd010a6b0 legacy=0x7fedd01045a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:48.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:48.110+0000 7fedcd7fa700 1 -- 172.21.15.90:0/2183438135 >> v1:172.21.15.90:6789/0 conn(0x7fedd010e250 legacy=0x7fedd01a8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:48.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:48.110+0000 7fedcd7fa700 1 -- 172.21.15.90:0/2183438135 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedd01ad4f0 con 0x7fedd01023f0 2024-04-14T15:27:48.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:48.110+0000 7fedd65de700 1 -- 172.21.15.90:0/2183438135 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fedd01ac4c0 con 0x7fedd01023f0 2024-04-14T15:27:48.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:48.111+0000 7fedd65de700 10 client.50056.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:27:48.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:48.111+0000 7fedd65de700 1 -- 172.21.15.90:0/2183438135 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fedd01ac980 con 0x7fedd01023f0 2024-04-14T15:27:48.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:48.111+0000 7fedd65de700 1 librados: init done 2024-04-14T15:27:48.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:48.111+0000 7fedcd7fa700 1 -- 172.21.15.90:0/2183438135 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fedc0003680 con 0x7fedd01023f0 2024-04-14T15:27:48.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:48.111+0000 7fedcd7fa700 1 -- 172.21.15.90:0/2183438135 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fedc0004ca0 con 0x7fedd01023f0 2024-04-14T15:27:48.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:48.111+0000 7fedd65de700 1 -- 172.21.15.90:0/2183438135 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fedbc005120 con 0x7fedd01023f0 2024-04-14T15:27:48.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:48.112+0000 7fedcd7fa700 1 -- 172.21.15.90:0/2183438135 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fedc001b8d0 con 0x7fedd01023f0 2024-04-14T15:27:48.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:48.112+0000 7fedcd7fa700 1 -- 172.21.15.90:0/2183438135 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(887..887 src has 467..887) ==== 7550+0+0 (unknown 1925516964 0 0) 0x7fedc0089520 con 0x7fedd01023f0 2024-04-14T15:27:48.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:48.112+0000 7fedcd7fa700 10 client.50056.objecter ms_dispatch 0x7fedd0001550 osd_map(887..887 src has 467..887) 2024-04-14T15:27:48.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:48.112+0000 7fedcd7fa700 3 client.50056.objecter handle_osd_map got epochs [887,887] > 0 2024-04-14T15:27:48.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:48.112+0000 7fedcd7fa700 3 client.50056.objecter handle_osd_map decoding full epoch 887 2024-04-14T15:27:48.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:48.113+0000 7fedcd7fa700 20 client.50056.objecter dump_active .. 0 homeless 2024-04-14T15:27:48.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:48.113+0000 7fedcd7fa700 10 client.50056.objecter ms_handle_connect 0x7fedb8071400 2024-04-14T15:27:48.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:48.117+0000 7fedcd7fa700 1 -- 172.21.15.90:0/2183438135 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fedc00561a0 con 0x7fedd01023f0 2024-04-14T15:27:48.155 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-04-14T15:27:48.155 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:27:48.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:48.332+0000 7fedd65de700 1 -- 172.21.15.90:0/2183438135 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd reweight", "id": 2, "weight": 1.0} v 0) -- 0x7fedbc005410 con 0x7fedd01023f0 2024-04-14T15:27:48.451 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:48 smithi092 ceph-mon[33966]: 5.15 scrub starts 2024-04-14T15:27:48.452 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:48 smithi092 ceph-mon[33966]: 5.15 scrub ok 2024-04-14T15:27:48.452 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:48 smithi092 ceph-mon[33966]: 11.b scrub starts 2024-04-14T15:27:48.452 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:48 smithi092 ceph-mon[33966]: 11.b scrub ok 2024-04-14T15:27:48.452 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:48 smithi092 ceph-mon[33966]: 11.1b scrub starts 2024-04-14T15:27:48.452 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:48 smithi092 ceph-mon[33966]: 11.1b scrub ok 2024-04-14T15:27:48.452 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:48 smithi092 ceph-mon[33966]: 1.b deep-scrub starts 2024-04-14T15:27:48.452 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:48 smithi092 ceph-mon[33966]: 1.b deep-scrub ok 2024-04-14T15:27:48.452 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:48 smithi092 ceph-mon[33966]: pgmap v1332: 180 pgs: 1 clean+premerge+peered, 176 active+clean, 3 active+remapped+backfill_toofull; 582 KiB data, 828 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:27:48.452 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:48 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:27:48.452 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:48 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:27:48.452 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:48 smithi092 ceph-mon[33966]: osdmap e887: 8 total, 8 up, 8 in 2024-04-14T15:27:48.691 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:48 smithi090 ceph-mon[30399]: 5.15 scrub starts 2024-04-14T15:27:48.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:48 smithi090 ceph-mon[30399]: 5.15 scrub ok 2024-04-14T15:27:48.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:48 smithi090 ceph-mon[30399]: 11.b scrub starts 2024-04-14T15:27:48.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:48 smithi090 ceph-mon[30399]: 11.b scrub ok 2024-04-14T15:27:48.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:48 smithi090 ceph-mon[30399]: 11.1b scrub starts 2024-04-14T15:27:48.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:48 smithi090 ceph-mon[30399]: 11.1b scrub ok 2024-04-14T15:27:48.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:48 smithi090 ceph-mon[30399]: 1.b deep-scrub starts 2024-04-14T15:27:48.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:48 smithi090 ceph-mon[30399]: 1.b deep-scrub ok 2024-04-14T15:27:48.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:48 smithi090 ceph-mon[30399]: pgmap v1332: 180 pgs: 1 clean+premerge+peered, 176 active+clean, 3 active+remapped+backfill_toofull; 582 KiB data, 828 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:27:48.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:48 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:27:48.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:48 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:27:48.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:48 smithi090 ceph-mon[30399]: osdmap e887: 8 total, 8 up, 8 in 2024-04-14T15:27:48.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:48 smithi090 ceph-mon[41670]: 5.15 scrub starts 2024-04-14T15:27:48.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:48 smithi090 ceph-mon[41670]: 5.15 scrub ok 2024-04-14T15:27:48.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:48 smithi090 ceph-mon[41670]: 11.b scrub starts 2024-04-14T15:27:48.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:48 smithi090 ceph-mon[41670]: 11.b scrub ok 2024-04-14T15:27:48.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:48 smithi090 ceph-mon[41670]: 11.1b scrub starts 2024-04-14T15:27:48.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:48 smithi090 ceph-mon[41670]: 11.1b scrub ok 2024-04-14T15:27:48.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:48 smithi090 ceph-mon[41670]: 1.b deep-scrub starts 2024-04-14T15:27:48.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:48 smithi090 ceph-mon[41670]: 1.b deep-scrub ok 2024-04-14T15:27:48.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:48 smithi090 ceph-mon[41670]: pgmap v1332: 180 pgs: 1 clean+premerge+peered, 176 active+clean, 3 active+remapped+backfill_toofull; 582 KiB data, 828 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:27:48.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:48 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:27:48.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:48 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:27:48.694 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:48 smithi090 ceph-mon[41670]: osdmap e887: 8 total, 8 up, 8 in 2024-04-14T15:27:48.754 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-14T15:27:48.994 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:27:49.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:49.061+0000 7fedcd7fa700 1 -- 172.21.15.90:0/2183438135 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 2, "weight": 1.0}]=0 reweighted osd.2 to 1 (10000) v888) ==== 113+0+0 (unknown 2823234695 0 0) 0x7fedc0059e50 con 0x7fedd01023f0 2024-04-14T15:27:49.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:49.065+0000 7fedd65de700 10 librados: watch_flush enter 2024-04-14T15:27:49.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:49.065+0000 7fedd65de700 10 librados: watch_flush exit 2024-04-14T15:27:49.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:49.065+0000 7fedd65de700 20 client.50056.objecter shutdown clearing up homeless session... 2024-04-14T15:27:49.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:49.065+0000 7fedd65de700 10 client.50056.objecter successfully canceled tick 2024-04-14T15:27:49.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:49.065+0000 7fedd65de700 1 -- 172.21.15.90:0/2183438135 >> v1:172.21.15.90:6800/420935427 conn(0x7fedb8071400 legacy=0x7fedb80738c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:49.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:49.065+0000 7fedd65de700 1 -- 172.21.15.90:0/2183438135 >> v1:172.21.15.92:6789/0 conn(0x7fedd01023f0 legacy=0x7fedd0103e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:49.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:49.065+0000 7fedd65de700 1 -- 172.21.15.90:0/2183438135 shutdown_connections 2024-04-14T15:27:49.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:49.065+0000 7fedd65de700 1 -- 172.21.15.90:0/2183438135 >> 172.21.15.90:0/2183438135 conn(0x7fedd0076aa0 msgr2=0x7fedd01106f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:27:49.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:49.066+0000 7fedd65de700 1 -- 172.21.15.90:0/2183438135 shutdown_connections 2024-04-14T15:27:49.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:49.066+0000 7fedd65de700 1 -- 172.21.15.90:0/2183438135 wait complete. 2024-04-14T15:27:49.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:49.066+0000 7fedd65de700 1 librados: shutdown 2024-04-14T15:27:49.067 INFO:teuthology.orchestra.run.smithi090.stderr:reweighted osd.2 to 1 (10000) 2024-04-14T15:27:49.368 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:49 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2183438135' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-14T15:27:49.369 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:49 smithi092 ceph-mon[33966]: from='client.50056 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-14T15:27:49.369 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:49 smithi092 ceph-mon[33966]: from='client.50056 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-14T15:27:49.369 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:49 smithi092 ceph-mon[33966]: osdmap e888: 8 total, 8 up, 8 in 2024-04-14T15:27:49.369 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:49 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-04-14T15:27:49.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:49 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2183438135' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-14T15:27:49.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:49 smithi090 ceph-mon[30399]: from='client.50056 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-14T15:27:49.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:49 smithi090 ceph-mon[30399]: from='client.50056 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-14T15:27:49.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:49 smithi090 ceph-mon[30399]: osdmap e888: 8 total, 8 up, 8 in 2024-04-14T15:27:49.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:49 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-04-14T15:27:49.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:49 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2183438135' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-14T15:27:49.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:49 smithi090 ceph-mon[41670]: from='client.50056 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-14T15:27:49.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:49 smithi090 ceph-mon[41670]: from='client.50056 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-14T15:27:49.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:49 smithi090 ceph-mon[41670]: osdmap e888: 8 total, 8 up, 8 in 2024-04-14T15:27:49.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:49 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-04-14T15:27:49.766 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd primary-affinity 2 1 2024-04-14T15:27:50.014 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:27:50.333 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:50 smithi092 ceph-mon[33966]: 4.1 scrub starts 2024-04-14T15:27:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:50 smithi092 ceph-mon[33966]: 4.1 scrub ok 2024-04-14T15:27:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:50 smithi092 ceph-mon[33966]: 6.9 scrub starts 2024-04-14T15:27:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:50 smithi092 ceph-mon[33966]: 6.9 scrub ok 2024-04-14T15:27:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:50 smithi092 ceph-mon[33966]: pgmap v1335: 179 pgs: 178 active+clean, 1 active+remapped+backfill_toofull; 582 KiB data, 832 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:27:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:50 smithi092 ceph-mon[33966]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-04-14T15:27:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:50 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]': finished 2024-04-14T15:27:50.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:50 smithi092 ceph-mon[33966]: osdmap e889: 8 total, 8 up, 8 in 2024-04-14T15:27:50.394 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:50 smithi090 ceph-mon[30399]: 4.1 scrub starts 2024-04-14T15:27:50.394 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:50 smithi090 ceph-mon[30399]: 4.1 scrub ok 2024-04-14T15:27:50.394 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:50 smithi090 ceph-mon[30399]: 6.9 scrub starts 2024-04-14T15:27:50.395 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:50 smithi090 ceph-mon[30399]: 6.9 scrub ok 2024-04-14T15:27:50.395 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:50 smithi090 ceph-mon[30399]: pgmap v1335: 179 pgs: 178 active+clean, 1 active+remapped+backfill_toofull; 582 KiB data, 832 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:27:50.395 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:50 smithi090 ceph-mon[30399]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-04-14T15:27:50.395 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:50 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]': finished 2024-04-14T15:27:50.395 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:50 smithi090 ceph-mon[30399]: osdmap e889: 8 total, 8 up, 8 in 2024-04-14T15:27:50.395 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:50 smithi090 ceph-mon[41670]: 4.1 scrub starts 2024-04-14T15:27:50.395 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:50 smithi090 ceph-mon[41670]: 4.1 scrub ok 2024-04-14T15:27:50.395 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:50 smithi090 ceph-mon[41670]: 6.9 scrub starts 2024-04-14T15:27:50.395 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:50 smithi090 ceph-mon[41670]: 6.9 scrub ok 2024-04-14T15:27:50.395 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:50 smithi090 ceph-mon[41670]: pgmap v1335: 179 pgs: 178 active+clean, 1 active+remapped+backfill_toofull; 582 KiB data, 832 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:27:50.395 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:50 smithi090 ceph-mon[41670]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-04-14T15:27:50.396 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:50 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]': finished 2024-04-14T15:27:50.396 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:50 smithi090 ceph-mon[41670]: osdmap e889: 8 total, 8 up, 8 in 2024-04-14T15:27:50.630 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-04-14T15:27:50.631 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:27:51.386 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-14T15:27:51.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:51.601+0000 7f5ae80cd700 1 -- 172.21.15.90:0/3288461241 >> v1:172.21.15.90:6789/0 conn(0x7f5ae0108890 legacy=0x7f5ae0108c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:51.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:51.601+0000 7f5ae80cd700 1 -- 172.21.15.90:0/3288461241 shutdown_connections 2024-04-14T15:27:51.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:51.601+0000 7f5ae80cd700 1 -- 172.21.15.90:0/3288461241 >> 172.21.15.90:0/3288461241 conn(0x7f5ae00fe380 msgr2=0x7f5ae01007a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:27:51.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:51.601+0000 7f5ae80cd700 1 -- 172.21.15.90:0/3288461241 shutdown_connections 2024-04-14T15:27:51.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:51.601+0000 7f5ae80cd700 1 -- 172.21.15.90:0/3288461241 wait complete. 2024-04-14T15:27:51.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:51.602+0000 7f5ae80cd700 1 librados: starting msgr at 2024-04-14T15:27:51.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:51.602+0000 7f5ae80cd700 1 librados: starting objecter 2024-04-14T15:27:51.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:51.602+0000 7f5ae80cd700 1 Processor -- start 2024-04-14T15:27:51.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:51.602+0000 7f5ae80cd700 1 -- start start 2024-04-14T15:27:51.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:51.602+0000 7f5ae80cd700 1 librados: setting wanted keys 2024-04-14T15:27:51.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:51.602+0000 7f5ae80cd700 1 librados: calling monclient init 2024-04-14T15:27:51.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:51.602+0000 7f5ae80cd700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5ae01a9d90 con 0x7f5ae010f310 2024-04-14T15:27:51.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:51.602+0000 7f5ae80cd700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5ae01aafb0 con 0x7f5ae010b740 2024-04-14T15:27:51.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:51.602+0000 7f5ae80cd700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5ae01ac1f0 con 0x7f5ae0108890 2024-04-14T15:27:51.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:51.602+0000 7f5ae666a700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f5ae010f310 0x7f5ae01a8450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:40766/0 (socket says 172.21.15.90:40766) 2024-04-14T15:27:51.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:51.602+0000 7f5ae666a700 1 -- 172.21.15.90:0/1531274100 learned_addr learned my addr 172.21.15.90:0/1531274100 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:27:51.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:51.603+0000 7f5ad6ffd700 10 client.?.objecter ms_handle_connect 0x7f5ae010f310 2024-04-14T15:27:51.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:51.603+0000 7f5ad6ffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:27:51.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:51.603+0000 7f5ad6ffd700 10 client.?.objecter ms_handle_connect 0x7f5ae010b740 2024-04-14T15:27:51.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:51.603+0000 7f5ad6ffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:27:51.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:51.603+0000 7f5ad6ffd700 10 client.?.objecter ms_handle_connect 0x7f5ae0108890 2024-04-14T15:27:51.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:51.603+0000 7f5ad6ffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:27:51.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:51.603+0000 7f5ad6ffd700 1 -- 172.21.15.90:0/1531274100 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3227963769 0 0) 0x7f5ae01a9d90 con 0x7f5ae010f310 2024-04-14T15:27:51.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:51.603+0000 7f5ad6ffd700 1 -- 172.21.15.90:0/1531274100 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5acc0035b0 con 0x7f5ae010f310 2024-04-14T15:27:51.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:51.603+0000 7f5ad6ffd700 1 -- 172.21.15.90:0/1531274100 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3571834495 0 0) 0x7f5ae01ac1f0 con 0x7f5ae0108890 2024-04-14T15:27:51.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:51.603+0000 7f5ad6ffd700 1 -- 172.21.15.90:0/1531274100 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ae01a9d90 con 0x7f5ae0108890 2024-04-14T15:27:51.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:51.603+0000 7f5ad6ffd700 1 -- 172.21.15.90:0/1531274100 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2765134407 0 0) 0x7f5ae01aafb0 con 0x7f5ae010b740 2024-04-14T15:27:51.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:51.603+0000 7f5ad6ffd700 1 -- 172.21.15.90:0/1531274100 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ae01ac1f0 con 0x7f5ae010b740 2024-04-14T15:27:51.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:51.603+0000 7f5ad6ffd700 1 -- 172.21.15.90:0/1531274100 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1912474590 0 0) 0x7f5acc0035b0 con 0x7f5ae010f310 2024-04-14T15:27:51.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:51.603+0000 7f5ad6ffd700 1 -- 172.21.15.90:0/1531274100 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5ae01aafb0 con 0x7f5ae010f310 2024-04-14T15:27:51.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:51.603+0000 7f5ad6ffd700 1 -- 172.21.15.90:0/1531274100 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2085981823 0 0) 0x7f5ae01a9d90 con 0x7f5ae0108890 2024-04-14T15:27:51.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:51.603+0000 7f5ad6ffd700 1 -- 172.21.15.90:0/1531274100 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5acc0035b0 con 0x7f5ae0108890 2024-04-14T15:27:51.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:51.603+0000 7f5ad6ffd700 1 -- 172.21.15.90:0/1531274100 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 585676323 0 0) 0x7f5ae01ac1f0 con 0x7f5ae010b740 2024-04-14T15:27:51.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:51.603+0000 7f5ad6ffd700 1 -- 172.21.15.90:0/1531274100 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5ae01a9d90 con 0x7f5ae010b740 2024-04-14T15:27:51.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:51.603+0000 7f5ad6ffd700 1 -- 172.21.15.90:0/1531274100 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f5ad8001c30 con 0x7f5ae010f310 2024-04-14T15:27:51.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:51.603+0000 7f5ad6ffd700 1 -- 172.21.15.90:0/1531274100 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f5ad0001430 con 0x7f5ae0108890 2024-04-14T15:27:51.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:51.603+0000 7f5ad6ffd700 1 -- 172.21.15.90:0/1531274100 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f5adc0017f0 con 0x7f5ae010b740 2024-04-14T15:27:51.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:51.603+0000 7f5ad6ffd700 1 -- 172.21.15.90:0/1531274100 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3773136572 0 0) 0x7f5ae01aafb0 con 0x7f5ae010f310 2024-04-14T15:27:51.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:51.603+0000 7f5ad6ffd700 1 -- 172.21.15.90:0/1531274100 >> v1:172.21.15.90:6790/0 conn(0x7f5ae0108890 legacy=0x7f5ae00767e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:51.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:51.604+0000 7f5ad6ffd700 1 -- 172.21.15.90:0/1531274100 >> v1:172.21.15.92:6789/0 conn(0x7f5ae010b740 legacy=0x7f5ae01a4bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:51.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:51.604+0000 7f5ad6ffd700 1 -- 172.21.15.90:0/1531274100 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ae01ad430 con 0x7f5ae010f310 2024-04-14T15:27:51.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:51.604+0000 7f5ae80cd700 1 -- 172.21.15.90:0/1531274100 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5ae01ab1e0 con 0x7f5ae010f310 2024-04-14T15:27:51.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:51.604+0000 7f5ae80cd700 10 client.38241.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:27:51.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:51.604+0000 7f5ae80cd700 1 -- 172.21.15.90:0/1531274100 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f5ae01ab6a0 con 0x7f5ae010f310 2024-04-14T15:27:51.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:51.604+0000 7f5ae80cd700 1 librados: init done 2024-04-14T15:27:51.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:51.604+0000 7f5ad6ffd700 1 -- 172.21.15.90:0/1531274100 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f5ad8003ea0 con 0x7f5ae010f310 2024-04-14T15:27:51.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:51.604+0000 7f5ad6ffd700 1 -- 172.21.15.90:0/1531274100 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f5ad8005f40 con 0x7f5ae010f310 2024-04-14T15:27:51.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:51.605+0000 7f5ae80cd700 1 -- 172.21.15.90:0/1531274100 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ac4005120 con 0x7f5ae010f310 2024-04-14T15:27:51.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:51.605+0000 7f5ad6ffd700 1 -- 172.21.15.90:0/1531274100 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f5ad801c950 con 0x7f5ae010f310 2024-04-14T15:27:51.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:51.606+0000 7f5ad6ffd700 1 -- 172.21.15.90:0/1531274100 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(890..890 src has 467..890) ==== 7550+0+0 (unknown 1342345585 0 0) 0x7f5ad808a9b0 con 0x7f5ae010f310 2024-04-14T15:27:51.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:51.606+0000 7f5ad6ffd700 10 client.38241.objecter ms_dispatch 0x7f5ae0001550 osd_map(890..890 src has 467..890) 2024-04-14T15:27:51.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:51.606+0000 7f5ad6ffd700 3 client.38241.objecter handle_osd_map got epochs [890,890] > 0 2024-04-14T15:27:51.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:51.606+0000 7f5ad6ffd700 3 client.38241.objecter handle_osd_map decoding full epoch 890 2024-04-14T15:27:51.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:51.606+0000 7f5ad6ffd700 20 client.38241.objecter dump_active .. 0 homeless 2024-04-14T15:27:51.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:51.606+0000 7f5ad6ffd700 10 client.38241.objecter ms_handle_connect 0x7f5acc0714b0 2024-04-14T15:27:51.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:51.611+0000 7f5ad6ffd700 1 -- 172.21.15.90:0/1531274100 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f5ad8057690 con 0x7f5ae010f310 2024-04-14T15:27:51.625 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:27:51.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:51.829+0000 7f5ae80cd700 1 -- 172.21.15.90:0/1531274100 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} v 0) -- 0x7f5ac4005c20 con 0x7f5ae010f310 2024-04-14T15:27:52.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:52.054+0000 7f5ad6ffd700 1 -- 172.21.15.90:0/1531274100 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]=0 set osd.2 primary-affinity to 1 (8655362) v891) ==== 133+0+0 (unknown 3162613738 0 0) 0x7f5ad805b340 con 0x7f5ae010f310 2024-04-14T15:27:52.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:52.058+0000 7f5ae80cd700 10 librados: watch_flush enter 2024-04-14T15:27:52.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:52.058+0000 7f5ae80cd700 10 librados: watch_flush exit 2024-04-14T15:27:52.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:52.058+0000 7f5ae80cd700 20 client.38241.objecter shutdown clearing up homeless session... 2024-04-14T15:27:52.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:52.058+0000 7f5ae80cd700 10 client.38241.objecter successfully canceled tick 2024-04-14T15:27:52.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:52.058+0000 7f5ae80cd700 1 -- 172.21.15.90:0/1531274100 >> v1:172.21.15.90:6800/420935427 conn(0x7f5acc0714b0 legacy=0x7f5acc073970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:52.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:52.058+0000 7f5ae80cd700 1 -- 172.21.15.90:0/1531274100 >> v1:172.21.15.90:6789/0 conn(0x7f5ae010f310 legacy=0x7f5ae01a8450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:52.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:52.059+0000 7f5ae80cd700 1 -- 172.21.15.90:0/1531274100 shutdown_connections 2024-04-14T15:27:52.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:52.059+0000 7f5ae80cd700 1 -- 172.21.15.90:0/1531274100 >> 172.21.15.90:0/1531274100 conn(0x7f5ae00fe380 msgr2=0x7f5ae0112830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:27:52.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:52.059+0000 7f5ae80cd700 1 -- 172.21.15.90:0/1531274100 shutdown_connections 2024-04-14T15:27:52.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:52.060+0000 7f5ae80cd700 1 -- 172.21.15.90:0/1531274100 wait complete. 2024-04-14T15:27:52.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:52.060+0000 7f5ae80cd700 1 librados: shutdown 2024-04-14T15:27:52.061 INFO:teuthology.orchestra.run.smithi090.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-04-14T15:27:52.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:52 smithi090 ceph-mon[41670]: 2.14 deep-scrub starts 2024-04-14T15:27:52.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:52 smithi090 ceph-mon[41670]: 2.14 deep-scrub ok 2024-04-14T15:27:52.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:52 smithi090 ceph-mon[41670]: 2.1c scrub starts 2024-04-14T15:27:52.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:52 smithi090 ceph-mon[41670]: 2.1c scrub ok 2024-04-14T15:27:52.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:52 smithi090 ceph-mon[41670]: osdmap e890: 8 total, 8 up, 8 in 2024-04-14T15:27:52.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:52 smithi090 ceph-mon[41670]: 4.1b scrub starts 2024-04-14T15:27:52.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:52 smithi090 ceph-mon[41670]: 4.1b scrub ok 2024-04-14T15:27:52.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:52 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1531274100' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-04-14T15:27:52.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:52 smithi090 ceph-mon[41670]: 5.7 scrub starts 2024-04-14T15:27:52.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:52 smithi090 ceph-mon[41670]: 5.7 scrub ok 2024-04-14T15:27:52.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:52 smithi090 ceph-mon[30399]: 2.14 deep-scrub starts 2024-04-14T15:27:52.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:52 smithi090 ceph-mon[30399]: 2.14 deep-scrub ok 2024-04-14T15:27:52.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:52 smithi090 ceph-mon[30399]: 2.1c scrub starts 2024-04-14T15:27:52.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:52 smithi090 ceph-mon[30399]: 2.1c scrub ok 2024-04-14T15:27:52.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:52 smithi090 ceph-mon[30399]: osdmap e890: 8 total, 8 up, 8 in 2024-04-14T15:27:52.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:52 smithi090 ceph-mon[30399]: 4.1b scrub starts 2024-04-14T15:27:52.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:52 smithi090 ceph-mon[30399]: 4.1b scrub ok 2024-04-14T15:27:52.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:52 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1531274100' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-04-14T15:27:52.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:52 smithi090 ceph-mon[30399]: 5.7 scrub starts 2024-04-14T15:27:52.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:52 smithi090 ceph-mon[30399]: 5.7 scrub ok 2024-04-14T15:27:52.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:52 smithi092 ceph-mon[33966]: 2.14 deep-scrub starts 2024-04-14T15:27:52.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:52 smithi092 ceph-mon[33966]: 2.14 deep-scrub ok 2024-04-14T15:27:52.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:52 smithi092 ceph-mon[33966]: 2.1c scrub starts 2024-04-14T15:27:52.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:52 smithi092 ceph-mon[33966]: 2.1c scrub ok 2024-04-14T15:27:52.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:52 smithi092 ceph-mon[33966]: osdmap e890: 8 total, 8 up, 8 in 2024-04-14T15:27:52.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:52 smithi092 ceph-mon[33966]: 4.1b scrub starts 2024-04-14T15:27:52.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:52 smithi092 ceph-mon[33966]: 4.1b scrub ok 2024-04-14T15:27:52.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:52 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1531274100' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-04-14T15:27:52.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:52 smithi092 ceph-mon[33966]: 5.7 scrub starts 2024-04-14T15:27:52.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:52 smithi092 ceph-mon[33966]: 5.7 scrub ok 2024-04-14T15:27:52.812 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd reweight 3 1 2024-04-14T15:27:53.052 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:27:53.085 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:53 smithi092 ceph-mon[33966]: pgmap v1338: 179 pgs: 178 active+clean, 1 active+remapped+backfill_toofull; 582 KiB data, 832 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:27:53.086 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:53 smithi092 ceph-mon[33966]: 1.0 scrub starts 2024-04-14T15:27:53.086 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:53 smithi092 ceph-mon[33966]: 1.0 scrub ok 2024-04-14T15:27:53.086 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:53 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1531274100' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-04-14T15:27:53.086 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:53 smithi092 ceph-mon[33966]: osdmap e891: 8 total, 8 up, 8 in 2024-04-14T15:27:53.321 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:53 smithi090 ceph-mon[41670]: pgmap v1338: 179 pgs: 178 active+clean, 1 active+remapped+backfill_toofull; 582 KiB data, 832 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:27:53.321 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:53 smithi090 ceph-mon[41670]: 1.0 scrub starts 2024-04-14T15:27:53.322 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:53 smithi090 ceph-mon[41670]: 1.0 scrub ok 2024-04-14T15:27:53.322 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:53 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1531274100' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-04-14T15:27:53.322 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:53 smithi090 ceph-mon[41670]: osdmap e891: 8 total, 8 up, 8 in 2024-04-14T15:27:53.322 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:53 smithi090 ceph-mon[30399]: pgmap v1338: 179 pgs: 178 active+clean, 1 active+remapped+backfill_toofull; 582 KiB data, 832 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:27:53.322 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:53 smithi090 ceph-mon[30399]: 1.0 scrub starts 2024-04-14T15:27:53.322 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:53 smithi090 ceph-mon[30399]: 1.0 scrub ok 2024-04-14T15:27:53.322 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:53 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1531274100' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-04-14T15:27:53.322 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:53 smithi090 ceph-mon[30399]: osdmap e891: 8 total, 8 up, 8 in 2024-04-14T15:27:53.382 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-04-14T15:27:53.383 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:27:54.034 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-14T15:27:54.176 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:27:53 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:27:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:27:54.176 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:54 smithi090 ceph-mon[41670]: 5.14 scrub starts 2024-04-14T15:27:54.176 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:54 smithi090 ceph-mon[41670]: 5.14 scrub ok 2024-04-14T15:27:54.176 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:54 smithi090 ceph-mon[41670]: osdmap e892: 8 total, 8 up, 8 in 2024-04-14T15:27:54.176 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:54 smithi090 ceph-mon[41670]: pgmap v1341: 178 pgs: 177 active+clean, 1 active+remapped+backfill_toofull; 582 KiB data, 832 MiB used, 714 GiB / 715 GiB avail 2024-04-14T15:27:54.177 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:54 smithi090 ceph-mon[30399]: 5.14 scrub starts 2024-04-14T15:27:54.177 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:54 smithi090 ceph-mon[30399]: 5.14 scrub ok 2024-04-14T15:27:54.177 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:54 smithi090 ceph-mon[30399]: osdmap e892: 8 total, 8 up, 8 in 2024-04-14T15:27:54.177 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:54 smithi090 ceph-mon[30399]: pgmap v1341: 178 pgs: 177 active+clean, 1 active+remapped+backfill_toofull; 582 KiB data, 832 MiB used, 714 GiB / 715 GiB avail 2024-04-14T15:27:54.282 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:27:54.337 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:54 smithi092 ceph-mon[33966]: 5.14 scrub starts 2024-04-14T15:27:54.337 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:54 smithi092 ceph-mon[33966]: 5.14 scrub ok 2024-04-14T15:27:54.337 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:54 smithi092 ceph-mon[33966]: osdmap e892: 8 total, 8 up, 8 in 2024-04-14T15:27:54.338 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:54 smithi092 ceph-mon[33966]: pgmap v1341: 178 pgs: 177 active+clean, 1 active+remapped+backfill_toofull; 582 KiB data, 832 MiB used, 714 GiB / 715 GiB avail 2024-04-14T15:27:54.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:54.666+0000 7f4dcc09d700 1 -- 172.21.15.90:0/29150756 >> v1:172.21.15.90:6789/0 conn(0x7f4dc410a740 legacy=0x7f4dc410cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:54.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:54.666+0000 7f4dcc09d700 1 -- 172.21.15.90:0/29150756 shutdown_connections 2024-04-14T15:27:54.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:54.666+0000 7f4dcc09d700 1 -- 172.21.15.90:0/29150756 >> 172.21.15.90:0/29150756 conn(0x7f4dc40fd290 msgr2=0x7f4dc40ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:27:54.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:54.666+0000 7f4dcc09d700 1 -- 172.21.15.90:0/29150756 shutdown_connections 2024-04-14T15:27:54.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:54.666+0000 7f4dcc09d700 1 -- 172.21.15.90:0/29150756 wait complete. 2024-04-14T15:27:54.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:54.666+0000 7f4dcc09d700 1 librados: starting msgr at 2024-04-14T15:27:54.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:54.666+0000 7f4dcc09d700 1 librados: starting objecter 2024-04-14T15:27:54.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:54.666+0000 7f4dcc09d700 1 Processor -- start 2024-04-14T15:27:54.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:54.666+0000 7f4dcc09d700 1 -- start start 2024-04-14T15:27:54.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:54.666+0000 7f4dcc09d700 1 librados: setting wanted keys 2024-04-14T15:27:54.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:54.666+0000 7f4dcc09d700 1 librados: calling monclient init 2024-04-14T15:27:54.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:54.667+0000 7f4dcc09d700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f4dc40ff2b0 con 0x7f4dc4107890 2024-04-14T15:27:54.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:54.667+0000 7f4dcc09d700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f4dc40ff4c0 con 0x7f4dc410e310 2024-04-14T15:27:54.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:54.667+0000 7f4dcc09d700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f4dc41ac330 con 0x7f4dc410a740 2024-04-14T15:27:54.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:54.667+0000 7f4dc9e39700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f4dc4107890 0x7f4dc40fd740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:40796/0 (socket says 172.21.15.90:40796) 2024-04-14T15:27:54.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:54.667+0000 7f4dc9e39700 1 -- 172.21.15.90:0/1195589221 learned_addr learned my addr 172.21.15.90:0/1195589221 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:27:54.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:54.667+0000 7f4dbaffd700 10 client.?.objecter ms_handle_connect 0x7f4dc4107890 2024-04-14T15:27:54.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:54.667+0000 7f4dbaffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:27:54.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:54.667+0000 7f4dbaffd700 10 client.?.objecter ms_handle_connect 0x7f4dc410a740 2024-04-14T15:27:54.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:54.667+0000 7f4dbaffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:27:54.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:54.667+0000 7f4dbaffd700 10 client.?.objecter ms_handle_connect 0x7f4dc410e310 2024-04-14T15:27:54.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:54.667+0000 7f4dbaffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:27:54.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:54.667+0000 7f4dbaffd700 1 -- 172.21.15.90:0/1195589221 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3712080755 0 0) 0x7f4dc40ff2b0 con 0x7f4dc4107890 2024-04-14T15:27:54.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:54.667+0000 7f4dbaffd700 1 -- 172.21.15.90:0/1195589221 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4db00035b0 con 0x7f4dc4107890 2024-04-14T15:27:54.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:54.667+0000 7f4dbaffd700 1 -- 172.21.15.90:0/1195589221 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 463675719 0 0) 0x7f4dc41ac330 con 0x7f4dc410a740 2024-04-14T15:27:54.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:54.667+0000 7f4dbaffd700 1 -- 172.21.15.90:0/1195589221 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4dc40ff2b0 con 0x7f4dc410a740 2024-04-14T15:27:54.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:54.667+0000 7f4dbaffd700 1 -- 172.21.15.90:0/1195589221 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 185317740 0 0) 0x7f4dc40ff4c0 con 0x7f4dc410e310 2024-04-14T15:27:54.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:54.667+0000 7f4dbaffd700 1 -- 172.21.15.90:0/1195589221 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4dc41ac330 con 0x7f4dc410e310 2024-04-14T15:27:54.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:54.667+0000 7f4dbaffd700 1 -- 172.21.15.90:0/1195589221 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 118548167 0 0) 0x7f4db00035b0 con 0x7f4dc4107890 2024-04-14T15:27:54.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:54.667+0000 7f4dbaffd700 1 -- 172.21.15.90:0/1195589221 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4dc40ff4c0 con 0x7f4dc4107890 2024-04-14T15:27:54.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:54.667+0000 7f4dbaffd700 1 -- 172.21.15.90:0/1195589221 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f4dc0001c90 con 0x7f4dc4107890 2024-04-14T15:27:54.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:54.667+0000 7f4dbaffd700 1 -- 172.21.15.90:0/1195589221 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 21085634 0 0) 0x7f4dc40ff2b0 con 0x7f4dc410a740 2024-04-14T15:27:54.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:54.667+0000 7f4dbaffd700 1 -- 172.21.15.90:0/1195589221 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4db00035b0 con 0x7f4dc410a740 2024-04-14T15:27:54.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:54.667+0000 7f4dbaffd700 1 -- 172.21.15.90:0/1195589221 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 205919508 0 0) 0x7f4dc41ac330 con 0x7f4dc410e310 2024-04-14T15:27:54.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:54.667+0000 7f4dbaffd700 1 -- 172.21.15.90:0/1195589221 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4dc40ff2b0 con 0x7f4dc410e310 2024-04-14T15:27:54.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:54.667+0000 7f4dbaffd700 1 -- 172.21.15.90:0/1195589221 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f4db4001370 con 0x7f4dc410a740 2024-04-14T15:27:54.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:54.667+0000 7f4dbaffd700 1 -- 172.21.15.90:0/1195589221 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f4dbc0017b0 con 0x7f4dc410e310 2024-04-14T15:27:54.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:54.668+0000 7f4dbaffd700 1 -- 172.21.15.90:0/1195589221 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 154916718 0 0) 0x7f4dc40ff4c0 con 0x7f4dc4107890 2024-04-14T15:27:54.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:54.668+0000 7f4dbaffd700 1 -- 172.21.15.90:0/1195589221 >> v1:172.21.15.90:6790/0 conn(0x7f4dc410a740 legacy=0x7f4dc40fde50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:54.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:54.668+0000 7f4dbaffd700 1 -- 172.21.15.90:0/1195589221 >> v1:172.21.15.92:6789/0 conn(0x7f4dc410e310 legacy=0x7f4dc41a8bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:54.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:54.668+0000 7f4dbaffd700 1 -- 172.21.15.90:0/1195589221 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dc41ad510 con 0x7f4dc4107890 2024-04-14T15:27:54.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:54.668+0000 7f4dcc09d700 1 -- 172.21.15.90:0/1195589221 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4dc41ac500 con 0x7f4dc4107890 2024-04-14T15:27:54.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:54.668+0000 7f4dcc09d700 10 client.38247.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:27:54.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:54.668+0000 7f4dcc09d700 1 -- 172.21.15.90:0/1195589221 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f4dc41ac9c0 con 0x7f4dc4107890 2024-04-14T15:27:54.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:54.668+0000 7f4dcc09d700 1 librados: init done 2024-04-14T15:27:54.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:54.668+0000 7f4dbaffd700 1 -- 172.21.15.90:0/1195589221 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f4dc0002060 con 0x7f4dc4107890 2024-04-14T15:27:54.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:54.668+0000 7f4dbaffd700 1 -- 172.21.15.90:0/1195589221 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f4dc0006170 con 0x7f4dc4107890 2024-04-14T15:27:54.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:54.669+0000 7f4dcc09d700 1 -- 172.21.15.90:0/1195589221 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4da8005150 con 0x7f4dc4107890 2024-04-14T15:27:54.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:54.669+0000 7f4dbaffd700 1 -- 172.21.15.90:0/1195589221 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f4dc001cb80 con 0x7f4dc4107890 2024-04-14T15:27:54.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:54.670+0000 7f4dbaffd700 1 -- 172.21.15.90:0/1195589221 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(892..892 src has 467..892) ==== 7550+0+0 (unknown 3713702793 0 0) 0x7f4dc008ab30 con 0x7f4dc4107890 2024-04-14T15:27:54.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:54.670+0000 7f4dbaffd700 10 client.38247.objecter ms_dispatch 0x7f4dc4001550 osd_map(892..892 src has 467..892) 2024-04-14T15:27:54.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:54.670+0000 7f4dbaffd700 3 client.38247.objecter handle_osd_map got epochs [892,892] > 0 2024-04-14T15:27:54.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:54.670+0000 7f4dbaffd700 3 client.38247.objecter handle_osd_map decoding full epoch 892 2024-04-14T15:27:54.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:54.670+0000 7f4dbaffd700 20 client.38247.objecter dump_active .. 0 homeless 2024-04-14T15:27:54.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:54.670+0000 7f4dbaffd700 10 client.38247.objecter ms_handle_connect 0x7f4db00713a0 2024-04-14T15:27:54.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:54.674+0000 7f4dbaffd700 1 -- 172.21.15.90:0/1195589221 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f4dc00577f0 con 0x7f4dc4107890 2024-04-14T15:27:54.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:54.885+0000 7f4dcc09d700 1 -- 172.21.15.90:0/1195589221 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd reweight", "id": 3, "weight": 1.0} v 0) -- 0x7f4da8005440 con 0x7f4dc4107890 2024-04-14T15:27:55.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:55.071+0000 7f4dbaffd700 1 -- 172.21.15.90:0/1195589221 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 3, "weight": 1.0}]=0 reweighted osd.3 to 1 (10000) v893) ==== 113+0+0 (unknown 1033869399 0 0) 0x7f4dc005b4a0 con 0x7f4dc4107890 2024-04-14T15:27:55.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:55.073+0000 7f4dcc09d700 10 librados: watch_flush enter 2024-04-14T15:27:55.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:55.073+0000 7f4dcc09d700 10 librados: watch_flush exit 2024-04-14T15:27:55.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:55.073+0000 7f4dcc09d700 20 client.38247.objecter shutdown clearing up homeless session... 2024-04-14T15:27:55.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:55.073+0000 7f4dcc09d700 10 client.38247.objecter successfully canceled tick 2024-04-14T15:27:55.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:55.073+0000 7f4dcc09d700 1 -- 172.21.15.90:0/1195589221 >> v1:172.21.15.90:6800/420935427 conn(0x7f4db00713a0 legacy=0x7f4db0073860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:55.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:55.073+0000 7f4dcc09d700 1 -- 172.21.15.90:0/1195589221 >> v1:172.21.15.90:6789/0 conn(0x7f4dc4107890 legacy=0x7f4dc40fd740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:55.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:55.074+0000 7f4dcc09d700 1 -- 172.21.15.90:0/1195589221 shutdown_connections 2024-04-14T15:27:55.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:55.074+0000 7f4dcc09d700 1 -- 172.21.15.90:0/1195589221 >> 172.21.15.90:0/1195589221 conn(0x7f4dc40fd290 msgr2=0x7f4dc4101780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:27:55.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:55.074+0000 7f4dcc09d700 1 -- 172.21.15.90:0/1195589221 shutdown_connections 2024-04-14T15:27:55.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:55.074+0000 7f4dcc09d700 1 -- 172.21.15.90:0/1195589221 wait complete. 2024-04-14T15:27:55.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:55.074+0000 7f4dcc09d700 1 librados: shutdown 2024-04-14T15:27:55.076 INFO:teuthology.orchestra.run.smithi090.stderr:reweighted osd.3 to 1 (10000) 2024-04-14T15:27:55.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:55 smithi090 ceph-mon[41670]: 2.d scrub starts 2024-04-14T15:27:55.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:55 smithi090 ceph-mon[41670]: 2.d scrub ok 2024-04-14T15:27:55.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:55 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:27:55.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:55 smithi090 ceph-mon[41670]: 11.1f scrub starts 2024-04-14T15:27:55.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:55 smithi090 ceph-mon[41670]: 11.1f scrub ok 2024-04-14T15:27:55.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:55 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1195589221' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-14T15:27:55.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:55 smithi090 ceph-mon[30399]: 2.d scrub starts 2024-04-14T15:27:55.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:55 smithi090 ceph-mon[30399]: 2.d scrub ok 2024-04-14T15:27:55.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:55 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:27:55.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:55 smithi090 ceph-mon[30399]: 11.1f scrub starts 2024-04-14T15:27:55.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:55 smithi090 ceph-mon[30399]: 11.1f scrub ok 2024-04-14T15:27:55.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:55 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1195589221' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-14T15:27:55.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:55 smithi092 ceph-mon[33966]: 2.d scrub starts 2024-04-14T15:27:55.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:55 smithi092 ceph-mon[33966]: 2.d scrub ok 2024-04-14T15:27:55.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:55 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:27:55.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:55 smithi092 ceph-mon[33966]: 11.1f scrub starts 2024-04-14T15:27:55.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:55 smithi092 ceph-mon[33966]: 11.1f scrub ok 2024-04-14T15:27:55.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:55 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1195589221' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-14T15:27:55.702 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:27:55 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:27:55.846 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd primary-affinity 3 1 2024-04-14T15:27:55.959 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-04-14T15:27:55.959 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:27:56.089 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:27:56.340 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:56 smithi090 ceph-mon[30399]: 1.2 scrub starts 2024-04-14T15:27:56.341 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:56 smithi090 ceph-mon[30399]: 1.2 scrub ok 2024-04-14T15:27:56.341 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:56 smithi090 ceph-mon[30399]: 5.1 scrub starts 2024-04-14T15:27:56.341 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:56 smithi090 ceph-mon[30399]: 5.1 scrub ok 2024-04-14T15:27:56.341 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:56 smithi090 ceph-mon[30399]: 2.9 scrub starts 2024-04-14T15:27:56.341 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:56 smithi090 ceph-mon[30399]: 2.9 scrub ok 2024-04-14T15:27:56.341 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:56 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1195589221' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-14T15:27:56.341 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:56 smithi090 ceph-mon[30399]: osdmap e893: 8 total, 8 up, 8 in 2024-04-14T15:27:56.341 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:56 smithi090 ceph-mon[30399]: 3.1 scrub starts 2024-04-14T15:27:56.341 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:56 smithi090 ceph-mon[30399]: 3.1 scrub ok 2024-04-14T15:27:56.341 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:56 smithi090 ceph-mon[30399]: pgmap v1343: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 582 KiB data, 833 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:27:56.341 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:56 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:27:56.341 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:56 smithi090 ceph-mon[30399]: 4.8 scrub starts 2024-04-14T15:27:56.341 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:56 smithi090 ceph-mon[30399]: 4.8 scrub ok 2024-04-14T15:27:56.341 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:56 smithi090 ceph-mon[30399]: osdmap e894: 8 total, 8 up, 8 in 2024-04-14T15:27:56.342 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:56 smithi090 ceph-mon[41670]: 1.2 scrub starts 2024-04-14T15:27:56.342 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:56 smithi090 ceph-mon[41670]: 1.2 scrub ok 2024-04-14T15:27:56.342 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:56 smithi090 ceph-mon[41670]: 5.1 scrub starts 2024-04-14T15:27:56.342 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:56 smithi090 ceph-mon[41670]: 5.1 scrub ok 2024-04-14T15:27:56.342 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:56 smithi090 ceph-mon[41670]: 2.9 scrub starts 2024-04-14T15:27:56.342 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:56 smithi090 ceph-mon[41670]: 2.9 scrub ok 2024-04-14T15:27:56.342 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:56 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1195589221' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-14T15:27:56.342 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:56 smithi090 ceph-mon[41670]: osdmap e893: 8 total, 8 up, 8 in 2024-04-14T15:27:56.342 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:56 smithi090 ceph-mon[41670]: 3.1 scrub starts 2024-04-14T15:27:56.342 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:56 smithi090 ceph-mon[41670]: 3.1 scrub ok 2024-04-14T15:27:56.342 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:56 smithi090 ceph-mon[41670]: pgmap v1343: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 582 KiB data, 833 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:27:56.343 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:56 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:27:56.343 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:56 smithi090 ceph-mon[41670]: 4.8 scrub starts 2024-04-14T15:27:56.343 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:56 smithi090 ceph-mon[41670]: 4.8 scrub ok 2024-04-14T15:27:56.343 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:56 smithi090 ceph-mon[41670]: osdmap e894: 8 total, 8 up, 8 in 2024-04-14T15:27:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:56 smithi092 ceph-mon[33966]: 1.2 scrub starts 2024-04-14T15:27:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:56 smithi092 ceph-mon[33966]: 1.2 scrub ok 2024-04-14T15:27:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:56 smithi092 ceph-mon[33966]: 5.1 scrub starts 2024-04-14T15:27:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:56 smithi092 ceph-mon[33966]: 5.1 scrub ok 2024-04-14T15:27:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:56 smithi092 ceph-mon[33966]: 2.9 scrub starts 2024-04-14T15:27:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:56 smithi092 ceph-mon[33966]: 2.9 scrub ok 2024-04-14T15:27:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:56 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1195589221' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-14T15:27:56.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:56 smithi092 ceph-mon[33966]: osdmap e893: 8 total, 8 up, 8 in 2024-04-14T15:27:56.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:56 smithi092 ceph-mon[33966]: 3.1 scrub starts 2024-04-14T15:27:56.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:56 smithi092 ceph-mon[33966]: 3.1 scrub ok 2024-04-14T15:27:56.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:56 smithi092 ceph-mon[33966]: pgmap v1343: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 582 KiB data, 833 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:27:56.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:56 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:27:56.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:56 smithi092 ceph-mon[33966]: 4.8 scrub starts 2024-04-14T15:27:56.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:56 smithi092 ceph-mon[33966]: 4.8 scrub ok 2024-04-14T15:27:56.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:56 smithi092 ceph-mon[33966]: osdmap e894: 8 total, 8 up, 8 in 2024-04-14T15:27:56.770 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-14T15:27:57.012 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:27:57.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:57 smithi090 ceph-mon[30399]: 4.0 scrub starts 2024-04-14T15:27:57.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:57 smithi090 ceph-mon[30399]: 4.0 scrub ok 2024-04-14T15:27:57.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:57 smithi090 ceph-mon[30399]: 11.6 scrub starts 2024-04-14T15:27:57.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:57 smithi090 ceph-mon[30399]: 11.6 scrub ok 2024-04-14T15:27:57.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:57 smithi090 ceph-mon[30399]: 4.1c scrub starts 2024-04-14T15:27:57.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:57 smithi090 ceph-mon[30399]: 4.1c scrub ok 2024-04-14T15:27:57.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:57 smithi090 ceph-mon[41670]: 4.0 scrub starts 2024-04-14T15:27:57.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:57 smithi090 ceph-mon[41670]: 4.0 scrub ok 2024-04-14T15:27:57.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:57 smithi090 ceph-mon[41670]: 11.6 scrub starts 2024-04-14T15:27:57.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:57 smithi090 ceph-mon[41670]: 11.6 scrub ok 2024-04-14T15:27:57.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:57 smithi090 ceph-mon[41670]: 4.1c scrub starts 2024-04-14T15:27:57.193 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:57 smithi090 ceph-mon[41670]: 4.1c scrub ok 2024-04-14T15:27:57.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:57 smithi092 ceph-mon[33966]: 4.0 scrub starts 2024-04-14T15:27:57.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:57 smithi092 ceph-mon[33966]: 4.0 scrub ok 2024-04-14T15:27:57.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:57 smithi092 ceph-mon[33966]: 11.6 scrub starts 2024-04-14T15:27:57.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:57 smithi092 ceph-mon[33966]: 11.6 scrub ok 2024-04-14T15:27:57.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:57 smithi092 ceph-mon[33966]: 4.1c scrub starts 2024-04-14T15:27:57.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:57 smithi092 ceph-mon[33966]: 4.1c scrub ok 2024-04-14T15:27:57.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:57.862+0000 7fd7619a5700 1 -- 172.21.15.90:0/834079950 >> v1:172.21.15.90:6789/0 conn(0x7fd75c10f430 legacy=0x7fd75c1118f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:57.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:57.862+0000 7fd7619a5700 1 -- 172.21.15.90:0/834079950 shutdown_connections 2024-04-14T15:27:57.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:57.862+0000 7fd7619a5700 1 -- 172.21.15.90:0/834079950 >> 172.21.15.90:0/834079950 conn(0x7fd75c076ab0 msgr2=0x7fd75c076ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:27:57.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:57.862+0000 7fd7619a5700 1 -- 172.21.15.90:0/834079950 shutdown_connections 2024-04-14T15:27:57.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:57.862+0000 7fd7619a5700 1 -- 172.21.15.90:0/834079950 wait complete. 2024-04-14T15:27:57.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:57.863+0000 7fd7619a5700 1 librados: starting msgr at 2024-04-14T15:27:57.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:57.863+0000 7fd7619a5700 1 librados: starting objecter 2024-04-14T15:27:57.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:57.863+0000 7fd7619a5700 1 Processor -- start 2024-04-14T15:27:57.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:57.863+0000 7fd7619a5700 1 -- start start 2024-04-14T15:27:57.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:57.863+0000 7fd7619a5700 1 librados: setting wanted keys 2024-04-14T15:27:57.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:57.863+0000 7fd7619a5700 1 librados: calling monclient init 2024-04-14T15:27:57.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:57.863+0000 7fd7619a5700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd75c103e10 con 0x7fd75c10b830 2024-04-14T15:27:57.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:57.863+0000 7fd7619a5700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd75c103fe0 con 0x7fd75c108a30 2024-04-14T15:27:57.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:57.863+0000 7fd7619a5700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd75c1ac420 con 0x7fd75c10f430 2024-04-14T15:27:57.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:57.863+0000 7fd75a7fc700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fd75c10b830 0x7fd75c102980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:40810/0 (socket says 172.21.15.90:40810) 2024-04-14T15:27:57.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:57.863+0000 7fd75a7fc700 1 -- 172.21.15.90:0/2243065274 learned_addr learned my addr 172.21.15.90:0/2243065274 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:27:57.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:57.864+0000 7fd7609a3700 10 client.?.objecter ms_handle_connect 0x7fd75c10b830 2024-04-14T15:27:57.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:57.864+0000 7fd7609a3700 10 client.?.objecter resend_mon_ops 2024-04-14T15:27:57.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:57.864+0000 7fd7609a3700 10 client.?.objecter ms_handle_connect 0x7fd75c10f430 2024-04-14T15:27:57.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:57.864+0000 7fd7609a3700 10 client.?.objecter resend_mon_ops 2024-04-14T15:27:57.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:57.864+0000 7fd7609a3700 10 client.?.objecter ms_handle_connect 0x7fd75c108a30 2024-04-14T15:27:57.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:57.864+0000 7fd7609a3700 10 client.?.objecter resend_mon_ops 2024-04-14T15:27:57.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:57.864+0000 7fd7609a3700 1 -- 172.21.15.90:0/2243065274 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1093895307 0 0) 0x7fd75c103fe0 con 0x7fd75c108a30 2024-04-14T15:27:57.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:57.864+0000 7fd7609a3700 1 -- 172.21.15.90:0/2243065274 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd7440035b0 con 0x7fd75c108a30 2024-04-14T15:27:57.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:57.864+0000 7fd7609a3700 1 -- 172.21.15.90:0/2243065274 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1416531759 0 0) 0x7fd75c103e10 con 0x7fd75c10b830 2024-04-14T15:27:57.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:57.864+0000 7fd7609a3700 1 -- 172.21.15.90:0/2243065274 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd75c103fe0 con 0x7fd75c10b830 2024-04-14T15:27:57.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:57.864+0000 7fd7609a3700 1 -- 172.21.15.90:0/2243065274 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 875589843 0 0) 0x7fd75c1ac420 con 0x7fd75c10f430 2024-04-14T15:27:57.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:57.864+0000 7fd7609a3700 1 -- 172.21.15.90:0/2243065274 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd75c103e10 con 0x7fd75c10f430 2024-04-14T15:27:57.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:57.864+0000 7fd7609a3700 1 -- 172.21.15.90:0/2243065274 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4265947527 0 0) 0x7fd7440035b0 con 0x7fd75c108a30 2024-04-14T15:27:57.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:57.864+0000 7fd7609a3700 1 -- 172.21.15.90:0/2243065274 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd75c1ac420 con 0x7fd75c108a30 2024-04-14T15:27:57.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:57.864+0000 7fd7609a3700 1 -- 172.21.15.90:0/2243065274 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fd74c0018b0 con 0x7fd75c108a30 2024-04-14T15:27:57.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:57.864+0000 7fd7609a3700 1 -- 172.21.15.90:0/2243065274 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2646098221 0 0) 0x7fd75c103fe0 con 0x7fd75c10b830 2024-04-14T15:27:57.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:57.864+0000 7fd7609a3700 1 -- 172.21.15.90:0/2243065274 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd7440035b0 con 0x7fd75c10b830 2024-04-14T15:27:57.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:57.864+0000 7fd7609a3700 1 -- 172.21.15.90:0/2243065274 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4152396179 0 0) 0x7fd75c103e10 con 0x7fd75c10f430 2024-04-14T15:27:57.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:57.864+0000 7fd7609a3700 1 -- 172.21.15.90:0/2243065274 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd75c103fe0 con 0x7fd75c10f430 2024-04-14T15:27:57.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:57.864+0000 7fd7609a3700 1 -- 172.21.15.90:0/2243065274 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fd750001830 con 0x7fd75c10b830 2024-04-14T15:27:57.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:57.864+0000 7fd7609a3700 1 -- 172.21.15.90:0/2243065274 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fd7540017b0 con 0x7fd75c10f430 2024-04-14T15:27:57.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:57.864+0000 7fd7609a3700 1 -- 172.21.15.90:0/2243065274 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2779598322 0 0) 0x7fd75c1ac420 con 0x7fd75c108a30 2024-04-14T15:27:57.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:57.864+0000 7fd7609a3700 1 -- 172.21.15.90:0/2243065274 >> v1:172.21.15.90:6790/0 conn(0x7fd75c10f430 legacy=0x7fd75c1a8ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:57.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:57.864+0000 7fd7609a3700 1 -- 172.21.15.90:0/2243065274 >> v1:172.21.15.90:6789/0 conn(0x7fd75c10b830 legacy=0x7fd75c102980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:57.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:57.864+0000 7fd7609a3700 1 -- 172.21.15.90:0/2243065274 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd75c1ad600 con 0x7fd75c108a30 2024-04-14T15:27:57.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:57.864+0000 7fd7619a5700 1 -- 172.21.15.90:0/2243065274 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd75c1ac5f0 con 0x7fd75c108a30 2024-04-14T15:27:57.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:57.864+0000 7fd7619a5700 10 client.50074.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:27:57.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:57.864+0000 7fd7619a5700 1 -- 172.21.15.90:0/2243065274 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd75c1acab0 con 0x7fd75c108a30 2024-04-14T15:27:57.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:57.864+0000 7fd7619a5700 1 librados: init done 2024-04-14T15:27:57.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:57.864+0000 7fd7609a3700 1 -- 172.21.15.90:0/2243065274 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fd74c003b40 con 0x7fd75c108a30 2024-04-14T15:27:57.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:57.864+0000 7fd7609a3700 1 -- 172.21.15.90:0/2243065274 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fd74c004a00 con 0x7fd75c108a30 2024-04-14T15:27:57.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:57.866+0000 7fd7619a5700 1 -- 172.21.15.90:0/2243065274 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd748005120 con 0x7fd75c108a30 2024-04-14T15:27:57.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:57.866+0000 7fd7609a3700 1 -- 172.21.15.90:0/2243065274 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fd74c0036a0 con 0x7fd75c108a30 2024-04-14T15:27:57.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:57.866+0000 7fd7609a3700 1 -- 172.21.15.90:0/2243065274 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(894..894 src has 467..894) ==== 7550+0+0 (unknown 2792186401 0 0) 0x7fd74c089460 con 0x7fd75c108a30 2024-04-14T15:27:57.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:57.866+0000 7fd7609a3700 10 client.50074.objecter ms_dispatch 0x7fd75c001550 osd_map(894..894 src has 467..894) 2024-04-14T15:27:57.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:57.866+0000 7fd7609a3700 3 client.50074.objecter handle_osd_map got epochs [894,894] > 0 2024-04-14T15:27:57.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:57.866+0000 7fd7609a3700 3 client.50074.objecter handle_osd_map decoding full epoch 894 2024-04-14T15:27:57.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:57.866+0000 7fd7609a3700 20 client.50074.objecter dump_active .. 0 homeless 2024-04-14T15:27:57.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:57.866+0000 7fd7609a3700 10 client.50074.objecter ms_handle_connect 0x7fd744071600 2024-04-14T15:27:57.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:57.871+0000 7fd7609a3700 1 -- 172.21.15.90:0/2243065274 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fd74c0560e0 con 0x7fd75c108a30 2024-04-14T15:27:58.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:58.086+0000 7fd7619a5700 1 -- 172.21.15.90:0/2243065274 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} v 0) -- 0x7fd748005c20 con 0x7fd75c108a30 2024-04-14T15:27:58.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:58.094+0000 7fd7609a3700 1 -- 172.21.15.90:0/2243065274 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]=0 set osd.3 primary-affinity to 1 (8655362) v895) ==== 133+0+0 (unknown 3006326438 0 0) 0x7fd74c059d90 con 0x7fd75c108a30 2024-04-14T15:27:58.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:58.097+0000 7fd7619a5700 10 librados: watch_flush enter 2024-04-14T15:27:58.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:58.097+0000 7fd7619a5700 10 librados: watch_flush exit 2024-04-14T15:27:58.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:58.097+0000 7fd7619a5700 20 client.50074.objecter shutdown clearing up homeless session... 2024-04-14T15:27:58.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:58.097+0000 7fd7619a5700 10 client.50074.objecter successfully canceled tick 2024-04-14T15:27:58.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:58.097+0000 7fd7619a5700 1 -- 172.21.15.90:0/2243065274 >> v1:172.21.15.90:6800/420935427 conn(0x7fd744071600 legacy=0x7fd744073ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:58.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:58.097+0000 7fd7619a5700 1 -- 172.21.15.90:0/2243065274 >> v1:172.21.15.92:6789/0 conn(0x7fd75c108a30 legacy=0x7fd75c102270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:27:58.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:58.097+0000 7fd7619a5700 1 -- 172.21.15.90:0/2243065274 shutdown_connections 2024-04-14T15:27:58.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:58.097+0000 7fd7619a5700 1 -- 172.21.15.90:0/2243065274 >> 172.21.15.90:0/2243065274 conn(0x7fd75c076ab0 msgr2=0x7fd75c1118b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:27:58.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:58.097+0000 7fd7619a5700 1 -- 172.21.15.90:0/2243065274 shutdown_connections 2024-04-14T15:27:58.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:58.097+0000 7fd7619a5700 1 -- 172.21.15.90:0/2243065274 wait complete. 2024-04-14T15:27:58.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:27:58.097+0000 7fd7619a5700 1 librados: shutdown 2024-04-14T15:27:58.099 INFO:teuthology.orchestra.run.smithi090.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-04-14T15:27:58.262 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:58 smithi092 ceph-mon[33966]: 11.14 scrub starts 2024-04-14T15:27:58.262 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:58 smithi092 ceph-mon[33966]: 11.14 scrub ok 2024-04-14T15:27:58.263 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:58 smithi092 ceph-mon[33966]: 5.0 scrub starts 2024-04-14T15:27:58.263 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:58 smithi092 ceph-mon[33966]: 5.0 scrub ok 2024-04-14T15:27:58.263 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:58 smithi092 ceph-mon[33966]: 6.1e scrub starts 2024-04-14T15:27:58.263 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:58 smithi092 ceph-mon[33966]: 6.1e scrub ok 2024-04-14T15:27:58.263 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:58 smithi092 ceph-mon[33966]: pgmap v1345: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 582 KiB data, 833 MiB used, 714 GiB / 715 GiB avail; 989 B/s rd, 0 op/s 2024-04-14T15:27:58.263 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:58 smithi092 ceph-mon[33966]: 6.19 scrub starts 2024-04-14T15:27:58.263 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:58 smithi092 ceph-mon[33966]: 6.19 scrub ok 2024-04-14T15:27:58.263 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:58 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2243065274' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-04-14T15:27:58.263 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:58 smithi092 ceph-mon[33966]: from='client.50074 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-04-14T15:27:58.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:58 smithi090 ceph-mon[41670]: 11.14 scrub starts 2024-04-14T15:27:58.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:58 smithi090 ceph-mon[41670]: 11.14 scrub ok 2024-04-14T15:27:58.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:58 smithi090 ceph-mon[41670]: 5.0 scrub starts 2024-04-14T15:27:58.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:58 smithi090 ceph-mon[41670]: 5.0 scrub ok 2024-04-14T15:27:58.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:58 smithi090 ceph-mon[41670]: 6.1e scrub starts 2024-04-14T15:27:58.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:58 smithi090 ceph-mon[41670]: 6.1e scrub ok 2024-04-14T15:27:58.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:58 smithi090 ceph-mon[41670]: pgmap v1345: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 582 KiB data, 833 MiB used, 714 GiB / 715 GiB avail; 989 B/s rd, 0 op/s 2024-04-14T15:27:58.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:58 smithi090 ceph-mon[41670]: 6.19 scrub starts 2024-04-14T15:27:58.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:58 smithi090 ceph-mon[41670]: 6.19 scrub ok 2024-04-14T15:27:58.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:58 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2243065274' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-04-14T15:27:58.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:58 smithi090 ceph-mon[41670]: from='client.50074 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-04-14T15:27:58.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:58 smithi090 ceph-mon[30399]: 11.14 scrub starts 2024-04-14T15:27:58.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:58 smithi090 ceph-mon[30399]: 11.14 scrub ok 2024-04-14T15:27:58.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:58 smithi090 ceph-mon[30399]: 5.0 scrub starts 2024-04-14T15:27:58.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:58 smithi090 ceph-mon[30399]: 5.0 scrub ok 2024-04-14T15:27:58.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:58 smithi090 ceph-mon[30399]: 6.1e scrub starts 2024-04-14T15:27:58.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:58 smithi090 ceph-mon[30399]: 6.1e scrub ok 2024-04-14T15:27:58.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:58 smithi090 ceph-mon[30399]: pgmap v1345: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 582 KiB data, 833 MiB used, 714 GiB / 715 GiB avail; 989 B/s rd, 0 op/s 2024-04-14T15:27:58.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:58 smithi090 ceph-mon[30399]: 6.19 scrub starts 2024-04-14T15:27:58.445 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:58 smithi090 ceph-mon[30399]: 6.19 scrub ok 2024-04-14T15:27:58.445 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:58 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2243065274' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-04-14T15:27:58.445 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:58 smithi090 ceph-mon[30399]: from='client.50074 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-04-14T15:27:58.671 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-04-14T15:27:58.671 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:27:58.784 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd reweight 4 1 2024-04-14T15:27:59.027 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:27:59.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:59 smithi092 ceph-mon[33966]: 11.1d scrub starts 2024-04-14T15:27:59.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:59 smithi092 ceph-mon[33966]: 11.1d scrub ok 2024-04-14T15:27:59.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:59 smithi092 ceph-mon[33966]: 1.c scrub starts 2024-04-14T15:27:59.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:59 smithi092 ceph-mon[33966]: 1.c scrub ok 2024-04-14T15:27:59.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:59 smithi092 ceph-mon[33966]: 4.7 scrub starts 2024-04-14T15:27:59.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:59 smithi092 ceph-mon[33966]: 4.7 scrub ok 2024-04-14T15:27:59.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:59 smithi092 ceph-mon[33966]: from='client.50074 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-04-14T15:27:59.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:59 smithi092 ceph-mon[33966]: osdmap e895: 8 total, 8 up, 8 in 2024-04-14T15:27:59.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:59 smithi092 ceph-mon[33966]: 4.18 scrub starts 2024-04-14T15:27:59.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:59 smithi092 ceph-mon[33966]: 4.18 scrub ok 2024-04-14T15:27:59.374 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:59 smithi092 ceph-mon[33966]: 6.10 scrub starts 2024-04-14T15:27:59.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:59 smithi092 ceph-mon[33966]: 6.10 scrub ok 2024-04-14T15:27:59.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:59 smithi092 ceph-mon[33966]: 11.2 scrub starts 2024-04-14T15:27:59.375 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:27:59 smithi092 ceph-mon[33966]: 11.2 scrub ok 2024-04-14T15:27:59.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:59 smithi090 ceph-mon[30399]: 11.1d scrub starts 2024-04-14T15:27:59.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:59 smithi090 ceph-mon[30399]: 11.1d scrub ok 2024-04-14T15:27:59.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:59 smithi090 ceph-mon[30399]: 1.c scrub starts 2024-04-14T15:27:59.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:59 smithi090 ceph-mon[30399]: 1.c scrub ok 2024-04-14T15:27:59.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:59 smithi090 ceph-mon[30399]: 4.7 scrub starts 2024-04-14T15:27:59.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:59 smithi090 ceph-mon[30399]: 4.7 scrub ok 2024-04-14T15:27:59.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:59 smithi090 ceph-mon[30399]: from='client.50074 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-04-14T15:27:59.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:59 smithi090 ceph-mon[30399]: osdmap e895: 8 total, 8 up, 8 in 2024-04-14T15:27:59.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:59 smithi090 ceph-mon[30399]: 4.18 scrub starts 2024-04-14T15:27:59.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:59 smithi090 ceph-mon[30399]: 4.18 scrub ok 2024-04-14T15:27:59.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:59 smithi090 ceph-mon[30399]: 6.10 scrub starts 2024-04-14T15:27:59.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:59 smithi090 ceph-mon[30399]: 6.10 scrub ok 2024-04-14T15:27:59.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:59 smithi090 ceph-mon[30399]: 11.2 scrub starts 2024-04-14T15:27:59.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:27:59 smithi090 ceph-mon[30399]: 11.2 scrub ok 2024-04-14T15:27:59.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:59 smithi090 ceph-mon[41670]: 11.1d scrub starts 2024-04-14T15:27:59.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:59 smithi090 ceph-mon[41670]: 11.1d scrub ok 2024-04-14T15:27:59.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:59 smithi090 ceph-mon[41670]: 1.c scrub starts 2024-04-14T15:27:59.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:59 smithi090 ceph-mon[41670]: 1.c scrub ok 2024-04-14T15:27:59.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:59 smithi090 ceph-mon[41670]: 4.7 scrub starts 2024-04-14T15:27:59.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:59 smithi090 ceph-mon[41670]: 4.7 scrub ok 2024-04-14T15:27:59.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:59 smithi090 ceph-mon[41670]: from='client.50074 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-04-14T15:27:59.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:59 smithi090 ceph-mon[41670]: osdmap e895: 8 total, 8 up, 8 in 2024-04-14T15:27:59.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:59 smithi090 ceph-mon[41670]: 4.18 scrub starts 2024-04-14T15:27:59.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:59 smithi090 ceph-mon[41670]: 4.18 scrub ok 2024-04-14T15:27:59.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:59 smithi090 ceph-mon[41670]: 6.10 scrub starts 2024-04-14T15:27:59.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:59 smithi090 ceph-mon[41670]: 6.10 scrub ok 2024-04-14T15:27:59.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:59 smithi090 ceph-mon[41670]: 11.2 scrub starts 2024-04-14T15:27:59.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:27:59 smithi090 ceph-mon[41670]: 11.2 scrub ok 2024-04-14T15:27:59.488 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-14T15:27:59.727 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.b/config 2024-04-14T15:28:00.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:00 smithi092 ceph-mon[33966]: 11.17 scrub starts 2024-04-14T15:28:00.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:00 smithi092 ceph-mon[33966]: 11.17 scrub ok 2024-04-14T15:28:00.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:00 smithi092 ceph-mon[33966]: 6.11 scrub starts 2024-04-14T15:28:00.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:00 smithi092 ceph-mon[33966]: 6.11 scrub ok 2024-04-14T15:28:00.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:00 smithi092 ceph-mon[33966]: pgmap v1347: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 582 KiB data, 833 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-14T15:28:00.399 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:00 smithi090 ceph-mon[41670]: 11.17 scrub starts 2024-04-14T15:28:00.399 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:00 smithi090 ceph-mon[41670]: 11.17 scrub ok 2024-04-14T15:28:00.399 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:00 smithi090 ceph-mon[41670]: 6.11 scrub starts 2024-04-14T15:28:00.400 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:00 smithi090 ceph-mon[41670]: 6.11 scrub ok 2024-04-14T15:28:00.400 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:00 smithi090 ceph-mon[41670]: pgmap v1347: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 582 KiB data, 833 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-14T15:28:00.400 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:00 smithi090 ceph-mon[30399]: 11.17 scrub starts 2024-04-14T15:28:00.400 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:00 smithi090 ceph-mon[30399]: 11.17 scrub ok 2024-04-14T15:28:00.400 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:00 smithi090 ceph-mon[30399]: 6.11 scrub starts 2024-04-14T15:28:00.400 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:00 smithi090 ceph-mon[30399]: 6.11 scrub ok 2024-04-14T15:28:00.400 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:00 smithi090 ceph-mon[30399]: pgmap v1347: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 582 KiB data, 833 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-14T15:28:00.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:00.642+0000 7fdb41a96700 1 -- 172.21.15.90:0/1255806646 >> v1:172.21.15.90:6790/0 conn(0x7fdb3c10a670 legacy=0x7fdb3c10ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:00.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:00.643+0000 7fdb41a96700 1 -- 172.21.15.90:0/1255806646 shutdown_connections 2024-04-14T15:28:00.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:00.643+0000 7fdb41a96700 1 -- 172.21.15.90:0/1255806646 >> 172.21.15.90:0/1255806646 conn(0x7fdb3c0fd2d0 msgr2=0x7fdb3c0ff6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:28:00.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:00.643+0000 7fdb41a96700 1 -- 172.21.15.90:0/1255806646 shutdown_connections 2024-04-14T15:28:00.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:00.643+0000 7fdb41a96700 1 -- 172.21.15.90:0/1255806646 wait complete. 2024-04-14T15:28:00.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:00.643+0000 7fdb41a96700 1 librados: starting msgr at 2024-04-14T15:28:00.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:00.643+0000 7fdb41a96700 1 librados: starting objecter 2024-04-14T15:28:00.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:00.643+0000 7fdb41a96700 1 Processor -- start 2024-04-14T15:28:00.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:00.643+0000 7fdb41a96700 1 -- start start 2024-04-14T15:28:00.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:00.643+0000 7fdb41a96700 1 librados: setting wanted keys 2024-04-14T15:28:00.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:00.643+0000 7fdb41a96700 1 librados: calling monclient init 2024-04-14T15:28:00.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:00.644+0000 7fdb41a96700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdb3c1a7a40 con 0x7fdb3c10e240 2024-04-14T15:28:00.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:00.644+0000 7fdb41a96700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdb3c1a8c60 con 0x7fdb3c10a670 2024-04-14T15:28:00.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:00.644+0000 7fdb41a96700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdb3c1a9ea0 con 0x7fdb3c1077c0 2024-04-14T15:28:00.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:00.644+0000 7fdb3b7fe700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7fdb3c1077c0 0x7fdb3c06b830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:48540/0 (socket says 172.21.15.90:48540) 2024-04-14T15:28:00.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:00.644+0000 7fdb3b7fe700 1 -- 172.21.15.90:0/1738260923 learned_addr learned my addr 172.21.15.90:0/1738260923 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:28:00.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:00.644+0000 7fdb38ff9700 10 client.?.objecter ms_handle_connect 0x7fdb3c10e240 2024-04-14T15:28:00.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:00.644+0000 7fdb38ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:28:00.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:00.644+0000 7fdb38ff9700 10 client.?.objecter ms_handle_connect 0x7fdb3c1077c0 2024-04-14T15:28:00.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:00.644+0000 7fdb38ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:28:00.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:00.644+0000 7fdb38ff9700 1 -- 172.21.15.90:0/1738260923 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2604327478 0 0) 0x7fdb3c1a7a40 con 0x7fdb3c10e240 2024-04-14T15:28:00.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:00.644+0000 7fdb38ff9700 1 -- 172.21.15.90:0/1738260923 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdb240039c0 con 0x7fdb3c10e240 2024-04-14T15:28:00.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:00.644+0000 7fdb38ff9700 10 client.?.objecter ms_handle_connect 0x7fdb3c10a670 2024-04-14T15:28:00.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:00.644+0000 7fdb38ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:28:00.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:00.644+0000 7fdb38ff9700 1 -- 172.21.15.90:0/1738260923 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 422033150 0 0) 0x7fdb3c1a9ea0 con 0x7fdb3c1077c0 2024-04-14T15:28:00.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:00.644+0000 7fdb38ff9700 1 -- 172.21.15.90:0/1738260923 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdb3c1a7a40 con 0x7fdb3c1077c0 2024-04-14T15:28:00.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:00.644+0000 7fdb38ff9700 1 -- 172.21.15.90:0/1738260923 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2297799463 0 0) 0x7fdb3c1a8c60 con 0x7fdb3c10a670 2024-04-14T15:28:00.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:00.644+0000 7fdb38ff9700 1 -- 172.21.15.90:0/1738260923 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdb3c1a9ea0 con 0x7fdb3c10a670 2024-04-14T15:28:00.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:00.644+0000 7fdb38ff9700 1 -- 172.21.15.90:0/1738260923 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3908307897 0 0) 0x7fdb240039c0 con 0x7fdb3c10e240 2024-04-14T15:28:00.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:00.644+0000 7fdb38ff9700 1 -- 172.21.15.90:0/1738260923 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdb3c1a8c60 con 0x7fdb3c10e240 2024-04-14T15:28:00.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:00.644+0000 7fdb38ff9700 1 -- 172.21.15.90:0/1738260923 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fdb34001c30 con 0x7fdb3c10e240 2024-04-14T15:28:00.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:00.644+0000 7fdb38ff9700 1 -- 172.21.15.90:0/1738260923 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2622467866 0 0) 0x7fdb3c1a7a40 con 0x7fdb3c1077c0 2024-04-14T15:28:00.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:00.644+0000 7fdb38ff9700 1 -- 172.21.15.90:0/1738260923 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdb240039c0 con 0x7fdb3c1077c0 2024-04-14T15:28:00.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:00.644+0000 7fdb38ff9700 1 -- 172.21.15.90:0/1738260923 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fdb2c001960 con 0x7fdb3c1077c0 2024-04-14T15:28:00.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:00.644+0000 7fdb38ff9700 1 -- 172.21.15.90:0/1738260923 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2088866104 0 0) 0x7fdb3c1a9ea0 con 0x7fdb3c10a670 2024-04-14T15:28:00.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:00.644+0000 7fdb38ff9700 1 -- 172.21.15.90:0/1738260923 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdb3c1a7a40 con 0x7fdb3c10a670 2024-04-14T15:28:00.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:00.644+0000 7fdb38ff9700 1 -- 172.21.15.90:0/1738260923 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fdb300014f0 con 0x7fdb3c10a670 2024-04-14T15:28:00.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:00.644+0000 7fdb38ff9700 1 -- 172.21.15.90:0/1738260923 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2335390523 0 0) 0x7fdb3c1a8c60 con 0x7fdb3c10e240 2024-04-14T15:28:00.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:00.644+0000 7fdb38ff9700 1 -- 172.21.15.90:0/1738260923 >> v1:172.21.15.90:6790/0 conn(0x7fdb3c1077c0 legacy=0x7fdb3c06b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:00.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:00.645+0000 7fdb38ff9700 1 -- 172.21.15.90:0/1738260923 >> v1:172.21.15.92:6789/0 conn(0x7fdb3c10a670 legacy=0x7fdb3c1a2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:00.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:00.645+0000 7fdb38ff9700 1 -- 172.21.15.90:0/1738260923 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb3c1ab0e0 con 0x7fdb3c10e240 2024-04-14T15:28:00.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:00.645+0000 7fdb41a96700 1 -- 172.21.15.90:0/1738260923 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdb3c1aa0d0 con 0x7fdb3c10e240 2024-04-14T15:28:00.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:00.645+0000 7fdb41a96700 10 client.38259.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:28:00.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:00.645+0000 7fdb41a96700 1 -- 172.21.15.90:0/1738260923 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fdb3c1aa590 con 0x7fdb3c10e240 2024-04-14T15:28:00.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:00.645+0000 7fdb41a96700 1 librados: init done 2024-04-14T15:28:00.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:00.645+0000 7fdb38ff9700 1 -- 172.21.15.90:0/1738260923 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fdb34003ec0 con 0x7fdb3c10e240 2024-04-14T15:28:00.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:00.645+0000 7fdb38ff9700 1 -- 172.21.15.90:0/1738260923 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fdb34004e00 con 0x7fdb3c10e240 2024-04-14T15:28:00.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:00.646+0000 7fdb41a96700 1 -- 172.21.15.90:0/1738260923 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb28005120 con 0x7fdb3c10e240 2024-04-14T15:28:00.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:00.646+0000 7fdb38ff9700 1 -- 172.21.15.90:0/1738260923 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fdb3401ba90 con 0x7fdb3c10e240 2024-04-14T15:28:00.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:00.646+0000 7fdb38ff9700 1 -- 172.21.15.90:0/1738260923 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(895..895 src has 467..895) ==== 7550+0+0 (unknown 2616631029 0 0) 0x7fdb34089780 con 0x7fdb3c10e240 2024-04-14T15:28:00.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:00.646+0000 7fdb38ff9700 10 client.38259.objecter ms_dispatch 0x7fdb3c001550 osd_map(895..895 src has 467..895) 2024-04-14T15:28:00.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:00.646+0000 7fdb38ff9700 3 client.38259.objecter handle_osd_map got epochs [895,895] > 0 2024-04-14T15:28:00.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:00.646+0000 7fdb38ff9700 3 client.38259.objecter handle_osd_map decoding full epoch 895 2024-04-14T15:28:00.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:00.647+0000 7fdb38ff9700 20 client.38259.objecter dump_active .. 0 homeless 2024-04-14T15:28:00.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:00.647+0000 7fdb38ff9700 10 client.38259.objecter ms_handle_connect 0x7fdb24071330 2024-04-14T15:28:00.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:00.651+0000 7fdb38ff9700 1 -- 172.21.15.90:0/1738260923 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fdb34056540 con 0x7fdb3c10e240 2024-04-14T15:28:00.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:00.859+0000 7fdb41a96700 1 -- 172.21.15.90:0/1738260923 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd reweight", "id": 4, "weight": 1.0} v 0) -- 0x7fdb28005410 con 0x7fdb3c10e240 2024-04-14T15:28:01.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:01.052+0000 7fdb38ff9700 1 -- 172.21.15.90:0/1738260923 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 4, "weight": 1.0}]=0 reweighted osd.4 to 1 (10000) v896) ==== 113+0+0 (unknown 1653914632 0 0) 0x7fdb3405a1f0 con 0x7fdb3c10e240 2024-04-14T15:28:01.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:01.056+0000 7fdb41a96700 10 librados: watch_flush enter 2024-04-14T15:28:01.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:01.056+0000 7fdb41a96700 10 librados: watch_flush exit 2024-04-14T15:28:01.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:01.056+0000 7fdb41a96700 20 client.38259.objecter shutdown clearing up homeless session... 2024-04-14T15:28:01.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:01.056+0000 7fdb41a96700 10 client.38259.objecter successfully canceled tick 2024-04-14T15:28:01.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:01.056+0000 7fdb41a96700 1 -- 172.21.15.90:0/1738260923 >> v1:172.21.15.90:6800/420935427 conn(0x7fdb24071330 legacy=0x7fdb240737f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:01.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:01.056+0000 7fdb41a96700 1 -- 172.21.15.90:0/1738260923 >> v1:172.21.15.90:6789/0 conn(0x7fdb3c10e240 legacy=0x7fdb3c1a6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:01.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:01.056+0000 7fdb41a96700 1 -- 172.21.15.90:0/1738260923 shutdown_connections 2024-04-14T15:28:01.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:01.056+0000 7fdb41a96700 1 -- 172.21.15.90:0/1738260923 >> 172.21.15.90:0/1738260923 conn(0x7fdb3c0fd2d0 msgr2=0x7fdb3c1003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:28:01.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:01.057+0000 7fdb41a96700 1 -- 172.21.15.90:0/1738260923 shutdown_connections 2024-04-14T15:28:01.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:01.057+0000 7fdb41a96700 1 -- 172.21.15.90:0/1738260923 wait complete. 2024-04-14T15:28:01.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:01.057+0000 7fdb41a96700 1 librados: shutdown 2024-04-14T15:28:01.058 INFO:teuthology.orchestra.run.smithi090.stderr:reweighted osd.4 to 1 (10000) 2024-04-14T15:28:01.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:01 smithi092 ceph-mon[33966]: 11.1 scrub starts 2024-04-14T15:28:01.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:01 smithi092 ceph-mon[33966]: 11.1 scrub ok 2024-04-14T15:28:01.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:01 smithi092 ceph-mon[33966]: 11.13 scrub starts 2024-04-14T15:28:01.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:01 smithi092 ceph-mon[33966]: 11.13 scrub ok 2024-04-14T15:28:01.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:01 smithi092 ceph-mon[33966]: 2.f scrub starts 2024-04-14T15:28:01.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:01 smithi092 ceph-mon[33966]: 2.f scrub ok 2024-04-14T15:28:01.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:01 smithi092 ceph-mon[33966]: 11.15 scrub starts 2024-04-14T15:28:01.122 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:01 smithi092 ceph-mon[33966]: 11.15 scrub ok 2024-04-14T15:28:01.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:01 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1738260923' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-14T15:28:01.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:01 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1738260923' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-14T15:28:01.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:01 smithi092 ceph-mon[33966]: osdmap e896: 8 total, 8 up, 8 in 2024-04-14T15:28:01.440 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:01 smithi090 ceph-mon[41670]: 11.1 scrub starts 2024-04-14T15:28:01.440 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:01 smithi090 ceph-mon[41670]: 11.1 scrub ok 2024-04-14T15:28:01.440 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:01 smithi090 ceph-mon[41670]: 11.13 scrub starts 2024-04-14T15:28:01.440 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:01 smithi090 ceph-mon[41670]: 11.13 scrub ok 2024-04-14T15:28:01.440 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:01 smithi090 ceph-mon[41670]: 2.f scrub starts 2024-04-14T15:28:01.440 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:01 smithi090 ceph-mon[41670]: 2.f scrub ok 2024-04-14T15:28:01.440 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:01 smithi090 ceph-mon[41670]: 11.15 scrub starts 2024-04-14T15:28:01.440 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:01 smithi090 ceph-mon[41670]: 11.15 scrub ok 2024-04-14T15:28:01.440 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:01 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1738260923' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-14T15:28:01.440 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:01 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1738260923' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-14T15:28:01.440 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:01 smithi090 ceph-mon[41670]: osdmap e896: 8 total, 8 up, 8 in 2024-04-14T15:28:01.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:01 smithi090 ceph-mon[30399]: 11.1 scrub starts 2024-04-14T15:28:01.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:01 smithi090 ceph-mon[30399]: 11.1 scrub ok 2024-04-14T15:28:01.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:01 smithi090 ceph-mon[30399]: 11.13 scrub starts 2024-04-14T15:28:01.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:01 smithi090 ceph-mon[30399]: 11.13 scrub ok 2024-04-14T15:28:01.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:01 smithi090 ceph-mon[30399]: 2.f scrub starts 2024-04-14T15:28:01.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:01 smithi090 ceph-mon[30399]: 2.f scrub ok 2024-04-14T15:28:01.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:01 smithi090 ceph-mon[30399]: 11.15 scrub starts 2024-04-14T15:28:01.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:01 smithi090 ceph-mon[30399]: 11.15 scrub ok 2024-04-14T15:28:01.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:01 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1738260923' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-14T15:28:01.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:01 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1738260923' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-14T15:28:01.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:01 smithi090 ceph-mon[30399]: osdmap e896: 8 total, 8 up, 8 in 2024-04-14T15:28:01.577 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-04-14T15:28:01.578 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-14T15:28:01.945 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd primary-affinity 4 1 2024-04-14T15:28:02.190 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:28:02.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:02 smithi090 ceph-mon[41670]: 2.1b scrub starts 2024-04-14T15:28:02.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:02 smithi090 ceph-mon[41670]: 2.1b scrub ok 2024-04-14T15:28:02.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:02 smithi090 ceph-mon[41670]: pgmap v1349: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 582 KiB data, 833 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:02.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:02 smithi090 ceph-mon[41670]: 5.a scrub starts 2024-04-14T15:28:02.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:02 smithi090 ceph-mon[41670]: 5.a scrub ok 2024-04-14T15:28:02.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:02 smithi090 ceph-mon[30399]: 2.1b scrub starts 2024-04-14T15:28:02.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:02 smithi090 ceph-mon[30399]: 2.1b scrub ok 2024-04-14T15:28:02.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:02 smithi090 ceph-mon[30399]: pgmap v1349: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 582 KiB data, 833 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:02.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:02 smithi090 ceph-mon[30399]: 5.a scrub starts 2024-04-14T15:28:02.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:02 smithi090 ceph-mon[30399]: 5.a scrub ok 2024-04-14T15:28:02.340 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:02 smithi092 ceph-mon[33966]: 2.1b scrub starts 2024-04-14T15:28:02.341 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:02 smithi092 ceph-mon[33966]: 2.1b scrub ok 2024-04-14T15:28:02.341 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:02 smithi092 ceph-mon[33966]: pgmap v1349: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 582 KiB data, 833 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:02.341 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:02 smithi092 ceph-mon[33966]: 5.a scrub starts 2024-04-14T15:28:02.341 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:02 smithi092 ceph-mon[33966]: 5.a scrub ok 2024-04-14T15:28:03.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:03 smithi092 ceph-mon[33966]: 2.3 scrub starts 2024-04-14T15:28:03.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:03 smithi092 ceph-mon[33966]: 2.3 scrub ok 2024-04-14T15:28:03.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:03 smithi090 ceph-mon[41670]: 2.3 scrub starts 2024-04-14T15:28:03.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:03 smithi090 ceph-mon[41670]: 2.3 scrub ok 2024-04-14T15:28:03.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:03 smithi090 ceph-mon[30399]: 2.3 scrub starts 2024-04-14T15:28:03.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:03 smithi090 ceph-mon[30399]: 2.3 scrub ok 2024-04-14T15:28:03.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:03.975+0000 7f047bc22700 1 -- 172.21.15.90:0/514897998 >> v1:172.21.15.90:6789/0 conn(0x7f047410f300 legacy=0x7f04741117c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:03.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:03.976+0000 7f047bc22700 1 -- 172.21.15.90:0/514897998 shutdown_connections 2024-04-14T15:28:03.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:03.976+0000 7f047bc22700 1 -- 172.21.15.90:0/514897998 >> 172.21.15.90:0/514897998 conn(0x7f04740fe3d0 msgr2=0x7f04741007f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:28:03.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:03.976+0000 7f047bc22700 1 -- 172.21.15.90:0/514897998 shutdown_connections 2024-04-14T15:28:03.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:03.976+0000 7f047bc22700 1 -- 172.21.15.90:0/514897998 wait complete. 2024-04-14T15:28:03.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:03.976+0000 7f047bc22700 1 librados: starting msgr at 2024-04-14T15:28:03.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:03.976+0000 7f047bc22700 1 librados: starting objecter 2024-04-14T15:28:03.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:03.976+0000 7f047bc22700 1 Processor -- start 2024-04-14T15:28:03.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:03.976+0000 7f047bc22700 1 -- start start 2024-04-14T15:28:03.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:03.976+0000 7f047bc22700 1 librados: setting wanted keys 2024-04-14T15:28:03.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:03.976+0000 7f047bc22700 1 librados: calling monclient init 2024-04-14T15:28:03.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:03.976+0000 7f047bc22700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f04741a9c10 con 0x7f047410f300 2024-04-14T15:28:03.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:03.976+0000 7f047bc22700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f04741aae30 con 0x7f0474108880 2024-04-14T15:28:03.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:03.976+0000 7f047bc22700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f04741ac070 con 0x7f047410b730 2024-04-14T15:28:03.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:03.977+0000 7f04791bd700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f047410b730 0x7f04741a4ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:48554/0 (socket says 172.21.15.90:48554) 2024-04-14T15:28:03.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:03.977+0000 7f04791bd700 1 -- 172.21.15.90:0/1977321893 learned_addr learned my addr 172.21.15.90:0/1977321893 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:28:03.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:03.977+0000 7f046affd700 10 client.?.objecter ms_handle_connect 0x7f047410b730 2024-04-14T15:28:03.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:03.977+0000 7f046affd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:28:03.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:03.977+0000 7f046affd700 10 client.?.objecter ms_handle_connect 0x7f047410f300 2024-04-14T15:28:03.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:03.977+0000 7f046affd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:28:03.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:03.977+0000 7f046affd700 10 client.?.objecter ms_handle_connect 0x7f0474108880 2024-04-14T15:28:03.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:03.977+0000 7f046affd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:28:03.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:03.977+0000 7f046affd700 1 -- 172.21.15.90:0/1977321893 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2949950127 0 0) 0x7f04741a9c10 con 0x7f047410f300 2024-04-14T15:28:03.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:03.977+0000 7f046affd700 1 -- 172.21.15.90:0/1977321893 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f04600031f0 con 0x7f047410f300 2024-04-14T15:28:03.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:03.977+0000 7f046affd700 1 -- 172.21.15.90:0/1977321893 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 321230005 0 0) 0x7f04741aae30 con 0x7f0474108880 2024-04-14T15:28:03.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:03.977+0000 7f046affd700 1 -- 172.21.15.90:0/1977321893 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f04741a9c10 con 0x7f0474108880 2024-04-14T15:28:03.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:03.977+0000 7f046affd700 1 -- 172.21.15.90:0/1977321893 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1434685924 0 0) 0x7f04741ac070 con 0x7f047410b730 2024-04-14T15:28:03.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:03.977+0000 7f046affd700 1 -- 172.21.15.90:0/1977321893 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f04741aae30 con 0x7f047410b730 2024-04-14T15:28:03.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:03.977+0000 7f046affd700 1 -- 172.21.15.90:0/1977321893 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1440816828 0 0) 0x7f04600031f0 con 0x7f047410f300 2024-04-14T15:28:03.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:03.977+0000 7f046affd700 1 -- 172.21.15.90:0/1977321893 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f04741ac070 con 0x7f047410f300 2024-04-14T15:28:03.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:03.977+0000 7f046affd700 1 -- 172.21.15.90:0/1977321893 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f046c001d60 con 0x7f047410f300 2024-04-14T15:28:03.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:03.977+0000 7f046affd700 1 -- 172.21.15.90:0/1977321893 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3619231150 0 0) 0x7f04741a9c10 con 0x7f0474108880 2024-04-14T15:28:03.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:03.977+0000 7f046affd700 1 -- 172.21.15.90:0/1977321893 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f04600031f0 con 0x7f0474108880 2024-04-14T15:28:03.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:03.977+0000 7f046affd700 1 -- 172.21.15.90:0/1977321893 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1993360755 0 0) 0x7f04741aae30 con 0x7f047410b730 2024-04-14T15:28:03.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:03.977+0000 7f046affd700 1 -- 172.21.15.90:0/1977321893 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f04741a9c10 con 0x7f047410b730 2024-04-14T15:28:03.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:03.977+0000 7f046affd700 1 -- 172.21.15.90:0/1977321893 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f04700017f0 con 0x7f0474108880 2024-04-14T15:28:03.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:03.977+0000 7f046affd700 1 -- 172.21.15.90:0/1977321893 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f0464001830 con 0x7f047410b730 2024-04-14T15:28:03.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:03.977+0000 7f046affd700 1 -- 172.21.15.90:0/1977321893 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4155902252 0 0) 0x7f04741ac070 con 0x7f047410f300 2024-04-14T15:28:03.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:03.977+0000 7f046affd700 1 -- 172.21.15.90:0/1977321893 >> v1:172.21.15.90:6790/0 conn(0x7f047410b730 legacy=0x7f04741a4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:03.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:03.977+0000 7f046affd700 1 -- 172.21.15.90:0/1977321893 >> v1:172.21.15.92:6789/0 conn(0x7f0474108880 legacy=0x7f047410eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:03.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:03.977+0000 7f046affd700 1 -- 172.21.15.90:0/1977321893 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04741ad2b0 con 0x7f047410f300 2024-04-14T15:28:03.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:03.977+0000 7f047bc22700 1 -- 172.21.15.90:0/1977321893 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f04741ac2a0 con 0x7f047410f300 2024-04-14T15:28:03.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:03.978+0000 7f047bc22700 10 client.38265.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:28:03.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:03.978+0000 7f047bc22700 1 -- 172.21.15.90:0/1977321893 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f04741ac760 con 0x7f047410f300 2024-04-14T15:28:03.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:03.978+0000 7f047bc22700 1 librados: init done 2024-04-14T15:28:03.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:03.978+0000 7f046affd700 1 -- 172.21.15.90:0/1977321893 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f046c004ae0 con 0x7f047410f300 2024-04-14T15:28:03.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:03.978+0000 7f046affd700 1 -- 172.21.15.90:0/1977321893 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f046c004fc0 con 0x7f047410f300 2024-04-14T15:28:03.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:03.978+0000 7f047bc22700 1 -- 172.21.15.90:0/1977321893 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0458005150 con 0x7f047410f300 2024-04-14T15:28:03.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:03.979+0000 7f046affd700 1 -- 172.21.15.90:0/1977321893 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f046c01b9d0 con 0x7f047410f300 2024-04-14T15:28:03.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:03.979+0000 7f046affd700 1 -- 172.21.15.90:0/1977321893 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(896..896 src has 467..896) ==== 7550+0+0 (unknown 1331507821 0 0) 0x7f046c089790 con 0x7f047410f300 2024-04-14T15:28:03.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:03.979+0000 7f046affd700 10 client.38265.objecter ms_dispatch 0x7f0474001550 osd_map(896..896 src has 467..896) 2024-04-14T15:28:03.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:03.979+0000 7f046affd700 3 client.38265.objecter handle_osd_map got epochs [896,896] > 0 2024-04-14T15:28:03.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:03.979+0000 7f046affd700 3 client.38265.objecter handle_osd_map decoding full epoch 896 2024-04-14T15:28:03.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:03.980+0000 7f046affd700 20 client.38265.objecter dump_active .. 0 homeless 2024-04-14T15:28:03.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:03.980+0000 7f046affd700 10 client.38265.objecter ms_handle_connect 0x7f0460071310 2024-04-14T15:28:03.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:03.984+0000 7f046affd700 1 -- 172.21.15.90:0/1977321893 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f046c056470 con 0x7f047410f300 2024-04-14T15:28:04.117 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:28:03 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:28:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:28:04.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:04.205+0000 7f047bc22700 1 -- 172.21.15.90:0/1977321893 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} v 0) -- 0x7f0458005c50 con 0x7f047410f300 2024-04-14T15:28:04.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:04 smithi092 ceph-mon[33966]: 5.c scrub starts 2024-04-14T15:28:04.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:04 smithi092 ceph-mon[33966]: 5.c scrub ok 2024-04-14T15:28:04.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:04 smithi092 ceph-mon[33966]: 6.2 scrub starts 2024-04-14T15:28:04.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:04 smithi092 ceph-mon[33966]: 6.2 scrub ok 2024-04-14T15:28:04.372 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:04 smithi092 ceph-mon[33966]: 11.9 scrub starts 2024-04-14T15:28:04.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:04 smithi092 ceph-mon[33966]: 11.9 scrub ok 2024-04-14T15:28:04.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:04 smithi092 ceph-mon[33966]: pgmap v1350: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 582 KiB data, 833 MiB used, 714 GiB / 715 GiB avail; 712 B/s rd, 0 op/s 2024-04-14T15:28:04.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:04 smithi092 ceph-mon[33966]: 6.14 scrub starts 2024-04-14T15:28:04.373 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:04 smithi092 ceph-mon[33966]: 6.14 scrub ok 2024-04-14T15:28:04.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:04 smithi090 ceph-mon[41670]: 5.c scrub starts 2024-04-14T15:28:04.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:04 smithi090 ceph-mon[41670]: 5.c scrub ok 2024-04-14T15:28:04.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:04 smithi090 ceph-mon[41670]: 6.2 scrub starts 2024-04-14T15:28:04.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:04 smithi090 ceph-mon[41670]: 6.2 scrub ok 2024-04-14T15:28:04.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:04 smithi090 ceph-mon[41670]: 11.9 scrub starts 2024-04-14T15:28:04.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:04 smithi090 ceph-mon[41670]: 11.9 scrub ok 2024-04-14T15:28:04.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:04 smithi090 ceph-mon[41670]: pgmap v1350: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 582 KiB data, 833 MiB used, 714 GiB / 715 GiB avail; 712 B/s rd, 0 op/s 2024-04-14T15:28:04.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:04 smithi090 ceph-mon[41670]: 6.14 scrub starts 2024-04-14T15:28:04.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:04 smithi090 ceph-mon[41670]: 6.14 scrub ok 2024-04-14T15:28:04.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:04 smithi090 ceph-mon[30399]: 5.c scrub starts 2024-04-14T15:28:04.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:04 smithi090 ceph-mon[30399]: 5.c scrub ok 2024-04-14T15:28:04.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:04 smithi090 ceph-mon[30399]: 6.2 scrub starts 2024-04-14T15:28:04.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:04 smithi090 ceph-mon[30399]: 6.2 scrub ok 2024-04-14T15:28:04.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:04 smithi090 ceph-mon[30399]: 11.9 scrub starts 2024-04-14T15:28:04.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:04 smithi090 ceph-mon[30399]: 11.9 scrub ok 2024-04-14T15:28:04.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:04 smithi090 ceph-mon[30399]: pgmap v1350: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 582 KiB data, 833 MiB used, 714 GiB / 715 GiB avail; 712 B/s rd, 0 op/s 2024-04-14T15:28:04.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:04 smithi090 ceph-mon[30399]: 6.14 scrub starts 2024-04-14T15:28:04.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:04 smithi090 ceph-mon[30399]: 6.14 scrub ok 2024-04-14T15:28:05.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:05.118+0000 7f046affd700 1 -- 172.21.15.90:0/1977321893 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]=0 set osd.4 primary-affinity to 1 (8655362) v897) ==== 133+0+0 (unknown 846630143 0 0) 0x7f046c05a120 con 0x7f047410f300 2024-04-14T15:28:05.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:05.122+0000 7f047bc22700 10 librados: watch_flush enter 2024-04-14T15:28:05.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:05.122+0000 7f047bc22700 10 librados: watch_flush exit 2024-04-14T15:28:05.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:05.122+0000 7f047bc22700 20 client.38265.objecter shutdown clearing up homeless session... 2024-04-14T15:28:05.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:05.122+0000 7f047bc22700 10 client.38265.objecter successfully canceled tick 2024-04-14T15:28:05.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:05.123+0000 7f047bc22700 1 -- 172.21.15.90:0/1977321893 >> v1:172.21.15.90:6800/420935427 conn(0x7f0460071310 legacy=0x7f04600737d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:05.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:05.123+0000 7f047bc22700 1 -- 172.21.15.90:0/1977321893 >> v1:172.21.15.90:6789/0 conn(0x7f047410f300 legacy=0x7f04741a82d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:05.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:05.123+0000 7f047bc22700 1 -- 172.21.15.90:0/1977321893 shutdown_connections 2024-04-14T15:28:05.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:05.123+0000 7f047bc22700 1 -- 172.21.15.90:0/1977321893 >> 172.21.15.90:0/1977321893 conn(0x7f04740fe3d0 msgr2=0x7f04741014a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:28:05.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:05.123+0000 7f047bc22700 1 -- 172.21.15.90:0/1977321893 shutdown_connections 2024-04-14T15:28:05.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:05.123+0000 7f047bc22700 1 -- 172.21.15.90:0/1977321893 wait complete. 2024-04-14T15:28:05.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:05.124+0000 7f047bc22700 1 librados: shutdown 2024-04-14T15:28:05.126 INFO:teuthology.orchestra.run.smithi090.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-04-14T15:28:05.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:05 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1977321893' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-04-14T15:28:05.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:05 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1977321893' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-04-14T15:28:05.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:05 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1977321893' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-04-14T15:28:05.865 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd reweight 5 1 2024-04-14T15:28:06.113 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:28:06.122 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:28:05 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:28:06.135 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:06 smithi090 ceph-mon[30399]: 11.12 scrub starts 2024-04-14T15:28:06.135 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:06 smithi090 ceph-mon[30399]: 11.12 scrub ok 2024-04-14T15:28:06.391 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:06 smithi090 ceph-mon[30399]: 11.0 scrub starts 2024-04-14T15:28:06.391 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:06 smithi090 ceph-mon[30399]: 11.0 scrub ok 2024-04-14T15:28:06.391 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:06 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1977321893' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-04-14T15:28:06.391 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:06 smithi090 ceph-mon[30399]: osdmap e897: 8 total, 8 up, 8 in 2024-04-14T15:28:06.391 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:06 smithi090 ceph-mon[30399]: 2.19 scrub starts 2024-04-14T15:28:06.392 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:06 smithi090 ceph-mon[30399]: 2.19 scrub ok 2024-04-14T15:28:06.392 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:06 smithi090 ceph-mon[30399]: 11.19 scrub starts 2024-04-14T15:28:06.392 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:06 smithi090 ceph-mon[30399]: 11.19 scrub ok 2024-04-14T15:28:06.392 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:06 smithi090 ceph-mon[30399]: pgmap v1352: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 582 KiB data, 833 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-14T15:28:06.392 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:06 smithi090 ceph-mon[30399]: 2.11 scrub starts 2024-04-14T15:28:06.392 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:06 smithi090 ceph-mon[30399]: 2.11 scrub ok 2024-04-14T15:28:06.392 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:06 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:28:06.393 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:06 smithi090 ceph-mon[30399]: osdmap e898: 8 total, 8 up, 8 in 2024-04-14T15:28:06.393 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:06 smithi090 ceph-mon[41670]: 11.12 scrub starts 2024-04-14T15:28:06.393 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:06 smithi090 ceph-mon[41670]: 11.12 scrub ok 2024-04-14T15:28:06.393 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:06 smithi090 ceph-mon[41670]: 11.0 scrub starts 2024-04-14T15:28:06.393 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:06 smithi090 ceph-mon[41670]: 11.0 scrub ok 2024-04-14T15:28:06.393 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:06 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1977321893' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-04-14T15:28:06.393 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:06 smithi090 ceph-mon[41670]: osdmap e897: 8 total, 8 up, 8 in 2024-04-14T15:28:06.393 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:06 smithi090 ceph-mon[41670]: 2.19 scrub starts 2024-04-14T15:28:06.393 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:06 smithi090 ceph-mon[41670]: 2.19 scrub ok 2024-04-14T15:28:06.393 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:06 smithi090 ceph-mon[41670]: 11.19 scrub starts 2024-04-14T15:28:06.394 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:06 smithi090 ceph-mon[41670]: 11.19 scrub ok 2024-04-14T15:28:06.394 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:06 smithi090 ceph-mon[41670]: pgmap v1352: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 582 KiB data, 833 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-14T15:28:06.394 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:06 smithi090 ceph-mon[41670]: 2.11 scrub starts 2024-04-14T15:28:06.394 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:06 smithi090 ceph-mon[41670]: 2.11 scrub ok 2024-04-14T15:28:06.394 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:06 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:28:06.394 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:06 smithi090 ceph-mon[41670]: osdmap e898: 8 total, 8 up, 8 in 2024-04-14T15:28:06.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:06 smithi092 ceph-mon[33966]: 11.12 scrub starts 2024-04-14T15:28:06.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:06 smithi092 ceph-mon[33966]: 11.12 scrub ok 2024-04-14T15:28:06.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:06 smithi092 ceph-mon[33966]: 11.0 scrub starts 2024-04-14T15:28:06.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:06 smithi092 ceph-mon[33966]: 11.0 scrub ok 2024-04-14T15:28:06.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:06 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1977321893' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-04-14T15:28:06.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:06 smithi092 ceph-mon[33966]: osdmap e897: 8 total, 8 up, 8 in 2024-04-14T15:28:06.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:06 smithi092 ceph-mon[33966]: 2.19 scrub starts 2024-04-14T15:28:06.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:06 smithi092 ceph-mon[33966]: 2.19 scrub ok 2024-04-14T15:28:06.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:06 smithi092 ceph-mon[33966]: 11.19 scrub starts 2024-04-14T15:28:06.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:06 smithi092 ceph-mon[33966]: 11.19 scrub ok 2024-04-14T15:28:06.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:06 smithi092 ceph-mon[33966]: pgmap v1352: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 582 KiB data, 833 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-14T15:28:06.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:06 smithi092 ceph-mon[33966]: 2.11 scrub starts 2024-04-14T15:28:06.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:06 smithi092 ceph-mon[33966]: 2.11 scrub ok 2024-04-14T15:28:06.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:06 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:28:06.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:06 smithi092 ceph-mon[33966]: osdmap e898: 8 total, 8 up, 8 in 2024-04-14T15:28:07.352 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:07 smithi090 ceph-mon[41670]: 2.12 scrub starts 2024-04-14T15:28:07.352 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:07 smithi090 ceph-mon[41670]: 2.12 scrub ok 2024-04-14T15:28:07.353 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:07 smithi090 ceph-mon[41670]: 11.1e scrub starts 2024-04-14T15:28:07.353 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:07 smithi090 ceph-mon[41670]: 11.1e scrub ok 2024-04-14T15:28:07.353 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:07 smithi090 ceph-mon[41670]: 2.1 scrub starts 2024-04-14T15:28:07.353 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:07 smithi090 ceph-mon[41670]: 2.1 scrub ok 2024-04-14T15:28:07.353 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:07 smithi090 ceph-mon[30399]: 2.12 scrub starts 2024-04-14T15:28:07.353 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:07 smithi090 ceph-mon[30399]: 2.12 scrub ok 2024-04-14T15:28:07.353 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:07 smithi090 ceph-mon[30399]: 11.1e scrub starts 2024-04-14T15:28:07.353 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:07 smithi090 ceph-mon[30399]: 11.1e scrub ok 2024-04-14T15:28:07.353 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:07 smithi090 ceph-mon[30399]: 2.1 scrub starts 2024-04-14T15:28:07.353 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:07 smithi090 ceph-mon[30399]: 2.1 scrub ok 2024-04-14T15:28:07.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:07 smithi092 ceph-mon[33966]: 2.12 scrub starts 2024-04-14T15:28:07.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:07 smithi092 ceph-mon[33966]: 2.12 scrub ok 2024-04-14T15:28:07.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:07 smithi092 ceph-mon[33966]: 11.1e scrub starts 2024-04-14T15:28:07.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:07 smithi092 ceph-mon[33966]: 11.1e scrub ok 2024-04-14T15:28:07.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:07 smithi092 ceph-mon[33966]: 2.1 scrub starts 2024-04-14T15:28:07.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:07 smithi092 ceph-mon[33966]: 2.1 scrub ok 2024-04-14T15:28:07.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:07.843+0000 7f2b26655700 1 -- 172.21.15.90:0/4098843580 >> v1:172.21.15.90:6790/0 conn(0x7f2b2010a690 legacy=0x7f2b2010ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:07.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:07.843+0000 7f2b26655700 1 -- 172.21.15.90:0/4098843580 shutdown_connections 2024-04-14T15:28:07.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:07.843+0000 7f2b26655700 1 -- 172.21.15.90:0/4098843580 >> 172.21.15.90:0/4098843580 conn(0x7f2b20076aa0 msgr2=0x7f2b20076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:28:07.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:07.843+0000 7f2b26655700 1 -- 172.21.15.90:0/4098843580 shutdown_connections 2024-04-14T15:28:07.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:07.843+0000 7f2b26655700 1 -- 172.21.15.90:0/4098843580 wait complete. 2024-04-14T15:28:07.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:07.844+0000 7f2b26655700 1 librados: starting msgr at 2024-04-14T15:28:07.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:07.844+0000 7f2b26655700 1 librados: starting objecter 2024-04-14T15:28:07.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:07.844+0000 7f2b26655700 1 Processor -- start 2024-04-14T15:28:07.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:07.844+0000 7f2b26655700 1 -- start start 2024-04-14T15:28:07.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:07.844+0000 7f2b26655700 1 librados: setting wanted keys 2024-04-14T15:28:07.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:07.844+0000 7f2b26655700 1 librados: calling monclient init 2024-04-14T15:28:07.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:07.844+0000 7f2b26655700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f2b201a9ae0 con 0x7f2b201023f0 2024-04-14T15:28:07.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:07.844+0000 7f2b26655700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f2b201aad00 con 0x7f2b2010e230 2024-04-14T15:28:07.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:07.844+0000 7f2b26655700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f2b201abf40 con 0x7f2b2010a690 2024-04-14T15:28:07.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:07.844+0000 7f2b1ffff700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f2b201023f0 0x7f2b20106b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:41550/0 (socket says 172.21.15.90:41550) 2024-04-14T15:28:07.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:07.845+0000 7f2b1ffff700 1 -- 172.21.15.90:0/4016915767 learned_addr learned my addr 172.21.15.90:0/4016915767 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:28:07.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:07.844+0000 7f2b1f7fe700 1 --1- 172.21.15.90:0/4016915767 >> v1:172.21.15.90:6790/0 conn(0x7f2b2010a690 0x7f2b201a4ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:48572/0 (socket says 172.21.15.90:48572) 2024-04-14T15:28:07.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:07.845+0000 7f2b1d7fa700 10 client.?.objecter ms_handle_connect 0x7f2b201023f0 2024-04-14T15:28:07.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:07.845+0000 7f2b1d7fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:28:07.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:07.845+0000 7f2b1d7fa700 10 client.?.objecter ms_handle_connect 0x7f2b2010a690 2024-04-14T15:28:07.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:07.845+0000 7f2b1d7fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:28:07.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:07.845+0000 7f2b1d7fa700 10 client.?.objecter ms_handle_connect 0x7f2b2010e230 2024-04-14T15:28:07.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:07.845+0000 7f2b1d7fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:28:07.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:07.845+0000 7f2b1d7fa700 1 -- 172.21.15.90:0/4016915767 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2220065050 0 0) 0x7f2b201a9ae0 con 0x7f2b201023f0 2024-04-14T15:28:07.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:07.845+0000 7f2b1d7fa700 1 -- 172.21.15.90:0/4016915767 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b08003aa0 con 0x7f2b201023f0 2024-04-14T15:28:07.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:07.845+0000 7f2b1d7fa700 1 -- 172.21.15.90:0/4016915767 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2575618075 0 0) 0x7f2b201abf40 con 0x7f2b2010a690 2024-04-14T15:28:07.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:07.845+0000 7f2b1d7fa700 1 -- 172.21.15.90:0/4016915767 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b201a9ae0 con 0x7f2b2010a690 2024-04-14T15:28:07.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:07.845+0000 7f2b1d7fa700 1 -- 172.21.15.90:0/4016915767 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2528169433 0 0) 0x7f2b201aad00 con 0x7f2b2010e230 2024-04-14T15:28:07.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:07.845+0000 7f2b1d7fa700 1 -- 172.21.15.90:0/4016915767 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b201abf40 con 0x7f2b2010e230 2024-04-14T15:28:07.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:07.845+0000 7f2b1d7fa700 1 -- 172.21.15.90:0/4016915767 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 524485969 0 0) 0x7f2b08003aa0 con 0x7f2b201023f0 2024-04-14T15:28:07.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:07.845+0000 7f2b1d7fa700 1 -- 172.21.15.90:0/4016915767 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2b201aad00 con 0x7f2b201023f0 2024-04-14T15:28:07.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:07.845+0000 7f2b1d7fa700 1 -- 172.21.15.90:0/4016915767 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f2b10001930 con 0x7f2b201023f0 2024-04-14T15:28:07.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:07.845+0000 7f2b1d7fa700 1 -- 172.21.15.90:0/4016915767 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3282457137 0 0) 0x7f2b201a9ae0 con 0x7f2b2010a690 2024-04-14T15:28:07.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:07.845+0000 7f2b1d7fa700 1 -- 172.21.15.90:0/4016915767 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2b08003aa0 con 0x7f2b2010a690 2024-04-14T15:28:07.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:07.845+0000 7f2b1d7fa700 1 -- 172.21.15.90:0/4016915767 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f2b14003820 con 0x7f2b2010a690 2024-04-14T15:28:07.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:07.845+0000 7f2b1d7fa700 1 -- 172.21.15.90:0/4016915767 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1172018713 0 0) 0x7f2b201abf40 con 0x7f2b2010e230 2024-04-14T15:28:07.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:07.845+0000 7f2b1d7fa700 1 -- 172.21.15.90:0/4016915767 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2b201a9ae0 con 0x7f2b2010e230 2024-04-14T15:28:07.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:07.845+0000 7f2b1d7fa700 1 -- 172.21.15.90:0/4016915767 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f2b18001590 con 0x7f2b2010e230 2024-04-14T15:28:07.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:07.845+0000 7f2b1d7fa700 1 -- 172.21.15.90:0/4016915767 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2180909835 0 0) 0x7f2b201aad00 con 0x7f2b201023f0 2024-04-14T15:28:07.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:07.845+0000 7f2b1d7fa700 1 -- 172.21.15.90:0/4016915767 >> v1:172.21.15.90:6790/0 conn(0x7f2b2010a690 legacy=0x7f2b201a4ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:07.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:07.845+0000 7f2b1d7fa700 1 -- 172.21.15.90:0/4016915767 >> v1:172.21.15.92:6789/0 conn(0x7f2b2010e230 legacy=0x7f2b201a81f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:07.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:07.845+0000 7f2b1d7fa700 1 -- 172.21.15.90:0/4016915767 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b201ad180 con 0x7f2b201023f0 2024-04-14T15:28:07.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:07.845+0000 7f2b26655700 1 -- 172.21.15.90:0/4016915767 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2b201ac170 con 0x7f2b201023f0 2024-04-14T15:28:07.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:07.845+0000 7f2b26655700 10 client.38271.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:28:07.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:07.845+0000 7f2b26655700 1 -- 172.21.15.90:0/4016915767 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f2b201ac630 con 0x7f2b201023f0 2024-04-14T15:28:07.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:07.845+0000 7f2b26655700 1 librados: init done 2024-04-14T15:28:07.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:07.845+0000 7f2b1d7fa700 1 -- 172.21.15.90:0/4016915767 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f2b10001d00 con 0x7f2b201023f0 2024-04-14T15:28:07.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:07.846+0000 7f2b1d7fa700 1 -- 172.21.15.90:0/4016915767 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f2b10005d70 con 0x7f2b201023f0 2024-04-14T15:28:07.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:07.846+0000 7f2b26655700 1 -- 172.21.15.90:0/4016915767 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b0c005150 con 0x7f2b201023f0 2024-04-14T15:28:07.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:07.847+0000 7f2b1d7fa700 1 -- 172.21.15.90:0/4016915767 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f2b10003950 con 0x7f2b201023f0 2024-04-14T15:28:07.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:07.847+0000 7f2b1d7fa700 1 -- 172.21.15.90:0/4016915767 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(898..898 src has 467..898) ==== 7550+0+0 (unknown 879414213 0 0) 0x7f2b1008a6b0 con 0x7f2b201023f0 2024-04-14T15:28:07.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:07.847+0000 7f2b1d7fa700 10 client.38271.objecter ms_dispatch 0x7f2b20001550 osd_map(898..898 src has 467..898) 2024-04-14T15:28:07.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:07.847+0000 7f2b1d7fa700 3 client.38271.objecter handle_osd_map got epochs [898,898] > 0 2024-04-14T15:28:07.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:07.847+0000 7f2b1d7fa700 3 client.38271.objecter handle_osd_map decoding full epoch 898 2024-04-14T15:28:07.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:07.848+0000 7f2b1d7fa700 20 client.38271.objecter dump_active .. 0 homeless 2024-04-14T15:28:07.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:07.848+0000 7f2b1d7fa700 10 client.38271.objecter ms_handle_connect 0x7f2b080713a0 2024-04-14T15:28:07.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:07.852+0000 7f2b1d7fa700 1 -- 172.21.15.90:0/4016915767 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f2b10057470 con 0x7f2b201023f0 2024-04-14T15:28:08.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:08.065+0000 7f2b26655700 1 -- 172.21.15.90:0/4016915767 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd reweight", "id": 5, "weight": 1.0} v 0) -- 0x7f2b0c005440 con 0x7f2b201023f0 2024-04-14T15:28:08.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:08.137+0000 7f2b1d7fa700 1 -- 172.21.15.90:0/4016915767 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 5, "weight": 1.0}]=0 reweighted osd.5 to 1 (10000) v899) ==== 113+0+0 (unknown 1442407222 0 0) 0x7f2b1005b120 con 0x7f2b201023f0 2024-04-14T15:28:08.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:08.139+0000 7f2b26655700 10 librados: watch_flush enter 2024-04-14T15:28:08.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:08.139+0000 7f2b26655700 10 librados: watch_flush exit 2024-04-14T15:28:08.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:08.139+0000 7f2b26655700 20 client.38271.objecter shutdown clearing up homeless session... 2024-04-14T15:28:08.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:08.139+0000 7f2b26655700 10 client.38271.objecter successfully canceled tick 2024-04-14T15:28:08.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:08.139+0000 7f2b26655700 1 -- 172.21.15.90:0/4016915767 >> v1:172.21.15.90:6800/420935427 conn(0x7f2b080713a0 legacy=0x7f2b08073860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:08.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:08.139+0000 7f2b26655700 1 -- 172.21.15.90:0/4016915767 >> v1:172.21.15.90:6789/0 conn(0x7f2b201023f0 legacy=0x7f2b20106b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:08.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:08.139+0000 7f2b26655700 1 -- 172.21.15.90:0/4016915767 shutdown_connections 2024-04-14T15:28:08.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:08.139+0000 7f2b26655700 1 -- 172.21.15.90:0/4016915767 >> 172.21.15.90:0/4016915767 conn(0x7f2b20076aa0 msgr2=0x7f2b201119f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:28:08.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:08.139+0000 7f2b26655700 1 -- 172.21.15.90:0/4016915767 shutdown_connections 2024-04-14T15:28:08.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:08.139+0000 7f2b26655700 1 -- 172.21.15.90:0/4016915767 wait complete. 2024-04-14T15:28:08.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:08.139+0000 7f2b26655700 1 librados: shutdown 2024-04-14T15:28:08.142 INFO:teuthology.orchestra.run.smithi090.stderr:reweighted osd.5 to 1 (10000) 2024-04-14T15:28:08.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:08 smithi090 ceph-mon[41670]: 6.a scrub starts 2024-04-14T15:28:08.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:08 smithi090 ceph-mon[41670]: 6.a scrub ok 2024-04-14T15:28:08.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:08 smithi090 ceph-mon[41670]: 11.c deep-scrub starts 2024-04-14T15:28:08.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:08 smithi090 ceph-mon[41670]: 11.c deep-scrub ok 2024-04-14T15:28:08.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:08 smithi090 ceph-mon[41670]: pgmap v1354: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 582 KiB data, 833 MiB used, 714 GiB / 715 GiB avail; 828 B/s rd, 0 op/s 2024-04-14T15:28:08.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:08 smithi090 ceph-mon[41670]: 5.2 scrub starts 2024-04-14T15:28:08.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:08 smithi090 ceph-mon[41670]: 5.2 scrub ok 2024-04-14T15:28:08.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:08 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/4016915767' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-14T15:28:08.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:08 smithi090 ceph-mon[30399]: 6.a scrub starts 2024-04-14T15:28:08.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:08 smithi090 ceph-mon[30399]: 6.a scrub ok 2024-04-14T15:28:08.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:08 smithi090 ceph-mon[30399]: 11.c deep-scrub starts 2024-04-14T15:28:08.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:08 smithi090 ceph-mon[30399]: 11.c deep-scrub ok 2024-04-14T15:28:08.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:08 smithi090 ceph-mon[30399]: pgmap v1354: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 582 KiB data, 833 MiB used, 714 GiB / 715 GiB avail; 828 B/s rd, 0 op/s 2024-04-14T15:28:08.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:08 smithi090 ceph-mon[30399]: 5.2 scrub starts 2024-04-14T15:28:08.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:08 smithi090 ceph-mon[30399]: 5.2 scrub ok 2024-04-14T15:28:08.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:08 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/4016915767' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-14T15:28:08.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:08 smithi092 ceph-mon[33966]: 6.a scrub starts 2024-04-14T15:28:08.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:08 smithi092 ceph-mon[33966]: 6.a scrub ok 2024-04-14T15:28:08.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:08 smithi092 ceph-mon[33966]: 11.c deep-scrub starts 2024-04-14T15:28:08.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:08 smithi092 ceph-mon[33966]: 11.c deep-scrub ok 2024-04-14T15:28:08.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:08 smithi092 ceph-mon[33966]: pgmap v1354: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 582 KiB data, 833 MiB used, 714 GiB / 715 GiB avail; 828 B/s rd, 0 op/s 2024-04-14T15:28:08.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:08 smithi092 ceph-mon[33966]: 5.2 scrub starts 2024-04-14T15:28:08.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:08 smithi092 ceph-mon[33966]: 5.2 scrub ok 2024-04-14T15:28:08.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:08 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/4016915767' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-14T15:28:08.922 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd primary-affinity 5 1 2024-04-14T15:28:09.168 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:28:09.412 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:09 smithi090 ceph-mon[41670]: 5.5 deep-scrub starts 2024-04-14T15:28:09.413 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:09 smithi090 ceph-mon[41670]: 5.5 deep-scrub ok 2024-04-14T15:28:09.413 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:09 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/4016915767' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-14T15:28:09.413 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:09 smithi090 ceph-mon[41670]: osdmap e899: 8 total, 8 up, 8 in 2024-04-14T15:28:09.413 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:09 smithi090 ceph-mon[30399]: 5.5 deep-scrub starts 2024-04-14T15:28:09.413 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:09 smithi090 ceph-mon[30399]: 5.5 deep-scrub ok 2024-04-14T15:28:09.413 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:09 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/4016915767' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-14T15:28:09.413 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:09 smithi090 ceph-mon[30399]: osdmap e899: 8 total, 8 up, 8 in 2024-04-14T15:28:09.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:09 smithi092 ceph-mon[33966]: 5.5 deep-scrub starts 2024-04-14T15:28:09.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:09 smithi092 ceph-mon[33966]: 5.5 deep-scrub ok 2024-04-14T15:28:09.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:09 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/4016915767' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-14T15:28:09.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:09 smithi092 ceph-mon[33966]: osdmap e899: 8 total, 8 up, 8 in 2024-04-14T15:28:10.413 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:10 smithi090 ceph-mon[41670]: 11.4 scrub starts 2024-04-14T15:28:10.413 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:10 smithi090 ceph-mon[41670]: 11.4 scrub ok 2024-04-14T15:28:10.413 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:10 smithi090 ceph-mon[41670]: 3.2 scrub starts 2024-04-14T15:28:10.413 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:10 smithi090 ceph-mon[41670]: 3.2 scrub ok 2024-04-14T15:28:10.413 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:10 smithi090 ceph-mon[41670]: pgmap v1356: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 582 KiB data, 833 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-14T15:28:10.414 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:10 smithi090 ceph-mon[41670]: 2.1d scrub starts 2024-04-14T15:28:10.414 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:10 smithi090 ceph-mon[41670]: 2.1d scrub ok 2024-04-14T15:28:10.414 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:10 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:28:10.414 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:10 smithi090 ceph-mon[41670]: 2.16 scrub starts 2024-04-14T15:28:10.414 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:10 smithi090 ceph-mon[41670]: 2.16 scrub ok 2024-04-14T15:28:10.414 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:10 smithi090 ceph-mon[30399]: 11.4 scrub starts 2024-04-14T15:28:10.414 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:10 smithi090 ceph-mon[30399]: 11.4 scrub ok 2024-04-14T15:28:10.414 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:10 smithi090 ceph-mon[30399]: 3.2 scrub starts 2024-04-14T15:28:10.414 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:10 smithi090 ceph-mon[30399]: 3.2 scrub ok 2024-04-14T15:28:10.414 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:10 smithi090 ceph-mon[30399]: pgmap v1356: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 582 KiB data, 833 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-14T15:28:10.415 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:10 smithi090 ceph-mon[30399]: 2.1d scrub starts 2024-04-14T15:28:10.415 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:10 smithi090 ceph-mon[30399]: 2.1d scrub ok 2024-04-14T15:28:10.415 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:10 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:28:10.415 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:10 smithi090 ceph-mon[30399]: 2.16 scrub starts 2024-04-14T15:28:10.415 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:10 smithi090 ceph-mon[30399]: 2.16 scrub ok 2024-04-14T15:28:10.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:10 smithi092 ceph-mon[33966]: 11.4 scrub starts 2024-04-14T15:28:10.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:10 smithi092 ceph-mon[33966]: 11.4 scrub ok 2024-04-14T15:28:10.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:10 smithi092 ceph-mon[33966]: 3.2 scrub starts 2024-04-14T15:28:10.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:10 smithi092 ceph-mon[33966]: 3.2 scrub ok 2024-04-14T15:28:10.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:10 smithi092 ceph-mon[33966]: pgmap v1356: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 582 KiB data, 833 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-14T15:28:10.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:10 smithi092 ceph-mon[33966]: 2.1d scrub starts 2024-04-14T15:28:10.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:10 smithi092 ceph-mon[33966]: 2.1d scrub ok 2024-04-14T15:28:10.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:10 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:28:10.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:10 smithi092 ceph-mon[33966]: 2.16 scrub starts 2024-04-14T15:28:10.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:10 smithi092 ceph-mon[33966]: 2.16 scrub ok 2024-04-14T15:28:10.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:10.906+0000 7f3fa025b700 1 -- 172.21.15.90:0/3645899837 >> v1:172.21.15.90:6789/0 conn(0x7f3f9810f2f0 legacy=0x7f3f981117b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:10.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:10.907+0000 7f3fa025b700 1 -- 172.21.15.90:0/3645899837 shutdown_connections 2024-04-14T15:28:10.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:10.907+0000 7f3fa025b700 1 -- 172.21.15.90:0/3645899837 >> 172.21.15.90:0/3645899837 conn(0x7f3f980fe3a0 msgr2=0x7f3f981007c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:28:10.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:10.907+0000 7f3fa025b700 1 -- 172.21.15.90:0/3645899837 shutdown_connections 2024-04-14T15:28:10.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:10.907+0000 7f3fa025b700 1 -- 172.21.15.90:0/3645899837 wait complete. 2024-04-14T15:28:10.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:10.907+0000 7f3fa025b700 1 librados: starting msgr at 2024-04-14T15:28:10.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:10.907+0000 7f3fa025b700 1 librados: starting objecter 2024-04-14T15:28:10.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:10.907+0000 7f3fa025b700 1 Processor -- start 2024-04-14T15:28:10.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:10.907+0000 7f3fa025b700 1 -- start start 2024-04-14T15:28:10.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:10.907+0000 7f3fa025b700 1 librados: setting wanted keys 2024-04-14T15:28:10.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:10.907+0000 7f3fa025b700 1 librados: calling monclient init 2024-04-14T15:28:10.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:10.907+0000 7f3fa025b700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3f981a9c20 con 0x7f3f9810f2f0 2024-04-14T15:28:10.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:10.907+0000 7f3fa025b700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3f981aae40 con 0x7f3f98108870 2024-04-14T15:28:10.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:10.907+0000 7f3fa025b700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3f981ac080 con 0x7f3f9810b720 2024-04-14T15:28:10.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:10.908+0000 7f3f9d7f6700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f3f9810b720 0x7f3f981a4b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:34696/0 (socket says 172.21.15.90:34696) 2024-04-14T15:28:10.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:10.908+0000 7f3f9d7f6700 1 -- 172.21.15.90:0/1107440711 learned_addr learned my addr 172.21.15.90:0/1107440711 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:28:10.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:10.908+0000 7f3f8effd700 10 client.?.objecter ms_handle_connect 0x7f3f98108870 2024-04-14T15:28:10.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:10.908+0000 7f3f8effd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:28:10.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:10.908+0000 7f3f8effd700 10 client.?.objecter ms_handle_connect 0x7f3f9810f2f0 2024-04-14T15:28:10.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:10.908+0000 7f3f8effd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:28:10.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:10.908+0000 7f3f8effd700 10 client.?.objecter ms_handle_connect 0x7f3f9810b720 2024-04-14T15:28:10.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:10.908+0000 7f3f8effd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:28:10.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:10.908+0000 7f3f8effd700 1 -- 172.21.15.90:0/1107440711 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1112876208 0 0) 0x7f3f981aae40 con 0x7f3f98108870 2024-04-14T15:28:10.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:10.908+0000 7f3f8effd700 1 -- 172.21.15.90:0/1107440711 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f840035b0 con 0x7f3f98108870 2024-04-14T15:28:10.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:10.908+0000 7f3f8effd700 1 -- 172.21.15.90:0/1107440711 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1969971589 0 0) 0x7f3f981ac080 con 0x7f3f9810b720 2024-04-14T15:28:10.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:10.908+0000 7f3f8effd700 1 -- 172.21.15.90:0/1107440711 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f981aae40 con 0x7f3f9810b720 2024-04-14T15:28:10.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:10.908+0000 7f3f8effd700 1 -- 172.21.15.90:0/1107440711 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2375676916 0 0) 0x7f3f981a9c20 con 0x7f3f9810f2f0 2024-04-14T15:28:10.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:10.908+0000 7f3f8effd700 1 -- 172.21.15.90:0/1107440711 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f981ac080 con 0x7f3f9810f2f0 2024-04-14T15:28:10.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:10.908+0000 7f3f8effd700 1 -- 172.21.15.90:0/1107440711 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3185531581 0 0) 0x7f3f840035b0 con 0x7f3f98108870 2024-04-14T15:28:10.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:10.908+0000 7f3f8effd700 1 -- 172.21.15.90:0/1107440711 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3f981a9c20 con 0x7f3f98108870 2024-04-14T15:28:10.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:10.908+0000 7f3f8effd700 1 -- 172.21.15.90:0/1107440711 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f3f940018b0 con 0x7f3f98108870 2024-04-14T15:28:10.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:10.908+0000 7f3f8effd700 1 -- 172.21.15.90:0/1107440711 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1811597638 0 0) 0x7f3f981ac080 con 0x7f3f9810f2f0 2024-04-14T15:28:10.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:10.908+0000 7f3f8effd700 1 -- 172.21.15.90:0/1107440711 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3f840035b0 con 0x7f3f9810f2f0 2024-04-14T15:28:10.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:10.908+0000 7f3f8effd700 1 -- 172.21.15.90:0/1107440711 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3256112339 0 0) 0x7f3f981aae40 con 0x7f3f9810b720 2024-04-14T15:28:10.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:10.908+0000 7f3f8effd700 1 -- 172.21.15.90:0/1107440711 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3f981ac080 con 0x7f3f9810b720 2024-04-14T15:28:10.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:10.908+0000 7f3f8effd700 1 -- 172.21.15.90:0/1107440711 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f3f900047f0 con 0x7f3f9810f2f0 2024-04-14T15:28:10.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:10.908+0000 7f3f8effd700 1 -- 172.21.15.90:0/1107440711 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f3f88001870 con 0x7f3f9810b720 2024-04-14T15:28:10.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:10.908+0000 7f3f8effd700 1 -- 172.21.15.90:0/1107440711 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3276128658 0 0) 0x7f3f981a9c20 con 0x7f3f98108870 2024-04-14T15:28:10.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:10.908+0000 7f3f8effd700 1 -- 172.21.15.90:0/1107440711 >> v1:172.21.15.90:6790/0 conn(0x7f3f9810b720 legacy=0x7f3f981a4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:10.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:10.908+0000 7f3f8effd700 1 -- 172.21.15.90:0/1107440711 >> v1:172.21.15.90:6789/0 conn(0x7f3f9810f2f0 legacy=0x7f3f981a8320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:10.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:10.908+0000 7f3f8effd700 1 -- 172.21.15.90:0/1107440711 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f981ad2c0 con 0x7f3f98108870 2024-04-14T15:28:10.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:10.908+0000 7f3fa025b700 1 -- 172.21.15.90:0/1107440711 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3f981a9e50 con 0x7f3f98108870 2024-04-14T15:28:10.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:10.909+0000 7f3fa025b700 10 client.50098.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:28:10.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:10.909+0000 7f3fa025b700 1 -- 172.21.15.90:0/1107440711 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f3f981aa310 con 0x7f3f98108870 2024-04-14T15:28:10.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:10.909+0000 7f3fa025b700 1 librados: init done 2024-04-14T15:28:10.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:10.909+0000 7f3f8effd700 1 -- 172.21.15.90:0/1107440711 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f3f94003b40 con 0x7f3f98108870 2024-04-14T15:28:10.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:10.909+0000 7f3f8effd700 1 -- 172.21.15.90:0/1107440711 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f3f94004af0 con 0x7f3f98108870 2024-04-14T15:28:10.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:10.910+0000 7f3fa025b700 1 -- 172.21.15.90:0/1107440711 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f7c005120 con 0x7f3f98108870 2024-04-14T15:28:10.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:10.910+0000 7f3f8effd700 1 -- 172.21.15.90:0/1107440711 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f3f940036a0 con 0x7f3f98108870 2024-04-14T15:28:10.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:10.911+0000 7f3f8effd700 1 -- 172.21.15.90:0/1107440711 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(899..899 src has 467..899) ==== 7550+0+0 (unknown 166803217 0 0) 0x7f3f94089770 con 0x7f3f98108870 2024-04-14T15:28:10.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:10.911+0000 7f3f8effd700 10 client.50098.objecter ms_dispatch 0x7f3f98001550 osd_map(899..899 src has 467..899) 2024-04-14T15:28:10.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:10.911+0000 7f3f8effd700 3 client.50098.objecter handle_osd_map got epochs [899,899] > 0 2024-04-14T15:28:10.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:10.911+0000 7f3f8effd700 3 client.50098.objecter handle_osd_map decoding full epoch 899 2024-04-14T15:28:10.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:10.911+0000 7f3f8effd700 20 client.50098.objecter dump_active .. 0 homeless 2024-04-14T15:28:10.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:10.911+0000 7f3f8effd700 10 client.50098.objecter ms_handle_connect 0x7f3f840714b0 2024-04-14T15:28:10.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:10.916+0000 7f3f8effd700 1 -- 172.21.15.90:0/1107440711 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f3f94056450 con 0x7f3f98108870 2024-04-14T15:28:11.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:11.165+0000 7f3fa025b700 1 -- 172.21.15.90:0/1107440711 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} v 0) -- 0x7f3f7c005c20 con 0x7f3f98108870 2024-04-14T15:28:11.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:11 smithi090 ceph-mon[30399]: 6.b scrub starts 2024-04-14T15:28:11.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:11 smithi090 ceph-mon[30399]: 6.b scrub ok 2024-04-14T15:28:11.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:11 smithi090 ceph-mon[30399]: 4.4 scrub starts 2024-04-14T15:28:11.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:11 smithi090 ceph-mon[30399]: 4.4 scrub ok 2024-04-14T15:28:11.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:11 smithi090 ceph-mon[30399]: osdmap e900: 8 total, 8 up, 8 in 2024-04-14T15:28:11.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:11 smithi090 ceph-mon[41670]: 6.b scrub starts 2024-04-14T15:28:11.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:11 smithi090 ceph-mon[41670]: 6.b scrub ok 2024-04-14T15:28:11.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:11 smithi090 ceph-mon[41670]: 4.4 scrub starts 2024-04-14T15:28:11.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:11 smithi090 ceph-mon[41670]: 4.4 scrub ok 2024-04-14T15:28:11.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:11 smithi090 ceph-mon[41670]: osdmap e900: 8 total, 8 up, 8 in 2024-04-14T15:28:11.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:11 smithi092 ceph-mon[33966]: 6.b scrub starts 2024-04-14T15:28:11.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:11 smithi092 ceph-mon[33966]: 6.b scrub ok 2024-04-14T15:28:11.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:11 smithi092 ceph-mon[33966]: 4.4 scrub starts 2024-04-14T15:28:11.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:11 smithi092 ceph-mon[33966]: 4.4 scrub ok 2024-04-14T15:28:11.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:11 smithi092 ceph-mon[33966]: osdmap e900: 8 total, 8 up, 8 in 2024-04-14T15:28:12.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:12.163+0000 7f3f8effd700 1 -- 172.21.15.90:0/1107440711 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]=0 set osd.5 primary-affinity to 1 (8655362) v901) ==== 133+0+0 (unknown 1036504499 0 0) 0x7f3f9405a100 con 0x7f3f98108870 2024-04-14T15:28:12.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:12.166+0000 7f3fa025b700 10 librados: watch_flush enter 2024-04-14T15:28:12.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:12.166+0000 7f3fa025b700 10 librados: watch_flush exit 2024-04-14T15:28:12.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:12.166+0000 7f3fa025b700 20 client.50098.objecter shutdown clearing up homeless session... 2024-04-14T15:28:12.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:12.166+0000 7f3fa025b700 10 client.50098.objecter successfully canceled tick 2024-04-14T15:28:12.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:12.166+0000 7f3fa025b700 1 -- 172.21.15.90:0/1107440711 >> v1:172.21.15.90:6800/420935427 conn(0x7f3f840714b0 legacy=0x7f3f84073970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:12.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:12.166+0000 7f3fa025b700 1 -- 172.21.15.90:0/1107440711 >> v1:172.21.15.92:6789/0 conn(0x7f3f98108870 legacy=0x7f3f98076670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:12.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:12.166+0000 7f3fa025b700 1 -- 172.21.15.90:0/1107440711 shutdown_connections 2024-04-14T15:28:12.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:12.166+0000 7f3fa025b700 1 -- 172.21.15.90:0/1107440711 >> 172.21.15.90:0/1107440711 conn(0x7f3f980fe3a0 msgr2=0x7f3f98112610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:28:12.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:12.166+0000 7f3fa025b700 1 -- 172.21.15.90:0/1107440711 shutdown_connections 2024-04-14T15:28:12.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:12.166+0000 7f3fa025b700 1 -- 172.21.15.90:0/1107440711 wait complete. 2024-04-14T15:28:12.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:12.166+0000 7f3fa025b700 1 librados: shutdown 2024-04-14T15:28:12.168 INFO:teuthology.orchestra.run.smithi090.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-04-14T15:28:12.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:12 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:28:12.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:12 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:28:12.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:12 smithi090 ceph-mon[41670]: 11.5 deep-scrub starts 2024-04-14T15:28:12.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:12 smithi090 ceph-mon[41670]: 11.5 deep-scrub ok 2024-04-14T15:28:12.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:12 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1107440711' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-04-14T15:28:12.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:12 smithi090 ceph-mon[41670]: from='client.50098 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-04-14T15:28:12.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:12 smithi090 ceph-mon[41670]: 2.a scrub starts 2024-04-14T15:28:12.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:12 smithi090 ceph-mon[41670]: 2.a scrub ok 2024-04-14T15:28:12.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:12 smithi090 ceph-mon[41670]: pgmap v1358: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 582 KiB data, 833 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:12.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:12 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:28:12.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:12 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:28:12.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:12 smithi090 ceph-mon[30399]: 11.5 deep-scrub starts 2024-04-14T15:28:12.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:12 smithi090 ceph-mon[30399]: 11.5 deep-scrub ok 2024-04-14T15:28:12.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:12 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1107440711' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-04-14T15:28:12.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:12 smithi090 ceph-mon[30399]: from='client.50098 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-04-14T15:28:12.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:12 smithi090 ceph-mon[30399]: 2.a scrub starts 2024-04-14T15:28:12.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:12 smithi090 ceph-mon[30399]: 2.a scrub ok 2024-04-14T15:28:12.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:12 smithi090 ceph-mon[30399]: pgmap v1358: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 582 KiB data, 833 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:12.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:12 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:28:12.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:12 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:28:12.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:12 smithi092 ceph-mon[33966]: 11.5 deep-scrub starts 2024-04-14T15:28:12.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:12 smithi092 ceph-mon[33966]: 11.5 deep-scrub ok 2024-04-14T15:28:12.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:12 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1107440711' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-04-14T15:28:12.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:12 smithi092 ceph-mon[33966]: from='client.50098 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-04-14T15:28:12.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:12 smithi092 ceph-mon[33966]: 2.a scrub starts 2024-04-14T15:28:12.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:12 smithi092 ceph-mon[33966]: 2.a scrub ok 2024-04-14T15:28:12.624 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:12 smithi092 ceph-mon[33966]: pgmap v1358: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 582 KiB data, 833 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:12.923 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd reweight 6 1 2024-04-14T15:28:13.174 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:28:13.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:13 smithi090 ceph-mon[30399]: from='client.50098 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-04-14T15:28:13.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:13 smithi090 ceph-mon[30399]: osdmap e901: 8 total, 8 up, 8 in 2024-04-14T15:28:13.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:13 smithi090 ceph-mon[30399]: 11.7 deep-scrub starts 2024-04-14T15:28:13.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:13 smithi090 ceph-mon[30399]: 11.7 deep-scrub ok 2024-04-14T15:28:13.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:13 smithi090 ceph-mon[41670]: from='client.50098 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-04-14T15:28:13.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:13 smithi090 ceph-mon[41670]: osdmap e901: 8 total, 8 up, 8 in 2024-04-14T15:28:13.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:13 smithi090 ceph-mon[41670]: 11.7 deep-scrub starts 2024-04-14T15:28:13.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:13 smithi090 ceph-mon[41670]: 11.7 deep-scrub ok 2024-04-14T15:28:13.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:13 smithi092 ceph-mon[33966]: from='client.50098 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-04-14T15:28:13.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:13 smithi092 ceph-mon[33966]: osdmap e901: 8 total, 8 up, 8 in 2024-04-14T15:28:13.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:13 smithi092 ceph-mon[33966]: 11.7 deep-scrub starts 2024-04-14T15:28:13.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:13 smithi092 ceph-mon[33966]: 11.7 deep-scrub ok 2024-04-14T15:28:14.441 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:28:13 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:28:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:28:14.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:14 smithi090 ceph-mon[41670]: 2.5 scrub starts 2024-04-14T15:28:14.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:14 smithi090 ceph-mon[41670]: 2.5 scrub ok 2024-04-14T15:28:14.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:14 smithi090 ceph-mon[41670]: pgmap v1360: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 582 KiB data, 833 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:14.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:14 smithi090 ceph-mon[30399]: 2.5 scrub starts 2024-04-14T15:28:14.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:14 smithi090 ceph-mon[30399]: 2.5 scrub ok 2024-04-14T15:28:14.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:14 smithi090 ceph-mon[30399]: pgmap v1360: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 582 KiB data, 833 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:14.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:14 smithi092 ceph-mon[33966]: 2.5 scrub starts 2024-04-14T15:28:14.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:14 smithi092 ceph-mon[33966]: 2.5 scrub ok 2024-04-14T15:28:14.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:14 smithi092 ceph-mon[33966]: pgmap v1360: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 582 KiB data, 833 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:15.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:15.259+0000 7f06a8db1700 1 -- 172.21.15.90:0/920896028 >> v1:172.21.15.90:6789/0 conn(0x7f06a4108870 legacy=0x7f06a4108c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:15.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:15.260+0000 7f06a8db1700 1 -- 172.21.15.90:0/920896028 shutdown_connections 2024-04-14T15:28:15.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:15.260+0000 7f06a8db1700 1 -- 172.21.15.90:0/920896028 >> 172.21.15.90:0/920896028 conn(0x7f06a40fe3c0 msgr2=0x7f06a41007e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:28:15.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:15.260+0000 7f06a8db1700 1 -- 172.21.15.90:0/920896028 shutdown_connections 2024-04-14T15:28:15.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:15.260+0000 7f06a8db1700 1 -- 172.21.15.90:0/920896028 wait complete. 2024-04-14T15:28:15.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:15.260+0000 7f06a8db1700 1 librados: starting msgr at 2024-04-14T15:28:15.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:15.260+0000 7f06a8db1700 1 librados: starting objecter 2024-04-14T15:28:15.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:15.260+0000 7f06a8db1700 1 Processor -- start 2024-04-14T15:28:15.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:15.261+0000 7f06a8db1700 1 -- start start 2024-04-14T15:28:15.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:15.261+0000 7f06a8db1700 1 librados: setting wanted keys 2024-04-14T15:28:15.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:15.261+0000 7f06a8db1700 1 librados: calling monclient init 2024-04-14T15:28:15.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:15.261+0000 7f06a8db1700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f06a4076a20 con 0x7f06a410f2f0 2024-04-14T15:28:15.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:15.261+0000 7f06a8db1700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f06a4076c50 con 0x7f06a4108870 2024-04-14T15:28:15.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:15.261+0000 7f06a8db1700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f06a41ac300 con 0x7f06a410b720 2024-04-14T15:28:15.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:15.261+0000 7f06a1d9b700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f06a410b720 0x7f06a40753f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:34710/0 (socket says 172.21.15.90:34710) 2024-04-14T15:28:15.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:15.261+0000 7f06a1d9b700 1 -- 172.21.15.90:0/3843761821 learned_addr learned my addr 172.21.15.90:0/3843761821 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:28:15.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:15.261+0000 7f06937fe700 10 client.?.objecter ms_handle_connect 0x7f06a410f2f0 2024-04-14T15:28:15.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:15.261+0000 7f06937fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:28:15.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:15.261+0000 7f06937fe700 10 client.?.objecter ms_handle_connect 0x7f06a410b720 2024-04-14T15:28:15.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:15.261+0000 7f06937fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:28:15.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:15.261+0000 7f06937fe700 10 client.?.objecter ms_handle_connect 0x7f06a4108870 2024-04-14T15:28:15.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:15.261+0000 7f06937fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:28:15.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:15.261+0000 7f06937fe700 1 -- 172.21.15.90:0/3843761821 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2433998057 0 0) 0x7f06a4076a20 con 0x7f06a410f2f0 2024-04-14T15:28:15.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:15.261+0000 7f06937fe700 1 -- 172.21.15.90:0/3843761821 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f068c0035b0 con 0x7f06a410f2f0 2024-04-14T15:28:15.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:15.261+0000 7f06937fe700 1 -- 172.21.15.90:0/3843761821 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1519917292 0 0) 0x7f06a4076c50 con 0x7f06a4108870 2024-04-14T15:28:15.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:15.261+0000 7f06937fe700 1 -- 172.21.15.90:0/3843761821 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f06a4076a20 con 0x7f06a4108870 2024-04-14T15:28:15.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:15.261+0000 7f06937fe700 1 -- 172.21.15.90:0/3843761821 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4286517877 0 0) 0x7f06a41ac300 con 0x7f06a410b720 2024-04-14T15:28:15.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:15.261+0000 7f06937fe700 1 -- 172.21.15.90:0/3843761821 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f06a4076c50 con 0x7f06a410b720 2024-04-14T15:28:15.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:15.261+0000 7f06937fe700 1 -- 172.21.15.90:0/3843761821 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1048368612 0 0) 0x7f06a4076a20 con 0x7f06a4108870 2024-04-14T15:28:15.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:15.261+0000 7f06937fe700 1 -- 172.21.15.90:0/3843761821 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f06a41ac300 con 0x7f06a4108870 2024-04-14T15:28:15.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:15.261+0000 7f06937fe700 1 -- 172.21.15.90:0/3843761821 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f06940015d0 con 0x7f06a4108870 2024-04-14T15:28:15.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:15.262+0000 7f06937fe700 1 -- 172.21.15.90:0/3843761821 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4288014821 0 0) 0x7f068c0035b0 con 0x7f06a410f2f0 2024-04-14T15:28:15.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:15.262+0000 7f06937fe700 1 -- 172.21.15.90:0/3843761821 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f06a4076a20 con 0x7f06a410f2f0 2024-04-14T15:28:15.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:15.262+0000 7f06937fe700 1 -- 172.21.15.90:0/3843761821 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2827063028 0 0) 0x7f06a4076c50 con 0x7f06a410b720 2024-04-14T15:28:15.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:15.262+0000 7f06937fe700 1 -- 172.21.15.90:0/3843761821 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f068c0035b0 con 0x7f06a410b720 2024-04-14T15:28:15.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:15.262+0000 7f06937fe700 1 -- 172.21.15.90:0/3843761821 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f069c001bb0 con 0x7f06a410f2f0 2024-04-14T15:28:15.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:15.262+0000 7f06937fe700 1 -- 172.21.15.90:0/3843761821 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f06980017b0 con 0x7f06a410b720 2024-04-14T15:28:15.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:15.262+0000 7f06937fe700 1 -- 172.21.15.90:0/3843761821 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 121844133 0 0) 0x7f06a41ac300 con 0x7f06a4108870 2024-04-14T15:28:15.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:15.262+0000 7f06937fe700 1 -- 172.21.15.90:0/3843761821 >> v1:172.21.15.90:6790/0 conn(0x7f06a410b720 legacy=0x7f06a40753f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:15.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:15.262+0000 7f06937fe700 1 -- 172.21.15.90:0/3843761821 >> v1:172.21.15.90:6789/0 conn(0x7f06a410f2f0 legacy=0x7f06a41a8bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:15.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:15.262+0000 7f06937fe700 1 -- 172.21.15.90:0/3843761821 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06a41ad4e0 con 0x7f06a4108870 2024-04-14T15:28:15.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:15.262+0000 7f06a8db1700 1 -- 172.21.15.90:0/3843761821 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f06a41ac530 con 0x7f06a4108870 2024-04-14T15:28:15.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:15.262+0000 7f06a8db1700 10 client.50104.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:28:15.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:15.262+0000 7f06a8db1700 1 -- 172.21.15.90:0/3843761821 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f06a41ac9f0 con 0x7f06a4108870 2024-04-14T15:28:15.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:15.262+0000 7f06a8db1700 1 librados: init done 2024-04-14T15:28:15.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:15.262+0000 7f06937fe700 1 -- 172.21.15.90:0/3843761821 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f0694003740 con 0x7f06a4108870 2024-04-14T15:28:15.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:15.262+0000 7f06937fe700 1 -- 172.21.15.90:0/3843761821 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f0694004e50 con 0x7f06a4108870 2024-04-14T15:28:15.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:15.263+0000 7f06a8db1700 1 -- 172.21.15.90:0/3843761821 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0684005120 con 0x7f06a4108870 2024-04-14T15:28:15.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:15.264+0000 7f06937fe700 1 -- 172.21.15.90:0/3843761821 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f0694001b70 con 0x7f06a4108870 2024-04-14T15:28:15.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:15.264+0000 7f06937fe700 1 -- 172.21.15.90:0/3843761821 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(901..901 src has 467..901) ==== 7550+0+0 (unknown 2225683945 0 0) 0x7f0694089550 con 0x7f06a4108870 2024-04-14T15:28:15.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:15.264+0000 7f06937fe700 10 client.50104.objecter ms_dispatch 0x7f06a4001550 osd_map(901..901 src has 467..901) 2024-04-14T15:28:15.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:15.264+0000 7f06937fe700 3 client.50104.objecter handle_osd_map got epochs [901,901] > 0 2024-04-14T15:28:15.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:15.264+0000 7f06937fe700 3 client.50104.objecter handle_osd_map decoding full epoch 901 2024-04-14T15:28:15.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:15.264+0000 7f06937fe700 20 client.50104.objecter dump_active .. 0 homeless 2024-04-14T15:28:15.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:15.264+0000 7f06937fe700 10 client.50104.objecter ms_handle_connect 0x7f068c071420 2024-04-14T15:28:15.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:15.269+0000 7f06937fe700 1 -- 172.21.15.90:0/3843761821 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f06940561d0 con 0x7f06a4108870 2024-04-14T15:28:15.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:15 smithi090 ceph-mon[30399]: 1.a scrub starts 2024-04-14T15:28:15.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:15 smithi090 ceph-mon[30399]: 1.a scrub ok 2024-04-14T15:28:15.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:15 smithi090 ceph-mon[30399]: 4.1e scrub starts 2024-04-14T15:28:15.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:15 smithi090 ceph-mon[30399]: 4.1e scrub ok 2024-04-14T15:28:15.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:15 smithi090 ceph-mon[30399]: 6.7 deep-scrub starts 2024-04-14T15:28:15.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:15 smithi090 ceph-mon[30399]: 6.7 deep-scrub ok 2024-04-14T15:28:15.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:15 smithi090 ceph-mon[41670]: 1.a scrub starts 2024-04-14T15:28:15.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:15 smithi090 ceph-mon[41670]: 1.a scrub ok 2024-04-14T15:28:15.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:15 smithi090 ceph-mon[41670]: 4.1e scrub starts 2024-04-14T15:28:15.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:15 smithi090 ceph-mon[41670]: 4.1e scrub ok 2024-04-14T15:28:15.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:15 smithi090 ceph-mon[41670]: 6.7 deep-scrub starts 2024-04-14T15:28:15.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:15 smithi090 ceph-mon[41670]: 6.7 deep-scrub ok 2024-04-14T15:28:15.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:15.481+0000 7f06a8db1700 1 -- 172.21.15.90:0/3843761821 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd reweight", "id": 6, "weight": 1.0} v 0) -- 0x7f0684005410 con 0x7f06a4108870 2024-04-14T15:28:15.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:15 smithi092 ceph-mon[33966]: 1.a scrub starts 2024-04-14T15:28:15.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:15 smithi092 ceph-mon[33966]: 1.a scrub ok 2024-04-14T15:28:15.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:15 smithi092 ceph-mon[33966]: 4.1e scrub starts 2024-04-14T15:28:15.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:15 smithi092 ceph-mon[33966]: 4.1e scrub ok 2024-04-14T15:28:15.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:15 smithi092 ceph-mon[33966]: 6.7 deep-scrub starts 2024-04-14T15:28:15.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:15 smithi092 ceph-mon[33966]: 6.7 deep-scrub ok 2024-04-14T15:28:16.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:16.063+0000 7f06937fe700 1 -- 172.21.15.90:0/3843761821 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 6, "weight": 1.0}]=0 reweighted osd.6 to 1 (10000) v902) ==== 113+0+0 (unknown 206504564 0 0) 0x7f0694059e80 con 0x7f06a4108870 2024-04-14T15:28:16.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:16.067+0000 7f06a8db1700 10 librados: watch_flush enter 2024-04-14T15:28:16.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:16.067+0000 7f06a8db1700 10 librados: watch_flush exit 2024-04-14T15:28:16.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:16.067+0000 7f06a8db1700 20 client.50104.objecter shutdown clearing up homeless session... 2024-04-14T15:28:16.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:16.067+0000 7f06a8db1700 10 client.50104.objecter successfully canceled tick 2024-04-14T15:28:16.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:16.068+0000 7f06a8db1700 1 -- 172.21.15.90:0/3843761821 >> v1:172.21.15.90:6800/420935427 conn(0x7f068c071420 legacy=0x7f068c0738e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:16.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:16.068+0000 7f06a8db1700 1 -- 172.21.15.90:0/3843761821 >> v1:172.21.15.92:6789/0 conn(0x7f06a4108870 legacy=0x7f06a4074ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:16.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:16.068+0000 7f06a8db1700 1 -- 172.21.15.90:0/3843761821 shutdown_connections 2024-04-14T15:28:16.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:16.068+0000 7f06a8db1700 1 -- 172.21.15.90:0/3843761821 >> 172.21.15.90:0/3843761821 conn(0x7f06a40fe3c0 msgr2=0x7f06a4112880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:28:16.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:16.068+0000 7f06a8db1700 1 -- 172.21.15.90:0/3843761821 shutdown_connections 2024-04-14T15:28:16.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:16.068+0000 7f06a8db1700 1 -- 172.21.15.90:0/3843761821 wait complete. 2024-04-14T15:28:16.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:16.068+0000 7f06a8db1700 1 librados: shutdown 2024-04-14T15:28:16.069 INFO:teuthology.orchestra.run.smithi090.stderr:reweighted osd.6 to 1 (10000) 2024-04-14T15:28:16.122 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:28:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:28:16.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:16 smithi090 ceph-mon[41670]: 5.1a scrub starts 2024-04-14T15:28:16.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:16 smithi090 ceph-mon[41670]: 5.1a scrub ok 2024-04-14T15:28:16.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:16 smithi090 ceph-mon[41670]: 6.0 scrub starts 2024-04-14T15:28:16.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:16 smithi090 ceph-mon[41670]: 6.0 scrub ok 2024-04-14T15:28:16.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:16 smithi090 ceph-mon[41670]: 6.1 scrub starts 2024-04-14T15:28:16.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:16 smithi090 ceph-mon[41670]: 6.1 scrub ok 2024-04-14T15:28:16.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:16 smithi090 ceph-mon[41670]: 2.1e scrub starts 2024-04-14T15:28:16.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:16 smithi090 ceph-mon[41670]: 2.1e scrub ok 2024-04-14T15:28:16.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:16 smithi090 ceph-mon[41670]: pgmap v1361: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 582 KiB data, 834 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-14T15:28:16.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:16 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3843761821' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-14T15:28:16.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:16 smithi090 ceph-mon[41670]: from='client.50104 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-14T15:28:16.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:16 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:28:16.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:16 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:28:16.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:16 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:28:16.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:16 smithi090 ceph-mon[41670]: from='client.50104 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-14T15:28:16.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:16 smithi090 ceph-mon[41670]: osdmap e902: 8 total, 8 up, 8 in 2024-04-14T15:28:16.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:16 smithi090 ceph-mon[30399]: 5.1a scrub starts 2024-04-14T15:28:16.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:16 smithi090 ceph-mon[30399]: 5.1a scrub ok 2024-04-14T15:28:16.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:16 smithi090 ceph-mon[30399]: 6.0 scrub starts 2024-04-14T15:28:16.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:16 smithi090 ceph-mon[30399]: 6.0 scrub ok 2024-04-14T15:28:16.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:16 smithi090 ceph-mon[30399]: 6.1 scrub starts 2024-04-14T15:28:16.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:16 smithi090 ceph-mon[30399]: 6.1 scrub ok 2024-04-14T15:28:16.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:16 smithi090 ceph-mon[30399]: 2.1e scrub starts 2024-04-14T15:28:16.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:16 smithi090 ceph-mon[30399]: 2.1e scrub ok 2024-04-14T15:28:16.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:16 smithi090 ceph-mon[30399]: pgmap v1361: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 582 KiB data, 834 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-14T15:28:16.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:16 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3843761821' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-14T15:28:16.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:16 smithi090 ceph-mon[30399]: from='client.50104 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-14T15:28:16.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:16 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:28:16.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:16 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:28:16.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:16 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:28:16.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:16 smithi090 ceph-mon[30399]: from='client.50104 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-14T15:28:16.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:16 smithi090 ceph-mon[30399]: osdmap e902: 8 total, 8 up, 8 in 2024-04-14T15:28:16.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:16 smithi092 ceph-mon[33966]: 5.1a scrub starts 2024-04-14T15:28:16.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:16 smithi092 ceph-mon[33966]: 5.1a scrub ok 2024-04-14T15:28:16.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:16 smithi092 ceph-mon[33966]: 6.0 scrub starts 2024-04-14T15:28:16.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:16 smithi092 ceph-mon[33966]: 6.0 scrub ok 2024-04-14T15:28:16.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:16 smithi092 ceph-mon[33966]: 6.1 scrub starts 2024-04-14T15:28:16.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:16 smithi092 ceph-mon[33966]: 6.1 scrub ok 2024-04-14T15:28:16.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:16 smithi092 ceph-mon[33966]: 2.1e scrub starts 2024-04-14T15:28:16.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:16 smithi092 ceph-mon[33966]: 2.1e scrub ok 2024-04-14T15:28:16.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:16 smithi092 ceph-mon[33966]: pgmap v1361: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 582 KiB data, 834 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-14T15:28:16.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:16 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3843761821' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-14T15:28:16.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:16 smithi092 ceph-mon[33966]: from='client.50104 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-14T15:28:16.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:16 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:28:16.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:16 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:28:16.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:16 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:28:16.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:16 smithi092 ceph-mon[33966]: from='client.50104 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-14T15:28:16.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:16 smithi092 ceph-mon[33966]: osdmap e902: 8 total, 8 up, 8 in 2024-04-14T15:28:17.015 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd primary-affinity 6 1 2024-04-14T15:28:17.266 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:28:17.387 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:17 smithi090 ceph-mon[41670]: 4.17 scrub starts 2024-04-14T15:28:17.388 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:17 smithi090 ceph-mon[41670]: 4.17 scrub ok 2024-04-14T15:28:17.388 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:17 smithi090 ceph-mon[41670]: osdmap e903: 8 total, 8 up, 8 in 2024-04-14T15:28:17.388 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:17 smithi090 ceph-mon[30399]: 4.17 scrub starts 2024-04-14T15:28:17.388 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:17 smithi090 ceph-mon[30399]: 4.17 scrub ok 2024-04-14T15:28:17.388 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:17 smithi090 ceph-mon[30399]: osdmap e903: 8 total, 8 up, 8 in 2024-04-14T15:28:17.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:17 smithi092 ceph-mon[33966]: 4.17 scrub starts 2024-04-14T15:28:17.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:17 smithi092 ceph-mon[33966]: 4.17 scrub ok 2024-04-14T15:28:17.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:17 smithi092 ceph-mon[33966]: osdmap e903: 8 total, 8 up, 8 in 2024-04-14T15:28:18.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:18 smithi090 ceph-mon[41670]: 11.a scrub starts 2024-04-14T15:28:18.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:18 smithi090 ceph-mon[41670]: 11.a scrub ok 2024-04-14T15:28:18.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:18 smithi090 ceph-mon[41670]: pgmap v1364: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 582 KiB data, 834 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:18.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:18 smithi090 ceph-mon[30399]: 11.a scrub starts 2024-04-14T15:28:18.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:18 smithi090 ceph-mon[30399]: 11.a scrub ok 2024-04-14T15:28:18.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:18 smithi090 ceph-mon[30399]: pgmap v1364: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 582 KiB data, 834 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:18.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:18 smithi092 ceph-mon[33966]: 11.a scrub starts 2024-04-14T15:28:18.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:18 smithi092 ceph-mon[33966]: 11.a scrub ok 2024-04-14T15:28:18.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:18 smithi092 ceph-mon[33966]: pgmap v1364: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 582 KiB data, 834 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:19.407 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:19 smithi090 ceph-mon[41670]: 6.1b deep-scrub starts 2024-04-14T15:28:19.407 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:19 smithi090 ceph-mon[41670]: 6.1b deep-scrub ok 2024-04-14T15:28:19.407 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:19 smithi090 ceph-mon[30399]: 6.1b deep-scrub starts 2024-04-14T15:28:19.407 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:19 smithi090 ceph-mon[30399]: 6.1b deep-scrub ok 2024-04-14T15:28:19.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:19 smithi092 ceph-mon[33966]: 6.1b deep-scrub starts 2024-04-14T15:28:19.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:19 smithi092 ceph-mon[33966]: 6.1b deep-scrub ok 2024-04-14T15:28:19.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:19.633+0000 7f04278b7700 1 -- 172.21.15.90:0/2733354196 >> v1:172.21.15.90:6790/0 conn(0x7f042010e1f0 legacy=0x7f04201106b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:19.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:19.633+0000 7f04278b7700 1 -- 172.21.15.90:0/2733354196 shutdown_connections 2024-04-14T15:28:19.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:19.633+0000 7f04278b7700 1 -- 172.21.15.90:0/2733354196 >> 172.21.15.90:0/2733354196 conn(0x7f04200fd280 msgr2=0x7f04200ff6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:28:19.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:19.633+0000 7f04278b7700 1 -- 172.21.15.90:0/2733354196 shutdown_connections 2024-04-14T15:28:19.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:19.633+0000 7f04278b7700 1 -- 172.21.15.90:0/2733354196 wait complete. 2024-04-14T15:28:19.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:19.633+0000 7f04278b7700 1 librados: starting msgr at 2024-04-14T15:28:19.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:19.633+0000 7f04278b7700 1 librados: starting objecter 2024-04-14T15:28:19.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:19.633+0000 7f04278b7700 1 Processor -- start 2024-04-14T15:28:19.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:19.633+0000 7f04278b7700 1 -- start start 2024-04-14T15:28:19.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:19.633+0000 7f04278b7700 1 librados: setting wanted keys 2024-04-14T15:28:19.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:19.633+0000 7f04278b7700 1 librados: calling monclient init 2024-04-14T15:28:19.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:19.634+0000 7f04278b7700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f04201a7880 con 0x7f0420107770 2024-04-14T15:28:19.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:19.634+0000 7f04278b7700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f04201a8ac0 con 0x7f042010e1f0 2024-04-14T15:28:19.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:19.634+0000 7f04278b7700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f04201a9d00 con 0x7f042010a620 2024-04-14T15:28:19.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:19.634+0000 7f0425653700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f0420107770 0x7f042010d990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50122/0 (socket says 172.21.15.90:50122) 2024-04-14T15:28:19.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:19.634+0000 7f0425653700 1 -- 172.21.15.90:0/2927560287 learned_addr learned my addr 172.21.15.90:0/2927560287 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:28:19.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:19.634+0000 7f04167fc700 10 client.?.objecter ms_handle_connect 0x7f0420107770 2024-04-14T15:28:19.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:19.634+0000 7f04167fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:28:19.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:19.634+0000 7f04167fc700 10 client.?.objecter ms_handle_connect 0x7f042010a620 2024-04-14T15:28:19.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:19.634+0000 7f04167fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:28:19.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:19.634+0000 7f04167fc700 10 client.?.objecter ms_handle_connect 0x7f042010e1f0 2024-04-14T15:28:19.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:19.634+0000 7f04167fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:28:19.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:19.634+0000 7f04167fc700 1 -- 172.21.15.90:0/2927560287 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4087856774 0 0) 0x7f04201a7880 con 0x7f0420107770 2024-04-14T15:28:19.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:19.634+0000 7f04167fc700 1 -- 172.21.15.90:0/2927560287 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f040c003aa0 con 0x7f0420107770 2024-04-14T15:28:19.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:19.634+0000 7f04167fc700 1 -- 172.21.15.90:0/2927560287 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3300260406 0 0) 0x7f04201a9d00 con 0x7f042010a620 2024-04-14T15:28:19.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:19.634+0000 7f04167fc700 1 -- 172.21.15.90:0/2927560287 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f04201a7880 con 0x7f042010a620 2024-04-14T15:28:19.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:19.634+0000 7f04167fc700 1 -- 172.21.15.90:0/2927560287 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 561466989 0 0) 0x7f04201a8ac0 con 0x7f042010e1f0 2024-04-14T15:28:19.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:19.634+0000 7f04167fc700 1 -- 172.21.15.90:0/2927560287 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f04201a9d00 con 0x7f042010e1f0 2024-04-14T15:28:19.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:19.634+0000 7f04167fc700 1 -- 172.21.15.90:0/2927560287 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 777580401 0 0) 0x7f040c003aa0 con 0x7f0420107770 2024-04-14T15:28:19.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:19.634+0000 7f04167fc700 1 -- 172.21.15.90:0/2927560287 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f04201a8ac0 con 0x7f0420107770 2024-04-14T15:28:19.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:19.634+0000 7f04167fc700 1 -- 172.21.15.90:0/2927560287 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f041c001a30 con 0x7f0420107770 2024-04-14T15:28:19.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:19.634+0000 7f04167fc700 1 -- 172.21.15.90:0/2927560287 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3661525886 0 0) 0x7f04201a7880 con 0x7f042010a620 2024-04-14T15:28:19.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:19.634+0000 7f04167fc700 1 -- 172.21.15.90:0/2927560287 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f040c003aa0 con 0x7f042010a620 2024-04-14T15:28:19.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:19.634+0000 7f04167fc700 1 -- 172.21.15.90:0/2927560287 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f0410001830 con 0x7f042010a620 2024-04-14T15:28:19.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:19.634+0000 7f04167fc700 1 -- 172.21.15.90:0/2927560287 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 11351257 0 0) 0x7f04201a9d00 con 0x7f042010e1f0 2024-04-14T15:28:19.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:19.635+0000 7f04167fc700 1 -- 172.21.15.90:0/2927560287 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f04201a7880 con 0x7f042010e1f0 2024-04-14T15:28:19.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:19.635+0000 7f04167fc700 1 -- 172.21.15.90:0/2927560287 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f0418004880 con 0x7f042010e1f0 2024-04-14T15:28:19.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:19.635+0000 7f04167fc700 1 -- 172.21.15.90:0/2927560287 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4006687455 0 0) 0x7f04201a8ac0 con 0x7f0420107770 2024-04-14T15:28:19.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:19.635+0000 7f04167fc700 1 -- 172.21.15.90:0/2927560287 >> v1:172.21.15.90:6790/0 conn(0x7f042010a620 legacy=0x7f04201a26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:19.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:19.635+0000 7f04167fc700 1 -- 172.21.15.90:0/2927560287 >> v1:172.21.15.92:6789/0 conn(0x7f042010e1f0 legacy=0x7f04201a5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:19.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:19.635+0000 7f04167fc700 1 -- 172.21.15.90:0/2927560287 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04201aaf40 con 0x7f0420107770 2024-04-14T15:28:19.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:19.635+0000 7f04278b7700 1 -- 172.21.15.90:0/2927560287 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f04201a9f30 con 0x7f0420107770 2024-04-14T15:28:19.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:19.635+0000 7f04278b7700 10 client.38289.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:28:19.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:19.635+0000 7f04278b7700 1 -- 172.21.15.90:0/2927560287 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f04201aa3f0 con 0x7f0420107770 2024-04-14T15:28:19.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:19.635+0000 7f04278b7700 1 librados: init done 2024-04-14T15:28:19.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:19.635+0000 7f04167fc700 1 -- 172.21.15.90:0/2927560287 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f041c001590 con 0x7f0420107770 2024-04-14T15:28:19.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:19.635+0000 7f04167fc700 1 -- 172.21.15.90:0/2927560287 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f041c005cd0 con 0x7f0420107770 2024-04-14T15:28:19.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:19.636+0000 7f04278b7700 1 -- 172.21.15.90:0/2927560287 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0404005150 con 0x7f0420107770 2024-04-14T15:28:19.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:19.636+0000 7f04167fc700 1 -- 172.21.15.90:0/2927560287 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f041c003930 con 0x7f0420107770 2024-04-14T15:28:19.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:19.637+0000 7f04167fc700 1 -- 172.21.15.90:0/2927560287 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(903..903 src has 467..903) ==== 7517+0+0 (unknown 2895349629 0 0) 0x7f041c08a7b0 con 0x7f0420107770 2024-04-14T15:28:19.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:19.637+0000 7f04167fc700 10 client.38289.objecter ms_dispatch 0x7f0420001550 osd_map(903..903 src has 467..903) 2024-04-14T15:28:19.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:19.637+0000 7f04167fc700 3 client.38289.objecter handle_osd_map got epochs [903,903] > 0 2024-04-14T15:28:19.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:19.637+0000 7f04167fc700 3 client.38289.objecter handle_osd_map decoding full epoch 903 2024-04-14T15:28:19.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:19.637+0000 7f04167fc700 20 client.38289.objecter dump_active .. 0 homeless 2024-04-14T15:28:19.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:19.637+0000 7f04167fc700 10 client.38289.objecter ms_handle_connect 0x7f040c071430 2024-04-14T15:28:19.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:19.641+0000 7f04167fc700 1 -- 172.21.15.90:0/2927560287 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f041c057490 con 0x7f0420107770 2024-04-14T15:28:19.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:19.854+0000 7f04278b7700 1 -- 172.21.15.90:0/2927560287 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} v 0) -- 0x7f0404005c50 con 0x7f0420107770 2024-04-14T15:28:20.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:20.181+0000 7f04167fc700 1 -- 172.21.15.90:0/2927560287 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]=0 set osd.6 primary-affinity to 1 (8655362) v904) ==== 133+0+0 (unknown 1509650549 0 0) 0x7f041c05b140 con 0x7f0420107770 2024-04-14T15:28:20.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:20.185+0000 7f04278b7700 10 librados: watch_flush enter 2024-04-14T15:28:20.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:20.185+0000 7f04278b7700 10 librados: watch_flush exit 2024-04-14T15:28:20.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:20.185+0000 7f04278b7700 20 client.38289.objecter shutdown clearing up homeless session... 2024-04-14T15:28:20.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:20.185+0000 7f04278b7700 10 client.38289.objecter successfully canceled tick 2024-04-14T15:28:20.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:20.185+0000 7f04278b7700 1 -- 172.21.15.90:0/2927560287 >> v1:172.21.15.90:6800/420935427 conn(0x7f040c071430 legacy=0x7f040c0738f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:20.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:20.185+0000 7f04278b7700 1 -- 172.21.15.90:0/2927560287 >> v1:172.21.15.90:6789/0 conn(0x7f0420107770 legacy=0x7f042010d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:20.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:20.186+0000 7f04278b7700 1 -- 172.21.15.90:0/2927560287 shutdown_connections 2024-04-14T15:28:20.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:20.186+0000 7f04278b7700 1 -- 172.21.15.90:0/2927560287 >> 172.21.15.90:0/2927560287 conn(0x7f04200fd280 msgr2=0x7f0420110e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:28:20.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:20.186+0000 7f04278b7700 1 -- 172.21.15.90:0/2927560287 shutdown_connections 2024-04-14T15:28:20.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:20.186+0000 7f04278b7700 1 -- 172.21.15.90:0/2927560287 wait complete. 2024-04-14T15:28:20.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:20.186+0000 7f04278b7700 1 librados: shutdown 2024-04-14T15:28:20.188 INFO:teuthology.orchestra.run.smithi090.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-04-14T15:28:20.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:20 smithi090 ceph-mon[30399]: 4.6 scrub starts 2024-04-14T15:28:20.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:20 smithi090 ceph-mon[30399]: 4.6 scrub ok 2024-04-14T15:28:20.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:20 smithi090 ceph-mon[30399]: 6.4 scrub starts 2024-04-14T15:28:20.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:20 smithi090 ceph-mon[30399]: 6.4 scrub ok 2024-04-14T15:28:20.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:20 smithi090 ceph-mon[30399]: pgmap v1365: 178 pgs: 178 active+clean; 582 KiB data, 834 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-14T15:28:20.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:20 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-04-14T15:28:20.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:20 smithi090 ceph-mon[30399]: 1.6 scrub starts 2024-04-14T15:28:20.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:20 smithi090 ceph-mon[30399]: 1.6 scrub ok 2024-04-14T15:28:20.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:20 smithi090 ceph-mon[30399]: 1.1 scrub starts 2024-04-14T15:28:20.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:20 smithi090 ceph-mon[30399]: 1.1 scrub ok 2024-04-14T15:28:20.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:20 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2927560287' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-04-14T15:28:20.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:20 smithi090 ceph-mon[41670]: 4.6 scrub starts 2024-04-14T15:28:20.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:20 smithi090 ceph-mon[41670]: 4.6 scrub ok 2024-04-14T15:28:20.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:20 smithi090 ceph-mon[41670]: 6.4 scrub starts 2024-04-14T15:28:20.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:20 smithi090 ceph-mon[41670]: 6.4 scrub ok 2024-04-14T15:28:20.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:20 smithi090 ceph-mon[41670]: pgmap v1365: 178 pgs: 178 active+clean; 582 KiB data, 834 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-14T15:28:20.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:20 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-04-14T15:28:20.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:20 smithi090 ceph-mon[41670]: 1.6 scrub starts 2024-04-14T15:28:20.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:20 smithi090 ceph-mon[41670]: 1.6 scrub ok 2024-04-14T15:28:20.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:20 smithi090 ceph-mon[41670]: 1.1 scrub starts 2024-04-14T15:28:20.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:20 smithi090 ceph-mon[41670]: 1.1 scrub ok 2024-04-14T15:28:20.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:20 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2927560287' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-04-14T15:28:20.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:20 smithi092 ceph-mon[33966]: 4.6 scrub starts 2024-04-14T15:28:20.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:20 smithi092 ceph-mon[33966]: 4.6 scrub ok 2024-04-14T15:28:20.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:20 smithi092 ceph-mon[33966]: 6.4 scrub starts 2024-04-14T15:28:20.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:20 smithi092 ceph-mon[33966]: 6.4 scrub ok 2024-04-14T15:28:20.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:20 smithi092 ceph-mon[33966]: pgmap v1365: 178 pgs: 178 active+clean; 582 KiB data, 834 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-14T15:28:20.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:20 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-04-14T15:28:20.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:20 smithi092 ceph-mon[33966]: 1.6 scrub starts 2024-04-14T15:28:20.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:20 smithi092 ceph-mon[33966]: 1.6 scrub ok 2024-04-14T15:28:20.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:20 smithi092 ceph-mon[33966]: 1.1 scrub starts 2024-04-14T15:28:20.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:20 smithi092 ceph-mon[33966]: 1.1 scrub ok 2024-04-14T15:28:20.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:20 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2927560287' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-04-14T15:28:21.082 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd reweight 7 1 2024-04-14T15:28:21.298 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:21 smithi090 ceph-mon[41670]: 4.11 scrub starts 2024-04-14T15:28:21.298 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:21 smithi090 ceph-mon[41670]: 4.11 scrub ok 2024-04-14T15:28:21.298 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:21 smithi090 ceph-mon[41670]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-04-14T15:28:21.298 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:21 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]': finished 2024-04-14T15:28:21.299 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:21 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2927560287' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-04-14T15:28:21.299 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:21 smithi090 ceph-mon[41670]: osdmap e904: 8 total, 8 up, 8 in 2024-04-14T15:28:21.299 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:21 smithi090 ceph-mon[41670]: 4.a deep-scrub starts 2024-04-14T15:28:21.299 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:21 smithi090 ceph-mon[41670]: 4.a deep-scrub ok 2024-04-14T15:28:21.299 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:21 smithi090 ceph-mon[30399]: 4.11 scrub starts 2024-04-14T15:28:21.299 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:21 smithi090 ceph-mon[30399]: 4.11 scrub ok 2024-04-14T15:28:21.299 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:21 smithi090 ceph-mon[30399]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-04-14T15:28:21.299 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:21 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]': finished 2024-04-14T15:28:21.299 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:21 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2927560287' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-04-14T15:28:21.299 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:21 smithi090 ceph-mon[30399]: osdmap e904: 8 total, 8 up, 8 in 2024-04-14T15:28:21.299 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:21 smithi090 ceph-mon[30399]: 4.a deep-scrub starts 2024-04-14T15:28:21.300 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:21 smithi090 ceph-mon[30399]: 4.a deep-scrub ok 2024-04-14T15:28:21.338 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:28:21.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:21 smithi092 ceph-mon[33966]: 4.11 scrub starts 2024-04-14T15:28:21.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:21 smithi092 ceph-mon[33966]: 4.11 scrub ok 2024-04-14T15:28:21.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:21 smithi092 ceph-mon[33966]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-04-14T15:28:21.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:21 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]': finished 2024-04-14T15:28:21.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:21 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2927560287' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-04-14T15:28:21.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:21 smithi092 ceph-mon[33966]: osdmap e904: 8 total, 8 up, 8 in 2024-04-14T15:28:21.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:21 smithi092 ceph-mon[33966]: 4.a deep-scrub starts 2024-04-14T15:28:21.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:21 smithi092 ceph-mon[33966]: 4.a deep-scrub ok 2024-04-14T15:28:22.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:22 smithi090 ceph-mon[30399]: 5.12 scrub starts 2024-04-14T15:28:22.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:22 smithi090 ceph-mon[30399]: 5.12 scrub ok 2024-04-14T15:28:22.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:22 smithi090 ceph-mon[30399]: osdmap e905: 8 total, 8 up, 8 in 2024-04-14T15:28:22.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:22 smithi090 ceph-mon[30399]: pgmap v1368: 178 pgs: 178 active+clean; 582 KiB data, 834 MiB used, 714 GiB / 715 GiB avail; 988 B/s rd, 0 op/s 2024-04-14T15:28:22.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:22 smithi090 ceph-mon[30399]: 5.6 scrub starts 2024-04-14T15:28:22.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:22 smithi090 ceph-mon[30399]: 5.6 scrub ok 2024-04-14T15:28:22.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:22 smithi090 ceph-mon[30399]: 1.8 scrub starts 2024-04-14T15:28:22.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:22 smithi090 ceph-mon[30399]: 1.8 scrub ok 2024-04-14T15:28:22.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:22 smithi090 ceph-mon[41670]: 5.12 scrub starts 2024-04-14T15:28:22.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:22 smithi090 ceph-mon[41670]: 5.12 scrub ok 2024-04-14T15:28:22.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:22 smithi090 ceph-mon[41670]: osdmap e905: 8 total, 8 up, 8 in 2024-04-14T15:28:22.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:22 smithi090 ceph-mon[41670]: pgmap v1368: 178 pgs: 178 active+clean; 582 KiB data, 834 MiB used, 714 GiB / 715 GiB avail; 988 B/s rd, 0 op/s 2024-04-14T15:28:22.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:22 smithi090 ceph-mon[41670]: 5.6 scrub starts 2024-04-14T15:28:22.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:22 smithi090 ceph-mon[41670]: 5.6 scrub ok 2024-04-14T15:28:22.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:22 smithi090 ceph-mon[41670]: 1.8 scrub starts 2024-04-14T15:28:22.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:22 smithi090 ceph-mon[41670]: 1.8 scrub ok 2024-04-14T15:28:22.523 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:22 smithi092 ceph-mon[33966]: 5.12 scrub starts 2024-04-14T15:28:22.523 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:22 smithi092 ceph-mon[33966]: 5.12 scrub ok 2024-04-14T15:28:22.524 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:22 smithi092 ceph-mon[33966]: osdmap e905: 8 total, 8 up, 8 in 2024-04-14T15:28:22.524 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:22 smithi092 ceph-mon[33966]: pgmap v1368: 178 pgs: 178 active+clean; 582 KiB data, 834 MiB used, 714 GiB / 715 GiB avail; 988 B/s rd, 0 op/s 2024-04-14T15:28:22.524 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:22 smithi092 ceph-mon[33966]: 5.6 scrub starts 2024-04-14T15:28:22.524 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:22 smithi092 ceph-mon[33966]: 5.6 scrub ok 2024-04-14T15:28:22.524 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:22 smithi092 ceph-mon[33966]: 1.8 scrub starts 2024-04-14T15:28:22.524 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:22 smithi092 ceph-mon[33966]: 1.8 scrub ok 2024-04-14T15:28:23.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:23.164+0000 7f78cd143700 1 -- 172.21.15.90:0/3373777711 >> v1:172.21.15.90:6789/0 conn(0x7f78c8108890 legacy=0x7f78c8108c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:23.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:23.164+0000 7f78cd143700 1 -- 172.21.15.90:0/3373777711 shutdown_connections 2024-04-14T15:28:23.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:23.164+0000 7f78cd143700 1 -- 172.21.15.90:0/3373777711 >> 172.21.15.90:0/3373777711 conn(0x7f78c80fe380 msgr2=0x7f78c81007a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:28:23.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:23.164+0000 7f78cd143700 1 -- 172.21.15.90:0/3373777711 shutdown_connections 2024-04-14T15:28:23.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:23.164+0000 7f78cd143700 1 -- 172.21.15.90:0/3373777711 wait complete. 2024-04-14T15:28:23.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:23.164+0000 7f78cd143700 1 librados: starting msgr at 2024-04-14T15:28:23.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:23.164+0000 7f78cd143700 1 librados: starting objecter 2024-04-14T15:28:23.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:23.165+0000 7f78cd143700 1 Processor -- start 2024-04-14T15:28:23.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:23.165+0000 7f78cd143700 1 -- start start 2024-04-14T15:28:23.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:23.165+0000 7f78cd143700 1 librados: setting wanted keys 2024-04-14T15:28:23.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:23.165+0000 7f78cd143700 1 librados: calling monclient init 2024-04-14T15:28:23.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:23.165+0000 7f78cd143700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f78c810efd0 con 0x7f78c810f310 2024-04-14T15:28:23.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:23.165+0000 7f78cd143700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f78c81a8f20 con 0x7f78c8108890 2024-04-14T15:28:23.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:23.165+0000 7f78cd143700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f78c81aa100 con 0x7f78c810b740 2024-04-14T15:28:23.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:23.165+0000 7f78c759e700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f78c810f310 0x7f78c81a67f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:45542/0 (socket says 172.21.15.90:45542) 2024-04-14T15:28:23.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:23.165+0000 7f78c759e700 1 -- 172.21.15.90:0/2673384232 learned_addr learned my addr 172.21.15.90:0/2673384232 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:28:23.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:23.165+0000 7f78affff700 10 client.?.objecter ms_handle_connect 0x7f78c810f310 2024-04-14T15:28:23.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:23.165+0000 7f78affff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:28:23.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:23.165+0000 7f78affff700 10 client.?.objecter ms_handle_connect 0x7f78c810b740 2024-04-14T15:28:23.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:23.165+0000 7f78affff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:28:23.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:23.165+0000 7f78affff700 10 client.?.objecter ms_handle_connect 0x7f78c8108890 2024-04-14T15:28:23.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:23.165+0000 7f78affff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:28:23.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:23.165+0000 7f78affff700 1 -- 172.21.15.90:0/2673384232 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3319888203 0 0) 0x7f78c810efd0 con 0x7f78c810f310 2024-04-14T15:28:23.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:23.165+0000 7f78affff700 1 -- 172.21.15.90:0/2673384232 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78b00035b0 con 0x7f78c810f310 2024-04-14T15:28:23.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:23.165+0000 7f78affff700 1 -- 172.21.15.90:0/2673384232 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2956811612 0 0) 0x7f78c81aa100 con 0x7f78c810b740 2024-04-14T15:28:23.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:23.165+0000 7f78affff700 1 -- 172.21.15.90:0/2673384232 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78c810efd0 con 0x7f78c810b740 2024-04-14T15:28:23.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:23.165+0000 7f78affff700 1 -- 172.21.15.90:0/2673384232 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1543689876 0 0) 0x7f78c81a8f20 con 0x7f78c8108890 2024-04-14T15:28:23.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:23.165+0000 7f78affff700 1 -- 172.21.15.90:0/2673384232 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78c81aa100 con 0x7f78c8108890 2024-04-14T15:28:23.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:23.166+0000 7f78affff700 1 -- 172.21.15.90:0/2673384232 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 622642705 0 0) 0x7f78b00035b0 con 0x7f78c810f310 2024-04-14T15:28:23.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:23.166+0000 7f78affff700 1 -- 172.21.15.90:0/2673384232 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f78c81a8f20 con 0x7f78c810f310 2024-04-14T15:28:23.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:23.166+0000 7f78affff700 1 -- 172.21.15.90:0/2673384232 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f78c0001c10 con 0x7f78c810f310 2024-04-14T15:28:23.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:23.166+0000 7f78affff700 1 -- 172.21.15.90:0/2673384232 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3727626390 0 0) 0x7f78c81aa100 con 0x7f78c8108890 2024-04-14T15:28:23.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:23.166+0000 7f78affff700 1 -- 172.21.15.90:0/2673384232 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f78b00035b0 con 0x7f78c8108890 2024-04-14T15:28:23.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:23.166+0000 7f78affff700 1 -- 172.21.15.90:0/2673384232 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1946944268 0 0) 0x7f78c810efd0 con 0x7f78c810b740 2024-04-14T15:28:23.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:23.166+0000 7f78affff700 1 -- 172.21.15.90:0/2673384232 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f78c81aa100 con 0x7f78c810b740 2024-04-14T15:28:23.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:23.166+0000 7f78affff700 1 -- 172.21.15.90:0/2673384232 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f78b8001430 con 0x7f78c8108890 2024-04-14T15:28:23.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:23.166+0000 7f78affff700 1 -- 172.21.15.90:0/2673384232 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f78bc0018b0 con 0x7f78c810b740 2024-04-14T15:28:23.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:23.166+0000 7f78affff700 1 -- 172.21.15.90:0/2673384232 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 977343308 0 0) 0x7f78c81a8f20 con 0x7f78c810f310 2024-04-14T15:28:23.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:23.166+0000 7f78affff700 1 -- 172.21.15.90:0/2673384232 >> v1:172.21.15.90:6790/0 conn(0x7f78c810b740 legacy=0x7f78c810c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:23.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:23.166+0000 7f78affff700 1 -- 172.21.15.90:0/2673384232 >> v1:172.21.15.92:6789/0 conn(0x7f78c8108890 legacy=0x7f78c810be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:23.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:23.166+0000 7f78affff700 1 -- 172.21.15.90:0/2673384232 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78c81ab2e0 con 0x7f78c810f310 2024-04-14T15:28:23.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:23.166+0000 7f78cd143700 1 -- 172.21.15.90:0/2673384232 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f78c81aa330 con 0x7f78c810f310 2024-04-14T15:28:23.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:23.166+0000 7f78cd143700 10 client.38295.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:28:23.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:23.166+0000 7f78cd143700 1 -- 172.21.15.90:0/2673384232 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f78c81aa7f0 con 0x7f78c810f310 2024-04-14T15:28:23.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:23.166+0000 7f78cd143700 1 librados: init done 2024-04-14T15:28:23.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:23.166+0000 7f78affff700 1 -- 172.21.15.90:0/2673384232 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f78c0003ec0 con 0x7f78c810f310 2024-04-14T15:28:23.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:23.166+0000 7f78affff700 1 -- 172.21.15.90:0/2673384232 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f78c0005fb0 con 0x7f78c810f310 2024-04-14T15:28:23.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:23.167+0000 7f78cd143700 1 -- 172.21.15.90:0/2673384232 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78b4005120 con 0x7f78c810f310 2024-04-14T15:28:23.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:23.168+0000 7f78affff700 1 -- 172.21.15.90:0/2673384232 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f78c001cbe0 con 0x7f78c810f310 2024-04-14T15:28:23.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:23.168+0000 7f78affff700 1 -- 172.21.15.90:0/2673384232 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(906..906 src has 467..906) ==== 7517+0+0 (unknown 4219044074 0 0) 0x7f78c008a8c0 con 0x7f78c810f310 2024-04-14T15:28:23.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:23.168+0000 7f78affff700 10 client.38295.objecter ms_dispatch 0x7f78c8001550 osd_map(906..906 src has 467..906) 2024-04-14T15:28:23.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:23.168+0000 7f78affff700 3 client.38295.objecter handle_osd_map got epochs [906,906] > 0 2024-04-14T15:28:23.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:23.168+0000 7f78affff700 3 client.38295.objecter handle_osd_map decoding full epoch 906 2024-04-14T15:28:23.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:23.168+0000 7f78affff700 20 client.38295.objecter dump_active .. 0 homeless 2024-04-14T15:28:23.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:23.168+0000 7f78affff700 10 client.38295.objecter ms_handle_connect 0x7f78b0071300 2024-04-14T15:28:23.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:23.173+0000 7f78affff700 1 -- 172.21.15.90:0/2673384232 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f78c00576a0 con 0x7f78c810f310 2024-04-14T15:28:23.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:23.392+0000 7f78cd143700 1 -- 172.21.15.90:0/2673384232 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd reweight", "id": 7, "weight": 1.0} v 0) -- 0x7f78b4005410 con 0x7f78c810f310 2024-04-14T15:28:23.539 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:23 smithi092 ceph-mon[33966]: 1.9 scrub starts 2024-04-14T15:28:23.539 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:23 smithi092 ceph-mon[33966]: 1.9 scrub ok 2024-04-14T15:28:23.539 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:23 smithi092 ceph-mon[33966]: osdmap e906: 8 total, 8 up, 8 in 2024-04-14T15:28:23.539 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:23 smithi092 ceph-mon[33966]: 6.f scrub starts 2024-04-14T15:28:23.539 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:23 smithi092 ceph-mon[33966]: 6.f scrub ok 2024-04-14T15:28:23.691 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:23 smithi090 ceph-mon[30399]: 1.9 scrub starts 2024-04-14T15:28:23.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:23 smithi090 ceph-mon[30399]: 1.9 scrub ok 2024-04-14T15:28:23.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:23 smithi090 ceph-mon[30399]: osdmap e906: 8 total, 8 up, 8 in 2024-04-14T15:28:23.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:23 smithi090 ceph-mon[30399]: 6.f scrub starts 2024-04-14T15:28:23.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:23 smithi090 ceph-mon[30399]: 6.f scrub ok 2024-04-14T15:28:23.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:23 smithi090 ceph-mon[41670]: 1.9 scrub starts 2024-04-14T15:28:23.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:23 smithi090 ceph-mon[41670]: 1.9 scrub ok 2024-04-14T15:28:23.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:23 smithi090 ceph-mon[41670]: osdmap e906: 8 total, 8 up, 8 in 2024-04-14T15:28:23.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:23 smithi090 ceph-mon[41670]: 6.f scrub starts 2024-04-14T15:28:23.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:23 smithi090 ceph-mon[41670]: 6.f scrub ok 2024-04-14T15:28:24.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:24.217+0000 7f78affff700 1 -- 172.21.15.90:0/2673384232 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 7, "weight": 1.0}]=0 reweighted osd.7 to 1 (10000) v908) ==== 113+0+0 (unknown 2286665342 0 0) 0x7f78c005b350 con 0x7f78c810f310 2024-04-14T15:28:24.218 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:28:23 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:28:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:28:24.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:24.222+0000 7f78cd143700 10 librados: watch_flush enter 2024-04-14T15:28:24.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:24.222+0000 7f78cd143700 10 librados: watch_flush exit 2024-04-14T15:28:24.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:24.222+0000 7f78cd143700 20 client.38295.objecter shutdown clearing up homeless session... 2024-04-14T15:28:24.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:24.222+0000 7f78cd143700 10 client.38295.objecter successfully canceled tick 2024-04-14T15:28:24.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:24.222+0000 7f78cd143700 1 -- 172.21.15.90:0/2673384232 >> v1:172.21.15.90:6800/420935427 conn(0x7f78b0071300 legacy=0x7f78b00737c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:24.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:24.222+0000 7f78cd143700 1 -- 172.21.15.90:0/2673384232 >> v1:172.21.15.90:6789/0 conn(0x7f78c810f310 legacy=0x7f78c81a67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:24.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:24.222+0000 7f78cd143700 1 -- 172.21.15.90:0/2673384232 shutdown_connections 2024-04-14T15:28:24.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:24.222+0000 7f78cd143700 1 -- 172.21.15.90:0/2673384232 >> 172.21.15.90:0/2673384232 conn(0x7f78c80fe380 msgr2=0x7f78c8112870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:28:24.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:24.223+0000 7f78cd143700 1 -- 172.21.15.90:0/2673384232 shutdown_connections 2024-04-14T15:28:24.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:24.223+0000 7f78cd143700 1 -- 172.21.15.90:0/2673384232 wait complete. 2024-04-14T15:28:24.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:24.224+0000 7f78cd143700 1 librados: shutdown 2024-04-14T15:28:24.225 INFO:teuthology.orchestra.run.smithi090.stderr:reweighted osd.7 to 1 (10000) 2024-04-14T15:28:24.603 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:24 smithi090 ceph-mon[30399]: 5.4 scrub starts 2024-04-14T15:28:24.603 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:24 smithi090 ceph-mon[30399]: 5.4 scrub ok 2024-04-14T15:28:24.603 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:24 smithi090 ceph-mon[30399]: 5.19 deep-scrub starts 2024-04-14T15:28:24.603 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:24 smithi090 ceph-mon[30399]: 5.19 deep-scrub ok 2024-04-14T15:28:24.603 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:24 smithi090 ceph-mon[30399]: osdmap e907: 8 total, 8 up, 8 in 2024-04-14T15:28:24.603 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:24 smithi090 ceph-mon[30399]: pgmap v1371: 177 pgs: 177 active+clean; 582 KiB data, 834 MiB used, 714 GiB / 715 GiB avail 2024-04-14T15:28:24.603 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:24 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-04-14T15:28:24.603 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:24 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2673384232' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-14T15:28:24.604 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:24 smithi090 ceph-mon[41670]: 5.4 scrub starts 2024-04-14T15:28:24.604 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:24 smithi090 ceph-mon[41670]: 5.4 scrub ok 2024-04-14T15:28:24.604 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:24 smithi090 ceph-mon[41670]: 5.19 deep-scrub starts 2024-04-14T15:28:24.604 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:24 smithi090 ceph-mon[41670]: 5.19 deep-scrub ok 2024-04-14T15:28:24.604 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:24 smithi090 ceph-mon[41670]: osdmap e907: 8 total, 8 up, 8 in 2024-04-14T15:28:24.604 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:24 smithi090 ceph-mon[41670]: pgmap v1371: 177 pgs: 177 active+clean; 582 KiB data, 834 MiB used, 714 GiB / 715 GiB avail 2024-04-14T15:28:24.604 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:24 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-04-14T15:28:24.604 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:24 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2673384232' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-14T15:28:24.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:24 smithi092 ceph-mon[33966]: 5.4 scrub starts 2024-04-14T15:28:24.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:24 smithi092 ceph-mon[33966]: 5.4 scrub ok 2024-04-14T15:28:24.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:24 smithi092 ceph-mon[33966]: 5.19 deep-scrub starts 2024-04-14T15:28:24.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:24 smithi092 ceph-mon[33966]: 5.19 deep-scrub ok 2024-04-14T15:28:24.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:24 smithi092 ceph-mon[33966]: osdmap e907: 8 total, 8 up, 8 in 2024-04-14T15:28:24.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:24 smithi092 ceph-mon[33966]: pgmap v1371: 177 pgs: 177 active+clean; 582 KiB data, 834 MiB used, 714 GiB / 715 GiB avail 2024-04-14T15:28:24.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:24 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-04-14T15:28:24.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:24 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2673384232' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-14T15:28:24.999 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd primary-affinity 7 1 2024-04-14T15:28:25.245 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:28:25.492 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:25 smithi090 ceph-mon[41670]: 2.15 deep-scrub starts 2024-04-14T15:28:25.493 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:25 smithi090 ceph-mon[41670]: 2.15 deep-scrub ok 2024-04-14T15:28:25.493 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:25 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]': finished 2024-04-14T15:28:25.493 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:25 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2673384232' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-14T15:28:25.493 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:25 smithi090 ceph-mon[41670]: osdmap e908: 8 total, 8 up, 8 in 2024-04-14T15:28:25.493 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:25 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:28:25.493 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:25 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:28:25.493 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:25 smithi090 ceph-mon[30399]: 2.15 deep-scrub starts 2024-04-14T15:28:25.493 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:25 smithi090 ceph-mon[30399]: 2.15 deep-scrub ok 2024-04-14T15:28:25.494 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:25 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]': finished 2024-04-14T15:28:25.494 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:25 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2673384232' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-14T15:28:25.494 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:25 smithi090 ceph-mon[30399]: osdmap e908: 8 total, 8 up, 8 in 2024-04-14T15:28:25.494 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:25 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:28:25.494 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:25 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:28:25.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:25 smithi092 ceph-mon[33966]: 2.15 deep-scrub starts 2024-04-14T15:28:25.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:25 smithi092 ceph-mon[33966]: 2.15 deep-scrub ok 2024-04-14T15:28:25.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:25 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]': finished 2024-04-14T15:28:25.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:25 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2673384232' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-14T15:28:25.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:25 smithi092 ceph-mon[33966]: osdmap e908: 8 total, 8 up, 8 in 2024-04-14T15:28:25.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:25 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:28:25.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:25 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:28:26.122 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:28:25 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:28:26.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:26 smithi090 ceph-mon[30399]: osdmap e909: 8 total, 8 up, 8 in 2024-04-14T15:28:26.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:26 smithi090 ceph-mon[30399]: pgmap v1374: 177 pgs: 177 active+clean; 582 KiB data, 834 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:28:26.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:26 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:28:26.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:26 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:28:26.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:26 smithi090 ceph-mon[30399]: osdmap e910: 8 total, 8 up, 8 in 2024-04-14T15:28:26.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:26 smithi090 ceph-mon[41670]: osdmap e909: 8 total, 8 up, 8 in 2024-04-14T15:28:26.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:26 smithi090 ceph-mon[41670]: pgmap v1374: 177 pgs: 177 active+clean; 582 KiB data, 834 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:28:26.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:26 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:28:26.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:26 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:28:26.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:26 smithi090 ceph-mon[41670]: osdmap e910: 8 total, 8 up, 8 in 2024-04-14T15:28:26.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:26 smithi092 ceph-mon[33966]: osdmap e909: 8 total, 8 up, 8 in 2024-04-14T15:28:26.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:26 smithi092 ceph-mon[33966]: pgmap v1374: 177 pgs: 177 active+clean; 582 KiB data, 834 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:28:26.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:26 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:28:26.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:26 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:28:26.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:26 smithi092 ceph-mon[33966]: osdmap e910: 8 total, 8 up, 8 in 2024-04-14T15:28:27.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:27.005+0000 7f14d448b700 1 -- 172.21.15.90:0/1548136740 >> v1:172.21.15.90:6789/0 conn(0x7f14cc10a630 legacy=0x7f14cc10ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:27.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:27.005+0000 7f14d448b700 1 -- 172.21.15.90:0/1548136740 shutdown_connections 2024-04-14T15:28:27.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:27.005+0000 7f14d448b700 1 -- 172.21.15.90:0/1548136740 >> 172.21.15.90:0/1548136740 conn(0x7f14cc0fd2b0 msgr2=0x7f14cc0ff6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:28:27.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:27.006+0000 7f14d448b700 1 -- 172.21.15.90:0/1548136740 shutdown_connections 2024-04-14T15:28:27.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:27.006+0000 7f14d448b700 1 -- 172.21.15.90:0/1548136740 wait complete. 2024-04-14T15:28:27.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:27.006+0000 7f14d448b700 1 librados: starting msgr at 2024-04-14T15:28:27.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:27.006+0000 7f14d448b700 1 librados: starting objecter 2024-04-14T15:28:27.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:27.007+0000 7f14d448b700 1 Processor -- start 2024-04-14T15:28:27.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:27.007+0000 7f14d448b700 1 -- start start 2024-04-14T15:28:27.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:27.007+0000 7f14d448b700 1 librados: setting wanted keys 2024-04-14T15:28:27.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:27.007+0000 7f14d448b700 1 librados: calling monclient init 2024-04-14T15:28:27.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:27.007+0000 7f14d448b700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f14cc10de50 con 0x7f14cc10e200 2024-04-14T15:28:27.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:27.007+0000 7f14d448b700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f14cc10e020 con 0x7f14cc107780 2024-04-14T15:28:27.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:27.007+0000 7f14d448b700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f14cc1a8e80 con 0x7f14cc10a630 2024-04-14T15:28:27.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:27.007+0000 7f14d1a26700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f14cc10a630 0x7f14cc10b460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:40574/0 (socket says 172.21.15.90:40574) 2024-04-14T15:28:27.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:27.007+0000 7f14d1a26700 1 -- 172.21.15.90:0/1390152864 learned_addr learned my addr 172.21.15.90:0/1390152864 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:28:27.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:27.007+0000 7f14c37fe700 10 client.?.objecter ms_handle_connect 0x7f14cc10e200 2024-04-14T15:28:27.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:27.007+0000 7f14c37fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:28:27.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:27.007+0000 7f14c37fe700 10 client.?.objecter ms_handle_connect 0x7f14cc10a630 2024-04-14T15:28:27.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:27.007+0000 7f14c37fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:28:27.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:27.007+0000 7f14c37fe700 10 client.?.objecter ms_handle_connect 0x7f14cc107780 2024-04-14T15:28:27.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:27.007+0000 7f14c37fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:28:27.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:27.007+0000 7f14c37fe700 1 -- 172.21.15.90:0/1390152864 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2056111140 0 0) 0x7f14cc1a8e80 con 0x7f14cc10a630 2024-04-14T15:28:27.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:27.007+0000 7f14c37fe700 1 -- 172.21.15.90:0/1390152864 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f14b80031f0 con 0x7f14cc10a630 2024-04-14T15:28:27.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:27.007+0000 7f14c37fe700 1 -- 172.21.15.90:0/1390152864 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2146915293 0 0) 0x7f14cc112000 con 0x7f14cc107780 2024-04-14T15:28:27.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:27.007+0000 7f14c37fe700 1 -- 172.21.15.90:0/1390152864 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f14cc1a8e80 con 0x7f14cc107780 2024-04-14T15:28:27.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:27.007+0000 7f14c37fe700 1 -- 172.21.15.90:0/1390152864 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2088587710 0 0) 0x7f14cc10de50 con 0x7f14cc10e200 2024-04-14T15:28:27.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:27.007+0000 7f14c37fe700 1 -- 172.21.15.90:0/1390152864 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f14cc112000 con 0x7f14cc10e200 2024-04-14T15:28:27.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:27.008+0000 7f14c37fe700 1 -- 172.21.15.90:0/1390152864 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1609128060 0 0) 0x7f14cc1a8e80 con 0x7f14cc107780 2024-04-14T15:28:27.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:27.008+0000 7f14c37fe700 1 -- 172.21.15.90:0/1390152864 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f14cc10de50 con 0x7f14cc107780 2024-04-14T15:28:27.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:27.008+0000 7f14c37fe700 1 -- 172.21.15.90:0/1390152864 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f14bc0018b0 con 0x7f14cc107780 2024-04-14T15:28:27.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:27.008+0000 7f14c37fe700 1 -- 172.21.15.90:0/1390152864 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 873161944 0 0) 0x7f14b80031f0 con 0x7f14cc10a630 2024-04-14T15:28:27.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:27.008+0000 7f14c37fe700 1 -- 172.21.15.90:0/1390152864 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f14cc1a8e80 con 0x7f14cc10a630 2024-04-14T15:28:27.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:27.008+0000 7f14c37fe700 1 -- 172.21.15.90:0/1390152864 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 262807259 0 0) 0x7f14cc112000 con 0x7f14cc10e200 2024-04-14T15:28:27.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:27.008+0000 7f14c37fe700 1 -- 172.21.15.90:0/1390152864 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f14b80031f0 con 0x7f14cc10e200 2024-04-14T15:28:27.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:27.008+0000 7f14c37fe700 1 -- 172.21.15.90:0/1390152864 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f14c8001430 con 0x7f14cc10a630 2024-04-14T15:28:27.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:27.008+0000 7f14c37fe700 1 -- 172.21.15.90:0/1390152864 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f14c4001bb0 con 0x7f14cc10e200 2024-04-14T15:28:27.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:27.008+0000 7f14c37fe700 1 -- 172.21.15.90:0/1390152864 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3744179800 0 0) 0x7f14cc10de50 con 0x7f14cc107780 2024-04-14T15:28:27.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:27.008+0000 7f14c37fe700 1 -- 172.21.15.90:0/1390152864 >> v1:172.21.15.90:6790/0 conn(0x7f14cc10a630 legacy=0x7f14cc10b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:27.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:27.008+0000 7f14c37fe700 1 -- 172.21.15.90:0/1390152864 >> v1:172.21.15.90:6789/0 conn(0x7f14cc10e200 legacy=0x7f14cc1a56e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:27.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:27.008+0000 7f14c37fe700 1 -- 172.21.15.90:0/1390152864 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14cc1aa0c0 con 0x7f14cc107780 2024-04-14T15:28:27.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:27.008+0000 7f14d448b700 1 -- 172.21.15.90:0/1390152864 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f14cc1a90b0 con 0x7f14cc107780 2024-04-14T15:28:27.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:27.008+0000 7f14d448b700 10 client.50122.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:28:27.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:27.008+0000 7f14d448b700 1 -- 172.21.15.90:0/1390152864 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f14cc1a9570 con 0x7f14cc107780 2024-04-14T15:28:27.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:27.008+0000 7f14d448b700 1 librados: init done 2024-04-14T15:28:27.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:27.008+0000 7f14c37fe700 1 -- 172.21.15.90:0/1390152864 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f14bc003b40 con 0x7f14cc107780 2024-04-14T15:28:27.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:27.008+0000 7f14c37fe700 1 -- 172.21.15.90:0/1390152864 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f14bc004980 con 0x7f14cc107780 2024-04-14T15:28:27.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:27.009+0000 7f14d448b700 1 -- 172.21.15.90:0/1390152864 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14b0005120 con 0x7f14cc107780 2024-04-14T15:28:27.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:27.010+0000 7f14c37fe700 1 -- 172.21.15.90:0/1390152864 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f14bc0036a0 con 0x7f14cc107780 2024-04-14T15:28:27.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:27.010+0000 7f14c37fe700 1 -- 172.21.15.90:0/1390152864 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(910..910 src has 467..910) ==== 7517+0+0 (unknown 488675711 0 0) 0x7f14bc0893a0 con 0x7f14cc107780 2024-04-14T15:28:27.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:27.010+0000 7f14c37fe700 10 client.50122.objecter ms_dispatch 0x7f14cc001550 osd_map(910..910 src has 467..910) 2024-04-14T15:28:27.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:27.010+0000 7f14c37fe700 3 client.50122.objecter handle_osd_map got epochs [910,910] > 0 2024-04-14T15:28:27.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:27.010+0000 7f14c37fe700 3 client.50122.objecter handle_osd_map decoding full epoch 910 2024-04-14T15:28:27.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:27.011+0000 7f14c37fe700 20 client.50122.objecter dump_active .. 0 homeless 2024-04-14T15:28:27.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:27.011+0000 7f14c37fe700 10 client.50122.objecter ms_handle_connect 0x7f14b8073730 2024-04-14T15:28:27.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:27.014+0000 7f14c37fe700 1 -- 172.21.15.90:0/1390152864 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f14cc10e020 con 0x7f14cc107780 2024-04-14T15:28:27.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:27.235+0000 7f14d448b700 1 -- 172.21.15.90:0/1390152864 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} v 0) -- 0x7f14b0005c20 con 0x7f14cc107780 2024-04-14T15:28:27.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:27 smithi090 ceph-mon[41670]: 6.c scrub starts 2024-04-14T15:28:27.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:27 smithi090 ceph-mon[41670]: 6.c scrub ok 2024-04-14T15:28:27.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:27 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:28:27.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:27 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:28:27.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:27 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:28:27.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:27 smithi090 ceph-mon[41670]: osdmap e911: 8 total, 8 up, 8 in 2024-04-14T15:28:27.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:27 smithi090 ceph-mon[30399]: 6.c scrub starts 2024-04-14T15:28:27.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:27 smithi090 ceph-mon[30399]: 6.c scrub ok 2024-04-14T15:28:27.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:27 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:28:27.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:27 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:28:27.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:27 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:28:27.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:27 smithi090 ceph-mon[30399]: osdmap e911: 8 total, 8 up, 8 in 2024-04-14T15:28:27.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:27 smithi092 ceph-mon[33966]: 6.c scrub starts 2024-04-14T15:28:27.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:27 smithi092 ceph-mon[33966]: 6.c scrub ok 2024-04-14T15:28:27.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:27 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:28:27.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:27 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:28:27.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:27 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:28:27.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:27 smithi092 ceph-mon[33966]: osdmap e911: 8 total, 8 up, 8 in 2024-04-14T15:28:28.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:28.240+0000 7f14c37fe700 1 -- 172.21.15.90:0/1390152864 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]=0 set osd.7 primary-affinity to 1 (8655362) v912) ==== 133+0+0 (unknown 2116449780 0 0) 0x7f14bc056040 con 0x7f14cc107780 2024-04-14T15:28:28.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:28.245+0000 7f14d448b700 10 librados: watch_flush enter 2024-04-14T15:28:28.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:28.245+0000 7f14d448b700 10 librados: watch_flush exit 2024-04-14T15:28:28.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:28.245+0000 7f14d448b700 20 client.50122.objecter shutdown clearing up homeless session... 2024-04-14T15:28:28.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:28.245+0000 7f14d448b700 10 client.50122.objecter successfully canceled tick 2024-04-14T15:28:28.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:28.245+0000 7f14d448b700 1 -- 172.21.15.90:0/1390152864 >> v1:172.21.15.90:6800/420935427 conn(0x7f14b8073730 legacy=0x7f14b8075bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:28.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:28.245+0000 7f14d448b700 1 -- 172.21.15.90:0/1390152864 >> v1:172.21.15.92:6789/0 conn(0x7f14cc107780 legacy=0x7f14cc10ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:28.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:28.245+0000 7f14d448b700 1 -- 172.21.15.90:0/1390152864 shutdown_connections 2024-04-14T15:28:28.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:28.245+0000 7f14d448b700 1 -- 172.21.15.90:0/1390152864 >> 172.21.15.90:0/1390152864 conn(0x7f14cc0fd2b0 msgr2=0x7f14cc1116e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:28:28.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:28.245+0000 7f14d448b700 1 -- 172.21.15.90:0/1390152864 shutdown_connections 2024-04-14T15:28:28.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:28.245+0000 7f14d448b700 1 -- 172.21.15.90:0/1390152864 wait complete. 2024-04-14T15:28:28.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:28.245+0000 7f14d448b700 1 librados: shutdown 2024-04-14T15:28:28.247 INFO:teuthology.orchestra.run.smithi090.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-04-14T15:28:28.565 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:28 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1390152864' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-04-14T15:28:28.565 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:28 smithi090 ceph-mon[41670]: from='client.50122 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-04-14T15:28:28.565 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:28 smithi090 ceph-mon[41670]: pgmap v1377: 176 pgs: 176 active+clean; 582 KiB data, 834 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:28:28.565 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:28 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-04-14T15:28:28.565 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:28 smithi090 ceph-mon[41670]: 11.3 scrub starts 2024-04-14T15:28:28.565 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:28 smithi090 ceph-mon[41670]: 11.3 scrub ok 2024-04-14T15:28:28.566 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:28 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1390152864' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-04-14T15:28:28.566 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:28 smithi090 ceph-mon[30399]: from='client.50122 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-04-14T15:28:28.566 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:28 smithi090 ceph-mon[30399]: pgmap v1377: 176 pgs: 176 active+clean; 582 KiB data, 834 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:28:28.566 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:28 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-04-14T15:28:28.566 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:28 smithi090 ceph-mon[30399]: 11.3 scrub starts 2024-04-14T15:28:28.566 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:28 smithi090 ceph-mon[30399]: 11.3 scrub ok 2024-04-14T15:28:28.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:28 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1390152864' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-04-14T15:28:28.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:28 smithi092 ceph-mon[33966]: from='client.50122 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-04-14T15:28:28.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:28 smithi092 ceph-mon[33966]: pgmap v1377: 176 pgs: 176 active+clean; 582 KiB data, 834 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:28:28.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:28 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-04-14T15:28:28.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:28 smithi092 ceph-mon[33966]: 11.3 scrub starts 2024-04-14T15:28:28.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:28 smithi092 ceph-mon[33966]: 11.3 scrub ok 2024-04-14T15:28:29.005 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-04-14T15:28:29.006 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-04-14T15:28:29.006 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-04-14T15:28:29.006 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-04-14T15:28:29.006 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-14T15:28:29.006 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd dump --format=json 2024-04-14T15:28:29.259 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:28:29.489 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:29 smithi090 ceph-mon[41670]: 1.5 scrub starts 2024-04-14T15:28:29.489 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:29 smithi090 ceph-mon[41670]: 1.5 scrub ok 2024-04-14T15:28:29.489 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:29 smithi090 ceph-mon[41670]: from='client.50122 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-04-14T15:28:29.489 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:29 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]': finished 2024-04-14T15:28:29.489 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:29 smithi090 ceph-mon[41670]: osdmap e912: 8 total, 8 up, 8 in 2024-04-14T15:28:29.490 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:29 smithi090 ceph-mon[41670]: 2.2 scrub starts 2024-04-14T15:28:29.490 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:29 smithi090 ceph-mon[41670]: 2.2 scrub ok 2024-04-14T15:28:29.490 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:29 smithi090 ceph-mon[30399]: 1.5 scrub starts 2024-04-14T15:28:29.490 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:29 smithi090 ceph-mon[30399]: 1.5 scrub ok 2024-04-14T15:28:29.490 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:29 smithi090 ceph-mon[30399]: from='client.50122 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-04-14T15:28:29.490 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:29 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]': finished 2024-04-14T15:28:29.490 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:29 smithi090 ceph-mon[30399]: osdmap e912: 8 total, 8 up, 8 in 2024-04-14T15:28:29.490 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:29 smithi090 ceph-mon[30399]: 2.2 scrub starts 2024-04-14T15:28:29.490 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:29 smithi090 ceph-mon[30399]: 2.2 scrub ok 2024-04-14T15:28:29.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:29 smithi092 ceph-mon[33966]: 1.5 scrub starts 2024-04-14T15:28:29.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:29 smithi092 ceph-mon[33966]: 1.5 scrub ok 2024-04-14T15:28:29.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:29 smithi092 ceph-mon[33966]: from='client.50122 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-04-14T15:28:29.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:29 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]': finished 2024-04-14T15:28:29.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:29 smithi092 ceph-mon[33966]: osdmap e912: 8 total, 8 up, 8 in 2024-04-14T15:28:29.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:29 smithi092 ceph-mon[33966]: 2.2 scrub starts 2024-04-14T15:28:29.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:29 smithi092 ceph-mon[33966]: 2.2 scrub ok 2024-04-14T15:28:30.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:30 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:28:30.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:30 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:28:30.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:30 smithi090 ceph-mon[30399]: 6.8 deep-scrub starts 2024-04-14T15:28:30.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:30 smithi090 ceph-mon[30399]: 6.8 deep-scrub ok 2024-04-14T15:28:30.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:30 smithi090 ceph-mon[30399]: pgmap v1379: 176 pgs: 2 peering, 174 active+clean; 582 KiB data, 835 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:28:30.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:30 smithi090 ceph-mon[30399]: osdmap e913: 8 total, 8 up, 8 in 2024-04-14T15:28:30.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:30 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:28:30.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:30 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:28:30.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:30 smithi090 ceph-mon[41670]: 6.8 deep-scrub starts 2024-04-14T15:28:30.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:30 smithi090 ceph-mon[41670]: 6.8 deep-scrub ok 2024-04-14T15:28:30.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:30 smithi090 ceph-mon[41670]: pgmap v1379: 176 pgs: 2 peering, 174 active+clean; 582 KiB data, 835 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:28:30.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:30 smithi090 ceph-mon[41670]: osdmap e913: 8 total, 8 up, 8 in 2024-04-14T15:28:30.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:30 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:28:30.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:30 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:28:30.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:30 smithi092 ceph-mon[33966]: 6.8 deep-scrub starts 2024-04-14T15:28:30.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:30 smithi092 ceph-mon[33966]: 6.8 deep-scrub ok 2024-04-14T15:28:30.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:30 smithi092 ceph-mon[33966]: pgmap v1379: 176 pgs: 2 peering, 174 active+clean; 582 KiB data, 835 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:28:30.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:30 smithi092 ceph-mon[33966]: osdmap e913: 8 total, 8 up, 8 in 2024-04-14T15:28:31.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:30.980+0000 7fc9055d0700 1 -- 172.21.15.90:0/1542094049 >> v1:172.21.15.90:6789/0 conn(0x7fc90010e1e0 legacy=0x7fc9001106a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:31.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:30.980+0000 7fc9055d0700 1 -- 172.21.15.90:0/1542094049 shutdown_connections 2024-04-14T15:28:31.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:30.980+0000 7fc9055d0700 1 -- 172.21.15.90:0/1542094049 >> 172.21.15.90:0/1542094049 conn(0x7fc9000fd290 msgr2=0x7fc9000ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:28:31.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:30.980+0000 7fc9055d0700 1 -- 172.21.15.90:0/1542094049 shutdown_connections 2024-04-14T15:28:31.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:30.980+0000 7fc9055d0700 1 -- 172.21.15.90:0/1542094049 wait complete. 2024-04-14T15:28:31.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:30.981+0000 7fc9055d0700 1 librados: starting msgr at 2024-04-14T15:28:31.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:30.981+0000 7fc9055d0700 1 librados: starting objecter 2024-04-14T15:28:31.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:30.981+0000 7fc9055d0700 1 Processor -- start 2024-04-14T15:28:31.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:30.981+0000 7fc9055d0700 1 -- start start 2024-04-14T15:28:31.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:30.981+0000 7fc9055d0700 1 librados: setting wanted keys 2024-04-14T15:28:31.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:30.981+0000 7fc9055d0700 1 librados: calling monclient init 2024-04-14T15:28:31.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:30.981+0000 7fc9055d0700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc9000ff4b0 con 0x7fc90010e1e0 2024-04-14T15:28:31.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:30.981+0000 7fc9055d0700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc9001ab1f0 con 0x7fc900107760 2024-04-14T15:28:31.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:30.981+0000 7fc9055d0700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc9001ac3d0 con 0x7fc90010a610 2024-04-14T15:28:31.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:30.981+0000 7fc8ff7fe700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fc90010e1e0 0x7fc9001a8ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:43848/0 (socket says 172.21.15.90:43848) 2024-04-14T15:28:31.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:30.981+0000 7fc8fe7fc700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7fc90010a610 0x7fc9000fde50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:39698/0 (socket says 172.21.15.90:39698) 2024-04-14T15:28:31.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:30.981+0000 7fc8ff7fe700 1 -- 172.21.15.90:0/695715463 learned_addr learned my addr 172.21.15.90:0/695715463 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:28:31.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:30.981+0000 7fc8e7fff700 10 client.?.objecter ms_handle_connect 0x7fc90010e1e0 2024-04-14T15:28:31.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:30.981+0000 7fc8e7fff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:28:31.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:30.981+0000 7fc8e7fff700 10 client.?.objecter ms_handle_connect 0x7fc90010a610 2024-04-14T15:28:31.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:30.981+0000 7fc8e7fff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:28:31.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:30.981+0000 7fc8e7fff700 10 client.?.objecter ms_handle_connect 0x7fc900107760 2024-04-14T15:28:31.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:30.981+0000 7fc8e7fff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:28:31.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:30.982+0000 7fc8e7fff700 1 -- 172.21.15.90:0/695715463 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1683754958 0 0) 0x7fc9000ff4b0 con 0x7fc90010e1e0 2024-04-14T15:28:31.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:30.982+0000 7fc8e7fff700 1 -- 172.21.15.90:0/695715463 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc8e80035b0 con 0x7fc90010e1e0 2024-04-14T15:28:31.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:30.982+0000 7fc8e7fff700 1 -- 172.21.15.90:0/695715463 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3448602738 0 0) 0x7fc9001ac3d0 con 0x7fc90010a610 2024-04-14T15:28:31.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:30.982+0000 7fc8e7fff700 1 -- 172.21.15.90:0/695715463 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9000ff4b0 con 0x7fc90010a610 2024-04-14T15:28:31.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:30.982+0000 7fc8e7fff700 1 -- 172.21.15.90:0/695715463 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3615977333 0 0) 0x7fc9001ab1f0 con 0x7fc900107760 2024-04-14T15:28:31.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:30.982+0000 7fc8e7fff700 1 -- 172.21.15.90:0/695715463 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9001ac3d0 con 0x7fc900107760 2024-04-14T15:28:31.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:30.982+0000 7fc8e7fff700 1 -- 172.21.15.90:0/695715463 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1606316543 0 0) 0x7fc8e80035b0 con 0x7fc90010e1e0 2024-04-14T15:28:31.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:30.982+0000 7fc8e7fff700 1 -- 172.21.15.90:0/695715463 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc9001ab1f0 con 0x7fc90010e1e0 2024-04-14T15:28:31.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:30.982+0000 7fc8e7fff700 1 -- 172.21.15.90:0/695715463 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fc8f8001950 con 0x7fc90010e1e0 2024-04-14T15:28:31.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:30.982+0000 7fc8e7fff700 1 -- 172.21.15.90:0/695715463 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2136236549 0 0) 0x7fc9000ff4b0 con 0x7fc90010a610 2024-04-14T15:28:31.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:30.982+0000 7fc8e7fff700 1 -- 172.21.15.90:0/695715463 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc8e80035b0 con 0x7fc90010a610 2024-04-14T15:28:31.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:30.982+0000 7fc8e7fff700 1 -- 172.21.15.90:0/695715463 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 611129257 0 0) 0x7fc9001ac3d0 con 0x7fc900107760 2024-04-14T15:28:31.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:30.982+0000 7fc8e7fff700 1 -- 172.21.15.90:0/695715463 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc9000ff4b0 con 0x7fc900107760 2024-04-14T15:28:31.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:30.982+0000 7fc8e7fff700 1 -- 172.21.15.90:0/695715463 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fc8f0001830 con 0x7fc90010a610 2024-04-14T15:28:31.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:30.982+0000 7fc8e7fff700 1 -- 172.21.15.90:0/695715463 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fc8f4001720 con 0x7fc900107760 2024-04-14T15:28:31.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:30.982+0000 7fc8e7fff700 1 -- 172.21.15.90:0/695715463 <== mon.1 v1:172.21.15.92:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 441768799 0 0) 0x7fc9000ff4b0 con 0x7fc900107760 2024-04-14T15:28:31.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:30.982+0000 7fc8e7fff700 1 -- 172.21.15.90:0/695715463 >> v1:172.21.15.90:6790/0 conn(0x7fc90010a610 legacy=0x7fc9000fde50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:31.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:30.982+0000 7fc8e7fff700 1 -- 172.21.15.90:0/695715463 >> v1:172.21.15.90:6789/0 conn(0x7fc90010e1e0 legacy=0x7fc9001a8ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:31.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:30.982+0000 7fc8e7fff700 1 -- 172.21.15.90:0/695715463 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9001ad5b0 con 0x7fc900107760 2024-04-14T15:28:31.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:30.982+0000 7fc9055d0700 1 -- 172.21.15.90:0/695715463 --> v1:172.21.15.92:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc9001ac5a0 con 0x7fc900107760 2024-04-14T15:28:31.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:30.982+0000 7fc9055d0700 10 client.50128.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:28:31.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:30.982+0000 7fc9055d0700 1 -- 172.21.15.90:0/695715463 --> v1:172.21.15.92:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc9001aca60 con 0x7fc900107760 2024-04-14T15:28:31.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:30.982+0000 7fc9055d0700 1 librados: init done 2024-04-14T15:28:31.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:30.982+0000 7fc8e7fff700 1 -- 172.21.15.90:0/695715463 <== mon.1 v1:172.21.15.92:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fc8f4001d30 con 0x7fc900107760 2024-04-14T15:28:31.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:30.982+0000 7fc8e7fff700 1 -- 172.21.15.90:0/695715463 <== mon.1 v1:172.21.15.92:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fc8f40049d0 con 0x7fc900107760 2024-04-14T15:28:31.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:30.983+0000 7fc9055d0700 1 -- 172.21.15.90:0/695715463 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8ec005120 con 0x7fc900107760 2024-04-14T15:28:31.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:30.984+0000 7fc8e7fff700 1 -- 172.21.15.90:0/695715463 <== mon.1 v1:172.21.15.92:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fc8f4003680 con 0x7fc900107760 2024-04-14T15:28:31.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:30.984+0000 7fc8e7fff700 1 -- 172.21.15.90:0/695715463 <== mon.1 v1:172.21.15.92:6789/0 8 ==== osd_map(914..914 src has 467..914) ==== 7517+0+0 (unknown 2776053878 0 0) 0x7fc8f4089260 con 0x7fc900107760 2024-04-14T15:28:31.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:30.984+0000 7fc8e7fff700 10 client.50128.objecter ms_dispatch 0x7fc900001550 osd_map(914..914 src has 467..914) 2024-04-14T15:28:31.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:30.984+0000 7fc8e7fff700 3 client.50128.objecter handle_osd_map got epochs [914,914] > 0 2024-04-14T15:28:31.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:30.984+0000 7fc8e7fff700 3 client.50128.objecter handle_osd_map decoding full epoch 914 2024-04-14T15:28:31.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:30.984+0000 7fc8e7fff700 20 client.50128.objecter dump_active .. 0 homeless 2024-04-14T15:28:31.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:30.984+0000 7fc8e7fff700 10 client.50128.objecter ms_handle_connect 0x7fc8e80712f0 2024-04-14T15:28:31.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:30.989+0000 7fc8e7fff700 1 -- 172.21.15.90:0/695715463 <== mon.1 v1:172.21.15.92:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fc8f4055f00 con 0x7fc900107760 2024-04-14T15:28:31.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:31.208+0000 7fc9055d0700 1 -- 172.21.15.90:0/695715463 --> v1:172.21.15.92:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc8ec005410 con 0x7fc900107760 2024-04-14T15:28:31.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:31.210+0000 7fc8e7fff700 1 -- 172.21.15.90:0/695715463 <== mon.1 v1:172.21.15.92:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v914) ==== 74+0+24146 (unknown 3349912027 0 4239114340) 0x7fc8f4059bb0 con 0x7fc900107760 2024-04-14T15:28:31.210 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:28:31.211 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":914,"fsid":"0e288098-fa70-11ee-bc8e-c7b262605968","created":"2024-04-14T15:04:05.808313+0000","modified":"2024-04-14T15:28:30.246330+0000","last_up_change":"2024-04-14T15:09:29.769352+0000","last_in_change":"2024-04-14T15:27:13.165962+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":119,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":345,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-14T15:07:23.959628+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":12,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":12,"last_pg_merge_meta":{"source_pgid":"1.c","ready_epoch":914,"last_epoch_started":913,"last_epoch_clean":913,"source_version":"91'130","target_version":"91'130"},"last_change":"914","last_force_op_resend":"0","last_force_op_resend_prenautilus":"914","last_force_op_resend_preluminous":"912","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-04-14T15:09:41.608420+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"datapool","create_time":"2024-04-14T15:09:41.744391+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":69,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-04-14T15:09:43.088157+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-04-14T15:09:45.042722+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-04-14T15:09:47.111336+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"ceph_test_rados_api_asio","create_time":"2024-04-14T15:12:54.225671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"76772cfc-89d1-4013-b617-32284ed8dfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":901,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6801","nonce":3761806885}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6802","nonce":3761806885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6804","nonce":3761806885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6803","nonce":3761806885}]},"public_addr":"172.21.15.90:6801/3761806885","cluster_addr":"172.21.15.90:6802/3761806885","heartbeat_back_addr":"172.21.15.90:6804/3761806885","heartbeat_front_addr":"172.21.15.90:6803/3761806885","state":["exists","up"]},{"osd":1,"uuid":"de4a7fa2-7d57-4ff3-a4fa-28c0cde8ef6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":884,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6805","nonce":4170361699}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6806","nonce":4170361699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6808","nonce":4170361699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6807","nonce":4170361699}]},"public_addr":"172.21.15.90:6805/4170361699","cluster_addr":"172.21.15.90:6806/4170361699","heartbeat_back_addr":"172.21.15.90:6808/4170361699","heartbeat_front_addr":"172.21.15.90:6807/4170361699","state":["exists","up"]},{"osd":2,"uuid":"9c39b5ff-6175-46ba-9d71-370581c6ff3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":865,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6809","nonce":1686184251}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6810","nonce":1686184251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6812","nonce":1686184251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6811","nonce":1686184251}]},"public_addr":"172.21.15.90:6809/1686184251","cluster_addr":"172.21.15.90:6810/1686184251","heartbeat_back_addr":"172.21.15.90:6812/1686184251","heartbeat_front_addr":"172.21.15.90:6811/1686184251","state":["exists","up"]},{"osd":3,"uuid":"efd2ca45-5b80-4013-a571-499e2cb5b2e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":912,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6813","nonce":2270166774}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6814","nonce":2270166774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6816","nonce":2270166774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6815","nonce":2270166774}]},"public_addr":"172.21.15.90:6813/2270166774","cluster_addr":"172.21.15.90:6814/2270166774","heartbeat_back_addr":"172.21.15.90:6816/2270166774","heartbeat_front_addr":"172.21.15.90:6815/2270166774","state":["exists","up"]},{"osd":4,"uuid":"e3d9013d-5eb6-433c-96b3-3fc38ba7ac8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":910,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6800","nonce":3887549733}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6801","nonce":3887549733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6803","nonce":3887549733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6802","nonce":3887549733}]},"public_addr":"172.21.15.92:6800/3887549733","cluster_addr":"172.21.15.92:6801/3887549733","heartbeat_back_addr":"172.21.15.92:6803/3887549733","heartbeat_front_addr":"172.21.15.92:6802/3887549733","state":["exists","up"]},{"osd":5,"uuid":"1ec41805-6f82-4139-81bc-75305ba9e958","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":883,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6804","nonce":3244561440}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6805","nonce":3244561440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6807","nonce":3244561440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6806","nonce":3244561440}]},"public_addr":"172.21.15.92:6804/3244561440","cluster_addr":"172.21.15.92:6805/3244561440","heartbeat_back_addr":"172.21.15.92:6807/3244561440","heartbeat_front_addr":"172.21.15.92:6806/3244561440","state":["exists","up"]},{"osd":6,"uuid":"ca977ab2-bf8d-4b3a-ae5e-0efa5bc7a3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":906,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6808","nonce":636125963}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6809","nonce":636125963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6811","nonce":636125963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6810","nonce":636125963}]},"public_addr":"172.21.15.92:6808/636125963","cluster_addr":"172.21.15.92:6809/636125963","heartbeat_back_addr":"172.21.15.92:6811/636125963","heartbeat_front_addr":"172.21.15.92:6810/636125963","state":["exists","up"]},{"osd":7,"uuid":"e5354645-79ec-4d4c-b6e5-f9b0653915ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":887,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6812","nonce":1773706716}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6813","nonce":1773706716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6815","nonce":1773706716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6814","nonce":1773706716}]},"public_addr":"172.21.15.92:6812/1773706716","cluster_addr":"172.21.15.92:6813/1773706716","heartbeat_back_addr":"172.21.15.92:6815/1773706716","heartbeat_front_addr":"172.21.15.92:6814/1773706716","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T15:27:41.248761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T15:27:47.498189+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T15:27:18.377262+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T15:27:41.356530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T15:28:10.850534+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T15:28:15.686886+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T15:27:15.597391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T15:28:28.608598+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.1d","mappings":[{"from":7,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.12","mappings":[{"from":1,"to":5}]},{"pgid":"4.15","mappings":[{"from":3,"to":2}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]},{"pgid":"11.1c","mappings":[{"from":2,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:0/508768479":"2024-04-15T15:10:22.918674+0000","172.21.15.90:0/1823072952":"2024-04-15T15:10:22.918674+0000","172.21.15.90:6800/491071118":"2024-04-15T15:10:22.918674+0000","172.21.15.90:0/1337064035":"2024-04-15T15:10:22.918674+0000","172.21.15.90:0/769462306":"2024-04-15T15:10:22.918674+0000","172.21.15.90:0/1139868610":"2024-04-15T15:05:05.882361+0000","172.21.15.90:0/93763408":"2024-04-15T15:10:22.918674+0000","172.21.15.90:0/977818204":"2024-04-15T15:04:39.251711+0000","172.21.15.90:6800/941091171":"2024-04-15T15:05:05.882361+0000","172.21.15.90:0/3671453561":"2024-04-15T15:05:05.882361+0000","172.21.15.90:0/2611856200":"2024-04-15T15:05:05.882361+0000","172.21.15.90:6800/2306258962":"2024-04-15T15:04:39.251711+0000","172.21.15.90:0/4051692380":"2024-04-15T15:04:39.251711+0000","172.21.15.90:0/2681502561":"2024-04-15T15:04:39.251711+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":5}]},{"pool":18,"snaps":[{"begin":2,"length":3}]},{"pool":39,"snaps":[{"begin":1,"length":1}]},{"pool":72,"snaps":[{"begin":1,"length":1}]},{"pool":99,"snaps":[{"begin":4,"length":2},{"begin":7,"length":2}]},{"pool":109,"snaps":[{"begin":1,"length":1}]},{"pool":128,"snaps":[{"begin":1,"length":2}]},{"pool":142,"snaps":[{"begin":3,"length":1}]},{"pool":149,"snaps":[{"begin":1,"length":4}]},{"pool":156,"snaps":[{"begin":2,"length":1}]},{"pool":157,"snaps":[{"begin":2,"length":2}]},{"pool":167,"snaps":[{"begin":2,"length":3}]},{"pool":172,"snaps":[{"begin":3,"length":4}]},{"pool":175,"snaps":[{"begin":2,"length":4}]},{"pool":176,"snaps":[{"begin":2,"length":2}]},{"pool":187,"snaps":[{"begin":2,"length":1}]},{"pool":192,"snaps":[{"begin":1,"length":4}]},{"pool":205,"snaps":[{"begin":2,"length":6}]},{"pool":213,"snaps":[{"begin":7,"length":1}]},{"pool":234,"snaps":[{"begin":8,"length":1}]},{"pool":247,"snaps":[{"begin":10,"length":1}]},{"pool":252,"snaps":[{"begin":11,"length":2}]},{"pool":265,"snaps":[{"begin":14,"length":1}]},{"pool":326,"snaps":[{"begin":2,"length":1}]},{"pool":327,"snaps":[{"begin":3,"length":1}]},{"pool":330,"snaps":[{"begin":4,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-14T15:28:31.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:31.212+0000 7fc9055d0700 10 librados: watch_flush enter 2024-04-14T15:28:31.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:31.212+0000 7fc9055d0700 10 librados: watch_flush exit 2024-04-14T15:28:31.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:31.212+0000 7fc9055d0700 20 client.50128.objecter shutdown clearing up homeless session... 2024-04-14T15:28:31.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:31.212+0000 7fc9055d0700 10 client.50128.objecter successfully canceled tick 2024-04-14T15:28:31.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:31.212+0000 7fc9055d0700 1 -- 172.21.15.90:0/695715463 >> v1:172.21.15.90:6800/420935427 conn(0x7fc8e80712f0 legacy=0x7fc8e80737b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:31.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:31.212+0000 7fc9055d0700 1 -- 172.21.15.90:0/695715463 >> v1:172.21.15.92:6789/0 conn(0x7fc900107760 legacy=0x7fc9000fd740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:31.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:31.212+0000 7fc9055d0700 1 -- 172.21.15.90:0/695715463 shutdown_connections 2024-04-14T15:28:31.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:31.212+0000 7fc9055d0700 1 -- 172.21.15.90:0/695715463 >> 172.21.15.90:0/695715463 conn(0x7fc9000fd290 msgr2=0x7fc900111050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:28:31.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:31.212+0000 7fc9055d0700 1 -- 172.21.15.90:0/695715463 shutdown_connections 2024-04-14T15:28:31.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:31.212+0000 7fc9055d0700 1 -- 172.21.15.90:0/695715463 wait complete. 2024-04-14T15:28:31.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:31.212+0000 7fc9055d0700 1 librados: shutdown 2024-04-14T15:28:31.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:31 smithi090 ceph-mon[41670]: osdmap e914: 8 total, 8 up, 8 in 2024-04-14T15:28:31.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:31 smithi090 ceph-mon[41670]: 6.d scrub starts 2024-04-14T15:28:31.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:31 smithi090 ceph-mon[41670]: 6.d scrub ok 2024-04-14T15:28:31.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:31 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/695715463' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-14T15:28:31.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:31 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-04-14T15:28:31.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:31 smithi090 ceph-mon[30399]: osdmap e914: 8 total, 8 up, 8 in 2024-04-14T15:28:31.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:31 smithi090 ceph-mon[30399]: 6.d scrub starts 2024-04-14T15:28:31.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:31 smithi090 ceph-mon[30399]: 6.d scrub ok 2024-04-14T15:28:31.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:31 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/695715463' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-14T15:28:31.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:31 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-04-14T15:28:31.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:31 smithi092 ceph-mon[33966]: osdmap e914: 8 total, 8 up, 8 in 2024-04-14T15:28:31.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:31 smithi092 ceph-mon[33966]: 6.d scrub starts 2024-04-14T15:28:31.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:31 smithi092 ceph-mon[33966]: 6.d scrub ok 2024-04-14T15:28:31.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:31 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/695715463' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-14T15:28:31.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:31 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-04-14T15:28:31.838 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-14T15:28:31.838 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd dump --format=json 2024-04-14T15:28:32.082 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:28:32.518 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:32 smithi090 ceph-mon[41670]: 4.10 scrub starts 2024-04-14T15:28:32.518 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:32 smithi090 ceph-mon[41670]: 4.10 scrub ok 2024-04-14T15:28:32.518 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:32 smithi090 ceph-mon[41670]: pgmap v1382: 175 pgs: 1 peering, 174 active+clean; 262 KiB data, 835 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:28:32.518 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:32 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-04-14T15:28:32.518 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:32 smithi090 ceph-mon[41670]: osdmap e915: 8 total, 8 up, 8 in 2024-04-14T15:28:32.518 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:32 smithi090 ceph-mon[41670]: osdmap e916: 8 total, 8 up, 8 in 2024-04-14T15:28:32.519 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:32 smithi090 ceph-mon[30399]: 4.10 scrub starts 2024-04-14T15:28:32.519 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:32 smithi090 ceph-mon[30399]: 4.10 scrub ok 2024-04-14T15:28:32.519 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:32 smithi090 ceph-mon[30399]: pgmap v1382: 175 pgs: 1 peering, 174 active+clean; 262 KiB data, 835 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:28:32.519 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:32 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-04-14T15:28:32.519 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:32 smithi090 ceph-mon[30399]: osdmap e915: 8 total, 8 up, 8 in 2024-04-14T15:28:32.519 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:32 smithi090 ceph-mon[30399]: osdmap e916: 8 total, 8 up, 8 in 2024-04-14T15:28:32.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:32 smithi092 ceph-mon[33966]: 4.10 scrub starts 2024-04-14T15:28:32.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:32 smithi092 ceph-mon[33966]: 4.10 scrub ok 2024-04-14T15:28:32.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:32 smithi092 ceph-mon[33966]: pgmap v1382: 175 pgs: 1 peering, 174 active+clean; 262 KiB data, 835 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:28:32.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:32 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-04-14T15:28:32.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:32 smithi092 ceph-mon[33966]: osdmap e915: 8 total, 8 up, 8 in 2024-04-14T15:28:32.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:32 smithi092 ceph-mon[33966]: osdmap e916: 8 total, 8 up, 8 in 2024-04-14T15:28:33.527 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:33 smithi090 ceph-mon[30399]: 6.5 deep-scrub starts 2024-04-14T15:28:33.528 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:33 smithi090 ceph-mon[30399]: 6.5 deep-scrub ok 2024-04-14T15:28:33.528 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:33 smithi090 ceph-mon[30399]: 6.6 scrub starts 2024-04-14T15:28:33.528 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:33 smithi090 ceph-mon[30399]: 6.6 scrub ok 2024-04-14T15:28:33.528 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:33 smithi090 ceph-mon[30399]: 2.1f scrub starts 2024-04-14T15:28:33.528 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:33 smithi090 ceph-mon[30399]: 2.1f scrub ok 2024-04-14T15:28:33.528 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:33 smithi090 ceph-mon[30399]: 5.11 scrub starts 2024-04-14T15:28:33.528 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:33 smithi090 ceph-mon[30399]: 5.11 scrub ok 2024-04-14T15:28:33.528 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:33 smithi090 ceph-mon[41670]: 6.5 deep-scrub starts 2024-04-14T15:28:33.528 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:33 smithi090 ceph-mon[41670]: 6.5 deep-scrub ok 2024-04-14T15:28:33.528 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:33 smithi090 ceph-mon[41670]: 6.6 scrub starts 2024-04-14T15:28:33.528 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:33 smithi090 ceph-mon[41670]: 6.6 scrub ok 2024-04-14T15:28:33.528 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:33 smithi090 ceph-mon[41670]: 2.1f scrub starts 2024-04-14T15:28:33.529 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:33 smithi090 ceph-mon[41670]: 2.1f scrub ok 2024-04-14T15:28:33.529 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:33 smithi090 ceph-mon[41670]: 5.11 scrub starts 2024-04-14T15:28:33.529 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:33 smithi090 ceph-mon[41670]: 5.11 scrub ok 2024-04-14T15:28:33.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:33 smithi092 ceph-mon[33966]: 6.5 deep-scrub starts 2024-04-14T15:28:33.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:33 smithi092 ceph-mon[33966]: 6.5 deep-scrub ok 2024-04-14T15:28:33.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:33 smithi092 ceph-mon[33966]: 6.6 scrub starts 2024-04-14T15:28:33.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:33 smithi092 ceph-mon[33966]: 6.6 scrub ok 2024-04-14T15:28:33.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:33 smithi092 ceph-mon[33966]: 2.1f scrub starts 2024-04-14T15:28:33.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:33 smithi092 ceph-mon[33966]: 2.1f scrub ok 2024-04-14T15:28:33.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:33 smithi092 ceph-mon[33966]: 5.11 scrub starts 2024-04-14T15:28:33.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:33 smithi092 ceph-mon[33966]: 5.11 scrub ok 2024-04-14T15:28:33.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:33.857+0000 7f3c092fd700 1 -- 172.21.15.90:0/483388133 >> v1:172.21.15.90:6789/0 conn(0x7f3c0410d9b0 legacy=0x7f3c0410fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:33.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:33.857+0000 7f3c092fd700 1 -- 172.21.15.90:0/483388133 shutdown_connections 2024-04-14T15:28:33.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:33.857+0000 7f3c092fd700 1 -- 172.21.15.90:0/483388133 >> 172.21.15.90:0/483388133 conn(0x7f3c04076aa0 msgr2=0x7f3c04076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:28:33.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:33.857+0000 7f3c092fd700 1 -- 172.21.15.90:0/483388133 shutdown_connections 2024-04-14T15:28:33.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:33.857+0000 7f3c092fd700 1 -- 172.21.15.90:0/483388133 wait complete. 2024-04-14T15:28:33.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:33.858+0000 7f3c092fd700 1 librados: starting msgr at 2024-04-14T15:28:33.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:33.858+0000 7f3c092fd700 1 librados: starting objecter 2024-04-14T15:28:33.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:33.858+0000 7f3c092fd700 1 Processor -- start 2024-04-14T15:28:33.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:33.858+0000 7f3c092fd700 1 -- start start 2024-04-14T15:28:33.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:33.858+0000 7f3c092fd700 1 librados: setting wanted keys 2024-04-14T15:28:33.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:33.858+0000 7f3c092fd700 1 librados: calling monclient init 2024-04-14T15:28:33.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:33.858+0000 7f3c092fd700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3c041a9e30 con 0x7f3c04109e10 2024-04-14T15:28:33.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:33.858+0000 7f3c092fd700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3c041ab050 con 0x7f3c0410d9b0 2024-04-14T15:28:33.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:33.858+0000 7f3c092fd700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3c041ac290 con 0x7f3c041012e0 2024-04-14T15:28:33.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:33.858+0000 7f3c02ffd700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f3c041012e0 0x7f3c04115290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:39710/0 (socket says 172.21.15.90:39710) 2024-04-14T15:28:33.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:33.858+0000 7f3c02ffd700 1 -- 172.21.15.90:0/435281984 learned_addr learned my addr 172.21.15.90:0/435281984 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:28:33.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:33.858+0000 7f3bebfff700 10 client.?.objecter ms_handle_connect 0x7f3c04109e10 2024-04-14T15:28:33.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:33.858+0000 7f3bebfff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:28:33.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:33.858+0000 7f3bebfff700 10 client.?.objecter ms_handle_connect 0x7f3c041012e0 2024-04-14T15:28:33.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:33.858+0000 7f3bebfff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:28:33.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:33.859+0000 7f3bebfff700 1 -- 172.21.15.90:0/435281984 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3024116530 0 0) 0x7f3c041a9e30 con 0x7f3c04109e10 2024-04-14T15:28:33.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:33.859+0000 7f3bebfff700 1 -- 172.21.15.90:0/435281984 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3bec0035b0 con 0x7f3c04109e10 2024-04-14T15:28:33.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:33.859+0000 7f3bebfff700 10 client.?.objecter ms_handle_connect 0x7f3c0410d9b0 2024-04-14T15:28:33.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:33.859+0000 7f3bebfff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:28:33.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:33.859+0000 7f3bebfff700 1 -- 172.21.15.90:0/435281984 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1320106252 0 0) 0x7f3c041ac290 con 0x7f3c041012e0 2024-04-14T15:28:33.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:33.859+0000 7f3bebfff700 1 -- 172.21.15.90:0/435281984 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3c041a9e30 con 0x7f3c041012e0 2024-04-14T15:28:33.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:33.859+0000 7f3bebfff700 1 -- 172.21.15.90:0/435281984 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1269551076 0 0) 0x7f3c041ab050 con 0x7f3c0410d9b0 2024-04-14T15:28:33.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:33.859+0000 7f3bebfff700 1 -- 172.21.15.90:0/435281984 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3c041ac290 con 0x7f3c0410d9b0 2024-04-14T15:28:33.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:33.859+0000 7f3bebfff700 1 -- 172.21.15.90:0/435281984 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4196939392 0 0) 0x7f3bec0035b0 con 0x7f3c04109e10 2024-04-14T15:28:33.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:33.859+0000 7f3bebfff700 1 -- 172.21.15.90:0/435281984 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3c041ab050 con 0x7f3c04109e10 2024-04-14T15:28:33.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:33.859+0000 7f3bebfff700 1 -- 172.21.15.90:0/435281984 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f3bf40018b0 con 0x7f3c04109e10 2024-04-14T15:28:33.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:33.859+0000 7f3bebfff700 1 -- 172.21.15.90:0/435281984 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1354528201 0 0) 0x7f3c041a9e30 con 0x7f3c041012e0 2024-04-14T15:28:33.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:33.859+0000 7f3bebfff700 1 -- 172.21.15.90:0/435281984 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3bec0035b0 con 0x7f3c041012e0 2024-04-14T15:28:33.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:33.859+0000 7f3bebfff700 1 -- 172.21.15.90:0/435281984 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f3bf8001970 con 0x7f3c041012e0 2024-04-14T15:28:33.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:33.859+0000 7f3bebfff700 1 -- 172.21.15.90:0/435281984 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4006050462 0 0) 0x7f3c041ab050 con 0x7f3c04109e10 2024-04-14T15:28:33.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:33.859+0000 7f3bebfff700 1 -- 172.21.15.90:0/435281984 >> v1:172.21.15.90:6790/0 conn(0x7f3c041012e0 legacy=0x7f3c04115290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:33.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:33.859+0000 7f3bebfff700 1 -- 172.21.15.90:0/435281984 >> v1:172.21.15.92:6789/0 conn(0x7f3c0410d9b0 legacy=0x7f3c041a84f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:33.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:33.859+0000 7f3bebfff700 1 -- 172.21.15.90:0/435281984 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c041ad4d0 con 0x7f3c04109e10 2024-04-14T15:28:33.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:33.859+0000 7f3c092fd700 1 -- 172.21.15.90:0/435281984 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3c041ac4c0 con 0x7f3c04109e10 2024-04-14T15:28:33.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:33.859+0000 7f3c092fd700 10 client.38313.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:28:33.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:33.859+0000 7f3c092fd700 1 -- 172.21.15.90:0/435281984 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f3c041ac980 con 0x7f3c04109e10 2024-04-14T15:28:33.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:33.859+0000 7f3c092fd700 1 librados: init done 2024-04-14T15:28:33.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:33.859+0000 7f3bebfff700 1 -- 172.21.15.90:0/435281984 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f3bf4001c80 con 0x7f3c04109e10 2024-04-14T15:28:33.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:33.859+0000 7f3bebfff700 1 -- 172.21.15.90:0/435281984 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f3bf4004b10 con 0x7f3c04109e10 2024-04-14T15:28:33.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:33.860+0000 7f3c092fd700 1 -- 172.21.15.90:0/435281984 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bf0005120 con 0x7f3c04109e10 2024-04-14T15:28:33.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:33.861+0000 7f3bebfff700 1 -- 172.21.15.90:0/435281984 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f3bf4003950 con 0x7f3c04109e10 2024-04-14T15:28:33.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:33.861+0000 7f3bebfff700 1 -- 172.21.15.90:0/435281984 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(917..917 src has 467..917) ==== 7517+0+0 (unknown 3390688648 0 0) 0x7f3bf4089530 con 0x7f3c04109e10 2024-04-14T15:28:33.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:33.861+0000 7f3bebfff700 10 client.38313.objecter ms_dispatch 0x7f3c04001550 osd_map(917..917 src has 467..917) 2024-04-14T15:28:33.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:33.861+0000 7f3bebfff700 3 client.38313.objecter handle_osd_map got epochs [917,917] > 0 2024-04-14T15:28:33.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:33.861+0000 7f3bebfff700 3 client.38313.objecter handle_osd_map decoding full epoch 917 2024-04-14T15:28:33.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:33.861+0000 7f3bebfff700 20 client.38313.objecter dump_active .. 0 homeless 2024-04-14T15:28:33.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:33.862+0000 7f3bebfff700 10 client.38313.objecter ms_handle_connect 0x7f3bec071120 2024-04-14T15:28:33.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:33.866+0000 7f3bebfff700 1 -- 172.21.15.90:0/435281984 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f3bf40561d0 con 0x7f3c04109e10 2024-04-14T15:28:34.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:34.068+0000 7f3c092fd700 1 -- 172.21.15.90:0/435281984 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3bf0005410 con 0x7f3c04109e10 2024-04-14T15:28:34.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:34.070+0000 7f3bebfff700 1 -- 172.21.15.90:0/435281984 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v917) ==== 74+0+24214 (unknown 885477452 0 2543525616) 0x7f3bf4059e80 con 0x7f3c04109e10 2024-04-14T15:28:34.071 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:28:34.071 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":917,"fsid":"0e288098-fa70-11ee-bc8e-c7b262605968","created":"2024-04-14T15:04:05.808313+0000","modified":"2024-04-14T15:28:33.261541+0000","last_up_change":"2024-04-14T15:09:29.769352+0000","last_in_change":"2024-04-14T15:27:13.165962+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":119,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":345,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-14T15:07:23.959628+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"1.b","ready_epoch":917,"last_epoch_started":916,"last_epoch_clean":916,"source_version":"91'130","target_version":"91'130"},"last_change":"917","last_force_op_resend":"0","last_force_op_resend_prenautilus":"917","last_force_op_resend_preluminous":"915","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.1800000667572021,"score_stable":2.1800000667572021,"optimal_score":1,"raw_score_acting":2.1800000667572021,"raw_score_stable":2.1800000667572021,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-04-14T15:09:41.608420+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"datapool","create_time":"2024-04-14T15:09:41.744391+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":69,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-04-14T15:09:43.088157+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-04-14T15:09:45.042722+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-04-14T15:09:47.111336+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"ceph_test_rados_api_asio","create_time":"2024-04-14T15:12:54.225671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"76772cfc-89d1-4013-b617-32284ed8dfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":901,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6801","nonce":3761806885}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6802","nonce":3761806885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6804","nonce":3761806885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6803","nonce":3761806885}]},"public_addr":"172.21.15.90:6801/3761806885","cluster_addr":"172.21.15.90:6802/3761806885","heartbeat_back_addr":"172.21.15.90:6804/3761806885","heartbeat_front_addr":"172.21.15.90:6803/3761806885","state":["exists","up"]},{"osd":1,"uuid":"de4a7fa2-7d57-4ff3-a4fa-28c0cde8ef6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":915,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6805","nonce":4170361699}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6806","nonce":4170361699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6808","nonce":4170361699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6807","nonce":4170361699}]},"public_addr":"172.21.15.90:6805/4170361699","cluster_addr":"172.21.15.90:6806/4170361699","heartbeat_back_addr":"172.21.15.90:6808/4170361699","heartbeat_front_addr":"172.21.15.90:6807/4170361699","state":["exists","up"]},{"osd":2,"uuid":"9c39b5ff-6175-46ba-9d71-370581c6ff3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":865,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6809","nonce":1686184251}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6810","nonce":1686184251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6812","nonce":1686184251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6811","nonce":1686184251}]},"public_addr":"172.21.15.90:6809/1686184251","cluster_addr":"172.21.15.90:6810/1686184251","heartbeat_back_addr":"172.21.15.90:6812/1686184251","heartbeat_front_addr":"172.21.15.90:6811/1686184251","state":["exists","up"]},{"osd":3,"uuid":"efd2ca45-5b80-4013-a571-499e2cb5b2e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":914,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6813","nonce":2270166774}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6814","nonce":2270166774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6816","nonce":2270166774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6815","nonce":2270166774}]},"public_addr":"172.21.15.90:6813/2270166774","cluster_addr":"172.21.15.90:6814/2270166774","heartbeat_back_addr":"172.21.15.90:6816/2270166774","heartbeat_front_addr":"172.21.15.90:6815/2270166774","state":["exists","up"]},{"osd":4,"uuid":"e3d9013d-5eb6-433c-96b3-3fc38ba7ac8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":910,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6800","nonce":3887549733}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6801","nonce":3887549733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6803","nonce":3887549733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6802","nonce":3887549733}]},"public_addr":"172.21.15.92:6800/3887549733","cluster_addr":"172.21.15.92:6801/3887549733","heartbeat_back_addr":"172.21.15.92:6803/3887549733","heartbeat_front_addr":"172.21.15.92:6802/3887549733","state":["exists","up"]},{"osd":5,"uuid":"1ec41805-6f82-4139-81bc-75305ba9e958","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":883,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6804","nonce":3244561440}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6805","nonce":3244561440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6807","nonce":3244561440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6806","nonce":3244561440}]},"public_addr":"172.21.15.92:6804/3244561440","cluster_addr":"172.21.15.92:6805/3244561440","heartbeat_back_addr":"172.21.15.92:6807/3244561440","heartbeat_front_addr":"172.21.15.92:6806/3244561440","state":["exists","up"]},{"osd":6,"uuid":"ca977ab2-bf8d-4b3a-ae5e-0efa5bc7a3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":906,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6808","nonce":636125963}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6809","nonce":636125963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6811","nonce":636125963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6810","nonce":636125963}]},"public_addr":"172.21.15.92:6808/636125963","cluster_addr":"172.21.15.92:6809/636125963","heartbeat_back_addr":"172.21.15.92:6811/636125963","heartbeat_front_addr":"172.21.15.92:6810/636125963","state":["exists","up"]},{"osd":7,"uuid":"e5354645-79ec-4d4c-b6e5-f9b0653915ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":887,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6812","nonce":1773706716}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6813","nonce":1773706716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6815","nonce":1773706716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6814","nonce":1773706716}]},"public_addr":"172.21.15.92:6812/1773706716","cluster_addr":"172.21.15.92:6813/1773706716","heartbeat_back_addr":"172.21.15.92:6815/1773706716","heartbeat_front_addr":"172.21.15.92:6814/1773706716","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T15:27:41.248761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T15:27:47.498189+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T15:27:18.377262+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T15:27:41.356530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T15:28:10.850534+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T15:28:15.686886+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T15:27:15.597391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T15:28:28.608598+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.1d","mappings":[{"from":7,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.12","mappings":[{"from":1,"to":5}]},{"pgid":"4.15","mappings":[{"from":3,"to":2}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]},{"pgid":"11.1c","mappings":[{"from":2,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:0/508768479":"2024-04-15T15:10:22.918674+0000","172.21.15.90:0/1823072952":"2024-04-15T15:10:22.918674+0000","172.21.15.90:6800/491071118":"2024-04-15T15:10:22.918674+0000","172.21.15.90:0/1337064035":"2024-04-15T15:10:22.918674+0000","172.21.15.90:0/769462306":"2024-04-15T15:10:22.918674+0000","172.21.15.90:0/1139868610":"2024-04-15T15:05:05.882361+0000","172.21.15.90:0/93763408":"2024-04-15T15:10:22.918674+0000","172.21.15.90:0/977818204":"2024-04-15T15:04:39.251711+0000","172.21.15.90:6800/941091171":"2024-04-15T15:05:05.882361+0000","172.21.15.90:0/3671453561":"2024-04-15T15:05:05.882361+0000","172.21.15.90:0/2611856200":"2024-04-15T15:05:05.882361+0000","172.21.15.90:6800/2306258962":"2024-04-15T15:04:39.251711+0000","172.21.15.90:0/4051692380":"2024-04-15T15:04:39.251711+0000","172.21.15.90:0/2681502561":"2024-04-15T15:04:39.251711+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":5}]},{"pool":18,"snaps":[{"begin":2,"length":3}]},{"pool":39,"snaps":[{"begin":1,"length":1}]},{"pool":72,"snaps":[{"begin":1,"length":1}]},{"pool":99,"snaps":[{"begin":4,"length":2},{"begin":7,"length":2}]},{"pool":109,"snaps":[{"begin":1,"length":1}]},{"pool":128,"snaps":[{"begin":1,"length":2}]},{"pool":142,"snaps":[{"begin":3,"length":1}]},{"pool":149,"snaps":[{"begin":1,"length":4}]},{"pool":156,"snaps":[{"begin":2,"length":1}]},{"pool":157,"snaps":[{"begin":2,"length":2}]},{"pool":167,"snaps":[{"begin":2,"length":3}]},{"pool":172,"snaps":[{"begin":3,"length":4}]},{"pool":175,"snaps":[{"begin":2,"length":4}]},{"pool":176,"snaps":[{"begin":2,"length":2}]},{"pool":187,"snaps":[{"begin":2,"length":1}]},{"pool":192,"snaps":[{"begin":1,"length":4}]},{"pool":205,"snaps":[{"begin":2,"length":6}]},{"pool":213,"snaps":[{"begin":7,"length":1}]},{"pool":234,"snaps":[{"begin":8,"length":1}]},{"pool":247,"snaps":[{"begin":10,"length":1}]},{"pool":252,"snaps":[{"begin":11,"length":2}]},{"pool":265,"snaps":[{"begin":14,"length":1}]},{"pool":326,"snaps":[{"begin":2,"length":1}]},{"pool":327,"snaps":[{"begin":3,"length":1}]},{"pool":330,"snaps":[{"begin":4,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-14T15:28:34.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:34.073+0000 7f3c092fd700 10 librados: watch_flush enter 2024-04-14T15:28:34.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:34.073+0000 7f3c092fd700 10 librados: watch_flush exit 2024-04-14T15:28:34.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:34.073+0000 7f3c092fd700 20 client.38313.objecter shutdown clearing up homeless session... 2024-04-14T15:28:34.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:34.073+0000 7f3c092fd700 10 client.38313.objecter successfully canceled tick 2024-04-14T15:28:34.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:34.073+0000 7f3c092fd700 1 -- 172.21.15.90:0/435281984 >> v1:172.21.15.90:6800/420935427 conn(0x7f3bec071120 legacy=0x7f3bec0735e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:34.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:34.073+0000 7f3c092fd700 1 -- 172.21.15.90:0/435281984 >> v1:172.21.15.90:6789/0 conn(0x7f3c04109e10 legacy=0x7f3c041159a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:34.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:34.073+0000 7f3c092fd700 1 -- 172.21.15.90:0/435281984 shutdown_connections 2024-04-14T15:28:34.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:34.073+0000 7f3c092fd700 1 -- 172.21.15.90:0/435281984 >> 172.21.15.90:0/435281984 conn(0x7f3c04076aa0 msgr2=0x7f3c0410e690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:28:34.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:34.073+0000 7f3c092fd700 1 -- 172.21.15.90:0/435281984 shutdown_connections 2024-04-14T15:28:34.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:34.073+0000 7f3c092fd700 1 -- 172.21.15.90:0/435281984 wait complete. 2024-04-14T15:28:34.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:34.073+0000 7f3c092fd700 1 librados: shutdown 2024-04-14T15:28:34.191 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:28:33 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:28:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:28:34.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:34 smithi092 ceph-mon[33966]: 6.17 scrub starts 2024-04-14T15:28:34.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:34 smithi092 ceph-mon[33966]: 6.17 scrub ok 2024-04-14T15:28:34.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:34 smithi092 ceph-mon[33966]: 1.4 scrub starts 2024-04-14T15:28:34.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:34 smithi092 ceph-mon[33966]: 1.4 scrub ok 2024-04-14T15:28:34.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:34 smithi092 ceph-mon[33966]: 5.1e scrub starts 2024-04-14T15:28:34.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:34 smithi092 ceph-mon[33966]: 5.1e scrub ok 2024-04-14T15:28:34.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:34 smithi092 ceph-mon[33966]: pgmap v1385: 175 pgs: 1 peering, 174 active+clean; 262 KiB data, 835 MiB used, 714 GiB / 715 GiB avail 2024-04-14T15:28:34.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:34 smithi092 ceph-mon[33966]: osdmap e917: 8 total, 8 up, 8 in 2024-04-14T15:28:34.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:34 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/435281984' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-14T15:28:34.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:34 smithi092 ceph-mon[33966]: osdmap e918: 8 total, 8 up, 8 in 2024-04-14T15:28:34.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:34 smithi090 ceph-mon[30399]: 6.17 scrub starts 2024-04-14T15:28:34.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:34 smithi090 ceph-mon[30399]: 6.17 scrub ok 2024-04-14T15:28:34.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:34 smithi090 ceph-mon[30399]: 1.4 scrub starts 2024-04-14T15:28:34.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:34 smithi090 ceph-mon[30399]: 1.4 scrub ok 2024-04-14T15:28:34.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:34 smithi090 ceph-mon[30399]: 5.1e scrub starts 2024-04-14T15:28:34.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:34 smithi090 ceph-mon[30399]: 5.1e scrub ok 2024-04-14T15:28:34.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:34 smithi090 ceph-mon[30399]: pgmap v1385: 175 pgs: 1 peering, 174 active+clean; 262 KiB data, 835 MiB used, 714 GiB / 715 GiB avail 2024-04-14T15:28:34.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:34 smithi090 ceph-mon[30399]: osdmap e917: 8 total, 8 up, 8 in 2024-04-14T15:28:34.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:34 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/435281984' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-14T15:28:34.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:34 smithi090 ceph-mon[30399]: osdmap e918: 8 total, 8 up, 8 in 2024-04-14T15:28:34.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:34 smithi090 ceph-mon[41670]: 6.17 scrub starts 2024-04-14T15:28:34.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:34 smithi090 ceph-mon[41670]: 6.17 scrub ok 2024-04-14T15:28:34.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:34 smithi090 ceph-mon[41670]: 1.4 scrub starts 2024-04-14T15:28:34.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:34 smithi090 ceph-mon[41670]: 1.4 scrub ok 2024-04-14T15:28:34.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:34 smithi090 ceph-mon[41670]: 5.1e scrub starts 2024-04-14T15:28:34.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:34 smithi090 ceph-mon[41670]: 5.1e scrub ok 2024-04-14T15:28:34.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:34 smithi090 ceph-mon[41670]: pgmap v1385: 175 pgs: 1 peering, 174 active+clean; 262 KiB data, 835 MiB used, 714 GiB / 715 GiB avail 2024-04-14T15:28:34.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:34 smithi090 ceph-mon[41670]: osdmap e917: 8 total, 8 up, 8 in 2024-04-14T15:28:34.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:34 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/435281984' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-14T15:28:34.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:34 smithi090 ceph-mon[41670]: osdmap e918: 8 total, 8 up, 8 in 2024-04-14T15:28:34.839 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-14T15:28:34.839 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-14T15:28:34.840 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-14T15:28:34.840 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-14T15:28:34.841 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-14T15:28:34.841 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-14T15:28:34.841 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-14T15:28:34.842 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-14T15:28:35.408 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:28:35.409 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:28:35.409 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:28:35.409 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:28:35.411 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:28:35.411 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:28:35.417 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:28:35.418 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:28:35.547 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:35 smithi090 ceph-mon[41670]: 6.1c scrub starts 2024-04-14T15:28:35.547 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:35 smithi090 ceph-mon[41670]: 6.1c scrub ok 2024-04-14T15:28:35.547 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:35 smithi090 ceph-mon[41670]: 2.c scrub starts 2024-04-14T15:28:35.547 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:35 smithi090 ceph-mon[41670]: 2.c scrub ok 2024-04-14T15:28:35.548 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:35 smithi090 ceph-mon[30399]: 6.1c scrub starts 2024-04-14T15:28:35.548 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:35 smithi090 ceph-mon[30399]: 6.1c scrub ok 2024-04-14T15:28:35.548 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:35 smithi090 ceph-mon[30399]: 2.c scrub starts 2024-04-14T15:28:35.548 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:35 smithi090 ceph-mon[30399]: 2.c scrub ok 2024-04-14T15:28:35.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:35 smithi092 ceph-mon[33966]: 6.1c scrub starts 2024-04-14T15:28:35.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:35 smithi092 ceph-mon[33966]: 6.1c scrub ok 2024-04-14T15:28:35.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:35 smithi092 ceph-mon[33966]: 2.c scrub starts 2024-04-14T15:28:35.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:35 smithi092 ceph-mon[33966]: 2.c scrub ok 2024-04-14T15:28:36.122 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:28:35 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:28:36.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:36 smithi090 ceph-mon[41670]: pgmap v1388: 174 pgs: 1 peering, 173 active+clean; 582 KiB data, 839 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 80 KiB/s, 0 objects/s recovering 2024-04-14T15:28:36.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:36 smithi090 ceph-mon[41670]: 2.10 scrub starts 2024-04-14T15:28:36.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:36 smithi090 ceph-mon[41670]: 2.10 scrub ok 2024-04-14T15:28:36.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:36 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:28:36.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:36 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:28:36.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:36 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:28:36.553 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:36 smithi090 ceph-mon[30399]: pgmap v1388: 174 pgs: 1 peering, 173 active+clean; 582 KiB data, 839 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 80 KiB/s, 0 objects/s recovering 2024-04-14T15:28:36.553 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:36 smithi090 ceph-mon[30399]: 2.10 scrub starts 2024-04-14T15:28:36.553 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:36 smithi090 ceph-mon[30399]: 2.10 scrub ok 2024-04-14T15:28:36.553 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:36 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:28:36.553 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:36 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:28:36.554 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:36 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:28:36.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:36 smithi092 ceph-mon[33966]: pgmap v1388: 174 pgs: 1 peering, 173 active+clean; 582 KiB data, 839 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 80 KiB/s, 0 objects/s recovering 2024-04-14T15:28:36.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:36 smithi092 ceph-mon[33966]: 2.10 scrub starts 2024-04-14T15:28:36.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:36 smithi092 ceph-mon[33966]: 2.10 scrub ok 2024-04-14T15:28:36.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:36 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:28:36.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:36 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:28:36.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:36 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:28:37.441 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:37 smithi090 ceph-mon[41670]: 4.c scrub starts 2024-04-14T15:28:37.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:37 smithi090 ceph-mon[41670]: 4.c scrub ok 2024-04-14T15:28:37.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:37 smithi090 ceph-mon[41670]: 6.3 scrub starts 2024-04-14T15:28:37.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:37 smithi090 ceph-mon[41670]: 6.3 scrub ok 2024-04-14T15:28:37.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:37 smithi090 ceph-mon[41670]: osdmap e919: 8 total, 8 up, 8 in 2024-04-14T15:28:37.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:37 smithi090 ceph-mon[41670]: 2.17 scrub starts 2024-04-14T15:28:37.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:37 smithi090 ceph-mon[41670]: 2.17 scrub ok 2024-04-14T15:28:37.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:37 smithi090 ceph-mon[30399]: 4.c scrub starts 2024-04-14T15:28:37.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:37 smithi090 ceph-mon[30399]: 4.c scrub ok 2024-04-14T15:28:37.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:37 smithi090 ceph-mon[30399]: 6.3 scrub starts 2024-04-14T15:28:37.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:37 smithi090 ceph-mon[30399]: 6.3 scrub ok 2024-04-14T15:28:37.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:37 smithi090 ceph-mon[30399]: osdmap e919: 8 total, 8 up, 8 in 2024-04-14T15:28:37.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:37 smithi090 ceph-mon[30399]: 2.17 scrub starts 2024-04-14T15:28:37.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:37 smithi090 ceph-mon[30399]: 2.17 scrub ok 2024-04-14T15:28:37.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:37 smithi092 ceph-mon[33966]: 4.c scrub starts 2024-04-14T15:28:37.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:37 smithi092 ceph-mon[33966]: 4.c scrub ok 2024-04-14T15:28:37.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:37 smithi092 ceph-mon[33966]: 6.3 scrub starts 2024-04-14T15:28:37.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:37 smithi092 ceph-mon[33966]: 6.3 scrub ok 2024-04-14T15:28:37.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:37 smithi092 ceph-mon[33966]: osdmap e919: 8 total, 8 up, 8 in 2024-04-14T15:28:37.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:37 smithi092 ceph-mon[33966]: 2.17 scrub starts 2024-04-14T15:28:37.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:37 smithi092 ceph-mon[33966]: 2.17 scrub ok 2024-04-14T15:28:38.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:38 smithi092 ceph-mon[33966]: 4.13 scrub starts 2024-04-14T15:28:38.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:38 smithi092 ceph-mon[33966]: 4.13 scrub ok 2024-04-14T15:28:38.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:38 smithi092 ceph-mon[33966]: pgmap v1390: 174 pgs: 1 peering, 173 active+clean; 582 KiB data, 839 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 64 KiB/s, 0 objects/s recovering 2024-04-14T15:28:38.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:38 smithi092 ceph-mon[33966]: 6.1d scrub starts 2024-04-14T15:28:38.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:38 smithi092 ceph-mon[33966]: 6.1d scrub ok 2024-04-14T15:28:38.691 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:38 smithi090 ceph-mon[30399]: 4.13 scrub starts 2024-04-14T15:28:38.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:38 smithi090 ceph-mon[30399]: 4.13 scrub ok 2024-04-14T15:28:38.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:38 smithi090 ceph-mon[30399]: pgmap v1390: 174 pgs: 1 peering, 173 active+clean; 582 KiB data, 839 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 64 KiB/s, 0 objects/s recovering 2024-04-14T15:28:38.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:38 smithi090 ceph-mon[30399]: 6.1d scrub starts 2024-04-14T15:28:38.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:38 smithi090 ceph-mon[30399]: 6.1d scrub ok 2024-04-14T15:28:38.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:38 smithi090 ceph-mon[41670]: 4.13 scrub starts 2024-04-14T15:28:38.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:38 smithi090 ceph-mon[41670]: 4.13 scrub ok 2024-04-14T15:28:38.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:38 smithi090 ceph-mon[41670]: pgmap v1390: 174 pgs: 1 peering, 173 active+clean; 582 KiB data, 839 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 64 KiB/s, 0 objects/s recovering 2024-04-14T15:28:38.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:38 smithi090 ceph-mon[41670]: 6.1d scrub starts 2024-04-14T15:28:38.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:38 smithi090 ceph-mon[41670]: 6.1d scrub ok 2024-04-14T15:28:39.471 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:39 smithi090 ceph-mon[41670]: 4.9 scrub starts 2024-04-14T15:28:39.472 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:39 smithi090 ceph-mon[41670]: 4.9 scrub ok 2024-04-14T15:28:39.472 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:39 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:28:39.472 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:39 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:28:39.472 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:39 smithi090 ceph-mon[41670]: 2.b scrub starts 2024-04-14T15:28:39.472 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:39 smithi090 ceph-mon[41670]: 2.b scrub ok 2024-04-14T15:28:39.472 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:39 smithi090 ceph-mon[41670]: osdmap e920: 8 total, 8 up, 8 in 2024-04-14T15:28:39.472 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:39 smithi090 ceph-mon[41670]: 5.16 scrub starts 2024-04-14T15:28:39.472 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:39 smithi090 ceph-mon[41670]: 5.16 scrub ok 2024-04-14T15:28:39.472 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:39 smithi090 ceph-mon[30399]: 4.9 scrub starts 2024-04-14T15:28:39.472 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:39 smithi090 ceph-mon[30399]: 4.9 scrub ok 2024-04-14T15:28:39.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:39 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:28:39.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:39 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:28:39.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:39 smithi090 ceph-mon[30399]: 2.b scrub starts 2024-04-14T15:28:39.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:39 smithi090 ceph-mon[30399]: 2.b scrub ok 2024-04-14T15:28:39.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:39 smithi090 ceph-mon[30399]: osdmap e920: 8 total, 8 up, 8 in 2024-04-14T15:28:39.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:39 smithi090 ceph-mon[30399]: 5.16 scrub starts 2024-04-14T15:28:39.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:39 smithi090 ceph-mon[30399]: 5.16 scrub ok 2024-04-14T15:28:39.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:39 smithi092 ceph-mon[33966]: 4.9 scrub starts 2024-04-14T15:28:39.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:39 smithi092 ceph-mon[33966]: 4.9 scrub ok 2024-04-14T15:28:39.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:39 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:28:39.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:39 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:28:39.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:39 smithi092 ceph-mon[33966]: 2.b scrub starts 2024-04-14T15:28:39.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:39 smithi092 ceph-mon[33966]: 2.b scrub ok 2024-04-14T15:28:39.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:39 smithi092 ceph-mon[33966]: osdmap e920: 8 total, 8 up, 8 in 2024-04-14T15:28:39.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:39 smithi092 ceph-mon[33966]: 5.16 scrub starts 2024-04-14T15:28:39.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:39 smithi092 ceph-mon[33966]: 5.16 scrub ok 2024-04-14T15:28:40.484 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:40 smithi090 ceph-mon[41670]: 5.3 deep-scrub starts 2024-04-14T15:28:40.484 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:40 smithi090 ceph-mon[41670]: 5.3 deep-scrub ok 2024-04-14T15:28:40.485 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:40 smithi090 ceph-mon[41670]: pgmap v1392: 174 pgs: 174 active+clean; 582 KiB data, 839 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 54 KiB/s, 0 objects/s recovering 2024-04-14T15:28:40.485 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:40 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:28:40.485 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:40 smithi090 ceph-mon[41670]: 5.d scrub starts 2024-04-14T15:28:40.485 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:40 smithi090 ceph-mon[41670]: 5.d scrub ok 2024-04-14T15:28:40.485 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:40 smithi090 ceph-mon[30399]: 5.3 deep-scrub starts 2024-04-14T15:28:40.485 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:40 smithi090 ceph-mon[30399]: 5.3 deep-scrub ok 2024-04-14T15:28:40.485 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:40 smithi090 ceph-mon[30399]: pgmap v1392: 174 pgs: 174 active+clean; 582 KiB data, 839 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 54 KiB/s, 0 objects/s recovering 2024-04-14T15:28:40.485 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:40 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:28:40.485 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:40 smithi090 ceph-mon[30399]: 5.d scrub starts 2024-04-14T15:28:40.485 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:40 smithi090 ceph-mon[30399]: 5.d scrub ok 2024-04-14T15:28:40.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:40 smithi092 ceph-mon[33966]: 5.3 deep-scrub starts 2024-04-14T15:28:40.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:40 smithi092 ceph-mon[33966]: 5.3 deep-scrub ok 2024-04-14T15:28:40.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:40 smithi092 ceph-mon[33966]: pgmap v1392: 174 pgs: 174 active+clean; 582 KiB data, 839 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 54 KiB/s, 0 objects/s recovering 2024-04-14T15:28:40.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:40 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:28:40.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:40 smithi092 ceph-mon[33966]: 5.d scrub starts 2024-04-14T15:28:40.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:40 smithi092 ceph-mon[33966]: 5.d scrub ok 2024-04-14T15:28:41.619 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:41 smithi090 ceph-mon[30399]: 4.b scrub starts 2024-04-14T15:28:41.620 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:41 smithi090 ceph-mon[30399]: 4.b scrub ok 2024-04-14T15:28:41.620 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:41 smithi090 ceph-mon[41670]: 4.b scrub starts 2024-04-14T15:28:41.620 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:41 smithi090 ceph-mon[41670]: 4.b scrub ok 2024-04-14T15:28:41.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:41 smithi092 ceph-mon[33966]: 4.b scrub starts 2024-04-14T15:28:41.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:41 smithi092 ceph-mon[33966]: 4.b scrub ok 2024-04-14T15:28:42.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:42 smithi090 ceph-mon[30399]: 6.1a deep-scrub starts 2024-04-14T15:28:42.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:42 smithi090 ceph-mon[30399]: 6.1a deep-scrub ok 2024-04-14T15:28:42.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:42 smithi090 ceph-mon[30399]: pgmap v1393: 174 pgs: 174 active+clean; 582 KiB data, 839 MiB used, 714 GiB / 715 GiB avail; 733 B/s rd, 0 op/s 2024-04-14T15:28:42.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:42 smithi090 ceph-mon[30399]: 4.f scrub starts 2024-04-14T15:28:42.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:42 smithi090 ceph-mon[30399]: 4.f scrub ok 2024-04-14T15:28:42.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:42 smithi090 ceph-mon[30399]: 4.1f scrub starts 2024-04-14T15:28:42.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:42 smithi090 ceph-mon[30399]: 4.1f scrub ok 2024-04-14T15:28:42.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:42 smithi090 ceph-mon[41670]: 6.1a deep-scrub starts 2024-04-14T15:28:42.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:42 smithi090 ceph-mon[41670]: 6.1a deep-scrub ok 2024-04-14T15:28:42.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:42 smithi090 ceph-mon[41670]: pgmap v1393: 174 pgs: 174 active+clean; 582 KiB data, 839 MiB used, 714 GiB / 715 GiB avail; 733 B/s rd, 0 op/s 2024-04-14T15:28:42.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:42 smithi090 ceph-mon[41670]: 4.f scrub starts 2024-04-14T15:28:42.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:42 smithi090 ceph-mon[41670]: 4.f scrub ok 2024-04-14T15:28:42.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:42 smithi090 ceph-mon[41670]: 4.1f scrub starts 2024-04-14T15:28:42.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:42 smithi090 ceph-mon[41670]: 4.1f scrub ok 2024-04-14T15:28:42.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:42 smithi092 ceph-mon[33966]: 6.1a deep-scrub starts 2024-04-14T15:28:42.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:42 smithi092 ceph-mon[33966]: 6.1a deep-scrub ok 2024-04-14T15:28:42.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:42 smithi092 ceph-mon[33966]: pgmap v1393: 174 pgs: 174 active+clean; 582 KiB data, 839 MiB used, 714 GiB / 715 GiB avail; 733 B/s rd, 0 op/s 2024-04-14T15:28:42.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:42 smithi092 ceph-mon[33966]: 4.f scrub starts 2024-04-14T15:28:42.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:42 smithi092 ceph-mon[33966]: 4.f scrub ok 2024-04-14T15:28:42.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:42 smithi092 ceph-mon[33966]: 4.1f scrub starts 2024-04-14T15:28:42.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:42 smithi092 ceph-mon[33966]: 4.1f scrub ok 2024-04-14T15:28:43.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:43 smithi092 ceph-mon[33966]: 4.1d scrub starts 2024-04-14T15:28:43.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:43 smithi092 ceph-mon[33966]: 4.1d scrub ok 2024-04-14T15:28:43.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:43 smithi092 ceph-mon[33966]: 4.2 scrub starts 2024-04-14T15:28:43.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:43 smithi092 ceph-mon[33966]: 4.2 scrub ok 2024-04-14T15:28:43.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:43 smithi090 ceph-mon[41670]: 4.1d scrub starts 2024-04-14T15:28:43.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:43 smithi090 ceph-mon[41670]: 4.1d scrub ok 2024-04-14T15:28:43.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:43 smithi090 ceph-mon[41670]: 4.2 scrub starts 2024-04-14T15:28:43.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:43 smithi090 ceph-mon[41670]: 4.2 scrub ok 2024-04-14T15:28:43.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:43 smithi090 ceph-mon[30399]: 4.1d scrub starts 2024-04-14T15:28:43.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:43 smithi090 ceph-mon[30399]: 4.1d scrub ok 2024-04-14T15:28:43.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:43 smithi090 ceph-mon[30399]: 4.2 scrub starts 2024-04-14T15:28:43.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:43 smithi090 ceph-mon[30399]: 4.2 scrub ok 2024-04-14T15:28:44.191 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:28:43 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:28:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:28:44.578 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:44 smithi090 ceph-mon[41670]: 5.18 scrub starts 2024-04-14T15:28:44.578 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:44 smithi090 ceph-mon[41670]: 5.18 scrub ok 2024-04-14T15:28:44.578 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:44 smithi090 ceph-mon[41670]: 2.0 scrub starts 2024-04-14T15:28:44.578 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:44 smithi090 ceph-mon[41670]: 2.0 scrub ok 2024-04-14T15:28:44.578 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:44 smithi090 ceph-mon[41670]: pgmap v1394: 174 pgs: 174 active+clean; 582 KiB data, 839 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-14T15:28:44.578 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:44 smithi090 ceph-mon[30399]: 5.18 scrub starts 2024-04-14T15:28:44.578 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:44 smithi090 ceph-mon[30399]: 5.18 scrub ok 2024-04-14T15:28:44.578 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:44 smithi090 ceph-mon[30399]: 2.0 scrub starts 2024-04-14T15:28:44.578 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:44 smithi090 ceph-mon[30399]: 2.0 scrub ok 2024-04-14T15:28:44.579 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:44 smithi090 ceph-mon[30399]: pgmap v1394: 174 pgs: 174 active+clean; 582 KiB data, 839 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-14T15:28:44.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:44 smithi092 ceph-mon[33966]: 5.18 scrub starts 2024-04-14T15:28:44.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:44 smithi092 ceph-mon[33966]: 5.18 scrub ok 2024-04-14T15:28:44.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:44 smithi092 ceph-mon[33966]: 2.0 scrub starts 2024-04-14T15:28:44.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:44 smithi092 ceph-mon[33966]: 2.0 scrub ok 2024-04-14T15:28:44.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:44 smithi092 ceph-mon[33966]: pgmap v1394: 174 pgs: 174 active+clean; 582 KiB data, 839 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-14T15:28:44.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.681+0000 7fc2bf300700 1 -- 172.21.15.90:0/3586784142 >> v1:172.21.15.90:6790/0 conn(0x7fc2b8103ac0 legacy=0x7fc2b8103ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:44.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.681+0000 7fc2bf300700 1 -- 172.21.15.90:0/3586784142 shutdown_connections 2024-04-14T15:28:44.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.681+0000 7fc2bf300700 1 -- 172.21.15.90:0/3586784142 >> 172.21.15.90:0/3586784142 conn(0x7fc2b80fd290 msgr2=0x7fc2b80ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:28:44.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.681+0000 7fc2bf300700 1 -- 172.21.15.90:0/3586784142 shutdown_connections 2024-04-14T15:28:44.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.681+0000 7fc2bf300700 1 -- 172.21.15.90:0/3586784142 wait complete. 2024-04-14T15:28:44.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.682+0000 7fc2bf300700 1 librados: starting msgr at 2024-04-14T15:28:44.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.682+0000 7fc2bf300700 1 librados: starting objecter 2024-04-14T15:28:44.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.682+0000 7fc2bf300700 1 Processor -- start 2024-04-14T15:28:44.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.682+0000 7fc2bf300700 1 -- start start 2024-04-14T15:28:44.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.682+0000 7fc2bf300700 1 librados: setting wanted keys 2024-04-14T15:28:44.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.682+0000 7fc2bf300700 1 librados: calling monclient init 2024-04-14T15:28:44.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.682+0000 7fc2bf300700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc2b81a7940 con 0x7fc2b810b8d0 2024-04-14T15:28:44.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.682+0000 7fc2bf300700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc2b81a8b60 con 0x7fc2b810f3e0 2024-04-14T15:28:44.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.683+0000 7fc2bf300700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc2b81a9da0 con 0x7fc2b8103ac0 2024-04-14T15:28:44.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.683+0000 7fc2bd09c700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7fc2b8103ac0 0x7fc2b810b100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:54064/0 (socket says 172.21.15.90:54064) 2024-04-14T15:28:44.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.683+0000 7fc2bc89b700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fc2b810b8d0 0x7fc2b81a2760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:40708/0 (socket says 172.21.15.90:40708) 2024-04-14T15:28:44.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.683+0000 7fc2bd09c700 1 -- 172.21.15.90:0/2084580762 learned_addr learned my addr 172.21.15.90:0/2084580762 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:28:44.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.683+0000 7fc2ae7fc700 10 client.?.objecter ms_handle_connect 0x7fc2b810b8d0 2024-04-14T15:28:44.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.683+0000 7fc2ae7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:28:44.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.683+0000 7fc2ae7fc700 10 client.?.objecter ms_handle_connect 0x7fc2b8103ac0 2024-04-14T15:28:44.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.683+0000 7fc2ae7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:28:44.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.683+0000 7fc2ae7fc700 10 client.?.objecter ms_handle_connect 0x7fc2b810f3e0 2024-04-14T15:28:44.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.683+0000 7fc2ae7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:28:44.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.683+0000 7fc2ae7fc700 1 -- 172.21.15.90:0/2084580762 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1945371005 0 0) 0x7fc2b81a7940 con 0x7fc2b810b8d0 2024-04-14T15:28:44.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.683+0000 7fc2ae7fc700 1 -- 172.21.15.90:0/2084580762 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc2a4003aa0 con 0x7fc2b810b8d0 2024-04-14T15:28:44.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.683+0000 7fc2ae7fc700 1 -- 172.21.15.90:0/2084580762 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3730220277 0 0) 0x7fc2b81a9da0 con 0x7fc2b8103ac0 2024-04-14T15:28:44.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.683+0000 7fc2ae7fc700 1 -- 172.21.15.90:0/2084580762 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc2b81a7940 con 0x7fc2b8103ac0 2024-04-14T15:28:44.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.683+0000 7fc2ae7fc700 1 -- 172.21.15.90:0/2084580762 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 234016823 0 0) 0x7fc2b81a8b60 con 0x7fc2b810f3e0 2024-04-14T15:28:44.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.683+0000 7fc2ae7fc700 1 -- 172.21.15.90:0/2084580762 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc2b81a9da0 con 0x7fc2b810f3e0 2024-04-14T15:28:44.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.683+0000 7fc2ae7fc700 1 -- 172.21.15.90:0/2084580762 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2247382119 0 0) 0x7fc2a4003aa0 con 0x7fc2b810b8d0 2024-04-14T15:28:44.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.683+0000 7fc2ae7fc700 1 -- 172.21.15.90:0/2084580762 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc2b81a8b60 con 0x7fc2b810b8d0 2024-04-14T15:28:44.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.683+0000 7fc2ae7fc700 1 -- 172.21.15.90:0/2084580762 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fc2b4001a30 con 0x7fc2b810b8d0 2024-04-14T15:28:44.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.683+0000 7fc2ae7fc700 1 -- 172.21.15.90:0/2084580762 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3616418922 0 0) 0x7fc2b81a7940 con 0x7fc2b8103ac0 2024-04-14T15:28:44.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.683+0000 7fc2ae7fc700 1 -- 172.21.15.90:0/2084580762 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc2a4003aa0 con 0x7fc2b8103ac0 2024-04-14T15:28:44.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.683+0000 7fc2ae7fc700 1 -- 172.21.15.90:0/2084580762 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fc2a80015d0 con 0x7fc2b8103ac0 2024-04-14T15:28:44.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.683+0000 7fc2ae7fc700 1 -- 172.21.15.90:0/2084580762 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3476757855 0 0) 0x7fc2b81a9da0 con 0x7fc2b810f3e0 2024-04-14T15:28:44.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.683+0000 7fc2ae7fc700 1 -- 172.21.15.90:0/2084580762 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc2b81a7940 con 0x7fc2b810f3e0 2024-04-14T15:28:44.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.683+0000 7fc2ae7fc700 1 -- 172.21.15.90:0/2084580762 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fc2b0001af0 con 0x7fc2b810f3e0 2024-04-14T15:28:44.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.683+0000 7fc2ae7fc700 1 -- 172.21.15.90:0/2084580762 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2825585111 0 0) 0x7fc2b81a8b60 con 0x7fc2b810b8d0 2024-04-14T15:28:44.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.683+0000 7fc2ae7fc700 1 -- 172.21.15.90:0/2084580762 >> v1:172.21.15.90:6790/0 conn(0x7fc2b8103ac0 legacy=0x7fc2b810b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:44.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.683+0000 7fc2ae7fc700 1 -- 172.21.15.90:0/2084580762 >> v1:172.21.15.92:6789/0 conn(0x7fc2b810f3e0 legacy=0x7fc2b81a6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:44.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.683+0000 7fc2ae7fc700 1 -- 172.21.15.90:0/2084580762 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2b81aafe0 con 0x7fc2b810b8d0 2024-04-14T15:28:44.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.683+0000 7fc2bf300700 1 -- 172.21.15.90:0/2084580762 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc2b81a7b70 con 0x7fc2b810b8d0 2024-04-14T15:28:44.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.683+0000 7fc2bf300700 10 client.38319.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:28:44.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.683+0000 7fc2bf300700 1 -- 172.21.15.90:0/2084580762 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc2b81a8030 con 0x7fc2b810b8d0 2024-04-14T15:28:44.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.683+0000 7fc2bf300700 1 librados: init done 2024-04-14T15:28:44.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.683+0000 7fc2ae7fc700 1 -- 172.21.15.90:0/2084580762 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fc2b4001c90 con 0x7fc2b810b8d0 2024-04-14T15:28:44.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.683+0000 7fc2ae7fc700 1 -- 172.21.15.90:0/2084580762 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fc2b4005b70 con 0x7fc2b810b8d0 2024-04-14T15:28:44.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.684+0000 7fc2bf300700 10 client.38319.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:28:44.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.684+0000 7fc2bf300700 15 client.38319.objecter _session_command_op_assign -1 1 2024-04-14T15:28:44.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.684+0000 7fc2bf300700 20 client.38319.objecter _get_session osd=-1 returning homeless 2024-04-14T15:28:44.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.684+0000 7fc2bf300700 10 client.38319.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:28:44.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.684+0000 7fc2bf300700 1 -- 172.21.15.90:0/2084580762 --> v1:172.21.15.90:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fc29c002920 con 0x7fc2b810b8d0 2024-04-14T15:28:44.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.685+0000 7fc2ae7fc700 1 -- 172.21.15.90:0/2084580762 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fc2b4003930 con 0x7fc2b810b8d0 2024-04-14T15:28:44.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.685+0000 7fc2ae7fc700 1 -- 172.21.15.90:0/2084580762 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(920..920 src has 467..920) ==== 7517+0+0 (unknown 2649999903 0 0) 0x7fc2b408a5c0 con 0x7fc2b810b8d0 2024-04-14T15:28:44.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.685+0000 7fc2ae7fc700 10 client.38319.objecter ms_dispatch 0x7fc2b8001550 osd_map(920..920 src has 467..920) 2024-04-14T15:28:44.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.685+0000 7fc2ae7fc700 3 client.38319.objecter handle_osd_map got epochs [920,920] > 0 2024-04-14T15:28:44.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.685+0000 7fc2ae7fc700 3 client.38319.objecter handle_osd_map decoding full epoch 920 2024-04-14T15:28:44.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.686+0000 7fc2ae7fc700 10 client.38319.objecter checking command 1 2024-04-14T15:28:44.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.686+0000 7fc2ae7fc700 20 client.38319.objecter _get_session s=0x7fc2a407d340 osd=7 3 2024-04-14T15:28:44.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.686+0000 7fc2ae7fc700 20 client.38319.objecter put_session s=0x7fc2a407d340 osd=7 3 2024-04-14T15:28:44.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.686+0000 7fc2ae7fc700 15 client.38319.objecter _session_command_op_remove -1 1 2024-04-14T15:28:44.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.686+0000 7fc2ae7fc700 20 client.38319.objecter _get_session s=0x7fc2a407d340 osd=7 3 2024-04-14T15:28:44.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.686+0000 7fc2ae7fc700 20 client.38319.objecter get_session s=0x7fc2a407d340 osd=7 3 2024-04-14T15:28:44.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.686+0000 7fc2ae7fc700 15 client.38319.objecter _session_command_op_assign 7 1 2024-04-14T15:28:44.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.686+0000 7fc2ae7fc700 20 client.38319.objecter put_session s=0x7fc2a407d340 osd=7 4 2024-04-14T15:28:44.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.686+0000 7fc2ae7fc700 10 client.38319.objecter _send_command 1 2024-04-14T15:28:44.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.686+0000 7fc2ae7fc700 1 -- 172.21.15.90:0/2084580762 --> v1:172.21.15.92:6812/1773706716 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc2a4081700 con 0x7fc2a407d9f0 2024-04-14T15:28:44.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.686+0000 7fc2ae7fc700 20 client.38319.objecter dump_active .. 0 homeless 2024-04-14T15:28:44.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.686+0000 7fc2ae7fc700 10 client.38319.objecter ms_handle_connect 0x7fc2a40712b0 2024-04-14T15:28:44.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.686+0000 7fc2ae7fc700 1 -- 172.21.15.90:0/2084580762 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_get_version_reply(handle=1 version=920) ==== 24+0+0 (unknown 622753966 0 0) 0x7fc2b408a9d0 con 0x7fc2b810b8d0 2024-04-14T15:28:44.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.686+0000 7fc2ae7fc700 10 client.38319.objecter ms_handle_connect 0x7fc2a407d9f0 2024-04-14T15:28:44.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.687+0000 7fc2ae7fc700 1 -- 172.21.15.90:0/2084580762 <== osd.7 v1:172.21.15.92:6812/1773706716 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc2a4081700 con 0x7fc2a407d9f0 2024-04-14T15:28:44.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.687+0000 7fc2ae7fc700 10 client.38319.objecter ms_dispatch 0x7fc2b8001550 command_reply(tid 1: 0 ) 2024-04-14T15:28:44.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.687+0000 7fc2ae7fc700 10 client.38319.objecter _finish_command 1 = system:0 2024-04-14T15:28:44.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.687+0000 7fc2ae7fc700 20 client.38319.objecter put_session s=0x7fc2a407d340 osd=7 4 2024-04-14T15:28:44.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.687+0000 7fc2ae7fc700 15 client.38319.objecter _session_command_op_remove 7 1 2024-04-14T15:28:44.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.703+0000 7fc2bf300700 10 client.38319.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-14T15:28:44.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.703+0000 7fc2bf300700 15 client.38319.objecter _session_command_op_assign -1 2 2024-04-14T15:28:44.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.703+0000 7fc2bf300700 20 client.38319.objecter _get_session s=0x7fc2a407d340 osd=7 3 2024-04-14T15:28:44.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.703+0000 7fc2bf300700 20 client.38319.objecter put_session s=0x7fc2a407d340 osd=7 3 2024-04-14T15:28:44.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.703+0000 7fc2bf300700 20 client.38319.objecter _get_session s=0x7fc2a407d340 osd=7 3 2024-04-14T15:28:44.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.703+0000 7fc2bf300700 15 client.38319.objecter _session_command_op_remove -1 2 2024-04-14T15:28:44.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.703+0000 7fc2bf300700 20 client.38319.objecter get_session s=0x7fc2a407d340 osd=7 3 2024-04-14T15:28:44.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.703+0000 7fc2bf300700 15 client.38319.objecter _session_command_op_assign 7 2 2024-04-14T15:28:44.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.703+0000 7fc2bf300700 20 client.38319.objecter put_session s=0x7fc2a407d340 osd=7 4 2024-04-14T15:28:44.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.703+0000 7fc2bf300700 10 client.38319.objecter _send_command 2 2024-04-14T15:28:44.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.703+0000 7fc2bf300700 1 -- 172.21.15.90:0/2084580762 --> v1:172.21.15.92:6812/1773706716 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc29c0011f0 con 0x7fc2a407d9f0 2024-04-14T15:28:44.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.703+0000 7fc2ae7fc700 1 -- 172.21.15.90:0/2084580762 <== osd.7 v1:172.21.15.92:6812/1773706716 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (unknown 0 0 3826523353) 0x7fc29c0011f0 con 0x7fc2a407d9f0 2024-04-14T15:28:44.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.703+0000 7fc2ae7fc700 10 client.38319.objecter ms_dispatch 0x7fc2b8001550 command_reply(tid 2: 0 ) 2024-04-14T15:28:44.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.703+0000 7fc2ae7fc700 10 client.38319.objecter _finish_command 2 = system:0 2024-04-14T15:28:44.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.703+0000 7fc2ae7fc700 20 client.38319.objecter put_session s=0x7fc2a407d340 osd=7 4 2024-04-14T15:28:44.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.703+0000 7fc2ae7fc700 15 client.38319.objecter _session_command_op_remove 7 2 2024-04-14T15:28:44.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.703+0000 7fc2bf300700 10 librados: watch_flush enter 2024-04-14T15:28:44.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.704+0000 7fc2bf300700 10 librados: watch_flush exit 2024-04-14T15:28:44.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.704+0000 7fc2bf300700 10 client.38319.objecter close_session for osd.7 2024-04-14T15:28:44.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.704+0000 7fc2bf300700 1 -- 172.21.15.90:0/2084580762 >> v1:172.21.15.92:6812/1773706716 conn(0x7fc2a407d9f0 legacy=0x7fc2a407fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:44.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.704+0000 7fc2bf300700 20 client.38319.objecter put_session s=0x7fc2a407d340 osd=7 1 2024-04-14T15:28:44.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.704+0000 7fc2bf300700 20 client.38319.objecter shutdown clearing up homeless session... 2024-04-14T15:28:44.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.704+0000 7fc2bf300700 10 client.38319.objecter successfully canceled tick 2024-04-14T15:28:44.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.704+0000 7fc2bf300700 1 -- 172.21.15.90:0/2084580762 >> v1:172.21.15.90:6800/420935427 conn(0x7fc2a40712b0 legacy=0x7fc2a4073770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:44.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.704+0000 7fc2bf300700 1 -- 172.21.15.90:0/2084580762 >> v1:172.21.15.90:6789/0 conn(0x7fc2b810b8d0 legacy=0x7fc2b81a2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:44.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.704+0000 7fc2bd89d700 1 -- 172.21.15.90:0/2084580762 reap_dead start 2024-04-14T15:28:44.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.704+0000 7fc2bf300700 1 -- 172.21.15.90:0/2084580762 shutdown_connections 2024-04-14T15:28:44.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.704+0000 7fc2bf300700 1 -- 172.21.15.90:0/2084580762 >> 172.21.15.90:0/2084580762 conn(0x7fc2b80fd290 msgr2=0x7fc2b80ff680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:28:44.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.704+0000 7fc2bf300700 1 -- 172.21.15.90:0/2084580762 shutdown_connections 2024-04-14T15:28:44.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.704+0000 7fc2bf300700 1 -- 172.21.15.90:0/2084580762 wait complete. 2024-04-14T15:28:44.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:44.704+0000 7fc2bf300700 1 librados: shutdown 2024-04-14T15:28:45.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:45 smithi092 ceph-mon[33966]: 4.d scrub starts 2024-04-14T15:28:45.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:45 smithi092 ceph-mon[33966]: 4.d scrub ok 2024-04-14T15:28:45.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:45 smithi092 ceph-mon[33966]: 5.1b scrub starts 2024-04-14T15:28:45.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:45 smithi092 ceph-mon[33966]: 5.1b scrub ok 2024-04-14T15:28:45.655 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:45 smithi090 ceph-mon[41670]: 4.d scrub starts 2024-04-14T15:28:45.655 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:45 smithi090 ceph-mon[41670]: 4.d scrub ok 2024-04-14T15:28:45.655 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:45 smithi090 ceph-mon[41670]: 5.1b scrub starts 2024-04-14T15:28:45.655 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:45 smithi090 ceph-mon[41670]: 5.1b scrub ok 2024-04-14T15:28:45.655 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:45 smithi090 ceph-mon[30399]: 4.d scrub starts 2024-04-14T15:28:45.655 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:45 smithi090 ceph-mon[30399]: 4.d scrub ok 2024-04-14T15:28:45.655 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:45 smithi090 ceph-mon[30399]: 5.1b scrub starts 2024-04-14T15:28:45.655 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:45 smithi090 ceph-mon[30399]: 5.1b scrub ok 2024-04-14T15:28:46.122 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:28:45 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:28:46.562 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:46 smithi090 ceph-mon[41670]: 5.e scrub starts 2024-04-14T15:28:46.562 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:46 smithi090 ceph-mon[41670]: 5.e scrub ok 2024-04-14T15:28:46.562 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:46 smithi090 ceph-mon[41670]: pgmap v1395: 174 pgs: 174 active+clean; 582 KiB data, 839 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T15:28:46.563 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:46 smithi090 ceph-mon[41670]: 5.1d scrub starts 2024-04-14T15:28:46.563 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:46 smithi090 ceph-mon[41670]: 5.1d scrub ok 2024-04-14T15:28:46.563 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:46 smithi090 ceph-mon[41670]: 11.11 scrub starts 2024-04-14T15:28:46.563 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:46 smithi090 ceph-mon[41670]: 11.11 scrub ok 2024-04-14T15:28:46.563 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:46 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:28:46.563 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:46 smithi090 ceph-mon[30399]: 5.e scrub starts 2024-04-14T15:28:46.563 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:46 smithi090 ceph-mon[30399]: 5.e scrub ok 2024-04-14T15:28:46.563 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:46 smithi090 ceph-mon[30399]: pgmap v1395: 174 pgs: 174 active+clean; 582 KiB data, 839 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T15:28:46.563 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:46 smithi090 ceph-mon[30399]: 5.1d scrub starts 2024-04-14T15:28:46.563 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:46 smithi090 ceph-mon[30399]: 5.1d scrub ok 2024-04-14T15:28:46.563 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:46 smithi090 ceph-mon[30399]: 11.11 scrub starts 2024-04-14T15:28:46.563 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:46 smithi090 ceph-mon[30399]: 11.11 scrub ok 2024-04-14T15:28:46.564 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:46 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:28:46.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:46 smithi092 ceph-mon[33966]: 5.e scrub starts 2024-04-14T15:28:46.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:46 smithi092 ceph-mon[33966]: 5.e scrub ok 2024-04-14T15:28:46.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:46 smithi092 ceph-mon[33966]: pgmap v1395: 174 pgs: 174 active+clean; 582 KiB data, 839 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T15:28:46.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:46 smithi092 ceph-mon[33966]: 5.1d scrub starts 2024-04-14T15:28:46.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:46 smithi092 ceph-mon[33966]: 5.1d scrub ok 2024-04-14T15:28:46.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:46 smithi092 ceph-mon[33966]: 11.11 scrub starts 2024-04-14T15:28:46.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:46 smithi092 ceph-mon[33966]: 11.11 scrub ok 2024-04-14T15:28:46.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:46 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:28:47.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.993+0000 7f236cc68700 1 -- 172.21.15.90:0/1771965235 >> v1:172.21.15.90:6789/0 conn(0x7f236810a630 legacy=0x7f236810ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:47.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.993+0000 7f236cc68700 1 -- 172.21.15.90:0/1771965235 shutdown_connections 2024-04-14T15:28:47.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.993+0000 7f236cc68700 1 -- 172.21.15.90:0/1771965235 >> 172.21.15.90:0/1771965235 conn(0x7f23680fd270 msgr2=0x7f23680ff690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:28:47.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.993+0000 7f236cc68700 1 -- 172.21.15.90:0/1771965235 shutdown_connections 2024-04-14T15:28:47.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.993+0000 7f236cc68700 1 -- 172.21.15.90:0/1771965235 wait complete. 2024-04-14T15:28:47.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.994+0000 7f236cc68700 1 librados: starting msgr at 2024-04-14T15:28:47.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.994+0000 7f236cc68700 1 librados: starting objecter 2024-04-14T15:28:47.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.994+0000 7f236cc68700 1 Processor -- start 2024-04-14T15:28:47.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.994+0000 7f236cc68700 1 -- start start 2024-04-14T15:28:47.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.994+0000 7f236cc68700 1 librados: setting wanted keys 2024-04-14T15:28:47.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.994+0000 7f236cc68700 1 librados: calling monclient init 2024-04-14T15:28:47.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.994+0000 7f236cc68700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f23680ff2f0 con 0x7f2368107780 2024-04-14T15:28:47.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.994+0000 7f236cc68700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f23680ff500 con 0x7f236810a630 2024-04-14T15:28:47.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.994+0000 7f236cc68700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f23681ac220 con 0x7f236810e200 2024-04-14T15:28:47.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.994+0000 7f236659c700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f2368107780 0x7f23680fd720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:40742/0 (socket says 172.21.15.90:40742) 2024-04-14T15:28:47.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.994+0000 7f236659c700 1 -- 172.21.15.90:0/1620318390 learned_addr learned my addr 172.21.15.90:0/1620318390 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:28:47.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.995+0000 7f235f7fe700 10 client.?.objecter ms_handle_connect 0x7f2368107780 2024-04-14T15:28:47.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.995+0000 7f235f7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:28:47.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.995+0000 7f235f7fe700 10 client.?.objecter ms_handle_connect 0x7f236810e200 2024-04-14T15:28:47.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.995+0000 7f235f7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:28:47.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.995+0000 7f235f7fe700 10 client.?.objecter ms_handle_connect 0x7f236810a630 2024-04-14T15:28:47.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.995+0000 7f235f7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:28:47.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.995+0000 7f235f7fe700 1 -- 172.21.15.90:0/1620318390 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1566193696 0 0) 0x7f23680ff2f0 con 0x7f2368107780 2024-04-14T15:28:47.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.995+0000 7f235f7fe700 1 -- 172.21.15.90:0/1620318390 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f23540031f0 con 0x7f2368107780 2024-04-14T15:28:47.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.995+0000 7f235f7fe700 1 -- 172.21.15.90:0/1620318390 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3499581563 0 0) 0x7f23681ac220 con 0x7f236810e200 2024-04-14T15:28:47.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.995+0000 7f235f7fe700 1 -- 172.21.15.90:0/1620318390 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f23680ff2f0 con 0x7f236810e200 2024-04-14T15:28:47.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.995+0000 7f235f7fe700 1 -- 172.21.15.90:0/1620318390 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3661097580 0 0) 0x7f23680ff500 con 0x7f236810a630 2024-04-14T15:28:47.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.995+0000 7f235f7fe700 1 -- 172.21.15.90:0/1620318390 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f23681ac220 con 0x7f236810a630 2024-04-14T15:28:47.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.995+0000 7f235f7fe700 1 -- 172.21.15.90:0/1620318390 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3726674185 0 0) 0x7f23540031f0 con 0x7f2368107780 2024-04-14T15:28:47.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.995+0000 7f235f7fe700 1 -- 172.21.15.90:0/1620318390 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f23680ff500 con 0x7f2368107780 2024-04-14T15:28:47.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.995+0000 7f235f7fe700 1 -- 172.21.15.90:0/1620318390 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f2350001930 con 0x7f2368107780 2024-04-14T15:28:47.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.995+0000 7f235f7fe700 1 -- 172.21.15.90:0/1620318390 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 451664186 0 0) 0x7f23680ff2f0 con 0x7f236810e200 2024-04-14T15:28:47.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.995+0000 7f235f7fe700 1 -- 172.21.15.90:0/1620318390 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f23540031f0 con 0x7f236810e200 2024-04-14T15:28:47.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.995+0000 7f235f7fe700 1 -- 172.21.15.90:0/1620318390 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f2360001590 con 0x7f236810e200 2024-04-14T15:28:47.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.995+0000 7f235f7fe700 1 -- 172.21.15.90:0/1620318390 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3481218214 0 0) 0x7f23681ac220 con 0x7f236810a630 2024-04-14T15:28:47.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.995+0000 7f235f7fe700 1 -- 172.21.15.90:0/1620318390 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f23680ff2f0 con 0x7f236810a630 2024-04-14T15:28:47.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.995+0000 7f235f7fe700 1 -- 172.21.15.90:0/1620318390 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f2358004470 con 0x7f236810a630 2024-04-14T15:28:47.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.995+0000 7f235f7fe700 1 -- 172.21.15.90:0/1620318390 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 224731259 0 0) 0x7f23680ff500 con 0x7f2368107780 2024-04-14T15:28:47.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.995+0000 7f235f7fe700 1 -- 172.21.15.90:0/1620318390 >> v1:172.21.15.90:6790/0 conn(0x7f236810e200 legacy=0x7f23681a8ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:47.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.995+0000 7f235f7fe700 1 -- 172.21.15.90:0/1620318390 >> v1:172.21.15.92:6789/0 conn(0x7f236810a630 legacy=0x7f23680fde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:47.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.995+0000 7f235f7fe700 1 -- 172.21.15.90:0/1620318390 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23681ad400 con 0x7f2368107780 2024-04-14T15:28:47.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.995+0000 7f236cc68700 1 -- 172.21.15.90:0/1620318390 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f23681ac450 con 0x7f2368107780 2024-04-14T15:28:47.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.995+0000 7f236cc68700 10 client.38331.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:28:47.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.995+0000 7f236cc68700 1 -- 172.21.15.90:0/1620318390 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f23681ac910 con 0x7f2368107780 2024-04-14T15:28:47.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.995+0000 7f236cc68700 1 librados: init done 2024-04-14T15:28:47.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.995+0000 7f235f7fe700 1 -- 172.21.15.90:0/1620318390 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f2350003b40 con 0x7f2368107780 2024-04-14T15:28:47.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.995+0000 7f235f7fe700 1 -- 172.21.15.90:0/1620318390 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f2350005d30 con 0x7f2368107780 2024-04-14T15:28:47.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.996+0000 7f236cc68700 10 client.38331.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:28:47.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.996+0000 7f236cc68700 15 client.38331.objecter _session_command_op_assign -1 1 2024-04-14T15:28:47.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.996+0000 7f236cc68700 20 client.38331.objecter _get_session osd=-1 returning homeless 2024-04-14T15:28:47.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.996+0000 7f236cc68700 10 client.38331.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:28:47.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.996+0000 7f236cc68700 1 -- 172.21.15.90:0/1620318390 --> v1:172.21.15.90:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f2348002920 con 0x7f2368107780 2024-04-14T15:28:47.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.997+0000 7f235f7fe700 1 -- 172.21.15.90:0/1620318390 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f23500036a0 con 0x7f2368107780 2024-04-14T15:28:47.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.997+0000 7f235f7fe700 1 -- 172.21.15.90:0/1620318390 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(920..920 src has 467..920) ==== 7517+0+0 (unknown 2649999903 0 0) 0x7f235008a9c0 con 0x7f2368107780 2024-04-14T15:28:47.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.997+0000 7f235f7fe700 10 client.38331.objecter ms_dispatch 0x7f2368001550 osd_map(920..920 src has 467..920) 2024-04-14T15:28:47.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.997+0000 7f235f7fe700 3 client.38331.objecter handle_osd_map got epochs [920,920] > 0 2024-04-14T15:28:47.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.997+0000 7f235f7fe700 3 client.38331.objecter handle_osd_map decoding full epoch 920 2024-04-14T15:28:47.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.997+0000 7f235f7fe700 10 client.38331.objecter checking command 1 2024-04-14T15:28:47.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.997+0000 7f235f7fe700 20 client.38331.objecter _get_session s=0x7f235407d3c0 osd=2 3 2024-04-14T15:28:47.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.997+0000 7f235f7fe700 20 client.38331.objecter put_session s=0x7f235407d3c0 osd=2 3 2024-04-14T15:28:47.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.997+0000 7f235f7fe700 15 client.38331.objecter _session_command_op_remove -1 1 2024-04-14T15:28:47.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.997+0000 7f235f7fe700 20 client.38331.objecter _get_session s=0x7f235407d3c0 osd=2 3 2024-04-14T15:28:47.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.997+0000 7f235f7fe700 20 client.38331.objecter get_session s=0x7f235407d3c0 osd=2 3 2024-04-14T15:28:47.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.997+0000 7f235f7fe700 15 client.38331.objecter _session_command_op_assign 2 1 2024-04-14T15:28:47.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.997+0000 7f235f7fe700 20 client.38331.objecter put_session s=0x7f235407d3c0 osd=2 4 2024-04-14T15:28:47.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.997+0000 7f235f7fe700 10 client.38331.objecter _send_command 1 2024-04-14T15:28:47.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.997+0000 7f235f7fe700 1 -- 172.21.15.90:0/1620318390 --> v1:172.21.15.90:6809/1686184251 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2358001080 con 0x7f235407da70 2024-04-14T15:28:47.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.997+0000 7f235f7fe700 20 client.38331.objecter dump_active .. 0 homeless 2024-04-14T15:28:47.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.997+0000 7f235f7fe700 1 -- 172.21.15.90:0/1620318390 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_get_version_reply(handle=1 version=920) ==== 24+0+0 (unknown 622753966 0 0) 0x7f235008ad90 con 0x7f2368107780 2024-04-14T15:28:47.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.997+0000 7f235f7fe700 10 client.38331.objecter ms_handle_connect 0x7f23540713e0 2024-04-14T15:28:47.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.998+0000 7f235f7fe700 10 client.38331.objecter ms_handle_connect 0x7f235407da70 2024-04-14T15:28:47.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.998+0000 7f235f7fe700 1 -- 172.21.15.90:0/1620318390 <== osd.2 v1:172.21.15.90:6809/1686184251 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2358001080 con 0x7f235407da70 2024-04-14T15:28:47.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.998+0000 7f235f7fe700 10 client.38331.objecter ms_dispatch 0x7f2368001550 command_reply(tid 1: 0 ) 2024-04-14T15:28:47.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.998+0000 7f235f7fe700 10 client.38331.objecter _finish_command 1 = system:0 2024-04-14T15:28:47.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.998+0000 7f235f7fe700 20 client.38331.objecter put_session s=0x7f235407d3c0 osd=2 4 2024-04-14T15:28:47.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.998+0000 7f235f7fe700 15 client.38331.objecter _session_command_op_remove 2 1 2024-04-14T15:28:47.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.014+0000 7f236cc68700 10 client.38331.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-14T15:28:47.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.014+0000 7f236cc68700 15 client.38331.objecter _session_command_op_assign -1 2 2024-04-14T15:28:47.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.014+0000 7f236cc68700 20 client.38331.objecter _get_session s=0x7f235407d3c0 osd=2 3 2024-04-14T15:28:47.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.014+0000 7f236cc68700 20 client.38331.objecter put_session s=0x7f235407d3c0 osd=2 3 2024-04-14T15:28:47.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.014+0000 7f236cc68700 20 client.38331.objecter _get_session s=0x7f235407d3c0 osd=2 3 2024-04-14T15:28:47.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.014+0000 7f236cc68700 15 client.38331.objecter _session_command_op_remove -1 2 2024-04-14T15:28:47.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.014+0000 7f236cc68700 20 client.38331.objecter get_session s=0x7f235407d3c0 osd=2 3 2024-04-14T15:28:47.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.014+0000 7f236cc68700 15 client.38331.objecter _session_command_op_assign 2 2 2024-04-14T15:28:47.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.014+0000 7f236cc68700 20 client.38331.objecter put_session s=0x7f235407d3c0 osd=2 4 2024-04-14T15:28:47.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.014+0000 7f236cc68700 10 client.38331.objecter _send_command 2 2024-04-14T15:28:47.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.014+0000 7f236cc68700 1 -- 172.21.15.90:0/1620318390 --> v1:172.21.15.90:6809/1686184251 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f23480011f0 con 0x7f235407da70 2024-04-14T15:28:47.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.014+0000 7f235f7fe700 1 -- 172.21.15.90:0/1620318390 <== osd.2 v1:172.21.15.90:6809/1686184251 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 435794203) 0x7f23480011f0 con 0x7f235407da70 2024-04-14T15:28:47.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.014+0000 7f235f7fe700 10 client.38331.objecter ms_dispatch 0x7f2368001550 command_reply(tid 2: 0 ) 2024-04-14T15:28:47.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.014+0000 7f235f7fe700 10 client.38331.objecter _finish_command 2 = system:0 2024-04-14T15:28:47.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.014+0000 7f235f7fe700 20 client.38331.objecter put_session s=0x7f235407d3c0 osd=2 4 2024-04-14T15:28:47.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.014+0000 7f235f7fe700 15 client.38331.objecter _session_command_op_remove 2 2 2024-04-14T15:28:47.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.015+0000 7f236cc68700 10 librados: watch_flush enter 2024-04-14T15:28:47.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.015+0000 7f236cc68700 10 librados: watch_flush exit 2024-04-14T15:28:47.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.015+0000 7f236cc68700 10 client.38331.objecter close_session for osd.2 2024-04-14T15:28:47.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.015+0000 7f236cc68700 1 -- 172.21.15.90:0/1620318390 >> v1:172.21.15.90:6809/1686184251 conn(0x7f235407da70 legacy=0x7f235407ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:47.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.015+0000 7f236cc68700 20 client.38331.objecter put_session s=0x7f235407d3c0 osd=2 1 2024-04-14T15:28:47.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.015+0000 7f236cc68700 20 client.38331.objecter shutdown clearing up homeless session... 2024-04-14T15:28:47.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.015+0000 7f236cc68700 10 client.38331.objecter successfully canceled tick 2024-04-14T15:28:47.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.015+0000 7f236cc68700 1 -- 172.21.15.90:0/1620318390 >> v1:172.21.15.90:6800/420935427 conn(0x7f23540713e0 legacy=0x7f23540738a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:47.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.015+0000 7f236cc68700 1 -- 172.21.15.90:0/1620318390 >> v1:172.21.15.90:6789/0 conn(0x7f2368107780 legacy=0x7f23680fd720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:47.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.015+0000 7f2366d9d700 1 -- 172.21.15.90:0/1620318390 reap_dead start 2024-04-14T15:28:47.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.015+0000 7f236cc68700 1 -- 172.21.15.90:0/1620318390 shutdown_connections 2024-04-14T15:28:47.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.015+0000 7f236cc68700 1 -- 172.21.15.90:0/1620318390 >> 172.21.15.90:0/1620318390 conn(0x7f23680fd270 msgr2=0x7f2368100510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:28:47.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.015+0000 7f236cc68700 1 -- 172.21.15.90:0/1620318390 shutdown_connections 2024-04-14T15:28:47.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.015+0000 7f236cc68700 1 -- 172.21.15.90:0/1620318390 wait complete. 2024-04-14T15:28:47.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.015+0000 7f236cc68700 1 librados: shutdown 2024-04-14T15:28:47.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:47 smithi092 ceph-mon[33966]: 6.13 scrub starts 2024-04-14T15:28:47.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:47 smithi092 ceph-mon[33966]: 6.13 scrub ok 2024-04-14T15:28:47.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:47 smithi092 ceph-mon[33966]: 6.15 scrub starts 2024-04-14T15:28:47.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:47 smithi092 ceph-mon[33966]: 6.15 scrub ok 2024-04-14T15:28:47.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:47 smithi092 ceph-mon[33966]: 2.1a scrub starts 2024-04-14T15:28:47.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:47 smithi092 ceph-mon[33966]: 2.1a scrub ok 2024-04-14T15:28:47.655 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:47 smithi090 ceph-mon[41670]: 6.13 scrub starts 2024-04-14T15:28:47.656 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:47 smithi090 ceph-mon[41670]: 6.13 scrub ok 2024-04-14T15:28:47.656 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:47 smithi090 ceph-mon[41670]: 6.15 scrub starts 2024-04-14T15:28:47.656 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:47 smithi090 ceph-mon[41670]: 6.15 scrub ok 2024-04-14T15:28:47.656 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:47 smithi090 ceph-mon[41670]: 2.1a scrub starts 2024-04-14T15:28:47.656 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:47 smithi090 ceph-mon[41670]: 2.1a scrub ok 2024-04-14T15:28:47.656 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:47 smithi090 ceph-mon[30399]: 6.13 scrub starts 2024-04-14T15:28:47.656 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:47 smithi090 ceph-mon[30399]: 6.13 scrub ok 2024-04-14T15:28:47.656 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:47 smithi090 ceph-mon[30399]: 6.15 scrub starts 2024-04-14T15:28:47.656 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:47 smithi090 ceph-mon[30399]: 6.15 scrub ok 2024-04-14T15:28:47.656 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:47 smithi090 ceph-mon[30399]: 2.1a scrub starts 2024-04-14T15:28:47.657 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:47 smithi090 ceph-mon[30399]: 2.1a scrub ok 2024-04-14T15:28:47.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.885+0000 7fbbc7e09700 1 -- 172.21.15.90:0/1989316669 >> v1:172.21.15.90:6789/0 conn(0x7fbbc0107760 legacy=0x7fbbc0107b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:47.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.885+0000 7fbbc7e09700 1 -- 172.21.15.90:0/1989316669 shutdown_connections 2024-04-14T15:28:47.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.885+0000 7fbbc7e09700 1 -- 172.21.15.90:0/1989316669 >> 172.21.15.90:0/1989316669 conn(0x7fbbc00fd290 msgr2=0x7fbbc00ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:28:47.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.885+0000 7fbbc7e09700 1 -- 172.21.15.90:0/1989316669 shutdown_connections 2024-04-14T15:28:47.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.885+0000 7fbbc7e09700 1 -- 172.21.15.90:0/1989316669 wait complete. 2024-04-14T15:28:47.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.886+0000 7fbbc7e09700 1 librados: starting msgr at 2024-04-14T15:28:47.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.886+0000 7fbbc7e09700 1 librados: starting objecter 2024-04-14T15:28:47.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.886+0000 7fbbc7e09700 1 Processor -- start 2024-04-14T15:28:47.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.886+0000 7fbbc7e09700 1 -- start start 2024-04-14T15:28:47.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.886+0000 7fbbc7e09700 1 librados: setting wanted keys 2024-04-14T15:28:47.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.886+0000 7fbbc7e09700 1 librados: calling monclient init 2024-04-14T15:28:47.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.886+0000 7fbbc7e09700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbbc01a7800 con 0x7fbbc010e1e0 2024-04-14T15:28:47.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.886+0000 7fbbc7e09700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbbc01a8a20 con 0x7fbbc0107760 2024-04-14T15:28:47.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.886+0000 7fbbc7e09700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbbc01a9c60 con 0x7fbbc010a610 2024-04-14T15:28:47.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.887+0000 7fbbc53a4700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7fbbc010a610 0x7fbbc01a26a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:54114/0 (socket says 172.21.15.90:54114) 2024-04-14T15:28:47.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.887+0000 7fbbc53a4700 1 -- 172.21.15.90:0/2922466057 learned_addr learned my addr 172.21.15.90:0/2922466057 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:28:47.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.887+0000 7fbbb6ffd700 10 client.?.objecter ms_handle_connect 0x7fbbc010e1e0 2024-04-14T15:28:47.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.887+0000 7fbbb6ffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:28:47.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.887+0000 7fbbb6ffd700 10 client.?.objecter ms_handle_connect 0x7fbbc010a610 2024-04-14T15:28:47.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.887+0000 7fbbb6ffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:28:47.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.887+0000 7fbbb6ffd700 10 client.?.objecter ms_handle_connect 0x7fbbc0107760 2024-04-14T15:28:47.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.887+0000 7fbbb6ffd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:28:47.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.887+0000 7fbbb6ffd700 1 -- 172.21.15.90:0/2922466057 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 710026504 0 0) 0x7fbbc01a7800 con 0x7fbbc010e1e0 2024-04-14T15:28:47.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.887+0000 7fbbb6ffd700 1 -- 172.21.15.90:0/2922466057 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbbac0031f0 con 0x7fbbc010e1e0 2024-04-14T15:28:47.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.887+0000 7fbbb6ffd700 1 -- 172.21.15.90:0/2922466057 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3406882954 0 0) 0x7fbbc01a9c60 con 0x7fbbc010a610 2024-04-14T15:28:47.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.887+0000 7fbbb6ffd700 1 -- 172.21.15.90:0/2922466057 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbbc01a7800 con 0x7fbbc010a610 2024-04-14T15:28:47.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.887+0000 7fbbb6ffd700 1 -- 172.21.15.90:0/2922466057 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1303664019 0 0) 0x7fbbc01a8a20 con 0x7fbbc0107760 2024-04-14T15:28:47.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.887+0000 7fbbb6ffd700 1 -- 172.21.15.90:0/2922466057 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbbc01a9c60 con 0x7fbbc0107760 2024-04-14T15:28:47.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.887+0000 7fbbb6ffd700 1 -- 172.21.15.90:0/2922466057 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3468336415 0 0) 0x7fbbac0031f0 con 0x7fbbc010e1e0 2024-04-14T15:28:47.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.887+0000 7fbbb6ffd700 1 -- 172.21.15.90:0/2922466057 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbbc01a8a20 con 0x7fbbc010e1e0 2024-04-14T15:28:47.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.887+0000 7fbbb6ffd700 1 -- 172.21.15.90:0/2922466057 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fbbb8001590 con 0x7fbbc010e1e0 2024-04-14T15:28:47.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.887+0000 7fbbb6ffd700 1 -- 172.21.15.90:0/2922466057 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4159326596 0 0) 0x7fbbc01a7800 con 0x7fbbc010a610 2024-04-14T15:28:47.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.887+0000 7fbbb6ffd700 1 -- 172.21.15.90:0/2922466057 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbbac0031f0 con 0x7fbbc010a610 2024-04-14T15:28:47.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.887+0000 7fbbb6ffd700 1 -- 172.21.15.90:0/2922466057 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fbbbc0018b0 con 0x7fbbc010a610 2024-04-14T15:28:47.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.887+0000 7fbbb6ffd700 1 -- 172.21.15.90:0/2922466057 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 906001481 0 0) 0x7fbbc01a9c60 con 0x7fbbc0107760 2024-04-14T15:28:47.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.887+0000 7fbbb6ffd700 1 -- 172.21.15.90:0/2922466057 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbbc01a7800 con 0x7fbbc0107760 2024-04-14T15:28:47.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.887+0000 7fbbb6ffd700 1 -- 172.21.15.90:0/2922466057 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fbbb0001880 con 0x7fbbc0107760 2024-04-14T15:28:47.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.887+0000 7fbbb6ffd700 1 -- 172.21.15.90:0/2922466057 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2592144989 0 0) 0x7fbbc01a8a20 con 0x7fbbc010e1e0 2024-04-14T15:28:47.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.887+0000 7fbbb6ffd700 1 -- 172.21.15.90:0/2922466057 >> v1:172.21.15.90:6790/0 conn(0x7fbbc010a610 legacy=0x7fbbc01a26a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:47.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.887+0000 7fbbb6ffd700 1 -- 172.21.15.90:0/2922466057 >> v1:172.21.15.92:6789/0 conn(0x7fbbc0107760 legacy=0x7fbbc010d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:47.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.887+0000 7fbbb6ffd700 1 -- 172.21.15.90:0/2922466057 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbc01aaea0 con 0x7fbbc010e1e0 2024-04-14T15:28:47.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.887+0000 7fbbc7e09700 1 -- 172.21.15.90:0/2922466057 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbbc01a9e90 con 0x7fbbc010e1e0 2024-04-14T15:28:47.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.887+0000 7fbbc7e09700 10 client.38337.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:28:47.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.887+0000 7fbbc7e09700 1 -- 172.21.15.90:0/2922466057 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fbbc01aa350 con 0x7fbbc010e1e0 2024-04-14T15:28:47.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.887+0000 7fbbc7e09700 1 librados: init done 2024-04-14T15:28:47.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.887+0000 7fbbb6ffd700 1 -- 172.21.15.90:0/2922466057 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fbbb8003ea0 con 0x7fbbc010e1e0 2024-04-14T15:28:47.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.887+0000 7fbbb6ffd700 1 -- 172.21.15.90:0/2922466057 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fbbb8004d70 con 0x7fbbc010e1e0 2024-04-14T15:28:47.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.888+0000 7fbbc7e09700 10 client.38337.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:28:47.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.888+0000 7fbbc7e09700 15 client.38337.objecter _session_command_op_assign -1 1 2024-04-14T15:28:47.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.888+0000 7fbbc7e09700 20 client.38337.objecter _get_session osd=-1 returning homeless 2024-04-14T15:28:47.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.888+0000 7fbbc7e09700 10 client.38337.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:28:47.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.888+0000 7fbbc7e09700 1 -- 172.21.15.90:0/2922466057 --> v1:172.21.15.90:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fbba4002950 con 0x7fbbc010e1e0 2024-04-14T15:28:47.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.889+0000 7fbbb6ffd700 1 -- 172.21.15.90:0/2922466057 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fbbb8003a00 con 0x7fbbc010e1e0 2024-04-14T15:28:47.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.889+0000 7fbbb6ffd700 1 -- 172.21.15.90:0/2922466057 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(920..920 src has 467..920) ==== 7517+0+0 (unknown 2649999903 0 0) 0x7fbbb8089960 con 0x7fbbc010e1e0 2024-04-14T15:28:47.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.889+0000 7fbbb6ffd700 10 client.38337.objecter ms_dispatch 0x7fbbc0001550 osd_map(920..920 src has 467..920) 2024-04-14T15:28:47.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.889+0000 7fbbb6ffd700 3 client.38337.objecter handle_osd_map got epochs [920,920] > 0 2024-04-14T15:28:47.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.889+0000 7fbbb6ffd700 3 client.38337.objecter handle_osd_map decoding full epoch 920 2024-04-14T15:28:47.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.889+0000 7fbbb6ffd700 10 client.38337.objecter checking command 1 2024-04-14T15:28:47.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.889+0000 7fbbb6ffd700 20 client.38337.objecter _get_session s=0x7fbbac07d4c0 osd=5 3 2024-04-14T15:28:47.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.889+0000 7fbbb6ffd700 20 client.38337.objecter put_session s=0x7fbbac07d4c0 osd=5 3 2024-04-14T15:28:47.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.890+0000 7fbbb6ffd700 15 client.38337.objecter _session_command_op_remove -1 1 2024-04-14T15:28:47.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.890+0000 7fbbb6ffd700 20 client.38337.objecter _get_session s=0x7fbbac07d4c0 osd=5 3 2024-04-14T15:28:47.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.890+0000 7fbbb6ffd700 20 client.38337.objecter get_session s=0x7fbbac07d4c0 osd=5 3 2024-04-14T15:28:47.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.890+0000 7fbbb6ffd700 15 client.38337.objecter _session_command_op_assign 5 1 2024-04-14T15:28:47.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.890+0000 7fbbb6ffd700 20 client.38337.objecter put_session s=0x7fbbac07d4c0 osd=5 4 2024-04-14T15:28:47.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.890+0000 7fbbb6ffd700 10 client.38337.objecter _send_command 1 2024-04-14T15:28:47.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.890+0000 7fbbb6ffd700 1 -- 172.21.15.90:0/2922466057 --> v1:172.21.15.92:6804/3244561440 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbbb0001080 con 0x7fbbac07db70 2024-04-14T15:28:47.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.890+0000 7fbbb6ffd700 20 client.38337.objecter dump_active .. 0 homeless 2024-04-14T15:28:47.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.890+0000 7fbbb6ffd700 10 client.38337.objecter ms_handle_connect 0x7fbbac071440 2024-04-14T15:28:47.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.890+0000 7fbbb6ffd700 1 -- 172.21.15.90:0/2922466057 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_get_version_reply(handle=1 version=920) ==== 24+0+0 (unknown 622753966 0 0) 0x7fbbb8089dd0 con 0x7fbbc010e1e0 2024-04-14T15:28:47.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.890+0000 7fbbb6ffd700 10 client.38337.objecter ms_handle_connect 0x7fbbac07db70 2024-04-14T15:28:47.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.891+0000 7fbbb6ffd700 1 -- 172.21.15.90:0/2922466057 <== osd.5 v1:172.21.15.92:6804/3244561440 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fbbb0001080 con 0x7fbbac07db70 2024-04-14T15:28:47.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.891+0000 7fbbb6ffd700 10 client.38337.objecter ms_dispatch 0x7fbbc0001550 command_reply(tid 1: 0 ) 2024-04-14T15:28:47.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.891+0000 7fbbb6ffd700 10 client.38337.objecter _finish_command 1 = system:0 2024-04-14T15:28:47.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.891+0000 7fbbb6ffd700 20 client.38337.objecter put_session s=0x7fbbac07d4c0 osd=5 4 2024-04-14T15:28:47.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.891+0000 7fbbb6ffd700 15 client.38337.objecter _session_command_op_remove 5 1 2024-04-14T15:28:47.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.907+0000 7fbbc7e09700 10 client.38337.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-14T15:28:47.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.907+0000 7fbbc7e09700 15 client.38337.objecter _session_command_op_assign -1 2 2024-04-14T15:28:47.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.907+0000 7fbbc7e09700 20 client.38337.objecter _get_session s=0x7fbbac07d4c0 osd=5 3 2024-04-14T15:28:47.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.907+0000 7fbbc7e09700 20 client.38337.objecter put_session s=0x7fbbac07d4c0 osd=5 3 2024-04-14T15:28:47.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.907+0000 7fbbc7e09700 20 client.38337.objecter _get_session s=0x7fbbac07d4c0 osd=5 3 2024-04-14T15:28:47.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.907+0000 7fbbc7e09700 15 client.38337.objecter _session_command_op_remove -1 2 2024-04-14T15:28:47.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.907+0000 7fbbc7e09700 20 client.38337.objecter get_session s=0x7fbbac07d4c0 osd=5 3 2024-04-14T15:28:47.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.907+0000 7fbbc7e09700 15 client.38337.objecter _session_command_op_assign 5 2 2024-04-14T15:28:47.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.907+0000 7fbbc7e09700 20 client.38337.objecter put_session s=0x7fbbac07d4c0 osd=5 4 2024-04-14T15:28:47.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.907+0000 7fbbc7e09700 10 client.38337.objecter _send_command 2 2024-04-14T15:28:47.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.907+0000 7fbbc7e09700 1 -- 172.21.15.90:0/2922466057 --> v1:172.21.15.92:6804/3244561440 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fbba4001220 con 0x7fbbac07db70 2024-04-14T15:28:47.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.907+0000 7fbbb6ffd700 1 -- 172.21.15.90:0/2922466057 <== osd.5 v1:172.21.15.92:6804/3244561440 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (unknown 0 0 548009954) 0x7fbba4001220 con 0x7fbbac07db70 2024-04-14T15:28:47.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.907+0000 7fbbb6ffd700 10 client.38337.objecter ms_dispatch 0x7fbbc0001550 command_reply(tid 2: 0 ) 2024-04-14T15:28:47.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.907+0000 7fbbb6ffd700 10 client.38337.objecter _finish_command 2 = system:0 2024-04-14T15:28:47.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.907+0000 7fbbb6ffd700 20 client.38337.objecter put_session s=0x7fbbac07d4c0 osd=5 4 2024-04-14T15:28:47.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.907+0000 7fbbb6ffd700 15 client.38337.objecter _session_command_op_remove 5 2 2024-04-14T15:28:47.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.907+0000 7fbbc7e09700 10 librados: watch_flush enter 2024-04-14T15:28:47.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.907+0000 7fbbc7e09700 10 librados: watch_flush exit 2024-04-14T15:28:47.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.907+0000 7fbbc7e09700 10 client.38337.objecter close_session for osd.5 2024-04-14T15:28:47.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.907+0000 7fbbc7e09700 1 -- 172.21.15.90:0/2922466057 >> v1:172.21.15.92:6804/3244561440 conn(0x7fbbac07db70 legacy=0x7fbbac080010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:47.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.907+0000 7fbbc7e09700 20 client.38337.objecter put_session s=0x7fbbac07d4c0 osd=5 1 2024-04-14T15:28:47.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.907+0000 7fbbc7e09700 20 client.38337.objecter shutdown clearing up homeless session... 2024-04-14T15:28:47.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.907+0000 7fbbc7e09700 10 client.38337.objecter successfully canceled tick 2024-04-14T15:28:47.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.907+0000 7fbbc7e09700 1 -- 172.21.15.90:0/2922466057 >> v1:172.21.15.90:6800/420935427 conn(0x7fbbac071440 legacy=0x7fbbac073900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:47.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.907+0000 7fbbc7e09700 1 -- 172.21.15.90:0/2922466057 >> v1:172.21.15.90:6789/0 conn(0x7fbbc010e1e0 legacy=0x7fbbc01a5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:47.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.907+0000 7fbbc63a6700 1 -- 172.21.15.90:0/2922466057 reap_dead start 2024-04-14T15:28:47.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.907+0000 7fbbc7e09700 1 -- 172.21.15.90:0/2922466057 shutdown_connections 2024-04-14T15:28:47.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.907+0000 7fbbc7e09700 1 -- 172.21.15.90:0/2922466057 >> 172.21.15.90:0/2922466057 conn(0x7fbbc00fd290 msgr2=0x7fbbc00ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:28:47.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.907+0000 7fbbc7e09700 1 -- 172.21.15.90:0/2922466057 shutdown_connections 2024-04-14T15:28:47.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.907+0000 7fbbc7e09700 1 -- 172.21.15.90:0/2922466057 wait complete. 2024-04-14T15:28:47.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:47.907+0000 7fbbc7e09700 1 librados: shutdown 2024-04-14T15:28:48.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.416+0000 7f05f4895700 1 -- 172.21.15.90:0/3127306514 >> v1:172.21.15.90:6789/0 conn(0x7f05f0103a40 legacy=0x7f05f0103e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:48.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.416+0000 7f05f4895700 1 -- 172.21.15.90:0/3127306514 shutdown_connections 2024-04-14T15:28:48.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.416+0000 7f05f4895700 1 -- 172.21.15.90:0/3127306514 >> 172.21.15.90:0/3127306514 conn(0x7f05f00fd2b0 msgr2=0x7f05f00ff6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:28:48.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.416+0000 7f05f4895700 1 -- 172.21.15.90:0/3127306514 shutdown_connections 2024-04-14T15:28:48.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.416+0000 7f05f4895700 1 -- 172.21.15.90:0/3127306514 wait complete. 2024-04-14T15:28:48.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.416+0000 7f05f4895700 1 librados: starting msgr at 2024-04-14T15:28:48.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.416+0000 7f05f4895700 1 librados: starting objecter 2024-04-14T15:28:48.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.417+0000 7f05f4895700 1 Processor -- start 2024-04-14T15:28:48.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.417+0000 7f05f4895700 1 -- start start 2024-04-14T15:28:48.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.417+0000 7f05f4895700 1 librados: setting wanted keys 2024-04-14T15:28:48.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.417+0000 7f05f4895700 1 librados: calling monclient init 2024-04-14T15:28:48.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.417+0000 7f05f4895700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f05f01a9d50 con 0x7f05f010f360 2024-04-14T15:28:48.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.417+0000 7f05f4895700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f05f01aaf30 con 0x7f05f010b850 2024-04-14T15:28:48.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.417+0000 7f05f4895700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f05f01ac110 con 0x7f05f0103a40 2024-04-14T15:28:48.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.417+0000 7f05eed9d700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f05f010f360 0x7f05f01a8630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:40736/0 (socket says 172.21.15.90:40736) 2024-04-14T15:28:48.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.417+0000 7f05eed9d700 1 -- 172.21.15.90:0/3869631630 learned_addr learned my addr 172.21.15.90:0/3869631630 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:28:48.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.417+0000 7f05df7fe700 10 client.?.objecter ms_handle_connect 0x7f05f010f360 2024-04-14T15:28:48.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.417+0000 7f05df7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:28:48.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.417+0000 7f05df7fe700 10 client.?.objecter ms_handle_connect 0x7f05f0103a40 2024-04-14T15:28:48.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.417+0000 7f05df7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:28:48.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.417+0000 7f05df7fe700 10 client.?.objecter ms_handle_connect 0x7f05f010b850 2024-04-14T15:28:48.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.417+0000 7f05df7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:28:48.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.417+0000 7f05df7fe700 1 -- 172.21.15.90:0/3869631630 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1763670318 0 0) 0x7f05f01a9d50 con 0x7f05f010f360 2024-04-14T15:28:48.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.417+0000 7f05df7fe700 1 -- 172.21.15.90:0/3869631630 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f05d80031f0 con 0x7f05f010f360 2024-04-14T15:28:48.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.417+0000 7f05df7fe700 1 -- 172.21.15.90:0/3869631630 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3486507952 0 0) 0x7f05f01ac110 con 0x7f05f0103a40 2024-04-14T15:28:48.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.417+0000 7f05df7fe700 1 -- 172.21.15.90:0/3869631630 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f05f01a9d50 con 0x7f05f0103a40 2024-04-14T15:28:48.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.417+0000 7f05df7fe700 1 -- 172.21.15.90:0/3869631630 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 262941911 0 0) 0x7f05f01aaf30 con 0x7f05f010b850 2024-04-14T15:28:48.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.417+0000 7f05df7fe700 1 -- 172.21.15.90:0/3869631630 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f05f01ac110 con 0x7f05f010b850 2024-04-14T15:28:48.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.418+0000 7f05df7fe700 1 -- 172.21.15.90:0/3869631630 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2042311559 0 0) 0x7f05f01a9d50 con 0x7f05f0103a40 2024-04-14T15:28:48.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.418+0000 7f05df7fe700 1 -- 172.21.15.90:0/3869631630 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f05f01aaf30 con 0x7f05f0103a40 2024-04-14T15:28:48.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.418+0000 7f05df7fe700 1 -- 172.21.15.90:0/3869631630 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3292840786 0 0) 0x7f05d80031f0 con 0x7f05f010f360 2024-04-14T15:28:48.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.418+0000 7f05df7fe700 1 -- 172.21.15.90:0/3869631630 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f05f01a9d50 con 0x7f05f010f360 2024-04-14T15:28:48.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.418+0000 7f05df7fe700 1 -- 172.21.15.90:0/3869631630 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f05e0003910 con 0x7f05f0103a40 2024-04-14T15:28:48.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.418+0000 7f05df7fe700 1 -- 172.21.15.90:0/3869631630 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1040612180 0 0) 0x7f05f01ac110 con 0x7f05f010b850 2024-04-14T15:28:48.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.418+0000 7f05df7fe700 1 -- 172.21.15.90:0/3869631630 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f05d80031f0 con 0x7f05f010b850 2024-04-14T15:28:48.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.418+0000 7f05df7fe700 1 -- 172.21.15.90:0/3869631630 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f05e8001c10 con 0x7f05f010f360 2024-04-14T15:28:48.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.418+0000 7f05df7fe700 1 -- 172.21.15.90:0/3869631630 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f05e4001830 con 0x7f05f010b850 2024-04-14T15:28:48.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.418+0000 7f05df7fe700 1 -- 172.21.15.90:0/3869631630 <== mon.2 v1:172.21.15.90:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4029208174 0 0) 0x7f05f01aaf30 con 0x7f05f0103a40 2024-04-14T15:28:48.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.418+0000 7f05df7fe700 1 -- 172.21.15.90:0/3869631630 >> v1:172.21.15.92:6789/0 conn(0x7f05f010b850 legacy=0x7f05f01a4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:48.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.418+0000 7f05df7fe700 1 -- 172.21.15.90:0/3869631630 >> v1:172.21.15.90:6789/0 conn(0x7f05f010f360 legacy=0x7f05f01a8630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:48.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.418+0000 7f05df7fe700 1 -- 172.21.15.90:0/3869631630 --> v1:172.21.15.90:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05f01ad2f0 con 0x7f05f0103a40 2024-04-14T15:28:48.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.418+0000 7f05f4895700 1 -- 172.21.15.90:0/3869631630 --> v1:172.21.15.90:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f05f01a9f80 con 0x7f05f0103a40 2024-04-14T15:28:48.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.418+0000 7f05f4895700 10 client.48275.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:28:48.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.418+0000 7f05f4895700 1 -- 172.21.15.90:0/3869631630 --> v1:172.21.15.90:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f05f01aa440 con 0x7f05f0103a40 2024-04-14T15:28:48.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.418+0000 7f05f4895700 1 librados: init done 2024-04-14T15:28:48.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.418+0000 7f05df7fe700 1 -- 172.21.15.90:0/3869631630 <== mon.2 v1:172.21.15.90:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f05e0003700 con 0x7f05f0103a40 2024-04-14T15:28:48.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.418+0000 7f05df7fe700 1 -- 172.21.15.90:0/3869631630 <== mon.2 v1:172.21.15.90:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f05e0004e40 con 0x7f05f0103a40 2024-04-14T15:28:48.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.419+0000 7f05f4895700 10 client.48275.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:28:48.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.419+0000 7f05f4895700 15 client.48275.objecter _session_command_op_assign -1 1 2024-04-14T15:28:48.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.419+0000 7f05f4895700 20 client.48275.objecter _get_session osd=-1 returning homeless 2024-04-14T15:28:48.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.419+0000 7f05f4895700 10 client.48275.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:28:48.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.419+0000 7f05f4895700 1 -- 172.21.15.90:0/3869631630 --> v1:172.21.15.90:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f05d0002920 con 0x7f05f0103a40 2024-04-14T15:28:48.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.419+0000 7f05df7fe700 1 -- 172.21.15.90:0/3869631630 <== mon.2 v1:172.21.15.90:6790/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f05e00018c0 con 0x7f05f0103a40 2024-04-14T15:28:48.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.420+0000 7f05df7fe700 1 -- 172.21.15.90:0/3869631630 <== mon.2 v1:172.21.15.90:6790/0 8 ==== osd_map(920..920 src has 467..920) ==== 7517+0+0 (unknown 2649999903 0 0) 0x7f05e00896a0 con 0x7f05f0103a40 2024-04-14T15:28:48.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.420+0000 7f05df7fe700 10 client.48275.objecter ms_dispatch 0x7f05f0001550 osd_map(920..920 src has 467..920) 2024-04-14T15:28:48.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.420+0000 7f05df7fe700 3 client.48275.objecter handle_osd_map got epochs [920,920] > 0 2024-04-14T15:28:48.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.420+0000 7f05df7fe700 3 client.48275.objecter handle_osd_map decoding full epoch 920 2024-04-14T15:28:48.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.420+0000 7f05df7fe700 10 client.48275.objecter checking command 1 2024-04-14T15:28:48.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.420+0000 7f05df7fe700 20 client.48275.objecter _get_session s=0x7f05d807d3e0 osd=6 3 2024-04-14T15:28:48.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.420+0000 7f05df7fe700 20 client.48275.objecter put_session s=0x7f05d807d3e0 osd=6 3 2024-04-14T15:28:48.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.420+0000 7f05df7fe700 15 client.48275.objecter _session_command_op_remove -1 1 2024-04-14T15:28:48.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.420+0000 7f05df7fe700 20 client.48275.objecter _get_session s=0x7f05d807d3e0 osd=6 3 2024-04-14T15:28:48.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.420+0000 7f05df7fe700 20 client.48275.objecter get_session s=0x7f05d807d3e0 osd=6 3 2024-04-14T15:28:48.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.420+0000 7f05df7fe700 15 client.48275.objecter _session_command_op_assign 6 1 2024-04-14T15:28:48.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.420+0000 7f05df7fe700 20 client.48275.objecter put_session s=0x7f05d807d3e0 osd=6 4 2024-04-14T15:28:48.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.420+0000 7f05df7fe700 10 client.48275.objecter _send_command 1 2024-04-14T15:28:48.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.420+0000 7f05df7fe700 1 -- 172.21.15.90:0/3869631630 --> v1:172.21.15.92:6808/636125963 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f05d80817a0 con 0x7f05d807da90 2024-04-14T15:28:48.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.420+0000 7f05df7fe700 20 client.48275.objecter dump_active .. 0 homeless 2024-04-14T15:28:48.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.420+0000 7f05df7fe700 1 -- 172.21.15.90:0/3869631630 <== mon.2 v1:172.21.15.90:6790/0 9 ==== mon_get_version_reply(handle=1 version=920) ==== 24+0+0 (unknown 622753966 0 0) 0x7f05e0089ac0 con 0x7f05f0103a40 2024-04-14T15:28:48.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.420+0000 7f05df7fe700 10 client.48275.objecter ms_handle_connect 0x7f05d8071380 2024-04-14T15:28:48.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.421+0000 7f05df7fe700 10 client.48275.objecter ms_handle_connect 0x7f05d807da90 2024-04-14T15:28:48.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.421+0000 7f05df7fe700 1 -- 172.21.15.90:0/3869631630 <== osd.6 v1:172.21.15.92:6808/636125963 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f05d80817a0 con 0x7f05d807da90 2024-04-14T15:28:48.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.421+0000 7f05df7fe700 10 client.48275.objecter ms_dispatch 0x7f05f0001550 command_reply(tid 1: 0 ) 2024-04-14T15:28:48.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.421+0000 7f05df7fe700 10 client.48275.objecter _finish_command 1 = system:0 2024-04-14T15:28:48.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.421+0000 7f05df7fe700 20 client.48275.objecter put_session s=0x7f05d807d3e0 osd=6 4 2024-04-14T15:28:48.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.421+0000 7f05df7fe700 15 client.48275.objecter _session_command_op_remove 6 1 2024-04-14T15:28:48.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.437+0000 7f05f4895700 10 client.48275.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-14T15:28:48.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.437+0000 7f05f4895700 15 client.48275.objecter _session_command_op_assign -1 2 2024-04-14T15:28:48.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.437+0000 7f05f4895700 20 client.48275.objecter _get_session s=0x7f05d807d3e0 osd=6 3 2024-04-14T15:28:48.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.437+0000 7f05f4895700 20 client.48275.objecter put_session s=0x7f05d807d3e0 osd=6 3 2024-04-14T15:28:48.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.437+0000 7f05f4895700 20 client.48275.objecter _get_session s=0x7f05d807d3e0 osd=6 3 2024-04-14T15:28:48.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.437+0000 7f05f4895700 15 client.48275.objecter _session_command_op_remove -1 2 2024-04-14T15:28:48.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.437+0000 7f05f4895700 20 client.48275.objecter get_session s=0x7f05d807d3e0 osd=6 3 2024-04-14T15:28:48.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.437+0000 7f05f4895700 15 client.48275.objecter _session_command_op_assign 6 2 2024-04-14T15:28:48.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.437+0000 7f05f4895700 20 client.48275.objecter put_session s=0x7f05d807d3e0 osd=6 4 2024-04-14T15:28:48.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.437+0000 7f05f4895700 10 client.48275.objecter _send_command 2 2024-04-14T15:28:48.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.437+0000 7f05f4895700 1 -- 172.21.15.90:0/3869631630 --> v1:172.21.15.92:6808/636125963 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f05d00011f0 con 0x7f05d807da90 2024-04-14T15:28:48.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.438+0000 7f05df7fe700 1 -- 172.21.15.90:0/3869631630 <== osd.6 v1:172.21.15.92:6808/636125963 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (unknown 0 0 387579342) 0x7f05d00011f0 con 0x7f05d807da90 2024-04-14T15:28:48.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.438+0000 7f05df7fe700 10 client.48275.objecter ms_dispatch 0x7f05f0001550 command_reply(tid 2: 0 ) 2024-04-14T15:28:48.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.438+0000 7f05df7fe700 10 client.48275.objecter _finish_command 2 = system:0 2024-04-14T15:28:48.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.438+0000 7f05df7fe700 20 client.48275.objecter put_session s=0x7f05d807d3e0 osd=6 4 2024-04-14T15:28:48.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.438+0000 7f05df7fe700 15 client.48275.objecter _session_command_op_remove 6 2 2024-04-14T15:28:48.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.438+0000 7f05f4895700 10 librados: watch_flush enter 2024-04-14T15:28:48.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.438+0000 7f05f4895700 10 librados: watch_flush exit 2024-04-14T15:28:48.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.438+0000 7f05f4895700 10 client.48275.objecter close_session for osd.6 2024-04-14T15:28:48.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.438+0000 7f05f4895700 1 -- 172.21.15.90:0/3869631630 >> v1:172.21.15.92:6808/636125963 conn(0x7f05d807da90 legacy=0x7f05d807ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:48.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.438+0000 7f05f4895700 20 client.48275.objecter put_session s=0x7f05d807d3e0 osd=6 1 2024-04-14T15:28:48.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.438+0000 7f05f4895700 20 client.48275.objecter shutdown clearing up homeless session... 2024-04-14T15:28:48.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.438+0000 7f05f4895700 10 client.48275.objecter successfully canceled tick 2024-04-14T15:28:48.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.438+0000 7f05f4895700 1 -- 172.21.15.90:0/3869631630 >> v1:172.21.15.90:6800/420935427 conn(0x7f05d8071380 legacy=0x7f05d8073840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:48.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.438+0000 7f05f4895700 1 -- 172.21.15.90:0/3869631630 >> v1:172.21.15.90:6790/0 conn(0x7f05f0103a40 legacy=0x7f05f01a00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:48.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.438+0000 7f05eed9d700 1 -- 172.21.15.90:0/3869631630 reap_dead start 2024-04-14T15:28:48.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.438+0000 7f05f4895700 1 -- 172.21.15.90:0/3869631630 shutdown_connections 2024-04-14T15:28:48.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.438+0000 7f05f4895700 1 -- 172.21.15.90:0/3869631630 >> 172.21.15.90:0/3869631630 conn(0x7f05f00fd2b0 msgr2=0x7f05f0110070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:28:48.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.438+0000 7f05f4895700 1 -- 172.21.15.90:0/3869631630 shutdown_connections 2024-04-14T15:28:48.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.438+0000 7f05f4895700 1 -- 172.21.15.90:0/3869631630 wait complete. 2024-04-14T15:28:48.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:46.438+0000 7f05f4895700 1 librados: shutdown 2024-04-14T15:28:48.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:48 smithi090 ceph-mon[30399]: pgmap v1396: 174 pgs: 174 active+clean; 582 KiB data, 839 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:28:48.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:48 smithi090 ceph-mon[30399]: 3.0 scrub starts 2024-04-14T15:28:48.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:48 smithi090 ceph-mon[30399]: 3.0 scrub ok 2024-04-14T15:28:48.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:48 smithi090 ceph-mon[30399]: 2.7 scrub starts 2024-04-14T15:28:48.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:48 smithi090 ceph-mon[30399]: 2.7 scrub ok 2024-04-14T15:28:48.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:48 smithi090 ceph-mon[30399]: 4.15 scrub starts 2024-04-14T15:28:48.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:48 smithi090 ceph-mon[30399]: 4.15 scrub ok 2024-04-14T15:28:48.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:48 smithi090 ceph-mon[41670]: pgmap v1396: 174 pgs: 174 active+clean; 582 KiB data, 839 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:28:48.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:48 smithi090 ceph-mon[41670]: 3.0 scrub starts 2024-04-14T15:28:48.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:48 smithi090 ceph-mon[41670]: 3.0 scrub ok 2024-04-14T15:28:48.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:48 smithi090 ceph-mon[41670]: 2.7 scrub starts 2024-04-14T15:28:48.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:48 smithi090 ceph-mon[41670]: 2.7 scrub ok 2024-04-14T15:28:48.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:48 smithi090 ceph-mon[41670]: 4.15 scrub starts 2024-04-14T15:28:48.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:48 smithi090 ceph-mon[41670]: 4.15 scrub ok 2024-04-14T15:28:48.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:48 smithi092 ceph-mon[33966]: pgmap v1396: 174 pgs: 174 active+clean; 582 KiB data, 839 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:28:48.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:48 smithi092 ceph-mon[33966]: 3.0 scrub starts 2024-04-14T15:28:48.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:48 smithi092 ceph-mon[33966]: 3.0 scrub ok 2024-04-14T15:28:48.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:48 smithi092 ceph-mon[33966]: 2.7 scrub starts 2024-04-14T15:28:48.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:48 smithi092 ceph-mon[33966]: 2.7 scrub ok 2024-04-14T15:28:48.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:48 smithi092 ceph-mon[33966]: 4.15 scrub starts 2024-04-14T15:28:48.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:48 smithi092 ceph-mon[33966]: 4.15 scrub ok 2024-04-14T15:28:48.969 INFO:teuthology.orchestra.run.smithi090.stdout:210453397742 2024-04-14T15:28:48.970 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-14T15:28:49.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:49 smithi092 ceph-mon[33966]: 6.e scrub starts 2024-04-14T15:28:49.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:49 smithi092 ceph-mon[33966]: 6.e scrub ok 2024-04-14T15:28:49.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:49 smithi092 ceph-mon[33966]: 4.16 scrub starts 2024-04-14T15:28:49.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:49 smithi092 ceph-mon[33966]: 4.16 scrub ok 2024-04-14T15:28:49.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:49 smithi092 ceph-mon[33966]: 1.3 scrub starts 2024-04-14T15:28:49.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:49 smithi092 ceph-mon[33966]: 1.3 scrub ok 2024-04-14T15:28:49.638 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:49 smithi090 ceph-mon[41670]: 6.e scrub starts 2024-04-14T15:28:49.638 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:49 smithi090 ceph-mon[41670]: 6.e scrub ok 2024-04-14T15:28:49.638 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:49 smithi090 ceph-mon[41670]: 4.16 scrub starts 2024-04-14T15:28:49.638 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:49 smithi090 ceph-mon[41670]: 4.16 scrub ok 2024-04-14T15:28:49.638 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:49 smithi090 ceph-mon[41670]: 1.3 scrub starts 2024-04-14T15:28:49.638 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:49 smithi090 ceph-mon[41670]: 1.3 scrub ok 2024-04-14T15:28:49.638 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:49 smithi090 ceph-mon[30399]: 6.e scrub starts 2024-04-14T15:28:49.638 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:49 smithi090 ceph-mon[30399]: 6.e scrub ok 2024-04-14T15:28:49.639 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:49 smithi090 ceph-mon[30399]: 4.16 scrub starts 2024-04-14T15:28:49.639 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:49 smithi090 ceph-mon[30399]: 4.16 scrub ok 2024-04-14T15:28:49.639 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:49 smithi090 ceph-mon[30399]: 1.3 scrub starts 2024-04-14T15:28:49.639 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:49 smithi090 ceph-mon[30399]: 1.3 scrub ok 2024-04-14T15:28:50.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.144+0000 7f6b80468700 1 -- 172.21.15.90:0/760334083 >> v1:172.21.15.92:6789/0 conn(0x7f6b7810e1e0 legacy=0x7f6b781106a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:50.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.144+0000 7f6b80468700 1 -- 172.21.15.90:0/760334083 shutdown_connections 2024-04-14T15:28:50.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.144+0000 7f6b80468700 1 -- 172.21.15.90:0/760334083 >> 172.21.15.90:0/760334083 conn(0x7f6b780fd290 msgr2=0x7f6b780ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:28:50.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.144+0000 7f6b80468700 1 -- 172.21.15.90:0/760334083 shutdown_connections 2024-04-14T15:28:50.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.144+0000 7f6b80468700 1 -- 172.21.15.90:0/760334083 wait complete. 2024-04-14T15:28:50.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.145+0000 7f6b80468700 1 librados: starting msgr at 2024-04-14T15:28:50.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.145+0000 7f6b80468700 1 librados: starting objecter 2024-04-14T15:28:50.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.145+0000 7f6b80468700 1 Processor -- start 2024-04-14T15:28:50.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.145+0000 7f6b80468700 1 -- start start 2024-04-14T15:28:50.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.145+0000 7f6b80468700 1 librados: setting wanted keys 2024-04-14T15:28:50.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.145+0000 7f6b80468700 1 librados: calling monclient init 2024-04-14T15:28:50.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.145+0000 7f6b80468700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6b7806b250 con 0x7f6b7810a610 2024-04-14T15:28:50.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.145+0000 7f6b80468700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6b781a8e00 con 0x7f6b78107760 2024-04-14T15:28:50.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.145+0000 7f6b80468700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6b781a9fe0 con 0x7f6b7810e1e0 2024-04-14T15:28:50.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.145+0000 7f6b7ea05700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f6b7810e1e0 0x7f6b781a66d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:54120/0 (socket says 172.21.15.90:54120) 2024-04-14T15:28:50.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.145+0000 7f6b7ea05700 1 -- 172.21.15.90:0/3208907871 learned_addr learned my addr 172.21.15.90:0/3208907871 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:28:50.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.145+0000 7f6b6b7fe700 10 client.?.objecter ms_handle_connect 0x7f6b7810e1e0 2024-04-14T15:28:50.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.145+0000 7f6b6b7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:28:50.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.145+0000 7f6b6b7fe700 10 client.?.objecter ms_handle_connect 0x7f6b7810a610 2024-04-14T15:28:50.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.145+0000 7f6b6b7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:28:50.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.145+0000 7f6b6b7fe700 1 -- 172.21.15.90:0/3208907871 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4145328630 0 0) 0x7f6b781a9fe0 con 0x7f6b7810e1e0 2024-04-14T15:28:50.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.146+0000 7f6b6b7fe700 1 -- 172.21.15.90:0/3208907871 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b640031f0 con 0x7f6b7810e1e0 2024-04-14T15:28:50.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.146+0000 7f6b6b7fe700 1 -- 172.21.15.90:0/3208907871 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1371851611 0 0) 0x7f6b7806b250 con 0x7f6b7810a610 2024-04-14T15:28:50.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.146+0000 7f6b6b7fe700 1 -- 172.21.15.90:0/3208907871 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b781a9fe0 con 0x7f6b7810a610 2024-04-14T15:28:50.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.146+0000 7f6b6b7fe700 10 client.?.objecter ms_handle_connect 0x7f6b78107760 2024-04-14T15:28:50.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.146+0000 7f6b6b7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:28:50.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.146+0000 7f6b6b7fe700 1 -- 172.21.15.90:0/3208907871 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2693975509 0 0) 0x7f6b781a8e00 con 0x7f6b78107760 2024-04-14T15:28:50.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.146+0000 7f6b6b7fe700 1 -- 172.21.15.90:0/3208907871 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b7806b250 con 0x7f6b78107760 2024-04-14T15:28:50.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.146+0000 7f6b6b7fe700 1 -- 172.21.15.90:0/3208907871 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1144725817 0 0) 0x7f6b640031f0 con 0x7f6b7810e1e0 2024-04-14T15:28:50.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.146+0000 7f6b6b7fe700 1 -- 172.21.15.90:0/3208907871 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6b781a8e00 con 0x7f6b7810e1e0 2024-04-14T15:28:50.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.146+0000 7f6b6b7fe700 1 -- 172.21.15.90:0/3208907871 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f6b70004810 con 0x7f6b7810e1e0 2024-04-14T15:28:50.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.146+0000 7f6b6b7fe700 1 -- 172.21.15.90:0/3208907871 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4258147082 0 0) 0x7f6b781a9fe0 con 0x7f6b7810a610 2024-04-14T15:28:50.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.146+0000 7f6b6b7fe700 1 -- 172.21.15.90:0/3208907871 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6b640031f0 con 0x7f6b7810a610 2024-04-14T15:28:50.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.146+0000 7f6b6b7fe700 1 -- 172.21.15.90:0/3208907871 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f6b74001870 con 0x7f6b7810a610 2024-04-14T15:28:50.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.146+0000 7f6b6b7fe700 1 -- 172.21.15.90:0/3208907871 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1365966912 0 0) 0x7f6b7806b250 con 0x7f6b78107760 2024-04-14T15:28:50.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.146+0000 7f6b6b7fe700 1 -- 172.21.15.90:0/3208907871 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6b781a9fe0 con 0x7f6b78107760 2024-04-14T15:28:50.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.146+0000 7f6b6b7fe700 1 -- 172.21.15.90:0/3208907871 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f6b6c0017a0 con 0x7f6b78107760 2024-04-14T15:28:50.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.146+0000 7f6b6b7fe700 1 -- 172.21.15.90:0/3208907871 <== mon.2 v1:172.21.15.90:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2619924331 0 0) 0x7f6b781a8e00 con 0x7f6b7810e1e0 2024-04-14T15:28:50.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.146+0000 7f6b6b7fe700 1 -- 172.21.15.90:0/3208907871 >> v1:172.21.15.92:6789/0 conn(0x7f6b78107760 legacy=0x7f6b7810cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:50.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.146+0000 7f6b6b7fe700 1 -- 172.21.15.90:0/3208907871 >> v1:172.21.15.90:6789/0 conn(0x7f6b7810a610 legacy=0x7f6b7810d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:50.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.146+0000 7f6b6b7fe700 1 -- 172.21.15.90:0/3208907871 --> v1:172.21.15.90:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b781ab1c0 con 0x7f6b7810e1e0 2024-04-14T15:28:50.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.146+0000 7f6b80468700 1 -- 172.21.15.90:0/3208907871 --> v1:172.21.15.90:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6b781a7e50 con 0x7f6b7810e1e0 2024-04-14T15:28:50.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.146+0000 7f6b80468700 10 client.48293.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:28:50.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.146+0000 7f6b80468700 1 -- 172.21.15.90:0/3208907871 --> v1:172.21.15.90:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f6b781a8310 con 0x7f6b7810e1e0 2024-04-14T15:28:50.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.146+0000 7f6b80468700 1 librados: init done 2024-04-14T15:28:50.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.146+0000 7f6b6b7fe700 1 -- 172.21.15.90:0/3208907871 <== mon.2 v1:172.21.15.90:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f6b70001440 con 0x7f6b7810e1e0 2024-04-14T15:28:50.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.146+0000 7f6b6b7fe700 1 -- 172.21.15.90:0/3208907871 <== mon.2 v1:172.21.15.90:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f6b700050a0 con 0x7f6b7810e1e0 2024-04-14T15:28:50.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.147+0000 7f6b80468700 10 client.48293.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:28:50.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.147+0000 7f6b80468700 15 client.48293.objecter _session_command_op_assign -1 1 2024-04-14T15:28:50.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.147+0000 7f6b80468700 20 client.48293.objecter _get_session osd=-1 returning homeless 2024-04-14T15:28:50.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.147+0000 7f6b80468700 10 client.48293.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:28:50.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.147+0000 7f6b80468700 1 -- 172.21.15.90:0/3208907871 --> v1:172.21.15.90:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f6b5c002920 con 0x7f6b7810e1e0 2024-04-14T15:28:50.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.148+0000 7f6b6b7fe700 1 -- 172.21.15.90:0/3208907871 <== mon.2 v1:172.21.15.90:6790/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f6b70005080 con 0x7f6b7810e1e0 2024-04-14T15:28:50.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.148+0000 7f6b6b7fe700 1 -- 172.21.15.90:0/3208907871 <== mon.2 v1:172.21.15.90:6790/0 8 ==== osd_map(920..920 src has 467..920) ==== 7517+0+0 (unknown 2649999903 0 0) 0x7f6b7008a9c0 con 0x7f6b7810e1e0 2024-04-14T15:28:50.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.148+0000 7f6b6b7fe700 10 client.48293.objecter ms_dispatch 0x7f6b78001550 osd_map(920..920 src has 467..920) 2024-04-14T15:28:50.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.148+0000 7f6b6b7fe700 3 client.48293.objecter handle_osd_map got epochs [920,920] > 0 2024-04-14T15:28:50.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.148+0000 7f6b6b7fe700 3 client.48293.objecter handle_osd_map decoding full epoch 920 2024-04-14T15:28:50.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.148+0000 7f6b6b7fe700 10 client.48293.objecter checking command 1 2024-04-14T15:28:50.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.149+0000 7f6b6b7fe700 20 client.48293.objecter _get_session s=0x7f6b6407d3b0 osd=0 3 2024-04-14T15:28:50.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.149+0000 7f6b6b7fe700 20 client.48293.objecter put_session s=0x7f6b6407d3b0 osd=0 3 2024-04-14T15:28:50.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.149+0000 7f6b6b7fe700 15 client.48293.objecter _session_command_op_remove -1 1 2024-04-14T15:28:50.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.149+0000 7f6b6b7fe700 20 client.48293.objecter _get_session s=0x7f6b6407d3b0 osd=0 3 2024-04-14T15:28:50.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.149+0000 7f6b6b7fe700 20 client.48293.objecter get_session s=0x7f6b6407d3b0 osd=0 3 2024-04-14T15:28:50.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.149+0000 7f6b6b7fe700 15 client.48293.objecter _session_command_op_assign 0 1 2024-04-14T15:28:50.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.149+0000 7f6b6b7fe700 20 client.48293.objecter put_session s=0x7f6b6407d3b0 osd=0 4 2024-04-14T15:28:50.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.149+0000 7f6b6b7fe700 10 client.48293.objecter _send_command 1 2024-04-14T15:28:50.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.149+0000 7f6b6b7fe700 1 -- 172.21.15.90:0/3208907871 --> v1:172.21.15.90:6801/3761806885 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6b64081770 con 0x7f6b6407da60 2024-04-14T15:28:50.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.149+0000 7f6b6b7fe700 20 client.48293.objecter dump_active .. 0 homeless 2024-04-14T15:28:50.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.149+0000 7f6b6b7fe700 1 -- 172.21.15.90:0/3208907871 <== mon.2 v1:172.21.15.90:6790/0 9 ==== mon_get_version_reply(handle=1 version=920) ==== 24+0+0 (unknown 622753966 0 0) 0x7f6b7008ae00 con 0x7f6b7810e1e0 2024-04-14T15:28:50.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.149+0000 7f6b6b7fe700 10 client.48293.objecter ms_handle_connect 0x7f6b64071320 2024-04-14T15:28:50.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.149+0000 7f6b6b7fe700 10 client.48293.objecter ms_handle_connect 0x7f6b6407da60 2024-04-14T15:28:50.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.149+0000 7f6b6b7fe700 1 -- 172.21.15.90:0/3208907871 <== osd.0 v1:172.21.15.90:6801/3761806885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6b64081770 con 0x7f6b6407da60 2024-04-14T15:28:50.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.149+0000 7f6b6b7fe700 10 client.48293.objecter ms_dispatch 0x7f6b78001550 command_reply(tid 1: 0 ) 2024-04-14T15:28:50.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.149+0000 7f6b6b7fe700 10 client.48293.objecter _finish_command 1 = system:0 2024-04-14T15:28:50.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.149+0000 7f6b6b7fe700 20 client.48293.objecter put_session s=0x7f6b6407d3b0 osd=0 4 2024-04-14T15:28:50.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.149+0000 7f6b6b7fe700 15 client.48293.objecter _session_command_op_remove 0 1 2024-04-14T15:28:50.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.165+0000 7f6b80468700 10 client.48293.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-14T15:28:50.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.165+0000 7f6b80468700 15 client.48293.objecter _session_command_op_assign -1 2 2024-04-14T15:28:50.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.165+0000 7f6b80468700 20 client.48293.objecter _get_session s=0x7f6b6407d3b0 osd=0 3 2024-04-14T15:28:50.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.165+0000 7f6b80468700 20 client.48293.objecter put_session s=0x7f6b6407d3b0 osd=0 3 2024-04-14T15:28:50.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.165+0000 7f6b80468700 20 client.48293.objecter _get_session s=0x7f6b6407d3b0 osd=0 3 2024-04-14T15:28:50.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.165+0000 7f6b80468700 15 client.48293.objecter _session_command_op_remove -1 2 2024-04-14T15:28:50.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.165+0000 7f6b80468700 20 client.48293.objecter get_session s=0x7f6b6407d3b0 osd=0 3 2024-04-14T15:28:50.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.165+0000 7f6b80468700 15 client.48293.objecter _session_command_op_assign 0 2 2024-04-14T15:28:50.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.165+0000 7f6b80468700 20 client.48293.objecter put_session s=0x7f6b6407d3b0 osd=0 4 2024-04-14T15:28:50.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.165+0000 7f6b80468700 10 client.48293.objecter _send_command 2 2024-04-14T15:28:50.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.165+0000 7f6b80468700 1 -- 172.21.15.90:0/3208907871 --> v1:172.21.15.90:6801/3761806885 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6b5c0011f0 con 0x7f6b6407da60 2024-04-14T15:28:50.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.166+0000 7f6b6b7fe700 1 -- 172.21.15.90:0/3208907871 <== osd.0 v1:172.21.15.90:6801/3761806885 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 226544643) 0x7f6b5c0011f0 con 0x7f6b6407da60 2024-04-14T15:28:50.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.166+0000 7f6b6b7fe700 10 client.48293.objecter ms_dispatch 0x7f6b78001550 command_reply(tid 2: 0 ) 2024-04-14T15:28:50.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.166+0000 7f6b6b7fe700 10 client.48293.objecter _finish_command 2 = system:0 2024-04-14T15:28:50.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.166+0000 7f6b6b7fe700 20 client.48293.objecter put_session s=0x7f6b6407d3b0 osd=0 4 2024-04-14T15:28:50.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.166+0000 7f6b6b7fe700 15 client.48293.objecter _session_command_op_remove 0 2 2024-04-14T15:28:50.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.166+0000 7f6b80468700 10 librados: watch_flush enter 2024-04-14T15:28:50.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.166+0000 7f6b80468700 10 librados: watch_flush exit 2024-04-14T15:28:50.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.166+0000 7f6b80468700 10 client.48293.objecter close_session for osd.0 2024-04-14T15:28:50.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.166+0000 7f6b80468700 1 -- 172.21.15.90:0/3208907871 >> v1:172.21.15.90:6801/3761806885 conn(0x7f6b6407da60 legacy=0x7f6b6407ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:50.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.166+0000 7f6b80468700 20 client.48293.objecter put_session s=0x7f6b6407d3b0 osd=0 1 2024-04-14T15:28:50.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.166+0000 7f6b80468700 20 client.48293.objecter shutdown clearing up homeless session... 2024-04-14T15:28:50.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.166+0000 7f6b80468700 10 client.48293.objecter successfully canceled tick 2024-04-14T15:28:50.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.166+0000 7f6b80468700 1 -- 172.21.15.90:0/3208907871 >> v1:172.21.15.90:6800/420935427 conn(0x7f6b64071320 legacy=0x7f6b640737e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:50.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.166+0000 7f6b80468700 1 -- 172.21.15.90:0/3208907871 >> v1:172.21.15.90:6790/0 conn(0x7f6b7810e1e0 legacy=0x7f6b781a66d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:50.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.166+0000 7f6b7ea05700 1 -- 172.21.15.90:0/3208907871 reap_dead start 2024-04-14T15:28:50.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.166+0000 7f6b80468700 1 -- 172.21.15.90:0/3208907871 shutdown_connections 2024-04-14T15:28:50.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.166+0000 7f6b80468700 1 -- 172.21.15.90:0/3208907871 >> 172.21.15.90:0/3208907871 conn(0x7f6b780fd290 msgr2=0x7f6b780ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:28:50.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.166+0000 7f6b80468700 1 -- 172.21.15.90:0/3208907871 shutdown_connections 2024-04-14T15:28:50.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.166+0000 7f6b80468700 1 -- 172.21.15.90:0/3208907871 wait complete. 2024-04-14T15:28:50.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:48.167+0000 7f6b80468700 1 librados: shutdown 2024-04-14T15:28:50.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:50 smithi092 ceph-mon[33966]: pgmap v1397: 174 pgs: 174 active+clean; 582 KiB data, 839 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-14T15:28:50.691 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:50 smithi090 ceph-mon[30399]: pgmap v1397: 174 pgs: 174 active+clean; 582 KiB data, 839 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-14T15:28:50.691 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:50 smithi090 ceph-mon[41670]: pgmap v1397: 174 pgs: 174 active+clean; 582 KiB data, 839 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-14T15:28:50.845 INFO:teuthology.orchestra.run.smithi090.stdout:176093659381 2024-04-14T15:28:50.845 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-14T15:28:51.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.336+0000 7fecf2973700 1 -- 172.21.15.90:0/1964529973 >> v1:172.21.15.90:6789/0 conn(0x7fecec1078e0 legacy=0x7fecec107cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:51.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.336+0000 7fecf2973700 1 -- 172.21.15.90:0/1964529973 shutdown_connections 2024-04-14T15:28:51.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.336+0000 7fecf2973700 1 -- 172.21.15.90:0/1964529973 >> 172.21.15.90:0/1964529973 conn(0x7fecec0fd2c0 msgr2=0x7fecec0ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:28:51.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.336+0000 7fecf2973700 1 -- 172.21.15.90:0/1964529973 shutdown_connections 2024-04-14T15:28:51.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.336+0000 7fecf2973700 1 -- 172.21.15.90:0/1964529973 wait complete. 2024-04-14T15:28:51.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.336+0000 7fecf2973700 1 librados: starting msgr at 2024-04-14T15:28:51.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.336+0000 7fecf2973700 1 librados: starting objecter 2024-04-14T15:28:51.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.336+0000 7fecf2973700 1 Processor -- start 2024-04-14T15:28:51.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.336+0000 7fecf2973700 1 -- start start 2024-04-14T15:28:51.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.336+0000 7fecf2973700 1 librados: setting wanted keys 2024-04-14T15:28:51.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.336+0000 7fecf2973700 1 librados: calling monclient init 2024-04-14T15:28:51.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.337+0000 7fecf2973700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fecec10e110 con 0x7fecec10a790 2024-04-14T15:28:51.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.337+0000 7fecf2973700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fecec1a8f80 con 0x7fecec1078e0 2024-04-14T15:28:51.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.337+0000 7fecf2973700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fecec1aa160 con 0x7fecec10e360 2024-04-14T15:28:51.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.337+0000 7feceb7fe700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fecec10a790 0x7fecec06b540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:43950/0 (socket says 172.21.15.90:43950) 2024-04-14T15:28:51.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.337+0000 7feceb7fe700 1 -- 172.21.15.90:0/1501637142 learned_addr learned my addr 172.21.15.90:0/1501637142 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:28:51.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.337+0000 7fece97fa700 10 client.?.objecter ms_handle_connect 0x7fecec10a790 2024-04-14T15:28:51.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.337+0000 7fece97fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:28:51.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.337+0000 7fece97fa700 10 client.?.objecter ms_handle_connect 0x7fecec10e360 2024-04-14T15:28:51.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.337+0000 7fece97fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:28:51.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.337+0000 7fece97fa700 10 client.?.objecter ms_handle_connect 0x7fecec1078e0 2024-04-14T15:28:51.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.337+0000 7fece97fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:28:51.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.337+0000 7fece97fa700 1 -- 172.21.15.90:0/1501637142 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 401142416 0 0) 0x7fecec10e110 con 0x7fecec10a790 2024-04-14T15:28:51.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.337+0000 7fece97fa700 1 -- 172.21.15.90:0/1501637142 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fecd40035b0 con 0x7fecec10a790 2024-04-14T15:28:51.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.337+0000 7fece97fa700 1 -- 172.21.15.90:0/1501637142 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3661410523 0 0) 0x7fecec1aa160 con 0x7fecec10e360 2024-04-14T15:28:51.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.337+0000 7fece97fa700 1 -- 172.21.15.90:0/1501637142 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fecec10e110 con 0x7fecec10e360 2024-04-14T15:28:51.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.337+0000 7fece97fa700 1 -- 172.21.15.90:0/1501637142 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1385236096 0 0) 0x7fecec1a8f80 con 0x7fecec1078e0 2024-04-14T15:28:51.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.337+0000 7fece97fa700 1 -- 172.21.15.90:0/1501637142 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fecec1aa160 con 0x7fecec1078e0 2024-04-14T15:28:51.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.337+0000 7fece97fa700 1 -- 172.21.15.90:0/1501637142 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1329071564 0 0) 0x7fecd40035b0 con 0x7fecec10a790 2024-04-14T15:28:51.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.337+0000 7fece97fa700 1 -- 172.21.15.90:0/1501637142 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fecec1a8f80 con 0x7fecec10a790 2024-04-14T15:28:51.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.337+0000 7fece97fa700 1 -- 172.21.15.90:0/1501637142 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fece00018d0 con 0x7fecec10a790 2024-04-14T15:28:51.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.337+0000 7fece97fa700 1 -- 172.21.15.90:0/1501637142 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 298751403 0 0) 0x7fecec10e110 con 0x7fecec10e360 2024-04-14T15:28:51.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.337+0000 7fece97fa700 1 -- 172.21.15.90:0/1501637142 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fecd40035b0 con 0x7fecec10e360 2024-04-14T15:28:51.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.338+0000 7fece97fa700 1 -- 172.21.15.90:0/1501637142 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fece4001af0 con 0x7fecec10e360 2024-04-14T15:28:51.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.338+0000 7fece97fa700 1 -- 172.21.15.90:0/1501637142 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3764730708 0 0) 0x7fecec1aa160 con 0x7fecec1078e0 2024-04-14T15:28:51.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.338+0000 7fece97fa700 1 -- 172.21.15.90:0/1501637142 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fecec10e110 con 0x7fecec1078e0 2024-04-14T15:28:51.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.338+0000 7fece97fa700 1 -- 172.21.15.90:0/1501637142 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fecdc001430 con 0x7fecec1078e0 2024-04-14T15:28:51.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.338+0000 7fece97fa700 1 -- 172.21.15.90:0/1501637142 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 657324317 0 0) 0x7fecec1a8f80 con 0x7fecec10a790 2024-04-14T15:28:51.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.338+0000 7fece97fa700 1 -- 172.21.15.90:0/1501637142 >> v1:172.21.15.90:6790/0 conn(0x7fecec10e360 legacy=0x7fecec1a6850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:51.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.338+0000 7fece97fa700 1 -- 172.21.15.90:0/1501637142 >> v1:172.21.15.92:6789/0 conn(0x7fecec1078e0 legacy=0x7fecec06ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:51.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.338+0000 7fece97fa700 1 -- 172.21.15.90:0/1501637142 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecec1ab380 con 0x7fecec10a790 2024-04-14T15:28:51.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.338+0000 7fecf2973700 1 -- 172.21.15.90:0/1501637142 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fecec1aa390 con 0x7fecec10a790 2024-04-14T15:28:51.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.338+0000 7fecf2973700 10 client.38349.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:28:51.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.338+0000 7fecf2973700 1 -- 172.21.15.90:0/1501637142 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fecec1aa850 con 0x7fecec10a790 2024-04-14T15:28:51.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.338+0000 7fecf2973700 1 librados: init done 2024-04-14T15:28:51.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.338+0000 7fece97fa700 1 -- 172.21.15.90:0/1501637142 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fece0003b20 con 0x7fecec10a790 2024-04-14T15:28:51.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.338+0000 7fece97fa700 1 -- 172.21.15.90:0/1501637142 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fece0005ce0 con 0x7fecec10a790 2024-04-14T15:28:51.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.338+0000 7fecf2973700 10 client.38349.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:28:51.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.338+0000 7fecf2973700 15 client.38349.objecter _session_command_op_assign -1 1 2024-04-14T15:28:51.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.338+0000 7fecf2973700 20 client.38349.objecter _get_session osd=-1 returning homeless 2024-04-14T15:28:51.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.338+0000 7fecf2973700 10 client.38349.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:28:51.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.338+0000 7fecf2973700 1 -- 172.21.15.90:0/1501637142 --> v1:172.21.15.90:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fecec0693d0 con 0x7fecec10a790 2024-04-14T15:28:51.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.339+0000 7fece97fa700 1 -- 172.21.15.90:0/1501637142 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fece0006fe0 con 0x7fecec10a790 2024-04-14T15:28:51.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.340+0000 7fece97fa700 1 -- 172.21.15.90:0/1501637142 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(920..920 src has 467..920) ==== 7517+0+0 (unknown 2649999903 0 0) 0x7fece008a750 con 0x7fecec10a790 2024-04-14T15:28:51.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.340+0000 7fece97fa700 10 client.38349.objecter ms_dispatch 0x7fecec001550 osd_map(920..920 src has 467..920) 2024-04-14T15:28:51.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.340+0000 7fece97fa700 3 client.38349.objecter handle_osd_map got epochs [920,920] > 0 2024-04-14T15:28:51.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.340+0000 7fece97fa700 3 client.38349.objecter handle_osd_map decoding full epoch 920 2024-04-14T15:28:51.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.340+0000 7fece97fa700 10 client.38349.objecter checking command 1 2024-04-14T15:28:51.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.340+0000 7fece97fa700 20 client.38349.objecter _get_session s=0x7fecd407d3c0 osd=3 3 2024-04-14T15:28:51.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.340+0000 7fece97fa700 20 client.38349.objecter put_session s=0x7fecd407d3c0 osd=3 3 2024-04-14T15:28:51.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.340+0000 7fece97fa700 15 client.38349.objecter _session_command_op_remove -1 1 2024-04-14T15:28:51.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.340+0000 7fece97fa700 20 client.38349.objecter _get_session s=0x7fecd407d3c0 osd=3 3 2024-04-14T15:28:51.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.340+0000 7fece97fa700 20 client.38349.objecter get_session s=0x7fecd407d3c0 osd=3 3 2024-04-14T15:28:51.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.340+0000 7fece97fa700 15 client.38349.objecter _session_command_op_assign 3 1 2024-04-14T15:28:51.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.340+0000 7fece97fa700 20 client.38349.objecter put_session s=0x7fecd407d3c0 osd=3 4 2024-04-14T15:28:51.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.340+0000 7fece97fa700 10 client.38349.objecter _send_command 1 2024-04-14T15:28:51.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.340+0000 7fece97fa700 1 -- 172.21.15.90:0/1501637142 --> v1:172.21.15.90:6813/2270166774 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fecd4081780 con 0x7fecd407da70 2024-04-14T15:28:51.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.340+0000 7fece97fa700 20 client.38349.objecter dump_active .. 0 homeless 2024-04-14T15:28:51.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.340+0000 7fece97fa700 1 -- 172.21.15.90:0/1501637142 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_get_version_reply(handle=1 version=920) ==== 24+0+0 (unknown 622753966 0 0) 0x7fece008aaf0 con 0x7fecec10a790 2024-04-14T15:28:51.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.340+0000 7fece97fa700 10 client.38349.objecter ms_handle_connect 0x7fecd4071440 2024-04-14T15:28:51.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.340+0000 7fece97fa700 10 client.38349.objecter ms_handle_connect 0x7fecd407da70 2024-04-14T15:28:51.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.341+0000 7fece97fa700 1 -- 172.21.15.90:0/1501637142 <== osd.3 v1:172.21.15.90:6813/2270166774 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fecd4081780 con 0x7fecd407da70 2024-04-14T15:28:51.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.341+0000 7fece97fa700 10 client.38349.objecter ms_dispatch 0x7fecec001550 command_reply(tid 1: 0 ) 2024-04-14T15:28:51.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.341+0000 7fece97fa700 10 client.38349.objecter _finish_command 1 = system:0 2024-04-14T15:28:51.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.341+0000 7fece97fa700 20 client.38349.objecter put_session s=0x7fecd407d3c0 osd=3 4 2024-04-14T15:28:51.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.341+0000 7fece97fa700 15 client.38349.objecter _session_command_op_remove 3 1 2024-04-14T15:28:51.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.357+0000 7fecf2973700 10 client.38349.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-14T15:28:51.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.357+0000 7fecf2973700 15 client.38349.objecter _session_command_op_assign -1 2 2024-04-14T15:28:51.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.357+0000 7fecf2973700 20 client.38349.objecter _get_session s=0x7fecd407d3c0 osd=3 3 2024-04-14T15:28:51.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.357+0000 7fecf2973700 20 client.38349.objecter put_session s=0x7fecd407d3c0 osd=3 3 2024-04-14T15:28:51.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.357+0000 7fecf2973700 20 client.38349.objecter _get_session s=0x7fecd407d3c0 osd=3 3 2024-04-14T15:28:51.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.357+0000 7fecf2973700 15 client.38349.objecter _session_command_op_remove -1 2 2024-04-14T15:28:51.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.357+0000 7fecf2973700 20 client.38349.objecter get_session s=0x7fecd407d3c0 osd=3 3 2024-04-14T15:28:51.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.357+0000 7fecf2973700 15 client.38349.objecter _session_command_op_assign 3 2 2024-04-14T15:28:51.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.357+0000 7fecf2973700 20 client.38349.objecter put_session s=0x7fecd407d3c0 osd=3 4 2024-04-14T15:28:51.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.357+0000 7fecf2973700 10 client.38349.objecter _send_command 2 2024-04-14T15:28:51.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.357+0000 7fecf2973700 1 -- 172.21.15.90:0/1501637142 --> v1:172.21.15.90:6813/2270166774 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fecec10d400 con 0x7fecd407da70 2024-04-14T15:28:51.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.358+0000 7fece97fa700 1 -- 172.21.15.90:0/1501637142 <== osd.3 v1:172.21.15.90:6813/2270166774 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (unknown 0 0 3350153181) 0x7fecec10d400 con 0x7fecd407da70 2024-04-14T15:28:51.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.358+0000 7fece97fa700 10 client.38349.objecter ms_dispatch 0x7fecec001550 command_reply(tid 2: 0 ) 2024-04-14T15:28:51.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.358+0000 7fece97fa700 10 client.38349.objecter _finish_command 2 = system:0 2024-04-14T15:28:51.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.358+0000 7fece97fa700 20 client.38349.objecter put_session s=0x7fecd407d3c0 osd=3 4 2024-04-14T15:28:51.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.358+0000 7fece97fa700 15 client.38349.objecter _session_command_op_remove 3 2 2024-04-14T15:28:51.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.358+0000 7fecf2973700 10 librados: watch_flush enter 2024-04-14T15:28:51.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.358+0000 7fecf2973700 10 librados: watch_flush exit 2024-04-14T15:28:51.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.358+0000 7fecf2973700 10 client.38349.objecter close_session for osd.3 2024-04-14T15:28:51.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.358+0000 7fecf2973700 1 -- 172.21.15.90:0/1501637142 >> v1:172.21.15.90:6813/2270166774 conn(0x7fecd407da70 legacy=0x7fecd407ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:51.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.358+0000 7fecf2973700 20 client.38349.objecter put_session s=0x7fecd407d3c0 osd=3 1 2024-04-14T15:28:51.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.358+0000 7fecf2973700 20 client.38349.objecter shutdown clearing up homeless session... 2024-04-14T15:28:51.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.358+0000 7fecf2973700 10 client.38349.objecter successfully canceled tick 2024-04-14T15:28:51.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.358+0000 7fecf2973700 1 -- 172.21.15.90:0/1501637142 >> v1:172.21.15.90:6800/420935427 conn(0x7fecd4071440 legacy=0x7fecd4073900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:51.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.358+0000 7fecf2973700 1 -- 172.21.15.90:0/1501637142 >> v1:172.21.15.90:6789/0 conn(0x7fecec10a790 legacy=0x7fecec06b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:51.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.358+0000 7fecf0f10700 1 -- 172.21.15.90:0/1501637142 reap_dead start 2024-04-14T15:28:51.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.358+0000 7fecf2973700 1 -- 172.21.15.90:0/1501637142 shutdown_connections 2024-04-14T15:28:51.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.358+0000 7fecf2973700 1 -- 172.21.15.90:0/1501637142 >> 172.21.15.90:0/1501637142 conn(0x7fecec0fd2c0 msgr2=0x7fecec111860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:28:51.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.358+0000 7fecf2973700 1 -- 172.21.15.90:0/1501637142 shutdown_connections 2024-04-14T15:28:51.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.358+0000 7fecf2973700 1 -- 172.21.15.90:0/1501637142 wait complete. 2024-04-14T15:28:51.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:51.358+0000 7fecf2973700 1 librados: shutdown 2024-04-14T15:28:51.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:51 smithi092 ceph-mon[33966]: 11.f scrub starts 2024-04-14T15:28:51.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:51 smithi092 ceph-mon[33966]: 11.f scrub ok 2024-04-14T15:28:51.651 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:51 smithi090 ceph-mon[41670]: 11.f scrub starts 2024-04-14T15:28:51.651 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:51 smithi090 ceph-mon[41670]: 11.f scrub ok 2024-04-14T15:28:51.651 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:51 smithi090 ceph-mon[30399]: 11.f scrub starts 2024-04-14T15:28:51.651 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:51 smithi090 ceph-mon[30399]: 11.f scrub ok 2024-04-14T15:28:52.117 INFO:teuthology.orchestra.run.smithi090.stdout:107374182655 2024-04-14T15:28:52.118 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-14T15:28:52.615 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:52 smithi090 ceph-mon[41670]: 4.14 scrub starts 2024-04-14T15:28:52.616 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:52 smithi090 ceph-mon[41670]: 4.14 scrub ok 2024-04-14T15:28:52.616 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:52 smithi090 ceph-mon[41670]: pgmap v1398: 174 pgs: 174 active+clean; 582 KiB data, 839 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:52.616 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:52 smithi090 ceph-mon[41670]: 2.4 scrub starts 2024-04-14T15:28:52.616 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:52 smithi090 ceph-mon[41670]: 2.4 scrub ok 2024-04-14T15:28:52.616 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:52 smithi090 ceph-mon[30399]: 4.14 scrub starts 2024-04-14T15:28:52.616 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:52 smithi090 ceph-mon[30399]: 4.14 scrub ok 2024-04-14T15:28:52.616 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:52 smithi090 ceph-mon[30399]: pgmap v1398: 174 pgs: 174 active+clean; 582 KiB data, 839 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:52.617 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:52 smithi090 ceph-mon[30399]: 2.4 scrub starts 2024-04-14T15:28:52.617 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:52 smithi090 ceph-mon[30399]: 2.4 scrub ok 2024-04-14T15:28:52.617 INFO:teuthology.orchestra.run.smithi090.stdout:34359738637 2024-04-14T15:28:52.618 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-14T15:28:52.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:52 smithi092 ceph-mon[33966]: 4.14 scrub starts 2024-04-14T15:28:52.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:52 smithi092 ceph-mon[33966]: 4.14 scrub ok 2024-04-14T15:28:52.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:52 smithi092 ceph-mon[33966]: pgmap v1398: 174 pgs: 174 active+clean; 582 KiB data, 839 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:52.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:52 smithi092 ceph-mon[33966]: 2.4 scrub starts 2024-04-14T15:28:52.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:52 smithi092 ceph-mon[33966]: 2.4 scrub ok 2024-04-14T15:28:53.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.072+0000 7fbbdaabd700 1 -- 172.21.15.90:0/4197766089 >> v1:172.21.15.90:6789/0 conn(0x7fbbd4108880 legacy=0x7fbbd4108c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:53.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.072+0000 7fbbdaabd700 1 -- 172.21.15.90:0/4197766089 shutdown_connections 2024-04-14T15:28:53.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.072+0000 7fbbdaabd700 1 -- 172.21.15.90:0/4197766089 >> 172.21.15.90:0/4197766089 conn(0x7fbbd40fe3d0 msgr2=0x7fbbd41007f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:28:53.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.072+0000 7fbbdaabd700 1 -- 172.21.15.90:0/4197766089 shutdown_connections 2024-04-14T15:28:53.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.072+0000 7fbbdaabd700 1 -- 172.21.15.90:0/4197766089 wait complete. 2024-04-14T15:28:53.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.073+0000 7fbbdaabd700 1 librados: starting msgr at 2024-04-14T15:28:53.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.073+0000 7fbbdaabd700 1 librados: starting objecter 2024-04-14T15:28:53.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.073+0000 7fbbdaabd700 1 Processor -- start 2024-04-14T15:28:53.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.073+0000 7fbbdaabd700 1 -- start start 2024-04-14T15:28:53.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.073+0000 7fbbdaabd700 1 librados: setting wanted keys 2024-04-14T15:28:53.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.073+0000 7fbbdaabd700 1 librados: calling monclient init 2024-04-14T15:28:53.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.073+0000 7fbbdaabd700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbbd410f080 con 0x7fbbd4108880 2024-04-14T15:28:53.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.073+0000 7fbbdaabd700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbbd41a8f10 con 0x7fbbd410b730 2024-04-14T15:28:53.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.073+0000 7fbbdaabd700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbbd41aa0f0 con 0x7fbbd410f300 2024-04-14T15:28:53.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.073+0000 7fbbd8859700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fbbd4108880 0x7fbbd410c010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:43964/0 (socket says 172.21.15.90:43964) 2024-04-14T15:28:53.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.073+0000 7fbbd8859700 1 -- 172.21.15.90:0/1738700284 learned_addr learned my addr 172.21.15.90:0/1738700284 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:28:53.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.073+0000 7fbbd1ffb700 10 client.?.objecter ms_handle_connect 0x7fbbd4108880 2024-04-14T15:28:53.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.073+0000 7fbbd1ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:28:53.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.073+0000 7fbbd1ffb700 10 client.?.objecter ms_handle_connect 0x7fbbd410f300 2024-04-14T15:28:53.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.073+0000 7fbbd1ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:28:53.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.073+0000 7fbbd1ffb700 10 client.?.objecter ms_handle_connect 0x7fbbd410b730 2024-04-14T15:28:53.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.073+0000 7fbbd1ffb700 10 client.?.objecter resend_mon_ops 2024-04-14T15:28:53.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.073+0000 7fbbd1ffb700 1 -- 172.21.15.90:0/1738700284 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 458195555 0 0) 0x7fbbd41aa0f0 con 0x7fbbd410f300 2024-04-14T15:28:53.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.073+0000 7fbbd1ffb700 1 -- 172.21.15.90:0/1738700284 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbbc40031f0 con 0x7fbbd410f300 2024-04-14T15:28:53.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.073+0000 7fbbd1ffb700 1 -- 172.21.15.90:0/1738700284 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3456005836 0 0) 0x7fbbd410f080 con 0x7fbbd4108880 2024-04-14T15:28:53.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.073+0000 7fbbd1ffb700 1 -- 172.21.15.90:0/1738700284 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbbd41aa0f0 con 0x7fbbd4108880 2024-04-14T15:28:53.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.073+0000 7fbbd1ffb700 1 -- 172.21.15.90:0/1738700284 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3909063086 0 0) 0x7fbbd41a8f10 con 0x7fbbd410b730 2024-04-14T15:28:53.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.073+0000 7fbbd1ffb700 1 -- 172.21.15.90:0/1738700284 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbbd410f080 con 0x7fbbd410b730 2024-04-14T15:28:53.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.073+0000 7fbbd1ffb700 1 -- 172.21.15.90:0/1738700284 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2018995722 0 0) 0x7fbbc40031f0 con 0x7fbbd410f300 2024-04-14T15:28:53.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.074+0000 7fbbd1ffb700 1 -- 172.21.15.90:0/1738700284 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbbd41a8f10 con 0x7fbbd410f300 2024-04-14T15:28:53.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.074+0000 7fbbd1ffb700 1 -- 172.21.15.90:0/1738700284 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fbbcc001c30 con 0x7fbbd410f300 2024-04-14T15:28:53.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.074+0000 7fbbd1ffb700 1 -- 172.21.15.90:0/1738700284 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4120612690 0 0) 0x7fbbd41aa0f0 con 0x7fbbd4108880 2024-04-14T15:28:53.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.074+0000 7fbbd1ffb700 1 -- 172.21.15.90:0/1738700284 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbbc40031f0 con 0x7fbbd4108880 2024-04-14T15:28:53.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.074+0000 7fbbd1ffb700 1 -- 172.21.15.90:0/1738700284 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3562455708 0 0) 0x7fbbd410f080 con 0x7fbbd410b730 2024-04-14T15:28:53.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.074+0000 7fbbd1ffb700 1 -- 172.21.15.90:0/1738700284 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbbd41aa0f0 con 0x7fbbd410b730 2024-04-14T15:28:53.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.074+0000 7fbbd1ffb700 1 -- 172.21.15.90:0/1738700284 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fbbc8001430 con 0x7fbbd4108880 2024-04-14T15:28:53.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.074+0000 7fbbd1ffb700 1 -- 172.21.15.90:0/1738700284 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fbbc0001830 con 0x7fbbd410b730 2024-04-14T15:28:53.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.074+0000 7fbbd1ffb700 1 -- 172.21.15.90:0/1738700284 <== mon.2 v1:172.21.15.90:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 23123089 0 0) 0x7fbbd41a8f10 con 0x7fbbd410f300 2024-04-14T15:28:53.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.074+0000 7fbbd1ffb700 1 -- 172.21.15.90:0/1738700284 >> v1:172.21.15.92:6789/0 conn(0x7fbbd410b730 legacy=0x7fbbd410c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:53.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.074+0000 7fbbd1ffb700 1 -- 172.21.15.90:0/1738700284 >> v1:172.21.15.90:6789/0 conn(0x7fbbd4108880 legacy=0x7fbbd410c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:53.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.074+0000 7fbbd1ffb700 1 -- 172.21.15.90:0/1738700284 --> v1:172.21.15.90:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbd41ab330 con 0x7fbbd410f300 2024-04-14T15:28:53.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.074+0000 7fbbdaabd700 1 -- 172.21.15.90:0/1738700284 --> v1:172.21.15.90:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbbd41a7f60 con 0x7fbbd410f300 2024-04-14T15:28:53.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.074+0000 7fbbd1ffb700 1 -- 172.21.15.90:0/1738700284 <== mon.2 v1:172.21.15.90:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fbbcc003ea0 con 0x7fbbd410f300 2024-04-14T15:28:53.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.074+0000 7fbbd1ffb700 1 -- 172.21.15.90:0/1738700284 <== mon.2 v1:172.21.15.90:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fbbcc004e30 con 0x7fbbd410f300 2024-04-14T15:28:53.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.074+0000 7fbbdaabd700 10 client.48305.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:28:53.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.074+0000 7fbbdaabd700 1 -- 172.21.15.90:0/1738700284 --> v1:172.21.15.90:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fbbd41a8420 con 0x7fbbd410f300 2024-04-14T15:28:53.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.074+0000 7fbbdaabd700 1 librados: init done 2024-04-14T15:28:53.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.075+0000 7fbbdaabd700 10 client.48305.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:28:53.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.075+0000 7fbbdaabd700 15 client.48305.objecter _session_command_op_assign -1 1 2024-04-14T15:28:53.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.075+0000 7fbbdaabd700 20 client.48305.objecter _get_session osd=-1 returning homeless 2024-04-14T15:28:53.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.075+0000 7fbbdaabd700 10 client.48305.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:28:53.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.075+0000 7fbbdaabd700 1 -- 172.21.15.90:0/1738700284 --> v1:172.21.15.90:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fbbd4103fc0 con 0x7fbbd410f300 2024-04-14T15:28:53.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.075+0000 7fbbd1ffb700 1 -- 172.21.15.90:0/1738700284 <== mon.2 v1:172.21.15.90:6790/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fbbcc003a00 con 0x7fbbd410f300 2024-04-14T15:28:53.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.076+0000 7fbbd1ffb700 1 -- 172.21.15.90:0/1738700284 <== mon.2 v1:172.21.15.90:6790/0 8 ==== osd_map(920..920 src has 467..920) ==== 7517+0+0 (unknown 2649999903 0 0) 0x7fbbcc089920 con 0x7fbbd410f300 2024-04-14T15:28:53.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.076+0000 7fbbd1ffb700 10 client.48305.objecter ms_dispatch 0x7fbbd4001550 osd_map(920..920 src has 467..920) 2024-04-14T15:28:53.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.076+0000 7fbbd1ffb700 3 client.48305.objecter handle_osd_map got epochs [920,920] > 0 2024-04-14T15:28:53.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.076+0000 7fbbd1ffb700 3 client.48305.objecter handle_osd_map decoding full epoch 920 2024-04-14T15:28:53.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.076+0000 7fbbd1ffb700 10 client.48305.objecter checking command 1 2024-04-14T15:28:53.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.076+0000 7fbbd1ffb700 20 client.48305.objecter _get_session s=0x7fbbc407d310 osd=4 3 2024-04-14T15:28:53.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.076+0000 7fbbd1ffb700 20 client.48305.objecter put_session s=0x7fbbc407d310 osd=4 3 2024-04-14T15:28:53.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.076+0000 7fbbd1ffb700 15 client.48305.objecter _session_command_op_remove -1 1 2024-04-14T15:28:53.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.076+0000 7fbbd1ffb700 20 client.48305.objecter _get_session s=0x7fbbc407d310 osd=4 3 2024-04-14T15:28:53.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.076+0000 7fbbd1ffb700 20 client.48305.objecter get_session s=0x7fbbc407d310 osd=4 3 2024-04-14T15:28:53.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.076+0000 7fbbd1ffb700 15 client.48305.objecter _session_command_op_assign 4 1 2024-04-14T15:28:53.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.076+0000 7fbbd1ffb700 20 client.48305.objecter put_session s=0x7fbbc407d310 osd=4 4 2024-04-14T15:28:53.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.076+0000 7fbbd1ffb700 10 client.48305.objecter _send_command 1 2024-04-14T15:28:53.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.076+0000 7fbbd1ffb700 1 -- 172.21.15.90:0/1738700284 --> v1:172.21.15.92:6800/3887549733 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbbc40816d0 con 0x7fbbc407d9c0 2024-04-14T15:28:53.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.076+0000 7fbbd1ffb700 20 client.48305.objecter dump_active .. 0 homeless 2024-04-14T15:28:53.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.076+0000 7fbbd1ffb700 1 -- 172.21.15.90:0/1738700284 <== mon.2 v1:172.21.15.90:6790/0 9 ==== mon_get_version_reply(handle=1 version=920) ==== 24+0+0 (unknown 622753966 0 0) 0x7fbbcc089d30 con 0x7fbbd410f300 2024-04-14T15:28:53.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.076+0000 7fbbd1ffb700 10 client.48305.objecter ms_handle_connect 0x7fbbc4071390 2024-04-14T15:28:53.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.076+0000 7fbbd1ffb700 10 client.48305.objecter ms_handle_connect 0x7fbbc407d9c0 2024-04-14T15:28:53.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.077+0000 7fbbd1ffb700 1 -- 172.21.15.90:0/1738700284 <== osd.4 v1:172.21.15.92:6800/3887549733 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fbbc40816d0 con 0x7fbbc407d9c0 2024-04-14T15:28:53.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.077+0000 7fbbd1ffb700 10 client.48305.objecter ms_dispatch 0x7fbbd4001550 command_reply(tid 1: 0 ) 2024-04-14T15:28:53.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.077+0000 7fbbd1ffb700 10 client.48305.objecter _finish_command 1 = system:0 2024-04-14T15:28:53.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.077+0000 7fbbd1ffb700 20 client.48305.objecter put_session s=0x7fbbc407d310 osd=4 4 2024-04-14T15:28:53.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.077+0000 7fbbd1ffb700 15 client.48305.objecter _session_command_op_remove 4 1 2024-04-14T15:28:53.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.095+0000 7fbbdaabd700 10 client.48305.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-14T15:28:53.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.095+0000 7fbbdaabd700 15 client.48305.objecter _session_command_op_assign -1 2 2024-04-14T15:28:53.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.095+0000 7fbbdaabd700 20 client.48305.objecter _get_session s=0x7fbbc407d310 osd=4 3 2024-04-14T15:28:53.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.095+0000 7fbbdaabd700 20 client.48305.objecter put_session s=0x7fbbc407d310 osd=4 3 2024-04-14T15:28:53.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.095+0000 7fbbdaabd700 20 client.48305.objecter _get_session s=0x7fbbc407d310 osd=4 3 2024-04-14T15:28:53.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.095+0000 7fbbdaabd700 15 client.48305.objecter _session_command_op_remove -1 2 2024-04-14T15:28:53.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.095+0000 7fbbdaabd700 20 client.48305.objecter get_session s=0x7fbbc407d310 osd=4 3 2024-04-14T15:28:53.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.095+0000 7fbbdaabd700 15 client.48305.objecter _session_command_op_assign 4 2 2024-04-14T15:28:53.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.095+0000 7fbbdaabd700 20 client.48305.objecter put_session s=0x7fbbc407d310 osd=4 4 2024-04-14T15:28:53.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.095+0000 7fbbdaabd700 10 client.48305.objecter _send_command 2 2024-04-14T15:28:53.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.095+0000 7fbbdaabd700 1 -- 172.21.15.90:0/1738700284 --> v1:172.21.15.92:6800/3887549733 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fbbd40693d0 con 0x7fbbc407d9c0 2024-04-14T15:28:53.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.095+0000 7fbbd1ffb700 1 -- 172.21.15.90:0/1738700284 <== osd.4 v1:172.21.15.92:6800/3887549733 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (unknown 0 0 1571423181) 0x7fbbd40693d0 con 0x7fbbc407d9c0 2024-04-14T15:28:53.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.095+0000 7fbbd1ffb700 10 client.48305.objecter ms_dispatch 0x7fbbd4001550 command_reply(tid 2: 0 ) 2024-04-14T15:28:53.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.095+0000 7fbbd1ffb700 10 client.48305.objecter _finish_command 2 = system:0 2024-04-14T15:28:53.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.095+0000 7fbbd1ffb700 20 client.48305.objecter put_session s=0x7fbbc407d310 osd=4 4 2024-04-14T15:28:53.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.095+0000 7fbbd1ffb700 15 client.48305.objecter _session_command_op_remove 4 2 2024-04-14T15:28:53.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.095+0000 7fbbdaabd700 10 librados: watch_flush enter 2024-04-14T15:28:53.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.095+0000 7fbbdaabd700 10 librados: watch_flush exit 2024-04-14T15:28:53.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.095+0000 7fbbdaabd700 10 client.48305.objecter close_session for osd.4 2024-04-14T15:28:53.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.095+0000 7fbbdaabd700 1 -- 172.21.15.90:0/1738700284 >> v1:172.21.15.92:6800/3887549733 conn(0x7fbbc407d9c0 legacy=0x7fbbc407fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:53.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.095+0000 7fbbdaabd700 20 client.48305.objecter put_session s=0x7fbbc407d310 osd=4 1 2024-04-14T15:28:53.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.095+0000 7fbbdaabd700 20 client.48305.objecter shutdown clearing up homeless session... 2024-04-14T15:28:53.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.095+0000 7fbbdaabd700 10 client.48305.objecter successfully canceled tick 2024-04-14T15:28:53.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.095+0000 7fbbdaabd700 1 -- 172.21.15.90:0/1738700284 >> v1:172.21.15.90:6800/420935427 conn(0x7fbbc4071390 legacy=0x7fbbc4073850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:53.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.095+0000 7fbbdaabd700 1 -- 172.21.15.90:0/1738700284 >> v1:172.21.15.90:6790/0 conn(0x7fbbd410f300 legacy=0x7fbbd41a67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:53.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.095+0000 7fbbd905a700 1 -- 172.21.15.90:0/1738700284 reap_dead start 2024-04-14T15:28:53.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.096+0000 7fbbdaabd700 1 -- 172.21.15.90:0/1738700284 shutdown_connections 2024-04-14T15:28:53.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.096+0000 7fbbdaabd700 1 -- 172.21.15.90:0/1738700284 >> 172.21.15.90:0/1738700284 conn(0x7fbbd40fe3d0 msgr2=0x7fbbd4102880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:28:53.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.096+0000 7fbbdaabd700 1 -- 172.21.15.90:0/1738700284 shutdown_connections 2024-04-14T15:28:53.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.096+0000 7fbbdaabd700 1 -- 172.21.15.90:0/1738700284 wait complete. 2024-04-14T15:28:53.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:53.096+0000 7fbbdaabd700 1 librados: shutdown 2024-04-14T15:28:53.552 INFO:teuthology.orchestra.run.smithi090.stdout:249108103403 2024-04-14T15:28:53.552 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-14T15:28:53.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:53 smithi092 ceph-mon[33966]: 5.17 scrub starts 2024-04-14T15:28:53.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:53 smithi092 ceph-mon[33966]: 5.17 scrub ok 2024-04-14T15:28:53.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:53 smithi092 ceph-mon[33966]: 2.8 scrub starts 2024-04-14T15:28:53.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:53 smithi092 ceph-mon[33966]: 2.8 scrub ok 2024-04-14T15:28:53.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:53 smithi092 ceph-mon[33966]: 5.8 deep-scrub starts 2024-04-14T15:28:53.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:53 smithi092 ceph-mon[33966]: 5.8 deep-scrub ok 2024-04-14T15:28:53.691 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:53 smithi090 ceph-mon[41670]: 5.17 scrub starts 2024-04-14T15:28:53.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:53 smithi090 ceph-mon[41670]: 5.17 scrub ok 2024-04-14T15:28:53.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:53 smithi090 ceph-mon[41670]: 2.8 scrub starts 2024-04-14T15:28:53.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:53 smithi090 ceph-mon[41670]: 2.8 scrub ok 2024-04-14T15:28:53.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:53 smithi090 ceph-mon[41670]: 5.8 deep-scrub starts 2024-04-14T15:28:53.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:53 smithi090 ceph-mon[41670]: 5.8 deep-scrub ok 2024-04-14T15:28:53.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:53 smithi090 ceph-mon[30399]: 5.17 scrub starts 2024-04-14T15:28:53.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:53 smithi090 ceph-mon[30399]: 5.17 scrub ok 2024-04-14T15:28:53.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:53 smithi090 ceph-mon[30399]: 2.8 scrub starts 2024-04-14T15:28:53.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:53 smithi090 ceph-mon[30399]: 2.8 scrub ok 2024-04-14T15:28:53.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:53 smithi090 ceph-mon[30399]: 5.8 deep-scrub starts 2024-04-14T15:28:53.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:53 smithi090 ceph-mon[30399]: 5.8 deep-scrub ok 2024-04-14T15:28:54.112 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:28:53 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:28:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:28:54.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.217+0000 7fd8e9c0b700 1 -- 172.21.15.90:0/3262548551 >> v1:172.21.15.90:6789/0 conn(0x7fd8e4103a20 legacy=0x7fd8e4103e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:54.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.217+0000 7fd8e9c0b700 1 -- 172.21.15.90:0/3262548551 shutdown_connections 2024-04-14T15:28:54.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.217+0000 7fd8e9c0b700 1 -- 172.21.15.90:0/3262548551 >> 172.21.15.90:0/3262548551 conn(0x7fd8e40fd290 msgr2=0x7fd8e40ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:28:54.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.217+0000 7fd8e9c0b700 1 -- 172.21.15.90:0/3262548551 shutdown_connections 2024-04-14T15:28:54.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.217+0000 7fd8e9c0b700 1 -- 172.21.15.90:0/3262548551 wait complete. 2024-04-14T15:28:54.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.217+0000 7fd8e9c0b700 1 librados: starting msgr at 2024-04-14T15:28:54.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.217+0000 7fd8e9c0b700 1 librados: starting objecter 2024-04-14T15:28:54.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.218+0000 7fd8e9c0b700 1 Processor -- start 2024-04-14T15:28:54.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.218+0000 7fd8e9c0b700 1 -- start start 2024-04-14T15:28:54.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.218+0000 7fd8e9c0b700 1 librados: setting wanted keys 2024-04-14T15:28:54.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.218+0000 7fd8e9c0b700 1 librados: calling monclient init 2024-04-14T15:28:54.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.218+0000 7fd8e9c0b700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd8e41a7910 con 0x7fd8e410b830 2024-04-14T15:28:54.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.218+0000 7fd8e9c0b700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd8e41a8b30 con 0x7fd8e4103a20 2024-04-14T15:28:54.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.218+0000 7fd8e9c0b700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd8e41a9d70 con 0x7fd8e410f340 2024-04-14T15:28:54.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.218+0000 7fd8e2ffd700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fd8e410b830 0x7fd8e41a2760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:43986/0 (socket says 172.21.15.90:43986) 2024-04-14T15:28:54.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.218+0000 7fd8e2ffd700 1 -- 172.21.15.90:0/380550825 learned_addr learned my addr 172.21.15.90:0/380550825 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:28:54.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.218+0000 7fd8e0ff9700 10 client.?.objecter ms_handle_connect 0x7fd8e410f340 2024-04-14T15:28:54.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.218+0000 7fd8e0ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:28:54.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.218+0000 7fd8e0ff9700 10 client.?.objecter ms_handle_connect 0x7fd8e410b830 2024-04-14T15:28:54.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.218+0000 7fd8e0ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:28:54.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.218+0000 7fd8e0ff9700 10 client.?.objecter ms_handle_connect 0x7fd8e4103a20 2024-04-14T15:28:54.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.218+0000 7fd8e0ff9700 10 client.?.objecter resend_mon_ops 2024-04-14T15:28:54.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.218+0000 7fd8e0ff9700 1 -- 172.21.15.90:0/380550825 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 282925212 0 0) 0x7fd8e41a9d70 con 0x7fd8e410f340 2024-04-14T15:28:54.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.218+0000 7fd8e0ff9700 1 -- 172.21.15.90:0/380550825 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd8cc0035b0 con 0x7fd8e410f340 2024-04-14T15:28:54.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.218+0000 7fd8e0ff9700 1 -- 172.21.15.90:0/380550825 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2954700551 0 0) 0x7fd8e41a7910 con 0x7fd8e410b830 2024-04-14T15:28:54.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.219+0000 7fd8e0ff9700 1 -- 172.21.15.90:0/380550825 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd8e41a9d70 con 0x7fd8e410b830 2024-04-14T15:28:54.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.219+0000 7fd8e0ff9700 1 -- 172.21.15.90:0/380550825 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2325614254 0 0) 0x7fd8e41a8b30 con 0x7fd8e4103a20 2024-04-14T15:28:54.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.219+0000 7fd8e0ff9700 1 -- 172.21.15.90:0/380550825 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd8e41a7910 con 0x7fd8e4103a20 2024-04-14T15:28:54.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.219+0000 7fd8e0ff9700 1 -- 172.21.15.90:0/380550825 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 230925693 0 0) 0x7fd8cc0035b0 con 0x7fd8e410f340 2024-04-14T15:28:54.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.219+0000 7fd8e0ff9700 1 -- 172.21.15.90:0/380550825 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd8e41a8b30 con 0x7fd8e410f340 2024-04-14T15:28:54.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.219+0000 7fd8e0ff9700 1 -- 172.21.15.90:0/380550825 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fd8dc001c30 con 0x7fd8e410f340 2024-04-14T15:28:54.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.219+0000 7fd8e0ff9700 1 -- 172.21.15.90:0/380550825 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1960685170 0 0) 0x7fd8e41a9d70 con 0x7fd8e410b830 2024-04-14T15:28:54.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.219+0000 7fd8e0ff9700 1 -- 172.21.15.90:0/380550825 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd8cc0035b0 con 0x7fd8e410b830 2024-04-14T15:28:54.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.219+0000 7fd8e0ff9700 1 -- 172.21.15.90:0/380550825 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 862366584 0 0) 0x7fd8e41a7910 con 0x7fd8e4103a20 2024-04-14T15:28:54.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.219+0000 7fd8e0ff9700 1 -- 172.21.15.90:0/380550825 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd8e41a9d70 con 0x7fd8e4103a20 2024-04-14T15:28:54.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.219+0000 7fd8e0ff9700 1 -- 172.21.15.90:0/380550825 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fd8d8001830 con 0x7fd8e410b830 2024-04-14T15:28:54.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.219+0000 7fd8e0ff9700 1 -- 172.21.15.90:0/380550825 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fd8d4001840 con 0x7fd8e4103a20 2024-04-14T15:28:54.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.219+0000 7fd8e0ff9700 1 -- 172.21.15.90:0/380550825 <== mon.2 v1:172.21.15.90:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 270763574 0 0) 0x7fd8e41a8b30 con 0x7fd8e410f340 2024-04-14T15:28:54.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.219+0000 7fd8e0ff9700 1 -- 172.21.15.90:0/380550825 >> v1:172.21.15.92:6789/0 conn(0x7fd8e4103a20 legacy=0x7fd8e410b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:54.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.219+0000 7fd8e0ff9700 1 -- 172.21.15.90:0/380550825 >> v1:172.21.15.90:6789/0 conn(0x7fd8e410b830 legacy=0x7fd8e41a2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:54.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.219+0000 7fd8e0ff9700 1 -- 172.21.15.90:0/380550825 --> v1:172.21.15.90:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8e41aafb0 con 0x7fd8e410f340 2024-04-14T15:28:54.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.219+0000 7fd8e9c0b700 1 -- 172.21.15.90:0/380550825 --> v1:172.21.15.90:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd8e41a7b40 con 0x7fd8e410f340 2024-04-14T15:28:54.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.219+0000 7fd8e9c0b700 10 client.48311.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:28:54.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.219+0000 7fd8e9c0b700 1 -- 172.21.15.90:0/380550825 --> v1:172.21.15.90:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fd8e41a8000 con 0x7fd8e410f340 2024-04-14T15:28:54.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.219+0000 7fd8e9c0b700 1 librados: init done 2024-04-14T15:28:54.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.220+0000 7fd8e0ff9700 1 -- 172.21.15.90:0/380550825 <== mon.2 v1:172.21.15.90:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fd8dc003ea0 con 0x7fd8e410f340 2024-04-14T15:28:54.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.220+0000 7fd8e0ff9700 1 -- 172.21.15.90:0/380550825 <== mon.2 v1:172.21.15.90:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fd8dc004db0 con 0x7fd8e410f340 2024-04-14T15:28:54.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.220+0000 7fd8e9c0b700 10 client.48311.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-14T15:28:54.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.220+0000 7fd8e9c0b700 15 client.48311.objecter _session_command_op_assign -1 1 2024-04-14T15:28:54.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.220+0000 7fd8e9c0b700 20 client.48311.objecter _get_session osd=-1 returning homeless 2024-04-14T15:28:54.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.220+0000 7fd8e9c0b700 10 client.48311.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:28:54.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.220+0000 7fd8e9c0b700 1 -- 172.21.15.90:0/380550825 --> v1:172.21.15.90:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fd8d0002920 con 0x7fd8e410f340 2024-04-14T15:28:54.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.220+0000 7fd8e0ff9700 1 -- 172.21.15.90:0/380550825 <== mon.2 v1:172.21.15.90:6790/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fd8dc004d90 con 0x7fd8e410f340 2024-04-14T15:28:54.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.221+0000 7fd8e0ff9700 1 -- 172.21.15.90:0/380550825 <== mon.2 v1:172.21.15.90:6790/0 8 ==== osd_map(920..920 src has 467..920) ==== 7517+0+0 (unknown 2649999903 0 0) 0x7fd8dc089750 con 0x7fd8e410f340 2024-04-14T15:28:54.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.221+0000 7fd8e0ff9700 10 client.48311.objecter ms_dispatch 0x7fd8e4001550 osd_map(920..920 src has 467..920) 2024-04-14T15:28:54.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.221+0000 7fd8e0ff9700 3 client.48311.objecter handle_osd_map got epochs [920,920] > 0 2024-04-14T15:28:54.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.221+0000 7fd8e0ff9700 3 client.48311.objecter handle_osd_map decoding full epoch 920 2024-04-14T15:28:54.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.221+0000 7fd8e0ff9700 10 client.48311.objecter checking command 1 2024-04-14T15:28:54.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.221+0000 7fd8e0ff9700 20 client.48311.objecter _get_session s=0x7fd8cc07d4f0 osd=1 3 2024-04-14T15:28:54.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.221+0000 7fd8e0ff9700 20 client.48311.objecter put_session s=0x7fd8cc07d4f0 osd=1 3 2024-04-14T15:28:54.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.221+0000 7fd8e0ff9700 15 client.48311.objecter _session_command_op_remove -1 1 2024-04-14T15:28:54.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.221+0000 7fd8e0ff9700 20 client.48311.objecter _get_session s=0x7fd8cc07d4f0 osd=1 3 2024-04-14T15:28:54.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.221+0000 7fd8e0ff9700 20 client.48311.objecter get_session s=0x7fd8cc07d4f0 osd=1 3 2024-04-14T15:28:54.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.221+0000 7fd8e0ff9700 15 client.48311.objecter _session_command_op_assign 1 1 2024-04-14T15:28:54.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.221+0000 7fd8e0ff9700 20 client.48311.objecter put_session s=0x7fd8cc07d4f0 osd=1 4 2024-04-14T15:28:54.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.221+0000 7fd8e0ff9700 10 client.48311.objecter _send_command 1 2024-04-14T15:28:54.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.221+0000 7fd8e0ff9700 1 -- 172.21.15.90:0/380550825 --> v1:172.21.15.90:6805/4170361699 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd8cc0818b0 con 0x7fd8cc07dba0 2024-04-14T15:28:54.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.221+0000 7fd8e0ff9700 20 client.48311.objecter dump_active .. 0 homeless 2024-04-14T15:28:54.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.221+0000 7fd8e0ff9700 1 -- 172.21.15.90:0/380550825 <== mon.2 v1:172.21.15.90:6790/0 9 ==== mon_get_version_reply(handle=1 version=920) ==== 24+0+0 (unknown 622753966 0 0) 0x7fd8dc089b60 con 0x7fd8e410f340 2024-04-14T15:28:54.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.221+0000 7fd8e0ff9700 10 client.48311.objecter ms_handle_connect 0x7fd8cc0713f0 2024-04-14T15:28:54.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.221+0000 7fd8e0ff9700 10 client.48311.objecter ms_handle_connect 0x7fd8cc07dba0 2024-04-14T15:28:54.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.222+0000 7fd8e0ff9700 1 -- 172.21.15.90:0/380550825 <== osd.1 v1:172.21.15.90:6805/4170361699 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd8cc0818b0 con 0x7fd8cc07dba0 2024-04-14T15:28:54.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.222+0000 7fd8e0ff9700 10 client.48311.objecter ms_dispatch 0x7fd8e4001550 command_reply(tid 1: 0 ) 2024-04-14T15:28:54.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.222+0000 7fd8e0ff9700 10 client.48311.objecter _finish_command 1 = system:0 2024-04-14T15:28:54.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.222+0000 7fd8e0ff9700 20 client.48311.objecter put_session s=0x7fd8cc07d4f0 osd=1 4 2024-04-14T15:28:54.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.222+0000 7fd8e0ff9700 15 client.48311.objecter _session_command_op_remove 1 1 2024-04-14T15:28:54.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.238+0000 7fd8e9c0b700 10 client.48311.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-14T15:28:54.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.238+0000 7fd8e9c0b700 15 client.48311.objecter _session_command_op_assign -1 2 2024-04-14T15:28:54.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.238+0000 7fd8e9c0b700 20 client.48311.objecter _get_session s=0x7fd8cc07d4f0 osd=1 3 2024-04-14T15:28:54.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.238+0000 7fd8e9c0b700 20 client.48311.objecter put_session s=0x7fd8cc07d4f0 osd=1 3 2024-04-14T15:28:54.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.238+0000 7fd8e9c0b700 20 client.48311.objecter _get_session s=0x7fd8cc07d4f0 osd=1 3 2024-04-14T15:28:54.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.238+0000 7fd8e9c0b700 15 client.48311.objecter _session_command_op_remove -1 2 2024-04-14T15:28:54.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.238+0000 7fd8e9c0b700 20 client.48311.objecter get_session s=0x7fd8cc07d4f0 osd=1 3 2024-04-14T15:28:54.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.238+0000 7fd8e9c0b700 15 client.48311.objecter _session_command_op_assign 1 2 2024-04-14T15:28:54.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.238+0000 7fd8e9c0b700 20 client.48311.objecter put_session s=0x7fd8cc07d4f0 osd=1 4 2024-04-14T15:28:54.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.238+0000 7fd8e9c0b700 10 client.48311.objecter _send_command 2 2024-04-14T15:28:54.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.238+0000 7fd8e9c0b700 1 -- 172.21.15.90:0/380550825 --> v1:172.21.15.90:6805/4170361699 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd8d00011f0 con 0x7fd8cc07dba0 2024-04-14T15:28:54.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.238+0000 7fd8e0ff9700 1 -- 172.21.15.90:0/380550825 <== osd.1 v1:172.21.15.90:6805/4170361699 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 29282121) 0x7fd8d00011f0 con 0x7fd8cc07dba0 2024-04-14T15:28:54.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.238+0000 7fd8e0ff9700 10 client.48311.objecter ms_dispatch 0x7fd8e4001550 command_reply(tid 2: 0 ) 2024-04-14T15:28:54.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.238+0000 7fd8e0ff9700 10 client.48311.objecter _finish_command 2 = system:0 2024-04-14T15:28:54.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.238+0000 7fd8e0ff9700 20 client.48311.objecter put_session s=0x7fd8cc07d4f0 osd=1 4 2024-04-14T15:28:54.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.238+0000 7fd8e0ff9700 15 client.48311.objecter _session_command_op_remove 1 2 2024-04-14T15:28:54.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.239+0000 7fd8e9c0b700 10 librados: watch_flush enter 2024-04-14T15:28:54.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.239+0000 7fd8e9c0b700 10 librados: watch_flush exit 2024-04-14T15:28:54.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.239+0000 7fd8e9c0b700 10 client.48311.objecter close_session for osd.1 2024-04-14T15:28:54.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.239+0000 7fd8e9c0b700 1 -- 172.21.15.90:0/380550825 >> v1:172.21.15.90:6805/4170361699 conn(0x7fd8cc07dba0 legacy=0x7fd8cc080040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:54.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.239+0000 7fd8e9c0b700 20 client.48311.objecter put_session s=0x7fd8cc07d4f0 osd=1 1 2024-04-14T15:28:54.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.239+0000 7fd8e9c0b700 20 client.48311.objecter shutdown clearing up homeless session... 2024-04-14T15:28:54.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.239+0000 7fd8e9c0b700 10 client.48311.objecter successfully canceled tick 2024-04-14T15:28:54.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.239+0000 7fd8e9c0b700 1 -- 172.21.15.90:0/380550825 >> v1:172.21.15.90:6800/420935427 conn(0x7fd8cc0713f0 legacy=0x7fd8cc0738b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:54.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.239+0000 7fd8e9c0b700 1 -- 172.21.15.90:0/380550825 >> v1:172.21.15.90:6790/0 conn(0x7fd8e410f340 legacy=0x7fd8e41a5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:28:54.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.239+0000 7fd8e3fff700 1 -- 172.21.15.90:0/380550825 reap_dead start 2024-04-14T15:28:54.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.239+0000 7fd8e9c0b700 1 -- 172.21.15.90:0/380550825 shutdown_connections 2024-04-14T15:28:54.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.239+0000 7fd8e9c0b700 1 -- 172.21.15.90:0/380550825 >> 172.21.15.90:0/380550825 conn(0x7fd8e40fd290 msgr2=0x7fd8e40ff680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:28:54.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.239+0000 7fd8e9c0b700 1 -- 172.21.15.90:0/380550825 shutdown_connections 2024-04-14T15:28:54.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.239+0000 7fd8e9c0b700 1 -- 172.21.15.90:0/380550825 wait complete. 2024-04-14T15:28:54.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:28:54.239+0000 7fd8e9c0b700 1 librados: shutdown 2024-04-14T15:28:54.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:54 smithi090 ceph-mon[30399]: 2.18 scrub starts 2024-04-14T15:28:54.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:54 smithi090 ceph-mon[30399]: 2.18 scrub ok 2024-04-14T15:28:54.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:54 smithi090 ceph-mon[30399]: pgmap v1399: 174 pgs: 174 active+clean; 582 KiB data, 839 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:54.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:54 smithi090 ceph-mon[30399]: 4.5 scrub starts 2024-04-14T15:28:54.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:54 smithi090 ceph-mon[30399]: 4.5 scrub ok 2024-04-14T15:28:54.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:54 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:28:54.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:54 smithi090 ceph-mon[41670]: 2.18 scrub starts 2024-04-14T15:28:54.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:54 smithi090 ceph-mon[41670]: 2.18 scrub ok 2024-04-14T15:28:54.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:54 smithi090 ceph-mon[41670]: pgmap v1399: 174 pgs: 174 active+clean; 582 KiB data, 839 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:54.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:54 smithi090 ceph-mon[41670]: 4.5 scrub starts 2024-04-14T15:28:54.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:54 smithi090 ceph-mon[41670]: 4.5 scrub ok 2024-04-14T15:28:54.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:54 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:28:54.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:54 smithi092 ceph-mon[33966]: 2.18 scrub starts 2024-04-14T15:28:54.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:54 smithi092 ceph-mon[33966]: 2.18 scrub ok 2024-04-14T15:28:54.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:54 smithi092 ceph-mon[33966]: pgmap v1399: 174 pgs: 174 active+clean; 582 KiB data, 839 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:54.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:54 smithi092 ceph-mon[33966]: 4.5 scrub starts 2024-04-14T15:28:54.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:54 smithi092 ceph-mon[33966]: 4.5 scrub ok 2024-04-14T15:28:54.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:54 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:28:55.041 INFO:teuthology.orchestra.run.smithi090.stdout:77309411588 2024-04-14T15:28:55.042 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-14T15:28:55.170 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:28:55.170 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:28:55.173 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:28:55.177 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:28:55.563 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:55 smithi090 ceph-mon[41670]: 5.13 scrub starts 2024-04-14T15:28:55.564 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:55 smithi090 ceph-mon[41670]: 5.13 scrub ok 2024-04-14T15:28:55.564 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:55 smithi090 ceph-mon[41670]: 5.9 scrub starts 2024-04-14T15:28:55.564 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:55 smithi090 ceph-mon[41670]: 5.9 scrub ok 2024-04-14T15:28:55.564 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:55 smithi090 ceph-mon[30399]: 5.13 scrub starts 2024-04-14T15:28:55.564 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:55 smithi090 ceph-mon[30399]: 5.13 scrub ok 2024-04-14T15:28:55.564 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:55 smithi090 ceph-mon[30399]: 5.9 scrub starts 2024-04-14T15:28:55.564 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:55 smithi090 ceph-mon[30399]: 5.9 scrub ok 2024-04-14T15:28:55.596 INFO:teuthology.orchestra.run.smithi090.stdout:55834575114 2024-04-14T15:28:55.596 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-14T15:28:55.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:55 smithi092 ceph-mon[33966]: 5.13 scrub starts 2024-04-14T15:28:55.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:55 smithi092 ceph-mon[33966]: 5.13 scrub ok 2024-04-14T15:28:55.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:55 smithi092 ceph-mon[33966]: 5.9 scrub starts 2024-04-14T15:28:55.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:55 smithi092 ceph-mon[33966]: 5.9 scrub ok 2024-04-14T15:28:56.033 INFO:teuthology.orchestra.run.smithi090.stdout:133143986425 2024-04-14T15:28:56.034 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-14T15:28:56.122 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:28:55 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:28:56.501 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:56 smithi090 ceph-mon[30399]: 4.3 scrub starts 2024-04-14T15:28:56.501 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:56 smithi090 ceph-mon[30399]: 4.3 scrub ok 2024-04-14T15:28:56.501 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:56 smithi090 ceph-mon[30399]: pgmap v1400: 174 pgs: 174 active+clean; 582 KiB data, 839 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:28:56.501 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:56 smithi090 ceph-mon[30399]: 11.d scrub starts 2024-04-14T15:28:56.501 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:56 smithi090 ceph-mon[30399]: 11.d scrub ok 2024-04-14T15:28:56.502 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:56 smithi090 ceph-mon[30399]: 1.7 scrub starts 2024-04-14T15:28:56.502 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:56 smithi090 ceph-mon[30399]: 1.7 scrub ok 2024-04-14T15:28:56.502 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:56 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:28:56.502 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:56 smithi090 ceph-mon[41670]: 4.3 scrub starts 2024-04-14T15:28:56.502 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:56 smithi090 ceph-mon[41670]: 4.3 scrub ok 2024-04-14T15:28:56.502 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:56 smithi090 ceph-mon[41670]: pgmap v1400: 174 pgs: 174 active+clean; 582 KiB data, 839 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:28:56.502 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:56 smithi090 ceph-mon[41670]: 11.d scrub starts 2024-04-14T15:28:56.502 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:56 smithi090 ceph-mon[41670]: 11.d scrub ok 2024-04-14T15:28:56.502 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:56 smithi090 ceph-mon[41670]: 1.7 scrub starts 2024-04-14T15:28:56.502 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:56 smithi090 ceph-mon[41670]: 1.7 scrub ok 2024-04-14T15:28:56.502 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:56 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:28:56.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:56 smithi092 ceph-mon[33966]: 4.3 scrub starts 2024-04-14T15:28:56.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:56 smithi092 ceph-mon[33966]: 4.3 scrub ok 2024-04-14T15:28:56.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:56 smithi092 ceph-mon[33966]: pgmap v1400: 174 pgs: 174 active+clean; 582 KiB data, 839 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:28:56.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:56 smithi092 ceph-mon[33966]: 11.d scrub starts 2024-04-14T15:28:56.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:56 smithi092 ceph-mon[33966]: 11.d scrub ok 2024-04-14T15:28:56.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:56 smithi092 ceph-mon[33966]: 1.7 scrub starts 2024-04-14T15:28:56.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:56 smithi092 ceph-mon[33966]: 1.7 scrub ok 2024-04-14T15:28:56.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:56 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:28:57.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:57 smithi092 ceph-mon[33966]: 11.e scrub starts 2024-04-14T15:28:57.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:57 smithi092 ceph-mon[33966]: 11.e scrub ok 2024-04-14T15:28:57.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:57 smithi092 ceph-mon[33966]: 4.12 deep-scrub starts 2024-04-14T15:28:57.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:57 smithi092 ceph-mon[33966]: 4.12 deep-scrub ok 2024-04-14T15:28:57.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:57 smithi092 ceph-mon[33966]: 2.13 scrub starts 2024-04-14T15:28:57.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:57 smithi092 ceph-mon[33966]: 2.13 scrub ok 2024-04-14T15:28:57.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:57 smithi092 ceph-mon[33966]: 5.10 scrub starts 2024-04-14T15:28:57.623 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:57 smithi092 ceph-mon[33966]: 5.10 scrub ok 2024-04-14T15:28:57.691 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:57 smithi090 ceph-mon[41670]: 11.e scrub starts 2024-04-14T15:28:57.691 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:57 smithi090 ceph-mon[41670]: 11.e scrub ok 2024-04-14T15:28:57.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:57 smithi090 ceph-mon[41670]: 4.12 deep-scrub starts 2024-04-14T15:28:57.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:57 smithi090 ceph-mon[41670]: 4.12 deep-scrub ok 2024-04-14T15:28:57.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:57 smithi090 ceph-mon[41670]: 2.13 scrub starts 2024-04-14T15:28:57.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:57 smithi090 ceph-mon[41670]: 2.13 scrub ok 2024-04-14T15:28:57.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:57 smithi090 ceph-mon[41670]: 5.10 scrub starts 2024-04-14T15:28:57.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:57 smithi090 ceph-mon[41670]: 5.10 scrub ok 2024-04-14T15:28:57.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:57 smithi090 ceph-mon[30399]: 11.e scrub starts 2024-04-14T15:28:57.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:57 smithi090 ceph-mon[30399]: 11.e scrub ok 2024-04-14T15:28:57.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:57 smithi090 ceph-mon[30399]: 4.12 deep-scrub starts 2024-04-14T15:28:57.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:57 smithi090 ceph-mon[30399]: 4.12 deep-scrub ok 2024-04-14T15:28:57.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:57 smithi090 ceph-mon[30399]: 2.13 scrub starts 2024-04-14T15:28:57.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:57 smithi090 ceph-mon[30399]: 2.13 scrub ok 2024-04-14T15:28:57.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:57 smithi090 ceph-mon[30399]: 5.10 scrub starts 2024-04-14T15:28:57.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:57 smithi090 ceph-mon[30399]: 5.10 scrub ok 2024-04-14T15:28:57.986 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:28:58.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:58 smithi092 ceph-mon[33966]: 11.1b scrub starts 2024-04-14T15:28:58.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:58 smithi092 ceph-mon[33966]: 11.1b scrub ok 2024-04-14T15:28:58.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:58 smithi092 ceph-mon[33966]: 4.19 scrub starts 2024-04-14T15:28:58.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:58 smithi092 ceph-mon[33966]: 4.19 scrub ok 2024-04-14T15:28:58.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:58 smithi092 ceph-mon[33966]: 11.1a scrub starts 2024-04-14T15:28:58.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:58 smithi092 ceph-mon[33966]: 11.1a scrub ok 2024-04-14T15:28:58.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:58 smithi092 ceph-mon[33966]: pgmap v1401: 174 pgs: 174 active+clean; 582 KiB data, 839 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:58.641 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:58 smithi090 ceph-mon[41670]: 11.1b scrub starts 2024-04-14T15:28:58.641 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:58 smithi090 ceph-mon[41670]: 11.1b scrub ok 2024-04-14T15:28:58.641 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:58 smithi090 ceph-mon[41670]: 4.19 scrub starts 2024-04-14T15:28:58.641 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:58 smithi090 ceph-mon[41670]: 4.19 scrub ok 2024-04-14T15:28:58.641 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:58 smithi090 ceph-mon[41670]: 11.1a scrub starts 2024-04-14T15:28:58.641 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:58 smithi090 ceph-mon[41670]: 11.1a scrub ok 2024-04-14T15:28:58.641 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:58 smithi090 ceph-mon[41670]: pgmap v1401: 174 pgs: 174 active+clean; 582 KiB data, 839 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:58.642 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:58 smithi090 ceph-mon[30399]: 11.1b scrub starts 2024-04-14T15:28:58.642 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:58 smithi090 ceph-mon[30399]: 11.1b scrub ok 2024-04-14T15:28:58.642 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:58 smithi090 ceph-mon[30399]: 4.19 scrub starts 2024-04-14T15:28:58.642 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:58 smithi090 ceph-mon[30399]: 4.19 scrub ok 2024-04-14T15:28:58.642 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:58 smithi090 ceph-mon[30399]: 11.1a scrub starts 2024-04-14T15:28:58.642 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:58 smithi090 ceph-mon[30399]: 11.1a scrub ok 2024-04-14T15:28:58.642 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:58 smithi090 ceph-mon[30399]: pgmap v1401: 174 pgs: 174 active+clean; 582 KiB data, 839 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:59.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:59 smithi090 ceph-mon[30399]: 6.12 scrub starts 2024-04-14T15:28:59.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:59 smithi090 ceph-mon[30399]: 6.12 scrub ok 2024-04-14T15:28:59.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:59 smithi090 ceph-mon[30399]: 2.9 scrub starts 2024-04-14T15:28:59.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:28:59 smithi090 ceph-mon[30399]: 2.9 scrub ok 2024-04-14T15:28:59.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:59 smithi090 ceph-mon[41670]: 6.12 scrub starts 2024-04-14T15:28:59.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:59 smithi090 ceph-mon[41670]: 6.12 scrub ok 2024-04-14T15:28:59.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:59 smithi090 ceph-mon[41670]: 2.9 scrub starts 2024-04-14T15:28:59.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:28:59 smithi090 ceph-mon[41670]: 2.9 scrub ok 2024-04-14T15:28:59.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:59 smithi092 ceph-mon[33966]: 6.12 scrub starts 2024-04-14T15:28:59.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:59 smithi092 ceph-mon[33966]: 6.12 scrub ok 2024-04-14T15:28:59.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:59 smithi092 ceph-mon[33966]: 2.9 scrub starts 2024-04-14T15:28:59.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:28:59 smithi092 ceph-mon[33966]: 2.9 scrub ok 2024-04-14T15:29:00.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:00 smithi090 ceph-mon[41670]: 4.e scrub starts 2024-04-14T15:29:00.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:00 smithi090 ceph-mon[41670]: 4.e scrub ok 2024-04-14T15:29:00.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:00 smithi090 ceph-mon[41670]: 6.1f scrub starts 2024-04-14T15:29:00.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:00 smithi090 ceph-mon[41670]: 6.1f scrub ok 2024-04-14T15:29:00.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:00 smithi090 ceph-mon[41670]: pgmap v1402: 174 pgs: 174 active+clean; 582 KiB data, 843 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:29:00.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:00 smithi090 ceph-mon[41670]: 11.1c scrub starts 2024-04-14T15:29:00.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:00 smithi090 ceph-mon[41670]: 11.1c scrub ok 2024-04-14T15:29:00.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:00 smithi090 ceph-mon[30399]: 4.e scrub starts 2024-04-14T15:29:00.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:00 smithi090 ceph-mon[30399]: 4.e scrub ok 2024-04-14T15:29:00.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:00 smithi090 ceph-mon[30399]: 6.1f scrub starts 2024-04-14T15:29:00.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:00 smithi090 ceph-mon[30399]: 6.1f scrub ok 2024-04-14T15:29:00.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:00 smithi090 ceph-mon[30399]: pgmap v1402: 174 pgs: 174 active+clean; 582 KiB data, 843 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:29:00.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:00 smithi090 ceph-mon[30399]: 11.1c scrub starts 2024-04-14T15:29:00.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:00 smithi090 ceph-mon[30399]: 11.1c scrub ok 2024-04-14T15:29:00.757 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:29:00.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:00 smithi092 ceph-mon[33966]: 4.e scrub starts 2024-04-14T15:29:00.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:00 smithi092 ceph-mon[33966]: 4.e scrub ok 2024-04-14T15:29:00.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:00 smithi092 ceph-mon[33966]: 6.1f scrub starts 2024-04-14T15:29:00.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:00 smithi092 ceph-mon[33966]: 6.1f scrub ok 2024-04-14T15:29:00.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:00 smithi092 ceph-mon[33966]: pgmap v1402: 174 pgs: 174 active+clean; 582 KiB data, 843 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:29:00.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:00 smithi092 ceph-mon[33966]: 11.1c scrub starts 2024-04-14T15:29:00.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:00 smithi092 ceph-mon[33966]: 11.1c scrub ok 2024-04-14T15:29:01.600 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:01 smithi090 ceph-mon[30399]: 1.0 scrub starts 2024-04-14T15:29:01.600 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:01 smithi090 ceph-mon[30399]: 1.0 scrub ok 2024-04-14T15:29:01.600 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:01 smithi090 ceph-mon[41670]: 1.0 scrub starts 2024-04-14T15:29:01.601 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:01 smithi090 ceph-mon[41670]: 1.0 scrub ok 2024-04-14T15:29:01.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:01 smithi092 ceph-mon[33966]: 1.0 scrub starts 2024-04-14T15:29:01.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:01 smithi092 ceph-mon[33966]: 1.0 scrub ok 2024-04-14T15:29:02.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.057+0000 7fa2e219d700 1 -- 172.21.15.90:0/2899001111 >> v1:172.21.15.90:6790/0 conn(0x7fa2dc1086c0 legacy=0x7fa2dc108aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:29:02.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.057+0000 7fa2e219d700 1 -- 172.21.15.90:0/2899001111 shutdown_connections 2024-04-14T15:29:02.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.057+0000 7fa2e219d700 1 -- 172.21.15.90:0/2899001111 >> 172.21.15.90:0/2899001111 conn(0x7fa2dc0fe1b0 msgr2=0x7fa2dc1005d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:29:02.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.057+0000 7fa2e219d700 1 -- 172.21.15.90:0/2899001111 shutdown_connections 2024-04-14T15:29:02.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.057+0000 7fa2e219d700 1 -- 172.21.15.90:0/2899001111 wait complete. 2024-04-14T15:29:02.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.057+0000 7fa2e219d700 1 librados: starting msgr at 2024-04-14T15:29:02.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.057+0000 7fa2e219d700 1 librados: starting objecter 2024-04-14T15:29:02.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.058+0000 7fa2e219d700 1 Processor -- start 2024-04-14T15:29:02.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.058+0000 7fa2e219d700 1 -- start start 2024-04-14T15:29:02.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.058+0000 7fa2e219d700 1 librados: setting wanted keys 2024-04-14T15:29:02.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.058+0000 7fa2e219d700 1 librados: calling monclient init 2024-04-14T15:29:02.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.058+0000 7fa2e219d700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa2dc1a7790 con 0x7fa2dc1086c0 2024-04-14T15:29:02.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.058+0000 7fa2e219d700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa2dc1a89b0 con 0x7fa2dc10f140 2024-04-14T15:29:02.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.058+0000 7fa2e219d700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa2dc1a9bf0 con 0x7fa2dc10b570 2024-04-14T15:29:02.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.058+0000 7fa2e099a700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7fa2dc10b570 0x7fa2dc1a25f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:38422/0 (socket says 172.21.15.90:38422) 2024-04-14T15:29:02.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.058+0000 7fa2e099a700 1 -- 172.21.15.90:0/2072023356 learned_addr learned my addr 172.21.15.90:0/2072023356 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:29:02.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.058+0000 7fa2e119b700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fa2dc1086c0 0x7fa2dc10e880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:40132/0 (socket says 172.21.15.90:40132) 2024-04-14T15:29:02.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.058+0000 7fa2d27fc700 10 client.?.objecter ms_handle_connect 0x7fa2dc1086c0 2024-04-14T15:29:02.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.058+0000 7fa2d27fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:29:02.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.058+0000 7fa2d27fc700 10 client.?.objecter ms_handle_connect 0x7fa2dc10b570 2024-04-14T15:29:02.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.058+0000 7fa2d27fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:29:02.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.058+0000 7fa2d27fc700 10 client.?.objecter ms_handle_connect 0x7fa2dc10f140 2024-04-14T15:29:02.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.058+0000 7fa2d27fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:29:02.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.058+0000 7fa2d27fc700 1 -- 172.21.15.90:0/2072023356 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1126873873 0 0) 0x7fa2dc1a9bf0 con 0x7fa2dc10b570 2024-04-14T15:29:02.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.058+0000 7fa2d27fc700 1 -- 172.21.15.90:0/2072023356 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa2c8003480 con 0x7fa2dc10b570 2024-04-14T15:29:02.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.058+0000 7fa2d27fc700 1 -- 172.21.15.90:0/2072023356 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2751566834 0 0) 0x7fa2dc1a7790 con 0x7fa2dc1086c0 2024-04-14T15:29:02.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.058+0000 7fa2d27fc700 1 -- 172.21.15.90:0/2072023356 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa2dc1a9bf0 con 0x7fa2dc1086c0 2024-04-14T15:29:02.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.058+0000 7fa2d27fc700 1 -- 172.21.15.90:0/2072023356 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2045660789 0 0) 0x7fa2dc1a89b0 con 0x7fa2dc10f140 2024-04-14T15:29:02.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.058+0000 7fa2d27fc700 1 -- 172.21.15.90:0/2072023356 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa2dc1a7790 con 0x7fa2dc10f140 2024-04-14T15:29:02.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.058+0000 7fa2d27fc700 1 -- 172.21.15.90:0/2072023356 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4284122448 0 0) 0x7fa2dc1a9bf0 con 0x7fa2dc1086c0 2024-04-14T15:29:02.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.058+0000 7fa2d27fc700 1 -- 172.21.15.90:0/2072023356 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa2dc1a89b0 con 0x7fa2dc1086c0 2024-04-14T15:29:02.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.058+0000 7fa2d27fc700 1 -- 172.21.15.90:0/2072023356 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fa2d8004550 con 0x7fa2dc1086c0 2024-04-14T15:29:02.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.058+0000 7fa2d27fc700 1 -- 172.21.15.90:0/2072023356 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4014141259 0 0) 0x7fa2c8003480 con 0x7fa2dc10b570 2024-04-14T15:29:02.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.058+0000 7fa2d27fc700 1 -- 172.21.15.90:0/2072023356 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa2dc1a9bf0 con 0x7fa2dc10b570 2024-04-14T15:29:02.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.058+0000 7fa2d27fc700 1 -- 172.21.15.90:0/2072023356 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fa2cc001900 con 0x7fa2dc10b570 2024-04-14T15:29:02.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.058+0000 7fa2d27fc700 1 -- 172.21.15.90:0/2072023356 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1888011773 0 0) 0x7fa2dc1a7790 con 0x7fa2dc10f140 2024-04-14T15:29:02.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.058+0000 7fa2d27fc700 1 -- 172.21.15.90:0/2072023356 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa2c8003480 con 0x7fa2dc10f140 2024-04-14T15:29:02.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.058+0000 7fa2d27fc700 1 -- 172.21.15.90:0/2072023356 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fa2d40013f0 con 0x7fa2dc10f140 2024-04-14T15:29:02.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.059+0000 7fa2d27fc700 1 -- 172.21.15.90:0/2072023356 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2861675680 0 0) 0x7fa2dc1a89b0 con 0x7fa2dc1086c0 2024-04-14T15:29:02.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.059+0000 7fa2d27fc700 1 -- 172.21.15.90:0/2072023356 >> v1:172.21.15.90:6790/0 conn(0x7fa2dc10b570 legacy=0x7fa2dc1a25f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:29:02.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.059+0000 7fa2d27fc700 1 -- 172.21.15.90:0/2072023356 >> v1:172.21.15.92:6789/0 conn(0x7fa2dc10f140 legacy=0x7fa2dc1a5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:29:02.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.059+0000 7fa2d27fc700 1 -- 172.21.15.90:0/2072023356 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2dc1aae30 con 0x7fa2dc1086c0 2024-04-14T15:29:02.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.059+0000 7fa2e219d700 1 -- 172.21.15.90:0/2072023356 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa2dc1a9e20 con 0x7fa2dc1086c0 2024-04-14T15:29:02.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.059+0000 7fa2e219d700 10 client.38367.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:29:02.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.059+0000 7fa2e219d700 1 -- 172.21.15.90:0/2072023356 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa2dc1aa2e0 con 0x7fa2dc1086c0 2024-04-14T15:29:02.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.059+0000 7fa2e219d700 1 librados: init done 2024-04-14T15:29:02.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.059+0000 7fa2d27fc700 1 -- 172.21.15.90:0/2072023356 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fa2d8001ab0 con 0x7fa2dc1086c0 2024-04-14T15:29:02.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.059+0000 7fa2d27fc700 1 -- 172.21.15.90:0/2072023356 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fa2d8004d80 con 0x7fa2dc1086c0 2024-04-14T15:29:02.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.059+0000 7fa2e219d700 1 -- 172.21.15.90:0/2072023356 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2c0005120 con 0x7fa2dc1086c0 2024-04-14T15:29:02.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.060+0000 7fa2d27fc700 1 -- 172.21.15.90:0/2072023356 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fa2d8005000 con 0x7fa2dc1086c0 2024-04-14T15:29:02.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.061+0000 7fa2d27fc700 1 -- 172.21.15.90:0/2072023356 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(920..920 src has 467..920) ==== 7517+0+0 (unknown 2649999903 0 0) 0x7fa2d808a7e0 con 0x7fa2dc1086c0 2024-04-14T15:29:02.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.061+0000 7fa2d27fc700 10 client.38367.objecter ms_dispatch 0x7fa2dc001550 osd_map(920..920 src has 467..920) 2024-04-14T15:29:02.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.061+0000 7fa2d27fc700 3 client.38367.objecter handle_osd_map got epochs [920,920] > 0 2024-04-14T15:29:02.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.061+0000 7fa2d27fc700 3 client.38367.objecter handle_osd_map decoding full epoch 920 2024-04-14T15:29:02.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.061+0000 7fa2d27fc700 20 client.38367.objecter dump_active .. 0 homeless 2024-04-14T15:29:02.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.061+0000 7fa2d27fc700 10 client.38367.objecter ms_handle_connect 0x7fa2c8071360 2024-04-14T15:29:02.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.065+0000 7fa2d27fc700 1 -- 172.21.15.90:0/2072023356 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fa2d8057480 con 0x7fa2dc1086c0 2024-04-14T15:29:02.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.270+0000 7fa2e219d700 1 -- 172.21.15.90:0/2072023356 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fa2c0005410 con 0x7fa2dc1086c0 2024-04-14T15:29:02.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.270+0000 7fa2d27fc700 1 -- 172.21.15.90:0/2072023356 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (unknown 574334944 0 1982002732) 0x7fa2d805b130 con 0x7fa2dc1086c0 2024-04-14T15:29:02.271 INFO:teuthology.orchestra.run.smithi090.stdout:34359738640 2024-04-14T15:29:02.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.272+0000 7fa2e219d700 10 librados: watch_flush enter 2024-04-14T15:29:02.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.272+0000 7fa2e219d700 10 librados: watch_flush exit 2024-04-14T15:29:02.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.272+0000 7fa2e219d700 20 client.38367.objecter shutdown clearing up homeless session... 2024-04-14T15:29:02.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.272+0000 7fa2e219d700 10 client.38367.objecter successfully canceled tick 2024-04-14T15:29:02.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.272+0000 7fa2e219d700 1 -- 172.21.15.90:0/2072023356 >> v1:172.21.15.90:6800/420935427 conn(0x7fa2c8071360 legacy=0x7fa2c8073820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:29:02.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.272+0000 7fa2e219d700 1 -- 172.21.15.90:0/2072023356 >> v1:172.21.15.90:6789/0 conn(0x7fa2dc1086c0 legacy=0x7fa2dc10e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:29:02.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.272+0000 7fa2e219d700 1 -- 172.21.15.90:0/2072023356 shutdown_connections 2024-04-14T15:29:02.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.272+0000 7fa2e219d700 1 -- 172.21.15.90:0/2072023356 >> 172.21.15.90:0/2072023356 conn(0x7fa2dc0fe1b0 msgr2=0x7fa2dc111e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:29:02.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.272+0000 7fa2e219d700 1 -- 172.21.15.90:0/2072023356 shutdown_connections 2024-04-14T15:29:02.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.272+0000 7fa2e219d700 1 -- 172.21.15.90:0/2072023356 wait complete. 2024-04-14T15:29:02.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.273+0000 7fa2e219d700 1 librados: shutdown 2024-04-14T15:29:02.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.354+0000 7f85fa623700 1 -- 172.21.15.90:0/3619628597 >> v1:172.21.15.90:6789/0 conn(0x7f85f410e1f0 legacy=0x7f85f41106b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:29:02.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.354+0000 7f85fa623700 1 -- 172.21.15.90:0/3619628597 shutdown_connections 2024-04-14T15:29:02.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.354+0000 7f85fa623700 1 -- 172.21.15.90:0/3619628597 >> 172.21.15.90:0/3619628597 conn(0x7f85f40fd280 msgr2=0x7f85f40ff6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:29:02.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.355+0000 7f85fa623700 1 -- 172.21.15.90:0/3619628597 shutdown_connections 2024-04-14T15:29:02.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.355+0000 7f85fa623700 1 -- 172.21.15.90:0/3619628597 wait complete. 2024-04-14T15:29:02.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.355+0000 7f85fa623700 1 librados: starting msgr at 2024-04-14T15:29:02.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.355+0000 7f85fa623700 1 librados: starting objecter 2024-04-14T15:29:02.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.355+0000 7f85fa623700 1 Processor -- start 2024-04-14T15:29:02.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.355+0000 7f85fa623700 1 -- start start 2024-04-14T15:29:02.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.355+0000 7f85fa623700 1 librados: setting wanted keys 2024-04-14T15:29:02.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.355+0000 7f85fa623700 1 librados: calling monclient init 2024-04-14T15:29:02.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.355+0000 7f85fa623700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f85f41a9d10 con 0x7f85f4107770 2024-04-14T15:29:02.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.355+0000 7f85fa623700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f85f41aaf30 con 0x7f85f410a620 2024-04-14T15:29:02.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.355+0000 7f85fa623700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f85f41ac170 con 0x7f85f410e1f0 2024-04-14T15:29:02.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.356+0000 7f85f3fff700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f85f4107770 0x7f85f410da30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:40156/0 (socket says 172.21.15.90:40156) 2024-04-14T15:29:02.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.356+0000 7f85f3fff700 1 -- 172.21.15.90:0/1993855349 learned_addr learned my addr 172.21.15.90:0/1993855349 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:29:02.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.356+0000 7f85f17fa700 10 client.?.objecter ms_handle_connect 0x7f85f4107770 2024-04-14T15:29:02.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.356+0000 7f85f17fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:29:02.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.356+0000 7f85f17fa700 10 client.?.objecter ms_handle_connect 0x7f85f410e1f0 2024-04-14T15:29:02.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.356+0000 7f85f17fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:29:02.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.356+0000 7f85f17fa700 10 client.?.objecter ms_handle_connect 0x7f85f410a620 2024-04-14T15:29:02.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.356+0000 7f85f17fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:29:02.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.356+0000 7f85f17fa700 1 -- 172.21.15.90:0/1993855349 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1682763301 0 0) 0x7f85f41a9d10 con 0x7f85f4107770 2024-04-14T15:29:02.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.356+0000 7f85f17fa700 1 -- 172.21.15.90:0/1993855349 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f85dc0031f0 con 0x7f85f4107770 2024-04-14T15:29:02.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.356+0000 7f85f17fa700 1 -- 172.21.15.90:0/1993855349 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 94706183 0 0) 0x7f85f41ac170 con 0x7f85f410e1f0 2024-04-14T15:29:02.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.356+0000 7f85f17fa700 1 -- 172.21.15.90:0/1993855349 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f85f41a9d10 con 0x7f85f410e1f0 2024-04-14T15:29:02.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.356+0000 7f85f17fa700 1 -- 172.21.15.90:0/1993855349 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 267275127 0 0) 0x7f85f41aaf30 con 0x7f85f410a620 2024-04-14T15:29:02.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.356+0000 7f85f17fa700 1 -- 172.21.15.90:0/1993855349 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f85f41ac170 con 0x7f85f410a620 2024-04-14T15:29:02.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.356+0000 7f85f17fa700 1 -- 172.21.15.90:0/1993855349 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4058409634 0 0) 0x7f85dc0031f0 con 0x7f85f4107770 2024-04-14T15:29:02.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.356+0000 7f85f17fa700 1 -- 172.21.15.90:0/1993855349 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f85f41aaf30 con 0x7f85f4107770 2024-04-14T15:29:02.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.356+0000 7f85f17fa700 1 -- 172.21.15.90:0/1993855349 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f85e40012d0 con 0x7f85f4107770 2024-04-14T15:29:02.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.356+0000 7f85f17fa700 1 -- 172.21.15.90:0/1993855349 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3237861052 0 0) 0x7f85f41a9d10 con 0x7f85f410e1f0 2024-04-14T15:29:02.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.356+0000 7f85f17fa700 1 -- 172.21.15.90:0/1993855349 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f85dc0031f0 con 0x7f85f410e1f0 2024-04-14T15:29:02.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.356+0000 7f85f17fa700 1 -- 172.21.15.90:0/1993855349 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f85ec001bc0 con 0x7f85f410e1f0 2024-04-14T15:29:02.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.356+0000 7f85f17fa700 1 -- 172.21.15.90:0/1993855349 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3646053160 0 0) 0x7f85f41ac170 con 0x7f85f410a620 2024-04-14T15:29:02.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.356+0000 7f85f17fa700 1 -- 172.21.15.90:0/1993855349 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f85f41a9d10 con 0x7f85f410a620 2024-04-14T15:29:02.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.356+0000 7f85f17fa700 1 -- 172.21.15.90:0/1993855349 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f85e8001870 con 0x7f85f410a620 2024-04-14T15:29:02.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.356+0000 7f85f17fa700 1 -- 172.21.15.90:0/1993855349 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 170838159 0 0) 0x7f85f41aaf30 con 0x7f85f4107770 2024-04-14T15:29:02.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.356+0000 7f85f17fa700 1 -- 172.21.15.90:0/1993855349 >> v1:172.21.15.90:6790/0 conn(0x7f85f410e1f0 legacy=0x7f85f41a83d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:29:02.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.356+0000 7f85f17fa700 1 -- 172.21.15.90:0/1993855349 >> v1:172.21.15.92:6789/0 conn(0x7f85f410a620 legacy=0x7f85f41a4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:29:02.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.356+0000 7f85f17fa700 1 -- 172.21.15.90:0/1993855349 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85f41ad3b0 con 0x7f85f4107770 2024-04-14T15:29:02.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.356+0000 7f85fa623700 1 -- 172.21.15.90:0/1993855349 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f85f41ac3a0 con 0x7f85f4107770 2024-04-14T15:29:02.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.356+0000 7f85fa623700 10 client.38373.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:29:02.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.356+0000 7f85fa623700 1 -- 172.21.15.90:0/1993855349 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f85f41ac860 con 0x7f85f4107770 2024-04-14T15:29:02.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.357+0000 7f85fa623700 1 librados: init done 2024-04-14T15:29:02.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.357+0000 7f85f17fa700 1 -- 172.21.15.90:0/1993855349 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f85e40015b0 con 0x7f85f4107770 2024-04-14T15:29:02.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.357+0000 7f85f17fa700 1 -- 172.21.15.90:0/1993855349 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f85e4005c60 con 0x7f85f4107770 2024-04-14T15:29:02.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.358+0000 7f85fa623700 1 -- 172.21.15.90:0/1993855349 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85e0005120 con 0x7f85f4107770 2024-04-14T15:29:02.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.358+0000 7f85f17fa700 1 -- 172.21.15.90:0/1993855349 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f85e401c8d0 con 0x7f85f4107770 2024-04-14T15:29:02.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.358+0000 7f85f17fa700 1 -- 172.21.15.90:0/1993855349 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(920..920 src has 467..920) ==== 7517+0+0 (unknown 2649999903 0 0) 0x7f85e408a6e0 con 0x7f85f4107770 2024-04-14T15:29:02.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.358+0000 7f85f17fa700 10 client.38373.objecter ms_dispatch 0x7f85f4001550 osd_map(920..920 src has 467..920) 2024-04-14T15:29:02.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.358+0000 7f85f17fa700 3 client.38373.objecter handle_osd_map got epochs [920,920] > 0 2024-04-14T15:29:02.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.358+0000 7f85f17fa700 3 client.38373.objecter handle_osd_map decoding full epoch 920 2024-04-14T15:29:02.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.359+0000 7f85f17fa700 20 client.38373.objecter dump_active .. 0 homeless 2024-04-14T15:29:02.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.359+0000 7f85f17fa700 10 client.38373.objecter ms_handle_connect 0x7f85dc071330 2024-04-14T15:29:02.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.363+0000 7f85f17fa700 1 -- 172.21.15.90:0/1993855349 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f85e4057380 con 0x7f85f4107770 2024-04-14T15:29:02.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.566+0000 7f85fa623700 1 -- 172.21.15.90:0/1993855349 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f85e0005410 con 0x7f85f4107770 2024-04-14T15:29:02.567 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:02 smithi090 ceph-mon[41670]: 5.1c scrub starts 2024-04-14T15:29:02.567 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:02 smithi090 ceph-mon[41670]: 5.1c scrub ok 2024-04-14T15:29:02.567 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:02 smithi090 ceph-mon[41670]: 5.7 deep-scrub starts 2024-04-14T15:29:02.567 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:02 smithi090 ceph-mon[41670]: 5.7 deep-scrub ok 2024-04-14T15:29:02.567 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:02 smithi090 ceph-mon[41670]: 11.10 scrub starts 2024-04-14T15:29:02.567 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:02 smithi090 ceph-mon[41670]: 11.10 scrub ok 2024-04-14T15:29:02.567 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:02 smithi090 ceph-mon[41670]: 2.1c deep-scrub starts 2024-04-14T15:29:02.568 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:02 smithi090 ceph-mon[41670]: 2.1c deep-scrub ok 2024-04-14T15:29:02.568 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:02 smithi090 ceph-mon[41670]: pgmap v1403: 174 pgs: 174 active+clean; 582 KiB data, 843 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:29:02.568 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:02 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2072023356' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-14T15:29:02.568 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:02 smithi090 ceph-mon[30399]: 5.1c scrub starts 2024-04-14T15:29:02.568 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:02 smithi090 ceph-mon[30399]: 5.1c scrub ok 2024-04-14T15:29:02.568 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:02 smithi090 ceph-mon[30399]: 5.7 deep-scrub starts 2024-04-14T15:29:02.568 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:02 smithi090 ceph-mon[30399]: 5.7 deep-scrub ok 2024-04-14T15:29:02.568 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:02 smithi090 ceph-mon[30399]: 11.10 scrub starts 2024-04-14T15:29:02.568 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:02 smithi090 ceph-mon[30399]: 11.10 scrub ok 2024-04-14T15:29:02.568 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:02 smithi090 ceph-mon[30399]: 2.1c deep-scrub starts 2024-04-14T15:29:02.568 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:02 smithi090 ceph-mon[30399]: 2.1c deep-scrub ok 2024-04-14T15:29:02.569 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:02 smithi090 ceph-mon[30399]: pgmap v1403: 174 pgs: 174 active+clean; 582 KiB data, 843 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:29:02.569 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:02 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2072023356' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-14T15:29:02.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.567+0000 7f85f17fa700 1 -- 172.21.15.90:0/1993855349 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (unknown 2131975755 0 887201929) 0x7f85e405b030 con 0x7f85f4107770 2024-04-14T15:29:02.569 INFO:teuthology.orchestra.run.smithi090.stdout:176093659384 2024-04-14T15:29:02.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.569+0000 7f85fa623700 10 librados: watch_flush enter 2024-04-14T15:29:02.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.569+0000 7f85fa623700 10 librados: watch_flush exit 2024-04-14T15:29:02.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.569+0000 7f85fa623700 20 client.38373.objecter shutdown clearing up homeless session... 2024-04-14T15:29:02.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.569+0000 7f85fa623700 10 client.38373.objecter successfully canceled tick 2024-04-14T15:29:02.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.569+0000 7f85fa623700 1 -- 172.21.15.90:0/1993855349 >> v1:172.21.15.90:6800/420935427 conn(0x7f85dc071330 legacy=0x7f85dc0737f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:29:02.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.569+0000 7f85fa623700 1 -- 172.21.15.90:0/1993855349 >> v1:172.21.15.90:6789/0 conn(0x7f85f4107770 legacy=0x7f85f410da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:29:02.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.569+0000 7f85fa623700 1 -- 172.21.15.90:0/1993855349 shutdown_connections 2024-04-14T15:29:02.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.569+0000 7f85fa623700 1 -- 172.21.15.90:0/1993855349 >> 172.21.15.90:0/1993855349 conn(0x7f85f40fd280 msgr2=0x7f85f41003e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:29:02.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.569+0000 7f85fa623700 1 -- 172.21.15.90:0/1993855349 shutdown_connections 2024-04-14T15:29:02.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.569+0000 7f85fa623700 1 -- 172.21.15.90:0/1993855349 wait complete. 2024-04-14T15:29:02.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:02.569+0000 7f85fa623700 1 librados: shutdown 2024-04-14T15:29:02.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:02 smithi092 ceph-mon[33966]: 5.1c scrub starts 2024-04-14T15:29:02.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:02 smithi092 ceph-mon[33966]: 5.1c scrub ok 2024-04-14T15:29:02.873 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:02 smithi092 ceph-mon[33966]: 5.7 deep-scrub starts 2024-04-14T15:29:02.873 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:02 smithi092 ceph-mon[33966]: 5.7 deep-scrub ok 2024-04-14T15:29:02.873 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:02 smithi092 ceph-mon[33966]: 11.10 scrub starts 2024-04-14T15:29:02.873 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:02 smithi092 ceph-mon[33966]: 11.10 scrub ok 2024-04-14T15:29:02.873 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:02 smithi092 ceph-mon[33966]: 2.1c deep-scrub starts 2024-04-14T15:29:02.873 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:02 smithi092 ceph-mon[33966]: 2.1c deep-scrub ok 2024-04-14T15:29:02.874 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:02 smithi092 ceph-mon[33966]: pgmap v1403: 174 pgs: 174 active+clean; 582 KiB data, 843 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:29:02.874 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:02 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2072023356' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-14T15:29:03.384 INFO:tasks.cephadm.ceph_manager.ceph:need seq 176093659381 got 176093659384 for osd.5 2024-04-14T15:29:03.384 DEBUG:teuthology.parallel:result is None 2024-04-14T15:29:03.610 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:03 smithi090 ceph-mon[30399]: 4.1a scrub starts 2024-04-14T15:29:03.610 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:03 smithi090 ceph-mon[30399]: 4.1a scrub ok 2024-04-14T15:29:03.610 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:03 smithi090 ceph-mon[30399]: 5.f scrub starts 2024-04-14T15:29:03.611 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:03 smithi090 ceph-mon[30399]: 5.f scrub ok 2024-04-14T15:29:03.611 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:03 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1993855349' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-14T15:29:03.611 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:03 smithi090 ceph-mon[41670]: 4.1a scrub starts 2024-04-14T15:29:03.611 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:03 smithi090 ceph-mon[41670]: 4.1a scrub ok 2024-04-14T15:29:03.611 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:03 smithi090 ceph-mon[41670]: 5.f scrub starts 2024-04-14T15:29:03.611 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:03 smithi090 ceph-mon[41670]: 5.f scrub ok 2024-04-14T15:29:03.611 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:03 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1993855349' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-14T15:29:03.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:03 smithi092 ceph-mon[33966]: 4.1a scrub starts 2024-04-14T15:29:03.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:03 smithi092 ceph-mon[33966]: 4.1a scrub ok 2024-04-14T15:29:03.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:03 smithi092 ceph-mon[33966]: 5.f scrub starts 2024-04-14T15:29:03.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:03 smithi092 ceph-mon[33966]: 5.f scrub ok 2024-04-14T15:29:03.622 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:03 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1993855349' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-14T15:29:04.028 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:29:03 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:29:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:29:04.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.031+0000 7f0e0d7e7700 1 -- 172.21.15.90:0/2757574320 >> v1:172.21.15.90:6789/0 conn(0x7f0e0810a740 legacy=0x7f0e0810cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:29:04.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.031+0000 7f0e0d7e7700 1 -- 172.21.15.90:0/2757574320 shutdown_connections 2024-04-14T15:29:04.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.031+0000 7f0e0d7e7700 1 -- 172.21.15.90:0/2757574320 >> 172.21.15.90:0/2757574320 conn(0x7f0e080fd290 msgr2=0x7f0e080ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:29:04.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.031+0000 7f0e0d7e7700 1 -- 172.21.15.90:0/2757574320 shutdown_connections 2024-04-14T15:29:04.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.031+0000 7f0e0d7e7700 1 -- 172.21.15.90:0/2757574320 wait complete. 2024-04-14T15:29:04.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.032+0000 7f0e0d7e7700 1 librados: starting msgr at 2024-04-14T15:29:04.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.032+0000 7f0e0d7e7700 1 librados: starting objecter 2024-04-14T15:29:04.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.032+0000 7f0e0d7e7700 1 Processor -- start 2024-04-14T15:29:04.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.032+0000 7f0e0d7e7700 1 -- start start 2024-04-14T15:29:04.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.032+0000 7f0e0d7e7700 1 librados: setting wanted keys 2024-04-14T15:29:04.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.032+0000 7f0e0d7e7700 1 librados: calling monclient init 2024-04-14T15:29:04.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.032+0000 7f0e0d7e7700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f0e08110390 con 0x7f0e0810e310 2024-04-14T15:29:04.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.032+0000 7f0e0d7e7700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f0e081105a0 con 0x7f0e0810a740 2024-04-14T15:29:04.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.032+0000 7f0e0d7e7700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f0e081107d0 con 0x7f0e08107890 2024-04-14T15:29:04.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.032+0000 7f0e06ffd700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f0e08107890 0x7f0e0810e890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:38454/0 (socket says 172.21.15.90:38454) 2024-04-14T15:29:04.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.032+0000 7f0e077fe700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f0e0810e310 0x7f0e081a8bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:40174/0 (socket says 172.21.15.90:40174) 2024-04-14T15:29:04.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.032+0000 7f0e06ffd700 1 -- 172.21.15.90:0/3360170866 learned_addr learned my addr 172.21.15.90:0/3360170866 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:29:04.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.032+0000 7f0deffff700 10 client.?.objecter ms_handle_connect 0x7f0e0810e310 2024-04-14T15:29:04.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.032+0000 7f0deffff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:29:04.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.032+0000 7f0deffff700 10 client.?.objecter ms_handle_connect 0x7f0e08107890 2024-04-14T15:29:04.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.032+0000 7f0deffff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:29:04.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.032+0000 7f0deffff700 10 client.?.objecter ms_handle_connect 0x7f0e0810a740 2024-04-14T15:29:04.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.032+0000 7f0deffff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:29:04.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.032+0000 7f0deffff700 1 -- 172.21.15.90:0/3360170866 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 890479040 0 0) 0x7f0e08110390 con 0x7f0e0810e310 2024-04-14T15:29:04.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.032+0000 7f0deffff700 1 -- 172.21.15.90:0/3360170866 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0df00035b0 con 0x7f0e0810e310 2024-04-14T15:29:04.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.033+0000 7f0deffff700 1 -- 172.21.15.90:0/3360170866 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3474074659 0 0) 0x7f0e081107d0 con 0x7f0e08107890 2024-04-14T15:29:04.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.033+0000 7f0deffff700 1 -- 172.21.15.90:0/3360170866 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0e08110390 con 0x7f0e08107890 2024-04-14T15:29:04.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.033+0000 7f0deffff700 1 -- 172.21.15.90:0/3360170866 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2013050951 0 0) 0x7f0e081105a0 con 0x7f0e0810a740 2024-04-14T15:29:04.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.033+0000 7f0deffff700 1 -- 172.21.15.90:0/3360170866 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0e081107d0 con 0x7f0e0810a740 2024-04-14T15:29:04.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.033+0000 7f0deffff700 1 -- 172.21.15.90:0/3360170866 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1935406299 0 0) 0x7f0df00035b0 con 0x7f0e0810e310 2024-04-14T15:29:04.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.033+0000 7f0deffff700 1 -- 172.21.15.90:0/3360170866 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0e081105a0 con 0x7f0e0810e310 2024-04-14T15:29:04.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.033+0000 7f0deffff700 1 -- 172.21.15.90:0/3360170866 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f0e00001bb0 con 0x7f0e0810e310 2024-04-14T15:29:04.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.033+0000 7f0deffff700 1 -- 172.21.15.90:0/3360170866 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1249171558 0 0) 0x7f0e08110390 con 0x7f0e08107890 2024-04-14T15:29:04.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.033+0000 7f0deffff700 1 -- 172.21.15.90:0/3360170866 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0df00035b0 con 0x7f0e08107890 2024-04-14T15:29:04.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.033+0000 7f0deffff700 1 -- 172.21.15.90:0/3360170866 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f0dfc0018b0 con 0x7f0e08107890 2024-04-14T15:29:04.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.033+0000 7f0deffff700 1 -- 172.21.15.90:0/3360170866 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1760060235 0 0) 0x7f0e081107d0 con 0x7f0e0810a740 2024-04-14T15:29:04.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.033+0000 7f0deffff700 1 -- 172.21.15.90:0/3360170866 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0e08110390 con 0x7f0e0810a740 2024-04-14T15:29:04.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.033+0000 7f0deffff700 1 -- 172.21.15.90:0/3360170866 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f0df8003910 con 0x7f0e0810a740 2024-04-14T15:29:04.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.033+0000 7f0deffff700 1 -- 172.21.15.90:0/3360170866 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2611314821 0 0) 0x7f0e081105a0 con 0x7f0e0810e310 2024-04-14T15:29:04.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.033+0000 7f0deffff700 1 -- 172.21.15.90:0/3360170866 >> v1:172.21.15.90:6790/0 conn(0x7f0e08107890 legacy=0x7f0e0810e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:29:04.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.033+0000 7f0deffff700 1 -- 172.21.15.90:0/3360170866 >> v1:172.21.15.92:6789/0 conn(0x7f0e0810a740 legacy=0x7f0e0810efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:29:04.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.033+0000 7f0deffff700 1 -- 172.21.15.90:0/3360170866 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e081ad380 con 0x7f0e0810e310 2024-04-14T15:29:04.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.033+0000 7f0e0d7e7700 1 -- 172.21.15.90:0/3360170866 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0e081aa370 con 0x7f0e0810e310 2024-04-14T15:29:04.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.033+0000 7f0e0d7e7700 10 client.38379.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:29:04.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.033+0000 7f0e0d7e7700 1 -- 172.21.15.90:0/3360170866 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f0e081aa830 con 0x7f0e0810e310 2024-04-14T15:29:04.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.033+0000 7f0e0d7e7700 1 librados: init done 2024-04-14T15:29:04.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.033+0000 7f0deffff700 1 -- 172.21.15.90:0/3360170866 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f0e00003ec0 con 0x7f0e0810e310 2024-04-14T15:29:04.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.033+0000 7f0deffff700 1 -- 172.21.15.90:0/3360170866 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f0e00006070 con 0x7f0e0810e310 2024-04-14T15:29:04.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.034+0000 7f0e0d7e7700 1 -- 172.21.15.90:0/3360170866 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0df4005120 con 0x7f0e0810e310 2024-04-14T15:29:04.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.035+0000 7f0deffff700 1 -- 172.21.15.90:0/3360170866 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f0e00003a20 con 0x7f0e0810e310 2024-04-14T15:29:04.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.035+0000 7f0deffff700 1 -- 172.21.15.90:0/3360170866 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(920..920 src has 467..920) ==== 7517+0+0 (unknown 2649999903 0 0) 0x7f0e0008aa00 con 0x7f0e0810e310 2024-04-14T15:29:04.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.035+0000 7f0deffff700 10 client.38379.objecter ms_dispatch 0x7f0e08001550 osd_map(920..920 src has 467..920) 2024-04-14T15:29:04.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.035+0000 7f0deffff700 3 client.38379.objecter handle_osd_map got epochs [920,920] > 0 2024-04-14T15:29:04.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.035+0000 7f0deffff700 3 client.38379.objecter handle_osd_map decoding full epoch 920 2024-04-14T15:29:04.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.035+0000 7f0deffff700 20 client.38379.objecter dump_active .. 0 homeless 2024-04-14T15:29:04.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.035+0000 7f0deffff700 10 client.38379.objecter ms_handle_connect 0x7f0df0071330 2024-04-14T15:29:04.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.039+0000 7f0deffff700 1 -- 172.21.15.90:0/3360170866 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f0e00006070 con 0x7f0e0810e310 2024-04-14T15:29:04.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.246+0000 7f0e0d7e7700 1 -- 172.21.15.90:0/3360170866 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f0df4005410 con 0x7f0e0810e310 2024-04-14T15:29:04.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.246+0000 7f0deffff700 1 -- 172.21.15.90:0/3360170866 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (unknown 383520633 0 547035881) 0x7f0e00057700 con 0x7f0e0810e310 2024-04-14T15:29:04.247 INFO:teuthology.orchestra.run.smithi090.stdout:107374182657 2024-04-14T15:29:04.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.248+0000 7f0e0d7e7700 10 librados: watch_flush enter 2024-04-14T15:29:04.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.248+0000 7f0e0d7e7700 10 librados: watch_flush exit 2024-04-14T15:29:04.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.248+0000 7f0e0d7e7700 20 client.38379.objecter shutdown clearing up homeless session... 2024-04-14T15:29:04.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.248+0000 7f0e0d7e7700 10 client.38379.objecter successfully canceled tick 2024-04-14T15:29:04.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.248+0000 7f0e0d7e7700 1 -- 172.21.15.90:0/3360170866 >> v1:172.21.15.90:6800/420935427 conn(0x7f0df0071330 legacy=0x7f0df00737f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:29:04.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.248+0000 7f0e0d7e7700 1 -- 172.21.15.90:0/3360170866 >> v1:172.21.15.90:6789/0 conn(0x7f0e0810e310 legacy=0x7f0e081a8bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:29:04.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.249+0000 7f0e0d7e7700 1 -- 172.21.15.90:0/3360170866 shutdown_connections 2024-04-14T15:29:04.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.249+0000 7f0e0d7e7700 1 -- 172.21.15.90:0/3360170866 >> 172.21.15.90:0/3360170866 conn(0x7f0e080fd290 msgr2=0x7f0e081006f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:29:04.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.249+0000 7f0e0d7e7700 1 -- 172.21.15.90:0/3360170866 shutdown_connections 2024-04-14T15:29:04.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.249+0000 7f0e0d7e7700 1 -- 172.21.15.90:0/3360170866 wait complete. 2024-04-14T15:29:04.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.249+0000 7f0e0d7e7700 1 librados: shutdown 2024-04-14T15:29:04.691 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:04 smithi090 ceph-mon[30399]: 5.15 scrub starts 2024-04-14T15:29:04.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:04 smithi090 ceph-mon[30399]: 5.15 scrub ok 2024-04-14T15:29:04.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:04 smithi090 ceph-mon[30399]: 2.e deep-scrub starts 2024-04-14T15:29:04.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:04 smithi090 ceph-mon[30399]: 2.e deep-scrub ok 2024-04-14T15:29:04.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:04 smithi090 ceph-mon[30399]: pgmap v1404: 174 pgs: 174 active+clean; 582 KiB data, 843 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:29:04.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:04 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3360170866' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-14T15:29:04.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:04 smithi090 ceph-mon[41670]: 5.15 scrub starts 2024-04-14T15:29:04.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:04 smithi090 ceph-mon[41670]: 5.15 scrub ok 2024-04-14T15:29:04.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:04 smithi090 ceph-mon[41670]: 2.e deep-scrub starts 2024-04-14T15:29:04.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:04 smithi090 ceph-mon[41670]: 2.e deep-scrub ok 2024-04-14T15:29:04.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:04 smithi090 ceph-mon[41670]: pgmap v1404: 174 pgs: 174 active+clean; 582 KiB data, 843 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:29:04.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:04 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3360170866' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-14T15:29:04.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:04 smithi092 ceph-mon[33966]: 5.15 scrub starts 2024-04-14T15:29:04.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:04 smithi092 ceph-mon[33966]: 5.15 scrub ok 2024-04-14T15:29:04.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:04 smithi092 ceph-mon[33966]: 2.e deep-scrub starts 2024-04-14T15:29:04.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:04 smithi092 ceph-mon[33966]: 2.e deep-scrub ok 2024-04-14T15:29:04.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:04 smithi092 ceph-mon[33966]: pgmap v1404: 174 pgs: 174 active+clean; 582 KiB data, 843 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:29:04.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:04 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3360170866' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-14T15:29:05.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.999+0000 7fe0e6a2d700 1 -- 172.21.15.90:0/3964890036 >> v1:172.21.15.90:6789/0 conn(0x7fe0e010a630 legacy=0x7fe0e010ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:29:05.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.999+0000 7fe0e6a2d700 1 -- 172.21.15.90:0/3964890036 shutdown_connections 2024-04-14T15:29:05.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.999+0000 7fe0e6a2d700 1 -- 172.21.15.90:0/3964890036 >> 172.21.15.90:0/3964890036 conn(0x7fe0e00fd270 msgr2=0x7fe0e00ff690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:29:05.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.999+0000 7fe0e6a2d700 1 -- 172.21.15.90:0/3964890036 shutdown_connections 2024-04-14T15:29:05.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:04.999+0000 7fe0e6a2d700 1 -- 172.21.15.90:0/3964890036 wait complete. 2024-04-14T15:29:05.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:05.000+0000 7fe0e6a2d700 1 librados: starting msgr at 2024-04-14T15:29:05.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:05.000+0000 7fe0e6a2d700 1 librados: starting objecter 2024-04-14T15:29:05.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:05.000+0000 7fe0e6a2d700 1 Processor -- start 2024-04-14T15:29:05.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:05.000+0000 7fe0e6a2d700 1 -- start start 2024-04-14T15:29:05.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:05.000+0000 7fe0e6a2d700 1 librados: setting wanted keys 2024-04-14T15:29:05.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:05.000+0000 7fe0e6a2d700 1 librados: calling monclient init 2024-04-14T15:29:05.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:05.000+0000 7fe0e6a2d700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe0e01a9bc0 con 0x7fe0e010a630 2024-04-14T15:29:05.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:05.000+0000 7fe0e6a2d700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe0e01aade0 con 0x7fe0e010e200 2024-04-14T15:29:05.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:05.000+0000 7fe0e6a2d700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe0e01ac020 con 0x7fe0e0107780 2024-04-14T15:29:05.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:05.001+0000 7fe0dffff700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7fe0e0107780 0x7fe0e010dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:38466/0 (socket says 172.21.15.90:38466) 2024-04-14T15:29:05.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:05.001+0000 7fe0dffff700 1 -- 172.21.15.90:0/3882646894 learned_addr learned my addr 172.21.15.90:0/3882646894 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:29:05.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:05.001+0000 7fe0dd7fa700 10 client.?.objecter ms_handle_connect 0x7fe0e0107780 2024-04-14T15:29:05.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:05.001+0000 7fe0dd7fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:29:05.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:05.001+0000 7fe0dd7fa700 10 client.?.objecter ms_handle_connect 0x7fe0e010e200 2024-04-14T15:29:05.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:05.001+0000 7fe0dd7fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:29:05.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:05.001+0000 7fe0dd7fa700 10 client.?.objecter ms_handle_connect 0x7fe0e010a630 2024-04-14T15:29:05.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:05.001+0000 7fe0dd7fa700 10 client.?.objecter resend_mon_ops 2024-04-14T15:29:05.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:05.001+0000 7fe0dd7fa700 1 -- 172.21.15.90:0/3882646894 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2707881752 0 0) 0x7fe0e01ac020 con 0x7fe0e0107780 2024-04-14T15:29:05.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:05.001+0000 7fe0dd7fa700 1 -- 172.21.15.90:0/3882646894 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe0c80035b0 con 0x7fe0e0107780 2024-04-14T15:29:05.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:05.001+0000 7fe0dd7fa700 1 -- 172.21.15.90:0/3882646894 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1325400360 0 0) 0x7fe0e01aade0 con 0x7fe0e010e200 2024-04-14T15:29:05.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:05.001+0000 7fe0dd7fa700 1 -- 172.21.15.90:0/3882646894 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe0e01ac020 con 0x7fe0e010e200 2024-04-14T15:29:05.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:05.001+0000 7fe0dd7fa700 1 -- 172.21.15.90:0/3882646894 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1695315768 0 0) 0x7fe0e01a9bc0 con 0x7fe0e010a630 2024-04-14T15:29:05.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:05.001+0000 7fe0dd7fa700 1 -- 172.21.15.90:0/3882646894 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe0e01aade0 con 0x7fe0e010a630 2024-04-14T15:29:05.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:05.001+0000 7fe0dd7fa700 1 -- 172.21.15.90:0/3882646894 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 245525494 0 0) 0x7fe0c80035b0 con 0x7fe0e0107780 2024-04-14T15:29:05.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:05.001+0000 7fe0dd7fa700 1 -- 172.21.15.90:0/3882646894 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe0e01a9bc0 con 0x7fe0e0107780 2024-04-14T15:29:05.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:05.001+0000 7fe0dd7fa700 1 -- 172.21.15.90:0/3882646894 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fe0d00018f0 con 0x7fe0e0107780 2024-04-14T15:29:05.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:05.001+0000 7fe0dd7fa700 1 -- 172.21.15.90:0/3882646894 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 865731834 0 0) 0x7fe0e01aade0 con 0x7fe0e010a630 2024-04-14T15:29:05.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:05.001+0000 7fe0dd7fa700 1 -- 172.21.15.90:0/3882646894 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe0c80035b0 con 0x7fe0e010a630 2024-04-14T15:29:05.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:05.001+0000 7fe0dd7fa700 1 -- 172.21.15.90:0/3882646894 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fe0d4004620 con 0x7fe0e010a630 2024-04-14T15:29:05.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:05.001+0000 7fe0dd7fa700 1 -- 172.21.15.90:0/3882646894 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 800644924 0 0) 0x7fe0e01ac020 con 0x7fe0e010e200 2024-04-14T15:29:05.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:05.001+0000 7fe0dd7fa700 1 -- 172.21.15.90:0/3882646894 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe0e01aade0 con 0x7fe0e010e200 2024-04-14T15:29:05.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:05.001+0000 7fe0dd7fa700 1 -- 172.21.15.90:0/3882646894 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fe0d8001b30 con 0x7fe0e010e200 2024-04-14T15:29:05.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:05.001+0000 7fe0dd7fa700 1 -- 172.21.15.90:0/3882646894 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4041848493 0 0) 0x7fe0c80035b0 con 0x7fe0e010a630 2024-04-14T15:29:05.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:05.001+0000 7fe0dd7fa700 1 -- 172.21.15.90:0/3882646894 >> v1:172.21.15.90:6790/0 conn(0x7fe0e0107780 legacy=0x7fe0e010dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:29:05.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:05.001+0000 7fe0dd7fa700 1 -- 172.21.15.90:0/3882646894 >> v1:172.21.15.92:6789/0 conn(0x7fe0e010e200 legacy=0x7fe0e01a8280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:29:05.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:05.001+0000 7fe0dd7fa700 1 -- 172.21.15.90:0/3882646894 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0e01ad260 con 0x7fe0e010a630 2024-04-14T15:29:05.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:05.001+0000 7fe0e6a2d700 1 -- 172.21.15.90:0/3882646894 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe0e01ab010 con 0x7fe0e010a630 2024-04-14T15:29:05.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:05.002+0000 7fe0e6a2d700 10 client.38385.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:29:05.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:05.002+0000 7fe0e6a2d700 1 -- 172.21.15.90:0/3882646894 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe0e01ab4d0 con 0x7fe0e010a630 2024-04-14T15:29:05.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:05.002+0000 7fe0e6a2d700 1 librados: init done 2024-04-14T15:29:05.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:05.002+0000 7fe0dd7fa700 1 -- 172.21.15.90:0/3882646894 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fe0d4001b70 con 0x7fe0e010a630 2024-04-14T15:29:05.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:05.002+0000 7fe0dd7fa700 1 -- 172.21.15.90:0/3882646894 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fe0d4004c90 con 0x7fe0e010a630 2024-04-14T15:29:05.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:05.003+0000 7fe0e6a2d700 1 -- 172.21.15.90:0/3882646894 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0cc005120 con 0x7fe0e010a630 2024-04-14T15:29:05.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:05.003+0000 7fe0dd7fa700 1 -- 172.21.15.90:0/3882646894 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fe0d401b6a0 con 0x7fe0e010a630 2024-04-14T15:29:05.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:05.003+0000 7fe0dd7fa700 1 -- 172.21.15.90:0/3882646894 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(920..920 src has 467..920) ==== 7517+0+0 (unknown 2649999903 0 0) 0x7fe0d4089570 con 0x7fe0e010a630 2024-04-14T15:29:05.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:05.003+0000 7fe0dd7fa700 10 client.38385.objecter ms_dispatch 0x7fe0e0001550 osd_map(920..920 src has 467..920) 2024-04-14T15:29:05.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:05.003+0000 7fe0dd7fa700 3 client.38385.objecter handle_osd_map got epochs [920,920] > 0 2024-04-14T15:29:05.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:05.003+0000 7fe0dd7fa700 3 client.38385.objecter handle_osd_map decoding full epoch 920 2024-04-14T15:29:05.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:05.004+0000 7fe0dd7fa700 20 client.38385.objecter dump_active .. 0 homeless 2024-04-14T15:29:05.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:05.004+0000 7fe0dd7fa700 10 client.38385.objecter ms_handle_connect 0x7fe0c8071350 2024-04-14T15:29:05.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:05.008+0000 7fe0dd7fa700 1 -- 172.21.15.90:0/3882646894 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fe0d4056250 con 0x7fe0e010a630 2024-04-14T15:29:05.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:05.215+0000 7fe0e6a2d700 1 -- 172.21.15.90:0/3882646894 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fe0cc005410 con 0x7fe0e010a630 2024-04-14T15:29:05.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:05.215+0000 7fe0dd7fa700 1 -- 172.21.15.90:0/3882646894 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (unknown 1274345170 0 2121097571) 0x7fe0d4059f00 con 0x7fe0e010a630 2024-04-14T15:29:05.216 INFO:teuthology.orchestra.run.smithi090.stdout:210453397745 2024-04-14T15:29:05.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:05.217+0000 7fe0e6a2d700 10 librados: watch_flush enter 2024-04-14T15:29:05.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:05.217+0000 7fe0e6a2d700 10 librados: watch_flush exit 2024-04-14T15:29:05.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:05.217+0000 7fe0e6a2d700 20 client.38385.objecter shutdown clearing up homeless session... 2024-04-14T15:29:05.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:05.217+0000 7fe0e6a2d700 10 client.38385.objecter successfully canceled tick 2024-04-14T15:29:05.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:05.217+0000 7fe0e6a2d700 1 -- 172.21.15.90:0/3882646894 >> v1:172.21.15.90:6800/420935427 conn(0x7fe0c8071350 legacy=0x7fe0c8073810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:29:05.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:05.217+0000 7fe0e6a2d700 1 -- 172.21.15.90:0/3882646894 >> v1:172.21.15.90:6789/0 conn(0x7fe0e010a630 legacy=0x7fe0e01a4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:29:05.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:05.218+0000 7fe0e6a2d700 1 -- 172.21.15.90:0/3882646894 shutdown_connections 2024-04-14T15:29:05.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:05.218+0000 7fe0e6a2d700 1 -- 172.21.15.90:0/3882646894 >> 172.21.15.90:0/3882646894 conn(0x7fe0e00fd270 msgr2=0x7fe0e006b720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:29:05.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:05.218+0000 7fe0e6a2d700 1 -- 172.21.15.90:0/3882646894 shutdown_connections 2024-04-14T15:29:05.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:05.218+0000 7fe0e6a2d700 1 -- 172.21.15.90:0/3882646894 wait complete. 2024-04-14T15:29:05.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:05.218+0000 7fe0e6a2d700 1 librados: shutdown 2024-04-14T15:29:05.451 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738637 got 34359738640 for osd.0 2024-04-14T15:29:05.451 DEBUG:teuthology.parallel:result is None 2024-04-14T15:29:05.533 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:05 smithi090 ceph-mon[41670]: 6.18 deep-scrub starts 2024-04-14T15:29:05.533 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:05 smithi090 ceph-mon[41670]: 6.18 deep-scrub ok 2024-04-14T15:29:05.533 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:05 smithi090 ceph-mon[41670]: 2.6 deep-scrub starts 2024-04-14T15:29:05.533 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:05 smithi090 ceph-mon[41670]: 2.6 deep-scrub ok 2024-04-14T15:29:05.533 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:05 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3882646894' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-14T15:29:05.534 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:05 smithi090 ceph-mon[30399]: 6.18 deep-scrub starts 2024-04-14T15:29:05.534 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:05 smithi090 ceph-mon[30399]: 6.18 deep-scrub ok 2024-04-14T15:29:05.534 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:05 smithi090 ceph-mon[30399]: 2.6 deep-scrub starts 2024-04-14T15:29:05.534 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:05 smithi090 ceph-mon[30399]: 2.6 deep-scrub ok 2024-04-14T15:29:05.534 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:05 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3882646894' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-14T15:29:05.688 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:05 smithi092 ceph-mon[33966]: 6.18 deep-scrub starts 2024-04-14T15:29:05.688 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:05 smithi092 ceph-mon[33966]: 6.18 deep-scrub ok 2024-04-14T15:29:05.688 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:05 smithi092 ceph-mon[33966]: 2.6 deep-scrub starts 2024-04-14T15:29:05.688 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:05 smithi092 ceph-mon[33966]: 2.6 deep-scrub ok 2024-04-14T15:29:05.688 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:05 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3882646894' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-14T15:29:06.122 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:29:05 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:29:06.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:06 smithi090 ceph-mon[41670]: 11.1f scrub starts 2024-04-14T15:29:06.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:06 smithi090 ceph-mon[41670]: 11.1f scrub ok 2024-04-14T15:29:06.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:06 smithi090 ceph-mon[41670]: pgmap v1405: 174 pgs: 174 active+clean; 582 KiB data, 843 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:29:06.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:06 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:29:06.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:06 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:29:06.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:06 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:29:06.442 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:06 smithi090 ceph-mon[30399]: 11.1f scrub starts 2024-04-14T15:29:06.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:06 smithi090 ceph-mon[30399]: 11.1f scrub ok 2024-04-14T15:29:06.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:06 smithi090 ceph-mon[30399]: pgmap v1405: 174 pgs: 174 active+clean; 582 KiB data, 843 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:29:06.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:06 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:29:06.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:06 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:29:06.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:06 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:29:06.526 INFO:tasks.cephadm.ceph_manager.ceph:need seq 210453397742 got 210453397745 for osd.6 2024-04-14T15:29:06.526 DEBUG:teuthology.parallel:result is None 2024-04-14T15:29:06.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:06 smithi092 ceph-mon[33966]: 11.1f scrub starts 2024-04-14T15:29:06.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:06 smithi092 ceph-mon[33966]: 11.1f scrub ok 2024-04-14T15:29:06.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:06 smithi092 ceph-mon[33966]: pgmap v1405: 174 pgs: 174 active+clean; 582 KiB data, 843 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:29:06.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:06 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:29:06.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:06 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:29:06.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:06 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:29:07.419 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:07 smithi090 ceph-mon[41670]: 6.16 scrub starts 2024-04-14T15:29:07.419 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:07 smithi090 ceph-mon[41670]: 6.16 scrub ok 2024-04-14T15:29:07.419 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:07 smithi090 ceph-mon[41670]: 1.2 scrub starts 2024-04-14T15:29:07.419 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:07 smithi090 ceph-mon[41670]: 1.2 scrub ok 2024-04-14T15:29:07.420 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:07 smithi090 ceph-mon[41670]: 5.1f scrub starts 2024-04-14T15:29:07.420 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:07 smithi090 ceph-mon[41670]: 5.1f scrub ok 2024-04-14T15:29:07.420 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:07 smithi090 ceph-mon[41670]: osdmap e921: 8 total, 8 up, 8 in 2024-04-14T15:29:07.420 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:07 smithi090 ceph-mon[41670]: 11.b scrub starts 2024-04-14T15:29:07.420 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:07 smithi090 ceph-mon[41670]: 11.b scrub ok 2024-04-14T15:29:07.420 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:07 smithi090 ceph-mon[30399]: 6.16 scrub starts 2024-04-14T15:29:07.420 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:07 smithi090 ceph-mon[30399]: 6.16 scrub ok 2024-04-14T15:29:07.420 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:07 smithi090 ceph-mon[30399]: 1.2 scrub starts 2024-04-14T15:29:07.420 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:07 smithi090 ceph-mon[30399]: 1.2 scrub ok 2024-04-14T15:29:07.420 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:07 smithi090 ceph-mon[30399]: 5.1f scrub starts 2024-04-14T15:29:07.420 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:07 smithi090 ceph-mon[30399]: 5.1f scrub ok 2024-04-14T15:29:07.420 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:07 smithi090 ceph-mon[30399]: osdmap e921: 8 total, 8 up, 8 in 2024-04-14T15:29:07.421 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:07 smithi090 ceph-mon[30399]: 11.b scrub starts 2024-04-14T15:29:07.421 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:07 smithi090 ceph-mon[30399]: 11.b scrub ok 2024-04-14T15:29:07.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.585+0000 7fcb4ffd8700 1 -- 172.21.15.90:0/1733890702 >> v1:172.21.15.90:6789/0 conn(0x7fcb4810e1e0 legacy=0x7fcb481106a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:29:07.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.585+0000 7fcb4ffd8700 1 -- 172.21.15.90:0/1733890702 shutdown_connections 2024-04-14T15:29:07.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.585+0000 7fcb4ffd8700 1 -- 172.21.15.90:0/1733890702 >> 172.21.15.90:0/1733890702 conn(0x7fcb480fd290 msgr2=0x7fcb480ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:29:07.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.585+0000 7fcb4ffd8700 1 -- 172.21.15.90:0/1733890702 shutdown_connections 2024-04-14T15:29:07.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.585+0000 7fcb4ffd8700 1 -- 172.21.15.90:0/1733890702 wait complete. 2024-04-14T15:29:07.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.586+0000 7fcb4ffd8700 1 librados: starting msgr at 2024-04-14T15:29:07.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.586+0000 7fcb4ffd8700 1 librados: starting objecter 2024-04-14T15:29:07.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.586+0000 7fcb4ffd8700 1 Processor -- start 2024-04-14T15:29:07.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.586+0000 7fcb4ffd8700 1 -- start start 2024-04-14T15:29:07.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.586+0000 7fcb4ffd8700 1 librados: setting wanted keys 2024-04-14T15:29:07.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.586+0000 7fcb4ffd8700 1 librados: calling monclient init 2024-04-14T15:29:07.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.586+0000 7fcb4ffd8700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fcb4810de90 con 0x7fcb48107760 2024-04-14T15:29:07.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.586+0000 7fcb4ffd8700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fcb481a7df0 con 0x7fcb4810e1e0 2024-04-14T15:29:07.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.586+0000 7fcb4ffd8700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fcb481a8fd0 con 0x7fcb4810a610 2024-04-14T15:29:07.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.586+0000 7fcb4d573700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7fcb4810a610 0x7fcb4810b3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:38488/0 (socket says 172.21.15.90:38488) 2024-04-14T15:29:07.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.586+0000 7fcb4d573700 1 -- 172.21.15.90:0/2676720458 learned_addr learned my addr 172.21.15.90:0/2676720458 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:29:07.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.586+0000 7fcb3effd700 10 client.?.objecter ms_handle_connect 0x7fcb48107760 2024-04-14T15:29:07.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.586+0000 7fcb3effd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:29:07.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.586+0000 7fcb3effd700 10 client.?.objecter ms_handle_connect 0x7fcb4810a610 2024-04-14T15:29:07.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.586+0000 7fcb3effd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:29:07.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.586+0000 7fcb3effd700 1 -- 172.21.15.90:0/2676720458 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1073065041 0 0) 0x7fcb481a8fd0 con 0x7fcb4810a610 2024-04-14T15:29:07.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.586+0000 7fcb3effd700 1 -- 172.21.15.90:0/2676720458 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcb340031f0 con 0x7fcb4810a610 2024-04-14T15:29:07.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.586+0000 7fcb3effd700 1 -- 172.21.15.90:0/2676720458 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3334940301 0 0) 0x7fcb4810de90 con 0x7fcb48107760 2024-04-14T15:29:07.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.586+0000 7fcb3effd700 1 -- 172.21.15.90:0/2676720458 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcb481a8fd0 con 0x7fcb48107760 2024-04-14T15:29:07.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.587+0000 7fcb3effd700 10 client.?.objecter ms_handle_connect 0x7fcb4810e1e0 2024-04-14T15:29:07.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.587+0000 7fcb3effd700 10 client.?.objecter resend_mon_ops 2024-04-14T15:29:07.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.587+0000 7fcb3effd700 1 -- 172.21.15.90:0/2676720458 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2304687233 0 0) 0x7fcb481a7df0 con 0x7fcb4810e1e0 2024-04-14T15:29:07.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.587+0000 7fcb3effd700 1 -- 172.21.15.90:0/2676720458 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcb4810de90 con 0x7fcb4810e1e0 2024-04-14T15:29:07.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.587+0000 7fcb3effd700 1 -- 172.21.15.90:0/2676720458 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 527464691 0 0) 0x7fcb340031f0 con 0x7fcb4810a610 2024-04-14T15:29:07.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.587+0000 7fcb3effd700 1 -- 172.21.15.90:0/2676720458 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcb481a7df0 con 0x7fcb4810a610 2024-04-14T15:29:07.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.587+0000 7fcb3effd700 1 -- 172.21.15.90:0/2676720458 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fcb440018b0 con 0x7fcb4810a610 2024-04-14T15:29:07.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.587+0000 7fcb3effd700 1 -- 172.21.15.90:0/2676720458 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3139931166 0 0) 0x7fcb481a8fd0 con 0x7fcb48107760 2024-04-14T15:29:07.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.587+0000 7fcb3effd700 1 -- 172.21.15.90:0/2676720458 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcb340031f0 con 0x7fcb48107760 2024-04-14T15:29:07.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.587+0000 7fcb3effd700 1 -- 172.21.15.90:0/2676720458 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fcb38001830 con 0x7fcb48107760 2024-04-14T15:29:07.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.587+0000 7fcb3effd700 1 -- 172.21.15.90:0/2676720458 <== mon.2 v1:172.21.15.90:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1313140479 0 0) 0x7fcb481a7df0 con 0x7fcb4810a610 2024-04-14T15:29:07.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.587+0000 7fcb3effd700 1 -- 172.21.15.90:0/2676720458 >> v1:172.21.15.92:6789/0 conn(0x7fcb4810e1e0 legacy=0x7fcb481a56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:29:07.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.587+0000 7fcb3effd700 1 -- 172.21.15.90:0/2676720458 >> v1:172.21.15.90:6789/0 conn(0x7fcb48107760 legacy=0x7fcb4810ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:29:07.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.587+0000 7fcb3effd700 1 -- 172.21.15.90:0/2676720458 --> v1:172.21.15.90:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb481aa1b0 con 0x7fcb4810a610 2024-04-14T15:29:07.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.587+0000 7fcb4ffd8700 1 -- 172.21.15.90:0/2676720458 --> v1:172.21.15.90:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fcb481a9200 con 0x7fcb4810a610 2024-04-14T15:29:07.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.587+0000 7fcb4ffd8700 10 client.48341.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:29:07.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.587+0000 7fcb4ffd8700 1 -- 172.21.15.90:0/2676720458 --> v1:172.21.15.90:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fcb481a96c0 con 0x7fcb4810a610 2024-04-14T15:29:07.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.587+0000 7fcb4ffd8700 1 librados: init done 2024-04-14T15:29:07.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.587+0000 7fcb3effd700 1 -- 172.21.15.90:0/2676720458 <== mon.2 v1:172.21.15.90:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fcb44003b40 con 0x7fcb4810a610 2024-04-14T15:29:07.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.587+0000 7fcb3effd700 1 -- 172.21.15.90:0/2676720458 <== mon.2 v1:172.21.15.90:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fcb44005d80 con 0x7fcb4810a610 2024-04-14T15:29:07.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.588+0000 7fcb4ffd8700 1 -- 172.21.15.90:0/2676720458 --> v1:172.21.15.90:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb2c005150 con 0x7fcb4810a610 2024-04-14T15:29:07.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.589+0000 7fcb3effd700 1 -- 172.21.15.90:0/2676720458 <== mon.2 v1:172.21.15.90:6790/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fcb440036a0 con 0x7fcb4810a610 2024-04-14T15:29:07.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.589+0000 7fcb3effd700 1 -- 172.21.15.90:0/2676720458 <== mon.2 v1:172.21.15.90:6790/0 8 ==== osd_map(921..921 src has 467..921) ==== 7517+0+0 (unknown 3843985862 0 0) 0x7fcb4408a850 con 0x7fcb4810a610 2024-04-14T15:29:07.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.589+0000 7fcb3effd700 10 client.48341.objecter ms_dispatch 0x7fcb48001550 osd_map(921..921 src has 467..921) 2024-04-14T15:29:07.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.589+0000 7fcb3effd700 3 client.48341.objecter handle_osd_map got epochs [921,921] > 0 2024-04-14T15:29:07.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.589+0000 7fcb3effd700 3 client.48341.objecter handle_osd_map decoding full epoch 921 2024-04-14T15:29:07.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.589+0000 7fcb3effd700 20 client.48341.objecter dump_active .. 0 homeless 2024-04-14T15:29:07.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.589+0000 7fcb3effd700 10 client.48341.objecter ms_handle_connect 0x7fcb340712d0 2024-04-14T15:29:07.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.596+0000 7fcb3effd700 1 -- 172.21.15.90:0/2676720458 <== mon.2 v1:172.21.15.90:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fcb440574f0 con 0x7fcb4810a610 2024-04-14T15:29:07.597 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:29:07.598 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:29:07.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.802+0000 7f8273a0d700 1 -- 172.21.15.90:0/1110086054 >> v1:172.21.15.90:6789/0 conn(0x7f826c10b730 legacy=0x7f826c10db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:29:07.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.802+0000 7f8273a0d700 1 -- 172.21.15.90:0/1110086054 shutdown_connections 2024-04-14T15:29:07.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.802+0000 7f8273a0d700 1 -- 172.21.15.90:0/1110086054 >> 172.21.15.90:0/1110086054 conn(0x7f826c0fe390 msgr2=0x7f826c1007b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:29:07.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.802+0000 7f8273a0d700 1 -- 172.21.15.90:0/1110086054 shutdown_connections 2024-04-14T15:29:07.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.802+0000 7f8273a0d700 1 -- 172.21.15.90:0/1110086054 wait complete. 2024-04-14T15:29:07.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.803+0000 7f8273a0d700 1 librados: starting msgr at 2024-04-14T15:29:07.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.803+0000 7f8273a0d700 1 librados: starting objecter 2024-04-14T15:29:07.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.803+0000 7f8273a0d700 1 Processor -- start 2024-04-14T15:29:07.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.803+0000 7f8273a0d700 1 -- start start 2024-04-14T15:29:07.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.803+0000 7f8273a0d700 1 librados: setting wanted keys 2024-04-14T15:29:07.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.803+0000 7f8273a0d700 1 librados: calling monclient init 2024-04-14T15:29:07.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.803+0000 7f8273a0d700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f826c1aa0d0 con 0x7f826c1a62f0 2024-04-14T15:29:07.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.803+0000 7f8273a0d700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f826c1ab2f0 con 0x7f826c10f300 2024-04-14T15:29:07.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.803+0000 7f8273a0d700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f826c1ac530 con 0x7f826c108880 2024-04-14T15:29:07.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.803+0000 7f82717a9700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f826c108880 0x7f826c10eb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:38498/0 (socket says 172.21.15.90:38498) 2024-04-14T15:29:07.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.803+0000 7f82717a9700 1 -- 172.21.15.90:0/2593399618 learned_addr learned my addr 172.21.15.90:0/2593399618 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:29:07.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.804+0000 7f82627fc700 10 client.?.objecter ms_handle_connect 0x7f826c108880 2024-04-14T15:29:07.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.804+0000 7f82627fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:29:07.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.804+0000 7f82627fc700 1 -- 172.21.15.90:0/2593399618 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1436031204 0 0) 0x7f826c1ac530 con 0x7f826c108880 2024-04-14T15:29:07.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.804+0000 7f82627fc700 1 -- 172.21.15.90:0/2593399618 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f82580035b0 con 0x7f826c108880 2024-04-14T15:29:07.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.804+0000 7f82627fc700 10 client.?.objecter ms_handle_connect 0x7f826c1a62f0 2024-04-14T15:29:07.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.804+0000 7f82627fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:29:07.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.804+0000 7f82627fc700 10 client.?.objecter ms_handle_connect 0x7f826c10f300 2024-04-14T15:29:07.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.804+0000 7f82627fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:29:07.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.804+0000 7f82627fc700 1 -- 172.21.15.90:0/2593399618 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 24983525 0 0) 0x7f826c1aa0d0 con 0x7f826c1a62f0 2024-04-14T15:29:07.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.804+0000 7f82627fc700 1 -- 172.21.15.90:0/2593399618 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f826c1ac530 con 0x7f826c1a62f0 2024-04-14T15:29:07.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.804+0000 7f82627fc700 1 -- 172.21.15.90:0/2593399618 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2083064449 0 0) 0x7f82580035b0 con 0x7f826c108880 2024-04-14T15:29:07.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.804+0000 7f82627fc700 1 -- 172.21.15.90:0/2593399618 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f826c1aa0d0 con 0x7f826c108880 2024-04-14T15:29:07.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.804+0000 7f82627fc700 1 -- 172.21.15.90:0/2593399618 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f82680019c0 con 0x7f826c108880 2024-04-14T15:29:07.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.804+0000 7f82627fc700 1 -- 172.21.15.90:0/2593399618 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3886538154 0 0) 0x7f826c1ab2f0 con 0x7f826c10f300 2024-04-14T15:29:07.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.804+0000 7f82627fc700 1 -- 172.21.15.90:0/2593399618 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f82580035b0 con 0x7f826c10f300 2024-04-14T15:29:07.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.804+0000 7f82627fc700 1 -- 172.21.15.90:0/2593399618 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1005269412 0 0) 0x7f826c1ac530 con 0x7f826c1a62f0 2024-04-14T15:29:07.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.804+0000 7f82627fc700 1 -- 172.21.15.90:0/2593399618 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f826c1ab2f0 con 0x7f826c1a62f0 2024-04-14T15:29:07.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.804+0000 7f82627fc700 1 -- 172.21.15.90:0/2593399618 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f8264001c30 con 0x7f826c1a62f0 2024-04-14T15:29:07.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.805+0000 7f82627fc700 1 -- 172.21.15.90:0/2593399618 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 58006632 0 0) 0x7f82580035b0 con 0x7f826c10f300 2024-04-14T15:29:07.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.805+0000 7f82627fc700 1 -- 172.21.15.90:0/2593399618 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f826c1ac530 con 0x7f826c10f300 2024-04-14T15:29:07.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.805+0000 7f82627fc700 1 -- 172.21.15.90:0/2593399618 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f825c001080 con 0x7f826c10f300 2024-04-14T15:29:07.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.805+0000 7f82627fc700 1 -- 172.21.15.90:0/2593399618 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1267664601 0 0) 0x7f826c1ab2f0 con 0x7f826c1a62f0 2024-04-14T15:29:07.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.805+0000 7f82627fc700 1 -- 172.21.15.90:0/2593399618 >> v1:172.21.15.90:6790/0 conn(0x7f826c108880 legacy=0x7f826c10eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:29:07.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.805+0000 7f82627fc700 1 -- 172.21.15.90:0/2593399618 >> v1:172.21.15.92:6789/0 conn(0x7f826c10f300 legacy=0x7f826c1a4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:29:07.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.805+0000 7f82627fc700 1 -- 172.21.15.90:0/2593399618 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f826c1ad770 con 0x7f826c1a62f0 2024-04-14T15:29:07.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.805+0000 7f8273a0d700 1 -- 172.21.15.90:0/2593399618 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f826c1ab520 con 0x7f826c1a62f0 2024-04-14T15:29:07.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.805+0000 7f82627fc700 1 -- 172.21.15.90:0/2593399618 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f8264003ec0 con 0x7f826c1a62f0 2024-04-14T15:29:07.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.805+0000 7f82627fc700 1 -- 172.21.15.90:0/2593399618 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f8264004e40 con 0x7f826c1a62f0 2024-04-14T15:29:07.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.805+0000 7f8273a0d700 10 client.38397.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:29:07.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.805+0000 7f8273a0d700 1 -- 172.21.15.90:0/2593399618 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f826c1ab9e0 con 0x7f826c1a62f0 2024-04-14T15:29:07.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.805+0000 7f8273a0d700 1 librados: init done 2024-04-14T15:29:07.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.806+0000 7f8273a0d700 1 -- 172.21.15.90:0/2593399618 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f826c103fc0 con 0x7f826c1a62f0 2024-04-14T15:29:07.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.807+0000 7f82627fc700 1 -- 172.21.15.90:0/2593399618 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f8264003a20 con 0x7f826c1a62f0 2024-04-14T15:29:07.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.807+0000 7f82627fc700 1 -- 172.21.15.90:0/2593399618 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(921..921 src has 467..921) ==== 7517+0+0 (unknown 3843985862 0 0) 0x7f8264089940 con 0x7f826c1a62f0 2024-04-14T15:29:07.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.807+0000 7f82627fc700 10 client.38397.objecter ms_dispatch 0x7f826c001550 osd_map(921..921 src has 467..921) 2024-04-14T15:29:07.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.807+0000 7f82627fc700 3 client.38397.objecter handle_osd_map got epochs [921,921] > 0 2024-04-14T15:29:07.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.807+0000 7f82627fc700 3 client.38397.objecter handle_osd_map decoding full epoch 921 2024-04-14T15:29:07.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.807+0000 7f82627fc700 20 client.38397.objecter dump_active .. 0 homeless 2024-04-14T15:29:07.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.808+0000 7f82627fc700 10 client.38397.objecter ms_handle_connect 0x7f82580713b0 2024-04-14T15:29:07.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.816+0000 7f82627fc700 1 -- 172.21.15.90:0/2593399618 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f8264056720 con 0x7f826c1a62f0 2024-04-14T15:29:07.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.828+0000 7fcb4ffd8700 1 -- 172.21.15.90:0/2676720458 --> v1:172.21.15.90:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fcb2c005440 con 0x7fcb4810a610 2024-04-14T15:29:07.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.828+0000 7fcb3effd700 1 -- 172.21.15.90:0/2676720458 <== mon.2 v1:172.21.15.90:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (unknown 92182286 0 4248624373) 0x7fcb4405b1a0 con 0x7fcb4810a610 2024-04-14T15:29:07.829 INFO:teuthology.orchestra.run.smithi090.stdout:77309411592 2024-04-14T15:29:07.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.830+0000 7fcb4ffd8700 10 librados: watch_flush enter 2024-04-14T15:29:07.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.831+0000 7fcb4ffd8700 10 librados: watch_flush exit 2024-04-14T15:29:07.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.831+0000 7fcb4ffd8700 20 client.48341.objecter shutdown clearing up homeless session... 2024-04-14T15:29:07.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.831+0000 7fcb4ffd8700 10 client.48341.objecter successfully canceled tick 2024-04-14T15:29:07.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.831+0000 7fcb4ffd8700 1 -- 172.21.15.90:0/2676720458 >> v1:172.21.15.90:6800/420935427 conn(0x7fcb340712d0 legacy=0x7fcb34073790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:29:07.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.831+0000 7fcb4ffd8700 1 -- 172.21.15.90:0/2676720458 >> v1:172.21.15.90:6790/0 conn(0x7fcb4810a610 legacy=0x7fcb4810b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:29:07.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.831+0000 7fcb4ffd8700 1 -- 172.21.15.90:0/2676720458 shutdown_connections 2024-04-14T15:29:07.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.831+0000 7fcb4ffd8700 1 -- 172.21.15.90:0/2676720458 >> 172.21.15.90:0/2676720458 conn(0x7fcb480fd290 msgr2=0x7fcb48109cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:29:07.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.831+0000 7fcb4ffd8700 1 -- 172.21.15.90:0/2676720458 shutdown_connections 2024-04-14T15:29:07.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.831+0000 7fcb4ffd8700 1 -- 172.21.15.90:0/2676720458 wait complete. 2024-04-14T15:29:07.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:07.831+0000 7fcb4ffd8700 1 librados: shutdown 2024-04-14T15:29:07.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:07 smithi092 ceph-mon[33966]: 6.16 scrub starts 2024-04-14T15:29:07.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:07 smithi092 ceph-mon[33966]: 6.16 scrub ok 2024-04-14T15:29:07.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:07 smithi092 ceph-mon[33966]: 1.2 scrub starts 2024-04-14T15:29:07.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:07 smithi092 ceph-mon[33966]: 1.2 scrub ok 2024-04-14T15:29:07.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:07 smithi092 ceph-mon[33966]: 5.1f scrub starts 2024-04-14T15:29:07.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:07 smithi092 ceph-mon[33966]: 5.1f scrub ok 2024-04-14T15:29:07.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:07 smithi092 ceph-mon[33966]: osdmap e921: 8 total, 8 up, 8 in 2024-04-14T15:29:07.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:07 smithi092 ceph-mon[33966]: 11.b scrub starts 2024-04-14T15:29:07.873 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:07 smithi092 ceph-mon[33966]: 11.b scrub ok 2024-04-14T15:29:08.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:08.024+0000 7f8273a0d700 1 -- 172.21.15.90:0/2593399618 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f826c0693d0 con 0x7f826c1a62f0 2024-04-14T15:29:08.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:08.024+0000 7f82627fc700 1 -- 172.21.15.90:0/2593399618 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (unknown 1482059429 0 2397923231) 0x7f826405a3d0 con 0x7f826c1a62f0 2024-04-14T15:29:08.025 INFO:teuthology.orchestra.run.smithi090.stdout:249108103407 2024-04-14T15:29:08.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:08.026+0000 7f8273a0d700 10 librados: watch_flush enter 2024-04-14T15:29:08.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:08.026+0000 7f8273a0d700 10 librados: watch_flush exit 2024-04-14T15:29:08.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:08.026+0000 7f8273a0d700 20 client.38397.objecter shutdown clearing up homeless session... 2024-04-14T15:29:08.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:08.026+0000 7f8273a0d700 10 client.38397.objecter successfully canceled tick 2024-04-14T15:29:08.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:08.026+0000 7f8273a0d700 1 -- 172.21.15.90:0/2593399618 >> v1:172.21.15.90:6800/420935427 conn(0x7f82580713b0 legacy=0x7f8258073870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:29:08.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:08.026+0000 7f8273a0d700 1 -- 172.21.15.90:0/2593399618 >> v1:172.21.15.90:6789/0 conn(0x7f826c1a62f0 legacy=0x7f826c1a8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:29:08.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:08.027+0000 7f8273a0d700 1 -- 172.21.15.90:0/2593399618 shutdown_connections 2024-04-14T15:29:08.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:08.027+0000 7f8273a0d700 1 -- 172.21.15.90:0/2593399618 >> 172.21.15.90:0/2593399618 conn(0x7f826c0fe390 msgr2=0x7f826c100f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:29:08.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:08.027+0000 7f8273a0d700 1 -- 172.21.15.90:0/2593399618 shutdown_connections 2024-04-14T15:29:08.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:08.027+0000 7f8273a0d700 1 -- 172.21.15.90:0/2593399618 wait complete. 2024-04-14T15:29:08.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:08.027+0000 7f8273a0d700 1 librados: shutdown 2024-04-14T15:29:08.614 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182655 got 107374182657 for osd.3 2024-04-14T15:29:08.614 DEBUG:teuthology.parallel:result is None 2024-04-14T15:29:08.691 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:08 smithi090 ceph-mon[41670]: 5.b scrub starts 2024-04-14T15:29:08.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:08 smithi090 ceph-mon[41670]: 5.b scrub ok 2024-04-14T15:29:08.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:08 smithi090 ceph-mon[41670]: 11.16 deep-scrub starts 2024-04-14T15:29:08.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:08 smithi090 ceph-mon[41670]: 11.16 deep-scrub ok 2024-04-14T15:29:08.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:08 smithi090 ceph-mon[41670]: pgmap v1407: 174 pgs: 174 active+clean; 582 KiB data, 843 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:29:08.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:08 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2676720458' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-14T15:29:08.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:08 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2593399618' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-14T15:29:08.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:08 smithi090 ceph-mon[30399]: 5.b scrub starts 2024-04-14T15:29:08.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:08 smithi090 ceph-mon[30399]: 5.b scrub ok 2024-04-14T15:29:08.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:08 smithi090 ceph-mon[30399]: 11.16 deep-scrub starts 2024-04-14T15:29:08.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:08 smithi090 ceph-mon[30399]: 11.16 deep-scrub ok 2024-04-14T15:29:08.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:08 smithi090 ceph-mon[30399]: pgmap v1407: 174 pgs: 174 active+clean; 582 KiB data, 843 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:29:08.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:08 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2676720458' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-14T15:29:08.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:08 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2593399618' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-14T15:29:08.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:08 smithi092 ceph-mon[33966]: 5.b scrub starts 2024-04-14T15:29:08.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:08 smithi092 ceph-mon[33966]: 5.b scrub ok 2024-04-14T15:29:08.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:08 smithi092 ceph-mon[33966]: 11.16 deep-scrub starts 2024-04-14T15:29:08.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:08 smithi092 ceph-mon[33966]: 11.16 deep-scrub ok 2024-04-14T15:29:08.873 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:08 smithi092 ceph-mon[33966]: pgmap v1407: 174 pgs: 174 active+clean; 582 KiB data, 843 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:29:08.873 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:08 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2676720458' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-14T15:29:08.873 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:08 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2593399618' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-14T15:29:09.691 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:09 smithi090 ceph-mon[41670]: 4.18 scrub starts 2024-04-14T15:29:09.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:09 smithi090 ceph-mon[41670]: 4.18 scrub ok 2024-04-14T15:29:09.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:09 smithi090 ceph-mon[41670]: 2.14 scrub starts 2024-04-14T15:29:09.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:09 smithi090 ceph-mon[41670]: 2.14 scrub ok 2024-04-14T15:29:09.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:09 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:29:09.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:09 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:29:09.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:09 smithi090 ceph-mon[41670]: 5.0 scrub starts 2024-04-14T15:29:09.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:09 smithi090 ceph-mon[41670]: 5.0 scrub ok 2024-04-14T15:29:09.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:09 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:29:09.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:09 smithi090 ceph-mon[30399]: 4.18 scrub starts 2024-04-14T15:29:09.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:09 smithi090 ceph-mon[30399]: 4.18 scrub ok 2024-04-14T15:29:09.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:09 smithi090 ceph-mon[30399]: 2.14 scrub starts 2024-04-14T15:29:09.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:09 smithi090 ceph-mon[30399]: 2.14 scrub ok 2024-04-14T15:29:09.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:09 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:29:09.694 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:09 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:29:09.695 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:09 smithi090 ceph-mon[30399]: 5.0 scrub starts 2024-04-14T15:29:09.695 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:09 smithi090 ceph-mon[30399]: 5.0 scrub ok 2024-04-14T15:29:09.695 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:09 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:29:09.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:09 smithi092 ceph-mon[33966]: 4.18 scrub starts 2024-04-14T15:29:09.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:09 smithi092 ceph-mon[33966]: 4.18 scrub ok 2024-04-14T15:29:09.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:09 smithi092 ceph-mon[33966]: 2.14 scrub starts 2024-04-14T15:29:09.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:09 smithi092 ceph-mon[33966]: 2.14 scrub ok 2024-04-14T15:29:09.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:09 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:29:09.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:09 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:29:09.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:09 smithi092 ceph-mon[33966]: 5.0 scrub starts 2024-04-14T15:29:09.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:09 smithi092 ceph-mon[33966]: 5.0 scrub ok 2024-04-14T15:29:09.873 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:09 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:29:10.058 INFO:tasks.cephadm.ceph_manager.ceph:need seq 249108103403 got 249108103407 for osd.7 2024-04-14T15:29:10.058 DEBUG:teuthology.parallel:result is None 2024-04-14T15:29:10.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:10 smithi090 ceph-mon[30399]: 6.19 scrub starts 2024-04-14T15:29:10.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:10 smithi090 ceph-mon[30399]: 6.19 scrub ok 2024-04-14T15:29:10.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:10 smithi090 ceph-mon[30399]: 11.15 scrub starts 2024-04-14T15:29:10.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:10 smithi090 ceph-mon[30399]: 11.15 scrub ok 2024-04-14T15:29:10.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:10 smithi090 ceph-mon[30399]: pgmap v1408: 174 pgs: 174 active+clean; 582 KiB data, 844 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:29:10.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:10 smithi090 ceph-mon[30399]: osdmap e922: 8 total, 8 up, 8 in 2024-04-14T15:29:10.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:10 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:29:10.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:10 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:29:10.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:10 smithi090 ceph-mon[41670]: 6.19 scrub starts 2024-04-14T15:29:10.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:10 smithi090 ceph-mon[41670]: 6.19 scrub ok 2024-04-14T15:29:10.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:10 smithi090 ceph-mon[41670]: 11.15 scrub starts 2024-04-14T15:29:10.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:10 smithi090 ceph-mon[41670]: 11.15 scrub ok 2024-04-14T15:29:10.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:10 smithi090 ceph-mon[41670]: pgmap v1408: 174 pgs: 174 active+clean; 582 KiB data, 844 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:29:10.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:10 smithi090 ceph-mon[41670]: osdmap e922: 8 total, 8 up, 8 in 2024-04-14T15:29:10.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:10 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:29:10.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:10 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:29:10.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:10 smithi092 ceph-mon[33966]: 6.19 scrub starts 2024-04-14T15:29:10.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:10 smithi092 ceph-mon[33966]: 6.19 scrub ok 2024-04-14T15:29:10.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:10 smithi092 ceph-mon[33966]: 11.15 scrub starts 2024-04-14T15:29:10.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:10 smithi092 ceph-mon[33966]: 11.15 scrub ok 2024-04-14T15:29:10.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:10 smithi092 ceph-mon[33966]: pgmap v1408: 174 pgs: 174 active+clean; 582 KiB data, 844 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:29:10.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:10 smithi092 ceph-mon[33966]: osdmap e922: 8 total, 8 up, 8 in 2024-04-14T15:29:10.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:10 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:29:10.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:10 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:29:11.691 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:11 smithi090 ceph-mon[30399]: osdmap e923: 8 total, 8 up, 8 in 2024-04-14T15:29:11.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:11 smithi090 ceph-mon[30399]: 11.9 scrub starts 2024-04-14T15:29:11.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:11 smithi090 ceph-mon[30399]: 11.9 scrub ok 2024-04-14T15:29:11.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:11 smithi090 ceph-mon[30399]: osdmap e924: 8 total, 8 up, 8 in 2024-04-14T15:29:11.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:11 smithi090 ceph-mon[41670]: osdmap e923: 8 total, 8 up, 8 in 2024-04-14T15:29:11.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:11 smithi090 ceph-mon[41670]: 11.9 scrub starts 2024-04-14T15:29:11.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:11 smithi090 ceph-mon[41670]: 11.9 scrub ok 2024-04-14T15:29:11.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:11 smithi090 ceph-mon[41670]: osdmap e924: 8 total, 8 up, 8 in 2024-04-14T15:29:11.823 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411588 got 77309411592 for osd.2 2024-04-14T15:29:11.823 DEBUG:teuthology.parallel:result is None 2024-04-14T15:29:11.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:11 smithi092 ceph-mon[33966]: osdmap e923: 8 total, 8 up, 8 in 2024-04-14T15:29:11.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:11 smithi092 ceph-mon[33966]: 11.9 scrub starts 2024-04-14T15:29:11.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:11 smithi092 ceph-mon[33966]: 11.9 scrub ok 2024-04-14T15:29:11.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:11 smithi092 ceph-mon[33966]: osdmap e924: 8 total, 8 up, 8 in 2024-04-14T15:29:12.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:12.024+0000 7f50a45a6700 1 -- 172.21.15.90:0/243564833 >> v1:172.21.15.90:6789/0 conn(0x7f509c10e200 legacy=0x7f509c1106c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:29:12.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:12.024+0000 7f50a45a6700 1 -- 172.21.15.90:0/243564833 shutdown_connections 2024-04-14T15:29:12.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:12.024+0000 7f50a45a6700 1 -- 172.21.15.90:0/243564833 >> 172.21.15.90:0/243564833 conn(0x7f509c0fd2b0 msgr2=0x7f509c0ff6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:29:12.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:12.024+0000 7f50a45a6700 1 -- 172.21.15.90:0/243564833 shutdown_connections 2024-04-14T15:29:12.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:12.024+0000 7f50a45a6700 1 -- 172.21.15.90:0/243564833 wait complete. 2024-04-14T15:29:12.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:12.025+0000 7f50a45a6700 1 librados: starting msgr at 2024-04-14T15:29:12.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:12.025+0000 7f50a45a6700 1 librados: starting objecter 2024-04-14T15:29:12.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:12.025+0000 7f50a45a6700 1 Processor -- start 2024-04-14T15:29:12.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:12.025+0000 7f50a45a6700 1 -- start start 2024-04-14T15:29:12.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:12.025+0000 7f50a45a6700 1 librados: setting wanted keys 2024-04-14T15:29:12.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:12.025+0000 7f50a45a6700 1 librados: calling monclient init 2024-04-14T15:29:12.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:12.025+0000 7f50a45a6700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f509c10de60 con 0x7f509c107780 2024-04-14T15:29:12.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:12.025+0000 7f50a45a6700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f509c1a7e10 con 0x7f509c10a630 2024-04-14T15:29:12.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:12.025+0000 7f50a45a6700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f509c1a8ff0 con 0x7f509c10e200 2024-04-14T15:29:12.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:12.025+0000 7f50a2342700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f509c107780 0x7f509c10ad40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:37484/0 (socket says 172.21.15.90:37484) 2024-04-14T15:29:12.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:12.025+0000 7f50a2342700 1 -- 172.21.15.90:0/2773422032 learned_addr learned my addr 172.21.15.90:0/2773422032 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:29:12.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:12.025+0000 7f509b7fe700 10 client.?.objecter ms_handle_connect 0x7f509c107780 2024-04-14T15:29:12.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:12.025+0000 7f509b7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:29:12.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:12.025+0000 7f509b7fe700 10 client.?.objecter ms_handle_connect 0x7f509c10e200 2024-04-14T15:29:12.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:12.025+0000 7f509b7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:29:12.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:12.025+0000 7f509b7fe700 1 -- 172.21.15.90:0/2773422032 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 82873784 0 0) 0x7f509c10de60 con 0x7f509c107780 2024-04-14T15:29:12.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:12.025+0000 7f509b7fe700 1 -- 172.21.15.90:0/2773422032 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f50840035b0 con 0x7f509c107780 2024-04-14T15:29:12.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:12.025+0000 7f509b7fe700 10 client.?.objecter ms_handle_connect 0x7f509c10a630 2024-04-14T15:29:12.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:12.025+0000 7f509b7fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:29:12.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:12.025+0000 7f509b7fe700 1 -- 172.21.15.90:0/2773422032 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1784365361 0 0) 0x7f509c1a8ff0 con 0x7f509c10e200 2024-04-14T15:29:12.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:12.026+0000 7f509b7fe700 1 -- 172.21.15.90:0/2773422032 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f509c10de60 con 0x7f509c10e200 2024-04-14T15:29:12.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:12.026+0000 7f509b7fe700 1 -- 172.21.15.90:0/2773422032 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 867588684 0 0) 0x7f509c1a7e10 con 0x7f509c10a630 2024-04-14T15:29:12.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:12.026+0000 7f509b7fe700 1 -- 172.21.15.90:0/2773422032 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f509c1a8ff0 con 0x7f509c10a630 2024-04-14T15:29:12.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:12.026+0000 7f509b7fe700 1 -- 172.21.15.90:0/2773422032 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3722632868 0 0) 0x7f50840035b0 con 0x7f509c107780 2024-04-14T15:29:12.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:12.026+0000 7f509b7fe700 1 -- 172.21.15.90:0/2773422032 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f509c1a7e10 con 0x7f509c107780 2024-04-14T15:29:12.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:12.026+0000 7f509b7fe700 1 -- 172.21.15.90:0/2773422032 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f5094001c30 con 0x7f509c107780 2024-04-14T15:29:12.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:12.026+0000 7f509b7fe700 1 -- 172.21.15.90:0/2773422032 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 626255930 0 0) 0x7f509c10de60 con 0x7f509c10e200 2024-04-14T15:29:12.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:12.026+0000 7f509b7fe700 1 -- 172.21.15.90:0/2773422032 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f50840035b0 con 0x7f509c10e200 2024-04-14T15:29:12.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:12.026+0000 7f509b7fe700 1 -- 172.21.15.90:0/2773422032 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f508c001430 con 0x7f509c10e200 2024-04-14T15:29:12.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:12.026+0000 7f509b7fe700 1 -- 172.21.15.90:0/2773422032 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2920993030 0 0) 0x7f509c1a7e10 con 0x7f509c107780 2024-04-14T15:29:12.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:12.026+0000 7f509b7fe700 1 -- 172.21.15.90:0/2773422032 >> v1:172.21.15.90:6790/0 conn(0x7f509c10e200 legacy=0x7f509c1a56e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:29:12.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:12.026+0000 7f509b7fe700 1 -- 172.21.15.90:0/2773422032 >> v1:172.21.15.92:6789/0 conn(0x7f509c10a630 legacy=0x7f509c10b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:29:12.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:12.026+0000 7f509b7fe700 1 -- 172.21.15.90:0/2773422032 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f509c1aa1d0 con 0x7f509c107780 2024-04-14T15:29:12.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:12.026+0000 7f50a45a6700 1 -- 172.21.15.90:0/2773422032 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f509c1a9220 con 0x7f509c107780 2024-04-14T15:29:12.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:12.026+0000 7f50a45a6700 10 client.38403.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:29:12.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:12.026+0000 7f50a45a6700 1 -- 172.21.15.90:0/2773422032 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f509c1a96e0 con 0x7f509c107780 2024-04-14T15:29:12.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:12.026+0000 7f50a45a6700 1 librados: init done 2024-04-14T15:29:12.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:12.026+0000 7f509b7fe700 1 -- 172.21.15.90:0/2773422032 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f5094003ec0 con 0x7f509c107780 2024-04-14T15:29:12.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:12.026+0000 7f509b7fe700 1 -- 172.21.15.90:0/2773422032 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f5094006020 con 0x7f509c107780 2024-04-14T15:29:12.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:12.027+0000 7f50a45a6700 1 -- 172.21.15.90:0/2773422032 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5088005120 con 0x7f509c107780 2024-04-14T15:29:12.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:12.027+0000 7f509b7fe700 1 -- 172.21.15.90:0/2773422032 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f5094003a20 con 0x7f509c107780 2024-04-14T15:29:12.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:12.028+0000 7f509b7fe700 1 -- 172.21.15.90:0/2773422032 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(924..924 src has 467..924) ==== 7517+0+0 (unknown 2074792842 0 0) 0x7f509408aa10 con 0x7f509c107780 2024-04-14T15:29:12.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:12.028+0000 7f509b7fe700 10 client.38403.objecter ms_dispatch 0x7f509c001550 osd_map(924..924 src has 467..924) 2024-04-14T15:29:12.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:12.028+0000 7f509b7fe700 3 client.38403.objecter handle_osd_map got epochs [924,924] > 0 2024-04-14T15:29:12.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:12.028+0000 7f509b7fe700 3 client.38403.objecter handle_osd_map decoding full epoch 924 2024-04-14T15:29:12.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:12.028+0000 7f509b7fe700 20 client.38403.objecter dump_active .. 0 homeless 2024-04-14T15:29:12.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:12.028+0000 7f509b7fe700 10 client.38403.objecter ms_handle_connect 0x7f50840710a0 2024-04-14T15:29:12.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:12.032+0000 7f509b7fe700 1 -- 172.21.15.90:0/2773422032 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f50940577f0 con 0x7f509c107780 2024-04-14T15:29:12.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:12.238+0000 7f50a45a6700 1 -- 172.21.15.90:0/2773422032 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f5088005410 con 0x7f509c107780 2024-04-14T15:29:12.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:12.238+0000 7f509b7fe700 1 -- 172.21.15.90:0/2773422032 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (unknown 832126871 0 1724774430) 0x7f509405b4a0 con 0x7f509c107780 2024-04-14T15:29:12.238 INFO:teuthology.orchestra.run.smithi090.stdout:55834575117 2024-04-14T15:29:12.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:12.240+0000 7f50a45a6700 10 librados: watch_flush enter 2024-04-14T15:29:12.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:12.240+0000 7f50a45a6700 10 librados: watch_flush exit 2024-04-14T15:29:12.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:12.240+0000 7f50a45a6700 20 client.38403.objecter shutdown clearing up homeless session... 2024-04-14T15:29:12.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:12.240+0000 7f50a45a6700 10 client.38403.objecter successfully canceled tick 2024-04-14T15:29:12.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:12.240+0000 7f50a45a6700 1 -- 172.21.15.90:0/2773422032 >> v1:172.21.15.90:6800/420935427 conn(0x7f50840710a0 legacy=0x7f5084073560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:29:12.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:12.240+0000 7f50a45a6700 1 -- 172.21.15.90:0/2773422032 >> v1:172.21.15.90:6789/0 conn(0x7f509c107780 legacy=0x7f509c10ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:29:12.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:12.240+0000 7f50a45a6700 1 -- 172.21.15.90:0/2773422032 shutdown_connections 2024-04-14T15:29:12.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:12.240+0000 7f50a45a6700 1 -- 172.21.15.90:0/2773422032 >> 172.21.15.90:0/2773422032 conn(0x7f509c0fd2b0 msgr2=0x7f509c111720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:29:12.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:12.240+0000 7f50a45a6700 1 -- 172.21.15.90:0/2773422032 shutdown_connections 2024-04-14T15:29:12.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:12.240+0000 7f50a45a6700 1 -- 172.21.15.90:0/2773422032 wait complete. 2024-04-14T15:29:12.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:12.240+0000 7f50a45a6700 1 librados: shutdown 2024-04-14T15:29:12.506 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:12 smithi090 ceph-mon[41670]: 6.9 deep-scrub starts 2024-04-14T15:29:12.506 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:12 smithi090 ceph-mon[41670]: 6.9 deep-scrub ok 2024-04-14T15:29:12.507 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:12 smithi090 ceph-mon[41670]: 11.18 scrub starts 2024-04-14T15:29:12.507 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:12 smithi090 ceph-mon[41670]: 11.18 scrub ok 2024-04-14T15:29:12.507 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:12 smithi090 ceph-mon[41670]: pgmap v1412: 174 pgs: 174 active+clean; 582 KiB data, 844 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-14T15:29:12.507 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:12 smithi090 ceph-mon[41670]: 11.5 deep-scrub starts 2024-04-14T15:29:12.507 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:12 smithi090 ceph-mon[41670]: 11.5 deep-scrub ok 2024-04-14T15:29:12.507 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:12 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/2773422032' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-14T15:29:12.507 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:12 smithi090 ceph-mon[30399]: 6.9 deep-scrub starts 2024-04-14T15:29:12.507 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:12 smithi090 ceph-mon[30399]: 6.9 deep-scrub ok 2024-04-14T15:29:12.507 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:12 smithi090 ceph-mon[30399]: 11.18 scrub starts 2024-04-14T15:29:12.507 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:12 smithi090 ceph-mon[30399]: 11.18 scrub ok 2024-04-14T15:29:12.507 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:12 smithi090 ceph-mon[30399]: pgmap v1412: 174 pgs: 174 active+clean; 582 KiB data, 844 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-14T15:29:12.507 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:12 smithi090 ceph-mon[30399]: 11.5 deep-scrub starts 2024-04-14T15:29:12.508 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:12 smithi090 ceph-mon[30399]: 11.5 deep-scrub ok 2024-04-14T15:29:12.508 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:12 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/2773422032' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-14T15:29:12.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:12 smithi092 ceph-mon[33966]: 6.9 deep-scrub starts 2024-04-14T15:29:12.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:12 smithi092 ceph-mon[33966]: 6.9 deep-scrub ok 2024-04-14T15:29:12.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:12 smithi092 ceph-mon[33966]: 11.18 scrub starts 2024-04-14T15:29:12.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:12 smithi092 ceph-mon[33966]: 11.18 scrub ok 2024-04-14T15:29:12.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:12 smithi092 ceph-mon[33966]: pgmap v1412: 174 pgs: 174 active+clean; 582 KiB data, 844 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-14T15:29:12.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:12 smithi092 ceph-mon[33966]: 11.5 deep-scrub starts 2024-04-14T15:29:12.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:12 smithi092 ceph-mon[33966]: 11.5 deep-scrub ok 2024-04-14T15:29:12.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:12 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/2773422032' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-14T15:29:13.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.031+0000 7f3c11a23700 1 -- 172.21.15.90:0/1196926860 >> v1:172.21.15.90:6789/0 conn(0x7f3c0c10f2f0 legacy=0x7f3c0c1117b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:29:13.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.031+0000 7f3c11a23700 1 -- 172.21.15.90:0/1196926860 shutdown_connections 2024-04-14T15:29:13.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.031+0000 7f3c11a23700 1 -- 172.21.15.90:0/1196926860 >> 172.21.15.90:0/1196926860 conn(0x7f3c0c076aa0 msgr2=0x7f3c0c076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:29:13.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.031+0000 7f3c11a23700 1 -- 172.21.15.90:0/1196926860 shutdown_connections 2024-04-14T15:29:13.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.031+0000 7f3c11a23700 1 -- 172.21.15.90:0/1196926860 wait complete. 2024-04-14T15:29:13.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.031+0000 7f3c11a23700 1 librados: starting msgr at 2024-04-14T15:29:13.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.031+0000 7f3c11a23700 1 librados: starting objecter 2024-04-14T15:29:13.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.031+0000 7f3c11a23700 1 Processor -- start 2024-04-14T15:29:13.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.031+0000 7f3c11a23700 1 -- start start 2024-04-14T15:29:13.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.031+0000 7f3c11a23700 1 librados: setting wanted keys 2024-04-14T15:29:13.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.031+0000 7f3c11a23700 1 librados: calling monclient init 2024-04-14T15:29:13.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.032+0000 7f3c11a23700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3c0c103e90 con 0x7f3c0c108980 2024-04-14T15:29:13.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.032+0000 7f3c11a23700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3c0c1040a0 con 0x7f3c0c10b6f0 2024-04-14T15:29:13.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.032+0000 7f3c11a23700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3c0c1ac300 con 0x7f3c0c10f2f0 2024-04-14T15:29:13.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.032+0000 7f3c0b7fe700 1 --1- >> v1:172.21.15.90:6790/0 conn(0x7f3c0c10f2f0 0x7f3c0c1a8bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:42554/0 (socket says 172.21.15.90:42554) 2024-04-14T15:29:13.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.032+0000 7f3c0b7fe700 1 -- 172.21.15.90:0/1119027449 learned_addr learned my addr 172.21.15.90:0/1119027449 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:29:13.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.032+0000 7f3c0affd700 1 --1- 172.21.15.90:0/1119027449 >> v1:172.21.15.90:6789/0 conn(0x7f3c0c108980 0x7f3c0c102260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:37510/0 (socket says 172.21.15.90:37510) 2024-04-14T15:29:13.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.032+0000 7f3c10a21700 10 client.?.objecter ms_handle_connect 0x7f3c0c10f2f0 2024-04-14T15:29:13.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.032+0000 7f3c10a21700 10 client.?.objecter resend_mon_ops 2024-04-14T15:29:13.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.032+0000 7f3c10a21700 10 client.?.objecter ms_handle_connect 0x7f3c0c108980 2024-04-14T15:29:13.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.032+0000 7f3c10a21700 10 client.?.objecter resend_mon_ops 2024-04-14T15:29:13.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.032+0000 7f3c10a21700 10 client.?.objecter ms_handle_connect 0x7f3c0c10b6f0 2024-04-14T15:29:13.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.032+0000 7f3c10a21700 10 client.?.objecter resend_mon_ops 2024-04-14T15:29:13.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.032+0000 7f3c10a21700 1 -- 172.21.15.90:0/1119027449 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3692872602 0 0) 0x7f3c0c103e90 con 0x7f3c0c108980 2024-04-14T15:29:13.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.032+0000 7f3c10a21700 1 -- 172.21.15.90:0/1119027449 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3bf80035b0 con 0x7f3c0c108980 2024-04-14T15:29:13.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.032+0000 7f3c10a21700 1 -- 172.21.15.90:0/1119027449 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1662765616 0 0) 0x7f3c0c1ac300 con 0x7f3c0c10f2f0 2024-04-14T15:29:13.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.032+0000 7f3c10a21700 1 -- 172.21.15.90:0/1119027449 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3c0c103e90 con 0x7f3c0c10f2f0 2024-04-14T15:29:13.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.032+0000 7f3c10a21700 1 -- 172.21.15.90:0/1119027449 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3966237343 0 0) 0x7f3c0c1040a0 con 0x7f3c0c10b6f0 2024-04-14T15:29:13.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.032+0000 7f3c10a21700 1 -- 172.21.15.90:0/1119027449 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3c0c1ac300 con 0x7f3c0c10b6f0 2024-04-14T15:29:13.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.032+0000 7f3c10a21700 1 -- 172.21.15.90:0/1119027449 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4002490137 0 0) 0x7f3bf80035b0 con 0x7f3c0c108980 2024-04-14T15:29:13.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.032+0000 7f3c10a21700 1 -- 172.21.15.90:0/1119027449 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3c0c1040a0 con 0x7f3c0c108980 2024-04-14T15:29:13.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.032+0000 7f3c10a21700 1 -- 172.21.15.90:0/1119027449 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f3bfc0018d0 con 0x7f3c0c108980 2024-04-14T15:29:13.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.032+0000 7f3c10a21700 1 -- 172.21.15.90:0/1119027449 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1299060166 0 0) 0x7f3c0c103e90 con 0x7f3c0c10f2f0 2024-04-14T15:29:13.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.032+0000 7f3c10a21700 1 -- 172.21.15.90:0/1119027449 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3bf80035b0 con 0x7f3c0c10f2f0 2024-04-14T15:29:13.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.033+0000 7f3c10a21700 1 -- 172.21.15.90:0/1119027449 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f3c040017b0 con 0x7f3c0c10f2f0 2024-04-14T15:29:13.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.033+0000 7f3c10a21700 1 -- 172.21.15.90:0/1119027449 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1742257732 0 0) 0x7f3c0c1ac300 con 0x7f3c0c10b6f0 2024-04-14T15:29:13.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.033+0000 7f3c10a21700 1 -- 172.21.15.90:0/1119027449 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3c0c103e90 con 0x7f3c0c10b6f0 2024-04-14T15:29:13.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.033+0000 7f3c10a21700 1 -- 172.21.15.90:0/1119027449 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f3bf40016a0 con 0x7f3c0c10b6f0 2024-04-14T15:29:13.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.033+0000 7f3c10a21700 1 -- 172.21.15.90:0/1119027449 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2876693985 0 0) 0x7f3c0c1040a0 con 0x7f3c0c108980 2024-04-14T15:29:13.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.033+0000 7f3c10a21700 1 -- 172.21.15.90:0/1119027449 >> v1:172.21.15.90:6790/0 conn(0x7f3c0c10f2f0 legacy=0x7f3c0c1a8bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:29:13.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.033+0000 7f3c10a21700 1 -- 172.21.15.90:0/1119027449 >> v1:172.21.15.92:6789/0 conn(0x7f3c0c10b6f0 legacy=0x7f3c0c102970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:29:13.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.033+0000 7f3c10a21700 1 -- 172.21.15.90:0/1119027449 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c0c1ad4e0 con 0x7f3c0c108980 2024-04-14T15:29:13.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.033+0000 7f3c11a23700 1 -- 172.21.15.90:0/1119027449 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3c0c1ac530 con 0x7f3c0c108980 2024-04-14T15:29:13.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.033+0000 7f3c11a23700 10 client.38409.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:29:13.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.033+0000 7f3c11a23700 1 -- 172.21.15.90:0/1119027449 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f3c0c1ac9f0 con 0x7f3c0c108980 2024-04-14T15:29:13.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.033+0000 7f3c11a23700 1 librados: init done 2024-04-14T15:29:13.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.033+0000 7f3c10a21700 1 -- 172.21.15.90:0/1119027449 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7f3bfc003b20 con 0x7f3c0c108980 2024-04-14T15:29:13.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.033+0000 7f3c10a21700 1 -- 172.21.15.90:0/1119027449 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7f3bfc005c50 con 0x7f3c0c108980 2024-04-14T15:29:13.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.034+0000 7f3c11a23700 1 -- 172.21.15.90:0/1119027449 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bec005120 con 0x7f3c0c108980 2024-04-14T15:29:13.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.034+0000 7f3c10a21700 1 -- 172.21.15.90:0/1119027449 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7f3bfc005c30 con 0x7f3c0c108980 2024-04-14T15:29:13.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.035+0000 7f3c10a21700 1 -- 172.21.15.90:0/1119027449 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(924..924 src has 467..924) ==== 7517+0+0 (unknown 2074792842 0 0) 0x7f3bfc08a680 con 0x7f3c0c108980 2024-04-14T15:29:13.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.035+0000 7f3c10a21700 10 client.38409.objecter ms_dispatch 0x7f3c0c001550 osd_map(924..924 src has 467..924) 2024-04-14T15:29:13.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.035+0000 7f3c10a21700 3 client.38409.objecter handle_osd_map got epochs [924,924] > 0 2024-04-14T15:29:13.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.035+0000 7f3c10a21700 3 client.38409.objecter handle_osd_map decoding full epoch 924 2024-04-14T15:29:13.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.035+0000 7f3c10a21700 20 client.38409.objecter dump_active .. 0 homeless 2024-04-14T15:29:13.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.035+0000 7f3c10a21700 10 client.38409.objecter ms_handle_connect 0x7f3bf8071470 2024-04-14T15:29:13.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.039+0000 7f3c10a21700 1 -- 172.21.15.90:0/1119027449 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7f3bfc057320 con 0x7f3c0c108980 2024-04-14T15:29:13.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.245+0000 7f3c11a23700 1 -- 172.21.15.90:0/1119027449 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f3bec005410 con 0x7f3c0c108980 2024-04-14T15:29:13.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.245+0000 7f3c10a21700 1 -- 172.21.15.90:0/1119027449 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (unknown 1823589948 0 3297004104) 0x7f3bfc05afd0 con 0x7f3c0c108980 2024-04-14T15:29:13.246 INFO:teuthology.orchestra.run.smithi090.stdout:133143986429 2024-04-14T15:29:13.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.247+0000 7f3c11a23700 10 librados: watch_flush enter 2024-04-14T15:29:13.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.247+0000 7f3c11a23700 10 librados: watch_flush exit 2024-04-14T15:29:13.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.247+0000 7f3c11a23700 20 client.38409.objecter shutdown clearing up homeless session... 2024-04-14T15:29:13.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.247+0000 7f3c11a23700 10 client.38409.objecter successfully canceled tick 2024-04-14T15:29:13.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.247+0000 7f3c11a23700 1 -- 172.21.15.90:0/1119027449 >> v1:172.21.15.90:6800/420935427 conn(0x7f3bf8071470 legacy=0x7f3bf8073930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:29:13.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.247+0000 7f3c11a23700 1 -- 172.21.15.90:0/1119027449 >> v1:172.21.15.90:6789/0 conn(0x7f3c0c108980 legacy=0x7f3c0c102260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:29:13.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.247+0000 7f3c11a23700 1 -- 172.21.15.90:0/1119027449 shutdown_connections 2024-04-14T15:29:13.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.247+0000 7f3c11a23700 1 -- 172.21.15.90:0/1119027449 >> 172.21.15.90:0/1119027449 conn(0x7f3c0c076aa0 msgr2=0x7f3c0c111770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:29:13.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.247+0000 7f3c11a23700 1 -- 172.21.15.90:0/1119027449 shutdown_connections 2024-04-14T15:29:13.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.247+0000 7f3c11a23700 1 -- 172.21.15.90:0/1119027449 wait complete. 2024-04-14T15:29:13.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:13.247+0000 7f3c11a23700 1 librados: shutdown 2024-04-14T15:29:13.674 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:13 smithi090 ceph-mon[41670]: 11.0 scrub starts 2024-04-14T15:29:13.675 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:13 smithi090 ceph-mon[41670]: 11.0 scrub ok 2024-04-14T15:29:13.675 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:13 smithi090 ceph-mon[41670]: 11.17 deep-scrub starts 2024-04-14T15:29:13.675 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:13 smithi090 ceph-mon[41670]: 11.17 deep-scrub ok 2024-04-14T15:29:13.675 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:13 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/1119027449' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-14T15:29:13.675 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:13 smithi090 ceph-mon[30399]: 11.0 scrub starts 2024-04-14T15:29:13.675 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:13 smithi090 ceph-mon[30399]: 11.0 scrub ok 2024-04-14T15:29:13.675 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:13 smithi090 ceph-mon[30399]: 11.17 deep-scrub starts 2024-04-14T15:29:13.675 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:13 smithi090 ceph-mon[30399]: 11.17 deep-scrub ok 2024-04-14T15:29:13.675 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:13 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/1119027449' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-14T15:29:13.701 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834575114 got 55834575117 for osd.1 2024-04-14T15:29:13.702 DEBUG:teuthology.parallel:result is None 2024-04-14T15:29:13.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:13 smithi092 ceph-mon[33966]: 11.0 scrub starts 2024-04-14T15:29:13.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:13 smithi092 ceph-mon[33966]: 11.0 scrub ok 2024-04-14T15:29:13.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:13 smithi092 ceph-mon[33966]: 11.17 deep-scrub starts 2024-04-14T15:29:13.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:13 smithi092 ceph-mon[33966]: 11.17 deep-scrub ok 2024-04-14T15:29:13.873 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:13 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/1119027449' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-14T15:29:14.429 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:29:13 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:29:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:29:14.462 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986425 got 133143986429 for osd.4 2024-04-14T15:29:14.462 DEBUG:teuthology.parallel:result is None 2024-04-14T15:29:14.462 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-04-14T15:29:14.462 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph pg dump --format=json 2024-04-14T15:29:14.691 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:14 smithi090 ceph-mon[41670]: pgmap v1413: 174 pgs: 174 active+clean; 582 KiB data, 844 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:29:14.691 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:14 smithi090 ceph-mon[41670]: 11.1d scrub starts 2024-04-14T15:29:14.691 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:14 smithi090 ceph-mon[41670]: 11.1d scrub ok 2024-04-14T15:29:14.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:14 smithi090 ceph-mon[30399]: pgmap v1413: 174 pgs: 174 active+clean; 582 KiB data, 844 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:29:14.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:14 smithi090 ceph-mon[30399]: 11.1d scrub starts 2024-04-14T15:29:14.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:14 smithi090 ceph-mon[30399]: 11.1d scrub ok 2024-04-14T15:29:14.706 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:29:14.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:14 smithi092 ceph-mon[33966]: pgmap v1413: 174 pgs: 174 active+clean; 582 KiB data, 844 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:29:14.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:14 smithi092 ceph-mon[33966]: 11.1d scrub starts 2024-04-14T15:29:14.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:14 smithi092 ceph-mon[33966]: 11.1d scrub ok 2024-04-14T15:29:15.691 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:15 smithi090 ceph-mon[30399]: 5.1 scrub starts 2024-04-14T15:29:15.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:15 smithi090 ceph-mon[30399]: 5.1 scrub ok 2024-04-14T15:29:15.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:15 smithi090 ceph-mon[30399]: 6.14 scrub starts 2024-04-14T15:29:15.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:15 smithi090 ceph-mon[30399]: 6.14 scrub ok 2024-04-14T15:29:15.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:15 smithi090 ceph-mon[30399]: 11.1 scrub starts 2024-04-14T15:29:15.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:15 smithi090 ceph-mon[30399]: 11.1 scrub ok 2024-04-14T15:29:15.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:15 smithi090 ceph-mon[41670]: 5.1 scrub starts 2024-04-14T15:29:15.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:15 smithi090 ceph-mon[41670]: 5.1 scrub ok 2024-04-14T15:29:15.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:15 smithi090 ceph-mon[41670]: 6.14 scrub starts 2024-04-14T15:29:15.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:15 smithi090 ceph-mon[41670]: 6.14 scrub ok 2024-04-14T15:29:15.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:15 smithi090 ceph-mon[41670]: 11.1 scrub starts 2024-04-14T15:29:15.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:15 smithi090 ceph-mon[41670]: 11.1 scrub ok 2024-04-14T15:29:15.698 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:15 smithi092 ceph-mon[33966]: 5.1 scrub starts 2024-04-14T15:29:15.698 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:15 smithi092 ceph-mon[33966]: 5.1 scrub ok 2024-04-14T15:29:15.699 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:15 smithi092 ceph-mon[33966]: 6.14 scrub starts 2024-04-14T15:29:15.699 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:15 smithi092 ceph-mon[33966]: 6.14 scrub ok 2024-04-14T15:29:15.699 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:15 smithi092 ceph-mon[33966]: 11.1 scrub starts 2024-04-14T15:29:15.699 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:15 smithi092 ceph-mon[33966]: 11.1 scrub ok 2024-04-14T15:29:16.122 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:29:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:29:16.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.411+0000 7fa6f447e700 1 -- 172.21.15.90:0/2048588812 >> v1:172.21.15.90:6790/0 conn(0x7fa6ec10e1e0 legacy=0x7fa6ec1106a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:29:16.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.411+0000 7fa6f447e700 1 -- 172.21.15.90:0/2048588812 shutdown_connections 2024-04-14T15:29:16.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.411+0000 7fa6f447e700 1 -- 172.21.15.90:0/2048588812 >> 172.21.15.90:0/2048588812 conn(0x7fa6ec0fd290 msgr2=0x7fa6ec0ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:29:16.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.412+0000 7fa6f447e700 1 -- 172.21.15.90:0/2048588812 shutdown_connections 2024-04-14T15:29:16.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.412+0000 7fa6f447e700 1 -- 172.21.15.90:0/2048588812 wait complete. 2024-04-14T15:29:16.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.412+0000 7fa6f447e700 1 librados: starting msgr at 2024-04-14T15:29:16.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.412+0000 7fa6f447e700 1 librados: starting objecter 2024-04-14T15:29:16.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.412+0000 7fa6f447e700 1 Processor -- start 2024-04-14T15:29:16.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.412+0000 7fa6f447e700 1 -- start start 2024-04-14T15:29:16.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.412+0000 7fa6f447e700 1 librados: setting wanted keys 2024-04-14T15:29:16.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.412+0000 7fa6f447e700 1 librados: calling monclient init 2024-04-14T15:29:16.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.412+0000 7fa6f447e700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa6ec1a78e0 con 0x7fa6ec107760 2024-04-14T15:29:16.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.412+0000 7fa6f447e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa6ec1a8b20 con 0x7fa6ec10e1e0 2024-04-14T15:29:16.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.412+0000 7fa6f447e700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa6ec1a9d60 con 0x7fa6ec10a610 2024-04-14T15:29:16.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.412+0000 7fa6f221a700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fa6ec107760 0x7fa6ec10d9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:37516/0 (socket says 172.21.15.90:37516) 2024-04-14T15:29:16.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.412+0000 7fa6f221a700 1 -- 172.21.15.90:0/219640531 learned_addr learned my addr 172.21.15.90:0/219640531 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:29:16.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.412+0000 7fa6e37fe700 10 client.?.objecter ms_handle_connect 0x7fa6ec107760 2024-04-14T15:29:16.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.413+0000 7fa6e37fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:29:16.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.413+0000 7fa6e37fe700 10 client.?.objecter ms_handle_connect 0x7fa6ec10a610 2024-04-14T15:29:16.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.413+0000 7fa6e37fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:29:16.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.413+0000 7fa6e37fe700 1 -- 172.21.15.90:0/219640531 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3342728685 0 0) 0x7fa6ec1a78e0 con 0x7fa6ec107760 2024-04-14T15:29:16.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.413+0000 7fa6e37fe700 1 -- 172.21.15.90:0/219640531 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa6d80039c0 con 0x7fa6ec107760 2024-04-14T15:29:16.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.413+0000 7fa6e37fe700 10 client.?.objecter ms_handle_connect 0x7fa6ec10e1e0 2024-04-14T15:29:16.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.413+0000 7fa6e37fe700 10 client.?.objecter resend_mon_ops 2024-04-14T15:29:16.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.413+0000 7fa6e37fe700 1 -- 172.21.15.90:0/219640531 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3578734493 0 0) 0x7fa6ec1a9d60 con 0x7fa6ec10a610 2024-04-14T15:29:16.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.413+0000 7fa6e37fe700 1 -- 172.21.15.90:0/219640531 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa6ec1a78e0 con 0x7fa6ec10a610 2024-04-14T15:29:16.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.413+0000 7fa6e37fe700 1 -- 172.21.15.90:0/219640531 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3500472428 0 0) 0x7fa6ec1a8b20 con 0x7fa6ec10e1e0 2024-04-14T15:29:16.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.413+0000 7fa6e37fe700 1 -- 172.21.15.90:0/219640531 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa6ec1a9d60 con 0x7fa6ec10e1e0 2024-04-14T15:29:16.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.413+0000 7fa6e37fe700 1 -- 172.21.15.90:0/219640531 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 848258482 0 0) 0x7fa6d80039c0 con 0x7fa6ec107760 2024-04-14T15:29:16.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.413+0000 7fa6e37fe700 1 -- 172.21.15.90:0/219640531 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa6ec1a8b20 con 0x7fa6ec107760 2024-04-14T15:29:16.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.413+0000 7fa6e37fe700 1 -- 172.21.15.90:0/219640531 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fa6dc001930 con 0x7fa6ec107760 2024-04-14T15:29:16.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.413+0000 7fa6e37fe700 1 -- 172.21.15.90:0/219640531 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3691472836 0 0) 0x7fa6ec1a78e0 con 0x7fa6ec10a610 2024-04-14T15:29:16.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.413+0000 7fa6e37fe700 1 -- 172.21.15.90:0/219640531 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa6d80039c0 con 0x7fa6ec10a610 2024-04-14T15:29:16.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.413+0000 7fa6e37fe700 1 -- 172.21.15.90:0/219640531 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fa6e8001840 con 0x7fa6ec10a610 2024-04-14T15:29:16.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.413+0000 7fa6e37fe700 1 -- 172.21.15.90:0/219640531 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3797868212 0 0) 0x7fa6ec1a9d60 con 0x7fa6ec10e1e0 2024-04-14T15:29:16.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.413+0000 7fa6e37fe700 1 -- 172.21.15.90:0/219640531 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa6ec1a78e0 con 0x7fa6ec10e1e0 2024-04-14T15:29:16.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.413+0000 7fa6e37fe700 1 -- 172.21.15.90:0/219640531 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fa6e40049a0 con 0x7fa6ec10e1e0 2024-04-14T15:29:16.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.413+0000 7fa6e37fe700 1 -- 172.21.15.90:0/219640531 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1534611296 0 0) 0x7fa6ec1a8b20 con 0x7fa6ec107760 2024-04-14T15:29:16.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.413+0000 7fa6e37fe700 1 -- 172.21.15.90:0/219640531 >> v1:172.21.15.90:6790/0 conn(0x7fa6ec10a610 legacy=0x7fa6ec1a26a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:29:16.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.413+0000 7fa6e37fe700 1 -- 172.21.15.90:0/219640531 >> v1:172.21.15.92:6789/0 conn(0x7fa6ec10e1e0 legacy=0x7fa6ec1a5fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:29:16.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.413+0000 7fa6e37fe700 1 -- 172.21.15.90:0/219640531 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6ec1aafa0 con 0x7fa6ec107760 2024-04-14T15:29:16.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.413+0000 7fa6f447e700 1 -- 172.21.15.90:0/219640531 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa6ec1a9f90 con 0x7fa6ec107760 2024-04-14T15:29:16.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.413+0000 7fa6f447e700 10 client.38415.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:29:16.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.413+0000 7fa6f447e700 1 -- 172.21.15.90:0/219640531 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa6ec1aa450 con 0x7fa6ec107760 2024-04-14T15:29:16.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.413+0000 7fa6f447e700 1 librados: init done 2024-04-14T15:29:16.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.413+0000 7fa6e37fe700 1 -- 172.21.15.90:0/219640531 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fa6dc001d00 con 0x7fa6ec107760 2024-04-14T15:29:16.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.413+0000 7fa6e37fe700 1 -- 172.21.15.90:0/219640531 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fa6dc005d70 con 0x7fa6ec107760 2024-04-14T15:29:16.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.414+0000 7fa6f447e700 1 -- 172.21.15.90:0/219640531 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6d0005150 con 0x7fa6ec107760 2024-04-14T15:29:16.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.415+0000 7fa6e37fe700 1 -- 172.21.15.90:0/219640531 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fa6dc003950 con 0x7fa6ec107760 2024-04-14T15:29:16.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.415+0000 7fa6e37fe700 1 -- 172.21.15.90:0/219640531 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(924..924 src has 467..924) ==== 7517+0+0 (unknown 2074792842 0 0) 0x7fa6dc08a590 con 0x7fa6ec107760 2024-04-14T15:29:16.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.415+0000 7fa6e37fe700 10 client.38415.objecter ms_dispatch 0x7fa6ec001550 osd_map(924..924 src has 467..924) 2024-04-14T15:29:16.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.415+0000 7fa6e37fe700 3 client.38415.objecter handle_osd_map got epochs [924,924] > 0 2024-04-14T15:29:16.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.415+0000 7fa6e37fe700 3 client.38415.objecter handle_osd_map decoding full epoch 924 2024-04-14T15:29:16.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.416+0000 7fa6e37fe700 20 client.38415.objecter dump_active .. 0 homeless 2024-04-14T15:29:16.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.416+0000 7fa6e37fe700 10 client.38415.objecter ms_handle_connect 0x7fa6d80712d0 2024-04-14T15:29:16.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.420+0000 7fa6e37fe700 1 -- 172.21.15.90:0/219640531 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fa6dc057370 con 0x7fa6ec107760 2024-04-14T15:29:16.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.612+0000 7fa6f447e700 1 -- 172.21.15.90:0/219640531 --> v1:172.21.15.90:6800/420935427 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa6d0000bc0 con 0x7fa6d80712d0 2024-04-14T15:29:16.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.624+0000 7fa6e37fe700 1 -- 172.21.15.90:0/219640531 <== mgr.14610 v1:172.21.15.90:6800/420935427 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+453560 (unknown 2965378022 0 4277126494) 0x7fa6d0000bc0 con 0x7fa6d80712d0 2024-04-14T15:29:16.633 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:29:16.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.629+0000 7fa6f447e700 10 librados: watch_flush enter 2024-04-14T15:29:16.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.629+0000 7fa6f447e700 10 librados: watch_flush exit 2024-04-14T15:29:16.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.629+0000 7fa6f447e700 20 client.38415.objecter shutdown clearing up homeless session... 2024-04-14T15:29:16.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.629+0000 7fa6f447e700 10 client.38415.objecter successfully canceled tick 2024-04-14T15:29:16.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.629+0000 7fa6f447e700 1 -- 172.21.15.90:0/219640531 >> v1:172.21.15.90:6800/420935427 conn(0x7fa6d80712d0 legacy=0x7fa6d8073790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:29:16.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.629+0000 7fa6f447e700 1 -- 172.21.15.90:0/219640531 >> v1:172.21.15.90:6789/0 conn(0x7fa6ec107760 legacy=0x7fa6ec10d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:29:16.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.629+0000 7fa6f447e700 1 -- 172.21.15.90:0/219640531 shutdown_connections 2024-04-14T15:29:16.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.629+0000 7fa6f447e700 1 -- 172.21.15.90:0/219640531 >> 172.21.15.90:0/219640531 conn(0x7fa6ec0fd290 msgr2=0x7fa6ec111420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:29:16.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.629+0000 7fa6f447e700 1 -- 172.21.15.90:0/219640531 shutdown_connections 2024-04-14T15:29:16.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.629+0000 7fa6f447e700 1 -- 172.21.15.90:0/219640531 wait complete. 2024-04-14T15:29:16.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:16.629+0000 7fa6f447e700 1 librados: shutdown 2024-04-14T15:29:16.639 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-14T15:29:16.691 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:16 smithi090 ceph-mon[41670]: 5.a deep-scrub starts 2024-04-14T15:29:16.691 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:16 smithi090 ceph-mon[41670]: 5.a deep-scrub ok 2024-04-14T15:29:16.691 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:16 smithi090 ceph-mon[41670]: pgmap v1414: 174 pgs: 174 active+clean; 582 KiB data, 844 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:29:16.691 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:16 smithi090 ceph-mon[41670]: 11.14 scrub starts 2024-04-14T15:29:16.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:16 smithi090 ceph-mon[41670]: 11.14 scrub ok 2024-04-14T15:29:16.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:16 smithi090 ceph-mon[41670]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:29:16.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:16 smithi090 ceph-mon[30399]: 5.a deep-scrub starts 2024-04-14T15:29:16.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:16 smithi090 ceph-mon[30399]: 5.a deep-scrub ok 2024-04-14T15:29:16.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:16 smithi090 ceph-mon[30399]: pgmap v1414: 174 pgs: 174 active+clean; 582 KiB data, 844 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:29:16.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:16 smithi090 ceph-mon[30399]: 11.14 scrub starts 2024-04-14T15:29:16.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:16 smithi090 ceph-mon[30399]: 11.14 scrub ok 2024-04-14T15:29:16.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:16 smithi090 ceph-mon[30399]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:29:16.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:16 smithi092 ceph-mon[33966]: 5.a deep-scrub starts 2024-04-14T15:29:16.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:16 smithi092 ceph-mon[33966]: 5.a deep-scrub ok 2024-04-14T15:29:16.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:16 smithi092 ceph-mon[33966]: pgmap v1414: 174 pgs: 174 active+clean; 582 KiB data, 844 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:29:16.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:16 smithi092 ceph-mon[33966]: 11.14 scrub starts 2024-04-14T15:29:16.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:16 smithi092 ceph-mon[33966]: 11.14 scrub ok 2024-04-14T15:29:16.873 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:16 smithi092 ceph-mon[33966]: from='client.24454 v1:172.21.15.92:0/240069196' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:29:17.464 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":1414,"stamp":"2024-04-14T15:29:15.254705+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596397,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":2309,"num_read_kb":2061,"num_write":729,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":169,"num_bytes_recovered":332641,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":922,"ondisk_log_size":922,"up":522,"acting":522,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":522,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":864120,"kb_used_data":66280,"kb_used_omap":12,"kb_used_meta":797683,"kb_avail":749097096,"statfs":{"total":767960285184,"available":767075426304,"internally_reserved":0,"allocated":67870720,"data_stored":35921191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":816827954},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001513"},"pg_stats":[{"pgid":"11.7","version":"0'0","reported_seq":561,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.157772+0000","last_change":"2024-04-14T15:28:12.845739+0000","last_active":"2024-04-14T15:29:11.157772+0000","last_peered":"2024-04-14T15:29:11.157772+0000","last_clean":"2024-04-14T15:29:11.157772+0000","last_became_active":"2024-04-14T15:26:27.058780+0000","last_became_peered":"2024-04-14T15:26:27.058780+0000","last_unstale":"2024-04-14T15:29:11.157772+0000","last_undegraded":"2024-04-14T15:29:11.157772+0000","last_fullsized":"2024-04-14T15:29:11.157772+0000","mapping_epoch":828,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":829,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:12.845555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:28:12.845555+0000","last_clean_scrub_stamp":"2024-04-14T15:28:12.845555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:15.174430+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"866'11","reported_seq":2052,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.113906+0000","last_change":"2024-04-14T15:28:54.104759+0000","last_active":"2024-04-14T15:29:11.113906+0000","last_peered":"2024-04-14T15:29:11.113906+0000","last_clean":"2024-04-14T15:29:11.113906+0000","last_became_active":"2024-04-14T15:27:14.215386+0000","last_became_peered":"2024-04-14T15:27:14.215386+0000","last_unstale":"2024-04-14T15:29:11.113906+0000","last_undegraded":"2024-04-14T15:29:11.113906+0000","last_fullsized":"2024-04-14T15:29:11.113906+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"866'11","last_scrub_stamp":"2024-04-14T15:28:54.104637+0000","last_deep_scrub":"85'9","last_deep_scrub_stamp":"2024-04-14T15:10:51.950580+0000","last_clean_scrub_stamp":"2024-04-14T15:28:54.104637+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:59.149921+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"479'34","reported_seq":720,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.155840+0000","last_change":"2024-04-14T15:27:55.858026+0000","last_active":"2024-04-14T15:29:11.155840+0000","last_peered":"2024-04-14T15:29:11.155840+0000","last_clean":"2024-04-14T15:29:11.155840+0000","last_became_active":"2024-04-14T15:27:14.219419+0000","last_became_peered":"2024-04-14T15:27:14.219419+0000","last_unstale":"2024-04-14T15:29:11.155840+0000","last_undegraded":"2024-04-14T15:29:11.155840+0000","last_fullsized":"2024-04-14T15:29:11.155840+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'34","last_scrub_stamp":"2024-04-14T15:27:55.857868+0000","last_deep_scrub":"430'22","last_deep_scrub_stamp":"2024-04-14T15:18:39.704883+0000","last_clean_scrub_stamp":"2024-04-14T15:27:55.857868+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:17.371799+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":1800,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.106132+0000","last_change":"2024-04-14T15:29:03.215761+0000","last_active":"2024-04-14T15:29:11.106132+0000","last_peered":"2024-04-14T15:29:11.106132+0000","last_clean":"2024-04-14T15:29:11.106132+0000","last_became_active":"2024-04-14T15:09:43.021460+0000","last_became_peered":"2024-04-14T15:09:43.021460+0000","last_unstale":"2024-04-14T15:29:11.106132+0000","last_undegraded":"2024-04-14T15:29:11.106132+0000","last_fullsized":"2024-04-14T15:29:11.106132+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:29:03.215687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:29:03.215687+0000","last_clean_scrub_stamp":"2024-04-14T15:29:03.215687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:14.584017+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":658,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.156260+0000","last_change":"2024-04-14T15:28:06.889444+0000","last_active":"2024-04-14T15:29:11.156260+0000","last_peered":"2024-04-14T15:29:11.156260+0000","last_clean":"2024-04-14T15:29:11.156260+0000","last_became_active":"2024-04-14T15:27:14.208471+0000","last_became_peered":"2024-04-14T15:27:14.208471+0000","last_unstale":"2024-04-14T15:29:11.156260+0000","last_undegraded":"2024-04-14T15:29:11.156260+0000","last_fullsized":"2024-04-14T15:29:11.156260+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:06.889313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:15:40.368219+0000","last_clean_scrub_stamp":"2024-04-14T15:28:06.889313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":1697,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.114285+0000","last_change":"2024-04-14T15:28:08.911776+0000","last_active":"2024-04-14T15:29:11.114285+0000","last_peered":"2024-04-14T15:29:11.114285+0000","last_clean":"2024-04-14T15:29:11.114285+0000","last_became_active":"2024-04-14T15:27:14.202698+0000","last_became_peered":"2024-04-14T15:27:14.202698+0000","last_unstale":"2024-04-14T15:29:11.114285+0000","last_undegraded":"2024-04-14T15:29:11.114285+0000","last_fullsized":"2024-04-14T15:29:11.114285+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:08.911665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:28:08.911665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:29.103837+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"479'15","reported_seq":1819,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.079537+0000","last_change":"2024-04-14T15:28:40.107670+0000","last_active":"2024-04-14T15:29:11.079537+0000","last_peered":"2024-04-14T15:29:11.079537+0000","last_clean":"2024-04-14T15:29:11.079537+0000","last_became_active":"2024-04-14T15:09:45.037179+0000","last_became_peered":"2024-04-14T15:09:45.037179+0000","last_unstale":"2024-04-14T15:29:11.079537+0000","last_undegraded":"2024-04-14T15:29:11.079537+0000","last_fullsized":"2024-04-14T15:29:11.079537+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'15","last_scrub_stamp":"2024-04-14T15:28:40.107560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:28:40.107560+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:55.659006+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":1793,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.079931+0000","last_change":"2024-04-14T15:27:54.031538+0000","last_active":"2024-04-14T15:29:11.079931+0000","last_peered":"2024-04-14T15:29:11.079931+0000","last_clean":"2024-04-14T15:29:11.079931+0000","last_became_active":"2024-04-14T15:09:43.020044+0000","last_became_peered":"2024-04-14T15:09:43.020044+0000","last_unstale":"2024-04-14T15:29:11.079931+0000","last_undegraded":"2024-04-14T15:29:11.079931+0000","last_fullsized":"2024-04-14T15:29:11.079931+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:27:54.031426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:27:54.031426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:14.053747+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1792,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.105275+0000","last_change":"2024-04-14T15:28:01.328416+0000","last_active":"2024-04-14T15:29:11.105275+0000","last_peered":"2024-04-14T15:29:11.105275+0000","last_clean":"2024-04-14T15:29:11.105275+0000","last_became_active":"2024-04-14T15:09:47.046153+0000","last_became_peered":"2024-04-14T15:09:47.046153+0000","last_unstale":"2024-04-14T15:29:11.105275+0000","last_undegraded":"2024-04-14T15:29:11.105275+0000","last_fullsized":"2024-04-14T15:29:11.105275+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:01.328334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:28:01.328334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1786,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.115514+0000","last_change":"2024-04-14T15:29:11.115555+0000","last_active":"2024-04-14T15:29:11.115514+0000","last_peered":"2024-04-14T15:29:11.115514+0000","last_clean":"2024-04-14T15:29:11.115514+0000","last_became_active":"2024-04-14T15:09:49.061495+0000","last_became_peered":"2024-04-14T15:09:49.061495+0000","last_unstale":"2024-04-14T15:29:11.115514+0000","last_undegraded":"2024-04-14T15:29:11.115514+0000","last_fullsized":"2024-04-14T15:29:11.115514+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:29:11.115418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:29:11.115418+0000","last_clean_scrub_stamp":"2024-04-14T15:29:11.115418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:37.190866+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":1705,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.271406+0000","last_change":"2024-04-14T15:29:11.271444+0000","last_active":"2024-04-14T15:29:11.271406+0000","last_peered":"2024-04-14T15:29:11.271406+0000","last_clean":"2024-04-14T15:29:11.271406+0000","last_became_active":"2024-04-14T15:27:14.202915+0000","last_became_peered":"2024-04-14T15:27:14.202915+0000","last_unstale":"2024-04-14T15:29:11.271406+0000","last_undegraded":"2024-04-14T15:29:11.271406+0000","last_fullsized":"2024-04-14T15:29:11.271406+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:29:11.271313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:29:11.271313+0000","last_clean_scrub_stamp":"2024-04-14T15:29:11.271313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:17.938909+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"479'44","reported_seq":154,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.157090+0000","last_change":"2024-04-14T15:28:20.450354+0000","last_active":"2024-04-14T15:29:11.157090+0000","last_peered":"2024-04-14T15:29:11.157090+0000","last_clean":"2024-04-14T15:29:11.157090+0000","last_became_active":"2024-04-14T15:27:14.210250+0000","last_became_peered":"2024-04-14T15:27:14.210250+0000","last_unstale":"2024-04-14T15:29:11.157090+0000","last_undegraded":"2024-04-14T15:29:11.157090+0000","last_fullsized":"2024-04-14T15:29:11.157090+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'44","last_scrub_stamp":"2024-04-14T15:28:20.450127+0000","last_deep_scrub":"479'44","last_deep_scrub_stamp":"2024-04-14T15:28:20.450127+0000","last_clean_scrub_stamp":"2024-04-14T15:28:20.450127+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:34.892050+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":1808,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.106187+0000","last_change":"2024-04-14T15:28:34.329860+0000","last_active":"2024-04-14T15:29:11.106187+0000","last_peered":"2024-04-14T15:29:11.106187+0000","last_clean":"2024-04-14T15:29:11.106187+0000","last_became_active":"2024-04-14T15:09:43.020268+0000","last_became_peered":"2024-04-14T15:09:43.020268+0000","last_unstale":"2024-04-14T15:29:11.106187+0000","last_undegraded":"2024-04-14T15:29:11.106187+0000","last_fullsized":"2024-04-14T15:29:11.106187+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:34.329762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:34.342793+0000","last_clean_scrub_stamp":"2024-04-14T15:28:34.329762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:57.790997+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1792,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.105162+0000","last_change":"2024-04-14T15:29:07.104656+0000","last_active":"2024-04-14T15:29:11.105162+0000","last_peered":"2024-04-14T15:29:11.105162+0000","last_clean":"2024-04-14T15:29:11.105162+0000","last_became_active":"2024-04-14T15:09:47.045542+0000","last_became_peered":"2024-04-14T15:09:47.045542+0000","last_unstale":"2024-04-14T15:29:11.105162+0000","last_undegraded":"2024-04-14T15:29:11.105162+0000","last_fullsized":"2024-04-14T15:29:11.105162+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:29:07.104582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:25:56.701776+0000","last_clean_scrub_stamp":"2024-04-14T15:29:07.104582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:12.232801+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1776,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.114600+0000","last_change":"2024-04-14T15:28:28.926657+0000","last_active":"2024-04-14T15:29:11.114600+0000","last_peered":"2024-04-14T15:29:11.114600+0000","last_clean":"2024-04-14T15:29:11.114600+0000","last_became_active":"2024-04-14T15:09:49.065397+0000","last_became_peered":"2024-04-14T15:09:49.065397+0000","last_unstale":"2024-04-14T15:29:11.114600+0000","last_undegraded":"2024-04-14T15:29:11.114600+0000","last_fullsized":"2024-04-14T15:29:11.114600+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:28.926558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:28:28.926558+0000","last_clean_scrub_stamp":"2024-04-14T15:28:28.926558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:29.568517+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"91'130","reported_seq":744,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.155745+0000","last_change":"2024-04-14T15:28:21.800962+0000","last_active":"2024-04-14T15:29:11.155745+0000","last_peered":"2024-04-14T15:29:11.155745+0000","last_clean":"2024-04-14T15:29:11.155745+0000","last_became_active":"2024-04-14T15:26:27.064317+0000","last_became_peered":"2024-04-14T15:26:27.064317+0000","last_unstale":"2024-04-14T15:29:11.155745+0000","last_undegraded":"2024-04-14T15:29:11.155745+0000","last_fullsized":"2024-04-14T15:29:11.155745+0000","mapping_epoch":828,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":829,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:28:21.800809+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:28:21.800809+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":70,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:32.782432+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":1697,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.105294+0000","last_change":"2024-04-14T15:27:58.390592+0000","last_active":"2024-04-14T15:29:11.105294+0000","last_peered":"2024-04-14T15:29:11.105294+0000","last_clean":"2024-04-14T15:29:11.105294+0000","last_became_active":"2024-04-14T15:12:56.474322+0000","last_became_peered":"2024-04-14T15:12:56.474322+0000","last_unstale":"2024-04-14T15:29:11.105294+0000","last_undegraded":"2024-04-14T15:29:11.105294+0000","last_fullsized":"2024-04-14T15:29:11.105294+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:27:58.390511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:27:58.390511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:18.100730+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"780'39","reported_seq":1896,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.074758+0000","last_change":"2024-04-14T15:28:43.884357+0000","last_active":"2024-04-14T15:29:11.074758+0000","last_peered":"2024-04-14T15:29:11.074758+0000","last_clean":"2024-04-14T15:29:11.074758+0000","last_became_active":"2024-04-14T15:13:12.933907+0000","last_became_peered":"2024-04-14T15:13:12.933907+0000","last_unstale":"2024-04-14T15:29:11.074758+0000","last_undegraded":"2024-04-14T15:29:11.074758+0000","last_fullsized":"2024-04-14T15:29:11.074758+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"780'39","last_scrub_stamp":"2024-04-14T15:28:43.884242+0000","last_deep_scrub":"780'39","last_deep_scrub_stamp":"2024-04-14T15:25:24.296779+0000","last_clean_scrub_stamp":"2024-04-14T15:28:43.884242+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:52.036846+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":1788,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.114038+0000","last_change":"2024-04-14T15:28:38.020549+0000","last_active":"2024-04-14T15:29:11.114038+0000","last_peered":"2024-04-14T15:29:11.114038+0000","last_clean":"2024-04-14T15:29:11.114038+0000","last_became_active":"2024-04-14T15:09:43.022351+0000","last_became_peered":"2024-04-14T15:09:43.022351+0000","last_unstale":"2024-04-14T15:29:11.114038+0000","last_undegraded":"2024-04-14T15:29:11.114038+0000","last_fullsized":"2024-04-14T15:29:11.114038+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:38.020454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:28:38.020454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:56.012668+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1785,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.079340+0000","last_change":"2024-04-14T15:28:03.078744+0000","last_active":"2024-04-14T15:29:11.079340+0000","last_peered":"2024-04-14T15:29:11.079340+0000","last_clean":"2024-04-14T15:29:11.079340+0000","last_became_active":"2024-04-14T15:09:47.042810+0000","last_became_peered":"2024-04-14T15:09:47.042810+0000","last_unstale":"2024-04-14T15:29:11.079340+0000","last_undegraded":"2024-04-14T15:29:11.079340+0000","last_fullsized":"2024-04-14T15:29:11.079340+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:03.078658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:28:03.078658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:22.969772+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1788,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.106699+0000","last_change":"2024-04-14T15:28:22.289204+0000","last_active":"2024-04-14T15:29:11.106699+0000","last_peered":"2024-04-14T15:29:11.106699+0000","last_clean":"2024-04-14T15:29:11.106699+0000","last_became_active":"2024-04-14T15:09:49.069594+0000","last_became_peered":"2024-04-14T15:09:49.069594+0000","last_unstale":"2024-04-14T15:29:11.106699+0000","last_undegraded":"2024-04-14T15:29:11.106699+0000","last_fullsized":"2024-04-14T15:29:11.106699+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:22.289129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:28:22.289129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:47.280657+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"91'130","reported_seq":2159,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.113541+0000","last_change":"2024-04-14T15:28:21.923496+0000","last_active":"2024-04-14T15:29:11.113541+0000","last_peered":"2024-04-14T15:29:11.113541+0000","last_clean":"2024-04-14T15:29:11.113541+0000","last_became_active":"2024-04-14T15:21:09.020300+0000","last_became_peered":"2024-04-14T15:21:09.020300+0000","last_unstale":"2024-04-14T15:29:11.113541+0000","last_undegraded":"2024-04-14T15:29:11.113541+0000","last_fullsized":"2024-04-14T15:29:11.113541+0000","mapping_epoch":189,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":544,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:28:21.923396+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:28:21.923396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:24.264160+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":142,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.156357+0000","last_change":"2024-04-14T15:28:27.378322+0000","last_active":"2024-04-14T15:29:11.156357+0000","last_peered":"2024-04-14T15:29:11.156357+0000","last_clean":"2024-04-14T15:29:11.156357+0000","last_became_active":"2024-04-14T15:27:14.215727+0000","last_became_peered":"2024-04-14T15:27:14.215727+0000","last_unstale":"2024-04-14T15:29:11.156357+0000","last_undegraded":"2024-04-14T15:29:11.156357+0000","last_fullsized":"2024-04-14T15:29:11.156357+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:27.378216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:28:27.378216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:53.912712+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"479'25","reported_seq":1854,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.074121+0000","last_change":"2024-04-14T15:28:35.940759+0000","last_active":"2024-04-14T15:29:11.074121+0000","last_peered":"2024-04-14T15:29:11.074121+0000","last_clean":"2024-04-14T15:29:11.074121+0000","last_became_active":"2024-04-14T15:27:14.209858+0000","last_became_peered":"2024-04-14T15:27:14.209858+0000","last_unstale":"2024-04-14T15:29:11.074121+0000","last_undegraded":"2024-04-14T15:29:11.074121+0000","last_fullsized":"2024-04-14T15:29:11.074121+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'25","last_scrub_stamp":"2024-04-14T15:28:35.940648+0000","last_deep_scrub":"479'25","last_deep_scrub_stamp":"2024-04-14T15:20:03.571160+0000","last_clean_scrub_stamp":"2024-04-14T15:28:35.940648+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:37.185385+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":1802,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.078313+0000","last_change":"2024-04-14T15:28:11.178171+0000","last_active":"2024-04-14T15:29:11.078313+0000","last_peered":"2024-04-14T15:29:11.078313+0000","last_clean":"2024-04-14T15:29:11.078313+0000","last_became_active":"2024-04-14T15:09:43.018871+0000","last_became_peered":"2024-04-14T15:09:43.018871+0000","last_unstale":"2024-04-14T15:29:11.078313+0000","last_undegraded":"2024-04-14T15:29:11.078313+0000","last_fullsized":"2024-04-14T15:29:11.078313+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:11.178093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:28:11.178093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:34.848304+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"400'9","reported_seq":2036,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.105961+0000","last_change":"2024-04-14T15:28:39.336656+0000","last_active":"2024-04-14T15:29:11.105961+0000","last_peered":"2024-04-14T15:29:11.105961+0000","last_clean":"2024-04-14T15:29:11.105961+0000","last_became_active":"2024-04-14T15:09:47.044085+0000","last_became_peered":"2024-04-14T15:09:47.044085+0000","last_unstale":"2024-04-14T15:29:11.105961+0000","last_undegraded":"2024-04-14T15:29:11.105961+0000","last_fullsized":"2024-04-14T15:29:11.105961+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'9","last_scrub_stamp":"2024-04-14T15:28:39.336522+0000","last_deep_scrub":"400'9","last_deep_scrub_stamp":"2024-04-14T15:17:54.845881+0000","last_clean_scrub_stamp":"2024-04-14T15:28:39.336522+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:41.291234+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1775,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.074648+0000","last_change":"2024-04-14T15:28:47.954798+0000","last_active":"2024-04-14T15:29:11.074648+0000","last_peered":"2024-04-14T15:29:11.074648+0000","last_clean":"2024-04-14T15:29:11.074648+0000","last_became_active":"2024-04-14T15:09:49.065274+0000","last_became_peered":"2024-04-14T15:09:49.065274+0000","last_unstale":"2024-04-14T15:29:11.074648+0000","last_undegraded":"2024-04-14T15:29:11.074648+0000","last_fullsized":"2024-04-14T15:29:11.074648+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:47.954691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:18.588546+0000","last_clean_scrub_stamp":"2024-04-14T15:28:47.954691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:57.549046+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"91'130","reported_seq":757,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.156097+0000","last_change":"2024-04-14T15:28:13.866095+0000","last_active":"2024-04-14T15:29:11.156097+0000","last_peered":"2024-04-14T15:29:11.156097+0000","last_clean":"2024-04-14T15:29:11.156097+0000","last_became_active":"2024-04-14T15:27:17.056550+0000","last_became_peered":"2024-04-14T15:27:17.056550+0000","last_unstale":"2024-04-14T15:29:11.156097+0000","last_undegraded":"2024-04-14T15:29:11.156097+0000","last_fullsized":"2024-04-14T15:29:11.156097+0000","mapping_epoch":868,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":869,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:28:13.865978+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:28:13.865978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:15.536292+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":1699,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:12.170892+0000","last_change":"2024-04-14T15:29:12.170913+0000","last_active":"2024-04-14T15:29:12.170892+0000","last_peered":"2024-04-14T15:29:12.170892+0000","last_clean":"2024-04-14T15:29:12.170892+0000","last_became_active":"2024-04-14T15:12:56.487615+0000","last_became_peered":"2024-04-14T15:12:56.487615+0000","last_unstale":"2024-04-14T15:29:12.170892+0000","last_undegraded":"2024-04-14T15:29:12.170892+0000","last_fullsized":"2024-04-14T15:29:12.170892+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:29:12.170836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:29:12.170836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:16.691346+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"479'34","reported_seq":1847,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.080036+0000","last_change":"2024-04-14T15:28:41.336454+0000","last_active":"2024-04-14T15:29:11.080036+0000","last_peered":"2024-04-14T15:29:11.080036+0000","last_clean":"2024-04-14T15:29:11.080036+0000","last_became_active":"2024-04-14T15:09:45.033331+0000","last_became_peered":"2024-04-14T15:09:45.033331+0000","last_unstale":"2024-04-14T15:29:11.080036+0000","last_undegraded":"2024-04-14T15:29:11.080036+0000","last_fullsized":"2024-04-14T15:29:11.080036+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'34","last_scrub_stamp":"2024-04-14T15:28:41.336355+0000","last_deep_scrub":"479'34","last_deep_scrub_stamp":"2024-04-14T15:24:35.378379+0000","last_clean_scrub_stamp":"2024-04-14T15:28:41.336355+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:04.937189+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":1802,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.078041+0000","last_change":"2024-04-14T15:28:58.418841+0000","last_active":"2024-04-14T15:29:11.078041+0000","last_peered":"2024-04-14T15:29:11.078041+0000","last_clean":"2024-04-14T15:29:11.078041+0000","last_became_active":"2024-04-14T15:09:43.020878+0000","last_became_peered":"2024-04-14T15:09:43.020878+0000","last_unstale":"2024-04-14T15:29:11.078041+0000","last_undegraded":"2024-04-14T15:29:11.078041+0000","last_fullsized":"2024-04-14T15:29:11.078041+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:58.418758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:28:58.418758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:13.356881+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"72'8","reported_seq":2022,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.074650+0000","last_change":"2024-04-14T15:28:44.883771+0000","last_active":"2024-04-14T15:29:11.074650+0000","last_peered":"2024-04-14T15:29:11.074650+0000","last_clean":"2024-04-14T15:29:11.074650+0000","last_became_active":"2024-04-14T15:09:47.050962+0000","last_became_peered":"2024-04-14T15:09:47.050962+0000","last_unstale":"2024-04-14T15:29:11.074650+0000","last_undegraded":"2024-04-14T15:29:11.074650+0000","last_fullsized":"2024-04-14T15:29:11.074650+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:28:44.883665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:28:44.883665+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:50.216166+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":652,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.157804+0000","last_change":"2024-04-14T15:28:30.732025+0000","last_active":"2024-04-14T15:29:11.157804+0000","last_peered":"2024-04-14T15:29:11.157804+0000","last_clean":"2024-04-14T15:29:11.157804+0000","last_became_active":"2024-04-14T15:26:27.065649+0000","last_became_peered":"2024-04-14T15:26:27.065649+0000","last_unstale":"2024-04-14T15:29:11.157804+0000","last_undegraded":"2024-04-14T15:29:11.157804+0000","last_fullsized":"2024-04-14T15:29:11.157804+0000","mapping_epoch":828,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":829,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:30.731930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:27:23.822301+0000","last_clean_scrub_stamp":"2024-04-14T15:28:30.731930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:36.499768+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"233'8","reported_seq":138,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.157223+0000","last_change":"2024-04-14T15:27:59.549354+0000","last_active":"2024-04-14T15:29:11.157223+0000","last_peered":"2024-04-14T15:29:11.157223+0000","last_clean":"2024-04-14T15:29:11.157223+0000","last_became_active":"2024-04-14T15:27:14.216287+0000","last_became_peered":"2024-04-14T15:27:14.216287+0000","last_unstale":"2024-04-14T15:29:11.157223+0000","last_undegraded":"2024-04-14T15:29:11.157223+0000","last_fullsized":"2024-04-14T15:29:11.157223+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'8","last_scrub_stamp":"2024-04-14T15:27:59.549269+0000","last_deep_scrub":"233'8","last_deep_scrub_stamp":"2024-04-14T15:15:01.295800+0000","last_clean_scrub_stamp":"2024-04-14T15:27:59.549269+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":5,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"479'24","reported_seq":1863,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.074360+0000","last_change":"2024-04-14T15:28:58.793618+0000","last_active":"2024-04-14T15:29:11.074360+0000","last_peered":"2024-04-14T15:29:11.074360+0000","last_clean":"2024-04-14T15:29:11.074360+0000","last_became_active":"2024-04-14T15:27:14.223836+0000","last_became_peered":"2024-04-14T15:27:14.223836+0000","last_unstale":"2024-04-14T15:29:11.074360+0000","last_undegraded":"2024-04-14T15:29:11.074360+0000","last_fullsized":"2024-04-14T15:29:11.074360+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'24","last_scrub_stamp":"2024-04-14T15:28:58.793497+0000","last_deep_scrub":"85'16","last_deep_scrub_stamp":"2024-04-14T15:10:47.456169+0000","last_clean_scrub_stamp":"2024-04-14T15:28:58.793497+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:05.252137+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":1797,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.113686+0000","last_change":"2024-04-14T15:28:52.070960+0000","last_active":"2024-04-14T15:29:11.113686+0000","last_peered":"2024-04-14T15:29:11.113686+0000","last_clean":"2024-04-14T15:29:11.113686+0000","last_became_active":"2024-04-14T15:09:43.018601+0000","last_became_peered":"2024-04-14T15:09:43.018601+0000","last_unstale":"2024-04-14T15:29:11.113686+0000","last_undegraded":"2024-04-14T15:29:11.113686+0000","last_fullsized":"2024-04-14T15:29:11.113686+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:52.070853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:28:52.070853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:56.127035+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":749,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.158042+0000","last_change":"2024-04-14T15:29:02.030422+0000","last_active":"2024-04-14T15:29:11.158042+0000","last_peered":"2024-04-14T15:29:11.158042+0000","last_clean":"2024-04-14T15:29:11.158042+0000","last_became_active":"2024-04-14T15:27:14.208646+0000","last_became_peered":"2024-04-14T15:27:14.208646+0000","last_unstale":"2024-04-14T15:29:11.158042+0000","last_undegraded":"2024-04-14T15:29:11.158042+0000","last_fullsized":"2024-04-14T15:29:11.158042+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:29:02.030326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:27:39.778818+0000","last_clean_scrub_stamp":"2024-04-14T15:29:02.030326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:13.927765+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1467,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.156839+0000","last_change":"2024-04-14T15:28:26.061582+0000","last_active":"2024-04-14T15:29:11.156839+0000","last_peered":"2024-04-14T15:29:11.156839+0000","last_clean":"2024-04-14T15:29:11.156839+0000","last_became_active":"2024-04-14T15:27:14.198110+0000","last_became_peered":"2024-04-14T15:27:14.198110+0000","last_unstale":"2024-04-14T15:29:11.156839+0000","last_undegraded":"2024-04-14T15:29:11.156839+0000","last_fullsized":"2024-04-14T15:29:11.156839+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:26.061472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:57.214046+0000","last_clean_scrub_stamp":"2024-04-14T15:28:26.061472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:30.621101+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"91'130","reported_seq":1985,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.156607+0000","last_change":"2024-04-14T15:28:33.001416+0000","last_active":"2024-04-14T15:29:11.156607+0000","last_peered":"2024-04-14T15:29:11.156607+0000","last_clean":"2024-04-14T15:29:11.156607+0000","last_became_active":"2024-04-14T15:28:31.265380+0000","last_became_peered":"2024-04-14T15:28:31.265380+0000","last_unstale":"2024-04-14T15:29:11.156607+0000","last_undegraded":"2024-04-14T15:29:11.156607+0000","last_fullsized":"2024-04-14T15:29:11.156607+0000","mapping_epoch":189,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":915,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:28:33.001281+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:28:33.001281+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":180,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:59.279973+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":1692,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.074322+0000","last_change":"2024-04-14T15:28:55.825563+0000","last_active":"2024-04-14T15:29:11.074322+0000","last_peered":"2024-04-14T15:29:11.074322+0000","last_clean":"2024-04-14T15:29:11.074322+0000","last_became_active":"2024-04-14T15:12:56.427276+0000","last_became_peered":"2024-04-14T15:12:56.427276+0000","last_unstale":"2024-04-14T15:29:11.074322+0000","last_undegraded":"2024-04-14T15:29:11.074322+0000","last_fullsized":"2024-04-14T15:29:11.074322+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:55.825466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:28:55.825466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:05.614908+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"255'10","reported_seq":1671,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.156282+0000","last_change":"2024-04-14T15:29:08.730140+0000","last_active":"2024-04-14T15:29:11.156282+0000","last_peered":"2024-04-14T15:29:11.156282+0000","last_clean":"2024-04-14T15:29:11.156282+0000","last_became_active":"2024-04-14T15:15:21.873747+0000","last_became_peered":"2024-04-14T15:15:21.873747+0000","last_unstale":"2024-04-14T15:29:11.156282+0000","last_undegraded":"2024-04-14T15:29:11.156282+0000","last_fullsized":"2024-04-14T15:29:11.156282+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'10","last_scrub_stamp":"2024-04-14T15:29:08.730015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:29:08.730015+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:11.722437+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"479'28","reported_seq":1870,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.073633+0000","last_change":"2024-04-14T15:27:49.103747+0000","last_active":"2024-04-14T15:29:11.073633+0000","last_peered":"2024-04-14T15:29:11.073633+0000","last_clean":"2024-04-14T15:29:11.073633+0000","last_became_active":"2024-04-14T15:27:14.212370+0000","last_became_peered":"2024-04-14T15:27:14.212370+0000","last_unstale":"2024-04-14T15:29:11.073633+0000","last_undegraded":"2024-04-14T15:29:11.073633+0000","last_fullsized":"2024-04-14T15:29:11.073633+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'28","last_scrub_stamp":"2024-04-14T15:27:49.103597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:27:49.103597+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:12.659020+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":142,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.157329+0000","last_change":"2024-04-14T15:28:47.360501+0000","last_active":"2024-04-14T15:29:11.157329+0000","last_peered":"2024-04-14T15:29:11.157329+0000","last_clean":"2024-04-14T15:29:11.157329+0000","last_became_active":"2024-04-14T15:27:14.212559+0000","last_became_peered":"2024-04-14T15:27:14.212559+0000","last_unstale":"2024-04-14T15:29:11.157329+0000","last_undegraded":"2024-04-14T15:29:11.157329+0000","last_fullsized":"2024-04-14T15:29:11.157329+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:47.360399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:28:47.360399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:15.273902+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1789,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.114296+0000","last_change":"2024-04-14T15:28:36.036651+0000","last_active":"2024-04-14T15:29:11.114296+0000","last_peered":"2024-04-14T15:29:11.114296+0000","last_clean":"2024-04-14T15:29:11.114296+0000","last_became_active":"2024-04-14T15:27:14.203240+0000","last_became_peered":"2024-04-14T15:27:14.203240+0000","last_unstale":"2024-04-14T15:29:11.114296+0000","last_undegraded":"2024-04-14T15:29:11.114296+0000","last_fullsized":"2024-04-14T15:29:11.114296+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:36.036555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:28:36.036555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:51.099129+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"91'130","reported_seq":2002,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.074061+0000","last_change":"2024-04-14T15:28:27.900740+0000","last_active":"2024-04-14T15:29:11.074061+0000","last_peered":"2024-04-14T15:29:11.074061+0000","last_clean":"2024-04-14T15:29:11.074061+0000","last_became_active":"2024-04-14T15:28:27.070386+0000","last_became_peered":"2024-04-14T15:28:27.070386+0000","last_unstale":"2024-04-14T15:29:11.074061+0000","last_undegraded":"2024-04-14T15:29:11.074061+0000","last_fullsized":"2024-04-14T15:29:11.074061+0000","mapping_epoch":885,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":911,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:28:27.900623+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:28:27.900623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:55.303648+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":1693,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.114892+0000","last_change":"2024-04-14T15:28:50.045865+0000","last_active":"2024-04-14T15:29:11.114892+0000","last_peered":"2024-04-14T15:29:11.114892+0000","last_clean":"2024-04-14T15:29:11.114892+0000","last_became_active":"2024-04-14T15:12:56.355254+0000","last_became_peered":"2024-04-14T15:12:56.355254+0000","last_unstale":"2024-04-14T15:29:11.114892+0000","last_undegraded":"2024-04-14T15:29:11.114892+0000","last_fullsized":"2024-04-14T15:29:11.114892+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:50.045738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:28:50.045738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:11.373315+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1788,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:13.978186+0000","last_change":"2024-04-14T15:29:13.978226+0000","last_active":"2024-04-14T15:29:13.978186+0000","last_peered":"2024-04-14T15:29:13.978186+0000","last_clean":"2024-04-14T15:29:13.978186+0000","last_became_active":"2024-04-14T15:09:47.044610+0000","last_became_peered":"2024-04-14T15:09:47.044610+0000","last_unstale":"2024-04-14T15:29:13.978186+0000","last_undegraded":"2024-04-14T15:29:13.978186+0000","last_fullsized":"2024-04-14T15:29:13.978186+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:29:13.978110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:39.394655+0000","last_clean_scrub_stamp":"2024-04-14T15:29:13.978110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:18.575306+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"783'38","reported_seq":1522,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.156221+0000","last_change":"2024-04-14T15:27:56.097704+0000","last_active":"2024-04-14T15:29:11.156221+0000","last_peered":"2024-04-14T15:29:11.156221+0000","last_clean":"2024-04-14T15:29:11.156221+0000","last_became_active":"2024-04-14T15:15:21.880565+0000","last_became_peered":"2024-04-14T15:15:21.880565+0000","last_unstale":"2024-04-14T15:29:11.156221+0000","last_undegraded":"2024-04-14T15:29:11.156221+0000","last_fullsized":"2024-04-14T15:29:11.156221+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"783'38","last_scrub_stamp":"2024-04-14T15:27:56.097590+0000","last_deep_scrub":"783'38","last_deep_scrub_stamp":"2024-04-14T15:25:27.804604+0000","last_clean_scrub_stamp":"2024-04-14T15:27:56.097590+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:21.215439+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'2","reported_seq":1816,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.079535+0000","last_change":"2024-04-14T15:29:04.339731+0000","last_active":"2024-04-14T15:29:11.079535+0000","last_peered":"2024-04-14T15:29:11.079535+0000","last_clean":"2024-04-14T15:29:11.079535+0000","last_became_active":"2024-04-14T15:27:14.201768+0000","last_became_peered":"2024-04-14T15:27:14.201768+0000","last_unstale":"2024-04-14T15:29:11.079535+0000","last_undegraded":"2024-04-14T15:29:11.079535+0000","last_fullsized":"2024-04-14T15:29:11.079535+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'2","last_scrub_stamp":"2024-04-14T15:29:04.339618+0000","last_deep_scrub":"91'2","last_deep_scrub_stamp":"2024-04-14T15:29:04.339618+0000","last_clean_scrub_stamp":"2024-04-14T15:29:04.339618+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:04.958836+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1776,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.074152+0000","last_change":"2024-04-14T15:28:03.148237+0000","last_active":"2024-04-14T15:29:11.074152+0000","last_peered":"2024-04-14T15:29:11.074152+0000","last_clean":"2024-04-14T15:29:11.074152+0000","last_became_active":"2024-04-14T15:09:49.065684+0000","last_became_peered":"2024-04-14T15:09:49.065684+0000","last_unstale":"2024-04-14T15:29:11.074152+0000","last_undegraded":"2024-04-14T15:29:11.074152+0000","last_fullsized":"2024-04-14T15:29:11.074152+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:03.148131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:28:03.148131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:29.138106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"91'130","reported_seq":1260,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.157066+0000","last_change":"2024-04-14T15:28:23.216706+0000","last_active":"2024-04-14T15:29:11.157066+0000","last_peered":"2024-04-14T15:29:11.157066+0000","last_clean":"2024-04-14T15:29:11.157066+0000","last_became_active":"2024-04-14T15:28:23.216417+0000","last_became_peered":"2024-04-14T15:28:23.216417+0000","last_unstale":"2024-04-14T15:29:11.157066+0000","last_undegraded":"2024-04-14T15:29:11.157066+0000","last_fullsized":"2024-04-14T15:29:11.157066+0000","mapping_epoch":902,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":907,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:28:19.473664+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:28:19.473664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:44.858236+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":1698,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.079207+0000","last_change":"2024-04-14T15:28:07.147963+0000","last_active":"2024-04-14T15:29:11.079207+0000","last_peered":"2024-04-14T15:29:11.079207+0000","last_clean":"2024-04-14T15:29:11.079207+0000","last_became_active":"2024-04-14T15:12:56.402023+0000","last_became_peered":"2024-04-14T15:12:56.402023+0000","last_unstale":"2024-04-14T15:29:11.079207+0000","last_undegraded":"2024-04-14T15:29:11.079207+0000","last_fullsized":"2024-04-14T15:29:11.079207+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:07.147878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:28:07.147878+0000","last_clean_scrub_stamp":"2024-04-14T15:28:07.147878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:18.144875+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":134,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.156945+0000","last_change":"2024-04-14T15:28:07.409341+0000","last_active":"2024-04-14T15:29:11.156945+0000","last_peered":"2024-04-14T15:29:11.156945+0000","last_clean":"2024-04-14T15:29:11.156945+0000","last_became_active":"2024-04-14T15:27:14.217451+0000","last_became_peered":"2024-04-14T15:27:14.217451+0000","last_unstale":"2024-04-14T15:29:11.156945+0000","last_undegraded":"2024-04-14T15:29:11.156945+0000","last_fullsized":"2024-04-14T15:29:11.156945+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:07.409246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:43.898835+0000","last_clean_scrub_stamp":"2024-04-14T15:28:07.409246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:37.206670+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"479'37","reported_seq":1854,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.079650+0000","last_change":"2024-04-14T15:28:55.031635+0000","last_active":"2024-04-14T15:29:11.079650+0000","last_peered":"2024-04-14T15:29:11.079650+0000","last_clean":"2024-04-14T15:29:11.079650+0000","last_became_active":"2024-04-14T15:09:45.179182+0000","last_became_peered":"2024-04-14T15:09:45.179182+0000","last_unstale":"2024-04-14T15:29:11.079650+0000","last_undegraded":"2024-04-14T15:29:11.079650+0000","last_fullsized":"2024-04-14T15:29:11.079650+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'37","last_scrub_stamp":"2024-04-14T15:28:55.031193+0000","last_deep_scrub":"102'27","last_deep_scrub_stamp":"2024-04-14T15:17:46.924330+0000","last_clean_scrub_stamp":"2024-04-14T15:28:55.031193+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:17.142255+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1789,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.114437+0000","last_change":"2024-04-14T15:28:12.930413+0000","last_active":"2024-04-14T15:29:11.114437+0000","last_peered":"2024-04-14T15:29:11.114437+0000","last_clean":"2024-04-14T15:29:11.114437+0000","last_became_active":"2024-04-14T15:09:43.021006+0000","last_became_peered":"2024-04-14T15:09:43.021006+0000","last_unstale":"2024-04-14T15:29:11.114437+0000","last_undegraded":"2024-04-14T15:29:11.114437+0000","last_fullsized":"2024-04-14T15:29:11.114437+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:12.930256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:28:12.930256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:38.970260+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1794,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.078096+0000","last_change":"2024-04-14T15:28:15.143755+0000","last_active":"2024-04-14T15:29:11.078096+0000","last_peered":"2024-04-14T15:29:11.078096+0000","last_clean":"2024-04-14T15:29:11.078096+0000","last_became_active":"2024-04-14T15:27:14.214481+0000","last_became_peered":"2024-04-14T15:27:14.214481+0000","last_unstale":"2024-04-14T15:29:11.078096+0000","last_undegraded":"2024-04-14T15:29:11.078096+0000","last_fullsized":"2024-04-14T15:29:11.078096+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:15.143609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:56.206649+0000","last_clean_scrub_stamp":"2024-04-14T15:28:15.143609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:26.910186+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"91'130","reported_seq":1999,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.078584+0000","last_change":"2024-04-14T15:28:48.320798+0000","last_active":"2024-04-14T15:29:11.078584+0000","last_peered":"2024-04-14T15:29:11.078584+0000","last_clean":"2024-04-14T15:29:11.078584+0000","last_became_active":"2024-04-14T15:28:34.269901+0000","last_became_peered":"2024-04-14T15:28:34.269901+0000","last_unstale":"2024-04-14T15:29:11.078584+0000","last_undegraded":"2024-04-14T15:29:11.078584+0000","last_fullsized":"2024-04-14T15:29:11.078584+0000","mapping_epoch":868,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":918,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:28:48.320719+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:28:48.320719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:02.388943+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":1415,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.156178+0000","last_change":"2024-04-14T15:29:10.676661+0000","last_active":"2024-04-14T15:29:11.156178+0000","last_peered":"2024-04-14T15:29:11.156178+0000","last_clean":"2024-04-14T15:29:11.156178+0000","last_became_active":"2024-04-14T15:27:14.221543+0000","last_became_peered":"2024-04-14T15:27:14.221543+0000","last_unstale":"2024-04-14T15:29:11.156178+0000","last_undegraded":"2024-04-14T15:29:11.156178+0000","last_fullsized":"2024-04-14T15:29:11.156178+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:29:10.676568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:29:10.676568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:39.947682+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":664,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.158075+0000","last_change":"2024-04-14T15:29:01.026804+0000","last_active":"2024-04-14T15:29:11.158075+0000","last_peered":"2024-04-14T15:29:11.158075+0000","last_clean":"2024-04-14T15:29:11.158075+0000","last_became_active":"2024-04-14T15:26:27.069614+0000","last_became_peered":"2024-04-14T15:26:27.069614+0000","last_unstale":"2024-04-14T15:29:11.158075+0000","last_undegraded":"2024-04-14T15:29:11.158075+0000","last_fullsized":"2024-04-14T15:29:11.158075+0000","mapping_epoch":828,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":829,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:29:01.026699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:29:01.026699+0000","last_clean_scrub_stamp":"2024-04-14T15:29:01.026699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:01.576701+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"479'30","reported_seq":1829,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.079382+0000","last_change":"2024-04-14T15:28:19.148165+0000","last_active":"2024-04-14T15:29:11.079382+0000","last_peered":"2024-04-14T15:29:11.079382+0000","last_clean":"2024-04-14T15:29:11.079382+0000","last_became_active":"2024-04-14T15:09:45.037231+0000","last_became_peered":"2024-04-14T15:09:45.037231+0000","last_unstale":"2024-04-14T15:29:11.079382+0000","last_undegraded":"2024-04-14T15:29:11.079382+0000","last_fullsized":"2024-04-14T15:29:11.079382+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'30","last_scrub_stamp":"2024-04-14T15:28:19.148035+0000","last_deep_scrub":"182'18","last_deep_scrub_stamp":"2024-04-14T15:14:10.383906+0000","last_clean_scrub_stamp":"2024-04-14T15:28:19.148035+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:31.614165+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1797,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.113784+0000","last_change":"2024-04-14T15:28:43.043408+0000","last_active":"2024-04-14T15:29:11.113784+0000","last_peered":"2024-04-14T15:29:11.113784+0000","last_clean":"2024-04-14T15:29:11.113784+0000","last_became_active":"2024-04-14T15:09:43.025095+0000","last_became_peered":"2024-04-14T15:09:43.025095+0000","last_unstale":"2024-04-14T15:29:11.113784+0000","last_undegraded":"2024-04-14T15:29:11.113784+0000","last_fullsized":"2024-04-14T15:29:11.113784+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:43.043313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:29.563759+0000","last_clean_scrub_stamp":"2024-04-14T15:28:43.043313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:12.097806+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1798,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.078473+0000","last_change":"2024-04-14T15:27:55.181206+0000","last_active":"2024-04-14T15:29:11.078473+0000","last_peered":"2024-04-14T15:29:11.078473+0000","last_clean":"2024-04-14T15:29:11.078473+0000","last_became_active":"2024-04-14T15:09:43.018509+0000","last_became_peered":"2024-04-14T15:09:43.018509+0000","last_unstale":"2024-04-14T15:29:11.078473+0000","last_undegraded":"2024-04-14T15:29:11.078473+0000","last_fullsized":"2024-04-14T15:29:11.078473+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:27:55.181100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:27:55.181100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:14.715967+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0018592680000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"355'2","reported_seq":1784,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.079348+0000","last_change":"2024-04-14T15:28:19.234847+0000","last_active":"2024-04-14T15:29:11.079348+0000","last_peered":"2024-04-14T15:29:11.079348+0000","last_clean":"2024-04-14T15:29:11.079348+0000","last_became_active":"2024-04-14T15:09:49.065243+0000","last_became_peered":"2024-04-14T15:09:49.065243+0000","last_unstale":"2024-04-14T15:29:11.079348+0000","last_undegraded":"2024-04-14T15:29:11.079348+0000","last_fullsized":"2024-04-14T15:29:11.079348+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'2","last_scrub_stamp":"2024-04-14T15:28:19.234755+0000","last_deep_scrub":"355'2","last_deep_scrub_stamp":"2024-04-14T15:20:04.842881+0000","last_clean_scrub_stamp":"2024-04-14T15:28:19.234755+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:36.830840+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":1399,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.156505+0000","last_change":"2024-04-14T15:28:17.202846+0000","last_active":"2024-04-14T15:29:11.156505+0000","last_peered":"2024-04-14T15:29:11.156505+0000","last_clean":"2024-04-14T15:29:11.156505+0000","last_became_active":"2024-04-14T15:27:14.208395+0000","last_became_peered":"2024-04-14T15:27:14.208395+0000","last_unstale":"2024-04-14T15:29:11.156505+0000","last_undegraded":"2024-04-14T15:29:11.156505+0000","last_fullsized":"2024-04-14T15:29:11.156505+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:17.202749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:28:17.202749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:40.255641+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1781,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.114386+0000","last_change":"2024-04-14T15:28:22.929418+0000","last_active":"2024-04-14T15:29:11.114386+0000","last_peered":"2024-04-14T15:29:11.114386+0000","last_clean":"2024-04-14T15:29:11.114386+0000","last_became_active":"2024-04-14T15:09:47.047201+0000","last_became_peered":"2024-04-14T15:09:47.047201+0000","last_unstale":"2024-04-14T15:29:11.114386+0000","last_undegraded":"2024-04-14T15:29:11.114386+0000","last_fullsized":"2024-04-14T15:29:11.114386+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:22.929296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:28:22.929296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:46.019860+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"866'38","reported_seq":185,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.156729+0000","last_change":"2024-04-14T15:28:53.344212+0000","last_active":"2024-04-14T15:29:11.156729+0000","last_peered":"2024-04-14T15:29:11.156729+0000","last_clean":"2024-04-14T15:29:11.156729+0000","last_became_active":"2024-04-14T15:27:14.217810+0000","last_became_peered":"2024-04-14T15:27:14.217810+0000","last_unstale":"2024-04-14T15:29:11.156729+0000","last_undegraded":"2024-04-14T15:29:11.156729+0000","last_fullsized":"2024-04-14T15:29:11.156729+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"866'38","last_scrub_stamp":"2024-04-14T15:28:53.344054+0000","last_deep_scrub":"90'24","last_deep_scrub_stamp":"2024-04-14T15:11:25.120692+0000","last_clean_scrub_stamp":"2024-04-14T15:28:53.344054+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:00.744833+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"91'130","reported_seq":2195,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.113624+0000","last_change":"2024-04-14T15:29:00.150163+0000","last_active":"2024-04-14T15:29:11.113624+0000","last_peered":"2024-04-14T15:29:11.113624+0000","last_clean":"2024-04-14T15:29:11.113624+0000","last_became_active":"2024-04-14T15:27:49.066511+0000","last_became_peered":"2024-04-14T15:27:49.066511+0000","last_unstale":"2024-04-14T15:29:11.113624+0000","last_undegraded":"2024-04-14T15:29:11.113624+0000","last_fullsized":"2024-04-14T15:29:11.113624+0000","mapping_epoch":868,"log_start":"91'130","ondisk_log_start":"91'130","created":19,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:29:00.150065+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:29:00.150065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:14.325815+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":664,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.156291+0000","last_change":"2024-04-14T15:28:01.874863+0000","last_active":"2024-04-14T15:29:11.156291+0000","last_peered":"2024-04-14T15:29:11.156291+0000","last_clean":"2024-04-14T15:29:11.156291+0000","last_became_active":"2024-04-14T15:26:27.063604+0000","last_became_peered":"2024-04-14T15:26:27.063604+0000","last_unstale":"2024-04-14T15:29:11.156291+0000","last_undegraded":"2024-04-14T15:29:11.156291+0000","last_fullsized":"2024-04-14T15:29:11.156291+0000","mapping_epoch":828,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":829,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:01.874778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:47.627305+0000","last_clean_scrub_stamp":"2024-04-14T15:28:01.874778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:22.976776+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"813'3","reported_seq":1498,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.156511+0000","last_change":"2024-04-14T15:28:09.152712+0000","last_active":"2024-04-14T15:29:11.156511+0000","last_peered":"2024-04-14T15:29:11.156511+0000","last_clean":"2024-04-14T15:29:11.156511+0000","last_became_active":"2024-04-14T15:27:14.204710+0000","last_became_peered":"2024-04-14T15:27:14.204710+0000","last_unstale":"2024-04-14T15:29:11.156511+0000","last_undegraded":"2024-04-14T15:29:11.156511+0000","last_fullsized":"2024-04-14T15:29:11.156511+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"813'3","last_scrub_stamp":"2024-04-14T15:28:09.152624+0000","last_deep_scrub":"813'3","last_deep_scrub_stamp":"2024-04-14T15:26:03.783312+0000","last_clean_scrub_stamp":"2024-04-14T15:28:09.152624+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:35.077143+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0058046169999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":652,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.158148+0000","last_change":"2024-04-14T15:28:14.831607+0000","last_active":"2024-04-14T15:29:11.158148+0000","last_peered":"2024-04-14T15:29:11.158148+0000","last_clean":"2024-04-14T15:29:11.158148+0000","last_became_active":"2024-04-14T15:26:27.070066+0000","last_became_peered":"2024-04-14T15:26:27.070066+0000","last_unstale":"2024-04-14T15:29:11.158148+0000","last_undegraded":"2024-04-14T15:29:11.158148+0000","last_fullsized":"2024-04-14T15:29:11.158148+0000","mapping_epoch":828,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":829,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:14.831474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:28:14.831474+0000","last_clean_scrub_stamp":"2024-04-14T15:28:14.831474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:15.707833+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"91'130","reported_seq":2182,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.113685+0000","last_change":"2024-04-14T15:29:06.228442+0000","last_active":"2024-04-14T15:29:11.113685+0000","last_peered":"2024-04-14T15:29:11.113685+0000","last_clean":"2024-04-14T15:29:11.113685+0000","last_became_active":"2024-04-14T15:27:42.065597+0000","last_became_peered":"2024-04-14T15:27:42.065597+0000","last_unstale":"2024-04-14T15:29:11.113685+0000","last_undegraded":"2024-04-14T15:29:11.113685+0000","last_fullsized":"2024-04-14T15:29:11.113685+0000","mapping_epoch":189,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":881,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:29:06.228304+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:22:17.518666+0000","last_clean_scrub_stamp":"2024-04-14T15:29:06.228304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:28.572455+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":1399,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.156125+0000","last_change":"2024-04-14T15:27:45.105160+0000","last_active":"2024-04-14T15:29:11.156125+0000","last_peered":"2024-04-14T15:29:11.156125+0000","last_clean":"2024-04-14T15:29:11.156125+0000","last_became_active":"2024-04-14T15:27:14.219271+0000","last_became_peered":"2024-04-14T15:27:14.219271+0000","last_unstale":"2024-04-14T15:29:11.156125+0000","last_undegraded":"2024-04-14T15:29:11.156125+0000","last_fullsized":"2024-04-14T15:29:11.156125+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:27:45.105085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:27:45.105085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:12.192724+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1792,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.106384+0000","last_change":"2024-04-14T15:28:21.262124+0000","last_active":"2024-04-14T15:29:11.106384+0000","last_peered":"2024-04-14T15:29:11.106384+0000","last_clean":"2024-04-14T15:29:11.106384+0000","last_became_active":"2024-04-14T15:09:47.049457+0000","last_became_peered":"2024-04-14T15:09:47.049457+0000","last_unstale":"2024-04-14T15:29:11.106384+0000","last_undegraded":"2024-04-14T15:29:11.106384+0000","last_fullsized":"2024-04-14T15:29:11.106384+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:21.262044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:28:21.262044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:23.619818+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"479'22","reported_seq":1827,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.080076+0000","last_change":"2024-04-14T15:27:58.079511+0000","last_active":"2024-04-14T15:29:11.080076+0000","last_peered":"2024-04-14T15:29:11.080076+0000","last_clean":"2024-04-14T15:29:11.080076+0000","last_became_active":"2024-04-14T15:09:45.033883+0000","last_became_peered":"2024-04-14T15:09:45.033883+0000","last_unstale":"2024-04-14T15:29:11.080076+0000","last_undegraded":"2024-04-14T15:29:11.080076+0000","last_fullsized":"2024-04-14T15:29:11.080076+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'22","last_scrub_stamp":"2024-04-14T15:27:58.079432+0000","last_deep_scrub":"308'18","last_deep_scrub_stamp":"2024-04-14T15:16:16.054032+0000","last_clean_scrub_stamp":"2024-04-14T15:27:58.079432+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:12.083598+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1801,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.105493+0000","last_change":"2024-04-14T15:28:06.319060+0000","last_active":"2024-04-14T15:29:11.105493+0000","last_peered":"2024-04-14T15:29:11.105493+0000","last_clean":"2024-04-14T15:29:11.105493+0000","last_became_active":"2024-04-14T15:09:43.021188+0000","last_became_peered":"2024-04-14T15:09:43.021188+0000","last_unstale":"2024-04-14T15:29:11.105493+0000","last_undegraded":"2024-04-14T15:29:11.105493+0000","last_fullsized":"2024-04-14T15:29:11.105493+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:06.318929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:28:06.318929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:30.761647+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"877'6","reported_seq":2995,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:12.412511+0000","last_change":"2024-04-14T15:28:47.314231+0000","last_active":"2024-04-14T15:29:12.412511+0000","last_peered":"2024-04-14T15:29:12.412511+0000","last_clean":"2024-04-14T15:29:12.412511+0000","last_became_active":"2024-04-14T15:27:14.214611+0000","last_became_peered":"2024-04-14T15:27:14.214611+0000","last_unstale":"2024-04-14T15:29:12.412511+0000","last_undegraded":"2024-04-14T15:29:12.412511+0000","last_fullsized":"2024-04-14T15:29:12.412511+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"877'6","last_scrub_stamp":"2024-04-14T15:28:47.314142+0000","last_deep_scrub":"877'6","last_deep_scrub_stamp":"2024-04-14T15:27:39.046415+0000","last_clean_scrub_stamp":"2024-04-14T15:28:47.314142+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:14.535075+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00056542099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1160,"num_read_kb":1155,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1789,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.114463+0000","last_change":"2024-04-14T15:28:31.910164+0000","last_active":"2024-04-14T15:29:11.114463+0000","last_peered":"2024-04-14T15:29:11.114463+0000","last_clean":"2024-04-14T15:29:11.114463+0000","last_became_active":"2024-04-14T15:27:14.203653+0000","last_became_peered":"2024-04-14T15:27:14.203653+0000","last_unstale":"2024-04-14T15:29:11.114463+0000","last_undegraded":"2024-04-14T15:29:11.114463+0000","last_fullsized":"2024-04-14T15:29:11.114463+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:31.910065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:28:31.910065+0000","last_clean_scrub_stamp":"2024-04-14T15:28:31.910065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:32.255005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"91'130","reported_seq":754,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.155821+0000","last_change":"2024-04-14T15:28:19.822693+0000","last_active":"2024-04-14T15:29:11.155821+0000","last_peered":"2024-04-14T15:29:11.155821+0000","last_clean":"2024-04-14T15:29:11.155821+0000","last_became_active":"2024-04-14T15:27:45.079088+0000","last_became_peered":"2024-04-14T15:27:45.079088+0000","last_unstale":"2024-04-14T15:29:11.155821+0000","last_undegraded":"2024-04-14T15:29:11.155821+0000","last_fullsized":"2024-04-14T15:29:11.155821+0000","mapping_epoch":828,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":884,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:28:19.822578+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:28:19.822578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:43.536556+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":1395,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.156556+0000","last_change":"2024-04-14T15:29:06.758617+0000","last_active":"2024-04-14T15:29:11.156556+0000","last_peered":"2024-04-14T15:29:11.156556+0000","last_clean":"2024-04-14T15:29:11.156556+0000","last_became_active":"2024-04-14T15:15:21.883717+0000","last_became_peered":"2024-04-14T15:15:21.883717+0000","last_unstale":"2024-04-14T15:29:11.156556+0000","last_undegraded":"2024-04-14T15:29:11.156556+0000","last_fullsized":"2024-04-14T15:29:11.156556+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:29:06.758512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:29:06.758512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:30.459459+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1783,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.079421+0000","last_change":"2024-04-14T15:28:08.136495+0000","last_active":"2024-04-14T15:29:11.079421+0000","last_peered":"2024-04-14T15:29:11.079421+0000","last_clean":"2024-04-14T15:29:11.079421+0000","last_became_active":"2024-04-14T15:09:47.040185+0000","last_became_peered":"2024-04-14T15:09:47.040185+0000","last_unstale":"2024-04-14T15:29:11.079421+0000","last_undegraded":"2024-04-14T15:29:11.079421+0000","last_fullsized":"2024-04-14T15:29:11.079421+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:08.136413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:28:08.136413+0000","last_clean_scrub_stamp":"2024-04-14T15:28:08.136413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:25.895647+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"479'40","reported_seq":2091,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.079248+0000","last_change":"2024-04-14T15:28:10.156157+0000","last_active":"2024-04-14T15:29:11.079248+0000","last_peered":"2024-04-14T15:29:11.079248+0000","last_clean":"2024-04-14T15:29:11.079248+0000","last_became_active":"2024-04-14T15:09:45.034168+0000","last_became_peered":"2024-04-14T15:09:45.034168+0000","last_unstale":"2024-04-14T15:29:11.079248+0000","last_undegraded":"2024-04-14T15:29:11.079248+0000","last_fullsized":"2024-04-14T15:29:11.079248+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'40","last_scrub_stamp":"2024-04-14T15:28:10.156043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:28:10.156043+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:33.151101+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":751,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.157381+0000","last_change":"2024-04-14T15:28:28.708689+0000","last_active":"2024-04-14T15:29:11.157381+0000","last_peered":"2024-04-14T15:29:11.157381+0000","last_clean":"2024-04-14T15:29:11.157381+0000","last_became_active":"2024-04-14T15:27:14.208574+0000","last_became_peered":"2024-04-14T15:27:14.208574+0000","last_unstale":"2024-04-14T15:29:11.157381+0000","last_undegraded":"2024-04-14T15:29:11.157381+0000","last_fullsized":"2024-04-14T15:29:11.157381+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:28.708595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:24.066275+0000","last_clean_scrub_stamp":"2024-04-14T15:28:28.708595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:30.871718+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1447,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.157138+0000","last_change":"2024-04-14T15:28:32.032028+0000","last_active":"2024-04-14T15:29:11.157138+0000","last_peered":"2024-04-14T15:29:11.157138+0000","last_clean":"2024-04-14T15:29:11.157138+0000","last_became_active":"2024-04-14T15:15:21.874742+0000","last_became_peered":"2024-04-14T15:15:21.874742+0000","last_unstale":"2024-04-14T15:29:11.157138+0000","last_undegraded":"2024-04-14T15:29:11.157138+0000","last_fullsized":"2024-04-14T15:29:11.157138+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:32.031897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:30.949569+0000","last_clean_scrub_stamp":"2024-04-14T15:28:32.031897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:59.867935+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"91'130","reported_seq":1264,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.156966+0000","last_change":"2024-04-14T15:28:55.360495+0000","last_active":"2024-04-14T15:29:11.156966+0000","last_peered":"2024-04-14T15:29:11.156966+0000","last_clean":"2024-04-14T15:29:11.156966+0000","last_became_active":"2024-04-14T15:27:53.072668+0000","last_became_peered":"2024-04-14T15:27:53.072668+0000","last_unstale":"2024-04-14T15:29:11.156966+0000","last_undegraded":"2024-04-14T15:29:11.156966+0000","last_fullsized":"2024-04-14T15:29:11.156966+0000","mapping_epoch":868,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":892,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:28:55.360339+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:27:22.667430+0000","last_clean_scrub_stamp":"2024-04-14T15:28:55.360339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:56.446427+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":1709,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.106136+0000","last_change":"2024-04-14T15:28:55.329952+0000","last_active":"2024-04-14T15:29:11.106136+0000","last_peered":"2024-04-14T15:29:11.106136+0000","last_clean":"2024-04-14T15:29:11.106136+0000","last_became_active":"2024-04-14T15:27:14.209189+0000","last_became_peered":"2024-04-14T15:27:14.209189+0000","last_unstale":"2024-04-14T15:29:11.106136+0000","last_undegraded":"2024-04-14T15:29:11.106136+0000","last_fullsized":"2024-04-14T15:29:11.106136+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:55.329853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:28:55.329853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:24.260048+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"866'11","reported_seq":2055,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.079199+0000","last_change":"2024-04-14T15:28:39.146093+0000","last_active":"2024-04-14T15:29:11.079199+0000","last_peered":"2024-04-14T15:29:11.079199+0000","last_clean":"2024-04-14T15:29:11.079199+0000","last_became_active":"2024-04-14T15:27:14.214459+0000","last_became_peered":"2024-04-14T15:27:14.214459+0000","last_unstale":"2024-04-14T15:29:11.079199+0000","last_undegraded":"2024-04-14T15:29:11.079199+0000","last_fullsized":"2024-04-14T15:29:11.079199+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"866'11","last_scrub_stamp":"2024-04-14T15:28:39.145971+0000","last_deep_scrub":"866'11","last_deep_scrub_stamp":"2024-04-14T15:28:39.145971+0000","last_clean_scrub_stamp":"2024-04-14T15:28:39.145971+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:58.014412+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"479'20","reported_seq":804,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.077942+0000","last_change":"2024-04-14T15:28:42.304006+0000","last_active":"2024-04-14T15:29:11.077942+0000","last_peered":"2024-04-14T15:29:11.077942+0000","last_clean":"2024-04-14T15:29:11.077942+0000","last_became_active":"2024-04-14T15:21:16.034027+0000","last_became_peered":"2024-04-14T15:21:16.034027+0000","last_unstale":"2024-04-14T15:29:11.077942+0000","last_undegraded":"2024-04-14T15:29:11.077942+0000","last_fullsized":"2024-04-14T15:29:11.077942+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'20","last_scrub_stamp":"2024-04-14T15:28:42.303902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:28:42.303902+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:46.695839+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1794,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.079127+0000","last_change":"2024-04-14T15:28:51.325087+0000","last_active":"2024-04-14T15:29:11.079127+0000","last_peered":"2024-04-14T15:29:11.079127+0000","last_clean":"2024-04-14T15:29:11.079127+0000","last_became_active":"2024-04-14T15:09:43.022597+0000","last_became_peered":"2024-04-14T15:09:43.022597+0000","last_unstale":"2024-04-14T15:29:11.079127+0000","last_undegraded":"2024-04-14T15:29:11.079127+0000","last_fullsized":"2024-04-14T15:29:11.079127+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:51.325007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:21:28.684818+0000","last_clean_scrub_stamp":"2024-04-14T15:28:51.325007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:04.281666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1779,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.078217+0000","last_change":"2024-04-14T15:28:15.050502+0000","last_active":"2024-04-14T15:29:11.078217+0000","last_peered":"2024-04-14T15:29:11.078217+0000","last_clean":"2024-04-14T15:29:11.078217+0000","last_became_active":"2024-04-14T15:09:49.066648+0000","last_became_peered":"2024-04-14T15:09:49.066648+0000","last_unstale":"2024-04-14T15:29:11.078217+0000","last_undegraded":"2024-04-14T15:29:11.078217+0000","last_fullsized":"2024-04-14T15:29:11.078217+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:15.050409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:28:15.050409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:34.922604+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":1687,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.079078+0000","last_change":"2024-04-14T15:27:56.147263+0000","last_active":"2024-04-14T15:29:11.079078+0000","last_peered":"2024-04-14T15:29:11.079078+0000","last_clean":"2024-04-14T15:29:11.079078+0000","last_became_active":"2024-04-14T15:12:56.563074+0000","last_became_peered":"2024-04-14T15:12:56.563074+0000","last_unstale":"2024-04-14T15:29:11.079078+0000","last_undegraded":"2024-04-14T15:29:11.079078+0000","last_fullsized":"2024-04-14T15:29:11.079078+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:27:56.147149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:27:56.147149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:14.678500+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1792,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.104974+0000","last_change":"2024-04-14T15:28:52.268312+0000","last_active":"2024-04-14T15:29:11.104974+0000","last_peered":"2024-04-14T15:29:11.104974+0000","last_clean":"2024-04-14T15:29:11.104974+0000","last_became_active":"2024-04-14T15:09:47.049005+0000","last_became_peered":"2024-04-14T15:09:47.049005+0000","last_unstale":"2024-04-14T15:29:11.104974+0000","last_undegraded":"2024-04-14T15:29:11.104974+0000","last_fullsized":"2024-04-14T15:29:11.104974+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:52.268228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:28:52.268228+0000","last_clean_scrub_stamp":"2024-04-14T15:28:52.268228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:54.311348+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"479'23","reported_seq":1828,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.073892+0000","last_change":"2024-04-14T15:28:37.903985+0000","last_active":"2024-04-14T15:29:11.073892+0000","last_peered":"2024-04-14T15:29:11.073892+0000","last_clean":"2024-04-14T15:29:11.073892+0000","last_became_active":"2024-04-14T15:09:45.032892+0000","last_became_peered":"2024-04-14T15:09:45.032892+0000","last_unstale":"2024-04-14T15:29:11.073892+0000","last_undegraded":"2024-04-14T15:29:11.073892+0000","last_fullsized":"2024-04-14T15:29:11.073892+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'23","last_scrub_stamp":"2024-04-14T15:28:37.903878+0000","last_deep_scrub":"183'17","last_deep_scrub_stamp":"2024-04-14T15:14:11.577574+0000","last_clean_scrub_stamp":"2024-04-14T15:28:37.903878+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:41.067500+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"677'6","reported_seq":1808,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.074433+0000","last_change":"2024-04-14T15:28:00.173214+0000","last_active":"2024-04-14T15:29:11.074433+0000","last_peered":"2024-04-14T15:29:11.074433+0000","last_clean":"2024-04-14T15:29:11.074433+0000","last_became_active":"2024-04-14T15:09:43.020490+0000","last_became_peered":"2024-04-14T15:09:43.020490+0000","last_unstale":"2024-04-14T15:29:11.074433+0000","last_undegraded":"2024-04-14T15:29:11.074433+0000","last_fullsized":"2024-04-14T15:29:11.074433+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"677'6","last_scrub_stamp":"2024-04-14T15:28:00.173087+0000","last_deep_scrub":"677'6","last_deep_scrub_stamp":"2024-04-14T15:23:38.773539+0000","last_clean_scrub_stamp":"2024-04-14T15:28:00.173087+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:27.862693+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":528,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1447,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.156744+0000","last_change":"2024-04-14T15:28:10.108032+0000","last_active":"2024-04-14T15:29:11.156744+0000","last_peered":"2024-04-14T15:29:11.156744+0000","last_clean":"2024-04-14T15:29:11.156744+0000","last_became_active":"2024-04-14T15:15:21.876410+0000","last_became_peered":"2024-04-14T15:15:21.876410+0000","last_unstale":"2024-04-14T15:29:11.156744+0000","last_undegraded":"2024-04-14T15:29:11.156744+0000","last_fullsized":"2024-04-14T15:29:11.156744+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:10.107941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:28:10.107941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:17.879492+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":1694,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.078337+0000","last_change":"2024-04-14T15:28:05.171891+0000","last_active":"2024-04-14T15:29:11.078337+0000","last_peered":"2024-04-14T15:29:11.078337+0000","last_clean":"2024-04-14T15:29:11.078337+0000","last_became_active":"2024-04-14T15:12:56.551177+0000","last_became_peered":"2024-04-14T15:12:56.551177+0000","last_unstale":"2024-04-14T15:29:11.078337+0000","last_undegraded":"2024-04-14T15:29:11.078337+0000","last_fullsized":"2024-04-14T15:29:11.078337+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:05.171817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:22:18.299330+0000","last_clean_scrub_stamp":"2024-04-14T15:28:05.171817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:19.005597+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"479'19","reported_seq":1815,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.079433+0000","last_change":"2024-04-14T15:28:48.017377+0000","last_active":"2024-04-14T15:29:11.079433+0000","last_peered":"2024-04-14T15:29:11.079433+0000","last_clean":"2024-04-14T15:29:11.079433+0000","last_became_active":"2024-04-14T15:09:45.180375+0000","last_became_peered":"2024-04-14T15:09:45.180375+0000","last_unstale":"2024-04-14T15:29:11.079433+0000","last_undegraded":"2024-04-14T15:29:11.079433+0000","last_fullsized":"2024-04-14T15:29:11.079433+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'19","last_scrub_stamp":"2024-04-14T15:28:48.017271+0000","last_deep_scrub":"347'13","last_deep_scrub_stamp":"2024-04-14T15:16:55.694599+0000","last_clean_scrub_stamp":"2024-04-14T15:28:48.017271+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:02.800393+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":1800,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.104864+0000","last_change":"2024-04-14T15:28:35.345923+0000","last_active":"2024-04-14T15:29:11.104864+0000","last_peered":"2024-04-14T15:29:11.104864+0000","last_clean":"2024-04-14T15:29:11.104864+0000","last_became_active":"2024-04-14T15:09:43.016788+0000","last_became_peered":"2024-04-14T15:09:43.016788+0000","last_unstale":"2024-04-14T15:29:11.104864+0000","last_undegraded":"2024-04-14T15:29:11.104864+0000","last_fullsized":"2024-04-14T15:29:11.104864+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:35.345754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:28:35.345754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:03.361232+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1466,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.157184+0000","last_change":"2024-04-14T15:28:51.799879+0000","last_active":"2024-04-14T15:29:11.157184+0000","last_peered":"2024-04-14T15:29:11.157184+0000","last_clean":"2024-04-14T15:29:11.157184+0000","last_became_active":"2024-04-14T15:15:21.873580+0000","last_became_peered":"2024-04-14T15:15:21.873580+0000","last_unstale":"2024-04-14T15:29:11.157184+0000","last_undegraded":"2024-04-14T15:29:11.157184+0000","last_fullsized":"2024-04-14T15:29:11.157184+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:51.799799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:28:51.799799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:01.688930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1796,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:14.163761+0000","last_change":"2024-04-14T15:29:14.163785+0000","last_active":"2024-04-14T15:29:14.163761+0000","last_peered":"2024-04-14T15:29:14.163761+0000","last_clean":"2024-04-14T15:29:14.163761+0000","last_became_active":"2024-04-14T15:09:49.064296+0000","last_became_peered":"2024-04-14T15:09:49.064296+0000","last_unstale":"2024-04-14T15:29:14.163761+0000","last_undegraded":"2024-04-14T15:29:14.163761+0000","last_fullsized":"2024-04-14T15:29:14.163761+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:29:14.163695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:39.410037+0000","last_clean_scrub_stamp":"2024-04-14T15:29:14.163695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:16.153138+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":1708,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.170131+0000","last_change":"2024-04-14T15:29:11.170177+0000","last_active":"2024-04-14T15:29:11.170131+0000","last_peered":"2024-04-14T15:29:11.170131+0000","last_clean":"2024-04-14T15:29:11.170131+0000","last_became_active":"2024-04-14T15:27:14.211630+0000","last_became_peered":"2024-04-14T15:27:14.211630+0000","last_unstale":"2024-04-14T15:29:11.170131+0000","last_undegraded":"2024-04-14T15:29:11.170131+0000","last_fullsized":"2024-04-14T15:29:11.170131+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:29:11.170027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:27:45.332536+0000","last_clean_scrub_stamp":"2024-04-14T15:29:11.170027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:15.143992+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.17","version":"479'12","reported_seq":1804,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.157145+0000","last_change":"2024-04-14T15:28:16.177386+0000","last_active":"2024-04-14T15:29:11.157145+0000","last_peered":"2024-04-14T15:29:11.157145+0000","last_clean":"2024-04-14T15:29:11.157145+0000","last_became_active":"2024-04-14T15:09:45.180037+0000","last_became_peered":"2024-04-14T15:09:45.180037+0000","last_unstale":"2024-04-14T15:29:11.157145+0000","last_undegraded":"2024-04-14T15:29:11.157145+0000","last_fullsized":"2024-04-14T15:29:11.157145+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'12","last_scrub_stamp":"2024-04-14T15:28:16.177258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:28:16.177258+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:44.619912+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":134,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.156802+0000","last_change":"2024-04-14T15:28:05.482212+0000","last_active":"2024-04-14T15:29:11.156802+0000","last_peered":"2024-04-14T15:29:11.156802+0000","last_clean":"2024-04-14T15:29:11.156802+0000","last_became_active":"2024-04-14T15:27:14.222112+0000","last_became_peered":"2024-04-14T15:27:14.222112+0000","last_unstale":"2024-04-14T15:29:11.156802+0000","last_undegraded":"2024-04-14T15:29:11.156802+0000","last_fullsized":"2024-04-14T15:29:11.156802+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:05.482096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:59.640565+0000","last_clean_scrub_stamp":"2024-04-14T15:28:05.482096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:33.428508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":664,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.157879+0000","last_change":"2024-04-14T15:28:38.700585+0000","last_active":"2024-04-14T15:29:11.157879+0000","last_peered":"2024-04-14T15:29:11.157879+0000","last_clean":"2024-04-14T15:29:11.157879+0000","last_became_active":"2024-04-14T15:26:27.067715+0000","last_became_peered":"2024-04-14T15:26:27.067715+0000","last_unstale":"2024-04-14T15:29:11.157879+0000","last_undegraded":"2024-04-14T15:29:11.157879+0000","last_fullsized":"2024-04-14T15:29:11.157879+0000","mapping_epoch":828,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":829,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:38.700449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:13.735024+0000","last_clean_scrub_stamp":"2024-04-14T15:28:38.700449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:58.269929+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1788,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.114659+0000","last_change":"2024-04-14T15:28:46.035515+0000","last_active":"2024-04-14T15:29:11.114659+0000","last_peered":"2024-04-14T15:29:11.114659+0000","last_clean":"2024-04-14T15:29:11.114659+0000","last_became_active":"2024-04-14T15:27:14.215736+0000","last_became_peered":"2024-04-14T15:27:14.215736+0000","last_unstale":"2024-04-14T15:29:11.114659+0000","last_undegraded":"2024-04-14T15:29:11.114659+0000","last_fullsized":"2024-04-14T15:29:11.114659+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:46.035420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:28:46.035420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:03.586018+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":1699,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.073932+0000","last_change":"2024-04-14T15:28:56.784656+0000","last_active":"2024-04-14T15:29:11.073932+0000","last_peered":"2024-04-14T15:29:11.073932+0000","last_clean":"2024-04-14T15:29:11.073932+0000","last_became_active":"2024-04-14T15:12:56.203577+0000","last_became_peered":"2024-04-14T15:12:56.203577+0000","last_unstale":"2024-04-14T15:29:11.073932+0000","last_undegraded":"2024-04-14T15:29:11.073932+0000","last_fullsized":"2024-04-14T15:29:11.073932+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:56.784550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:46.369231+0000","last_clean_scrub_stamp":"2024-04-14T15:28:56.784550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:01.539197+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"479'20","reported_seq":1500,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.156872+0000","last_change":"2024-04-14T15:28:50.797133+0000","last_active":"2024-04-14T15:29:11.156872+0000","last_peered":"2024-04-14T15:29:11.156872+0000","last_clean":"2024-04-14T15:29:11.156872+0000","last_became_active":"2024-04-14T15:15:21.877667+0000","last_became_peered":"2024-04-14T15:15:21.877667+0000","last_unstale":"2024-04-14T15:29:11.156872+0000","last_undegraded":"2024-04-14T15:29:11.156872+0000","last_fullsized":"2024-04-14T15:29:11.156872+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'20","last_scrub_stamp":"2024-04-14T15:28:50.797032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:28:50.797032+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:02.284320+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":656,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.156394+0000","last_change":"2024-04-14T15:28:05.921075+0000","last_active":"2024-04-14T15:29:11.156394+0000","last_peered":"2024-04-14T15:29:11.156394+0000","last_clean":"2024-04-14T15:29:11.156394+0000","last_became_active":"2024-04-14T15:26:27.065734+0000","last_became_peered":"2024-04-14T15:26:27.065734+0000","last_unstale":"2024-04-14T15:29:11.156394+0000","last_undegraded":"2024-04-14T15:29:11.156394+0000","last_fullsized":"2024-04-14T15:29:11.156394+0000","mapping_epoch":828,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":829,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:05.920902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:28:05.920902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"829'10","reported_seq":779,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.157707+0000","last_change":"2024-04-14T15:29:03.040455+0000","last_active":"2024-04-14T15:29:11.157707+0000","last_peered":"2024-04-14T15:29:11.157707+0000","last_clean":"2024-04-14T15:29:11.157707+0000","last_became_active":"2024-04-14T15:26:27.065792+0000","last_became_peered":"2024-04-14T15:26:27.065792+0000","last_unstale":"2024-04-14T15:29:11.157707+0000","last_undegraded":"2024-04-14T15:29:11.157707+0000","last_fullsized":"2024-04-14T15:29:11.157707+0000","mapping_epoch":828,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":829,"parent":"0.0","parent_split_bits":0,"last_scrub":"829'10","last_scrub_stamp":"2024-04-14T15:29:03.040354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:29:03.040354+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:23.554431+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1778,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.079286+0000","last_change":"2024-04-14T15:29:06.080391+0000","last_active":"2024-04-14T15:29:11.079286+0000","last_peered":"2024-04-14T15:29:11.079286+0000","last_clean":"2024-04-14T15:29:11.079286+0000","last_became_active":"2024-04-14T15:09:49.066068+0000","last_became_peered":"2024-04-14T15:09:49.066068+0000","last_unstale":"2024-04-14T15:29:11.079286+0000","last_undegraded":"2024-04-14T15:29:11.079286+0000","last_fullsized":"2024-04-14T15:29:11.079286+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:29:06.080315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:54.724681+0000","last_clean_scrub_stamp":"2024-04-14T15:29:06.080315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:30.256693+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":569,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.157271+0000","last_change":"2024-04-14T15:28:56.946641+0000","last_active":"2024-04-14T15:29:11.157271+0000","last_peered":"2024-04-14T15:29:11.157271+0000","last_clean":"2024-04-14T15:29:11.157271+0000","last_became_active":"2024-04-14T15:26:27.068033+0000","last_became_peered":"2024-04-14T15:26:27.068033+0000","last_unstale":"2024-04-14T15:29:11.157271+0000","last_undegraded":"2024-04-14T15:29:11.157271+0000","last_fullsized":"2024-04-14T15:29:11.157271+0000","mapping_epoch":828,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":829,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:56.946545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:28:56.946545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:24.320804+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"479'19","reported_seq":925,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.157601+0000","last_change":"2024-04-14T15:28:47.826008+0000","last_active":"2024-04-14T15:29:11.157601+0000","last_peered":"2024-04-14T15:29:11.157601+0000","last_clean":"2024-04-14T15:29:11.157601+0000","last_became_active":"2024-04-14T15:26:27.062451+0000","last_became_peered":"2024-04-14T15:26:27.062451+0000","last_unstale":"2024-04-14T15:29:11.157601+0000","last_undegraded":"2024-04-14T15:29:11.157601+0000","last_fullsized":"2024-04-14T15:29:11.157601+0000","mapping_epoch":828,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":829,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'19","last_scrub_stamp":"2024-04-14T15:28:47.825887+0000","last_deep_scrub":"183'13","last_deep_scrub_stamp":"2024-04-14T15:14:11.755252+0000","last_clean_scrub_stamp":"2024-04-14T15:28:47.825887+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:07.579857+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1798,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.079543+0000","last_change":"2024-04-14T15:28:56.033939+0000","last_active":"2024-04-14T15:29:11.079543+0000","last_peered":"2024-04-14T15:29:11.079543+0000","last_clean":"2024-04-14T15:29:11.079543+0000","last_became_active":"2024-04-14T15:09:43.019301+0000","last_became_peered":"2024-04-14T15:09:43.019301+0000","last_unstale":"2024-04-14T15:29:11.079543+0000","last_undegraded":"2024-04-14T15:29:11.079543+0000","last_fullsized":"2024-04-14T15:29:11.079543+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:56.033841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:28:56.033841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:05.569556+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"255'10","reported_seq":1664,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.157075+0000","last_change":"2024-04-14T15:27:53.037018+0000","last_active":"2024-04-14T15:29:11.157075+0000","last_peered":"2024-04-14T15:29:11.157075+0000","last_clean":"2024-04-14T15:29:11.157075+0000","last_became_active":"2024-04-14T15:15:21.884468+0000","last_became_peered":"2024-04-14T15:15:21.884468+0000","last_unstale":"2024-04-14T15:29:11.157075+0000","last_undegraded":"2024-04-14T15:29:11.157075+0000","last_fullsized":"2024-04-14T15:29:11.157075+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'10","last_scrub_stamp":"2024-04-14T15:27:53.036891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:27:53.036891+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:20.315770+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1775,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.074657+0000","last_change":"2024-04-14T15:28:32.941088+0000","last_active":"2024-04-14T15:29:11.074657+0000","last_peered":"2024-04-14T15:29:11.074657+0000","last_clean":"2024-04-14T15:29:11.074657+0000","last_became_active":"2024-04-14T15:09:49.065242+0000","last_became_peered":"2024-04-14T15:09:49.065242+0000","last_unstale":"2024-04-14T15:29:11.074657+0000","last_undegraded":"2024-04-14T15:29:11.074657+0000","last_fullsized":"2024-04-14T15:29:11.074657+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:32.940953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:28:32.940953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:44.206768+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":1692,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:13.263492+0000","last_change":"2024-04-14T15:29:13.263518+0000","last_active":"2024-04-14T15:29:13.263492+0000","last_peered":"2024-04-14T15:29:13.263492+0000","last_clean":"2024-04-14T15:29:13.263492+0000","last_became_active":"2024-04-14T15:12:56.389204+0000","last_became_peered":"2024-04-14T15:12:56.389204+0000","last_unstale":"2024-04-14T15:29:13.263492+0000","last_undegraded":"2024-04-14T15:29:13.263492+0000","last_fullsized":"2024-04-14T15:29:13.263492+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:29:13.263410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:29:13.263410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:15.116162+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"479'19","reported_seq":640,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.156034+0000","last_change":"2024-04-14T15:28:55.902406+0000","last_active":"2024-04-14T15:29:11.156034+0000","last_peered":"2024-04-14T15:29:11.156034+0000","last_clean":"2024-04-14T15:29:11.156034+0000","last_became_active":"2024-04-14T15:27:14.204854+0000","last_became_peered":"2024-04-14T15:27:14.204854+0000","last_unstale":"2024-04-14T15:29:11.156034+0000","last_undegraded":"2024-04-14T15:29:11.156034+0000","last_fullsized":"2024-04-14T15:29:11.156034+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'19","last_scrub_stamp":"2024-04-14T15:28:55.902288+0000","last_deep_scrub":"479'19","last_deep_scrub_stamp":"2024-04-14T15:28:55.902288+0000","last_clean_scrub_stamp":"2024-04-14T15:28:55.902288+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:22.172278+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":142,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.156479+0000","last_change":"2024-04-14T15:29:08.282168+0000","last_active":"2024-04-14T15:29:11.156479+0000","last_peered":"2024-04-14T15:29:11.156479+0000","last_clean":"2024-04-14T15:29:11.156479+0000","last_became_active":"2024-04-14T15:27:14.204682+0000","last_became_peered":"2024-04-14T15:27:14.204682+0000","last_unstale":"2024-04-14T15:29:11.156479+0000","last_undegraded":"2024-04-14T15:29:11.156479+0000","last_fullsized":"2024-04-14T15:29:11.156479+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:29:08.282081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:27:50.505612+0000","last_clean_scrub_stamp":"2024-04-14T15:29:08.282081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:36.736107+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1458,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.156644+0000","last_change":"2024-04-14T15:28:53.872899+0000","last_active":"2024-04-14T15:29:11.156644+0000","last_peered":"2024-04-14T15:29:11.156644+0000","last_clean":"2024-04-14T15:29:11.156644+0000","last_became_active":"2024-04-14T15:15:21.882643+0000","last_became_peered":"2024-04-14T15:15:21.882643+0000","last_unstale":"2024-04-14T15:29:11.156644+0000","last_undegraded":"2024-04-14T15:29:11.156644+0000","last_fullsized":"2024-04-14T15:29:11.156644+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:53.872816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:33.485411+0000","last_clean_scrub_stamp":"2024-04-14T15:28:53.872816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:56.975133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1781,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.078189+0000","last_change":"2024-04-14T15:27:58.172404+0000","last_active":"2024-04-14T15:29:11.078189+0000","last_peered":"2024-04-14T15:29:11.078189+0000","last_clean":"2024-04-14T15:29:11.078189+0000","last_became_active":"2024-04-14T15:09:49.059711+0000","last_became_peered":"2024-04-14T15:09:49.059711+0000","last_unstale":"2024-04-14T15:29:11.078189+0000","last_undegraded":"2024-04-14T15:29:11.078189+0000","last_fullsized":"2024-04-14T15:29:11.078189+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:27:58.172330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:27:58.172330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:24.855336+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":142,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.156821+0000","last_change":"2024-04-14T15:28:59.415776+0000","last_active":"2024-04-14T15:29:11.156821+0000","last_peered":"2024-04-14T15:29:11.156821+0000","last_clean":"2024-04-14T15:29:11.156821+0000","last_became_active":"2024-04-14T15:27:14.218875+0000","last_became_peered":"2024-04-14T15:27:14.218875+0000","last_unstale":"2024-04-14T15:29:11.156821+0000","last_undegraded":"2024-04-14T15:29:11.156821+0000","last_fullsized":"2024-04-14T15:29:11.156821+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:59.415615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:28:59.415615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:09.387385+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"479'24","reported_seq":1855,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.074047+0000","last_change":"2024-04-14T15:28:36.910524+0000","last_active":"2024-04-14T15:29:11.074047+0000","last_peered":"2024-04-14T15:29:11.074047+0000","last_clean":"2024-04-14T15:29:11.074047+0000","last_became_active":"2024-04-14T15:27:14.223844+0000","last_became_peered":"2024-04-14T15:27:14.223844+0000","last_unstale":"2024-04-14T15:29:11.074047+0000","last_undegraded":"2024-04-14T15:29:11.074047+0000","last_fullsized":"2024-04-14T15:29:11.074047+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'24","last_scrub_stamp":"2024-04-14T15:28:36.910422+0000","last_deep_scrub":"464'16","last_deep_scrub_stamp":"2024-04-14T15:19:17.589999+0000","last_clean_scrub_stamp":"2024-04-14T15:28:36.910422+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:58.983098+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":1801,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.079852+0000","last_change":"2024-04-14T15:28:24.128742+0000","last_active":"2024-04-14T15:29:11.079852+0000","last_peered":"2024-04-14T15:29:11.079852+0000","last_clean":"2024-04-14T15:29:11.079852+0000","last_became_active":"2024-04-14T15:09:43.023583+0000","last_became_peered":"2024-04-14T15:09:43.023583+0000","last_unstale":"2024-04-14T15:29:11.079852+0000","last_undegraded":"2024-04-14T15:29:11.079852+0000","last_fullsized":"2024-04-14T15:29:11.079852+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:24.128613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:28:24.128613+0000","last_clean_scrub_stamp":"2024-04-14T15:28:24.128613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:40.104962+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1785,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.078380+0000","last_change":"2024-04-14T15:28:21.180951+0000","last_active":"2024-04-14T15:29:11.078380+0000","last_peered":"2024-04-14T15:29:11.078380+0000","last_clean":"2024-04-14T15:29:11.078380+0000","last_became_active":"2024-04-14T15:09:47.041683+0000","last_became_peered":"2024-04-14T15:09:47.041683+0000","last_unstale":"2024-04-14T15:29:11.078380+0000","last_undegraded":"2024-04-14T15:29:11.078380+0000","last_fullsized":"2024-04-14T15:29:11.078380+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:21.180874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:08.486217+0000","last_clean_scrub_stamp":"2024-04-14T15:28:21.180874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:33.674236+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1454,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.155985+0000","last_change":"2024-04-14T15:27:59.074687+0000","last_active":"2024-04-14T15:29:11.155985+0000","last_peered":"2024-04-14T15:29:11.155985+0000","last_clean":"2024-04-14T15:29:11.155985+0000","last_became_active":"2024-04-14T15:15:21.889567+0000","last_became_peered":"2024-04-14T15:15:21.889567+0000","last_unstale":"2024-04-14T15:29:11.155985+0000","last_undegraded":"2024-04-14T15:29:11.155985+0000","last_fullsized":"2024-04-14T15:29:11.155985+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:27:59.074610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:27:59.074610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:25.024323+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":1705,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.106541+0000","last_change":"2024-04-14T15:29:05.178562+0000","last_active":"2024-04-14T15:29:11.106541+0000","last_peered":"2024-04-14T15:29:11.106541+0000","last_clean":"2024-04-14T15:29:11.106541+0000","last_became_active":"2024-04-14T15:12:56.387723+0000","last_became_peered":"2024-04-14T15:12:56.387723+0000","last_unstale":"2024-04-14T15:29:11.106541+0000","last_undegraded":"2024-04-14T15:29:11.106541+0000","last_fullsized":"2024-04-14T15:29:11.106541+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:29:05.178478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:03.685045+0000","last_clean_scrub_stamp":"2024-04-14T15:29:05.178478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:22.494936+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.10","version":"479'8","reported_seq":1822,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.156035+0000","last_change":"2024-04-14T15:28:31.057158+0000","last_active":"2024-04-14T15:29:11.156035+0000","last_peered":"2024-04-14T15:29:11.156035+0000","last_clean":"2024-04-14T15:29:11.156035+0000","last_became_active":"2024-04-14T15:27:14.219371+0000","last_became_peered":"2024-04-14T15:27:14.219371+0000","last_unstale":"2024-04-14T15:29:11.156035+0000","last_undegraded":"2024-04-14T15:29:11.156035+0000","last_fullsized":"2024-04-14T15:29:11.156035+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'8","last_scrub_stamp":"2024-04-14T15:28:31.057050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:28:31.057050+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:50.993432+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":670,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.157704+0000","last_change":"2024-04-14T15:28:09.858748+0000","last_active":"2024-04-14T15:29:11.157704+0000","last_peered":"2024-04-14T15:29:11.157704+0000","last_clean":"2024-04-14T15:29:11.157704+0000","last_became_active":"2024-04-14T15:27:14.202117+0000","last_became_peered":"2024-04-14T15:27:14.202117+0000","last_unstale":"2024-04-14T15:29:11.157704+0000","last_undegraded":"2024-04-14T15:29:11.157704+0000","last_fullsized":"2024-04-14T15:29:11.157704+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:09.858655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:08.712289+0000","last_clean_scrub_stamp":"2024-04-14T15:28:09.858655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:35.246948+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":142,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.157172+0000","last_change":"2024-04-14T15:28:32.362435+0000","last_active":"2024-04-14T15:29:11.157172+0000","last_peered":"2024-04-14T15:29:11.157172+0000","last_clean":"2024-04-14T15:29:11.157172+0000","last_became_active":"2024-04-14T15:27:14.219333+0000","last_became_peered":"2024-04-14T15:27:14.219333+0000","last_unstale":"2024-04-14T15:29:11.157172+0000","last_undegraded":"2024-04-14T15:29:11.157172+0000","last_fullsized":"2024-04-14T15:29:11.157172+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:32.362297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:28:32.362297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:56.724312+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1776,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.114081+0000","last_change":"2024-04-14T15:28:58.134672+0000","last_active":"2024-04-14T15:29:11.114081+0000","last_peered":"2024-04-14T15:29:11.114081+0000","last_clean":"2024-04-14T15:29:11.114081+0000","last_became_active":"2024-04-14T15:09:49.061359+0000","last_became_peered":"2024-04-14T15:09:49.061359+0000","last_unstale":"2024-04-14T15:29:11.114081+0000","last_undegraded":"2024-04-14T15:29:11.114081+0000","last_fullsized":"2024-04-14T15:29:11.114081+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:58.134566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:27:31.732339+0000","last_clean_scrub_stamp":"2024-04-14T15:28:58.134566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:02.746731+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":1698,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.114620+0000","last_change":"2024-04-14T15:28:05.992275+0000","last_active":"2024-04-14T15:29:11.114620+0000","last_peered":"2024-04-14T15:29:11.114620+0000","last_clean":"2024-04-14T15:29:11.114620+0000","last_became_active":"2024-04-14T15:27:14.202407+0000","last_became_peered":"2024-04-14T15:27:14.202407+0000","last_unstale":"2024-04-14T15:29:11.114620+0000","last_undegraded":"2024-04-14T15:29:11.114620+0000","last_fullsized":"2024-04-14T15:29:11.114620+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:05.992175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:15:43.680852+0000","last_clean_scrub_stamp":"2024-04-14T15:28:05.992175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:33.603407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"479'24","reported_seq":1495,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.156324+0000","last_change":"2024-04-14T15:28:20.146179+0000","last_active":"2024-04-14T15:29:11.156324+0000","last_peered":"2024-04-14T15:29:11.156324+0000","last_clean":"2024-04-14T15:29:11.156324+0000","last_became_active":"2024-04-14T15:27:14.219501+0000","last_became_peered":"2024-04-14T15:27:14.219501+0000","last_unstale":"2024-04-14T15:29:11.156324+0000","last_undegraded":"2024-04-14T15:29:11.156324+0000","last_fullsized":"2024-04-14T15:29:11.156324+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'24","last_scrub_stamp":"2024-04-14T15:28:20.146086+0000","last_deep_scrub":"102'16","last_deep_scrub_stamp":"2024-04-14T15:12:35.999326+0000","last_clean_scrub_stamp":"2024-04-14T15:28:20.146086+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:37.390831+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":142,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.157454+0000","last_change":"2024-04-14T15:28:36.313957+0000","last_active":"2024-04-14T15:29:11.157454+0000","last_peered":"2024-04-14T15:29:11.157454+0000","last_clean":"2024-04-14T15:29:11.157454+0000","last_became_active":"2024-04-14T15:27:14.213337+0000","last_became_peered":"2024-04-14T15:27:14.213337+0000","last_unstale":"2024-04-14T15:29:11.157454+0000","last_undegraded":"2024-04-14T15:29:11.157454+0000","last_fullsized":"2024-04-14T15:29:11.157454+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:36.313807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:30.484673+0000","last_clean_scrub_stamp":"2024-04-14T15:28:36.313807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:46.980513+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1792,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.113456+0000","last_change":"2024-04-14T15:28:56.113120+0000","last_active":"2024-04-14T15:29:11.113456+0000","last_peered":"2024-04-14T15:29:11.113456+0000","last_clean":"2024-04-14T15:29:11.113456+0000","last_became_active":"2024-04-14T15:27:14.215548+0000","last_became_peered":"2024-04-14T15:27:14.215548+0000","last_unstale":"2024-04-14T15:29:11.113456+0000","last_undegraded":"2024-04-14T15:29:11.113456+0000","last_fullsized":"2024-04-14T15:29:11.113456+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:56.113024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:27:44.912947+0000","last_clean_scrub_stamp":"2024-04-14T15:28:56.113024+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:15.477954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1474,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.156790+0000","last_change":"2024-04-14T15:28:45.874358+0000","last_active":"2024-04-14T15:29:11.156790+0000","last_peered":"2024-04-14T15:29:11.156790+0000","last_clean":"2024-04-14T15:29:11.156790+0000","last_became_active":"2024-04-14T15:27:14.202684+0000","last_became_peered":"2024-04-14T15:27:14.202684+0000","last_unstale":"2024-04-14T15:29:11.156790+0000","last_undegraded":"2024-04-14T15:29:11.156790+0000","last_fullsized":"2024-04-14T15:29:11.156790+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:45.874274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:27:25.021273+0000","last_clean_scrub_stamp":"2024-04-14T15:28:45.874274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:59.303768+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":1696,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.105326+0000","last_change":"2024-04-14T15:28:45.338663+0000","last_active":"2024-04-14T15:29:11.105326+0000","last_peered":"2024-04-14T15:29:11.105326+0000","last_clean":"2024-04-14T15:29:11.105326+0000","last_became_active":"2024-04-14T15:12:56.335230+0000","last_became_peered":"2024-04-14T15:12:56.335230+0000","last_unstale":"2024-04-14T15:29:11.105326+0000","last_undegraded":"2024-04-14T15:29:11.105326+0000","last_fullsized":"2024-04-14T15:29:11.105326+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:45.338583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:28:45.338583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:05.098064+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"776'25","reported_seq":1822,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.079639+0000","last_change":"2024-04-14T15:28:14.047717+0000","last_active":"2024-04-14T15:29:11.079639+0000","last_peered":"2024-04-14T15:29:11.079639+0000","last_clean":"2024-04-14T15:29:11.079639+0000","last_became_active":"2024-04-14T15:09:45.181456+0000","last_became_peered":"2024-04-14T15:09:45.181456+0000","last_unstale":"2024-04-14T15:29:11.079639+0000","last_undegraded":"2024-04-14T15:29:11.079639+0000","last_fullsized":"2024-04-14T15:29:11.079639+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"776'25","last_scrub_stamp":"2024-04-14T15:28:14.047615+0000","last_deep_scrub":"776'25","last_deep_scrub_stamp":"2024-04-14T15:25:21.063579+0000","last_clean_scrub_stamp":"2024-04-14T15:28:14.047615+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:27.182951+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":672,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.155870+0000","last_change":"2024-04-14T15:28:52.894885+0000","last_active":"2024-04-14T15:29:11.155870+0000","last_peered":"2024-04-14T15:29:11.155870+0000","last_clean":"2024-04-14T15:29:11.155870+0000","last_became_active":"2024-04-14T15:26:27.058655+0000","last_became_peered":"2024-04-14T15:26:27.058655+0000","last_unstale":"2024-04-14T15:29:11.155870+0000","last_undegraded":"2024-04-14T15:29:11.155870+0000","last_fullsized":"2024-04-14T15:29:11.155870+0000","mapping_epoch":828,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":829,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:52.894796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:27:30.830543+0000","last_clean_scrub_stamp":"2024-04-14T15:28:52.894796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:02.812727+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"866'10","reported_seq":171,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.157521+0000","last_change":"2024-04-14T15:29:06.253222+0000","last_active":"2024-04-14T15:29:11.157521+0000","last_peered":"2024-04-14T15:29:11.157521+0000","last_clean":"2024-04-14T15:29:11.157521+0000","last_became_active":"2024-04-14T15:27:14.219712+0000","last_became_peered":"2024-04-14T15:27:14.219712+0000","last_unstale":"2024-04-14T15:29:11.157521+0000","last_undegraded":"2024-04-14T15:29:11.157521+0000","last_fullsized":"2024-04-14T15:29:11.157521+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"866'10","last_scrub_stamp":"2024-04-14T15:29:06.253113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:29:06.253113+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:23.997020+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"72'1","reported_seq":1784,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.114727+0000","last_change":"2024-04-14T15:28:33.970559+0000","last_active":"2024-04-14T15:29:11.114727+0000","last_peered":"2024-04-14T15:29:11.114727+0000","last_clean":"2024-04-14T15:29:11.114727+0000","last_became_active":"2024-04-14T15:09:49.062051+0000","last_became_peered":"2024-04-14T15:09:49.062051+0000","last_unstale":"2024-04-14T15:29:11.114727+0000","last_undegraded":"2024-04-14T15:29:11.114727+0000","last_fullsized":"2024-04-14T15:29:11.114727+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-04-14T15:28:33.970444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:28:33.970444+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:52.332757+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":1698,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.078436+0000","last_change":"2024-04-14T15:29:01.043551+0000","last_active":"2024-04-14T15:29:11.078436+0000","last_peered":"2024-04-14T15:29:11.078436+0000","last_clean":"2024-04-14T15:29:11.078436+0000","last_became_active":"2024-04-14T15:27:14.215274+0000","last_became_peered":"2024-04-14T15:27:14.215274+0000","last_unstale":"2024-04-14T15:29:11.078436+0000","last_undegraded":"2024-04-14T15:29:11.078436+0000","last_fullsized":"2024-04-14T15:29:11.078436+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:29:01.043471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:29.418183+0000","last_clean_scrub_stamp":"2024-04-14T15:29:01.043471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:30.779896+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"519'24","reported_seq":154,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.157326+0000","last_change":"2024-04-14T15:28:41.374815+0000","last_active":"2024-04-14T15:29:11.157326+0000","last_peered":"2024-04-14T15:29:11.157326+0000","last_clean":"2024-04-14T15:29:11.157326+0000","last_became_active":"2024-04-14T15:27:14.213577+0000","last_became_peered":"2024-04-14T15:27:14.213577+0000","last_unstale":"2024-04-14T15:29:11.157326+0000","last_undegraded":"2024-04-14T15:29:11.157326+0000","last_fullsized":"2024-04-14T15:29:11.157326+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"519'24","last_scrub_stamp":"2024-04-14T15:28:41.374691+0000","last_deep_scrub":"519'24","last_deep_scrub_stamp":"2024-04-14T15:20:44.763417+0000","last_clean_scrub_stamp":"2024-04-14T15:28:41.374691+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:41.898370+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"684'4","reported_seq":1499,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.156146+0000","last_change":"2024-04-14T15:28:05.163497+0000","last_active":"2024-04-14T15:29:11.156146+0000","last_peered":"2024-04-14T15:29:11.156146+0000","last_clean":"2024-04-14T15:29:11.156146+0000","last_became_active":"2024-04-14T15:27:14.219431+0000","last_became_peered":"2024-04-14T15:27:14.219431+0000","last_unstale":"2024-04-14T15:29:11.156146+0000","last_undegraded":"2024-04-14T15:29:11.156146+0000","last_fullsized":"2024-04-14T15:29:11.156146+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"684'4","last_scrub_stamp":"2024-04-14T15:28:05.163399+0000","last_deep_scrub":"684'4","last_deep_scrub_stamp":"2024-04-14T15:23:46.931560+0000","last_clean_scrub_stamp":"2024-04-14T15:28:05.163399+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1914,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1783,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.079733+0000","last_change":"2024-04-14T15:28:33.055080+0000","last_active":"2024-04-14T15:29:11.079733+0000","last_peered":"2024-04-14T15:29:11.079733+0000","last_clean":"2024-04-14T15:29:11.079733+0000","last_became_active":"2024-04-14T15:09:47.045730+0000","last_became_peered":"2024-04-14T15:09:47.045730+0000","last_unstale":"2024-04-14T15:29:11.079733+0000","last_undegraded":"2024-04-14T15:29:11.079733+0000","last_fullsized":"2024-04-14T15:29:11.079733+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:33.054998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:27:22.882334+0000","last_clean_scrub_stamp":"2024-04-14T15:28:33.054998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:46.556521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1780,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.079662+0000","last_change":"2024-04-14T15:28:37.328960+0000","last_active":"2024-04-14T15:29:11.079662+0000","last_peered":"2024-04-14T15:29:11.079662+0000","last_clean":"2024-04-14T15:29:11.079662+0000","last_became_active":"2024-04-14T15:09:49.057545+0000","last_became_peered":"2024-04-14T15:09:49.057545+0000","last_unstale":"2024-04-14T15:29:11.079662+0000","last_undegraded":"2024-04-14T15:29:11.079662+0000","last_fullsized":"2024-04-14T15:29:11.079662+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:37.328839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:30.343106+0000","last_clean_scrub_stamp":"2024-04-14T15:28:37.328839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:03.203875+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":561,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.156465+0000","last_change":"2024-04-14T15:27:59.880061+0000","last_active":"2024-04-14T15:29:11.156465+0000","last_peered":"2024-04-14T15:29:11.156465+0000","last_clean":"2024-04-14T15:29:11.156465+0000","last_became_active":"2024-04-14T15:26:27.059738+0000","last_became_peered":"2024-04-14T15:26:27.059738+0000","last_unstale":"2024-04-14T15:29:11.156465+0000","last_undegraded":"2024-04-14T15:29:11.156465+0000","last_fullsized":"2024-04-14T15:29:11.156465+0000","mapping_epoch":828,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":829,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:27:59.879976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:27:59.879976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:16.538197+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"479'27","reported_seq":1830,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.106318+0000","last_change":"2024-04-14T15:27:56.392763+0000","last_active":"2024-04-14T15:29:11.106318+0000","last_peered":"2024-04-14T15:29:11.106318+0000","last_clean":"2024-04-14T15:29:11.106318+0000","last_became_active":"2024-04-14T15:09:45.032887+0000","last_became_peered":"2024-04-14T15:09:45.032887+0000","last_unstale":"2024-04-14T15:29:11.106318+0000","last_undegraded":"2024-04-14T15:29:11.106318+0000","last_fullsized":"2024-04-14T15:29:11.106318+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'27","last_scrub_stamp":"2024-04-14T15:27:56.392615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:27:56.392615+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:23.336007+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":142,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.156582+0000","last_change":"2024-04-14T15:28:46.394175+0000","last_active":"2024-04-14T15:29:11.156582+0000","last_peered":"2024-04-14T15:29:11.156582+0000","last_clean":"2024-04-14T15:29:11.156582+0000","last_became_active":"2024-04-14T15:27:14.222196+0000","last_became_peered":"2024-04-14T15:27:14.222196+0000","last_unstale":"2024-04-14T15:29:11.156582+0000","last_undegraded":"2024-04-14T15:29:11.156582+0000","last_fullsized":"2024-04-14T15:29:11.156582+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:46.394030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:27:27.745213+0000","last_clean_scrub_stamp":"2024-04-14T15:28:46.394030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:54.018805+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1792,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.079846+0000","last_change":"2024-04-14T15:28:45.312738+0000","last_active":"2024-04-14T15:29:11.079846+0000","last_peered":"2024-04-14T15:29:11.079846+0000","last_clean":"2024-04-14T15:29:11.079846+0000","last_became_active":"2024-04-14T15:09:47.042733+0000","last_became_peered":"2024-04-14T15:09:47.042733+0000","last_unstale":"2024-04-14T15:29:11.079846+0000","last_undegraded":"2024-04-14T15:29:11.079846+0000","last_fullsized":"2024-04-14T15:29:11.079846+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:45.312658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:25:22.689718+0000","last_clean_scrub_stamp":"2024-04-14T15:28:45.312658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:56.876051+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1788,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.074470+0000","last_change":"2024-04-14T15:27:57.115562+0000","last_active":"2024-04-14T15:29:11.074470+0000","last_peered":"2024-04-14T15:29:11.074470+0000","last_clean":"2024-04-14T15:29:11.074470+0000","last_became_active":"2024-04-14T15:27:14.223736+0000","last_became_peered":"2024-04-14T15:27:14.223736+0000","last_unstale":"2024-04-14T15:29:11.074470+0000","last_undegraded":"2024-04-14T15:29:11.074470+0000","last_fullsized":"2024-04-14T15:29:11.074470+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:27:57.115489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:54.453174+0000","last_clean_scrub_stamp":"2024-04-14T15:27:57.115489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:26.600941+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":1684,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.113685+0000","last_change":"2024-04-14T15:28:04.968660+0000","last_active":"2024-04-14T15:29:11.113685+0000","last_peered":"2024-04-14T15:29:11.113685+0000","last_clean":"2024-04-14T15:29:11.113685+0000","last_became_active":"2024-04-14T15:12:56.392352+0000","last_became_peered":"2024-04-14T15:12:56.392352+0000","last_unstale":"2024-04-14T15:29:11.113685+0000","last_undegraded":"2024-04-14T15:29:11.113685+0000","last_fullsized":"2024-04-14T15:29:11.113685+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:04.968559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:28:04.968559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:30.122612+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"479'18","reported_seq":1502,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.156686+0000","last_change":"2024-04-14T15:28:41.993464+0000","last_active":"2024-04-14T15:29:11.156686+0000","last_peered":"2024-04-14T15:29:11.156686+0000","last_clean":"2024-04-14T15:29:11.156686+0000","last_became_active":"2024-04-14T15:15:21.877838+0000","last_became_peered":"2024-04-14T15:15:21.877838+0000","last_unstale":"2024-04-14T15:29:11.156686+0000","last_undegraded":"2024-04-14T15:29:11.156686+0000","last_fullsized":"2024-04-14T15:29:11.156686+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'18","last_scrub_stamp":"2024-04-14T15:28:41.993355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:28:41.993355+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:08.178104+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":1478,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.156667+0000","last_change":"2024-04-14T15:28:01.146869+0000","last_active":"2024-04-14T15:29:11.156667+0000","last_peered":"2024-04-14T15:29:11.156667+0000","last_clean":"2024-04-14T15:29:11.156667+0000","last_became_active":"2024-04-14T15:27:14.204179+0000","last_became_peered":"2024-04-14T15:27:14.204179+0000","last_unstale":"2024-04-14T15:29:11.156667+0000","last_undegraded":"2024-04-14T15:29:11.156667+0000","last_fullsized":"2024-04-14T15:29:11.156667+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:01.146789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:28:01.146789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:28.765987+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1786,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.074164+0000","last_change":"2024-04-14T15:29:00.879787+0000","last_active":"2024-04-14T15:29:11.074164+0000","last_peered":"2024-04-14T15:29:11.074164+0000","last_clean":"2024-04-14T15:29:11.074164+0000","last_became_active":"2024-04-14T15:09:47.048045+0000","last_became_peered":"2024-04-14T15:09:47.048045+0000","last_unstale":"2024-04-14T15:29:11.074164+0000","last_undegraded":"2024-04-14T15:29:11.074164+0000","last_fullsized":"2024-04-14T15:29:11.074164+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:29:00.879688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:29:00.879688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:08.583065+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1476,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.157130+0000","last_change":"2024-04-14T15:28:58.824212+0000","last_active":"2024-04-14T15:29:11.157130+0000","last_peered":"2024-04-14T15:29:11.157130+0000","last_clean":"2024-04-14T15:29:11.157130+0000","last_became_active":"2024-04-14T15:27:14.208409+0000","last_became_peered":"2024-04-14T15:27:14.208409+0000","last_unstale":"2024-04-14T15:29:11.157130+0000","last_undegraded":"2024-04-14T15:29:11.157130+0000","last_fullsized":"2024-04-14T15:29:11.157130+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:58.824136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:27:44.071208+0000","last_clean_scrub_stamp":"2024-04-14T15:28:58.824136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:15.763311+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":1704,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.106241+0000","last_change":"2024-04-14T15:29:09.118127+0000","last_active":"2024-04-14T15:29:11.106241+0000","last_peered":"2024-04-14T15:29:11.106241+0000","last_clean":"2024-04-14T15:29:11.106241+0000","last_became_active":"2024-04-14T15:12:56.375278+0000","last_became_peered":"2024-04-14T15:12:56.375278+0000","last_unstale":"2024-04-14T15:29:11.106241+0000","last_undegraded":"2024-04-14T15:29:11.106241+0000","last_fullsized":"2024-04-14T15:29:11.106241+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:29:09.118049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:25:19.668662+0000","last_clean_scrub_stamp":"2024-04-14T15:29:09.118049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:19.630779+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"479'19","reported_seq":1819,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.073841+0000","last_change":"2024-04-14T15:29:01.920385+0000","last_active":"2024-04-14T15:29:11.073841+0000","last_peered":"2024-04-14T15:29:11.073841+0000","last_clean":"2024-04-14T15:29:11.073841+0000","last_became_active":"2024-04-14T15:09:45.030093+0000","last_became_peered":"2024-04-14T15:09:45.030093+0000","last_unstale":"2024-04-14T15:29:11.073841+0000","last_undegraded":"2024-04-14T15:29:11.073841+0000","last_fullsized":"2024-04-14T15:29:11.073841+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'19","last_scrub_stamp":"2024-04-14T15:29:01.920260+0000","last_deep_scrub":"276'13","last_deep_scrub_stamp":"2024-04-14T15:15:43.969775+0000","last_clean_scrub_stamp":"2024-04-14T15:29:01.920260+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:03.252951+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":1493,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.113398+0000","last_change":"2024-04-14T15:29:01.194106+0000","last_active":"2024-04-14T15:29:11.113398+0000","last_peered":"2024-04-14T15:29:11.113398+0000","last_clean":"2024-04-14T15:29:11.113398+0000","last_became_active":"2024-04-14T15:14:30.857727+0000","last_became_peered":"2024-04-14T15:14:30.857727+0000","last_unstale":"2024-04-14T15:29:11.113398+0000","last_undegraded":"2024-04-14T15:29:11.113398+0000","last_fullsized":"2024-04-14T15:29:11.113398+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:29:01.194007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:29:01.194007+0000","last_clean_scrub_stamp":"2024-04-14T15:29:01.194007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:03.732070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":664,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.156250+0000","last_change":"2024-04-14T15:28:44.725756+0000","last_active":"2024-04-14T15:29:11.156250+0000","last_peered":"2024-04-14T15:29:11.156250+0000","last_clean":"2024-04-14T15:29:11.156250+0000","last_became_active":"2024-04-14T15:26:27.068043+0000","last_became_peered":"2024-04-14T15:26:27.068043+0000","last_unstale":"2024-04-14T15:29:11.156250+0000","last_undegraded":"2024-04-14T15:29:11.156250+0000","last_fullsized":"2024-04-14T15:29:11.156250+0000","mapping_epoch":828,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":829,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:44.725657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:28:44.725657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:02.238641+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1786,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.078610+0000","last_change":"2024-04-14T15:29:04.085955+0000","last_active":"2024-04-14T15:29:11.078610+0000","last_peered":"2024-04-14T15:29:11.078610+0000","last_clean":"2024-04-14T15:29:11.078610+0000","last_became_active":"2024-04-14T15:09:49.061756+0000","last_became_peered":"2024-04-14T15:09:49.061756+0000","last_unstale":"2024-04-14T15:29:11.078610+0000","last_undegraded":"2024-04-14T15:29:11.078610+0000","last_fullsized":"2024-04-14T15:29:11.078610+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:29:04.085812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:29:04.085812+0000","last_clean_scrub_stamp":"2024-04-14T15:29:04.085812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:16.288367+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":134,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.156652+0000","last_change":"2024-04-14T15:27:56.629104+0000","last_active":"2024-04-14T15:29:11.156652+0000","last_peered":"2024-04-14T15:29:11.156652+0000","last_clean":"2024-04-14T15:29:11.156652+0000","last_became_active":"2024-04-14T15:27:14.213348+0000","last_became_peered":"2024-04-14T15:27:14.213348+0000","last_unstale":"2024-04-14T15:29:11.156652+0000","last_undegraded":"2024-04-14T15:29:11.156652+0000","last_fullsized":"2024-04-14T15:29:11.156652+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:27:56.629012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:27:56.629012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:13.676232+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1b","version":"479'9","reported_seq":1797,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.074295+0000","last_change":"2024-04-14T15:27:51.172348+0000","last_active":"2024-04-14T15:29:11.074295+0000","last_peered":"2024-04-14T15:29:11.074295+0000","last_clean":"2024-04-14T15:29:11.074295+0000","last_became_active":"2024-04-14T15:09:45.028667+0000","last_became_peered":"2024-04-14T15:09:45.028667+0000","last_unstale":"2024-04-14T15:29:11.074295+0000","last_undegraded":"2024-04-14T15:29:11.074295+0000","last_fullsized":"2024-04-14T15:29:11.074295+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'9","last_scrub_stamp":"2024-04-14T15:27:51.172269+0000","last_deep_scrub":"87'7","last_deep_scrub_stamp":"2024-04-14T15:11:10.510785+0000","last_clean_scrub_stamp":"2024-04-14T15:27:51.172269+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:17.237916+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":1281,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.105787+0000","last_change":"2024-04-14T15:28:09.282466+0000","last_active":"2024-04-14T15:29:11.105787+0000","last_peered":"2024-04-14T15:29:11.105787+0000","last_clean":"2024-04-14T15:29:11.105787+0000","last_became_active":"2024-04-14T15:27:14.208738+0000","last_became_peered":"2024-04-14T15:27:14.208738+0000","last_unstale":"2024-04-14T15:29:11.105787+0000","last_undegraded":"2024-04-14T15:29:11.105787+0000","last_fullsized":"2024-04-14T15:29:11.105787+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:09.282383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:22:15.944005+0000","last_clean_scrub_stamp":"2024-04-14T15:28:09.282383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:36.169459+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1779,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.113982+0000","last_change":"2024-04-14T15:28:14.957788+0000","last_active":"2024-04-14T15:29:11.113982+0000","last_peered":"2024-04-14T15:29:11.113982+0000","last_clean":"2024-04-14T15:29:11.113982+0000","last_became_active":"2024-04-14T15:09:47.046517+0000","last_became_peered":"2024-04-14T15:09:47.046517+0000","last_unstale":"2024-04-14T15:29:11.113982+0000","last_undegraded":"2024-04-14T15:29:11.113982+0000","last_fullsized":"2024-04-14T15:29:11.113982+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:14.957694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:28:14.957694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:34.128715+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":660,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.156064+0000","last_change":"2024-04-14T15:29:09.099762+0000","last_active":"2024-04-14T15:29:11.156064+0000","last_peered":"2024-04-14T15:29:11.156064+0000","last_clean":"2024-04-14T15:29:11.156064+0000","last_became_active":"2024-04-14T15:26:27.065443+0000","last_became_peered":"2024-04-14T15:26:27.065443+0000","last_unstale":"2024-04-14T15:29:11.156064+0000","last_undegraded":"2024-04-14T15:29:11.156064+0000","last_fullsized":"2024-04-14T15:29:11.156064+0000","mapping_epoch":828,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":829,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:29:09.099620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:56.268086+0000","last_clean_scrub_stamp":"2024-04-14T15:29:09.099620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:17.541761+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":1704,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:12.241814+0000","last_change":"2024-04-14T15:29:12.241841+0000","last_active":"2024-04-14T15:29:12.241814+0000","last_peered":"2024-04-14T15:29:12.241814+0000","last_clean":"2024-04-14T15:29:12.241814+0000","last_became_active":"2024-04-14T15:27:14.214682+0000","last_became_peered":"2024-04-14T15:27:14.214682+0000","last_unstale":"2024-04-14T15:29:12.241814+0000","last_undegraded":"2024-04-14T15:29:12.241814+0000","last_fullsized":"2024-04-14T15:29:12.241814+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:29:12.241729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:29:12.241729+0000","last_clean_scrub_stamp":"2024-04-14T15:29:12.241729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:17.963819+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.18","version":"806'19","reported_seq":1852,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.073469+0000","last_change":"2024-04-14T15:29:07.987282+0000","last_active":"2024-04-14T15:29:11.073469+0000","last_peered":"2024-04-14T15:29:11.073469+0000","last_clean":"2024-04-14T15:29:11.073469+0000","last_became_active":"2024-04-14T15:27:14.211957+0000","last_became_peered":"2024-04-14T15:27:14.211957+0000","last_unstale":"2024-04-14T15:29:11.073469+0000","last_undegraded":"2024-04-14T15:29:11.073469+0000","last_fullsized":"2024-04-14T15:29:11.073469+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"806'19","last_scrub_stamp":"2024-04-14T15:29:07.987120+0000","last_deep_scrub":"806'19","last_deep_scrub_stamp":"2024-04-14T15:25:56.405833+0000","last_clean_scrub_stamp":"2024-04-14T15:29:07.987120+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:21.848137+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":1468,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.157101+0000","last_change":"2024-04-14T15:28:15.145281+0000","last_active":"2024-04-14T15:29:11.157101+0000","last_peered":"2024-04-14T15:29:11.157101+0000","last_clean":"2024-04-14T15:29:11.157101+0000","last_became_active":"2024-04-14T15:15:21.891926+0000","last_became_peered":"2024-04-14T15:15:21.891926+0000","last_unstale":"2024-04-14T15:29:11.157101+0000","last_undegraded":"2024-04-14T15:29:11.157101+0000","last_fullsized":"2024-04-14T15:29:11.157101+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:15.145203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:28:15.145203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:41.361332+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1785,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.077983+0000","last_change":"2024-04-14T15:28:23.153515+0000","last_active":"2024-04-14T15:29:11.077983+0000","last_peered":"2024-04-14T15:29:11.077983+0000","last_clean":"2024-04-14T15:29:11.077983+0000","last_became_active":"2024-04-14T15:09:47.041486+0000","last_became_peered":"2024-04-14T15:09:47.041486+0000","last_unstale":"2024-04-14T15:29:11.077983+0000","last_undegraded":"2024-04-14T15:29:11.077983+0000","last_fullsized":"2024-04-14T15:29:11.077983+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:23.153427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:28:23.153427+0000","last_clean_scrub_stamp":"2024-04-14T15:28:23.153427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:35.509872+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1774,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.073710+0000","last_change":"2024-04-14T15:28:40.934665+0000","last_active":"2024-04-14T15:29:11.073710+0000","last_peered":"2024-04-14T15:29:11.073710+0000","last_clean":"2024-04-14T15:29:11.073710+0000","last_became_active":"2024-04-14T15:09:49.058904+0000","last_became_peered":"2024-04-14T15:09:49.058904+0000","last_unstale":"2024-04-14T15:29:11.073710+0000","last_undegraded":"2024-04-14T15:29:11.073710+0000","last_fullsized":"2024-04-14T15:29:11.073710+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:40.934551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:28:40.934551+0000","last_clean_scrub_stamp":"2024-04-14T15:28:40.934551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:57.015993+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":569,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.157927+0000","last_change":"2024-04-14T15:29:07.117269+0000","last_active":"2024-04-14T15:29:11.157927+0000","last_peered":"2024-04-14T15:29:11.157927+0000","last_clean":"2024-04-14T15:29:11.157927+0000","last_became_active":"2024-04-14T15:26:27.068240+0000","last_became_peered":"2024-04-14T15:26:27.068240+0000","last_unstale":"2024-04-14T15:29:11.157927+0000","last_undegraded":"2024-04-14T15:29:11.157927+0000","last_fullsized":"2024-04-14T15:29:11.157927+0000","mapping_epoch":828,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":829,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:29:07.117128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:29:07.117128+0000","last_clean_scrub_stamp":"2024-04-14T15:29:07.117128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:08.063967+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"479'34","reported_seq":1520,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.156413+0000","last_change":"2024-04-14T15:28:56.794095+0000","last_active":"2024-04-14T15:29:11.156413+0000","last_peered":"2024-04-14T15:29:11.156413+0000","last_clean":"2024-04-14T15:29:11.156413+0000","last_became_active":"2024-04-14T15:15:21.884310+0000","last_became_peered":"2024-04-14T15:15:21.884310+0000","last_unstale":"2024-04-14T15:29:11.156413+0000","last_undegraded":"2024-04-14T15:29:11.156413+0000","last_fullsized":"2024-04-14T15:29:11.156413+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'34","last_scrub_stamp":"2024-04-14T15:28:56.793981+0000","last_deep_scrub":"306'22","last_deep_scrub_stamp":"2024-04-14T15:16:13.898106+0000","last_clean_scrub_stamp":"2024-04-14T15:28:56.793981+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:06.482098+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":1791,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.080029+0000","last_change":"2024-04-14T15:28:32.100163+0000","last_active":"2024-04-14T15:29:11.080029+0000","last_peered":"2024-04-14T15:29:11.080029+0000","last_clean":"2024-04-14T15:29:11.080029+0000","last_became_active":"2024-04-14T15:09:43.017351+0000","last_became_peered":"2024-04-14T15:09:43.017351+0000","last_unstale":"2024-04-14T15:29:11.080029+0000","last_undegraded":"2024-04-14T15:29:11.080029+0000","last_fullsized":"2024-04-14T15:29:11.080029+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:32.100084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:28:32.100084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:33.999154+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1791,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.073713+0000","last_change":"2024-04-14T15:28:42.918254+0000","last_active":"2024-04-14T15:29:11.073713+0000","last_peered":"2024-04-14T15:29:11.073713+0000","last_clean":"2024-04-14T15:29:11.073713+0000","last_became_active":"2024-04-14T15:27:14.212502+0000","last_became_peered":"2024-04-14T15:27:14.212502+0000","last_unstale":"2024-04-14T15:29:11.073713+0000","last_undegraded":"2024-04-14T15:29:11.073713+0000","last_fullsized":"2024-04-14T15:29:11.073713+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:42.918122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:32.403077+0000","last_clean_scrub_stamp":"2024-04-14T15:28:42.918122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:50.282002+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1789,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.156506+0000","last_change":"2024-04-14T15:28:18.170713+0000","last_active":"2024-04-14T15:29:11.156506+0000","last_peered":"2024-04-14T15:29:11.156506+0000","last_clean":"2024-04-14T15:29:11.156506+0000","last_became_active":"2024-04-14T15:27:14.221125+0000","last_became_peered":"2024-04-14T15:27:14.221125+0000","last_unstale":"2024-04-14T15:29:11.156506+0000","last_undegraded":"2024-04-14T15:29:11.156506+0000","last_fullsized":"2024-04-14T15:29:11.156506+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:18.170628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:28:18.170628+0000","last_clean_scrub_stamp":"2024-04-14T15:28:18.170628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:36.341370+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":5,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":2006,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1160,"num_read_kb":1155,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1119,"num_read_kb":876,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":152,"num_bytes_recovered":2134,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":811,"ondisk_log_size":811,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103409,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":93712,"kb_used_data":8380,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":93651440,"statfs":{"total":95995035648,"available":95899074560,"internally_reserved":0,"allocated":8581120,"data_stored":4596332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":87357898},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397748,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":97552,"kb_used_data":8060,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":93647600,"statfs":{"total":95995035648,"available":95895142400,"internally_reserved":0,"allocated":8253440,"data_stored":4268950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":91617734},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575118,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":111460,"kb_used_data":8088,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93633692,"statfs":{"total":95995035648,"available":95880900608,"internally_reserved":0,"allocated":8282112,"data_stored":4268956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738643,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":119772,"kb_used_data":8072,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93625380,"statfs":{"total":95995035648,"available":95872389120,"internally_reserved":0,"allocated":8265728,"data_stored":4269310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":114358727},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411594,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":120008,"kb_used_data":8312,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93625144,"statfs":{"total":95995035648,"available":95872147456,"internally_reserved":0,"allocated":8511488,"data_stored":4531075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182660,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":98160,"kb_used_data":8668,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":93646992,"statfs":{"total":95995035648,"available":95894519808,"internally_reserved":0,"allocated":8876032,"data_stored":4859650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":91617735},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986430,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":111444,"kb_used_data":8068,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93633708,"statfs":{"total":95995035648,"available":95880916992,"internally_reserved":0,"allocated":8261632,"data_stored":4268548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659387,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":112012,"kb_used_data":8632,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93633140,"statfs":{"total":95995035648,"available":95880335360,"internally_reserved":0,"allocated":8839168,"data_stored":4858370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-14T15:29:17.472 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph pg dump --format=json 2024-04-14T15:29:17.691 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:17 smithi090 ceph-mon[41670]: 2.12 scrub starts 2024-04-14T15:29:17.691 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:17 smithi090 ceph-mon[41670]: 2.12 scrub ok 2024-04-14T15:29:17.691 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:17 smithi090 ceph-mon[41670]: 4.7 scrub starts 2024-04-14T15:29:17.691 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:17 smithi090 ceph-mon[41670]: 4.7 scrub ok 2024-04-14T15:29:17.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:17 smithi090 ceph-mon[41670]: from='client.38415 v1:172.21.15.90:0/219640531' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:29:17.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:17 smithi090 ceph-mon[30399]: 2.12 scrub starts 2024-04-14T15:29:17.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:17 smithi090 ceph-mon[30399]: 2.12 scrub ok 2024-04-14T15:29:17.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:17 smithi090 ceph-mon[30399]: 4.7 scrub starts 2024-04-14T15:29:17.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:17 smithi090 ceph-mon[30399]: 4.7 scrub ok 2024-04-14T15:29:17.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:17 smithi090 ceph-mon[30399]: from='client.38415 v1:172.21.15.90:0/219640531' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:29:17.716 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:29:17.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:17 smithi092 ceph-mon[33966]: 2.12 scrub starts 2024-04-14T15:29:17.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:17 smithi092 ceph-mon[33966]: 2.12 scrub ok 2024-04-14T15:29:17.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:17 smithi092 ceph-mon[33966]: 4.7 scrub starts 2024-04-14T15:29:17.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:17 smithi092 ceph-mon[33966]: 4.7 scrub ok 2024-04-14T15:29:17.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:17 smithi092 ceph-mon[33966]: from='client.38415 v1:172.21.15.90:0/219640531' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:29:18.842 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:18 smithi090 ceph-mon[41670]: 6.a scrub starts 2024-04-14T15:29:18.843 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:18 smithi090 ceph-mon[41670]: 6.a scrub ok 2024-04-14T15:29:18.843 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:18 smithi090 ceph-mon[41670]: 11.6 scrub starts 2024-04-14T15:29:18.843 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:18 smithi090 ceph-mon[41670]: 11.6 scrub ok 2024-04-14T15:29:18.843 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:18 smithi090 ceph-mon[41670]: pgmap v1415: 174 pgs: 174 active+clean; 582 KiB data, 844 MiB used, 714 GiB / 715 GiB avail; 652 B/s rd, 0 op/s 2024-04-14T15:29:18.843 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:18 smithi090 ceph-mon[41670]: 2.19 scrub starts 2024-04-14T15:29:18.843 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:18 smithi090 ceph-mon[41670]: 2.19 scrub ok 2024-04-14T15:29:18.843 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:18 smithi090 ceph-mon[41670]: purged_snaps scrub starts 2024-04-14T15:29:18.843 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:18 smithi090 ceph-mon[41670]: purged_snaps scrub ok 2024-04-14T15:29:18.843 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:18 smithi090 ceph-mon[30399]: 6.a scrub starts 2024-04-14T15:29:18.843 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:18 smithi090 ceph-mon[30399]: 6.a scrub ok 2024-04-14T15:29:18.843 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:18 smithi090 ceph-mon[30399]: 11.6 scrub starts 2024-04-14T15:29:18.843 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:18 smithi090 ceph-mon[30399]: 11.6 scrub ok 2024-04-14T15:29:18.844 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:18 smithi090 ceph-mon[30399]: pgmap v1415: 174 pgs: 174 active+clean; 582 KiB data, 844 MiB used, 714 GiB / 715 GiB avail; 652 B/s rd, 0 op/s 2024-04-14T15:29:18.844 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:18 smithi090 ceph-mon[30399]: 2.19 scrub starts 2024-04-14T15:29:18.844 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:18 smithi090 ceph-mon[30399]: 2.19 scrub ok 2024-04-14T15:29:18.844 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:18 smithi090 ceph-mon[30399]: purged_snaps scrub starts 2024-04-14T15:29:18.844 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:18 smithi090 ceph-mon[30399]: purged_snaps scrub ok 2024-04-14T15:29:18.873 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:18 smithi092 ceph-mon[33966]: 6.a scrub starts 2024-04-14T15:29:18.873 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:18 smithi092 ceph-mon[33966]: 6.a scrub ok 2024-04-14T15:29:18.873 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:18 smithi092 ceph-mon[33966]: 11.6 scrub starts 2024-04-14T15:29:18.873 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:18 smithi092 ceph-mon[33966]: 11.6 scrub ok 2024-04-14T15:29:18.873 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:18 smithi092 ceph-mon[33966]: pgmap v1415: 174 pgs: 174 active+clean; 582 KiB data, 844 MiB used, 714 GiB / 715 GiB avail; 652 B/s rd, 0 op/s 2024-04-14T15:29:18.873 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:18 smithi092 ceph-mon[33966]: 2.19 scrub starts 2024-04-14T15:29:18.873 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:18 smithi092 ceph-mon[33966]: 2.19 scrub ok 2024-04-14T15:29:18.873 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:18 smithi092 ceph-mon[33966]: purged_snaps scrub starts 2024-04-14T15:29:18.873 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:18 smithi092 ceph-mon[33966]: purged_snaps scrub ok 2024-04-14T15:29:19.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.363+0000 7fca5b839700 1 -- 172.21.15.90:0/2650160633 >> v1:172.21.15.90:6789/0 conn(0x7fca54108880 legacy=0x7fca54108c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:29:19.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.363+0000 7fca5b839700 1 -- 172.21.15.90:0/2650160633 shutdown_connections 2024-04-14T15:29:19.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.363+0000 7fca5b839700 1 -- 172.21.15.90:0/2650160633 >> 172.21.15.90:0/2650160633 conn(0x7fca540fe3d0 msgr2=0x7fca541007f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:29:19.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.363+0000 7fca5b839700 1 -- 172.21.15.90:0/2650160633 shutdown_connections 2024-04-14T15:29:19.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.363+0000 7fca5b839700 1 -- 172.21.15.90:0/2650160633 wait complete. 2024-04-14T15:29:19.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.364+0000 7fca5b839700 1 librados: starting msgr at 2024-04-14T15:29:19.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.364+0000 7fca5b839700 1 librados: starting objecter 2024-04-14T15:29:19.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.364+0000 7fca5b839700 1 Processor -- start 2024-04-14T15:29:19.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.364+0000 7fca5b839700 1 -- start start 2024-04-14T15:29:19.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.364+0000 7fca5b839700 1 librados: setting wanted keys 2024-04-14T15:29:19.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.364+0000 7fca5b839700 1 librados: calling monclient init 2024-04-14T15:29:19.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.364+0000 7fca5b839700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fca540769e0 con 0x7fca5410b730 2024-04-14T15:29:19.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.364+0000 7fca5b839700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fca54076c10 con 0x7fca5410f300 2024-04-14T15:29:19.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.364+0000 7fca5b839700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fca541ac310 con 0x7fca54108880 2024-04-14T15:29:19.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.364+0000 7fca58dd4700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fca5410b730 0x7fca540753f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:37544/0 (socket says 172.21.15.90:37544) 2024-04-14T15:29:19.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.364+0000 7fca58dd4700 1 -- 172.21.15.90:0/3505720627 learned_addr learned my addr 172.21.15.90:0/3505720627 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:29:19.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.364+0000 7fca595d5700 1 --1- 172.21.15.90:0/3505720627 >> v1:172.21.15.90:6790/0 conn(0x7fca54108880 0x7fca54074ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6790/0 says I am v1:172.21.15.90:42586/0 (socket says 172.21.15.90:42586) 2024-04-14T15:29:19.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.364+0000 7fca4a7fc700 10 client.?.objecter ms_handle_connect 0x7fca5410b730 2024-04-14T15:29:19.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.364+0000 7fca4a7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:29:19.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.364+0000 7fca4a7fc700 10 client.?.objecter ms_handle_connect 0x7fca54108880 2024-04-14T15:29:19.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.364+0000 7fca4a7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:29:19.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.364+0000 7fca4a7fc700 10 client.?.objecter ms_handle_connect 0x7fca5410f300 2024-04-14T15:29:19.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.364+0000 7fca4a7fc700 10 client.?.objecter resend_mon_ops 2024-04-14T15:29:19.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.365+0000 7fca4a7fc700 1 -- 172.21.15.90:0/3505720627 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1143495752 0 0) 0x7fca541ac310 con 0x7fca54108880 2024-04-14T15:29:19.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.365+0000 7fca4a7fc700 1 -- 172.21.15.90:0/3505720627 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fca400031f0 con 0x7fca54108880 2024-04-14T15:29:19.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.365+0000 7fca4a7fc700 1 -- 172.21.15.90:0/3505720627 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1638419006 0 0) 0x7fca540769e0 con 0x7fca5410b730 2024-04-14T15:29:19.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.365+0000 7fca4a7fc700 1 -- 172.21.15.90:0/3505720627 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fca541ac310 con 0x7fca5410b730 2024-04-14T15:29:19.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.365+0000 7fca4a7fc700 1 -- 172.21.15.90:0/3505720627 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1387191234 0 0) 0x7fca54076c10 con 0x7fca5410f300 2024-04-14T15:29:19.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.365+0000 7fca4a7fc700 1 -- 172.21.15.90:0/3505720627 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fca540769e0 con 0x7fca5410f300 2024-04-14T15:29:19.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.365+0000 7fca4a7fc700 1 -- 172.21.15.90:0/3505720627 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1228559977 0 0) 0x7fca541ac310 con 0x7fca5410b730 2024-04-14T15:29:19.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.365+0000 7fca4a7fc700 1 -- 172.21.15.90:0/3505720627 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fca54076c10 con 0x7fca5410b730 2024-04-14T15:29:19.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.365+0000 7fca4a7fc700 1 -- 172.21.15.90:0/3505720627 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fca44001890 con 0x7fca5410b730 2024-04-14T15:29:19.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.365+0000 7fca4a7fc700 1 -- 172.21.15.90:0/3505720627 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3146804592 0 0) 0x7fca400031f0 con 0x7fca54108880 2024-04-14T15:29:19.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.365+0000 7fca4a7fc700 1 -- 172.21.15.90:0/3505720627 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fca541ac310 con 0x7fca54108880 2024-04-14T15:29:19.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.365+0000 7fca4a7fc700 1 -- 172.21.15.90:0/3505720627 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 772808765 0 0) 0x7fca540769e0 con 0x7fca5410f300 2024-04-14T15:29:19.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.365+0000 7fca4a7fc700 1 -- 172.21.15.90:0/3505720627 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fca400031f0 con 0x7fca5410f300 2024-04-14T15:29:19.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.365+0000 7fca4a7fc700 1 -- 172.21.15.90:0/3505720627 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fca50001430 con 0x7fca54108880 2024-04-14T15:29:19.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.365+0000 7fca4a7fc700 1 -- 172.21.15.90:0/3505720627 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fca4c001af0 con 0x7fca5410f300 2024-04-14T15:29:19.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.365+0000 7fca4a7fc700 1 -- 172.21.15.90:0/3505720627 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3860936582 0 0) 0x7fca54076c10 con 0x7fca5410b730 2024-04-14T15:29:19.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.365+0000 7fca4a7fc700 1 -- 172.21.15.90:0/3505720627 >> v1:172.21.15.90:6790/0 conn(0x7fca54108880 legacy=0x7fca54074ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:29:19.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.365+0000 7fca4a7fc700 1 -- 172.21.15.90:0/3505720627 >> v1:172.21.15.92:6789/0 conn(0x7fca5410f300 legacy=0x7fca541a8bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:29:19.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.365+0000 7fca4a7fc700 1 -- 172.21.15.90:0/3505720627 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca541ad4f0 con 0x7fca5410b730 2024-04-14T15:29:19.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.365+0000 7fca5b839700 1 -- 172.21.15.90:0/3505720627 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fca541ac540 con 0x7fca5410b730 2024-04-14T15:29:19.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.365+0000 7fca5b839700 10 client.38421.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:29:19.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.365+0000 7fca5b839700 1 -- 172.21.15.90:0/3505720627 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fca541aca00 con 0x7fca5410b730 2024-04-14T15:29:19.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.365+0000 7fca5b839700 1 librados: init done 2024-04-14T15:29:19.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.365+0000 7fca4a7fc700 1 -- 172.21.15.90:0/3505720627 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fca44003b20 con 0x7fca5410b730 2024-04-14T15:29:19.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.365+0000 7fca4a7fc700 1 -- 172.21.15.90:0/3505720627 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fca44005dd0 con 0x7fca5410b730 2024-04-14T15:29:19.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.366+0000 7fca5b839700 1 -- 172.21.15.90:0/3505720627 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca38005150 con 0x7fca5410b730 2024-04-14T15:29:19.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.367+0000 7fca4a7fc700 1 -- 172.21.15.90:0/3505720627 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fca44003680 con 0x7fca5410b730 2024-04-14T15:29:19.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.367+0000 7fca4a7fc700 1 -- 172.21.15.90:0/3505720627 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(925..925 src has 467..925) ==== 7517+0+0 (unknown 55019603 0 0) 0x7fca4408a730 con 0x7fca5410b730 2024-04-14T15:29:19.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.367+0000 7fca4a7fc700 10 client.38421.objecter ms_dispatch 0x7fca54001550 osd_map(925..925 src has 467..925) 2024-04-14T15:29:19.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.367+0000 7fca4a7fc700 3 client.38421.objecter handle_osd_map got epochs [925,925] > 0 2024-04-14T15:29:19.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.367+0000 7fca4a7fc700 3 client.38421.objecter handle_osd_map decoding full epoch 925 2024-04-14T15:29:19.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.367+0000 7fca4a7fc700 20 client.38421.objecter dump_active .. 0 homeless 2024-04-14T15:29:19.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.367+0000 7fca4a7fc700 10 client.38421.objecter ms_handle_connect 0x7fca40071400 2024-04-14T15:29:19.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.372+0000 7fca4a7fc700 1 -- 172.21.15.90:0/3505720627 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fca44057430 con 0x7fca5410b730 2024-04-14T15:29:19.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.567+0000 7fca5b839700 1 -- 172.21.15.90:0/3505720627 --> v1:172.21.15.90:6800/420935427 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fca38000bc0 con 0x7fca40071400 2024-04-14T15:29:19.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.579+0000 7fca4a7fc700 1 -- 172.21.15.90:0/3505720627 <== mgr.14610 v1:172.21.15.90:6800/420935427 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+453526 (unknown 2965378022 0 1953870961) 0x7fca38000bc0 con 0x7fca40071400 2024-04-14T15:29:19.585 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-14T15:29:19.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.582+0000 7fca5b839700 10 librados: watch_flush enter 2024-04-14T15:29:19.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.582+0000 7fca5b839700 10 librados: watch_flush exit 2024-04-14T15:29:19.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.582+0000 7fca5b839700 20 client.38421.objecter shutdown clearing up homeless session... 2024-04-14T15:29:19.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.582+0000 7fca5b839700 10 client.38421.objecter successfully canceled tick 2024-04-14T15:29:19.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.582+0000 7fca5b839700 1 -- 172.21.15.90:0/3505720627 >> v1:172.21.15.90:6800/420935427 conn(0x7fca40071400 legacy=0x7fca400738c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:29:19.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.582+0000 7fca5b839700 1 -- 172.21.15.90:0/3505720627 >> v1:172.21.15.90:6789/0 conn(0x7fca5410b730 legacy=0x7fca540753f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:29:19.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.582+0000 7fca5b839700 1 -- 172.21.15.90:0/3505720627 shutdown_connections 2024-04-14T15:29:19.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.582+0000 7fca5b839700 1 -- 172.21.15.90:0/3505720627 >> 172.21.15.90:0/3505720627 conn(0x7fca540fe3d0 msgr2=0x7fca541128b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:29:19.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.582+0000 7fca5b839700 1 -- 172.21.15.90:0/3505720627 shutdown_connections 2024-04-14T15:29:19.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.582+0000 7fca5b839700 1 -- 172.21.15.90:0/3505720627 wait complete. 2024-04-14T15:29:19.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:19.582+0000 7fca5b839700 1 librados: shutdown 2024-04-14T15:29:19.589 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-14T15:29:19.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:19 smithi092 ceph-mon[33966]: 4.1 scrub starts 2024-04-14T15:29:19.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:19 smithi092 ceph-mon[33966]: 4.1 scrub ok 2024-04-14T15:29:19.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:19 smithi092 ceph-mon[33966]: 2.d scrub starts 2024-04-14T15:29:19.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:19 smithi092 ceph-mon[33966]: 2.d scrub ok 2024-04-14T15:29:19.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:19 smithi092 ceph-mon[33966]: osdmap e925: 8 total, 8 up, 8 in 2024-04-14T15:29:19.893 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:19 smithi090 ceph-mon[41670]: 4.1 scrub starts 2024-04-14T15:29:19.893 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:19 smithi090 ceph-mon[41670]: 4.1 scrub ok 2024-04-14T15:29:19.893 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:19 smithi090 ceph-mon[41670]: 2.d scrub starts 2024-04-14T15:29:19.893 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:19 smithi090 ceph-mon[41670]: 2.d scrub ok 2024-04-14T15:29:19.893 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:19 smithi090 ceph-mon[41670]: osdmap e925: 8 total, 8 up, 8 in 2024-04-14T15:29:19.894 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:19 smithi090 ceph-mon[30399]: 4.1 scrub starts 2024-04-14T15:29:19.894 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:19 smithi090 ceph-mon[30399]: 4.1 scrub ok 2024-04-14T15:29:19.894 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:19 smithi090 ceph-mon[30399]: 2.d scrub starts 2024-04-14T15:29:19.894 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:19 smithi090 ceph-mon[30399]: 2.d scrub ok 2024-04-14T15:29:19.894 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:19 smithi090 ceph-mon[30399]: osdmap e925: 8 total, 8 up, 8 in 2024-04-14T15:29:20.549 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":1417,"stamp":"2024-04-14T15:29:19.256012+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596397,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":2314,"num_read_kb":2066,"num_write":729,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":169,"num_bytes_recovered":332641,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":922,"ondisk_log_size":922,"up":522,"acting":522,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":522,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":864072,"kb_used_data":66304,"kb_used_omap":12,"kb_used_meta":797683,"kb_avail":749097144,"statfs":{"total":767960285184,"available":767075475456,"internally_reserved":0,"allocated":67895296,"data_stored":35936725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":816827954},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.181415"},"pg_stats":[{"pgid":"11.7","version":"0'0","reported_seq":561,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.157772+0000","last_change":"2024-04-14T15:28:12.845739+0000","last_active":"2024-04-14T15:29:11.157772+0000","last_peered":"2024-04-14T15:29:11.157772+0000","last_clean":"2024-04-14T15:29:11.157772+0000","last_became_active":"2024-04-14T15:26:27.058780+0000","last_became_peered":"2024-04-14T15:26:27.058780+0000","last_unstale":"2024-04-14T15:29:11.157772+0000","last_undegraded":"2024-04-14T15:29:11.157772+0000","last_fullsized":"2024-04-14T15:29:11.157772+0000","mapping_epoch":828,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":829,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:12.845555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:28:12.845555+0000","last_clean_scrub_stamp":"2024-04-14T15:28:12.845555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:15.174430+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"866'11","reported_seq":2053,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:13.289939+0000","last_change":"2024-04-14T15:28:54.104759+0000","last_active":"2024-04-14T15:29:13.289939+0000","last_peered":"2024-04-14T15:29:13.289939+0000","last_clean":"2024-04-14T15:29:13.289939+0000","last_became_active":"2024-04-14T15:27:14.215386+0000","last_became_peered":"2024-04-14T15:27:14.215386+0000","last_unstale":"2024-04-14T15:29:13.289939+0000","last_undegraded":"2024-04-14T15:29:13.289939+0000","last_fullsized":"2024-04-14T15:29:13.289939+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"866'11","last_scrub_stamp":"2024-04-14T15:28:54.104637+0000","last_deep_scrub":"85'9","last_deep_scrub_stamp":"2024-04-14T15:10:51.950580+0000","last_clean_scrub_stamp":"2024-04-14T15:28:54.104637+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:59.149921+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"479'34","reported_seq":720,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.155840+0000","last_change":"2024-04-14T15:27:55.858026+0000","last_active":"2024-04-14T15:29:11.155840+0000","last_peered":"2024-04-14T15:29:11.155840+0000","last_clean":"2024-04-14T15:29:11.155840+0000","last_became_active":"2024-04-14T15:27:14.219419+0000","last_became_peered":"2024-04-14T15:27:14.219419+0000","last_unstale":"2024-04-14T15:29:11.155840+0000","last_undegraded":"2024-04-14T15:29:11.155840+0000","last_fullsized":"2024-04-14T15:29:11.155840+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'34","last_scrub_stamp":"2024-04-14T15:27:55.857868+0000","last_deep_scrub":"430'22","last_deep_scrub_stamp":"2024-04-14T15:18:39.704883+0000","last_clean_scrub_stamp":"2024-04-14T15:27:55.857868+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:17.371799+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":1802,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:19.105071+0000","last_change":"2024-04-14T15:29:03.215761+0000","last_active":"2024-04-14T15:29:19.105071+0000","last_peered":"2024-04-14T15:29:19.105071+0000","last_clean":"2024-04-14T15:29:19.105071+0000","last_became_active":"2024-04-14T15:09:43.021460+0000","last_became_peered":"2024-04-14T15:09:43.021460+0000","last_unstale":"2024-04-14T15:29:19.105071+0000","last_undegraded":"2024-04-14T15:29:19.105071+0000","last_fullsized":"2024-04-14T15:29:19.105071+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:29:03.215687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:29:03.215687+0000","last_clean_scrub_stamp":"2024-04-14T15:29:03.215687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:14.584017+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":666,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:17.067980+0000","last_change":"2024-04-14T15:29:17.068023+0000","last_active":"2024-04-14T15:29:17.067980+0000","last_peered":"2024-04-14T15:29:17.067980+0000","last_clean":"2024-04-14T15:29:17.067980+0000","last_became_active":"2024-04-14T15:27:14.208471+0000","last_became_peered":"2024-04-14T15:27:14.208471+0000","last_unstale":"2024-04-14T15:29:17.067980+0000","last_undegraded":"2024-04-14T15:29:17.067980+0000","last_fullsized":"2024-04-14T15:29:17.067980+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:29:17.067892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:15:40.368219+0000","last_clean_scrub_stamp":"2024-04-14T15:29:17.067892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:32.592496+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":1697,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.114285+0000","last_change":"2024-04-14T15:28:08.911776+0000","last_active":"2024-04-14T15:29:11.114285+0000","last_peered":"2024-04-14T15:29:11.114285+0000","last_clean":"2024-04-14T15:29:11.114285+0000","last_became_active":"2024-04-14T15:27:14.202698+0000","last_became_peered":"2024-04-14T15:27:14.202698+0000","last_unstale":"2024-04-14T15:29:11.114285+0000","last_undegraded":"2024-04-14T15:29:11.114285+0000","last_fullsized":"2024-04-14T15:29:11.114285+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:08.911665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:28:08.911665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:29.103837+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"479'15","reported_seq":1821,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:18.450469+0000","last_change":"2024-04-14T15:28:40.107670+0000","last_active":"2024-04-14T15:29:18.450469+0000","last_peered":"2024-04-14T15:29:18.450469+0000","last_clean":"2024-04-14T15:29:18.450469+0000","last_became_active":"2024-04-14T15:09:45.037179+0000","last_became_peered":"2024-04-14T15:09:45.037179+0000","last_unstale":"2024-04-14T15:29:18.450469+0000","last_undegraded":"2024-04-14T15:29:18.450469+0000","last_fullsized":"2024-04-14T15:29:18.450469+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'15","last_scrub_stamp":"2024-04-14T15:28:40.107560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:28:40.107560+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:55.659006+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":1803,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:18.450734+0000","last_change":"2024-04-14T15:29:18.109734+0000","last_active":"2024-04-14T15:29:18.450734+0000","last_peered":"2024-04-14T15:29:18.450734+0000","last_clean":"2024-04-14T15:29:18.450734+0000","last_became_active":"2024-04-14T15:09:43.020044+0000","last_became_peered":"2024-04-14T15:09:43.020044+0000","last_unstale":"2024-04-14T15:29:18.450734+0000","last_undegraded":"2024-04-14T15:29:18.450734+0000","last_fullsized":"2024-04-14T15:29:18.450734+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:29:18.109648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:29:18.109648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:32.113503+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1802,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:19.104626+0000","last_change":"2024-04-14T15:29:15.142803+0000","last_active":"2024-04-14T15:29:19.104626+0000","last_peered":"2024-04-14T15:29:19.104626+0000","last_clean":"2024-04-14T15:29:19.104626+0000","last_became_active":"2024-04-14T15:09:47.046153+0000","last_became_peered":"2024-04-14T15:09:47.046153+0000","last_unstale":"2024-04-14T15:29:19.104626+0000","last_undegraded":"2024-04-14T15:29:19.104626+0000","last_fullsized":"2024-04-14T15:29:19.104626+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:29:15.142713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:29:15.142713+0000","last_clean_scrub_stamp":"2024-04-14T15:29:15.142713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:39.150631+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1788,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:18.450220+0000","last_change":"2024-04-14T15:29:11.115555+0000","last_active":"2024-04-14T15:29:18.450220+0000","last_peered":"2024-04-14T15:29:18.450220+0000","last_clean":"2024-04-14T15:29:18.450220+0000","last_became_active":"2024-04-14T15:09:49.061495+0000","last_became_peered":"2024-04-14T15:09:49.061495+0000","last_unstale":"2024-04-14T15:29:18.450220+0000","last_undegraded":"2024-04-14T15:29:18.450220+0000","last_fullsized":"2024-04-14T15:29:18.450220+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:29:11.115418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:29:11.115418+0000","last_clean_scrub_stamp":"2024-04-14T15:29:11.115418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:37.190866+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":1705,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.271406+0000","last_change":"2024-04-14T15:29:11.271444+0000","last_active":"2024-04-14T15:29:11.271406+0000","last_peered":"2024-04-14T15:29:11.271406+0000","last_clean":"2024-04-14T15:29:11.271406+0000","last_became_active":"2024-04-14T15:27:14.202915+0000","last_became_peered":"2024-04-14T15:27:14.202915+0000","last_unstale":"2024-04-14T15:29:11.271406+0000","last_undegraded":"2024-04-14T15:29:11.271406+0000","last_fullsized":"2024-04-14T15:29:11.271406+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:29:11.271313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:29:11.271313+0000","last_clean_scrub_stamp":"2024-04-14T15:29:11.271313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:17.938909+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"479'44","reported_seq":154,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.157090+0000","last_change":"2024-04-14T15:28:20.450354+0000","last_active":"2024-04-14T15:29:11.157090+0000","last_peered":"2024-04-14T15:29:11.157090+0000","last_clean":"2024-04-14T15:29:11.157090+0000","last_became_active":"2024-04-14T15:27:14.210250+0000","last_became_peered":"2024-04-14T15:27:14.210250+0000","last_unstale":"2024-04-14T15:29:11.157090+0000","last_undegraded":"2024-04-14T15:29:11.157090+0000","last_fullsized":"2024-04-14T15:29:11.157090+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'44","last_scrub_stamp":"2024-04-14T15:28:20.450127+0000","last_deep_scrub":"479'44","last_deep_scrub_stamp":"2024-04-14T15:28:20.450127+0000","last_clean_scrub_stamp":"2024-04-14T15:28:20.450127+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:34.892050+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":1810,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:19.105045+0000","last_change":"2024-04-14T15:28:34.329860+0000","last_active":"2024-04-14T15:29:19.105045+0000","last_peered":"2024-04-14T15:29:19.105045+0000","last_clean":"2024-04-14T15:29:19.105045+0000","last_became_active":"2024-04-14T15:09:43.020268+0000","last_became_peered":"2024-04-14T15:09:43.020268+0000","last_unstale":"2024-04-14T15:29:19.105045+0000","last_undegraded":"2024-04-14T15:29:19.105045+0000","last_fullsized":"2024-04-14T15:29:19.105045+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:34.329762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:13:34.342793+0000","last_clean_scrub_stamp":"2024-04-14T15:28:34.329762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:57.790997+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1794,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:19.104607+0000","last_change":"2024-04-14T15:29:07.104656+0000","last_active":"2024-04-14T15:29:19.104607+0000","last_peered":"2024-04-14T15:29:19.104607+0000","last_clean":"2024-04-14T15:29:19.104607+0000","last_became_active":"2024-04-14T15:09:47.045542+0000","last_became_peered":"2024-04-14T15:09:47.045542+0000","last_unstale":"2024-04-14T15:29:19.104607+0000","last_undegraded":"2024-04-14T15:29:19.104607+0000","last_fullsized":"2024-04-14T15:29:19.104607+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:29:07.104582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:25:56.701776+0000","last_clean_scrub_stamp":"2024-04-14T15:29:07.104582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:12.232801+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1777,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.114645+0000","last_change":"2024-04-14T15:28:28.926657+0000","last_active":"2024-04-14T15:29:11.114645+0000","last_peered":"2024-04-14T15:29:11.114645+0000","last_clean":"2024-04-14T15:29:11.114645+0000","last_became_active":"2024-04-14T15:09:49.065397+0000","last_became_peered":"2024-04-14T15:09:49.065397+0000","last_unstale":"2024-04-14T15:29:11.114645+0000","last_undegraded":"2024-04-14T15:29:11.114645+0000","last_fullsized":"2024-04-14T15:29:11.114645+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:28.926558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:28:28.926558+0000","last_clean_scrub_stamp":"2024-04-14T15:28:28.926558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:29.568517+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"91'130","reported_seq":744,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.155745+0000","last_change":"2024-04-14T15:28:21.800962+0000","last_active":"2024-04-14T15:29:11.155745+0000","last_peered":"2024-04-14T15:29:11.155745+0000","last_clean":"2024-04-14T15:29:11.155745+0000","last_became_active":"2024-04-14T15:26:27.064317+0000","last_became_peered":"2024-04-14T15:26:27.064317+0000","last_unstale":"2024-04-14T15:29:11.155745+0000","last_undegraded":"2024-04-14T15:29:11.155745+0000","last_fullsized":"2024-04-14T15:29:11.155745+0000","mapping_epoch":828,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":829,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:28:21.800809+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:28:21.800809+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":70,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:32.782432+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":1699,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:19.104569+0000","last_change":"2024-04-14T15:27:58.390592+0000","last_active":"2024-04-14T15:29:19.104569+0000","last_peered":"2024-04-14T15:29:19.104569+0000","last_clean":"2024-04-14T15:29:19.104569+0000","last_became_active":"2024-04-14T15:12:56.474322+0000","last_became_peered":"2024-04-14T15:12:56.474322+0000","last_unstale":"2024-04-14T15:29:19.104569+0000","last_undegraded":"2024-04-14T15:29:19.104569+0000","last_fullsized":"2024-04-14T15:29:19.104569+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:27:58.390511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:27:58.390511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"780'39","reported_seq":1898,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:18.450273+0000","last_change":"2024-04-14T15:28:43.884357+0000","last_active":"2024-04-14T15:29:18.450273+0000","last_peered":"2024-04-14T15:29:18.450273+0000","last_clean":"2024-04-14T15:29:18.450273+0000","last_became_active":"2024-04-14T15:13:12.933907+0000","last_became_peered":"2024-04-14T15:13:12.933907+0000","last_unstale":"2024-04-14T15:29:18.450273+0000","last_undegraded":"2024-04-14T15:29:18.450273+0000","last_fullsized":"2024-04-14T15:29:18.450273+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"780'39","last_scrub_stamp":"2024-04-14T15:28:43.884242+0000","last_deep_scrub":"780'39","last_deep_scrub_stamp":"2024-04-14T15:25:24.296779+0000","last_clean_scrub_stamp":"2024-04-14T15:28:43.884242+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:52.036846+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":1788,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.114038+0000","last_change":"2024-04-14T15:28:38.020549+0000","last_active":"2024-04-14T15:29:11.114038+0000","last_peered":"2024-04-14T15:29:11.114038+0000","last_clean":"2024-04-14T15:29:11.114038+0000","last_became_active":"2024-04-14T15:09:43.022351+0000","last_became_peered":"2024-04-14T15:09:43.022351+0000","last_unstale":"2024-04-14T15:29:11.114038+0000","last_undegraded":"2024-04-14T15:29:11.114038+0000","last_fullsized":"2024-04-14T15:29:11.114038+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:38.020454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:28:38.020454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:56.012668+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1787,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:18.449543+0000","last_change":"2024-04-14T15:28:03.078744+0000","last_active":"2024-04-14T15:29:18.449543+0000","last_peered":"2024-04-14T15:29:18.449543+0000","last_clean":"2024-04-14T15:29:18.449543+0000","last_became_active":"2024-04-14T15:09:47.042810+0000","last_became_peered":"2024-04-14T15:09:47.042810+0000","last_unstale":"2024-04-14T15:29:18.449543+0000","last_undegraded":"2024-04-14T15:29:18.449543+0000","last_fullsized":"2024-04-14T15:29:18.449543+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:03.078658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:28:03.078658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:22.969772+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1790,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:19.105279+0000","last_change":"2024-04-14T15:28:22.289204+0000","last_active":"2024-04-14T15:29:19.105279+0000","last_peered":"2024-04-14T15:29:19.105279+0000","last_clean":"2024-04-14T15:29:19.105279+0000","last_became_active":"2024-04-14T15:09:49.069594+0000","last_became_peered":"2024-04-14T15:09:49.069594+0000","last_unstale":"2024-04-14T15:29:19.105279+0000","last_undegraded":"2024-04-14T15:29:19.105279+0000","last_fullsized":"2024-04-14T15:29:19.105279+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:22.289129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:28:22.289129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:47.280657+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"91'130","reported_seq":2159,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.113541+0000","last_change":"2024-04-14T15:28:21.923496+0000","last_active":"2024-04-14T15:29:11.113541+0000","last_peered":"2024-04-14T15:29:11.113541+0000","last_clean":"2024-04-14T15:29:11.113541+0000","last_became_active":"2024-04-14T15:21:09.020300+0000","last_became_peered":"2024-04-14T15:21:09.020300+0000","last_unstale":"2024-04-14T15:29:11.113541+0000","last_undegraded":"2024-04-14T15:29:11.113541+0000","last_fullsized":"2024-04-14T15:29:11.113541+0000","mapping_epoch":189,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":544,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:28:21.923396+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:28:21.923396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:24.264160+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":143,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.156420+0000","last_change":"2024-04-14T15:28:27.378322+0000","last_active":"2024-04-14T15:29:11.156420+0000","last_peered":"2024-04-14T15:29:11.156420+0000","last_clean":"2024-04-14T15:29:11.156420+0000","last_became_active":"2024-04-14T15:27:14.215727+0000","last_became_peered":"2024-04-14T15:27:14.215727+0000","last_unstale":"2024-04-14T15:29:11.156420+0000","last_undegraded":"2024-04-14T15:29:11.156420+0000","last_fullsized":"2024-04-14T15:29:11.156420+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:27.378216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:28:27.378216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:53.912712+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"479'25","reported_seq":1856,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:18.449763+0000","last_change":"2024-04-14T15:28:35.940759+0000","last_active":"2024-04-14T15:29:18.449763+0000","last_peered":"2024-04-14T15:29:18.449763+0000","last_clean":"2024-04-14T15:29:18.449763+0000","last_became_active":"2024-04-14T15:27:14.209858+0000","last_became_peered":"2024-04-14T15:27:14.209858+0000","last_unstale":"2024-04-14T15:29:18.449763+0000","last_undegraded":"2024-04-14T15:29:18.449763+0000","last_fullsized":"2024-04-14T15:29:18.449763+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'25","last_scrub_stamp":"2024-04-14T15:28:35.940648+0000","last_deep_scrub":"479'25","last_deep_scrub_stamp":"2024-04-14T15:20:03.571160+0000","last_clean_scrub_stamp":"2024-04-14T15:28:35.940648+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:37.185385+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":1804,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:18.448853+0000","last_change":"2024-04-14T15:28:11.178171+0000","last_active":"2024-04-14T15:29:18.448853+0000","last_peered":"2024-04-14T15:29:18.448853+0000","last_clean":"2024-04-14T15:29:18.448853+0000","last_became_active":"2024-04-14T15:09:43.018871+0000","last_became_peered":"2024-04-14T15:09:43.018871+0000","last_unstale":"2024-04-14T15:29:18.448853+0000","last_undegraded":"2024-04-14T15:29:18.448853+0000","last_fullsized":"2024-04-14T15:29:18.448853+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:11.178093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:28:11.178093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:34.848304+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"400'9","reported_seq":2039,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:19.104961+0000","last_change":"2024-04-14T15:28:39.336656+0000","last_active":"2024-04-14T15:29:19.104961+0000","last_peered":"2024-04-14T15:29:19.104961+0000","last_clean":"2024-04-14T15:29:19.104961+0000","last_became_active":"2024-04-14T15:09:47.044085+0000","last_became_peered":"2024-04-14T15:09:47.044085+0000","last_unstale":"2024-04-14T15:29:19.104961+0000","last_undegraded":"2024-04-14T15:29:19.104961+0000","last_fullsized":"2024-04-14T15:29:19.104961+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'9","last_scrub_stamp":"2024-04-14T15:28:39.336522+0000","last_deep_scrub":"400'9","last_deep_scrub_stamp":"2024-04-14T15:17:54.845881+0000","last_clean_scrub_stamp":"2024-04-14T15:28:39.336522+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:41.291234+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1777,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:18.450311+0000","last_change":"2024-04-14T15:28:47.954798+0000","last_active":"2024-04-14T15:29:18.450311+0000","last_peered":"2024-04-14T15:29:18.450311+0000","last_clean":"2024-04-14T15:29:18.450311+0000","last_became_active":"2024-04-14T15:09:49.065274+0000","last_became_peered":"2024-04-14T15:09:49.065274+0000","last_unstale":"2024-04-14T15:29:18.450311+0000","last_undegraded":"2024-04-14T15:29:18.450311+0000","last_fullsized":"2024-04-14T15:29:18.450311+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:47.954691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:18.588546+0000","last_clean_scrub_stamp":"2024-04-14T15:28:47.954691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:57.549046+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"91'130","reported_seq":757,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.156097+0000","last_change":"2024-04-14T15:28:13.866095+0000","last_active":"2024-04-14T15:29:11.156097+0000","last_peered":"2024-04-14T15:29:11.156097+0000","last_clean":"2024-04-14T15:29:11.156097+0000","last_became_active":"2024-04-14T15:27:17.056550+0000","last_became_peered":"2024-04-14T15:27:17.056550+0000","last_unstale":"2024-04-14T15:29:11.156097+0000","last_undegraded":"2024-04-14T15:29:11.156097+0000","last_fullsized":"2024-04-14T15:29:11.156097+0000","mapping_epoch":868,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":869,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:28:13.865978+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:28:13.865978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:15.536292+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":1701,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:18.448756+0000","last_change":"2024-04-14T15:29:12.170913+0000","last_active":"2024-04-14T15:29:18.448756+0000","last_peered":"2024-04-14T15:29:18.448756+0000","last_clean":"2024-04-14T15:29:18.448756+0000","last_became_active":"2024-04-14T15:12:56.487615+0000","last_became_peered":"2024-04-14T15:12:56.487615+0000","last_unstale":"2024-04-14T15:29:18.448756+0000","last_undegraded":"2024-04-14T15:29:18.448756+0000","last_fullsized":"2024-04-14T15:29:18.448756+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:29:12.170836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:29:12.170836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:16.691346+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"479'34","reported_seq":1849,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:18.449859+0000","last_change":"2024-04-14T15:28:41.336454+0000","last_active":"2024-04-14T15:29:18.449859+0000","last_peered":"2024-04-14T15:29:18.449859+0000","last_clean":"2024-04-14T15:29:18.449859+0000","last_became_active":"2024-04-14T15:09:45.033331+0000","last_became_peered":"2024-04-14T15:09:45.033331+0000","last_unstale":"2024-04-14T15:29:18.449859+0000","last_undegraded":"2024-04-14T15:29:18.449859+0000","last_fullsized":"2024-04-14T15:29:18.449859+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'34","last_scrub_stamp":"2024-04-14T15:28:41.336355+0000","last_deep_scrub":"479'34","last_deep_scrub_stamp":"2024-04-14T15:24:35.378379+0000","last_clean_scrub_stamp":"2024-04-14T15:28:41.336355+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:04.937189+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":1804,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:18.449016+0000","last_change":"2024-04-14T15:28:58.418841+0000","last_active":"2024-04-14T15:29:18.449016+0000","last_peered":"2024-04-14T15:29:18.449016+0000","last_clean":"2024-04-14T15:29:18.449016+0000","last_became_active":"2024-04-14T15:09:43.020878+0000","last_became_peered":"2024-04-14T15:09:43.020878+0000","last_unstale":"2024-04-14T15:29:18.449016+0000","last_undegraded":"2024-04-14T15:29:18.449016+0000","last_fullsized":"2024-04-14T15:29:18.449016+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:58.418758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:28:58.418758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:13.356881+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"72'8","reported_seq":2025,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:18.450171+0000","last_change":"2024-04-14T15:28:44.883771+0000","last_active":"2024-04-14T15:29:18.450171+0000","last_peered":"2024-04-14T15:29:18.450171+0000","last_clean":"2024-04-14T15:29:18.450171+0000","last_became_active":"2024-04-14T15:09:47.050962+0000","last_became_peered":"2024-04-14T15:09:47.050962+0000","last_unstale":"2024-04-14T15:29:18.450171+0000","last_undegraded":"2024-04-14T15:29:18.450171+0000","last_fullsized":"2024-04-14T15:29:18.450171+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-04-14T15:28:44.883665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:28:44.883665+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:50.216166+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":652,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.157804+0000","last_change":"2024-04-14T15:28:30.732025+0000","last_active":"2024-04-14T15:29:11.157804+0000","last_peered":"2024-04-14T15:29:11.157804+0000","last_clean":"2024-04-14T15:29:11.157804+0000","last_became_active":"2024-04-14T15:26:27.065649+0000","last_became_peered":"2024-04-14T15:26:27.065649+0000","last_unstale":"2024-04-14T15:29:11.157804+0000","last_undegraded":"2024-04-14T15:29:11.157804+0000","last_fullsized":"2024-04-14T15:29:11.157804+0000","mapping_epoch":828,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":829,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:30.731930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:27:23.822301+0000","last_clean_scrub_stamp":"2024-04-14T15:28:30.731930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:36.499768+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"233'8","reported_seq":147,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:14.337619+0000","last_change":"2024-04-14T15:29:14.337619+0000","last_active":"2024-04-14T15:29:14.337619+0000","last_peered":"2024-04-14T15:29:14.337619+0000","last_clean":"2024-04-14T15:29:14.337619+0000","last_became_active":"2024-04-14T15:27:14.216287+0000","last_became_peered":"2024-04-14T15:27:14.216287+0000","last_unstale":"2024-04-14T15:29:14.337619+0000","last_undegraded":"2024-04-14T15:29:14.337619+0000","last_fullsized":"2024-04-14T15:29:14.337619+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'8","last_scrub_stamp":"2024-04-14T15:29:14.337484+0000","last_deep_scrub":"233'8","last_deep_scrub_stamp":"2024-04-14T15:15:01.295800+0000","last_clean_scrub_stamp":"2024-04-14T15:29:14.337484+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:19.023356+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":5,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"479'24","reported_seq":1865,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:18.449868+0000","last_change":"2024-04-14T15:28:58.793618+0000","last_active":"2024-04-14T15:29:18.449868+0000","last_peered":"2024-04-14T15:29:18.449868+0000","last_clean":"2024-04-14T15:29:18.449868+0000","last_became_active":"2024-04-14T15:27:14.223836+0000","last_became_peered":"2024-04-14T15:27:14.223836+0000","last_unstale":"2024-04-14T15:29:18.449868+0000","last_undegraded":"2024-04-14T15:29:18.449868+0000","last_fullsized":"2024-04-14T15:29:18.449868+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'24","last_scrub_stamp":"2024-04-14T15:28:58.793497+0000","last_deep_scrub":"85'16","last_deep_scrub_stamp":"2024-04-14T15:10:47.456169+0000","last_clean_scrub_stamp":"2024-04-14T15:28:58.793497+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:05.252137+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":1797,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.113686+0000","last_change":"2024-04-14T15:28:52.070960+0000","last_active":"2024-04-14T15:29:11.113686+0000","last_peered":"2024-04-14T15:29:11.113686+0000","last_clean":"2024-04-14T15:29:11.113686+0000","last_became_active":"2024-04-14T15:09:43.018601+0000","last_became_peered":"2024-04-14T15:09:43.018601+0000","last_unstale":"2024-04-14T15:29:11.113686+0000","last_undegraded":"2024-04-14T15:29:11.113686+0000","last_fullsized":"2024-04-14T15:29:11.113686+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:52.070853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:28:52.070853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:56.127035+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":749,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.158042+0000","last_change":"2024-04-14T15:29:02.030422+0000","last_active":"2024-04-14T15:29:11.158042+0000","last_peered":"2024-04-14T15:29:11.158042+0000","last_clean":"2024-04-14T15:29:11.158042+0000","last_became_active":"2024-04-14T15:27:14.208646+0000","last_became_peered":"2024-04-14T15:27:14.208646+0000","last_unstale":"2024-04-14T15:29:11.158042+0000","last_undegraded":"2024-04-14T15:29:11.158042+0000","last_fullsized":"2024-04-14T15:29:11.158042+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:29:02.030326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:27:39.778818+0000","last_clean_scrub_stamp":"2024-04-14T15:29:02.030326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:13.927765+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1469,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:19.140956+0000","last_change":"2024-04-14T15:28:26.061582+0000","last_active":"2024-04-14T15:29:19.140956+0000","last_peered":"2024-04-14T15:29:19.140956+0000","last_clean":"2024-04-14T15:29:19.140956+0000","last_became_active":"2024-04-14T15:27:14.198110+0000","last_became_peered":"2024-04-14T15:27:14.198110+0000","last_unstale":"2024-04-14T15:29:19.140956+0000","last_undegraded":"2024-04-14T15:29:19.140956+0000","last_fullsized":"2024-04-14T15:29:19.140956+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:26.061472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:57.214046+0000","last_clean_scrub_stamp":"2024-04-14T15:28:26.061472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:30.621101+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"91'130","reported_seq":1987,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:19.141219+0000","last_change":"2024-04-14T15:28:33.001416+0000","last_active":"2024-04-14T15:29:19.141219+0000","last_peered":"2024-04-14T15:29:19.141219+0000","last_clean":"2024-04-14T15:29:19.141219+0000","last_became_active":"2024-04-14T15:28:31.265380+0000","last_became_peered":"2024-04-14T15:28:31.265380+0000","last_unstale":"2024-04-14T15:29:19.141219+0000","last_undegraded":"2024-04-14T15:29:19.141219+0000","last_fullsized":"2024-04-14T15:29:19.141219+0000","mapping_epoch":189,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":915,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:28:33.001281+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:28:33.001281+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":180,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:59.279973+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":1694,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:18.449623+0000","last_change":"2024-04-14T15:28:55.825563+0000","last_active":"2024-04-14T15:29:18.449623+0000","last_peered":"2024-04-14T15:29:18.449623+0000","last_clean":"2024-04-14T15:29:18.449623+0000","last_became_active":"2024-04-14T15:12:56.427276+0000","last_became_peered":"2024-04-14T15:12:56.427276+0000","last_unstale":"2024-04-14T15:29:18.449623+0000","last_undegraded":"2024-04-14T15:29:18.449623+0000","last_fullsized":"2024-04-14T15:29:18.449623+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:55.825466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:28:55.825466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:05.614908+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"255'10","reported_seq":1674,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:19.139142+0000","last_change":"2024-04-14T15:29:08.730140+0000","last_active":"2024-04-14T15:29:19.139142+0000","last_peered":"2024-04-14T15:29:19.139142+0000","last_clean":"2024-04-14T15:29:19.139142+0000","last_became_active":"2024-04-14T15:15:21.873747+0000","last_became_peered":"2024-04-14T15:15:21.873747+0000","last_unstale":"2024-04-14T15:29:19.139142+0000","last_undegraded":"2024-04-14T15:29:19.139142+0000","last_fullsized":"2024-04-14T15:29:19.139142+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'10","last_scrub_stamp":"2024-04-14T15:29:08.730015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:29:08.730015+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:11.722437+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"479'28","reported_seq":1880,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:18.448956+0000","last_change":"2024-04-14T15:29:17.968781+0000","last_active":"2024-04-14T15:29:18.448956+0000","last_peered":"2024-04-14T15:29:18.448956+0000","last_clean":"2024-04-14T15:29:18.448956+0000","last_became_active":"2024-04-14T15:27:14.212370+0000","last_became_peered":"2024-04-14T15:27:14.212370+0000","last_unstale":"2024-04-14T15:29:18.448956+0000","last_undegraded":"2024-04-14T15:29:18.448956+0000","last_fullsized":"2024-04-14T15:29:18.448956+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'28","last_scrub_stamp":"2024-04-14T15:29:17.968663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:29:17.968663+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:37.237008+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":142,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.157329+0000","last_change":"2024-04-14T15:28:47.360501+0000","last_active":"2024-04-14T15:29:11.157329+0000","last_peered":"2024-04-14T15:29:11.157329+0000","last_clean":"2024-04-14T15:29:11.157329+0000","last_became_active":"2024-04-14T15:27:14.212559+0000","last_became_peered":"2024-04-14T15:27:14.212559+0000","last_unstale":"2024-04-14T15:29:11.157329+0000","last_undegraded":"2024-04-14T15:29:11.157329+0000","last_fullsized":"2024-04-14T15:29:11.157329+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:47.360399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:28:47.360399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:15.273902+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1789,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.114296+0000","last_change":"2024-04-14T15:28:36.036651+0000","last_active":"2024-04-14T15:29:11.114296+0000","last_peered":"2024-04-14T15:29:11.114296+0000","last_clean":"2024-04-14T15:29:11.114296+0000","last_became_active":"2024-04-14T15:27:14.203240+0000","last_became_peered":"2024-04-14T15:27:14.203240+0000","last_unstale":"2024-04-14T15:29:11.114296+0000","last_undegraded":"2024-04-14T15:29:11.114296+0000","last_fullsized":"2024-04-14T15:29:11.114296+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:36.036555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:28:36.036555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:51.099129+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"91'130","reported_seq":2004,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:18.449491+0000","last_change":"2024-04-14T15:28:27.900740+0000","last_active":"2024-04-14T15:29:18.449491+0000","last_peered":"2024-04-14T15:29:18.449491+0000","last_clean":"2024-04-14T15:29:18.449491+0000","last_became_active":"2024-04-14T15:28:27.070386+0000","last_became_peered":"2024-04-14T15:28:27.070386+0000","last_unstale":"2024-04-14T15:29:18.449491+0000","last_undegraded":"2024-04-14T15:29:18.449491+0000","last_fullsized":"2024-04-14T15:29:18.449491+0000","mapping_epoch":885,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":911,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:28:27.900623+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:28:27.900623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:55.303648+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":1693,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.114892+0000","last_change":"2024-04-14T15:28:50.045865+0000","last_active":"2024-04-14T15:29:11.114892+0000","last_peered":"2024-04-14T15:29:11.114892+0000","last_clean":"2024-04-14T15:29:11.114892+0000","last_became_active":"2024-04-14T15:12:56.355254+0000","last_became_peered":"2024-04-14T15:12:56.355254+0000","last_unstale":"2024-04-14T15:29:11.114892+0000","last_undegraded":"2024-04-14T15:29:11.114892+0000","last_fullsized":"2024-04-14T15:29:11.114892+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:50.045738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:28:50.045738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:11.373315+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1790,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:18.449180+0000","last_change":"2024-04-14T15:29:13.978226+0000","last_active":"2024-04-14T15:29:18.449180+0000","last_peered":"2024-04-14T15:29:18.449180+0000","last_clean":"2024-04-14T15:29:18.449180+0000","last_became_active":"2024-04-14T15:09:47.044610+0000","last_became_peered":"2024-04-14T15:09:47.044610+0000","last_unstale":"2024-04-14T15:29:18.449180+0000","last_undegraded":"2024-04-14T15:29:18.449180+0000","last_fullsized":"2024-04-14T15:29:18.449180+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:29:13.978110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:39.394655+0000","last_clean_scrub_stamp":"2024-04-14T15:29:13.978110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:18.575306+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"783'38","reported_seq":1524,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:19.139104+0000","last_change":"2024-04-14T15:27:56.097704+0000","last_active":"2024-04-14T15:29:19.139104+0000","last_peered":"2024-04-14T15:29:19.139104+0000","last_clean":"2024-04-14T15:29:19.139104+0000","last_became_active":"2024-04-14T15:15:21.880565+0000","last_became_peered":"2024-04-14T15:15:21.880565+0000","last_unstale":"2024-04-14T15:29:19.139104+0000","last_undegraded":"2024-04-14T15:29:19.139104+0000","last_fullsized":"2024-04-14T15:29:19.139104+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"783'38","last_scrub_stamp":"2024-04-14T15:27:56.097590+0000","last_deep_scrub":"783'38","last_deep_scrub_stamp":"2024-04-14T15:25:27.804604+0000","last_clean_scrub_stamp":"2024-04-14T15:27:56.097590+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:21.215439+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'2","reported_seq":1818,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:18.449192+0000","last_change":"2024-04-14T15:29:04.339731+0000","last_active":"2024-04-14T15:29:18.449192+0000","last_peered":"2024-04-14T15:29:18.449192+0000","last_clean":"2024-04-14T15:29:18.449192+0000","last_became_active":"2024-04-14T15:27:14.201768+0000","last_became_peered":"2024-04-14T15:27:14.201768+0000","last_unstale":"2024-04-14T15:29:18.449192+0000","last_undegraded":"2024-04-14T15:29:18.449192+0000","last_fullsized":"2024-04-14T15:29:18.449192+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'2","last_scrub_stamp":"2024-04-14T15:29:04.339618+0000","last_deep_scrub":"91'2","last_deep_scrub_stamp":"2024-04-14T15:29:04.339618+0000","last_clean_scrub_stamp":"2024-04-14T15:29:04.339618+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:04.958836+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1778,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:18.449621+0000","last_change":"2024-04-14T15:28:03.148237+0000","last_active":"2024-04-14T15:29:18.449621+0000","last_peered":"2024-04-14T15:29:18.449621+0000","last_clean":"2024-04-14T15:29:18.449621+0000","last_became_active":"2024-04-14T15:09:49.065684+0000","last_became_peered":"2024-04-14T15:09:49.065684+0000","last_unstale":"2024-04-14T15:29:18.449621+0000","last_undegraded":"2024-04-14T15:29:18.449621+0000","last_fullsized":"2024-04-14T15:29:18.449621+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:03.148131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:28:03.148131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:29.138106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"91'130","reported_seq":1260,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.157066+0000","last_change":"2024-04-14T15:28:23.216706+0000","last_active":"2024-04-14T15:29:11.157066+0000","last_peered":"2024-04-14T15:29:11.157066+0000","last_clean":"2024-04-14T15:29:11.157066+0000","last_became_active":"2024-04-14T15:28:23.216417+0000","last_became_peered":"2024-04-14T15:28:23.216417+0000","last_unstale":"2024-04-14T15:29:11.157066+0000","last_undegraded":"2024-04-14T15:29:11.157066+0000","last_fullsized":"2024-04-14T15:29:11.157066+0000","mapping_epoch":902,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":907,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:28:19.473664+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:28:19.473664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:44.858236+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":1708,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:19.106894+0000","last_change":"2024-04-14T15:29:19.106914+0000","last_active":"2024-04-14T15:29:19.106894+0000","last_peered":"2024-04-14T15:29:19.106894+0000","last_clean":"2024-04-14T15:29:19.106894+0000","last_became_active":"2024-04-14T15:12:56.402023+0000","last_became_peered":"2024-04-14T15:12:56.402023+0000","last_unstale":"2024-04-14T15:29:19.106894+0000","last_undegraded":"2024-04-14T15:29:19.106894+0000","last_fullsized":"2024-04-14T15:29:19.106894+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:29:19.106838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:28:07.147878+0000","last_clean_scrub_stamp":"2024-04-14T15:29:19.106838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:32.259067+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":135,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.156993+0000","last_change":"2024-04-14T15:28:07.409341+0000","last_active":"2024-04-14T15:29:11.156993+0000","last_peered":"2024-04-14T15:29:11.156993+0000","last_clean":"2024-04-14T15:29:11.156993+0000","last_became_active":"2024-04-14T15:27:14.217451+0000","last_became_peered":"2024-04-14T15:27:14.217451+0000","last_unstale":"2024-04-14T15:29:11.156993+0000","last_undegraded":"2024-04-14T15:29:11.156993+0000","last_fullsized":"2024-04-14T15:29:11.156993+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:07.409246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:43.898835+0000","last_clean_scrub_stamp":"2024-04-14T15:28:07.409246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:37.206670+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"479'37","reported_seq":1856,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:18.450702+0000","last_change":"2024-04-14T15:28:55.031635+0000","last_active":"2024-04-14T15:29:18.450702+0000","last_peered":"2024-04-14T15:29:18.450702+0000","last_clean":"2024-04-14T15:29:18.450702+0000","last_became_active":"2024-04-14T15:09:45.179182+0000","last_became_peered":"2024-04-14T15:09:45.179182+0000","last_unstale":"2024-04-14T15:29:18.450702+0000","last_undegraded":"2024-04-14T15:29:18.450702+0000","last_fullsized":"2024-04-14T15:29:18.450702+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'37","last_scrub_stamp":"2024-04-14T15:28:55.031193+0000","last_deep_scrub":"102'27","last_deep_scrub_stamp":"2024-04-14T15:17:46.924330+0000","last_clean_scrub_stamp":"2024-04-14T15:28:55.031193+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:17.142255+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1789,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.114437+0000","last_change":"2024-04-14T15:28:12.930413+0000","last_active":"2024-04-14T15:29:11.114437+0000","last_peered":"2024-04-14T15:29:11.114437+0000","last_clean":"2024-04-14T15:29:11.114437+0000","last_became_active":"2024-04-14T15:09:43.021006+0000","last_became_peered":"2024-04-14T15:09:43.021006+0000","last_unstale":"2024-04-14T15:29:11.114437+0000","last_undegraded":"2024-04-14T15:29:11.114437+0000","last_fullsized":"2024-04-14T15:29:11.114437+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:12.930256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:28:12.930256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:38.970260+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1796,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:18.449105+0000","last_change":"2024-04-14T15:28:15.143755+0000","last_active":"2024-04-14T15:29:18.449105+0000","last_peered":"2024-04-14T15:29:18.449105+0000","last_clean":"2024-04-14T15:29:18.449105+0000","last_became_active":"2024-04-14T15:27:14.214481+0000","last_became_peered":"2024-04-14T15:27:14.214481+0000","last_unstale":"2024-04-14T15:29:18.449105+0000","last_undegraded":"2024-04-14T15:29:18.449105+0000","last_fullsized":"2024-04-14T15:29:18.449105+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:15.143609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:56.206649+0000","last_clean_scrub_stamp":"2024-04-14T15:28:15.143609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:26.910186+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"91'130","reported_seq":2001,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:18.449272+0000","last_change":"2024-04-14T15:28:48.320798+0000","last_active":"2024-04-14T15:29:18.449272+0000","last_peered":"2024-04-14T15:29:18.449272+0000","last_clean":"2024-04-14T15:29:18.449272+0000","last_became_active":"2024-04-14T15:28:34.269901+0000","last_became_peered":"2024-04-14T15:28:34.269901+0000","last_unstale":"2024-04-14T15:29:18.449272+0000","last_undegraded":"2024-04-14T15:29:18.449272+0000","last_fullsized":"2024-04-14T15:29:18.449272+0000","mapping_epoch":868,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":918,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:28:48.320719+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:28:48.320719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:02.388943+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":1417,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:19.139539+0000","last_change":"2024-04-14T15:29:10.676661+0000","last_active":"2024-04-14T15:29:19.139539+0000","last_peered":"2024-04-14T15:29:19.139539+0000","last_clean":"2024-04-14T15:29:19.139539+0000","last_became_active":"2024-04-14T15:27:14.221543+0000","last_became_peered":"2024-04-14T15:27:14.221543+0000","last_unstale":"2024-04-14T15:29:19.139539+0000","last_undegraded":"2024-04-14T15:29:19.139539+0000","last_fullsized":"2024-04-14T15:29:19.139539+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:29:10.676568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:29:10.676568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:39.947682+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":664,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.158075+0000","last_change":"2024-04-14T15:29:01.026804+0000","last_active":"2024-04-14T15:29:11.158075+0000","last_peered":"2024-04-14T15:29:11.158075+0000","last_clean":"2024-04-14T15:29:11.158075+0000","last_became_active":"2024-04-14T15:26:27.069614+0000","last_became_peered":"2024-04-14T15:26:27.069614+0000","last_unstale":"2024-04-14T15:29:11.158075+0000","last_undegraded":"2024-04-14T15:29:11.158075+0000","last_fullsized":"2024-04-14T15:29:11.158075+0000","mapping_epoch":828,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":829,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:29:01.026699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:29:01.026699+0000","last_clean_scrub_stamp":"2024-04-14T15:29:01.026699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:01.576701+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"479'30","reported_seq":1831,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:18.450685+0000","last_change":"2024-04-14T15:28:19.148165+0000","last_active":"2024-04-14T15:29:18.450685+0000","last_peered":"2024-04-14T15:29:18.450685+0000","last_clean":"2024-04-14T15:29:18.450685+0000","last_became_active":"2024-04-14T15:09:45.037231+0000","last_became_peered":"2024-04-14T15:09:45.037231+0000","last_unstale":"2024-04-14T15:29:18.450685+0000","last_undegraded":"2024-04-14T15:29:18.450685+0000","last_fullsized":"2024-04-14T15:29:18.450685+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'30","last_scrub_stamp":"2024-04-14T15:28:19.148035+0000","last_deep_scrub":"182'18","last_deep_scrub_stamp":"2024-04-14T15:14:10.383906+0000","last_clean_scrub_stamp":"2024-04-14T15:28:19.148035+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:31.614165+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1797,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.113784+0000","last_change":"2024-04-14T15:28:43.043408+0000","last_active":"2024-04-14T15:29:11.113784+0000","last_peered":"2024-04-14T15:29:11.113784+0000","last_clean":"2024-04-14T15:29:11.113784+0000","last_became_active":"2024-04-14T15:09:43.025095+0000","last_became_peered":"2024-04-14T15:09:43.025095+0000","last_unstale":"2024-04-14T15:29:11.113784+0000","last_undegraded":"2024-04-14T15:29:11.113784+0000","last_fullsized":"2024-04-14T15:29:11.113784+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:43.043313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:29.563759+0000","last_clean_scrub_stamp":"2024-04-14T15:28:43.043313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:12.097806+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1800,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:18.449831+0000","last_change":"2024-04-14T15:27:55.181206+0000","last_active":"2024-04-14T15:29:18.449831+0000","last_peered":"2024-04-14T15:29:18.449831+0000","last_clean":"2024-04-14T15:29:18.449831+0000","last_became_active":"2024-04-14T15:09:43.018509+0000","last_became_peered":"2024-04-14T15:09:43.018509+0000","last_unstale":"2024-04-14T15:29:18.449831+0000","last_undegraded":"2024-04-14T15:29:18.449831+0000","last_fullsized":"2024-04-14T15:29:18.449831+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:27:55.181100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:27:55.181100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0018592680000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"355'2","reported_seq":1786,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:18.449630+0000","last_change":"2024-04-14T15:28:19.234847+0000","last_active":"2024-04-14T15:29:18.449630+0000","last_peered":"2024-04-14T15:29:18.449630+0000","last_clean":"2024-04-14T15:29:18.449630+0000","last_became_active":"2024-04-14T15:09:49.065243+0000","last_became_peered":"2024-04-14T15:09:49.065243+0000","last_unstale":"2024-04-14T15:29:18.449630+0000","last_undegraded":"2024-04-14T15:29:18.449630+0000","last_fullsized":"2024-04-14T15:29:18.449630+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'2","last_scrub_stamp":"2024-04-14T15:28:19.234755+0000","last_deep_scrub":"355'2","last_deep_scrub_stamp":"2024-04-14T15:20:04.842881+0000","last_clean_scrub_stamp":"2024-04-14T15:28:19.234755+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:36.830840+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":1401,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:19.139445+0000","last_change":"2024-04-14T15:28:17.202846+0000","last_active":"2024-04-14T15:29:19.139445+0000","last_peered":"2024-04-14T15:29:19.139445+0000","last_clean":"2024-04-14T15:29:19.139445+0000","last_became_active":"2024-04-14T15:27:14.208395+0000","last_became_peered":"2024-04-14T15:27:14.208395+0000","last_unstale":"2024-04-14T15:29:19.139445+0000","last_undegraded":"2024-04-14T15:29:19.139445+0000","last_fullsized":"2024-04-14T15:29:19.139445+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:17.202749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:28:17.202749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:40.255641+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1781,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.114386+0000","last_change":"2024-04-14T15:28:22.929418+0000","last_active":"2024-04-14T15:29:11.114386+0000","last_peered":"2024-04-14T15:29:11.114386+0000","last_clean":"2024-04-14T15:29:11.114386+0000","last_became_active":"2024-04-14T15:09:47.047201+0000","last_became_peered":"2024-04-14T15:09:47.047201+0000","last_unstale":"2024-04-14T15:29:11.114386+0000","last_undegraded":"2024-04-14T15:29:11.114386+0000","last_fullsized":"2024-04-14T15:29:11.114386+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:22.929296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:28:22.929296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:46.019860+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"866'38","reported_seq":187,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:18.290929+0000","last_change":"2024-04-14T15:28:53.344212+0000","last_active":"2024-04-14T15:29:18.290929+0000","last_peered":"2024-04-14T15:29:18.290929+0000","last_clean":"2024-04-14T15:29:18.290929+0000","last_became_active":"2024-04-14T15:27:14.217810+0000","last_became_peered":"2024-04-14T15:27:14.217810+0000","last_unstale":"2024-04-14T15:29:18.290929+0000","last_undegraded":"2024-04-14T15:29:18.290929+0000","last_fullsized":"2024-04-14T15:29:18.290929+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"866'38","last_scrub_stamp":"2024-04-14T15:28:53.344054+0000","last_deep_scrub":"90'24","last_deep_scrub_stamp":"2024-04-14T15:11:25.120692+0000","last_clean_scrub_stamp":"2024-04-14T15:28:53.344054+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:00.744833+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"91'130","reported_seq":2195,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.113624+0000","last_change":"2024-04-14T15:29:00.150163+0000","last_active":"2024-04-14T15:29:11.113624+0000","last_peered":"2024-04-14T15:29:11.113624+0000","last_clean":"2024-04-14T15:29:11.113624+0000","last_became_active":"2024-04-14T15:27:49.066511+0000","last_became_peered":"2024-04-14T15:27:49.066511+0000","last_unstale":"2024-04-14T15:29:11.113624+0000","last_undegraded":"2024-04-14T15:29:11.113624+0000","last_fullsized":"2024-04-14T15:29:11.113624+0000","mapping_epoch":868,"log_start":"91'130","ondisk_log_start":"91'130","created":19,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:29:00.150065+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:29:00.150065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:14.325815+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":664,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.156291+0000","last_change":"2024-04-14T15:28:01.874863+0000","last_active":"2024-04-14T15:29:11.156291+0000","last_peered":"2024-04-14T15:29:11.156291+0000","last_clean":"2024-04-14T15:29:11.156291+0000","last_became_active":"2024-04-14T15:26:27.063604+0000","last_became_peered":"2024-04-14T15:26:27.063604+0000","last_unstale":"2024-04-14T15:29:11.156291+0000","last_undegraded":"2024-04-14T15:29:11.156291+0000","last_fullsized":"2024-04-14T15:29:11.156291+0000","mapping_epoch":828,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":829,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:01.874778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:47.627305+0000","last_clean_scrub_stamp":"2024-04-14T15:28:01.874778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:22.976776+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"813'3","reported_seq":1500,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:19.139542+0000","last_change":"2024-04-14T15:28:09.152712+0000","last_active":"2024-04-14T15:29:19.139542+0000","last_peered":"2024-04-14T15:29:19.139542+0000","last_clean":"2024-04-14T15:29:19.139542+0000","last_became_active":"2024-04-14T15:27:14.204710+0000","last_became_peered":"2024-04-14T15:27:14.204710+0000","last_unstale":"2024-04-14T15:29:19.139542+0000","last_undegraded":"2024-04-14T15:29:19.139542+0000","last_fullsized":"2024-04-14T15:29:19.139542+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"813'3","last_scrub_stamp":"2024-04-14T15:28:09.152624+0000","last_deep_scrub":"813'3","last_deep_scrub_stamp":"2024-04-14T15:26:03.783312+0000","last_clean_scrub_stamp":"2024-04-14T15:28:09.152624+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:35.077143+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0058046169999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":652,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.158148+0000","last_change":"2024-04-14T15:28:14.831607+0000","last_active":"2024-04-14T15:29:11.158148+0000","last_peered":"2024-04-14T15:29:11.158148+0000","last_clean":"2024-04-14T15:29:11.158148+0000","last_became_active":"2024-04-14T15:26:27.070066+0000","last_became_peered":"2024-04-14T15:26:27.070066+0000","last_unstale":"2024-04-14T15:29:11.158148+0000","last_undegraded":"2024-04-14T15:29:11.158148+0000","last_fullsized":"2024-04-14T15:29:11.158148+0000","mapping_epoch":828,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":829,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:14.831474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:28:14.831474+0000","last_clean_scrub_stamp":"2024-04-14T15:28:14.831474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:15.707833+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"91'130","reported_seq":2182,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.113685+0000","last_change":"2024-04-14T15:29:06.228442+0000","last_active":"2024-04-14T15:29:11.113685+0000","last_peered":"2024-04-14T15:29:11.113685+0000","last_clean":"2024-04-14T15:29:11.113685+0000","last_became_active":"2024-04-14T15:27:42.065597+0000","last_became_peered":"2024-04-14T15:27:42.065597+0000","last_unstale":"2024-04-14T15:29:11.113685+0000","last_undegraded":"2024-04-14T15:29:11.113685+0000","last_fullsized":"2024-04-14T15:29:11.113685+0000","mapping_epoch":189,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":881,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:29:06.228304+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:22:17.518666+0000","last_clean_scrub_stamp":"2024-04-14T15:29:06.228304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:28.572455+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":1401,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:19.138974+0000","last_change":"2024-04-14T15:27:45.105160+0000","last_active":"2024-04-14T15:29:19.138974+0000","last_peered":"2024-04-14T15:29:19.138974+0000","last_clean":"2024-04-14T15:29:19.138974+0000","last_became_active":"2024-04-14T15:27:14.219271+0000","last_became_peered":"2024-04-14T15:27:14.219271+0000","last_unstale":"2024-04-14T15:29:19.138974+0000","last_undegraded":"2024-04-14T15:29:19.138974+0000","last_fullsized":"2024-04-14T15:29:19.138974+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:27:45.105085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:27:45.105085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1795,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:19.105268+0000","last_change":"2024-04-14T15:28:21.262124+0000","last_active":"2024-04-14T15:29:19.105268+0000","last_peered":"2024-04-14T15:29:19.105268+0000","last_clean":"2024-04-14T15:29:19.105268+0000","last_became_active":"2024-04-14T15:09:47.049457+0000","last_became_peered":"2024-04-14T15:09:47.049457+0000","last_unstale":"2024-04-14T15:29:19.105268+0000","last_undegraded":"2024-04-14T15:29:19.105268+0000","last_fullsized":"2024-04-14T15:29:19.105268+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:21.262044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:28:21.262044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:23.619818+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"479'22","reported_seq":1838,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:18.450227+0000","last_change":"2024-04-14T15:29:16.155154+0000","last_active":"2024-04-14T15:29:18.450227+0000","last_peered":"2024-04-14T15:29:18.450227+0000","last_clean":"2024-04-14T15:29:18.450227+0000","last_became_active":"2024-04-14T15:09:45.033883+0000","last_became_peered":"2024-04-14T15:09:45.033883+0000","last_unstale":"2024-04-14T15:29:18.450227+0000","last_undegraded":"2024-04-14T15:29:18.450227+0000","last_fullsized":"2024-04-14T15:29:18.450227+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'22","last_scrub_stamp":"2024-04-14T15:29:16.155050+0000","last_deep_scrub":"308'18","last_deep_scrub_stamp":"2024-04-14T15:16:16.054032+0000","last_clean_scrub_stamp":"2024-04-14T15:29:16.155050+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:37.260920+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1803,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:19.104630+0000","last_change":"2024-04-14T15:28:06.319060+0000","last_active":"2024-04-14T15:29:19.104630+0000","last_peered":"2024-04-14T15:29:19.104630+0000","last_clean":"2024-04-14T15:29:19.104630+0000","last_became_active":"2024-04-14T15:09:43.021188+0000","last_became_peered":"2024-04-14T15:09:43.021188+0000","last_unstale":"2024-04-14T15:29:19.104630+0000","last_undegraded":"2024-04-14T15:29:19.104630+0000","last_fullsized":"2024-04-14T15:29:19.104630+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:06.318929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:28:06.318929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:30.761647+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"877'6","reported_seq":3002,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:18.448952+0000","last_change":"2024-04-14T15:28:47.314231+0000","last_active":"2024-04-14T15:29:18.448952+0000","last_peered":"2024-04-14T15:29:18.448952+0000","last_clean":"2024-04-14T15:29:18.448952+0000","last_became_active":"2024-04-14T15:27:14.214611+0000","last_became_peered":"2024-04-14T15:27:14.214611+0000","last_unstale":"2024-04-14T15:29:18.448952+0000","last_undegraded":"2024-04-14T15:29:18.448952+0000","last_fullsized":"2024-04-14T15:29:18.448952+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"877'6","last_scrub_stamp":"2024-04-14T15:28:47.314142+0000","last_deep_scrub":"877'6","last_deep_scrub_stamp":"2024-04-14T15:27:39.046415+0000","last_clean_scrub_stamp":"2024-04-14T15:28:47.314142+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:14.535075+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00056542099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1165,"num_read_kb":1160,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1789,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.114463+0000","last_change":"2024-04-14T15:28:31.910164+0000","last_active":"2024-04-14T15:29:11.114463+0000","last_peered":"2024-04-14T15:29:11.114463+0000","last_clean":"2024-04-14T15:29:11.114463+0000","last_became_active":"2024-04-14T15:27:14.203653+0000","last_became_peered":"2024-04-14T15:27:14.203653+0000","last_unstale":"2024-04-14T15:29:11.114463+0000","last_undegraded":"2024-04-14T15:29:11.114463+0000","last_fullsized":"2024-04-14T15:29:11.114463+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:31.910065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:28:31.910065+0000","last_clean_scrub_stamp":"2024-04-14T15:28:31.910065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:32.255005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"91'130","reported_seq":754,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.155821+0000","last_change":"2024-04-14T15:28:19.822693+0000","last_active":"2024-04-14T15:29:11.155821+0000","last_peered":"2024-04-14T15:29:11.155821+0000","last_clean":"2024-04-14T15:29:11.155821+0000","last_became_active":"2024-04-14T15:27:45.079088+0000","last_became_peered":"2024-04-14T15:27:45.079088+0000","last_unstale":"2024-04-14T15:29:11.155821+0000","last_undegraded":"2024-04-14T15:29:11.155821+0000","last_fullsized":"2024-04-14T15:29:11.155821+0000","mapping_epoch":828,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":884,"parent":"0.0","parent_split_bits":4,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:28:19.822578+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:11:28.875570+0000","last_clean_scrub_stamp":"2024-04-14T15:28:19.822578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:43.536556+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":1397,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:19.140421+0000","last_change":"2024-04-14T15:29:06.758617+0000","last_active":"2024-04-14T15:29:19.140421+0000","last_peered":"2024-04-14T15:29:19.140421+0000","last_clean":"2024-04-14T15:29:19.140421+0000","last_became_active":"2024-04-14T15:15:21.883717+0000","last_became_peered":"2024-04-14T15:15:21.883717+0000","last_unstale":"2024-04-14T15:29:19.140421+0000","last_undegraded":"2024-04-14T15:29:19.140421+0000","last_fullsized":"2024-04-14T15:29:19.140421+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:29:06.758512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:29:06.758512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:30.459459+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1785,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:18.449205+0000","last_change":"2024-04-14T15:28:08.136495+0000","last_active":"2024-04-14T15:29:18.449205+0000","last_peered":"2024-04-14T15:29:18.449205+0000","last_clean":"2024-04-14T15:29:18.449205+0000","last_became_active":"2024-04-14T15:09:47.040185+0000","last_became_peered":"2024-04-14T15:09:47.040185+0000","last_unstale":"2024-04-14T15:29:18.449205+0000","last_undegraded":"2024-04-14T15:29:18.449205+0000","last_fullsized":"2024-04-14T15:29:18.449205+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:08.136413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:28:08.136413+0000","last_clean_scrub_stamp":"2024-04-14T15:28:08.136413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:25.895647+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"479'40","reported_seq":2094,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:18.449238+0000","last_change":"2024-04-14T15:28:10.156157+0000","last_active":"2024-04-14T15:29:18.449238+0000","last_peered":"2024-04-14T15:29:18.449238+0000","last_clean":"2024-04-14T15:29:18.449238+0000","last_became_active":"2024-04-14T15:09:45.034168+0000","last_became_peered":"2024-04-14T15:09:45.034168+0000","last_unstale":"2024-04-14T15:29:18.449238+0000","last_undegraded":"2024-04-14T15:29:18.449238+0000","last_fullsized":"2024-04-14T15:29:18.449238+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'40","last_scrub_stamp":"2024-04-14T15:28:10.156043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:28:10.156043+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:33.151101+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":751,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.157381+0000","last_change":"2024-04-14T15:28:28.708689+0000","last_active":"2024-04-14T15:29:11.157381+0000","last_peered":"2024-04-14T15:29:11.157381+0000","last_clean":"2024-04-14T15:29:11.157381+0000","last_became_active":"2024-04-14T15:27:14.208574+0000","last_became_peered":"2024-04-14T15:27:14.208574+0000","last_unstale":"2024-04-14T15:29:11.157381+0000","last_undegraded":"2024-04-14T15:29:11.157381+0000","last_fullsized":"2024-04-14T15:29:11.157381+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:28.708595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:19:24.066275+0000","last_clean_scrub_stamp":"2024-04-14T15:28:28.708595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:30.871718+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1449,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:19.141107+0000","last_change":"2024-04-14T15:28:32.032028+0000","last_active":"2024-04-14T15:29:19.141107+0000","last_peered":"2024-04-14T15:29:19.141107+0000","last_clean":"2024-04-14T15:29:19.141107+0000","last_became_active":"2024-04-14T15:15:21.874742+0000","last_became_peered":"2024-04-14T15:15:21.874742+0000","last_unstale":"2024-04-14T15:29:19.141107+0000","last_undegraded":"2024-04-14T15:29:19.141107+0000","last_fullsized":"2024-04-14T15:29:19.141107+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:32.031897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:30.949569+0000","last_clean_scrub_stamp":"2024-04-14T15:28:32.031897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:59.867935+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"91'130","reported_seq":1264,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.156966+0000","last_change":"2024-04-14T15:28:55.360495+0000","last_active":"2024-04-14T15:29:11.156966+0000","last_peered":"2024-04-14T15:29:11.156966+0000","last_clean":"2024-04-14T15:29:11.156966+0000","last_became_active":"2024-04-14T15:27:53.072668+0000","last_became_peered":"2024-04-14T15:27:53.072668+0000","last_unstale":"2024-04-14T15:29:11.156966+0000","last_undegraded":"2024-04-14T15:29:11.156966+0000","last_fullsized":"2024-04-14T15:29:11.156966+0000","mapping_epoch":868,"log_start":"91'130","ondisk_log_start":"91'130","created":184,"last_epoch_clean":892,"parent":"0.0","parent_split_bits":3,"last_scrub":"91'130","last_scrub_stamp":"2024-04-14T15:28:55.360339+0000","last_deep_scrub":"91'130","last_deep_scrub_stamp":"2024-04-14T15:27:22.667430+0000","last_clean_scrub_stamp":"2024-04-14T15:28:55.360339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:56.446427+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":1711,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:19.105188+0000","last_change":"2024-04-14T15:28:55.329952+0000","last_active":"2024-04-14T15:29:19.105188+0000","last_peered":"2024-04-14T15:29:19.105188+0000","last_clean":"2024-04-14T15:29:19.105188+0000","last_became_active":"2024-04-14T15:27:14.209189+0000","last_became_peered":"2024-04-14T15:27:14.209189+0000","last_unstale":"2024-04-14T15:29:19.105188+0000","last_undegraded":"2024-04-14T15:29:19.105188+0000","last_fullsized":"2024-04-14T15:29:19.105188+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:55.329853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:28:55.329853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:24.260048+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"866'11","reported_seq":2058,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:18.450362+0000","last_change":"2024-04-14T15:28:39.146093+0000","last_active":"2024-04-14T15:29:18.450362+0000","last_peered":"2024-04-14T15:29:18.450362+0000","last_clean":"2024-04-14T15:29:18.450362+0000","last_became_active":"2024-04-14T15:27:14.214459+0000","last_became_peered":"2024-04-14T15:27:14.214459+0000","last_unstale":"2024-04-14T15:29:18.450362+0000","last_undegraded":"2024-04-14T15:29:18.450362+0000","last_fullsized":"2024-04-14T15:29:18.450362+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"866'11","last_scrub_stamp":"2024-04-14T15:28:39.145971+0000","last_deep_scrub":"866'11","last_deep_scrub_stamp":"2024-04-14T15:28:39.145971+0000","last_clean_scrub_stamp":"2024-04-14T15:28:39.145971+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:58.014412+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"479'20","reported_seq":806,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:18.448570+0000","last_change":"2024-04-14T15:28:42.304006+0000","last_active":"2024-04-14T15:29:18.448570+0000","last_peered":"2024-04-14T15:29:18.448570+0000","last_clean":"2024-04-14T15:29:18.448570+0000","last_became_active":"2024-04-14T15:21:16.034027+0000","last_became_peered":"2024-04-14T15:21:16.034027+0000","last_unstale":"2024-04-14T15:29:18.448570+0000","last_undegraded":"2024-04-14T15:29:18.448570+0000","last_fullsized":"2024-04-14T15:29:18.448570+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'20","last_scrub_stamp":"2024-04-14T15:28:42.303902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:28:42.303902+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:46.695839+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1796,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:18.449147+0000","last_change":"2024-04-14T15:28:51.325087+0000","last_active":"2024-04-14T15:29:18.449147+0000","last_peered":"2024-04-14T15:29:18.449147+0000","last_clean":"2024-04-14T15:29:18.449147+0000","last_became_active":"2024-04-14T15:09:43.022597+0000","last_became_peered":"2024-04-14T15:09:43.022597+0000","last_unstale":"2024-04-14T15:29:18.449147+0000","last_undegraded":"2024-04-14T15:29:18.449147+0000","last_fullsized":"2024-04-14T15:29:18.449147+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:51.325007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:21:28.684818+0000","last_clean_scrub_stamp":"2024-04-14T15:28:51.325007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:04.281666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1781,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:18.449659+0000","last_change":"2024-04-14T15:28:15.050502+0000","last_active":"2024-04-14T15:29:18.449659+0000","last_peered":"2024-04-14T15:29:18.449659+0000","last_clean":"2024-04-14T15:29:18.449659+0000","last_became_active":"2024-04-14T15:09:49.066648+0000","last_became_peered":"2024-04-14T15:09:49.066648+0000","last_unstale":"2024-04-14T15:29:18.449659+0000","last_undegraded":"2024-04-14T15:29:18.449659+0000","last_fullsized":"2024-04-14T15:29:18.449659+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:15.050409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:28:15.050409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:34.922604+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":1697,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:18.448781+0000","last_change":"2024-04-14T15:29:17.190309+0000","last_active":"2024-04-14T15:29:18.448781+0000","last_peered":"2024-04-14T15:29:18.448781+0000","last_clean":"2024-04-14T15:29:18.448781+0000","last_became_active":"2024-04-14T15:12:56.563074+0000","last_became_peered":"2024-04-14T15:12:56.563074+0000","last_unstale":"2024-04-14T15:29:18.448781+0000","last_undegraded":"2024-04-14T15:29:18.448781+0000","last_fullsized":"2024-04-14T15:29:18.448781+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:29:17.190220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:29:17.190220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:29.822883+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1795,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:19.104484+0000","last_change":"2024-04-14T15:28:52.268312+0000","last_active":"2024-04-14T15:29:19.104484+0000","last_peered":"2024-04-14T15:29:19.104484+0000","last_clean":"2024-04-14T15:29:19.104484+0000","last_became_active":"2024-04-14T15:09:47.049005+0000","last_became_peered":"2024-04-14T15:09:47.049005+0000","last_unstale":"2024-04-14T15:29:19.104484+0000","last_undegraded":"2024-04-14T15:29:19.104484+0000","last_fullsized":"2024-04-14T15:29:19.104484+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:52.268228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:28:52.268228+0000","last_clean_scrub_stamp":"2024-04-14T15:28:52.268228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:54.311348+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"479'23","reported_seq":1831,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:18.449158+0000","last_change":"2024-04-14T15:28:37.903985+0000","last_active":"2024-04-14T15:29:18.449158+0000","last_peered":"2024-04-14T15:29:18.449158+0000","last_clean":"2024-04-14T15:29:18.449158+0000","last_became_active":"2024-04-14T15:09:45.032892+0000","last_became_peered":"2024-04-14T15:09:45.032892+0000","last_unstale":"2024-04-14T15:29:18.449158+0000","last_undegraded":"2024-04-14T15:29:18.449158+0000","last_fullsized":"2024-04-14T15:29:18.449158+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'23","last_scrub_stamp":"2024-04-14T15:28:37.903878+0000","last_deep_scrub":"183'17","last_deep_scrub_stamp":"2024-04-14T15:14:11.577574+0000","last_clean_scrub_stamp":"2024-04-14T15:28:37.903878+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:41.067500+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"677'6","reported_seq":1810,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:18.449983+0000","last_change":"2024-04-14T15:28:00.173214+0000","last_active":"2024-04-14T15:29:18.449983+0000","last_peered":"2024-04-14T15:29:18.449983+0000","last_clean":"2024-04-14T15:29:18.449983+0000","last_became_active":"2024-04-14T15:09:43.020490+0000","last_became_peered":"2024-04-14T15:09:43.020490+0000","last_unstale":"2024-04-14T15:29:18.449983+0000","last_undegraded":"2024-04-14T15:29:18.449983+0000","last_fullsized":"2024-04-14T15:29:18.449983+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"677'6","last_scrub_stamp":"2024-04-14T15:28:00.173087+0000","last_deep_scrub":"677'6","last_deep_scrub_stamp":"2024-04-14T15:23:38.773539+0000","last_clean_scrub_stamp":"2024-04-14T15:28:00.173087+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:27.862693+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":528,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1449,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:19.140868+0000","last_change":"2024-04-14T15:28:10.108032+0000","last_active":"2024-04-14T15:29:19.140868+0000","last_peered":"2024-04-14T15:29:19.140868+0000","last_clean":"2024-04-14T15:29:19.140868+0000","last_became_active":"2024-04-14T15:15:21.876410+0000","last_became_peered":"2024-04-14T15:15:21.876410+0000","last_unstale":"2024-04-14T15:29:19.140868+0000","last_undegraded":"2024-04-14T15:29:19.140868+0000","last_fullsized":"2024-04-14T15:29:19.140868+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:10.107941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:28:10.107941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":1696,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:18.449685+0000","last_change":"2024-04-14T15:28:05.171891+0000","last_active":"2024-04-14T15:29:18.449685+0000","last_peered":"2024-04-14T15:29:18.449685+0000","last_clean":"2024-04-14T15:29:18.449685+0000","last_became_active":"2024-04-14T15:12:56.551177+0000","last_became_peered":"2024-04-14T15:12:56.551177+0000","last_unstale":"2024-04-14T15:29:18.449685+0000","last_undegraded":"2024-04-14T15:29:18.449685+0000","last_fullsized":"2024-04-14T15:29:18.449685+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:05.171817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:22:18.299330+0000","last_clean_scrub_stamp":"2024-04-14T15:28:05.171817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:19.005597+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"479'19","reported_seq":1817,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:18.450481+0000","last_change":"2024-04-14T15:28:48.017377+0000","last_active":"2024-04-14T15:29:18.450481+0000","last_peered":"2024-04-14T15:29:18.450481+0000","last_clean":"2024-04-14T15:29:18.450481+0000","last_became_active":"2024-04-14T15:09:45.180375+0000","last_became_peered":"2024-04-14T15:09:45.180375+0000","last_unstale":"2024-04-14T15:29:18.450481+0000","last_undegraded":"2024-04-14T15:29:18.450481+0000","last_fullsized":"2024-04-14T15:29:18.450481+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'19","last_scrub_stamp":"2024-04-14T15:28:48.017271+0000","last_deep_scrub":"347'13","last_deep_scrub_stamp":"2024-04-14T15:16:55.694599+0000","last_clean_scrub_stamp":"2024-04-14T15:28:48.017271+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:02.800393+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":1802,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:19.104434+0000","last_change":"2024-04-14T15:28:35.345923+0000","last_active":"2024-04-14T15:29:19.104434+0000","last_peered":"2024-04-14T15:29:19.104434+0000","last_clean":"2024-04-14T15:29:19.104434+0000","last_became_active":"2024-04-14T15:09:43.016788+0000","last_became_peered":"2024-04-14T15:09:43.016788+0000","last_unstale":"2024-04-14T15:29:19.104434+0000","last_undegraded":"2024-04-14T15:29:19.104434+0000","last_fullsized":"2024-04-14T15:29:19.104434+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:35.345754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:28:35.345754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:03.361232+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1468,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:19.141327+0000","last_change":"2024-04-14T15:28:51.799879+0000","last_active":"2024-04-14T15:29:19.141327+0000","last_peered":"2024-04-14T15:29:19.141327+0000","last_clean":"2024-04-14T15:29:19.141327+0000","last_became_active":"2024-04-14T15:15:21.873580+0000","last_became_peered":"2024-04-14T15:15:21.873580+0000","last_unstale":"2024-04-14T15:29:19.141327+0000","last_undegraded":"2024-04-14T15:29:19.141327+0000","last_fullsized":"2024-04-14T15:29:19.141327+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:51.799799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:28:51.799799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:01.688930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1798,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:19.104924+0000","last_change":"2024-04-14T15:29:14.163785+0000","last_active":"2024-04-14T15:29:19.104924+0000","last_peered":"2024-04-14T15:29:19.104924+0000","last_clean":"2024-04-14T15:29:19.104924+0000","last_became_active":"2024-04-14T15:09:49.064296+0000","last_became_peered":"2024-04-14T15:09:49.064296+0000","last_unstale":"2024-04-14T15:29:19.104924+0000","last_undegraded":"2024-04-14T15:29:19.104924+0000","last_fullsized":"2024-04-14T15:29:19.104924+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:29:14.163695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:26:39.410037+0000","last_clean_scrub_stamp":"2024-04-14T15:29:14.163695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:16.153138+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":1710,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:19.104349+0000","last_change":"2024-04-14T15:29:11.170177+0000","last_active":"2024-04-14T15:29:19.104349+0000","last_peered":"2024-04-14T15:29:19.104349+0000","last_clean":"2024-04-14T15:29:19.104349+0000","last_became_active":"2024-04-14T15:27:14.211630+0000","last_became_peered":"2024-04-14T15:27:14.211630+0000","last_unstale":"2024-04-14T15:29:19.104349+0000","last_undegraded":"2024-04-14T15:29:19.104349+0000","last_fullsized":"2024-04-14T15:29:19.104349+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:29:11.170027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:27:45.332536+0000","last_clean_scrub_stamp":"2024-04-14T15:29:11.170027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:15.143992+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.17","version":"479'12","reported_seq":1806,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:19.141322+0000","last_change":"2024-04-14T15:28:16.177386+0000","last_active":"2024-04-14T15:29:19.141322+0000","last_peered":"2024-04-14T15:29:19.141322+0000","last_clean":"2024-04-14T15:29:19.141322+0000","last_became_active":"2024-04-14T15:09:45.180037+0000","last_became_peered":"2024-04-14T15:09:45.180037+0000","last_unstale":"2024-04-14T15:29:19.141322+0000","last_undegraded":"2024-04-14T15:29:19.141322+0000","last_fullsized":"2024-04-14T15:29:19.141322+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'12","last_scrub_stamp":"2024-04-14T15:28:16.177258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:28:16.177258+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:44.619912+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":134,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.156802+0000","last_change":"2024-04-14T15:28:05.482212+0000","last_active":"2024-04-14T15:29:11.156802+0000","last_peered":"2024-04-14T15:29:11.156802+0000","last_clean":"2024-04-14T15:29:11.156802+0000","last_became_active":"2024-04-14T15:27:14.222112+0000","last_became_peered":"2024-04-14T15:27:14.222112+0000","last_unstale":"2024-04-14T15:29:11.156802+0000","last_undegraded":"2024-04-14T15:29:11.156802+0000","last_fullsized":"2024-04-14T15:29:11.156802+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:05.482096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:59.640565+0000","last_clean_scrub_stamp":"2024-04-14T15:28:05.482096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:33.428508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":664,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.157879+0000","last_change":"2024-04-14T15:28:38.700585+0000","last_active":"2024-04-14T15:29:11.157879+0000","last_peered":"2024-04-14T15:29:11.157879+0000","last_clean":"2024-04-14T15:29:11.157879+0000","last_became_active":"2024-04-14T15:26:27.067715+0000","last_became_peered":"2024-04-14T15:26:27.067715+0000","last_unstale":"2024-04-14T15:29:11.157879+0000","last_undegraded":"2024-04-14T15:29:11.157879+0000","last_fullsized":"2024-04-14T15:29:11.157879+0000","mapping_epoch":828,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":829,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:38.700449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:13.735024+0000","last_clean_scrub_stamp":"2024-04-14T15:28:38.700449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:58.269929+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1788,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.114659+0000","last_change":"2024-04-14T15:28:46.035515+0000","last_active":"2024-04-14T15:29:11.114659+0000","last_peered":"2024-04-14T15:29:11.114659+0000","last_clean":"2024-04-14T15:29:11.114659+0000","last_became_active":"2024-04-14T15:27:14.215736+0000","last_became_peered":"2024-04-14T15:27:14.215736+0000","last_unstale":"2024-04-14T15:29:11.114659+0000","last_undegraded":"2024-04-14T15:29:11.114659+0000","last_fullsized":"2024-04-14T15:29:11.114659+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:46.035420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:28:46.035420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:03.586018+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":1701,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:18.449420+0000","last_change":"2024-04-14T15:28:56.784656+0000","last_active":"2024-04-14T15:29:18.449420+0000","last_peered":"2024-04-14T15:29:18.449420+0000","last_clean":"2024-04-14T15:29:18.449420+0000","last_became_active":"2024-04-14T15:12:56.203577+0000","last_became_peered":"2024-04-14T15:12:56.203577+0000","last_unstale":"2024-04-14T15:29:18.449420+0000","last_undegraded":"2024-04-14T15:29:18.449420+0000","last_fullsized":"2024-04-14T15:29:18.449420+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:56.784550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:46.369231+0000","last_clean_scrub_stamp":"2024-04-14T15:28:56.784550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:01.539197+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"479'20","reported_seq":1502,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:19.141073+0000","last_change":"2024-04-14T15:28:50.797133+0000","last_active":"2024-04-14T15:29:19.141073+0000","last_peered":"2024-04-14T15:29:19.141073+0000","last_clean":"2024-04-14T15:29:19.141073+0000","last_became_active":"2024-04-14T15:15:21.877667+0000","last_became_peered":"2024-04-14T15:15:21.877667+0000","last_unstale":"2024-04-14T15:29:19.141073+0000","last_undegraded":"2024-04-14T15:29:19.141073+0000","last_fullsized":"2024-04-14T15:29:19.141073+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'20","last_scrub_stamp":"2024-04-14T15:28:50.797032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:28:50.797032+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:02.284320+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":664,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:16.085459+0000","last_change":"2024-04-14T15:29:16.085505+0000","last_active":"2024-04-14T15:29:16.085459+0000","last_peered":"2024-04-14T15:29:16.085459+0000","last_clean":"2024-04-14T15:29:16.085459+0000","last_became_active":"2024-04-14T15:26:27.065734+0000","last_became_peered":"2024-04-14T15:26:27.065734+0000","last_unstale":"2024-04-14T15:29:16.085459+0000","last_undegraded":"2024-04-14T15:29:16.085459+0000","last_fullsized":"2024-04-14T15:29:16.085459+0000","mapping_epoch":828,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":829,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:29:16.085373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:29:16.085373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:19.767433+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"829'10","reported_seq":780,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:13.289960+0000","last_change":"2024-04-14T15:29:03.040455+0000","last_active":"2024-04-14T15:29:13.289960+0000","last_peered":"2024-04-14T15:29:13.289960+0000","last_clean":"2024-04-14T15:29:13.289960+0000","last_became_active":"2024-04-14T15:26:27.065792+0000","last_became_peered":"2024-04-14T15:26:27.065792+0000","last_unstale":"2024-04-14T15:29:13.289960+0000","last_undegraded":"2024-04-14T15:29:13.289960+0000","last_fullsized":"2024-04-14T15:29:13.289960+0000","mapping_epoch":828,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":829,"parent":"0.0","parent_split_bits":0,"last_scrub":"829'10","last_scrub_stamp":"2024-04-14T15:29:03.040354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:29:03.040354+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:23.554431+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1780,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:18.450370+0000","last_change":"2024-04-14T15:29:06.080391+0000","last_active":"2024-04-14T15:29:18.450370+0000","last_peered":"2024-04-14T15:29:18.450370+0000","last_clean":"2024-04-14T15:29:18.450370+0000","last_became_active":"2024-04-14T15:09:49.066068+0000","last_became_peered":"2024-04-14T15:09:49.066068+0000","last_unstale":"2024-04-14T15:29:18.450370+0000","last_undegraded":"2024-04-14T15:29:18.450370+0000","last_fullsized":"2024-04-14T15:29:18.450370+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:29:06.080315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:16:54.724681+0000","last_clean_scrub_stamp":"2024-04-14T15:29:06.080315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:30.256693+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":569,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.157271+0000","last_change":"2024-04-14T15:28:56.946641+0000","last_active":"2024-04-14T15:29:11.157271+0000","last_peered":"2024-04-14T15:29:11.157271+0000","last_clean":"2024-04-14T15:29:11.157271+0000","last_became_active":"2024-04-14T15:26:27.068033+0000","last_became_peered":"2024-04-14T15:26:27.068033+0000","last_unstale":"2024-04-14T15:29:11.157271+0000","last_undegraded":"2024-04-14T15:29:11.157271+0000","last_fullsized":"2024-04-14T15:29:11.157271+0000","mapping_epoch":828,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":829,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:56.946545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:28:56.946545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:24.320804+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"479'19","reported_seq":925,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.157601+0000","last_change":"2024-04-14T15:28:47.826008+0000","last_active":"2024-04-14T15:29:11.157601+0000","last_peered":"2024-04-14T15:29:11.157601+0000","last_clean":"2024-04-14T15:29:11.157601+0000","last_became_active":"2024-04-14T15:26:27.062451+0000","last_became_peered":"2024-04-14T15:26:27.062451+0000","last_unstale":"2024-04-14T15:29:11.157601+0000","last_undegraded":"2024-04-14T15:29:11.157601+0000","last_fullsized":"2024-04-14T15:29:11.157601+0000","mapping_epoch":828,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":829,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'19","last_scrub_stamp":"2024-04-14T15:28:47.825887+0000","last_deep_scrub":"183'13","last_deep_scrub_stamp":"2024-04-14T15:14:11.755252+0000","last_clean_scrub_stamp":"2024-04-14T15:28:47.825887+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:07.579857+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1800,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:18.450579+0000","last_change":"2024-04-14T15:28:56.033939+0000","last_active":"2024-04-14T15:29:18.450579+0000","last_peered":"2024-04-14T15:29:18.450579+0000","last_clean":"2024-04-14T15:29:18.450579+0000","last_became_active":"2024-04-14T15:09:43.019301+0000","last_became_peered":"2024-04-14T15:09:43.019301+0000","last_unstale":"2024-04-14T15:29:18.450579+0000","last_undegraded":"2024-04-14T15:29:18.450579+0000","last_fullsized":"2024-04-14T15:29:18.450579+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:56.033841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:28:56.033841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:05.569556+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"255'10","reported_seq":1667,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:19.141354+0000","last_change":"2024-04-14T15:27:53.037018+0000","last_active":"2024-04-14T15:29:19.141354+0000","last_peered":"2024-04-14T15:29:19.141354+0000","last_clean":"2024-04-14T15:29:19.141354+0000","last_became_active":"2024-04-14T15:15:21.884468+0000","last_became_peered":"2024-04-14T15:15:21.884468+0000","last_unstale":"2024-04-14T15:29:19.141354+0000","last_undegraded":"2024-04-14T15:29:19.141354+0000","last_fullsized":"2024-04-14T15:29:19.141354+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'10","last_scrub_stamp":"2024-04-14T15:27:53.036891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:27:53.036891+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:20.315770+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1777,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:18.450126+0000","last_change":"2024-04-14T15:28:32.941088+0000","last_active":"2024-04-14T15:29:18.450126+0000","last_peered":"2024-04-14T15:29:18.450126+0000","last_clean":"2024-04-14T15:29:18.450126+0000","last_became_active":"2024-04-14T15:09:49.065242+0000","last_became_peered":"2024-04-14T15:09:49.065242+0000","last_unstale":"2024-04-14T15:29:18.450126+0000","last_undegraded":"2024-04-14T15:29:18.450126+0000","last_fullsized":"2024-04-14T15:29:18.450126+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:32.940953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:28:32.940953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:44.206768+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":1692,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:13.263492+0000","last_change":"2024-04-14T15:29:13.263518+0000","last_active":"2024-04-14T15:29:13.263492+0000","last_peered":"2024-04-14T15:29:13.263492+0000","last_clean":"2024-04-14T15:29:13.263492+0000","last_became_active":"2024-04-14T15:12:56.389204+0000","last_became_peered":"2024-04-14T15:12:56.389204+0000","last_unstale":"2024-04-14T15:29:13.263492+0000","last_undegraded":"2024-04-14T15:29:13.263492+0000","last_fullsized":"2024-04-14T15:29:13.263492+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:29:13.263410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:29:13.263410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:15.116162+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"479'19","reported_seq":640,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.156034+0000","last_change":"2024-04-14T15:28:55.902406+0000","last_active":"2024-04-14T15:29:11.156034+0000","last_peered":"2024-04-14T15:29:11.156034+0000","last_clean":"2024-04-14T15:29:11.156034+0000","last_became_active":"2024-04-14T15:27:14.204854+0000","last_became_peered":"2024-04-14T15:27:14.204854+0000","last_unstale":"2024-04-14T15:29:11.156034+0000","last_undegraded":"2024-04-14T15:29:11.156034+0000","last_fullsized":"2024-04-14T15:29:11.156034+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'19","last_scrub_stamp":"2024-04-14T15:28:55.902288+0000","last_deep_scrub":"479'19","last_deep_scrub_stamp":"2024-04-14T15:28:55.902288+0000","last_clean_scrub_stamp":"2024-04-14T15:28:55.902288+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:22.172278+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":142,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.156479+0000","last_change":"2024-04-14T15:29:08.282168+0000","last_active":"2024-04-14T15:29:11.156479+0000","last_peered":"2024-04-14T15:29:11.156479+0000","last_clean":"2024-04-14T15:29:11.156479+0000","last_became_active":"2024-04-14T15:27:14.204682+0000","last_became_peered":"2024-04-14T15:27:14.204682+0000","last_unstale":"2024-04-14T15:29:11.156479+0000","last_undegraded":"2024-04-14T15:29:11.156479+0000","last_fullsized":"2024-04-14T15:29:11.156479+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:29:08.282081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:27:50.505612+0000","last_clean_scrub_stamp":"2024-04-14T15:29:08.282081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:36.736107+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1460,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:19.140619+0000","last_change":"2024-04-14T15:28:53.872899+0000","last_active":"2024-04-14T15:29:19.140619+0000","last_peered":"2024-04-14T15:29:19.140619+0000","last_clean":"2024-04-14T15:29:19.140619+0000","last_became_active":"2024-04-14T15:15:21.882643+0000","last_became_peered":"2024-04-14T15:15:21.882643+0000","last_unstale":"2024-04-14T15:29:19.140619+0000","last_undegraded":"2024-04-14T15:29:19.140619+0000","last_fullsized":"2024-04-14T15:29:19.140619+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:53.872816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:18:33.485411+0000","last_clean_scrub_stamp":"2024-04-14T15:28:53.872816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:56.975133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1783,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:18.449558+0000","last_change":"2024-04-14T15:27:58.172404+0000","last_active":"2024-04-14T15:29:18.449558+0000","last_peered":"2024-04-14T15:29:18.449558+0000","last_clean":"2024-04-14T15:29:18.449558+0000","last_became_active":"2024-04-14T15:09:49.059711+0000","last_became_peered":"2024-04-14T15:09:49.059711+0000","last_unstale":"2024-04-14T15:29:18.449558+0000","last_undegraded":"2024-04-14T15:29:18.449558+0000","last_fullsized":"2024-04-14T15:29:18.449558+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:27:58.172330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:27:58.172330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:24.855336+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":142,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.156821+0000","last_change":"2024-04-14T15:28:59.415776+0000","last_active":"2024-04-14T15:29:11.156821+0000","last_peered":"2024-04-14T15:29:11.156821+0000","last_clean":"2024-04-14T15:29:11.156821+0000","last_became_active":"2024-04-14T15:27:14.218875+0000","last_became_peered":"2024-04-14T15:27:14.218875+0000","last_unstale":"2024-04-14T15:29:11.156821+0000","last_undegraded":"2024-04-14T15:29:11.156821+0000","last_fullsized":"2024-04-14T15:29:11.156821+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:59.415615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:28:59.415615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:09.387385+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"479'24","reported_seq":1857,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:18.449509+0000","last_change":"2024-04-14T15:28:36.910524+0000","last_active":"2024-04-14T15:29:18.449509+0000","last_peered":"2024-04-14T15:29:18.449509+0000","last_clean":"2024-04-14T15:29:18.449509+0000","last_became_active":"2024-04-14T15:27:14.223844+0000","last_became_peered":"2024-04-14T15:27:14.223844+0000","last_unstale":"2024-04-14T15:29:18.449509+0000","last_undegraded":"2024-04-14T15:29:18.449509+0000","last_fullsized":"2024-04-14T15:29:18.449509+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'24","last_scrub_stamp":"2024-04-14T15:28:36.910422+0000","last_deep_scrub":"464'16","last_deep_scrub_stamp":"2024-04-14T15:19:17.589999+0000","last_clean_scrub_stamp":"2024-04-14T15:28:36.910422+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:58.983098+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":1803,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:18.450659+0000","last_change":"2024-04-14T15:28:24.128742+0000","last_active":"2024-04-14T15:29:18.450659+0000","last_peered":"2024-04-14T15:29:18.450659+0000","last_clean":"2024-04-14T15:29:18.450659+0000","last_became_active":"2024-04-14T15:09:43.023583+0000","last_became_peered":"2024-04-14T15:09:43.023583+0000","last_unstale":"2024-04-14T15:29:18.450659+0000","last_undegraded":"2024-04-14T15:29:18.450659+0000","last_fullsized":"2024-04-14T15:29:18.450659+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:24.128613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:28:24.128613+0000","last_clean_scrub_stamp":"2024-04-14T15:28:24.128613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:40.104962+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1787,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:18.448945+0000","last_change":"2024-04-14T15:28:21.180951+0000","last_active":"2024-04-14T15:29:18.448945+0000","last_peered":"2024-04-14T15:29:18.448945+0000","last_clean":"2024-04-14T15:29:18.448945+0000","last_became_active":"2024-04-14T15:09:47.041683+0000","last_became_peered":"2024-04-14T15:09:47.041683+0000","last_unstale":"2024-04-14T15:29:18.448945+0000","last_undegraded":"2024-04-14T15:29:18.448945+0000","last_fullsized":"2024-04-14T15:29:18.448945+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:21.180874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:08.486217+0000","last_clean_scrub_stamp":"2024-04-14T15:28:21.180874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:33.674236+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1456,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:19.139322+0000","last_change":"2024-04-14T15:27:59.074687+0000","last_active":"2024-04-14T15:29:19.139322+0000","last_peered":"2024-04-14T15:29:19.139322+0000","last_clean":"2024-04-14T15:29:19.139322+0000","last_became_active":"2024-04-14T15:15:21.889567+0000","last_became_peered":"2024-04-14T15:15:21.889567+0000","last_unstale":"2024-04-14T15:29:19.139322+0000","last_undegraded":"2024-04-14T15:29:19.139322+0000","last_fullsized":"2024-04-14T15:29:19.139322+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:27:59.074610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:27:59.074610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:25.024323+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":1707,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:19.105096+0000","last_change":"2024-04-14T15:29:05.178562+0000","last_active":"2024-04-14T15:29:19.105096+0000","last_peered":"2024-04-14T15:29:19.105096+0000","last_clean":"2024-04-14T15:29:19.105096+0000","last_became_active":"2024-04-14T15:12:56.387723+0000","last_became_peered":"2024-04-14T15:12:56.387723+0000","last_unstale":"2024-04-14T15:29:19.105096+0000","last_undegraded":"2024-04-14T15:29:19.105096+0000","last_fullsized":"2024-04-14T15:29:19.105096+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:29:05.178478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:03.685045+0000","last_clean_scrub_stamp":"2024-04-14T15:29:05.178478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:22.494936+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.10","version":"479'8","reported_seq":1824,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:19.138839+0000","last_change":"2024-04-14T15:28:31.057158+0000","last_active":"2024-04-14T15:29:19.138839+0000","last_peered":"2024-04-14T15:29:19.138839+0000","last_clean":"2024-04-14T15:29:19.138839+0000","last_became_active":"2024-04-14T15:27:14.219371+0000","last_became_peered":"2024-04-14T15:27:14.219371+0000","last_unstale":"2024-04-14T15:29:19.138839+0000","last_undegraded":"2024-04-14T15:29:19.138839+0000","last_fullsized":"2024-04-14T15:29:19.138839+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'8","last_scrub_stamp":"2024-04-14T15:28:31.057050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:28:31.057050+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:50.993432+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":670,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.157704+0000","last_change":"2024-04-14T15:28:09.858748+0000","last_active":"2024-04-14T15:29:11.157704+0000","last_peered":"2024-04-14T15:29:11.157704+0000","last_clean":"2024-04-14T15:29:11.157704+0000","last_became_active":"2024-04-14T15:27:14.202117+0000","last_became_peered":"2024-04-14T15:27:14.202117+0000","last_unstale":"2024-04-14T15:29:11.157704+0000","last_undegraded":"2024-04-14T15:29:11.157704+0000","last_fullsized":"2024-04-14T15:29:11.157704+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:09.858655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:11:08.712289+0000","last_clean_scrub_stamp":"2024-04-14T15:28:09.858655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:35.246948+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":142,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.157172+0000","last_change":"2024-04-14T15:28:32.362435+0000","last_active":"2024-04-14T15:29:11.157172+0000","last_peered":"2024-04-14T15:29:11.157172+0000","last_clean":"2024-04-14T15:29:11.157172+0000","last_became_active":"2024-04-14T15:27:14.219333+0000","last_became_peered":"2024-04-14T15:27:14.219333+0000","last_unstale":"2024-04-14T15:29:11.157172+0000","last_undegraded":"2024-04-14T15:29:11.157172+0000","last_fullsized":"2024-04-14T15:29:11.157172+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:32.362297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:28:32.362297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:56.724312+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1776,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.114081+0000","last_change":"2024-04-14T15:28:58.134672+0000","last_active":"2024-04-14T15:29:11.114081+0000","last_peered":"2024-04-14T15:29:11.114081+0000","last_clean":"2024-04-14T15:29:11.114081+0000","last_became_active":"2024-04-14T15:09:49.061359+0000","last_became_peered":"2024-04-14T15:09:49.061359+0000","last_unstale":"2024-04-14T15:29:11.114081+0000","last_undegraded":"2024-04-14T15:29:11.114081+0000","last_fullsized":"2024-04-14T15:29:11.114081+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:58.134566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:27:31.732339+0000","last_clean_scrub_stamp":"2024-04-14T15:28:58.134566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:02.746731+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":1698,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.114620+0000","last_change":"2024-04-14T15:28:05.992275+0000","last_active":"2024-04-14T15:29:11.114620+0000","last_peered":"2024-04-14T15:29:11.114620+0000","last_clean":"2024-04-14T15:29:11.114620+0000","last_became_active":"2024-04-14T15:27:14.202407+0000","last_became_peered":"2024-04-14T15:27:14.202407+0000","last_unstale":"2024-04-14T15:29:11.114620+0000","last_undegraded":"2024-04-14T15:29:11.114620+0000","last_fullsized":"2024-04-14T15:29:11.114620+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:05.992175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:15:43.680852+0000","last_clean_scrub_stamp":"2024-04-14T15:28:05.992175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:33.603407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"479'24","reported_seq":1497,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:19.139703+0000","last_change":"2024-04-14T15:28:20.146179+0000","last_active":"2024-04-14T15:29:19.139703+0000","last_peered":"2024-04-14T15:29:19.139703+0000","last_clean":"2024-04-14T15:29:19.139703+0000","last_became_active":"2024-04-14T15:27:14.219501+0000","last_became_peered":"2024-04-14T15:27:14.219501+0000","last_unstale":"2024-04-14T15:29:19.139703+0000","last_undegraded":"2024-04-14T15:29:19.139703+0000","last_fullsized":"2024-04-14T15:29:19.139703+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'24","last_scrub_stamp":"2024-04-14T15:28:20.146086+0000","last_deep_scrub":"102'16","last_deep_scrub_stamp":"2024-04-14T15:12:35.999326+0000","last_clean_scrub_stamp":"2024-04-14T15:28:20.146086+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:37.390831+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":142,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.157454+0000","last_change":"2024-04-14T15:28:36.313957+0000","last_active":"2024-04-14T15:29:11.157454+0000","last_peered":"2024-04-14T15:29:11.157454+0000","last_clean":"2024-04-14T15:29:11.157454+0000","last_became_active":"2024-04-14T15:27:14.213337+0000","last_became_peered":"2024-04-14T15:27:14.213337+0000","last_unstale":"2024-04-14T15:29:11.157454+0000","last_undegraded":"2024-04-14T15:29:11.157454+0000","last_fullsized":"2024-04-14T15:29:11.157454+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:36.313807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:30.484673+0000","last_clean_scrub_stamp":"2024-04-14T15:28:36.313807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:46.980513+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1792,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.113456+0000","last_change":"2024-04-14T15:28:56.113120+0000","last_active":"2024-04-14T15:29:11.113456+0000","last_peered":"2024-04-14T15:29:11.113456+0000","last_clean":"2024-04-14T15:29:11.113456+0000","last_became_active":"2024-04-14T15:27:14.215548+0000","last_became_peered":"2024-04-14T15:27:14.215548+0000","last_unstale":"2024-04-14T15:29:11.113456+0000","last_undegraded":"2024-04-14T15:29:11.113456+0000","last_fullsized":"2024-04-14T15:29:11.113456+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:56.113024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:27:44.912947+0000","last_clean_scrub_stamp":"2024-04-14T15:28:56.113024+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:15.477954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1476,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:19.140870+0000","last_change":"2024-04-14T15:28:45.874358+0000","last_active":"2024-04-14T15:29:19.140870+0000","last_peered":"2024-04-14T15:29:19.140870+0000","last_clean":"2024-04-14T15:29:19.140870+0000","last_became_active":"2024-04-14T15:27:14.202684+0000","last_became_peered":"2024-04-14T15:27:14.202684+0000","last_unstale":"2024-04-14T15:29:19.140870+0000","last_undegraded":"2024-04-14T15:29:19.140870+0000","last_fullsized":"2024-04-14T15:29:19.140870+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:45.874274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:27:25.021273+0000","last_clean_scrub_stamp":"2024-04-14T15:28:45.874274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:59.303768+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":1699,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:19.104560+0000","last_change":"2024-04-14T15:28:45.338663+0000","last_active":"2024-04-14T15:29:19.104560+0000","last_peered":"2024-04-14T15:29:19.104560+0000","last_clean":"2024-04-14T15:29:19.104560+0000","last_became_active":"2024-04-14T15:12:56.335230+0000","last_became_peered":"2024-04-14T15:12:56.335230+0000","last_unstale":"2024-04-14T15:29:19.104560+0000","last_undegraded":"2024-04-14T15:29:19.104560+0000","last_fullsized":"2024-04-14T15:29:19.104560+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:45.338583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:28:45.338583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:05.098064+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"776'25","reported_seq":1824,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:18.450680+0000","last_change":"2024-04-14T15:28:14.047717+0000","last_active":"2024-04-14T15:29:18.450680+0000","last_peered":"2024-04-14T15:29:18.450680+0000","last_clean":"2024-04-14T15:29:18.450680+0000","last_became_active":"2024-04-14T15:09:45.181456+0000","last_became_peered":"2024-04-14T15:09:45.181456+0000","last_unstale":"2024-04-14T15:29:18.450680+0000","last_undegraded":"2024-04-14T15:29:18.450680+0000","last_fullsized":"2024-04-14T15:29:18.450680+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"776'25","last_scrub_stamp":"2024-04-14T15:28:14.047615+0000","last_deep_scrub":"776'25","last_deep_scrub_stamp":"2024-04-14T15:25:21.063579+0000","last_clean_scrub_stamp":"2024-04-14T15:28:14.047615+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:27.182951+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":672,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.155870+0000","last_change":"2024-04-14T15:28:52.894885+0000","last_active":"2024-04-14T15:29:11.155870+0000","last_peered":"2024-04-14T15:29:11.155870+0000","last_clean":"2024-04-14T15:29:11.155870+0000","last_became_active":"2024-04-14T15:26:27.058655+0000","last_became_peered":"2024-04-14T15:26:27.058655+0000","last_unstale":"2024-04-14T15:29:11.155870+0000","last_undegraded":"2024-04-14T15:29:11.155870+0000","last_fullsized":"2024-04-14T15:29:11.155870+0000","mapping_epoch":828,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":829,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:52.894796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:27:30.830543+0000","last_clean_scrub_stamp":"2024-04-14T15:28:52.894796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:02.812727+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"866'10","reported_seq":173,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:18.290671+0000","last_change":"2024-04-14T15:29:06.253222+0000","last_active":"2024-04-14T15:29:18.290671+0000","last_peered":"2024-04-14T15:29:18.290671+0000","last_clean":"2024-04-14T15:29:18.290671+0000","last_became_active":"2024-04-14T15:27:14.219712+0000","last_became_peered":"2024-04-14T15:27:14.219712+0000","last_unstale":"2024-04-14T15:29:18.290671+0000","last_undegraded":"2024-04-14T15:29:18.290671+0000","last_fullsized":"2024-04-14T15:29:18.290671+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"866'10","last_scrub_stamp":"2024-04-14T15:29:06.253113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:29:06.253113+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:23.997020+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"72'1","reported_seq":1784,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.114727+0000","last_change":"2024-04-14T15:28:33.970559+0000","last_active":"2024-04-14T15:29:11.114727+0000","last_peered":"2024-04-14T15:29:11.114727+0000","last_clean":"2024-04-14T15:29:11.114727+0000","last_became_active":"2024-04-14T15:09:49.062051+0000","last_became_peered":"2024-04-14T15:09:49.062051+0000","last_unstale":"2024-04-14T15:29:11.114727+0000","last_undegraded":"2024-04-14T15:29:11.114727+0000","last_fullsized":"2024-04-14T15:29:11.114727+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-04-14T15:28:33.970444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:48.039252+0000","last_clean_scrub_stamp":"2024-04-14T15:28:33.970444+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:52.332757+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":1701,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:18.449794+0000","last_change":"2024-04-14T15:29:01.043551+0000","last_active":"2024-04-14T15:29:18.449794+0000","last_peered":"2024-04-14T15:29:18.449794+0000","last_clean":"2024-04-14T15:29:18.449794+0000","last_became_active":"2024-04-14T15:27:14.215274+0000","last_became_peered":"2024-04-14T15:27:14.215274+0000","last_unstale":"2024-04-14T15:29:18.449794+0000","last_undegraded":"2024-04-14T15:29:18.449794+0000","last_fullsized":"2024-04-14T15:29:18.449794+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:29:01.043471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:29.418183+0000","last_clean_scrub_stamp":"2024-04-14T15:29:01.043471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:30.779896+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"519'24","reported_seq":155,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.157393+0000","last_change":"2024-04-14T15:28:41.374815+0000","last_active":"2024-04-14T15:29:11.157393+0000","last_peered":"2024-04-14T15:29:11.157393+0000","last_clean":"2024-04-14T15:29:11.157393+0000","last_became_active":"2024-04-14T15:27:14.213577+0000","last_became_peered":"2024-04-14T15:27:14.213577+0000","last_unstale":"2024-04-14T15:29:11.157393+0000","last_undegraded":"2024-04-14T15:29:11.157393+0000","last_fullsized":"2024-04-14T15:29:11.157393+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"519'24","last_scrub_stamp":"2024-04-14T15:28:41.374691+0000","last_deep_scrub":"519'24","last_deep_scrub_stamp":"2024-04-14T15:20:44.763417+0000","last_clean_scrub_stamp":"2024-04-14T15:28:41.374691+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:41.898370+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"684'4","reported_seq":1509,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:19.139483+0000","last_change":"2024-04-14T15:29:17.530237+0000","last_active":"2024-04-14T15:29:19.139483+0000","last_peered":"2024-04-14T15:29:19.139483+0000","last_clean":"2024-04-14T15:29:19.139483+0000","last_became_active":"2024-04-14T15:27:14.219431+0000","last_became_peered":"2024-04-14T15:27:14.219431+0000","last_unstale":"2024-04-14T15:29:19.139483+0000","last_undegraded":"2024-04-14T15:29:19.139483+0000","last_fullsized":"2024-04-14T15:29:19.139483+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"684'4","last_scrub_stamp":"2024-04-14T15:29:17.530093+0000","last_deep_scrub":"684'4","last_deep_scrub_stamp":"2024-04-14T15:23:46.931560+0000","last_clean_scrub_stamp":"2024-04-14T15:29:17.530093+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:45.089805+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1914,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1785,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:18.450782+0000","last_change":"2024-04-14T15:28:33.055080+0000","last_active":"2024-04-14T15:29:18.450782+0000","last_peered":"2024-04-14T15:29:18.450782+0000","last_clean":"2024-04-14T15:29:18.450782+0000","last_became_active":"2024-04-14T15:09:47.045730+0000","last_became_peered":"2024-04-14T15:09:47.045730+0000","last_unstale":"2024-04-14T15:29:18.450782+0000","last_undegraded":"2024-04-14T15:29:18.450782+0000","last_fullsized":"2024-04-14T15:29:18.450782+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:33.054998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:27:22.882334+0000","last_clean_scrub_stamp":"2024-04-14T15:28:33.054998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:46.556521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1782,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:18.450565+0000","last_change":"2024-04-14T15:28:37.328960+0000","last_active":"2024-04-14T15:29:18.450565+0000","last_peered":"2024-04-14T15:29:18.450565+0000","last_clean":"2024-04-14T15:29:18.450565+0000","last_became_active":"2024-04-14T15:09:49.057545+0000","last_became_peered":"2024-04-14T15:09:49.057545+0000","last_unstale":"2024-04-14T15:29:18.450565+0000","last_undegraded":"2024-04-14T15:29:18.450565+0000","last_fullsized":"2024-04-14T15:29:18.450565+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:37.328839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:30.343106+0000","last_clean_scrub_stamp":"2024-04-14T15:28:37.328839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:03.203875+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":561,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.156465+0000","last_change":"2024-04-14T15:27:59.880061+0000","last_active":"2024-04-14T15:29:11.156465+0000","last_peered":"2024-04-14T15:29:11.156465+0000","last_clean":"2024-04-14T15:29:11.156465+0000","last_became_active":"2024-04-14T15:26:27.059738+0000","last_became_peered":"2024-04-14T15:26:27.059738+0000","last_unstale":"2024-04-14T15:29:11.156465+0000","last_undegraded":"2024-04-14T15:29:11.156465+0000","last_fullsized":"2024-04-14T15:29:11.156465+0000","mapping_epoch":828,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":829,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:27:59.879976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:27:59.879976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:16.538197+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"479'27","reported_seq":1832,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:19.105077+0000","last_change":"2024-04-14T15:27:56.392763+0000","last_active":"2024-04-14T15:29:19.105077+0000","last_peered":"2024-04-14T15:29:19.105077+0000","last_clean":"2024-04-14T15:29:19.105077+0000","last_became_active":"2024-04-14T15:09:45.032887+0000","last_became_peered":"2024-04-14T15:09:45.032887+0000","last_unstale":"2024-04-14T15:29:19.105077+0000","last_undegraded":"2024-04-14T15:29:19.105077+0000","last_fullsized":"2024-04-14T15:29:19.105077+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'27","last_scrub_stamp":"2024-04-14T15:27:56.392615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:27:56.392615+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:23.336007+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":142,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.156582+0000","last_change":"2024-04-14T15:28:46.394175+0000","last_active":"2024-04-14T15:29:11.156582+0000","last_peered":"2024-04-14T15:29:11.156582+0000","last_clean":"2024-04-14T15:29:11.156582+0000","last_became_active":"2024-04-14T15:27:14.222196+0000","last_became_peered":"2024-04-14T15:27:14.222196+0000","last_unstale":"2024-04-14T15:29:11.156582+0000","last_undegraded":"2024-04-14T15:29:11.156582+0000","last_fullsized":"2024-04-14T15:29:11.156582+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:46.394030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:27:27.745213+0000","last_clean_scrub_stamp":"2024-04-14T15:28:46.394030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:54.018805+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1794,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:18.450670+0000","last_change":"2024-04-14T15:28:45.312738+0000","last_active":"2024-04-14T15:29:18.450670+0000","last_peered":"2024-04-14T15:29:18.450670+0000","last_clean":"2024-04-14T15:29:18.450670+0000","last_became_active":"2024-04-14T15:09:47.042733+0000","last_became_peered":"2024-04-14T15:09:47.042733+0000","last_unstale":"2024-04-14T15:29:18.450670+0000","last_undegraded":"2024-04-14T15:29:18.450670+0000","last_fullsized":"2024-04-14T15:29:18.450670+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:45.312658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:25:22.689718+0000","last_clean_scrub_stamp":"2024-04-14T15:28:45.312658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:56.876051+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1790,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:18.449921+0000","last_change":"2024-04-14T15:27:57.115562+0000","last_active":"2024-04-14T15:29:18.449921+0000","last_peered":"2024-04-14T15:29:18.449921+0000","last_clean":"2024-04-14T15:29:18.449921+0000","last_became_active":"2024-04-14T15:27:14.223736+0000","last_became_peered":"2024-04-14T15:27:14.223736+0000","last_unstale":"2024-04-14T15:29:18.449921+0000","last_undegraded":"2024-04-14T15:29:18.449921+0000","last_fullsized":"2024-04-14T15:29:18.449921+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:27:57.115489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:17:54.453174+0000","last_clean_scrub_stamp":"2024-04-14T15:27:57.115489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:26.600941+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":1685,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.113790+0000","last_change":"2024-04-14T15:28:04.968660+0000","last_active":"2024-04-14T15:29:11.113790+0000","last_peered":"2024-04-14T15:29:11.113790+0000","last_clean":"2024-04-14T15:29:11.113790+0000","last_became_active":"2024-04-14T15:12:56.392352+0000","last_became_peered":"2024-04-14T15:12:56.392352+0000","last_unstale":"2024-04-14T15:29:11.113790+0000","last_undegraded":"2024-04-14T15:29:11.113790+0000","last_fullsized":"2024-04-14T15:29:11.113790+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:04.968559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:28:04.968559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:30.122612+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"479'18","reported_seq":1504,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:19.139977+0000","last_change":"2024-04-14T15:28:41.993464+0000","last_active":"2024-04-14T15:29:19.139977+0000","last_peered":"2024-04-14T15:29:19.139977+0000","last_clean":"2024-04-14T15:29:19.139977+0000","last_became_active":"2024-04-14T15:15:21.877838+0000","last_became_peered":"2024-04-14T15:15:21.877838+0000","last_unstale":"2024-04-14T15:29:19.139977+0000","last_undegraded":"2024-04-14T15:29:19.139977+0000","last_fullsized":"2024-04-14T15:29:19.139977+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'18","last_scrub_stamp":"2024-04-14T15:28:41.993355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:44.008350+0000","last_clean_scrub_stamp":"2024-04-14T15:28:41.993355+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:08.178104+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":1480,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:19.140711+0000","last_change":"2024-04-14T15:28:01.146869+0000","last_active":"2024-04-14T15:29:19.140711+0000","last_peered":"2024-04-14T15:29:19.140711+0000","last_clean":"2024-04-14T15:29:19.140711+0000","last_became_active":"2024-04-14T15:27:14.204179+0000","last_became_peered":"2024-04-14T15:27:14.204179+0000","last_unstale":"2024-04-14T15:29:19.140711+0000","last_undegraded":"2024-04-14T15:29:19.140711+0000","last_fullsized":"2024-04-14T15:29:19.140711+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:01.146789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:28:01.146789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:28.765987+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1788,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:18.449821+0000","last_change":"2024-04-14T15:29:00.879787+0000","last_active":"2024-04-14T15:29:18.449821+0000","last_peered":"2024-04-14T15:29:18.449821+0000","last_clean":"2024-04-14T15:29:18.449821+0000","last_became_active":"2024-04-14T15:09:47.048045+0000","last_became_peered":"2024-04-14T15:09:47.048045+0000","last_unstale":"2024-04-14T15:29:18.449821+0000","last_undegraded":"2024-04-14T15:29:18.449821+0000","last_fullsized":"2024-04-14T15:29:18.449821+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:29:00.879688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:29:00.879688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:08.583065+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1478,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:19.141217+0000","last_change":"2024-04-14T15:28:58.824212+0000","last_active":"2024-04-14T15:29:19.141217+0000","last_peered":"2024-04-14T15:29:19.141217+0000","last_clean":"2024-04-14T15:29:19.141217+0000","last_became_active":"2024-04-14T15:27:14.208409+0000","last_became_peered":"2024-04-14T15:27:14.208409+0000","last_unstale":"2024-04-14T15:29:19.141217+0000","last_undegraded":"2024-04-14T15:29:19.141217+0000","last_fullsized":"2024-04-14T15:29:19.141217+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:58.824136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:27:44.071208+0000","last_clean_scrub_stamp":"2024-04-14T15:28:58.824136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:15.763311+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":1706,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:19.105074+0000","last_change":"2024-04-14T15:29:09.118127+0000","last_active":"2024-04-14T15:29:19.105074+0000","last_peered":"2024-04-14T15:29:19.105074+0000","last_clean":"2024-04-14T15:29:19.105074+0000","last_became_active":"2024-04-14T15:12:56.375278+0000","last_became_peered":"2024-04-14T15:12:56.375278+0000","last_unstale":"2024-04-14T15:29:19.105074+0000","last_undegraded":"2024-04-14T15:29:19.105074+0000","last_fullsized":"2024-04-14T15:29:19.105074+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:29:09.118049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:25:19.668662+0000","last_clean_scrub_stamp":"2024-04-14T15:29:09.118049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:19.630779+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"479'19","reported_seq":1821,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:18.449247+0000","last_change":"2024-04-14T15:29:01.920385+0000","last_active":"2024-04-14T15:29:18.449247+0000","last_peered":"2024-04-14T15:29:18.449247+0000","last_clean":"2024-04-14T15:29:18.449247+0000","last_became_active":"2024-04-14T15:09:45.030093+0000","last_became_peered":"2024-04-14T15:09:45.030093+0000","last_unstale":"2024-04-14T15:29:18.449247+0000","last_undegraded":"2024-04-14T15:29:18.449247+0000","last_fullsized":"2024-04-14T15:29:18.449247+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'19","last_scrub_stamp":"2024-04-14T15:29:01.920260+0000","last_deep_scrub":"276'13","last_deep_scrub_stamp":"2024-04-14T15:15:43.969775+0000","last_clean_scrub_stamp":"2024-04-14T15:29:01.920260+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:03.252951+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":1493,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.113398+0000","last_change":"2024-04-14T15:29:01.194106+0000","last_active":"2024-04-14T15:29:11.113398+0000","last_peered":"2024-04-14T15:29:11.113398+0000","last_clean":"2024-04-14T15:29:11.113398+0000","last_became_active":"2024-04-14T15:14:30.857727+0000","last_became_peered":"2024-04-14T15:14:30.857727+0000","last_unstale":"2024-04-14T15:29:11.113398+0000","last_undegraded":"2024-04-14T15:29:11.113398+0000","last_fullsized":"2024-04-14T15:29:11.113398+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:29:01.194007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:29:01.194007+0000","last_clean_scrub_stamp":"2024-04-14T15:29:01.194007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:03.732070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":664,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.156250+0000","last_change":"2024-04-14T15:28:44.725756+0000","last_active":"2024-04-14T15:29:11.156250+0000","last_peered":"2024-04-14T15:29:11.156250+0000","last_clean":"2024-04-14T15:29:11.156250+0000","last_became_active":"2024-04-14T15:26:27.068043+0000","last_became_peered":"2024-04-14T15:26:27.068043+0000","last_unstale":"2024-04-14T15:29:11.156250+0000","last_undegraded":"2024-04-14T15:29:11.156250+0000","last_fullsized":"2024-04-14T15:29:11.156250+0000","mapping_epoch":828,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":829,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:44.725657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:28:44.725657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:02.238641+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1788,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:18.450163+0000","last_change":"2024-04-14T15:29:04.085955+0000","last_active":"2024-04-14T15:29:18.450163+0000","last_peered":"2024-04-14T15:29:18.450163+0000","last_clean":"2024-04-14T15:29:18.450163+0000","last_became_active":"2024-04-14T15:09:49.061756+0000","last_became_peered":"2024-04-14T15:09:49.061756+0000","last_unstale":"2024-04-14T15:29:18.450163+0000","last_undegraded":"2024-04-14T15:29:18.450163+0000","last_fullsized":"2024-04-14T15:29:18.450163+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:29:04.085812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:29:04.085812+0000","last_clean_scrub_stamp":"2024-04-14T15:29:04.085812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:16.288367+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":142,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:15.361617+0000","last_change":"2024-04-14T15:29:15.361644+0000","last_active":"2024-04-14T15:29:15.361617+0000","last_peered":"2024-04-14T15:29:15.361617+0000","last_clean":"2024-04-14T15:29:15.361617+0000","last_became_active":"2024-04-14T15:27:14.213348+0000","last_became_peered":"2024-04-14T15:27:14.213348+0000","last_unstale":"2024-04-14T15:29:15.361617+0000","last_undegraded":"2024-04-14T15:29:15.361617+0000","last_fullsized":"2024-04-14T15:29:15.361617+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:29:15.361539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:12:55.034316+0000","last_clean_scrub_stamp":"2024-04-14T15:29:15.361539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:21.697984+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1b","version":"479'9","reported_seq":1799,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:18.449676+0000","last_change":"2024-04-14T15:27:51.172348+0000","last_active":"2024-04-14T15:29:18.449676+0000","last_peered":"2024-04-14T15:29:18.449676+0000","last_clean":"2024-04-14T15:29:18.449676+0000","last_became_active":"2024-04-14T15:09:45.028667+0000","last_became_peered":"2024-04-14T15:09:45.028667+0000","last_unstale":"2024-04-14T15:29:18.449676+0000","last_undegraded":"2024-04-14T15:29:18.449676+0000","last_fullsized":"2024-04-14T15:29:18.449676+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'9","last_scrub_stamp":"2024-04-14T15:27:51.172269+0000","last_deep_scrub":"87'7","last_deep_scrub_stamp":"2024-04-14T15:11:10.510785+0000","last_clean_scrub_stamp":"2024-04-14T15:27:51.172269+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":1283,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:19.104872+0000","last_change":"2024-04-14T15:28:09.282466+0000","last_active":"2024-04-14T15:29:19.104872+0000","last_peered":"2024-04-14T15:29:19.104872+0000","last_clean":"2024-04-14T15:29:19.104872+0000","last_became_active":"2024-04-14T15:27:14.208738+0000","last_became_peered":"2024-04-14T15:27:14.208738+0000","last_unstale":"2024-04-14T15:29:19.104872+0000","last_undegraded":"2024-04-14T15:29:19.104872+0000","last_fullsized":"2024-04-14T15:29:19.104872+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:09.282383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:22:15.944005+0000","last_clean_scrub_stamp":"2024-04-14T15:28:09.282383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:36.169459+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1779,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.113982+0000","last_change":"2024-04-14T15:28:14.957788+0000","last_active":"2024-04-14T15:29:11.113982+0000","last_peered":"2024-04-14T15:29:11.113982+0000","last_clean":"2024-04-14T15:29:11.113982+0000","last_became_active":"2024-04-14T15:09:47.046517+0000","last_became_peered":"2024-04-14T15:09:47.046517+0000","last_unstale":"2024-04-14T15:29:11.113982+0000","last_undegraded":"2024-04-14T15:29:11.113982+0000","last_fullsized":"2024-04-14T15:29:11.113982+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:14.957694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:46.023663+0000","last_clean_scrub_stamp":"2024-04-14T15:28:14.957694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:34.128715+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":660,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.156064+0000","last_change":"2024-04-14T15:29:09.099762+0000","last_active":"2024-04-14T15:29:11.156064+0000","last_peered":"2024-04-14T15:29:11.156064+0000","last_clean":"2024-04-14T15:29:11.156064+0000","last_became_active":"2024-04-14T15:26:27.065443+0000","last_became_peered":"2024-04-14T15:26:27.065443+0000","last_unstale":"2024-04-14T15:29:11.156064+0000","last_undegraded":"2024-04-14T15:29:11.156064+0000","last_fullsized":"2024-04-14T15:29:11.156064+0000","mapping_epoch":828,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":829,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:29:09.099620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:14:56.268086+0000","last_clean_scrub_stamp":"2024-04-14T15:29:09.099620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:17.541761+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":1704,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:12.241814+0000","last_change":"2024-04-14T15:29:12.241841+0000","last_active":"2024-04-14T15:29:12.241814+0000","last_peered":"2024-04-14T15:29:12.241814+0000","last_clean":"2024-04-14T15:29:12.241814+0000","last_became_active":"2024-04-14T15:27:14.214682+0000","last_became_peered":"2024-04-14T15:27:14.214682+0000","last_unstale":"2024-04-14T15:29:12.241814+0000","last_undegraded":"2024-04-14T15:29:12.241814+0000","last_fullsized":"2024-04-14T15:29:12.241814+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:29:12.241729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:29:12.241729+0000","last_clean_scrub_stamp":"2024-04-14T15:29:12.241729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:17.963819+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.18","version":"806'19","reported_seq":1854,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:18.448853+0000","last_change":"2024-04-14T15:29:07.987282+0000","last_active":"2024-04-14T15:29:18.448853+0000","last_peered":"2024-04-14T15:29:18.448853+0000","last_clean":"2024-04-14T15:29:18.448853+0000","last_became_active":"2024-04-14T15:27:14.211957+0000","last_became_peered":"2024-04-14T15:27:14.211957+0000","last_unstale":"2024-04-14T15:29:18.448853+0000","last_undegraded":"2024-04-14T15:29:18.448853+0000","last_fullsized":"2024-04-14T15:29:18.448853+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"806'19","last_scrub_stamp":"2024-04-14T15:29:07.987120+0000","last_deep_scrub":"806'19","last_deep_scrub_stamp":"2024-04-14T15:25:56.405833+0000","last_clean_scrub_stamp":"2024-04-14T15:29:07.987120+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:21.848137+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":1470,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:19.141080+0000","last_change":"2024-04-14T15:28:15.145281+0000","last_active":"2024-04-14T15:29:19.141080+0000","last_peered":"2024-04-14T15:29:19.141080+0000","last_clean":"2024-04-14T15:29:19.141080+0000","last_became_active":"2024-04-14T15:15:21.891926+0000","last_became_peered":"2024-04-14T15:15:21.891926+0000","last_unstale":"2024-04-14T15:29:19.141080+0000","last_undegraded":"2024-04-14T15:29:19.141080+0000","last_fullsized":"2024-04-14T15:29:19.141080+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:15.145203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:28:15.145203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:41.361332+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1787,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:18.448920+0000","last_change":"2024-04-14T15:28:23.153515+0000","last_active":"2024-04-14T15:29:18.448920+0000","last_peered":"2024-04-14T15:29:18.448920+0000","last_clean":"2024-04-14T15:29:18.448920+0000","last_became_active":"2024-04-14T15:09:47.041486+0000","last_became_peered":"2024-04-14T15:09:47.041486+0000","last_unstale":"2024-04-14T15:29:18.448920+0000","last_undegraded":"2024-04-14T15:29:18.448920+0000","last_fullsized":"2024-04-14T15:29:18.448920+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:23.153427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:28:23.153427+0000","last_clean_scrub_stamp":"2024-04-14T15:28:23.153427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:35.509872+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1776,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:18.449227+0000","last_change":"2024-04-14T15:28:40.934665+0000","last_active":"2024-04-14T15:29:18.449227+0000","last_peered":"2024-04-14T15:29:18.449227+0000","last_clean":"2024-04-14T15:29:18.449227+0000","last_became_active":"2024-04-14T15:09:49.058904+0000","last_became_peered":"2024-04-14T15:09:49.058904+0000","last_unstale":"2024-04-14T15:29:18.449227+0000","last_undegraded":"2024-04-14T15:29:18.449227+0000","last_fullsized":"2024-04-14T15:29:18.449227+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:40.934551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:28:40.934551+0000","last_clean_scrub_stamp":"2024-04-14T15:28:40.934551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:57.015993+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":569,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-04-14T15:29:11.157927+0000","last_change":"2024-04-14T15:29:07.117269+0000","last_active":"2024-04-14T15:29:11.157927+0000","last_peered":"2024-04-14T15:29:11.157927+0000","last_clean":"2024-04-14T15:29:11.157927+0000","last_became_active":"2024-04-14T15:26:27.068240+0000","last_became_peered":"2024-04-14T15:26:27.068240+0000","last_unstale":"2024-04-14T15:29:11.157927+0000","last_undegraded":"2024-04-14T15:29:11.157927+0000","last_fullsized":"2024-04-14T15:29:11.157927+0000","mapping_epoch":828,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":829,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:29:07.117128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:29:07.117128+0000","last_clean_scrub_stamp":"2024-04-14T15:29:07.117128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:08.063967+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"479'34","reported_seq":1522,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:19.139756+0000","last_change":"2024-04-14T15:28:56.794095+0000","last_active":"2024-04-14T15:29:19.139756+0000","last_peered":"2024-04-14T15:29:19.139756+0000","last_clean":"2024-04-14T15:29:19.139756+0000","last_became_active":"2024-04-14T15:15:21.884310+0000","last_became_peered":"2024-04-14T15:15:21.884310+0000","last_unstale":"2024-04-14T15:29:19.139756+0000","last_undegraded":"2024-04-14T15:29:19.139756+0000","last_fullsized":"2024-04-14T15:29:19.139756+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'34","last_scrub_stamp":"2024-04-14T15:28:56.793981+0000","last_deep_scrub":"306'22","last_deep_scrub_stamp":"2024-04-14T15:16:13.898106+0000","last_clean_scrub_stamp":"2024-04-14T15:28:56.793981+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:06.482098+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":1793,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:18.450504+0000","last_change":"2024-04-14T15:28:32.100163+0000","last_active":"2024-04-14T15:29:18.450504+0000","last_peered":"2024-04-14T15:29:18.450504+0000","last_clean":"2024-04-14T15:29:18.450504+0000","last_became_active":"2024-04-14T15:09:43.017351+0000","last_became_peered":"2024-04-14T15:09:43.017351+0000","last_unstale":"2024-04-14T15:29:18.450504+0000","last_undegraded":"2024-04-14T15:29:18.450504+0000","last_fullsized":"2024-04-14T15:29:18.450504+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:32.100084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:09:42.004317+0000","last_clean_scrub_stamp":"2024-04-14T15:28:32.100084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:33.999154+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1793,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:18.449052+0000","last_change":"2024-04-14T15:28:42.918254+0000","last_active":"2024-04-14T15:29:18.449052+0000","last_peered":"2024-04-14T15:29:18.449052+0000","last_clean":"2024-04-14T15:29:18.449052+0000","last_became_active":"2024-04-14T15:27:14.212502+0000","last_became_peered":"2024-04-14T15:27:14.212502+0000","last_unstale":"2024-04-14T15:29:18.449052+0000","last_undegraded":"2024-04-14T15:29:18.449052+0000","last_fullsized":"2024-04-14T15:29:18.449052+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:42.918122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:24:32.403077+0000","last_clean_scrub_stamp":"2024-04-14T15:28:42.918122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:50.282002+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1791,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-04-14T15:29:19.140327+0000","last_change":"2024-04-14T15:28:18.170713+0000","last_active":"2024-04-14T15:29:19.140327+0000","last_peered":"2024-04-14T15:29:19.140327+0000","last_clean":"2024-04-14T15:29:19.140327+0000","last_became_active":"2024-04-14T15:27:14.221125+0000","last_became_peered":"2024-04-14T15:27:14.221125+0000","last_unstale":"2024-04-14T15:29:19.140327+0000","last_undegraded":"2024-04-14T15:29:19.140327+0000","last_fullsized":"2024-04-14T15:29:19.140327+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:28:18.170628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:28:18.170628+0000","last_clean_scrub_stamp":"2024-04-14T15:28:18.170628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:29:36.341370+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":5,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":2006,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1165,"num_read_kb":1160,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1119,"num_read_kb":876,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":152,"num_bytes_recovered":2134,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":811,"ondisk_log_size":811,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103410,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":93700,"kb_used_data":8380,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":93651452,"statfs":{"total":95995035648,"available":95899086848,"internally_reserved":0,"allocated":8581120,"data_stored":4596332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":87357898},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397749,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":97540,"kb_used_data":8060,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":93647612,"statfs":{"total":95995035648,"available":95895154688,"internally_reserved":0,"allocated":8253440,"data_stored":4268950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":91617734},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575119,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":111472,"kb_used_data":8100,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93633680,"statfs":{"total":95995035648,"available":95880888320,"internally_reserved":0,"allocated":8294400,"data_stored":4276723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738644,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":119760,"kb_used_data":8072,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93625392,"statfs":{"total":95995035648,"available":95872401408,"internally_reserved":0,"allocated":8265728,"data_stored":4269310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":114358727},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411595,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":120000,"kb_used_data":8312,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93625152,"statfs":{"total":95995035648,"available":95872155648,"internally_reserved":0,"allocated":8511488,"data_stored":4531075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182661,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":98148,"kb_used_data":8668,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":93647004,"statfs":{"total":95995035648,"available":95894532096,"internally_reserved":0,"allocated":8876032,"data_stored":4859650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":91617735},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986431,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":111452,"kb_used_data":8080,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93633700,"statfs":{"total":95995035648,"available":95880908800,"internally_reserved":0,"allocated":8273920,"data_stored":4276315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659388,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":112000,"kb_used_data":8632,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93633152,"statfs":{"total":95995035648,"available":95880347648,"internally_reserved":0,"allocated":8839168,"data_stored":4858370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-14T15:29:20.556 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-04-14T15:29:20.556 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 shell --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 -- ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-04-14T15:29:20.783 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:20 smithi090 ceph-mon[41670]: 11.c scrub starts 2024-04-14T15:29:20.783 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:20 smithi090 ceph-mon[41670]: 11.c scrub ok 2024-04-14T15:29:20.784 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:20 smithi090 ceph-mon[41670]: pgmap v1417: 174 pgs: 174 active+clean; 582 KiB data, 844 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:29:20.784 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:20 smithi090 ceph-mon[41670]: from='client.38421 v1:172.21.15.90:0/3505720627' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:29:20.784 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:20 smithi090 ceph-mon[41670]: 11.2 scrub starts 2024-04-14T15:29:20.784 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:20 smithi090 ceph-mon[41670]: 11.2 scrub ok 2024-04-14T15:29:20.784 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:20 smithi090 ceph-mon[30399]: 11.c scrub starts 2024-04-14T15:29:20.784 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:20 smithi090 ceph-mon[30399]: 11.c scrub ok 2024-04-14T15:29:20.784 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:20 smithi090 ceph-mon[30399]: pgmap v1417: 174 pgs: 174 active+clean; 582 KiB data, 844 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:29:20.784 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:20 smithi090 ceph-mon[30399]: from='client.38421 v1:172.21.15.90:0/3505720627' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:29:20.784 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:20 smithi090 ceph-mon[30399]: 11.2 scrub starts 2024-04-14T15:29:20.784 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:20 smithi090 ceph-mon[30399]: 11.2 scrub ok 2024-04-14T15:29:20.807 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:29:20.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:20 smithi092 ceph-mon[33966]: 11.c scrub starts 2024-04-14T15:29:20.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:20 smithi092 ceph-mon[33966]: 11.c scrub ok 2024-04-14T15:29:20.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:20 smithi092 ceph-mon[33966]: pgmap v1417: 174 pgs: 174 active+clean; 582 KiB data, 844 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:29:20.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:20 smithi092 ceph-mon[33966]: from='client.38421 v1:172.21.15.90:0/3505720627' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:29:20.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:20 smithi092 ceph-mon[33966]: 11.2 scrub starts 2024-04-14T15:29:20.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:20 smithi092 ceph-mon[33966]: 11.2 scrub ok 2024-04-14T15:29:21.627 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:21 smithi090 ceph-mon[41670]: 4.1b scrub starts 2024-04-14T15:29:21.627 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:21 smithi090 ceph-mon[41670]: 4.1b scrub ok 2024-04-14T15:29:21.628 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:21 smithi090 ceph-mon[41670]: 11.7 scrub starts 2024-04-14T15:29:21.628 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:21 smithi090 ceph-mon[41670]: 11.7 scrub ok 2024-04-14T15:29:21.628 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:21 smithi090 ceph-mon[41670]: 11.8 scrub starts 2024-04-14T15:29:21.628 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:21 smithi090 ceph-mon[41670]: 11.8 scrub ok 2024-04-14T15:29:21.628 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:21 smithi090 ceph-mon[41670]: osdmap e926: 8 total, 8 up, 8 in 2024-04-14T15:29:21.628 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:21 smithi090 ceph-mon[30399]: 4.1b scrub starts 2024-04-14T15:29:21.628 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:21 smithi090 ceph-mon[30399]: 4.1b scrub ok 2024-04-14T15:29:21.628 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:21 smithi090 ceph-mon[30399]: 11.7 scrub starts 2024-04-14T15:29:21.628 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:21 smithi090 ceph-mon[30399]: 11.7 scrub ok 2024-04-14T15:29:21.628 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:21 smithi090 ceph-mon[30399]: 11.8 scrub starts 2024-04-14T15:29:21.628 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:21 smithi090 ceph-mon[30399]: 11.8 scrub ok 2024-04-14T15:29:21.629 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:21 smithi090 ceph-mon[30399]: osdmap e926: 8 total, 8 up, 8 in 2024-04-14T15:29:21.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:21 smithi092 ceph-mon[33966]: 4.1b scrub starts 2024-04-14T15:29:21.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:21 smithi092 ceph-mon[33966]: 4.1b scrub ok 2024-04-14T15:29:21.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:21 smithi092 ceph-mon[33966]: 11.7 scrub starts 2024-04-14T15:29:21.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:21 smithi092 ceph-mon[33966]: 11.7 scrub ok 2024-04-14T15:29:21.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:21 smithi092 ceph-mon[33966]: 11.8 scrub starts 2024-04-14T15:29:21.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:21 smithi092 ceph-mon[33966]: 11.8 scrub ok 2024-04-14T15:29:21.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:21 smithi092 ceph-mon[33966]: osdmap e926: 8 total, 8 up, 8 in 2024-04-14T15:29:22.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.529+0000 7fb049115700 1 -- 172.21.15.90:0/2512775637 >> v1:172.21.15.90:6789/0 conn(0x7fb044103a40 legacy=0x7fb044103e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:29:22.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.529+0000 7fb049115700 1 -- 172.21.15.90:0/2512775637 shutdown_connections 2024-04-14T15:29:22.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.529+0000 7fb049115700 1 -- 172.21.15.90:0/2512775637 >> 172.21.15.90:0/2512775637 conn(0x7fb0440fd290 msgr2=0x7fb0440ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:29:22.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.529+0000 7fb049115700 1 -- 172.21.15.90:0/2512775637 shutdown_connections 2024-04-14T15:29:22.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.529+0000 7fb049115700 1 -- 172.21.15.90:0/2512775637 wait complete. 2024-04-14T15:29:22.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.529+0000 7fb049115700 1 librados: starting msgr at 2024-04-14T15:29:22.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.529+0000 7fb049115700 1 librados: starting objecter 2024-04-14T15:29:22.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.530+0000 7fb049115700 1 Processor -- start 2024-04-14T15:29:22.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.530+0000 7fb049115700 1 -- start start 2024-04-14T15:29:22.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.530+0000 7fb049115700 1 librados: setting wanted keys 2024-04-14T15:29:22.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.530+0000 7fb049115700 1 librados: calling monclient init 2024-04-14T15:29:22.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.530+0000 7fb049115700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb044103370 con 0x7fb04410b850 2024-04-14T15:29:22.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.530+0000 7fb049115700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb044103580 con 0x7fb044103a40 2024-04-14T15:29:22.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.530+0000 7fb049115700 1 -- --> v1:172.21.15.90:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb0441ac300 con 0x7fb04410f360 2024-04-14T15:29:22.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.530+0000 7fb04259c700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fb04410b850 0x7fb044101db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:58290/0 (socket says 172.21.15.90:58290) 2024-04-14T15:29:22.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.530+0000 7fb04259c700 1 -- 172.21.15.90:0/3290801562 learned_addr learned my addr 172.21.15.90:0/3290801562 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-14T15:29:22.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.530+0000 7fb02bfff700 10 client.?.objecter ms_handle_connect 0x7fb04410b850 2024-04-14T15:29:22.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.530+0000 7fb02bfff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:29:22.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.530+0000 7fb02bfff700 10 client.?.objecter ms_handle_connect 0x7fb04410f360 2024-04-14T15:29:22.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.530+0000 7fb02bfff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:29:22.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.530+0000 7fb02bfff700 10 client.?.objecter ms_handle_connect 0x7fb044103a40 2024-04-14T15:29:22.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.530+0000 7fb02bfff700 10 client.?.objecter resend_mon_ops 2024-04-14T15:29:22.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.530+0000 7fb02bfff700 1 -- 172.21.15.90:0/3290801562 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2263539932 0 0) 0x7fb044103370 con 0x7fb04410b850 2024-04-14T15:29:22.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.531+0000 7fb02bfff700 1 -- 172.21.15.90:0/3290801562 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb02c0035b0 con 0x7fb04410b850 2024-04-14T15:29:22.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.531+0000 7fb02bfff700 1 -- 172.21.15.90:0/3290801562 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2619433312 0 0) 0x7fb044103580 con 0x7fb044103a40 2024-04-14T15:29:22.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.531+0000 7fb02bfff700 1 -- 172.21.15.90:0/3290801562 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb044103370 con 0x7fb044103a40 2024-04-14T15:29:22.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.531+0000 7fb02bfff700 1 -- 172.21.15.90:0/3290801562 <== mon.2 v1:172.21.15.90:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3038293922 0 0) 0x7fb0441ac300 con 0x7fb04410f360 2024-04-14T15:29:22.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.531+0000 7fb02bfff700 1 -- 172.21.15.90:0/3290801562 --> v1:172.21.15.90:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb044103580 con 0x7fb04410f360 2024-04-14T15:29:22.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.531+0000 7fb02bfff700 1 -- 172.21.15.90:0/3290801562 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4159416366 0 0) 0x7fb02c0035b0 con 0x7fb04410b850 2024-04-14T15:29:22.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.531+0000 7fb02bfff700 1 -- 172.21.15.90:0/3290801562 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb0441ac300 con 0x7fb04410b850 2024-04-14T15:29:22.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.531+0000 7fb02bfff700 1 -- 172.21.15.90:0/3290801562 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fb034001870 con 0x7fb04410b850 2024-04-14T15:29:22.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.531+0000 7fb02bfff700 1 -- 172.21.15.90:0/3290801562 <== mon.2 v1:172.21.15.90:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2475122419 0 0) 0x7fb044103580 con 0x7fb04410f360 2024-04-14T15:29:22.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.531+0000 7fb02bfff700 1 -- 172.21.15.90:0/3290801562 --> v1:172.21.15.90:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb02c0035b0 con 0x7fb04410f360 2024-04-14T15:29:22.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.531+0000 7fb02bfff700 1 -- 172.21.15.90:0/3290801562 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1843550715 0 0) 0x7fb044103370 con 0x7fb044103a40 2024-04-14T15:29:22.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.531+0000 7fb02bfff700 1 -- 172.21.15.90:0/3290801562 --> v1:172.21.15.92:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb044103580 con 0x7fb044103a40 2024-04-14T15:29:22.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.531+0000 7fb02bfff700 1 -- 172.21.15.90:0/3290801562 <== mon.2 v1:172.21.15.90:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fb03c001bb0 con 0x7fb04410f360 2024-04-14T15:29:22.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.531+0000 7fb02bfff700 1 -- 172.21.15.90:0/3290801562 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fb038001430 con 0x7fb044103a40 2024-04-14T15:29:22.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.531+0000 7fb02bfff700 1 -- 172.21.15.90:0/3290801562 <== mon.0 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1608754664 0 0) 0x7fb0441ac300 con 0x7fb04410b850 2024-04-14T15:29:22.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.531+0000 7fb02bfff700 1 -- 172.21.15.90:0/3290801562 >> v1:172.21.15.90:6790/0 conn(0x7fb04410f360 legacy=0x7fb0441a8bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:29:22.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.531+0000 7fb02bfff700 1 -- 172.21.15.90:0/3290801562 >> v1:172.21.15.92:6789/0 conn(0x7fb044103a40 legacy=0x7fb0441016a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:29:22.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.531+0000 7fb02bfff700 1 -- 172.21.15.90:0/3290801562 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0441ad520 con 0x7fb04410b850 2024-04-14T15:29:22.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.531+0000 7fb049115700 1 -- 172.21.15.90:0/3290801562 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb0441ac530 con 0x7fb04410b850 2024-04-14T15:29:22.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.531+0000 7fb049115700 10 client.38427.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-14T15:29:22.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.531+0000 7fb049115700 1 -- 172.21.15.90:0/3290801562 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb0441ac9f0 con 0x7fb04410b850 2024-04-14T15:29:22.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.531+0000 7fb049115700 1 librados: init done 2024-04-14T15:29:22.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.531+0000 7fb02bfff700 1 -- 172.21.15.90:0/3290801562 <== mon.0 v1:172.21.15.90:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1040907098 0 0) 0x7fb034001c40 con 0x7fb04410b850 2024-04-14T15:29:22.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.531+0000 7fb02bfff700 1 -- 172.21.15.90:0/3290801562 <== mon.0 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 399874170 0 0) 0x7fb034005d30 con 0x7fb04410b850 2024-04-14T15:29:22.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.532+0000 7fb049115700 1 -- 172.21.15.90:0/3290801562 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb030005120 con 0x7fb04410b850 2024-04-14T15:29:22.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.533+0000 7fb02bfff700 1 -- 172.21.15.90:0/3290801562 <== mon.0 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 22) ==== 92680+0+0 (unknown 3183958143 0 0) 0x7fb03401c9c0 con 0x7fb04410b850 2024-04-14T15:29:22.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.533+0000 7fb02bfff700 1 -- 172.21.15.90:0/3290801562 <== mon.0 v1:172.21.15.90:6789/0 8 ==== osd_map(926..926 src has 467..926) ==== 7517+0+0 (unknown 2322621496 0 0) 0x7fb03408a750 con 0x7fb04410b850 2024-04-14T15:29:22.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.533+0000 7fb02bfff700 10 client.38427.objecter ms_dispatch 0x7fb044001550 osd_map(926..926 src has 467..926) 2024-04-14T15:29:22.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.533+0000 7fb02bfff700 3 client.38427.objecter handle_osd_map got epochs [926,926] > 0 2024-04-14T15:29:22.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.533+0000 7fb02bfff700 3 client.38427.objecter handle_osd_map decoding full epoch 926 2024-04-14T15:29:22.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.533+0000 7fb02bfff700 20 client.38427.objecter dump_active .. 0 homeless 2024-04-14T15:29:22.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.533+0000 7fb02bfff700 10 client.38427.objecter ms_handle_connect 0x7fb02c071370 2024-04-14T15:29:22.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.538+0000 7fb02bfff700 1 -- 172.21.15.90:0/3290801562 <== mon.0 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188297 (unknown 1092875540 0 2267177403) 0x7fb034003950 con 0x7fb04410b850 2024-04-14T15:29:22.691 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:22 smithi090 ceph-mon[41670]: pgmap v1419: 174 pgs: 174 active+clean; 582 KiB data, 844 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:29:22.691 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:22 smithi090 ceph-mon[41670]: 6.b scrub starts 2024-04-14T15:29:22.691 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:22 smithi090 ceph-mon[41670]: 6.b scrub ok 2024-04-14T15:29:22.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:22 smithi090 ceph-mon[30399]: pgmap v1419: 174 pgs: 174 active+clean; 582 KiB data, 844 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:29:22.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:22 smithi090 ceph-mon[30399]: 6.b scrub starts 2024-04-14T15:29:22.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:22 smithi090 ceph-mon[30399]: 6.b scrub ok 2024-04-14T15:29:22.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.739+0000 7fb049115700 1 -- 172.21.15.90:0/3290801562 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} v 0) -- 0x7fb030005410 con 0x7fb04410b850 2024-04-14T15:29:22.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.742+0000 7fb02bfff700 1 -- 172.21.15.90:0/3290801562 <== mon.0 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]=0 v33) ==== 118+0+0 (unknown 3727294725 0 0) 0x7fb034057450 con 0x7fb04410b850 2024-04-14T15:29:22.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.745+0000 7fb049115700 10 librados: watch_flush enter 2024-04-14T15:29:22.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.745+0000 7fb049115700 10 librados: watch_flush exit 2024-04-14T15:29:22.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.745+0000 7fb049115700 20 client.38427.objecter shutdown clearing up homeless session... 2024-04-14T15:29:22.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.745+0000 7fb049115700 10 client.38427.objecter successfully canceled tick 2024-04-14T15:29:22.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.745+0000 7fb049115700 1 -- 172.21.15.90:0/3290801562 >> v1:172.21.15.90:6800/420935427 conn(0x7fb02c071370 legacy=0x7fb02c073830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:29:22.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.745+0000 7fb049115700 1 -- 172.21.15.90:0/3290801562 >> v1:172.21.15.90:6789/0 conn(0x7fb04410b850 legacy=0x7fb044101db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T15:29:22.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.745+0000 7fb049115700 1 -- 172.21.15.90:0/3290801562 shutdown_connections 2024-04-14T15:29:22.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.745+0000 7fb049115700 1 -- 172.21.15.90:0/3290801562 >> 172.21.15.90:0/3290801562 conn(0x7fb0440fd290 msgr2=0x7fb0440ff680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T15:29:22.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.745+0000 7fb049115700 1 -- 172.21.15.90:0/3290801562 shutdown_connections 2024-04-14T15:29:22.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.745+0000 7fb049115700 1 -- 172.21.15.90:0/3290801562 wait complete. 2024-04-14T15:29:22.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-14T15:29:22.745+0000 7fb049115700 1 librados: shutdown 2024-04-14T15:29:22.772 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:22 smithi092 ceph-mon[33966]: pgmap v1419: 174 pgs: 174 active+clean; 582 KiB data, 844 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:29:22.772 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:22 smithi092 ceph-mon[33966]: 6.b scrub starts 2024-04-14T15:29:22.772 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:22 smithi092 ceph-mon[33966]: 6.b scrub ok 2024-04-14T15:29:23.054 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:29:22 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: [14/Apr/2024:15:29:22] ENGINE Bus STOPPING 2024-04-14T15:29:23.054 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:29:22 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: [14/Apr/2024:15:29:22] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-14T15:29:23.054 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:29:22 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: [14/Apr/2024:15:29:22] ENGINE Bus STOPPED 2024-04-14T15:29:23.054 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:29:22 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: [14/Apr/2024:15:29:22] ENGINE Bus STARTING 2024-04-14T15:29:23.054 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:29:22 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: [14/Apr/2024:15:29:22] ENGINE Serving on http://:::9283 2024-04-14T15:29:23.054 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:29:22 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: [14/Apr/2024:15:29:22] ENGINE Bus STARTED 2024-04-14T15:29:23.490 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-14T15:29:23.504 INFO:tasks.cephadm:Teardown begin 2024-04-14T15:29:23.505 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-14T15:29:23.534 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-14T15:29:23.568 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-14T15:29:23.568 DEBUG:teuthology.orchestra.run.smithi090:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-14T15:29:23.590 DEBUG:teuthology.orchestra.run.smithi092:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-14T15:29:23.625 INFO:tasks.cephadm:Stopping all daemons... 2024-04-14T15:29:23.625 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-14T15:29:23.625 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ceph-0e288098-fa70-11ee-bc8e-c7b262605968@mon.a 2024-04-14T15:29:23.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:23 smithi092 ceph-mon[33966]: 5.14 scrub starts 2024-04-14T15:29:23.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:23 smithi092 ceph-mon[33966]: 5.14 scrub ok 2024-04-14T15:29:23.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:23 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3290801562' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-04-14T15:29:23.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:23 smithi092 ceph-mon[33966]: from='client.? v1:172.21.15.90:0/3290801562' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-04-14T15:29:23.873 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:23 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:29:23.873 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:23 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:29:23.873 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:23 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:29:23.873 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:23 smithi092 ceph-mon[33966]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:29:23.903 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:23 smithi090 ceph-mon[41670]: 5.14 scrub starts 2024-04-14T15:29:23.904 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:23 smithi090 ceph-mon[41670]: 5.14 scrub ok 2024-04-14T15:29:23.904 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:23 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3290801562' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-04-14T15:29:23.904 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:23 smithi090 ceph-mon[41670]: from='client.? v1:172.21.15.90:0/3290801562' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-04-14T15:29:23.904 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:23 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:29:23.904 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:23 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:29:23.904 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:23 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:29:23.905 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:23 smithi090 ceph-mon[41670]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:29:23.905 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:23 smithi090 ceph-mon[30399]: 5.14 scrub starts 2024-04-14T15:29:23.905 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:23 smithi090 ceph-mon[30399]: 5.14 scrub ok 2024-04-14T15:29:23.905 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:23 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3290801562' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-04-14T15:29:23.906 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:23 smithi090 ceph-mon[30399]: from='client.? v1:172.21.15.90:0/3290801562' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-04-14T15:29:23.906 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:23 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:29:23.906 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:23 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:29:23.906 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:23 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:29:23.906 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:23 smithi090 ceph-mon[30399]: from='mgr.14610 v1:172.21.15.90:0/1178040626' entity='mgr.y' 2024-04-14T15:29:23.906 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:23 smithi090 systemd[1]: Stopping Ceph mon.a for 0e288098-fa70-11ee-bc8e-c7b262605968... 2024-04-14T15:29:24.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:23 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mon-a[30376]: 2024-04-14T15:29:23.902+0000 7f016c3b3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T15:29:24.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:23 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mon-a[30376]: 2024-04-14T15:29:23.902+0000 7f016c3b3700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-14T15:29:24.192 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:23 smithi090 podman[292269]: 2024-04-14 15:29:23.964432707 +0000 UTC m=+0.146767152 container died fa32a6dcdb6ac5d02ad3385108a5352762c685236cdccde67fdc46db0001ec1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mon-a, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-a971fd5, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-14T15:29:24.192 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:29:23 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: [14/Apr/2024:15:29:23] ENGINE Bus STOPPING 2024-04-14T15:29:24.192 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:29:23 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: [14/Apr/2024:15:29:23] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-14T15:29:24.192 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:29:23 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: [14/Apr/2024:15:29:23] ENGINE Bus STOPPED 2024-04-14T15:29:24.192 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:29:23 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: [14/Apr/2024:15:29:23] ENGINE Bus STARTING 2024-04-14T15:29:24.192 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:29:23 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: ::ffff:172.21.15.92 - - [14/Apr/2024:15:29:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-14T15:29:24.192 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:29:24 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: [14/Apr/2024:15:29:24] ENGINE Serving on http://:::9283 2024-04-14T15:29:24.192 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:29:24 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: [14/Apr/2024:15:29:24] ENGINE Bus STARTED 2024-04-14T15:29:24.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:24 smithi090 podman[292269]: 2024-04-14 15:29:24.444420887 +0000 UTC m=+0.626755383 container cleanup fa32a6dcdb6ac5d02ad3385108a5352762c685236cdccde67fdc46db0001ec1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mon-a, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, RELEASE=main-a971fd5, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-14T15:29:24.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 14 15:29:24 smithi090 bash[292269]: ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mon-a 2024-04-14T15:29:24.853 DEBUG:teuthology.orchestra.run.smithi090:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0e288098-fa70-11ee-bc8e-c7b262605968@mon.a.service' 2024-04-14T15:29:24.895 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-14T15:29:24.895 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-14T15:29:24.895 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-04-14T15:29:24.895 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ceph-0e288098-fa70-11ee-bc8e-c7b262605968@mon.c 2024-04-14T15:29:25.260 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:24 smithi090 systemd[1]: Stopping Ceph mon.c for 0e288098-fa70-11ee-bc8e-c7b262605968... 2024-04-14T15:29:25.541 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:25 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mon-c[41647]: 2024-04-14T15:29:25.260+0000 7f9001c0d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T15:29:25.542 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:25 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mon-c[41647]: 2024-04-14T15:29:25.260+0000 7f9001c0d700 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-04-14T15:29:25.542 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:25 smithi090 podman[292412]: 2024-04-14 15:29:25.281834682 +0000 UTC m=+0.165463786 container died bc18858f949463086f3893ab3d98f541b737bb3019d18609ec36256b6a69944b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mon-c, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=main-a971fd5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=) 2024-04-14T15:29:25.900 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:25 smithi090 podman[292412]: 2024-04-14 15:29:25.641070424 +0000 UTC m=+0.524699510 container cleanup bc18858f949463086f3893ab3d98f541b737bb3019d18609ec36256b6a69944b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, RELEASE=main-a971fd5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-14T15:29:25.900 INFO:journalctl@ceph.mon.c.smithi090.stdout:Apr 14 15:29:25 smithi090 bash[292412]: ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mon-c 2024-04-14T15:29:26.064 DEBUG:teuthology.orchestra.run.smithi090:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0e288098-fa70-11ee-bc8e-c7b262605968@mon.c.service' 2024-04-14T15:29:26.101 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-14T15:29:26.101 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-04-14T15:29:26.101 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-04-14T15:29:26.101 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ceph-0e288098-fa70-11ee-bc8e-c7b262605968@mon.b 2024-04-14T15:29:26.122 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:29:25 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug there is no tcmu-runner data available 2024-04-14T15:29:26.445 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:26 smithi092 systemd[1]: Stopping Ceph mon.b for 0e288098-fa70-11ee-bc8e-c7b262605968... 2024-04-14T15:29:26.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:26 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mon-b[33943]: 2024-04-14T15:29:26.444+0000 7fb9099d3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T15:29:26.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:26 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mon-b[33943]: 2024-04-14T15:29:26.444+0000 7fb9099d3700 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-04-14T15:29:26.872 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:26 smithi092 podman[151853]: 2024-04-14 15:29:26.568545499 +0000 UTC m=+0.271735958 container died 20e24def0d377ff86771074a51111996756791ca0d91c9254ac1aef978b70d70 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mon-b, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=main-a971fd5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-14T15:29:27.268 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:27 smithi092 podman[151853]: 2024-04-14 15:29:27.07560306 +0000 UTC m=+0.778793520 container cleanup 20e24def0d377ff86771074a51111996756791ca0d91c9254ac1aef978b70d70 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mon-b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=main-a971fd5, org.label-schema.build-date=20240403, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0) 2024-04-14T15:29:27.268 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:27 smithi092 bash[151853]: ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mon-b 2024-04-14T15:29:27.504 DEBUG:teuthology.orchestra.run.smithi092:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0e288098-fa70-11ee-bc8e-c7b262605968@mon.b.service' 2024-04-14T15:29:27.569 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:27 smithi092 podman[151872]: 2024-04-14 15:29:27.268738649 +0000 UTC m=+0.699711244 container remove 20e24def0d377ff86771074a51111996756791ca0d91c9254ac1aef978b70d70 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mon-b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240403, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=main-a971fd5) 2024-04-14T15:29:27.569 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:27 smithi092 systemd[1]: ceph-0e288098-fa70-11ee-bc8e-c7b262605968@mon.b.service: Deactivated successfully. 2024-04-14T15:29:27.570 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:27 smithi092 systemd[1]: Stopped Ceph mon.b for 0e288098-fa70-11ee-bc8e-c7b262605968. 2024-04-14T15:29:27.570 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 14 15:29:27 smithi092 systemd[1]: ceph-0e288098-fa70-11ee-bc8e-c7b262605968@mon.b.service: Consumed 31.738s CPU time. 2024-04-14T15:29:27.593 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-14T15:29:27.593 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-04-14T15:29:27.593 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-14T15:29:27.593 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ceph-0e288098-fa70-11ee-bc8e-c7b262605968@mgr.y 2024-04-14T15:29:27.940 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:29:27 smithi090 systemd[1]: Stopping Ceph mgr.y for 0e288098-fa70-11ee-bc8e-c7b262605968... 2024-04-14T15:29:28.191 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:29:27 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y[30656]: 2024-04-14T15:29:27.939+0000 7f562b58b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T15:29:28.191 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:29:27 smithi090 podman[292542]: 2024-04-14 15:29:27.971356688 +0000 UTC m=+0.187150821 container died eb9ab6f9fe2001a8c8f0c236fc55a58b39133bc2d933d65b2f688b6110e3fff6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=main-a971fd5, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-04-14T15:29:28.657 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:29:28 smithi090 podman[292542]: 2024-04-14 15:29:28.455837758 +0000 UTC m=+0.671631888 container cleanup eb9ab6f9fe2001a8c8f0c236fc55a58b39133bc2d933d65b2f688b6110e3fff6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y, CEPH_POINT_RELEASE=, RELEASE=main-a971fd5, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-14T15:29:28.657 INFO:journalctl@ceph.mgr.y.smithi090.stdout:Apr 14 15:29:28 smithi090 bash[292542]: ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-y 2024-04-14T15:29:28.876 DEBUG:teuthology.orchestra.run.smithi090:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0e288098-fa70-11ee-bc8e-c7b262605968@mgr.y.service' 2024-04-14T15:29:28.912 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-14T15:29:28.912 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-14T15:29:28.912 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-14T15:29:28.912 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ceph-0e288098-fa70-11ee-bc8e-c7b262605968@mgr.x 2024-04-14T15:29:29.253 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:29:28 smithi092 systemd[1]: Stopping Ceph mgr.x for 0e288098-fa70-11ee-bc8e-c7b262605968... 2024-04-14T15:29:29.539 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:29:29 smithi092 podman[151983]: 2024-04-14 15:29:29.253795542 +0000 UTC m=+0.149914723 container died 503f8922bf5434a3ec443850ec202ae388254ad5c982bd674a19ec1e1176deb8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, RELEASE=main-a971fd5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-14T15:29:29.872 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:29:29 smithi092 podman[151983]: 2024-04-14 15:29:29.658344326 +0000 UTC m=+0.554463514 container cleanup 503f8922bf5434a3ec443850ec202ae388254ad5c982bd674a19ec1e1176deb8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-x, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=main-a971fd5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-14T15:29:29.872 INFO:journalctl@ceph.mgr.x.smithi092.stdout:Apr 14 15:29:29 smithi092 bash[151983]: ceph-0e288098-fa70-11ee-bc8e-c7b262605968-mgr-x 2024-04-14T15:29:30.013 DEBUG:teuthology.orchestra.run.smithi092:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0e288098-fa70-11ee-bc8e-c7b262605968@mgr.x.service' 2024-04-14T15:29:30.051 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-14T15:29:30.051 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-14T15:29:30.051 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-14T15:29:30.051 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ceph-0e288098-fa70-11ee-bc8e-c7b262605968@osd.0 2024-04-14T15:29:30.412 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 14 15:29:30 smithi090 systemd[1]: Stopping Ceph osd.0 for 0e288098-fa70-11ee-bc8e-c7b262605968... 2024-04-14T15:29:30.691 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 14 15:29:30 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-0[46039]: 2024-04-14T15:29:30.411+0000 7fb4b4eb4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T15:29:30.691 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 14 15:29:30 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-0[46039]: 2024-04-14T15:29:30.411+0000 7fb4b4eb4700 -1 osd.0 926 *** Got signal Terminated *** 2024-04-14T15:29:30.691 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 14 15:29:30 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-0[46039]: 2024-04-14T15:29:30.411+0000 7fb4b4eb4700 -1 osd.0 926 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-14T15:29:36.191 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 14 15:29:35 smithi090 podman[292673]: 2024-04-14 15:29:35.91775416 +0000 UTC m=+5.672919757 container died 2a3c4fd68da9ad787b8a0a163c6cf2063abd5539ff655ecb362b72c202a48fbf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=main-a971fd5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-14T15:29:36.691 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 14 15:29:36 smithi090 podman[292673]: 2024-04-14 15:29:36.610196268 +0000 UTC m=+6.365361859 container cleanup 2a3c4fd68da9ad787b8a0a163c6cf2063abd5539ff655ecb362b72c202a48fbf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-0, RELEASE=main-a971fd5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-14T15:29:36.691 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 14 15:29:36 smithi090 bash[292673]: ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-0 2024-04-14T15:29:37.160 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 14 15:29:36 smithi090 podman[292696]: 2024-04-14 15:29:36.861872318 +0000 UTC m=+0.939211857 container remove 2a3c4fd68da9ad787b8a0a163c6cf2063abd5539ff655ecb362b72c202a48fbf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a971fd5, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-14T15:29:37.414 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 14 15:29:37 smithi090 podman[292795]: 2024-04-14 15:29:37.063611313 +0000 UTC m=+0.017565276 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608 2024-04-14T15:29:37.414 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 14 15:29:37 smithi090 podman[292795]: 2024-04-14 15:29:37.179416916 +0000 UTC m=+0.133370874 container create 63efc706e090d3752485b9643f3db73afb7c41b0cc56713974a7b71221c4c62f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=main-a971fd5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True) 2024-04-14T15:29:37.414 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 14 15:29:37 smithi090 podman[292795]: 2024-04-14 15:29:37.408409475 +0000 UTC m=+0.362363428 container init 63efc706e090d3752485b9643f3db73afb7c41b0cc56713974a7b71221c4c62f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=main-a971fd5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, GIT_CLEAN=True) 2024-04-14T15:29:37.414 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 14 15:29:37 smithi090 podman[292795]: 2024-04-14 15:29:37.414229322 +0000 UTC m=+0.368183278 container start 63efc706e090d3752485b9643f3db73afb7c41b0cc56713974a7b71221c4c62f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, RELEASE=main-a971fd5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-14T15:29:37.691 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 14 15:29:37 smithi090 podman[292795]: 2024-04-14 15:29:37.508460149 +0000 UTC m=+0.462414108 container attach 63efc706e090d3752485b9643f3db73afb7c41b0cc56713974a7b71221c4c62f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-0-deactivate, RELEASE=main-a971fd5, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-14T15:29:38.060 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 14 15:29:37 smithi090 podman[292795]: 2024-04-14 15:29:37.707018854 +0000 UTC m=+0.660972806 container died 63efc706e090d3752485b9643f3db73afb7c41b0cc56713974a7b71221c4c62f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=main-a971fd5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-14T15:29:38.453 DEBUG:teuthology.orchestra.run.smithi090:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0e288098-fa70-11ee-bc8e-c7b262605968@osd.0.service' 2024-04-14T15:29:38.496 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-14T15:29:38.497 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-14T15:29:38.497 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-14T15:29:38.497 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ceph-0e288098-fa70-11ee-bc8e-c7b262605968@osd.1 2024-04-14T15:29:38.691 INFO:journalctl@ceph.osd.1.smithi090.stdout:Apr 14 15:29:38 smithi090 systemd[1]: Stopping Ceph osd.1 for 0e288098-fa70-11ee-bc8e-c7b262605968... 2024-04-14T15:29:39.191 INFO:journalctl@ceph.osd.1.smithi090.stdout:Apr 14 15:29:38 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-1[52165]: 2024-04-14T15:29:38.825+0000 7f18c4ad5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T15:29:39.191 INFO:journalctl@ceph.osd.1.smithi090.stdout:Apr 14 15:29:38 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-1[52165]: 2024-04-14T15:29:38.825+0000 7f18c4ad5700 -1 osd.1 926 *** Got signal Terminated *** 2024-04-14T15:29:39.192 INFO:journalctl@ceph.osd.1.smithi090.stdout:Apr 14 15:29:38 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-1[52165]: 2024-04-14T15:29:38.825+0000 7f18c4ad5700 -1 osd.1 926 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-14T15:29:44.593 INFO:journalctl@ceph.osd.1.smithi090.stdout:Apr 14 15:29:44 smithi090 podman[292925]: 2024-04-14 15:29:44.322382521 +0000 UTC m=+5.598001032 container died b94492f3b4e4113eae61d97461c7158342d1b59f4201709f2dacf9de40c20e45 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-1, RELEASE=main-a971fd5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240403) 2024-04-14T15:29:44.941 INFO:journalctl@ceph.osd.1.smithi090.stdout:Apr 14 15:29:44 smithi090 podman[292925]: 2024-04-14 15:29:44.767964061 +0000 UTC m=+6.043582570 container cleanup b94492f3b4e4113eae61d97461c7158342d1b59f4201709f2dacf9de40c20e45 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-1, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a971fd5, io.buildah.version=1.33.5) 2024-04-14T15:29:44.941 INFO:journalctl@ceph.osd.1.smithi090.stdout:Apr 14 15:29:44 smithi090 bash[292925]: ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-1 2024-04-14T15:29:45.374 INFO:journalctl@ceph.osd.1.smithi090.stdout:Apr 14 15:29:45 smithi090 podman[292959]: 2024-04-14 15:29:45.052937139 +0000 UTC m=+0.730224875 container remove b94492f3b4e4113eae61d97461c7158342d1b59f4201709f2dacf9de40c20e45 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-1, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=main-a971fd5, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, ceph=True) 2024-04-14T15:29:45.691 INFO:journalctl@ceph.osd.1.smithi090.stdout:Apr 14 15:29:45 smithi090 podman[293057]: 2024-04-14 15:29:45.277707593 +0000 UTC m=+0.017764244 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608 2024-04-14T15:29:45.692 INFO:journalctl@ceph.osd.1.smithi090.stdout:Apr 14 15:29:45 smithi090 podman[293057]: 2024-04-14 15:29:45.378943913 +0000 UTC m=+0.119000606 container create 9b7d000cc378f1e84812a583fb833e5b7b28c546f8b565dd37b31b2b5c9d2dd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-1-deactivate, RELEASE=main-a971fd5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403) 2024-04-14T15:29:45.692 INFO:journalctl@ceph.osd.1.smithi090.stdout:Apr 14 15:29:45 smithi090 podman[293057]: 2024-04-14 15:29:45.614957078 +0000 UTC m=+0.355013728 container init 9b7d000cc378f1e84812a583fb833e5b7b28c546f8b565dd37b31b2b5c9d2dd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240403, GIT_BRANCH=HEAD, RELEASE=main-a971fd5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True) 2024-04-14T15:29:45.692 INFO:journalctl@ceph.osd.1.smithi090.stdout:Apr 14 15:29:45 smithi090 podman[293057]: 2024-04-14 15:29:45.620403582 +0000 UTC m=+0.360460233 container start 9b7d000cc378f1e84812a583fb833e5b7b28c546f8b565dd37b31b2b5c9d2dd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-1-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=main-a971fd5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240403) 2024-04-14T15:29:46.191 INFO:journalctl@ceph.osd.1.smithi090.stdout:Apr 14 15:29:45 smithi090 podman[293057]: 2024-04-14 15:29:45.740171093 +0000 UTC m=+0.480227746 container attach 9b7d000cc378f1e84812a583fb833e5b7b28c546f8b565dd37b31b2b5c9d2dd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-1-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=main-a971fd5) 2024-04-14T15:29:46.191 INFO:journalctl@ceph.osd.1.smithi090.stdout:Apr 14 15:29:45 smithi090 podman[293057]: 2024-04-14 15:29:45.910277063 +0000 UTC m=+0.650333715 container died 9b7d000cc378f1e84812a583fb833e5b7b28c546f8b565dd37b31b2b5c9d2dd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=main-a971fd5, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403) 2024-04-14T15:29:46.612 DEBUG:teuthology.orchestra.run.smithi090:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0e288098-fa70-11ee-bc8e-c7b262605968@osd.1.service' 2024-04-14T15:29:46.654 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-14T15:29:46.654 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-14T15:29:46.654 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-14T15:29:46.654 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ceph-0e288098-fa70-11ee-bc8e-c7b262605968@osd.2 2024-04-14T15:29:46.941 INFO:journalctl@ceph.osd.2.smithi090.stdout:Apr 14 15:29:46 smithi090 systemd[1]: Stopping Ceph osd.2 for 0e288098-fa70-11ee-bc8e-c7b262605968... 2024-04-14T15:29:47.441 INFO:journalctl@ceph.osd.2.smithi090.stdout:Apr 14 15:29:46 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-2[58450]: 2024-04-14T15:29:46.973+0000 7f7929a74700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T15:29:47.441 INFO:journalctl@ceph.osd.2.smithi090.stdout:Apr 14 15:29:46 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-2[58450]: 2024-04-14T15:29:46.973+0000 7f7929a74700 -1 osd.2 926 *** Got signal Terminated *** 2024-04-14T15:29:47.441 INFO:journalctl@ceph.osd.2.smithi090.stdout:Apr 14 15:29:46 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-2[58450]: 2024-04-14T15:29:46.973+0000 7f7929a74700 -1 osd.2 926 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-14T15:29:52.775 INFO:journalctl@ceph.osd.2.smithi090.stdout:Apr 14 15:29:52 smithi090 podman[293187]: 2024-04-14 15:29:52.50415546 +0000 UTC m=+5.629192604 container died 9159c3788b8ca620f402e2def481a697edd051d68a4bae35b308975e8fbb4623 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-2, io.buildah.version=1.33.5, RELEASE=main-a971fd5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403) 2024-04-14T15:29:53.100 INFO:journalctl@ceph.osd.2.smithi090.stdout:Apr 14 15:29:52 smithi090 podman[293187]: 2024-04-14 15:29:52.932249877 +0000 UTC m=+6.057287071 container cleanup 9159c3788b8ca620f402e2def481a697edd051d68a4bae35b308975e8fbb4623 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=main-a971fd5, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-14T15:29:53.100 INFO:journalctl@ceph.osd.2.smithi090.stdout:Apr 14 15:29:52 smithi090 bash[293187]: ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-2 2024-04-14T15:29:53.441 INFO:journalctl@ceph.osd.2.smithi090.stdout:Apr 14 15:29:53 smithi090 podman[293209]: 2024-04-14 15:29:53.100346657 +0000 UTC m=+0.598705369 container remove 9159c3788b8ca620f402e2def481a697edd051d68a4bae35b308975e8fbb4623 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240403, RELEASE=main-a971fd5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-14T15:29:53.871 INFO:journalctl@ceph.osd.2.smithi090.stdout:Apr 14 15:29:53 smithi090 podman[293310]: 2024-04-14 15:29:53.535943262 +0000 UTC m=+0.114682673 container create a158728354c8031b37804c534e36b94884290b52fef0340731538e25065493c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-2-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a971fd5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-14T15:29:53.871 INFO:journalctl@ceph.osd.2.smithi090.stdout:Apr 14 15:29:53 smithi090 podman[293310]: 2024-04-14 15:29:53.438943383 +0000 UTC m=+0.017682803 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608 2024-04-14T15:29:53.872 INFO:journalctl@ceph.osd.2.smithi090.stdout:Apr 14 15:29:53 smithi090 podman[293310]: 2024-04-14 15:29:53.771751189 +0000 UTC m=+0.350490600 container init a158728354c8031b37804c534e36b94884290b52fef0340731538e25065493c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-2-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=main-a971fd5, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS) 2024-04-14T15:29:53.872 INFO:journalctl@ceph.osd.2.smithi090.stdout:Apr 14 15:29:53 smithi090 podman[293310]: 2024-04-14 15:29:53.777355289 +0000 UTC m=+0.356094706 container start a158728354c8031b37804c534e36b94884290b52fef0340731538e25065493c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-2-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=main-a971fd5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-14T15:29:53.872 INFO:journalctl@ceph.osd.2.smithi090.stdout:Apr 14 15:29:53 smithi090 podman[293310]: 2024-04-14 15:29:53.871821818 +0000 UTC m=+0.450561249 container attach a158728354c8031b37804c534e36b94884290b52fef0340731538e25065493c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-2-deactivate, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, RELEASE=main-a971fd5, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-14T15:29:54.191 INFO:journalctl@ceph.osd.2.smithi090.stdout:Apr 14 15:29:54 smithi090 podman[293310]: 2024-04-14 15:29:54.061821119 +0000 UTC m=+0.640560521 container died a158728354c8031b37804c534e36b94884290b52fef0340731538e25065493c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=main-a971fd5, maintainer=Guillaume Abrioux ) 2024-04-14T15:29:54.731 DEBUG:teuthology.orchestra.run.smithi090:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0e288098-fa70-11ee-bc8e-c7b262605968@osd.2.service' 2024-04-14T15:29:54.772 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-14T15:29:54.772 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-14T15:29:54.773 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-14T15:29:54.773 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ceph-0e288098-fa70-11ee-bc8e-c7b262605968@osd.3 2024-04-14T15:29:55.089 INFO:journalctl@ceph.osd.3.smithi090.stdout:Apr 14 15:29:54 smithi090 systemd[1]: Stopping Ceph osd.3 for 0e288098-fa70-11ee-bc8e-c7b262605968... 2024-04-14T15:29:55.441 INFO:journalctl@ceph.osd.3.smithi090.stdout:Apr 14 15:29:55 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-3[64642]: 2024-04-14T15:29:55.088+0000 7f57def36700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T15:29:55.441 INFO:journalctl@ceph.osd.3.smithi090.stdout:Apr 14 15:29:55 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-3[64642]: 2024-04-14T15:29:55.088+0000 7f57def36700 -1 osd.3 926 *** Got signal Terminated *** 2024-04-14T15:29:55.441 INFO:journalctl@ceph.osd.3.smithi090.stdout:Apr 14 15:29:55 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-3[64642]: 2024-04-14T15:29:55.088+0000 7f57def36700 -1 osd.3 926 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-14T15:29:56.622 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:29:56 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6[52258]: 2024-04-14T15:29:56.390+0000 7fb5db8ea700 -1 osd.6 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:31.616672+0000 front 2024-04-14T15:29:31.616871+0000 (oldest deadline 2024-04-14T15:29:56.316800+0000) 2024-04-14T15:29:56.622 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:29:56 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:29:56.220+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:34.221673+0000 front 2024-04-14T15:29:34.221775+0000 (oldest deadline 2024-04-14T15:29:55.921555+0000) 2024-04-14T15:29:56.623 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:29:56 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[70312]: ts=2024-04-14T15:29:56.189Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.90:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.90:8765: connect: connection refused" 2024-04-14T15:29:56.623 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:29:56 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[70312]: ts=2024-04-14T15:29:56.190Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.90:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.90:8765: connect: connection refused" 2024-04-14T15:29:56.623 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:29:56 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[70312]: ts=2024-04-14T15:29:56.192Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.90:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.90:8765: connect: connection refused" 2024-04-14T15:29:56.623 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:29:56 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[70312]: ts=2024-04-14T15:29:56.192Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.90:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.90:8765: connect: connection refused" 2024-04-14T15:29:56.623 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:29:56 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[70312]: ts=2024-04-14T15:29:56.192Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.90:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.90:8765: connect: connection refused" 2024-04-14T15:29:57.622 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:29:57 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6[52258]: 2024-04-14T15:29:57.423+0000 7fb5db8ea700 -1 osd.6 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:31.616672+0000 front 2024-04-14T15:29:31.616871+0000 (oldest deadline 2024-04-14T15:29:56.316800+0000) 2024-04-14T15:29:57.622 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:29:57 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:29:57.182+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:34.221673+0000 front 2024-04-14T15:29:34.221775+0000 (oldest deadline 2024-04-14T15:29:55.921555+0000) 2024-04-14T15:29:58.438 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:29:58 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:29:58.217+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:34.221673+0000 front 2024-04-14T15:29:34.221775+0000 (oldest deadline 2024-04-14T15:29:55.921555+0000) 2024-04-14T15:29:58.438 INFO:journalctl@ceph.osd.5.smithi092.stdout:Apr 14 15:29:58 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-5[45851]: 2024-04-14T15:29:58.141+0000 7f20edfee700 -1 osd.5 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:33.315664+0000 front 2024-04-14T15:29:33.315504+0000 (oldest deadline 2024-04-14T15:29:58.015582+0000) 2024-04-14T15:29:58.872 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:29:58 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6[52258]: 2024-04-14T15:29:58.438+0000 7fb5db8ea700 -1 osd.6 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:31.616672+0000 front 2024-04-14T15:29:31.616871+0000 (oldest deadline 2024-04-14T15:29:56.316800+0000) 2024-04-14T15:29:59.191 INFO:journalctl@ceph.osd.3.smithi090.stdout:Apr 14 15:29:58 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-3[64642]: 2024-04-14T15:29:58.709+0000 7f57db52a700 -1 osd.3 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:32.286816+0000 front 2024-04-14T15:29:32.286840+0000 (oldest deadline 2024-04-14T15:29:58.187032+0000) 2024-04-14T15:29:59.460 INFO:journalctl@ceph.osd.5.smithi092.stdout:Apr 14 15:29:59 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-5[45851]: 2024-04-14T15:29:59.175+0000 7f20edfee700 -1 osd.5 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:33.315664+0000 front 2024-04-14T15:29:33.315504+0000 (oldest deadline 2024-04-14T15:29:58.015582+0000) 2024-04-14T15:29:59.461 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:29:59 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:29:59.255+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:34.221673+0000 front 2024-04-14T15:29:34.221775+0000 (oldest deadline 2024-04-14T15:29:55.921555+0000) 2024-04-14T15:29:59.873 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:29:59 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6[52258]: 2024-04-14T15:29:59.459+0000 7fb5db8ea700 -1 osd.6 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:31.616672+0000 front 2024-04-14T15:29:31.616871+0000 (oldest deadline 2024-04-14T15:29:56.316800+0000) 2024-04-14T15:29:59.941 INFO:journalctl@ceph.osd.3.smithi090.stdout:Apr 14 15:29:59 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-3[64642]: 2024-04-14T15:29:59.687+0000 7f57db52a700 -1 osd.3 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:32.286816+0000 front 2024-04-14T15:29:32.286840+0000 (oldest deadline 2024-04-14T15:29:58.187032+0000) 2024-04-14T15:30:00.622 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:00 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6[52258]: 2024-04-14T15:30:00.440+0000 7fb5db8ea700 -1 osd.6 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:31.616672+0000 front 2024-04-14T15:29:31.616871+0000 (oldest deadline 2024-04-14T15:29:56.316800+0000) 2024-04-14T15:30:00.622 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:00 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:00.270+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:34.221673+0000 front 2024-04-14T15:29:34.221775+0000 (oldest deadline 2024-04-14T15:29:55.921555+0000) 2024-04-14T15:30:00.623 INFO:journalctl@ceph.osd.5.smithi092.stdout:Apr 14 15:30:00 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-5[45851]: 2024-04-14T15:30:00.195+0000 7f20edfee700 -1 osd.5 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:33.315664+0000 front 2024-04-14T15:29:33.315504+0000 (oldest deadline 2024-04-14T15:29:58.015582+0000) 2024-04-14T15:30:00.691 INFO:journalctl@ceph.osd.3.smithi090.stdout:Apr 14 15:30:00 smithi090 podman[293441]: 2024-04-14 15:30:00.473948837 +0000 UTC m=+5.485228038 container died b2078bb85a511ddb6a6cadb14035e922a74a081a9d8ad920c6cf85c1617ef86e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-3, org.label-schema.build-date=20240403, GIT_CLEAN=True, RELEASE=main-a971fd5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-14T15:30:01.131 INFO:journalctl@ceph.osd.3.smithi090.stdout:Apr 14 15:30:00 smithi090 podman[293441]: 2024-04-14 15:30:00.905528858 +0000 UTC m=+5.916808069 container cleanup b2078bb85a511ddb6a6cadb14035e922a74a081a9d8ad920c6cf85c1617ef86e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-3, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, RELEASE=main-a971fd5, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-14T15:30:01.131 INFO:journalctl@ceph.osd.3.smithi090.stdout:Apr 14 15:30:00 smithi090 bash[293441]: ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-3 2024-04-14T15:30:01.441 INFO:journalctl@ceph.osd.3.smithi090.stdout:Apr 14 15:30:01 smithi090 podman[293463]: 2024-04-14 15:30:01.132075334 +0000 UTC m=+0.657540066 container remove b2078bb85a511ddb6a6cadb14035e922a74a081a9d8ad920c6cf85c1617ef86e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-3, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, RELEASE=main-a971fd5, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403) 2024-04-14T15:30:01.622 INFO:journalctl@ceph.osd.5.smithi092.stdout:Apr 14 15:30:01 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-5[45851]: 2024-04-14T15:30:01.225+0000 7f20edfee700 -1 osd.5 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:33.315664+0000 front 2024-04-14T15:29:33.315504+0000 (oldest deadline 2024-04-14T15:29:58.015582+0000) 2024-04-14T15:30:01.622 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:01 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6[52258]: 2024-04-14T15:30:01.416+0000 7fb5db8ea700 -1 osd.6 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:31.616672+0000 front 2024-04-14T15:29:31.616871+0000 (oldest deadline 2024-04-14T15:29:56.316800+0000) 2024-04-14T15:30:01.622 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:01 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:01.226+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:34.221673+0000 front 2024-04-14T15:29:34.221775+0000 (oldest deadline 2024-04-14T15:29:55.921555+0000) 2024-04-14T15:30:01.876 INFO:journalctl@ceph.osd.3.smithi090.stdout:Apr 14 15:30:01 smithi090 podman[293558]: 2024-04-14 15:30:01.558072473 +0000 UTC m=+0.111673071 container create 058e2c5d018b6d515d5584d57df3867e576239543e8416eac30d9cdb31ca9ec5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, ceph=True, RELEASE=main-a971fd5, GIT_BRANCH=HEAD) 2024-04-14T15:30:01.877 INFO:journalctl@ceph.osd.3.smithi090.stdout:Apr 14 15:30:01 smithi090 podman[293558]: 2024-04-14 15:30:01.463977639 +0000 UTC m=+0.017578207 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608 2024-04-14T15:30:01.877 INFO:journalctl@ceph.osd.3.smithi090.stdout:Apr 14 15:30:01 smithi090 podman[293558]: 2024-04-14 15:30:01.768819678 +0000 UTC m=+0.322420234 container init 058e2c5d018b6d515d5584d57df3867e576239543e8416eac30d9cdb31ca9ec5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=main-a971fd5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-14T15:30:01.877 INFO:journalctl@ceph.osd.3.smithi090.stdout:Apr 14 15:30:01 smithi090 podman[293558]: 2024-04-14 15:30:01.774571945 +0000 UTC m=+0.328172504 container start 058e2c5d018b6d515d5584d57df3867e576239543e8416eac30d9cdb31ca9ec5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-3-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=main-a971fd5, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-14T15:30:01.877 INFO:journalctl@ceph.osd.3.smithi090.stdout:Apr 14 15:30:01 smithi090 podman[293558]: 2024-04-14 15:30:01.87693325 +0000 UTC m=+0.430533807 container attach 058e2c5d018b6d515d5584d57df3867e576239543e8416eac30d9cdb31ca9ec5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, RELEASE=main-a971fd5, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-14T15:30:02.191 INFO:journalctl@ceph.osd.3.smithi090.stdout:Apr 14 15:30:02 smithi090 podman[293558]: 2024-04-14 15:30:02.06514116 +0000 UTC m=+0.618741717 container died 058e2c5d018b6d515d5584d57df3867e576239543e8416eac30d9cdb31ca9ec5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, RELEASE=main-a971fd5, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux ) 2024-04-14T15:30:02.192 INFO:journalctl@ceph.osd.4.smithi092.stdout:Apr 14 15:30:01 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-4[39413]: 2024-04-14T15:30:01.890+0000 7f7456ddd700 -1 osd.4 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:35.290878+0000 front 2024-04-14T15:29:35.290925+0000 (oldest deadline 2024-04-14T15:30:01.190799+0000) 2024-04-14T15:30:02.622 INFO:journalctl@ceph.osd.5.smithi092.stdout:Apr 14 15:30:02 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-5[45851]: 2024-04-14T15:30:02.191+0000 7f20edfee700 -1 osd.5 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:33.315664+0000 front 2024-04-14T15:29:33.315504+0000 (oldest deadline 2024-04-14T15:29:58.015582+0000) 2024-04-14T15:30:02.623 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:02 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:02.254+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:34.221673+0000 front 2024-04-14T15:29:34.221775+0000 (oldest deadline 2024-04-14T15:29:55.921555+0000) 2024-04-14T15:30:02.623 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:02 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6[52258]: 2024-04-14T15:30:02.405+0000 7fb5db8ea700 -1 osd.6 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:31.616672+0000 front 2024-04-14T15:29:31.616871+0000 (oldest deadline 2024-04-14T15:29:56.316800+0000) 2024-04-14T15:30:02.702 DEBUG:teuthology.orchestra.run.smithi090:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0e288098-fa70-11ee-bc8e-c7b262605968@osd.3.service' 2024-04-14T15:30:02.743 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-14T15:30:02.743 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-14T15:30:02.744 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-14T15:30:02.744 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ceph-0e288098-fa70-11ee-bc8e-c7b262605968@osd.4 2024-04-14T15:30:03.032 INFO:journalctl@ceph.osd.4.smithi092.stdout:Apr 14 15:30:02 smithi092 systemd[1]: Stopping Ceph osd.4 for 0e288098-fa70-11ee-bc8e-c7b262605968... 2024-04-14T15:30:03.032 INFO:journalctl@ceph.osd.4.smithi092.stdout:Apr 14 15:30:02 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-4[39413]: 2024-04-14T15:30:02.849+0000 7f7456ddd700 -1 osd.4 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:35.290878+0000 front 2024-04-14T15:29:35.290925+0000 (oldest deadline 2024-04-14T15:30:01.190799+0000) 2024-04-14T15:30:03.372 INFO:journalctl@ceph.osd.5.smithi092.stdout:Apr 14 15:30:03 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-5[45851]: 2024-04-14T15:30:03.188+0000 7f20edfee700 -1 osd.5 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:33.315664+0000 front 2024-04-14T15:29:33.315504+0000 (oldest deadline 2024-04-14T15:29:58.015582+0000) 2024-04-14T15:30:03.372 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:03 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:03.274+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:34.221673+0000 front 2024-04-14T15:29:34.221775+0000 (oldest deadline 2024-04-14T15:29:55.921555+0000) 2024-04-14T15:30:03.372 INFO:journalctl@ceph.osd.4.smithi092.stdout:Apr 14 15:30:03 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-4[39413]: 2024-04-14T15:30:03.031+0000 7f745a7e9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T15:30:03.373 INFO:journalctl@ceph.osd.4.smithi092.stdout:Apr 14 15:30:03 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-4[39413]: 2024-04-14T15:30:03.031+0000 7f745a7e9700 -1 osd.4 926 *** Got signal Terminated *** 2024-04-14T15:30:03.373 INFO:journalctl@ceph.osd.4.smithi092.stdout:Apr 14 15:30:03 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-4[39413]: 2024-04-14T15:30:03.031+0000 7f745a7e9700 -1 osd.4 926 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-14T15:30:03.846 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:03 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6[52258]: 2024-04-14T15:30:03.397+0000 7fb5db8ea700 -1 osd.6 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:31.616672+0000 front 2024-04-14T15:29:31.616871+0000 (oldest deadline 2024-04-14T15:29:56.316800+0000) 2024-04-14T15:30:03.941 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:30:03 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-alertmanager-a[84115]: ts=2024-04-14T15:30:03.571Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi090.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.90:8443: connect: connection refused" 2024-04-14T15:30:03.942 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:30:03 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-alertmanager-a[84115]: ts=2024-04-14T15:30:03.571Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi092.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.92:8443: connect: connection refused" 2024-04-14T15:30:04.122 INFO:journalctl@ceph.osd.4.smithi092.stdout:Apr 14 15:30:03 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-4[39413]: 2024-04-14T15:30:03.846+0000 7f7456ddd700 -1 osd.4 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:35.290878+0000 front 2024-04-14T15:29:35.290925+0000 (oldest deadline 2024-04-14T15:30:01.190799+0000) 2024-04-14T15:30:04.622 INFO:journalctl@ceph.osd.5.smithi092.stdout:Apr 14 15:30:04 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-5[45851]: 2024-04-14T15:30:04.171+0000 7f20edfee700 -1 osd.5 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:33.315664+0000 front 2024-04-14T15:29:33.315504+0000 (oldest deadline 2024-04-14T15:29:58.015582+0000) 2024-04-14T15:30:04.622 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:04 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:04.297+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:34.221673+0000 front 2024-04-14T15:29:34.221775+0000 (oldest deadline 2024-04-14T15:29:55.921555+0000) 2024-04-14T15:30:04.622 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:04 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6[52258]: 2024-04-14T15:30:04.420+0000 7fb5db8ea700 -1 osd.6 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:31.616672+0000 front 2024-04-14T15:29:31.616871+0000 (oldest deadline 2024-04-14T15:29:56.316800+0000) 2024-04-14T15:30:05.155 INFO:journalctl@ceph.osd.4.smithi092.stdout:Apr 14 15:30:04 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-4[39413]: 2024-04-14T15:30:04.879+0000 7f7456ddd700 -1 osd.4 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:35.290878+0000 front 2024-04-14T15:29:35.290925+0000 (oldest deadline 2024-04-14T15:30:01.190799+0000) 2024-04-14T15:30:05.462 INFO:journalctl@ceph.osd.5.smithi092.stdout:Apr 14 15:30:05 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-5[45851]: 2024-04-14T15:30:05.154+0000 7f20edfee700 -1 osd.5 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:33.315664+0000 front 2024-04-14T15:29:33.315504+0000 (oldest deadline 2024-04-14T15:29:58.015582+0000) 2024-04-14T15:30:05.463 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:05 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:05.281+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:34.221673+0000 front 2024-04-14T15:29:34.221775+0000 (oldest deadline 2024-04-14T15:29:55.921555+0000) 2024-04-14T15:30:05.831 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:05 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6[52258]: 2024-04-14T15:30:05.462+0000 7fb5db8ea700 -1 osd.6 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:31.616672+0000 front 2024-04-14T15:29:31.616871+0000 (oldest deadline 2024-04-14T15:29:56.316800+0000) 2024-04-14T15:30:06.122 INFO:journalctl@ceph.osd.4.smithi092.stdout:Apr 14 15:30:05 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-4[39413]: 2024-04-14T15:30:05.829+0000 7f7456ddd700 -1 osd.4 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:35.290878+0000 front 2024-04-14T15:29:35.290925+0000 (oldest deadline 2024-04-14T15:30:01.190799+0000) 2024-04-14T15:30:06.511 INFO:journalctl@ceph.osd.5.smithi092.stdout:Apr 14 15:30:06 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-5[45851]: 2024-04-14T15:30:06.153+0000 7f20edfee700 -1 osd.5 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:33.315664+0000 front 2024-04-14T15:29:33.315504+0000 (oldest deadline 2024-04-14T15:29:58.015582+0000) 2024-04-14T15:30:06.511 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:06 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:06.298+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:34.221673+0000 front 2024-04-14T15:29:34.221775+0000 (oldest deadline 2024-04-14T15:29:55.921555+0000) 2024-04-14T15:30:06.791 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:06 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6[52258]: 2024-04-14T15:30:06.511+0000 7fb5db8ea700 -1 osd.6 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:31.616672+0000 front 2024-04-14T15:29:31.616871+0000 (oldest deadline 2024-04-14T15:29:56.316800+0000) 2024-04-14T15:30:07.122 INFO:journalctl@ceph.osd.4.smithi092.stdout:Apr 14 15:30:06 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-4[39413]: 2024-04-14T15:30:06.790+0000 7f7456ddd700 -1 osd.4 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:35.290878+0000 front 2024-04-14T15:29:35.290925+0000 (oldest deadline 2024-04-14T15:30:01.190799+0000) 2024-04-14T15:30:07.503 INFO:journalctl@ceph.osd.5.smithi092.stdout:Apr 14 15:30:07 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-5[45851]: 2024-04-14T15:30:07.131+0000 7f20edfee700 -1 osd.5 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:33.315664+0000 front 2024-04-14T15:29:33.315504+0000 (oldest deadline 2024-04-14T15:29:58.015582+0000) 2024-04-14T15:30:07.503 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:07 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:07.290+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:34.221673+0000 front 2024-04-14T15:29:34.221775+0000 (oldest deadline 2024-04-14T15:29:55.921555+0000) 2024-04-14T15:30:07.504 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:07 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:07.290+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6807 osd.1 since back 2024-04-14T15:29:42.923224+0000 front 2024-04-14T15:29:42.923341+0000 (oldest deadline 2024-04-14T15:30:06.423218+0000) 2024-04-14T15:30:07.798 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:07 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6[52258]: 2024-04-14T15:30:07.503+0000 7fb5db8ea700 -1 osd.6 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:31.616672+0000 front 2024-04-14T15:29:31.616871+0000 (oldest deadline 2024-04-14T15:29:56.316800+0000) 2024-04-14T15:30:07.798 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:07 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6[52258]: 2024-04-14T15:30:07.503+0000 7fb5db8ea700 -1 osd.6 926 heartbeat_check: no reply from 172.21.15.90:6807 osd.1 since back 2024-04-14T15:29:41.518416+0000 front 2024-04-14T15:29:41.518436+0000 (oldest deadline 2024-04-14T15:30:07.418462+0000) 2024-04-14T15:30:08.122 INFO:journalctl@ceph.osd.4.smithi092.stdout:Apr 14 15:30:07 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-4[39413]: 2024-04-14T15:30:07.797+0000 7f7456ddd700 -1 osd.4 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:35.290878+0000 front 2024-04-14T15:29:35.290925+0000 (oldest deadline 2024-04-14T15:30:01.190799+0000) 2024-04-14T15:30:08.122 INFO:journalctl@ceph.osd.4.smithi092.stdout:Apr 14 15:30:07 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-4[39413]: 2024-04-14T15:30:07.797+0000 7f7456ddd700 -1 osd.4 926 heartbeat_check: no reply from 172.21.15.90:6807 osd.1 since back 2024-04-14T15:29:42.291778+0000 front 2024-04-14T15:29:42.291844+0000 (oldest deadline 2024-04-14T15:30:06.991756+0000) 2024-04-14T15:30:08.439 INFO:journalctl@ceph.osd.5.smithi092.stdout:Apr 14 15:30:08 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-5[45851]: 2024-04-14T15:30:08.163+0000 7f20edfee700 -1 osd.5 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:33.315664+0000 front 2024-04-14T15:29:33.315504+0000 (oldest deadline 2024-04-14T15:29:58.015582+0000) 2024-04-14T15:30:08.439 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:08 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:08.307+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:34.221673+0000 front 2024-04-14T15:29:34.221775+0000 (oldest deadline 2024-04-14T15:29:55.921555+0000) 2024-04-14T15:30:08.440 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:08 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:08.307+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6807 osd.1 since back 2024-04-14T15:29:42.923224+0000 front 2024-04-14T15:29:42.923341+0000 (oldest deadline 2024-04-14T15:30:06.423218+0000) 2024-04-14T15:30:08.761 INFO:journalctl@ceph.osd.4.smithi092.stdout:Apr 14 15:30:08 smithi092 podman[152129]: 2024-04-14 15:30:08.439762503 +0000 UTC m=+5.504659574 container died cb3f8419f15b858f338542170ffb56ed75ccb7d99382a0699ef892e12602f879 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-4, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=main-a971fd5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True) 2024-04-14T15:30:08.761 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:08 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6[52258]: 2024-04-14T15:30:08.492+0000 7fb5db8ea700 -1 osd.6 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:31.616672+0000 front 2024-04-14T15:29:31.616871+0000 (oldest deadline 2024-04-14T15:29:56.316800+0000) 2024-04-14T15:30:08.761 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:08 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6[52258]: 2024-04-14T15:30:08.492+0000 7fb5db8ea700 -1 osd.6 926 heartbeat_check: no reply from 172.21.15.90:6807 osd.1 since back 2024-04-14T15:29:41.518416+0000 front 2024-04-14T15:29:41.518436+0000 (oldest deadline 2024-04-14T15:30:07.418462+0000) 2024-04-14T15:30:09.122 INFO:journalctl@ceph.osd.4.smithi092.stdout:Apr 14 15:30:08 smithi092 podman[152129]: 2024-04-14 15:30:08.761865507 +0000 UTC m=+5.826762575 container cleanup cb3f8419f15b858f338542170ffb56ed75ccb7d99382a0699ef892e12602f879 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-4, RELEASE=main-a971fd5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-14T15:30:09.122 INFO:journalctl@ceph.osd.4.smithi092.stdout:Apr 14 15:30:08 smithi092 bash[152129]: ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-4 2024-04-14T15:30:09.122 INFO:journalctl@ceph.osd.4.smithi092.stdout:Apr 14 15:30:08 smithi092 podman[152151]: 2024-04-14 15:30:08.971403266 +0000 UTC m=+0.531226093 container remove cb3f8419f15b858f338542170ffb56ed75ccb7d99382a0699ef892e12602f879 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-4, RELEASE=main-a971fd5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-04-14T15:30:09.461 INFO:journalctl@ceph.osd.5.smithi092.stdout:Apr 14 15:30:09 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-5[45851]: 2024-04-14T15:30:09.191+0000 7f20edfee700 -1 osd.5 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:33.315664+0000 front 2024-04-14T15:29:33.315504+0000 (oldest deadline 2024-04-14T15:29:58.015582+0000) 2024-04-14T15:30:09.461 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:09 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:09.352+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:34.221673+0000 front 2024-04-14T15:29:34.221775+0000 (oldest deadline 2024-04-14T15:29:55.921555+0000) 2024-04-14T15:30:09.461 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:09 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:09.352+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6807 osd.1 since back 2024-04-14T15:29:42.923224+0000 front 2024-04-14T15:29:42.923341+0000 (oldest deadline 2024-04-14T15:30:06.423218+0000) 2024-04-14T15:30:09.462 INFO:journalctl@ceph.osd.4.smithi092.stdout:Apr 14 15:30:09 smithi092 podman[152248]: 2024-04-14 15:30:09.275409035 +0000 UTC m=+0.018057151 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608 2024-04-14T15:30:09.462 INFO:journalctl@ceph.osd.4.smithi092.stdout:Apr 14 15:30:09 smithi092 podman[152248]: 2024-04-14 15:30:09.38886322 +0000 UTC m=+0.131511326 container create d6542e5d8376002adbb4d4173b9292d91575937f6c35412f284cfd62f8ee0e59 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=main-a971fd5, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403) 2024-04-14T15:30:09.872 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:09 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6[52258]: 2024-04-14T15:30:09.461+0000 7fb5db8ea700 -1 osd.6 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:31.616672+0000 front 2024-04-14T15:29:31.616871+0000 (oldest deadline 2024-04-14T15:29:56.316800+0000) 2024-04-14T15:30:09.873 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:09 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6[52258]: 2024-04-14T15:30:09.461+0000 7fb5db8ea700 -1 osd.6 926 heartbeat_check: no reply from 172.21.15.90:6807 osd.1 since back 2024-04-14T15:29:41.518416+0000 front 2024-04-14T15:29:41.518436+0000 (oldest deadline 2024-04-14T15:30:07.418462+0000) 2024-04-14T15:30:09.873 INFO:journalctl@ceph.osd.4.smithi092.stdout:Apr 14 15:30:09 smithi092 podman[152248]: 2024-04-14 15:30:09.608219452 +0000 UTC m=+0.350867571 container init d6542e5d8376002adbb4d4173b9292d91575937f6c35412f284cfd62f8ee0e59 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, RELEASE=main-a971fd5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403) 2024-04-14T15:30:09.873 INFO:journalctl@ceph.osd.4.smithi092.stdout:Apr 14 15:30:09 smithi092 podman[152248]: 2024-04-14 15:30:09.613693453 +0000 UTC m=+0.356341565 container start d6542e5d8376002adbb4d4173b9292d91575937f6c35412f284cfd62f8ee0e59 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-4-deactivate, RELEASE=main-a971fd5, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-14T15:30:09.873 INFO:journalctl@ceph.osd.4.smithi092.stdout:Apr 14 15:30:09 smithi092 podman[152248]: 2024-04-14 15:30:09.69949888 +0000 UTC m=+0.442147000 container attach d6542e5d8376002adbb4d4173b9292d91575937f6c35412f284cfd62f8ee0e59 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-4-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=main-a971fd5, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240403) 2024-04-14T15:30:10.193 INFO:journalctl@ceph.osd.4.smithi092.stdout:Apr 14 15:30:09 smithi092 podman[152248]: 2024-04-14 15:30:09.909224468 +0000 UTC m=+0.651872578 container died d6542e5d8376002adbb4d4173b9292d91575937f6c35412f284cfd62f8ee0e59 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=main-a971fd5, org.label-schema.build-date=20240403, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-14T15:30:10.485 INFO:journalctl@ceph.osd.5.smithi092.stdout:Apr 14 15:30:10 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-5[45851]: 2024-04-14T15:30:10.206+0000 7f20edfee700 -1 osd.5 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:33.315664+0000 front 2024-04-14T15:29:33.315504+0000 (oldest deadline 2024-04-14T15:29:58.015582+0000) 2024-04-14T15:30:10.485 INFO:journalctl@ceph.osd.5.smithi092.stdout:Apr 14 15:30:10 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-5[45851]: 2024-04-14T15:30:10.206+0000 7f20edfee700 -1 osd.5 926 heartbeat_check: no reply from 172.21.15.90:6807 osd.1 since back 2024-04-14T15:29:43.817351+0000 front 2024-04-14T15:29:43.817339+0000 (oldest deadline 2024-04-14T15:30:09.717259+0000) 2024-04-14T15:30:10.485 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:10 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6[52258]: 2024-04-14T15:30:10.411+0000 7fb5db8ea700 -1 osd.6 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:31.616672+0000 front 2024-04-14T15:29:31.616871+0000 (oldest deadline 2024-04-14T15:29:56.316800+0000) 2024-04-14T15:30:10.486 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:10 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6[52258]: 2024-04-14T15:30:10.411+0000 7fb5db8ea700 -1 osd.6 926 heartbeat_check: no reply from 172.21.15.90:6807 osd.1 since back 2024-04-14T15:29:41.518416+0000 front 2024-04-14T15:29:41.518436+0000 (oldest deadline 2024-04-14T15:30:07.418462+0000) 2024-04-14T15:30:10.486 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:10 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:10.384+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:34.221673+0000 front 2024-04-14T15:29:34.221775+0000 (oldest deadline 2024-04-14T15:29:55.921555+0000) 2024-04-14T15:30:10.486 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:10 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:10.384+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6807 osd.1 since back 2024-04-14T15:29:42.923224+0000 front 2024-04-14T15:29:42.923341+0000 (oldest deadline 2024-04-14T15:30:06.423218+0000) 2024-04-14T15:30:10.503 DEBUG:teuthology.orchestra.run.smithi092:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0e288098-fa70-11ee-bc8e-c7b262605968@osd.4.service' 2024-04-14T15:30:10.545 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-14T15:30:10.545 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-14T15:30:10.546 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-14T15:30:10.546 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ceph-0e288098-fa70-11ee-bc8e-c7b262605968@osd.5 2024-04-14T15:30:10.866 INFO:journalctl@ceph.osd.5.smithi092.stdout:Apr 14 15:30:10 smithi092 systemd[1]: Stopping Ceph osd.5 for 0e288098-fa70-11ee-bc8e-c7b262605968... 2024-04-14T15:30:10.866 INFO:journalctl@ceph.osd.5.smithi092.stdout:Apr 14 15:30:10 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-5[45851]: 2024-04-14T15:30:10.865+0000 7f20f5a02700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T15:30:10.866 INFO:journalctl@ceph.osd.5.smithi092.stdout:Apr 14 15:30:10 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-5[45851]: 2024-04-14T15:30:10.865+0000 7f20f5a02700 -1 osd.5 926 *** Got signal Terminated *** 2024-04-14T15:30:11.122 INFO:journalctl@ceph.osd.5.smithi092.stdout:Apr 14 15:30:10 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-5[45851]: 2024-04-14T15:30:10.865+0000 7f20f5a02700 -1 osd.5 926 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-14T15:30:11.450 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:11 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:11.379+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:34.221673+0000 front 2024-04-14T15:29:34.221775+0000 (oldest deadline 2024-04-14T15:29:55.921555+0000) 2024-04-14T15:30:11.450 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:11 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:11.379+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6807 osd.1 since back 2024-04-14T15:29:42.923224+0000 front 2024-04-14T15:29:42.923341+0000 (oldest deadline 2024-04-14T15:30:06.423218+0000) 2024-04-14T15:30:11.451 INFO:journalctl@ceph.osd.5.smithi092.stdout:Apr 14 15:30:11 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-5[45851]: 2024-04-14T15:30:11.165+0000 7f20edfee700 -1 osd.5 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:33.315664+0000 front 2024-04-14T15:29:33.315504+0000 (oldest deadline 2024-04-14T15:29:58.015582+0000) 2024-04-14T15:30:11.451 INFO:journalctl@ceph.osd.5.smithi092.stdout:Apr 14 15:30:11 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-5[45851]: 2024-04-14T15:30:11.165+0000 7f20edfee700 -1 osd.5 926 heartbeat_check: no reply from 172.21.15.90:6807 osd.1 since back 2024-04-14T15:29:43.817351+0000 front 2024-04-14T15:29:43.817339+0000 (oldest deadline 2024-04-14T15:30:09.717259+0000) 2024-04-14T15:30:11.872 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:11 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6[52258]: 2024-04-14T15:30:11.450+0000 7fb5db8ea700 -1 osd.6 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:31.616672+0000 front 2024-04-14T15:29:31.616871+0000 (oldest deadline 2024-04-14T15:29:56.316800+0000) 2024-04-14T15:30:11.872 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:11 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6[52258]: 2024-04-14T15:30:11.450+0000 7fb5db8ea700 -1 osd.6 926 heartbeat_check: no reply from 172.21.15.90:6807 osd.1 since back 2024-04-14T15:29:41.518416+0000 front 2024-04-14T15:29:41.518436+0000 (oldest deadline 2024-04-14T15:30:07.418462+0000) 2024-04-14T15:30:12.407 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:12 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:12.369+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:34.221673+0000 front 2024-04-14T15:29:34.221775+0000 (oldest deadline 2024-04-14T15:29:55.921555+0000) 2024-04-14T15:30:12.407 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:12 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:12.369+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6807 osd.1 since back 2024-04-14T15:29:42.923224+0000 front 2024-04-14T15:29:42.923341+0000 (oldest deadline 2024-04-14T15:30:06.423218+0000) 2024-04-14T15:30:12.407 INFO:journalctl@ceph.osd.5.smithi092.stdout:Apr 14 15:30:12 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-5[45851]: 2024-04-14T15:30:12.142+0000 7f20edfee700 -1 osd.5 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:33.315664+0000 front 2024-04-14T15:29:33.315504+0000 (oldest deadline 2024-04-14T15:29:58.015582+0000) 2024-04-14T15:30:12.407 INFO:journalctl@ceph.osd.5.smithi092.stdout:Apr 14 15:30:12 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-5[45851]: 2024-04-14T15:30:12.142+0000 7f20edfee700 -1 osd.5 926 heartbeat_check: no reply from 172.21.15.90:6807 osd.1 since back 2024-04-14T15:29:43.817351+0000 front 2024-04-14T15:29:43.817339+0000 (oldest deadline 2024-04-14T15:30:09.717259+0000) 2024-04-14T15:30:12.872 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:12 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6[52258]: 2024-04-14T15:30:12.406+0000 7fb5db8ea700 -1 osd.6 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:31.616672+0000 front 2024-04-14T15:29:31.616871+0000 (oldest deadline 2024-04-14T15:29:56.316800+0000) 2024-04-14T15:30:12.872 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:12 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6[52258]: 2024-04-14T15:30:12.406+0000 7fb5db8ea700 -1 osd.6 926 heartbeat_check: no reply from 172.21.15.90:6807 osd.1 since back 2024-04-14T15:29:41.518416+0000 front 2024-04-14T15:29:41.518436+0000 (oldest deadline 2024-04-14T15:30:07.418462+0000) 2024-04-14T15:30:13.622 INFO:journalctl@ceph.osd.5.smithi092.stdout:Apr 14 15:30:13 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-5[45851]: 2024-04-14T15:30:13.173+0000 7f20edfee700 -1 osd.5 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:33.315664+0000 front 2024-04-14T15:29:33.315504+0000 (oldest deadline 2024-04-14T15:29:58.015582+0000) 2024-04-14T15:30:13.622 INFO:journalctl@ceph.osd.5.smithi092.stdout:Apr 14 15:30:13 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-5[45851]: 2024-04-14T15:30:13.173+0000 7f20edfee700 -1 osd.5 926 heartbeat_check: no reply from 172.21.15.90:6807 osd.1 since back 2024-04-14T15:29:43.817351+0000 front 2024-04-14T15:29:43.817339+0000 (oldest deadline 2024-04-14T15:30:09.717259+0000) 2024-04-14T15:30:13.623 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:13 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:13.393+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:34.221673+0000 front 2024-04-14T15:29:34.221775+0000 (oldest deadline 2024-04-14T15:29:55.921555+0000) 2024-04-14T15:30:13.623 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:13 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:13.393+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6807 osd.1 since back 2024-04-14T15:29:42.923224+0000 front 2024-04-14T15:29:42.923341+0000 (oldest deadline 2024-04-14T15:30:06.423218+0000) 2024-04-14T15:30:13.623 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:13 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6[52258]: 2024-04-14T15:30:13.403+0000 7fb5db8ea700 -1 osd.6 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:31.616672+0000 front 2024-04-14T15:29:31.616871+0000 (oldest deadline 2024-04-14T15:29:56.316800+0000) 2024-04-14T15:30:13.623 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:13 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6[52258]: 2024-04-14T15:30:13.403+0000 7fb5db8ea700 -1 osd.6 926 heartbeat_check: no reply from 172.21.15.90:6807 osd.1 since back 2024-04-14T15:29:41.518416+0000 front 2024-04-14T15:29:41.518436+0000 (oldest deadline 2024-04-14T15:30:07.418462+0000) 2024-04-14T15:30:13.623 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:13 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6[52258]: 2024-04-14T15:30:13.403+0000 7fb5db8ea700 -1 osd.6 926 heartbeat_check: no reply from 172.21.15.90:6811 osd.2 since back 2024-04-14T15:29:50.819830+0000 front 2024-04-14T15:29:50.819846+0000 (oldest deadline 2024-04-14T15:30:12.519819+0000) 2024-04-14T15:30:13.852 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:30:13 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-alertmanager-a[84115]: ts=2024-04-14T15:30:13.570Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi092.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.92:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi090.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.90:8443: connect: connection refused" 2024-04-14T15:30:13.853 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:30:13 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-alertmanager-a[84115]: ts=2024-04-14T15:30:13.571Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi090.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.90:8443: connect: connection refused" 2024-04-14T15:30:13.853 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:30:13 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-alertmanager-a[84115]: ts=2024-04-14T15:30:13.571Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi092.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.92:8443: connect: connection refused" 2024-04-14T15:30:14.623 INFO:journalctl@ceph.osd.5.smithi092.stdout:Apr 14 15:30:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-5[45851]: 2024-04-14T15:30:14.184+0000 7f20edfee700 -1 osd.5 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:33.315664+0000 front 2024-04-14T15:29:33.315504+0000 (oldest deadline 2024-04-14T15:29:58.015582+0000) 2024-04-14T15:30:14.624 INFO:journalctl@ceph.osd.5.smithi092.stdout:Apr 14 15:30:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-5[45851]: 2024-04-14T15:30:14.184+0000 7f20edfee700 -1 osd.5 926 heartbeat_check: no reply from 172.21.15.90:6807 osd.1 since back 2024-04-14T15:29:43.817351+0000 front 2024-04-14T15:29:43.817339+0000 (oldest deadline 2024-04-14T15:30:09.717259+0000) 2024-04-14T15:30:14.624 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:14.378+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:34.221673+0000 front 2024-04-14T15:29:34.221775+0000 (oldest deadline 2024-04-14T15:29:55.921555+0000) 2024-04-14T15:30:14.624 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:14.378+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6807 osd.1 since back 2024-04-14T15:29:42.923224+0000 front 2024-04-14T15:29:42.923341+0000 (oldest deadline 2024-04-14T15:30:06.423218+0000) 2024-04-14T15:30:14.624 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6[52258]: 2024-04-14T15:30:14.425+0000 7fb5db8ea700 -1 osd.6 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:31.616672+0000 front 2024-04-14T15:29:31.616871+0000 (oldest deadline 2024-04-14T15:29:56.316800+0000) 2024-04-14T15:30:14.624 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6[52258]: 2024-04-14T15:30:14.425+0000 7fb5db8ea700 -1 osd.6 926 heartbeat_check: no reply from 172.21.15.90:6807 osd.1 since back 2024-04-14T15:29:41.518416+0000 front 2024-04-14T15:29:41.518436+0000 (oldest deadline 2024-04-14T15:30:07.418462+0000) 2024-04-14T15:30:14.624 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:14 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6[52258]: 2024-04-14T15:30:14.425+0000 7fb5db8ea700 -1 osd.6 926 heartbeat_check: no reply from 172.21.15.90:6811 osd.2 since back 2024-04-14T15:29:50.819830+0000 front 2024-04-14T15:29:50.819846+0000 (oldest deadline 2024-04-14T15:30:12.519819+0000) 2024-04-14T15:30:15.467 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:15.375+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:34.221673+0000 front 2024-04-14T15:29:34.221775+0000 (oldest deadline 2024-04-14T15:29:55.921555+0000) 2024-04-14T15:30:15.467 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:15.375+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6807 osd.1 since back 2024-04-14T15:29:42.923224+0000 front 2024-04-14T15:29:42.923341+0000 (oldest deadline 2024-04-14T15:30:06.423218+0000) 2024-04-14T15:30:15.467 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:15.375+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6811 osd.2 since back 2024-04-14T15:29:49.924376+0000 front 2024-04-14T15:29:49.924338+0000 (oldest deadline 2024-04-14T15:30:14.624363+0000) 2024-04-14T15:30:15.467 INFO:journalctl@ceph.osd.5.smithi092.stdout:Apr 14 15:30:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-5[45851]: 2024-04-14T15:30:15.167+0000 7f20edfee700 -1 osd.5 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:33.315664+0000 front 2024-04-14T15:29:33.315504+0000 (oldest deadline 2024-04-14T15:29:58.015582+0000) 2024-04-14T15:30:15.468 INFO:journalctl@ceph.osd.5.smithi092.stdout:Apr 14 15:30:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-5[45851]: 2024-04-14T15:30:15.167+0000 7f20edfee700 -1 osd.5 926 heartbeat_check: no reply from 172.21.15.90:6807 osd.1 since back 2024-04-14T15:29:43.817351+0000 front 2024-04-14T15:29:43.817339+0000 (oldest deadline 2024-04-14T15:30:09.717259+0000) 2024-04-14T15:30:15.872 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6[52258]: 2024-04-14T15:30:15.466+0000 7fb5db8ea700 -1 osd.6 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:31.616672+0000 front 2024-04-14T15:29:31.616871+0000 (oldest deadline 2024-04-14T15:29:56.316800+0000) 2024-04-14T15:30:15.872 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6[52258]: 2024-04-14T15:30:15.466+0000 7fb5db8ea700 -1 osd.6 926 heartbeat_check: no reply from 172.21.15.90:6807 osd.1 since back 2024-04-14T15:29:41.518416+0000 front 2024-04-14T15:29:41.518436+0000 (oldest deadline 2024-04-14T15:30:07.418462+0000) 2024-04-14T15:30:15.872 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:15 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6[52258]: 2024-04-14T15:30:15.466+0000 7fb5db8ea700 -1 osd.6 926 heartbeat_check: no reply from 172.21.15.90:6811 osd.2 since back 2024-04-14T15:29:50.819830+0000 front 2024-04-14T15:29:50.819846+0000 (oldest deadline 2024-04-14T15:30:12.519819+0000) 2024-04-14T15:30:16.622 INFO:journalctl@ceph.osd.5.smithi092.stdout:Apr 14 15:30:16 smithi092 podman[152378]: 2024-04-14 15:30:16.357782235 +0000 UTC m=+5.598572030 container died e4bb180ef4049b83bafefce6e3d1662577cfbfb0aeb1d1ffff2c87aa67c2bd51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-5, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=main-a971fd5, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-14T15:30:16.622 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:16 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6[52258]: 2024-04-14T15:30:16.514+0000 7fb5db8ea700 -1 osd.6 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:31.616672+0000 front 2024-04-14T15:29:31.616871+0000 (oldest deadline 2024-04-14T15:29:56.316800+0000) 2024-04-14T15:30:16.623 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:16 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6[52258]: 2024-04-14T15:30:16.514+0000 7fb5db8ea700 -1 osd.6 926 heartbeat_check: no reply from 172.21.15.90:6807 osd.1 since back 2024-04-14T15:29:41.518416+0000 front 2024-04-14T15:29:41.518436+0000 (oldest deadline 2024-04-14T15:30:07.418462+0000) 2024-04-14T15:30:16.623 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:16 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6[52258]: 2024-04-14T15:30:16.514+0000 7fb5db8ea700 -1 osd.6 926 heartbeat_check: no reply from 172.21.15.90:6811 osd.2 since back 2024-04-14T15:29:50.819830+0000 front 2024-04-14T15:29:50.819846+0000 (oldest deadline 2024-04-14T15:30:12.519819+0000) 2024-04-14T15:30:16.623 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:16 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:16.373+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:34.221673+0000 front 2024-04-14T15:29:34.221775+0000 (oldest deadline 2024-04-14T15:29:55.921555+0000) 2024-04-14T15:30:16.623 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:16 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:16.373+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6807 osd.1 since back 2024-04-14T15:29:42.923224+0000 front 2024-04-14T15:29:42.923341+0000 (oldest deadline 2024-04-14T15:30:06.423218+0000) 2024-04-14T15:30:16.623 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:16 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:16.373+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6811 osd.2 since back 2024-04-14T15:29:49.924376+0000 front 2024-04-14T15:29:49.924338+0000 (oldest deadline 2024-04-14T15:30:14.624363+0000) 2024-04-14T15:30:17.005 INFO:journalctl@ceph.osd.5.smithi092.stdout:Apr 14 15:30:16 smithi092 podman[152378]: 2024-04-14 15:30:16.754445615 +0000 UTC m=+5.995235411 container cleanup e4bb180ef4049b83bafefce6e3d1662577cfbfb0aeb1d1ffff2c87aa67c2bd51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-5, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-a971fd5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-14T15:30:17.005 INFO:journalctl@ceph.osd.5.smithi092.stdout:Apr 14 15:30:16 smithi092 bash[152378]: ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-5 2024-04-14T15:30:17.317 INFO:journalctl@ceph.osd.5.smithi092.stdout:Apr 14 15:30:17 smithi092 podman[152401]: 2024-04-14 15:30:17.005879776 +0000 UTC m=+0.649865653 container remove e4bb180ef4049b83bafefce6e3d1662577cfbfb0aeb1d1ffff2c87aa67c2bd51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-5, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-a971fd5, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-14T15:30:17.599 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:17 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:17.421+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:34.221673+0000 front 2024-04-14T15:29:34.221775+0000 (oldest deadline 2024-04-14T15:29:55.921555+0000) 2024-04-14T15:30:17.600 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:17 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:17.421+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6807 osd.1 since back 2024-04-14T15:29:42.923224+0000 front 2024-04-14T15:29:42.923341+0000 (oldest deadline 2024-04-14T15:30:06.423218+0000) 2024-04-14T15:30:17.600 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:17 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:17.421+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6811 osd.2 since back 2024-04-14T15:29:49.924376+0000 front 2024-04-14T15:29:49.924338+0000 (oldest deadline 2024-04-14T15:30:14.624363+0000) 2024-04-14T15:30:17.600 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:17 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6[52258]: 2024-04-14T15:30:17.501+0000 7fb5db8ea700 -1 osd.6 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:31.616672+0000 front 2024-04-14T15:29:31.616871+0000 (oldest deadline 2024-04-14T15:29:56.316800+0000) 2024-04-14T15:30:17.600 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:17 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6[52258]: 2024-04-14T15:30:17.501+0000 7fb5db8ea700 -1 osd.6 926 heartbeat_check: no reply from 172.21.15.90:6807 osd.1 since back 2024-04-14T15:29:41.518416+0000 front 2024-04-14T15:29:41.518436+0000 (oldest deadline 2024-04-14T15:30:07.418462+0000) 2024-04-14T15:30:17.600 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:17 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6[52258]: 2024-04-14T15:30:17.501+0000 7fb5db8ea700 -1 osd.6 926 heartbeat_check: no reply from 172.21.15.90:6811 osd.2 since back 2024-04-14T15:29:50.819830+0000 front 2024-04-14T15:29:50.819846+0000 (oldest deadline 2024-04-14T15:30:12.519819+0000) 2024-04-14T15:30:17.600 INFO:journalctl@ceph.osd.5.smithi092.stdout:Apr 14 15:30:17 smithi092 podman[152501]: 2024-04-14 15:30:17.221407514 +0000 UTC m=+0.018122857 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608 2024-04-14T15:30:17.601 INFO:journalctl@ceph.osd.5.smithi092.stdout:Apr 14 15:30:17 smithi092 podman[152501]: 2024-04-14 15:30:17.323345392 +0000 UTC m=+0.120060749 container create f8fb3e9419a333bc5ca22e0c7ae3a6f6c779b537880de14b946074ba2e6d3daf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=main-a971fd5, org.label-schema.build-date=20240403, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-14T15:30:17.601 INFO:journalctl@ceph.osd.5.smithi092.stdout:Apr 14 15:30:17 smithi092 podman[152501]: 2024-04-14 15:30:17.51729354 +0000 UTC m=+0.314008856 container init f8fb3e9419a333bc5ca22e0c7ae3a6f6c779b537880de14b946074ba2e6d3daf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, RELEASE=main-a971fd5) 2024-04-14T15:30:17.601 INFO:journalctl@ceph.osd.5.smithi092.stdout:Apr 14 15:30:17 smithi092 podman[152501]: 2024-04-14 15:30:17.522666668 +0000 UTC m=+0.319381986 container start f8fb3e9419a333bc5ca22e0c7ae3a6f6c779b537880de14b946074ba2e6d3daf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=main-a971fd5) 2024-04-14T15:30:17.601 INFO:journalctl@ceph.osd.5.smithi092.stdout:Apr 14 15:30:17 smithi092 podman[152501]: 2024-04-14 15:30:17.600346131 +0000 UTC m=+0.397061446 container attach f8fb3e9419a333bc5ca22e0c7ae3a6f6c779b537880de14b946074ba2e6d3daf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-a971fd5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-14T15:30:17.872 INFO:journalctl@ceph.osd.5.smithi092.stdout:Apr 14 15:30:17 smithi092 podman[152501]: 2024-04-14 15:30:17.811177305 +0000 UTC m=+0.607892613 container died f8fb3e9419a333bc5ca22e0c7ae3a6f6c779b537880de14b946074ba2e6d3daf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=main-a971fd5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-14T15:30:18.411 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:18 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:18.402+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:34.221673+0000 front 2024-04-14T15:29:34.221775+0000 (oldest deadline 2024-04-14T15:29:55.921555+0000) 2024-04-14T15:30:18.411 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:18 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:18.402+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6807 osd.1 since back 2024-04-14T15:29:42.923224+0000 front 2024-04-14T15:29:42.923341+0000 (oldest deadline 2024-04-14T15:30:06.423218+0000) 2024-04-14T15:30:18.411 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:18 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:18.402+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6811 osd.2 since back 2024-04-14T15:29:49.924376+0000 front 2024-04-14T15:29:49.924338+0000 (oldest deadline 2024-04-14T15:30:14.624363+0000) 2024-04-14T15:30:18.431 DEBUG:teuthology.orchestra.run.smithi092:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0e288098-fa70-11ee-bc8e-c7b262605968@osd.5.service' 2024-04-14T15:30:18.473 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-14T15:30:18.473 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-14T15:30:18.473 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-14T15:30:18.473 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ceph-0e288098-fa70-11ee-bc8e-c7b262605968@osd.6 2024-04-14T15:30:18.801 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:18 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6[52258]: 2024-04-14T15:30:18.506+0000 7fb5db8ea700 -1 osd.6 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:31.616672+0000 front 2024-04-14T15:29:31.616871+0000 (oldest deadline 2024-04-14T15:29:56.316800+0000) 2024-04-14T15:30:18.801 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:18 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6[52258]: 2024-04-14T15:30:18.506+0000 7fb5db8ea700 -1 osd.6 926 heartbeat_check: no reply from 172.21.15.90:6807 osd.1 since back 2024-04-14T15:29:41.518416+0000 front 2024-04-14T15:29:41.518436+0000 (oldest deadline 2024-04-14T15:30:07.418462+0000) 2024-04-14T15:30:18.801 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:18 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6[52258]: 2024-04-14T15:30:18.506+0000 7fb5db8ea700 -1 osd.6 926 heartbeat_check: no reply from 172.21.15.90:6811 osd.2 since back 2024-04-14T15:29:50.819830+0000 front 2024-04-14T15:29:50.819846+0000 (oldest deadline 2024-04-14T15:30:12.519819+0000) 2024-04-14T15:30:18.802 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:18 smithi092 systemd[1]: Stopping Ceph osd.6 for 0e288098-fa70-11ee-bc8e-c7b262605968... 2024-04-14T15:30:18.802 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:18 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6[52258]: 2024-04-14T15:30:18.800+0000 7fb5df2f6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T15:30:18.802 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:18 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6[52258]: 2024-04-14T15:30:18.800+0000 7fb5df2f6700 -1 osd.6 926 *** Got signal Terminated *** 2024-04-14T15:30:18.802 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:18 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6[52258]: 2024-04-14T15:30:18.800+0000 7fb5df2f6700 -1 osd.6 926 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-14T15:30:19.872 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:19 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6[52258]: 2024-04-14T15:30:19.475+0000 7fb5db8ea700 -1 osd.6 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:31.616672+0000 front 2024-04-14T15:29:31.616871+0000 (oldest deadline 2024-04-14T15:29:56.316800+0000) 2024-04-14T15:30:19.873 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:19 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6[52258]: 2024-04-14T15:30:19.475+0000 7fb5db8ea700 -1 osd.6 926 heartbeat_check: no reply from 172.21.15.90:6807 osd.1 since back 2024-04-14T15:29:41.518416+0000 front 2024-04-14T15:29:41.518436+0000 (oldest deadline 2024-04-14T15:30:07.418462+0000) 2024-04-14T15:30:19.873 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:19 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6[52258]: 2024-04-14T15:30:19.475+0000 7fb5db8ea700 -1 osd.6 926 heartbeat_check: no reply from 172.21.15.90:6811 osd.2 since back 2024-04-14T15:29:50.819830+0000 front 2024-04-14T15:29:50.819846+0000 (oldest deadline 2024-04-14T15:30:12.519819+0000) 2024-04-14T15:30:19.873 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:19 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:19.389+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:34.221673+0000 front 2024-04-14T15:29:34.221775+0000 (oldest deadline 2024-04-14T15:29:55.921555+0000) 2024-04-14T15:30:19.873 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:19 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:19.389+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6807 osd.1 since back 2024-04-14T15:29:42.923224+0000 front 2024-04-14T15:29:42.923341+0000 (oldest deadline 2024-04-14T15:30:06.423218+0000) 2024-04-14T15:30:19.873 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:19 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:19.389+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6811 osd.2 since back 2024-04-14T15:29:49.924376+0000 front 2024-04-14T15:29:49.924338+0000 (oldest deadline 2024-04-14T15:30:14.624363+0000) 2024-04-14T15:30:20.622 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:20 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6[52258]: 2024-04-14T15:30:20.510+0000 7fb5db8ea700 -1 osd.6 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:31.616672+0000 front 2024-04-14T15:29:31.616871+0000 (oldest deadline 2024-04-14T15:29:56.316800+0000) 2024-04-14T15:30:20.622 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:20 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6[52258]: 2024-04-14T15:30:20.510+0000 7fb5db8ea700 -1 osd.6 926 heartbeat_check: no reply from 172.21.15.90:6807 osd.1 since back 2024-04-14T15:29:41.518416+0000 front 2024-04-14T15:29:41.518436+0000 (oldest deadline 2024-04-14T15:30:07.418462+0000) 2024-04-14T15:30:20.622 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:20 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6[52258]: 2024-04-14T15:30:20.510+0000 7fb5db8ea700 -1 osd.6 926 heartbeat_check: no reply from 172.21.15.90:6811 osd.2 since back 2024-04-14T15:29:50.819830+0000 front 2024-04-14T15:29:50.819846+0000 (oldest deadline 2024-04-14T15:30:12.519819+0000) 2024-04-14T15:30:20.622 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:20 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:20.343+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:34.221673+0000 front 2024-04-14T15:29:34.221775+0000 (oldest deadline 2024-04-14T15:29:55.921555+0000) 2024-04-14T15:30:20.622 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:20 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:20.343+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6807 osd.1 since back 2024-04-14T15:29:42.923224+0000 front 2024-04-14T15:29:42.923341+0000 (oldest deadline 2024-04-14T15:30:06.423218+0000) 2024-04-14T15:30:20.623 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:20 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:20.343+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6811 osd.2 since back 2024-04-14T15:29:49.924376+0000 front 2024-04-14T15:29:49.924338+0000 (oldest deadline 2024-04-14T15:30:14.624363+0000) 2024-04-14T15:30:21.622 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:21 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6[52258]: 2024-04-14T15:30:21.535+0000 7fb5db8ea700 -1 osd.6 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:31.616672+0000 front 2024-04-14T15:29:31.616871+0000 (oldest deadline 2024-04-14T15:29:56.316800+0000) 2024-04-14T15:30:21.623 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:21 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6[52258]: 2024-04-14T15:30:21.535+0000 7fb5db8ea700 -1 osd.6 926 heartbeat_check: no reply from 172.21.15.90:6807 osd.1 since back 2024-04-14T15:29:41.518416+0000 front 2024-04-14T15:29:41.518436+0000 (oldest deadline 2024-04-14T15:30:07.418462+0000) 2024-04-14T15:30:21.623 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:21 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6[52258]: 2024-04-14T15:30:21.535+0000 7fb5db8ea700 -1 osd.6 926 heartbeat_check: no reply from 172.21.15.90:6811 osd.2 since back 2024-04-14T15:29:50.819830+0000 front 2024-04-14T15:29:50.819846+0000 (oldest deadline 2024-04-14T15:30:12.519819+0000) 2024-04-14T15:30:21.623 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:21 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:21.345+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:34.221673+0000 front 2024-04-14T15:29:34.221775+0000 (oldest deadline 2024-04-14T15:29:55.921555+0000) 2024-04-14T15:30:21.623 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:21 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:21.345+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6807 osd.1 since back 2024-04-14T15:29:42.923224+0000 front 2024-04-14T15:29:42.923341+0000 (oldest deadline 2024-04-14T15:30:06.423218+0000) 2024-04-14T15:30:21.623 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:21 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:21.345+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6811 osd.2 since back 2024-04-14T15:29:49.924376+0000 front 2024-04-14T15:29:49.924338+0000 (oldest deadline 2024-04-14T15:30:14.624363+0000) 2024-04-14T15:30:22.622 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:22 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:22.341+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:34.221673+0000 front 2024-04-14T15:29:34.221775+0000 (oldest deadline 2024-04-14T15:29:55.921555+0000) 2024-04-14T15:30:22.622 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:22 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:22.341+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6807 osd.1 since back 2024-04-14T15:29:42.923224+0000 front 2024-04-14T15:29:42.923341+0000 (oldest deadline 2024-04-14T15:30:06.423218+0000) 2024-04-14T15:30:22.622 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:22 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:22.341+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6811 osd.2 since back 2024-04-14T15:29:49.924376+0000 front 2024-04-14T15:29:49.924338+0000 (oldest deadline 2024-04-14T15:30:14.624363+0000) 2024-04-14T15:30:22.622 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:22 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6[52258]: 2024-04-14T15:30:22.511+0000 7fb5db8ea700 -1 osd.6 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:31.616672+0000 front 2024-04-14T15:29:31.616871+0000 (oldest deadline 2024-04-14T15:29:56.316800+0000) 2024-04-14T15:30:22.623 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:22 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6[52258]: 2024-04-14T15:30:22.511+0000 7fb5db8ea700 -1 osd.6 926 heartbeat_check: no reply from 172.21.15.90:6807 osd.1 since back 2024-04-14T15:29:41.518416+0000 front 2024-04-14T15:29:41.518436+0000 (oldest deadline 2024-04-14T15:30:07.418462+0000) 2024-04-14T15:30:22.623 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:22 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6[52258]: 2024-04-14T15:30:22.511+0000 7fb5db8ea700 -1 osd.6 926 heartbeat_check: no reply from 172.21.15.90:6811 osd.2 since back 2024-04-14T15:29:50.819830+0000 front 2024-04-14T15:29:50.819846+0000 (oldest deadline 2024-04-14T15:30:12.519819+0000) 2024-04-14T15:30:22.623 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:22 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6[52258]: 2024-04-14T15:30:22.511+0000 7fb5db8ea700 -1 osd.6 926 heartbeat_check: no reply from 172.21.15.90:6815 osd.3 since back 2024-04-14T15:29:58.921097+0000 front 2024-04-14T15:29:58.921125+0000 (oldest deadline 2024-04-14T15:30:21.821057+0000) 2024-04-14T15:30:23.622 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:23 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6[52258]: 2024-04-14T15:30:23.529+0000 7fb5db8ea700 -1 osd.6 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:31.616672+0000 front 2024-04-14T15:29:31.616871+0000 (oldest deadline 2024-04-14T15:29:56.316800+0000) 2024-04-14T15:30:23.622 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:23 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6[52258]: 2024-04-14T15:30:23.529+0000 7fb5db8ea700 -1 osd.6 926 heartbeat_check: no reply from 172.21.15.90:6807 osd.1 since back 2024-04-14T15:29:41.518416+0000 front 2024-04-14T15:29:41.518436+0000 (oldest deadline 2024-04-14T15:30:07.418462+0000) 2024-04-14T15:30:23.623 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:23 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6[52258]: 2024-04-14T15:30:23.529+0000 7fb5db8ea700 -1 osd.6 926 heartbeat_check: no reply from 172.21.15.90:6811 osd.2 since back 2024-04-14T15:29:50.819830+0000 front 2024-04-14T15:29:50.819846+0000 (oldest deadline 2024-04-14T15:30:12.519819+0000) 2024-04-14T15:30:23.623 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:23 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6[52258]: 2024-04-14T15:30:23.529+0000 7fb5db8ea700 -1 osd.6 926 heartbeat_check: no reply from 172.21.15.90:6815 osd.3 since back 2024-04-14T15:29:58.921097+0000 front 2024-04-14T15:29:58.921125+0000 (oldest deadline 2024-04-14T15:30:21.821057+0000) 2024-04-14T15:30:23.623 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:23 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:23.306+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:34.221673+0000 front 2024-04-14T15:29:34.221775+0000 (oldest deadline 2024-04-14T15:29:55.921555+0000) 2024-04-14T15:30:23.623 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:23 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:23.306+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6807 osd.1 since back 2024-04-14T15:29:42.923224+0000 front 2024-04-14T15:29:42.923341+0000 (oldest deadline 2024-04-14T15:30:06.423218+0000) 2024-04-14T15:30:23.623 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:23 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:23.306+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6811 osd.2 since back 2024-04-14T15:29:49.924376+0000 front 2024-04-14T15:29:49.924338+0000 (oldest deadline 2024-04-14T15:30:14.624363+0000) 2024-04-14T15:30:23.941 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:30:23 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-alertmanager-a[84115]: ts=2024-04-14T15:30:23.571Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi090.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.90:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi092.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.92:8443: connect: connection refused" 2024-04-14T15:30:23.941 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:30:23 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-alertmanager-a[84115]: ts=2024-04-14T15:30:23.571Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi090.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.90:8443: connect: connection refused" 2024-04-14T15:30:23.942 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:30:23 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-alertmanager-a[84115]: ts=2024-04-14T15:30:23.572Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi092.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.92:8443: connect: connection refused" 2024-04-14T15:30:24.540 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:24 smithi092 podman[152633]: 2024-04-14 15:30:24.212366736 +0000 UTC m=+5.519136713 container died cb4dd1d4b350bc4eb07ccf0a4ca96969a3e4932badde0c39b875b263528ce23b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=main-a971fd5, org.label-schema.build-date=20240403, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-14T15:30:24.540 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:24 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:24.274+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:34.221673+0000 front 2024-04-14T15:29:34.221775+0000 (oldest deadline 2024-04-14T15:29:55.921555+0000) 2024-04-14T15:30:24.540 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:24 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:24.274+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6807 osd.1 since back 2024-04-14T15:29:42.923224+0000 front 2024-04-14T15:29:42.923341+0000 (oldest deadline 2024-04-14T15:30:06.423218+0000) 2024-04-14T15:30:24.540 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:24 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:24.274+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6811 osd.2 since back 2024-04-14T15:29:49.924376+0000 front 2024-04-14T15:29:49.924338+0000 (oldest deadline 2024-04-14T15:30:14.624363+0000) 2024-04-14T15:30:24.540 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:24 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:24.274+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6815 osd.3 since back 2024-04-14T15:29:59.325252+0000 front 2024-04-14T15:29:59.325337+0000 (oldest deadline 2024-04-14T15:30:23.425312+0000) 2024-04-14T15:30:24.815 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:24 smithi092 podman[152633]: 2024-04-14 15:30:24.613695765 +0000 UTC m=+5.920465750 container cleanup cb4dd1d4b350bc4eb07ccf0a4ca96969a3e4932badde0c39b875b263528ce23b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6, RELEASE=main-a971fd5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-04-14T15:30:24.816 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:24 smithi092 bash[152633]: ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6 2024-04-14T15:30:25.106 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:24 smithi092 podman[152654]: 2024-04-14 15:30:24.814855416 +0000 UTC m=+0.602708235 container remove cb4dd1d4b350bc4eb07ccf0a4ca96969a3e4932badde0c39b875b263528ce23b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=main-a971fd5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, GIT_BRANCH=HEAD) 2024-04-14T15:30:25.372 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:25 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:25.258+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:34.221673+0000 front 2024-04-14T15:29:34.221775+0000 (oldest deadline 2024-04-14T15:29:55.921555+0000) 2024-04-14T15:30:25.372 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:25 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:25.258+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6807 osd.1 since back 2024-04-14T15:29:42.923224+0000 front 2024-04-14T15:29:42.923341+0000 (oldest deadline 2024-04-14T15:30:06.423218+0000) 2024-04-14T15:30:25.372 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:25 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:25.258+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6811 osd.2 since back 2024-04-14T15:29:49.924376+0000 front 2024-04-14T15:29:49.924338+0000 (oldest deadline 2024-04-14T15:30:14.624363+0000) 2024-04-14T15:30:25.372 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:25 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:25.258+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6815 osd.3 since back 2024-04-14T15:29:59.325252+0000 front 2024-04-14T15:29:59.325337+0000 (oldest deadline 2024-04-14T15:30:23.425312+0000) 2024-04-14T15:30:25.372 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:25 smithi092 podman[152752]: 2024-04-14 15:30:25.107362232 +0000 UTC m=+0.114974241 container create ca8ed53779f786f4e85d49d12bf2ec8de8ab40213ad5fa17b8c2e4a7fb2d1a66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=main-a971fd5, org.label-schema.schema-version=1.0) 2024-04-14T15:30:25.373 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:25 smithi092 podman[152752]: 2024-04-14 15:30:25.010184682 +0000 UTC m=+0.017796704 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608 2024-04-14T15:30:25.373 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:25 smithi092 podman[152752]: 2024-04-14 15:30:25.334866858 +0000 UTC m=+0.342478867 container init ca8ed53779f786f4e85d49d12bf2ec8de8ab40213ad5fa17b8c2e4a7fb2d1a66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, RELEASE=main-a971fd5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True) 2024-04-14T15:30:25.373 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:25 smithi092 podman[152752]: 2024-04-14 15:30:25.340217729 +0000 UTC m=+0.347829739 container start ca8ed53779f786f4e85d49d12bf2ec8de8ab40213ad5fa17b8c2e4a7fb2d1a66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, RELEASE=main-a971fd5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-14T15:30:25.875 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:25 smithi092 podman[152752]: 2024-04-14 15:30:25.426156468 +0000 UTC m=+0.433768484 container attach ca8ed53779f786f4e85d49d12bf2ec8de8ab40213ad5fa17b8c2e4a7fb2d1a66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, RELEASE=main-a971fd5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=) 2024-04-14T15:30:25.875 INFO:journalctl@ceph.osd.6.smithi092.stdout:Apr 14 15:30:25 smithi092 podman[152752]: 2024-04-14 15:30:25.642468834 +0000 UTC m=+0.650080844 container died ca8ed53779f786f4e85d49d12bf2ec8de8ab40213ad5fa17b8c2e4a7fb2d1a66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-6-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=main-a971fd5, io.buildah.version=1.33.5) 2024-04-14T15:30:26.354 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:26 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:26.305+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:34.221673+0000 front 2024-04-14T15:29:34.221775+0000 (oldest deadline 2024-04-14T15:29:55.921555+0000) 2024-04-14T15:30:26.354 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:26 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:26.305+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6807 osd.1 since back 2024-04-14T15:29:42.923224+0000 front 2024-04-14T15:29:42.923341+0000 (oldest deadline 2024-04-14T15:30:06.423218+0000) 2024-04-14T15:30:26.354 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:26 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:26.305+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6811 osd.2 since back 2024-04-14T15:29:49.924376+0000 front 2024-04-14T15:29:49.924338+0000 (oldest deadline 2024-04-14T15:30:14.624363+0000) 2024-04-14T15:30:26.354 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:26 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:26.305+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6815 osd.3 since back 2024-04-14T15:29:59.325252+0000 front 2024-04-14T15:29:59.325337+0000 (oldest deadline 2024-04-14T15:30:23.425312+0000) 2024-04-14T15:30:26.377 DEBUG:teuthology.orchestra.run.smithi092:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0e288098-fa70-11ee-bc8e-c7b262605968@osd.6.service' 2024-04-14T15:30:26.423 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-14T15:30:26.423 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-14T15:30:26.423 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-14T15:30:26.423 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ceph-0e288098-fa70-11ee-bc8e-c7b262605968@osd.7 2024-04-14T15:30:26.622 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:26 smithi092 systemd[1]: Stopping Ceph osd.7 for 0e288098-fa70-11ee-bc8e-c7b262605968... 2024-04-14T15:30:27.122 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:26 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:26.752+0000 7fa00d5da700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T15:30:27.122 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:26 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:26.752+0000 7fa00d5da700 -1 osd.7 926 *** Got signal Terminated *** 2024-04-14T15:30:27.122 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:26 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:26.752+0000 7fa00d5da700 -1 osd.7 926 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-14T15:30:27.622 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:27 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:27.343+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:34.221673+0000 front 2024-04-14T15:29:34.221775+0000 (oldest deadline 2024-04-14T15:29:55.921555+0000) 2024-04-14T15:30:27.622 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:27 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:27.343+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6807 osd.1 since back 2024-04-14T15:29:42.923224+0000 front 2024-04-14T15:29:42.923341+0000 (oldest deadline 2024-04-14T15:30:06.423218+0000) 2024-04-14T15:30:27.622 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:27 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:27.343+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6811 osd.2 since back 2024-04-14T15:29:49.924376+0000 front 2024-04-14T15:29:49.924338+0000 (oldest deadline 2024-04-14T15:30:14.624363+0000) 2024-04-14T15:30:27.622 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:27 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:27.343+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6815 osd.3 since back 2024-04-14T15:29:59.325252+0000 front 2024-04-14T15:29:59.325337+0000 (oldest deadline 2024-04-14T15:30:23.425312+0000) 2024-04-14T15:30:28.622 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:28 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:28.340+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:34.221673+0000 front 2024-04-14T15:29:34.221775+0000 (oldest deadline 2024-04-14T15:29:55.921555+0000) 2024-04-14T15:30:28.622 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:28 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:28.340+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6807 osd.1 since back 2024-04-14T15:29:42.923224+0000 front 2024-04-14T15:29:42.923341+0000 (oldest deadline 2024-04-14T15:30:06.423218+0000) 2024-04-14T15:30:28.622 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:28 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:28.340+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6811 osd.2 since back 2024-04-14T15:29:49.924376+0000 front 2024-04-14T15:29:49.924338+0000 (oldest deadline 2024-04-14T15:30:14.624363+0000) 2024-04-14T15:30:28.622 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:28 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:28.340+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6815 osd.3 since back 2024-04-14T15:29:59.325252+0000 front 2024-04-14T15:29:59.325337+0000 (oldest deadline 2024-04-14T15:30:23.425312+0000) 2024-04-14T15:30:29.622 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:29 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:29.345+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:34.221673+0000 front 2024-04-14T15:29:34.221775+0000 (oldest deadline 2024-04-14T15:29:55.921555+0000) 2024-04-14T15:30:29.622 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:29 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:29.345+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6807 osd.1 since back 2024-04-14T15:29:42.923224+0000 front 2024-04-14T15:29:42.923341+0000 (oldest deadline 2024-04-14T15:30:06.423218+0000) 2024-04-14T15:30:29.622 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:29 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:29.345+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6811 osd.2 since back 2024-04-14T15:29:49.924376+0000 front 2024-04-14T15:29:49.924338+0000 (oldest deadline 2024-04-14T15:30:14.624363+0000) 2024-04-14T15:30:29.622 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:29 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:29.345+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6815 osd.3 since back 2024-04-14T15:29:59.325252+0000 front 2024-04-14T15:29:59.325337+0000 (oldest deadline 2024-04-14T15:30:23.425312+0000) 2024-04-14T15:30:30.622 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:30 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:30.344+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:34.221673+0000 front 2024-04-14T15:29:34.221775+0000 (oldest deadline 2024-04-14T15:29:55.921555+0000) 2024-04-14T15:30:30.622 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:30 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:30.344+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6807 osd.1 since back 2024-04-14T15:29:42.923224+0000 front 2024-04-14T15:29:42.923341+0000 (oldest deadline 2024-04-14T15:30:06.423218+0000) 2024-04-14T15:30:30.622 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:30 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:30.344+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6811 osd.2 since back 2024-04-14T15:29:49.924376+0000 front 2024-04-14T15:29:49.924338+0000 (oldest deadline 2024-04-14T15:30:14.624363+0000) 2024-04-14T15:30:30.622 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:30 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:30.344+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6815 osd.3 since back 2024-04-14T15:29:59.325252+0000 front 2024-04-14T15:29:59.325337+0000 (oldest deadline 2024-04-14T15:30:23.425312+0000) 2024-04-14T15:30:31.622 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:31 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:31.311+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6803 osd.0 since back 2024-04-14T15:29:34.221673+0000 front 2024-04-14T15:29:34.221775+0000 (oldest deadline 2024-04-14T15:29:55.921555+0000) 2024-04-14T15:30:31.622 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:31 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:31.311+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6807 osd.1 since back 2024-04-14T15:29:42.923224+0000 front 2024-04-14T15:29:42.923341+0000 (oldest deadline 2024-04-14T15:30:06.423218+0000) 2024-04-14T15:30:31.622 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:31 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:31.311+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6811 osd.2 since back 2024-04-14T15:29:49.924376+0000 front 2024-04-14T15:29:49.924338+0000 (oldest deadline 2024-04-14T15:30:14.624363+0000) 2024-04-14T15:30:31.622 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:31 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7[58229]: 2024-04-14T15:30:31.311+0000 7fa009bce700 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.90:6815 osd.3 since back 2024-04-14T15:29:59.325252+0000 front 2024-04-14T15:29:59.325337+0000 (oldest deadline 2024-04-14T15:30:23.425312+0000) 2024-04-14T15:30:32.391 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:32 smithi092 podman[152884]: 2024-04-14 15:30:32.135744865 +0000 UTC m=+5.485482926 container died 6fde7adb5111e20ec7e559eecda957a0e3dc0944933bcaa5d02c35b7756452cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=main-a971fd5, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-14T15:30:32.656 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:32 smithi092 podman[152884]: 2024-04-14 15:30:32.473037529 +0000 UTC m=+5.822775586 container cleanup 6fde7adb5111e20ec7e559eecda957a0e3dc0944933bcaa5d02c35b7756452cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a971fd5, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-14T15:30:32.657 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:32 smithi092 bash[152884]: ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7 2024-04-14T15:30:33.062 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:32 smithi092 podman[152922]: 2024-04-14 15:30:32.657363996 +0000 UTC m=+0.517545509 container remove 6fde7adb5111e20ec7e559eecda957a0e3dc0944933bcaa5d02c35b7756452cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7, RELEASE=main-a971fd5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, io.buildah.version=1.33.5) 2024-04-14T15:30:33.343 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:33 smithi092 podman[153024]: 2024-04-14 15:30:32.965380689 +0000 UTC m=+0.018065079 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608 2024-04-14T15:30:33.343 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:33 smithi092 podman[153024]: 2024-04-14 15:30:33.074923567 +0000 UTC m=+0.127607951 container create 9700a6a3063a3253a68c443afe6526181b8a3a87d2d21bfde56e446127eb128e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7-deactivate, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, RELEASE=main-a971fd5, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-14T15:30:33.343 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:33 smithi092 podman[153024]: 2024-04-14 15:30:33.26062724 +0000 UTC m=+0.313311627 container init 9700a6a3063a3253a68c443afe6526181b8a3a87d2d21bfde56e446127eb128e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-a971fd5, io.buildah.version=1.33.5) 2024-04-14T15:30:33.343 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:33 smithi092 podman[153024]: 2024-04-14 15:30:33.266346092 +0000 UTC m=+0.319030479 container start 9700a6a3063a3253a68c443afe6526181b8a3a87d2d21bfde56e446127eb128e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=main-a971fd5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-14T15:30:33.343 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:33 smithi092 podman[153024]: 2024-04-14 15:30:33.343470864 +0000 UTC m=+0.396155241 container attach 9700a6a3063a3253a68c443afe6526181b8a3a87d2d21bfde56e446127eb128e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, RELEASE=main-a971fd5, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-14T15:30:33.622 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 14 15:30:33 smithi092 podman[153024]: 2024-04-14 15:30:33.563456344 +0000 UTC m=+0.616140732 container died 9700a6a3063a3253a68c443afe6526181b8a3a87d2d21bfde56e446127eb128e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-osd-7-deactivate, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=main-a971fd5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-14T15:30:33.941 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:30:33 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-alertmanager-a[84115]: ts=2024-04-14T15:30:33.572Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi092.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.92:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi090.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.90:8443: connect: connection refused" 2024-04-14T15:30:33.942 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:30:33 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-alertmanager-a[84115]: ts=2024-04-14T15:30:33.572Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi090.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.90:8443: connect: connection refused" 2024-04-14T15:30:33.942 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:30:33 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-alertmanager-a[84115]: ts=2024-04-14T15:30:33.573Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi092.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.92:8443: connect: connection refused" 2024-04-14T15:30:34.229 DEBUG:teuthology.orchestra.run.smithi092:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0e288098-fa70-11ee-bc8e-c7b262605968@osd.7.service' 2024-04-14T15:30:34.270 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-14T15:30:34.270 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-14T15:30:34.270 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-04-14T15:30:34.270 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ceph-0e288098-fa70-11ee-bc8e-c7b262605968@rgw.foo.a 2024-04-14T15:30:34.622 INFO:journalctl@ceph.rgw.foo.a.smithi090.stdout:Apr 14 15:30:34 smithi090 systemd[1]: Stopping Ceph rgw.foo.a for 0e288098-fa70-11ee-bc8e-c7b262605968... 2024-04-14T15:30:34.941 INFO:journalctl@ceph.rgw.foo.a.smithi090.stdout:Apr 14 15:30:34 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-rgw-foo-a[70053]: 2024-04-14T15:30:34.621+0000 7fbac0a04700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T15:30:34.942 INFO:journalctl@ceph.rgw.foo.a.smithi090.stdout:Apr 14 15:30:34 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-rgw-foo-a[70053]: 2024-04-14T15:30:34.621+0000 7fbacf28bac0 -1 shutting down 2024-04-14T15:30:43.941 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:30:43 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-alertmanager-a[84115]: ts=2024-04-14T15:30:43.573Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi090.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.90:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi092.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.92:8443: connect: connection refused" 2024-04-14T15:30:43.942 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:30:43 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-alertmanager-a[84115]: ts=2024-04-14T15:30:43.573Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi090.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.90:8443: connect: connection refused" 2024-04-14T15:30:43.942 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:30:43 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-alertmanager-a[84115]: ts=2024-04-14T15:30:43.573Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi092.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.92:8443: connect: connection refused" 2024-04-14T15:30:44.921 INFO:journalctl@ceph.rgw.foo.a.smithi090.stdout:Apr 14 15:30:44 smithi090 bash[293702]: time="2024-04-14T15:30:44Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-0e288098-fa70-11ee-bc8e-c7b262605968-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-04-14T15:30:44.921 INFO:journalctl@ceph.rgw.foo.a.smithi090.stdout:Apr 14 15:30:44 smithi090 podman[293702]: 2024-04-14 15:30:44.646528202 +0000 UTC m=+10.178899041 container died fc3f1f3469ebe7f93ba3030a48e39b911fa0f7188cea2aa30a085198b9a7c5f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-rgw-foo-a, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-a971fd5, org.label-schema.build-date=20240403, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-14T15:30:45.561 INFO:journalctl@ceph.rgw.foo.a.smithi090.stdout:Apr 14 15:30:45 smithi090 podman[293702]: 2024-04-14 15:30:45.213023619 +0000 UTC m=+10.745394449 container cleanup fc3f1f3469ebe7f93ba3030a48e39b911fa0f7188cea2aa30a085198b9a7c5f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-rgw-foo-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=main-a971fd5, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-14T15:30:45.562 INFO:journalctl@ceph.rgw.foo.a.smithi090.stdout:Apr 14 15:30:45 smithi090 bash[293702]: ceph-0e288098-fa70-11ee-bc8e-c7b262605968-rgw-foo-a 2024-04-14T15:30:45.562 INFO:journalctl@ceph.rgw.foo.a.smithi090.stdout:Apr 14 15:30:45 smithi090 podman[293738]: 2024-04-14 15:30:45.455967419 +0000 UTC m=+0.803049558 container remove fc3f1f3469ebe7f93ba3030a48e39b911fa0f7188cea2aa30a085198b9a7c5f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-rgw-foo-a, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=main-a971fd5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-14T15:30:45.562 INFO:journalctl@ceph.rgw.foo.a.smithi090.stdout:Apr 14 15:30:45 smithi090 systemd[1]: ceph-0e288098-fa70-11ee-bc8e-c7b262605968@rgw.foo.a.service: Main process exited, code=exited, status=137/n/a 2024-04-14T15:30:45.576 DEBUG:teuthology.orchestra.run.smithi090:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0e288098-fa70-11ee-bc8e-c7b262605968@rgw.foo.a.service' 2024-04-14T15:30:45.615 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-14T15:30:45.615 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-04-14T15:30:45.616 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-04-14T15:30:45.616 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ceph-0e288098-fa70-11ee-bc8e-c7b262605968@prometheus.a 2024-04-14T15:30:45.925 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:30:45 smithi092 systemd[1]: Stopping Ceph prometheus.a for 0e288098-fa70-11ee-bc8e-c7b262605968... 2024-04-14T15:30:46.223 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:30:45 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[70312]: ts=2024-04-14T15:30:45.925Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-04-14T15:30:46.223 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:30:45 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[70312]: ts=2024-04-14T15:30:45.926Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-04-14T15:30:46.223 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:30:45 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[70312]: ts=2024-04-14T15:30:45.926Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-04-14T15:30:46.223 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:30:45 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[70312]: ts=2024-04-14T15:30:45.926Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-04-14T15:30:46.223 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:30:45 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[70312]: ts=2024-04-14T15:30:45.926Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-04-14T15:30:46.223 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:30:45 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[70312]: ts=2024-04-14T15:30:45.926Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-04-14T15:30:46.223 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:30:45 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[70312]: ts=2024-04-14T15:30:45.926Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-04-14T15:30:46.223 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:30:45 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[70312]: ts=2024-04-14T15:30:45.926Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-04-14T15:30:46.223 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:30:45 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[70312]: ts=2024-04-14T15:30:45.926Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-04-14T15:30:46.223 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:30:45 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[70312]: ts=2024-04-14T15:30:45.926Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-04-14T15:30:46.224 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:30:45 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[70312]: ts=2024-04-14T15:30:45.926Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-04-14T15:30:46.224 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:30:45 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a[70312]: ts=2024-04-14T15:30:45.926Z caller=main.go:1147 level=info msg="See you next time!" 2024-04-14T15:30:46.224 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:30:45 smithi092 podman[153154]: 2024-04-14 15:30:45.936290582 +0000 UTC m=+0.125563207 container died 380bcfffec5aad23cb566afcb38e91626b634d53028d6979cda36041fdc7210a (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T15:30:46.622 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:30:46 smithi092 podman[153154]: 2024-04-14 15:30:46.422688494 +0000 UTC m=+0.611961162 container cleanup 380bcfffec5aad23cb566afcb38e91626b634d53028d6979cda36041fdc7210a (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T15:30:46.622 INFO:journalctl@ceph.prometheus.a.smithi092.stdout:Apr 14 15:30:46 smithi092 bash[153154]: ceph-0e288098-fa70-11ee-bc8e-c7b262605968-prometheus-a 2024-04-14T15:30:46.783 DEBUG:teuthology.orchestra.run.smithi092:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0e288098-fa70-11ee-bc8e-c7b262605968@prometheus.a.service' 2024-04-14T15:30:46.821 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-14T15:30:46.821 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-04-14T15:30:46.821 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 --force --keep-logs 2024-04-14T15:30:47.015 INFO:teuthology.orchestra.run.smithi090.stdout:Deleting cluster with fsid: 0e288098-fa70-11ee-bc8e-c7b262605968 2024-04-14T15:30:48.692 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:30:48 smithi090 systemd[1]: Stopping Ceph alertmanager.a for 0e288098-fa70-11ee-bc8e-c7b262605968... 2024-04-14T15:30:49.191 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:30:48 smithi090 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-alertmanager-a[84115]: ts=2024-04-14T15:30:48.862Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-04-14T15:30:49.191 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:30:48 smithi090 podman[294223]: 2024-04-14 15:30:48.904116845 +0000 UTC m=+0.152208184 container died a1404ca24609bf7cccd9a82746fda5c420e34341a6dfd2db0b46113dda9ee0b0 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-14T15:30:49.621 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:30:49 smithi090 podman[294223]: 2024-04-14 15:30:49.361254447 +0000 UTC m=+0.609345829 container cleanup a1404ca24609bf7cccd9a82746fda5c420e34341a6dfd2db0b46113dda9ee0b0 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-14T15:30:49.621 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:30:49 smithi090 bash[294223]: ceph-0e288098-fa70-11ee-bc8e-c7b262605968-alertmanager-a 2024-04-14T15:30:49.941 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:30:49 smithi090 podman[294242]: 2024-04-14 15:30:49.622207075 +0000 UTC m=+0.711842531 container remove a1404ca24609bf7cccd9a82746fda5c420e34341a6dfd2db0b46113dda9ee0b0 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-14T15:30:49.941 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:30:49 smithi090 podman[294242]: 2024-04-14 15:30:49.697740881 +0000 UTC m=+0.787376295 volume remove 72c3f1ebb2f88129542df071edf80dc669c248e37d196c184d271ea1c320de81 2024-04-14T15:30:49.941 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:30:49 smithi090 systemd[1]: ceph-0e288098-fa70-11ee-bc8e-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-04-14T15:30:49.941 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:30:49 smithi090 systemd[1]: Stopped Ceph alertmanager.a for 0e288098-fa70-11ee-bc8e-c7b262605968. 2024-04-14T15:30:49.941 INFO:journalctl@ceph.alertmanager.a.smithi090.stdout:Apr 14 15:30:49 smithi090 systemd[1]: ceph-0e288098-fa70-11ee-bc8e-c7b262605968@alertmanager.a.service: Consumed 1.697s CPU time. 2024-04-14T15:30:50.430 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:30:50 smithi090 systemd[1]: Stopping Ceph node-exporter.a for 0e288098-fa70-11ee-bc8e-c7b262605968... 2024-04-14T15:30:50.684 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:30:50 smithi090 podman[294358]: 2024-04-14 15:30:50.430677566 +0000 UTC m=+0.267868413 container died 5f2702315825ca14b91f0cc47d486173ceed9ffcad5cdac18bfba74183f27629 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-14T15:30:50.941 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:30:50 smithi090 podman[294358]: 2024-04-14 15:30:50.85939814 +0000 UTC m=+0.696589029 container cleanup 5f2702315825ca14b91f0cc47d486173ceed9ffcad5cdac18bfba74183f27629 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-14T15:30:50.941 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:30:50 smithi090 bash[294358]: ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-a 2024-04-14T15:30:51.303 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:30:51 smithi090 podman[294377]: 2024-04-14 15:30:51.044013755 +0000 UTC m=+0.614014632 container remove 5f2702315825ca14b91f0cc47d486173ceed9ffcad5cdac18bfba74183f27629 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-14T15:30:51.303 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:30:51 smithi090 systemd[1]: ceph-0e288098-fa70-11ee-bc8e-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-04-14T15:30:51.303 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:30:51 smithi090 systemd[1]: ceph-0e288098-fa70-11ee-bc8e-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-04-14T15:30:51.304 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:30:51 smithi090 systemd[1]: Stopped Ceph node-exporter.a for 0e288098-fa70-11ee-bc8e-c7b262605968. 2024-04-14T15:30:51.304 INFO:journalctl@ceph.node-exporter.a.smithi090.stdout:Apr 14 15:30:51 smithi090 systemd[1]: ceph-0e288098-fa70-11ee-bc8e-c7b262605968@node-exporter.a.service: Consumed 3.616s CPU time. 2024-04-14T15:30:51.922 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 --force --keep-logs 2024-04-14T15:30:52.112 INFO:teuthology.orchestra.run.smithi092.stdout:Deleting cluster with fsid: 0e288098-fa70-11ee-bc8e-c7b262605968 2024-04-14T15:30:53.612 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:30:53 smithi092 systemd[1]: Stopping Ceph iscsi.iscsi.a for 0e288098-fa70-11ee-bc8e-c7b262605968... 2024-04-14T15:30:53.872 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:30:53 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a[63506]: debug Shutdown received 2024-04-14T15:31:03.872 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:31:03 smithi092 bash[153573]: time="2024-04-14T15:31:03Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a in 10 seconds, resorting to SIGKILL" 2024-04-14T15:31:03.873 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:31:03 smithi092 podman[153573]: 2024-04-14 15:31:03.634047291 +0000 UTC m=+10.119702937 container died 5d5759575ca229bf030cf7ac396d66ce82e747187a59f1cf64ea0b55b0e0906a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-a971fd5, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-14T15:31:04.272 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:31:04 smithi092 podman[153573]: 2024-04-14 15:31:04.032752458 +0000 UTC m=+10.518408111 container cleanup 5d5759575ca229bf030cf7ac396d66ce82e747187a59f1cf64ea0b55b0e0906a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, RELEASE=main-a971fd5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-14T15:31:04.272 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:31:04 smithi092 bash[153573]: ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a 2024-04-14T15:31:04.622 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:31:04 smithi092 podman[153598]: 2024-04-14 15:31:04.359452161 +0000 UTC m=+0.727465389 container remove 5d5759575ca229bf030cf7ac396d66ce82e747187a59f1cf64ea0b55b0e0906a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-iscsi-iscsi-a, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240403, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=main-a971fd5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-14T15:31:04.622 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:31:04 smithi092 systemd[1]: ceph-0e288098-fa70-11ee-bc8e-c7b262605968@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-04-14T15:31:04.622 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:31:04 smithi092 systemd[1]: ceph-0e288098-fa70-11ee-bc8e-c7b262605968@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-04-14T15:31:04.622 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:31:04 smithi092 systemd[1]: Stopped Ceph iscsi.iscsi.a for 0e288098-fa70-11ee-bc8e-c7b262605968. 2024-04-14T15:31:04.622 INFO:journalctl@ceph.iscsi.iscsi.a.smithi092.stdout:Apr 14 15:31:04 smithi092 systemd[1]: ceph-0e288098-fa70-11ee-bc8e-c7b262605968@iscsi.iscsi.a.service: Consumed 3.603s CPU time. 2024-04-14T15:31:06.122 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:31:06 smithi092 systemd[1]: Stopping Ceph grafana.a for 0e288098-fa70-11ee-bc8e-c7b262605968... 2024-04-14T15:31:06.517 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:31:06 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=server t=2024-04-14T15:31:06.244864473Z level=info msg="Shutdown started" reason="System signal: terminated" 2024-04-14T15:31:06.517 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:31:06 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=ticker t=2024-04-14T15:31:06.245274704Z level=info msg=stopped last_tick=2024-04-14T15:31:00Z 2024-04-14T15:31:06.517 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:31:06 smithi092 ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a[67596]: logger=tracing t=2024-04-14T15:31:06.245314022Z level=info msg="Closing tracing" 2024-04-14T15:31:06.517 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:31:06 smithi092 podman[153887]: 2024-04-14 15:31:06.255405806 +0000 UTC m=+0.114172415 container died 176d192f7fa174bfd5cd7ffe9d1b180411f2e8347142e95e140739b2470aff40 (image=quay.io/ceph/grafana:9.4.12, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-04-14T15:31:06.872 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:31:06 smithi092 podman[153887]: 2024-04-14 15:31:06.60834654 +0000 UTC m=+0.467113192 container cleanup 176d192f7fa174bfd5cd7ffe9d1b180411f2e8347142e95e140739b2470aff40 (image=quay.io/ceph/grafana:9.4.12, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-04-14T15:31:06.872 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:31:06 smithi092 bash[153887]: ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a 2024-04-14T15:31:07.211 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:31:06 smithi092 podman[153906]: 2024-04-14 15:31:06.900805781 +0000 UTC m=+0.647238128 container remove 176d192f7fa174bfd5cd7ffe9d1b180411f2e8347142e95e140739b2470aff40 (image=quay.io/ceph/grafana:9.4.12, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-04-14T15:31:07.212 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:31:07 smithi092 systemd[1]: ceph-0e288098-fa70-11ee-bc8e-c7b262605968@grafana.a.service: Deactivated successfully. 2024-04-14T15:31:07.212 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:31:07 smithi092 systemd[1]: Stopped Ceph grafana.a for 0e288098-fa70-11ee-bc8e-c7b262605968. 2024-04-14T15:31:07.212 INFO:journalctl@ceph.grafana.a.smithi092.stdout:Apr 14 15:31:07 smithi092 systemd[1]: ceph-0e288098-fa70-11ee-bc8e-c7b262605968@grafana.a.service: Consumed 14.136s CPU time. 2024-04-14T15:31:07.467 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:31:07 smithi092 systemd[1]: Stopping Ceph node-exporter.b for 0e288098-fa70-11ee-bc8e-c7b262605968... 2024-04-14T15:31:07.728 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:31:07 smithi092 podman[154023]: 2024-04-14 15:31:07.468006421 +0000 UTC m=+0.120916691 container died 3096ddbef6327091889e6a95eb5d901afc92a9b765fdd00a4d9350e17718db9d (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-14T15:31:08.078 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:31:07 smithi092 podman[154023]: 2024-04-14 15:31:07.828216873 +0000 UTC m=+0.481127127 container cleanup 3096ddbef6327091889e6a95eb5d901afc92a9b765fdd00a4d9350e17718db9d (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-14T15:31:08.079 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:31:07 smithi092 bash[154023]: ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-b 2024-04-14T15:31:08.372 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:31:08 smithi092 podman[154042]: 2024-04-14 15:31:08.079439306 +0000 UTC m=+0.615470005 container remove 3096ddbef6327091889e6a95eb5d901afc92a9b765fdd00a4d9350e17718db9d (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-0e288098-fa70-11ee-bc8e-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-14T15:31:08.372 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:31:08 smithi092 systemd[1]: ceph-0e288098-fa70-11ee-bc8e-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-04-14T15:31:08.372 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:31:08 smithi092 systemd[1]: ceph-0e288098-fa70-11ee-bc8e-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-04-14T15:31:08.372 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:31:08 smithi092 systemd[1]: Stopped Ceph node-exporter.b for 0e288098-fa70-11ee-bc8e-c7b262605968. 2024-04-14T15:31:08.372 INFO:journalctl@ceph.node-exporter.b.smithi092.stdout:Apr 14 15:31:08 smithi092 systemd[1]: ceph-0e288098-fa70-11ee-bc8e-c7b262605968@node-exporter.b.service: Consumed 3.545s CPU time. 2024-04-14T15:31:08.832 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-14T15:31:08.867 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-14T15:31:08.897 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-14T15:31:08.899 DEBUG:teuthology.misc:Transferring archived files from smithi090:/var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-08_20:08:14-orch-main-distro-default-smithi/7646059/remote/smithi090/crash 2024-04-14T15:31:08.900 DEBUG:teuthology.orchestra.run.smithi090:> sudo tar c -f - -C /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/crash -- . 2024-04-14T15:31:08.953 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/crash: Cannot open: No such file or directory 2024-04-14T15:31:08.953 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2024-04-14T15:31:08.955 DEBUG:teuthology.misc:Transferring archived files from smithi092:/var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-08_20:08:14-orch-main-distro-default-smithi/7646059/remote/smithi092/crash 2024-04-14T15:31:08.955 DEBUG:teuthology.orchestra.run.smithi092:> sudo tar c -f - -C /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/crash -- . 2024-04-14T15:31:08.992 INFO:teuthology.orchestra.run.smithi092.stderr:tar: /var/lib/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/crash: Cannot open: No such file or directory 2024-04-14T15:31:08.993 INFO:teuthology.orchestra.run.smithi092.stderr:tar: Error is not recoverable: exiting now 2024-04-14T15:31:08.994 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-14T15:31:08.994 DEBUG:teuthology.orchestra.run.smithi090:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'reached quota' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | head -n 1 2024-04-14T15:31:09.025 INFO:teuthology.orchestra.run.smithi090.stdout:2024-04-14T15:11:49.180327+0000 mon.a (mon.0) 865 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:31:09.027 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-04-14T15:31:09.027 DEBUG:teuthology.orchestra.run.smithi090:> sudo egrep '\[SEC\]' /var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'reached quota' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | head -n 1 2024-04-14T15:31:09.099 DEBUG:teuthology.orchestra.run.smithi090:> sudo egrep '\[ERR\]' /var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'reached quota' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | head -n 1 2024-04-14T15:31:09.170 DEBUG:teuthology.orchestra.run.smithi090:> sudo egrep '\[WRN\]' /var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'reached quota' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | head -n 1 2024-04-14T15:31:09.240 INFO:teuthology.orchestra.run.smithi090.stdout:2024-04-14T15:11:49.180327+0000 mon.a (mon.0) 865 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-14T15:31:09.241 INFO:tasks.cephadm:Compressing logs... 2024-04-14T15:31:09.242 DEBUG:teuthology.orchestra.run.smithi090:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-14T15:31:09.284 DEBUG:teuthology.orchestra.run.smithi092:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-14T15:31:09.309 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose --: No such file or directory /var/log/ceph/cephadm.log 2024-04-14T15:31:09.309 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-04-14T15:31:09.310 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph-mon.a.log 2024-04-14T15:31:09.310 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph.audit.log 2024-04-14T15:31:09.310 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph-client.rgw.foo.a.log 2024-04-14T15:31:09.311 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph-osd.1.log 2024-04-14T15:31:09.311 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph-client.rgw.foo.a.log: gzip -5 --verbose -- /var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph-osd.3.log 2024-04-14T15:31:09.311 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph-mgr.y.log 2024-04-14T15:31:09.312 INFO:teuthology.orchestra.run.smithi092.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-14T15:31:09.312 INFO:teuthology.orchestra.run.smithi092.stderr:: No such file or directory 2024-04-14T15:31:09.312 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph-osd.6.log 2024-04-14T15:31:09.312 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph.audit.log 2024-04-14T15:31:09.312 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph-osd.4.log 2024-04-14T15:31:09.313 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph-osd.7.log 2024-04-14T15:31:09.314 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose --/var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph-osd.4.log: /var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph-osd.5.log 2024-04-14T15:31:09.314 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph-mgr.x.log 2024-04-14T15:31:09.315 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph-osd.0.log 2024-04-14T15:31:09.315 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph-mgr.y.log: /var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph.cephadm.log 2024-04-14T15:31:09.317 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/tcmu-runner.log 2024-04-14T15:31:09.318 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph.cephadm.log 2024-04-14T15:31:09.318 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/tcmu-runner.log: 66.8% -- replaced with /var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/tcmu-runner.log.gz 2024-04-14T15:31:09.318 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph.log 2024-04-14T15:31:09.318 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph.cephadm.log: 81.6% -- replaced with /var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph.cephadm.log.gz 2024-04-14T15:31:09.318 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph-volume.log 2024-04-14T15:31:09.323 INFO:teuthology.orchestra.run.smithi090.stderr: 94.7% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-14T15:31:09.324 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph.log 2024-04-14T15:31:09.325 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph.log: 91.7% -- replaced with /var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph-mgr.x.log.gz 2024-04-14T15:31:09.325 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph-mon.b.log 2024-04-14T15:31:09.326 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph-volume.log: 91.9% -- replaced with /var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph.audit.log.gz 2024-04-14T15:31:09.327 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph.cephadm.log: 89.2% -- replaced with /var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph.cephadm.log.gz 2024-04-14T15:31:09.327 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph-osd.2.log 2024-04-14T15:31:09.328 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph-mon.b.log: 92.0% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-14T15:31:09.328 INFO:teuthology.orchestra.run.smithi092.stderr: 85.8% -- replaced with /var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph.log.gz 2024-04-14T15:31:09.332 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph.log: 95.0% -- replaced with /var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph.audit.log.gz 2024-04-14T15:31:09.332 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph-volume.log 2024-04-14T15:31:09.335 INFO:teuthology.orchestra.run.smithi092.stderr: 90.7% -- replaced with /var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph-volume.log.gz 2024-04-14T15:31:09.342 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph-mon.c.log 2024-04-14T15:31:09.343 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph-volume.log: 91.8% -- replaced with /var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph.log.gz 2024-04-14T15:31:09.357 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph-mon.c.log: 90.8% -- replaced with /var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph-volume.log.gz 2024-04-14T15:31:09.532 INFO:teuthology.orchestra.run.smithi090.stderr: 93.4% -- replaced with /var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph-client.rgw.foo.a.log.gz 2024-04-14T15:31:09.780 INFO:teuthology.orchestra.run.smithi090.stderr: 90.1% -- replaced with /var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph-mgr.y.log.gz 2024-04-14T15:31:10.723 INFO:teuthology.orchestra.run.smithi092.stderr: 92.0% -- replaced with /var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph-mon.b.log.gz 2024-04-14T15:31:10.935 INFO:teuthology.orchestra.run.smithi090.stderr: 91.8% -- replaced with /var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph-mon.c.log.gz 2024-04-14T15:31:12.695 INFO:teuthology.orchestra.run.smithi090.stderr: 91.4% -- replaced with /var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph-mon.a.log.gz 2024-04-14T15:31:15.233 INFO:teuthology.orchestra.run.smithi092.stderr: 93.9% -- replaced with /var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph-osd.6.log.gz 2024-04-14T15:31:16.862 INFO:teuthology.orchestra.run.smithi090.stderr: 94.0% -- replaced with /var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph-osd.3.log.gz 2024-04-14T15:31:17.028 INFO:teuthology.orchestra.run.smithi092.stderr: 94.0% -- replaced with /var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph-osd.5.log.gz 2024-04-14T15:31:17.564 INFO:teuthology.orchestra.run.smithi092.stderr: 94.0% -- replaced with /var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph-osd.4.log.gz 2024-04-14T15:31:17.669 INFO:teuthology.orchestra.run.smithi090.stderr: 94.0% -- replaced with /var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph-osd.2.log.gz 2024-04-14T15:31:17.870 INFO:teuthology.orchestra.run.smithi090.stderr: 94.0% -- replaced with /var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph-osd.1.log.gz 2024-04-14T15:31:18.073 INFO:teuthology.orchestra.run.smithi092.stderr: 94.0% -- replaced with /var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph-osd.7.log.gz 2024-04-14T15:31:18.075 INFO:teuthology.orchestra.run.smithi092.stderr: 2024-04-14T15:31:18.075 INFO:teuthology.orchestra.run.smithi092.stderr:real 0m8.776s 2024-04-14T15:31:18.075 INFO:teuthology.orchestra.run.smithi092.stderr:user 0m30.922s 2024-04-14T15:31:18.075 INFO:teuthology.orchestra.run.smithi092.stderr:sys 0m1.150s 2024-04-14T15:31:18.193 INFO:teuthology.orchestra.run.smithi090.stderr: 94.0% -- replaced with /var/log/ceph/0e288098-fa70-11ee-bc8e-c7b262605968/ceph-osd.0.log.gz 2024-04-14T15:31:18.194 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-04-14T15:31:18.195 INFO:teuthology.orchestra.run.smithi090.stderr:real 0m8.898s 2024-04-14T15:31:18.195 INFO:teuthology.orchestra.run.smithi090.stderr:user 0m37.633s 2024-04-14T15:31:18.195 INFO:teuthology.orchestra.run.smithi090.stderr:sys 0m1.376s 2024-04-14T15:31:18.195 INFO:tasks.cephadm:Archiving logs... 2024-04-14T15:31:18.195 DEBUG:teuthology.misc:Transferring archived files from smithi090:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-08_20:08:14-orch-main-distro-default-smithi/7646059/remote/smithi090/log 2024-04-14T15:31:18.196 DEBUG:teuthology.orchestra.run.smithi090:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-14T15:31:23.292 DEBUG:teuthology.misc:Transferring archived files from smithi092:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-08_20:08:14-orch-main-distro-default-smithi/7646059/remote/smithi092/log 2024-04-14T15:31:23.293 DEBUG:teuthology.orchestra.run.smithi092:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-14T15:31:27.764 INFO:tasks.cephadm:Removing cluster... 2024-04-14T15:31:27.764 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 --force 2024-04-14T15:31:27.956 INFO:teuthology.orchestra.run.smithi090.stdout:Deleting cluster with fsid: 0e288098-fa70-11ee-bc8e-c7b262605968 2024-04-14T15:31:28.257 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 0e288098-fa70-11ee-bc8e-c7b262605968 --force 2024-04-14T15:31:28.450 INFO:teuthology.orchestra.run.smithi092.stdout:Deleting cluster with fsid: 0e288098-fa70-11ee-bc8e-c7b262605968 2024-04-14T15:31:28.745 INFO:tasks.cephadm:Removing cephadm ... 2024-04-14T15:31:28.745 DEBUG:teuthology.orchestra.run.smithi090:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-14T15:31:28.761 DEBUG:teuthology.orchestra.run.smithi092:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-14T15:31:28.776 INFO:tasks.cephadm:Teardown complete 2024-04-14T15:31:28.777 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-14T15:31:28.787 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-14T15:31:28.788 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-14T15:31:28.804 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-14T15:31:28.893 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-14T15:31:28.903 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-14T15:31:28.904 DEBUG:teuthology.orchestra.run.smithi090:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-14T15:31:28.906 DEBUG:teuthology.orchestra.run.smithi092:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-14T15:31:28.917 INFO:teuthology.orchestra.run.smithi090.stderr:bash: line 1: ntpq: command not found 2024-04-14T15:31:28.920 INFO:teuthology.orchestra.run.smithi092.stderr:bash: line 1: ntpq: command not found 2024-04-14T15:31:28.922 INFO:teuthology.orchestra.run.smithi090.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-14T15:31:28.922 INFO:teuthology.orchestra.run.smithi090.stdout:=============================================================================== 2024-04-14T15:31:28.922 INFO:teuthology.orchestra.run.smithi090.stdout:^* hv01.front.sepia.ceph.com 3 6 377 33 +969us[ +971us] +/- 54ms 2024-04-14T15:31:28.922 INFO:teuthology.orchestra.run.smithi090.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 96 -1771us[-1770us] +/- 74ms 2024-04-14T15:31:28.922 INFO:teuthology.orchestra.run.smithi090.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 31 +368us[ +368us] +/- 76ms 2024-04-14T15:31:28.923 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-14T15:31:28.923 INFO:teuthology.orchestra.run.smithi092.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-14T15:31:28.924 INFO:teuthology.orchestra.run.smithi092.stdout:=============================================================================== 2024-04-14T15:31:28.924 INFO:teuthology.orchestra.run.smithi092.stdout:^* hv01.front.sepia.ceph.com 3 6 377 32 +1056us[+1143us] +/- 54ms 2024-04-14T15:31:28.924 INFO:teuthology.orchestra.run.smithi092.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 36 -1636us[-1550us] +/- 74ms 2024-04-14T15:31:28.924 INFO:teuthology.orchestra.run.smithi092.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 34 +468us[ +555us] +/- 76ms 2024-04-14T15:31:28.924 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-14T15:31:28.925 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-14T15:31:28.934 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-14T15:31:28.935 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-14T15:31:28.944 DEBUG:teuthology.orchestra.run.smithi090:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-14T15:31:29.033 DEBUG:teuthology.orchestra.run.smithi092:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-14T15:31:29.076 DEBUG:teuthology.orchestra.run.smithi090:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-14T15:31:29.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:31:29.106 DEBUG:teuthology.orchestra.run.smithi092:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-14T15:31:29.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:31:29.144 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-14T15:31:29.154 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-14T15:31:29.197 INFO:teuthology.task.internal:Duration was 2182.125319 seconds 2024-04-14T15:31:29.197 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-14T15:31:29.208 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-14T15:31:29.208 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-14T15:31:29.211 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-14T15:31:29.251 INFO:teuthology.orchestra.run.smithi090.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-14T15:31:29.253 INFO:teuthology.orchestra.run.smithi092.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-14T15:31:29.654 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-14T15:31:29.654 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi090.front.sepia.ceph.com 2024-04-14T15:31:29.655 DEBUG:teuthology.orchestra.run.smithi090:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-14T15:31:29.674 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi092.front.sepia.ceph.com 2024-04-14T15:31:29.674 DEBUG:teuthology.orchestra.run.smithi092:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-14T15:31:29.721 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-14T15:31:29.721 DEBUG:teuthology.orchestra.run.smithi090:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-14T15:31:29.724 DEBUG:teuthology.orchestra.run.smithi092:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-14T15:31:29.790 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-14T15:31:29.790 DEBUG:teuthology.orchestra.run.smithi090:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-14T15:31:29.793 DEBUG:teuthology.orchestra.run.smithi092:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-14T15:31:32.248 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-14T15:31:32.258 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-14T15:31:32.259 DEBUG:teuthology.orchestra.run.smithi090:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-14T15:31:32.286 DEBUG:teuthology.orchestra.run.smithi092:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-14T15:31:32.318 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-14T15:31:32.329 DEBUG:teuthology.orchestra.run.smithi090:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-14T15:31:32.331 DEBUG:teuthology.orchestra.run.smithi092:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-14T15:31:32.355 INFO:teuthology.orchestra.run.smithi090.stdout:kernel.core_pattern = core 2024-04-14T15:31:32.387 INFO:teuthology.orchestra.run.smithi092.stdout:kernel.core_pattern = core 2024-04-14T15:31:32.439 DEBUG:teuthology.orchestra.run.smithi090:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-14T15:31:32.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:31:32.454 DEBUG:teuthology.orchestra.run.smithi092:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-14T15:31:32.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:31:32.495 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-14T15:31:32.506 INFO:teuthology.task.internal:Transferring archived files... 2024-04-14T15:31:32.507 DEBUG:teuthology.misc:Transferring archived files from smithi090:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-08_20:08:14-orch-main-distro-default-smithi/7646059/remote/smithi090 2024-04-14T15:31:32.507 DEBUG:teuthology.orchestra.run.smithi090:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-14T15:31:32.619 DEBUG:teuthology.misc:Transferring archived files from smithi092:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-08_20:08:14-orch-main-distro-default-smithi/7646059/remote/smithi092 2024-04-14T15:31:32.620 DEBUG:teuthology.orchestra.run.smithi092:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-14T15:31:32.663 INFO:teuthology.task.internal:Removing archive directory... 2024-04-14T15:31:32.663 DEBUG:teuthology.orchestra.run.smithi090:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-14T15:31:32.665 DEBUG:teuthology.orchestra.run.smithi092:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-14T15:31:32.712 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-14T15:31:32.728 INFO:teuthology.task.internal:Not uploading archives. 2024-04-14T15:31:32.728 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-14T15:31:32.738 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-14T15:31:32.738 DEBUG:teuthology.orchestra.run.smithi090:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-14T15:31:32.741 DEBUG:teuthology.orchestra.run.smithi092:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-14T15:31:32.753 INFO:teuthology.orchestra.run.smithi090.stdout: 265318 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 14 15:31 /home/ubuntu/cephtest 2024-04-14T15:31:32.768 INFO:teuthology.orchestra.run.smithi092.stdout: 265306 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 14 15:31 /home/ubuntu/cephtest 2024-04-14T15:31:32.769 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-14T15:31:32.780 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-14T15:31:32.803 INFO:teuthology.nuke:Checking targets against current locks 2024-04-14T15:31:32.830 DEBUG:teuthology.nuke:shortname: smithi090 2024-04-14T15:31:32.830 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-14T15:31:32.852 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi090.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-08_20:08:14-orch-main-distro-default-smithi/7646059', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-14 14:45:31.756234', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF4r8vt0uUbDwufn8Oy7xXkk8cNtSGrn7iFpqL1+81e2O4nTg2MD0Wo8y2GwsCJ7wc/Tey3H3cn9h7qhcBDldRo='} 2024-04-14T15:31:32.856 DEBUG:teuthology.nuke:shortname: smithi092 2024-04-14T15:31:32.856 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-14T15:31:32.880 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi092.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-08_20:08:14-orch-main-distro-default-smithi/7646059', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-14 14:45:31.758510', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLpstLYsuLzhOW3itcBt92l/uZdJxnpz1kSmx0N5abFELPL7u07DE0DV9prrJGKdyyTPwim2WmYIu/WHGK4FB/U='} 2024-04-14T15:31:32.910 INFO:teuthology.orchestra.console.smithi090:Power off 2024-04-14T15:31:32.911 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-14T15:31:32.936 DEBUG:teuthology.orchestra.console.smithi090:power off output: Chassis Power Control: Down/Off 2024-04-14T15:31:32.937 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-14T15:31:32.953 INFO:teuthology.orchestra.console.smithi092:Power off 2024-04-14T15:31:32.953 DEBUG:teuthology.orchestra.console.smithi092:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-14T15:31:32.963 DEBUG:teuthology.orchestra.console.smithi090:check power output: Chassis Power is on 2024-04-14T15:31:32.977 DEBUG:teuthology.orchestra.console.smithi092:power off output: Chassis Power Control: Down/Off 2024-04-14T15:31:32.977 DEBUG:teuthology.orchestra.console.smithi092:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-14T15:31:33.005 DEBUG:teuthology.orchestra.console.smithi092:check power output: Chassis Power is on 2024-04-14T15:31:36.963 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-14T15:31:37.005 DEBUG:teuthology.orchestra.console.smithi092:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-14T15:31:37.074 DEBUG:teuthology.orchestra.console.smithi090:check power output: Chassis Power is on 2024-04-14T15:31:37.116 DEBUG:teuthology.orchestra.console.smithi092:check power output: Chassis Power is on 2024-04-14T15:31:41.078 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-14T15:31:41.117 DEBUG:teuthology.orchestra.console.smithi092:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-14T15:31:41.191 DEBUG:teuthology.orchestra.console.smithi090:check power output: Chassis Power is off 2024-04-14T15:31:41.229 DEBUG:teuthology.orchestra.console.smithi092:check power output: Chassis Power is off 2024-04-14T15:31:41.292 INFO:teuthology.orchestra.console.smithi090:Power off completed 2024-04-14T15:31:41.330 INFO:teuthology.orchestra.console.smithi092:Power off completed 2024-04-14T15:31:41.434 INFO:teuthology.run:Summary data: description: orch/cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v1only root} duration: 2182.1253185272217 failure_reason: '"2024-04-14T15:11:49.180327+0000 mon.a (mon.0) 865 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS)" in cluster log' flavor: default owner: scheduled_teuthology@teuthology success: false 2024-04-14T15:31:41.435 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-14T15:31:41.523 INFO:teuthology.run:FAIL